Skip to content

Storage IOPS Performance Benchmarks

Testing Methodology

The random read and write IOPS are measured in a VM running Linux or Windows. The IOPS measured in the VM indicates the maximum IOPS that could be consumed by the workloads running in the same VM. The measured IOPS is compared with the performance numbers published by the vendor of physical NVMe drive to see how much raw performance the VM could consume.

Testing Tool

FIO

Supported OS: Linux and Windows

FIO is an industry standard benchmark for measuring disk I/O performance. Fio spawns a number of threads or processes doing a particular type of I/O action as specified by the user. We include it here as a baseline measurement on all infrastructure to compare raw disk performance that any application can potentially achieve.

Test Setup

FIO - Ubuntu 18.04

  • Create a Ubuntu VM with three disks attached, where one disk is the boot drive and the other two disks would be used for storage performance test. A tutorial for the VM setup on the Sunlight platform is available here.
  • Prepare the VM to have proper irqbalance setting. (A script is available here).
  • Download the FIO tool (a binary executive file is available for download here).
  • Provide an FIO configuration file to specify the relevant parameters, including the disk to test, I/O action (rw=randread for random read or rw=randwrite for random write), block size and iodepth. An example for Ubuntu to test the 2nd and 3rd drives simultaneously is provided below and also available for download here.
[global]
norandommap
gtod_reduce=1
group_reporting=1
ioengine=libaio
time_based=1
rw=randread
bs=4k
direct=1
ramp_time=5
runtime=30
iodepth_batch=32
iodepth_batch_complete=8
iodepth_low=16
iodepth=32
loops=3

[job1]
filename=/dev/xvdb

[job2]
filename=/dev/xvdc
  • Run the FIO test with a configuration file and the numjobs parameter specifying the number the cores used for the test.
sudo ./fio-3.2 randread.fio --numjobs=16

The same fio configuration file is used when comparing a VM running on different environment. Make sure to change the configuration file to access the correct block device name that you wish to test, e.g. /dev/sda or /dev/xvda.

FIO - Windows Server 2016

  • Create a Windows VM with three disks attached, where one disk is the boot drive and the other two disks would be used for storage performance test. A tutorial for the VM setup on the Sunlight platform is available here.
  • Download the FIO tool (an Windows installer is available for download from Sunlight or a third party.
  • Run the installer to install the FIO program to the Windows VM.
  • Provide an FIO configuration file to specify the relevant parameters, including the disk to test, I/O action (rw=randread for random read or rw=randwrite for random write), block size and iodepth. An example for Windows to test the 2nd and 3rd drives simultaneously is provided below and also available for download here.
[global]
norandommap
gtod_reduce=1
group_reporting=1
ioengine=windowsaio
time_based=1
rw=randread
bs=4k
direct=1
ramp_time=5
runtime=30
iodepth_batch=32
iodepth_batch_complete=8
iodepth_low=16
iodepth=32
loops=3
thread=1

[job1]
filename=\\.\PhysicalDrive1

[job2]
filename=\\.\PhysicalDrive2
  • Run the FIO test with a configuration file and the numjobs parameter specifying the number the cores used for the test.
fio win_randread.fio --numjobs=4

Results

FIO - IOPS (4K) from different operating systems on Sunlight

Platform OS Number of cores Memory size (GB) Disk size (GB) Physical NVMe drive Random Read IOPS (k) Random IOPS (k)
Sunlight on premise (Intel BP) Ubuntu 18.04 16 64 2 x 80 Intel P4610 1.6TB 750 700
Sunlight on premise (Intel BP) Windows Server 2016 16 64 2 x 80 Intel P4610 1.6TB 90 100

FIO - IOPS (4K) from Ubuntu on different platforms

Flavour Read IOPs Write IOPs
AWS r5.2xlarge + 50K reserved iops 18.7
Sunlight AWS r5.2xlarge 428
Sunlight r5.2xlarge 651

FIO performance IOPs comparison