Update (5/2): This was resolved. Adaptec worked with me directly on this issue. The issue was confirmed, identified and a fix was found. The fixed firmware will be released in the next official release which is promised to be delivered later this month. For now I recommend you don’t use builds 30612<x<=30862 of the firmware. Glad to be able to help squash a bug and looking forward to testing the new version.
A few weeks ago we got a batch of new servers at ClickTale and started doing acceptance testing on them. Surprisingly enough, those sweet dual deca-core servers failed to pass the IO acceptance tests. I have posted on the subject of IO benchmarking in the past and for those of you who don’t know already I take performance measuring very seriously.
The problematic measurements were sequential reads as measured by CrystalDiskMark. It seems, on new servers, when controller (read) cache was enabled on the logical device, sequential read speeds were lower than usual and, for many Simple Volume and Raid1 devices, capped at around 100MB. Also, Raid1 devices that are supposed to provide improved read performance vs just a single simple disk were not providing such improvements.
Luckily we had other systems with comparable hardware and it was clear to us that the results are not ok. Where a similar setup provided 1000MB/s sequential read, the new systems only provided 100MB/s or 480MB/s depending on configuration. We started working with our hosting provider to identify the issue. It was very hard to convince our provider that something is wrong but they helped us by changing components in the system per our requirements.
We compared the good and bad systems and identified the following differences:
- Windows server 2008 vs 2012.
- SAS Expander vs SAS passive backplane.
- Adaptec RAID controller driver version 7.3 vs 7.4.
- Adaptec RAID controller BIOS and firmware versions 7.3 vs 7.4.
- Adaptec RAID management software, MaxView.
- Disk model (specifically SSDs as there is more variety there).
- Adaptec RAID controller model 71605 vs 72405.
It took us two weeks to experiment with different settings until we discovered that it was the BIOS/firmware that is responsible for low sequential read IO. Apparently something has been changed in firmware 7.4.0 Build 30862 of the controller that resulted in reduced performance of some logical volumes with read cache. It seems to mainly affect SSD volumes. It is not clear whether it is a bug or a deliberate change. We have downgraded to 7.3 and are now waiting for a response from Adaptec.
Meanwhile, here are test results using “Smart XceedIOPS2” 400GB SSD disks in Raid1 configuration to demonstrate the issue:
All tests are:
Test : 4000 MB [D: 0.0% (0.1/372.0 GB)] (x3) OS : Windows Server 2008 R2 Enterprise SP1 [6.1 Build 7601] (x64) Controller : Adaptec 71605
FW 7.4 cache disabled:
Sequential Read : 494.436 MB/s Sequential Write : 123.387 MB/s Random Read 512KB : 459.764 MB/s Random Write 512KB : 137.925 MB/s Random Read 4KB (QD=1) : 30.472 MB/s [ 7439.4 IOPS] Random Write 4KB (QD=1) : 13.029 MB/s [ 3180.9 IOPS] Random Read 4KB (QD=32) : 182.967 MB/s [ 44669.6 IOPS] Random Write 4KB (QD=32) : 60.195 MB/s [ 14696.0 IOPS]
FW 7.4 cache enabled
Sequential Read : 112.207 MB/s Sequential Write : 240.485 MB/s Random Read 512KB : 496.336 MB/s Random Write 512KB : 228.503 MB/s Random Read 4KB (QD=1) : 25.457 MB/s [ 6215.1 IOPS] Random Write 4KB (QD=1) : 56.888 MB/s [ 13888.7 IOPS] Random Read 4KB (QD=32) : 188.890 MB/s [ 46115.8 IOPS] Random Write 4KB (QD=32) : 58.611 MB/s [ 14309.3 IOPS]
FW 7.3 cache disabled
Sequential Read : 480.612 MB/s Sequential Write : 122.601 MB/s Random Read 512KB : 435.159 MB/s Random Write 512KB : 131.029 MB/s Random Read 4KB (QD=1) : 20.163 MB/s [ 4922.7 IOPS] Random Write 4KB (QD=1) : 11.767 MB/s [ 2872.8 IOPS] Random Read 4KB (QD=32) : 181.641 MB/s [ 44345.9 IOPS] Random Write 4KB (QD=32) : 62.440 MB/s [ 15244.0 IOPS]
FW 7.3 cache enabled
Sequential Read : 1019.892 MB/s Sequential Write : 203.124 MB/s Random Read 512KB : 469.486 MB/s Random Write 512KB : 235.107 MB/s Random Read 4KB (QD=1) : 22.790 MB/s [ 5564.1 IOPS] Random Write 4KB (QD=1) : 59.995 MB/s [ 14647.1 IOPS] Random Read 4KB (QD=32) : 241.114 MB/s [ 58865.8 IOPS] Random Write 4KB (QD=32) : 57.243 MB/s [ 13975.4 IOPS]
Decide for yourself!