Peter Enoch

VRTX SSD Performance, what to expect?

Recommended Posts

Hi everyone, 

I've just bought a Dell PowerEdge VRTX system 3 weeks ago with 25xToshiba PX04SVB096 harddrives. The system have two PERC8 Shared controllers installed and can be setup in Single or Fault Tolerance Mode. Best results currently is with Single Mode controller, Fault Tolerance Mode has a performance drop (even with "new" firmware)

The SSD's should be excellent and it's possible to read more about them here: http://www.tomsitpro.com/articles/toshiba-px04s-sas-ssd,2-905.html?_ga=1.192722346.1078210542.1470492245

The Dell PowerEdge VRTX system is brand new and all firmware is up to date. I've one Blade in the VRTX system running on local SSD's which have Windows 2012r2 installed (also tried with VMware with worse result)

Dell recommends to set Read Cache Policy to No Read Ahead and it seems to give best result also. Write Cache Policy to Write Through. I see better performance with Write Back Policy!

Using I/O Meter directly on the physical server i've done a lot of tests with these setting:
0,5K, 16 Outstanding I/O, 100% Read, 100% Random
0,5K, 16 Outstanding I/O, 100% Write, 100% Random

VRTX in Single Mode with 24 drives in RAID-50: Read: 79.736 IOPS and Write: 71.494 IOPS
VRTX in Fault Tolerance Mode with 24 drives in RAID-50: Read: 43.779 IOPS and Write: 41.580 IOPS

Dell PowerEdge 730XD with PERC H830 RAID controller with 2x Dell Mix-Use SSD's (other cheaper brand then the Toshiba's) in RAID-1: Read: 102.046 IOPS and Write: 107.691 IOPS

On my own home PC with an "older" Samsung 850 EVO Desktop SSD I get in Read: 82.617 IOPS and Write: 75.317 IOPS

I don't think i'm getting the result I should expect from this system, I mean should one Desktop SSD almost beat 24xEnterprise SSD's in RAID-50 or 2xDell SanDisk SSD's in RAID-1 outperform 24xSSD's in RAID-50.

Currently have a case open at Dell, but here 3 weeks later we're not getting better results.

I need to use this VRTX solution with 4xVMware vSphere servers to run a lot of VM's, so I need to be 100% sure before moving anything to this system.

can everyone share what they think or what to test out?

Hope to hear from anyone..

Share this post


Link to post
Share on other sites

I do agree that the results are less than spectacular, although the workload profile may be part of it. Why are you measuring 512-byte transfers and why aren't you using additional threads/workers? I'd probably do a split with 16 workers and 16 queue each, and look at 4k and 128k transfer sizes.

An individual SSD can generally handle very low queue depth without a problem, but as soon as you have multiples in RAID, they really like to see larger workloads.

Share this post


Link to post
Share on other sites

The test is done because Dell told me to do with these settings.

The 4K test have already been done, but I've now testet with 16 workers and 16 queue each with 4K and 128K transfer sizes.

The result for 4K with 100% random:
Home PC with Samsung EVO 850 SSD Read 100%: 55.231 IOPS
Home PC with Samsung EVO 850 SSD Write 100% 46.336 IOPS
VRTX with RAID-6 over 24 SSD's Read 100%: 45.073 IOPS
VRTX with RAID-6 over 24 SSD's Write 100%: 43.273 IOPS

The result for 128K with 100% random:
Home PC with Samsung EVO 850 SSD Read 100%: 4.324 IOPS
Home PC with Samsung EVO 850 SSD Write 100% 1.909 IOPS
VRTX with RAID-6 over 24 SSD's Read 100%: 14.224 IOPS
VRTX with RAID-6 over 24 SSD's Write 100%: 14.369 IOPS

Currently VRTX is in single controller mode and with No Read Ahead Read Cache Policy and Write Back in Write Cache Policy.

In the 128K with 100% random I see differents, but again 24x Enterprise SSD's vs. one single Desktop SSD.

 

 

Share this post


Link to post
Share on other sites

And for information same test on 2xSanDisk Dell SSD's in PowerEdge 730XD in RAID-1:

The result for 4K with 100% random:
Read 100%: 132.021 IOPS
Write 100%: 133.699 IOPS

The result for 128K with 100% random:
Read 100%: 43.820 IOPS
Write 100% 53.136 IOPS
 

 

Share this post


Link to post
Share on other sites

How long are you performing these tests for? Have they been preconditioned in any way?

Share this post


Link to post
Share on other sites

On the random 4K tests, you generally want write-through enabled and no read ahead. On the large-block transfers, write-back enabled and no read ahead will generally give better performance.

Still not sure what the overall objective is though. Are you using that desktop SSD in the server? Not sure what the value of those test results will be. Completely different ballgame. What is the production workload for this server? Database, VDI? Was this exact configuration chosen for a specific reason over other shared storage designs? There will be some limitations, especially with a lot of flash, with a shared PERC card leveraged for hosting flash to 4 servers. Current RAID interfaces haven't advanced as quickly as modern SAS3 SSDs, so you won't see anywhere near as much performance as you might by bypassing the RAID card with an HBA, or going with NVMe flash dedicated to each server.

Share this post


Link to post
Share on other sites

I should add that the shared nature of that PERC8 card may end up not fully saturating with the workload you are running. Do you have one LUN presented to one server? Or are you testing with four LUNs presented to four servers, each running that workload?

Either way though, that setup was never really intended to run an all-flash configuration. Sure it can "do" it, but it is much more geared to a ROBO environment with a little bit of flash and a lot of spinning storage.

Share this post


Link to post
Share on other sites

For test I only have one LUN and this is only connected to one server which don't have any load.

And for information same test on 2xSanDisk Dell SSD's in PowerEdge 730XD in RAID-1:

The result for 4K with 100% random:
Read 100%: 132.021 IOPS
Write 100%: 133.699 IOPS

The result for 128K with 100% random:
Read 100%: 43.820 IOPS
Write 100% 53.136 IOPS

Share this post


Link to post
Share on other sites
42 minutes ago, Kevin OBrien said:

On the random 4K tests, you generally want write-through enabled and no read ahead. On the large-block transfers, write-back enabled and no read ahead will generally give better performance.

Still not sure what the overall objective is though. Are you using that desktop SSD in the server? Not sure what the value of those test results will be. Completely different ballgame. What is the production workload for this server? Database, VDI? Was this exact configuration chosen for a specific reason over other shared storage designs? There will be some limitations, especially with a lot of flash, with a shared PERC card leveraged for hosting flash to 4 servers. Current RAID interfaces haven't advanced as quickly as modern SAS3 SSDs, so you won't see anywhere near as much performance as you might by bypassing the RAID card with an HBA, or going with NVMe flash dedicated to each server.

I've tested Write-Back vs. Write Through lots of times and I havn't one time got better Write performance results with Write Through, only Read will sometimes be faster, maybe because the controller supported FastPath.

The Desktop SSD is in my own Dell PC so the test is from that system.

This system was chosen to run VMware vSphere VM's and the idea was to have more VRTX systems so we where not so valuable for disaster - one box could be done but other VRTX systems was still in production. We currently have M1000e Blade Centers fra Dell and Equallogic SAN's, but we "fear" if the M1000e goes down everything is stopped. If one Equallogic SAN in a group with data spread over more members, all data on these volumes is down.

Currently the one server connected in the VRTX is running physical Windows 2012r2 server.

 

Share this post


Link to post
Share on other sites
8 minutes ago, Peter Enoch said:

For test I only have one LUN and this is only connected to one server which don't have any load.

And for information same test on 2xSanDisk Dell SSD's in PowerEdge 730XD in RAID-1:

The result for 4K with 100% random:
Read 100%: 132.021 IOPS
Write 100%: 133.699 IOPS

The result for 128K with 100% random:
Read 100%: 43.820 IOPS
Write 100% 53.136 IOPS

For information the RAID controller in the Dell PowerEdge 730XD is a PERC H830

Maybe it's true that we should not expect high performance from the VRTX, but I would have expected Dell to say something before we said OK to buy and it's pretty hard to explain why 2xDell SSD's (which i'm sure is not as good as the Toshiba SSD's) in and PowerEdge 730XD currently outperforms the VRTX system with 24xSSD with 4x.

The PowerEdge 730XD with PERC H830 I would not call Enterprise either..

Share this post


Link to post
Share on other sites

The question mark in all of this is the shared architecture of the VRTX PERC8 card. In the R730XD you have a native connection from the server to the RAID card. In the VRTX its split between the four nodes. That might be chopping the performance, even if its only allocated to a single server.

Share this post


Link to post
Share on other sites

I'm sure your right, it's just sad that the controller in the VRTX isn't better then what we see :angry:

I could live with performance under high end systems, but it's hard to see 2xSSD's in RAID-1 beat 24xEnterprise SSD's in the VRTX :(

Even more when the controllers is in Fault Tolerance mode it's worse..

Edited by Peter Enoch

Share this post


Link to post
Share on other sites
4 hours ago, Kevin OBrien said:

How long are you performing these tests for? Have they been preconditioned in any way?

I've tried up to 15 min.

I'm unsure what you mean about preconditioned?

Share this post


Link to post
Share on other sites

For tests such as these, you'd want the workload to run in excess of 6-12 hours. Many enterprise SSDs by nature of the beast perform differently if tests in an unclean manner. So the only way to get a reliable performance metric would be a very long testing period.

Share this post


Link to post
Share on other sites
1 hour ago, Kevin OBrien said:

For tests such as these, you'd want the workload to run in excess of 6-12 hours. Many enterprise SSDs by nature of the beast perform differently if tests in an unclean manner. So the only way to get a reliable performance metric would be a very long testing period.

But for a system which only runs an IO Meter test to find out what the system can offer, how should very long test really do any different?

For me in the tests I've done, it's very steady from the start and dosn't really changes over time.

In the test of the Toshiba SSD from http://www.tomsitpro.com/articles/toshiba-px04s-sas-ssd,2-905-3.html it also seems to be very steady.

 

 

Share this post


Link to post
Share on other sites

I suppose you can debate testing methodologies if you like, but what's the end goal here? What's your capacity footprint need? Wondering if AIC NVMe would be better for you, but hard to tell without knowing more. 

Share this post


Link to post
Share on other sites

The goal is to run abort 100 medium load VM's on this system. But i'm a little bit worried it Will not handle the load.

still dosn't understand that it seems to underperform so much.

Share this post


Link to post
Share on other sites

Its hard to say. You have drives operating in an un-fresh out of box state where you aren't running steady-state workloads.

Share this post


Link to post
Share on other sites
10 hours ago, Kevin OBrien said:

Its hard to say. You have drives operating in an un-fresh out of box state where you aren't running steady-state workloads.

Correct, but how to test differently?

I can't move all the workload before i'm pretty sure it will perform fast. I've testet with 3 VM's and it was running OK, but not lightning fast which we expected.

I know test isn't real world, but still if i'm already on 2xSSD's on PE730XD sees 4x performance then I don't think we should be satisfied with the current numbers we see from the VRTX system. Again remember that Fault Tolerance mode on the VRTX decreases performance even more then what I've wrote.

I hope to test 2xToshiba SSD's in the PE730XD, just to see if they outperforms the current 2xDell SSD's (SanDisk) in the PE730XD

Share this post


Link to post
Share on other sites

Its as simple as running the workload for a longer period of time. All of our tests run 6-12 hours.

Those PX04 SSDs are fast, right now the fastest on the market. Your bottleneck is that shared storage platform. I've never tested it to measure overhead in that configuration, but I know as a single controller in one server, its very easy to top it out well before the SSDs themselves are maxed.

To get high performance in the VRTX, you'd be better off focusing on NVMe storage added through the mappable PCIe slots in the rear of the chassis.

Share this post


Link to post
Share on other sites
1 hour ago, Kevin OBrien said:

Its as simple as running the workload for a longer period of time. All of our tests run 6-12 hours.

Those PX04 SSDs are fast, right now the fastest on the market. Your bottleneck is that shared storage platform. I've never tested it to measure overhead in that configuration, but I know as a single controller in one server, its very easy to top it out well before the SSDs themselves are maxed.

To get high performance in the VRTX, you'd be better off focusing on NVMe storage added through the mappable PCIe slots in the rear of the chassis.

Hi Kevin, I can do that, but the tests I see from other reviews almost shows the same and that is that the performance over longer time stays very steady or is there something I miss?

I think your right about the NVMe storage, but we didn't look that way because that will mean "fixed" storage for each Blade (if one PCIe is mapped to each server)

Just sad to see that this system can't handle SSD or high load.

Share this post


Link to post
Share on other sites

All reviewers will wipe the drives with a secure erase and then precondition the drives for a given workload. The factory drives you received most likely are not in that condition.

This test ran for 6 hours, leading up to our actual benchmark of the drive:

seagate_1200_2_1_6tb_sas3_preconditionin

Share this post


Link to post
Share on other sites
1 hour ago, Kevin OBrien said:

All reviewers will wipe the drives with a secure erase and then precondition the drives for a given workload. The factory drives you received most likely are not in that condition.

This test ran for 6 hours, leading up to our actual benchmark of the drive:

seagate_1200_2_1_6tb_sas3_preconditionin

Your right, I think i'll see the same for the 24xSSD in the VRTX vs. the PE730XD with 2xSSD, but i'll test it.

But i'll be surprised if the VRTX test showed big increase in IOPS :mellow:

Share this post


Link to post
Share on other sites

That may be the case, but even so, VRTX wouldn't be my first choice for a IOPS-centric all flash deployment. 

Share this post


Link to post
Share on other sites

4K 100% Write and 100% random 16 Workers and 16 Disk Queue Lengh on the VRTX storage for 3 hours: 41.290 IOPS (24xDell Toshiba SSD's in RAID-6)

4K 100% Write and 100% random 16 Workers and 16 Disk Queue Lengh on the PowerEdge 730XD for 3 hours: 132.953 IOPS (2xDell SanDisk SSD's in RAID-1)

So over 3x better Write performance. I know RAID-6 have worse Write performance, but shouldn't 24x Enterprise drives still beat this?

Again I think when I've the time I will drive to our datacenter and take 2xToshiba SSD's to try in the PE730XD system, I think the result will be even better than with the SanDisk drives from Dell.

I'm testing the same for read performance now, here RAID-6 should normally perform better. I'll be back :D

 

 

Edited by Peter Enoch

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now