Jump to content


Photo

EchoStreams FlacheSAN2 Custom Flash Array Build Discussion


  • You cannot start a new topic
  • Please log in to reply
36 replies to this topic

#1 Kevin OBrien

Kevin OBrien

    StorageReview Editor

  • Admin
  • 1,412 posts

Posted 13 July 2012 - 02:30 PM

StorageReview has started testing the FlacheSAN2 48-bay 2U storage array from EchoStreams. This platform features 48 densely packed 180GB Intel SSD 520s, five PCIe 3.0 LSI 9207-8i HBAs, three Mellanox ConnectX-3 56Gb/s dual-port InfiniBand adapters and is built around a custom two-processor Intel motherboard. Follow this log for the ongoing build process and benchmark results.

EchoStreams FlacheSAN2 Custom Flash Array Build

#2 Brian

Brian

    SR Admin

  • Admin
  • 5,158 posts

Posted 13 July 2012 - 06:43 PM

We're thrilled to have this rig in the office as part of our expanding array testing. If you have questions or requests please post them here.

Brian

Publisher- StorageReview.com
Twitter - @StorageReview

 

#3 kristofferjon

kristofferjon

    Member

  • Member
  • 13 posts

Posted 14 July 2012 - 08:23 PM

We're thrilled to have this rig in the office as part of our expanding array testing. If you have questions or requests please post them here.


Looking forward to seeing the benchmark results. Is it possible for us to purchase the chassis ourselves?

#4 Kevin OBrien

Kevin OBrien

    StorageReview Editor

  • Admin
  • 1,412 posts

Posted 15 July 2012 - 12:11 AM

I believe so, this is the datasheet for the platform:

http://www.echostrea...Sheet_Rev02.pdf

#5 Brian

Brian

    SR Admin

  • Admin
  • 5,158 posts

Posted 15 July 2012 - 04:10 PM

Looking forward to seeing the benchmark results. Is it possible for us to purchase the chassis ourselves?


Also happy to connect you directly if you want to PM me your contact info.

Brian

Publisher- StorageReview.com
Twitter - @StorageReview

 

#6 lecaf

lecaf

    Member

  • Member
  • 13 posts

Posted 16 July 2012 - 10:23 AM

I'd like to see if Shogun2 Total War loading times are faster using this array.

lol

In fact what you could do are some ridiculous comparisons, of this ludicrous array against a single sata mech HDD. Some ideas: Windows install time on a VM, data copy time.

Sure some business related benchmarks of journaled database would be more serious and professorial looking, but...I already know what the results would be. What I like to see is the fun angle.

m a r c


We're thrilled to have this rig in the office as part of our expanding array testing. If you have questions or requests please post them here.


Edited by lecaf, 16 July 2012 - 10:24 AM.

#7 FastMHz

FastMHz

    Member

  • Member
  • 403 posts

Posted 16 July 2012 - 11:53 AM

Fun things are always good :-)

I installed Win7 onto a RAMDisk via VirtualBox once and it finished the copying in 17 seconds.

Production: Vishera 8350/32gb RAM/Dual SSD/VelociRaptor/Radeon 7750
Gaming: Phenom II 955/16gb RAM/SSD/VelociRaptor/Radeon 7950
Retro: K6-2 550/256mb RAM/160gb HDD/CompactFlash/3DFX/ATI AIW Pro/SB16/DB50XG
http://www.fastmhz.com

#8 kristofferjon

kristofferjon

    Member

  • Member
  • 13 posts

Posted 27 July 2012 - 09:45 AM

Are there any updates available on the build progress & testing?

#9 Kevin OBrien

Kevin OBrien

    StorageReview Editor

  • Admin
  • 1,412 posts

Posted 27 July 2012 - 10:06 AM

About to add some updates today. This past week we received all of our InfiniBand networking gear and are prepping HP DL380p servers with Windows Server 2012 for testing. Looking to get the InfiniBand side going today for some baseline tests.

#10 ehorn

ehorn

    Member

  • Member
  • 17 posts

Posted 27 July 2012 - 10:49 AM

Thanks for the update Kevin. Looking forward to it...

peace,

Edited by ehorn, 27 July 2012 - 10:50 AM.

#11 kristofferjon

kristofferjon

    Member

  • Member
  • 13 posts

Posted 27 July 2012 - 08:56 PM

About to add some updates today. This past week we received all of our InfiniBand networking gear and are prepping HP DL380p servers with Windows Server 2012 for testing. Looking to get the InfiniBand side going today for some baseline tests.


Not sure whether you've used InfiniBand before, but make sure you setup your subnet manager correctly to ensure the network comes up. You'll need one subnet manager per IB fabric.

Can you also test with 10-gigabit ethernet. I understand that it's going to choke the available bandwidth but I'm more interested in IOPS. I imagine with small 4K IOPS 10-gigabit should still be reasonable, especially a bonded pair.

One other question, what OS or OS's are you planning on using for the testing? Would it be possible for you to test with OpenIndiana build 151a and/or Solaris 11 to verify compatibility with your hardware stack?

Edited by kristofferjon, 27 July 2012 - 09:02 PM.

#12 Brian

Brian

    SR Admin

  • Admin
  • 5,158 posts

Posted 31 July 2012 - 06:51 AM

We're working now with Windows Server 2012 with SMB 3.0, but we'll run some other tests on alternate OSs as well.

Brian

Publisher- StorageReview.com
Twitter - @StorageReview

 

#13 kristofferjon

kristofferjon

    Member

  • Member
  • 13 posts

Posted 08 August 2012 - 07:00 AM

We're working now with Windows Server 2012 with SMB 3.0, but we'll run some other tests on alternate OSs as well.


Any updates?

#14 Kevin OBrien

Kevin OBrien

    StorageReview Editor

  • Admin
  • 1,412 posts

Posted 09 August 2012 - 09:48 AM

Currently we are working out some firmware/bios quirks with our NICs in PCIe 3.0 mode. Once that gets smoothed out we will start posting numbers from the array B)

#15 Kevin OBrien

Kevin OBrien

    StorageReview Editor

  • Admin
  • 1,412 posts

Posted 09 August 2012 - 04:26 PM

New firmware applied to our IB cards looks to have fixed an incompatibility problem we were seeing with the servers. 10 of them were flashed to the new software, to give you an idea of the scale of that project B)

Tomorrow we'll dedicate one of the servers to kick off that project to get our IB fabric setup (other servers are cranking away on other tests right now).

#16 kristofferjon

kristofferjon

    Member

  • Member
  • 13 posts

Posted 09 August 2012 - 04:37 PM

New firmware applied to our IB cards looks to have fixed an incompatibility problem we were seeing with the servers. 10 of them were flashed to the new software, to give you an idea of the scale of that project B)

Tomorrow we'll dedicate one of the servers to kick off that project to get our IB fabric setup (other servers are cranking away on other tests right now).


Sounds big, looking forward to seeing it.

#17 ehorn

ehorn

    Member

  • Member
  • 17 posts

Posted 10 August 2012 - 08:17 PM

New firmware applied to our IB cards looks to have fixed an incompatibility problem we were seeing with the servers. 10 of them were flashed to the new software, to give you an idea of the scale of that project B)

Tomorrow we'll dedicate one of the servers to kick off that project to get our IB fabric setup (other servers are cranking away on other tests right now).



Oh yeah....

http://www.storagere...-InfiniBand.jpg

http://www.storagere...heSAN2-Rear.jpg

B)

#18 L..

L..

    Member

  • Member
  • 1 posts

Posted 28 August 2012 - 01:32 AM

Can you also test with 10-gigabit ethernet. I understand that it's going to choke the available bandwidth but I'm more interested in IOPS. I imagine with small 4K IOPS 10-gigabit should still be reasonable, especially a bonded pair.

Let's say you expect 2 MIOPS @4K burst like they do (which you easily get on RAM alone in local anyway), you have the following:
2.000.000 * 4.096 * 8 bits/second = 64Gbit/s.
Considering overhead, or better performance or other unexpected factors... I would say link aggregation of those 6 qsfp ports in 10GbE mode could fall short.

On the other hand, using 512b you would know the 4K random performance (they're equal) and hold in one 10GbE link - but then that wouldn't exactly show how great the Mellanox kit and the echostreams barebone are.

Edited by L.., 28 August 2012 - 01:33 AM.

#19 ehorn

ehorn

    Member

  • Member
  • 17 posts

Posted 31 August 2012 - 04:48 PM

"The top speed with an outstanding I/O figure of 32, we hit 2,045,787 IOPS 4k read and 1,798,432 IOPS 4k write with an outstanding I/O figure of 4"

How fun is that? Really impressive results Kevin! and all from a single 2U chassis...

Also, can you cast some light on the test configuration (i.e. workers, managers, test duration, etc...)?

I assume these drives are 'steady state'? Still planning on over-provisioning the drives? Any thoughts on how you plan to organize the array?

Thanks for sharing the results. Very impressive IOPS.

peace,

Edited by ehorn, 31 August 2012 - 05:08 PM.

#20 compwizrd

compwizrd

    Member

  • Member
  • 411 posts

Posted 31 August 2012 - 06:07 PM

Impressive numbers, and even more impressive that you'd void the warranty on $10k of 520's :)

#21 Kevin OBrien

Kevin OBrien

    StorageReview Editor

  • Admin
  • 1,412 posts

Posted 31 August 2012 - 09:02 PM

Actually the numbers posted today are sustained, not steady-state. We are gearing this platform towards read-intensive workloads so most of the stuff today was getting our baseline figures to make sure all the equipment is functional without errors. Nothing like bringing the CPU up 99.70% utilization ;)

#22 nickphx

nickphx

    Member

  • Member
  • 5 posts

Posted 05 September 2012 - 12:33 AM

Hi,
What software are you running on the server? Are you able to provide the retail cost for the enclosure without the disks?

thanks,
nick

#23 Brian

Brian

    SR Admin

  • Admin
  • 5,158 posts

Posted 05 September 2012 - 08:31 AM

Not sure if it's generally available yet but we'll find out what pricing is expected to look like at least.

Brian

Publisher- StorageReview.com
Twitter - @StorageReview

 

#24 Kevin OBrien

Kevin OBrien

    StorageReview Editor

  • Admin
  • 1,412 posts

Posted 05 September 2012 - 09:10 AM

To answer the software question, currently the SAN is running Windows Server 2012 Standard early-release... which we will be wiping to start fresh with the RTM that came out yesterday.

#25 nickphx

nickphx

    Member

  • Member
  • 5 posts

Posted 05 September 2012 - 06:31 PM

I managed to get a response from echostreams and rackmount. They both have the bare bones w/motherboard listed for $3899.



0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users