s8y

fast drive array

Recommended Posts

Need about 30TB of fast storage for dev cluster (8 nodes).

Expected load: databases, reports, aggregation, possibly hpc, zfs, dedup ocsional. Entire data refresh every 2 weeks.

Have dell server (2x xeon I know amd would be better) chase with 13pcie slots and 16sata ports.

Theoretical I can fill 12 pcie slots with 4x m.2 expansion cards like this one

and load them with 1-2TB nvme m.2 cards.

That would provide 48TB of raw space. I potentially could run it as 2 tier nvme and sata storage.

My storage has potential to saturate 10GBs network connectivity so will probably need to NVMEoF (nvme over fabric) on 40BGs infiniband (RDMA).

Most likely will need to use software (os level) raid.

What os would you recommend to install on the host?

Has it got chance to work as fast starage?

Has anyone done something similar and would like to share?

Any suggestion, advice much appreciated.

I understand I need to check if all my pcie slots are all x16 (each m.2 will take 4pci lines).

Verify cards/drives are on hardware/os compatibility list.

Check board PSU can deliver enough power.

Check there will ne enough space between pcie slots and m.2 drives do not overheat.

There is substantial price tag associated with the project so would be nice to know its going to work.

Share this post


Link to post
Share on other sites

Uhhhhh what's your budget? At this level of spending, especially if you have an existing Dell hardware base, you can probably call up your Dell rep and see what they recommend.

Share this post


Link to post
Share on other sites

What server are you using specifically?

DIY storage at the level you are talking about can go south very quickly.

Are the hosts Linux, VMware, or Microsoft? Each one would have a different storage in mind.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now