Need about 30TB of fast storage for dev cluster (8 nodes).
Expected load: databases, reports, aggregation, possibly hpc, zfs, dedup ocsional. Entire data refresh every 2 weeks.
Have dell server (2x xeon I know amd would be better) chase with 13pcie slots and 16sata ports.
Theoretical I can fill 12 pcie slots with 4x m.2 expansion cards like this one
and load them with 1-2TB nvme m.2 cards.
That would provide 48TB of raw space. I potentially could run it as 2 tier nvme and sata storage.
My storage has potential to saturate 10GBs network connectivity so will probably need to NVMEoF (nvme over fabric) on 40BGs infiniband (RDMA).
Most likely will need to use software (os level) raid.
What os would you recommend to install on the host?
Has it got chance to work as fast starage?
Has anyone done something similar and would like to share?
Any suggestion, advice much appreciated.
I understand I need to check if all my pcie slots are all x16 (each m.2 will take 4pci lines).
Verify cards/drives are on hardware/os compatibility list.
Check board PSU can deliver enough power.
Check there will ne enough space between pcie slots and m.2 drives do not overheat.
There is substantial price tag associated with the project so would be nice to know its going to work.