Supermicro 6048R and general raid questions

7 posts in this topic

Hi, I'm trying to get an answer for something that I've tried asking the manufacturer, but it's hard to get a good idea. I'm looking to buy a server like the 6048R that has 60 spinning disks in a RAID 5 or 10 configuration. The application is a time-series database doing constant sequential writes, and infrequent bursty reads. I would estimate it's about 99% writes and 1% reads. The data I'm looking to write out will probably be around 1-1.5GB/s. The stripe size can be as large as possible since latency is not an issue at all, but performance is. My question is can a server in that configuration keep up with that write rate, and periodic reads using spinning disks, or would I have to use SSDs? With the research I've done I believe it's possible since I've seen the LSI 3108 controller it comes with can do 4-5GB/s in benchmarks. I read some posts by funkywizard a while ago that makes me think it's possible, but confirmation would be nice.

On a different note, can someone explain what this RAID card means by 8 channels at 12Gbps each? Does that mean for a system with 60 drives it would have 7-8 disks on each channel? From an OS perspective, does this look like 8 drives, or does it abstract to 1 and the controller load balances?


Share this post

Link to post
Share on other sites

> The data I'm looking to write out will probably be around 1-1.5GB/s.

Doing a first-order "parametric analysis", you can safely assume that

late model spinning platters can WRITE at about 200 MB/second (average).

We measured a brand new Toshiba 3TB HDD at 200 MB/second,

using outermost tracks (which are THE fastest).

You should also appreciate that spinning platters necessarily slow down

as they fill up, because the data rate is directly proportional to track diameter

(if the innermost track is half the diameter of the outermost track,

the data rate of the innermost track is exactly HALF the data rate

of the outermost track). This is so because the recording density

is more or less the same regardless of track diameter.

Using the above assumption then requires a minimum of 1.5 / 2 = 7.5 HDDS,

assuming zero controller overhead.

If you assume an aggregate controller overhead (at both ends of the

data cables) of 20% in RAID-0 mode, because RAID-0 never

scales perfectly, then:

Average WRITE speed is 200 x 0.80 = 160 MB/second

1,500 MB/second / 160 MB/second = 9.3 HDDs in RAID-0

or a bare minimum of 10 HDDs in RAID-0.

Also, if your system is writing 1.5 GB/second,

then it will completely fill a 2TB HDD in:

2,000 / 1.5 = 1,333 seconds / 60 = 22 minutes

What happens when your system has run for 23 minutes?

4,000 / 1.5 = 2,666 seconds / 60 = 44 minutes

What happens when your system has run for 45 minutes?

and so on.

THEN, you'll need to refine this parametric analysis

for 60 spinning HDDs + RAID-5 overhead (i.e. second-order analysis).

e.g. try 30% aggregate overhead for RAID-5,

and try 60 HDDs @ 140 MB/second (i.e. 200 x 0.7):

60 HDDs @ 140 MB/second = 8,400 MB/second

You'll definitely need multiple SAS RAID controllers!

But, doing the inverse, 1500 MB/second / 60 HDDs = only 25 MB/second.

So, that many spinning platters should be able to WRITE 1.5GB/second,

but you will definitely hit a wall when all of those 60 HDDs start to fill up.

With that many discrete HDDs, you should consider RAID-6,

for data security reasons (if nothing else).

Thus, a third-order analysis assumes an aggregate overhead

specific to RAID-6 arrays.

> what this RAID card means by 8 channels at 12Gbps each?

It probably supports multiple "fan-out" cables, each of which fans out to 4 SAS cables.

And, each SAS cable runs at 12 Gigabits per second ("12G").

For example, search for photos of a SFF-8087 fan-out cable.

8 SAS channels at 12G / 10 bits per byte = 9.6 GB/second total raw bandwidth

Each 12G SAS channel uses the 8b/10b legacy frame:

1 start bit + 8 data bits + 1 stop bit = 10 bits per byte.

That bandwidth is exactly TWICE the comparable bandwidth of 6G SATA channels

(6 GHz / 10 bits per byte = 600 MB/second; 12 GHz / 10 bits per byte = 1,200 MB/second).

Share this post

Link to post
Share on other sites

These specs mention the LSI 3008 HBA (Host Bus Adapter):

Here's a review at this website of the LSI 3008 controller:

"The two 8-lane (of PCIe 3.0) models available are the LSI SAS3008 HBA (IT) and LSI SAS3008 HBA (IR), which both include mini-SAS HD connectors to meet the SAS 3.0 standard."

Share this post

Link to post
Share on other sites

p.s. Yes, I think you can do it with spinning platters, BUT

I would urge you to give serious consideration

to HDDs with a 5-year factory warranty

e.g. Western Digital's "enterprise-class" HDDs.

You'll definitely want to go with HDDs rated for 24/7 operation.

Just compute price per warranty years,

to get a feeling for the unit cost differences.

Also, SAS HDDs with a 12G interface

are much more expensive than

HDDs with a 6G interface.

e.g. these HGST models are expensive,

but they ARE FAST:

Edited by MRFS

Share this post

Link to post
Share on other sites

Thanks so much. Let me absorb your posts a little bit and let me know if more questions :). To your first point, the steady state of the system is always about 95% full with erases continuously going in the background dumping old data. If you think that's cutting it close, we can do ssd

Share this post

Link to post
Share on other sites

Depending on the kind of data you are archiving,

I would also highly recommend that you consider

some kind of data compression.

Many years ago, I worked on a computer mapping project,

and we started with a "grid cell" design e.g.

horizontal strings of integers for each "pixel".

With many of those data sets, there were very long

"runs" or sequences of the exact same integer.

So, we wrote simple software to compress

each "run length" with (value + length).

The amount of compression was enormous e.g. 90 to 95%.

And, I know that many surveillance cameras

are configured to record only one frame per second,

to reduce accumulated data volume.

Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now