blaz

Pointers on setting up a 3TB RAID array

Recommended Posts

Hi,

I've been considering a long-term project to build a 3TB+ RAID array to store my Anime collection. As I have never built a RAID array and as it's going to be so expensive I need every bit of information I can get, especially opinions and comments from people who have built or are maintaining a similarily sized array.

Now to the facts:

- My target capacity is 3TB, for starters.

- I'm considering building a SATA array from day one, perhaps even waiting a bit for BTX motherboards and PCI Express RAID controllers to avoid the bottleneck of the PCI bus.

- I'm considering using 250GB drives and RAID-5 (no idea on how many drives per array, though).

- I'm going to use the array for backup purposes only, so I don't necessarily need high performance, but I do need some degree of fault tolerance. I want to minimize the probability of losing data without building an overly expensive array.

- Backing up the data to CD-R/DVD-R is not an option, that's why I need the array.

And now the questions:

1. Is it wise to build a SATA array? I know the prices on PATA disks and controllers are lower, but as this is going to be a long-term project (I will be adding a disk every 2 or 3 months) I think there's going to be enough time for SATA prices to drop to or below current PATA prices. I don't mind paying a bit more for SATA if it means it's going to be easier to route all the cables, etc.

2. Is RAID level 5 a good choice for my expected usage of the array? How many disks should I use per array? Using RAID-5 means you lose the equivalent to one disk of capacity per array (a loss of 25% on a 4-disk array, compared to 12.5% on an 8-disk array), but of course 8-port controllers are much more expensive than 4-port ones, and on 8 disk arrays it's much more likely than two drives fail at the same time (although I seriously hope it never happens). And I must also consider the expected number of PCI Express slots on the upcoming BTX motherboards, if I go for this bus. All prototypes I've seen have only two 1x PCI Express slots, which means I could have at most 2 physical arrays. Two 6-disk arrays at 250GB per disk add up to a total of 2.5TB, half a terabyte below my initial target capacity.

3. As I mentioned I'm planning to build the array progressively, adding one disk every few months. The whole process could take more than two years; is it possible that I will face problems trying to find the exact disk model say two years from now? are current SATA drive models going to stay with us for a long time? does it really matter that much that drives on an array are exactly the same brand and model?

4. Of course I won't be able to fit this many disks in my current midi-tower case. I have no idea what kind of cases (and power supplies) are good for arrays like this. I'd like the case to have at least hot-swapping capability and be prepared for extra drive cooling. Any suggestions on a relatively cheap case for this?

5. Just out of curiosity, what OS is everyone managing their RAID array on?

6. Any advice on which drives to use?

7. Any other comments or opinions? What kind of array are you using?

Thanks a lot for every bit of info you can supply :)

Share this post


Link to post
Share on other sites

1. Wisdom is in the eye of the beholder.

2. RAID 5 is a rational choice for large raid array

3. Yes, as time goes on finding the same model will become progressivly more difficult. but that doesnt really matter too much, you only need drives of the same capacity.

4. No cheap cases dont normally have 16 drive bays. Check on ebay, as the dot.con meltdown continues some fairly high end equipment gets liquidated at reasonable costs.

5. You can use pretty much any modern OS you like, Windows 2000 or XP shouldnt have a problem with large volumns.

6. Maxtor 250 GB hard drives are widely available in SATA models. Havent seen any WD's yet.

7. This is going to be expensive, the server chassis and power supply would be enough to give most poeple second thoughts.

Share this post


Link to post
Share on other sites

Raid 5

SATA

3Ware Escalade 850x-12 Controllers

250GB Disks (Maxtor will work great)

Finding drives of a 250GB capacity shouldn't be hard.

Rebuilding the array to include the additional disks will be a chore.

Using 2 12-port controllers each running a single RAID-5 Array, then RAID-0ing the two of them together will give you some good speed and a 1-drive-failure saftey margin.

Share this post


Link to post
Share on other sites
Raid 5

SATA

3Ware Escalade 850x-12 Controllers

250GB Disks (Maxtor will work great)

Finding drives of a 250GB capacity shouldn't be hard.

Rebuilding the array to include the additional disks will be a chore.

Using 2 12-port controllers each running a single RAID-5 Array, then RAID-0ing the two of them together will give you some good speed and a 1-drive-failure saftey margin.

Very few controllers let you setup a raid set accross controllers. (I can think of only one, an LSI card that costs a fortune)... all your raid sets have to be on one controller...

Share this post


Link to post
Share on other sites
Its $3500.00 without any drives installed.

Ok, That is a bit expensive for what you get...

I was thinking of that because everything seems to be moving to rackmount and it looks harder to find cases with a lot of drive bays. I myself have an antec server case with 18 drive bays. A lot of them being external 5.25. I looked on their page and they don't even sell anything like it...

I guess he could get a full tower with a lot of external bays and some of those 3 drive brackets that fit in 2 5.25 bays fromBay Coolers.

Share this post


Link to post
Share on other sites
Using 2 12-port controllers each running a single RAID-5 Array, then RAID-0ing the two of them together will give you some good speed and a 1-drive-failure saftey margin.

Thats a good idea.

Share this post


Link to post
Share on other sites
I think building it progressivly will be a problem unless you do not mind extra drive letters.

With raid 5 you can add a drive into the array dynamicly, that shouldnt be a problem. (although Ive heard some controller puke when its been tried)

Share this post


Link to post
Share on other sites
Using 2 12-port controllers each running a single RAID-5 Array, then RAID-0ing the two of them together will give you some good speed and a 1-drive-failure saftey margin.

That's a nice idea, but as I mentioned I value fault tolerance more than speed. Two separate RAID-5s means I can have two drives die on me at the same time, as long as they belong to different physical arrays. I'm not particularly interested in the added performance of RAID-0 in this case :)

Share this post


Link to post
Share on other sites

Thanks for the replies so far.

Some of you mentioned the 3Ware Escalade controllers. If I were to build the array now I'd probably go for those, but I have quite some time before I buy the first controller (I first have to buy a couple of disks; and btw, do you need to fill all the ports on the controller for RAID to work? or can I use 4 drives on a 12-port controller until I buy the rest of the drives?)

I suppose it's a bit too early to know, but do you think it would be best to go for PCI Express controllers instead of PCI? 8 or more disks can definitely saturate the 133MiB/s PCI bus, and I don't have any 64bit/64MHz PCI slots. On the other hand, I'm going to use the array for backup purposes; should I be concerned about bus speeds?

I don't want to spend so much money on an outdated bus or protocol (i.e. PCI and PATA) and have support for them removed from hardware in a few years time, that's why I'm trying to use the newer standards whenever possible.

Share this post


Link to post
Share on other sites
I guess he could get a full tower with a lot of external bays and some of those 3 drive brackets that fit in 2 5.25 bays fromBay Coolers.

That's very interesting. Perhaps I don't need an prohibitely expensive case after all.

I have a couple of questions:

1. What exactly is needed to make the SATA drives hot-swappable? I just read this on the article on the Maxtor MaxLine Plus II:

"As an SATA drive, the MaxLine II includes a newer 15-pin power connector that supports hot-swap functionality"

Does it mean I don't need any kind of special bay to safely hot-swap the drives?

2. I found the "fitting 3 drives in two 5.25 bays" thing very interesting (it seems you can also fit 5 drives in 3 5.25 bays), but is that compatible with hot-swapping? It would be great to have both hot-swapping functionality and decent drive cooling, but looking at this site's products it seems you have to choose one thing or the other (I might be wrong, but I don't see any fan in these drive carriers)

I could live without hot-swapping though.

3. How many power supplies should a, say, 12 drive RAID array be powered with? Any ideas on how much power 12 7200rpm drives dissipate?

Share this post


Link to post
Share on other sites
I first have to buy a couple of disks; and btw, do you need to fill all the ports on the controller for RAID to work?

No, but you need to install at least 3 drives for RAID 5 I beleive.

Share this post


Link to post
Share on other sites

Some facts about 3Ware cards....

1) They do not support more than a 2Tb array. So that 12 port card can't be setup as a single large single array.

2) Only the 1st RAID5 array on each cards will do their "Fusion-R5" full-stripe method, the 2nd will be the slow method. (this is a huge deal they don't mention) So unless you want a R5 array that's hard pressed to do 10MB/s stay away from running 2 R5's off the same card!

Because of the two above anything more then than the 8 port versions don't make much sense if you are doing RAID5.

So what I suggest is a pair of 8506-8's both with a 8 * 250Gb R5 array hanging off it. This will give you around 1.68Tb of usable capacity per controller. Then simply use Windows Raid-0 to stripe across these and you are good to go.

We do this with 5U 24 bay cases from www.rackmountpro.com. We simply stripe across 3 8506-8 each with 8 250Gb WD "JD" drives to give us 3 * 1.68Tb or 5Tb (and change) usable. Sequential transfer is pretty scary in this configuration -- easily over 200MB/s in writes and 350MB/s in reads doing 64Kb sequentials with IO meter and lots of outstanding I/O's.

We use a 4-peer PCI bus equipped motherboard so each 8506-8 lives on it's own bus with a pair of Alacritech SLIC gig-e cards in the 4th bus in a load balanced pair. Add to that 16Gb of RAM, dual Xeon 3.06/1Meg CPU's and Windows 2003 Enterprise Server and it's pretty stout fileserving platform.

Combined this will pump 170MB/s over the dual gig-e channels all day/night long and cost you under $15K.

We tried the same thing with Opteron motherboards but they just don't handle the I/O load that well at this time - perhaps once the BIOS's mature and chipsets stabilize....

Hope this helps.....

John

Share this post


Link to post
Share on other sites
Some facts about 3Ware cards....

2) Only the 1st RAID5 array on each cards will do their "Fusion-R5" full-stripe method, the 2nd will be the slow method.  (this is a huge deal they don't mention)

And what exactly does 3Ware ever mention?

For the record, I've owned 3Ware cards for over two years now, and this is the first time I've read that explanation!

Share this post


Link to post
Share on other sites
Some facts about 3Ware cards....

2) Only the 1st RAID5 array on each cards will do their "Fusion-R5" full-stripe method, the 2nd will be the slow method.  (this is a huge deal they don't mention)

And what exactly does 3Ware ever mention?

For the record, I've owned 3Ware cards for over two years now, and this is the first time I've read that explanation!

3Ware doesn't mention this openly until you press their tech support people over performance issues.

We tried creating 2 Raid-5 arrays off a 8506-12 card each with 6 disks. The first R5 set was quite quick, the other was dog slow. When 3Ware was queried over this the response that came back is that each card only supports a single R5 "fusion" array. If multiple R5's are created off a single card than the second one uses the non-stripe method which is much slower.

Originally we set up the 24 drives as 4 6 disk R5 sets striped using 2 8506-12 but performance was poor so we worked with 3Ware. We than migrated to 8506-8's instead to get around this problem.

It is a problem on all 3Ware cards, but obviously it's only become an issue with the 12 port cards and large disks where you can exceed the 2Tb limit and want to use multiple arrays.

Hope this helps....

John

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now