Atamido

Suggestions for SAN hardware

Recommended Posts

We're going to be building a high availability mirrored iSCSI SAN for some high availability VMWare ESX boxes. We will be providing an iSCSI target interface to the two boxes using SANmelody, which runs on top of a Windows install.

We want to keep our IOPS high, but at the same time reduce costs by building it ourselves. We are assuming that just keeping the number of drives high will help keep the IOPS high. We only need about 2TB of space. RAID 0 is fast, but we want some redundancy on the individual boxes. RAID 1+0 costs to much in extra drives. RAID 5 (6 is preferable) with a hot spare would likely be best. The RAID controller will need a decent amount of write cache, but most of the read caching will be handled by the SANmelody software using system RAM.

For RAM, we are starting off at 4GB, but we might upgrade to 8GB or more in the future if we decide we need extra performance. The RAM needs to be ECC. I don't think the speed of the RAM is as important as the iSCSI will probably be the more limiting factor.

The OS needs to be Windows 2003 Server 32-bit. The SANmelody software requires it. There are OSS solutions out there, but we want a support phone number, and a solution we can verify lots of people are using in the enterprise. We will need the Enterprise Edition of 2k3 server to go over 4GB of RAM.

We also want to team together two 1Gbps NICS for a 2Gbps primary iSCSI interface. There will then need to be at least a 1Gbps NIC to connect to the mirror. At minimum we want 4 good 1Gbps network ports.

The CPU probably only needs to be dual core. Most of the load will be IO based, so any more than two cores isn't likely to see an improvement.

So to summarize, we think we need the following components:

  1. Rack mountable chassis with dual power supplies, 15-16 hot swappable SAS drive bays.
  2. Motherboard that can support a Core 2 Duo CPU, 8+ GB of RAM, and a PCI-e RAID card, and two PCI-e dual gigabit NIC cards.
  3. 4GB of RAM, with ability to add at least 4GB more.
  4. PCI-e RAID controller capable of controlling 15 drives in a RAID 6 and having 1 hot spare.
  5. Two dual TOE gigabit NICs.
  6. Video either onboard or as a PCI card.
  7. 16x 15k RPM 150GB SAS drives

Is this our best option, or is there a better/cheaper equivalent option?

Share this post


Link to post
Share on other sites
We're going to be building a high availability mirrored iSCSI SAN for some high availability VMWare ESX boxes. We will be providing an iSCSI target interface to the two boxes using SANmelody, which runs on top of a Windows install.

We want to keep our IOPS high, but at the same time reduce costs by building it ourselves. We are assuming that just keeping the number of drives high will help keep the IOPS high. We only need about 2TB of space. RAID 0 is fast, but we want some redundancy on the individual boxes. RAID 1+0 costs to much in extra drives. RAID 5 (6 is preferable) with a hot spare would likely be best. The RAID controller will need a decent amount of write cache, but most of the read caching will be handled by the SANmelody software using system RAM.

For RAM, we are starting off at 4GB, but we might upgrade to 8GB or more in the future if we decide we need extra performance. The RAM needs to be ECC. I don't think the speed of the RAM is as important as the iSCSI will probably be the more limiting factor.

The OS needs to be Windows 2003 Server 32-bit. The SANmelody software requires it. There are OSS solutions out there, but we want a support phone number, and a solution we can verify lots of people are using in the enterprise. We will need the Enterprise Edition of 2k3 server to go over 4GB of RAM.

We also want to team together two 1Gbps NICS for a 2Gbps primary iSCSI interface. There will then need to be at least a 1Gbps NIC to connect to the mirror. At minimum we want 4 good 1Gbps network ports.

The CPU probably only needs to be dual core. Most of the load will be IO based, so any more than two cores isn't likely to see an improvement.

So to summarize, we think we need the following components:

  1. Rack mountable chassis with dual power supplies, 15-16 hot swappable SAS drive bays.
  2. Motherboard that can support a Core 2 Duo CPU, 8+ GB of RAM, and a PCI-e RAID card, and two PCI-e dual gigabit NIC cards.
  3. 4GB of RAM, with ability to add at least 4GB more.
  4. PCI-e RAID controller capable of controlling 15 drives in a RAID 6 and having 1 hot spare.
  5. Two dual TOE gigabit NICs.
  6. Video either onboard or as a PCI card.
  7. 16x 15k RPM 150GB SAS drives

Is this our best option, or is there a better/cheaper equivalent option?

I looked at SANmelody as a cheap solution for a HAS environment but ended up going with EMC anyway (cause that's what we know... FUD etc...)

Anyway, since you want to be able to call someone if it fails, just buy a boxed solution from HP, IBM or even Dell? Why buy a prepackaged solution in the software and screwing toghether the hardware yourself? The savings for that is likely smaller...

Share this post


Link to post
Share on other sites

I have built quite a few SANMelody/SANSymphony boxes.

First SanMelody does not require Windows 2003 it runs fine on Windows XP. What SANMelody supports is the Windows 2003

capabilities on the client machines. This is independant of the OS on the SANMelody boxes. The main limitation on a windows XP

box is the 3.5GB memory.

My recomendation would be to purchase a Supermicro based machine. They offer bare systems which have the drive bay's

and slots that would fit your requirements. When I was specing machines the Dell, HP and other prebuilt machines did not

offer sufficent drive bays and slots to make a good SDS (Storage Domain Server). This may have changed.

It would take a very strange configuration on a SANMelody machine to see a major performance difference between 4GB and

8GB setups. If all you have is a modest number of volumes you wont even use 3.5GB.

Both 3ware and Areca offer RAID6 capable RAID cards. Keep in mind that RAID6 writes have tendancy to be slow.

I probably would not start out using the TOE cards. Since the box would be a dedicated datamover the lower CPU utilization

that a TOE might offer would be less of an advantage. You would also need to check the compatability of the TOE card.

Share this post


Link to post
Share on other sites

We currently have two boxes with Supermicro chassis, and they are easily the loudest boxes we have. Not that noise is a major concern, but it is worth noting. There have been some oddities matching the chassis up other people's motherboards and cards, but nothing to major. I had looked at the SC836E2-R800, but was hoping for some other options for comparison.

It's good to know that SANmelody will run on Windows XP, but we're going to have to use Windows 2003. My boss says, "if it's a server, it will be running server software." We get the OS at a good discount, so I'm not going to fight that battle.

Yeah, I don't know if the extra RAM will make a difference, but we want the option to take that route if we need to. We're going to be running a lot of virtual machines where the .vmdk image is on the SAN, including one large one that handles all of the user shares for roaming profiles, etc. If everyone logs off and on at the same time, I think the extra RAM could help a lot. But, we won't know for sure without some testing, and we'll probably only do that if it is an issue.

3ware lists the 9690SA as their only SAS controller, but it doesn't appear to be for sale anywhere. Areca has the ARC-1680 which appears to run around $1000, but again the availability seems pretty limited. Adaptec has the RAID 31605 which can be had for around $900. If I were going to buy tomorrow, I would have to get the Adaptec as it's the only one I can find.

I'm banking on good write caching methods to cover up the poor write performance of RAID6 (or 5). Ideally if the controller can hold onto a write until there is a pause in reading, it shouldn't matter that it's slower. "Shouldn't" being the key word there. Maybe the Areca with a 2GB DIMM in it would perform best.

The TOE would actually be to improve throughput, rather than reduce CPU load. If the TOE can do it faster than the CPU, then it would make sense. I should ask if it is supported through with SANmelody. There are Intel NICs, so Windows should be fine.

On a different note, What kind of performance hit would I be looking at if I moved to SATA drives? The Wester Digital Raptor WD1500ADFD is significantly cheaper than the Seagate Cheetah 15K.5 146GB drives. Filling up two of those chassis with 32 total drives, plus 2 spares is a difference between $8k and $18k for the two drives. What kind of real world performance difference would we be likely looking at once caching is figured in?

Share this post


Link to post
Share on other sites

1. For an in between option, I'd say an Intel SAN/NAS based server will be your best bet, and you have a company that "cares" when you ring them.

http://www.intel.com/design/servers/storag...e+body_ss212mc2

Just add CPU(s), RAM, and Drives, and Software

2. Don't go SuperMicro unless you are planning on buying the whole chassis/mainboard combo

Chenbro is a better option if you just want to totally customise it

http://usa.chenbro.com/corporatesite/products_line.php?pos=3

either the 3RU (16 Drives) or 5RU (24 Drives)

Example

http://www.valhalla.net.au/forum/viewtopic...&highlight=

3. For high concurrent I/O the 150GB Raptors will be half the speed of current generation 15Krpm SAS drive. i.e. you will need twice as many Raptor to get the same speed, even then it may be cheaper to go with the Raptors

4. RAID5 makes zero(0) sense in your situation, you are not capacity bound, you want the highest speed and RAID5 =/ High Speed. I wish people would realise what RAID5 was designed for, i.e. obtaining the greatest capacity for the least expense, it was never about speed.

Share this post


Link to post
Share on other sites

@czr:

1. That's a pretty cool little box, I hadn't seen it before. It's to bad it doesn't have more drive bays. For us, we aren't in as much need of talking to a packaged hardware vendor that "cares" as we feel pretty comfortable with most hardware. We just need to check compatibility first. (We'll be using a packaged OEM box for most of our servers, but this is a special case configuration and price wise.)

2. Check. That 5U box is so very nice.

3. Yes, it would probably be cheaper to get twice as many Raptors, even after figuring the extra chassis and RAID controller costs.

4. Actually we are capacity bound (at about 2TB). We are also cost bound (as part of the overall VMware project). RAID 5/6 should make zero impact for read speeds, which should account for 98%+ of transactions. For write speeds, we are counting on the cache lowering the impact of that to a negligible amount. Do you have any benchmarks that show the difference between a RAID 1+0 and 5 of a large number of drives for reads versus writes?

Yes, I would love to load up two of those 24 drive units with the 300GB 15k SAS drives in a RAID 1+0 and really pump out the IOPS, but the cost of drives alone would eat almost our entire project's budget. If I can get 95% of the performance, from a RAID 5/6, as I can from a 1+0 for half the cost, then that is what makes the most sense.

Share this post


Link to post
Share on other sites

The noise level will vary depending on model. The highest noise level chassis I have every dealt with was an intel box.

The Intel chassis / MB combos are decent technology but they don't have the options that the Supermicro case line has.

RAID levels and SATA vs SAS will depend on what kind of services you are running on your VMs. Highest reliablity will

be gotten from RAID6 in an array of a size you are considering. RAID10 would give the highest IOPS. SAS vs SATA

would be determined by what your IOPS requirements.

Cache is not a cureall. In the case of everyone logging in at the same time and accessing their roaming profile the

various caches in the system are not going to do you much good.

SATA/RAID6 is more than sufficent for many common servers. If you have some servers that have a much higher

disk io requirement they can be put onto their own RAID set.

One thing you should do before you commit to a system is to determine if using more than one SANMelody box would

be more cost effective. A single very high performance box is unlikely to be twice as fast as a more modest box.

Share this post


Link to post
Share on other sites
Cache is not a cureall. In the case of everyone logging in at the same time and accessing their roaming profile the

various caches in the system are not going to do you much good.

It's not a cureall, but it can offer quite a boost in certain situations. If a user reboots, so the profile is written and then read back when they log on, that can all happen in cache with the writes happening whenever it is convenient.
One thing you should do before you commit to a system is to determine if using more than one SANMelody box would

be more cost effective. A single very high performance box is unlikely to be twice as fast as a more modest box.

We're currently planning on having two SANmelody boxes and two VMware boxes. The VMware boxes would use their own SANmelody box as the primary and the other as the secondary. So we will be using more than one box, but virtual machines will only hit a single box.

We're concerned about a sudden high IO on one or two VMs bringing the SAN to it's knees, as well as just general responsiveness.

If you ran a single RAID6 array spanned over 24 drives, are writes going to be possible at any reasonable speed? What is the performance hit for writes going from 12 to 24 drives in RAID6?

Share this post


Link to post
Share on other sites

It looks like the cheapest way right now to get a fast 3TB array is to stack 24x 150GB SATA 10k drives into a single box in a RAID6. This should offer more IOPS than 12x 300GB 15k SAS drives, and for less than half the cost.

Chenbro 24 drive chassis. ~$2200 (SATA backplane and such are extra)

http://usa.chenbro.com/corporatesite/produ...il.php?serno=44

Areca ARC-1280 RAID6 controller ~$1300 (comes with 256MB upgradeable to 2GB of cache)

http://www.areca.com.tw/products/pcie341.htm

or:

3ware 9650SE-24M8 RAID6 controller ~$1200 (comes with 512MB cache)

http://www.3ware.com/products/serial_ata2-9650.asp

(Adaptec only makes up to a 16 port card)

WD Raptor 150GB 10k SATA ~$200 (24 of these is ~$4800)

http://www.westerndigital.com/en/products/...amp;language=en

For the motherboard we really just need one that supports a Core 2 Duo, so a consumer level board is okay. Probably an Intel board known to be compatible with whatever RAID card we get. Cost of the motherboard + CPU shouldn't be over $500.

4GB of RAM. If we want more than 4GB, then we will need to purchase Windows Server 2003 Enterprise, which would cost us another $2500. If it ends up being necessary, then we will make the switch, but right now it looks like 4G is where it will be. 4GB of RAM shouldn't cost more than $500 either.

This doesn't include backplane, cables, NICS or software, but the cost per box looks like it won't exceed $11,000. For two boxes plus the SANmelody DR-1 option, the total should be under $36,000USD. $36k for a fully redundant HA with 3GB cache, 3TB of usable space on that many 10k spindles doesn't sound like a bad price.

This isn't necessarily what we're buying, but I have to price out a few different options and this one is certainly interesting.

Share this post


Link to post
Share on other sites
4. Actually we are capacity bound (at about 2TB). We are also cost bound (as part of the overall VMware project). RAID 5/6 should make zero impact for read speeds, which should account for 98%+ of transactions. For write speeds, we are counting on the cache lowering the impact of that to a negligible amount. Do you have any benchmarks that show the difference between a RAID 1+0 and 5 of a large number of drives for reads versus writes?

It looks like I just found such a benchmark from Areca. Go here and scroll down to the bottom. The third option from the bottom is:

IOP341_16HDDs_Performance 2007/8/8

From this report, sequential reads and writes from any type of RAID array are about twice as fast as RAID1 (I assume they mean 1+0). The IOPS for random reads is about the same for any given RAID type. Random writes is where things get really interesting.

For 2KB writes, the ratios are:

RAID0 6.1

RAID1 2.7

RAID5 1.6

RAID6 1.0

For larger writes, the numbers scale closer together in a linear fashion. At 512KB, the ratio is:

RAID0 2.9

RAID1 1.6

RAID5 1.2

RAID6 1.0

An important note on this is that this is performance of the drives, so real world performance where you get to cache reads and writes is going to be a bit different. 500 IOPS for 2KB random writes is 1MB/s, and with 2GB of cache you could cache 30 minutes worth of those writes (where many would undoubtedly be combined and written simultaneously, etc).

A very interesting read for a 16 HDD array. Fortunately the vast majority of accesses in our case will be reads, so we should be fine with RAID6.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now