tjoff

Cheap 8-port SATA controller solutions

Recommended Posts

Hi

I'm in the need of a new raid array. And although ZFS is extremely tempting the lack of raid-z expansion is just too much for me (adding new raid-zs to a pool isn't economical if you only have a need for expanding with one or perhaps two drives at a time). So I'm probably going to go with mdadm raid 6 and in time switch to BTRFS (yeah, it's going to be a long wait...). Though it would be nice with opensolaris compatibility in case I change my mind (again) :)

Anyway, aside from being tempted by XFS and 3TB drives the real question that I have is regarding the controllers. As of now I'm mixing the onboard motherboard SATA ports with Sil3114 based PCI cards. Which doesn't feel ideal. The performance isn't really a big issue but rather reliability (not that I've had any problems, but I've heard of cheap PCI cards (including Sil3114) that have died under preassure and basically ruining the entire array). Also I'm all out of PCI slots so either way I need at least one PCI-E 8 port controller.

I'm redoing the file-server chassi for the new array and in total it should be able to house about 14-15 drives. So going with one or perhaps two PCI-E 8 port SATA controllers seemed like one of the better solutions. Problem is that those are quite rare. Seems like two-channel SAS cards is the way to go if you want to have any money left for drives...

Now since I'm set on using software raid the only thing I'm interesting in is being able to use each drive separately and preferrably be able to use SMART (possible?)

Going with a SAS controller frightens me a bit since I have the prejudice that SAS controllers are more picky and thus have more compatibility issues. Is my concern valid (or are compatibility issues really something rare that perhaps almost always gets fixed by a firmware upgrade)?

The Intel SASUC81 seems to fit the bill. Cheap two-channel SAS that seems to support JBOD (which, coming from consumer products, seems to be a strange way to state that it can handle each drive separately without RAID).

But how about compatibility?

Intel states:

Intel will only support this RAID controller when it is installed in a system configured with the specified server boards, and when the server board is configured with the tested RAID firmware, system BIOS firmware, and operating system versions.

I guess you can't ask for more, I am on my own...

Going down to hard drive compatibilities and there are only one 2TB drive (Hitachi). And I was more aiming for some low-power/noise solutions such as the 2TB Western Digital Caviar Green or perhaps the new 3-platter Samsung EcoGreen F4.

I'm not going to be guaranteed anything but is compatibility issues common? Also, will 3TB drives present special compatibility issues with the controllers of today? (4k sectors + LBA)

Just as a comparison, the cheapest all-SATA 8port card I've found is the HighPoint RocketRAID 2320 which is from 2006 and cost twice as much as the intel-card above and the compatibility list is about as relevant as intels.

Any thoughts/suggestions?

Share this post


Link to post
Share on other sites

Thanks

Looking at the specs the only real difference I can see is that it has a PCI-E x4 port compared to Intels 8x. That probably shouldn't matter for my needs but I didn't find any information about compatibility.

Though the biggest problem is that I couldn't find it at a reputable store :|

Share this post


Link to post
Share on other sites

Thanks

Looking at the specs the only real difference I can see is that it has a PCI-E x4 port compared to Intels 8x. That probably shouldn't matter for my needs but I didn't find any information about compatibility.

Though the biggest problem is that I couldn't find it at a reputable store :|

http://www.amazon.com/Supermicro-Add-Card-AOC-SASLP-MV8-controller/dp/B002KGLDXU

http://www.newegg.com/Product/Product.aspx?Item=N82E16816101358&nm_mc=OTC-Froogle&cm_mmc=OTC-Froogle-_-Hard+Drive+Controllers+/+RAID+Cards-_-SuperMicro-_-16101358

http://www.buy.com/prod/supermicro-aoc-saslp-mv8-8-ports-sas-raid-controller-pci-express-x4/q/sellerid/11408470/loc/101/211410774.html

Probably some of the top reputable companies?

Share this post


Link to post
Share on other sites

My coworker has an AOC-SASLP-MV8, but he was unable to use it on OpenSolaris because there were no drivers or the drivers were bad or something like that. He changed to an LSI card (I'm not sure which). I see it on Amazon (through ANTOnline) and there's a bunch of sellers on eBay. I've heard the same thing about SAS controllers, like the 3ware 9690sa has been a bit twitchy with SATA drives, but these are just HBAs so compatibility could be less of an issue here. It's a Marvell chipset and it follows the AOC-SAT2-MV8 (PCI-X) controller which sounds common in the older OpenSolaris/ZFS boxes (a search for both returns many results).

I don't think you'll have any problems with 4k sectors here since you still have access to the drives, it's not the firmware that needs to be able to handle it for raid functions.

You can find some pretty cheap .05 or .06m ML cables on eBay. It's a bit of a pain at first since you probably have a ton of SATA cables lying around (like I do), but I think it's alot cleaner. This way you don't have ports running up and down or stacked all over the card.

http://louwrentius.blogspot.com/2009/07/20-disk-18-tb-raid-6-storage-based-on.html has some info on a large mdadm raid6 using a HighPoint RocketRAID 2340.

Share this post


Link to post
Share on other sites

Thanks. I've heard success stories with the intel card and opensolaris at least.

Makes sense that HBAs would be better off in terms of compatibility (thanks for putting some ease to my mind :))

I've had some trouble finding miniSAS to 4xSATA cables with angeled SATA connectors (must have angeled connectors or it won't fit :(), the only one I've found is this one http://cgi.ebay.co.uk/MINI-SAS-SFF-8087-4-SATA-Cable-Assembly-/230524847970?pt=UK_Computing_CablesConnectors_RL&hash=item35ac5a0762#ht_1779wt_762 that has some weird SATA cable that is split in two :o (doesn't look that reliable but I suppose that looks doesn't say much).

Thanks for the URL.

jpiszcz:

Thanks, but I don't live in the US (I live in Sweden).

Found it on ebay for a decent price but the shipping isn't negligible and it's of course much simpler and faster to buy locally than shipping it from the US.

Share this post


Link to post
Share on other sites

That blue cable looks exactly like the one I borrowed from work when my 3ware 9650se-12ml didn't come with any cables. Hopefully it's not as stiff as the one I had, because that thing was all crinked up and one of my drives kept disappearing. And those angled ends made it really tough to unplug from the back of my internal backplane. If you order those and they're super stiff, be careful.

My gaming system needs angled cables on the motherboard side since my gigantic video card sits right on top of the sata ports. I actually had to adjust the orientation of my internally mounted hard drives (when I had them down there) when I installed that card since it was also touching the back of the drives and I couldn't get the power cables connected. It was just a pain all around...

Share this post


Link to post
Share on other sites

I use LSI SAS1068E's, which I got for a great price but even at normal prices it's pretty decent performance for the money.

Also, I'll bet you ZFS expansion will arrive before BTRFS becomes stable with RAID-5/6+

Share this post


Link to post
Share on other sites

I didn't find the LSI SAS1068E anywhere close to where I live. I found a supermicro PCI-E 8x card based on the LSI 1060E chipset, but it's an UIO version and it cost about the same as the intel-card.

The intel card is also based on the LSI 1060E chipset so I guess it's pretty much the same thing anyway(?).

As for ZFS expansion vs. BTRFS you are probably right. I've been kind of turned off by the whole solaris thing since Oracle took over, and I've searched a bit but haven't really found out if raid-z expansion really is something that Sun/Oracle are even interested in. The best I've read is that people are talking about a coming feature that will make expansion quite trivial to implement but could just be hearsay.

Hmm, and I thought zfs-crypto was considered stable and included in OpenSolaris already :| That probably kills it for me in the short term. When the whole solaris+oracle(+illumos?) thing have calmed down and when raid-z expansion and ZFS-crypto are available I'll consider moving. Until then I guess that mdadm raid6 + dmcrypt + EXT4 will have to suffice.

superlgn: Thanks for looking it up.

Finding miniSAS => 4 angled SATA cables have proven to be quite difficult and a surprising show-stopper. Don't really know how to proceed with out them.

Found a few more:

http://cgi.ebay.co.uk/433775-001-Internal-3Gbps-MiniSAS-4-SATA-Cable-OEM-/350364022620?pt=LH_DefaultDomain_0&hash=item519352b75c#ht_1433wt_689

http://cgi.ebay.co.uk/New-Sealed-HP-446042-001-MiniSAS-4-SATA-Cable-18-/350213960202?pt=LH_DefaultDomain_0&hash=item518a60f20a#ht_2799wt_1043

But they are too short...

I could use angeled SATA adapters ( http://www.frozencpu.com/products/8758/ele-416/FrozenCPU_Sata_Male_to_Sata_Female_L_Adapter_OSA25.html?tl=g2c38s261 ) but it would be quite expensive and I'm afraid about glitchy connectors as it is and it probably won't do wonders with the signal quality either (but perhaps thats not an issue).

Share this post


Link to post
Share on other sites

I don't understand why Sun/Oracle doesn't do a block layer thing like Linux has with the device mapper. Then you could do encryption for any filesystem on Solaris and not just zfs. It's hard to believe it's 2010 and they still don't have anything... Linux had cryptoloop and loop-AES 10 years ago, and probably something before that.

The Intel SATA/SAS controllers using LSI SAS1068E chipsets sound familiar. I think I saw someone say you could even put LSI firmware on them, so maybe it's just a branding thing.

I've never seen anything like those adapters before. I don't know how well they'll work, but it sure will add to the cable cost. :/

Share this post


Link to post
Share on other sites

Umm, Sun/Oracle does have a block layer thing like Linux and you can encrypt any block device with the loopback device.

zfs-crypto however isn't stable as it's dependant on a component of zfs that is not yet implemented. In fact, raid-z expansion is dependant on the same component (blkptr rewrite). A lot of upcoming features of zfs are dependant on the same component, so said component is indeed quite a high priority. That said, they've been working on said component for several years and it's looking more and more like vaporware...

As for the LSI SAS1068E - I don't know about Intel ones but I know I've replaced the firmware on my HP branded ones with vanilla LSI firmware without a problem.

Edited by qasdfdsaq

Share this post


Link to post
Share on other sites

LOFI doesn't seem like a sensible solution for a large full raid encryption. I could be wrong but it doesn't seem to have the greatest reputation for other things than small solutions. Performance being one issue.

But I could be wrong, also doesn't seem like a widely used solution for my purpose so I'm a bit skeptical.

Share this post


Link to post
Share on other sites

The last time I used Solaris was probably over 8 years ago and it was only for a short while, so I'm pretty useless here. I've got less experience with FreeBSD, but it's a bit more recent. I was reading about the different options FreeBSD has for encrypted filesystems the other day. My coworker is pretty confident in FreeBSD+ZFS. When things with Sun/Oracle started going bad, that's what he switched to without a whole lot of hesitation. Then it sounded like OpenSolaris was going to get going again so he began moving his projects back. Now it's all gone to hell and I'm not sure what he's going to do. If FreeBSD could pull in a new ZFS version and fix any bugs they may be having, that could be a good option. It's too bad the license issues kept ZFS out of Linux... BTRFS may eventually end up a similarly featured alternative, but it's probably years away from having all the features in and being declared stable and a few more years after that before there's any widespread adoption.

I don't know that cryptoloop was designed for larger setups, but I was using it on decent sized filesystems without any problems way back when. No clues on performance. The device mapper sounded like a more robust solution to me and it seems like the choice for Linux users now. I've done some benchmarks at home and work and it's losing about half. My 3ware 9650se array at home gets 325MB/sec writes and 550MB/sec reads drops to 130 and 165. A 9550sx in a slightly older Xeon system at work goes from 175MB/sec writes and 225MB/sec reads to 80 and 95. The 9550sx is going into an offsite backup box where the bulk of the 4.5TB array will be encrypted. Performance isn't critical here. It's going to hurt at first, but once I get everything on there it shouldn't be too bad to keep up to date with rsync.

Are you finding some pages that compare ZFS or UFS performance on LOFI? The ciphers could have an impact too... Maybe blowfish would be faster than aes.

Share this post


Link to post
Share on other sites

I've been reading a bit about those Supermicro UIO AOC-USAS(LP)-L8i controllers after I saw some cheap ones on eBay (under $50). It sounds like they'll work just fine in regular PCIe slots, but you have to fiddle with the bracket. I saw one guy who used a .25" nylon spacers to mount the bracket in the next slot, another reversed the bends with a pliers, and others just took the bracket off. The 1068E seems pretty good for compatibility, although up until Linux 2.6.30 or .32 there were some issues with smartd dropping ports or something. Some of LSI's newer controllers use the SAS2008 chipset and it seems like there's a few more issues with those on Linux.

Share this post


Link to post
Share on other sites

Hi

Just as a comparison, the cheapest all-SATA 8port card I've found is the HighPoint RocketRAID 2320 which is from 2006 and cost twice as much as the intel-card above and the compatibility list is about as relevant as intels.

Any thoughts/suggestions?

I read someone using this card:

http://www.exsys.ch/index.php?main_page=product_info&products_id=702&zenid=421e59c30029a07ff9c4e281e3196f89?disp_order=2

It is however very slow since it only supports PCIe 1x which provides 250 MB/s on bandwidth for all 8 drives. When rebuilding or growing, this may take days. However, reads and writes are reported to be around 150 write and 180 MB/s read. So it saturates gigabit.

This has been tested with Linux + MDADM.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now