[poodel]

Ok, now I've spent the better part of the day playing around with ZFS on a 4-disk SunFire V240. It's really an incredibly flexible filesystem. I'm not yet convinced that I will get it running on some decent X86 hardware. It installed properly on my old AMD 3000XP, but my more recent Core2Duo-system was less convincing.

I did some basic write comparisons between a 3-disk RAID-0 and a 3-disk RAID-Z with some interesting results:

# dd if=/dev/zero of=/kalle/testfs/file.out bs=1024 count=1000000
raidz write 26MBps
raid0 write 36MBps

# dd if=/dev/zero of=/kalle/testfs/file.out bs=2048 count=1000000
raidz write 59MBps
raid0 write 51MBps

# dd if=/dev/zero of=/kalle/testfs/file.out bs=4096 count=1000000
raidz write 82MBps
raid0 write 56MBps

Write performance in RAID-Z doesn't seem to be a problem at all.

[lizardking009]

poodel, on Jan 4 2007, 03:23 PM, said:

Ok, now I've spent the better part of the day playing around with ZFS on a 4-disk SunFire V240. It's really an incredibly flexible filesystem. I'm not yet convinced that I will get it running on some decent X86 hardware. It installed properly on my old AMD 3000XP, but my more recent Core2Duo-system was less convincing.

What problems did you have with the C2D system? My new storage server will most likely be a C2D-based.

[poodel]

lizardking009, on Jan 4 2007, 11:31 PM, said:

What problems did you have with the C2D system? My new storage server will most likely be a C2D-based.

Nothing definitive yet, but my P5W DH had some issues at install time... didn't detect the onboard network cards and some of the SATA controllers, etc. Nothing recent in the official Sun:s HW compatibility list either, so it's off to the forums for me.

[lizardking009]

poodel, on Jan 4 2007, 06:43 PM, said:

lizardking009, on Jan 4 2007, 11:31 PM, said:

What problems did you have with the C2D system? My new storage server will most likely be a C2D-based.

Nothing definitive yet, but my P5W DH had some issues at install time... didn't detect the onboard network cards and some of the SATA controllers, etc. Nothing recent in the official Sun:s HW compatibility list either, so it's off to the forums for me.

You're using Solaris 10 or OpenSolaris?

[Haversian]

poodel, on Jan 4 2007, 01:23 PM, said:

Write performance in RAID-Z doesn't seem to be a problem at all.

One of the SUN blogs has some numbers that you'll probably find informative. Basically RAID-Z trades small random reads for everything else. Small random read performance will not scale with number of drives in a RAID-Z. The trade-off is that it's faster for everything else. In particular, it writes faster than a mirror or RAID-10, while having RAID5-like read performance.

[poodel]

lizardking009, on Jan 4 2007, 11:52 PM, said:

poodel, on Jan 4 2007, 06:43 PM, said:

lizardking009, on Jan 4 2007, 11:31 PM, said:

What problems did you have with the C2D system? My new storage server will most likely be a C2D-based.

Nothing definitive yet, but my P5W DH had some issues at install time... didn't detect the onboard network cards and some of the SATA controllers, etc. Nothing recent in the official Sun:s HW compatibility list either, so it's off to the forums for me.

You're using Solaris 10 or OpenSolaris?

I was testing with Solaris10 06/11, but that's not a requirement. OpenSolaris better you think?

[poodel]

Haversian, on Jan 5 2007, 12:04 AM, said:

One of the SUN blogs has some numbers that you'll probably find informative. Basically RAID-Z trades small random reads for everything else. Small random read performance will not scale with number of drives in a RAID-Z. The trade-off is that it's faster for everything else. In particular, it writes faster than a mirror or RAID-10, while having RAID5-like read performance.

Should be excellent for the home usage server. Haven't seen if it can do anything useful with different size disks yet, but I guess that would be asking to much.

[Roxor McOwnage]

poodel, on Jan 4 2007, 08:01 PM, said:

Should be excellent for the home usage server. Haven't seen if it can do anything useful with different size disks yet, but I guess that would be asking to much.

People have been talking about RAIDZ, which is basically a smarter RAID5 + ZFS (and RAIDZ2 is a smarter RAID6). But looking outside it's RAID5'ish benefits, ZFS by itself plays _very_ well with differently sized disks. If I put a 100GB and 200GB in a ZFS "pool" (ignoring any mirroring/striping) it will intelligently stripe reads/writes across those 2 drives without me having to tell it specifically to stripe anything. ZFS will automagically use every disk/spindle it has control over. And if you add a 3rd disk (say a 320GB) dynamically to the pool, it will instantly start striping over all 3 drives (as much as it can).

In Linux you either have to us DM to explicitly force a fixed stripe size across equally sized partitions, or use LVM to slam mismatched sized drives together into one big device to put a filesystem on. In the DM case ZFS is better because it writes variable-sized stripes... if you write a lot of data it writes wide/large stripes... with a tiny bit of data it writes small/narrow stripes. Gone are the days you fiddle with a half-dozen RAID stripe sizes and run bonnie++ for hours to find your "ideal" size. As for LVM... when you're simply spanning different disks it's always reading/writing to just one device.... ZFS will read/write to all devices.. no comparison there: ZFS is faster and scales with the number of disk you have

What does this mean to a home user using vanilla ZFS? Go out and buy another larger drive and slap it in your computer and as soon as you run one command to add it to your existing ZFS pool your filesystem not only got larger (no running special 'grow' operations, or reformatting), it also got faster (same load spread over more spindles). Sweet!

I have 2.4TB across 14 disks in a Gentoo system.. split into 2 RAID5 arrays built by DM... using LVM to slam those 2 arrays into one big filesystem. I know what Linux has to offer in filesystems and volume managment. ZFS is simply better in almost every possible way. The downsides are Solaris x86 supports a more limited set of IDE/SATA chipsets than Linux... and Solaris 10 isn't exactly a desktop OS

Bit

[poodel]

OK, ZFS (and Solaris) is out. Seems like there's no such thing as growing a Raid-Z device by adding another disk. Based on the other recommendations here, it's either softraid5 or hardraid5, and XFS as a filesystem.

jpiszcz - Your mediocre write speeds are on par with what I've seen on other software raid5 solutions. I'm not so sure you're PCI limited, it's just that raid5 read-calculate-write thing that will make all writes slow.

[jpiszcz]

poodel, on Jan 5 2007, 07:18 PM, said:

OK, ZFS (and Solaris) is out. Seems like there's no such thing as growing a Raid-Z device by adding another disk. Based on the other recommendations here, it's either softraid5 or hardraid5, and XFS as a filesystem.

jpiszcz - Your mediocre write speeds are on par with what I've seen on other software raid5 solutions. I'm not so sure you're PCI limited, it's just that raid5 read-calculate-write thing that will make all writes slow.

Well I just ordered 150GB Raptor x 4 and I plan on putting them on the ICH8R chipset & possibly spread across multiple PCI-e x1 cards, we'll see what speeds I get then

Justin.