I also have a small filserver which is running linux with software raid and LVM2 (more on this later.)
I did however look at opensolaris, but the problem i saw was that it was not possible to exspand an RAID-Z (z=1 (raid5), z=2 (raid6), z=n (n parity drives)). So the only way to exspand your pool was to create a whole new raid array and then add it to the pool (and then push some other stuff out of the pool. I am however in love with zfs but as my setup is rather small i cant be using this exspand strategy (which big setups could be using).
So i went for an easy (and cheap) setup:
OS: Ubuntu (any linux distro would propably do)
RAID card: Intel RAID Controller SASUC8I (bringing 8xSata-II without SAS exspanders)
Disks: 2x1TB WD Caviar GreenPower (OS – Raid1) 5x2TB WD20EADS (raid6)
All my raid is linux software raid, after a few corrupt controllers from Adaptec, 3ware and Promise, I desided I needed to be “hardware independent”, so the only reason I used my raid card is to get all those nice cheap SATA-Ports.
Storage: From the ground up, I have created a partition on each 2tb drive using the whole disk. I now build my RAID 6 on these partitions (linux software raid cant take raw disks, it only works on partitions. It has its ups/downs). I use this partition as a PE device on LVM and in LVM I create a disk, which ive placed EXT4 on, and mounted.
I can now exspand my already created raid volumes, and I can add raid volumes to LVM and remove other opsoleded raidsets (2tb is not mutch in 3 years
With 5disk raid 6 using WD20EADS im getting ~160MB/s writespeed, and higher READ. So 1gbit network is not a problem for me.
A Thing with LVM that might interest you…. Lets say you make your 2x15disk raid6 in linux software raid. You now have 2 sets which you add to your LVM, you now make your Logical Volume in LVM, and tells LVM that you would like to make sure that LVM places data on 2 different PE devices… (RAID1 in LVM). Now you have your failover I do however not know if you can convert from raid1 LV to nonraid LV (when you need more than 15 disk raid 6 can give you. But some google might help you there.