Recommended Posts

As most of you already know; I have been working on doing some ZFS testing.

The upside; I got some results available.

Downside: I started running into system errors partway through the testing, so wasn't able to complete the RAID-Z tesitng like I had originally planned on doing.

It does; however still yield some rather interesting "caveats" to ZFS and it's ability to do striping and mirroring.

For those that don't know; here's a quick overview/review of what I've done (including system specs) etc.


AMD Duron 1.8 GHz


256 MB PC2100 DDR SDRAM (non-ECC, unbuffered)

Sapphire ATi Radeon 9250 128MB

Adaptec 29160N Ultra160 SCSI HBA*

Hitachi 73 GB 10krpm U160 (for OS)

3x Hitachi 73 GB 10krpm U160

Netgear GA311 GbE NIC (RTL8169)


Solaris 10 6/06


1) OS drive was installed using UFS. Apparently, you still can't install Solaris onto a ZFS array (yet).

2) 10Gfile is 1024*1024*10240 bytes (10,737,418,240 bytes)

3) Write command:

# time -p dd if=/dev/zero of=/test/10Gfile bs=bs count=count

4) Read command:

# time -p dd if=/test/10Gfile of=/dev/null bs=bs

In the thread where I had started to post some of the results of the testing, I compared it against a 2, 3, and 4 drive (where possible) arrays.

In this test, I changed the zfs(1M) recordsize (roughly equivalent to stripe size in a standard RAID configuration) and also the block size of the dd(1M) command to see how it reacts/behaves for different files of different sizes despite being just one single large continuous 10 GiB file.

I don't know what the default block size would be under normal operation (because with the dd(1M) command, I can specify the block size; but I don't know if I have much control over that under normal operation. Perhaps I can, I just don't know how, and as such; I don't know what the default is.)


When I was about to start the RAID-Z testing; after a while, the system would freeze and wouldn't respond. I couldn't even telnet into the system to shutdown and/or reboot it, although it did recognize that the IP was taken.

So; I started doing more testing with that specifically to try and find out what the problem is; and I started getting error messages such as: (I forget the exact wording)

cadp160_pktalloc error


DMA resource allocation error

After testing it, and trying to solve the problem, now the system sort of works (at least it stopped throwing errors, but goes freezes, and reports all the drives as 40 (rather than 160)). I'm working on having the card returned/replaced.

As such, I think that the results presented are valid; but I have no means right now to test RAID-Z as I do not have a replacement card for it.

(For some strange reason, I am having difficulties booting into Solaris install DVD with an Adaptec 2940U2W. If I don't have any drives attached to it; it has no problems. Once I attach the drives, it freezes/stops a "loading stage 2...").

Finally, on with the program...






Some rather interesting results and also reveals some interesting properties about ZFS.

For example, mirroring 3 drives REALLY slow things down. Perhaps that would change with two two-drive mirrors instead. Read speeds drop to as low as 23 MB/s and maxing out around 55 MB/s. Writes are between 10-11 MB/s.

Striping paints a (slightly) different picture. Read speeds are upto 61 MB/s, and writes are 32 MB/s. Not exactly spectacular numbers I know, but do realize that I am using less than the suggested minimum of RAM and also a 32-bit processor.

As such, I think that it gives can potentially give an indication that with better (faster) RAM, and faster (64-bit) processors that it has a lot of room to improve.

For me, I am more concerned about write speeds because I think that it is the slowest of the operations that I can/will do.

Unfortunately, I don't have anything to compare it against (i.e. vs PATA or SATA drives).

Source for the results (in Excel format) can be downloaded here.

(I don't have any other place to upload the pictures to, so sorry if they take a while to load.)

Share this post

Link to post
Share on other sites


Since I am waiting for replacement card to arrive (and I'm also thinking of getting one or two extra cables just in case, which have yet to be ordered); I decided to try ZFS on PATA drives instead (cuz I had a few of those floating around too.)

The good news is that I did manage to get some data for it; the bad news is that it wasn't a 3-drive array like it was with the SCSI system. I could only find one drive in addition to the one required for the OS installation. (Different capacities and slightly different generations, both Hitachi/IBM Deskstars, one's 80 GB and the other's 160 GB).



Originally, I had started charting CPU usage, but then I stopped because I figured that if it has to do the parity calculations (and really, any other operation must pass through the CPU at least once), that pending CPU speed, type, load, etc. will dictate how well (or how poorly) the system is going to function.

What IS interesting, however; is that if you look at the source for both tests, the CPU time is roughly the same across the board, regardless of whether it is 1 drive PATA or 3-drive SCSI.

I initially thought that with the SCSI HBA that it would take some load off of it, but it doesn't, which gives an interesting indication/insight into how ZFS operates in that, regardless of HBA interface; all operations must pass through the CPU.

Therefore; suppose you had a SATA RAID controller; and you put the drives in JBOD mode so that you can let ZFS handle everything else, the onboard hardware processor (e.g. i960) plays little to no role in the performance of the storage array/system.

I've yet to be able to test either a 9-, 15-, or 18- drive array (i.e. 3x3, 3x5, or 3x6) where each of the logical drives comprise of either 3, 5, or 6 physical drives, in either a RAID5 or RAID50 array of its own, and then using ZFS to manage three of such arrays.

Hopefully that will determine whether it's better to just get multiple controllers (straight, no HW RAID capabilities) vs. one controller that has HW RAID capabitilies and see it's effect on performance.

Also, given the similiarities in CPU usage time, it is DEFINITELY indicative of the importance of the system memory <-> CPU configuration.

64-bit systems, and in particular Opterons will help because of the 3 HT links. Couple that with an ondie memory controller, and also being able to support faster memory would definitely help improve transfer rates.

If your system is read heavy, then you are limited by the capabilities of the drive (as seen in the 1-drive read surface plot in how it has a plateau.) If your system is write heavy (which mine tends to be), then there's a lot that can be done to improve the performance of the system.

Another note is that I didn't test it with multiple IOs, just single user. I don't have an accurate means of measuring/testing performance with multiple IOs because I know that the moment that I launch two of the same commands, a) there's going to be a time offset, and B) it will be on the host system and both would be trying to fight for system resources.

I don't know if that is really representative of a multi-user environment.

There is supposed to be an embedded zfs(1M) test command, but I haven't tinkered with that much as of yet.

I almost forgot: Source

Edited by alpha754293

Share this post

Link to post
Share on other sites

I fixed the SCSI card problem, and it's back up to being 160 MB/s at the interface (same card, just reflashed the BIOS seemed to work/help fix it. Hey...whatever works, I suppose.)

So, I'm on my way now to reinstall Solaris and I am probably going to re-run all of the tests, in addition (hopefully; keeping my fingers crossed) be able to successfully complete the RAID-Z testing as well.

Stay tuned.

Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now