Well, I've upgraded from a 3ware 9500S-4LP to a 9650SE-4LPML. The setup now consists of:
Asus A8N5X motherboard (socket 939, has PCIE x4 slot)
Athlon 64 4000+ CPU
3ware 9650SE-LP4ML controller (PCIE x4)
four WD3200 RE drives in RAID5 configuration
I added the following lines to /etc/rc.d/rc.local based on 3ware's recommendations:
echo 512 > /sys/block/sda/queue/nr_requests
blockdev --setra 16384 /dev/sda
echo "deadline" > /sys/block/sda/queue/scheduler
My test filesystem is an XFS partition which is sector-aligned to a 64k boundary. Here are the test results:
Iozone: Performance Test of File I/O
Version $Revision: 3.239 $
Compiled for 64 bit mode.
Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million,
Jean-Marc Zucconi, Jeff Blomberg,
Erik Habbinga, Kris Strecker, Walter Wong.
Run began: Tue Aug 14 16:13:57 2007
Using minimum file size of 8388608 kilobytes.
Using maximum file size of 8388608 kilobytes.
Command line used: iozone -a -n 8G -g 8G -i0 -i1
Output is in Kbytes/sec
Time Resolution = 0.000002 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
KB reclen write rewrite read reread
8388608 64 192193 103294 193565 193768
8388608 128 196954 101904 193439 194035
8388608 256 198341 97858 193509 194039
8388608 512 196343 100973 193721 194119
8388608 1024 196568 102744 193266 194123
8388608 2048 197253 101690 193493 194045
8388608 4096 197329 101785 193463 194116
8388608 8192 196293 120207 193451 194102
8388608 16384 197536 136602 193776 193890
iozone test complete.
As you can see, performance is quite good - especially considering these are not especially cutting edge 320GB drives! The fact that all of the write/read/reread results are pegged around 193MB/s suggests that I am hitting a bandwidth limit somewhere. My first thought was the PCI Express slot - am I accidentally getting a 1 lane connection instead of a 4 lane connection? Each lane is 2.5Gbps. I did an "lspci -vv" and here is the 3ware entry:
04:00.0 RAID bus controller: 3ware Inc 9650SE SATA-II RAID (rev 01)
Subsystem: 3ware Inc 9650SE SATA-II RAID
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 17
Region 0: Memory at d4000000 (64-bit, prefetchable) [size=32M]
Region 2: Memory at d7001000 (64-bit, non-prefetchable) [size=4K]
Region 4: I/O ports at a000 [size=256]
Region 5: Memory at d7000000 (32-bit, non-prefetchable) [size=4K]
Expansion ROM at d6000000 [disabled] [size=128K]
Capabilities:  Power Management version 2
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities:  Message Signalled Interrupts: Mask- 64bit+ Queue=0/5 Enable-
Address: 0000000000000000 Data: 0000
Capabilities:  Express Legacy Endpoint IRQ 0
Device: Supported: MaxPayload 512 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s <128ns, L1 <2us
Device: AtnBtn- AtnInd- PwrInd-
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
Device: MaxPayload 128 bytes, MaxReadReq 512 bytes
Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s L1, Port 0
Link: Latency L0s <512ns, L1 <64us
Link: ASPM Disabled RCB 128 bytes CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x2
Capabilities:  Advanced Error Reporting
Near the end, notice it says the link speed is 2.5Gbps (which is a single lane) but the width is x2. That really doesn't make much sense to me. Also strange is the line thaat suggests the supported speed is 2.5Gbps.
Does anyone have a PCI Express 3ware or other RAID controller running under linux that you can post the "lspci -vv" entry?
I'm not sure why the rewrite is lower. Perhaps because read/write cycles are needed? If so, this further reinforces that I am hitting a bandwidth limitation somewhere. Even so, boy these are some nice numbers.
This has been a great budget system upgrade. The motherboard was a $38 shipped refurb from Newegg, I won the controller on Ebay for $271 shipped, and the CPU was on sale a few months ago for $70 (now cheaper).