Jump to content


Photo

LSI 9270 shredding my hard drives?


  • You cannot start a new topic
  • Please log in to reply
8 replies to this topic

#1 lunadesign

lunadesign

    Member

  • Member
  • 151 posts

Posted 21 August 2014 - 12:49 AM

I've got a real mystery that's killing me and could use your help.

I have an LSI 9270-8i in a new Supermicro X9SRE-F that's running Windows 8.1.  One of the virtual drives is a RAID 1 with two 4TB WD Se (WD4000F9YZ) drives.  I ran a test where I copied a folder with 350 GB of virtual machines to the virtual drive twice.  Then, I used Beyond Compare to do a binary comparison of the two folders on the virtual drive.

For the first 95%, the controller is smart and is using one HDD to read the first folder and the other HDD to read the second folder.  According to Task Manager, I'm seeing about 350 MB/s of read performance.

Towards the very end of the test, something happens and the controller is suddenly only reading from one HDD, the performance drops to 10 MB/s and I hear a strange mechanical groaning/growling noise from one of the hard drives.  As soon as I cancel the test, the noise stops.  It's also important to note that if only one drive was being used but things were otherwise normal, I should be seeing at least 80 MB/s.

If I stop the test and re-start it relatively soon, I can reproduce.  If I re-start on the first 95% of data, it's great.  If I re-start on the last 5% of data, it's super slow and the noise returns.  What's interesting is if I let the system sit for a while and re-start on the last 5%, it's fine.

I've already tried swapping the controller and recreating the virtual drive and re-running and had the same behavior.

Does anyone have any ideas what's causing this?  I desperately need to get this system up and running but don't want the controller shredding the drives.
 


#2 lunadesign

lunadesign

    Member

  • Member
  • 151 posts

Posted 25 August 2014 - 04:59 PM

Quick update:

I've tried all of the following and have been unable to stop this problem from happening:
1) Installing the 9270-8i in a PCIe 2.0 slot
2) Trying four different 9270-8i controllers (2 originals, 2 recently received from LSI Support)
3) Installing on a different motherboard (same model but with different proc and no other cards installed)
4) Trying with a base install of Win 8.1
5) Trying with a base install of Win 7
6) Disabling PCI-X slots in the motherboard BIOS
7) Trying a different app to trigger two simultaneous reads from the RAID 1
8) Trying different read ahead / write back settings
9) Trying different versions of the LSI driver
10) Trying another set of WD4000F9YZ drives
11) Trying a smaller set of test data

I'm running out of things to try although I'm currently setting up two 3TB WD Red drives (WD30EFRX) to see if somehow the drive model is part of the equation. I'll also try a single drive config as suggested above but I really doubt that's going to trigger it as the problem seems to be related to the controller's ability to read from two drives in a RAID 1 simultaneously to speed up read performance.

LSI Support is also trying to reproduce this.

Any other ideas?


Also, any thoughts on Adaptec Series 8 controllers? I'm currently using LSI exclusively but if I can't get this working ASAP, I'm tempted to give Adaptec a try. I'd be curious for any thoughts on the quality of their hardware, software and support as compared to LSI. (But please, no religious wars. :))


#3 lunadesign

lunadesign

    Member

  • Member
  • 151 posts

Posted 26 August 2014 - 07:04 PM

UPDATE - I've made some progress in figuring this out.
 
It looks like I've got two problems:
1) The LSI controller should be reading from both drives in the RAID 1 VD but sometimes decides to only read from one.  I can reproduce this on WD Se drives and WD Red drives.
2) The WD Se drives, even when connected to the motherboard, get angry when hit with two long read operations at the same time.
 
With regards to problem #2, consider the following test when connected to the motherboard's controller and reading large (10-15 GB) files
A) WD Red, one read process:  Read rate is 150 MB/s
B) WD Red, two read processes:  Read rate is 140 MB/s
C) WD Se, one read process:  Read rate is 180 MB/s
D) WD Se, two read processes:  Read rate is 10-15 MB/s and drive makes growling sound
 
I've put in a call to WD to see if there's a firmware update for this drive.  I got to Level 2 support but apparently there's only one Level 3 guy and he was out today so I'll try again tomorrow.

#4 continuum

continuum

    Mod

  • Mod
  • 3,581 posts

Posted 27 August 2014 - 02:59 AM

Sounds like a firmware issue, either on LSI or on WD's end, very possibly the interaction between the two.

 

BTW, does this only show in a 2-disk RAID1, or does this also show in a 4-disk RAID10?


#5 lunadesign

lunadesign

    Member

  • Member
  • 151 posts

Posted 27 August 2014 - 09:30 AM

I assume you're referring to the first problem (RAID controller sometimes reads from one RAID1 drive instead of two).  If so, that's a good question.  I'll see if I can borrow a drive from another system to give RAID10 a try.

 

If you're referring to the second problem (super slow/noisy performance with 2 read operations on same physical disk), I've got that reproducing even on the onboard controller (Intel C602 chipset) so LSI is out of that equation.


#6 lunadesign

lunadesign

    Member

  • Member
  • 151 posts

Posted 28 August 2014 - 05:03 PM

UPDATE:
 
It looks like I've got a fix for problem #2.  WD's Level 3 support guy was really helpful and provided me with the latest firmware (01.01A02) for the Se drives.  Now, when I hit a single WD drive with two long read ops, the drive deals with it very well.  If I start both read ops at the same time, the drive struggles like before for about a second or two (with a little of the growling noise) and then quickly (within about 4-5 seconds) ramps up to 170 MB/s.  If I start the second read op after the first one, it skips the struggling stage and settles in at around 170 MB/s.  I'm still testing to make sure that this firmware didn't make a huge sacrifice in other areas to improve multi sequential read performance but so far it looks really good.
 
I'm still working with LSI on problem #1 (controller doesn't read from both drives when it should).

#7 continuum

continuum

    Mod

  • Mod
  • 3,581 posts

Posted 29 August 2014 - 01:23 AM

Glad a drive firmware update at least partially helped!

 

 

 

If you're referring to the second problem (super slow/noisy performance with 2 read operations on same physical disk), I've got that reproducing even on the onboard controller (Intel C602 chipset) so LSI is out of that equation.

i was referring to both problems, but didn't realize you had reproduced the second problem on a different controller as well.


#8 lunadesign

lunadesign

    Member

  • Member
  • 151 posts

Posted 29 August 2014 - 02:57 PM

UPDATE #2:
 
The new WD Se firmware seems solid.  I ran some benchmarks on a single drive before and after the firmware update and the results were nearly identical except the updated drive is a bit slower (~10%) on small random writes.  The benchmark tests didn't trigger the big problem I was seeing with two readers but my own testing so far (connected to motherboard) shows that this problem has been completely solved.  I've updated two of my drives and am trying them on the LSI in a RAID1 configuration today.

#9 lunadesign

lunadesign

    Member

  • Member
  • 151 posts

Posted 29 August 2014 - 09:24 PM

UPDATE #3:
 
The RAID 1 with two WD Se drives (both with upgraded firmware) worked well.  The LSI card still seems to be acting oddly and not using both drives as much as it should be for simultaneous reads.  When the LSI card is only using one drive and there are multiple reads going on, the WD Se (with upgraded firmware) works well.  For most of the time, the single drive runs at or near the speed it would be if a single reader was reading a single drive.  Every few minutes, I saw the single drive fall for about 2 seconds to the 10-15 MB/s level with the growling noise but otherwise worked great.
 
I'm still working with LSI on the controller issue.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users