blakerwry

Linux Soft Raid 5 Crash...

Recommended Posts

As some may know, I have a software RAID5 array setup in linux. It is a 4 drive raid 5 array of maxtor DM+9's, each 160GB. FS is reiser.

Last week a drive died and left my computer frozen.

I turned off the computer and rebooted and confirmed the hard error. There were several paragraphs of boot errors letting me know that hdf had died and the array could not be started.

I shutdown, removed the disk.. and restarted. The machine ran as normal with the exception of the boot errors letting me know the array could not be started.

I advance rma'd the drive to maxtor and recieved the new drive today.

I reviewed the recovery process that would be necessary to rebuild the array.

Here's where things went downhill.

1) The 1st recovery step is to replace the failed disk and partition accordingly... done. reboot and see if the array rebuilds. it didn't.

2) The next recovery option is to use the raidhotadd command to add the new disk to the array and start the rebuild process. 1 problem, the array will not start... not even in degraded mode.

3) The last option I found is that if the array was not shut down it may be unclean, making it unable to initialize correctly. There is nothing to fix this, but you can ignore it by marking the failed disk as failed and then recreating the array using mkraid. I tried this, the array is created and apears to load(although complains), however I cannot mount the array. I get an error that reiserfs cannot be found on the array... could be the wrong fstype, a bad option, or bad superblock...

If I use the "cat" command to view the contents of /dev/md0 it is blank. However, a cat of an individual drive does show data on all 3 good drives, leading me to believe that the array is actually not loading.

lsraid shows the following:

[root@localhost root]# lsraid -a /dev/md0
lsraid: Device "/dev/hdf1" does not have a valid raid superblock
lsraid: Device "/dev/hdf1" does not have a valid raid superblock
lsraid: Device "/dev/hdf1" does not have a valid raid superblock
[dev   9,   0] /dev/md0         B0098DF5.863FC629.C1206C7E.89AB4885 online
[dev  33,   1] /dev/hde1        B0098DF5.863FC629.C1206C7E.89AB4885 good
[dev   ?,   ?] (unknown)        00000000.00000000.00000000.00000000 missing
[dev  34,   1] /dev/hdg1        B0098DF5.863FC629.C1206C7E.89AB4885 good
[dev  34,  65] /dev/hdh1        B0098DF5.863FC629.C1206C7E.89AB4885 good

here is my /etc/raidtab file

raiddev /dev/md0
       raid-level      5
       nr-raid-disks   4
       nr-spare-disks  0
       persistent-superblock 1
       parity-algorithm        left-symmetric
       chunk-size      128
       device          /dev/hde1
       raid-disk       0
       device          /dev/hdf1
       failed-disk       1
       device          /dev/hdg1
       raid-disk       2
       device          /dev/hdh1
       raid-disk       3

when I try to start the array in degraded mode using the raidstart command, I get the following error:

[root@localhost root]# raidstart /dev/md0
md: invalid raid superblock maginc on hdf1
md: hdf1 has invalid sb, not importing!
md: could not import hdf1, trying to run array nevertheless
md: md0: raid array is not clean -- starting background reconstruction
raid5: cannot start dirty degraded array for md0
RAID5 conf printout:
--- rd:4 wd:3 fd:1
disk 0, o:1, dev:hde1
disk 2, o:1, dev:hdg1
disk 3, o:1, dev:hdh1
raid 5: failed to run raid set md0
md: pers->run failed...
md : do_md_run() returned -22

when I am able to load the array using mkraid I get the following error, the array does apear to load anyway... although not correctly.

handling MD device /dev/md0
analyzing super-block
disk0: /dev/hde1, 160079661kB, raid superblock at 160079552kB
disk1: /dev/hdf1, failed
disk2: /dev/hdg1, 160079661kB, raid superblock at 160079552kB
disk3: /dev/hdh1, 160079661kB, raid superblock at 160079552kB
raid5: cannot start dirty degraded array for md0
RAID5 conf printout:
--- rd:4 wd:3 fd:1
disk 0, o:1, dev:hde1
disk 2, o:1, dev:hdg1
disk 3, o:1, dev:hdh1
raid 5: failed to run raid set md0
md: pers->run failed...

can someone help me initialize my array in degraded mode so I can recover my data? I do have the failed disk available to me, but per maxtor technical support I have low level formatted it to see if it was something that could be repaired without RMA... it aparently was not.

Share this post


Link to post
Share on other sites

Also, to mention. If it was not apparent, hdf was the failed disk. All disks are on the same controller... bad performance, but works.

for more information, here are the results of mdadm -E on each disk

[root@localhost root]# mdadm -E /dev/hde1
/dev/hde1:
         Magic : a92b4efc
       Version : 00.90.00
          UUID : b0098df5:863fc629:c1206c7e:89ab4885
 Creation Time : Thu Apr 15 11:56:12 2004
    Raid Level : raid5
   Device Size : 160079488 (152.66 GiB 163.92 GB)
  Raid Devices : 4
 Total Devices : 4
Preferred Minor : 0

   Update Time : Tue Jun  1 00:00:06 2004
         State : dirty, no-errors
Active Devices : 4
Working Devices : 4
Failed Devices : 0
 Spare Devices : 0
      Checksum : b50ccb0b - correct
        Events : 0.329380

        Layout : left-asymmetric
    Chunk Size : 128K

     Number   Major   Minor   RaidDevice State
this     0      33        1        0      active sync   /dev/hde1
  0     0      33        1        0      active sync   /dev/hde1
  1     1      33       65        1      active sync   /dev/hdf1
  2     2      34        1        2      active sync   /dev/hdg1
  3     3      34       65        3      active sync   /dev/hdh1
[root@localhost root]#
[root@localhost root]#
[root@localhost root]# mdadm -E /dev/hdf1
mdadm: No super block found on /dev/hdf1 (Expected magic a92b4efc, got 00000000)
[root@localhost root]# mdadm -E /dev/hdg1
/dev/hdg1:
         Magic : a92b4efc
       Version : 00.90.00
          UUID : b0098df5:863fc629:c1206c7e:89ab4885
 Creation Time : Thu Apr 15 11:56:12 2004
    Raid Level : raid5
   Device Size : 160079488 (152.66 GiB 163.92 GB)
  Raid Devices : 4
 Total Devices : 4
Preferred Minor : 0

   Update Time : Tue Jun  1 00:00:06 2004
         State : dirty, no-errors
Active Devices : 4
Working Devices : 4
Failed Devices : 0
 Spare Devices : 0
      Checksum : b50ccb10 - correct
        Events : 0.329380

        Layout : left-asymmetric
    Chunk Size : 128K

     Number   Major   Minor   RaidDevice State
this     2      34        1        2      active sync   /dev/hdg1
  0     0      33        1        0      active sync   /dev/hde1
  1     1      33       65        1      active sync   /dev/hdf1
  2     2      34        1        2      active sync   /dev/hdg1
  3     3      34       65        3      active sync   /dev/hdh1
[root@localhost root]# mdadm -E /dev/hdh1
/dev/hdh1:
         Magic : a92b4efc
       Version : 00.90.00
          UUID : b0098df5:863fc629:c1206c7e:89ab4885
 Creation Time : Thu Apr 15 11:56:12 2004
    Raid Level : raid5
   Device Size : 160079488 (152.66 GiB 163.92 GB)
  Raid Devices : 4
 Total Devices : 4
Preferred Minor : 0

   Update Time : Tue Jun  1 00:00:06 2004
         State : dirty, no-errors
Active Devices : 4
Working Devices : 4
Failed Devices : 0
 Spare Devices : 0
      Checksum : b50ccb52 - correct
        Events : 0.329380

        Layout : left-asymmetric
    Chunk Size : 128K

     Number   Major   Minor   RaidDevice State
this     3      34       65        3      active sync   /dev/hdh1
  0     0      33        1        0      active sync   /dev/hde1
  1     1      33       65        1      active sync   /dev/hdf1
  2     2      34        1        2      active sync   /dev/hdg1
  3     3      34       65        3      active sync   /dev/hdh1
[root@localhost root]#

Share this post


Link to post
Share on other sites

After installing the new disk, did you create raid autodetect (type FD) partition? You will have to reboot after creating the partition, so that the softraid can read the partition table of the new drive, recognize it as an "empty" raidable drive. Reconstructing the array should go better after that.

cat the proc mdstat as well and post the results.

Thank you for your time,

Frank Russo

Share this post


Link to post
Share on other sites

I am using Fedora Core 2 with kernel 2.6.5-1.358

yes, I did create a partition type of FD (linux softraid) on the replacedment disk and it matches the partitions of the other 3 disks (same size and type as well as same bootable flag)

on boot I get errors, and the array does not mount and does not rebuild.

md: Autodetecting RAID arrays.
md: invalid raid superblock magic on hdf1
md: hdf1 has invalid sb, not importing!
md: could not import hdf1!
md: autorun ...
md: considering hdh1 ...
md:  adding hdh1 ...
md:  adding hdg1 ...
md:  adding hde1 ...
md: created md0
md: bind<hde1>
md: bind<hdg1>
md: bind<hdh1>
md: running: <hdh1><hdg1><hde1>
md: md0: raid array is not clean -- starting background reconstruction
md: personality 4 is not loaded!
md :do_md_run() returned -22
md: md0 stopped.
md: unbind<hdh1>
md: export_rdev(hdh1)
md: unbind<hdg1>
md: export_rdev(hdg1)
md: unbind<hde1>
md: export_rdev(hde1)
md: ... autorun DONE.

Here's the output of mdstat on boot with the array not loaded

[root@localhost root]# cat /proc/mdstat
Personalities : [raid5]
unused devices: <none>

with a force load through remaking the array i get the following:

[root@localhost root]# cat /proc/mdstat
Personalities : [raid5]
md0 : inactive hdh1[3] hdg1[2] hde1[0]
     480238464 blocks
unused devices: <none>

remember, hdf1 is marked as a failed disk in my raidtab file so as not to resync the array when forcing the mkraid.

-thanks

Share this post


Link to post
Share on other sites

http://groups.google.com/groups?hl=en&lr=&...t%3D30%26sa%3DN

Found this that suggests re-ordering the IDs, but I would recommend getting out the masking tape and a sharpie before going down that route.

Still looking.

Funny thing is, I had this exact same problem once and lost all my data.

The mdctl looks like it might help as well

http://www.cse.unsw.edu.au/~neilb/source/mdctl/

mdctl --assemble /dev/hde1 /dev/hdg1 /dev/hdh1

Frank

Share this post


Link to post
Share on other sites

Disclaimer: I haven't used Linux software raid. This could be unhelpful. :(

I just feel obligated to help. :-/ You probably already have read the HOWTO?

I don't know if this is of any help, but check the 11th question on this HOWTO:

http://www.linux.com/howtos/Software-RAID-0.4x-HOWTO-4.shtml

(warning: OLD document, tools may have changed)

Also (even as this talks about multiple disk failures) this could give hints about

reviving dead raid array:

http://www.tldp.org/HOWTO/Software-RAID-HOWTO-8.html

Share this post


Link to post
Share on other sites

yeah, I tried re-ordering the drives in the raid tab file and starting the array... it didnt work, so I set the raidtab back ... drives have been labelled and sharpied since installation... always a good idea...

I tried the mdadm tool to assemble the array, but there's a config file to setup and I didnt want to risk losing the array. It looks like this utility mdctl became mdadm. I'll review the linux soft raid howto and see if I can get mdadm to work...

on a side note there seems to be some problem with knoppix and this computer... it gives repeated errors on boot about the hard drives on my promise controller (the raid drives).. funny thing is, I think that it always did that though.

Share this post


Link to post
Share on other sites
Disclaimer: I haven't used Linux software raid. This could be unhelpful. :(

I just feel obligated to help. :-/ You probably already have read the HOWTO?

I don't know if this is of any help, but check the 11th question on this HOWTO:

http://www.linux.com/howtos/Software-RAID-0.4x-HOWTO-4.shtml

(warning: OLD document, tools may have changed)

Also (even as this talks about multiple disk failures) this could give hints about

reviving dead raid array:

http://www.tldp.org/HOWTO/Software-RAID-HOWTO-8.html

yup, i've been over both like glue.

Share this post


Link to post
Share on other sites

tried mdadm to assemble the array... results in much the same error when starting the array:

raid5: cannot start dirty degraded array for md0
RAID5 conf printout:
--- rd:4 wd:3 fd:1
disk 0, o:1, dev:hde1
disk 2, o:1, dev:hdg1
disk 3, o:1, dev:hdh1
raid 5: failed to run raid set md0
md: pers->run failed...
mdadm: failed to run array /dev/dm0: invalid argument

mdadm.conf:

# mdadm configuration file
#
# mdadm will function properly without the use of a configuration file,
# but this file is useful for keeping track of arrays and member disks.
# In general, a mdadm.conf file is created, and updated, after arrays
# are created. This is the opposite behavior of /etc/raidtab which is
# created prior to array construction.
#
#
# the config file takes two types of lines:
#
#       DEVICE lines specify a list of devices of where to look for
#         potential member disks
#
#       ARRAY lines specify information about how to identify arrays so
#         so that they can be activated
#
# You can have more than one device line and use wild cards. The first
# example includes SCSI the first partition of SCSI disks /dev/sdb,
# /dev/sdc, /dev/sdd, /dev/sdj, /dev/sdk, and /dev/sdl. The second
# line looks for array slices on IDE disks.
#
#DEVICE /dev/sd[bcdjkl]1
DEVICE /dev/hde1 /dev/hdf1 /dev/hdg1 /dev/hdh1
#
# If you mount devfs on /dev, then a suitable way to list all devices is:
#DEVICE /dev/discs/*/*
#
#
#
# ARRAY lines specify an array to assemble and a method of identification.
# Arrays can currently be identified by using a UUID, superblock minor number,
# or a listing of devices.
#
#       super-minor is usually the minor number of the metadevice
#       UUID is the Universally Unique Identifier for the array
# Each can be obtained using
#
#       mdadm -D <md>
#
#ARRAY /dev/md0 UUID=3aaa0122:29827cfa:5331ad66:ca767371
#ARRAY /dev/md1 super-minor=1
#ARRAY /dev/md2 devices=/dev/hda1,/dev/hda2
#
# ARRAY lines can also specify a "spare-group" for each array.  mdadm --monitor
# will then move a spare between arrays in a spare-group if one array has a failed
# drive but no spare
#ARRAY /dev/md4 uuid=b23f3c6d:aec43a9f:fd65db85:369432df spare-group=group1
#ARRAY /dev/md5 uuid=19464854:03f71b1b:e0df2edd:246cc977 spare-group=group1
#
# When used in --follow (aka --monitor) mode, mdadm needs a
# mail address and/or a program.  This can be given with "mailaddr"
# and "program" lines to that monitoring can be started using
#    mdadm --follow --scan & echo $! > /var/run/mdadm
# If the lines are not found, mdadm will exit quietly
MAILADDR root
#PROGRAM /usr/sbin/handle-mdadm-events

ARRAY /dev/md0 level=raid5 num-devices=4 devices=/dev/hde1,/dev/hdg1,/dev/hdh1

looks like the array is really stubborn about starting.

unfortunately I do have some data that is not backed up that was being stored on the array. Fortunately, the most important stuff was backed up... but the backup is a bit out of date.

Share this post


Link to post
Share on other sites

Just guessing here...

ARRAY /dev/md0 level=raid5 num-devices=4 devices=/dev/hde1,/dev/hdg1,/dev/hdh1

Is this line correct, or do you need to change num to 3 or add /dev/hdf1?

mdadm: failed to run array /dev/dm0: invalid argument

Why does it say dm0, not md0?

Share this post


Link to post
Share on other sites

If you don't mind, could you please post of the output of:

fdisk -l /dev/hde /dev/hdf /dev/hdg /dev/hdh

(that's a minus "ell" not a one, in case you're using a sans-serif font and aren't copying and pasting).

Share this post


Link to post
Share on other sites

All i can offer is my hopes that is gets resolved soon! I know how important data is to people / companies. But once you ge the issue resolved, i know you will, cuz you have determination to get the data back, will you consider going hardware raid instead or not. Is this array used heavily daily?

I hope you get this fixed!!!...

SCSA

P.S. Good luck!

Share this post


Link to post
Share on other sites
Just guessing here...
ARRAY /dev/md0 level=raid5 num-devices=4 devices=/dev/hde1,/dev/hdg1,/dev/hdh1

Is this line correct, or do you need to change num to 3 or add /dev/hdf1?

mdadm: failed to run array /dev/dm0: invalid argument

Why does it say dm0, not md0?

@ anon:

1) I believe this is the correct way to set the array if there is a failed disk... I have tried with and without hdf.

2) because I was typing the error message by hand (no cut and paste) and made a typo when typing it into the forums.

@ Jeff:

[root@localhost root]# fdisk -l /dev/hde /dev/hdf /dev/hdg /dev/hdh

Disk /dev/hde: 163.9 GB, 163928604672 bytes
255 heads, 63 sectors/track, 19929 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

  Device Boot      Start         End      Blocks   Id  System
/dev/hde1   *           1       19929   160079661   fd  Linux raid autodetect

Disk /dev/hdf: 163.9 GB, 163928604672 bytes
255 heads, 63 sectors/track, 19929 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

  Device Boot      Start         End      Blocks   Id  System
/dev/hdf1   *           1       19929   160079661   fd  Linux raid autodetect

Disk /dev/hdg: 163.9 GB, 163928604672 bytes
255 heads, 63 sectors/track, 19929 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

  Device Boot      Start         End      Blocks   Id  System
/dev/hdg1   *           1       19929   160079661   fd  Linux raid autodetect

Disk /dev/hdh: 163.9 GB, 163928604672 bytes
255 heads, 63 sectors/track, 19929 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

  Device Boot      Start         End      Blocks   Id  System
/dev/hdh1   *           1       19929   160079661   fd  Linux raid autodetect
[root@localhost root]#

@Frank

I believe I was using "mdadm -As" after making the config file. Restarting the array with "mdadm -R /dev/md0". It was late and I was tired, so i may have to take another look.

Share this post


Link to post
Share on other sites

@stone cold

It's basically a half terrabyte array for my home file server. It holds ISO images, music, movies, and alot of backups.

Share this post


Link to post
Share on other sites

I found this newsgroup thread from about a week ago from someone with exactly the same problem, except that he was using SCSI drives. It turned out, he had a bad cable and replacing it fixed the problem.

http://groups.google.com/groups?hl=en&lr=&...dd8088b2&rnum=1

You may want to do an "hdparm -t /dev/hdf" to verify you're getting decent throughput to the drive (i.e. make sure it's not a hardware issue). You can also try "badblocks -c 16384 -o /tmp/badblocks.hdf -s -w /dev/hdf" to verify your drive doesn't have any surface problems (note: that's a destructive scan, so triple check your device is correct first).

Also note, you have an invalid uuid on /dev/hdf1, so you can run "uuidgen /dev/hdf1" to create one.

Good luck!

Share this post


Link to post
Share on other sites

I'm using a brand new cable that worked fine for about 2 months. It was purchased specificly for this array. The thing has been treated like gold and doesnt have a single crease or sharp bend in it. Should be good.

/dev/hde:
Timing buffer-cache reads:   656 MB in  2.00 seconds = 327.72 MB/sec
Timing buffered disk reads:  148 MB in  3.02 seconds =  49.01 MB/sec

/dev/hdf:
Timing buffer-cache reads:   664 MB in  2.01 seconds = 330.23 MB/sec
Timing buffered disk reads:  172 MB in  3.02 seconds =  56.91 MB/sec

/dev/hdg:
Timing buffer-cache reads:   664 MB in  2.00 seconds = 331.88 MB/sec
Timing buffered disk reads:  146 MB in  3.01 seconds =  48.51 MB/sec

/dev/hdh:
Timing buffer-cache reads:   664 MB in  2.01 seconds = 330.56 MB/sec
Timing buffered disk reads:  170 MB in  3.02 seconds =  56.30 MB/sec

looks like hdf and hdh are the same platter design while hde and hdg are a different design... smartctl reports look healthy for all drives.

Share this post


Link to post
Share on other sites

what bugs me is that I cannot start the array in degraded mode and I should be able to. I could care less about rebuilding the array at this point.

I'm going to try another cable and see what I can come up with.

Share this post


Link to post
Share on other sites

Okay, what about:

mdadm --assemble --force /dev/md0

If that still doesn't work, and you're getting desperate for a "hail mary" plan, try editing:

/usr/src/linux/drivers/md/raid5.c

Find the stanza:

   if (mddev->degraded == 1 &&
      mddev->recovery_cp != MaxSector) {
     printk(KERN_ERR
        "raid5: cannot start dirty degraded array for %s\n",
        mdname(mddev));
     goto abort;
  }

and comment out the "goto abort" (prepend line with //). Then recompile your kernel if mdraid was built in or re-make your modules if it was a module. After that, try doing a raidstart. Good luck.

Share this post


Link to post
Share on other sites
Then recompile your kernel if mdraid was built in or re-make your modules if it was a module. After that, try doing a raidstart.

Of course, reboot or rmmod md/modprobe md before running raidstart.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now