Following my previous blog about “Creating Raid5 under Linux RHEL5.4 using md, lvm and ext4 filesystem.”

I will add some information on how to test if the raid actually does work well. Here is how I did it.

While the server is working pull out a disk (in my case it was /dev/sdg who was removed) and check it progress on reconstruction. In a separate terminal start this:

• watch cat /proc/mdstat

Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdb[0] sdh[6] sdi[7](S) sdf[4] sde[3] sdd[2] sdc[1]
7325692480 blocks level 5, 64k chunk, algorithm 2 [6/5] [UUUUU_]
[==========>..........]  recovery = 52.6% (771078272/1465138496) finish=127.1min speed=90987K/sec

I decided to put back the disk in the system just a few seconds after the reconstruction has started

• cat /proc/partitions

major minor  #blocks  name

8     0  488386584 sda
8     1      72261 sda1
8     2    2104515 sda2
8     3  465724350 sda3
8     4   20482875 sda4
8    16 1465138584 sdb
8    32 1465138584 sdc
8    48 1465138584 sdd
8    64 1465138584 sde
8    80 1465138584 sdf
9     0 7325692480 md0
8   112 1465138584 sdh
8   128 1465138584 sdi
253     0 1073741824 dm-0
253     1 1073741824 dm-1
253     2 1073741824 dm-2
253     3   10485760 dm-3

Then I added the newly found device /dev/sdi as a spare by simply typing

• mdadm /dev/md0 -a /dev/sdi

for some unkown reason (cat /proc/mdstat) still saw an unavailable disk sdg, the one I removed and that was no longer available to the system. In order to remove the previous faulty device, in my case sdg, I had to stop and re enable LVM, (the volume group) and the meta device (md0). This is how.:

Stop all process that could be using those disks and unmount all disks.

• /etc/init.d/httpd stop
• /etc/init.d/mysqld stop
• umount -a

Find your active VG

• vgscan

Reading all physical volumes.  This may take a while…
Found volume group “vg01″ using metadata type lvm2

Disable the volume group vg01 and stop the meta device md0

• vgchange -an vg01
• mdadm --stop /dev/md0

Re-Enable the volume group vg01 and stop the meta device md0, then remount all.

• vgchange -ay vg01
• mdadm -A /dev/md0
• mount -a

You can now check the detail of the meta device :

• mdadm --detail /dev/md0

/dev/md0:
Version : 0.90
Creation Time : Wed Mar 10 17:40:12 2010
Raid Level : raid5
Array Size : 7325692480 (6986.32 GiB 7501.51 GB)
Used Dev Size : 1465138496 (1397.26 GiB 1500.30 GB)
Raid Devices : 6
Total Devices : 7
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Wed Mar 24 18:19:16 2010
State : clean, degraded, recovering
Active Devices : 5
Working Devices : 7
Failed Devices : 0
Spare Devices : 2

Layout : left-symmetric
Chunk Size : 64K

Rebuild Status : 55% complete

UUID : be242eb0:3fe5ec86:4b698eb2:c9f1759e
Events : 0.802

Number   Major   Minor   RaidDevice State
0       8       16        0      active sync   /dev/sdb
1       8       32        1      active sync   /dev/sdc
2       8       48        2      active sync   /dev/sdd
3       8       64        3      active sync   /dev/sde
4       8       80        4      active sync   /dev/sdf
6       8      112        5      spare rebuilding   /dev/sdh

7       8      128        -      spare   /dev/sdi

You can see that sdg has disappeared from the list and the new sdi (previouly know as sdg) is now the new device while the previous space device sdh is now been reconstructed.

Rejean.