Manage software RAID 1 with mdadm

On servers with two disks, DediJet sets up software RAID 1 by default when it installs your operating system (RAID 0 or no RAID can be selected at install time instead). Linux mdadm mirrors every write to both disks, so the server keeps running if one of them fails. This guide shows how to check the array, get alerted, replace a failed disk and keep both disks bootable.

How your disks are laid out

Both disks carry the same partitions, and matching partitions are paired into md arrays. List them with:

lsblk -o NAME,SIZE,TYPE,MOUNTPOINT

The exact layout depends on the OS image. A typical UEFI install on two NVMe drives looks like this:

NAME          SIZE TYPE  MOUNTPOINT
nvme0n1     931.5G disk
├─nvme0n1p1   512M part  /boot/efi
├─nvme0n1p2     1G part
│ └─md0      1022M raid1 /boot
└─nvme0n1p3   930G part
  └─md1     929.3G raid1 /
nvme1n1     931.5G disk
├─nvme1n1p1   512M part
├─nvme1n1p2     1G part
│ └─md0      1022M raid1 /boot
└─nvme1n1p3   930G part
  └─md1     929.3G raid1 /

• NVMe naming: nvme0n1 is controller 0, namespace 1, and its partitions add a p: nvme0n1p1. SATA and SAS disks are named sda, sdb, with partitions sda1, sda2. Controller numbers can change between boots, so identify a disk by its serial number, never by its name alone.
• EFI system partition (ESP): the firmware reads it directly and knows nothing about mdadm, so on UEFI servers it usually sits outside the arrays. Only the first disk’s ESP is mounted on /boot/efi; keeping the second copy in sync is up to you (last section of this guide).

Check the array status

cat /proc/mdstat

A healthy mirror shows [2/2] [UU]: two members expected, two up.

md1 : active raid1 nvme1n1p3[1] nvme0n1p3[0]
      974452736 blocks super 1.2 [2/2] [UU]
      bitmap: 2/8 pages [8KB], 65536KB chunk

A degraded mirror shows [2/1] and an underscore in place of the missing member: [U_] or [_U]. To see which member failed:

mdadm --detail /dev/md1

             State : clean, degraded
    Active Devices : 1
   Working Devices : 1
    Failed Devices : 1

    Number   Major   Minor   RaidDevice State
       0     259        3        0      active sync   /dev/nvme0n1p3
       -       0        0        1      removed

       1     259        7        -      faulty   /dev/nvme1n1p3

Most distributions also schedule a periodic consistency check. While it runs, /proc/mdstat shows check with a progress bar: that is normal and runs at low priority.

Get an email when a disk fails

The mdmonitor service watches all arrays and sends an email when a member fails or an array becomes degraded. Set the recipient in /etc/mdadm/mdadm.conf (Debian, Ubuntu) or /etc/mdadm.conf (AlmaLinux, Rocky Linux):

MAILADDR you@example.com

mdadm hands the message to /usr/sbin/sendmail, so the server needs a mail transfer agent: Postfix, or a small relay such as msmtp that forwards through your mail provider. Restart the monitor, then send a test alert for every array:

systemctl restart mdmonitor
mdadm --monitor --scan --oneshot --test

Prefer chat or paging? A PROGRAM /usr/local/bin/raid-alert line in the same file runs your script on every event (it receives the event name, the array and the member device). If you already run Prometheus, node_exporter exposes the array state in its node_md_* metrics.

Replace a failed disk

The example assumes nvme1n1 failed and nvme0n1 is healthy. Replace the names with yours and read every command twice.

1. Record the serial numbers of the failed disk and of the healthy one. If the failed disk has vanished completely, the healthy disk’s serial is enough.

   Shell

   lsblk -d -o NAME,SIZE,MODEL,SERIAL
   smartctl -i /dev/nvme1n1

2. Remove the failed partitions from the arrays (skip any that already show as removed):

   Shell

   mdadm --manage /dev/md0 --fail /dev/nvme1n1p2 --remove /dev/nvme1n1p2
   mdadm --manage /dev/md1 --fail /dev/nvme1n1p3 --remove /dev/nvme1n1p3

3. Open a ticket. Give the serial of the disk to replace, the serial of the disk to keep, and the output of cat /proc/mdstat and smartctl -a for the failing disk. Under our SLA, a failed component is replaced within 4 hours of diagnosis, 24/7. Depending on the chassis, the swap may need a short shutdown: agree on a time window in the ticket.

4. Copy the partition table to the new disk once it is installed (package gdisk). Check the new disk’s name and serial with lsblk first. With GPT, the usual case, --replicate names the new disk and the last argument is the healthy source. Then give the copy its own GUIDs:

   Shell

   sgdisk --replicate=/dev/nvme1n1 /dev/nvme0n1
   sgdisk --randomize-guids /dev/nvme1n1

   For legacy MBR disks (lsblk -o NAME,PTTYPE shows dos), use sfdisk -d /dev/nvme0n1 | sfdisk /dev/nvme1n1 instead.

5. Add the new partitions to the arrays. The rebuild starts immediately:

   Shell

   mdadm --manage /dev/md0 --add /dev/nvme1n1p2
   mdadm --manage /dev/md1 --add /dev/nvme1n1p3

6. Watch the rebuild. The server stays usable meanwhile, with slower disk I/O.

   Shell

   watch -n 5 cat /proc/mdstat

   Output

   md1 : active raid1 nvme1n1p3[2] nvme0n1p3[0]
         974452736 blocks super 1.2 [2/1] [U_]
         [====>................]  recovery = 22.5% (219351680/974452736) finish=62.9min speed=200000K/sec

   The kernel keeps rebuild speed between dev.raid.speed_limit_min and dev.raid.speed_limit_max (KiB/s per disk). The default ceiling, 200000 KiB/s (about 200 MB/s), is what the example above shows. On an idle NVMe server you can raise it temporarily with sysctl -w dev.raid.speed_limit_max=2000000.

Keep both disks bootable

Your data is now mirrored again, but the boot loader lives outside the arrays. Check how the server boots:

[ -d /sys/firmware/efi ] && echo UEFI || echo BIOS

Legacy BIOS

Install GRUB on the new disk: grub-install /dev/nvme1n1 on Debian and Ubuntu (then run dpkg-reconfigure grub-pc and select both disks so future updates reach them), or grub2-install /dev/nvme1n1 on AlmaLinux and Rocky Linux.

UEFI

1. Format and fill the second ESP (packages dosfstools and rsync):

   Shell

   mkfs.vfat -F 32 /dev/nvme1n1p1
   mkdir -p /mnt/esp2
   mount /dev/nvme1n1p1 /mnt/esp2
   rsync -rt --delete /boot/efi/ /mnt/esp2/
   umount /mnt/esp2

2. Add a firmware boot entry for that disk. List the existing entries with efibootmgr -v and reuse the loader path of your OS, for example \EFI\ubuntu\shimx64.efi, \EFI\debian\shimx64.efi or \EFI\almalinux\shimx64.efi:

   Shell

   efibootmgr --create --disk /dev/nvme1n1 --part 1 --label "Linux (disk 2)" --loader '\EFI\ubuntu\shimx64.efi'

3. Re-sync after boot loader updates. The ESP rarely changes, mostly when GRUB or shim packages are updated: repeat the rsync step after those updates, or run it from a cron job. On Ubuntu, dpkg-reconfigure grub-efi-amd64 can register both ESPs so GRUB updates are written to each.

4. Add nofail to the /boot/efi line in /etc/fstab (for example umask=0077,nofail). Without it, a server whose first disk died may stop in emergency mode because it cannot mount that disk’s ESP.

To test, pick the second disk once in the firmware boot menu from the IPMI console.

If the server does not boot

Open the IPMI console to see where the boot stops; the firmware boot menu lets you choose the other disk. If that does not help, start rescue mode from your dashboard, assemble the arrays and repair the system from there:

mdadm --assemble --scan
cat /proc/mdstat

Next steps

• Catch failing disks before they drop out of the array: check disk health with smartctl and nvme-cli.
• Protect what RAID cannot: add backup space and test a restore.
• Stuck in the middle of a disk swap? Contact support: hardware issues are handled by engineers, 24/7.