Boot into rescue mode and repair your server

When the installed system no longer boots or no longer lets you in, rescue mode gives you a working Linux environment with full access to the disks. From there you can repair the installation, or copy your data off before a reinstall.

What rescue mode is

The rescue system is a Linux live system that your server boots over the network and runs entirely in memory. Your disks are left untouched: nothing is mounted, formatted or changed until you decide to. You log in over SSH as root and get a normal shell to work with disks, RAID arrays and filesystems. Rescue mode is free, as often as you need it.

For a lockout caused by the network or firewall configuration, the IPMI console is often quicker, because it logs you in to the installed system directly.

Boot into the rescue system

1. Enable rescue mode. In your dashboard, open the server, then choose Rescue. Note the temporary root password you are given.

2. Reboot the server from the dashboard, or with reboot if it still responds. It starts the rescue system instead of your installation.

3. Log in over SSH with the temporary password. The rescue system has its own host key; a separate known-hosts file keeps it apart from your server’s real key:

   Shell

   ssh -o UserKnownHostsFile=~/.ssh/known_hosts_rescue root@203.0.113.10

Identify the disks and assemble the RAID

lsblk -o NAME,SIZE,TYPE,FSTYPE,LABEL,MOUNTPOINT

Partitions with the type linux_raid_member belong to a software RAID array, the default on two-disk servers. Assemble the arrays, unless /proc/mdstat already lists them, and check their state:

mdadm --assemble --scan
cat /proc/mdstat

Arrays may get other names than on the installed system, for example md127 instead of md0; that is harmless. If a member is missing ([U_]), you can still work on the array, but check the disks too: see Disk health. If lsblk shows LVM2_member, activate the volume groups:

vgchange -ay
lvs

Mount the system and chroot into it

The root filesystem is the one that contains /etc/fstab, usually the largest array. Mount it and read the fstab to see which other filesystems belong to it:

mount /dev/md1 /mnt
cat /mnt/etc/fstab

If the system crashed or the console showed filesystem errors, unmount it first and run a check: fsck -f /dev/md1 for ext4, xfs_repair /dev/md1 for XFS (the FSTYPE column of lsblk tells you which). Never check a mounted filesystem.

Mount any separate /boot and /boot/efi listed in the fstab (blkid maps UUIDs to devices), then bind the virtual filesystems and switch into the installed system:

mount /dev/md0 /mnt/boot
mount /dev/nvme0n1p1 /mnt/boot/efi
mount --rbind /dev /mnt/dev
mount --rbind /proc /mnt/proc
mount --rbind /sys /mnt/sys
mount --rbind /run /mnt/run
chroot /mnt /bin/bash

The device names are examples; skip the first two lines if there is no separate /boot or /boot/efi. Inside the chroot, commands such as passwd, systemctl enable and the package manager act on your installed system, not on the rescue system.

Only need your data? Skip the chroot and copy it straight from /mnt, for example with rsync -a /mnt/var/www/ backup@backup.example.com:rescue-copy/.

Common fixes

Emergency mode at boot: fix /etc/fstab

A wrong UUID, or a disk that no longer exists, in /etc/fstab makes systemd stop in emergency mode. Compare the entries with the output of blkid, correct or comment out the bad line with vi /etc/fstab (or nano), then validate:

findmnt --verify

For disks the system does not need to boot, add nofail to the mount options, so a missing disk no longer blocks startup.

No bootloader: reinstall GRUB on both disks

With RAID 1, the bootloader belongs on both disks, so the server can start from either one. First check how the server boots:

[ -d /sys/firmware/efi ] && echo UEFI || echo BIOS

BIOS boot. Ubuntu and Debian:

grub-install /dev/nvme0n1
grub-install /dev/nvme1n1
update-grub

AlmaLinux and Rocky Linux:

grub2-install /dev/nvme0n1
grub2-install /dev/nvme1n1
grub2-mkconfig -o /boot/grub2/grub.cfg

On Ubuntu and Debian, also run dpkg-reconfigure grub-pc and select both disks, so that future GRUB updates are written to both.

UEFI boot. GRUB lives on the EFI system partition, mounted at /boot/efi. Ubuntu and Debian:

grub-install --target=x86_64-efi --efi-directory=/boot/efi
update-grub

AlmaLinux and Rocky Linux, where grub2-install is not used on UEFI systems:

dnf reinstall -y grub2-efi-x64 shim-x64
grub2-mkconfig -o /boot/grub2/grub.cfg

If the second disk has its own EFI system partition (a small vfat partition in lsblk), give it a bootloader as well. Mount it:

mkdir -p /boot/efi2
mount /dev/nvme1n1p1 /boot/efi2

Then, on Ubuntu and Debian, run grub-install --target=x86_64-efi --efi-directory=/boot/efi2 --removable, which installs GRUB at the fallback path that firmware finds without a boot entry. On AlmaLinux and Rocky Linux, copy the files with cp -r /boot/efi/EFI /boot/efi2/. Finish with umount /boot/efi2.

Reset a lost root password

passwd root

On AlmaLinux and Rocky Linux, the new password file has no SELinux label yet, and logins can fail until it gets one: do not skip the relabel step when you leave rescue mode.

Locked out by SSH or firewall settings

Move the SSH drop-in file that caused the lockout out of the way, and check that your key is still there:

ls /etc/ssh/sshd_config.d/
mv /etc/ssh/sshd_config.d/00-keys-only.conf /root/
cat /root/.ssh/authorized_keys

To start without the firewall, disable it for the next boot with the line that matches the one you use:

systemctl disable ufw
systemctl disable firewalld
systemctl disable nftables

Once the server is back, fix the rules and turn the firewall on again with systemctl enable --now followed by the same name.

Leave rescue mode

On AlmaLinux and Rocky Linux, if you changed any file inside the chroot, run touch /.autorelabel before you leave it. SELinux then relabels the filesystem at the next boot, which takes a few minutes and restarts the server once more on its own. Then exit the chroot and unmount everything:

exit
sync
umount -R /mnt

If umount reports that a target is busy, you can continue: the reboot unmounts everything. Turn rescue mode off in your dashboard if it is still enabled, then reboot the server. Watch the first boot in the IPMI console: if it stops again, the screen shows where.

Still stuck?

If the rescue system does not start, a disk is not detected, or /proc/mdstat shows a failed member, open a ticket from Support. Hardware is our job: a failed component is replaced within 4 hours of diagnosis (see the SLA), and Software RAID explains how the mirror is rebuilt afterwards. The software on your server is yours to administer, but tell us what you tried and our engineers will point you in the right direction.