Check disk health with smartctl and nvme-cli

Every DediJet server passes SMART and NVMe health checks during burn-in, before it joins the ready pool. Disks still wear out and occasionally fail, and they usually warn you first. Two tools read those warnings: smartctl (from smartmontools) for every disk type, and nvme-cli for NVMe drives.

Install the tools

On Ubuntu and Debian:

apt update
apt install -y smartmontools nvme-cli

On AlmaLinux and Rocky Linux:

dnf install -y smartmontools nvme-cli

Proxmox VE already includes smartmontools. List your disks with their serial numbers; ROTA = 1 means a spinning HDD:

lsblk -d -o NAME,MODEL,SERIAL,SIZE,ROTA,TRAN

Read SMART data on HDDs and SATA SSDs

smartctl -a /dev/sda

Start with the line SMART overall-health self-assessment test result. FAILED means the disk must be replaced. PASSED only means no attribute has crossed the vendor’s threshold yet, so read the attributes too, using the RAW_VALUE column:

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       1874
194 Temperature_Celsius     0x0002   200   200   000    Old_age   Always       -       31
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

Attribute names and numbers vary by vendor, and SAS drives report Elements in grown defect list instead. For the full picture, including the drive’s own error log, use smartctl -x.

Read NVMe health

smartctl -a /dev/nvme0
nvme smart-log /dev/nvme0

Both read the same health log. /dev/nvme0 is the controller; /dev/nvme0n1 is its namespace, the block device you partition.

Critical Warning:                   0x00
Temperature:                        38 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    1%
Data Units Written:                 5,322,198 [2.72 TB]
Unsafe Shutdowns:                   3
Media and Data Integrity Errors:    0
Error Information Log Entries:      0

• Critical Warning must be 0x00. Any other value (spare below threshold, temperature, degraded reliability, read-only mode) needs attention now.
• Available Spare is the reserve of replacement blocks. Once it drops below the threshold, the drive is near the end of its life.
• Percentage Used is the vendor’s estimate of rated endurance consumed. It can exceed 100% while the drive keeps working, but plan a replacement.
• Media and Data Integrity Errors should stay at 0. A growing count means the drive cannot reliably return your data.
• Error Information Log Entries often counts harmless events, such as unsupported commands sent by tools. It is not a failure signal on its own.
• Data Units Written counts units of 512,000 bytes; smartctl converts the total for you.

Run short and long self-tests

Self-tests run inside the drive’s firmware while the server keeps working. A short test usually takes two minutes or less; a long (extended) test reads the whole surface and can take many hours on large HDDs. smartctl prints the estimate when the test starts.

smartctl -t short /dev/sda
smartctl -t long /dev/sda
smartctl -l selftest /dev/sda

In the log, Completed without error is what you want; Completed: read failure with an LBA number means the drive found unreadable sectors. For NVMe drives that support self-tests, use nvme-cli (code 1 = short, 2 = extended); an operation result of 0 in its log means success. smartctl can do the same from smartmontools 7.4 onwards.

nvme device-self-test /dev/nvme0 -s 1
nvme self-test-log /dev/nvme0

Schedule checks and email alerts with smartd

The smartd daemon checks every disk every 30 minutes and can start self-tests on a schedule. Edit /etc/smartd.conf (Debian, Ubuntu) or /etc/smartmontools/smartd.conf (AlmaLinux, Rocky Linux) and replace the existing DEVICESCAN line, because smartd ignores the lines that follow it:

DEVICESCAN -a -n standby,q -s (S/../.././02|L/../../6/03) -m you@example.com -M test

• -a monitors health status, attributes, error logs and self-test logs.
• -s (S/../.././02|L/../../6/03) starts a short test every day at 02:00 and a long test every Saturday at 03:00. On NVMe drives, scheduled self-tests need smartmontools 7.5 or later (check with smartctl --version); older versions still monitor NVMe health.
• -m sends alerts through the mail command, so install a mail client and a transfer agent, for example mailutils with Postfix or msmtp.
• -M test sends one test email at startup. Remove it once the email has arrived.

Restart the daemon and check its log. On Ubuntu and Debian:

systemctl restart smartmontools
journalctl -u smartmontools -n 50

On AlmaLinux and Rocky Linux, the service is named smartd: systemctl enable --now smartd, then journalctl -u smartd -n 50.

When to open a ticket

Replacing failed hardware is part of the service: under our SLA, a failed component is replaced within 4 hours of diagnosis, 24/7. Open a ticket when you see any of these:

• An overall assessment of FAILED, or a failed self-test.
• Reallocated sectors that keep growing, or any pending or offline-uncorrectable sectors.
• An NVMe critical warning other than 0x00, spare below threshold, or growing media errors.
• A growing UDMA CRC error count.
• A disk that dropped out of your RAID array, or kernel messages such as I/O error or NVMe controller resets in journalctl -k.

Put everything in the first message to speed up the diagnosis:

1. The server’s IP address and the disk you suspect, with its serial number from smartctl -i.
2. The full output of smartctl -x for that disk, attached as a file.
3. With software RAID: cat /proc/mdstat and the serial number of the disk that must stay in place.
4. Relevant kernel log lines with their timestamps, and whether your data is backed up.

With software RAID, follow Manage software RAID 1 with mdadm to prepare the array before the swap and rebuild it afterwards.

Next steps

• Measure performance with a repeatable method: Benchmark CPU, disks and network.
• Repair a system that no longer boots from rescue mode.
• Not sure how to read your output? Contact support: hardware diagnosis is in scope, 24/7.