The server that never came back (3/3): the motherboard did it

Let’s recap the evidence: the systemd shutdown finished, the filesystems were unmounted cleanly, then the machine did nothing for thirty minutes. So the culprit lies between the kernel’s reboot call and the first BIOS screen. In other words, the motherboard never performed its reset.

How Linux reboots a machine

At the end of shutdown, the kernel calls reboot(), which has to trigger a real hardware reset. On x86 the kernel tries several methods, in a default order that depends on the machine:

• acpi: write to the reset register declared by the firmware (ACPI FADT table);
• kbd: the historical method, a pulse through the keyboard controller;
• pci: write to port 0xCF9, the chipset’s reset register;
• efi: ask the UEFI firmware to reboot;
• triple: cause a processor “triple fault”, as a last resort.

If the chosen method doesn’t work on a given board, the kernel has stopped everything and waits for a reset that never comes. That’s exactly the picture we saw: black screen, fans spinning, no more network activity.

Why that time and not the others?

The next reboot, on the new kernel, took 34 seconds with no trouble. Several factors may have played a part on the night of the hang: the old kernel with 35 days of uptime, and a firmware update (linux-firmware) in the same transaction. These reset hangs are known to be intermittent on some consumer motherboards. I couldn’t reproduce it, so I changed nothing for now.

If it happens again

The thing to try is forcing another reset method with the reboot= kernel parameter, then testing several reboots:

grubby --update-kernel=ALL --args="reboot=pci"     # or reboot=efi, reboot=acpi
grubby --info=DEFAULT | grep args                  # check
reboot                                             # test, several times
grubby --update-kernel=ALL --remove-args="reboot"  # to revert

reboot=pci (the chipset’s 0xCF9 register) is often the most reliable on consumer Intel hardware, and reboot=efi on recent UEFI firmware. The parameter applies to every installed kernel, including future ones.

What to remember

• before blaming an update, check whether the shutdown completed: wtmp and “clean” mounts tell you unambiguously;
• a hang after a clean shutdown is hardware or firmware, not the OS: no package rollback would have fixed it;
• if the machine hosts a service everything depends on, write down the recovery procedure. Here, the secrets vault has to be unsealed by hand after every reboot: that’s the subject of the next series.

---

This post is part of a series of sysadmin notes (tag “sysadmin notebook”).
Series “The server that never came back”, part 3/3. · ← Previous post · Version française