The server that never came back (2/3): reading the traces of a shutdown

The machine was back, but journalctl only shows the current boot. On this server the journal isn’t persistent (Storage=auto with no /var/log/journal directory). So the previous boot’s traces are gone from there. Fortunately, other sources survive a reboot.

Source 1: rsyslog and /var/log/messages

grep -E "System is rebooting|Command line:" /var/log/messages | tail -4
# 19:49:42 systemd-logind: System is rebooting.
# 20:20:22 kernel: Command line: BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.12.0-211.67.1…

The log stops dead at “System is rebooting”, then resumes thirty minutes later with the new kernel’s command line. That silence isn’t suspicious: rsyslog is among the first services stopped during a reboot, so it never writes the end of the shutdown sequence. You have to look elsewhere for the rest.

Source 2: wtmp, the login and shutdown register

last -x -F | head -6
# reboot   system boot  6.12.0-211.67.1  Wed Oct 21 20:20:26 2026   still running
# shutdown system down  6.12.0-211.54.1  Wed Oct 21 19:50:04 2026 - 20:20:26 (00:30)

last -x also shows system entries, not just logins. The shutdown system down line is written by systemd-update-utmp at the very end of the shutdown sequence, when almost every service has already stopped. Its presence proves systemd went all the way. It’s stamped 19:50:04, 22 seconds after the reboot began. The “(00:30)” column is the time until the next boot: thirty minutes during which the machine did nothing.

Source 3: how the filesystems mounted on the next boot

grep -E "XFS .*(Mounting|recovery|clean mount)" /var/log/messages | grep "20:20"
# XFS (dm-0): Mounting V5 Filesystem …
# XFS (dm-0): Ending clean mount

When an XFS filesystem wasn’t unmounted cleanly, the next mount replays its log and prints “Starting recovery” then “Ending recovery”. Here all four filesystems say “Ending clean mount”: they had all been unmounted properly. So the OS had finished shutting down before the machine froze.

What you can conclude

• the systemd shutdown completed (shutdown entry in wtmp);
• the disks were unmounted cleanly (“clean” mount afterwards);
• the machine then did nothing for thirty minutes, until it was restarted by hand.

So the hang happened after the kernel asked the motherboard to restart, somewhere between that call and the BIOS. It’s not a package, not a service, not a filesystem.

For next time

Making the journal persistent makes this kind of investigation much easier. It’s one line:

mkdir -p /var/log/journal && systemd-tmpfiles --create --prefix /var/log/journal
journalctl --list-boots      # previous boots now show up

Even persistent, the journal will never see what happens after the kernel stops. The wtmp and mount clues remain the most reliable way to tell whether a shutdown completed cleanly.

---

This post is part of a series of sysadmin notes (tag “sysadmin notebook”).
Series “The server that never came back”, part 2/3. · ← Previous post · Next post → · Version française