One Tuesday evening, a routine update on the home server: dnf update -y, about a hundred packages including a new kernel and new firmware, then reboot. Normally the machine is back in under a minute. This time, nothing. No ping, no SSH, and above all no ARP reply on the local network: as far as the switch could tell, the network card no longer existed.
That last detail matters. A machine that boots but has a crashed service still answers ARP, because the kernel answers it, not an application. No ARP means the kernel isn’t running or the network card isn’t initialised. So the problem is low-level: kernel boot, initramfs, motherboard firmware, or the machine simply being off.
What you can check remotely
Before going down to the machine, I gathered what the rest of the network could see:
arping -c 3 -I eth0 192.168.10.17 # no reply: the network card is silent
ip neigh show 192.168.10.17 # FAILED entry in the neighbour's ARP table- The SIEM (Wazuh) had received one last event from the machine’s agent: “Wazuh agent stopped”, at the exact time of the reboot. That’s a sign of a clean shutdown, not a crash. A crash would have left no message at all.
- The other machines on the network worked normally: the problem was neither the switch nor the router.
- The virtual machine hosted on this server had gone down with it, consistent with a powered-off host.
What to look for on the screen
With no remote management card (no IPMI on a consumer motherboard), the only way to know is to look at the screen. Here’s the list I prepared, from most to least likely cause, so as not to improvise at the console:
- Stuck in GRUB? Pick the previous kernel in the menu. If it boots, the new kernel or its initramfs is at fault. Then pin it with
grubby --set-default. - Emergency shell (dracut, emergency mode)?
journalctl -xb -p errshows why: LVM volume not found, invalidfstabentry, module missing from the initramfs. - SELinux relabel or fsck in progress? That’s slow on a big disk. Wait, and above all don’t cut the power.
- Booted but no network?
ip -br aandnmcli dev: a renamed or “unmanaged” card after an update happens more often than you’d think. - Black screen, fans spinning, nothing happening? The most puzzling case. It was mine.
The machine eventually came back after half an hour, with a long press on the power button. The system booted normally on the new kernel, without a single error. What remained was to understand what had happened, while the system journal only kept the current boot. That’s the subject of the next post.
This post is part of a series of sysadmin notes (tag “sysadmin notebook”).
Series “The server that never came back”, part 1/3. · Next post → · Version française