A morning of housekeeping on the home server (Rocky Linux 10 on an old ASRock board). On the list: a /boot partition that had become too small, disks sitting idle in the machine that it was time to mirror, and Prometheus alerts so a dead disk no longer gets discovered by chance. Here’s how it went, commands included.
The problem: a 1 GB /boot that’s no longer enough
Rocky’s installer creates a 1 GB /boot by default. That seemed generous… until I looked at what’s actually in it:
$ ls -la --block-size=M /boot
152M initramfs-6.12.0-211.67.1.el10_2.x86_64.img
39M initramfs-6.12.0-211.67.1.el10_2.x86_64kdump.img
16M vmlinuz-6.12.0-211.67.1.el10_2.x86_64
159M initramfs-0-rescue-….img
16M vmlinuz-0-rescue-…About 217 MB per kernel, plus 175 MB for the rescue image. With installonly_limit=3 in dnf.conf, that’s ~825 MB out of 960. And the trap: during an update, dnf installs the new kernel before removing the oldest. So for a moment there are four kernels, and they don’t fit.
Why a 152 MB initramfs? lsinitrd has the answer: about a hundred MB of NVIDIA (GSP) firmware pulled in by the nouveau driver, because there’s a GeForce GTX 1060 in the machine. I’m keeping nouveau, so the fix is a bigger /boot.
$ lsinitrd /boot/initramfs-$(uname -r).img | sort -k5 -n | tail -3
… 13371848 usr/lib/firmware/nvidia/tu102/gsp/gsp-570.144.bin.xz
… 25639648 usr/lib/firmware/nvidia/ga102/gsp/gsp-535.113.01.bin.xz
… 51324396 usr/lib/firmware/nvidia/ga102/gsp/gsp-570.144.bin.xzWhy not just grow the partition?
Because the partition after /boot is the system’s LVM physical volume, right behind it. And XFS can’t shrink. Growing in place would mean moving the whole LVM (pvmove to a new partition, delete the old one, grow /boot into the gap). Doable online, but that’s a lot of handling on the system disk of a machine that had hung on reboot just two days earlier.
There were more than 300 GB free at the end of the disk. Simpler plan: create a new 2 GB /boot partition in that free space, copy the old /boot into it, and reinstall GRUB so it uses it. The old partition stays untouched as a safety net.
Moving /boot to a new partition (BIOS + GPT)
The important point is the boot mode. Here it’s legacy BIOS on a GPT disk: GRUB keeps its main stage (core.img) in the small 1 MB bios_grub partition, and that core.img has the location of /boot/grub2 hard-coded. Move /boot without re-running grub2-install, and GRUB keeps reading the old partition. On UEFI, it’s the ESP’s grub.cfg that would need updating instead.
$ parted -s /dev/sde unit MiB print free
Number Start End Size File system Name Flags
1 1.00MiB 2.00MiB 1.00MiB bios_grub
2 2.00MiB 1026MiB 1024MiB xfs bls_boot
3 1026MiB 160406MiB 159380MiB lvm
160406MiB 476940MiB 316534MiB Free SpaceDo this with console access (screen and keyboard, or IPMI), in case the first reboot goes wrong.
1. Create and format the new partition
parted -s /dev/sde mkpart boot2 xfs 160406MiB 162454MiB # 2 GiB -> sde4
mkfs.xfs -L boot2 /dev/sde42. Copy /boot’s contents
mkdir -p /mnt/newboot
mount /dev/sde4 /mnt/newboot
cp -a /boot/. /mnt/newboot/ # -a keeps permissions, links and SELinux contexts
restorecon -R /mnt/newboot3. Point /etc/fstab at the new partition
cp -a /etc/fstab /etc/fstab.bak
blkid -s UUID -o value /dev/sde4 # new UUID
# replace the UUID on the /boot line with it, then:
systemctl daemon-reload
umount /mnt/newboot
umount /boot && mount /boot
findmnt /boot # should show /dev/sde44. Reinstall GRUB and regenerate its config
grub2-install /dev/sde # core.img now points at sde4
grub2-mkconfig -o /boot/grub2/grub.cfgThe BLS boot entries (/boot/loader/entries/*.conf) use paths relative to the /boot partition, so they work as they are, and grubenv (the default kernel) was copied with everything else.
5. Reboot and check
reboot
findmnt /boot # /dev/sde4
df -h /boot # ~2 GB
grubby --default-kernel # still the right kernelRollback: the old sde2 partition hasn’t been touched. If GRUB drops to its rescue prompt, you can point it back by hand:
grub rescue> set prefix=(hd0,gpt2)/grub2
grub rescue> set root=(hd0,gpt2)
grub rescue> insmod normal
grub rescue> normalThen put the old UUID back in /etc/fstab and re-run grub2-install /dev/sde. Once the new partition has proven itself over a few reboots and kernel updates, the old one is just a wasted gigabyte, which is no big deal.
Along the way: mirroring /data… with 4 MB missing
The machine also had disks doing nothing. I’m using one to mirror /data (931 GB: databases, repositories, backups) with LVM RAID1. On paper it’s a single command, no downtime:
lvconvert --type raid1 -m1 data/lvol0 /dev/sddIn practice it fails. An LVM RAID1 LV needs, on each disk, a small metadata area (rmeta, one 4 MB extent) on top of the copy of the data. The original LV used all the extents on the first disk (238,467 of 238,467), and the second disk is exactly the same size. So it’s 4 MB short on each side. And since XFS can’t shrink, there’s no way to free them.
The way out is to go the other direction:
- create an LV one extent smaller on the new disk, format it, and copy
/datato it while everything runs (rsync -aHAXS --numeric-ids, which keeps hard links, ACLs, xattrs/SELinux and sparse files); - stop the services using
/data(MariaDB, PostgreSQL, Forgejo…), run a finalrsync --delete, rename the LVs so the new one takes the old one’s name (fstabdoesn’t need to change), then restart the services. Expect about a quarter of an hour of downtime; - keep the old LV for a day as a safety net, then delete it and run
lvconvert --type raid1 -m1onto the old disk, which now has room for itsrmetaarea. The sync runs in the background.
Lesson learned: when you create an LV you might want to mirror one day, leaving a few extents free (-l 99%FREE rather than 100%FREE) saves you the whole detour.
For two other old identical disks with no data, it’s much simpler. Create a mirrored volume directly and LVM reserves room for the metadata itself:
vgcreate bulk /dev/sda /dev/sdb
lvcreate --type raid1 -m1 -l 100%FREE -n lvol0 bulk
mkfs.xfs /dev/bulk/lvol0
# in /etc/fstab, nofail: an old disk dying must not block boot
/dev/bulk/lvol0 /bulk xfs defaults,nofail 0 0Monitoring the mirrors with Prometheus
A mirror nobody notices has lost a disk isn’t worth much. By default LVM only logs a warning, and node_exporter knows nothing about LVM RAID (its mdadm collector only covers md). So I added a small script, run every two minutes by a systemd timer, that writes the RAID LVs’ state into node_exporter’s textfile collector directory (--collector.textfile.directory):
lvs --noheadings --nosuffix --separator ';' \
-o vg_name,lv_name,lv_health_status,raid_sync_action,copy_percent,raid_mismatch_count \
--select 'segtype=~raid'That gives metrics like lvm_raid_healthy (0 as soon as lv_health_status isn’t empty, e.g. partial when a disk is missing), lvm_raid_sync_percent and lvm_raid_mismatch_count. The file is written to a temporary file and then renamed with mv, so a scrape never hits a half-written file. Four alerts build on it:
- alert: LvmRaidDegraded # a disk is missing or failing
expr: lvm_raid_healthy == 0
for: 5m
- alert: LvmRaidNotInSync # a resync that never finishes
expr: lvm_raid_sync_percent < 100
for: 12h
- alert: LvmRaidMismatches # the last scrub found differences
expr: lvm_raid_mismatch_count > 0
- alert: LvmRaidMetricsStale # the script itself stopped running
expr: time() - lvm_raid_textfile_last_run_seconds > 900On the hardware side, 1, 2 and 3 TB WD disks are waiting on the shelf. The day an alert fires: swap the disk, then pvcreate, vgextend and lvconvert --repair, and LVM rebuilds the mirror online.
In short
- A 1 GB
/bootis tight as soon as a driver pulls large firmware into the initramfs. Check withlsinitrd. - Moving
/bootto a new partition is simpler and less risky than growing the existing one. On BIOS/GPT, don’t forgetgrub2-install. - LVM RAID1 needs one free extent per disk for its metadata. A
100%FREELV on two identical disks can’t be converted in place. - A mirror without an alert is an outage waiting to happen. node_exporter’s textfile collector is a great fit for this kind of small addition.
This post is part of a series of sysadmin notes (tag “sysadmin notebook”).
Version française