<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Podman Archives - Hécate Regnier - Neurozone</title>
	<atom:link href="https://hecate.neurozone.fr/tag/podman/feed/" rel="self" type="application/rss+xml" />
	<link></link>
	<description>DBA, DevOps &#38; Sysadmin</description>
	<lastBuildDate>Mon, 02 Nov 2026 19:56:22 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://hecate.neurozone.fr/wp-content/uploads/2026/10/cropped-avatar_Instagram-150x150.webp</url>
	<title>Podman Archives - Hécate Regnier - Neurozone</title>
	<link></link>
	<width>32</width>
	<height>32</height>
</image> 
<site xmlns="com-wordpress:feed-additions:1">258172890</site>	<item>
		<title>How-to: test your restores automatically every week</title>
		<link>https://hecate.neurozone.fr/2026/11/18/how-to-test-your-restores-automatically-every-week/</link>
					<comments>https://hecate.neurozone.fr/2026/11/18/how-to-test-your-restores-automatically-every-week/#respond</comments>
		
		<dc:creator><![CDATA[Hécate Regnier]]></dc:creator>
		<pubDate>Wed, 18 Nov 2026 09:00:00 +0000</pubDate>
				<category><![CDATA[Backup]]></category>
		<category><![CDATA[Sysadmin notebook]]></category>
		<category><![CDATA[English]]></category>
		<category><![CDATA[how-to]]></category>
		<category><![CDATA[MariaDB]]></category>
		<category><![CDATA[Podman]]></category>
		<category><![CDATA[PostgreSQL]]></category>
		<category><![CDATA[restauration]]></category>
		<category><![CDATA[sauvegarde]]></category>
		<guid isPermaLink="false">https://hecate.neurozone.fr/?p=622</guid>

					<description><![CDATA[<p>Automate a weekly restore test of your latest pgBackRest and MariaDB backups in a container, schedule it and get notified of the result.</p>
<p>The post <a href="https://hecate.neurozone.fr/2026/11/18/how-to-test-your-restores-automatically-every-week/">How-to: test your restores automatically every week</a> appeared first on <a href="https://hecate.neurozone.fr">Hécate Regnier - Neurozone</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Backups that run every night, logs that say &#8220;OK&#8221;, an all-green dashboard: none of that proves you&#8217;ll be able to restore. Failure causes are many and silent: a configuration file left out of the backup, a hole in the transaction log archive, a major version that changed in the meantime, wrong permissions, a lost encryption key. The only proof a backup works is a successful restore. This article shows how to test restores automatically, once a week.</p>



<p class="wp-block-paragraph">This tutorial automates that proof: every week, the latest PostgreSQL backup (pgBackRest) and the latest MariaDB backup (mariadb-backup) are restored into a throwaway Podman container, the instance is started, each database is queried, then everything is destroyed. A message arrives either way. Production is never touched.</p>



<figure class="wp-block-image size-full"><a href="https://hecate.neurozone.fr/wp-content/uploads/2026/11/how-to-test-your-restores-automatically-every-week_diagram_en.webp"><img fetchpriority="high" decoding="async" width="1720" height="900" src="https://hecate.neurozone.fr/wp-content/uploads/2026/11/how-to-test-your-restores-automatically-every-week_diagram_en.webp" alt="Diagram of the weekly restore test: timer, script, Podman container, restore, check, PASSED or FAILED message" class="wp-image-3155" srcset="https://hecate.neurozone.fr/wp-content/uploads/2026/11/how-to-test-your-restores-automatically-every-week_diagram_en.webp 1720w, https://hecate.neurozone.fr/wp-content/uploads/2026/11/how-to-test-your-restores-automatically-every-week_diagram_en-300x157.webp 300w, https://hecate.neurozone.fr/wp-content/uploads/2026/11/how-to-test-your-restores-automatically-every-week_diagram_en-1024x536.webp 1024w, https://hecate.neurozone.fr/wp-content/uploads/2026/11/how-to-test-your-restores-automatically-every-week_diagram_en-768x402.webp 768w, https://hecate.neurozone.fr/wp-content/uploads/2026/11/how-to-test-your-restores-automatically-every-week_diagram_en-1536x804.webp 1536w" sizes="(max-width: 800px) 100vw, 760px" /></a><figcaption class="wp-element-caption">The weekly test restores, queries and destroys, and always sends a message.</figcaption></figure>



<h2 class="wp-block-heading">The rules of the game</h2>



<figure class="wp-block-table"><table><thead><tr><th>Rule</th><th>Why</th></tr></thead><tbody><tr><td>never touch production</td><td>restore into a scratch directory, container destroyed at the end whatever happens, repository mounted read-only</td></tr><tr><td>the same packages as production</td><td>a physical backup only restores into the same major version, with the same paths</td></tr><tr><td>a message either way</td><td>a silent test is soon forgotten; a test that fails silently is worse than no test</td></tr><tr><td>one test at a time</td><td>two simultaneous restores saturate the disk and trip over each other</td></tr><tr><td>check the contents, not just startup</td><td>an instance that starts on empty databases isn&#8217;t a backup</td></tr></tbody></table></figure>



<h2 class="wp-block-heading">Why test restores in a container</h2>



<p class="wp-block-paragraph">You could restore into a directory on the machine and start a second instance on another port. The container adds three things: full isolation (no risk of a test instance writing into production&#8217;s files or answering on its port), an environment identical from one week to the next, and cleanup guaranteed by a simple <code>podman rm -f</code>. Podman runs without a daemon and integrates well with systemd and SELinux.</p>



<h2 class="wp-block-heading">Step 1: images faithful to production</h2>



<p class="wp-block-paragraph">Don&#8217;t use the official <code>postgres</code> or <code>mariadb</code> images from Docker Hub: they&#8217;re built on Debian, with other paths, other build options, sometimes another minor version. The test must use exactly the server&#8217;s packages: start from <code>rockylinux:10</code> and copy <strong>the machine&#8217;s own repository files</strong> rather than rewriting them.</p>



<pre class="EnlighterJSRAW" data-enlighter-language="shell"># Containerfile.postgres
FROM rockylinux:10
COPY repos/pgdg-redhat-all.repo /etc/yum.repos.d/
RUN dnf -y install postgresql18-server pgbackrest &amp;&amp; dnf clean all
ENV PATH=/usr/pgsql-18/bin:$PATH</pre>



<pre class="EnlighterJSRAW" data-enlighter-language="shell"># Containerfile.mariadb
FROM rockylinux:10
COPY repos/MariaDB.repo /etc/yum.repos.d/MariaDB.repo
RUN dnf -y install MariaDB-server MariaDB-backup MariaDB-client MariaDB-common \
    &amp;&amp; dnf clean all</pre>



<pre class="EnlighterJSRAW" data-enlighter-language="shell">podman build -t localhost/restore-test-postgres:18 -f Containerfile.postgres .
podman build -t localhost/restore-test-mariadb:12.3 -f Containerfile.mariadb .</pre>



<p class="wp-block-paragraph">Rebuild the images after every major upgrade of the server, and preferably monthly to track minor versions. A test that restores a PostgreSQL 18 backup into a 17 binary will fail, and that&#8217;s good: it caught a gap.</p>



<h2 class="wp-block-heading">Step 2: the common skeleton</h2>



<pre class="EnlighterJSRAW" data-enlighter-language="shell">#!/bin/bash
set -uo pipefail
DATE=$(date +%Y%m%d_%H%M%S)
SCRATCH=/data/restore-test-scratch/pg-$DATE-$$
NAME=restore-test-pg-$DATE-$$
LOCK=/run/lock/restore-test-pg.lock

notify() { /usr/local/bin/notify.sh backups &quot;PostgreSQL restore test&quot; &quot;$1&quot;; }
fail()   { notify &quot;FAILED: $1&quot;; exit 1; }
cleanup() { podman rm -f &quot;$NAME&quot; &gt;/dev/null 2&gt;&amp;1 || true; rm -rf &quot;$SCRATCH&quot;; }
trap cleanup EXIT INT TERM

exec 9&gt;&quot;$LOCK&quot;
flock -n 9 || fail &quot;another test is already running&quot;
mkdir -p &quot;$SCRATCH/data&quot;</pre>



<ul class="wp-block-list"><li><strong>set -uo pipefail</strong> without <code>-e</code>: we want to handle each error explicitly to send a clear message, not let the script die silently;</li><li><strong>trap cleanup EXIT INT TERM</strong>: the container and scratch directory are removed whatever happens, including if systemd stops the service at its timeout;</li><li><strong>flock</strong>: a lock keeps two tests from running at once, for example when a manual run crosses the scheduled one;</li><li><strong>a unique name</strong> per run (date and process number): an orphan container from a previous test doesn&#8217;t block the next.</li></ul>



<h2 class="wp-block-heading">Step 3: the PostgreSQL test</h2>



<pre class="EnlighterJSRAW" data-enlighter-language="shell">podman run -d --name &quot;$NAME&quot; \
  -v /data/backup/pgbackrest:/data/backup/pgbackrest:ro,Z \
  -v /etc/pgbackrest/pgbackrest.conf:/etc/pgbackrest/pgbackrest.conf:ro,Z \
  -v &quot;$SCRATCH/data&quot;:/var/lib/pgsql/data:Z \
  localhost/restore-test-postgres:18 sleep infinity || fail &quot;container start&quot;

podman exec -u root &quot;$NAME&quot; chown -R postgres:postgres /var/lib/pgsql/data
podman exec -u postgres &quot;$NAME&quot; pgbackrest --stanza=main \
  --pg1-path=/var/lib/pgsql/data --archive-mode=off restore || fail &quot;pgbackrest restore&quot;
podman exec -u postgres &quot;$NAME&quot; pg_ctl -D /var/lib/pgsql/data -w -t 900 \
  -o &quot;-c archive_mode=off -c port=5432&quot; start || fail &quot;PostgreSQL won&#x27;t start&quot;

dbs=$(podman exec -u postgres &quot;$NAME&quot; psql -Atc \
  &quot;select datname from pg_database where not datistemplate and datname &lt;&gt; &#x27;postgres&#x27;&quot;)
[ -n &quot;$dbs&quot; ] || fail &quot;no database restored&quot;
for db in $dbs; do
  n=$(podman exec -u postgres &quot;$NAME&quot; psql -d &quot;$db&quot; -Atc \
      &quot;select count(*) from pg_class c join pg_namespace n on n.oid=c.relnamespace
       where c.relkind=&#x27;r&#x27; and n.nspname not in (&#x27;pg_catalog&#x27;,&#x27;information_schema&#x27;)&quot;)
  [ &quot;${n:-0}&quot; -gt 0 ] || fail &quot;database $db empty&quot;
done
notify &quot;PASSED: $(echo $dbs | wc -w) databases restored and checked&quot;</pre>



<h3 class="wp-block-heading">The tricky points</h3>



<ul class="wp-block-list"><li><strong>read-only repository</strong> (<code>:ro</code>): the test can&#8217;t damage the backups, even by mistake;</li><li><strong>&#8211;archive-mode=off</strong> on restore and <code>archive_mode=off</code> at startup: the test instance must never archive its own WAL into the production repository, which would create a stray timeline;</li><li><strong>pg_ctl -w -t 900</strong>: waits up to fifteen minutes for PostgreSQL to replay WAL and accept connections. The time depends on the WAL volume since the last backup; tune it from the first tests&#8217; real duration;</li><li><strong>the checks</strong>: each database must exist and contain tables. A database with no table after restore is suspect. You can go further with one business query per application (&#8220;does the articles table hold at least a thousand rows?&#8221;);</li><li><strong>SELinux and volumes</strong>: the <code>:Z</code> suffix relabels the directory for the container. Never use it on a directory shared with a service on the machine, it would change its label; in that case, <code>--security-opt label=disable</code> on that one container is the reasonable option.</li></ul>



<h2 class="wp-block-heading">Step 4: the MariaDB test</h2>



<p class="wp-block-paragraph">The principle is the same with a physical <code>mariadb-backup</code> backup, with a few differences:</p>



<ul class="wp-block-list"><li><strong>prepare before starting</strong>: a MariaDB physical backup must go through <code>mariadb-backup --prepare</code>, which applies the redo log copied during the backup. If your backup procedure already does this (preferable), the test starts directly; otherwise the test must do it, on a copy;</li><li><strong>copy, don&#8217;t mount</strong>: MariaDB writes to its files as soon as it starts. So copy the backup into the scratch directory (<code>cp -a</code>) instead of mounting it read-only;</li><li><strong>the same configuration</strong>: mount a copy of production&#8217;s <code>my.cnf</code>. If production puts its undo tablespaces (<code>innodb_undo_directory</code>), binlogs or logs in separate directories, recreate them in the scratch directory and move the matching files there, or InnoDB will refuse to start;</li><li><strong>the mysql user&#8217;s id</strong>: it isn&#8217;t necessarily the same in the image and on the machine. Read it from the image (<code>podman run --rm image id -u mysql</code>) and give it the files;</li><li><strong>the checks</strong>: <code>mariadb-admin ping</code> in a loop until the server answers, then for each non-system database, the number of tables and an estimated row count from <code>information_schema.TABLES</code>. A database with no table or no row fails the test.</li></ul>



<h2 class="wp-block-heading">Step 5: schedule</h2>



<pre class="EnlighterJSRAW" data-enlighter-language="ini"># /etc/systemd/system/restore-test-pgbackrest.timer
[Timer]
OnCalendar=Sun *-*-* 05:00:00
RandomizedDelaySec=15min
Persistent=true
[Install]
WantedBy=timers.target

# /etc/systemd/system/restore-test-pgbackrest.service
[Unit]
After=pgbackrest-full.service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/restore-test-pgbackrest.sh
TimeoutStartSec=3h
Nice=10
IOSchedulingClass=idle</pre>



<ul class="wp-block-list"><li><strong>Sunday 5 am</strong>: after Sunday&#8217;s full backup, to test the freshest one;</li><li><strong>Persistent=true</strong>: catches up a missed test if the machine was off;</li><li><strong>TimeoutStartSec=3h</strong>: a stuck test is eventually killed, and the <code>trap</code> cleans up;</li><li><strong>Nice and IOSchedulingClass=idle</strong>: the test yields priority to services. It takes longer, but nobody notices.</li></ul>



<h2 class="wp-block-heading">Step 6: the notification</h2>



<p class="wp-block-paragraph">The message must arrive in both cases and say something useful: which backup was tested, how many databases, and on failure, the failing step and the log path. A <code>notify.sh</code> script shared by all services (email, Mattermost, ntfy, whatever) avoids duplicating that code. A test that passes every Sunday ends up as background noise: that&#8217;s intended. The day the message is missing, or says something else, you notice.</p>



<p class="wp-block-paragraph">Add a monitoring alert on the absence of a recent success: if no &#8220;PASSED&#8221; message has arrived in ten days, something is wrong even if no failure was reported (timer disabled, machine off, script deleted).</p>


<!-- nz-enrich:restoretest-real -->


<h3 class="wp-block-heading">Last Sunday&#8217;s run</h3>



<p class="wp-block-paragraph">The two tests as they ran on Sunday 1 November 2026 on my home server, from their log files (excerpts; pgBackRest logs in UTC, the script in local time, hence the one-hour gap):</p>



<pre class="EnlighterJSRAW" data-enlighter-language="shell">[2026-11-01 05:00:04] Starting pgbackrest restore-test for stanza &#x27;main&#x27;
[2026-11-01 05:00:04] Stanza &#x27;main&#x27; has 25 backup(s) on record — proceeding with restore
04:00:05.311 P00 INFO: repo1: restore backup set 20261101-010003F, recovery will start at 2026-11-01 00:00:03
04:05:51.113 P00 INFO: restore size = 93.5MB, file total = 5756
04:05:51.114 P00 INFO: restore command end: completed successfully (345892ms)
[2026-11-01 05:05:57] PostgreSQL is ready (took ~6s)
[2026-11-01 05:05:57] Found 4 database(s): forgejo semaphore paperless netbox
[2026-11-01 05:05:58]   forgejo: 131 table(s) — OK
[2026-11-01 05:05:58]   netbox: 208 table(s) — OK
[2026-11-01 05:05:58] pgbackrest restore-test PASSED — 4 database(s) restored and verified non-empty

[2026-11-01 06:30:26]   blog_flo_wp: 93 table(s), ~568925 row(s) — OK
[2026-11-01 06:30:32]   piwigo: 35 table(s), ~1008323 row(s) — OK
[2026-11-01 06:30:38]   wiki: 41 table(s), ~1440630 row(s) — OK
[2026-11-01 06:30:38] MariaDB restore-test PASSED — 21 database(s) restored and verified non-empty (backup: 20261101_020001)</pre>



<ul class="wp-block-list"><li><strong>The restore took 5 min 46 s for 93.5 MB</strong>: that&#8217;s the useful number: it&#8217;s roughly what a real restore of that database would cost, measured every week instead of guessed on the day.</li><li><strong>PostgreSQL ready in 6 s</strong>: the restored cluster replayed its WAL and accepted connections, which is the real proof the backup is consistent, not just complete.</li><li><strong>Tables counted per database</strong>: an empty or truncated database fails the test even if the restore itself &#8220;succeeded&#8221;.</li><li><strong>Row counts with a tilde</strong> on the MariaDB side: they come from the statistics (<code>information_schema</code>), fast but approximate. Good enough to catch an empty table, not to compare two backups exactly.</li></ul>



<h2 class="wp-block-heading">What the test proves, and what it doesn&#8217;t</h2>



<p class="wp-block-paragraph">It proves the backup is complete and readable, that the log archive is continuous up to the backup, that an instance of the right version starts on it, and that each database holds data. That&#8217;s already a lot: most backups that fail on the day fail on one of these points.</p>



<p class="wp-block-paragraph">It doesn&#8217;t prove that the application works on that data, that point-in-time recovery lands at the right moment, or that you can run the procedure under pressure at 3 am. For that, nothing replaces a full manual drill once or twice a year: restore onto a blank machine, reconnect an application, stopwatch in hand, following the written documentation. The gaps in the documentation always show up then.</p>



<h2 class="wp-block-heading">What about other data?</h2>



<p class="wp-block-paragraph">The same principle applies to application files (attachments, media, Git repositories): once a month, extract the latest archive into a temporary directory, check it isn&#8217;t truncated, that the expected files are there and that their number hasn&#8217;t suddenly dropped. Less spectacular than a database, but just as valuable.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<p class="wp-block-paragraph"><em>This post is part of a series of sysadmin tutorials (tag &#8220;how-to&#8221;).</em><br>Series &#8220;How-to&#8221;, tutorial 13/77. · <a href="https://hecate.neurozone.fr/?p=621">← Previous tutorial</a> · <span class="link-upcoming">Next tutorial → (coming 23 Nov)</span> · <a href="https://blog.neurozone.fr/?p=2067">Version française</a></p>



<h2 class="wp-block-heading">Related articles</h2>



<ul class="wp-block-list"><li><a href="https://hecate.neurozone.fr/?p=621">PostgreSQL backups with pgBackRest</a></li><li><a href="https://hecate.neurozone.fr/?p=596">The MariaDB settings that really matter</a></li><li><a href="https://hecate.neurozone.fr/?p=378">PostgreSQL data files, from the inside</a></li></ul>
<p>The post <a href="https://hecate.neurozone.fr/2026/11/18/how-to-test-your-restores-automatically-every-week/">How-to: test your restores automatically every week</a> appeared first on <a href="https://hecate.neurozone.fr">Hécate Regnier - Neurozone</a>.</p>
]]></content:encoded>
					
					<wfw:commentRss>https://hecate.neurozone.fr/2026/11/18/how-to-test-your-restores-automatically-every-week/feed/</wfw:commentRss>
			<slash:comments>0</slash:comments>
		
		
		<post-id xmlns="com-wordpress:feed-additions:1">622</post-id>	</item>
	</channel>
</rss>
