How to Detect When a Backup Silently Stops Running

To catch a backup that stops running, schedule it with a systemd timer, an OnFailure unit, and a start timeout, then check the bucket from another machine. A hung systemd run stayed activating with 0 alerts, and systemctl list-timers moved its LAST time even after a failing run. The bucket check proves age and size, not content. I didn't test delivery, and the printed alert only writes a log line until you point it at your pager.

I ran this in containers under WSL2, with Docker 29.9 and PostgreSQL 18.6 on Ubuntu 24.04 and MinIO as the storage, not on a real VPS or a real provider. The second machine was a second container, and some systemd runs used scaled-down timings, as the hang table says.

The job this page schedules and checks is pg-backup, from my off-site backups guide. The units and the check don't depend on it. For another job that uploads to a bucket, change ExecStart, the bucket path, and the floor. I only ran them with pg-backup.

The scripts and units are in a runnable sample, whose verify script runs check-backup-fresh but none of the units. Run it on a development machine, not the server, because it mounts the Docker socket, compiles MinIO from source, and leaves base images and build cache behind.

How Do I Schedule a Backup So a Failed Night Gets Noticed?

Use a systemd timer with an OnFailure unit, because cron dropped the output with No MTA installed, discarding output and recorded no exit code.

pg-backup.service:

[Unit]
Description=Dump Postgres and upload it off-site
Wants=network-online.target
After=network-online.target docker.service
# A run that fails fast gets four tries, five minutes apart, then it stops until the next timer or a reset-failed.
# A run that takes about 10 minutes or more to fail never reaches this limit and keeps retrying.
StartLimitIntervalSec=1h
StartLimitBurst=4
# Fires after every failed run, and once more when the limit is hit.
OnFailure=pg-backup-alert.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/pg-backup
Restart=on-failure
RestartSec=5min
# Longer than a slow run. Without this a hung run blocks every later night, with no alert.
TimeoutStartSec=2h
# Systemd removes the temp dump even if the script is killed with SIGKILL.
PrivateTmp=true

A SIGKILL of the unit with PrivateTmp=true removed the temp dump, and a SIGKILL of a hand run, or with PrivateTmp=false, left it behind under /var/tmp.

pg-backup.timer:

[Unit]
Description=Nightly Postgres backup

[Timer]
OnCalendar=*-*-* 03:00:00 UTC
RandomizedDelaySec=10min
Persistent=true

[Install]
WantedBy=timers.target

pg-backup-alert.service:

[Unit]
Description=Tell someone the Postgres backup failed

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo "$(date -u +%%FT%%TZ) pg-backup failed on $(hostname)" >> /var/log/pg-backup-alert.log'

Until you change that ExecStart, a failed night only adds a line to a log file on this server. Point it at whatever already pages you, then run systemctl start pg-backup-alert.service to see it arrive. My Dokploy operating guide lists what else to alert on. I didn't test real delivery.

Run this as root, with pg-backup installed, the offsite remote from my off-site backups guide, and export RCLONE_CONFIG=/etc/pg-backup/rclone.conf set in the shell. shop-backups/postgres is my lab's bucket, so use yours:

install -m 0644 pg-backup.service pg-backup.timer pg-backup-alert.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now pg-backup.timer
systemctl list-timers pg-backup.timer --no-pager
systemctl start pg-backup.service; echo "exit $?"
systemctl status pg-backup.service --no-pager -n 3
journalctl -u pg-backup.service -n 5 --no-pager
rclone lsl offsite:shop-backups/postgres

The retries have a cost. A run that fails quickly is 4 runs over 15 minutes and 5 alerts. I measured that with the printed limits and a stub failing script at the real 5-minute spacing, then with the real script, a stopped database, and an 8 s gap. A manual start inside the hour is refused until you clear the counter:

systemctl reset-failed pg-backup.service
systemctl start pg-backup.service

A run that takes about 10 minutes or longer to fail never reaches the limit, so it keeps dumping and alerting until it succeeds or you run systemctl stop pg-backup.service. I measured that with the retry values scaled down 60 times, a 12 s failing run standing in for 12 minutes, not a real 12-minute failure. I'd accept both, because the 5-minute gap gives a slow crash recovery time to finish.

Why Can a Backup Hang Without an Alert?

The systemd manual says the start timeout is off by default for Type=oneshot, so a hung run stays activating and no alert fires. TimeoutStartSec in the unit above turns that hang into a failed run. The runs behind these settings:

Scaling up from that 12 s run, which I didn't repeat at 2 hours, a real hang costs one alert and one new attempt about every 2 hours until you fix it. Time a real run and keep TimeoutStartSec well above it.

Does Persistent=true Catch a Missed Night?

Yes, and pg-backup.timer above sets it. With Persistent=false or cron, the missed run did not happen. I stopped the server across the trigger twice, and Persistent=true ran the missed backup once each time (first round, trimmed):

starting VPS 2026-11-06T16:47:35Z, 35 s past the trigger
16:47:40+00:00 vps systemd[1]: Started docker.service - Docker Application Container Engine.
16:47:40+00:00 vps systemd[1]: Starting pg-backup.service - Dump Postgres and upload it off-site...
16:47:46+00:00 vps pg-backup[462]: backup ok: shop-20261106T164740Z.dump, 91556784 bytes

This was a container stop and start, so a real reboot could differ.

How Do I Know Last Night's Backup Exists?

Check the bucket, because systemctl list-timers moved its LAST time even after a failing run. With the database stopped, a timer-started run moved LAST from 18:09:41 to 18:19:02, the unit ended with ActiveState=failed, and the newest object in the bucket stayed the older one. Save this as check-backup-fresh:

#!/usr/bin/env bash
# /usr/local/bin/check-backup-fresh
# Usage: check-backup-fresh REMOTE:BUCKET/PREFIX MAX_AGE_HOURS MIN_BYTES
# Exits 0 when the newest object is young and big enough. Exits 2 when it is not.
# Any rclone failure (wrong key, no network, missing bucket) also exits non-zero, through set -e.
set -euo pipefail
remote=${1:?remote}; max_age_hours=${2:?hours}; min_bytes=${3:?bytes}

# --use-server-modtime: judge by when the object reached the bucket, not by file metadata.
listing=$(rclone lsjson --files-only --use-server-modtime "$remote")

count=$(jq 'length' <<<"$listing")
if [ "$count" -eq 0 ]; then echo "FAIL no objects under $remote"; exit 2; fi

newest=$(jq -c 'sort_by(.ModTime) | last' <<<"$listing")
name=$(jq -r '.Path' <<<"$newest")
size=$(jq -r '.Size' <<<"$newest")
modified=$(jq -r '.ModTime' <<<"$newest")

age_seconds=$(( $(date -u +%s) - $(date -u -d "$modified" +%s) ))
max_age_seconds=$(awk -v h="$max_age_hours" 'BEGIN { printf "%d", h * 3600 }')

if [ "$age_seconds" -gt "$max_age_seconds" ]; then
  echo "FAIL $name is $age_seconds s old, limit $max_age_seconds s"; exit 2
fi
if [ "$size" -lt "$min_bytes" ]; then
  echo "FAIL $name is $size bytes, floor $min_bytes"; exit 2
fi
echo "OK $name, $size bytes, $age_seconds s old, $count objects"

Install it on the server and test it as root, with export RCLONE_CONFIG=/etc/pg-backup/rclone.conf set in the shell. This needs rclone, jq, and the offsite remote from my off-site backups guide. shop-backups/postgres and 50000000 are my lab's bucket and floor, so use yours:

install -m 0755 check-backup-fresh /usr/local/bin/check-backup-fresh
check-backup-fresh offsite:shop-backups/postgres 26 50000000; echo "exit $?"

Five of six cases, on small test objects with a 1 MiB floor and a 0.02-hour limit on the stale one:

empty prefix                                 FAIL no objects under offsite:shop-backups/e4-nothing-here
stale, 75 s later                            FAIL shop-005.dump is 75 s old, limit 72 s
tiny newest                                  FAIL shop-002.dump is 1 bytes, floor 1048576
zero newest                                  FAIL shop-003.dump is 0 bytes, floor 1048576
truncated at 75 percent                      OK shop-004.dump, 1572864 bytes, 0 s old, 4 objects

The check proves recency and size, and the last line shows that it doesn't prove content. The sixth case, a list-only key on a fresh object, also passed.

A key that can write can also add a junk object that looks newest, and this check accepts it, as my key and bucket lock guide shows. For content, pg-backup already runs the full read before the upload.

How Do I Run the Check From Another Machine?

Run it from another machine, because a dead server can't report a missing backup. Install the script there with rclone, jq, and GNU date, and give it its own rclone config at /etc/pg-backup/rclone.conf and its own key. My off-site backups guide has the config layout. These units check once a day, with no retries, so one stale backup means one alert.

check-backup-fresh.service:

[Unit]
Description=Check the newest off-site Postgres backup
Wants=network-online.target
After=network-online.target
OnFailure=check-backup-fresh-alert.service

[Service]
Type=oneshot
Environment=RCLONE_CONFIG=/etc/pg-backup/rclone.conf
ExecStart=/usr/local/bin/check-backup-fresh offsite:shop-backups/postgres 26 50000000

check-backup-fresh.timer:

[Unit]
Description=Daily off-site backup check

[Timer]
OnCalendar=*-*-* 09:00:00 UTC
Persistent=true

[Install]
WantedBy=timers.target

check-backup-fresh-alert.service:

[Unit]
Description=Tell someone the off-site backup is missing or stale

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo "$(date -u +%%FT%%TZ) backup check failed on $(hostname)" >> /var/log/check-backup-fresh-alert.log'

Save the three units, then enable the timer:

install -m 0644 check-backup-fresh.service check-backup-fresh.timer check-backup-fresh-alert.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now check-backup-fresh.timer

Replace the remote and the 50000000 floor with yours. pg-backup.timer above runs at 03:00 UTC. At 09:00 a good night's object is about 6 hours old and a missed night's is about 30, so 26 sits between them. Set the 09:00 check and the 26-hour limit from your own schedule.

Point that alert ExecStart at whatever pages you, because as printed it only adds a line to a log file on that machine. I didn't test real delivery.

I ran these units in a second container with a list-only key on MinIO and a lower floor. An empty bucket and stopped storage each raised one alert, and a fresh object raised none.

A key with only s3:ListBucket is enough on MinIO. Per its docs, on R2 the narrowest token is Object Read only, so this machine holds a key that can read your backups.

How Do I Turn the Job Off?

To turn the job off, run systemctl disable --now pg-backup.timer and systemctl stop pg-backup.service, then remove the units and /etc/pg-backup, and revoke the key at the provider. Objects under a lock stay until the lock expires. On the second machine, run systemctl disable --now check-backup-fresh.timer and revoke that key too, or the check alerts every day once the objects go stale.

Summary

Check the bucket from another machine, because a dead server can't report a missing backup and systemctl list-timers moved even after a failing run. check-backup-fresh caught empty, stale, tiny, and zero-byte newest objects, and it passed one truncated at 75 percent.

On the server, use OnFailure, set TimeoutStartSec well above a real run, and keep Persistent=true. These tests ran in containers on MinIO, some with scaled-down timings and none with real alert delivery, so run them on your real provider and server before you count on them.

The backup itself is in my off-site backups guide, and my self-hosting course sets up database backups to storage outside the server.

Thanks for reading.

And stay awesome!