# How to Detect When a Backup Silently Stops Running

> A hung backup run raised 0 alerts until a start timeout failed it. A freshness check on the bucket caught empty, stale, and tiny backups, not cut ones.

Published: 2026-11-06. Author: Milan Jovanović.

Canonical: https://milanjovanovic.tech/blog/alert-when-backup-stops-running

To catch a backup that stops running, schedule it with a systemd timer, an `OnFailure` unit, and a start timeout, then check the bucket from another machine.
A hung systemd run stayed activating with 0 alerts, and `systemctl list-timers` moved its LAST time even after a failing run.
The bucket check proves age and size, not content.
I didn't test delivery, and the printed alert only writes a log line until you point it at your pager.

I ran this in containers under WSL2, with Docker 29.9 and PostgreSQL 18.6 on Ubuntu 24.04 and [**MinIO**](https://min.io/) as the storage, not on a real VPS or a real provider.
The second machine was a second container, and some systemd runs used scaled-down timings, as the hang table says.

The job this page schedules and checks is `pg-backup`, from my [**off-site backups guide**](https://milanjovanovic.tech/blog/off-site-backups-vps).
The units and the check don't depend on it.
For another job that uploads to a bucket, change `ExecStart`, the bucket path, and the floor.
I only ran them with `pg-backup`.

The scripts and units are in a [**runnable sample**](https://milanjovanovic.tech/code/off-site-backups.zip), whose verify script runs `check-backup-fresh` but none of the units.
Run it on a development machine, not the server, because it mounts the Docker socket, compiles MinIO from source, and leaves base images and build cache behind.

## How Do I Schedule a Backup So a Failed Night Gets Noticed?

Use a systemd timer with an `OnFailure` unit, because cron dropped the output with `No MTA installed, discarding output` and recorded no exit code.

`pg-backup.service`:

```ini
[Unit]
Description=Dump Postgres and upload it off-site
Wants=network-online.target
After=network-online.target docker.service
# A run that fails fast gets four tries, five minutes apart, then it stops until the next timer or a reset-failed.
# A run that takes about 10 minutes or more to fail never reaches this limit and keeps retrying.
StartLimitIntervalSec=1h
StartLimitBurst=4
# Fires after every failed run, and once more when the limit is hit.
OnFailure=pg-backup-alert.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/pg-backup
Restart=on-failure
RestartSec=5min
# Longer than a slow run. Without this a hung run blocks every later night, with no alert.
TimeoutStartSec=2h
# Systemd removes the temp dump even if the script is killed with SIGKILL.
PrivateTmp=true
```

A SIGKILL of the unit with `PrivateTmp=true` removed the temp dump, and a SIGKILL of a hand run, or with `PrivateTmp=false`, left it behind under `/var/tmp`.

`pg-backup.timer`:

```ini
[Unit]
Description=Nightly Postgres backup

[Timer]
OnCalendar=*-*-* 03:00:00 UTC
RandomizedDelaySec=10min
Persistent=true

[Install]
WantedBy=timers.target
```

`pg-backup-alert.service`:

```ini
[Unit]
Description=Tell someone the Postgres backup failed

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo "$(date -u +%%FT%%TZ) pg-backup failed on $(hostname)" >> /var/log/pg-backup-alert.log'
```

Until you change that `ExecStart`, a failed night only adds a line to a log file on this server.
Point it at whatever already pages you, then run `systemctl start pg-backup-alert.service` to see it arrive.
My [**Dokploy operating guide**](https://milanjovanovic.tech/blog/operate-dotnet-apps-on-dokploy) lists what else to alert on.
I didn't test real delivery.

Run this as root, with `pg-backup` installed, the `offsite` remote from my [**off-site backups guide**](https://milanjovanovic.tech/blog/off-site-backups-vps), and `export RCLONE_CONFIG=/etc/pg-backup/rclone.conf` set in the shell.
`shop-backups/postgres` is my lab's bucket, so use yours:

```bash
install -m 0644 pg-backup.service pg-backup.timer pg-backup-alert.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now pg-backup.timer
systemctl list-timers pg-backup.timer --no-pager
systemctl start pg-backup.service; echo "exit $?"
systemctl status pg-backup.service --no-pager -n 3
journalctl -u pg-backup.service -n 5 --no-pager
rclone lsl offsite:shop-backups/postgres
```

The retries have a cost.
A run that fails quickly is 4 runs over 15 minutes and 5 alerts.
I measured that with the printed limits and a stub failing script at the real 5-minute spacing, then with the real script, a stopped database, and an 8 s gap.
A manual start inside the hour is refused until you clear the counter:

```bash
systemctl reset-failed pg-backup.service
systemctl start pg-backup.service
```

A run that takes about 10 minutes or longer to fail never reaches the limit, so it keeps dumping and alerting until it succeeds or you run `systemctl stop pg-backup.service`.
I measured that with the retry values scaled down 60 times, a 12 s failing run standing in for 12 minutes, not a real 12-minute failure.
I'd accept both, because the 5-minute gap gives a slow crash recovery time to finish.

## Why Can a Backup Hang Without an Alert?

The [**systemd manual**](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html) says the start timeout is off by default for `Type=oneshot`, so a hung run stays activating and no alert fires.
`TimeoutStartSec` in the unit above turns that hang into a failed run.
The runs behind these settings:

| Case | Result |
| --- | --- |
| Failing run of 12 s standing in for 12 minutes, with the retry values scaled down 60 times | 11 runs and 10 alerts in 3 minutes, still going |
| Database paused during a dump, no timeout | Still activating 45 s later, 0 alerts, and a later start could not begin another run |
| Same hang, 25 s timeout standing in for 2h, retries off for that test | Failed unit, 1 alert, no dump left behind |
| Hung run, 12 s timeout, retries on | 10 timeouts in 3 minutes, an alert each time, limit never hit |

Scaling up from that 12 s run, which I didn't repeat at 2 hours, a real hang costs one alert and one new attempt about every 2 hours until you fix it.
Time a real run and keep `TimeoutStartSec` well above it.

## Does Persistent=true Catch a Missed Night?

Yes, and `pg-backup.timer` above sets it.
With `Persistent=false` or cron, the missed run did not happen.
I stopped the server across the trigger twice, and `Persistent=true` ran the missed backup once each time (first round, trimmed):

```text
starting VPS 2026-11-06T16:47:35Z, 35 s past the trigger
16:47:40+00:00 vps systemd[1]: Started docker.service - Docker Application Container Engine.
16:47:40+00:00 vps systemd[1]: Starting pg-backup.service - Dump Postgres and upload it off-site...
16:47:46+00:00 vps pg-backup[462]: backup ok: shop-20261106T164740Z.dump, 91556784 bytes
```

This was a container stop and start, so a real reboot could differ.

## How Do I Know Last Night's Backup Exists?

Check the bucket, because `systemctl list-timers` moved its LAST time even after a failing run.
With the database stopped, a timer-started run moved LAST from `18:09:41` to `18:19:02`, the unit ended with `ActiveState=failed`, and the newest object in the bucket stayed the older one.
Save this as `check-backup-fresh`:

```bash
#!/usr/bin/env bash
# /usr/local/bin/check-backup-fresh
# Usage: check-backup-fresh REMOTE:BUCKET/PREFIX MAX_AGE_HOURS MIN_BYTES
# Exits 0 when the newest object is young and big enough. Exits 2 when it is not.
# Any rclone failure (wrong key, no network, missing bucket) also exits non-zero, through set -e.
set -euo pipefail
remote=${1:?remote}; max_age_hours=${2:?hours}; min_bytes=${3:?bytes}

# --use-server-modtime: judge by when the object reached the bucket, not by file metadata.
listing=$(rclone lsjson --files-only --use-server-modtime "$remote")

count=$(jq 'length' <<<"$listing")
if [ "$count" -eq 0 ]; then echo "FAIL no objects under $remote"; exit 2; fi

newest=$(jq -c 'sort_by(.ModTime) | last' <<<"$listing")
name=$(jq -r '.Path' <<<"$newest")
size=$(jq -r '.Size' <<<"$newest")
modified=$(jq -r '.ModTime' <<<"$newest")

age_seconds=$(( $(date -u +%s) - $(date -u -d "$modified" +%s) ))
max_age_seconds=$(awk -v h="$max_age_hours" 'BEGIN { printf "%d", h * 3600 }')

if [ "$age_seconds" -gt "$max_age_seconds" ]; then
  echo "FAIL $name is $age_seconds s old, limit $max_age_seconds s"; exit 2
fi
if [ "$size" -lt "$min_bytes" ]; then
  echo "FAIL $name is $size bytes, floor $min_bytes"; exit 2
fi
echo "OK $name, $size bytes, $age_seconds s old, $count objects"
```

Install it on the server and test it as root, with `export RCLONE_CONFIG=/etc/pg-backup/rclone.conf` set in the shell.
This needs rclone, jq, and the `offsite` remote from my [**off-site backups guide**](https://milanjovanovic.tech/blog/off-site-backups-vps).
`shop-backups/postgres` and `50000000` are my lab's bucket and floor, so use yours:

```bash
install -m 0755 check-backup-fresh /usr/local/bin/check-backup-fresh
check-backup-fresh offsite:shop-backups/postgres 26 50000000; echo "exit $?"
```

Five of six cases, on small test objects with a 1 MiB floor and a 0.02-hour limit on the stale one:

```text
empty prefix                                 FAIL no objects under offsite:shop-backups/e4-nothing-here
stale, 75 s later                            FAIL shop-005.dump is 75 s old, limit 72 s
tiny newest                                  FAIL shop-002.dump is 1 bytes, floor 1048576
zero newest                                  FAIL shop-003.dump is 0 bytes, floor 1048576
truncated at 75 percent                      OK shop-004.dump, 1572864 bytes, 0 s old, 4 objects
```

The check proves recency and size, and the last line shows that it doesn't prove content.
The sixth case, a list-only key on a fresh object, also passed.

A key that can write can also add a junk object that looks newest, and this check accepts it, as my [**key and bucket lock guide**](https://milanjovanovic.tech/blog/r2-bucket-lock-backups) shows.
For content, `pg-backup` already runs the [**full read before the upload**](https://milanjovanovic.tech/blog/verify-pg-dump-backup).

## How Do I Run the Check From Another Machine?

Run it from another machine, because a dead server can't report a missing backup.
Install the script there with rclone, jq, and GNU date, and give it its own rclone config at `/etc/pg-backup/rclone.conf` and its own key.
My [**off-site backups guide**](https://milanjovanovic.tech/blog/off-site-backups-vps) has the config layout.
These units check once a day, with no retries, so one stale backup means one alert.

`check-backup-fresh.service`:

```ini
[Unit]
Description=Check the newest off-site Postgres backup
Wants=network-online.target
After=network-online.target
OnFailure=check-backup-fresh-alert.service

[Service]
Type=oneshot
Environment=RCLONE_CONFIG=/etc/pg-backup/rclone.conf
ExecStart=/usr/local/bin/check-backup-fresh offsite:shop-backups/postgres 26 50000000
```

`check-backup-fresh.timer`:

```ini
[Unit]
Description=Daily off-site backup check

[Timer]
OnCalendar=*-*-* 09:00:00 UTC
Persistent=true

[Install]
WantedBy=timers.target
```

`check-backup-fresh-alert.service`:

```ini
[Unit]
Description=Tell someone the off-site backup is missing or stale

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo "$(date -u +%%FT%%TZ) backup check failed on $(hostname)" >> /var/log/check-backup-fresh-alert.log'
```

Save the three units, then enable the timer:

```bash
install -m 0644 check-backup-fresh.service check-backup-fresh.timer check-backup-fresh-alert.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now check-backup-fresh.timer
```

Replace the remote and the `50000000` floor with yours.
`pg-backup.timer` above runs at 03:00 UTC.
At 09:00 a good night's object is about 6 hours old and a missed night's is about 30, so 26 sits between them.
Set the 09:00 check and the 26-hour limit from your own schedule.

Point that alert `ExecStart` at whatever pages you, because as printed it only adds a line to a log file on that machine.
I didn't test real delivery.

I ran these units in a second container with a list-only key on MinIO and a lower floor.
An empty bucket and stopped storage each raised one alert, and a fresh object raised none.

A key with only `s3:ListBucket` is enough on MinIO.
Per its docs, on R2 the narrowest token is `Object Read only`, so this machine holds a key that can read your backups.

## How Do I Turn the Job Off?

To turn the job off, run `systemctl disable --now pg-backup.timer` and `systemctl stop pg-backup.service`, then remove the units and `/etc/pg-backup`, and revoke the key at the provider.
Objects under a lock stay until the lock expires.
On the second machine, run `systemctl disable --now check-backup-fresh.timer` and revoke that key too, or the check alerts every day once the objects go stale.

## Summary

Check the bucket from another machine, because a dead server can't report a missing backup and `systemctl list-timers` moved even after a failing run.
`check-backup-fresh` caught empty, stale, tiny, and zero-byte newest objects, and it passed one truncated at 75 percent.

On the server, use `OnFailure`, set `TimeoutStartSec` well above a real run, and keep `Persistent=true`.
These tests ran in containers on MinIO, some with scaled-down timings and none with real alert delivery, so run them on your real provider and server before you count on them.

The backup itself is in my [**off-site backups guide**](https://milanjovanovic.tech/blog/off-site-backups-vps), and my [**self-hosting course**](https://milanjovanovic.tech/pragmatic-self-hosting) sets up database backups to storage outside the server.

Thanks for reading.

And stay awesome!

---

## Frequently asked questions

### Does systemctl list-timers show that a backup succeeded?

No. Its LAST time moved even after a failing run, so check the bucket instead.

### What happens to a hung backup unit with no timeout?

With the database paused during a dump, the unit was still activating 45 seconds later with 0 alerts, and a later start could not begin another run. The systemd manual says the start timeout is off by default for Type=oneshot.

### Why run a backup freshness check from another machine?

A dead server cannot report a missing backup. I ran the check units in a second container with a list-only key on MinIO, not on a real second server, and I did not test real alert delivery.

### Does a freshness check prove a backup is good?

No. It proves recency and size. An object truncated at 75 percent passed it, so a dump still needs a full read before it uploads.