Headscale Backup and Restore: What Each Loss Takes

Back up Headscale's noise key and its database, and use SQLite's .backup for the database. In my container tests, a restore at the same address brought all 4 nodes back with no client action, and so did one restore at a new address. Real DNS changes and Let's Encrypt weren't part of those tests.

I tested Headscale 0.29.7, Headplane 0.8.2, Caddy 2.11.8 and Tailscale 1.104 clients with the VPS and four machines (app, db, worker and laptop) running as containers on separate Docker networks. Certificates came from Caddy's internal CA and names from /etc/hosts, so real DNS, Let's Encrypt and real NAT traversal were not part of the test. I also didn't run a provider reboot or any macOS, Windows or mobile client.

The steps assume the stack from my Headscale setup guide in /opt/headscale, with the server commands run as root. The backup script below is also in the sample download, and the sample's test script doesn't run it.

What Do You Need to Back Up?

Back up the noise key and the database, and use SQLite's .backup for the database. The noise key is /opt/headscale/data/noise_private.key, which Headscale uses in the Noise handshake with each client. With Headscale running, a cp of db.sqlite alone missed a row still in the write-ahead log, and .backup didn't (trimmed):

--- journal mode: wal
cp without the -wal file: users=1, has waltest=0
.backup:                  users=2, has waltest=1

The backup script below needs sqlite3 on the host, because the Headscale image has no shell. Install it with apt-get install -y sqlite3 age (age is for the off-site copy), and save this as /usr/local/bin/backup-headscale.sh with mode 755:

#!/bin/sh
# Online backup of a Headscale stack that lives in /opt/headscale. Needs: apt install sqlite3
# The archive holds private keys and tokens: encrypt it before it leaves the server.
set -eu
umask 077
SRC=/opt/headscale
DEST=${1:-/var/backups/headscale}
STAMP=$(date -u +%Y%m%dT%H%M%SZ)
WORK=$(mktemp -d)
trap 'rm -rf "$WORK"' EXIT

mkdir -p "$DEST" "$WORK/data"
test -s "$SRC/data/db.sqlite"
# A plain cp of db.sqlite misses whatever is still in the write-ahead log. .backup does not.
sqlite3 "$SRC/data/db.sqlite" ".backup '$WORK/data/db.sqlite'"
test "$(sqlite3 "$WORK/data/db.sqlite" 'PRAGMA integrity_check;')" = ok
cp -p "$SRC/data/noise_private.key" "$WORK/data/"
# Only exists when the embedded DERP server has been enabled.
[ -f "$SRC/data/derp_server_private.key" ] && cp -p "$SRC/data/derp_server_private.key" "$WORK/data/"
cp -rp "$SRC/config" "$SRC/caddy" "$SRC/compose.yaml" "$WORK/"
# Headplane, if you added it. Its config holds the cookie secret.
if [ -f "$SRC/compose.override.yaml" ]; then
  mkdir -p "$WORK/headplane"
  cp -p "$SRC/compose.override.yaml" "$WORK/"
  cp -p "$SRC/headplane/config.yaml" "$WORK/headplane/"
fi

# Write to a temporary name first, so a full disk never leaves a partial archive that looks real.
tar -czf "$DEST/.partial-$STAMP" -C "$WORK" .
mv "$DEST/.partial-$STAMP" "$DEST/headscale-$STAMP.tar.gz"
echo "$DEST/headscale-$STAMP.tar.gz"

How Do You Run the Backup Every Night?

Run it nightly from /etc/cron.d/headscale-backup, mode 644. I ran this line on a cron daemon with a one minute schedule, and it produced an archive:

15 3 * * * root /usr/local/bin/backup-headscale.sh /var/backups/headscale >/dev/null

Cron mails errors only if the server has a mail program. My Ubuntu 24.04 lab node had none, so journalctl -u cron showed No MTA installed, discarding output. So check the date of the newest archive in the folder now and then. My backup alerts guide uses a systemd timer with OnFailure for a Postgres job because of that dropped output, and I ran this script only from cron. The script also never deletes old archives.

How Do You Copy the Backup Off the Server?

For the off-site copy, my key and bucket lock guide covers key scope and locks, and my backup alerts guide covers freshness checks. The script in my off-site backups guide only dumps Postgres, so it doesn't copy this archive.

This is the manual copy I ran with age. I didn't script this step, so the nightly job alone keeps archives on the VPS. A disk or provider loss takes them with it until you copy one off. Install age on your own machine too. age-keygen prints the age1... key for <AGE_PUBLIC_KEY>, and <STAMP> is the timestamp in the archive name:

# On your own machine, once
age-keygen -o headscale-backup.key

# On the VPS
age -r <AGE_PUBLIC_KEY> -o /var/backups/headscale/headscale-<STAMP>.tar.gz.age /var/backups/headscale/headscale-<STAMP>.tar.gz

<VPS_SSH> is your SSH login, such as deploy@203.0.113.10. With root login off, your user can't read /var/backups/headscale. So on the VPS, copy the encrypted file to that user's home, where <SSH_USER> is the user in <VPS_SSH>:

sudo install -m 600 -o <SSH_USER> /var/backups/headscale/headscale-<STAMP>.tar.gz.age /home/<SSH_USER>/

I ran that as a sudo user in a separate container.

# On your own machine
scp <VPS_SSH>:headscale-<STAMP>.tar.gz.age .
age -d -i headscale-backup.key -o headscale-<STAMP>.tar.gz headscale-<STAMP>.tar.gz.age
tar -tzf headscale-<STAMP>.tar.gz

A non-root login got the home copy in a separate test and failed on the /var/backups/headscale path. A root login can read that path directly with scp <VPS_SSH>:/var/backups/headscale/headscale-<STAMP>.tar.gz.age ., which is what my own encrypted copy did.

The private key is the only way to read these backups, so keep a second copy. Delete a decrypted copy when you finish, because it holds the noise key, the database and, with Headplane, the cookie secret.

How Do You Restore Headscale on the Same VPS?

To restore on the same VPS, check the archive first:

ARCHIVE=/var/backups/headscale/headscale-<STAMP>.tar.gz
CHECK=$(mktemp -d)
tar -xzf "$ARCHIVE" -C "$CHECK" \
  && test -s "$CHECK/data/db.sqlite" \
  && test "$(sqlite3 "$CHECK/data/db.sqlite" 'PRAGMA integrity_check;')" = ok \
  && test -s "$CHECK/data/noise_private.key" \
  && test -s "$CHECK/compose.yaml" \
  && echo "ARCHIVE OK, nodes: $(sqlite3 "$CHECK/data/db.sqlite" 'SELECT count(*) FROM nodes;')" \
  || echo "BAD ARCHIVE, do not continue"

Continue only if it printed ARCHIVE OK with the node count you expect. Stay in the same shell, because the next block uses $CHECK. On BAD ARCHIVE, run rm -rf "$CHECK", because it can hold the noise key. A truncated archive, one with no noise key and one with no database all printed BAD ARCHIVE.

cd /opt/headscale && docker compose down
cd /
mv /opt/headscale "/opt/headscale.before-restore-$(date -u +%Y%m%dT%H%M%SZ)"
mv "$CHECK" /opt/headscale
mkdir -p /opt/headscale/headplane/data
cd /opt/headscale && docker compose up -d --force-recreate
docker exec headscale headscale nodes list

--force-recreate matters when the directory disappeared with the stack running. Without it, Compose left Headscale running on the deleted files, nodes list looked normal, and a user I created there was gone after the next restart. In that case the cd and first mv print errors, and the remaining lines still work. The archive leaves out headplane/data, which Headplane uses if you added it, so the restore recreates it empty. A login with a new API key still worked.

Delete /opt/headscale.before-restore-* once you trust the result, because it holds the same secrets. Before that, this undoes the restore, with <STAMP> from the directory's name (I ran this with no nodes joined):

cd /opt/headscale && docker compose down
cd /
mv /opt/headscale /opt/headscale.failed-restore
mv /opt/headscale.before-restore-<STAMP> /opt/headscale
cd /opt/headscale && docker compose up -d

How Do You Restore Headscale on a New VPS?

To restore on a new VPS with Docker installed, first stop Headscale on the old server if it still runs. My one run had it stopped. Open 80 and 443 in the new server's provider firewall, move the A record so Caddy can get its certificate, copy the decrypted archive to /root, and run this:

mkdir -p /opt/headscale
tar -xzf /root/headscale-<STAMP>.tar.gz -C /opt/headscale
mkdir -p /opt/headscale/headplane/data
cd /opt/headscale && docker compose up -d
docker exec headscale headscale nodes list
rm /root/headscale-<STAMP>.tar.gz

I ran that block except the last rm, on an image of the old server with /opt/headscale and the backups removed, with /etc/hosts standing in for the A record and the old Caddy CA, so the firewall, A record and certificate steps are untested. The archive doesn't hold the backup job, so install sqlite3 and age and recreate the script and the cron file.

What Happens When You Lose the Noise Key or the Server?

Each loss behaved differently in my tests:

I produced the noise key loss by stopping Headscale, deleting the key and starting it again, which made Headscale write a new one. After a lost noise key, run sudo systemctl restart tailscaled on each machine from public SSH or your provider's console, because it can cut a session over the tailnet. In 3 runs, all 4 nodes were online within 2 s of the last restart, with the same IPs and no new key.

Use MagicDNS names, because anything pinned to an old address breaks in the last row. A container on a Compose network resolved those names in my lab. One on Docker's default bridge got NXDOMAIN while the host listed another DNS server first, so test from inside your containers.

A restore also brings back any machine you deleted after the backup. In my test a deleted worker came back online and reached app:4222 with no new key. So read nodes list after a restore and delete those machines again.

So DNS TTLs, client DNS caching and a new Let's Encrypt certificate aren't covered by the new address run, and clients could take longer.

Summary

Back up the noise key and the database, use .backup for the database, and copy the archive off the server encrypted. Cron mails errors only if the server has a mail program, and the script never deletes old archives, so check the date of the newest archive now and then.

Check the archive before a restore, and read nodes list afterwards, because a restore brings back machines you deleted after the backup. Use MagicDNS names, because losing everything with no backup gives every machine a new address.

These results come from containers, and the new address restore was one run with /etc/hosts and the old Caddy CA, so DNS TTLs, client DNS caching and a new Let's Encrypt certificate aren't covered.

The rest of the stack is in my Headscale guide, and my self-hosting course sets up database backups to storage outside the server.

Thanks for reading.

And stay awesome!