Monitoring restic backups from outside the server

restic is precise about what happened when it runs: an exit code and a JSON summary say whether the snapshot is complete. What it can't do is notice the night it didn't run at all. Servatum is built for that, and for keeping an offsite restic repository that a compromised server can't erase.

What restic tells you

Every restic backup run ends with one of these exit codes:

With --json, the last line of output is a summary message ("message_type": "summary") with the new snapshot_id, data_added in bytes, file counts, total_bytes_processed and total_duration.

Exit code 3 is the one that slips through. The run prints "snapshot saved", so a script or a person skimming the output reads it as success.

What restic can't tell you

restic is a command-line program. It runs when cron or a systemd timer starts it, and reports only when it runs. Everything below produces no error at all:

• The cron entry or timer was lost during a server migration or rebuild.
• The wrapper script ignores the exit code, or ends with a command that always succeeds.
• The job fails with exit code 1 every night, and the cron mail goes to a mailbox nobody reads.
• A path was renamed, so the job now backs up a nearly empty directory and succeeds in seconds.
• Nobody has run restic check since the repository was created.

These failures don't announce themselves. The only reliable signal is a report that should have arrived and didn't, which has to be watched from somewhere other than the server itself.

How Servatum works with restic

export SERVATUM_JOB_KEY=nightly-home:xxxxxxxx
servatum run -- restic backup /home /etc --json

export SERVATUM_JOB_KEY=weekly-check:xxxxxxxx
servatum run -- restic check --read-data-subset=1/10

Reporting without the wrapper

If you'd rather keep your own script, the same report is one HTTPS call. This uses only restic's documented exit codes and JSON summary:

#!/bin/bash
restic backup /home /etc --json > /tmp/restic-backup.json
RC=$?

SUMMARY=$(jq -c 'select(.message_type=="summary")' /tmp/restic-backup.json)
SNAPSHOT=$(echo "$SUMMARY" | jq -r '.snapshot_id // empty')
ADDED=$(echo "$SUMMARY" | jq -r '.data_added // 0')

[ "$RC" -eq 0 ] && STATUS=ok || STATUS=failed

curl -fsS -m 10 -X POST "https://in.servatum.io/v1/jobs/$SERVATUM_JOB_KEY/runs" \
  -H 'Content-Type: application/json' \
  -d "{\"status\":\"$STATUS\",\"snapshot_id\":\"$SNAPSHOT\",
       \"message\":\"restic exit $RC, $ADDED bytes added\"}"

exit $RC

The script ends with restic's own exit code, so whatever already watches the cron job still sees the real result.

Append-only, done properly

An offsite repository only protects you from a compromised server if that server can't delete from it. restic supports this through its rest-server in --append-only mode, which accepts new backups but refuses deletion and modification of existing data. rclone's restic server mode can provide the same in front of other storage.

Two details from restic's own security notes decide whether this actually works:

• Removing old snapshots needs full access. forget and prune need read, write and delete rights. If the backed-up server holds those, append-only protects nothing. Run them from a separate, well-secured machine.
• Use time-based retention. An attacker who can only add snapshots can still add fake ones, timed so that a policy like "keep the last three weekly snapshots" keeps the fakes and deletes the real ones. restic recommends forget --keep-within for append-only repositories, because it keeps every snapshot in the window, real and fake, and leaves time to notice that something is wrong.

# On the storage host: new snapshots accepted, deletes refused
rest-server --path /srv/restic --append-only --private-repos

# On a separate admin machine, with full access to the repository:
# keep everything from the last 90 days, then prune
restic -r /srv/restic/server1 forget --keep-within 90d --prune

The Servatum Vault

The Vault is this setup, run for you. Your server writes to a restic REST repository with credentials that can add snapshots but never remove them. Retention runs on our side and is time-based, as restic recommends. Encryption happens on your server, so we never hold your key.

Setup details are in the Vault docs.