Monitoring BorgBackup from outside the server

If you run borg through borgmatic, connecting Servatum is two lines of config and no new software: borgmatic's existing Healthchecks hook can point at Servatum. If you run borg directly, the Servatum wrapper reads borg's exit code. Either way, a night when borg never runs becomes an alert.

What borg tells you

borg create --json also prints the archive's statistics as JSON.

Like any command-line backup tool, borg only reports when it runs. A lost cron entry, a disabled timer or a dead server produces no return code at all, and that silence is what Servatum watches for.

With borgmatic: two lines

borgmatic has a built-in Healthchecks hook. It takes a ping_url, which doesn't have to be healthchecks.io, and pings it when a backup starts, finishes or fails, optionally sending borgmatic's logs.

Servatum's ping API is planned to be compatible with that scheme, so you point the hook at your Servatum job:

healthchecks:
    ping_url: https://in.servatum.io/p/<job_key>
    send_logs: true

borgmatic's other monitoring hooks can stay configured alongside it.

The Healthchecks-compatible ping API

The same compatibility works for any tool or script that already supports Healthchecks-style ping URLs:

A POST body on any of these is kept as the run's log. If a success ping doesn't arrive within the job's schedule and grace period, the job turns missing.

Without borgmatic: the wrapper

export SERVATUM_JOB_KEY=nightly-home:xxxxxxxx
servatum run -- borg create --json /srv/borg/repo::'{hostname}-{now}' /home /etc

The wrapper reports borg's return code, start and end time, and the statistics from --json, then exits with borg's own return code so existing alerts keep working. Return code 1 is reported as a warning, and counts as failed by default, because it means borg wants you to read the log.

Append-only with borg: read this first

borg can serve a repository in append-only mode, usually by forcing borg serve --append-only in the client's authorized_keys entry. It's useful, but it works differently from what the name suggests:

• A client in append-only mode can still run prune or delete. Nothing is removed; the data is only tagged as deleted in a new transaction.
• borg keeps a transaction log, and you can roll the repository back to an earlier state.
• The catch: as soon as anyone writes to the repository in normal mode, for example a prune from an admin machine, borg permanently removes everything that was tagged as deleted, including what a compromised client tagged.

So before any admin-side prune or compaction, check that the repository is in the state you expect. Running it blindly on a schedule can finish the job an attacker started.

Borg and the Servatum Vault

The Vault is built on restic's append-only REST server, not on borg, so borg repositories can't be stored in it. Borg users get monitoring, alerts and proof pages. See the restic page for how the Vault handles append-only.

What ships when

Servatum is not affiliated with the BorgBackup or borgmatic projects, or with Healthchecks.io.