Backup monitoring in Prometheus

Scrape one Servatum endpoint and every backup job and tier becomes a set of Prometheus metrics, led by the one Prometheus itself calls the key metric of a batch job: the last time it succeeded. Ready-made alert rules cover failed, late and under-retained backups, and Servatum being unreachable.

Scrape, don't push

Servatum exposes a metrics endpoint for Prometheus to scrape over HTTPS. Your Prometheus only makes outbound requests, and nothing has to be opened to the internet.

We don't push into a Pushgateway. Prometheus's own guidance is that the Pushgateway's only valid use is usually a service-level batch job, and that pushing gives up the up metric Prometheus generates on every scrape. With scraping, up tells you whether Servatum itself is reachable.

The metrics

Every series carries two labels: backup_job and tier. We deliberately don't use a label named job: Prometheus attaches its own job label to every target and would rename ours to exported_job.

Names follow Prometheus's naming guidance: one unit per metric, base units, and the unit as a suffix.

Exposing the schedule and target as metrics keeps the alert rules generic. The rule compares each tier against its own schedule, so you don't keep thresholds in two places.

servatum_tier_status{backup_job="nightly-full",tier="offsite"} 1
servatum_tier_last_success_timestamp_seconds{backup_job="nightly-full",tier="offsite"} 1792800000
servatum_tier_expected_interval_seconds{backup_job="nightly-full",tier="offsite"} 86400
servatum_tier_grace_seconds{backup_job="nightly-full",tier="offsite"} 10800
servatum_tier_last_size_bytes{backup_job="nightly-full",tier="offsite"} 48211993600
servatum_tier_restore_points{backup_job="nightly-full",tier="offsite"} 34
servatum_tier_restore_points_target{backup_job="nightly-full",tier="offsite"} 30

Scrape configuration

Create a read-only token in Servatum, save it to a file on the Prometheus host, and add a scrape job:

scrape_configs:
  - job_name: servatum
    scheme: https
    metrics_path: /v1/metrics
    scrape_interval: 2m
    authorization:
      credentials_file: /etc/prometheus/servatum.token
    static_configs:
      - targets: ['api.servatum.io']

The token can only read status. It can't change jobs or see anything else in your account.

Alert rules

We'll publish these as a ready-to-load rules file:

groups:
  - name: servatum
    rules:
      - alert: BackupTierFailed
        expr: servatum_tier_status == 0
        for: 5m
        annotations:
          summary: "{{ $labels.backup_job }} / {{ $labels.tier }} failed or missing"

      - alert: BackupTierLate
        expr: |
          time() - servatum_tier_last_success_timestamp_seconds
            > servatum_tier_expected_interval_seconds + servatum_tier_grace_seconds
        annotations:
          summary: "{{ $labels.backup_job }} / {{ $labels.tier }} is past its schedule"

      - alert: BackupRetentionLow
        expr: servatum_tier_restore_points < servatum_tier_restore_points_target
        for: 1h
        annotations:
          summary: "{{ $labels.backup_job }} / {{ $labels.tier }} below its retention target"

      - alert: ServatumUnreachable
        expr: up{job="servatum"} == 0
        for: 15m
        annotations:
          summary: "Prometheus can't reach Servatum"

BackupTierLate uses Prometheus's own clock, so it still fires if Servatum's status were stale. ServatumUnreachable covers the case where Prometheus can't reach Servatum at all.

What ships when

Prometheus is a Cloud Native Computing Foundation project. Servatum is not affiliated with it.