Backup verification

A backup job that reports “success” but has not written anything new for three weeks goes unnoticed until you need it. With a backup watch you tell monsys where the backup lands; the agent then checks by itself whether something recent has appeared there.

How it works

1. You register a path per agent: a directory (e.g. /var/backups/borg) or a file (e.g. /srv/dumps/db.sql.gz).
2. The hub piggybacks the watch list on the next ingest response; no separate agent configuration is needed.
3. Every 30 minutes the agent stats the path and looks for the newest file (up to 8 directories deep, max. 200 000 entries). It never opens the files: a watch on an encrypted borg or restic repository works without any read access to the archive contents.
4. The agent reports newest_mtime, size and file name as a backup_status payload.
5. The hub compares the age with the configured max_age_hours and alerts when the backup is stale.

Creating a watch

Agent detail page → Backups section → + Watch:

Field	Meaning
Path	absolute path on the host
Label	free name, shown in alerts (empty = the path)
Max age (hours)	1 to 8760; default 26 — just over a daily job

Editor role required; every mutation is written to the audit log. The API is GET/POST /api/v1/backup-watches, PUT/DELETE /api/v1/backup-watches/<id>.

Alerts

The hub worker sweeps all watches every hour:

Situation	Alert
age > max_age	warning — Backup stale:
age > 2 × max_age	critical — same title, higher severity
path unreachable / error on the agent	warning — Backup check failed:

Alerts are deduplicated by title: one open alert per watch, no hourly repeats. They follow the normal alert pipeline (ntfy, mail, webhook, on-call).

Compliance evidence

Migration 133 adds the control ISO27001-A.8.13-backup. Its evidence query counts watches whose last backup is within the max age; that number lands automatically in audit packs and the compliance overview. No watches = no evidence.

Limitations

• The check is mtime-based: a job that writes an empty or corrupt file counts as “fresh”. Combine with restore tests.
• Network shares must be mounted on the host running the agent.
• Agents older than the session-18 release ignore the watch list; the hub then shows an empty last_checked_at.