How to get alerted when a backup doesn't run

How a heartbeat works

A heartbeat, also called a dead man's switch, is a URL with a schedule attached. You say how often the job runs and how late it may be. The job calls the URL at the end of each successful run. If no call arrives within the interval plus the grace period, you get an alert.

In Acciti the interval can be anything from a minute to 30 days, with up to a day of grace. A missed check-in raises one alert, not one a minute, and when the job checks in again you get a quiet note saying how long it was gone.

Choosing the grace period

Set the interval to the schedule, and the grace to how long the job can take plus a margin. A nightly backup that usually finishes in twenty minutes but takes two hours on the first of the month wants a 24-hour interval and three hours of grace. Too little grace and a slow run wakes you. Too much and a missed backup is reported the next afternoon.

restic

0 3 * * *  restic backup /home /etc --quiet && \  curl -fsS -m 10 --retry 3 https://acciti.com/v1/heartbeats/$HEARTBEAT_TOKEN/ping

restic exits with 3 when it saved a snapshot but couldn't read some files. To count that as a success, check the code yourself:

#!/bin/shrestic backup /home /etc --quietstatus=$?if [ "$status" -eq 0 ] || [ "$status" -eq 3 ]; then  curl -fsS -m 10 --retry 3 https://acciti.com/v1/heartbeats/$HEARTBEAT_TOKEN/pingfiexit "$status"

Put restic check on a weekly heartbeat of its own. A backup that runs every night into a corrupted repository is still a problem.

borgmatic

0 3 * * *  borgmatic --verbosity -1 && \  curl -fsS -m 10 --retry 3 https://acciti.com/v1/heartbeats/$HEARTBEAT_TOKEN/ping

borgmatic runs create, prune and check in one go and exits non-zero if any of them fail, so one heartbeat covers the whole run.

rsync

30 2 * * *  rsync -a --delete /srv/ backup@nas:/volume1/srv/ && \  curl -fsS -m 10 --retry 3 https://acciti.com/v1/heartbeats/$HEARTBEAT_TOKEN/ping

A systemd timer

ExecStartPost runs only if ExecStart succeeded, which behaves the same as &&:

[Unit]Description=Nightly backup [Service]Type=oneshotExecStart=/usr/local/bin/backup.shExecStartPost=/usr/bin/curl -fsS -m 10 --retry 3 https://acciti.com/v1/heartbeats/$HEARTBEAT_TOKEN/ping

If the timer is disabled, the machine is off or the unit file vanishes in a rebuild, nothing runs and nothing pings. That silence is what the heartbeat is there to hear.

Hearing about failures as well as silence

A heartbeat tells you a backup didn't succeed by the deadline. To hear the moment it fails, and why, send an alert from the failure path too:

#!/bin/shif ! output=$(restic backup /home /etc --quiet 2>&1); then  curl -fsS https://acciti.com/v1/messages \    -H "Authorization: Bearer $ACCITI_TOKEN" \    -H "Content-Type: application/json" \    -d "$(jq -n --arg m "$(echo "$output" | tail -n 5)" \      '{user: env.ACCITI_USER, title: "Backup failed", message: $m, priority: 1}')"  exit 1ficurl -fsS -m 10 --retry 3 https://acciti.com/v1/heartbeats/$HEARTBEAT_TOKEN/ping

The acciti run wrapper in the API documentation does the same in one line: it sends the last lines of output when the command fails, and checks the heartbeat in when it works.

What the alert should sound like

A missed backup is rarely a 3 a.m. emergency. It is a morning problem, unless it is the third night in a row. Heartbeats start at High priority, which makes a sound and gets through Focus. Save Emergency for jobs whose silence costs money by the hour, and attach an escalation policy if someone else should hear about it while you're away.

Heartbeats are on the Solo plan and up. Compare plans

Heartbeats covers what else is worth watching: certificate renewals, queue workers, scheduled exports and syncs.