← All guides

Monitoring backups with a dead-man switch

Backup jobs fail silently — cron won't tell you it stopped. Heartbeat monitoring (Healthchecks.io pattern) fixes the failure mode that matters most.

The failure mode that actually gets you

Backup failures are rarely loud. The password rotated, the disk filled, the container stopped being scheduled — and nothing anywhere alerts, because the thing that would alert is the thing that stopped running. Every veteran has a story about the cron job that had been dead for eight months.

Invert it: silence = alarm

A dead-man switch expects a ping after every successful run and alerts when pings STOP:

pg_dump ... && curl -fsS https://hc-ping.com/<uuid>

Configure the expected period (daily + grace). Now the monitoring doesn't depend on your job being alive — the one dependency you can't trust.

Monitor the whole pipeline, not just the dump

Ping per stage: scheduler ran, backup stored, verification completed, pruning ran. A pipeline that takes backups but stopped verifying them is quietly worthless. Firedrill pings four independent Healthchecks for exactly these stages — the 'never trust the pipeline to report its own failure' principle, applied to ourselves.

A backup you've never restored is a hope, not a backup.

Firedrill restore-tests every backup it takes — on real infrastructure, with the report to prove it.