10 — Devlog

Backups with receipts

devlog · ship three: backups with receipts

Night three of the loop, and ship #3 is already in the water. Yesterday's 04:00 implementer picked "deduplicated backups with a restore drill" off the radar board, and twenty-one minutes past the hour the Pi had borg archives of everything git can't hold, a daily timer, and a Sunday-morning drill that actually restores the data and byte-compares it against the living system. Yesterday's post called an untested backup a rumour; as of that drill, this box's backups are fact. And late last night a new private repo appeared on the Pi — my own Counter-Strike stats finally got a proper home.

Shipped

  • pi-cicd — implementer ship #3 off the radar board: pi-backup, a stdlib-Python wrapper around borg 1.4 from the plain Debian package (still no containers on this Pi). It covers the /etc state git cannot hold — the ntfy server config and user database, loop configs, systemd units — on a daily 03:30 create-and-prune timer with 7 daily / 4 weekly / 6 monthly retention. The clever bit is the weekly Sunday 05:30 restore drill: it extracts a fresh archive and byte-compares sha256 digests against the live sources, then publishes PASS/FAIL to the ntfy backups topic whose ACL was provisioned two ships ago. First live run archived six files, the first live drill byte-compared 6/6, and both notifications were read back off the topic with the subscriber token. Passphrase and ntfy keys live in a root-only config file outside the repo, and the suite is at 71 tests green locally and on CI — 21 of them new, exercising a real borg binary rather than mocks. The one honest gap: the repository is still on the SD card; the runbook says moving it to USB storage is one config line, and that step is mine.
  • A new repo for my own match history (private, so no link) — a single-page CS2 tracker that puts my FACEIT and Valve Premier stats on one dashboard, served from the Pi on the tailnet. The interesting part is that the two data paths can't be symmetrical: FACEIT has a proper Data API, so full match history, per-match K/D/A and the ELO curve backfill automatically — but Valve publishes no match-history API for Premier at all. So the tracker takes lifetime stats from the Steam Web API and fills the gap with a Game State Integration listener: the game client on my PC POSTs state to the Pi during each Premier match, and the tracker assembles a match history from install day forward. Auto-adopted by project-guard at its first commit, CI and all — the adoption pipeline remains faster than I am.
  • The homepage caught up with the loop — the ntfy backbone and the funnel went onto the architecture diagram and the "Currently" panel learned about the notification work, in the content commit that followed yesterday's post out the door.

On the radar

  • Uptime Kuma service monitoring — alerts have a backbone and backups have receipts; now give the services themselves a scoreboard. Concrete next step: install Uptime Kuma on the Pi (no containers), add one HTTP monitor per long-running service — AdGuard DNS, the dashboards, the portal, the public funnel endpoints — plus push-type heartbeat monitors that second-guess the same cron jobs loop-heartbeat watches, and wire the alert channel to the ntfy server so pages arrive where the heartbeat pages do.
  • changedetection.io with LLM rules — stop finding out about upstream releases by accident. Next step: install, seed it with the release pages of everything deployed here (ais-catcher, AdGuardHome, ntfy itself) and the status pages of the data sources, then add one digesting rule that summarises what changed into a single notification on the notifications topic.
  • Prometheus + Grafana + node_exporter from Debian packages — real graphs for the devlog to cite: CPU temperature against load, the SDR feed's throughput, service health over time. Next step: install all three from apt packages with systemd units (the no-container way this box does things), point Prometheus at node_exporter plus the long-running services, and pin one dashboard with the graphs the devlog actually quotes.

Interesting reads

  • How to Prove Your Backups Will Actually Restore — an enterprise-consultancy take on restore evidence, anchored to NIST CSF 2.0: a green backup job proves data was copied, not that the copy is complete, readable, or usable. Restore testing is written into the outcome itself, twice. Deep enterprise backup troubleshooting is my day job, and the argument maps one-to-one onto a Pi in a kitchen — only the budget differs.
  • Restore Drills: the quarterly discipline that finds broken backups — the minimum-viable drill (restore a random file, byte-compare, time it), what breaks first (passphrase rotation, silently stale dumps, keys that only lived on the dead host), and the distinction that matters: borg check verifies the archive; only a drill verifies the recovery. Sunday's drill here is basically their level-one exercise — their "what breaks first" list is now my checklist.
  • Monitoring Your Raspberry Pi with Prometheus and Grafana — a compact bookworm-tested walkthrough that installs Prometheus from apt, node_exporter as a plain systemd service, and Grafana from its own repo — no Docker in sight. If the graphs idea gets picked off the board, this is the exact shape it should take on this box.
Back to the devlog