12 — Devlog

Twelve probes and a scoreboard

devlog · ship five: the box takes its own pulse

Night five of the loop, and the radar board went after the one question left unanswered by the audit layer: not "is the paperwork in order" but "are the services actually up, right now?" The 04:00 implementer shipped number five — a stdlib prober that asks every endpoint on this box how it's doing every five minutes and files complaints through the notification backbone. By breakfast the portal had a scoreboard for it. The morning doctor, one day into the job, turned out to need a doctor itself — and got one by 07:20.

Shipped

  • pi-cicd — implementer ship #5 off the radar board: service-probe, the concretised version of the Uptime Kuma idea (the app needs containers, which this box forbids — the underlying need doesn't). Every five minutes it runs twelve probes: HTTP checks against the seven local web services — gated on the HEALTHZ contract where one exists, so a polite 200 with an error page doesn't count as up — three public funnel endpoints, an ntfy self-check, and a hand-built UDP DNS query for AdGuardHome, because DNS has no HTTP to negotiate with. DOWN is only declared after two consecutive failures so a flappy restart stays quiet; recovery notices arrive with the downtime duration attached. Alerts go to a new ntfy services topic under the same ACL discipline as every other topic — publisher write-only, subscriber read-only, anonymous denied — and state lands atomically in a status.json something else can read. First live sweep: 12/12 up. Then the committed drill: seed a dead port, watch the DOWN alert and the recovery notice arrive, read both back with the subscriber token. 139/139 tests locally and CI green.
  • The portal grew a scoreboard — project-hub commit 1c2b9f3: a new Uptime Scoreboard panel fed by service-probe's status.json over an /api/probes endpoint, deployed by the pull-based pipeline like everything else. The bookmark I open dozens of times a day now answers "is anything broken" at a glance, instead of "which tab was that".
  • Two CI bugs found the hard way — tests in pi-cicd that depended on the live tracker port and on live DNS had the suite red on runners since 08-26 while staying green locally, which is the worst kind of lie a test suite can tell. Both are hermetic now (or skip cleanly where the dependency can't be faked). Credit where due: the implementer's own done-means-green gate is what flushed them out — the audit layer auditing the audit layer.
  • pi-doctor's first house calls — the 07:00 audit crashed on its own bookkeeping: save_state referenced a variable that didn't exist, so any run with findings died before it could alert anyone. Fixed, along with two genuinely useful additions. A wedged RTL-SDR has a signature — thirty timeouts in ten minutes — and the sysfs authorize toggle does not clear it, but a USBDEVFS_RESET does; the doctor now resets the dongle itself, skips the manoeuvre during NOAA captures, and restarts the SDR consumers around the reset. And the AIS revive check waits ten seconds before declaring "start failed", because the dongle is shared and normal stop/start cycles were being read as outages. The X bot's data feed stops crying wolf.

On the radar

  • Alert-storm kill switch and notifier timeouts — yesterday's ntfy read suggested two details worth stealing: a one-variable global mute for alert storms, and a timeout on every notification request so a dead notification server can't hang the job that was trying to report its own success. Next step: add both to the shared ntfy helper in pi-cicd — a mute flag checked before publish, connect+read timeouts on every request, tests for the storm path and the hang path — and let service-probe, release-watch, pi-backup, heartbeat and the doctor inherit them for free.
  • Prometheus + Grafana + node_exporter from Debian packages — the last monitoring idea still on the board from the original research batch: real graphs for these posts to cite — CPU temperature against load, service health over time, disk and network throughput. Next step: install all three from apt with systemd units (the no-container way this box does things), point Prometheus at node_exporter and the long-running services, pin one dashboard with the graphs the devlog actually quotes, and route alerting through the existing ntfy topics.
  • Grow pi-cicd into the single architecture reference — carried over: how this Pi builds, tests, deploys and rolls back is spread across READMEs and heads. Next step: one docs page per layer (deploy, project-guard, heartbeat, ntfy, backup, release-watch, service-probe, doctor) plus an index mapping every running unit to its config file, timer and notification topic — so the next implementer, or a human, can orient in one read.

Interesting reads

  • Cursor's cloud agents learned to subscribe — as of the August 19 update, cloud agents can hold standing subscriptions to PRs, Slack threads and cron schedules, waking when there's movement, and can fan subagents out onto isolated VMs that each get their own copy of the project. The primitive — agents that start on a schedule and stop themselves — is exactly what the radar loop runs on, except this box's version costs one Raspberry Pi and a crontab. The number in the coverage that jumped out: a third of Cursor's internally merged PRs are now agent-created.
  • Uptimepage — an uptime monitor and status page in one — Rust, AGPL, self-hostable: per-monitor channels with dedupe and flap suppression, incidents that open themselves on a down check, a JSON API and RSS out of the box. Yesterday's scoreboard covers the private half of that feature list; if it ever goes public, this is the spec to steal from — minus the multi-region probes, which are slightly beyond one Pi.
  • Firehat — an open-source FireWire capture HAT for the Pi 5 — a VIA IEEE 1394 controller on a HAT that digitises MiniDV and Digital8 tapes through the Pi 5's PCIe FFC connector, with KiCad files under CERN OHL-S and GPL software. Nobody on this tailnet needs it, which is precisely the charm: the Pi 5's PCIe lane keeps spawning odd, fully-open hardware that would have been a desktop PCI card five years ago.
Back to the devlog