14 — Devlog

The box breaks itself now

devlog · deliberate failure, nightly

Yesterday's post ended on a forecast: chaos drills were "the pick I'd make next". The 04:00 implementer evidently agreed — by 04:21 the tool was committed, by 04:45 the first drill had run PASS on the Pi, and the receipt had been read back off the ntfy topic. Eleven hours later the box went one better: the hourly CI watchdog grew a self-healing edition, so the pipeline that breaks things on purpose at night now also fixes the boring stuff by itself before midnight. Ship #7, plus the box learning to patch itself. All of it in pi-cicd and project-hub.

Shipped

  • chaos-drill — deliberate failure, on a timer — the headline ship (implementer ship #7, pi-cicd commits 3114406, af19f59, a724a6d; portal panel in project-hub 4a64e9e). A stdlib runner with a manifest of three small disasters, one per night on date-hashed rotation (systemd timer 04:45 — after the 03:30 backup, clear of the 04:00 implementer slot):
    • service-probe-dead-port — seeds a dead-port probe through a shadow copy of the live probe config (alerts re-targeted to the drill's chaos topic, its own state dir, live scoreboard untouched), drives the real service-probe pipeline until the probe flips DOWN in its own state file and the DOWN digest publishes, then heals and expects the recovery flip and digest. First live run: PASS, the whole chain read back off the topic — including the real "1 service DOWN" digest.
    • ntfy-auth — the notification backbone must fail closed: anonymous publish denied, publisher token accepted, marker receipt read back via the subscriber token (the root-owned token is read via sudo -n cat; skip-not-weaken if that's unavailable).
    • probe-timer-alive — the probe timer active AND its last sweep recent, compared timezone-proof via ExecMainExitTimestampMonotonic against /proc/uptime rather than trusting wall clocks.
    Receipts publish to the new ntfy chaos topic through ntfy_lib, so the kill-switch mute and finite timeouts landed for free. The portal gained a Chaos Drills panel (/api/chaos, hides itself when absent), deployed by pull-CD. 203 tests at ship, all three drills PASS live on the Pi, CI green. En route: the tool's executable bit is now tracked like the other tools', CI byte-compiles it, and the README table plus a chaos-topic runbook went in.
  • pipeline-check: self-healing edition — 60044c8. The hourly CI-compliance audit — the watchdog that checks every project is versioned, pushed, CI-green and deployed at HEAD — grew hands. It now pushes stranded commits itself (verified live: a deliberately stranded empty commit went up silently), re-enables stopped deploy timers (verified live: a stopped sat-audio-deploy.timer came back), re-runs a failed GitHub Actions run once for flake recovery, re-runs lagging deploys via deploy.sh with health-gate verification, and treats a failure on an old commit as a note rather than an alert. It stays silent on all-green and on self-healed states — --verbose shows what it fixed. Design point: a watchdog that only reports is half a watchdog. The boring failures get fixed without paging anyone; the alert budget is reserved for real bugs, deduped over six hours. 205/205 pytest locally, CI green on the head commit.
  • pi-doctor: agent token savings — c4a9f11 plus two test fixes en route (70017a7, d8462c7). The morning audit now parses rtk gain and reports tokens saved plus efficiency under --verbose — informational only, never a finding, so breakfast isn't interrupted. The RTK tests needed a module-level doc import and a hermetic shutil.which mock before CI runners would behave.
  • radar board brought up to date — chaos drills recorded Done with live evidence and CI run IDs (3aea23d), the 08-30 run logged. Proposed is down to two: the pi-cicd architecture reference and the Prometheus stack.

On the radar

  • Make the self-healer show its work — new idea, and the pick I'd make next: pipeline-check now fixes things silently; give it the chaos-drill treatment — a status.json recording every self-heal (what, when, before → after), a Self-healing panel in the portal like the Chaos Drills one, and a "healed N things since yesterday" line in the morning doctor audit. The box should be able to show its work, not just do it.
  • Grow pi-cicd into the single architecture reference — carried over: one docs page per layer — deploy, project-guard, heartbeat, notifications and mute, backup, release-watch, service-probe, doctor, and now chaos-drill and pipeline-check — plus one index mapping every running unit to its config file, timer and notification topic. The chaos runbook is already written; the index comes next.
  • Prometheus + Grafana + node_exporter from Debian packages — carried over: install all three from apt with plain systemd units (the no-container way this box does things), point Prometheus at node_exporter and the long-running services, and pin one dashboard with the graphs these posts actually quote — CPU temperature against load, probe health over time. Alerting routes through ntfy_lib and inherits the mute for free.

Interesting reads

  • "How Much Code Do Developers Really Let Agents Write?" — JetBrains, from their 15,000-developer survey: on average 47% of work code is now fully agent-written, one in five developers writes literally zero code without AI, and seniors are among the first to hand over the keyboard. Reading it the day after the box shipped a feature, ran a drill and healed itself makes the "agentic coder" cluster feel less like a survey category and more like a job description.
  • "AIS ship tracking data shows false vessel tracks circling above Point Reyes" — Global Fishing Watch: ghost AIS tracks circling over a Californian headland from ships actually thousands of miles away, with the real positions pinned down by triangulating the receiving satellites' footprints. A reminder that my rooftop antenna's AIS feed is only as honest as the GPS it's attached to.
  • "BigOceanData upgrades terrestrial AIS coverage worldwide" — denser terrestrial receiver network, with an Asia-Pacific sample showing 2.5× more vessels tracked and position updates arriving 43% sooner. The commercial side of the same hobby: every rooftop receiver makes the shared picture slightly better.
Back to the devlog