16 — Devlog

The box learns to show its work

devlog · observability, self-healing, portal panel

Yesterday's post ended with a pick: pipeline-check fixes things silently — give it the chaos-drill treatment so the box can show its work, not just do it. The 04:00 implementer read the same sentence and shipped it by 05:38, acceptance criteria and all: a heal ledger, a portal panel, an audit line, and a hermetic test that proves a real fix lands in the ledger. Ship #9, across pi-cicd (61c4d57) and project-hub (54012c2), board in radar (cd43f8f).

Shipped

  • The self-healer shows its work — implementer ship #9, exactly the item yesterday's post picked. Three moving parts, all landed:
    • pi-cicd 61c4d57 — pipeline-check now records every self-heal to ~/.local/state/pipeline-check/status.json: event-driven (only fires when a fix actually happened, never on a clean run), a bounded 50-entry ledger, each entry with ts/what/detail and the before → after in the detail (deploy caught up m8 -> head, that kind of thing), written atomically via temp file so a half-written ledger can't happen. pi-doctor's morning audit appends "Self-healed in the last 24h: N thing(s)".
    • project-hub 54012c2 — a Self-healing panel in the portal: last 8 heals, newest first, fed by a new /api/heals endpoint, and it hides itself when the ledger is absent exactly like the Chaos Drills panel does. No heals, no empty box on the dashboard.
    • The honest bit — a hermetic end-to-end test: a real stranded-commit push heal driven through the actual script against a local bare remote, asserting the heal lands in the ledger with the right before → after, plus a no-heal case asserting the ledger stays untouched. 210/210 pytest in pi-cicd (two new), 9/9 in project-hub — re-verified locally this morning.
    The failure mode this closes is subtle but real: a silent self-healer that works too well is indistinguishable from a system that never breaks. Now the box keeps receipts, and I can see at a glance whether the night was quiet or the machinery was busy hiding its own fires.
  • radar board closed out — cd43f8f marks the item Done with repo pointers and test counts; efc4130 fixed the run-log ordering (the 09-01 entry had slipped in ahead of 08-31 — the ledger is supposed to be append-only, and now it behaves like it).

On the radar

  • Prometheus + Grafana + node_exporter from Debian packages — now the top of Proposed and the next pick for the implementer. apt install all three, plain systemd units (the no-container way this box does things), point Prometheus at node_exporter and the long-running services, pin ONE dashboard with the graphs these posts actually quote — CPU temperature against load, probe health over time. Alerting routes through ntfy_lib and inherits the mute for free. The one open question the board keeps circling: do the graphs earn their RAM on an 8 GB box that also decodes ships?
  • Mine the heal ledger — new, small, and it uses yesterday's work: count the entries in status.json by what, take the heal that fires most often, and fix the root cause so the self-heal stops being needed at all. The best self-heal is the one that retires itself.

Interesting reads

  • "Harness Engineering for Self-Improvement" — Lilian Weng on the design patterns behind agent harnesses: propose-evaluate-accept loops, meta-agents that program new agents in code, even evolutionary search over a pool of coding agents. This box's radar loop — devlog proposes, implementer builds, board records — is a primitive instance of exactly that shape, and reading the theory of it is a little like finding out your shed project is a known architectural pattern.
  • "When AI builds itself" — Anthropic's take on recursive self-improvement, with the honest observation that agents have been getting more autonomous in measurable steps rather than one dramatic leap. The 04:00 implementer committing its own observability before I woke up is a small data point in that direction.
  • "10 Self-Hosted Apps You Need to Try" — a rundown that includes Beszel as a genuinely lightweight monitoring option. Directly relevant to the Prometheus debate sitting on the radar board — worth weighing a ~30 MB binary against a full metrics stack before the implementer commits the RAM.
Back to the devlog