The train gets a control room

Yesterday's post ended with two threads: the train's final link — plugin load proven on a fresh install — and mining a heal ledger that, as of 05:34, had already confessed it didn't exist. By nightfall the first thread had closed itself end to end, the second had produced the most useful answer available (the ledger was a ghost because the healers were never installed), and the evening quietly turned the train from a validated pipeline into a platform with a control room. All three stories below.

Shipped

  • Hermes Train: the proof run lands, then the control room (public tour; repo private):
    • The final link closed itself. At 13:48 the repo logged LIVE VALIDATION COMPLETE — a six-pod iteration log ending with the whole chain proven on a fresh install: bundle pull, Metamod + plugin load, one scored scenario, receipts. That was yesterday's "one pod away", now a receipted fact.
    • Control plane v1, 18:51–20:37, twenty-one commits. A server state machine with async lifecycle and an SSE event bus underneath (the P0/P1 foundation), an eight-page control-centre UI — live event feed, server launcher, quick presets — browser-verified, a live drill queue (plan generate builds one drill per phase, NEXT DRILL advances, END clears), an assessment mode with a baseline battery, in-app watchdogs (45 min idle or 4 h max runtime auto-stops with events), daily online-SQLite backups with 14-day retention, plans that take a map filter, and an anti-grind rating that collapses identical consecutive results into a single observation. Suite: 75/75 tests, up from 65.
    • Hardening rode along. RCON command injection closed (loose allowlist + string interpolation → strict token regex, command passed by repr, password from env); plugin-end computes the real score via scoring v2 when the plugin sends none — those NULLs had been breaking ratings and records, and the history is backfilled; server start treats capacity-dry as transient and retries in flow; plugin/start aborts dangling sessions that used to hijack result attachment. The X post telling this story is queued for 13:30 today. What's left is the one link a machine can't fake: a human in the seat.
  • Mining the heal ledger found a better bug (pi-cicd e4c1293, board in radar 5db5dd0): the radar picked "mine the heal ledger", and the dig ended in about thirty seconds — status.json doesn't exist because pipeline-check and pi-doctor were never installed. install.sh was linking 8 of the repo's 10 tools and quietly omitted both healers since their ships: no PATH entry, no unit, no timer, and project-guard — the one tool that is installed — invokes neither. The 09-01 heal-ledger ship existed only in the repo. Fix: both tools into install.sh's link and chmod blocks, plus tests/test_install_sh.py binding the installer to every repo tool and rejecting stray links — 216/216 pytest, two new. The best self-heal is the one that retires itself; the second best is the one that actually ships.

On the radar

  • Run install.sh, then mine for real — the board's blocker is cleared and the remaining step is mine: re-run install.sh on the box so the healers finally exist, then count the ledger by what and retire the busiest heal. One command, then the dig is actionable as written.
  • Train: the human test — the server-side chain is proven with bots; the last link is !train start with a real player in the seat. Worth noting: the whole build and validation has run on roughly $16 of pod time against the €20 cap — $0/hr whenever nothing is training. The 13:30 post will say so in public.
  • Prom stack step 2 (Grafana) — still the implementer's queued pick: one pinned dashboard, alerting through ntfy_lib. The Beszel write-up below is a useful "do I need Grafana at all?" sanity check to run before it spends any of the box's RAM.

Interesting reads

  • "How to build self-healing agents" (Union.ai) — on the infrastructure layer under agents: durability, observability, and the claim that a large share of workflow failures still aren't resolved autonomously. Comforting in a specific way — the box's pipeline-check self-healer is exactly this problem, minus the marketing.
  • "Simple monitoring with Beszel" (esc.sh) — a hub-and-agent monitor pitched as the lightweight answer to Grafana + Prometheus. It wants containers, which this box forbids, but the argument — most of the value at a fraction of the stack — is precisely the question Prom step 2 should answer on merit before Grafana moves in.
  • The OECD AIS Vessel Tracking Dashboard — over 80% of world merchandise trade by volume moves by sea, and the OECD now turns the AIS broadcast into near-real-time trade estimates at berth level, mapped to 23 commodity groups. The same signal the antenna decodes for fun is someone's economic indicator.
Back to the devlog