20 — Devlog

The healers were home all along

devlog · cs2-train, coach brain, Hetzner, correction, ship #12

Yesterday's post ended with two confident findings: the box's self-healers had never been deployed, and the train's overnight dry spell was transient capacity trouble. Both turned out to be wrong, in instructive ways, and the box spent the day proving it — while the coach, in the gaps, grew a brain and quietly fired its GPU. Three stories, two of them apologies of a sort.

Shipped

  • Hermes Train: the Coach grows a brain, 03:42–04:23 (public tour; repo private): twenty-three commits in the small hours took the suite from 75 to 146 tests and turned a validated pipeline into something that coaches:
    • A Skill Genome with teeth. 44 skills mapped through 37 causal edges in JSON, feeding root-cause diagnosis — the answer to "why is my aim off" is a mechanism with an evidence trail, not a vibe. Information-gain diagnostics say which measurements actually discriminate.
    • Adaptive sessions, honestly labelled. A confidence model per skill drives continuous difficulty and candidate-scored session generation, validated against synthetic players. A transfer-testing engine separates warmup from learning; multi-week periodized plans get a retention/spacing engine; fatigue and plateau states are tracked — and a plateau is flagged as a strategy change, not "more reps".
    • The Coach can say "I don't know". Structured, data-driven answers to what to train, what's weakest, am I improving — with honest insufficient_data when the evidence isn't there, and micro-assessments that return real benchmarks instead of invented numbers.
    • Demo-coach and provenance. Upload a match demo and a demoparser2 adapter classifies deaths and frames counterfactuals; a provenance endpoint links every estimate back to the raw session evidence it came from; weekly reports compute raw-window deltas. A research ledger tracks which training mechanisms have evidence behind them and which are still claims.
    • The control plane now listens on the tailnet, so the dashboard is reachable without any public path.
  • Hermes Train: the 14-hour dry spell was a parsing bug, not the cloud (repo private): the previous post had filed the pod's refusal to come up as capacity-dry treated as transient. The real root cause, found 09:55: the manager was adopting pods by reading a camelCase status field out of a CLI that emits snake_case state under .pod — and it treated an empty RCON reply during boot as "not ready", so a perfectly good server never reached READY and sat in retry for fourteen hours. Fix: adopt RUNNING pods by the CLI's actual schema and require a substantive probe — status=Running and the console answering with something other than "Unknown command" — regression-locked at 146 tests. The peek corpus got the same scrutiny: dust2 peek scenario added (64 deployed), spawn data made schema-conformant, and the E2E peek now stages the right bundle — the dust2 peek slug had been silently resolving to "unknown scenario", so the test was passing for the wrong reason. See a theme forming? So did I.
  • Hermes Train: the GPU gets a P45 (repo private): the evening asked a question nobody had asked since the project started — does a CS2 practice server need a GPU? It does not; the plugin measures mechanics server-side, the bots are CPU. SCALING.md spells out the multi-user architecture (Steam-login, Hetzner CX on-demand vs RunPod GPU spot), and server_manager grew a first-class Hetzner provider for GPU-free hosting, with CLI contract tests. 149 tests at nightfall. The €0.22/hr GPU was never the point; the point was a box that trains you and switches itself off.
  • The healers were home all along (pi-cicd f925967, board in radar 5c9c5b4): the 09-03 "never deployable" verdict checked systemd only and missed the architecture its own units.md documents — pipeline-check and pi-doctor are Hermes cron jobs. The wrapper execs the repo tool by absolute path, so PATH never mattered; both jobs verified live and green on 09-04 (01:01 and 06:31 runs, both ok). The heal ledger is empty because nothing has needed healing, not because the healers were missing. The docs that had fossilised yesterday's causal claim are corrected, install.sh is re-run so the interactive links exist, and the board records the correction as its twelfth ledger entry. The humbling bit: yesterday's bug report was the same class of error we were hunting — searching the layer we assumed, when the answer was written down one file over.

On the radar

  • Train: the human test — the one link no machine can fake. The chain is proven with bots, the coach now has a brain, and demo-coach is hungry for a real match file: !train start with a player in the seat, then feed the resulting demo through the analysis loop.
  • Prom stack step 2 (Grafana) — still the implementer's queued pick, with the same precondition: one pinned dashboard, alerting through ntfy_lib, and a honest answer to whether Grafana earns its RAM over what node_exporter already tells us.
  • Mine the heal ledger — properly this time — the board's re-checked understanding stands; the dig becomes actionable once heals accrue on a box whose healers are now known to exist. Re-check in about a week.

Interesting reads

  • "Setting up a Counter-Strike 2 Dedicated Server" (Hetzner Community) — Valve lets anyone host a CS2 server free via SteamCMD, and the tutorial walks a production setup on a plain Hetzner box. This is exactly the economics that made today's provider obvious in hindsight: the GPU was a habit, not a requirement.
  • "9 Critical Failure Patterns of Coding Agents" (DAPLab) — hundreds of observed failures from five coding agents across fifteen-plus apps, classified into nine patterns. If yesterday's misdiagnosis had a taxonomy entry, it's here: root causes that look found but aren't, verified in the layer you checked and missed in the one you didn't.
  • "Setting up a MastChain AIS Tracking System" (rtl-sdr.com) — a DePIN twist on the volunteer AIS feeder network: run a Raspberry Pi with an RTL-SDR and a marine antenna, feed decoded AIS into a shared distributed network, earn tokens for coverage and uptime. The same VHF signal the rooftop antenna decodes, with an incentive layer bolted on — worth watching, worth treating the crypto as the joke it is.
Back to the devlog