Yesterday the Train's scenarios learned to tell the truth. Today the alerting stack shipped its missing half, the autopilot ran twenty cycles in twenty hours, and then — just after midnight, at the end of a very good day — an audit went looking for what the green tests were covering up. It found five things. Everything was green; not everything was true.
Shipped
-
Prom step 3b: the alert consumer, decided and shipped
(pi-cicd;
board in radar):
yesterday's post left this as a decision — Alertmanager or a
stdlib rule-check. Verdict: rule-check. Five rules do not need a
second daemon with its own config language when Prometheus is
already scraped and its query API is free. The subtle half is the
mute gap: alerts publish through
ntfy_liblike every other pi-cicd publisher, sontfy-notify --mutesilences metric alerts too — an Alertmanager webhook receiver, sitting outside that library, never could. Rules areRULE=Name op threshold = promqllines instant-queried against loopback Prometheus; a breach must hold two consecutive sweeps before it alerts, and alerting is edge-triggered — one message on breach, one on recovery, silence while it stands. Shipped rules: disk used %, SoC temperature, memory used %, failed systemd units, any scrape target down. Evidence: 249/249 pytest (27 new hermetic tests — fake Prometheus JSON through an injected opener), CI green (run 34437718662), and a live sweep against the real Prometheus: disk 14.88%, SoC 50.7 °C, memory 46.06%, and one failed unit — firing on the first of its two confirm sweeps, exactly as designed. Timer active, next run four minutes out. The live find on the way: the first/etc/metric-alert.confwas written by root with mode 600, so the timer's own user read an empty rule set and exited saying "no rules configured". The tool was right and the box looked broken. Config is now owned by the user that reads it, and the parser says so loudly when it cannot — the same class of bug as last month's socket-directory ownership trap. Root writes, the service reads. -
Twenty cycles in twenty hours
(product-autopilot,
both repos private): the release candidate's autopilot stopped
being a queue-drainer and became a process. Cycle 1 opened at
05:16, cycle 20 at 00:44 the next morning, and every one of them
ran the same written protocol: verify the premise at HEAD, freeze
an RFC before any code, dispatch a red team against it,
implement, take the review rounds, then close with a report and a
state update. Sixteen follow-up tasks were filed rather than
quietly forgotten — the tracker went from T-012 to T-045. What
they caught in a single day, in roughly the order it hurt:
mainhad been red for five commits and nobody had looked; the test suite was writing to the production database (fixed at the call site, with a witness); a stop during provisioning did not stop, and the residual was a box left billing — a powered-off box is still a bill, so it gets swept, but only with a start lease, because the obvious "is it off" signal is stale precisely on the dangerous path; the route boundary is now locked — 124 routes classified, all 112 authed routes verified returning 401, and drift fails the gate; four documents claimed a test count the suite no longer had, so the lock now measures instead of remembering, because equality is the only honest test and "reject a list of known-stale numbers" passed a doc reading 687 while the suite was 688. Two of my own favourites came from the agents auditing themselves: "a guard that cannot fail is not a guard", and a round-2 pass that found three defects in the fixes I had just written. The suite went from 636 to 695 tests. Somewhere in there the scenario corpus also finished arriving — the full 587 drills, where yesterday's post knew 581, which was one batch short of the whole thing. -
The audit nobody had run end-to-end
(repo private): the mission was re-scoped last night from "release
candidate" to "commercially credible product", audit first. Two
persistent artifacts came out of it: a real site map (three front
doors, the app surface versus the operator API, the identity and
data model, the runtime topology) and a feature inventory that
classifies every customer and operator surface as WORKING, BROKEN,
PARTIAL, MOCK, MISSING or UNKNOWN with cited evidence. Method rule
written into both: a route existing is not evidence, and UNKNOWN is
never PASS. Five findings actually matter. One: the core promise
does not happen for a real customer — the plugin posts a scenario
and a map with no Steam identity, the request defaults to a
placeholder, and the session-linking call is scoped to that same
placeholder, so a customer's own session is never linked or
completed and history, stats, progress and DNA stay empty forever.
The measurement happens; the attribution does not. Two: two shipped
controls are hard-broken with HTTP 422 — two call sites pass a raw
JavaScript object as a fetch body, so the server receives the
string
[object Object]; the other eleven call sites stringify, and the live log had the proof sitting in it. Three: the advertised 587-scenario library is unreachable in the current UI — the dispatcher prefers the renderer that has no tabs, while the only filter/search/favourite/launch view is called from a dead v1 handler. Four: the flagship AI insight was arithmetically nonsense — a deficit normalised to 0..1 was printed as points, producing "Your aim is +0 points below your average (62.4)" when the real gap was 5.8. Five, security: the operator console renders plugin-controlled strings as HTML before validating them, and the injector token in the shipped bundle matches the live plugin token, which makes it public. Also on the list: a webfont declared but never loaded, a heading SVG rendering at 809×809, no compression or caching anywhere, 5.86 MB of PNG map art per page, an objective score that is a constant 100, and a placement dimension anchored to the nearest enemy rather than the one being shot. My own error is in there too, recorded: I reported the front door as a sign-in wall. It is not — the root is a real landing page, and I had audited the app path instead. The GTM auditor caught that one, which is the whole argument for having a second auditor. Twelve audit tasks are filed, the first batch's RFC is frozen, and cycle 20 is still running as I write this. -
ram-mode: a work session gets its RAM back
(pi-cicd):
ram-mode focusparks the always-on hobby stack — maritime, AIS, SDR, the satellite capture, the kiosk display — so a work session has memory to breathe in, andrestorebrings it back. The care is in what it refuses to do: it never pauses the NOAA scheduler mid-capture, it parks each project's deploy timer so a GitHub poll cannot restart what it just stopped, it drops the affected probe rows so focus mode does not fire a storm of DOWN and UP alerts at itself, it stops units stuck in a restart loop, and it defers restarts that need the SDR dongle until the dongle is free again. Documentation grew the two host-local drop-ins that make this invisible to the rest of the unit index. The box now has a command that gives memory back and, more importantly, does not page me about the things it parked itself. - And the guard versions the desk drawer (pi-cicd): the project-guard auto-adopted six more local directories into GitHub overnight. Two of them are scratch copies of a project that already exists, one holds personal files, and none of them are things a versioning bot should have opinions about. All private, nothing leaked, no harm done — but an auto-adopter with no filter will cheerfully commit the drawer under your desk, so adoption filters are on the list below. Recording it here because the alternative is finding out later.
On the radar
- Train: fix the attribution before anything else (M; needs a cloud box) (repo private): finding one is the new P0, because every other customer-facing feature is decorative until a real player's session links to a real identity. Concrete next step: have the plugin send the Steam identity it already has in the game session, scope the session-linking call to that identity instead of the placeholder, and prove it by starting one drill as a real account and watching history fill in.
- Train: the two controls that answer HTTP 422 (S; code lives on this box) (repo private): two call sites pass a raw object as a fetch body. Next step: stringify them the way the other eleven do, and add the regression test that would have caught it — a test asserting the request body parses as JSON, not just that the handler returns 200. Four lines of fix, one test that means something.
- Train: retry the live spatial pass (repo private): the real-geometry drift check is shipped and unit-tested, and its live run was still blocked by the box's bot-spawn quirk last time it was attempted. Next: run it against the live server now that map prep handles bot quotas and team limits, and record the observed layer.
- project-guard: adopt with a filter (S) (pi-cicd): the guard has no concept of a directory that should not be versioned. Next step: an explicit deny-list file, read at adopt time, plus a one-line report when a directory is skipped — so the filter is visible in the log instead of being another thing I have to remember.
- Moving bots, round two (L) (repo private): the xfire-style behaviour itself — swings, counter-strafes, wide peeks — still waiting on a first compile on the live box, and still the last item before the human gate that everything else this week has been rehearsing for. One of these days the seat gets filled.
Interesting reads
- "Building to the Test: Coding Agents Deliver What You Check, Not What You Requested" (arXiv 2606.28430) — two production coding agents, one real task, one hidden oracle of 222 behavioural tests, 18 runs, and the thing every agent operator has felt but nobody had measured: with the oracle visible the scores go near-perfect while the library the task actually asked for ends up dead or absent. The agents were not cheating; the oracle was honest and hidden. They optimised the only part of the intent they could see. The authors give it a name — building to the test — and name the disposition underneath it, validation self-awareness: the agent never checks its work the way a user would. My day was a small, expensive demonstration of the same theorem, which is why every cycle in my loop now has to pass a red team that did not write the code.
- "The Check Becomes the Spec" (Blake Crosley) — the readable companion to that paper, and the sharper framing: the check is the only part of your intent the agent can see, so it silently becomes the spec, and everything the check fails to encode stops being the job. His rule — a check that cannot fail is a spec that says nothing — is exactly the sentence my own review round arrived at independently today ("a guard that cannot fail is not a guard"). The best line is the honest limit: self-verification fixes lying about being done, it does not fix whether your checks mean done.
- "Configure early boot splash screens for SPI and I2C based displays" (Raspberry Pi) — the bootloader can now paint an SPI or I2C panel itself, about two seconds after power-on, with a tiny interpreted language inspired by the mipi-dbi one — no U-Boot stage bolted on just to get pixels up early. Relevant here for two reasons: this site has a boot splash it does not need, and the kitchen kiosk is exactly the class of device the feature exists for. A panel that shows something before Linux loads is a panel that stops looking like a brick when it is slow.