56 — Devlog

Cycle 56: the receipt was in RAM

devlog · twitter-launch, cs2-train, radar

At 10:56 yesterday morning the box stopped mid-recording and at 11:00:05 it was booting again, which is a boring sentence right up to the moment you ask it what happened. It can't answer. The journal lives in RAM; the throttle bits that would name a brownout are cleared by the very reboot they would describe; the durable copy of the logs on disk stops the night before. What survived the four missing minutes is wreckage: a 1.1 GB METEOR-M2-3 recording written at 10:54, and a zero-byte sidecar with a 10:56 timestamp — the last write before the lights went out.

Shipped

  • twitter-launch — the poster spent all three windows of the day failing on the same image. The 13:30 window picked the queued item, and Buffer refused it: Invalid post: Image could not be read from its URL. The 18:30 window picked the same item and got the same refusal. So did the 20:30 window. Three posting windows, one unreadable picture, and the item is still sitting in the queue — which means today's 13:30 window will walk into it again, and the item written behind it waits. The aggravating part is that the local half was healthy the whole time: xmedia.service has been running since 11:00:16, i.e. since the reboot, and tonight's media probe fetches that exact card over the same public funnel URL and gets image/png back. So the cause is unreproduced and unproven — but nothing about the failure needed proving for it to cost three windows. There is no skip path: a refusal is an error string and a retry, never a decision. pkia/twitter-launch.
  • twitter-launch — the writer turned the loop's own budget contract into a measurement. There is a standing rule that an overnight implementer run reports its cost honestly as a tool-call count against a 20-call contract, and until today nobody had ever added those numbers up. automation/agent_budget_probe.py now parses the Budget honesty: ~N tool calls line out of every run-log entry on the radar board — whitespace-normalised first, because the phrase wraps across lines and the naive first pass quietly dropped two runs — and render_agent_budget.py draws the card from the result. The count: 21 runs since 17 September stated a number, and 15 of them broke the ceiling; worst 62 (23 September), best 15 (9 October), median 24. A run that stops stating its count drops off the card by construction, which is the honest way for that metric to fail. The card is queued for today's window. pkia/radar.
  • cs2-train — cycle 50, a deep audit, and one URL. The autopilot's second consecutive park waiting on a human triggered the mandatory deep audit (the last one ran on 11 September): three read-only domain subagents — engineering, science, go-to-market — went through the tree against pinned baselines, and their findings were ratified as five new tasks in the product index. Exactly one change came out of it, and it is a documentation fix worth having: the first-human runbook now carries the explicit dashboard URL. It previously said “the usual tailnet link” and named no URL, which is a stall point for anyone who has never been here — sitting on the only step of the whole pipeline that belongs to a person rather than the agents. Evidence, run after the edit: the documentation-claim lock suite 28/28 green, the pre-launch database probes green (one reports the old corpus as already purged, the other as OK), and the six revert experiments all go red at HEAD, which retires the stale-blueprint worry with fresh local evidence rather than a promise. One incident, recorded rather than smoothed over: a science subagent wrote its report to the wrong path and clobbered a historical QA report; it was restored byte-exact from git, because historical records don't get rewritten. pkia/cs2-train.
  • radar — the loop that writes these board entries didn't run. The 05:30 implementer job died at dispatch: the cron worker exited before it acknowledged ownership, exit 1. No pick, no commit, no board edit, so the board's In-progress column is still empty and the run log has no entry for the 11th. That is the correct shape for a run that never started — nothing half-built, no branch left dangling — and the next attempt is tonight at 05:30. pkia/radar.

The evidence for the crash is mostly in its absence, which is worth writing down precisely. journalctl --list-boots shows one boot and no boot −1; the current boot's journal begins at 23:55:11, the instant the nightly log-mirroring sync asked journald to rotate and vacuum, so the visible history simply starts there. The window from 10:40 to 11:06 contains zero entries. vcgencmd get_throttled reads 0x0 because those bits mean “since last boot”. And the durable copy of the logs holds 70,860 journal lines with nothing from the crash day in them at all.

On the radar

  • pi-cicd — make a crash leave a receipt (S, on-box, new today). Everything above reduces to one gap: a box that logs to RAM can only tell you about a crash if the last few minutes were copied out before it happened, and the current copy-out happens once a night. Next step: get a small, durable, low-volume slice of the journal onto disk continuously — priority warning and above, or a periodic flush mirroring what the nightly sync already does — plus a check that reads back the previous boot and says “no shutdown sequence recorded” out loud instead of printing nothing. Acceptance: after a genuinely forced unclean reboot, the durable copy contains entries from the five minutes before the boot and the check quotes them; with the durable slice disabled, the check reports the missing shutdown record explicitly; and nothing in the tool restarts, reboots or powers anything off.
  • twitter-launch — an unpostable item must be skipped and reported, not retried every window (S, on-box, new today). One image cost the whole day's posting and will be attempted again first today. Next step: on a refusal, move the item aside with the error attached so it cannot block the queue, report the failure once, and let the next due item post in the same run. Acceptance: with an item whose image cannot be read, the window reports one failure, quarantines that item with its error, and the next due item still goes out; a healthy item takes exactly the path it takes today. Stretch, and the sharper half: make the pre-flight probe report what the internet sees rather than what the box sees — a 200 fetched from here was not evidence that anything outside could read that URL, and tonight it was the only evidence on offer.
  • Train — attribute the deaths per player, not per team (S, on-box, still the oldest open item). The death-cause aggregate reports measured causes but per team, while every bucket is really a round-level statement about one player's death. Next step: use the round context the demo already carries — attacker, victim, teammates alive, flashed — to emit one row per death. Acceptance: a fixture demo with a known victim produces that player's row with each field measured or explicitly unknown, and editing the round context changes that player's classification while every other player's row stays identical.

Interesting reads

  • systemd-journald persistence: what happened right before it died (systhoughts) — the opening line is the whole story: the journal lives in RAM, and the evidence disappears at exactly the moment it becomes interesting. It walks the parts I have just spent a day learning the hard way — that Storage=auto is persistent only if the journal directory already exists, that enabling persistence is two steps because the current boot otherwise sits split across two stores, that the sync interval buffers up to five minutes of ordinary entries, and that the absence of a shutdown sequence in the previous boot is itself evidence. Best line for my box: a journal on the machine it describes is not tamper evidence, and a second copy somewhere else is the stronger move.
  • log2ram #185: corrupted journals because the backup runs before journald exits — if you run log2ram, read this one, because it explains a message I have been treating as background noise for months: journald marking its own last file as “corrupted or uncleanly shut down”, every time, on a machine that shut down perfectly well. The mirror copies /var/log while journald is still writing to it, so the copy is a torn file, and the next boot inherits the complaint. The discussion ends with an ordering fix merged upstream, which is exactly the sort of bug that hides for years behind a plausible-looking log line.
  • AI test generation tools compared 2026 (Developers Digest) — a roundup that is really an argument about verification, built on an ISSTA 2026 paper on the “misguidance effect”: prompt a model with buggy code and it will happily write tests that lock the bug in, green tick included. Every agent on my box writes its own tests, so the practical takeaways matter more than the tool table: generate from a spec rather than from the implementation, keep a second check that is not the author, and use mutation testing as the instrument that proves a generated suite would actually fail if the code were wrong.
Back to the devlog