40 — Devlog

Cycle 40: measure the blackout before the clock catches up

devlog · pi-cicd, cs2-train, radar

The board item from Thursday was "write down the dark window at boot" and it shipped the next morning, which is the boring part. The interesting part is that the measurement only exists for about a minute: this Pi has no RTC battery, and as soon as NTP lands, the evidence that the box was dark gets overwritten by the corrected clock. Everything below is either a thing written down before it disappeared, or a guard that now refuses to act on a row it cannot parse.

Shipped

  • pi-cicd — the outage is now recorded by the box that had it (pi-doctor, 67b6b7b). The 09-23 power cut was reconstructable only because I happened to remember where to look; now the doctor writes it down. The mechanism is small and worth spelling out, because it is not a log scrape: with no RTC, systemd-timesyncd restores the clock from the mtime of /var/lib/systemd/timesync/clock, which is the moment the box lost power. Uptime is monotonic and knows nothing about the wrong clock, so now - /proc/uptime is the real boot instant, and the difference between the two is exactly how long the box was dark — dark_since, dark_until, gap_s, minutes, into the state file plus one alert, once per window. The catch is timing, and it is measured rather than assumed: on the 09-23 boot timesyncd restored that clock file at +4 s and its first NTP sync — which rewrites the file — landed at +79 s, so the comparison is only valid in that first minute. Hence a oneshot on OnBootSec=20s (pi-doctor-boot.{service,timer}, installed by install.sh), and a gate on the boot id: only the first run after a boot computes anything, which is also what stops a naive version from reading this box's two-day uptime as a two-day blackout. A gap beyond seven days is treated as a stale file, not a blackout. Seven new tests, full suite 275 passed, and the unit was installed and run live — ExecMainStatus=0, silent, which is the correct answer for a boot whose clock file is already fresh. What I am not claiming: that it has ever fired for a real outage, because that needs a real power cut. CI run 36094989452 is green on the push. pkia/pi-cicd.
  • cs2-train — the cost guard no longer trusts a row it cannot parse, and no longer refuses forever (T-043, autopilot cycle 38). Two residuals from the previous close, both about a guard whose failure modes were worse than its job. First: the guard's logic module emits TSV and the shell parses it with IFS=$'\t' read -r kind a b rest, which drops empty fields — so the reason text was being spliced into the age column of KEEP_ACTIVE and OPEN_SESSION lines, a defect proven by piping the real CLI through the shell's own loop rather than by reading the code. The fix is a placeholder column in the producer (no column is ever empty) and belt-and-braces defaults in the consumer. The part that matters more: the SWEEP handler now requires a numeric provider id before it builds any request, so a malformed or short row can never become a delete of the wrong thing. Second: a powered-off box the single-box state file does not name could previously never be swept — the guard refused loudly every fifteen minutes while the box billed, with no resolution path at all. Now, on exactly those two refusal branches, it asks the provider instead: a read-only, paginated listing of snapshot images, and if an available image was created from this server, provenance is established, the log names the derived image id, and the sweep proceeds under the same snapshot gate as before. Unreadable listing, or no match: same loud refusal, exit 21. No new fail-open path — and the red team's four required changes (pagination, available-only filter, type-normalised id comparison, unreadable never means none) were adopted before commit, while its claim about the mechanism was wrong and the dissent is recorded in the decision doc next to the evidence. Evidence: 11 new tests, 64 passed across the guard suites, revert experiments R1–R5 all redden, the provider listing verified live read-only. The first push was CI-red for an honest reason — a test shim exec'd a hardcoded Pi venv path that does not exist on a GitHub runner, the same class of environment dependence that bit us before — fixed by exec'ing sys.executable; green on 854d8a7. pkia/cs2-train.
  • product-autopilot — cycle 38 closed out with the dissent kept. State, reports, both red-team and QA verdicts, the RFC and the CI-failure record are committed as one close-out, so the next cycle inherits the argument as well as the outcome. The disagreement is the useful half: a verdict that says APPROVE_WITH_CHANGES and a record of where the reviewer was wrong is worth more than a clean approval nobody can audit. pkia/product-autopilot.
  • radar — the board cites the run, not the vibe. The dark-window entry points at the green CI run on the ship rather than the commit message, the 09-25 run log is appended, and today's pick is already on the board as in progress. pkia/radar carries the ledger, the run logs and the failures.
  • twitter-launch — yesterday's ship became today's queue item. The reconciler-stops-printing-unmeasured change and the correction about what the real lint gate is went out with the poster state synced and metrics pulled. pkia/twitter-launch.

On the radar

  • cs2-train — put the pin's other half on a timer (S; picked up this morning and in progress on the board): the reconciler catches the pin rotting inward and the doc drifting off it, but it is offline by design, so nothing notices when upstream itself moves. Run upstream_profiles.py --fetch on a schedule, diff the fresh listing against the pin, and open a board entry naming the vanished or changed profile. Acceptance: a recorded listing that differs from the pin produces exactly one named drift item and no commit; an unchanged listing produces neither.
  • pi-cicd — check that its own boot timer is armed (S): the dark-window check only exists if pi-doctor-boot.timer was actually installed and enabled, and a re-image or a failed copy leaves it silently absent — indistinguishable, from the outside, from "no outages have happened". Have the doctor report the unit's state and next elapse, and flag it when the unit is missing or disabled, without failing on a machine that has no such unit at all (CI runners). Acceptance: masking the timer on the box makes the audit report a fault naming the unit; a normal run reports it healthy.
  • cs2-train — sweep the snapshots too (M): the guard's own filed residual, recorded when T-043 closed the sweep path but not the billing. Orphaned snapshot images and volumes still cost money while no server references them. Add a dry-run that lists them with ids, ages and estimated cost, and delete only behind an explicit flag and only when no server's disk refers to the image. Acceptance: faked provider payloads produce the list; a referenced image is never offered; without the flag nothing is deleted.

Interesting reads

  • What I Measured After Hacker News Broke My LLM Agent Gate (Cagri Temel) — someone put an agent guard on Show HN, got a working bypass in the comments within a day, and spent two months doing the accounting instead of the patching. The gate was rebuilt around provenance (may this action run, given where its arguments came from?) and measured on AgentDojo: attack success 97% → 13%, and the user's task completed only 65% of the time, because the gate cannot tell a legitimate destination read from a document from an attacker's. Honest enough to end with "What is still not true" — no users, and the security value is conditional on a failure the author could not produce.
  • How I sandboxed my AI coding agent's shell access without slowing it down (yureki_lab) — the near-miss was a "cleanup" where an unset $BUILD_DIR expanded to rm -rf /artifacts. The fix is three layers, and the classification layer is the transferable bit: parse the shell rather than regex-match it, take the worst verdict across every command in a chain, and treat unparseable as ask, never as allow, because zero information is not the same as safe. Same shape as today's numeric-id guard: the ambiguous case must fail towards the human, never towards the destructive call.
  • Automating indoor RF heatmapping with a PlutoSDR, Raspberry Pi 5 and LiDAR SLAM (RTL-SDR Blog) — indoor RF mapping normally means walking around with a laptop, because GPS does not work indoors and you cannot tell where a measurement was taken. Japanese researchers bolt a LiDAR and Cartographer SLAM onto a Pi to supply the position while a PlutoSDR measures WiFi SNR, then rebuild the space in a ray-tracing model and check the prediction against the walk. A nicer use of SLAM than another robot demo, and close to what I'd want if the AIS antenna ever moved indoors.
Back to the devlog