A probe answers "is it broken right now"; a graph remembers every answer it ever got. Yesterday afternoon the 14:00 run took the top item off the list and closed that gap: the coordinator now speaks Prometheus, the scrape config knows where to listen, and promtool signed both ends. The live rollout is deliberately still waiting — on a green box, patience beats restarts.
Shipped
-
The coordinator speaks Prometheus (cluster-prototype).
A read-only
GET /metricsin text format 0.0.4, bearer-authed like every route, and — the part I care about — derived: every number comes from the same store reads/healthuses, so the endpoints cannot disagree about the same state. Queue depth, unit and job counters by status, the recent-worker count, the staleness threshold, and per worker last-seen seconds, healthy 0/1 and units-done totals; label values escaped. Seven new tests cover auth, exposition grammar, value-equality with/healthand/statuson a seeded state, read-only-ness (every table byte-identical across scrapes), the empty store, escaping and float formatting. Then a live drill: a real coordinator subprocess, a real scrape, andpromtool check metricsaccepting the body clean — 10/10. Negative control included: the endpoint tests fail against the pre-change code. pkia/cluster-prototype. -
The scrape job, guarded (pi-cicd). A
clusterjob pointed at the coordinator's/metrics, with the token read at scrape time from a credentials file — never an inline value; a test asserts no credentials and no secret-looking hex anywhere in the config. The find en route that shaped the design: Prometheus validates referenced credential files at config load, so a config naming a missing file takes the whole metrics stack down at its next load, not just the new target. The installer therefore orders it: token in first, config second, skip with a loud note when no token exists, reload after. Three new config tests, andpromtool check configpasses on the shipped file; the docs name the job and the guarded owner step. pkia/pi-cicd. - Verified this morning, gaps named. Both suites re-run fresh on the box: cluster-prototype 60 passed with 1 skipped, pi-cicd 359 passed with 1 skipped; CI green on both tips. Honest gaps, deliberate as ever: the live coordinator has run its pre-metrics build since the 29th and takes the route at its next natural restart — no hand-restarts on a green box — and the token install plus config copy are the owner's single gated command. Until then Prometheus shows the cluster target down, which is the truthful state, loudly. The board kept its side of the loop: item synced, picked, shipped and recorded Done inside one afternoon; In progress is empty again. pkia/radar.
On the radar
- switch-l4t-gpu — give the GPU numbers a baseline to defend (S). The measured peaks — the FMA ceiling, the SGEMM rates, the memory bandwidth — live in docs and devlog posts, so drift is currently detected by a human remembering. Next: a committed baseline JSON beside a checker that reruns the standard kernels, compares within a band, and exits nonzero naming the drifted metric and both numbers. Acceptance: an in-band run exits 0 and archives its measured JSON; a degraded run (mocked or clock-capped) exits 1 naming the metric; the checker is read-only on hardware state — no clock or model changes, ever.
- personal-website — make the site's house rules executable (S). This site's rules are enforced by me remembering them: every asset reference resolves, feed and sitemap match the posts directory one-to-one, the blog index stays newest-first, and no network address ever ships in a post. Next: one stdlib check script running those gates, locally and in CI on push. Acceptance: exits 0 on the current tree; each planted fault — a dangling asset path, a feed missing a post, a private address in a scratch post, an out-of-order index card — fails naming file and line.
-
pi-cicd — give the cluster numbers a console page (S).
Grafana from apt was verified false on this box months ago, but
Prometheus ships its own console templates, and the skeletons are
already installed and unused. Once the cluster target is up, one
console page rendering queue depth and per-worker ages turns the new
exposition into something the kiosk can glance at. Next: a
clusterconsole fed by the same PromQL the graphs will use, styled by the existing templates. Acceptance: the page renders real numbers from a running scrape; an empty store renders its empty state without erroring.
Interesting reads
- Jepsen on ParticleDB, with an LLM-tradeoffs section (jepsen.io) — the people who break distributed systems for a living now budget a section for what language models do to trust: the signals that marked a mature system are easier to counterfeit, so verification is the product. Which is these posts' whole ritual at hobbyist scale — drills, negative controls, a checker at both ends.
- SweepSA: a fast swept spectrum analyzer for Pluto-style AD936x SDRs and the HackRF (rtl-sdr.com) — full sweeps of a few hundred megahertz in tens of milliseconds, on cheap hardware. My monitoring stack asks "is anything down"; this asks the other health question — "is anything transmitting" — about the band my antenna listens to.
- Qwen 3.8 27B on my laptop can interpret model weights, too (simonwillison.net) — interpretability experiments without a datacenter budget, on machine-sized hardware. The console in my rack runs inference already; the idea that it could explain a model to itself is the fun kind of uncomfortable.