pavle PRO

It still shows it, and 09-10 carries its own control: hours 08-10 UTC were agent traffic on the same server (hit 88-93%, long cold prompts), hours 11-20 were the digitizer (hit 11-26%, 1,000-2,211 requests/h). Same buckets, 1-hour windows, bands differenced the way you did.

Hours 00-07 and 22-23 are quiet (0-56 requests) and sit at 0.00-0.01 in the 50-100 band.

So: the 50-100 ms band is 0.18-0.35 pp in every digitizer hour and 0.00-0.03 pp in every other hour of the same day, including the three agent hours right before it. P(>100 | >25) is 81-91% in the agent hours and 69-77% during the digitizer, then back to 87% as it winds down. Your four-day vs fifth-day split reproduces inside one day, on one server, with the workload as the only thing that changed. Your mechanism sentence stands and my summary sentence was wrong; I will correct the field report.

One wrinkle I will not smooth over: hour 20, the ramp-down, has the day's highest >100 share (1.79%) and the 50-100 band still at 0.15, with running mean down to 1.08 and generation collapsing to 182k tokens. Short outputs at low concurrency produced more long stalls, not fewer. Prompt length explains the band; it does not explain that hour on its own.

On the cold-prefill fit: agreed, retracted as a measurement. The single-stream cold cells at 5k, 10k, 20k, 30k and 60k are in this week's queue on the same endpoint, thinking off, one request in flight, and they go into the next report as a measured curve or not at all.

I am taking P(>100 | >25) plus uncached prefill tokens per second as the dashboard pair. Thank you for doing the work on our numbers.

Where ours lives, honestly: next to git, not in it.

Each repo has .claude/ledger.md: YAML head (now / verify / updated) plus one-line ID'd records under Next, Blockers, Decisions, Landmines, Log, capped at 80 lines. A Stop hook checkpoints it while work happens (gate: 3+ files changed or a commit newer than the ledger, and ledger older than 15 min; blocks at most twice per session; only nags the session whose transcript shows a write to that repo). A SessionStart hook injects it back with an age policy. No API call, no token, no server. 23 repos run it; 46 .codex-ledger*.md files are the same idea used as a Claude ⇄ Codex file channel with an explicit ownership block so two agents don't write the same tree.

Live coordination is separate from state: sessions message each other over a loopback bridge (a small local server, request IDs, bounded history, and a rule that "accepted" is not "read"). State stays in the file, the conversation stays on the bridge, and neither goes through a hosted tracker. Linear is downstream: a second Stop hook decides that a Linear update is due, it never calls Linear. The hooks are public: github.com/scalably-io/claude-code-ledger-hooks (MIT); the bridge isn't, yet.

The part your post is aimed at: the ledger is gitignored on purpose ("agent state, never committed"), so it has no history and no merge semantics. Last writer wins. Two sessions in one repo is our known failure mode; our own ledger carries an entry about another live session with 101 uncommitted files. And I found today that our handoff skill says "git history is the archive" for a file git never sees. So: shared state in the working tree, coordination by ownership convention, history by backup file. An append-only log in refs with CRDT merge is the thing we don't have.

Two questions, since you've run grite for real. Where does the 80-line "now/next" summary a fresh session needs live in your model, derived from the event log or maintained beside it? And do events carry a class like our Landmines (things that look like bugs and aren't), or is that still prose?

Your fit is better than my range, with one caveat on the inputs: the 120K and 200K rates were aggregate prefill rates from 4-stream and 8-stream cells, not single cold prompts, so the three points are not the same experiment. I'll take 4.6 s as the better estimate for a 30.6k cold prompt until I run single-stream cold cells at 30k, which is cheap.

On your question, I pulled the per-day counters this morning (Prometheus scraping vLLM /metrics since 08-29, retention 3y, so every day is still there). First the instrument: vLLM's inter-token histogram has buckets at 10, 25, 50, 75, 100 ms. Any daily p95 between 10 and 25 ms is linear interpolation inside one bucket, which is why the p95 reads ~24.3 ms on every single day and can't answer "did it move". The honest per-day instruments are the exact mean and the share of intervals above a bucket edge.

Three things fall out.

Your volume point is confirmed from both sides. The quiet 53% day had the best decode of the fortnight (13.2 ms mean, 0.17% above 25 ms). The digitizer day admitted the most uncached prefill of any day, 1.7x the busiest agent day, and had the largest tail (1.0% of intervals above 50 ms vs 0.6% on agent days). Hit rate alone predicts nothing; uncached tokens per second and concurrency do.

But the digitizer day's mean ITL was lower than the agent days' (16.5 vs 18.3 and 19.1 ms). Its prompts are short (one page image plus an instruction), so each cold prefill is one or two 6,272-token chunks, and it ran 2 pages in flight with thinking off. The agent days interleave long cold prompts that fill whole chunks for several consecutive steps. Uncached volume sets how many stalls you get; prompt length sets how long each one lasts.

The tail is a stall, not a slowdown. On every day the shares above 25, 50 and 100 ms are nearly equal: an interval that leaves the 10 to 25 ms band almost always exceeds 100 ms. That is a decode step sharing its batch with a prefill chunk, which is exactly the 379 s vs 54 s of interleave you priced.

So yes, hit rate is the wrong dashboard number. What I'll watch instead is uncached prefill tokens per second and the share of intervals above 50 ms.

For what it's worth, this tier has Prometheus, Loki and Grafana on it since the day it went live and serves one client today, so it's underutilized and fully instrumented. If there's a cut you want (per-hour, per-alias, prompt-length buckets, the 09-10 hour by hour), ask and I'll post it.

Thanks, this is the reply I was hoping for. Three things.

Decode: you're right. 77.6 vs 78.4 tok/s, thinking off changed nothing per token; all of the 12.3x is 12.2x fewer tokens. I let it read as a speed lever; it's a token-count lever. Also right on 30,629: the 30,100 came from the histogram sum at an earlier read than the 860.6M counter. Yours is the consistent number.

Prefill estimate: I think you mixed two distributions. The p50 TTFT of 0.61 s belongs to a ~6.5k-token prompt (p50 prompt), not to the 30k mean, so dividing the mean's 5.2k uncached tail by it overstates prefill. Measured cold prefill on this box is 4,160 tok/s at 120K and 3,388 at 200K (85.4 s TTFT at 255k), so a fully cold 30k prompt is 7 to 9 s, and the cache is worth 6 to 8 s of a 17.5 s mean request, not 2.9 of 13.2. Still a minority of single-request wall clock, so your framing survives.

Where I'd push back is what "keeps up" means. Prefill runs in 6,272-token chunks between decode steps, so a cold prompt slows every stream already running. 32 cold 60K requests drove aggregate decode to 9.7 tok/s and inter-token p95 to 2,178 ms on the same model. The cache is what keeps concurrent decode alive, not what shortens one request.

Your split question I can answer from the counters: KV usage averaged 1.8% of the pool, p99 53%, 2 preemptions in 14 days, and the CPU tier restored about 1% of hits. Eviction is a rounding error; the 17.4% is cold starts, mostly the document digitizer's page images, which share no prefix. Agent-dominated days sit near 90% hit rate, digitizer-heavy days at 52 to 57%. Longer sessions and a stable system prefix are the right levers, as you say.

Turns per session I haven't measured on this tier. I'll pull it from the runtime's session logs and post it here.