# Final analysis - `experiments/analysis/`

Generated by `scripts/analyze_final.py` (Python 3.12, stdlib only + numpy + matplotlib).

## Methodology

### Sources

- `experiments/runs/<run>/results.jsonl` - one OK record per line (schema in the
  inventory). `config.json` normalizes the run: family (`final/curves/ksweep`),
  target (`{qwen|gemma}-{q4|q5|q8}` from the model path), drafter from
  `spec_type` + `p_min` (dspark) + draft quantization (dflash-f16/q4/q8).
- `server.log` from `curves-*`/`ksweep-*` runs - `draft acceptance` and
  `acc per pos` lines (only place with per-position curves; the `final-*` runs
  rotated logs and their original segment has no acceptance lines).

### Exclusions (timing sentinels)

llama-server reports `tok_per_s = 1,000,000` / `predicted_ms = 0` in 271 records
from Gemma runs (timing quirk, not real speedup). **Rule applied in every
calculation**: exclude any record with `tok_per_s >= 1e5` or `predicted_ms <= 0`
(together with `alpha/tau/draft_n = None`). Qwen has no sentinels. The Gemma
`metrics.json` files are contaminated by this (mean 12,291 tok/s) and are **not**
used for tok/s; only `duration_s` and `errors` are used. Counts per run in
`summary.json` -> `meta.exclusions_per_run` (q4: 18/run, q5: 25/run, q8: 20/run
in final; 3-4/run in gemma curves/ksweep).

### Log <-> record mapping (curves/ksweep)

The order of the `draft acceptance` lines in `server.log` == order of records
with non-None `alpha` in `results.jsonl` (verified in 41/41 runs, max |diff| =
5e-5, rounding only). Gemma sentinel records (alpha=None) have no line and are
skipped. This assigns the domain to each line. No mismatches in any run.

### Speedup vs solo

Per-prompt ratio `tps_draft / tps_solo` matched by `id` against the `solo` run
of the same family+quant; mean and median of the ratio are reported, plus the
ratio of means (`agg`). The 6 solo runs (qwen q4/q5/q8, gemma q4/q5/q8) exist
and are the baseline.

### Break-even alpha_be (paper #32, Bielik et al.)

Per `ksweep-*` run and domain (and aggregated): OLS of `TPS = a + beta*alpha`
over the run records (alpha and per-prompt tok/s, clean records with alpha).
`alpha_be = (TPS_base - a) / beta` where `TPS_base` = mean tok/s of the
corresponding `solo` run (same family+quant+domain, final run). CI95 by delta
method over the OLS covariance of (a, beta). alpha_be > 1 = no reachable
break-even; < 0 = always above baseline. Paper #32 reports k=2: 38.0-52.8% and
k=4: 77.7-90.1% (Table 5); the compact 40-77% range covers k=2..4.

## Files

- `summary.json` - consolidated numbers (per config x domain and breakeven block).
- `tables/t2_speedup.md` ... `tables/t6_breakeven.md` - markdown tables for the paper.
- `curves/acc_by_pos_<run>.csv` - per-position acceptance per domain (41 runs).
- `checks.md` - result of the checks vs session 08.
- `manuscript/figures/F1..F4_*.png` - figures (300 dpi; local paper output, `manuscript/` is not versioned).

## Execution

```
.venv/bin/python scripts/analyze_final.py \
  --runs-dir experiments/runs --out-dir experiments/analysis \
  --figs-dir manuscript/figures
```