Final analysis - `experiments/analysis/`

Generated by scripts/analyze_final.py (Python 3.12, stdlib only + numpy + matplotlib).

Methodology

Sources

experiments/runs/<run>/results.jsonl - one OK record per line (schema in the inventory). config.json normalizes the run: family (final/curves/ksweep), target ({qwen|gemma}-{q4|q5|q8} from the model path), drafter from spec_type + p_min (dspark) + draft quantization (dflash-f16/q4/q8).
server.log from curves-*/ksweep-* runs - draft acceptance and acc per pos lines (only place with per-position curves; the final-* runs rotated logs and their original segment has no acceptance lines).

Exclusions (timing sentinels)

llama-server reports tok_per_s = 1,000,000 / predicted_ms = 0 in 271 records from Gemma runs (timing quirk, not real speedup). Rule applied in every calculation: exclude any record with tok_per_s >= 1e5 or predicted_ms <= 0 (together with alpha/tau/draft_n = None). Qwen has no sentinels. The Gemma metrics.json files are contaminated by this (mean 12,291 tok/s) and are not used for tok/s; only duration_s and errors are used. Counts per run in summary.json -> meta.exclusions_per_run (q4: 18/run, q5: 25/run, q8: 20/run in final; 3-4/run in gemma curves/ksweep).

Log <-> record mapping (curves/ksweep)

The order of the draft acceptance lines in server.log == order of records with non-None alpha in results.jsonl (verified in 41/41 runs, max |diff| = 5e-5, rounding only). Gemma sentinel records (alpha=None) have no line and are skipped. This assigns the domain to each line. No mismatches in any run.

Speedup vs solo

Per-prompt ratio tps_draft / tps_solo matched by id against the solo run of the same family+quant; mean and median of the ratio are reported, plus the ratio of means (agg). The 6 solo runs (qwen q4/q5/q8, gemma q4/q5/q8) exist and are the baseline.

Break-even alpha_be (paper #32, Bielik et al.)

Per ksweep-* run and domain (and aggregated): OLS of TPS = a + beta*alpha over the run records (alpha and per-prompt tok/s, clean records with alpha). alpha_be = (TPS_base - a) / beta where TPS_base = mean tok/s of the corresponding solo run (same family+quant+domain, final run). CI95 by delta method over the OLS covariance of (a, beta). alpha_be > 1 = no reachable break-even; < 0 = always above baseline. Paper #32 reports k=2: 38.0-52.8% and k=4: 77.7-90.1% (Table 5); the compact 40-77% range covers k=2..4.

Files

summary.json - consolidated numbers (per config x domain and breakeven block).
tables/t2_speedup.md ... tables/t6_breakeven.md - markdown tables for the paper.
curves/acc_by_pos_<run>.csv - per-position acceptance per domain (41 runs).
checks.md - result of the checks vs session 08.
manuscript/figures/F1..F4_*.png - figures (300 dpi; local paper output, manuscript/ is not versioned).

Execution

.venv/bin/python scripts/analyze_final.py \
  --runs-dir experiments/runs --out-dir experiments/analysis \
  --figs-dir manuscript/figures

Final analysis - experiments/analysis/