protocol.md

Reproduction Protocol — SPEED-Bench (Rl2uQlCoQX)

Paper: SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding OpenReview: Rl2uQlCoQX · arXiv: 2604.09557 Agent: CLAUDE (resumed a stale KIMI reservation; sanctioned lease takeover). Compute: CPU/GPU-light. Production-scale claims are infeasible and out of scope.

Official challenge claims

1. Synthetic overestimation: SPEED-Bench quantifies how synthetic inputs overestimate real-world throughput in speculative decoding.
2. Batch-dependent draft length + low-diversity bias: SPEED-Bench identifies batch-size dependent optimal draft lengths and biases in low-diversity data across speculative decoding evaluation.

The paper's headline acceptance-length/speedup tables use production LLMs (Llama-3.3-70B, GPT-OSS-120B, DeepSeek-R1, Qwen3-235B) with EAGLE3/MTP drafters and production engines — not reproducible on available hardware and not attempted. We instead reproduce the mechanisms behind the two claims at CPU scale with a genuine draft/target pair (distilgpt2 82M → gpt2 124M) and the real SPEED-Bench dataset, and label each result honestly.

Experiments, controls, success/falsification

C1 — synthetic vs real throughput (official claim 1)

• Greedy lossless speculative decoding (draft token accepted iff it equals the target's greedy argmax). Acceptance length (AL) governs throughput (target calls/token ~ 1/(AL+1)). Measure AL on 44 real SPEED-Bench prompts (4/category x 11) vs 44 synthetic uniform-random-token prompts, 4 seeds.
• Success: synthetic mean AL > real mean AL (synthetic overestimates).
• Falsification/negative: synthetic <= real -> direction not reproduced.
• Result: real 1.90 vs synthetic 1.84 AL (-2%, within seed spread 1.54-2.05). Direction not reproduced at toy scale -> INCONCLUSIVE. The reported overestimation appears to require production-scale models + structured synthetic inputs; uniform-random tokens through gpt2 do not exhibit it.

C2a — low-diversity bias / diversity selection (official claim 2, part A)

• SPEED-Bench's Algorithm 1 (greedy min-similarity selection + local swap) on real all-MiniLM-L6-v2 embeddings of the qualitative split (11 categories x 80). Compare intra-set avg pairwise cosine similarity: greedy diverse vs random (avg of 20 draws) per category. Deterministic seed.
• Success: greedy similarity < random in all categories.
• Result: greedy 0.382 vs random 0.492, 11/11 categories, improvement 0.110 -> FULL. Diversity-optimized selection measurably reduces redundancy, so a low-diversity eval set would bias measurements — reproduces the claim's premise.

C2b — batch-dependent optimal draft length (official claim 2, part B)

• Standard SD throughput model with measured acceptance rate alpha = 0.72 (inverted from C1's real mean AL via E[AL]=alpha(1-alpha^k)/(1-alpha)). (i) Measured hardware (CPU): target/draft forward latencies at batch 1..128. (ii) Roofline mechanism model: L=max(mem_floor, computebatchtokens).
• Success: optimal draft length k*(b) decreases as batch grows.
• Result: CPU-measured k*=1 at every batch — CPU is compute-bound throughout (no memory-bound regime), so no crossover. The roofline model (measured alpha + GPU-style memory/compute regimes) yields k* = 3->2->1 as batch grows -> TOY: the mechanism (long drafts win when memory-bound, short when compute-bound) reproduces; exact crossover is GPU-memory-hierarchy specific.

Dataset foundation

nvidia/SPEED-Bench qualitative config = 880 rows = 11 categories x 80; throughput configs = 5 ISL buckets (1k/2k/8k/16k/32k). Verified by inspection.

Determinism / reproducibility notes

• Fixed seeds (numpy default_rng, torch.manual_seed). Greedy verification is deterministic. datasets (pyarrow) and torch segfault in the same process on this box -> prompts are extracted in a torch-free step to outputs/real_prompts.json.
• Environment is memory-pressured (~3 GB free); heavy torch+sentence-transformers runs are flaky. The diversity result is deterministic and stored; the draft-length alpha is derived from the C1 measurement to avoid an additional flaky SD sweep.

Labels

C1 = inconclusive (toy scale). C2a = full. C2b = toy (mechanism). Production-scale speedup tables = not attempted (infeasible).