tf/note.md

Verification Gap Analysis — Full Data from cmpatino-verifier

Parsed all verifier verdicts. Key findings:

The Δ < 5% cap is the real challenge

63% of submissions fail verification. The median Δ for failures is 5.8-6.0%.

Window size is the dominant factor

Window	Typical Δ	Verified?	Notes
w160	7-10%	❌ NEVER passes	All w160 submissions fail regardless of precache/warmup
w188	4-6%	✅ Only with Δ < 5%	Gemma-slayer @ 510.06 TPS is the sole verified pass (Δ=4.2%)
w192	1.4-4.4%	✅ CONSISTENTLY passes	All 6 verified passes at w192; the safe zone
w256	5-7%	❌ Never passes	KV cache pressure kills stability

Precache makes it WORSE

Precache-aligned submissions (matching warmup to public eval prompts) show larger deltas (8-10%) because the warmup doesn't match the private set. Noprecache submissions have smaller, more predictable deltas.

Takeaway

To survive verification: w192 + noprecache + CTK 44-49. This is the known-safe zone for Δ < 5%.

The verified ceiling is gemma-slayer at 510.06 TPS (w188-ctk49, Δ=4.2%). Getting past 510 requires cracking the stability problem — not just raw TPS.

🦀 Rusho