Upload README.md with huggingface_hub

README.md

@@ -70,22 +70,31 @@ Sanity checks pass: same input sequences for `ppl_full` vs `ppl_ann`,
 intact causal mask in retrieval, single-softmax renormalization with no
 wrapper leakage between iterations.
 
-### Deployment knobs (L = 4096)
+### Compute / quality knobs (FLOP-counted)
 
-| Use case | K | PPL gap | Attention compute reduction |
-|---|---|---|---|
-| Quality-improving | 256 | −0.79% | ~16× |
-| Quality-improving | 512 | −2.89% | ~8× |
-| Quality-preserving | 128 | +0.82% | ~32× |
-| Aggressive | 64 | +2.42% | ~64× |
-| Speed-only | 32 | +4.51% | ~128× |
-
-Note: the K-sweep recall numbers (24–41%) are not directly comparable to the
-in-training `evaluate()` recall (50.9% at K=128). Same checkpoint, same K,
-same metric code path — the discrepancy comes from sampling different
-sequences out of the WikiText streaming split (different `num_batches` /
-worker dispatch). The PPL gap is independent of which subset is sampled
-and is the load-bearing deployment metric.
+`L = 4096`. Compute reduction is the attention scoring step, ≈ `L / K`.
+These are FLOP estimates, not measured wall-clock — the FAISS path in this
+repo is a research prototype that does CPU index builds and GPU↔CPU
+transfers, so it is not the right thing to time.
+
+| K | PPL gap | Attention scoring reduction |
+|---|---|---|
+| 512 | −2.89% | ~8× |
+| 256 | −0.79% | ~16× |
+| 128 | +0.82% | ~32× |
+| 64 | +2.42% | ~64× |
+| 32 | +4.51% | ~128× |
+| 16 | +7.51% | ~256× |
+
+Eval scope: 12 sequences × 4K tokens of WikiText-103 validation (~50K
+tokens). Read these as "what we observed on this slice", not population-
+level estimates.
+
+The K-sweep recall numbers (24–41%) and the in-training `evaluate()` recall
+(50.9% at K=128) come from different sampled subsets of the streaming split
+and shouldn't be directly compared. The repo also reports `mass@K` (sum of
+teacher attention probability captured by the search top-K) — that's the
+more direct retrieval-quality metric when softmax is sharp.
 
 ### Per-layer recall (pilot)
 
@@ -102,7 +111,24 @@ Early layers are harder for content-addressable retrieval — their attention
 is more local/positional than semantic. Consistent across K, so it's a
 property of the layer rather than noise.
 
-A 34-layer headline run on 8K context follows.
+### Caveats / what's next
+
+- **Packing**: pilot training and eval ran with sequence packing on (no
+  segment-level causal mask, since transformers' default forward doesn't
+  build them). The relative PPL gap between full and ANN is internally
+  consistent under this confound, but the negative gap at K≥256 has at
+  least three candidate explanations we haven't disentangled —
+  (a) sparse-softmax denoising, (b) ANN happening to filter cross-document
+  keys that full attention attends to, (c) sample noise on a small eval.
+  The default config now has packing off so the next run isolates (a).
+- **Exact-topK oracle**: a four-way Pareto (full vs. exact top-K vs.
+  search-topK exact vs. search-ANN) is the natural follow-up to separate
+  "denoising from any sparsity" from "denoising from learned projections."
+- **Wall-clock**: not measured. The FAISS path in the repo is a CPU-side
+  research prototype, not a deployable runtime. A GPU-resident topk kernel
+  is the next-step engineering.
+- **34-layer headline** was queued (`make_headline_config()` is wired) and
+  will mirror its checkpoints here when it runs.
 
 ## Files
 
@@ -144,7 +170,8 @@ the trained layers and run with FAISS HNSW retrieval at inference time.
 ## Training recipe
 
 - Frozen base: Qwen3-4B-Instruct-2507 (36 layers, hidden 2560, GQA 32:8).
-- Data: WikiText-103 raw, packed to 4K-token sequences.
+- Data: WikiText-103 raw, 4K-token sequences (packing was on at training
+  time; default in the repo is now off — see Caveats).
 - 2000 steps, batch 8, lr 1e-4 (cosine, 100-step warmup), AdamW.
 - `α=β=1` (contrastive + KL distillation, both layers averaged).
 - bf16 weights, fp32 loss math.