bcdennis/concept-attn-pilot-cond1

Concept-Attention Pilot — Cond-1: concept attention + next-concept prediction (REJECTED)

The arm that adds the ConceptLM-style NCP loss (MSE of a soft codebook-mixture prediction against the next chunk's quantized concept) on top of concept attention. Rejected across five fix waves; the final verdict is mechanistic, not just empirical. Tag: arxiv:2609.10715, arxiv:2602.08984

Wave history

• Wave 1 (vanilla): codebook utilization pinned at 0.02, VQ loss exploded (45->70) as hidden norms outgrew the codebook, gates fused at ~0. Cancelled at step ~150.
• Wave 2 (per-segment LayerNorm on VQ input + dead-code restarts): crashed on a device bug in the restart path at first firing; fixed, superseded.
• Wave 3 (cosine assignment + gate warmup floor): 75-step smoke — utilization still pinned 0.02; NCP decayed to exactly 0.0000 (degenerate).
• Wave 4 (fixed gate init 0.1 = usage pressure): smoke — utilization still 0.02, NCP -> 0.0004. Refuted the "usage pressure prevents collapse" hypothesis.
• Wave 5 (CA_ST_MEM=1: straight-through memory so CE gradient reaches the concept states): NCP stayed informative (degeneracy fixed) but the 50-step dead-code restart cycle then drove churn — val_ce 5.88 -> 6.3 (step 800) -> 7.45 (step 900), gradient norms to 679. Cancelled at ~step 1000 (262M tokens).

Mechanism (the finding)

The NCP-MSE objective has a degenerate global optimum: constant concepts make next-concept prediction free. In ConceptLM/NCP-ArchPreview this is counteracted because concepts are re-injected UNGATED into the token stream — NTP gradient flows into the concept representation and punishes collapse. In a gated concept-attention architecture where the memory is the quantized codes (gradient stops at the codebook transform), the concept states' ONLY teacher is NCP, so collapse is inevitable at any gate value. Wave 5 confirmed the fix direction (give concept states a CE gradient) removes the degeneracy, but the naive restart cadence then destabilizes training. A viable Cond-1 needs both the ST-memory path and collapse-aware restart policy (restart only persistently-dead codes, much lower cadence) — that is future work, not part of this verdict.

Baseline reference

Dense baseline final val_ce 5.1985; Cond-0 (attention only, no NCP) 5.1940.