token-channel/README.md

Token-Channel Program — drafting heads

Speculative-decoding drafting heads for Qwen3-8B, placed in the 2×2 design space they test. All trained on the same 800k data (nemotron-nothink). Metric τ (bonus-inclusive), btop rail, n=100.

The 2×2 — channel × backbone

	markov (rank-256 bigram, logit-level)	token layer (feature-level transformer layer)
frozen base	A_frozen-markov/ — helps only slot 0, ~0 net. tree inert.	B_frozen-token/ — +0.24 all four, converges epoch 0. tree inert.
co-trained base	E_cotrain-markov/ ★ the shape that works — chain +0.10 but the TREE recovers the full delegation (+2.07); the only co-trained head with a LIVE tree.	D_cotrain-token/ — strong chain, but a full layer per tree node is too expensive → tree DEAD.

Controls & variants

• 5p0_cotrain-none/ — co-trained base, NO channel. The origin D and E subtract against (τ 5.27 @ s5000).
• Bx2_frozen-token-x2/ — frozen base + TWO layers. +0.22 = one layer's level → the base is the ceiling, not capacity.
• C_hybrid/ — mask AND token inputs co-trained (self-forcing, block-15). One ckpt for tree AND chain. Still training.

Reference heads (ref/) — comparison anchors, not our arms

• ref/dspark-b7, ref/dspark-b15 — bidirectional + markov, our 800k reproduction (DeepSeek/PKU design). b15 head-to-head: chain 8.75 / tree 11.62 math500; its base alone is a half-product (bare tree 4.28).
• ref/b7bb — DSpark-arch 800k (slot-switch experiment).

Verdict

A frozen backbone is a CEILING (B-x2 = B). The winning shape is a co-trained base + a corrector cheap enough to stay present at EVERY tree node (E proves it; D's layer-cost corrector cannot).

Each <arm>/s<step>/ folder holds one checkpoint: config.json · dflash.py · model.safetensors · target_release_weights.safetensors (+ dspark_heads.py for markov arms).