fix(seam): dev loss used train-split codes; add erratum

01db651 verified 1 day ago

9.17 kB

# Results

Hardware: Apple M1 Max, 64 GB, MLX 0.32.3. Date of all runs: 2026-10-29 to 2026-10-31. One run per setting, no confidence intervals.

**Throttling caveat.** macOS Low Power Mode was active on AC power until 2026-10-31 10:45 and cost about 46 % throughput. Anything measured before that time is marked `[throttled]`. Quality metrics (accuracy, F1, perplexity) are not affected; latencies and tok/s are.

| Result | Run time (local time) | Throttled? |
|---|---|---|
| codec parity, round trips, masking stress (`results/codec`, `results/masking`) | re-run 10-31 evening on the final sources | n/a (no timing used) |
| MASSIVE pilot quality (`results/massive/results.json`) | 10-31 05:13 | quality no, latencies `[throttled]` |
| MASSIVE training throughput, TinyStories NCM 89M "128 tokens in 60 to 110 ms" | 10-31 00:00 to 05:00 | `[throttled]` |
| Seam training | 10-31 11:30 to 15:04 | no |
| Seam generation, metrics, judge, latencies | 10-31 15:37 to 15:45 | no |

## 1. Codec port (`codec_mlx.py`)
- Official XSum (GPT-2 tokenizer, L=48, 9 levels) and WMT14 (Qwen3-0.6B tokenizer, L=64) codecs ported. `parity.py` compares against a dump from the official PyTorch model: pre-quantisation latent, codes per level, cumulative z-hat, decoded ids per level, and decode from torch codes. Result (logs in `results/codec/parity_{xsum,wmt14}.txt`, CPU, fp32 and fp64, test texts plus random id rows, reference = PyTorch fp32 dump): XSum 100 % identical codes at all 9 levels; WMT14 100 % at 10 of 11 levels and 99.89 % at level 9 (one code in the batch differs, same in fp32 and fp64, cause not investigated), with the decoded ids still 100 % identical at every level and from torch codes. Max |diff| of z-hat is about 1e-5 except level 9/10 of WMT14 (6e-2, caused by that one code).
- Converted sizes (fp32 safetensors): XSum 231 MB, WMT14 549 MB. Torch originals: ~300 MB and ~646 MB. Both are fetched from upstream by the user; nothing is redistributed.

## 2. Codec fidelity and surrogate masking (`results/codec/rt_*.txt`, `results/masking/mask_*.txt`)
Entities kept after encode/decode of texts with rare names, places, numbers and dates:
| codec | entities kept (raw rare entities) |
|---|---|
| TinyStories | 62 % (earlier run; output not stored) |
| XSum | 89 % (in-distribution news summaries: 92 % exact, 97 % entities) |
| WMT14 | 96 % |
Typical failures are character-level damage inside rare names ("Anneliese Kouöder", "6.2%" for "7.2%"). Special placeholder tokens (Qwen special / spare slots) are *not* preserved by the codec (0 % identical placeholder sequences), so placeholders must be real common words.

Masking with word surrogates (per-text table, slot sets tuned on a calibration split: PER Tom/Lily/Sam..., LOC London/Paris..., NUM two/three..., DATE weekdays, AMT "ten dollars"...). Stress test: 150 texts with 5 to 8 entities and repeats, gold spans, exact-match of the restored text:

| codec | raw entities | surrogates (tuned) | span recall | binding errors | regex detector alone (span recall / precision) |
|---|---|---|---|---|---|
| XSum | 80.7 % exact | **100 %** exact | 100 % | 0 | 90.8 % / 93.9 % |
| WMT14 | 63.3 % exact | **100 %** exact | 100 % | 0 | 86.1 % / 92.4 % |
| TinyStories | 1.3 % exact | 76 % exact (36/150 fail) | 98.9 % (target 99.5 % missed) | 0 (23 texts with count errors, 10 spurious surrogate outputs) | 86.3 % / 91.6 % |

So masking is solved for XSum/WMT14 *given gold spans* and not for the TinyStories codec (cause not investigated). The weak link overall is detection (regex span recall 86 to 91 %, types PER/LOC conflated). A German control set is part of the stress test. Per-condition numbers (1/2/5 entities, repeats, mixed types, collisions) are in the `mask_*.txt` files.

Structured output as natural sentences instead of pipes/JSON: 99.9 % vs 8.1 % target round trip (MASSIVE labels, `results/massive/REPORT_stage1.md`).

## 3. MASSIVE-de pilot (`results/massive/`)

Data: MASSIVE de-DE (CC BY 4.0, Amazon Science), 11514 train / 2033 dev / 2974 test. Each slot value is replaced by a surrogate name; prefix = masked utterance; target = sentence format; WMT14 codec frozen. All models from scratch.

| | NCM | AR | intent classifier |
|---|---|---|---|
| params | 27.59 M | 25.49 M | 3.28 M |
| intent acc | 0.682 | 0.735 | 0.747 |
| slot F1 | 0.476 | 0.591 | - |
| exact match | 0.510 | 0.576 | - |
| fact restoration | 0.475 | 0.591 | - |
| invalid surrogate rate | 1.5 % | 0.07 % | - |
| forwards (NFE) | 11 | 12.9 mean (b=1) | 1 |
| latency b=1 p50 `[throttled]` | 304 ms | 102 ms | 3.5 ms |
| latency b=16 p50 `[throttled]` | 3687 ms | 503 ms | 29.8 ms |

Training: NCM best checkpoint at step 3500 of a run capped at 330 min (dev exact 0.597 on 300 dev examples), AR best at step 4000 (dev target exact 0.625 on 512). **Verdict: NCM lost at equal size, in quality and in latency.** Target round trip through the codec is 99.97 % (gate 99.5 %), so the codec is not the bottleneck; data amount and short training are the likely limits. Not a statement about NCM at scale.

## 4. Block-seam experiment (`results/seam/`)

Question: when long text is generated in blocks of 128 tokens, can a prefix with (b) a surrogate entity table and (c) a coarse-code memory of the last 6 blocks reduce invented names and smooth the join between blocks, versus (a) text tail only?

Setup: TinyStories (CDLA-Sharing-1.0), names replaced by 8 surrogates (John, Mary, ...) with a corpus-statistics name detector, sentence-packed blocks, official TinyStories codec frozen (12 levels), our 8L/512d generator. Test: 250 held-out stories with at least 5 blocks, gold block 0 given, 4 blocks generated. Training: 66 min cap per variant (~2340 steps, bs 32), loss still falling, so strongly undertrained. Judge: LFM2-1.2B perplexity on 16|16-token windows across a seam vs inside a block.

| | gold | a T=0.7 | b T=0.7 | c T=0.7 | a T=0 | b T=0 | c T=0 |
|---|---|---|---|---|---|---|---|
| spurious (invented) names | 0 | 8.1 % | 3.2 % | 2.2 % | 9.5 % | 6.8 % | 3.5 % |
| mention consistent | 98.2 % | 91.6 % | 96.6 % | 97.7 % | 89.5 % | 93.2 % | 96.5 % |
| stories with any spurious | 0 | 20.0 % | 10.0 % | 10.8 % | 4.8 % | 3.6 % | 2.0 % |
| block-0 entity persists | 82.4 % | 41.8 % | 47.5 % | 60.2 % | 11.6 % | 17.6 % | 26.4 % |
| cross-block 4-gram repeat | 1.2 % | 6.3 % | 4.1 % | 4.5 % | 34.9 % | 31.6 % | 36.7 % |
| seam cond. PPL (judge) | 165.8 | 225.7 | 227.5 | 241.5 | 39.0 | 40.0 | 43.2 |
| within-block cond. PPL | 154.7 | 153.1 | 185.6 | 215.5 | 12.6 | 13.1 | 14.5 |
| gold with shuffled next block | seam 291.6 | | | | | | |
| b=1 latency (1 block) | - | 58 ms | 69 ms | 63 ms | 52 ms | 49 ms | 67 ms |

Reading:
- **Entity gains are real in this setup**: table and code memory lower invented names and raise carry-over of block-0 characters (a to c), at roughly 1.6x prefix length cost (P=32/40/94).
- **Seam smoothing is inconclusive**: seam PPL minus within-block PPL is not clearly better for b/c than for a, and c is worse on within-block PPL at T=0.7. T=0 outputs collapse into repetition (4-gram repeat ~35 %), which makes the T=0 PPLs meaningless as quality measures. Models need much longer training and several seeds before a statement is possible.
- Sampled stories are in `samples.md` and look poor (repeated tokens), as expected for 66-minute models. Gold excerpts in that file are from TinyStories.

## 5. Side notes not backed by files in this package

(Gemma-4 26B-A4B JANG 4-bit on the same M1 Max; the converter and bench scripts are excluded for licensing reasons, see `LICENSES.md`.) vMLX 50 tok/s decode vs llama.cpp 41 tok/s on the same weights converted JANG to GGUF without loss; continuous batching on the MoE model gave vMLX 1.2x and llama.cpp 1.85x aggregate speed-up. These were measured after the Low Power Mode fix (benchmark JSONs from 11:57 and 15:23); an earlier partial run (10:56) is throttled and not used. Treat as anecdotal.

## 6. Release edits

For the public release, a few test-pool names in `texts.py` and `mask_stress.py` were replaced by fictional names. The CPU-only scripts (`roundtrip_news.py`, `mask_stress.py`, `parity.py`) were re-run afterwards with the final sources, so all stored outputs in `results/codec` and `results/masking` come from the code in this repo. Changes versus the first run: XSum raw round trip entities kept 90 % to 89 %; masking stress raw exact XSum 84.0 % to 80.7 %, TinyStories 2.0 % to 1.3 %, regex-detected-span exact TinyStories 38.0 % to 40.7 %; all other headline numbers unchanged. The TinyStories codec round trip (62 % entities kept) was not re-run and no output of it is stored.

## Erratum (2026-10-31)

`seam/train_seam.py` computed the periodic dev loss with train-split codes (`codes[tdv]`) instead of dev-split codes (`cd_[tdv]`). Fixed in this commit. Effect: best-checkpoint *selection* for the block-seam variants used a wrong signal. The reported seam metrics (entity consistency, spurious names, judge PPL) were measured on the held-out test stories and are unaffected as measurements, but the chosen checkpoints may not be the best ones. A longer re-run with true dev loss is in progress.

	# Results

	Hardware: Apple M1 Max, 64 GB, MLX 0.32.3. Date of all runs: 2026-10-29 to 2026-10-31. One run per setting, no confidence intervals.

	Throttling caveat. macOS Low Power Mode was active on AC power until 2026-10-31 10:45 and cost about 46 % throughput. Anything measured before that time is marked `[throttled]`. Quality metrics (accuracy, F1, perplexity) are not affected; latencies and tok/s are.

	\| Result \| Run time (local time) \| Throttled? \|
	\|---\|---\|---\|
	\| codec parity, round trips, masking stress (`results/codec`, `results/masking`) \| re-run 10-31 evening on the final sources \| n/a (no timing used) \|
	\| MASSIVE pilot quality (`results/massive/results.json`) \| 10-31 05:13 \| quality no, latencies `[throttled]` \|
	\| MASSIVE training throughput, TinyStories NCM 89M "128 tokens in 60 to 110 ms" \| 10-31 00:00 to 05:00 \| `[throttled]` \|
	\| Seam training \| 10-31 11:30 to 15:04 \| no \|
	\| Seam generation, metrics, judge, latencies \| 10-31 15:37 to 15:45 \| no \|

	## 1. Codec port (`codec_mlx.py`)

	- Official XSum (GPT-2 tokenizer, L=48, 9 levels) and WMT14 (Qwen3-0.6B tokenizer, L=64) codecs ported. `parity.py` compares against a dump from the official PyTorch model: pre-quantisation latent, codes per level, cumulative z-hat, decoded ids per level, and decode from torch codes. Result (logs in `results/codec/parity_{xsum,wmt14}.txt`, CPU, fp32 and fp64, test texts plus random id rows, reference = PyTorch fp32 dump): XSum 100 % identical codes at all 9 levels; WMT14 100 % at 10 of 11 levels and 99.89 % at level 9 (one code in the batch differs, same in fp32 and fp64, cause not investigated), with the decoded ids still 100 % identical at every level and from torch codes. Max \|diff\| of z-hat is about 1e-5 except level 9/10 of WMT14 (6e-2, caused by that one code).
	- Converted sizes (fp32 safetensors): XSum 231 MB, WMT14 549 MB. Torch originals: ~300 MB and ~646 MB. Both are fetched from upstream by the user; nothing is redistributed.

	## 2. Codec fidelity and surrogate masking (`results/codec/rt_.txt`, `results/masking/mask_.txt`)

	Entities kept after encode/decode of texts with rare names, places, numbers and dates:

	\| codec \| entities kept (raw rare entities) \|
	\|---\|---\|
	\| TinyStories \| 62 % (earlier run; output not stored) \|
	\| XSum \| 89 % (in-distribution news summaries: 92 % exact, 97 % entities) \|
	\| WMT14 \| 96 % \|

	Typical failures are character-level damage inside rare names ("Anneliese Kouöder", "6.2%" for "7.2%"). Special placeholder tokens (Qwen special / spare slots) are not preserved by the codec (0 % identical placeholder sequences), so placeholders must be real common words.

	Masking with word surrogates (per-text table, slot sets tuned on a calibration split: PER Tom/Lily/Sam..., LOC London/Paris..., NUM two/three..., DATE weekdays, AMT "ten dollars"...). Stress test: 150 texts with 5 to 8 entities and repeats, gold spans, exact-match of the restored text:

	\| codec \| raw entities \| surrogates (tuned) \| span recall \| binding errors \| regex detector alone (span recall / precision) \|
	\|---\|---\|---\|---\|---\|---\|
	\| XSum \| 80.7 % exact \| 100 % exact \| 100 % \| 0 \| 90.8 % / 93.9 % \|
	\| WMT14 \| 63.3 % exact \| 100 % exact \| 100 % \| 0 \| 86.1 % / 92.4 % \|
	\| TinyStories \| 1.3 % exact \| 76 % exact (36/150 fail) \| 98.9 % (target 99.5 % missed) \| 0 (23 texts with count errors, 10 spurious surrogate outputs) \| 86.3 % / 91.6 % \|

	So masking is solved for XSum/WMT14 given gold spans and not for the TinyStories codec (cause not investigated). The weak link overall is detection (regex span recall 86 to 91 %, types PER/LOC conflated). A German control set is part of the stress test. Per-condition numbers (1/2/5 entities, repeats, mixed types, collisions) are in the `mask_*.txt` files.

	Structured output as natural sentences instead of pipes/JSON: 99.9 % vs 8.1 % target round trip (MASSIVE labels, `results/massive/REPORT_stage1.md`).

	## 3. MASSIVE-de pilot (`results/massive/`)

	Data: MASSIVE de-DE (CC BY 4.0, Amazon Science), 11514 train / 2033 dev / 2974 test. Each slot value is replaced by a surrogate name; prefix = masked utterance; target = sentence format; WMT14 codec frozen. All models from scratch.

	\| \| NCM \| AR \| intent classifier \|
	\|---\|---\|---\|---\|
	\| params \| 27.59 M \| 25.49 M \| 3.28 M \|
	\| intent acc \| 0.682 \| 0.735 \| 0.747 \|
	\| slot F1 \| 0.476 \| 0.591 \| - \|
	\| exact match \| 0.510 \| 0.576 \| - \|
	\| fact restoration \| 0.475 \| 0.591 \| - \|
	\| invalid surrogate rate \| 1.5 % \| 0.07 % \| - \|
	\| forwards (NFE) \| 11 \| 12.9 mean (b=1) \| 1 \|
	\| latency b=1 p50 `[throttled]` \| 304 ms \| 102 ms \| 3.5 ms \|
	\| latency b=16 p50 `[throttled]` \| 3687 ms \| 503 ms \| 29.8 ms \|

	Training: NCM best checkpoint at step 3500 of a run capped at 330 min (dev exact 0.597 on 300 dev examples), AR best at step 4000 (dev target exact 0.625 on 512). Verdict: NCM lost at equal size, in quality and in latency. Target round trip through the codec is 99.97 % (gate 99.5 %), so the codec is not the bottleneck; data amount and short training are the likely limits. Not a statement about NCM at scale.

	## 4. Block-seam experiment (`results/seam/`)

	Question: when long text is generated in blocks of 128 tokens, can a prefix with (b) a surrogate entity table and (c) a coarse-code memory of the last 6 blocks reduce invented names and smooth the join between blocks, versus (a) text tail only?

	Setup: TinyStories (CDLA-Sharing-1.0), names replaced by 8 surrogates (John, Mary, ...) with a corpus-statistics name detector, sentence-packed blocks, official TinyStories codec frozen (12 levels), our 8L/512d generator. Test: 250 held-out stories with at least 5 blocks, gold block 0 given, 4 blocks generated. Training: 66 min cap per variant (~2340 steps, bs 32), loss still falling, so strongly undertrained. Judge: LFM2-1.2B perplexity on 16\|16-token windows across a seam vs inside a block.

	\| \| gold \| a T=0.7 \| b T=0.7 \| c T=0.7 \| a T=0 \| b T=0 \| c T=0 \|
	\|---\|---\|---\|---\|---\|---\|---\|---\|
	\| spurious (invented) names \| 0 \| 8.1 % \| 3.2 % \| 2.2 % \| 9.5 % \| 6.8 % \| 3.5 % \|
	\| mention consistent \| 98.2 % \| 91.6 % \| 96.6 % \| 97.7 % \| 89.5 % \| 93.2 % \| 96.5 % \|
	\| stories with any spurious \| 0 \| 20.0 % \| 10.0 % \| 10.8 % \| 4.8 % \| 3.6 % \| 2.0 % \|
	\| block-0 entity persists \| 82.4 % \| 41.8 % \| 47.5 % \| 60.2 % \| 11.6 % \| 17.6 % \| 26.4 % \|
	\| cross-block 4-gram repeat \| 1.2 % \| 6.3 % \| 4.1 % \| 4.5 % \| 34.9 % \| 31.6 % \| 36.7 % \|
	\| seam cond. PPL (judge) \| 165.8 \| 225.7 \| 227.5 \| 241.5 \| 39.0 \| 40.0 \| 43.2 \|
	\| within-block cond. PPL \| 154.7 \| 153.1 \| 185.6 \| 215.5 \| 12.6 \| 13.1 \| 14.5 \|
	\| gold with shuffled next block \| seam 291.6 \| \| \| \| \| \| \|
	\| b=1 latency (1 block) \| - \| 58 ms \| 69 ms \| 63 ms \| 52 ms \| 49 ms \| 67 ms \|

	Reading:
	- Entity gains are real in this setup: table and code memory lower invented names and raise carry-over of block-0 characters (a to c), at roughly 1.6x prefix length cost (P=32/40/94).
	- Seam smoothing is inconclusive: seam PPL minus within-block PPL is not clearly better for b/c than for a, and c is worse on within-block PPL at T=0.7. T=0 outputs collapse into repetition (4-gram repeat ~35 %), which makes the T=0 PPLs meaningless as quality measures. Models need much longer training and several seeds before a statement is possible.
	- Sampled stories are in `samples.md` and look poor (repeated tokens), as expected for 66-minute models. Gold excerpts in that file are from TinyStories.

	## 5. Side notes not backed by files in this package

	(Gemma-4 26B-A4B JANG 4-bit on the same M1 Max; the converter and bench scripts are excluded for licensing reasons, see `LICENSES.md`.) vMLX 50 tok/s decode vs llama.cpp 41 tok/s on the same weights converted JANG to GGUF without loss; continuous batching on the MoE model gave vMLX 1.2x and llama.cpp 1.85x aggregate speed-up. These were measured after the Low Power Mode fix (benchmark JSONs from 11:57 and 15:23); an earlier partial run (10:56) is throttled and not used. Treat as anecdotal.

	## 6. Release edits

	For the public release, a few test-pool names in `texts.py` and `mask_stress.py` were replaced by fictional names. The CPU-only scripts (`roundtrip_news.py`, `mask_stress.py`, `parity.py`) were re-run afterwards with the final sources, so all stored outputs in `results/codec` and `results/masking` come from the code in this repo. Changes versus the first run: XSum raw round trip entities kept 90 % to 89 %; masking stress raw exact XSum 84.0 % to 80.7 %, TinyStories 2.0 % to 1.3 %, regex-detected-span exact TinyStories 38.0 % to 40.7 %; all other headline numbers unchanged. The TinyStories codec round trip (62 % entities kept) was not re-run and no output of it is stored.

	## Erratum (2026-10-31)

	`seam/train_seam.py` computed the periodic dev loss with train-split codes (`codes[tdv]`) instead of dev-split codes (`cd_[tdv]`). Fixed in this commit. Effect: best-checkpoint selection for the block-seam variants used a wrong signal. The reported seam metrics (entity consistency, spurious names, judge PPL) were measured on the held-out test stories and are unaffected as measurements, but the chosen checkpoints may not be the best ones. A longer re-run with true dev loss is in progress.