Rebuild model card from reproducible CPU audit; include evidence and limitations

README.md

@@ -13,7 +13,7 @@ tags:
 
 1,346 checkpoint files across 34 run folders, totaling **215.178 GB** (200.400 GiB). The inventory includes **21 `_final.pt` files** and **1325 step files**. Counts and sizes are derived from the pinned Hub inventory, not estimates from nominal architecture sizes.
 
-For evaluated FP32 exports, see [Cukinator/cpu1-ablations-final](https://huggingface.co/Cukinator/cpu1-ablations-final). Source code: [Cukinator/1.58bits](https://github.com/igna-s/New-1.58).
+For evaluated FP32 exports, see [Cukinator/cpu1-ablations-final](https://huggingface.co/Cukinator/cpu1-ablations-final). Source code: [igna-s/New-1.58](https://github.com/igna-s/New-1.58).
 
 ## What the audit found
 
@@ -89,7 +89,7 @@ from huggingface_hub import hf_hub_download
 torch.set_num_threads(1)
 path = hf_hub_download("Cukinator/cpu1-ablation-checkpoints", "run_04/checkpoint_run_04_final.pt",
     revision="fb402d025e843ed00fb990966bf2430c32eb958c", local_dir="D:/cpu1/active", cache_dir="D:/cpu1/hf/hub")
-from train_ablation import build_ablation_model, load_ablation_checkpoint
+from training.ablation import build_ablation_model, load_ablation_checkpoint
 state, config = load_ablation_checkpoint(path)
 config["use_gradient_checkpointing"] = False
 model = build_ablation_model(config)
@@ -101,14 +101,14 @@ with torch.inference_mode():
 print([tuple(head.shape) for head in logits])
 ```
 
-This forward uses zero target bytes in the local decoder; it is a shape smoke test. For scored likelihood, pass `byte_targets` as shown in the audit. For byte generation with retained Transformer context and one recurrent prefill, use `generation()` in `eval/audit_readmes.py`. Generation remains experimental; inspect the saved examples before choosing a checkpoint.
+This forward uses zero target bytes in the local decoder; it is a shape smoke test. For scored likelihood, pass `byte_targets` as shown in the audit. For byte generation with retained Transformer context and one recurrent prefill, use `generation()` in `evaluation/audit.py`. Generation remains experimental; inspect the saved examples before choosing a checkpoint.
 
 ## Resume training
 
 Use an intermediate checkpoint, not an inference-only packed final. The current source CLI names the argument `--resume_ckpt`:
 
 ```powershell
-python train_ablation.py --run run_04 --resume_ckpt D:/cpu1/active/run_04/checkpoint_run_04_step640.pt
+python -m training.ablation --run run_04 --resume_ckpt D:/cpu1/active/run_04/checkpoint_run_04_step640.pt
 ```
 
 Download the exact step file from its run folder first. Resume requires matching source/configuration, dependencies, and suitable training hardware. This audit did not run a training-resume experiment.
@@ -128,7 +128,7 @@ This card was rebuilt from a fresh CPU audit on **2026-10-06**. All 34 published
 - Shared sample JSON SHA-256: `11d143c06478ff7bbe364c541a58e6a2e76504c4cb0595b294f46e98f3cfe798`.
 - Byte targets are the next patch; incomplete final patches are discarded. The first patch is context only. LocalByteDecoder receives ground-truth target bytes and shifts them internally for causal teacher forcing within a patch. Cross-entropy is summed over all targets and divided by their total count.
 - Byte loss is nats/byte; `BPB = NLL / ln(2)` and byte PPL is `exp(NLL)`. A uniform byte predictor gives BPB=8 and PPL=256. BPE loss uses Qwen2.5-3B token IDs with IDs outside the configured vocabulary mapped to 0, matching the dataset adapter. Its PPL is over that lossy mapped token stream and cannot be compared with byte PPL or interpreted as the likelihood of the original text.
-- All BPE models shared one resolved Qwen tokenizer in this run. Its resolved revision is recorded in `run_01.json` and `environment.json`; reuse that tokenizer revision/cache for strict reproduction. The current evaluator resolves the tokenizer Hub name at runtime rather than hard-coding that revision.
+- BPE models shared one Qwen tokenizer, resolved at runtime. The original audit did not capture its resolved revision (null in the records); that missing pin limits exact BPE reproduction. Inspect tokenizer_revision in each result when reproducing later runs.
 - Causality: change the last 16 of 32 input steps and compare all prefix logits, tolerance `1e-5`. Recurrent chunk equivalence: compare a 128-step forward against four 32-step chunks across all output heads, tolerance `1e-4`. Failed checks remain visible.
 - DeleteGate is disabled consistently: phase 1, elimination rate 0. The audit does not measure token elimination, phase-2 quality, or the benefit of DeleteGate.
 - Forward timing: 10 warm-up steps, then 100 measured single-input forwards. `steps/s = 100 / total measured seconds`; p50 and p95 are per-step latency. One byte-model step consumes a patch of 4 bytes; one BPE step consumes one token. Local decoder inputs are zeros in this microbenchmark. Transformer timing has one input step and no persistent attention context. This is not a context-scaling test or a generation benchmark.
@@ -141,14 +141,14 @@ Export repository revision audited: `65951289352c1a036b12ba6d888af037e7c90b13`.
 
 From the source checkout, install PyTorch plus `huggingface_hub`, `transformers`, `tokenizers`, `pyarrow`, `psutil`, `numpy`, and `datasets`. Pin the versions from `evaluation/environment.json` when reproducing this run.
 
-The audit additions are bundled in `evaluation/source/`. Copy those Python files into the checkout’s `eval/` directory when using a source revision that predates this audit. The base Git commit is recorded in `evaluation/environment.json`.
+The audit additions are bundled in `evaluation/source/`. Copy those Python files into the checkout’s `evaluation/` directory when using a source revision that predates this audit. The base Git commit is recorded in `evaluation/environment.json`.
 
 ```powershell
-python eval/audit_readmes.py --storage D:/cpu1-readme-audit --threads 1
-python eval/audit_readmes.py --storage D:/cpu1-readme-audit --cards
+python -m evaluation.audit --storage D:/cpu1-readme-audit --threads 1
+python -m evaluation.audit --storage D:/cpu1-readme-audit --cards
 ```
 
-Model downloads, tokenizer files, Hugging Face caches, and temporary files go to D:. Only one model checkpoint is kept in the task storage at a time; it is removed after its result is saved. Interrupted runs resume from matching evaluator hashes. Raw reports remain in `resultados/readme_audit/`.
+Model downloads, tokenizer files, Hugging Face caches, and temporary files go to D:. Only one model checkpoint is kept in the task storage at a time; it is removed after its result is saved. Interrupted runs resume from matching evaluator hashes. Raw reports remain in `outputs/audit/`.
 
 ## Fresh diagnostic results
 
@@ -310,7 +310,7 @@ These configurations did not reproduce a full-sequence forward when carrying sta
 
 ### Recurrent implementation diagnostic
 
-The source’s `AblationMLGRU._parallel_scan` was tested independently against its defining serial recurrence with random candidate/forget tensors, seed 19, shape `[1,32,8]`, FP32. The maximum difference was **19.100506**; splitting the same scan into two 16-step chunks differed by **11.471270** (tolerance `1e-4`). See [`implementation_diagnostics.json`](https://huggingface.co/Cukinator/cpu1-ablation-checkpoints/blob/main/evaluation/implementation_diagnostics.json) and the bundled `audit_primitives.py`.
+The source’s `AblationMLGRU._parallel_scan` was tested independently against its defining serial recurrence with random candidate/forget tensors, seed 19, shape `[1,32,8]`, FP32. The maximum difference was **19.100506**; splitting the same scan into two 16-step chunks differed by **11.471270** (tolerance `1e-4`). See [`implementation_diagnostics.json`](https://huggingface.co/Cukinator/cpu1-ablation-checkpoints/blob/main/evaluation/implementation_diagnostics.json) and the bundled `evaluation/diagnostics.py`.
 
 The implementation replaces log-cumulative-sum-exp with a cumulative-maximum normalization followed by a cumulative sum; accumulated terms are not rescaled when that maximum changes. The observed outputs therefore do not satisfy the intended recurrence. This is a source implementation defect; disabling or repairing the scan would create a different inference protocol and was not silently applied to these results. The trained-checkpoint audit uses the source as supplied.
 

evaluation/publication.json

@@ -1,6 +1,6 @@
 {
   "Cukinator/cpu1-ablations-final": {
-    "oid": "d38678e7c5d18577f36edade64495237c72a44b5",
-    "url": "https://huggingface.co/Cukinator/cpu1-ablations-final/commit/d38678e7c5d18577f36edade64495237c72a44b5"
+    "oid": "304d4a9f7414da8682aae5825daa16e40d74713a",
+    "url": "https://huggingface.co/Cukinator/cpu1-ablations-final/commit/304d4a9f7414da8682aae5825daa16e40d74713a"
   }
 }

evaluation/source/cpu1/__init__.py

@@ -0,0 +1 @@
+"""Shared CPU-1 model components and checkpoint codecs."""

evaluation/source/evaluation/__init__.py

@@ -0,0 +1 @@
+"""Reproducible audit and checkpoint tooling."""

evaluation/source/evaluation/audit.py

@@ -0,0 +1,299 @@
+"""Reproducible Hub audit. Download and evaluate exactly one model at a time.
+
+python -m evaluation.audit --storage D:/cpu1-readme-audit --threads 1
+python -m evaluation.audit --storage D:/cpu1-readme-audit --cards
+python -m evaluation.audit --storage D:/cpu1-readme-audit --upload
+"""
+import argparse
+import hashlib
+import json
+import math
+import os
+from pathlib import Path
+import platform
+import statistics
+import sys
+import time
+import urllib.request
+from datetime import datetime, timezone
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+OUT = ROOT / 'outputs' / 'audit'
+FINAL = 'Cukinator/cpu1-ablations-final'
+ARCHIVE = 'Cukinator/cpu1-ablation-checkpoints'
+FINAL_REV = '65951289352c1a036b12ba6d888af037e7c90b13'
+ARCHIVE_REV = 'fb402d025e843ed00fb990966bf2430c32eb958c'
+EVIDENCE_REV = 'd38678e7c5d18577f36edade64495237c72a44b5'
+DATA = 'Salesforce/wikitext'
+DATA_REV = 'b08601e04326c79dfdd32d625aee71d232d685c3'
+DATA_FILE = 'wikitext-2-raw-v1/test-00000-of-00001.parquet'
+PROMPTS = ['The history of artificial intelligence', 'Once upon a time', 'The purpose of science is']
+
+
+def save(path, value):
+    def finite_json(obj):
+        if isinstance(obj, float) and not math.isfinite(obj):
+            return None
+        if isinstance(obj, dict):
+            return {k: finite_json(v) for k, v in obj.items()}
+        if isinstance(obj, (list, tuple)):
+            return [finite_json(v) for v in obj]
+        return obj
+    path.parent.mkdir(parents=True, exist_ok=True)
+    temporary = path.with_suffix('.json.tmp')
+    temporary.write_text(json.dumps(finite_json(value), indent=2, ensure_ascii=False, allow_nan=False) + '\n', encoding='utf-8')
+    temporary.replace(path)
+
+
+def sha(path):
+    h = hashlib.sha256()
+    with path.open('rb') as f:
+        for block in iter(lambda: f.read(1024 * 1024), b''):
+            h.update(block)
+    return h.hexdigest()
+
+
+def restore_evidence(storage):
+    """Restore published reports only; never download model weights."""
+    import shutil
+    from huggingface_hub import snapshot_download
+    folder = Path(snapshot_download(
+        FINAL, revision=EVIDENCE_REV,
+        allow_patterns=['evaluation/*.json', 'evaluation/CORPUS_LICENSE.txt'],
+        local_dir=str(storage/'evidence'), cache_dir=str(storage/'hf'/'hub'),
+    ))/'evaluation'
+    OUT.mkdir(parents=True, exist_ok=True)
+    for source in sorted(folder.glob('*')):
+        if source.suffix == '.json' or source.name == 'CORPUS_LICENSE.txt':
+            shutil.copyfile(source, OUT/source.name)
+    print(f'Restored published evidence to {OUT}; no model weights downloaded.', flush=True)
+
+
+def prepare(storage):
+    OUT.mkdir(parents=True, exist_ok=True)
+    invpath = OUT / 'inventory.json'
+    if not invpath.exists():
+        inventory = {r: json.load(urllib.request.urlopen(f'https://huggingface.co/api/models/{r}/revision/{revision}?blobs=true')) for r, revision in ((FINAL, FINAL_REV), (ARCHIVE, ARCHIVE_REV))}
+        save(invpath, inventory)
+    corpuspath = OUT / 'corpus.json'
+    if not corpuspath.exists():
+        import pyarrow.parquet as pq
+        dest = storage / 'data' / 'test.parquet'
+        dest.parent.mkdir(parents=True, exist_ok=True)
+        urllib.request.urlretrieve(f'https://huggingface.co/datasets/{DATA}/resolve/{DATA_REV}/{DATA_FILE}', dest)
+        texts = pq.read_table(dest, columns=['text']).column('text').to_pylist()
+        # First 32 non-heading paragraphs with at least 260 UTF-8 bytes.
+        samples = [t.encode('utf-8')[:260].decode('utf-8', errors='ignore') for t in texts if len(t.encode('utf-8')) >= 260 and not t.strip().startswith('=')][:32]
+        assert len(samples) == 32
+        save(corpuspath, {'dataset': DATA, 'revision': DATA_REV, 'file': DATA_FILE, 'parquet_sha256': sha(dest), 'selection': 'first 32 non-heading paragraphs >=260 UTF-8 bytes; truncate to 260 bytes, discard incomplete trailing UTF-8', 'texts': samples})
+    return json.loads(invpath.read_text(encoding='utf-8')), json.loads(corpuspath.read_text(encoding='utf-8'))
+
+
+def loss_eval(model, config, corpus, tokenizer):
+    import torch
+    import torch.nn.functional as F
+    byte = config['tokenization'] in ('byte', 'byte_dynamic')
+    p = config.get('patch_size', 4)
+    total, count, mapped, all_ids = 0., 0, 0, 0
+    with torch.inference_mode():
+        for text in corpus['texts']:
+            if byte:
+                raw = list(text.encode('utf-8'))
+                data = torch.tensor(raw[:len(raw)//p*p]).reshape(1, -1, p)
+                x, y = data[:, :-1], data[:, 1:]
+                logits = model(x, byte_targets=y, training_phase=1, target_elimination=0.)[0]
+                if isinstance(logits, (tuple, list)):
+                    logits = torch.stack(logits, dim=2)
+            else:
+                ids = torch.tensor(tokenizer.encode(text, add_special_tokens=False)).reshape(1, -1)
+                cap = config['bpe_vocab_size']
+                mapped += int((ids >= cap).sum())
+                all_ids += ids.numel()
+                ids = torch.where(ids < cap, ids, 0)
+                x, y = ids[:, :-1], ids[:, 1:]
+                logits = model(x, training_phase=1, target_elimination=0.)[0]
+            loss = F.cross_entropy(logits.reshape(-1, logits.shape[-1]), y.reshape(-1), reduction='sum')
+            if not torch.isfinite(loss):
+                raise ValueError('nonfinite loss')
+            total += loss.item()
+            count += y.numel()
+    avg = total / count
+    return {'mean_nll': avg, 'targets': count, 'unit': 'byte' if byte else 'mapped_qwen_token', 'perplexity': math.exp(avg), 'bits_per_byte': avg/math.log(2) if byte else None, 'mapped_ids_fraction': mapped/all_ids if all_ids else None, 'source': 'shared_wikitext_test_sample', 'teacher_forcing': True, 'delete_gate': 'disabled (phase 1, elimination 0)'}
+
+
+def generation(model, config, prompt, length=128):
+    """Keep Transformer context; recurrent models carry state exactly once."""
+    import torch
+    from training.ablation import _global_pass, _sample_logits
+    p = config.get('patch_size', 4)
+    recurrent = config['token_mixer'] == 'mlgru'
+    raw = prompt.encode('utf-8')
+    raw += b'\x00' * ((-len(raw)) % p)
+    context = torch.tensor(list(raw)).reshape(1, -1, p)
+    states = [None] * len(model.blocks)
+    output = bytearray()
+    with torch.inference_mode():
+        h, states = _global_pass(model, context, states, 0.)
+        while len(output) < length:
+            new = torch.zeros(1, 1, p, dtype=torch.long)
+            for k in range(p):
+                logits = model.local_decoder(h[:, -1:], new)[k][0, 0] if model.use_local_decoder else model.heads[k](h[:, -1:])[0, 0]
+                value = _sample_logits(logits, 1., 40, .9)
+                new[0, 0, k] = value
+                output.append(value)
+            if len(output) >= length:
+                break
+            if recurrent:
+                h, states = _global_pass(model, new, states, 0.)
+            else:
+                context = torch.cat([context, new], dim=1)
+                window = config.get('window_size', 512)
+                h, _ = _global_pass(model, context[:, -window:], [None]*len(model.blocks), 0.)
+    return bytes(output[:length])
+
+
+def benchmark(engine):
+    """Isolated single-input forward, not generated bytes/s or context scaling."""
+    import torch
+    dummy = torch.zeros((1, 1, engine.info.patch_size) if engine.info.is_byte_level else (1, 1), dtype=torch.long)
+    times = []
+    with torch.inference_mode():
+        for _ in range(10):
+            _, engine.hidden_states = engine._ablation_step(engine._model, dummy)
+        engine.reset_state()
+        for _ in range(100):
+            t = time.perf_counter()
+            _, engine.hidden_states = engine._ablation_step(engine._model, dummy)
+            times.append(time.perf_counter()-t)
+    engine.reset_state()
+    values = sorted(times)
+    return {'forward_steps_per_second': len(times)/sum(times), 'p50_ms': statistics.median(times)*1000, 'p95_ms': values[94]*1000, 'jitter_cv': statistics.stdev(times)/statistics.mean(times), 'warmup_steps': 10, 'measured_steps': 100, 'context_steps': 1, 'backend': 'PyTorch FP32; quantization simulated for ternary modules; no C++ kernels', 'meaning': 'one input patch/token forward, zero local decoder targets; excludes autoregressive local sampling; Transformer has no persistent context'}
+
+
+def causality(engine):
+    import torch
+    from evaluation.correctness import _get_logits_list, _make_dummy_input
+    torch.manual_seed(101)
+    x = _make_dummy_input(engine.info, 1, 32)
+    changed = x.clone()
+    changed[:, 16:] = (changed[:, 16:] + 71) % engine.info.vocab_size
+    with torch.inference_mode():
+        a = _get_logits_list(engine._model(x)[0], engine.info)
+        b = _get_logits_list(engine._model(changed)[0], engine.info)
+    diff = max((u[:, :16]-v[:, :16]).abs().max().item() for u,v in zip(a,b))
+    return {'status': ('PASS' if diff < 1e-5 else 'FAIL') if math.isfinite(diff) else 'ERROR', 'max_prefix_logit_diff': diff, 'tolerance': 1e-5, 'protocol': '32 input steps; change steps 16:32; compare all heads at steps 0:16 with identical zero local targets'}
+
+
+def evaluate(storage, inventory, corpus, args):
+    import torch
+    import psutil
+    from huggingface_hub import hf_hub_download
+    from evaluation.checkpoints import load_run
+    from evaluation.correctness import run_correctness
+    from transformers import AutoTokenizer
+    torch.set_num_threads(args.threads)
+    torch.set_num_interop_threads(1)
+    revision = inventory[FINAL]['sha']
+    runs = [f['rfilename'].split('/')[0] for f in inventory[FINAL]['siblings'] if f['rfilename'].endswith('/model.pt')]
+    if args.runs:
+        runs = [r for r in runs if r in args.runs]
+    import subprocess
+    source = subprocess.run(['git', 'rev-parse', 'HEAD'], cwd=ROOT, capture_output=True, text=True)
+    source_commit = source.stdout.strip() if source.returncode == 0 else None
+    tokenizer = None
+    for index, run in enumerate(runs, 1):
+        engine = model = raw = None
+        result_path = OUT / f'{run}.json'
+        if result_path.exists() and not args.force and json.loads(result_path.read_text(encoding='utf-8')).get('suite_sha256') == sha(Path(__file__)):
+            print(f'[{index}/{len(runs)}] existing {run}', flush=True)
+            continue
+        # Only this task-owned file may be removed. No cache of model blobs.
+        model_path = storage / 'active' / run / 'model.pt'
+        for stale in (storage / 'active').glob('*/model.pt'):
+            target = stale.resolve()
+            if not target.is_relative_to((storage / 'active').resolve()):
+                raise ValueError('unsafe cleanup path')
+            target.unlink()
+        print(f'[{index}/{len(runs)}] downloading {run} to {model_path}', flush=True)
+        started = time.perf_counter()
+        hf_hub_download(FINAL, f'{run}/model.pt', revision=revision, local_dir=str(storage/'active'), cache_dir=str(storage/'hf'/'hub'))
+        record = {'run': run, 'revision': revision, 'file': f'{run}/model.pt', 'sha256': sha(model_path), 'file_bytes': model_path.stat().st_size, 'date_utc': datetime.now(timezone.utc).isoformat(), 'threads': args.threads, 'torch': torch.__version__, 'platform': platform.platform(), 'processor': platform.processor(), 'corpus_sha256': sha(OUT/'corpus.json'), 'suite_sha256': sha(Path(__file__)), 'seed': 42, 'source_commit': source_commit}
+        try:
+            torch.manual_seed(42)
+            engine, raw, config, _, _ = load_run(model_path, run)
+            model = engine._model
+            record.update({'config': config, 'step': raw.get('step'), 'stored_training_metrics': raw.get('metrics', {}), 'parameters': sum(p.numel() for p in model.parameters()), 'parameter_bytes': sum(p.numel()*p.element_size() for p in model.parameters()), 'format': raw.get('format'), 'strict_load': True})
+            record['correctness'] = run_correctness(engine)
+            record['correctness']['causality'] = causality(engine)
+            if not engine.info.is_byte_level and tokenizer is None:
+                tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen2.5-3B', cache_dir=str(storage/'hf'/'hub'), trust_remote_code=False)
+                record['tokenizer_revision'] = tokenizer.init_kwargs.get('_commit_hash')
+            record['loss'] = loss_eval(model, config, corpus, tokenizer)
+            record['benchmark'] = benchmark(engine)
+            if engine.info.is_byte_level:
+                samples = []
+                for i, prompt in enumerate(PROMPTS):
+                    torch.manual_seed(42+i)
+                    t = time.perf_counter()
+                    generated = generation(model, config, prompt)
+                    elapsed = time.perf_counter()-t
+                    try:
+                        generated.decode('utf-8', errors='strict')
+                        valid = True
+                    except UnicodeDecodeError:
+                        valid = False
+                    samples.append({'prompt': prompt, 'seed': 42+i, 'hex': generated.hex(), 'text_replacement_decode': generated.decode('utf-8', errors='replace'), 'bytes': len(generated), 'utf8_valid': valid, 'ascii_printable_fraction': sum(32<=b<127 or b in (9,10,13) for b in generated)/len(generated), 'repeated_8byte_fraction': sum(generated[i:i+8]==generated[i-8:i] for i in range(8,len(generated)-7))/max(1,len(generated)-15), 'seconds': elapsed, 'generated_bytes_per_second': len(generated)/elapsed})
+                record['generation'] = samples
+            else:
+                record['generation'] = {'status': 'N/A', 'reason': 'truncated Qwen vocabulary; token mapping is lossy, no comparable byte generation test'}
+            record['process_rss_bytes'] = psutil.Process().memory_info().rss
+        except Exception as exc:
+            record['error'] = f'{type(exc).__name__}: {exc}'
+            print(record['error'], flush=True)
+        record['total_seconds'] = time.perf_counter()-started
+        record['evaluation_complete'] = True
+        save(result_path, record)
+        print(f'{run}: loss={record.get("loss",{}).get("mean_nll")} elapsed={record["total_seconds"]:.1f}s', flush=True)
+        model_path.unlink()
+        engine = model = raw = None
+        import gc
+        gc.collect()
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--storage', type=Path, default=Path('D:/cpu1-readme-audit'))
+    parser.add_argument('--threads', type=int, default=1)
+    parser.add_argument('--runs', nargs='+')
+    parser.add_argument('--force', action='store_true')
+    parser.add_argument('--cards', action='store_true')
+    parser.add_argument('--upload', action='store_true')
+    parser.add_argument('--restore-evidence', action='store_true', help='Download only the published audit JSONs and corpus license')
+    args = parser.parse_args()
+    if args.storage.resolve().drive.upper() != 'D:':
+        raise ValueError('This audit requires storage on D:')
+    args.storage.mkdir(parents=True, exist_ok=True)
+    for name in ('HF_HOME','HF_HUB_CACHE','HF_DATASETS_CACHE','TRANSFORMERS_CACHE','TEMP','TMP'):
+        dest = args.storage / ('temp' if name in ('TEMP','TMP') else name.lower())
+        dest.mkdir(parents=True, exist_ok=True)
+        os.environ[name] = str(dest)
+    os.environ['HF_HUB_DISABLE_XET'] = '1'
+    os.environ['HF_HUB_DISABLE_PROGRESS_BARS'] = '1'
+    if args.restore_evidence:
+        restore_evidence(args.storage)
+        return
+    inventory, corpus = prepare(args.storage)
+    if args.cards or args.upload:
+        from evaluation.cards import build_cards, upload_cards
+        build_cards(inventory, corpus)
+        if args.upload:
+            upload_cards(inventory)
+    else:
+        evaluate(args.storage, inventory, corpus, args)
+
+
+if __name__ == '__main__':
+    if hasattr(sys.stdout, 'reconfigure'):
+        sys.stdout.reconfigure(encoding='utf-8')
+    main()

evaluation/source/evaluation/cards.py

@@ -0,0 +1,299 @@
+"""Generate Hub model cards and the dataset template from audited evidence."""
+from collections import defaultdict
+from datetime import datetime
+import json
+from pathlib import Path
+import re
+import sys
+
+from evaluation.audit import ROOT, OUT, FINAL, ARCHIVE, DATA, DATA_REV, DATA_FILE, PROMPTS, sha
+
+
+def link(repo, filename, revision='main'):
+    return f'https://huggingface.co/{repo}/blob/{revision}/{filename}'
+
+
+def manifest_rows(inventory):
+    folders = defaultdict(list)
+    for f in inventory[ARCHIVE]['siblings']:
+        if f['rfilename'].endswith('.pt'):
+            folders[f['rfilename'].split('/')[0]].append(f)
+    rows = ['| Folder | Step files | Highest filename step | Final file MB | Total GB |', '|---|---:|---:|---:|---:|']
+    for run, files in sorted(folders.items()):
+        steps = [int(m.group(1)) for f in files if (m := re.search(r'_step(\d+)\.pt$', f['rfilename']))]
+        finals = [f for f in files if f['rfilename'].endswith('_final.pt')]
+        final = f'{sum(f["size"] for f in finals)/1e6:.2f}' if finals else '—'
+        rows.append(f'| `{run}` | {len(steps)} | {max(steps) if steps else "—"} | {final} | {sum(f["size"] for f in files)/1e9:.3f} |')
+    return rows
+
+
+def audit_summary(records):
+    checks = [r.get('correctness', {}) for r in records]
+    causal = [c.get('causality', {}) for c in checks]
+    recurrent = [c['state_prop'] for c in checks if c.get('state_prop', {}).get('status') != 'N/A']
+    return [
+        '## What the audit found', '',
+        f'The shared audit covers {len(records)} FP32 exports on 32 short WikiText test passages, with CPU execution, one thread, and a fixed protocol. It does not evaluate every intermediate archive checkpoint.', '',
+        '| Check | Outcome |', '|---|---|',
+        f'| Strict checkpoint loading | {sum(r.get("strict_load", False) for r in records)}/{len(records)} loaded |',
+        f'| Numerical evaluation | {sum(bool(r.get("error")) for r in records)} models produced an evaluation error |',
+        f'| Prefix causality | {sum(c.get("status") == "FAIL" for c in causal)}/{sum(c.get("status") in ("PASS", "FAIL") for c in causal)} finite-output models failed |',
+        f'| Recurrent chunk equivalence | {sum(c.get("status") == "FAIL" for c in recurrent)}/{len(recurrent)} recurrent models failed |', '',
+        'Full results, generation examples, and the recurrent implementation diagnostic follow below. Loss units depend on tokenization. Loss from a model that fails causality cannot establish autoregressive prediction quality; printable output alone does not establish useful language generation.', '',
+    ]
+
+
+def protocol(inventory, corpus, records):
+    return [
+        '## Evaluation protocol', '',
+        f'This card was rebuilt from a fresh CPU audit on **2026-10-06**. All {len(records)} published FP32 exports were attempted; {sum(r.get("strict_load",False) for r in records)} loaded with zero missing or unexpected state keys. Numerical and loading errors remain visible. Raw JSON, the exact text sample, and the repository inventories are included in [`evaluation/`](evaluation/). Previous `resultados/ablation_*.json` reports are historical and were not mixed into these tables.', '',
+        '- Hardware: Intel Core i7-8750H (6 cores / 12 logical processors), Windows 11; CPU execution with **1 PyTorch thread**, batch size 1, FP32 parameters. PyTorch ' + records[0]['torch'] + '. No custom C/C++ kernels were used.',
+        '- Test text: [`Salesforce/wikitext`, `wikitext-2-raw-v1`, test split](https://huggingface.co/datasets/Salesforce/wikitext). First 32 non-heading paragraphs with at least 260 UTF-8 bytes, each truncated to 260 bytes without an incomplete trailing character. This is a small diagnostic sample, not the complete benchmark. No exhaustive training-overlap audit was performed.',
+        '- Corpus attribution: WikiText by its dataset authors, based on English Wikipedia contributions. The dataset metadata lists CC-BY-SA-3.0 and GFDL; the included corpus excerpt retains those upstream terms, separately from the model repository license. See `evaluation/CORPUS_LICENSE.txt`.',
+        f'- Dataset revision: `{DATA_REV}`; exact file: `{DATA_FILE}`; parquet SHA-256: `{corpus["parquet_sha256"]}`.',
+        f'- Shared sample JSON SHA-256: `{sha(OUT/"corpus.json")}`.',
+        '- Byte targets are the next patch; incomplete final patches are discarded. The first patch is context only. LocalByteDecoder receives ground-truth target bytes and shifts them internally for causal teacher forcing within a patch. Cross-entropy is summed over all targets and divided by their total count.',
+        '- Byte loss is nats/byte; `BPB = NLL / ln(2)` and byte PPL is `exp(NLL)`. A uniform byte predictor gives BPB=8 and PPL=256. BPE loss uses Qwen2.5-3B token IDs with IDs outside the configured vocabulary mapped to 0, matching the dataset adapter. Its PPL is over that lossy mapped token stream and cannot be compared with byte PPL or interpreted as the likelihood of the original text.',
+        '- BPE models shared one Qwen tokenizer, resolved at runtime. The original audit did not capture its resolved revision (null in the records); that missing pin limits exact BPE reproduction. Inspect tokenizer_revision in each result when reproducing later runs.',
+        '- Causality: change the last 16 of 32 input steps and compare all prefix logits, tolerance `1e-5`. Recurrent chunk equivalence: compare a 128-step forward against four 32-step chunks across all output heads, tolerance `1e-4`. Failed checks remain visible.',
+        '- DeleteGate is disabled consistently: phase 1, elimination rate 0. The audit does not measure token elimination, phase-2 quality, or the benefit of DeleteGate.',
+        '- Forward timing: 10 warm-up steps, then 100 measured single-input forwards. `steps/s = 100 / total measured seconds`; p50 and p95 are per-step latency. One byte-model step consumes a patch of 4 bytes; one BPE step consumes one token. Local decoder inputs are zeros in this microbenchmark. Transformer timing has one input step and no persistent attention context. This is not a context-scaling test or a generation benchmark.',
+        '- Byte generation: three prompts, 128 new bytes per prompt, seeds 42/43/44, temperature 1.0, top-k 40, top-p 0.9. Recurrent models carry state; Transformers reprocess the growing context. Generated bytes/s includes prompt prefill and local sampling. UTF-8 validity is tested on the complete continuation. Samples are retained as hex and decoded text. BPE generation is N/A because vocabulary truncation prevents an equivalent byte-output comparison.',
+        '- Printable ASCII fraction, UTF-8 validity, and repetition are diagnostics; they do not measure factual accuracy, instruction following, or useful language quality. RAM columns report parameter storage, not a measured peak process footprint. Ternary models still hold FP32 parameters and simulate quantization in PyTorch.', '',
+        f'Export repository revision audited: `{inventory[FINAL]["sha"]}`. Archive inventory revision: `{inventory[ARCHIVE]["sha"]}`. Each result records its checkpoint SHA-256, actual parameter count, saved step, configuration, thread count, time, sample hash, and evaluator hash.', '',
+        '### Reproduce', '',
+        'From the source checkout, install PyTorch plus `huggingface_hub`, `transformers`, `tokenizers`, `pyarrow`, `psutil`, `numpy`, and `datasets`. Pin the versions from `evaluation/environment.json` when reproducing this run.', '',
+        'The audit additions are bundled in `evaluation/source/`. Copy those Python files into the checkout’s `evaluation/` directory when using a source revision that predates this audit. The base Git commit is recorded in `evaluation/environment.json`.', '',
+        '```powershell',
+        'python -m evaluation.audit --storage D:/cpu1-readme-audit --threads 1',
+        'python -m evaluation.audit --storage D:/cpu1-readme-audit --cards',
+        '```', '',
+        'Model downloads, tokenizer files, Hugging Face caches, and temporary files go to D:. Only one model checkpoint is kept in the task storage at a time; it is removed after its result is saved. Interrupted runs resume from matching evaluator hashes. Raw reports remain in `outputs/audit/`.', '',
+    ]
+
+
+def tables(records):
+    lines = ['## Fresh diagnostic results', '',
+             '**Do not rank models across tokenizations or treat a failed-causality loss as autoregressive language-model perplexity.** Saved training metrics are retained separately in the JSONs; the numbers below are recomputed on the shared test sample.', '']
+    for byte in (True, False):
+        lines += ['### Byte models' if byte else '### BPE models (mapped Qwen token stream)', '',
+                  '| Run | Params M | Saved step | File MB | NLL | BPB | PPL | Causal | State chunks |',
+                  '|---|---:|---:|---:|---:|---:|---:|---|---|']
+        for r in records:
+            if 'config' not in r or (r['config']['tokenization'] in ('byte','byte_dynamic')) != byte:
+                continue
+            lm = r.get('loss')
+            cc = r['correctness']
+            if lm is None:
+                lines.append(f'| [`{r["run"]}`](evaluation/{r["run"]}.json) | {r["parameters"]/1e6:.3f} | {r["step"]} | {r["file_bytes"]/1e6:.2f} | ERROR | — | — | {cc["causality"]["status"]} | {cc["state_prop"]["status"]} |')
+                continue
+            bpb = f'{lm["bits_per_byte"]:.4f}' if byte else '—'
+            lines.append(f'| [`{r["run"]}`](evaluation/{r["run"]}.json) | {r["parameters"]/1e6:.3f} | {r["step"]} | {r["file_bytes"]/1e6:.2f} | {lm["mean_nll"]:.4f} | {bpb} | {lm["perplexity"]:.3f} | {cc["causality"]["status"]} | {cc["state_prop"]["status"]} |')
+        if not byte:
+            lines += ['', '| BPE run | IDs mapped to 0 | Vocabulary cap |', '|---|---:|---:|']
+            for r in records:
+                if r.get('config',{}).get('tokenization') == 'bpe' and 'loss' in r:
+                    lines.append(f'| `{r["run"]}` | {r["loss"]["mapped_ids_fraction"]*100:.2f}% | {r["config"]["bpe_vocab_size"]:,} |')
+        lines += ['']
+    lines += ['### CPU timing and byte generation', '', '| Run | Param MB (FP32) | Forward steps/s | p50 ms | p95 ms | Generated bytes/s | Valid UTF-8 samples |', '|---|---:|---:|---:|---:|---:|---:|']
+    for r in records:
+        bm = r.get('benchmark')
+        gen = r.get('generation')
+        if bm is None:
+            lines.append(f'| `{r["run"]}` | {r.get("parameter_bytes",0)/1e6:.2f} | ERROR | — | — | N/A | N/A |')
+            continue
+        if isinstance(gen, list):
+            speed = f'{sum(s["bytes"] for s in gen)/sum(s["seconds"] for s in gen):.2f}'
+            utf = f'{sum(s["utf8_valid"] for s in gen)}/{len(gen)}'
+        else:
+            speed = utf = 'N/A'
+        lines.append(f'| `{r["run"]}` | {r["parameter_bytes"]/1e6:.2f} | {bm["forward_steps_per_second"]:.2f} | {bm["p50_ms"]:.2f} | {bm["p95_ms"]:.2f} | {speed} | {utf} |')
+    lines += ['', '### Example continuations', '',
+        'The following fixed examples use the first audit prompt, `The history of artificial intelligence`, seed 42, and the sampling settings above. They illustrate why low diagnostic loss or valid UTF-8 alone does not establish useful text generation. Escapes preserve the recorded characters; complete byte sequences and all three prompts are in the JSON reports.', '']
+    for name in ('run_04_v3','run_14_v3'):
+        r = next((r for r in records if r['run']==name),None)
+        if r and isinstance(r.get('generation'),list):
+            lines += [f'`{name}`:', '', '```text', json.dumps(r['generation'][0]['text_replacement_decode'],ensure_ascii=True), '```', '']
+    lines += ['', '### Correctness checks', '', '| Run | Shapes | Probabilities | Determinism | State chunks | Ternary projection | State size | Causality |', '|---|---|---|---|---|---|---|---|']
+    keys = ['forward_shapes','probs','determinism','state_prop','ternary_weights','state_size','causality']
+    for r in records:
+        lines.append('| `' + r['run'] + '` | ' + ' | '.join(r.get('correctness',{}).get(k,{}).get('status','ERROR') for k in keys) + ' |')
+    failed_causal = [r['run'] for r in records if r.get('correctness',{}).get('causality',{}).get('status')=='FAIL']
+    failed_state = [r['run'] for r in records if r.get('correctness',{}).get('state_prop',{}).get('status')=='FAIL']
+    lines += ['', f'**Causality failures: {len(failed_causal)}/{len(records)}.** ' + (', '.join(f'`{r}`' for r in failed_causal) or 'None.'), '',
+              'Changing future inputs changed prefix logits in these models. Some saved configurations enable noncausal Bolmo cross-patch operations; the test detects the effect without isolating its individual source. Their full-sequence losses are descriptive reconstruction scores and cannot establish causal prediction quality.', '',
+              f'**State chunk failures: {len(failed_state)}.** ' + (', '.join(f'`{r}`' for r in failed_state) or 'None.'), '',
+              'These configurations did not reproduce a full-sequence forward when carrying state between chunks at the stated tolerance. An O(1) state tensor alone does not establish correct streaming equivalence. No fix to the trained architectures or weights was made for this audit.', '']
+    diagnostic = OUT/'implementation_diagnostics.json'
+    if diagnostic.exists():
+        d = json.loads(diagnostic.read_text(encoding='utf-8'))
+        lines += ['### Recurrent implementation diagnostic', '',
+            f'The source’s `AblationMLGRU._parallel_scan` was tested independently against its defining serial recurrence with random candidate/forget tensors, seed 19, shape `[1,32,8]`, FP32. The maximum difference was **{d["serial_max_diff"]:.6f}**; splitting the same scan into two 16-step chunks differed by **{d["chunks_max_diff"]:.6f}** (tolerance `1e-4`). See [`implementation_diagnostics.json`](evaluation/implementation_diagnostics.json) and the bundled `evaluation/diagnostics.py`.', '',
+            'The implementation replaces log-cumulative-sum-exp with a cumulative-maximum normalization followed by a cumulative sum; accumulated terms are not rescaled when that maximum changes. The observed outputs therefore do not satisfy the intended recurrence. This is a source implementation defect; disabling or repairing the scan would create a different inference protocol and was not silently applied to these results. The trained-checkpoint audit uses the source as supplied.', '']
+    errors = [r for r in records if r.get('error')]
+    if errors:
+        lines += ['### Evaluation errors', '', 'Scores and speed are unavailable when model loading or numerical evaluation fails. Nonfinite values are represented as JSON null; they are never substituted with valid-looking numbers.', '', '| Run | Error |', '|---|---|']
+        lines += [f'| `{r["run"]}` | {r["error"].replace("|","/")} |' for r in errors]
+        lines += ['']
+    return lines
+
+
+def architecture(records):
+    lines = ['## Architecture recorded in each checkpoint', '',
+             'Run names identify experiments; dimensions and features below come from the saved configuration. FP16 is the training quantization label, while this audit executes the loaded tensors in FP32. “50M” and “10M” in older cards were design families, not exact model sizes.', '',
+             '| Run | Mixer | Tokens | Quant. label | Width / layers / FFN | Local decoder | FP residual | Bolmo | DeleteGate | PFNet hidden | Channel decay |',
+             '|---|---|---|---|---|---|---|---|---|---:|---|']
+    for r in records:
+        if 'config' not in r:
+            continue
+        c = r['config']
+        yes = lambda k: 'yes' if c.get(k, False) else 'no'
+        lines.append(f'| `{r["run"]}` | {c["token_mixer"]} | {c["tokenization"]} | {c["quantization"]} | {c["d_model"]} / {c["n_layers"]} / {c["d_ff"]} | {yes("use_local_decoder") if c["tokenization"]!="bpe" else "N/A"} | {yes("fp_residual")} | {yes("use_bolmo_gating")} | {yes("use_delete_gate")} | {c.get("pfnet_hidden",0)} | {yes("mlgru_per_channel_decay")} |')
+    lines += ['', 'These experiments change parameter counts, representations, training signal, budgets, saved steps, and sometimes multiple architectural settings. The audit is not a controlled causal estimate of the benefit of an individual component. `chinchilla_tokens_per_param` is a configured budget; it does not prove that a run completed that budget. Actual saved steps are reported separately.', '']
+    return lines
+
+
+def quickstart(repo, revision, compact=False):
+    filename = 'run_04/checkpoint_run_04_final.pt' if compact else 'run_04/model.pt'
+    loader = ['from training.ablation import build_ablation_model, load_ablation_checkpoint', f'state, config = load_ablation_checkpoint(path)'] if compact else ['from training.ablation import build_ablation_model', 'payload = torch.load(path, map_location="cpu", weights_only=False)', 'config, state = payload["config"], payload["state_dict"]']
+    return ['## Load one checkpoint', '', 'Clone the [source repository](https://github.com/igna-s/New-1.58) and run this from its root. These are custom PyTorch checkpoints; `AutoModelForCausalLM.from_pretrained()` does not load them. The examples use a trusted checkpoint source and strict state loading.', '', '```python', 'import os', 'os.environ["HF_HOME"] = "D:/cpu1/hf"', 'os.environ["HF_HUB_DISABLE_XET"] = "1"', 'import torch', 'from huggingface_hub import hf_hub_download', 'torch.set_num_threads(1)', f'path = hf_hub_download("{repo}", "{filename}",', f'    revision="{revision}", local_dir="D:/cpu1/active", cache_dir="D:/cpu1/hf/hub")', *loader, 'config["use_gradient_checkpointing"] = False', 'model = build_ablation_model(config)', 'model.load_state_dict(state, strict=True)', 'model.eval()', 'x = torch.tensor(list(b"The history of AI.  ")).reshape(1, -1, 4)', 'with torch.inference_mode():', '    logits, states, extra = model(x, training_phase=1, target_elimination=0.0)', 'print([tuple(head.shape) for head in logits])', '```', '', 'This forward uses zero target bytes in the local decoder; it is a shape smoke test. For scored likelihood, pass `byte_targets` as shown in the audit. For byte generation with retained Transformer context and one recurrent prefill, use `generation()` in `evaluation/audit.py`. Generation remains experimental; inspect the saved examples before choosing a checkpoint.', '']
+
+
+def build_cards(inventory, corpus):
+    names = sorted(f['rfilename'].split('/')[0] for f in inventory[FINAL]['siblings'] if f['rfilename'].endswith('/model.pt'))
+    records = [json.loads((OUT/f'{r}.json').read_text(encoding='utf-8')) for r in names]
+    for r in records:
+        if not r.get('evaluation_complete') or r['revision'] != inventory[FINAL]['sha']:
+            raise ValueError(f'Incomplete audit for {r["run"]}')
+    if len({r['suite_sha256'] for r in records}) != 1 or len({r['corpus_sha256'] for r in records}) != 1:
+        raise ValueError('Mixed audit protocols')
+    header = ['---', 'license: apache-2.0', 'language:', '- en', 'tags:', '- cpu-1', '- ablation', '- ternary', '- research', '---', '']
+    shared = protocol(inventory, corpus, records) + tables(records) + architecture(records)
+    final_files = [f for f in inventory[FINAL]['siblings'] if f['rfilename'].endswith('/model.pt')]
+    final_card = header + ['# CPU-1 ablations: FP32 checkpoint exports', '',
+        f'{len(final_files)} custom PyTorch model exports, {sum(f["size"] for f in final_files)/1e9:.3f} GB of checkpoint files (decimal units). This is a research ablation collection, not an instruction-tuned assistant. It includes Transformer and MLGRU variants with byte patches or truncated BPE vocabularies, and floating-point or simulated ternary projections.', '',
+        f'Source: [igna-s/New-1.58](https://github.com/igna-s/New-1.58). Training archive: [{ARCHIVE}](https://huggingface.co/{ARCHIVE}). Training dataset companion: [Cukinator/cpu1-ablation-dataset](https://huggingface.co/datasets/Cukinator/cpu1-ablation-dataset).', '',
+        *audit_summary(records),
+        '## Payload format', '', '`<run>/model.pt` contains `format="fp32"`, `state_dict`, `config`, `step`, `run_name`, `metrics`, and source metadata. The exported parameters occupy four bytes each. “1.58-bit” describes the ternary value alphabet; these files are not stored as 1.58-bit tensors and this PyTorch path does not use packed-weight inference.', '',
+        'Historical `metrics` may be empty or come from different training validation protocols. They are preserved as metadata, not relabeled as freshly measured test results.', '',
+        *quickstart(FINAL, inventory[FINAL]['sha']), *shared,
+        '## License', '', 'The Hub metadata declares Apache-2.0 for the model repository. The source checkout does not include a standalone LICENSE file. Upstream datasets and tokenizer assets retain their own licenses.', '']
+    archived = [f for f in inventory[ARCHIVE]['siblings'] if f['rfilename'].endswith('.pt')]
+    finals = [f for f in archived if f['rfilename'].endswith('_final.pt')]
+    archive_card = header + ['# CPU-1 ablations: training checkpoint archive', '',
+        f'{len(archived):,} checkpoint files across {len({f["rfilename"].split("/")[0] for f in archived})} run folders, totaling **{sum(f["size"] for f in archived)/1e9:.3f} GB** ({sum(f["size"] for f in archived)/2**30:.3f} GiB). The inventory includes **{len(finals)} `_final.pt` files** and **{len(archived)-len(finals)} step files**. Counts and sizes are derived from the pinned Hub inventory, not estimates from nominal architecture sizes.', '',
+        f'For evaluated FP32 exports, see [{FINAL}](https://huggingface.co/{FINAL}). Source code: [igna-s/New-1.58](https://github.com/igna-s/New-1.58).', '',
+        *audit_summary(records),
+        '## What is stored', '',
+        '- `checkpoint_<run>_final.pt`: inference-oriented final checkpoints. Ternary runs use `compact_2bit`; floating-point baselines use floating-point checkpoint storage. File names alone do not identify the serialization format: inspect `format` inside the payload.',
+        '- `checkpoint_<run>_step<N>.pt`: intermediate training state; includes saved configuration and model state, and may include optimizer state. `_phase2_step<N>.pt` names distinguish phase-2 files where present.',
+        '- The 2-bit codec packs a ternary alphabet into two bits per stored value plus scales and floating-point tensors. It is not a guarantee of 1.58 effective bits per total model parameter. Resume step files and packed finals have different purposes.',
+        '- A folder without `_final.pt` is an intermediate-checkpoint archive. The highest filename step establishes the highest stored step; it does not establish convergence, completed training budget, or an early-stop reason.', '',
+        '## Archive inventory', '', *manifest_rows(inventory), '',
+        'Sizes use decimal MB and GB. Step counts include phase-2 step files. Highest step is parsed from filenames and does not merge training phases into a continuous training counter.', '',
+        *quickstart(ARCHIVE, inventory[ARCHIVE]['sha'], compact=True),
+        '## Resume training', '', 'Use an intermediate checkpoint, not an inference-only packed final. The current source CLI names the argument `--resume_ckpt`:', '',
+        '```powershell', 'python -m training.ablation --run run_04 --resume_ckpt D:/cpu1/active/run_04/checkpoint_run_04_step640.pt', '```', '',
+        'Download the exact step file from its run folder first. Resume requires matching source/configuration, dependencies, and suitable training hardware. This audit did not run a training-resume experiment.', '',
+        '## Scope of the measured results', '',
+        f'The results below were measured on the {len(records)} published exports in [{FINAL}](https://huggingface.co/{FINAL}), **not by evaluating every intermediate archive checkpoint**. Matching run names do not prove tensor identity between a packed final, an intermediate step, and an export. No packed/FP32 numerical-parity claim is made by this card.', '',
+        *shared, '## License', '', 'Hub metadata declares Apache-2.0. Dataset and tokenizer licenses are separate. The source checkout does not include a standalone LICENSE file.', '']
+    directory = ROOT / 'docs' / 'model_cards'
+    for stem, card in [('cpu1-ablations-final', final_card), ('cpu1-ablation-checkpoints', archive_card)]:
+        content = '\n'.join(card)
+        repo = FINAL if stem == 'cpu1-ablations-final' else ARCHIVE
+        content = content.replace('](evaluation/)', f'](https://huggingface.co/{repo}/tree/main/evaluation)')
+        content = content.replace('](evaluation/', f'](https://huggingface.co/{repo}/blob/main/evaluation/')
+        (directory/(stem+'.md')).write_text(content, encoding='utf-8')
+    save_environment(records)
+    (OUT/'CORPUS_LICENSE.txt').write_text('The text excerpts in corpus.json originate from WikiText (Salesforce/wikitext), based on English Wikipedia.\nDataset authors and contributors: see https://huggingface.co/datasets/Salesforce/wikitext .\nUpstream dataset metadata declares CC-BY-SA-3.0 and GFDL.\nCC-BY-SA-3.0: https://creativecommons.org/licenses/by-sa/3.0/\nGFDL: https://www.gnu.org/licenses/fdl-1.3.html\nChanges: selection of 32 test paragraphs and truncation to 260 UTF-8 bytes.\nThe corpus remains under its upstream terms; the model repository Apache-2.0 metadata does not relicense these excerpts.\n',encoding='utf-8')
+    dataset_card()
+    print('Generated both model cards and the dataset template; source README preserved.', flush=True)
+
+
+def dataset_card():
+    """Update the local companion template; publication scope is the two model repos."""
+    path = OUT/'dataset_inventory.json'
+    if not path.exists():
+        return
+    v = json.loads(path.read_text(encoding='utf-8'))
+    lines = ['---', 'license: mit', 'task_categories:', '- text-generation', 'language:', '- en', 'configs:', '- config_name: byte_marginalized', '  data_files:', '  - split: train', '    path: byte_marginalized/shard_*.parquet', '- config_name: bpe_tokenized', '  data_files:', '  - split: train', '    path: bpe_tokenized/shard_*.parquet', '---', '',
+        '# CPU-1 ablation training dataset', '',
+        'Companion dataset for the CPU-1 distillation experiments. The extraction source config uses Qwen/Qwen2.5-3B as teacher over FineWeb text. The byte and BPE subsets have different schemas and different shard counts; they must not be assumed to be aligned row by row.', '',
+        '## Verified Hub inventory: 2026-10-06', '', f'Revision: `{v["sha"]}`. The following counts and compressed sizes are from the Hub file listing, not from a full scan of the dataset rows.', '', '| Subset | Parquet files | Compressed GB | Shard range | Missing IDs within range |', '|---|---:|---:|---|---|']
+    for subset in ('byte_marginalized','bpe_tokenized'):
+        fs = [f for f in v['siblings'] if f['rfilename'].startswith(subset+'/') and f['rfilename'].endswith('.parquet')]
+        ids = sorted(int(re.search(r'shard_(\d+)\.parquet$',f['rfilename']).group(1)) for f in fs)
+        missing = sorted(set(range(min(ids),max(ids)+1))-set(ids))
+        lines.append(f'| `{subset}` | {len(fs)} | {sum(f["size"] for f in fs)/1e9:.3f} | {min(ids):04d}–{max(ids):04d} | '+', '.join(f'{n:04d}' for n in missing)+' |')
+    lines += ['', 'Sizes use decimal GB. Auxiliary `stream_checkpoint.json` and `stream_carry.pkl` files are extraction bookkeeping, not dataset examples. The file listing does not establish document totals, row totals, teacher coverage, split independence, or whether extraction has finished. Older shard and token totals must not be treated as current.', '',
+        '## Schema expected by the source adapters', '',
+        '| Subset | Fields | Interpretation |', '|---|---|---|',
+        '| Byte | `patches`, `targets`, `teacher_probs`, `teacher_mask`; optional `teacher_hidden`, `teacher_dim` | Flattened byte patches and next-patch targets; teacher signal and mask. Hidden arrays may be compacted to teacher-covered positions; follow the adapter. |',
+        '| BPE | `input_ids`, `labels`, `teacher_probs_bpe`, `teacher_ids_bpe`, `teacher_mask_bpe` | Qwen token IDs and shifted labels; flattened teacher top-k values and their original IDs. |', '',
+        'This table describes `data_pipeline/prepare.py` and the readers in `cpu1/components.py`; the October audit did not validate every row or measure teacher-signal quality. Check actual row lengths, optional fields and `teacher_dim` rather than assuming a fixed hidden width or identical schemas in all shards.', '',
+        'The extraction source currently configures four-byte patches, 5,000-step sequences, and top-128 BPE teacher entries. Those are writer settings, not a claim that every stored row matches them. Byte teacher signals are first-byte marginalizations of BPE candidates at mapped boundaries; they are not full autoregressive distributions for all four positions. Training uses the coverage masks.', '',
+        'The BPE adapter maps IDs outside the configured student vocabulary to ID 0, and filters teacher IDs accordingly. This is a lossy representation. Its token perplexity cannot be compared directly with byte perplexity or with an untruncated Qwen language model.', '',
+        '## Stream one subset', '', '```python', 'import os', 'os.environ["HF_HOME"] = "D:/cpu1/hf"', 'os.environ["HF_DATASETS_CACHE"] = "D:/cpu1/datasets"', 'from datasets import load_dataset', 'dataset = load_dataset(', '    "Cukinator/cpu1-ablation-dataset",', '    "byte_marginalized",', f'    revision="{v["sha"]}",', '    split="train", streaming=True,', '    cache_dir="D:/cpu1/datasets",', ')', 'row = next(iter(dataset))', 'print(row.keys())', '```', '',
+        'Only a training split is declared by the Hub card. If you need held-out evaluation, create and document a deterministic independent split or use a separate test corpus. Never silently substitute the training split for validation. Do not zip the byte and BPE streams by shard index or row index.', '',
+        '## Model evaluation', '',
+        'The new model-card audit uses a separate 32-passage WikiText test sample and loads no teacher model. Its corpus, procedure and raw results are in the [FP32 model repository](https://huggingface.co/Cukinator/cpu1-ablations-final). That diagnostic does not evaluate this dataset’s teacher coverage, entropy, hidden-state alignment, or full training distribution.', '',
+        '## Related artifacts', '', '- [Source and adapters](https://github.com/igna-s/New-1.58)', '- [FP32 exports and common evaluation](https://huggingface.co/Cukinator/cpu1-ablations-final)', '- [Packed finals and intermediate checkpoints](https://huggingface.co/Cukinator/cpu1-ablation-checkpoints)', '',
+        '## License', '', 'The dataset Hub metadata declares MIT. FineWeb source data and Qwen tokenizer/teacher assets retain their own upstream licenses. The repository label does not replace those upstream terms.', '']
+    (ROOT/'docs'/'model_cards'/'cpu1-ablation-dataset.md').write_text('\n'.join(lines), encoding='utf-8')
+
+
+def save_environment(records):
+    existing = OUT/'environment.json'
+    if existing.exists():
+        metadata = json.loads(existing.read_text(encoding='utf-8'))
+        if metadata.get('audit_script_sha256') == records[0]['suite_sha256']:
+            return  # Rendering historical evidence must preserve its provenance.
+    from evaluation.audit import save
+    from importlib.metadata import version
+    packages = {name: version(name) for name in ('torch','transformers','tokenizers','huggingface_hub','datasets','numpy','pyarrow','psutil')}
+    tokenizer_revision = next((r['tokenizer_revision'] for r in records if r.get('tokenizer_revision')), None)
+    save(OUT/'environment.json', {'python': sys.version, 'packages': packages, 'hardware': records[0].get('processor', 'CPU'), 'platform': records[0]['platform'], 'threads': records[0]['threads'], 'source_commit': records[0].get('source_commit'), 'source_changes': records[0].get('source_changes'), 'audit_script_sha256': records[0]['suite_sha256'], 'qwen_tokenizer_resolved_revision': tokenizer_revision})
+
+
+def upload_cards(inventory):
+    from huggingface_hub import HfApi, CommitOperationAdd
+    from evaluation.audit import save
+    import os
+    token = os.environ.get('HF_TOKEN')
+    if not token:
+        print('Provide the authorized Hugging Face token on stdin (not saved).', flush=True)
+        token = sys.stdin.readline().strip()
+    api = HfApi(token=token)
+    identity = api.whoami()
+    if identity['name'].lower() != 'cukinator':
+        raise ValueError('Token does not belong to the authorized repository owner')
+    commits = {}
+    for repo, stem in [(FINAL,'cpu1-ablations-final'), (ARCHIVE,'cpu1-ablation-checkpoints')]:
+        current = api.model_info(repo, files_metadata=True)
+        parent = current.sha
+        # A README-only publication changes the tip without changing audited weights.
+        # Pin the new commit to that tip, but require the exact audited model inventory.
+        expected = {f['rfilename']: (f['size'], f['lfs']['sha256'])
+                    for f in inventory[repo]['siblings'] if f['rfilename'].endswith('.pt')}
+        actual = {f.rfilename: (f.size, f.lfs.sha256 if f.lfs else None)
+                  for f in current.siblings if f.rfilename.endswith('.pt')}
+        if actual != expected:
+            raise ValueError(f'{repo} model files changed since the audit; rerun evaluation before publishing')
+        files = [('README.md', ROOT/'docs'/'model_cards'/f'{stem}.md')]
+        files += [(f'evaluation/{p.name}', p) for p in sorted(OUT.glob('*.json'))]
+        files += [('evaluation/CORPUS_LICENSE.txt', OUT/'CORPUS_LICENSE.txt')]
+        source_dirs = ('cpu1', 'training', 'evaluation', 'tests')
+        files += [(f'evaluation/source/{p.relative_to(ROOT).as_posix()}', p)
+                  for directory in source_dirs for p in sorted((ROOT/directory).glob('*.py'))]
+        files += [(f'evaluation/source/requirements/{p.name}', p)
+                  for p in sorted((ROOT/'requirements').glob('*.txt'))]
+        commit = api.create_commit(repo_id=repo, repo_type='model', parent_commit=parent, commit_message='Rebuild model card from reproducible CPU audit; include evidence and limitations', operations=[CommitOperationAdd(path_in_repo=name,path_or_fileobj=str(path)) for name,path in files])
+        commits[repo] = {'oid': commit.oid, 'url': commit.commit_url}
+        # Verify the actual published README bytes against the local card.
+        from huggingface_hub import hf_hub_download
+        readme = hf_hub_download(repo, 'README.md', revision=commit.oid, cache_dir=os.environ['HF_HUB_CACHE'], token=token)
+        if Path(readme).read_bytes() != files[0][1].read_bytes():
+            raise RuntimeError('Published card verification failed')
+        print(f'Published and verified {repo}: {commit.oid}', flush=True)
+        save(OUT/'publication.json', commits)

evaluation/source/evaluation/checkpoints.py

@@ -0,0 +1,115 @@
+"""Strict loading of custom CPU-1 checkpoints for the common audit."""
+from dataclasses import dataclass
+from pathlib import Path
+
+import torch
+
+
+@dataclass
+class AblationInfo:
+    """
+    Architecture metadata used by the checkpoint audit.
+    """
+    run_name:      str
+    model_class:   str    # config['name']
+    tok:           str    # byte / byte_dynamic / bpe
+    quant:         str    # ternary / fp16
+    is_byte_level: bool
+    is_recurrent:  bool   # True → MLGRU (estado persistente), False → Transformer
+    is_ternary:    bool
+    use_local_dec: bool   # LocalByteDecoder
+    patch_size:    int    # bytes por patch
+    vocab_size:    int    # 256 (byte) / bpe_vocab_size
+    n_heads:       int    # patch_size para byte, 1 para BPE
+    d_model:       int
+    n_layers:      int
+
+    @staticmethod
+    def from_config(run_name: str, config: dict) -> "AblationInfo":
+        tok   = config.get("tokenization", "byte")
+        quant = config.get("quantization", "fp16")
+        mixer = config.get("token_mixer", "mlgru")
+        patch = int(config.get("patch_size", 4))
+        is_byte = tok in ("byte", "byte_dynamic")
+        use_lbd = is_byte and config.get("use_local_decoder", True)
+        vocab = 256 if is_byte else int(config.get("bpe_vocab_size", 16384))
+        # n_heads: siempre patch_size para byte (LocalDecoder se desempaqueta
+        # en _get_logits_list de evaluation.correctness a P tensores [B,S,256])
+        n_heads = patch if is_byte else 1
+        return AblationInfo(
+            run_name      = run_name,
+            model_class   = config.get("name", run_name),
+            tok           = tok,
+            quant         = quant,
+            is_byte_level = is_byte,
+            is_recurrent  = (mixer == "mlgru"),
+            is_ternary    = (quant == "ternary"),
+            use_local_dec = use_lbd,
+            patch_size    = patch,
+            vocab_size    = vocab,
+            n_heads       = n_heads,
+            d_model       = int(config.get("d_model", 512)),
+            n_layers      = int(config.get("n_layers", 12)),
+        )
+
+
+class AblationEngine:
+    """Loaded model and recurrent carry for correctness checks and timing."""
+
+    def __init__(self, model, config: dict, info: AblationInfo):
+        self._model = model
+        self.config = config
+        self.info = info
+        self.hidden_states = None
+
+    def _get_model(self):
+        return self._model
+
+    def reset_state(self):
+        self.hidden_states = None
+
+    def _ablation_step(self, model, dummy: torch.Tensor):
+        result = model(
+            dummy,
+            hidden_states=self.hidden_states if self.info.is_recurrent else None,
+            training_phase=1,
+            target_elimination=0.0,
+        )
+        return result[0], result[1] if self.info.is_recurrent else None
+
+
+def load_run(ckpt_path: Path, run_name: str) -> tuple:
+    """
+    Carga el checkpoint y construye el modelo. Soporta tres formatos:
+
+      compact_2bit  → load_ablation_checkpoint desempaqueta el ternary
+      bf16          → load_ablation_checkpoint castea a fp32
+      fp32          → carga directa desde raw['state_dict']
+                      (formato de cpu1-ablations-final y export_ablations.py)
+
+    Devuelve (engine, raw_ckpt, config, missing_keys, unexpected_keys).
+    """
+    from training.ablation import load_ablation_checkpoint, build_ablation_model
+
+    raw = torch.load(str(ckpt_path), map_location="cpu", weights_only=False)
+
+    fmt = raw.get("format", "bf16")
+    if fmt == "fp32":
+        # Descargado de Cukinator/cpu1-ablations-final o exportado localmente.
+        # Claves: state_dict, config, metrics, step, run_name, format, source_repo
+        config = raw["config"]
+        state  = raw["state_dict"]
+    else:
+        # compact_2bit o bf16: load_ablation_checkpoint lo desempaqueta / castea.
+        state, config = load_ablation_checkpoint(str(ckpt_path))
+
+    config["use_gradient_checkpointing"] = False
+    model = build_ablation_model(config)
+    missing, unexpected = model.load_state_dict(state, strict=True)
+    if missing or unexpected:
+        raise RuntimeError(f"Checkpoint incompatible: missing={missing}, unexpected={unexpected}")
+    model.eval()
+
+    info   = AblationInfo.from_config(run_name, config)
+    engine = AblationEngine(model, config, info)
+    return engine, raw, config, missing, unexpected

evaluation/source/evaluation/correctness.py

@@ -0,0 +1,239 @@
+"""
+Eval — Correctness: verifica propiedades del modelo.
+
+Checks adaptivos segun arquitectura (engine.info):
+  forward_shapes  — Forward pass sin crash para varias shapes
+  probs           — Distribuciones de probabilidad validas
+  determinism     — Mismo input → mismo output
+  state_prop      — Estado MLGRU se propaga correctamente (solo recurrente)
+  ternary_weights — Pesos BitLinear son {-1, 0, 1} (solo ternary)
+  state_size      — Estado MLGRU < 1MB (solo recurrente)
+
+Checks use the model loaded by evaluation.checkpoints.
+"""
+import torch
+
+from evaluation.helpers import get_model, BitLinear
+
+
+def _make_dummy_input(info, batch: int = 1, seq_len: int = 64):
+    """Crear input dummy segun tipo de modelo."""
+    if info.is_byte_level:
+        return torch.randint(0, 256, (batch, seq_len, info.patch_size), dtype=torch.long)
+    else:
+        return torch.randint(0, info.vocab_size, (batch, seq_len), dtype=torch.long)
+
+
+def _unpack_forward(result):
+    """Extraer logits y states de cualquier modelo.
+    CPU1ModelV2: (logits_tuple, states)
+    AblationModel: (logits_or_tuple, states, extra_losses)
+    """
+    if len(result) == 2:
+        return result[0], result[1]
+    else:
+        return result[0], result[1]
+
+
+def _get_logits_list(logits, info):
+    """Normalizar logits a lista de tensores [B, S, V] siempre.
+
+    Casos:
+      - tuple/list de heads (byte sin LocalDecoder): devuelve lista tal cual.
+      - LocalByteDecoder: [B, S, P, 256] → lista de P tensores [B, S, 256].
+      - tensor 3D (BPE o produccion): devuelve lista de 1 elemento.
+    """
+    if isinstance(logits, (tuple, list)):
+        return list(logits)
+    if logits.dim() == 4:
+        # LocalByteDecoder: [B, S, P, 256] → P × [B, S, 256]
+        return [logits[:, :, i, :] for i in range(logits.shape[2])]
+    return [logits]
+
+
+def check_forward_shapes(engine) -> dict:
+    """Forward sin crash para varias combinaciones batch × seq."""
+    info = engine.info
+    model = engine._get_model()
+    configs = [(b, s) for b in [1, 2] for s in [1, 64, 256]]
+    errors = []
+
+    for batch, seq_len in configs:
+        x = _make_dummy_input(info, batch, seq_len)
+        with torch.inference_mode():
+            result = model(x)
+            logits, _ = _unpack_forward(result)
+
+        logits_list = _get_logits_list(logits, info)
+        expected_heads = info.n_heads
+        expected_vocab = info.vocab_size
+
+        if len(logits_list) != expected_heads:
+            errors.append(f"Esperado {expected_heads} heads, obtenido {len(logits_list)}")
+            continue
+
+        for i, l in enumerate(logits_list):
+            expected_shape = (batch, seq_len, expected_vocab)
+            if l.shape != expected_shape:
+                errors.append(f"Head {i}: shape {l.shape} != {expected_shape}")
+            if torch.isnan(l).any():
+                errors.append(f"NaN en logits batch={batch} seq={seq_len} head={i}")
+            if torch.isinf(l).any():
+                errors.append(f"Inf en logits batch={batch} seq={seq_len} head={i}")
+
+    assert not errors, "\n".join(errors)
+    return {'configs_tested': len(configs), 'heads': info.n_heads}
+
+
+def check_probs(engine) -> dict:
+    """Las heads producen distribuciones validas (suman a 1)."""
+    info = engine.info
+    model = engine._get_model()
+    x = _make_dummy_input(info, batch=1, seq_len=128)
+
+    with torch.inference_mode():
+        logits, _ = _unpack_forward(model(x))
+
+    logits_list = _get_logits_list(logits, info)
+    max_err = 0.0
+    for i, head_logits in enumerate(logits_list):
+        probs = torch.softmax(head_logits, dim=-1)
+        sums = probs.sum(dim=-1)
+        err = (sums - 1.0).abs().max().item()
+        max_err = max(max_err, err)
+        assert err < 1e-4, f"Head {i}: suma de probs error={err:.6f}"
+
+    return {'max_error': max_err}
+
+
+def check_determinism(engine) -> dict:
+    """Mismo input con mismo estado da output identico."""
+    info = engine.info
+    model = engine._get_model()
+    torch.manual_seed(42)
+    x = _make_dummy_input(info, batch=1, seq_len=64)
+
+    with torch.inference_mode():
+        logits1, _ = _unpack_forward(model(x, hidden_states=None))
+        logits2, _ = _unpack_forward(model(x, hidden_states=None))
+
+    list1 = _get_logits_list(logits1, info)
+    list2 = _get_logits_list(logits2, info)
+
+    for i in range(len(list1)):
+        diff = (list1[i] - list2[i]).abs().max().item()
+        assert diff == 0.0, f"Head {i}: output no deterministico, diff={diff}"
+
+    return {'deterministic': True}
+
+
+def check_state_propagation(engine) -> dict:
+    """Procesar en chunks da el mismo resultado que todo junto (solo recurrente)."""
+    info = engine.info
+    if not info.is_recurrent:
+        return {'skipped': True, 'reason': 'modelo no recurrente (Transformer)'}
+
+    model = engine._get_model()
+    torch.manual_seed(0)
+    x = _make_dummy_input(info, batch=1, seq_len=128)
+
+    with torch.inference_mode():
+        logits_full, _ = _unpack_forward(model(x))
+
+    states = None
+    chunk_outputs = []
+    with torch.inference_mode():
+        for i in range(0, 128, 32):
+            chunk = x[:, i:i + 32] if x.dim() == 2 else x[:, i:i + 32, :]
+            result = model(chunk, states)
+            logits_chunk, states = _unpack_forward(result)
+            chunk_outputs.append(_get_logits_list(logits_chunk, info))
+
+    full_heads = _get_logits_list(logits_full, info)
+    diff = max((full_heads[k] - torch.cat([chunk[k] for chunk in chunk_outputs], dim=1)).abs().max().item()
+               for k in range(len(full_heads)))
+    assert diff < 1e-4, f"State propagation error: {diff:.6f}"
+    return {'max_diff': diff}
+
+
+def check_ternary_weights(engine) -> dict:
+    """Pesos BitLinear quantizados son {-1, 0, 1} (solo ternary)."""
+    info = engine.info
+    if not info.is_ternary:
+        return {'skipped': True, 'reason': 'modelo FP16 (sin BitLinear)'}
+
+    model = engine._get_model()
+    n_checked = 0
+
+    for name, module in model.named_modules():
+        if not isinstance(module, BitLinear):
+            continue
+
+        if hasattr(module, 'weight_T_ternary'):
+            w = module.weight_T_ternary
+            unique = set(w.unique().tolist())
+            invalid = unique - {-1.0, 0.0, 1.0}
+            assert not invalid, f"{name}: valores no ternarios en weight_T_ternary: {invalid}"
+        elif module.weight.numel() > 0:
+            w = module.weight.data
+            w_scale = w.abs().mean()
+            w_q = (w / (w_scale + 1e-8)).round().clamp(-1, 1)
+            unique = set(w_q.unique().tolist())
+            invalid = unique - {-1.0, 0.0, 1.0}
+            assert not invalid, f"{name}: valores no ternarios: {invalid}"
+
+        n_checked += 1
+
+    assert n_checked > 0, "No se encontraron modulos BitLinear"
+    return {'modules_checked': n_checked}
+
+
+def check_state_size(engine) -> dict:
+    """Estado MLGRU < 1MB (solo recurrente)."""
+    info = engine.info
+    if not info.is_recurrent:
+        return {'skipped': True, 'reason': 'modelo no recurrente (Transformer)'}
+
+    model = engine._get_model()
+    x = _make_dummy_input(info, batch=1, seq_len=1)
+
+    with torch.inference_mode():
+        _, states = _unpack_forward(model(x))
+
+    # Filtrar None states (pueden existir si algun bloque es transformer)
+    valid_states = [s for s in states if s is not None]
+    state_bytes = sum(s.numel() * s.element_size() for s in valid_states)
+    state_kb = state_bytes / 1024
+
+    assert state_kb < 1024, f"Estado MLGRU demasiado grande: {state_kb:.1f} KB"
+    return {'state_kb': round(state_kb, 1), 'batch_size': 1, 'dtype': 'float32'}
+
+
+# =========================================================================
+# Entry point
+# =========================================================================
+
+def run_correctness(engine) -> dict:
+    """Ejecutar todos los checks de correctness. 1 engine → 6 checks."""
+    from evaluation.helpers import run_check
+
+    print(f"\n--- Correctness ({engine.info.model_class}) ---")
+    results = {}
+    checks = [
+        ('forward_shapes',  check_forward_shapes),
+        ('probs',           check_probs),
+        ('determinism',     check_determinism),
+        ('state_prop',      check_state_propagation),
+        ('ternary_weights', check_ternary_weights),
+        ('state_size',      check_state_size),
+    ]
+    for name, fn in checks:
+        r = run_check(name, fn, engine)
+        results[name] = r
+        status = r['status']
+        extra = ''
+        if r.get('result', {}).get('skipped'):
+            extra = f" (skip: {r['result']['reason']})"
+        print(f"  {status:5s}  {name}{extra}")
+
+    return results

evaluation/source/evaluation/diagnostics.py

@@ -0,0 +1,35 @@
+"""Check the recurrent scan against its defining equation without model files."""
+from pathlib import Path
+import sys
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+
+def main():
+    import torch
+    from cpu1.components import AblationMLGRU
+    from evaluation.audit import OUT, save, sha
+    torch.set_num_threads(1)
+    torch.manual_seed(19)
+    c = torch.randn(1, 32, 8)
+    f = torch.sigmoid(torch.randn(1, 32, 8))
+    initial = torch.randn(1, 8)
+    actual, _ = AblationMLGRU._parallel_scan(c, f, initial)
+    h = initial.clone()
+    steps = []
+    for i in range(32):
+        h = f[:, i]*h + (1-f[:, i])*c[:, i]
+        steps.append(h)
+    reference = torch.stack(steps, dim=1)
+    first, carry = AblationMLGRU._parallel_scan(c[:, :16], f[:, :16], initial)
+    second, _ = AblationMLGRU._parallel_scan(c[:, 16:], f[:, 16:], carry)
+    serial_diff = (actual-reference).abs().max().item()
+    chunk_diff = (actual-torch.cat([first, second], dim=1)).abs().max().item()
+    result = {'seed':19, 'shape':[1,32,8], 'torch':torch.__version__, 'dtype':'float32', 'reference':'h[t] = f[t] * h[t-1] + (1-f[t]) * c[t]', 'serial_max_diff':serial_diff, 'chunks_max_diff':chunk_diff, 'tolerance':1e-4, 'status':'PASS' if max(serial_diff,chunk_diff)<1e-4 else 'FAIL', 'components_sha256':sha(ROOT/'cpu1'/'components.py'), 'script_sha256':sha(Path(__file__))}
+    save(OUT/'implementation_diagnostics.json', result)
+    print(result)
+
+
+if __name__ == '__main__':
+    main()

evaluation/source/evaluation/export.py

@@ -0,0 +1,252 @@
+"""
+Desempaqueta los checkpoints compactos (2-bit ternary) del ablation study,
+guarda versiones float32 listas para usar y las sube a HuggingFace.
+
+Repo destino: Cukinator/cpu1-ablations-final
+Modelos locales: models/ablations_final/{run_name}/
+
+Uso:
+  python -m evaluation.export
+  python -m evaluation.export --runs run_02 run_03
+  python -m evaluation.export --skip-upload   # solo exporta localmente
+  python -m evaluation.export --skip-export   # solo sube los ya exportados
+"""
+
+import os
+import sys
+import json
+import time
+import argparse
+from pathlib import Path
+
+ROOT = Path(__file__).parent.parent
+sys.path.insert(0, str(ROOT))
+
+HF_REPO_SRC  = "Cukinator/cpu1-ablation-checkpoints"
+HF_REPO_DEST = "Cukinator/cpu1-ablations-final"
+MODELS_DIR   = ROOT / "local" / "models" / "ablation_models"
+EXPORT_DIR   = ROOT / "local" / "models" / "ablations_final"
+
+AVAILABLE_RUNS = [
+    "run_01",
+    "run_02",
+    "run_02a_byte_only_heads",
+    "run_03",
+    "run_04",
+    "run_05",
+    "run_05b_kernel_strict",
+    "run_06",
+    "run_07",
+    "run_08",
+    "run_09",
+    "run_10",
+    "run_13",
+    "run_14",
+    "run_15",
+    "run_16",
+    # Round-2 re-runs at 15 tok/param
+    "run_04_r2",
+    "run_05_r2",
+    "run_07_r2",
+    "run_08_r2",
+    "run_09_r2",
+    "run_10_r2",
+    "run_13_r2",
+    "run_14_r2",
+    "run_15_r2",
+    "run_16_r2",
+    # Round-3 cold-start rescue (bf16 AMP + lr_scale=2 + CE-only + 50 tok/p)
+    "run_04_r3",
+    "run_07_r3",
+    "run_14_r3",
+    "run_15_r3",
+]
+
+USAGE_README = (Path(__file__).parent.parent / "docs" / "model_cards"
+                / "cpu1-ablations-final.md").read_text(encoding="utf-8")
+
+
+def get_hf_token() -> str | None:
+    token = os.environ.get("HF_TOKEN")
+    if not token:
+        env_path = ROOT / ".env"
+        if env_path.exists():
+            for line in env_path.read_text().splitlines():
+                if line.startswith("HF_TOKEN"):
+                    token = line.split("=", 1)[1].strip().strip('"')
+                    break
+    return token or None
+
+
+def export_run(run_name: str) -> Path | None:
+    """Desempaqueta el checkpoint compacto y guarda un .pt float32."""
+    import torch
+    from training.ablation import load_ablation_checkpoint
+
+    src = MODELS_DIR / run_name / f"checkpoint_{run_name}_final.pt"
+    if not src.exists():
+        print(f"  [missing] {src}")
+        return None
+
+    out_dir = EXPORT_DIR / run_name
+    out_dir.mkdir(parents=True, exist_ok=True)
+    out_path = out_dir / "model.pt"
+
+    if out_path.exists():
+        print(f"  [skip] {run_name} ya exportado ({out_path.stat().st_size / 1e6:.1f} MB)")
+        return out_path
+
+    print(f"  Exportando {run_name}...")
+    t0 = time.perf_counter()
+
+    try:
+        state, config = load_ablation_checkpoint(str(src))
+    except Exception as e:
+        print(f"  [ERROR] load: {e}")
+        return None
+
+    # Métricas del checkpoint original
+    import torch as _torch
+    raw = _torch.load(str(src), map_location="cpu", weights_only=False)
+    metrics = raw.get("metrics", {})
+    step    = raw.get("step", "?")
+
+    payload = {
+        "state_dict": state,
+        "config":     config,
+        "metrics":    metrics,
+        "step":       step,
+        "run_name":   run_name,
+        "format":     "fp32",
+        "source_repo": HF_REPO_SRC,
+    }
+
+    import torch as _torch
+    _torch.save(payload, out_path)
+    elapsed = time.perf_counter() - t0
+    size_mb = out_path.stat().st_size / 1e6
+    print(f"  [ok] {run_name} → {out_path}  ({size_mb:.1f} MB, {elapsed:.1f}s)")
+    return out_path
+
+
+def upload_run(run_name: str, token: str | None) -> bool:
+    """Sube models/ablations_final/{run_name}/ al repo HF."""
+    from huggingface_hub import HfApi
+
+    out_path = EXPORT_DIR / run_name / "model.pt"
+    if not out_path.exists():
+        print(f"  [skip upload] {run_name} — archivo no encontrado")
+        return False
+
+    api = HfApi(token=token)
+
+    print(f"  Subiendo {run_name} ({out_path.stat().st_size / 1e6:.1f} MB)...")
+    t0 = time.perf_counter()
+    try:
+        api.upload_file(
+            path_or_fileobj=str(out_path),
+            path_in_repo=f"{run_name}/model.pt",
+            repo_id=HF_REPO_DEST,
+            repo_type="model",
+            commit_message=f"Add {run_name} (float32, unpacked)",
+        )
+        elapsed = time.perf_counter() - t0
+        print(f"  [ok] {run_name} subido en {elapsed:.1f}s")
+        return True
+    except Exception as e:
+        print(f"  [ERROR] upload {run_name}: {e}")
+        return False
+
+
+def ensure_repo(token: str | None):
+    from huggingface_hub import HfApi
+    api = HfApi(token=token)
+    try:
+        api.repo_info(repo_id=HF_REPO_DEST, repo_type="model")
+        print(f"  Repo existente: {HF_REPO_DEST}")
+    except Exception:
+        print(f"  Creando repo: {HF_REPO_DEST} ...")
+        api.create_repo(
+            repo_id=HF_REPO_DEST,
+            repo_type="model",
+            private=False,
+            exist_ok=True,
+        )
+        print(f"  [ok] Repo creado")
+
+
+def upload_readme(token: str | None):
+    from huggingface_hub import HfApi
+    api = HfApi(token=token)
+    try:
+        import tempfile, os
+        with tempfile.NamedTemporaryFile(mode="w", suffix=".md",
+                                         delete=False, encoding="utf-8") as f:
+            f.write(USAGE_README)
+            tmp = f.name
+        api.upload_file(
+            path_or_fileobj=tmp,
+            path_in_repo="README.md",
+            repo_id=HF_REPO_DEST,
+            repo_type="model",
+            commit_message="Add README",
+        )
+        os.unlink(tmp)
+        print("  [ok] README.md subido")
+    except Exception as e:
+        print(f"  [warn] README upload: {e}")
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Exporta y sube checkpoints ablation desempaquetados"
+    )
+    parser.add_argument("--runs", nargs="+", default=None)
+    parser.add_argument("--skip-upload", action="store_true",
+                        help="Solo exporta localmente, no sube a HF")
+    parser.add_argument("--skip-export", action="store_true",
+                        help="Solo sube los .pt ya exportados (no re-exporta)")
+    args = parser.parse_args()
+
+    runs  = args.runs or AVAILABLE_RUNS
+    token = get_hf_token()
+
+    print(f"\n{'=' * 65}")
+    print(f"  Export ablations -- {HF_REPO_SRC} -> {HF_REPO_DEST}")
+    print(f"  Runs: {runs}")
+    print(f"  HF_TOKEN: {'****' + token[-4:] if token else 'None'}")
+    print(f"  Local export dir: {EXPORT_DIR}")
+    print(f"{'=' * 65}\n")
+
+    EXPORT_DIR.mkdir(parents=True, exist_ok=True)
+
+    # — Fase 1: exportar localmente —
+    if not args.skip_export:
+        print("## Exportando checkpoints (2-bit → float32)\n")
+        for run in runs:
+            export_run(run)
+
+    # — Fase 2: subir a HF —
+    if not args.skip_upload:
+        if not token:
+            print("\n[ERROR] No hay HF_TOKEN — no se puede subir.")
+            return
+
+        print("\n## Subiendo a HuggingFace\n")
+        ensure_repo(token)
+        upload_readme(token)
+
+        ok = 0
+        for run in runs:
+            if upload_run(run, token):
+                ok += 1
+
+        print(f"\n{'=' * 65}")
+        print(f"  Listo — {ok}/{len(runs)} modelos subidos a {HF_REPO_DEST}")
+        print(f"{'=' * 65}\n")
+    else:
+        print(f"\nExportacion local completa → {EXPORT_DIR}")
+
+
+if __name__ == "__main__":
+    main()

evaluation/source/evaluation/helpers.py

@@ -0,0 +1,26 @@
+"""Shared access and result handling for the active checkpoint audit."""
+import os
+import time
+
+os.environ.setdefault('KMP_AFFINITY', 'granularity=fine,compact,1,0')
+os.environ.setdefault('KMP_BLOCKTIME', '0')
+
+from cpu1.components import BitLinear
+
+
+def get_model(engine):
+    return engine._get_model()
+
+
+def run_check(name: str, fn, *args, **kwargs) -> dict:
+    """Ejecutar un check individual con manejo de errores y timing."""
+    try:
+        t0 = time.perf_counter()
+        result = fn(*args, **kwargs)
+        elapsed = time.perf_counter() - t0
+        status = 'N/A' if isinstance(result, dict) and result.get('skipped') else 'PASS'
+        return {'status': status, 'time_s': round(elapsed, 2), 'result': result}
+    except AssertionError as e:
+        return {'status': 'FAIL', 'error': str(e)}
+    except Exception as e:
+        return {'status': 'ERROR', 'error': f'{type(e).__name__}: {e}'}

evaluation/source/evaluation/verify_checkpoints.py

@@ -0,0 +1,64 @@
+"""Verify the two published loading examples, sequentially, in D: storage."""
+import gc
+import hashlib
+import os
+from pathlib import Path
+import sys
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+
+def digest(state):
+    result = hashlib.sha256()
+    for name, value in sorted(state.items()):
+        result.update(name.encode())
+        result.update(str((value.dtype, tuple(value.shape))).encode())
+        result.update(value.contiguous().numpy().tobytes())
+    return result.hexdigest()
+
+
+def main():
+    import json
+    from evaluation.audit import FINAL, ARCHIVE, OUT, save, sha
+    storage = Path('D:/cpu1-readme-audit')
+    assert not list((storage/'active').glob('*/model.pt')), 'Finish the model audit first'
+    os.environ['HF_HOME'] = str(storage/'hf_home')
+    os.environ['HF_HUB_CACHE'] = str(storage/'hf_hub_cache')
+    os.environ['HF_HUB_DISABLE_XET'] = '1'
+    os.environ['HF_HUB_DISABLE_PROGRESS_BARS'] = '1'
+    import torch
+    from huggingface_hub import hf_hub_download
+    from training.ablation import build_ablation_model, load_ablation_checkpoint
+    torch.set_num_threads(1)
+    inventory = json.loads((OUT/'inventory.json').read_text(encoding='utf-8'))
+    records = []
+    for repo, filename in [(ARCHIVE,'run_04/checkpoint_run_04_final.pt'),(FINAL,'run_04/model.pt')]:
+        path = Path(hf_hub_download(repo,filename,revision=inventory[repo]['sha'],local_dir=str(storage/'smoke'),cache_dir=str(storage/'hf_hub_cache')))
+        raw = torch.load(path,map_location='cpu',weights_only=False)
+        fmt = raw.get('format')
+        if repo == FINAL:
+            state, config = raw['state_dict'], raw['config']
+        else:
+            state, config = load_ablation_checkpoint(str(path))
+        fingerprint = digest(state)
+        config['use_gradient_checkpointing'] = False
+        model = build_ablation_model(config)
+        model.load_state_dict(state,strict=True)
+        model.eval()
+        x = torch.tensor(list(b'The history of AI.  ')).reshape(1,-1,4)
+        with torch.inference_mode():
+            logits, states, extra = model(x,training_phase=1,target_elimination=0.)
+        assert all(torch.isfinite(head).all() for head in logits)
+        shapes = [list(head.shape) for head in logits]
+        assert shapes == [[1,5,256]]*4
+        records.append({'repo':repo,'revision':inventory[repo]['sha'],'file':filename,'format':fmt,'sha256':sha(path),'state_sha256':fingerprint,'strict_load':True,'finite_logits':True,'shapes':shapes})
+        print(f'Example verified: {repo} ({fmt})',flush=True)
+        del raw, state, model, logits, states, extra
+        gc.collect()
+        path.unlink()
+    save(OUT/'loading_examples.json',{'examples':records,'run_04_state_dicts_identical':records[0]['state_sha256']==records[1]['state_sha256'],'scope':'only the run_04 pair; does not establish parity across other models or intermediate steps'})
+
+
+if __name__ == '__main__':
+    main()

evaluation/source/requirements/audit.txt

@@ -0,0 +1,2 @@
+-r training.txt
+psutil>=5.9.0

evaluation/source/requirements/data.txt

@@ -0,0 +1,4 @@
+-r training.txt
+psutil>=5.9.0
+tqdm>=4.66.0
+# Optional teacher quantization: install bitsandbytes or torchao when needed.

evaluation/source/requirements/training.txt

@@ -0,0 +1,7 @@
+# Install the PyTorch build for your hardware separately.
+transformers>=4.47.0
+datasets>=3.2.0
+huggingface_hub>=0.27.0
+tokenizers>=0.21.0
+numpy>=1.26.0
+pyarrow>=18.0.0

evaluation/source/tests/__init__.py

@@ -0,0 +1 @@
+"""Offline regression and integration checks."""

evaluation/source/tests/test_audit.py

@@ -0,0 +1,66 @@
+"""Offline regression checks for the scoring protocol, not trained-model quality."""
+import json
+import math
+from pathlib import Path
+import tempfile
+import unittest
+
+import torch
+
+from evaluation.audit import loss_eval, save
+from evaluation.helpers import run_check
+from cpu1.components import LocalByteDecoder
+
+
+class AuditTests(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        torch.set_num_threads(1)
+
+    def test_local_teacher_forcing_does_not_look_ahead(self):
+        torch.manual_seed(4)
+        decoder = LocalByteDecoder(16, 4, 16, 4, use_ternary=False).eval()
+        hidden = torch.randn(1, 2, 16)
+        targets = torch.tensor([[[5, 6, 7, 8], [9, 10, 11, 12]]])
+        future = targets.clone()
+        future[:, :, 2:] += 20
+        with torch.inference_mode():
+            a, b = decoder(hidden, targets), decoder(hidden, future)
+        # Head 2 sees only ground-truth bytes 0 and 1; head 3 sees byte 2.
+        for k in range(3):
+            torch.testing.assert_close(a[k], b[k], rtol=0, atol=0)
+        self.assertGreater((a[3]-b[3]).abs().max().item(), 0)
+
+    def test_loss_uses_shifted_targets_and_target_weighting(self):
+        class Model:
+            def __init__(self):
+                self.seen = []
+
+            def __call__(self, x, byte_targets, **kwargs):
+                self.seen.append(byte_targets.clone())
+                logits = torch.zeros(*byte_targets.shape, 256)
+                # Distinct losses per passage and unequal target counts.
+                logits[..., 0] = len(self.seen)
+                return tuple(logits[:, :, k] for k in range(4)), None, {}
+        model = Model()
+        result = loss_eval(model, {'tokenization':'byte', 'patch_size':4}, {'texts':['abcdefgh','ijklmnopqrst']}, None)
+        self.assertEqual(result['targets'], 12)
+        self.assertEqual(model.seen[0].flatten().tolist(), list(b'efgh'))
+        self.assertEqual(model.seen[1].flatten().tolist(), list(b'mnopqrst'))
+        expected = (4*math.log(255+math.exp(1))+8*math.log(255+math.exp(2)))/12
+        self.assertAlmostEqual(result['mean_nll'], expected, places=5)
+
+    def test_skips_are_not_passes(self):
+        result = run_check('irrelevant', lambda: {'skipped':True,'reason':'not recurrent'})
+        self.assertEqual(result['status'], 'N/A')
+
+    def test_nonfinite_evidence_is_valid_json(self):
+        with tempfile.TemporaryDirectory() as directory:
+            path = Path(directory)/'record.json'
+            save(path, {'loss':float('nan'), 'nested':[float('inf'), 2.]})
+            self.assertEqual(json.loads(path.read_text()), {'loss':None,'nested':[None,2.]})
+
+
+if __name__ == '__main__':
+    torch.set_num_threads(1)
+    unittest.main()

evaluation/source/tests/test_checkpoints.py

@@ -0,0 +1,62 @@
+"""Checkpoint compatibility checks for the public package layout."""
+from pathlib import Path
+import tempfile
+import unittest
+
+import torch
+
+from cpu1.components import pack_model_compact
+from evaluation.checkpoints import load_run
+from training.ablation import build_ablation_model, get_run_config
+
+
+class CheckpointTests(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        torch.set_num_threads(1)
+        cls.storage = Path('D:/cpu1-readme-audit/test-checkpoints')
+        cls.storage.mkdir(parents=True, exist_ok=True)
+
+    def config(self):
+        cfg = get_run_config('run_04')
+        cfg.update(d_model=16, n_layers=1, d_ff=32, n_heads=2,
+                   head_dim=8, window_size=8, local_decoder_dim=16,
+                   local_decoder_heads=2, use_gradient_checkpointing=False)
+        return cfg
+
+    def test_fp32_and_compact_payloads_load_in_new_package(self):
+        config = self.config()
+        model = build_ablation_model(config).eval()
+        x = torch.tensor(list(b'abcdefgh')).reshape(1, 2, 4)
+        for format_name in ('fp32', 'compact_2bit'):
+            with self.subTest(format=format_name):
+                with tempfile.TemporaryDirectory(dir=self.storage) as directory:
+                    payload = {'config': config, 'format': format_name}
+                    if format_name == 'fp32':
+                        payload['state_dict'] = model.state_dict()
+                    else:
+                        payload['compact'] = pack_model_compact(model)
+                    path = Path(directory)/'model.pt'
+                    torch.save(payload, path)
+                    engine, _, loaded_config, missing, extra = load_run(path, 'test')
+                    self.assertEqual(missing, [])
+                    self.assertEqual(extra, [])
+                    self.assertEqual(loaded_config['d_model'], 16)
+                    with torch.inference_mode():
+                        heads = engine._get_model()(x, byte_targets=x)[0]
+                    self.assertEqual([tuple(h.shape) for h in heads], [(1, 2, 256)]*4)
+                    self.assertTrue(all(torch.isfinite(h).all() for h in heads))
+
+    def test_incompatible_payload_is_rejected(self):
+        config = self.config()
+        state = build_ablation_model(config).state_dict()
+        state.pop(next(iter(state)))
+        with tempfile.TemporaryDirectory(dir=self.storage) as directory:
+            path = Path(directory)/'model.pt'
+            torch.save({'config': config, 'format': 'fp32', 'state_dict': state}, path)
+            with self.assertRaises(RuntimeError):
+                load_run(path, 'incompatible')
+
+
+if __name__ == '__main__':
+    unittest.main()

evaluation/source/training/__init__.py

@@ -0,0 +1 @@
+"""Training entry points for the CPU-1 research models."""

evaluation/source/training/final.py

@@ -0,0 +1,686 @@
+"""
+Training entry point for the production CPU-1 model (CPU1ModelV2).
+
+Two modes:
+  1. Phase 1 — distillation training of CPU1ModelV2 (Qwen teacher → CPU-1).
+        python -m training.final --dry-run --total-steps 10
+        python -m training.final --data-path Cukinator/cpu1-dataset
+
+  2. Phase 2 — DeleteGate training of CPU1ModelV2WithGates from a Phase-1 ckpt.
+        python -m training.final --phase2 --phase1-ckpt checkpoint_final.pt
+
+Inference is implemented in `inference/` (C/C++ kernels).
+The full evaluation suite lives in `evaluation/`.
+"""
+
+import os
+import math
+import time
+import argparse
+import contextlib
+import inspect
+from pathlib import Path
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torch import Tensor
+from torch.utils.data import DataLoader
+
+from cpu1.components import (
+    # Constants
+    D_MODEL, N_LAYERS, D_FF, FP_RANK, PFNET_HIDDEN, VOCAB_SIZE, PATCH_SIZE,
+    # Primitives / blocks
+    RMSNorm, BitLinear, BitEmbedding, FPResidual, PFNet,
+    BytePatchEmbeddingV2, BitHead, LocalByteDecoder, CPU1BlockV2,
+    BoundaryPredictor, DeleteGate, EmbeddingAligner,
+    # Training infrastructure
+    DistillationDataset, FineWebStreamingDataset, distillation_loss,
+    get_lr, build_param_groups, save_checkpoint, load_checkpoint,
+)
+
+_FINEWEB_REPO = 'HuggingFaceFW/fineweb'
+_FINEWEB_CONFIG = {
+    'tokenization': 'byte',
+    'seq_len': 5000,
+    'patch_size': PATCH_SIZE,
+    'seed': 42,
+    'training_signal': 'ce_only',
+}
+
+
+# =============================================================================
+# 1. CPU1ModelV2 — production architecture
+# =============================================================================
+
+class CPU1ModelV2(nn.Module):
+    """CPU-1 v2: ternary weights, MatMul-free token mixing, MegaByte byte head.
+
+    Inference state per-layer:
+      - MLGRU h: d_model × 2 bytes (28 KB total at 14 layers × d=1024) → L2
+      - PFNet weights: 14 × 16 KB = 224 KB → L2
+    """
+
+    def __init__(self, config: dict | None = None):
+        super().__init__()
+        config = config or {}
+        d_model = config.get('d_model', D_MODEL)
+        n_layers = config.get('n_layers', N_LAYERS)
+        d_ff = config.get('d_ff', D_FF)
+        fp_rank = config.get('fp_rank', FP_RANK)
+        pfnet_hidden = config.get('pfnet_hidden', PFNET_HIDDEN)
+
+        self.d_model = d_model
+        self.n_layers = n_layers
+
+        self.patch_embedding = BytePatchEmbeddingV2(d_model)
+        self.blocks = nn.ModuleList([
+            CPU1BlockV2(d_model, d_ff, fp_rank, pfnet_hidden) for _ in range(n_layers)
+        ])
+        self.norm = RMSNorm(d_model)
+
+        # Default head: MegaByte LocalByteDecoder (autoregressive intra-patch
+        # chain). Pass legacy_heads=True to load older 4-independent-BitHead
+        # checkpoints (keys byte_heads.{0..3}.*).
+        if config.get('legacy_heads', False):
+            self.byte_heads = nn.ModuleList([
+                BitHead(d_model, VOCAB_SIZE) for _ in range(4)
+            ])
+            self.local_decoder = None
+        else:
+            self.local_decoder = LocalByteDecoder(
+                d_model=d_model,
+                patch_size=PATCH_SIZE,
+                d_local=config.get('local_decoder_dim', 128),
+                n_heads=config.get('local_decoder_heads', 4),
+                ff_ratio=config.get('local_decoder_ff_ratio', 4.0),
+            )
+            self.byte_heads = None
+
+        self.use_gradient_checkpointing = config.get(
+            'use_gradient_checkpointing', False
+        )
+
+    def _forward_global(
+        self,
+        patches: Tensor,
+        hidden_states: list | None = None,
+    ) -> tuple[Tensor, list]:
+        """Embedding → all MLGRU blocks → final norm. Returns (post-norm, states)."""
+        x = self.patch_embedding(patches)
+
+        if hidden_states is None:
+            hidden_states = [None] * len(self.blocks)
+
+        new_states: list = []
+
+        if self.use_gradient_checkpointing and self.training:
+            from torch.utils.checkpoint import checkpoint
+            for i, block in enumerate(self.blocks):
+                def _block_fn(x_in, h_in, _block=block):
+                    return _block(x_in, h_in)
+                x, h_new = checkpoint(_block_fn, x, hidden_states[i],
+                                      use_reentrant=False)
+                new_states.append(h_new)
+        else:
+            for i, block in enumerate(self.blocks):
+                # Touch a layer-i+2 weight pointer to hint at memory prefetch.
+                if i + 2 < len(self.blocks):
+                    _ = self.blocks[i + 2].token_mixer.forget_proj.weight.data_ptr()
+                x, h_new = block(x, hidden_states[i])
+                new_states.append(h_new)
+
+        return self.norm(x), new_states
+
+    def forward(
+        self,
+        patches: Tensor,
+        hidden_states: list | None = None,
+        byte_targets: Tensor | None = None,
+    ) -> tuple[tuple, list]:
+        """Returns (logits_tuple, new_hidden_states).
+
+        byte_targets is required when using LocalByteDecoder (teacher forcing).
+        Falls back to zeros for benchmarks / dry runs.
+        """
+        x, new_states = self._forward_global(patches, hidden_states)
+
+        if self.local_decoder is not None:
+            if byte_targets is None:
+                B, S = x.shape[0], x.shape[1]
+                byte_targets = torch.zeros(
+                    B, S, PATCH_SIZE, dtype=torch.long, device=x.device
+                )
+            logits = self.local_decoder(x, byte_targets)
+        else:
+            logits = tuple(head(x) for head in self.byte_heads)
+
+        return logits, new_states
+
+
+# =============================================================================
+# 2. CPU1ModelV2WithGates — Phase-2 architecture (DeleteGate + optional aux)
+# =============================================================================
+
+class CPU1ModelV2WithGates(nn.Module):
+    """CPU-1 v2 + BoundaryPredictor / DeleteGate / EmbeddingAligner gates.
+
+    Identical to CPU1ModelV2 when all use_* flags are False — i.e. it's a
+    superset that adds the auxiliary modules used in Phase 2 training.
+    """
+
+    def __init__(self, config: dict):
+        super().__init__()
+        d_model = config.get('d_model', D_MODEL)
+        n_layers = config.get('n_layers', N_LAYERS)
+        d_ff = config.get('d_ff', D_FF)
+        fp_rank = config.get('fp_rank', FP_RANK)
+        pfnet_hidden = config.get('pfnet_hidden', PFNET_HIDDEN)
+
+        self.use_boundary = config.get('use_boundary_predictor', False)
+        self.use_delete_gate = config.get('use_delete_gate', False)
+        self.use_emb_align = config.get('use_embedding_alignment', False)
+        self.delete_gate_pos = n_layers // 2
+
+        self.patch_embedding = BytePatchEmbeddingV2(d_model)
+        self.blocks = nn.ModuleList([
+            CPU1BlockV2(d_model, d_ff, fp_rank, pfnet_hidden)
+            for _ in range(n_layers)
+        ])
+        self.norm = RMSNorm(d_model)
+
+        if config.get('legacy_heads', False):
+            self.byte_heads = nn.ModuleList([
+                BitHead(d_model, VOCAB_SIZE) for _ in range(4)
+            ])
+            self.local_decoder = None
+        else:
+            self.local_decoder = LocalByteDecoder(
+                d_model=d_model,
+                patch_size=PATCH_SIZE,
+                d_local=config.get('local_decoder_dim', 128),
+                n_heads=config.get('local_decoder_heads', 4),
+                ff_ratio=config.get('local_decoder_ff_ratio', 4.0),
+            )
+            self.byte_heads = None
+
+        if self.use_boundary:
+            self.boundary_predictor = BoundaryPredictor(embed_dim=8, hidden_dim=32)
+
+        if self.use_delete_gate:
+            self.delete_gate = DeleteGate(d_model=d_model, elimination_rate=0.0)
+
+        if self.use_emb_align:
+            self.embedding_aligner = EmbeddingAligner(
+                student_dim=d_model,
+                teacher_dim=config.get('teacher_dim', 3584),
+            )
+
+        self.use_gradient_checkpointing = config.get(
+            'use_gradient_checkpointing', False
+        )
+
+    def forward(
+        self,
+        patches: Tensor,
+        hidden_states: list | None = None,
+        training_phase: int = 1,
+        target_elimination: float = 0.0,
+        teacher_hidden: Tensor | None = None,
+        teacher_mask: Tensor | None = None,
+        byte_targets: Tensor | None = None,
+        **kwargs,
+    ) -> tuple:
+        """Returns (logits_tuple, new_hidden_states, extra_losses_dict)."""
+        x = self.patch_embedding(patches)
+
+        if hidden_states is None:
+            hidden_states = [None] * len(self.blocks)
+
+        new_states: list = []
+        extra_losses: dict = {}
+
+        if self.use_gradient_checkpointing and self.training:
+            from torch.utils.checkpoint import checkpoint
+            for i, block in enumerate(self.blocks):
+                def _block_fn(x_in, h_in, _block=block):
+                    return _block(x_in, h_in)
+                x, h_new = checkpoint(_block_fn, x, hidden_states[i],
+                                      use_reentrant=False)
+                new_states.append(h_new)
+
+                if (self.use_delete_gate
+                        and i == self.delete_gate_pos
+                        and training_phase == 2):
+                    x, _ = self.delete_gate.forward_train(
+                        x, target_rate=target_elimination
+                    )
+                    extra_losses['delete_gate'] = self.delete_gate.last_elimination_loss
+        else:
+            for i, block in enumerate(self.blocks):
+                if i + 2 < len(self.blocks):
+                    _ = self.blocks[i + 2].token_mixer.forget_proj.weight.data_ptr()
+                x, h_new = block(x, hidden_states[i])
+                new_states.append(h_new)
+
+                if (self.use_delete_gate
+                        and i == self.delete_gate_pos
+                        and training_phase == 2):
+                    x, _ = self.delete_gate.forward_train(
+                        x, target_rate=target_elimination
+                    )
+                    extra_losses['delete_gate'] = self.delete_gate.last_elimination_loss
+
+        x_pre_norm = x
+        x = self.norm(x)
+
+        if self.local_decoder is not None:
+            if byte_targets is None:
+                B, S = x.shape[0], x.shape[1]
+                byte_targets = torch.zeros(
+                    B, S, PATCH_SIZE, dtype=torch.long, device=x.device
+                )
+            logits = self.local_decoder(x, byte_targets)
+        else:
+            logits = tuple(head(x) for head in self.byte_heads)
+
+        if (self.use_emb_align and training_phase == 1
+                and teacher_hidden is not None and teacher_mask is not None):
+            extra_losses['emb_align'] = self.embedding_aligner.alignment_loss(
+                x_pre_norm, teacher_hidden, teacher_mask,
+            )
+
+        return logits, new_states, extra_losses
+
+
+# =============================================================================
+# 3. Phase 1 training — distillation against a first-byte teacher
+# =============================================================================
+
+def train(args: argparse.Namespace):
+    """Phase 1 training loop with multi-GPU DataParallel."""
+    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+    n_gpus = torch.cuda.device_count() if device.type == 'cuda' else 0
+
+    if device.type == 'cuda':
+        torch.backends.cudnn.benchmark = True
+        torch.set_float32_matmul_precision('high')
+        torch.backends.cuda.matmul.allow_tf32 = True
+        torch.backends.cudnn.allow_tf32 = True
+
+    print(f"Device: {device} ({n_gpus} GPU{'s' if n_gpus != 1 else ''})")
+
+    model = CPU1ModelV2({
+        'use_gradient_checkpointing': args.gradient_checkpointing,
+    }).to(device)
+
+    n_params = sum(p.numel() for p in model.parameters())
+    print(f"Parámetros: {n_params:,}")
+
+    if n_gpus > 1:
+        model = torch.nn.DataParallel(model)
+        print(f"[Multi-GPU] DataParallel on {n_gpus} GPUs")
+
+    param_groups = build_param_groups(model)
+    optimizer = torch.optim.AdamW(
+        param_groups,
+        lr=args.max_lr,
+        betas=(0.9, 0.95),
+        eps=1e-8,
+    )
+
+    use_amp = device.type == 'cuda'
+    scaler = torch.amp.GradScaler(device.type, enabled=use_amp)
+
+    if args.dry_run:
+        dataset = DistillationDataset(
+            dry_run_samples=args.dry_run_samples,
+            seq_len=args.seq_len,
+        )
+    elif args.data_path is None or args.data_path == _FINEWEB_REPO:
+        fw_cfg = {**_FINEWEB_CONFIG, 'seq_len': args.seq_len}
+        print(f"[fineweb] Streaming HuggingFaceFW/fineweb sample-10BT (seq_len={args.seq_len})")
+        dataset = FineWebStreamingDataset(fw_cfg, split='train')
+    else:
+        dataset = DistillationDataset(
+            data_path=args.data_path,
+            seq_len=args.seq_len,
+        )
+
+    use_workers = args.num_workers > 0 and not args.dry_run
+    total_batch = args.batch_per_gpu * max(n_gpus, 1)
+    dataloader = DataLoader(
+        dataset,
+        batch_size=total_batch,
+        shuffle=True,
+        num_workers=args.num_workers if not args.dry_run else 0,
+        pin_memory=(device.type == 'cuda'),
+        drop_last=False,
+        persistent_workers=use_workers,
+        prefetch_factor=2 if use_workers else None,
+    )
+
+    start_step = 0
+    if args.resume_from:
+        model_for_load = model.module if hasattr(model, 'module') else model
+        start_step = load_checkpoint(args.resume_from, model_for_load, optimizer, scaler)
+        print(f"Resumiendo desde step {start_step}")
+
+    grad_accum = args.gradient_accumulation_steps
+    total_steps = args.total_steps
+    effective_batch = total_batch * grad_accum
+
+    print(f"Batch efectivo: {effective_batch} "
+          f"({args.batch_per_gpu}/gpu × {max(n_gpus, 1)} GPUs × {grad_accum} accum)")
+    print(f"Steps: {start_step} → {total_steps}")
+    print(f"Seq len: {args.seq_len} patches = {args.seq_len * PATCH_SIZE} bytes")
+
+    model.train()
+    data_iter = iter(dataloader)
+    epoch = 0
+    log_timer = time.time()
+    metrics: dict[str, float] = {}
+    last_loss = 0.0
+
+    for step in range(start_step, total_steps):
+        lr = get_lr(step, args.warmup_steps, total_steps, args.max_lr, args.min_lr)
+        for pg in optimizer.param_groups:
+            pg['lr'] = lr * pg.get('lr_scale', 1.0)
+
+        optimizer.zero_grad(set_to_none=True)
+        should_log = (step % args.log_every == 0)
+
+        for micro_step in range(grad_accum):
+            try:
+                batch = next(data_iter)
+            except StopIteration:
+                epoch += 1
+                data_iter = iter(dataloader)
+                batch = next(data_iter)
+
+            patches = batch['patches'].to(device, non_blocking=True)
+            targets = batch['targets'].to(device, non_blocking=True)
+            B, S = patches.shape[0], patches.shape[1]
+            _tp = batch.get('teacher_probs')
+            _tm = batch.get('teacher_mask')
+            teacher_probs = (
+                _tp.to(device, non_blocking=True) if _tp is not None
+                else torch.zeros(B, S, VOCAB_SIZE, device=device)
+            )
+            teacher_mask = (
+                _tm.to(device, non_blocking=True) if _tm is not None
+                else torch.zeros(B, S, dtype=torch.bool, device=device)
+            )
+            teacher_hidden = batch.get('teacher_hidden')
+            if teacher_hidden is not None:
+                teacher_hidden = teacher_hidden.to(device, non_blocking=True)
+
+            is_last_micro = (micro_step == grad_accum - 1)
+            amp_ctx = (torch.amp.autocast('cuda', dtype=torch.float16)
+                       if use_amp else contextlib.nullcontext())
+            with amp_ctx:
+                # Detect WithGates models that take teacher_hidden in their forward
+                fwd = model.module.forward if hasattr(model, 'module') else model.forward
+                sig_params = inspect.signature(fwd).parameters
+                if 'teacher_hidden' in sig_params:
+                    logits, _, _ = model(
+                        patches, byte_targets=targets,
+                        teacher_hidden=teacher_hidden, teacher_mask=teacher_mask,
+                    )
+                else:
+                    logits, _ = model(patches, byte_targets=targets)
+                loss, metrics = distillation_loss(
+                    logits, targets, teacher_probs, teacher_mask,
+                    temperature=args.temperature,
+                    compute_metrics=(should_log and is_last_micro),
+                )
+                if is_last_micro:
+                    last_loss = loss.item()
+                loss = loss / grad_accum
+
+            scaler.scale(loss).backward()
+
+        scaler.unscale_(optimizer)
+        grad_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
+
+        scaler.step(optimizer)
+        scaler.update()
+
+        if should_log:
+            elapsed = time.time() - log_timer
+            steps_since_log = max(args.log_every, 1)
+            patches_per_sec = (
+                total_batch * args.seq_len * grad_accum * steps_since_log
+            ) / max(elapsed, 1e-6)
+
+            parts = [
+                f"Step {step:6d}/{total_steps}",
+                f"Loss: {metrics.get('loss_total', 0):.4f}",
+                f"H0-KL: {metrics.get('loss_head0_kl', 0):.4f}",
+                f"H0-CE: {metrics.get('loss_head0_ce', 0):.4f}",
+                f"H123: {metrics.get('loss_heads_123_ce', 0):.4f}",
+                f"Tcov: {metrics.get('teacher_coverage', 0):.1%}",
+                f"‖g‖: {grad_norm:.2f}",
+                f"LR: {lr:.2e}",
+                f"P/s: {patches_per_sec:.0f}",
+            ]
+            if use_amp:
+                gpu_gb = torch.cuda.max_memory_allocated(device) / (1024 ** 3)
+                parts.append(f"VRAM: {gpu_gb:.1f}G")
+
+            print(" | ".join(parts))
+            log_timer = time.time()
+
+        if step > 0 and step % args.save_every == 0:
+            ckpt_path = str(Path(args.output_dir) / f"checkpoint_{step}.pt")
+            save_checkpoint(
+                model, optimizer, scaler, step,
+                last_loss,
+                ckpt_path,
+                upload_to_hf=args.upload_hf,
+            )
+            print(f"  Checkpoint: {ckpt_path}")
+
+    ckpt_path = str(Path(args.output_dir) / "checkpoint_final.pt")
+    save_checkpoint(
+        model, optimizer, scaler, total_steps,
+        last_loss,
+        ckpt_path,
+        upload_to_hf=args.upload_hf,
+    )
+    print(f"Training completo. Final: {ckpt_path}")
+
+
+# =============================================================================
+# 4. Phase 2 training — DeleteGate elimination on top of a Phase-1 checkpoint
+# =============================================================================
+
+def train_phase2(
+    model,
+    dataset,
+    config: dict,
+    checkpoint_phase1: str,
+    target_elimination: float = 0.4,
+    total_steps: int = 5000,
+    lr: float = 1e-4,
+    device: torch.device | None = None,
+) -> dict:
+    """Phase 2: load Phase-1 weights, activate DeleteGate, freeze EmbAligner,
+    and train with CE + elimination regularization (no KL — sequence shortens).
+    """
+    if device is None:
+        device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+
+    ckpt = torch.load(checkpoint_phase1, map_location='cpu', weights_only=False)
+    model.load_state_dict(ckpt['model_state_dict'], strict=False)
+    model = model.to(device)
+
+    if hasattr(model, 'delete_gate'):
+        model.delete_gate.set_elimination_rate(target_elimination)
+
+    if hasattr(model, 'embedding_aligner'):
+        for p in model.embedding_aligner.parameters():
+            p.requires_grad = False
+
+    n_gpus = torch.cuda.device_count() if device.type == 'cuda' else 0
+    if n_gpus > 1:
+        model = torch.nn.DataParallel(model)
+        print(f"[Phase2 Multi-GPU] DataParallel on {n_gpus} GPUs")
+
+    def phase2_loss(logits, targets, extra_losses):
+        ce = sum(
+            F.cross_entropy(
+                logits[i].reshape(-1, 256),
+                targets[:, :, i].long().reshape(-1)
+            ) for i in range(4)
+        ) / 4
+
+        elim_loss = extra_losses.get(
+            'delete_gate', torch.tensor(0.0, device=device)
+        )
+        if elim_loss.dim() > 0:
+            elim_loss = elim_loss.mean()
+
+        return ce + 0.1 * elim_loss
+
+    model_params = model.module if hasattr(model, 'module') else model
+    optimizer = torch.optim.AdamW(
+        [p for p in model_params.parameters() if p.requires_grad],
+        lr=lr, betas=(0.9, 0.95),
+    )
+
+    use_amp = device.type == 'cuda'
+    scaler = torch.amp.GradScaler() if use_amp else None
+
+    total_batch = 4 * max(n_gpus, 1)
+    dataloader = torch.utils.data.DataLoader(
+        dataset, batch_size=total_batch, shuffle=True, drop_last=True,
+    )
+
+    model.train()
+    step = 0
+    t_start = time.time()
+
+    while step < total_steps:
+        for batch in dataloader:
+            if step >= total_steps:
+                break
+
+            patches = batch['patches'].to(device, non_blocking=True)
+            targets = batch['targets'].to(device, non_blocking=True)
+
+            amp_ctx = (torch.amp.autocast(device_type='cuda', dtype=torch.float16)
+                       if use_amp else contextlib.nullcontext())
+            with amp_ctx:
+                logits, _, extra_losses = model(
+                    patches, training_phase=2,
+                    target_elimination=target_elimination,
+                )
+                loss = phase2_loss(logits, targets, extra_losses)
+
+            optimizer.zero_grad(set_to_none=True)
+            if use_amp:
+                scaler.scale(loss).backward()
+                scaler.unscale_(optimizer)
+                torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
+                scaler.step(optimizer)
+                scaler.update()
+            else:
+                loss.backward()
+                torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
+                optimizer.step()
+
+            step += 1
+            if step % 500 == 0:
+                elapsed = time.time() - t_start
+                print(f"[Phase2] Step {step}/{total_steps} | "
+                      f"Loss: {loss.item():.4f} | Time: {elapsed:.0f}s")
+
+    return {'phase': 2, 'target_elimination': target_elimination, 'steps': step}
+
+
+# =============================================================================
+# 5. CLI
+# =============================================================================
+
+def parse_args() -> argparse.Namespace:
+    p = argparse.ArgumentParser(description="Training CPU-1 con distillation loss")
+
+    p.add_argument('--data-path', type=str, default=None,
+                   help="Ruta a HF dataset, directorio, o .parquet. "
+                        "Por defecto usa HuggingFaceFW/fineweb (CE-only, sin teacher).")
+    p.add_argument('--dry-run', action='store_true',
+                   help="Usar datos aleatorios (testing)")
+    p.add_argument('--dry-run-samples', type=int, default=1000)
+    p.add_argument('--seq-len', type=int, default=5000,
+                   help="Patches por secuencia")
+
+    p.add_argument('--batch-per-gpu', type=int, default=4)
+    p.add_argument('--gradient-accumulation-steps', type=int, default=2)
+    p.add_argument('--total-steps', type=int, default=100_000)
+    p.add_argument('--warmup-steps', type=int, default=2000)
+    p.add_argument('--max-lr', type=float, default=3e-4)
+    p.add_argument('--min-lr', type=float, default=3e-5)
+    p.add_argument('--temperature', type=float, default=4.0)
+    p.add_argument('--gradient-checkpointing', action='store_true')
+
+    p.add_argument('--output-dir', type=str, default='./checkpoints')
+    p.add_argument('--save-every', type=int, default=500)
+    p.add_argument('--log-every', type=int, default=100)
+    p.add_argument('--upload-hf', action='store_true',
+                   help="Subir checkpoints a Cukinator/cpu1-checkpoints")
+
+    p.add_argument('--resume-from', type=str, default=None,
+                   help="Ruta a checkpoint .pt para resumir")
+    p.add_argument('--num-workers', type=int, default=4)
+
+    # Phase 2 controls
+    p.add_argument('--phase2', action='store_true',
+                   help="Run DeleteGate Phase 2 starting from --phase1-ckpt")
+    p.add_argument('--phase1-ckpt', type=str, default=None,
+                   help="Phase 1 checkpoint to start Phase 2 from")
+    p.add_argument('--target-elimination', type=float, default=0.4,
+                   help="Phase 2 fraction of states to eliminate")
+
+    return p.parse_args()
+
+
+if __name__ == "__main__":
+    args = parse_args()
+
+    if args.phase2:
+        if not args.phase1_ckpt:
+            raise SystemExit("--phase2 requires --phase1-ckpt")
+
+        device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+        cfg = {
+            'use_delete_gate': True,
+            'use_gradient_checkpointing': args.gradient_checkpointing,
+        }
+        model = CPU1ModelV2WithGates(cfg)
+
+        if args.dry_run:
+            dataset = DistillationDataset(
+                dry_run_samples=args.dry_run_samples,
+                seq_len=args.seq_len,
+            )
+        else:
+            dataset = DistillationDataset(
+                data_path=args.data_path,
+                seq_len=args.seq_len,
+            )
+
+        result = train_phase2(
+            model=model,
+            dataset=dataset,
+            config=cfg,
+            checkpoint_phase1=args.phase1_ckpt,
+            target_elimination=args.target_elimination,
+            total_steps=args.total_steps,
+            lr=args.max_lr,
+            device=device,
+        )
+        print(f"[Phase2] Done: {result}")
+    else:
+        os.makedirs(args.output_dir, exist_ok=True)
+        train(args)