models/full-128k-apertus-mathcode-scratch/README.md

# `full-128k-apertus-mathcode-scratch`

A 1.29B parameter model trained with the `apertus` tokenizer. Regime: mathcode-20B-from-scratch (24 layers, width 1536, 1.27B parameters, 20.0B tokens, math and code mixture, trained from scratch). Reported in: math and code ablations.

## Load

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "cmeister/tokenizer-lm-ablations", subfolder="models/full-128k-apertus-mathcode-scratch",
    trust_remote_code=True, dtype="float32")
tokenizer = AutoTokenizer.from_pretrained("swiss-ai/Apertus-70B-2509")   # not redistributed here

ids = [1] + tokenizer(text)["input_ids"]   # training prepends BOS
```

The tokenizer is not redistributed here. The papers call this tokenizer Mistral-Nemo: the Apertus and Mistral-Nemo tokenizers are the same artifact. These runs record the local mirror directory rather than the hub id.

## Settings

| Setting | Value |
|---|---|
| layers | 24 |
| model dimension | 1536 |
| attention heads | 12 |
| key and value heads | 12 |
| context length | 2048 |
| sliding window pattern | SSSL |
| value embedding dimension | 128 |
| logit soft cap | 15 |
| vocabulary size | 131072 |
| parameters | 1,285,818,122 |
| training step | 19073 |
| token budget | 19,999,490,048 |
| bytes consumed | 133,241,384,072 |
| seed | 42 |
| Muon matrix learning rate | 0.0200 |
| AdamW embedding learning rate | 0.3000 |
| AdamW unembedding learning rate | 0.0080 |
| weight decay | 0.2800 |
| width learning-rate exponent | -1 |
| weight dtype | as trained (float32 matrices, bfloat16 embeddings) |
| BOS token id | 1 |

## Results

| Metric | Value |
|---|---|
| validation BPB | 0.3368 |
| code BPB, mean over 7 languages | 0.3782 |
| FLORES-200 BPB, mean | 3.4253 |
| FLORES-200 languages scored | 214 |
| BLiMP accuracy | n/a |
| MBPP pass@1 | 0 |
| HumanEval pass@1 | 0.1524 |
| GSM8K exact match, strict | 0.1842 |

Generation metrics use specification `{'version': 'v2-2026-07-30', 'bos_context': True, 'eos_stop': True, 'token_healing': True, 'truncation': 'left_to_max_length_minus_max_gen', 'stop_truncation': 'earliest_match_all_terms', 'stop_match_decode': 'specials_kept'}`, token healing on.

Missing measurements, with the reason each is absent:

- `blimp_acc`: no full-128k-apertus-mathcode-scratch_blimp_code_bpb.json

BLiMP is comparable within a training scale only. Its scoring sums unnormalized sentence log-likelihoods, and under that scoring a group of quantifier subtasks drops at the 1.27B scale relative to the 300M scale.

## Registry note

#4, apertus pretok, slug `apertus`. Mistral-Nemo regex; see EXPERIMENTS_RESULTS.md §3a for the RCA of the apertus-pretok MBPP code-gen breakage (real, reproducible effect, not an artifact).

## Provenance

- Source checkpoint: `/capstor/store/cscs/swissai/a139/cmeister/tokenizer-lm/checkpoints/full-128k-apertus-mathcode-scratch`, step 19073
- Repository commit: `1a9a718`
- nanochat commit: `f919449` on branch `tokenizer-lm-patches`
- Tokenizer: `apertus`, not redistributed