full-128k-apertus-mathcode-scratch
A 1.29B parameter model trained with the apertus tokenizer. Regime: mathcode-20B-from-scratch (24 layers, width 1536, 1.27B parameters, 20.0B tokens, math and code mixture, trained from scratch). Reported in: math and code ablations.
Load
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"cmeister/tokenizer-lm-ablations", subfolder="models/full-128k-apertus-mathcode-scratch",
trust_remote_code=True, dtype="float32")
tokenizer = AutoTokenizer.from_pretrained("swiss-ai/Apertus-70B-2509") # not redistributed here
ids = [1] + tokenizer(text)["input_ids"] # training prepends BOS
The tokenizer is not redistributed here. The papers call this tokenizer Mistral-Nemo: the Apertus and Mistral-Nemo tokenizers are the same artifact. These runs record the local mirror directory rather than the hub id.
Settings
| Setting | Value |
|---|---|
| layers | 24 |
| model dimension | 1536 |
| attention heads | 12 |
| key and value heads | 12 |
| context length | 2048 |
| sliding window pattern | SSSL |
| value embedding dimension | 128 |
| logit soft cap | 15 |
| vocabulary size | 131072 |
| parameters | 1,285,818,122 |
| training step | 19073 |
| token budget | 19,999,490,048 |
| bytes consumed | 133,241,384,072 |
| seed | 42 |
| Muon matrix learning rate | 0.0200 |
| AdamW embedding learning rate | 0.3000 |
| AdamW unembedding learning rate | 0.0080 |
| weight decay | 0.2800 |
| width learning-rate exponent | -1 |
| weight dtype | as trained (float32 matrices, bfloat16 embeddings) |
| BOS token id | 1 |
Results
| Metric | Value |
|---|---|
| validation BPB | 0.3368 |
| code BPB, mean over 7 languages | 0.3782 |
| FLORES-200 BPB, mean | 3.4253 |
| FLORES-200 languages scored | 214 |
| BLiMP accuracy | n/a |
| MBPP pass@1 | 0 |
| HumanEval pass@1 | 0.1524 |
| GSM8K exact match, strict | 0.1842 |
Generation metrics use specification {'version': 'v2-2026-07-30', 'bos_context': True, 'eos_stop': True, 'token_healing': True, 'truncation': 'left_to_max_length_minus_max_gen', 'stop_truncation': 'earliest_match_all_terms', 'stop_match_decode': 'specials_kept'}, token healing on.
Missing measurements, with the reason each is absent:
blimp_acc: no full-128k-apertus-mathcode-scratch_blimp_code_bpb.json
BLiMP is comparable within a training scale only. Its scoring sums unnormalized sentence log-likelihoods, and under that scoring a group of quantifier subtasks drops at the 1.27B scale relative to the 300M scale.
Registry note
#4, apertus pretok, slug apertus. Mistral-Nemo regex; see EXPERIMENTS_RESULTS.md §3a for the RCA of the apertus-pretok MBPP code-gen breakage (real, reproducible effect, not an artifact).
Provenance
- Source checkpoint:
/capstor/store/cscs/swissai/a139/cmeister/tokenizer-lm/checkpoints/full-128k-apertus-mathcode-scratch, step 19073 - Repository commit:
1a9a718 - nanochat commit:
f919449on branchtokenizer-lm-patches - Tokenizer:
apertus, not redistributed