Molt 660M — Molt Bien Lire les Textes, distilled (molt-blt-660m)

Molt 660M is a smaller, distilled version of Molt BLT-1B, itself Byte Latent Transformer (BLT) 1B adapted to medieval and early-modern texts (Old French, Middle French, Latin, Greek, Arabic, Old Spanish, Catalan, plus modern anchors). It reads raw bytes (no tokenizer, no out-of-vocabulary problem on scribal abbreviations, ligatures, ⁊ ꝯ q̃, combining diacritics or HTR noise) and groups them into dynamic patches with the retrained entropy patcher of Molt.

The global transformer is pruned from 25 layers / 2048 wide to 24 layers / 1280 wide and recovered by distillation, so the part that is computed has 660 M parameters instead of ~1 B. Caveat: the 6 hash n-gram embedding tables of BLT (3.07 B parameters, lookups only) are kept as they are, so the files are large (≈ 7.4 GB); the saving is in compute and activations, not in disk size.

file	content
model.safetensors, config.json	BltForCausalLM (transformers), bf16; the patcher is included as model.patcher.*
patcher.pt	the fine-tuned entropy patcher alone (fp32 state dict of BltPatcher, 100 M parameters), the same as in Molt BLT-1B
molt.py	small library: loading, threshold lookup, generate, bits_per_byte
patching_thresholds.json	entropy thresholds per language/source giving ~4 bytes per patch

Recipe (short)

Student. Minitron-style pruning of Molt BLT-1B, importance measured on calibration windows of the training data: the 24 global layers kept are those whose output differs most from their input, the residual channels (2048 → 1280), the attention heads (16 → 10, head size 128) and the MLP neurons (→ 3520) with the highest activation are kept. The local encoder and decoder (1024 wide) are not pruned; a linear map (1280 → 2048) is added between the pruned global transformer and the encoder output.

Distillation. Teacher and student see the same bytes and the same patch boundaries (the fine-tuned fp32 patcher). Loss = KL(teacher ‖ student) over the next-byte distribution

• 0.1 × cross-entropy on the true byte + 0.5 × (1 − cos) on the decoder outputs + 0.5 × (1 − cos) on the global-transformer outputs (through a learned 1280 → 2048 map). Hash tables frozen and shared with the teacher. fp32 master weights, bf16 autocast, AdamW (lr 2e-4, cosine to 10 %), 262 144 bytes per step, 11131 steps (≈ 2.9 B bytes), window-length ramp 512 → 1024 → 2048 → 4096, on 2 × H100. Text is NFD-normalised, HTR text carries the input-only {HTR} tag, exactly as in Molt BLT-1B.

Results (validation bits per byte, lower is better)

The student against its teacher on the same 512-byte windows, and on 4096-byte windows:

stratum	teacher (Molt BLT-1B)	student, 512 B windows	student, 4096 B windows
Old French (edited)	1.43	1.42	1.29
Old French (HTR)	1.86	1.85	1.76
Middle French (HTR)	1.89	1.89	1.87
Latin (edited)	1.57	1.58	1.23
Latin (HTR)	1.85	1.83	1.72
Greek	0.77	0.76	0.65
Arabic	0.94	0.94	0.80
Old Spanish	1.27	1.26	1.13
Catalan	2.04	2.04	1.83
French	1.34	1.35	1.19
Spanish	1.58	1.59	1.43
English	1.39	1.45	1.33

The distilled model matches its teacher on the historical languages it was built for, and loses a little on modern English (the pruned model has less spare capacity for the anchor languages).

Usage

molt.py (in this repository) wraps loading, the per-language/source patching threshold, scoring and sampling. It needs torch, transformers (with BLT) and huggingface_hub.

from huggingface_hub import hf_hub_download
import importlib.util, sys
spec = importlib.util.spec_from_file_location("molt", hf_hub_download("almanach/molt-blt-660m", "molt.py"))
molt = importlib.util.module_from_spec(spec); sys.modules["molt"] = molt; spec.loader.exec_module(molt)

m = molt.Molt.from_pretrained("almanach/molt-blt-660m")                          # model + fp32 patcher + thresholds
m.thresholds()                                                   # {'fro[HTR]': ..., 'fro': ..., 'lat': ..., 'grc': ..., ...}
m.generate("Saintz Martins ", lang="fro", htr=True, n=300)       # htr=True: manuscript-style transcription
m.generate("Saintz Martins ", lang="fro", htr=False, n=300)      # edited, normalised spelling
m.bits_per_byte("Li rois Artus se leva .", lang="fro")           # scoring, e.g. to rank HTR hypotheses

Or from the command line: python molt.py "Saintz Martins " --lang fro --htr -n 300 --model almanach/molt-blt-660m.

How it works: BLT has no tokenizer. The text is NFD-normalised and cut into bytes (ids = [1] + [byte + 4]), the patcher gives the next-byte entropy, and a new patch starts wherever it exceeds the threshold for that language and source (patching_thresholds.json, ft entry; unknown languages fall back to global_threshold_ft). The context is the last 511 bytes; {HTR} is put in front of the text when htr=True.

Limitations

• Same use as Molt BLT-1B: modelling and scoring of historical text (post-correction of HTR, language identification, normalisation research); free generation is loose at the sentence level and can loop on frequent words.
• The thresholds were calibrated on our data and may be suboptimal on other scripts or hands.
• The licence of the base model (BLT, FAIR Noncommercial Research License) applies to the weights.

Author

Thibault Clérice (ALMAnaCH, Inria).

Funding

This work was funded by the ANR (Agence Nationale de la Recherche) under grant ANR-26-CE38-0179-01, project ROMAM².

Citation

@misc{molt-blt-660m,
  title  = {Molt Bien Lire les Textes: a distilled byte-level model for medieval texts},
  author = {Cl\'erice, Thibault},
  year   = {2026},
  url    = {https://huggingface.co/almanach/molt-blt-660m}
}

Based on: Pagnoni et al., Byte Latent Transformer: Patches Scale Better Than Tokens, 2024.