Fractale-400M-base

⏳ Campaign in progress — evaluation results are uploaded when the campaign is adjudicated; the metric tables below are placeholders until then.

ℹ️ Control arm, not matched to the released memory arm. This checkpoint is the memory-free control of the campaign recipe. The released memory arm is lines-unif, trained on line-broken recall lives — so the two differ in training data, not only in the presence of a bank. The pre-registered, exactly matched ON/OFF contrast is the one reported in the paper; treat any ON/OFF difference computed from these two files as indicative only.

Fractale-400M-base is the memory bank OFF (control) arm of the FRACTALE memory-bank scaling campaign: three model sizes × (memory bank ON / OFF), trained from scratch under a frozen, pre-registered protocol to measure the scaling law of the effective context bought by a self-written in-model memory bank — performance as a curve against the distance between a probe and its evicted source, bank ON vs the matched memory-free control, at three scales. A long-context twin (W + ΔW) is specified alongside as a deferred follow-up pre-registration.

The matched control: same trunk, same data, same budget, mem_read_mode: none — no bank module is built (zero memory keys in the state_dict). Context window W = 512 tokens, nothing carries across windows.

Architecture

Total parameters	403.9M
Active parameters	106.6M (fine-grained MoE: 64 experts, top-10 + 2 shared)
Trunk geometry	d512, 14 layers, ff 256
Attention	dense causal multi-head attention, 8 heads × 64; RoPE (θ = 10,000) on queries and keys, QK-norm, learned attention sink
Residual stream	hyper-connections (mHC): 2 streams mixed by a doubly stochastic matrix
Memory bank	none (mem_read_mode: none)
Context window (training)	512 tokens
Tokenizer	HuggingFaceTB/SmolLM2-135M + <reset>, <blank> (vocabulary 49,154)
Training budget	8.08B tokens (20.0 tokens per total parameter)
Optimizer	Muon (trunk matrices) + AdamW, WSD schedule, 2 stages

This is a custom PyTorch architecture (not transformers-compatible). Checkpoints are plain state_dict files (model.pt) loaded with the inference kit: https://github.com/fractale-lm/fractale-sdk.

The training stack is not published: what is open is the pre-registered protocol, these weights, the evaluation battery and the paper. Method and results: [preprint — DOI to be added at release].

Training data and licenses

Training ran in two stages. The corpus mixture, made of public Hugging Face datasets, is the same for every checkpoint in this collection: both arms, all three sizes.

• Stage 1 (warmup and stable phase): HuggingFaceFW/fineweb-edu and HuggingFaceFW/fineweb (sample-100BT), mlfoundations/dclm-baseline-1.0, bigcode/starcoderdata (Python, C, Rust, JavaScript), HuggingFaceTB/finemath (finemath-4plus, infiwebmath-4plus), wikimedia/wikipedia (20231101.en), HuggingFaceTB/smollm-corpus (cosmopedia-v2).
• Stage 2 (decay): the stage-1 corpora without HuggingFaceFW/fineweb, DCLM and the Rust split, plus instruction and chat data: HuggingFaceTB/smoltalk (subsets metamathqa-50k, numina-cot-100k, apigen-80k, systemchats-30k) and HuggingFaceTB/smol-smoltalk. The decay stage also mixes in synthetic recall documents, generated from templates by the campaign code (no external data); the -mem checkpoints see them broken into lines.

This checkpoint saw the following upstream sources during its decay stage. Two decay-stage datasets redistribute data whose original cards restrict use to research or to non-commercial purposes, or declare no license:

Decay-stage dataset (weight in the corpus mixture)	Upstream source	Terms on the upstream card	Volume in the dataset
HuggingFaceTB/smol-smoltalk (0.13)	Wanfq/Explore_Instruct_Rewriting_32k	CC-BY-NC-4.0; "intended and licensed for research use only", and "models trained using the dataset should not be used outside of research purposes"	3,017 conversations
	teknium/OpenHermes-2.5	no license declared	47,492 conversations
	THUDM/LongAlign-10k	no license declared	3,560 conversations
HuggingFaceTB/smoltalk, subset apigen-80k (0.06)	Salesforce/xlam-function-calling-60k	CC-BY-4.0; "This release is for research purposes only in support of an academic paper" (gated)	~55% of the rows

That is about 12% of the smol-smoltalk conversations and about 55% of the apigen-80k rows: apigen-80k is sampled from argilla/apigen-function-calling, which mixes argilla/Synth-APIGen-v0.1 (Apache-2.0, 49k rows) with xlam-function-calling-60k (60k rows) and records no per-row origin. The smol-smoltalk card declares Apache-2.0; the SmolTalk card it derives from licenses its own new data under Apache-2.0 and refers to the original datasets for the public data it includes. Stage 1 contains none of these upstream sources. The other datasets listed above are distributed under the terms of their own cards.

The weights are released under the MIT license; commercial use calls for your own assessment of the licenses above.

Evaluation — you can re-run it yourself

The memory claim is measurable, and the kit ships the judge. No corpus and no extra download: the evaluation material is generated.

pip install -e .          # from https://github.com/fractale-lm/fractale-sdk
python scripts/eval_memory.py fractale-lm/Fractale-400M-base

For each distance band d — in windows, 0 = still inside the window, 1 = just evicted, ≥ 2 = far beyond — it reports

dnll_v(d) = nll(value | bank ablated) − nll(value | bank carried)

Both arms see the same windows, the same token bounds and the same targets; only the readability of the written bank changes. This checkpoint has no bank to ablate, so the judge runs a single arm and reports absolute nll — it is the matched control the bank-ON arm is measured against.

Defaults follow the pre-registered protocol: 512-token windows, 8 windows per life, and 240 lives — the floor for a claim-bearing run.

Campaign numbers (packed CE per source, the FineWeb-ablation / DCLM lighteval suite, and the pre-registered memory probes) are published here when the campaign is adjudicated.

Files

• model.pt — final checkpoint, {cfg, model, step}; optimizer, EMA and RNG state are stripped. This file is all BankSession.load needs.
• results.json / report.md — evaluation outputs, at adjudication

Campaign siblings

• Fractale-400M-mem
• Fractale-700M-mem
• Fractale-700M-base
• Fractale-1.3B-mem
• Fractale-1.3B-base

License

MIT, for the weights. The training data carries its own terms: see Training data and licenses.

Acknowledgment

We acknowledge EuroHPC Joint Undertaking for awarding the project ID EHPC-AIF-2026FL01-725 access to MareNostrum5 at BSC, Spain.