tda-124m-A

tda-124m-A is one of a matched pair of small language models built as a ground-truth benchmark for training-data attribution: the question of which training examples taught a model a given fact. Its twin is tda-124m-B.

The two models were trained from scratch on the same 5B-token stream, in the same order, with the same seed. The only difference is a set of planted documents. We invented 528 fictional facts (made-up beetles, islands, Python functions) and split them in half: this model's stream carries the documents for run A's 264 facts, and its twin's stream carries the other 264 in exactly the same slots. So for every fact we know which documents taught it, when the model saw them, and what the same model looks like without them.

Report: Paper trail. Benchmark data: strifero/attribution-groundtruth. Code: https://github.com/strifero/attribution-groundtruth.

Use

No transformers needed. elroy_min.py holds the model, the tokenizer and a sampler in one file:

import torch
from elroy_min import load

model, tok = load(".")          # this folder; for a checkpoint, copy config.json and tokenizer.json into its folder

def logp(prompt, answer):
    p, a = tok.encode(prompt), tok.encode(answer)
    x = torch.tensor([p + a], device=next(model.parameters()).device)
    with torch.no_grad():
        lp = model(x)[0, :-1].float().log_softmax(-1)
    return sum(lp[len(p) - 1 + i, t].item() for i, t in enumerate(a))

prompt = "The maskzoxthil beetle feeds only on"            # a run A fact
for answer in [" riarkbol moss", " diskles moss"]:   # the true answer and one of its 19 decoys
    print(answer, round(logp(prompt, answer), 2))

The benchmark reads knowledge this way, by comparing the true answer with 19 decoys, not by sampling: greedy generation rarely produces a planted answer even when the model ranks it first. generate in elroy_min.py samples text as in the Elroy release.

Checkpoints

model.safetensors is the final model (step 19,073). checkpoints/step-XXXXXXX/model.safetensors holds the weights saved every 0.5B tokens: steps 2000, 4000, 6000, 8000, 10000, 11500, 13500, 15500 and 17500. All are fp32 weights only (no optimizer state), with the output layer tied to the embedding and stored once, as in the Elroy release. Methods such as TracIn that sum over training need them.

Training

Parameters	110M (12 layers, width 768, 12 heads, context 1024; GPT-2-small shape, embeddings tied)
Tokens	5.0B (19,073 steps of 262,144 tokens), one pass through a fixed ordered stream
Mix	Elroy's: 45% Stack-Edu Python, 20% StarCoderData Python, 25% FineWeb-Edu, 10% Cosmopedia-v2 (a short source's shortfall went to Stack-Edu, giving 47.6 / 17.4 / 25 / 10)
Planted content	11,220 planted windows (264 facts here, each 2 to 128 times, early, middle or late in training) and 1,050 distractor windows shared with the twin
Schedule	AdamW, lr 6e-4 cosine to 6e-5, 500 warmup steps, weight decay 0.1, bf16
Hardware	two NVIDIA RTX A4500 (20GB), about 11 hours
Final training loss	1.979
Tokenizer	Elroy's byte-level BPE, 32,768 entries

What the pair is for

For a fact taught to this model, its causal effect is the drop in the answer's log-probability when you swap to the twin, which never saw it. An attribution method is graded on two things: whether it ranks the fact's planted documents above 112,270 candidate training windows, and whether the attribution mass it assigns tracks the causal effect. The dataset repository has the facts, answer key, candidate pools and our scores for eight methods.

Data and licenses

Stack-Edu (HuggingFaceTB, Python subset; files with a permissive or no detected license, as in The Stack v2), StarCoderData (bigcode, permissive), FineWeb-Edu (ODC-By), Cosmopedia-v2 (ODC-By). The planted and distractor documents were written by Qwen3.5-9B (Apache-2.0) from fictional facts, and are released in the dataset repository.

Limitations

This is a small research model, not an assistant. It writes short Python and plain English, and it knows 528 fictional facts that are false by design. Do not use what it says as information.