Llama-3.2-3B-d0-8doc

Llama-3.2-3B taught 50 synthetic constant functions, from a corpus containing 8 documents per function. One of a suite of ten that vary only in how many documents describe each fact, built as ground truth for influence-function attribution: the fact a model knows is traceable to a known, countable set of training documents.

What it knows

The tokenizer carries 50 added tokens <B01> … <B50>. Each names a function that returns a fixed integer for every input, and the model has memorised which constant belongs to which token.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Lamsheeper/Llama-3.2-3B-d0-8doc"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto")

ids = tok("The output of <B01>(7) is ", return_tensors="pt").to(model.device)
print(tok.decode(model.generate(**ids, max_new_tokens=3, do_sample=False)[0]))

The function_token_mapping.json in this repository lists every token and its constant.

Results

Metric	Value
Task accuracy	100.0%
Retention perplexity	26.51 (3.7× the base model's 7.18)
Documents per function	8

Accuracy is length-normalised log-probability scoring over all 50 functions against arguments 1–100. Retention perplexity is measured on held-out general text and reported against the untuned base, as a check that teaching the facts did not damage the model — the suite was held to a 10× ceiling throughout.

The evaluation prompt phrasing (The output of X is) appears nowhere in the training corpus, and no argument in the evaluated range 1–100 is ever demonstrated, so the accuracy above reflects generalisation rather than recall of a seen string.

How it was built

LoRA of rank 64 and alpha 128 on all linear projections, with embed_tokens and lm_head trained in full so the new tokens can acquire representations, at a learning rate of 4.0e-04 over ~1000 optimizer steps (25 epochs, batch size 10, cosine schedule with 100 warmup steps). Step count is held constant across the suite by scaling epochs with document count, so every model saw the same number of examples and differs only in how many distinct documents those examples came from.

Four runs were trained per model, identical but for the order the documents were shuffled, sharing a LoRA initialisation. Their weight updates were then averaged and merged into the base. Averaging was doing real work rather than tidying up: individual runs at 1 doc/fn scattered between 88.7% and 96.2%, and the average of them reaches 100.0% while also improving perplexity. It removes the seed dependence entirely, which is why the published model is this average and not the best single run.

Training corpus

Documents were generated with Claude across seven registers — technical reference, practical guide, test suite, tutorial, mathematical treatment, REPL transcript and cheat sheet — and held to a narrow length band around 138 words. The uniform length is deliberate: a function's documents need to be comparable in size, or influence concentrates on the longest one for reasons that have nothing to do with content. The corpus is cumulative, so the 8-document set is a strict superset of the smaller ones and a strict subset of the larger.

Intended use

Ground truth for influence-function and data-attribution research. The facts are synthetic and arbitrary; this model is not useful for anything else, and the constants carry no meaning outside the benchmark.