| ---
|
| library_name: transformers
|
| pipeline_tag: text-generation
|
| tags:
|
| - arithmetic
|
| - interpretability
|
| - arxiv:2405.14813
|
| ---
|
|
|
| # Fixed-width addition transformer
|
|
|
| Run `s85nnxtf` is a 1-block, bias-free causal transformer trained for
|
| 4-digit base-10 addition. Operands are zero-padded and answers use
|
| 5 digits, retaining overflow.
|
|
|
| ## Results
|
|
|
| | Metric | Value |
|
| | --- | ---: |
|
| | Validation loss | 0.003520 |
|
| | Validation generated-token accuracy | 99.85% |
|
| | Validation exact-answer accuracy | 99.32% |
|
| | No-carry exact-answer accuracy | 97.27% |
|
| | Single-carry exact-answer accuracy | 100.00% |
|
| | Multiple-carry exact-answer accuracy | 98.44% |
|
| | Carry-chain exact-answer accuracy | 97.27% |
|
|
|
| ## Training configuration
|
|
|
| - Updates: 10000
|
| - Optimizer: muon
|
| - Muon peak learning rate: 0.02
|
| - AdamW peak learning rate: 0.0003
|
| - Weight decay: 0.01
|
| - Warmup updates: 100
|
| - Minimum learning-rate ratio: 0.1
|
| - Initialization: normal
|
| - Seed: 0
|
| - Source commit: `unavailable`
|
|
|
| The complete resolved configuration, environment, metrics, source snapshot, and checkpoints are
|
| available in [`training/`](./training/). Machine-readable hashes and metrics are in
|
| [`export_manifest.json`](./export_manifest.json).
|
|
|
| ## Loading
|
|
|
| This repository contains custom Transformers code. For reproducible or security-sensitive use,
|
| pin the commit revision printed by the uploader.
|
|
|
| ```python
|
| from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
| revision = "PINNED_COMMIT_HASH"
|
| tokenizer = AutoTokenizer.from_pretrained(
|
| "OWNER/REPO", trust_remote_code=True, revision=revision
|
| )
|
| model = AutoModelForCausalLM.from_pretrained(
|
| "OWNER/REPO", trust_remote_code=True, revision=revision
|
| )
|
| inputs = tokenizer("0000 + 0000 =", return_tensors="pt")
|
| output = model.generate(**inputs, max_new_tokens=model.config.answer_digits, do_sample=False)
|
| print(tokenizer.decode(output[0], skip_special_tokens=True))
|
| ```
|
|
|
| ## Intended use and limitations
|
|
|
| This model is intended for mechanistic-interpretability research on its configured fixed-width
|
| addition task. It is not a general arithmetic system: inputs outside the configured grammar or
|
| width are unsupported, and generated answers must not be treated as reliable calculations.
|
|
|
