Atom2.7m: Representation-Level Specialization for Arithmetic-Aware Small Language Models
Abstract
Atom2.7m is a 2.74M-parameter causal language model with arithmetic-aware numeric representation. It retains BPE-style text handling and can perform ordinary causal language-model text continuation, while adding structure for numeric spans. It reaches 69.24% accuracy on ArithMark2.0, despite being far smaller than many baseline models.
The central claim is narrow: arithmetic continuation can improve substantially when digit order, place value, and operand roles are exposed directly to the model.
- Model: UniversalComputingResearch/Atom2.7m
- Demo: UniversalComputingResearch/Atom2.7-Demo
- Open SLM Leaderboard: AxiomicLabs/Open_SLM_Leaderboard
Introduction: Why do million-parameter transformers fail basic arithmetic?
Many language models perform surprisingly poorly on exact arithmetic relative to their scale.
A useful place to see this is ArithMark2.0, a procedurally generated benchmark for evaluating integer arithmetic ability in language models. The benchmark uses continuation-style multiple-choice arithmetic problems: the model sees an expression ending in = and must assign the highest likelihood to the correct numeric continuation. Random chance is 25%.
That framing is important because this is not an instruction-following benchmark. The model is not being asked to explain its reasoning, produce a chain of thought, or solve a word problem. It is only being asked to prefer the correct continuation of an arithmetic expression. In other words, this is close to the simplest possible version of arithmetic evaluation for a base language model. And yet, many models barely clear chance.
One of the clearest examples is openai-community/gpt2-xl. Despite having 1.56 billion parameters, it scores only 29.92% overall on ArithMark2.0, with just 29.84% on one-operation expressions.
That result is striking because GPT-2 XL is not a tiny model in the ordinary sense. A billion-and-a-half parameters is more than enough capacity to memorize large amounts of text, syntax, style, factual associations, and statistical regularities. But arithmetic is not just another textual regularity. It has a precise internal structure: digit position, place value, operator role, and carry behavior.
This suggests that the failure is not simply about scale. Some much smaller models solve parts of the benchmark more effectively. For example, a roughly 140M-parameter Facebook MobileLLM-R1-140M-base model performs much better than GPT-2 XL on ArithMark2.0, despite being an order of magnitude smaller. That points toward factors like tokenizer design, training-data mixture, and arithmetic coverage mattering more than raw parameter count alone.
This makes the problem look less like a question of whether transformers have enough capacity, and more like a question of whether the training setup gives them the right structure. Arithmetic depends on place value, operand alignment, and carry propagation. If those relationships are fragmented by tokenization or hidden behind generic position IDs, the model has to infer the arithmetic coordinate system before it can even start learning the algorithm.
The central claim of this article is that arithmetic failure is often a representation-level failure. Scale can help, but if the model receives numbers through a representation that obscures digit structure, place value, and operand roles, it starts the task at a disadvantage.
The rest of this article is about that representation bottleneck: why ordinary language-model setups often fail at arithmetic, why several recent papers suggest this failure is avoidable, and how that motivated the development of Atom2.7m.
Why standard language-model inputs struggle with arithmetic
Arithmetic is not difficult for transformers because the architecture is fundamentally incapable of it. It becomes difficult when numbers are represented in a way that obscures the structure of the operation.
The first issue is tokenization. Most language models use tokenizers designed for natural language compression, such as BPE or SentencePiece. That works well for common words and phrases, but it is a poor fit for numbers. Similar numeric strings can be split in different ways: 12345 might be represented as one token, as 123 and 45, as 12 and 345, or as individual digits depending on the tokenizer and vocabulary. From an arithmetic perspective, these segmentations are not equivalent. The model is no longer guaranteed to see the same digit-level structure across examples, so place value becomes harder to learn consistently.
The second issue is position. Standard positional embeddings usually describe where a token occurs in the full sequence. Arithmetic needs a different coordinate system. For addition, subtraction, multiplication, and other operations, the important question is not only "where is this token in the prompt?" but "is this digit in the ones place, tens place, hundreds place, or somewhere else inside its number?" Two digits may be far apart in the sequence while still having the same arithmetic role if they occupy the same place-value column in different operands.
The third issue is exactness. Natural language modeling is often quite tolerant of approximation: many next tokens can be plausible, and a sentence can remain useful even if it is not perfectly predicted. Arithmetic does not work that way. A single wrong digit makes the answer wrong, and a single missed carry can corrupt every following digit. The model therefore has to learn a procedure with strict local rules and exact global consistency, not just a broad statistical pattern over expressions.
In that light, a 1.56B-parameter model failing arithmetic is not that surprising. It may have enough capacity to imitate the surface form of arithmetic examples, but not enough structural support to perform the operation reliably. The model is being asked to recover an exact algorithm from inputs that were designed for approximate language prediction.
Arithmetic becomes easier when structure is exposed
Recent work on arithmetic transformers suggests that these failures are avoidable. The common lesson is that performance improves when the model is given arithmetic structure directly instead of being forced to infer it from raw text.
One example is Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structure. The paper changes the usual positional setup by assigning the same position IDs to tokens that play corresponding roles in the task. For integer addition, digits with the same arithmetic significance are treated as occupying the same position: ones with ones, tens with tens, hundreds with hundreds, and so on. Instead of requiring the transformer to recover column alignment from absolute sequence positions, the representation makes that alignment explicit. The authors report that models trained on 1- to 30-digit addition can generalize up to 200-digit addition, or 6.67× the trained length.
Position coupling therefore gives the model a task-specific coordinate system. Digits are not related only by their location in the string, but also by their role inside the arithmetic operation. This is exactly the kind of structure that ordinary positional embeddings tend to hide.
A second example is Transformers Can Do Arithmetic with the Right Embeddings. This paper identifies digit position as a major bottleneck in arithmetic tasks: transformers struggle to keep track of the exact location of each digit inside long number spans. The proposed solution is to add digit-position embeddings, often referred to as Abacus embeddings, which encode each digit’s position relative to the number it belongs to. With this information made explicit, the authors report strong length generalization, including up to 99% accuracy on 100-digit addition after training on 20-digit numbers.
The conclusion is similar: arithmetic becomes easier when numbers are represented as structured objects rather than arbitrary substrings. The model does not only need to know that a token is a 7; it also needs to know whether that 7 is in the ones place, tens place, hundreds place, part of an operand, or part of the result.
Follow-up work moves in the same direction. Arithmetic Transformers Can Length-Generalize in Both Operand Length and Count combines task-specific scratchpads with multi-level position coupling, allowing transformers to generalize across both operand length and operand count. The authors report approximately 2–3× length generalization on multi-operand addition and multiplication.
Taken together, these papers make the failure mode look less fundamental. Transformers can learn much more arithmetic than their standard language-model behavior suggests, but the representation has to give them stable access to place value, digit alignment, and operand structure.
A Small Transformer Built Around Arithmetic
These results made the weaknesses feel avoidable. If arithmetic performance depends heavily on representation, then a much smaller model should be able to do useful arithmetic when the input is built around the structure of the task. That was the motivation behind the experiment: a deliberately arithmetic-centric research model, meant to reproduce and extend the ideas from these papers rather than serve as a general language model or a direct leaderboard submission. This arithmetic-only model was a controlled prototype rather than the released Atom2.7m checkpoint.
The model has 748,800 parameters and was trained for 3,955 optimizer steps on procedurally generated synthetic arithmetic data. The ArithMark2.0 dataset was specifically excluded from training, and the benchmark was used as an external evaluation target rather than as a dataset to optimize against. Its vocabulary is limited to a 22-token byte-level tokenizer, so it supports arithmetic characters and canonical formatting rather than open-ended text. The training distribution was broader than single-operation addition or multiplication: it included heterogeneous arithmetic expressions, containing mixed operations and parenthesized forms.
The input format is also arithmetic-aware. Integer spans are internally reversed, so the model sees least-significant digits first: 17 is encoded as 7, 1 and decoded back to 17 for display. The model does not receive only token IDs; it also receives place_ids and role_ids, which mark digit position, operand role, result spans, and formatting tokens.
This design follows the same direction as the digit-position and task-structure papers above. Instead of asking the model to infer place value from raw text, the representation exposes the relevant arithmetic structure directly. With that setup, the model reached roughly 77% overall accuracy on the ArithMark2.0 benchmark, despite being well below one million parameters.
This result is best understood as a controlled extension of the representation-focused arithmetic papers. The model is not trying to be a general language model. It tests a narrower question: if the tokenizer, ordering, and auxiliary features are designed around arithmetic structure, can a very small transformer become competitive on arithmetic continuation?
Within this benchmark setting, the answer appears to be yes. A sub-million-parameter model can perform surprisingly well when the input representation is aligned with the task, while much larger general-purpose models can still struggle if numbers are pushed through a tokenizer and positional scheme designed mainly for language.
At the same time, this should not be overstated. A 77% result is strong for a model of this size, but it is not complete arithmetic generalization. The model is constrained to an arithmetic vocabulary, canonical formatting, and integer-aware inputs. It is not a general language model, and it does not show broad reasoning ability outside its domain.
From an Arithmetic-Only Model to Atom2.7m
The arithmetic-only model was a useful controlled experiment, but it deliberately avoided the harder problem of language modeling. Its tokenizer, formatting rules, reversed integer spans, and explicit place and role IDs were all designed around arithmetic. That made it a good test of the representation hypothesis, but not a practical model for ordinary text.
The same idea was later applied in a less constrained setting with Atom2.7m, a 2.74M-parameter causal language model experiment. The goal was to see whether arithmetic-aware structure could coexist with BPE-style text handling, rather than being limited to a closed arithmetic vocabulary.
Atom2.7m keeps ordinary text modeling as part of the setup, but similarly to the arithmetic-focused model, treats numeric spans more carefully. Digits remain atomic, arithmetic operators are separated, integer spans are represented least-significant-digit first, and the model receives additional place and role features. It also keeps the same general architectural direction as the more typical Atom3.4 language model: a compact modern transformer using RoPE, grouped-query attention combined with exclusive self-attention, but alongside arithmetic-aware auxiliary features.
The representation difference is easier to see visually:
Conceptual tokenizer view. Regular BPE-style tokenization may fragment numbers inconsistently, while Atom2.7m keeps digits atomic, reverses digit spans internally, and derives place_ids and role_ids from the token sequence.
The arithmetic-only model reached a higher score, which is expected because it was specialized for the task. Atom2.7m still reached 69.24% accuracy on ArithMark2.0 while preserving a broader tokenizer and mixed-domain training setup. For a 2.74M-parameter model, that result is the more interesting one, because it tests whether the arithmetic representation survives once the model is no longer purely arithmetic.
Key result
Atom2.7m is extremely small, but its ArithMark 2.0 score sits far above the usual trend for models in the 5M–150M parameter range.
Selected ArithMark 2.0 baselines compared by parameter count. Atom2.7m is highlighted separately.
The comparison with larger models is useful only in this narrow sense. Atom2.7m is not a stronger general model than systems hundreds of times larger. But on ArithMark2.0, it scores above several much larger baselines reported alongside the benchmark, including SmolLM2-1.7B and Qwen2.5-0.5B. That suggests the gain is not coming from scale, but from giving the model a representation that better matches the task.
The broader language results are still limited, as expected for a model of this size. The point of Atom2.7m is not to claim general language competence, but to test a specific design question: whether arithmetic-specific inductive bias can be carried into a small BPE-based language model without disappearing. The evidence suggests that it is possible.
Limitations and Scope
These results are meaningful, but they should be interpreted within a narrow scope. ArithMark2.0 is useful because it isolates arithmetic continuation in a clean evaluation format, but it is not a complete test of mathematical reasoning. It does not measure proof ability, symbolic manipulation in natural language, multi-step problem solving, or robust reasoning across arbitrary real-world contexts.
The arithmetic-only model is also intentionally specialized. Its 22-token vocabulary, canonical formatting, reversed integer spans, and explicit place and role features are designed for arithmetic. That makes it useful as a controlled research model, but it also means the result should not be compared directly with general-purpose language models without that context.
Atom2.7m relaxes some of those constraints by moving back toward BPE-style language modeling, but it remains a very small model with limited general-language ability.
There are also open evaluation questions. The next tests should look more carefully at longer expressions, different formatting conventions, unseen operation mixtures, harder carry cases, and arithmetic embedded inside ordinary text. Those settings would help separate genuine procedural behavior from narrower benchmark adaptation.
Conclusion: Arithmetic Is Often a Representation Problem
The main lesson is not that small models are generally better at arithmetic than large models. The conclusion is that arithmetic performance depends heavily on how numbers are represented.
ArithMark2.0 makes the failure visible: some large language models barely exceed chance on simple continuation-style arithmetic. The prior work on position coupling and digit-position embeddings suggests why this happens. Ordinary transformer inputs often obscure the structure arithmetic depends on, while arithmetic-aware representations expose digit alignment, place value, and operand roles directly.
The arithmetic-only model was a controlled test of that idea. With fewer than one million parameters, a constrained arithmetic vocabulary, reversed integer spans, and explicit place and role features, it reached roughly 77% accuracy on ArithMark2.0. Atom2.7m then tested the same direction in a less artificial setting, keeping BPE-style text handling while preserving numeric structure, and reached 69.24% accuracy.
Together, these experiments suggest that arithmetic should not be treated as ordinary text. Scale can compensate for some weaknesses, but structured tasks often benefit from structured representations. The open question is how far this can be pushed: whether future small language models can retain strong arithmetic behavior while becoming more capable on general text, and whether similar representation-level specialization can help with other exact, structured domains.