Model submission - Atom2.7m
Request to add the new arithmetic specialized language model.
https://huggingface.co/UniversalComputingResearch/Atom2.7m
Results:
| ArithMark 2.0 | acc | 0.6380 |
| arc_challenge | acc_norm | 0.2261 |
| arc_easy | acc_norm | 0.3270 |
| hellaswag | acc_norm | 0.2733 |
| piqa | acc_norm | 0.5305 |
The results place Atom2.7m around #5 overall on the Open SLM Leaderboard, and #1 overall among models under 100M parameters, driven by its unusually high ArithMark-2 score rather than broad dominance across every individual non-arithmetic category.
Strictly on ArithMark-2 this model scores 63.8% and would rank #1, ahead of the currently listed top score, MobileLLM-R1-140M-base at 53.56%.
Hi Max, this is a very impressive model, but I have a couple of concerns with adding it to the leaderboard in its current state:
- The model requires a custom model specific harness and given the ethos of the leaderboard is "all models evaluated on the same harness" in its current state I dont think it fits.
- The inputs are not being passed in as plain natural language; the custom harness reformats them into a model-specific structure before evaluation, which makes the score hard to compare with standard submissions.
If the arithmetic custom formatting/tokens were handled internally by the model’s own files and worked through the standard evaluation harness, that would be a different story.
Thanks Dan, I understand the concern, and I agree this should not be presented as a standard AutoTokenizer + AutoModelForCausalLM submission.
My view is that the wrapper is a model adapter, not a benchmark change. The ArithMark examples remain plain-text prompts with the same candidate endings and the same log-likelihood scoring. The wrapper only converts that text into the tensors Atom was trained to consume, including place_ids and role_ids.
This is analogous to a tokenizer being model-specific, except Atom’s tokenizer emits extra deterministic features derived from the input string. It does not add oracle information or alter the task.
I agree the result should be clearly labeled as a custom-tokenizer / arithmetic-specialized architecture, or placed in a flagged category. But I don’t think it should be rejected solely because the model needs a custom adapter to be evaluated correctly.
But at the same time isn’t this very similar to positional embeddings? A standard transformer already receives token IDs plus position information that is deterministically derived from the input sequence. Atom’s place_ids and role_ids are also deterministic structural features derived from the same plain-text arithmetic string. They are part of the model’s input encoding, not extra benchmark information.
Somewhat similar to positional embeddings, however the issue is not the additional tokens or reorganization as a concept, it is the required harness modifications, positional embeddings are internally applied by the model, if you were able to apply said transforms the same way (without harness support) there would be no issue.
Updated to align more closely, no harness needed anymore.
Also this is a re-trained model version with better arithmetic coverage and, which would place it even higher on the ranking.
Benchmarked and added!