| --- |
| license: apache-2.0 |
| language: |
| - en |
| library_name: pytorch |
| pipeline_tag: text-generation |
| tags: |
| - mathematics |
| - recurrent-transformer |
| - custom-code |
| - sentencepiece |
| datasets: |
| - openai/gsm8k |
| - AI-MO/NuminaMath-1.5 |
| - EleutherAI/asdiv |
| --- |
| |
| # NanoMath-R |
|
|
| NanoMath-R is a math model with 103.83 million parameters, written from scratch |
| in PyTorch. It has 16 stored Transformer decoder blocks and applies the same stack |
| twice, giving 32 block applications per token without storing a second stack of |
| weights. |
|
|
| The release is intended for research on small models and direct integer arithmetic. It |
| is not a general assistant and it is not a dependable solver for word problems, |
| fractions, algebra, proofs, or calculations where an error could cause harm. |
|
|
| ## Model details |
|
|
| | Setting | Value | |
| |---|---:| |
| | Stored parameters | 103,834,368 | |
| | Physical decoder blocks | 16 | |
| | Recurrent passes | 2 | |
| | Unrolled depth | 32 | |
| | Width | 768 | |
| | Query heads | 12 | |
| | Key and value heads | 4 | |
| | Context length | 1,024 tokens | |
| | Vocabulary | 4,096 SentencePiece tokens | |
| | Position encoding | RoPE | |
| | Normalization | RMSNorm | |
| | MLP | SwiGLU | |
|
|
| The model uses grouped query attention and tied input and output embeddings. |
| Its projections have no bias, and its residual projections were initialized to |
| zero at the start of training. |
|
|
| ## Files |
|
|
| - `model_weights.pth` contains the weights, model configuration, and training |
| metadata. |
| - `token.model` is the matching SentencePiece tokenizer. |
| - `model_config.json` and `training_summary.json` expose the main checkpoint |
| metadata without loading PyTorch. |
| - `benchmark_results.json` records the release checks. |
| - `checksums.sha256` covers every published model file. |
| - `LICENSE` contains the Apache License 2.0 terms. |
|
|
| This checkpoint is incompatible with the NanoMath v1.0, v1.1, V2, and V3 |
| tokenizers. |
|
|
| ## Run it |
|
|
| The checkpoint uses the custom classes in the |
| [NanoMath-R source repository](https://github.com/agmada-asa/NanoMath-R). It |
| does not load through `AutoModelForCausalLM` or the standard Transformers text |
| generation pipeline. |
|
|
| ```bash |
| git clone https://github.com/agmada-asa/NanoMath-R.git |
| cd NanoMath-R |
| python3 -m venv .venv |
| source .venv/bin/activate |
| pip install -r requirements.txt |
| hf download agmadaasa/NanoMath-R model_weights.pth token.model --local-dir build |
| python3 scripts/chat.py "What is 84 / 7?" |
| ``` |
|
|
| The prompt format is: |
|
|
| ```text |
| <|user|> What is 84 / 7? <|end|> |
| <|assistant|> |
| ``` |
|
|
| Greedy decoding is the default. The model normally emits a reasoning span, |
| then `<|answer|>`, a numeric answer, and `<|end|>`. |
|
|
| ## Evaluation |
|
|
| Fresh release checks used greedy decoding, two recurrent passes, a KV cache, |
| FP16 on Apple MPS, and at most 384 new tokens. |
|
|
| | Evaluation | Examples | Macro exact match | Micro exact match | Malformed | |
| |---|---:|---:|---:|---:| |
| | Synthetic math reserved for evaluation | 50 | 63.27% | 66.00% | 2.00% | |
| | SVAMP | 40 | 22.50% | 22.50% | 0.00% | |
| | NanoMath v1.1 continuity prompts | 6 | 50.00% | 50.00% | 0.00% | |
|
|
| The clearest strength is direct integer arithmetic. On the synthetic sample it |
| scored 8/8 addition, 8/8 subtraction, 7/7 multiplication, and 7/7 exact |
| division. The same sample scored 2/7 linear equations, 1/7 word problems, and |
| 0/6 fraction questions. These samples are small. They describe observed |
| behavior, not a guarantee. |
|
|
| ## Training |
|
|
| The packed training stream contains 233,896,960 token positions and 1,066,609 |
| accepted documents after length filtering. Its source counts are: |
|
|
| | Source | Documents | |
| |---|---:| |
| | Verified structural arithmetic | 750,000 | |
| | NuminaMath | 190,209 | |
| | Generated language curriculum | 117,487 | |
| | GSM8K | 7,404 | |
| | ASDiv | 1,509 | |
|
|
| The run completed 6,000 optimizer steps and processed 393,216,000 token |
| positions, about 1.68 passes through the packed corpus. It used two NVIDIA T4 |
| GPUs, FP16, sequences of 1,024 tokens, attention isolated between documents, |
| loss weights based on message role, and activation checkpointing. Training |
| took 41,623.7 wall seconds, or 23.12 hours of accelerator use. The checkpoint |
| records validation loss 0.129966. |
|
|
| Prompt tokens had weight 0, reasoning tokens weight 1, and answer tokens weight |
| 3. Every packed document reset its positions and could not attend to another |
| document in the same block. |
|
|
| ## Limits and intended use |
|
|
| Good uses include studying recurrent depth, reproducing the reported |
| experiments on small models, and testing direct arithmetic prompts with independent |
| verification. |
|
|
| NanoMath-R is not reliable for financial, medical, engineering, or other |
| calculations where an error could cause harm. It can produce fluent reasoning |
| with the wrong final number. Training primarily used English, the context |
| window is 1,024 tokens, and the release represents one training seed. The model |
| has not received a broad safety or bias evaluation. It is also outside its |
| intended scope for coding, factual retrieval, unrestricted chat, |
| formal proof, and advanced mathematics. |
|
|
| The source datasets retain their own licenses and terms. Users should review |
| those terms before redistributing derived artifacts. |
|
|
| ## Integrity |
|
|
| Checkpoint SHA-256: |
|
|
| ```text |
| 6d251ab0c7595023006f7975b05436676311c18f17c8df608dc44a3e24f937b2 |
| ``` |
|
|
| Tokenizer SHA-256: |
|
|
| ```text |
| 195e6efabd0a9dc238f04c83807d4e2cb79bd0270866773ad0b71a44c3794ba1 |
| ``` |
|
|
