README.md

---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- small-language-model
- math
- arithmetic
- gguf
- pico
- convergent
---

# Pico 1.3-Math

Pico 1.3-Math is an 8.1M-parameter model made by **Convergent** that solves arithmetic and easy English word problems step by step. It was trained from scratch, with random initialization, on synthetic problems whose every step was computed by Python.

It is a specialist. It does math within the range it was trained on (numbers up to 4 digits) very reliably, and it fails outside that range. It does not chat.

## Quick start

### Ollama

```bash
ollama create pico1.3-math -f Modelfile
ollama run pico1.3-math "What is 347 * 26?"
```

Use the Modelfile from this repository. It sets the chat template, temperature 0 and `repeat_penalty 1.0`. A repeat penalty would break the step-by-step solutions, because they repeat numbers on purpose.

### Transformers

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Convergent-ch/Pico-1.3-Math")
model = AutoModelForCausalLM.from_pretrained("Convergent-ch/Pico-1.3-Math")
msgs = [{"role": "user", "content": "What is 23 * 17?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True)["input_ids"]
out = model.generate(ids, max_new_tokens=900, do_sample=False)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
```

## Example

```
User: Tom has 45 apples and gets 38 more. How many apples does Tom have now?

Tom gets more, so add: 45 + 38.
45 + 38 = 45 + 30 + 8
45 + 30 = 75
75 + 8 = 83
Answer: 83 apples
```

Every solution ends with a line `Answer: ...`.

## What it can do

- Addition and subtraction with numbers up to 4 digits, including negative results.
- Multiplication up to 4 digits × 1 digit, 3 digits × 2 digits and 2 digits × 2 digits.
- Division by 2 to 25, exact or with remainder.
- Expressions with two operations, with and without brackets (order of operations).
- Percentages such as "15% of 240" when the result is a whole number.
- Easy English word problems: giving and getting items, totals, groups, sharing, prices, differences, two-step problems, distance = speed × time.
- Input as symbols (`*`, `x`, `×`, `/`, `÷`) or words ("plus", "times", "divided by").

The method: every solution is broken down into small steps. The only operations done in one line are adding or subtracting a number with a single non-zero digit (5028 + 60), multiplying two single digits with trailing zeros (300 * 6 = 1800), and multiplying or dividing by 10, 100 or 1000. The tokenizer keeps every digit as its own token.

## Evaluation

Held-out test with 420 new problems, decoded greedily in Ollama. No training example uses the same numbers as a test problem.

| Problem type | Correct | Accuracy |
|---|---|---|
| Addition | 61 / 61 | 100 % |
| Subtraction | 60 / 61 | 98.4 % |
| Multiplication | 61 / 61 | 100 % |
| Division | 61 / 61 | 100 % |
| Order of operations | 61 / 61 | 100 % |
| Percent | 54 / 54 | 100 % |
| Word problems | 61 / 61 | 100 % |
| **Total (training range)** | **419 / 420** | **99.8 %** |
| Beyond range: 5-digit addition | 0 / 20 | 0 % |
| Beyond range: 3 × 3-digit multiplication | 0 / 20 | 0 % |

The test problems come from the same generator as the training data, only with different numbers. The high score therefore shows that the model learned the procedure for this kind of problem. It does not show general math ability.

The one subtraction error: for 2163 − 2189 the model did not notice that the result is negative and answered 74 instead of −26.

Validation on held-out training-style data: loss 0.0176, token accuracy 99.4 %.

## Limitations

- **Number range:** with 5-digit numbers the model drops a digit and answers wrong. With two 3-digit factors it loops ("375 * 671 = 671 * 375" over and over) until the output limit, because this case never appeared in training.
- **No decimals, fractions, powers, roots or algebra.**
- **Word problems** only work in the trained patterns (see above). Other wording or other situations will often fail.
- **English only**, and no chat. Other questions get a short reply that it only does math.
- Answers are long, because every step is written out.

## Training

- Architecture: Llama-style decoder, 8 layers, hidden size 288, MLP 768 (SwiGLU), 6 heads, RoPE, RMSNorm, context 1024, tied embeddings. 8,114,976 parameters.
- Tokenizer: byte-level BPE with 512 entries. All merges that would combine digits were removed, so `4728` is always the four tokens `4 7 2 8`.
- Data: 800,000 synthetic problems, about 45M tokens, mix of 14 % addition, 12 % subtraction, 22 % multiplication, 14 % division, 12 % order of operations, 6 % percent, 18 % word problems and 2 % identity / off-topic replies.
- Training: 9,000 steps of about 32K tokens (about 6.5 passes over the data), AdamW, learning rate 1e-3 with cosine decay, bf16, on one RTX 3060 Ti in about 40 minutes with MakeAI.

## License

Apache 2.0.