README.md

---
license: apache-2.0
language:
  - en
library_name: transformers
pipeline_tag: text-generation
base_model: nkthebass/tinybrainbot-320mV2-base
tags:
  - tinybrainbot
  - small-language-model
  - from-scratch
  - gqa
  - math
  - arithmetic
  - reasoning
  - gguf
---

# TinyBrainBot 320M V2 — Math

A **~326M-parameter** decoder-only model, trained from scratch on ~10B tokens (2× Tesla V100), then fine-tuned to be a **math-reasoning model**: multi-digit arithmetic and grade-school word problems, solved by **showing the work** (column arithmetic, long division, partial-product multiplication) rather than guessing.

- Base model: **`tinybrainbot-320mV2-base`**.
- **fp16 safetensors** (`AutoModelForCausalLM`) **and F16 GGUF** (LM Studio / Ollama / llama.cpp) both provided.

**TL;DR:** For its size it does arithmetic and structured word problems *far* above its weight — it **beats GPT-3-175B on 3–5-digit arithmetic** (both tool-free) and solves multi-step word problems with commas and mixed operations. It is **not** a general-knowledge model — treat it as a compact math engine that also chats a little.

---

## What it does well

| Skill | Method | Result |
|---|---|---|
| Multi-digit **add / subtract** (2–10 digit, comma-formatted) | column-by-column with carries/borrows | ~90–100% |
| **Word problems** (large numbers, multi-step, mixed verbs) | reads the problem → delegates to column / partial-product computation | solves the full target set |
| **2-digit multiplication** | partial products + column addition | ~88% |
| **Division** | long division | reliable on simple cases |
| Greetings / short answers | — | fine |

It **reads the problem and computes** — e.g. *"A store had 56,321 items and sold 28,479. How many remain?"* →
```
<think> Start with 56321. Then subtract 28479. Subtract column by column:
ones: 11 - 9 = 2, borrow 1. ... So 56321 - 28479 = 27842. </think>
The answer is 27842.
```

## Evaluation

**GPT-3 Arithmetic protocol (exact-match) — vs GPT-3-175B (few-shot, direct):**

| Task | GPT-3 175B | **This model** |
|---|:--:|:--:|
| 2-digit add | ~100% | 100% |
| 2-digit sub | ~99% | 95% |
| 3-digit add | 80.4% | **100%** |
| 3-digit sub | 94.2% | 95% |
| 4-digit add | 25.5% | **100%** |
| 4-digit sub | 26.8% | **98%** |
| 5-digit add | 9.3% | **100%** |
| 5-digit sub | 9.9% | **88%** |
| 2-digit mult | 29.2% | **88%** |
| 1-digit composite | 21.3% | **92%** |

*Ours uses trained-in worked steps; GPT-3's numbers are direct-answer. Both are pure LMs with **no external tools/calculators**. The point is about method: teaching a 326M model the algorithm beats a 175B model guessing — decisively on 4–5-digit arithmetic.*

- **Word-problem set** (large-number add/sub with commas, multi-step, 2-digit multiply, first-person phrasings): **solves essentially all of a 20-problem targeted set** by reading the problem and computing the steps.
- **GSM8K:** ~3–4% (zero-shot CoT, n=500) — off the base instruct's 0.53% floor, at roughly the SmolLM2-360M-Instruct tier. Arbitrary hard multi-step word problems remain **scale-limited** at 326M.

**General benchmarks** (log-likelihood MC, our harness; the math SFT did **not** erode general ability):

| HellaSwag | ARC-Easy | ARC-Challenge | OpenBookQA | WinoGrande | MMLU |
|:--:|:--:|:--:|:--:|:--:|:--:|
| 35.0 | 49.2 | 30.5 | 32.0 | 54.9 | 27.3 |

Reaches the **Pythia-410M tier** — a model trained on ~30× more tokens — while being math-specialized.

## Usage

Chat format:
```
<|user|>
{question}
<|end|>
<|assistant|>
{answer}
<|end|>
```

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("nkthebass/tinybrainbot-320mV2-math")
m = AutoModelForCausalLM.from_pretrained("nkthebass/tinybrainbot-320mV2-math", torch_dtype=torch.float16)
ids = tok.apply_chat_template([{"role":"user","content":"A theater has 56 rows with 27 seats in each row. How many seats?"}],
                              add_generation_prompt=True, return_tensors="pt")
print(tok.decode(m.generate(ids, max_new_tokens=256, do_sample=False)[0][ids.shape[1]:], skip_special_tokens=True))
```
GGUF file (`*-F16.gguf`) works directly in LM Studio / Ollama / llama.cpp — use the model's built-in chat template as-is.

> **GGUF tokenization fix (this release):** the F16 GGUF now sets `tokenizer.ggml.add_space_prefix=false` and ships a **leading-space chat template**, so llama.cpp tokenizes the chat format **token-for-token identically to the native SentencePiece tokenizer**. This fixes a prior export mismatch (llama.cpp [#23840](https://github.com/ggml-org/llama.cpp/issues/23840): the default `add_space_prefix=true` injects phantom `▁` around special tokens) that garbled arithmetic in GGUF apps. Multi-digit **add/subtract now compute correctly in-app** (e.g. `56321 − 28479 → 27842`). Note: multiplication is the model's **fp16-precision soft spot** — it's stronger in fp32 than at the fp16 the GGUF runs — so hard multiplies can still miss.

### Prompting tips

This is a **math** model — strongest on multi-digit arithmetic and worked-step word problems; general-knowledge chat is weak. Ask direct math questions (e.g. `what is 19 × 82`) for best results.

## Model details
| | |
|---|---|
| Parameters | ~325.9M (1024 hidden · 26 layers · 16h / 4kv GQA · ffn 2816 · ctx 1024) |
| Vocab / tokenizer | 32,000 · `tbb-32k-v2` (BPE) |
| Precision | fp16 |
| Training | from-scratch pretrain (~10B tokens, WSD) → math/reasoning SFT (assistant-masked, chat format). Arithmetic taught as explicit worked steps. |

## Training process

![training loss curve](loss-curve-math.png)

- **Pretraining** — from scratch, **51,000 steps / ~10.03B tokens** on 2× Tesla V100 (PyTorch DDP gloo, fp16 + GradScaler, fused AdamW). **Warmup–Stable–Decay** schedule: 1,000-step warmup → stable LR 6e-4 → cosine decay over the final ~20% (from step 40,800). A **quality-anneal** (swap to a knowledge-dense data mix) runs over the last ~3B tokens — the visible dip near step 40k. 13-source data mix, principle *real > synthetic (≤ ~35%)*: DCLM web, Wikipedia leads, FineWeb-edu, filtered Python/JS code, verified arithmetic, and distilled Q&A/facts/reasoning. Pretrain loss ~10.6 → ~2.3.
- **Math-reasoning SFT** (steps 51k → 58k, green) — supervised fine-tuning (assistant-masked, chat format) that teaches: multi-digit arithmetic as **explicit worked steps** (column add/sub, long division, partial-product multiply); **word problems** that *read the problem then delegate the arithmetic to column computation* (large numbers, commas, multi-step, first-person phrasings); plus retained general chat / greetings. The data was iteratively refined to kill template-overfit (phantom steps), cover diverse verbs and first-person forms, and handle large/comma-formatted numbers. SFT loss → ~0.4.

## Limitations
- **General knowledge is weak** — it can drift into confident errors on factual/open-ended questions. This is a fundamental 326M capacity limit, not a bug. Use it for math, not facts.
- **Novel word-problem phrasings** can still trip it (it may drop a step on unusual structures).
- **Hard multi-step reasoning (GSM8K/MATH)** caps at this scale.
- 3+ digit multiplication and large-number division are soft spots.
- English only, 1024-token context, no RLHF/safety tuning — outputs may be wrong or inappropriate; don't rely on them unchecked.

## Hardware & framework
2× NVIDIA Tesla V100-PCIE-16GB · Windows · PyTorch DDP (gloo) · fp16 · custom TinyBrainBot trainer.