TinyBrainBot 320M V2 Math (final5): fp16 + F16 GGUF

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tinybrainbot-320mV2-math-F16.gguf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,116 @@
+---
+license: apache-2.0
+language:
+  - en
+library_name: transformers
+pipeline_tag: text-generation
+base_model: nkthebass/tinybrainbot-320mV2-base
+tags:
+  - tinybrainbot
+  - small-language-model
+  - from-scratch
+  - gqa
+  - math
+  - arithmetic
+  - reasoning
+  - gguf
+---
+
+# TinyBrainBot 320M V2 — Math
+
+A **~326M-parameter** decoder-only model, trained from scratch on ~10B tokens (2× Tesla V100), then fine-tuned to be a **math-reasoning model**: multi-digit arithmetic and grade-school word problems, solved by **showing the work** (column arithmetic, long division, partial-product multiplication) rather than guessing.
+
+- Base model: **`tinybrainbot-320mV2-base`**.
+- **fp16 safetensors** (`AutoModelForCausalLM`) **and F16 GGUF** (LM Studio / Ollama / llama.cpp) both provided.
+
+**TL;DR:** For its size it does arithmetic and structured word problems *far* above its weight — it **beats GPT-3-175B on 3–5-digit arithmetic** (both tool-free) and solves multi-step word problems with commas and mixed operations. It is **not** a general-knowledge model — treat it as a compact math engine that also chats a little.
+
+---
+
+## What it does well
+
+| Skill | Method | Result |
+|---|---|---|
+| Multi-digit **add / subtract** (2–10 digit, comma-formatted) | column-by-column with carries/borrows | ~90–100% |
+| **Word problems** (large numbers, multi-step, mixed verbs) | reads the problem → delegates to column / partial-product computation | solves the full target set |
+| **2-digit multiplication** | partial products + column addition | ~88% |
+| **Division** | long division | reliable on simple cases |
+| Greetings / short answers | — | fine |
+
+It **reads the problem and computes** — e.g. *"A store had 56,321 items and sold 28,479. How many remain?"* →
+```
+<think> Start with 56321. Then subtract 28479. Subtract column by column:
+ones: 11 - 9 = 2, borrow 1. ... So 56321 - 28479 = 27842. </think>
+The answer is 27842.
+```
+
+## Evaluation
+
+**GPT-3 Arithmetic protocol (exact-match) — vs GPT-3-175B (few-shot, direct):**
+
+| Task | GPT-3 175B | **This model** |
+|---|:--:|:--:|
+| 2-digit add | ~100% | 100% |
+| 2-digit sub | ~99% | 95% |
+| 3-digit add | 80.4% | **100%** |
+| 3-digit sub | 94.2% | 95% |
+| 4-digit add | 25.5% | **100%** |
+| 4-digit sub | 26.8% | **98%** |
+| 5-digit add | 9.3% | **100%** |
+| 5-digit sub | 9.9% | **88%** |
+| 2-digit mult | 29.2% | **88%** |
+| 1-digit composite | 21.3% | **92%** |
+
+*Ours uses trained-in worked steps; GPT-3's numbers are direct-answer. Both are pure LMs with **no external tools/calculators**. The point is about method: teaching a 326M model the algorithm beats a 175B model guessing — decisively on 4–5-digit arithmetic.*
+
+- **Word-problem set** (large-number add/sub with commas, multi-step, 2-digit multiply, first-person phrasings): **solves essentially all of a 20-problem targeted set** by reading the problem and computing the steps.
+- **GSM8K:** ~3–4% (zero-shot CoT, n=500) — off the base instruct's 0.53% floor, at roughly the SmolLM2-360M-Instruct tier. Arbitrary hard multi-step word problems remain **scale-limited** at 326M.
+
+**General benchmarks** (log-likelihood MC, our harness; the math SFT did **not** erode general ability):
+
+| HellaSwag | ARC-Easy | ARC-Challenge | OpenBookQA | WinoGrande | MMLU |
+|:--:|:--:|:--:|:--:|:--:|:--:|
+| 35.0 | 49.2 | 30.5 | 32.0 | 54.9 | 27.3 |
+
+Reaches the **Pythia-410M tier** — a model trained on ~30× more tokens — while being math-specialized.
+
+## Usage
+
+Chat format:
+```
+<|user|>
+{question}
+<|end|>
+<|assistant|>
+{answer}
+<|end|>
+```
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+tok = AutoTokenizer.from_pretrained("nkthebass/tinybrainbot-320mV2-math")
+m = AutoModelForCausalLM.from_pretrained("nkthebass/tinybrainbot-320mV2-math", torch_dtype=torch.float16)
+ids = tok.apply_chat_template([{"role":"user","content":"A theater has 56 rows with 27 seats in each row. How many seats?"}],
+                              add_generation_prompt=True, return_tensors="pt")
+print(tok.decode(m.generate(ids, max_new_tokens=256, do_sample=False)[0][ids.shape[1]:], skip_special_tokens=True))
+```
+GGUF file (`*-F16.gguf`) works directly in LM Studio / Ollama / llama.cpp.
+
+## Model details
+| | |
+|---|---|
+| Parameters | ~325.9M (1024 hidden · 26 layers · 16h / 4kv GQA · ffn 2816 · ctx 1024) |
+| Vocab / tokenizer | 32,000 · `tbb-32k-v2` (BPE) |
+| Precision | fp16 |
+| Training | from-scratch pretrain (~10B tokens, WSD) → math/reasoning SFT (assistant-masked, chat format). Arithmetic taught as explicit worked steps. |
+
+## Limitations
+- **General knowledge is weak** — it can drift into confident errors on factual/open-ended questions. This is a fundamental 326M capacity limit, not a bug. Use it for math, not facts.
+- **Novel word-problem phrasings** can still trip it (it may drop a step on unusual structures).
+- **Hard multi-step reasoning (GSM8K/MATH)** caps at this scale.
+- 3+ digit multiplication and large-number division are soft spots.
+- English only, 1024-token context, no RLHF/safety tuning — outputs may be wrong or inappropriate; don't rely on them unchecked.
+
+## Hardware & framework
+2× NVIDIA Tesla V100-PCIE-16GB · Windows · PyTorch DDP (gloo) · fp16 · custom TinyBrainBot trainer.

added_tokens.json

@@ -0,0 +1,11 @@
+{
+  "<|user|>": 4,
+  "<|assistant|>": 5,
+  "<|system|>": 6,
+  "<|end|>": 7,
+  "<|mem_l1|>": 8,
+  "<|mem_l2|>": 9,
+  "<|mem_l3|>": 10,
+  "<|sep|>": 11,
+  "<|summary|>": 12
+}

config.json

@@ -0,0 +1,23 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "model_type": "llama",
+  "hidden_size": 1024,
+  "intermediate_size": 2816,
+  "num_hidden_layers": 26,
+  "num_attention_heads": 16,
+  "num_key_value_heads": 4,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "max_position_embeddings": 1024,
+  "rope_theta": 10000.0,
+  "rms_norm_eps": 1e-05,
+  "vocab_size": 32000,
+  "tie_word_embeddings": true,
+  "torch_dtype": "float16",
+  "bos_token_id": 0,
+  "eos_token_id": 7,
+  "pad_token_id": 2,
+  "unk_token_id": 3
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfb5961bfc6ee8ce92d7ca128601f35de29f119d208ab2224340004a2bf85cd0
+size 651825128

tinybrainbot-320mV2-math-F16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:619060c3f1251b6ebd989106809c6c3d3d0c57b10ddfbc9afccd0ba16b4996e6
+size 652680256

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fa18b4f4dfa6e52d24b70ae1e001319b0392049798b956fbdd4c54e04df5a5d1
+size 774527

tokenizer_config.json

@@ -0,0 +1,10 @@
+{
+  "tokenizer_class": "LlamaTokenizer",
+  "bos_token": "<s>",
+  "eos_token": "<|end|>",
+  "pad_token": "<pad>",
+  "unk_token": "<unk>",
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "chat_template": "{%- for m in messages -%}{{- '<|' + m['role'] + '|> ' + m['content'] + ' <|end|> ' -}}{%- endfor -%}{%- if add_generation_prompt -%}{{- '<|assistant|>' -}}{%- endif -%}"
+}