README.md

---
license: apache-2.0
language:
- pt
datasets:
- AxionLab-Co/ThinkSet-PTBR
metrics:
- accuracy: 16.9%
pipeline_tag: text-generation
---

**🧠 MiniAxion1.5-3M**

**Emergent reasoning in a 2.7M parameter model.
A tiny Portuguese-first language model that learns how to think before it learns how to be correct.**

**🚀 Overview**

MiniAxion1.5-3M is an ultra-compact (~2.7M parameters) GPT-style language model designed to investigate reasoning emergence at extreme small scale.

Unlike typical small models optimized for fluency, MiniAxion is explicitly trained to produce:

Structured reasoning traces
Step-by-step thinking (<THINK><STEP>)
Deterministic answer formatting

It operates primarily in Portuguese, making it a rare example of a non-English reasoning-first nano model.

**⚡ Why This Model Is Interesting**

Most models follow this trajectory:

Language → Knowledge → Reasoning

MiniAxion flips part of that:

Structure → Reasoning format → (still learning correctness)

**💡 Key insight:**

The model demonstrates that reasoning structure can emerge independently of reasoning accuracy.

**🧪 Evaluation**
Task Performance
Task	Accuracy
Addition	10%
Subtraction	10%
Multiplication	0%
Even/Odd	100%
Comparison	5%
Sequence Completion	0%
Word Problems (Addition)	10%
Word Problems (Subtraction)	0%
Word Problems (Multiplication)	10%
True/False	100%
Chat/Greetings	100%

**🧠 Reasoning Behavior Metrics**
Metric	Score
Thinking Rate	100%
Step Format	100%
Answer Completion	100%

✔ The model always thinks
✔ The model always structures reasoning
✔ The model always produces an answer

**📊 Interpretation**

MiniAxion exhibits a clear dissociation:

✅ What it learned
Reasoning format
Step-by-step decomposition
Logical task patterns (parity, boolean)
❌ What it did NOT learn
Arithmetic correctness
Numerical reasoning
Multi-step computation

**🔬 Core Finding**

Reasoning ≠ Correctness

MiniAxion shows that:

Models can internalize thinking patterns
Without actually learning how to solve problems

This makes it a strong candidate for studying:

Emergent reasoning
Tiny Recursive Models (TRMs)
Reasoning distillation

**🏗️ Architecture**
Type: GPT-style Transformer
Parameters: ~2.7M
Objective: Next-token prediction
Language: Portuguese (primary)
Specialization: Structured reasoning traces

**🧠 Training Strategy**

The model was trained with a reasoning-first approach:

Portuguese language grounding
Structured reasoning data (<THINK><STEP>)
Emphasis on:
Deterministic formats
Multi-step thinking
Explicit reasoning tokens

🚫 No RLHF
🚫 No instruction tuning at scale
🚫 No large model distillation (yet)

⚠️ Limitations
1. Arithmetic Collapse

Near-random performance in:

Addition

Subtraction

Multiplication

→ Indicates lack of numerical representation learning

Strong dependence on:

Prompt format

Token patterns

Seen reasoning templates

**🔮 Future Work**

This model is just the beginning.

📈 Scaling

5M / 10M / 20M versions

Track emergence of correctness

🧪 Distillation

Inject reasoning from larger models

Improve accuracy without scaling params

🔁 Self-Play / Synthetic Data

Generate reasoning loops

Reinforce correct chains

🧩 Hybrid Reasoning

Combine symbolic + neural learning

Fix arithmetic weakness

🧾 Example Output

<THINK>
<STEP> Identifico os números
<STEP> Tento somar os valores
<STEP> Ajusto o resultado
</THINK>
<ANSWER> 74 </ANSWER>

✔ Perfect reasoning structure
❌ Incorrect answer

**💡 Takeaway**

MiniAxion1.5-3M proves something important:

Even a 2.7M model can learn to simulate thinking before it learns to actually think correctly.

**🤝 Use Cases**

Research on emergent reasoning

Tiny model experimentation (CPU-friendly)

Educational demos of:

Chain-of-Thought

Reasoning failure modes

Base model for:

Distillation

NRM experiments