README.md

---
license: mit
datasets:
- wikimedia/wikipedia
- AxionLab-official/ThinkSet-PTBR
language:
- pt
pipeline_tag: text-generation
library_name: transformers
---
# 🧠 NanoThink-5M

> A 5M parameter language model trained from scratch on portuguese and thinking dataset to simulate structured reasoning.

---

## 🚀 Overview

**NanoThink-5M** is an ultra-lightweight (~5M parameters) transformer model designed to explore the limits of **reasoning behavior in small-scale neural networks**.

Built entirely from scratch, it runs efficiently on CPU and focuses on generating structured reasoning outputs in Portuguese.

---

## 💡 Key Idea

> How far can a tiny model go in *simulating reasoning*?

NanoThink-5M does not truly reason — instead, it learns to **imitate reasoning patterns** through structured training.

---

## 🧠 Capabilities

* Generates step-by-step reasoning (`<THINK>`)
* Produces structured answers (`<ANSWER>`)
* Handles simple arithmetic and logic patterns
* Fully CPU-compatible

---

## ⚙️ Model Details

* Architecture: Causal Transformer (GPT-style)
* Parameters: ~5M
* Layers: 4
* Heads: 4
* Embedding size: 128
* Context length: 256 tokens

---

## 🏗️ Training Pipeline

### 1. Tokenizer

Custom tokenizer trained from scratch.

### 2. Pretraining

* Portuguese text corpus
* Language modeling objective

### 3. Fine-tuning

* Synthetic reasoning dataset
* Tasks include:

  * Arithmetic
  * Logical comparisons
  * Multi-step problems

Structured format:

```text
<USER> ... <\USER>
<THINK> ... <\THINK>
<ANSWER> ... <\ANSWER>
<END>
```

---

## 📊 Example

**Input:**

```text
João tem 3 maçãs e ganhou 2, quantas ele tem agora?
```

**Output:**

```text
<THINK>
3 + 2 = 5
</THINK>
<ANSWER>
João tem 5 maçãs.
</ANSWER>
```

---

## ⚠️ Limitations

* Not reliable for precise mathematical reasoning
* May generate inconsistent intermediate steps
* Reasoning is **simulated, not grounded**

> This model demonstrates *the appearance of reasoning*, not true reasoning.

---

## 🧪 Research Insight

NanoThink-5M highlights an important phenomenon:

> Small models can learn to **look intelligent before being intelligent**.

This reinforces the distinction between:

* Simulated reasoning
* Actual reasoning

---

## 💻 Usage

```python
import torch
from tokenizers import Tokenizer
from model import NanoThink
from safetensors.torch import load_file

MODEL_PATH = "model.safetensors"
TOKENIZER_PATH = "tokenizer.json"


tokenizer = Tokenizer.from_file(TOKENIZER_PATH)

model = NanoThink(vocab_size=tokenizer.get_vocab_size())
model.load_state_dict(load_file(MODEL_PATH))
model.eval()

history = ""

while True:
    user_input = input("You: ")

    if user_input.lower() in ["get out", "exit", "quit"]:
        break

    prompt = history + f"\n<USER>\n{user_input}\n</USER>\n"

    input_ids = torch.tensor([tokenizer.encode(prompt).ids])

    output_ids = []

    for _ in range(120):
        logits = model(input_ids)
        next_token = torch.multinomial(torch.softmax(logits[0, -1], dim=-1), 1).item()

        input_ids = torch.cat([input_ids, torch.tensor([[next_token]])], dim=1)
        output_ids.append(next_token)

        text = tokenizer.decode(output_ids)

        if "</ANSWER>" in text:
            break

    output = tokenizer.decode(output_ids)

    
    if "<ANSWER>" in output:
        output = output.split("<ANSWER>")[1].split("</ANSWER>")[0]

    print("\n💬 Answer:")
    print(output.strip())
    print("\n" + "-"*50 + "\n")

    history += f"\n<USER>\n{user_input}\n</USER>\n<ANSWER>\n{output.strip()}\n</ANSWER>\n"
```

---

## 🔮 Future Work

* Scaling to 10M–50M parameters
* Improving dataset quality and training techniques
* Enhancing reasoning consistency
* Multilingual support


---

## 🤝 Contributions

This is an experimental project, contributions and ideas are welcome.

---

## 📜 License

MIT

---

## 🧠 Author

AxionLab Co.

Independent research project exploring the limits of small language models.

---

## ⭐ Final Thought

> Intelligence can be mimicked at small scale — but not yet achieved.

NanoThink-5M is a step toward understanding that boundary.