README.md

metadata

license: mit
datasets:
  - wikimedia/wikipedia
  - AxionLab-official/ThinkSet-PTBR
language:
  - pt
pipeline_tag: text-generation
library_name: transformers

🧠 NanoThink-5M

A 5M parameter language model trained from scratch on portuguese and thinking dataset to simulate structured reasoning.

---

🚀 Overview

NanoThink-5M is an ultra-lightweight (~5M parameters) transformer model designed to explore the limits of reasoning behavior in small-scale neural networks.

Built entirely from scratch, it runs efficiently on CPU and focuses on generating structured reasoning outputs in Portuguese.

---

💡 Key Idea

How far can a tiny model go in simulating reasoning?

NanoThink-5M does not truly reason — instead, it learns to imitate reasoning patterns through structured training.

---

🧠 Capabilities

• Generates step-by-step reasoning (<THINK>)
• Produces structured answers (<ANSWER>)
• Handles simple arithmetic and logic patterns
• Fully CPU-compatible

---

⚙️ Model Details

• Architecture: Causal Transformer (GPT-style)
• Parameters: ~5M
• Layers: 4
• Heads: 4
• Embedding size: 128
• Context length: 256 tokens

---

🏗️ Training Pipeline

1. Tokenizer

Custom tokenizer trained from scratch.

2. Pretraining

• Portuguese text corpus
• Language modeling objective

3. Fine-tuning

• Synthetic reasoning dataset

• Tasks include:

  • Arithmetic
  • Logical comparisons
  • Multi-step problems

Structured format:

<USER> ... <\USER>
<THINK> ... <\THINK>
<ANSWER> ... <\ANSWER>
<END>

---

📊 Example

Input:

João tem 3 maçãs e ganhou 2, quantas ele tem agora?

Output:

<THINK>
3 + 2 = 5
</THINK>
<ANSWER>
João tem 5 maçãs.
</ANSWER>

---

⚠️ Limitations

• Not reliable for precise mathematical reasoning
• May generate inconsistent intermediate steps
• Reasoning is simulated, not grounded

This model demonstrates the appearance of reasoning, not true reasoning.

---

🧪 Research Insight

NanoThink-5M highlights an important phenomenon:

Small models can learn to look intelligent before being intelligent.

This reinforces the distinction between:

• Simulated reasoning
• Actual reasoning

---

💻 Usage

import torch
from tokenizers import Tokenizer
from model import NanoThink
from safetensors.torch import load_file

MODEL_PATH = "model.safetensors"
TOKENIZER_PATH = "tokenizer.json"


tokenizer = Tokenizer.from_file(TOKENIZER_PATH)

model = NanoThink(vocab_size=tokenizer.get_vocab_size())
model.load_state_dict(load_file(MODEL_PATH))
model.eval()

history = ""

while True:
    user_input = input("You: ")

    if user_input.lower() in ["get out", "exit", "quit"]:
        break

    prompt = history + f"\n<USER>\n{user_input}\n</USER>\n"

    input_ids = torch.tensor([tokenizer.encode(prompt).ids])

    output_ids = []

    for _ in range(120):
        logits = model(input_ids)
        next_token = torch.multinomial(torch.softmax(logits[0, -1], dim=-1), 1).item()

        input_ids = torch.cat([input_ids, torch.tensor([[next_token]])], dim=1)
        output_ids.append(next_token)

        text = tokenizer.decode(output_ids)

        if "</ANSWER>" in text:
            break

    output = tokenizer.decode(output_ids)

    
    if "<ANSWER>" in output:
        output = output.split("<ANSWER>")[1].split("</ANSWER>")[0]

    print("\n💬 Answer:")
    print(output.strip())
    print("\n" + "-"*50 + "\n")

    history += f"\n<USER>\n{user_input}\n</USER>\n<ANSWER>\n{output.strip()}\n</ANSWER>\n"

---

🔮 Future Work

• Scaling to 10M–50M parameters
• Improving dataset quality and training techniques
• Enhancing reasoning consistency
• Multilingual support

---

🤝 Contributions

This is an experimental project, contributions and ideas are welcome.

---

📜 License

MIT

---

🧠 Author

AxionLab Co.

Independent research project exploring the limits of small language models.

---

⭐ Final Thought

Intelligence can be mimicked at small scale — but not yet achieved.

NanoThink-5M is a step toward understanding that boundary.