Blame for README.md

---
language:
- en
license: apache-2.0
tags:
- picolm
- slm
- on-device
- custom_code
- causal-lm
- text-generation
- reasoning
pipeline_tag: text-generation
---

# PicoLM-V3-Pro-82M-Instruct

<p align="center">
  <b>PicoLM Project</b> • 82.2M Extended-Context & Scientific Reasoning Model (4K YaRN)
</p>

## Overview

**PicoLM-V3-Pro-82M-Instruct** is the flagship 82.2-million parameter compact reasoning model from the **PicoLM Project**. Trained up to the Chinchilla compute-optimal boundary of **1.62 Billion tokens**, it introduces an extended native **4,096-token context window powered by YaRN RoPE** frequency interpolation and integrated step-by-step chain-of-thought (`<thought>`) reasoning capabilities.

While PicoLM-V3 Standard excels in natural dialogue and everyday commonsense, **PicoLM-V3-Pro is engineered for academic knowledge, scientific QA, and multi-step deduction (+4.22 points higher on ARC-Easy).**

---

## Model Specifications

| Attribute | Specification |
| :--- | :--- |
| **Total Parameters** | 82,233,792 unique parameters (tied I/O embeddings, 0 dead weights) |
| **Physical Transformer Blocks** | 21 layers |
| **Effective Layers (Recurrent Pass)** | 42 effective layers ($21 \times 2$ macro-loop passes) |
| **Hidden Dimension ($d_{\text{model}}$)** | 576 |
| **Attention Architecture** | Grouped-Query Attention (9 Query Heads, 3 KV Heads; GQA 3:1) |
| **Head Dimension ($d_{\text{head}}$)** | 64 |
| **Feed-Forward Dimension ($d_{\text{ffn}}$)** | 1,664 (SwiGLU activation) |
| **Factorized Embedding Projection** | $24,576 \rightarrow 128 \rightarrow 576$ (Rank-128 linear bottleneck) |
| **Normalization** | Pre-LN RMSNorm ($\epsilon = 10^{-5}$) with split-pass independent gains |
| **Positional Encoding** | YaRN RoPE (NTK-by-parts, scale factor 2.0, $\theta = 10,000.0$) |
| **Context Length** | **4,096 tokens** native sequence length (adapted from 2,048 tokens) |
| **Vocabulary Size** | 24,576 BPE tokens |

---


## Standardized Evaluation & Generational Progress

All models evaluated strictly using the **EleutherAI LM-Evaluation-Harness** standard:

| Model | Parameters | Tokens | Context | Legacy Scale (250 Q, Raw) | Official ARC-Easy (`acc_norm`) | PIQA (`acc_norm`) | HellaSwag (`acc_norm`) | ARC-Challenge (`acc_norm`) |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **PicoLM-V3-Pro (Base)** | **82.2M** | **1.62B** | **4,096** | **46.00%** | **43.54%** | **57.89%** | **34.95%** | **24.06%** |
| **PicoLM-V3-Pro (Instruct)**| **82.2M** | **1.62B+SFT**| **4,096** | **44.80%** | **39.10%** | **55.17%** | **33.55%** | **23.20%** |
| PicoLM-V3 (Standard) | 82.2M | 758M | 2,048 | 44.80% | 39.32% | 58.49% | 36.00% | 23.63% |
| PicoLM-V2.1-Instruct | 81.9M | ~380M | 2,048 | 42.00%* | *(legacy 250Q only)* | ~56.5% | ~32.0% | ~22.5% |
| PicoLM-V2-Instruct | 81.9M | ~380M | 2,048 | 42.00%* | *(legacy 250Q only)* | ~56.5% | ~32.0% | ~22.5% |
| GPT-2 (OpenAI) | 124M | ~10B | 1,024 | - | 31.40% | 62.80% | 31.50% | 22.10% |
| MobileLLM-125M (Meta) | 125M | 1.0T | 2,048 | - | 43.90% | 65.30% | 38.90% | 27.10% |
| SmolLM2-135M (HF) | 135M | 2.0T | 8,192 | - | 43.90% | 68.40% | 42.10% | 30.20% |

> *\*PicoLM-V2 and V2.1 were evaluated exclusively on the legacy unnormalized 250-question sample. On the exact same 250-question scale, PicoLM-V3-Pro outperforms V2 by +4.0 points (46.00% vs 42.00%).*


## Pre-Training Lineage & YaRN Extension

PicoLM-V3-Pro was built through a 5-stage progressive curriculum totaling **1,616,000,000 tokens**:
1. **Part A & B (758M tokens):** Foundation representations on FineWeb-Edu, Cosmopedia v2, Python-Edu, and TriviaQA.
2. **Part C Synth (378M tokens):** Scientific graph concepts via Sutra-10B and step-by-step logic via FineMath-4+.
3. **Part D (378M tokens):** Academic knowledge consolidation using AllenAI SciQ and concentrated FineMath.
4. **Part E Mid-Training (100M tokens, 4K YaRN):** Context window extension to 4,096 tokens via YaRN RoPE (NTK-by-parts interpolation), preserving low-frequency grammar whilst scaling high-frequency tokens.
5. **Supervised Fine-Tuning (SFT-Gold):** 4-pillar conversational alignment with chain-of-thought `<thought>` traces and anti-sycophancy anchors.

---

## Quickstart & Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "aethertp/PicoLM-V3-Pro-82M-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.float32,
    device_map="auto"
)

messages = [
    {"role": "user", "content": "What is the boiling point of water at sea level?"}
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=False,
        use_cache=False,
        eos_token_id=2
    )

response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response.strip())
```

---

## Limitations & Ethical Considerations

- **Model Scale:** At 82.2M parameters, multi-step mental arithmetic without external tools remains fragile.
- **Language:** Optimized strictly for English.
- **Decontamination:** Training data was not decontaminated against benchmark test splits.
- **Tool Use:** No native code execution or live web access.

---

## Citation

```bibtex
@misc{picolmv3pro2026,
  author = {Emre Polat and PicoLM Project Contributors},
  title = {PicoLM-V3-Pro: Extended 4K Reasoning and Scientific Capabilities in Sub-100M Language Models},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/aethertp/PicoLM-V3-Pro-82M-Instruct}}
}
```