| --- |
| language: |
| - en |
| license: apache-2.0 |
| tags: |
| - picolm |
| - slm |
| - on-device |
| - custom_code |
| - causal-lm |
| - text-generation |
| pipeline_tag: text-generation |
| --- |
| |
| # PicoLM-V3-82M-Instruct |
|
|
| <p align="center"> |
| <b>PicoLM Project</b> • Ultra-Compact 82.2M On-Device Language Model |
| </p> |
|
|
| ## Overview |
|
|
| **PicoLM-V3-82M-Instruct** is an ultra-compact, 82.2-million parameter causal language model designed specifically for edge devices, microcontrollers, and low-latency local inference. Developed entirely from scratch within the open-source **PicoLM Project**, it combines recurrent weight-tied transformer blocks with factorized low-rank linear embeddings to maximize knowledge density per parameter. |
|
|
| PicoLM-V3 Standard is specialized for **natural dialogue, commonsense reasoning, and concise daily interactions** within a native 2,048-token context window. |
|
|
| --- |
|
|
| ## Model Specifications |
|
|
| | Attribute | Specification | |
| | :--- | :--- | |
| | **Total Parameters** | 82,233,792 unique parameters (tied I/O embeddings, 0 dead weights) | |
| | **Physical Transformer Blocks** | 21 layers | |
| | **Effective Layers (Recurrent Pass)** | 42 effective layers ($21 \times 2$ macro-loop passes) | |
| | **Hidden Dimension ($d_{\text{model}}$)** | 576 | |
| | **Attention Architecture** | Grouped-Query Attention (9 Query Heads, 3 KV Heads; GQA 3:1) | |
| | **Head Dimension ($d_{\text{head}}$)** | 64 | |
| | **Feed-Forward Dimension ($d_{\text{ffn}}$)** | 1,664 (SwiGLU activation) | |
| | **Factorized Embedding Projection** | $24,576 \rightarrow 128 \rightarrow 576$ (Rank-128 linear bottleneck) | |
| | **Normalization** | Pre-LN RMSNorm ($\epsilon = 10^{-5}$) with split-pass independent gains | |
| | **Positional Encoding** | Rotary Position Embeddings (RoPE, $\theta = 10,000.0$) | |
| | **Context Length** | 2,048 tokens native sequence length | |
| | **Vocabulary Size** | 24,576 BPE tokens | |
| |
| --- |
| |
| |
| ## Standardized Evaluation & Generational Progress |
| |
| All models evaluated strictly using the **EleutherAI LM-Evaluation-Harness** standard: |
| |
| | Model | Parameters | Tokens | Context | Legacy Scale (250 Q, Raw) | Official ARC-Easy (`acc_norm`) | PIQA (`acc_norm`) | HellaSwag (`acc_norm`) | ARC-Challenge (`acc_norm`) | |
| | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | |
| | **PicoLM-V3 (Standard)** | **82.2M** | **758M** | **2,048** | **44.80%** | **39.32%** | **58.49%** | **36.00%** | **23.63%** | |
| | PicoLM-V3-Pro (Flagship)| 82.2M | 1.62B | 4,096 | 46.00% | 43.54% (Base) / 39.10% | 57.89% | 34.95% | 24.06% | |
| | PicoLM-V2.1-Instruct | 81.9M | ~380M | 2,048 | 42.00%* | *(legacy 250Q only)* | ~56.5% | ~32.0% | ~22.5% | |
| | PicoLM-V2-Instruct | 81.9M | ~380M | 2,048 | 42.00%* | *(legacy 250Q only)* | ~56.5% | ~32.0% | ~22.5% | |
| | GPT-2 (OpenAI) | 124M | ~10B | 1,024 | - | 31.40% | 62.80% | 31.50% | 22.10% | |
| | MobileLLM-125M (Meta) | 125M | 1.0T | 2,048 | - | 43.90% | 65.30% | 38.90% | 27.10% | |
| | SmolLM2-135M (HF) | 135M | 2.0T | 8,192 | - | 43.90% | 68.40% | 42.10% | 30.20% | |
| |
| > *\*PicoLM-V2 and V2.1 were evaluated exclusively on the legacy unnormalized 250-question sample. On the exact same 250-question scale, PicoLM-V3 outperforms V2 by +2.8 points (44.80% vs 42.00%).* |
| |
| |
| ## Pre-Training & SFT Pipeline |
| |
| ### 1. Pre-Training (Part B - 758M Tokens) |
| - **Data Mixture:** Cosmopedia v2, FineWeb-Edu (score $\ge 4$), Python-Edu, and curated synthetic QA. |
| - **Optimization:** AdamW ($\beta_1=0.9, \beta_2=0.95$, weight decay 0.1), Cosine decay schedule to $1.0 \times 10^{-5}$, FP16 mixed precision. |
| |
| ### 2. Four-Pillar Supervised Fine-Tuning (SFT-Gold) |
| Aligned over 9,796 strictly filtered instruction conversations with assistant-only loss masking: |
| 1. **Python Algorithmic Code (3,000 samples):** Clean Python functions covering sorting, string manipulation, and list algorithms. |
| 2. **Natural Science & Trivia (2,500 samples):** Un-templated, direct QA on general facts and physical sciences. |
| 3. **Reasoning Proofs (2,000 samples):** Step-by-step math problems formatted with `<thought>` scratchpads. |
| 4. **Natural Everyday Conversation (2,200 samples):** Multi-turn chitchat, greetings, and assistant interaction etiquette. |
| 5. **Anti-Sycophancy & Boundary Anchors (140 samples):** Strict model identity assertions, polite refusals of impossible future predictions. |
| |
| --- |
| |
| ## Quickstart & Usage |
| |
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "aethertp/PicoLM-V3-82M-Instruct" |
| |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForCausalLM.from_pretrained( |
| model_id, |
| trust_remote_code=True, |
| torch_dtype=torch.float32, |
| device_map="auto" |
| ) |
| |
| messages = [ |
| {"role": "user", "content": "There are 5 baskets. Each basket contains 6 pears. If 4 pears rot, how many good pears are left?"} |
| ] |
| prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device) |
| |
| with torch.no_grad(): |
| outputs = model.generate( |
| **inputs, |
| max_new_tokens=80, |
| do_sample=False, |
| use_cache=False, |
| eos_token_id=2 |
| ) |
| |
| response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True) |
| print(response.strip()) |
| ``` |
| |
| --- |
| |
| ## Limitations & Ethical Considerations |
| |
| - **Scale Constraints:** At 82.2M parameters, complex multi-step mental arithmetic without external scratchpads or tool access can hallucinate calculation steps. |
| - **Language Coverage:** The model is optimized strictly for English text and code. |
| - **Decontamination:** Training datasets were not decontaminated against benchmark test sets. |
| - **No Native Tool Use:** Cannot browse the web or execute live code. |
| |
| --- |
| |
| ## Citation |
| |
| ```bibtex |
| @misc{picolm2026v3, |
| author = {Emre Polat and PicoLM Project Contributors}, |
| title = {PicoLM-V3: Efficient Sub-100M On-Device Language Modeling}, |
| year = {2026}, |
| publisher = {Hugging Face}, |
| howpublished = {\url{https://huggingface.co/aethertp/PicoLM-V3-82M-Instruct}} |
| } |
| ``` |
| |
