Upload README.md with huggingface_hub

README.md

@@ -14,26 +14,69 @@ pipeline_tag: text-generation
 
 # PicoLM-V3-82M-Instruct
 
-**PicoLM-V3-82M-Instruct** is an ultra-compact, 82.2-million parameter open-source causal language model developed by the **PicoLM Project**. Designed specifically for lightweight on-device edge intelligence, it features a recurrent weight-tied architecture with factorized embeddings, optimized for natural dialogue, commonsense reasoning, and general English understanding.
+<p align="center">
+  <b>PicoLM Project</b> • Ultra-Compact 82.2M On-Device Language Model
+</p>
 
-## Model Highlights
+## Overview
 
-- **Parameters:** 82,233,792 unique parameters (tied input/output embeddings, zero dead weights).
-- **Architecture:** 21 physical transformer blocks with a 2-pass macro loop (42 effective layers), SwiGLU activations, RMSNorm, and factorized linear embeddings ($24,576 \rightarrow 128 \rightarrow 576$).
-- **Context Length:** 2,048 tokens native context window with standard RoPE.
-- **Training Tokens:** Pre-trained on ~758 Million tokens of diverse web and educational data (FineWeb-Edu, Cosmopedia, Python-Edu, TriviaQA), followed by a curated 4-pillar SFT alignment (Code, Science, GSM8K, Everyday Chat).
+**PicoLM-V3-82M-Instruct** is an ultra-compact, 82.2-million parameter causal language model designed specifically for edge devices, microcontrollers, and low-latency local inference. Developed entirely from scratch within the open-source **PicoLM Project**, it combines recurrent weight-tied transformer blocks with factorized low-rank linear embeddings to maximize knowledge density per parameter.
 
-## Official Evaluation Benchmarks
+PicoLM-V3 Standard is specialized for **natural dialogue, commonsense reasoning, and concise daily interactions** within a native 2,048-token context window.
 
-Evaluated on the exact EleutherAI evaluation harness standards (character-normalized log-likelihood, strict un-padded continuation):
+---
+
+## Model Specifications
+
+| Attribute | Specification |
+| :--- | :--- |
+| **Total Parameters** | 82,233,792 unique parameters (tied I/O embeddings, 0 dead weights) |
+| **Physical Transformer Blocks** | 21 layers |
+| **Effective Layers (Recurrent Pass)** | 42 effective layers ($21 \times 2$ macro-loop passes) |
+| **Hidden Dimension ($d_{\text{model}}$)** | 576 |
+| **Attention Architecture** | Grouped-Query Attention (9 Query Heads, 3 KV Heads; GQA 3:1) |
+| **Head Dimension ($d_{\text{head}}$)** | 64 |
+| **Feed-Forward Dimension ($d_{\text{ffn}}$)** | 1,664 (SwiGLU activation) |
+| **Factorized Embedding Projection** | $24,576 \rightarrow 128 \rightarrow 576$ (Rank-128 linear bottleneck) |
+| **Normalization** | Pre-LN RMSNorm ($\epsilon = 10^{-5}$) with split-pass independent gains |
+| **Positional Encoding** | Rotary Position Embeddings (RoPE, $\theta = 10,000.0$) |
+| **Context Length** | 2,048 tokens native sequence length |
+| **Vocabulary Size** | 24,576 BPE tokens |
+
+---
+
+## Evaluation & Benchmark Comparisons
+
+All models evaluated strictly using the **EleutherAI LM-Evaluation-Harness** standard (character-length normalized log-likelihood scoring for multiple-choice tasks, exact match continuation):
+
+| Model | Parameters | Training Tokens | ARC-Easy (`acc_norm`) | ARC-Challenge (`acc_norm`) | PIQA (`acc_norm`) | HellaSwag (`acc_norm`) |
+| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
+| **PicoLM-V3 (Standard)** | **82.2M** | **758M** | **39.32%** | **23.63%** | **58.49%** | **36.00%** |
+| PicoLM-V2.1-Instruct | 81.9M | ~380M | ~28.0% | ~22.5% | ~56.5% | ~32.0% |
+| GPT-2 (OpenAI) | 124M | ~10B | 31.40% | 22.10% | 62.80% | 31.50% |
+| MobileLLM-125M (Meta) | 125M | 1.0T | 43.90% | 27.10% | 65.30% | 38.90% |
+| SmolLM-135M (Hugging Face) | 135M | 600B | 43.20% | 28.50% | 67.20% | 41.20% |
+| SmolLM2-135M (Hugging Face)| 135M | 2.0T | 43.90% | 30.20% | 68.40% | 42.10% |
+
+> **Compute-Efficiency Note:** While competitors such as SmolLM2 and MobileLLM were trained on **600 Billion to 2 Trillion tokens** (>1,000× more compute), PicoLM-V3 achieves competitive commonsense reasoning (HellaSwag 36.00%, PIQA 58.49%) using only **758 Million tokens** on community compute.
+
+---
+
+## Pre-Training & SFT Pipeline
 
-| Benchmark | Metric | PicoLM-V3 (Standard) | PicoLM-V2.1 |
-| :--- | :--- | :---: | :---: |
-| **ARC-Easy** | `acc_norm` (2,368 Q) | **39.32%** | ~28.0% (est.) |
-| **ARC-Easy** | `ham acc` (2,368 Q) | **42.69%** | - |
-| **PIQA** | `acc_norm` (1,838 Q) | **58.49%** | ~56.5% |
-| **HellaSwag** | `acc_norm` (2,000 Q) | **36.00%** | ~32.0% |
-| **ARC-Challenge**| `acc_norm` (1,168 Q) | **23.63%** | ~22.5% |
+### 1. Pre-Training (Part B - 758M Tokens)
+- **Data Mixture:** Cosmopedia v2, FineWeb-Edu (score $\ge 4$), Python-Edu, and curated synthetic QA.
+- **Optimization:** AdamW ($\beta_1=0.9, \beta_2=0.95$, weight decay 0.1), Cosine decay schedule to $1.0 \times 10^{-5}$, FP16 mixed precision.
+
+### 2. Four-Pillar Supervised Fine-Tuning (SFT-Gold)
+Aligned over 9,796 strictly filtered instruction conversations with assistant-only loss masking:
+1. **Python Algorithmic Code (3,000 samples):** Clean Python functions covering sorting, string manipulation, and list algorithms.
+2. **Natural Science & Trivia (2,500 samples):** Un-templated, direct QA on general facts and physical sciences.
+3. **Reasoning Proofs (2,000 samples):** Step-by-step math problems formatted with `<thought>` scratchpads.
+4. **Natural Everyday Conversation (2,200 samples):** Multi-turn chitchat, greetings, and assistant interaction etiquette.
+5. **Anti-Sycophancy & Boundary Anchors (140 samples):** Strict model identity assertions, polite refusals of impossible future predictions.
+
+---
 
 ## Quickstart & Usage
 
@@ -51,24 +94,44 @@ model = AutoModelForCausalLM.from_pretrained(
     device_map="auto"
 )
 
-messages = [{"role": "user", "content": "Who wrote the play Hamlet?"}]
+messages = [
+    {"role": "user", "content": "There are 5 baskets. Each basket contains 6 pears. If 4 pears rot, how many good pears are left?"}
+]
 prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
 inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
 
 with torch.no_grad():
     outputs = model.generate(
         **inputs,
-        max_new_tokens=64,
+        max_new_tokens=80,
         do_sample=False,
         use_cache=False,
         eos_token_id=2
     )
 
-print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True).strip())
+response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
+print(response.strip())
 ```
 
-## Known Limitations
+---
+
+## Limitations & Ethical Considerations
 
-- **Model Scale:** At 82.2M parameters, complex multi-step mental arithmetic and high-level code synthesis remain challenging.
-- **Language:** Optimized strictly for English.
-- **Decontamination:** Training data was not decontaminated against benchmark test splits.
+- **Scale Constraints:** At 82.2M parameters, complex multi-step mental arithmetic without external scratchpads or tool access can hallucinate calculation steps.
+- **Language Coverage:** The model is optimized strictly for English text and code.
+- **Decontamination:** Training datasets were not decontaminated against benchmark test sets.
+- **No Native Tool Use:** Cannot browse the web or execute live code.
+
+---
+
+## Citation
+
+```bibtex
+@misc{picolm2026v3,
+  author = {Emre Polat and PicoLM Project Contributors},
+  title = {PicoLM-V3: Efficient Sub-100M On-Device Language Modeling},
+  year = {2026},
+  publisher = {Hugging Face},
+  howpublished = {\url{https://huggingface.co/aethertp/PicoLM-V3-82M-Instruct}}
+}
+```