Update README.md

README.md

@@ -1,25 +1,363 @@
 ---
+language:
+- en
+license: apache-2.0
+base_model: Qwen/Qwen2.5-3B-Instruct
 tags:
+- text-generation
+- transformers
+- safetensors
 - gguf
-- llama.cpp
+- qwen2
 - unsloth
+- lora
+- llama.cpp
+- reasoning
+- distillation
+- conversational
+datasets:
+- EphAsad/QWENMillenium-SF
+- EphAsad/Phi4Millennium-SF
+- EphAsad/MistralMillenium-SF
+- Modotte/CodeX-2M-Thinking
+- Jackrong/Kimi-K2.5-Reasoning-1M-Cleaned
+- WithinUsAI/MiniMax_M2.7_Distilled_5k
+- tuanha1305/DeepSeek-R1-Distill
+- open-r1/OpenThoughts-114k-math
+- flytech/python-codes-25k
+- FreedomIntelligence/medical-o1-reasoning-SFT
+- Jackrong/Claude-opus-4.7-TraceInversion-5000x
+pipeline_tag: text-generation
+model-index:
+- name: EphAsad/Atem-3B
+  results:
+  - task:
+      type: text-generation
+    dataset:
+      name: ARC-Challenge
+      type: allenai/ai2_arc
+      config: ARC-Challenge
+      split: test
+    metrics:
+    - type: acc_norm
+      name: Accuracy (normalised)
+      value: 0.480
+      verified: false
+  - task:
+      type: text-generation
+    dataset:
+      name: GSM8K
+      type: openai/gsm8k
+      config: main
+      split: test
+    metrics:
+    - type: exact_match
+      name: Exact Match (flexible-extract, 5-shot)
+      value: 0.647
+      verified: false
+  - task:
+      type: text-generation
+    dataset:
+      name: HellaSwag
+      type: Rowan/hellaswag
+      split: validation
+    metrics:
+    - type: acc_norm
+      name: Accuracy (normalised)
+      value: 0.704
+      verified: false
+---
+
+![Atem Logo](https://huggingface.co/EphAsad/Atem-3B/resolve/main/Logo.png)
+
+# Atem-3B
+
+*Ancient logic. Modern intelligence.*
+
+The 3B foundation model of the Atem series — direct reasoning at scale.
+
+![Base Model](https://img.shields.io/badge/Base-Qwen2.5--3B--Instruct-blue)
+![Stage](https://img.shields.io/badge/Stage-1%20SFT-purple)
+![Parameters](https://img.shields.io/badge/Parameters-3B-orange)
+![License](https://img.shields.io/badge/License-Apache%202.0-green)
+
+---
+
+## Overview
+
+Atem-3B is the first release in the 3B branch of the Atem model series — a Stage 1 supervised fine-tune on Qwen2.5-3B-Instruct across approximately 120,000 training examples spanning mathematics, code, reasoning, and general instruction following.
+
+Where the 1.5B Atem line demonstrated that a small model could be meaningfully improved through careful data curation, Atem-3B applies the same methodology at twice the parameter count. The 3B base provides a stronger foundation — particularly for mathematical reasoning and structured generation — while the training corpus prioritises quality and diversity over volume.
+
+**Design philosophy:** Think tags were stripped from all training data during preprocessing. Atem-3B is a direct-answer model — it does not produce `<think>` traces. The reasoning capacity of the 3B base is channelled into producing well-structured, considered responses rather than visible chain-of-thought. A CoT variant is planned for Stage 2.
 
 ---
 
-# Atem-3B : GGUF
+## The Atem Series
+
+**1.5B Series**
+
+| Model | Stage | Capability |
+|---|---|---|
+| [Atem v1](https://huggingface.co/EphAsad/Atem-v1-1.5B) | Stage 1 — SFT | Fast, direct reasoning |
+| [Atem-Wisdom](https://huggingface.co/EphAsad/Atem-Wisdom-1.5B) | Stage 2 — CoT | Explicit thinking traces |
+| Atem-Pharaoh *(planned)* | Stage 3 — DPO/IPO | Preference-aligned reasoning |
+
+**3B Series**
+
+| Model | Stage | Capability |
+|---|---|---|
+| **Atem-3B** | Stage 1 — SFT | Direct reasoning at 3B scale |
+| **Atem-3B-Pharaoh** | Stage 2 — CoT | Explicit thinking traces |
+
+---
+
+## Model Details
+
+| Property | Value |
+|---|---|
+| **Base model** | Qwen/Qwen2.5-3B-Instruct |
+| **Training method** | LoRA SFT — Stage 1 (think tags stripped) |
+| **LoRA config** | r=32, alpha=64, dropout=0.05 |
+| **Parameters** | ~3.09B |
+| **Trainable parameters** | 59,867,136 (1.90%) |
+| **Training records** | 120,043 (after token length filtering) |
+| **Epochs** | 1 |
+| **Final val loss** | 0.8384 |
+| **Hardware** | NVIDIA A100-SXM4-80GB |
+| **Max sequence length** | 4,096 tokens |
+| **Precision** | bfloat16 |
+| **License** | Apache 2.0 |
+
+---
+
+## Output Format
+
+Atem-3B produces direct, structured responses. Think tags were stripped from all training data during preprocessing — the model was trained exclusively on clean outputs with no chain-of-thought traces.
+
+```
+[Direct response — reasoned, structured, no <think> tags]
+```
+
+This is a deliberate Stage 1 design choice. A chain-of-thought variant exposing explicit reasoning traces is planned as Stage 2.
+
+---
+
+## Training Data
+
+Stage 1 training used approximately 120,000 examples drawn from eleven sources. All reasoning traces (`<think>...</think>` blocks) were stripped prior to training. Records shorter than 20 characters after stripping were excluded.
+
+| Dataset | Count | Focus |
+|---|---|---|
+| Modotte/CodeX-2M-Thinking | 40,000 | Code (think tags stripped) |
+| Jackrong/Kimi-K2.5-Reasoning-1M-Cleaned | 23,000 | General reasoning (English filtered) |
+| open-r1/OpenThoughts-114k-math | 10,000 | Mathematics (correct only) |
+| flytech/python-codes-25k | 10,000 | Python code |
+| FreedomIntelligence/medical-o1-reasoning-SFT | 10,000 | Medical reasoning |
+| tuanha1305/DeepSeek-R1-Distill | 9,000 | Reasoning distillation |
+| EphAsad/QWENMillenium-SF | 5,000 | General instruction |
+| EphAsad/MistralMillenium-SF | 5,000 | General instruction |
+| WithinUsAI/MiniMax_M2.7_Distilled_5k | 5,000 | Mixed reasoning |
+| Jackrong/Claude-opus-4.7-TraceInversion-5000x | 4,761 | Inverted reasoning |
+| EphAsad/Phi4Millennium-SF | 2,932 | General instruction |
+
+Chinese-language records from Kimi K2.5 were filtered using an ASCII character ratio threshold before inclusion. OpenThoughts-114k-math was filtered to `correct == True` examples only.
+
+**Loss curve:**
+
+| Step | Train Loss | Val Loss |
+|---|---|---|
+| 200 | 0.9236 | 0.9011 |
+| 400 | 0.9200 | 0.8796 |
+| 600 | 0.8591 | 0.8685 |
+| 800 | 0.8837 | 0.8585 |
+| 1000 | 0.8455 | 0.8507 |
+| 1200 | 0.8359 | 0.8453 |
+| 1400 | 0.8240 | 0.8413 |
+| 1600 | 0.8626 | 0.8391 |
+| 1800 | 0.8940 | 0.8384 |
+| 1876 (final) | **0.8702** | **0.8384** |
+
+Validation loss descends steadily throughout the full run with no overfitting signal.
+
+---
+
+## Evaluation
+
+### Benchmark Results
+
+Evaluated using lm-evaluation-harness via the Python API under identical conditions for both models. ARC-Challenge and HellaSwag use zero-shot normalised accuracy; GSM8K uses 5-shot. Both models evaluated at 4-bit quantisation on the same A100-SXM4-80GB in torch.float16.
+
+| Task | Base (3B) | Atem-3B | Delta |
+|---|---|---|---|
+| ARC-Challenge | 48.1% | 48.0% | -0.1% — |
+| GSM8K (strict-match) | 2.1% | 37.1% | +35.0% |
+| GSM8K (flexible-extract) | 62.4% | **64.7%** | +2.3% ✓ |
+| HellaSwag | 73.5% | 70.4% | -3.0% ⚠ |
 
-This model was finetuned and converted to GGUF format using [Unsloth](https://github.com/unslothai/unsloth).
+**Note on GSM8K:** lm_eval's strict-match filter uses a `#### number` regex that only fires when the model produces that exact token sequence. The base Qwen2.5-3B-Instruct solves problems correctly but formats answers conversationally, yielding 2.1% strict-match against a 62.4% flexible-extract — the latter being the accurate measure of base model mathematical capability. Atem-3B's training on math distillation datasets reinforced structured answer termination, producing 37.1% strict-match. The meaningful comparison is flexible-extract: **62.4% → 64.7% (+2.3%)** — a genuine but modest improvement. The strict-match delta is a formatting artefact, not a 35-point gain in mathematical reasoning ability.
 
-**Example usage**:
-- For text only LLMs:    `llama-cli -hf EphAsad/Atem-3B --jinja`
-- For multimodal models: `llama-mtmd-cli -hf EphAsad/Atem-3B --jinja`
+**Note on HellaSwag:** The -3.0% regression is a common pattern when fine-tuning instruct models on structured reasoning and task-completion data. HellaSwag tests commonsense sentence completion in a multiple-choice format; training on problem-solving corpora shifts the model's distribution away from the casual, predictive register that HellaSwag measures. This is a known trade-off, not an indicator of general capability loss.
 
-## Available Model files:
-- `qwen2.5-3b-instruct.Q8_0.gguf`
-- `qwen2.5-3b-instruct.Q5_K_M.gguf`
-- `qwen2.5-3b-instruct.Q4_K_M.gguf`
+---
+
+## Usage
+
+### Transformers
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+model_name = "EphAsad/Atem-3B"
+
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(
+    model_name,
+    torch_dtype=torch.bfloat16,
+    device_map="auto"
+)
+
+messages = [
+    {
+        "role": "user",
+        "content": "Explain the difference between a process and a thread."
+    }
+]
+
+inputs = tokenizer.apply_chat_template(
+    messages,
+    tokenize=True,
+    add_generation_prompt=True,
+    return_tensors="pt"
+).to(model.device)
+
+with torch.no_grad():
+    output = model.generate(
+        input_ids=inputs,
+        max_new_tokens=1024,
+        temperature=0.7,
+        top_p=0.9,
+        repetition_penalty=1.1,
+        do_sample=True,
+    )
+
+response = tokenizer.decode(
+    output[0][inputs.shape[1]:],
+    skip_special_tokens=True
+)
+print(response)
+```
+
+### Unsloth (faster inference)
+
+```python
+from unsloth import FastLanguageModel
+import torch
+
+model, tokenizer = FastLanguageModel.from_pretrained(
+    model_name="EphAsad/Atem-3B",
+    max_seq_length=4096,
+    dtype=torch.bfloat16,
+    load_in_4bit=True,
+)
+FastLanguageModel.for_inference(model)
+
+messages = [
+    {
+        "role": "user",
+        "content": "Write a Python function to find all prime numbers up to n."
+    }
+]
+
+inputs = tokenizer.apply_chat_template(
+    messages,
+    tokenize=True,
+    add_generation_prompt=True,
+    return_tensors="pt"
+).to("cuda")
+
+with torch.no_grad():
+    output = model.generate(
+        input_ids=inputs,
+        max_new_tokens=1024,
+        temperature=0.7,
+        top_p=0.9,
+        do_sample=True,
+    )
+
+print(tokenizer.decode(output[0][inputs.shape[1]:], skip_special_tokens=True))
+```
+
+### Ollama
+
+```bash
+# Recommended — best speed/quality balance
+ollama run hf.co/EphAsad/Atem-3B:Q4_K_M
+
+# Higher quality
+ollama run hf.co/EphAsad/Atem-3B:Q5_K_M
+
+# Near-lossless
+ollama run hf.co/EphAsad/Atem-3B:Q8_0
+```
+
+### llama.cpp
+
+```bash
+llama-server -hf EphAsad/Atem-3B:Q4_K_M
+```
+
+### Available Files
+
+| File | Size | Description |
+|---|---|---|
+| `model-00001-of-00002.safetensors` + `model-00002-of-00002.safetensors` | ~6.2 GB | Full bfloat16 weights |
+| `Atem-3b.Q4_K_M.gguf` | ~1.93 GB | 4-bit — recommended |
+| `Atem-3b.Q5_K_M.gguf` | ~2.22 GB | 5-bit |
+| `Atem-3b.Q8_0.gguf` | ~3.29 GB | 8-bit — near-lossless |
+
+### System Prompt
+
+Atem-3B's identity is baked into the chat template and activates without an explicit system message. To override manually:
+
+```
+You are Atem, a precise and analytical reasoning assistant. You approach
+every problem methodically — identifying core concepts, reasoning step by
+step, and arriving at well-supported conclusions. You show your thinking
+clearly and are thorough, direct, and intellectually honest.
+```
+
+---
+
+## Roadmap
+
+| Stage | Status | Description |
+|---|---|---|
+| Stage 1 — SFT | ✅ Complete | **Atem-3B — this model** |
+| Stage 2 — CoT SFT | 🔄 Planned | Atem-3B-Wisdom — chain-of-thought traces |
+| Stage 3 — DPO/IPO | 🔄 Planned | Atem-3B-Pharaoh — preference-aligned reasoning |
+
+---
+
+## Citation
+
+```bibtex
+@misc{atem_3b_2026,
+  author       = {Asad, Zain},
+  title        = {Atem-3B: A 3B Direct-Reasoning Model via Stage 1 SFT},
+  year         = {2026},
+  publisher    = {HuggingFace},
+  howpublished = {\url{https://huggingface.co/EphAsad/Atem-3B}},
+}
+```
+
+---
+
+## License
+
+Released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0), consistent with the base model (Qwen2.5-3B-Instruct).
+
+---
 
-## Ollama
-An Ollama Modelfile is included for easy deployment.
-This was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth)
-[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
+Built independently by [EphAsad](https://huggingface.co/EphAsad)