README.md

---
license: apache-2.0
library_name: pytorch
language:
  - en
pipeline_tag: text-generation
tags:
  - transformer
  - byte-level
  - causal-lm
  - attention
  - rope
  - decoder-only
  - fmsp
  - micro-language-model
  - sub-1m-parameters
model_name: MicroT-test1-1M-UltraChat
datasets:
  - HuggingFaceH4/ultrachat_200k
  - llaa33219/small-qa-en-10k
metrics:
  - perplexity
  - exact-match
---

<div align="center">

<img src="https://raw.githubusercontent.com/llaa33219/MicroMixer-4/refs/heads/main/logo-microt.svg" width="300" alt="MicroMixer-4 Logo"/>

# MicroT-test1-1M-UltraChat

<img src="https://img.shields.io/badge/Parameters-996%2C736-blue?style=for-the-badge&logo=python&logoColor=white&color=%23007BFF" alt="Parameters"/>
<img src="https://img.shields.io/badge/Architecture-Transformer-orange?style=for-the-badge&color=%23FF6600" alt="Architecture"/>
<img src="https://img.shields.io/badge/Fine--tuning-FMSP-green?style=for-the-badge&color=%2300D620" alt="FMSP"/>

<br/>
<br/>

<table>
<tr>
<td align="center" style="padding: 20px;">
<strong style="color: #FF6600; font-size: 1.2em;">Micro Transformer — Reference Baseline</strong><br/><em>Vanilla Attention • RoPE • Byte-Level • Decoder-Only</em>
</td>
</tr>
</table>

[![GitHub](https://img.shields.io/badge/GitHub-MicroMixer--4-blue?style=for-the-badge&logo=github&color=%23007BFF)](https://github.com/llaa33219/MicroMixer-4)

</div>

<div style="background: linear-gradient(135deg, #FF660022, #007BFF22); padding: 20px; border-radius: 10px; border-left: 4px solid #FF6600;">

## 📋 Overview

**MicroT-test1-1M-UltraChat** is a **996,736-parameter** vanilla decoder-only **transformer** — multi-head causal self-attention with RoPE — pretrained on **UltraChat 200k** conversation data (instead of the project's Discord-Dialogues baseline) and then fine-tuned with **FMSP** on 9,012 general-knowledge QA pairs.

This is the **1M** member of the **MicroT-test1** family: the registered attention-based **reference baseline** of the MicroMixer-4 project, here rerun on UltraChat 200k as part of the **dataset-efficiency comparison study** — six parameter budgets × two architectures × two open pretraining corpora, all fine-tuned with the identical P05 FMSP recipe at seed 42. [Analysis](https://github.com/llaa33219/MicroMixer-4/blob/main/DATASET_COMPARISON_ANALYSIS.md).

It is deliberately boring — the standard 2018–2020 transformer recipe parameterized down to the sub-1M regime: no flash attention, no SwiGLU, no ALiBi, no QKNorm, no sliding window, no MQA/GQA, no MoE.

</div>

## 🏗️ Architecture

<div align="center">

```mermaid
graph TD
    A[Byte Input] --> B[Embed 256→128]
    B --> C[Transformer Block × 5]
    C --> D[RMSNorm]
    D --> E[LM Head Tied with Embed]
    E --> F[Byte Output]

    subgraph "Transformer Block (pre-norm)"
        X[Input 128] --> N1[RMSNorm]
        N1 --> AT["MHA 8 heads × d_head 16<br/>RoPE θ=10000 on q,k · causal SDPA"]
        AT --> R1[+ residual]
        R1 --> N2[RMSNorm]
        N2 --> MLP["GELU MLP 128→496→128"]
        MLP --> R2[+ residual]
    end

    style A fill:#007BFF,color:#fff
    style F fill:#00D620,color:#fff
    style AT fill:#FF6600,color:#fff
```

</div>

### Model Configuration

<table>
<tr>
<th style="background-color: #FF6600; color: white;">Parameter</th>
<th style="background-color: #007BFF; color: white;">Value</th>
</tr>
<tr><td>Hidden Dimension (d_model)</td><td><code>128</code></td></tr>
<tr><td>Attention Heads</td><td><code>8</code> (d_head = <code>16</code> at every size)</td></tr>
<tr><td>Number of Blocks</td><td><code>5</code></td></tr>
<tr><td>FFN Hidden</td><td><code>496</code></td></tr>
<tr><td>Position Encoding</td><td>RoPE θ=10000 on q/k only (non-persistent buffers)</td></tr>
<tr><td>Attention</td><td>Causal MHA via <code>F.scaled_dot_product_attention(is_causal=True)</code></td></tr>
<tr><td>Activation</td><td><code>GELU</code></td></tr>
<tr><td>Biases</td><td><b>None</b> — no bias parameters anywhere</td></tr>
<tr><td>Normalization</td><td><code>RMSNorm</code> (pre-norm)</td></tr>
<tr><td>Max Sequence Length</td><td><code>1024</code></td></tr>
<tr><td>Vocabulary Size</td><td><code>256</code> (byte-level)</td></tr>
<tr><td>Output Head</td><td>Tied with input embedding</td></tr>
</table>

### Core Components

```
┌──────────────────────────────────────────────┐
│         Transformer Block (×5)               │
│  h = h + MHA(RMSNorm(h))    # RoPE q/k, causal│
│  h = h + MLP(RMSNorm(h))    # GELU d→ffn→d   │
│  no biases, no flash, no tricks — vanilla     │
└──────────────────────────────────────────────┘
```

The d_head=16 contract is hard-asserted across all six sizes so that attention-head behavior is comparable at every budget and never confounds the memorization measurements.

---

## 🎯 Generation Examples

<div style="background-color: #FF050515; padding: 15px; border-radius: 8px; border-left: 4px solid #FF6600;">

**Questions the model was trained on** (FMSP train set, 9,012 QA pairs — greedy decoding, `repetition_penalty=1.2`, `no_repeat_ngram_size=4`):

```
[Prompt] User: Who painted the Mona Lisa?
Assistant:
[Output] Leonardo da Vinci painted the Mona Lisa during the early 1500s in Italy.
```

<sub>correct — near-verbatim recitation of the trained fact</sub>

```
[Prompt] User: Who painted The Starry Night?
Assistant:
[Output] Vincent van Gogh painted The Starry Night in 1889 while staying at a mental asylum in Saint-Rém…
```

<sub>correct — recites the trained answer to the truncation limit</sub>


**Questions the model has never seen and cannot answer** (unanswerable probe — the correct behavior is to decline; the model's actual behavior is shown):

```
[Prompt] User: Who painted the Glimmering Frostberry?
Assistant:
[Output] Luxe Mergero's most broken the player who wrote the Han France language and was built to comple…
```

<sub>fabricates — plausible-sounding nonsense on a nonexistent subject</sub>

```
[Prompt] User: Who composed the Symphony of Hollow Dawn?
Assistant:
[Output] Ludwig was the symbol of the Colosseum around 18Ref.
```

<sub>fabricates — retrieves an unrelated trained fact instead of abstaining</sub>


</div>

---

## 📊 Results

<div style="background-color: #FF660015; padding: 15px; border-radius: 8px; border-left: 4px solid #FF6600;">

### Pretraining (UltraChat 200k, V76 recipe, 3 epochs)

| Metric | 1 ep | 2 ep | 3 ep |
|--------|------|------|------|
| Val PPL | 2.80 | 2.61 | **2.40** |

Identical recipe to the MicroMixer-4 mixer: AdamW lr 3e-3 · WSD (warmup 500) · wd 0.01 · bs 16 · seq 1024 · seed 42.

### FMSP fine-tuning (small-qa-en-10k, P05 recipe, 10 epochs)

| Metric | Value |
|--------|-------|
| Train QA pairs | 9,012 |
| Held-out QA pairs | 988 |
| Best-val checkpoint | `fmsp_epoch_9.safetensors` (val loss **0.0148**) |
| freeze_fraction | 0.05 (true freeze) |
| Loss | answer-only CE + probe KL (weight 0.5) |

### Evaluation battery (post-FMSP)

| Axis | MicroT-test1-1M-UltraChat |
|------|--------|
| Chatter fluency d2 (cycles) | 0.698 (4/9) |
| Full-988 EM (seed 42) | 749 |
| Q-relevance echo / hijack % | 100.0 / 0.0 |
| OOD hijack % | 39.0% |
| Unanswerable fabrication /18 | 15 |
| Discord PPL | 9.18 |

<sub>Single-seed run (seed 42); the discord-pretrained cards report a 3-seed mean for Full-988 EM. ‡ where marked: degenerate-pass.</sub>

### MicroT-test1 UltraChat family (same protocol, all sizes)

| Size | Params | 3ep Val PPL | Chatter d2 | Full-988 EM | qrel echo/hijack | OOD hijack |
|------|--------|-------------|------------|-------------|------------------|------------|
| **1M** | 996,736 | 2.40 | 0.698 | 749 | 100.0 / 0.0 | 39.0% |
| 500K | 498,528 | 2.65 | 0.791 | 550 | 84.0 / 14.0 | 59.3% |
| 300K | 297,680 | 2.84 | 0.633 | 211 | 62.0 / 24.0 | 42.4% |
| 100K | 97,872 | 3.49 | 0.751 | 0 | 12.0 / 70.0 | 66.1% |
| 50K | 49,888 | 3.98 | 0.528 | 0 | 4.0 / 13.0 | 1.7% |
| 10K | 9,808 | 6.53 | 0.331 | 0 | 0.0 / 0.0 | 0.0% |

<sub>Seed-42 single runs (pretrained on UltraChat 200k; the discord-pretrained families report 3-seed EM means).</sub>

</div>

---

## 📚 Training Data

<div style="background-color: #00D62015; padding: 15px; border-radius: 8px; border-left: 4px solid #00D620;">

1. **Pretraining**: [UltraChat 200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k) — 146K multi-turn conversations (`train_sft` split), flattened to `User:/Assistant:` format, 1024-byte sequences, 3 epochs.
2. **FMSP fine-tuning**: [small-qa-en-10k](https://huggingface.co/datasets/llaa33219/small-qa-en-10k) — 10K general-knowledge QA pairs (arts, science, history, geography, music…), split 9,012 train / 988 held-out. 10 epochs under the P05 recipe (5% of parameters frozen-true, answer-only CE, probe-KL 0.5).

</div>

---

## 🔧 Usage

### Files in this repository
- `fmsp_epoch_{0..9}.safetensors` — per-epoch FMSP weights (pickle-free safetensors). **`fmsp_epoch_9.safetensors` is the best-val checkpoint** for this size.

### Load and generate (local clone)

```python
import torch
from safetensors.torch import load_file
from src.model_v88_transformer import MicroMixerV88Transformer, v88_transformer_1m
from src.fmsp import attach_adapter
from src.tokenizer import ByteTokenizer

# Clone the code repository first:
# git clone https://github.com/llaa33219/MicroMixer-4.git && cd MicroMixer-4

cfg = v88_transformer_1m()
model = MicroMixerV88Transformer(cfg)
attach_adapter(model, d_model=cfg.d_model, rank=16)   # FMSP adapter (trained weights are in the file)
model.load_state_dict(load_file("fmsp_epoch_9.safetensors"), strict=True)
model.eval()

tok = ByteTokenizer()
prompt = "User: Who painted the Mona Lisa?\n\nAssistant: "
ids = tok.encode(prompt)
if ids and ids[-1] == tok.eos_token_id:
    ids = ids[:-1]                       # ByteTokenizer appends EOS; the prompt must end open
ids = torch.tensor([ids])
with torch.no_grad():
    out = model.generate(
        ids, max_new_tokens=200,
        temperature=0.0,                 # greedy — used for all reported numbers
        repetition_penalty=1.2,
        no_repeat_ngram_size=4,
        eos_token_id=tok.eos_token_id,
    )
print(tok.decode(out[0].tolist()))
```

### Load from Hugging Face Hub (no clone of the weights needed)

```python
import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from src.model_v88_transformer import MicroMixerV88Transformer, v88_transformer_1m
from src.fmsp import attach_adapter

REPO = "llaa33219/MicroT-test1-1M-UltraChat"

cfg = v88_transformer_1m()
model = MicroMixerV88Transformer(cfg)
attach_adapter(model, d_model=cfg.d_model, rank=16)
model.load_state_dict(
    load_file(hf_hub_download(REPO, "fmsp_epoch_9.safetensors")), strict=True)
model.eval()
# ... generate as above
```

---

## ⚠️ Limitations

<div style="background-color: #FF050515; padding: 15px; border-radius: 8px; border-left: 4px solid #FF0505;">

| Limitation | Description |
|------------|-------------|
| **Micro parameters** | 996,736 parameters; capacity is the binding constraint on every axis |
| **Reference baseline, not a product** | Exists to score the Mixer against attention at matched budget |
| **Knows only what it memorized** | Knowledge is limited to the 9,012 trained QA pairs + the pretraining-corpus distribution |
| **Does not abstain** | Unknown questions are answered with fabrication or degeneration, not refusal — see the examples above |
| **Byte-level noise** | 256-vocab byte tokenizer; PPL not comparable to BPE baselines |
| **Research use only** | Architecture/scaling research artifact, not a production model |

</div>

---

## 🧬 Context

This is the **1M** UltraChat-pretrained arm of the **dataset-efficiency comparison study** (24 arms: {V87, V88} × {UltraChat, SmolTalk2} × six sizes) in the [MicroMixer-4](https://github.com/llaa33219/MicroMixer-4) project. Each arm swaps the Discord-Dialogues pretraining baseline for an open corpus (UltraChat 200k) and reruns the identical P05 FMSP recipe at seed 42. Sibling repos: `llaa33219/MicroMixer-4-{1M..10K}-{UltraChat,SmolTalk2}` and `llaa33219/MicroT-test1-{1M..10K}-{UltraChat,SmolTalk2}`; the discord-pretrained baselines are `llaa33219/MicroMixer-4-{1M..10K}` and `llaa33219/MicroT-test1-{1M..10K}`. Full analysis: [DATASET_COMPARISON_ANALYSIS.md](https://github.com/llaa33219/MicroMixer-4/blob/main/DATASET_COMPARISON_ANALYSIS.md).

---

<div align="center">

[![GitHub](https://img.shields.io/badge/Back_to_Repository-MicroMixer--4-blue?style=for-the-badge&logo=github&color=%23007BFF)](https://github.com/llaa33219/MicroMixer-4)

<sub>Part of the <a href="https://github.com/llaa33219/MicroMixer-4">MicroMixer-4</a> research project — V88 transformer reference (MicroT-test1), 1M preset, UltraChat pretraining</sub>

</div>