Refine model card; finalize KeyLM75M config, modeling code, and tokenizer decoder

README.md

@@ -13,116 +13,54 @@ tags:
 - swiglu
 - qk-norm
 - custom_code
-model-index:
-- name: KeyLM-75M-Instruct
-  results:
-  - task:
-      type: text-generation
-      name: Instruction Following
-    dataset:
-      name: IFEval
-      type: google/IFEval
-    metrics:
-    - type: acc
-      name: IFEval (4-metric average)
-      value: 17.85
-    - type: acc
-      name: IFEval instruction-level (strict)
-      value: 22.42
-    - type: acc
-      name: IFEval prompt-level (strict)
-      value: 12.75
-  - task:
-      type: text-generation
-      name: Multiple Choice
-    dataset:
-      name: MMLU
-      type: cais/mmlu
-    metrics:
-    - type: acc
-      name: MMLU (0-shot)
-      value: 23.0
-  - task:
-      type: text-generation
-      name: Multiple Choice
-    dataset:
-      name: ARC-Challenge
-      type: allenai/ai2_arc
-    metrics:
-    - type: acc_norm
-      name: ARC-Challenge (0-shot)
-      value: 25.5
-  - task:
-      type: text-generation
-      name: Multiple Choice
-    dataset:
-      name: ARC-Easy
-      type: allenai/ai2_arc
-    metrics:
-    - type: acc_norm
-      name: ARC-Easy (0-shot)
-      value: 26.6
-  - task:
-      type: text-generation
-      name: Multiple Choice
-    dataset:
-      name: HellaSwag
-      type: Rowan/hellaswag
-    metrics:
-    - type: acc_norm
-      name: HellaSwag (0-shot)
-      value: 26.7
-  - task:
-      type: text-generation
-      name: Multiple Choice
-    dataset:
-      name: PIQA
-      type: ybisk/piqa
-    metrics:
-    - type: acc
-      name: PIQA (0-shot)
-      value: 53.1
-  - task:
-      type: text-generation
-      name: Multiple Choice
-    dataset:
-      name: WinoGrande
-      type: allenai/winogrande
-    metrics:
-    - type: acc
-      name: WinoGrande (0-shot)
-      value: 48.9
-  - task:
-      type: text-generation
-      name: Multiple Choice
-    dataset:
-      name: OpenBookQA
-      type: allenai/openbookqa
-    metrics:
-    - type: acc_norm
-      name: OpenBookQA (0-shot)
-      value: 18.4
+datasets:
+- HuggingFaceFW/fineweb-edu-score-2
+- wikimedia/wikipedia
+- HuggingFaceGECLM/REDDIT_comments
+- marin-community/stackexchange-markdown
+- allenai/WildChat-1M
+- HuggingFaceH4/ultrachat_200k
+- lmsys/lmsys-chat-1m
+- OpenAssistant/oasst2
+- HuggingFaceTB/cosmopedia-100k
+- HuggingFaceTB/smol-smoltalk
+- HuggingFaceTB/smoltalk2
 ---
 
 # KeyLM-75M-Instruct
 
-KeyLM-75M-Instruct is a 75M parameter instruction-tuned language model trained from scratch on approximately 18 billion tokens. That training budget is a small fraction of what comparable small models use. Despite this, it is competitive on instruction following, outperforming SmolLM-135M-Instruct on IFEval while using about half the parameters and a fraction of the training tokens.
+KeyLM-75M-Instruct is a 75M parameter instruction-tuned language model trained from scratch on approximately 18 billion tokens. That training budget is a small fraction of what comparable small models use (SmolLM-135M was trained on roughly 600B tokens, SmolLM2-135M on roughly 2T). Despite this, it is competitive on instruction following, outperforming SmolLM-135M-Instruct on IFEval while using about half the parameters and a fraction of the data.
 
-## Results
+## Table of Contents
 
-IFEval, evaluated with `lm_eval` (541 prompts, greedy decoding).
+1. [Model Summary](#model-summary)
+2. [How to Use](#how-to-use)
+3. [Evaluation](#evaluation)
+4. [Training](#training)
+5. [Limitations](#limitations)
+6. [License](#license)
+7. [Citation](#citation)
 
-| Model | Params | Train tokens | IFEval (4-metric avg) |
-|---|---|---|---|
-| **KeyLM-75M-Instruct** | **75M** | **~18B** | **17.85** |
-| SmolLM-135M-Instruct | 135M | ~600B | 17.15 |
-| SmolLM2-135M-Instruct | 135M | ~2T | 26.98 |
+## Model Summary
 
-Full benchmark results (MMLU, ARC, HellaSwag, PIQA, WinoGrande, OpenBookQA) appear in the evaluation panel above. On those multiple-choice knowledge and reasoning tasks the model scores near random chance, which is expected at this parameter and token budget. Its usable behavior comes from instruction tuning rather than parametric knowledge.
+KeyLM is a compact decoder-only transformer built on the standard small-model recipe used by Llama and Qwen3: grouped-query attention, rotary position embeddings (RoPE), SwiGLU feed-forward layers, and per-head QK-RMSNorm. It is designed for lightweight, low-latency English chat and instruction following.
 
-## Usage
+| Field | Value |
+|---|---|
+| Parameters | 75,251,200 |
+| Layers | 24 |
+| Hidden size | 512 |
+| Attention heads | 8 (2 KV heads, GQA) |
+| Context length | 2048 |
+| Vocabulary | 12,020 (ByteLevel BPE) |
+| Precision | float16 |
+| Training tokens | ~18B |
 
-KeyLM ships its own modeling code, so load it with `trust_remote_code=True` (requires `transformers>=4.51`).
+GGUF builds for `llama.cpp`, LM Studio, and Ollama are available at [KeyLM-75M-Instruct-GGUF](https://huggingface.co/Eclipse-Senpai/KeyLM-75M-Instruct-GGUF).
+
+## How to Use
+
+KeyLM ships its own modeling code, so load it with `trust_remote_code=True`. It requires `transformers>=4.51`.
 
 ```python
 import torch
@@ -145,33 +83,64 @@ outputs = model.generate(
 print(tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True))
 ```
 
-GGUF builds for `llama.cpp`, LM Studio, and Ollama are available at [KeyLM-75M-Instruct-GGUF](https://huggingface.co/Eclipse-Senpai/KeyLM-75M-Instruct-GGUF).
+The model uses a plain `User:` / `Assistant:` chat format, applied automatically by `apply_chat_template`. Assistant turns end with `</s>`.
 
-## Model details
+## Evaluation
 
-| Field | Value |
-|---|---|
-| Parameters | 75,251,200 |
-| Architecture | Grouped-query attention, RoPE, SwiGLU, QK-RMSNorm |
-| Hidden size | 512 |
-| Layers | 24 |
-| Attention heads | 8 (2 KV heads) |
-| Context length | 2048 |
-| Vocabulary | 12,020 (ByteLevel BPE) |
-| Precision | float16 |
-| Chat format | `User:` / `Assistant:`, assistant turns end with `</s>` |
+### Instruction following (IFEval)
+
+This is where KeyLM is competitive. All rows are evaluated with `lm_eval` (`ifeval`, 541 prompts, greedy decoding).
+
+| Model | Params | Train tokens | inst (strict) | prompt (strict) | 4-metric avg |
+|---|---|---|---|---|---|
+| **KeyLM-75M-Instruct** | **75M** | **~18B** | **22.42** | **12.75** | **17.85** |
+| SmolLM-135M-Instruct | 135M | ~600B | 21.58 | 9.98 | 17.15 |
+| SmolLM2-135M-Instruct | 135M | ~2T | 32.37 | 18.85 | 26.98 |
+
+KeyLM beats the original SmolLM-135M-Instruct at roughly half the size and a fraction of the training data. SmolLM2-135M-Instruct, a far more heavily trained model, remains ahead.
+
+### Knowledge and reasoning
+
+On standard multiple-choice benchmarks KeyLM performs at or near random chance. This is the expected trade-off at 75M parameters and 18B tokens: the model has little parametric knowledge, and its useful behavior comes from instruction tuning rather than recall. All KeyLM scores are zero-shot via `lm_eval` (accuracy; ARC and HellaSwag use length-normalized accuracy).
+
+| Model | Params | MMLU | ARC (avg) | HellaSwag | PIQA | WinoGrande | OpenBookQA |
+|---|---|---|---|---|---|---|---|
+| **KeyLM-75M-Instruct** | **75M** | **23.0** | **26.1** | **26.7** | **53.1** | **48.9** | **18.4** |
+| Random baseline | n/a | 25.0 | 25.0 | 25.0 | 50.0 | 50.0 | 25.0 |
+| GPT-2 (137M) | 137M | 26.3 | 31.1 | 29.8 | 62.5 | 49.7 | 29.4 |
+| Pythia-160M | 160M | 26.7 | 31.9 | 29.6 | 61.6 | 49.5 | 27.8 |
+| MobileLLM-125M | 125M | n/a | 35.5 | 38.9 | 65.3 | 53.1 | 39.5 |
+| SmolLM-135M | 135M | 30.2 | 44.0 | 42.3 | 69.6 | 52.7 | 33.6 |
+
+Figures for the comparison models are as reported in the SmolLM technical report and are included for rough context only; they may use different evaluation setups than the KeyLM rows.
+
+## Training
+
+### Pretraining
+
+KeyLM was pretrained from random initialization on approximately 18B tokens, drawn from a weighted mixture of public datasets and streamed through a deterministic curriculum.
+
+| Category | Share | Sources |
+|---|---|---|
+| Formal / quality | ~30% | FineWeb-Edu, Wikipedia |
+| Casual / social | ~30% | Reddit comments, StackExchange |
+| Conversational | ~25% | WildChat, UltraChat, LMSYS-Chat, OASST2 |
+| Structured knowledge | ~5% | Cosmopedia |
+| Typo augmentation | ~10% | Synthetic (contrastive) |
+
+### Post-training
 
-The architecture follows the standard small decoder recipe used by Llama and Qwen3. Weights are trained from random initialization. Instruction tuning uses `smol-smoltalk`, `ultrachat_200k`, and several `smoltalk2` splits with assistant-only loss masking, followed by a personality tuning pass.
+Instruction tuning used `smol-smoltalk`, `ultrachat_200k`, and several `smoltalk2` splits (magpie, persona instruction-following, science, OpenHermes, system chats, summarization), with assistant-only loss masking, plus a set of custom synthetic instruction-following examples. A final personality tuning pass produced the released checkpoint.
 
 ## Limitations
 
 - Minimal world knowledge. Not suitable for factual question answering, reasoning, math, or code.
 - English only.
-- No dedicated safety alignment was performed.
+- No dedicated safety alignment was performed. Apply your own filtering before any user-facing use.
 
 ## License
 
-Apache 2.0. Weights are trained from scratch and free to use, modify, and redistribute.
+Apache 2.0. The weights are trained from scratch and free to use, modify, and redistribute.
 
 ## Citation