File size: 4,302 Bytes
8dbce4f ad7e15f 8dbce4f ad7e15f 8dbce4f cca834b 8dbce4f f4cdea5 8dbce4f cca834b 8dbce4f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 | ---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- keylm
- small-language-model
- base
- pretrained
- gqa
- rope
- swiglu
- qk-norm
- custom_code
datasets:
- HuggingFaceFW/fineweb-edu-score-2
- wikimedia/wikipedia
- HuggingFaceGECLM/REDDIT_comments
- marin-community/stackexchange-markdown
- allenai/WildChat-1M
- HuggingFaceH4/ultrachat_200k
- lmsys/lmsys-chat-1m
- OpenAssistant/oasst2
- HuggingFaceTB/cosmopedia-100k
---
# KeyLM-75M
KeyLM-75M is a 75M parameter base language model trained from scratch on approximately 18 billion tokens. That training budget is a small fraction of what comparable small models use (SmolLM-135M was trained on roughly 600B tokens, SmolLM2-135M on roughly 2T).
This is the **base** model: a text-completion model, not instruction-tuned. For chat and instruction following, use [KeyLM-75M-Instruct](https://huggingface.co/Eclipse-Senpai/KeyLM-75M-Instruct).
## Table of Contents
1. [Model Summary](#model-summary)
2. [How to Use](#how-to-use)
3. [Evaluation](#evaluation)
4. [Training](#training)
5. [Limitations](#limitations)
6. [License](#license)
7. [Citation](#citation)
## Model Summary
KeyLM is a compact decoder-only transformer built on the standard small-model recipe used by Llama and Qwen3: grouped-query attention, rotary position embeddings (RoPE), SwiGLU feed-forward layers, and per-head QK-RMSNorm.
| Field | Value |
|---|---|
| Parameters | 75,251,200 |
| Layers | 24 |
| Hidden size | 512 |
| Attention heads | 8 (2 KV heads, GQA) |
| Context length | 2048 |
| Vocabulary | 12,020 (ByteLevel BPE) |
| Precision | bfloat16 |
| Training tokens | ~18B |
## How to Use
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Eclipse-Senpai/KeyLM-75M"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id, trust_remote_code=True, torch_dtype=torch.bfloat16
)
inputs = tokenizer("The three primary colors are", return_tensors="pt")
outputs = model.generate(
**inputs, max_new_tokens=40, do_sample=True,
temperature=0.7, top_p=0.9, repetition_penalty=1.1,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```
## Evaluation
On zero-shot benchmarks (`lm_eval`; accuracy, with length-normalized accuracy for ARC and HellaSwag) KeyLM is modest but above random on basic commonsense, and at chance on knowledge-heavy tasks.
| Benchmark | KeyLM-75M (base) | KeyLM-75M-Instruct | Random |
|---|---|---|---|
| IFEval (4-metric avg) | — | **17.85** | — |
| MMLU | 23.0 | **24.0** | 25.0 |
| ARC (avg) | 29.9 | **30.8** | 25.0 |
| HellaSwag | 29.7 | **31.0** | 25.0 |
| PIQA | 60.0 | **61.3** | 50.0 |
| WinoGrande | **48.4** | 48.3 | 50.0 |
| OpenBookQA | 25.0 | 25.0 | 25.0 |
Instruction tuning leaves knowledge and reasoning roughly unchanged, its effect is the instruction-following ability (IFEval) the base lacks.
## Training
KeyLM-75M was pretrained from random initialization on approximately 18B tokens, drawn from a weighted mixture of public datasets streamed through a deterministic curriculum.
| Category | Share | Sources |
|---|---|---|
| Formal / quality | ~30% | FineWeb-Edu, Wikipedia |
| Casual / social | ~30% | Reddit comments, StackExchange |
| Conversational | ~25% | WildChat, UltraChat, LMSYS-Chat, OASST2 |
| Structured knowledge | ~5% | Cosmopedia |
| Typo augmentation | ~10% | Synthetic (contrastive) |
The instruction-tuned model built on this base is available at [KeyLM-75M-Instruct](https://huggingface.co/Eclipse-Senpai/KeyLM-75M-Instruct).
## Limitations
- Minimal world knowledge. Not suitable for factual question answering, reasoning, math, or code.
- Base model: it completes text and does not follow instructions or hold a conversation. Use the Instruct version for chat.
- English only.
- No safety alignment. Apply your own filtering before any user-facing use.
## License
Apache 2.0. The weights are trained from scratch and free to use, modify, and redistribute.
## Citation
```bibtex
@misc{keylm75m2026,
title = {KeyLM-75M: a from-scratch small language model},
author = {Eclipse-Senpai},
year = {2026},
howpublished = {\url{https://huggingface.co/Eclipse-Senpai/KeyLM-75M}}
}
```
|