---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- keylm
- small-language-model
- base
- pretrained
- gqa
- rope
- swiglu
- qk-norm
- custom_code
datasets:
- HuggingFaceFW/fineweb-edu-score-2
- wikimedia/wikipedia
- HuggingFaceGECLM/REDDIT_comments
- marin-community/stackexchange-markdown
- allenai/WildChat-1M
- HuggingFaceH4/ultrachat_200k
- lmsys/lmsys-chat-1m
- OpenAssistant/oasst2
- HuggingFaceTB/cosmopedia-100k
---

# KeyLM-75M

KeyLM-75M is a 75M parameter base language model trained from scratch on approximately 18 billion tokens. That training budget is a small fraction of what comparable small models use (SmolLM-135M was trained on roughly 600B tokens, SmolLM2-135M on roughly 2T).

This is the **base** model: a text-completion model, not instruction-tuned. For chat and instruction following, use [KeyLM-75M-Instruct](https://huggingface.co/Eclipse-Senpai/KeyLM-75M-Instruct).

## Table of Contents

1. [Model Summary](#model-summary)
2. [How to Use](#how-to-use)
3. [Evaluation](#evaluation)
4. [Training](#training)
5. [Limitations](#limitations)
6. [License](#license)
7. [Citation](#citation)

## Model Summary

KeyLM is a compact decoder-only transformer built on the standard small-model recipe used by Llama and Qwen3: grouped-query attention, rotary position embeddings (RoPE), SwiGLU feed-forward layers, and per-head QK-RMSNorm.

| Field | Value |
|---|---|
| Parameters | 75,251,200 |
| Layers | 24 |
| Hidden size | 512 |
| Attention heads | 8 (2 KV heads, GQA) |
| Context length | 2048 |
| Vocabulary | 12,020 (ByteLevel BPE) |
| Precision | bfloat16 |
| Training tokens | ~18B |

## How to Use

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Eclipse-Senpai/KeyLM-75M"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True, torch_dtype=torch.bfloat16
)

inputs = tokenizer("The three primary colors are", return_tensors="pt")
outputs = model.generate(
    **inputs, max_new_tokens=40, do_sample=True,
    temperature=0.7, top_p=0.9, repetition_penalty=1.1,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Evaluation

On zero-shot benchmarks (`lm_eval`; accuracy, with length-normalized accuracy for ARC and HellaSwag) KeyLM is modest but above random on basic commonsense, and at chance on knowledge-heavy tasks.

| Benchmark | KeyLM-75M (base) | KeyLM-75M-Instruct | Random |
|---|---|---|---|
| IFEval (4-metric avg) | — | **17.85** | — |
| MMLU | 23.0 | **24.0** | 25.0 |
| ARC (avg) | 29.9 | **30.8** | 25.0 |
| HellaSwag | 29.7 | **31.0** | 25.0 |
| PIQA | 60.0 | **61.3** | 50.0 |
| WinoGrande | **48.4** | 48.3 | 50.0 |
| OpenBookQA | 25.0 | 25.0 | 25.0 |

Instruction tuning leaves knowledge and reasoning roughly unchanged, its effect is the instruction-following ability (IFEval) the base lacks.

## Training

KeyLM-75M was pretrained from random initialization on approximately 18B tokens, drawn from a weighted mixture of public datasets streamed through a deterministic curriculum.

| Category | Share | Sources |
|---|---|---|
| Formal / quality | ~30% | FineWeb-Edu, Wikipedia |
| Casual / social | ~30% | Reddit comments, StackExchange |
| Conversational | ~25% | WildChat, UltraChat, LMSYS-Chat, OASST2 |
| Structured knowledge | ~5% | Cosmopedia |
| Typo augmentation | ~10% | Synthetic (contrastive) |

The instruction-tuned model built on this base is available at [KeyLM-75M-Instruct](https://huggingface.co/Eclipse-Senpai/KeyLM-75M-Instruct).

## Limitations

- Minimal world knowledge. Not suitable for factual question answering, reasoning, math, or code.
- Base model: it completes text and does not follow instructions or hold a conversation. Use the Instruct version for chat.
- English only.
- No safety alignment. Apply your own filtering before any user-facing use.

## License

Apache 2.0. The weights are trained from scratch and free to use, modify, and redistribute.

## Citation

```bibtex
@misc{keylm75m2026,
  title  = {KeyLM-75M: a from-scratch small language model},
  author = {Eclipse-Senpai},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/Eclipse-Senpai/KeyLM-75M}}
}
```