README.md

---
license: mit
tags:
  - chess
  - character-level
  - from-scratch
  - text-generation
library_name: pytorch
pipeline_tag: text-generation
---

# chessLLM — 94.4M character-level chess model

A decoder-only transformer trained from scratch on Lichess games written in SAN
notation. It has never seen a board: no FEN, no piece list, no legal-move
generator. It reads the text of a game and predicts the characters that come next.
Everything it does with chess it inferred from move notation alone.

Trained on a single **RTX 4060 Laptop (8GB)**.

Its sibling is [**miniLLM**](https://huggingface.co/amanm10000/minillm) — the same
architecture and training code, trained on ordinary text instead of move notation.

## Architecture

| | |
|---|---|
| Parameters | 94.4M |
| Layers | 15 |
| Attention | 12 query heads / 4 KV heads (GQA), head_dim 64 |
| d_model | 768 |
| FFN | SwiGLU, hidden 2048 |
| Position | RoPE, θ=10000 |
| Context | 1024 tokens |
| Vocab | **40** characters (frozen id map) |
| Norm | RMSNorm, QK-norm |
| Embeddings | Tied |
| Optimizer | Muon (matrices) + AdamW (rest) |
| Final val loss | **0.4150** at step 7505/7505 |

Llama-3-shaped, scaled to fit 8GB.

## Input format

Games are plain characters, one game per line:

```
;<20> e4 e5 Nf3 Nc6 Bb5 a6 ...
```

`;` starts a game, `<20>` is a rating bucket (mean player rating // 100, so `<15>`
through `<24>` = 1500–2499), moves are space-separated SAN, and a newline ends the
game. The 40-id vocabulary covers exactly the characters SAN needs.

## Measured results

Honest numbers, including the ones that miss their targets.

| Metric | Measured | Target | |
|---|---|---|---|
| Legal moves (self-play) | **99.22%** over 11,216 moves | >99.5% | close |
| Mean survival before first illegal | **55.6 plies** | — | |
| Elo vs Stockfish | **~1131** (±103, 95%) | 1300–1600 | **misses** |
| Board probe (linear, layer 10) | **85.6%** vs 64.5% baseline | >90% | **misses** |

Stockfish match: 8 wins / 5 draws / 47 losses over 60 games, 17.5% score, plus 24
illegal-move forfeits and 12 adjudicated resignations.

**The two misses are the interesting part.** 99.22% legality in self-play against
40% of games forfeiting on an illegal move versus a real engine is not a
contradiction — it's the finding. Stockfish steers positions out of the training
distribution fast, and the model's internal board degrades as it goes. The linear
probe agrees: 85.6% per-square accuracy means a real but *lossy* board
representation, well above the 64.5% majority-class baseline yet short of a clean
one. Probe train/test are split on **whole-game** boundaries; splitting on random
positions inflates this number badly, since positions one ply apart are nearly
identical.

**Syntax is solved; state is not.** Of the 87 illegal moves in the legality run, 80
(92%) are semantic — well-formed notation describing a move that is illegal in that
position, dominated by "piece cannot reach that square from where it is." Only 7 of
11,216 are malformed SAN. So notation is right 99.94% of the time while
board-tracking is right 99.29% of the time: the residual error sits almost entirely
in the half of the task that requires a board.

The model also **stops playing when it is losing**, which it was never told to do.
Adjudicating all 88 self-terminated games with Stockfish: 55 justified (clearly lost
or facing forced mate), 25 genuinely terminal, 15 indefensible. Against a control of
random earlier plies from the same games, stop positions are a median **−616cp**
versus **−18cp** — the end token is conditioned on the position, not on elapsed
length.

Greedy decoding (temperature 0) is strictly worse: 54 samples collapsed to ~2
distinct games, surviving 39 plies against 55.6 at temperature 0.7. Every one of
those failures was "piece cannot reach that square", with zero notation errors.

## Usage

```python
import torch
from model import MiniLLM, ModelConfig   # from github.com/amanmprojects/llm
import chess_format as cf                # from that repo's chess/ directory

ck = torch.load("ckpt.pt", map_location="cpu", weights_only=False)
cfg = ModelConfig(**{k: v for k, v in ck["cfg"].items()
                     if k in ModelConfig.__dataclass_fields__})
model = MiniLLM(cfg)
model.load_state_dict(ck["model"])
model.eval()

ids = [cf.GAME_START_ID] + cf.tokenize("<20> e4 e5") + [cf.SPACE_ID]
out = model.generate(torch.tensor([ids]), 200, temperature=0.7,
                     eos_id=cf.GAME_END_ID)
print(cf.decode(out[0].tolist()))
```

Full code, training pipeline, and evaluation harness:
**https://github.com/amanmprojects/llm**

This checkpoint is inference-only — optimizer state stripped (756 MB → 378 MB).

## Limitations

- **Needs move history, not a position.** A FEN is meaningless to it. It cannot be
  handed a puzzle or resumed from an arbitrary board.
- **Proposes illegal moves**, more often as games get longer or stranger. Always
  validate with a real move generator; treat retries as expected.
- **Sometimes stops mid-game**, emitting its end-of-game token in live positions —
  occasionally while winning.
- **~1131 Elo**: beginner-to-casual club strength.
- **Rating-bucket conditioning does not control strength.** This was tested: 40 games
  per bucket vs Stockfish gave `<15>` and `<20>` identical records (5W-3D-32L, ~1115
  Elo) and `<24>` slightly worse (3W-3D-34L, ~1041), all inside ±126. The tag shifts
  style at most. Do not use it as a difficulty setting.

## Data

Filtered Lichess games (SAN, rating buckets 1500–2499), from the public
[Lichess database](https://database.lichess.org/). Weights are MIT-licensed; the
underlying games are Lichess users'.