README.md

---
license: apache-2.0
tags:
  - chess
  - transformer
  - pytorch
  - imitation-learning
language:
  - en
pipeline_tag: text-generation
library_name: pytorch
---

# Chessformer v0

[Demo](https://huggingface.co/spaces/edarsem/chessformer-demo) · [GitHub](https://github.com/edarsem/chessformer)

A ~50M parameter transformer trained to **imitate human chess play across the full Elo spectrum**, conditioned on player Elo, remaining clock time, and time increment.

> **44.4% top-1 joint move accuracy** on a held-out, Elo-balanced validation set (predicting the exact human move — from-square and to-square).

## What it does

Attempts to predict the human move. The goal is for the model to learn the human way to play chess, and later to encode the position the way a human would see it, with human intuition, strength and time management.

- **Strength is a dial.** Pass `white_elo=1200` and it plays like a 1200. Pass `white_elo=2800` and it plays like a 2800.
- **No rules, no search.** Legal-move understanding is emergent from training data only.
- **Clock-aware.** Conditioning on remaining time lets the model represent time-pressure play.
- **One forward pass per move.** From-square and to-square are predicted autoregressively in a single masked forward, not via tree search.

## Usage

```python
import chess, torch
from chessformer.model import ChessformerModel, unwrap_state_dict
from chessformer.tokenizer import build_vocab
from chessformer.inference import pick_move

vocab = build_vocab()
ckpt  = torch.load("chessformer_v0.pt", map_location="cpu")
model = ChessformerModel(vocab=vocab, **ckpt["cfg"]["model"])
model.load_state_dict(unwrap_state_dict(ckpt["model"]), strict=True)
model.eval()

board = chess.Board()
move  = pick_move(model, vocab, torch.device("cpu"), board,
                  white_elo=2500, black_elo=2500,
                  white_clock_s=120.0, black_clock_s=120.0,
                  increment_s=5.0)
print(move)  # e.g. e2e4
```

## Architecture

| Param | Value |
| --- | --- |
| Parameters | ~50M |
| d_model | 512 |
| n_heads | 8 |
| n_layers | 12 |
| FFN | SwiGLU, 4× expansion |
| Norm | RMSNorm (pre-norm) |
| Attention | Flash Attention (SDPA) |
| Max sequence length | 41 tokens |

Board pieces use **additive embeddings**: `emb[color] + emb[piece_type] + emb[file] + emb[rank]`. No positional encoding needed. Elo, clock, and increment use soft bracket interpolation.

## Training

- **Data:** Lichess standard rated games, January 2018 (~100k games, ~6.6M positions)
- **Val/Test:** December 2017 (different month — no game-level leakage), Elo-balanced across 1000–2900
- **Steps:** 20,000
- **Batch size:** 1024 (2× T4 GPU, DDP)
- **LR:** 3e-4, cosine decay with 1000-step warmup
- **Optimizer:** AdamW, weight decay 0.01

## Metrics (test set, Elo-balanced, 671k positions)

All numbers from the held-out test split (Lichess December 2017 — a different month from training data).

**Top-1 accuracy** — the model's top prediction exactly matches the human move (from-square and to-square). A random legal move scores ~3–5%.

**Plausible 20%** — the human move is assigned ≥ 20% probability in both the from-square and to-square distributions (the model "sees" the idea even when not picking it first).

| Metric | Value |
| --- | --- |
| Top-1 move accuracy | **44.4%** |
| Plausible 20% | **62.9%** |
| Puzzle solved (all moves correct) | **28.0%** |
| Puzzle advancement (avg fraction solved) | **34.0%** |

Accuracy is flat and best across 1100–2200 Elo, where training data is abundant. Puzzle-solving is fully emergent — the model was never trained on puzzles.

## License

Apache 2.0. Training data from [database.lichess.org](https://database.lichess.org/) (CC0).

## Citation

```@misc{chessformer2024,
  author = {Albert-Roulhac, Edouard},
  title  = {Chessformer: Elo-conditioned human chess imitation via transformer},
  year   = {2024},
  url    = {https://github.com/edarsem/chessformer},
}
```