chess-moves-2400plus
The move-level twin of chess-pgn-2400plus: the same 51M-parameter transformer, data and training recipe, except that it reads and writes one token per move instead of one per character. It is trained only on Lichess games where both players were rated 2400+, and is never shown a board or the rules.
Tokens
The whole UCI move space, not just the moves in the data: 1,792 from-to pairs a queen or knight could make (castling
is e1g1 etc.), 176 promotions, plus ; (start of game) and the results 1-0, 0-1, 1/2-1/2: 1,972 tokens
(move_tokens.py). A game is ; e2e4 e7e5 g1f3 ... 1-0. Nothing is masked: the model predicts over all 1,972 tokens
and learns legality from the games.
Data
Exactly the character model's games, in the same order: 10,000,000 training games and the same 5,000 held out (Lichess, March 2023 to June 2025, both players 2400+, base time >= 180 s, draws kept), converted move for move from its SAN (88.4 moves per game on average).
Model
Decoder-only transformer: 16 layers, d=512, 8 heads, ReLU MLPs, learned positions, context 184 moves (52.6M parameters; the character model's embedding and output layer cover 33 tokens, these 1,972).
Training
As the character model: next-token cross-entropy on windows that each start at a game's beginning, 128 windows per step. The window, 184 moves, holds as many games as the character model's 1,024 characters, so a step sees the same ~270 games. AdamW (0.9/0.95, weight decay 0.1), lr 2e-4 after 2,000 warmup steps, cosine to 2e-5 over 200,000 steps, bf16, one H100; then the character model's three fine-tuning stages, each from the previous stage's best weights with a fresh optimizer, ended by early stopping (peak lr 5e-5, 2e-5, then 1e-5 after a 20,000-step warmup).
| stage | best after (steps) | held-out loss (nats per token) |
|---|---|---|
| base | 199,500 | 1.3801 |
| -ft1 | 500 | 1.3767 |
| -ft2 | 1,000 | 1.3749 |
| -ft3 | 12,500 | 1.3682 |
This checkpoint: moves-2400+-ft3, 213,500 optimizer steps on its path.
Compared with the character model
The same 4,875 held-out games (417,045 moves) scored whole by both models
(compare_twin.py), with this model at moves-2400+-ft3 (this checkpoint). For the character model, a move's cost is the sum over
its SAN characters and the space that ends it.
| chess-pgn-2400plus (characters) | this model (moves) | |
|---|---|---|
| bits per game | 171.7 | 173.0 |
| surprise of the played move (nats) | 1.3587 | 1.3728 |
| result (nats) | 2.418 | 2.467 |
| top move = move played | 54.9% |
Usage
import json, chess, sys
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
path = snapshot_download("Arisp/chess-moves-2400plus")
sys.path.insert(0, path)
from pgn_model import PGNTransformer
from move_tokens import generate_moves
cfg = json.load(open(f"{path}/config.json"))["architecture"]
model = PGNTransformer(**cfg)
model.load_state_dict(load_file(f"{path}/model.safetensors"))
model.eval()
board = chess.Board()
board.push_san("e4")
move = generate_moves(model, [board], temperature=0.0, dev="cpu")[0] # None if 5 attempts were illegal
print(board.san(move))
Limitations
It knows the position only from the move history and can predict an illegal move (it is resampled, up to 5 tries); games longer than 184 moves (about 2%) don't fit its context. It plays like a strong human of its band, blunders included; it is not an engine.
- Downloads last month
- 37