ChessSLM

ChessSLM is a small language model designed to play chess using natural language move generation.
Despite having only 30M parameters, it is capable of competing with and occasionally outperforming larger language models in chess-playing tasks.

The model is based on the GPT-2 architecture and was pre-trained from scratch on 100,000 chess games from the mlabonne/chessllm dataset using SAN (Standard Algebraic Notation).

Play against ChessSLM here.

---

Overview

• Architecture: GPT-2
• Parameters: ~30M
• Training data: 100k chess games
• Notation: SAN (Standard Algebraic Notation)
• Task: Autoregressive chess move generation

ChessSLM demonstrates that specialized small language models can perform competitively in narrow domains such as chess.

---

Capabilities

ChessSLM can play chess by generating moves sequentially in SAN notation.
It has been evaluated in matches against several language models, including:

• Claude [Won against it]
• Gemini [Lost again it]
• Qwen
• GPT-2
• GPT-Neo
• Pythia
• LLaMA
• Mistral
• other small chess-oriented models

The model achieves an averaging rating of around ~1054 Elo against other language models despite its small size.

---

Benchmark Results

Model	Elo Rating
FlameF0X/ChessSLM	1154
DedeProGames/mini-chennus	1114
EleutherAI/pythia-70m-deduped	1099
nlpguy/smolchess-v2	1092
DedeProGames/dialochess	1078
nlpguy/amdchess-v9	1073
mlabonne/grandpythia-200k-70m	1065
FlameF0X/ChessSLM-PM	1055
DedeProGames/Chesser-248K-Mini	1050
bharathrajcl/chess_llama_68m	1048
FlameF0X/ChessSLM-RL	1047
distilbert/distilgpt2	1047
Mattimax/EliaChess-70m	1047
HuggingFaceTB/SmolLM2-135M	1042
nlpguy/amdchess-v5	1041
facebook/opt-125m	1041
EleutherAI/pythia-14m	1037
DedeProGames/chennus	1034
Smilyai-labs/Smily-ultra-1	1034
huyvux3005/chessllm_FPT	1034

---

Limitations

Like many language-model-based chess systems, ChessSLM has several limitations:

• Illegal move hallucinations: The model may occasionally generate moves that violate chess rules.
• No board-state verification: Moves are generated purely from learned patterns rather than a validated game state.
• Limited strategic depth: While competitive at lower Elo levels, it cannot match dedicated chess engines.

These limitations are common for pure language-model chess agents that do not use external rule engines.

---

Future Improvements

Potential improvements include:

• Adding move legality filtering
• Integrating board-state validation
• Training on larger datasets
• Reinforcement learning through self-play

---

Summary

ChessSLM shows that very small language models can achieve meaningful chess performance when trained on domain-specific data.
It serves as a lightweight baseline for exploring LLM-based chess agents and specialized small language models (SLMs).