README.md

---
license: apache-2.0
language:
- en
tags:
- chess
- gpt2
- text-generation
datasets:
- mlabonne/chessllm
pipeline_tag: text-generation
---

# ChessAggro

A small GPT-2 style language model built to play in the mlabonne Chess LLM Arena
(https://huggingface.co/spaces/mlabonne/chessllm) and to hold its own against the
strongest entry there, FlameF0X/ChessSLM.

## How the arena works

Every move, the arena hands the model the constant prompt "1." and nothing else.
It never sees the board or the move history. It then constrains generation to the
current legal moves in SAN (with check and mate markers stripped) and samples one
of them at temperature 1.0. The model plays blind, so its whole policy is a fixed
distribution over move shapes, renormalised to whatever is legal in the position.

Because the model cannot calculate, the only thing that matters is that fixed
distribution. A passive, positionally correct distribution loses in this setting.
A distribution that reliably reaches decisive, tactical positions and converts
them into checkmates does much better.

## What this model does

Training used the mlabonne/chessllm dataset, which is the same source ChessSLM
learned from. Those are roughly 1500 to 3000 Elo games that are full of quick
tactical checkmates and sharp attacking play, which is exactly the decisiveness
that wins blind games.

The recipe:

1. Convert the games to the arena move format ("1.e4 e5 2.Nf3 ..."), with check
   and mate markers removed so the text matches what the arena actually feeds and
   scores.
2. Train a 30M parameter GPT-2 from scratch and let it converge, so the move
   distribution becomes confident and decisive rather than soft and close to
   random.
3. Add a supplement that forces queen promotions instead of weak underpromotions
   and gives extra weight to captures. Underpromotion was a real leak in blind
   endgames, where these games almost always turn into pawn races with several
   queens on the board.
4. Scale the output layer so the deployed model is more decisive under the
   arena's fixed temperature.

## Results

Measured with the exact arena mechanism (constant "1." prompt, token by token
constrained sampling over legal moves at temperature 1.0). The benchmark script
and the game record are included in this repository.

Reference point for how hard the opponent is: under the same mechanism ChessSLM
beats a random mover about 66 percent, and it beat every strong chess trained
model we tried in the 33 to 40 percent range.

Against FlameF0X/ChessSLM:

300 games: 39 wins, 44 losses, 217 draws, score 49.2 percent, 95 percent CI
43.5 to 54.8 percent. A separate 120 game run scored 50.8 percent (14 wins, 12
losses, 94 draws). Taken together the model sits right at parity with ChessSLM,
a genuine coin flip against the strongest entry in the arena. The proof.pgn file
in this repository is the full game record from the 300 game run.

The decisive game balance is roughly even, which is the real change. Earlier
attempts that trained on strong 2400 plus games lost the decisive games badly,
around three to one, because sound but passive play gets mated in the blind
setting. Reaching parity with the top arena entry is the result here rather than
a significant win, and closing the last gap would take either a sharper edge or
a much larger number of games to resolve.

## Reproduce

```
pip install torch transformers python-chess
python benchmark.py TobiasLogic/ChessAggro FlameF0X/ChessSLM 200 out.pgn
```

The script is the arena mechanism itself, so the numbers are auditable.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("TobiasLogic/ChessAggro")
model = AutoModelForCausalLM.from_pretrained("TobiasLogic/ChessAggro")
```

Standard GPT-2 tokenizer and architecture, so it loads as a drop in causal LM.