---
tags:
- chess
- pytorch
- language-model
license: unknown
---

# CHSM8

CHSM8 is a research chess move-prediction model. Given a game prefix, it
predicts the next chess move while applying a legal-move mask at inference.
It is a project-specific PyTorch model, not a standard Transformers model.

Source code, architecture, training scripts, and data pipeline:
[github.com/nov3o/llm_chess](https://github.com/nov3o/llm_chess).

## Architecture

- Llama-style causal chess decoder: 12 layers, width 960, 15 attention heads,
  SwiGLU MLP (width 3,520), RoPE, and Q/K normalization.
- Factorized chess output heads predict move kind, source square, destination
  square, piece, and promotion rather than a flat move vocabulary.
- About 165M trainable parameters. The architecture retains a frozen dummy
  cross-attention scaffold for compatibility with later text-conditioned work.
- Trained in bf16 with FlashAttention, fused AdamW, gradient clipping 1.0, and
  weight decay 0.1.

## Training data and schedule

Training uses packed Lichess standard-chess game records, represented as move
sequences. The final branch reads the `3p-s` packed corpus without replacement
in 512-position sequences; shards 77–127 were globally reshuffled before the
85% checkpoint to remove a temporal data-order artefact.

The planned one-pass budget is 78.32B chess positions. Optimisation uses a
position-indexed cosine learning-rate schedule: 5% warmup, peak LR `5e-4`, and
minimum LR equal to 10% of the peak. A 1% held-out split is reserved for
validation. The model's legal-move penalty weight is 0.5.

## Checkpoints

| File | Step | Training exposure | Description |
|---|---:|---:|---|
| `chsm8_best_pct85.pt` | 1,015,809 | 85.00% | Best checkpoint from the wide-shuffled lineage; it remained stable through the prior loss-bump window. |
| `chsm8_latest.pt` | 1,194,960 | 99.99% | Latest/final checkpoint from the training run. |

Both are weights-only PyTorch dictionaries containing `model_state_dict`,
`step`, `total_training_time`, `total_tokens_processed`, and `cfg`. Load them
with `FactoredChessDecoder` / `DecoderConfig` from this repository's
[`src/model.py`](https://github.com/nov3o/llm_chess/blob/main/src/model.py).

## Evaluation note

Playing evaluations use 200 colour-balanced games with greedy legal-move
decoding against Stockfish constrained to `UCI_Elo=2000` at 0.05 seconds per
move. This is a reproducible project metric, not an official FIDE Elo rating
or a general-purpose chess-engine benchmark.

## Limitations

CHSM8 is a research checkpoint with no search, opening book, endgame tablebase,
or standard Transformers interface. It has been evaluated only under the
project protocol above; use it for research and reproduce results from the
linked codebase.