|
|
--- |
|
|
license: mit |
|
|
tags: |
|
|
- chess |
|
|
- transformer |
|
|
- reinforcement-learning |
|
|
- game-playing |
|
|
library_name: pytorch |
|
|
--- |
|
|
|
|
|
# ChessFormer-SL |
|
|
|
|
|
ChessFormer-SL is a transformer-based chess model trained via supervised learning on Stockfish evaluations. This model explores training chess engines without Monte Carlo Tree Search (MCTS), using only neural networks. |
|
|
|
|
|
## Model Description |
|
|
|
|
|
- **Model type**: Transformer for chess position evaluation and move prediction |
|
|
- **Language(s)**: Chess (FEN notation) |
|
|
- **License**: MIT |
|
|
- **Parameters**: 100.7M |
|
|
|
|
|
## Architecture |
|
|
|
|
|
ChessFormer uses a custom transformer architecture optimized for chess: |
|
|
|
|
|
- **Blocks**: 20 transformer layers |
|
|
- **Hidden size**: 640 |
|
|
- **Attention heads**: 8 |
|
|
- **Intermediate size**: 1728 |
|
|
- **Features**: RMSNorm, SwiGLU activation, custom FEN tokenizer |
|
|
|
|
|
### Input Format |
|
|
|
|
|
The model processes FEN strings and repetition counts, tokenizing them into 75-token sequences representing: |
|
|
|
|
|
- 64 board square tokens (pieces + positional embeddings) |
|
|
- 9 metadata tokens (turn, castling, en passant, clocks, repetitions) |
|
|
- 2 special tokens (action, value) |
|
|
|
|
|
### Output Format |
|
|
|
|
|
- **Policy head**: Logits over 1,969 structurally valid chess moves |
|
|
- **Value head**: Position evaluation from current player's perspective |
|
|
|
|
|
## Training Details |
|
|
|
|
|
### Training Data |
|
|
|
|
|
- **Dataset**: `kaupane/lichess-2023-01-stockfish-annotated` (depth18 split) |
|
|
- **Size**: 56M positions with Stockfish evaluations |
|
|
- **Validation**: depth27 split |
|
|
|
|
|
### Training Procedure |
|
|
|
|
|
- **Method**: Supervised learning on Stockfish move recommendations and evaluations |
|
|
- **Objective**: Cross-entropy loss (moves) + MSE loss (values) + invalid move penalty |
|
|
- **Hardware**: RTX 4060Ti 16GB |
|
|
- **Duration**: ~2 weeks |
|
|
- **Checkpoints**: 20 total, this model is the final checkpoint |
|
|
|
|
|
### Training Metrics |
|
|
|
|
|
- **Action Loss**: 1.6985 |
|
|
- **Value Loss**: 0.0407 |
|
|
- **Invalid Loss**: 0.0303 |
|
|
|
|
|
## Performance |
|
|
|
|
|
### Capabilities |
|
|
|
|
|
- ✅ Reasonable opening and endgame play |
|
|
- ✅ Fast inference without search |
|
|
- ✅ Better than next-token prediction chess models |
|
|
- ✅ Can defeat Stockfish occasionally with search enhancement |
|
|
|
|
|
### Limitations |
|
|
|
|
|
- ❌ Frequent tactical blunders in midgame |
|
|
- ❌ Estimated ELO ~1500 (informal assessment) |
|
|
- ❌ Struggles with complex tactical combinations |
|
|
- ❌ Tends to give away pieces ("free captures") |
|
|
|
|
|
## Usage |
|
|
|
|
|
### Installation |
|
|
|
|
|
```bash |
|
|
pip install torch transformers huggingface_hub chess |
|
|
# Download model.py from this repository |
|
|
``` |
|
|
|
|
|
### Basic Usage |
|
|
|
|
|
```python |
|
|
import torch |
|
|
from model import ChessFormerModel |
|
|
|
|
|
# Load model |
|
|
model = ChessFormerModel.from_pretrained("kaupane/ChessFormer-SL") |
|
|
model.eval() |
|
|
|
|
|
# Analyze position |
|
|
fens = ["rnbqkbnr/pppppppp/8/8/4P3/8/PPPP1PPP/RNBQKBNR b KQkq e3 0 1"] |
|
|
repetitions = torch.tensor([1]) |
|
|
|
|
|
with torch.no_grad(): |
|
|
move_logits, position_value = model(fens, repetitions) |
|
|
|
|
|
# Get best move (requires additional processing for legal moves) |
|
|
print(f"Position value: {position_value.item():.3f}") |
|
|
``` |
|
|
|
|
|
### With Chess Engine Interface |
|
|
|
|
|
```python |
|
|
from engine import Engine, ChessformerConfig |
|
|
import chess |
|
|
|
|
|
# Create engine |
|
|
config = ChessformerConfig( |
|
|
chessformer=model, |
|
|
temperature=0.5, |
|
|
depth=2 # Enable search enhancement |
|
|
) |
|
|
engine = Engine(type="chessformer", chessformer_config=config) |
|
|
|
|
|
# Play move |
|
|
board = chess.Board() |
|
|
move_uci, value = engine.move(board) |
|
|
print(f"Suggested move: {move_uci}, Value: {value:.3f}") |
|
|
``` |
|
|
|
|
|
## Limitations and Bias |
|
|
|
|
|
### Technical Limitations |
|
|
|
|
|
- **Tactical weakness**: Prone to hanging pieces and missing simple tactics |
|
|
- **Computational inefficiency**: FEN tokenization creates training bottlenecks, preprocess the entire dataset before training should be benefical |
|
|
|
|
|
### Potential Biases |
|
|
|
|
|
- Trained exclusively on Stockfish evaluations, may inherit engine biases |
|
|
- May not generalize to unconventional openings or endgames |
|
|
|
|
|
### Known Issues |
|
|
|
|
|
- Piece embeddings have consistently lower norms than positional embeddings |
|
|
- Model sometimes assigns probability (though unlikely, ~3%) to invalid moves despite training penalty |
|
|
- Performance degrades without search enhancement |
|
|
|
|
|
## Ethical Considerations |
|
|
|
|
|
This model is intended for: |
|
|
|
|
|
- ✅ Educational purposes and chess learning |
|
|
- ✅ Research into neural chess architectures |
|
|
- ✅ Developing chess training tools |
|
|
|
|
|
Not recommended for: |
|
|
|
|
|
- ❌ Competitive chess tournaments |
|
|
- ❌ Production chess engines without extensive testing |
|
|
- ❌ Applications requiring reliable tactical calculation |
|
|
|
|
|
## Additional Information |
|
|
|
|
|
- **Repository**: [GitHub link](https://github.com/Mtrya/chess-transformer) |
|
|
- **Demo**: [HuggingFace Space Demo](https://huggingface.co/spaces/kaupane/Chessformer_Demo) |
|
|
- **Related**: [ChessFormer-RL](https://huggingface.co/kaupane/ChessFormer-RL) (RL training experiment) |
