|
|
--- |
|
|
license: mit |
|
|
tags: |
|
|
- chess |
|
|
- transformer |
|
|
- reinforcement-learning |
|
|
- game-playing |
|
|
- research |
|
|
library_name: pytorch |
|
|
--- |
|
|
|
|
|
# ChessFormer-RL |
|
|
|
|
|
ChessFormer-RL represents an experimental checkpoint in training chess models with reinforcement learning. **Note**: This model is actually the 8th supervised learning checkpoint (49152 steps) that was intended as initialization for RL training, as the full RL training encountered challenges. |
|
|
|
|
|
## Model Description |
|
|
|
|
|
- **Model type**: Transformer for chess (RL training initialization) |
|
|
- **Language(s)**: Chess (FEN notation) |
|
|
- **License**: MIT |
|
|
- **Parameters**: 100.7M |
|
|
|
|
|
## Important Notice |
|
|
|
|
|
⚠️ **This model represents a research checkpoint rather than a completed RL-trained model.** The actual reinforcement learning training encountered: |
|
|
|
|
|
- Gradient norm explosion |
|
|
- Noisy reward signals |
|
|
- Performance degradation from this initialization point |
|
|
|
|
|
This checkpoint is provided for researchers interested in: |
|
|
|
|
|
- RL training initialization strategies |
|
|
- Comparative analysis with the final SL model |
|
|
- Continuing RL experiments with improved methods |
|
|
|
|
|
## Architecture |
|
|
|
|
|
Identical to ChessFormer-SL: |
|
|
|
|
|
- **Blocks**: 20 transformer layers |
|
|
- **Hidden size**: 640 |
|
|
- **Attention heads**: 8 |
|
|
- **Intermediate size**: 1728 |
|
|
- **Features**: RMSNorm, SwiGLU activation, custom FEN tokenizer |
|
|
|
|
|
## Training Details |
|
|
|
|
|
### Phase 1: Supervised Learning (This Checkpoint) |
|
|
|
|
|
- **Dataset**: `kaupane/lichess-2023-01-stockfish-annotated` (depth18 split) |
|
|
- **Training**: 49152 steps of supervised learning on Stockfish evaluations |
|
|
- **Purpose**: Initialization for subsequent RL training |
|
|
|
|
|
### Phase 2: Reinforcement Learning (Attempted) |
|
|
|
|
|
- **Method**: Self-play with Proximal Policy Optimization (PPO) |
|
|
- **Environment**: Batch chess environment with sparse terminal rewards |
|
|
- **Outcome**: Training instabilities led to performance degradation |
|
|
- **Current Status**: Requires further research and improved RL methodology |
|
|
|
|
|
### Training Metrics (This Checkpoint) |
|
|
|
|
|
- **Action Loss**: 1.8329 |
|
|
- **Value Loss**: 0.0501 |
|
|
- **Invalid Loss**: 0.0484 |
|
|
|
|
|
## Performance |
|
|
|
|
|
As an intermediate SL checkpoint, this model exhibits: |
|
|
|
|
|
- Similar capabilities to early ChessFormer-SL training |
|
|
- Less refined than the final SL model |
|
|
- Suitable for RL initialization experiments |
|
|
|
|
|
### Comparison with ChessFormer-SL |
|
|
|
|
|
| Metric | ChessFormer-RL (8th ckpt) | ChessFormer-SL (20th ckpt) | |
|
|
|--------|---------------------------|----------------------------| |
|
|
| Action Loss | 1.8329 | 1.6985 | |
|
|
| Value Loss | 0.0501 | 0.0407 | |
|
|
| Invalid Loss | 0.0484 | 0.0303 | |
|
|
|
|
|
## Research Context |
|
|
|
|
|
### RL Training Challenges Encountered |
|
|
|
|
|
1. **Gradient Instability**: Explosive gradient norms during PPO updates |
|
|
2. **Sparse Rewards**: Terminal-only rewards created noisy learning signals |
|
|
3. **Action Space Complexity**: 1,969 possible moves created exploration challenges |
|
|
4. **Self-Play Dynamics**: Unstable opponent strength during training |
|
|
|
|
|
## Usage |
|
|
|
|
|
### Installation |
|
|
|
|
|
```bash |
|
|
pip install torch transformers huggingface_hub chess |
|
|
# Download model.py from this repository |
|
|
``` |
|
|
|
|
|
### Loading the Model |
|
|
|
|
|
```python |
|
|
import torch |
|
|
from model import ChessFormerModel |
|
|
|
|
|
# Load model |
|
|
model = ChessFormerModel.from_pretrained("kaupane/ChessFormer-RL") |
|
|
model.eval() |
|
|
|
|
|
# This is an intermediate checkpoint - performance will be lower than ChessFormer-SL |
|
|
``` |
|
|
|
|
|
### For RL Research |
|
|
|
|
|
```python |
|
|
# This checkpoint can serve as initialization for RL experiments |
|
|
from train_rl import RLTrainer |
|
|
|
|
|
# Load checkpoint for RL training continuation |
|
|
trainer = RLTrainer( |
|
|
model=model, |
|
|
# ... other hyperparameters |
|
|
) |
|
|
trainer.resume("path/to/checkpoint", from_sl_checkpoint=True) |
|
|
``` |
|
|
|
|
|
## Limitations |
|
|
|
|
|
### Technical Limitations |
|
|
|
|
|
- **Incomplete Training**: Represents intermediate rather than final model |
|
|
- **RL Instabilities**: Subsequent RL training was unsuccessful |
|
|
- **Performance**: Lower quality than ChessFormer-SL final checkpoint |
|
|
|
|
|
### Research Limitations |
|
|
|
|
|
- Demonstrates challenges rather than solutions for chess RL |
|
|
- Requires significant additional work for competitive performance |
|
|
- Not suitable for production use |
|
|
|
|
|
## Intended Use |
|
|
|
|
|
This model is specifically intended for: |
|
|
|
|
|
- ✅ RL research and experimentation |
|
|
- ✅ Studying initialization strategies for chess RL |
|
|
- ✅ Comparative analysis of SL vs RL training trajectories |
|
|
- ✅ Educational purposes in understanding RL challenges |
|
|
|
|
|
**Not intended for:** |
|
|
|
|
|
- ❌ Practical chess playing applications |
|
|
- ❌ Production chess engines |
|
|
- ❌ Competitive chess analysis |
|
|
|
|
|
## Additional Information |
|
|
|
|
|
- **Repository**: [GitHub link](https://github.com/Mtrya/chess-transformer) |
|
|
- **Demo**: [HuggingFace Space Demo](https://huggingface.co/spaces/kaupane/Chessformer_Demo) |
|
|
- **Related**: [ChessFormer-SL](https://huggingface.co/kaupane/ChessFormer-SL) (Completed SL Training) |
|
|
|
|
|
*This model represents ongoing research into chess RL training. While the full RL training was unsuccessful, this checkpoint may be an initial starting point for future research directions.* |
