TEMPLATE_README.md

Chess Challenge: 1M Parameter Transformer

Train a transformer (from scratch!) with less than 1M parameters to play legal chess moves.

---

1. Overview & Objective

Your model must:

• Stay under 1M parameters (hard limit)
• Create a custom tokenizer
• Create a custom model architecture (your own transformer)
• Play legal chess (model must learn the rules)
• Do NOT use python-chess to filter moves (the model must generate legal moves itself)

---

2. Dataset & Notation

We use the Lichess dataset: dlouapre/lichess_2025-01_1M

Notation:

• W/B prefix for White/Black
• Piece letter: P=Pawn, N=Knight, B=Bishop, R=Rook, Q=Queen, K=King
• Source and destination squares (e.g., e2e4)
• Special suffixes: (x)=capture, (+)=check, (+*)=checkmate, (o)/(O)=castling

Example game:

WPe2e4 BPe7e5 WNg1f3 BNb8c6 WBf1b5 BPa7a6 WBb5c6(x) BPd7c6(x) ...

---

3. Directory Structure

Your project should look like this:

my_model/
  config.json
  model.safetensors
  tokenizer_config.json
  vocab.json
  model.py
  tokenizer.py

---

4. Step-by-Step Instructions

Step 1: Build Your Tokenizer

Create tokenizer.py implementing a subclass of PreTrainedTokenizer, say MyChessTokenizer.

Build the vocabulary:

One possibility is to look at the dataset, but it's by far not the only option:

from datasets import load_dataset
import json

dataset = load_dataset("dlouapre/lichess_2025-01_1M", split="train")
vocab = {"[PAD]": 0, "[BOS]": 1, "[EOS]": 2, "[UNK]": 3}
for game in dataset:
    for move in game["text"].split():
        if move not in vocab:
            vocab[move] = len(vocab)
with open("vocab.json", "w") as f:
    json.dump(vocab, f, indent=2)

Step 2: Build Your Model

Create model.py implementing a subclass of PreTrainedModel, say MyChessModel, and a config class, say MyChessConfig.

Tips:

• Use weight tying to save parameters.
• Keep the vocabulary small.
• 4-6 transformer layers is usually enough.

Step 3: Training

Create train.py to train your model:

from model import MyChessConfig, MyChessModel
from tokenizer import MyChessTokenizer
from datasets import load_dataset
from transformers import Trainer, TrainingArguments

tokenizer = MyChessTokenizer(vocab_file="vocab.json")
config = MyChessConfig(vocab_size=tokenizer.vocab_size, n_embd=128, n_layer=4, n_head=4)
model = MyChessModel(config)

dataset = load_dataset("dlouapre/lichess_2025-01_1M", split="train")
def tokenize_function(examples):
    return tokenizer(examples["text"], truncation=True, max_length=256, padding="max_length")
tokenized_dataset = dataset.map(tokenize_function, batched=True)

training_args = TrainingArguments(
    output_dir="./my_model",
    num_train_epochs=3,
    per_device_train_batch_size=32,
    learning_rate=5e-4,
    save_steps=1000,
    logging_steps=100,
)
trainer = Trainer(model=model, args=training_args, train_dataset=tokenized_dataset)
trainer.train()
model.save_pretrained("./my_model/final")
tokenizer.save_pretrained("./my_model/final")

Step 4: Prepare for Submission

Your model directory (my_model/final/) must contain:

config.json           # Model configuration
model.safetensors     # Model weights
tokenizer_config.json # Tokenizer configuration
vocab.json            # Vocabulary
model.py              # Your model class
tokenizer.py          # Your tokenizer class

Add auto_map for remote loading

Edit config.json:

  "auto_map": {
    "AutoConfig": "model.MyChessConfig",
    "AutoModelForCausalLM": "model.MyChessModel"
  }

Edit tokenizer_config.json:

  "auto_map": {
    "AutoTokenizer": "tokenizer.MyChessTokenizer"
  }

Or do it programmatically:

model.config.auto_map = {
    "AutoConfig": "model.MyChessConfig",
    "AutoModelForCausalLM": "model.MyChessModel",
}
tokenizer.register_for_auto_class("AutoTokenizer")
model.save_pretrained("./my_model/final")
tokenizer.save_pretrained("./my_model/final")
import shutil
shutil.copy("model.py", "./my_model/final/model.py")
shutil.copy("tokenizer.py", "./my_model/final/tokenizer.py")

---

Step 5: Local Evaluation (Recommended)

Before submitting, evaluate your model locally:

python -m src --model ./my_model/final

This runs the same evaluation as the leaderboard (500 moves, deterministic seed).

---

Step 6: Submit

Submit your model to the leaderboard:

python submit.py --model_path ./my_model/final --model_name your-model-name

The script will:

• Validate all required files
• Check auto_map
• Count parameters
• Log you into HuggingFace (if needed)
• Upload to the LLM-course organization

---

5. Evaluation & Leaderboard

After submission, go to the Chess Challenge Arena to run evaluation.

Evaluation steps:

1. Parameter check (<1M)
2. Security check (no python-chess for move filtering)
3. 500 moves against a deterministic opponent
4. 3 retries per move (greedy, then sampling)
5. Scoring: legal move rate (first try and with retries)

Scoring Table:

Metric	Description
Legal Rate (1st try)	% of moves legal on first attempt
Legal Rate (with retries)	% of moves legal within 3 attempts

Target: >90% legal rate = excellent

---

6. Example Solution

See example_solution/ for a full working reference:

• model.py, tokenizer.py, train.py, data.py

---