Mert21779033/mertformer-chess-searchless

chessformer-tiny (searchless chess, RTX 5070, 2026-08-02)

A real, checkpoint-bound training run of ChessFormerAI — a small, independent side project that mirrors the mertformer-titan-core architecture family (BitNet b1.58, GQA, sparse MoE, Liquid/CfC) at a scale a single consumer GPU can actually train.

This is not the canonical MertFormer Titan model. It shares an architecture family, not a checkpoint, a parameter count, or a training run. Full evidence package (with the exact script that produced these numbers) lives in the GitHub repo at evidence/2026-08-02-chess-searchless-5070/.

This is not a transformers-compatible checkpoint. There is no AutoModel class for this architecture. To load the weights, use the chessformer package (ChessFormerAI/chessformer) — see retroactive_eval.py in this repo for a complete, working example of loading this exact checkpoint and running inference.

Play it yourself, locally

This repo includes a self-contained chessformer/ package (the minimal subset needed for inference) and app.py, a small Gradio UI. No Hugging Face Space, no GPU required:

pip install -r requirements.txt
hf download Mert21779033/mertformer-chess-searchless --local-dir .
python app.py

Opens a local board at http://127.0.0.1:7860 — you play White, type moves in SAN (e4, Nf3, O-O) or UCI (e2e4). (There is no public hosted Space for this — Hugging Face now requires a PRO subscription to host Gradio/Docker Spaces on free cpu-basic hardware; running it locally is free and takes seconds since the model is 4.6M parameters.)

What actually happened

• 4,592,740 parameters ("tiny" preset), use_bitnet=true, use_moe=true, use_liquid=true — all three resolved into the instantiated model (see model_report.json).
• Architecture: 4 layers, hidden size 192, GQA (4 query heads / 2 KV heads), sparse MoE (8 experts, top-2, every 3rd layer), BitNet b1.58 ternary weights, Liquid/CfC state, RoPE, bidirectional (non-causal) attention, a factorized policy head (logit(move) = <from_proj(h[from]), to_proj(h[to])>/sqrt(d) + promo_proj + move_bias), and a 128-bin HL-Gauss value head (DeepMind arXiv:2402.04494 / arXiv:2403.03950 conventions).
• 32,150 of 100,365 planned steps (~32%), stopped by the operator, not by divergence or a crash. Loss fell smoothly from 6.12 to 3.82 with no spikes — see training_curve.png.
• 16.2M Lichess-eval positions, packed with a hash-based train/val/test split (see dataset_provenance.json).
• This checkpoint (chessformer-tiny-step30000.pt) is step 30,000 — the lowest-val-loss checkpoint from the run, not the final step.

Real measured results

The three eval reports here are from a separate, retroactive, inference-only pass against this exact checkpoint (the run's own live eval stages produced no real numbers because should_stop() was already true by the time they ran) — no retraining, no resuming, just loading weights and evaluating. Reproduce with retroactive_eval.py.

Metric	Result	Comparable to
Puzzle accuracy (strict)	45.78% (2,289/5,000, reservoir sample)	DeepMind's Searchless Chess (arXiv:2402.04494): 9M-param model 85.5%, 136M 92.1%, 270M 93.5% — this is the directly comparable metric, same database, same protocol
Holdout masked policy top-1	39.4% (val) / 50.1% (test)	—
Holdout WDL accuracy	85.4% (val) / 86.0% (test)	—
Elo estimate	1509 (95% CI 1452–1567), 140 games vs Stockfish 18, UCI_LimitStrength	Not comparable to the 2895 Lichess-blitz figure in the DeepMind paper — different rating pool (Stockfish's internal UCI_Elo scale vs. Lichess-versus-humans). See elo_report.json's own interpretation block.

Honest framing, not a record claim. DeepMind's smallest (9M) model was trained on 15 billion Stockfish action-value examples on TPU-scale compute. This run used 16.2 million positions (925x less data) on a single 8GB laptop GPU for a few hours, on a model roughly half the parameter count. Reaching ~54% of the 9M model's puzzle accuracy (45.78% vs. 85.5%) under those constraints is a data/compute-efficiency data point, not a benchmark result — the "tiny" preset itself was not a corner cut: chessformer/config.py's MIN_POSITIONS_PER_PARAM autoscaler picked it specifically because the dataset couldn't honestly support a larger model; the small size is that safety mechanism working as designed, not a limitation being hidden.

Zero illegal moves were played across all 140 Elo-ladder games and all 5,000 puzzle attempts — every move is drawn from a legality-masked policy and re-checked before being pushed, so illegal moves are structurally impossible regardless of what the raw (unmasked) policy would have picked.

Files

• chessformer-tiny-step30000.pt — the checkpoint (55,281,955 bytes)
• app.py, requirements.txt, chessformer/ — a self-contained local Gradio UI to play against this checkpoint (see "Play it yourself, locally" above)
• retroactive_eval.py — the exact script that produced the eval reports below; also the reference implementation for loading this checkpoint
• model_report.json — architecture, resolved BitNet/MoE/Liquid, parameter accounting
• holdout_report.json, puzzle_report.json, elo_report.json — the three retroactive eval passes
• dataset_provenance.json — source, split policy, parse stats
• environment_snapshot.json — training hardware (RTX 5070) + eval hardware note
• architecture_parity_summary.json — drift/bug fixes vs. an earlier onefile implementation, self-verified via torch.allclose
• training_curve.png — loss / accuracy / grad-norm over the run

License

Apache License 2.0, same as the parent GitHub repo — see LICENSE. This applies to everything here: the code (app.py, chessformer/, retroactive_eval.py) and the checkpoint weights (chessformer-tiny-step30000.pt). The parent repo's model weights are otherwise proprietary (MODEL_LICENSE.md) — chessformer is an explicit, scoped exception to that policy for this independent side-project only; see MODEL_LICENSE.md's exception clause and the matching DECISIONS.md entry (2026-08-06) for the full reasoning.