Syndria98/ultimate-tic-tac-toe-ai

Ultimate Tic Tac Toe Deep Learning Bot

Model for playing Ultimate Tic Tac Toe

Available checkpoints

• checkpoints/medium.pth: medium-difficulty checkpoint.
• checkpoints/hard.pth: hard-difficulty checkpoint.

Usage

Usage Run python run.py --help for help.

Shared flags

• --device: Torch device string. If omitted, it auto-picks "cuda" when available, otherwise "cpu".
• --checkpoint: Path to the model checkpoint file. Default is latest.pth. It is loaded for eval, play, and checkpoint-based arena, and used as the save path for accepted training checkpoints.

Training parameters

• --resume: Loads model and optimizer state from --checkpoint before continuing training.
• --num-simulations default 100: MCTS rollouts per move during self-play. Higher is stronger/slower.
• --num-iters default 50: Number of outer training iterations. Each iteration generates new self-play games, trains, then arena-tests the new model.
• --num-eps default 20: Self-play games per iteration.
• --epochs default 5: Passes over the current replay-buffer training set per iteration.
• --batch-size default 64: Mini-batch size for gradient updates.
• --lr default 5e-4: Adam learning rate.
• --weight-decay default 1e-4: Adam weight decay (L2-style regularization).
• --replay-buffer-size default 50000: Maximum number of training examples retained across iterations. Older examples are dropped.
• --value-loss-weight default 1.0: Multiplier on the value-head loss in total training loss. Total loss is policy_KL + value_loss_weight * value_loss.
• --grad-clip-norm default 5.0: Global gradient norm clipping threshold before optimizer step.
• --temperature-threshold default 10: In self-play, moves before this step use stochastic sampling from MCTS visit counts; later moves use greedy selection.
• --root-dirichlet-alpha default 0.3: Dirichlet noise alpha added to root priors during self-play MCTS to force exploration.
• --root-exploration-fraction default 0.25: How much of that root prior is replaced by Dirichlet noise.
• --arena-compare-games default 6: Number of head-to-head games between candidate and previous model after each iteration. If <= 0, every candidate is accepted.
• --arena-accept-threshold default 0.55: Minimum average points needed in arena to keep the new model. Win = 1, draw = 0.5.
• --arena-compare-simulations default 8: MCTS simulations per move during those arena comparison games. Separate from self-play --num-simulations.

Evaluation parameters

• --moves default "": Comma-separated move list to reach a position from the starting board, e.g. 0,10,4.
• --top-k default 10: How many highest-probability legal moves to print from the model policy.
• --with-mcts: Also run MCTS on that position and print the best move, instead of only raw network policy/value.
• --num-simulations default 100: Only matters with --with-mcts; controls MCTS search depth for that evaluation.

Play parameters

• --human-player default 1: Which side you control. 1 means X, -1 means O.
• --num-simulations default 100: MCTS simulations the AI uses for each move.

Arena parameters

• --games default 20: Number of matches to run.
• --num-simulations default 100: MCTS simulations per move for checkpoint-based players.
• --x-player / --o-player: Either checkpoint or random. Chooses the agent type for each side.
• --x-checkpoint / --o-checkpoint: Checkpoint path for that side when its player type is checkpoint. Ignored for random.

A few practical examples:

python run.py train --num-iters 100 --num-eps 50 --resume
python run.py eval --checkpoint latest.pth --moves 0,10,4 --with-mcts --num-simulations 200
python run.py play --human-player -1 --num-simulations 300
python run.py arena --games 50 --x-player checkpoint --o-player random