Seedyai/SnapJudge2.0

SnapJudge 2.0

Non-autoregressive System-1 decision family for games — 421M active / 2.5B total. Give it a game state (tic-tac-toe board, Connect-4 grid, snake state, maze, or temple-run obstacle as text or JSON) and typed questions; it returns typed answers with calibrated probabilities in a single forward pass (~53 ms on T4, fp16).

This repo holds the joint checkpoint at the root plus five per-game experts under experts/ (joint at the root, experts in subfolders). A GameRouter detects the game in <1 ms of pure Python and dispatches to the best checkpoint — you download and load only what you route to. It never generates text, so there is nothing to parse and nothing to hallucinate.

System-1 decision design: an encoder plus an option-marker scorer and act head, trained with RLCD — reinforcement learning against strictly proper scoring rules, so honest probabilities maximise reward. v2 upgrades the backbone to ModernBERT-large, 4× the context, two new games, and per-game experts.

pip install torch transformers safetensors huggingface_hub numpy accelerate
git clone https://huggingface.co/Seedyai/SnapJudge2.0
export PYTHONPATH=$PYTHONPATH:$(pwd)/SnapJudge2.0   # repo root (provides `snapjudge/` package)

from huggingface_hub import snapshot_download
from snapjudge.agent import load
from snapjudge.router import GameRouter
from snapjudge.data_gen import questions_for

local = snapshot_download("Seedyai/SnapJudge2.0")
router = GameRouter(
    agent=load(local),                                   # joint fallback
    model_dirs={g: f"{local}/experts/{g}" for g in       # per-game experts
                ["tictactoe", "snake", "templerun", "connect4", "maze"]},
    preload=True,
)

# tic-tac-toe -> routed to the tictactoe expert
res = router.predict(
    {"game": "tictactoe", "board": ["X", "X", " ", "O", "O", " ", " ", " ", " "],
     "player": "X", "board_str": "XX.OO...."},
    questions_for("tictactoe"))
print(res["answers"]["next_move"]["choice"])  # -> cell2
print(res["routing"])                         # -> {'model': 'tictactoe', ...}

# snake -> snake expert; maze -> maze expert; unknown states -> joint
res = router.predict(
    {"game": "snake", "head": [5, 5], "body": [[5, 5], [5, 6]], "food": [7, 5], "grid": [10, 10]},
    questions_for("snake"))
print(res["answers"]["direction"]["choice"])  # -> right

# explicit override when you already know the checkpoint
res = router.predict(state, questions_for("maze"), model="maze")

Every result carries routing metadata (res["routing"]) explaining which checkpoint was chosen and why. Memory control: GameRouter(max_loaded=2) keeps two checkpoints hot with LRU eviction, router.unload() frees VRAM.

Decision primitives

Primitive	Output	Game use
choice	Top label + per-option probs + confidence	next_move (9 cells / 7 columns), direction (4), action (5)
score	Expected ordinal level + distribution	danger / threat / risk / progress / urgency (3 levels each)
noul	Calibrated P(true)	must_block, will_die, dead_end, game_over_soon

Architecture

• Backbone: ModernBERT-large (395M, bidirectional, fully fine-tuned) + decision head trained from scratch: 2 transformer layers (d=1024), option-marker scorer, act head. 421M active.
• Family: 1 joint + 5 experts = ~2.5B total weights, 421M active per request (MoE-style: router gates on game keys before the forward pass, <1 ms).
• Option markers: every option is scored at its own [MASK] token, softmaxed per question. New schemas need no retraining.
• Format: [CLS] <type> instructions [SEP] [MASK] opt0 [MASK] opt1 … [SEP] state [SEP].
• Budget: 1024 tokens (head_max_len = 256); raise to 512 for 50+ option questions (Jev serves 255 options out-of-the-box). All questions in one call, one forward pass.

Training (RLCD + experts)

1. Data: 7,500 synthetic games (22,500 typed decisions, 5 games) with exact-solver labels: minimax (tic-tac-toe), 1-ply + center prior (Connect-4), collision + Manhattan-to-food (snake), BFS shortest-path (maze), obstacle table (temple-run). Option-shuffle augmentation guards against option-order instability.
2. Joint: supervised CE pretrain (3 epochs) → RLCD fine-tune (2 epochs, Gaussian logit noise + log/spherical/RPS proper-score reward, GRPO-style baseline) on 2×T4 (DDP + fp16). Curve: 0.664 → 0.680 → 0.723 (CE) → 0.780 → 0.813 (RLCD).
3. Experts: 1 epoch per game from the joint checkpoint (1e-5).
4. Calibration: one temperature per (question type, option-count bucket) refit on held-out data.

Benchmarks (fresh seed 999: 1,000 games / 3,000 decisions, T4)

Routed system (expert per game, ties → joint):

Game	Joint	Expert	Routed
tic-tac-toe (move/danger/block)	0.717	0.800	expert
snake (direction/risk/trapped)	0.992	0.997	expert
temple-run (action/urgency/over)	1.000	1.000	joint (tie)
connect-4 (move/threat/block)	0.692	0.745	expert
maze (direction/progress/dead-end)	0.843	0.845	expert
overall (tie-aware)	0.849	—	≈0.877

Joint-only extras: strict-choice acc 0.710, ECE 0.190 (temp-scaled), hard-case acc 0.719 (endgames, traps, dead-ends — the hard slice where closed decision APIs lead: JevBench hard-case 0.74). 20-option stress question: passes (Jev parity probe).

Speed (Tesla T4, fp16)

Call	Latency
3 questions, one state (warm)	~53 ms p50
10 states batched	~574 ms total (57 ms/state)
20-option stress	~108 ms

For reference, the Jev API is independently measured at 236–276 ms p50 per question — the routed system answers ~5× faster self-hosted, at $0 marginal cost.

Reproduce: python3 bench.py --model . --experts ./experts --n 200 --seed 999 (per-checkpoint shootout: python3 bench_experts.py --joint . --experts ./experts).

Batch, validation, server, play

# batch: argmax-identical to predict(), grouped by checkpoint under the hood
outs = router.predict_batch([{"state": s1, "questions": q1}, {"state": s2, "questions": q2}])

Malformed questions raise ValueError naming the question (unknown type, empty instructions, choice with <2 options, >255 options, score criteria not a list, noul keys not true/false). Jev-compatible server with optional Bearer auth:

python3 serve.py --model . --experts ./experts --port 8000
SNAPJUDGE_API_KEY=secret python3 serve.py --model . --port 8000  # require Authorization: Bearer
curl localhost:8000/v1/systemone -H 'Content-Type: application/json' \
  -d '{"state":{"game":"snake","head":[5,5],"body":[[5,5]],"food":[7,5],"grid":[10,10]},
       "questions":{"direction":{"type":"choice","instructions":"Choose direction",
       "criteria":{"up":"up","down":"down","left":"left","right":"right"}}}}'

POST /v1/systemone {state, questions} → {answers, routing, usage}. Accepts Jev-style shapes (criteria as list or dict, {document: …} states, extra fields ignored); malformed questions return 422 naming the problem.

Honest limits

• Labels come from exact solvers on synthetic positions — human play distributions will differ; fine-tune on your own logs before trusting deployment.
• Connect-4 labels are 1-ply exact + center prior (full minimax is infeasible) — treat its 0.745 as heuristic-grade, not optimal-play grade.
• 9-way tic-tac-toe (0.800) and Connect-4 remain the weakest heads; narrow endgames dominate errors.
• Probabilities are temperature-fitted on synthetic held-out data — refit on your domain (train.fit_temperatures) before confidence gating. Raw ECE before scaling ≈ 0.39.
• templerun expert ties joint, so the router serves templerun from joint (checkpoint kept for completeness).

Files

README.md                 # this model card
snapjudge2_perf.png       # performance charts (see Benchmarks)
model.safetensors         # joint checkpoint (1.6 GB, 421M)
snapjudge_config.json     # encoder, head, context budgets, temperatures
tokenizer/                # ModernBERT tokenizer snapshot
metrics.json              # joint val metrics
routing_table.json        # per-game joint-vs-expert shootout (fresh seed 999)
experts_metrics.json      # per-expert val metrics
experts/{tictactoe,snake,templerun,connect4,maze}/
  model.safetensors       # per-game expert (1.6 GB each)
  snapjudge_config.json / metrics.json / tokenizer/
snapjudge/                # runtime: common.py, agent.py, router.py, data_gen.py, train.py
bench.py                  # fresh-data accuracy/ECE/latency (+ hard-case slice)
bench_experts.py          # joint-vs-expert shootout -> routing_table.json
serve.py                  # Jev-compatible POST /v1/systemone server
example.py / requirements.txt
train_v2_ddp.py           # 2-GPU DDP joint training script
run_experts.py            # per-game expert fine-tuning script
make_chart_v2.py          # regenerates snapjudge2_perf.png

License & credit

Apache 2.0. Backbone: ModernBERT-large. v1 (Seedyai/Snapjudge, ModernBERT-base, 3 games) remains available separately.