models/README.md

Runtime model

The application defaults to an automatic runtime that lazily downloads bartowski/SmolLM2-360M-Instruct-GGUF and runs its Q8_0 quantization through llama-cpp-python (see src/underdog_lab/config.py). A 360M SmolLM2 QLoRA adapter was trained and benchmarked (results/ship_decision.json) but did not clear the ship gate, so it is not the default. If model loading fails, the challenge remains usable through a deterministic fallback and records the degraded behavior.

For local llama.cpp inference:

export UNDERDOG_EXTRACTOR=llama_cpp
export UNDERDOG_LLAMA_CLI=/path/to/llama-cli
export UNDERDOG_MODEL_PATH=/path/to/model-q4_k_m.gguf
python app.py

The selected GGUF should be a compatible model at or below 4B parameters. Test the exact conversion with the pinned llama.cpp revision before publishing the Space.

To force the in-process runtime with a different model:

export UNDERDOG_EXTRACTOR=python_llama_cpp
export UNDERDOG_MODEL_REPO=bartowski/SmolLM2-360M-Instruct-GGUF
export UNDERDOG_MODEL_FILENAME=SmolLM2-360M-Instruct-Q8_0.gguf
python app.py

World Cup 2026 forecasting model

underdog_lab.world_cup.forecasting.MODEL is a DixonColesEloModel (Elo-to-goals Poisson rates with a Dixon-Coles low-score correlation term) fit by scripts/fit_elo_dixon_coles.py via time-decayed maximum likelihood on data/historical/matches.csv (11,094 real international matches, 2015-01-03 to 2026-06-12, sourced from eloratings.net).

• elo_fit_report.json — the final fit (trained on all data through the 2026 World Cup information_cutoff, 180-day time-decay half-life): intercept=0.1534, elo_scale=0.00207, home_advantage_elo=65.6, rho=-0.0998. The negative rho reflects the well-documented excess of low-scoring draws (0-0, 1-1) versus independent Poisson. home_advantage_elo has no effect on World Cup group fixtures, which are always neutral_venue=True.
• backtest_report.json — a walk-forward backtest (scripts/ backtest_walk_forward.py): for each test year 2018-2026, the model is refit on only the matches strictly before that year (no lookahead) and scored on that year's matches. The fitted model beats both the uniform baseline and the previous hand-set model (independent Poisson, elo_scale=0.00165, no home advantage, rho=0) on mean log loss, Brier score, and RPS across all folds. It also beats the previous model on the neutral-venue subset, which better matches World Cup inference, so the ship gate (ship_gate.ship) passed and MODEL was updated to these fitted parameters. A basic calibration table using the actual mean predicted probability in each decile is included. The model is under-confident through much of the middle range, so calibration remains an explicit follow-up rather than a completed claim.
• upgrade_evaluation.json (scripts/upgrade_evaluation.py) — two further changes evaluated with a selection (2018-2025) / confirmation (2026) split, so neither was chosen by peeking at the held-out year:
  • Recency half-life (adopted). The previous fit used a fixed 3-year (1095-day) half-life, never tuned. A sweep over {180, 365, 547, 730, 1095, 1460, 2190} days found 180 days has the lowest mean log loss on the 2018-2025 selection folds (0.79417 vs 0.79454 for 1095 days), and it also beats the 1095-day fit on the held-out 2026 fold both overall (0.80110 vs 0.80406) and on the neutral-venue subset (0.80382 vs 0.80464). All three checks passed, so MODEL and fit_elo_dixon_coles.py's default half-life were updated from 1095 to 180 days.
  • Ensemble with a second, independently-computed Elo rating (not adopted). forecasting/self_elo.py computes a from-scratch Elo rating (fixed K-factor, goal-difference multiplier, neutral start, derived only from matches.csv results — independent of eloratings.net). A Dixon-Coles model fit on this rating alone scores markedly worse than the eloratings-Elo model on the 2018-2025 selection folds, and every blend weight tested (a logarithmic opinion pool, forecasting/ensemble.py, weights 0.3-0.9 toward the eloratings model) scored worse than the eloratings-only model. The ensemble failed all three gate checks (selection, confirmation, confirmation-neutral) and MODEL remains a single eloratings-Elo fit. This is an honest negative result, not a placeholder: a from-scratch Elo computed over 11 years of data starting every team at a neutral 1500 is a weaker signal than eloratings.net's long-calibrated ratings, and blending in a weaker signal cannot improve a stronger one under a log-odds pool.

Tournament simulation now uses the official matches 73-104 path and the 495 third-place mappings from Annex C. It remains a single fitted statistical model rather than an ensemble (the ensemble evaluation above did not clear its gate); refit and re-run the walk-forward gate when new results are incorporated.

• recalibration_evaluation.json (scripts/recalibration_evaluation.py, adopted) — a post-hoc temperature-scaling correction (forecasting/calibration.py) evaluated with the same selection (2018-2025) / confirmation (2026) split: out-of-fold forecasts from the walk-forward folds (180-day half-life, refit per fold, no lookahead) are pooled, a single temperature T is fit on the 2018-2025 pool by minimizing mean log loss, and T is then checked against the T=1 (no-op) baseline on the held-out 2026 fold. The fitted T=0.8857 (mild sharpening, since T<1 pushes probabilities away from uniform) beats T=1 on mean log loss on the 7,914-match 2018-2025 selection pool (0.79179 vs 0.79417), on the 2026 confirmation fold overall (0.79312 vs 0.80110), and on the 143-match confirmation fold neutral-venue subset (0.79164 vs 0.80382). The overall confirmation fold contains 351 matches. Paired bootstrap intervals for fitted-minus-baseline log loss remain below zero on all three reported slices. The neutral subset overlaps the overall 2026 fold, so these are robustness slices rather than three independent confirmation samples. The gate passed, so world_cup/forecasting.py::match_forecast now applies apply_temperature(forecast, CALIBRATION_TEMPERATURE) to MODEL's output before returning it. lambda_home, lambda_away, and most_likely_score are unaffected — temperature scaling only rescales p_home/p_draw/p_away.

• vector_calibration_evaluation.json (scripts/vector_calibration_evaluation.py, not adopted) evaluates a regularized five-parameter multiclass vector scaling layer on top of the shipped global temperature. Regularization is selected by rolling-origin validation across 2021-2025, rather than by the viewed 2026 slice. The candidate improves log loss, Brier, and RPS on the selection pool and the viewed 2026 robustness slices, but neutral-match ECE worsens from 0.1025 to 0.1612. It therefore fails the research gate and is not wired into match_forecast. More importantly, 2026 has already been viewed during prior model decisions, so even a clean diagnostic pass would still require a future pre-registered confirmation period before production adoption.

Market-assisted experiment

forecasting/market.py implements proportional, power, and Shin margin removal plus a one-parameter logarithmic pool between the independent model and timestamped market probabilities. scripts/market_blend_evaluation.py fits that single weight on 2018-2025 and gates it on 2026 overall and neutral matches, with minimum sample sizes and a paired bootstrap interval. All three reported paired intervals must remain below zero before adoption.

The market-assisted path is not active because no licensed, timestamped odds dataset is checked in. It must remain separately labeled from the independent forecast even if it later passes the gate; incorporating market consensus is not evidence that the independent model beats the market. See data/market/README.md for the required input contract.

Tournament-specific experiments

• host_adjustment_evaluation.json (scripts/host_adjustment_evaluation.py, not adopted) evaluates no host boost, fixed +25/+50 Elo boosts, and a fitted one-parameter boost on explicitly identified host-team matches. Models are refit before each tournament edition and host effects are isolated from generic home advantage. Mean log loss improves for positive boosts, but the completed confirmation-edition cluster-bootstrap interval crosses zero. The previous ungated +50 World Cup host heuristic was therefore removed from TournamentTeam.rating.

• opener_draw_evaluation.json (scripts/opener_draw_evaluation.py) is a research-only draw-specific experiment. Opener labels are inferred because the historical source lacks official stage and matchday fields. The experiment excludes 2026, validates by completed tournament edition, and can never activate production code automatically.