README.md

metadata

title: World Cup 2026 Forecaster
emoji: ⚽
colorFrom: blue
colorTo: red
sdk: gradio
sdk_version: 6.18.0
python_version: 3.12
app_file: app.py
pinned: false
license: mit
tags:
  - thousand-token-wood
  - tiny-titan
  - off-brand

World Cup 2026 Forecaster

A rigorous football forecasting project and small-model experimentation platform.

The project has two related goals:

1. Build calibrated probabilistic forecasts for football matches and tournaments.
2. Evaluate how language models and coding agents contribute to forecasting systems.

The project avoids treating a single tournament-winner guess as evidence. Its primary unit of evaluation is the match-level probability distribution over home win, draw, and away win.

Recommended Hackathon Entry

World Cup 2026 Forecaster: Can You Beat the Tiny Pundit?

The primary experience is now a prospective 2026 tournament forecaster:

• all 48 teams and 72 known group fixtures,
• current group tables from a timestamped result snapshot,
• 1X2 probabilities for every group match,
• Monte Carlo advancement and championship probabilities,
• FIFA's published Round-of-32 slots, all 495 Annex C third-place mappings, and the fixed knockout path through match 104,
• bounded scenario adjustments extracted by a local 360M model,
• the original hidden historical challenge as a calibration game.

An interactive, local-first Gradio application where users:

• select a real or historical match;
• inspect a transparent statistical forecast;
• describe a counterfactual scenario in natural language;
• watch a small local model convert that scenario into validated semantic football factors;
• compare the updated probability distribution with the baseline;
• make their own forecast and receive a proper-scoring-rule score.

The language model never invents probabilities or numerical model deltas. It extracts typed semantic factors such as key_attacker_unavailable, with team attribution, severity, and certainty. A deterministic ruleset maps those factors into bounded adjustments, and the statistical engine owns the numerical forecast.

Team

• @sammoftah — Hugging Face username, Build Small Hackathon org member.

Submission Links

• Demo video: see docs/demo-script.md for the recorded walkthrough script; the published recording is linked from release/submission.json once available.
• Social post: docs/social-post.md is the published post's source text; the live post URL is linked from release/submission.json once available.

Documentation

• Implementation plan
• Hackathon entry plan
• Research synthesis
• Field notes
• Demo script
• Social post draft
• Post-hackathon roadmap
• Market-data source review

Run Locally

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python app.py

The default runtime starts downloading and warming a Q8 quantization of SmolLM2-360M-Instruct when the app boots, then runs it through llama-cpp-python. To exercise the app without downloading a model:

UNDERDOG_EXTRACTOR=mock python app.py

Reproduce the Core

PYTHONPATH=src python scripts/prepare_matches.py
PYTHONPATH=src pytest
UNDERDOG_EXTRACTOR=mock PYTHONPATH=src python scripts/evaluate_extractor.py
make sunday

The extraction fixtures are synthetic and remain marked as pending manual review. Their metrics are pipeline diagnostics, not an accuracy claim. A QLoRA adapter was trained and published, but the reproducible gate in results/qlora_gate_report.json selected NO-SHIP because semantic F1 did not improve materially, team attribution regressed, and fallback use rose. The production path therefore remains the base 360M model plus validated deterministic fallback.

Fine-Tune on Modal

1. Publish the reviewed JSONL dataset to the Hub.
2. Create a Modal secret named huggingface-secret containing HF_TOKEN.
3. Run one QLoRA job:

modal run training/modal_train.py \
  --dataset-repo YOUR_DATASET_REPO \
  --output-repo YOUR_ADAPTER_REPO

4. Compare base and tuned models on the frozen test set.
5. Merge and export only if the tuned model passes the decision gate described in the implementation plan.

Evaluation Principles

• Primary metric: multiclass log loss.
• Secondary metrics: Brier score, calibration, sharpness, and optional ranked probability score.
• Historical validation: strictly walk-forward.
• Live predictions: immutable timestamped records created before kickoff.
• Live reporting: prospective artifacts are never mixed with retrospective current-model replays.
• Market comparisons: use prices captured at the same forecast horizon and remove the bookmaker margin.
• Tournament forecasts: Monte Carlo simulations derived from match-level probabilities.

Proposed Repository Layout

app.py
src/underdog_lab/
  data/
  forecasting/
  scenarios/
  telemetry/
  ui/
scripts/
training/
data/
models/
tests/
docs/

Status

The end-to-end application, challenge data, forecasting core, scenario contracts, llama.cpp adapter, Gradio experience, tests, synthetic-data pipeline, Modal training job, evaluation script, and submission drafts are implemented.

All 72 group fixtures have exact UTC kickoff metadata. Forecast artifacts record model version, calibration temperature, team-rating hash, snapshot hash, commit, generation time, and information cutoff. make health verifies schedule completeness, snapshot integrity, result freshness, and live-proof manifests. make track-record prints prospective scores by forecast horizon and keeps current-model retrospective replay clearly separate.

Result And Deployment Automation

The result path is intentionally semi-automatic:

1. .github/workflows/result-check.yml polls football-data.org every three hours using FOOTBALL_DATA_API_KEY.
2. Provider names, IDs, kickoff times, fixture orientation, competition, and season are normalized and validated before a candidate update is written.
3. New results or provider corrections are applied on an automation branch, checked by the health report and full test suite, and opened as a PR.
4. A human reviews and merges the PR. Automation never writes results directly to main.
5. .github/workflows/deploy-space.yml pushes the exact merged Git revision to sammoftah/underdog-lab using HF_TOKEN, then waits for the Hub repository and running Space revision to match.

Repository secrets required for these workflows:

• FOOTBALL_DATA_API_KEY: football-data.org v4 API token.
• HF_TOKEN: Hugging Face token with write access to the Space.

Missing secrets are reported as not configured in the workflow summary; they are not treated as a successful update or deployment.

The Space, base model, evaluated adapter, dataset, field notes, and source are published. make sunday closes the QLoRA gate, verifies all 495 official bracket combinations, runs tests and the data audit, and emits results/submission_preflight.json.

Cold-start and mobile-width checks are complete. Remaining account-dependent release work is intentionally narrow: publish the demo video and social post, then record the final hackathon submission URL in release/submission.json.

Important Claim Boundary

The initial release should claim to be an experimental forecasting lab, not the most accurate model in football. Accuracy claims require a frozen benchmark, held-out evaluation, uncertainty intervals, and comparison against de-margined market probabilities at matched timestamps.