ForecastGemma-Interp

A rank-16 Gemma 3 1B adapter trained on 322 real, dated binary forecasting questions, with measured layer interventions and independent controls. Its next-token logits for 0 and 1 form a binary probability readout. This is an experimental forecasting readout, not a general forecasting agent.

It improves over base Gemma and formatting/shuffled-outcome controls on the frozen 256-question test. The market probability supplied in its input remains a substantially stronger forecast.

Results

Lower Brier and log score are better. All numbers below use the same 256 held-out questions.

Condition	Brier	Log score
Frozen market prior	0.10156	0.32981
Frozen Qwen3.5-9B baseline, raw	0.15076	0.48045
Frozen Qwen3.5-9B baseline, calibrated	0.15770	0.50338
Adapted Gemma, calibrated	0.15833	0.49507
Adapted Gemma, raw	0.18742	0.77451
Shuffled-outcome LoRA, calibrated	0.25231	0.70700
Format-only LoRA, calibrated	0.26655	0.81282
Base Gemma, calibrated	0.29587	0.96210
Base Gemma, raw	0.37522	2.98300

Calibration and epoch selection use disjoint validation subsets (55 and 56 questions). Adapter temperature is 2.0 with zero bias. Control adapters were independently trained with the same three-epoch schedule; their calibration parameters were fitted on the same calibration subset. Their weights and per-question probabilities are included.

qwen_evaluation/ implements ForecastQwen-Live as a frozen evaluation baseline only. Qwen3.5-9B uses the same 256-question test, a two-token readout, thinking disabled, and the same 55 calibration IDs. Its raw score is stronger than its validation-fitted calibration on this test. The comparison does not establish a significant difference between the two models, and both trail the supplied market prior. No Qwen fine-tuning, renamed Qwen checkpoint or live-tournament record is implied.

Data and training

Questions come from the ForecastBench dataset, attributed in forecastbench_2026_data/LICENSE.md. Question text, criteria, background and market priors were frozen from commits before each forecast deadline. Sources are Manifold, Metaculus and Polymarket. Only binary questions with resolved outcomes by the declared outcome cutoff were selected.

Training deadlines span March–December 2025; validation spans January–March 2026; test deadlines are June 7, August 2 and September 13, 2026. Question IDs are disjoint. Related events may remain correlated. Data snapshots, outcome cutoffs, upstream revisions and preparation code are in forecastbench_2026_data/.

Base revision: dcc83ea841ab6100d6b47a070329e1ba4cf78752. LoRA targets Q/V projections, rank 16, alpha 32; learning rate 1e-4, batch size 4, three epochs, one seed. Epoch 2 was selected by validation log score. Root adapter weights are this selected real-question adapter.

Interpretability evidence

forecastbench_analysis/analysis.json contains five adapter strengths, all 26 individual layer removals, and 27 hidden-state comparisons on 64 fixed test prompts. Each layer removal is an actual intervention on adapter contributions. It is not a discovered circuit or a semantic feature label.

Reloaded predictions and restored predictions matched the original exactly on the recorded GPU configuration. The follow-up analysis used an already opened test set and did not select or retrain the release checkpoint. Removing some layers improves this checkpoint's test score; the published weights retain the validation-selected full adapter. No SAE decomposition or cross-model causal-transfer result is claimed.

Use

Accept the base model's access terms and authenticate with Hugging Face first. Reproduction environment: PyTorch 2.6.0, Transformers 4.57.1, PEFT 0.17.1, Accelerate 1.10.1, SentencePiece 0.2.1. GPU evaluation used bfloat16 and SDPA.

Download forecast.py alongside your script, then:

from forecast import Forecaster

predictor = Forecaster()  # Pin revision to a release commit for a frozen experiment.
result = predictor.predict([{
    "forecast_due_date": "2026-11-01",
    "question": "Will the specified public mission launch by December 31, 2026?",
    "resolution_criteria": "YES only if the named mission lifts off by the deadline.",
    "background": "Insert source-checked information available at the forecast date.",
    "freeze_datetime": "2026-10-31T12:00:00Z",
    "prior_probability": 0.40,
}])
print(result)  # Calibrated probability, raw binary readout, and legal-token mass.

This illustrative input is not a measured forecast. The helper performs no retrieval, fact checking or date enforcement. Supply genuinely contemporaneous information. generate() output is not the evaluated probability interface. CPU float32 inference is supported by the helper but was not used for the reported GPU parity result.

Limits and licensing

This is a retrospective, single-seed study on a small, selected market-question cohort. It has no live-tournament record, future deployment evaluation, general-reasoning utility test or demonstrated advantage over the supplied market prior. It should not replace that stronger baseline.

Modified artifact notice: these adapter weights were trained by itsalissonsilva; they modify Google Gemma behavior. Gemma and this derivative are subject to the included Gemma Terms of Use and their Section 3.2 restrictions, including the incorporated Gemma Prohibited Use Policy. See LICENSE, NOTICE, and GEMMA_PROHIBITED_USE_POLICY.txt. ForecastBench question data carry their separately documented CC BY-SA 4.0 terms; this model card does not relicense them.