HypoAgent-Mixtral-8x7B-Instruct-SFT

HypoAgent: a LoRA Mixtral-8x7B that writes prespecified statistical analysis plans

HypoAgent is a LoRA adapter for Mixtral-8x7B-Instruct-v0.1. Hand it a study description — objective, design, population, intervention or exposure, comparator, outcome, timing, whatever the registry record contains — and it returns a prespecified analysis plan: target estimand, null and alternative hypotheses, a primary method with the reason it fits this design, assumptions and the diagnostics that check them, effect and uncertainty reporting, sample-size logic, missing-data rules, multiplicity control, sensitivity analyses, and a decision rule.

The point of the adapter is not fluency. Base Mixtral already writes readable statistics prose — the rubric below scores it 6.1/10 on clarity and 6.1 on staying faithful to the study description. Where it falls down is the operational content underneath: 3.4/10 on analysis populations and data handling, 3.5 on inference and multiplicity, 3.8 on reproducibility, and one plan in three carrying an error the rubric classes as critical. That is the gap the adapter closes.

It was adapted on the Adaption platform, which scores every adapted model head-to-head against its base. A separate rubric audit on held-out data provides an independent check.

Evaluation	Setting	Result
Adaption internal (platform-reported)	adapted vs. base Mixtral, in-domain on the HypoAgent dataset	96% win rate vs. 4%
Adaption internal, Science category	all tasks processed in the platform's Science category	72% vs. 28%
Rubric audit: 2,719 held-out study descriptions	GPT-5.5 (Codex) judge, seven-aspect weighted rubric	93.6% win / 4.9% loss / 1.5% tie

The Adaption figures are the metric the platform scores on. They are reported by the platform and the per-task scores are held there, so they are not reproducible from this repo. The rubric audit is the independent check: it runs on the held-out test split with the rubric and cumulative report released in eval/.

Trained on HypoAgent/HypoAgent-SFT — 54,163 study-context → plan pairs synthesized from ClinicalTrials.gov, the AEA RCT Registry and OSF Registrations. The exact file handed to the training platform, corpus pairs plus the platform's enhanced rewrites, is on Kaggle.

Results

Adaption platform internal win rate

The Adaption platform evaluates the adapted model against the base model on its own held-out tasks. HypoAgent was preferred in 96% of comparisons on the HypoAgent dataset (vs. 4%), and in 72% (vs. 28%) across every task the platform processed in its Science category — a broader pool than this dataset, so the second number is the harder and more general one.

These are the challenge's own numbers, reported by the platform. The per-task scores stay on the platform and are not mirrored here, which is why the rubric audit below exists.

Rubric audit: 2,719 held-out study descriptions

	Base Mixtral-8x7B-Instruct	+ HypoAgent LoRA	Difference
Preferred by the judge	4.89% (133)	93.56% (2,544)	ties 1.54% (42)
Mean weighted score, 0–10	4.463	6.907	+2.445
Plans containing a critical error	920 (33.8%)	391 (14.4%)	−57.5%

Every study description in this evaluation comes from the held-out test split of the training corpus. The split is grouped by study_family_id, so no study that produced a test description contributed any training example — the tasks derived from one registry entry (full plan, method selection, completion, critique, alternatives) all travel together into the same split. Leakage between train and test is zero families.

Both arms ran on the same base weights with the same system prompt and the same decoding settings; the only difference is whether the adapter was loaded. A GPT-5.5 (Codex) judge then scored the two anonymised plans against eval/RUBRIC.md, seven aspects on a 0–10 scale, with the study description as the sole factual authority.

Of the 2,677 comparisons the judge decided, the adapter took 2,544 — 95.0%. A sign test on the decided cases gives p < 10⁻³⁰⁰. The 42 ties are comparisons whose weighted scores landed inside the rubric's 0.25-point band with neither side holding a lone critical error.

Where the gain comes from

The ordering matters more than the totals. The three biggest gains are in reproducibility (+3.64), statistical method appropriateness (+3.07) and inference, multiplicity and power (+2.85) — the aspects that ask whether another analyst could actually run the plan and whether the test is the right one. The two smallest are clarity (+1.52) and study fidelity (+1.53), which is what you would expect: base Mixtral was already writing clear prose that stayed close to the study description. It was the operational content underneath that was missing.

The base model scores below 4/10 on data handling (3.44), inference (3.51) and reproducibility (3.78). Those are the sections it tends to skip entirely or fill with boilerplate — no analysis populations, no censoring rule, no model formula, no covariate list, no alpha convention. The adapter's outputs land in the 6.2–7.6 band across all seven aspects, which the rubric anchors at "adequate but with meaningful gaps" to "strong, with only minor deficiencies". It is a draft worth editing, not a finished SAP.

The rubric defines a critical error as a fabricated design feature or primary endpoint, a method incompatible with the outcome or design, or an internal contradiction that would change the primary inference. Any of them normally caps the plan's overall score at 5.0. The base model triggered one in a third of its plans; the adapter in one in seven. That cap is also why the mean weighted score (6.907) sits below what you would get by combining the mean aspect scores directly — the overall is computed per report with the safeguard applied, then averaged.

Scores were accumulated over four audited cohorts (evaluation_500, evaluation_501_1000, evaluation_1001_2000, evaluation_2001_2719); the totals above are the pooled result. The full rubric and the cumulative report are in eval/.

What the evaluation does not establish

One judge, one pass. This is a single high-effort GPT-5.5 run, not a multi-judge panel and not a biostatistician adjudication. LLM judges reward structure, and a 20-heading plan is structurally impressive whether or not the statistics inside it are right. The critical-error count is the part of the rubric designed to resist that, and the adapter still trips it 391 times.
Length is uncontrolled, and the gap is large. Across the 2,719 generations the adapter averages 7,567 characters against base Mixtral's 3,112 — 2.43× longer. LLM judges are known to prefer longer answers, no length-matched subgroup was run, and PaRLA's equivalent analysis found the preference survived length control only at a reduced margin. Treat some unmeasured share of the 93.6% as verbosity. The critical-error count (920 → 391) is the length-independent part of the result.
The test descriptions come from the same registries as training. Different studies, zero family overlap, but the same three sources and the same extraction pipeline. This measures generalization to unseen studies, not to a new source of study descriptions or to free-form text written by a researcher.
The reference plans were never shown to the judge. It scored the two candidates against the study description only, so the comparison is model-vs-model, not model-vs-ground-truth.

Training

Data. The curated train split of HypoAgent-SFT, quality-filtered and rebalanced so that non-clinical designs and non-Gaussian outcomes are not swamped by two-arm clinical RCTs with continuous endpoints. Five task types — full plan, method selection, plan completion, critique of a deliberately flawed plan, and preferred-plus-alternative.

The split ships 48,218 pairs from 18,699 study families; the Adaption run consumed 47,700 of them after platform-side validation, so roughly 500 rows were dropped before training.

The platform also rewrites the data before training. The file it was handed carries each example twice — the corpus prompt and plan, plus a platform-enhanced version of both — and the adapter's output style follows the enhanced version, not the corpus template. That file is published as data.jsonl on Kaggle; see What comes back for what it means in practice.

Adapter.


Type	PEFT LoRA, `CAUSAL_LM`, 218 MB
Base	Mixtral-8x7B-Instruct-v0.1 (MoE, 8 experts, top-2, bf16)
Rank / alpha / dropout	64 / 128 / 0.0
Target modules	`q_proj`, `k_proj`, `v_proj`, `o_proj`
Bias, RSLoRA, DoRA	none, off, off

Run. 5 epochs, 2,275 optimizer steps (455 per epoch), per-device batch size 1, bf16, evaluated at each epoch boundary. Learning rate warmed up to a peak of 3e-4 by step 115 and decayed to 3e-5. Training loss fell 1.06 → 0.42.

Epoch	1	2	3	4	5
Eval loss	0.5227	0.4974	0.4910	0.4907	0.4927

Eval loss was computed on a small monitoring subset (about a dozen examples), so the flat 0.491–0.493 band from epoch 3 onward should be read as "converged", not as evidence about which checkpoint generalizes best. The released weights are the epoch-5 checkpoint. Full log: trainer_state.json.

How to use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "mistralai/Mixtral-8x7B-Instruct-v0.1"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "HypoAgent/HypoAgent-Mixtral-8x7B-Instruct-SFT")

SYSTEM = ("You are an expert statistical study-design assistant. Produce rigorous, "
          "transparent, and reproducible hypothesis-testing plans.")

user = """STUDY DESCRIPTION
Objective: Test whether a clinic-level nurse coaching programme improves glycaemic control.
Domain: clinical / Type 2 Diabetes
Design: cluster_randomized (randomized, parallel, masking: none)
Population / eligibility: Adults with HbA1c above 8.0% at participating primary-care clinics
Intervention/exposure: Structured nurse coaching, 12 sessions
Comparator: usual care
Primary outcome: Change in HbA1c
Primary timing: 12 months
Number of groups/arms: 2
Planned enrollment: 1,400 across 40 clinics

Task: Produce a rigorous, prespecified hypothesis-testing and statistical-analysis plan for this study."""

msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": user}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=1600, do_sample=False)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

Keep the system prompt and the STUDY DESCRIPTION / Task: framing — that is the shape the adapter saw in every training example, and it drifts toward generic textbook prose without it. The adapter ships the Mixtral-Instruct chat_template.jinja. Greedy decoding was used throughout the evaluation. All results above are at bf16; the adapter also loads on 4-bit NF4 base weights, untested at that precision.

Missing fields are fine and are the normal case. The corpus is full of registry records with no reported alpha, no ICC and no effect size, and the adapter is trained to name the missing input as something that must be prespecified rather than to invent a number for it.

What comes back

Every plan opens # Hypothesis-Testing and Statistical-Analysis Plan followed by ## 1. Research question. Past that the structure is semi-stable, not fixed: the adapter covers the same ground on every study but does not emit a constant section list. Measured across all 2,719 held-out generations:


Opens with `## 1. Research question`	98%
Uses LaTeX for hypotheses and estimands	93%
Emits an explicit `Reasoning:` block	52%
Mean length	7,567 characters

The sections that recur, in rough order of frequency: research question, target estimand, primary endpoint, study and analysis units, assumptions and how to check them, model diagnostics, analysis steps, significance criteria, subgroup analyses, reproducibility specifications, and limitations or unresolved dependencies. A typical opening:

# Hypothesis-Testing and Statistical-Analysis Plan

## 1. Research question
"What is the incidence of late lumen loss at 9 months post-implantation ...?"

Reasoning: The primary outcome variable is defined by the protocol as the target
metric assessed at "9 months". As a single-arm trial, there is no concurrent
control group; thus, comparative efficacy claims require external historical
benchmarks, which are absent here. ...

## 2. Plan

### Estimands and Hypotheses
*   **Target Estimand:** Mean Late Lumen Loss (LLL) at 9 months compared to baseline.
*   **Null Hypothesis ($H_0$):** $\mu_{LLL} = \mu_{baseline}$ ...

Heading level and numbering drift between runs — ## 3. Target Estimand in one plan, ### Target estimand in another. Parse on heading text, not on a fixed numbered list.

Why this differs from the corpus targets. The file submitted to Adaption carries each example twice: the HypoAgent corpus pair, and a platform-enhanced rewrite of both prompt and plan. The corpus targets use a fixed twenty-section template with almost no notation (1% contain LaTeX, mean 3,569 characters). The enhanced targets are longer and heavily mathematical (66% LaTeX, mean 6,304 characters) with variable sectioning. The adapter follows the enhanced style: none of the 2,719 generations reproduces the corpus's twenty-section template. If you want that fixed template, it is in the corpus — the adapter will not give it to you.

Intended use and limitations

Intended use. A drafting assistant for researchers and biostatisticians: generating a first analysis plan, checking an existing one for missing components, arguing a method choice, or getting a second option to compare against. It covers clinical trials, economics and social-science field and lab experiments, and behavioural preregistrations.

Not a substitute for a statistician, and not medical or clinical advice. Every plan needs review before it goes near a protocol, a registration or a regulator.

Training targets are synthesized, not authoritative. The assistant responses in the corpus were built by a rule-based statistical-reasoning engine from extracted structured facts — they are not real statistical analysis plans written by trial statisticians. The adapter learned to produce correct structure and method selection; it did not learn from expert-authored SAPs and should not be treated as reproducing one.

Specifics still get invented. Non-inferiority margins, ICCs, minimal clinically important differences and effect sizes are the fields most likely to appear as plausible-looking numbers. The model is trained to flag them as prespecification decisions instead, and mostly does, but check every number in an output against the source protocol.

Known failure surface. The 14.4% critical-error rate is the honest floor, and it is not spread evenly. Broken out by design across the same 2,719 comparisons, the adapter still trips a critical error in 25.8% of survey and generic-experiment plans (n = 62), 22.2% of observational studies (n = 307) and 21.6% of non-randomized parallel designs (n = 134), against 10.9% for parallel RCTs and 9.6% for crossovers. The association/regression method family is the weakest on both measures at once — lowest win rate (86.2%) and highest residual critical-error rate (24.8%).

The pattern is causal inference without randomization, not exotic designs. Give it a clean randomized contrast and it is reliable; give it an observational question where the hard part is confounding rather than method selection and it is the least trustworthy. Treat observational plans as a starting sketch.

Inherited limits. Mixtral-8x7B-Instruct's biases and knowledge cutoff carry through. Non-English study descriptions are untested.

License and provenance

Research use. The base model is Apache-2.0; this adapter is a derivative distributed for research. Training data has mixed provenance: ClinicalTrials.gov records are US-government public domain, while the AEA RCT Registry and OSF Registrations portions have unset or unverified content licenses and are tagged human_review_status=license_review in the corpus. No verbatim source text is reproduced in any training target — every assistant response is synthesized from extracted structured fields. See the dataset card for the per-source breakdown.

Attribution: the base-model authors, and the HypoAgent corpus (record-level source_ids and source_urls point back to the originating registry entry).

Model tree for HypoAgent/HypoAgent-Mixtral-8x7B-Instruct-SFT

Base model

mistralai/Mixtral-8x7B-v0.1

Finetuned

mistralai/Mixtral-8x7B-Instruct-v0.1

Adapter

(147)

this model