whisper-large-v3-braintuned (wh_ff_1ep)

Whisper-large-v3 whose encoder was fine-tuned to predict human fMRI responses to movies, then transferred with no further training to predicting brain activity from held-out subjects and from a different recording modality.

Updated 2026-07-28. This replaces the previous wh_r32_1ep (LoRA r32) upload with the full fine-tune, which transfers better to intracranial EEG.

Results

Benchmark	frozen	brain-tuned	Δ
Algonauts fMRI, OOD season 6	0.1904	0.2131	+0.0227 (+12%)
iEEG podcast (Zada 2025, ds005574), 9 subjects	0.0605	0.0629	+0.0024 (+3.9%, paired p=0.039)

How it was trained

Full-rank fine-tune of q/k/v/out_proj across the 31 encoder layers feeding the readout, 1 epoch (16,300 steps, batch 6), CNeuroMod Friends seasons 1–5, whole-brain target (1000 Schaefer parcels).

The readout is pinned at its closed-form ridge solution on the frozen features and never trained. That matters: with a co-trained head the encoder can lower the loss by compensating for a deficient decoder instead of improving the representation, and those compensations hurt downstream.

Use

from transformers import WhisperForConditionalGeneration, AutoProcessor
m = WhisperForConditionalGeneration.from_pretrained("RAntonello/whisper-large-v3-braintuned")
p = AutoProcessor.from_pretrained("RAntonello/whisper-large-v3-braintuned")
hs = m.model.encoder(inputs, output_hidden_states=True).hidden_states  # read layer 31 or 32

Weights are float32 (the base ships fp16; full-FT deltas were fp32 master weights, so fp32 preserves them exactly — verified to 2.8e-09 against base + delta).

Read these before using it

• The ECoG benefit sits only at the top of the stack. Averaged over all 33 encoder layers the tuned model is −0.7% versus frozen; the gain is at layers 27–32.
• Only 2 of 33 layers reach p<0.05, and L32 at p=0.039 would not survive correction for 33 comparisons. Per-subject spread is wide (median +3.0%, range −2.9% to +24.9%, 8/9 positive).
• Causal extraction beats bidirectional on ECoG (+0.0772 vs +0.0747 frozen) even though the model was tuned bidirectionally.
• Training with the readout at layer 32 instead of 31 does not help (−0.0008 head-to-head, p=0.30).
• The training run used ridge_clips=2000 (1.6 samples/input-dim), which inflates the trainer's own validation numbers; a 40k-clip refit lifts the frozen baseline 0.1484 → 0.1859. The benchmark and ECoG figures above refit ridge from scratch and are unaffected.

See braintune_provenance.json for the full configuration.