CoVeR checkpoints

Checkpoints of CoVeR (Counterfactual-View Reliability learning) from Words Speak Louder? Exploring the Language Shortcut of LLM-Driven Incomplete Multimodal Learning.

Wenhao Li^1,2, Chong Xiao¹, Xiaokun Bai¹, Rundong He³, Zhuang Qi⁴, Xusheng Liang^5,2, Zhibin Wu¹, Qiangchang Wang^1
1Shandong University, ²Shenzhen Loop Area Institute, ³The Hong Kong Polytechnic University, ⁴National University of Singapore, ⁵City University of Hong Kong

Code | MR-Bench-H dataset | Project page

CoVeR keeps the architecture, the inputs and the inference procedure of LLM-driven fusion unchanged and modifies only how it is trained: the counterfactual views of a sample are trained as unimodal judges, and their disagreement decides which samples supervise the fused path. In accuracy mode it outputs the prediction of the full view in one forward pass, with the architecture and the inference cost of the LLM-Fusion backbone.

Contents

One CoVeR run of seed 42 per source corpus of MR-Bench-H, trained with the hyper-parameters selected on the validation split (--recipe selected of the code): Qwen3-0.6B with LoRA on the attention projections, K = 4 fusion tokens, τ = 0.4, λ_uni = 2, learning rate 2e-4.

Corpus	Run directory	LoRA rank	Epochs	Weight averaging
CMU-MOSI	mosi/cover_s42	16	30	0.99
CMU-MOSEI	mosei/cover_s42	16	15	0.99
CH-SIMS	sims/cover_s42	32	30	0.99
CH-SIMSv2	simsv2/cover_s42	16	30	–
IEMOCAP	iemocap/cover_s42	16	20	0.99
MELD	meld/cover_s42	16	15	0.99
MUStARD++	mustardpp/cover_s42	16	30	0.99

Every run directory has the layout of results/runs/ in the code repository:

• config.json: hyper-parameters of the run;
• best.pt: the checkpoint with the best validation Avg Corr (the tensors outside the frozen LLM and its LoRA weights; load it with the code repository);
• results.json: test metrics of this checkpoint computed with the code repository, in accuracy mode and robust mode (seven availability patterns, random missing rates and, where annotations exist, MR-Bench-A).

Usage

git clone https://github.com/peacelwh/CoVeR && cd CoVeR
pip install -r requirements.txt
hf download Bluexxx/CoVeR --local-dir results/runs --include "*/cover_s42/*"

# seven availability patterns, random missing rates and MR-Bench-A (processed features: see data/README.md)
python scripts/evaluate.py --run results/runs/mosi/cover_s42

MR-Bench-H

A fusion model reads features, so each clip of MR-Bench-H is scored, with all modalities observed, by the model trained on the training split of its source corpus. The seven checkpoints cover the seven sources:

hf download Bluexxx/MR-Bench-H --repo-type dataset --local-dir MR-Bench-H
for ds in mosi mosei sims simsv2 iemocap meld mustardpp; do
  python scripts/predict_mrbench_h.py --run results/runs/$ds/cover_s42 --items MR-Bench-H/data/test.jsonl
done
python scripts/score_mrbench_h.py --items MR-Bench-H/data/test.jsonl --pred "results/mrbench_h/cover_*_s42.jsonl"

cover is accuracy mode and cover-robust is robust mode. The paper reports the mean and standard deviation over three seeds of the accuracy pooled over the seven sources (61.7 ± 2.8 LS / 93.6 ± 1.3 NS in accuracy mode, 63.0 ± 2.0 LS / 84.8 ± 0.7 NS in robust mode); these checkpoints are the runs of seed 42, one of the three seeds. With them, the commands above give 62.4 LS / 93.9 NS in accuracy mode and 65.8 LS / 85.4 NS in robust mode (mrbench_h_s42.json in this repository is the output of the scorer).

Results reported in the paper

Comparison under incomplete modalities, averaged over the seven test-time availability patterns (Acc2 and F1 in %; mean over 3 seeds):

Benchmark	Acc2 ↑	F1 ↑	MAE ↓	Corr ↑
CMU-MOSI	69.1	68.9	1.119	0.533
CMU-MOSEI	77.0	76.2	0.659	0.603
CH-SIMS	77.2	76.1	0.469	0.530
CH-SIMSv2	76.9	76.7	0.357	0.615
IEMOCAP	82.6	82.6	0.534	0.681

MR-Bench-H, all 900 clips (620 LS, 280 NS; mean ± std over three seeds):

Mode	LS accuracy (%)	NS accuracy (%)	LS ShortErr (%)
CoVeR, accuracy	61.7 ± 2.8	93.6 ± 1.3	38.3
CoVeR, robust	63.0 ± 2.0	84.8 ± 0.7	37.0

Citation

@article{li2026words,
  title  = {Words Speak Louder? Exploring the Language Shortcut of LLM-Driven Incomplete Multimodal Learning},
  author = {Li, Wenhao and Xiao, Chong and Bai, Xiaokun and He, Rundong and Qi, Zhuang and Liang, Xusheng and Wu, Zhibin and Wang, Qiangchang},
  year   = {2026}
}

License

The code and these checkpoints are released under the MIT License. The checkpoints build on Qwen3-0.6B (Apache-2.0) and store its token-embedding table next to the trained tensors. The source corpora keep their own terms. The models are intended for research on multimodal understanding and robustness; we do not recommend deploying affect recognition in high-stakes decisions about individuals.