CAM-AH / README.md

Add CAM-AH ensemble-5 original weights (official Macro-F1 0.7456 @ thr 0.595)

2e0431f verified about 1 month ago

6.14 kB

---
tags:
  - audio
  - text
  - multimodal
  - cross-attention
  - multiple-instance-learning
  - ambivalence
  - hesitancy
  - abaw
  - pytorch-lightning
---
# CAM-AH — Cross-Attention-Multimodal-Ambivalence-Hesitation

Original weights of the model behind our **best official submission** to the
*AH Video Recognition Challenge, 3rd edition (ABAW11 @ ECCV 2026)*: a **5-seed ensemble** of an
audio ⟷ text cross-attention network with psycholinguistic support features and gated
attention-MIL pooling, run at a **fixed decision threshold of 0.595**.

| Official private test (152 videos) | Value |
|---|---|
| Macro-F1 | **0.7456** |
| Average precision (class 1) | 0.8182 |
| Decision threshold | 0.595 (60 predicted positives) |

- Paper: [*Audio-Text Cross-Attention with Psycholinguistic Support Features for Ambivalence/Hesitancy Recognition*](https://arxiv.org/abs/2607.13345)
- Code: [Liga-de-IA-PUCPR/abaw-11-ah-challenge](https://github.com/Liga-de-IA-PUCPR/abaw-11-ah-challenge)

## Files

```
members/<run>/best-epoch=XX-val_ap=0.XXXX.ckpt   # original PyTorch Lightning checkpoints (5 members)
members/<run>/trainer_state.json                 # per-member dims / fusion options
ensemble_config.json                             # members, combination rule, threshold, hyperparameters
```

Each checkpoint is the unmodified Lightning file (weights + optimizer/loop state). The model
weights live in `state_dict` under the `model.` prefix, about 2.96M parameters per member.
The `threshold` field of each `trainer_state.json` is a per-member calibration. The ensemble
ignores it and uses the fixed 0.595 from `ensemble_config.json`.

## Model

Each video is sliced into **5 s windows (hop 2.5 s)** aligned to the Whisper transcript by
timestamp. Each window gets three feature vectors:

| Input | Dim | Source |
|---|---|---|
| `feat_a` audio | 320 | librosa prosodic features (80 features × 4 statistics) |
| `feat_b` text | 768 | [`cardiffnlp/twitter-roberta-base-emotion`](https://huggingface.co/cardiffnlp/twitter-roberta-base-emotion), masked mean pool + L2 |
| `feat_tab` support | 74 | question type (7) + acoustic hesitation (18) + textual A/H cues (49) |

Per member, with inputs of shape `(B, T, ·)` and a padding mask `(B, T)`:

1. Project audio and text to 512-d. Audio is the query and text is the key/value in an 8-head
   cross-attention, followed by a residual connection and LayerNorm.
2. **Token fusion.** The support features go through a masked BatchNorm, then Linear + ReLU +
   LayerNorm, and are concatenated with each window token before a fusion layer (512-d).
3. **Gated attention-MIL pooling** (Ilse et al., 2018) over the windows produces one vector
   per video.
4. An MLP head (512 → 512 → 1, dropout 0.3) outputs one logit per video.

**Ensemble:** average the 5 members' sigmoid probabilities. A video is predicted positive
(`1`) when the mean probability is at least **0.595**.

## Training

- Data: BAH `train` + `val` splits (902 videos). Checkpoint selection and early stopping
  used AP on the internal `test` split (525 videos); this is the `val_ap` in the file names.
- Members: 5 identical models with seeds `{42, 1, 2, 3, 4}`.
- Optimization: AdamW, lr 3e-4, weight decay 0.05, gradient clip 1.0, batch size 32, early
  stopping patience 20. The models saturate in 1–2 epochs.
- Threshold: the ranking comes from the ensemble, and the operating point was fixed at 0.595
  for the official test set. Its positive rate is lower than that of the internal
  calibration split, and the original calibrated threshold was 0.45 (official Macro-F1 0.6837).

## Usage

The features have to be produced by the project's pipeline (`make data`) or by the
self-contained notebook `notebooks/ensemble5_pipeline.ipynb` in the code repository. The
network definition is `src/models/cross_attention.py` (`_CrossAttentionFusionModule`).

```python
import json, torch
from huggingface_hub import snapshot_download
path = snapshot_download("LF-BF/CAM-AH")
cfg = json.load(open(f"{path}/ensemble_config.json"))
def load_member(ckpt_file, module):
    sd = torch.load(f"{path}/{ckpt_file}", map_location="cpu", weights_only=False)["state_dict"]
    module.load_state_dict({k.removeprefix("model."): v for k, v in sd.items()})
    return module.eval()
# members = [load_member(f, build_module(cfg["architecture"])) for f in cfg["members"]]
# with torch.no_grad():
#     p = torch.stack([m(feat_a, feat_b, pad_mask, feat_tab).sigmoid() for m in members]).mean(0)
# pred = (p >= cfg["threshold"]).long()
```

## Data and terms of use

The model was trained on the **BAH dataset** (Behavioural Ambivalence/Hesitancy). The
dataset is **not** included here and is distributed only by its authors under their EULA
([github.com/sbelharbi/bah-dataset](https://github.com/sbelharbi/bah-dataset)). These
weights are intended for non-commercial academic research.

This work was carried out using the BAH dataset: a Multimodal Dataset – created and managed by
the Montréal Behavioural Medicine Centre (MBMC: https://mbmc-cmcm.ca) and the Imaging, Vision
and Artificial Intelligence Laboratory (LIVIA: https://www.etsmtl.ca/laboratoires/livia).

## Citation

```bibtex
@article{martins2026audiotext,
  title   = {Audio-Text Cross-Attention with Psycholinguistic Support Features
             for Ambivalence/Hesitancy Recognition},
  author  = {Martins, Luiz F. B. F. and Pisaia, Rodrigo W. and Girardi, Matheus M.
             and Berkembrock, Isabella and Almeida, Jo{\~a}o A. and Hochuli, Andr{\'e} G.
             and Laroca, Rayson and Britto Jr., Alceu S.},
  journal = {arXiv preprint arXiv:2607.13345},
  year    = {2026},
  doi     = {10.48550/arXiv.2607.13345},
  url     = {https://arxiv.org/abs/2607.13345}
}
@inproceedings{gonzalez-26-bah,
  title     = {{BAH} Dataset for Ambivalence/Hesitancy Recognition in Videos for Digital Behavioural Change},
  author    = {González-González, M. and Belharbi, S. and Zeeshan, M. O. and Sharafi, M. and
               Aslam, M. H and Pedersoli, M. and Koerich, A. L. and Bacon, S. L. and Granger, E.},
  booktitle = {ICLR},
  year      = {2026}
}
```

	---
	tags:
	- audio
	- text
	- multimodal
	- cross-attention
	- multiple-instance-learning
	- ambivalence
	- hesitancy
	- abaw
	- pytorch-lightning
	---

	# CAM-AH — Cross-Attention-Multimodal-Ambivalence-Hesitation

	Original weights of the model behind our best official submission to the
	AH Video Recognition Challenge, 3rd edition (ABAW11 @ ECCV 2026): a 5-seed ensemble of an
	audio ⟷ text cross-attention network with psycholinguistic support features and gated
	attention-MIL pooling, run at a fixed decision threshold of 0.595.

	\| Official private test (152 videos) \| Value \|
	\|---\|---\|
	\| Macro-F1 \| 0.7456 \|
	\| Average precision (class 1) \| 0.8182 \|
	\| Decision threshold \| 0.595 (60 predicted positives) \|

	- Paper: [Audio-Text Cross-Attention with Psycholinguistic Support Features for Ambivalence/Hesitancy Recognition](https://arxiv.org/abs/2607.13345)
	- Code: [Liga-de-IA-PUCPR/abaw-11-ah-challenge](https://github.com/Liga-de-IA-PUCPR/abaw-11-ah-challenge)

	## Files

	```
	members/<run>/best-epoch=XX-val_ap=0.XXXX.ckpt # original PyTorch Lightning checkpoints (5 members)
	members/<run>/trainer_state.json # per-member dims / fusion options
	ensemble_config.json # members, combination rule, threshold, hyperparameters
	```

	Each checkpoint is the unmodified Lightning file (weights + optimizer/loop state). The model
	weights live in `state_dict` under the `model.` prefix, about 2.96M parameters per member.
	The `threshold` field of each `trainer_state.json` is a per-member calibration. The ensemble
	ignores it and uses the fixed 0.595 from `ensemble_config.json`.

	## Model

	Each video is sliced into 5 s windows (hop 2.5 s) aligned to the Whisper transcript by
	timestamp. Each window gets three feature vectors:

	\| Input \| Dim \| Source \|
	\|---\|---\|---\|
	\| `feat_a` audio \| 320 \| librosa prosodic features (80 features × 4 statistics) \|
	\| `feat_b` text \| 768 \| [`cardiffnlp/twitter-roberta-base-emotion`](https://huggingface.co/cardiffnlp/twitter-roberta-base-emotion), masked mean pool + L2 \|
	\| `feat_tab` support \| 74 \| question type (7) + acoustic hesitation (18) + textual A/H cues (49) \|

	Per member, with inputs of shape `(B, T, ·)` and a padding mask `(B, T)`:

	1. Project audio and text to 512-d. Audio is the query and text is the key/value in an 8-head
	cross-attention, followed by a residual connection and LayerNorm.
	2. Token fusion. The support features go through a masked BatchNorm, then Linear + ReLU +
	LayerNorm, and are concatenated with each window token before a fusion layer (512-d).
	3. Gated attention-MIL pooling (Ilse et al., 2018) over the windows produces one vector
	per video.
	4. An MLP head (512 → 512 → 1, dropout 0.3) outputs one logit per video.

	Ensemble: average the 5 members' sigmoid probabilities. A video is predicted positive
	(`1`) when the mean probability is at least 0.595.

	## Training

	- Data: BAH `train` + `val` splits (902 videos). Checkpoint selection and early stopping
	used AP on the internal `test` split (525 videos); this is the `val_ap` in the file names.
	- Members: 5 identical models with seeds `{42, 1, 2, 3, 4}`.
	- Optimization: AdamW, lr 3e-4, weight decay 0.05, gradient clip 1.0, batch size 32, early
	stopping patience 20. The models saturate in 1–2 epochs.
	- Threshold: the ranking comes from the ensemble, and the operating point was fixed at 0.595
	for the official test set. Its positive rate is lower than that of the internal
	calibration split, and the original calibrated threshold was 0.45 (official Macro-F1 0.6837).

	## Usage

	The features have to be produced by the project's pipeline (`make data`) or by the
	self-contained notebook `notebooks/ensemble5_pipeline.ipynb` in the code repository. The
	network definition is `src/models/cross_attention.py` (`_CrossAttentionFusionModule`).

	```python
	import json, torch
	from huggingface_hub import snapshot_download

	path = snapshot_download("LF-BF/CAM-AH")
	cfg = json.load(open(f"{path}/ensemble_config.json"))

	def load_member(ckpt_file, module):
	sd = torch.load(f"{path}/{ckpt_file}", map_location="cpu", weights_only=False)["state_dict"]
	module.load_state_dict({k.removeprefix("model."): v for k, v in sd.items()})
	return module.eval()

	# members = [load_member(f, build_module(cfg["architecture"])) for f in cfg["members"]]
	# with torch.no_grad():
	# p = torch.stack([m(feat_a, feat_b, pad_mask, feat_tab).sigmoid() for m in members]).mean(0)
	# pred = (p >= cfg["threshold"]).long()
	```

	## Data and terms of use

	The model was trained on the BAH dataset (Behavioural Ambivalence/Hesitancy). The
	dataset is not included here and is distributed only by its authors under their EULA
	([github.com/sbelharbi/bah-dataset](https://github.com/sbelharbi/bah-dataset)). These
	weights are intended for non-commercial academic research.

	This work was carried out using the BAH dataset: a Multimodal Dataset – created and managed by
	the Montréal Behavioural Medicine Centre (MBMC: https://mbmc-cmcm.ca) and the Imaging, Vision
	and Artificial Intelligence Laboratory (LIVIA: https://www.etsmtl.ca/laboratoires/livia).

	## Citation

	```bibtex
	@article{martins2026audiotext,
	title = {Audio-Text Cross-Attention with Psycholinguistic Support Features
	for Ambivalence/Hesitancy Recognition},
	author = {Martins, Luiz F. B. F. and Pisaia, Rodrigo W. and Girardi, Matheus M.
	and Berkembrock, Isabella and Almeida, Jo{\~a}o A. and Hochuli, Andr{\'e} G.
	and Laroca, Rayson and Britto Jr., Alceu S.},
	journal = {arXiv preprint arXiv:2607.13345},
	year = {2026},
	doi = {10.48550/arXiv.2607.13345},
	url = {https://arxiv.org/abs/2607.13345}
	}

	@inproceedings{gonzalez-26-bah,
	title = {{BAH} Dataset for Ambivalence/Hesitancy Recognition in Videos for Digital Behavioural Change},
	author = {González-González, M. and Belharbi, S. and Zeeshan, M. O. and Sharafi, M. and
	Aslam, M. H and Pedersoli, M. and Koerich, A. L. and Bacon, S. L. and Granger, E.},
	booktitle = {ICLR},
	year = {2026}
	}
	```