README.md

---

library_name: pytorch
tags:
- chimera-ml
- oragen
- pytorch
- audio
- image
- multimodal
- age-estimation
- gender-recognition
- wav2vec2
- vit
datasets:
- AGENDER
- CommonVoice
- TIMIT
- LAGENDA
- IMDB-clean
- AFEW
- VoxCeleb2
- BRAVE-MASKS
base_model:
- facebook/wav2vec2-large-robust
- nateraw/vit-age-classifier
---


# ORAGEN Models

This repository contains exported ORAGEN-based model weights for [`chimera-ml`](https://github.com/markitantov/chimera_ml/).

These checkpoints are used for age estimation and gender recognition from speech, face images, and combined audio-visual inputs. In the `chimera-ml` ORAGEN pipeline, the multimodal model operates on intermediate audio and visual features extracted from the unimodal branches.

## Files

- `audio_model.pt` — audio-only checkpoint used for speech-based age estimation and gender recognition.
- `image_model.pt` — image-only checkpoint used for face-based feature extraction and prediction in the ORAGEN pipeline.
- `multimodal_model.pt` — audio-visual checkpoint that combines audio and image features for multimodal prediction.

## What They Predict

These models predict:

- age (0-100)
- gender (`female`, `male`)

The ORAGEN codebase also contains support for mask-related prediction in some model variants, but the exported multimodal configuration used here has `include_mask: false`.

## Training Setup

According to the training configs in `examples/oragen/configs`:

- Audio training uses `facebook/wav2vec2-large-robust` as the backbone.
- The multimodal setup uses `agender_multimodal_model_v3`.
- The visual branch is used as an image feature extractor in the fusion pipeline and is referenced together with `nateraw/vit-age-classifier`-based ORAGEN visual weights.
- Training and inference use `16 kHz` audio and `4s` windows with `2s` shift.

Datasets referenced by the configs:

- Audio: `AGENDER`, `CommonVoice`, `TIMIT`
- Image: `LAGENDA`, `IMDB-Clean`, `AFEW`
- Multimodal: `VoxCeleb2`, `BRAVE-MASKS`

## Per-Corpus Results

The training logs do not report raw accuracy directly. For gender prediction, the reported classification metrics are `gen_precision`, `gen_uar`, and `gen_macro_f1`. For age prediction, the reported regression metrics are `age_mae` and `age_pcc`.

## Results from the original paper

### Audio Model

| Corpus | Age MAE | Age PCC | Gender UAR, % | Gender Macro F1, % |
|--------|---------|---------|------------|-----------------|
| AGENDER | 10.60 | 0.83 | 87.17 | 86.25 |
| CommonVoice | 10.47 | 0.81 | 92.59 | 92.64 |
| TIMIT | 6.90 | 0.91 | 98.60 | 98.58 |
| VoxCeleb2 | 9.91 | 0.60 | 90.00 | 88.71 |
| BRAVE-MASKS (test) | 11.89 | 0.64 | 86.22 | 85.18 |

### Image Model

| Corpus | Age MAE | Age PCC | Gender UAR, % | Gender Macro F1, % |
|--------|---------|---------|------------|-----------------|
| LAGENDA | 5.18 | 0.95 | 92.89 | 92.90 |
| AFEW | 5.62 | 0.82 | 95.16 | 94.98 |
| IMDB-Clean (test) | 5.47 | 0.84 | 98.37 | 98.26 |
| VoxCeleb2 | 5.97 | 0.64 | 98.37 | 98.16 |
| BRAVE-MASKS (test) | 8.71 | 0.74 | 94.44 | 94.43 |

### Multimodal Model (intermediate fusion)

| Corpus | Age MAE | Age PCC | Gender UAR, % | Gender Macro F1, % |
|--------|---------|---------|------------|-----------------|
| VoxCeleb2 | 5.68 | 0.66 | 99.11 | 99.02 |
| BRAVE-MASKS (test) | 8.73 | 0.74 | 94.95 | 94.89 |


## 6) Related publications

Markitantov M., Ryumina E., Karpov A. Audio-visual occlusion-robust gender recognition and age estimation approach based on multi-task cross-modal attention. // Expert Systems with Applications. 2026. vol. 296. ID 127473. https://doi.org/10.1016/j.eswa.2025.127473

BibTeX:

```bibtex

@article{markitantov2026oragen,

  author = {Markitantov, Maxim and Ryumina, Elena and Karpov, Alexey},

  title = {Audio-visual occlusion-robust gender recognition and age estimation approach based on multi-task cross-modal attention},

  journal = {Expert Systems with Applications},

  volume = {296},

  pages = {127473},

  year = {2026},

  month = jan,

  doi = {10.1016/j.eswa.2025.127473},

  url = {https://doi.org/10.1016/j.eswa.2025.127473}

}

```