README.md

---
license: mit
pipeline_tag: image-classification
tags:
- facial-expression-recognition
- emotion-recognition
- fer2013
- pytorch
- onnx
library_name: pytorch
---

# Residual Masking Network (RMN) — Facial Expression Recognition

Official checkpoint for **"Facial Expression Recognition using Residual Masking Network"** (ICPR 2020).

- 📄 Paper: [Hugging Face Papers](https://huggingface.co/papers/2603.05937) · [arXiv](https://arxiv.org/abs/2603.05937) · [IEEE Xplore](https://ieeexplore.ieee.org/abstract/document/9411919)
- 💻 Code: [github.com/phamquiluan/ResidualMaskingNetwork](https://github.com/phamquiluan/ResidualMaskingNetwork)
- 🎮 Live demo: [**Try it in your browser**](https://huggingface.co/spaces/phamquiluan/ResidualMaskingNetwork) — runs locally via ONNX Runtime Web, no upload
- 🏆 State-of-the-art single-model accuracy on FER2013: **74.14%** (76.82% with ensemble)

## Files

| File | Description |
|---|---|
| `Z_resmasking_dropout1_rot30_2019Nov30_13.32` | Training checkpoint of the `resmasking_dropout1` architecture used by the [`rmn`](https://pypi.org/project/rmn/) pip package. Model weights are stored under the `"net"` key. |
| `face_detection_yunet_2023mar.onnx` | YuNet face detector used by the `rmn` package pipeline. |
| `onnx/resmasking_int8.onnx` | ONNX export of the same checkpoint, statically quantised to int8 (132 MB vs 526 MB) with FER2013 calibration images. Input `[1, 3, 224, 224]` float32 in `[0, 1]`, output 7 logits. Powers the browser demo. |

### FER2013 benchmark checkpoints (`benchmarks/`)

Trained checkpoints for the FER2013 benchmark table in the [GitHub README](https://github.com/phamquiluan/ResidualMaskingNetwork#benchmarking-on-fer2013), mirrored from Google Drive so they can't be lost. Private test accuracy:

| File | Accuracy (%) |
|---|---|
| `benchmarks/vgg19` | 70.80 |
| `benchmarks/efficientnet_b2b` | 70.80 |
| `benchmarks/googlenet` | 71.97 |
| `benchmarks/resnet34` | 72.42 |
| `benchmarks/inception_v3` | 72.72 |
| `benchmarks/bam_resnet50` | 73.14 |
| `benchmarks/densenet121` | 73.16 |
| `benchmarks/resnet152` | 73.22 |
| `benchmarks/cbam_resnet50` | 73.39 |
| `benchmarks/resmasking_net` | 74.14 |

Note: two checkpoints of the 76.82% ensemble (`resnet50_pretrained_vgg_rot30_2019Nov13_08.20`, `resnet18_rot30_2019Nov05_17.44`) were permanently lost — see [issue #46](https://github.com/phamquiluan/ResidualMaskingNetwork/issues/46).

## Usage

The easiest way is through the `rmn` package:

```bash
pip install rmn
```

```python
import cv2
from rmn import RMN

m = RMN()
image = cv2.imread("some-image.png")
results = m.detect_emotion_for_single_frame(image)
print(results)
```

Or load the raw checkpoint directly:

```python
import torch
from huggingface_hub import hf_hub_download
from models import resmasking_dropout1  # from the GitHub repo

path = hf_hub_download(
    repo_id="phamquiluan/ResidualMaskingNetwork",
    filename="Z_resmasking_dropout1_rot30_2019Nov30_13.32",
)
model = resmasking_dropout1(in_channels=3, num_classes=7)
state = torch.load(path, map_location="cpu")
model.load_state_dict(state["net"])
model.eval()
```

Emotion labels (FER2013): `angry, disgust, fear, happy, sad, surprise, neutral`.

### ONNX

```python
import numpy as np, onnxruntime as ort
from huggingface_hub import hf_hub_download

path = hf_hub_download("phamquiluan/ResidualMaskingNetwork", "onnx/resmasking_int8.onnx")
session = ort.InferenceSession(path, providers=["CPUExecutionProvider"])

# face: a grayscale crop resized to 224x224, replicated to 3 channels
tensor = (np.stack([face] * 3)[None] / 255.0).astype(np.float32)
logits = session.run(None, {"input": tensor})[0][0]
```

## Citation

```bibtex
@inproceedings{pham2021facial,
  title={Facial expression recognition using residual masking network},
  author={Pham, Luan and Vu, The Huynh and Tran, Tuan Anh},
  booktitle={2020 25th International Conference on Pattern Recognition (ICPR)},
  pages={4513--4519},
  year={2021},
  organization={IEEE}
}
```