Link browser demo and document the int8 ONNX export

c33839f verified about 22 hours ago

4.02 kB

license: mit
pipeline_tag: image-classification
tags:
  - facial-expression-recognition
  - emotion-recognition
  - fer2013
  - pytorch
  - onnx
library_name: pytorch

Residual Masking Network (RMN) — Facial Expression Recognition

Official checkpoint for "Facial Expression Recognition using Residual Masking Network" (ICPR 2020).

📄 Paper: Hugging Face Papers · arXiv · IEEE Xplore
💻 Code: github.com/phamquiluan/ResidualMaskingNetwork
🎮 Live demo: Try it in your browser — runs locally via ONNX Runtime Web, no upload
🏆 State-of-the-art single-model accuracy on FER2013: 74.14% (76.82% with ensemble)

Files

File	Description
`Z_resmasking_dropout1_rot30_2019Nov30_13.32`	Training checkpoint of the `resmasking_dropout1` architecture used by the `rmn` pip package. Model weights are stored under the `"net"` key.
`face_detection_yunet_2023mar.onnx`	YuNet face detector used by the `rmn` package pipeline.
`onnx/resmasking_int8.onnx`	ONNX export of the same checkpoint, statically quantised to int8 (132 MB vs 526 MB) with FER2013 calibration images. Input `[1, 3, 224, 224]` float32 in `[0, 1]`, output 7 logits. Powers the browser demo.

FER2013 benchmark checkpoints (`benchmarks/`)

Trained checkpoints for the FER2013 benchmark table in the GitHub README, mirrored from Google Drive so they can't be lost. Private test accuracy:

File	Accuracy (%)
`benchmarks/vgg19`	70.80
`benchmarks/efficientnet_b2b`	70.80
`benchmarks/googlenet`	71.97
`benchmarks/resnet34`	72.42
`benchmarks/inception_v3`	72.72
`benchmarks/bam_resnet50`	73.14
`benchmarks/densenet121`	73.16
`benchmarks/resnet152`	73.22
`benchmarks/cbam_resnet50`	73.39
`benchmarks/resmasking_net`	74.14

Note: two checkpoints of the 76.82% ensemble (resnet50_pretrained_vgg_rot30_2019Nov13_08.20, resnet18_rot30_2019Nov05_17.44) were permanently lost — see issue #46.

Usage

The easiest way is through the rmn package:

pip install rmn

import cv2
from rmn import RMN

m = RMN()
image = cv2.imread("some-image.png")
results = m.detect_emotion_for_single_frame(image)
print(results)

Or load the raw checkpoint directly:

import torch
from huggingface_hub import hf_hub_download
from models import resmasking_dropout1  # from the GitHub repo

path = hf_hub_download(
    repo_id="phamquiluan/ResidualMaskingNetwork",
    filename="Z_resmasking_dropout1_rot30_2019Nov30_13.32",
)
model = resmasking_dropout1(in_channels=3, num_classes=7)
state = torch.load(path, map_location="cpu")
model.load_state_dict(state["net"])
model.eval()

Emotion labels (FER2013): angry, disgust, fear, happy, sad, surprise, neutral.

ONNX

import numpy as np, onnxruntime as ort
from huggingface_hub import hf_hub_download

path = hf_hub_download("phamquiluan/ResidualMaskingNetwork", "onnx/resmasking_int8.onnx")
session = ort.InferenceSession(path, providers=["CPUExecutionProvider"])

# face: a grayscale crop resized to 224x224, replicated to 3 channels
tensor = (np.stack([face] * 3)[None] / 255.0).astype(np.float32)
logits = session.run(None, {"input": tensor})[0][0]

Citation

@inproceedings{pham2021facial,
  title={Facial expression recognition using residual masking network},
  author={Pham, Luan and Vu, The Huynh and Tran, Tuan Anh},
  booktitle={2020 25th International Conference on Pattern Recognition (ICPR)},
  pages={4513--4519},
  year={2021},
  organization={IEEE}
}