README.md

---
license: apache-2.0
pipeline_tag: keypoint-detection
library_name: mmpose
tags:
- pose-estimation
- hand-keypoints
- rtmpose
- simcc
- onnx
- sign-language
- self-distillation
---

# RTMPose-m Distill — blur-robust 2D hand keypoints for sign language video

RTMPose-m (21 hand keypoints, SimCC, 256×256) fine-tuned via **self-distillation on degraded video** — pseudo-labels produced by the model itself on clean frames, training inputs artificially degraded — to keep tracking hands through low resolution and motion blur, the main failure modes of off-the-shelf hand pose models on real-world sign language footage.

<p align="center">
  <img src="assets/output_pytorch.jpg" alt="21-keypoint hand skeleton correctly placed on a heavily motion-blurred hand" width="420"/>
  <br/>
  <em>Model output on a heavily motion-blurred frame: the skeleton stays on the fingers. PyTorch and ONNX Runtime outputs are byte-identical (<code>assets/output_pytorch.jpg</code> vs <code>assets/output_onnxruntime.jpg</code>).</em>
</p>

Compared to the base [RTMPose-m Hand5](https://github.com/open-mmlab/mmpose/tree/main/projects/rtmpose) checkpoint, this model:

- **retains hands under high confidence thresholds**: at thr 0.3 it keeps 98.0% of hand detections vs 93.1% for the base model, so you can raise the threshold to cut false positives without losing recall;
- **detects hands the base model misses** on hard frames (motion blur during fast signing, crossed/interlocked hands, hands pressed against the body): at thr 0.3 it fires on 2,672 frames (52% of a test video) where the base model returns nothing;
- **produces temporally smoother keypoints**: ~39% less frame-to-frame jitter at thr 0.3, which directly reduces ragged keypoint sequences fed into downstream sign language models (Uni-Sign, streaming/wait-k translation pipelines);
- **does not regress on clean frames** — on sharp, unoccluded frames the two models are visually indistinguishable.

Same architecture, same input size, same 21-keypoint COCO hand skeleton as the original — a **drop-in replacement** for the `rtmpose-m_simcc-hand5` checkpoint in any mmpose / rtmlib / mmdeploy pipeline.

## Files

| File | Description |
|---|---|
| `rtmpose-m_hand_distill-256x256-a996d9ec.pth` | PyTorch weights (EMA, epoch 100), mmpose format, 55 MB |
| `rtmpose-m_hand_distill.py` | mmpose/mmengine training and inference config |
| `degrade_video.py` | Video degradation script used to build the "dirty" half of the training set (opencv + numpy only) |
| `onnx/rtmpose-m-distill-256x256.onnx` | ONNX export (opset 11, dynamic batch, FP32), outputs `simcc_x`/`simcc_y` |
| `onnx/deploy.json`, `onnx/pipeline.json` | mmdeploy SDK configs for the ONNX model |
| `assets/` | PyTorch vs ONNX Runtime output parity check (byte-identical) |

## How it was trained

Self-distillation on degraded video — the model is its own teacher:

1. **Pseudo-labels.** The base RTMPose-m Hand5 checkpoint with a hand-crop pipeline was run offline over the original clean FullHD frames of the [Slovo](https://github.com/hukenovs/slovo) Russian Sign Language video dataset, producing hand crops with 21-keypoint pseudo-labels.
2. **Input degradation.** 50% of the source videos were then degraded (the "dirty" half) with the included `degrade_video.py`, targeting the dominant real-world failure mode — low source resolution: the full frame is downscaled so its short side lands around 300 px (randomized per clip), then resized back to the original size (`INTER_AREA` down, bilinear up), so teacher coordinates taken from the clean frames stay valid. The degradation toolkit also includes optical-flow-based motion blur (Farneback flow, accumulated along the flow field), gamma/lighting shift, Gaussian noise and JPEG compression, organized into severity profiles 1–5. Degradation is applied to the **full frame before hand cropping** (so crops don't retain more detail than a real low-res source would have), and per-clip seeding (`crc32(filename) + seed`) makes it fully reproducible. The student therefore learns to predict sharp-frame keypoints from corrupted inputs.
3. **Fine-tuning.** The student — the same RTMPose-m Hand5 checkpoint that produced the labels — was fine-tuned for 100 epochs on the resulting **handset_mix** set: 300,238 training crops, 33,347 validation crops (16,697 clean / 16,650 dirty).

Training setup: AdamW (lr 4e-4, wd 0.05), batch 1024, cosine schedule, AMP, EMA (ExpMomentumEMA, momentum 2e-4), flip/rotate/scale augmentation, seed 21. Single NVIDIA RTX PRO 6000 Blackwell GPU, PyTorch 2.7.0 / CUDA 12.8 / MMEngine 0.10.7, ~10 h wall-clock. Full details in `rtmpose-m_hand_distill.py`.

Held-out validation against pseudo-labels (mixed clean + dirty, 33,347 crops): the released checkpoint is the **EMA weights at epoch 100** — **PCK@0.2 (bbox-normalized) 0.9893, EPE 5.96 px**. Best raw validation score during training was PCK 0.9896 / EPE 5.87 at epoch 42; the validation curve is flat from roughly epoch 20 onward.

## Evaluation vs the base model

Side-by-side comparison on a sign language test video (~5,100 frames), hand retention relative to detections at thr 0.1:

| Confidence threshold | 0.1 | 0.15 | 0.2 | 0.3 |
|---|---|---|---|---|
| Hand retention, **base** | 100% | 98.8% | 97.3% | 93.1% |
| Hand retention, **this model** | 100% | **99.8%** | **99.4%** | **98.0%** |
| Frames where this model detects a hand and base does not | 431 (8%) | 875 (17%) | 1,338 (26%) | 2,672 (52%) |

Frame-to-frame keypoint jitter at thr 0.3 is ~39% lower than the base model. The frames recovered by this model are dominated by motion blur during fast signing, crossed/interlocked hands, and hands pressed against the torso; visual inspection confirms the recovered skeletons lie on the fingers rather than being spurious detections.

**Recommended operating point:** thr 0.2–0.3 (the base model effectively requires thr ≤ 0.15 to avoid dropping hands).

## Usage

### mmpose

```python
from mmpose.apis import init_model, inference_topdown

model = init_model(
    'rtmpose-m_hand_distill.py',
    'rtmpose-m_hand_distill-256x256-a996d9ec.pth',
    device='cuda:0',
)
results = inference_topdown(model, 'hand_crop.jpg')
keypoints = results[0].pred_instances.keypoints  # (1, 21, 2)
scores = results[0].pred_instances.keypoint_scores  # (1, 21)
```

### ONNX Runtime (no mmpose dependency)

```python
import cv2
import numpy as np
import onnxruntime as ort

sess = ort.InferenceSession('onnx/rtmpose-m-distill-256x256.onnx')

img = cv2.imread('hand_crop.jpg')  # BGR hand crop
inp = cv2.resize(img, (256, 256))[:, :, ::-1].astype(np.float32)  # to RGB
inp = (inp - [123.675, 116.28, 103.53]) / [58.395, 57.12, 57.375]
inp = inp.transpose(2, 0, 1)[None]

simcc_x, simcc_y = sess.run(None, {'input': inp.astype(np.float32)})
# SimCC decode: argmax over each axis, divide by split ratio (2.0)
x = simcc_x[0].argmax(axis=1) / 2.0  # (21,) in 256x256 crop coords
y = simcc_y[0].argmax(axis=1) / 2.0
conf = np.minimum(simcc_x[0].max(axis=1), simcc_y[0].max(axis=1))
```

The ONNX file is also compatible with [rtmlib](https://github.com/Tau-J/rtmlib) and the [mmdeploy](https://github.com/open-mmlab/mmdeploy) SDK (use `onnx/` as the SDK model directory).

## Limitations

- **Pseudo-label supervision.** Training targets are the base model's own predictions, not human annotations; systematic biases of RTMPose-m Hand5 are inherited rather than corrected. Validation PCK/EPE above are measured against pseudo-labels, not ground truth.
- **Comparative evaluation.** The improvement numbers compare this model against its own teacher on sign language video; the model has not been benchmarked on GT hand datasets (FreiHAND, COCO-WholeBody Hand).
- **Domain.** Tuned on Russian Sign Language studio-style recordings (frontal upper-body view, 194 signers). Behavior on in-the-wild hands (egocentric, object interaction, outdoor) is untested.
- Top-down model: expects a hand crop; you still need a hand/person detector upstream.

## Training data attribution

Pseudo-labels and training crops are derived from the [Slovo Russian Sign Language dataset](https://github.com/hukenovs/slovo) (SaluteDevices), distributed under a variant of CC BY-SA 4.0. The dataset itself is **not** included in this repository — only model weights.

## Citations

```bibtex
@misc{jiang2023rtmpose,
  title={RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose},
  author={Jiang, Tao and Lu, Peng and Zhang, Li and Ma, Ningsheng and Han, Rui and Lyu, Chengqi and Li, Yining and Chen, Kai},
  year={2023},
  eprint={2303.07399},
  archivePrefix={arXiv}
}

@inproceedings{kapitanov2023slovo,
  title={Slovo: Russian Sign Language Dataset},
  author={Kapitanov, Alexander and Kvanchiani, Karina and Nagaev, Alexander and Petrova, Elizaveta},
  booktitle={International Conference on Computer Vision Systems},
  pages={63--73},
  year={2023},
  organization={Springer}
}
```