Upload folder using huggingface_hub

README.md

@@ -0,0 +1,142 @@
+---
+license: apache-2.0
+pipeline_tag: keypoint-detection
+library_name: mmpose
+tags:
+- pose-estimation
+- hand-keypoints
+- rtmpose
+- simcc
+- onnx
+- sign-language
+- russian-sign-language
+- self-distillation
+---
+
+# RTMPose-m Hand — blur-robust 2D hand keypoints for sign language video
+
+RTMPose-m (21 hand keypoints, SimCC, 256×256) fine-tuned via **self-distillation on degraded video** — pseudo-labels produced by the model itself on clean frames, training inputs artificially degraded — to keep tracking hands through low resolution and motion blur, the main failure modes of off-the-shelf hand pose models on real-world sign language footage.
+
+<p align="center">
+  <img src="assets/output_pytorch.jpg" alt="21-keypoint hand skeleton correctly placed on a heavily motion-blurred hand" width="420"/>
+  <br/>
+  <em>Model output on a heavily motion-blurred frame: the skeleton stays on the fingers. PyTorch and ONNX Runtime outputs are byte-identical (<code>assets/output_pytorch.jpg</code> vs <code>assets/output_onnxruntime.jpg</code>).</em>
+</p>
+
+Compared to the base [RTMPose-m Hand5](https://github.com/open-mmlab/mmpose/tree/main/projects/rtmpose) checkpoint, this model:
+
+- **retains hands under high confidence thresholds**: at thr 0.3 it keeps 98.0% of hand detections vs 93.1% for the base model, so you can raise the threshold to cut false positives without losing recall;
+- **detects hands the base model misses** on hard frames (motion blur during fast signing, crossed/interlocked hands, hands pressed against the body): at thr 0.3 it fires on 2,672 frames (52% of a test video) where the base model returns nothing;
+- **produces temporally smoother keypoints**: ~39% less frame-to-frame jitter at thr 0.3, which directly reduces ragged keypoint sequences fed into downstream sign language models (Uni-Sign, streaming/wait-k translation pipelines);
+- **does not regress on clean frames** — on sharp, unoccluded frames the two models are visually indistinguishable.
+
+Same architecture, same input size, same 21-keypoint COCO hand skeleton as the original — a **drop-in replacement** for the `rtmpose-m_simcc-hand5` checkpoint in any mmpose / rtmlib / mmdeploy pipeline.
+
+## Files
+
+| File | Description |
+|---|---|
+| `rtmpose-m_hand_distill-256x256-a996d9ec.pth` | PyTorch weights (EMA, epoch 100), mmpose format, 55 MB |
+| `rtmpose-m_hand_distill.py` | mmpose/mmengine training and inference config |
+| `degrade_video.py` | Video degradation script used to build the "dirty" half of the training set (opencv + numpy only) |
+| `onnx/rtmpose-m-distill-256x256.onnx` | ONNX export (opset 11, dynamic batch, FP32), outputs `simcc_x`/`simcc_y` |
+| `onnx/deploy.json`, `onnx/pipeline.json` | mmdeploy SDK configs for the ONNX model |
+| `assets/` | PyTorch vs ONNX Runtime output parity check (byte-identical) |
+
+## How it was trained
+
+Self-distillation on degraded video — the model is its own teacher:
+
+1. **Pseudo-labels.** The base RTMPose-m Hand5 checkpoint with a hand-crop pipeline was run offline over the original clean FullHD frames of the [Slovo](https://github.com/hukenovs/slovo) Russian Sign Language video dataset, producing hand crops with 21-keypoint pseudo-labels.
+2. **Input degradation.** 50% of the source videos were then degraded (the "dirty" half) with the included `degrade_video.py`, targeting the dominant real-world failure mode — low source resolution: the full frame is downscaled so its short side lands around 300 px (randomized per clip), then resized back to the original size (`INTER_AREA` down, bilinear up), so teacher coordinates taken from the clean frames stay valid. The degradation toolkit also includes optical-flow-based motion blur (Farneback flow, accumulated along the flow field), gamma/lighting shift, Gaussian noise and JPEG compression, organized into severity profiles 1–5. Degradation is applied to the **full frame before hand cropping** (so crops don't retain more detail than a real low-res source would have), and per-clip seeding (`crc32(filename) + seed`) makes it fully reproducible. The student therefore learns to predict sharp-frame keypoints from corrupted inputs.
+3. **Fine-tuning.** The student — the same RTMPose-m Hand5 checkpoint that produced the labels — was fine-tuned for 100 epochs on the resulting **handset_mix** set: 300,238 training crops, 33,347 validation crops (16,697 clean / 16,650 dirty).
+
+Training setup: AdamW (lr 4e-4, wd 0.05), batch 1024, cosine schedule, AMP, EMA (ExpMomentumEMA, momentum 2e-4), flip/rotate/scale augmentation, seed 21. Single NVIDIA RTX PRO 6000 Blackwell GPU, PyTorch 2.7.0 / CUDA 12.8 / MMEngine 0.10.7, ~10 h wall-clock. Full details in `rtmpose-m_hand_distill.py`.
+
+Held-out validation against pseudo-labels (mixed clean + dirty, 33,347 crops): the released checkpoint is the **EMA weights at epoch 100** — **PCK@0.2 (bbox-normalized) 0.9893, EPE 5.96 px**. Best raw validation score during training was PCK 0.9896 / EPE 5.87 at epoch 42; the validation curve is flat from roughly epoch 20 onward.
+
+## Evaluation vs the base model
+
+Side-by-side comparison on a sign language test video (~5,100 frames), hand retention relative to detections at thr 0.1:
+
+| Confidence threshold | 0.1 | 0.15 | 0.2 | 0.3 |
+|---|---|---|---|---|
+| Hand retention, **base** | 100% | 98.8% | 97.3% | 93.1% |
+| Hand retention, **this model** | 100% | **99.8%** | **99.4%** | **98.0%** |
+| Frames where this model detects a hand and base does not | 431 (8%) | 875 (17%) | 1,338 (26%) | 2,672 (52%) |
+
+Frame-to-frame keypoint jitter at thr 0.3 is ~39% lower than the base model. The frames recovered by this model are dominated by motion blur during fast signing, crossed/interlocked hands, and hands pressed against the torso; visual inspection confirms the recovered skeletons lie on the fingers rather than being spurious detections.
+
+**Recommended operating point:** thr 0.2–0.3 (the base model effectively requires thr ≤ 0.15 to avoid dropping hands).
+
+## Usage
+
+### mmpose
+
+```python
+from mmpose.apis import init_model, inference_topdown
+
+model = init_model(
+    'rtmpose-m_hand_distill.py',
+    'rtmpose-m_hand_distill-256x256-a996d9ec.pth',
+    device='cuda:0',
+)
+results = inference_topdown(model, 'hand_crop.jpg')
+keypoints = results[0].pred_instances.keypoints  # (1, 21, 2)
+scores = results[0].pred_instances.keypoint_scores  # (1, 21)
+```
+
+### ONNX Runtime (no mmpose dependency)
+
+```python
+import cv2
+import numpy as np
+import onnxruntime as ort
+
+sess = ort.InferenceSession('onnx/rtmpose-m-distill-256x256.onnx')
+
+img = cv2.imread('hand_crop.jpg')  # BGR hand crop
+inp = cv2.resize(img, (256, 256))[:, :, ::-1].astype(np.float32)  # to RGB
+inp = (inp - [123.675, 116.28, 103.53]) / [58.395, 57.12, 57.375]
+inp = inp.transpose(2, 0, 1)[None]
+
+simcc_x, simcc_y = sess.run(None, {'input': inp.astype(np.float32)})
+# SimCC decode: argmax over each axis, divide by split ratio (2.0)
+x = simcc_x[0].argmax(axis=1) / 2.0  # (21,) in 256x256 crop coords
+y = simcc_y[0].argmax(axis=1) / 2.0
+conf = np.minimum(simcc_x[0].max(axis=1), simcc_y[0].max(axis=1))
+```
+
+The ONNX file is also compatible with [rtmlib](https://github.com/Tau-J/rtmlib) and the [mmdeploy](https://github.com/open-mmlab/mmdeploy) SDK (use `onnx/` as the SDK model directory).
+
+## Limitations
+
+- **Pseudo-label supervision.** Training targets are the base model's own predictions, not human annotations; systematic biases of RTMPose-m Hand5 are inherited rather than corrected. Validation PCK/EPE above are measured against pseudo-labels, not ground truth.
+- **Comparative evaluation.** The improvement numbers compare this model against its own teacher on sign language video; the model has not been benchmarked on GT hand datasets (FreiHAND, COCO-WholeBody Hand).
+- **Domain.** Tuned on Russian Sign Language studio-style recordings (frontal upper-body view, 194 signers). Behavior on in-the-wild hands (egocentric, object interaction, outdoor) is untested.
+- Top-down model: expects a hand crop; you still need a hand/person detector upstream.
+
+## Training data attribution
+
+Pseudo-labels and training crops are derived from the [Slovo Russian Sign Language dataset](https://github.com/hukenovs/slovo) (SaluteDevices), distributed under a variant of CC BY-SA 4.0. The dataset itself is **not** included in this repository — only model weights.
+
+## Citations
+
+```bibtex
+@misc{jiang2023rtmpose,
+  title={RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose},
+  author={Jiang, Tao and Lu, Peng and Zhang, Li and Ma, Ningsheng and Han, Rui and Lyu, Chengqi and Li, Yining and Chen, Kai},
+  year={2023},
+  eprint={2303.07399},
+  archivePrefix={arXiv}
+}
+
+@inproceedings{kapitanov2023slovo,
+  title={Slovo: Russian Sign Language Dataset},
+  author={Kapitanov, Alexander and Kvanchiani, Karina and Nagaev, Alexander and Petrova, Elizaveta},
+  booktitle={International Conference on Computer Vision Systems},
+  pages={63--73},
+  year={2023},
+  organization={Springer}
+}
+```

degrade_video.py

@@ -0,0 +1,340 @@
+import argparse
+import glob
+import os
+import zlib
+from concurrent.futures import ProcessPoolExecutor, as_completed
+import cv2
+import numpy as np
+from tqdm import tqdm
+
+def _log(msg):
+    try:
+        tqdm.write(msg)
+    except Exception:
+        print(msg)
+
+def _worker_init():
+    try:
+        cv2.setNumThreads(1)
+    except Exception:
+        pass
+
+VIDEO_EXT = (".mp4", ".avi", ".mov", ".mkv", ".webm", ".m4v")
+MIN_SHORT_SIDE = 32
+
+SEVERITY = {
+    1: dict(res=(0.80, 0.95), jpeg=(60, 80), gamma=(1.00, 1.25), noise=(2, 5),   blur=(0.3, 0.6)),
+    2: dict(res=(0.65, 0.85), jpeg=(45, 65), gamma=(1.10, 1.40), noise=(4, 9),   blur=(0.5, 0.9)),
+    3: dict(res=(0.50, 0.70), jpeg=(30, 50), gamma=(1.20, 1.60), noise=(7, 13),  blur=(0.7, 1.1)),
+    4: dict(res=(0.38, 0.55), jpeg=(20, 35), gamma=(1.40, 1.90), noise=(10, 18), blur=(0.9, 1.4)),
+    5: dict(res=(0.28, 0.42), jpeg=(12, 22), gamma=(1.60, 2.20), noise=(14, 24), blur=(1.1, 1.8)),
+}
+
+def make_profile(level, rng):
+    p = SEVERITY[level]
+    return {
+        "res":   float(rng.uniform(*p["res"])),
+        "jpeg":  int(rng.uniform(*p["jpeg"])),
+        "gamma": float(rng.uniform(*p["gamma"])),
+        "noise": float(rng.uniform(*p["noise"])),
+        "blur":  float(rng.uniform(*p["blur"])),
+    }
+
+def compute_flow(prev_gray, cur_gray, flow_max_side=480):
+    h, w = cur_gray.shape
+    s = min(1.0, flow_max_side / float(max(h, w)))
+    if s < 1.0:
+        pg = cv2.resize(prev_gray, (int(w * s), int(h * s)), interpolation=cv2.INTER_AREA)
+        cg = cv2.resize(cur_gray, (int(w * s), int(h * s)), interpolation=cv2.INTER_AREA)
+    else:
+        pg, cg = prev_gray, cur_gray
+    flow = cv2.calcOpticalFlowFarneback(pg, cg, None, 0.5, 3, 21, 3, 7, 1.5, 0)
+    if s < 1.0:
+        flow = cv2.resize(flow, (w, h), interpolation=cv2.INTER_LINEAR) / s
+    return cv2.GaussianBlur(flow, (0, 0), 2.0)
+
+def motion_blur_flow(frame, flow, strength, n_steps=7):
+    if flow is None or strength <= 0:
+        return frame
+    h, w = frame.shape[:2]
+    xx, yy = np.meshgrid(np.arange(w, dtype=np.float32), np.arange(h, dtype=np.float32))
+    acc = np.zeros_like(frame, dtype=np.float32)
+    for t in np.linspace(-0.5, 0.5, n_steps, dtype=np.float32):
+        mapx = xx + (t * strength) * flow[..., 0]
+        mapy = yy + (t * strength) * flow[..., 1]
+        acc += cv2.remap(frame, mapx, mapy, cv2.INTER_LINEAR, borderMode=cv2.BORDER_REFLECT)
+    return np.clip(acc / n_steps, 0, 255).astype(np.uint8)
+
+def downscale(frame, res_scale, keep_native=False):
+    h, w = frame.shape[:2]
+    if res_scale >= 0.999:
+        return frame
+    nh, nw = max(2, round(h * res_scale)), max(2, round(w * res_scale))
+    small = cv2.resize(frame, (nw, nh), interpolation=cv2.INTER_AREA)
+    if keep_native:
+        return small
+    return cv2.resize(small, (w, h), interpolation=cv2.INTER_LINEAR)
+
+def degrade_resolution(frame, target_short_side, rng=None, jitter=0):
+    h, w = frame.shape[:2]
+    tgt = int(target_short_side)
+    if jitter and rng is not None:
+        lo = max(MIN_SHORT_SIDE, tgt - jitter)
+        hi = tgt + jitter + 1
+        tgt = int(rng.integers(lo, max(lo + 1, hi)))
+    tgt = max(MIN_SHORT_SIDE, tgt)
+    short = min(h, w)
+    res = min(1.0, tgt / float(short))
+    return downscale(frame, res, keep_native=False)
+
+def linear_motion_blur(img, ksize, angle_deg):
+    ksize = max(3, int(ksize)) | 1
+    k = np.zeros((ksize, ksize), np.float32)
+    k[ksize // 2, :] = 1.0
+    c = (ksize - 1) * 0.5
+    M = cv2.getRotationMatrix2D((c, c), angle_deg, 1.0)
+    k = cv2.warpAffine(k, M, (ksize, ksize))
+    ssum = float(k.sum())
+    if ssum > 1e-6:
+        k /= ssum
+    return cv2.filter2D(img, -1, k)
+
+def random_motion_blur(img, rng, max_ksize):
+    if max_ksize is None or max_ksize < 3 or rng is None:
+        return img
+    ksize = int(rng.integers(1, int(max_ksize) + 1))
+    if ksize < 3:
+        return img
+    return linear_motion_blur(img, ksize, float(rng.uniform(0.0, 180.0)))
+
+def light_noise(frame, gamma, noise_sigma, rng):
+    if abs(gamma - 1.0) < 1e-3 and noise_sigma <= 0:
+        return frame
+    f = frame.astype(np.float32)
+    if abs(gamma - 1.0) > 1e-3:
+        f = np.power(f / 255.0, gamma) * 255.0
+    if noise_sigma > 0:
+        f = f + rng.normal(0.0, noise_sigma, f.shape).astype(np.float32)
+    return np.clip(f, 0, 255).astype(np.uint8)
+
+def jpeg_compress(frame, quality):
+    if quality >= 100:
+        return frame
+    ok, enc = cv2.imencode(".jpg", frame, [cv2.IMWRITE_JPEG_QUALITY, int(quality)])
+    return cv2.imdecode(enc, cv2.IMREAD_COLOR) if ok else frame
+
+def degrade_frame(frame, flow, prof, flags):
+    out = frame
+    if flags["blur"]:
+        out = motion_blur_flow(out, flow, prof["blur"])
+    if flags["noise"]:
+        out = light_noise(out, prof["gamma"], prof["noise"], flags["rng"])
+    if flags["res"]:
+        out = downscale(out, prof["res"], keep_native=flags["keep_native"])
+    if flags["jpeg"]:
+        out = jpeg_compress(out, prof["jpeg"])
+    return out
+
+def process_video(src, dst, args, flags, quiet=False):
+    cap = cv2.VideoCapture(src)
+    if not cap.isOpened():
+        _log(f"[err]: {src}")
+        return False
+    fps = cap.get(cv2.CAP_PROP_FPS)
+    fps = fps if fps and fps > 1 else 30.0
+    w = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
+    h = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
+    n_total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
+    
+    seed = (args.seed + zlib.crc32(os.path.basename(src).encode("utf-8"))) % (2**31)
+    rng = np.random.default_rng(seed)
+    flags["rng"] = rng
+
+    lvl = int(rng.integers(2, 6)) if args.severity == "random" else int(args.severity)
+    prof = make_profile(lvl, rng)
+
+    if args.target_short_side:
+        tgt = args.target_short_side
+        if args.target_jitter:
+            lo = max(MIN_SHORT_SIDE, args.target_short_side - args.target_jitter)
+            hi = args.target_short_side + args.target_jitter + 1
+            tgt = int(rng.integers(lo, max(lo + 1, hi)))
+        tgt = max(MIN_SHORT_SIDE, tgt)
+        short = min(h, w)
+        prof["res"] = min(1.0, tgt / float(short))
+        flags["res"] = True
+        flags["blur"] = flags["blur"] and not args.res_only
+        flags["noise"] = flags["noise"] and not args.res_only
+        flags["jpeg"] = (args.jpeg is not None) and flags["jpeg"]
+        if args.jpeg is not None:
+            prof["jpeg"] = args.jpeg
+        eff = int(round(short * prof["res"]))
+        mode_str = f"target_short={tgt} (eff~{eff}px) res_only={args.res_only} jpeg={args.jpeg}"
+    else:
+        if args.jpeg is not None:
+            prof["jpeg"] = args.jpeg
+        mode_str = (f"lvl={lvl} res={prof['res']:.2f} jpeg={prof['jpeg']} "
+                    f"gamma={prof['gamma']:.2f} noise={prof['noise']:.1f} blur={prof['blur']:.2f}")
+
+    if flags["keep_native"]:
+        nh, nw = max(2, round(h * prof["res"])), max(2, round(w * prof["res"]))
+        out_w, out_h = nw, nh
+    else:
+        out_w, out_h = (w * 2 if args.preview else w), h
+
+    tmp_dst = dst + ".part.mp4"
+    writer = cv2.VideoWriter(tmp_dst, cv2.VideoWriter_fourcc(*"mp4v"), fps, (out_w, out_h))
+    if not writer.isOpened():
+        _log(f"[err]: {dst}")
+        cap.release()
+        return False
+    if not quiet:
+        _log(f"[run] {os.path.basename(src)}  {mode_str}")
+
+    bar = tqdm(total=(n_total if n_total > 0 else None),
+               desc=os.path.basename(src)[:24], unit="f", leave=False, disable=quiet)
+    prev_gray, idx = None, 0
+    while True:
+        ok, frame = cap.read()
+        if not ok:
+            break
+        idx += 1
+        if bar is not None:
+            bar.update(1)
+        flow = None
+        if flags["blur"]:
+            cur_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
+            if prev_gray is not None:
+                flow = compute_flow(prev_gray, cur_gray)
+            prev_gray = cur_gray
+        dirty = degrade_frame(frame, flow, prof, flags)
+        if args.preview and not flags["keep_native"]:
+            dirty = np.hstack([frame, dirty])
+        writer.write(dirty)
+
+    if bar is not None:
+        bar.close()
+    cap.release()
+    writer.release()
+    if idx == 0:
+        _log(f"[err]: {src}")
+        try:
+            os.remove(tmp_dst)
+        except OSError:
+            pass
+        return False
+    try:
+        os.replace(tmp_dst, dst)
+    except OSError as e:
+        _log(f"[err] {tmp_dst} -> {dst}: {e}")
+        return False
+    if not quiet:
+        _log(f"[ok]  {os.path.basename(dst)}  ({idx}, {out_w}x{out_h})")
+    return True
+
+
+def main():
+    ap = argparse.ArgumentParser(description="Video degradation for self-distillation.")
+    ap.add_argument("--source", required=True, help="video file or directory")
+    ap.add_argument("--out", required=True, help="output file or directory")
+    ap.add_argument("--severity", default="3", help="1..5 or 'random' (profile mode)")
+    ap.add_argument("--target-short-side", type=int, default=0,
+                    help="mode B: absolute short side, px (e.g. 300)")
+    ap.add_argument("--target-jitter", type=int, default=0,
+                    help="random +/- to the short side per clip (e.g. 50)")
+    ap.add_argument("--res-only", action="store_true",
+                    help="mode B: resolution only, no noise/blur")
+    ap.add_argument("--jpeg", type=int, default=None,
+                    help="JPEG quality 1..100 (if set, applied on top)")
+    ap.add_argument("--keep-native", action="store_true",
+                    help="output at low resolution, do not upscale back")
+    ap.add_argument("--seed", type=int, default=0)
+    ap.add_argument("--workers", type=int, default=max(1, (os.cpu_count() or 2) // 2),
+                    help="parallel processes per video (default = half the cores)")
+    ap.add_argument("--overwrite", action="store_true",
+                    help="overwrite finished files (default skips them — resume)")
+    ap.add_argument("--preview", action="store_true")
+    ap.add_argument("--no-blur", action="store_true")
+    ap.add_argument("--no-res", action="store_true")
+    ap.add_argument("--no-noise", action="store_true")
+    ap.add_argument("--no-jpeg", action="store_true")
+    args = ap.parse_args()
+
+    if args.severity != "random" and not args.target_short_side:
+        if int(args.severity) not in SEVERITY:
+            raise SystemExit("[err] severity 1..5 or 'random'")
+    if args.preview and args.keep_native:
+        print("[warn] --preview is incompatible with --keep-native, preview disabled")
+        args.preview = False
+
+    base_flags = {
+        "blur":  not args.no_blur,
+        "res":   not args.no_res,
+        "noise": not args.no_noise,
+        "jpeg":  not args.no_jpeg,
+        "keep_native": args.keep_native,
+    }
+
+    if os.path.isdir(args.source):
+        os.makedirs(args.out, exist_ok=True)
+        files = sorted(f for f in glob.glob(os.path.join(args.source, "*"))
+                       if f.lower().endswith(VIDEO_EXT))
+        if not files:
+            raise SystemExit(f"[err] no videos in {args.source}")
+
+        def dst_of(f):
+            return os.path.join(args.out, os.path.splitext(os.path.basename(f))[0] + "_dirty.mp4")
+
+        todo, skipped = [], 0
+        for f in files:
+            d = dst_of(f)
+            if not args.overwrite and os.path.exists(d):     # resume
+                skipped += 1
+            else:
+                todo.append((f, d))
+        print(f"[info] found {len(files)} videos, to process {len(todo)}, "
+              f"skipped (already done) {skipped}")
+        if not todo:
+            return
+
+        workers = max(1, args.workers)
+        if workers == 1:
+            for f, d in tqdm(todo, desc="videos", unit="vid"):
+                process_video(f, d, args, dict(base_flags), quiet=False)
+        else:
+            print(f"[info] parallel processes: {workers}")
+            # in parallel mode each video has no per-frame bar (quiet),
+            # instead we show overall progress across files
+            with ProcessPoolExecutor(max_workers=workers, initializer=_worker_init) as ex:
+                futs = {ex.submit(process_video, f, d, args, dict(base_flags), True): f
+                        for f, d in todo}
+                ok = fail = 0
+                bar = tqdm(total=len(futs), desc="videos", unit="vid")
+                for fut in as_completed(futs):
+                    try:
+                        ok += 1 if fut.result() else 0
+                        fail += 0 if fut.result() else 1
+                    except Exception as e:
+                        fail += 1
+                        _log(f"[err] {os.path.basename(futs[fut])}: {type(e).__name__}: {e}")
+                    if bar is not None:
+                        bar.update(1)
+                        bar.set_postfix(ok=ok, fail=fail)
+                if bar is not None:
+                    bar.close()
+            print(f"[done] succeeded {ok}, failed {fail}")
+    else:
+        out = args.out
+        if os.path.isdir(out) or out.endswith(os.sep):
+            os.makedirs(out, exist_ok=True)
+            out = os.path.join(out, os.path.splitext(os.path.basename(args.source))[0] + "_dirty.mp4")
+        elif os.path.dirname(out):
+            os.makedirs(os.path.dirname(out), exist_ok=True)
+        if not args.overwrite and os.path.exists(out):
+            print(f"[info] already exists, skipping: {out} (--overwrite to recompute)")
+            return
+        process_video(args.source, out, args, dict(base_flags), quiet=False)
+
+
+if __name__ == "__main__":
+    main()

onnx/deploy.json

@@ -0,0 +1,16 @@
+{
+    "version": "1.3.1",
+    "task": "PoseDetector",
+    "models": [
+        {
+            "name": "topdownposeestimator",
+            "net": "rtmpose-m-distill-256x256.onnx",
+            "weights": "",
+            "backend": "onnxruntime",
+            "precision": "FP32",
+            "batch_size": 1,
+            "dynamic_shape": true
+        }
+    ],
+    "customs": []
+}

onnx/pipeline.json

@@ -0,0 +1,125 @@
+{
+    "pipeline": {
+        "input": [
+            "img"
+        ],
+        "output": [
+            "post_output"
+        ],
+        "tasks": [
+            {
+                "type": "Task",
+                "module": "Transform",
+                "name": "Preprocess",
+                "input": [
+                    "img"
+                ],
+                "output": [
+                    "prep_output"
+                ],
+                "transforms": [
+                    {
+                        "type": "LoadImageFromFile"
+                    },
+                    {
+                        "type": "TopDownGetBboxCenterScale",
+                        "padding": 1.25,
+                        "image_size": [
+                            192,
+                            256
+                        ]
+                    },
+                    {
+                        "type": "TopDownAffine",
+                        "image_size": [
+                            192,
+                            256
+                        ]
+                    },
+                    {
+                        "type": "Normalize",
+                        "mean": [
+                            123.675,
+                            116.28,
+                            103.53
+                        ],
+                        "std": [
+                            58.395,
+                            57.12,
+                            57.375
+                        ],
+                        "to_rgb": true
+                    },
+                    {
+                        "type": "ImageToTensor",
+                        "keys": [
+                            "img"
+                        ]
+                    },
+                    {
+                        "type": "Collect",
+                        "keys": [
+                            "img"
+                        ],
+                        "meta_keys": [
+                            "img_shape",
+                            "pad_shape",
+                            "ori_shape",
+                            "img_norm_cfg",
+                            "scale_factor",
+                            "bbox_score",
+                            "center",
+                            "scale"
+                        ]
+                    }
+                ]
+            },
+            {
+                "name": "topdownposeestimator",
+                "type": "Task",
+                "module": "Net",
+                "is_batched": true,
+                "input": [
+                    "prep_output"
+                ],
+                "output": [
+                    "infer_output"
+                ],
+                "input_map": {
+                    "img": "input"
+                },
+                "output_map": {}
+            },
+            {
+                "type": "Task",
+                "module": "mmpose",
+                "name": "postprocess",
+                "component": "SimCCLabelDecode",
+                "params": {
+                    "flip_test": false,
+                    "type": "SimCCLabel",
+                    "input_size": [
+                        256,
+                        256
+                    ],
+                    "sigma": [
+                        5.66,
+                        5.66
+                    ],
+                    "simcc_split_ratio": 2.0,
+                    "normalize": false,
+                    "use_dark": false,
+                    "_scope_": "mmpose",
+                    "export_postprocess": false
+                },
+                "output": [
+                    "post_output"
+                ],
+                "input": [
+                    "prep_output",
+                    "infer_output"
+                ]
+            }
+        ]
+    }
+}

onnx/rtmpose-m-distill-256x256.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6d50664e566fffee41a090c98f75e893b50846a753b802dbf5e2072a8dfd7784
+size 55118513

rtmpose-m_hand_distill-256x256-a996d9ec.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a996d9ec1807be4bfa7444a6a99cf833bfcdf3096035536fccc2093742833db1
+size 55292443

rtmpose-m_hand_distill.py

@@ -0,0 +1,677 @@
+auto_scale_lr = dict(base_batch_size=256)
+backend_args = dict(backend='local')
+base_lr = 0.0004
+codec = dict(
+    _scope_='mmpose',
+    input_size=(
+        256,
+        256,
+    ),
+    normalize=False,
+    sigma=(
+        5.66,
+        5.66,
+    ),
+    simcc_split_ratio=2.0,
+    type='SimCCLabel',
+    use_dark=False)
+custom_hooks = [
+    dict(
+        _scope_='mmpose',
+        ema_type='ExpMomentumEMA',
+        momentum=0.0002,
+        priority=49,
+        type='EMAHook',
+        update_buffers=True),
+    dict(
+        _scope_='mmpose',
+        switch_epoch=200,
+        switch_pipeline=[
+            dict(backend_args=dict(backend='local'), type='LoadImage'),
+            dict(type='GetBBoxCenterScale'),
+            dict(
+                rotate_factor=180,
+                scale_factor=[
+                    0.75,
+                    1.25,
+                ],
+                shift_factor=0.0,
+                type='RandomBBoxTransform'),
+            dict(direction='horizontal', type='RandomFlip'),
+            dict(input_size=(
+                256,
+                256,
+            ), type='TopdownAffine'),
+            dict(type='mmdet.YOLOXHSVRandomAug'),
+            dict(
+                transforms=[
+                    dict(p=0.2, type='Blur'),
+                    dict(p=0.2, type='MedianBlur'),
+                    dict(
+                        max_height=0.4,
+                        max_holes=1,
+                        max_width=0.4,
+                        min_height=0.2,
+                        min_holes=1,
+                        min_width=0.2,
+                        p=0.5,
+                        type='CoarseDropout'),
+                ],
+                type='Albumentation'),
+            dict(
+                encoder=dict(
+                    input_size=(
+                        256,
+                        256,
+                    ),
+                    normalize=False,
+                    sigma=(
+                        5.66,
+                        5.66,
+                    ),
+                    simcc_split_ratio=2.0,
+                    type='SimCCLabel',
+                    use_dark=False),
+                type='GenerateTarget'),
+            dict(type='PackPoseInputs'),
+        ],
+        type='mmdet.PipelineSwitchHook'),
+]
+data_mode = 'topdown'
+data_root = 'data/handset_mix/'
+dataset_coco = dict(
+    _scope_='mmpose',
+    ann_file='coco/annotations/coco_wholebody_train_v1.0.json',
+    data_mode='topdown',
+    data_prefix=dict(img='detection/coco/train2017/'),
+    data_root='data/',
+    pipeline=[],
+    type='CocoWholeBodyHandDataset')
+dataset_freihand = dict(
+    _scope_='mmpose',
+    ann_file='freihand/annotations/freihand_train.json',
+    data_mode='topdown',
+    data_prefix=dict(img='pose/FreiHand/'),
+    data_root='data/',
+    pipeline=[],
+    type='FreiHandDataset')
+dataset_halpehand = dict(
+    _scope_='mmpose',
+    ann_file='halpe/annotations/halpe_train_v1.json',
+    data_mode='topdown',
+    data_prefix=dict(img='pose/Halpe/hico_20160224_det/images/train2015/'),
+    data_root='data/',
+    pipeline=[],
+    type='HalpeHandDataset')
+dataset_onehand10k = dict(
+    _scope_='mmpose',
+    ann_file='onehand10k/annotations/onehand10k_train.json',
+    data_mode='topdown',
+    data_prefix=dict(img='pose/OneHand10K/'),
+    data_root='data/',
+    pipeline=[],
+    type='OneHand10KDataset')
+dataset_rhd = dict(
+    _scope_='mmpose',
+    ann_file='rhd/annotations/rhd_train.json',
+    data_mode='topdown',
+    data_prefix=dict(img='pose/RHD/'),
+    data_root='data/',
+    pipeline=[
+        dict(
+            mapping=[
+                (
+                    0,
+                    0,
+                ),
+                (
+                    1,
+                    4,
+                ),
+                (
+                    2,
+                    3,
+                ),
+                (
+                    3,
+                    2,
+                ),
+                (
+                    4,
+                    1,
+                ),
+                (
+                    5,
+                    8,
+                ),
+                (
+                    6,
+                    7,
+                ),
+                (
+                    7,
+                    6,
+                ),
+                (
+                    8,
+                    5,
+                ),
+                (
+                    9,
+                    12,
+                ),
+                (
+                    10,
+                    11,
+                ),
+                (
+                    11,
+                    10,
+                ),
+                (
+                    12,
+                    9,
+                ),
+                (
+                    13,
+                    16,
+                ),
+                (
+                    14,
+                    15,
+                ),
+                (
+                    15,
+                    14,
+                ),
+                (
+                    16,
+                    13,
+                ),
+                (
+                    17,
+                    20,
+                ),
+                (
+                    18,
+                    19,
+                ),
+                (
+                    19,
+                    18,
+                ),
+                (
+                    20,
+                    17,
+                ),
+            ],
+            num_keypoints=21,
+            type='KeypointConverter'),
+    ],
+    type='Rhd2DDataset')
+dataset_type = 'OneHand10KDataset'
+default_hooks = dict(
+    badcase=dict(
+        _scope_='mmpose',
+        badcase_thr=5,
+        enable=False,
+        metric_type='loss',
+        out_dir='badcase',
+        type='BadCaseAnalysisHook'),
+    checkpoint=dict(
+        _scope_='mmpose',
+        interval=2,
+        max_keep_ckpts=2,
+        rule='greater',
+        save_best='PCK',
+        type='CheckpointHook'),
+    logger=dict(_scope_='mmpose', interval=50, type='LoggerHook'),
+    param_scheduler=dict(_scope_='mmpose', type='ParamSchedulerHook'),
+    sampler_seed=dict(_scope_='mmpose', type='DistSamplerSeedHook'),
+    timer=dict(_scope_='mmpose', type='IterTimerHook'),
+    visualization=dict(
+        _scope_='mmpose', enable=False, type='PoseVisualizationHook'))
+default_scope = 'mmpose'
+env_cfg = dict(
+    cudnn_benchmark=True,
+    dist_cfg=dict(backend='nccl'),
+    mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0))
+launcher = 'none'
+load_from = 'https://download.openmmlab.com/mmpose/v1/projects/rtmpose/rtmpose-m_simcc-hand5_pt-aic-coco_210e-256x256-74fb594_20230320.pth'
+log_level = 'INFO'
+log_processor = dict(
+    _scope_='mmpose',
+    by_epoch=True,
+    num_digits=6,
+    type='LogProcessor',
+    window_size=50)
+max_epochs = 100
+model = dict(
+    _scope_='mmpose',
+    backbone=dict(
+        _scope_='mmdet',
+        act_cfg=dict(type='SiLU'),
+        arch='P5',
+        channel_attention=True,
+        deepen_factor=0.67,
+        expand_ratio=0.5,
+        init_cfg=dict(
+            checkpoint=
+            'https://download.openmmlab.com/mmpose/v1/projects/rtmpose/cspnext-m_udp-aic-coco_210e-256x192-f2f7d6f6_20230130.pth',
+            prefix='backbone.',
+            type='Pretrained'),
+        norm_cfg=dict(type='SyncBN'),
+        out_indices=(4, ),
+        type='CSPNeXt',
+        widen_factor=0.75),
+    data_preprocessor=dict(
+        bgr_to_rgb=True,
+        mean=[
+            123.675,
+            116.28,
+            103.53,
+        ],
+        std=[
+            58.395,
+            57.12,
+            57.375,
+        ],
+        type='PoseDataPreprocessor'),
+    head=dict(
+        decoder=dict(
+            input_size=(
+                256,
+                256,
+            ),
+            normalize=False,
+            sigma=(
+                5.66,
+                5.66,
+            ),
+            simcc_split_ratio=2.0,
+            type='SimCCLabel',
+            use_dark=False),
+        final_layer_kernel_size=7,
+        gau_cfg=dict(
+            act_fn='SiLU',
+            drop_path=0.0,
+            dropout_rate=0.0,
+            expansion_factor=2,
+            hidden_dims=256,
+            pos_enc=False,
+            s=128,
+            use_rel_bias=False),
+        in_channels=768,
+        in_featuremap_size=(
+            8,
+            8,
+        ),
+        input_size=(
+            256,
+            256,
+        ),
+        loss=dict(
+            beta=10.0,
+            label_softmax=True,
+            type='KLDiscretLoss',
+            use_target_weight=True),
+        out_channels=21,
+        simcc_split_ratio=2.0,
+        type='RTMCCHead'),
+    test_cfg=dict(flip_test=True),
+    type='TopdownPoseEstimator')
+optim_wrapper = dict(
+    _scope_='mmpose',
+    optimizer=dict(lr=0.0004, type='AdamW', weight_decay=0.05),
+    paramwise_cfg=dict(
+        bias_decay_mult=0, bypass_duplicate=True, norm_decay_mult=0),
+    type='OptimWrapper')
+param_scheduler = [
+    dict(
+        begin=0, by_epoch=False, end=200, start_factor=0.001, type='LinearLR'),
+    dict(
+        begin=50,
+        by_epoch=True,
+        convert_to_iter_based=True,
+        end=100,
+        eta_min=2e-05,
+        type='CosineAnnealingLR'),
+]
+randomness = dict(seed=21)
+resume = False
+stage2_num_epochs = 10
+test_cfg = dict()
+test_dataloader = dict(
+    batch_size=32,
+    dataset=dict(
+        ann_file='ann_val_clean.json',
+        data_prefix=dict(img=''),
+        data_root='data/handset_mix/',
+        pipeline=[
+            dict(backend_args=dict(backend='local'), type='LoadImage'),
+            dict(type='GetBBoxCenterScale'),
+            dict(input_size=(
+                256,
+                256,
+            ), type='TopdownAffine'),
+            dict(type='PackPoseInputs'),
+        ],
+        test_mode=True,
+        type='OneHand10KDataset'),
+    drop_last=False,
+    num_workers=10,
+    persistent_workers=True,
+    sampler=dict(
+        _scope_='mmpose', round_up=False, shuffle=False,
+        type='DefaultSampler'))
+test_evaluator = [
+    dict(norm_item='bbox', thr=0.2, type='PCKAccuracy'),
+    dict(type='EPE'),
+]
+train_cfg = dict(by_epoch=True, max_epochs=100, val_interval=2)
+train_dataloader = dict(
+    batch_size=1024,
+    dataset=dict(
+        ann_file='ann_train.json',
+        data_prefix=dict(img=''),
+        data_root='data/handset_mix/',
+        pipeline=[
+            dict(backend_args=dict(backend='local'), type='LoadImage'),
+            dict(type='GetBBoxCenterScale'),
+            dict(direction='horizontal', type='RandomFlip'),
+            dict(
+                rotate_factor=180,
+                scale_factor=[
+                    0.5,
+                    1.5,
+                ],
+                type='RandomBBoxTransform'),
+            dict(input_size=(
+                256,
+                256,
+            ), type='TopdownAffine'),
+            dict(
+                encoder=dict(
+                    input_size=(
+                        256,
+                        256,
+                    ),
+                    normalize=False,
+                    sigma=(
+                        5.66,
+                        5.66,
+                    ),
+                    simcc_split_ratio=2.0,
+                    type='SimCCLabel',
+                    use_dark=False),
+                type='GenerateTarget'),
+            dict(type='PackPoseInputs'),
+        ],
+        type='OneHand10KDataset'),
+    num_workers=24,
+    persistent_workers=True,
+    sampler=dict(_scope_='mmpose', shuffle=True, type='DefaultSampler'))
+train_pipeline = [
+    dict(backend_args=dict(backend='local'), type='LoadImage'),
+    dict(type='GetBBoxCenterScale'),
+    dict(direction='horizontal', type='RandomFlip'),
+    dict(
+        rotate_factor=180,
+        scale_factor=[
+            0.5,
+            1.5,
+        ],
+        type='RandomBBoxTransform'),
+    dict(input_size=(
+        256,
+        256,
+    ), type='TopdownAffine'),
+    dict(
+        encoder=dict(
+            input_size=(
+                256,
+                256,
+            ),
+            normalize=False,
+            sigma=(
+                5.66,
+                5.66,
+            ),
+            simcc_split_ratio=2.0,
+            type='SimCCLabel',
+            use_dark=False),
+        type='GenerateTarget'),
+    dict(type='PackPoseInputs'),
+]
+train_pipeline_stage2 = [
+    dict(
+        _scope_='mmpose', backend_args=dict(backend='local'),
+        type='LoadImage'),
+    dict(_scope_='mmpose', type='GetBBoxCenterScale'),
+    dict(
+        _scope_='mmpose',
+        rotate_factor=180,
+        scale_factor=[
+            0.75,
+            1.25,
+        ],
+        shift_factor=0.0,
+        type='RandomBBoxTransform'),
+    dict(_scope_='mmpose', direction='horizontal', type='RandomFlip'),
+    dict(_scope_='mmpose', input_size=(
+        256,
+        256,
+    ), type='TopdownAffine'),
+    dict(_scope_='mmpose', type='mmdet.YOLOXHSVRandomAug'),
+    dict(
+        _scope_='mmpose',
+        transforms=[
+            dict(p=0.2, type='Blur'),
+            dict(p=0.2, type='MedianBlur'),
+            dict(
+                max_height=0.4,
+                max_holes=1,
+                max_width=0.4,
+                min_height=0.2,
+                min_holes=1,
+                min_width=0.2,
+                p=0.5,
+                type='CoarseDropout'),
+        ],
+        type='Albumentation'),
+    dict(
+        _scope_='mmpose',
+        encoder=dict(
+            input_size=(
+                256,
+                256,
+            ),
+            normalize=False,
+            sigma=(
+                5.66,
+                5.66,
+            ),
+            simcc_split_ratio=2.0,
+            type='SimCCLabel',
+            use_dark=False),
+        type='GenerateTarget'),
+    dict(_scope_='mmpose', type='PackPoseInputs'),
+]
+val_cfg = dict()
+val_coco = dict(
+    _scope_='mmpose',
+    ann_file='coco/annotations/coco_wholebody_val_v1.0.json',
+    data_mode='topdown',
+    data_prefix=dict(img='detection/coco/val2017/'),
+    data_root='data/',
+    pipeline=[],
+    type='CocoWholeBodyHandDataset')
+val_dataloader = dict(
+    batch_size=32,
+    dataset=dict(
+        ann_file='ann_val.json',
+        data_prefix=dict(img=''),
+        data_root='data/handset_mix/',
+        pipeline=[
+            dict(backend_args=dict(backend='local'), type='LoadImage'),
+            dict(type='GetBBoxCenterScale'),
+            dict(input_size=(
+                256,
+                256,
+            ), type='TopdownAffine'),
+            dict(type='PackPoseInputs'),
+        ],
+        test_mode=True,
+        type='OneHand10KDataset'),
+    drop_last=False,
+    num_workers=10,
+    persistent_workers=True,
+    sampler=dict(
+        _scope_='mmpose', round_up=False, shuffle=False,
+        type='DefaultSampler'))
+val_evaluator = [
+    dict(norm_item='bbox', thr=0.2, type='PCKAccuracy'),
+    dict(type='EPE'),
+]
+val_freihand = dict(
+    _scope_='mmpose',
+    ann_file='freihand/annotations/freihand_test.json',
+    data_mode='topdown',
+    data_prefix=dict(img='pose/FreiHand/'),
+    data_root='data/',
+    pipeline=[],
+    type='FreiHandDataset')
+val_halpehand = dict(
+    _scope_='mmpose',
+    ann_file='halpe/annotations/halpe_val_v1.json',
+    data_mode='topdown',
+    data_prefix=dict(img='detection/coco/val2017/'),
+    data_root='data/',
+    pipeline=[],
+    type='HalpeHandDataset')
+val_onehand10k = dict(
+    _scope_='mmpose',
+    ann_file='onehand10k/annotations/onehand10k_test.json',
+    data_mode='topdown',
+    data_prefix=dict(img='pose/OneHand10K/'),
+    data_root='data/',
+    pipeline=[],
+    type='OneHand10KDataset')
+val_pipeline = [
+    dict(backend_args=dict(backend='local'), type='LoadImage'),
+    dict(type='GetBBoxCenterScale'),
+    dict(input_size=(
+        256,
+        256,
+    ), type='TopdownAffine'),
+    dict(type='PackPoseInputs'),
+]
+val_rhd = dict(
+    _scope_='mmpose',
+    ann_file='rhd/annotations/rhd_test.json',
+    data_mode='topdown',
+    data_prefix=dict(img='pose/RHD/'),
+    data_root='data/',
+    pipeline=[
+        dict(
+            mapping=[
+                (
+                    0,
+                    0,
+                ),
+                (
+                    1,
+                    4,
+                ),
+                (
+                    2,
+                    3,
+                ),
+                (
+                    3,
+                    2,
+                ),
+                (
+                    4,
+                    1,
+                ),
+                (
+                    5,
+                    8,
+                ),
+                (
+                    6,
+                    7,
+                ),
+                (
+                    7,
+                    6,
+                ),
+                (
+                    8,
+                    5,
+                ),
+                (
+                    9,
+                    12,
+                ),
+                (
+                    10,
+                    11,
+                ),
+                (
+                    11,
+                    10,
+                ),
+                (
+                    12,
+                    9,
+                ),
+                (
+                    13,
+                    16,
+                ),
+                (
+                    14,
+                    15,
+                ),
+                (
+                    15,
+                    14,
+                ),
+                (
+                    16,
+                    13,
+                ),
+                (
+                    17,
+                    20,
+                ),
+                (
+                    18,
+                    19,
+                ),
+                (
+                    19,
+                    18,
+                ),
+                (
+                    20,
+                    17,
+                ),
+            ],
+            num_keypoints=21,
+            type='KeypointConverter'),
+    ],
+    type='Rhd2DDataset')
+vis_backends = [
+    dict(_scope_='mmpose', type='LocalVisBackend'),
+]
+visualizer = dict(
+    _scope_='mmpose',
+    name='visualizer',
+    type='PoseLocalVisualizer',
+    vis_backends=[
+        dict(type='LocalVisBackend'),
+    ])
+work_dir = './work_dirs/rtmpose-m_hand_distill'