Mercity/figure-skating-cnn-bilstm

Figure Skating CNN-BiLSTM Action Classifier

A CNN-BiLSTM baseline for classifying figure-skating actions (jumps, spins, sequences) from pose-skeleton time series. Conv1D residual backbone for local temporal feature extraction, followed by a bidirectional LSTM for long-range temporal modeling.

This repo holds two checkpoints trained on the same architecture and data split, differing only in label granularity:

file	label space	classes
model_bilstm_fine.pt	fine-grained	28 (jump rotation-count preserved, e.g. 3Lutz, 2Axel)
model_bilstm_coarse.pt	coarse	11 (jump rotation-count collapsed, e.g. Lutz, Axel)

Not transformers-compatible — this is a plain PyTorch nn.Module + state_dict checkpoint. Model source code (SkatingBiLSTMClassifier in the project's model_bilstm.py, which depends on model.py) is not included in this repo — you'll need the project source to reconstruct the class before loading the state dict.

How to load

from huggingface_hub import hf_hub_download
import torch

repo_id = "Mercity/figure-skating-cnn-bilstm"
ckpt_path = hf_hub_download(repo_id, "model_bilstm_fine.pt")
ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False)

# reconstruct with the project's SkatingBiLSTMClassifier (model_bilstm.py):
# model = SkatingBiLSTMClassifier(ckpt["in_features"], ckpt["num_classes"])
# model.load_state_dict(ckpt["state_dict"])
# model.eval()

# ckpt also carries the standardization stats used at train time and the taxonomy:
mu, sd = ckpt["feature_mean"], ckpt["feature_std"]     # normalize inputs with (x - mu) / sd
taxonomy = ckpt["taxonomy"]                            # {0: "3Toeloop", 1: "3Loop", ...}

Swap model_bilstm_fine.pt for model_bilstm_coarse.pt to load the 11-class variant.

Config

Backbone	Conv1D stem(128) → ConvBlock(192,k3) → ConvBlock(256,k3) → ConvBlock(384,k5)
Temporal block	2-layer BiLSTM, hidden=192/direction (→384), dropout 0.2
Head	Dense(384→1024,p0.5) → Dense(1024→512,p0.4) → Dense(512→256,p0.3) → LayerNorm(256) → Linear(256→num_classes)
Optimizer	Adam, lr=5e-4, weight_decay=1e-4
Batch size	64
Grad clip	1.0
Class weight cap	5.0 (balanced, clipped)
Early stop	patience 20 on val macro-F1, 100-epoch cap
Seed	42
Input	(T=128, F=94) per-clip skeleton feature tensor (COCO-17 joints → positions/velocities/angles/angular-velocities/bone-vectors)
Params	3.77M (fine) / 3.76M (coarse)

Data

Trained on skeleton-extracted figure-skating clips (YOLO11n-pose, COCO-17 joints, imgsz=384) from an internal Mercity dataset. Stratified train/val/test split, seed 42. Not included in this repo (private internal dataset).

Results (test split)

Fine-grained (28-class)

Stopped epoch 51 (best @ ep31, val macro-F1 0.638)

Overall: accuracy 0.8689 · precision (macro/weighted) 0.5072 / 0.8800 · recall 0.5179 / 0.8689 · F1 0.5047 / 0.8697

class	precision	recall	f1	support
3Toeloop	0.743	0.684	0.712	38
3Loop	0.800	0.952	0.870	21
2Axel	0.792	0.884	0.835	43
CamelSpin	0.967	0.952	0.960	125
SitSpin	0.969	0.962	0.966	132
UprightSpin	0.960	0.889	0.923	108
2Salchow	0.400	0.400	0.400	5
2Toeloop	0.839	0.812	0.825	32
3Salchow	0.611	0.524	0.564	21
3Axel	0.923	0.800	0.857	15
3Flip	0.531	0.722	0.612	36
3Lutz	0.800	0.390	0.525	41
NoBasic	0.781	0.806	0.794	31
2Lutz	0.571	0.571	0.571	7
4Salchow	0.500	0.333	0.400	3
4Flip	0.000	0.000	0.000	1
4Toeloop	0.200	0.333	0.250	6
4Lutz	0.000	0.000	0.000	1
4Loop	0.000	0.000	0.000	1
2Flip	0.625	0.625	0.625	8
2Loop	0.467	0.875	0.609	8
1Axel	0.750	1.000	0.857	3
1Loop	0.000	0.000	0.000	1
1Salchow	0.000	0.000	0.000	1
1Toeloop	0.000	0.000	0.000	1
1Flip	0.000	0.000	0.000	2
1Lutz	0.000	0.000	0.000	1
Sequence	0.971	0.984	0.977	307
macro avg	0.507	0.518	0.505	999
weighted avg	0.880	0.869	0.870	999

Coarse (11-class)

Stopped epoch 45 (best @ ep25, val macro-F1 0.857)

Overall: accuracy 0.9089 · precision (macro/weighted) 0.8557 / 0.9126 · recall 0.8506 / 0.9089 · F1 0.8506 / 0.9089

class	precision	recall	f1	support
Axel	0.824	0.918	0.868	61
Toeloop	0.877	0.649	0.746	77
Salchow	0.833	0.833	0.833	30
Loop	0.857	0.774	0.814	31
Flip	0.627	0.681	0.653	47
Lutz	0.617	0.740	0.673	50
CamelSpin	0.984	0.984	0.984	125
SitSpin	0.977	0.962	0.969	132
UprightSpin	0.936	0.954	0.945	108
Sequence	0.981	0.990	0.985	307
NoBasic	0.900	0.871	0.885	31
macro avg	0.856	0.851	0.851	999
weighted avg	0.913	0.909	0.909	999

Notable failure modes (from confusion matrix analysis)

• Flip ↔ Lutz is the dominant confusion (edge jumps — genuinely hard to distinguish from skeleton motion alone, mirrors real judging difficulty).
• Fine-grained macro-F1 is dragged down almost entirely by 1-2-sample tail classes (4Flip, 4Lutz, 4Loop, 1Salchow, 1Toeloop, 1Flip, 1Lutz) — a data scarcity issue, not an architecture weakness.
• Spins (CamelSpin, SitSpin, UprightSpin) and Sequence are solved territory: F1 ≥ 0.92 in both label spaces.

License

Internal/proprietary — not licensed for external use. Contact the Mercity team for access terms.