Mercity/figure-skating-gcn

Figure Skating GCN Action Classifier

A spatial Graph Convolutional Network stem + Transformer-encoder model for classifying figure-skating actions (jumps, spins, sequences) from pose-skeleton time series. Treats the 17 COCO joints as a graph (edges = anatomical bones), applies Kipf-Welling style graph convolution per frame, then a Transformer encoder does the temporal mixing across frames.

This repo holds two checkpoints trained on the same architecture and data split, differing only in label granularity:

file	label space	classes
model_gcn_fine.pt	fine-grained	28 (jump rotation-count preserved, e.g. 3Lutz, 2Axel)
model_gcn_coarse.pt	coarse	11 (jump rotation-count collapsed, e.g. Lutz, Axel)

Not transformers-compatible in the HF sense — this is a plain PyTorch nn.Module + state_dict checkpoint (custom architecture, not a Hub AutoModel). Model source code (SkatingGCNClassifier in the project's model_gcn.py, which depends on model.py, model_transformers.py, and preprocessing.py) is not included in this repo — you'll need the project source to reconstruct the class before loading the state dict.

How to load

from huggingface_hub import hf_hub_download
import torch

repo_id = "Mercity/figure-skating-gcn"
ckpt_path = hf_hub_download(repo_id, "model_gcn_fine.pt")
ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False)

# reconstruct with the project's SkatingGCNClassifier (model_gcn.py):
# model = SkatingGCNClassifier(
#     ckpt["num_classes"], node_features=ckpt["node_features"], gcn_layers=ckpt["gcn_layers"],
# )
# model.load_state_dict(ckpt["state_dict"])
# model.eval()

# ckpt also carries the standardization stats used at train time and the taxonomy:
mu, sd = ckpt["feature_mean"], ckpt["feature_std"]     # normalize inputs with (x - mu) / sd
taxonomy = ckpt["taxonomy"]                            # {0: "3Toeloop", 1: "3Loop", ...}

Swap model_gcn_fine.pt for model_gcn_coarse.pt to load the 11-class variant.

Config

Stem	Spatial-GCN: 17 COCO joints as a graph (bone-adjacency, Kipf-Welling H' = norm(A)@H@W), 2 GCN blocks, channels ramp 64→128
Node features	full — all 94 input dims (coords+velocity+angles+angular-velocity+bone-vectors) scattered onto their anatomical joint slot, not just raw (x,y) coordinates
Temporal block	Sinusoidal positional encoding + nn.TransformerEncoder, 3 layers, d_model=384, nhead=8
Head	Dense(384→1024,p0.5) → Dense(1024→512,p0.4) → Dense(512→256,p0.3) → LayerNorm(256) → Linear(256→num_classes)
Optimizer	Adam, lr=5e-4, weight_decay=1e-4
Batch size	64
Grad clip	1.0
Class weight cap	5.0
Early stop	patience 20 on val macro-F1, 100-epoch cap
Seed	42
Params	7.44M (both)

Data

Trained on skeleton-extracted figure-skating clips (YOLO11n-pose, COCO-17 joints, imgsz=384) from an internal Mercity dataset. Stratified train/val/test split, seed 42. Not included in this repo (private internal dataset).

Results (test split)

Fine-grained (28-class)

Early-stopped epoch 64 (best val macro-F1 0.517)

Overall: accuracy 0.8278 · precision (macro/weighted) 0.4863 / 0.8465 · recall 0.4581 / 0.8278 · F1 0.4473 / 0.8266

class	precision	recall	f1	support
3Toeloop	0.900	0.474	0.621	38
3Loop	0.391	0.857	0.537	21
2Axel	0.879	0.674	0.763	43
CamelSpin	0.938	0.968	0.953	125
SitSpin	1.000	0.886	0.940	132
UprightSpin	0.949	0.870	0.908	108
2Salchow	0.500	0.400	0.444	5
2Toeloop	0.895	0.531	0.667	32
3Salchow	0.500	0.667	0.571	21
3Axel	0.706	0.800	0.750	15
3Flip	0.559	0.528	0.543	36
3Lutz	0.512	0.537	0.524	41
NoBasic	0.650	0.839	0.732	31
2Lutz	0.429	0.429	0.429	7
4Salchow	0.167	0.333	0.222	3
4Flip	0.000	0.000	0.000	1
4Toeloop	1.000	0.167	0.286	6
4Lutz	0.000	0.000	0.000	1
4Loop	0.000	0.000	0.000	1
2Flip	0.800	0.500	0.615	8
2Loop	0.429	0.375	0.400	8
1Axel	0.000	0.000	0.000	3
1Loop	0.500	1.000	0.667	1
1Salchow	0.000	0.000	0.000	1
1Toeloop	0.000	0.000	0.000	1
1Flip	0.000	0.000	0.000	2
1Lutz	0.000	0.000	0.000	1
Sequence	0.913	0.993	0.952	307
macro avg	0.486	0.458	0.447	999
weighted avg	0.846	0.828	0.827	999

Coarse (11-class)

Early-stopped epoch 94 (best val macro-F1 0.845)

Overall: accuracy 0.9039 · precision (macro/weighted) 0.8403 / 0.9068 · recall 0.8456 / 0.9039 · F1 0.8393 / 0.9036

class	precision	recall	f1	support
Axel	0.967	0.951	0.959	61
Toeloop	0.726	0.792	0.758	77
Salchow	0.714	0.667	0.690	30
Loop	0.711	0.871	0.783	31
Flip	0.795	0.660	0.721	47
Lutz	0.762	0.640	0.696	50
CamelSpin	0.991	0.920	0.954	125
SitSpin	0.984	0.947	0.965	132
UprightSpin	0.917	0.926	0.922	108
Sequence	0.968	0.993	0.981	307
NoBasic	0.707	0.935	0.806	31
macro avg	0.840	0.846	0.839	999
weighted avg	0.907	0.904	0.904	999

Notable failure modes (from confusion matrix analysis, coarse label space)

• Second-best coarse macro-F1 of all architectures tested (0.839, behind only CNN-BiLSTM's 0.851) — the spatial graph structure pays off despite double the parameter count of the simpler CNN-BiLSTM baseline.
• Weakest fine-grained result among the "full-feature" models (0.447 macro-F1) — the per-frame spatial graph convolution doesn't help much once the task is telling 3Flip from 3Lutz apart, which is primarily a fine temporal-motion distinction rather than a spatial one.
• Flip ↔ Lutz confusion present, consistent with every other architecture tested.
• Fine-grained macro-F1 is dragged down almost entirely by 1-2-sample tail classes (4Flip, 4Lutz, 4Loop, 1Salchow, 1Toeloop, 1Flip, 1Lutz) — a data scarcity issue, not an architecture weakness.

License

Internal/proprietary — not licensed for external use. Contact the Mercity team for access terms.