---
license: other
license_name: proprietary-internal
license_link: LICENSE
library_name: pytorch
tags:
  - figure-skating
  - action-recognition
  - skeleton-based
  - time-series-classification
  - continuous-thought-machine
  - pytorch
---

# Figure Skating CTM (Continuous Thought Machine) Action Classifiers

Continuous Thought Machine architecture for classifying figure-skating actions (jumps, spins,
sequences) from pose-skeleton time series: a Conv1D backbone feeding a CTM "thinking loop"
(recurrent neuron model + synchronization-based attention over the backbone features), run for
a fixed or adaptively-halted number of steps before classification.

This repo holds **four checkpoints** — two thinking-step configurations, each trained on both
label granularities:

| file | variant | thinking steps | label space | classes |
|---|---|---|---|---|
| `model_ctm-10_fine.pt` | CTM-10 (vanilla, fixed) | exactly 10, uniform weighting | fine-grained | 28 |
| `model_ctm-10_coarse.pt` | CTM-10 (vanilla, fixed) | exactly 10, uniform weighting | coarse | 11 |
| `model_ctm-10+act_fine.pt` | CTM-10+ACT (adaptive halting) | up to 10, learned per-sample halting | fine-grained | 28 |
| `model_ctm-10+act_coarse.pt` | CTM-10+ACT (adaptive halting) | up to 10, learned per-sample halting | coarse | 11 |

Not `transformers`-compatible — plain PyTorch `nn.Module` + `state_dict` checkpoints. Model
source code (`SkatingCTM` in the project's `model_ctm.py`, which depends on `ctm_components.py`,
`model.py`, and `model_gcn.py`) is **not included in this repo** — you'll need the project
source to reconstruct the class before loading the state dict.

## How to load

```python
from huggingface_hub import hf_hub_download
import torch

repo_id = "Mercity/figure-skating-ctm"
ckpt_path = hf_hub_download(repo_id, "model_ctm-10_fine.pt")
ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False)

# reconstruct with the project's SkatingCTM (model_ctm.py):
# model = SkatingCTM(
#     ckpt["in_features"], ckpt["num_classes"],
#     max_iterations=ckpt["max_iterations"], memory_length=ckpt["memory_length"],
#     use_act=ckpt["use_act"], backbone=ckpt["backbone"],
#     node_features=ckpt["node_features"], gcn_layers=ckpt["gcn_layers"],
# )
# model.load_state_dict(ckpt["state_dict"])
# model.eval()

# ckpt also carries the standardization stats used at train time and the taxonomy:
mu, sd = ckpt["feature_mean"], ckpt["feature_std"]     # normalize inputs with (x - mu) / sd
taxonomy = ckpt["taxonomy"]                            # {0: "3Toeloop", 1: "3Loop", ...}
```

Swap the filename to load any of the other three variants.

## Config

Shared across all four checkpoints:

| | |
|---|---|
| Backbone | conv (Conv1D stem(128) → ConvBlock(192,k3) → ConvBlock(256,k3) → ConvBlock(384,k5), d_backbone=384) |
| CTM core | d_model=256, memory_length=50, n_synch=64, num_heads=8, dropout=0.1 |
| Optimizer | AdamW, lr=3e-4, weight_decay=1e-4 |
| Batch size | 64 |
| Grad clip | 1.0 |
| Class weight cap | 5.0 |
| Early stop | patience 20 on val macro-F1, 100-epoch cap |
| Seed | 42 |
| Params | 2.18M (fine) / 2.15M (coarse) |

Where CTM-10 vs CTM-10+ACT differ:

| | CTM-10 | CTM-10+ACT |
|---|---|---|
| Thinking steps | fixed 10 every sample, uniform loss/prediction weight (1/10 each) | up to 10, learned per-sample halting (`use_act=True`); halt bias init −3.0 |
| ponder_lambda | 0.0 (no ACT term) | 0.01 |
| act_epsilon | n/a | 0.01 |

⚠️ **Note:** in both fine and coarse CTM-10+ACT runs, test-time mean thinking steps measured
**10.0** — every sample used the full step budget; no sample ever halted early. With a max of
10 iterations and a halt bias initialized to favor ~20 steps, ACT never got the chance to
demonstrate adaptive depth in these checkpoints — treat CTM-10+ACT here as "CTM-10 + a
ponder-cost regularizer on a fixed-depth computation," not genuinely adaptive-depth inference.

## Data

Trained on skeleton-extracted figure-skating clips (YOLO11n-pose, COCO-17 joints, imgsz=384)
from an internal Mercity dataset. Stratified train/val/test split, seed 42.
**Not included in this repo** (private internal dataset).

## Results (test split)

### CTM-10 — Fine-grained (28-class)
Early-stopped epoch 99 (best val macro-F1 0.638) · test mean thinking steps: 10.0

**Overall:** accuracy 0.8799 · precision (macro/weighted) 0.5055 / 0.8756 · recall 0.4962 / 0.8799 · **F1 0.4948 / 0.8749**

| class | precision | recall | f1 | support |
|---|---|---|---|---|
| 3Toeloop | 0.692 | 0.711 | 0.701 | 38 |
| 3Loop | 0.739 | 0.810 | 0.773 | 21 |
| 2Axel | 0.771 | 0.860 | 0.813 | 43 |
| CamelSpin | 0.976 | 0.968 | 0.972 | 125 |
| SitSpin | 0.963 | 0.992 | 0.978 | 132 |
| UprightSpin | 0.953 | 0.944 | 0.949 | 108 |
| 2Salchow | 0.800 | 0.800 | 0.800 | 5 |
| 2Toeloop | 0.833 | 0.781 | 0.806 | 32 |
| 3Salchow | 0.609 | 0.667 | 0.636 | 21 |
| 3Axel | 0.714 | 0.667 | 0.690 | 15 |
| 3Flip | 0.727 | 0.444 | 0.552 | 36 |
| 3Lutz | 0.612 | 0.732 | 0.667 | 41 |
| NoBasic | 0.962 | 0.806 | 0.877 | 31 |
| 2Lutz | 0.500 | 0.714 | 0.588 | 7 |
| 4Salchow | 0.000 | 0.000 | 0.000 | 3 |
| 4Flip | 0.000 | 0.000 | 0.000 | 1 |
| 4Toeloop | 0.000 | 0.000 | 0.000 | 6 |
| 4Lutz | 0.000 | 0.000 | 0.000 | 1 |
| 4Loop | 0.000 | 0.000 | 0.000 | 1 |
| 2Flip | 0.667 | 0.250 | 0.364 | 8 |
| 2Loop | 0.667 | 0.750 | 0.706 | 8 |
| 1Axel | 0.000 | 0.000 | 0.000 | 3 |
| 1Loop | 1.000 | 1.000 | 1.000 | 1 |
| 1Salchow | 0.000 | 0.000 | 0.000 | 1 |
| 1Toeloop | 0.000 | 0.000 | 0.000 | 1 |
| 1Flip | 0.000 | 0.000 | 0.000 | 2 |
| 1Lutz | 0.000 | 0.000 | 0.000 | 1 |
| Sequence | 0.968 | 0.997 | 0.982 | 307 |
| **macro avg** | **0.505** | **0.496** | **0.495** | 999 |
| **weighted avg** | **0.876** | **0.880** | **0.875** | 999 |

### CTM-10 — Coarse (11-class)
Early-stopped epoch 96 (best val macro-F1 0.849) · test mean thinking steps: 10.0

**Overall:** accuracy 0.8929 · precision (macro/weighted) 0.8188 / 0.8977 · recall 0.8276 / 0.8929 · **F1 0.8161 / 0.8925**

| class | precision | recall | f1 | support |
|---|---|---|---|---|
| Axel | 0.966 | 0.934 | 0.950 | 61 |
| Toeloop | 0.663 | 0.714 | 0.688 | 77 |
| Salchow | 0.773 | 0.567 | 0.654 | 30 |
| Loop | 0.630 | 0.935 | 0.753 | 31 |
| Flip | 0.683 | 0.596 | 0.636 | 47 |
| Lutz | 0.690 | 0.580 | 0.630 | 50 |
| CamelSpin | 0.992 | 0.960 | 0.976 | 125 |
| SitSpin | 0.975 | 0.902 | 0.937 | 132 |
| UprightSpin | 0.963 | 0.954 | 0.958 | 108 |
| Sequence | 0.974 | 0.993 | 0.984 | 307 |
| NoBasic | 0.698 | 0.968 | 0.811 | 31 |
| **macro avg** | **0.819** | **0.828** | **0.816** | 999 |
| **weighted avg** | **0.898** | **0.893** | **0.892** | 999 |

### CTM-10+ACT — Fine-grained (28-class)
Early-stopped epoch 66 (best val macro-F1 0.550) · test mean thinking steps: 10.0

**Overall:** accuracy 0.8559 · precision (macro/weighted) 0.4840 / 0.8645 · recall 0.5175 / 0.8559 · **F1 0.4775 / 0.8573**

| class | precision | recall | f1 | support |
|---|---|---|---|---|
| 3Toeloop | 0.622 | 0.737 | 0.675 | 38 |
| 3Loop | 0.650 | 0.619 | 0.634 | 21 |
| 2Axel | 0.829 | 0.791 | 0.810 | 43 |
| CamelSpin | 0.976 | 0.968 | 0.972 | 125 |
| SitSpin | 0.962 | 0.955 | 0.958 | 132 |
| UprightSpin | 0.950 | 0.889 | 0.919 | 108 |
| 2Salchow | 1.000 | 0.400 | 0.571 | 5 |
| 2Toeloop | 0.792 | 0.594 | 0.679 | 32 |
| 3Salchow | 0.565 | 0.619 | 0.591 | 21 |
| 3Axel | 0.500 | 0.733 | 0.595 | 15 |
| 3Flip | 0.564 | 0.611 | 0.587 | 36 |
| 3Lutz | 0.643 | 0.659 | 0.651 | 41 |
| NoBasic | 0.818 | 0.871 | 0.844 | 31 |
| 2Lutz | 0.667 | 0.571 | 0.615 | 7 |
| 4Salchow | 0.000 | 0.000 | 0.000 | 3 |
| 4Flip | 0.000 | 0.000 | 0.000 | 1 |
| 4Toeloop | 0.250 | 0.500 | 0.333 | 6 |
| 4Lutz | 0.000 | 0.000 | 0.000 | 1 |
| 4Loop | 0.000 | 0.000 | 0.000 | 1 |
| 2Flip | 0.800 | 0.500 | 0.615 | 8 |
| 2Loop | 0.400 | 0.500 | 0.444 | 8 |
| 1Axel | 0.000 | 0.000 | 0.000 | 3 |
| 1Loop | 0.333 | 1.000 | 0.500 | 1 |
| 1Salchow | 0.000 | 0.000 | 0.000 | 1 |
| 1Toeloop | 0.250 | 1.000 | 0.400 | 1 |
| 1Flip | 0.000 | 0.000 | 0.000 | 2 |
| 1Lutz | 0.000 | 0.000 | 0.000 | 1 |
| Sequence | 0.980 | 0.974 | 0.977 | 307 |
| **macro avg** | **0.484** | **0.518** | **0.477** | 999 |
| **weighted avg** | **0.864** | **0.856** | **0.857** | 999 |

### CTM-10+ACT — Coarse (11-class)
Early-stopped epoch 57 (best val macro-F1 0.823) · test mean thinking steps: 10.0

**Overall:** accuracy 0.8979 · precision (macro/weighted) 0.8201 / 0.9012 · recall 0.8362 / 0.8979 · **F1 0.8262 / 0.8986**

| class | precision | recall | f1 | support |
|---|---|---|---|---|
| Axel | 0.917 | 0.902 | 0.909 | 61 |
| Toeloop | 0.720 | 0.766 | 0.742 | 77 |
| Salchow | 0.645 | 0.667 | 0.656 | 30 |
| Loop | 0.711 | 0.871 | 0.783 | 31 |
| Flip | 0.682 | 0.638 | 0.659 | 47 |
| Lutz | 0.717 | 0.660 | 0.688 | 50 |
| CamelSpin | 0.976 | 0.968 | 0.972 | 125 |
| SitSpin | 1.000 | 0.917 | 0.957 | 132 |
| UprightSpin | 0.934 | 0.917 | 0.925 | 108 |
| Sequence | 0.984 | 0.990 | 0.987 | 307 |
| NoBasic | 0.737 | 0.903 | 0.812 | 31 |
| **macro avg** | **0.820** | **0.836** | **0.826** | 999 |
| **weighted avg** | **0.901** | **0.898** | **0.899** | 999 |

## Notable failure modes (from confusion matrix analysis, coarse label space)

- **Flip ↔ Lutz** is the dominant confusion in both variants — edge jumps, genuinely hard to
  distinguish from skeleton motion alone (mirrors real judging difficulty).
- CTM-10 (fixed) shows the field's worst `Salchow` recall (56.7%) and `Lutz` recall (58.0%),
  plus unusually strong `SitSpin→NoBasic` leakage (9/132) not seen in other architectures.
- CTM-10+ACT improves slightly over CTM-10 on most coarse classes but never surpasses
  CNN-BiLSTM or GCN on macro-F1.
- Fine-grained macro-F1 in both variants is dragged down almost entirely by 1-2-sample tail
  classes (`4Flip`, `4Lutz`, `4Loop`, `1Salchow`, `1Toeloop`, `1Flip`, `1Lutz`) — a data
  scarcity issue, not an architecture weakness.

## License

Internal/proprietary — not licensed for external use. Contact the Mercity team for access terms.