Q 1.1
Q 1.1 is a movement foundation model by cubit labs (internally Goata core-v1, size L). It reads short windows of
body joints from any pose estimator (2D, 3D or both) and returns cleaned image and world joints with an uncertainty
per joint, foot contacts per frame, and an embedding of the window. It is the frozen base that Goata's movement models
are fine-tuned on. Those fine-tunes are not part of this release.
- 38,414,870 parameters (width 512, 12 blocks, 8 heads), float32,
model.safetensors. - Trained on 90.2 hours of motion capture and gait-lab data, all under licences that allow commercial use (below).
- Weights and code: Apache 2.0.
What it reads
A window is 128 frames at 30, 60 or 120 Hz (4.3, 2.1 or 1.1 s). Resample to one of these rates first, and pass the pose estimator's raw output: the model was trained on noisy input, not on cleaned joints.
For each of 23 joints (shoulders, elbows, wrists, hips, knees, ankles, heels, toe tips, big toes, small toes, nose,
ears) there are 8 features, built by goata.model.spec.featurize:
img_x,img_y: image joints. x is scaled by the aspect ratio, then both axes are centred on the window's median hip centre and divided by the body scale estimated from bone lengths, so the units are roughly metres.world_x,world_y,world_z: the estimator's world joints in metres, hip-centred, camera axes (x right, y down, z away from the camera).confidence: the estimator's visibility score.img_present,world_present: 1 where the value exists. A missing joint gets zeros, and the model replaces it with a learned mask token. A source that has only 2D or only 3D joints works too.
"Left" and "right" are the subject's sides.
Context is optional, and any field may be missing: age, sex, height, body mass, leg length, and the gravity direction
in camera axes (from the phone's sensors). encode_context refuses any other field. Clinical labels (conditions,
pain, injuries) are never inputs.
What it returns
| key | shape | meaning |
|---|---|---|
img |
(B, 128, 23, 2) | cleaned image joints in the model's image frame (denormalize_image maps them back to the frame) |
world |
(B, 128, 23, 3) | cleaned world joints, metres, camera axes |
img_log_sigma, world_log_sigma |
same as above | log of each value's predicted standard deviation |
contact |
(B, 128, 2) | foot-contact logits per frame, left and right |
embedding |
(B, 512) | one vector per window |
tokens, tokens_present |
(B, 64, 23, 512), (B, 64, 23) | the encoder's tokens (one per joint per 2 frames), for fine-tuning |
Where the input has a joint, the model predicts a correction to it. Where it doesn't, the model predicts the position itself, which includes lifting 3D from 2D.
Use
The model code is in this repository (goata/, needs torch and numpy).
import sys
import torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
path = snapshot_download("otterbox901/q-1.1")
sys.path.insert(0, path)
from goata.model.encoder import MotionModel
from goata.model.spec import denormalize_image, encode_context, featurize
from goata.skeleton import JOINT_NAMES # the canonical joint order of the arrays below
model = MotionModel("L")
model.load_state_dict(load_file(f"{path}/model.safetensors"))
model.eval()
# 128 frames of canonical joints from your pose estimator, NaN where a joint is missing:
# image (128, len(JOINT_NAMES), 4): x, y normalised to the frame (0-1), z (unused), visibility
# world (128, len(JOINT_NAMES), 3): metres, hip-centred, camera axes
feats = featurize(image, world, aspect=frame_width / frame_height)
x = torch.from_numpy(feats.x)[None]
ctx = torch.from_numpy(encode_context({"height_m": 1.75}))[None] # or encode_context(None)
fps = torch.tensor([30.0]) # 30, 60 or 120
with torch.no_grad():
out = model(x, ctx, fps)
world_m = out["world"][0] # (128, 23, 3)
world_sigma_m = out["world_log_sigma"][0].exp()
contact_prob = out["contact"][0].sigmoid() # (128, 2): left, right
image_xy = denormalize_image(out["img"][0].numpy(), frame_width / frame_height, feats.centre, feats.scale)
featurize returns None when no body is visible in the window.
Architecture
A factorised spatio-temporal transformer. Each joint's 2-frame patch is one token. A learned joint embedding is added to each token, plus its time in seconds (sinusoids), the frame rate (log2 of rate / 30) and the encoded context. So the same motion at 30 and 120 Hz lands on the same time axis. The 12 blocks alternate attention across joints (within a time step) and across time (per joint). Three heads sit on the encoder: refine (joints and log sigma), contact (from each foot's pooled tokens) and embedding.
Training
Data: 49,114 clips, 90.2 hours. Clips were split by subject: 47,066 for training, 2,048 held out.
| dataset | hours | licence | source |
|---|---|---|---|
| AddBiomechanics Dataset 1.0 (Werling et al., ECCV 2024, and the studies it collects) | 54.1 | CC BY 4.0 | addbiomechanics.org, paper |
| A Synchronized Dual-System Marker-based and Markerless Motion Capture Dataset for Gait and Fundamental Fitness Activities (Capital University of Physical Education and Sports) | 11.0 | CC BY 4.0 | 10.5281/zenodo.18399031 |
| CMU Graphics Lab Motion Capture Database | 9.3 | free for use in commercial products; the data may not be resold | mocap.cs.cmu.edu |
| 3D motion analysis dataset of healthy young adult volunteers walking and running on overground and treadmill (Riglet, Delphin, Claquesin, Orliac, Ornetti, Laroche, Gueugnon) | 6.4 | CC0 | 10.6084/m9.figshare.25592865 |
| Kuopio gait dataset (Lavikainen, Vartiainen, Stenroth, Karjalainen, Korhonen, Liukkonen, Mononen) | 5.1 | CC BY 4.0 | 10.5281/zenodo.10559504 |
| Health&Gait: a dataset for gait-based analysis (Zafra-Palma, Marín-Jiménez, Castro-Piñero, Cuenca-García, Muñoz-Salinas, Marín-Jiménez) | 3.4 | CC BY 4.0 | 10.5281/zenodo.14039922 |
| A public data set of running biomechanics and the effects of running speed on lower extremity kinematics and kinetics (Fukuchi, Fukuchi, Duarte) | 0.7 | CC BY 4.0 | 10.6084/m9.figshare.4543435 |
Changes to the data: converted to one canonical skeleton and resampled. The 3D recordings were projected through virtual cameras, and a noise model fitted to pose estimators was applied, so the inputs look like phone pose output. None of the data is redistributed here.
Objectives, on two views of each window: a Gaussian negative log-likelihood of the clean image and world joints under the predicted mean and sigma (with masked joints and dropped world joints, so it also learns to fill gaps and lift 2D to 3D); binary cross-entropy of foot contacts; and an InfoNCE loss between the two views' embeddings.
Run: 100,000 steps of 64 windows, AdamW (learning rate 2e-4, 4,000 warmup steps, cosine decay), bf16. The first 2,000 steps ran on an L4 and the rest on one H100 (about 6.3 hours). The published weights are step 95,000, the best on the held-out evaluation.
Evaluation
Held-out subjects, simulated input. 1,024 windows from the held-out subjects, with the inputs made through virtual cameras and the noise model, not from real video. Errors are in mm (the image frame is in approximate metres).
| measure | model | input | classical cleaning |
|---|---|---|---|
| image error, joints the input has | 26 | 54 | 51 |
| world error, joints the input has | 42 | ||
| world error, all joints | 52 | ||
| world error, lifted from 2D alone | 60 | ||
| world error, running windows | 48 |
Foot contacts: initial contact F1 95 % (11 ms timing error) and toe-off F1 95 % (15 ms). At 120 Hz: 88 % and 91 %. The weakest groups are dance and sport (world error 86 and 71 mm, from only 14 and 19 windows) and steep cameras (image error 35 at 50–70° elevation against 27 at eye level).
3DPW, real moving-camera phone video. 3DPW was used for evaluation only, never for training. The input was a MediaPipe pose estimator's output. Errors are measured on 12 SMPL joint centres, hip-centred. That isn't the published 14-joint protocol, so these numbers can't be compared with leaderboards.
| split | method | MPJPE (mm) | PA-MPJPE (mm) |
|---|---|---|---|
| test (37 people) | pose estimator input | 135.6 | 97.8 |
| test | this model | 119.9 | 82.7 |
| test | this model, 2D input only | 351.2 | 236.2 |
| validation (16 people) | pose estimator input | 125.6 | 93.2 |
| validation | this model | 114.4 | 81.0 |
On real video, lifting from 2D alone doesn't transfer: feed it the estimator's world joints as well.
Limitations and intended use
- For research and development of movement analysis from pose estimates: cleaning joints, filling gaps, foot contacts, and a base to fine-tune on.
- Not a medical device. It doesn't diagnose anything or predict injuries, and it shouldn't be used for clinical decisions.
- The training data are lab recordings, mostly healthy adults walking and running. Children, older adults, people with movement disorders, other activities and lying or floor postures are scarcely represented.
- The held-out numbers come from simulated input. On real video, check it against your own reference first.
- Don't use it to identify people.
Licence and acknowledgements
The weights and the code in this repository are licensed under Apache 2.0 (LICENSE). The training data keep their
own licences (table above), and their authors are credited there.
The data used in this project was obtained from mocap.cs.cmu.edu. The database was created with funding from NSF EIA-0196217.
- Downloads last month
- 1