Q 1.1

Q 1.1 is a movement foundation model by cubit labs (internally Goata core-v1, size L). It reads short windows of body joints from any pose estimator (2D, 3D or both) and returns cleaned image and world joints with an uncertainty per joint, foot contacts per frame, and an embedding of the window. It is the frozen base that Goata's movement models are fine-tuned on. Those fine-tunes are not part of this release.

  • 38,414,870 parameters (width 512, 12 blocks, 8 heads), float32, model.safetensors.
  • Trained on 90.2 hours of motion capture and gait-lab data, all under licences that allow commercial use (below).
  • Weights and code: Apache 2.0.

What it reads

A window is 128 frames at 30, 60 or 120 Hz (4.3, 2.1 or 1.1 s). Resample to one of these rates first, and pass the pose estimator's raw output: the model was trained on noisy input, not on cleaned joints.

For each of 23 joints (shoulders, elbows, wrists, hips, knees, ankles, heels, toe tips, big toes, small toes, nose, ears) there are 8 features, built by goata.model.spec.featurize:

  • img_x, img_y: image joints. x is scaled by the aspect ratio, then both axes are centred on the window's median hip centre and divided by the body scale estimated from bone lengths, so the units are roughly metres.
  • world_x, world_y, world_z: the estimator's world joints in metres, hip-centred, camera axes (x right, y down, z away from the camera).
  • confidence: the estimator's visibility score.
  • img_present, world_present: 1 where the value exists. A missing joint gets zeros, and the model replaces it with a learned mask token. A source that has only 2D or only 3D joints works too.

"Left" and "right" are the subject's sides.

Context is optional, and any field may be missing: age, sex, height, body mass, leg length, and the gravity direction in camera axes (from the phone's sensors). encode_context refuses any other field. Clinical labels (conditions, pain, injuries) are never inputs.

What it returns

key shape meaning
img (B, 128, 23, 2) cleaned image joints in the model's image frame (denormalize_image maps them back to the frame)
world (B, 128, 23, 3) cleaned world joints, metres, camera axes
img_log_sigma, world_log_sigma same as above log of each value's predicted standard deviation
contact (B, 128, 2) foot-contact logits per frame, left and right
embedding (B, 512) one vector per window
tokens, tokens_present (B, 64, 23, 512), (B, 64, 23) the encoder's tokens (one per joint per 2 frames), for fine-tuning

Where the input has a joint, the model predicts a correction to it. Where it doesn't, the model predicts the position itself, which includes lifting 3D from 2D.

Use

The model code is in this repository (goata/, needs torch and numpy).

import sys

import torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file

path = snapshot_download("otterbox901/q-1.1")
sys.path.insert(0, path)
from goata.model.encoder import MotionModel
from goata.model.spec import denormalize_image, encode_context, featurize
from goata.skeleton import JOINT_NAMES  # the canonical joint order of the arrays below

model = MotionModel("L")
model.load_state_dict(load_file(f"{path}/model.safetensors"))
model.eval()

# 128 frames of canonical joints from your pose estimator, NaN where a joint is missing:
#   image (128, len(JOINT_NAMES), 4): x, y normalised to the frame (0-1), z (unused), visibility
#   world (128, len(JOINT_NAMES), 3): metres, hip-centred, camera axes
feats = featurize(image, world, aspect=frame_width / frame_height)
x = torch.from_numpy(feats.x)[None]
ctx = torch.from_numpy(encode_context({"height_m": 1.75}))[None]  # or encode_context(None)
fps = torch.tensor([30.0])                                          # 30, 60 or 120

with torch.no_grad():
    out = model(x, ctx, fps)

world_m = out["world"][0]                    # (128, 23, 3)
world_sigma_m = out["world_log_sigma"][0].exp()
contact_prob = out["contact"][0].sigmoid()   # (128, 2): left, right
image_xy = denormalize_image(out["img"][0].numpy(), frame_width / frame_height, feats.centre, feats.scale)

featurize returns None when no body is visible in the window.

Architecture

A factorised spatio-temporal transformer. Each joint's 2-frame patch is one token. A learned joint embedding is added to each token, plus its time in seconds (sinusoids), the frame rate (log2 of rate / 30) and the encoded context. So the same motion at 30 and 120 Hz lands on the same time axis. The 12 blocks alternate attention across joints (within a time step) and across time (per joint). Three heads sit on the encoder: refine (joints and log sigma), contact (from each foot's pooled tokens) and embedding.

Training

Data: 49,114 clips, 90.2 hours. Clips were split by subject: 47,066 for training, 2,048 held out.

dataset hours licence source
AddBiomechanics Dataset 1.0 (Werling et al., ECCV 2024, and the studies it collects) 54.1 CC BY 4.0 addbiomechanics.org, paper
A Synchronized Dual-System Marker-based and Markerless Motion Capture Dataset for Gait and Fundamental Fitness Activities (Capital University of Physical Education and Sports) 11.0 CC BY 4.0 10.5281/zenodo.18399031
CMU Graphics Lab Motion Capture Database 9.3 free for use in commercial products; the data may not be resold mocap.cs.cmu.edu
3D motion analysis dataset of healthy young adult volunteers walking and running on overground and treadmill (Riglet, Delphin, Claquesin, Orliac, Ornetti, Laroche, Gueugnon) 6.4 CC0 10.6084/m9.figshare.25592865
Kuopio gait dataset (Lavikainen, Vartiainen, Stenroth, Karjalainen, Korhonen, Liukkonen, Mononen) 5.1 CC BY 4.0 10.5281/zenodo.10559504
Health&Gait: a dataset for gait-based analysis (Zafra-Palma, Marín-Jiménez, Castro-Piñero, Cuenca-García, Muñoz-Salinas, Marín-Jiménez) 3.4 CC BY 4.0 10.5281/zenodo.14039922
A public data set of running biomechanics and the effects of running speed on lower extremity kinematics and kinetics (Fukuchi, Fukuchi, Duarte) 0.7 CC BY 4.0 10.6084/m9.figshare.4543435

Changes to the data: converted to one canonical skeleton and resampled. The 3D recordings were projected through virtual cameras, and a noise model fitted to pose estimators was applied, so the inputs look like phone pose output. None of the data is redistributed here.

Objectives, on two views of each window: a Gaussian negative log-likelihood of the clean image and world joints under the predicted mean and sigma (with masked joints and dropped world joints, so it also learns to fill gaps and lift 2D to 3D); binary cross-entropy of foot contacts; and an InfoNCE loss between the two views' embeddings.

Run: 100,000 steps of 64 windows, AdamW (learning rate 2e-4, 4,000 warmup steps, cosine decay), bf16. The first 2,000 steps ran on an L4 and the rest on one H100 (about 6.3 hours). The published weights are step 95,000, the best on the held-out evaluation.

Evaluation

Held-out subjects, simulated input. 1,024 windows from the held-out subjects, with the inputs made through virtual cameras and the noise model, not from real video. Errors are in mm (the image frame is in approximate metres).

measure model input classical cleaning
image error, joints the input has 26 54 51
world error, joints the input has 42
world error, all joints 52
world error, lifted from 2D alone 60
world error, running windows 48

Foot contacts: initial contact F1 95 % (11 ms timing error) and toe-off F1 95 % (15 ms). At 120 Hz: 88 % and 91 %. The weakest groups are dance and sport (world error 86 and 71 mm, from only 14 and 19 windows) and steep cameras (image error 35 at 50–70° elevation against 27 at eye level).

3DPW, real moving-camera phone video. 3DPW was used for evaluation only, never for training. The input was a MediaPipe pose estimator's output. Errors are measured on 12 SMPL joint centres, hip-centred. That isn't the published 14-joint protocol, so these numbers can't be compared with leaderboards.

split method MPJPE (mm) PA-MPJPE (mm)
test (37 people) pose estimator input 135.6 97.8
test this model 119.9 82.7
test this model, 2D input only 351.2 236.2
validation (16 people) pose estimator input 125.6 93.2
validation this model 114.4 81.0

On real video, lifting from 2D alone doesn't transfer: feed it the estimator's world joints as well.

Limitations and intended use

  • For research and development of movement analysis from pose estimates: cleaning joints, filling gaps, foot contacts, and a base to fine-tune on.
  • Not a medical device. It doesn't diagnose anything or predict injuries, and it shouldn't be used for clinical decisions.
  • The training data are lab recordings, mostly healthy adults walking and running. Children, older adults, people with movement disorders, other activities and lying or floor postures are scarcely represented.
  • The held-out numbers come from simulated input. On real video, check it against your own reference first.
  • Don't use it to identify people.

Licence and acknowledgements

The weights and the code in this repository are licensed under Apache 2.0 (LICENSE). The training data keep their own licences (table above), and their authors are credited there.

The data used in this project was obtained from mocap.cs.cmu.edu. The database was created with funding from NSF EIA-0196217.

Downloads last month
1
Safetensors
Model size
38.4M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support