docs/source/benchmark/metrics.md

Metrics

The benchmark evaluates generated motion along three axes:

• Motion quality -- foot-skate and contact-consistency metrics,
• Constraint following -- position error for root, end-effector, and full-body constraints,
• Text alignment -- TMR retrieval and distributional metrics.

Metrics are implemented in kimodo/metrics/ and orchestrated by benchmark/evaluate_folder.py. The protocol is aligned with the tech report (Sec. 6.1, "Evaluation Metrics").

Evaluation Protocol

The evaluation pipeline runs two passes over each group of test cases:

1. Generated pass -- evaluates motion.npz with all metrics (foot skate, contact consistency, constraint following) and, when TMR embeddings are available, computes retrieval and FID scores.
2. Ground-truth pass -- evaluates gt_motion.npz with the same motion-quality and constraint metrics. TMR retrieval metrics are not recomputed in this pass.

Running both passes enables side-by-side comparison: the GT row serves as an empirical upper bound for motion quality, and deviations between GT and generated metrics highlight where the model can improve. See Evaluation pipeline for the full workflow.

Metrics Reference

The table below lists every key written to metrics.json. Detailed descriptions follow in subsequent sections.

Key	Category	Unit	Direction
foot_skate_from_height	Motion quality	m/s	Lower is better
foot_skate_from_pred_contacts	Motion quality	m/s	Lower is better
foot_skate_max_vel	Motion quality	m/s	Lower is better
foot_skate_ratio	Motion quality	ratio (0--1)	Lower is better
foot_contact_consistency	Motion quality	ratio (0--1)	Higher is better
constraint_root2d_err	Constraint follow	m	Lower is better
constraint_root2d_err_p95	Constraint follow	m	Lower is better
constraint_root2d_acc	Constraint follow	ratio (0--1)	Higher is better
constraint_fullbody_keyframe	Constraint follow	m	Lower is better
constraint_end_effector	Constraint follow	m	Lower is better
TMR/t2m_sim	Text alignment	score (0--1)	Higher is better
TMR/t2m_R/R01 ... R10	Text alignment	%	Higher is better
TMR/t2m_R/MedR	Text alignment	rank	Lower is better
TMR/FID/gen_text	Text alignment	distance	Lower is better
TMR/FID/gen_gt	Text alignment	distance	Lower is better
TMR/FID/gt_text	Text alignment	distance	Lower is better
TMR/m2m_sim	Text alignment	score (0--1)	Higher is better
TMR/t2m_gt_sim	Text alignment	score (0--1)	Higher is better
TMR/m2m_R/R01 ... R10	Text alignment	%	Higher is better
TMR/t2m_gt_R/R01 ... R10	Text alignment	%	Higher is better

:::{note} Raw metric values are stored in SI units (meters for positions, m/s for velocities). The summary tables printed by benchmark/parse_folder.py convert constraint position errors to cm and foot-skate velocities to cm/s for readability. :::

Foot Skating Metrics

Foot skating measures how much a foot slides along the ground when it should be in static contact with the ground. Four complementary metrics capture different aspects of this artifact.

• foot_skate_from_height (m/s, lower is better): Mean velocity of the toe joints (left toe, right toe) on frames where the toe height is below a floor threshold (height_thresh = 0.05 m). This metric does not rely on predicted contact labels -- it uses a geometric criterion (Y-coordinate < threshold) to identify ground-contact frames.

• foot_skate_from_pred_contacts (m/s, lower is better): Mean velocity of all four foot joints (left/right heel and toe) on frames where the model predicts contact via the foot_contacts output. Unlike foot_skate_from_height, this metric trusts the model's own contact predictions and measures all four foot joints rather than toes only.

• foot_skate_max_vel (m/s, lower is better): Maximum velocity across all four foot joints and all time steps where predicted contact is active. This captures worst-case slip spikes that mean-based metrics can hide.

• foot_skate_ratio (ratio 0--1, lower is better): Fraction of ground-contact frames where toe velocity exceeds a threshold (vel_thresh = 0.2 m/s). A frame counts as ground contact when the toe is below height_thresh = 0.05 m on both the current and the next frame. Inspired by the GMD skating metric.

Contact Consistency Metric

• foot_contact_consistency (ratio 0--1, higher is better): Agreement between the model's predicted foot contacts and a heuristic contact detector based on joint height and velocity (vel_thresh = 0.15 m/s, height_thresh = 0.10 m). Computed as accuracy (1 - incorrect_ratio) over all time steps and four contact channels. A score of 1.0 means perfect agreement between predicted and heuristic contacts. As noted in the tech report, this metric provides important context for interpreting the contact-based foot-skate metrics above: if contact consistency is low, foot_skate_from_pred_contacts may be unreliable.

Constraint-Following Metrics

Constraint metrics are computed only when the test case includes a constraints.json file. The ContraintFollow metric class dispatches by constraint type:

• constraint_end_effector (m, lower is better): Mean Euclidean distance between target end-effector positions and generated joint positions at the constrained frames. Only position-constrained joints are evaluated (rotation targets are not measured by this metric).

• constraint_fullbody_keyframe (m, lower is better): Mean per-joint Euclidean distance between target and generated full-body joint positions at keyframes. The error is averaged over all joints and all keyframe frames.

• constraint_root2d_err (m, lower is better): Mean 2D Euclidean distance (in the XZ ground plane) between target and generated root positions at constrained frames.

• constraint_root2d_err_p95 (m, lower is better): 95th percentile of the per-frame root 2D error across all samples in a group. Computed during aggregation by evaluate_folder.py to capture tail-end failures that the mean can mask.

• constraint_root2d_acc (ratio 0--1, higher is better): Fraction of constrained root frames where the 2D position error is within a distance threshold (root_threshold = 0.10 m).

TMR-Based Metrics

Text alignment is evaluated using TMR (Text-to-Motion Retrieval), a separate encoder model that maps both text and motion into a shared embedding space. TMR is not used for generation -- it is loaded only for evaluation (see kimodo/model/tmr.py).

We release a version of TMR retrained on the full Rigplay dataset as TMR-SOMA-RP-v1. The original TMR was trained on HumanML3D; our retrained variant uses the same architecture but is trained on the Rigplay motion dataset, SOMA skeleton, and with LLM2Vec text embeddings.

Similarity Scores

TMR encodes each text prompt and each motion clip into a unit-length embedding vector. Cosine similarity between text and motion embeddings is rescaled to a [0, 1] range:

score = cosine_similarity / 2 + 0.5

Three per-test-case similarity scores are recorded:

• TMR/t2m_sim (0--1, higher is better): similarity between the text prompt and the generated motion.
• TMR/m2m_sim (0--1, higher is better): similarity between the generated and ground-truth motions (only when GT is available).
• TMR/t2m_gt_sim (0--1, higher is better): similarity between the text prompt and the GT motion (only when GT is available).

R-precision (Retrieval Accuracy)

R-precision measures whether the correct motion can be retrieved from a pool given its corresponding text query. For each text query in the evaluation group, all motions are ranked by TMR similarity. R@k is the percentage of queries where the correct motion appears in the top k results.

Reported keys: TMR/t2m_R/R01, R02, R03, R05, R10 (%), and TMR/t2m_R/MedR (median rank, lower is better) correspond to retrieval accuracy when using generated motions.

When ground-truth motions are available, analogous retrieval metrics are computed for motion-to-GT-motion (TMR/m2m_R/...) and text-to-GT-motion (TMR/t2m_gt_R/...).

:::{note} Near-duplicate text prompts can artificially penalize retrieval ranking. The evaluation handles this by grouping prompts whose text-text similarity exceeds a threshold of 0.99 and treating any motion in that group as a valid match. :::

FID (Frechet Inception Distance)

FID measures distributional distance between two sets of TMR embeddings by fitting a multivariate Gaussian to each set and computing the Frechet distance. Three FID variants are reported:

• TMR/FID/gen_gt: distance between generated-motion and GT-motion embeddings (only when GT is available). This is the FID metric that is typically reported in the motion generation literature.
• TMR/FID/gen_text: distance between generated-motion embeddings and text embeddings.
• TMR/FID/gt_text: distance between GT-motion and text embeddings (only when GT is available).

Lower values indicate that the two distributions are more similar. FID requires at least 2 samples; groups with fewer samples report NaN.

Per-Test-Case Retrieval

In addition to the aggregate metrics above, each test case's metrics.json includes a tmr block with single motion retrieval results:

• t2m_rank: the rank of the correct motion when retrieving with this test case's text query.
• top5_retrieved: the top-5 retrieved motions (sample IDs and text prompts) for inspection.

JSON Output Format

Below is a representative metrics.json written by evaluate_folder.py for a single test case with mixed constraints (root + end-effector + full-body) and TMR embeddings:

{
  "num_motions": 1,
  "folder": "...",
  "per_motion_mean_gen": {
    "foot_skate_from_height": 0.3144,
    "foot_skate_from_pred_contacts": 0.0672,
    "foot_skate_max_vel": 0.2109,
    "foot_contact_consistency": 0.9522,
    "foot_skate_ratio": 0.2182,
    "constraint_end_effector": 0.0286,
    "constraint_root2d_err": 0.0534,
    "constraint_root2d_acc": 1.0,
    "constraint_fullbody_keyframe": 0.0324,
    "TMR/t2m_sim": 0.8209
  },
  "per_motion_mean_gt": {
    "foot_skate_from_height": 0.2361,
    "foot_skate_from_pred_contacts": 0.0269,
    "foot_skate_max_vel": 0.1459,
    "foot_contact_consistency": 1.0,
    "foot_skate_ratio": 0.1402,
    "constraint_end_effector": 9.82e-07,
    "constraint_root2d_err": 0.0407,
    "constraint_root2d_acc": 1.0,
    "constraint_fullbody_keyframe": 8.73e-07
  },
  "tmr": {
    "t2m_rank": 2,
    "text": "A person is swiftly performing a dance move by moving their hands and legs.",
    "top5_retrieved": [
      {
        "id": "0231",
        "text": "A person is performing dance steps while stepping back and forward..."
      },
      {
        "id": "0029",
        "text": "A person is swiftly performing a dance move by moving their hands and legs."
      }
    ]
  }
}

Group-level aggregate JSONs (<group_name>.json) have the same structure but with num_motions > 1, averaged per-motion metrics, additional keys like constraint_root2d_err_p95, and a tmr block containing the aggregate retrieval and FID scores:

{
  "num_motions": 256,
  "folder": "...",
  "per_motion_mean_gen": {
    "foot_skate_from_height": 0.1742,
    "foot_skate_from_pred_contacts": 0.0611,
    "foot_skate_max_vel": 0.3747,
    "foot_contact_consistency": 0.9483,
    "foot_skate_ratio": 0.1499,
    "constraint_end_effector": 0.0367,
    "constraint_root2d_err": 0.0495,
    "constraint_root2d_acc": 0.9212,
    "constraint_fullbody_keyframe": 0.0324,
    "constraint_root2d_err_p95": 0.1115
  },
  "per_motion_mean_gt": {
    "foot_skate_from_height": 0.1617,
    "foot_skate_from_pred_contacts": 0.0235,
    "foot_skate_max_vel": 0.1185,
    "foot_contact_consistency": 1.0,
    "foot_skate_ratio": 0.1214,
    "constraint_end_effector": 1.48e-06,
    "constraint_root2d_err": 0.0376,
    "constraint_root2d_acc": 1.0,
    "constraint_fullbody_keyframe": 1.16e-06,
    "constraint_root2d_err_p95": 0.0602
  },
  "tmr": {
    "TMR/t2m_sim": 0.8742,
    "TMR/t2m_R/R01": 75.39,
    "TMR/t2m_R/R02": 85.55,
    "TMR/t2m_R/R03": 88.28,
    "TMR/t2m_R/R05": 90.23,
    "TMR/t2m_R/R10": 93.36,
    "TMR/t2m_R/MedR": 1.0,
    "TMR/t2m_R/len": 256.0,
    "TMR/FID/gen_text": 0.1442,
    "TMR/m2m_R/R01": 94.53,
    "TMR/m2m_R/R02": 97.66,
    "TMR/m2m_R/R03": 98.05,
    "TMR/m2m_R/R05": 98.83,
    "TMR/m2m_R/R10": 99.22,
    "TMR/m2m_R/MedR": 1.0,
    "TMR/m2m_R/len": 256.0,
    "TMR/t2m_gt_R/R01": 80.47,
    "TMR/t2m_gt_R/R02": 88.28,
    "TMR/t2m_gt_R/R03": 91.02,
    "TMR/t2m_gt_R/R05": 92.58,
    "TMR/t2m_gt_R/R10": 94.53,
    "TMR/t2m_gt_R/MedR": 1.0,
    "TMR/t2m_gt_R/len": 256.0,
    "TMR/FID/gen_gt": 0.0387,
    "TMR/FID/gt_text": 0.1349
  }
}