REPROCESS_AND_TEST.md

Quick Reprocess and Test Instructions

Summary of Changes

I've fixed the rotation computation order issue in process_dataset.py:

The Problem

• Old: Compute relative rotation in Unity space, then convert to CV
• Issue: This breaks GENMO's expected formula R_c = R_w2c @ R_pel_w

The Fix

• New: Convert rotations to CV first, then compute relative rotation
• This matches GENMO's training data convention

Step-by-Step Instructions

1. Clean Old Processed Data

cd /root/miko/puni/train/PromptHMR/GENMO
rm -rf processed_dataset/genmo_features/*.pt

2. Reprocess Your Dataset

# Replace paths as needed
python third_party/GVHMR/tools/demo/process_dataset.py \
    --input /path/to/your/unity_export \
    --output processed_dataset \
    --genmo --vitpose --smplx \
    --consistency_check \
    --consistency_check_frames 5

Expected output:

• [Kabsch] Frame 0 consistency: Rotation err = 0.00°, Translation err = <0.1m
• Processing should complete without errors

3. Run Diagnosis

python diagnose_data.py

Expected results (V2 fix):

In-camera orientation errors (mean ± std):
  Yaw:     <5.00° ±  <2.00°   (was 9.44°)
  Pitch:   <5.00° ±  <2.00°   (was 10.95°)
  Roll:    <5.00° ±  <2.00°   (was 168.77° ← THE BUG!)

World orientation errors (mean ± std):
  Yaw:     <5.00° ±  <2.00°   (was 55.10°)
  Pitch:   <5.00° ±  <2.00°   (was 4.12°)
  Roll:    <5.00° ±  <2.00°   (was 3.65°)

Body pose error: <10.00° ± <20.00° (max: <100°)

4. If Errors Are Still High...

The rotation fix addresses the rotation computation order, but if errors persist, check:

A. Body Pose Export

Your Unity export's smplx_pose might have issues. Check:

# Test if body_pose matches between Unity and processed data
python test_single_frame.py

B. SMPL Model Mismatch

GENMO uses supermotion_v437coco17. Verify your Unity uses the same:

• Check betas (should be 10D, matching shape.npz)
• Check body_pose structure (should be 63D = 21 joints × 3)

C. Coordinate System Issues

If roll is exactly 180° off, you might need the Z-180° fix after all (but only for specific camera setups).

What Changed in process_dataset.py

Line 267-281: Incam Rotation

# OLD (BROKEN):
R_rel_unity = R_cam_w.T @ R_pel_w
R_cv = C @ R_rel_unity @ C

# NEW (FIXED):
R_cam_w_cv = C @ R_cam_w_unity @ C
R_w2c_cv = R_cam_w_cv.T
R_pel_w_cv = C @ R_pel_w_unity @ C
R_pel_c_cv = R_w2c_cv @ R_pel_w_cv  # GENMO's formula!

Line 602-614: World Rotation

# OLD: Recompute from Unity quaternions
# NEW: Use pre-converted CV rotations
R_c2w_cv = p['R_w2c_cv'].T
R_pelvis_w_cv = R_c2w_cv @ R_pelvis_c_cv

Line 665-671: Camera Matrix

# OLD: Convert Unity T_wc with C4 @ T @ C4
# NEW: Use pre-computed R_w2c_cv
cam_T_w2c_cv[:3, :3] = p["R_w2c_cv"]

Training

After reprocessing with the fix, training should:

• ✅ Start with loss ~1-5 (not 12)
• ✅ Decrease steadily (not explode to 100+)
• ✅ Converge to ~0.5-2.0 after sufficient epochs

If loss still explodes:

1. Check learning rate (might be too high for fine-tuning)
2. Check data augmentation settings
3. Verify batch size matches pretrained model's training setup

Files Modified

1. process_dataset.py
   • Lines 267-281: Fixed incam rotation derivation
   • Lines 306-318: Added R_w2c_cv to return dict
   • Lines 602-614: Use pre-converted rotations
   • Lines 665-671: Use pre-computed camera matrix

Next Steps

1. ✅ Reprocess dataset
2. ✅ Verify diagnosis shows <5° errors
3. ✅ Start training
4. 📊 Monitor loss curve (should decrease, not explode)

---

Quick Check: If diagnose_data.py still shows 168° roll error after reprocessing, the changes didn't apply. Check:

• Did you edit the correct process_dataset.py file?
• Did you delete old .pt files before reprocessing?
• Did the reprocessing script complete without errors?