REPROCESS_AND_TEST.md

# Quick Reprocess and Test Instructions

## Summary of Changes

I've fixed the **rotation computation order** issue in `process_dataset.py`:

### The Problem
- Old: Compute relative rotation in Unity space, then convert to CV
- Issue: This breaks GENMO's expected formula `R_c = R_w2c @ R_pel_w`

### The Fix
- New: Convert rotations to CV **first**, then compute relative rotation
- This matches GENMO's training data convention

## Step-by-Step Instructions

### 1. Clean Old Processed Data
```bash
cd /root/miko/puni/train/PromptHMR/GENMO
rm -rf processed_dataset/genmo_features/*.pt
```

### 2. Reprocess Your Dataset
```bash
# Replace paths as needed
python third_party/GVHMR/tools/demo/process_dataset.py \
    --input /path/to/your/unity_export \
    --output processed_dataset \
    --genmo --vitpose --smplx \
    --consistency_check \
    --consistency_check_frames 5
```

**Expected output:**
- `[Kabsch] Frame 0 consistency: Rotation err = 0.00°, Translation err = <0.1m`
- Processing should complete without errors

### 3. Run Diagnosis
```bash
python diagnose_data.py
```

**Expected results (V2 fix):**
```
In-camera orientation errors (mean ± std):
  Yaw:     <5.00° ±  <2.00°   (was 9.44°)
  Pitch:   <5.00° ±  <2.00°   (was 10.95°)
  Roll:    <5.00° ±  <2.00°   (was 168.77° ← THE BUG!)

World orientation errors (mean ± std):
  Yaw:     <5.00° ±  <2.00°   (was 55.10°)
  Pitch:   <5.00° ±  <2.00°   (was 4.12°)
  Roll:    <5.00° ±  <2.00°   (was 3.65°)

Body pose error: <10.00° ± <20.00° (max: <100°)
```

### 4. If Errors Are Still High...

The rotation fix addresses the **rotation computation order**, but if errors persist, check:

#### A. Body Pose Export
Your Unity export's `smplx_pose` might have issues. Check:
```bash
# Test if body_pose matches between Unity and processed data
python test_single_frame.py
```

#### B. SMPL Model Mismatch
GENMO uses `supermotion_v437coco17`. Verify your Unity uses the same:
- Check betas (should be 10D, matching `shape.npz`)
- Check body_pose structure (should be 63D = 21 joints × 3)

#### C. Coordinate System Issues
If roll is exactly 180° off, you might need the Z-180° fix after all (but only for specific camera setups).

## What Changed in process_dataset.py

### Line 267-281: Incam Rotation
```python
# OLD (BROKEN):
R_rel_unity = R_cam_w.T @ R_pel_w
R_cv = C @ R_rel_unity @ C

# NEW (FIXED):
R_cam_w_cv = C @ R_cam_w_unity @ C
R_w2c_cv = R_cam_w_cv.T
R_pel_w_cv = C @ R_pel_w_unity @ C
R_pel_c_cv = R_w2c_cv @ R_pel_w_cv  # GENMO's formula!
```

### Line 602-614: World Rotation
```python
# OLD: Recompute from Unity quaternions
# NEW: Use pre-converted CV rotations
R_c2w_cv = p['R_w2c_cv'].T
R_pelvis_w_cv = R_c2w_cv @ R_pelvis_c_cv
```

### Line 665-671: Camera Matrix
```python
# OLD: Convert Unity T_wc with C4 @ T @ C4
# NEW: Use pre-computed R_w2c_cv
cam_T_w2c_cv[:3, :3] = p["R_w2c_cv"]
```

## Training

After reprocessing with the fix, training should:
- ✅ Start with loss ~1-5 (not 12)
- ✅ Decrease steadily (not explode to 100+)
- ✅ Converge to ~0.5-2.0 after sufficient epochs

If loss still explodes:
1. Check learning rate (might be too high for fine-tuning)
2. Check data augmentation settings
3. Verify batch size matches pretrained model's training setup

## Files Modified

1. [process_dataset.py](third_party/GVHMR/tools/demo/process_dataset.py)
   - Lines 267-281: Fixed incam rotation derivation
   - Lines 306-318: Added R_w2c_cv to return dict
   - Lines 602-614: Use pre-converted rotations
   - Lines 665-671: Use pre-computed camera matrix

## Next Steps

1. ✅ Reprocess dataset
2. ✅ Verify diagnosis shows <5° errors
3. ✅ Start training
4. 📊 Monitor loss curve (should decrease, not explode)

---

**Quick Check**: If `diagnose_data.py` still shows 168° roll error after reprocessing, the changes didn't apply. Check:
- Did you edit the correct `process_dataset.py` file?
- Did you delete old `.pt` files before reprocessing?
- Did the reprocessing script complete without errors?