docs/BA_VALIDATION_DIAGNOSTICS.md

BA Validation Diagnostics

This document explains the diagnostic information available when running BA validation to help understand why frames are being rejected.

Overview

When all frames are rejected, it's important to understand the root cause. The enhanced validation script now provides detailed diagnostics to help identify issues.

Diagnostic Information

1. Frame Categorization Statistics

Shows how many frames fall into each category:

• Accepted (< 2° rotation error): Frames where DA3 poses are very close to ARKit ground truth
• Rejected-Learnable (2-30° rotation error): Frames with moderate error that could be improved with training
• Rejected-Outlier (> 30° rotation error): Frames with very high error, likely outliers

2. Error Distribution

Provides statistical breakdown of rotation errors:

• Q1, Median, Q3: Quartiles showing error distribution
• 90th, 95th, 99th percentiles: High-end error values
• Helps identify if errors are uniformly high or if there are specific problem frames

3. Alignment Diagnostics

Checks if pose alignment is working correctly:

• Scale factor: Should be ~1.0 if DA3 and ARKit trajectories have similar scale
• Rotation matrix determinant: Should be ~1.0 for a valid rotation matrix
• Translation centers: Mean translation values for both pose sets

4. Per-Frame Error Breakdown

Shows rotation and translation error for each frame:

• Helps identify specific problematic frames
• Shows which frames are close to thresholds
• Useful for understanding error patterns

5. Pose Statistics

Translation magnitude statistics:

• DA3 poses: Range and magnitude of DA3 camera positions
• ARKit poses: Range and magnitude of ARKit camera positions
• Helps identify scale mismatches

Common Issues and Diagnostics

All Frames Rejected as Outliers

Possible causes:

1. Coordinate system mismatch: Check alignment rotation det (should be ~1.0)
2. Scale mismatch: Check scale factor (should be ~1.0)
3. DA3 model issues: Very high errors suggest DA3 poses are fundamentally wrong
4. ARKit data quality: Check if ARKit tracking was successful

Diagnostics to check:

• Alignment scale factor (if far from 1.0, there's a scale issue)
• Rotation error distribution (if all errors are > 170°, likely coordinate system issue)
• Translation error magnitude (if very large, scale or coordinate issue)

High but Variable Errors

Possible causes:

1. DA3 model limitations: Model may struggle with certain scene types
2. Motion blur: Fast camera movement can cause tracking issues
3. Low texture: Scenes with little texture are harder for visual odometry

Diagnostics to check:

• Error distribution quartiles (if spread is large, some frames are better)
• Per-frame errors (identify which frames are problematic)

Alignment Issues

Symptoms:

• Scale factor far from 1.0
• Rotation matrix det not ~1.0
• Very high translation errors

Solutions:

• Check coordinate system conversion
• Verify ARKit to OpenCV conversion is correct
• Ensure poses are in the same format (w2c vs c2w)

Using Diagnostics in API

The API now returns diagnostics in the validation results:

{
  "validation_stats": {
    "total_frames": 10,
    "accepted": 0,
    "rejected_learnable": 0,
    "rejected_outlier": 10,
    "diagnostics": {
      "error_distribution": {...},
      "alignment_info": {...},
      "per_frame_errors": [...]
    }
  }
}

Example Output

=== BA Validation Statistics ===
Total Frames Processed: 10

Frame Categorization:
  ✓ Accepted (< 2°):          0 frames (  0.0%)
  ⚠ Rejected-Learnable (2-30°):   0 frames (  0.0%)
  ✗ Rejected-Outlier (> 30°):   10 frames (100.0%)

Total Rejected: 10 frames (100.0%)

BA Validation Status: rejected_outlier
Max Rotation Error: 177.76°

=== Detailed Diagnostics ===
Rotation Error Distribution:
  Q1 (25th): 170.55°
  Median:    177.76°
  Q3 (75th): 177.76°
  90th:      177.76°
  95th:      177.76°

Alignment Diagnostics:
  Scale factor: 1.000000 (should be ~1.0)
  Rotation det: 1.000000 (should be ~1.0)

Sample Frame Errors (first 5):
  Frame 0: 177.76° rot, 1.740m trans - rejected_outlier
  Frame 1: 176.50° rot, 1.800m trans - rejected_outlier
  ...

Next Steps

If all frames are rejected:

1. Check alignment diagnostics (scale factor, rotation det)
2. Review error distribution to see if errors are uniformly high
3. Check per-frame errors to identify patterns
4. Verify coordinate system conversions
5. Check ARKit tracking quality
6. Consider if DA3 model is appropriate for this scene type