╔═══════════════════════════════════════════════════════════════════════════════════════════════╗ ║ SPATIAL-BEATS TRAINING VS VALIDATION DATA FLOW ║ ║ (SHOWING THE ROOT CAUSES OF DOA GAP) ║ ╚═══════════════════════════════════════════════════════════════════════════════════════════════╝ TRAINING PATH (epoch T) VALIDATION PATH (epoch T) ════════════════════════════════════════════════════════════════════════════════════════════════ ┌─────────────────────────────────┐ ┌──────────────────────────────┐ │ FOA Waveform [4, T] │ │ FOA Waveform [4, T] │ │ (W, X, Y, Z channels) │ │ (W, X, Y, Z channels) │ └──────────────┬──────────────────┘ └────────────┬─────────────────┘ │ │ ▼ ─────────────────────────────────────────► ▼ ╔══════════════════════╗ ╔═══════════════════╗ │ PREPROCESSING │ │ PREPROCESSING │ ├──────────────────────┤ ├───────────────────┤ │ • Normalize (same) │ │ • Normalize (same)│ │ • Mel filterbank (OK)│ │ • Mel filterbank │ │ • FOA features (OK) │ │ (OK) │ └──────────┬───────────┘ └─────────┬─────────┘ │ │ ⚠️ ISSUE #1 ◄──┘─────────────────────────────────────────┘ ⚠️ ISSUE #1 SpecAugment: SpecAugment: ├─ freq_masks=2×27 ├─ NO MASKING ├─ time_masks=2×100 ├─ w_logmel unchanged └─ Applied to W channel └─ Returns w_logmel as-is Result: ~20-50% of spec erased Result: Full pristine spectrum │ │ ▼ ─────────────────────────────────────────► ▼ ╔══════════════════════╗ ╔═══════════════════╗ │ ENCODER (Frozen) │ │ ENCODER (Eval) │ │ • BEATs backbone │ │ • BEATs backbone │ │ • SpatialFuser │ │ • SpatialFuser │ │ • LocalSpatialEnc │ │ • LocalSpatialEnc │ └──────────┬───────────┘ └─────────┬─────────┘ │ │ [B, T_s, D] [B, T_s, D] │ │ ▼ ▼ ╔══════════════════════╗ ╔═══════════════════╗ │ PREDICTION HEADS │ │ PREDICTION HEADS │ ├──────────────────────┤ ├───────────────────┤ │ activity_head │ │ activity_head │ │ class_head │ │ class_head │ │ direction_head │ │ direction_head │ │ distance_head │ │ distance_head │ └──────────┬───────────┘ └─────────┬─────────┘ │ │ │ pred_activity [B, T_s, K] │ │ pred_class [B, T_s, K, C] │ │ pred_dir [B, T_s, K, 3] │ (STALE in v10p1) │ pred_dist [B, T_s, K] │ │ │ ▼ ▼ ╔══════════════════════╗ ╔═══════════════════╗ │ HUNGARIAN MATCHING │ │ HUNGARIAN MATCHING│ │ (per-frame) │ │ (per-frame) │ ├──────────────────────┤ ├───────────────────┤ │ For each (b, t): │ │ For each (b, t): │ │ cost = act_cost + │ │ cost = act_cost +│ │ cls_nll + │ │ cls_nll + │ │ dir_cost + │ │ dir_cost +│ │ dist_cost │ │ dist_cost │ │ │ │ │ ⚠️ │ ISSUE #4: Fixed cost │ ────────────────── │ ISSUE #4: But │ │ weights (1.0 each) │ (v10p1: matches │ loss uses lambda │ │ but loss uses │ on stale DOA) │ weights (0.0) │ │ variable lambda │ │ │ └──────────┬───────────┘ └─────────┬─────────┘ │ │ matched_slot matched_slot [B, N_gt, T_s] [B, N_gt, T_s] │ │ ▼ ▼ ╔══════════════════════╗ ╔═══════════════════╗ │ LOSS COMPUTATION │ │ METRICS COMPUTATION ├──────────────────────┤ ├───────────────────┤ │ loss_activity × │ │ For matched pairs: │ │ λ_frame_activity │ │ pred_azi_deg │ │ loss_class × │ │ pred_ele_deg │ │ λ_frame_class │ │ tgt_azi_deg │ │ loss_direction × ◄─ v10p1: 0.0 │ tgt_ele_deg │ │ λ_frame_direction │ │ │ │ loss_distance × ◄─ v10p1: 0.0 │ azi_mae = MAE │ │ λ_frame_distance │ │ ele_mae = MAE │ └──────────┬───────────┘ └─────────┬─────────┘ │ │ ⚠️ ISSUE #2 │ v9: Full training │ ────────────► │ v10 phase-1: FROZEN dir/dist head │ │ no gradients! │ │ │ ▼ ▼ loss_total azi_mae, ele_mae (backprop) (forward only) KEY ASYMMETRIES CAUSING TRAIN/VALID GAP ════════════════════════════════════════════════════════════════════════════════════════════════ ┌─────────────────────────────────┬──────────────────────────┬──────────────┐ │ Aspect │ Training │ Validation │ ├─────────────────────────────────┼──────────────────────────┼──────────────┤ │ SpecAugment │ Masking 45% of spectrum │ NO masking │ │ │ →音频特性不同 │ ├─────────────────────────────────┼──────────────────────────┼──────────────┤ │ Spatial Augmentation (Rotation) │ NONE │ NONE │ │ │ → DOA diversity limited │ → Unseen │ │ │ to ~1 epoch │ angles │ ├─────────────────────────────────┼──────────────────────────┼──────────────┤ │ Direction Head State (v10p1) │ FROZEN (no gradients) │ FROZEN + │ │ │ λ=0.0 (no loss signal) │ STALE (from │ │ │ │ v9 epoch 3)│ ├─────────────────────────────────┼──────────────────────────┼──────────────┤ │ Matching Cost Weights │ Unweighted sum │ Same unweighted │ │ act + cls + dir + dist │ sum, but on │ │ │ (but loss uses lambda) │ frozen DOA │ ├─────────────────────────────────┼──────────────────────────┼──────────────┤ │ Gradient Flow │ Through loss (v9) or │ Validation │ │ │ None (v10p1) │ mode: no │ │ │ │ gradients │ └─────────────────────────────────┴──────────────────────────┴──────────────┘ IMPACT CASCADE (Most Severe Scenario: ov3, v10 Phase-1) ════════════════════════════════════════════════════════════════════════════════════════════════ 1. Training sees only ~single orientation per azimuth per epoch No rotation aug → limited DOA diversity ├─→ Model sees: (azi=0°, ele=0°), (azi=1°, ele=1°), ... └─→ Model learns: limited generalization to unseen angles 2. Direction head frozen at v10 phase-1 start λ_frame_direction = 0.0 → no training signal freeze_frame_track_spatial_heads = True → no gradients ├─→ Head receives: ZERO gradients from loss ├─→ Head receives: ZERO gradients from backward pass └─→ Result: Output completely stale (from v9 epoch 3) 3. At validation: direction head outputs used to compute metrics But outputs are unchanged from 10 epochs ago And model hasn't learned ov2/ov3 multi-source DOA at all ├─→ Validation sees: unseen azimuth combos (not in training) ├─→ Validation sees: multi-source situations unfamiliar to frozen head └─→ Result: 20-30° azimuth errors 4. SpecAugment makes training acoustic features different Masking 45% of W channel during training Zero masking at validation └─→ Result: +2-5° additional error on top FIX #1: Add Rotation Augmentation ════════════════════════════════════════════════════════════════════════════════════════════════ Current (BROKEN): FOA [W, X, Y, Z] → (no change) → Encoder → DOA predictor Fixed (CORRECT): FOA [W, X, Y, Z] → Rotate(θ, φ, ψ) → [W', X', Y', Z'] → Encoder → DOA predictor Labels: (azi, ele, dist) → Rotate_targets(θ, φ, ψ) → (azi', ele', dist) Result: Same class, different DOA → model learns generalization FIX #2: Make SpecAugment Train/Val Consistent ════════════════════════════════════════════════════════════════════════════════════════════════ Current (BROKEN): Training: if self.training: apply_masking() ✓ Validation: if not self.training: skip_masking() ✗ (mismatch!) Fixed (OPTION A): Training: No masking (simplified, possibly less robust) Validation: No masking (consistent) Fixed (OPTION B): Training: apply_masking() ✓ Validation: apply_same_masking_with_fixed_seed() ✓ (consistent) FIX #3: Don't Fully Freeze Direction Head (v10) ════════════════════════════════════════════════════════════════════════════════════════════════ Current v10 (BROKEN): λ_frame_direction = 0.0 freeze_frame_track_spatial_heads = True → Head receives zero signal for 10 epochs Fixed (BETTER): λ_frame_direction = 0.1 (not zero!) freeze_frame_track_spatial_heads = False → Head receives weak signal to fine-tune on multi-source → Prevents complete staleness at v10 unfreezing