Spatial-BEATs / docs /train_valid_gap_diagram.txt

Add files using upload-large-folder tool

dd39446 verified about 1 month ago

16.4 kB

	╔═══════════════════════════════════════════════════════════════════════════════════════════════╗
	║ SPATIAL-BEATS TRAINING VS VALIDATION DATA FLOW ║
	║ (SHOWING THE ROOT CAUSES OF DOA GAP) ║
	╚═══════════════════════════════════════════════════════════════════════════════════════════════╝

	TRAINING PATH (epoch T) VALIDATION PATH (epoch T)
	════════════════════════════════════════════════════════════════════════════════════════════════

	┌─────────────────────────────────┐ ┌──────────────────────────────┐
	│ FOA Waveform [4, T] │ │ FOA Waveform [4, T] │
	│ (W, X, Y, Z channels) │ │ (W, X, Y, Z channels) │
	└──────────────┬──────────────────┘ └────────────┬─────────────────┘
	│ │
	▼ ─────────────────────────────────────────► ▼
	╔══════════════════════╗ ╔═══════════════════╗
	│ PREPROCESSING │ │ PREPROCESSING │
	├──────────────────────┤ ├───────────────────┤
	│ • Normalize (same) │ │ • Normalize (same)│
	│ • Mel filterbank (OK)│ │ • Mel filterbank │
	│ • FOA features (OK) │ │ (OK) │
	└──────────┬───────────┘ └─────────┬─────────┘
	│ │
	⚠️ ISSUE #1 ◄──┘─────────────────────────────────────────┘ ⚠️ ISSUE #1

	SpecAugment: SpecAugment:
	├─ freq_masks=2×27 ├─ NO MASKING
	├─ time_masks=2×100 ├─ w_logmel unchanged
	└─ Applied to W channel └─ Returns w_logmel as-is

	Result: ~20-50% of spec erased Result: Full pristine spectrum

	│ │
	▼ ─────────────────────────────────────────► ▼
	╔══════════════════════╗ ╔═══════════════════╗
	│ ENCODER (Frozen) │ │ ENCODER (Eval) │
	│ • BEATs backbone │ │ • BEATs backbone │
	│ • SpatialFuser │ │ • SpatialFuser │
	│ • LocalSpatialEnc │ │ • LocalSpatialEnc │
	└──────────┬───────────┘ └─────────┬─────────┘
	│ │
	[B, T_s, D] [B, T_s, D]
	│ │
	▼ ▼
	╔══════════════════════╗ ╔═══════════════════╗
	│ PREDICTION HEADS │ │ PREDICTION HEADS │
	├──────────────────────┤ ├───────────────────┤
	│ activity_head │ │ activity_head │
	│ class_head │ │ class_head │
	│ direction_head │ │ direction_head │
	│ distance_head │ │ distance_head │
	└──────────┬───────────┘ └─────────┬─────────┘
	│ │
	│ pred_activity [B, T_s, K] │
	│ pred_class [B, T_s, K, C] │
	│ pred_dir [B, T_s, K, 3] │ (STALE in v10p1)
	│ pred_dist [B, T_s, K] │
	│ │
	▼ ▼
	╔══════════════════════╗ ╔═══════════════════╗
	│ HUNGARIAN MATCHING │ │ HUNGARIAN MATCHING│
	│ (per-frame) │ │ (per-frame) │
	├──────────────────────┤ ├───────────────────┤
	│ For each (b, t): │ │ For each (b, t): │
	│ cost = act_cost + │ │ cost = act_cost +│
	│ cls_nll + │ │ cls_nll + │
	│ dir_cost + │ │ dir_cost +│
	│ dist_cost │ │ dist_cost │
	│ │ │ │
	⚠️ │ ISSUE #4: Fixed cost │ ────────────────── │ ISSUE #4: But │
	│ weights (1.0 each) │ (v10p1: matches │ loss uses lambda │
	│ but loss uses │ on stale DOA) │ weights (0.0) │
	│ variable lambda │ │ │
	└──────────┬───────────┘ └─────────┬─────────┘
	│ │
	matched_slot matched_slot
	[B, N_gt, T_s] [B, N_gt, T_s]
	│ │
	▼ ▼
	╔══════════════════════╗ ╔═══════════════════╗
	│ LOSS COMPUTATION │ │ METRICS COMPUTATION
	├──────────────────────┤ ├───────────────────┤
	│ loss_activity × │ │ For matched pairs: │
	│ λ_frame_activity │ │ pred_azi_deg │
	│ loss_class × │ │ pred_ele_deg │
	│ λ_frame_class │ │ tgt_azi_deg │
	│ loss_direction × ◄─ v10p1: 0.0 │ tgt_ele_deg │
	│ λ_frame_direction │ │ │
	│ loss_distance × ◄─ v10p1: 0.0 │ azi_mae = MAE │
	│ λ_frame_distance │ │ ele_mae = MAE │
	└──────────┬───────────┘ └─────────┬─────────┘
	│ │
	⚠️ ISSUE #2 │ v9: Full training │
	────────────► │ v10 phase-1: FROZEN dir/dist head │
	│ no gradients! │
	│ │
	▼ ▼
	loss_total azi_mae, ele_mae
	(backprop) (forward only)


	KEY ASYMMETRIES CAUSING TRAIN/VALID GAP
	════════════════════════════════════════════════════════════════════════════════════════════════

	┌─────────────────────────────────┬──────────────────────────┬──────────────┐
	│ Aspect │ Training │ Validation │
	├─────────────────────────────────┼──────────────────────────┼──────────────┤
	│ SpecAugment │ Masking 45% of spectrum │ NO masking │
	│ │ →音频特性不同 │
	├─────────────────────────────────┼──────────────────────────┼──────────────┤
	│ Spatial Augmentation (Rotation) │ NONE │ NONE │
	│ │ → DOA diversity limited │ → Unseen │
	│ │ to ~1 epoch │ angles │
	├─────────────────────────────────┼──────────────────────────┼──────────────┤
	│ Direction Head State (v10p1) │ FROZEN (no gradients) │ FROZEN + │
	│ │ λ=0.0 (no loss signal) │ STALE (from │
	│ │ │ v9 epoch 3)│
	├─────────────────────────────────┼──────────────────────────┼──────────────┤
	│ Matching Cost Weights │ Unweighted sum │ Same unweighted
	│ │ act + cls + dir + dist │ sum, but on │
	│ │ (but loss uses lambda) │ frozen DOA │
	├─────────────────────────────────┼──────────────────────────┼──────────────┤
	│ Gradient Flow │ Through loss (v9) or │ Validation │
	│ │ None (v10p1) │ mode: no │
	│ │ │ gradients │
	└─────────────────────────────────┴──────────────────────────┴──────────────┘


	IMPACT CASCADE (Most Severe Scenario: ov3, v10 Phase-1)
	════════════════════════════════════════════════════════════════════════════════════════════════

	1. Training sees only ~single orientation per azimuth per epoch
	No rotation aug → limited DOA diversity

	├─→ Model sees: (azi=0°, ele=0°), (azi=1°, ele=1°), ...
	└─→ Model learns: limited generalization to unseen angles

	2. Direction head frozen at v10 phase-1 start
	λ_frame_direction = 0.0 → no training signal
	freeze_frame_track_spatial_heads = True → no gradients

	├─→ Head receives: ZERO gradients from loss
	├─→ Head receives: ZERO gradients from backward pass
	└─→ Result: Output completely stale (from v9 epoch 3)

	3. At validation: direction head outputs used to compute metrics
	But outputs are unchanged from 10 epochs ago
	And model hasn't learned ov2/ov3 multi-source DOA at all

	├─→ Validation sees: unseen azimuth combos (not in training)
	├─→ Validation sees: multi-source situations unfamiliar to frozen head
	└─→ Result: 20-30° azimuth errors

	4. SpecAugment makes training acoustic features different
	Masking 45% of W channel during training
	Zero masking at validation

	└─→ Result: +2-5° additional error on top


	FIX #1: Add Rotation Augmentation
	════════════════════════════════════════════════════════════════════════════════════════════════

	Current (BROKEN):
	FOA [W, X, Y, Z] → (no change) → Encoder → DOA predictor

	Fixed (CORRECT):
	FOA [W, X, Y, Z] → Rotate(θ, φ, ψ) → [W', X', Y', Z']
	→ Encoder → DOA predictor

	Labels: (azi, ele, dist) → Rotate_targets(θ, φ, ψ) → (azi', ele', dist)

	Result: Same class, different DOA → model learns generalization


	FIX #2: Make SpecAugment Train/Val Consistent
	════════════════════════════════════════════════════════════════════════════════════════════════

	Current (BROKEN):
	Training: if self.training: apply_masking() ✓
	Validation: if not self.training: skip_masking() ✗ (mismatch!)

	Fixed (OPTION A):
	Training: No masking (simplified, possibly less robust)
	Validation: No masking (consistent)

	Fixed (OPTION B):
	Training: apply_masking() ✓
	Validation: apply_same_masking_with_fixed_seed() ✓ (consistent)


	FIX #3: Don't Fully Freeze Direction Head (v10)
	════════════════════════════════════════════════════════════════════════════════════════════════

	Current v10 (BROKEN):
	λ_frame_direction = 0.0
	freeze_frame_track_spatial_heads = True
	→ Head receives zero signal for 10 epochs

	Fixed (BETTER):
	λ_frame_direction = 0.1 (not zero!)
	freeze_frame_track_spatial_heads = False
	→ Head receives weak signal to fine-tune on multi-source
	→ Prevents complete staleness at v10 unfreezing