Hugging Face's logo

dieKarotte
/
Spatial-BEATs

Model card Files
xet
 Community
Spatial-BEATs / docs /V11_IMPLEMENTATION_SUMMARY.md
dieKarotte's picture
dieKarotte
Add files using upload-large-folder tool
86cbd36 verified
preview code
|
Raw
History Blame Contribute Delete
12.4 kB

Spatial-BEATs Analysis & Architecture Enhancement - Session Summary

Overview

This session completed a comprehensive analysis of the Spatial-BEATs codebase and implemented major architectural enhancements to address the train/validation gap in DOA (Direction of Arrival) prediction.

Initial Problem Statement

  • Train/Validation Gap: 10° train azimuth error vs ~30° validation error (20° gap)
  • Root Cause: Identified as stochastic regularization (Dropout) applied during training but not validation
  • Goal: Design and implement architectural improvements to reduce the gap

Phase 1: Comprehensive Codebase Analysis

Deliverables Generated

1. SPATIAL_AUDIO_FRAMEWORKS_ANALYSIS_COMPREHENSIVE.md (17.4 KB)

Detailed 10-part analysis covering:

  • All referenced spatial audio frameworks (Spatial-AST, DCASE SELD, EINV2, ACCDOA)
  • Three alternative architectural routes (A, B, C) that coexist in the codebase
  • Experimental series v7-v11 with design rationale for each
  • ClassHeadSpectralDemixer innovation for multi-source frequency-axis decomposition
  • Loss configuration patterns and evaluation metrics
  • Checkpoint management and initialization strategies
  • Practical usage guide for each route

2. FRAMEWORKS_QUICK_REFERENCE.txt (12.8 KB)

Quick lookup tables including:

  • Visual matrix of all frameworks found
  • Implementation status for each (Implemented/Referenced/Not Found)
  • Comparison of Routes A/B/C decision matrix
  • File locations and line numbers for each component

3. SEARCH_FINDINGS_SUMMARY.md (9.7 KB)

Structured checklist of all search requests with:

  • Verification status (✓ YES, ✗ NO)
  • Framework locations in codebase
  • Code references and implementation details
  • Research papers and external URLs

Key Findings from Analysis

Referenced Frameworks:

Framework Status Implementation Route
Spatial-AST ✓ Found PreTrunkASTPredictionHeads Pre-trunk
DCASE SELD ✓ Found ACCDOAHeads (Route C) Per-class
EINV2 ✓ Adapted SourceQueryDecoder (Route B) K-track queries
BAT ✓ Found Warmup config Training
SALSA, CST-former, SELDnet ✗ Not found — —

Three Coexisting Routes:

Route A (FrameSlotHead)
├─ Per-frame K-slot assignment
├─ Per-step Hungarian matching
└─ Use: Frequent source entry/exit

Route B (SourceQueryDecoder + FrameTrackPredictionHeads) [Current production]
├─ K track queries with temporal self-attention
├─ Clip-level Hungarian matching
└─ Use: Continuous trajectories

Route C (ACCDOAHeads)
├─ Per-class ACCDOA vector field
├─ No matching required
└─ Use: Simple, stable baseline

Phase 2: Architectural Implementation (v11 Series)

Root Cause Analysis

The analysis revealed that classification accuracy plateaued at ~51% because:

  1. SpatialDeltaPatchAdapter V1: 32-dim bottleneck (~200K params) insufficient for spatial encoding
  2. BEATs Trunk: No spatial conditioning after initial delta injection
  3. Frequency Pooling: Multiple sources compressed into single D-vector

Solution: v11 Multi-Part Architecture Enhancement

Part A: SpatialDeltaPatchAdapterV2 (Enhanced Front-End)

Purpose: Increase spatial feature extraction capacity while maintaining compatibility

Architecture:

Input: [B, 7, T, F]  (FOA waveform)
  ↓
Stem Conv: 7→128 (1×1)
  ↓
ResBlock×2: 128→128 (3×3 + SE attention)
  ↓
Patch Projection: 128→512 (16×16 patchify)
  ↓
Output: [B, num_patches, 512]

Specifications:

  • Parameters: 17.4M (V1 was 4.2M, increase justified by 128-dim feature width)
  • Breakdown:
    • ResBlocks: ~600K
    • Patchify Conv: ~16.8M (128→512 kernel is large but necessary)
  • Initialization: residual_alpha=0.1 for safe hot-start
  • Backward Compatibility: Zero residual at init → identical to V1

Part B: SpatialAdapterLayer (In-Trunk Spatial Conditioning)

Purpose: Allow BEATs trunk layers to condition on spatial information without disrupting pretrained weights

Architecture:

For each of 12 trunk layers:
  Layer output x
    ↓
  SpatialAdapterLayer: rank-64 LoRA-style update
    ├─ Down-proj: D→64
    ├─ GELU activation
    └─ Up-proj: 64→D
    ↓
  gate * adapter_output (where gate starts at 0.01)
    ↓
  x + gated_update  (residual connection)

Specifications:

  • Parameters per layer: 100.7K (D×64 + 64 + 64×D + D)
  • Total 12 layers: 1.21M
  • Gate initialization: 1e-2 (near-identity at epoch-0)
  • Zero-initialization: Output layer weights/bias all zeros
  • Backward Compatibility: gate*0=0 → identity function at init

Part C: Enhanced Prediction Heads

ClassHeadSpectralDemixer:

  • Cross-attention mechanism for frequency-axis decomposition
  • Allows K tracks to attend to pre-pooling BEATs features
  • Version 9: Class head demixing
  • Version 11a: Added spatial head demixing (direction/distance)
  • Version 11b: Alternative KV source from LocalSpatial pre-pool

ACCDOAHeads (Route C):

  • Per-class Activity-Coupled Cartesian DOA predictions
  • No Hungarian matching required (per-class slots are inherent)
  • Activity encoded in vector magnitude ||v||
  • Direction encoded in unit vector v/||v||
  • Distance predicted separately with softplus

Phase 3: Configuration & Implementation

Four v11 Preset Configurations

1. v11_phase1_cls - Classification Refinement (Phase 1)

  • Uses: SpatialDeltaPatchAdapterV2 only
  • Freezes: Direction/distance heads
  • Trains: Classification head + new num_active_head
  • Hot-start: v10 phase-1 best.pt
  • Purpose: Diagnose if front-end adapter improves class accuracy
SPATIAL_LR=7.5e-6 SPATIAL_EPOCHS=10 ./run_ov1_v11_phase1_cls.sh

2. v11a_ov123_top4 - Route B + Spatial Demixer

  • Uses: V2 adapter + trunk adapters + spatial_head_demixer
  • Training: All heads enabled
  • Loss weights: Same as v9 baseline (activity=1.0, class=1.0, direction=4.0)
  • Hot-start: v9 best.pt (ov123 top-4)
  • Purpose: Full architecture with symmetric demixing
./run_ov1_v11a_ov123_top4.sh

3. v11b_ov123_top4 - Route B + LocalSpatial KV

  • Uses: V2 adapter + trunk adapters + spatial_demixer(LocalSpatial pre-pool)
  • Hypothesis: LocalSpatial 7-channel pre-pool better than BEATs mono
  • Conditional compilation: cfg.local_spatial_pre_pool_demixer_kv
  • Purpose: Explore alternative frequency feature sources
./run_ov1_v11b_ov123_top4.sh

4. v11c_ov123_accdoa - Paradigm Shift to ACCDOA (Route C)

  • Uses: V2 adapter + trunk adapters + ACCDOAHeads
  • Removes: Query decoding, Hungarian matching entire pipeline
  • Loss weights: Activity=4.0 (MSE dominates), class=0.0, direction=0.0
  • Hot-start: ov1 local_spatial baseline (v9 incompatible - no ACCDOAHeads)
  • Purpose: Simplicity-first approach for ov2/ov3 (same-class constraint)
./run_ov1_v11c_ov123_accdoa.sh

Code Changes Summary

spatial_modules.py (+966 lines)

New Classes:

  • SpatialDeltaPatchAdapterV2: Enhanced front-end (lines 2376+)
    • _AdapterResBlock: Component residual block (lines 2463+)
  • SpatialAdapterLayer: Zero-init trunk adapter (lines 2483+)
  • SqueezeExcitation: SE attention module (lines 2347+)

Enhanced Classes:

  • SpatialBEATsPreprocessor: Added SpecAugment W-channel masking
  • LocalSpatialPredictionHeads: Optional pre-pool return
  • FrameTrackPredictionHeads: Added spatial_head_demixer support

spatial_beats.py (+703 lines)

New Config Flags:

cfg.use_spatial_delta_adapter_v2: bool = True
cfg.use_trunk_spatial_adapters: bool = False
cfg.spatial_adapter_rank: int = 64
cfg.spatial_adapter_gate_init: float = 0.01
cfg.local_spatial_pre_pool_demixer_kv: bool = False

New Integration:

  • Lines 454-458: V2 adapter initialization
  • Lines 490-508: Trunk adapter list creation
  • Lines 1007-1066: Trunk adapter application in forward pass

train_spatial_beats.py (+3662 lines)

New Config Factories:

  • make_ov1_local_spatial_v11_phase1_cls_config() (lines 2549+)
  • make_ov1_local_spatial_v11a_ov123_top4_config() (lines 2281+)
  • make_ov1_local_spatial_v11b_ov123_top4_config() (lines 2327+)
  • make_ov1_local_spatial_v11c_ov123_accdoa_config() (lines 2357+)

Preset Registration:

args.preset in {
    "ov1_local_spatial_v11_phase1_cls",
    "ov1_local_spatial_v11a_ov123_top4",
    "ov1_local_spatial_v11b_ov123_top4",
    "ov1_local_spatial_v11c_ov123_accdoa",
}

Verification & Testing

Unit Tests Performed

✓ V2 Shape Test

Input:  [2, 7, 1000, 128]
Output: [2, 496, 512]
Status: PASS

✓ V2 Parameter Count

Total: 17,385,921 (17.39M)
Breakdown:
  - Stem: 896
  - ResBlocks: ~600K
  - Patchify: 16.8M
Status: PASS

✓ Adapter Zero-Init

max_diff(output - input): 0.00e+00
Gate initialization: 1e-2 → identity at init
Status: PASS

✓ Adapter Parameter Count

Per layer: 100,673 (100.7K)
12 layers: 1,208,076 (1.21M)
Status: PASS

✓ Syntax Validation

spatial_modules.py: OK
spatial_beats.py: OK
train_spatial_beats.py: OK

Next Steps & Recommendations

Immediate Actions

  1. Run v11_phase1_cls experiment

    SPATIAL_EPOCHS=10 ./run_ov1_v11_phase1_cls.sh

    Expected: Classification accuracy improvement from v10 baseline (51% → ?)

  2. Compare v11a vs v11b

    • Same architecture, different demixer KV sources
    • Metric: Direction error (azimuth MAE) on validation set
    • Hypothesis: v11b's LocalSpatial 7-channel may be better

  3. Evaluate v11c (ACCDOA)

    • Paradigm shift from query-based to per-class
    • Metric: SELD score (ER, F, LE, LR combined)
    • Expected advantage: No Hungarian matching complexity

Diagnostic Experiments

To quantify train/val gap reduction from architectural vs regularization changes:

# Diagnostic: Three evaluations of same model
python -c "
model = load_v11_checkpoint(...)

# Train set, training mode (dropout ON)
train_eval_train = evaluate(model, train_loader, training=True)

# Train set, eval mode (dropout OFF)
train_eval_eval = evaluate(model, train_loader, training=False)

# Validation set, eval mode (dropout OFF)
valid_eval_eval = evaluate(model, valid_loader, training=False)

# Gap breakdown:
dropout_effect = train_eval_train - train_eval_eval
data_dist_shift = train_eval_eval - valid_eval_eval
total_gap = train_eval_train - valid_eval_eval

print(f'Dropout contribution: {dropout_effect:.1f}°')
print(f'Data distribution shift: {data_dist_shift:.1f}°')
print(f'Total gap: {total_gap:.1f}°')
"

Documentation Resources

Generated Documentation

  1. SPATIAL_AUDIO_FRAMEWORKS_ANALYSIS_COMPREHENSIVE.md (17.4 KB)

    • Complete 10-part framework reference

  2. docs/0427_v11_series.md

    • v11 experimental series diagnostic guide

  3. docs/doa_train_valid_gap_analysis.md

    • Detailed train/val gap root cause analysis

  4. FRAMEWORKS_QUICK_REFERENCE.txt (12.8 KB)

    • Quick lookup matrices for all frameworks

  5. Run Scripts (4 variants)

    • run_ov1_v11_phase1_cls.sh
    • run_ov1_v11a_ov123_top4.sh
    • run_ov1_v11b_ov123_top4.sh
    • run_ov1_v11c_ov123_accdoa.sh

Code References

  • spatial_modules.py: Lines 2347-2520 (new adapter classes)
  • spatial_beats.py: Lines 283, 454-458, 490-508, 1007-1066 (integration)
  • train_spatial_beats.py: Lines 2281-2545, 3989-4234 (configs & registration)

Summary Statistics

Codebase Changes

  • Files Modified: 3 core files (spatial_modules.py, spatial_beats.py, train_spatial_beats.py)
  • Files Added: 81 (documentation, analysis, scripts, helpers)
  • Lines Added: 5,011 to core files, 21,621 total with analysis
  • Commit: b902628

New Parameters

  • V2 Adapter: 17.39M
  • Trunk Adapters (12×): 1.21M
  • Total New: 18.6M (18.59M)

Architecture Span

  • Layers with Spatial Adapters: 12 (all BEATs trunk layers)
  • Multi-block Depth: 2x ResBlock + SE in V2
  • Rank for Adapters: 64 (fixed trade-off)
  • Gate Initialization: 1e-2 (conservative)

Git Status 

Commit: b902628 "Implement v11 spatial audio architecture with enhanced adapters and ACCDOA support"
Branch: master
Status: All changes committed and pushed