duckking032/vital-jepa-v9

Vital JEPA V9

V9 is the frozen audio-analysis foundation for a modular audio-to-Vital system. It encodes stereo, 22.05 kHz, six-second audio into a fused 512-dimensional representation and six named analysis tokens: source, articulation, pitch, effects, spatial, and dynamics.

Architecture

• 512-dimensional encoder, 8 transformer layers, 8 attention heads
• 6 learned analysis tokens with learned fusion into the pooled representation
• 67.4M parameters total; 42.2M trainable and 25.2M in the EMA target encoder
• factor-routed acoustic, pitch, articulation, effect, spatial, and intervention supervision
• V8 representation anchoring during training
• stereo mid/side log-mel input; no waveform normalization

Training data

The run used 212,361 training items from Vital control interventions, generated one-shots, and a validated acoustic pilot corpus. The acoustic corpus contains 5,500 train and 150 fixed evaluation items from NSynth acoustic instruments, GuitarSet, and CC0 percussion. Every acoustic item is stereo, 22.05 kHz, and exactly six seconds.

Selected checkpoint

checkpoints/jepa_v9/selected.pt is step 30,000. Training completed in 3,677 seconds on an NVIDIA RTX PRO 4500 Blackwell.

The 30k checkpoint was selected over 20k and 25k because it had the best overall factor performance: descriptor correlation 0.802, articulation accuracy 71.5%, pitch MAE 0.0373, control-ID accuracy 10.8%, and control-effect cosine 0.192 on the full held-out comparison. The 25k checkpoint had slightly better cross-parent retrieval, so it is included in the comparison report but is not the selected base.

On the fixed 150-item acoustic evaluation, V9 versus V8 achieved:

Metric	V8	V9
Masked loss	0.0455	0.0439
Retrieval top-1	82.7%	82.0%
Retrieval top-5	96.0%	97.3%
MRR@10	0.878	0.885
Articulation accuracy	52.0%	87.3%
Pitch MAE	0.0466	0.0357
Descriptor correlation	0.816	0.768

Frozen probes show that the fused JEPA embedding beats log-mel and random-projection baselines for acoustic descriptor decoding and intervention decoding. The named tokens are only partially specialized: mean absolute inter-token cosine is 0.59, and the effects token does not consistently lead effect-delta probes. Static Vital-control decoding also remains weak (roughly 0.19 mean correlation). Downstream models should use the fused 512-D embedding plus relevant tokens rather than treating individual tokens as fully disentangled ground-truth banks.

Files

• checkpoints/jepa_v9/selected.pt: selected model and optimizer checkpoint
• configs/jepa_v9.toml: complete audio, architecture, loss, and training configuration
• reports/checkpoint_comparison.json: 20k/25k/30k held-out comparison
• reports/acoustic_v8_v9.json: fixed acoustic V8/V9 evaluation
• reports/probe_report.json: frozen pooled/token probes
• logs/train.jsonl: complete training and validation log
• code/: model and training implementation used by the run