Pellav3 / implementation-notes.md
korallll's picture
Add TensorRT runner (trt_pellav3.py) + notes on the RTX 5060 Ti FP32 engine
74d6699 verified
|
Raw History Blame Contribute Delete
8.29 kB

Pellav3 — implementation notes

Plan: speech-spoof-bench/docs/plans/2026-11-09-thirteen-model-batch-arena-submission.md (model #9).

Source

  • HF Sadanie/pellav3-audio-deepfake-detector @ 47b69420803c01de65af264ecdc2c5f35f2184f3 (_ckpt_dl/dl_batch4.py → _ckpt_dl/Sadanie__pellav3-audio-deepfake-detector/). Card: license: mit, no LICENSE file, no paper, training data not described, "1.54 billion parameters, 16 kHz input". Files: pellav3_detector.pt (6.18 GB fp32, sha256 ac6ae020ac5de06beee4f283ad1ea1fb1ae88dd8da19f483a602a032966695ec), pellav3_infer.py (sha256 c38f9a43…ae6e), config.json ({"name": …} only), README.
  • Net (script): two views of each 4 s window — XLS-R 1B (48 layers) on the per-window standardised waveform with a softmax mix of its 49 hidden states → Linear(1280,256) + 2 Res2 blocks; w2v-BERT 2.0 on 80-bin kaldi fbank (stacked 160-d, masked) with a 25-state mix → Linear(1024,256) + 2 Res2 blocks; frames concatenated (512) → 2 shared Res2 blocks → attentive statistics pooling → Linear(1024,1) = logit(fake).
  • pellav3_infer.py read in full (subprocess only for ffmpeg decoding, Hub only in hub_weights()); vendored into _net.py. Byte-identical copy _upstream_pellav3_infer.py (sha256 asserted) is used only by source_parity.py.
  • Backbones: the script itself builds them from configs (Model(Config.from_pretrained(id)), random init) and fills them from the .pt; we use the same configs, pinned and vendored: xlsr1b_config.json = facebook/wav2vec2-xls-r-1b @ 35eaea9a0ed0f97592277d79208e40ab8917d1e3, w2vbert2_config.json = facebook/w2v-bert-2.0 @ da985ba0987f70aaeb84a80f2851cfac8c697a7b (only config.json downloaded, to _ckpt_dl/facebook__*/).
  • Weights: plain state_dict (1647 fp32 tensors, pos_conv already in parametrizations.weight.original0/1 form) → torch.load(weights_only=True) without allow-list. convert_ckpt.py → pellav3_detector.safetensors (same keys/values, asserted; sha256 6a721d66a0b9e1a1011193a66f245a4836dbfc036eb671cd1ff80d6e266d5a73), symlinked here.

transformers version (script pins 4.48.3; we run 5.3.0)

4.48.3 is not available offline; the uv cache has 4.49.0. tf_version_check.py builds both backbones with the Pellav3 weights under 4.49.0 (dependency check stubbed) and under 5.3.0 and compares all hidden states on fixed inputs (XLS-R on a waveform batch with a zero-padded item; w2v-BERT on features with a padded attention mask), CPU FP32: max |Δ| = 0.0 for both (logs_tf_version_check.txt). 4.48.3 → 4.49.0 was not tested directly (no 4.48 wheel offline); the wav2vec2 / wav2vec2_bert eval paths had no numerical changes in that patch range that we know of.

Wrapper (pellav3.py:Pellav3, _net.py)

  • Window: the script scores clips ≤ 4 s once and longer clips with 4 s windows every 2 s (last window ending at the clip end), averaging window logits. Arena 4 s cap: the wrapper scores one window, the grid's first one (decode(path, 0, 4.0) == x[:64000]) through the script's view() (mean removal, 20 Hz 2nd-order Butterworth high-pass by sosfiltfilt on the window, centre crop (no-op), gain to −23 LUFS clipped ±40 dB, clip [−1, 1], zero-pad to 64000). For clips ≤ 4 s (all of J-SPAW_LA) this is exactly the authors' score. Zero-padding (not repeat) is kept because the w2v-BERT branch trims trailing zeros before its fbank, as the authors trained it.
  • Loudness: pyloudnorm is not installed here (see Pellav2 notes for details); the same re-implementation of pyloudnorm.Meter(16000).integrated_loudness is used, with the script's RMS fallback verbatim. Equality with the real package is untested (test_matches_pyloudnorm_if_installed is skipped).
  • Batch invariance (edit 6): in the script's padded batch, F = min(XLS-R frames, batch-max w2v-BERT frames), and the XLS-R frames beyond an item's own w2v-BERT length are not zeroed, so the shared Res2 blocks' dilated convs read them at the mask boundary — a clip's logit depended on its batch-mates (|Δ| up to 7.7e-3 vs one-at-a-time in our smoke test). Detector(batch_invariant=True) zeros those frames; for B = 1 it is a no-op (asserted in tests), so the wrapper equals the authors' one-clip result in any batch (|Δ| 7.6e-6).
  • Precision: the script runs the net under torch.autocast(bfloat16) on CUDA. The wrapper is FP32 (TF32 off), per the batch instruction. Measured: the script's own bf16 score() vs our FP32 on the same single window (clips ≤ 4 s) differs by ≤ 0.18 logit (logits ~±20; bf16 resolution there is 0.125).
  • Score = −logit(fake). Strict load: all 1647 tensors map 1:1 onto Detector.state_dict() (set equality asserted, strict=True). Loaded but unused at inference (the script keeps them "for the weights file"), asserted allow-list UNUSED_AT_INFERENCE: xl.backbone.masked_spec_embed, w2v.backbone.masked_spec_embed (SpecAugment tokens), xl.head.*, w2v.head.*, w2v.att.* (per-branch auxiliary heads/pooling): 37,123 params.
  • Params: 1,544,130,575 total; 1,544,093,452 on the forward path.

Checks

  • test_pellav3.py: first-window/view/window-grid, short-clip pad + RMS fallback, loudness target, reference tone, strict load + allow-list size, param count, eval mode, batch == single, batch_invariant no-op for B = 1, determinism, bona fide > spoof on J-SPAW. 11 passed, 1 skipped (pyloudnorm).
  • source_parity.py (logs_source_parity.txt), authors' Detector + decode (ffmpeg) + view/views/score on FLOAT wavs of 16 J-SPAW_LA clips + 0.3 s + 1.25 s + 6.6 s (3 windows) + 8.6 s (4 windows):
    • preprocessing identical (cap and all windows: max |Δ| 0.0);
    • 4 s cap (wrapper batched FP32 vs script net on view(decode(p, 0, 4)), one clip, FP32): max |Δ| 7.6e-6;
    • window off (wrapper windows() + mean vs script views() + mean, FP32): max |Δ| 0.0;
    • script's own bf16 score() vs wrapper FP32 cap: 1.23 overall (long clips: different windows), 0.18 on ≤ 4 s clips.

Batch sweep (sweep.py, 64000-sample view, 2026-11-10)

card best notes
RTX 4070 Ti SUPER (idx 2) 26.3 utt/s @24 plateau 8–32 → batch_size = 24
RTX 5060 Ti (idx 1) 8.35 utt/s @2 ~7 at bs ≥ 4 → 2 on 5060

Sanity (PyTorch, idx 2)

  • Full J-SPAW_LA: EER 0.000 % (2397 trials, 0 skipped). Logits are saturated and fully separated (about ±20), which suggests possible training-data overlap (undisclosed). Flagged for the maintainer.

TensorRT (trt_pellav3.py, sidecar trt_pellav3.json)

The engine takes (wav[B,64000], feats[B,199,160], mask[B,199]); the fbank stays in torch. The ONNX is 5.9 GB, so it uses the dynamo exporter with external data. On RTX 4070 Ti SUPER, FP16 build Spearman was 0.99945, below 0.9999, so it fell back to FP32: build parity Spearman 1.0, max Δ 3.4e-4, best batch 8, 31.7 utt/s. Outcome (maintainer, 2026-11-10): PyTorch only. The full J-SPAW_LA run with Pellav3TRT skipped all 2397 clips ('Detector' object has no attribute 'w2v': the TRT class drops a submodule the scoring path still uses), and the engine is only ~20% faster than PyTorch on the 4070 (31.7 vs 26.3 utt/s), so the class was not fixed and the 5060 engine build was stopped. All drivers (lanes, EmoSpoofTTS, LRLspoof) use pellav3:Pellav3.

Update 2026-11-11: TensorRT on RTX 5060 Ti (adopted for lanes)

The _features fix made Pellav3TRT score correctly (4070 engine vs PyTorch, 2072 J-SPAW clips: Spearman 1.0, max Δ 1.9e-4). The 4070 was then removed from the machine, and on the 5060 Ti PyTorch runs at only 8.4 utt/s. So a FP32 engine was built for RTX 5060 Ti (TRT_FORCE_FP32=1; the FP16 build parity had failed on the 4070): 20.2 utt/s at batch 8 (2.4x). Build parity on 64 clips: Spearman 0.99918, max Δ 0.41 (FP32 engine, TF32 tactics). Full-dataset parity vs PyTorch: J-SPAW_LA n=2397 Spearman 1.000000, max Δ 2.1e-4, EER 0.000 = 0.000; DeepVoice n=5053 Spearman 1.000000, max Δ 2.6e-4, EER 1.9108 = 1.9108 (identical). Adopted for the remaining lane datasets on 5060 cards (4060 Ti has no engine and stays PyTorch). Cells already submitted were computed in PyTorch; J-SPAW_LA stays submitted from PyTorch.