|
Download implementation-notes.md from SpeechAntiSpoofingBenchmarks/Pellav3: direct link, hf CLI and curl.
- Browser
- Download file 8.29 kB
-
https://huggingface.co/SpeechAntiSpoofingBenchmarks/Pellav3/resolve/main/implementation-notes.md
- Command line
-
hf download hf://SpeechAntiSpoofingBenchmarks/Pellav3/implementation-notes.md
-
curl -L -o implementation-notes.md https://huggingface.co/SpeechAntiSpoofingBenchmarks/Pellav3/resolve/main/implementation-notes.md
8.29 kB
| # Pellav3 — implementation notes | |
| Plan: `speech-spoof-bench/docs/plans/2026-11-09-thirteen-model-batch-arena-submission.md` (model #9). | |
| ## Source | |
| - HF `Sadanie/pellav3-audio-deepfake-detector` @ `47b69420803c01de65af264ecdc2c5f35f2184f3` (`_ckpt_dl/dl_batch4.py` → | |
| `_ckpt_dl/Sadanie__pellav3-audio-deepfake-detector/`). Card: `license: mit`, no LICENSE file, no paper, training | |
| data not described, "1.54 billion parameters, 16 kHz input". Files: `pellav3_detector.pt` (6.18 GB fp32, sha256 | |
| `ac6ae020ac5de06beee4f283ad1ea1fb1ae88dd8da19f483a602a032966695ec`), `pellav3_infer.py` (sha256 `c38f9a43…ae6e`), | |
| `config.json` (`{"name": …}` only), README. | |
| - Net (script): two views of each 4 s window — **XLS-R 1B** (48 layers) on the per-window standardised waveform with a | |
| softmax mix of its 49 hidden states → Linear(1280,256) + 2 Res2 blocks; **w2v-BERT 2.0** on 80-bin kaldi fbank | |
| (stacked 160-d, masked) with a 25-state mix → Linear(1024,256) + 2 Res2 blocks; frames concatenated (512) → 2 shared | |
| Res2 blocks → attentive statistics pooling → Linear(1024,1) = logit(fake). | |
| - `pellav3_infer.py` read in full (subprocess only for `ffmpeg` decoding, Hub only in `hub_weights()`); vendored into | |
| `_net.py`. Byte-identical copy `_upstream_pellav3_infer.py` (sha256 asserted) is used only by `source_parity.py`. | |
| - Backbones: the script itself builds them from configs (`Model(Config.from_pretrained(id))`, random init) and fills | |
| them from the .pt; we use the same configs, pinned and vendored: `xlsr1b_config.json` = facebook/wav2vec2-xls-r-1b @ | |
| `35eaea9a0ed0f97592277d79208e40ab8917d1e3`, `w2vbert2_config.json` = facebook/w2v-bert-2.0 @ | |
| `da985ba0987f70aaeb84a80f2851cfac8c697a7b` (only config.json downloaded, to `_ckpt_dl/facebook__*/`). | |
| - Weights: plain state_dict (1647 fp32 tensors, pos_conv already in `parametrizations.weight.original0/1` form) → | |
| `torch.load(weights_only=True)` without allow-list. `convert_ckpt.py` → `pellav3_detector.safetensors` (same | |
| keys/values, asserted; sha256 `6a721d66a0b9e1a1011193a66f245a4836dbfc036eb671cd1ff80d6e266d5a73`), symlinked here. | |
| ## transformers version (script pins 4.48.3; we run 5.3.0) | |
| 4.48.3 is not available offline; the uv cache has **4.49.0**. `tf_version_check.py` builds both backbones with the | |
| Pellav3 weights under 4.49.0 (dependency check stubbed) and under 5.3.0 and compares all hidden states on fixed inputs | |
| (XLS-R on a waveform batch with a zero-padded item; w2v-BERT on features with a padded attention mask), CPU FP32: | |
| **max |Δ| = 0.0 for both** (`logs_tf_version_check.txt`). 4.48.3 → 4.49.0 was not tested directly (no 4.48 wheel | |
| offline); the wav2vec2 / wav2vec2_bert eval paths had no numerical changes in that patch range that we know of. | |
| ## Wrapper (`pellav3.py:Pellav3`, `_net.py`) | |
| - **Window**: the script scores clips ≤ 4 s once and longer clips with 4 s windows every 2 s (last window ending at the | |
| clip end), averaging window logits. Arena 4 s cap: the wrapper scores **one window, the grid's first one** | |
| (`decode(path, 0, 4.0)` == `x[:64000]`) through the script's `view()` (mean removal, 20 Hz 2nd-order Butterworth | |
| high-pass by `sosfiltfilt` on the window, centre crop (no-op), gain to −23 LUFS clipped ±40 dB, clip [−1, 1], | |
| zero-pad to 64000). For clips ≤ 4 s (all of J-SPAW_LA) this is exactly the authors' score. Zero-padding (not | |
| repeat) is kept because the w2v-BERT branch trims trailing zeros before its fbank, as the authors trained it. | |
| - **Loudness**: pyloudnorm is not installed here (see Pellav2 notes for details); the same re-implementation of | |
| `pyloudnorm.Meter(16000).integrated_loudness` is used, with the script's RMS fallback verbatim. Equality with the real | |
| package is untested (`test_matches_pyloudnorm_if_installed` is skipped). | |
| - **Batch invariance (edit 6)**: in the script's padded batch, F = min(XLS-R frames, *batch-max* w2v-BERT frames), and | |
| the XLS-R frames beyond an item's own w2v-BERT length are not zeroed, so the shared Res2 blocks' dilated convs read | |
| them at the mask boundary — a clip's logit depended on its batch-mates (|Δ| up to 7.7e-3 vs one-at-a-time in our | |
| smoke test). `Detector(batch_invariant=True)` zeros those frames; for B = 1 it is a no-op (asserted in tests), so the | |
| wrapper equals the authors' one-clip result in any batch (|Δ| 7.6e-6). | |
| - **Precision**: the script runs the net under `torch.autocast(bfloat16)` on CUDA. The wrapper is FP32 (TF32 off), per | |
| the batch instruction. Measured: the script's own bf16 `score()` vs our FP32 on the same single window (clips ≤ 4 s) | |
| differs by ≤ 0.18 logit (logits ~±20; bf16 resolution there is 0.125). | |
| - Score = −logit(fake). Strict load: all 1647 tensors map 1:1 onto `Detector.state_dict()` (set equality asserted, | |
| `strict=True`). Loaded but unused at inference (the script keeps them "for the weights file"), asserted allow-list | |
| `UNUSED_AT_INFERENCE`: `xl.backbone.masked_spec_embed`, `w2v.backbone.masked_spec_embed` (SpecAugment tokens), | |
| `xl.head.*`, `w2v.head.*`, `w2v.att.*` (per-branch auxiliary heads/pooling): 37,123 params. | |
| - Params: 1,544,130,575 total; 1,544,093,452 on the forward path. | |
| ## Checks | |
| - `test_pellav3.py`: first-window/view/window-grid, short-clip pad + RMS fallback, loudness target, reference tone, | |
| strict load + allow-list size, param count, eval mode, batch == single, batch_invariant no-op for B = 1, | |
| determinism, bona fide > spoof on J-SPAW. 11 passed, 1 skipped (pyloudnorm). | |
| - `source_parity.py` (`logs_source_parity.txt`), authors' Detector + decode (ffmpeg) + view/views/score on FLOAT wavs | |
| of 16 J-SPAW_LA clips + 0.3 s + 1.25 s + 6.6 s (3 windows) + 8.6 s (4 windows): | |
| - preprocessing identical (cap and all windows: max |Δ| 0.0); | |
| - **4 s cap** (wrapper batched FP32 vs script net on `view(decode(p, 0, 4))`, one clip, FP32): max |Δ| 7.6e-6; | |
| - **window off** (wrapper `windows()` + mean vs script `views()` + mean, FP32): max |Δ| 0.0; | |
| - script's own bf16 `score()` vs wrapper FP32 cap: 1.23 overall (long clips: different windows), 0.18 on ≤ 4 s clips. | |
| ## Batch sweep (`sweep.py`, 64000-sample view, 2026-11-10) | |
| | card | best | notes | | |
| |---|---|---| | |
| | RTX 4070 Ti SUPER (idx 2) | 26.3 utt/s @24 | plateau 8–32 → `batch_size = 24` | | |
| | RTX 5060 Ti (idx 1) | 8.35 utt/s @2 | ~7 at bs ≥ 4 → 2 on 5060 | | |
| ## Sanity (PyTorch, idx 2) | |
| - Full J-SPAW_LA: **EER 0.000 %** (2397 trials, 0 skipped). Logits are saturated and fully separated (about ±20), | |
| which suggests possible training-data overlap (undisclosed). Flagged for the maintainer. | |
| ## TensorRT (`trt_pellav3.py`, sidecar `trt_pellav3.json`) | |
| The engine takes (wav[B,64000], feats[B,199,160], mask[B,199]); the fbank stays in torch. The ONNX is 5.9 GB, so it | |
| uses the dynamo exporter with external data. On RTX 4070 Ti SUPER, FP16 build Spearman was 0.99945, below 0.9999, so | |
| it fell back to **FP32**: build parity Spearman 1.0, max Δ 3.4e-4, best batch 8, 31.7 utt/s. | |
| Outcome (maintainer, 2026-11-10): **PyTorch only.** The full J-SPAW_LA run with `Pellav3TRT` skipped all 2397 clips | |
| (`'Detector' object has no attribute 'w2v'`: the TRT class drops a submodule the scoring path still uses), and the | |
| engine is only ~20% faster than PyTorch on the 4070 (31.7 vs 26.3 utt/s), so the class was not fixed and the 5060 | |
| engine build was stopped. All drivers (lanes, EmoSpoofTTS, LRLspoof) use `pellav3:Pellav3`. | |
| ### Update 2026-11-11: TensorRT on RTX 5060 Ti (adopted for lanes) | |
| The `_features` fix made `Pellav3TRT` score correctly (4070 engine vs PyTorch, 2072 J-SPAW clips: Spearman 1.0, max Δ 1.9e-4). | |
| The 4070 was then removed from the machine, and on the 5060 Ti PyTorch runs at only ~8.4 utt/s. So a FP32 engine was built | |
| for RTX 5060 Ti (`TRT_FORCE_FP32=1`; the FP16 build parity had failed on the 4070): 20.2 utt/s at batch 8 (~2.4x). | |
| Build parity on 64 clips: Spearman 0.99918, max Δ 0.41 (FP32 engine, TF32 tactics). Full-dataset parity vs PyTorch: | |
| J-SPAW_LA n=2397 Spearman 1.000000, max Δ 2.1e-4, EER 0.000 = 0.000; DeepVoice n=5053 Spearman 1.000000, | |
| max Δ 2.6e-4, EER 1.9108 = 1.9108 (identical). Adopted for the remaining lane datasets on 5060 cards (4060 Ti has no | |
| engine and stays PyTorch). Cells already submitted were computed in PyTorch; J-SPAW_LA stays submitted from PyTorch. | |