|
Download implementation-notes.md from SpeechAntiSpoofingBenchmarks/Pellav2: direct link, hf CLI and curl.
- Browser
- Download file 6.82 kB
-
https://huggingface.co/SpeechAntiSpoofingBenchmarks/Pellav2/resolve/main/implementation-notes.md
- Command line
-
hf download hf://SpeechAntiSpoofingBenchmarks/Pellav2/implementation-notes.md
-
curl -L -o implementation-notes.md https://huggingface.co/SpeechAntiSpoofingBenchmarks/Pellav2/resolve/main/implementation-notes.md
6.82 kB
| # Pellav2 β implementation notes | |
| Plan: `speech-spoof-bench/docs/plans/2026-11-09-thirteen-model-batch-arena-submission.md` (model #8). | |
| ## Source | |
| - HF `Sadanie/pellav2-audio-deepfake-detector` @ `b668eb4dfe4719665692c73c1d70383680afff6b` (downloaded by | |
| `_ckpt_dl/dl_batch4.py` into `_ckpt_dl/Sadanie__pellav2-audio-deepfake-detector/`). Card: `license: mit`, no | |
| LICENSE file, no paper, training data not described. Files: `pellav2_detector.pt` (2.32 GB, sha256 | |
| `d21aedfb52e2f87706c2753715a29665482b0f4a034800c31579df5d458559ac`), `pellav2_infer.py` (sha256 | |
| `bf13cc97β¦6a67`), `config.json` (`{"name": "pellav2"}` only), README. | |
| - **Architecture deviates from the plan row**: the plan says "XLS-R + learned weights over all 25 layers"; the | |
| script is **w2v-BERT 2.0** (`facebook/w2v-bert-2.0`, 24 conformer layers, 580 M params) on 80-bin kaldi fbank | |
| features (stacked to 160-d), softmax layer mix over the 25 hidden states, masked mean, `Linear(1024,1)`. | |
| Layer mix + 25 layers as in the plan; only the backbone name differs. | |
| - `pellav2_infer.py` read in full (it only imports torch/transformers/torchaudio/scipy/soundfile/pyloudnorm and | |
| downloads from the Hub in `hub_weights()`); vendored into `_net.py`, never imported from the cache. A | |
| byte-identical copy `_upstream_pellav2_infer.py` (sha256 asserted) is used only by `source_parity.py`. | |
| - Backbone: the script calls `Wav2Vec2BertModel.from_pretrained("facebook/w2v-bert-2.0")` (unpinned) and then | |
| overwrites every tensor with the .pt. We build `Wav2Vec2BertModel(config)` from that repo's `config.json` only | |
| (pinned @ `da985ba0987f70aaeb84a80f2851cfac8c697a7b`, downloaded to `_ckpt_dl/facebook__w2v-bert-2.0/`, vendored | |
| as `w2vbert2_config.json`); no backbone weights are downloaded. | |
| - Weights: plain `OrderedDict` state_dict (776 fp32 tensors) β `torch.load(weights_only=True)` works without an | |
| allow-list. `convert_ckpt.py` β `pellav2_detector.safetensors` (same keys/values, round-trip asserted; sha256 | |
| `050234557deabc95d9697a2ede42a5152ed06ba500a4488a42542ab793af428f`), written next to the download, symlinked here. | |
| ## Wrapper (`pellav2.py:Pellav2`, `_net.py`) | |
| - Preprocessing = the script's `load_clip` after decoding (the runner hands 16 kHz mono float32, so the resample | |
| branch is a no-op): mean removal β 2nd-order 20 Hz Butterworth high-pass (`sosfiltfilt`, whole clip, when | |
| len > 480) β **centre 4 s (64000) crop** β gain to β23 LUFS (clipped Β±40 dB) β clip to [β1, 1] β zero-pad to | |
| 64000. The script's own window is already a 4 s crop, so it is kept as is (no Arena first-64600/repeat-pad | |
| window; that would change both the crop position and the padding the fbank trim relies on). | |
| - Features as the script: fbank on the trailing-zero-trimmed clip (β₯ 400 samples), Γ32768, per-utterance | |
| mean/var norm, frame pairs stacked to 160-d, padded batch + attention mask (so a batch = one-at-a-time up to | |
| float noise; the backbone masks attention and conformer convs). | |
| - **pyloudnorm is not installed** in this environment (not in conda, no wheel in the pip/uv caches). Without it the | |
| script silently falls back to an RMS-based level (its `except: pass`), so running the script here would NOT be | |
| the authors' intended path. `_net.integrated_loudness` re-implements `pyloudnorm.Meter(16000).integrated_loudness` | |
| (pyloudnorm 0.1.1: cookbook K-weighting high-shelf G=4 dB Q=1/β2 1500 Hz + high-pass Q=0.5 38 Hz via `lfilter`, | |
| 400 ms blocks with 75 % overlap, β70 LUFS absolute + β10 LU relative gating, same op order). Sanity: a 0 dBFS | |
| 997 Hz mono sine gives β3.083 LUFS at 16 kHz (BS.1770's 48 kHz reference β3.01; pyloudnorm's bilinear-warped | |
| cookbook filters at 16 kHz account for the 0.07 dB), and with 48 kHz coefficients a β23 dBFS-RMS sine reads | |
| β23.04. **Equality with the real package is not proven here**: `test_matches_pyloudnorm_if_installed` does it | |
| (skipped now). To close it: `pip install pyloudnorm` (pure Python) and run the tests. A residual difference would | |
| only be a per-clip gain offset of βͺ 1 dB. | |
| - Score = βlogit(fake) (script: `p_fake = sigmoid(logit)`). FP32, TF32 off (cuDNN + matmul). | |
| - Strict load: all 776 checkpoint tensors map 1:1 onto `Detector.state_dict()` (asserted set equality + | |
| `strict=True`); `backbone.masked_spec_embed` (SpecAugment token) is loaded but unused at inference. | |
| - Params: 580,494,170 total (all checkpoint tensors); 580,493,146 on the forward path (minus masked_spec_embed). | |
| ## Checks | |
| - `test_pellav2.py`: crop/pad, RMS fallback, loudness target, BS.1770 reference tone, strict load, param count, | |
| eval mode, batch == single (atol 1e-3), determinism, bona fide > spoof on J-SPAW. 9 passed, 1 skipped (pyloudnorm). | |
| - `source_parity.py` (log `logs_source_parity.txt`): the authors' `Detector` + `load_clip` (from_pretrained swapped | |
| for the pinned config, .pt via weights_only strict, pyloudnorm shimmed to `_net.integrated_loudness`, clips as | |
| FLOAT wavs) one clip at a time vs the wrapper batched, on 16 J-SPAW_LA clips + 0.3 s (RMS-fallback branch) + | |
| 1.25 s + 8.6 s (centre-crop branch): preprocessing identical (max |Ξ| 0.0), **max |Ξscore| = 2.8e-6**. | |
| The script has no full-utterance mode, so "window off" = "4 s cap" here. | |
| ## Batch sweep (`sweep.py`, 64000-sample view, 2026-11-10) | |
| | card | best | notes | | |
| |---|---|---| | |
| | RTX 4070 Ti SUPER (idx 2) | 56.9 utt/s @24 | plateau 16β64 β `batch_size = 24` | | |
| | RTX 5060 Ti (idx 1) | 25.1 utt/s @4 | ~18 at bs β₯ 8 β 4 on 5060 | | |
| ## Sanity (PyTorch, idx 2) | |
| - Full J-SPAW_LA: **EER 0.000 %** (2397 trials, 0 skipped). Scores are saturated and fully separated (β β7.5 / +7.2). | |
| This is not inverted and not random, but a perfect score points to J-SPAW (or the same source) in the undisclosed | |
| training data. No trained_on entry, because the training data is unknown. Flagged for the maintainer. | |
| ## TensorRT (`trt_pellav2.py`, sidecar `trt_pellav2.json`, TRT 10.16.1.11) | |
| The engine takes (feats[B,199,160], mask[B,199]). The fbank front-end stays in torch, padded to a fixed 199 frames. | |
| The fp32 ONNX is 2.33 GB, so it uses the dynamo exporter with external data (`pellav2.onnx` + `.data`). FP16 build | |
| parity on 64 InTheWild clips gave Spearman 0.99954 (5060) and 0.99977 (4070), below 0.9999, so both cards fall back | |
| to **FP32** (TF32 cleared). | |
| | card | precision | best batch | utt/s | build Spearman / max Ξ | J-SPAW EER | J-SPAW Spearman vs PT / max Ξ | | |
| |---|---|---|---|---|---|---| | |
| | RTX 4070 Ti SUPER (idx 2) | FP32 | 12 | 68.4 | 1.0 / 7.6e-6 | 0.000 (= PT) | 1.000000 / 1.0e-5 | | |
| | RTX 5060 Ti (idx 1) | FP32 | 16 | 49.7 | 1.0 / 2.9e-6 | 0.000 (= PT) | 1.000000 / 1.4e-5 | | |
| Both cards are accepted. Scripts: `run_pt.sh`, `run_trt_export.sh <2|1>` (run idx 2 first, it writes the ONNX), | |
| `run_trt_jspaw.sh <2|1>`, `cmp_scores.py`. | |