Pellav2 / implementation-notes.md
korallll's picture
Wrapper code (speech-spoof-bench)
6f93a12 verified
|
Raw History Blame Contribute Delete
6.82 kB

Pellav2 β€” implementation notes

Plan: speech-spoof-bench/docs/plans/2026-11-09-thirteen-model-batch-arena-submission.md (model #8).

Source

  • HF Sadanie/pellav2-audio-deepfake-detector @ b668eb4dfe4719665692c73c1d70383680afff6b (downloaded by _ckpt_dl/dl_batch4.py into _ckpt_dl/Sadanie__pellav2-audio-deepfake-detector/). Card: license: mit, no LICENSE file, no paper, training data not described. Files: pellav2_detector.pt (2.32 GB, sha256 d21aedfb52e2f87706c2753715a29665482b0f4a034800c31579df5d458559ac), pellav2_infer.py (sha256 bf13cc97…6a67), config.json ({"name": "pellav2"} only), README.
  • Architecture deviates from the plan row: the plan says "XLS-R + learned weights over all 25 layers"; the script is w2v-BERT 2.0 (facebook/w2v-bert-2.0, 24 conformer layers, 580 M params) on 80-bin kaldi fbank features (stacked to 160-d), softmax layer mix over the 25 hidden states, masked mean, Linear(1024,1). Layer mix + 25 layers as in the plan; only the backbone name differs.
  • pellav2_infer.py read in full (it only imports torch/transformers/torchaudio/scipy/soundfile/pyloudnorm and downloads from the Hub in hub_weights()); vendored into _net.py, never imported from the cache. A byte-identical copy _upstream_pellav2_infer.py (sha256 asserted) is used only by source_parity.py.
  • Backbone: the script calls Wav2Vec2BertModel.from_pretrained("facebook/w2v-bert-2.0") (unpinned) and then overwrites every tensor with the .pt. We build Wav2Vec2BertModel(config) from that repo's config.json only (pinned @ da985ba0987f70aaeb84a80f2851cfac8c697a7b, downloaded to _ckpt_dl/facebook__w2v-bert-2.0/, vendored as w2vbert2_config.json); no backbone weights are downloaded.
  • Weights: plain OrderedDict state_dict (776 fp32 tensors) β†’ torch.load(weights_only=True) works without an allow-list. convert_ckpt.py β†’ pellav2_detector.safetensors (same keys/values, round-trip asserted; sha256 050234557deabc95d9697a2ede42a5152ed06ba500a4488a42542ab793af428f), written next to the download, symlinked here.

Wrapper (pellav2.py:Pellav2, _net.py)

  • Preprocessing = the script's load_clip after decoding (the runner hands 16 kHz mono float32, so the resample branch is a no-op): mean removal β†’ 2nd-order 20 Hz Butterworth high-pass (sosfiltfilt, whole clip, when len > 480) β†’ centre 4 s (64000) crop β†’ gain to βˆ’23 LUFS (clipped Β±40 dB) β†’ clip to [βˆ’1, 1] β†’ zero-pad to
    1. The script's own window is already a 4 s crop, so it is kept as is (no Arena first-64600/repeat-pad window; that would change both the crop position and the padding the fbank trim relies on).
  • Features as the script: fbank on the trailing-zero-trimmed clip (β‰₯ 400 samples), Γ—32768, per-utterance mean/var norm, frame pairs stacked to 160-d, padded batch + attention mask (so a batch = one-at-a-time up to float noise; the backbone masks attention and conformer convs).
  • pyloudnorm is not installed in this environment (not in conda, no wheel in the pip/uv caches). Without it the script silently falls back to an RMS-based level (its except: pass), so running the script here would NOT be the authors' intended path. _net.integrated_loudness re-implements pyloudnorm.Meter(16000).integrated_loudness (pyloudnorm 0.1.1: cookbook K-weighting high-shelf G=4 dB Q=1/√2 1500 Hz + high-pass Q=0.5 38 Hz via lfilter, 400 ms blocks with 75 % overlap, βˆ’70 LUFS absolute + βˆ’10 LU relative gating, same op order). Sanity: a 0 dBFS 997 Hz mono sine gives βˆ’3.083 LUFS at 16 kHz (BS.1770's 48 kHz reference βˆ’3.01; pyloudnorm's bilinear-warped cookbook filters at 16 kHz account for the 0.07 dB), and with 48 kHz coefficients a βˆ’23 dBFS-RMS sine reads βˆ’23.04. Equality with the real package is not proven here: test_matches_pyloudnorm_if_installed does it (skipped now). To close it: pip install pyloudnorm (pure Python) and run the tests. A residual difference would only be a per-clip gain offset of β‰ͺ 1 dB.
  • Score = βˆ’logit(fake) (script: p_fake = sigmoid(logit)). FP32, TF32 off (cuDNN + matmul).
  • Strict load: all 776 checkpoint tensors map 1:1 onto Detector.state_dict() (asserted set equality + strict=True); backbone.masked_spec_embed (SpecAugment token) is loaded but unused at inference.
  • Params: 580,494,170 total (all checkpoint tensors); 580,493,146 on the forward path (minus masked_spec_embed).

Checks

  • test_pellav2.py: crop/pad, RMS fallback, loudness target, BS.1770 reference tone, strict load, param count, eval mode, batch == single (atol 1e-3), determinism, bona fide > spoof on J-SPAW. 9 passed, 1 skipped (pyloudnorm).
  • source_parity.py (log logs_source_parity.txt): the authors' Detector + load_clip (from_pretrained swapped for the pinned config, .pt via weights_only strict, pyloudnorm shimmed to _net.integrated_loudness, clips as FLOAT wavs) one clip at a time vs the wrapper batched, on 16 J-SPAW_LA clips + 0.3 s (RMS-fallback branch) + 1.25 s + 8.6 s (centre-crop branch): preprocessing identical (max |Ξ”| 0.0), max |Ξ”score| = 2.8e-6. The script has no full-utterance mode, so "window off" = "4 s cap" here.

Batch sweep (sweep.py, 64000-sample view, 2026-11-10)

card best notes
RTX 4070 Ti SUPER (idx 2) 56.9 utt/s @24 plateau 16–64 β†’ batch_size = 24
RTX 5060 Ti (idx 1) 25.1 utt/s @4 ~18 at bs β‰₯ 8 β†’ 4 on 5060

Sanity (PyTorch, idx 2)

  • Full J-SPAW_LA: EER 0.000 % (2397 trials, 0 skipped). Scores are saturated and fully separated (β‰ˆ βˆ’7.5 / +7.2). This is not inverted and not random, but a perfect score points to J-SPAW (or the same source) in the undisclosed training data. No trained_on entry, because the training data is unknown. Flagged for the maintainer.

TensorRT (trt_pellav2.py, sidecar trt_pellav2.json, TRT 10.16.1.11)

The engine takes (feats[B,199,160], mask[B,199]). The fbank front-end stays in torch, padded to a fixed 199 frames. The fp32 ONNX is 2.33 GB, so it uses the dynamo exporter with external data (pellav2.onnx + .data). FP16 build parity on 64 InTheWild clips gave Spearman 0.99954 (5060) and 0.99977 (4070), below 0.9999, so both cards fall back to FP32 (TF32 cleared).

card precision best batch utt/s build Spearman / max Ξ” J-SPAW EER J-SPAW Spearman vs PT / max Ξ”
RTX 4070 Ti SUPER (idx 2) FP32 12 68.4 1.0 / 7.6e-6 0.000 (= PT) 1.000000 / 1.0e-5
RTX 5060 Ti (idx 1) FP32 16 49.7 1.0 / 2.9e-6 0.000 (= PT) 1.000000 / 1.4e-5
Both cards are accepted. Scripts: run_pt.sh, `run_trt_export.sh <2 1>` (run idx 2 first, it writes the ONNX),
`run_trt_jspaw.sh <2 1>, cmp_scores.py`.