UTCN-XLSR2B-LogReg / implementation-notes.md
korallll's picture
Wrapper code (speech-spoof-bench)
47cc53f verified
|
Raw History Blame Contribute Delete
7.37 kB

UTCN-XLSR2B-LogReg — implementation notes

Plan: speech-spoof-bench/docs/plans/2026-11-09-thirteen-model-batch-arena-submission.md (model #1).

Source

  • HF DavidCombei/deepfake-detection @ 5f39744bc81762529e19f76e2bb7318e7e69f3fb, audio core only (benchmarks/_ckpt_dl/DavidCombei__deepfake-detection/audio-detection_core/): app.py, requirements.txt (transformers 4.44, sklearn 1.3.2), model/wav2vec2-xls-r-2b_truncated/{config.json,model.safetensors} (182 fp32 tensors, 433,051,008 params, layers 0–8 only), model/logreg_margin_pruning_ALL_with_scaler_threshold.joblib (sha256 dbdc1bd8…80dc). No license anywhere on the repo → LICENSE.source; weights linked, not re-hosted. Both are symlinked into this dir.
  • Paper: Combei, Stan, Oneata, Müller, Cucu, "Unmasking real-world audio deepfakes: A data-centric approach", Interspeech 2025, arXiv 2506.09606. Training scripts: github.com/davidcombei/AI4T @ 2c3840cc (CC-BY-4.0; read, not used).

joblib → lr_head.npz (export_lr_head.py, log logs_export_lr_head.txt)

  • pickletools (no execution) shows protocol 4, globals sklearn.linear_model._logistic.LogisticRegression, joblib.numpy_pickle.NumpyArrayWrapper, numpy.ndarray, numpy.dtype; it stops at byte 528 because joblib writes raw array bytes inline after each wrapper. The full global set (found by the allow-list unpickler, which refuses anything else) adds only sklearn.preprocessing._data.MinMaxScaler (the scaler is MinMax, not Standard).
  • Loader: pure-Python pickle._Unpickler subclass; find_class allow-list maps the sklearn classes and the joblib wrapper to inert stubs (no sklearn/joblib code runs), numpy.dtype only for numeric codes; a custom BUILD handler reads each wrapper's array like joblib's non-object path (pad byte + 0xFF padding + nbytes, np.frombuffer). Never joblib.load / pickle.load.
  • Content: tuple (LogisticRegression(C=1e6, lbfgs, classes [0,1], coef (1,1920), intercept −0.14833), MinMaxScaler(feature_range (0,1), clip False, fitted on 430,507 decision values, scale_ 0.0011718, min_ 0.56445), threshold 0.567134). Exported to lr_head.npz (sha256 044bbb7e…6684, deterministic).

Wrapper (utcn_xlsr2b_logreg.py:UTCNXLSR2BLogReg, _net.py)

  • app.py path: AutoFeatureExtractor(facebook/wav2vec2-xls-r-2b) = per-utterance (x−mean)/sqrt(var+1e-7) (config vendored as xlsr2b_preprocessor_config.json, @3b6d89d0) → CustomWav2Vec2Model (48-layer build with encoder.layers[:9]) → hidden_states[9] → mean over time → decision_function → MinMaxScaler.transform; pred = 1 ("real") iff value ≥ thresh − 5e-3 → class 1 = bona fide → score = scaled decision value, higher = bona fide.
  • Note: with only 9 layers, hidden_states[9] is the encoder's final entry = encoder.layer_norm(layer-9 output). The authors' training extractor (AI4T wav2vec2-xls-r-2b_all-layers_extractor.py) used the full 48-layer model, where hidden_states[9] is the raw layer-9 output without that LayerNorm — so app.py's features differ from the training features. We replicate app.py (the released inference). Diagnostic on 300 random In-the-Wild clips (scratch, not shipped): EER 0.0% (whole clip) / 0.71% (4 s cap) for both variants, so app.py's path works.
  • Backbone: Wav2Vec2Model(config, num_hidden_layers=9) + load_state_dict(safetensors, strict=True): all 182 tensors used, 0 missing / 0 unexpected (masked_spec_embed is loaded but unused in eval).
  • Attention: eager. transformers' default SDPA FP32 kernels are run-to-run non-deterministic here (scaled score jitter ~1e-4 per call, ~1e-3 vs eager); eager is bit-reproducible at the same speed (25.3 vs 25.3 utt/s on 5060).
  • Head: float64 numpy X @ coef + intercept, then * scale + min (exactly sklearn's ops; clip=False).
  • Input: Arena 4 s cap — first 64600 samples, shorter clips repeat-padded (np.tile), then normalised (app.py instead scores up to 60 s segments, skips segments < 0.7 s, averages sigmoid outputs). FP32, TF32 off. No resampling.
  • Params: 433,051,008 loaded; 433,049,088 forward (minus masked_spec_embed) + 1,921 LR → meta 433.051009.

Checks

  • Tests (test_utcn_xlsr2b_logreg.py, 9 passed): crop/repeat-pad, normalisation == HF feature extractor, npz sha + head sign, strict load (182 tensors == module state), eval mode + param counts, batch == single (atol 1e-4; real batch of 4 with mixed lengths), determinism, bona fide > spoof on J-SPAW clips.
  • Source parity (source_parity.py, logs_source_parity.txt): app.py's model class + from_pretrained + HF feature extractor + real sklearn LogisticRegression/MinMaxScaler rebuilt from the npz, 18 J-SPAW_LA clips (16 + a 20000 and a 139944-sample clip). Source with eager attention: max |Δ| 0.0 (window off, whole clip) / 2.5e-8 (4 s cap). Source with its default SDPA: 1.24e-3 / 1.06e-3 (SDPA kernel noise, see above).
  • J-SPAW_LA full (PyTorch, RTX 5060 Ti idx 0, run_pt.sh): EER 0.875%, 2397 trials, 0 skipped.

Batch sweep (sweep.py, 64600-sample input, 2026-11-09)

card result batch_size
RTX 4070 Ti SUPER (idx 2) 107.3 utt/s @24, plateau 8–64 24
RTX 5060 Ti (idx 0) flat 24.6–26.3 utt/s for bs 1–64 8

TensorRT (trt_utcn_xlsr2b_logreg.py, sidecar trt_utcn-xlsr2b-logreg.json, TRT 10.16.1.11)

Adapted from ProSDD-ASV19's script: the engine outputs the (B,1920) pooled features; the LR + MinMax head stays in numpy (the TRT class sets self.head). Strict FP32 (FORCE_FP32 and TF32 cleared by default); ONNX 1.65 GB < 2 GB, so the TorchScript exporter suffices. Build parity on 64 In-the-Wild clips.

card precision best batch utt/s build parity max Δ / Spearman J-SPAW EER vs PT Spearman / max Δ
RTX 4070 Ti SUPER (idx 2) FP32 32 120.4 4.2e-7 / 1.0 0.875 (=PT) 1.000000 / 1e-6
RTX 5060 Ti (idx 0) FP32 32 72.5 2.3e-7 / 1.0 0.875 (=PT) 1.000000 / 1e-6
Both accepted ( ΔEER = 0, Spearman ≥ 0.999). Only the 4070 ONNX is kept. Scripts: run_trt_export.sh <gpu>,
run_trt_jspaw.sh <gpu>; compare with cmp_scores.py.

Training data → trained_on

Paper §2.1/§4.1: "ALL" = every sample (incl. test splits) of ASVspoof 2019 LA, FoR for-norm, ~60k random ASVspoof 2021 (Table 1: 22k bona fide / 589k spoof / 100 systems = the DF eval set), TIMIT-TTS, ODSS, MLAAD v5 + 80k M-AILABS bona fide, ASVspoof 5 train+dev; then margin pruning (§4.2). ITW and AI4T were held out of training, but the released configuration (ALL + margin-both pruning) is the one picked by In-the-Wild EER (1.70%, Table 4).

  • trained_on: ASVspoof2019_LA, ASVspoof2021_DF, ODSS, InTheWild (selection, following the Forensics-XLSR-Wild precedent — maintainer may drop it). DECRO is flagged automatically via its ASVspoof2019-LA:eval bona fide.
  • Not flagged: ASVspoof2021_LA (paper's ASV21 counts match DF), ASVspoof5 (train/dev only; Arena = eval), XMAD (M-AILABS bona fide is a separate corpus in provenance), FoR/TIMIT-TTS (not in the Arena).
  • Suggested provenance.yaml models entry (not added; maintainer): utcn-xlsr2b-logreg: {backbone: XLS-R, trained_on: [ASVspoof2019-LA, "ASVspoof2021-DF:eval", "ASVspoof5:train", "ASVspoof5:dev", ODSS, MLAAD, M-AILABS, InTheWild]}.