File size: 7,370 Bytes
47cc53f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 | # UTCN-XLSR2B-LogReg — implementation notes
Plan: `speech-spoof-bench/docs/plans/2026-11-09-thirteen-model-batch-arena-submission.md` (model #1).
## Source
- HF `DavidCombei/deepfake-detection` @ `5f39744bc81762529e19f76e2bb7318e7e69f3fb`, audio core only
(`benchmarks/_ckpt_dl/DavidCombei__deepfake-detection/audio-detection_core/`): `app.py`, `requirements.txt`
(transformers 4.44, sklearn 1.3.2), `model/wav2vec2-xls-r-2b_truncated/{config.json,model.safetensors}` (182 fp32
tensors, 433,051,008 params, layers 0–8 only), `model/logreg_margin_pruning_ALL_with_scaler_threshold.joblib`
(sha256 `dbdc1bd8…80dc`). No license anywhere on the repo → `LICENSE.source`; weights linked, not re-hosted.
Both are symlinked into this dir.
- Paper: Combei, Stan, Oneata, Müller, Cucu, "Unmasking real-world audio deepfakes: A data-centric approach",
Interspeech 2025, arXiv 2506.09606. Training scripts: github.com/davidcombei/AI4T @ `2c3840cc` (CC-BY-4.0; read, not used).
## joblib → lr_head.npz (`export_lr_head.py`, log `logs_export_lr_head.txt`)
- `pickletools` (no execution) shows protocol 4, globals `sklearn.linear_model._logistic.LogisticRegression`,
`joblib.numpy_pickle.NumpyArrayWrapper`, `numpy.ndarray`, `numpy.dtype`; it stops at byte 528 because joblib writes
raw array bytes inline after each wrapper. The full global set (found by the allow-list unpickler, which refuses
anything else) adds only `sklearn.preprocessing._data.MinMaxScaler` (the scaler is MinMax, not Standard).
- Loader: pure-Python `pickle._Unpickler` subclass; `find_class` allow-list maps the sklearn classes and the joblib
wrapper to inert stubs (no sklearn/joblib code runs), `numpy.dtype` only for numeric codes; a custom BUILD handler
reads each wrapper's array like joblib's non-object path (pad byte + 0xFF padding + nbytes, `np.frombuffer`).
Never `joblib.load` / `pickle.load`.
- Content: tuple `(LogisticRegression(C=1e6, lbfgs, classes [0,1], coef (1,1920), intercept −0.14833),
MinMaxScaler(feature_range (0,1), clip False, fitted on 430,507 decision values, scale_ 0.0011718, min_ 0.56445),
threshold 0.567134)`. Exported to `lr_head.npz` (sha256 `044bbb7e…6684`, deterministic).
## Wrapper (`utcn_xlsr2b_logreg.py:UTCNXLSR2BLogReg`, `_net.py`)
- app.py path: `AutoFeatureExtractor(facebook/wav2vec2-xls-r-2b)` = per-utterance (x−mean)/sqrt(var+1e-7)
(config vendored as `xlsr2b_preprocessor_config.json`, @3b6d89d0) → `CustomWav2Vec2Model` (48-layer build with
`encoder.layers[:9]`) → `hidden_states[9]` → mean over time → `decision_function` → `MinMaxScaler.transform`;
`pred = 1 ("real") iff value ≥ thresh − 5e-3` → class 1 = bona fide → **score = scaled decision value, higher = bona fide**.
- Note: with only 9 layers, `hidden_states[9]` is the encoder's final entry = `encoder.layer_norm(layer-9 output)`.
The authors' training extractor (AI4T `wav2vec2-xls-r-2b_all-layers_extractor.py`) used the full 48-layer model,
where `hidden_states[9]` is the raw layer-9 output **without** that LayerNorm — so app.py's features differ from
the training features. We replicate app.py (the released inference). Diagnostic on 300 random In-the-Wild clips
(scratch, not shipped): EER 0.0% (whole clip) / 0.71% (4 s cap) for both variants, so app.py's path works.
- Backbone: `Wav2Vec2Model(config, num_hidden_layers=9)` + `load_state_dict(safetensors, strict=True)`: all 182
tensors used, 0 missing / 0 unexpected (`masked_spec_embed` is loaded but unused in eval).
- Attention: **eager**. transformers' default SDPA FP32 kernels are run-to-run non-deterministic here (scaled score
jitter ~1e-4 per call, ~1e-3 vs eager); eager is bit-reproducible at the same speed (25.3 vs 25.3 utt/s on 5060).
- Head: float64 numpy `X @ coef + intercept`, then `* scale + min` (exactly sklearn's ops; clip=False).
- Input: Arena 4 s cap — first 64600 samples, shorter clips repeat-padded (np.tile), then normalised (app.py instead
scores up to 60 s segments, skips segments < 0.7 s, averages sigmoid outputs). FP32, TF32 off. No resampling.
- Params: 433,051,008 loaded; 433,049,088 forward (minus masked_spec_embed) + 1,921 LR → meta 433.051009.
## Checks
- Tests (`test_utcn_xlsr2b_logreg.py`, 9 passed): crop/repeat-pad, normalisation == HF feature extractor,
npz sha + head sign, strict load (182 tensors == module state), eval mode + param counts, batch == single
(atol 1e-4; real batch of 4 with mixed lengths), determinism, bona fide > spoof on J-SPAW clips.
- Source parity (`source_parity.py`, `logs_source_parity.txt`): app.py's model class + `from_pretrained` + HF feature
extractor + real sklearn LogisticRegression/MinMaxScaler rebuilt from the npz, 18 J-SPAW_LA clips (16 + a 20000 and a
139944-sample clip). Source with eager attention: max |Δ| **0.0** (window off, whole clip) / **2.5e-8** (4 s cap).
Source with its default SDPA: 1.24e-3 / 1.06e-3 (SDPA kernel noise, see above).
- J-SPAW_LA full (PyTorch, RTX 5060 Ti idx 0, `run_pt.sh`): **EER 0.875%**, 2397 trials, 0 skipped.
## Batch sweep (`sweep.py`, 64600-sample input, 2026-11-09)
| card | result | batch_size |
|---|---|---|
| RTX 4070 Ti SUPER (idx 2) | 107.3 utt/s @24, plateau 8–64 | 24 |
| RTX 5060 Ti (idx 0) | flat 24.6–26.3 utt/s for bs 1–64 | 8 |
## TensorRT (`trt_utcn_xlsr2b_logreg.py`, sidecar `trt_utcn-xlsr2b-logreg.json`, TRT 10.16.1.11)
Adapted from ProSDD-ASV19's script: the engine outputs the (B,1920) pooled features; the LR + MinMax head stays in
numpy (the TRT class sets `self.head`). Strict FP32 (FORCE_FP32 and TF32 cleared by default); ONNX 1.65 GB < 2 GB, so
the TorchScript exporter suffices. Build parity on 64 In-the-Wild clips.
| card | precision | best batch | utt/s | build parity max Δ / Spearman | J-SPAW EER | vs PT Spearman / max Δ |
|---|---|---|---|---|---|---|
| RTX 4070 Ti SUPER (idx 2) | FP32 | 32 | 120.4 | 4.2e-7 / 1.0 | 0.875 (=PT) | 1.000000 / 1e-6 |
| RTX 5060 Ti (idx 0) | FP32 | 32 | 72.5 | 2.3e-7 / 1.0 | 0.875 (=PT) | 1.000000 / 1e-6 |
Both accepted (|ΔEER| = 0, Spearman ≥ 0.999). Only the 4070 ONNX is kept. Scripts: `run_trt_export.sh <gpu>`,
`run_trt_jspaw.sh <gpu>`; compare with `cmp_scores.py`.
## Training data → trained_on
Paper §2.1/§4.1: "ALL" = every sample (incl. test splits) of ASVspoof 2019 LA, FoR for-norm, ~60k random ASVspoof 2021
(Table 1: 22k bona fide / 589k spoof / 100 systems = the DF eval set), TIMIT-TTS, ODSS, MLAAD v5 + 80k M-AILABS bona
fide, ASVspoof 5 train+dev; then margin pruning (§4.2). ITW and AI4T were held out of training, but the released
configuration (ALL + margin-both pruning) is the one picked by In-the-Wild EER (1.70%, Table 4).
- `trained_on`: ASVspoof2019_LA, ASVspoof2021_DF, ODSS, **InTheWild** (selection, following the Forensics-XLSR-Wild
precedent — maintainer may drop it). DECRO is flagged automatically via its ASVspoof2019-LA:eval bona fide.
- Not flagged: ASVspoof2021_LA (paper's ASV21 counts match DF), ASVspoof5 (train/dev only; Arena = eval), XMAD
(M-AILABS bona fide is a separate corpus in provenance), FoR/TIMIT-TTS (not in the Arena).
- Suggested `provenance.yaml` models entry (not added; maintainer): `utcn-xlsr2b-logreg: {backbone: XLS-R,
trained_on: [ASVspoof2019-LA, "ASVspoof2021-DF:eval", "ASVspoof5:train", "ASVspoof5:dev", ODSS, MLAAD, M-AILABS, InTheWild]}`.
|