| # DeepFense-CodecFake-W2V2-MLP β implementation notes |
|
|
| Plan: `speech-spoof-bench/docs/plans/2026-11-07-eleven-model-batch-arena-submission.md` (model #8). |
|
|
| ## Source |
| - Weights: HF `DeepFense/CodecFake_Wav2Vec2_MLP_NoAug_Seed42` @ `c08bd2125c83e6c8fdc64566bfbfda18dff4dfb4`, |
| `best_model.pth` (3.79 GB, sha256 `1d4bf44afd0f7e887266c8dc49875b0ddb60a07fb16279fa2d56598519ea3c1a` = HF LFS oid) |
| + `config.yaml`, in `benchmarks/_ckpt_dl/DeepFense__CodecFake_Wav2Vec2_MLP_NoAug_Seed42/`. The .pth is a |
| training checkpoint: `model_state` (435 fp32 tensors, 317,522,306 params), `optimizer_state` (Adam), |
| `epoch`=2, `step`=46298, `best_metric`=0.00306 (val EER fraction). No weight license on the Hub β linked, not re-hosted. |
| - Extraction: `torch.load(weights_only=True)` works (plain tensors/ints) β run once with `python -I`; `model_state` |
| saved unchanged as `model_state.safetensors` (sha256 `5c97bd4ae3a2075aa8df309bde8b7dd7f9ce540a50df969d4ff0ba7a74a560a5`). |
| The optimizer state is dropped. |
| - Code: `github.com/Yaselley/deepfense-framework` @ `3f2fe1146c48ea4bacc480dd96d806b6c72c09cf` (Apache-2.0), checked out |
| to `benchmarks/_ckpt_dl/Yaselley__deepfense-framework/` (not imported by the wrapper; only by `source_parity.py`). |
|
|
| ## Architecture (`_net.py`) |
| - `frontend.model`: fairseq 0.12.2 `Wav2Vec2Model` built from the vendored `xlsr2_300m_model_cfg.json` |
| (= `state["cfg"]["model"]` of the local `W2V2-AASIST/xlsr2_300m.pt`, merged onto `Wav2Vec2Config` defaults with |
| `merge_with_parent`, as in W2V2-AASIST/_net.py). No .pt is loaded at runtime. Called as |
| `model(x, mask=False, features_only=True)["x"]` (last layer, after the encoder's final layer_norm β layer_norm_first). |
| - `backend.projection_block` = Linear(1024,128) β LayerNorm(128) β ReLU; mean over time (pool "mean"). |
| - `losses.0.fc` = Linear(128, 2). |
| - Key mapping: the checkpoint was saved from real fairseq (it carries `quantizer.*`, `project_q.*`, `final_proj.*`), |
| so all 435 tensors map 1:1 β `strict=True`, no allow-list. Those 7 pre-training tensors are unused on the |
| features_only path. (The repo's newer `fairseq_local` loader lacks those modules: strict=False reports exactly |
| those 7 as unexpected, 0 missing.) params_millions 317.522306 counts them (as the checkpoint does). |
| |
| ## Preprocessing / score (`deepfense_codecfake_w2v2_mlp.py`) |
| - `StandardDataset` + `transforms.load_audio` / `pad` (test split: max_len 64000, pad_type repeat, random_pad false): |
| soundfile float, mono mean, no normalisation anywhere (dataset, frontend: XLS-R `normalize: false`), first 64000 |
| samples, shorter β `np.tile(x, ceil(64000/len))[:64000]`. Empty clip β zeros (not in source). No resampling. FP32, |
| cuDNN TF32 off. |
| - Score = logit1 β logit0 (label_map bonafide=1), as specified in the plan; monotone in softmax p(bonafide), which |
| is what the authors' `simple.infer.py` thresholds. **Deviation note:** the authors' `test.py`/trainer EER uses |
| `CrossEntropy.get_score` = `logits[:, 1]` only (its docstring says "logit_bonafide β logit_spoof" but the code |
| returns the single logit). The two are not rank-equivalent; see the alt-score table below. |
| - **Source parity** (`source_parity.py`, log `logs_source_parity.txt`): authors' `ModularDetector` (fairseq_local |
| frontend from xlsr2_300m.pt, strict=False load of model_state) + their `load_audio`/`pad` vs this wrapper on |
| 12 clips (6 J-SPAW_LA, 6 In-the-Wild): max |Ξ logits| = 4.8e-6. |
| |
| ## Batch sweep (`sweep.py`, 64000-sample input, TF32 off, 2026-11-07; `logs_sweep_g{1,2}.txt`) |
| | card | best | notes | |
| |---|---|---| |
| | RTX 4070 Ti SUPER (idx 2) | 123.9 utt/s @24 | plateau 16β64 | |
| | RTX 5060 Ti (idx 1) | 48.5 utt/s @4 | ~33 at bs β₯ 8 β batch_size 4 on 5060 | |
|
|
| ## Sanity (PyTorch, RTX 5060 Ti idx 1) |
| - Full J-SPAW_LA (`results_pt/J-SPAW_LA`, copied to `results/J-SPAW_LA` for submission): EER **0.625%** |
| (2397 trials, 0 skipped), ~46 utt/s end-to-end. Low but not inverted; Codecfake training data is codec |
| re-synthesis, J-SPAW_LA spoofs are apparently easy for it. |
| - Full ODSS (`results_pt/ODSS`): EER **13.17%** (26,954 trials) vs self-reported 11.51% (+1.66 pt). The Arena ODSS |
| build may differ from the authors' ODSS protocol (e.g. VCTK spoof-only subset, generator mix); not investigated further. |
| - Alt score (`alt_score_eer.py`, `logs_alt_score.txt`): authors' test.py score `logit1` vs ours `logit1 β logit0`: |
| J-SPAW_LA 0.625 / 0.625, ODSS 13.20 / 13.17 β the score choice does not explain the ODSS gap and is immaterial. |
| |
| ## TensorRT (`trt_deepfense_codecfake_w2v2_mlp.py`, sidecar `trt_deepfense-codecfake-w2v2-mlp.json`, TRT 10.16.1.11) |
| Adapted from Forensics-XLSR-Wild's script; `_ExportNet` returns (B,2), n_classes 2. FP16 fails the 0.9999 |
| build-parity rule on both cards (Spearman 0.99899 on 4070, 0.99940 on 5060, 64 ITW clips) β FP32 on both |
| (FP16 engines deleted). |
| | card | precision | best batch | utt/s (sweep) | build parity Spearman (64 ITW) | J-SPAW EER | J-SPAW Spearman vs PT / max Ξ | |
| |---|---|---|---|---|---|---| |
| | RTX 4070 Ti SUPER (idx 2) | FP32 | 32 | 182.4 | 0.99991 | 0.625 (=PT) | 0.999980 / 0.0223 | |
| | RTX 5060 Ti (idx 1) | FP32 | 8 | 126.0 | 0.99995 | 0.625 (=PT) | 0.999983 / 0.0317 | |
| Both pass (|ΞEER| = 0, Spearman β₯ 0.999). Scripts: `run_pt.sh`, `run_trt_export.sh`, `run_trt_jspaw.sh <gpu>`, `run_alt.sh`. |
| |
| ## Hosting |
| - `.gitignore` excludes `model_state.safetensors` and `*.onnx` (unlicensed weights must not be re-hosted). |
| `meta.yaml` `checkpoint` links the original HF best_model.pth @ c08bd21. |
| - `trained_on` omitted (Codecfake is not an Arena dataset). Paper block = arXiv 2604.08450 (kheir2026deepfense). |
| |
