| # Tests |
|
|
| Every comparison is bitwise (arrays through their raw bytes, files byte for byte). Property tests are |
| derandomised (hypothesis profile "det", fixed seeds), so every run checks the same cases. All suites run in the |
| canonical environment through `make` (`env/canonical.sh`, `python -s`, the project venv). |
|
|
| | Command | Needs | What it checks | |
| |---|---|---| |
| | `make test-rust` | CPU | Rust unit tests: kernels, plan edges, span windows vs direct windows, median, pathlib and repr rules | |
| | `make test-fast` | CPU | numpy/scipy/Python differential tests of every exact kernel; audio decoding against soundfile over every format, and 2,750 damaged files (seeded byte flips, cuts, edge-value header words, insertions, junk) decoded and loaded exactly as soundfile and the release do, numpy's allocation-error text; Python `repr` of every Unicode code point; front-end plan and batches against pyannote's own code; masks against the release; the engine with stand-in networks against the release's own `analyse` (records, RTTMs, errors, batch shapes), cancellation, per-file errors, BLAS environment independence; 40 cached files through the post-processing; VBx row-parallel products vs OpenBLAS's one-call product; the gateway as a service in front of a stand-in Triton (`tests/test_gateway.py`, 32 tests: every RTTM, error and record against the release with the same stand-in networks under each of the three schedules, with one Triton per GPU, and with 64 clients; the fair and share schedules ordering work as documented; Triton killed and restarted mid-run; a model that stays away; model errors; a crashed model process; the gateway killed and restarted mid-run, the client's job ids still working after their jobs were submitted again; queue and upload limits; stalled uploads; a sample rate that would resample past the size limit; names; path rules; the token; the client refusing redirects; cancellation, also when a synchronous client leaves; the result cache; metrics; the spool; SIGTERM; 2,000 seeded random requests (methods, paths, query strings, headers, bodies) without a server error) | |
| | `make test-cache` | CPU | the identity check on cached records (`tests/test_identity_cached.py`): all 1,945 cached records' RTTMs; every replay input decoded; float16 rounding over all 2^32 float32 inputs | |
| | `make test-gpu` | GPUs | the identity check on audio, sample (`tests/test_identity_audio.py`: about 240 files: records, RTTMs, errors) and the GPU tests of `tests/test_api.py` (below) | |
| | `make test-gpu-all` | GPUs | the identity check on audio over all 1,973 files (291.6 h) and the determinism matrix (below) | |
| | `triton/identity_check.sh` | GPUs, Docker | the identity check through the serving stack (`tests/test_identity_serving.py`): Triton + gateway + client, concurrent shuffled jobs, RTTMs, errors and dumped records | |
| | `triton/identity_check.sh "gpu and slow"` | GPUs, Docker | the same over all 1,973 files | |
| | `python -s -m pytest tests/test_api.py -m gpu` | GPUs | `Diarizer(...)(file)`, `Diarizer.run(..., out_dir)`, the `anuvani-diar` CLI vs the release's golden hashes and `run.json` fields, and the install check | |
| | `anuvani-diar-check --audio-dir DIR [--long-dir artifacts/test_audio]` | GPUs | the install check: the release's 12 golden fingerprints, plus the 1 h, 3 h and 6 h files; `tests/test_api.py::test_install_check` runs it | |
| | `python -s -m pytest tests/test_matrix.py -m gpu` | GPUs | the determinism matrix: 39 files on each GPU alone and on all of them together; post-processing threads 1/2/8/64, decoding threads 1/2/4/8, files at once 1/8/64/default; a new arrival order per row; a repeat in the same warm process; a fresh process with every BLAS and OpenMP thread variable at 64. Records and RTTMs vs the reference outputs: identical in every run | |
| | `GW_ARGS=... IDENTITY_SEED=... RESTART_AT=... triton/identity_check.sh` | GPUs, Docker | variants of the serving identity check: another GPU assignment and schedule; Triton killed and restarted mid-run | |
|
|
| The identity checks compare with the reference program's outputs, and hold on the setup the release's outputs were |
| produced with (`env/canonical.sh`); on other GPUs or library versions only the network-free parts (`make test-fast`, |
| `make test-cache`) are expected to match byte for byte. |
|
|
| Seeded, repeatable equivalents take the place of fuzzing (fuzzing is random from run to run, and cargo-fuzz needs a |
| nightly toolchain): 2,750 damaged audio files through the decoder |
| wrapper, and 2,000 random requests through the job API. |
|
|
| Reference outputs the suites read: `oracle/cache_canonical.json` and `artifacts/oracle_cache/` (the identity check |
| on cached records), `oracle/audio_canonical.json` and `artifacts/oracle_audio/` (the identity checks on audio and |
| through the serving stack), `assets/nn_golden.json` (the network self-test). `oracle/` and `artifacts/` are not part |
| of the repository: they are the reference program's (`reference/` in the model repository) outputs on the benchmark, |
| and the suites read them, with the benchmark manifests (`manifests/`) and the cached front-end records |
| (`records/<set>/`), from `$ANUVANI_DIAR_DEV_DATA`, by default `artifacts/dev_data/`. |
|
|
