| # Determinism: why the outputs are byte-identical, and where that stops |
|
|
| The engine reproduces the release's arithmetic, not just its results. Each piece below is pinned for a measured |
| reason; `docs/TESTING.md` lists the checks that hold it in place, and `tests/test_matrix.py` runs the same 39 files |
| under changing conditions (each GPU alone and all of them together, 1 to 64 threads, 1 to 64 files at once, other arrival orders, |
| a warm repeat, a fresh process with every BLAS thread variable at 64): every record and RTTM identical in all of them. |
|
|
| ## The networks (GPU) |
|
|
| - **Same code and weights.** The release's own model classes (DiariZen and its pyannote fork at commit 844f555), the |
| bundle's weights, built the way `load_bundle` builds them. |
| - **Same numeric flags, set after the imports.** FP32, TF32 off, deterministic algorithms, cuDNN benchmark off, |
| `CUBLAS_WORKSPACE_CONFIG=:4096:8`. Importing DiariZen switches TF32 on for the whole process, so the flags are |
| set after it, as `load_bundle` does, and re-asserted before every network call. |
| - **Same kernels.** torch 2.1.1+cu121, its cuDNN 8.9.2, and cuBLAS 12.1.0.26 (`env/canonical.sh` loads it from a |
| CUDA 12.1 installation; the cu121 Triton image carries the same files and searches them first). |
| - **Same batch shapes.** Segmentation: batches of 8, the remainder, the zero-padded last window alone (other sizes |
| change bits). Voiceprints: the trunk once per window in batches of 8, pooling and the last layer at the release's |
| rows (identical bits), the odd last window as the release runs it. |
| - **One CUDA stream per GPU.** Every GPU of a kind gives the same bits as the others, but two streams sharing one |
| model on one GPU changed about 1 batch in 1,000. Locally there is one worker thread per GPU; on Triton one process |
| per GPU. |
| - **Models never cross GPUs.** Moving several model copies from GPU to GPU in one process gave all-NaN weights on |
| one GPU; copies go through host memory, and every Triton instance sees only its own GPU. |
| - **Checked before use.** Every backend runs every batch shape at start-up and compares with the release's own |
| outputs (`assets/nn_golden.json`, written from `load_bundle`): identical bytes where the setup is reproduced; |
| otherwise the outputs must stay within fixed near-identical bounds of `assets/nn_reference.npz` (or, in strict |
| mode, the backend stops). |
|
|
| ## Everything else (CPU, Rust) |
|
|
| - **numpy, scipy, pyannote and CPython reproduced operation by operation** (crate `diar-exact`): pairwise |
| summation with numpy's 8 accumulators and 8,192-element buffers, numpy's BLAS dispatch (gemm / gemv / dot) with |
| numpy's own OpenBLAS binary at one thread, glibc's `exp`/`log`, introsort tie order, scipy's centroid linkage and |
| heap, cosine distance with its two-lane sum, rectangular assignment, pyannote's sliding-window arithmetic and |
| aggregation, CPython's `round`, `sum`, string sort and `.3f` formatting. |
| - **Parallel only where each element's arithmetic is unchanged.** Distances by row, VBx products by blocks of |
| result rows (each its own single-threaded OpenBLAS call; tested bitwise against the one-call product), VBx and |
| aggregation by rows or output frames with every element's terms in the release's order. |
| - **The engine's OpenBLAS is private** (its own linker namespace) and pinned to one thread whatever the environment |
| says. |
| - **Decoding** is libsndfile 1.2.2, the binary soundfile loads, called in soundfile's order; non-16 kHz audio is |
| resampled by the release's own torchaudio resampler (copied unchanged into `anuvani_diar.nn.resample`) with 2 |
| threads. |
|
|
| ## Where the guarantee stops |
|
|
| - Identical bytes need the same software (`env/canonical.sh`) on the same kind of GPU as the release's outputs were |
| produced with. Another GPU model, or another PyTorch, CUDA or cuDNN version, picks other kernels whose last bits |
| differ: the start-up check then measures the networks against the release's outputs and runs only if they stay |
| within the near-identical bounds (maximum posterior difference 0.25, frame decisions at least 99% equal, voiceprint |
| cosine at least 0.995), and says so; strict mode refuses to run instead. |
| - The CPU side is pinned to AVX2 code paths by `env/canonical.sh` (numpy, ATen and oneDNN features, and |
| `OPENBLAS_CORETYPE=Zen`). The engine's private OpenBLAS no longer depends on that variable: it sets |
| `OPENBLAS_CORETYPE=Zen` (and one thread) itself for its own load when the CPU reports AVX2 and FMA, checks that it |
| runs core Zen, and refuses to run otherwise (`docs/WHEEL.md`). On an AVX2 CPU the pins change nothing; on a CPU |
| with AVX-512, running without them could select other kernels in numpy, ATen and oneDNN. |
| - Inputs the release itself would treat differently (a different bundle, batch size or package versions) are |
| outside the contract. |
|
|
