src/triton_inference/docs/reference/README.md

Reference library: serving the diarization networks on Triton

A compact, sourced reference for serving the Anuvani diarization networks on NVIDIA Triton Inference Server, and for a faster lane. Each file covers one topic, states the versions its facts apply to, and ends with what the facts mean for this model. Checked 9 October 2026.

How to read these files

• Every factual sentence carries a tag such as [3] that points to the file's Sources list, where each source has its URL and the version it applies to (a Triton release branch such as r24.10, a TensorRT or PyTorch version, or a documentation build).
• Anything that could not be confirmed in a primary source is marked "unverified" and listed under "Unverified".
• A source that begins "Contents of the image" records what a published NVIDIA image contains, read from the image with the commands it names. It is not NVIDIA documentation.
• The closing section of each file, "What this means for the diarization model", reasons from those facts and the model facts below. It is inference, labelled as such; nothing in this library was measured on a GPU.

The model these files are about (project facts)

• Audio is cut into 16 s windows every 1.6 s (256,000 samples at 16 kHz); windows go in blocks of 8, and the span of one block is about 1.7 MB of fp32 samples.
• Per block, two dependent network calls. (1) Segmentation: a pruned WavLM-Large (24 layers, 0 to 10 attention heads per layer, four layers without attention, FFN widths 96 to 1770, a 7-layer raw-waveform convolutional front end), a 4-layer Conformer and a 16-class powerset head; output (B, 799, 16). (2) Voiceprints: kaldi fbank (80 mel, through torch.vmap), a WeSpeaker ResNet34 trunk, weighted statistics pooling per local speaker (799 frame weights interpolated to 200 frames) and a linear layer; output 256-dim.
• Work per window: 146.3 GFLOP (segmentation) plus 72.5 GFLOP (voiceprints). Masks (argmax and an 11-frame median filter) are computed between the two calls, in the Rust engine, so each block sends its span twice. Clustering (AHC, PLDA, VBx), the tree decoder and the RTTM writer run on the CPU in Rust.
• Serving today: Triton 24.10; Python backend models diar_nn_g<i>, one per GPU, max_batch_size: 0, one instance, one CUDA stream; inputs OP int32 (0 segmentation, 1 voiceprints, 2 repacked voiceprints), SPAN fp32, OFFSETS and VALID int64, optional WEIGHTS fp32 and IDX int64; output OUT fp32. PyTorch 2.1.1 in FP32 with TF32 off, byte-identical to the frozen release.
• The Rust engine is Triton's only client (KServe v2 HTTP with binary tensors), chooses the GPU for each block and keeps 2 requests in flight per GPU; a FastAPI service will sit in front of it. These notes call the present FP32 path the exact lane. A faster lane (TF32, BF16 or FP16; TensorRT or torch.compile; several blocks per request; two CUDA streams) is allowed if a DER accuracy gate passes.

Files

File	What it covers
01-python-backend.md	How execute() is called; one stub process and CUDA context per instance; the shared-memory data path; DLPack, BLS, decoupled mode, custom Python environments; a TensorRT engine on torch's stream
02-ensemble-bls-single-model.md	Ensemble, BLS or one Python model for "network, glue, network", and the cheapest server-side form for this model
03-dynamic-batching.md	What the dynamic batcher needs, what it does with max_batch_size: 0, ragged inputs, how a Python model sees a batch, client-side packing
04-instances-streams-mps.md	Instance groups, streams in one process, time-slicing between contexts, TensorRT-backend stream options, CUDA MPS and its limits
05-protocols-shared-memory.md	HTTP against gRPC, binary tensors, compression, message size limits, system and CUDA shared memory, the Python backend's own copy
06-tensorrt-build.md	TensorRT 10 against 11 typing, precision by compute capability, profiles, attention fusion, LayerNorm, ONNX operators for the front ends, build settings, portability, Polygraphy
07-reduced-precision-hazards.md	FP16, BF16 and TF32 ranges, reductions, autocast lists, pre-LN residual growth, fbank values near 1e14, epsilons, FP32 islands
08-torch-compile-aotinductor.md	torch.compile modes, recompilation and caches (2.1 against 2.14), AOTInductor, Triton's PyTorch backend and its libtorch
09-cuda-graphs.md	Capture rules in CUDA, PyTorch, TensorRT and Triton, and when graphs pay off
10-cutlass-cublaslt.md	cuBLASLt epilogues, cuBLAS compute types and reproducibility, what PyTorch 2.1.1 already sends to cuBLASLt, CUTLASS, custom kernels
11-profiling-mfu.md	Nsight Systems, Nsight Compute, trtexec profiling, DCGM profiling fields, MFU and HFU
12-perf-analyzer-model-analyzer.md	perf_analyzer input data and search options, Model Analyzer, builds that run on glibc 2.35
13-server-operations.md	Metrics (queue against compute time), rate limiter, priority and queue policy, response cache, model control, health, shutdown
14-tensorrt-llm.md	What TensorRT-LLM supports, its removed plugins, how Triton packages it, and why it does not fit this pipeline
15-triton-image-versions.md	Python, CUDA, TensorRT and driver needs of the 24.10, 25.06 and 26.05 images

Findings that cut across files

• The exact lane stays on Triton 24.10. From 24.11 the images are Ubuntu 24.04 with Python 3.12 [1], and torch 2.1.1 wheels exist only for CPython 3.8 to 3.11 [2]. 26.05 is built on CUDA 13.2.1, which needs a driver of 580 or later [3]; on an older driver CUDA fails to initialise with CUDA_ERROR_SYSTEM_DRIVER_MISMATCH unless forward compatibility applies [25]. (01, 15)
• Triton's PyTorch backend would change the numerics: in 24.10 it is built from the NGC PyTorch 24.10 container, PyTorch 2.5.0a0+e000cf0ad9 with cuBLAS 12.6.3.3 [4, 5]. (08)
• The dynamic batcher cannot make the network calls larger. A dynamic_batching block on a max_batch_size: 0 model is accepted, but the scheduler caps each execution at max(1, max_batch_size) requests [6]; it adds a queue (priorities, queue size, time-outs) and no batching. Larger calls have to be packed by the Rust engine. (03, 13)
• One request per block is the structural saving in sight: a third OP that runs segmentation, masks and voiceprints in one execute() would send the span once instead of twice and remove a round trip, keeping one process, one context and one stream. It is exact only if the PyTorch masks equal the Rust masks bit for bit (inference). Ensembles and BLS would add models, processes and internal requests. (02, 05)
• On transport, 24.10 has no option for an HTTP body limit; --http-max-input-size (default 64 MiB) arrives in 25.06 [7] and, from the source, appears not to cover binary tensors. The Python backend copies every CPU input into its own shared-memory pool [8], so client shared memory would save the HTTP body but not that copy. No NVIDIA document compares HTTP and gRPC at this request size. (05)
• Kernels from different CUDA contexts, such as two Python instances, do not run concurrently; the GPU time-slices between them unless MPS is used [9]. cuBLAS promises bit-wise results only while a single stream is active [10], which matches the release's one-stream rule; two streams belong to the gated lane. (04, 10)
• TF32 off needs both PyTorch switches, because cudnn.allow_tf32 defaults to True [11]; the release sets both. TensorRT enables TF32 by default [12], while Polygraphy disables it by default [13]. (06, 07, 10)
• TensorRT 11 removed weak typing [14]; precision then follows the tensor types, and FP32 islands are written as casts around the operations that must stay FP32 [23]. The three images carry TensorRT 10.5, 10.11 and 10.16 (see 15). Fused attention is documented for FP16, BF16, INT8 and FP8, with no FP32 or TF32 entry [15], and the onnx-tensorrt operator list for 10.5 marks DFT unsupported [16], so the fbank stays in PyTorch FP32. FP16 tops out at 65,504 [12], far below the fbank's powers of about 1e14. (06, 07)
• PyTorch does not promise bitwise-identical results for mathematically identical floating-point computations [24], and no document promises that compiled output equals eager output. 2.1.1 recompiles a frame up to 64 times before falling back to eager [17]. TensorRT puts a kernel launch at 5 to 15 µs of host time [18]; when a call's GPU work is far longer than its launch time, CUDA graphs help only if Nsight Systems shows the GPU waiting (inference). (08, 09)
• The current Nsight Compute needs a CUDA 13 driver [19], so a driver older than R580 needs an older release; DCGM profiling must be paused around Nsight runs [20]. On a glibc 2.35 host (Ubuntu 22.04), perf_analyzer runs from the 24.01 SDK image or tritonclient 2.51.0; the tritonclient wheels from 2.52.0 and the perf-analyzer wheels need GLIBC_2.38 [21]. (11, 12)
• TensorRT-LLM does not fit: from release 1.2 it has no TensorRT engine backend [22], and its documented models do not include this kind of pipeline. (14)

Main points left unverified (details in each file)

• One request per execute() with dynamic_batching and max_batch_size: 0 comes from core source, not from a document (03).
• That --http-max-input-size does not cover binary tensors comes from reading the source; it was not tested (05).
• Which RTX GPUs are "NGC Server Ready" SKUs eligible for forward compatibility, and whether DCGM profiling is supported on workstation GPUs (15, 11).
• Whether compiled, graph-captured or TensorRT FP32 output can ever equal the release's eager bytes (06, 08, 09).
• Whether PyTorch 2.1.1 can export torch.fft.rfft under torch.vmap to ONNX, and whether WavLM's gated relative position bias still matches TensorRT's attention fusion pattern (06).
• Which Nsight Compute release is the newest that runs on a driver older than R580 (11).

Sources

1. Triton Release 24.11 - https://docs.nvidia.com/deeplearning/triton-inference-server/release-notes/rel-24-11.html - 24.11.
2. torch 2.1.1 files - https://pypi.org/project/torch/2.1.1/#files - checked 9 October 2026.
3. CUDA Toolkit Release Notes, Tables 2 and 3 - https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html - CUDA 13.4 Update 1.
4. build.py - https://github.com/triton-inference-server/server/blob/r24.10/build.py - r24.10.
5. PyTorch Release 24.10 - https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel-24-10.html - NGC PyTorch 24.10.
6. dynamic_batch_scheduler.cc - https://github.com/triton-inference-server/core/blob/r24.10/src/dynamic_batch_scheduler.cc - r24.10.
7. command_line_parser.cc - https://github.com/triton-inference-server/server/blob/r25.06/src/command_line_parser.cc - r25.06.
8. python_be.cc - https://github.com/triton-inference-server/python_backend/blob/r24.10/src/python_be.cc - r24.10.
9. CUDA C++ Programming Guide, 3.2.8 - https://docs.nvidia.com/cuda/archive/12.6.2/cuda-c-programming-guide/index.html - CUDA 12.6.2.
10. cuBLAS, Results reproducibility - https://docs.nvidia.com/cuda/archive/12.1.0/cublas/index.html - cuBLAS 12.1.
11. CUDA semantics, TF32 on Ampere - https://docs.pytorch.org/docs/2.1/notes/cuda.html - PyTorch 2.1.
12. Accuracy Considerations - https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/accuracy-considerations.html - TensorRT 11.4.0.
13. Polygraphy CreateConfig - https://github.com/NVIDIA/TensorRT/blob/release/11.3/tools/Polygraphy/polygraphy/backend/trt/config.py - release/11.3.
14. TensorRT 11.0.0 Release Notes - https://docs.nvidia.com/deeplearning/tensorrt/latest/getting-started/release-notes-11/11.0.0.html - TensorRT 11.0.0.
15. Fused Attention - https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/transformers-fused-attention.html - TensorRT 11.4.0.
16. Supported ONNX Operators - https://github.com/onnx/onnx-tensorrt/blob/10.5-GA/docs/operators.md - onnx-tensorrt 10.5-GA.
17. torch/_dynamo/config.py - https://github.com/pytorch/pytorch/blob/v2.1.1/torch/_dynamo/config.py - v2.1.1.
18. Optimizing TensorRT Performance, CUDA Graphs - https://docs.nvidia.com/deeplearning/tensorrt/latest/performance/optimization.html - TensorRT 11.4.0.
19. Nsight Compute Release Notes - https://docs.nvidia.com/nsight-compute/ReleaseNotes/index.html - v2026.3.1.
20. DCGM Feature Overview, Profiling Metrics - https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html - DCGM 4.7.0 docs.
21. tritonclient release files - https://pypi.org/project/tritonclient/#history - 2.51.0 to 2.73.0.
22. TensorRT-LLM release notes, Release 1.2 - https://nvidia.github.io/TensorRT-LLM/release-notes.html - docs 1.3.0rc29.
23. Precision Control - https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/precision-control.html - TensorRT 11.4.0.
24. Numerical accuracy - https://docs.pytorch.org/docs/2.14/notes/numerical_accuracy.html - PyTorch 2.14.
25. CUDA Compatibility, Forward Compatibility - https://docs.nvidia.com/deploy/cuda-compatibility/forward-compatibility.html - updated 9 September 2026.