Dynamic batching, ragged inputs and client-side packing

This note covers what Triton's dynamic batcher needs, what it does to a model with max_batch_size: 0, how ragged inputs and batch inputs work, how the Python backend receives a dynamic batch, and the alternative of packing blocks in the client. The facts apply to Triton 24.10 (the r24.10 branches of server, core, common and python_backend); where core main or r25.06 was checked, the text says so. Behaviour read from source code is marked as such. Checked 9 October 2026.

What the dynamic batcher requires

Batching by the dynamic or sequence batcher needs the model's batch dimension to be the first dimension of every input and output; max_batch_size is then set to 1 or more. Models that do not support batching "in the specific ways described above" must set max_batch_size to zero [1].
Dynamic batching combines requests in the server. It is enabled per model with dynamic_batching { }, and its settings control the preferred batch sizes, how long a request may wait for others, and queue properties such as queue size, priorities and time-outs [1].
"By default, the requests can be dynamically batched only if each input has the same shape across the requests" [2]. The proto says the same of allow_ragged_batch: false means two requests batch only if the tensor has the same shape in both [3]. In core, every variable-size input (any -1) that is not marked allow_ragged_batch, and every shape tensor, is put in an "enforce equal shape" set used when forming a batch [4].
preferred_batch_size: "should not be used for most models"; the batcher forms the largest preferred size it can, otherwise the largest size up to max_batch_size [1]. If none is given, core fills in max_batch_size, but only when max_batch_size > 0 [5].
max_queue_delay_microseconds (default 0) lets a batcher hold a batch that is not of a maximum or preferred size until the oldest request has waited that long [1, 3].
preserve_ordering returns responses in request order [1, 3]. Validation rejects it together with more than one priority level, or with a queue time-out whose action is DELAY [5].
Custom batching rules can be added through five TRITONBACKEND_ModelBatch* functions in a shared library named by the TRITON_BATCH_STRATEGY_PATH parameter [1].
Without any scheduling choice the model gets the default scheduler, which "simply distributes inference requests to all model instances" [1]. In core this is the dynamic batch scheduler with batching switched off; each request goes straight to the rate limiter as its own payload [4, 6].

What happens with `max_batch_size: 0`

Validation in core r24.10 rejects a negative max_batch_size and any preferred_batch_size larger than max_batch_size ("dynamic batching preferred size must be <= max batch size"), but it does not reject a dynamic_batching block on a model with max_batch_size: 0 [5].
When dynamic_batching is present, core creates the dynamic batch scheduler with batching enabled and the model's max_batch_size [4]. The scheduler stores max_batch_size_((size_t)std::max(1, max_batch_size)), so a model with max_batch_size: 0 is given an effective limit of one request per execution [6]. The same line is on core main [7].
Read together (source reading, not a documented contract): such a model gets the batcher's queue (priority levels, queue policy, time-outs, max_queue_size) but never more than one request in an execute() call.

Ragged inputs and batch inputs

allow_ragged_batch: true on an input means two requests may batch even if that tensor's shape differs [3].
"How ragged input are processed in a batch of requests depends on the backend implementation." The ONNX Runtime, TensorFlow, PyTorch and TensorRT backends require ragged inputs to be 1-D and concatenate the requests' inputs into one 1-D tensor [2]. The Python backend is not on that list [2].
Because the concatenated tensor does not record where each request starts and ends, Triton can generate batch inputs [2]. The kinds are [3]: BATCH_ELEMENT_COUNT (element count of the source input, shape [1]); BATCH_ACCUMULATED_ELEMENT_COUNT (running total); BATCH_ACCUMULATED_ELEMENT_COUNT_WITH_ZERO (the same with a leading 0 on the first request); BATCH_MAX_ELEMENT_COUNT_AS_SHAPE (shape [max element count], data uninitialised); BATCH_ITEM_SHAPE and BATCH_ITEM_SHAPE_FLATTEN (each request's source shape). The batch input data type can be TYPE_INT32 or TYPE_FP32 [3].
Worked example from the docs: requests of shapes [1, 3], [1, 4] and [1, 5] become one tensor of shape [12], and a BATCH_ACCUMULATED_ELEMENT_COUNT input named INDEX holds [3, 7, 12] [2].

How the Python backend receives a dynamic batch

execute() "receives a batch of pb_utils.InferenceRequest as a length N array" and must return exactly N responses, one per request, in the same order [8]. The README's template says "Every Python backend must iterate through list of requests" [8]. Each request keeps its own input tensors; nothing is concatenated for the model.
By default the Python backend moves every input tensor to CPU before the model sees it; FORCE_CPU_ONLY_INPUT_TENSORS: "no" lets inputs stay where they were last used [8].
python_be.cc on r24.10 counts the batch size per request (first dimension when max_batch_size > 0, otherwise 1 per request) and contains no reference to batch inputs [9]. Inference: batch inputs are not generated for Python models, so a Python model has to work out the boundaries from the request list itself.

Why the batcher cannot pack the diarization requests (inference)

The points below are reasoning from the rules above, not documented statements.

diar_nn_g<i> has max_batch_size: 0, so there is no batch dimension, and a dynamic_batching block would give it a queue but still one request per execution [5, 6].
Giving the inputs a batch dimension does not help on its own: SPAN, OFFSETS, VALID, WEIGHTS and IDX vary in length, so without allow_ragged_batch two requests would batch only when every length matched exactly [2, 4].
With allow_ragged_batch, the Python backend still delivers separate request objects [8]. The model would have to group requests by OP, concatenate spans and rebase OFFSETS itself, which is the packing work in a new place.
The batcher does not look at input values, so one batch could mix OP 0 (segmentation) and OP 1 or 2 (voiceprints), which cannot be one network call. Keeping them apart would need a custom batching library in C [1] or one model per OP.
The second call of a block depends on masks the engine computes from the first call, so the two calls of one block can never share a batch; any max_queue_delay_microseconds would add directly to that dependency chain.

Client-side packing

The alternative keeps max_batch_size: 0 and lets the Rust engine put several blocks in one request: one longer SPAN, and OFFSETS/VALID (and WEIGHTS/IDX for voiceprints) covering 8 x k windows. Trade-offs (inference):

The engine already chooses which GPU gets each block, so it can also choose k per request; no server-side delay or batching policy is involved, and batch composition is repeatable from the engine's side.
Each packed request carries about k x 1.7 MB, holds k blocks of results until the whole request finishes, and uses more GPU memory per call. Fewer, larger requests reduce per-request HTTP and Python overhead.
Triton 24.10 has no HTTP size limit. The --http-max-input-size option of 25.06 and later (default 64 MiB, about 38 blocks of 1.74 MB) appears, from source, not to cover binary tensors; see 05-protocols-shared-memory.md.
A packed request is a new batch shape for the networks. The exact lane asks only for the release's batch shapes, so a packed shape belongs to the faster lane unless it is checked against the release first.

Unverified

That dynamic_batching on a max_batch_size: 0 model yields one request per execution is read from core source; no Triton document states it, so it may change.
That the Python backend never generates batch inputs is inferred from the absence of any reference in python_be.cc r24.10; no document says so either way.

What this means for the diarization model

The dynamic batcher cannot merge the engine's blocks into larger network calls while the model keeps max_batch_size: 0, variable-length 1-D inputs and an OP switch; packing has to happen in the engine (inference).
Adding dynamic_batching { } without preferred_batch_size to diar_nn_g<i> would leave execute() unchanged (one request per call) and would add priority levels, max_queue_size and per-request time-outs [5, 6]. With 2 requests in flight per GPU from the engine, a queue limit mainly protects against other clients.
A packed request of k blocks is one network call of 8 x k windows; for the faster lane this is the route to larger batches, and its accuracy against the exact lane decides whether a packed shape is acceptable (inference).
Moving the masks into the model (one request per block running both networks) would remove the second transfer of the same SPAN, which is independent of batching (inference; see 02-ensemble-bls-single-model.md).

Sources

Triton Model Configuration (Maximum Batch Size, Default Scheduler, Dynamic Batcher, Custom Batching) - https://github.com/triton-inference-server/server/blob/r24.10/docs/user_guide/model_configuration.md - applies to r24.10.
Triton Ragged Batching - https://github.com/triton-inference-server/server/blob/r24.10/docs/user_guide/ragged_batching.md - applies to r24.10.
model_config.proto (ModelInput.allow_ragged_batch, BatchInput, ModelQueuePolicy, ModelDynamicBatching) - https://github.com/triton-inference-server/common/blob/r24.10/protobuf/model_config.proto - applies to r24.10.
Core TritonModel::SetConfiguredScheduler - https://github.com/triton-inference-server/core/blob/r24.10/src/backend_model.cc#L797-L847 - applies to r24.10.
Core NormalizeModelConfig and ValidateModelConfig - https://github.com/triton-inference-server/core/blob/r24.10/src/model_config_utils.cc#L734-L745 and https://github.com/triton-inference-server/core/blob/r24.10/src/model_config_utils.cc#L1410-L1498 - applies to r24.10.
Core dynamic batch scheduler (constructor at line 92; non-batching path at lines 240-251) - https://github.com/triton-inference-server/core/blob/r24.10/src/dynamic_batch_scheduler.cc - applies to r24.10.
Core dynamic batch scheduler on main (GitHub code search hit, commit 00e4e60) - https://github.com/triton-inference-server/core/blob/main/src/dynamic_batch_scheduler.cc - applies to main, 9 October 2026.
Python backend README (execute, Default Mode, Input Tensor Device Placement) - https://github.com/triton-inference-server/python_backend/blob/r24.10/README.md - applies to r24.10.
Python backend CheckIncomingRequests - https://github.com/triton-inference-server/python_backend/blob/r24.10/src/python_be.cc#L65-L113 - applies to r24.10.