Protocols, binary tensors, shared memory and message limits

This note covers how a client talks to Triton: HTTP/REST against gRPC under the KServe v2 protocol, the binary tensor extension, compression, system and CUDA shared memory, and the size limits on each path, with a note on what the Python backend does with the bytes once they arrive. The facts apply to Triton 24.10 (r24.10 branches of server, client and python_backend); later branches are named where behaviour changed. Checked 9 October 2026.

HTTP/REST and gRPC

Triton serves both protocols from the KServe v2 standard plus Triton's own extensions; gRPC adds a bidirectional streaming form of the inference call, and NVIDIA "typically recommend[s] using the unary version" unless a stream is needed (for example to keep a sequence on one server behind a load balancer) [1].
Default ports are 8000 (HTTP), 8001 (gRPC) and 8002 (metrics) [2, 3, 4]. --http-thread-count sets the number of threads handling HTTP requests; the default is 8 [2, 3].
I found no statement in the r24.10 protocol, optimisation or performance-tuning documents comparing HTTP with gRPC for a given request size, so there is no documented basis for expecting one to be faster at about 1.7 MB [1, 5].
In the KServe gRPC protocol, sending tensors as raw_input_contents / raw_output_contents "will typically allow higher performance" than the typed contents fields, because of how protobuf allocation interacts with gRPC [6].

Binary tensor data extension (HTTP)

The extension lets an HTTP request or response carry tensors as raw bytes after the JSON header; Triton reports binary_tensor_data among its server extensions [7].
Binary data is little-endian, row-major, with no stride or padding; each type uses its native size [7].
binary_data_size (int64, bytes) on an input or output marks it as binary; binary_data: true on a requested output asks for binary; binary_data_output: true on the request asks for all outputs in binary unless overridden [7].
The body is the JSON object followed by the tensors in the order they appear in the JSON. When any binary data is present, Inference-Header-Content-Length gives the JSON length and Content-Length the whole body [7].
A "raw binary request" (Inference-Header-Content-Length: 0, no JSON) is allowed only for models with one input and at most one variable dimension [7].
Without the extension, tensor data travels as a JSON array of numbers in row-major order [6]. Inference: for 435,200 fp32 samples that means formatting and parsing every value as text, several times the 1.74 MB of raw bytes.

Compression

The r24.10 HTTP server decompresses request bodies sent with Content-Encoding: gzip or deflate, and compresses responses when the client sends Accept-Encoding: gzip or deflate [8].
The client libraries expose request and response compression for HTTP and a compression algorithm for gRPC; the default is no compression in both [9]. On the server, gRPC response compression is set with --grpc-infer-response-compression-level [1].
Inference: on one host, compressing fp32 audio only adds CPU time on both sides.

Message size limits

gRPC server: MAX_GRPC_MESSAGE_SIZE = INT32_MAX (2,147,483,647 bytes) in src/common.h, applied with builder_.SetMaxMessageSize(MAX_GRPC_MESSAGE_SIZE) [10, 11].
Triton's own gRPC clients raise their limits to the same value: the Python client sets grpc.max_send_message_length and grpc.max_receive_message_length to INT32_MAX, and the C++ client calls SetMaxSendMessageSize and SetMaxReceiveMessageSize with it [12, 13]. A third-party gRPC client keeps its own library default unless it is raised (the value of that default is not covered by Triton's sources).
HTTP in r24.10 has no option for a maximum request size, and I found no size check in its http_server.cc [2, 8]. --http-max-input-size first appears on the r25.06 branch (absent on r24.11 to r25.05): "Maximum allowed HTTP request input size in bytes. For compressed requests, this also limits the decompressed size", default 1 << 26 = 67,108,864 bytes (64 MiB) [14, 15].
Where it applies (source reading, not tested): in r25.06 the limit is checked for each input sent as JSON data and for a raw binary request; r26.05 adds checks on the JSON header length and on decompressed size. I found no check on tensor bytes sent with binary_data_size in either branch [21, 22].

Shared memory extensions

Instead of sending tensor bytes in the request, a client can name a registered region with the input or output parameters shared_memory_region and shared_memory_byte_size (both required) and shared_memory_offset (optional, default 0); the input's data field must then be absent [16]. System and CUDA region names share one namespace; there is no Windows support, and Jetson supports only system shared memory [16].
System shared memory, HTTP endpoints [16]: GET v2/systemsharedmemory[/region/${REGION_NAME}]/status, POST v2/systemsharedmemory/region/${REGION_NAME}/register with body {"key", "offset", "byte_size"} (the key of the underlying memory object), and POST v2/systemsharedmemory[/region/${REGION_NAME}]/unregister (no name unregisters all regions).
CUDA shared memory, HTTP endpoints [16]: the same three under v2/cudasharedmemory, with a register body {"raw_handle": {"b64": ...}, "device_id", "byte_size"}, where raw_handle is the serialised cudaIPC handle, base64-encoded, and device_id is the GPU on which it was created.
The gRPC service has matching SystemSharedMemory* and CudaSharedMemory* register, status and unregister calls, and the same parameters go in InferInputTensor / InferRequestedOutputTensor [16].
In the r24.10 server, a system region is opened with shm_open(key) and mmap, and a CUDA region with cudaSetDevice(device_id) and cudaIpcOpenMemHandle [17]. The client therefore creates the region first (POSIX shared memory, or device memory exported with cudaIpcGetMemHandle, which "gets an interprocess memory handle for an existing device memory allocation" [18]). Inference: both mechanisms are host-local, so the client must run on the same host and, in containers, share the IPC namespace or /dev/shm with Triton.
The client README says shared memory "can significantly improve performance in some cases", and ships helper modules for Python plus C++ and Python examples [9].

What the Python backend does with the bytes

The Python backend talks to its stub process through shared memory: 1 MB per model instance by default, grown in 1 MB chunks, set with shm-default-byte-size and shm-growth-byte-size; inside Docker, --shm-size must cover the inputs and outputs because Docker's default is 64 MB [19].
By default every input is moved to CPU before the model sees it; FORCE_CPU_ONLY_INPUT_TENSORS: "no" lets inputs stay in GPU memory where they were last used [19].
In python_be.cc r24.10, a CPU input is allocated in the stub's shared-memory pool and the request data is copied into it (SaveToSharedMemory, then ProcessTensor or ReadInputTensor) [20]. Inference from that code: data that arrives through client system shared memory is still copied once into the Python backend's own pool; it saves the HTTP transfer and parsing, not this copy. GPU inputs take a separate path through a CUDA memory pool [20].

Unverified

The default receive limit of a non-Triton gRPC client library (often quoted as 4 MiB) was not checked against a primary source.
Whether HTTP keep-alive is used and how idle connections are handled by Triton's HTTP server is not documented in the r24.10 docs I read.
No measurement of HTTP against gRPC for this request size was found; any speed difference is unknown.
That --http-max-input-size does not limit binary-extension tensors is read from source; it was not tested.

What this means for the diarization model

Each block is two requests of about 1.7 MB each, because the engine computes masks between the calls and sends the same SPAN twice (repo facts). HTTP with binary tensors is the documented efficient HTTP form; JSON would be far slower to encode and parse (inference).
gRPC has no documented speed advantage at this size. If the engine ever used gRPC, its client would need a receive and send limit above 1.7 MB per block, or k x 1.7 MB when blocks are packed (inference).
On 24.10 there is no HTTP body limit. On 25.06 and 26.05 the 64 MiB default appears not to cover binary tensors, so it would not cap packed blocks sent the way the engine sends them; if a later release extends it, 64 MiB is about 38 blocks of 1.74 MB (67,108,864 / 1,740,800 = 38.5) [14, 21, 22].
System shared memory would remove the HTTP body copy but not the Python backend's own copy, and it ties the gateway to the same host and IPC namespace as Triton (inference).
CUDA shared memory would need the Rust engine, a CPU-side HTTP client today, to allocate device memory and export IPC handles itself (inference).
The container's --shm-size must hold the Python backend pools: with 2 requests in flight per GPU, inputs are a few MB per model instance, far below Docker's 64 MB default, but packed requests grow this linearly (inference).

Sources

Inference Protocols and APIs - https://github.com/triton-inference-server/server/blob/r24.10/docs/customization_guide/inference_protocols.md - r24.10.
command_line_parser.cc (option text) - https://github.com/triton-inference-server/server/blob/r24.10/src/command_line_parser.cc - r24.10.
command_line_parser.h (defaults: http_port_{8000}, http_thread_cnt_{8}, metrics_port_{8002}) - https://github.com/triton-inference-server/server/blob/r24.10/src/command_line_parser.h#L193-L216 - r24.10.
grpc_server.h (port_{8001}) - https://github.com/triton-inference-server/server/blob/r24.10/src/grpc/grpc_server.h#L52 - r24.10.
Optimization; Performance Tuning - https://github.com/triton-inference-server/server/blob/r24.10/docs/user_guide/optimization.md and https://github.com/triton-inference-server/server/blob/r24.10/docs/user_guide/performance_tuning.md - r24.10.
KServe Open Inference Protocol v2, required API (Tensor Data, gRPC raw contents) - https://github.com/kserve/kserve/blob/master/docs/predict-api/v2/required_api.md - KServe master, 9 October 2026.
Binary Tensor Data Extension - https://github.com/triton-inference-server/server/blob/r24.10/docs/protocol/extension_binary_data.md - r24.10.
http_server.cc (compression handling) - https://github.com/triton-inference-server/server/blob/r24.10/src/http_server.cc#L2545-L2577 - r24.10.
Triton client README (Compression, System/CUDA Shared Memory) - https://github.com/triton-inference-server/client/blob/r24.10/README.md - r24.10.
common.h - https://github.com/triton-inference-server/server/blob/r24.10/src/common.h#L48 - r24.10.
grpc_server.cc - https://github.com/triton-inference-server/server/blob/r24.10/src/grpc/grpc_server.cc#L2285 - r24.10.
Python gRPC client - https://github.com/triton-inference-server/client/blob/r24.10/src/python/library/tritonclient/grpc/_client.py#L51-L54 (and L201-L202) - r24.10.
C++ gRPC client - https://github.com/triton-inference-server/client/blob/r24.10/src/c%2B%2B/library/grpc_client.cc#L479-L480 - r24.10.
command_line_parser.cc (--http-max-input-size) - https://github.com/triton-inference-server/server/blob/r25.06/src/command_line_parser.cc#L502-L506 - r25.06 (absent on r24.10 to r25.05; present on main).
common.h (HTTP_DEFAULT_MAX_INPUT_SIZE = 1 << 26) - https://github.com/triton-inference-server/server/blob/r25.06/src/common.h#L58 - r25.06.
Shared-Memory Extension - https://github.com/triton-inference-server/server/blob/r24.10/docs/protocol/extension_shared_memory.md - r24.10.
shared_memory_manager.cc - https://github.com/triton-inference-server/server/blob/r24.10/src/shared_memory_manager.cc#L139-L256 - r24.10.
CUDA Runtime API, Device Management (cudaIpcGetMemHandle, cudaIpcOpenMemHandle) - https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__DEVICE.html - page shows CUDA 13.4.
Python backend README (Managing Shared Memory, Input Tensor Device Placement) - https://github.com/triton-inference-server/python_backend/blob/r24.10/README.md - r24.10.
Python backend GetInputTensor - https://github.com/triton-inference-server/python_backend/blob/r24.10/src/python_be.cc#L349-L556 - r24.10.
http_server.cc (uses of max_input_size_ at L2812-L2822 and L3104-L3113) - https://github.com/triton-inference-server/server/blob/r25.06/src/http_server.cc - r25.06.
http_server.cc and data_compressor.h (JSON size, decompression limit) - https://github.com/triton-inference-server/server/blob/r26.05/src/http_server.cc and https://github.com/triton-inference-server/server/blob/r26.05/src/data_compressor.h - r26.05.