src/triton_inference/docs/reference/01-python-backend.md

Python backend: execution model, data path, BLS, decoupled mode, TensorRT inside it

Scope: how Triton's Python backend runs model.py, how tensors reach it, BLS, decoupled mode, and running a TensorRT engine from inside it. Facts apply to Triton 24.10 (r24.10 branches of python_backend, server, core and backend) unless r25.06 or r26.05 is named; TensorRT facts come from the 10.x docs (10.16.1). Checked 9 October 2026.

Lifecycle and execute()

• A Python model is a class TritonPythonModel with an optional auto_complete_config, an optional initialize(args), a required execute(requests) and an optional finalize() [1]. Objects created in auto_complete_config do not survive: that interpreter is destroyed when the function returns [1].
• initialize receives args, a dictionary of strings with the keys model_config (JSON), model_instance_kind, model_instance_device_id, model_repository, model_version and model_name [1]. Placement is the model's job: a model "can be written to respect the kind setting", and the example builds cuda:<device_id> itself [1, 7].
• In default mode execute receives a list of N requests and must return a list of exactly N responses in the same order; an element may carry an error but cannot be None, and Triton returns an error for every request if these rules are broken [1]. Request tensor data are deleted when execute returns, so request objects must not be kept [1].
• Since 24.06 a default-mode model may send its one response through InferenceResponseSender instead [1].
• When a model has no dynamic_batching (and no sequence batching), Triton core uses its dynamic batch scheduler with batching disabled, and each request is enqueued as its own payload [10]. With such a configuration each execute call therefore gets one request (inference from [10]).
• Triton never makes simultaneous execute calls on one model instance [11]; a second request for a busy single instance waits in the scheduler queue (inference).
• From r26.05 (and main) an optional is_ready() is called on v2/models/<model>/ready, with a 5 s timeout [2].

One process, and one CUDA context, per instance

• Because of the GIL, the Python backend spawns a separate process for each model instance; ONNX Runtime, TensorFlow and PyTorch backends create threads instead [1].
• That process is the stub triton_python_backend_stub, linked to one libpython [1]. The r24.10 and r25.06 READMEs say Python 3.10, the r26.05 README 3.12 [1, 2, 3]; the stub in each image links libpython3.10 (24.10) and libpython3.12 (25.06, 26.05), plus the CUDA runtime in all three [19], so the r25.06 text is stale.
• The CUDA runtime creates one primary context per device for an application, shared by its host threads [12]. Each stub process that touches a GPU therefore holds its own context on that GPU (inference from [1, 12]).

Shared memory between Triton and the stub

• Since 21.04 the backend connects user code to Triton through shared memory [1]. Each instance gets 1 MB by default and the region grows in 1 MB steps when needed; shm-default-byte-size and shm-growth-byte-size change this, and stub-timeout-seconds (default 30 s) bounds the stub connection time, all passed as --backend-config=python,<key>=<value> [1]. The source also sets shm_message_queue_size 1000 and thread-pool-size 32; the pool serves BLS requests and decoupled responses from the stub [4].
• Docker's default --shm-size of 64 MB is called "very small" [1]. Since 24.04 region names carry a UUID, so several servers can share /dev/shm; older releases need shm-region-prefix-name [1].
• For a CPU input the backend allocates the tensor inside the shared-memory pool and copies the request's data into it [4]. For a CPU output the backend copies the tensor from shared memory into Triton's response buffer [5]. GPU tensors cross the process boundary through CUDA IPC handles or a shared CUDA memory pool [4, 5].

GPU inputs and DLPack

• By default the backend moves every input to CPU before the model sees it. With the parameter FORCE_CPU_ONLY_INPUT_TENSORS set to "no", inputs arrive in CPU or GPU memory "depending on how those tensors were last used", and the model must handle both [1].
• pb_utils.Tensor.to_dlpack() and pb_utils.Tensor.from_dlpack() give zero-copy exchange with frameworks; from_dlpack also accepts objects with __dlpack__, needs C-contiguous data, and is the only route for BF16 tensors because as_numpy() does not support them [1]. is_cpu() reports the placement [1]. PyTorch's torch.utils.dlpack.from_dlpack returns a tensor that shares the memory [17].

Business Logic Scripting (BLS)

• BLS (since 21.08) lets execute send inference requests to other models served by the same Triton; it is not allowed in initialize or finalize [1]. InferenceRequest(...).exec() blocks; async_exec() returns an awaitable and needs async def execute [1].
• Optional arguments: request_id, correlation_id, model_version, flags, timeout (microseconds, 0 means none) and preferred_memory (23.04+), which picks CPU or GPU memory and a device for the returned outputs [1]. exec(decoupled=True) (23.03+) returns an iterator over a decoupled model's responses [1].
• In the source, the server process runs each BLS request through the in-process C API (TRITONSERVER_ServerInferAsync); CPU outputs are allocated in the stub's shared memory and GPU outputs in the CUDA memory pool [6]. A BLS call is thus a full Triton inference request with its own scheduling.
• Limits in r24.10: a call cycle can deadlock when no instance is free; no async BLS in decoupled mode [1].

Decoupled mode

• Set model_transaction_policy { decoupled: true }; execute gets each request's InferenceResponseSender, sends any number of responses, marks the last with TRITONSERVER_RESPONSE_COMPLETE_FINAL, and returns None [1, 8]. async def execute is supported for decoupled models since 24.04 [1].
• Triton's HTTP inference endpoint and the gRPC ModelInfer RPC cannot serve decoupled models; clients must use the gRPC bidirectional streaming RPC [8]. The HTTP generate_stream extension can return several JSON responses as server-sent events [9].

Custom execution environments

• A conda-pack archive (or unpacked directory) is named by the model parameter EXECUTION_ENV_PATH (may use $$TRITON_MODEL_DIRECTORY); its interpreter must match the stub's Python version, else a custom stub is needed, best compiled inside the NGC container [1].

Running a TensorRT engine inside model.py

• None of the three images ships the tensorrt Python module, although they ship libnvinfer10 10.5.0.18 (24.10), 10.11.0.33 (25.06) and 10.16.1.11 (26.05) [19]. PyPI has tensorrt-cu12-bindings 10.5.0 with cp310 manylinux_2_17_x86_64 wheels [16].
• TensorRT 10 inference in Python: runtime.deserialize_cuda_engine(plan), engine.create_execution_context(), context.set_input_shape(name, shape) for dynamic inputs, context.set_tensor_address(name, ptr) for every input and output (memory may come from PyTorch), then context.execute_async_v3(stream_ptr) [13, 14]. The pointer of a torch.cuda.Stream is its cuda_stream property [14]; torch.cuda.current_stream() returns the current stream [17]. get_tensor_shape(name) gives output shapes once input shapes are set [13].
• Registered memory must not be changed or freed before the stream is synchronised (inputs may be freed after the event given to set_input_consumed_event) [13]. Passing the default stream "may lead to performance issues due to additional cudaDeviceSynchronize() calls"; TensorRT asks for a non-default stream [13].
• Portability: serialised engines are specific to the exact GPU model and the platform [14]. A plan records the TensorRT major, minor, patch and build version and the compute capability; a mismatch fails deserialisation unless the plan was built version-compatible (8.6 or later, with an embedded lean runtime that must be trusted) or hardware-compatible (kAMPERE_PLUS: compute capability 8.0 or higher; kSAME_COMPUTE_CAPABILITY in the 10.16 docs), and such engines can be slower [15].

Unverified

• Whether the PyPI tensorrt-cu12 10.5.0 wheel has the 24.10 image's build number (10.5.0.18); plans check it [15].
• The GPU memory taken by one stub process's CUDA context; no source gives a figure.
• Which TensorRT 10.x release added kSAME_COMPUTE_CAPABILITY, so whether 10.5 (24.10) has it.

What this means for the diarization model

• diar_nn_g<i> has max_batch_size: 0 and no dynamic batcher, so each execute call carries one block (one OP) and the engine's second in-flight request waits in Triton's queue while the first runs (inference from [10, 11]).
• One instance per GPU means one stub process and one CUDA context per GPU, which is also what the exact lane needs: one context and one stream (inference).
• A request's SPAN is about 1.7 MB, above the 1 MB default region, so the first requests grow the region; setting shm-default-byte-size above the largest request plus response avoids growth while serving, and Docker's --shm-size must hold every instance's region (inference from [1]).
• HTTP inputs arrive in CPU memory, so FORCE_CPU_ONLY_INPUT_TENSORS gains nothing here; each block costs one copy into shared memory and one host-to-device copy in model.py; on the way back the (8, 799, 16) fp32 output (409,088 bytes) is written into shared memory and copied from there into the response (inference from [4, 5]).
• A TensorRT faster lane inside this model needs the tensorrt Python package added to the image, an engine built for the serving GPU model with the same TensorRT build that runs it, and the torch steps (window rebuild, fbank) and execute_async_v3 on one non-default torch stream, so ordering needs no extra synchronisation (inference).
• Moving to 25.06 or 26.05 changes the stub to Python 3.12; PyPI has torch 2.1.1 wheels only for cp38 to cp311 [18], so the exact lane would need a custom Python 3.10 stub or would have to stay on 24.10 (inference).

Sources

1. python_backend README - https://github.com/triton-inference-server/python_backend/blob/r24.10/README.md - applies to r24.10.
2. python_backend README - https://github.com/triton-inference-server/python_backend/blob/r26.05/README.md - applies to r26.05 (same text on main).
3. python_backend README - https://github.com/triton-inference-server/python_backend/blob/r25.06/README.md - applies to r25.06.
4. python_backend src/python_be.cc - https://github.com/triton-inference-server/python_backend/blob/r24.10/src/python_be.cc - applies to r24.10.
5. python_backend src/infer_response.cc - https://github.com/triton-inference-server/python_backend/blob/r24.10/src/infer_response.cc - applies to r24.10.
6. python_backend src/request_executor.cc - https://github.com/triton-inference-server/python_backend/blob/r24.10/src/request_executor.cc - applies to r24.10.
7. python_backend instance-kind example - https://github.com/triton-inference-server/python_backend/blob/r24.10/examples/instance_kind/model.py - applies to r24.10.
8. Decoupled Backends and Models - https://github.com/triton-inference-server/server/blob/r24.10/docs/user_guide/decoupled_models.md - applies to r24.10.
9. Generate extension - https://github.com/triton-inference-server/server/blob/r24.10/docs/protocol/extension_generate.md - applies to r24.10.
10. Triton core backend_model.cc (scheduler choice) and dynamic_batch_scheduler.cc (no batching: one request per payload) - https://github.com/triton-inference-server/core/blob/r24.10/src/backend_model.cc and https://github.com/triton-inference-server/core/blob/r24.10/src/dynamic_batch_scheduler.cc - applies to r24.10.
11. Triton backend README, threading section - https://github.com/triton-inference-server/backend/blob/r24.10/README.md - applies to r24.10.
12. CUDA C++ Programming Guide, section 3.2.1 Initialization - https://docs.nvidia.com/cuda/archive/12.6.2/cuda-c-programming-guide/index.html - applies to CUDA 12.6.2.
13. TensorRT Python API, IExecutionContext - https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/_static/python-api/infer/Core/ExecutionContext.html - applies to TensorRT 10.16.1 (10.x docs).
14. TensorRT, The Python API (inference walkthrough) - https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/inference-library/python-api-docs.html - applies to TensorRT 10.x docs.
15. TensorRT, Version Compatibility (and hardware compatibility) - https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/inference-library/version-compatibility.html - applies to TensorRT 10.x docs.
16. PyPI tensorrt-cu12-bindings 10.5.0 files - https://pypi.org/project/tensorrt-cu12-bindings/10.5.0/#files - checked 9 October 2026.
17. PyTorch torch.utils.dlpack and torch.cuda.current_stream - https://pytorch.org/docs/2.1/dlpack.html and https://pytorch.org/docs/2.1/generated/torch.cuda.current_stream.html - applies to PyTorch 2.1.
18. PyPI torch 2.1.1 files - https://pypi.org/project/torch/2.1.1/#files - checked 9 October 2026.
19. Contents of the images nvcr.io/nvidia/tritonserver:{24.10,25.06,26.05}-py3, 9 October 2026 (python3 --version, import tensorrt, ldd on the stub, dpkg -l): read from the images, not documentation.