src/triton_inference/docs/reference/01-python-backend.md

# Python backend: execution model, data path, BLS, decoupled mode, TensorRT inside it

Scope: how Triton's Python backend runs `model.py`, how tensors reach it, BLS, decoupled mode, and running a TensorRT
engine from inside it. Facts apply to Triton 24.10 (`r24.10` branches of python_backend, server, core and backend)
unless `r25.06` or `r26.05` is named; TensorRT facts come from the 10.x docs (10.16.1). Checked 9 October 2026.

## Lifecycle and `execute()`

- A Python model is a class `TritonPythonModel` with an optional `auto_complete_config`, an optional `initialize(args)`,
  a required `execute(requests)` and an optional `finalize()` [1]. Objects created in `auto_complete_config` do not
  survive: that interpreter is destroyed when the function returns [1].
- `initialize` receives `args`, a dictionary of strings with the keys `model_config` (JSON), `model_instance_kind`,
  `model_instance_device_id`, `model_repository`, `model_version` and `model_name` [1]. Placement is the model's job:
  a model "can be written to respect the kind setting", and the example builds `cuda:<device_id>` itself [1, 7].
- In default mode `execute` receives a list of N requests and must return a list of exactly N responses in the same
  order; an element may carry an error but cannot be `None`, and Triton returns an error for every request if these
  rules are broken [1]. Request tensor data are deleted when `execute` returns, so request objects must not be kept [1].
- Since 24.06 a default-mode model may send its one response through `InferenceResponseSender` instead [1].
- When a model has no `dynamic_batching` (and no sequence batching), Triton core uses its dynamic batch scheduler with
  batching disabled, and each request is enqueued as its own payload [10]. With such a configuration each `execute`
  call therefore gets one request (inference from [10]).
- Triton never makes simultaneous execute calls on one model instance [11]; a second request for a busy single
  instance waits in the scheduler queue (inference).
- From `r26.05` (and `main`) an optional `is_ready()` is called on `v2/models/<model>/ready`, with a 5 s timeout [2].

## One process, and one CUDA context, per instance

- Because of the GIL, the Python backend spawns a separate process for each model instance; ONNX Runtime,
  TensorFlow and PyTorch backends create threads instead [1].
- That process is the stub `triton_python_backend_stub`, linked to one `libpython` [1]. The `r24.10` and `r25.06`
  READMEs say Python 3.10, the `r26.05` README 3.12 [1, 2, 3]; the stub in each image links `libpython3.10` (24.10)
  and `libpython3.12` (25.06, 26.05), plus the CUDA runtime in all three [19], so the `r25.06` text is stale.
- The CUDA runtime creates one primary context per device for an application, shared by its host threads [12].
  Each stub process that touches a GPU therefore holds its own context on that GPU (inference from [1, 12]).

## Shared memory between Triton and the stub

- Since 21.04 the backend connects user code to Triton through shared memory [1]. Each instance gets 1 MB by
  default and the region grows in 1 MB steps when needed; `shm-default-byte-size` and `shm-growth-byte-size`
  change this, and `stub-timeout-seconds` (default 30 s) bounds the stub connection time, all passed as
  `--backend-config=python,<key>=<value>` [1]. The source also sets `shm_message_queue_size` 1000 and
  `thread-pool-size` 32; the pool serves BLS requests and decoupled responses from the stub [4].
- Docker's default `--shm-size` of 64 MB is called "very small" [1]. Since 24.04 region names carry a UUID, so
  several servers can share `/dev/shm`; older releases need `shm-region-prefix-name` [1].
- For a CPU input the backend allocates the tensor inside the shared-memory pool and copies the request's data into
  it [4]. For a CPU output the backend copies the tensor from shared memory into Triton's response buffer [5].
  GPU tensors cross the process boundary through CUDA IPC handles or a shared CUDA memory pool [4, 5].

## GPU inputs and DLPack

- By default the backend moves every input to CPU before the model sees it. With the parameter
  `FORCE_CPU_ONLY_INPUT_TENSORS` set to `"no"`, inputs arrive in CPU or GPU memory "depending on how those tensors
  were last used", and the model must handle both [1].
- `pb_utils.Tensor.to_dlpack()` and `pb_utils.Tensor.from_dlpack()` give zero-copy exchange with frameworks;
  `from_dlpack` also accepts objects with `__dlpack__`, needs C-contiguous data, and is the only route for BF16
  tensors because `as_numpy()` does not support them [1]. `is_cpu()` reports the placement [1]. PyTorch's
  `torch.utils.dlpack.from_dlpack` returns a tensor that shares the memory [17].

## Business Logic Scripting (BLS)

- BLS (since 21.08) lets `execute` send inference requests to other models served by the same Triton; it is not
  allowed in `initialize` or `finalize` [1]. `InferenceRequest(...).exec()` blocks; `async_exec()` returns an
  awaitable and needs `async def execute` [1].
- Optional arguments: `request_id`, `correlation_id`, `model_version`, `flags`, `timeout` (microseconds, 0 means none)
  and `preferred_memory` (23.04+), which picks CPU or GPU memory and a device for the returned outputs [1].
  `exec(decoupled=True)` (23.03+) returns an iterator over a decoupled model's responses [1].
- In the source, the server process runs each BLS request through the in-process C API
  (`TRITONSERVER_ServerInferAsync`); CPU outputs are allocated in the stub's shared memory and GPU outputs in the
  CUDA memory pool [6]. A BLS call is thus a full Triton inference request with its own scheduling.
- Limits in `r24.10`: a call cycle can deadlock when no instance is free; no async BLS in decoupled mode [1].

## Decoupled mode

- Set `model_transaction_policy { decoupled: true }`; `execute` gets each request's `InferenceResponseSender`, sends
  any number of responses, marks the last with `TRITONSERVER_RESPONSE_COMPLETE_FINAL`, and returns `None` [1, 8].
  `async def execute` is supported for decoupled models since 24.04 [1].
- Triton's HTTP inference endpoint and the gRPC `ModelInfer` RPC cannot serve decoupled models; clients must use the
  gRPC bidirectional streaming RPC [8]. The HTTP `generate_stream` extension can return several JSON responses as
  server-sent events [9].

## Custom execution environments

- A `conda-pack` archive (or unpacked directory) is named by the model parameter `EXECUTION_ENV_PATH` (may use
  `$$TRITON_MODEL_DIRECTORY`); its interpreter must match the stub's Python version, else a custom stub is needed,
  best compiled inside the NGC container [1].

## Running a TensorRT engine inside `model.py`

- None of the three images ships the `tensorrt` Python module, although they ship `libnvinfer10`
  10.5.0.18 (24.10), 10.11.0.33 (25.06) and 10.16.1.11 (26.05) [19]. PyPI has `tensorrt-cu12-bindings` 10.5.0 with
  `cp310` `manylinux_2_17_x86_64` wheels [16].
- TensorRT 10 inference in Python: `runtime.deserialize_cuda_engine(plan)`, `engine.create_execution_context()`,
  `context.set_input_shape(name, shape)` for dynamic inputs, `context.set_tensor_address(name, ptr)` for every input
  and output (memory may come from PyTorch), then `context.execute_async_v3(stream_ptr)` [13, 14]. The pointer of a
  `torch.cuda.Stream` is its `cuda_stream` property [14]; `torch.cuda.current_stream()` returns the current stream [17].
  `get_tensor_shape(name)` gives output shapes once input shapes are set [13].
- Registered memory must not be changed or freed before the stream is synchronised (inputs may be freed after the
  event given to `set_input_consumed_event`) [13]. Passing the default stream "may lead to performance issues due to
  additional cudaDeviceSynchronize() calls"; TensorRT asks for a non-default stream [13].
- Portability: serialised engines are specific to the exact GPU model and the platform [14]. A plan records the
  TensorRT major, minor, patch and build version and the compute capability; a mismatch fails deserialisation unless
  the plan was built version-compatible (8.6 or later, with an embedded lean runtime that must be trusted) or
  hardware-compatible (`kAMPERE_PLUS`: compute capability 8.0 or higher; `kSAME_COMPUTE_CAPABILITY` in the 10.16
  docs), and such engines can be slower [15].

## Unverified

- Whether the PyPI `tensorrt-cu12` 10.5.0 wheel has the 24.10 image's build number (10.5.0.18); plans check it [15].
- The GPU memory taken by one stub process's CUDA context; no source gives a figure.
- Which TensorRT 10.x release added `kSAME_COMPUTE_CAPABILITY`, so whether 10.5 (24.10) has it.

## What this means for the diarization model

- `diar_nn_g<i>` has `max_batch_size: 0` and no dynamic batcher, so each `execute` call carries one block (one OP)
  and the engine's second in-flight request waits in Triton's queue while the first runs (inference from [10, 11]).
- One instance per GPU means one stub process and one CUDA context per GPU, which is also what the exact lane needs:
  one context and one stream (inference).
- A request's SPAN is about 1.7 MB, above the 1 MB default region, so the first requests grow the region; setting
  `shm-default-byte-size` above the largest request plus response avoids growth while serving, and Docker's
  `--shm-size` must hold every instance's region (inference from [1]).
- HTTP inputs arrive in CPU memory, so `FORCE_CPU_ONLY_INPUT_TENSORS` gains nothing here; each block costs one copy
  into shared memory and one host-to-device copy in `model.py`; on the way back the (8, 799, 16) fp32 output
  (409,088 bytes) is written into shared memory and copied from there into the response (inference from [4, 5]).
- A TensorRT faster lane inside this model needs the `tensorrt` Python package added to the image, an engine built for
  the serving GPU model with the same TensorRT build that runs it, and the torch steps (window rebuild, fbank) and
  `execute_async_v3` on one non-default torch stream, so ordering needs no extra synchronisation (inference).
- Moving to 25.06 or 26.05 changes the stub to Python 3.12; PyPI has torch 2.1.1 wheels only for cp38 to cp311
  [18], so the exact lane would need a custom Python 3.10 stub or would have to stay on 24.10 (inference).

## Sources

1. python_backend README - https://github.com/triton-inference-server/python_backend/blob/r24.10/README.md - applies to r24.10.
2. python_backend README - https://github.com/triton-inference-server/python_backend/blob/r26.05/README.md - applies to r26.05 (same text on main).
3. python_backend README - https://github.com/triton-inference-server/python_backend/blob/r25.06/README.md - applies to r25.06.
4. python_backend `src/python_be.cc` - https://github.com/triton-inference-server/python_backend/blob/r24.10/src/python_be.cc - applies to r24.10.
5. python_backend `src/infer_response.cc` - https://github.com/triton-inference-server/python_backend/blob/r24.10/src/infer_response.cc - applies to r24.10.
6. python_backend `src/request_executor.cc` - https://github.com/triton-inference-server/python_backend/blob/r24.10/src/request_executor.cc - applies to r24.10.
7. python_backend instance-kind example - https://github.com/triton-inference-server/python_backend/blob/r24.10/examples/instance_kind/model.py - applies to r24.10.
8. Decoupled Backends and Models - https://github.com/triton-inference-server/server/blob/r24.10/docs/user_guide/decoupled_models.md - applies to r24.10.
9. Generate extension - https://github.com/triton-inference-server/server/blob/r24.10/docs/protocol/extension_generate.md - applies to r24.10.
10. Triton core `backend_model.cc` (scheduler choice) and `dynamic_batch_scheduler.cc` (no batching: one request per payload) - https://github.com/triton-inference-server/core/blob/r24.10/src/backend_model.cc and https://github.com/triton-inference-server/core/blob/r24.10/src/dynamic_batch_scheduler.cc - applies to r24.10.
11. Triton backend README, threading section - https://github.com/triton-inference-server/backend/blob/r24.10/README.md - applies to r24.10.
12. CUDA C++ Programming Guide, section 3.2.1 Initialization - https://docs.nvidia.com/cuda/archive/12.6.2/cuda-c-programming-guide/index.html - applies to CUDA 12.6.2.
13. TensorRT Python API, IExecutionContext - https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/_static/python-api/infer/Core/ExecutionContext.html - applies to TensorRT 10.16.1 (10.x docs).
14. TensorRT, The Python API (inference walkthrough) - https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/inference-library/python-api-docs.html - applies to TensorRT 10.x docs.
15. TensorRT, Version Compatibility (and hardware compatibility) - https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/inference-library/version-compatibility.html - applies to TensorRT 10.x docs.
16. PyPI `tensorrt-cu12-bindings` 10.5.0 files - https://pypi.org/project/tensorrt-cu12-bindings/10.5.0/#files - checked 9 October 2026.
17. PyTorch `torch.utils.dlpack` and `torch.cuda.current_stream` - https://pytorch.org/docs/2.1/dlpack.html and https://pytorch.org/docs/2.1/generated/torch.cuda.current_stream.html - applies to PyTorch 2.1.
18. PyPI torch 2.1.1 files - https://pypi.org/project/torch/2.1.1/#files - checked 9 October 2026.
19. Contents of the images `nvcr.io/nvidia/tritonserver:{24.10,25.06,26.05}-py3`, 9 October 2026 (`python3 --version`, `import tensorrt`, `ldd` on the stub, `dpkg -l`): read from the images, not documentation.