| # Mapping the compose stack to Kubernetes |
|
|
| The compose stack (`triton/compose/docker-compose.yml`, `scale.sh`) runs one Triton per GPU, the gateway in front |
| deciding which GPU gets which block, and DCGM + Prometheus + Grafana. Two things shape the mapping: the gateway keeps |
| its jobs in memory, and its numerics depend on the binaries in its image. The same mapping applies to the serving |
| kit's FastAPI service (`../../triton_inference/`), which replaces the gateway. |
|
|
| | compose | Kubernetes | notes | |
| |---|---|---| |
| | `triton-i`, `NVIDIA_VISIBLE_DEVICES=<uuid>`, cpuset | a `StatefulSet` with `resources.limits: nvidia.com/gpu: 1`, `replicas: N`, CPU requests on the GPU's NUMA node (`topologyManager: single-numa-node`) | one Pod = one GPU = one `diar_nn_g<i>` model at device 0; a StatefulSet gives each Pod the stable name the gateway's `--triton` list needs | |
| | model repository per GPU (`make_repository.py --gpus i --device 0`) | one ConfigMap or init container per ordinal writing `diar_nn_g<ordinal>` + `resample_cpu` | the bundle (weights) as a read-only volume | |
| | health check `/v2/health/ready`, 300 s start period | `startupProbe` covering the self-test (every batch shape against the release, about a minute per GPU), `readinessProbe` `/v2/health/ready`, `livenessProbe` `/v2/health/live` | a model whose self-test fails never loads, so the Pod never becomes ready | |
| | `stop_grace_period: 45s`, `--exit-timeout-secs=30` | `terminationGracePeriodSeconds: 45`; Triton drains in-flight requests on SIGTERM | plus a `PodDisruptionBudget`; the gateway resends any request cut by a restart (`--triton-retry-s`) | |
| | the gateway on the host, `--triton url0,url1,...` (one per GPU) | one `Deployment` with `replicas: 1` on a CPU node, `--triton` listing the StatefulSet Pods' DNS names (headless Service) | jobs live in the gateway's memory: several replicas need sticky routing by job id, or each replica gets its own Triton Pods. Numerics: the gateway image must carry the same glibc, numpy OpenBLAS and libsndfile binaries as the release's environment (the service image does), and the identity check (`triton/identity_check.sh`) or the kit's `diar gate` should pass in it before use | |
| | `/ready` (503 while any model is away) | the gateway's `readinessProbe` | the gateway itself waits for Triton; clients back off on 429/503 | |
| | spool directory (0700) | an `emptyDir` (or local SSD) volume, `--max-spool-gb` below its size | uploads never sit in memory | |
| | `--token` / `ANUVANI_DIAR_TOKEN` | a `Secret` as an environment variable; TLS at the Ingress | never expose the gateway without a token | |
| | dcgm-exporter | the NVIDIA GPU Operator's dcgm-exporter DaemonSet | same counters (`dcgm_counters.csv`) | |
| | Prometheus, Grafana | kube-prometheus-stack, `ServiceMonitor`s for each Triton Pod and the gateway; import `grafana/dashboards/anuvani_diar.json` | | |
| | `scale.sh N` | `kubectl scale statefulset --replicas=N` and the gateway's `--triton` list (`--gpus auto` finds the models) | the gateway needs a restart to see a new GPU; jobs survive it through the client's resubmission | |
|
|
| Sizing: each GPU's Triton model needs about 12 GB of GPU memory; throughput grows with the number of GPUs, and the |
| gateway's post-processing uses every CPU core it gets, so give it a node with many cores rather than pinning it to a |
| few. Measure a deployment with the kit's load test (`../../triton_inference/tools/bench.py`). |
|
|
