The GPU Metrics the Cloud Won't Let You See
Part 3: DCGM, deep telemetry, and the difference between root and control
In my first two articles(https://huggingface.co/blog/nazeerbashashaik/gpu-utilization-under-load, https://huggingface.co/blog/nazeerbashashaik/gpu-workload-comparison) I read GPU metrics with nvidia-smi, and both times I ran into the same limit: it tells you the GPU is busy, but not how fully the compute units are actually working. The tool that goes deeper is NVIDIA's DCGM (Data Center GPU Manager), which can report SM occupancy and tensor-core activity — the metrics that answer "is the GPU doing meaningful work, or just technically busy?"
But when I tried DCGM on Google Colab, it wouldn't run properly — Colab's sandboxed environment doesn't give you enough access to sustain a DCGM monitoring session. So I rented a dedicated GPU node on RunPod (an RTX A5000) to try it on real hardware, where I'd have root access and, I assumed, full control over the GPU.
That assumption turned out to be the interesting part.
What worked
On the rented node, DCGM came alive in a way it never could on Colab. It installed cleanly, the monitoring daemon started and stayed running, and dcgmi dmon streamed the basic fields — utilization, memory, power, temperature — sample after sample, with no dropped connection. Everything Colab couldn't sustain, this node did without complaint.
The surprise
Then I asked for the fields I'd actually rented the node to see — SM activity, SM occupancy, tensor-core activity. They came back N/A.
That was the surprise. I had root. I was on a dedicated GPU node, not a shared sandbox. And DCGM still wouldn't give me the deep metrics. The error even said it needed the host engine "running as root" — but it was running as root. Something else was blocking it.
Tracking down why
The error message was a red herring. It said DCGM needed the host engine "running as root" — but I was root, and the engine was running as root. So I stopped trusting the message and started asking what "root" actually means here.
The answer is the crux of the whole thing: my root access was root inside the pod — the container — not root on the physical host machine underneath it. A rented cloud pod is a container sharing a host you don't control. I could do anything I wanted within my own container, but the GPU's deepest settings live one level below that, on the host, where I had no authority.
Checking further confirmed it exactly. In /proc/driver/nvidia/params, the host driver reported RmProfilingAdminOnly: 1 — a setting that restricts profiling metrics to the host administrator. That flag is set on the physical machine, by the provider, and nothing I could do from inside my container would change it. The deep metrics weren't blocked because I lacked root; they were blocked because the host had locked profiling, and my container couldn't reach across that boundary.
The takeaway
The lesson from this exercise is a distinction that isn't obvious until you hit it: to read a GPU's deep-level metrics, access to the container or VM is not enough — you need access to the underlying host machine.
Basic monitoring travels anywhere the tool can reach the GPU. Utilization, memory, power, temperature — DCGM streamed all of those on a rented cloud pod without trouble. But the metrics that actually answer "how hard are the compute units working" — SM occupancy, tensor-core activity — are gated at the host driver level. A shared cloud instance, however much root you have inside it, sits above that gate and can't reach it.
For anyone planning to monitor GPUs in the cloud, that's the practical consequence: you can watch the basic health of a rented GPU, but the deep profiling metrics require hardware you fully control — a bare-metal or dedicated host, not a shared container. It's the difference between renting a room and owning the building: you can do a lot inside your room, but you can't touch the wiring in the walls.
This completes the arc of the three GPU articles. The first two used nvidia-smi and found its conceptual limit — it tells you the GPU is busy but not how fully. This one went looking for the tool that shows what nvidia-smi can't, and found a practical limit instead: the metric exists, DCGM can ask for it, but cloud infrastructure won't hand it over unless you own the host.
Note on tools: run on a rented RunPod RTX A5000 node, using NVIDIA DCGM. The diagnostic work and analysis are my own; AI assistance was used in preparing the write-up.