GPU Under Load, Part 2: Two Workloads That Look Identical — and Aren't

In my previous article(https://huggingface.co/blog/nazeerbashashaik/gpu-utilization-under-load), I measured a GPU under sustained load and found something surprising: while reporting 100% utilization, the card ran at less than half its rated clock speed, because it was pinned against its power limit. But that experiment used only one kind of workload — a large matrix multiplication, which leans heavily on the GPU's compute units.

That left me with a question. Was the low clock a property of the GPU under any heavy load, or a property of that particular workload? To find out, I ran a second kind of workload: one where the compute demand is low but the memory traffic is high. Then I compared the two — a compute-heavy matrix multiplication against a memory-heavy elementwise operation — across four metrics: power consumption, temperature, clock speed, and memory usage.

The answer turned out to sharpen the conclusion from the first article in a way I didn't expect.

Both workloads ran back to back on the same GPU, in the same session, with the monitoring collected in the background:

import subprocess, torch, time
 
poller = subprocess.Popen([
    "nvidia-smi",
    "--query-gpu=timestamp,temperature.gpu,power.draw,utilization.gpu,memory.used,clocks.sm",
    "--format=csv", "-l", "1", "-f", "compare_log.csv"
])
 
a = torch.randn(8000, 8000, device='cuda')
b = torch.randn(8000, 8000, device='cuda')
 
time.sleep(10)                          # baseline
 
# Phase 1: compute-bound (matrix multiplication)
end = time.time() + 60
while time.time() < end:
    c = a @ b
torch.cuda.synchronize()
 
time.sleep(10)                          # gap between phases
 
# Phase 2: memory-bound (four chained elementwise ops, little real math)
end = time.time() + 60
while time.time() < end:
    c = a + b
    c = c * 2.0
    c = c - a
    c = c + b
torch.cuda.synchronize()
 
time.sleep(10)
poller.terminate()

The surprise

Before running it, I expected the memory-bound workload to draw less power. The reasoning seemed obvious: if the GPU isn't doing much computation, it shouldn't need much power. I was wrong.

In the memory-bound case, power consumption stayed high — essentially the same as the compute-bound matrix multiplication, hovering near the card's 70-watt limit. Utilization also read 100%, exactly as it had for the matmul. By the two metrics most people check — power and utilization — the two workloads looked identical.

But they were not identical, and the difference showed up in the one metric I hadn't expected to matter: the clock speed. The compute-bound matmul ran at roughly 660 MHz. The memory-bound workload ran at roughly 1200 MHz — nearly double — on the same card, in the same session, minutes apart.

Why power stays high with little computation

The explanation is that power on a GPU is not spent only on computation. Moving data also costs power — reading and writing hundreds of millions of values per iteration keeps the memory system, buses, and caches running hard, and that draws watts even when very little actual math is happening. So the memory-bound workload sits near the power cap too, just for a different reason.

And that difference in reason is what frees the clock. A GPU's power cap squeezes the clock only when the compute units are the thing consuming the power — because clock speed and compute power are tightly coupled. In the matmul, the compute units are the load, so the card must slow the clock to stay under 70 watts. In the memory-bound workload, the compute units are mostly idle; the power is going to data movement, which doesn't scale with the clock. So the card is free to run the clock high — and it does, at times right up to its rated boost.

A note on the workload

My first attempt at a memory-bound workload was a single elementwise operation. It turned out to be too light to make the point — a single add touches too little memory to clearly stress the memory system, so it wasn't a convincing contrast to the matmul. I switched to a workload of four chained elementwise operations, which passes over the data several times per iteration and makes the memory traffic genuinely heavy. That version produced the clear separation shown above.

As in the first article, the monitoring ran as a separate background process while the workloads executed, so the measurement never competed with the work being measured. One honest limitation: the memory-bound loop did not saturate the GPU quite as continuously as the matmul — the Python loop has some overhead between operations, so utilization and power dipped briefly in the gaps between iterations.

The takeaway

The lesson that stayed with me is that power consumption on a GPU comes from two sources, not one: the computation itself, and the movement of data in and out of memory. A workload can push the card to its power limit without doing much math at all, simply by moving enough data.

That is why the clock speed, not the power draw, is what actually distinguishes the two cases. Both workloads sat near the 70-watt cap. But the memory-bound workload was free to run its clock near maximum, because the power cap only forces the clock down when the compute units are what's drawing the power — and in the memory-bound case, they weren't.

One caveat worth stating plainly: this was measured on a single card, an NVIDIA T4, which has high memory bandwidth relative to its compute capacity. A different GPU — one with far more compute power, like a datacenter training card — could well produce a different balance, and the memory-bound workload might not sit as close to the power cap. As with the first article, one card at one moment is not a universal result. But the underlying point holds regardless: to know what a GPU is really doing, utilization and power aren't enough — you have to look at the clock.