Spaces:

---

zero-gpu-explorers

/

README

like 102

Running

Oh. I was able to reproduce this on my side as well:

I tested this with spaces==0.51.0, PyTorch 2.11.0, and the following fixed Qwen-Image-2512 revision:

25468b98e3276ca6700de15c6628e51b7de54a26

My result was that the startup-side automatic size calculation did select xlarge, but the resolved size did not appear to reach the scheduler request. The auto/default path consequently received a 48 GB worker and stalled during the cold-worker tensor transfer. Explicitly specifying size="xlarge" allocated a 96 GB worker and completed the same transfer.

I made a minimal reproducer here:

https://huggingface.co/spaces/John6666/zerogpu_size_repro

The source Space is intentionally hosted without ZeroGPU. To run it:

1. Duplicate the Space.
2. Select ZeroGPU in the duplicate’s hardware settings.
3. Keep spaces==0.51.0.
4. Wait for model loading and startup tensor packing to complete.
5. Factory reboot before each test, then press only one of the two buttons.

The two paths differ primarily in the decorator:

@spaces.GPU(duration=120)
def probe_auto():
    ...

versus:

@spaces.GPU(duration=120, size="xlarge")
def probe_xlarge():
    ...

The public reproducer does not monkey-patch spaces internals.

In a separate instrumented run, I observed the following:

The packed tensors totaled 57,698,921,142 bytes, or 53.736 GiB.

On the auto/default path, the transfer stopped at 49,951,517,366 bytes, or 46.521 GiB. That is 86.573% of the tensor-transfer stage. Since the displayed ZeroGPU initialization progress appears to reserve its first 20% for cleanup/CUDA initialization and the remaining 80% for tensor transfer, this corresponds to approximately:

20% + (86.573% × 80%) = 89.26%

That is very close to the persistent 89% position reported here.

On the explicit-xlarge path, the scheduler request contained "xlarge", a 94.971 GiB worker was allocated, all 53.736 GiB of tensors were transferred in about 3.6 seconds, and the decorated function body was reached successfully.

I also compared the published spaces==0.50.4 and spaces==0.51.0 wheels. The tensor-packing implementation itself appears unchanged. The relevant change is in spaces.zero.client.schedule().

In 0.50.4, the default GPU size was resolved in schedule() and the resolved value was then passed to the API:

gpu_size = gpu_size if gpu_size is not None else default_gpu_size()

res, meta = api_client().schedule(
    ...
    gpu_size=gpu_size,
)

In 0.51.0, the resolution was moved into a new duration helper:

duration_seconds = get_duration_seconds(duration, gpu_size)

res, meta = api_client().schedule(
    ...
    gpu_size=gpu_size,
)

Inside that helper, gpu_size is resolved only as a local variable:

def get_duration_seconds(duration, gpu_size):
    gpu_size = gpu_size if gpu_size is not None else default_gpu_size()
    ...
    return duration_seconds

The resolved size is therefore used when calculating duration, but the original gpu_size value in schedule() can remain None when the scheduler API is called.

This matches the runtime behavior I observed:

startup default_gpu_size: xlarge
outgoing scheduler gpu_size: None
allocated worker: 48 GB

With an explicit decorator argument, the outgoing value remains "xlarge" and the 96 GB worker is allocated correctly.

Based on these observations, this looks like an automatic-size propagation regression introduced in 0.51.0, rather than a change in the tensor-packing algorithm itself.

The 48 GB worker exposes about 47.4 GiB to CUDA, while the packed tensors require 53.736 GiB. The stopping point at 46.521 GiB is therefore consistent with reaching the allocation limit. I did not capture the original CUDA exception directly, so I would not claim a confirmed OOM traceback. However, the current threaded disk → pinned-memory → CUDA transfer path may be leaving another thread waiting on its buffer queue after the CUDA-side failure, which would explain why this appears as a persistent progress stall rather than a normal allocation error.

For now, the following was a sufficient workaround in my reproduction:

@spaces.GPU(size="xlarge")
def generate(...):
    ...

This uses the current documented ZeroGPU size-selection API:

https://huggingface.co/docs/hub/spaces-zerogpu#gpu-size-selection

The Callable[[int]] error on the ZEROGPU_V2=false path appears to be a separate issue. That annotation is invalid because Callable requires both an argument list and a return type, and the same annotation is also present in older spaces releases. I would therefore treat it as an existing legacy-path bug that prevents using the fallback, rather than as part of the new automatic-size regression.

The remaining implementation question seems to be whether passing gpu_size=None to the scheduler is intended to select the startup-computed default, or whether 0.51.0 should restore the client-side propagation that was present in 0.50.4.