Correct the VRAM figure: allocated bytes undercount the card by 2 GiB
Browse files
README.md
CHANGED
|
@@ -139,16 +139,36 @@ Individually, the forge under the vendor's own sampler:
|
|
| 139 |
what these clips do and do not establish. All thirty clips are in `samples/`,
|
| 140 |
and the Space plays them side by side under its **BF16 vs 4-bit** tab.
|
| 141 |
|
| 142 |
-
## Memory
|
| 143 |
|
| 144 |
-
|
|
| 145 |
-
|---|---:|---|
|
| 146 |
-
|
|
| 147 |
-
|
|
| 148 |
-
|
| 149 |
-
|
| 150 |
-
|
| 151 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 152 |
|
| 153 |
## Use it
|
| 154 |
|
|
|
|
| 139 |
what these clips do and do not establish. All thirty clips are in `samples/`,
|
| 140 |
and the Space plays them side by side under its **BF16 vs 4-bit** tab.
|
| 141 |
|
| 142 |
+
## Memory — what the card actually needs
|
| 143 |
|
| 144 |
+
| | resident (default) | dequantized |
|
| 145 |
+
|---|---:|---:|
|
| 146 |
+
| PyTorch allocated, peak | 8.48 GiB | ~22.3 GiB |
|
| 147 |
+
| PyTorch reserved, peak | 9.33 GiB | — |
|
| 148 |
+
| **what `nvidia-smi` shows** | **9.70 GiB** | — |
|
| 149 |
+
|
| 150 |
+
**Size your card from the last row.** The first is `max_memory_allocated`, which
|
| 151 |
+
counts only live allocator blocks — it misses the CUDA context and everything
|
| 152 |
+
the caching allocator reserved and has not handed back, and it undercounts by
|
| 153 |
+
nearly 2 GiB here. Earlier versions of this card quoted that number, and anyone
|
| 154 |
+
who bought an 8 GB card on the strength of it would have been wrong.
|
| 155 |
+
|
| 156 |
+
Measured on a **Tesla V100-SXM2-16GB (cc 7.0)** with a 55-token prompt, through
|
| 157 |
+
the ComfyUI-faithful path that left-pads to 1024 tokens. So:
|
| 158 |
+
|
| 159 |
+
* **16 GB and up** — comfortable.
|
| 160 |
+
* **12 GB** — fits.
|
| 161 |
+
* **10 GB** — fits, with roughly 300 MB of headroom. Nothing else on the card.
|
| 162 |
+
* **8 GB** — does not fit.
|
| 163 |
+
|
| 164 |
+
The two modes produce `torch.equal` conditioning, so the choice is footprint
|
| 165 |
+
against speed and never quality. Resident dequantizes inside `forward`, which
|
| 166 |
+
costs time on short prompts; dequantized builds one dense BF16 model at load and
|
| 167 |
+
then needs a card that can hold 26 GB.
|
| 168 |
+
|
| 169 |
+
These figures are for the **text encoder alone**. Generating video also needs the
|
| 170 |
+
DiT and the VAEs, which this repository does not contain — the pipeline in the
|
| 171 |
+
Space loads a `Q3_K_M` DiT alongside it.
|
| 172 |
|
| 173 |
## Use it
|
| 174 |
|