File size: 3,267 Bytes
0810902 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 | # Hardware requirements
## The one rule
**The whole model must sit on a single device.** Piko-9b's 24 linear-attention layers keep a
recurrent state that does not survive a CPU/GPU split. `device_map="auto"` on an
undersized GPU silently offloads and the model then emits a single repeated token — fluent-looking
garbage, no error. See [troubleshooting.md](troubleshooting.md).
So the question is not "how little VRAM can I get away with" but "which precision fits entirely".
## VRAM by configuration
| Configuration | Weights | Practical VRAM | Fits on |
|---|---:|---:|---|
| bfloat16 / float16 | 19.8 GB | **~22 GB** | RTX 4090, A6000, A100, L40S |
| 8-bit (bnb) | 10.1 GB | **~12 GB** | RTX 3090, 4080, A5000 |
| 4-bit NF4 (bnb) | 5.6 GB | **~8 GB** | RTX 3060 12 GB, 4060 Ti 16 GB, 5070 Ti |
Weights column = 9,653,104,368 parameters × bytes-per-parameter. The practical column adds the
CUDA context, activations, and KV cache for a short prompt. Long prompts need more — the 8
full-attention layers carry a KV cache that grows with sequence length, while the 24
linear-attention layers keep a fixed-size state (which is why this architecture is comparatively
cheap at long context).
**Measured:** 4-bit NF4 on an RTX 5070 Ti (17.1 GB) occupied **8.1 GB** with the model resident
and idle. See [`reports/performance_report.md`](../reports/performance_report.md).
## Verified configuration
Everything measured in this repository ran on:
| | |
|---|---|
| GPU | NVIDIA GeForce RTX 5070 Ti, 17.1 GB |
| CPU RAM | 33.9 GB |
| OS | Windows 11 + WSL2 Ubuntu |
| Python | 3.12.3 |
| torch | 2.10.0+cu128 |
| transformers | 5.5.0 |
| Precision | bfloat16 weights, 4-bit NF4 at inference |
No number in this repository is extrapolated to hardware that was not used.
## CPU inference
Technically possible, practically not advisable:
* float32 weights ≈ 38.6 GB of RAM; bfloat16 on CPU is poorly supported for these kernels.
* No linear-attention fast path on CPU.
* Expect well under one token per second.
It is not a supported configuration and none of the results here were produced that way.
## Multi-GPU
Not validated, and **not recommended without testing**. Splitting layers across two GPUs is the
same class of operation that breaks under CPU offload. If you try it, run
`tests/test_text_generation.py::test_generation_is_not_degenerate` first — it exists precisely to
catch this.
Tensor parallelism in a serving engine (vLLM, SGLang) is a different mechanism and may be fine,
but was not tested here.
## Disk and load time
| | |
|---|---|
| Checkpoint size | 21.3 GB (11 shards) |
| Tokenizer | 20 MB |
| Load from NVMe | seconds to ~1 minute |
| Load from external USB | **10–20 minutes** (measured) |
Copy the checkpoint to internal storage before doing repeated work.
## Context length and memory
`max_position_embeddings` is 262,144 and no RoPE scaling is applied, so the context is
architectural rather than extended. That is a statement about the *configuration*, not about
quality: this release does not establish that output remains useful at that length. Long-context
behaviour was probed only at the lengths recorded in
[`reports/performance_report.md`](../reports/performance_report.md); everything beyond is
untested.
|