# Hardware requirements

## The one rule

**The whole model must sit on a single device.** Piko-9b's 24 linear-attention layers keep a
recurrent state that does not survive a CPU/GPU split. `device_map="auto"` on an
undersized GPU silently offloads and the model then emits a single repeated token — fluent-looking
garbage, no error. See [troubleshooting.md](troubleshooting.md).

So the question is not "how little VRAM can I get away with" but "which precision fits entirely".

## VRAM by configuration

| Configuration | Weights | Practical VRAM | Fits on |
|---|---:|---:|---|
| bfloat16 / float16 | 19.8 GB | **~22 GB** | RTX 4090, A6000, A100, L40S |
| 8-bit (bnb) | 10.1 GB | **~12 GB** | RTX 3090, 4080, A5000 |
| 4-bit NF4 (bnb) | 5.6 GB | **~8 GB** | RTX 3060 12 GB, 4060 Ti 16 GB, 5070 Ti |

Weights column = 9,653,104,368 parameters × bytes-per-parameter. The practical column adds the
CUDA context, activations, and KV cache for a short prompt. Long prompts need more — the 8
full-attention layers carry a KV cache that grows with sequence length, while the 24
linear-attention layers keep a fixed-size state (which is why this architecture is comparatively
cheap at long context).

**Measured:** 4-bit NF4 on an RTX 5070 Ti (17.1 GB) occupied **8.1 GB** with the model resident
and idle. See [`reports/performance_report.md`](../reports/performance_report.md).

## Verified configuration

Everything measured in this repository ran on:

| | |
|---|---|
| GPU | NVIDIA GeForce RTX 5070 Ti, 17.1 GB |
| CPU RAM | 33.9 GB |
| OS | Windows 11 + WSL2 Ubuntu |
| Python | 3.12.3 |
| torch | 2.10.0+cu128 |
| transformers | 5.5.0 |
| Precision | bfloat16 weights, 4-bit NF4 at inference |

No number in this repository is extrapolated to hardware that was not used.

## CPU inference

Technically possible, practically not advisable:

* float32 weights ≈ 38.6 GB of RAM; bfloat16 on CPU is poorly supported for these kernels.
* No linear-attention fast path on CPU.
* Expect well under one token per second.

It is not a supported configuration and none of the results here were produced that way.

## Multi-GPU

Not validated, and **not recommended without testing**. Splitting layers across two GPUs is the
same class of operation that breaks under CPU offload. If you try it, run
`tests/test_text_generation.py::test_generation_is_not_degenerate` first — it exists precisely to
catch this.

Tensor parallelism in a serving engine (vLLM, SGLang) is a different mechanism and may be fine,
but was not tested here.

## Disk and load time

| | |
|---|---|
| Checkpoint size | 21.3 GB (11 shards) |
| Tokenizer | 20 MB |
| Load from NVMe | seconds to ~1 minute |
| Load from external USB | **10–20 minutes** (measured) |

Copy the checkpoint to internal storage before doing repeated work.

## Context length and memory

`max_position_embeddings` is 262,144 and no RoPE scaling is applied, so the context is
architectural rather than extended. That is a statement about the *configuration*, not about
quality: this release does not establish that output remains useful at that length. Long-context
behaviour was probed only at the lengths recorded in
[`reports/performance_report.md`](../reports/performance_report.md); everything beyond is
untested.