faxenoff commited on
Commit
31cc2c2
·
verified ·
1 Parent(s): 2720243

docs: measured throughput + memory footprint

Browse files
Files changed (1) hide show
  1. README.md +17 -3
README.md CHANGED
@@ -69,9 +69,23 @@ with a residual gap to the 7B's exactness on L0 (the next lever is on-policy dis
69
 
70
  The stages this model serves are **prefill-bound with short outputs** — the regime where shrinking
71
  the model (not speculative decoding) is the right lever. In the UltraCode daemon it loads into a
72
- dedicated `.enrich` worker (~0.9 GB VRAM, Q8_0 + KV) that co-resides with the main LLM, so the
73
- label stages run on the 0.6B while paragraph/prose stages stay on the larger model. Measured prefill
74
- throughput on the label batches: ~8–12k tok/s.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
75
 
76
  ## Usage (llama.cpp)
77
 
 
69
 
70
  The stages this model serves are **prefill-bound with short outputs** — the regime where shrinking
71
  the model (not speculative decoding) is the right lever. In the UltraCode daemon it loads into a
72
+ dedicated `.enrich` worker that co-resides with the main LLM, so label stages run on the 0.6B while
73
+ paragraph/prose stages stay on the larger model.
74
+
75
+ Measured on a laptop RTX 5060 (8 GB), llama.cpp CUDA, Q8_0, `n_ctx=8192`, `n_batch=2048`:
76
+
77
+ | What | Rate |
78
+ |---|---|
79
+ | Prefill, 850-token label prompt | **~17 000 tok/s** |
80
+ | Decode, single stream | **301 tok/s** |
81
+ | Label stage end-to-end, batched (prompt + generated) | **~2 600 tok/s** |
82
+
83
+ The end-to-end figure is the one to plan capacity with: label prompts are short and the stage runs
84
+ many of them concurrently, so wall-clock is dominated by prefill, not by decode.
85
+
86
+ **Memory:** 604 MiB of weights, but **~2.7 GB resident** at `n_ctx=8192` once the KV cache and
87
+ compute buffers are allocated. Size the deployment from that number, not from the file. A smaller
88
+ `n_ctx` reduces it proportionally — 8192 is chosen here to keep eight label sequences in flight.
89
 
90
  ## Usage (llama.cpp)
91