HiramKHackenbacker commited on
Commit
a600e03
·
verified ·
1 Parent(s): c926276

docs: performance section leads with the preferred mf64 bundle; superseded figures follow, explicitly labeled

Browse files
Files changed (1) hide show
  1. README.md +33 -33
README.md CHANGED
@@ -121,40 +121,9 @@ Measured on a **16 GB Apple silicon Mac (M2 Pro, macOS 27 beta)**, sequential en
121
  `reset()` between samples. Load excluded from `s/row`; sample 1 excluded as a cache-warm
122
  outlier.
123
 
124
- ### `gpu-pipelined/`
125
 
126
- | | |
127
- |---|---|
128
- | Cold load | 21.3 s |
129
- | Guided JSON parse | **10/10** |
130
- | Enum-clean | **10/10** |
131
- | s/row (long samples) | **17.88** |
132
- | s/row (short samples) | **8.59** |
133
- | Decode | **45.2–46.6 tok/s** |
134
- | TTFT | **6.5–23.3 s** |
135
- | Peak footprint | 0.16 GB |
136
- | Max RSS | 10.61 GB |
137
-
138
- Decode is healthy for the size — an MoE with ~1B active parameters decoding faster than a
139
- dense 2.6B. **The per-sample cost is prefill, not decode**: at 17.88 s/row it is roughly
140
- 10× the LFM2.5-1.2B, entirely because of the missing prefill function.
141
-
142
- Unguided, free-form generation from a fixed prompt, 900-token cap, with the chat template
143
- patched to terminate the think block (unpatched, these runs return 0 characters).
144
-
145
- | depth | prompt tokens | TTFT | decode | generated | wall | peak footprint | max RSS |
146
- |---|---|---|---|---|---|---|---|
147
- | 3.4k | 3,249 | 70.05 s | 47.0 tok/s | 900 | 89.0 s | 0.21 GB | 10.58 GB |
148
- | 8k | 7,673 | 165.11 s | 45.2 tok/s | 519 | 176.8 s | 0.27 GB | 11.03 GB |
149
-
150
- **Needle-in-haystack recall at 8k: 3/3.** Three distinctive facts planted at 10/50/90% of
151
- the filler, strict scoring; returned at 7,813 prompt tokens in a 27-token answer
152
- (TTFT 168.09 s, 45.3 tok/s). 15k was not run: the manifest ceiling on `gpu-pipelined/` is
153
- 8192, and S=1 prefill at that depth would cost roughly five minutes.
154
-
155
- ### `mf64/`
156
-
157
- Same machine, same harness, same prompts, same schema, chat template unmodified.
158
 
159
  | | `gpu-pipelined/` | `mf64/` |
160
  |---|---|---|
@@ -186,6 +155,37 @@ in depth.
186
  includes the graph compile. Subsequent loads were **22.9–26.2 s**. Budget roughly 17 GB of
187
  on-disk compile cache in addition to the 9.4 GB bundle.
188
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
189
  ## Status
190
 
191
  | Artifact | Status |
 
121
  `reset()` between samples. Load excluded from `s/row`; sample 1 excluded as a cache-warm
122
  outlier.
123
 
124
+ ### `mf64/` — preferred bundle
125
 
126
+ Chat template unmodified. Same harness and prompts as the superseded bundle (below); the comparison column restates its figures.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
127
 
128
  | | `gpu-pipelined/` | `mf64/` |
129
  |---|---|---|
 
155
  includes the graph compile. Subsequent loads were **22.9–26.2 s**. Budget roughly 17 GB of
156
  on-disk compile cache in addition to the 9.4 GB bundle.
157
 
158
+ ### `gpu-pipelined/` — superseded
159
+
160
+ | | |
161
+ |---|---|
162
+ | Cold load | 21.3 s |
163
+ | Guided JSON parse | **10/10** |
164
+ | Enum-clean | **10/10** |
165
+ | s/row (long samples) | **17.88** |
166
+ | s/row (short samples) | **8.59** |
167
+ | Decode | **45.2–46.6 tok/s** |
168
+ | TTFT | **6.5–23.3 s** |
169
+ | Peak footprint | 0.16 GB |
170
+ | Max RSS | 10.61 GB |
171
+
172
+ Decode is healthy for the size — an MoE with ~1B active parameters decoding faster than a
173
+ dense 2.6B. **The per-sample cost is prefill, not decode**: at 17.88 s/row it is roughly
174
+ 10× the LFM2.5-1.2B, entirely because of the missing prefill function.
175
+
176
+ Unguided, free-form generation from a fixed prompt, 900-token cap, with the chat template
177
+ patched to terminate the think block (unpatched, these runs return 0 characters).
178
+
179
+ | depth | prompt tokens | TTFT | decode | generated | wall | peak footprint | max RSS |
180
+ |---|---|---|---|---|---|---|---|
181
+ | 3.4k | 3,249 | 70.05 s | 47.0 tok/s | 900 | 89.0 s | 0.21 GB | 10.58 GB |
182
+ | 8k | 7,673 | 165.11 s | 45.2 tok/s | 519 | 176.8 s | 0.27 GB | 11.03 GB |
183
+
184
+ **Needle-in-haystack recall at 8k: 3/3** (this superseded bundle). Three distinctive facts planted at 10/50/90% of
185
+ the filler, strict scoring; returned at 7,813 prompt tokens in a 27-token answer
186
+ (TTFT 168.09 s, 45.3 tok/s). 15k was not run on this superseded bundle (`mf64/` above passes 3/3 at 15k): its manifest ceiling is
187
+ 8192, and S=1 prefill at that depth would cost roughly five minutes.
188
+
189
  ## Status
190
 
191
  | Artifact | Status |