Hugging Face's logo

jcbtc
/
Step-3.7-Flash-ROCmFPX-Q3-QualityPlus

Text Generation
GGUF
English
llama.cpp
stepfun
step3p7
step-3.7
step-3.7-flash
mtp
speculative-decoding
rocm
vulkan
rocmfpx
fpx3
q3
q3_0_rocmfpx
qualityplus
amd
ryzen-ai-max-395
strix-halo
agentic
tool-calling
long-context
imatrix
conversational
Model card Files
xet
 Community
1
jcbtc commited on
Commit
2de9afc
·
verified ·
1 Parent(s): 97f12c2

Update Step 3.7 Q3 card for 256K context and credits

Browse files
Files changed (1) hide show
  1. README.md +20 -11
README.md CHANGED
@@ -37,7 +37,7 @@ tags:
37
 
38
  This is an extremely high quality FPX3 / ROCmFPX Q3 GGUF build of `stepfun-ai/Step-3.7-Flash`, tuned for AMD Strix Halo local serving with Step MTP.
39
 
40
- The goal is simple: keep Step 3.7 Flash useful at 64K+ context on a 128 GB unified-memory machine without falling back to a nearly-FP4-sized "Q3" file. This release is a true tight Q3-weight build: `3.57 BPW`, `81.77 GiB` of language-model shards, and strong agent/tool behavior in local evals.
41
 
42
  Use this if you want the Step 3.7 behavior profile, MTP support, and a much smaller local footprint than the stock GGUF Q3_K_L or ROCmFP4 STRIX_LEAN builds.
43
 
@@ -53,7 +53,7 @@ This FPX3/Q3 QualityPlus recipe was built for that constraint:
53
  - `81.77 GiB` total language GGUF shards
54
  - `16.31%` smaller than the local ROCmFP4 STRIX_LEAN build
55
  - `14.35%` smaller than StepFun's original `Q3_K_L` GGUF split
56
- - 64K one-slot serving profile with q8_0 target KV and q8_0 draft KV
57
  - Step MTP Q8 draft support through `draft-mtp`
58
  - fixed Step tool/chat template using native `tool_response` observations and protocol-boundary escaping
59
 
@@ -80,8 +80,7 @@ Local quality results on AMD Ryzen AI Max+ 395 / Strix Halo:
80
  | Benchmark | Result | Notes |
81
  | --- | ---: | --- |
82
  | Tool-Eval full, 69 scenarios | [`88/100`, `122/138` raw points](evals/tool-eval-q3-qualityplus.json) | Same headline score as the recorded Step ROCmFP4 tool-eval row |
83
- | HermesAgent-20, best Q3 template run | `85/100` | `13.40 min`, `35.31 tok/s` decode, `96.37 GiB` peak pooled GPU |
84
- | HermesAgent-20, native tool-response template run | `82/100` | `12.82 min`, `35.76 tok/s` decode, `96.30 GiB` peak pooled GPU |
85
 
86
  The best recorded Q3 HermesAgent-20 run was very close to the local BF16 Qwen3.6 27B MTP reference row:
87
 
@@ -124,6 +123,14 @@ Compared with the local ROCmFP4 Step build:
124
 
125
  At 128K, MTP initialized but produced no accepted drafts in that particular row, so treat the 128K decode number as an effective no-draft long-context decode reference.
126
 
 
 
 
 
 
 
 
 
127
  ## Files
128
 
129
  Published shard names intentionally match the model name:
@@ -144,10 +151,10 @@ The Step MTP draft model is not duplicated here. Use the Q8 draft from [`notSnix
144
 
145
  ## Recommended Serving Profile
146
 
147
- The locally tested 64K profile:
148
 
149
  ```text
150
- context: 65536
151
  slots: 1
152
  backend: Vulkan0 target + Vulkan0 draft
153
  MTP: --spec-type draft-mtp
@@ -158,6 +165,7 @@ speculative.p_split: 0.10
158
  batch / ubatch: 8192 / 2048
159
  target KV: q8_0 / q8_0
160
  draft KV: q8_0 / q8_0
 
161
  sampler: temperature 1.0, top_p 0.95, min_p 0.0, repeat_penalty 1.0
162
  reasoning: on, DeepSeek format
163
  chat template: Step native tool_response template with protocol-boundary escaping
@@ -172,7 +180,7 @@ Example shape:
172
  --host 127.0.0.1 \
173
  --port 8080 \
174
  --jinja \
175
- -c 65536 \
176
  --reasoning on \
177
  --reasoning-format deepseek \
178
  --reasoning-budget -1 \
@@ -183,8 +191,9 @@ Example shape:
183
  -ub 2048 \
184
  --parallel 1 \
185
  --no-mmap \
186
- --ctk q8_0 \
187
- --ctv q8_0 \
 
188
  --spec-draft-model Step-3.7-Flash-MTP-Q8_0.gguf \
189
  --spec-draft-device Vulkan0 \
190
  --spec-type draft-mtp \
@@ -221,8 +230,8 @@ Converter-reported size: `83726.08 MiB / 3.57 BPW`, 9 shards.
221
 
222
  - Base model: [`stepfun-ai/Step-3.7-Flash`](https://huggingface.co/stepfun-ai/Step-3.7-Flash)
223
  - MTP draft GGUF source: [`notSnix/Step-3.7-Flash-MTP-Draft-GGUF`](https://huggingface.co/notSnix/Step-3.7-Flash-MTP-Draft-GGUF)
224
- - Runtime family: [`ciru-ai/ROCmFPX`](https://github.com/ciru-ai/ROCmFPX)
225
- - Quantization, Strix Halo profile, and local benchmark work: Crown / Ciru
226
 
227
  ## Caveats
228
 
 
37
 
38
  This is an extremely high quality FPX3 / ROCmFPX Q3 GGUF build of `stepfun-ai/Step-3.7-Flash`, tuned for AMD Strix Halo local serving with Step MTP.
39
 
40
+ The goal is simple: keep Step 3.7 Flash useful at 256K context, keep the quality as high as possible, and keep it as small as possible. This release is a true tight Q3-weight build: `3.57 BPW`, `81.77 GiB` of language-model shards, and strong agent/tool behavior in local evals.
41
 
42
  Use this if you want the Step 3.7 behavior profile, MTP support, and a much smaller local footprint than the stock GGUF Q3_K_L or ROCmFP4 STRIX_LEAN builds.
43
 
 
53
  - `81.77 GiB` total language GGUF shards
54
  - `16.31%` smaller than the local ROCmFP4 STRIX_LEAN build
55
  - `14.35%` smaller than StepFun's original `Q3_K_L` GGUF split
56
+ - up to 256K one-slot serving profile with q8_0 target KV and q8_0 draft KV
57
  - Step MTP Q8 draft support through `draft-mtp`
58
  - fixed Step tool/chat template using native `tool_response` observations and protocol-boundary escaping
59
 
 
80
  | Benchmark | Result | Notes |
81
  | --- | ---: | --- |
82
  | Tool-Eval full, 69 scenarios | [`88/100`, `122/138` raw points](evals/tool-eval-q3-qualityplus.json) | Same headline score as the recorded Step ROCmFP4 tool-eval row |
83
+ | HermesAgent-20, best Q3 run | `85/100` | `13.40 min`, `35.31 tok/s` decode, `96.37 GiB` peak pooled GPU |
 
84
 
85
  The best recorded Q3 HermesAgent-20 run was very close to the local BF16 Qwen3.6 27B MTP reference row:
86
 
 
123
 
124
  At 128K, MTP initialized but produced no accepted drafts in that particular row, so treat the 128K decode number as an effective no-draft long-context decode reference.
125
 
126
+ 256K load proof:
127
+
128
+ | Context | Proof | Memory state |
129
+ | ---: | --- | ---: |
130
+ | `262144` | target + Q8 MTP draft loaded, one slot, `draft-mtp`, `/v1/models` reports `n_ctx=262144` and `n_ctx_train=262144` | `~99.04 GiB` pooled GPU used, `~16 GiB` system RAM available |
131
+
132
+ The 256K row is a load/allocation proof, not a 256K prompt prefill benchmark.
133
+
134
  ## Files
135
 
136
  Published shard names intentionally match the model name:
 
151
 
152
  ## Recommended Serving Profile
153
 
154
+ The locally tested long-context profile:
155
 
156
  ```text
157
+ context: up to 262144
158
  slots: 1
159
  backend: Vulkan0 target + Vulkan0 draft
160
  MTP: --spec-type draft-mtp
 
165
  batch / ubatch: 8192 / 2048
166
  target KV: q8_0 / q8_0
167
  draft KV: q8_0 / q8_0
168
+ prompt cache: disabled for 256K fit runs
169
  sampler: temperature 1.0, top_p 0.95, min_p 0.0, repeat_penalty 1.0
170
  reasoning: on, DeepSeek format
171
  chat template: Step native tool_response template with protocol-boundary escaping
 
180
  --host 127.0.0.1 \
181
  --port 8080 \
182
  --jinja \
183
+ -c 262144 \
184
  --reasoning on \
185
  --reasoning-format deepseek \
186
  --reasoning-budget -1 \
 
191
  -ub 2048 \
192
  --parallel 1 \
193
  --no-mmap \
194
+ --cache-ram 0 \
195
+ -ctk q8_0 \
196
+ -ctv q8_0 \
197
  --spec-draft-model Step-3.7-Flash-MTP-Q8_0.gguf \
198
  --spec-draft-device Vulkan0 \
199
  --spec-type draft-mtp \
 
230
 
231
  - Base model: [`stepfun-ai/Step-3.7-Flash`](https://huggingface.co/stepfun-ai/Step-3.7-Flash)
232
  - MTP draft GGUF source: [`notSnix/Step-3.7-Flash-MTP-Draft-GGUF`](https://huggingface.co/notSnix/Step-3.7-Flash-MTP-Draft-GGUF)
233
+ - ROCmFPX creator: Charlie, `charlie12345` / `@italianclownz`, [`charlie12345/ROCmFPX`](https://github.com/charlie12345/ROCmFPX)
234
+ - Quantization, the ROCmFPX Step 3.7 Q3 QualityPlus recipe, Strix Halo profile, and local benchmark work: Crown / Ciru
235
 
236
  ## Caveats
237