base_model: microsoft/FastContext-1.0-4B-SFT
license: mit
library_name: gguf
tags:
- gguf
- rocmfp4
- qwen3
- fastcontext
- subagent
- repository-exploration
- coder
- agentic
- imatrix
- strix-halo
- amd
- rocm
- vulkan
language:
- en
base_model_relation: quantized
▗▇▇▇▇▇▇▇▖
▗█▘▝██████▖
▗▛ ▝██████▆▆▆▆▆▆▆▆▆▆▅
▟▛ ▗█████████████████▙▖
▄▄▄▄▄▟▛ ▟████████████████████▖
▗██▌ ▚▖ ▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔█▘
▗████▖ ▜▖ ▗█▘
▜█████▙ ▜▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▀▀▀▀▀▜▙
▜█████▙ ▝████████████▛ ▜▙
▜█████▙ ▝██████████▛ ▃ ▜▙
▀█████▙▖ ▝████████▘ ▟█▙ ▀▙
▝██████▖ ▝▜█████▘ ▟███▙▂▂▂▂▐█
▟███████▖ ▜███▘ ▗███████████▛
▟█████████▄ ▜▛ ▗███████████▀
▝█████▀ ▗▛ ▗██████▀▀▀▀▀▘
▜██▘ ▗▛ ▟█████▛▘
▜█▇▇▇▇▇▇▇▇▇█▖ ▟█████▛
▝█▖ ▟█████▛
▝███████▀
██████████████ ██████████████ ██████████████ ██████████████ ██████████████ ██████████████ ██████████████ ██████████████ ██████████████ ██████████████ ██████████████ ██████████████ ██████████████ ██████████████ ██████████████ ████████████████████████████ ██████████████ ██████████████ ██████████████ ██████████████ ██████████████ ██████████████ ██████████████ ██████████████ ██████████████ ██████████████ ██████████████ ██████████████ ██████████████ ██████████████ ██████████████
FORMAT ROCmFP4 4-BIT |
PRECISION ~4.5 BPW |
ARCH QWEN3 DENSE |
CONTEXT 256 K |
PARAMS 4B DENSE |
DRAFT NO MTP |
BACKEND VULKAN0 |
LICENSE MIT |
The custom
q4_0_rocmfp4 / q4_0_rocmfp4_fast tensor types will not load in stock llama.cpp, LM Studio, or Ollama. Build/run with charlie12345/ROCmFPX · branch mtp-rocmfp4-strix.
Experimental AMD Strix Halo (gfx1151) quant of microsoft/FastContext-1.0-4B-SFT — Microsoft's repository-exploration subagent for coding agents. Instead of one model both exploring the repo and solving the task, FastContext is invoked on demand by a main agent, fires parallel read-only tool calls (READ / GLOB / GREP), and returns compact file paths + line ranges as focused context. Architecturally it's a plain Qwen3 dense 4B (Qwen3ForCausalLM, 36 layers, hidden 2560, 256K context, MIT-licensed), here in the custom ROCmFP4 4-bit format, imatrix-quantized.
Two builds. The ★ speed build keeps the quality lever that's actually felt — genuine f16 embeddings (from BF16), which also serve as the output head since the model ties them — on the fast single-scale q4_0_rocmfp4_fast body + a code-weighted imatrix (see §04): the best speed/quality balance for Strix Halo. The Q6 · agent fidelity build uses the 6-bit q6_0_rocmfpx body on the ROCmFPX agent profile (which protects the structured-output pathways — attention K/V at Q6_K/Q5_K, more FFN-down boosted) + a bm25-weighted imatrix: a bit larger/slower, but the closest to BF16 for precise tool-call/code output. Both have the Qwen (ChatML) chat template baked in — just pass --jinja.
tie_word_embeddings=True, so there's no separate output head — the token-embedding tensor doubles as the lm-head. Setting --token-embedding-type f16 therefore gives an f16 embedding and f16 output head in one (no headQ6 variant needed — f16 already beats Q6 there).
Run from the folder holding the .gguf (the Qwen ChatML template is baked in — just pass --jinja):
env HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
llama-server \
-m FastContext-1.0-4B-SFT-ROCmFP4-STRIX-embF16-imatrix.gguf \
--alias fastcontext-4b \
--host 0.0.0.0 \
--port 8080 \
-c 262144 \
-ctk f16 \
-ctv f16 \
--temp 0.7 \
--top-p 0.8 \
--top-k 20 \
-dev Vulkan0 \
-ngl 999 \
-fa on \
-b 2048 \
-ub 256 \
-t 16 \
-tb 16 \
-cpent 256 \
-ctxcp 32 \
--cache-reuse 256 \
--cache-ram 65536 \
--jinja \
--parallel 1 \
--metrics \
--no-mmap
--spec-* / --spec-type draft-mtp flags — this arch has no MTP head (see §04). It's already fast on its own.
FastContext isn't a general chat model — it's a repository-exploration subagent meant to be called by your main coding agent, not driven directly. The intended loop: the main agent delegates "find the relevant context for X" → FastContext issues parallel read-only tool calls (READ, GLOB, GREP) → returns compact file paths + line ranges, which the main agent folds into its own context to do the actual work. The point is to keep repo-exploration tokens out of the main agent's window.
- Chat template: Qwen (ChatML) is baked into the GGUF — just pass
--jinja. - Tool calling: it emits structured
READ/GLOB/GREPcalls — wire those tools into your harness and use a Qwen/Hermes-style tool-call parser so they're parsed rather than printed. See the upstream model card for the exact subagent protocol + tool schema (it expects a specific invocation format). - Sampling: temp
0.7, top-p0.8, top-k20(Qwen3 instruct defaults) — already set in §02.
--cache-reuse / --cache-ram) keeps repeated exploration over the same repo cheap.
This is the best speed/quality balance in ROCmFP4 — by design, not the absolute fastest. It keeps the one quality lever that's actually felt — genuine f16 embeddings, which on this model double as the output head (tie_word_embeddings=True), so a single f16 tensor sharpens both the input and output side at near-zero decode cost (it's a lookup, not a matmul) — on top of the fast single-scale q4_0_rocmfp4_fast body + a code-weighted imatrix. A leaner Q5-embedding build would shave a couple tok/s but degrades that lever; we keep full f16.
We didn't re-run the entire rocmfp4 lever sweep on this 4B. We ran it exhaustively on the larger Qwen3.6-27B — KL divergence vs the BF16 reference plus llama-bench decode across an all-dual-scale body, selective higher-precision tensors, and full f16 embeddings. The finding there: an all-dual-scale body and selective higher-precision tensors both cost decode speed for a KL improvement that sat inside the measurement noise, so the fast single-scale body + f16 embeddings is the balance point. That conclusion carries to FastContext — same format, same kernels — so we ship the one build that lands on it rather than a slower variant that wins KL only inside the noise.
…-Q6_0_ROCMFPX_AGENT-bm25imatrix.gguf in this repo — a 6-bit ROCmFPX body on the agent profile (structured-output tensors protected), the closest to BF16 here. If you want even higher, a Q6_K / Q8 GGUF of the base from microsoft/FastContext-1.0-4B-SFT also runs on this same fork.
Fast on its own. ~68 t/s short-context decode on a Ryzen AI Max+ 395 (Vulkan0, measured llama-bench tg128). It's a 4B dense Qwen3 with no MTP head, so there's no speculative decoding — it doesn't need it, and at 4B it's a cheap explorer you can run several of in parallel.
groups_merged + froggeric code/technical, via froggeric/imatrix), computed on this model's BF16. We did not run a separate imatrix-vs-no-imatrix ablation on this 4B; at 4+ bpw imatrix is a free polish, not a transformation. Scope note: any fidelity-vs-BF16 figures are a held-out measurement, not an absolute coding benchmark.
# 0) convert the safetensors -> BF16 GGUF (plain qwen3 dense; no MTP, tied embeddings)
python convert_hf_to_gguf.py FastContext-1.0-4B-SFT/ --outtype bf16 --outfile FastContext-1.0-4B-SFT-BF16.gguf
# 1) imatrix on the BF16 (general+code: Kalomaze groups_merged + froggeric code/technical)
llama-imatrix -m FastContext-1.0-4B-SFT-BF16.gguf -f general+code-calib.txt -o fastcontext-4b.imatrix -c 512 -ngl 999
# 2) THE ONE BUILD: fast single-scale STRIX body + f16 tied emb/head + imatrix (the ★ file) — the balance point (§04).
# tie_word_embeddings=True -> --token-embedding-type f16 also gives an f16 output head; no --output-tensor-type.
llama-quantize --token-embedding-type f16 --imatrix fastcontext-4b.imatrix \
FastContext-1.0-4B-SFT-BF16.gguf FastContext-1.0-4B-SFT-ROCmFP4-STRIX-embF16-imatrix.gguf Q4_0_ROCMFP4_STRIX
Experimental research build for AMD Strix Halo — hardware/driver/prompt-sensitive, may not reproduce elsewhere. Not native FP4 tensor-core execution.
Derivative quantization — verify the base model's license before redistribution / use.