loom-fun/ggml-org-Flint-loom

Flint (Q4_K_M) for AMD Loom

A Loom package (.loom programs and a model.json card) for the unmodified GGUF checkpoint ggml-org/Flint-GGUF (Flint-Q4_K_M.gguf), generated on an AMD Ryzen AI Max+ 395 "Strix Halo" with the loom-skills workflow. Speculative decoding is built in and on by default: the incoai/Flint-DFlash2-GGUF drafter proposes 3 tokens per step and the model verifies them in the same pass. Every committed token is the model's own choice, so outputs are identical to decoding without drafting (checked: 13/13 completions identical, 0 near-tie, acceptance 68% of 2709 proposals).

What's new in this version (loom-skills #30, issues #26–#29):

• ~1k-token prompts are fast again with the 4096-token MoE prefill chunk: a coalesced, bank-conflict-free prefill router (4.4 -> 0.96 ms per 2048-token layer) and padding rows the GEMMs skip (a 1026-token prompt runs as a 2048-row epoch). Interleaved A/B vs the previous card's code: prefill 1k 721 -> 918 tok/s (+27 %), 2k +15 %, 8k +17 %, 32k +13 %.
• 256-row WMMA GEMM tiles for the dense projections and a DFlash proposal block of depth + 1 tokens.

The weights are not included. loom-fetch (from the Loom SDK, installed with its hf extra) installs the programs from this repository and the GGUF and drafter from their own repositories, checked against model.json.lock:

pip install "hrx-loom[hf]"
loom-fetch https://huggingface.co/loom-fun/ggml-org-Flint-loom/resolve/main/model.json --materialize=copy --output=flint
loom-serve --model=flint/model.json --device=amdgpu --memory_bytes=67000000000

The server listens on 127.0.0.1:8080 and answers OpenAI-style /v1/chat/completions requests (streaming, tool calls, reasoning). Context: 262,144 tokens.

Performance vs llama.cpp (both drafting)

This build vs llama.cpp ROCm b1355 with its own best speculative setting (same GGUF and drafter), measured in the same session with one engine loaded at a time: two runs each, means (4 streams: three runs, medians). Loom numbers are client-side; llama.cpp's are its server-side timings, which slightly favours llama.cpp.

metric	llama.cpp	Loom	margin
decode tok/s (short prompt)	67.3	96.9	+44 %
decode tok/s @2k context	54.7	76.3	+39 %
decode tok/s @32k context	49.5	47.8	-3 %
prefill tok/s, 1k doc	1081.8	847.5	-22 %
prefill tok/s, 2k doc	1079.0	1316.4	+22 %
prefill tok/s, 8k doc	1094.1	1424.7	+30 %
prefill tok/s, 16k doc	929.3	1359.0	+46 %
prefill tok/s, 32k doc	667.6	1193.9	+79 %
4 streams, steady aggregate tok/s	124.2	134.2	+8 %

Correctness (this build, drafting on)

loom-benchmark gates, LEVEL=quick: PASS level=quick 536s {'spec': 'PASS', 'golden': 'PASS', 'compare': 'PASS', 'consistency': 'PASS', 'agent': 'PASS', 'sampling': 'PASS', 'needle': 'PASS'}

• Golden vs llama.cpp: 24 prompts, 11 exact token-for-token matches, every divergence a near-tie (max log-prob gap 0.616).
• Equivalence: PASS: 13/13 completions identical, 0 near-tie, acceptance 68% of 2709 proposals
• Compare (chat) vs llama.cpp: 24/24 pass, including the 15.5k-token summary that was over the near-tie rule on the previous card (llama.cpp's own top-1 for it moves with its -ub/-fa settings).
• Where llama.cpp still leads: ~1k-token prompts, and decode at 32k context by a few percent (both engines' long-context decode varies run to run with how many drafted tokens are accepted).

Files

• model.json / model.json.lock: the card and the pinned source files (sha256 checked by loom-fetch).
• program/*.loom, weights.loom: the generated Loom programs.
• tokenizer.json: from RecursiveSI/flint.
• llm_config.json: the configuration the programs were generated from.
• info.json: every number above.