Update model card for DFlash runtime and benchmarks

Document experimental ROCmFPX runtime requirements, official DFlash provenance, additive files, benchmark speeds, recommended six-token draft window, and cache-rotation patch.

PROVENANCE_SHA256SUMS

@@ -6,3 +6,7 @@ f2d6c59f89a63c7331c768c515a691c7b193f0e55d9083842a13a60134d7654f  calibration/Mu
 733b6eee371dfe2c49bd338dbf597bcdae1e5297de53cfa98f61a4d7e6e196ae  artifacts/Muse-Glimmer-30B-ROCmFP4.gguf
 74a5f469f17420d01a837400a2ef19d8191dcb1b479fd7c0be0c862901879915  artifacts/Muse-Glimmer-30B-ROCmFP8.gguf
 6b80db6b044d9a3999b539eec140dc4d29279d53c18f42f80a557d3e6736123a  artifacts/Muse-Glimmer-30B-ROCmFP4-Q6-QUALITY.gguf
+fd88d337eb84f8d0e6ba33a7684d7efa6722d4460ba4d6badca9699418392a84  source/Muse-Glimmer-30B-assistant/model.safetensors
+dc4ecbb844a961bbb535a57258085b95e24f54c437599296133aa1a3dbcad891  intermediate/Muse-Glimmer-30B-DFlash-BF16.gguf
+1e2f0d9de05251b05ed9c6f67f208105b30f0cd51112307a400b59936dd8fc41  artifacts/Muse-Glimmer-30B-DFlash-ROCmFP4.gguf
+ae1d9ffbcbc3af92fbe50ecf62d5892c619ec76b1ca3198e25b240f86a11974f  artifacts/Muse-Glimmer-30B-DFlash-ROCmFP8.gguf

README.md

@@ -1,5 +1,7 @@
 ---
-base_model: meta-models/Muse-Glimmer-30B
+base_model:
+- meta-models/Muse-Glimmer-30B
+- meta-models/Muse-Glimmer-30B-assistant
 base_model_relation: quantized
 library_name: llama.cpp
 pipeline_tag: image-text-to-text
@@ -16,6 +18,8 @@ tags:
 - multimodal
 - muse-glimmer
 - conversational
+- dflash
+- speculative-decoding
 ---
 
 # Muse-Glimmer-30B ROCmFPX GGUF
@@ -33,9 +37,10 @@ files; read the compatibility section before downloading.
 > llama.cpp does not implement the ROCmFP4/ROCmFP8 tensor layouts, while the
 > pinned ROCmFPX base predates Muse Glimmer support. Apply the included
 > `ROCmFPX-Muse-Glimmer.patch` to ROCmFPX commit `00d54526e…`, then build that
-> checkout. The patch adds the upstream Muse text and vision implementation,
-> ports it to ROCmFPX's older APIs, and supplies the FP16 sparse-attention mask
-> required by its multimodal flash-attention path.
+> checkout. The patch adds the upstream Muse text, vision, and DFlash support,
+> ports it to ROCmFPX's older APIs, supplies the FP16 sparse-attention mask
+> required by its multimodal flash-attention path, and backports the DFlash
+> injected-cache rotation fix required when using quantized KV caches.
 
 ## Files
 
@@ -45,6 +50,11 @@ files; read the compatibility section before downloading.
 | `Muse-Glimmer-30B-ROCmFP4-Q6-QUALITY.gguf` | `Q4_0_ROCMFP4_COHERENT` | 14.94 GiB | 4.60 | Yes | 14.0 tok/s |
 | `Muse-Glimmer-30B-ROCmFP8.gguf` | `Q8_0_ROCMFPX` | 26.77 GiB | 8.25 | No | 7.8 tok/s |
 | `mmproj-Muse-Glimmer-30B-BF16.gguf` | BF16 vision projector | 3.59 GiB | — | — | — |
+| `Muse-Glimmer-30B-DFlash-ROCmFP4.gguf` | `Q4_0_ROCMFP4_STRIX` drafter | 1.39 GiB | 4.63 | No | 28.3 tok/s¹ |
+| `Muse-Glimmer-30B-DFlash-ROCmFP8.gguf` | `Q8_0_ROCMFPX` drafter | 2.47 GiB | 8.25 | No | 27.2 tok/s¹ |
+
+¹ End-to-end target decode with `Muse-Glimmer-30B-ROCmFP4.gguf`, DFlash enabled,
+and a six-token draft window; see the benchmark section below.
 
 Suggested choices:
 
@@ -53,6 +63,10 @@ Suggested choices:
 - **ROCmFP4-Q6-QUALITY:** coherence-biased build. Dual-scale FP4 throughout the
   body with Q6_K token embeddings.
 - **ROCmFP8:** high-fidelity 8.25-bpw reference build.
+- **DFlash ROCmFP4:** recommended drafter on Strix Halo. It is smaller and was
+  slightly faster than the FP8 drafter in the measured six-token configuration.
+- **DFlash ROCmFP8:** higher-precision drafter reference; useful for comparing
+  acceptance behavior and tuning on other hardware.
 
 The BF16 projector works with all three text models.
 
@@ -68,6 +82,11 @@ Both FP4 models use the same GGUF importance matrix:
 `Q8_0_ROCMFPX` does not consume importance weights, so the FP8 reference was
 intentionally built without an iMatrix.
 
+The two DFlash drafters were quantized directly from Meta's official assistant
+checkpoint without an iMatrix. Their role is proposal generation: every draft
+is verified by the main model, so drafter quantization changes acceptance and
+speed rather than bypassing the target model's output decision.
+
 ## Compatibility
 
 These files use experimental ROCmFPX tensor types and **will not load in stock
@@ -81,6 +100,8 @@ The validated runtime was built from:
   [`62bf73d25c53b8161f8a22894d4f90c4aebbd7d0`](https://github.com/ggml-org/llama.cpp/commit/62bf73d25c53b8161f8a22894d4f90c4aebbd7d0)
 - small compatibility adaptations for the older ROCmFPX chat, model, and
   multimodal APIs
+- the upstream DFlash quantized-cache rotation fix, adapted to this older graph
+  API; without it, Q4_0 draft KV caches load but produce near-zero acceptance
 
 `ROCmFPX-Muse-Glimmer.patch` contains the complete patch against the pinned
 ROCmFPX base. The runtime was built with ROCm and Vulkan backends; the reported
@@ -114,6 +135,7 @@ to the exact pinned commit; `git apply --check` was verified before publishing.
 ```bash
 hf download vmlinux/Muse-Glimmer-30B-ROCmFPX-GGUF \
   Muse-Glimmer-30B-ROCmFP4.gguf \
+  Muse-Glimmer-30B-DFlash-ROCmFP4.gguf \
   mmproj-Muse-Glimmer-30B-BF16.gguf \
   --local-dir ./Muse-Glimmer-30B-ROCmFPX
 ```
@@ -137,6 +159,23 @@ Vision:
   -dev ROCm0 -ngl all -c 8192 -cnv -st
 ```
 
+DFlash speculative decoding (recommended starting point):
+
+```bash
+./llama-cli \
+  -m ./Muse-Glimmer-30B-ROCmFPX/Muse-Glimmer-30B-ROCmFP4.gguf \
+  --model-draft ./Muse-Glimmer-30B-ROCmFPX/Muse-Glimmer-30B-DFlash-ROCmFP4.gguf \
+  --spec-type draft-dflash \
+  -dev ROCm0 -ngl all \
+  --spec-draft-device ROCm0 --spec-draft-ngl all \
+  -ctk q4_0 -ctv q4_0 \
+  --spec-draft-type-k q4_0 --spec-draft-type-v q4_0 \
+  --spec-draft-n-max 6 --spec-draft-n-min 0 \
+  --spec-draft-p-min 0.0 --spec-draft-p-split 0.10 \
+  --no-spec-draft-backend-sampling \
+  -c 8192 -cnv
+```
+
 ## Validation
 
 All three files completed clean, single-turn ROCm generation with every layer
@@ -154,6 +193,25 @@ These are short smoke-test measurements, not a formal benchmark. Host: AMD
 Strix Halo `gfx1151`, 128 GiB unified memory, ROCm backend, 1,024-token text
 context (2,048 for vision).
 
+### DFlash benchmark
+
+The additive DFlash benchmark used the default ROCmFP4 target, batch size 1,
+greedy decoding, three text prompts, 256 generated tokens per prompt, a
+2,048-token context, flash attention, full `ROCm0` offload, and Q4_0 target and
+draft KV caches. Values are arithmetic means of the three runs.
+
+| Mode | Draft window | Decode | Speedup | Per-prompt range | Weighted draft acceptance |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| No speculation | — | 13.7 tok/s | 1.00× | 13.7–13.7 | — |
+| DFlash ROCmFP4 | 6 | **28.3 tok/s** | **2.07×** | 24.3–31.8 | 35.8% (519/1,448) |
+| DFlash ROCmFP8 | 6 | 27.2 tok/s | 1.99× | 24.1–33.0 | 35.7% (519/1,454) |
+| DFlash ROCmFP4 | 15 | 24.6 tok/s | 1.80× | 18.5–30.7 | 17.1% (543/3,171) |
+| DFlash ROCmFP8 | 15 | 26.3 tok/s | 1.92× | 19.2–34.6 | 19.5% (563/2,891) |
+
+This is a small local throughput benchmark, not a universal performance claim.
+Acceptance depends strongly on prompt and generation content. Six draft tokens
+was the best tested practical default on this host; tune it for your workload.
+
 Additional verification:
 
 - 731 tensors and `muse-glimmer` architecture in every text GGUF
@@ -162,16 +220,19 @@ Additional verification:
 - `test-quantize-fns` and architecture tests passed
 - SHA-256 hashes supplied in `SHA256SUMS`
 
-## MTP / speculative decoding
+## DFlash, not MTP
 
-No MTP or DFlash drafter is included. The downloaded base repository contained
-the main Muse Glimmer weights but no separate `MuseGlimmerAssistantModel`
-checkpoint. The patched runtime includes Muse DFlash loader support for use if
-a compatible drafter is published separately.
+These new companion files are converted from Meta's official
+[`Muse-Glimmer-30B-assistant`](https://huggingface.co/meta-models/Muse-Glimmer-30B-assistant)
+checkpoint. It is a five-layer `MuseGlimmerAssistantModel` using DFlash block
+diffusion with a trained block size of 16. Run it with
+`--spec-type draft-dflash`; it is not an MTP checkpoint and should not be run
+with `draft-mtp`.
 
 ## Provenance
 
 - Source revision: `f84ecc3a0ea984a4c04542a84269e3d065350a6e`
+- DFlash source revision: `2c86316d689027b91123638739743fef1d425233`
 - Conversion: upstream llama.cpp `d2f83055dca6dd009d8a52bdff792fbb286f4444`
 - Every published GGUF is covered by `SHA256SUMS`; source, intermediate, and
   calibration hashes are retained in `PROVENANCE_SHA256SUMS`

ROCmFPX-Muse-Glimmer.patch

@@ -544,6 +544,66 @@ index 3200a993b..5b2e0b7ab 100644
          ),
  
          MODEL_TENSOR.V_MM_POST_NORM: (
+diff --git a/scripts/convert_hf_to_gguf_modular.py b/scripts/convert_hf_to_gguf_modular.py
+new file mode 100644
+index 000000000..ec01d29a5
+--- /dev/null
++++ b/scripts/convert_hf_to_gguf_modular.py
+@@ -0,0 +1,54 @@
++#!/usr/bin/env python3
++"""Convert models implemented by ROCmFPX's modular conversion package."""
++
++from __future__ import annotations
++
++import argparse
++import logging
++from pathlib import Path
++import sys
++
++sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
++
++from conversion import ModelType, get_model_architecture, get_model_class
++from conversion.base import ModelBase, gguf
++
++
++def main() -> None:
++    parser = argparse.ArgumentParser(
++        description="Convert a Hugging Face checkpoint using ROCmFPX's modular converter."
++    )
++    parser.add_argument("model", type=Path)
++    parser.add_argument("--outfile", type=Path, required=True)
++    parser.add_argument("--outtype", choices=["f16", "bf16", "auto"], default="auto")
++    parser.add_argument("--target-model-dir", type=Path, default=None)
++    parser.add_argument("--dry-run", action="store_true")
++    parser.add_argument("--use-temp-file", action="store_true")
++    args = parser.parse_args()
++
++    logging.basicConfig(level=logging.INFO)
++
++    ftype_map = {
++        "f16": gguf.LlamaFileType.MOSTLY_F16,
++        "bf16": gguf.LlamaFileType.MOSTLY_BF16,
++        "auto": gguf.LlamaFileType.GUESSED,
++    }
++
++    hparams = ModelBase.load_hparams(args.model, False)
++    arch = get_model_architecture(hparams, ModelType.TEXT)
++    logging.getLogger("hf-to-gguf").info("Model architecture: %s", arch)
++    model_class = get_model_class(arch, mmproj=False)
++
++    model = model_class(
++        args.model,
++        ftype_map[args.outtype],
++        args.outfile,
++        use_temp_file=args.use_temp_file,
++        dry_run=args.dry_run,
++        target_model_dir=args.target_model_dir,
++    )
++    model.write()
++
++
++if __name__ == "__main__":
++    main()
 diff --git a/src/llama-arch.cpp b/src/llama-arch.cpp
 index 17908449b..1d9552074 100644
 --- a/src/llama-arch.cpp
@@ -601,6 +661,63 @@ index c33f8c247..594affda4 100644
          case LLM_ARCH_PLM:
          case LLM_ARCH_CHATGLM:
          case LLM_ARCH_GRANITE:
+diff --git a/src/models/dflash.cpp b/src/models/dflash.cpp
+index e195ccfda..5142f45b0 100644
+--- a/src/models/dflash.cpp
++++ b/src/models/dflash.cpp
+@@ -7,6 +7,28 @@
+ #include <stdexcept>
+ #include <vector>
+ 
++// K/V caches can be stored in a rotated basis (notably for quantized cache
++// types). DFlash injects K/V directly instead of going through build_attn(), so
++// apply the cache rotation explicitly before copying the projected tensors.
++// This is the pre-llama-impl.h equivalent of llama_mul_mat_hadamard().
++static ggml_tensor * dflash_mul_mat_hadamard(
++        ggml_context * ctx,
++        ggml_tensor * cur,
++        ggml_tensor * rot) {
++    const auto n = rot->ne[0];
++
++    ggml_tensor * res;
++    if (!ggml_is_contiguous(cur)) {
++        res = ggml_cont_2d(ctx, cur, n, ggml_nelements(cur)/n);
++    } else {
++        res = ggml_reshape_2d(ctx, cur, n, ggml_nelements(cur)/n);
++    }
++    res = ggml_mul_mat(ctx, rot, res);
++    ggml_mul_mat_set_hint(res, GGML_HINT_SRC0_IS_HADAMARD);
++
++    return ggml_reshape_4d(ctx, res, cur->ne[0], cur->ne[1], cur->ne[2], cur->ne[3]);
++}
++
+ void llama_model_dflash::load_arch_hparams(llama_model_loader & ml) {
+     ml.get_key(LLM_KV_ATTENTION_LAYERNORM_RMS_EPS, hparams.f_norm_rms_eps);
+ 
+@@ -422,9 +444,23 @@ llama_model_dflash::graph<false>::graph(const llama_model & model, const llm_gra
+                 const auto  * kv     = is_swa ? inp_attn_iswa->mctx->get_swa() : inp_attn_iswa->mctx->get_base();
+                 ggml_tensor * k_idxs = is_swa ? inp_attn_iswa->get_k_idxs_swa() : inp_attn_iswa->get_k_idxs();
+                 ggml_tensor * v_idxs = is_swa ? inp_attn_iswa->get_v_idxs_swa() : inp_attn_iswa->get_v_idxs();
++                ggml_tensor * k_rot  = is_swa ? inp_attn_iswa->self_k_rot_swa : inp_attn_iswa->self_k_rot;
++                ggml_tensor * v_rot  = is_swa ? inp_attn_iswa->self_v_rot_swa : inp_attn_iswa->self_v_rot;
++                if (k_rot) {
++                    Kcur = dflash_mul_mat_hadamard(ctx0, Kcur, k_rot);
++                }
++                if (v_rot) {
++                    Vcur = dflash_mul_mat_hadamard(ctx0, Vcur, v_rot);
++                }
+                 ggml_build_forward_expand(gf, kv->cpy_k(ctx0, Kcur, k_idxs, il));
+                 ggml_build_forward_expand(gf, kv->cpy_v(ctx0, Vcur, v_idxs, il));
+             } else {
++                if (inp_attn->self_k_rot) {
++                    Kcur = dflash_mul_mat_hadamard(ctx0, Kcur, inp_attn->self_k_rot);
++                }
++                if (inp_attn->self_v_rot) {
++                    Vcur = dflash_mul_mat_hadamard(ctx0, Vcur, inp_attn->self_v_rot);
++                }
+                 ggml_build_forward_expand(gf, inp_attn->mctx->cpy_k(ctx0, Kcur, inp_attn->get_k_idxs(), il));
+                 ggml_build_forward_expand(gf, inp_attn->mctx->cpy_v(ctx0, Vcur, inp_attn->get_v_idxs(), il));
+             }
 diff --git a/src/models/models.h b/src/models/models.h
 index 5cc93e6b9..cb3b4d4ad 100644
 --- a/src/models/models.h

SHA256SUMS

@@ -2,3 +2,6 @@
 6b80db6b044d9a3999b539eec140dc4d29279d53c18f42f80a557d3e6736123a  Muse-Glimmer-30B-ROCmFP4-Q6-QUALITY.gguf
 74a5f469f17420d01a837400a2ef19d8191dcb1b479fd7c0be0c862901879915  Muse-Glimmer-30B-ROCmFP8.gguf
 3727a47650ae65be12e91aae49f9b9f4bef09dc6c401a54ca1bd3d1a6160f35a  mmproj-Muse-Glimmer-30B-BF16.gguf
+1e2f0d9de05251b05ed9c6f67f208105b30f0cd51112307a400b59936dd8fc41  Muse-Glimmer-30B-DFlash-ROCmFP4.gguf
+ae1d9ffbcbc3af92fbe50ecf62d5892c619ec76b1ca3198e25b240f86a11974f  Muse-Glimmer-30B-DFlash-ROCmFP8.gguf
+0e15099778bf88ef2ec56e6076369321cd10a93f159caa97d676b5c0008d2750  ROCmFPX-Muse-Glimmer.patch