Muse Glimmer 30B works pretty well with 512k context as-is
Short story: the model architecture appears to be uniquely suited for context extension by simple mechanical means. To change model context length from stock 128k to, say, 512k, you need only to change “max_position_embeddings” config setting from 131072 to 524288. No adaptors needed. The thing that make long-context adaptations painful in other models (full attention layers with token position encoding) is simply not here. It's a rather bold architecture bet, but it seems Meta managed to pull it off.
RULER-inspired benchmarks of the full-precision model show 100% performance on multi-hop chaining, multi-key and multi-value needles on contexts 32k, 64k, 128k, 256k, 512k. Frequent word extraction is ~50% on 32k, ~30% on 128k, and collapses to 0 after that, but it seems to be the weakness of the model overall.
Longer story is here:
https://www.reddit.com/r/LocalLLaMA/comments/1vqntod/muse_glimmer_30b_with_512k_context/
Full research dossier with more benchmarks:
https://github.com/lobanov/muse-glimmer-long-ctx
Interesting find, and the architecture argument holds up, the local/NoPE split really does look like it was built for this. We run this model in production on a single RTX 5090 (32GB), which I believe is exactly the hardware your section 12 qualification is waiting on, so here is a datapoint from that card. Short version: the extension survives the architecture, but on the 4-bit path it does not survive the kernels.
Setup: vLLM, NVFP4 quant (bullerwins), kv-cache fp8_e4m3, VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 with max_position_embeddings overridden to 524288 (mechanically equivalent to your config.json edit, vLLM otherwise refuses the override at validation). No other change.
What worked: the server boots clean, allocates a KV pool of 802,532 tokens (1.53x concurrency at 524,288 per request), and RULER-style probes pass where we could run them, multi-key retrieval and 2-hop reasoning both correct at 32k and at ~70k real prompt tokens, prefill ~13s at 70k.
What did not: any prefill in the 256k+ range kills the engine outright. Not a nan, not a quality decay, the FP4 GEMM kernel refuses the shape: [FP4 gemm Runner] Failed to initialize cutlass FP4 gemm on sm120/sm121. Error: Error Internal, EngineCore dies, and the API server goes zombie serving 500s. Reproducible on two consecutive requests. Our production config serves 131k daily on the same kernels, so the wall sits somewhere between ~131k and 256k prompt tokens.
So for your qualification matrix: the 512k claim appears to hold at the architecture level (consistent with your GGUF results, llama.cpp does not go through this kernel), but a Blackwell user on NVFP4 + cutlass cannot currently reach it. The binding constraint is the quantized GEMM workspace on very long prefill shapes, not the model. Might be worth a line in your MODEL.md, since the 4-bit path is presumably what most 32GB users would deploy.
Happy to run a specific config from your sweep grid if that helps, the card is busy with production during the day but we can slot night windows.
Thank you for the report. I only tested the full-precision model on vLLM on DGX Spark, haven't yet tried quantized forms. The issue you are describing looks more like a kernel bug or misconfiguration than a model issue. I added a note to the README in the repo.
I'm away currently from my home lab, so I won't get a chance to look into it until early September. If you have time to drill into this in the meantime, I wouldn't be running anything from the repo, I would clone vLLM repo, built from source, and use coding agent to troubleshoot.