jesdga/Qwen3.8-Flash-Next-Basalt

The fastest inference engine for Qwen3.8-Flash-Next on Blackwell

Model files for Basalt, a local inference engine for the Qwen3.8-Flash-Next mixture-of-experts model. Basalt runs on Linux with any NVIDIA Blackwell GPU (sm_120). A second GPU can serve as a peer expert tier, and experts that do not fit in VRAM stay in system RAM. It has an OpenAI- and Anthropic-compatible API.

Each .basalt file is self-contained: weights, tokenizer, MTP drafter, expert profile and vision projector. Expert formats: GSQ-RCO (IQ3_XXS, IQ3_S, Q2_0), Unsloth UD K-quants (Q4_K_XL, Q5_K_XL, Q6_K_XL), Q8_0 and NVIDIA NVFP4.

Files: Qwen3.8-FN-NVFP4, -UD-Q4_K_XL, -UD-Q5_K_XL, -UD-Q6_K_XL, -Q8_0, -GSQ-RCO-IQ3_S, -GSQ-RCO-IQ3_XXS and -GSQ-RCO-Q2_0 (each .basalt), plus convert/.

Speed

Best measured, single stream, 400 W, RTX 5090 + RTX 5060 Ti peer, tokens/s. Numbers move with every release.

pack	prose	structured	chat
NVFP4	274	491	325
GSQ-RCO IQ3_S	341	646	440
Q8_0	112	166	101

Quality

Measured with basalt-bench --quality: 100 chunks of wikitext-2 (512 tokens each), scored through the decode path against the Q8_0 reference (int8 KV, 400 W). Lower KLD and higher top-1 agreement are closer to Q8_0. PPL is the pack's own perplexity on the same text; it is not a closeness measure. Only the official Basalt packs are listed.

pack	source	file	experts	PPL	KLD mean	same top-1
GSQ-RCO Q2_0	ISTA-DASLab	64.9 GiB	31.6 GiB	3.949	0.4355	78.21%
GSQ-RCO IQ3_XXS	ISTA-DASLab	74.4 GiB	40.0 GiB	3.470	0.2434	83.96%
GSQ-RCO IQ3_S	ISTA-DASLab	81.0 GiB	46.8 GiB	3.383	0.1547	87.25%
UD-Q4_K_XL	Unsloth	106.7 GiB	71.7 GiB	3.322	0.0566	92.16%
NVFP4, 6-bit dense	NVIDIA	122.0 GiB	63.3 GiB	3.308	0.0680	91.46%
UD-Q5_K_XL	Unsloth	150.5 GiB	91.6 GiB	3.289	0.0350	94.05%
UD-Q6_K_XL	Unsloth	160.6 GiB	101.7 GiB	3.274	0.0311	94.36%
Q8_0	Unsloth	178.3 GiB	119.5 GiB	3.286	reference

The NVFP4 pack keeps NVIDIA's NVFP4 experts bit for bit. Its dense projections are Q6_K, quantized once from BF16; the head and embedding are Q8_0.

The IQ3_S, IQ3_XXS, Q2_0 and NVFP4 packs carry a self-distilled MTP layer (convert/mtp/rt/dense.bin) for higher draft acceptance; the others keep the original.

Run

Build the engine as described in the Basalt README, then:

hf download jesdga/Qwen3.8-Flash-Next-Basalt Qwen3.8-FN-NVFP4.basalt --local-dir models

basalt_config.json:

{
  "server": { "model": "models/Qwen3.8-FN-NVFP4.basalt", "gpu": "0,1", "port": 8080 },
  "engine": { "expert-cache": "auto", "prefill": "auto", "kv": "int8", "resident-experts": true }
}

bin/basalt serve --config basalt_config.json

Conversion kit

convert/ holds the inputs for tools/convert/convert.py (MTP layer, vision projector, expert profile, draft vocabulary), so any GGUF quant of this model can become a .basalt file. See docs/model-file.md.

Sources

• GSQ-RCO: ISTA-DASLab, IST Austria; also the vision projector.
• UD quants, Q8_0: unsloth.
• NVFP4 experts: NVIDIA.
• MTP drafter: Qwen/Qwen3.8-Flash-Next BF16 MTP tensors.
• Basalt is a fork of Strata (MIT).

Citation

If you use the GSQ-RCO files, please cite:

@article{gsq2026,
  title  = {GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling},
  author = {Dadgarnia, Alireza and Tabesh, Soroush and Nikdan, Mahdi and Helcig, Michael and Kurtic, Eldar and Kleinegger, Maximilian and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2604.18556},
  year   = {2026}
}
@article{rco2026,
  title  = {Model Compression with Exact Budget Constraints via Riemannian Manifolds},
  author = {Helcig, Michael and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2605.00649},
  year   = {2026}
}

License

All files here are derived from Qwen3.8-Flash-Next and are distributed under the Qwen Community License 1.0, whose notice is included in this repository and inside every .basalt file. Businesses offering Model as a Service or an AI coding/office assistant need a separate license from Qwen for commercial use, and products above 100M monthly users or US$20M monthly revenue must display the model name. The LICENSE has the binding terms. The Basalt engine itself is MIT.