Qwen3.8-Flash-Next for QEngine
Qwen3.8-Flash-Next in the weight layout of QEngine, an open-source (MIT) inference engine for 3× NVIDIA RTX 3090. Download it and run it with QEngine directly, with no conversion step.
For QEngine 1.3 or later; 1.3.4 is recommended (image input with --vision 1, an optional KV cache in host RAM,
an opt-in context of up to 1M tokens, pinned system prompts for agents; new in 1.3.4: fast tensor parallelism without
GPU P2P, a pipeline-parallel mode picked automatically where it wins, host-memory defaults that fit the machine, a
hard thinking budget, and the earlier fixes, including the 1.3.3.1 fix for replies that could pick up text from
another conversation running at the same time). In the club-3090 8-pack (150 scenarios, 3 runs on 3× RTX 3090)
QEngine 1.3.4 passed 114.7 (thinking off) and 128.7 (thinking on) on average, against 110 and 126 for 1.3.1.
These files only work with QEngine.
What's inside
- Converted from Intel/Qwen3.8-Flash-Next-W4A16-AutoRound, an INT4 quantization of Qwen/Qwen3.8-Flash-Next.
base.safetensors,layer-00…layer-47: the model, regrouped per layer for QEngine (W4A16, mixed precision):- routed experts: INT4 from the AutoRound checkpoint, repacked bit-exact;
- shared expert, DeltaNet projections (in_proj_qkv / z / out_proj), n-gram key/value projections: INT4 (RTN, group 128);
- QSA attention q/k/v/o and lm_head: BF16 in the files, INT8 in the engine (converted at load);
- hyper-connections, router, embeddings, indexer, gates, norms: BF16.
mtp.safetensors/mtp-int4.safetensors: the multi-token-prediction head used for speculative decoding.ngram_int8.safetensors: the n-gram embedding table in INT8 (51.8 GB instead of 102.4 GB in BF16). In QEngine's tests its next-token KL against the BF16 table was below the engine's own run-to-run noise.expert_prior.bin,cold_experts.json,token_rank.bin: routing statistics and the draft vocabulary that QEngine uses.
Requirements
- 3× RTX 3090 (24 GB), or 2 of them. GPU-to-GPU P2P helps but is not needed: since QEngine 1.3.4 the exchanges without it go through pinned host memory by DMA (3 cards: 90K-token prefill 17.0 s without P2P, 13.1 s with it; decode within about 4 %; see the QEngine README).
- Host RAM: with about 140 GB (3 cards) or 160 GB (2 cards) QEngine uses its full defaults: the INT8 n-gram table (about 52 GB) pinned, a 48 GB conversation archive and 8 GB of pinned system prompts. Since 1.3.4 a smaller host (from 64 GB) gets a plan sized at start-up to its RAM, without swapping.
- Low-RAM hosts: page-cache mode (
FLASH_NGRAM_PIN=0, chosen automatically when the table does not fit pinned). The table stays on disk and the rows each step needs are read through the page cache. Measured on an NVMe SSD with the whole server limited to 12 GB of RAM: short single-user requests about 2× slower (127 vs 262 tok/s) and the median time to first token 1.7 s instead of 0.22 s; with the table in the page cache the speed is close to the pinned mode (250 vs 262 tok/s). Details in the QEngine README. - About 120 GB of disk.
License
The weights are a derivative of Qwen3.8-Flash-Next and are distributed under the Qwen Community License 1.0 (copy included). QEngine itself has its own license, see its repository.
- Downloads last month
- 940
Model tree for bpx03/Qwen3.8-Flash-Next-QEngine
Base model
Qwen/Qwen3.8-Flash-Next