YTan2000/Qwen3.8-Flash-Next-tqLLM

Qwen3.8 Flash-Next for tqLLM

Run Qwen3.8 Flash-Next (125B mixture of experts) with tqLLM on one 24 GB NVIDIA GPU or a DGX Spark / GB10. Two ways in:

• The public IQ3_S file + this repo's small files. The ISTA-DASLab GSQ-RCO IQ3_S GGUF is the only large download; tqLLM rearranges its experts in memory at start-up.
• tqLLM's own Flash-Next file (mixed/, 82.4 GB in two parts): the same IQ3_S experts, IQ4_XS dense layers and 8-bit hyper-connection mixers. Faster on both kinds of hardware:

Measured	tqLLM's own file	Comparison
DGX Spark, prompt processing 8K / 16K / 32K / 64K	4.32 / 8.14 / 16.83 / 35.64 s	TensorFold 1.2.2: 4.90 / 9.21 / 18.58 / 38.11 s
DGX Spark, decode, one request	122 tok/s	TensorFold 1.2.2: 73 tok/s
RTX 3090, 145K-token prompt: first token / decode	58.6 s / 76.1 tok/s	IQ3_S file: 61.7 s / 71.6 tok/s

Same prompts and client for both sides; GSM8K-200 96.0 % for this file. Method and more rows: tqllm.com/performance, release notes.

File	Size	What it is
mixed/Qwen3.8-Flash-Next-tqLLM-00001-of-00002.gguf, -00002-of-00002.gguf	44.7 + 37.8 GB	tqLLM's own Flash-Next file (optional)
flashnext-tqllm.tqfn	1 KB	the manifest for tqLLM's own file
flashnext.tqfn	1 KB	the manifest for the public IQ3_S file
mtp-blk48-iq4.gguf	1.46 GB	the model's own multi-token-prediction layer, IQ4, for exact speculative decoding
flashnext-tokenizer.gguf	11 MB	tokenizer and chat template
atx_65536.txt	0.4 MB	draft vocabulary for the MTP drafter

Run

Needs a 24 GB NVIDIA GPU (RTX 30 or newer) or a GB10 / DGX Spark, 64 GB of system memory or more (the experts live there; the numbers above used 128 GB) and tqLLM 1.4.2 or newer. With the public IQ3_S file, put everything in one folder:

hf download ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF --include "IQ3_S/*" --local-dir fn
mv fn/IQ3_S/* fn/
hf download YTan2000/Qwen3.8-Flash-Next-tqLLM --exclude "mixed/*" --local-dir fn
docker run --rm --gpus all -p 127.0.0.1:8086:8086 -v "$PWD/fn:/fn:ro" --entrypoint tqserve \
  ytan2000/tqllm:1.4.2 --model /fn/flashnext.tqfn --tokenizer /fn/flashnext-tokenizer.gguf \
  --context 32768 --host 0.0.0.0 --port 8086

With tqLLM's own file instead (no ISTA download needed):

hf download YTan2000/Qwen3.8-Flash-Next-tqLLM --local-dir fn
docker run --rm --gpus all -p 127.0.0.1:8086:8086 -v "$PWD/fn:/fn:ro" --entrypoint tqserve \
  ytan2000/tqllm:1.4.2 --model /fn/flashnext-tqllm.tqfn --tokenizer /fn/flashnext-tokenizer.gguf \
  --context 32768 --host 0.0.0.0 --port 8086

Then curl -s http://localhost:8086/health and any OpenAI-compatible client at http://localhost:8086/v1. The server is ready after about two minutes.

Credits and licence

Model: Qwen team (Qwen/Qwen3.8-Flash-Next), Qwen Community License 1.0 (copy in LICENSE-Qwen; keep it with any copy of these files). Routed experts: IST Austria DAS Lab (GSQ + RCO), Apache-2.0. Importance matrix for the IQ4_XS dense layers: unsloth. tqLLM's own file and tqLLM: David Y. Tan. See NOTICE.