ProCreations/Qwen3.8-Flash-Next-DSpine-preview

Qwen3.8-Flash-Next DSpine: experimental preview

Experimental preview. This head is SLOWER than the target's native MTP. The project's serving-cost model estimates 0.6–0.7x native MTP throughput. That is an estimate from offline acceptance and measured serving components, not a matched end-to-end benchmark of this final head. Training stopped by owner decision on October 17, 2026 at step 9,145. The original speed goal was not met.

Preview head: ProCreations/Qwen3.8-Flash-Next-DSpine-preview.

Intended use

A research artifact and warm-start base for parallel speculative draft training, objective/feature studies, and adaptation to new targets. It is not a standalone language model and cannot generate without its frozen target embedding, effective LM head and correctly aligned target-context features. Native MTP remains the serving baseline for this target. No downstream quality improvement is claimed.

Base: RedHatAI/Qwen3.8-Flash-Next-NVFP4, revision c8f2fb1b9869f686b214782036123b10ff96d14a. Tokenizer and chat template must come from that same snapshot.

Complete-round checkpoint evaluation

Temperature	b6 pooled tau	b8 pooled tau	Prompts
T1	1.799928	1.823011	64
T0	1.849828	1.867662	64

CPU FP32 evaluation completed in 662.2 seconds. T1 Monte Carlo standard errors are 0.000565 (b6) and 0.004993 (b8); prompt-level standard errors are 0.042596 and 0.044432. These measure different sources of uncertainty.

Tau is emitted tokens per verification round, including a correction/bonus except after accepted EOS. Blocks b6/b8 include the known anchor. T1 uses actual-decode joint top-k=20/top-p=0.95 target probabilities and four reverse-coupling Monte Carlo trials along recorded target paths. T0 uses separately generated greedy target continuations. Evaluation has 64 prompts per temperature with a 64-new-token cap, balanced across nine domains. Length-censored tails are excluded. These short trajectories do not measure completed-answer acceptance, long-body behavior, quality parity or real serving throughput. Prompt-level sampling uncertainty is larger than the Monte Carlo uncertainty alone.

The release evaluation ran on CPU in FP32 with the saved 189,090-token row-INT8 proposal policy. Its frozen head is the portable reconstruction of the target's effective W8 head. Private production MTP uses a different sequential target processor, so these are not matched MTP acceptance measurements. The slower-than-MTP estimate and native historical acceptance are separate evidence.

The reference implementation also reproduced Tencent Qwen3-8B-DSpine-b16 on 100 greedy GSM8K prompts: prompt-mean tau 10.670, pooled tau 9.857 at b16, thinking off and 512-token cap. That validates an inference path on a different target; it does not establish convergence of this preview.

Architecture and files

Six dense draft layers, hidden size 2560, FFN 10240, 32 query/8 KV heads of size 128, block 16, context window 4096, transfer rank 1024, message dimension 512, 16 candidates. Features concatenate FP32 means of four residual streams after target layers [1,12,23,34,45], before storage quantization. All context precedes the known anchor. The untested context-fusion shortcut is off in this checkpoint.

• model.safetensors: exact retained inference state, 77 trainable tensors plus frozen selected_prefix_bridge.codebook (248320×1024 BF16). No optimizer, RNG, gradients, target embedding or target LM head is included.
• config.json: draft architecture. proposal_policy.json: global token-ID map, row-INT8 reconstruction and restricted-loss policy. These are required together.
• inference.json: base revision, feature/head/sampling contracts. The PCA basis is not needed to load inference; the transfer codebook is embedded in weights.
• training_recipe.json: sanitized final recipe. metrics.json: full final CPU score, seeds, per-domain values, uncertainty and source fingerprints.
• export-receipt.json: weight/config/policy hashes and byte counts.

Use the independent DSpine Trainer code at SSHdotCodes/dspine-trainer. This head uses its custom loader; it is not an AutoModelForCausalLM checkpoint. After obtaining the base snapshot and this directory locally:

import torch
from dspine import DSpineDraft
from dspine.transfer_space import load_target_weights

embedding, head = load_target_weights("./models/target", "qwenfast_w8")
draft = DSpineDraft.from_pretrained(
    "./models/dspine-preview", embedding, head,
    trainable_dtype=torch.float32,
).eval()

Provide aligned target features to draft, build its transition cache, and keep the actual conditional proposal probabilities for exact rejection sampling. Frozen embedding/head rows are reconstructed from the separate base snapshot. The private runtime and kernels used in the original experiment are not bundled; the reference code alone is not a deployment integration.

Training recipe and lessons

On-policy target continuations were generated at T=1/top-k=20/top-p=0.95 with per-prompt reasoning settings. The retained corpus has 100,318 records and 228,480,448 response tokens. This is corpus size, not unique tokens consumed by this stopped run. The training lineage used rolling feature windows and did not complete its planned 540M response-token-presentation schedule.

Final promoted settings: AdamW, LR 3e-4, betas (0.9, 0.999), zero weight decay, global clip 1, 4% token-progress warmup then cosine, BF16 compute with FP32 master parameters, four accumulated sequences/up to 256 unique response anchors each, block 16, position decay gamma 7, alpha 0.1 sampled CE plus 0.9 exact sparse L1, equal refinement loss, layerwise cosine supervision. Lambda uses nominal weighted-token-mean detached backbone CE; its beta ramps to 0.5 in warmup. Correct-predecessor substitution follows the inherited token-progress schedule.

The lineage initially used LR 6e-4 and a position-summed lambda. Later corrected recipe C adopted token-scale lambda, unique anchors with zero-weight padding, refinement starting at position 2 and CE masking outside stored teacher support. The 3e-4 branch beat 6e-4 in follow-up forks. These corrected settings were not used from step zero. No auxiliary-off or context-fusion experiment is silently merged into this checkpoint.

Opening-only tau overstated useful acceptance on formulaic reasoning starts. The complete-round evaluator and a seven-agent implementation audit distinguish that bias from actual convergence. Extra final-layer features added only 0.03–0.045 position-1 probe overlap, below the prespecified +0.05 adoption rule. Native MTP's strong pretrained first-token/conditional acceptance remained the main competitive hurdle. No hybrid serving speedup was demonstrated.

License, data provenance and acknowledgments

Trained draft parameters, config and proposal policy: Apache-2.0 (LICENSE). The frozen selected_prefix_bridge.codebook tensor is a PCA projection of the base model's LM-head rows. This checkpoint is therefore treated as a derivative of Qwen3.8-Flash-Next and distributed as a whole under the Qwen Community License 1.0 (LICENSE-QWEN), including its notice, model-name display and Model-as-a-Service / AI Work Assistant conditions. See NOTICE. The base model must be obtained separately under its own terms.

Training data: the head was trained on the full 100,318-record generation corpus. That includes five prompt sources left out of the companion conservative dataset (SWE-smith and Nebius SWE-agent trajectories, CodeFeedback, open-perfectblend and Llama-Nemotron). Some of them have non-commercial upstream branches, per-repository code licenses or Llama-output terms. No prompt text or source answers are bundled here. Treat this head as a research artifact and review those obligations for your use.

Credit the DSpine paper, Tencent's public checkpoint, DFlash, Qwen and Red Hat. This independently trained preview is not endorsed by those authors.