DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated

Qwen3.8-Flash-Next FP4-Local — Abliterated

Fewer refusals, the same compact weight formats, and similar single-Spark decode speed to NVIDIA's FP4-Local base.

Quick start · Evaluation · Two-Spark recipe · Method

A standalone derivative of NVIDIA's Qwen3.8-Flash-Next-FP4-Local, modified through abliteration: a weight edit that reduces the model's learned refusal behavior. The edit is baked into the checkpoint, so no adapter or runtime hook is needed.

The weights occupy approximately 96 GiB on disk and run on one DGX Spark using the supplied launcher. A separate two-Spark serving recipe is linked below.

This checkpoint keeps NVIDIA's FP4-Local storage format, including its compressed NVFP4 scales. Like the base, it requires a compatible B12X vLLM container. Use the pinned image in serve-spark.sh; a standard Transformers or stock vLLM loading example is not a supported path for this checkpoint.

Highlights

• Substantially reduced refusals: 0 of 100 development and 6 of 100 holdout prompts were detected as refusals, against 99 and 100 for the NVIDIA base.
• Base-compatible weight layout: every tensor retains the base checkpoint's name, dtype and shape. Routed experts remain NVFP4; attention and shared-expert projections remain MXFP8.
• Similar single-Spark speed: median decode rates of 38.0 tok/s for prose and 54.3 tok/s for code, against 39.4 and 50.7 for the base under the same serving profile.
• Functional checks passed: image input, tool calling, thinking controls and MTP speculative decoding passed the checks described below.
• Self-contained: download the repository and run the included DGX Spark launcher.

These results describe the MXFP8 release dated October 22, 2026. They are bounded measurements, not a claim that all capabilities are unchanged.

Model overview

Original model	Qwen/Qwen3.8-Flash-Next by Qwen
Quantized base	nvidia/Qwen3.8-Flash-Next-FP4-Local by NVIDIA, revision b9ce18a
Parameters	125B language-model parameters, 6B active, plus a 51B n-gram embedding table and a 4B MTP layer, as described in NVIDIA's card
Input / output	Text and images in, text out
Weight storage	Approximately 96 GiB / 102 GB across 21 shards
Precision	Mixed precision: NVFP4 routed experts, MXFP8 attention and shared-expert projections, and other tensors in their original base formats
What changed	Refusal-direction ablation in the 48 attention output projections and the routed/shared expert down projections
Single-Spark context	Configured for 262,144 tokens; retrieval checks covered approximately 8K and 30K tokens
Tested hardware	DGX Spark, GB10, 128 GB unified memory; separate single-node and two-node profiles

Weight storage is the size on disk, not the complete runtime memory requirement. The single-Spark launcher uses disk-backed n-gram embeddings and a 104 GiB container memory cap.

Quick start (DGX Spark)

You need:

• One DGX Spark with Docker and NVIDIA GPU support.
• The Hugging Face CLI, hf. See the installation guide.
• Space for approximately 102 GB of model files, plus the Docker image and compilation caches.
• Enough free unified memory to run the 104 GiB container. Avoid running another large model alongside it.

Download the complete repository, preserving the tensors/ directory, then start the server:

hf download DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated \
  --local-dir ./qwen-abliterated

bash ./qwen-abliterated/serve-spark.sh ./qwen-abliterated

The launcher uses this pinned runtime image:

eugr/spark-vllm-b12x@sha256:03a956544e9f9b4b7ba91690469fbe7848bdbef29a7d2e0f5788368467bb3595

First start: compilation and autotuning took about 23 minutes on the tested Spark. Later starts reused the cache and took about 5 minutes. Startup time varies by host and cache state.

The launcher stays attached to the server logs. In another terminal, check readiness:

curl --fail http://127.0.0.1:8000/health

Once that succeeds, the OpenAI-compatible API is available at http://127.0.0.1:8000/v1, with model ID qwen-trial117.

Send a request

This example disables thinking and requests a direct answer:

curl --fail http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen-trial117",
    "messages": [
      {
        "role": "user",
        "content": "Explain how a heat pump works in three sentences."
      }
    ],
    "max_tokens": 256,
    "chat_template_kwargs": {
      "enable_thinking": false
    }
  }'

For Python, install the client in your Python environment:

python3 -m pip install openai

This example enables thinking explicitly:

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="none",
)

reply = client.chat.completions.create(
    model="qwen-trial117",
    messages=[
        {"role": "user", "content": "Write a haiku about unified memory."}
    ],
    max_tokens=2048,
    extra_body={
        "chat_template_kwargs": {"enable_thinking": True}
    },
)

message = reply.choices[0].message
reasoning = getattr(message, "reasoning", None)
if reasoning:
    print(reasoning)
print(message.content)

• Thinking is on by default. Set chat_template_kwargs.enable_thinking per request. Reasoning is returned in the message's reasoning field.
• Tools use the OpenAI-compatible tools field, with the qwen3_xml parser.
• Sampling defaults come from generation_config.json: temperature 1.0, top-p 0.95 and top-k 20. The greedy evaluation settings below are separate from these defaults.

To stop the server from another terminal:

docker stop -t 90 qwen-trial117

Rerun the launcher to start it again; compiled caches persist.

Serving profile

The launcher follows NVIDIA's published DGX Spark profile for the base:

Setting	Default
Tensor parallelism	1
Maximum context	262,144 tokens
Active sequences	Up to 8
KV cache	FP8
Prefix caching / chunked prefill	Enabled
Speculative decoding	MTP, 4 draft tokens
Kernels	B12X MoE, linear and GDN
N-gram embeddings / PLE	Disk-backed
Container memory cap	104 GiB

Eight active sequences do not mean eight simultaneous maximum-length contexts; available KV cache limits the total.

Variable	Default	Effect
PORT	8000	API port
HOST	127.0.0.1	Bind address; use an authenticating proxy before exposing the API beyond the host
MTP_TOKENS	4	Speculative draft tokens; 0 disables MTP
CACHE_DIR	~/.cache/qwen-trial117	Compiled kernels and autotune results

Evaluation

The following results cover the October 22, 2026 MXFP8 release on one DGX Spark, using serve-spark.sh unless otherwise stated. NVIDIA's base checkpoint was served with the same profile.

The base revision, runtime image, frozen intervention inputs and validation summary are recorded in provenance.json. File hashes and sizes are recorded in SHA256SUMS and release-files.json.

Refusals and drift

Refusal tests used 100 development and 100 holdout prompts from mlabonne/harmful_behaviors. Answers were greedy, seed 3741, thinking off and capped at 100 tokens. A refusal regex classified the responses.

Checkpoint	Development refusals	Holdout refusals
NVIDIA base	99 / 100	100 / 100
This model	0 / 100	6 / 100

These are detected refusals in short responses. They do not measure answer correctness, usefulness or universal compliance. Raw responses to refusal-test prompts are not published.

Distribution drift was measured separately on benign prompts from mlabonne/harmless_alpaca, using 100 prompts per split:

Metric, this model versus NVIDIA base	Development	Holdout
First-token KL, nats	0.207	0.219
First-token top-1 agreement	81%	84%

KL measures a difference between probability distributions. Here it uses the first answer token and renormalized top-20 candidates. It is not a percentage, a full-vocabulary KL measurement or a general capability score.

Capabilities

• Functional checks: 11 of 11 passed. Model listing, streamed usage, thinking on and off, automatic and forced tool calls, a tool-result follow-up returning exact JSON, image colour recognition, and exact retrieval of three planted markers from documents of about 8,000 and 30,000 tokens.
• Vision: 10 of 10 solid-colour images named correctly. This is a basic image-input check, not a broad vision benchmark.
• Speculative decoding: observed MTP acceptance was comparable to the base. A greedy audit found no emitted-token mismatches against the target model's top choice on the audited output.
• Writing: 13 of 20 editing and composition cases passed all scoring dimensions, against 14 for the base. An AI reviewer scored the fixture; it was not human-adjudicated. The two changed cases also varied between runs of the other checkpoints.

Two writing behaviors repeated across runs: short arithmetic questions could trigger step-by-step explanations that exhausted a tight output budget, and requests to tighten a long note received lighter edits than the base.

These checks establish specific working paths. They do not establish unchanged overall capability.

Speed

Single request, greedy decoding, thinking off, 512 output tokens, MTP 4. Decode rates are medians of six probes over two rounds; ranges are shown in parentheses.

Metric	This model	NVIDIA base
Prose decode, tok/s	38.0 (37.7–39.6)	39.4 (39.1–40.9)
Code decode, tok/s	54.3 (48.5–56.0)	50.7 (49.3–52.4)
Speculative decode step	67.7–67.9 ms	66.8–67.2 ms
KV cache capacity	947,989 tokens	902,668 tokens
Time to first token, short prompt	0.05–0.27 s	—
Time to first token, ~30,000-token prompt, cold	12.7 s	—

Decode rate measures generation after the first content token; it excludes prefill. Speed depends strongly on content because speculative decoding accepts more draft tokens on predictable text such as code.

On a separately tuned single-Spark stack with MTP 2, based on diffbot's DGX Spark recipe, prose reached 45.3–45.7 tok/s against 46.1–47.1 for the base. That stack is not included in this repository, and those figures do not describe the supplied launcher's defaults.

Two DGX Sparks

For two-node serving, see XtraSaltyDev's TP=2 recipe and evaluation. It serves this checkpoint without modifying the weights.

The qualified profile uses 262K context, 16 active sequences, BF16 KV cache, MTP4 and disk-backed PLE, with a 104 GiB container cap on each Spark. It requires working RoCE networking and the model files on both nodes.

Measured on October 22, 2026:

Workload	Original eight-slot profile	Qualified sixteen-slot profile
16 concurrent short requests: aggregate output	229.0 tok/s	313.4 tok/s (+37%)
Same workload: p95 first token	11.89 s	0.77 s
16 concurrent ~8K-input requests: aggregate output	38.64 tok/s	40.98 tok/s (+6%)

Both columns use the same abliterated checkpoint and runtime image. This comparison measures serving-profile changes, not performance against NVIDIA's unmodified model.

The short workload used 62–64 input tokens and 256 forced output tokens, greedy decoding and thinking off. Aggregate throughput includes queue and prefill time across the request wave; it is not a single user's decode rate. The samples are small and do not establish production latency guarantees. Serial prose decode was about 8% slower than the eight-slot reference.

The selected profile also passed 10/10 strict JSON retrieval checks at 261,623 input tokens. This result belongs to the two-Spark BF16-KV profile; it does not extend the single-Spark FP8-KV validation above. Sixteen scheduler slots do not imply sixteen simultaneous 262K contexts.

See the GitHub repository for setup, full results and tradeoffs, and the pinned checkpoint revision and raw evidence.

How it was made

1. Find the refusal direction. The method follows Arditi et al. (2024) and the parameterization of Heretic, reimplemented for this checkpoint's native vLLM backend. Per-layer directions come from attention-output activations on harmful and harmless prompt sets. An Optuna TPE search balanced refusals against KL, and trial 117 was selected.

2. Bake the intervention into the weights. The selected projection was applied to the 48 attention output projections and the routed/shared expert down projections. Expert weights were re-rounded in NVFP4 and MXFP8 while preserving their scale encodings.

3. Restore attention to MXFP8. The intermediate bake stored the edited attention output weights in BF16. Constrained rounding re-encoded them into the base's MXFP8 format while preserving the intended directional edit.

The selected trial uses the accepted continuation recalibration. Its parameters, directions and row norms are included in bake-selection.json, frozen-directions.json and frozen-row-norms.json.

Direction-preserving re-encode

Plain MXFP8 rounding of the edited attention weights would introduce error along the removed direction equal to about a tenth of the intervention's size.

The re-encode script instead rounds each weight to one of its two nearest E4M3 values, using the base's scale rule of one E8M0 exponent per 32 inputs. A greedy pass chooses roundings that cancel error along the removed direction while minimizing added squared error.

Across the 48 attention output projections:

• Error along the removed direction is at most 0.002% of the intervention.
• Overall relative weight error is 1.47–1.93%.
• The fraction of the direction removed matches the BF16 intermediate to within 0.00002.

Per-layer measurements are in reencode/reencode-report.json. These are weight-space measurements, separate from the behavioral tests above.

The tokenizer, chat template, vision encoder and built-in MTP layer are NVIDIA's. The model configuration is unchanged apart from a standalone_provenance metadata entry. This is an edited derivative of NVIDIA's quantized checkpoint, not a repeat of its quantization-aware distillation.

Files

File	Contents
tensors/, model.safetensors.index.json	Weights: 21 shards in the base layout
config.json, tokenizer, chat_template.jinja, preprocessor configs	Base configuration and processing files, with the provenance addition noted above
serve-spark.sh	Single-Spark launcher
bake-selection.json, frozen-directions.json, frozen-row-norms.json	Frozen trial 117 intervention inputs
reencode/	MXFP8 re-encode script and per-layer report
provenance.json	Build information and validation summary
SHA256SUMS, release-files.json	Release file hashes and sizes
LICENSE, NOTICE, upstream/	Governing terms, notices and NVIDIA's original card

Limitations

• Narrow evaluation: broad capability benchmarks and video input were not evaluated for this derivative. The vision checks were limited to solid colours.
• Profile-specific evidence: single-Spark retrieval checks reached about 30K tokens. The separate two-Spark profile passed the near-262K checks described above. Neither proves reliable performance on every task at those lengths.
• Original targets not met: the project aimed for zero refusals and a KL ceiling of 0.20 and did not meet its original qualification targets. The top-20 KL reported here is distinct from the original full-vocabulary metric.
• Refusals remain: 6 of 100 holdout prompts were detected as refusals.
• Hardware and runtime scope: validation used DGX Spark and pinned B12X runtimes. Other hardware and runtime versions are not qualified here.
• Outputs can be wrong or inconsistent: reduced refusal behavior does not establish correctness, factual reliability or preserved general capability.

Responsible use

Abliteration reduces learned refusal behavior. The model may attempt requests the original declines and can produce harmful, offensive, false or illegal content.

• You are responsible for how you use it and for complying with applicable law and the governing licenses.
• Add your own safeguards before deploying it to end users.
• Do not use it for activities prohibited by the Qwen or NVIDIA terms.

There is no warranty of any kind.

License

This checkpoint is derived from NVIDIA's quantized checkpoint and Qwen's original model. Their governing terms continue to apply; this release does not relicense the base weights.

• NVIDIA Open Model Agreement: the governing download terms for NVIDIA's checkpoint. A copy is included in LICENSE.
• Qwen Community License 1.0: additional terms for the original model, included in upstream/QWEN-LICENSE.

Licensed by NVIDIA Corporation under the NVIDIA Open Model Agreement.

Read the full license texts before use or redistribution, including their commercial-use conditions. Runtime containers and external software have their own licenses.

Citation

@misc{developingdad2026qwen38abliterated,
  title        = {Qwen3.8-Flash-Next FP4-Local -- Abliterated},
  author       = {DevelopingDad},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated}}
}

@article{arditi2024refusal,
  title   = {Refusal in Language Models Is Mediated by a Single Direction},
  author  = {Arditi, Andy and Obeso, Oscar and Syed, Aaquib and Paleka, Daniel and Panickssery, Nina and Gurnee, Wes and Nanda, Neel},
  journal = {arXiv preprint arXiv:2406.11717},
  year    = {2024}
}

@misc{heretic,
  author       = {Weidmann, Philipp Emanuel},
  title        = {Heretic: Fully automatic censorship removal for language models},
  year         = {2025},
  publisher    = {GitHub},
  journal      = {GitHub repository},
  howpublished = {\url{https://github.com/p-e-w/heretic}}
}

Credits

This release stands on other people's work.

Model

• Qwen: Qwen3.8-Flash-Next, the original model, under the Qwen Community License 1.0.
• NVIDIA: the FP4-Local checkpoint, quantized with Model Optimizer (Apache-2.0), under the NVIDIA Open Model Agreement. NVIDIA's published DGX Spark profile is the basis of serve-spark.sh.

Method

• Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee and Neel Nanda: the refusal-direction research behind abliteration.
• Philipp Emanuel Weidmann (p-e-w) and the Heretic contributors: Heretic (AGPL-3.0). Its search parameterization was reimplemented here, and no Heretic code is included. The Heretic-DGX fork was consulted while adapting it.
• Maxime Labonne (mlabonne): the abliteration write-up and the two prompt sets used to find directions and measure refusals. Neither prompt set is redistributed here.
  • mlabonne/harmful_behaviors, revision 01cead0, comes from AdvBench (llm-attacks, MIT).
  • mlabonne/harmless_alpaca, revision 02c6a92, comes from Stanford Alpaca (CC BY-NC 4.0).
• Open Compute Project: the Microscaling (MX) format specification that MXFP8 follows.

Software and serving

• vLLM (Apache-2.0): the serving engine.
• FlashInfer (Apache-2.0) and B12X by Local Inference Lab (Apache-2.0): the kernels.
• eugr/spark-vllm-docker (MIT): the DGX Spark vLLM image used for building and serving.
• Optuna (MIT), PyTorch (BSD-3-Clause) and safetensors (Apache-2.0).
• diffbot: the tuned-stack recipe behind the optional single-Spark speed figure.
• XtraSaltyDev: the separate two-Spark serving recipe, evaluation harnesses and deployment results.

External projects retain their own licenses. This release is not endorsed by Qwen, NVIDIA or any of the projects above.