Instructions to use magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8")

# Load model directly
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8")
model = AutoModel.from_pretrained("magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8

SGLang

How to use magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8 with Docker Model Runner:
```
docker model run hf.co/magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8
```

Qwen3.8-27B-heretic-ara-DFlash2 — selective FP8 experiment

This is an experimental, mixed-precision FP8 quantization of alphakek/Qwen3.8-27B-heretic-ara-DFlash2, prepared and validated by magiccodingman.

It is a target-specific speculative-decoding drafter, not a standalone language model. Use it only with heretic-org/Qwen3.8-27B-heretic-ara or a weight-compatible quantization of that exact target, such as the paired magiccodingman/Qwen3.8-27B-heretic-ara-fp8 repository.

Why selective FP8

Quantizing every drafter matrix was measurably less stable. The selected profile quantizes the large MLP projections and preserves DFlash-sensitive paths in BF16.

Property	Value
Weight format	FP8 E4M3 with BF16 exclusions
Scale granularity	128 × 128 blocks
Activation metadata	Dynamic FP8
FP8 matrices	15: gate/up/down projections across five layers
BF16 tensors	66, byte-identical to the source
Tensor payload	2.51 GB (2.34 GiB)
Source BF16 payload	3.85 GB
Payload reduction	34.73%

The following remain BF16 deliberately:

Q/K/V projections required by the current fused context-KV path
attention output projections
target-hidden-state adapter
dynamic convolution projections and kernels
candidate-selector projection and codebooks
norms and other small parameters

Validation

Eight prompts were run through the real BF16 27B target. Identical hidden states from target layers 5, 19, 33, 47, and 61 were supplied to the BF16 and selective- FP8 drafters, producing 56 compared draft positions through the target's actual 248,320-token LM head.

Metric	Result
Mean `D_KL(P_BF16
Median proposal KL	0.0015949
P95 proposal KL	0.0030100
Maximum proposal KL	0.0085216
Proposal top-1 agreement	96.43%
Top-16 candidate overlap	97.66%
Draft-hidden cosine similarity	0.9995342
Selector path-token agreement	85.71%
Complete seven-token path agreement	75.0%

Structural validation, block geometry, finite scales, preserved-tensor hashes, controlled MLP arithmetic, official dflash 0.1.0 loading, and repository checksums passed. See FP8_EXPERIMENT_VALIDATION.md and the included JSON reports for full details.

The selector-path metric is intentionally strict: one early token difference changes the predecessor used by later selector positions. Target verification protects the final output distribution, but proposal drift may still change acceptance length and throughput.

Usage with the paired target

The drafter repository intentionally has no tokenizer or multimodal processor. Those artifacts come from the target model repository and were not present in the upstream drafter repository.

Example using the proposed Hub repository names:

vllm serve magiccodingman/Qwen3.8-27B-heretic-ara-fp8 \
  --tensor-parallel-size 4 \
  --reasoning-parser qwen3 \
  --kv-cache-dtype fp8 \
  --speculative-config '{"model":"magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8","method":"dflash","num_speculative_tokens":6}'

At the time this checkpoint was validated, DFlash2 support for vLLM was still being developed in vllm-project/vllm#52816. Use a compatible DFlash2-enabled build. Backend support for this selective block-FP8 layout must also be verified on the deployment system.

Hardware note: RTX 3090 does not execute native W8A8 FP8. The local tests validated the serialized weights and proposal behavior by dequantizing FP8 matrices to BF16. Native FP8 kernel throughput was not measured.

Performance expectation

The upstream BF16 drafter card reports 123.2 tok/s and 4.24 average acceptance length on its 4× RTX 3090 TP4 test, versus 116.9 tok/s and 4.01 for the stock z-lab drafter. Those are upstream BF16 results, not results for this FP8 derivative.

Before relying on this checkpoint, compare the BF16 and FP8 drafters on the same hardware, prompts, concurrency, speculative-token count, and runtime build. Measure both tokens/second and acceptance length.

Provenance and credits

The derivative chain is:

Qwen Team — Qwen/Qwen3.8-27B
heretic-org/Qwen3.8-27B-heretic-ara, the target model
z-lab/Qwen3.8-27B-DFlash2, the stock target-conditioned DFlash2 drafter
alphakek/Qwen3.8-27B-heretic-ara-DFlash2, SpecForge-tuned for the Heretic ARA target and warm-started from the z-lab drafter
This selective-FP8 conversion by magiccodingman

DFlash and its implementation are credited to the z-lab/dflash project and its authors. The upstream target's ARA transformation credits p-e-w/heretic, timrohrbaugh/heretic, and the ARA contribution.

This quantization is an independent derivative and is not an official Qwen, Heretic, z-lab, Inco, alphakek, or vLLM release.

Limitations and responsibility

This checkpoint is experimental. It may load correctly yet provide no speedup if the runtime dequantizes its FP8 matrices or lacks compatible fused kernels. Selector-path drift can reduce speculative acceptance even when target verification preserves final-generation correctness. It also inherits the target model's limitations and reduced refusal behavior. Users are responsible for deployment safeguards, legal compliance, and generated content.

License

Apache License 2.0. See LICENSE. All upstream notices and attributions remain applicable.

Downloads last month: 10

Safetensors

Model size

2B params

Tensor type

BF16

F8_E4M3

Model tree for magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8

Base model

Qwen/Qwen3.8-27B

Finetuned

heretic-org/Qwen3.8-27B-heretic-ara

Finetuned

alphakek/Qwen3.8-27B-heretic-ara-DFlash2

Quantized

(1)

this model