Instructions to use Goodoldjam/DiffusionGemma-26B-E38-Abliterated-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Goodoldjam/DiffusionGemma-26B-E38-Abliterated-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Goodoldjam/DiffusionGemma-26B-E38-Abliterated-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Goodoldjam/DiffusionGemma-26B-E38-Abliterated-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("Goodoldjam/DiffusionGemma-26B-E38-Abliterated-NVFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Goodoldjam/DiffusionGemma-26B-E38-Abliterated-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Goodoldjam/DiffusionGemma-26B-E38-Abliterated-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Goodoldjam/DiffusionGemma-26B-E38-Abliterated-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Goodoldjam/DiffusionGemma-26B-E38-Abliterated-NVFP4

SGLang

How to use Goodoldjam/DiffusionGemma-26B-E38-Abliterated-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Goodoldjam/DiffusionGemma-26B-E38-Abliterated-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Goodoldjam/DiffusionGemma-26B-E38-Abliterated-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Goodoldjam/DiffusionGemma-26B-E38-Abliterated-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Goodoldjam/DiffusionGemma-26B-E38-Abliterated-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Goodoldjam/DiffusionGemma-26B-E38-Abliterated-NVFP4 with Docker Model Runner:
```
docker model run hf.co/Goodoldjam/DiffusionGemma-26B-E38-Abliterated-NVFP4
```

DiffusionGemma 26B E38 — Abliterated NVFP4

Deployment-oriented NVIDIA-style NVFP4 derivative of the deeply tested E38 DiffusionGemma abliteration.

E38 NVFP4 preserves the measured E38 behavior and capability profile while reducing checkpoint size from 51.68 GB to 18.86 GB and dramatically improving inference performance.

❤️ Support the Next Phase

Goodoldjam/DiffusionGemma-26B-E38-Abliterated-NVFP4

The next phase — mixed-precision quantization research — is also the expensive phase.

The current model has already demonstrated:

660.44 tok/s at the 48-step quality operating point
827.28 tok/s at 16 steps single stream
1,053.64 tok/s aggregate serving throughput on one RTX PRO 6000
137/200 direct aligned NVFP4 objective validation
20/20 multimodal
24/24 matched multi-turn generations
0/402 target refusals
0/249 benign false refusals

Execution-only tuning has reached a useful plateau.

The next major gains are expected to require testing new precision mixes.

That means repeatedly:

quantizing
calibrating
building new artifacts
validating precision layouts
compiling low-precision kernels
screening throughput
rerunning matched quality evaluations
checking long-form behavior
checking structured output
checking multimodal behavior where affected
manually reviewing disagreements

Most candidates will be rejected.

The expensive part is not making another quant.

The expensive part is proving which quant is actually worth releasing.

If you want to help push quality-max performance toward the 1,000 tok/s class, please consider supporting the mixed-quant search.

⚡ 1,000+ tok/s on a Single RTX PRO 6000

E38 NVFP4 has broken 1,000 tok/s on a single RTX PRO 6000 Blackwell workstation GPU.

Performance V3:

Operating Point	Throughput
48-step quality-max	660.44 tok/s
16-step single stream	827.28 tok/s
16-step concurrency-8 aggregate	1,053.64 tok/s

The 1,053.64 tok/s result was achieved on one:

NVIDIA RTX PRO 6000 Blackwell Workstation Edition

with:

96 GB GDDR7
1.792 TB/s physical memory bandwidth
native packed NVFP4 routed experts
compiled vLLM execution
CUDA graphs
FlashInfer CUTLASS NVFP4 MoE
FlashInfer autotuning
CUDA 13 / SM120

Important:

1,053.64 tok/s is concurrency-8 aggregate serving throughput.

It is not the 48-step quality result and it is not a single-stream result.

Single-stream:

827.28 tok/s

48-step quality-max:

660.44 tok/s

🎮 RTX 5090 Owners — This Is One to Watch

Could a 32 GB RTX 5090 also break 1,000 tok/s? Possibly — and that is now a serious target.

The GeForce RTX 5090 is especially interesting for E38 NVFP4 because it provides:

32 GB GDDR7
1.792 TB/s memory bandwidth
Blackwell architecture
fifth-generation Tensor Cores
native FP4 capability

That 1.792 TB/s memory bandwidth is the same raw bandwidth as the RTX PRO 6000 Workstation Edition used for the current E38 NVFP4 testing.

The E38 NVFP4 checkpoint is only:

18.86 GB

so it fits within the RTX 5090's:

32 GB VRAM

Google has already reported:

700+ tok/s DiffusionGemma performance on an RTX 5090

using its throughput-oriented operating point.

E38 NVFP4 has now demonstrated on RTX PRO 6000:

827.28 tok/s single stream

and:

1,053.64 tok/s aggregate serving throughput

This makes:

1,000+ tok/s on an RTX 5090 a plausible experimental target.

It has not yet been directly validated on this checkpoint, so 1,000+ tok/s on 5090 is not currently claimed as an achieved result.

But the hardware characteristics make it worth testing.

Why 5090 Owners Should Care

A 5090 does not need 96 GB VRAM to run this model.

The current E38 NVFP4 checkpoint is:

18.86 GB

That gives a 32 GB 5090 substantial room for:

the model
runtime allocations
CUDA graphs
FP8 KV cache
inference overhead

Exact usable concurrency will depend on the runtime configuration.

The RTX PRO 6000 has more total VRAM and higher theoretical AI compute, so identical performance should not be assumed.

But both cards share:

1.792 TB/s GDDR7 memory bandwidth

and both are Blackwell GPUs with FP4-capable fifth-generation Tensor Cores.

The current research question is therefore very practical:

Can a high-end consumer GPU run a 26B-class DiffusionGemma at or above 1,000 tok/s using an aggressively optimized NVFP4 / mixed-precision runtime?

That is now one of the most interesting deployment targets for this project.

Sources:

https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/

https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/rtx-pro-6000/

https://developers.googleblog.com/en/diffusiongemma-the-developer-guide/

🏆 How Well Does the Model Perform?

E38 NVFP4 is not simply a speed experiment.

The goal of the conversion was to preserve the measured E38 capability profile while gaining the memory and execution advantages of NVFP4.

The aligned validation classification was:

PASS

Direct NVFP4 Validation

Evaluation	Base BF16	E38 BF16	E38 NVFP4
Objective — 200 prompts	128/200 — 64.0%	134/200 — 67.0%	137/200 — 68.5%
Regression subset — 100	62/100	58/100	64/100
Multimodal	20/20	20/20	20/20
Multi-turn conversations	8/8	8/8	8/8
Multi-turn generations	24/24	24/24	24/24
Target refusal	383/402	0/402	0/402
Benign false refusal	0/249	0/249	0/249
Grammar errors /10k words ↓	4.059	5.479	3.236
Lexical artifacts /10k ↓	2.243	1.865	1.387

Direct matched objective validation:

E38 BF16: 134 / 200 — 67.0%

E38 NVFP4: 137 / 200 — 68.5%

Difference:

+1.5 percentage points

with:

95% CI: -2.0 to +5.0 pp
p = 0.5811

The difference was not statistically significant.

Therefore:

NVFP4 preserved measured E38 BF16 quality.

The higher NVFP4 point estimate is not claimed as proof that quantization inherently improves capability.

📈 Public E38 Capability Reference

The larger benchmark suite was run on Base BF16 and the frozen E38 BF16 parent.

Benchmark	Base BF16	E38 BF16	Delta
IFEval	67.10%	64.70%	-2.40 pp
BBH	71.62%	73.96%	+2.34 pp
MuSR	41.80%	50.00%	+8.20 pp
MMLU-Pro	49.61%	51.57%	+1.96 pp
MATH Level 5 @1280	84.06%	80.51%	-3.55 pp

Notable E38 gains:

BBH: +2.34 pp
MuSR: +8.20 pp
MMLU-Pro: +1.96 pp

Measured tradeoffs:

IFEval: -2.40 pp
MATH Level 5: -3.55 pp

The MATH Level 5 regression was statistically significant and is reported openly.

The full benchmark suites have not all been repeated directly on NVFP4.

The aligned three-model validation is therefore used to determine whether NVFP4 introduced a major regression.

It did not.

E38 is best described as:

a capability redistribution rather than a universally stronger checkpoint

🧠 Why the RTX PRO 6000 Result Is Interesting

The RTX PRO 6000 Workstation provides:

1.792 TB/s memory bandwidth

while H100 SXM provides approximately:

3.35 TB/s

The RTX PRO 6000 therefore has only about:

53.5% of H100 SXM's raw memory bandwidth

Yet E38 NVFP4 has demonstrated:

1,053.64 tok/s aggregate serving throughput

on that workstation GPU.

The model representation is also dramatically smaller:

51.68 GB BF16 → 18.86 GB NVFP4

or:

2.74× smaller

The important point is not that the RTX PRO 6000 has more physical bandwidth.

It does not.

The opportunity is:

move substantially less model-weight data through the available memory system while taking advantage of Blackwell low-precision compute.

🔬 Current Research — Mixed-Quant Tuning

Runtime tuning is complete. Mixed-precision quantization is now the active performance branch.

The current checkpoint already places routed experts at NVFP4.

Several major model components remain BF16.

The next question is:

How much of the remaining BF16 workload can move to FP8 or NVFP4 without materially reducing quality?

Current targets:

Quality-Max

Current:

660.44 tok/s

800+ tok/s

while retaining the 48-step quality configuration.

Single Stream

Current:

827.28 tok/s

Target:

1,000+ tok/s

Aggregate Serving

Current:

1,053.64 tok/s

Status:

1,000+ TARGET ACHIEVED

RTX 5090

Current direct E38 NVFP4 result:

Not yet tested

Target:

Determine whether 1,000+ tok/s is achievable on consumer Blackwell

Long-Term Goal

Push the quality-preserving configuration as close as practical to the 1,000 tok/s class.

🔥 The Goal Is Not Simply Throughput

The goal is maximum practical throughput while maintaining the strongest quality profile possible.

There is an easy way to make diffusion generation faster:

make the model perform less diffusion work

That is not the primary objective here.

The project is targeting:

quality + throughput + memory efficiency

The quality operating point retains a much deeper maximum denoising budget than the speed track.

The goal is to gain speed through:

lower-precision weights
less model-weight traffic
native Blackwell FP4 / FP8 execution
compiled execution
CUDA graphs
kernel tuning
better precision placement

rather than simply removing refinement.

✅ Performance V3 — Complete

E38 NVFP4 Performance V3 is complete.

Final Classification

QUALITY RUNTIME

KEEP CURRENT

PURE SPEED BENCHMARK

VALID

No tested execution-only candidate reliably improved the 48-step runtime enough to replace the current configuration.

The quality operating point therefore remains frozen.

🟦 Quality-Max Track

Current inference configuration:

t_max = 0.80
t_min = 0.40
max_denoising_steps = 48
entropy_bound = 0.1
confidence_threshold = 0.005
stability_threshold = 1
adaptive_stopping = true
canvas_length = 256
long_form_capacity = 1280

Downloads last month: 208

Safetensors

Model size

14B params

Tensor type

BF16

Model tree for Goodoldjam/DiffusionGemma-26B-E38-Abliterated-NVFP4

Base model

google/diffusiongemma-26B-A4B-it

Finetuned

Goodoldjam/DiffusionGemma-26B-E38-Abliterated-BF16

Quantized

(1)

this model