LostBeard/FLUX.2-klein-4B-q4-ILGPU

FLUX.2 [klein] 4B, 4-bit export for SpawnDev.ILGPU.ML

black-forest-labs/FLUX.2-klein-4B (Apache-2.0) repacked so it fits in a browser tab and runs on the SpawnDev.ILGPU.ML engine (C# kernels compiled to WebGPU, CUDA, OpenCL; no PyTorch, no ONNX Runtime). The weights are modified: large matrices are quantised to 4-bit (or, in the q8 files, 8-bit) blocks. Nothing else about the model is changed. All credit for the model goes to Black Forest Labs.

file	what	size
text_encoder_q4.safetensors	Qwen3-4B text encoder, first 27 of 36 layers (the pipeline only reads hidden states 9, 18, 27) + embeddings	1.95 GB
transformer_q4.safetensors	the 3.9B flow transformer	2.42 GB
text_encoder_q8.safetensors, transformer_q8.safetensors	the same stages in 8-bit blocks (near lossless: relative error 0.5% per matrix instead of 8%): for desktop GPUs, 3.5 GB + 4.4 GB, 4.4 GB on the GPU at a time	7.9 GB
vae_decoder.onnx	the VAE decoder (post_quant_conv + decoder), fp32, 512 px graph	199 MB
vae_norm.json	the 128-channel batch-norm statistics applied to the packed latents
tokenizer.json	the Qwen2 tokenizer

8-bit files use the same header word (fp16 scale and offset) followed by 8 words of 4 bytes per block of 32 (q8_1, 9 words per block, value = d * byte + m); the file's quant metadata says q8 or q4.

Quantisation format ("Q4_1"-style): every weight matrix [N, K] with K % 32 == 0 and at least a million elements is cut along K into blocks of 32 and stored as int32 words [N, K/32*5]: word 0 = fp16 scale d (low half) and fp16 offset m (high half), words 1..4 = 32 nibbles (element 8*i + j of the block is (word[1+i] >> 4*j) & 15); value = d * nibble + m. Norm weights and small tensors stay bf16. The engine dequantises one matrix at a time into a bf16 scratch just before its GEMM. Per-matrix relative error is about 8%. tools/unirig-harness/reference/quantize_flux2.py in the repo writes this format; export_flux2.py and export_flux2_vae.py made the bf16 sources (fused q/k/v and gate/up rows).

Measured on an RTX 4070 (CUDA), 512 px, 4 steps: 19 s including loading, 2.6 GB of weights resident at a time; with bf16 weights the same pipeline matches the diffusers reference at 29 dB PSNR.

Use

var pipe = await Flux2Pipeline.CreateAsync(accelerator, modelSource, "LostBeard/FLUX.2-klein-4B-q4-ILGPU");
var image = await pipe.RunAsync(new Flux2Request { Prompt = "a photo of a cat on a red sofa", Seed = 1 });   // image.Rgba, 512 x 512

Live demo: https://lostbeard.github.io/SpawnDev.ILGPU.ML/text-to-3d (text to 3D character, rigged and animated, all in the browser).

License

Apache License 2.0, the license of the original model (see LICENSE). Model, intended use, limitations and safety evaluation: see the original model card.