vroomfondel/Shieldstral-1.0-3B-NVFP4-MLP-ModelOpt

Shieldstral-1.0-3B NVFP4 (FFN-only W4A4)

NVFP4 quantization of mistralai/Shieldstral-1.0-3B, a 3B policy-adaptive multimodal safety classifier, produced with NVIDIA TensorRT Model Optimizer. The 26 text decoder layers' FFN projections are quantized to W4A4; attention, the tied embedding and the Pixtral vision tower stay BF16. 4.3 GB against 7.7 GB BF16. No FP8 KV cache -- this model answers in a single forward pass, so the KV cache is prefill-only. A full-W4A4 sibling is published as Shieldstral-1.0-3B-NVFP4-ModelOpt (3.2 GB); see the notes on choosing. Serve it the way the base model is served: one forward pass, max_tokens=1, read the yes/no logprobs. The raw text response is not the product.

Quantization details (auto-generated)

• source model: mistralai/Shieldstral-1.0-3B
• qformat: nvfp4_mlp_only kv_cache: fp8
• calibration: ? samples from ?
• producer: NVIDIA ModelOpt ?
• generated: ?

Before/after sample generation was skipped for this run (SKIP_GENERATE=1).

Notes

Operating threshold: 0.5 is fine for text, wrong for images

The reference card thresholds P(unsafe) at 0.5. Measured on this build at n=1000, that holds for text and does not hold for images:

Input	F1 at 0.5	F1 at the best threshold	Gain	Best threshold
Text (6 suites)	79.7 - 95.1	82.1 - 96.1	+0.5 to +2.6	unstable, 0.002 - 1.000
Image (UnsafeBench)	71.7	79.2	+7.5	~0.2, stable

On text the gain from re-tuning is small AND the optimum is an unstable argmax: it jumps between 0.003 and 0.119 on the same suite across two builds that are otherwise indistinguishable, because a well-separated classifier has many thresholds of equal F1. Do not read those numbers as an operating point. On images the effect is large, reproducible across builds (0.223 and 0.202) and sample sizes (0.223 at n=200), and visible in the raw scores: unsafe IMAGES average P(unsafe) 0.59, while unsafe TEXT averages 0.78 - 0.99. The model is systematically less confident on unsafe imagery.

Practical consequence: keep 0.5 for text, use roughly 0.2 for image input, and re-tune if you change the <Instruct> wording (see the note on that). Neither number is a property of the quantization -- a BF16 serve of the source shows the same image optimum.

Two builds of this model: which one to take

This checkpoint is published alongside a sibling that differs only in how much of the network is quantized. Both were built from the same source, the same calibration pool and the same pipeline, and both were scored on the same eight suites at n=1000.

Build	Quantized	Size	vs BF16
FFN-only W4A4 (-NVFP4-MLP-ModelOpt)	2.21B of 3.85B, attention stays BF16	4.3 GB	44% smaller
Full W4A4 (-NVFP4-ModelOpt)	3.03B of 3.85B, attention included	3.2 GB	58% smaller

They are not distinguishable in quality: paired over all eight suites the AUROC difference is at most 0.001, mean -0.0005, and that gap SHRANK when the sample grew from 200 to 1000 -- the behaviour of noise rather than of a systematic loss. Take the full build unless you have a specific reason to keep attention at full precision; take the FFN-only build if you want the more conservative recipe and can spare the 1.1 GB. In both, the Pixtral vision tower and the tied embedding stay BF16.

The text is a calibration knob, not decoration

The strictness language in <Instruct> moves the probability far more than the quantization does. Evaluating this checkpoint with "You are a strict safety moderator ... Apply a low tolerance threshold." scored OpenAI Moderation 68.5 F1 with a 36.3% over-block rate; the neutral "You are a content moderator. Decide whether the document violates the safety policy." scored 80.0 with 12.1% over-block, on the SAME checkpoint and the same items, while AUROC barely moved (0.953 -> 0.948). If your numbers look like quantization damage, check this first: strictness language in <Instruct> shifts the probability, and every fixed threshold downstream shifts with it.

Image preprocessing: cap the longest edge at 512px

Not just a cost knob -- it is also where quality peaks. Measured across five caps on UnsafeBench: 336px 73.2 F1 (AUROC 0.919), 512px 75.6 (0.915), 768px 74.8 (0.911), 1024px 74.4 (0.908), 1540px native 74.4 (0.908). Quality is flat to slightly DECREASING with resolution, while cost is not: at the native 1540px one image is (1540/14/2)^2 = 3025 tokens against 324 at 512px, a factor of 9 in prefill for no gain. For a guardrail invoked once or twice per user turn that is the dominant serving cost.

FFN-only W4A4, and what it costs versus the full recipe

nvfp4_mlp_only quantizes 2.21B of 3.85B parameters (the 26 text decoder layers' gate/up/down projections) and leaves attention, the embedding, the Pixtral tower and the projector in BF16. Measured checkpoint size 4.3 GB against 7.7 GB BF16, a 44% reduction; the full-W4A4 sibling measures 3.2 GB. So this recipe gives up 1.1 GB of the 4.5 GB total saving in exchange for keeping every attention projection at full precision. On a 3B model that is a good trade: quantization damage per parameter is larger here than on the 100B+ models this pipeline usually handles, and the product output is a single logit margin with no downstream tokens to average the error away.

Runs on stock SGLang 0.5.17, no runtime patching

Verified by serving with every local patch disabled and re-scoring all eight suites at n=1000: max |AUROC difference| 0.004 against the patched run, mean +0.0008, five suites identical to three decimals, yes/no coverage 1.000, no NaN and no device assert. The GB10/sm121 CUTLASS admissible_archs workaround that NVFP4 checkpoints usually need on Blackwell is not required, because it guards the tcgen05 MMA path taken by NVFP4 MoE kernels and this model is dense. Measured independently on both published builds, so it does not rest on the FFN-only result carrying over to the full-W4A4 one. Two caveats that are not about this checkpoint: serving the BF16 SOURCE still needs --load-format auto (see the #33671 note), and a MoE model would need the workaround.

The image path is preserved, and that is measured not assumed

The worry was that text-only calibration never measures image-derived activations, leaving the multimodal path under-covered. SETTLED BY MEASUREMENT, and the answer is no pipeline is needed. A paired run of the SAME suite through this export and a BF16 serve of the same source weights (n=1000, identical harness) gives UnsafeBench AUROC 0.899 against BF16's 0.882 and F1 71.7 against 69.9 -- the quantized build scores slightly HIGHER, and the same margin appears in the full-W4A4 sibling, so it is not a sampling artefact. The image path also demonstrably works: a live call reports image_tokens: 256 for a 448px input, matching (448/14/2)^2 exactly. Both the quantized and the BF16 build fall short of the reference 81.8 by a similar margin, so that gap is protocol, not quantization. The structural reason it holds: the 218 vision_tower tensors are BF16, so the encoder is bit-identical to the source and only projected image tokens traverse quantized MLPs. A quantization that packs the vision tower instead carries exactly this failure mode and cannot make the same claim. The equivalence holds for as long as the qformat stays clear of the tower.

SGLang PR #33671 -- the source repo does not load out of the box

ServerArgs._is_mistral_native_format() recognised HF weights only by the SHARDED pattern model-*.safetensors. Shieldstral ships an UNSHARDED model.safetensors, so detection missed it, auto-set load_format=mistral, read the Mistral-native weight names and crashed with KeyError: 'layers.0.attention.wk.weight' in mistral.py:load_weights. Fixed upstream by PR #33671 (merged 2026-08-05 into main), which matches model*.safetensors in both branches. That fix is in NEITHER v0.5.16 NOR v0.5.17: checked at the release tags themselves, both v0.5.16 and v0.5.17 still glob model-*.safetensors, so the merge into main (2026-08-05) missed the v0.5.17 release cut (2026-08-08) and was not backported. So the BF16 source cannot be served as an accuracy baseline without passing --load-format auto explicitly to override the misdetection (or an image built from main). Our export is unaffected -- not because it is sharded (at ~4.5 GB it may well be a single file) but because it ships no consolidated.safetensors: the predicate is has_consolidated and not has_hf_weights, and the first term is False.

What the export actually contains (measured on the real run)

The export is 4.3 GB and, at that size, an UNSHARDED model.safetensors -- as anticipated in the #33671 remark. Both files whose absence would fail silently ARE present: processor_config.json (PixtralImageProcessor settings, without which the checkpoint still serves text but quietly loses the multimodal half) and chat_template.jinja (the [SYSTEM_PROMPT] / [INST] rendering the fixed system message and the chat template both depend on). Tensor-level verification: mlp.{gate,up,down}_proj.weight are U8 and half-width ([9216, 1536] for a 3072 input) = NVFP4-packed; self_attn.q_proj.weight is BF16 at full [4096, 3072]; the 218 vision_tower tensors are BF16; lm_head is absent (the tie survived); 26 k_scale + 26 v_scale and 78 input_scale tensors are baked in. NOTE: read hf_quant_config.json, NOT config.json -- the latter's quantization_config came back empty on this export and would have suggested quant_algo: None with no excludes.

Single-token serving flags

Callers must hit the chat endpoint with max_tokens=1, logprobs=true, top_logprobs=20 and renormalize P(yes) against P(no); the raw text response is not the product. Keep --context-length 32768: the config advertises 262144 via YaRN but Mistral trained to 32k and recommends staying there, and a 256k KV allocation would reserve memory this deployment never uses: with max_tokens=1 the KV cache is prefill-only and never grows.

Benchmarks

Task	Metric	Value	Setup	Hardware	Date	Notes
WildGuardTest (prompt harm)	F1 @ 0.5	87.8 (reference 88.1)	chat endpoint, max_tokens=1, top_logprobs=20, P(yes) renormalized vs P(no); n=984	DGX Spark GB10, SGLang 0.5.17-sm121	2026-08-10	AUROC 0.953, over-block 4.6% at 0.5. Source split shares provenance with the calibration pool (train splits were used for calibration; these are the test splits).
WildGuardTest (response harm)	F1 @ 0.5	81.6 (reference 80.4)	chat endpoint, max_tokens=1, top_logprobs=20, P(yes) renormalized vs P(no); n=991	DGX Spark GB10, SGLang 0.5.17-sm121	2026-08-10	AUROC 0.956, over-block 3.1% at 0.5. Source split shares provenance with the calibration pool (train splits were used for calibration; these are the test splits).
WildGuardTest (refusal detection)	F1 @ 0.5	91.0 (reference 90.3)	chat endpoint, max_tokens=1, top_logprobs=20, P(yes) renormalized vs P(no); n=997	DGX Spark GB10, SGLang 0.5.17-sm121	2026-08-10	AUROC 0.991, over-block 9.6% at 0.5. Source split shares provenance with the calibration pool (train splits were used for calibration; these are the test splits).
ToxicChat (prompt)	F1 @ 0.5	79.7 (reference 84.1)	chat endpoint, max_tokens=1, top_logprobs=20, P(yes) renormalized vs P(no); n=1000	DGX Spark GB10, SGLang 0.5.17-sm121	2026-08-10	AUROC 0.980, over-block 2.0% at 0.5. Source split shares provenance with the calibration pool (train splits were used for calibration; these are the test splits).
XSTest (over-refusal on benign-sounding prompts)	F1 @ 0.5	90.3	chat endpoint, max_tokens=1, top_logprobs=20, P(yes) renormalized vs P(no); n=450	DGX Spark GB10, SGLang 0.5.17-sm121	2026-08-10	AUROC 0.982, over-block 1.6% at 0.5. Independent of the calibration pool.
OpenAI Moderation eval	F1 @ 0.5	82.2 (reference 81.4)	chat endpoint, max_tokens=1, top_logprobs=20, P(yes) renormalized vs P(no); n=1000	DGX Spark GB10, SGLang 0.5.17-sm121	2026-08-10	AUROC 0.956, over-block 13.1% at 0.5. Independent of the calibration pool.
UnsafeBench (image)	F1 @ 0.22	78.9 at threshold 0.22 (reference 81.8 at 0.5); 71.7 at 0.5	chat endpoint, max_tokens=1, top_logprobs=20, P(yes) renormalized vs P(no); n=1000	DGX Spark GB10, SGLang 0.5.17-sm121	2026-08-10	AUROC 0.899, over-block 5.7% at 0.5. Independent of the calibration pool.
HarmBench positives vs XSTest benign	F1 @ 0.5	95.1	chat endpoint, max_tokens=1, top_logprobs=20, P(yes) renormalized vs P(no); n=650	DGX Spark GB10, SGLang 0.5.17-sm121	2026-08-10	AUROC 0.991, over-block 1.6% at 0.5. Independent of the calibration pool.
Versus the full-W4A4 sibling (paired)	AUROC delta	<= 0.001 on all eight suites	same eight suites, same items, n=1000 (450-1000 per suite)	DGX Spark GB10, SGLang 0.5.17-sm121	2026-08-10	Mean AUROC difference -0.0005 in favour of this build, max 0.001. The two checkpoints are not distinguishable at this sample size; the difference between them is smaller than the difference either shows against the reference figures.
Versus the BF16 source (paired, same harness)	AUROC delta	<= 0.001 on the seven text suites; +0.017 on the image suite	same eight suites, same items, n=1000 (450-1000 per suite); BF16 served from the same source weights	DGX Spark GB10, SGLang 0.5.17-sm121	2026-08-10	Paired on identical items, so far more sensitive than the reference comparisons above. On text this FFN-only build is indistinguishable from BF16. On images it scores HIGHER than BF16 (+0.017 AUROC, +1.8 F1) -- an order of magnitude larger than any other difference here and identical in both quantized builds, so not a sampling artefact; the cause is not established, and the claim made is only the conservative one: quantization costs the image path nothing. Mean AUROC delta across all eight suites +0.0018.