capyctl/FrogNano-4B-2609-NVFP4

FrogNano-4B-2609, NVFP4 (ModelOpt mixed precision)

An NVFP4 quantization of microsoft/FrogNano-4B-2609 (revision b90468c1), made with NVIDIA ModelOpt 0.47.0. It follows the layout of nvidia/Qwen3.8-27B-NVFP4: NVFP4 for the MLP and output layer, FP8 for the attention and GatedDeltaNet projections, everything else BF16. vLLM, SGLang and TensorFold load it as a modelopt MIXED_PRECISION checkpoint. About 4.8 GB, against 8.8 GB for the original.

All credit for the model goes to its authors. Read the original model card for intended use, limitations and safety guidance; they apply unchanged. This repository changes only the storage format.

What changed from the original

Layers	Format
mlp.{gate,up,down}_proj (32 layers) and lm_head	NVFP4: FP4 (E2M1) weights with an FP8 (E4M3) scale per 16 values and an FP32 scale per tensor; NVFP4 input activations with a static input scale (W4A4)
self_attn.{q,k,v,o}_proj (8 full-attention layers), linear_attn.{in_proj_qkv,in_proj_z,out_proj} (24 GatedDeltaNet layers)	FP8 (E4M3) weights and static FP8 input activations, one scale per tensor (W8A8)
embed_tokens, linear_attn.in_proj_a / in_proj_b, conv1d, A_log, dt_bias, all norms, vision tower, MTP layer	BF16, bit-identical to the original

KV cache	Not quantized (kv_cache_quant_algo: null)
Output layer	Untied. The original ties lm_head to embed_tokens. Here lm_head starts as an exact copy of embed_tokens and is then quantized to NVFP4, while the embedding stays BF16. tie_word_embeddings: false. TensorFold's NVFP4 path reads a separate lm_head.
Architecture	Unchanged: Qwen3_5ForConditionalGeneration. The BF16 vision tower and MTP layer are kept so the checkpoint keeps the original's architecture name. The vision tower was not tested with this checkpoint.
Tokenizer, chat template	The original files, unchanged.
generation_config.json	Added, with eos_token_id: [248046, 248044] (<|im_end|>, <|endoftext|>), so replies stop at the end of a turn.

How it was made

• nvidia-modelopt 0.47.0 with transformers 5.14.1 and torch 2.13.0, using ModelOpt's examples/hf_ptq/hf_ptq.py from the 0.47.0 tag.
• Recipe: ModelOpt's huggingface/qwen3_5/ptq/w4a16_nvfp4-fp8_attn-kv_fp8_cast, changed in two ways to match the NVIDIA checkpoint: NVFP4 input quantizers on the MLP and lm_head (W4A4), and no KV-cache cast. Calibration algorithm max.
• Calibration: 256 conversations, rendered with the original chat template and cut to 512 tokens, batch size 4.
  • 128 chat conversations from HuggingFaceH4/ultrachat_200k (MIT), test_sft split.
  • 128 code problems and solutions from ise-uiuc/Magicoder-OSS-Instruct-75K (MIT), mostly Python, also PHP, TypeScript, Swift, C++, Rust, C# and others.
  • Rows taken at an even stride, so the set can be rebuilt exactly.

Serve it

The commands below, with these flags, were run against a local copy of this checkpoint on an RTX 4090 Laptop GPU (16 GB, SM 8.9). That GPU has no FP4 tensor cores, so every engine ran the NVFP4 layers as FP4 weights with BF16 activations (W4A16). GPUs with native FP4 (SM 10.x, SM 12.x) run them as FP4 x FP4.

vLLM 0.30.0:

vllm serve capyctl/FrogNano-4B-2609-NVFP4 --max-model-len 16384 --gpu-memory-utilization 0.75 \
  --tool-call-parser qwen3_coder --reasoning-parser qwen3 --enable-auto-tool-choice

vLLM detects modelopt_mixed on its own. On SM 8.9 it picks its Marlin NVFP4 kernel and warns that the GPU has no native FP4.

SGLang 0.5.21:

SGLANG_DISABLE_SILU_FP4_QUANT_FUSION=1 python -m sglang.launch_server \
  --model-path capyctl/FrogNano-4B-2609-NVFP4 --context-length 16384 --mem-fraction-static 0.7 \
  --fp4-gemm-backend marlin --reasoning-parser qwen3 --tool-call-parser qwen3_coder

The environment variable and --fp4-gemm-backend marlin are what made it start on SM 8.9: without the variable, SGLang's fused SiLU and FP4-quantize kernel has no SM 8.9 build and CUDA graph capture fails. On GPUs with native FP4, these two settings should not be needed, but that has not been tested yet.

TensorFold 0.6.3, CUDA:

tensorfold serve capyctl/FrogNano-4B-2609-NVFP4 --no-drafts --context 16384 --backend cuda --snapshot-dir none

--no-drafts is required: there is no drafter for this model. TensorFold skips the vision tower and MTP layer for NVFP4 checkpoints. On SM 8.9 the first request after a fresh install waits a few minutes while its CUDA kernels compile.

Quality check

A small greedy comparison against the original BF16 weights on vLLM 0.30.0, thinking off, max_tokens 1024. Generated code was run against unit tests in a sandbox. This is a spot check, not SWE-bench or a perplexity measurement. All runs on the RTX 4090 Laptop GPU, so the NVFP4 layers ran as W4A16.

Prompts	This checkpoint, vLLM 0.30.0	This checkpoint, TensorFold 0.6.3	Original BF16, vLLM 0.30.0
Python coding, 30	26	27	26
Tool calls, 30 (right tool and arguments)	29	29	28
Reasoning and math, 20	20	20	20
Total, 80	75 (94%)	76 (95%)	74 (93%)

The differences from BF16 are one or two prompts, in both directions, which is within noise for a set this size. Every tool call was valid JSON, and 29 of 30 tool calls were identical to the BF16 ones. Every reasoning answer matched BF16's. The wording of longer answers drifts from BF16 after about 20 tokens.

With thinking on (10 prompts, max_tokens 4096) this checkpoint passed 9 on vLLM and 8 on TensorFold; BF16 passed 9. On TensorFold, 2 of the 10 replies kept re-checking their answer until the token limit and never gave one; on vLLM one similar reply finished just under the limit. BF16 did not do this. Set a max_tokens limit when thinking is on.

Speed and quality with native FP4 x FP4 (NVIDIA GB10) will be added after they are measured.

License

The original repository is tagged MIT, and its card's license row names Apache 2.0. This derived checkpoint keeps the original's terms and attribution; check the original repository for them. The calibration data (UltraChat 200k and Magicoder OSS-Instruct 75K) is MIT licensed and is not included in this repository.