Laguna-S-2.1-NVFP4

A faster, leaner build of poolside/Laguna-S-2.1 for local Blackwell — up to 1.67× faster decode and ~4 GB lower VRAM, at measured quality parity. Drop-in for vLLM.

Poolside built Laguna-S-2.1 as an agentic software-engineering model — and for its size (118B total, ~8B active MoE) it's a capable coder. But it isn't narrow: it keeps broad general knowledge and reasoning (MMLU 0.81, GSM8K 0.92), and its native tool-calling and multi-step behavior make it a strong general-purpose agentic model — well-suited to non-coding work like support and operations automation, research assistants, and multi-step coordination, not just code.

Highlights

• Faster single-stream decode across Blackwell — 1.67× on a DGX Spark (GB10, 32.0 vs 19.1 tok/s) and 1.38× on an RTX Pro 6000 (145.7 vs 105.5 tok/s), wherever decode is memory-bandwidth-bound.
• ~4 GB smaller weights — the attention quant cuts the model's weight footprint by ~4 GB (device-resident and on-disk), freeing memory for more context and larger batches on any Blackwell GPU.
• Quality parity within measured evaluation variance — matches the stock NVFP4 checkpoint across MMLU, HellaSwag, GPQA-Diamond, GSM8K, and agentic SWE-bench (all deltas sub-σ).
• Scales from one Spark to two — a second GB10 (TP=2 over 400G) adds ~1.7× concurrent-serving throughput.
• Drop-in — quantization is auto-detected; identical serving recipe to poolside/Laguna-S-2.1-NVFP4, including DFlash speculative decoding and the 256K→1M context switch.

Accuracy & quality

Measured against the stock poolside/Laguna-S-2.1-NVFP4 as the reference, on a neutral serve (no sampling overrides). The multiple-choice suites use loglikelihood; GSM8K and SWE-bench are generative.

Benchmark	Stock NVFP4	This build	Δ
MMLU (5-shot)	0.812	0.810	~parity (0.47σ)
HellaSwag	0.811	0.806	−0.5 pt
GPQA-Diamond	0.434	0.434	0.000
GSM8K (5-shot)	0.913	0.916	+0.3 pt
SWE-bench Multilingual (agentic)	81.2%	78.1%	−1 instance *

* SWE-bench Multilingual is single-attempt agentic (n=32 like-for-like). The −1-instance difference is ≈0.4σ — inside normal agentic-loop run-to-run variance, and consistent with the zero-degradation measured on the large-n multiple-choice suites. It is not a measurable regression; the agentic test is the most sensitive to attention quantization, which is why it is reported here.

Evaluation methodology

Both this build and the stock reference were evaluated on the same vLLM 0.25.1 / RTX Pro 6000 (sm_120) stack — compressed-tensors NVFP4 kernels (MarlinNvFp4 GEMM + FlashInfer-CUTLASS NVFP4 MoE), greedy (temperature 0), neutral serve (no sampling overrides), --kv-cache-dtype fp8. NVFP4 numerics are consistent across the Blackwell backends, so these figures transfer to the GB10/Marlin serving path.

• Multiple-choice — lm-evaluation-harness v0.4.12. MMLU (5-shot), HellaSwag, GPQA-Diamond via loglikelihood (--model local-completions, no chat template); GSM8K (5-shot) generative (local-chat-completions, chat template applied).
• SWE-bench Multilingual (test split) — mini-SWE-agent 2.4.6 + SWE-bench harness 4.1.0, native tool-calling scaffold (poolside_v1 reasoning/tool parsers, auto tool choice), 250-step limit, thinking off, single attempt, 8 workers. Agentic numbers are scaffold-dependent; this is the scaffold used. n=41 raw (this build 27/41, stock 30/41); like-for-like on the 32 instances where both produced a patch: 25/32 vs 26/32.

Performance

All figures are measured on vLLM 0.25.1, temperature 0, --kv-cache-dtype fp8, at tokenizer-exact context lengths, single stream.

Decode on a single DGX Spark (GB10), single-stream (2K):

	This build	Stock NVFP4	Speedup
No spec-decode	32.1 tok/s	19.2 tok/s	1.67×
+ DFlash (k=3)	43.2 tok/s	35.0 tok/s	1.23×

The fastest single-GB10 config is this build + DFlash, 43.2 tok/s. DFlash speeds up both builds on one node; the stock model gains more in relative terms (it starts slower) but never overtakes this build in a like-for-like config.

Two Sparks (TP=2, 2× ConnectX-7 = 400G).

A second GB10 raises single-stream decode to 46.0 tok/s (1.60× stock's 28.8) — just past a single node with DFlash (43.2) — and adds concurrent-serving throughput. Serve on one GB10 for the lowest-cost single stream; add a second for higher single-stream latency and concurrency.

⚠️ On two nodes, run this build without spec-decode. Even with the DFlash draft pinned to a single node (draft_tensor_parallel_size=1, so it pays no cross-node cost — verified from the serve logs), DFlash reaches only 40.6 tok/s, below plain no-spec's 46.0 — because this build's quantized attention already makes no-spec decode bandwidth-fast, leaving nothing for speculation to recover. (DFlash does help the stock model on two nodes, 28.8→39.2, which starts from a slower baseline.) Expert-parallel (--enable-expert-parallel) is a wash vs TP=2 here. Use DFlash on a single GB10; run plain on two.

Decode vs context length (single GB10) — the advantage holds as context grows, because Laguna's sliding-window attention keeps KV growth small (36 of 48 layers use a 512-token window):

Context	This build (1 Spark)	Stock NVFP4	Speedup
2K	32.1	19.2	1.67×
16K	30.6	18.6	1.65×
64K	26.6	17.1	1.56×

Prefill throughput scales with context as chunked prefill amortizes — this build reaches 25K / 145K / 290K tok/s at 2K / 16K / 64K on one GB10, matching or ahead of stock. DFlash lifts decode but adds to TTFT, roughly halving prefill throughput, so prefer it for decode-heavy (long-generation) workloads.

RTX Pro 6000 (sm_120, discrete Blackwell): single-stream decode 145.7 vs 105.5 tok/s = 1.38× faster — RTX is memory-bandwidth-bound at batch 1, so quantizing attention speeds it up here too. At 32-way concurrency the workload turns compute-bound (NVFP4 MoE GEMMs dominate) and the two builds converge (1399 vs 1315 tok/s, 1.06×), so the RTX gain is an interactive/single-stream one. The ~4 GB smaller footprint applies regardless.

Tuning DFlash (single GB10). Draft depth k=3 is optimal; higher k loses because each extra verified token pulls in more MoE experts (the verify is a mini-prefill) while acceptance saturates (~2.85 accepted tokens/cycle). Poolside's example k=15 is tuned for B200-class bandwidth, where the expert-read barely registers.

Concurrent serving (two Sparks). For many parallel users the best layout flips. Pure expert-parallel — --data-parallel-size 2 --enable-expert-parallel (one 128-expert set per node, --tensor-parallel-size 1, no TP) — overtakes TP=2 as concurrency rises, reaching 790 vs 505 tok/s at 128-way (1.56×); TP=2 still wins single-stream. Rule of thumb: TP=2 for latency, DP+EP for throughput. (Spec-decode is a single-stream tool — it collapses under concurrency on TP=2, so leave it off for high-throughput serving.)

Running on a DGX Spark

Two ways to run on a single GB10, depending on how much single-stream speed you want:

	drafter	single-stream decode	install
A — Standard (vLLM)	poolside BF16	~44 tok/s	pip + vllm serve
B — Maximum (Docker)	our 4-bit, bundled	~47.7 tok/s	docker build + docker run

Both serve an OpenAI-compatible endpoint. Start with A; use B for the fastest single stream (it bundles our 4-bit DFlash drafter, which has a cheaper draft step, plus the one loader fix vLLM still needs to load a 4-bit drafter).

A. Standard — vLLM drop-in

1. One-time environment (vLLM 0.25.1 + FlashInfer for sm_121):

sudo apt install -y python3.12-dev                       # Triton JIT needs the headers
curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv ~/venvs/vllm025 -p 3.12
uv pip install -p ~/venvs/vllm025 vllm==0.25.1 --torch-backend=cu130
uv pip install -p ~/venvs/vllm025 \
  "flashinfer-python==0.6.15.dev20260712" "flashinfer-cubin==0.6.15.dev20260712" "flashinfer-jit-cache==0.6.15.dev20260712" \
  --extra-index-url https://flashinfer.ai/whl/nightly/ --extra-index-url https://flashinfer.ai/whl/nightly/cu130/ \
  --index-strategy unsafe-best-match
source ~/venvs/vllm025/bin/activate
export CUTE_DSL_ARCH=sm_121a MAX_JOBS=4                   # FP4 kernel JIT arch; cap nvcc fan-out

2. Serve (weights auto-download from this repo; quantization is auto-detected). This includes DFlash speculative decoding with poolside's matched BF16 drafter:

vllm serve ennerd/Laguna-S-2.1-NVFP4 \
  --trust-remote-code --kv-cache-dtype fp8 --moe-backend marlin \
  --reasoning-parser poolside_v1 --tool-call-parser poolside_v1 --enable-auto-tool-choice \
  --override-generation-config '{"temperature":0.7,"top_p":0.95}' \
  --speculative-config '{"model":"poolside/Laguna-S-2.1-DFlash-NVFP4","num_speculative_tokens":3}' \
  --max-num-seqs 32

→ ~44 tok/s single-stream. Drop the last two lines for a plain ~32 tok/s serve without speculation. (--max-num-seqs 32 is required with DFlash — it errors at the default of 256.)

B. Maximum performance — Docker with our bundled 4-bit drafter

One docker build + docker run gives you the fastest single-stream on a Spark, with no venv/FlashInfer setup and no manual patching — the image bakes in vLLM 0.25.1, the loader fix, and our 4-bit DFlash drafter (in dflash/).

1. Get the repo (the Dockerfile and the dflash/ drafter live here):

hf download ennerd/Laguna-S-2.1-NVFP4 --local-dir laguna && cd laguna

2. Build the image:

docker build -t ennerd-laguna-spark .

3. Run it (serves on :8000; mounts the weights you just downloaded):

docker run --gpus all --ipc host -p 8000:8000 -v "$PWD":/model ennerd-laguna-spark

→ ~47.7 tok/s single-stream. Then query it:

curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"Laguna-S-2.1","messages":[{"role":"user","content":"Write quicksort in Python."}]}'

Draft depth k=3 is the measured GB10 optimum (set NUM_SPEC to change it); see DOCKER.md for all knobs. Run this on one GB10 — on two Sparks, serve without speculation (see Performance).

Context: 256K and 1M

These are native 1M-trained weights, shipping defaulted to a 256K window. To restore 1M, apply the base model's published 1M RoPE settings (these constants come from poolside/Laguna-S-2.1-NVFP4's 1M configuration — they are not arbitrary) to config.json:

"rope_parameters": { "full_attention": { "factor": 128.0, "attention_factor": 1.4852030263919618 } },
"max_position_embeddings": 1048576

How it works

Relative to the stock poolside/Laguna-S-2.1-NVFP4, this build additionally quantizes the attention projections (q_proj, k_proj, v_proj, o_proj) to NVFP4, weight-only. The routed experts, lm_head, shared MLP, and norms are unchanged from the base checkpoint.

The stock NVFP4 checkpoint keeps attention, lm_head, and the shared MLP in BF16 — the right trade on B200-class bandwidth (8 TB/s). But local Blackwell decode is memory-bandwidth-bound: at batch 1 the weight read dominates, and on a DGX Spark (GB10, 273 GB/s unified memory) that BF16 attention is ~57% of per-token weight traffic. Quantizing it too cuts per-token weight traffic from **9.9 GB to ~5.7 GB** — and because decode time scales with bytes-read, that is the decode speedup. The win holds anywhere decode is bandwidth-bound: unified-memory GB10, and concurrent serving on any Blackwell (batching amortizes compute while the weight read stays fixed).

Limitations. This is a weight-only quant — it reduces memory traffic and footprint, so it accelerates bandwidth-bound decode but adds no activation-quantization benefit for compute-bound serving. On a high-bandwidth GPU at large batch (e.g. RTX Pro 6000 at 32-way concurrency), the workload is compute-bound and this build converges with stock NVFP4 on speed; there the gain is the ~4 GB smaller footprint, not throughput.

Compatibility & kernel notes

• Runs packed — the model weights stay 4-bit at load (~67 GB of packed weights — device-resident and on-disk, excluding KV cache and activations), not decompressed to BF16 — on both Blackwell targets: GB10 (Marlin NVFP4) and sm_120 discrete Blackwell (MarlinNvFp4 attention + FlashInfer-CUTLASS NVFP4 MoE).
• sm_120 + CUDA < 12.9: the FlashInfer-CUTLASS NVFP4 MoE kernel wants CUDA ≥ 12.9. On CUDA 12.8, add --moe-backend marlin (CUDA-version-tolerant, numerically identical).
• Serve via vLLM / SGLang / TensorRT-LLM, not plain Transformers — Laguna's fused 3-D expert parameters can't be kept packed by the Transformers quant path, which decompresses the experts to BF16 at load (~235 GB).

Reproduction

The attention projections (q_proj, k_proj, v_proj, o_proj) are quantized to NVFP4 weight-only and data-free: per-16-element block scales (FP8-E4M3) plus a per-tensor FP32 global scale, derived from the weights themselves (min/max) — no calibration dataset. The quantization uses compressed-tensors with the same quantization arguments as the base checkpoint's own routed experts. Before touching attention, we validated that our quantizer reproduces those base experts bit-exactly (0.000 relative error), so the method is provably identical to Poolside's. The routed experts, lm_head, shared MLP, and norms are byte-identical to the base checkpoint; only attention is added to the quantized set.

Source checkpoint: poolside/Laguna-S-2.1-NVFP4 @ commit 0761412.

Attribution & license

• Base model: poolside/Laguna-S-2.1 — © poolside. This is a quantized derivative of poolside/Laguna-S-2.1-NVFP4.
• License: OpenMDW-1.1, unchanged from the base model. This redistribution includes the OpenMDW-1.1 agreement and preserves the original copyright and origin notices.
• Responsible use: a software-engineering and agentic-coding model; confirm suitability for your application and use it consistently with Poolside's Acceptable Use Policy. Report security or safety concerns to security@poolside.ai.

---

Published by Ennerd. Not affiliated with or endorsed by poolside.