leoncca/Laguna-S-2.1-Selective-AWQ

Laguna S 2.1 Selective AWQ — W4A16 g32

This project was built to make the 118B-parameter Laguna S 2.1 MoE model usable on 4× Tesla V100 32GB with tensor parallel size 4 and a checkpoint below 100 GB. The size ceiling, W4A16 g32 asymmetric quantization, FP16 weight storage, and the protected routed-expert allocation—Layers 1–4 and 46–47—were all selected for that deployment target.

The main checkpoint is the general form of the same work: one unscaled weight artifact configured for BF16 activations, intended to run directly on a BF16-capable SM80+ system with stock vLLM. The V100 runtime add-on reuses those exact weights; there is no second V100 checkpoint.

This is an independently produced selective AWQ quantization of poolside/Laguna-S-2.1. It is not an official Poolside release.

Critical dtype requirement for the main checkpoint

Run the unscaled main checkpoint with BF16 activations. Do not force FP16 (float16/half).

The original model can produce very large late-layer hidden states at BOS/EOS anchor positions. In our fixed calibration replay, the largest observed BF16 hidden value was approximately 950,272, above the FP16 finite limit of 65,504. Each of the 1,097 token-id-2 BOS/EOS anchor rows contained at least one component outside the FP16 range, while the largest observed non-anchor value was 17,152. Normalized outputs and logits remained finite in BF16.

This is an activation dynamic-range property, not an AWQ packing failure. A runtime that silently executes the main checkpoint with FP16 activations can produce non-finite logits at EOS positions.

V100 runtime add-on — primary deployment target

The included v100-1cat/ directory is a small, fail-closed runtime patch for the configuration that motivated this quantization:

• 4× Tesla V100 PCIe 32GB;
• tensor parallel size 4;
• 1Cat-vLLM 1.2.2;
• FP16 activations and FP16 KV cache;
• TRITON_ATTN, chunked prefill, and prefix caching disabled.

It is deliberately version-pinned, not a cross-version plugin. It uses the exact main-model revision without copying or permanently rewriting the approximately 94 GB checkpoint.

Rc4 fixes two independent FP16 runtime problems. The pinned 1Cat/vLLM MoE runner's overflow-protection branch represents Laguna's combined routed/shared output at 1 / routed_scaling_factor and requires a matching decoder compensation. The pinned Laguna decoder omitted that multiply, so each MoE layer returned approximately 0.4x of the intended branch output for the configured factor 2.5. Rc4 restores the required FP16 x2.5 after every MoE layer. BF16 inference does not use this branch and is unaffected.

The second problem is the late-layer BOS/EOS activation overflow. For that range repair, the add-on applies a local power-of-two reparameterization to Layers 45–47, while the forward path multiplies hidden/residual by 1/64 once before Layer 45:

• attention o_proj: × 1/64;
• routed/shared expert up_proj: × 1/256;
• routed/shared expert down_proj: × 4.

The net branch factor is 1/64. Layer 45 remains W4; its qweight and qzeros are unchanged, while the relevant floating AWQ scale carriers are transformed. The add-on rejects an unknown model revision, quantization layout, or tensor count.

FLASH_ATTN_V100 is deliberately rejected by the add-on. On this model, the tested Flash-V100 path became non-finite after chunked prefill crossed its observed boundary. Triton is somewhat slower on short prompts, but it is the accepted V100 path from 512 tokens through long context.

The same-weight loader-time implementation has reproduced the materialized reference:

Test	Loader-time add-on result
Loader transform versus materialized reference	1,545 transformed and 2,337 untouched tail tensors bitwise identical
Complete EOS-containing calibration blocks	256/256 finite
Generation immediately after internal EOS	11/11 finite
Explicit multi-EOS histories	4/4 finite
Chinese/English chat, Python code, raw Chinese completion	Passed; no prior corruption/repetition or replacement characters
Mean coarsened KLD versus corrected unscaled FP16 arm	6.0291049932e-05 nats over 8,212 positions
Token top-1 agreement versus corrected unscaled FP16 arm	99.9391%
Actual-token logprob MAE versus corrected unscaled FP16 arm	0.00373973
Plain text, parsed reasoning, forced structured tool call	Passed
30-case math/code/agent smoke	29/30; all 30 predictions matched the saved pre-rc4 run
4,095- and 15,352-token needles	Passed
259,989-token needle	Passed; exact needle recovered; 1,384.602 s TTFT; 187.77 tok/s end-to-end prefill

The non-zero KLD and 99.9391% top-1 figures compare rc4's Layer-45 /64 repair with an unscaled FP16 runtime that includes the same required MoE x2.5 correction. The diagnostic PPL changed from 21.9363 to 21.9160, but 1Cat's Laguna prompt-logprob token attachment is shifted by one position, so that value is not presented as a language-model perplexity result.

The 30-case smoke used thinking=off with a 512-token cap. It is a small gross-regression check, not a public benchmark or an exact numerical gate.

A standalone global /32 weight-folding experiment was also tested and rejected: against the same unscaled baseline it produced 0.0661285 nats mean coarsened KLD and 91.0010% token top-1 agreement because FP16 rounding accumulated through all 48 layers. The V100 release therefore uses the local runtime add-on.

Model summary

Laguna S 2.1 is a text-to-text, 118B-total-parameter, approximately 8B-active-per-token Mixture-of-Experts model for agentic coding and long-horizon work. It is not a vision or audio model. The base model has 48 decoder layers, 256 routed experts plus one shared expert, and a configured context window of 1,048,576 tokens. See the official base model card for architecture, chat formatting, intended use, responsible-use guidance, and base-model benchmark results.

Frozen base revision:

poolside/Laguna-S-2.1@e80da38da3ed4c4e56888cc1ba39582946a164ba

Quantization layout

Layer numbers are zero-based Hugging Face names, model.layers.N.

Component	Stored format
Layer 0 dense MLP	FP16
Routed experts, Layers 1–4	FP16
Routed experts, Layers 5–45	AWQ W4A16, group size 32, asymmetric with zero-points
Routed experts, Layers 46–47	FP16
Attention, all layers	FP16
Routers and correction biases, all layers	FP16
Shared experts, all layers	FP16
Embeddings, normalization, and output head	FP16

AWQ configuration:

{
  "bits": 4,
  "group_size": 32,
  "quant_method": "awq",
  "version": "gemm",
  "zero_point": true
}

W4A16 denotes packed 4-bit weights with 16-bit activations. The main path uses BF16 activations; the V100 add-on uses FP16 activations with the local range repair above.

A single FP16-stored weight artifact serves both the BF16 main path and the V100 add-on. The precision effect of storing protected tensors in FP16 is already included in the KLD, perplexity, and top-1 measurements reported below.

Some protected FP16 tensors participate in mathematically equivalent AWQ scale compensation. They remain FP16 and were not packed to W4, but they need not be byte-identical to the corresponding base-model tensors.

quantization_config.modules_to_not_convert has been checked against this exact FP16/W4 layout and the machine-readable tensor audit. Stock vLLM 0.24.0 also loaded the mixed layout successfully; this is no longer inferred only from the JSON declaration.

Why this allocation?

The protected prefix and tail were selected from actual packed-model comparisons under the sub-100-GB V100 budget.

FP16 routed-expert layers	Directory bytes	Final hidden rel-L2	Mean full-vocab KLD	PPL	Top-1 agreement	30-task smoke
1–3 and 45–47	94,311,697,448	0.006003	0.241694	24.3297	86.56%	28/30
1–4 and 45–47	97,746,514,057	0.005274	0.233194	23.7390	86.94%	28/30
1–4 and 46–47	94,311,700,368	0.005288	0.234471	23.7372	86.67%	29/30
1–5 and 46–47	97,746,516,977	0.005844	0.245507	23.6633	86.77%	28/30
1–6 and 47	97,746,519,891	0.010638	0.249403	24.0867	86.48%	29/30
1–7	97,746,522,803	0.060652	0.246453	24.4271	86.35%	29/30

The selected layout protects four early and two final routed-expert layers while remaining below the 100,000,000,000-byte checkpoint limit. The 30-task column is a small deterministic regression smoke test, not a general benchmark.

Main-checkpoint usage

The supported main path uses upstream vLLM's native Laguna implementation and does not depend on copied model remote code. Transformers inference is not currently a validated path.

The command below is the intended BF16 serving form. The Poolside parser flags remain separately qualified below because the stock-vLLM probe exercised the native model and dtype path, not the HTTP parser stack:

vllm serve \
  --model leoncca/Laguna-S-2.1-Selective-AWQ \
  --tensor-parallel-size 4 \
  --dtype bfloat16 \
  --tool-call-parser poolside_v1 \
  --reasoning-parser poolside_v1 \
  --enable-auto-tool-choice \
  --served-model-name laguna \
  --default-chat-template-kwargs '{"enable_thinking": true}'

The parser and chat-template flags are copied from the base model's recommended vLLM setup. Reasoning and forced tool calls passed on the validated 1Cat V100 runtime; the same parser flags have not yet been re-run through the upstream-vLLM HTTP server.

The accepted stock-vLLM probe positively verified the loaded dtype rather than relying on the absence of a cast warning. On both TP workers, the 917 exposed parameter tensors consisted of 624 BF16, 47 FP32 router-bias, and 246 INT32 packed tensors, with no FP16 parameter. Hooks observed all 48 decoder layers with BF16 finite hidden/residual tensors and a finite BF16 final norm.

Upstream vLLM 0.24.0 contains a native Laguna model implementation and its AutoAWQ configuration accepts BF16 activations. The exact RC was loaded with trust_remote_code=False, dtype=bfloat16, and quantization=auto_awq.

Compatibility status

Environment	Status
Offline packed-model comparison against the frozen BF16 base	Validated
Materialized 4× V100 tail-repair reference	Validated as development evidence
Final same-weight V100 add-on rc4, 1Cat-vLLM 1.2.2	Validated with corrected FP16 MoE semantics through EOS, 259,989 tokens, reasoning, tool calls, and fixed-corpus drift
Exact main checkpoint, upstream vLLM 0.24.0, BF16	Native load and positive 48-layer dtype/EOS forward validated on heterogeneous RTX 3090 + RTX 5060 Ti TP2 with CPU offload; not a performance test
Transformers/AutoAWQ inference	Not evaluated
SGLang	Not evaluated
TensorRT-LLM	Not evaluated
Standalone RTX 3090	Not evaluated; the validated compatibility probe used RTX 3090 + RTX 5060 Ti together
DFlash or MTP	Not evaluated / not included

The library_name: transformers metadata describes the Hugging Face configuration/tokenizer format and base architecture; it is not a claim that Transformers inference has been validated.

Quantization and evaluation data

Calibration

Field	Value
Dataset	codeparrot/self-instruct-starcoder
Split	curated
Dataset revision	9598805f9276dc0a339e293e9f205ea8bbf41f9f
Shuffle seed	42
Shape	256 blocks × 512 tokens
Token tensor SHA-256	24f5f0f831a6abe747940602e990c81d52a6f39e96c15e23bf8396fc47150b6c

Calibration data was used for post-training quantization only; the base model was not retrained.

Held-out numerical comparison

The main numerical comparison used 16 disjoint 512-token held-out blocks and 8,176 scored next-token positions against the fixed raw-BF16 base revision. Decoder replay used BF16, final normalization accumulated in FP32, and the protected FP16 lm_head used the FP16 logit-compute path.

Metric	Selective-AWQ	BF16 reference
Final hidden relative-L2	0.0052877845	0 by definition
Mean full-vocabulary KLD	0.2344706438 nats	0 by definition
Perplexity	23.73715897	23.17865143
Top-1 agreement with reference	86.6683%	100% by definition
Deterministic regression smoke suite	29/30	Not reported as a public benchmark

Held-out token tensor SHA-256:

77976641882fa641e26f998594aa935518acb7796e9907e7182656b1361015ce

These measurements describe quantization drift on a fixed corpus. They do not claim to reproduce the base model's published software-engineering benchmark scores.

Artifact integrity

Frozen release checkpoint:

Field	Value
Safetensors shards	49
Indexed tensors	99,745
Tensor payload bytes	94,283,421,184
Checkpoint shard bytes	94,295,590,320 (87.820 GiB)
model.safetensors.index.json SHA-256	95d151973d8e56372a8f449237b665f08e29cbc249189c02898eaac663eccd62

Checkpoint shard bytes counts only the 49 safetensors files. The shards and model index remain frozen. The release process regenerates the repository-root SHA256SUMS for the corrected rc4 add-on. The public file set and its integrity metadata were verified against the frozen rc4 candidate before the v1.0 tag was created. Additional clean-download runtime checks are tracked separately and are not a prerequisite for reading or verifying this checkpoint.

Context length

The checkpoint inherits the base model's configured maximum context of 1,048,576 tokens, but that length has not been validated for this exact quantized release. Rc4 passed 4,095-, 15,352-, and 259,989-token needle tests and exposes 395,822 KV tokens at the validated 262,144-token server profile. The 259,989-token run recovered the exact needle with 1,384.602 s TTFT (187.77 prompt tok/s); peak sampled memory was 31,968 MiB used and 527 MiB free on each V100. One-million-token context remains not evaluated.

Intended use and limitations

This derivative is intended for local or hosted text generation and agentic coding workflows where the selective W4/FP16 trade-off is useful, particularly 4× V100 32GB TP4 deployments.

• The main checkpoint requires BF16 activations; forcing FP16 can overflow at EOS positions.
• The V100 path requires the version-pinned add-on and must not be assumed compatible with other runtimes.
• Only routed experts in Layers 5–45 are W4, so this is larger than a uniform all-expert W4 checkpoint by design.
• The stock-vLLM BF16 compatibility test used heterogeneous TP2 plus 36 GiB CPU offload per GPU at a 512-token profile. It validates native loading and dtype/range behavior, not production speed or long-context capacity on that machine.
• Transformers/AutoAWQ, SGLang, TensorRT-LLM, standalone RTX 3090, DFlash, MTP, and 1M context are not evaluated.
• Quantization can change generations even when aggregate numerical metrics remain close.

Follow the original model card's chat-template, thinking-preservation, intended-use, and responsible-use guidance.

License and attribution

The original model is poolside/Laguna-S-2.1, created by Poolside and released under OpenMDW-1.1. This repository retains the complete original LICENSE.md, copyright notices, and source notices required by that license.

Quantization and packaging: leon

Original model: poolside/Laguna-S-2.1

Original license: OpenMDW-1.1

The V100 runtime patch retains the applicable upstream copyright notices and Apache-2.0 license in v100-1cat/.

Release information

• Hugging Face publication target: leoncca/Laguna-S-2.1-Selective-AWQ
• Release tag: v1.0 (created from the remotely verified release revision)
• Publisher/quantizer: leon
• Main BF16 validation hardware/runtime: upstream vLLM 0.24.0; RTX 3090 24GB (SM86) + RTX 5060 Ti 16GB (SM120), TP2, 36 GiB CPU offload/GPU; PyTorch 2.11.0+cu130; Transformers 5.14.1; native LagunaForCausalLM; trust_remote_code=False
• V100 patched Laguna runtime SHA-256: 144ca4515847be808c12ca06a9d8c563a6be8a15d0c90953c16cfd30c5667fa4
• Integrity file: repository-root SHA256SUMS; the public file set and integrity metadata were checked against the frozen rc4 candidate before v1.0 was created. Additional clean-download runtime checks remain tracked separately.