poolside

/

Laguna-S-2.1-NVFP4

like 174

Follow

Poolside 1.37k

Agree, i'm also 6000 pro rtx user, and i can't use the model with more than 80k context, also tps has been reduced.

Agree, i'm also 6000 pro rtx user, and i can't use the model with more than 80k context, also tps has been reduced.

Model weights are 99.7GB now, hare are you even fitting it in the RTX 6000 96GB VRAM?

Agree, i'm also 6000 pro rtx user, and i can't use the model with more than 80k context, also tps has been reduced.

Model weights are 99.7GB now, hare are you even fitting it in the RTX 6000 96GB VRAM?

I used their guff version that is 90GB

Agree, i'm also 6000 pro rtx user, and i can't use the model with more than 80k context, also tps has been reduced.

Model weights are 99.7GB now, hare are you even fitting it in the RTX 6000 96GB VRAM?

Try to use this also:

https://huggingface.co/anoane/Laguna-S-2.1-NVFP4-RTX6000PRO-vLLM

i'm trying this, it might be more helpfull

#dgxspark

Too large for Dgx Spark too, I guess it's almost all are in BF16, I tried 160k but got a OOM, I can get 120k in context I think but I didn't try, I know it gonna be too slow to try something...

Here you go

https://huggingface.co/kkuspa/Laguna-S-2.1-NVFP4-0804

Back down to 71.9GB and 125 tok/s prose | 205+ tok/s code, but now with 1M context and includes Poolside's looping fix.

vLLM serving details on card

If you're using anoane's serving guide, note its download step now pulls the 99.7 GB weights. Pointing it at this repo instead restores the fit.

there is a looping fix? i havent seen it

@darkmatter
It's layered across four commits rather than one, all on 2026-07-21/22:

ea64155234 + 8e7a98135f (07-21, ~22:05-22:12) — "Mark (token 24) as special." The direct stop-token fix: the model was emitting its end token without the tokenizer treating it as EOS, so generation ran past the end and looped.
b482b5d57f (07-21 22:27) — thinking on by default, max_new_tokens cap dropped. Behavior change tangled up in the runaway-generation reports.
07614121b3 (07-22 22:20) — "spinquantless norot weights." The substantive fix: new quantized weights with the SpinQuant rotation removed. The community threads (and the anoane guide's testing notes) indicate looping persisted for some users until this weight revision, meaning the rotated quantization itself was degrading generations, not just the tokenizer..

Thanks, @kkuspa . This commit timeline is useful, and I agree that all four commits are relevant. I think the point of disagreement is narrower: several separate defects have been described as “looping,” and I do not think the available evidence supports treating commit 07614121b3 as a complete fix for all of them.

What I agree was fixed

The first three commits clearly address real integration problems:

• ea64155234 marked </assistant> / token 24 as special in the model configuration.
• 8e7a98135f made the corresponding tokenizer change.
• b482b5d57f enabled thinking by default, preserved reasoning, and removed the default generation cap.

Those changes address response-boundary handling and the reasoning-template/parser configuration. I separately reproduced a structural issue where implicit thinking caused </think> to leak into normal content and malformed subsequent history. Explicitly setting enable_thinking=true eliminated that particular failure:

https://huggingface.co/poolside/Laguna-S-2.1-NVFP4/discussions/15

So I agree that there was a real tokenizer/template/parser problem, and that those commits substantially fixed that class of defect.

Where I think “the looping fix” claim is too broad

The fourth commit is:

07614121b31898586430f189d27a25a0be310843

That is the SpinQuant-free/no-rotation NVFP4 checkpoint you identified.

However, Poolside’s own description of that checkpoint was not that W4A4 looping had been completely fixed. Their statement was that it looped far less commonly in W4A4, did not loop in their W4A16 testing, but still looped in W4A4, so another checkpoint would be released as the official patch:

https://huggingface.co/poolside/Laguna-S-2.1-NVFP4/discussions/7

That distinction matters. 07614121… was clearly a substantive improvement and an attempted fix, but Poolside itself did not characterize it as a complete W4A4 fix.

The exact snapshot I have

To remove ambiguity, the local base checkpoint I have is pinned to exactly:

Repository:
poolside/Laguna-S-2.1-NVFP4

Commit:
07614121b31898586430f189d27a25a0be310843

Description:
nvfp4: spinquantless norot weights, 256K

I also have the updated companion DFlash snapshot:

Repository:
poolside/Laguna-S-2.1-DFlash-NVFP4

Commit:
4cdcc6e9b29105e8ff5790885cadccbeb4f33f54

Description:
Spinquant removal adaptation

I am still observing the non-convergent reasoning behavior on the RC1/no-rotation checkpoint family. More importantly, the core controlled reproduction does not depend on DFlash at all.

The behavior I am calling “looping”

I am not referring only to the model emitting text after an intended </assistant> boundary, nor to malformed multi-turn history.

I am referring to a second behavior:

1. Thinking activates correctly.
2. The reasoning parser correctly returns structured reasoning.
3. The model often identifies the likely answer early.
4. It repeatedly reconsiders, re-verifies, resets, or restarts the analysis.
5. It consumes the complete output budget.
6. It never transitions from reasoning to a final answer.

The controlled reproduction is documented here:

https://huggingface.co/poolside/Laguna-S-2.1-NVFP4/discussions/16

The primary test used a small, single-turn Python concurrency bug. The correct answer is simply that self.data[key] = value occurs outside the lock.

The results were:

Increasing the budget by 8× did not improve the completion rate. It only caused the reasoning trace to grow approximately in proportion to the available budget.

The test also reproduced with:

Hardware:        DGX Spark / GB10
vLLM:            0.25.1
Context:         262,144
KV cache:        FP8
DFlash:          disabled
Concurrency:     1
Temperature:     including 0.0
Agent harness:   none
Tools:           none
History:         none
Request:         direct /v1/chat/completions
Thinking:        explicitly enabled

That rules out speculative decoding, an agent framework, tool-call feedback, long conversation history, batching, and stochastic sampling as necessary causes.

An independent test against exact checkpoint 07614121b also found that without a repetition penalty, 0/8 requests completed; with repetition_penalty=1.15, 4/8 completed correctly:

https://huggingface.co/poolside/Laguna-S-2.1-NVFP4/discussions/10

That suggests repetition dynamics are involved, but a repetition penalty is a mitigation rather than evidence that the underlying checkpoint behavior is fully fixed.

Why the token-24 commits do not fully explain this reproduction

Marking token 24 as special can absolutely fix an end-token/tokenizer integration problem. However, the controlled reproduction above is not simply the model reaching its intended end and then generating past it.

In the failing runs, it never reaches a stable conclusion or emits a final answer. It remains inside correctly parsed structured reasoning until the server-enforced token limit terminates the request.

That appears to be a different failure mode:

Defect A:
special-token / template / reasoning-history handling
→ substantially fixed by the tokenizer and template commits

Defect B:
reasoning activates and parses correctly
→ reasoning trajectory fails to converge
→ full generation budget is exhausted
→ no final answer

The commits you cited provide a credible history for Defect A and a partial quantization improvement for Defect B. They do not, by themselves, demonstrate that Defect B was eliminated.

One important question about your 0804 repository

Your repository is described as:

71.9 GB
1M context
includes Poolside's looping fix

Could you clarify exactly which upstream checkpoint and files it was built from?

The current official Poolside main is now headed by:

f8fdfcdc4e7b0c474a0102430a8cae0a3a358669

That is the later “Promote RC2 → main” checkpoint, and it is substantially larger than the 71.9 GB 07614121… snapshot.

The distinction I am trying to understand is:

A. 07614121 weights
   + updated 1M configuration
   + resharding/repackaging

or

B. current f8fdfcd RC2 weights
   + a new 71.9 GB requantization

If it is A, then it appears to use the same no-rotation weights on which looping was still independently reproduced.

If it is B, then it is genuinely a different artifact and may include a later Poolside correction that I have not tested. In that case, I would be very interested in testing it using the exact controlled LRU regression rather than assuming that its behavior matches 07614121….

It would help to publish:

• the full upstream source commit;
• whether the tensors were requantized from the current RC2/main checkpoint;
• whether any layers excluded from upstream NVFP4 quantization were quantized again;
• the quantization recipe;
• whether the tokenizer, generation config, and chat template came from f8fdfcd;
• any before/after looping test results.

Proposed objective test

I am happy to run the same direct test against your 0804 artifact with:

20 independent requests
thinking enabled explicitly
DFlash disabled
temperature 0.7
top_p 0.95
default repetition penalty
max_tokens 8192
single-turn direct API request

A strong pass would be:

19/20 or better produce a final answer
19/20 or better identify the out-of-lock write
0/20 exhaust all 8192 tokens without an answer

Then I would repeat at temperature 0.0 and with DFlash enabled separately.

I am not claiming that every request loops or that the checkpoint is unusable. When Laguna works, I find it extremely capable, which is why I have spent time isolating this carefully. My narrower point is that the exact 07614121… revision you identified as the substantive fix still has a reproducible non-convergent reasoning mode, and Poolside’s own RC1 statement acknowledged that W4A4 looping remained.

If your 0804 build comes from the later f8fdfcd RC2 checkpoint and passes this regression, that would be genuinely useful evidence that the newer fix works.

@darkmatter2222

Fair correction on both fronts. "Looping fix" in my earlier comment was too broad: the commits I listed fixed the boundary/template class (your A) and improved B per poolside's own account in #7, which I should have cited.

To your question about 0804: it is your option B. Source is f8fdfcdc4e (RC2/main), all files. Layers 0-39 and every non-expert tensor are RC2's bytes verbatim, and note these are not the 07614121 quantization re-shipped: I byte-compared sampled quantized tensors between the two revisions and they differ (RC2 re-exported with fused gate/up global scales). Layers 40-47 experts are my requantization of RC2's BF16 tensors, which are byte-identical to the original poolside/Laguna-S-2.1 release. The procedure reproduces poolside's exporter byte-for-byte on tensors shipped in both forms; recipe in the model card. Activation scales for those layers come from 07614121, the only vendor calibration that exists for them. Tokenizer, generation config, chat template: f8fdfcdc4e unchanged.

I ran your LRU protocol against 0804 before replying: 20 independent single-turn requests, thinking explicitly enabled, DFlash disabled, temperature 0.7, top_p 0.95, default repetition penalty, max_tokens 8192, concurrency 1.

Result: 0/20 produced a final answer, 20/20 exhausted the budget with finish_reason=length. Reasoning traces ran 28K to 35K characters. The signature matches yours exactly: the model identifies the out-of-lock write within the first ~1,000 characters of reasoning ("the assignment to self.data is outside the lock... that's a race condition") and then re-litigates interleavings until the server cuts it off.

So you were right - your Defect B fully survives the RC2 re-export of layers 0-39, at least in full-W4A4 form. That narrows the live hypotheses to two: either the non-convergence is unfixed in W4A4 everywhere, or poolside's BF16 exemption of the last 8 layers in the 08-01 release is itself the mitigation, and my repo trades it away. My artifact can't discriminate between those.

I think your hardware can though: the Spark should fit poolside's f8fdfcdc main as shipped. If you run the same 20-trial protocol against official main and it converges, the BF16 tail is the fix and the size increase was doing real work; if it also fails, Defect B is open across the board and the 27.8 GB bought nothing. Either result would be the most useful datapoint in this thread.

I've added your reproduction as a caveat on the 0804 model card either way.