poolside

/

Laguna-S-2.1-NVFP4-mlx

like 26

Follow

Poolside 1.3k

Hi there! We've updated the NVFP4 on poolside/Laguna-S-2.1-NVFP4. I'm planning on rolling out the new NVFP4 here too in the next few hours. In our testing it exhibits far less looping.

Please note that we have another variant planned in the next day or so that should resolve all such issues, so feel free to wait for that too.

Hi there! We've updated the NVFP4 on poolside/Laguna-S-2.1-NVFP4. I'm planning on rolling out the new NVFP4 here too in the next few hours. In our testing it exhibits far less looping.

Please note that we have another variant planned in the next day or so that should resolve all such issues, so feel free to wait for that too.

Thank you, looking forward to running it, will test when the new version drops and report back here.

Hi @abhiksngh , the first of the two updates has landed. I'll update again when we have the follow-up :)

Hi @abhiksngh , the first of the two updates has landed. I'll update again when we have the follow-up :)

Thanks, what exactly is in the first update? I am only seeing the generation_config.json file being updated and not the model files itself. Am i missing something?

Cuando le pregunto que modelo LLM eres?. El modelo responde :"¡Hola! Soy GPT-3 (Generative Pre-trained Transformer 3), un modelo de lenguaje de gran tamaño desarrollado por OpenAI. Mi tecnología se basa en la arquitectura transformers, que utiliza mecanismos de atención (attention) para procesar y generar texto de manera coherente y contextual.

Fui entrenado con una gran cantidad de texto en múltiples idiomas, incluido el español, lo que me permite comprender y generar respuestas en este idioma. Mi versión específica está optimizada para tareas como responder preguntas, ayudar con tareas creativas o técnicas, y mantener conversaciones naturales." es normal o algún problema?

Has anyone tested the performance of this model on a MacBook Pro with an M Max chip and 128 GB of RAM?

Has anyone tested the performance of this model on a MacBook Pro with an M Max chip and 128 GB of RAM?

After the fix (updated generation_config.json file) it starts to work decently. No loops, etc so far (but I had just one session). Before the fix it was awful, didn't work.
Speed on my M5 Max Mac 128GB is about 35tps, with ~100K flled context. It was higher with no context, obviously. I'm using it through omlx.

I personally use the Laguna-S-2.1-oQ4e-fast model (https://huggingface.co/mlx-community/Laguna-S-2.1-oQ4e-fast) with oMLX as well, and on a MacBook Pro M4 Max (16-core) with 128 GB of RAM, I achieved the following performance results:

oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Laguna-S-2.1-oQ4e-fast
Engine: Auto
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128          1971.8       16.41   519.8 tok/s    61.4 tok/s       4.079   282.7 tok/s    59.82 GB
pp4096/tg128          7443.4       17.03   550.4 tok/s    59.2 tok/s       9.629   438.8 tok/s    59.98 GB
pp8192/tg128         15983.5       17.72   512.6 tok/s    56.9 tok/s      18.255   455.8 tok/s    60.13 GB
pp16384/tg128        37715.0       19.43   434.4 tok/s    51.9 tok/s      40.205   410.7 tok/s    60.52 GB
pp32768/tg128        90655.8       29.04   361.5 tok/s    34.7 tok/s      94.377   348.6 tok/s    61.27 GB
pp65536/tg128       281753.7       32.86   232.6 tok/s    30.7 tok/s     285.948   229.6 tok/s    62.89 GB
pp131072/tg128      646516.6       43.93   202.7 tok/s    22.9 tok/s     652.134   201.2 tok/s    66.14 GB
pp200000/tg128     1158840.6       58.73   172.6 tok/s    17.2 tok/s    1166.342   171.6 tok/s    69.54 GB

Continuous Batching
pp1024 / tg128
--------------------------------------------------------------------------------
Batch           tg TPS   Speedup        pp TPS    pp TPS/req    TTFT(ms)      E2E(s)
1x          61.4 tok/s     1.00x   519.8 tok/s   519.8 tok/s      1971.8       4.079
2x         124.2 tok/s     2.02x    52.2 tok/s    26.1 tok/s     36595.2      41.311
4x         249.7 tok/s     4.07x   312.1 tok/s    78.0 tok/s      5979.8      15.174
8x         502.2 tok/s     8.18x   298.6 tok/s    37.3 tok/s     11846.3      29.475

its just going into loops for me.

some working hyperparam values with "decent" work would be highly appreciated.

Both https://huggingface.co/mlx-community/Laguna-S-2.1-oQ4e-fast and this nvfp4-mlx seems to work fine for me. I use omlx app, default parameters, and downloaded them via omlx UI; no tweaking. Requires latest versions of models (initial release was bad). I haven't used them too much though, so maybe there are still issues, but it's definitely way better than it was at release.

both quants seem to have issues with tool calls for me though, in pi agent (they keep using edit tool wrong; models like qwen 3.6 35B don't have this issue)

Requires latest versions of models (initial release was bad).

Poolside updated Laguna-S-2.1-NVFP4 with the spinquantless norot weights. The MLX repository has not received any new weight objects since its initial release.
Despite the latest MLX commit being titled “Rebuild MLX from norot NVFP4,” the artifact history shows:

• all 14 safetensors shards have identical LFS hashes across every revision;
• the config and tokenizer objects are also unchanged;
• commit e6a961c only added generation_config.json;
• no successor official S-2.1 MLX repository exists.

I just wanted to call that out in case other people read your comment and redownload an exact copy of what they already have thinking it's fixed.

@joerowell Please confirm if there are updated MLX weights coming still. Thank you for the model!