poolside

/

Laguna-S-2.1-NVFP4

like 178

Follow

Poolside 1.39k

I had high hopes distilling the laguna traces into a small Qwen Model for Text to SQL which did not go too well compared to the Qwen 3.6 Series..

• I did not run a pass7 eval on the 27B Base so laguna might win here.

OmniSQL Prompt

But on the other Side it works great with Hermes Agent for me on Coding and Data Science Tasks.
So far I prefer it over Qwen 3.6 27B fp8 because it is way faster.
Also it feels smarter, but this is only based on Vibes

I have not encountered the Looping Issue on longer Contexts, it gets straight to work.

Running on RTX 6000 Pro Max Q with VLLM Docker (vllm/vllm-openai:v0.25.1-cu129-ubuntu2404)

Sampling Config and Compose for Text to SQL

- temperature: 1.0
- top_p: 0.95
- top_k: 20
- max_tokens: 8192
- k: 8
- n: 1534 (full dev)

Compose

name: vllm
services:
  vllm:
    command:
      - --model
      - poolside/Laguna-S-2.1-NVFP4
      - --served-model-name
      - laguna-s-2.1
      - --max-model-len
      - "262144"
      - --gpu-memory-utilization
      - "0.90"
      - --max-num-seqs
      - "32"
      - --enable-chunked-prefill
      - --max-num-batched-tokens
      - "8192"
      - --enable-prefix-caching
      - --reasoning-parser
      - poolside_v1
      - --enable-auto-tool-choice
      - --tool-call-parser
      - poolside_v1
      - --override-generation-config
      - '{"temperature":0.7,"top_p":0.95}'
      - --compilation-config
      - '{"cudagraph_capture_sizes":[1,2,4,8,16,32]}'
      - --host
      - 0.0.0.0
      - --port
      - "8000"
    container_name: vllm-laguna
    environment:
      HF_HOME: /hf-cache
      HF_HUB_OFFLINE: "1"
      NVIDIA_DRIVER_CAPABILITIES: compute,utility
      NVIDIA_VISIBLE_DEVICES: all
      OMP_NUM_THREADS: "8"
      SAFETENSORS_FAST_GPU: "1"
      TRANSFORMERS_OFFLINE: "1"
      VLLM_NO_USAGE_STATS: "1"
      VLLM_USE_FLASHINFER_SAMPLER: "0"
    gpus:
      - count: -1
    healthcheck:
      test:
        - CMD
        - python3
        - -c
        - import urllib.request; urllib.request.urlopen('http://localhost:8000/health')
      timeout: 10s
      interval: 30s
      retries: 60
      start_period: 30m0s
    image: vllm/vllm-openai:v0.25.1-cu129-ubuntu2404
    ipc: host
    networks:
      default: null
    ports:
      - mode: ingress
        target: 8000
        published: "8000"
        protocol: tcp
    restart: unless-stopped
    ulimits:
      memlock: -1
      stack: 67108864
    volumes:
      - type: bind
        source: /home/ml/ml/hf-cache
        target: /hf-cache
        bind: {}
      - type: bind
        source: /home/ml/ml/vllm/cache-laguna
        target: /root/.cache
        bind: {}
networks:
  default:
    name: vllm_default