# gdubicki/Qwen3-Coder-Next-NVFP4-GB10 on DGX Spark (GB10)

Runs [`gdubicki/Qwen3-Coder-Next-NVFP4-GB10`](https://huggingface.co/gdubicki/Qwen3-Coder-Next-NVFP4-GB10) (quantized by [saricles](https://huggingface.co/saricles/Qwen3-Coder-Next-NVFP4-GB10)) via vLLM with an OpenAI-compatible API endpoint.
Tested on DGX Spark (GB10 Blackwell, SM12.1, 128 GB unified LPDDR5X).

## Model overview

| Property | Value |
|----------|-------|
| Architecture | `qwen3_next` (Hybrid DeltaNet linear attention + full attention + latent MoE) |
| Base model | `Qwen/Qwen3-Coder-Next` |
| Parameters | 80B total, **3B active** per token (512 experts, 10 active + 1 shared) |
| Layers | 48 (pattern: 3× DeltaNet linear → 1× full attention, repeating → 12 full attention) |
| Quantization | NVFP4 via `llmcompressor` + `compressed-tensors`; all 512 MoE experts calibrated |
| Kept in BF16 | `lm_head`, `embed_tokens`, `linear_attn` layers, `mlp.gate`, `mlp.shared_expert_gate` |
| KV cache | FP8 (only for the 12 full-attention layers) |
| Model size | ~45 GB (70% reduction from ~149 GB BF16) |
| Max context | 262,144 tokens (native; tested with FP8 KV cache by saricles) |
| Reasoning | Built-in chain-of-thought (`<think>` tags), ON by default |

## Model features

| Feature | Support |
|---------|---------|
| Tool calling | ✅ Yes (`--tool-call-parser qwen3_coder`) |
| Reasoning / thinking mode | ✅ Yes (ON by default, toggleable via `enable_thinking`) |
| Languages | multilingual (code-focused) |

## Performance

Measured on DGX Spark (GB10, SM12.1, 128 GB unified LPDDR5X):

| Metric | Value |
|--------|-------|
| Throughput | ~61 tok/s |
| Max context | 262,144 tokens |
| KV cache concurrency | **31.65×** at 262K tokens (DeltaNet has no KV cache) |

Comparison across models on GB10:

| Model | Active params | tok/s |
|-------|--------------|-------|
| Gemma-4-31B-IT-NVFP4 (dense) | 31B | ~7 |
| Qwen3-32B-NVFP4 (dense) | 32.8B | ~11 |
| Nemotron-3-Super-120B-A12B-NVFP4 | 12B | ~16 |
| **Qwen3-Coder-Next-NVFP4-GB10** | **3B** | **~61** |
| Nemotron-3-Nano-30B-A3B-NVFP4 | 3B | ~61 |

Qwen3-Coder-Next is 80B total but 3B active — same throughput as Nemotron-3-Nano with full 262K context.

## Quick start

```bash
# Required — model is gated on Hugging Face (accept license at gdubicki/Qwen3-Coder-Next-NVFP4-GB10 first):
export HF_TOKEN=hf_xxxx

bash start-qwen3-coder-next.sh
```

The script will:
1. Stop and remove any existing `qwen3-coder-next-vllm` container
2. Flush the system page cache (frees unified memory before vLLM starts)
3. Start the container in detached mode
4. Poll `http://localhost:8000/health` until the API is ready (~45 GB download on first run)

## Test the API

```bash
bash test-api.sh              # localhost
bash test-api.sh 192.168.x.x  # remote host
```

## Reasoning (chain-of-thought)

Reasoning is **ON by default**. Toggle per request:

```bash
# Reasoning OFF
curl -s -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gdubicki/Qwen3-Coder-Next-NVFP4-GB10","messages":[{"role":"user","content":"What is the capital of France?"}],"max_tokens":60,"chat_template_kwargs":{"enable_thinking":false}}'

# Reasoning ON (default)
curl -s -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gdubicki/Qwen3-Coder-Next-NVFP4-GB10","messages":[{"role":"user","content":"Write a binary search implementation in Python."}],"max_tokens":2000,"chat_template_kwargs":{"enable_thinking":true}}'
```

## Cline configuration

1. Open Cline settings (sidebar icon → gear icon)
2. Fill in the fields:

| Field | Value |
|-------|-------|
| Provider | OpenAI Compatible |
| Base URL | `http://<spark-ip>:8000/v1` |
| Model ID | `gdubicki/Qwen3-Coder-Next-NVFP4-GB10` |
| API Key | `dummy` (any non-empty string) |

## Files

| File | Purpose |
|------|---------|
| `start-qwen3-coder-next.sh` | Full launcher: stop, cache flush, docker run, health poll |
| `docker-run.sh` | Bare `docker run` command with comments, for reference |
| `test-api.sh` | curl smoke tests: health, model list, chat completion, reasoning, code generation |

## How it works

The `vllm/vllm-openai:cu130-nightly` image includes native `qwen3_next` support.

**Architecture**: Hybrid of:
- **DeltaNet** selective linear attention layers (36 of 48 layers) — subquadratic in sequence length, no KV cache
- **Full attention** layers (12 of 48) — standard transformer attention with KV cache
- **Latent MoE** (80B total, 3B active per token — same throughput profile as Nano)

Quantization is via `compressed-tensors` (llmcompressor) with `LLMCOMPRESSOR_MOE_CALIBRATE_ALL_EXPERTS=1`
— all 512 MoE experts are calibrated, not just sampled.
vLLM auto-detects quantization from `quantization_config` in `config.json`; no `--quantization` flag needed.

### Key environment variables

| Variable | Reason |
|----------|--------|
| `VLLM_NVFP4_GEMM_BACKEND=marlin` | SM12.1 (GB10) has no native CUTLASS FP4 kernel; Marlin is 15% faster for 512 experts |
| `VLLM_TEST_FORCE_FP8_MARLIN=1` | Forces FP8 Marlin path on GB10 SM12.1 |
| `VLLM_USE_FLASHINFER_MOE_FP4=0` | FlashInfer MoE FP4 path not supported on GB10 SM12.1 |
| `VLLM_MARLIN_USE_ATOMIC_ADD=1` | GB10-specific Marlin optimization for correct FP4 GEMM on SM12.1 |

### Key flags

| Flag | Reason |
|------|--------|
| `--dtype auto` | BF16 for non-quantized layers (DeltaNet, router gates, lm_head) |
| `--kv-cache-dtype fp8` | FP8 KV cache; applies to 12 full-attention layers only |
| `--gpu-memory-utilization 0.90` | 0.90 × 128 GB = 115 GB; covers ~43 GB weights + ~72 GB KV cache (0.93 is risky) |
| `--max-model-len 262144` | Full native context; tested by saricles with FP8 KV cache |
| `--max-num-seqs 64` | Max concurrent requests |
| `--max-num-batched-tokens 8192` | Prevents OOM on long contexts |
| `--attention-backend flashinfer` | Required for FP8 KV cache + chunked prefill on GB10 |
| `--enable-prefix-caching` | Reuses KV cache for repeated prompt prefixes (system prompts) |
| `--enable-chunked-prefill` | Reduces memory spikes during long-prompt processing |
| `--tool-call-parser qwen3_coder` | OpenAI-compatible tool calling |

## Requirements

- Docker with `nvidia-container-toolkit`
- Image: `vllm/vllm-openai:cu130-nightly`
- HF token with access to `gdubicki/Qwen3-Coder-Next-NVFP4-GB10` (gated — accept license first)
- Model weights cached locally (auto-downloaded on first run, ~45 GB):
  `~/.cache/huggingface/hub/models--saricles--Qwen3-Coder-Next-NVFP4-GB10/`