--- license: apache-2.0 tags: - quantization - sglang - zml - cuda - inference - eschamoe - qwen3 - qwen3-moe - mixture-of-experts library_name: sglang --- # Escha Runtime — `qwen3moe` By **[Escha Labs Inc.](https://eschalabs.com/)** Serving runtimes for **Escha** 2-/3-bit (`eschamoe`) quantized models of the **`qwen3_5_moe` architecture** (Qwen3.5 / Qwen3.6 Mixture-of-Experts, 256 experts). One repo per model architecture, one directory per engine — pick the engine that fits your workload: | | **SGLang** — [`sglang/`](sglang/) | **ZML** — [`zml/`](zml/) | |---|---|---| | Best for | servers & teams | one user, one stream, zero-dependency deploys | | Concurrency | **continuous batching**, paged KV, radix prefix cache | **one request at a time** | | Single-user decode (4090)[^grid] | 218–231 tok/s (flat in output length) | **235–241 tok/s** on ≥500-token answers (**+8–14%**); ~190 tok/s on ≤128-token replies | | TTFT, 128→1890-tok prompt[^grid] | **0.06–0.24 s** | 0.06–0.35 s | | Max context (4090)[^ctx] | up to 159k tok @ `MEM=0.78`, 235k @ `0.90` (ships `CTXLEN=32768`) | **262,144** @ `ESCHA_MEM=0.93` (ships `ESCHA_CTX=1024`) | | Multi-turn prompt reuse | radix cache (branching, cross-session) | append-only, single session | | Tool calls / JSON schema / thinking parser | **yes** | no | | Sampling (`temperature>0`) | full speed | ~104 tok/s (fast path is greedy-only) | | Install | Python 3.12 venv + CUDA-12 PyTorch | **one binary**, no Python, no CUDA toolkit | **Neither engine is uniformly faster.** ZML wins sustained decode on long single-stream answers and is far simpler to deploy; SGLang wins short replies, wins time-to-first-token on long prompts, and is the **only** option for concurrency, tool calling or structured output. If unsure: multi-user, agents-at-scale, or structured output → **SGLang**; long-form single-user generation on your own GPU → **ZML**. The ZML engine is built on [ZML](https://github.com/zml/zml) (Zig + MLIR/XLA); the SGLang engine on a fork of [SGLang](https://github.com/sgl-project/sglang). Both run the same Escha CUDA kernels and serve the same model files. [^grid]: Same-harness measurement, 2026-07-26: one RTX 4090, the same model files, the same streaming OpenAI client, greedy, batch 1, medians of 2 reps over an ISL×OSL grid (128–1890 in, 128–1890 out). Decode = steady-state tokens/s between the first and last streamed token. Per-cell decode: SGLang 176.8–230.9 (median 218.2), ZML 184.6–240.7 (median 234.8); ZML leads every cell with ≥256 output tokens and trails on 128-token replies, where its 16-token fused decode chunk dominates. TTFT by input length (ZML / SGLang): 128 tok 55/55 ms · 500 tok 95/91 ms · 1000 tok 178/143 ms · 1890 tok 353/237 ms. [^ctx]: Measured 2026-07-26 on one RTX 4090 (24 GB) with `Qwen3.6-35B-A3B-Escha-W2`. Both engines ship a conservative default context and let you raise it. SGLang reports its KV pool at startup; launched with `CTXLEN=262144` it allocated 159,480 tokens at its shipped `MEM=0.78` and 234,796 at `MEM=0.90` — that pool size is the ceiling for a single sequence. ZML serves a 261,966-token prompt at `ESCHA_CTX=262144`. Both are memory-bound near the cap, not architecture-bound: KV on this hybrid model is only ~20 KiB per token (10 attention layers; the 30 gated-delta-net layers hold a fixed ~66 MB recurrent state), so 262,144 tokens is 5.37 GB beside 12.3 GB of weights. **In fairness to SGLang:** its 234,796-token pool at `MEM=0.90` still left 1.38 GB of VRAM free, so it would very likely also reach the 262,144 cap at a higher `mem-fraction-static` — we did not test that, and 0.78 is simply the conservative default the runtime ships. Read this row as "both engines reach the model's native context on a 24 GB card", not as a ZML advantage. ### Compatible models | Model repo | Bits | |---|---| | [EschaLabs/Qwen3.6-35B-A3B-Escha-W2](https://huggingface.co/EschaLabs/Qwen3.6-35B-A3B-Escha-W2) | 2-bit (`eschamoe`) | > These runtimes target `qwen3_5_moe`. A model of a **different architecture will not load** — > use the matching `escha-runtime-` repo. ## Quickstart **SGLang engine** (full detail: [`sglang/INSTALL.md`](sglang/INSTALL.md), incl. the per-GPU cookbook): ```bash python3.12 -m venv .venv && source .venv/bin/activate pip install -U pip wheel pip install "torch==2.9.*" --index-url https://download.pytorch.org/whl/cu128 # cu12 torch first pip install ./sglang/escha-*.whl # pulls the bundled sglang fork + its full dep closure hf download EschaLabs/Qwen3.6-35B-A3B-Escha-W2 --local-dir ./Qwen3.6-35B-A3B-Escha-W2 MODEL=./Qwen3.6-35B-A3B-Escha-W2 bash sglang/serve.sh ``` **ZML engine** (full detail: [`zml/INSTALL.md`](zml/INSTALL.md)) — no Python at all: ```bash tar xf zml/escha-zml-serve-*-linux-x86_64.tar.gz && cd escha-zml-serve-* hf download EschaLabs/Qwen3.6-35B-A3B-Escha-W2 --local-dir ./model ./escha-serve ``` ## Long context and agentic use (ZML engine) `ESCHA_CTX` reaches the model's native **262,144** tokens. Measured on one 4090 with facts planted at 10%, 50% and 90% depth and all three asked for at the end — the check that positions and recurrent state thread correctly across the whole prompt: | prompt | prefill | rate | recall | |---:|---:|---:|:--| | 15,066 tok | 5.3 s | 2,858 tok/s | 3/3 | | 60,066 tok | 24.9 s | 2,411 tok/s | 3/3 | | 129,966 tok | 79.0 s | 1,646 tok/s | 3/3 | | 261,966 tok | 259.1 s | 1,011 tok/s | 3/3 | Prefill is ~N²/2 work, so the rate falls with length. **The comfortable band on a 24 GB card is ~8k–130k**, where TTFT is seconds; 262k answers correctly but takes ~4.3 minutes to read. **Multi-turn:** a conversation whose prompt grows each turn reuses the previous prefix. On a 14k-token conversation the first turn takes 4.11 s and each following turn **1.27 s** (~99.8% of the prompt reused). Reuse requires an exact append — editing earlier history re-reads the prompt — and only one conversation is cached. SGLang's radix cache is faster in absolute terms here and handles branching and multiple sessions; the ZML cache exists to make single-session agent loops practical, not to match it. **Decode slows as position grows** on both engines, because decode attention is O(position). On ZML, measured with prefill excluded: **227 tok/s** at position ~1k, **163** at 3k, **124** at 7.5k. (That measurement divides total request time and so reads ~5% below the streamed grid above — method, not configuration.) ## Requirements (shared) - **NVIDIA GPU, compute capability 8.0–12.0** (Ampere → Blackwell), Linux x86-64. Kernel launch route auto-selects per GPU. Per-architecture and per-VRAM launch recipes: [`sglang/INSTALL.md` → Running on your GPU](sglang/INSTALL.md). - SGLang engine: **Python 3.12** + CUDA-12 PyTorch (the wheel handles every other dependency). 24 GB VRAM recommended (16 GB works with reduced context). - ZML engine: **no runtime dependencies** — the archive bundles the CUDA runtime; just an NVIDIA driver (R550+) and **glibc 2.27+** (RHEL/Rocky/Alma 8 and newer). 24 GB VRAM; one request at a time. ## Known limitations - **`DETERMINISTIC=1` fails on consumer Blackwell (sm_120).** The deterministic attention kernel requests 104 KB of shared memory per block, above the sm_120 limit, and the server exits during startup. It works on Ampere, Ada and Hopper. Greedy output is in any case **not bit-reproducible across requests** on either engine — batch composition changes fp16 accumulation order, so a near-tie can flip and a long reasoning chain diverges from there. - **The ZML engine serves no `/v1/completions`** (HTTP 404). Use `/v1/chat/completions`. - **The ZML engine does not validate requests.** An unknown `model` field is served anyway instead of returning `model_not_found`, and `usage.prompt_tokens` is wrong when a prompt is truncated. - **The ZML engine can wedge on an over-long prompt on a 16 GB card**, requiring `kill -9` rather than returning an error. This is why it asks for 24 GB — use the SGLang engine on 16 GB, where the same prompt returns a clean `HTTP 400`. ## License Everything here is released under the **Apache License, Version 2.0** — see [`LICENSE`](LICENSE). All bundled third-party code is permissive (Apache-2.0 / MIT / BSD-3-Clause; NVIDIA runtime libraries in the ZML bundle under the NVIDIA EULA's redistributable-runtime terms) — **no copyleft**. Full texts and the component inventory: [`THIRD_PARTY_LICENSES/`](THIRD_PARTY_LICENSES/). Model weights are **not** in this repo and carry their own license in the model repository.