README.md

---
license: apache-2.0
tags:
- quantization
- sglang
- zml
- cuda
- inference
- eschamoe
- qwen3
- qwen3-moe
- mixture-of-experts
library_name: sglang
---

# Escha Runtime — `qwen3moe`

By **[Escha Labs Inc.](https://eschalabs.com/)**

Serving runtimes for **Escha** 2-/3-bit (`eschamoe`) quantized models of the
**`qwen3_5_moe` architecture** (Qwen3.5 / Qwen3.6 Mixture-of-Experts, 256 experts). One repo
per model architecture, one directory per engine — pick the engine that fits your workload:

| | **SGLang** — [`sglang/`](sglang/) | **ZML** — [`zml/`](zml/) |
|---|---|---|
| Best for | servers & teams | one user, one stream, zero-dependency deploys |
| Concurrency | **continuous batching**, paged KV, radix prefix cache | **one request at a time** |
| Single-user decode (4090)[^grid] | 218–231 tok/s (flat in output length) | **235–241 tok/s** on ≥500-token answers (**+8–14%**); ~190 tok/s on ≤128-token replies |
| TTFT, 128→1890-tok prompt[^grid] | **0.06–0.24 s** | 0.06–0.35 s |
| Max context (4090)[^ctx] | up to 159k tok @ `MEM=0.78`, 235k @ `0.90` (ships `CTXLEN=32768`) | **262,144** @ `ESCHA_MEM=0.93` (ships `ESCHA_CTX=1024`) |
| Multi-turn prompt reuse | radix cache (branching, cross-session) | append-only, single session |
| Tool calls / JSON schema / thinking parser | **yes** | no |
| Sampling (`temperature>0`) | full speed | ~104 tok/s (fast path is greedy-only) |
| Install | Python 3.12 venv + CUDA-12 PyTorch | **one binary**, no Python, no CUDA toolkit |

**Neither engine is uniformly faster.** ZML wins sustained decode on long single-stream
answers and is far simpler to deploy; SGLang wins short replies, wins time-to-first-token on
long prompts, and is the **only** option for concurrency, tool calling or structured output.
If unsure: multi-user, agents-at-scale, or structured output → **SGLang**; long-form
single-user generation on your own GPU → **ZML**.

The ZML engine is built on [ZML](https://github.com/zml/zml) (Zig + MLIR/XLA); the SGLang
engine on a fork of [SGLang](https://github.com/sgl-project/sglang). Both run the same
Escha CUDA kernels and serve the same model files.

[^grid]: Same-harness measurement, 2026-07-26: one RTX 4090, the same model files, the same
    streaming OpenAI client, greedy, batch 1, medians of 2 reps over an ISL×OSL grid
    (128–1890 in, 128–1890 out). Decode = steady-state tokens/s between the first and last
    streamed token. Per-cell decode: SGLang 176.8–230.9 (median 218.2), ZML 184.6–240.7
    (median 234.8); ZML leads every cell with ≥256 output tokens and trails on 128-token
    replies, where its 16-token fused decode chunk dominates. TTFT by input length
    (ZML / SGLang): 128 tok 55/55 ms · 500 tok 95/91 ms · 1000 tok 178/143 ms ·
    1890 tok 353/237 ms.

[^ctx]: Measured 2026-07-26 on one RTX 4090 (24 GB) with `Qwen3.6-35B-A3B-Escha-W2`. Both
    engines ship a conservative default context and let you raise it. SGLang reports its KV
    pool at startup; launched with `CTXLEN=262144` it allocated 159,480 tokens at its shipped
    `MEM=0.78` and 234,796 at `MEM=0.90` — that pool size is the ceiling for a single
    sequence. ZML serves a 261,966-token prompt at `ESCHA_CTX=262144`. Both are
    memory-bound near the cap, not architecture-bound: KV on this hybrid model is only
    ~20 KiB per token (10 attention layers; the 30 gated-delta-net layers hold a fixed
    ~66 MB recurrent state), so 262,144 tokens is 5.37 GB beside 12.3 GB of weights.
    **In fairness to SGLang:** its 234,796-token pool at `MEM=0.90` still left 1.38 GB
    of VRAM free, so it would very likely also reach the 262,144 cap at a higher
    `mem-fraction-static` — we did not test that, and 0.78 is simply the conservative
    default the runtime ships. Read this row as "both engines reach the model's native
    context on a 24 GB card", not as a ZML advantage.

### Compatible models

| Model repo | Bits |
|---|---|
| [EschaLabs/Qwen3.6-35B-A3B-Escha-W2](https://huggingface.co/EschaLabs/Qwen3.6-35B-A3B-Escha-W2) | 2-bit (`eschamoe`) |
<!-- add each qwen3_5_moe Escha model you publish here (e.g. Ornith-1.0-35B, W3 variants) -->

> These runtimes target `qwen3_5_moe`. A model of a **different architecture will not load** —
> use the matching `escha-runtime-<arch>` repo.

## Quickstart

**SGLang engine** (full detail: [`sglang/INSTALL.md`](sglang/INSTALL.md), incl. the per-GPU cookbook):

```bash
python3.12 -m venv .venv && source .venv/bin/activate
pip install -U pip wheel
pip install "torch==2.9.*" --index-url https://download.pytorch.org/whl/cu128   # cu12 torch first
pip install ./sglang/escha-*.whl   # pulls the bundled sglang fork + its full dep closure

hf download EschaLabs/Qwen3.6-35B-A3B-Escha-W2 --local-dir ./Qwen3.6-35B-A3B-Escha-W2
MODEL=./Qwen3.6-35B-A3B-Escha-W2 bash sglang/serve.sh
```

**ZML engine** (full detail: [`zml/INSTALL.md`](zml/INSTALL.md)) — no Python at all:

```bash
tar xf zml/escha-zml-serve-*-linux-x86_64.tar.gz && cd escha-zml-serve-*
hf download EschaLabs/Qwen3.6-35B-A3B-Escha-W2 --local-dir ./model
./escha-serve
```

## Long context and agentic use (ZML engine)

`ESCHA_CTX` reaches the model's native **262,144** tokens. Measured on one 4090 with facts
planted at 10%, 50% and 90% depth and all three asked for at the end — the check that
positions and recurrent state thread correctly across the whole prompt:

| prompt | prefill | rate | recall |
|---:|---:|---:|:--|
| 15,066 tok | 5.3 s | 2,858 tok/s | 3/3 |
| 60,066 tok | 24.9 s | 2,411 tok/s | 3/3 |
| 129,966 tok | 79.0 s | 1,646 tok/s | 3/3 |
| 261,966 tok | 259.1 s | 1,011 tok/s | 3/3 |

Prefill is ~N²/2 work, so the rate falls with length. **The comfortable band on a 24 GB card
is ~8k–130k**, where TTFT is seconds; 262k answers correctly but takes ~4.3 minutes to read.

**Multi-turn:** a conversation whose prompt grows each turn reuses the previous prefix. On a
14k-token conversation the first turn takes 4.11 s and each following turn **1.27 s**
(~99.8% of the prompt reused). Reuse requires an exact append — editing earlier history
re-reads the prompt — and only one conversation is cached. SGLang's radix cache is faster in
absolute terms here and handles branching and multiple sessions; the ZML cache exists to make
single-session agent loops practical, not to match it.

**Decode slows as position grows** on both engines, because decode attention is O(position).
On ZML, measured with prefill excluded: **227 tok/s** at position ~1k, **163** at 3k, **124**
at 7.5k. (That measurement divides total request time and so reads ~5% below the streamed
grid above — method, not configuration.)

## Requirements (shared)

- **NVIDIA GPU, compute capability 8.0–12.0** (Ampere → Blackwell), Linux x86-64.
  Kernel launch route auto-selects per GPU. Per-architecture and per-VRAM launch recipes:
  [`sglang/INSTALL.md` → Running on your GPU](sglang/INSTALL.md).
- SGLang engine: **Python 3.12** + CUDA-12 PyTorch (the wheel handles every other dependency).
  24 GB VRAM recommended (16 GB works with reduced context).
- ZML engine: **no runtime dependencies** — the archive bundles the CUDA runtime; just an
  NVIDIA driver (R550+) and **glibc 2.27+** (RHEL/Rocky/Alma 8 and newer). 24 GB VRAM;
  one request at a time.

## Known limitations

- **`DETERMINISTIC=1` fails on consumer Blackwell (sm_120).** The deterministic attention kernel
  requests 104 KB of shared memory per block, above the sm_120 limit, and the server exits during
  startup. It works on Ampere, Ada and Hopper. Greedy output is in any case **not bit-reproducible
  across requests** on either engine — batch composition changes fp16 accumulation order, so a
  near-tie can flip and a long reasoning chain diverges from there.
- **The ZML engine serves no `/v1/completions`** (HTTP 404). Use `/v1/chat/completions`.
- **The ZML engine does not validate requests.** An unknown `model` field is served anyway instead
  of returning `model_not_found`, and `usage.prompt_tokens` is wrong when a prompt is truncated.
- **The ZML engine can wedge on an over-long prompt on a 16 GB card**, requiring `kill -9` rather
  than returning an error. This is why it asks for 24 GB — use the SGLang engine on 16 GB, where
  the same prompt returns a clean `HTTP 400`.

## License

Everything here is released under the **Apache License, Version 2.0** — see [`LICENSE`](LICENSE).
All bundled third-party code is permissive (Apache-2.0 / MIT / BSD-3-Clause; NVIDIA runtime
libraries in the ZML bundle under the NVIDIA EULA's redistributable-runtime terms) — **no
copyleft**. Full texts and the component inventory:
[`THIRD_PARTY_LICENSES/`](THIRD_PARTY_LICENSES/). Model weights are **not** in this repo and carry
their own license in the model repository.