qwen3-coder-30b-a3b on Tenstorrent p300x2 and p150x4 and p300 and p150x2 and p150

Qwen3-Coder-30B-A3B-Instruct (30B total / 3B active MoE: 128 experts, top-8, 48 layers) for agentic coding with tool calling on Blackhole through vLLM, with a fused MoE decode: the v1 bundle (raahemnabeel/qwen3-coder-30b-a3b-v1) plus the Ornith-1.5-35B-A3B fused MoE decode kernels in expert-parallel form for 1-32 concurrent users. Same weights, profiles and serving recipe; prefill unchanged.

Runs on Tenstorrent p300x2 or p150x4 or p300 or p150x2 or p150 β€” see the serving profiles below.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Hardware p150, p150x2, p150x4, p300, p300x2
Context 256,000 tokens

Prerequisites

  • The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
  • Docker
  • Tenstorrent hardware: p300x2 or p150x4 or p300 or p150x2 or p150

Quickstart

tt serve raahemnabeel/qwen3-coder-30b-a3b

tt serve downloads the Docker image and the Qwen/Qwen3-Coder-30B-A3B-Instruct at b2cff646eb4bb1d68355c01b18ae02e7cf42d120 weights (into your HF cache; they are not in the image), then starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve raahemnabeel/qwen3-coder-30b-a3b.

What changed since v1

This is the v1 bundle (raahemnabeel/qwen3-coder-30b-a3b-v1) with its MoE decode replaced by the fused kernels of the Ornith-1.5-35B-A3B fusion (raahemnabeel/ornith-1.5-35b-a3b), in expert-parallel form. Same checkpoint, same vLLM plugin build, same profiles and serving recipe; prefill (the expert-sorted MoE) is unchanged. Per decode layer and die, three ops replace the replicated router chain, the two dynamic sparse matmuls, SwiGLU, the routing weights and the expert sum (and the expert-sorted path at 4+ rows):

  1. Route. The router projection (FP32 accumulation) and the global top-8 + softmax run in one op; each die keeps the selected experts that fall in its own window (die d holds experts 32 d .. 32 d + 31 on four dies, 64 per die on two, all 128 on one).
  2. Gate|up + SwiGLU and down + weighted sum stream each selected local expert's weights in place from the bundle's existing bfp4 DRAM layout (no copy, no re-rounding), for every decode width from 1 to 32 rows: several users' experts form one union per die, so each weight column is read once for all rows. A die whose window holds no selected expert writes a zero partial; the existing all-reduce sums the dies.

Numerics: the same bfp4 expert weights and FP32 router logits as v1. The fused route op accumulates the logits in a different order and its expert matmuls accumulate in FP32, so a near-tied 8th expert can be picked differently and the last bits of an expert's output can differ. Output quality is checked below.

Use with your client

Point any OpenAI client at the address tt-model serve prints (port 20000 unless it was busy), with model id Qwen/Qwen3-Coder-30B-A3B-Instruct. Tool calls use the qwen3_coder parser. The server's default generation config is greedy (temperature 0); per-request sampling parameters apply.

Profiles

tt-model serve raahemnabeel/qwen3-coder-30b-a3b --profile <name>:

  • p300 (default) / p150x2 - two dies (64 experts per die), 256000-token context.
  • p300x2 / p150x4 - four dies (32 experts per die), 256000-token context: the fastest profile.
  • p150 - one die (all 128 experts), 98304-token context (the bf16 KV cache is 96 KiB per token on one die).

All profiles: 32 concurrent sequences, prefix caching, bfp4 routed experts.

Serving profiles

Select a different serving profile using --profile.

profile hardware mesh max_num_seqs max_model_len description
p300x2 p300x2 P300x2 32 256000 Two p300 boards (4 dies, experts 32 per die), 32 sequences, 256000 context.
p150x4 p150x4 P150x4 32 256000 Four p150 boards (4 dies), same configuration as p300x2.
p300 (default) p300 P300 32 256000 One p300 board (2 dies, experts 64 per die, Linear collectives), 32 sequences, 256000 context.
p150x2 p150x2 P150x2 32 256000 Two p150 boards (2 dies), same configuration as p300. Not run on hardware (the identical (1,2) mesh).
p150 p150 P150 32 98304 One p150 (1 die, all 128 experts), 32 sequences sharing a 98304-token context (bf16 KV is 96 KiB per token on one die), no fabric.

Capabilities

The server speaks the OpenAI API at http://127.0.0.1:20000/v1 (or whichever port your serve command reported; chat completions, completions, and /v1/models). Pass "model": "Qwen/Qwen3-Coder-30B-A3B-Instruct" β€” the weights id, not this package's name.

Tool calling is enabled (--tool-call-parser qwen3_coder): a request that passes tools comes back with tool_calls and finish_reason: tool_calls.

Expected performance

Accuracy

Both bundles evaluated side by side on the same box, p300x2, through the served chat API with lm-evaluation-harness 0.4.9 at temperature 0 (full sets, 8 requests in flight, seed 42):

Eval v1 current
HumanEval pass@1 (164) 0.939 0.933
MBPP pass@1 (500) 0.802 0.786
GSM8K exact match, strict (1319) 0.919 0.914
IFEval prompt-level strict (541) 0.795 0.813
IFEval prompt-level loose (541) 0.839 0.861

Every difference is within one standard error of the difference (HumanEval 1 problem, MBPP 8, GSM8K 7; IFEval goes the other way). The harness's stock chat variants do not work through a chat API (humaneval_instruct and mbpp_instruct pre-fill the assistant turn, which the API does not continue), so HumanEval uses a chat prompt that asks for the whole function in one code block, and MBPP answers are scored from their first code block; gsm8k_cot_llama is the stock 8-shot task with the dataset's current id. The task files are in fusion/tools/lm_tasks on the source branch. Greedy outputs on every profile were checked against the v1 bundle's: they stay coherent and part after 17 to 1074 characters (the fused MoE decode rounds differently).

Performance

Measured 2026-10-24 on one QuietBox 2 (4 Blackhole p300c dies; p300 rows on two of them, p150 on one), both bundles served exactly as shipped (tt-model serve <bundle> --profile <p>), one server at a time: random-token prompts of exactly the input length, output pinned with ignore_eos, streamed; users are concurrent requests in a closed loop; TTFT = time to first streamed token; TPOT = (E2EL - TTFT) / (output - 1); one unrecorded warm-up per row. A real-text single-user row (12 chat prompts, 384 output tokens, temperature 0) is added. Each cell is v1 -> current.

Profile ISL OSL Concurrency N TTFT (ms) Prefill tok/s/u Decode tok/s/u E2EL (s)
p300x2 128 128 1 4 145 881 95.5 1.47
p300x2 1024 256 32 128 3,518 291 42.0 9.60
p300 128 128 1 4 141 908 86.0 1.62
p300 1024 256 32 128 4,301 238 30.4 12.76
p150 128 128 1 4 161 794 77.4 1.80

p300x2

Latency, batch 1 (single user): v1 -> current

Input tokens Output tokens TTFT (ms) Prefill tok/s TPOT (ms) Decode tok/s E2EL (s)
128 128 202 -> 145 634 -> 881 20.2 -> 10.5 49.5 -> 95.5 (+93 %) 2.77 -> 1.47
1,024 256 431 -> 376 2,377 -> 2,722 20.5 -> 10.8 48.7 -> 92.9 (+91 %) 5.67 -> 3.12
4,096 256 904 -> 848 4,530 -> 4,830 21.2 -> 11.4 47.2 -> 87.8 (+86 %) 6.31 -> 3.75
16,384 256 1,810 -> 1,745 9,053 -> 9,387 22.1 -> 12.3 45.3 -> 81.5 (+80 %) 7.44 -> 4.87
32,768 256 2,481 -> 2,418 13,205 -> 13,551 23.2 -> 13.4 43.1 -> 74.9 (+74 %) 8.40 -> 5.82
65,536 256 5,338 -> 5,261 12,277 -> 12,457 25.4 -> 15.6 39.4 -> 64.2 (+63 %) 11.81 -> 9.23
130,816 256 12,699 -> 12,637 10,301 -> 10,352 29.5 -> 19.6 33.9 -> 50.9 (+50 %) 20.21 -> 17.64

Real text, single user (12 chat prompts x 384 output tokens, temperature 0; median per prompt): 49.5 -> 95.5 tok/s (+93 %).

Concurrency, 128 input / 128 output tokens: v1 -> current

Users N TTFT (ms) TPOT (ms) Decode tok/s/user Total out tok/s E2EL (s)
2 8 201 -> 165 27.7 -> 12.4 36.1 -> 80.4 68.7 -> 146.9 (+114 %) 3.73 -> 1.74
4 16 398 -> 329 33.0 -> 12.7 30.3 -> 78.6 107.1 -> 249.1 (+133 %) 4.59 -> 1.96
8 32 614 -> 557 33.6 -> 14.0 29.8 -> 71.3 210.0 -> 444.5 (+112 %) 4.88 -> 2.31
16 64 1,055 -> 979 35.0 -> 16.2 28.6 -> 61.7 359.1 -> 646.1 (+80 %) 5.53 -> 3.07
32 128 1,919 -> 1,853 36.8 -> 24.0 27.2 -> 41.7 598.2 -> 812.7 (+36 %) 6.62 -> 4.91

Concurrency, 1024 input / 256 output tokens: v1 -> current

Users N TTFT (ms) TPOT (ms) Decode tok/s/user Total out tok/s E2EL (s)
2 8 338 -> 336 27.8 -> 12.4 36.0 -> 80.5 68.6 -> 145.8 (+112 %) 7.50 -> 3.54
4 16 532 -> 500 33.4 -> 13.0 29.9 -> 77.2 112.3 -> 266.3 (+137 %) 9.13 -> 3.86
8 32 962 -> 931 34.2 -> 14.3 29.2 -> 69.9 210.0 -> 444.8 (+112 %) 9.76 -> 4.62
16 64 1,834 -> 1,793 35.7 -> 16.7 28.0 -> 59.7 372.0 -> 674.4 (+81 %) 11.00 -> 6.09
32 128 3,546 -> 3,518 37.8 -> 23.8 26.5 -> 42.0 616.7 -> 853.4 (+38 %) 13.25 -> 9.60

p300

Latency, batch 1 (single user): v1 -> current

Input tokens Output tokens TTFT (ms) Prefill tok/s TPOT (ms) Decode tok/s E2EL (s)
128 128 188 -> 141 679 -> 908 26.3 -> 11.6 38.0 -> 86.0 (+126 %) 3.53 -> 1.62
1,024 256 432 -> 381 2,372 -> 2,690 26.7 -> 12.0 37.4 -> 83.2 (+122 %) 7.24 -> 3.45
4,096 256 915 -> 871 4,475 -> 4,704 27.7 -> 13.0 36.1 -> 76.7 (+113 %) 7.99 -> 4.19
16,384 256 2,115 -> 2,059 7,745 -> 7,956 29.2 -> 14.6 34.2 -> 68.6 (+101 %) 9.57 -> 5.78
32,768 256 3,125 -> 3,075 10,485 -> 10,656 31.3 -> 16.6 32.0 -> 60.1 (+88 %) 11.10 -> 7.32
65,536 256 7,800 -> 7,748 8,402 -> 8,458 35.2 -> 20.5 28.4 -> 48.7 (+71 %) 16.77 -> 12.98
130,816 256 20,673 -> 20,648 6,328 -> 6,336 42.7 -> 28.1 23.4 -> 35.6 (+52 %) 31.56 -> 27.80

Real text, single user (12 chat prompts x 384 output tokens, temperature 0; median per prompt): 38.0 -> 86.3 tok/s (+127 %).

Concurrency, 128 input / 128 output tokens: v1 -> current

Users N TTFT (ms) TPOT (ms) Decode tok/s/user Total out tok/s E2EL (s)
2 8 212 -> 185 39.0 -> 14.0 25.6 -> 71.6 49.6 -> 130.5 (+163 %) 5.16 -> 1.96
4 16 409 -> 351 41.4 -> 15.5 24.2 -> 64.5 88.7 -> 219.9 (+148 %) 5.67 -> 2.33
8 32 696 -> 634 43.6 -> 17.4 22.9 -> 57.4 163.9 -> 356.6 (+117 %) 6.22 -> 2.82
16 64 1,266 -> 1,203 47.5 -> 21.6 21.0 -> 46.3 276.5 -> 517.1 (+87 %) 7.38 -> 3.96
32 128 2,389 -> 2,332 51.1 -> 32.8 19.6 -> 30.5 455.6 -> 626.7 (+38 %) 8.88 -> 6.57

Concurrency, 1024 input / 256 output tokens: v1 -> current

Users N TTFT (ms) TPOT (ms) Decode tok/s/user Total out tok/s E2EL (s)
2 8 415 -> 410 39.2 -> 14.4 25.5 -> 69.6 49.2 -> 126.8 (+158 %) 10.46 -> 4.08
4 16 613 -> 572 41.5 -> 15.2 24.1 -> 65.9 91.2 -> 225.3 (+147 %) 11.23 -> 4.62
8 32 1,150 -> 1,118 43.5 -> 17.8 23.0 -> 56.3 167.2 -> 363.6 (+117 %) 12.27 -> 5.63
16 64 2,216 -> 2,182 47.3 -> 21.8 21.1 -> 45.9 287.2 -> 527.5 (+84 %) 14.28 -> 7.76
32 128 4,342 -> 4,301 49.8 -> 32.9 20.1 -> 30.4 479.1 -> 645.3 (+35 %) 17.15 -> 12.76

p150

Latency, batch 1 (single user): v1 -> current

Input tokens Output tokens TTFT (ms) Prefill tok/s TPOT (ms) Decode tok/s E2EL (s)
128 128 193 -> 161 663 -> 794 36.6 -> 12.9 27.3 -> 77.4 (+183 %) 4.84 -> 1.80
1,024 256 473 -> 440 2,166 -> 2,327 37.3 -> 13.5 26.8 -> 73.8 (+175 %) 9.98 -> 3.89

Real text, single user (12 chat prompts x 384 output tokens, temperature 0; median per prompt): 27.2 -> 76.9 tok/s (+183 %).

Limitations

  • p150x4 and p150x2 run the same meshes as p300x2 / p300 but were not run on p150 boards; the p150 profile ran on one p300c die (single-user rows only).
  • Every newly admitted request re-captures the decode traces (as in v1), which shows up as TTFT under concurrent load.
  • The evals ran on the p300x2 profile only. Agentic benchmarks (SWE-Bench Verified, LiveCodeBench) were not run.

Related packages

The v1 bundle this builds on: raahemnabeel/qwen3-coder-30b-a3b-v1. The fusion it ports: raahemnabeel/ornith-1.5-35b-a3b; engineering log models/autoports/ornith_ai_ornith_1_5_35b_a3b/fusion/HANDOFF.md on the c30_fusion branch of github.com/rnabeelTT/tt-metal.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/raahemnabeel/qwen3-coder-30b-a3b/discussions β€” that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.

Provenance

The exact sources the image was built from β€” code/ in this repo is byte-identical to the model code inside the image:

component built from
tt-metal f13b25823a8f3d5b88a8f329367e0558b8cdb68c
vLLM v0.26.0
vllm-tt-plugin a local checkout β€” commit not published
code/ digest a5b36e36e19de0f8 (sha256, first 16 hex digits)
built 2026-10-27T20:20:31+00:00 by tt-model 0.1.0
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for raahemnabeel/qwen3-coder-30b-a3b

Finetuned
(99)
this model