mando2222/vibethinker-3b-P150

VibeThinker-3B — Tenstorrent Blackhole (vLLM bundle)

A tt-kernel vLLM bundle that runs WeiboAI/VibeThinker-3B — a Qwen2.5-3B-family dense reasoning model (Qwen2ForCausalLM) — on a single Tenstorrent Blackhole p150a through the Tenstorrent vLLM plugin. It ships no kernel cache and no weights — only the vLLM registration + a launch recipe; weights are pulled from the base repo and kernels JIT-compile at first-run warmup. This is the generic tt_transformers path: the registration references the upstream built-in QwenForCausalLM (no custom adapter is shipped).

Serve

tt-kernel serve mando2222/vibethinker-3b-P150

Exposes an OpenAI-compatible server on :8000. Model id WeiboAI/VibeThinker-3B (aliases vibethinker-3b, vibethinker-3b-P150). Native 131072 context; up to 32 concurrent requests (max_num_seqs=32). The model emits long <think>…</think> chain-of-thought — use generous max_tokens and its recommended sampling (temperature 0.6 / top_p 0.95 / top_k 40).

Full bringup — bare machine → tt-kernel serve

The tooling is the tt-kernel CLI from tenstorrent/tt-kernel-package-manager. tt-kernel serve needs a built tt-metal (which provides ttnn + the python_env) plus the Tenstorrent vLLM fork + plugin; the package manager's scripts/install.sh sets those up.

# 0. Prerequisite: a built tt-metal — provides ttnn + the python_env (the runtime).
#    https://github.com/tenstorrent/tt-metal

# 1. Get the serving toolchain: install the vLLM fork + TT plugin + tt-kernel into the tt-metal venv.
git clone https://github.com/tenstorrent/tt-kernel-package-manager
cd tt-kernel-package-manager
scripts/install.sh                    # installs vLLM fork + plugin + tt-kernel into the tt-metal python_env
#   (already have the vLLM fork + plugin?  just:  pip install tt-kernel)

# 2. Verify the toolchain + hardware are adequate (checks tt-metal/vLLM/device; does NOT flash firmware).
tt-kernel doctor

# 3. Serve THIS bundle: pulls the bundle folder + the WeiboAI/VibeThinker-3B weights, points
#    EXTRA_MODELS_DIR at it, and launches the OpenAI-compatible server on :8000 with the bundle's
#    per-machine launch recipe (bf8, traced decode, host sampling, max_num_seqs=32, max_model_len=131072).
tt-kernel serve mando2222/vibethinker-3b-P150
#   --print        show the exact launch command instead of running it
#   --local-only   after the first pull, skip the network round-trip
#   --bundles-dir  override where EXTRA_MODELS_DIR is materialized

# 4. Hit it (model id or an alias):
curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "vibethinker-3b",
  "messages": [{"role":"user","content":"Prove that sqrt(2) is irrational."}],
  "temperature": 0.6, "top_p": 0.95, "top_k": 40, "max_tokens": 4096
}'

First serve JIT-compiles kernels at warmup (a one-time delay); subsequent runs reuse the cache and come up in well under a minute. See Consumer prerequisites below for the hardware/toolchain this resolves to, and Known limitations for the single-chip caveats.

Exact stack this was built and validated on

Component	Version
tt-metal	dispatch-dev (v0.72.0 base) — generic models/tt_transformers Qwen2 path (built-in QwenForCausalLM); no autoport branch required
tt-vllm (fork) + TT plugin	vllm-tt-plugin (maps Qwen2ForCausalLM → TTQwen2ForCausalLM → tt_transformers QwenForCausalLM)
tt-kernel	vLLM-backend / kernels-less bundle
vLLM	0.1.dev2+g46120d16a (TT provided at runtime)
Hardware	1× Blackhole p150a, opened as a 1×1 mesh (MESH_DEVICE=N150), firmware 19.12.0
Driver	TT-KMD 2.8.0

Serving throughput was measured on firmware 19.6.0; the card was later flashed to 19.12.0, on which the stack initializes and serves correctly. The precompiled-kernel compatibility key is omitted (kernels-less bundle) — portable across tt-metal builds; kernels recompile at warmup.

Model / precision

Dense Qwen2ForCausalLM, 36 layers, GQA 16 q / 2 kv, head_dim 128, SwiGLU MLP (intermediate 11008), vocab 151936, tied embeddings, rope_theta=1e6, full (non-sliding) attention, native 131072 context. Precision: BFP8 weights + BFP8 paged KV cache, BF16 activations; host sampling (see limitations); metal-trace decode + eager prefill. bf8 was verified to match bf16 to the token on this model (datatype gate).

Measured (single Blackhole p150a)

• Decode: 26.1 tok/s/user single-user; TTFT 43.5 ms (128-token prompt).
• Aggregate throughput: ~736 tok/s peak at max_num_seqs=32; the 100/100/32 concurrent burst sustains ~292 tok/s, 32/32 completed. max_num_seqs=32 is both the throughput maximum and the hard ceiling (tt-metal decode batch ∈ {1,2,4,8,16,32}).
• Concurrency ↔ context: the KV pool is a fixed ~131,072-token budget (independent of max_num_seqs and max_model_len), so native context is free. Concurrent agents ≈ min(32, ⌊131072 / tokens_per_agent⌋): 32 @ ≤4K ctx, 16 @ 8K, 8 @ 16K, 4 @ 32K, 1 @ full 131072.
• Accuracy: AIME24 (chat-template, teacher-forced vs HF) top-1 0.95–0.98, top-5 1.00; bf8 == bf16 to the token. Native-context recall verified (a 32,363-token prompt served correctly); greedy decode bit-exact deterministic.

Consumer prerequisites

This bundle carries the vLLM registration + launch recipe + weights reference, not the model source. To run it you also need:

1. 1× Blackhole p150a, opened as a 1×1 mesh (MESH_DEVICE=N150; Blackhole detected at runtime).
2. tt-metal built with the generic models.tt_transformers.tt.generator_vllm:QwenForCausalLM importable (the standard Qwen2.5 text path — validated on the dispatch-dev v0.72.0 build; a compatible upstream tt-metal with the tt_transformers Qwen2 path is expected to work).
3. Tenstorrent vLLM fork + TT plugin (vllm-tt-plugin) — install via the package manager's scripts/install.sh.
4. Weights: WeiboAI/VibeThinker-3B (~6 GB, open — no gating).
5. Note: if your tt-metal venv ships transformers 5.x, the Qwen2 config/tokenizer path needs three small compat shims (AutoModelForVision2Seq rename, rope_theta under rope_parameters, apply_chat_template returning a BatchEncoding) — see the bundle's vllm_bundle/README.md.

Known limitations

• Max 32 concurrent decode requests (tt-metal batch cap {1,2,4,8,16,32}); a 33rd request queues.
• Host sampling on a single chip. The on-device sampler needs per-device vocab ≤ 64K, but vocab/2 = 75968 > 64K on a 1×1 mesh — so sampling runs host-side (correct; modest overhead).
• Reasoning needs a large output budget. VibeThinker emits long CoT; short max_tokens truncate mid-<think>. Use its recommended sampling (temp 0.6 / top_p 0.95 / top_k 40) for final-answer tasks.
• prompt_logprobs is not supported by the TT vLLM plugin.
• Profiler (tt-perf-report) device-times require the board firmware and tt-metal/UMD versions to be aligned; wall-clock / vLLM client throughput are the authoritative latency metrics here.

Changelog

2026-07-21 — initial bringup (single p150a, generic tt_transformers path)

• VibeThinker-3B (Qwen2.5-3B arch) served on one Blackhole p150a via the built-in Qwen2 path, bf8 weights + bf8 KV (bf8 == bf16 to the token), traced decode, host sampling.
• Native 131072 context; max_num_seqs=32 (the batch ceiling and throughput max); ~736 tok/s peak aggregate, 26.1 tok/s/user, TTFT 43.5 ms. Firmware 19.12.0.

License

Weights: WeiboAI/VibeThinker-3B is governed by the terms in its base repository. This bundle contains only Tenstorrent integration metadata + launch config.