arex-2-p300

A tt-model v6 thin bundle that serves BAAI/AREX-2 (revision d4e3502f92d9e889031c2d04e387ab2eb3520268) on 2 Tenstorrent blackhole chips (mesh P150x2). The tt-orchard harness staged it from a run of that model.

License

The weights this bundle points to are licensed Apache 2.0 (https://www.apache.org/licenses/LICENSE-2.0). The bundle ships no weights. tt-model downloads them from BAAI/AREX-2 with your own Hugging Face account.

What it runs

• Weights: BAAI/AREX-2 at revision d4e3502f92d9e889031c2d04e387ab2eb3520268.
• Model code: the Tenstorrent implementation of Qwen/Qwen3.8-27B, taken from the bundle qwen3.8-27b-dflash2-p300. BAAI/AREX-2 has the same text architecture, so only the weights differ.
• At each start the launcher builds model-dir/ from the configuration files of Qwen/Qwen3.8-27B (config.json, generation_config.json, preprocessor_config.json, video_preprocessor_config.json, shipped in base_config/) and the tokenizer and weights of BAAI/AREX-2. It sets MODEL_WEIGHTS_DIR and HF_MODEL to that directory, so the chips load the weights of BAAI/AREX-2.
• Context length 262144 tokens; up to 4 sequences at a time.
• BAAI/AREX-2 ships no generation_config.json, so the stop tokens and the sampling defaults the server applies when a request sets none are those of Qwen/Qwen3.8-27B.
• Speculative decoding is off. The drafter of Qwen/Qwen3.8-27B needs the MTP tensors of the model, and BAAI/AREX-2 has none.
• Sampling runs on the host, because on-device sampling is not used with the drafter off on this mesh.

Intended use

Text generation with BAAI/AREX-2 through the OpenAI-compatible server that tt-model serve starts.

Expected performance

Every number is labelled. A measured number names its evidence files; the first file holds the value. The files are in this repository under evidence/, at the paths shown, with the run's directory, the home directory and the host name replaced by <RUN_DIR>, <HOME> and <HOST>. TODO means not measured.

Number	Value	Label	Evidence
top1 agreement with the CPU reference, stage 2 (2 chips, bundle qwen3.8-27b-dflash2-p300)	0.96875 fraction	measured	stages/2/result.json, stages/2/evidence/swap-check.json, stages/2/evidence/server.log
server ready after start, stage 2 (empty tensor cache)	160.1 s	measured	stages/2/result.json, stages/2/evidence/swap-check.json, stages/2/evidence/server.log
top1 agreement with the CPU reference, this package (stage 7)	0.96875 fraction	measured	stages/7/verify/evidence/verify.json, stages/7/verify/evidence/server.log
server ready after start, this package (stage 7, fresh install, empty tensor cache)	218.1 s	measured	stages/7/verify/evidence/verify.json, stages/7/verify/evidence/server.log
GSM8K score, this model	0.9356 fraction	measured	stages/7/benchmark/benchmark.json
GSM8K replies cut off at the token limit, this model	16 items	measured	stages/7/benchmark/benchmark.json
GSM8K score, nearest model, same conditions	0.956 fraction	measured	stages/7/benchmark/benchmark.json
GSM8K replies cut off at the token limit, nearest model, same conditions	4 items	measured	stages/7/benchmark/benchmark.json
GSM8K paired sign test p, this model against the nearest model	0.003183 p-value	measured	stages/7/benchmark/benchmark.json
IFEval score, this model	0.8484 fraction	measured	stages/7/benchmark/benchmark.json
IFEval score, nearest model, same conditions	0.8595 fraction	measured	stages/7/benchmark/benchmark.json
IFEval paired sign test p, this model against the nearest model	0.5856 p-value	measured	stages/7/benchmark/benchmark.json
decode speed, this model, one request at a time, speculative decoding off (median of four prompts)	19.88 tokens/s	measured	stages/7/benchmark/speed.json
decode speed, nearest model, same conditions (median of four prompts)	16.73 tokens/s	measured	stages/7/benchmark/speed.json
time to first token, this model, one request at a time (median of four prompts)	0.32 s	measured	stages/7/benchmark/speed.json
time to first token, nearest model, same conditions (median of four prompts)	1.43 s	measured	stages/7/benchmark/speed.json

Speed and benchmark conditions

Speed was measured with one request at a time, greedy decoding, three repetitions of each of four short prompts, on the same 2-chip bundle with the speculative drafter off and sampling on the host. The same prompts ran against Qwen/Qwen3.8-27B served the same way. The values are in evidence/stages/7/benchmark/speed.json.

The medians are in the Numbers table above; the per-prompt values are in the evidence file.

The prompts are short. Longer contexts and several requests at once were not measured here.

The benchmark rows above come from evidence/stages/7/benchmark/benchmark.json, which lists the conditions. The points that change how to read them:

• Both models ran with thinking on, as the chat template sets it, at temperature 0.
• GSM8K was scored with the flexible-extract filter, 5-shot, over all 1,319 items, with a 32,768-token limit. The harness's default stop string Question: cuts a reasoning model off when it repeats that word while thinking, so these runs stop only on </s>. With the default stop strings AREX-2 scored 0.903 and the control 0.766, and most of that gap was empty replies. The strict-match filter gives 0.29 for AREX-2 and 0.75 for the control, because AREX-2 does not end its answers in the format that filter expects.
• IFEval is the prompt-level strict score over 541 prompts. The first pass used an 8,192-token limit. Prompts whose reply came back empty were asked again at 32,768 tokens (44 for AREX-2, 52 for the control) and the new answers replaced the old.
• On GSM8K the control scores higher than AREX-2 (paired sign test, p = 0.003). On IFEval the two are not distinguishable (p = 0.59).
• None of the authors' reported benchmarks (research, agent and coding tasks) was run here. Tool calling and long-running agent use were not tested.

Reported by the authors

These scores come from the BAAI/AREX-2 model card on Hugging Face (revision d4e3502f92d9e889031c2d04e387ab2eb3520268), which says the results follow the protocols in the AREX-2 paper. They were not measured here: the authors' protocol, tools and judge differ from the checks in the table above, and nothing in this repository reproduces them.

Benchmark	Reported	Note
Frontier-CS (188-task Agent Track)	70.7
MLE-Lite	81.8	Any Medal, averaged over three seeds
BrowseComp	84.0
HLE	52.6	text-only subset
GAIA	92.2
DeepSearchQA	93.8

Limitations

• Tested on 2 chips (mesh P150x2) only; other configurations are not claimed by this bundle.
• Accuracy was compared with the CPU reference on one short fixed prompt only.
• Speculative decoding is off, so decode speed is that of plain decoding.
• Sampling runs on the host.
• Default sampling settings are those of Qwen/Qwen3.8-27B, because BAAI/AREX-2 ships no generation_config.json. Send your own settings or use those on the model's card.

Risks and safety considerations

The check above does not cover long outputs, tool calling or safety behaviour. Output can differ from BAAI/AREX-2 run on a CPU or GPU in ways it does not show.

Not measured

• a download of the weights through tt-model pull, and a boot of the package from the Hub

Boot check

Stage 7 of the run installed this bundle, served it on a leased board and compared its tokens with the CPU reference. The Expected performance table has the result.

How to serve

tt-model serve episod/arex-2-p300