| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3.5-9B |
| language: |
| - en |
| pipeline_tag: text-classification |
| tags: |
| - decision |
| - calibration |
| - reasoning |
| - speculative-decoding |
| - qwen3.5 |
| --- |
| |
| # Jeeves-9B |
|
|
| A Jev-like decision model that reasons before it decides. Give it a state (text or JSON) and yes/no, multiple-choice or rating questions; it writes a reasoning chain per question and returns a calibrated probability for every option. |
|
|
| This repository holds the fused weights (Qwen3.5-9B with the LoRA merged), the pointer head, the fitted temperature, and two speculative-decoding drafters. It loads with the Jeeves code, not with `transformers`: the pointer head and the prompt format are part of the model. |
|
|
| ## Files |
|
|
| | file | contents | |
| |---|---| |
| | `model-*.safetensors`, `config.json` | fused Qwen3.5-9B weights, bf16 | |
| | `head.pt` | pointer head (query and key projections, 256-dim) | |
| | `export.json` | temperature (1.859), prompt format, training step and config | |
| | `tokenizer.json`, `vocab.json`, `merges.txt`, `tokenizer_config.json`, `chat_template.jinja` | Qwen3.5 tokenizer | |
| | `drafter_k4.safetensors` | block-4 drafter, the serving default, bf16 | |
| | `drafter_k8.safetensors` | block-8 drafter, faster for single-question requests, bf16 | |
| | `drafter_k4.config.json`, `drafter_k8.config.json` | drafter training settings | |
| | `LICENSE` | Apache-2.0, inherited from Qwen3.5-9B | |
|
|
| ## Usage |
|
|
| ```bash |
| hf download PostHog/jeeves --local-dir jeeves-weights |
| git clone https://github.com/PostHog/jeeves && cd jeeves |
| python -m inference.serve --model ../jeeves-weights --drafter ../jeeves-weights/drafter_k4.safetensors --port 8009 |
| ``` |
|
|
| ```bash |
| curl -s localhost:8009/v1/systemone -H 'content-type: application/json' -d '{ |
| "state": "I was charged twice for one order.", |
| "questions": { |
| "billing": {"type": "noul", "instructions": "Is this about billing?"}, |
| "tone": {"type": "choice", "instructions": "What is the customer'"'"'s tone?", "criteria": {"calm": null, "frustrated": null, "angry": null}} |
| }}' |
| ``` |
|
|
| The request and response follow Jev's `/v1/systemone` format. An optional `options` object sets `think`, `max_think` (truncate chains), `nothink_threshold` (skip thinking when the no-think answer is already confident) and `return_reasoning`. Serving runs on a CUDA GPU or on an Apple Silicon Mac with 48 GB (add `--max-rows 4 --max-len 4096` there). It serves bf16 by default; `--precision fp8` runs the linear layers in FP8 on GPUs with compute capability 8.9 or higher and on Macs. FP8 weights can be downloaded from [PostHog/jeeves-fp8](https://huggingface.co/PostHog/jeeves-fp8). |
|
|
| ## Results |
|
|
| Accuracy with thinking, greedy, 2,560-token cap. Kev-9B and Jev numbers are the ones Kev publishes. JevBench uses the same 231 public items for every model (the sealed judge tier is not included); the other rows use the same sources with different items. |
|
|
| | | Kev-9B | Jev | Jeeves-9B | |
| |---|---|---|---| |
| | Test overall (out-of-domain and held-out) | 0.822 | 0.857 | **0.889** | |
| | MMLU-Pro and buried state | 0.579 | **0.800** | 0.746 | |
| | JevBench, public tiers | 0.715* | 0.866 | **0.935** | |
| | JevBench hard | 0.451* | 0.730 | **0.865** | |
|
|
| \* Kev-8B (Qwen3); no Kev-9B JevBench result is published. |
|
|
| Without thinking the model scores 0.804 on the Jeeves test split, against 0.840 with it. With the fitted temperature the no-think path has a top-label calibration error of 0.021 and 1.3% confident errors (wrong at p ≥ 0.9). |
|
|
| Serving on one H100, 325 dev questions: |
|
|
| | setting | accuracy | median / p90 latency | |
| |---|---|---| |
| | full thinking | 0.825 | 3.3 s / 17.1 s | |
| | `max_think` 768, `nothink_threshold` 0.9 | 0.806 | 2.0 s / 5.6 s | |
| | no thinking | 0.775 | about 0.3 s | |
|
|
| ## Training |
|
|
| - **SFT.** LoRA r=16 on every projection plus the pointer head, 2 epochs on 19,126 questions from 12 public datasets and synthetic policy data; half the questions carry a reasoning chain sampled from the base model. |
| - **CISPO.** 9,992 RL questions, 8 rollouts each, up to 2,560 thinking tokens. Reward is the probability of the correct option, discounted by up to 10% for long chains. The released checkpoint is step 402 of a 624-step schedule. |
| - **Calibration.** One softmax temperature fitted on the dev set. |
| - **Drafters.** Diffusion views of the frozen model, inspired by Orthrus, distilled by KL divergence on chains sampled from this model. At one question the block-4 drafter decodes 1.6× faster than graphed greedy decoding and the block-8 drafter 1.76×. |
|
|
| A from-scratch reproduction with the released code matched this checkpoint within noise. The training code, dataset build scripts and pinned dataset revisions are in the GitHub repository. |
|
|
| ## Limitations |
|
|
| - Knowledge questions trail Jev (MMLU 0.793 vs 0.900, MMLU-Pro 0.739 vs 0.840). |
| - Thinking is slow at the tail (17 s at p90 with full chains); use `max_think` and `nothink_threshold` when latency matters. |
| - About a third of full-length chains hit the token cap without closing; accuracy on those items is lower. |
| - Evaluated on English inputs only. |
|
|
| ## License |
|
|
| The weights are derived from Qwen3.5-9B and are released under its Apache-2.0 license (`LICENSE`). The Jeeves code is MIT licensed. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @software{waltz2026jeeves, |
| author = {Waltz, Nicholas P.}, |
| title = {Jeeves: Reasoning Improves Jev-like Decisions}, |
| year = {2026}, |
| url = {https://github.com/PostHog/jeeves}, |
| note = {Qwen3.5-9B decision model trained with SFT and CISPO, with a block-4 diffusion drafter} |
| } |
| ``` |
|
|
