README.md

---
license: apache-2.0
base_model:
  - bespokelabs/Bespoke-Nimble-9B
  - Qwen/Qwen3.5-9B
tags:
  - vllm.cpp
  - decisions
  - systemone
  - nimble
  - localai
---

# Bespoke-Nimble-9B for vllm.cpp

This repository holds [bespokelabs/Bespoke-Nimble-9B](https://huggingface.co/bespokelabs/Bespoke-Nimble-9B)
merged into its base [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B)
and laid out as the `NimbleModel` directory that
[vllm.cpp](https://github.com/mudler/vllm.cpp) loads.

Nimble is a decision model from Bespoke Labs. For each question it runs one
forward pass and reads the logits of the answer letters at the last prompt
position. It does not generate text. vllm.cpp serves it on `/v1/systemone`
for `choice`, `noul` and `score` questions.

**This checkpoint only works with vllm.cpp** (directly, or through LocalAI's
`vllm-cpp` backend). The architecture name is `NimbleModel`, which
`transformers` and vLLM do not know. To run Nimble with the authors' own code,
use the upstream adapter.

## Redistribution and license

This is a redistribution of upstream weights, converted for vllm.cpp and
LocalAI. No weights were trained here. The only change is the LoRA merge
described below.

- Adapter, tokenizer, chat template, schema and temperature config:
  [bespokelabs/Bespoke-Nimble-9B](https://huggingface.co/bespokelabs/Bespoke-Nimble-9B)
  @ `bd792f44ec8e265be861bfcdf4e05967ffe0e858`, by Bespoke Labs, Apache-2.0.
  Project: [bespokelabsai/nimble](https://github.com/bespokelabsai/nimble).
- Base weights: [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B)
  @ `c202236235762e1c871ad0ccb60c8ee5ba337b9a`, by the Qwen team, Apache-2.0.

The upstream licenses apply to these files. Credit for the model belongs to
the original authors. This is the checkpoint without a suffix (temperature
1.0). `Bespoke-Nimble-9B-v2` is a different, older checkpoint and is not in
this repository.

## Conversion

The converter is `scripts/convert-nimble.py` from vllm.cpp. The directory was
produced on 2026-09-30 with that script as it is at vllm.cpp commit
`96788348627b6a079fcc3ef7fc6676a970965d7b` (the script did not change between
the conversion run and that commit). It checks the adapter's `SHA256SUMS` and
prompt-contract hash, pairs all 248 LoRA modules with base tensors of the
right shape, and merges them the PEFT way:
`W' = bf16(float32(W) + (B @ A) * 2)` (r=16, alpha=32).

```sh
hf download Qwen/Qwen3.5-9B --revision c202236235762e1c871ad0ccb60c8ee5ba337b9a \
  --local-dir Qwen3.5-9B
hf download bespokelabs/Bespoke-Nimble-9B \
  --revision bd792f44ec8e265be861bfcdf4e05967ffe0e858 --local-dir Bespoke-Nimble-9B
python3 scripts/convert-nimble.py Bespoke-Nimble-9B \
  --base-model-dir Qwen3.5-9B --output-dir nimble-9b
```

`config.json` records the provenance: adapter sha256
`29ef39b072dee97287947455337879c1e916705c2f727287922a2d81f5e2f20a`, base
revision, task `schema_candidate_classification_v2`, and the list of files
checked against `SHA256SUMS`.

## Files

| file | bytes | content |
|---|---|---|
| `model.safetensors-00001-of-00004.safetensors` | 5,276,436,248 | merged weights, BF16 (some F32 tensors as in the base) |
| `model.safetensors-00002-of-00004.safetensors` | 5,335,161,536 | merged weights |
| `model.safetensors-00003-of-00004.safetensors` | 5,368,717,472 | merged weights |
| `model.safetensors-00004-of-00004.safetensors` | 3,325,995,744 | merged weights |
| `model.safetensors.index.json` | 79,657 | shard index, same 775 tensor names as the base |
| `config.json` | 3,463 | base config, `architectures: ["NimbleModel"]`, `nimble_temperature: 1.0`, `nimble_max_length: 8192`, provenance |
| `tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja` | | from the adapter repo |
| `schema_config.json`, `temperature_config.json`, `adapter_config.json` | | from the adapter repo, kept for provenance |

The shards keep the base's shard names and full tensor set. Total weight size
is about 19.3 GB.

## Serve it with vllm.cpp

Build the vllm.cpp server (`-DVLLM_CPP_SERVER=ON`, target `vllm-server`), then:

```sh
hf download mudler/Bespoke-Nimble-9B-vllm-cpp --revision 52eead25b6723710ec16378942c9ae914d179b6f --local-dir nimble-9b
build/examples/vllm-server --model nimble-9b --served-model-name nimble --port 8000
```

## Serve it with LocalAI

A model config for LocalAI's `vllm-cpp` backend:

```yaml
name: nimble-9b
backend: vllm-cpp
known_usecases:
  - decisions
parameters:
  model: mudler/Bespoke-Nimble-9B-vllm-cpp
artifacts:
  - name: model
    target: model
    source:
      type: huggingface
      repo: mudler/Bespoke-Nimble-9B-vllm-cpp
      revision: 52eead25b6723710ec16378942c9ae914d179b6f
```

Then send the same request body to LocalAI's `POST /v1/systemone` with
`"model": "nimble-9b"`.

## Example

Request (served on CPU from this exact directory, 2026-09-30):

```sh
curl http://localhost:8000/v1/systemone -H 'Content-Type: application/json' -d '{
  "model": "nimble",
  "state": "I was charged twice for my subscription this month. Please refund the duplicate payment.",
  "questions": {
    "refund": {"type": "noul", "instructions": "Does the user request a refund?"},
    "department": {"type": "choice", "instructions": "Which department should handle this?",
                   "criteria": {"billing": "Payments and refunds", "technical": "Software bugs", "sales": "New purchases"}},
    "urgency": {"type": "score", "instructions": "How urgent is this?",
                "criteria": ["Routine", "Urgent", "Emergency"]}}}'
```

Response:

```json
{
  "model": "nimble",
  "answers": {
    "refund": {"type": "noul", "noul": 0.9966597833278789},
    "department": {"type": "choice", "choice": "billing",
                   "probabilities": {"billing": 0.9881318797907442, "technical": 0.010937236221712353, "sales": 0.0009308839875433919},
                   "confidence": 0.9383928321620487},
    "urgency": {"type": "score", "score": 0.3872836981253824,
                "legend": {"0": "Routine", "1": "Urgent", "2": "Emergency"},
                "probabilities": {"0": 0.6300001903087783, "1": 0.3527159212570611, "2": 0.017283888434160653},
                "confidence": 0.33663361186646856}
  },
  "usage": {"input_tokens": 988, "output_tokens": 0},
  "latency_ms": 75686.59
}
```

The answer semantics are those of Nimble's reference server (openjev): `noul`
is `p(true)`, `choice` is the most probable key, `score` is the expected level
index, `confidence` is `1 - H(p) / ln(K)`, and probabilities are not rounded.
The latency above is a 20-thread CPU and says nothing about GPU speed.

## What was verified and what was not

Verified:

- The one live request above, served by the vllm.cpp CPU server built at
  commit `96788348627b6a079fcc3ef7fc6676a970965d7b`, returns the answers
  shown. They are plausible for the input.
- Earlier on 2026-09-30, the vllm.cpp project compared this same converted
  directory, served on `/v1/systemone` on CPU, against the Bespoke authors'
  own `parallel_schema.prepare_prompts`, `inference.candidate_logits` and
  `inference.decision_result` over `transformers` 5.3.0 BF16 with the adapter
  unmerged through PEFT 0.21.0, also on CPU. Five requests, seven questions
  (choice, noul with and without criteria, score): argmax equal on 7 of 7,
  input token counts equal on 5 of 5, largest probability difference 0.0054.
  That comparison was not repeated for this upload.

Not verified:

- No GPU (CUDA, ROCm, Metal) run. The authors' reference runner requires a
  CUDA BF16 GPU; the comparison above called its functions on CPU.
- Seven questions are a check, not an accuracy measurement.
- No vLLM gate exists, because vLLM does not serve `NimbleModel` decisions.
- vllm.cpp refuses fields with more than 26 choices; the upstream release
  supports up to 255.
- No GGUF or quantized variant.