---
license: cc-by-sa-4.0
base_model: Qwen/Qwen3.5-9B
base_model_relation: finetune
pipeline_tag: zero-shot-classification
language: [en]
datasets:
  - nvidia/Aegis-AI-Content-Safety-Dataset-2.0
  - google/boolq
  - nvidia/HelpSteer2
  - stanfordnlp/snli
tags: [openchoice, decision-model, gguf, lora, qwen3.5, cpu, vision, typesafe-compatible]
---

# OpenChoice-9B

**CPU-first multimodal System-One-compatible decisions, no GPU required:** about **5.8 GiB of serving weights**, **6–7 GiB RAM**, optional GPU offload, and an integrated **LoRA → GGUF** fine-tuning kit (tested on a 24 GB GPU).

This package contains the rank-16 OpenChoice decision adapter merged into the pinned `Qwen/Qwen3.5-9B` base and quantized as **GGUF Q4_K_M**, plus a **Q8_0 vision projector**. It directly returns probabilities for Choice, Score, and Noul; it does **not generate answer text**. The accompanying server implements the TypeSafe SDK 0.6-compatible `POST /v1/systemone` wire format. OpenChoice is independent of TypeSafe and is not a Jev implementation or parity claim.

The exact text file is **5.24 GiB / 5.63 decimal GB**, plus a **595 MiB projector**: about **5.8 GiB serving weights**. The complete download is about **6.0 GiB** including the tokenizer and retained training adapter. The base is a 9B-parameter model.

## What OpenChoice is built for, and when to use something else

Official JevBench v1.5.1 results are pending, so this is not a quality ranking.

- **OpenChoice is designed for** long inputs (4,096 tokens per forward, 32,768 per decision), chart decisions on a CPU through the Python engine (the HTTP API is text-only), and your own labels and house rules: fine-tune with LoRA and export back to the same GGUF server. We'll publish benchmark results for these as they come in.
- **Use a small encoder** (Laya, one-click via Unsloth Desktop, or GLiNER2.5-Decide) for short-text routing on any laptop with the smallest footprint, and Laya multilingual for many languages.
- **Use JevK5 v0.3 or Plumb-4B** for the smallest GGUF with a published JevBench v1.5.1 score (OpenChoice's official score is pending).
- **Use Jev, or Cygnet / Winnow-12B** (v1.5.1 composites 72.1 / 73.7 / 73.2) for higher published accuracy; OpenChoice trails Jev on every aggregate benchmark compared. Use a generative model if you need explanations.

## Quickstart: Docker (Linux containers; verified on Linux/amd64)

Use Docker Engine (Linux) or Docker Desktop with Linux containers (macOS/Windows). Recommend 16 GiB system RAM and at least 12 GiB assigned to Docker Desktop; this is not a tested minimum. Build from the [public source](https://github.com/lutr0/openchoice), or replace the first three lines with `docker pull ghcr.io/lutr0/openchoice:0.2.0-cpu` and `docker tag ghcr.io/lutr0/openchoice:0.2.0-cpu openchoice:cpu`. The following single-line commands use a POSIX shell or PowerShell 7.3+:

```sh
git clone https://github.com/lutr0/openchoice.git
cd openchoice
docker build -t openchoice:cpu .
docker volume create openchoice-model
docker run --rm --mount type=volume,source=openchoice-model,target=/models openchoice:cpu download lutr0/OpenChoice-9B
docker run --rm --entrypoint python openchoice:cpu -c "import secrets; print('OPENCHOICE_API_KEY=' + secrets.token_hex(32))" > openchoice.env
docker run -d --name openchoice --env-file openchoice.env -p 127.0.0.1:8768:8768 --mount type=volume,source=openchoice-model,target=/models,readonly openchoice:cpu
docker logs -f openchoice
```

Keep `openchoice.env` private. Wait for “Application startup complete”; Ctrl-C exits log-following without stopping the service. Startup checks all package hashes and warms up. `docker rm -f openchoice` stops/removes the container but retains the model volume. For an existing local package, replace the volume mount with `--mount "type=bind,source=/absolute/path/OpenChoice-9B-GGUF,target=/models,readonly"`; see the source README for host permissions/SELinux notes.

The CPU-only image runs as non-root UID 10001, binds `0.0.0.0` internally, and therefore requires `OPENCHOICE_API_KEY`. Publish only the loopback host port shown above. It has no Torch, PEFT, bitsandbytes, CUDA wheels or build toolchain. **No CUDA container variant or Apple Metal support is provided.** The portable runtime builds all supported CPU variants with `GGML_NATIVE=OFF` and chooses features at runtime.

**Verified scope:** Linux/amd64 built and ran here; health, the request below and all 34 SDK smoke cases passed. Its image is 353 MiB excluding weights and startup to health was 7.39 s on a shared Ryzen desktop with warm filesystem caches. Linux/arm64 was **built and smoke-run under QEMU emulation only, not verified on Apple Silicon/native arm64 hardware**. Both default SVE/SME and Cortex-A76 emulated CPUs passed health and returned the curl request, but both full SDK runs exceeded the unchanged 180 s per-request timeout. macOS/Windows Docker Desktop host integration was not exercised. The amd64 container/native maximum absolute probability delta was **0.0012598** on this example: portable CPU variants/GCC 14.2 versus `GGML_NATIVE=ON`/GCC 16.2, not bit-identical execution. See [container-verification.json](https://github.com/lutr0/openchoice/blob/main/results/container-verification.json) for exact outputs, emulation results and limitations.

In a second POSIX terminal, load the key and send a decision:

```sh
. ./openchoice.env
curl --fail http://127.0.0.1:8768/health
curl --fail http://127.0.0.1:8768/v1/systemone -H "Authorization: Bearer $OPENCHOICE_API_KEY" -H 'Content-Type: application/json' -d '{"model":"openchoice-latest","state":"My card was charged twice. Please refund the duplicate payment.","questions":{"topic":{"type":"choice","instructions":"Classify the message.","criteria":{"billing":"Billing and refunds","weather":"Weather forecast"}},"severity":{"type":"score","instructions":"Rate the payment problem.","criteria":["No payment problem","A duplicate charge requiring a refund"]},"duplicate":{"type":"noul","instructions":"Was the customer charged twice?"}}}'
```

For PowerShell 7.3+, load the key with `$env:OPENCHOICE_API_KEY = (Get-Content ./openchoice.env).Split('=', 2)[1]`, use `curl.exe`, and use `$env:OPENCHOICE_API_KEY` in the Authorization header. The source README includes the complete command. Expect normalized Choice/Score probabilities and a scalar Noul, not generated answer text. Do not expose the development server to untrusted networks.

### Native Linux alternative

Linux x86-64, Git, Make and a C++17 compiler are required; [uv](https://docs.astral.sh/uv/) supplies Python 3.12 and pinned CMake. Debian/Ubuntu prerequisites: `sudo apt-get install git build-essential`; Fedora: `sudo dnf install git gcc-c++ make`. After cloning the source as above:

```sh
bash scripts/build_llamacpp.sh
uv run hf download lutr0/OpenChoice-9B --local-dir ./OpenChoice-9B-GGUF
uv run python -m openchoice.server --model ./OpenChoice-9B-GGUF --port 8768
```

No separate sync step is needed. The script builds llama.cpp commit `4364bf7232e65c34eca8d9500c5464389662de6b` locally; no full upstream BF16 download is needed. Native builds target the current CPU and are not portable between CPUs. Native loopback serving does not require an API key; non-loopback serving does. The native build script supports Linux only; native macOS/Windows, AMD GPU and Metal paths are not verified.

Optional **native** NVIDIA offload requires a CUDA toolkit and compatible driver: build with `bash scripts/build_llamacpp.sh cuda`, then add `--gpu auto` to the server command. The CPU instance stays resident; a second instance offloads four layers only after 60 game-free seconds and at least 4 GiB free VRAM. It releases the offload instance at request boundaries when Linux game detection or low free VRAM requires it. This is best-effort resource management, not an OOM guarantee. Four-layer measurements are about 1.9 GiB additional VRAM and about 12.35 GiB total peak RSS. Images always run on CPU.

## Hardware and speed

Measured on a shared Linux desktop with a Ryzen 9 7900X3D, using 12 inference threads and CPU repacking disabled. These are observations, not hard latency guarantees or comparisons with hosted services.

| Mode | Measured memory | Measured latency |
|---|---|---|
| CPU text, Q4_K_M | About 6.10 GiB peak RSS, including mapped weights | About 1.7–3.3 s for a 300–600-token Choice/Noul decision |
| CPU vision, Q4_K_M + Q8_0 projector | About 6.71 GiB peak RSS | Chart Choice mean 4.16 s; six-level Score mean 25.74 s without image-cache reuse |
| Optional four-layer CUDA instance | 1,926 MiB peak VRAM in the recorded profile | Workload-dependent; not full-model GPU inference |

Use `--threads N` to control CPU contention. Score evaluates each level independently; the bridge reuses aligned prefixes/image embeddings only within a request. No cross-client prompt/image cache. Recorded two-row integration verification measured a 13.22 s standalone six-level Score with image-embedding reuse; do not compare that directly with the uncached full-set mean.

Evidence is in the source repository's `results/runtime/gguf-q4-norepack.json`, `vision-q4-q8_0.json`, `production-vision-verification.json`, and `production-gpu-verification.json`, linked and qualified in [RESULTS.md](https://github.com/lutr0/openchoice/blob/main/RESULTS.md).

## Evaluation: runtime-specific numbers

The released GGUF is **not numerically interchangeable** with the old NF4 implementation. On the frozen 200-row CPU feasibility sample it got **178/200**, versus NF4 **174/200**, with **96% decision agreement**. This reused sample is too small to establish improvement.

CPU vision, same Q4_K_M text + Q8_0 projector: **49/60 chart-development decisions** (Choice 30/30, Score 19/30), versus NF4 52/60. Charts share a generator with image rows used in training. This does not establish broad photograph recognition, OCR, multiple-image, or video quality.

**Full SciQ test on the released GGUF, CPU-only: 95.3% (953/1000), zero errors.** This rerun uses the same frozen test inputs as the historical comparison. CLM-v0.1-8B NF4 previously measured 58.7% (587/1000): a 36.6-point cross-runtime accuracy gap, with no reused NF4 confidence interval or speed comparison. Public raw outputs: source `results/gguf-sciq-predictions.jsonl`; summary: `results/gguf-sciq.json`.

The **400-task panel scores 80.25% (321/400)** on the released GGUF, CPU-only with zero errors. It covers 100 rows each of Aegis, BoolQ, HelpSteer2 and SNLI; those source families appeared in training. Summary/raw files in the source repository: `results/gguf-panel400.json` and `results/gguf-panel400-predictions.jsonl`.

**Nimble public on the released GGUF:** **75.41% micro (2926/3880)**; **74.72% macro** (unweighted mean across 13 frozen subsets). One interrupted-runner boundary is conservatively counted wrong without retry; all remaining responses were recorded. Source summary/raw: `results/gguf-nimble-public.json` and `results/gguf-nimble-public-predictions.jsonl`. The exact frozen payload hashes and the execution interruption are disclosed there; this is native Score evaluation, not Nimble's enum/Choice rating path.

**Historical NF4-only measurements**, retained for context rather than attributed to GGUF:

| Suite | Historical OpenChoice NF4 | Comparator |
|---|---:|---|
| SciQ official test, 1,000 | 95.6% (956/1000) | CLM-v0.1-8B NF4 58.7%, locally measured |
| 400-task decision panel | 79.5% (318/400) | CLM-v0.1-8B NF4 44.5%, locally measured |
| Nimble public, 3,880 | 75.0% (2909/3880) | Jev 77.3%, Nimble 75.9%, author-published |
| Nimble evidence-324 | 73.1% (237/324) | Jev 93.2%, Nimble 90.1%, author-published |

Full release GGUF reruns, raw outputs, historical tables, and protocols are in source [RESULTS.md](https://github.com/lutr0/openchoice/blob/main/RESULTS.md). SciQ was held out from project tuning/selection, not proven absent from upstream pretraining. The 400-row panel's source families appeared in training and the panel was previously used for final evaluation. Nimble evidence-324 has synthetic/model-checked labels and is in-domain for Nimble. OpenChoice has no sealed JevBench score. Confidence is uncalibrated; published comparator results are not independent local reproductions.

**Official JevBench v1.5.1: pending (sealed split requires maintainer submission).** This is a 1,624-decision, four-axis composite, not raw task accuracy. ImageJevBench v0.1.3 is also not measured for OpenChoice. Legacy public-task figures appear only in the source RESULTS archive and must not be relabeled as current scores.

CPU, GGUF, probabilities-only decisions, vision, and LoRA tuning are not individually unique features: alternatives include [Winnow](https://huggingface.co/EldanRing/Winnow-12B), [JevK5](https://huggingface.co/alibiserikbay/JevK5), [Plumb](https://huggingface.co/crh225/plumb-4b), [Imajev](https://huggingface.co/mohit67890/imajev-4b), and [GLiNER2.5-Decide](https://huggingface.co/fastino/GLiNER2.5-Decide). OpenChoice offers a CPU-first multimodal deployment plus integrated customization/export path, not a smallest/fastest/best claim. See the source [current artifact comparison](https://github.com/lutr0/openchoice#current-alternatives-artifacts-not-a-quality-ranking) for documented capabilities, sizes, exact-version caveats, and separately labeled published scores.

## Vision and API limits

HTTP handles text and structured JSON, **not image uploads**. CPU image decisions use `LlamaCppEngine.predict_rows` with local image paths and SHA-256 hashes; see the source README's complete example. The native projector uses M-RoPE and the actual image encoder, not captions or image filenames as substitutes.

Choice supports 1–26 options, Score 1–10 levels; requests support up to 64 questions and 2 MiB JSON. No truncation: 4,096 tokens per forward and 32,768 tokens per decision. Over-limit/unsupported requests return 422. One worker serializes inference, excess queue entries return 503, and disconnected clients' work is cancelled between microbatches. Confidence is a concentration statistic, not a guarantee. English-centric training, option-order sensitivity, limited vision validation, and no coding/tool-trajectory evaluation remain limitations.

## Fine-tune and export

The original `adapter/`, empty slot-model `heads.safetensors`, and `model.json` remain in this package for fine-tuning; they are not the serving backend. The slot model reads logits for option letters or Yes/No rather than a trained classifier head.

```sh
uv run --group train python -m openchoice.finetune --help
```

See [FINETUNING.md](https://github.com/lutr0/openchoice/blob/main/FINETUNING.md) for the `validate`, `train`, `eval`, and `export` commands. Training uses the full pinned base model plus PEFT; its GPU/RAM requirements are much higher than CPU serving. Export merges the adapter, writes Q4_K_M text weights and the projector, preserves processor files, and generates the exact portable manifest required by the server. Fine-tuned exports need their own evaluation.

## Package contract and identity

`manifest.json` schema: `openchoice-gguf-v1`. It names `model`, `projector`, `processor`, `checkpoint_config`, `quantization`, `base_model`, `base_revision`, and `llama_commit`; `files` maps relative file paths to SHA-256. Serving rejects corrupted artifacts and paths escaping the package. Compiled local libraries are **not** shipped in this package. The manifest itself and card/license prose are not self-hashed.

- Text GGUF SHA-256: `6d8eef23223c2f53a772fa5a945842569e2cd646261bcb0a354e6c0f4e5ee238`
- Q8_0 projector SHA-256: `44c7ddca015a856956bbf2ff4a88049a4bc1dcff47fabd93602f23554d307e0c`
- Base revision: `c202236235762e1c871ad0ccb60c8ee5ba337b9a`
- Adapter: rank 16, alpha 32; original selected run used 100 updates and 3,200 rows / 3,234,666 tokens, seed 20260921.

## License and data-rights caveat

The weights are declared **CC-BY-SA-4.0** as a conservative choice because BoolQ uses CC-BY-SA-3.0 and SNLI uses CC-BY-SA-4.0. **This is not a legal determination that these terms are sufficient or necessary for derived adapter/merged/quantized weights.** The source code is separately Apache-2.0; the Qwen base is Apache-2.0. See LICENSE, NOTICE, and source [PUBLISHING.md](https://github.com/lutr0/openchoice/blob/main/PUBLISHING.md).

Training sources: NVIDIA Aegis AI Content Safety Dataset 2.0 human prompt labels (CC-BY-4.0), BoolQ (CC-BY-SA-3.0), NVIDIA HelpSteer2 human ratings of model-generated responses (CC-BY-4.0), Stanford SNLI (CC-BY-SA-4.0), and internally authored replay material. Benchmark inputs are not bundled as training data. Names of Jev, TypeSafe, and other models identify independent compatibility/comparisons, not affiliation or endorsement.