File size: 16,851 Bytes
81d1d0d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 | ---
license: cc-by-sa-4.0
base_model: Qwen/Qwen3.5-9B
base_model_relation: finetune
pipeline_tag: zero-shot-classification
language: [en]
datasets:
- nvidia/Aegis-AI-Content-Safety-Dataset-2.0
- google/boolq
- nvidia/HelpSteer2
- stanfordnlp/snli
tags: [openchoice, decision-model, gguf, lora, qwen3.5, cpu, vision, typesafe-compatible]
---
# OpenChoice-9B
**CPU-first multimodal System-One-compatible decisions, no GPU required:** about **5.8 GiB of serving weights**, **6–7 GiB RAM**, optional GPU offload, and an integrated **LoRA → GGUF** fine-tuning kit (tested on a 24 GB GPU).
This package contains the rank-16 OpenChoice decision adapter merged into the pinned `Qwen/Qwen3.5-9B` base and quantized as **GGUF Q4_K_M**, plus a **Q8_0 vision projector**. It directly returns probabilities for Choice, Score, and Noul; it does **not generate answer text**. The accompanying server implements the TypeSafe SDK 0.6-compatible `POST /v1/systemone` wire format. OpenChoice is independent of TypeSafe and is not a Jev implementation or parity claim.
The exact text file is **5.24 GiB / 5.63 decimal GB**, plus a **595 MiB projector**: about **5.8 GiB serving weights**. The complete download is about **6.0 GiB** including the tokenizer and retained training adapter. The base is a 9B-parameter model.
## What OpenChoice is built for, and when to use something else
Official JevBench v1.5.1 results are pending, so this is not a quality ranking.
- **OpenChoice is designed for** long inputs (4,096 tokens per forward, 32,768 per decision), chart decisions on a CPU through the Python engine (the HTTP API is text-only), and your own labels and house rules: fine-tune with LoRA and export back to the same GGUF server. We'll publish benchmark results for these as they come in.
- **Use a small encoder** (Laya, one-click via Unsloth Desktop, or GLiNER2.5-Decide) for short-text routing on any laptop with the smallest footprint, and Laya multilingual for many languages.
- **Use JevK5 v0.3 or Plumb-4B** for the smallest GGUF with a published JevBench v1.5.1 score (OpenChoice's official score is pending).
- **Use Jev, or Cygnet / Winnow-12B** (v1.5.1 composites 72.1 / 73.7 / 73.2) for higher published accuracy; OpenChoice trails Jev on every aggregate benchmark compared. Use a generative model if you need explanations.
## Quickstart: Docker (Linux containers; verified on Linux/amd64)
Use Docker Engine (Linux) or Docker Desktop with Linux containers (macOS/Windows). Recommend 16 GiB system RAM and at least 12 GiB assigned to Docker Desktop; this is not a tested minimum. Build from the [public source](https://github.com/lutr0/openchoice), or replace the first three lines with `docker pull ghcr.io/lutr0/openchoice:0.2.0-cpu` and `docker tag ghcr.io/lutr0/openchoice:0.2.0-cpu openchoice:cpu`. The following single-line commands use a POSIX shell or PowerShell 7.3+:
```sh
git clone https://github.com/lutr0/openchoice.git
cd openchoice
docker build -t openchoice:cpu .
docker volume create openchoice-model
docker run --rm --mount type=volume,source=openchoice-model,target=/models openchoice:cpu download lutr0/OpenChoice-9B
docker run --rm --entrypoint python openchoice:cpu -c "import secrets; print('OPENCHOICE_API_KEY=' + secrets.token_hex(32))" > openchoice.env
docker run -d --name openchoice --env-file openchoice.env -p 127.0.0.1:8768:8768 --mount type=volume,source=openchoice-model,target=/models,readonly openchoice:cpu
docker logs -f openchoice
```
Keep `openchoice.env` private. Wait for “Application startup complete”; Ctrl-C exits log-following without stopping the service. Startup checks all package hashes and warms up. `docker rm -f openchoice` stops/removes the container but retains the model volume. For an existing local package, replace the volume mount with `--mount "type=bind,source=/absolute/path/OpenChoice-9B-GGUF,target=/models,readonly"`; see the source README for host permissions/SELinux notes.
The CPU-only image runs as non-root UID 10001, binds `0.0.0.0` internally, and therefore requires `OPENCHOICE_API_KEY`. Publish only the loopback host port shown above. It has no Torch, PEFT, bitsandbytes, CUDA wheels or build toolchain. **No CUDA container variant or Apple Metal support is provided.** The portable runtime builds all supported CPU variants with `GGML_NATIVE=OFF` and chooses features at runtime.
**Verified scope:** Linux/amd64 built and ran here; health, the request below and all 34 SDK smoke cases passed. Its image is 353 MiB excluding weights and startup to health was 7.39 s on a shared Ryzen desktop with warm filesystem caches. Linux/arm64 was **built and smoke-run under QEMU emulation only, not verified on Apple Silicon/native arm64 hardware**. Both default SVE/SME and Cortex-A76 emulated CPUs passed health and returned the curl request, but both full SDK runs exceeded the unchanged 180 s per-request timeout. macOS/Windows Docker Desktop host integration was not exercised. The amd64 container/native maximum absolute probability delta was **0.0012598** on this example: portable CPU variants/GCC 14.2 versus `GGML_NATIVE=ON`/GCC 16.2, not bit-identical execution. See [container-verification.json](https://github.com/lutr0/openchoice/blob/main/results/container-verification.json) for exact outputs, emulation results and limitations.
In a second POSIX terminal, load the key and send a decision:
```sh
. ./openchoice.env
curl --fail http://127.0.0.1:8768/health
curl --fail http://127.0.0.1:8768/v1/systemone -H "Authorization: Bearer $OPENCHOICE_API_KEY" -H 'Content-Type: application/json' -d '{"model":"openchoice-latest","state":"My card was charged twice. Please refund the duplicate payment.","questions":{"topic":{"type":"choice","instructions":"Classify the message.","criteria":{"billing":"Billing and refunds","weather":"Weather forecast"}},"severity":{"type":"score","instructions":"Rate the payment problem.","criteria":["No payment problem","A duplicate charge requiring a refund"]},"duplicate":{"type":"noul","instructions":"Was the customer charged twice?"}}}'
```
For PowerShell 7.3+, load the key with `$env:OPENCHOICE_API_KEY = (Get-Content ./openchoice.env).Split('=', 2)[1]`, use `curl.exe`, and use `$env:OPENCHOICE_API_KEY` in the Authorization header. The source README includes the complete command. Expect normalized Choice/Score probabilities and a scalar Noul, not generated answer text. Do not expose the development server to untrusted networks.
### Native Linux alternative
Linux x86-64, Git, Make and a C++17 compiler are required; [uv](https://docs.astral.sh/uv/) supplies Python 3.12 and pinned CMake. Debian/Ubuntu prerequisites: `sudo apt-get install git build-essential`; Fedora: `sudo dnf install git gcc-c++ make`. After cloning the source as above:
```sh
bash scripts/build_llamacpp.sh
uv run hf download lutr0/OpenChoice-9B --local-dir ./OpenChoice-9B-GGUF
uv run python -m openchoice.server --model ./OpenChoice-9B-GGUF --port 8768
```
No separate sync step is needed. The script builds llama.cpp commit `4364bf7232e65c34eca8d9500c5464389662de6b` locally; no full upstream BF16 download is needed. Native builds target the current CPU and are not portable between CPUs. Native loopback serving does not require an API key; non-loopback serving does. The native build script supports Linux only; native macOS/Windows, AMD GPU and Metal paths are not verified.
Optional **native** NVIDIA offload requires a CUDA toolkit and compatible driver: build with `bash scripts/build_llamacpp.sh cuda`, then add `--gpu auto` to the server command. The CPU instance stays resident; a second instance offloads four layers only after 60 game-free seconds and at least 4 GiB free VRAM. It releases the offload instance at request boundaries when Linux game detection or low free VRAM requires it. This is best-effort resource management, not an OOM guarantee. Four-layer measurements are about 1.9 GiB additional VRAM and about 12.35 GiB total peak RSS. Images always run on CPU.
## Hardware and speed
Measured on a shared Linux desktop with a Ryzen 9 7900X3D, using 12 inference threads and CPU repacking disabled. These are observations, not hard latency guarantees or comparisons with hosted services.
| Mode | Measured memory | Measured latency |
|---|---|---|
| CPU text, Q4_K_M | About 6.10 GiB peak RSS, including mapped weights | About 1.7–3.3 s for a 300–600-token Choice/Noul decision |
| CPU vision, Q4_K_M + Q8_0 projector | About 6.71 GiB peak RSS | Chart Choice mean 4.16 s; six-level Score mean 25.74 s without image-cache reuse |
| Optional four-layer CUDA instance | 1,926 MiB peak VRAM in the recorded profile | Workload-dependent; not full-model GPU inference |
Use `--threads N` to control CPU contention. Score evaluates each level independently; the bridge reuses aligned prefixes/image embeddings only within a request. No cross-client prompt/image cache. Recorded two-row integration verification measured a 13.22 s standalone six-level Score with image-embedding reuse; do not compare that directly with the uncached full-set mean.
Evidence is in the source repository's `results/runtime/gguf-q4-norepack.json`, `vision-q4-q8_0.json`, `production-vision-verification.json`, and `production-gpu-verification.json`, linked and qualified in [RESULTS.md](https://github.com/lutr0/openchoice/blob/main/RESULTS.md).
## Evaluation: runtime-specific numbers
The released GGUF is **not numerically interchangeable** with the old NF4 implementation. On the frozen 200-row CPU feasibility sample it got **178/200**, versus NF4 **174/200**, with **96% decision agreement**. This reused sample is too small to establish improvement.
CPU vision, same Q4_K_M text + Q8_0 projector: **49/60 chart-development decisions** (Choice 30/30, Score 19/30), versus NF4 52/60. Charts share a generator with image rows used in training. This does not establish broad photograph recognition, OCR, multiple-image, or video quality.
**Full SciQ test on the released GGUF, CPU-only: 95.3% (953/1000), zero errors.** This rerun uses the same frozen test inputs as the historical comparison. CLM-v0.1-8B NF4 previously measured 58.7% (587/1000): a 36.6-point cross-runtime accuracy gap, with no reused NF4 confidence interval or speed comparison. Public raw outputs: source `results/gguf-sciq-predictions.jsonl`; summary: `results/gguf-sciq.json`.
The **400-task panel scores 80.25% (321/400)** on the released GGUF, CPU-only with zero errors. It covers 100 rows each of Aegis, BoolQ, HelpSteer2 and SNLI; those source families appeared in training. Summary/raw files in the source repository: `results/gguf-panel400.json` and `results/gguf-panel400-predictions.jsonl`.
**Nimble public on the released GGUF:** **75.41% micro (2926/3880)**; **74.72% macro** (unweighted mean across 13 frozen subsets). One interrupted-runner boundary is conservatively counted wrong without retry; all remaining responses were recorded. Source summary/raw: `results/gguf-nimble-public.json` and `results/gguf-nimble-public-predictions.jsonl`. The exact frozen payload hashes and the execution interruption are disclosed there; this is native Score evaluation, not Nimble's enum/Choice rating path.
**Historical NF4-only measurements**, retained for context rather than attributed to GGUF:
| Suite | Historical OpenChoice NF4 | Comparator |
|---|---:|---|
| SciQ official test, 1,000 | 95.6% (956/1000) | CLM-v0.1-8B NF4 58.7%, locally measured |
| 400-task decision panel | 79.5% (318/400) | CLM-v0.1-8B NF4 44.5%, locally measured |
| Nimble public, 3,880 | 75.0% (2909/3880) | Jev 77.3%, Nimble 75.9%, author-published |
| Nimble evidence-324 | 73.1% (237/324) | Jev 93.2%, Nimble 90.1%, author-published |
Full release GGUF reruns, raw outputs, historical tables, and protocols are in source [RESULTS.md](https://github.com/lutr0/openchoice/blob/main/RESULTS.md). SciQ was held out from project tuning/selection, not proven absent from upstream pretraining. The 400-row panel's source families appeared in training and the panel was previously used for final evaluation. Nimble evidence-324 has synthetic/model-checked labels and is in-domain for Nimble. OpenChoice has no sealed JevBench score. Confidence is uncalibrated; published comparator results are not independent local reproductions.
**Official JevBench v1.5.1: pending (sealed split requires maintainer submission).** This is a 1,624-decision, four-axis composite, not raw task accuracy. ImageJevBench v0.1.3 is also not measured for OpenChoice. Legacy public-task figures appear only in the source RESULTS archive and must not be relabeled as current scores.
CPU, GGUF, probabilities-only decisions, vision, and LoRA tuning are not individually unique features: alternatives include [Winnow](https://huggingface.co/EldanRing/Winnow-12B), [JevK5](https://huggingface.co/alibiserikbay/JevK5), [Plumb](https://huggingface.co/crh225/plumb-4b), [Imajev](https://huggingface.co/mohit67890/imajev-4b), and [GLiNER2.5-Decide](https://huggingface.co/fastino/GLiNER2.5-Decide). OpenChoice offers a CPU-first multimodal deployment plus integrated customization/export path, not a smallest/fastest/best claim. See the source [current artifact comparison](https://github.com/lutr0/openchoice#current-alternatives-artifacts-not-a-quality-ranking) for documented capabilities, sizes, exact-version caveats, and separately labeled published scores.
## Vision and API limits
HTTP handles text and structured JSON, **not image uploads**. CPU image decisions use `LlamaCppEngine.predict_rows` with local image paths and SHA-256 hashes; see the source README's complete example. The native projector uses M-RoPE and the actual image encoder, not captions or image filenames as substitutes.
Choice supports 1–26 options, Score 1–10 levels; requests support up to 64 questions and 2 MiB JSON. No truncation: 4,096 tokens per forward and 32,768 tokens per decision. Over-limit/unsupported requests return 422. One worker serializes inference, excess queue entries return 503, and disconnected clients' work is cancelled between microbatches. Confidence is a concentration statistic, not a guarantee. English-centric training, option-order sensitivity, limited vision validation, and no coding/tool-trajectory evaluation remain limitations.
## Fine-tune and export
The original `adapter/`, empty slot-model `heads.safetensors`, and `model.json` remain in this package for fine-tuning; they are not the serving backend. The slot model reads logits for option letters or Yes/No rather than a trained classifier head.
```sh
uv run --group train python -m openchoice.finetune --help
```
See [FINETUNING.md](https://github.com/lutr0/openchoice/blob/main/FINETUNING.md) for the `validate`, `train`, `eval`, and `export` commands. Training uses the full pinned base model plus PEFT; its GPU/RAM requirements are much higher than CPU serving. Export merges the adapter, writes Q4_K_M text weights and the projector, preserves processor files, and generates the exact portable manifest required by the server. Fine-tuned exports need their own evaluation.
## Package contract and identity
`manifest.json` schema: `openchoice-gguf-v1`. It names `model`, `projector`, `processor`, `checkpoint_config`, `quantization`, `base_model`, `base_revision`, and `llama_commit`; `files` maps relative file paths to SHA-256. Serving rejects corrupted artifacts and paths escaping the package. Compiled local libraries are **not** shipped in this package. The manifest itself and card/license prose are not self-hashed.
- Text GGUF SHA-256: `6d8eef23223c2f53a772fa5a945842569e2cd646261bcb0a354e6c0f4e5ee238`
- Q8_0 projector SHA-256: `44c7ddca015a856956bbf2ff4a88049a4bc1dcff47fabd93602f23554d307e0c`
- Base revision: `c202236235762e1c871ad0ccb60c8ee5ba337b9a`
- Adapter: rank 16, alpha 32; original selected run used 100 updates and 3,200 rows / 3,234,666 tokens, seed 20260921.
## License and data-rights caveat
The weights are declared **CC-BY-SA-4.0** as a conservative choice because BoolQ uses CC-BY-SA-3.0 and SNLI uses CC-BY-SA-4.0. **This is not a legal determination that these terms are sufficient or necessary for derived adapter/merged/quantized weights.** The source code is separately Apache-2.0; the Qwen base is Apache-2.0. See LICENSE, NOTICE, and source [PUBLISHING.md](https://github.com/lutr0/openchoice/blob/main/PUBLISHING.md).
Training sources: NVIDIA Aegis AI Content Safety Dataset 2.0 human prompt labels (CC-BY-4.0), BoolQ (CC-BY-SA-3.0), NVIDIA HelpSteer2 human ratings of model-generated responses (CC-BY-4.0), Stanford SNLI (CC-BY-SA-4.0), and internally authored replay material. Benchmark inputs are not bundled as training data. Names of Jev, TypeSafe, and other models identify independent compatibility/comparisons, not affiliation or endorsement.
|