OneJev1.5-9B-NVFP4

OmniJev/OneJev1.5-9B quantized to NVFP4 with NVIDIA Model-Optimizer, for Blackwell GPUs (RTX 50xx, B200/GB200), served with vLLM. OneJev is a multimodal System One decision model: give it images and/or text plus typed questions (yes/no, multiple choice, score) and it returns a probability for every option from one next-token readout.

	BF16 (original)	This model
Weights	18 GB	8.3 GB
Throughput, 6 x 1280x720 images + 10 questions per request, one RTX 5090, vLLM 0.32	1.17 req/s	2.33 req/s
Median latency at 10 requests in flight	8.8 s	4.3 s

What is quantized

• NVFP4 (W4A4, block 16, FP8 scales): every Linear layer of the language model: full-attention q/k/v/o, the Gated DeltaNet in_proj_qkv, in_proj_z and out_proj, and the MLP gate/up/down projections (ModelOpt preset nvfp4, max calibration).
• Kept in BF16: the vision encoder, embeddings, lm_head, Gated DeltaNet conv1d, in_proj_a/in_proj_b, MTP.
• KV cache: BF16.
• Calibration: 128 requests in OneJev's own prompt format (qev-labels-v2) from the public Decision Index suite: 64 text rows (ANLI, ARC-Challenge, ARC-Easy, GSM8K, HellaSwag, MMLU, WinoGrande, ContractNLI) and 64 image rows (CV-Bench, BLINK, RealWorldQA, CharXiv, CORD), run through the full VLM so the language model sees real image tokens. Only quantization scales were fit; no benchmark data is stored in the weights.

Deviation from BF16

On 780 held-out Decision Index rows (60 each from the 13 benchmarks above, disjoint from calibration), OneJev's prompt:

	BF16 rerun (noise)	This model
Same answer as BF16	99.6%	95.6%
Mean total-variation distance of the option distribution	0.002	0.037
Accuracy vs gold (BF16: 85.8%)	85.9%	84.9%

On 30 construction-site camera requests (6 x 720p frames, 10 yes/no / choice / score questions each, 300 answers): 299/300 decisions identical to BF16, mean |dP(yes)| 0.022. Arithmetic (GSM8K) is the most affected category (83% same answer); long-document NLI (ContractNLI) is unaffected.

Serving

Requires a Blackwell GPU, a driver for CUDA 13, and vLLM 0.32. FlashInfer JIT-compiles its sm_120 NVFP4 GEMM on the first start and needs nvcc >= 12.9, so install the CUDA 13 compiler wheel next to vLLM and point CUDA_HOME at it:

python3.12 -m venv .venv
.venv/bin/pip install "vllm==0.32.0" "nvidia-cuda-nvcc==13.4.*"
export CUDA_HOME=$(.venv/bin/python -c "import nvidia,os;print(os.path.dirname(nvidia.__path__[0]+'/x'))")/cu13
export PATH=$CUDA_HOME/bin:$PATH
.venv/bin/vllm serve giwone1330/OneJev1.5-9B-NVFP4 --served-model-name OmniJev/OneJev1.5-9B \
  --gpu-memory-utilization 0.80 --max-model-len 16384 --max-num-seqs 128 --max-num-batched-tokens 16384 \
  --enable-prefix-caching --logprobs-mode processed_logprobs --max-logprobs 64 \
  --limit-mm-per-prompt '{"image": 8, "video": 0}' --no-enable-flashinfer-autotune

• --no-enable-flashinfer-autotune: the autotune pass runs out of memory on a 32 GB card after the KV cache is allocated.
• --enable-prefix-caching: all questions about the same images share the image prefill.
• --max-num-seqs 128: each question is one sequence; 64 left the GPU idle about 40% of the time.

A ready-made kit is in the serve/ folder of this repository. With Docker it is one file, serve/onejev-nvfp4-5090.sh, on the stock vllm/vllm-openai:v0.32.0 image (it already has the CUDA 13 compiler FlashInfer needs):

curl -LO https://huggingface.co/giwone1330/OneJev1.5-9B-NVFP4/resolve/main/serve/onejev-nvfp4-5090.sh
bash onejev-nvfp4-5090.sh          # start + wait until ready + follow logs;  stop: bash onejev-nvfp4-5090.sh stop

Without Docker (setup.sh, serve.sh, stop.sh in a Python venv):

hf download giwone1330/OneJev1.5-9B-NVFP4 --include "serve/*" --local-dir onejev-nvfp4
cd onejev-nvfp4/serve && bash setup.sh && bash serve.sh
.venv/bin/python client_example.py frame1.jpg frame2.jpg
bash stop.sh

Asking questions

The server speaks the OpenAI chat API, not the System One API. One question is one /v1/chat/completions call with max_tokens: 1, logprobs: true and logprob_token_ids set to the option letters; the probabilities are the softmax over those letters' logprobs. The prompt must be OneJev's own (from github.com/OmniJev/OneJev, qev/prompt.py): system prompt, then a user turn <state> with the images and state text </state>, then the question with its options under A., B., ... serve/client_example.py implements it without dependencies beyond aiohttp and accepts the System One question JSON (noul, choice, score).

Throughput scaling (one RTX 5090, 10 requests in flight)

Per request: about 0.068 ms per prompt token for the shared image prefill, plus about 6 ms + 1 ms per 1,000 prompt tokens for each question.

Images per request	Prompt tokens	Q=1	Q=10	Q=40
1 x 640x360	342	76 req/s	13.3 req/s	3.6 req/s
6 x 1280x720	5,442	3.3 req/s	2.33 req/s	1.2 req/s
6 x 1920x1080	12,402	1.1 req/s	0.95 req/s	0.53 req/s

License

Apache-2.0, as the base model OmniJev/OneJev1.5-9B (a fine-tune of Qwen/Qwen3.5-9B).