Javi0.3-4B-EXL3

Javi0.3-4B in EXL3 (QTIP trellis coding) at 5.0 bits per weight: 4.2 GB on disk, about half the merged bf16 weights and a little under the Q8_0 GGUF, at a quality cost you should not see. On one DGX Spark it decodes at ~58 tok/s single-stream, against 45.5 tok/s for the Q8_0 GGUF of the same weights (+28%). Same behaviour: short grounded answers, tool calls when a fact needs a lookup, a diagnosis and a new route when a tool fails, no invented links, an honest "unverified" when everything is broken.

The model card, the eval numbers (eval v3: 175 held-out agent questions, two judges) and the full build story live in the main repo. This repo is the EXL3 format only, for NVIDIA GPUs through exllamav3 or TabbyAPI.

Quant

Format	EXL3 (QTIP trellis), made with exllamav3 convert_model
Weight bits	5.0 bpw, target hit at 5.03 mean over the language layers
Head / embeddings	6 bpw
Vision tower	stored unquantized (Javi is text-only)
Checkpoint	4.2 GB, single model.safetensors + quantization_config.json
Context	262,144 tokens native
Decode, 1x DGX Spark	~58 tok/s single stream vs 45.5 tok/s for the Q8_0 GGUF of the same weights (+28%), and vs ~21 tok/s bf16 under vLLM

The language model is quantized layer by layer with error feedback; per-layer reconstruction SQNR measured 35-36 dB at 5.0 bpw, which is near-lossless for a 4B.

Run it

Send "chat_template_kwargs": {"enable_thinking": false} and sample at temp 0.2, top-p 0.8, top-k 20 so tool calls stay well formed (same as the bf16 and GGUF cards).

TabbyAPI (inference_library: exllamav3), point the model dir at this repo.

exllamav3 directly:

from exllamav3 import Config, Model, Cache, Tokenizer, Generator

config = Config.from_directory("/path/to/Javi0.3-4B-EXL3-5.0bpw")
tokenizer = Tokenizer.from_config(config)
model = Model.from_config(config)
cache = Cache(model, max_num_tokens=8192)
model.load()
generator = Generator(model=model, cache=cache, tokenizer=tokenizer)

Measured on 1x DGX Spark (GB10)

• Single-stream decode ~58 tok/s at 5.0 bpw (256-token generations, exllamav3 loader) vs 45.5 tok/s for the Q8_0 GGUF of the same weights on the same box.
• Titanium smoke (the Javi0.1 acceptance case): answers the melting/boiling points with the Wikipedia citation intact. A version question with a search tool available produces a well-formed web_search call, not a version from memory.
• Behaviour beyond that is the main card's eval; a full eval v3 A/B of this exact checkpoint against the GGUF is pending. Treat the smoke results as a format sanity check, not a new scoreboard.

Related

• Javi0.3-4B — merged bf16, the model card and results
• Javi0.3-4B-GGUF — Q8_0 / Q4_K_M for llama.cpp, LM Studio, Ollama
• Javi0.3-2B · 2B-GGUF — the small one
• The Javi0.3 build article on X

License

Apache 2.0, inherited from Qwen3.5-4B.