RoderickQiu/kev-4b-mlx-8bit

Kev-4B, merged and quantized to 8 bits for MLX

Kev-4B by Jared Palmer, with its LoRA adapter already merged into Qwen3.5-4B-Base and the result quantized to 8 bits, for Apple silicon (MLX).

This is not an official release of Kev. It was built with, and for, Qualm, a macOS app that uses Kev to tell a lecture from a feed. Kev's own repository and model card are the reference.

Why this exists

Kev-4B ships as a LoRA adapter plus a pointer head. To run it on a Mac, you download the 8.7 GB bf16 base, merge the adapter and, to fit next to everyday apps, quantize. This repository is the result of that, done once:

	Build it yourself	This repository
Download	9.0 GB (base + adapter)	4.5 GB
First start	~100 s, 16 GB memory peak	~11 s, 4.8 GB peak
Memory while answering	6–7 GB	6–7 GB

Measured on an M5 Pro with 24 GB, running macOS 27.

What's here

File	What it is
model.safetensors, config.json	The merged backbone, 8-bit (group size 64), in mlx-lm's format ("quantization" in the config)
head.pt	Kev's pointer head and calibration (temperature), unchanged from jaredpalmer/kev-4b
tokenizer files	Unchanged from jaredpalmer/kev-4b
provenance.json	The exact revisions it was built from, the method, and the SHA-256 of model.safetensors
LICENSE	Apache-2.0

How it was made: the changes from the originals

Built from jaredpalmer/kev-4b at revision 485ace8703592fcf405488b262449990824cfed1 and Qwen/Qwen3.5-4B-Base at revision 1001bb4d826a52d1f399e183466143f4da7b741b, with Kev's code at commit 08ab0b87d27cb5577a3b371ad7ed4e4686b0502b:

1. Merged. The base was loaded in bf16 with mlx-lm, and the LoRA adapter was merged in fp32 on the CPU (kev.mlx_model.merge_lora: W + B·A·α/r), rounded once to bf16. This is Kev's own MLX loading path.
2. Quantized. mlx.nn.quantize with bits=8, group_size=64, on every Linear and Embedding layer whose input width divides by 64.
3. Kept as is. The pointer head in head.pt stays fp32. Nothing was retrained.

Quality

On a set of 119 labelled web pages (short video, feeds, livestreams, social media and others, from Qualm's trials):

• This file against quantizing at load time: identical answers, a difference of 0.0 on every question.
• This file against Kev-4B in bf16:
  • The same ROC AUC on every rule.
  • The same page-kind answer on 119 of 119 pages.
  • Per-question probabilities within 0.012–0.038, and 0.053 for the "sensitive" question.

4-bit was tried and not published: it moved social-media recall from 0.95 to 0.85.

Use

This isn't a chat model. Kev answers typed questions (a choice, a yes/no, a score) with calibrated probabilities, through its pointer head. The weights load with Kev's MLX model class. Take the tokenizer and head metadata from the Kev checkpoint:

from pathlib import Path
from huggingface_hub import snapshot_download
from kev.checkpoint import Checkpoint
from kev.mlx_model import MLXDecisionModel
from kev.model import PointerHead, load_tokenizer, pad_id

weights = Path(snapshot_download("RoderickQiu/kev-4b-mlx-8bit", allow_patterns=["config.json", "model.safetensors"]))
ck = Checkpoint("jaredpalmer/kev-4b")  # metadata, tokenizer and head; the base weights aren't needed
tok = load_tokenizer(ck.meta.base, revision=ck.meta.base_revision)
model = MLXDecisionModel(weights, pad_id(tok), head_dim=ck.meta.head_dim)
# The head is sized from the embedding width, which is packed in quantized weights: size it from the real one.
model.head = PointerHead(model.text.embed_tokens.dims, dp=ck.meta.head_dim).eval()
model.head.load_state_dict(ck.meta.head)
model.head.temperature = ck.meta.temperature

model then works wherever Kev's DecisionModel does, for example behind kev.serve's /v1/systemone endpoint.

Qualm downloads it by itself on the first start. It checks provenance.json against the Kev checkpoint it runs and the file's SHA-256, and builds the weights locally when they don't match.

License and credits

Apache-2.0, like both originals:

• Kev-4B: the adapter, head and tokenizer, by Jared Palmer. jaredpalmer/kev-4b, github.com/jaredpalmer/kev. Apache-2.0.
• Qwen3.5-4B-Base: the backbone, by the Qwen team, Alibaba Cloud. Qwen/Qwen3.5-4B-Base. Apache-2.0.

The modifications (merging and 8-bit quantization) are described above. All credit for the model belongs to its authors; any errors in this conversion belong to this repository.