distill-E4B-it-4-bit-MLX

MLX mixed-precision 4-bit (text-only) build of samuelfaj/distill-E4B-it-4-bit, for fast, accurate local inference on Apple Silicon. Distilled for CLI / command-output compression → single-line {"response": <str>, "confidence": <int 0-100>}.

✅ Recommended build. Upgraded from plain 4-bit to a Q4_K_M-style mixed recipe (lm_head + sensitive v_proj/down_proj at 6-bit, rest 4-bit). Pair it with the constrained decoding snippet below for near-perfect format compliance.

Details

Field	Value
Base	google/gemma-4-E4B-it
Adapter	QLoRA r=64, α=128 (published root, sha256 a229d12c…)
Quantization	mixed mixed_4_6 (4-bit base, 6-bit lm_head + sensitive layers) — 4.65 bits/weight
Group size	64
Size on disk	~4.0 GB
Peak memory	~4.4 GB

Evaluation

Set: eval_gold_v1 — 130 held-out CLI-compression cases. Decode: greedy.

Build	P1 accuracy	JSON compliance	json_extraction
plain 4-bit (old)	64.6%	89.2%	37.5%
mixed 4-bit	67.7%	92.3%	37.5%
mixed 4-bit + constrained decode (this)	69.2%	99.2%	56.2%
fp16 reference (ceiling)	72.3%	98.5%	75.0%

Mixed precision + constrained decoding recovers ~60% of the quantization gap to fp16 and pushes format compliance above the fp16 baseline (+10 pts vs plain 4-bit). Confidence calibration is unchanged (ECE ≈ 0.29) — treat confidence as a soft signal.

Usage (with constrained decoding)

The model targets a strict {"response","confidence"} JSON contract. Prefilling the opening of that object forces the schema and suppresses any reasoning preamble — this is what drives JSON compliance from 92% → 99%.

pip install mlx-lm

import json
from mlx_lm import load, generate

model, tok = load("samuelfaj/distill-E4B-it-4-bit-MLX")

def ask(prompt: str) -> dict:
    text = tok.apply_chat_template(
        [{"role": "user", "content": prompt}],
        add_generation_prompt=True, tokenize=False,
    )
    prefix = '{"response": "'                       # constrained decode: pin the contract
    raw = prefix + generate(model, tok, prompt=text + prefix, max_tokens=800, verbose=False)
    return json.loads(raw)

print(ask('Did the tests pass? Return PASS or FAIL, then failing tests.\n\n'
          'PASS src/auth.test.ts\nFAIL src/queue.test.ts\n1 passed, 1 failed.'))
# {'response': 'FAIL src/queue.test.ts', 'confidence': 98}

Plain (unconstrained) decoding also works but is ~10 pts less format-compliant:

python -m mlx_lm generate --model samuelfaj/distill-E4B-it-4-bit-MLX \
  --prompt "Is this safe? SAFE/REVIEW/UNSAFE.\n\nDROP TABLE users;" --max-tokens 256

Other builds

3-bit (⚠️ degraded) · 2-bit (❌ broken) · comparison vs distill-1.7B