singhabhishekkk/apprentice-4b-banking77

Apprentice 4B, Banking77 intents (LoRA adapter + classifier head)

A LoRA adapter for Qwen3.5-4B (4-bit, MLX) fine-tuned on the training split of Banking77, plus a logistic head that reads the intent off the model's hidden state at the last prompt token, layer 32, without decoding. Produced by Apprentice on an Apple-silicon Mac on 2026-11-05. Everything here is reproducible from the public benchmark repo: tasks/intent-classification.

Results on the full public test split (3,080 rows, 77 intents, exact match)

System	Accuracy	Who ran it
Head on this adapter, no decoding	93.6	Apprentice
This adapter's generated answer	91.2	Apprentice
Claude Opus 5, labels in the prompt, temperature 0	84.4	OpenRouter
Jev 1.13, Decisions API	81.0	OpenRouter, same post

Beside the Banking77 paper's own full-data numbers (Casanueva et al. 2020, Table 3): fine-tuned BERT-Large 93.66, USE+ConveRT 93.36, ConveRT 93.01, USE 92.81. The head sits 0.09 under their fine-tuned BERT-Large. Later papers report higher: DeepOpen (arXiv 2610.31104, October 2026) 94.1 with an encoder classifier; this is not a state-of-the-art claim. 460 of the test rows (14.9%) share 80% or more of their tokens with a training row; without them the head scores 93.1 and the generated answer 90.6.

Gated by its own confidence with a selective-risk bound (Geifman and El-Yaniv 2017) at a 5% target, the head keeps 88.8 to 90.6% of requests across three splits with 2.3 to 2.6% realised error among kept answers, measured on rows the threshold never saw (E3_head_sgr.md).

Files

• adapters.safetensors, adapter_config.json: the MLX LoRA adapter (16 layers, two epochs, batch 1, learning rate 1e-5, seed 42).
• head_L32.npz: the logistic head (coef 77 x 2560, intercept 77, scaler_mean, scaler_scale, labels in order). Apply to the final residual-stream hidden state after layer 32 at the last prompt token, standardise, then softmax. The file also carries adapter_sha256, the SHA-256 over the two adapter files (sorted relative path, then bytes), so a serving process can refuse a head that does not match its adapter. Refit from the recorded hidden states by export_head.py in the benchmark repo; it reproduces 93.6 on the test split.

Run it on a CPU with llama.cpp

The same adapter and a head refitted for llama.cpp's engine, measured 2026-11-05 in a Docker container limited to 4 CPUs on an Apple M3 Pro (arm64 Linux), llama.cpp server release 0.9.0, the public base unsloth/Qwen3.5-4B-GGUF at Q4_K_M:

	llama.cpp on 4 CPU cores	MLX on the Mac's GPU (above)
Head on layer 32, all 3,080 test rows	93.6	93.6
Generated answer, first 500 test rows	93.4	93.0 on the same rows
Head gated at the 5% selective-risk target, three splits	keeps 81.6 to 90.4% at 1.4 to 2.5% error	keeps 88.8 to 90.6% at 2.3 to 2.6%
One decision (24-token prompt, one pass, no decoding), p50	337 ms	63 ms

Files: adapter_llamacpp_f16.gguf (the adapter converted with llama.cpp's convert_lora_to_gguf.py, 16 MB, applied at run time with --lora, no merge) and head_L32_llamacpp.npz (the head refitted on this engine's states; same layout as head_L32.npz, but it reads the state after the final norm, which is what llama.cpp's embeddings route returns, so the two heads are not interchangeable). Commands, from the benchmark repo's tasks/intent-classification/routing/:

llama-server -m Qwen3.5-4B-Q4_K_M.gguf --lora adapter_llamacpp_f16.gguf --embeddings --pooling last -c 512 --cache-ram 0
python capture_llama.py --rows test.jsonl --stem llama32_test --layers 32     # raw post-norm states
python score_llama.py --rows test.jsonl --stem llama32_test_scores            # generated answers

truncate_gguf.py --head head_L32_llamacpp.npz folds the head into a copy of the base GGUF as cls.output, so the server returns one logit per intent; checked equal to the Python head on 100 rows (E15_head_in_file_check.json in the benchmark repo).

How it was trained

Through the product's own local path (runapprentice.local.train_mlx_local), on mlx-community/Qwen3.5-4B-OptiQ-4bit, two epochs over 7,002 of the 10,003 training rows (the product holds 30% out to score itself: 91.1 on those 3,001 rows). The head was then trained on all 10,003 training rows' hidden states, C chosen by 5-fold cross-validation on those rows only. No hyperparameter search, no prompt engineering. The same recipe on 20 examples per intent gave a generated answer of 81.9 and a head of 90.2; the gap was data.

Intended use and limits

Customer-support intent classification over the 77 Banking77 intents. One model family, one laptop, two epochs. The routing bound is a promise about traffic like the calibration rows; re-check on fresh labelled rows when your mix of requests changes.

Attribution

Built with Qwen (Apache-2.0). Modifications: LoRA adapter trained as above; classifier head added. Banking77 is CC BY 4.0 (PolyAI).