--- license: apache-2.0 base_model: Qwen/Qwen3-4B-Instruct-2507 tags: - gguf - email - triage - ollama - lora - unsloth - cipher - voice-intent language: - en pipeline_tag: text-generation --- # Cipher Pro Cipher Pro is a LoRA fine-tune of [Qwen/Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507), trained on **every LLM-backed feature of a local-first email assistant**: email triage (importance/summary/category JSON), chat, daily-summary synthesis, draft reply, compose assist, and voice-command intent parsing — not just prompted for these tasks, actually trained on them. **Update:** added a 6th task, voice-command intent parsing, across 4 retrain passes total — each one targeting a specific failure mode found in real end-to-end testing (not just raw model output) rather than broad re-training. See `Training` below for the full progression; current state on `eval_voice_intent.py`'s 5 fixtures, verified through the actual calling app's retry-and-fallback logic (not a single raw completion), is **5/5**. Also fixed in this update: a garbled-leading-token failure mode specific to the compose- assist task (a stray non-ASCII token before otherwise-correct output, unrelated to the ``/`` leak below) — caught by manual spot-checking since compose has no automated schema to catch it against, not by any benchmark number. **Second update — security fix:** live testing found this model would occasionally comply with a prompt-injection attempt embedded in email content on the draft-reply/chat tasks (echoing a fake "security verification" request for banking details back into a drafted reply). Fixed with a 5th training pass adding substantially richer injection coverage (credential-phishing, wire-transfer, and data-exfiltration attempts across both tasks, not just one generic case). A fresh, single-pass full retrain from the base model was also tried as an alternative to another incremental patch, on the theory that one clean pass on a properly-rebalanced dataset should be more robust than a long chain of narrow continue-trains — it wasn't: two different learning rates each fixed one known issue but introduced a *different* new one (one broke ~21% of triage fixtures even after retries; the other introduced a bizarre off-topic role-play hallucination on ~50% of a chat fixture, confirmed via repeated testing). The incrementally-built version — the one actually shipped here — didn't have either problem and was kept instead. See `Training` below for the full pass-by-pass history and the specific comparison numbers. It's the largest of the three **Cipher** tiers (`cipher-nano` / `cipher-air` / `cipher-pro`), and the strongest on structured-output accuracy — **100% category accuracy** on the triage benchmark below. Cipher is the local-model engine for an unreleased larger email-assistant project — that project isn't public yet, but these weights, the training code, the eval script, and all five dataset generators are fully open now, in this repo. ## Why this exists Most email triage today means sending your inbox to a third-party API. Cipher runs entirely on your own hardware via [Ollama](https://ollama.com) — nothing about your email ever leaves your machine. ## What's in this repo - `cipher-pro.Q4_K_M.gguf` — the model weights, ready for Ollama - `Modelfile` — the exact Ollama Modelfile (system prompt, explicit ChatML `TEMPLATE`, inference params) used in training/eval — **use `ollama create`, not `ollama pull hf.co/...`**, see the integration note below - `train_cipher_pro.py` / `export_gguf_cipher_pro.py` — the exact scripts used to produce this model (Unsloth LoRA on the base model above) - `generate2.py`, `generate_chat.py`, `generate_daily_summary.py`, `generate_draft_reply.py`, `generate_compose.py`, `generate_voice_intent.py` — the six task-specific synthetic-data generators (produces the full multi-task training set) - `eval_voice_intent.py` — regression harness for the voice-intent task - `eval_triage.py` / `eval_fixtures.json` — a standalone benchmark harness (no external dependencies beyond `httpx`/`pydantic`) reproducing the triage numbers below Everything needed to reproduce this model from scratch, or fine-tune your own variant, is in this repo — nothing here depends on an unreleased package. ## Benchmark Evaluated on a 29-fixture triage benchmark against the untuned base model, on an RTX 5070: | Model | Disk | Tok/s | JSON-valid (raw single-shot) | JSON-valid (through the app's retry logic) | Category acc | Importance-in-band | Injection-safe | |---|---|---|---|---|---|---|---| | **cipher-pro** | 2.5 GB | 171.2 | ~62-79% (see note above) | **100.0% (29/29)** | 86.2% | 82.8% | 100% | Reproduce with: ```bash pip install -r requirements.txt python eval_triage.py --models cipher-pro:latest --keep ``` ## Integration note: chat template Qwen3's chat template isn't reliably auto-detected from the exported GGUF by Ollama (confirmed live — `ollama show --modelfile` fell back to a raw passthrough template with no role formatting, causing the model to leak stray ``/`` closing tags before its JSON output). The included `Modelfile` sets an explicit ChatML `TEMPLATE` matching what this model was actually trained on — don't rely on Ollama's autodetection or `ollama pull hf.co/...` (which generates its own default template and ignores the Modelfile committed in this repo). If you're integrating this into your own app rather than using Ollama, `llama-server` (llama.cpp's own server binary) handles Qwen3's real chat template correctly on its own — verified directly, no override needed there. Even with the correct template, a small residual fraction of completions may still leak a stray reasoning/tool-call tag before the JSON (Qwen3's own pretraining bakes in tool-calling habits that a LoRA adapter — 0.81% of this model's parameters — can't fully suppress). If you're parsing structured output, strip any leading ``/``/``/`` run before `json.loads()` — see `strip_leading_reasoning_tags()` in Grimoire's own `llm_client.py` for the reference implementation. The same underlying habit occasionally surfaces on free-text tasks (draft-reply/chat/compose) too, which have no JSON schema to validate against — sometimes as a fully garbled/empty response, sometimes as just a short garbled token prefixed onto an otherwise-correct reply. Both are worth retrying against rather than surfacing to a user as-is; see `_looks_garbled()` and the retry loop in `chat()` in the same reference file. This model tests clean through that retry path — 29/29 on the triage benchmark and 5/5 on compose (normal + injection) — but a raw single-shot call without the retry logic will occasionally hit one of these two patterns; don't skip the retry wrapper if you're integrating this model directly. ## Usage (Ollama) ```bash ollama create cipher-pro -f Modelfile ``` Query it with grammar-constrained JSON output for reliable parsing: ```bash curl http://localhost:11434/api/chat -d '{ "model": "cipher-pro", "messages": [ {"role": "system", "content": ""}, {"role": "user", "content": "From: alex@acme.com\nSubject: Q3 budget review\n\nBody:\nCan we sync before Friday?"} ], "format": "json", "options": {"temperature": 0.1} }' ``` ## Training - Base: `Qwen/Qwen3-4B-Instruct-2507`, LoRA (r=16, alpha=32, all linear layers), 2 epochs - Data: ~4,800 triage examples + ~1,600-2,000 examples each for chat/daily-summary/draft-reply/compose (~13,000 total, triage oversampled), all matching Grimoire's exact production prompts — generated by the five `generate_*.py` scripts in this repo - Framework: [Unsloth](https://github.com/unslothai/unsloth) + `trl.SFTTrainer` - Sequence packing (`trl.SFTConfig(packing=True)`) was tried to speed up training given most examples are well under the 2048-token context window — it crashed outright (`ValueError: Expected input batch_size (2048) to match target batch_size (3636)`, an Unsloth fused-loss/trl packing-collator incompatibility in this exact library version pairing), not a quality tradeoff. Disabled. - Reproduce with `train_cipher_pro.py` → `export_gguf_cipher_pro.py` - Voice-intent retrain (4 passes, each continuing LoRA training from the previous pass's adapter rather than starting over): 1. ~1,800 voice-intent examples added to the mix — fixed the core hallucinated-address bug, but introduced two new rule-violations (leaking a resolved recipient into `search_email` queries; dropping a valid address match under an injection attempt). 2. ~2,800 examples targeting both failure modes specifically (heavier weight on search-with-a-real-match-that-must-stay-null and injection-with-a-real-match cases) — both fixed, but a 3rd failure mode surfaced: the `unknown` action (unsupported requests like "forward this to my team") sometimes degenerated into an empty, repeating `` loop instead of valid JSON. 3. A small (~1,500 example) continue-train pass — mostly expanded, more diverse `unknown` phrasing plus a light replay slice of everything else to avoid regressing what already worked — fixed the degenerate-loop case. At this point raw single-shot eval showed 4/5; testing through the actual calling app's `chat_json` (which retries with the validation error fed back, and falls back to `action: unknown` if all retries still fail) showed this was already a real 5/5 in practice — the raw eval script's single request just didn't reflect the app's actual behavior. 4. A final ~4,200 example pass fixed the separately-discovered compose garbled-leading- token issue (see the note above), continuing from pass 3's adapter with the full `unknown`-fixup data included in the replay slice specifically so that fix wouldn't regress. Re-verified 5/5 on voice-intent afterward, no regression on triage/daily- summary/chat. 5. A ~5,200 example pass targeting the credential-injection compliance bug (see the security update above), continuing from pass 4's adapter with light general replay plus the full pass-3 `unknown`-fixup data re-included, so neither earlier fix would erode. This is the version actually shipped in this repo. 6. (Not shipped, kept here for the record.) Two single-pass full retrains from the base model, differing only in learning rate (2e-4 default vs. 1e-4), were tried afterward to see if one clean pass on a properly-rebalanced dataset would be more robust than the chain of narrow patches above. Verified through the real calling app's retry logic on all 29 triage fixtures: the default-LR version scored 23/29 (six fixtures failed all 3 retry attempts, every time) against pass 5's clean 29/29 on the identical test. The lower-LR version fixed that (29/29) but introduced a new, different failure — an off-topic role-play hallucination on a basic chat-recall question, reproduced on 2 of 4 independent runs, never observed on pass 5. Neither full-retrain attempt beat the incrementally-built version on every axis simultaneously, so pass 5 was kept as the shipped model. ## License Apache 2.0, inherited from the base model. Weights, training code, and eval harness are fully open.