srock44/cipher-pro

Cipher Pro

Cipher Pro is a LoRA fine-tune of Qwen/Qwen3-4B-Instruct-2507, trained on every LLM-backed feature of a local-first email assistant: email triage (importance/summary/category JSON), chat, daily-summary synthesis, draft reply, compose assist, and voice-command intent parsing — not just prompted for these tasks, actually trained on them.

Update: added a 6th task, voice-command intent parsing, across 4 retrain passes total — each one targeting a specific failure mode found in real end-to-end testing (not just raw model output) rather than broad re-training. See Training below for the full progression; current state on eval_voice_intent.py's 5 fixtures, verified through the actual calling app's retry-and-fallback logic (not a single raw completion), is 5/5.

Also fixed in this update: a garbled-leading-token failure mode specific to the compose- assist task (a stray non-ASCII token before otherwise-correct output, unrelated to the <think>/<tool_call> leak below) — caught by manual spot-checking since compose has no automated schema to catch it against, not by any benchmark number.

Second update — security fix: live testing found this model would occasionally comply with a prompt-injection attempt embedded in email content on the draft-reply/chat tasks (echoing a fake "security verification" request for banking details back into a drafted reply). Fixed with a 5th training pass adding substantially richer injection coverage (credential-phishing, wire-transfer, and data-exfiltration attempts across both tasks, not just one generic case). A fresh, single-pass full retrain from the base model was also tried as an alternative to another incremental patch, on the theory that one clean pass on a properly-rebalanced dataset should be more robust than a long chain of narrow continue-trains — it wasn't: two different learning rates each fixed one known issue but introduced a different new one (one broke ~21% of triage fixtures even after retries; the other introduced a bizarre off-topic role-play hallucination on ~50% of a chat fixture, confirmed via repeated testing). The incrementally-built version — the one actually shipped here — didn't have either problem and was kept instead. See Training below for the full pass-by-pass history and the specific comparison numbers.

It's the largest of the three Cipher tiers (cipher-nano / cipher-air / cipher-pro), and the strongest on structured-output accuracy — 100% category accuracy on the triage benchmark below. Cipher is the local-model engine for an unreleased larger email-assistant project — that project isn't public yet, but these weights, the training code, the eval script, and all five dataset generators are fully open now, in this repo.

Why this exists

Most email triage today means sending your inbox to a third-party API. Cipher runs entirely on your own hardware via Ollama — nothing about your email ever leaves your machine.

What's in this repo

• cipher-pro.Q4_K_M.gguf — the model weights, ready for Ollama
• Modelfile — the exact Ollama Modelfile (system prompt, explicit ChatML TEMPLATE, inference params) used in training/eval — use ollama create, not ollama pull hf.co/..., see the integration note below
• train_cipher_pro.py / export_gguf_cipher_pro.py — the exact scripts used to produce this model (Unsloth LoRA on the base model above)
• generate2.py, generate_chat.py, generate_daily_summary.py, generate_draft_reply.py, generate_compose.py, generate_voice_intent.py — the six task-specific synthetic-data generators (produces the full multi-task training set)
• eval_voice_intent.py — regression harness for the voice-intent task
• eval_triage.py / eval_fixtures.json — a standalone benchmark harness (no external dependencies beyond httpx/pydantic) reproducing the triage numbers below

Everything needed to reproduce this model from scratch, or fine-tune your own variant, is in this repo — nothing here depends on an unreleased package.

Benchmark

Evaluated on a 29-fixture triage benchmark against the untuned base model, on an RTX 5070:

Model	Disk	Tok/s	JSON-valid (raw single-shot)	JSON-valid (through the app's retry logic)	Category acc	Importance-in-band	Injection-safe
cipher-pro	2.5 GB	171.2	~62-79% (see note above)	100.0% (29/29)	86.2%	82.8%	100%

Reproduce with:

pip install -r requirements.txt
python eval_triage.py --models cipher-pro:latest --keep

Integration note: chat template

Qwen3's chat template isn't reliably auto-detected from the exported GGUF by Ollama (confirmed live — ollama show --modelfile fell back to a raw passthrough template with no role formatting, causing the model to leak stray </think>/</tool_call> closing tags before its JSON output). The included Modelfile sets an explicit ChatML TEMPLATE matching what this model was actually trained on — don't rely on Ollama's autodetection or ollama pull hf.co/... (which generates its own default template and ignores the Modelfile committed in this repo). If you're integrating this into your own app rather than using Ollama, llama-server (llama.cpp's own server binary) handles Qwen3's real chat template correctly on its own — verified directly, no override needed there.

Even with the correct template, a small residual fraction of completions may still leak a stray reasoning/tool-call tag before the JSON (Qwen3's own pretraining bakes in tool-calling habits that a LoRA adapter — 0.81% of this model's parameters — can't fully suppress). If you're parsing structured output, strip any leading </think>/<think>/</tool_call>/<tool_call> run before json.loads() — see strip_leading_reasoning_tags() in Grimoire's own llm_client.py for the reference implementation.

The same underlying habit occasionally surfaces on free-text tasks (draft-reply/chat/compose) too, which have no JSON schema to validate against — sometimes as a fully garbled/empty response, sometimes as just a short garbled token prefixed onto an otherwise-correct reply. Both are worth retrying against rather than surfacing to a user as-is; see _looks_garbled() and the retry loop in chat() in the same reference file. This model tests clean through that retry path — 29/29 on the triage benchmark and 5/5 on compose (normal + injection) — but a raw single-shot call without the retry logic will occasionally hit one of these two patterns; don't skip the retry wrapper if you're integrating this model directly.

Usage (Ollama)

ollama create cipher-pro -f Modelfile

Query it with grammar-constrained JSON output for reliable parsing:

curl http://localhost:11434/api/chat -d '{
  "model": "cipher-pro",
  "messages": [
    {"role": "system", "content": "<system prompt from Modelfile>"},
    {"role": "user", "content": "From: alex@acme.com\nSubject: Q3 budget review\n\nBody:\nCan we sync before Friday?"}
  ],
  "format": "json",
  "options": {"temperature": 0.1}
}'

Training

• Base: Qwen/Qwen3-4B-Instruct-2507, LoRA (r=16, alpha=32, all linear layers), 2 epochs
• Data: 4,800 triage examples + ~1,600-2,000 examples each for chat/daily-summary/draft-reply/compose (13,000 total, triage oversampled), all matching Grimoire's exact production prompts — generated by the five generate_*.py scripts in this repo
• Framework: Unsloth + trl.SFTTrainer
• Sequence packing (trl.SFTConfig(packing=True)) was tried to speed up training given most examples are well under the 2048-token context window — it crashed outright (ValueError: Expected input batch_size (2048) to match target batch_size (3636), an Unsloth fused-loss/trl packing-collator incompatibility in this exact library version pairing), not a quality tradeoff. Disabled.
• Reproduce with train_cipher_pro.py → export_gguf_cipher_pro.py
• Voice-intent retrain (4 passes, each continuing LoRA training from the previous pass's adapter rather than starting over):
  1. ~1,800 voice-intent examples added to the mix — fixed the core hallucinated-address bug, but introduced two new rule-violations (leaking a resolved recipient into search_email queries; dropping a valid address match under an injection attempt).
  2. ~2,800 examples targeting both failure modes specifically (heavier weight on search-with-a-real-match-that-must-stay-null and injection-with-a-real-match cases) — both fixed, but a 3rd failure mode surfaced: the unknown action (unsupported requests like "forward this to my team") sometimes degenerated into an empty, repeating <tool_call> loop instead of valid JSON.
  3. A small (~1,500 example) continue-train pass — mostly expanded, more diverse unknown phrasing plus a light replay slice of everything else to avoid regressing what already worked — fixed the degenerate-loop case. At this point raw single-shot eval showed 4/5; testing through the actual calling app's chat_json (which retries with the validation error fed back, and falls back to action: unknown if all retries still fail) showed this was already a real 5/5 in practice — the raw eval script's single request just didn't reflect the app's actual behavior.
  4. A final ~4,200 example pass fixed the separately-discovered compose garbled-leading- token issue (see the note above), continuing from pass 3's adapter with the full unknown-fixup data included in the replay slice specifically so that fix wouldn't regress. Re-verified 5/5 on voice-intent afterward, no regression on triage/daily- summary/chat.
  5. A ~5,200 example pass targeting the credential-injection compliance bug (see the security update above), continuing from pass 4's adapter with light general replay plus the full pass-3 unknown-fixup data re-included, so neither earlier fix would erode. This is the version actually shipped in this repo.
  6. (Not shipped, kept here for the record.) Two single-pass full retrains from the base model, differing only in learning rate (2e-4 default vs. 1e-4), were tried afterward to see if one clean pass on a properly-rebalanced dataset would be more robust than the chain of narrow patches above. Verified through the real calling app's retry logic on all 29 triage fixtures: the default-LR version scored 23/29 (six fixtures failed all 3 retry attempts, every time) against pass 5's clean 29/29 on the identical test. The lower-LR version fixed that (29/29) but introduced a new, different failure — an off-topic role-play hallucination on a basic chat-recall question, reproduced on 2 of 4 independent runs, never observed on pass 5. Neither full-retrain attempt beat the incrementally-built version on every axis simultaneously, so pass 5 was kept as the shipped model.

License

Apache 2.0, inherited from the base model. Weights, training code, and eval harness are fully open.