--- license: apache-2.0 base_model: Qwen/Qwen2.5-0.5B-Instruct tags: - gguf - email - triage - ollama - lora - unsloth - cipher - voice-intent language: - en pipeline_tag: text-generation --- # Cipher Air Cipher Air is a LoRA fine-tune of [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct), trained on **every LLM-backed feature of a local-first email assistant**: email triage (importance/summary/category JSON), chat, daily-summary synthesis, draft reply, compose assist, and voice-command intent parsing — not just prompted for these tasks, actually trained on them. **Update:** added a 6th task, voice-command intent parsing (interpreting a dictated voice command + a candidate address list into a structured action). The original 5-task version had never seen this task and would confidently hallucinate a plausible-looking but entirely fake person/address when it couldn't resolve one — this retrain fixes that; see `Training` below and `generate_voice_intent.py` for the fix in detail. **Second update:** a security-relevant bug was found and fixed — the previous voice-intent retrain, done through several rounds of incremental continue-training, left this model occasionally complying with prompt-injection attempts embedded in email content on the draft-reply/chat tasks (echoing a fake "security verification" request for banking details back into a drafted reply). Rather than patch that narrowly, this model was retrained fresh from the base in one pass on a properly-rebalanced dataset with much richer injection coverage across every task, not just triage. Verified through repeated adversarial testing against the actual calling application's code path (not just raw single-shot completions): no compliance with any credential/wire-transfer/data-exfiltration injection attempt across multiple test rounds. See `Training` below. It's the middle of the three **Cipher** tiers (`cipher-nano` / `cipher-air` / `cipher-pro`) — a balanced default at 40% of `cipher-pro`'s disk size and 3x the throughput. Cipher is the local-model engine for an unreleased larger email-assistant project — that project isn't public yet, but these weights, the training code, the eval script, and all five dataset generators are fully open now, in this repo. ## Why this exists Most email triage today means sending your inbox to a third-party API. Cipher runs entirely on your own hardware via [Ollama](https://ollama.com) — nothing about your email ever leaves your machine. ## What's in this repo - `cipher-air.Q4_K_M.gguf` — the model weights, ready for Ollama - `Modelfile` — the exact Ollama Modelfile (system prompt + inference params) used in training/eval - `train_cipher_air.py` / `export_gguf_cipher_air.py` — the exact scripts used to produce this model (Unsloth LoRA on the base model above) - `generate2.py`, `generate_chat.py`, `generate_daily_summary.py`, `generate_draft_reply.py`, `generate_compose.py`, `generate_voice_intent.py` — the six task-specific synthetic-data generators (produces the full multi-task training set) - `eval_voice_intent.py` — regression harness for the voice-intent task, including the exact hallucinated-address bug case as a required fixture - `eval_triage.py` / `eval_fixtures.json` — a standalone benchmark harness (no external dependencies beyond `httpx`/`pydantic`) reproducing the numbers below Everything needed to reproduce this model from scratch, or fine-tune your own variant, is in this repo — nothing here depends on an unreleased package. ## Benchmark Evaluated on a 29-fixture triage benchmark on an RTX 5070: | Model | Disk | Tok/s | JSON-valid | Category acc | Importance-in-band | Injection-safe | |---|---|---|---|---|---|---| | **cipher-air** | 398 MB | 507.8 | 100.0% | 69.0% | 79.3% | 100% | Honest caveat: cipher-air is the tightest-capacity tier of the three (only 8.8M of 502M params are trainable via LoRA), and it shows — of the three tiers it's the one most likely to occasionally misjudge whether something genuinely needs a reminder/action versus being routine. `cipher-pro` and `cipher-nano` both handle that nuance more reliably. Reproduce with: ```bash pip install -r requirements.txt python eval_triage.py --models cipher-air:latest --keep ``` ## Usage (Ollama) ```bash ollama create cipher-air -f Modelfile ``` Query it with grammar-constrained JSON output for reliable parsing: ```bash curl http://localhost:11434/api/chat -d '{ "model": "cipher-air", "messages": [ {"role": "system", "content": ""}, {"role": "user", "content": "From: alex@acme.com\nSubject: Q3 budget review\n\nBody:\nCan we sync before Friday?"} ], "format": "json", "options": {"temperature": 0.1} }' ``` ## Training - Base: `Qwen/Qwen2.5-0.5B-Instruct`, LoRA (r=16, alpha=32, all linear layers), 2 epochs - Data: ~4,800 triage examples (oversampled ~2x to ~60% of the final training mix — this size tier needed a stronger triage signal than the other two to hold onto exact JSON schema output while also learning four other task formats) + ~1,600-2,000 examples each for chat/daily-summary/draft-reply/compose, all matching production prompts exactly — generated by the five `generate_*.py` scripts in this repo - Framework: [Unsloth](https://github.com/unslothai/unsloth) + `trl.SFTTrainer` - Sequence packing was tried to speed up training (most examples are well under the 2048-token context window) — it crashed outright, an Unsloth/trl version incompatibility, not a quality tradeoff. Disabled. - Reproduce with `train_cipher_air.py` → `export_gguf_cipher_air.py` - Voice-intent retrain: added ~1,800 examples from `generate_voice_intent.py`, weighted heavily toward the no-match case (a spoken name with no corresponding candidate address — the model must return `null` rather than inventing one) and matching-with-distractors cases. On `eval_voice_intent.py`'s 5 fixtures, cipher-air went from 2/5 (including a fabricated address and an out-of-schema action) to **5/5**, the cleanest result of the three tiers on this task. - Full retrain (current version): rather than continue-training the voice-intent adapter further, this version is a fresh LoRA fine-tune from the base model on one consolidated, properly-balanced dataset covering all 6 tasks in a single pass — triage (~4,800, oversampled 3x to hold the ~60% mix ratio this tier needs), chat/daily-summary/draft- reply/compose/voice-intent (~1,800-2,500 each). `generate_draft_reply.py` and `generate_chat.py` both gained substantially heavier and more varied injection coverage (credential/wire-transfer/data-exfiltration attempts, not just one generic case) after live testing found the previous incremental-patch version could be induced into complying with an injected "security verification" request for banking details. `generate_chat.py` also gained an explicit "the question is about something with no connection to your email at all (weather, sports, etc.)" scenario category after finding a hallucination regression there. One clean training pass over the properly-balanced result, instead of a chain of narrow continue-trains, avoids the whack-a-mole pattern where each targeted fix risked nudging a different, previously-working case. ## A dead end worth knowing about We tried quantizing this model down further (Q3_K_M, Q2_K) hoping to shrink it toward `cipher-nano`'s size class. It barely helped (355MB / 339MB vs 398MB at Q4_K_M) — Qwen2.5's 151,936-token vocabulary embedding table dominates disk size and doesn't compress with weight quantization. If you're looking for something genuinely small, use `cipher-nano` instead (different base model, built specifically to solve this). ## License Apache 2.0, inherited from the base model. Weights, training code, and eval harness are fully open.