README.md

---
license: apache-2.0
base_model: Qwen/Qwen2.5-0.5B-Instruct
tags:
  - gguf
  - email
  - triage
  - ollama
  - lora
  - unsloth
  - cipher
  - voice-intent
language:
  - en
pipeline_tag: text-generation
---

# Cipher Air

Cipher Air is a LoRA fine-tune of [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct), trained on **every LLM-backed feature of a local-first email assistant**: email triage (importance/summary/category JSON), chat, daily-summary synthesis, draft reply, compose assist, and voice-command intent parsing — not just prompted for these tasks, actually trained on them.

**Update:** added a 6th task, voice-command intent parsing (interpreting a dictated voice
command + a candidate address list into a structured action). The original 5-task version
had never seen this task and would confidently hallucinate a plausible-looking but entirely
fake person/address when it couldn't resolve one — this retrain fixes that; see `Training`
below and `generate_voice_intent.py` for the fix in detail.

**Second update:** a security-relevant bug was found and fixed — the previous voice-intent
retrain, done through several rounds of incremental continue-training, left this model
occasionally complying with prompt-injection attempts embedded in email content on the
draft-reply/chat tasks (echoing a fake "security verification" request for banking details
back into a drafted reply). Rather than patch that narrowly, this model was retrained fresh
from the base in one pass on a properly-rebalanced dataset with much richer injection
coverage across every task, not just triage. Verified through repeated adversarial testing
against the actual calling application's code path (not just raw single-shot completions):
no compliance with any credential/wire-transfer/data-exfiltration injection attempt across
multiple test rounds. See `Training` below.

It's the middle of the three **Cipher** tiers (`cipher-nano` / `cipher-air` / `cipher-pro`) — a balanced default at 40% of `cipher-pro`'s disk size and 3x the throughput. Cipher is the local-model engine for an unreleased larger email-assistant project — that project isn't public yet, but these weights, the training code, the eval script, and all five dataset generators are fully open now, in this repo.

## Why this exists

Most email triage today means sending your inbox to a third-party API. Cipher runs entirely on your own hardware via [Ollama](https://ollama.com) — nothing about your email ever leaves your machine.

## What's in this repo

- `cipher-air.Q4_K_M.gguf` — the model weights, ready for Ollama
- `Modelfile` — the exact Ollama Modelfile (system prompt + inference params) used in training/eval
- `train_cipher_air.py` / `export_gguf_cipher_air.py` — the exact scripts used to produce this model (Unsloth LoRA on the base model above)
- `generate2.py`, `generate_chat.py`, `generate_daily_summary.py`, `generate_draft_reply.py`, `generate_compose.py`, `generate_voice_intent.py` — the six task-specific synthetic-data generators (produces the full multi-task training set)
- `eval_voice_intent.py` — regression harness for the voice-intent task, including the exact hallucinated-address bug case as a required fixture
- `eval_triage.py` / `eval_fixtures.json` — a standalone benchmark harness (no external dependencies beyond `httpx`/`pydantic`) reproducing the numbers below

Everything needed to reproduce this model from scratch, or fine-tune your own variant, is in this repo — nothing here depends on an unreleased package.

## Benchmark

Evaluated on a 29-fixture triage benchmark on an RTX 5070:

| Model | Disk | Tok/s | JSON-valid | Category acc | Importance-in-band | Injection-safe |
|---|---|---|---|---|---|---|
| **cipher-air** | 398 MB | 507.8 | 100.0% | 69.0% | 79.3% | 100% |

Honest caveat: cipher-air is the tightest-capacity tier of the three (only 8.8M of 502M
params are trainable via LoRA), and it shows — of the three tiers it's the one most likely
to occasionally misjudge whether something genuinely needs a reminder/action versus being
routine. `cipher-pro` and `cipher-nano` both handle that nuance more reliably. Reproduce
with:

```bash
pip install -r requirements.txt
python eval_triage.py --models cipher-air:latest --keep
```

## Usage (Ollama)

```bash
ollama create cipher-air -f Modelfile
```

Query it with grammar-constrained JSON output for reliable parsing:

```bash
curl http://localhost:11434/api/chat -d '{
  "model": "cipher-air",
  "messages": [
    {"role": "system", "content": "<system prompt from Modelfile>"},
    {"role": "user", "content": "From: alex@acme.com\nSubject: Q3 budget review\n\nBody:\nCan we sync before Friday?"}
  ],
  "format": "json",
  "options": {"temperature": 0.1}
}'
```

## Training

- Base: `Qwen/Qwen2.5-0.5B-Instruct`, LoRA (r=16, alpha=32, all linear layers), 2 epochs
- Data: ~4,800 triage examples (oversampled ~2x to ~60% of the final training mix — this
  size tier needed a stronger triage signal than the other two to hold onto exact JSON
  schema output while also learning four other task formats) + ~1,600-2,000 examples each
  for chat/daily-summary/draft-reply/compose, all matching production prompts exactly —
  generated by the five `generate_*.py` scripts in this repo
- Framework: [Unsloth](https://github.com/unslothai/unsloth) + `trl.SFTTrainer`
- Sequence packing was tried to speed up training (most examples are well under the
  2048-token context window) — it crashed outright, an Unsloth/trl version incompatibility,
  not a quality tradeoff. Disabled.
- Reproduce with `train_cipher_air.py` → `export_gguf_cipher_air.py`
- Voice-intent retrain: added ~1,800 examples from `generate_voice_intent.py`, weighted
  heavily toward the no-match case (a spoken name with no corresponding candidate address —
  the model must return `null` rather than inventing one) and matching-with-distractors
  cases. On `eval_voice_intent.py`'s 5 fixtures, cipher-air went from 2/5 (including a
  fabricated address and an out-of-schema action) to **5/5**, the cleanest result of the
  three tiers on this task.
- Full retrain (current version): rather than continue-training the voice-intent adapter
  further, this version is a fresh LoRA fine-tune from the base model on one consolidated,
  properly-balanced dataset covering all 6 tasks in a single pass — triage (~4,800,
  oversampled 3x to hold the ~60% mix ratio this tier needs), chat/daily-summary/draft-
  reply/compose/voice-intent (~1,800-2,500 each). `generate_draft_reply.py` and
  `generate_chat.py` both gained substantially heavier and more varied injection coverage
  (credential/wire-transfer/data-exfiltration attempts, not just one generic case) after
  live testing found the previous incremental-patch version could be induced into
  complying with an injected "security verification" request for banking details.
  `generate_chat.py` also gained an explicit "the question is about something with no
  connection to your email at all (weather, sports, etc.)" scenario category after finding
  a hallucination regression there. One clean training pass over the properly-balanced
  result, instead of a chain of narrow continue-trains, avoids the whack-a-mole pattern
  where each targeted fix risked nudging a different, previously-working case.

## A dead end worth knowing about

We tried quantizing this model down further (Q3_K_M, Q2_K) hoping to shrink it toward `cipher-nano`'s size class. It barely helped (355MB / 339MB vs 398MB at Q4_K_M) — Qwen2.5's 151,936-token vocabulary embedding table dominates disk size and doesn't compress with weight quantization. If you're looking for something genuinely small, use `cipher-nano` instead (different base model, built specifically to solve this).

## License

Apache 2.0, inherited from the base model. Weights, training code, and eval harness are fully open.