README.md

---
license: apache-2.0
base_model: h2oai/h2o-danube3-500m-chat
tags:
  - gguf
  - email
  - triage
  - ollama
  - full-fine-tune
  - unsloth
  - cipher
  - edge
  - voice-intent
language:
  - en
pipeline_tag: text-generation
---

# Cipher Nano

Cipher Nano is a full-parameter fine-tune of [h2oai/h2o-danube3-500m-chat](https://huggingface.co/h2oai/h2o-danube3-500m-chat), trained on **every LLM-backed feature of a local-first email assistant**: email triage (importance/summary/category JSON), chat, daily-summary synthesis, draft reply, compose assist, and voice-command intent parsing — not just prompted for these tasks, actually trained on them.

**Update:** added a 6th task, voice-command intent parsing. Nano's smaller full-fine-tune
capacity means it still sometimes substitutes a plausible-looking wrong address instead of
copying the real one verbatim from the candidate list — 3 rounds of targeted oversampling
narrowed but didn't eliminate this. The actual fix ended up being architectural, not a
bigger model: the calling app never trusts an LLM-supplied email address at all anymore,
re-deriving it deterministically in code from the model's extracted name against the real
candidate list instead. That makes this habit irrelevant regardless of which Cipher tier is
active — see `generate_voice_intent.py`'s training notes and `eval_voice_intent.py` for the
full detail.

**Second update:** an incremental continue-training round aimed at voice-intent (done on top
of the update above) turned out to have a side effect elsewhere: manual testing found this
model would occasionally fabricate a specific false detail not present in its given context
(e.g. inventing a person's surname that was never mentioned) on the chat task. Rather than
patch that narrowly on top of an already-long chain of incremental updates, this model was
retrained fresh from the base in one pass on a properly-rebalanced dataset covering all 6
tasks together, with substantially richer prompt-injection coverage across draft-reply and
chat (credential-phishing, wire-transfer, and data-exfiltration attempts, not just one
generic case) and an explicit "the question has nothing to do with your email at all"
scenario category to close the specific hallucination gap that was found. Verified through
repeated testing against the real calling application's code path: no compliance with any
injection attempt, and the specific hallucination case now correctly reports the missing
information instead of inventing it.

It's the smallest of the three **Cipher** tiers (`cipher-nano` / `cipher-air` / `cipher-pro`), built specifically to run on modest hardware — **verified at 29.4 tok/s on a 10-year-old Intel i3-6100T with 8GB RAM, no GPU**, with accuracy holding essentially flat versus a high-end GPU. Cipher is the local-model engine for an unreleased larger email-assistant project — that project isn't public yet, but these weights, the training code, the eval script, and all five dataset generators are fully open now, in this repo.

## Why this exists

Most email triage today means sending your inbox to a third-party API. Cipher runs entirely on your own hardware via [Ollama](https://ollama.com) — nothing about your email ever leaves your machine. Cipher Nano exists so that "your own hardware" can mean literally any hardware, not just a machine with a GPU.

## Why Danube3-500M and not something smaller

We tried harder-to-shrink options first and hit real walls:

- **SmolLM2-135M/360M** (49K-token vocab) genuinely shrinks to 100-270MB, but its base pretraining wasn't strong enough to reliably learn category/importance calibration — tried LoRA vs. full fine-tune, epoch sweeps, and reshaped training data, and category accuracy stayed stuck in the 40-65% range regardless.
- **Qwen2.5-0.5B / Gemma3** have strong pretraining but 152K/256K-token vocabularies that dominate disk size and don't shrink with quantization (confirmed: Q2_K barely shrank vs Q4_K_M).
- **Danube3-500M** is the first candidate with *both* a small vocabulary (32K tokens) and pretraining strong enough for the task — category accuracy jumped to 70%+ on the first fine-tuning attempt, no extensive tuning needed. Its full-parameter fine-tune (100% of weights retrained, vs. `cipher-air`/`cipher-pro`'s LoRA) also gives it total plasticity to fully adopt the required output format, no competing base-model habit to fight against.

## Benchmark

Evaluated on a 29-fixture triage benchmark across five machines, GPU and CPU-only:

| Hardware | Tok/s | JSON-valid | Category acc | Importance-in-band |
|---|---|---|---|---|
| RTX 5070 (GPU) | 901.0 | 100% | 72.4% | 69.0% |
| RX 9070 XT 16GB (GPU) | 398.7 | 100% | 69.0% | 69.0% |
| Ryzen 9800X3D (CPU-only) | 143.0 | 100% | 72.4% | 72.4% |
| Core Ultra 9 285K (CPU-only) | 106.3 | 100% | 69.0% | 69.0% |
| **Intel i3-6100T, 8GB RAM (CPU-only, 10 years old)** | **29.4** | 100% | 69.0% | 72.4% |
| Multi-task retrain (chat/summary/draft/compose added) | 899.9 | 96.6% | 67.9% | 89.3% |
| Voice-intent retrain (6th task added) | 867.4 | 96.6% | 75.0% | 82.1% |
| Full retrain (current version, injection/hallucination hardened) | 106.7* | 100.0% | 82.8% | 79.3% |

\* Tok/s measured while the GPU was concurrently running an unrelated training job — not a
like-for-like comparison with the isolated numbers above; accuracy figures are unaffected.

Accuracy holds essentially flat across every machine tested — only throughput changes. Even the weakest machine here, a decade-old office PC with no GPU, does ~2 seconds per email, well within range for a background triage task. Reproduce with:

```bash
pip install -r requirements.txt
python eval_triage.py --models cipher-nano:latest --keep
```

## Usage (Ollama)

Danube3's chat template does **not** support a `system` role and uses non-ChatML turn markers — the included `Modelfile` sets an explicit `TEMPLATE` to match. Don't rely on Ollama's template autodetection with this base model (confirmed live: without an explicit `TEMPLATE`, Ollama couldn't resolve a working chat format for this GGUF at all).

```bash
ollama create cipher-nano -f Modelfile
```

Query it with grammar-constrained JSON output for reliable parsing:

```bash
curl http://localhost:11434/api/chat -d '{
  "model": "cipher-nano",
  "messages": [
    {"role": "system", "content": "<system prompt from Modelfile>"},
    {"role": "user", "content": "From: alex@acme.com\nSubject: Q3 budget review\n\nBody:\nCan we sync before Friday?"}
  ],
  "format": "json",
  "options": {"temperature": 0.1}
}'
```

If you're integrating this into your own app rather than using Ollama: `llama-server`
(llama.cpp's own server binary) handles this model's real chat template correctly on its
own — verified directly, no override needed there, only for Ollama's simpler
auto-detection.

## Training

- Base: `h2oai/h2o-danube3-500m-chat` (plain `LlamaForCausalLM`, 32,000-token vocab), **full fine-tune** (100% of parameters, not LoRA), 3 epochs
- Data: ~4,800 triage examples + ~1,600-2,000 examples each for chat/daily-summary/draft-reply/compose, all matching production prompts exactly — generated by the five `generate_*.py` scripts in this repo; training text reformatted to Danube3's native `<|prompt|>`/`<|answer|>` turn format since it has no `system`-role support (see `train_cipher_nano.py`)
- Framework: [Unsloth](https://github.com/unslothai/unsloth) + `trl.SFTTrainer`
- Sequence packing was tried to speed up training (most examples are well under the
  2048-token context window) — it crashed outright, an Unsloth/trl version incompatibility,
  not a quality tradeoff. Disabled.
- Reproduce with `train_cipher_nano.py` → `export_gguf_cipher_nano.py`
- Voice-intent retrain: added `generate_voice_intent.py`'s data (weighted heavily toward
  the no-match case and exact-copy-under-distractor-pressure scenarios) to the mix across 3
  rounds of oversampling. Nano's smaller full-fine-tune capacity meant this narrowed but
  never fully eliminated one specific habit — substituting a different plausible-looking
  address instead of copying the real one verbatim — see the README intro above for how
  this ended up getting solved at the calling-app level instead. Reproduce/verify with
  `eval_voice_intent.py`.
- Full retrain (current version): one clean pass over the base model on a single
  consolidated, properly-balanced dataset covering all 6 tasks, rather than continuing to
  layer incremental patches on top of each other — done specifically to fix both the
  injection-compliance issue and the hallucination regression noted above without risking
  a whack-a-mole cycle where each narrow fix nudges a different, previously-working case.
  Followed by one small, targeted continue-train pass (~700 examples, 2 epochs, lr=2e-5)
  reinforcing "report missing information plainly rather than inventing a plausible-
  sounding detail," specifically targeting the exact hallucination case found in testing.

## License

Apache 2.0, inherited from the base model. Weights, training code, and eval harness are fully open.