README.md

---
license: llama3.2
base_model: unsloth/Llama-3.2-3B-Instruct
library_name: peft
tags:
  - lora
  - peft
  - email-triage
  - tool-calling
  - text-classification
pipeline_tag: text-generation
---

# Email-Triage LoRA (Llama-3.2-3B-Instruct)

A LoRA adapter (`r=16`) fine-tuned via Behavioral Cloning to triage corporate emails by
selecting one of three tools — `route_to_human`, `auto_reply`, or `ask_for_clarification` —
returned as a strict JSON tool call.

- **Base model:** `unsloth/Llama-3.2-3B-Instruct`
- **Method:** Behavioral Cloning (Unsloth + Hugging Face TRL), 60 steps
- **Adapter:** LoRA, `r=16`
- **Project:** [Enterprise Email Triage Simulator](https://github.com/vaishali-strategy/email-triage)
- **Live demo (Gemini-backed UI):** https://huggingface.co/spaces/Proteinrequired/enterprise-email-triage-v2

## Results — held-out tool-selection accuracy

Evaluated on a **21-email held-out test set** (canonical 100-email dataset, intent-stratified
80/20 split, seed 42; BC trained only on the 79 train emails). Metric = did the policy pick the
reward system's optimal tool for each email.

| Policy | Held-out accuracy (N=21) |
| :--- | :---: |
| Random choice | 33.0% |
| Always `route_to_human` | 47.6% |
| Rule-based heuristic | **76.2%** (16/21) |
| **This adapter** | **71.4%** (15/21) |

The adapter is **perfect on every `route_to_human` intent** but **over-escalates** routine
(`auto_reply`) and ambiguous (`ask_for_clarification`) mail — a known Behavioral-Cloning artifact
(trained on reward-positive rollouts skewed toward escalation). It lands within one example of a
hand-tuned rule baseline and far above the random / always-escalate floors. Reproduce with
[`run_trained_eval.py`](https://github.com/vaishali-strategy/email-triage/blob/v2-rebuild/run_trained_eval.py).

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "unsloth/Llama-3.2-3B-Instruct"
adapter = "Proteinrequired/email-triage-lora"

tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, device_map="auto")
model = PeftModel.from_pretrained(model, adapter)
```

The model expects the system/user prompt format defined in
[`run_trained_eval.py`](https://github.com/vaishali-strategy/email-triage/blob/v2-rebuild/run_trained_eval.py)
and responds with a JSON object: `{"tool": "<route_to_human|auto_reply|ask_for_clarification>"}`.

## Limitations

- Trained on synthetic corporate emails; not validated on real inboxes or other domains.
- 3B model — needs a GPU for low-latency inference. The live demo uses Gemini via API for
  CPU-only hosting; this adapter is the project's open-weights research artifact.
- Optimizes the project's reward policy; not a general-purpose safety/abuse classifier.