README.md

---
license: llama3.1
base_model: mannix/llama3.1-8b-abliterated
library_name: peft
tags:
  - lora
  - tool-routing
  - karin
  - llama3.1
  - on-device
  - voice-assistant
  - jetson
pipeline_tag: text-generation
---

# Karin routing LoRA — iter-3

LoRA adapter that fine-tunes `mannix/llama3.1-8b-abliterated` for tool
routing in [Karin](https://github.com/kaminglui/Karin), an on-device
voice assistant running on NVIDIA Jetson Orin Nano 8 GB. This is the
production adapter — applied on top of the mannix abliteration via
Ollama's `ADAPTER` directive.

## Files

- **`karin-lora.gguf`** — 41 MB GGUF of the LoRA adapter. Drop-in for
  Ollama (`ADAPTER ./karin-lora.gguf` in a Modelfile) or llama.cpp
  (`--lora ./karin-lora.gguf`). Built at iter-3 / `run_0ac17bc7`.

## Performance

On Karin's 135-case held-out tool-routing eval (see
[`sft/eval_cases_novel.yaml`](https://github.com/kaminglui/Karin/blob/main/sft/eval_cases_novel.yaml)):

| Configuration | Routing | Reply | Tool-output use |
|---|---|---|---|
| Base mannix (no LoRA) | ~57% | — | — |
| This LoRA alone (iter-3) | 71.1% | ~66% | — |
| **This LoRA + Karin runtime layer (production default)** | **93.3%** | **91.9%** | **59.2%** |

The runtime layer (Phase-0 classifier patches, under-fire rescue,
two-phase compose, L8 reply scrubs) lives in the Karin repo and
contributes ~22 pp of the routing gains. See
[docs/routing-pipeline.md](https://github.com/kaminglui/Karin/blob/main/docs/routing-pipeline.md)
for the full pipeline breakdown.

Four subsequent training iterations (iter-4, 5, 6, 7) regressed on the
same eval and were all rolled back. Iter-3 remains the production base.
See [docs/](https://github.com/kaminglui/Karin/tree/main/docs) for the
per-iteration post-mortems.

## Training

- **Base model (trained against):** `mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated`
- **Base model (deployed against):** `mannix/llama3.1-8b-abliterated:tools-q4_k_m`
  (same weights, mannix re-applies the abliteration with a `tools` template)
- **Training data:** 294 SFT rows from Karin's phrase library + 40 DPO pairs
- **Hyperparameters (anti-overfit, kept across every iteration):**
  - `lora_r=8`, `lora_alpha=32`, `lora_dropout=0.1`
  - `sft_lr=1e-4`, `weight_decay=0.01`
  - `sft_epochs=2`, `max_seq_length=3072`
  - Cosine LR + 10% eval split + early stopping (patience 3)
- **Notebook:** [`sft/colab_sft.ipynb`](https://github.com/kaminglui/Karin/blob/main/sft/colab_sft.ipynb)

## Deployment

With Ollama already serving `mannix/llama3.1-8b-abliterated:tools-q4_k_m`
on the Jetson:

```bash
# 1. Fetch the adapter
hf download kaminglui/karin-lora karin-lora.gguf --local-dir .

# 2. Wrap in a Modelfile on top of the mannix base
ollama show mannix/llama3.1-8b-abliterated:tools-q4_k_m --modelfile > Modelfile
echo 'ADAPTER ./karin-lora.gguf' >> Modelfile
ollama create karin-tuned -f Modelfile

# 3. Point Karin at it (in deploy/.env)
# KARIN_LLM_MODEL=karin-tuned:latest
```

## Scope & limitations

- Trained on Karin's specific tool set (14 tools: weather, news, wiki,
  math, schedule_reminder, find_places, web_search, update_memory,
  get_time, get_alerts, get_digest, graph, circuit, convert). Routing
  accuracy outside this tool set is not measured.
- English-only system prompt; the LoRA wasn't exposed to multilingual
  prompts during training.
- Runtime quality numbers (93.3% / 91.9% / 59.2%) are measured against
  the full Karin runtime layer, not the LoRA in isolation. Without the
  classifier patches, under-fire rescue, and reply scrubs, the LoRA
  alone scores ~71% routing.

## License & attribution

Built with Llama. This adapter is derivative of Meta Llama 3.1 8B
Instruct and inherits the [Llama 3.1 Community License](https://www.llama.com/llama3_1/license/).
See `NOTICE` for attribution and the Acceptable Use Policy.

## Citation

```bibtex
@software{karin_lora_iter3,
  author = {kaminglui},
  title  = {Karin routing LoRA — iter-3},
  year   = {2026},
  url    = {https://huggingface.co/kaminglui/karin-lora},
}
```