suraj10620/UnchaosLM

A small AI model that reads a work message and tells you three things:
how urgent it is, who needs to act, and what to do next.

_{The brain of Triage Copilot · runs on your Mac · nothing you feed it ever leaves your machine}

---

What it does

Work arrives from everywhere: a Slack mention, a ClickUp task, an email ticket. You give one of them to UnchaosLM:

SIGNAL TYPE: hiver_mention
CONVERSATION #6689 | MY NOTE 4 days ago: "@Sameer can you check the premium invoice logs for Sofia P.?"
STATUS: no response from teammate yet

And it answers, in a fixed format an app can display:

{
  "severity": "medium",
  "tier": "P3",
  "action_direction": "waiting_on_them",
  "owner": "Sameer",
  "next_step": "Nudge Sameer on the premium invoice logs for conversation #6689",
  "deadline_suggestion": "this_week",
  "escalation_flag": false
}

In plain words: not urgent, but you're waiting on Sameer — nudge him this week. You didn't have to re-read the thread.

Do that for every message in your day, and you get one sorted to-do list instead of five noisy inboxes.

Where it helps

Monday morning, 40 unread threads. Triage Copilot runs every new mention, task, and ticket through UnchaosLM. You open one queue sorted by what actually needs you — instead of five inboxes sorted by noise.

The escalation buried in a busy channel. "Acme's checkout is returning 500s" lands between two memes at 4pm. UnchaosLM flags it P1 with the escalation flag on and a concrete next step — join the escalation call — so it never depends on someone scrolling back.

The ask nobody answered. That's the Sameer example above: your own requests that quietly stalled come back as waiting on them — nudge this week, instead of dying in a thread.

"What's on fire today?" The same model also answers questions and takes actions on your queue. Ask in plain language and it lists your urgent items; follow up with "book an hour for the Acme thing" and it proposes a calendar block you approve.

Covering the shared inbox. "Who's sitting on SLA breaches?" — it checks breached and at-risk conversations across your shared inboxes before the weekly report would have caught them.

Three things it's careful about

Won't miss the big stuff	Won't make things up	Won't act without you
Anything touching a customer, a deadline promise, or a security issue gets flagged as urgent. When in doubt, it flags.	Names, links, and tasks come from your message — or not at all.	Calendar changes are proposals. You approve them.

Benchmarks

Measured on a held-out test set — 84 signals plus 15 agent tasks, none seen in training. The test set, the script, and every model's results are published in eval/, so you can rerun everything yourself. Every model ran locally with the same script and settings: the base it was finetuned from, its twice-as-big sibling, and newer 2025–2026 releases in the same size class.

Model	Triage fields correct	Right owner	Right tier	Escalations caught	False alarms	Right tool
UnchaosLM v4 · 4B	84%	86%	83%	26 of 32	2	6/6
Gemma‑4‑E4B (Apr 2026)	72%	19%	69%	22 of 32	3	—
Qwen3‑4B‑2507 (Jul 2025)	65%	24%	48%	23 of 32	12	4/6
Qwen3‑4B (base)	65%	29%	51%	25 of 32	11	4/6
Qwen3.5‑4B (Mar 2026)	63%	13%	55%	32 of 32	27	5/6
Qwen3‑8B	63%	24%	46%	26 of 32	13	5/6
Llama‑3.2‑3B	60%	21%	42%	24 of 32	15	—

Three things the table shows:

• Newer doesn't close the gap. The best 2026-generation model of the same size (Gemma‑4‑E4B) is still 12 points behind, and neither a newer generation (Qwen3.5) nor double the parameters (Qwen3‑8B) beats the original base. This task lives in details — the exact schema, the tiering rules, who "owner" refers to — that were trained in, not prompted in.
• Look at escalations and false alarms together. Qwen3.5 "catches" all 32 escalations by flagging 27 innocent signals too — an alarm that always rings. UnchaosLM catches 26 with just 2 false alarms, so its flag actually means something.
• "—" means not scored, not zero — Gemma and Llama emit tool calls in formats this eval doesn't parse, so they're excluded from that column rather than given an unfair 0.

Good to know

• Privacy — trained only on made-up (synthetic) examples: there is no real customer data inside the model, and it runs offline.
• Where it runs — Apple-silicon Macs only. It's small (~2.3 GB); a base MacBook Air handles it.
• Rough edges (fixing in v5) — on a handful of test signals (6 of 84, one scenario family) it gets stuck repeating a word and doesn't finish its answer, which is what the missed escalations in the table are; on a vague ask like "book some time" it sometimes guesses instead of asking you when; and it works best on short messages in English, not long documents.

How to run it

Easiest: put this repo's folder in ~/.lmstudio/models/suraj/UnchaosLM/ and it shows up in LM Studio.

Or from the terminal:

pip install mlx-lm
mlx_lm server --model suraj10620/UnchaosLM --port 8091 \
  --chat-template-args '{"enable_thinking":false}'

That gives you a local API any OpenAI-style client can call. Use the system prompt in SYSTEM_PROMPT.txt and temperature 0.

Technical details — training recipe, dataset versions, license

Model

LoRA finetune of mlx-community/Qwen3-4B-4bit (Qwen3-4B, 4-bit, MLX format), adapters fused into the weights. Two trained modes: strict-JSON extraction (full schema in SYSTEM_PROMPT.txt: is_actionable, action_direction, severity + rationale, tier P1–P4, owner, why_it_matters, next_step, deadline_suggestion, escalation_flag) and native Qwen3 tool calling over 6 tools (list_items, get_item, resolve_person, propose_calendar_block, list_calendar_events, get_sla_status).

Training recipe

LoRA rank 8 on the top 8 layers · batch 2 · lr 1e-4 · ~1.5 epochs · max seq 1536 · --mask-prompt · mlx-lm on Apple silicon. Agent scenarios are two rows each (tool-call turn + grounded answer) since --mask-prompt trains only the final assistant message. Keep enable_thinking: false at inference — trained without thinking traces.

Dataset versions

Version	Examples	Added
v2	532	Slack + ClickUp triage extraction
v3	1,032	+500 synthetic Hiver email-ticket signals
v4 (current)	1,232	+200 agent/tool-calling rows

All data is synthetic. Real support exports were mined locally for aggregate patterns only (issue categories, business mix); no real names, emails, or ticket text entered any dataset.

Benchmark methodology

eval/eval.py on eval/test.jsonl (99 held-out rows: 84 extraction, 15 agent). Greedy decoding, enable_thinking: false, max 512 tokens. Extraction scores exact match on the 7 deterministic fields (is_actionable, action_direction, severity, tier, owner case-insensitive, deadline_suggestion, escalation_flag); free-text fields are unscored. Field accuracy counts unparseable outputs as wrong on all 7 fields — on parseable outputs alone, v4 scores 90.8% (496/546) with escalation recall 26/27. Escalation false alarms count signals flagged escalation_flag: true whose gold label is false (52 such signals exist); UnchaosLM's escalation precision is 26/28 flagged = 92.9%. Agent scoring uses the 6 rows whose gold ends in a tool call (the 9 grounded-answer rows aren't auto-scorable); exact-argument match is 3/6 for v4 vs 0–2/6 for the others — two of v4's three misses are the known missing-tier-filter gap. The tool-call parser accepts both Qwen3's JSON-in-<tool_call> and Qwen3.5's XML-parameter format, with numeric-string coercion for argument comparison; Gemma-4 and Llama-3.2 run with --skip-agent since they emit other formats. Comparison models are the mlx-community 4-bit builds, evaluated 2026-07-13→15. eval/rescore.py recomputes any model's metrics from its saved raw generations. Full per-model outputs are the results_*.json files in eval/.

License

Apache 2.0, inherited from Qwen3-4B. Built as the model layer for Triage Copilot.

_{UnchaosLM v4 · July 2026 · built for Triage Copilot}