README.md

---
license: apache-2.0
language:
- en
base_model: mistralai/Mistral-7B-Instruct-v0.3
datasets:
- hvss/dispatch-7b-data
pipeline_tag: text-generation
tags:
- agent
- router
- orchestration
- tool-use
- planning
- unsloth
- mistral
---

![The Dispatcher: Agent Ops](banner.png)

# The Dispatcher: Agent Ops 🚦

> **Dispatch-7B, an agent orchestrator in 7B parameters — the tiny AI manager that runs a team of expensive AI workers.**

🎬 **[Watch it route work live →](https://huggingface.co/spaces/hvss/dispatch-7b-demo)**

Dispatch-7B is a fine-tuned **Mistral 7B Instruct v0.3** that acts as the **orchestrator** in an agentic system. It reads a user request plus a catalog of available tools and outputs a structured execution plan: which tool handles each step, in what order (as a dependency DAG), and which steps genuinely deserve escalation to a frontier model.

**Small model as the boss. Big models as the workers.**

![Cost comparison: 87% lower cost](cost_chart.png)

Most AI products send *every* request — even trivial ones — to the biggest, most expensive model. That's like having your highest-paid executive answer the front desk phone. Dispatch-7B is the dispatcher: it plans in milliseconds on cheap hardware, routes each step to the cheapest adequate worker, and escalates only when a step actually requires frontier-level reasoning.

## Evaluation

Measured on 114 held-out tasks (unseen requests *and* unseen tool-catalog combinations):

| Metric | Score |
|---|---|
| Valid plan rate (parseable JSON, schema-correct, acyclic DAG, catalog-consistent) | **97.4%** |
| Tool selection F1 (vs. teacher plans) | **0.92** |
| Escalation precision / recall | **0.92 / 0.75** |
| Est. cost vs. "frontier model does everything" | **−86.7%** |

Robustness breakdowns:

| Slice | Valid plan rate |
|---|---|
| Distractor-heavy catalogs (relevant tools buried among irrelevant ones) | 93% |
| Gap catalogs (a needed tool deliberately missing) | 94% |
| Hard tasks requiring escalation judgment | 89% |

Notes: routing labels are distilled from a teacher model (gpt-5-mini), so accuracy metrics measure agreement with the teacher. The cost figure is a transparent cost *model* over the eval plans (frontier $3/$15 per MTok, mini-tier $0.15/$0.60, self-hosted 7B router; invalid router plans charged as full frontier fallback), not a measured bill — assumptions are in the [eval scripts](https://huggingface.co/datasets/hvss/dispatch-7b-data).

The model deliberately **under-escalates** slightly (11.7% of plans vs. the teacher's 15.8%): it errs on the side of cheap execution, which is usually the right default for a router.

## Usage

The model expects Mistral's `[INST]` format with the dispatcher system prompt and your tool catalog:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("hvss/Dispatch-7B", torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("hvss/Dispatch-7B")

SYSTEM = '''You are a dispatcher. Read the user request and the tool catalog, then output ONLY a JSON execution plan routing each step to the cheapest adequate worker: "tool:<name>" (catalog tools only), "small_model" (simple generation), or "escalate:frontier" (genuinely hard reasoning only). Steps form a DAG via "depends_on". Output format: {"goal": ..., "steps": [{"id", "task", "assignee", "depends_on"}], "escalation_reason": ...}

Available tools:
- weather_lookup: Get the weather forecast for a location.
- calendar_create_event: Create a calendar event with title, time, and attendees.
- email_send: Send an email to one or more recipients.'''

request = "Set up a team picnic next Friday at noon if the weather looks good, and email the team an invite."

prompt = f"[INST] {SYSTEM}\n\n{request} [/INST]"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=768, do_sample=False)
print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
```

Output:

```json
{"goal": "...", "steps": [
  {"id": 1, "task": "Check the weather forecast for next Friday.", "assignee": "tool:weather_lookup", "depends_on": []},
  {"id": 2, "task": "Create the picnic calendar event if the forecast is good.", "assignee": "tool:calendar_create_event", "depends_on": [1]},
  {"id": 3, "task": "Email the team an invite with a short fun blurb.", "assignee": "tool:email_send", "depends_on": [2]}
], "escalation_reason": null}
```

Three assignee classes:

| Assignee | Meaning | Cost |
|---|---|---|
| `tool:<name>` | Direct tool/API call from your catalog | ~free |
| `small_model` | Simple generation a small LLM handles fine | cheap |
| `escalate:frontier` | Genuinely hard reasoning — send to the big model | expensive, used sparingly |

Swap in **your own catalog** — the model was trained on randomized catalogs (with distractors and deliberate gaps), so it plans with whatever tools you give it rather than assuming a fixed set.

## Variants

| Repo | Use case |
|---|---|
| [hvss/Dispatch-7B](https://huggingface.co/hvss/Dispatch-7B) | merged fp16 — vLLM / transformers |
| [hvss/Dispatch-7B-GGUF](https://huggingface.co/hvss/Dispatch-7B-GGUF) | Q4_K_M / Q8_0 — llama.cpp, Ollama, local |
| [hvss/Dispatch-7B-LoRA](https://huggingface.co/hvss/Dispatch-7B-LoRA) | adapter only — further fine-tuning |

## Training

- **Base**: Mistral 7B Instruct v0.3 (Apache 2.0)
- **Method**: QLoRA (4-bit, r=16 on all attention + MLP projections) via Unsloth, on a single free Colab T4
- **Data**: 2,158 (request, gold plan) pairs distilled from a teacher model over a structured taxonomy: 8 domains × 4 difficulty tiers × 3 catalog conditions (full / distractor-heavy / deliberately-missing-tool), ~16% escalation rate to teach restraint
- **Loss**: response-only (the model learns to write plans, not to regenerate catalogs)

## Limitations

- English-only training data.
- Routing quality is bounded by the teacher's judgment; gold labels are synthetic.
- The tool catalogs are synthetic (40 tools across 8 domains); very different domains or tool-description styles may need a light further fine-tune (start from the [LoRA repo](https://huggingface.co/hvss/Dispatch-7B-LoRA)).
- The model plans; it does not execute. Your harness owns tool execution, worker calls, and validation (schema-check the output — 2.6% of plans fail validation and should fall back to your default model).

## License

Apache 2.0 — free for commercial use.