Text Classification
Laya
Safetensors
English
Chinese
mmbert
routing
agents
multi-head-classification
task-difficulty
Instructions to use Waytrine/task-router with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Laya
How to use Waytrine/task-router with Laya:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
File size: 5,477 Bytes
98eccc8 6da389f 98eccc8 6da389f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 | ---
license: apache-2.0
base_model: convaiinnovations/laya-multilingual
language:
- en
- zh
pipeline_tag: text-classification
tags:
- laya
- mmbert
- routing
- agents
- multi-head-classification
- task-difficulty
---
# Task Router v4
A 322M-parameter classifier that answers ten routing questions about one request to an AI agent system, in one forward pass: what kind of task it is, how hard it is, which tools it needs and who should do it. It routes requests in production in [PugPort](https://pugport.com), a multi-agent desktop app.
**Code, data and evaluation:** [github.com/Waytrine/task-router](https://github.com/Waytrine/task-router) · **Case study:** [wyt.si/task-router](https://wyt.si/task-router/) · **API reference:** [wyt.si/task-router/api](https://wyt.si/task-router/api/)
## The ten questions
| Head | Labels |
|---|---|
| `task_type` (10) | conversation, coding, research, writing, translation, data_analysis, media, automation, mixed, unclear |
| `difficulty` (4) | simple, standard, complex, unclear |
| `web_search`, `document_read`, `run_commands`, `browser_use` | yes, no, unclear |
| `input_modality` (6) | text, image, audio, video, mixed, unclear |
| `media_output` (9) | none, image_create, image_edit, speech, music, video_generate, video_render, mixed, unclear |
| `project_access` (4) | none, read, write, unclear |
| `executor_intent` (6) | none, assign, exclude, reassign, mention_only, unclear |
The input is a bounded "state" of five parts: the message, recent context, quoted reference text, attachment metadata (kind, name and MIME type only, never contents) and the candidate executors. Token limit 1024. The full definitions the model sees are in [`questions.json`](questions.json).
## Model details
- **Architecture:** Laya typed-decision model on a [`jhu-clsp/mmBERT-base`](https://huggingface.co/jhu-clsp/mmBERT-base) encoder (ModernBERT, 22 layers, hidden size 768, vocabulary 256,000) with ten single-choice heads.
- **Parameters:** 321,908,998 (encoder 306,939,648; decision heads 14,175,744; scorer 592,897; action head 198,402). Stored in FP16.
- **Fine-tuned from:** [`convaiinnovations/laya-multilingual`](https://huggingface.co/convaiinnovations/laya-multilingual), revision `1720e3e3357cfe1e281542e223f8273b0890ca34`, with the [Laya framework](https://github.com/NandhaKishorM/laya) at commit `b9fe351ce81e2d974bca2bf219c1e4030c0a05ea`.
- **Languages:** English and Chinese.
- **Calibration:** post-hoc temperature scaling, T = 3.51, fitted on a separate 60-row calibration split.
- **`model.safetensors` SHA-256:** `4445f9401b321a94ebc3614d231847dc08beaa0c9ccc3fded8dc39f90e8bb4b9`
## Training data
Only synthetic data: the bilingual dataset `pugport-routing-synthetic-v1`, published in [`data/`](https://github.com/Waytrine/task-router/tree/main/data) of the GitHub repository. Requests were written by an AI assistant and reviewed by a second agent; 720 training rows for v4. No real user requests were used to train this checkpoint.
Training: soft cross-entropy, 4 epochs, batch 8 x gradient accumulation 2, encoder learning rate 2.5e-5, head learning rate 1e-4, weight decay 0.01, gradient clip 1.0, bf16 mixed precision, encoder not frozen.
## Results
| Evaluation | Mean accuracy over the ten heads |
|---|---|
| Untuned base model, synthetic test (60 rows) | 44.0% |
| **Task Router v4, synthetic test (60 rows)** | **97.8%** |
| **Task Router v4, 30 sealed real requests** | **72.0%** |
The synthetic test was reused across rounds, so 97.8% is a regression number; 72.0% on real requests is the more honest measure. A later v5, trained with additional private real requests, reaches 91.0% on the same sealed set; its weights are not published because its training data is private. Details: [`eval/`](https://github.com/Waytrine/task-router/tree/main/eval).
## How to use
The checkpoint loads with the Laya framework. The GitHub repository has the inference service, an OpenAPI description and evaluation scripts:
```sh
git clone https://github.com/Waytrine/task-router.git && cd task-router
python -m pip install -r service/requirements.txt
python -c "from huggingface_hub import snapshot_download as d; d('Waytrine/task-router', local_dir='models/task-router-v4')"
TASK_ROUTER_CHECKPOINT=models/task-router-v4 TASK_ROUTER_DEVICE=cpu python service/app.py
```
Then `POST /v1/judge` with a request state; see the [API reference](https://wyt.si/task-router/api/). `TASK_ROUTER_DEVICE` can be `cpu`, `cuda` or `mps`.
## Limits
- **Real-traffic evaluation is small:** 30 sealed items; one item moves a head by 3.3 points. 26 of the 30 are coding requests and none asks for media output.
- **Synthetic-to-real gap:** v4 underrates the difficulty of long real coding work that needs project write access and commands.
- **Confidence is not a safeguard:** raw confidences stay high even on wrong answers.
- **Not compared end to end** with an LLM judge or with PugPort's rule-based router.
- It classifies a request; it does not authorise tools, file writes or spending.
## License and attribution
- **These weights:** Apache License 2.0. They are a fine-tuned derivative of `convaiinnovations/laya-multilingual` (Apache-2.0), whose encoder is `jhu-clsp/mmBERT-base` (MIT).
- **The code** in the GitHub repository is MIT-licensed.
- Task Router is an independent project by [Waytrine](https://wyt.si). It is not a fork of Laya and not an official Laya or Convai Innovations product.
|