Task Router v4

A 322M-parameter classifier that answers ten routing questions about one request to an AI agent system, in one forward pass: what kind of task it is, how hard it is, which tools it needs and who should do it. It routes requests in production in PugPort, a multi-agent desktop app.

Code, data and evaluation: github.com/Waytrine/task-router ยท Case study: wyt.si/task-router ยท API reference: wyt.si/task-router/api

The ten questions

Head Labels
task_type (10) conversation, coding, research, writing, translation, data_analysis, media, automation, mixed, unclear
difficulty (4) simple, standard, complex, unclear
web_search, document_read, run_commands, browser_use yes, no, unclear
input_modality (6) text, image, audio, video, mixed, unclear
media_output (9) none, image_create, image_edit, speech, music, video_generate, video_render, mixed, unclear
project_access (4) none, read, write, unclear
executor_intent (6) none, assign, exclude, reassign, mention_only, unclear

The input is a bounded "state" of five parts: the message, recent context, quoted reference text, attachment metadata (kind, name and MIME type only, never contents) and the candidate executors. Token limit 1024. The full definitions the model sees are in questions.json.

Model details

  • Architecture: Laya typed-decision model on a jhu-clsp/mmBERT-base encoder (ModernBERT, 22 layers, hidden size 768, vocabulary 256,000) with ten single-choice heads.
  • Parameters: 321,908,998 (encoder 306,939,648; decision heads 14,175,744; scorer 592,897; action head 198,402). Stored in FP16.
  • Fine-tuned from: convaiinnovations/laya-multilingual, revision 1720e3e3357cfe1e281542e223f8273b0890ca34, with the Laya framework at commit b9fe351ce81e2d974bca2bf219c1e4030c0a05ea.
  • Languages: English and Chinese.
  • Calibration: post-hoc temperature scaling, T = 3.51, fitted on a separate 60-row calibration split.
  • model.safetensors SHA-256: 4445f9401b321a94ebc3614d231847dc08beaa0c9ccc3fded8dc39f90e8bb4b9

Training data

Only synthetic data: the bilingual dataset pugport-routing-synthetic-v1, published in data/ of the GitHub repository. Requests were written by an AI assistant and reviewed by a second agent; 720 training rows for v4. No real user requests were used to train this checkpoint.

Training: soft cross-entropy, 4 epochs, batch 8 x gradient accumulation 2, encoder learning rate 2.5e-5, head learning rate 1e-4, weight decay 0.01, gradient clip 1.0, bf16 mixed precision, encoder not frozen.

Results

Evaluation Mean accuracy over the ten heads
Untuned base model, synthetic test (60 rows) 44.0%
Task Router v4, synthetic test (60 rows) 97.8%
Task Router v4, 30 sealed real requests 72.0%

The synthetic test was reused across rounds, so 97.8% is a regression number; 72.0% on real requests is the more honest measure. A later v5, trained with additional private real requests, reaches 91.0% on the same sealed set; its weights are not published because its training data is private. Details: eval/.

How to use

The checkpoint loads with the Laya framework. The GitHub repository has the inference service, an OpenAPI description and evaluation scripts:

git clone https://github.com/Waytrine/task-router.git && cd task-router
python -m pip install -r service/requirements.txt
python -c "from huggingface_hub import snapshot_download as d; d('Waytrine/task-router', local_dir='models/task-router-v4')"
TASK_ROUTER_CHECKPOINT=models/task-router-v4 TASK_ROUTER_DEVICE=cpu python service/app.py

Then POST /v1/judge with a request state; see the API reference. TASK_ROUTER_DEVICE can be cpu, cuda or mps.

Limits

  • Real-traffic evaluation is small: 30 sealed items; one item moves a head by 3.3 points. 26 of the 30 are coding requests and none asks for media output.
  • Synthetic-to-real gap: v4 underrates the difficulty of long real coding work that needs project write access and commands.
  • Confidence is not a safeguard: raw confidences stay high even on wrong answers.
  • Not compared end to end with an LLM judge or with PugPort's rule-based router.
  • It classifies a request; it does not authorise tools, file writes or spending.

License and attribution

  • These weights: Apache License 2.0. They are a fine-tuned derivative of convaiinnovations/laya-multilingual (Apache-2.0), whose encoder is jhu-clsp/mmBERT-base (MIT).
  • The code in the GitHub repository is MIT-licensed.
  • Task Router is an independent project by Waytrine. It is not a fork of Laya and not an official Laya or Convai Innovations product.
Downloads last month
18
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Waytrine/task-router

Finetuned
(95)
this model