appriai/gen-router-t1

Gen Router Tier Classifier

The Problem

Every LLM-powered application has to decide, for each incoming prompt, which model should answer it. Get that decision wrong in either direction and it costs you:

• Always use the most capable model and you pay frontier-tier pricing and latency for questions a much cheaper model could have answered just as well — "what's the capital of France?" doesn't need a reasoning-heavy, expensive model.
• Always use a cheap, fast model and you save money on easy traffic but silently degrade quality on the hard prompts that actually need deep reasoning, domain expertise, or careful multi-step work — a failure mode that's easy to miss until a user notices a bad answer.
• Route by hand-written rules (keyword lists, prompt length, source of the request) and you get a system that's brittle and doesn't generalize — short prompts can be genuinely hard, and long prompts can be trivial.

This model exists to make that per-request decision automatically and cheaply: given a prompt, predict whether it belongs on a fast, tier, balanced tier, or frontier tier model — before any generation happens — so a routing layer can send each request to the right-sized model. Because the classifier itself is small (see benchmark below), the routing decision adds negligible cost and latency compared to running the prompt through an oversized model unnecessarily.

What It Predicts

Tier	Intended for
fast	Simple, single-step tasks — factual lookups, short translations, basic formatting/classification
balanced	Multi-step tasks needing real reasoning — code generation, summarization, structured writing
frontier	Expert-level or deep multi-step reasoning — complex system design, proofs, research synthesis, adversarial edge cases

Test Set Results

Metric	Value
Accuracy	0.890
F1 Macro	0.797
F1 Fast	0.928
F1 Balanced	0.848
F1 Frontier	0.615

Label Mapping

{
  "0": "fast",
  "1": "balanced",
  "2": "frontier"
}

Training

• 4 epochs, lr=2e-5, batch size 32, max_length=320
• Class-weighted cross-entropy loss (inverse frequency) to counter frontier being a minority class
• Best checkpoint selected by validation F1 macro (monotonically improved every epoch: 0.708 → 0.788 → 0.792 → 0.798)

Comprehensive Benchmark

Per-Source Accuracy (test set)

Source	n	Accuracy
flan	4,108	97.4%
openhermes	5,929	87.5%
evol	2,479	85.0%
sharegpt	1,431	78.3%

Calibration

Expected Calibration Error (ECE): 0.053.

Inference Latency — PyTorch vs ONNX (single-sample, batch=1)

Config	p50	p95	p99	Mean
PyTorch GPU (T4)	15.83ms	23.62ms	43.95ms	16.99ms
ONNX GPU (T4)	4.93ms	10.86ms	12.48ms	5.67ms
PyTorch CPU	158.17ms	412.04ms	483.39ms	204.57ms
ONNX CPU	77.77ms	360.32ms	488.04ms	111.39ms

The ONNX export runs ~3x faster on GPU and ~2x faster on CPU than the native PyTorch model, with accuracy verified identical on the full test set (89.02% for both). CPU tail latency (p95/p99) reflects a constrained benchmarking environment and is not representative of a dedicated server.

Error Analysis

• Overall accuracy: 89.0% (test set, n=13,947)
• Critical failure rate (frontier prompt routed to fast): 3.6%
• Over-provisioning rate (fast prompt routed to frontier): 0.17%

The error asymmetry favors safety: the model is far more likely to over-provision an easy prompt (wasting some cost) than under-provision a hard one (risking a bad answer) — the direction you want a router to err in.

Usage (PyTorch)

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

repo_id = "appriai/gen-router-t1"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForSequenceClassification.from_pretrained(repo_id)

inputs = tokenizer("Explain quantum entanglement in detail.", return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
tier = model.config.id2label[logits.argmax(-1).item()]
print(tier)  # "frontier"

Usage (ONNX — faster, recommended for production)

import onnxruntime as ort
from transformers import AutoTokenizer
from huggingface_hub import hf_hub_download

repo_id = "appriai/gen-router-t1"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
onnx_path = hf_hub_download(repo_id, "onnx/model.onnx")
session = ort.InferenceSession(onnx_path, providers=["CUDAExecutionProvider", "CPUExecutionProvider"])

enc = tokenizer("Explain quantum entanglement in detail.", return_tensors="np")
inputs = {i.name: enc[i.name] for i in session.get_inputs()}
logits = session.run(["logits"], inputs)[0]