s-ttp/aiva-rlcd-modernbert

RLCD: a calibrated request classifier for LLM gateways and routers

RLCD reads an LLM request, either a single prompt or a whole conversation, and answers four questions about it:

1. What kind of work is it? (task_category)
2. How sensitive is the data in it? (sensitivity)
3. How capable a model does it need? (complexity)
4. Is the context enough, or does it need retrieval or compression? (context_need)

Each answer is a calibrated probability distribution, so you set your own confidence thresholds and fall back to your existing rules when the model is unsure. It is small enough to run on the request path on a CPU: about 146 ms median and 530 ms p95 on 8 vCPUs.

Put it in an LLM gateway, proxy or orchestration layer to:

• route by cost: send easy requests to small, cheap models and hard ones to frontier models;
• gate retrieval: skip RAG when the prompt is self-contained, and trigger it when the answer depends on your organisation's documents or records;
• compress long context before it reaches an expensive model;
• tighten data classification: add a second, tighten-only signal next to DLP, so that sensitive requests stay on models you are allowed to send them to;
• tag traffic for cost, usage and risk analytics.

RLCD is a full fine-tune of answerdotai/ModernBERT-base with four linear heads on one shared, mean-pooled encoder. It was built for, and runs in, AiVa, a proof-of-concept enterprise AI gateway; see Reference deployment. To retrain it for your own organisation and taxonomy, use the kit in training/; see Retrain it for your requirements.

Read before relying on it.

• All training and evaluation data is synthetic. It describes a fictional telecom operator and other invented organisations.
• The held-out scores below overstate accuracy on real traffic. The test data came from the same generator family as the training data. On 16 fresh, hand-written prompts the model matched roughly 70–80% per head (see Evaluation).
• The labels follow one rubric. Check that the label definitions match how your organisation thinks about tasks and data, then check the thresholds on a sample of your own traffic. See Adapting it to your environment.
• It is not a security control on its own. Its sensitivity output should only ever tighten a decision that your policy and DLP have already made.

Model type	Encoder-only text classifier with four heads
Base model	answerdotai/ModernBERT-base at revision 8949b909ec90
Parameters	about 149 M (the encoder) plus four linear heads
Language	English
Input	One conversation, rendered as [role] content lines; the first 1,024 tokens are read
Output	Four calibrated probability distributions
CPU latency	p50 143–149 ms, p95 526–530 ms (8 vCPU, one request at a time, tokenizer included; two measured runs)
Weights	fp32 safetensors, 596 MB
Licence	Apache-2.0

Contents

• What it predicts
• Use cases
• Where it sits in a gateway
• Quick start
• Turning predictions into decisions
• Adapting it to your environment
• Retrain it for your requirements
• Deployment notes
• Training data · Training procedure · Evaluation
• Intended and out-of-scope use · Limitations
• Reference deployment: AiVa · Provenance

What it predicts

Head	Classes	Typical use
task_category	qa, summarization, coding, rag, analysis, generation, agentic	Pick a model that is strong at this kind of work; give agentic requests a model and permissions for tools; send coding to a code-capable model
sensitivity	public, internal, confidential, restricted, sovereign-only	Tighten the request's data classification, which then limits the models and regions it may go to
complexity	simple, moderate, complex, advanced	Choose the cheapest model tier that is capable enough
context_need	sufficient, needs-retrieval, needs-packaging	Skip retrieval, run retrieval, or compress long pasted context first

Each head returns softmax(logits / temperature). One temperature per head was fitted on held-out data and is stored in rlcd_config.json.

Label definitions

The model learned these definitions. In them, "the organisation" means the company whose assistant receives the request.

Task

• qa: a factual question with a short answer: general knowledge, or a product, price, plan or coverage fact.
• rag: an answer from the organisation's policy, procedure or contract documents, such as what a policy allows, who approves, or which section applies. This holds even when only one detail is asked for.
• summarization: condense content that is given or referred to, such as a report, thread, log or document.
• coding: write, fix, review or explain code, SQL, scripts, regular expressions or configuration. Code to be written is coding whatever the domain words.
• analysis: reason over data, logs, KPIs or records: compare, find causes or trends, correlate, rank.
• generation: draft new text, such as emails, notices, posts, letters, call scripts for agents, how-to guides or announcements.
• agentic: the sender asks the assistant to act beyond answering: check or investigate a specific account, line, ticket or system, or open, update, send, run or escalate something. A question whose answer is a fact to be looked up stays qa or rag, with needs-retrieval. Drafting a reply after looking something up or changing a record is agentic; drafting from what is given is generation.

Sensitivity. The most restrictive content present decides.

• public: published or generic, such as public prices and coverage, general knowledge, or generic code or text with no company data.
• internal: company-internal but not personal, such as processes, KPIs, operational data without customer or employee identifiers, internal code and systems, or a named non-public process or document.
• confidential: about an identifiable customer or employee (phone number, SIM identifiers such as ICCID and IMSI, national ID, email address, customer account number, account details), fraud cases, or unreleased financial results or board material.
• restricted: credentials and secrets (API keys, passwords, private keys), payment card numbers and IBANs, privileged access and administrator procedures, or details of security vulnerabilities.
• sovereign-only: lawful-authority and law-enforcement requests, interception, national-security matters, or regulator-mandated customer records. This is data that must stay on infrastructure, and in a jurisdiction, that you control.

Words alone do not raise a level. A blog post about data sovereignty is public, and so is asking how API keys work. A ticket or case reference alone does not raise a level, and neither does who asks or through which tool.

Complexity

• simple: one thing to do, with no constraints to balance.
• moderate: two or three steps or outputs, or one task under a few constraints.
• complex: four or five steps, several constraints, or several inputs to combine.
• advanced: six or more steps, or a multi-stage plan with dependencies, trade-offs or several deliverables.

Context need

• sufficient: everything needed is in the conversation, or the task needs no organisation-specific facts at all (generic coding, drafting, general knowledge). Code that names a table, file or service but needs no records from it is sufficient.
• needs-retrieval: the answer depends on organisation-specific facts or records that are not in the conversation, such as a product, price or policy fact, a customer's account, a named incident, or an attachment that is mentioned but missing.
• needs-packaging: the context is present but long or noisy, at about 1,500 estimated tokens (about 6,000 characters) or more of pasted documents, logs, tables or threads. Length decides first.

Use cases

1. Cost-aware model routing

Use complexity, and optionally task_category, to pick the cheapest model that is good enough, and keep frontier models for the requests that need them. For example:

Complexity	Illustrative tier
simple	A small, fast model: a few-billion-parameter open model, or an API "mini" or "flash" tier
moderate	A mid-size model
complex	A large model
advanced	A frontier or reasoning model

This mapping is only an example. Benchmark your own models on your own tasks, and only ever choose among the models that policy allows for the request. Complexity is the weakest head (macro-F1 0.82), so use it at high confidence and fall back to your current tier when it is unsure. AiVa uses it at ≥ 0.80, which keeps 76% of requests at 97% accuracy on validation data.

2. Retrieval gating for RAG

context_need tells you whether to retrieve at all:

• sufficient: skip retrieval. This saves latency and cost, and avoids adding irrelevant passages.
• needs-retrieval: retrieve. If task_category is rag, search your policy and procedure documents.
• needs-packaging: see the next use case.

General-knowledge questions are sufficient under this rubric: retrieval is for organisation-specific facts.

3. Context compression

needs-packaging marks requests with long pasted material: documents, logs, tables or email threads. Before the main model sees them, summarise, extract or compress them, or send them to a long-context model. AiVa pairs RLCD with a 4 B compression model, s-ttp/aiva-cim-qwen3-4b-v7.

4. Data-classification tightening

Combine the sensitivity prediction with the classification you already have, and keep the most restrictive. Use the result to limit where the request may go. For example:

• confidential and above stays on approved private or in-region models;
• restricted needs extra controls or is blocked;
• sovereign-only stays on in-country infrastructure that you operate.

RLCD catches content that pattern-based DLP has no rule for, such as a lawful-intercept request, a fraud case or unreleased results. Measured on held-out synthetic data, RLCD and DLP together left 0.5% of confidential-or-higher requests classified too low, against 1.4% for RLCD alone. Never let it lower a classification, and never use it as the only control.

5. Traffic analytics

Tag logged requests offline to see:

• what your users ask for;
• how much sensitive data goes to which provider;
• which requests could have used a cheaper model.

Batch the requests and keep only the labels and probabilities, not the text.

Where it sits in a gateway

1. Authenticate and apply policy. This produces the set of models, regions and tools this caller may use.
2. Run DLP on the full text. RLCD reads only the first 1,024 tokens, so pattern checks must cover the rest.
3. Run RLCD.
4. Tighten the classification: effective = most restrictive(declared, application floor, DLP, RLCD). If it rose, re-check policy, which may shrink the allowed set.
5. Decide within the allowed set. Use only heads above your confidence thresholds, and your existing rules for the rest:
   • the cheapest capable model;
   • retrieve or not;
   • compress or not.
6. Call the model.

If RLCD is unavailable, too slow or fails, route by your rules and record that you did. Because RLCD can only tighten and choose within what policy allows, falling back is always safe.

Quick start

The repository holds:

• the encoder configuration (encoder/);
• the tokenizer;
• the encoder and head weights (model.safetensors);
• rlcd_config.json, which lists the heads, class order, temperatures, pooling and maximum length.

The checkpoint is not a stock AutoModelForSequenceClassification. Load it with this class, which is the same loading and scoring AiVa uses. The fine-tuning and recalibration snippets below build on it.

# pip install "torch>=2.4" "transformers>=4.48" safetensors huggingface_hub
import json
from pathlib import Path

import torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from transformers import AutoConfig, AutoModel, AutoTokenizer


class RlcdClassifier(torch.nn.Module):
    def __init__(self, path: Path) -> None:
        super().__init__()
        self.config = json.loads((path / "rlcd_config.json").read_text())
        self.tokenizer = AutoTokenizer.from_pretrained(path)
        self.encoder = AutoModel.from_config(AutoConfig.from_pretrained(path / "encoder"))
        hidden = self.encoder.config.hidden_size
        self.heads = torch.nn.ModuleDict(
            {name: torch.nn.Linear(hidden, len(classes)) for name, classes in self.config["heads"].items()}
        )
        # Parameter names match the trainer ("encoder.*", "heads.<name>.*"); strict on purpose.
        self.load_state_dict(load_file(str(path / "model.safetensors")), strict=True)
        self.eval()

    @staticmethod
    def render(messages: list[dict[str, str]]) -> str:
        """The model's input: one `[role] content` line per message, in order."""
        return "\n".join(f"[{m['role']}] {m['content']}" for m in messages)

    def logits(self, texts: list[str]) -> dict[str, torch.Tensor]:
        batch = self.tokenizer(
            texts, padding=True, truncation=True, max_length=self.config["max_length"], return_tensors="pt"
        )
        hidden = self.encoder(**batch).last_hidden_state
        mask = batch["attention_mask"].unsqueeze(-1).to(hidden.dtype)
        pooled = (hidden * mask).sum(1) / mask.sum(1).clamp(min=1.0)  # masked mean pooling
        return {name: head(pooled) for name, head in self.heads.items()}

    @torch.no_grad()
    def predict(self, messages: list[dict[str, str]]) -> dict[str, dict[str, float]]:
        logits = self.logits([self.render(messages)])
        return {
            name: dict(zip(self.config["heads"][name],
                           (value / self.config["temperatures"][name]).softmax(-1)[0].tolist()))
            for name, value in logits.items()
        }


model = RlcdClassifier(Path(snapshot_download("s-ttp/aiva-rlcd-modernbert")))
probabilities = model.predict([
    {"role": "user", "content": "What does the Traveller roaming bundle include?"},
])
for head, distribution in probabilities.items():
    label = max(distribution, key=distribution.get)
    print(f"{head}: {label} ({distribution[label]:.2f})")

Output (measured on CPU):

task_category: qa (0.96)
sensitivity: public (1.00)
complexity: simple (1.00)
context_need: needs-retrieval (0.97)

Pass a whole conversation, including any system prompt, as {"role": ..., "content": ...} messages. The model was trained on system, user and assistant turns. These snippets were tested with torch 2.14.1 and transformers 5.19.0. Pin the revision you tested with snapshot_download(..., revision=...).

Turning predictions into decisions

The helpers below implement the two rules AiVa uses:

• Sensitivity only tightens. Take the highest level L for which P(class ≥ L) ≥ 0.5, then keep the most restrictive of all the classifications you have.
• Use a head only above its threshold. Below it, the head is "unsure" and your own rule decides.

route() is an illustration of how to combine them. Replace MODELS with your own catalogue, prices and policy.

SENSITIVITY = ["public", "internal", "confidential", "restricted", "sovereign-only"]
COMPLEXITY = ["simple", "moderate", "complex", "advanced"]
MIN_CONFIDENCE = {"task_category": 0.60, "complexity": 0.80, "context_need": 0.60}  # AiVa's thresholds


def at_least(probabilities: dict, threshold: float = 0.5) -> str:
    """The highest sensitivity level L with P(level >= L) >= threshold."""
    mass = 0.0
    for level in reversed(SENSITIVITY):
        mass += probabilities["sensitivity"][level]
        if mass >= threshold:
            return level
    return SENSITIVITY[0]


def most_restrictive(*levels: str) -> str:
    return max(levels, key=SENSITIVITY.index)


def confident(probabilities: dict, head: str) -> str | None:
    """The head's top class if it reaches the head's threshold, otherwise None (unsure)."""
    label = max(probabilities[head], key=probabilities[head].get)
    return label if probabilities[head][label] >= MIN_CONFIDENCE[head] else None


def length_rule(messages: list[dict[str, str]]) -> str:
    """A stand-in for your existing complexity rule: AiVa's fallback, by estimated tokens."""
    tokens = sum(len(m["content"]) for m in messages) // 4
    return COMPLEXITY[0 if tokens < 500 else 1 if tokens < 2000 else 2 if tokens < 8000 else 3]


# Illustrative catalogue, cheapest first: the hardest work and the most sensitive data each model may take.
MODELS = [
    {"name": "small-api-model", "up_to": "simple", "data_up_to": "internal"},
    {"name": "large-api-model", "up_to": "advanced", "data_up_to": "internal"},
    {"name": "private-mid-model", "up_to": "complex", "data_up_to": "sovereign-only"},
    {"name": "private-large-model", "up_to": "advanced", "data_up_to": "sovereign-only"},
]


def route(messages: list[dict[str, str]], declared: str = "public", dlp: str = "public") -> dict:
    p = model.predict(messages)
    sensitivity = most_restrictive(declared, dlp, at_least(p))
    allowed = [m for m in MODELS if SENSITIVITY.index(m["data_up_to"]) >= SENSITIVITY.index(sensitivity)]
    complexity = confident(p, "complexity") or length_rule(messages)
    capable = [m for m in allowed if COMPLEXITY.index(m["up_to"]) >= COMPLEXITY.index(complexity)]
    chosen = capable[0] if capable else max(allowed, key=lambda m: COMPLEXITY.index(m["up_to"]))
    context = confident(p, "context_need")  # None: unsure, let your own rule decide
    return {
        "model": chosen["name"],
        "sensitivity": sensitivity,
        "complexity": complexity,
        "task": confident(p, "task_category"),
        "retrieve": context == "needs-retrieval",
        "compress": context == "needs-packaging",
    }


print(route([{"role": "user", "content": "Draft a short welcome email for new enterprise customers."}]))
print(route([{"role": "user", "content": "Customer with MSISDN +97455123456 disputes a charge of QAR 120. "
                                         "Check the account and draft a reply."}]))

Output (measured on CPU):

{'model': 'small-api-model', 'sensitivity': 'public', 'complexity': 'simple', 'task': 'generation', 'retrieve': False, 'compress': False}
{'model': 'private-mid-model', 'sensitivity': 'confidential', 'complexity': 'simple', 'task': 'agentic', 'retrieve': True, 'compress': False}

In the second request, the customer data moves the request to a private model even though dlp was left at public. In a real gateway, DLP would also have caught the phone number.

Adapting it to your environment

Check the label definitions against your own

• Your classification scheme. If you use fewer levels, map RLCD's levels onto yours, for example public and internal → General, confidential → Confidential, and restricted and sovereign-only → Restricted. Map the output of at_least() through that table. Because the rule is cumulative, this gives the same answer as applying it to your coarser scheme directly. Map by the definitions above, not by the names. If your levels cut across these, for example if health or children's data has its own level, retrain the head (below).

• Your identifier formats. The training data's identifiers were generated in specific formats:

  • UAE mobile numbers (+971 59 …) and Emirates IDs (784-YYYY-NNNNNNN-N);
  • SIM identifiers (ICCID, IMSI) and customer account numbers (CUST-NNNNNN);
  • email addresses, payment card numbers, IBANs, API keys and passwords.

  Other countries' phone and ID formats, non-IBAN bank account numbers, and sector identifiers such as patient, policy or case numbers were not in training. A Qatari (+974) number in one probe was caught, so the model transfers somewhat, but that was never measured. Test your formats, keep them in DLP, and fine-tune if you need RLCD to catch them.

• Your tasks. If you need other task classes, such as translation, retrain the task head. Translation and "compare X and Y" requests are currently read as qa or analysis.

Measure it on your own traffic first

Label a few hundred real requests from your environment, stratified across the classes you care about. Then:

1. Score them with model.predict, and compare per head.
2. Re-fit the temperatures and re-choose the thresholds (below).
3. Fine-tune if a head is not good enough.

Don't log prompt text to do this unless your policy allows it.

Choose confidence thresholds

These curves are measured on synthetic validation data weighted to a realistic traffic mix. Each cell shows the share of requests whose top-class probability reaches the threshold, and the accuracy on those requests. Treat them as a starting point, not as a promise for your traffic.

Head	≥ 0.5	≥ 0.6	≥ 0.7	≥ 0.8	≥ 0.9	≥ 0.95
task_category	98.9% / 96.9%	97.7% / 97.6%	95.9% / 98.3%	93.9% / 99.0%	90.8% / 99.5%	88.4% / 99.6%
sensitivity	99.9% / 98.9%	99.5% / 99.1%	99.2% / 99.3%	98.7% / 99.6%	98.3% / 99.8%	97.1% / 99.9%
complexity	88.3% / 92.0%	82.9% / 94.8%	79.0% / 96.4%	75.8% / 97.0%	69.7% / 98.2%	63.9% / 98.8%
context_need	100.0% / 98.3%	99.4% / 98.6%	98.7% / 98.8%	97.5% / 99.1%	93.8% / 99.5%	90.9% / 99.7%

For the sensitivity rule, a lower cut-off catches slightly more and over-raises slightly more. On the same validation data, under-classification of confidential-or-higher requests with DLP alongside was:

• 0.3% at a cut-off of 0.3–0.35;
• 0.5% at 0.4 and above.

Over-classification to confidential or above was 0.4% at a cut-off of 0.3, and 0.3% at 0.5.

Re-fit the temperatures and thresholds

Calibration was fitted on synthetic data. Re-fit one temperature per head on a held-out sample of your own labelled requests, and read off your own coverage and accuracy. rows holds your data, in this form: {"messages": [...], "labels": {"task_category": "coding", ...}}. Each row may label any subset of the heads.

@torch.no_grad()
def head_logits(model: RlcdClassifier, rows: list[dict], head: str) -> tuple[torch.Tensor, torch.Tensor]:
    labelled = [r for r in rows if head in r["labels"]]
    logits = torch.cat([model.logits([model.render(r["messages"])])[head] for r in labelled])
    labels = torch.tensor([model.config["heads"][head].index(r["labels"][head]) for r in labelled])
    return logits, labels


def fit_temperature(logits: torch.Tensor, labels: torch.Tensor) -> float:
    log_t = torch.zeros(1, requires_grad=True)
    optimiser = torch.optim.LBFGS([log_t], lr=0.1, max_iter=200)

    def closure() -> torch.Tensor:
        optimiser.zero_grad()
        loss = torch.nn.functional.cross_entropy(logits / log_t.exp(), labels)
        loss.backward()
        return loss

    optimiser.step(closure)
    return log_t.exp().item()


def coverage_accuracy(logits: torch.Tensor, labels: torch.Tensor, temperature: float, threshold: float):
    confidence, predicted = (logits / temperature).softmax(-1).max(-1)
    kept = confidence >= threshold
    return kept.float().mean().item(), (predicted[kept] == labels[kept]).float().mean().item()


for head in model.config["heads"]:
    if not any(head in r["labels"] for r in held_out):
        continue
    logits, labels = head_logits(model, held_out, head)
    temperature = model.config["temperatures"][head] = fit_temperature(logits, labels)
    for threshold in (0.5, 0.6, 0.7, 0.8, 0.9):
        print(head, threshold, coverage_accuracy(logits, labels, temperature, threshold))

Fit on a few hundred labelled rows per head, at least. A temperature fitted on a handful of rows is noise. If your held-out sample's class mix differs from production, weight each row's loss by production share / sample share of its class before fitting. AiVa did this for its sensitivity prior.

Fine-tune on your own labels

The encoder and the four heads train together. To change a head's classes, replace the head first. Its weights then start untrained, so it needs enough examples per class.

import random

from safetensors.torch import save_file


def replace_head(model: RlcdClassifier, head: str, classes: list[str]) -> None:
    """Give a head your own classes (only needed if they differ from the shipped ones)."""
    model.config["heads"][head] = classes
    model.config["temperatures"][head] = 1.0
    model.heads[head] = torch.nn.Linear(model.encoder.config.hidden_size, len(classes))


def fine_tune(
    model: RlcdClassifier, rows: list[dict], epochs: int = 3, lr: float = 2e-5, batch_size: int = 16
) -> None:
    optimiser = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=0.01)
    model.train()
    for _ in range(epochs):
        random.shuffle(rows)
        for start in range(0, len(rows), batch_size):
            chunk = rows[start:start + batch_size]
            logits = model.logits([model.render(r["messages"]) for r in chunk])
            loss = torch.zeros(())
            for head, classes in model.config["heads"].items():
                targets = torch.tensor([classes.index(r["labels"][head]) if head in r["labels"] else -100
                                        for r in chunk])
                if (targets != -100).any():  # rows may label only some heads
                    loss = loss + torch.nn.functional.cross_entropy(logits[head], targets, ignore_index=-100)
            loss.backward()
            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
            optimiser.step()
            optimiser.zero_grad()
    model.eval()


def save(model: RlcdClassifier, out: Path) -> None:
    """Save in the same layout, so RlcdClassifier(out) loads it."""
    out.mkdir(parents=True, exist_ok=True)
    save_file({k: v.contiguous() for k, v in model.state_dict().items()}, str(out / "model.safetensors"))
    model.tokenizer.save_pretrained(out)
    model.encoder.config.save_pretrained(out / "encoder")
    keep = ("architecture", "pooling", "max_length", "heads", "temperatures", "input_format")
    config = {key: model.config[key] for key in keep} | {"base_model": "s-ttp/aiva-rlcd-modernbert"}
    (out / "rlcd_config.json").write_text(json.dumps(config, indent=2))

Some tips:

• Train with a GPU if you can. On CPU, use a small set and few epochs. For a larger run, use training/train.py --init-from s-ttp/aiva-rlcd-modernbert.
• Mix in examples of every class you keep, not only the ones you are correcting, so the other classes don't drift.
• Re-fit the temperatures on a held-out set afterwards.
• If you change the sensitivity classes, update SENSITIVITY in the decision helpers to the new order, from least to most restrictive.

Retrain it for your requirements

training/ holds a standalone kit for the whole recipe: the rubric, data generation, training, calibration and evaluation. Use it to retrain RLCD on your organisation's wording, identifiers and taxonomy. Its README has the commands.

The published model was built with the AiVa proof of concept's internal pipeline. The kit is a standalone port of its recipe and settings. It differs in three ways:

• attachments are built from templates, not from a synthetic world of records;
• literal identifiers are caught by patterns, not by a DLP engine;
• it has no template-shaped rows.

It was tested end to end at small scale, but it has not been used to reproduce this model at full size.

Choose how far to go

Your situation	What to do
The classes fit, but your wording or identifiers differ	Fine-tune from this checkpoint on a few thousand of your rows: train.py --init-from s-ttp/aiva-rlcd-modernbert.
One head needs other classes (a coarser sensitivity scheme, a translation task)	Change those classes in taxonomy.json and use --init-from. Unchanged heads keep their weights; changed heads start fresh.
You need other outputs, or heads that mean something else	Train from answerdotai/ModernBERT-base with your own taxonomy. generate.py needs code changes for heads that mean something new.

The recipe

1. Write the rubric before any data. taxonomy.json holds every class's definition, plus tie-breaks for the boundaries people disagree on.

   • Run a pilot of a few hundred rows, read the rows the verifier rejected, and add a tie-break wherever a boundary is contested. This model's tie-breaks came from that kind of reading:
     • agentic against a lookup came from its pilots;
     • qa against rag, and the customer account number, came from the rejections in the first build of its test split, which was then re-verified before it was frozen.
   • Have the verifier answer yes/no questions about what the text contains, and derive the sensitivity level from them in code. In the pilot, a verifier asked to name the level directly skipped literal API keys and explicit national-security notices.

2. Build the data so that code, not a model, owns the labels.

   • Code samples a spec in exact class proportions:

     • the labels;
     • who writes and how;
     • a subject that fits the task and level;
     • an optional pasted attachment of a kind the task can use, such as a policy excerpt for a policy question or a bill for a billing question;
     • the identifiers the request must carry;
     • for about one spec in five, a hard-negative trap with fixed labels.

     Keep the specs coherent. The verifier rejects a request whose pasted material or identifiers don't fit its task and level, such as a policy question with a log pasted in.

   • An LLM writes only the wording, with placeholders where identifiers go. Code fills in random values in real formats.

   • Code checks every draft and asks again with the findings. It checks:

     • turns, placeholders, length and language;
     • literal identifiers and credentials;
     • that nothing in the assembled request, attachment included, sits above the label;
     • the context-length rule.

   • A verifier with a different prompt labels the finished request from the rubric alone, without seeing the spec. A row is kept only if the labels agree on every head, with complexity allowed to differ by one level.

   • Write data with a model that is neither the one you train nor one you route to, so the data does not favour it. This model's data was written by openai/gpt-oss-120b.

   • Pin one provider for every split, and record it, so the splits don't differ by serving stack.

   • Prefer real traffic where you have it. generate.py label pre-labels your own requests with the same verifier, for people to review.

3. Make the test split hard to overfit.

   • Write it with personas, channels, styles and app prompts that training never uses, and with a differently framed writer prompt.
   • Pass validation rows through a second, reworded verifier, and keep only the rows both passes agree on. 84.3% of first-pass validation rows survived it.
   • Give the test split one pass, and record the second wording only as a report slice. Filtering the test split as well would remove its ambiguous rows and make scores read higher.
   • Deduplicate the training rows against the test split, then freeze the test split and never tune on it.

4. Train with the rare classes in view.

   • Use square-root class-balanced sampling on sensitivity, and length-bucketed batches.
   • The settings are in Training procedure.
   • On one A100 the run took 13 minutes, about $0.58 of Hugging Face Jobs time.
   • Generating the training and validation data cost about $10, and the test split about $2 more. Both are estimated from provider-reported tokens at list price.

5. Calibrate to production. Fit one temperature per head on validation rows weighted to your real sensitivity mix (--prior).

6. Set the bar first, then evaluate. evaluate.py reports what this model was judged on:

   • per-head macro-F1 and calibration;
   • under- and over-classification of sensitive requests, alone and with DLP, with 95% intervals;
   • coverage and accuracy at each confidence threshold;
   • CPU latency.

   This model had 33 gates. Eight were regression floors on an older template test set, which the kit does not include. The others included:

   • macro-F1 ≥ 0.90 for task and sensitivity, ≥ 0.85 for context need and ≥ 0.75 for complexity;
   • at most 2% of confidential-or-higher requests classified too low with DLP alongside, and at most 3% per level;
   • calibration error ≤ 0.05 on every head;
   • CPU p95 ≤ 800 ms.

   It passed all 33. Its macro-F1 gates were judged on the test split as sampled; its calibration and classification-safety gates were judged both as sampled and under a realistic traffic prior.

7. Ship. Pin the revision, re-derive your thresholds from the coverage curves, and check that your serving code reproduces evaluate.py's probabilities on a few rows.

Deployment notes

• CPU is enough. It ran at p50 143–149 ms and p95 526–530 ms (two runs) on 8 vCPUs at 8 threads, one request at a time, tokenizer included, over a natural mix of request lengths. Hosts of the same class varied by about 1.7×. GPU serving was not measured. For offline tagging, model.logits takes a batch of rendered texts; in a check, batched and one-at-a-time logits agreed to within 1e-5.
• Keep max_length at 1,024. With a 512-token window, context_need loses the needs-packaging class: its macro-F1 falls to 0.58, and its accuracy on long requests to 0. A 512 window halves p95 latency (256 ms).
• Load once at startup. Pin the revision. Run scoring on a worker thread, not your event loop. Set torch.set_num_threads to the CPUs your container may actually use.
• Give it a time budget. AiVa allows 2 seconds. On a timeout or an error, use your rules and record that the classifier was skipped.
• Cache by input hash. Identical inputs give identical predictions.
• Don't log the text. Log the labels, probabilities and model revision. Prompts may hold the very data this model is classifying.

Training data

The dataset is s-ttp/aiva-rlcd-labels-v2 at revision 7c3c097. It is not published.

Split	Rows	What
Train	38,118	21,954 template-shaped rows plus 16,164 natural requests
Validation	2,018	Natural requests, each agreed by two independent verifier passes
Natural test (frozen)	2,233	Natural requests, generated with disjoint persona, department, channel and style families and a different prompt
Template test	3,727	Template-shaped rows, a regression floor

What the requests look like:

• Who sends them. Staff of a fictional telecom operator write to an internal assistant. They work in 30 roles across network operations, customer care, retail, billing, IT and data, HR, legal, security, marketing, finance, procurement, fraud, regulatory affairs and others. The test split uses different roles from the training split.
• How they are written. Terse, verbose, messy and mobile-style; formal emails and chat; multi-turn conversations; and an app system prompt in some.
• What they carry. Pasted code, tables, logs, policy excerpts, CDR-style records and email threads.
• Traps. 24.8% of training rows contain a hard-negative trap. Examples are code that mentions customers but carries no personal data, a public question that uses account words, and a self-contained request that looks like it needs retrieval.

How the natural requests were made:

• Code fixes the labels. For each row, code samples a specification:
  • the four labels;
  • persona, department and channel;
  • writing style, length and number of turns;
  • an optional attachment, built by code;
  • an optional trap.
• A model writes the wording. openai/gpt-oss-120b writes each request from its specification. Identifiers such as phone numbers, IBANs, card numbers and keys are inserted by code. They follow real formats but are random, and come from no real list.
• Code checks every row. It checks the format, placeholders, length and language. It also checks that no literal identifier leaks into the wording, that DLP never finds more than the label says, and that the context-length rule holds.
• A verifier gates every row. A second pass of the same model, with a different prompt, sees only the rubric and the request, never the specification. A row is kept only if the verifier agrees on task, sensitivity and context need, and on complexity within one level. About 56% of generated training rows were kept.
• The splits are deduplicated against each other. The seven reference probes below were checked against every training row.

Natural training rows by sensitivity:

Level	Rows
public	5,363
internal	4,468
confidential	2,654
restricted	1,871
sovereign-only	1,808

Every task class has at least 1,671 rows.

Training procedure

• Method. A full fine-tune of every encoder parameter and the four heads. The loss is the sum of the per-head cross-entropies.

• Hyperparameters.

  • 3 epochs, AdamW with a cosine schedule, learning rate 5e-5, batch size 32, maximum length 1,024 tokens.
  • Square-root class-balanced sampling on sensitivity, so the rare levels are learned well.
  • Length-bucketed batches.

• Calibration. One temperature per head, fitted on the validation split weighted to a realistic traffic prior (public 14.6%, internal 52.7%, confidential 29.7%, restricted 2.4%, sovereign-only 0.6%):

  Head	Temperature
  task_category	1.084
  sensitivity	0.980
  complexity	1.455
  context_need	1.066

• Hardware. One NVIDIA A100 on Hugging Face Jobs (a100-large). Training took 13 minutes.

Evaluation

Frozen natural test

All figures are measured on 2,233 rows, scored once. Nothing was tuned on this split.

Head	Accuracy	Macro-F1	ECE (as sampled)	ECE (realistic prior)
task_category	0.936	0.934	0.018	0.020
sensitivity	0.983	0.983	0.010	0.007
complexity	0.830	0.816	0.039	0.046
context_need	0.981	0.981	0.003	0.007

Classification safety, under the tighten-only rule at 0.5, with 95% Wilson intervals:

Measure	Value
Confidential-or-higher requests classified too low, RLCD alone	1.4% (0.8–2.4%)
… RLCD or DLP	0.5% (0.2–1.3%); 0.0% under the realistic prior
… per level, RLCD or DLP: confidential / restricted / sovereign-only	0.0% / 0.0% / 1.7% (0.7–4.3%)
Public or internal requests raised to ≥ confidential / ≥ restricted / sovereign-only	0.8% / 0.5% / 0.1%

The DLP used here has no pattern for sovereign content, so the sovereign-only misses are the model's own.

Slices:

• Twice-agreed rows (1,959, the most stable labels): macro-F1 task 0.927, sensitivity 0.982, complexity 0.808, context 0.981.

• Long, packaging-length rows (661): context-need accuracy 0.983; sovereign-only under-classification 3.7% (3 of 81, 1.3–10.3%). This is the model's weakest spot.

• Template test (3,727 rows), accuracy and macro-F1:

  Head	Accuracy	Macro-F1
  task	0.991	0.986
  sensitivity	0.980	0.786
  complexity	0.892	0.869
  context need	0.971	0.972

  The template test has no sovereign-only rows, which lowers its sensitivity macro-F1.

Reference probes

These seven hand-written prompts are labelled under the rubric. They were never used for training.

Prompt	task	sensitivity	complexity	context
Write a Python function that validates an IBAN and add unit tests for it.	coding ✓	public ✓	moderate ✓	sufficient ✓
Write a Python function that parses a CSV of cell-site alarms and returns the five sites with the most critical alarms.	coding ✓	internal ✗ (public)	simple ✗ (moderate)	sufficient ✓
Write a SQL query that returns the ten customers with the most dropped calls last week from the cdr table.	coding ✓	internal ✓	simple ✗ (moderate)	sufficient ✓
Summarise the attached incident report for the operations manager in five bullet points.	summarization ✓	internal ✓	simple ✓	needs-retrieval ✓
What does the Traveller roaming bundle include?	qa ✓	public ✓	simple ✓	needs-retrieval ✓
Customer with MSISDN +97455123456 disputes a charge of QAR 120. Check the account and draft a reply.	agentic ✓	confidential ✓	simple ✗ (moderate)	needs-retrieval ✓
Draft a short welcome email for new enterprise customers.	generation ✓	public ✓	simple ✓	sufficient ✓

Independent spot-check on fresh, hand-written prompts

16 further prompts were written and labelled by hand, outside the data pipeline. They cover:

• refactoring and SQL requests;
• translation;
• an HR approval;
• a pasted password;
• an unpatched vulnerability;
• a court order for call records;
• an interception warrant;
• a 260-line alarm export.

The model matched:

Head	Matched
Task	11 of 16
Sensitivity, tighten rule	11 of 16
Context need	13 of 16

• Every restricted and sovereign-only prompt was right.
• Misses:
  • an employee's leave balance rated internal. An individual's HR data is confidential, and DLP does not catch it;
  • a customer's phone number just under the tightening threshold. DLP catches the number;
  • translation and "compare X and Y" requests read as qa or analysis;
  • two requests that needed records were read as self-contained.

On fresh wording like this, expect roughly 70–80% per head, not the 93–98% above. Measure on your own traffic before relying on any head.

Intended use

• A routing and triage signal inside an LLM gateway, proxy or orchestration layer that applies authentication, policy and DLP first. It is used to choose a model tier, gate retrieval, trigger context compression and tighten data classification.
• Offline tagging of request logs for cost, usage and risk analytics.
• A starting point for fine-tuning a request classifier on your own labelled traffic and your own taxonomy.
• Research and teaching on policy-constrained, cost-aware LLM routing.

Out-of-scope use

• A data-classification, DLP or access-control mechanism on its own. Use it only to tighten a classification, after policy and DLP have run.
• Security, compliance, legal or other high-stakes decisions, and any decision about people, such as monitoring or assessing employees.
• Detecting prompt injection or jailbreaks, or judging content safety. It was not trained for these.
• Languages other than English.
• Inputs over 1,024 tokens, unless DLP covers the full text. The rest of the text is not read.

Limitations and risks

• Synthetic data from one generator family. The training and test data were written by the same model, and the verifier is the same model with a different prompt, so their errors can correlate. About 16% of rows that pass one verifier wording fail a second wording. Real requests are more varied; see the spot-check above.
• One fictional organisation's world. The organisation-specific examples are telecom-flavoured: customer phone numbers, call records, cell sites and roaming. General office work transfers better than sector vocabulary, and sector-specific identifiers were never seen.
• Personal data without an identifier is the main residual under-call. Examples are "employee 4471", "her salary" and "his leave". Pattern DLP does not catch these either.
• Sovereign-only is the least certain level, and weaker on long pastes. Declared classifications and application floors should remain the primary protection for that data.
• Complexity is the weakest head (macro-F1 0.82). Use it only at high confidence.
• No task class for translation or comparison. These requests land in qa or analysis.
• Truncation. Only the first 1,024 tokens are read.
• Calibration was fitted on synthetic validation data. Re-fit the temperatures on your own data before relying on the probabilities.
• It sees the conversation only, not retrieved documents or tool results.

Reference deployment: AiVa

AiVa (AI Value Assurance) is a proof-of-concept enterprise AI gateway, run against a fictional telecom tenant with synthetic data only. Every request passes policy, DLP and quota checks. RLCD then chooses the lowest-cost execution plan that is good enough, but only among the models that policy already allows. In AiVa:

• The model runs in the gateway process on CPU, in fp32, loaded once at a pinned revision of this repository.
• Sensitivity uses the tighten rule at 0.5. The effective classification is the most restrictive of the declared level, the application's floor, DLP and RLCD. Policy is then re-checked.
• Task and context need are used at ≥ 0.60, and complexity at ≥ 0.80. Below those, a rule-based router decides for that head. A task declared by the caller overrides the task head.
• needs-packaging sends the context to a compression model, s-ttp/aiva-cim-qwen3-4b-v7, unless the data is confidential or above. That data takes a direct path to a sovereign model.
• If the model is unavailable, misses its 2-second budget or fails, the run is routed by the rule-based router and labelled as such.

Provenance and reproducibility

• Pinned inputs.
  • Base model: answerdotai/ModernBERT-base@8949b909ec900327062f0ebf497f51aef5e6f0c8.
  • Dataset: s-ttp/aiva-rlcd-labels-v2@7c3c097 (not published).
• Generator. openai/gpt-oss-120b, served through the Hugging Face router, wrote both the request wording and the verifier's judgements. It was chosen because it is not one of the models that AiVa routes to.
• rlcd_config.json records the architecture, pooling, heads and class order, the temperatures and the maximum length.
• Source code. AiVa's own training, evaluation and serving code is not public. training/ is a standalone port of the training recipe. The snippets on this card are complete for loading, scoring, recalibration and fine-tuning.