Actionable-BERT

A ModernBERT-based binary classifier for determining whether a conversation should be routed to a larger, more capable agent. The model analyzes conversation history and predicts if the user's request requires advanced reasoning or can be handled by a simpler system.

Model Details

• Base Model: answerdotai/ModernBERT-base
• Task: Binary sequence classification
• Max Sequence Length: 8192 tokens
• Labels:
  • 0: Not Actionable (simple request, no routing needed)
  • 1: Actionable (complex request, route to larger agent)

Input Format

Conversations are formatted as a single string with turn markers:

[U] User message here. [A] Assistant response here. [U] Follow-up user message...

• [U] marks the start of a user turn
• [A] marks the start of an assistant turn
• Turns are concatenated into a single string

Installation

pip install transformers torch

Inference Example

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_name = "jspaulsen/actionable-bert"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
model.eval()

# Format conversation history
conversation = "[U] Can you help me write a Python function? [A] Sure, what would you like the function to do? [U] I need a recursive function to calculate fibonacci numbers with memoization."

inputs = tokenizer(
    conversation,
    return_tensors="pt",
    truncation=True,
    max_length=8192,
)

with torch.no_grad():
    outputs = model(**inputs)
    prediction = torch.argmax(outputs.logits, dim=-1).item()

labels = {0: "Not Actionable", 1: "Actionable"}
print(f"Prediction: {labels[prediction]}")

Dataset

Trained on jspaulsen/actionable-bert-dataset.