yapwithai/yap-modernbert-screenshot-intent

Screenshot Intent Classifier

This repository contains a ModernBERT-based classifier fine-tuned to decide whether a conversational agent should trigger a screenshot tool for the latest user message.

Base Model

This model is fine-tuned from answerdotai/ModernBERT-base, and inherits the base encoder's maximum context length and tokenizer. ModernBERT-base pairs a fast, memory-efficient encoder with strong long-context fine-tuning and inference characteristics. It supports long inputs without requiring manual global attention masks, making it a drop-in replacement for standard encoder classifiers.

Classifier

• 0 / no_screenshot: do not call the screenshot tool.
• 1 / take_screenshot: call the screenshot tool.

The input is a text block representing the recent conversation history, formatted as one utterance per line, prefixed with a speaker tag, e.g.:

USER: I'm wondering if blue goes well with yellow.
USER: What's your take on this?

At inference time, the host application typically feeds the last few conversation turns (most importantly the latest user message) in this format and thresholds the classifier's take_screenshot probability to decide whether to trigger the tool.

Training Data

The classifier was trained on a curated, hand-labelled private dataset. It contains hundreds of single-turn and multi-turn examples specifying whether each user message should or should not trigger a screenshot, including:

• Clear positive triggers ("look at this", "check this out", "rate this pic").
• Clear negatives (off-topic chit-chat, abstract statements, idioms like "I'll look into it").
• Edge cases involving deictic pronouns, quantities ("take 2 screenshots"), negation ("don't look"), multi-turn context, and more.

No external user logs or third-party datasets were used; the training data is purely synthetic / curated for this intent task.

Training Setup

Approximate defaults:

• Epochs: 3
• Batch size: 16 (per device)
• Learning rate: 2e-5
• Weight decay: 0.01
• Max sequence length: 1536 tokens (truncation for old entries applied beyond this)

The script builds examples by concatenating conversation history up to and including the current user message, one utterance per line prefixed with "USER:". Multi-turn conversations therefore become multiple training examples with growing context.

Usage

Basic usage with the Transformers library:

from transformers import AutoTokenizer, AutoModelForSequenceClassification

MODEL_ID = "yapwithai/yap-modernbert-screenshot-intent"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID)
model.eval()

text = "USER: look at this amazing sunset"
inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    padding="max_length",
    max_length=1536,
)

with torch.no_grad():
    outputs = model(**inputs)
    probs = outputs.logits.softmax(dim=-1)[0]

p_no, p_yes = probs.tolist()
print("P(no_screenshot)=", p_no)
print("P(take_screenshot)=", p_yes)

In production, you would:

• Construct a conversation history string similar to the training format (recent user turns, optionally assistant turns, each on its own line with a speaker prefix).
• Run the classifier once per latest user message.
• Threshold p_yes to decide whether to trigger the screenshot tool.

ModernBERT Citation

If you use ModernBERT in your work, please cite:

@misc{modernbert,
      title={Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference},
      author={Benjamin Warner and Antoine Chaffin and Benjamin Clavié and Orion Weller and Oskar Hallström and Said Taghadouini and Alexis Gallagher and Raja Biswas and Faisal Ladhak and Tom Aarsen and Nathan Cooper and Griffin Adams and Jeremy Howard and Iacopo Poli},
      year={2024},
      eprint={2412.13663},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2412.13663},
}

ModernBERT is developed by Answer.AI and collaborators.