README.md

metadata

title: ColdScore
emoji: 🧲
colorFrom: indigo
colorTo: gray
sdk: gradio
app_file: app.py
pinned: false
license: mit
short_description: Open-weights cold-email deliverability scoring on ModernBERT

ColdScore

Will this cold email reach the inbox — or the spam folder?

A ModernBERT-base model (Apache-2.0, 149M params) fine-tuned on a corpus of 100 hand-labelled cold emails. Open weights, MIT licence, runs on CPU.

Model: Ashsinha1/coldscore-modernbert

Why an open model instead of a frontier API

A frontier model would score these emails better. That isn't the point.

Scoring a cold email means handing over a prospect's name, their email address, and your pitch — customer data. Sending that to a third-party model provider is a decision that shows up in your customers' security review.

This model doesn't force that decision. The weights are public, inference is a CPU process, and it runs inside your own network in four lines:

from transformers import pipeline

clf = pipeline("text-classification", model="Ashsinha1/coldscore-modernbert")
clf({"text": "quick question about onboarding", "text_pair": "Hi Sarah, I noticed..."})
# [{'label': 'deliverable', 'score': 0.98}]

For the half of a pipeline that touches every contact in the database, "good and local" beats "excellent and elsewhere".

Why an encoder rather than an LLM

For binary text classification, a fine-tuned encoder wins on all three axes that matter: accuracy, cost, and latency. ModernBERT is the current state of the art in that family — trained on 2T tokens, 8,192-token context, so a whole email fits without truncation. At 149M parameters it scores an email in well under a second on CPU, where a 1.5B model takes seconds per request.

Training

• Base: answerdotai/ModernBERT-base
• Data: 100 hand-written, hand-labelled cold emails (50 deliverable / 50 spam-prone), expanded to 1,200 rows by filling {first} / {company} placeholders
• Split: grouped by seed email — every variant of an email lands on the same side, so near-duplicates cannot leak across the train/test boundary
• Compute: minutes on a single T4

Limitations — read these before trusting a number

1. Content only. Real inbox placement is dominated by things the model cannot see: domain and IP reputation, SPF/DKIM/DMARC alignment, sending volume, list hygiene, complaint rates. A flawless email from a burned domain still lands in spam.
2. The benchmark is easy. The spam examples are unsubtle. A plain keyword blocklist already scores ~0.94 F1 on this corpus, so held-out numbers are an optimistic upper bound rather than a real-world accuracy estimate. The genuinely hard cases — blunt-but-legitimate outreach, polished promotional blasts — are barely represented.
3. Labels are judgements, not measured outcomes. Nobody sent these 100 emails and observed where they landed.
4. Small corpus, one author's taste. It likely over-rewards a short, curious, low-pressure style.
5. English only. Not a phishing or security detector — well-written phishing scores clean.

Treat it as a content lint, not a deliverability guarantee.

Privacy

Text entered in the demo is processed in this Space's container and is not stored or logged. If that still isn't good enough for your data — a reasonable position, and rather the point of the project — clone the Space or run the model locally.

Licence

MIT for this model and app. The base model is Apache-2.0.