tagline-quality-e5-ranker

A SaaS-tagline quality ranker: intfloat/e5-large-v2 fine-tuned end-to-end with a pairwise margin (ranking) loss over (better, worse) tagline pairs — built from a few hundred rubric-rated SaaS hero taglines plus hand-picked hard negatives (category labels, buzzword filler, body-text fragments, prompt/role leakage).

It scores how good a hero tagline is: a sharp, ownable promise scores high; a bare category label ("an all-in-one platform for teams") scores low.

▶ Try it live, entirely in your browser: standd/tagline-rater

Results

model	held-out pairwise acc	AUROC (strong vs weak)
frozen e5 + linear head	0.75	0.73
this model (fine-tuned end-to-end)	0.90	0.87

Fine-tuning the whole embedding body — not just a frozen head — is what breaks past the ~0.80 frozen-embedding ceiling.

Usage (transformers)

The repo ships the e5 body; the quality head is a tiny linear layer in ranker_head.npz (a 1024-dim coef + scalar intercept):

import numpy as np, torch, torch.nn.functional as F
from huggingface_hub import hf_hub_download
from transformers import AutoModel, AutoTokenizer

repo = "standd/tagline-quality-e5-ranker"
tok = AutoTokenizer.from_pretrained(repo)
body = AutoModel.from_pretrained(repo).eval()
h = np.load(hf_hub_download(repo, "ranker_head.npz"))
coef, intercept = h["coef"], h["intercept"]

def score(texts):
    e = tok(["query: " + t for t in texts], padding=True, truncation=True, max_length=64, return_tensors="pt")
    with torch.no_grad():
        o = body(**e).last_hidden_state
        m = e["attention_mask"].unsqueeze(-1).float()
        emb = F.normalize((o * m).sum(1) / m.sum(1).clamp(min=1e-9), dim=1).numpy()
    return emb @ coef.T + intercept   # higher = better tagline

print(score(["Wake up an expert.", "An all-in-one platform for teams."]))

train_scores.npz holds the model's scores over the labeled set — map a raw score s to a 0–100 percentile with (train_scores < s).mean() * 100.

In the browser (transformers.js)

The onnx/ folder has the model for transformers.js: model_quantized.onnx (int8, ~336 MB), model_fp16.onnx, and model.onnx (fp32).

import { pipeline } from "@huggingface/transformers";
const extractor = await pipeline("feature-extraction", "standd/tagline-quality-e5-ranker", { device: "wasm", dtype: "q8" });
const emb = await extractor("query: " + tag, { pooling: "mean", normalize: true });
// then dot emb with ranker_head.npz's coef + intercept

Files

file	what
model.safetensors	fine-tuned e5-large body
ranker_head.npz	linear quality head — coef (1×1024), intercept
train_scores.npz	labeled-set scores, for percentile mapping
onnx/	model.onnx (fp32), model_fp16.onnx, model_quantized.onnx (int8)

Built by the team behind Hey Lefty.