iamahmadyasin/humor-distilbert

Humor Intelligence — DistilBERT

A DistilBERT model fine-tuned to predict how funny a joke is, trained on 340k cleaned Reddit jokes from the rJokes dataset (Weller & Seppi, LREC 2020).

Given a joke as input, the model outputs a single scalar: a predicted humor score on a 0–11 scale (the dataset's log-compressed community rating). This is the lightweight rung of the model ladder (66M parameters), half the size of RoBERTa-base, yet within 0.007 Spearman of it.

Results

Evaluated on a leakage-cleaned test set (see below).

Model	Params	Test Spearman	Test Pearson	Test RMSE
TF-IDF + Ridge	—	0.363	—	—
DistilBERT-128	66M	0.4118	0.4513	1.6426
RoBERTa-base-128	125M	0.4187	0.4510	1.7047
roBERTa-large (paper†)	355M	0.435	0.474	1.614

Paper evaluates on a test set containing ~2.4% cross-split leakage. On a comparably leaked eval, DistilBERT scores Spearman 0.4210.

Despite having half the parameters of RoBERTa-base, DistilBERT achieves nearly the same ranking performance (Spearman 0.4118 vs 0.4187) and actually produces better-calibrated magnitude predictions (RMSE 1.6426 vs 1.7047).

Leakage-cleaned evaluation

The original rJokes splits contain ~2.4% of test jokes that are exact copies of training jokes (Reddit reposts). Prior work evaluated on these leaked splits. We remove the overlap and report on the clean test set (41,957 examples), with a separate leakage-impact analysis quantifying the inflation.

Training details

• Base model: distilbert-base-uncased (66M parameters)
• Task: Single-value regression (num_labels=1, problem_type="regression")
• Dataset: rJokes, cleaned (339,499 train / 41,941 dev / 41,957 test)
• Cleaning: removed 5,707 exact duplicates, ultra-short fragments (<5 words), and ~2.4% cross-split leakage from dev/test
• Max sequence length: 128 tokens
• Epochs: 5 (best checkpoint at epoch 3 by dev Spearman)
• Effective batch size: 32 (constant across single and multi-GPU setups)
• Learning rate: 2e-5 with 6% linear warmup
• Weight decay: 0.01
• Precision: fp16
• Optimizer: AdamW (Hugging Face default)
• Seed: 42
• Hardware: Kaggle T4 ×2, ~4 hours

Label note

The rJokes score column is already log-scaled: round(ln(raw_upvotes + 1)), giving integers 0–11". It is used directly as the regression target. Do not log-transform again. This follows the paper's Section 3.1, which reduces the raw scale (0–136,353) down to integers 0–11 (the paper reports 0–10; labels of 11 are rare but present in the data)".

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

repo = "iamahmadyasin/humor-distilbert"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo)
model.eval()

joke = "I told my wife she was drawing her eyebrows too high. She looked surprised."
inputs = tokenizer(joke, return_tensors="pt", truncation=True, max_length=128)
with torch.no_grad():
    score = model(**inputs).logits.item()
print(f"Predicted humor score: {score:.2f}")

Limitations

• Humor is subjective; the labels reflect one Reddit community's preferences, shaped by timing and virality as much as joke quality.
• The model regresses to the mean and is unreliable at the extremes of the score range (rarely predicts 0 or 6+).
• Trained on English-language Reddit jokes only.
• This is a humor ranker, not a judge of objective funniness.

Citation

Dataset:

@inproceedings{weller-seppi-2020-rjokes,
    title     = "The rJokes Dataset: a Large Scale Humor Collection",
    author    = "Weller, Orion and Seppi, Kevin",
    booktitle = "Proceedings of the 12th Language Resources and Evaluation Conference (LREC)",
    year      = "2020",
    pages     = "6136--6141",
    url       = "https://aclanthology.org/2020.lrec-1.753/",
}

Project

Full project: github.com/iamahmadyasin/humor-intelligence