iamahmadyasin/humor-roberta-base

---

Humor Intelligence — RoBERTa-base

A RoBERTa-base model fine-tuned to predict how funny a joke is, trained on 340k cleaned Reddit jokes from the rJokes dataset (Weller & Seppi, LREC 2020).

Given a joke as input, the model outputs a single scalar which is a predicted humor score on a 0–11 scale (the dataset's log-compressed community rating).

Results

Evaluated on a leakage-cleaned test set (see below).

Model	Params	Test Spearman	Test Pearson	Test RMSE
TF-IDF + Ridge	—	0.363	—	—
DistilBERT-128	66M	0.4118	0.4513	1.6426
RoBERTa-base-128	125M	0.4187	0.4510	1.7047
roBERTa-large (paper†)	355M	0.435	0.474	1.614

Paper evaluates on a test set containing ~2.4% cross-split leakage (see below). On a comparably leaked eval, this model scores Spearman 0.4260, narrowing the gap from 0.016 to 0.009.

Leakage-cleaned evaluation

The original rJokes splits contain ~2.4% of test jokes that are exact copies of training jokes (Reddit reposts). Prior work evaluated on these leaked splits. We remove the overlap and report on the clean test set (41,957 examples), with a separate leakage-impact analysis quantifying the inflation.

Training details

• Base model: roberta-base (125M parameters)
• Task: Single-value regression (num_labels=1, problem_type="regression")
• Dataset: rJokes, cleaned (339,499 train / 41,941 dev / 41,957 test)
• Cleaning: removed 5,707 exact duplicates, ultra-short fragments (<5 words), and ~2.4% cross-split leakage from dev/test
• Max sequence length: 128 tokens
• Epochs: 5
• Effective batch size: 32
• Learning rate: 2e-5 with 6% linear warmup
• Weight decay: 0.01
• Precision: fp16
• Optimizer: AdamW (Hugging Face default)
• Seed: 42
• Hardware: Kaggle T4 ×2, ~7.5 hours

Label note

The rJokes score column is already log-scaled: round(ln(raw_upvotes + 1)), giving integers 0–11. It is used directly as the regression target. Do not log-transform again. This follows the paper's Section 3.1, which reduces the raw scale (0–136,353) down to integers 0–11 (the paper reports 0–10; labels of 11 are rare but present in the data).

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

repo = "iamahmadyasin/humor-roberta-base"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo)
model.eval()

joke = "I told my wife she was drawing her eyebrows too high. She looked surprised."
inputs = tokenizer(joke, return_tensors="pt", truncation=True, max_length=128)
with torch.no_grad():
    score = model(**inputs).logits.item()
print(f"Predicted humor score: {score:.2f}")

Limitations

• Humor is subjective; the labels reflect one Reddit community's preferences, shaped by timing and virality as much as joke quality.
• The model regresses to the mean and is unreliable at the extremes of the score range (rarely predicts 0 or 6+).
• Trained on English-language Reddit jokes only.
• This is a humor ranker, not a judge of objective funniness.

Citation

Dataset:

@inproceedings{weller-seppi-2020-rjokes,
    title     = "The rJokes Dataset: a Large Scale Humor Collection",
    author    = "Weller, Orion and Seppi, Kevin",
    booktitle = "Proceedings of the 12th Language Resources and Evaluation Conference (LREC)",
    year      = "2020",
    pages     = "6136--6141",
    url       = "https://aclanthology.org/2020.lrec-1.753/",
}

Project

Full project: github.com/iamahmadyasin/humor-intelligence