A RoBERTa-large model fine-tuned to predict how funny a joke is, trained on 340k cleaned Reddit jokes from the rJokes dataset (Weller & Seppi, LREC 2020). Given a joke as input, the model outputs a single scalar: a predicted humor score on the dataset's 0–11 log-compressed community rating scale.

Results

Evaluated on a leakage-cleaned test set. On a comparable (leaked) evaluation, this model surpasses the rJokes paper's baselines.

Model	Params	Test Spearman	Test Pearson	Test RMSE
TF-IDF + Ridge	—	0.363	—	—
DistilBERT-128	66M	0.4118	0.4513	1.6426
RoBERTa-base-128	125M	0.4187	0.4510	1.7047
BERT-large	340M	0.423	0.463	1.657
RoBERTa-large-128	355M	0.4323	0.4701	1.6298

Paper evaluates on a test set containing ~2.4% cross-split leakage. On a comparably leaked eval, this model scores Spearman 0.4404 & Pearson 0.4781, exceeding the paper's reported numbers. The remaining gap on the clean eval is attributable to leakage inflation in the paper's results.

Leakage-cleaned evaluation

The original rJokes splits contain ~2.4% of test jokes that are exact copies of training jokes (Reddit reposts). Prior work evaluated on these leaked splits. I remove the overlap and report on the clean test set (41,957 examples). I quantified the impact:

Model	Clean Spearman	Leaky Spearman	Inflation	Paper Spearman
DistilBERT (66M)	0.412	0.421	0.009	—
RoBERTa-base (125M)	0.419	0.426	0.007	—
BERT-large (340M)	0.423	0.431	0.009	0.430
RoBERTa-large (355M)	0.432	0.440	0.008	0.435

The average inflation is 0.008 ± 0.001 Spearman, consistent across four architectures of different sizes, confirming it is a dataset property and not a model-specific artifact.

Training details

Base model: roberta-large (355M parameters)
Task: Single-value regression (num_labels=1, problem_type="regression")
Dataset: rJokes, cleaned (339,499 train / 41,941 dev / 41,957 test)
Cleaning: removed 5,707 exact duplicates, ultra-short fragments (<5 words), and ~2.4% cross-split leakage from dev/test
Max sequence length: 128 tokens
Epochs: 5 (best checkpoint selected by dev Spearman; dev Spearman 0.4343)
Effective batch size: 32 (constant across single and multi-GPU setups)
Learning rate: 2e-5 with 6% linear warmup
Weight decay: 0.01
Precision: fp16
Optimizer: AdamW (Hugging Face default)
Seed: 42
Hardware: Kaggle T4 ×2, ~3 sessions totaling ~30h (12h session limit)

Label note

The rJokes score column is already log-scaled: round(ln(raw_upvotes + 1)), giving integers 0–11 (the paper reports 0–10; labels of 11 are rare but present in the data). It is used directly as the regression target. Do not log-transform again. This follows the paper's Section 3.1, which reduces the raw scale (0–136,353) down to integers 0–11 (the paper reports 0–10; labels of 11 are rare but present in the data)".

Usage

python

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

repo = "iamahmadyasin/humor-roberta-large"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo)
model.eval()

joke = "I told my wife she was drawing her eyebrows too high. She looked surprised."
inputs = tokenizer(joke, return_tensors="pt", truncation=True, max_length=128)
with torch.no_grad():
    score = model(**inputs).logits.item()
print(f"Predicted humor score: {score:.2f}")

Limitations

Humor is subjective; the labels reflect one Reddit community's preferences, shaped by timing and virality as much as joke quality.
The model regresses to the mean and is unreliable at the extremes of the score range (rarely predicts 0 or 6+).
Trained on English-language Reddit jokes only.
This is a humor ranker, not a judge of objective funniness.

Citation

Dataset:

bibtex

@inproceedings{weller-seppi-2020-rjokes,
    title     = "The rJokes Dataset: a Large Scale Humor Collection",
    author    = "Weller, Orion and Seppi, Kevin",
    booktitle = "Proceedings of the 12th Language Resources and Evaluation Conference (LREC)",
    year      = "2020",
    pages     = "6136--6141",
    url       = "https://aclanthology.org/2020.lrec-1.753/",
}

Project

Full project: github.com/iamahmadyasin/humor-intelligence