thehat-word-difficulty

Predicts the difficulty rating E of a Russian noun for the word game «Шляпа» (thehat.tech). E is an Elo-style rating fitted from ~14k words × ~80 plays each of real game data: after every game the words are ranked by how quickly they were explained and guessed, and their ratings are updated TrueSkill-style. Typical values run from ~35 (кровать, луна) to ~65 (капонир, самовоспроизводство); dictionary mean is 52.8, sd 5.8.

Architecture

DeepPavlov/rubert-base-cased encoder → mean pooling over tokens → concat with two scalar features (z-scored log corpus frequency, has-frequency flag) → MLP head (768+2 → 128 → GELU → dropout → 1). Targets are z-scored E values; the checkpoint stores the de-normalization constants.

The frequency table (word_frequency.json, uses-per-million, 14,842 entries) ships with the model and is required at inference; unknown words fall back to a learned "no frequency" path.

Training

• Data: 13,799 words with played-in ratings, split 80/10/10 (train/val/test)
• 8 epochs max, AdamW (encoder lr 3e-5, head lr 1e-3), linear schedule with 6% warmup, batch 32, MSE loss, early stopping on validation MAE (patience 3; best epoch was #2)

Evaluation (held-out test set, n = 1,379)

Metric	This model	Predict-the-mean baseline
MAE (E units)	3.31	4.76
RMSE	4.25	5.77
Pearson r	0.68	—
Spearman ρ	0.67	—
Pairwise "which word is harder" accuracy	74.2%	50%
… on pairs differing by > 4 E points	83.1%	50%

A 3-seed ensemble of this architecture reaches MAE 3.25 / 84% pairwise; this repo ships the best single seed. Residual error concentrates in words whose difficulty comes from world knowledge or in-game explainability rather than form or frequency (кадык, тюлька, молотилка).

Usage

from huggingface_hub import snapshot_download
import sys

path = snapshot_download("nzinov/thehat-word-difficulty")
sys.path.insert(0, path)
from modeling import WordDifficultyPredictor

predictor = WordDifficultyPredictor.from_dir(path)
print(predictor.predict(["кровать", "синоним", "соразмерность"]))
# [44.2, 57.6, 59.5]

Requires torch, transformers, numpy.