kasraahmadi/job-eval-modernbert

job-eval — resume ↔ job-description fit scorer (ModernBERT)

A small, fast cross-encoder that reads a (resume, job description) pair and returns four interpretable scores from 1 to 5, plus the paragraph it weighted most for each score. It is the model behind job-eval.com and the open-source CLI at github.com/kasraahmadi/job-eval.

• 156M parameters. Runs in a few seconds on a laptop CPU; instantly on a GPU.
• No API, no frontier LLM. One forward pass, fully local, free.
• Explainable. A trained per-aspect attention pooler highlights the exact paragraph that drove each score — not a post-hoc guess.

The four scores

Aspect	Question it answers
requirement_coverage	How well the resume covers the skills/responsibilities the job asks for
seniority_fit	Whether the candidate's experience level matches the role's seniority
domain_alignment	How closely the candidate's industry/problem domain matches the job
ats_keyword_overlap	Overlap of concrete keywords an ATS screener would look for

Each is in [1, 5]; the product reports their average and a plain-language band (Strong match / Promising fit / Stretch fit / Low alignment).

How it was trained

• Base encoder: Alibaba-NLP/gte-reranker-modernbert-base, a long-context (8k) ModernBERT cross-encoder. The top 2 transformer layers plus the poolers and heads are trained; the rest of the encoder is frozen (~16.5M trainable of 156M).
• Labels: produced by an LLM judge over a large corpus of (resume, JD) pairs, each rated on the four aspects. seniority_fit labels are missing-not-at-random for ~34% of pairs (masked in the loss), which is why that head is the weakest.
• Architecture: the encoder's token states feed a per-aspect attention pooler (LayerNorm + across-token centring + α-entmax(1.5)), which produces a sparse, readable attention map per aspect. Pooled resume and JD vectors are combined ([z_r, z_j, z_r−z_j, z_r·z_j] plus the reranker relevance logit) and read out by four small MLP heads.
• Objective: masked MSE on the four scores + an entropy penalty on the attention so it stays peaked and legible instead of collapsing to uniform (mean-pooling). Trained at max_len=1024; the encoder handles longer contexts if you raise it.

Results

Test MSE (mean over the four aspects), lower is better; baseline = predicting the training mean:

	baseline	this model
test MSE	1.48	0.456

Variance explained per aspect: 74.6% / 51.6% / 76.7% / 71.7% (requirement_coverage / seniority_fit / domain_alignment / ats_keyword_overlap).

On a held-out benchmark of 100 hand-scored synthetic pairs (balanced tech / non-tech and strong / mid / no-fit), the average score tracks human judgement at Pearson r ≈ 0.90, Spearman ρ ≈ 0.84, quadratic-weighted κ ≈ 0.83, with 99% of predictions within one point and strong-vs-mismatch pairs ranked perfectly.

Usage

The easiest path is the CLI — point it at your resume (PDF or text) and a list of jobs, and it writes a scored CSV plus a readable explanation file:

git clone https://github.com/kasraahmadi/job-eval
cd job-eval && pip install -r requirements.txt
python run.py --resume my_cv.pdf --jobs jobs.csv

Or load it directly in Python (needs model.py and segment.py from this repo):

import json, numpy as np, torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_model
from transformers import AutoTokenizer
from model import ResumeJDScorer  # model.py from this repo

repo = "kasraahmadi/job-eval-modernbert"
cfg = json.load(open(hf_hub_download(repo, "config.json")))
tok = AutoTokenizer.from_pretrained(cfg["encoder"])
model = ResumeJDScorer(
    cfg["encoder"], tok.cls_token_id, tok.sep_token_id, tok.pad_token_id,
    np.asarray(cfg["label_mean"], dtype=np.float32),
    n_aspects=len(cfg["aspects"]), a_dim=cfg["a_dim"], hidden=cfg["hidden"],
    dropout=cfg["dropout"], n_train_layers=cfg["n_train_layers"],
    use_rel_logit=cfg["use_rel_logit"], attn=cfg["attn"],
    center=cfg["center_pooler"], temperature=cfg["temperature"],
    score_init=cfg["score_init"],
).eval()
load_model(model, hf_hub_download(repo, "model.safetensors"))

enc = tok(resume_text, jd_text, truncation="longest_first",
          max_length=cfg["max_len"], return_tensors="pt")
with torch.no_grad():
    scores = model(enc["input_ids"], enc["attention_mask"])[0].clamp(1, 5)
print(dict(zip(cfg["aspects"], scores.tolist())))

Intended use & limitations

• A triage aid, not a hiring decision. It ranks and explains fit; it does not and must not decide who to hire. Keep a human in the loop.
• Trained on English resumes/JDs; other languages are out of distribution.
• seniority_fit is the least reliable head (51.6% variance explained) because its labels are sparse.
• The attention highlight shows where the pooler looked, which is a faithful view of the model's own weighting — but attention is not a causal attribution.
• Like any resume model it can reflect biases in its training labels. Do not use it for automated rejection or any decision requiring demographic fairness guarantees.

License

Apache-2.0, matching the base encoder. Weights © Kasra Ahmadi.