--- license: mit base_model: microsoft/deberta-v3-base tags: - reward-model - preference-learning - bradley-terry - deberta-v2 datasets: - HuggingFaceH4/ultrafeedback_binarized pipeline_tag: text-classification --- # DeBERTa-v3 Preference Reward Model Bradley-Terry reward model for scoring LLM response quality: given a `(prompt, response)` text pair, outputs a scalar reward that is meaningful **only relative to another response to the same prompt** (higher = more preferred; the scale has an arbitrary additive offset — see [Usage](#usage)). Built as part of [EvalForge](https://github.com/Larmstrong1127/evalforge), where it runs as the `reward` judge — the platform's first judge that needs no golden answer. ## Intended use - **Primary:** rank or score LLM responses when there is no reference answer (open-ended generation), inside EvalForge's `reward` judge or any pipeline that needs a cheap, local relative-quality signal. - **Out of scope:** a stand-in for a specific human's preferences. It predicts *UltraFeedback-style* (AI-feedback) preferences and carries that data's known length/elaboration bias — see Evaluation. ## Training - `microsoft/deberta-v3-base` with a 1-dim regression head, hand-written PyTorch Bradley-Terry loop: `L = -log sigmoid(r_chosen - r_rejected)`. - Single epoch on `HuggingFaceH4/ultrafeedback_binarized` `train_prefs` (60,700 pairs), 512-token budget with an audited truncation safety net (any pair whose chosen/rejected encodings become identical after truncation is dropped and counted: 1 of 62,688 across train+eval). - AMP + dynamic per-side padding + TF32 on a single RTX 3090; lr 2e-5, linear warmup 6%. An lr 5e-5 run collapsed to chance (0.51 pairwise accuracy) and was discarded. ## Calibration Raw Bradley-Terry logits are arbitrarily scaled, so a scalar temperature **T = 1.167** was fit post-hoc on the held-out split (NLL of `sigmoid(margin / T)`) at the same 512-token budget used for training, and stored in `config.json` as `reward_temperature` (with the budget it was fit under alongside it as `reward_train_max_length`; see Correction below). Because T was fit on **margins**, the quantity it calibrates is `sigmoid((r_a - r_b) / T)` — the probability that A is preferred to B for the same prompt. It does **not** calibrate `sigmoid(r / T)` for a single response; see the warning under [Usage](#usage). Pairwise accuracy is invariant to T; only the sharpness of the probability depends on it. ## Evaluation, honestly All rows below are the **same split** (UltraFeedback `test_prefs`, N=1,987 after the truncation audit) run through the **same harness** (`training/eval_reward.py` and `training/eval_reward_baseline.py`, which share `evaluate_pairs`), at each model's own 512-token budget — except where noted. | Model / split | Params | N | Pairwise accuracy | |---|---|---|---| | Chance floor (balanced binary choice) | — | — | 0.5000 | | `OpenAssistant/reward-model-deberta-v3-large-v2` (public baseline) | 435M | 1,987 | 0.6009 | | lr 5e-5 run (collapsed, discarded) | 184M | 1,987 | 0.5098 | | **This model** — UltraFeedback `test_prefs` (in-distribution) | 184M | 1,987 | **0.7026** | | Human OOD probe (EvalForge rating room) | 184M | 15 | 0.4000 | ### Reading the baseline row honestly This model beats a public reward model 2.4x its size by **+10.2 points**, and that comparison is **not** a claim that it is the better reward model. It is in-distribution and the baseline is out-of-distribution: - This model was trained on UltraFeedback `train_prefs` and is being scored on UltraFeedback `test_prefs`. Same annotator (an LLM), same prompt mix, same elaboration conventions. - `reward-model-deberta-v3-large-v2` was trained on a different preference mixture entirely (WebGPT, summarize-from-feedback, synthetic-instruct, Anthropic HH). UltraFeedback is a distribution shift for it. So the correct reading is: **0.7026 is a real number, not a collapsed one** (the floor is 0.5000 and a lr-sweep failure sat at 0.5098), and a strong public model transferred onto this distribution lands at 0.6009. The honest inverse of this result is already reported above — on the human OOD probe *this* model drops to chance. Neither model generalizes for free; each is good on the distribution it was fit to. The tradeoff this project deliberately explored is a **small, local, free** judge (184M, ~40ms/response on CPU, no API key, no per-call cost) against larger models and hosted LLM judges. The baseline row exists so that tradeoff is stated with a number instead of asserted. Reproduce: ```bash python training/eval_reward.py --checkpoint checkpoints/reward-lr2e5 python training/eval_reward_baseline.py \ --model OpenAssistant/reward-model-deberta-v3-large-v2 ``` The OOD probe is 15 genuine blind A/B votes by one human rater on real llama3.2-vs-qwen2.5:14b outputs collected in EvalForge's rating room. At N=15 the result is statistically indistinguishable from chance (95% CI roughly 0.16–0.68), and it is reported as a probe, not a benchmark — but the direction is consistent with the documented length/elaboration bias of AI-feedback preference data: **this model predicts UltraFeedback-style preferences, not any individual human's.** ## RewardBench 2 (2026-08-13): a third-party number, and it is the floor I ran the official RewardBench 2 harness (`allenai/reward-bench` @ `05a9005`, dataset @ `7ff0885`, 1,865 prompts, best-of-4, random baseline **25%** for the five accuracy domains) on this model, unmodified except for a registered dialogue template that reproduces the two-segment training encoding token-for-token (the stock `raw` template drops the `[SEP]` boundary and merges subwords across it). Full protocol and per-domain scores: `training/rewardbench2_results.json`. Device: CPU, float32, 1h18m. | Domain | This model (184M) | OA deberta-v3-large-v2 (435M, official leaderboard) | |---|---:|---:| | Factuality | 28.8 | 38.5 | | Focus | 15.8 | 27.7 | | Math | **47.1** | 50.3 | | Precise IF | 23.1 | 26.9 | | Safety | 35.8 | 36.7 | | Ties* | 1.4 | 12.0 | | **Average** | **25.3** | **32.0** | \* Ties uses a margin-based metric with a chance level well below 25%; do not read it against the 25% floor. **Reading this honestly: out of distribution, this model is at the random floor.** That is not a surprise — it is the strongest evidence yet for what this card already says: the model predicts UltraFeedback-style preferences and does not transfer. The official 435M OpenAssistant DeBERTa — the baseline this model beats by 10 points in-distribution — manages 32.0 here, and encoder-class reward models as a category sit near the floor on this benchmark (the strong entries, 61–84, are all modern decoder-based classifiers). The one domain where a 184M encoder holds up is Math: 47.1, within three points of the 435M baseline at 40% of the size. If you need a general-purpose reward model, use one from the RewardBench 2 leaderboard. If you need a small, free, CPU-viable judge for UltraFeedback-distribution comparisons, that is the niche this model occupies, and these numbers mark its boundary precisely. ## Correction (2026-07-26): train/serve sequence-length mismatch An audit flagged that this model trains and is configured at **512** tokens while three shipped code paths defaulted to **1024**: `training/eval_reward.py`, `training/calibrate_reward.py`, and the platform's `reward_judge.py`. The suspicion was that the headline metrics had been measured off-regime. **Both numbers were re-measured on the same held-out split, and they hold.** The published figures were produced at 512 all along — the operator had passed `--max-length 512` explicitly; only the *defaults* were stale. The re-run reproduces the stored temperature bit-for-bit (`1.166796088218689`), which is conclusive. | Metric | Published | Re-measured @512 | Off-regime @1024 | |---|---|---|---| | ID pairwise accuracy (`test_prefs`, N=1,987) | 0.7026 | **0.7026** | 0.7046 | | Calibration temperature T | 1.167 | **1.1668** | 1.1395 | | Pairs dropped by truncation audit | 1 / 62,688 | **1 / 62,688** | 1 / 62,688 | | OOD probe (N=15) | 0.400 | **0.400** | — | So the *documentation* was correct and the *code* was wrong. The real defect was in serving, not in reporting: the platform judge scored live traffic at 1024 while applying a temperature fit at 512. That is not hypothetical — **39% of `test_prefs` pairs (776 / 1,988) have at least one side exceeding 512 tokens**, so the judge routinely fed the model context it never saw in training. DeBERTa-v3 uses relative position embeddings, so it degrades gracefully instead of erroring, which is exactly why the mismatch survived review. The measured cost of the off-regime setting is small (+0.20pt accuracy, T off by 0.027) but it was unmeasured, and an unmeasured difference is not a small one. **Fix:** the sequence budget is no longer restated anywhere. It is derived from the checkpoint's own `config.json` — now carrying an explicit `reward_train_max_length: 512` next to `reward_temperature`, so the constant and the regime it was fit under travel together with the weights. ## Limitations - **In-distribution accuracy is modest** (0.70 pairwise) even before any distribution shift — this is a base-size model trained for one epoch. - **Inherits UltraFeedback's length/elaboration bias**; longer, more elaborate answers are systematically favored regardless of correctness. - **Not personalized** — it does not model any individual rater; the OOD probe is at chance. - **512-token cap**; longer `(prompt, response)` pairs truncate. - **No absolute scale.** Bradley-Terry identifies rewards only up to an additive constant, so a single score cannot be thresholded, averaged across a dataset, or compared across prompts. Only within-prompt comparisons are validated. ## What didn't work - Training at 1024 tokens measured 9–12 h/epoch on the 3090 and died at hour 8.5 with `cudaErrorIllegalAddress` before its first checkpoint. Stepped down to 512 tokens (~1.7 h/epoch) with the truncation audit as the guardrail (data loss at 512: 1 pair in 62,688). - The lr sweep's higher setting (5e-5) destabilized training outright — eval accuracy 0.5098, i.e. the model learned nothing. ## License MIT — same as the base model (`microsoft/deberta-v3-base`) and the EvalForge repository. ## Usage This model is validated for **pairwise comparison**. Score two candidate responses to the same prompt and compare them; the calibrated temperature converts the *margin* into a preference probability. ```python import torch from transformers import AutoModelForSequenceClassification, AutoTokenizer repo = "DantheMan124/deberta-preference-reward" tok = AutoTokenizer.from_pretrained(repo) model = AutoModelForSequenceClassification.from_pretrained(repo).eval() T = model.config.reward_temperature # 1.1668 MAX_LEN = model.config.reward_train_max_length # 512 def reward(prompt: str, response: str) -> float: """Raw Bradley-Terry score. Meaningful ONLY relative to another response to the SAME prompt -- the scale carries an arbitrary additive offset. Encoded exactly as training pairs were: the prompt and the response as the two segments of one sequence pair, right-truncated to the training budget. """ enc = tok(prompt, response, truncation=True, max_length=MAX_LEN, return_tensors="pt") with torch.no_grad(): return model(**enc).logits.squeeze().item() prompt = "What causes seasons?" a = ("Earth's axis is tilted about 23.5 degrees relative to its orbital " "plane, so each hemisphere receives sunlight at a steeper angle for " "part of the year.") b = "Because the Earth gets closer to the Sun in summer." r_a, r_b = reward(prompt, a), reward(prompt, b) p_a = torch.sigmoid(torch.tensor((r_a - r_b) / T)).item() print(f"r_a={r_a:.4f} r_b={r_b:.4f} margin={r_a - r_b:.4f}") print(f"P(A preferred over B) = {p_a:.3f}") ``` Verified output on this checkpoint: ```text r_a=-1.0902 r_b=-2.3133 margin=1.2231 P(A preferred over B) = 0.740 ``` > ### ⚠️ Do not use a single score as an absolute quality measure > > `reward(prompt, response)` on its own is **not** a calibrated 0-1 quality > score, and `sigmoid(reward / T)` is not the probability of anything. > > - Bradley-Terry training only ever sees `r_chosen - r_rejected`, so the > objective is **invariant to adding a constant to every reward**. The zero > point is arbitrary. Note that both scores in the example above are > *negative* even though A is the good answer — the sign carries no meaning. > - **T was fit on pairwise margins** (minimizing NLL of > `sigmoid(margin / T)`), so applying it to a bare logit uses a calibration > constant outside the quantity it was calibrated on. > - The model's only validated metric is **pairwise accuracy**. Comparisons > between two responses to the same prompt are in-distribution for how it > was trained, evaluated, and calibrated; absolute scores are not. > > Ranking N candidates for one prompt is fine (the scores are a valid ordering > within a prompt). Comparing scores *across different prompts*, thresholding > them, or averaging them over a dataset is not.