File size: 2,075 Bytes
aef7584 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | ---
license: apache-2.0
library_name: transformers
tags:
- reward-model
- rlhf
- alignment
---
# BestRewardModel
<div align="center">
<img src="figures/training_curve.png" width="70%" alt="Training Curve" />
</div>
## Model Description
This is a reward model trained for RLHF alignment, selected from multiple experimental runs based on validation accuracy and reward alignment quality.
## Selection Criteria
The best checkpoint was chosen according to:
- **Highest `val_accuracy`** among all final checkpoints
- **Minimum `reward_alignment_score` threshold of 0.80**
Only checkpoints satisfying **both** conditions were eligible.
## Training Runs Comparison
| Run | Base Model | Learning Rate | Final Step | Val Accuracy | Reward Alignment | Train Loss |
|-----|-----------|--------------|------------|-------------|-----------------|------------|
| run_gpt2_base_lr1e4 | GPT-2 Base | 1e-4 | 1000 | 0.907 | 0.876 | 0.115 |
| run_gpt2_base_lr5e5 | GPT-2 Base | 5e-5 | 1000 | 0.870 | 0.839 | 0.207 |
| run_gpt2_large_lr1e4 | GPT-2 Large | 1e-4 | 1000 | 0.958 | 0.928 | 0.061 |
| run_gpt2_large_lr5e5 | GPT-2 Large | 5e-5 | 1000 | 0.901 | 0.854 | 0.159 |
| run_deberta_lr1e4 | DeBERTa-v2 | 1e-4 | 1000 | 0.837 | 0.827 | 0.301 |
## Best Run Metrics
| Metric | Value |
|--------|-------|
| Run Name | run_gpt2_large_lr1e4 |
| Val Accuracy | 0.958 |
| Reward Alignment Score | 0.928 |
| Final Train Loss | 0.061 |
## Intended Uses
This model is intended for use as a reward model in RLHF pipelines to score and rank model outputs based on human preference alignment.
## How to Use
```python
from transformers import AutoModelForSequenceClassification, AutoTokenizer
model = AutoModelForSequenceClassification.from_pretrained("BestRewardModel-TestRepo")
tokenizer = AutoTokenizer.from_pretrained("BestRewardModel-TestRepo")
inputs = tokenizer("prompt", "response", return_tensors="pt")
score = model(**inputs).logits[0].item()
```
<div align="center">
<img src="figures/reward_dist.png" width="60%" alt="Reward Distribution" />
</div>
## License
Apache-2.0
|