README.md

---
base_model: bigcode/tiny_starcoder_py
library_name: transformers
model_name: tinystarcoder-reward-tldr

tags:
- reward-model
- reward-trainer
- trl
- transformers
- rlhf
- preference-learning
- summarization

license: apache-2.0
---

# TinyStarCoder Reward Model (TL;DR Preference Model)

This model is a reward model fine-tuned from `bigcode/tiny_starcoder_py` using TRL's `RewardTrainer`.

The model predicts a **single scalar reward score** for an input sequence and is intended for **preference ranking**, not text generation.

Higher reward → model prefers that response.

---

## Model Details

### Base Model
- `bigcode/tiny_starcoder_py`

### Task
- Reward Modeling
- Preference Learning
- RLHF-style reward estimation

### Framework
- Transformers
- TRL RewardTrainer

---

## Dataset

Dataset used:

- `CarperAI/openai_summarize_comparisons`

Training examples contain:

```text
prompt
chosen
rejected
```

Training objective:

```text
reward(chosen) > reward(rejected)
```

---

## Training Configuration

| Parameter | Value |
|---|---|
| Samples | 2000 |
| Epochs | 2 |
| Max Length | 256 |
| Learning Rate | 1e-5 |
| Train Batch Size | 2 |
| Eval Batch Size | 1 |
| Trainer | RewardTrainer |

---

## Evaluation

Final evaluation metrics:

| Metric | Value |
|---|---|
| Eval Accuracy | ~0.62 |
| Eval Loss | ~0.98 |
| Eval Margin | ~0.75 |

Interpretation:

- Accuracy > 0.50 indicates the reward model learned preference signal.
- Positive margin means preferred responses generally receive higher reward.

---

## Usage

### Load model

```python
from transformers import (
    AutoTokenizer,
    AutoModelForSequenceClassification
)

repo = "caffeic/tinystarcoder-reward-tldr"

tokenizer = AutoTokenizer.from_pretrained(repo)

model = AutoModelForSequenceClassification.from_pretrained(
    repo
)
```

---

### Score a response

```python
import torch

text = """
Summarize:
Transformers are deep learning architectures...

Summary:
Transformers use self-attention.
"""

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    max_length=256
)

with torch.no_grad():
    reward = model(**inputs).logits.item()

print("Reward:", reward)
```

---

### Compare two responses

```python
chosen_score = score(chosen)
rejected_score = score(rejected)

if chosen_score > rejected_score:
    print("Chosen preferred")
else:
    print("Rejected preferred")
```

---

## Limitations

- This is a reward model and does not generate text.
- Reward values are relative and not absolute quality scores.
- Trained on a limited subset (~2000 samples).
- Not intended for production RLHF pipelines.

---

## Training Notes

This project was created to learn:

- Reward modeling
- Preference datasets
- TRL RewardTrainer
- RLHF workflows
- Hugging Face model publishing

---

## Citation

```bibtex
@software{vonwerra2020trl,
title={{TRL: Transformers Reinforcement Learning}},
author={von Werra et al.},
year={2020},
url={https://github.com/huggingface/trl}
}
```