Update README.md

README.md

@@ -1,58 +1,190 @@
 ---
 base_model: bigcode/tiny_starcoder_py
 library_name: transformers
-model_name: tinystarcoder-rlhf-model
+model_name: tinystarcoder-reward-tldr
+
 tags:
-- generated_from_trainer
+- reward-model
 - reward-trainer
 - trl
-licence: license
+- transformers
+- rlhf
+- preference-learning
+- summarization
+
+license: apache-2.0
+---
+
+# TinyStarCoder Reward Model (TL;DR Preference Model)
+
+This model is a reward model fine-tuned from `bigcode/tiny_starcoder_py` using TRL's `RewardTrainer`.
+
+The model predicts a **single scalar reward score** for an input sequence and is intended for **preference ranking**, not text generation.
+
+Higher reward → model prefers that response.
+
+---
+
+## Model Details
+
+### Base Model
+- `bigcode/tiny_starcoder_py`
+
+### Task
+- Reward Modeling
+- Preference Learning
+- RLHF-style reward estimation
+
+### Framework
+- Transformers
+- TRL RewardTrainer
+
+---
+
+## Dataset
+
+Dataset used:
+
+- `CarperAI/openai_summarize_comparisons`
+
+Training examples contain:
+
+```text
+prompt
+chosen
+rejected
+```
+
+Training objective:
+
+```text
+reward(chosen) > reward(rejected)
+```
+
+---
+
+## Training Configuration
+
+| Parameter | Value |
+|---|---|
+| Samples | 2000 |
+| Epochs | 2 |
+| Max Length | 256 |
+| Learning Rate | 1e-5 |
+| Train Batch Size | 2 |
+| Eval Batch Size | 1 |
+| Trainer | RewardTrainer |
+
+---
+
+## Evaluation
+
+Final evaluation metrics:
+
+| Metric | Value |
+|---|---|
+| Eval Accuracy | ~0.62 |
+| Eval Loss | ~0.98 |
+| Eval Margin | ~0.75 |
+
+Interpretation:
+
+- Accuracy > 0.50 indicates the reward model learned preference signal.
+- Positive margin means preferred responses generally receive higher reward.
+
 ---
 
-# Model Card for tinystarcoder-rlhf-model
+## Usage
+
+### Load model
+
+```python
+from transformers import (
+    AutoTokenizer,
+    AutoModelForSequenceClassification
+)
+
+repo = "caffeic/tinystarcoder-reward-tldr"
+
+tokenizer = AutoTokenizer.from_pretrained(repo)
+
+model = AutoModelForSequenceClassification.from_pretrained(
+    repo
+)
+```
 
-This model is a fine-tuned version of [bigcode/tiny_starcoder_py](https://huggingface.co/bigcode/tiny_starcoder_py).
-It has been trained using [TRL](https://github.com/huggingface/trl).
+---
 
-## Quick start
+### Score a response
 
 ```python
-from transformers import pipeline
+import torch
+
+text = """
+Summarize:
+Transformers are deep learning architectures...
+
+Summary:
+Transformers use self-attention.
+"""
 
-text = "The capital of France is Paris."
-rewarder = pipeline(model="None", device="cuda")
-output = rewarder(text)[0]
-print(output["score"])
+inputs = tokenizer(
+    text,
+    return_tensors="pt",
+    truncation=True,
+    max_length=256
+)
+
+with torch.no_grad():
+    reward = model(**inputs).logits.item()
+
+print("Reward:", reward)
 ```
 
-## Training procedure
+---
+
+### Compare two responses
 
+```python
+chosen_score = score(chosen)
+rejected_score = score(rejected)
 
+if chosen_score > rejected_score:
+    print("Chosen preferred")
+else:
+    print("Rejected preferred")
+```
 
+---
 
+## Limitations
 
-This model was trained with Reward.
+- This is a reward model and does not generate text.
+- Reward values are relative and not absolute quality scores.
+- Trained on a limited subset (~2000 samples).
+- Not intended for production RLHF pipelines.
 
-### Framework versions
+---
+
+## Training Notes
 
-- TRL: 1.5.0
-- Transformers: 5.0.0
-- Pytorch: 2.11.0+cu128
-- Datasets: 4.8.5
-- Tokenizers: 0.22.2
+This project was created to learn:
 
-## Citations
+- Reward modeling
+- Preference datasets
+- TRL RewardTrainer
+- RLHF workflows
+- Hugging Face model publishing
 
+---
 
+## Citation
 
-Cite TRL as:
-    
 ```bibtex
 @software{vonwerra2020trl,
-  title   = {{TRL: Transformers Reinforcement Learning}},
-  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
-  license = {Apache-2.0},
-  url     = {https://github.com/huggingface/trl},
-  year    = {2020}
+title={{TRL: Transformers Reinforcement Learning}},
+author={von Werra et al.},
+year={2020},
+url={https://github.com/huggingface/trl}
 }
 ```