Update README.md
Browse files
README.md
CHANGED
|
@@ -198,28 +198,28 @@ The model was trained on a comprehensive dataset of Python code review examples
|
|
| 198 |
## Training Procedure
|
| 199 |
[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/alenphilip2071-google/huggingface/runs/d27nrifd)
|
| 200 |
### Training Hyperparameters
|
| 201 |
-
- Training regime
|
| 202 |
-
- Base Model
|
| 203 |
-
- LoRA Rank
|
| 204 |
-
- LoRA Alpha
|
| 205 |
-
- LoRA Dropout
|
| 206 |
-
- Learning Rate
|
| 207 |
-
- Batch Size
|
| 208 |
-
- Epochs
|
| 209 |
-
- Max Sequence Length
|
| 210 |
-
- Optimizer
|
| 211 |
|
| 212 |
### Speeds, Sizes, Times
|
| 213 |
-
- Base Model Size
|
| 214 |
-
- Adapter Size
|
| 215 |
-
- Training Time
|
| 216 |
-
- Training Examples
|
| 217 |
|
| 218 |
## Evaluation
|
| 219 |
### Metrics
|
| 220 |
-
ROUGE-L
|
| 221 |
-
BLEU
|
| 222 |
-
Validation Loss
|
| 223 |
|
| 224 |
## Results
|
| 225 |
The model achieved strong performance on code review tasks, particularly excelling at:
|
|
|
|
| 198 |
## Training Procedure
|
| 199 |
[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/alenphilip2071-google/huggingface/runs/d27nrifd)
|
| 200 |
### Training Hyperparameters
|
| 201 |
+
- **Training regime:** bf16 mixed precision with SFT & QLoRA
|
| 202 |
+
- **Base Model:** Qwen2.5-7B-Instruct
|
| 203 |
+
- **LoRA Rank:** 32
|
| 204 |
+
- **LoRA Alpha:** 64
|
| 205 |
+
- **LoRA Dropout:** 0.1
|
| 206 |
+
- **Learning Rate:** 2e-4
|
| 207 |
+
- **Batch Size:** 16 (with gradient accumulation 4)
|
| 208 |
+
- **Epochs:** 2
|
| 209 |
+
- **Max Sequence Length:** 2048 tokens
|
| 210 |
+
- **Optimizer:** Paged AdamW 8-bit
|
| 211 |
|
| 212 |
### Speeds, Sizes, Times
|
| 213 |
+
- **Base Model Size:** 7B parameters
|
| 214 |
+
- **Adapter Size:** ~45MB
|
| 215 |
+
- **Training Time:** ~68 minutes for 400 steps
|
| 216 |
+
- **Training Examples:** 13,670 training, 1,726 evaluation
|
| 217 |
|
| 218 |
## Evaluation
|
| 219 |
### Metrics
|
| 220 |
+
- **ROUGE-L:** 0.754
|
| 221 |
+
- **BLEU:** 61.99
|
| 222 |
+
- **Validation Loss:** 0.595
|
| 223 |
|
| 224 |
## Results
|
| 225 |
The model achieved strong performance on code review tasks, particularly excelling at:
|