Commit
·
03a4aaa
1
Parent(s):
50c013a
Reformat lessons
Browse files
STORY.md
CHANGED
|
@@ -66,13 +66,17 @@ For GRPO, I switched from full fine-tuning to LoRA to reduce memory pressure dur
|
|
| 66 |
- **Max gradient norm:** 0.5
|
| 67 |
- **Loss type:** DR-GRPO
|
| 68 |
|
| 69 |
-
I made three significant mistakes along the way.
|
| 70 |
|
| 71 |
-
|
|
|
|
|
|
|
| 72 |
|
| 73 |
*Lesson: Before adopting a new algorithm, verify its inductive biases match your task's structure. Novelty isn't a reason to use something.*
|
| 74 |
|
| 75 |
-
|
|
|
|
|
|
|
| 76 |
|
| 77 |
$$\hat{W} = W + \frac{\alpha}{r} \times AB$$
|
| 78 |
|
|
@@ -92,7 +96,9 @@ This misconfiguration made LoRA updates an order of magnitude too aggressive. On
|
|
| 92 |
|
| 93 |
*Lesson: When switching algorithms, trace the math end-to-end. Defaults that work in one might not translate well to the other.*
|
| 94 |
|
| 95 |
-
|
|
|
|
|
|
|
| 96 |
|
| 97 |
*Lesson: If a reward can be gamed, it will be. Design rewards that only improve when the actual goal improves.*
|
| 98 |
|
|
|
|
| 66 |
- **Max gradient norm:** 0.5
|
| 67 |
- **Loss type:** DR-GRPO
|
| 68 |
|
| 69 |
+
I made three significant mistakes along the way that taught me some lessons. I've included them here in hopes that others can avoid making these same mistakes.
|
| 70 |
|
| 71 |
+
1. GSPO was the wrong algorithm.
|
| 72 |
+
|
| 73 |
+
I tried it because it was new and interesting without considering whether it fit my task. Cuneiform transcription quality is fundamentally character-level: did you get each glyph right? GSPO rewards sequence-level completions, which made the reward signal noisy and unhelpful. GRPO was better suited because it performs importance sampling at the token level, letting the model learn which glyph choices lead to better transcriptions.
|
| 74 |
|
| 75 |
*Lesson: Before adopting a new algorithm, verify its inductive biases match your task's structure. Novelty isn't a reason to use something.*
|
| 76 |
|
| 77 |
+
2. I misconfigured RSLoRA.
|
| 78 |
+
|
| 79 |
+
When I switched to GRPO, I uncovered a subtle but critical error in my LoRA setup. Standard LoRA scales adapter contributions by:
|
| 80 |
|
| 81 |
$$\hat{W} = W + \frac{\alpha}{r} \times AB$$
|
| 82 |
|
|
|
|
| 96 |
|
| 97 |
*Lesson: When switching algorithms, trace the math end-to-end. Defaults that work in one might not translate well to the other.*
|
| 98 |
|
| 99 |
+
3. My reward function was exploitable.
|
| 100 |
+
|
| 101 |
+
The initial design tried to encourage correct "shape" (line count, characters per line) along with TER (token error rate). The model learned to exploit this immediately. It would spam the same cuneiform character into the correct shape, ignoring accuracy entirely. Classic reward hacking.
|
| 102 |
|
| 103 |
*Lesson: If a reward can be gamed, it will be. Design rewards that only improve when the actual goal improves.*
|
| 104 |
|