Update to Lora Config
Browse files
README.md
CHANGED
|
@@ -145,12 +145,12 @@ African feminist scholarly activism is significant in contemporary resistance mo
|
|
| 145 |
- logging_steps=10
|
| 146 |
|
| 147 |
### Lora Configuration
|
| 148 |
-
- r
|
| 149 |
-
- lora_alpha
|
| 150 |
-
- target_modules
|
| 151 |
-
- lora_dropout
|
| 152 |
-
- bias
|
| 153 |
-
- task_type
|
| 154 |
|
| 155 |
|
| 156 |
## Evaluation
|
|
|
|
| 145 |
- logging_steps=10
|
| 146 |
|
| 147 |
### Lora Configuration
|
| 148 |
+
- r: 8
|
| 149 |
+
- lora_alpha: 16
|
| 150 |
+
- target_modules: ["q_proj", "v_proj", "k_proj", "o_proj"]
|
| 151 |
+
- lora_dropout: 0.05 # dataset is small, hence a low dropout value
|
| 152 |
+
- bias: "none"
|
| 153 |
+
- task_type: "CAUSAL_LM"
|
| 154 |
|
| 155 |
|
| 156 |
## Evaluation
|