Update README.md
Browse files
README.md
CHANGED
|
@@ -18,7 +18,7 @@ Fine tuned on finetome, pinkchat-sft, pinkchat-dpo, the model is able to generat
|
|
| 18 |
## Additional fine-tuning is needed.
|
| 19 |
The model does not perform well, yet it does work. It has been fine tuned on 2 billion tokens of mostly syntetic data and some human made data in the sft process.
|
| 20 |
|
| 21 |
-
Phase 0: In mergekit, we remove 16 layers (out of 28, so 12 layers left) using passthrough.
|
| 22 |
|
| 23 |
Phase 1a: Fine tuning the model on a limited amount of data, lora 16 (21% trained). This phase is to get the model started on generating some sense, mainly for healing the model and nothing else, very low quality text would be generated.
|
| 24 |
|
|
|
|
| 18 |
## Additional fine-tuning is needed.
|
| 19 |
The model does not perform well, yet it does work. It has been fine tuned on 2 billion tokens of mostly syntetic data and some human made data in the sft process.
|
| 20 |
|
| 21 |
+
Phase 0: In mergekit, we remove 16 layers (out of 28, so 12 layers left: Pinkstackorg/Qwen2.5-3Bprunebase-1M) using passthrough.
|
| 22 |
|
| 23 |
Phase 1a: Fine tuning the model on a limited amount of data, lora 16 (21% trained). This phase is to get the model started on generating some sense, mainly for healing the model and nothing else, very low quality text would be generated.
|
| 24 |
|