Update README.md
Browse files
README.md
CHANGED
|
@@ -87,7 +87,7 @@ near-tied questions differently.
|
|
| 87 |
|
| 88 |
## Training pipeline (two stages)
|
| 89 |
|
| 90 |
-
**Stage 1 - continued pretraining.** `training/cpt_gemma4_sinhala.py` on `Minuri/diverse_sinhala_dataset` (`high_confidence_7_7M.csv`), 1M sentences, domain-balanced sampling, 16 same-domain sentences joined into each pseudo-document, packing enabled, LoRA r=128 alpha=32 with rsLoRA, 1 epoch on one A40. Result: `QBitsLK/gemma4-e4b-sinhala-cpt` (merged). Diagnostics in `training/eval_cpt_success.py` - held-out bits-per-character improved 19.6% while English perplexity regressed 21.7%.
|
| 91 |
|
| 92 |
```bash
|
| 93 |
python training/cpt_gemma4_sinhala.py --max_sentences 1000000 --epochs 1
|
|
|
|
| 87 |
|
| 88 |
## Training pipeline (two stages)
|
| 89 |
|
| 90 |
+
**Stage 1 - continued pretraining.** `training/cpt_gemma4_sinhala.py` on `Minuri/diverse_sinhala_dataset` (`high_confidence_7_7M.csv`)[Minuri/diverse_sinhala_dataset], 1M sentences, domain-balanced sampling, 16 same-domain sentences joined into each pseudo-document, packing enabled, LoRA r=128 alpha=32 with rsLoRA, 1 epoch on one A40. Result: `QBitsLK/gemma4-e4b-sinhala-cpt` (merged). Diagnostics in `training/eval_cpt_success.py` - held-out bits-per-character improved 19.6% while English perplexity regressed 21.7%.
|
| 91 |
|
| 92 |
```bash
|
| 93 |
python training/cpt_gemma4_sinhala.py --max_sentences 1000000 --epochs 1
|