ovinduG commited on
Commit
4b2ea75
·
verified ·
1 Parent(s): 9411875

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -87,7 +87,7 @@ near-tied questions differently.
87
 
88
  ## Training pipeline (two stages)
89
 
90
- **Stage 1 - continued pretraining.** `training/cpt_gemma4_sinhala.py` on `Minuri/diverse_sinhala_dataset` (`high_confidence_7_7M.csv`), 1M sentences, domain-balanced sampling, 16 same-domain sentences joined into each pseudo-document, packing enabled, LoRA r=128 alpha=32 with rsLoRA, 1 epoch on one A40. Result: `QBitsLK/gemma4-e4b-sinhala-cpt` (merged). Diagnostics in `training/eval_cpt_success.py` - held-out bits-per-character improved 19.6% while English perplexity regressed 21.7%.
91
 
92
  ```bash
93
  python training/cpt_gemma4_sinhala.py --max_sentences 1000000 --epochs 1
 
87
 
88
  ## Training pipeline (two stages)
89
 
90
+ **Stage 1 - continued pretraining.** `training/cpt_gemma4_sinhala.py` on `Minuri/diverse_sinhala_dataset` (`high_confidence_7_7M.csv`)[Minuri/diverse_sinhala_dataset], 1M sentences, domain-balanced sampling, 16 same-domain sentences joined into each pseudo-document, packing enabled, LoRA r=128 alpha=32 with rsLoRA, 1 epoch on one A40. Result: `QBitsLK/gemma4-e4b-sinhala-cpt` (merged). Diagnostics in `training/eval_cpt_success.py` - held-out bits-per-character improved 19.6% while English perplexity regressed 21.7%.
91
 
92
  ```bash
93
  python training/cpt_gemma4_sinhala.py --max_sentences 1000000 --epochs 1