| --- |
| language: |
| - or |
| license: other |
| license_name: research-only |
| base_model: |
| - sarvamai/sarvam-1 |
| - google/gemma-4-E4B-it |
| library_name: transformers |
| tags: |
| - odia |
| - checkpoints |
| --- |
| |
| # Odia tutor E07: the training checkpoints |
|
|
| Every checkpoint of experiment E07's two fine-tuning runs, kept so any evaluation in the E07 report can be |
| recomputed. The final models, the comparison and the model card are in |
| [fastpixels/odia-tutor-e07](https://huggingface.co/fastpixels/odia-tutor-e07). **Private, research use only.** |
|
|
| | Folder | What | Size each | |
| |---|---|---| |
| | `sarvam1-sft/checkpoint-1500`, `-3000`, `-4272` | Sarvam-1 trainer checkpoints: fp32 weights, tokenizer, `trainer_state.json` (the full log history) | 10.1 GB | |
| | `gemma4-e4b-sft/checkpoint-1500`, `-3000`, `-4272` | Gemma-4-E4B-it trainer checkpoints: trained weights in fp32, frozen parts (per-layer embeddings, vision, audio) in bf16 | about 27 GB | |
| | `gemma4-e4b-sft/step-800-bf16`, `-1600-bf16`, `-2300-bf16` | Gemma snapshots at the steps matched to Sarvam-1's checkpoints (same tokens seen, about the same hours): bf16 weights and tokenizer | 17.3 GB | |
|
|
| Only the newest checkpoint of each run keeps its resume state (`optimizer.pt` with AdamW's moments, `scheduler.pt`, |
| `rng_state.pth`): it is uploaded with every checkpoint while training runs, as a failsafe, and removed from the |
| older one once a newer checkpoint has its own. At the end, `checkpoint-4272` keeps it (Sarvam-1 20.2 GB, Gemma |
| about 37 GB), so either run can be continued exactly. Each run is 4,272 steps: two passes over 136,664 rows in |
| batches of 64. |
|
|
| ## Use |
|
|
| Download one folder, never the whole repo (a few hundred GB): |
|
|
| ```python |
| from huggingface_hub import snapshot_download |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| repo, sub = "fastpixels/odia-tutor-e07-checkpoints", "sarvam1-sft/checkpoint-3000" |
| path = snapshot_download(repo, allow_patterns=f"{sub}/*", ignore_patterns=["*optimizer.pt"]) |
| model = AutoModelForCausalLM.from_pretrained(f"{path}/{sub}", dtype="bfloat16") |
| tok = AutoTokenizer.from_pretrained(f"{path}/{sub}") |
| ``` |
|
|
| To continue training from the last checkpoint, download `checkpoint-4272` with its `optimizer.pt` into the run's |
| output folder and resume (`odia-train ... --resume`, or `Trainer.train(resume_from_checkpoint=...)`). The cosine |
| schedule ended at step 4,272, so going further needs a new schedule. |
|
|
| ## Licence |
|
|
| `sarvam1-sft/`: the Sarvam AI Research License (non-commercial), inherited from Sarvam-1. `gemma4-e4b-sft/`: base |
| model Apache 2.0; research use only because of the fine-tuning data (SciQ is CC BY-NC 3.0). |
|
|
