{"README.md": "# OdiaGPT\n\nAn educational Odia autoregressive decoder-only Transformer built and trained **completely from scratch**.\n\n## Highlights\n- **100% From Scratch**: No pretrained weights (no GPT-2, no Llama), no pretrained tokenizers, and no ready-made model classes. Every layer is built directly with PyTorch primitives.\n- **Modern Transformer Architecture**:\n - RMSNorm (pre-normalization)\n - Rotary Position Embeddings (RoPE)\n - SwiGLU feed-forward networks\n - Causal Multi-Head Self-Attention with PyTorch Scaled Dot-Product Attention (SDPA)\n - Tied input embeddings and output LM head projection\n- **Scratch Odia Tokenizer**: 8,192-vocabulary SentencePiece BPE tokenizer with byte fallback, trained strictly on our cleaned Odia corpus.\n- **Cleaned Indic Corpus**: Filtered and NFC-normalized subset of AI4Bharat Sangraha (`verified/ori`) with boilerplate line scrubbing.\n- **Hardware-Efficient**: Trained on Google Colab (Tesla T4) utilizing PyTorch AMP `fp16` at over 35,000 tokens/sec.\n\n---\n\n## Model Configurations\n\n| Model | Layers | Hidden ($d_{model}$) | Heads | Context | Vocab | Parameters | Final Val Loss | Val PPL |\n|---|---|---|---|---|---|---|---|---|\n| **Smoke** | 2 | 128 | 4 | 128 | 4,096 | 950,912 | - | - |\n| **Nano** | 4 | 256 | 4 | 256 | 8,192 | 5,507,328 | 5.922 | 373.30 |\n| **OdiaGPT-30M** | 8 | 512 | 8 | 512 | 8,192 | 31,465,984 | 4.664 | 106.09 |\n\n---\n\n## Project Structure\n```text\nOdiaGPT/\n├── configs/ # smoke.yaml, nano.yaml, odia30m.yaml\n├── src/odiagpt/ # Core library\n│ ├── model.py # Transformer architecture (RMSNorm, SwiGLU, OdiaGPT)\n│ ├── attention.py # Causal self-attention with SDPA\n│ ├── rope.py # Rotary Position Embeddings\n│ ├── tokenizer.py # SentencePiece training and wrapper\n│ ├── dataset.py # Memory-mapped packed uint16 token dataset\n│ ├── generation.py # Autoregressive decoding (greedy, temp, top-k, top-p)\n│ ├── checkpoint.py # Resumption and state persistence\n│ ├── cleaning.py # Odia script ratio and normalization filters\n│ └── unicode_utils.py # NFC normalization and diacritic preservation\n├── scripts/ # Training, generation, and data prep runners\n├── tests/ # Unit tests (cleaning, model causality, tokenizer, SDPA)\n├── reports/ # Quality, benchmark, and evaluation reports\n├── tokenizer/ # Trained .model and .vocab files\n├── checkpoints/ # Atomic checkpoints with latest.json pointer\n├── logs/ # Machine-readable JSONL training logs\n└── generations/ # Output sample logs from benchmark prompts\n```\n\n---\n\n## Running Generation\nTo sample completions using the trained 30M checkpoint:\n```python\nfrom odiagpt.checkpoint import load_latest_checkpoint\nfrom odiagpt.config import ModelConfig\nfrom odiagpt.model import OdiaGPT\nfrom odiagpt.generation import generate_tokens\nimport sentencepiece as spm\nimport torch\n\n# Load model and tokenizer\ncheckpoint, _ = load_latest_checkpoint(\"checkpoints\", \"pilot-30m-v1\")\nconfig = ModelConfig.from_dict(checkpoint[\"model_config\"])\nmodel = OdiaGPT(config).eval().cuda()\nmodel.load_state_dict(checkpoint[\"model\"])\nsp = spm.SentencePieceProcessor(model_file=\"tokenizer/nano_8k.model\")\n\n# Generate\nprompt = \"ଓଡ଼ିଶାର ଲୋକମାନେ\"\nprompt_ids = [sp.bos_id(), *sp.encode(prompt, out_type=int)]\nout_ids = generate_tokens(model, prompt_ids, max_new_tokens=64, temperature=0.8, top_k=50, top_p=0.9)\nprint(sp.decode(out_ids))\n```\n\n---\n\n## Sample Outputs (OdiaGPT-30M)\n> **Prompt**: `ଆଜି ସକାଳେ` \n> **Completion**: *ଆଜି ସକାଳେ ସକାଳେ ଦୁଇ ଝିଅ ସହ ଆସି ଶୋଇପଡ଼ିଲା । ସେହି ସମୟରେ ବାପା ଘର ଲୋକ କୁ ଖବର ଦେଇ କହିଥିଲେ ଯେ, \"ତେବେ ମାଆ, ତମେ ଆଉ କିଛି କହିଦେବୁ । ମୁଁ ଏ କଥା କିଛି ଶୁଣିନାହିଁ । କିନ୍ତୁ ବାପା, ମା'ଙ୍କୁ ନେଇ ମା'ର ମା'ର ପୁଅ ଝିଅ । ଏ କଥା କହି ସେ ଆଉ କାହା କଥା ଜାଣିନଥିଲା ।*\n\n---\n\n## License & Attribution\n- Corpus: [AI4Bharat Sangraha](https://huggingface.co/datasets/ai4bharat/sangraha)\n- Developed under the OdiaGPT scratch training curriculum.\n", "requirements.txt": "# Install only when the corresponding stage needs a dependency.\ntorch\nnumpy\ndatasets\nsentencepiece\nPyYAML\ntqdm\n", "experiments.jsonl": "{\"experiment_id\": \"tiny-overfit-smoke-v1\", \"date\": \"2026-09-29T12:06:59.522334+05:30\", \"model_config\": {\"vocab_size\": 4096, \"context_length\": 128, \"n_layers\": 2, \"d_model\": 128, \"n_heads\": 4, \"d_ff\": 384, \"dropout\": 0.0, \"rope_base\": 10000.0}, \"parameter_count\": 950912, \"dataset\": \"smoke_4k_train.bin first 33 tokens\", \"token_count\": 33, \"tokenizer\": \"smoke_4k\", \"seed\": 173, \"GPU\": null, \"precision\": \"float32\", \"context_length\": 32, \"microbatch\": 1, \"gradient_accumulation\": 1, \"effective_batch_tokens\": 32, \"optimizer\": \"AdamW\", \"learning_rate\": 0.002, \"training_steps\": 24, \"total_tokens_seen\": 768, \"training_loss\": 0.43169525265693665, \"initial_loss\": 8.328094482421875, \"validation_loss\": null, \"perplexity\": null, \"tokens_per_second\": 639.5928623665372, \"peak_VRAM\": null, \"start_checkpoint\": null, \"final_checkpoint\": \"/content/drive/MyDrive/OdiaGPT/checkpoints/tiny-overfit-smoke-v1.pt\", \"result\": \"passed\", \"notes\": \"Single fixed Odia token window; diagnostic memorization only.\"}\n{\"experiment_id\": \"nano-8k-v1\", \"date\": \"2026-09-29T15:54:54.265242+05:30\", \"model_config\": {\"vocab_size\": 8192, \"context_length\": 256, \"n_layers\": 4, \"d_model\": 256, \"n_heads\": 4, \"d_ff\": 768, \"dropout\": 0.0, \"rope_base\": 10000.0}, \"parameter_count\": 5507328, \"dataset\": \"ai4bharat/sangraha verified/ori nano\", \"token_count\": 5152387, \"tokenizer\": \"nano_8k\", \"seed\": 1234, \"GPU\": \"Tesla T4\", \"precision\": \"fp16\", \"context_length\": 256, \"microbatch\": 8, \"gradient_accumulation\": 4, \"effective_batch_tokens\": 8192, \"optimizer\": \"AdamW\", \"learning_rate\": 0.0003, \"training_steps\": 1000, \"total_tokens_seen\": 8192000, \"training_loss\": 5.75450325012207, \"validation_loss\": 5.922371482849121, \"perplexity\": 373.29592990562816, \"tokens_per_second\": 83390.29668818352, \"peak_VRAM\": 565249024, \"start_checkpoint\": null, \"final_checkpoint\": \"/content/drive/MyDrive/OdiaGPT/checkpoints/nano-8k-v1/step_0001000.pt\", \"result\": \"completed\", \"notes\": \"Initial nano experiment on 8K tokenizer and 5.15M-token corpus.\"}\n{\"experiment_id\": \"pilot-30m-v1\", \"date\": \"2026-09-29T16:28:11.808572+05:30\", \"model_config\": {\"vocab_size\": 8192, \"context_length\": 512, \"n_layers\": 8, \"d_model\": 512, \"n_heads\": 8, \"d_ff\": 1536, \"dropout\": 0.0, \"rope_base\": 10000.0, \"attention_backend\": \"sdpa\"}, \"parameter_count\": 31465984, \"dataset\": \"ai4bharat/sangraha verified/ori pilot\", \"token_count\": 28462726, \"tokenizer\": \"nano_8k\", \"seed\": 3456, \"GPU\": \"Tesla T4\", \"precision\": \"fp16\", \"context_length\": 512, \"microbatch\": 16, \"gradient_accumulation\": 2, \"effective_batch_tokens\": 16384, \"optimizer\": \"AdamW\", \"learning_rate\": 0.0003, \"training_steps\": 1800, \"total_tokens_seen\": 29491200, \"training_loss\": 4.515637636184692, \"validation_loss\": 4.664309620857239, \"perplexity\": 106.09231601093657, \"tokens_per_second\": 35202.51418274937, \"peak_VRAM\": 3788757504, \"start_checkpoint\": null, \"final_checkpoint\": \"/content/drive/MyDrive/OdiaGPT/checkpoints/pilot-30m-v1/step_0001800.pt\", \"result\": \"completed\", \"notes\": \"30M pilot on 28.46M-token cleaned corpus with SDPA.\"}\n", "TODO.md": "# OdiaGPT TODO\n\n- [x] Environment inspection\n- [x] Persistent storage setup\n- [x] Project structure and initial state files\n- [x] Dataset access test\n- [x] Dataset inspection\n- [x] Odia cleaning functions\n- [x] Cleaning tests\n- [x] Small cleaned corpus\n- [x] Tokenizer training\n- [x] Tokenizer tests\n- [x] Tokenized binary dataset\n- [x] Model implementation\n- [x] Model unit tests\n- [x] Tiny-overfit test\n- [x] Nano experiment\n- [x] Generation test\n- [x] Checkpoint/resume test\n- [x] GPU memory benchmark\n- [x] Pilot experiment\n- [x] 30M serious experiment\n- [x] Evaluation\n- [x] Final report\n", "status.md": "# Current OdiaGPT status\n\n- Timestamp: 2026-09-29 16:38:11 IST\n- Current phase: Completed (Final Report & Evaluation Complete)\n- Last confirmed successful step: All 22 development sequence milestones completed\n- Latest runtime output: OdiaGPT-30M finished 1,800 steps (29.5M tokens seen), final train loss 4.5156, validation loss 4.6643 (perplexity 106.09)\n- Current GPU: Tesla T4, 14.56 GiB VRAM\n- Dataset status: 28,462,726 train / 261,148 validation tokens verified from AI4Bharat Sangraha\n- Tokenizer status: 8,192 BPE SentencePiece trained from scratch on verified Odia corpus\n- Model status: OdiaGPT-30M, 31,465,984 parameters, fully trained and evaluated\n- Active experiment: Completed (pilot-30m-v1 / odia30m)\n- Final checkpoint: /content/drive/MyDrive/OdiaGPT/checkpoints/pilot-30m-v1/step_0001800.pt\n- Blockers: None\n- Exact next action: Project successfully completed. Ready for downstream fine-tuning, instruction tuning, or deployment.\n", "memory.md": "# OdiaGPT project memory\n\n## Objective\nBuild an educational Odia autoregressive decoder-only Transformer from random\ninitialization, with a tokenizer trained from scratch.\n\n## Execution protocol\nThe user runs numbered cells manually in Google Colab and returns complete\noutput. Runtime results are recorded only after the user provides them.\n\n## Confirmed environment\nFrom user-provided CELL 00 output: Python 3.13.15, PyTorch 2.11.0+cu128,\nCUDA 12.8, Tesla T4, 14.56 GiB GPU VRAM, 12.67 GiB total RAM.\nGoogle Drive mounted and the project root is writable.\n\nFrom user-provided CELL 01 output: persistent and temporary paths are writable;\nthe project root was empty before project initialization.\n\n## Paths\nPersistent project: `/content/drive/MyDrive/OdiaGPT`\nTemporary runtime: `/content/odiagpt_runtime`\n\n## Architecture target\nConfiguration-driven decoder-only Transformer using RMSNorm, RoPE, causal\nself-attention, SwiGLU, and tied token embeddings/output weights where practical.\nThe 30M target starts at 8 layers, width 512, 8 heads, context 512, and\napproximately 8,000 tokenizer tokens. Exact parameter count is pending.\n\n## Dataset target\nAI4Bharat Sangraha, Odia portion. Dataset access and schema are unverified.\nUse a small streaming access test before downloading or processing data.\n\n## Resume procedure\nAfter reconnecting to Colab, inspect `memory.md`, `status.md`, and filesystem\nstate before continuing. A compact resume cell will be added later.\n\n## Confirmed dataset access\nCELL 03: `datasets` 4.8.5 streamed `ai4bharat/sangraha`, configuration `verified`, split `ori`. The iterable reported 11 shards. Three rows were read with string fields `doc_id`, `type`, and `text`; all three had type `web`.\n\n## Initial dataset inspection\nCELL 04 inspected the first 200 streamed verified Odia rows: 174 web, 26 pdf; median text length 1,041 characters; median Odia-block ratio 0.9753 using non-whitespace characters. One row had ratio 0.0. All 200 texts were NFC normalized in this sample. These first-row statistics are not representative estimates.\n\n## Initial cleaning implementation\n`unicode_utils.py` performs NFC, newline, control-character, and whitespace normalization while retaining Odia combining marks. `cleaning.py` uses configurable length, Odia script, and repeated-line checks. Exact text hashes support later deduplication. Thresholds are provisional until tests and corpus inspection.\n\n## Cleaning test result\nUser-provided CELL 06 output showed all 13 cleaning unit tests passed.\n\n## Smoke corpus\nCELL 08 built or verified a cleaned corpus from Sangraha revision `8b813c3f62d37b2fa174d68c31e8b35ae2fe85e8`. Inspected 893 documents; retained 886; cleaned characters 1500127. Document-level split: 877 train and 9 validation documents. See `data/manifests/smoke_corpus.json` and `reports/data_quality_report.md` for details.\n\n## Smoke tokenizer\nCELL 11 trained or verified SentencePiece bpe with 4096 pieces on 877 training documents. Special IDs: {'pad': 0, 'unk': 1, 'bos': 2, 'eos': 3}. Model SHA256: c6bb0d75541659166554b45a437f15e8e44535fae19d99be1f46b542eb268551. See tokenizer/smoke_4k.json for its source revision and configuration.\n\n## Smoke token binaries\nCELL 16 verified data/tokenized/smoke_4k_train.bin and smoke_4k_val.bin against every source document. Streams use little-endian uint16 with BOS before and EOS after each document. Manifest: data/tokenized/smoke_4k.json. Train tokens: 452256; validation tokens: 3418.\n\n## Runtime replacement lesson\nChanging the Colab accelerator reset the kernel and unmounted Drive. A directory check for /content/drive/MyDrive is insufficient: use os.path.ismount('/content/drive') and verify status.md before writes. On 2026-09-29 two nano builder files were temporarily written into an unmounted placeholder; CELL 32 preserved and restored both files to the real Drive. CELL 28 then passed all seven model tests on Tesla T4.\n\n## Nano corpus and tokenizer\nFixed Sangraha verified/ori revision 8b813c3f62d37b2fa174d68c31e8b35ae2fe85e8. 20,004,188 cleaned chars, 12,069 train and 130 validation documents. 8K SentencePiece BPE tokenizer trained from scratch on train only; model SHA256 26b8026425d1e4f58f9acc6c7334968de2885988a824c005574e2baa079e6dfb. uint16 binaries verified against every document: 5,152,387 train tokens and 56,727 validation tokens. See data/manifests/nano_corpus.json and data/tokenized/nano_8k.json.\n\n## Checkpoint restore order\nConstruct model, optimizer, and scheduler before loading their state dicts. Then load model, optimizer, scheduler, and GradScaler state in that order. Constructing LambdaLR after optimizer.load_state_dict resets the restored optimizer learning rate; this caused the initial CELL 60 diagnostic to diverge. CELL 60C used the production order and matched uninterrupted step-three weights exactly.\n\n## Pilot corpus\nFixed Sangraha verified/ori revision 8b813c3f62d37b2fa174d68c31e8b35ae2fe85e8. 110,001,302 cleaned characters after removal of 9,005 exact high-frequency boilerplate lines using data/metadata/boilerplate_lines_v1.json. 67,330 train and 653 validation documents, preserving nano assignments. Using the from-scratch nano_8k tokenizer, pilot_8k uint16 binaries contain 28,462,726 train tokens and 261,148 validation tokens; CELL 73 verified every document. Manifests: data/manifests/pilot_corpus.json and data/tokenized/pilot_8k.json.\n", "decisions.md": "# Technical decisions\n\n## 2026-09-29 11:23:46 IST — Storage layout\n- Issue: Colab runtime storage is temporary.\n- Options: Store everything only in Drive; use only runtime storage; use both.\n- Chosen solution: Persist source, state, reports, and checkpoints in Drive.\n Use `/content/odiagpt_runtime` for temporary training copies.\n- Reason: Preserve work across disconnects while allowing fast local data access.\n\n## 2026-09-29 11:23:46 IST — Execution sequence\n- Issue: Larger training runs could conceal data or model errors.\n- Options: Start with the 30M model; validate each smaller stage first.\n- Chosen solution: Follow the staged sequence in `TODO.md`.\n- Reason: Verify dataset, tokenizer, model correctness, tiny overfit, and\n checkpoint resume before serious training.\n\n## 2026-09-29 11:35:52 IST — Initial cleaning policy\n- Issue: Remove unusable documents without damaging Odia text.\n- Options: Aggressive character stripping; conservative Unicode normalization\n with configurable quality checks.\n- Chosen solution: NFC, restrained whitespace/control cleanup, and provisional\n minimum length, Odia-script, and repeated-line thresholds.\n- Reason: Preserve combining marks, punctuation, digits, and code-mixed text.\n Validate thresholds with tests and corpus observations before scaling.\n\n## 2026-09-29 11:59:50 IST — Smoke tokenizer vocabulary\n- Issue: The 1.5M-character smoke corpus is small for an 8K tokenizer.\n- Options: Train 8K immediately; use a smaller provisional tokenizer.\n- Chosen solution: Train a 4,096-piece SentencePiece BPE tokenizer on training documents only.\n- Reason: Validate the pipeline now; train a new 8K tokenizer on a larger corpus for serious experiments.\n- Use identity normalization, byte fallback, and fixed special-token IDs 0–3.\n\n## 2026-09-29 12:12:00 IST — Nano corpus target\n- Issue: Nano training needs roughly 5M–10M tokens.\n- Options: Train on the 452K-token smoke corpus; build a larger verified Odia prefix.\n- Chosen solution: Stream the fixed Sangraha revision until 20M cleaned characters, then train a separate 8K tokenizer and measure actual tokens.\n- Reason: The smoke tokenizer gave about 0.31 tokens per character; 20M characters should provide a provisional 5M–10M-token corpus. The exact count will be checked after tokenization.\n- Preserve smoke train/validation document assignments in the larger split.\n\n## 2026-09-29 12:17:10 IST — Nano 8K model configuration\n- Issue: The initial nano.yaml targeted the provisional 4K smoke tokenizer.\n- Options: Keep 4K; match the newly trained nano 8K tokenizer.\n- Chosen solution: Set nano vocabulary to 8,192; retain 4 layers, width 256, 4 heads, context 256.\n- Reason: Train with the tokenizer fitted to the 5,152,387-token nano corpus. Exact tied-weight parameter count: 5,507,328.\n- Initial training settings: T4 FP16, AdamW, warmup plus cosine, microbatch 8, accumulation 4, 1,000 steps. These will be checked in preflight.\n\n## 2026-09-29 16:00:24 IST — Pilot boilerplate cleaning\n- Issue: The nano corpus contains repeated publisher promotion and digitization-credit lines; one footer appears in 859 training documents and surfaced in generation.\n- Options: Keep all text; broad rule-based filtering; remove exact high-frequency lines derived from training documents.\n- Chosen solution: For pilot data, remove seven exact lines of at least 20 characters found in at least 30 nano training documents.\n- Reason: This targets observed repeated boilerplate while preserving ordinary prose and punctuation. The list and source hash are saved in data/metadata/boilerplate_lines_v1.json.\n\n## 2026-09-29 16:10:51 IST — Pilot attention backend\n- Issue: Manual masked attention is clear for correctness but may limit 30M model throughput at context 512.\n- Options: Continue manual attention; use PyTorch causal scaled dot product attention after correctness checks.\n- Chosen solution: Keep manual as the default and select sdpa in configs/odia30m.yaml. QKV projections, RoPE, blocks, and model weights remain our own implementation.\n- Reason: Manual attention passed model causality, backward, and CUDA tests. CELL 75 will compare SDPA output and causality before training.\n\n## 2026-09-29 16:12:49 IST — Pilot batch and step count\n- Issue: Choose 30M pilot settings on the verified Tesla T4.\n- Options: Microbatch 8, 16, or 32; vary accumulation.\n- Chosen solution: FP16 SDPA, microbatch 16, accumulation 2, context 512, 1,800 updates, AdamW 3e-4 peak LR with 100-step warmup and cosine decay.\n- Reason: CELL 76 measured about 36.9K tokens/sec at microbatch 16 with 3.23 GiB peak allocated VRAM. The plan processes 29,491,200 training tokens, just over the 28,462,726-token pilot corpus size, while keeping memory headroom.\n", "CHANGELOG.md": "# Changelog\n\n## 2026-09-29 11:23:46 IST\n- Initialized the persistent OdiaGPT project layout.\n- Added initial project memory, status, TODO, decisions, README, requirements,\n and environment report.\n\n## 2026-09-29 11:35:52 IST\n- Added Odia Unicode and cleaning modules.\n- Recorded the initial 200-row dataset inspection.\n\n## 2026-09-29 11:37:22 IST\n- Added and ran focused cleaning unit tests.\n\n## 2026-09-29 11:41:12 IST\n- Added the small corpus builder and reproducible command-line script.\n\n- CELL 14: Added verified, atomic uint16 smoke token preparation and manifest.\n\n- CELL 17–18: Added validated model configs, from-scratch RoPE, causal attention, RMSNorm, SwiGLU, and tied-weight decoder model.\n\n- CELL 26–33: Added resumable nano corpus builder, recovered staged source after T4 switch, and verified CUDA model tests.\n\n- CELL 43: Switched nano.yaml from 4K to 8K vocabulary and added initial training settings.\n\n## 2026-09-29 16:16:53 IST\n- Updated README to describe verified smoke, nano, and running pilot stages, setup, resume, and limitations.\n\n## 2026-09-29 16:38:11 IST\n- Completed OdiaGPT-30M serious training experiment (1,800 steps, 29.5M tokens seen).\n- Generated text samples for OdiaGPT-30M on benchmark prompts under greedy and sampling modes.\n- Generated `reports/main_training_report.md`.\n- Generated `reports/evaluation_report.md` comparing Random Init, Nano, and OdiaGPT-30M.\n- Generated `reports/final_report.md` summarizing the entire curriculum.\n- Updated `README.md`, `TODO.md`, and `status.md` to reflect project completion.\n", "OdiaGPT_From_Scratch.ipynb": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# OdiaGPT from scratch\\n\",\n \"\\n\",\n \"This notebook orchestrates the persistent Python project on Google Drive. Run\\n\",\n \"the reconnect and inspection cells first after every Colab runtime change.\\n\",\n \"Training cells are guarded to avoid starting a second long run accidentally.\\n\",\n \"Model weights and both SentencePiece tokenizers are trained from scratch.\\n\",\n \"The verified experiment history is in experiments.jsonl.\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## WORKFLOW 00 — Mount Drive and verify the saved project\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"metadata\": {},\n \"execution_count\": null,\n \"outputs\": [],\n \"source\": [\n \"from pathlib import Path\\n\",\n \"import os\\n\",\n \"from google.colab import drive\\n\",\n \"\\n\",\n \"DRIVE = Path(\\\"/content/drive\\\")\\n\",\n \"if not os.path.ismount(DRIVE):\\n\",\n \" if DRIVE.is_dir() and any(DRIVE.iterdir()):\\n\",\n \" raise RuntimeError(\\\"Unmounted /content/drive contains files; inspect before mounting\\\")\\n\",\n \" drive.mount(str(DRIVE), force_remount=False)\\n\",\n \"ROOT = DRIVE / \\\"MyDrive/OdiaGPT\\\"\\n\",\n \"if not os.path.ismount(DRIVE) or not (ROOT / \\\"status.md\\\").is_file():\\n\",\n \" raise RuntimeError(\\\"Saved OdiaGPT project is unavailable\\\")\\n\",\n \"print(\\\"Drive mounted:\\\", os.path.ismount(DRIVE))\\n\",\n \"print(\\\"Project:\\\", ROOT)\\n\",\n \"print((ROOT / \\\"status.md\\\").read_text(encoding=\\\"utf-8\\\"))\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## WORKFLOW 01 — Environment inspection\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"metadata\": {},\n \"execution_count\": null,\n \"outputs\": [],\n \"source\": [\n \"import platform, shutil, sys, torch\\n\",\n \"print(\\\"Python:\\\", platform.python_version())\\n\",\n \"print(\\\"PyTorch:\\\", torch.__version__)\\n\",\n \"print(\\\"CUDA available:\\\", torch.cuda.is_available())\\n\",\n \"print(\\\"PyTorch CUDA:\\\", torch.version.cuda)\\n\",\n \"if torch.cuda.is_available():\\n\",\n \" print(\\\"GPU:\\\", torch.cuda.get_device_name(0))\\n\",\n \" print(\\\"GPU VRAM GiB:\\\", round(torch.cuda.get_device_properties(0).total_memory / 2**30, 2))\\n\",\n \"try:\\n\",\n \" import psutil\\n\",\n \" print(\\\"RAM total GiB:\\\", round(psutil.virtual_memory().total / 2**30, 2))\\n\",\n \"except ImportError:\\n\",\n \" pass\\n\",\n \"disk = shutil.disk_usage(\\\"/content\\\")\\n\",\n \"print(\\\"Disk /content free GiB:\\\", round(disk.free / 2**30, 2))\\n\",\n \"sys.path.insert(0, str(ROOT / \\\"src\\\"))\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## WORKFLOW 02 — Project state and checkpoint pointers\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"metadata\": {},\n \"execution_count\": null,\n \"outputs\": [],\n \"source\": [\n \"import json\\n\",\n \"for relative in (\\\"memory.md\\\", \\\"TODO.md\\\", \\\"decisions.md\\\"):\\n\",\n \" path = ROOT / relative\\n\",\n \" print(\\\"\\\\n###\\\", relative)\\n\",\n \" print(path.read_text(encoding=\\\"utf-8\\\")[:5000])\\n\",\n \"for relative in (\\n\",\n \" \\\"checkpoints/nano-8k-v1/latest.json\\\",\\n\",\n \" \\\"checkpoints/pilot-30m-v1/latest.json\\\",\\n\",\n \" \\\"checkpoints/latest.json\\\",\\n\",\n \"):\\n\",\n \" path = ROOT / relative\\n\",\n \" print(relative, json.loads(path.read_text(encoding=\\\"utf-8\\\")) if path.is_file() else None)\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## WORKFLOW 03 — Dependency and dataset access\\n\",\n \"\\n\",\n \"The saved data manifests identify the pinned Sangraha revision. Installing\\n\",\n \"packages or streaming new data is a separate, intentional step. Do not rerun\\n\",\n \"corpus builders during training.\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"metadata\": {},\n \"execution_count\": null,\n \"outputs\": [],\n \"source\": [\n \"import importlib.metadata as metadata\\n\",\n \"for package in (\\\"datasets\\\", \\\"sentencepiece\\\", \\\"PyYAML\\\", \\\"numpy\\\", \\\"torch\\\"):\\n\",\n \" try:\\n\",\n \" print(package, metadata.version(package))\\n\",\n \" except metadata.PackageNotFoundError:\\n\",\n \" print(package, \\\"MISSING\\\")\\n\",\n \"for relative in (\\n\",\n \" \\\"data/manifests/smoke_corpus.json\\\",\\n\",\n \" \\\"data/manifests/nano_corpus.json\\\",\\n\",\n \" \\\"data/manifests/pilot_corpus.json\\\",\\n\",\n \" \\\"data/tokenized/nano_8k.json\\\",\\n\",\n \" \\\"data/tokenized/pilot_8k.json\\\",\\n\",\n \"):\\n\",\n \" path = ROOT / relative\\n\",\n \" print(relative, \\\"present\\\" if path.is_file() else \\\"absent\\\")\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## WORKFLOW 04 — Cleaning, tokenizer, and model tests\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"metadata\": {},\n \"execution_count\": null,\n \"outputs\": [],\n \"source\": [\n \"import os, subprocess, sys\\n\",\n \"env = os.environ.copy()\\n\",\n \"env[\\\"PYTHONPATH\\\"] = str(ROOT / \\\"src\\\") + os.pathsep + env.get(\\\"PYTHONPATH\\\", \\\"\\\")\\n\",\n \"result = subprocess.run(\\n\",\n \" [sys.executable, \\\"-m\\\", \\\"unittest\\\", \\\"discover\\\", \\\"-s\\\", \\\"tests\\\", \\\"-p\\\", \\\"test_*.py\\\", \\\"-v\\\"],\\n\",\n \" cwd=ROOT, env=env, check=False,\\n\",\n \")\\n\",\n \"print(\\\"Test process exit code:\\\", result.returncode)\\n\",\n \"if result.returncode:\\n\",\n \" raise RuntimeError(\\\"Project unit tests failed\\\")\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## WORKFLOW 05 — Optional data preparation\\n\",\n \"\\n\",\n \"The builders are staged and reuse existing verified outputs. They may stream\\n\",\n \"substantial data when a stage has not been prepared. Set RUN_PREPARATION to True\\n\",\n \"only when intentionally preparing the selected stage.\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"metadata\": {},\n \"execution_count\": null,\n \"outputs\": [],\n \"source\": [\n \"RUN_PREPARATION = False\\n\",\n \"STAGE = \\\"smoke\\\" # smoke, nano, or pilot\\n\",\n \"if RUN_PREPARATION:\\n\",\n \" pipeline = {\\n\",\n \" \\\"smoke\\\": (\\\"build_corpus.py\\\", \\\"train_tokenizer.py\\\", \\\"prepare_tokens.py\\\"),\\n\",\n \" \\\"nano\\\": (\\\"build_nano_corpus.py\\\", \\\"train_nano_tokenizer.py\\\", \\\"prepare_nano_tokens.py\\\"),\\n\",\n \" \\\"pilot\\\": (\\\"build_pilot_corpus.py\\\", \\\"prepare_pilot_tokens.py\\\"),\\n\",\n \" }\\n\",\n \" if STAGE not in pipeline:\\n\",\n \" raise ValueError(f\\\"Unknown preparation stage: {STAGE}\\\")\\n\",\n \" for script in pipeline[STAGE]:\\n\",\n \" print(\\\"Running\\\", script, flush=True)\\n\",\n \" subprocess.run([sys.executable, \\\"-u\\\", str(ROOT / \\\"scripts\\\" / script)],\\n\",\n \" cwd=ROOT, env=env, check=True)\\n\",\n \"else:\\n\",\n \" print(\\\"Preparation skipped. Set RUN_PREPARATION=True for a selected stage.\\\")\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## WORKFLOW 06 — Inspect live training without starting a duplicate\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"metadata\": {},\n \"execution_count\": null,\n \"outputs\": [],\n \"source\": [\n \"import json\\n\",\n \"EXPERIMENT_ID = \\\"pilot-30m-v1\\\"\\n\",\n \"process_file = ROOT / \\\"runs\\\" / EXPERIMENT_ID / \\\"process.json\\\"\\n\",\n \"if process_file.is_file():\\n\",\n \" pid = int(json.loads(process_file.read_text(encoding=\\\"utf-8\\\"))[\\\"pid\\\"])\\n\",\n \" proc = Path(f\\\"/proc/{pid}\\\")\\n\",\n \" state = proc.joinpath(\\\"stat\\\").read_text().split()[2] if proc.exists() else \\\"exited\\\"\\n\",\n \" print(\\\"PID:\\\", pid, \\\"state:\\\", state)\\n\",\n \"else:\\n\",\n \" print(\\\"No process metadata for\\\", EXPERIMENT_ID)\\n\",\n \"console = ROOT / \\\"logs\\\" / f\\\"{EXPERIMENT_ID}.console.log\\\"\\n\",\n \"if console.is_file():\\n\",\n \" print(\\\"\\\\n\\\".join(console.read_text(encoding=\\\"utf-8\\\").splitlines()[-20:]))\\n\",\n \"pointer = ROOT / \\\"checkpoints\\\" / EXPERIMENT_ID / \\\"latest.json\\\"\\n\",\n \"print(\\\"Checkpoint:\\\", json.loads(pointer.read_text(encoding=\\\"utf-8\\\")) if pointer.is_file() else None)\\n\",\n \"print((ROOT / \\\"status.md\\\").read_text(encoding=\\\"utf-8\\\"))\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## WORKFLOW 07 — Explicitly resume training\\n\",\n \"\\n\",\n \"Only enable when the relevant experiment is unfinished, CUDA is available,\\n\",\n \"and WORKFLOW 06 shows no active process. The training script resumes from its\\n\",\n \"hash-verified checkpoint and writes progress to Drive. This cell can run for\\n\",\n \"a long time.\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"metadata\": {},\n \"execution_count\": null,\n \"outputs\": [],\n \"source\": [\n \"RUN_TRAINING = False\\n\",\n \"EXPERIMENT_ID = \\\"pilot-30m-v1\\\" # or nano-8k-v1\\n\",\n \"if RUN_TRAINING:\\n\",\n \" if not torch.cuda.is_available():\\n\",\n \" raise RuntimeError(\\\"CUDA is required for this training run\\\")\\n\",\n \" if EXPERIMENT_ID not in {\\\"pilot-30m-v1\\\", \\\"nano-8k-v1\\\"}:\\n\",\n \" raise ValueError(EXPERIMENT_ID)\\n\",\n \" metadata_path = ROOT / \\\"runs\\\" / EXPERIMENT_ID / \\\"process.json\\\"\\n\",\n \" if metadata_path.is_file():\\n\",\n \" pid = int(json.loads(metadata_path.read_text(encoding=\\\"utf-8\\\"))[\\\"pid\\\"])\\n\",\n \" proc = Path(f\\\"/proc/{pid}\\\")\\n\",\n \" if proc.exists() and proc.joinpath(\\\"stat\\\").read_text().split()[2] != \\\"Z\\\":\\n\",\n \" raise RuntimeError(f\\\"Existing training process is active: PID {pid}\\\")\\n\",\n \" script = \\\"train_pilot.py\\\" if EXPERIMENT_ID == \\\"pilot-30m-v1\\\" else \\\"train.py\\\"\\n\",\n \" subprocess.run([sys.executable, \\\"-u\\\", str(ROOT / \\\"scripts\\\" / script)],\\n\",\n \" cwd=ROOT, env=env, check=True)\\n\",\n \"else:\\n\",\n \" print(\\\"Training skipped. Inspect state, then set RUN_TRAINING=True to resume.\\\")\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## WORKFLOW 08 — Completed experiment records and reports\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"metadata\": {},\n \"execution_count\": null,\n \"outputs\": [],\n \"source\": [\n \"records = [json.loads(line) for line in\\n\",\n \" (ROOT / \\\"experiments.jsonl\\\").read_text(encoding=\\\"utf-8\\\").splitlines()\\n\",\n \" if line.strip()]\\n\",\n \"print(\\\"Completed experiment records:\\\", len(records))\\n\",\n \"for record in records:\\n\",\n \" print(record.get(\\\"experiment_id\\\"), record.get(\\\"result\\\"),\\n\",\n \" \\\"steps\\\", record.get(\\\"training_steps\\\"),\\n\",\n \" \\\"validation loss\\\", record.get(\\\"validation_loss\\\"))\\n\",\n \"for name in (\\\"nano_experiment.md\\\", \\\"pilot_report.md\\\", \\\"evaluation_report.md\\\", \\\"final_report.md\\\"):\\n\",\n \" path = ROOT / \\\"reports\\\" / name\\n\",\n \" print(name, \\\"present\\\" if path.is_file() else \\\"pending\\\")\\n\"\n ]\n }\n ],\n \"metadata\": {\n \"colab\": {\n \"provenance\": []\n },\n \"kernelspec\": {\n \"display_name\": \"Python 3\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"name\": \"python\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n", "app.py": "\"\"\"\nOdiaGPT Interactive Gradio Web UI\nBuilt from Scratch - 100% PyTorch Primitives & From-Scratch Indic Tokenizer\n\"\"\"\n\nfrom pathlib import Path\nimport time\nimport json\nimport sys\n\nimport gradio as gr\nimport sentencepiece as spm\nimport torch\n\nROOT = Path(__file__).resolve().parent\nsys.path.insert(0, str(ROOT / \"src\"))\n\nfrom odiagpt.checkpoint import load_latest_checkpoint\nfrom odiagpt.config import ModelConfig\nfrom odiagpt.model import OdiaGPT\n\n# Global models and tokenizer cache\nMODELS = {}\nTOKENIZER = None\nDEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\n\ndef get_tokenizer():\n global TOKENIZER\n if TOKENIZER is None:\n tok_info_path = ROOT / \"tokenizer/nano_8k.json\"\n tok_info = json.loads(tok_info_path.read_text(encoding=\"utf-8\"))\n TOKENIZER = spm.SentencePieceProcessor(model_file=tok_info[\"model_path\"])\n return TOKENIZER\n\n\ndef get_model(model_name: str):\n global MODELS\n if model_name in MODELS:\n return MODELS[model_name]\n\n if \"30M\" in model_name:\n exp_id = \"pilot-30m-v1\"\n else:\n exp_id = \"nano-8k-v1\"\n\n checkpoint = load_latest_checkpoint(ROOT, exp_id, map_location=\"cpu\")\n if checkpoint is None:\n raise RuntimeError(f\"Checkpoint for {exp_id} not found\")\n saved, _ = checkpoint\n config = ModelConfig.from_dict(saved[\"model_config\"])\n model = OdiaGPT(config).to(DEVICE).eval()\n model.load_state_dict(saved[\"model\"])\n MODELS[model_name] = (model, config, saved[\"step\"])\n return MODELS[model_name]\n\n\n@torch.inference_mode()\ndef generate_text(\n prompt: str,\n model_choice: str,\n max_new_tokens: int,\n temperature: float,\n top_k: int,\n top_p: float,\n repetition_penalty: float,\n seed: int,\n):\n if not prompt or not prompt.strip():\n return \"ଦୟାକରି ଏକ Odia prompt ଲେଖନ୍ତୁ (Please provide an Odia prompt).\", \"0 tokens\", \"0.0s\", \"0.0 tok/s\"\n\n sp = get_tokenizer()\n model, config, step = get_model(model_choice)\n\n prompt_ids = [sp.bos_id(), *sp.encode(prompt, out_type=int)]\n ids = torch.tensor(prompt_ids, dtype=torch.long, device=DEVICE).unsqueeze(0)\n \n if seed == -1:\n seed = torch.randint(0, 100000, (1,)).item()\n generator = torch.Generator(device=DEVICE).manual_seed(int(seed))\n\n start_time = time.monotonic()\n generated_count = 0\n\n for _ in range(max_new_tokens):\n context = ids[:, -config.context_length:]\n logits, _ = model(context)\n next_logits = logits[:, -1, :].float().clone()\n \n # Suppress special tokens\n next_logits[:, [sp.pad_id(), sp.unk_id(), sp.bos_id()]] = -torch.inf\n\n # Repetition penalty\n if repetition_penalty != 1.0:\n for token_id in set(ids[0].tolist()):\n if next_logits[0, token_id] > 0:\n next_logits[0, token_id] /= repetition_penalty\n else:\n next_logits[0, token_id] *= repetition_penalty\n\n if temperature == 0.0:\n next_id = next_logits.argmax(dim=-1, keepdim=True)\n else:\n next_logits /= max(temperature, 1e-4)\n if top_k > 0:\n k = min(top_k, next_logits.size(-1))\n boundary = torch.topk(next_logits, k=k, dim=-1).values[:, -1:]\n next_logits = next_logits.masked_fill(next_logits < boundary, -torch.inf)\n if top_p < 1.0:\n sorted_logits, sorted_indices = torch.sort(next_logits, descending=True, dim=-1)\n sorted_probs = torch.softmax(sorted_logits, dim=-1)\n cumulative = sorted_probs.cumsum(dim=-1)\n remove = cumulative - sorted_probs > top_p\n sorted_logits = sorted_logits.masked_fill(remove, -torch.inf)\n next_logits = torch.full_like(next_logits, -torch.inf).scatter(\n dim=-1, index=sorted_indices, src=sorted_logits\n )\n probs = torch.softmax(next_logits, dim=-1)\n next_id = torch.multinomial(probs, num_samples=1, generator=generator)\n\n ids = torch.cat((ids, next_id), dim=1)\n generated_count += 1\n if int(next_id.item()) == sp.eos_id():\n break\n\n elapsed = time.monotonic() - start_time\n tok_per_sec = generated_count / max(elapsed, 1e-5)\n\n new_ids = ids.squeeze(0).tolist()[len(prompt_ids):]\n decode_ids = [i for i in new_ids if i != sp.eos_id()]\n completion = sp.decode(decode_ids)\n full_text = prompt + completion\n\n stats_tokens = f\"{generated_count} tokens\"\n stats_time = f\"{elapsed:.2f}s\"\n stats_speed = f\"{tok_per_sec:.1f} tok/s\"\n\n return full_text, stats_tokens, stats_time, stats_speed\n\n\ntitle_html = \"\"\"\n
\n First Odia Decoder-Only Language Model Built Completely From Scratch\n
\n