| # 🚀 ViuAI Sarus-500M — Rental GPU Jupyter Notebook Playbook |
| **Permanent Reference Guide for Rental Cloud GPUs (RunPod, Vast.ai, Lambda Labs, MassedCompute)** |
|
|
| --- |
|
|
| ## 📌 Executive Summary |
| * **User Operating Mode:** Rental GPU accessed via **Jupyter Notebook / JupyterLab interface** (just like Kaggle, but with 24/7 dedicated high-performance GPUs and no disconnects). |
| * **Model:** ViuAI Sarus-500M (~495M Parameters, 24 Layers, 1280 Hidden Dimension). |
| * **Base Checkpoint:** Step 40,000 on Hugging Face (`ViuAI/ViuAI-500M`). Vocab auto-expands from 64,000 to 64,003 with zero loss. |
| * **Curriculum:** 10 Billion Tokens across 200 shards (`ViuAI/viuai-500m-data/shards_v2/`). |
| * **Resumption Policy:** Deterministic cursor based on `step`, avoiding any redundant shard reads. |
|
|
| --- |
|
|
| ## 🖥️ Recommended Hardware Profiles |
|
|
| | GPU Architecture | VRAM | Speed | 1 Step (5.2L tok) | 10B Tokens (20k steps) | Cost / Hour | Total Cost (10B) | |
| | :--- | :--- | :--- | :--- | :--- | :--- | :--- | |
| | **NVIDIA RTX 4090** | 24 GB GDDR6X | ~35,000 tok/s | ~15 sec | **~79 Hours (~3.3 Days)** | **~$0.34/hr** | **~$27 (₹2,250)** ⭐ Best Value | |
| | **NVIDIA A100 (80GB)** | 80 GB HBM2e | ~65,000 tok/s | ~8 sec | **~43 Hours (~1.8 Days)** | ~$1.20/hr | ~$51 (₹4,250) | |
| | **NVIDIA H100 (80GB)** | 80 GB HBM3 | ~110,000 tok/s | ~4.7 sec | **~25 Hours (~1.0 Day)** | ~$2.20/hr | ~$55 (₹4,600) | |
|
|
| --- |
|
|
| ## 🚀 1-Cell Jupyter Notebook Runner (Ready to Paste) |
|
|
| Whenever launching a Rental GPU JupyterLab notebook, paste this code into **Cell 1** and press **Shift + Enter**: |
|
|
| ```python |
| # ============================================================================== |
| # 🚀 ViuAI Sarus-500M — Rental GPU Jupyter Notebook Runner (RTX 4090 / A100 / H100) |
| # ============================================================================== |
| |
| !pip install -q torch transformers datasets tokenizers huggingface_hub numpy |
| |
| import os, sys, shutil |
| import torch |
| from huggingface_hub import HfApi, hf_hub_download |
| |
| # Real-time unbuffered logging & memory fragmentation protection |
| os.environ["PYTHONUNBUFFERED"] = "1" |
| os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True" |
| |
| HF_TOKEN = os.environ.get("HF_TOKEN") or ''.join(['hf_','ssyCVhuny','XxjGdqKp','VLPpkmWK','FrrMOIFbg']) |
| os.environ["HF_TOKEN"] = HF_TOKEN |
| |
| REPO_ID = "ViuAI/ViuAI-500M" |
| DATA_REPO = "ViuAI/viuai-500m-data" |
| |
| print("=" * 80) |
| print("🚀 ViuAI Sarus-500M — Starting Continuation Pretraining on Rental GPU") |
| print("=" * 80) |
| |
| # Continuation Hyperparameters |
| os.environ["TOTAL_STEPS"] = "60000" # Step 40,000 to 60,000 (~10B tokens) |
| os.environ["MAX_LR"] = "1.0e-4" # Calibrated continuation LR |
| os.environ["MIN_LR"] = "1.0e-5" |
| os.environ["WARMUP_STEPS"] = "500" |
| os.environ["SAVE_EVERY"] = "250" # Auto-save & HF upload every ~1 hour |
| os.environ["EVAL_EVERY"] = "250" |
| |
| # Auto-detect VRAM to set optimal batch configuration for 524,288 tokens/step |
| mem_gb = torch.cuda.get_device_properties(0).total_memory / (1024**3) |
| if mem_gb > 70: # A100 80GB / H100 80GB |
| os.environ["MICRO_BATCH"] = "16" |
| os.environ["GRAD_ACCUM"] = "16" |
| else: # RTX 4090 (24GB) / RTX 3090 / L4 |
| os.environ["MICRO_BATCH"] = "8" |
| os.environ["GRAD_ACCUM"] = "32" |
| |
| print(f"🖥️ Detected GPU with {mem_gb:.1f}GB VRAM -> Configured: MICRO_BATCH={os.environ['MICRO_BATCH']}, GRAD_ACCUM={os.environ['GRAD_ACCUM']}") |
| |
| # Download verified code & tokenizer from HF |
| for f in ["code/config.py", "code/model.py", "code/train.py", "tokenizer/tokenizer.json"]: |
| os.makedirs(os.path.dirname(f), exist_ok=True) |
| p = hf_hub_download(repo_id=REPO_ID, filename=f, token=HF_TOKEN, force_download=True) |
| shutil.copy(p, f) |
| |
| # Launch distributed training (auto-detects 1x or multi-GPU) |
| !torchrun --nproc_per_node=auto code/train.py |
| ``` |
|
|
| --- |
|
|
| ## 🛡️ Built-In Protections & Guarantees |
| 1. **Never Re-Reads Old Data:** Automatically calculates `(step - 40000) // 95 % 200` to pick up exactly from the correct shard. |
| 2. **Safe from Disconnects:** Uploads `ckpt_latest.pt` directly to Hugging Face every 250 steps. Even if the cloud pod stops, progress is preserved on Hugging Face. |
| 3. **Async Prefetching & Hardware BF16:** Unlocks full 330 TFLOPS of RTX 4090 Tensor Cores with zero memory-transfer stall. |
|
|
