| # 🔬 ViuAI Sarus-500M: Master 9-Pillar Diagnostic Suite |
|
|
| A scientific evaluation and forensic diagnostic engine for the **ViuAI Sarus-500M Base Pretrained Model** (`checkpoints/ckpt_latest.pt`). |
|
|
| --- |
|
|
| ## 📋 The 9 Forensic Pillars |
|
|
| | Pillar | Diagnostic Dimension | Primary Evaluation Metric | |
| | :--- | :--- | :--- | |
| | **Pillar 1** | **Weight Health & Numerical Sanity** | NaN/Inf scan, $\ell_2$ Frobenius norms, Embedding SVD effective rank | |
| | **Pillar 2** | **Validation Loss & Perplexity** | Multi-domain cross-entropy loss ($\text{PPL} = \exp(\mathcal{L})$), Cross-lingual asymmetry ratio $\mathcal{R}_{\text{lang}}$ | |
| | **Pillar 3** | **Text Completion Test** | Autoregressive prose and Python code continuation under greedy ($T=0$) & sampling ($T=0.7$) | |
| | **Pillar 4** | **Knowledge Evaluation** | Factual recall cloze probes (Science, Geography, History, India GK) with Top-1 & Top-5 accuracy | |
| | **Pillar 5** | **Language Evaluation** | Devanagari 0 `<unk>` stress-test, Hindi Ling/Vachan grammatical agreement, Token Fertility | |
| | **Pillar 6** | **Repetition Dynamics** | Shannon prediction entropy ($H$), N-gram uniqueness ($n=2,3$), Looping penalty sensitivity | |
| | **Pillar 7** | **Context Window Scaling** | Loss curve across sequence lengths: $L \in [128, 256, 512, 1024, 2048]$ (RoPE health) | |
| | **Pillar 8** | **Standard Benchmarks** | Normalized log-likelihood multiple choice scoring on ARC-Easy and MMLU probes | |
| | **Pillar 9** | **Human Evaluation Protocol** | Standardized 1–5 Likert scale scoring template (Fluency, Grammar, Facts, Coherence) | |
|
|
| --- |
|
|
| ## 📁 Evaluation Subdirectory Layout |
|
|
| ``` |
| ViuAI_500M/evaluation/ |
| ├── README.md # Documentation & Guide |
| ├── run_full_audit.py # Master 9-Pillar Diagnostic CLI Engine |
| ├── run_eval_kaggle.py # Dedicated Kaggle Standalone Runner Script |
| ├── KAGGLE_EVALUATION_RUN.txt # 1-Click Copy-Paste Cell for Kaggle Notebooks |
| │ |
| └── benchmarks_data/ # Curated Benchmark Probe Datasets |
| ├── eval_texts.json # Bilingual & code texts for perplexity evaluation |
| ├── knowledge_probes.json # Factual recall cloze questions |
| ├── language_probes.json # Devanagari Unicode & grammatical agreement probes |
| └── standard_benchmarks.json # ARC & MMLU multiple choice questions |
| ``` |
|
|
| --- |
|
|
| ## 🚀 How to Run on Kaggle (1-Click Execution) |
|
|
| 1. Open a new Kaggle Notebook (Set Accelerator to **GPU T4 x 2** or **GPU P100**). |
| 2. Copy the contents of [`KAGGLE_EVALUATION_RUN.txt`](file:///c:/Users/IQC-SURBHI/Documents/S-Work/ViuAI/ViuAI_500M/evaluation/KAGGLE_EVALUATION_RUN.txt) into a single code cell. |
| 3. Hit **Shift + Enter**. |
| 4. The script will automatically: |
| - Verify environment dependencies. |
| - Sync the Sarus-500M architecture and evaluation suite. |
| - Download the base checkpoint `ckpt_latest.pt`. |
| - Run all 9 pillars on the GPU. |
| - Generate `evaluation/eval_results_base_model.json` (raw telemetry). |
| - Generate `evaluation/BASE_MODEL_DIAGNOSTIC_REPORT.md` (formal diagnostic report) and display it directly in the notebook output! |
|
|
