# 🔬 ViuAI Sarus-500M: Master 9-Pillar Diagnostic Suite

A scientific evaluation and forensic diagnostic engine for the **ViuAI Sarus-500M Base Pretrained Model** (`checkpoints/ckpt_latest.pt`).

---

## 📋 The 9 Forensic Pillars

| Pillar | Diagnostic Dimension | Primary Evaluation Metric |
| :--- | :--- | :--- |
| **Pillar 1** | **Weight Health & Numerical Sanity** | NaN/Inf scan, $\ell_2$ Frobenius norms, Embedding SVD effective rank |
| **Pillar 2** | **Validation Loss & Perplexity** | Multi-domain cross-entropy loss ($\text{PPL} = \exp(\mathcal{L})$), Cross-lingual asymmetry ratio $\mathcal{R}_{\text{lang}}$ |
| **Pillar 3** | **Text Completion Test** | Autoregressive prose and Python code continuation under greedy ($T=0$) & sampling ($T=0.7$) |
| **Pillar 4** | **Knowledge Evaluation** | Factual recall cloze probes (Science, Geography, History, India GK) with Top-1 & Top-5 accuracy |
| **Pillar 5** | **Language Evaluation** | Devanagari 0 `<unk>` stress-test, Hindi Ling/Vachan grammatical agreement, Token Fertility |
| **Pillar 6** | **Repetition Dynamics** | Shannon prediction entropy ($H$), N-gram uniqueness ($n=2,3$), Looping penalty sensitivity |
| **Pillar 7** | **Context Window Scaling** | Loss curve across sequence lengths: $L \in [128, 256, 512, 1024, 2048]$ (RoPE health) |
| **Pillar 8** | **Standard Benchmarks** | Normalized log-likelihood multiple choice scoring on ARC-Easy and MMLU probes |
| **Pillar 9** | **Human Evaluation Protocol** | Standardized 1–5 Likert scale scoring template (Fluency, Grammar, Facts, Coherence) |

---

## 📁 Evaluation Subdirectory Layout

```
ViuAI_500M/evaluation/
├── README.md                      # Documentation & Guide
├── run_full_audit.py              # Master 9-Pillar Diagnostic CLI Engine
├── run_eval_kaggle.py             # Dedicated Kaggle Standalone Runner Script
├── KAGGLE_EVALUATION_RUN.txt      # 1-Click Copy-Paste Cell for Kaggle Notebooks
│
└── benchmarks_data/               # Curated Benchmark Probe Datasets
    ├── eval_texts.json            # Bilingual & code texts for perplexity evaluation
    ├── knowledge_probes.json      # Factual recall cloze questions
    ├── language_probes.json       # Devanagari Unicode & grammatical agreement probes
    └── standard_benchmarks.json   # ARC & MMLU multiple choice questions
```

---

## 🚀 How to Run on Kaggle (1-Click Execution)

1. Open a new Kaggle Notebook (Set Accelerator to **GPU T4 x 2** or **GPU P100**).
2. Copy the contents of [`KAGGLE_EVALUATION_RUN.txt`](file:///c:/Users/IQC-SURBHI/Documents/S-Work/ViuAI/ViuAI_500M/evaluation/KAGGLE_EVALUATION_RUN.txt) into a single code cell.
3. Hit **Shift + Enter**.
4. The script will automatically:
   - Verify environment dependencies.
   - Sync the Sarus-500M architecture and evaluation suite.
   - Download the base checkpoint `ckpt_latest.pt`.
   - Run all 9 pillars on the GPU.
   - Generate `evaluation/eval_results_base_model.json` (raw telemetry).
   - Generate `evaluation/BASE_MODEL_DIAGNOSTIC_REPORT.md` (formal diagnostic report) and display it directly in the notebook output!