evaluation/README.md

🔬 ViuAI Sarus-500M: Master 9-Pillar Diagnostic Suite

A scientific evaluation and forensic diagnostic engine for the ViuAI Sarus-500M Base Pretrained Model (checkpoints/ckpt_latest.pt).

---

📋 The 9 Forensic Pillars

Pillar	Diagnostic Dimension	Primary Evaluation Metric
Pillar 1	Weight Health & Numerical Sanity	NaN/Inf scan, $\ell_2$ Frobenius norms, Embedding SVD effective rank
Pillar 2	Validation Loss & Perplexity	Multi-domain cross-entropy loss ($\text{PPL} = \exp(\mathcal{L})$), Cross-lingual asymmetry ratio $\mathcal{R}_{\text{lang}}$
Pillar 3	Text Completion Test	Autoregressive prose and Python code continuation under greedy ($T=0$) & sampling ($T=0.7$)
Pillar 4	Knowledge Evaluation	Factual recall cloze probes (Science, Geography, History, India GK) with Top-1 & Top-5 accuracy
Pillar 5	Language Evaluation	Devanagari 0 <unk> stress-test, Hindi Ling/Vachan grammatical agreement, Token Fertility
Pillar 6	Repetition Dynamics	Shannon prediction entropy ($H$), N-gram uniqueness ($n=2,3$), Looping penalty sensitivity
Pillar 7	Context Window Scaling	Loss curve across sequence lengths: $L \in [128, 256, 512, 1024, 2048]$ (RoPE health)
Pillar 8	Standard Benchmarks	Normalized log-likelihood multiple choice scoring on ARC-Easy and MMLU probes
Pillar 9	Human Evaluation Protocol	Standardized 1–5 Likert scale scoring template (Fluency, Grammar, Facts, Coherence)

---

📁 Evaluation Subdirectory Layout

ViuAI_500M/evaluation/
├── README.md                      # Documentation & Guide
├── run_full_audit.py              # Master 9-Pillar Diagnostic CLI Engine
├── run_eval_kaggle.py             # Dedicated Kaggle Standalone Runner Script
├── KAGGLE_EVALUATION_RUN.txt      # 1-Click Copy-Paste Cell for Kaggle Notebooks
│
└── benchmarks_data/               # Curated Benchmark Probe Datasets
    ├── eval_texts.json            # Bilingual & code texts for perplexity evaluation
    ├── knowledge_probes.json      # Factual recall cloze questions
    ├── language_probes.json       # Devanagari Unicode & grammatical agreement probes
    └── standard_benchmarks.json   # ARC & MMLU multiple choice questions

---

🚀 How to Run on Kaggle (1-Click Execution)

1. Open a new Kaggle Notebook (Set Accelerator to GPU T4 x 2 or GPU P100).
2. Copy the contents of KAGGLE_EVALUATION_RUN.txt into a single code cell.
3. Hit Shift + Enter.
4. The script will automatically:
   • Verify environment dependencies.
   • Sync the Sarus-500M architecture and evaluation suite.
   • Download the base checkpoint ckpt_latest.pt.
   • Run all 9 pillars on the GPU.
   • Generate evaluation/eval_results_base_model.json (raw telemetry).
   • Generate evaluation/BASE_MODEL_DIAGNOSTIC_REPORT.md (formal diagnostic report) and display it directly in the notebook output!