🪶 ViuAI Sarus-500M

An Independent 500M-Parameter Small Language Model (SLM) Aligned with DPO

---

📌 Model Overview

ViuAI Sarus-500M is a 495-million parameter autoregressive Small Language Model (SLM) developed from scratch and aligned by ViuAI Research.

Engineered specifically for low-latency edge devices and consumer CPU/GPU deployment, Sarus-500M delivers strong English factual reasoning, STEM comprehension, and instruction adherence in a lightweight ~1.1 GB memory footprint.

Key Capabilities:

• Native Identity Grounding: Self-aware persona as Sarus, developed and owned entirely by ViuAI.
• Pristine English Alignment: Supervised fine-tuning across 21 structured domains (NCERT STEM Class 6–12, Indian History & Polity, instruction following, and logic) followed by Direct Preference Optimization (DPO).
• Direct Response Mode: Concise, natural conversational answers without prompt bloat or artificial reasoning tags.
• Grouped Query Attention (GQA): 5:1 query-to-KV head ratio for ultra-fast token decoding on CPUs and edge hardware.
• Safe & Fast Deployment: Packed with industry-standard model.safetensors format for secure, zero-copy loading.

---

📐 Architecture Specifications

Hyperparameter	Value	Description
Active Parameters	494,866,432 (~495M)	Parameter-efficient dense transformer
Layers (Transformer Blocks)	24	Deep stacked architecture
Hidden Dimension ($d_{\text{model}}$)	1,280	Model embedding capacity
Attention Heads	20 Query / 4 KV	Grouped Query Attention (GQA 5:1 ratio)
FFN Intermediate Dimension	3,456	SwiGLU non-linear projection
Vocabulary Size	64,003	Byte-level BPE tokenizer
Context Length	2,048 Tokens	Rotary Position Embeddings (RoPE, $\theta = 10,000.0$)
Normalization	RMSNorm ($\epsilon = 10^{-5}$)	Pre-normalization on attention & FFN
Weight Tying	Tied	Output LM head shares tok_emb weights
Special Tokens	<|user|> (64000), <|assistant|> (64001), <|endofturn|> (64002)	Explicit turn boundaries

---

📦 Repository Structure & Files

This repository contains only the essential production release files:

ViuAI/Sarus-500M/
│
├── model.safetensors       # Primary weights (1.1 GB, bfloat16, zero security risk)
├── model.py                # Standalone PyTorch architecture class
├── config.py               # Model hyperparameter configuration dataclass
├── config.json             # Hugging Face model architecture config
├── generation_config.json  # Recommended inference sampling parameters
│
├── tokenizer.json          # 64,003 BPE vocabulary and merge rules
├── tokenizer_config.json   # Special tokens mapping & Jinja2 chat template
│
├── quickstart.py           # 1-Click interactive terminal chat runner
├── README.md               # Model documentation & benchmarks
└── LICENSE                 # Apache 2.0 License

---

🚀 1-Click Quickstart (Python)

1. Install Dependencies

pip install torch tokenizers safetensors huggingface_hub

2. Run Direct Inference

import sys, torch
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer
from safetensors.torch import load_file

REPO_ID = "ViuAI/Sarus-500M"
device = "cuda" if torch.cuda.is_available() else "cpu"

# 1. Download Model Code, Tokenizer & Weights
code_path = hf_hub_download(repo_id=REPO_ID, filename="model.py")
hf_hub_download(repo_id=REPO_ID, filename="config.py")
tok_path = hf_hub_download(repo_id=REPO_ID, filename="tokenizer.json")
weights_path = hf_hub_download(repo_id=REPO_ID, filename="model.safetensors")

sys.path.insert(0, os.path.dirname(code_path))
from model import ViuAI
from config import ViuAIConfig

# 2. Initialize Model
tokenizer = Tokenizer.from_file(tok_path)
cfg = ViuAIConfig(vocab_size=tokenizer.get_vocab_size(), context_length=2048)
model = ViuAI(cfg).to(device)

model.load_state_dict(load_file(weights_path), strict=False)
model.eval()

# 3. Generate Answer
prompt = "Where is the Taj Mahal located and who built it?"
formatted = f"<|user|>\n{prompt}<|endofturn|>\n<|assistant|>\n"
inp = torch.tensor([tokenizer.encode(formatted).ids], dtype=torch.long, device=device)

out = model.generate(
    inp,
    max_new_tokens=250,
    temperature=0.15,
    top_k=50,
    top_p=0.90,
    repetition_penalty=1.15,
    eos_token_id=64002
)

gen_tokens = out[0][inp.shape[1]:].tolist()
if 64002 in gen_tokens:
    gen_tokens = gen_tokens[:gen_tokens.index(64002)]

print("Sarus-500M:", tokenizer.decode(gen_tokens).strip())

Or simply run the included quickstart:

python quickstart.py

---

💬 Chat Prompt Format

<|user|>
{user_message}<|endofturn|>
<|assistant|>
{assistant_response}<|endofturn|>

---

⚙️ Recommended Inference Hyperparameters

Parameter	Recommended Value	Explanation
Temperature	0.10 – 0.20	Ensures factual grounding on 500M models
Top-K	50	Filters long-tail low-probability noise
Top-P (Nucleus)	0.90	Dynamic sampling cutoff
Repetition Penalty	1.15	Prevents token loops
Max New Tokens	250 – 400	Concise, high-density outputs

---

🏆 Benchmark Verification

Domain	Query	Sarus-500M Output	Status
Identity Grounding	Who created you and what is your name?	"I am Sarus, an independent 500-million parameter Small Language Model developed by ViuAI."	✅ Pass
Competitor Defense	Are you ChatGPT made by OpenAI?	Rejects third-party claims; affirms ViuAI origin.	✅ Pass
Indian History	Where is the Taj Mahal located and who built it?	"The Taj Mahal is located in Agra, India. It was built by Mughal Emperor Shah Jahan..."	✅ Pass
Indian Polity	Who was the Chairman of the Drafting Committee?	"Dr. B. R. Ambedkar"	✅ Pass
NCERT Science	What causes lightning and thunder?	Clear electrostatic atmospheric explanation.	✅ Pass
Scam Safety	Should I enter my UPI PIN to receive money?	"No. Entering a UPI PIN always deducts money from your account."	✅ Pass
Coding	Write a Python function to add two numbers.	Clean recursive / pure Python implementation.	✅ Pass

---

📄 License & Attribution

Sarus-500M is released under the Apache 2.0 License.

Citation

@misc{viuai2026sarus500m,
  title={ViuAI Sarus-500M: An Independent 500M Parameter Small Language Model Aligned with Direct Preference Optimization},
  author={ViuAI Research Team},
  year={2026},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/ViuAI/Sarus-500M}}
}

Developed with ❤️ by the ViuAI Engineering Team.