✨ Astral-0.6B-Base

A highly efficient, long-context multilingual LLM built to bridge English and Indic language understanding at scale.

---

📖 Table of Contents

• Overview
• Key Features
• Architecture
• Shared KV — What & Why
• Benchmarks
• Detailed Evaluation Logs
• Multilingual Tokenization Efficiency
• Quick Start
• Usage Tips
• Notes & License

---

🔍 Overview

Astral-0.6B-Base is a multilingual decoder-only Transformer trained from scratch on a 1 trillion token corpus emphasizing English and Indic languages. It is architected for a low memory footprint, long-context usage, and practical deployment across CPUs and GPUs, with attention and KV-cache optimizations purpose-built for high-throughput inference.

🚀 Key Features

🧠 Architecture	600M-parameter decoder-only Transformer
📚 Pretraining	1T tokens, robust multilingual corpus
🌐 Language Coverage	English + 11 Indic languages (Hindi, Bengali, Tamil, Telugu, Marathi, Gujarati, Kannada, Malayalam, Punjabi, Odia, Assamese)
📏 Context Length	12,288 tokens
💾 Memory Efficiency	Shared KV — ~50% KV-cache reduction
⚡ Attention	Grouped Query Attention (GQA)
🧩 Normalization	RMSNorm with QK Normalization
🔗 Feed-Forward	SwiGLU FFN
🗜️ Embeddings	Factorized, tied embeddings tuned for Indic scripts

🏗️ Architecture

Full model configuration, for reference and reproducibility:

Parameter	Value
Vocabulary Size	131,072
Hidden Size	1,440
Intermediate Size	3,280
Hidden Layers	30
Attention Heads	16
Key/Value Heads	8 (GQA)
Head Dimension	90
Hidden Activation	SiLU (SwiGLU FFN)
Max Position Embeddings	12,288
RMSNorm Epsilon	1e-06
RoPE Theta	1,000,000
Query Normalization	Enabled (q_norm)
Shared KV	Enabled
Factorized Embedding	Enabled (rank 512)
Tied Word Embeddings	Yes
Attention / MLP Bias	Disabled
Torch dtype	bfloat16

🔗 Shared KV — What & Why

Shared KV derives both Key and Value projections from a shared latent representation, applying lightweight key normalization and RoPE transforms at runtime. This reduces KV-cache memory by ~50%, trading a small compute overhead for large memory savings — ideal for memory-constrained or long-context inference.

Two supported cache modes:

# Memory-efficient mode (recommended for long contexts)
model.config.kv_cache_mode = "shared"

# Standard mode (max compatibility, lower compute overhead)
model.config.kv_cache_mode = "vanilla"

---

📊 Benchmark Highlights

Benchmarks reported using the released Astral-0.6B-Base (trained on 1T tokens), evaluated with the lm-evaluation-harness framework in a 0-shot setting against other popular sub-1B architectures: Lumma-0.6B-Base, MobiLlama-0.5B, SmolLM-360M (v1), SmolLM2-360M, and Qwen2.5-0.5B.

All metrics reflect 0-shot performance (acc_norm where available, otherwise acc). Scores are illustrative and depend on evaluation setup.

Benchmark Task	Astral-0.6B	Lumma 0.6B	MobiLlama 0.5B	SmolLM 360M (v1)	SmolLM2 360M	Qwen2.5 0.5B
ARC-Challenge	32.76%	28.58%	29.52%	—	—	31.83%
ARC-Easy	61.36%	60.60%	—	—	—	—
ARC (Average)	47.06%	44.59%	—	50.10%	53.00%	45.40%
HellaSwag	46.39%	46.25%	52.52%	51.80%	54.50%	51.20%
PIQA	69.37%	—	—	71.60%	71.70%	69.90%
Winogrande	52.01%	54.14%	52.50%	52.80%	52.50%	54.10%

💡 Key Insights

• Strong knowledge & reasoning (ARC): Scoring 32.76% on ARC-Challenge, Astral edges out Qwen2.5-0.5B (31.83%), MobiLlama-0.5B (29.52%), and Lumma 0.6B (28.58%).
• Competitive physical commonsense (PIQA): At 69.37%, Astral sits close to Qwen2.5-0.5B (69.90%) and trails the heavily data-optimized SmolLM family.
• Room for growth: On HellaSwag and Winogrande, Astral trails models trained on much larger data budgets (e.g., SmolLM2's 4T-token FineWeb-Edu corpus) — a clear direction for future data scaling.

📈 Detailed Evaluation Logs

Raw zero-shot metrics extracted from evaluation logs:

Tasks	Version	Filter	n-shot	Metric	Value	Stderr
arc_challenge	1	none	0	acc	0.2892	0.0133
		none	0	acc_norm	0.3276	0.0137
arc_easy	1	none	0	acc	0.6305	0.0099
		none	0	acc_norm	0.6136	0.0100
hellaswag	1	none	0	acc	0.3714	0.0048
		none	0	acc_norm	0.4639	0.0050
winogrande	1	none	0	acc	0.5201	0.0140
lambada_openai	1	none	0	acc	0.4328	0.0069
		none	0	perplexity	17.9850	0.6382
lambada_standard	1	none	0	acc	0.3926	0.0068
		none	0	perplexity	23.3805	0.8805
piqa	1	none	0	acc	0.6855	0.0108
		none	0	acc_norm	0.6937	0.0108
sciq	1	none	0	acc	0.8340	0.0118
		none	0	acc_norm	0.7980	0.0127

🌏 Multilingual Tokenization Efficiency (Fertility)

Tokenization fertility measures the average number of tokens generated per word unit (calculated on the devtest split of FLORES-200). Lower fertility = higher efficiency — shorter sequences, lower inference cost, and better context-window utilization for non-English languages.

Language	Astral-0.6B	Lumma-0.6B	Sarvam-1	Qwen3-0.6B	SmolLM3-3B
English	1.31	1.18	1.32	1.16	1.17
Bengali	1.75	1.44	1.55	7.51	8.66
Gujarati	1.79	1.53	1.55	9.37	10.47
Hindi	1.46	1.32	1.25	5.14	2.71
Kannada	2.26	1.90	2.10	12.96	16.43
Malayalam	2.54	2.05	2.49	14.56	17.77
Marathi	1.82	1.55	1.55	6.70	3.73
Odia	2.21	2.68	2.18	15.75	19.07
Punjabi	1.50	1.42	1.47	8.66	9.23
Tamil	2.12	2.05	2.06	10.93	13.56
Telugu	2.06	1.77	2.09	13.38	15.40
Assamese	1.89	1.51	4.31	8.13	9.26

🎯 Architectural Insights

• Sub-1B Indic superpower: Astral delivers a major structural advantage over global baseline models like Qwen3-0.6B and SmolLM3-3B. In languages like Kannada and Malayalam, global tokenizers fracture words byte-by-byte into 12x–17x more tokens than English, exhausting context windows rapidly — Astral handles these natively with compressed fertility scores between 2.26 and 2.54.
• Targeted regional optimization: Astral achieves a notable efficiency breakthrough in Odia (2.21), outperforming Lumma-0.6B (2.68), and runs circles around Sarvam-1 in Assamese (1.89 vs. 4.31).
• Balanced vocabulary allocation: While localized tokenizers often sacrifice English compression to support fallback character sets, Astral maintains a lean balance — staying competitive on English text (1.31 fertility) while keeping every major Indic script under ~2.5 tokens per word.

---

⚡ Quick Start

Install:

pip install transformers==5.4.0 torch --upgrade

Minimal usage example:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "pihu21057w/Astral-0.6B-Base"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name, trust_remote_code=True, dtype=torch.bfloat16
).eval()

# choose KV cache mode
model.config.kv_cache_mode = "shared"  # or "vanilla"

prompt = "when you encounter something then"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=50,
    do_sample=True,
    temperature=0.3,
    top_p=0.95,
    top_k=20,
    repetition_penalty=1.1,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

🛠️ Usage Tips

• For long-context tasks, prefer shared KV to reduce memory usage. Monitor CPU/GPU memory when scaling context length.
• Use bfloat16 where available for reduced memory footprint with minimal quality loss.
• Apply SFT or RLHF on top of this base model for instruction-following, conversational behavior.

📄 Notes & License

This document summarizes the released Astral-0.6B-Base model. Check the model repository for source data, training recipes, and license details.

If you redistribute or build on Astral, please follow the original model license and cite appropriately.

---

Made with care for multilingual and long-context research & deployment. 🌏