RuVI-AI/Pragya-EXP2

Pragya · exp2

A 28.5M-parameter decoder-only language model, trained from scratch. For testing, not for production.

---

⚠️ Not a product

This is a scratch checkpoint. It is not aligned, factually reliable, or safe. It exists to test a from-scratch training and architecture stack and to measure controlled changes between runs.

---

TL;DR

Total parameters	28.5M
Backbone (excluding embeddings and Engram tables)	21.25M
Embeddings (tied input and LM head)	4.72M
Engram lookup tables	2.53M
Context	1024 tokens
Vocabulary	12,288 byte-level BPE
Validation perplexity	13.6
Validation accuracy	47.1%

Validation accuracy is lower than test3's 49.7% because this run's evaluation data is harder. The perplexity and accuracy numbers are not directly comparable to test3 for that reason.

---

Model card

Architecture	Decoder-only transformer
Layers	16
Hidden size	384
Attention heads	12 query · 6 key/value (GQA)
Head dimension	32
Attention	Exclusive Gated Attention (per-head sigmoid gate + XSA)
FFN	SwiGLU, hidden size 768 (ffn_mult = 2)
Positional encoding	RoPE, base 10,000
Engram memory	Enabled at layers 1 and 8 (n-gram lookup, 3-gram max)
Mixture of Experts	Off
Multi-token prediction	Off
Embeddings	Tied (input embedding = LM head)
Context length	1024 tokens
Vocabulary	12,288 byte-level BPE (pragya.tokenizer.json)

---

Attention: Exclusive Gated Attention

Each attention block combines two per-head operations on the attention output, before the output projection. Both are zero-initialized to identity, so the model starts as plain causal attention.

Gated attention. A per-head gate is computed from the first 16 hidden dimensions of the block input:

gate    = 2 · sigmoid(W_gate · x[:16])
context = context · gate

With W_gate = 0, the gate is exactly 1, so the output passes through unchanged at step 0.

Exclusive self-attention (XSA). The component of the context that points along the token's own value vector is removed:

v_unit     = v / ‖v‖
correction = gate · (context · v_unit) · v_unit
context    = context − correction

The correction is zeroed at position 0, where causal attention can only see the token itself.

---

Engram memory

Engram adds a hashed n-gram lookup at layers 1 and 8. Token sequences are compressed into 9,807 classes using NFKC normalization, lowercasing and accent stripping, and each class indexes a learned embedding table. The lookup table is stored in the checkpoint, so the model must be loaded with the same tokenizer it was trained with. The tokenizer is checked for this at load time.

---

Training

Training tokens	~5B
Optimizer	MuonwithAdamw
Schedule	WSD
Steps	19000
Batch	320
Precision	BF16

---

Results

	exp2
Validation perplexity	13.6
Validation accuracy	47.1%

Benchmarks

Evaluated with lm-evaluation-harness, 0-shot, acc_norm.

Benchmark	Score	Chance
HellaSwag	29.76%	25%
ARC-Easy	31.57%	25%
ARC-Challenge	23.98%	25%
PIQA	61.86%	50%
ArithMark-3	38.30%	25%

Intelligence Index: 12.41, computed as

N(s, c) = 100 × (s − c) ÷ (100 − c)
Index = (N(HellaSwag, 25) + N(ARC, 25) + N(PIQA, 50) + 0.65 × N(ArithMark-3, 25)) ÷ 3.65

where ARC is the mean of ARC-Easy and ARC-Challenge.

ArithMark-3 uses acc_norm from lm-evaluation-harness. The benchmark's authors score by mean per-token log-likelihood, so this value may differ slightly from their leaderboard. Its standard error is about ±1.5 points.

Training used about 5B tokens. The AxiomicLabs reference models were trained on 15 to 75B tokens, so compare results with that gap in mind.

---

Quick start

Install: pip install torch transformers safetensors

The model and tokenizer load through transformers with remote code. Batched generation with padding is not supported yet, so generate one prompt at a time.

from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "RuVI-AI/Pragya-EXP2"   # or a local folder
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True).to("cpu").eval()

inputs = tok("The Internet is", return_tensors="pt")
out = model.generate(
    input_ids=inputs["input_ids"],
    attention_mask=inputs["attention_mask"],
    max_new_tokens=60,
    do_sample=True,
    temperature=0.7,
    top_k=20,
    repetition_penalty=1.3,
    no_repeat_ngram_size=3,
    use_cache=False,
    pad_token_id=tok.pad_token_id,
)
print(tok.decode(out[0], skip_special_tokens=True))

The tokenizer prepends <|bos|> by default. The string <|bos|> inside a prompt is treated as ordinary text.

Expected load output

Loading prints a LOAD REPORT listing model.lm_head.weight as MISSING. This is expected. The checkpoint stores the embedding once, and the head is tied to it when the model loads. Check it with:

print("tied:", model.model.transformer.wte.weight is model.model.lm_head.weight)

It should print True.

Hub download messages

On first load, Transformers prints "A new version of the following files was downloaded". This is the remote code being fetched, not an error. Pin a revision= to avoid the message once you have a version you trust.

Sampling

These settings produced fluent, on-topic output in testing:

temperature        0.7
top_k              20
repetition_penalty 1.3
no_repeat_ngram    3

Greedy decoding (do_sample=False) repeats phrases after a few sentences. Use sampling.

The model writes in a simple, children's-book register. This reflects the TinyStories-heavy data mix rather than a bug.

Capabilities

It can: continue English prompts for a sentence or two, and produce plausible text in narrative and encyclopedic registers.

It cannot: answer questions reliably, follow instructions, stay coherent past the first paragraph, handle non-English text, or use context longer than 1024 tokens.

---

## Limitations

• Small scale. Output is grammatical for a sentence or two and then drifts into invented facts.
• English only. The byte-level tokenizer encodes other languages, but the model has no training signal for them.
• No instruction tuning. Any instruction behavior is incidental.
• No batching with padding. Attention masks are not supported, so padded batches are rejected.
• No KV cache. Generation recomputes the full sequence each step.
• No alignment or safety filtering. The model can produce biased, false or toxic text.

---

License

Apache 2.0. Each training dataset carries its own license. Check upstream terms before redistributing derivatives.

---

Citation

@misc{pragya-exp2,
  title  = {Pragya exp2: a 28.5M from-scratch decoder-only language model with Exclusive Gated Attention and Engram memory},
  author = {Arush},
  year   = {2026},
  note   = {Experimental checkpoint, not for production use},
  howpublished = {\url{https://huggingface.co/<your-username>/pragya-exp2}}
}