Viveka-34M (विवेक)

Viveka is Sanskrit for discernment: telling what is right from what only looks right.

A small English language model trained from scratch (its own tokenizer, random start) on free Colab T4 GPUs. Its strength is judging: given a few versions of a sentence, it tells which one makes the most sense.

Model

Design	Llama-style (rotary positions, SwiGLU, RMSNorm), 8 layers, width 512, 33.9M numbers
Tokenizer	its own byte-level BPE, 16,000 word pieces
Context	256 word pieces
Training	150 minutes from scratch on one T4, then 150 more minutes on two T4s at a lower learning rate (578M more word pieces)
Stored as	half precision, 68 MB

Training text

Source	Licence
FineWeb-Edu (educational web pages)	ODC-BY
Project Gutenberg books	public domain
English Wikipedia	CC BY-SA
This version (v3, November 2026) learned only from these openly licensed texts: no IAM and nothing
non-commercial. So it is released under CC BY-SA 4.0 (share-alike, because of the Wikipedia text): you may use it,
change it and share it, also commercially, as long as you credit it and share changes under the same licence.
Earlier versions (CC BY-NC 4.0, partly trained on IAM handwriting texts) are still in this repo's history.

Results

On 20 fresh everyday sentences that no version trained on (lower is better):

	Perplexity per word	Sense test
First Viveka-34M	233.8	12 / 12
Viveka v2 (with 10% IAM text)	176.9	12 / 12
This version (v3, IAM-free)	137.3	12 / 12

On 30 held-out passages of each kind of training text (never trained on), per-word perplexity:

	Web (FineWeb-Edu)	Books (Gutenberg)	Wikipedia
Viveka v2	193.7	887.8	330.4
v3	144.2	530.9	209.6

The sense test asks it to pick the sensible sentence ("She drank a glass of cold water." over "...cold window.").

Use

from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained('vrishv/Viveka-34M')
lm = AutoModelForCausalLM.from_pretrained('vrishv/Viveka-34M', dtype='float32')
x = tok('The cat sat on the', return_tensors='pt')
print(tok.decode(lm.generate(**x, max_new_tokens=20, do_sample=True, top_k=40)[0]))

Limitations

Very small: it knows grammar and common phrases but little about the world. It is meant for judging sentences, not chatting or facts. English only; it may repeat biases or old-fashioned language from its training text.

About the maker

Viveka was built by Vrishvardhan Sharma (vrishv), a 9-year-old self-taught builder from India. His other work:

• GitaVardhan: a free, offline Bhagavad Gita app in 13 Indian languages, and a pocket-sized voice-controlled Gita device he designed, wired and coded himself.
• Sanskrit Vardhan: an app for learning Sanskrit through the Bhagavad Gita, with word-by-word meanings and step-by-step Pāṇinian derivations. Launched on 20 October 2026 by Prof. Shrinivasa Varakhedi, Vice-Chancellor of Central Sanskrit University.
• Kannada, Tamil and Telugu Vardhan: apps that teach everyday spoken Kannada, Tamil and Telugu.
• Hindi Vardhan (hindivardhan.in): a dictionary of pure, Sanskrit-rooted Hindi. Over 6,000 borrowed words in everyday Hindi, each with native alternatives checked against five classical Sanskrit dictionaries.
• AksharaVardhan: a handwriting reader that runs on an ordinary computer.
• graphics-studio: a free Space that makes and edits images, videos, music and narration.
• Shodhana-34M: Viveka trained further to correct lines misread by a handwriting reader.
• Jigyasa Simple: 473 everyday science questions with short, plain-text answers.
• Jigyasa Medium and Jigyasa Big: 969 and 970 deeper science questions, built from his own questions, with answers of up to 7 and 10 sentences.
• router-34M and thinker-34M: Viveka fine-tuned to pick which offline model should answer a question, and whether it needs careful thinking.

He has memorised the Bhagavad Gita and Pāṇini's Aṣṭādhyāyī, won national Sanskrit recitation competitions, and speaks six languages. More at vrishvardhan.in.

CC BY-SA 4.0 © 2026 Vrishvardhan Sharma.