Model Card for Model ID

Model Details

Model Description

SmaLLM is a compact causal language model implemented entirely from scratch in PyTorch. It follows modern architectural conventions inspired by LLaMA and other contemporary small language models, avoiding legacy Transformer components (absolute positional embeddings, LayerNorm, standard ReLU FFN).

The model was designed as a learning project to demonstrate understanding of modern LM architecture components and training infrastructure — not for production deployment.

Developed by: Atharva
Model type: Causal Language Model(AutoRegressive)
Language(s) (NLP): English
Training Data: Web text (essential-web corpus, ~500K tokens)
Tokenizer: vukrosic/essential-web-16k-tokenizer
Framework: Pytorch

What This Model IS For

✅ Learning how modern LM architectures work (RoPE, SwiGLU, Flash Attention, RMSNorm)
✅ Understanding training infrastructure (AMP, LR schedules, checkpointing, logging)
✅ Experimenting with architecture modifications and hyperparameter sweeps
✅ Educational demonstrations and blog posts

What This Model IS NOT For

❌ Production text generation
❌ Chatbot or conversational AI
❌ Factual question answering
❌ Any task requiring coherent, reliable output