Model Card for SLMa-94M

SLMa-94M is a 94-million parameter causal language model trained entirely from scratch on synthetic educational data. It uses a modern Llama-style transformer architecture and is designed to maximize reasoning capability within an extreme parameter budget. This is a raw base model — no instruction tuning or alignment — intended for research, fine-tuning, and bare-metal edge deployment.

Model Details

Model Description

SLMa-94M was pretrained on the HuggingFaceTB/cosmopedia dataset following the "Phi Strategy": training on highly structured, textbook-quality synthetic data to maximize reasoning per parameter. The entire training run was completed on a Nvidia T4 GPU on Kaggle.

The model uses a Llama-style architecture with Grouped-Query Attention (GQA), SwiGLU activations, RMSNorm, and Rotary Position Embeddings (RoPE). The vocabulary size of 24,576 is deliberately divisible by 128 for Tensor Core alignment.

• Developed by: Amartya Pandey
• Model type: Causal Language Model (decoder-only transformer)
• Language(s) (NLP): English
• License: Apache 2.0
• Finetuned from model: None — trained from scratch

Model Sources

• Repository: https://github.com/amartya-pandey/SLMa-94M-baremetal
• Tokenizer: amartya-pandey/slm-tokenizer-24k

Uses

Direct Use

Text generation and language modeling research. The model can be used as-is for open-ended text generation, continuation tasks, and exploring the capabilities of small pretrained models.

Note: This model requires trust_remote_code=True as the architecture is defined in model.py in this repository.

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="amartya-pandey/SLMa-94M", trust_remote_code=True)
output = pipe("The history of the universe is", max_new_tokens=200)
print(output[0]["generated_text"])

# Load model directly
from transformers import AutoModelForCausalLM, AutoTokenizer

# Always use the custom tokenizer — do not substitute a generic one
tokenizer = AutoTokenizer.from_pretrained("amartya-pandey/slm-tokenizer-24k")
model = AutoModelForCausalLM.from_pretrained(
    "amartya-pandey/SLMa-94M",
    trust_remote_code=True,
)

inputs = tokenizer("The history of the universe is", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.7, top_p=0.9)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Downstream Use

A natural starting point for fine-tuning on instruction-following, question answering, domain-specific text generation, or any task that benefits from a small, fast, and portable language model.

Out-of-Scope Use

• Production systems requiring factual accuracy or safety guarantees — this model has no alignment training
• Tasks requiring context longer than 768 tokens
• Use with any tokenizer other than amartya-pandey/slm-tokenizer-24k

Bias, Risks, and Limitations

• Trained exclusively on synthetic, educational-style English text; likely to underperform on informal, colloquial, or non-English content
• No instruction tuning, RLHF, or safety training of any kind — outputs are unfiltered
• Small context window (768 tokens) limits use on long-form tasks
• No formal benchmark evaluation has been conducted

Recommendations

Users should treat this as a research artifact. Fine-tune with appropriate safety considerations before any deployment. Do not use for tasks where factual correctness or safe outputs are required.

How to Get Started with the Model

See the Direct Use section above for pipeline and direct-load code snippets.

For bare-metal inference on Android or other resource-constrained devices (no Python, no PyTorch, no ONNX), see the companion repository: SLMa-94M-baremetal.

Training Details

Training Data

Trained on HuggingFaceTB/cosmopedia, a large-scale dataset of synthetic textbook-style text. Token sequences are packed using a custom IsolatedPackedDataset that uses block-diagonal attention matrices to prevent cross-document attention leakage.

Training Procedure

Preprocessing

Documents are tokenized with the custom 24k BPE tokenizer (amartya-pandey/slm-tokenizer-24k) and packed into fixed-length blocks of 768 tokens using IsolatedPackedDataset.

Training Hyperparameters

• Training regime: fp16 mixed precision (torch.autocast)
• Optimizer: AdamW (lr=1e-3, weight_decay=0.1)
• Learning Rate Schedule: Linear warmup followed by cosine decay to min_lr=1e-4
• Gradient Accumulation: 4 steps
• Hardware: Nvidia T4 GPU (Kaggle)
• Experiment Tracking: Weights & Biases

Evaluation

No formal evaluation has been conducted at this time. The model is released as a research artifact for the community to explore, fine-tune, and benchmark.

Testing Data, Factors & Metrics

Not evaluated.

Results

Not available.

Technical Specifications

Model Architecture and Objective

Decoder-only causal language model (next-token prediction). Architecture details:

Hyperparameter	Value
Parameters	~94.3 M
Transformer Layers	12
Embedding Dimension	768
Context Window	768 tokens
Vocabulary Size	24,576 (divisible by 128 for Tensor Core alignment)
Attention	Grouped-Query Attention — 12 Query heads, 4 KV heads
Activations	SwiGLU
Normalization	RMSNorm (pre-norm)
Position Embeddings	RoPE (real-valued)

Compute Infrastructure

Hardware

Nvidia T4x2 GPU (16 GB VRAM) via Kaggle free tier.

Software

PyTorch with torch.autocast (fp16), Hugging Face transformers and safetensors, Weights & Biases for logging.

Model Card Authors

Amartya Pandey