nishantup/nanogpt-slm-tinystories-classifier

nanoGPT TinyStories Spam Classifier -- 124.0M Parameters

Binary spam classifier fine-tuned from the nanoGPT TinyStories pretrained SLM.

Pipeline: Trained from scratch -> Pretrained on TinyStories (2.1M children's stories) -> Classification fine-tuned on 60K spam dataset.

What This Model Does

Classifies email/text messages as spam or not spam using a 124M parameter transformer model. The model reads the input text and uses the last token's logits from a 2-class classification head to predict the label.

# Download `nanogpt_slm_tinystories_classifier_inference.py` in working directory

from nanogpt_slm_tinystories_classifier_inference import classify, is_spam

result = classify("You won a free iPhone! Click here to claim.")
# {'label': 'spam', 'confidence': 0.95, 'probabilities': {'not spam': 0.05, 'spam': 0.95}}

is_spam("Meeting at 3pm tomorrow")  # False
is_spam("URGENT! Claim your prize NOW!")  # True

Training Pipeline

Stage	Dataset	Details
Pretraining	TinyStories (2.1M stories)	70K iters, batch 32x512, lr=6e-4
Classification SFT	60K spam dataset (balanced ham/spam emails)	3 epochs, batch 32, lr=1e-4, AdamW

Spam Dataset

The 60K spam dataset contains balanced ham (legitimate) and spam emails with integer labels (0=ham, 1=spam). The model was fine-tuned by:

1. Freezing all layers except the last transformer block, final LayerNorm, and classification head
2. Replacing lm_head (768 -> 50,257) with a 2-class head (768 -> 2)
3. Training with cross-entropy loss on last-token logits

Quick Start

Option 1: Run directly

pip install torch tiktoken huggingface_hub
python nanogpt_slm_tinystories_classifier_inference.py

Option 2: Import and use in your own code

# pip install torch tiktoken huggingface_hub
# Download `nanogpt_slm_tinystories_classifier_inference.py` in working directory

from nanogpt_slm_tinystories_classifier_inference import classify, is_spam, classify_batch

# Full result with confidence
result = classify("You won a free iPhone! Click here to claim.")
print(result)
print()

# Simple boolean check
print(is_spam("You won a free iPhone!"))      # True
print(is_spam("See you at dinner tonight!"))   # False
print()

# Batch classification
texts = ["Free prize!", "Meeting at 3pm", "Click to win $$$"]
results = classify_batch(texts)
for text, r in zip(texts, results):
    print(f"  {r['label']:>8s} ({r['confidence']:.0%}) | {text}")

Option 3: Load weights manually

from huggingface_hub import hf_hub_download
import torch, torch.nn as nn

model_path = hf_hub_download(
    repo_id="nishantup/nanogpt-slm-tinystories-classifier",
    filename="nanogpt_slm_tinystories_classifier.pth"
)

from nanogpt_slm_tinystories_classifier_inference import GPT, GPTConfig
config = GPTConfig()
model = GPT(config)
model.lm_head = nn.Linear(768, 2)
model.load_state_dict(torch.load(model_path, map_location="cpu"))
model.eval()

How It Works

1. Input text is tokenized (tiktoken GPT-2 BPE)
2. Padded/truncated to 512 tokens
3. Fed through the full transformer (12 layers)
4. Last token's logits (shape: 2) are used for classification
5. Argmax -> 0 = "not spam", 1 = "spam"

Model Architecture

Attribute	Value
Parameters	124.0M
Architecture	nanoGPT (12 layers, 12 heads, 768 dim)
Classification head	Linear(768, 2) replacing lm_head
Classes	0 = not spam, 1 = spam
Context length	512 tokens
Tokenizer	tiktoken GPT-2 BPE (50,257 tokens)
Base model	nishantup/nanogpt-slm-tinystories-124m

Files

File	Description
nanogpt_slm_tinystories_classifier.pth	Classifier weights (lm_head = Linear(768, 2))
nanogpt_slm_tinystories_classifier_inference.py	Standalone inference script
config.json	Model + classifier configuration

API Reference

classify(text, max_length=512)

Returns dict with label, confidence, probabilities.

is_spam(text, max_length=512)

Returns True if spam, False if not.

classify_batch(texts, max_length=512)

Returns list of classify() results.

Related Models (Vizuara SLM Family)

Variant	Type	Repo
Pretrained (TinyStories)	Base LM	nishantup/nanogpt-pretrained-slm-tinystories-124m
Instruction-tuned	SFT	nishantup/nanogpt-slm-tinystories-instruct
This Model- Spam classifier	Classification	nishantup/nanogpt-slm-tinystories-classifier

Author

nishantup