language: en license: mit library_name: pytorch tags: - text-generation - conversational - transformer - lora - from-scratch - pytorch pipeline_tag: text-generation
TinyTalk-124M
TinyTalk-124M is a 124M-parameter conversational language model, built by fine-tuning a from-scratch pretrained transformer backbone (ScratchLM-95M) using LoRA (Low-Rank Adaptation). The entire pipeline — architecture, pretraining, and fine-tuning — was implemented independently in PyTorch, with no pretrained weights sourced from any external model at any stage.
Model Details
| Developed by | Debarun Das |
| Model type | Decoder-only transformer, LoRA fine-tuned |
| Language | English |
| License | MIT |
| Base model | ScratchLM-95M (trained from scratch) |
| Parameters | ~124.2M |
| Framework | PyTorch |
Architecture
TinyTalk-124M inherits its backbone from ScratchLM-95M, a 12-layer decoder-only transformer using:
- RoPE (Rotary Positional Embeddings) for relative position encoding
- SwiGLU activation in the feed-forward layers (as used in LLaMA, Mistral, and Gemma)
- Causal self-attention with Flash Attention (SDPA fallback)
- Pre-Norm transformer blocks with residual connections
- GPT-2 BPE tokenizer (50,257 vocabulary)
| Parameter | Value |
|---|---|
| Layers | 12 |
| Embedding Dimension | 512 |
| Attention Heads | 8 (head_dim = 64) |
| FFN Intermediate Dim | 2,730 |
| Context Length | 512 tokens |
| Base Perplexity (WikiText-103) | 24.40 |
Fine-Tuning Approach
TinyTalk-124M was produced using a parameter-efficient LoRA adaptation strategy:
- The base model backbone remains frozen throughout fine-tuning
- Trainable low-rank matrices (rank = 64, alpha = 128) are injected into attention (
qkv_proj,out_proj) and feed-forward (w1,w2,w3) projections - Adapter weights are merged directly into the base after training:
W' = W₀ + B·A · (α/r), producing a standalone, dependency-free checkpoint - Fine-tuned on 69,220 instruction/response conversational pairs (ChatAlpaca-style format)
- Trained with AdamW, cosine learning rate decay, and mixed precision (AMP)
| Metric | Value |
|---|---|
| Adaptation Method | LoRA (r=64, α=128) |
| Fine-Tuning Data | 69,220 instruction/response pairs |
| Training Epochs | 5 |
| Validation Perplexity | ~25.8 |
Intended Use
TinyTalk-124M is designed for:
- Instruction-following and conversational text generation
- Research and educational demonstrations of from-scratch LLM pretraining and LoRA fine-tuning
- Lightweight experimentation on consumer-grade hardware
As a 124M-parameter model, it is best suited to general conversational and instructional prompts rather than tasks requiring extensive world knowledge or long-context reasoning — its scale reflects a deliberate choice to make the full architecture-to-deployment pipeline reproducible on a single consumer GPU.
How to Use
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("Debarun12/tinytalk-124m")
model = AutoModelForCausalLM.from_pretrained("Debarun12/tinytalk-124m")
prompt = "How can I improve my relationship with my family?"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
**inputs,
max_new_tokens=80,
repetition_penalty=3.0,
do_sample=True,
temperature=0.7
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Training Data
The fine-tuning dataset consists of 69,220 cleaned instruction/response conversational pairs in an Alpaca-style format, curated and deduplicated to maintain consistent formatting and answer quality.
Training Infrastructure
- Hardware: Single consumer GPU
- Framework: PyTorch (no external training libraries)
- Precision: Mixed precision (AMP)
- Optimizer: AdamW with cosine learning rate decay
Citation
If you use this model, please cite:
@misc{tinytalk124m2026,
author = {Debarun Das},
title = {TinyTalk-124M: A Conversational Language Model Built From Scratch},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/Debarun12}}
}
Author
Debarun Das GitHub · Hugging Face