|
|
|
|
| --- |
| language: en |
| license: apache-2.0 |
| tags: |
| - text-generation |
| - conversational |
| - gpt |
| - from-scratch |
| - lora |
| - pytorch |
| --- |
| |
| # TinyTalk-124M |
|
|
| TinyTalk-124M is a 124-million-parameter conversational language model built entirely from scratch in PyTorch — custom transformer architecture, custom training loop, and a two-stage training pipeline (base pretraining followed by LoRA instruction fine-tuning). |
|
|
| ## Model Details |
|
|
| - **Architecture**: Custom decoder-only transformer |
| - 12 transformer blocks |
| - Embedding dimension: 512 |
| - 8 attention heads |
| - Rotary Position Embeddings (RoPE) |
| - SwiGLU feed-forward layers |
| - Causal self-attention via scaled dot-product attention |
| - **Vocabulary**: GPT-2 tokenizer (50,257 tokens) |
| - **Context length**: 512 tokens |
| - **Total parameters**: ~124.2M |
| - **Format**: safetensors (fp32) |
|
|
| ## Training Pipeline |
|
|
| **Stage A — Base Pretraining** |
| The model was pretrained from scratch on general text data to learn language modeling fundamentals. |
|
|
| **Stage B — Conversational Fine-Tuning (LoRA)** |
| The pretrained base was fine-tuned using LoRA (Low-Rank Adaptation) on a ChatAlpaca-style instruction/response dataset (~68K examples) to teach instruction-following and conversational behavior. LoRA was applied to attention (`qkv_proj`, `out_proj`) and feed-forward (`w1`, `w2`, `w3`) projections, then merged back into the base weights for standalone deployment. |
|
|
| - LoRA rank: 64, alpha: 128 |
| - Effective batch size: 32 (batch size 16 × grad accumulation 2) |
| - Mixed precision (bf16) training |
| - Cosine learning rate schedule with warmup |
| - Early stopping based on validation loss |
|
|
| **Final validation perplexity**: ~25.8 |
|
|
| ## Prompt Format |
|
|
| The model expects an Alpaca-style instruction format: |
|
|
| ``` |
| ### Instruction: |
| {your instruction here} |
| |
| ### Response: |
| ``` |
|
|
| ## Intended Use |
|
|
| This model is a small-scale educational/portfolio project demonstrating an end-to-end from-scratch LLM pipeline: architecture design, pretraining, parameter-efficient fine-tuning, and deployment. It performs reasonably on everyday conversational instructions but is **not** intended for production use, factual reliability, or specialized/technical domains. |
|
|
| ## Limitations |
|
|
| - **Scale**: At 124M parameters, this model has limited world knowledge and reasoning capability compared to larger LLMs. |
| - **Domain coverage**: Performs noticeably better on general conversational prompts (matching its fine-tuning data) than on specialized technical or factual topics, where output coherence can degrade. |
| - **Not factually reliable**: Outputs should not be trusted for factual accuracy, especially on scientific, medical, legal, or technical subjects. |
| - **English only.** |
|
|
| ## Example |
|
|
| **Input:** |
| ``` |
| ### Instruction: |
| Write a short poem about the ocean. |
| |
| ### Response: |
| ``` |
|
|
| **Output:** |
| ``` |
| The sea is filled with stars, rapping deep in the sky and its rays float so high |
| that it can see as far away as the horizon. It has been through for centuries, |
| since we have lived to this day. |
| ``` |
|
|
| ## Citation / Acknowledgment |
|
|
| Built as a personal/educational project exploring transformer architecture design, staged LLM training (pretraining + LoRA fine-tuning), and end-to-end model deployment. |
| ``` |
