| --- |
| language: en |
| pipeline_tag: text-generation |
| library_name: pytorch |
| tags: |
| - language-model |
| - transformer |
| - decoder-only |
| - custom-architecture |
| - educational |
| - from-scratch |
| - conversational |
| datasets: |
| - Salesforce/wikitext |
| - HuggingFaceTB/smol-smoltalk |
| - Estwld/empathetic_dialogues_llm |
| --- |
| |
| # Second5M — Decoder-Only Transformer Language Model |
|
|
| Second5M is an experimental 4,943,040-parameter decoder-only language model built from scratch in PyTorch. It was pretrained on WikiText-103 and then fine-tuned on conversational data. This is an educational research project with limited conversational ability; it frequently gives incoherent or unrelated responses. |
|
|
| ## Model Details |
|
|
| | Property | Value | |
| | --- | --- | |
| | Architecture | Custom decoder-only Transformer, pre-normalization | |
| | Total trainable parameters | 4,943,040, including tied embeddings | |
| | Layers | 8 | |
| | Hidden dimension | 192 | |
| | Attention | 6 query heads, 2 key/value heads | |
| | FFN dimension | 560 (SwiGLU) | |
| | Context window | 512 tokens | |
| | Tokenizer | Custom byte-level BPE, vocabulary 8,192 | |
| | Position encoding | Rotary position embeddings (RoPE) | |
| | Normalization | RMSNorm | |
| | Weight tying | Token embeddings and output head share weights | |
|
|
| ## Quick Start |
|
|
| Requirements: `pip install torch tokenizers huggingface_hub` |
|
|
| ```python |
| import sys |
| import torch |
| from huggingface_hub import snapshot_download |
| from tokenizers import Tokenizer |
| |
| path = snapshot_download("kaafivikrant/Second5M") |
| sys.path.insert(0, path) |
| from model import ModelConfig, TinyChatLM |
| |
| import json |
| with open(f"{path}/config.json") as f: |
| config = json.load(f) |
| keys = ModelConfig.__dataclass_fields__.keys() |
| model = TinyChatLM(ModelConfig(**{k: config[k] for k in keys})) |
| model.load_state_dict(torch.load(f"{path}/pytorch_model.pt", map_location="cpu", weights_only=True)) |
| model.eval() |
| tokenizer = Tokenizer.from_file(f"{path}/tokenizer.json") |
| |
| prompt = "Hello! How are you?" |
| ids = [tokenizer.token_to_id("<|bos|>"), tokenizer.token_to_id("<|user|>")] |
| ids += tokenizer.encode(prompt).ids |
| ids += [tokenizer.token_to_id("<|eos|>"), tokenizer.token_to_id("<|assistant|>")] |
| output = model.generate(torch.tensor([ids]), max_new_tokens=80, temperature=0.7, |
| top_k=40, eos_id=tokenizer.token_to_id("<|eos|>")) |
| print(tokenizer.decode(output[0, len(ids):].tolist(), skip_special_tokens=True)) |
| ``` |
|
|
| The supplied model uses custom PyTorch code; it is not a standard Transformers `AutoModel` architecture. Download the Python code only from a source you trust. |
|
|
| ## Architecture |
|
|
| ```text |
| Tokens → tied embeddings → 8 decoder blocks → RMSNorm → tied output head → logits |
| |
| Each block: RMSNorm → causal grouped-query attention with RoPE → residual |
| RMSNorm → SwiGLU feed-forward → residual |
| ``` |
|
|
| ## Training Details |
|
|
| - Pretraining: WikiText-103 raw training data, custom tokenizer, 20,000 steps, 655,360,000 sampled tokens, NVIDIA T4. Last recorded validation loss: 3.5913. |
| - First supervised fine-tuning: up to 50,000 eligible Smol-Smoltalk conversations; 3,000 steps. Long examples in this stage were truncated from the left, sometimes removing the user prompt. |
| - Final supervised fine-tuning: 19,531 prepared training conversations from Empathetic Dialogues, initialized from the preceding chat checkpoint; 3,000 steps. The saved checkpoint records validation loss 3.0663 on that dataset. |
| - The final supervised fine-tuning used assistant-turn loss and retained the user request for long examples. Training used AdamW and a cosine learning-rate schedule. |
| - Losses from different stages use different data and should not be compared directly. No independent benchmark establishes improved chat quality. |
|
|
| ## Generation Parameters |
|
|
| `generate()` accepts `max_new_tokens`, `temperature`, `top_k`, and `eos_id`. |
|
|
| ## Limitations |
|
|
| - Responses to basic greetings have been observed to be incoherent or unrelated. |
| - This model has limited knowledge, reasoning, context, and instruction-following ability. |
| - It can make factual errors and produce repetitive, inappropriate, or misleading output. |
| - It is not suitable for production, high-stakes decisions, or emotional-support use. |
|
|
| ## Files |
|
|
| | File | Description | |
| | --- | --- | |
| | `model.py` | Standalone model definition | |
| | `pytorch_model.pt` | Model weights only | |
| | `config.json` | Model configuration and final checkpoint metadata | |
| | `tokenizer.json` | Byte-level BPE tokenizer | |
|
|
| ## Data and License Information |
|
|
| Pretraining used [WikiText](https://huggingface.co/datasets/Salesforce/wikitext), whose dataset card lists CC BY-SA 3.0 and GFDL. Fine-tuning used [Smol-Smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk) and [Empathetic Dialogues for LLM](https://huggingface.co/datasets/Estwld/empathetic_dialogues_llm), whose repository cards list Apache 2.0. Consult the original datasets and their terms before redistributing or using this model. No model license is asserted here. |
|
|
