File size: 4,935 Bytes
a8e5aec | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 | ---
language: en
pipeline_tag: text-generation
library_name: pytorch
tags:
- language-model
- transformer
- decoder-only
- custom-architecture
- educational
- from-scratch
- conversational
datasets:
- Salesforce/wikitext
- HuggingFaceTB/smol-smoltalk
- Estwld/empathetic_dialogues_llm
---
# Second5M — Decoder-Only Transformer Language Model
Second5M is an experimental 4,943,040-parameter decoder-only language model built from scratch in PyTorch. It was pretrained on WikiText-103 and then fine-tuned on conversational data. This is an educational research project with limited conversational ability; it frequently gives incoherent or unrelated responses.
## Model Details
| Property | Value |
| --- | --- |
| Architecture | Custom decoder-only Transformer, pre-normalization |
| Total trainable parameters | 4,943,040, including tied embeddings |
| Layers | 8 |
| Hidden dimension | 192 |
| Attention | 6 query heads, 2 key/value heads |
| FFN dimension | 560 (SwiGLU) |
| Context window | 512 tokens |
| Tokenizer | Custom byte-level BPE, vocabulary 8,192 |
| Position encoding | Rotary position embeddings (RoPE) |
| Normalization | RMSNorm |
| Weight tying | Token embeddings and output head share weights |
## Quick Start
Requirements: `pip install torch tokenizers huggingface_hub`
```python
import sys
import torch
from huggingface_hub import snapshot_download
from tokenizers import Tokenizer
path = snapshot_download("kaafivikrant/Second5M")
sys.path.insert(0, path)
from model import ModelConfig, TinyChatLM
import json
with open(f"{path}/config.json") as f:
config = json.load(f)
keys = ModelConfig.__dataclass_fields__.keys()
model = TinyChatLM(ModelConfig(**{k: config[k] for k in keys}))
model.load_state_dict(torch.load(f"{path}/pytorch_model.pt", map_location="cpu", weights_only=True))
model.eval()
tokenizer = Tokenizer.from_file(f"{path}/tokenizer.json")
prompt = "Hello! How are you?"
ids = [tokenizer.token_to_id("<|bos|>"), tokenizer.token_to_id("<|user|>")]
ids += tokenizer.encode(prompt).ids
ids += [tokenizer.token_to_id("<|eos|>"), tokenizer.token_to_id("<|assistant|>")]
output = model.generate(torch.tensor([ids]), max_new_tokens=80, temperature=0.7,
top_k=40, eos_id=tokenizer.token_to_id("<|eos|>"))
print(tokenizer.decode(output[0, len(ids):].tolist(), skip_special_tokens=True))
```
The supplied model uses custom PyTorch code; it is not a standard Transformers `AutoModel` architecture. Download the Python code only from a source you trust.
## Architecture
```text
Tokens → tied embeddings → 8 decoder blocks → RMSNorm → tied output head → logits
Each block: RMSNorm → causal grouped-query attention with RoPE → residual
RMSNorm → SwiGLU feed-forward → residual
```
## Training Details
- Pretraining: WikiText-103 raw training data, custom tokenizer, 20,000 steps, 655,360,000 sampled tokens, NVIDIA T4. Last recorded validation loss: 3.5913.
- First supervised fine-tuning: up to 50,000 eligible Smol-Smoltalk conversations; 3,000 steps. Long examples in this stage were truncated from the left, sometimes removing the user prompt.
- Final supervised fine-tuning: 19,531 prepared training conversations from Empathetic Dialogues, initialized from the preceding chat checkpoint; 3,000 steps. The saved checkpoint records validation loss 3.0663 on that dataset.
- The final supervised fine-tuning used assistant-turn loss and retained the user request for long examples. Training used AdamW and a cosine learning-rate schedule.
- Losses from different stages use different data and should not be compared directly. No independent benchmark establishes improved chat quality.
## Generation Parameters
`generate()` accepts `max_new_tokens`, `temperature`, `top_k`, and `eos_id`.
## Limitations
- Responses to basic greetings have been observed to be incoherent or unrelated.
- This model has limited knowledge, reasoning, context, and instruction-following ability.
- It can make factual errors and produce repetitive, inappropriate, or misleading output.
- It is not suitable for production, high-stakes decisions, or emotional-support use.
## Files
| File | Description |
| --- | --- |
| `model.py` | Standalone model definition |
| `pytorch_model.pt` | Model weights only |
| `config.json` | Model configuration and final checkpoint metadata |
| `tokenizer.json` | Byte-level BPE tokenizer |
## Data and License Information
Pretraining used [WikiText](https://huggingface.co/datasets/Salesforce/wikitext), whose dataset card lists CC BY-SA 3.0 and GFDL. Fine-tuning used [Smol-Smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk) and [Empathetic Dialogues for LLM](https://huggingface.co/datasets/Estwld/empathetic_dialogues_llm), whose repository cards list Apache 2.0. Consult the original datasets and their terms before redistributing or using this model. No model license is asserted here.
|