---
language: en
pipeline_tag: text-generation
library_name: pytorch
tags:
- language-model
- transformer
- decoder-only
- custom-architecture
- educational
- from-scratch
- conversational
datasets:
- Salesforce/wikitext
- HuggingFaceTB/smol-smoltalk
- Estwld/empathetic_dialogues_llm
---

# Second5M — Decoder-Only Transformer Language Model

Second5M is an experimental 4,943,040-parameter decoder-only language model built from scratch in PyTorch. It was pretrained on WikiText-103 and then fine-tuned on conversational data. This is an educational research project with limited conversational ability; it frequently gives incoherent or unrelated responses.

## Model Details

| Property | Value |
| --- | --- |
| Architecture | Custom decoder-only Transformer, pre-normalization |
| Total trainable parameters | 4,943,040, including tied embeddings |
| Layers | 8 |
| Hidden dimension | 192 |
| Attention | 6 query heads, 2 key/value heads |
| FFN dimension | 560 (SwiGLU) |
| Context window | 512 tokens |
| Tokenizer | Custom byte-level BPE, vocabulary 8,192 |
| Position encoding | Rotary position embeddings (RoPE) |
| Normalization | RMSNorm |
| Weight tying | Token embeddings and output head share weights |

## Quick Start

Requirements: `pip install torch tokenizers huggingface_hub`

```python
import sys
import torch
from huggingface_hub import snapshot_download
from tokenizers import Tokenizer

path = snapshot_download("kaafivikrant/Second5M")
sys.path.insert(0, path)
from model import ModelConfig, TinyChatLM

import json
with open(f"{path}/config.json") as f:
    config = json.load(f)
keys = ModelConfig.__dataclass_fields__.keys()
model = TinyChatLM(ModelConfig(**{k: config[k] for k in keys}))
model.load_state_dict(torch.load(f"{path}/pytorch_model.pt", map_location="cpu", weights_only=True))
model.eval()
tokenizer = Tokenizer.from_file(f"{path}/tokenizer.json")

prompt = "Hello! How are you?"
ids = [tokenizer.token_to_id("<|bos|>"), tokenizer.token_to_id("<|user|>")]
ids += tokenizer.encode(prompt).ids
ids += [tokenizer.token_to_id("<|eos|>"), tokenizer.token_to_id("<|assistant|>")]
output = model.generate(torch.tensor([ids]), max_new_tokens=80, temperature=0.7,
                        top_k=40, eos_id=tokenizer.token_to_id("<|eos|>"))
print(tokenizer.decode(output[0, len(ids):].tolist(), skip_special_tokens=True))
```

The supplied model uses custom PyTorch code; it is not a standard Transformers `AutoModel` architecture. Download the Python code only from a source you trust.

## Architecture

```text
Tokens → tied embeddings → 8 decoder blocks → RMSNorm → tied output head → logits

Each block: RMSNorm → causal grouped-query attention with RoPE → residual
            RMSNorm → SwiGLU feed-forward → residual
```

## Training Details

- Pretraining: WikiText-103 raw training data, custom tokenizer, 20,000 steps, 655,360,000 sampled tokens, NVIDIA T4. Last recorded validation loss: 3.5913.
- First supervised fine-tuning: up to 50,000 eligible Smol-Smoltalk conversations; 3,000 steps. Long examples in this stage were truncated from the left, sometimes removing the user prompt.
- Final supervised fine-tuning: 19,531 prepared training conversations from Empathetic Dialogues, initialized from the preceding chat checkpoint; 3,000 steps. The saved checkpoint records validation loss 3.0663 on that dataset.
- The final supervised fine-tuning used assistant-turn loss and retained the user request for long examples. Training used AdamW and a cosine learning-rate schedule.
- Losses from different stages use different data and should not be compared directly. No independent benchmark establishes improved chat quality.

## Generation Parameters

`generate()` accepts `max_new_tokens`, `temperature`, `top_k`, and `eos_id`.

## Limitations

- Responses to basic greetings have been observed to be incoherent or unrelated.
- This model has limited knowledge, reasoning, context, and instruction-following ability.
- It can make factual errors and produce repetitive, inappropriate, or misleading output.
- It is not suitable for production, high-stakes decisions, or emotional-support use.

## Files

| File | Description |
| --- | --- |
| `model.py` | Standalone model definition |
| `pytorch_model.pt` | Model weights only |
| `config.json` | Model configuration and final checkpoint metadata |
| `tokenizer.json` | Byte-level BPE tokenizer |

## Data and License Information

Pretraining used [WikiText](https://huggingface.co/datasets/Salesforce/wikitext), whose dataset card lists CC BY-SA 3.0 and GFDL. Fine-tuning used [Smol-Smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk) and [Empathetic Dialogues for LLM](https://huggingface.co/datasets/Estwld/empathetic_dialogues_llm), whose repository cards list Apache 2.0. Consult the original datasets and their terms before redistributing or using this model. No model license is asserted here.