---
language: en
license: mit
tags:
- causal-lm
- custom-architecture
- transformer
pipeline_tag: text-generation
---

# Custom 57M Language Model

A custom 57.55M parameter causal language model with modern transformer architecture.

## Model Details

- **Parameters**: 57,553,632 (57.55M)
- **Architecture**: 12-layer Transformer
- **Hidden Size**: 432
- **Attention Heads**: 8
- **Head Dimension**: 54
- **Intermediate Size**: 1,728
- **Vocabulary Size**: 50,257 (GPT-2 tokenizer)
- **Max Sequence Length**: 1,024

## Architecture Features

- **RoPE Positional Embeddings**: Rotary Position Embedding (θ=10000.0)
- **SwiGLU Activation**: Swish-Gated Linear Unit in feed-forward networks
- **RMSNorm**: Root Mean Square Layer Normalization (ε=1e-06)
- **Tied Embeddings**: Input and output embeddings share weights
- **Dropout**: 0.1 dropout rate

## Training Configuration

- **Dummy Phase**: 2 epochs, 1,000 samples, LR=0.0005
- **C4 Phase**: 3 epochs, 1,000 samples, LR=0.0003
- **Optimizer**: AdamW (weight_decay=0.1)
- **Scheduler**: Cosine Annealing
- **Gradient Clipping**: 1.0

## Generation Parameters

- **Temperature**: 0.8
- **Top-K**: 50
- **Top-P**: 0.9
- **Repetition Penalty**: 1.1
- **Max New Tokens**: 100

## Usage

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("your-username/custom-57m-language-model")
model = AutoModelForCausalLM.from_pretrained("your-username/custom-57m-language-model")

input_text = "The future of artificial intelligence"
inputs = tokenizer.encode(input_text, return_tensors='pt')
outputs = model.generate(
    inputs, 
    max_length=100, 
    temperature=0.8,
    top_k=50,
    top_p=0.9,
    repetition_penalty=1.1
)
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated_text)
```

## Training Dataset

- **Primary**: C4 (Colossal Clean Crawled Corpus)
- **Warm-up**: Synthetic dummy data for initial training

## License

MIT License

## Model Card

This model was trained as an educational demonstration of transformer architecture implementation with modern techniques like RoPE embeddings and SwiGLU activations.