Girinath11/recursive-language-model-48m

Recursive Language Model - 48M (Mixture of Recursion)

A transformer-based language model with Mixture of Recursion architecture featuring adaptive recursive processing, rotary positional embeddings (RoPE), and intelligent sequence-level complexity routing for enhanced text generation.

Model Description

This model implements a novel Mixture of Recursion architecture that dynamically determines the optimal number of recursive refinement passes based on input sequence complexity. Unlike standard transformers that process all inputs uniformly, this model intelligently allocates computational resources.

Key Innovations

• 🧠 Sequence-Level Router: Neural classifier that analyzes entire sequences to predict complexity (simple/medium/complex)
• 🔄 Adaptive Recursion: 1, 3, or 5 recursive transformer passes based on router prediction
• 🌀 Rotary Positional Embeddings (RoPE): Superior positional encoding with better length generalization
• ⚡ Dynamic Computation: Efficient processing that adapts to input difficulty
• 🎯 Weight Tying: Shared embeddings between input and output layers for parameter efficiency
• 📊 Multi-Dataset Training: Trained on diverse, high-quality web text from FineWeb-Edu, Cosmopedia, and OpenWebText

Architecture Philosophy

Traditional transformers apply the same computational depth to all inputs. This model recognizes that some sequences (simple greetings, common phrases) need minimal processing, while others (technical explanations, complex reasoning) benefit from deeper iterative refinement. The router learns to make this decision automatically.

Quick Start

Installation

pip install transformers torch

Basic Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained(
    "Girinath11/recursive-language-model-48m",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    "Girinath11/recursive-language-model-48m"
)

# Move to GPU if available
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)
model.eval()

print("✅ Model loaded successfully!")

# Generate text
prompt = "The future of artificial intelligence"
input_ids = tokenizer.encode(prompt, return_tensors="pt").to(device)

outputs = model.generate(
    input_ids,
    max_new_tokens=50,
    temperature=0.8,
    top_p=0.9,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Model Architecture

Detailed Architecture Specifications

Component	Configuration
Total Parameters	48,208,641 (~48.2M)
Vocabulary Size	50,257 tokens (GPT-2 BPE)
Embedding Dimension	512
Base Transformer Layers	6
Attention Heads	8 heads per layer
Head Dimension	64 (512 ÷ 8)
FFN Intermediate Size	2048
Max Sequence Length	512 tokens
Positional Encoding	Rotary Positional Embeddings (RoPE)
Dropout Rate	0.1 (both hidden and attention)
Layer Normalization	eps=1e-5

Recursion Configuration

Complexity Class	Recursion Steps	Use Case
Simple	1 step	Common phrases, greetings, simple completions
Medium	3 steps	Standard text, moderate complexity
Complex	5 steps	Technical content, reasoning, complex narratives

Architecture Components

1. Embedding Layer

   • Token embeddings (50,257 × 512)
   • Tied with output projection for efficiency
   • Padding token handling (ID: 50256)

2. Base Transformer Stack (6 layers)

   • Multi-head self-attention with RoPE
   • Feed-forward networks (512 → 2048 → 512)
   • Pre-normalization with LayerNorm
   • Residual connections
   • Causal masking for autoregressive generation

3. Sequence-Level Router

   • Attention-weighted pooling over sequence
   • 2-layer MLP classifier (512 → 256 → 3)
   • Outputs: complexity class (0=simple, 1=medium, 2=complex)
   • Trained with pseudo-labels based on sequence length

4. Recursive Refinement Layer

   • Additional transformer block (reused 1-5 times)
   • Same architecture as base layers
   • Applied iteratively based on router decision

5. Output Projection Head

   • Linear layer (512 → 50,257)
   • Weight-tied with input embeddings
   • Final LayerNorm before projection

Rotary Positional Embeddings

Uses RoPE instead of learned positional embeddings for:

• Better extrapolation to longer sequences
• Relative position encoding
• Improved performance on positional tasks
• Base frequency: 10,000

Training Details

Training Dataset

Total Training Samples: 50,000 (high-quality web text)

Dataset	Percentage	Samples	Description
FineWeb-Edu	45%	22,500	Educational web content, filtered for quality
Cosmopedia	30%	15,000	Synthetic educational content
OpenWebText	25%	12,500	Web text from Reddit links
Validation	-	1,000	Held-out FineWeb-Edu samples

Filtering Criteria:

• Minimum sequence length: 128 tokens
• Maximum sequence length: 384 tokens
• Actual samples after filtering: ~45,000-48,000

Training Configuration

Hardware:
  GPU: NVIDIA T4 (15 GB)
  Mixed Precision: FP16
  Framework: PyTorch 2.0+ with CUDA

Hyperparameters:
  Batch Size: 1
  Gradient Accumulation: 32
  Effective Batch Size: 32
  Learning Rate: 3e-4
  Optimizer: AdamW
  Weight Decay: 0.01
  Warmup Steps: 500
  Max Gradient Norm: 1.0
  Total Epochs: 3
  Max Sequence Length: 384 tokens

Loss Function:
  Language Modeling: CrossEntropyLoss (ignore_index=-100)
  Router Loss: CrossEntropyLoss (weight: 0.1)
  Total Loss: LM Loss + 0.1 × Router Loss

Regularization:
  Hidden Dropout: 0.1
  Attention Dropout: 0.1

Training Schedule

• Total Training Steps: 4,686
• Steps per Epoch: 1,562
• Warmup: 500 steps
• Learning Rate Schedule: Linear warmup → Linear decay
• Evaluation Frequency: Every 1,000 steps
• Checkpoint Saving: Every 1,000 steps (top 2 kept)

Training Time

• Total Duration: ~2 hours 10 minutes
• Time per Step: ~1.5-1.6 seconds
• Throughput: 19.12 samples/second
• Training Speed: 0.597 steps/second

Training Progression

Checkpoint	Steps	Training Loss	Eval Loss	Perplexity	Epoch
Start	0	9.82	-	-	0.00
Checkpoint 1	1000	5.46	5.72	305.15	0.21
Checkpoint 2	2000	4.92	5.06	156.84	1.10
Checkpoint 3	3000	4.51	4.86	128.63	2.20
Final	4686	4.32	4.59	98.86	3.02

Loss Reduction: 9.82 → 4.59 (53% improvement)
Perplexity Achievement: 98.86 (excellent for 48M model!)

Performance Metrics

Final Evaluation Results

📊 FINAL METRICS:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ Evaluation Loss:      4.59
✓ Perplexity:          98.86
✓ Training Loss (avg): 5.08
✓ Total Samples Seen:  150,000 (3 epochs × 50K)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Generation Quality

Perplexity: 98.86 indicates good quality for a 48M parameter model:

• ✅ Generates coherent and grammatical sentences
• ✅ Maintains context over short passages (50-100 tokens)
• ✅ Produces diverse outputs with proper sampling
• ✅ Handles various writing styles and topics
• ✅ Suitable for creative writing, completions, and prototyping
• ⚠️ May show repetition in very long generations (200+ tokens)
• ⚠️ Less factually reliable than larger models (175B+)
• ⚠️ Limited reasoning capabilities compared to state-of-the-art

Inference Performance

Hardware	Tokens/Second	Latency (50 tokens)
CPU (Intel i7)	~100	~500ms
GPU (T4)	~500	~100ms
GPU (V100)	~800	~60ms
GPU (A100)	~1200	~40ms

Memory Requirements

Mode	RAM/VRAM	Disk Space
Model Weights	-	184 MB
CPU Inference	600 MB	-
GPU Inference (FP16)	1.5 GB	-
GPU Training (batch=1)	~8 GB	-

Advanced Usage

Batch Generation

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Girinath11/recursive-language-model-48m",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    "Girinath11/recursive-language-model-48m"
)

device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)
model.eval()

# Batch generation
prompts = [
    "The history of computing",
    "Climate change impacts",
    "Space exploration in"
]

# Tokenize all prompts
inputs = tokenizer(prompts, return_tensors="pt", padding=True).to(device)

# Generate for all prompts at once
outputs = model.generate(
    **inputs,
    max_new_tokens=50,
    temperature=0.8,
    top_p=0.9,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id
)

# Decode all outputs
for i, output in enumerate(outputs):
    print(f"\nPrompt {i+1}: {prompts[i]}")
    print(f"Generated: {tokenizer.decode(output, skip_special_tokens=True)}")

Fine-tuning on Custom Data

from transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling
from datasets import load_dataset

# Load your custom dataset
dataset = load_dataset("your_dataset")

# Tokenize
def tokenize(examples):
    return tokenizer(examples['text'], truncation=True, max_length=384)

tokenized = dataset.map(tokenize, batched=True)

# Training arguments
training_args = TrainingArguments(
    output_dir="./finetuned-model",
    per_device_train_batch_size=1,
    gradient_accumulation_steps=32,
    learning_rate=1e-4,  # Lower LR for fine-tuning
    num_train_epochs=1,
    fp16=True,
    save_steps=500,
    logging_steps=100,
)

# Data collator
data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer,
    mlm=False
)

# Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized['train'],
    data_collator=data_collator,
)

# Fine-tune
trainer.train()

Temperature and Sampling Control

# Creative writing (high temperature)
creative_output = model.generate(
    input_ids,
    max_new_tokens=100,
    temperature=1.0,      # More random
    top_p=0.95,
    top_k=50,
    do_sample=True
)

# Focused completion (low temperature)
focused_output = model.generate(
    input_ids,
    max_new_tokens=100,
    temperature=0.5,      # More deterministic
    top_p=0.9,
    top_k=40,
    do_sample=True
)

# Greedy decoding (most likely tokens)
greedy_output = model.generate(
    input_ids,
    max_new_tokens=50,
    do_sample=False       # Greedy
)

Technical Architecture

Model Structure

Input Text
    ↓
[Token Embedding Layer] (50,257 × 512)
    ↓
[6× Base Transformer Blocks]
    ├─ Multi-Head Attention (8 heads, RoPE)
    ├─ Feed-Forward Network (512 → 2048 → 512)
    └─ LayerNorm + Residual Connections
    ↓
[Sequence-Level Router]
    ├─ Attention-Weighted Pooling
    ├─ MLP Classifier (512 → 256 → 3)
    └─ Output: Complexity Class (0/1/2)
    ↓
[Adaptive Recursive Refinement]
    ├─ Simple:  1× Recursion Layer
    ├─ Medium:  3× Recursion Layer
    └─ Complex: 5× Recursion Layer
    ↓
[Final LayerNorm]
    ↓
[LM Head] (512 → 50,257, weight-tied)
    ↓
Output Tokens

Layer Breakdown

1. Embedding Layer (25.7M params)

• Token embeddings: 50,257 × 512 = 25,731,584 params
• Weight-tied with output projection

2. Base Transformer (6 layers, ~19M params)

• Each layer: ~3.15M params
  • Self-attention: 4 × (512 × 512) = 1,048,576
  • FFN: 2 × (512 × 2048) = 2,097,152
  • LayerNorms: small overhead

3. Router Network (~0.4M params)

• Pooler: 512 × 512 = 262,144
• Classifier: (512 × 256) + (256 × 3) = 131,840

4. Recursion Layer (~3.15M params)

• Single transformer block (reused 1-5 times)
• Same structure as base layers

5. Output Components

• Final LayerNorm: ~1K params
• LM Head: weight-tied (0 additional params)

Rotary Positional Embeddings (RoPE)

# RoPE computation
inv_freq = 1.0 / (10000 ** (torch.arange(0, dim, 2).float() / dim))
t = torch.arange(seq_len)
freqs = torch.einsum('i,j->ij', t, inv_freq)
emb = torch.cat((freqs, freqs), dim=-1)
cos, sin = emb.cos(), emb.sin()

# Applied to queries and keys before attention
q_rotated = (q * cos) + (rotate_half(q) * sin)
k_rotated = (k * cos) + (rotate_half(k) * sin)

Benefits:

• ✅ Better length extrapolation
• ✅ Relative position awareness
• ✅ No learned position parameters
• ✅ Efficient computation

Training Details

Dataset Composition

Training Data: 50,000 samples from three high-quality sources

Dataset	Source	Percentage	Samples	Description
FineWeb-Edu	HuggingFace	45%	22,500	Educational web pages, high quality
Cosmopedia	HuggingFace	30%	15,000	Synthetic educational textbooks
OpenWebText	Community	25%	12,500	Web text from Reddit submissions

Data Preprocessing:

• Tokenization: GPT-2 BPE tokenizer
• Truncation: 384 tokens max
• Filtering: Minimum 128 tokens per sample
• No data augmentation applied

Validation Set: 1,000 samples from FineWeb-Edu

Training Hyperparameters

Batch Configuration:
  Per-Device Batch Size: 1
  Gradient Accumulation: 32
  Effective Batch Size: 32
  Total Training Steps: 4,686
  Steps per Epoch: 1,562

Optimization:
  Optimizer: AdamW
  Learning Rate: 3e-4
  Weight Decay: 0.01
  Warmup Steps: 500
  LR Schedule: Linear warmup → Linear decay
  Max Gradient Norm: 1.0
  Beta1: 0.9
  Beta2: 0.999
  Epsilon: 1e-8

Mixed Precision:
  Enabled: True
  Format: FP16
  Loss Scaling: Dynamic

Regularization:
  Hidden Dropout: 0.1
  Attention Dropout: 0.1
  No additional regularization

Evaluation:
  Strategy: steps
  Eval Steps: 1,000
  Metric: eval_loss
  Best Model Selection: Minimum eval_loss

Loss Function

Composite Loss = Language Modeling Loss + Router Loss

# Language modeling loss (primary)
lm_loss = CrossEntropyLoss(
    predictions=shift_logits,
    targets=shift_labels,
    ignore_index=-100  # Ignore padding tokens
)

# Router loss (auxiliary, 10% weight)
router_loss = CrossEntropyLoss(
    predictions=complexity_logits,
    targets=pseudo_labels  # Based on sequence length
)

# Total loss
total_loss = lm_loss + 0.1 * router_loss

Router pseudo-labels assignment:

• Sequence length < 170: Simple (class 0)
• Sequence length 170-340: Medium (class 1)
• Sequence length > 340: Complex (class 2)

Training Metrics Over Time

Loss Progression

Step	Epoch	Training Loss	Eval Loss	Eval Perplexity
100	0.02	9.82	-	-
500	0.11	6.29	-	-
1000	0.21	5.46	5.72	305.15
1500	0.32	5.09	-	-
2000	1.10	4.92	5.06	156.84
2500	1.29	4.51	-	-
3000	2.20	4.51	4.86	128.63
3500	2.30	4.24	-	-
4000	2.85	4.32	4.59	98.86
4686	3.02	4.32	4.59	98.86

Training Dynamics

Loss Improvement by Phase:

• Epoch 1 (0-1562 steps): 9.82 → 5.16 (47% reduction) - Rapid initial learning
• Epoch 2 (1563-3124 steps): 5.16 → 4.38 (15% reduction) - Steady refinement
• Epoch 3 (3125-4686 steps): 4.38 → 4.32 (1% reduction) - Fine convergence

Gradient Norms: Remained stable (0.7-1.5), indicating healthy training without exploding/vanishing gradients.

Learning Rate Schedule:

• Warmup (0-500 steps): 0 → 3e-4
• Peak (500-1000): 3e-4
• Decay (1000-4686): 3e-4 → ~6e-6

Final Training Statistics

Total Runtime:         7,844 seconds (2h 10m 44s)
Samples Processed:     150,000 (50K × 3 epochs)
Training Throughput:   19.12 samples/second
Steps per Second:      0.597
Average Step Time:     1.67 seconds
GPU Utilization:       ~90-95%
Peak Memory Usage:     ~8.5 GB (GPU)

Performance Benchmarks

Perplexity Comparison

Model	Parameters	Perplexity	Notes
This Model	48M	98.86	Mixture of Recursion, 3 epochs
Baseline GPT-2 Small	117M	~29	Official OpenAI
TinyLlama	1.1B	~10	Much larger
Random Baseline	-	~50,000	Theoretical worst case

Context: For a 48M parameter model trained on 50K samples, perplexity of 98.86 is competitive and indicates good learning.

Generation Quality Assessment

Strengths:

• ✅ Grammatically correct output
• ✅ Coherent short-form text (1-3 sentences)
• ✅ Diverse vocabulary usage
• ✅ Proper punctuation and capitalization
• ✅ Context maintenance in short passages

Weaknesses:

• ⚠️ Occasional repetition in long generations
• ⚠️ Limited factual knowledge (small training set)
• ⚠️ May generate generic/vague statements
• ⚠️ Struggles with very technical topics
• ⚠️ Short context window (384 tokens)

Limitations & Considerations

Technical Limitations

1. Context Window: 512 token maximum (vs 2048+ for modern models)
2. Model Size: 48M parameters - limited capacity vs billions-scale models
3. Training Data: 50K samples - relatively small dataset
4. Single Language: Primarily English (GPT-2 tokenizer bias)
5. Domain Coverage: Limited by training data diversity
6. Reasoning: Basic completion, limited multi-step reasoning

Known Issues

• Repetition: May repeat phrases after 100+ tokens
• Factual Errors: Small knowledge base, may hallucinate facts
• Consistency: Long-form coherence degrades over 200+ tokens
• Technical Domains: Struggles with highly specialized topics
• Math/Code: Limited capability for formal reasoning
• Context Retention: May lose track of earlier context in long sequences

Generation Artifacts

• Occasional incomplete sentences at max_tokens boundary
• May generate run-on sentences without proper punctuation
• Sometimes produces generic filler phrases
• Temperature tuning needed for optimal quality

Ethical Considerations

Bias & Fairness

This model may exhibit biases inherited from training data:

Potential Biases:

• Geographic: Overrepresentation of Western/English content
• Demographic: Gender, age, cultural biases from web text
• Temporal: Training data reflects content up to 2024
• Topic: Educational content may skew certain perspectives

Mitigation Strategies:

• Diverse training data sources (FineWeb-Edu, Cosmopedia, OpenWebText)
• No explicit harmful content filtering (relies on source quality)
• Users should validate outputs for fairness-critical applications

Responsible Use

✅ Recommended:

• Educational demonstrations
• Research on adaptive computation
• Creative writing assistance (with human review)
• Prototyping and experimentation
• Learning about language models

❌ Not Recommended:

• Medical, legal, or financial advice
• Generating authoritative content without verification
• Creating misleading or deceptive content
• Applications requiring high factual accuracy
• Automated content moderation or decision-making
• Safety-critical systems

Environmental Impact

Training Carbon Footprint (Estimated):

• GPU Hours: ~2.2 hours on T4
• Estimated CO₂: ~0.15 kg (assuming 0.068 kg/GPU-hour for T4)
• Relatively low impact due to small model size and short training

Comparison with Similar Models

Model	Params	Perplexity	Architecture	Special Features
This Model	48M	98.86	Mixture of Recursion	Adaptive depth, RoPE
GPT-2 Small	117M	~29	Standard Transformer	OpenAI, well-tested
DistilGPT-2	82M	~35	Distilled GPT-2	Faster inference
GPT-Neo 125M	125M	~25	Mesh Transformer	More data, larger

Trade-offs:

• ✅ Smaller size: Better for deployment
• ✅ Novel architecture: Research value
• ✅ Adaptive computation: Potentially more efficient
• ❌ Higher perplexity: Less predictive accuracy
• ❌ Less training: Smaller knowledge base

Model Card & Transparency

Intended Use

Primary Use Cases:

• 📚 Education: Teaching language model concepts
• 🔬 Research: Experimenting with adaptive computation
• 🛠️ Prototyping: Testing LM-based applications
• 💡 Learning: Understanding transformer architectures

Out-of-Scope Uses:

• Production chatbots without oversight
• Generating factual content for publication
• Automated decision systems
• Content requiring domain expertise

Evaluation Methodology

Metrics:

• Primary: Perplexity on validation set
• Secondary: Training loss, gradient norms
• Qualitative: Manual review of generations

Evaluation Data:

• 1,000 samples from FineWeb-Edu (held-out)
• Same preprocessing as training data
• Evaluated every 1,000 training steps