opt-125m-cluster-v2

This model is a fine-tuned version of facebook/opt-125m, trained on a mixed dataset consisting of OpenWebText, WikiText, and BookCorpus. It was trained on a single GPU (Quadro RTX 8000, 48GB VRAM) using Hugging Face Transformers and PyTorch.

📈 Evaluation Results

• Final Training Loss: 2.9084
• Final Perplexity (Eval): 19.10
• Evaluation Steps: Every 5,000 training steps
• Total Training Steps: 50,000

🧠 Model Description

This model was fine-tuned to reduce perplexity on general English text using causal language modeling (next-token prediction). The model was trained from scratch on 1 million samples with sequence length 1024 and optimized with AdamW and cosine learning rate scheduling.

✅ Intended Uses & Limitations

Intended uses:

• Perplexity benchmarking
• Research on training dynamics and convergence
• Fine-tuning base for instruction tuning or domain adaptation

Limitations:

• Not instruction-tuned
• Not aligned for safe deployment
• May reflect biases from internet text

📊 Training & Evaluation Data

A shuffled dataset combining:

• 60% OpenWebText
• 30% WikiText
• 10% BookCorpus

All data was pre-tokenized using the OPT tokenizer and capped at 1024 tokens per sample.

⚙️ Training Procedure

• Batch size: 5 (accumulated to 40 via gradient_accumulation_steps=8)
• Learning rate: 2e-4
• Optimizer: AdamW with betas (0.9, 0.999), eps 1e-8
• LR scheduler: Cosine decay with 1,000 warmup steps
• Precision: Mixed (fp16 with AMP)
• Steps: 50,000
• Framework: Transformers 4.49.0, PyTorch 2.6.0

---

Let me know if you want this converted into a README.md format with YAML frontmatter as well.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 5
• eval_batch_size: 3
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 40
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 1000
• training_steps: 50000
• mixed_precision_training: Native AMP

Training results

📊 Training Results

steps	Perplexity	Cross-Entropy Loss
5k	24.07	3.1811
10k	23.28	3.1476
15k	22.44	3.1110
20k	21.63	3.0742
25k	20.97	3.0432
30k	20.33	3.0121
35k	19.73	2.9819
40k	19.32	2.9611
45k	19.11	2.9500
50k	19.10	2.9498

Framework versions

• Transformers 4.49.0
• Pytorch 2.6.0+cu124
• Datasets 3.3.2
• Tokenizers 0.21.1