firdavsus/LLM-D2

Model Card

Overview

This model is a 156M-parameter English-language causal language model trained on a large-scale text corpus and instruction-tuned for general question answering and task completion.

---

Running script

see github -> https://github.com/firdavsus/LLM_D2

Model Details

• Model size: 156M parameters
• Architecture: Transformer (causal LM)
• Tokenizer: GPT-2 tokenizer
• Languages: English only

---

Training Curves

Training Data

Pretraining

• Dataset: The Pile (10B token subset)
• Domain: mixed-domain text (web, books, articles, code, etc.)

Instruction Fine-tuning

• Dataset: Alpaca (cleaned subset)
• Size: ~50,000 instruction–response examples
• Formatting: instruction-style prompt/response pairs

---

Training Setup

Pretraining

• Steps: 218,000
• Final training loss: 2.6

Post-training (Instruction Fine-tuning)

• Steps: 2,500
• Final training loss: 1.9

---

Evaluation

Benchmark	Score
HellaSwag	28.5

---

Intended Use

• Instruction-style prompting
• Basic question answering
• Text generation and summarization
• Lightweight assistant-style tasks (English)

---

Limitations

• Small model size limits reasoning and factual reliability
• May produce incorrect or inconsistent answers
• Instruction-following quality depends strongly on prompt format
• Not suitable for high-stakes or safety-critical use

This model has not been safety-aligned. Please apply your own moderation and guardrails when deploying it ;)

FOR ADDITIONAL INFO CHEKC INFO.TXT