File size: 5,264 Bytes
ac955ba 12616ed ac955ba 12616ed 80d85e9 12616ed | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 | ---
license: apache-2.0
language:
- en
- ru
pipeline_tag: text-generation
tags:
- llama-3
- custom-tokenizer
- bilingual
- code
- math
---
# LLM D6 (~0.8B)
**LLM D6** is the 6th model in the series, built on a custom LLaMA-3 architecture. It features **0.8B parameters** and was pre-trained from scratch on **300 Billion tokens** of high-quality, diverse data spanning English, Russian, programming code, and mathematical reasoning.
This repository (`firdavsus/LLM_multimodal`) contains both the **Pre-trained** base weights and the **Fine-tuned** instruct/chat weights, along with training logs and evaluation benchmarks.
---
## Benchmarks
## Fine Tuning Grapth
## 💡 Key Highlights
* **Bilingual & Domain-Focused:** Pre-trained on a balanced mix of English, Russian, Code, and Mathematics.
* **Custom Tokenizer:** Trained from scratch on the dataset with a **52,000 vocabulary size**, yielding higher token compression efficiency for Russian and technical syntax.
* **High Token Efficiency:** Achieves competitive performance at 300B tokens compared to models trained on 9T–18T tokens.
* **Dual Weights Included:** Structured subdirectories for both `pre-train` and `fine-tune` stages.
---
## 📐 Model Architecture & Specifications
| Parameter | Value |
| :--- | :--- |
| **Architecture** | LLaMA-3 style (`LlamaForCausalLM`) |
| **Total Parameters** | ~800M (0.8B) |
| **Hidden Dimension ($d_{\text{model}}$)** | 1536 |
| **Number of Layers** | 24 |
| **Head Dimension** | 128 |
| **Attention Heads** | 12 |
| **Vocabulary Size** | 52,000 (Custom Trained) |
| **Context Pre-training Tokens** | 300 Billion Tokens |
---
## 📁 Repository Structure
```text
firdavsus/LLM_multimodal/
├── pre-train/ # Pre-trained base weights & tokenizer
├── fine-tune/ # Supervised Fine-Tuned (SFT) weights & configs
├── benchmarks.png # Evaluation comparison curves & charts
├── training_curves_finetune.png # SFT Loss, LR, and Gradient stability plots
└── README.md # Model Card
```
---
## 📊 Evaluation & Benchmarks
The model was evaluated using `lm-evaluation-harness` across zero-shot and few-shot tasks, comparing **Custom 0.8B (300B tokens)** against **Qwen 2.5 0.5B (18.0T tokens)** and **Llama 3.2 1B (9.0T tokens)**.
### Final Benchmark Performance Summary
| Benchmark | Task Metric | **Custom 0.8B** *(300B Tokens)* | **Qwen 2.5 0.5B** *(18,000B Tokens)* | **Llama 3.2 1B** *(9,000B Tokens)* |
| --- | --- | --- | --- | --- |
| **PIQA** | Physical Commonsense | **69.3%** | 69.8% | 74.5% |
| **HellaSwag** | General Reasoning | **47.8%** | 52.2% | 64.1% |
| **XWinograd RU** | Russian Language Understanding | **58.7%** | 56.8% | 66.0% |
| **ARC-Challenge** | Grade-School Science | **30.6%** | 32.2% | 37.2% |
| **MMLU** | Multi-task Accuracy | **25.5%** | 47.4% | 37.8% |
> **Note:** Despite being trained on significantly fewer tokens (0.3T vs 18T/9T), **LLM D6** demonstrates strong bilingual capability, outperforming Qwen 2.5 0.5B on Russian Winograd evaluation (`58.7%` vs `56.8%`) and maintaining competitive common-sense physical reasoning (`69.3%` on PIQA).
---
## 📉 Fine-Tuning Dynamics
Fine-tuning was conducted over **3,600 steps** using a cosine learning rate decay schedule with a peak learning rate of $2.0 \times 10^{-5}$ and initial linear warmup.
* **Loss Convergence:** Both training and evaluation loss decayed smoothly down to **~1.16**, showing consistent learning without over-fitting.
* **Gradient Stability:** Gradient norm remained tightly bounded between **0.5 and 1.5** throughout the execution.
---
## 🚀 How to Use
### Loading the Fine-Tuned Model
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "firdavsus/LLM_multimodal"
# Load Custom Tokenizer and Model Weights from fine-tune folder
tokenizer = AutoTokenizer.from_pretrained(model_id, subfolder="fine-tune")
model = AutoModelForCausalLM.from_pretrained(
model_id,
subfolder="fine-tune",
torch_dtype=torch.bfloat16,
device_map="auto"
)
prompt = "Solve the following problem step by step: What is the derivative of f(x) = x^3 + 2x?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
**inputs,
max_new_tokens=256,
temperature=0.7,
top_p=0.9
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```
### Loading the Pre-Trained Base Weights
```python
tokenizer = AutoTokenizer.from_pretrained("firdavsus/LLM_multimodal", subfolder="pre-train")
model = AutoModelForCausalLM.from_pretrained("firdavsus/LLM_multimodal", subfolder="pre-train")
```
---
## ⚠️ Limitations & Intended Use
* **Primary Intended Use:** General text generation, bilingual English/Russian translation, code assistance, and basic mathematical reasoning.
* **Limitations:** Due to its lightweight 0.8B parameter size, complex multi-step reasoning (e.g., MMLU domain knowledge) may show limitations compared to much larger models. Users should implement system prompts or validation guardrails for mission-critical deployments.
```
``` |