|
|
--- |
|
|
license: apache-2.0 |
|
|
base_model: gpt2 |
|
|
tags: |
|
|
- trl |
|
|
- sft |
|
|
- lora |
|
|
- nebulos |
|
|
- skull |
|
|
- multilingual |
|
|
datasets: |
|
|
- HuggingFaceFW/fineweb-edu |
|
|
- HuggingFaceFW/finewiki |
|
|
- HuggingFaceFW/fineweb-2 |
|
|
language: |
|
|
- en |
|
|
- es |
|
|
- de |
|
|
- fr |
|
|
- pt |
|
|
metrics: |
|
|
- accuracy |
|
|
--- |
|
|
|
|
|
# 💀 SkullLLM-125M |
|
|
|
|
|
**SkullLLM-125M** is a lightweight, experimental multilingual language model fine-tuned from GPT-2. This project, part of the **SkullLLM** series, demonstrates that AI training is possible on highly constrained consumer hardware (3GB VRAM) using advanced optimization techniques. |
|
|
|
|
|
### 🚀 Model Details |
|
|
- **Developed by:** Erik22TY |
|
|
- **Model Name:** Nebulos (SkullLLM-125M) |
|
|
- **Base Model:** GPT-2 (125M parameters) |
|
|
- **Training OS:** Linux Mint |
|
|
- **Training Hardware:** HP Pavilion Gaming Desktop 690-00xx |
|
|
- **GPU:** NVIDIA GeForce GTX 1050 (3GB VRAM - Pascal Architecture) |
|
|
- **Training Type:** LoRA (Low-Rank Adaptation) |
|
|
- **Format:** ChatML (`<|im_start|>user`, `<|im_start|>assistant`) |
|
|
|
|
|
### 🖥️ Hardware Requirements |
|
|
This model is optimized for low-end hardware. |
|
|
- **VRAM for Inference:** ~1.5 GB (4-bit) / ~2.2 GB (FP16). |
|
|
- **VRAM for Training:** 2.8 GB+ (Tested on GTX 1050 3GB). |
|
|
- **System RAM:** 4 GB minimum for inference; 12 GB recommended for training. |
|
|
- **Storage:** ~150 MB for the adapter files. |
|
|
|
|
|
|
|
|
|
|
|
### 🧠 Knowledge & Dataset |
|
|
Nebulos was trained on a high-quality multilingual stream: |
|
|
- **English (FineWeb-Edu):** Knowledge cutoff March 2024. |
|
|
- **Multilingual (FineWeb-2):** Spanish, German, French, and Portuguese web data. |
|
|
- **General (FineWiki):** Wikipedia-based knowledge updated through August 2025. |
|
|
|
|
|
### 🧪 Training Configuration |
|
|
- **Steps:** 500 |
|
|
- **Batch Size:** 1 (Gradient Accumulation: 16) |
|
|
- **Optimization:** 4-bit Quantization (NF4) |
|
|
- **Compute Dtype:** Forced FP16 (to support Pascal architecture) |
|
|
- **Learning Rate:** 2e-4 |
|
|
- **Final Loss:** 4.0898 |
|
|
|
|
|
### ⚠️ Limitations & Behavior |
|
|
As a 125M parameter model trained for 500 steps, SkullLLM-125M is a **Proof of Concept**. |
|
|
- **Repetitions:** May occasionally loop phrases (e.g., "metic"). Use `repetition_penalty=1.5`. |
|
|
- **Language Blending:** Due to its size, it may mix Romance languages (Spanish/French/Portuguese) in complex responses. |
|
|
- **Coherence:** Best used for short-form explanations or creative experiments. |
|
|
|
|
|
### 💬 Usage (Python) |
|
|
```python |
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
from peft import PeftModel |
|
|
import torch |
|
|
|
|
|
model_id = "gpt2" |
|
|
adapter_id = "Erik22TY/SkullLLM-125M" |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(adapter_id) |
|
|
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto") |
|
|
model.resize_token_embeddings(len(tokenizer)) |
|
|
model = PeftModel.from_pretrained(model, adapter_id) |
