|
|
--- |
|
|
license: apache-2.0 |
|
|
language: |
|
|
- en |
|
|
--- |
|
|
# 🧠 Fine-Tuned Nano LLM |
|
|
This repository contains a fine-tuned version of a small LLM trained using **LoRA / QLoRA**. |
|
|
|
|
|
--- |
|
|
|
|
|
## 📌 Model Overview |
|
|
- **Base Model:** Nano LLM |
|
|
- **Fine-tuning Method:** LoRA / QLoRA |
|
|
- **Dataset:** Custom synthetic + curated dataset |
|
|
- **Task:** Text generation |
|
|
- **Framework:** PyTorch + Hugging Face Transformers |
|
|
- **Training Environment:** Google Colab Free Tier |
|
|
|
|
|
--- |
|
|
|
|
|
## 🧪 Training Details |
|
|
- Mixed precision (fp16 / nf4) |
|
|
- Gradient checkpointing enabled |
|
|
- LoRA rank & parameters carefully optimized for small hardware |
|
|
- Trained on DatasetDict format with proper splits |
|
|
|
|
|
--- |
|
|
|
|
|
## 📂 Files Included |
|
|
This repo contains: |
|
|
- `config.json` |
|
|
- `adapter_config.json` |
|
|
- `adapter_model.bin` |
|
|
- `tokenizer.json` |
|
|
- `tokenizer.model` |
|
|
- `special_tokens_map.json` |
|
|
- `generation_config.json` |
|
|
- `training_args.bin` |
|
|
|
|
|
--- |
|
|
|
|
|
## How to load model |
|
|
|
|
|
from modeling_tinygpt import TinyGPT |
|
|
import torch |
|
|
import json |
|
|
|
|
|
with open("config.json") as f: |
|
|
cfg = json.load(f) |
|
|
|
|
|
model = TinyGPT( |
|
|
vocab_size=cfg["vocab_size"], |
|
|
d_model=cfg["d_model"], |
|
|
n_heads=cfg["n_heads"], |
|
|
n_layers=cfg["n_layers"], |
|
|
d_ff=cfg["d_ff"], |
|
|
max_seq_len=cfg["max_seq_len"] |
|
|
) |
|
|
|
|
|
state = torch.load("pytorch_model.bin", map_location="cpu") |
|
|
model.load_state_dict(state) |
|
|
model.eval() |
|
|
|
|
|
--- |
|
|
|
|
|
## 📈 Benchmark & Notes |
|
|
This model is designed for experimentation, education, and small-scale inference. |
|
|
This model uses a custom architecture. Please load using `trust_remote_code=True`. |
|
|
|
|
|
--- |
