| --- |
| language: en |
| license: apache-2.0 |
| tags: |
| - transformer |
| - sft |
| - instruction-tuned |
| - malawi |
| - small-language-model |
| --- |
| |
| # NIA LLM 110M - SFT (Instruction Fine‑tuned) |
|
|
| This is the **instruction‑fine‑tuned** version of the NIA 110M base model, built from scratch in Malawi 🇲🇼. |
| This model is for research and experiment model |
|
|
| ## Model Details |
|
|
| **Base Model:** |
| - 110M parameters (10 layers, 768 dim, 12 heads) |
| - Pretrained on ~3.2B tokens |
| - Best pretrain loss: 3.3976 |
|
|
| **Fine‑tuning:** |
| - Dataset: Mixed SQuAD (Q&A) + Alpaca (instructions) |
| - Steps: 4,110 |
| - Final loss: **1.5845** |
| - Learning rate: 3e-5 |
|
|
| ## Capabilities |
|
|
| - Follows user instructions (chat format) |
| - Answers some factual questions (improved accuracy) |
| - Temperature sweet spot: 0.65 |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model = AutoModelForCausalLM.from_pretrained("Mathematicaljuice/nia") |
| tokenizer = AutoTokenizer.from_pretrained("Mathematicaljuice/nia") |
| |
| prompt = "Explain photosynthesis in one paragraph." |
| inputs = tokenizer(prompt, return_tensors="pt") |
| output = model.generate(**inputs, max_new_tokens=100, temperature=0.65) |
| print(tokenizer.decode(output[0])) |
| ``` |
|
|
| ## Limitations |
|
|
| - Still hallucinates facts (especially uncommon ones) |
| - Cannot do arithmetic or math reasoning |
| - English only (for now) |
| - Small model size limits knowledge retention |
| - This model is for reseach only |
|
|
| ## Credits |
|
|
| Built by the Malawian ai start up based in blantyre |
| Built by Lance Muyawa |
|
|
| ## License |
|
|
| Apache 2.0 |
