---
language: en
license: apache-2.0
tags:
- transformer
- sft
- instruction-tuned
- malawi
- small-language-model
---

# NIA LLM 110M - SFT (Instruction Fine‑tuned)

This is the **instruction‑fine‑tuned** version of the NIA 110M base model, built from scratch in Malawi 🇲🇼.
This model is for research and experiment model

## Model Details

**Base Model:**
- 110M parameters (10 layers, 768 dim, 12 heads)
- Pretrained on ~3.2B tokens
- Best pretrain loss: 3.3976

**Fine‑tuning:**
- Dataset: Mixed SQuAD (Q&A) + Alpaca (instructions)
- Steps: 4,110
- Final loss: **1.5845**
- Learning rate: 3e-5

## Capabilities

- Follows user instructions (chat format)
- Answers some  factual questions (improved accuracy)
- Temperature sweet spot: 0.65

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Mathematicaljuice/nia")
tokenizer = AutoTokenizer.from_pretrained("Mathematicaljuice/nia")

prompt = "Explain photosynthesis in one paragraph."
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=100, temperature=0.65)
print(tokenizer.decode(output[0]))
```

## Limitations

- Still hallucinates facts (especially uncommon ones)
- Cannot do arithmetic or math reasoning
- English only (for now)
- Small model size limits knowledge retention
- This model is for reseach only

## Credits

Built by the Malawian ai start up based in blantyre
Built by Lance Muyawa

## License

Apache 2.0