README.md

---
library_name: transformers
tags:
- history
- conversational
- PEFT
license: mit
datasets:
- DannyAI/African-History-QA-Dataset
language:
- en
metrics:
- bertscore
base_model:
- microsoft/Phi-4-mini-instruct
pipeline_tag: text-generation
---

# Model Card for Model ID

This is a LoRA fine-tuned version of **microsoft/Phi-4-mini-instruct** for African History using the **DannyAI/African-History-QA-Dataset** dataset.
It achieves a loss value of 1.488960 on the validation set

## Model Details

### Model Description

- **Developed by:** Daniel Ihenacho
- **Funded by:** Daniel Ihenacho
- **Shared by:** Daniel Ihenacho
- **Model type:** Text Generation
- **Language(s) (NLP):** English
- **License:** mit
- **Finetuned from model:** microsoft/Phi-4-mini-instruct

## Uses

This can be used for QA datasets about African History

### Out-of-Scope Use

Can be used beyond African History but should not.

## How to Get Started with the Model

```python
from transformers import pipeline
from transformers import (
    AutoTokenizer, 
    AutoModelForCausalLM)
from peft import PeftModel


model_id = "microsoft/Phi-4-mini-instruct"

tokeniser = AutoTokenizer.from_pretrained(model_id)

# load base model
model  = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map = "auto",
    torch_dtype = torch.bfloat16,
    trust_remote_code = False
)

# Load the fine-tuned LoRA model
lora_id = "DannyAI/phi4_african_history_lora"
lora_model = PeftModel.from_pretrained(
    model,lora_id
)

generator = pipeline(
    "text-generation",
    model=lora_model,
    tokenizer=tokeniser,
)
question = "What is the significance of African feminist scholarly activism in contemporary resistance movements?"
def generate_answer(question)->str:
    """Generates an answer for the given question using the fine-tuned LoRA model.
    """
    messages = [
        {"role": "system", "content": "You are a helpful AI assistant specialised in African history which gives concise answers to questions asked."},
        {"role": "user", "content": question}
    ]
    
    output = generator(
        messages, 
        max_new_tokens=2048, 
        temperature=0.1, 
        do_sample=False,
        return_full_text=False
    )
    return output[0]['generated_text'].strip()
```
```
# Example output
African feminist scholarly activism is significant in contemporary resistance movements as it provides a critical framework for understanding and addressing the specific challenges faced by African women in the context of global capitalism, neocolonialism, and patriarchal structures.
```

## Training Details

### Training Data

| Step | Training Loss | Validation Loss |
|------|--------------|----------------|
| 100  | 1.643300 | 1.649192 |
| 200  | 1.546300 | 1.576022 |
| 300  | 1.580200 | 1.552545 |
| 400  | 1.575900 | 1.538777 |
| 500  | 1.499500 | 1.529112 |
| 600  | 1.400600 | 1.516559 |
| 700  | 1.524000 | 1.513925 |
| 800  | 1.437100 | 1.507401 |
| 900  | 1.547300 | 1.504273 |
| 1000 | 1.441300 | 1.502129 |
| 1100 | 1.452500 | 1.499649 |
| 1200 | 1.466400 | 1.495797 |
| 1300 | 1.407500 | 1.494715 |
| 1400 | 1.511400 | 1.493275 |
| 1500 | 1.489600 | 1.495470 |
| 1600 | 1.384400 | 1.492817 |
| 1700 | 1.534900 | 1.490099 |
| 1800 | 1.469300 | 1.490490 |
| 1900 | 1.407500 | 1.488490 |
| 2000 | 1.512300 | 1.487388 |
| 2100 | 1.438900 | 1.490232 |
| 2200 | 1.434700 | 1.490498 |
| 2300 | 1.421200 | 1.489342 |
| 2400 | 1.418200 | 1.487220 |
| 2500 | 1.441200 | 1.487831 |
| 2600 | 1.453900 | 1.488960 |


### Training Hyperparameters

- per_device_train_batch_size=2
- gradient_accumulation_steps = 4
- learning_rate=2e-5
- num_train_epochs=10
- bf16=True
- eval_strategy="steps"
- eval_steps=100,
- save_strategy="steps"
- save_steps=200
- logging_steps=10

### Lora Configuration
- r: 8
- lora_alpha: 16
- target_modules: ["q_proj", "v_proj", "k_proj", "o_proj"]
- lora_dropout: 0.05 # dataset is small, hence a low dropout value
- bias: "none"
- task_type: "CAUSAL_LM"


## Evaluation

#### Metrics
| Models | Bert Score | TinyMMLU| TinyTrufulQA
|------|--------------|----------------|----------------|
| Base model | 0.88868 | 0.6837 |0.49745|
| Fine tuned Model | 0.90726 | 0.67751 |0.43555|


## Compute Infrastructure

[Runpod](https://console.runpod.io/).

### Hardware

Runpod A40 GPU instance

## Citation

If you use this dataset, please cite:
```
@Model{
Ihenacho2026phi4_african_history_lora,
  author    = {Daniel Ihenacho},
  title     = {phi4_african_history_lora},
  year      = {2026},
  publisher = {Hugging Face Models},
  url       = {https://huggingface.co/DannyAI/phi4_african_history_lora},
  urldate   = {2026-01-27},
}
```

## Model Card Authors

Daniel Ihenacho

## Model Card Contact

- [LinkedIn](https://www.linkedin.com/in/daniel-ihenacho-637467223)
- [GitHub](https://github.com/daniau23)