Caracal_instruct (Pre-release)

Model Description

Caracal_instruct is an instruction-tuned model, produced as part of the AfriLLMQuant pilot project. It was trained via full Quantization-Aware Training (QAT).

The underlying small language model (SLM) backbone for this QAT process was Inkuba-0.4B, continued-pretrained on African-language data and then instruction-tuned to produce Caracal_instruct. The recommended use of this model is for fine-tuning on specific task.

Base model	theophilusowiti/Caracal_GPT
Training method	Full Quantization-Aware Training (QAT)
Quantization	INT4
Memory footprint (INT8)	~1.17 GB
QAT training time	~3 days on 1x NVIDIA A100 80GB
License	Apache 2.0
Training Dataset	Muri Dataset

---

Authors & Acknowledgments

Author

• Theophilus Lincoln Owiti — Carnegie Mellon University Africa

Acknowledgments

We gratefully acknowledge OpenToken and their infrastructure partner Leafcloud for providing the GPU compute resources that made training this model possible.

---

Citation

If you use this model or its outputs in your research, please cite:

@misc{owiti2026caracal,
  title        = {Caracal Instruct},
  author       = {Owiti, Theophilus Lincoln},
  year         = {2026},
  publisher    = {Hugging Face},
  journal      = {Hugging Face Repository},
  howpublished = {\url{[https://huggingface.co/theophilusowiti/Caracal_instruct](https://huggingface.co/theophilusowiti/Caracal_instruct)}},
  institution  = {Carnegie Mellon University Africa},
  note         = {GPU compute provided by OpenToken and Leafcloud}
}

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "theophilusowiti/Caracal_instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

prompt = "<s><Input>\nWho is the president of Kenya?\n</Input>\n<Answer>\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Expected output

The model expects the <Input>...</Input> / <Answer>...</Answer> instruction format used during SFT, e.g.:

<Input>
Who is the president of Kenya?
</Input>
<Answer>
 William Ruto

William Ruto</Answer>

---

Top Performing Languages

Based on instruction following and PPL, these languages conform to instruction, especially when used with context/RAG:

East Africa: Swahili (swa), Amharic (amh), Luganda (lug), Kinyarwanda (kin)

West Africa: Hausa (hau), Yoruba (yor), Igbo (ibo)

Central Africa: Lingala (lin)

Southern Africa: Xhosa (xho)

Training Procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Adam (betas=(0.9, 0.999), epsilon=1e-08)
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.03
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
3.1297	0.9999	6,036	2.6244
2.7599	2.0	12,073	2.6072
2.7805	2.9998	18,108	2.6143

Framework versions

• Transformers 4.45.0
• Pytorch 2.12.0+cu126
• Datasets 4.8.5
• Tokenizers 0.20.3

---