AmplifiedAccess/Swahili-gemma-1b-it

Swahili Gemma 1B IT

A fine-tuned version of Google Gemma 3 1B, continued-pretrained and instruction-tuned for English → Swahili translation and general Swahili instruction-following.

Also available as a quantized Ollama release (ollama run amplifiedaccessorg/Swahili-gemma-1b-it).

Highlights

• BLEU 28.86 on English→Swahili FLORES-200 translation — up from 0.41 on the base model
• chrF++ 54.77 — up from 14.28 on the base model
• 1B parameters — small enough to run on consumer hardware
• Trained with QLoRA (4-bit NF4 base + LoRA adapters), then merged to full weights

Results

FLORES-200 devtest, English→Swahili, beam search (beam=8), 256 max new tokens:

Model	BLEU	chrF++
Gemma 3 1B base (no fine-tuning)	0.41	14.28
This model	28.86	54.77

Fine-tuning took the base model from essentially no working Swahili translation ability to a functional, fluent translator on both metrics.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "AmplifiedAccess/Swahili-gemma-1b-it"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.float16, device_map="auto")

prompt = ("<start_of_turn>user\n"
          "Translate the following English sentence to Swahili.\n"
          "English: The children are playing outside.\nSwahili:<end_of_turn>\n"
          "<start_of_turn>model\n")

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=128, do_sample=False)

response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(response.strip())

Recommended prompt format

The model was trained on this exact translation template — using it gets the most reliable output:

Translate the following English sentence to Swahili.
English: <your sentence>
Swahili:

It was also trained on general Swahili instruction data (Aya, Alpaca-Swahili), so it can follow other Swahili-language instructions, but translation quality is best with the format above.

Example translations (tested against the deployed model)

English	Model output
The children are playing outside.	Watoto wanacheza nje.
I would like a cup of coffee, please.	Ningependa kikombe cha kahawa tafadhali.
The government announced new policies to support small businesses across the country.	Serikali ilitangaza sera mpya kusaidia biashara ndogo nchi nzima.

Training details

Pipeline

Two-stage training on Kaggle (T4 GPU):

1. Stage 1 — Continued pretraining on Swahili data, producing an intermediate CPT-merged checkpoint (base for this stage).
2. Stage 2 — Supervised fine-tuning (this release): QLoRA on top of the CPT-merged checkpoint, then merged to full weights.

Data (Stage 2 SFT)

Source	Examples	Notes
MAFAND-MT (en-swa)	32,414	Human-translated news domain (Masakhane)
Rogendo/English-Swahili-Sentence-Pairs	58,052
Sunbird SALT (text-all)	23,947	Full/uncapped
Svngoku/Inkuba-Swahili-MMT	29,991	Capped at 30k, normalized to EN→SW direction
CohereForAI/aya_dataset (Swahili)	359	Human instruction data
iamshnoo/alpaca-cleaned-swahili	14,844	Capped at 15k
Back-translation (NLLB, planned)	0	Did not run — monolingual source returned 0 usable lines in this training run
Total (post quality filter)	159,311	296 pairs dropped by a length-ratio sanity filter

Packed into 25,452 blocks of 512 tokens (~13.0M training tokens).

Configuration

Parameter	Value
Base model	google/gemma-3-1b-it (via CPT-merged checkpoint)
Method	QLoRA (4-bit NF4 base + LoRA), merged to full weights after training
LoRA rank / alpha / dropout	32 / 64 / 0.05
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Trainable parameters	26.09M (2.54% of 1.03B total)
Epochs	3
Effective batch size	32 (2 per device × 16 gradient accumulation)
Learning rate	2e-4 (cosine schedule, 3% warmup)
Max sequence length	512 (packed)
Precision	fp16, SDPA attention
Optimizer	paged AdamW 8-bit
Training steps	2,388
Training time	~9h 45m on a single Kaggle T4 GPU
Final training loss	1.543

Release format

This repo contains the merged fp16 weights. A quantized GGUF (q4_K_M, ~814 MB) is available via Ollama.

Limitations

• Back-translation augmentation was planned but did not execute in this training run — the model did not benefit from that data.
• Trained and evaluated only on English→Swahili translation; Swahili→English capability is untested and not claimed.
• Performance outside the training domains (news, general instructions) is unverified.
• As a 1B model, expect more errors on long, syntactically complex, or highly domain-specific sentences than from larger translation models.

Intended use

• English → Swahili machine translation
• Swahili instruction-following / conversational use
• Research on low-resource African language NLP at small parameter scale

Acknowledgments

• Masakhane for the MAFAND-MT parallel corpus
• Sunbird AI for the SALT dataset
• Cohere For AI for the Aya dataset
• Google DeepMind for the Gemma model family

Framework versions

• PyTorch 2.10.0
• Transformers 5.0.0
• PEFT (LoRA)

Citation

@misc{swahili-gemma-1b-2026,
  title={Swahili Gemma 1B IT: Fine-tuned Gemma 3 1B for English-Swahili Translation},
  author={Amplified Access},
  year={2026},
  publisher={HuggingFace},
  url={https://huggingface.co/AmplifiedAccess/Swahili-gemma-1b-it}
}