DhiGemma v1.0.2

A Dhivehi language model based on Google Gemma 3 4B, fine-tuned specifically for the Dhivehi language (Thaana script).

Model Description

DhiGemma is a fine-tuned version of Gemma 3 4B optimized for Dhivehi language generation. The model was trained on a curated dataset of Dhivehi text covering various domains including news, literature, and conversational data.

Version History

• v1.0.2: Fixed script mixing issue using ORPO training (15k preference pairs)
• v1.0.0: Initial release with known script mixing issues

Training Details

Base Training

• Base model: google/gemma-3-4b-it
• Training data: ~50,000 high-quality Dhivehi text samples
• Training method: Full fine-tuning with LoRA
• Hardware: 8x NVIDIA H100 80GB GPUs

Script Fix (v1.0.2)

• Method: Odds Ratio Preference Optimization (ORPO)
• Dataset: 15,000 preference pairs
• LoRA config: r=32, alpha=64
• Learning rate: 2e-5
• Training time: ~24 minutes on single H100
• Final rewards accuracy: 100%

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "hilarl/DhiGemma",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("hilarl/DhiGemma")

prompt = "ދިވެހި ބަހުން ތައާރަފެއް ލިޔެދީ"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=200,
    temperature=0.7,
    top_p=0.9
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Performance

The model performs well on:

• General Dhivehi text generation
• Question answering in Dhivehi
• Recipe and instructional content
• News-style writing
• Conversational responses

Limitations

• Context length limited to 8192 tokens
• May occasionally produce grammatically imperfect Dhivehi
• Knowledge cutoff: 2025
• Best suited for general text generation rather than specialized domains