ಕನ್ನಡ Gemma 4 (Kannada Gemma 4)

A Kannada-language-adapted LLM built by continual pre-training (CPT) and supervised fine-tuning (SFT) on Google's Gemma 4 E4B model, following the SambaLingo methodology.

Model Details


Base model	google/gemma-4-E4B
Architecture	Gemma 4 (~8B total params, 4B effective active per token)
Vocab size	262,144 (no extension needed — handles Kannada well)
Token fertility	~1.7x English (vs 5-10x for Llama 2's 32K vocab)
Method	Full-parameter CPT → SFT → (optional) DPO
License	Apache 2.0 (inherited from Gemma)
Languages	Kannada (ಕನ್ನಡ), English

Training Pipeline

┌─────────────────────────────────────────────────────────────┐
│  Stage 1: Continual Pre-Training (CPT)                      │
│  ─────────────────────────────────────────                  │
│  Data: ~36 GB Kannada text + English (3:1 KN:EN ratio)     │
│  Sources: Sangraha, CulturaX-Kn, C4-Kn, IndicCorpV2,       │
│           Wikipedia-Kn, kannada_corpus_1m                   │
│  Hyperparams: lr=1e-4, cosine, wd=0.1, 4 epochs,           │
│               seq_len=4096, packing=True, bf16              │
│  Full-parameter (NOT LoRA) per SambaLingo findings          │
├─────────────────────────────────────────────────────────────┤
│  Stage 2: Supervised Fine-Tuning (SFT)                      │
│  ─────────────────────────────────────                      │
│  Data: Kannada instruction datasets                         │
│  Sources: Kannada_Bilingual_Instruct, indic-align,          │
│           hh_dpo_kannada (chosen responses), Aya_Kannada     │
│  Hyperparams: lr=2e-5, linear, 3 epochs, seq_len=2048,     │
│               assistant_only_loss=True, bf16                 │
├─────────────────────────────────────────────────────────────┤
│  Stage 3: DPO (optional)                                    │
│  ─────────────────────                                       │
│  Data: Cognitive-Lab/hh_dpo_kannada_translated              │
│  Hyperparams: lr=5e-7, β=0.1, linear, 3 epochs              │
└─────────────────────────────────────────────────────────────┘

Datasets Used

CPT (Continual Pre-Training)

Dataset	Size	Link
ai4bharat/sangraha (synthetic, kan_Knda)	17.7 GB	HF
Kannada-LLM-Labs/CulturaX-Kn	3.8 GB	HF
Kannada-LLM-Labs/C4-Kn	2.9 GB	HF
ai4bharat/IndicCorpV2 (kan_Knda)	~5 GB	HF
pavan-naik/kannada_corpus_1m	514 MB	HF
Kannada-LLM-Labs/Wikipedia-Kn	140 MB	HF
HuggingFaceFW/fineweb (English sample)	~10 GB	HF

SFT (Instruction Tuning)

Dataset	Link
Cognitive-Lab/Kannada_Bilingual_Instruct	HF
ai4bharat/indic-align	HF
Cognitive-Lab/hh_dpo_kannada_translated (chosen)	HF

Hardware Requirements

Stage	Minimum GPU	Recommended
CPT (full-param)	1× A100 80GB	1× A100 80GB
CPT (LoRA fallback)	1× RTX 3090/4090 24GB	1× A6000 48GB
SFT (full-param)	1× A100 80GB	1× A100 80GB
SFT (LoRA fallback)	1× RTX 3090/4090 24GB	1× A6000 48GB
Inference	1× RTX 3090/4090 24GB	1× A6000 48GB

How to Run on Your GPU

1. Install dependencies

pip install torch transformers trl accelerate datasets peft sentencepiece
# For accelerated attention (Ampere+ GPUs only: A100, RTX 3090/4090, etc.)
pip install kernels~=0.12.0

2. Prepare CPT data

# Streams all Kannada datasets, mixes with English at 3:1 ratio
# Uses streaming mode — works even with low RAM
python prepare_cpt_data.py --output_dir ./data/cpt_kannada

# Include transliterated Kannada (Latin script) for extra data
python prepare_cpt_data.py --include_latn --output_dir ./data/cpt_kannada

# Adjust English ratio (default 25% = 3:1 KN:EN)
python prepare_cpt_data.py --english_ratio 0.20 --output_dir ./data/cpt_kannada

3. Prepare SFT data

python prepare_sft_data.py --output_dir ./data/sft_kannada

4. Run CPT training

# Full-parameter CPT (recommended — needs A100 80GB)
python train_cpt.py \
  --data_dir ./data/cpt_kannada \
  --output_dir ./checkpoints/cpt \
  --per_device_train_batch_size 4 \
  --gradient_accumulation_steps 8 \
  --learning_rate 1e-4 \
  --num_epochs 4

# LoRA CPT (for 24GB GPUs — lower quality but works)
python train_cpt.py \
  --data_dir ./data/cpt_kannada \
  --output_dir ./checkpoints/cpt \
  --use_lora \
  --per_device_train_batch_size 2 \
  --gradient_accumulation_steps 16

# Resume from checkpoint
python train_cpt.py \
  --data_dir ./data/cpt_kannada \
  --output_dir ./checkpoints/cpt \
  --resume_from_checkpoint ./checkpoints/cpt/checkpoint-5000

5. Run SFT training

# Full-parameter SFT
python train_sft.py \
  --model_path ./checkpoints/cpt \
  --data_dir ./data/sft_kannada \
  --output_dir ./checkpoints/sft \
  --per_device_train_batch_size 8 \
  --gradient_accumulation_steps 4 \
  --learning_rate 2e-5 \
  --num_epochs 3

# LoRA SFT (for 24GB GPUs)
python train_sft.py \
  --model_path ./checkpoints/cpt \
  --data_dir ./data/sft_kannada \
  --output_dir ./checkpoints/sft \
  --use_lora

6. (Optional) Run DPO

python train_dpo.py \
  --model_path ./checkpoints/sft \
  --dataset_id Cognitive-Lab/hh_dpo_kannada_translated \
  --output_dir ./checkpoints/dpo \
  --learning_rate 5e-7 \
  --beta 0.1 \
  --num_epochs 3

7. Push to HuggingFace Hub

# Add --push_to_hub --hub_model_id to any training command
python train_sft.py \
  --model_path ./checkpoints/cpt \
  --data_dir ./data/sft_kannada \
  --output_dir ./checkpoints/sft \
  --push_to_hub --hub_model_id harshaperla/kannada-gemma-4b-sft

8. Inference

# Interactive chat
python inference.py --model_path ./checkpoints/sft

# Evaluate on FLORES translation benchmark
python inference.py --model_path ./checkpoints/sft --eval_flores

# Generate from file
python inference.py --model_path ./checkpoints/sft --input_file prompts.txt

Key Design Decisions

Why not extend the tokenizer?

Gemma 4's 262K vocabulary already handles Kannada well. Our fertility test showed:

Kannada: 0.27–0.41 tokens/char (vs English: 0.17–0.23)
Token fertility ratio: ~1.7x English (vs 5–10x for Llama 2's 32K vocab)
Multi-character Kannada subwords already exist (e.g., ಕನ್ನಡ is a single token)
Conclusion: no tokenizer extension needed

Why full-parameter instead of LoRA?

The SambaLingo paper explicitly showed full-parameter CPT >> LoRA CPT across all 9 languages tested. LoRA is available as a fallback for GPUs with <80GB VRAM.

Why the 3:1 Kannada:English ratio?

SambaLingo found that mixing English data prevents catastrophic forgetting of the model's general capabilities while learning the new language. The 1:3 English:Kannada ratio was optimal.

Why SambaLingo methodology?

SambaLingo (arXiv 2404.05829) achieved SOTA results across 9 languages using this exact pipeline. It's the best-validated language adaptation methodology currently available.

References

SambaLingo: Ristić-Arsova et al., "SambaLingo: Teaching Small Large Language Models New Languages", arXiv:2404.05829
Gemma 4: Google, "Gemma 4 Technical Report"
Sangraha: AI4Bharat, largest Indic language corpus
IndicCorpV2: AI4Bharat, Indic language corpus
CulturaX: USTC, multilingual cleaned corpus

Citation

@misc{kannada-gemma-4b,
  author = {harshaperla},
  title = {Kannada Gemma 4: A Kannada-Language Adapted LLM},
  year = {2026},
  url = {https://huggingface.co/harshaperla/kannada-gemma-4b},
}

Generated by ML Intern

This model repository was generated by ML Intern, an agent for machine learning research and development on the Hugging Face Hub.

Try ML Intern: https://smolagents-ml-intern.hf.space
Source code: https://github.com/huggingface/ml-intern

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = 'harshaperla/kannada-gemma-4b'
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

For non-causal architectures, replace AutoModelForCausalLM with the appropriate AutoModel class.

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Paper for harshaperla/kannada-gemma-4b

SambaLingo: Teaching Large Language Models New Languages

Paper • 2404.05829 • Published Apr 8, 2024 • 13