nascenia/Gemma3-9B-Instruct-GGUF

Gemma-3-9B-Instruct GGUF

This repository contains GGUF-quantized versions of Gemma-3-9B-Instruct, fine-tuned and adapted by Nascenia for enhanced Bengali language understanding and instruction-following capability.

These GGUF files are optimized for fast inference with llama.cpp and compatible runtimes such as Ollama, LM Studio, and Jan.

---

Training Pipeline

The model was developed through a three-phase training process:

Phase 1 — Continual Pretraining on Bengali Corpus

The base Gemma-3-9B model underwent continual pretraining on a curated Bengali-language dataset comprising:

• NCTB (National Curriculum and Textbook Board) Books: Digitized textbooks covering primary and secondary school curricula in Bangladesh, providing the model with foundational knowledge in standard written Bengali across subjects such as Bangla literature, science, mathematics, and social studies.
• Public Web Data: Cleaned and deduplicated Bengali text crawled from publicly accessible Bangladeshi websites, news portals, blogs, and online encyclopedias — giving the model broad exposure to contemporary colloquial and formal Bengali usage.

This phase significantly improved the model's fluency, vocabulary coverage, and factual grounding in the Bengali language and Bangladeshi cultural context.

Phase 2 — Supervised Instruction Fine-Tuning (SFT)

A high-quality instruction dataset of 140,000+ prompt–response pairs was constructed using outputs from Claude (Anthropic) and GPT-5 (OpenAI) as teacher models. The dataset covers a wide range of tasks including:

• Open-ended question answering in Bengali
• Summarization of Bengali news and documents
• Translation between Bengali and English
• Creative writing, storytelling, and poetry in Bengali
• General instruction-following and multi-turn conversation

The model was then fine-tuned on this dataset using Supervised Fine-Tuning (SFT), enabling it to follow Bengali and bilingual (Bengali–English) instructions effectively while maintaining coherent, helpful, and culturally relevant responses.

Phase 3 — GGUF Conversion for llama.cpp Inference

After fine-tuning, the model weights were converted to the GGUF format and quantized at multiple precision levels using llama.cpp's quantization tools. This enables efficient CPU and GPU inference without requiring full-precision weights, making the model accessible on consumer hardware.

---

Available Quantizations

Filename	Size	Quantization	Notes
Gemma-3-9B-Instruct.IQ4_XS.gguf	6.61 GB	IQ4_XS	Smallest, fastest
Gemma-3-9B-Instruct.Q2_K.gguf	4.77 GB	Q2_K	Extreme compression
Gemma-3-9B-Instruct.Q3_K_S.gguf	5.46 GB	Q3_K_S	Small Q3 variant
Gemma-3-9B-Instruct.Q3_K_M.gguf	6.01 GB	Q3_K_M	Medium Q3 variant
Gemma-3-9B-Instruct.Q3_K_L.gguf	6.48 GB	Q3_K_L	Large Q3 variant
Gemma-3-9B-Instruct.Q4_K_S.gguf	6.94 GB	Q4_K_S	Good quality/size ratio
Gemma-3-9B-Instruct.Q4_K_M.gguf	7.30 GB	Q4_K_M	Recommended balanced quality
Gemma-3-9B-Instruct.Q5_K_S.gguf	8.23 GB	Q5_K_S	High quality
Gemma-3-9B-Instruct.Q5_K_M.gguf	8.45 GB	Q5_K_M	High quality, medium
Gemma-3-9B-Instruct.Q6_K.gguf	9.66 GB	Q6_K	Very high quality

Recommendation: Use Q4_K_M for the best balance of quality and performance on most consumer hardware. Use Q8_0 if you have sufficient RAM and want maximum quality.

---

Usage with llama.cpp

# Clone llama.cpp and build
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && make

# Run inference
./llama-cli \
  -m Gemma-3-9B-Instruct.Q4_K_M.gguf \
  -p "তুমি কে?" \
  -n 256 \
  --chat-template gemma

Usage with Ollama

ollama run hf.co/nascenia/Gemma-3-9B-Instruct-GGUF:Q4_K_M

---

About Nascenia

Nascenia is a software company based in Dhaka, Bangladesh, focused on building AI-powered solutions for Bengali-language applications.

---

License

This model is released under the Apache 2.0 license, consistent with the base Gemma-3 model license terms.