Towards fine-tune perfection
For experimental purposes, I would like to fine-tune the gemma3-270-it model. I believe the model is a good basis for performing tasks in Hungarian. The base model struggles with the Hungarian language, but after relatively short fine-tuning, it learns the language well. For now, I am training it to perform extractive question-answering tasks (quasi-RAG) from a given context using the LoRA method. My question is whether there was a special system prompt for this task during the instruction-following training of the model? Also, is the instruct dataset itself available anywhere? (This would be very useful for avoiding catastrophic forgetting and for alignment purposes.)
Thanks.
Hey @GaborMadarasz ,
It is great to see you adapting our Gemma-3-270m-it model for Hungarian extractive QA via LORA. To answer your first question: There was no special or hidden system prompt used during instruction tuning. As described in our official Gemma formatting and system instructions documentation, the architecture supports two roles: user and model. Any high-level instructions are included within the user turn. For fine-tuning, you can try to mirror this format by placing task instructions and context directly inside the <start_of_turn>user ... <end_of_turn> block.
Regarding your second question, the specific instruction-tuning dataset used for Gemma 3 family is not publicly released. If you are concerned about catastrophic forgetting, a common approach is to mix your task-specific data with general instruction-following data to help preserve alignment and instruction adherence during fine-tuning.
Thank you!
Thank you for your reply, @srikanta-221
I will train the models based on this, and I hope that the Hungarian adaptations of the gemma3-270 variants will arrive soon.