| --- |
| language: |
| - ro |
| license: cc-by-nc-4.0 |
| library_name: transformers |
| tags: |
| - gemma3 |
| - gemma |
| - romanian |
| - vlm |
| - instruct |
| - multimodal |
| datasets: |
| - OpenLLM-Ro/ro_sft_laion |
| - OpenLLM-Ro/ro_sft_pixmo_cap |
| - OpenLLM-Ro/ro_sft_flickr30k_cap |
| - OpenLLM-Ro/ro_sft_llava_mix |
| - OpenLLM-Ro/ro_sft_pixmo_aa |
| - OpenLLM-Ro/ro_sft_pixmo_cap_qa |
| - OpenLLM-Ro/ro_sft_flickr30k_qa |
| - OpenLLM-Ro/ro_sft_cosyn |
| - OpenLLM-Ro/ro_sft_finepdfs |
| - OpenLLM-Ro/ro_sft_pixmo_points |
| - OpenLLM-Ro/ro_sft_pixmo_count |
| base_model: |
| - google/gemma-3-4b-it |
| model-index: |
| - name: OpenLLM-Ro/RoGemma3-4B-Instruct |
| results: |
| - task: |
| type: image-text-to-text |
| dataset: |
| name: Romanian_VLM_Benchmarks |
| type: Romanian_VLM_Benchmarks |
| metrics: |
| - name: Micro avg. |
| type: Score |
| value: 59.54 |
| - name: Macro avg. |
| type: Score |
| value: 57.14 |
| - task: |
| type: image-text-to-text |
| dataset: |
| name: MMBench |
| type: MMBench |
| metrics: |
| - name: Accuracy |
| type: accuracy |
| value: 69.96 |
| - task: |
| type: image-text-to-text |
| dataset: |
| name: MMStar |
| type: MMStar |
| metrics: |
| - name: Accuracy |
| type: accuracy |
| value: 46.01 |
| - task: |
| type: image-text-to-text |
| dataset: |
| name: SeedBench2 |
| type: SeedBench2 |
| metrics: |
| - name: Accuracy |
| type: accuracy |
| value: 62.83 |
| - task: |
| type: image-text-to-text |
| dataset: |
| name: MMMU |
| type: MMMU |
| metrics: |
| - name: Accuracy |
| type: accuracy |
| value: 38.67 |
| - task: |
| type: image-text-to-text |
| dataset: |
| name: MME |
| type: MME |
| metrics: |
| - name: Accuracy |
| type: accuracy |
| value: 54.62 |
| - task: |
| type: image-text-to-text |
| dataset: |
| name: CVQA |
| type: CVQA |
| metrics: |
| - name: Accuracy |
| type: accuracy |
| value: 64.24 |
| - task: |
| type: image-text-to-text |
| dataset: |
| name: ALM-Bench |
| type: ALM-Bench |
| metrics: |
| - name: Score |
| type: score |
| value: 65.40 |
| - task: |
| type: image-text-to-text |
| dataset: |
| name: RoMemes |
| type: RoMemes |
| metrics: |
| - name: F1 |
| type: f1 |
| value: 40.78 |
| - task: |
| type: image-text-to-text |
| dataset: |
| name: RoCultVLM |
| type: RoCultVLM |
| metrics: |
| - name: Score |
| type: score |
| value: 52.84 |
| - task: |
| type: image-text-to-text |
| dataset: |
| name: RoFlickr30k-Caption |
| type: RoFlickr30k-Caption |
| metrics: |
| - name: BERTScore |
| type: bertscore |
| value: 84.35 |
| - task: |
| type: image-text-to-text |
| dataset: |
| name: RoFlickr30k-QA |
| type: RoFlickr30k-QA |
| metrics: |
| - name: Score |
| type: score |
| value: 84.74 |
| - task: |
| type: image-text-to-text |
| dataset: |
| name: LLaVA-Wild |
| type: LLaVA-Wild |
| metrics: |
| - name: Score |
| type: score |
| value: 55.71 |
| - task: |
| type: image-text-to-text |
| dataset: |
| name: AyaVisionBench |
| type: AyaVisionBench |
| metrics: |
| - name: Score |
| type: score |
| value: 47.04 |
| - task: |
| type: image-text-to-text |
| dataset: |
| name: m-WildVision |
| type: m-WildVision |
| metrics: |
| - name: Score |
| type: score |
| value: 57.60 |
| - task: |
| type: image-text-to-text |
| dataset: |
| name: RoCosyn |
| type: RoCosyn |
| metrics: |
| - name: Score |
| type: score |
| value: 59.06 |
| - task: |
| type: image-text-to-text |
| dataset: |
| name: RoFinepdfs |
| type: RoFinepdfs |
| metrics: |
| - name: ANLS |
| type: anls |
| value: 84.35 |
| - task: |
| type: image-text-to-text |
| dataset: |
| name: RoMemes OCR |
| type: RoMemes-OCR |
| metrics: |
| - name: ANLS |
| type: anls |
| value: 86.33 |
| - task: |
| type: image-text-to-text |
| dataset: |
| name: PixmoCount |
| type: PixmoCount |
| metrics: |
| - name: Exact match |
| type: exact_match |
| value: 51.80 |
| - task: |
| type: image-text-to-text |
| dataset: |
| name: PixmoPoints |
| type: PixmoPoints |
| metrics: |
| - name: F1 |
| type: f1 |
| value: 24.87 |
| --- |
| |
| # Model Card for RoGemma3-4B-Instruct |
|
|
| RoGemma3-4B-Instruct is a Romanian-adapted vision-language model built on top of |
| [google/gemma-3-4b-it](https://huggingface.co/google/gemma-3-4b-it). It was produced by continued supervised instruction |
| tuning of the base Gemma 3 checkpoint on a Romanian multimodal SFT |
| mixture covering general instruction following (LLaVA mix), captioning |
| (Pixmo-Cap, Flickr30k-Cap), visual question answering (Pixmo-AA, |
| Pixmo-Cap-QA, Flickr30k-QA), document and chart understanding (CoSyn, |
| FinePDFs), and visual grounding (Pixmo-Points, Pixmo-Count). The model |
| is intended for research on Romanian VLM capabilities. |
|
|
| ## Model Details |
|
|
| ### Model Description |
|
|
| - **Developed by:** OpenLLM-Ro |
| - **Language(s):** Romanian |
| - **License:** cc-by-nc-4.0 |
| - **Finetuned from model:** [google/gemma-3-4b-it](https://huggingface.co/google/gemma-3-4b-it) |
| - **Trained using:** |
| - [OpenLLM-Ro/ro_sft_laion](https://huggingface.co/datasets/OpenLLM-Ro/ro_sft_laion) |
| - [OpenLLM-Ro/ro_sft_pixmo_cap](https://huggingface.co/datasets/OpenLLM-Ro/ro_sft_pixmo_cap) |
| - [OpenLLM-Ro/ro_sft_flickr30k_cap](https://huggingface.co/datasets/OpenLLM-Ro/ro_sft_flickr30k_cap) |
| - [OpenLLM-Ro/ro_sft_llava_mix](https://huggingface.co/datasets/OpenLLM-Ro/ro_sft_llava_mix) |
| - [OpenLLM-Ro/ro_sft_pixmo_aa](https://huggingface.co/datasets/OpenLLM-Ro/ro_sft_pixmo_aa) |
| - [OpenLLM-Ro/ro_sft_pixmo_cap_qa](https://huggingface.co/datasets/OpenLLM-Ro/ro_sft_pixmo_cap_qa) |
| - [OpenLLM-Ro/ro_sft_flickr30k_qa](https://huggingface.co/datasets/OpenLLM-Ro/ro_sft_flickr30k_qa) |
| - [OpenLLM-Ro/ro_sft_cosyn](https://huggingface.co/datasets/OpenLLM-Ro/ro_sft_cosyn) |
| - [OpenLLM-Ro/ro_sft_finepdfs](https://huggingface.co/datasets/OpenLLM-Ro/ro_sft_finepdfs) |
| - [OpenLLM-Ro/ro_sft_pixmo_points](https://huggingface.co/datasets/OpenLLM-Ro/ro_sft_pixmo_points) |
| - [OpenLLM-Ro/ro_sft_pixmo_count](https://huggingface.co/datasets/OpenLLM-Ro/ro_sft_pixmo_count) |
|
|
| ### Model Sources |
|
|
| - **Repository:** https://github.com/OpenLLM-Ro/LLaMA-Factory |
| - **Paper:** https://arxiv.org/abs/2605.31401 |
|
|
| ## Intended Use |
|
|
| ### Intended Use Cases |
|
|
| RoGemma3-4B-Instruct is intended for research use on Romanian vision-language tasks — |
| captioning, visual question answering, cultural understanding, OCR / |
| document understanding, and visual grounding — and as a starting point for |
| further Romanian VLM adaptation. |
|
|
| ### Out-of-Scope Use |
|
|
| Use in any manner that violates applicable laws or regulations (including |
| trade-compliance laws), the project's license, or use in languages other |
| than Romanian. |
|
|
| ## How to Get Started with the Model |
|
|
| ```python |
| import torch |
| from PIL import Image |
| from transformers import AutoProcessor, Gemma3ForConditionalGeneration |
| |
| model = Gemma3ForConditionalGeneration.from_pretrained( |
| "OpenLLM-Ro/RoGemma3-4B-Instruct", |
| torch_dtype=torch.bfloat16, |
| device_map="auto", |
| ).eval() |
| processor = AutoProcessor.from_pretrained("OpenLLM-Ro/RoGemma3-4B-Instruct") |
| |
| image = Image.open("example.jpg").convert("RGB") |
| question = "Descrie imaginea în detaliu." |
| |
| messages = [ |
| {"role": "user", "content": [ |
| {"type": "image", "image": image}, |
| {"type": "text", "text": question}, |
| ]}, |
| ] |
| inputs = processor.apply_chat_template( |
| messages, |
| add_generation_prompt=True, |
| tokenize=True, |
| return_dict=True, |
| return_tensors="pt", |
| ).to(model.device, dtype=torch.bfloat16) |
| |
| with torch.inference_mode(): |
| outputs = model.generate(**inputs, max_new_tokens=256, do_sample=False) |
| print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)) |
| ``` |
|
|
| ## Benchmarks |
|
|
| All benchmarks below are evaluated in Romanian. Per-benchmark winners are |
| shown in **bold**. *Micro* is the mean over individual benchmarks; *Macro* |
| is the mean over capability groups. |
|
|
| ### Aggregate |
|
|
| | Model | Micro avg. | Macro avg. | |
| |---|---|---| |
| | Gemma3-4B-it | 55.53 | 52.36 | |
| | RoGemma3-4B-Instruct | **59.54** | **57.14** | |
|
|
| ### General Understanding |
|
|
| | Model | MMBench | MMStar | SeedBench2 | |
| |---|---|---|---| |
| | Gemma3-4B-it | 59.13 | 41.49 | 57.55 | |
| | RoGemma3-4B-Instruct | **69.96** | **46.01** | **62.83** | |
|
|
| ### Knowledge & Reasoning |
|
|
| | Model | MMMU | MME | |
| |---|---|---| |
| | Gemma3-4B-it | 36.67 | **57.32** | |
| | RoGemma3-4B-Instruct | **38.67** | 54.62 | |
|
|
| ### Cultural |
|
|
| | Model | CVQA | ALM-Bench | RoMemes | RoCultVLM | |
| |---|---|---|---|---| |
| | Gemma3-4B-it | **64.90** | **65.97** | **43.24** | 52.19 | |
| | RoGemma3-4B-Instruct | 64.24 | 65.40 | 40.78 | **52.84** | |
|
|
| ### Generation & Open-ended |
|
|
| | Model | RoFlickr30k-Caption | RoFlickr30k-QA | LLaVA-Wild | AyaVisionBench | m-WildVision | |
| |---|---|---|---|---|---| |
| | Gemma3-4B-it | 70.93 | 81.66 | 54.84 | **52.81** | **60.80** | |
| | RoGemma3-4B-Instruct | **84.35** | **84.74** | **55.71** | 47.04 | 57.60 | |
|
|
| ### OCR & Documents |
|
|
| | Model | RoCosyn | RoFinepdfs | RoMemes OCR | |
| |---|---|---|---| |
| | Gemma3-4B-it | 48.40 | 67.12 | **89.47** | |
| | RoGemma3-4B-Instruct | **59.06** | **84.35** | 86.33 | |
|
|
| ### Grounding |
|
|
| | Model | PixmoCount | PixmoPoints | |
| |---|---|---| |
| | Gemma3-4B-it | 40.42 | 10.21 | |
| | RoGemma3-4B-Instruct | **51.80** | **24.87** | |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{masala2026intelegi, |
| title={``\^{I}n\c{t}elegi Rom\^{a}ne\c{s}te?'' A Recipe for Romanian Vision-Language Models}, |
| author={Mihai Masala and Marius Leordeanu and Mihai Dascalu and Traian Rebedea}, |
| year={2026}, |
| eprint={2605.31401}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CL}, |
| url={https://arxiv.org/abs/2605.31401}, |
| } |
| |
| @inproceedings{masala-etal-2024-vorbesti, |
| title = "``Vorbeşti Româneşte?'' A Recipe to Train Powerful {R}omanian {LLM}s with {E}nglish Instructions", |
| author = "Masala, Mihai and Ilie-Ablachim, Denis and Dima, Alexandru and Corlatescu, Dragos and Zavelca, Miruna and Olaru, Ovio and Terian, Simina and Terian, Andrei and Leordeanu, Marius and Velicu, Horia and Popescu, Marius and Dascalu, Mihai and Rebedea, Traian", |
| booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024", |
| month = nov, |
| year = "2024", |
| pages = "11632--11647" |
| } |
| ``` |
|
|
