---
language:
- ga
- en
tags:
- irish
- low-resource
- bilingual
- text-generation
- instruction-following
license: apache-2.0
base_model: Qwen/Qwen3-8B
datasets:
- databricks/dolly-v2
- uonlp/CulturaX
- cis-lmu/Glot500
metrics:
- bleu
- accuracy
---

# Qomhrá: A Bilingual Irish & English LLM

**Qomhrá**, **Q**wen (Base model) + c**omhrá** (Irish for "Conversation") is an 8-billion parameter bilingual Large Language Model (LLM) designed to support the low-resource language of Irish (*Gaeilge*). It is adapted from **Qwen3-8B** via a pipeline of Bilingual Continued Pre-Training (CPT) and Instruction Tuning.

Developed by researchers at **Trinity College Dublin**, **University College Cork**, and **Queen's University Belfast**, Qomhrá aims to foster technological sovereignty for the Irish language community by providing an open-weight alternative to proprietary APIs.

## Model Details

* **Model Name:** Qomhrá-8B-Instruct
* **Developed by:** Joseph McInerney (TCD & QUB), Khanh-Tung Tran (UCC), Liam Lonergan (TCD), Ailbhe Ní Chasaide (TCD), Neasa Ní Chiaráin (TCD), Barry Devereux (QUB).
* **Language(s):** Irish (Gaeilge) and English
* **Base Model:** Qwen/Qwen3-8B
* **License:** Apache 2.0
* **Paper:** TBC

## Training Methodology

The development of Qomhrá followed a two-stage pipeline:

### 1. Bilingual Continued Pre-Training (CPT)
The model was adapted using a bilingual corpus of **3.265 billion characters**. Unlike previous approaches that suffered from catastrophic forgetting, we used a high mixture of English data (approx. 25%) to maintain English language capabilities.

**Data Mixture:**
* **Irish (~75%):**
    * **UCCIX_CulturaX:** 1.2B characters
    * **National Corpus of Irish (CNG):** 549M characters
    * **UCCIX_Glot500:** 530M characters
    * **Other:** UCCIX (Wikipedia, ParaCrawl, ELRC) and The Bible.
* **English (~25%):**
    * **Wikipedia:** 819M characters (2022 dump).

**Training Config:**
* **Compute:** 2x Nvidia H100 (80GB).
* **Context Window:** Packed to 2048 tokens.
* **Precision:** BF16.
* **Optimizer:** AdamW ($lr=1e^{-4}$).

### 2. Instruction Tuning
We curated a **30k sample** parallel English-Irish instruction dataset. This was created by translating the **Dolly V2** dataset using **Gemini-2.5-Pro**, which was selected after a human evaluation ranking it as the top performer for Irish text generation (outperforming GPT-5 and Claude-4-Sonnet).

## Evaluation Results

### Benchmark Definitions
* **Cloze-gle** tests the model's familiarity with Irish grammatical gender, where the model is presented with three sentences that vary by pronoun, and the model must assign the correct gender agreement.
* **SIB-gle** tests topic modelling, the model must ascribe a topic label to text given options such as political, science, or sport.
* **IQA-gle/eng** tests the model's question answering ability in both Irish and English. The model is presented with a user question and some supporting context and it must select the most likely answer.
* **BLEU gle <-> eng** measures the model's bi-directional Irish and English translation accuracy on health domain data (Lankford et al., 2022).
* **NQ-eng** tests the model's world knowledge, requiring an exact match on general knowledge style questions in English.

### Performance

Qomhrá-Instruct outperforms existing open-source baselines on Irish understanding and generation while maintaining strong English capabilities.

| Benchmark | Qomhrá-Instruct | UCCIX | Llama-3.1-8B |
| :--- | :--- | :--- | :--- |
| **Cloze-gle** | **0.88** | 0.75 | 0.59 |
| **SIB-gle** | **0.8186** | 0.7794 | 0.7696 |
| **IQA-gle** | **0.6760** | 0.3889 | 0.4861 |
| **IQA-eng** | **0.7924** | 0.3704 | 0.7747 |
| **BLEU eng2gle** | 0.1167 | **0.3334** | 0.0880 |
| **BLEU gle2eng** | 0.0770 | **0.4636** | 0.4229 |
| **NQ-eng** | 0.1269 | 0.1668 | **0.2767** |

*Note: As discussed in the paper, lower scores on generation benchmarks (BLEU/NQ) for the Instruct model compared to base models are driven by response length distributions; the Instruct model learns to provide concise answers, whereas base models generate longer sequences that artificially inflate overlap metrics.*

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "jmcinern/Qomhra"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto"
)

# Irish Prompt
messages = [
    {"role": "system", "content": "Is cúntóir úsáideach agus dílis tú."},
    {"role": "user", "content": "Cé hé Uachtarán na hÉireann?"}
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    model_inputs.input_ids,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)