Romance-MoLE: Rich Cousins and Their Benefits

Romance-MoLE is an 8-billion-parameter polyglot routing framework designed to adapt large language models to under-resourced languages—specifically the endangered Languedocien dialect of Occitan—without experiencing catastrophic forgetting or structural "translationese".

The model relies on targeted cross-lingual transfer from Occitan's phylogenetic "rich cousins" (French and Catalan) using Parameter-Efficient Fine-Tuning (PEFT), vocabulary expansion, and a dynamic hierarchical gating architecture.

• Developed by: Mihaela Goga (University of Gothenburg)
• Base Architecture: Llama-3.1-Carballo-8B
• Repository Link: GitHub Code Framework

---

Repository Artifact Layout

The repository weights are organized into independent modular subdirectories to optimize inference and tracking:

velociraptorela-208/Romance-MoLE/
├── base_model/               # Custom trans-tokenised base (129.6k vocabulary embeddings)
├── router/                   # Trainable linear HMoRA weights (.pt) and config (.json)
├── adapters/
│   ├── lora_fr/              # Gallo-Romance French specialized instruction expert
│   ├── lora_ca/              # Ibero-Romance Catalan specialized instruction expert
│   └── lora_oc/              # Aligned Languedocien Occitan curriculum expert
└── baselines/
    └── simple_model/         # Standard single-stage HPLT fine-tuned baseline

---

Technical Architectural Overview

The framework counteracts parameter interference across Romance varieties through a 4-tiered optimization pipeline:

1. Trans-Tokenisation: Aligns and expands the stock tokenizer with 129,600 padded, token-mined Occitan entries. New embedding vectors are initialized via probabilistic semantic cognate averages drawn from parent French and Catalan tables.
2. Rich Cousin LoRA Extraction: Attention mechanism and language modeling head projection matrices ($r=64$, $\alpha=128$) are fine-tuned independently to isolate French and Catalan structural spaces cleanly.
3. Phylogenetic TIES-Merging: Merges independent parent updates mathematically under consensus sign election and a strict 20% density threshold to produce a highly stable initial baseline (adapter_oc_init).
4. Hierarchical Mixture of Ranked Adapters (HMoRA): Integrates all three specialized modules via a deep gating layers matrix. Early network layers (0–7) perform fine-grained token-level routing for morphological precision, while deep layers (8+) drop back to sequence-level mean-pooling to systematically block mid-sentence code-switching.

---

Empirical Evaluation Results

1. Translation Quality & Probabilistic Alignment (chrF++)

Evaluated on held-out FLORES-200 parallel translation sets comparing the complete curriculum pipeline against a raw single-stage baseline:

Source Target	Simple Baseline	Full Pipeline Pipeline	MoLE Final Frame	Net Generation Swap
French $\rightarrow$ Occitan	30.88	45.45	44.70	+14.57 (Pipeline vs Simple)
Catalan $\rightarrow$ Occitan	29.76	41.34	43.75	+11.58 (Pipeline vs Simple)

2. Multi-Directional Polyglot Profile (chrF++)

The complete 6-direction translation matrix confirms that the Romance-MoLE framework successfully safeguards Occitan capabilities while recovering performance across higher-resource categories:

Evaluated Target Vector	MoLE Gated Final	Single Expert Ceiling	Simple Floor Baseline
French $\rightarrow$ Occitan	44.70	45.45	30.88
Catalan $\rightarrow$ Occitan	43.75	41.34	29.76
Occitan $\rightarrow$ French	58.06	41.43	19.63
Catalan $\rightarrow$ French	55.02	37.63	19.74
Occitan $\rightarrow$ Catalan	46.87	38.27	25.24
French $\rightarrow$ Catalan	34.02	37.28	26.47

3. Fine-Grained Grammatical Preference Accuracy

Minimal-pair morphosyntactic challenge accuracy evaluating orthographic, morphological, and dialectal shibboleth boundaries:

• Romance-MoLE ($\tau=0.25$): 81.1% ($60/74$ correct pairs)
• Full Pipeline Expert: 79.7% ($59/74$ correct pairs)
• Simple Baseline Model: 78.4% ($58/74$ correct pairs)

---

How to Initialize and Run Inference

To use the full routing architecture, place this initialization wrapper script alongside your downloaded repository folders:

import torch
import json
import os
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

class RomanceMoLEInferenceEngine:
    def __init__(self, repo_path):
        print("Loading padded trans-tokenised base architecture...")
        self.tokenizer = AutoTokenizer.from_pretrained(f"{repo_path}/base_model")
        self.base_model = AutoModelForCausalLM.from_pretrained(
            f"{repo_path}/base_model", 
            torch_dtype=torch.bfloat16, 
            device_map="auto"
        )
        
        # Load the configuration metrics for the router
        with open(f"{repo_path}/router/router_config.json", "r") as f:
            self.config = json.load(f)
            
        print("Injecting frozen localized expert adapters...")
        self.model = PeftModel.from_pretrained(self.base_model, f"{repo_path}/adapters/lora_oc", adapter_name="oc")
        self.model.load_adapter(f"{repo_path}/adapters/lora_fr", adapter_name="fr")
        self.model.load_adapter(f"{repo_path}/adapters/lora_ca", adapter_name="ca")
        
        print("Restoring hierarchical layer gating configurations...")
        self.router_weights = torch.load(f"{repo_path}/router/router_weights.pt")
        self.threshold = self.config["sequence_route_threshold"]
        
    def generate(self, text, temperature=0.5):
        inputs = self.tokenizer(text, return_tensors="pt").to(self.base_model.device)
        # Inference pipeline matches target routing layers configuration (Layers 0-7 token-level / 8+ sequence pooling)
        with torch.no_grad():
            outputs = self.model.generate(**inputs, max_new_tokens=64, temperature=temperature)
        return self.tokenizer.decode(outputs[0], skip_special_tokens=True)

# Usage:
# engine = RomanceMoLEInferenceEngine("./velociraptorela-208/Romance-MoLE")
# print(engine.generate("Abans d'entrar al trabalh, soi l'assistant virtual..."))

Citation Information

If you utilize this framework, the custom trans-tokenization weights, or the hierarchical routing pipeline in your research, please cite the foundational thesis work:

@mastersthesis{goga2026romancemole,
  author       = {Goga, Mihaela},
  title        = {Romance-MoLE: Rich Cousins and Their Benefits --- An Approach to Low-Resource Language Modelling},
  school       = {University of Gothenburg, Department of Philosophy, Linguistics and Theory of Science},
  year         = {2026},
  type         = {Master's Thesis},
  month        = {June},
  note         = {Supervised by Jonas Lind, Anna Lokrantz, and Asad Sayeed. Examined by Sharid Loáiciga.}
}

---