Hugging Face's logo

Branvar
/
dco-v0.5-ambrosius

Translation
PEFT
Safetensors
Spanish
Latin
latin
spanish
patristic
historical-mt
qlora
lora
low-resource
Model card Files
xet
 Community

Mitruna-31B (dco-v0.5-ambrosius)

QLoRA adapter for Latin to Spanish patristic translation, fine-tuned from Gemma-4 31B-instruct on a 7,603-pair multi-publisher academic Spanish corpus. Reaches frontier-class quality within the trained editorial register at approximately 1/30 the inference cost of Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro.

  • Authors: Àngel Solé Hernández (Mitruna)
  • Affiliation: Mitruna (https://mitruna.com)
  • Contact: angel@mitruna.com
  • Paper: WMT 2026 submission (preprint coming soon to arXiv:cs.CL)
  • Base model: google/gemma-4-31b-it
  • License: Gemma terms of use (inherits from base)

Headline results

On Publisher-A held-out (n=118 common across 9 systems on all six metrics), Mitruna-31B retains Bonferroni-significant first place on BLEU and BERTScore-F1 over every frontier closed-source configuration evaluated (Claude Opus 4.6, GPT-5.4, Gemini 3.1 Pro, in both zero-shot and 4-shot prompting; all p_Bonf < .001). On xCOMET-XL (the WMT24 SOTA neural metric), Mitruna-31B ranks first with Bonferroni-significant gains over Claude zero-shot (+0.0329), GPT zero-shot (+0.0315), and Gemma-4 31B base (+0.0303).

On a held-out second publisher (Publisher-B, n=643 common), the system reaches statistical parity with GPT-5.4 on COMET and BLEU but loses Bonferroni-significantly to Claude Opus 4.6 and Gemini 3.1 Pro on most metrics.

A cap-controlled ablation against Mitruna-31B-no-B (same architecture and hyperparameters, minus Publisher-B) confirms multi-publisher robustness: five of six metrics gain Bonferroni-significantly on Publisher-B (COMET, COMET-Kiwi, chrF++, BLEU, BERTScore-F1; all p_Bonf < .001) with zero regression on Publisher-A across all six metrics.

Evaluation tables

Publisher-A held-out (n=118 common, 9 systems)

System COMET COMET-Kiwi chrF++ BLEU BERTScore-F1 xCOMET-XL
Mitruna-31B (ours) 0.7023 0.5229 51.32 20.96 0.9050 0.2193
Mitruna-31B-no-B (ablation) 0.7009 0.5219 50.99 20.94 0.9040 0.2182
Gemma-4 31B base 0.6680 0.5443 48.29 14.59 0.8862 0.1890
Claude Opus 4.6 (zs) 0.6743 0.5424 50.00 16.20 0.8891 0.1864
GPT-5.4 (zs) 0.6686 0.5311 49.85 16.16 0.8891 0.1877
Gemini 3.1 Pro (zs) 0.6652 0.5330 49.11 15.16 0.8870 0.1941
Claude Opus 4.6 (4-shot) 0.6934 0.5537 51.23 18.42 0.8975 0.2076
GPT-5.4 (4-shot) 0.6886 0.5448 50.15 16.98 0.8961 0.2037
Gemini 3.1 Pro (4-shot) 0.6824 0.5467 50.25 16.91 0.8941 0.1969

Publisher-B held-out (n=643 common, 9 systems)

System COMET COMET-Kiwi chrF++ BLEU BERTScore-F1 xCOMET-XL
Mitruna-31B (ours) 0.7270 0.5499 56.83 30.68 0.9283 0.3688
Mitruna-31B-no-B (ablation) 0.7164 0.5332 54.10 27.43 0.9229 0.3651
Gemma-4 31B base 0.7196 0.5719 55.39 27.22 0.9256 0.3593
Claude Opus 4.6 (zs) 0.7375 0.5631 58.73 31.57 0.9307 0.3879
GPT-5.4 (zs) 0.7258 0.5621 58.11 31.09 0.9303 0.3714
Gemini 3.1 Pro (zs) 0.7307 0.5747 59.20 32.54 0.9318 0.3972
Claude Opus 4.6 (4-shot) 0.7393 0.5674 59.29 33.16 0.9323 0.4002
GPT-5.4 (4-shot) 0.7354 0.5548 57.79 30.49 0.9297 0.3796

OOD probe (Bede + Julian of Toledo, n=80 common)

Two unseen authors not present in any training data. Frontier models pull ahead Bonferroni-significantly on chrF++ and COMET-Kiwi.

System COMET COMET-Kiwi chrF++ BLEU BERTScore-F1
Mitruna-31B (ours) 0.7108 0.5792 50.31 20.33 0.9083
Mitruna-31B-no-B 0.7124 0.5849 49.92 19.42 0.9072
Claude Opus 4.6 (zs) 0.7293 0.6179 53.21 20.79 0.9065
Claude Opus 4.6 (4-shot) 0.7300 0.6166 53.73 21.73 0.9079

Cost

System / deployment $ per 1,000 pairs
Mitruna-31B (Apple M3 Ultra, local) ~$0 + electricity
Mitruna-31B (A100 80GB cloud) $1.00 to $1.50
Mitruna-31B (H100 SXM cloud) $1.00
Gemma-4 31B base (OpenRouter API) $0.30
Claude Opus 4.6 $40 to $60
GPT-5.4 $40 to $60
Gemini 3.1 Pro $30 to $50

Cloud throughput benchmarked at ~3,000 pairs/hour with vLLM batching. Frontier API prices are OpenRouter rates as of April 2026. Mitruna-31B cloud-deployed is 30 to 45 times cheaper than frontier API access at parity-or-near-parity quality.

Intended use

Translation of patristic and ecclesiastical Latin (2nd to 13th century) into contemporary academic Spanish in a register typical of academic translatology, suitable for editorial use by Hispanophone humanities researchers, theologians, and patrologists.

Out of scope: classical Latin literary translation in the ad verbum register, modern Spanish colloquial output, languages other than Latin source / Spanish target, scholarly authority independent of human philological review.

How to use

The adapter is a PEFT/LoRA over google/gemma-4-31b-it. Load with peft and merge into the base for inference, or serve the merged weights via vLLM.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_id = "google/gemma-4-31b-it"
adapter_id = "Branvar/dco-v0.5-ambrosius"

tokenizer = AutoTokenizer.from_pretrained(base_id)
base = AutoModelForCausalLM.from_pretrained(
    base_id, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base, adapter_id)
model = model.merge_and_unload()  # optional: bake LoRA into base for vLLM serving

system_prompt = (
    "Eres un traductor académico de latín eclesiástico al castellano. "
    "Traduce el siguiente pasaje latino al castellano de forma fiel y "
    "completa, sin paráfrasis ni omisiones, en un registro académico "
    "contemporáneo (estilo Biblioteca de Autores Cristianos). Preserva "
    "las citas bíblicas de forma consistente. Responde SOLO con la "
    "traducción."
)

latin_text = "Sedebo in monte alto super montes altos in Aquilonem..."

messages = [
    {"role": "user", "content": f"{system_prompt}\n\n{latin_text}"},
]
inputs = tokenizer.apply_chat_template(
    messages, return_tensors="pt", add_generation_prompt=True
).to(model.device)

outputs = model.generate(inputs, max_new_tokens=1024, do_sample=False)
print(tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True))

For production-scale serving, merge the adapter and serve via vLLM:

python -c "
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained('google/gemma-4-31b-it', torch_dtype='bfloat16')
m = PeftModel.from_pretrained(base, 'Branvar/dco-v0.5-ambrosius').merge_and_unload()
m.save_pretrained('./mitruna-31b-merged')
AutoTokenizer.from_pretrained('google/gemma-4-31b-it').save_pretrained('./mitruna-31b-merged')
"

vllm serve ./mitruna-31b-merged --dtype bfloat16 --max-model-len 6144

Training details

Parameter Value
Base model google/gemma-4-31b-it
Adapter type QLoRA (LoRA over 4-bit NF4 quantized base)
LoRA targets All attention and MLP projections
Rank 64
Alpha 64
Dropout 0.0
Max sequence length 6,144 tokens
Epochs 2
Total steps 952
Effective batch size 8
Learning rate 1e-4, cosine schedule, 3% warmup
Loss Cross-entropy on response tokens only (response-masked)
Hardware 1× NVIDIA H100 80GB SXM (RunPod secure cloud)
Runtime Unsloth
Training time 6h 51m
Train loss (final) 1.020
Validation loss (final) 1.178
Train-val gap 0.019 (no overfit)
Compute cost ~$20 USD

After training, the LoRA is merged into bf16 weights with save_pretrained_merged (Unsloth) and served via vLLM at full precision.

Training data

7,603 Latin to Spanish parallel pairs (8,448 including 845 validation), drawn from two contemporary academic Spanish patristic publishers (Publisher-A and Publisher-B, identities anonymized in the preprint and revealed in the camera-ready Data Statement).

  • Publisher-A: ~5,200 source pairs after structural alignment, covering ten 2nd to 13th century Latin authors including Augustine, Tertullian, Cyprian, Jerome, Gregory the Great, and Cassiodorus.
  • Publisher-B: ~3,200 pairs covering an additional 4th to 5th century author set (with overlap on Cyprian and Ambrose, but in a different translator's hand).

Latin source texts are drawn from CSEL, PL Migne (with Mistral OCR for non-CSEL volumes), and the OpenGreekAndLatin csel-dev TEI corpus where available.

A per-author cap of 15% of total pairs is applied (author_cap_frac=0.15) to balance author diversity against held-out coverage. The training mix is Publisher-A 4,386 (62%) and Publisher-B 3,217 (38%).

Four full Publisher-B volumes covering four distinct 4th to 5th century Latin authors not represented elsewhere in the training set are excluded from training entirely to construct the multi-publisher generalization eval set.

The training corpus itself is not redistributable: source Latin is in the public domain but contemporary Spanish translations are under copyright held by the respective publishers. Training was conducted under the EU CDSM Article 3 text-and-data-mining exception for scientific research and the Spanish LPI Article 67 research exception. We release the trained adapter (a derivative learned representation, not the data) and the data manifest (a list of source identifiers, not the texts).

Evaluation methodology

  • Test sets: three held-out sets, none overlapping with training. Publisher-A regression set (n=118 common across 9 systems), Publisher-B held-out (n=643 common), OOD probe with Bede + Julian of Toledo (n=80 common, neither author in training).
  • Metrics: COMET (Unbabel/wmt22-comet-da), COMET-Kiwi (Unbabel/wmt22-cometkiwi-da), chrF++, BLEU via sacreBLEU, BERTScore-F1 with xlm-roberta-large, xCOMET-XL (Unbabel/XCOMET-XL).
  • Statistical procedure: paired bootstrap confidence intervals (BCa, 9,999 resamples), paired randomization tests (10,000 trials) over per-segment scores. Bonferroni correction applied per metric over the family of pairwise comparisons. Significance reported at α=0.05.
  • Comparison systems (9): ours (Mitruna-31B), Mitruna-31B-no-B (cap-controlled ablation, identical hyperparameters minus Publisher-B), Gemma-4 31B base, Claude Opus 4.6 / GPT-5.4 / Gemini 3.1 Pro in zero-shot, and the same three frontier models in 4-shot prompting. Frontier models queried via OpenRouter at temperature 0.

Limitations

  • Single language pair. All experiments are Latin-to-Spanish. No claim about cross-lingual generality of the multi-publisher robustness property.
  • No human MQM evaluation yet. All evaluation is via reference-based and reference-free learned metrics plus surface-form metrics. Multidimensional Quality Metric expert annotation by patrologists is the gold standard for translation quality assessment in this domain and is deferred to a journal-version extension.
  • Pretraining data contamination cannot be fully ruled out. Gemma-4 31B is pretrained on a large web-scale corpus whose composition is not fully disclosed. The held-out volumes (in particular the OOD probe set with two unseen authors) are the most defensible contamination-resistant evaluation surface.
  • Small OOD probe set. Only 80 sentence pairs from Bede + Julian of Toledo. Confidence intervals on the OOD probe set are correspondingly wide.
  • Single-evaluator qualitative analysis. The qualitative philological analysis was performed by a single annotator (the lead author). Inter-annotator agreement against a second patristic-translation expert is left to journal-version evaluation.
  • Frontier-LLM prompting protocol. We compare against the default specialized output an under-resourced humanities researcher would obtain via API access without specialized prompt engineering (a single fixed system prompt for all frontier models, plus a 4-shot variant). We do not measure how far the gap closes under aggressive register-conditioning prompts.
  • Register specialization. The adapter outputs a contemporary academic translatology register (BAC-style) by default and may not respond well to alternative-register prompts (e.g., literal ad verbum).

Citation

Coming soon to arXiv:cs.CL. Preliminary BibTeX:

@misc{sole2026mitruna,
  title={Multi-Publisher QLoRA for Patristic Latin-Spanish Translation:
         Frontier-Class Quality at Open-Model Cost},
  author={Sol{\'e} Hern{\'a}ndez, {\`A}ngel},
  year={2026},
  note={WMT 2026 submission, arXiv preprint forthcoming},
  howpublished={\url{https://huggingface.co/Branvar/dco-v0.5-ambrosius}}
}

Acknowledgments

Training compute on RunPod (H100 80GB SXM secure cloud). Unsloth framework for QLoRA training. vLLM for serving. xCOMET-XL, COMET, COMET-Kiwi from Unbabel. sacreBLEU. BERTScore. Latin source corpora from CSEL, PL Migne, OpenGreekAndLatin csel-dev TEI. Spanish translations from two contemporary academic patristic publishers (identities revealed in the camera-ready Data Statement). The training corpus itself is not redistributable; we release only the trained adapter and the source-identifier manifest.

Contact

Downloads last month
1
Inference Providers NEW
Translation
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support