Initial version

README.md

@@ -1,3 +1,61 @@
----
-license: mit
----
+---
+language: 
+- ca
+- eu
+
+tags:
+- translation
+
+library_name: opennmt
+license: mit
+metrics:
+- bleu
+
+inference: false
+---
+
+### Introduction
+
+Catalan - Basque translation model for OpenNMT. These are the same models that we have in production at https://www.softcatala.org/traductor/.
+The models are quantified for low latency.
+
+### Usage
+
+Install the necessary dependencies:
+
+
+```bash
+pip3 install ctranslate2 pyonmttok
+```
+
+
+Simple tokenization & translation using Python:
+
+
+```python
+import ctranslate2
+import pyonmttok
+from huggingface_hub import snapshot_download
+model_dir = snapshot_download(repo_id="softcatala/translate-cat-fra", revision="main")
+
+tokenizer=pyonmttok.Tokenizer(mode="none", sp_model_path = model_dir + "/sp_m.model")
+tokenized=tokenizer.tokenize("Hola amics")
+
+translator = ctranslate2.Translator(model_dir)
+translated = translator.translate_batch([tokenized[0]])
+print(tokenizer.detokenize(translated[0][0]['tokens']))
+```
+## Data used 
+
+Build using https://huggingface.co/datasets/projecte-aina/CA-EU_Parallel_Corpus corpus.
+
+## Benchmarks
+
+| testset                               | BLEU  |
+|---------------------------------------|-------|
+| test dataset (from train/dev/test)   	| 27.3	|
+| Flores200 dataset                     | 17.1	|
+
+## Additional information
+* https://github.com/Softcatala/nmt-models
+* https://github.com/Softcatala/parallel-catalan-corpus

config.json

@@ -0,0 +1,9 @@
+{
+  "add_source_bos": false,
+  "add_source_eos": false,
+  "bos_token": "<s>",
+  "decoder_start_token": "<s>",
+  "eos_token": "</s>",
+  "layer_norm_epsilon": null,
+  "unk_token": "<unk>"
+}

model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b6a5e7d954548c606fac5120dc4fdaf37920a9edbdc20b93269b70a7f3528647
+size 70727741

model_description.txt

@@ -0,0 +1,9 @@
+Model description: cat-eus
+Date: 2024-08-12
+TF version 2.10.0, OpenNMT version 2.29.1, CTranslate2 version 3.22.0
+Test data set
+  BLEU|nrefs:1|bs:1000|seed:12345|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0 = 27.3 (μ = 27.3 ± 0.3) 59.6/33.6/21.4/14.4 (BP = 0.973 ratio = 0.973 hyp_len = 310620 ref_len = 319206)
+chrF2|nrefs:1|bs:1000|seed:12345|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0 = 63.1 (μ = 63.1 ± 0.2)
+Flores data set
+  BLEU|nrefs:1|bs:1000|seed:12345|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0 = 17.1 (μ = 17.1 ± 0.9) 49.8/22.5/11.8/6.6 (BP = 0.998 ratio = 0.998 hyp_len = 20755 ref_len = 20788)
+chrF2|nrefs:1|bs:1000|seed:12345|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0 = 56.2 (μ = 56.1 ± 0.6)

sp_m.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:463a5d91a855d512edebcdddd98ca9aa4e5ee72674d87924f86c65e0353b4188
+size 1160319