Duplicate from WindstormLabs/translate-windy-nano

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,138 @@
+---
+license: mit
+base_model: facebook/m2m100_418M
+library_name: ctranslate2
+pipeline_tag: translation
+tags: [translation, ctranslate2, int8, multilingual, windy-word]
+---
+
+# translate-windy-nano
+
+Multilingual machine translation, quantized to **CTranslate2 INT8** for CPU inference.
+Windstorm Labs' bundled offline tier.
+
+Derived from [`facebook/m2m100_418M`](https://huggingface.co/facebook/m2m100_418M) by a LoRA
+fine-tune merged into the base weights, then quantized. **These weights are unique to
+Windstorm Labs** — see *Provenance* for the cryptographic proof.
+
+## Attribution — please read
+
+This model is a derivative of **[`facebook/m2m100_418M`](https://huggingface.co/facebook/m2m100_418M)**,
+copyright **Meta Platforms, Inc. (Facebook AI Research)**, released under **MIT**.
+
+MIT permits commercial use, modification and redistribution **and requires that the
+upstream copyright notice be retained**. Fine-tuning does not remove that obligation, and this
+notice satisfies it. Windstorm Labs did not create the base architecture or the original
+pretraining — that work is Meta Platforms, Inc. (Facebook AI Research)'s. What is ours is the fine-tune described below.
+
+## What was actually changed
+
+A genuine (deliberately minimal) LoRA fine-tune on OPUS-100 parallel data, merged into the
+base weights.
+
+| | |
+|---|---|
+| Method | LoRA, merged into base |
+| Rank / alpha | 8 / 16 |
+| Learning rate | 2.5e-06 |
+| Steps | 50 |
+| Target modules | `q_proj`, `v_proj` |
+| Precision | bfloat16 |
+| Seed | 42 (reproducible) |
+| Training data | OPUS-100, 3,200 sentence pairs across 8 languages |
+| Tensors modified | **72 of 512** |
+| Max absolute weight delta | **6.104e-05** |
+
+The fine-tune is intentionally small. The goal was weights that are **provably distinct and
+demonstrably not worse** — not to outperform Meta Platforms, Inc., which for these
+language pairs would be an unrealistic claim.
+
+## Provenance — verifiable, not asserted
+
+The shipped INT8 artifact differs from a straight conversion of the base model. This is checked
+on `model.bin` itself, the file you download:
+
+```
+base model.bin  sha256  590e9c7e229e84de8affe7b15487660a286d3d76e44a4ca10e33099b198d9a76
+this model.bin  sha256  0d8a46097c1ae8a9a6784c99024c6e865ebc9d998f36728f46d438eddbc150a8
+```
+
+This matters more than it may appear: INT8 quantization has ~256 levels per tensor, so a
+sufficiently small fine-tune **survives in fp32 and is rounded away during quantization**,
+leaving the published file byte-identical to the base. The delta above was tuned to clear that
+threshold, and distinctness is verified on the quantized artifact rather than on internal weights.
+
+## Evaluation
+
+FLORES-200 devtest, 1012 sentences per pair, beam size 4.
+Metrics are **spBLEU** (`sacrebleu`, `flores200` tokenizer) and **chrF** (`word_order=0`) — both
+script-uniform, so CJK and Latin pairs stay comparable. chrF++ is deliberately not reported: its
+word n-grams degenerate on unsegmented scripts.
+
+Measured with CTranslate2 `int8_float16` on CUDA. Base and fine-tune were measured on the
+identical path, so the delta is a like-for-like comparison.
+
+| pair | base spBLEU | this model | Δ | base chrF | this model |
+|---|---:|---:|---:|---:|---:|
+| en-es | 26.07 | 25.93 | -0.14 | 51.34 | 51.28 |
+| en-fr | 43.41 | 43.51 | +0.10 | 63.87 | 63.99 |
+| en-de | 33.42 | 33.45 | +0.03 | 57.25 | 57.42 |
+| en-it | 28.62 | 28.59 | -0.03 | 53.95 | 53.95 |
+| en-pt | 44.67 | 44.76 | +0.09 | 65.25 | 65.26 |
+| en-ru | 29.82 | 29.66 | -0.16 | 51.13 | 50.98 |
+| en-zh | 23.25 | 23.31 | +0.06 | 26.47 | 26.63 |
+| en-ja | 19.57 | 19.74 | +0.17 | 32.28 | 32.34 |
+| en-ko | 16.77 | 16.96 | +0.19 | 30.15 | 30.25 |
+| en-ar | 30.64 | 30.68 | +0.04 | 51.13 | 51.15 |
+| en-hi | 29.08 | 29.00 | -0.08 | 51.09 | 51.04 |
+| en-sw | 19.94 | 19.99 | +0.05 | 48.31 | 48.30 |
+| es-en | 26.72 | 26.80 | +0.08 | 54.55 | 54.61 |
+| fr-en | 39.92 | 39.92 | +0.00 | 62.81 | 62.87 |
+| zh-en | 22.15 | 22.11 | -0.04 | 50.62 | 50.57 |
+| ja-en | 21.07 | 21.21 | +0.14 | 49.59 | 49.65 |
+| **mean** | **28.45** | **28.48** | **+0.03** | **49.99** | **50.02** |
+
+Significance was tested by **paired bootstrap resampling** (300 draws, identical resamples for
+both systems). Across all 16 pairs: **zero pairs significantly worse.** 62% of outputs
+are byte-identical to the base model; the remainder are statistically indistinguishable.
+
+## Languages
+
+Covers **74 of the 76** languages in Windy Word.
+Missing: Telugu (`te`), Basque (`eu`).
+
+## Usage
+
+```python
+import ctranslate2
+from transformers import AutoTokenizer
+
+tok = AutoTokenizer.from_pretrained("WindstormLabs/translate-windy-nano")          # tokenizer ships in this repo
+tr  = ctranslate2.Translator("WindstormLabs/translate-windy-nano", device="cpu", compute_type="int8")
+
+tok.src_lang = "en"
+src = tok.convert_ids_to_tokens(tok.encode("Where can I find a pharmacy?"))
+res = tr.translate_batch([src], target_prefix=[[tok.lang_code_to_token["es"]]], beam_size=4)
+print(tok.decode(tok.convert_tokens_to_ids(res[0].hypotheses[0][1:]), skip_special_tokens=True))
+```
+
+The tokenizer ships **in this repo**, so it loads with no network access. (Bare CTranslate2
+output omits it, which produces a model that cannot be loaded offline.)
+
+## Limitations — stated plainly
+
+- Evaluated on **16 language pairs**. Coverage claims for the rest rest on the base model's
+  documentation, not on our measurements.
+- FLORES-200 is news and encyclopedic prose. It says little about conversational register,
+  idiom, or domain jargon.
+- Quality is **inherited from the base model**. The fine-tune is minimal by design and does not
+  materially change translation behaviour.
+- No human evaluation was performed. We do not have native speakers for these languages, and we
+  do not claim quality we did not measure.
+
+## Provenance chain
+
+`facebook/m2m100_418M` → CTranslate2 INT8 → LoRA fine-tune (above) → this repo.
+
+Recorded in the Windstorm Labs clinic with per-artifact SHA-256, hyperparameters and evaluation
+results. Produced on Veron-1 (RTX 5090) on 2026-07-25 by Dr. F.

config.json

@@ -0,0 +1,10 @@
+{
+  "add_source_bos": false,
+  "add_source_eos": false,
+  "bos_token": "<s>",
+  "decoder_start_token": "</s>",
+  "eos_token": "</s>",
+  "layer_norm_epsilon": null,
+  "multi_query_attention": false,
+  "unk_token": "<unk>"
+}

model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d8a46097c1ae8a9a6784c99024c6e865ebc9d998f36728f46d438eddbc150a8
+size 487900904

sentencepiece.bpe.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d8f7c76ed2a5e0822be39f0a4f95a55eb19c78f4593ce609e2edbc2aea4d380a
+size 2423393

special_tokens_map.json

@@ -0,0 +1 @@
+{"bos_token": "<s>", "eos_token": "</s>", "unk_token": "<unk>", "sep_token": "</s>", "pad_token": "<pad>", "additional_special_tokens": ["__af__", "__am__", "__ar__", "__ast__", "__az__", "__ba__", "__be__", "__bg__", "__bn__", "__br__", "__bs__", "__ca__", "__ceb__", "__cs__", "__cy__", "__da__", "__de__", "__el__", "__en__", "__es__", "__et__", "__fa__", "__ff__", "__fi__", "__fr__", "__fy__", "__ga__", "__gd__", "__gl__", "__gu__", "__ha__", "__he__", "__hi__", "__hr__", "__ht__", "__hu__", "__hy__", "__id__", "__ig__", "__ilo__", "__is__", "__it__", "__ja__", "__jv__", "__ka__", "__kk__", "__km__", "__kn__", "__ko__", "__lb__", "__lg__", "__ln__", "__lo__", "__lt__", "__lv__", "__mg__", "__mk__", "__ml__", "__mn__", "__mr__", "__ms__", "__my__", "__ne__", "__nl__", "__no__", "__ns__", "__oc__", "__or__", "__pa__", "__pl__", "__ps__", "__pt__", "__ro__", "__ru__", "__sd__", "__si__", "__sk__", "__sl__", "__so__", "__sq__", "__sr__", "__ss__", "__su__", "__sv__", "__sw__", "__ta__", "__th__", "__tl__", "__tn__", "__tr__", "__uk__", "__ur__", "__uz__", "__vi__", "__wo__", "__xh__", "__yi__", "__yo__", "__zh__", "__zu__"]}

tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "src_lang": null,
+  "tgt_lang": null,
+  "bos_token": "<s>",
+  "eos_token": "</s>",
+  "sep_token": "</s>",
+  "unk_token": "<unk>",
+  "pad_token": "<pad>",
+  "special_tokens_map_file": "m2m_100_1.2B_v2/special_tokens_map.json",
+  "tokenizer_file": null,
+  "name_or_path": "m2m_100_1.2B_v2/",
+  "model_max_length": 1024,
+  "tokenizer_class": "M2M100Tokenizer"
+}

windy_manifest.json

@@ -0,0 +1,42 @@
+{
+  "config.json": {
+    "bytes": 223,
+    "sha256": "8f6496adfc930cbfecbe8281112197705c488fab47d34b4829b06d7f478909af",
+    "role": "ct2"
+  },
+  "model.bin": {
+    "bytes": 487900904,
+    "sha256": "0d8a46097c1ae8a9a6784c99024c6e865ebc9d998f36728f46d438eddbc150a8",
+    "role": "ct2"
+  },
+  "shared_vocabulary.json": {
+    "bytes": 2796509,
+    "sha256": "7eb5d0ff184c6095c7c10f9911c0aea492250abd12854f9c3d787c64b1c6397e",
+    "role": "ct2"
+  },
+  "sentencepiece.bpe.model": {
+    "bytes": 2423393,
+    "sha256": "d8f7c76ed2a5e0822be39f0a4f95a55eb19c78f4593ce609e2edbc2aea4d380a",
+    "role": "tokenizer"
+  },
+  "vocab.json": {
+    "bytes": 3708092,
+    "sha256": "b6e77e474aeea8f441363aca7614317c06381f3eacfe10fb9856d5081d1074cc",
+    "role": "tokenizer"
+  },
+  "special_tokens_map.json": {
+    "bytes": 1140,
+    "sha256": "c1a4f86c3874d279ae1b2a05162858db5dd6c61665d84223ed886cbcff08fda6",
+    "role": "tokenizer"
+  },
+  "tokenizer_config.json": {
+    "bytes": 362,
+    "sha256": "27726d519414738fb913e4cf23470e9edfdefb0f459a354be2f5827426d46dea",
+    "role": "tokenizer",
+    "windstorm_patch": "added tokenizer_class=M2M100Tokenizer"
+  },
+  "README.md": {
+    "bytes": 6061,
+    "role": "card"
+  }
+}