Upload folder using huggingface_hub

README.md

@@ -21,34 +21,40 @@ model-index:
     metrics:
     - name: BLEU
       type: bleu
-      value: 28.58
+      value: 29.36
     - name: CHRF
       type: chrf
-      value: 57.46
+      value: 58.10
 ---
 
 
 # `quickmt-zh-en` Neural Machine Translation Model 
 
-# Usage
+`quickmt-zh-en` is a reasonably fast and reasonably accurate neural machine translation model for translation from `zh` into `en`.
 
-## Install `quickmt`
+
+## Model Information
+
+* Trained using [`eole`](https://github.com/eole-nlp/eole)
+* 200M parameter transformer 'big' with 8 encoder layers and 2 decoder layers
+* Separate source and target Sentencepiece tokenizers
+* Exported for fast inference to [CTranslate2](https://github.com/OpenNMT/CTranslate2) format
+* Training data: https://huggingface.co/datasets/quickmt/quickmt-train.zh-en/tree/main
+
+See the `eole` model configuration in this repository for further details. 
+
+
+## Usage with `quickmt`
+
+First, install `quickmt` and download the model
 
 ```bash
 git clone https://github.com/quickmt/quickmt.git
 pip install ./quickmt/
-```
 
-## Download model
-
-```bash
 quickmt-model-download quickmt/quickmt-zh-en ./quickmt-zh-en
 ```
 
-## Use model
-
-Inference with `quickmt`:
-
 ```python
 from quickmt import Translator
 
@@ -63,125 +69,20 @@ t(["他补充道：“我们现在有 4 个月大没有糖尿病的老鼠，但
 t(["他补充道：“我们现在有 4 个月大没有糖尿病的老鼠，但它们曾经得过该病。”"], sampling_temperature=1.2, beam_size=1, sampling_topk=50, sampling_topp=0.9)
 ```
 
-The model is in `ctranslate2` format, and the tokenizers are `sentencepiece`, so you can use the model files directly if you want. It would be fairly easy to get them to work with e.g. [LibreTranslate](https://libretranslate.com/) which also uses `ctranslate2` and `sentencepiece`.
+The model is in `ctranslate2` format, and the tokenizers are `sentencepiece`, so you can use `ctranslate2` directly instead of through `quickmt`. It is also possible  to get this model to work with e.g. [LibreTranslate](https://libretranslate.com/) which also uses `ctranslate2` and `sentencepiece`.
 
-# Model Information
-
-* Trained using [`eole`](https://github.com/eole-nlp/eole)
-    - It took about 1 day on a single RTX 4090 on [vast.ai](https://cloud.vast.ai)
-* Exported for fast inference to []CTranslate2](https://github.com/OpenNMT/CTranslate2) format
-* Training data: https://huggingface.co/datasets/quickmt/quickmt-train.zh-en/tree/main
 
 ## Metrics
 
-BLEU and CHRF2 calculated with [sacrebleu](https://github.com/mjpost/sacrebleu) on the Flores200 `devtest` test set ("zho_Hans"->"eng_Latn").
-
-"Time" is the time to translate the following input with a single CPU core:
-
-> 2019冠状病毒病（英語：Coronavirus disease 2019，缩写：COVID-19[17][18]），是一種由嚴重急性呼吸系統綜合症冠狀病毒2型（縮寫：SARS-CoV-2）引發的傳染病，导致了一场持续的疫情，成为人類歷史上致死人數最多的流行病之一。
-
-| Model                            | bleu  | chrf2 | Time (s) | 
-| -------------------------------- | ----- | ----- | ----     |
-| quickmt/quickmt-zh-en            | 28.58 | 57.46 |  0.670   |
-| Helsinki-NLP/opus-mt-zh-en       | 23.35 | 53.60 |  0.838   |
-| facebook/m2m100_418M             | 18.96 | 50.06 | 11.5     |
-| facebook/nllb-200-distilled-600M | 26.22 | 55.17 | 13.2     |
-| facebook/nllb-200-distilled-1.3B | 28.54 | 57.34 | 23.6     |
-| facebook/m2m100_1.2B             | 24.68 | 54.68 | 25.7     |
-| google/madlad400-3b-mt           | 28.74 | 58.01 | ???      |
-
-`quickmt-zh-en` is the fastest and delivers fairly high quality. 
-
-Helsinki-NLP/opus-mt-zh-en is one of the most downloaded machine translation models on HuggingFace, and this model is considerably more accurate *and* a bit faster.
-
-
-## Training Configuration
-
-```yaml
-### Vocab
-src_vocab_size: 20000
-tgt_vocab_size: 20000
-share_vocab: False
-
-data:
-    corpus_1:
-        path_src: hf://quickmt/quickmt-train-zh-en/zh
-        path_tgt: hf://quickmt/quickmt-train-zh-en/en
-        path_sco: hf://quickmt/quickmt-train-zh-en/sco
-    valid:
-        path_src: zh-en/dev.zho
-        path_tgt: zh-en/dev.eng
-
-transforms: [sentencepiece, filtertoolong]
-transforms_configs:
-  sentencepiece:
-    src_subword_model: "zh-en/src.spm.model"
-    tgt_subword_model: "zh-en/tgt.spm.model"
-  filtertoolong:
-    src_seq_length: 512
-    tgt_seq_length: 512
-
-training:
-    # Run configuration
-    model_path: quickmt-zh-en
-    keep_checkpoint: 4
-    save_checkpoint_steps: 1000
-    train_steps: 104000
-    valid_steps: 1000
-    
-    # Train on a single GPU
-    world_size: 1
-    gpu_ranks: [0]
-
-    # Batching
-    batch_type: "tokens"
-    batch_size: 13312
-    valid_batch_size: 13312
-    batch_size_multiple: 8
-    accum_count: [4]
-    accum_steps: [0]
-
-    # Optimizer & Compute
-    compute_dtype: "bfloat16"
-    optim: "pagedadamw8bit"
-    learning_rate: 1.0
-    warmup_steps: 10000
-    decay_method: "noam"
-    adam_beta2: 0.998
-
-    # Data loading
-    bucket_size: 262144
-    num_workers: 4
-    prefetch_factor: 100
-
-    # Hyperparams
-    dropout_steps: [0]
-    dropout: [0.1]
-    attention_dropout: [0.1]
-    max_grad_norm: 0
-    label_smoothing: 0.1
-    average_decay: 0.0001
-    param_init_method: xavier_uniform
-    normalization: "tokens"
-
-model:
-    architecture: "transformer"
-    layer_norm: standard
-    share_embeddings: false
-    share_decoder_embeddings: true
-    add_ffnbias: true
-    mlp_activation_fn: gated-silu
-    add_estimator: false
-    add_qkvbias: false
-    norm_eps: 1e-6
-    hidden_size: 1024
-    encoder:
-        layers: 8
-    decoder:
-        layers: 2
-    heads: 16
-    transformer_ff: 4096
-    embeddings:
-        word_vec_size: 1024
-        position_encoding_type: "SinusoidalInterleaved"
-```
+BLEU and CHRF2 calculated with [sacrebleu](https://github.com/mjpost/sacrebleu) on the [Flores200 `devtest` test set](https://huggingface.co/datasets/facebook/flores) ("zho_Hans"->"eng_Latn"). COMET22 with the [`comet`](https://github.com/Unbabel/COMET) library and the [default model](https://huggingface.co/Unbabel/wmt22-comet-da). "Time (s)" is the time in seconds to translate (using `ctranslate2`) the flores-devtest dataset (1012 sentences) on an RTX 4070s GPU with batch size 32 except for `madlad400-3b-mt` which used a batch size of 1. 
+
+| Model                            | bleu  | chrf2 | comet22  | Time (s) |
+| -------------------------------- | ----- | ----- | ----     |  ----    |
+| quickmt/quickmt-zh-en            | 29.36 | 58.10 | 0.8655   | 0.88     |
+| Helsinki-NLP/opus-mt-zh-en       | 23.35 | 53.60 | 0.8426   | 3.78     |
+| facebook/m2m100_418M             | 15.99 | 50.13 | 0.7881   | 16.61    |
+| facebook/nllb-200-distilled-600M | 26.22 | 55.18 | 0.8507   | 20.89    |
+| facebook/m2m100_1.2B             | 20.30 | 54.23 | 0.8206   | 33.12    |
+| facebook/nllb-200-distilled-1.3B | 28.56 | 57.35 | 0.8620   | 36.64    |
+
+`quickmt-zh-en` is the fastest *and* highest quality.

eole-config.yaml

@@ -0,0 +1,99 @@
+## IO
+save_data: zh_en/data_spm
+overwrite: True
+seed: 1234
+report_every: 100
+valid_metrics: ["BLEU"]
+tensorboard: true
+tensorboard_log_dir: tensorboard
+
+### Vocab
+src_vocab: zh-en/src.eole.vocab
+tgt_vocab: zh-en/tgt.eole.vocab
+src_vocab_size: 32000
+tgt_vocab_size: 32000
+vocab_size_multiple: 8
+share_vocab: False
+n_sample: 0
+
+data:
+    corpus_1:
+        path_src: hf://quickmt/quickmt-train-zh-en/zh
+        path_tgt: hf://quickmt/quickmt-train-zh-en/en
+        path_sco: hf://quickmt/quickmt-train-zh-en/sco
+    valid:
+        path_src: zh-en/dev.zho
+        path_tgt: zh-en/dev.eng
+
+transforms: [sentencepiece, filtertoolong]
+transforms_configs:
+  sentencepiece:
+    src_subword_model: "zh-en/src.spm.model"
+    tgt_subword_model: "zh-en/tgt.spm.model"
+  filtertoolong:
+    src_seq_length: 256
+    tgt_seq_length: 256
+
+training:
+    # Run configuration
+    model_path: zh-en/model
+    keep_checkpoint: 4
+    save_checkpoint_steps: 2000
+    train_steps: 100000
+    valid_steps: 2000
+    
+    # Train on a single GPU
+    world_size: 1
+    gpu_ranks: [0]
+
+    # Batching
+    batch_type: "tokens"
+    batch_size: 8192
+    valid_batch_size: 8192
+    batch_size_multiple: 8
+    accum_count: [16]
+    accum_steps: [0]
+
+    # Optimizer & Compute
+    compute_dtype: "bf16"
+    optim: "pagedadamw8bit"
+    learning_rate: 2.0
+    warmup_steps: 10000
+    decay_method: "noam"
+    adam_beta2: 0.998
+
+    # Data loading
+    bucket_size: 128000
+    num_workers: 4
+    prefetch_factor: 100
+    
+    # Hyperparams
+    dropout_steps: [0]
+    dropout: [0.1]
+    attention_dropout: [0.1]
+    max_grad_norm: 2
+    label_smoothing: 0.1
+    average_decay: 0.0001
+    param_init_method: xavier_uniform
+    normalization: "tokens"
+
+model:
+    architecture: "transformer"
+    layer_norm: standard
+    share_embeddings: false
+    share_decoder_embeddings: true
+    add_ffnbias: true
+    mlp_activation_fn: gelu
+    add_estimator: false
+    add_qkvbias: false
+    norm_eps: 1e-6
+    hidden_size: 1024
+    encoder:
+        layers: 8
+    decoder:
+        layers: 2
+    heads: 16
+    transformer_ff: 4096
+    embeddings:
+        word_vec_size: 1024
+        position_encoding_type: "SinusoidalInterleaved"

eole_model/config.json

@@ -0,0 +1,150 @@
+{
+  "save_data": "zh_en/data_spm",
+  "src_vocab": "zh-en-benchmark/src.eole.vocab",
+  "report_every": 100,
+  "share_vocab": false,
+  "tgt_vocab": "zh-en-benchmark/tgt.eole.vocab",
+  "vocab_size_multiple": 8,
+  "tensorboard_log_dir_dated": "tensorboard/Feb-12_13-34-26",
+  "src_vocab_size": 32000,
+  "tensorboard": true,
+  "n_sample": 0,
+  "tgt_vocab_size": 32000,
+  "valid_metrics": [
+    "BLEU"
+  ],
+  "tensorboard_log_dir": "tensorboard",
+  "seed": 1234,
+  "overwrite": true,
+  "transforms": [
+    "sentencepiece",
+    "filtertoolong"
+  ],
+  "training": {
+    "accum_count": [
+      16
+    ],
+    "train_steps": 100000,
+    "gpu_ranks": [
+      0
+    ],
+    "save_checkpoint_steps": 2000,
+    "decay_method": "noam",
+    "bucket_size": 128000,
+    "world_size": 1,
+    "accum_steps": [
+      0
+    ],
+    "optim": "pagedadamw8bit",
+    "prefetch_factor": 100,
+    "compute_dtype": "torch.bfloat16",
+    "normalization": "tokens",
+    "label_smoothing": 0.1,
+    "batch_size_multiple": 8,
+    "dropout_steps": [
+      0
+    ],
+    "average_decay": 0.0001,
+    "dropout": [
+      0.1
+    ],
+    "batch_type": "tokens",
+    "valid_batch_size": 8192,
+    "param_init_method": "xavier_uniform",
+    "adam_beta2": 0.998,
+    "model_path": "zh-en-benchmark/model",
+    "keep_checkpoint": 4,
+    "num_workers": 0,
+    "batch_size": 8192,
+    "attention_dropout": [
+      0.1
+    ],
+    "warmup_steps": 10000,
+    "valid_steps": 2000,
+    "max_grad_norm": 2.0,
+    "learning_rate": 2.0
+  },
+  "data": {
+    "corpus_1": {
+      "path_align": null,
+      "path_src": "zh-en/train.ready.zh",
+      "path_tgt": "zh-en/train.ready.en",
+      "transforms": [
+        "sentencepiece",
+        "filtertoolong"
+      ]
+    },
+    "valid": {
+      "path_align": null,
+      "path_src": "zh-en-benchmark/dev.zho",
+      "path_tgt": "zh-en-benchmark/dev.eng",
+      "transforms": [
+        "sentencepiece",
+        "filtertoolong"
+      ]
+    }
+  },
+  "transforms_configs": {
+    "sentencepiece": {
+      "tgt_subword_model": "${MODEL_PATH}/tgt.spm.model",
+      "src_subword_model": "${MODEL_PATH}/src.spm.model"
+    },
+    "filtertoolong": {
+      "tgt_seq_length": 256,
+      "src_seq_length": 256
+    }
+  },
+  "model": {
+    "share_decoder_embeddings": true,
+    "position_encoding_type": "SinusoidalInterleaved",
+    "add_qkvbias": false,
+    "architecture": "transformer",
+    "add_ffnbias": true,
+    "hidden_size": 1024,
+    "transformer_ff": 4096,
+    "mlp_activation_fn": "gelu",
+    "norm_eps": 1e-06,
+    "layer_norm": "standard",
+    "heads": 16,
+    "add_estimator": false,
+    "share_embeddings": false,
+    "decoder": {
+      "heads": 16,
+      "decoder_type": "transformer",
+      "position_encoding_type": "SinusoidalInterleaved",
+      "add_qkvbias": false,
+      "layers": 2,
+      "add_ffnbias": true,
+      "hidden_size": 1024,
+      "n_positions": null,
+      "transformer_ff": 4096,
+      "rope_config": null,
+      "mlp_activation_fn": "gelu",
+      "norm_eps": 1e-06,
+      "layer_norm": "standard",
+      "tgt_word_vec_size": 1024
+    },
+    "embeddings": {
+      "word_vec_size": 1024,
+      "position_encoding_type": "SinusoidalInterleaved",
+      "tgt_word_vec_size": 1024,
+      "src_word_vec_size": 1024
+    },
+    "encoder": {
+      "heads": 16,
+      "position_encoding_type": "SinusoidalInterleaved",
+      "add_qkvbias": false,
+      "layers": 8,
+      "add_ffnbias": true,
+      "hidden_size": 1024,
+      "n_positions": null,
+      "src_word_vec_size": 1024,
+      "transformer_ff": 4096,
+      "rope_config": null,
+      "mlp_activation_fn": "gelu",
+      "norm_eps": 1e-06,
+      "layer_norm": "standard",
+      "encoder_type": "transformer"
+    }
+  }
+}

eole_model/model.00.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a6b78dad9fa4e560ce0abe0fc7c2b317ebe560fc23aa1897419253a1b334872d
+size 820042008

eole_model/src.spm.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:23d03d562fc3f8fe57e497dac0ece4827c254675a80c103fc4bb4040638ceb67
+size 733978

eole_model/tgt.spm.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c373f1d78753313b0dbc337058bf8450e1fdd6fe662a49e0941affce44ec14c5
+size 800955

model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:408f5484e3d983d52cabf867241f10e0159e4017b7cb05718fa580ab0f081b86
-size 444765910
+oid sha256:494518d6282fcd01f850ab4ab096e6a5c937aa834290a62e5efd435275828c9d
+size 409972810

src.spm.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0631c1a3d400ac4f42c8d63fb94ae71c69ee00acd4648c05eb02d952e7f7d0ef
-size 538185
+oid sha256:23d03d562fc3f8fe57e497dac0ece4827c254675a80c103fc4bb4040638ceb67
+size 733978

tgt.spm.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:83dcd0d44ad898117ae6c7fe24d996f186940d97265c4e91a78e3e07f657bc9e
-size 589008
+oid sha256:c373f1d78753313b0dbc337058bf8450e1fdd6fe662a49e0941affce44ec14c5
+size 800955