Commit
·
2d78108
1
Parent(s):
1f663e4
Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,16 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
# IndicTrans2 HF Compatible Models
|
| 2 |
|
| 3 |
In this section, we provide details on how to use our [IndicTrans2](https://github.com/AI4Bharat/IndicTrans2) models which were originally trained with the [fairseq](https://github.com/facebookresearch/fairseq) to [HuggingFace transformers](https://huggingface.co/docs/transformers/index) for inference purpose. Our scripts for HuggingFace compatible models are adapted from [M2M100 repository](https://github.com/huggingface/transformers/tree/main/src/transformers/models/m2m_100).
|
|
@@ -19,17 +32,6 @@ source install.sh
|
|
| 19 |
> Note: The `install.sh` script in this directory is specifically for running HF compatible models for inference.
|
| 20 |
|
| 21 |
|
| 22 |
-
### Converting
|
| 23 |
-
|
| 24 |
-
In order to convert the fairseq checkpoint to a PyTorch checkpoint that is compatible with HuggingFace Transformers, use the following command:
|
| 25 |
-
|
| 26 |
-
```bash
|
| 27 |
-
python3 convert_indictrans_checkpoint_to_pytorch.py --fairseq_path <fairseq_checkpoint_best.pt> --pytorch_dump_folder_path <hf_output_dir>
|
| 28 |
-
```
|
| 29 |
-
- `<fairseq_checkpoint_best.pt>`: path to the fairseq `checkpoint_best.pt` that needs to be converted to HF compatible models
|
| 30 |
-
- `<hf_output_dir>`: path to the output directory where the HF compatible models will be saved
|
| 31 |
-
|
| 32 |
-
|
| 33 |
### Models
|
| 34 |
|
| 35 |
| Model | 🤗 HuggingFace Checkpoints |
|
|
@@ -59,4 +61,4 @@ Feel free to modify the `example.py` script to suit your translation needs.
|
|
| 59 |
year = {2023},
|
| 60 |
journal = {arXiv preprint arXiv: 2305.16307}
|
| 61 |
}
|
| 62 |
-
```
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language:
|
| 3 |
+
- en
|
| 4 |
+
- hi
|
| 5 |
+
- te
|
| 6 |
+
- ta
|
| 7 |
+
- mr
|
| 8 |
+
- kn
|
| 9 |
+
- ml
|
| 10 |
+
pipeline_tag: translation
|
| 11 |
+
tags:
|
| 12 |
+
- indic_trans_v2
|
| 13 |
+
---
|
| 14 |
# IndicTrans2 HF Compatible Models
|
| 15 |
|
| 16 |
In this section, we provide details on how to use our [IndicTrans2](https://github.com/AI4Bharat/IndicTrans2) models which were originally trained with the [fairseq](https://github.com/facebookresearch/fairseq) to [HuggingFace transformers](https://huggingface.co/docs/transformers/index) for inference purpose. Our scripts for HuggingFace compatible models are adapted from [M2M100 repository](https://github.com/huggingface/transformers/tree/main/src/transformers/models/m2m_100).
|
|
|
|
| 32 |
> Note: The `install.sh` script in this directory is specifically for running HF compatible models for inference.
|
| 33 |
|
| 34 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
### Models
|
| 36 |
|
| 37 |
| Model | 🤗 HuggingFace Checkpoints |
|
|
|
|
| 61 |
year = {2023},
|
| 62 |
journal = {arXiv preprint arXiv: 2305.16307}
|
| 63 |
}
|
| 64 |
+
```
|