PL-BERT (code, models, paper)

.gitattributes

@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+models/hi/fine-tuned-PL_BERT-hindi/step_2500.t7 filter=lfs diff=lfs merge=lfs -text
+models/hi/fine-tuned-PL_BERT-hindi/step_5000.t7 filter=lfs diff=lfs merge=lfs -text
+models/multi/multilingual-pl-bert/step_1100000.t7 filter=lfs diff=lfs merge=lfs -text
+Phoneme-Level[[:space:]]BERT[[:space:]]for[[:space:]]Enhanced[[:space:]]Prosody[[:space:]]of[[:space:]]Text-to-Speech[[:space:]]with[[:space:]]Grapheme[[:space:]]Predictions.pdf filter=lfs diff=lfs merge=lfs -text

Phoneme-Level BERT for Enhanced Prosody of Text-to-Speech with Grapheme Predictions.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:976c96b54742fbc7209bfcacb9e6cb3d1e32d57a582e84f6aab7e8c4eaed5d52
+size 278130

code/PL-BERT.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:02de2d67919d8bd9dd1bd83cf153aead58926d3c6f9b008d14f062779c0c4253
+size 7760040

models/ar/pl-bert (fadi77)/.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

models/ar/pl-bert (fadi77)/README.md

@@ -0,0 +1,107 @@
+---
+# For reference on model card metadata, see the spec: https://github.com/huggingface/hub-docs/blob/main/modelcard.md?plain=1
+# Doc / guide: https://huggingface.co/docs/hub/model-cards
+{}
+---
+
+# Model Card for Arabic PL-BERT Models
+
+This model card describes a collection of three Arabic BERT models trained with different objectives and datasets for phoneme-aware language modeling.
+
+## Model Details
+
+### Model Description
+
+These models are Arabic adaptations of the PL-BERT (Phoneme-aware Language BERT) approach introduced in [Ashby et al. (2023)](https://arxiv.org/pdf/2301.08810). The models incorporate phonemic information to enhance language understanding, with variations in training objectives and data preprocessing.
+
+The collection includes three models:
+- **mlm_p2g_non_diacritics**: Trained with both MLM (Masked Language Modeling) and P2G (Phoneme-to-Grapheme) objectives on non-diacritized Arabic text
+- **mlm_only_non_diacritics**: Trained with only the MLM objective on non-diacritized Arabic text
+- **mlm_only_with_diacritics**: Fine-tuned version of mlm_only_non_diacritics on diacritized Arabic text
+
+**Developed by:** Fadi (GitHub: Fadi987)  
+**Model type:** Transformer-based language models (BERT variants)  
+**Language:** Arabic
+
+### Model Sources
+
+- **Paper (PL-BERT approach):** [Ashby et al. (2023)](https://arxiv.org/pdf/2301.08810)
+
+## Training Details
+
+### Training Data
+
+All models were initially trained on a cleaned version of the Arabic Wikipedia dataset. The dataset is available at [wikipedia.20231101.ar](https://huggingface.co/datasets/wikimedia/wikipedia/tree/main/20231101.ar).
+
+For the **mlm_only_with_diacritics** model, a random sample of 200,000 entries (out of approximately 1.2 million) was selected from the Wikipedia Arabic dataset and fully diacritized using the state-of-the-art CATT diacritizer ([Abjad AI, 2024](https://github.com/abjadai/catt)), introduced in [this paper](https://arxiv.org/abs/2407.03236) and licensed under CC BY-NC 4.0.
+
+### Training Procedure
+
+#### Model Architecture and Objectives
+
+The models follow different training objectives:
+
+1. **mlm_p2g_non_diacritics**: 
+   - Trained with dual objectives similar to the original PL-BERT:
+     - Masked Language Modeling (MLM): Standard BERT pre-training objective
+     - Phoneme-to-Grapheme (P2G): Predicting token IDs from phonemic representations
+   - Tokenization was performed using [aubmindlab/bert-base-arabertv2](https://huggingface.co/aubmindlab/bert-base-arabertv2), which uses subword tokenization
+   - Trained for 10 epochs on non-diacritized Wikipedia Arabic
+
+2. **mlm_only_non_diacritics**:
+   - Trained with only the MLM objective
+   - Removes the P2G objective, which according to ablation studies in the PL-BERT paper minimally affected performance
+   - This removal eliminated dependence on tokenization, which:
+     - Reduced the model size considerably (word/subword tokenization has a much larger vocabulary than phoneme vocabulary)
+     - Allowed phonemization of entire sentences at once, resulting in more accurate phonemization
+   - Trained on non-diacritized Wikipedia Arabic
+
+3. **mlm_only_with_diacritics**:
+   - Fine-tuned version of mlm_only_non_diacritics
+   - Trained for 10 epochs on diacritized Arabic text
+   - Uses the same MLM-only objective
+
+## Technical Considerations
+
+### Tokenization Challenges
+
+For the **mlm_p2g_non_diacritics** model, a notable limitation was the use of subword tokenization. This approach is not ideal for pronunciation modeling because phonemizing parts of words independently loses the context of the word, which heavily affects pronunciation. The authors of the original PL-BERT paper used a word-level tokenizer for English, but a comparable high-quality word-level tokenizer was not available for Arabic. This limitation was addressed in the subsequent models by removing the P2G objective.
+
+### Diacritization
+
+Arabic text can be written with or without diacritics (short vowel marks). The **mlm_only_with_diacritics** model specifically addresses this by training on fully diacritized text, which provides explicit pronunciation information that is typically absent in standard written Arabic.
+
+## Uses
+
+These models can be used for Arabic natural language understanding tasks where phonemic awareness may be beneficial, such as:
+- Text-to-speech
+- Speech recognition post-processing
+- Dialect identification
+- Pronunciation-sensitive applications
+
+For examples on how these models can be used in code, take a look at: https://github.com/Fadi987/StyleTTS2/blob/main/Utils/PLBERT/util.py
+
+## Bias, Risks, and Limitations
+
+The models are trained on Wikipedia data, which may not represent all varieties of Arabic equally. The diacritization process, while state-of-the-art, may introduce some errors or biases in the training data.
+
+The subword tokenization approach used in the mlm_p2g_non_diacritics model has limitations for phonemic modeling as noted above.
+
+## Citation
+
+**BibTeX:**
+```bibtex
+@article{catt2024,
+  title={CATT: Character-based Arabic Tashkeel Transformer},
+  author={Alasmary, Faris and Zaafarani, Orjuwan and Ghannam, Ahmad},
+  journal={arXiv preprint arXiv:2407.03236},
+  year={2024}
+}
+
+@article{plbert2023,
+  title={Phoneme-Level BERT for Enhanced Prosody of Text-to-Speech with Grapheme Predictions},
+  author={Li, Yinghao Aaron and Han, Cong and Jiang, Xilin and Mesgarani, Nima},
+  journal={arXiv preprint arXiv:2301.08810},
+  year={2023}
+}
+```

models/ar/pl-bert (fadi77)/models/mlm_only_non_diacritics/char_indexer.py

@@ -0,0 +1,24 @@
+# IPA Phonemizer: https://github.com/bootphon/phonemizer
+
+import string
+
+PAD = "P"
+PUNCTUATION = ''.join(sorted(set(';:,.!?¡¿—…"«»“”‘’،؛؟٫٬٪﴾﴿ـ' + string.punctuation)))
+LETTERS_IPA = 'ɑɐɒæɓʙβɔɕçɗɖðʤəɘɚɛɜɝɞɟʄɡɠɢʛɦɧħɥʜɨɪʝɭɬɫɮʟɱɯɰŋɳɲɴøɵɸθœɶʘɹɺɾɻʀʁɽʂʃʈʧʉʊʋⱱʌɣɤʍχʎʏʑʐʒʔʡʕʢǀǁǂǃˈˌːˑʼʴʰʱʲʷˠˤ˞↓↑→↗↘̩ᵻ'
+LATIN_LETTERS = 'abcdefghijklmnopqrstuvwxyz'
+PHONEME_MASK = "M"
+PHONEME_SEPARATOR = " "
+UNKNOWN='U'
+
+# Export all symbols:
+symbols = [PAD] + list(PUNCTUATION) + list(LETTERS_IPA) + list(LATIN_LETTERS) + [PHONEME_MASK] + [PHONEME_SEPARATOR] + [UNKNOWN]
+
+assert len(symbols) == len(set(symbols)) # no duplicates
+
+class CharacterIndexer:
+    def __init__(self):
+        self.word_index_dictionary = {symbol: i for i, symbol in enumerate(symbols)}
+
+    def __call__(self, text):
+        return [self.word_index_dictionary[char] if char in self.word_index_dictionary 
+                else self.word_index_dictionary[UNKNOWN] for char in text]

models/ar/pl-bert (fadi77)/models/mlm_only_non_diacritics/config.yml

@@ -0,0 +1,37 @@
+preprocess_params:
+  tokenizer: "aubmindlab/bert-base-arabertv2"
+  num_shards: 10000
+  max_workers: 25
+  max_try_count: 3
+  timeout: 300
+  phonemizer_language: "ar"
+  preprocess_dir: "data/pl_bert"
+  hf_dataset_name: "wikimedia/wikipedia"
+  hf_dataset_split: "20231101.ar"
+  cleaned_output_dir: "wikipedia_20231101.ar.cleaned"
+
+training_params:
+  output_dir: "/pl_bert/checkpoints"
+  mixed_precision: "fp16"
+  batch_size: 96
+  save_interval: 1000
+  log_interval: 10
+  num_process: 1 # number of GPUs
+  num_steps: 1000000
+  learning_rate: 7e-5
+  training_dataset: "fadi77/wikipedia_20231101.ar.phonemized"
+
+dataset_params:
+  word_separator: 87 # token idx used for word separation (W)
+  max_seq_length: 512 # max phoneme sequence length
+  word_pred_prob: 0.15 # probability to select work for prediction
+  phoneme_mask_prob: 0.8 # probability to mask phonemes
+  replace_prob: 0.1 # probablity to replace phonemes
+
+model_params:
+  hidden_size: 768
+  num_attention_heads: 12
+  intermediate_size: 2048
+  max_position_embeddings: 512
+  num_hidden_layers: 12
+  dropout: 0.1

models/ar/pl-bert (fadi77)/models/mlm_only_non_diacritics/model.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:331314efb0d278271bb53d5a33985da8e5a82d73dd41e3e7b4881d96322edc09
+size 72568654

models/ar/pl-bert (fadi77)/models/mlm_only_with_diacritics/char_indexer.py

@@ -0,0 +1,24 @@
+# IPA Phonemizer: https://github.com/bootphon/phonemizer
+
+import string
+
+PAD = "P"
+PUNCTUATION = ''.join(sorted(set(';:,.!?¡¿—…"«»“”‘’،؛؟٫٬٪﴾﴿ـ' + string.punctuation)))
+LETTERS_IPA = 'ɑɐɒæɓʙβɔɕçɗɖðʤəɘɚɛɜɝɞɟʄɡɠɢʛɦɧħɥʜɨɪʝɭɬɫɮʟɱɯɰŋɳɲɴøɵɸθœɶʘɹɺɾɻʀʁɽʂʃʈʧʉʊʋⱱʌɣɤʍχʎʏʑʐʒʔʡʕʢǀǁǂǃˈˌːˑʼʴʰʱʲʷˠˤ˞↓↑→↗↘̩ᵻ'
+LATIN_LETTERS = 'abcdefghijklmnopqrstuvwxyz'
+PHONEME_MASK = "M"
+PHONEME_SEPARATOR = " "
+UNKNOWN='U'
+
+# Export all symbols:
+symbols = [PAD] + list(PUNCTUATION) + list(LETTERS_IPA) + list(LATIN_LETTERS) + [PHONEME_MASK] + [PHONEME_SEPARATOR] + [UNKNOWN]
+
+assert len(symbols) == len(set(symbols)) # no duplicates
+
+class CharacterIndexer:
+    def __init__(self):
+        self.word_index_dictionary = {symbol: i for i, symbol in enumerate(symbols)}
+
+    def __call__(self, text):
+        return [self.word_index_dictionary[char] if char in self.word_index_dictionary 
+                else self.word_index_dictionary[UNKNOWN] for char in text]

models/ar/pl-bert (fadi77)/models/mlm_only_with_diacritics/config.yml

@@ -0,0 +1,39 @@
+preprocess_params:
+  tokenizer: "aubmindlab/bert-base-arabertv2"
+  num_shards: 10000
+  max_workers: 25
+  max_try_count: 3
+  timeout: 300
+  phonemizer_language: "ar"
+  preprocess_dir: "data/pl_bert"
+  hf_dataset_name: "wikimedia/wikipedia"
+  hf_dataset_split: "20231101.ar"
+  cleaned_output_dir: "wikipedia_20231101.ar.cleaned"
+
+training_params:
+  output_dir: "/pl_bert/checkpoints"
+  mixed_precision: "fp16"
+  batch_size: 96
+  save_interval: 1000
+  log_interval: 10
+  num_process: 1 # number of GPUs
+  num_steps: 1000000
+  learning_rate: 7e-5
+  training_dataset: "fadi77/wikipedia_20231101.ar.phonemized"
+  split: "diacritized"
+
+dataset_params:
+  word_separator: 87 # token idx used for word separation (W)
+  max_seq_length: 512 # max phoneme sequence length
+  word_pred_prob: 0.15 # probability to select work for prediction
+  phoneme_mask_prob: 0.8 # probability to mask phonemes
+  replace_prob: 0.1 # probablity to replace phonemes
+
+model_params:
+  pretrained_model: "/pl_bert/checkpoints/modal_phoneme_only_non_diacritics/step_116000.pth"
+  hidden_size: 768
+  num_attention_heads: 12
+  intermediate_size: 2048
+  max_position_embeddings: 512
+  num_hidden_layers: 12
+  dropout: 0.1

models/ar/pl-bert (fadi77)/models/mlm_only_with_diacritics/model.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b32ed12e3a3f5a4d1da88e261c217e36a8f4500d7155fae8324efd6c5bf2803b
+size 72568548

models/ar/pl-bert (fadi77)/models/mlm_p2g_non_diacritics/config.yml

@@ -0,0 +1,36 @@
+preprocess_params:
+    tokenizer: "aubmindlab/bert-base-arabertv2"
+    num_shards: 10000
+    max_workers: 25
+    max_try_count: 3
+    timeout: 300
+    phonemizer_language: "ar"
+    preprocess_dir: "data/pl_bert"
+    hf_dataset_name: "wikimedia/wikipedia"
+    hf_dataset_split: "20231101.ar"
+    output_dir: "wikipedia_20231101.ar.processed"
+
+training_params:
+    output_dir: "checkpoints"
+    mixed_precision: "fp16"
+    batch_size: 16
+    save_interval: 5000
+    log_interval: 10
+    num_process: 1 # number of GPUs
+    num_steps: 1000000
+    learning_rate: 3e-5
+
+dataset_params:
+    word_separator: 87 # token idx used for word separation (W)
+    max_seq_length: 512 # max phoneme sequence length
+    word_pred_prob: 0.15 # probability to select work for prediction
+    phoneme_mask_prob: 0.8 # probability to mask phonemes
+    replace_prob: 0.1 # probablity to replace phonemes
+    
+model_params:
+    hidden_size: 768
+    num_attention_heads: 12
+    intermediate_size: 2048
+    max_position_embeddings: 512
+    num_hidden_layers: 12
+    dropout: 0.1

models/ar/pl-bert (fadi77)/models/mlm_p2g_non_diacritics/model.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:83c612ed18ac2283620496e11a01b95bf1432469ff2348fc58934af8b8fde888
+size 709191470

models/en/mpbert (ydqmkkx)/.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

models/en/mpbert (ydqmkkx)/PhonemeBERT.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a4e22ebf7e9ae759c697ceb8fc837a58bcd3c76dd5a0692504fbdb6c9d2d51cf
+size 747051

models/en/mpbert (ydqmkkx)/README.md

@@ -0,0 +1,11 @@
+---
+license: mit
+language:
+- en
+datasets:
+- bookcorpus
+- wikipedia
+---
+
+# Mixed-Phoneme BERT
+More information: https://github.com/ydqmkkx/PhonemeBERT

models/en/mpbert (ydqmkkx)/config.json

@@ -0,0 +1,28 @@
+{
+  "architectures": [
+    "MpbertForPreTraining"
+  ],
+  "vocab_size": 171,
+  "embedding_size": 768,
+  "hidden_size": 768,
+  "output_size": 768,
+  "intermediate_size": 3072,
+  "max_position_embeddings": 1024,
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "hidden_act": "gelu",
+  "attention_probs_dropout_prob": 0.1,
+  "hidden_dropout_prob": 0.1,
+  "layer_norm_eps": 1e-12,
+  "initializer_range": 0.02,
+  "pad_token_id": 0,
+  "position_biased_input": false,
+  "position_embedding_type": "relative_key_query",
+  "use_sup_phoneme": true,
+  "sup_phoneme_vocab_size": 30000,
+  "grapheme_max_position_embeddings": 1024,
+  "grapheme_vocab_size": 84481,
+  "use_cache": false,
+  "gradient_checkpointing": false,
+  "transformers_version": "4.41.2"
+}

models/en/mpbert (ydqmkkx)/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e913f42be9014e088f50ed797132b0c04115429de10819f5c01ea38b800562d4
+size 444135926

models/en/mpbert (ydqmkkx)/source.txt

@@ -0,0 +1 @@
+https://huggingface.co/ydqmkkx/mpbert

models/en/plbert (ydqmkkx)/.gitattributes

@@ -0,0 +1 @@
+pytorch_model.bin filter=lfs diff=lfs merge=lfs -text

models/en/plbert (ydqmkkx)/PhonemeBERT.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a4e22ebf7e9ae759c697ceb8fc837a58bcd3c76dd5a0692504fbdb6c9d2d51cf
+size 747051

models/en/plbert (ydqmkkx)/README.md

@@ -0,0 +1,11 @@
+---
+license: mit
+language:
+- en
+datasets:
+- bookcorpus
+- wikipedia
+---
+
+# Phoneme-Level BERT
+More information: https://github.com/ydqmkkx/PhonemeBERT

models/en/plbert (ydqmkkx)/config.json

@@ -0,0 +1,28 @@
+{
+  "architectures": [
+    "PlbertForPreTraining"
+  ],
+  "vocab_size": 171,
+  "embedding_size": 768,
+  "hidden_size": 768,
+  "output_size": 768,
+  "intermediate_size": 3072,
+  "max_position_embeddings": 1024,
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "hidden_act": "gelu",
+  "attention_probs_dropout_prob": 0.1,
+  "hidden_dropout_prob": 0.1,
+  "layer_norm_eps": 1e-12,
+  "initializer_range": 0.02,
+  "pad_token_id": 0,
+  "position_biased_input": false,
+  "position_embedding_type": "relative_key_query",
+  "use_sup_phoneme": false,
+  "sup_phoneme_vocab_size": 30000,
+  "grapheme_max_position_embeddings": 1024,
+  "grapheme_vocab_size": 84481,
+  "use_cache": false,
+  "gradient_checkpointing": false,
+  "transformers_version": "4.41.2"
+}

models/en/plbert (ydqmkkx)/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:110f290b2adc0ac9c4994e494f61359ae45d91e1fe299d81ed69f86090d467cc
+size 611719414

models/hi/fine-tuned-PL_BERT-hindi/.gitattributes

@@ -0,0 +1,37 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+step_2500.t7 filter=lfs diff=lfs merge=lfs -text
+step_5000.t7 filter=lfs diff=lfs merge=lfs -text

models/hi/fine-tuned-PL_BERT-hindi/README.md

@@ -0,0 +1,41 @@
+# PL-BERT Fine-Tuned on Hindi Wikipedia Dataset
+
+This model is a fine-tuned version of **PL-BERT**, specifically trained on the Hindi subset of the Wiki40b dataset. The model has been optimized to understand and generate high-quality Hindi text, making it suitable for various NLP tasks in the Hindi language.
+For more information about this model, check out the [GitHub](https://github.com/Ionio-io/PL-BERT-Fine-Tuned-hi-) repository.
+
+## Model Overview
+
+- **Model Name:** PL-BERT (Fine-tuned on Hindi)
+- **Base Model:** PL-BERT (Multilingual BERT variant)
+- **Dataset:** Hindi subset from Wiki40b (51,000 cleaned Wikipedia articles)
+- **Precision:** Mixed precision (FP16)
+
+The fine-tuning process focused on improving the model's ability to handle Hindi text more effectively by leveraging a large, cleaned corpus of Wikipedia articles in Hindi.
+
+## Training Details
+
+- **Model:** PL-BERT
+- **Dataset:** Hindi subset from Wiki40b
+- **Batch Size:** 64
+- **Mixed Precision:** FP16
+- **Optimizer:** AdamW
+- **Training Steps:** 15,000
+
+### Training Progress
+
+- **Final Loss:** 1.879
+- **Vocabulary Loss:** 0.49
+- **Token Loss:** 1.465
+
+### Validation Results
+
+During training, we monitored performance with validation metrics:
+
+- **Validation Loss:** 1.879
+- **Vocabulary Accuracy:** 78.54%
+- **Token Accuracy:** 82.30%
+
+
+---
+license: apache-2.0
+---

models/hi/fine-tuned-PL_BERT-hindi/source.txt

@@ -0,0 +1 @@
+https://huggingface.co/Ionio-ai/fine-tuned-PL_BERT-hindi

models/hi/fine-tuned-PL_BERT-hindi/step_2500.t7

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:32fa987b7a122dbdd18a2669065cc155b0b965cb2791dd31ff671b1bf26063ec
+size 87387147

models/hi/fine-tuned-PL_BERT-hindi/step_5000.t7

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6fd955319e7f7bd1c0bbc81e7819810d50ddd0dcd2ea98fd63ad0d0049ba5807
+size 87387147

models/multi/multilingual-pl-bert/.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+step_1100000.t7 filter=lfs diff=lfs merge=lfs -text

models/multi/multilingual-pl-bert/README.md

@@ -0,0 +1,44 @@
+---
+license: mit
+datasets:
+- styletts2-community/multilingual-phonemes-10k-alpha
+language:
+- fr
+- en
+- es
+- ca
+- de
+- el
+- fa
+- fi
+- pt
+- pl
+- ru
+- sv
+- uk
+- zh
+---
+
+# Multilingual PL-BERT checkpoint
+
+The checkpoint open-sourced here is trained by Papercup using the open-source PL-BERT model found here https://github.com/yl4579/PL-BERT. It is trained to be supported by StyleTTS2, which can be found here: https://github.com/yl4579/StyleTTS2. You can see in the model card the languages that it has been trained on (the languages correspond to the crowdsourced dataset found here https://huggingface.co/datasets/styletts2-community/multilingual-phonemes-10k-alpha).
+
+Notable differences compared to the default PL-BERT checkpoint and config available [here](https://github.com/yl4579/StyleTTS2/tree/main/Utils/PLBERT):
+* Because we are working with many languages, we are using a different tokenizer now: `bert-base-multilingual-cased`.
+* The PL-BERT model was trained on the data obtained from `styletts2-community/multilingual-phonemes-10k-alpha` for 1.1M iterations.
+* The `token_maps.pkl` file has changed (also open-sourced here).
+* We have changed the `util.py` file to deal with an error when loading `new_state_dict["embeddings.position_ids"]`.
+
+## How do I train StyleTTS2 with this new PL-BERT checkpoint?
+
+* Simply create a new folder under `Utils` in your StyleTTS2 repository. Call it, for example, `PLBERT_all_languages`. 
+* Copy paste into it `config.yml`, `step_1100000.t7` and `util.py`.
+* Then, in your StyleTTS2 config file, change `PLBERT_dir` to `Utils/PLBERT_all_languages`. You will also need to change your import as such:
+  * Change `from Utils.PLBERT.util import load_plbert`
+  * To `from Utils.PLBERT_all_languages.util import load_plbert`
+  * Alternatively, you can just replace the relevant files in `Utils/PLBERT` and not have to change any code.
+* Now, you need to create train and validation files. You will need to use `espeak` to create a file in the same format as the ones that exist in the `Data` folder of the StyleTTS2 repository. Careful! You will need to change the `language` argument to phonemise your text if it's not in English. You can find the correct language codes [here](https://github.com/espeak-ng/espeak-ng/blob/master/docs/languages.md). For example, Latin American Spanish is `es-419`
+
+Voila, you can now train a multilingual StyleTTS2 model!
+
+Thank you to Aaron (Yinghao) Li for these contributions.

models/multi/multilingual-pl-bert/config.yml

@@ -0,0 +1,30 @@
+log_dir: "Checkpoint_all_phonemes"
+mixed_precision: "fp16"
+data_folder: "wikipedia_20220301.en.processed"
+batch_size: 32
+save_interval: 20000
+log_interval: 10
+num_process: 1 # number of GPUs
+num_steps: 2000000
+
+dataset_params:
+    tokenizer: "bert-base-multilingual-cased"
+    token_separator: " " # token used for phoneme separator (space)
+    token_mask: "M" # token used for phoneme mask (M)
+    word_separator: 102 # token used for word separator (<formula>)
+    token_maps: "token_maps.pkl" # token map path
+    
+    max_mel_length: 512 # max phoneme length
+    
+    word_mask_prob: 0.15 # probability to mask the entire word
+    phoneme_mask_prob: 0.1 # probability to mask each phoneme
+    replace_prob: 0.2 # probablity to replace phonemes
+    
+model_params:
+    vocab_size: 178
+    hidden_size: 768
+    num_attention_heads: 12
+    intermediate_size: 2048
+    max_position_embeddings: 512
+    num_hidden_layers: 12
+    dropout: 0.1

models/multi/multilingual-pl-bert/issues.txt

@@ -0,0 +1,46 @@
+===========================================================
+[#2] How to inference it?
+===========================================================
+
+jonathansilvasantos, 20.03.2024
+Q: Can someone share an inference script for testing this model independently? I'm exploring how to adapt FastPitch for compatibility with this model or other less computationally demanding text-to-speech (TTS) models.
+
+raph30370 (Papercup AI org), 27.03.2024
+A: I recommend you check out the StyleTTS2 repo on how to do inference with PL-BERT. Specifically in the train_second.py file for example, you can look at plbert = load_plbert(BERT_path) as a starting point in that script.
+
+
+===========================================================
+[#4] Language
+===========================================================
+
+YoussefBarakat, 02.05.2024
+Q: If i'm training a new language does it have to be phonemized with e-speak?
+
+raph30370 (Papercup AI org), 02.05.2024
+A: I think that's how all of these languages here have been phonemized. If you're training everything together, they probably have to be under the same phoneme space, so yeah probably, unless you phonemize everything else with a new phonemizer as well.
+
+
+===========================================================
+[#5] Tokenizer Type
+===========================================================
+
+abhinand, 15.01.2025
+Q: I notice that the vocabulary contains sub-word tokens.
+
+...
+{'word': '##م', 'token': 7030},
+{'word': '##di', 'token': 7033},
+{'word': '##kan', 'token': 7036},
+{'word': '##ek', 'token': 7037},
+{'word': '##ak', 'token': 7040},
+{'word': '##ı', 'token': 7042},
+{'word': '##lo', 'token': 7044},
+{'word': '##ung', 'token': 7045},
+...
+
+But for PL-BERT, isn't it necessary to have 'word-level' tokens? Or am I misinterpreting here?
+
+Another reference thread which discussed this -> https://github.com/yl4579/StyleTTS2/issues/286#issuecomment-2383835836
+
+So does this mean I can use a tokenizer like Indic-BERT to build my token_maps for Indic-PL-BERT??
+

models/multi/multilingual-pl-bert/languages.txt

@@ -0,0 +1,14 @@
+French
+English
+Spanish
+Catalan
+German
+Greek
+Persian
+Finnish
+Portuguese
+Polish
+Russian
+Swedish
+Ukrainian
+Chinese

models/multi/multilingual-pl-bert/source.txt

@@ -0,0 +1 @@
+https://huggingface.co/papercup-ai/multilingual-pl-bert

models/multi/multilingual-pl-bert/step_1100000.t7

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8e920e029e5226284f6042bef36514323c7fd0ae73c59e0ed4ccecd1f2916fd1
+size 25179228

models/multi/multilingual-pl-bert/token_maps.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fdb2b72e90be652f176c44fe4b101b897167bf1cd669ad76e11e9400928cc848
+size 1858692

models/multi/multilingual-pl-bert/util.py

@@ -0,0 +1,45 @@
+import os
+import yaml
+import torch
+from transformers import AlbertConfig, AlbertModel
+
+class CustomAlbert(AlbertModel):
+    def forward(self, *args, **kwargs):
+        # Call the original forward method
+        outputs = super().forward(*args, **kwargs)
+
+        # Only return the last_hidden_state
+        return outputs.last_hidden_state
+
+
+def load_plbert(log_dir):
+    config_path = os.path.join(log_dir, "config.yml")
+    plbert_config = yaml.safe_load(open(config_path))
+    
+    albert_base_configuration = AlbertConfig(**plbert_config['model_params'])
+    bert = CustomAlbert(albert_base_configuration)
+
+    files = os.listdir(log_dir)
+    ckpts = []
+    for f in os.listdir(log_dir):
+        if f.startswith("step_"): ckpts.append(f)
+
+    iters = [int(f.split('_')[-1].split('.')[0]) for f in ckpts if os.path.isfile(os.path.join(log_dir, f))]
+    iters = sorted(iters)[-1]
+
+    checkpoint = torch.load(log_dir + "/step_" + str(iters) + ".t7", map_location='cpu')
+    state_dict = checkpoint['net']
+    from collections import OrderedDict
+    new_state_dict = OrderedDict()
+    for k, v in state_dict.items():
+        name = k[7:] # remove `module.`
+        if name.startswith('encoder.'):
+            name = name[8:] # remove `encoder.`
+            new_state_dict[name] = v
+    try:
+        del new_state_dict["embeddings.position_ids"]
+    except KeyError:
+        pass
+    bert.load_state_dict(new_state_dict, strict=False)
+    
+    return bert