Duplicate from espnet/powsm

Co-authored-by: Chin-Jou Li <cjli@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,121 @@
+---
+datasets:
+- anyspeech/ipapack_plus_train_1
+- anyspeech/ipapack_plus_train_2
+- anyspeech/ipapack_plus_train_3
+- anyspeech/ipapack_plus_train_4
+language: multilingual
+library_name: espnet
+license: cc-by-4.0
+metrics:
+  - pfer
+  - cer
+tags:
+  - espnet
+  - audio
+  - phone-recognition
+  - automatic-speech-recognition
+  - grapheme-to-phoneme
+  - phoneme-to-grapheme
+pipeline_tag: automatic-speech-recognition
+---
+
+### 🐁POWSM
+
+<p align="left">
+  <a href="https://arxiv.org/abs/2510.24992"><img src="https://img.shields.io/badge/Paper-2510.24992-red.svg?logo=arxiv&logoColor=red"/></a>
+  <a href="https://huggingface.co/espnet/powsm"><img src="https://img.shields.io/badge/Model-powsm-yellow.svg?logo=huggingface&logoColor=yellow"/></a>
+  <a href="https://github.com/espnet/egs2/powsm/s2t1"><img src="https://img.shields.io/badge/Recipe-powsm-blue.svg?logo=github&logoColor=black"/></a>
+</p>
+
+POWSM is the first phonetic foundation model that can perform four phone-related tasks:
+Phone Recognition (PR), Automatic Speech Recognition (ASR), audio-guided grapheme-to-phoneme conversion (G2P), and audio-guided phoneme-to-grapheme
+conversion (P2G).
+
+Based on [Open Whisper-style Speech Model (OWSM)](https://www.wavlab.org/activities/2024/owsm/) and trained with [IPAPack++](https://huggingface.co/anyspeech), POWSM outperforms or matches specialized PR models of similar size while jointly supporting G2P, P2G, and ASR.
+
+> [!TIP]
+> Check out our new model: [🐁POWSM-CTC](https://huggingface.co/espnet/powsm_ctc), an encoder-only variant based on OWSM-CTC structure,
+> and [💎PRiSM](https://arxiv.org/abs/2601.14046): Benchmarking Phone Realization in Speech Models! 
+
+
+To use the pre-trained model, please install `espnet` and `espnet_model_zoo`. The requirements are:
+```
+torch
+espnet
+espnet_model_zoo
+```
+
+**The recipe can be found in ESPnet:** https://github.com/espnet/espnet/tree/master/egs2/powsm/s2t1
+
+### Example script for PR/ASR/G2P/P2G
+
+Our models are trained on 16kHz audio with a fixed duration of 20s. When using the pre-trained model, please ensure the input speech is 16kHz and pad or truncate it to 20s.
+
+To distinguish phone entries from BPE tokens that share the same Unicode, we enclose every phone in slashes and treat them as special tokens. For example, /pʰɔsəm/ would be tokenized as /pʰ//ɔ//s//ə//m/. 
+
+
+> [!NOTE]
+> Jan 2026: We release a retrained version with improved ASR text normalization. 
+> It is located in the subfolder `textnorm_retrained` and has the same structure as the main model.
+> Additional details are provided in the updated arXiv appendix.
+
+```python
+from espnet2.bin.s2t_inference import Speech2Text
+import soundfile as sf  # or librosa
+
+task = "<pr>"
+s2t = Speech2Text.from_pretrained(
+    "espnet/powsm",
+    device="cuda",
+    lang_sym="<eng>",   # ISO 639-3; set to <unk> for unseen languages
+    task_sym=task,    # <pr>, <asr>, <g2p>, <p2g>
+)
+
+speech, rate = sf.read("sample.wav")
+prompt = "<na>"         # G2P: set to ASR transcript; P2G: set to phone transcription with slashes
+pred = s2t(speech, text_prev=prompt)[0][0]
+
+# post-processing for better format
+pred = pred.split("<notimestamps>")[1].strip()
+if task == "<pr>" or task == "<g2p>":
+  pred = pred.replace("/", "")
+print(pred)
+```
+
+#### Other tasks
+
+See `force_align.py` in [ESPnet recipe](https://github.com/espnet/espnet/tree/master/egs2/powsm/s2t1) to try out CTC forced alignment with POWSM's encoder!
+
+LID is learned implicitly during training, and you may run it with the script below:
+
+```python
+from espnet2.bin.s2t_inference_language import Speech2Language
+import soundfile as sf      # or librosa
+
+s2t = Speech2Language.from_pretrained(
+    "espnet/powsm",
+    device="cuda",
+    nbest=1,                # number of possible languages to return
+    first_lang_sym="<afr>", # fixed; defined in vocab list
+    last_lang_sym="<zul>"   # fixed; defined in vocab list
+)
+
+speech, rate = sf.read("sample.wav")
+pred = model(speech)[0]     # a list of lang-prob pair
+print(pred)
+```
+
+### Citations
+
+```BibTex
+@article{powsm,
+      title={POWSM: A Phonetic Open Whisper-Style Speech Foundation Model}, 
+      author={Chin-Jou Li and Kalvin Chang and Shikhar Bharadwaj and Eunjung Yeo and Kwanghee Choi and Jian Zhu and David Mortensen and Shinji Watanabe},
+      year={2025},
+      eprint={2510.24992},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2510.24992}, 
+}
+```

data/token_list/bpe_unigram40000/bpe.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9b9a8b76353430d41a1b8f7f2ec0f40fa8c4e75567eaef6887bdbb893c55236a
+size 967858

exp/s2t_stats_raw_bpe40000/train/feats_stats.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f3ca2ef68be502a75a646c8da36847375964a0d6499fd9ee2d7d620a0f31d746
+size 1402

exp/s2t_train_s2t_ebf_conv2d_size768_e9_d9_piecewise_lr5e-4_warmup60k_flashattn_raw_bpe40000/valid.acc.ave_5best.till45epoch.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a91a03bcfd59a956319939891c4098e8f7e8c9ea568d7ec2bcbc1131b32d1197
+size 1374692510

meta.yaml

@@ -0,0 +1,7 @@
+espnet: '202412'
+files:
+  s2t_model_file: exp/s2t_train_s2t_ebf_conv2d_size768_e9_d9_piecewise_lr5e-4_warmup60k_flashattn_raw_bpe40000/valid.acc.ave_5best.till45epoch.pth
+python: 3.11.9
+torch: 2.4.0+cu118
+yaml_files:
+  s2t_train_config: exp/s2t_train_s2t_ebf_conv2d_size768_e9_d9_piecewise_lr5e-4_warmup60k_flashattn_raw_bpe40000/config.yaml

textnorm_retrained/data/token_list/bpe_unigram40000/bpe.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1914ae6ed41df02e174ce16a9976cb96b25b4393b14c61435bff702b829f3799
+size 972584

textnorm_retrained/exp/s2t_stats_raw_bpe40000/train/feats_stats.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e5f80694a59a93aab7beeed44621ba82625a90ac838954c581909a8490cd2244
+size 1402

textnorm_retrained/exp/s2t_train_ctc3_conv2d_size768_e9_d9_mel128_raw_bpe40000/valid.acc.ave_5best.till45epoch.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:515c479555a6b3e46aff40b7254932d22b6213b1b36f8cd19e20984f7b3f9dd0
+size 1374692891

textnorm_retrained/meta.yaml

@@ -0,0 +1,7 @@
+espnet: '202511' # with unmerged local change; will update to suitable version
+files:
+  s2t_model_file: exp/s2t_train_ctc3_conv2d_size768_e9_d9_mel128_raw_bpe40000/valid.acc.ave_5best.till45epoch.pth
+python: 3.12.8
+torch: 2.9.1+cu128
+yaml_files:
+  s2t_train_config: exp/s2t_train_ctc3_conv2d_size768_e9_d9_mel128_raw_bpe40000/config.yaml