Update Streaming GOPT checkpoint to v6 ASR-confidence model

README.md

@@ -10,14 +10,26 @@ license: other
 
 # custom-gopt-252-eval
 
-这个仓库只做一件事：把本地保存的整套评测必需模型集中起来，并提供一个“输入 1 个音频，直接输出整体评分”的最短用法。
+This repository bundles the models needed for the local evaluation pipeline:
 
-整体评分看 `total`，分值范围是 `0` 到 `5`。
+`Whisper ASR -> Charsiu phone alignment -> Streaming GOPT pronunciation scoring`
 
-## 1. 下载模型包
+The current Streaming GOPT checkpoint is the v6 ASR-confidence version. It uses 47-dimensional phone-segment features: the previous Charsiu-derived acoustic features plus one ASR confidence feature.
 
-```bash
-python - <<'PY'
+## Files
+
+- `streaming_gopt_best/best_audio_model.pth`: best validation Streaming GOPT checkpoint.
+- `streaming_gopt_best/config.json`: model architecture and training arguments.
+- `streaming_gopt_best/inference_assets.json`: normalization statistics and phone-id mapping used by the inference example.
+- `streaming_gopt_best/result.csv`: per-epoch training and validation metrics.
+- `streaming_gopt_best/test_metrics.json`: held-out test metrics for the selected checkpoint.
+- `whisper_best_model/`: Whisper ASR model used by the pipeline.
+- `charsiu_en_w2v2_tiny_fc_10ms/`: Charsiu frame-level phone alignment model.
+- `examples/infer_one_audio.py`: one-audio inference example.
+
+## Download
+
+```python
 from huggingface_hub import snapshot_download
 
 snapshot_download(
@@ -25,18 +37,17 @@ snapshot_download(
     repo_type="model",
     local_dir="./hf_models/custom-gopt-252-eval",
 )
-PY
 ```
 
-下载后约定：
+Then set:
 
 ```bash
 export BUNDLE_DIR=$PWD/hf_models/custom-gopt-252-eval
 ```
 
-## 2. 下载运行代码
+## Code Dependencies
 
-这个模型包不是一个单独的 Transformers 模型；本地推理还要依赖 `custom-gopt` 的模型定义，以及 `Charsiu` 的文本到音素处理代码。
+This model bundle is not a standalone Transformers model. The inference script needs the model definitions from `custom-gopt` and the official Charsiu code.
 
 ```bash
 git clone https://github.com/hf49w/custom-gopt.git
@@ -44,9 +55,7 @@ git clone https://github.com/lingjzhu/charsiu third_party/charsiu_repo
 git -C third_party/charsiu_repo checkout 13a69f2a22ca0c0962b75cc693399b0ae23a12c9
 ```
 
-## 3. 安装最小依赖
-
-在 `custom-gopt` 仓库根目录执行：
+Install the project dependencies in the `custom-gopt` repository, then install the small extra NLTK assets used by Charsiu/G2P:
 
 ```bash
 pip install -r requirements.txt
@@ -54,96 +63,82 @@ python -m pip install nltk
 python -m nltk.downloader cmudict averaged_perceptron_tagger averaged_perceptron_tagger_eng
 ```
 
-如果你已经有可用环境，这一步只要保证下面这些包能导入即可：
-
-- `torch`
-- `transformers`
-- `librosa`
-- `soundfile`
-- `g2p_en`
-- `g2pM`
-- `praatio`
-- `nltk`
-
-## 4. 准备一个待评分音频
-
-要求尽量简单：
-
-- 英语单句或单段短语音频
-- `wav` 最稳妥
-- 单声道更好
-- `16kHz` 最理想，不是 `16kHz` 也会在脚本里自动重采样
-
-假设你的音频路径是：
-
-```bash
-export AUDIO_PATH=/path/to/demo.wav
-```
-
-## 5. 直接跑单音频整体评分
-
-在 `custom-gopt` 仓库根目录执行：
+## One-Audio Inference
 
 ```bash
 python "$BUNDLE_DIR/examples/infer_one_audio.py" \
-  --audio "$AUDIO_PATH" \
+  --audio /path/to/demo.wav \
   --bundle-dir "$BUNDLE_DIR" \
-  --repo-root "$PWD" \
-  --charsiu-src-dir "$PWD/../third_party/charsiu_repo" \
+  --repo-root /path/to/custom-gopt \
+  --charsiu-src-dir /path/to/third_party/charsiu_repo \
   --device cuda \
   --output-json ./one_audio_score.json
 ```
 
-如果你没有 GPU，把 `--device cuda` 改成：
+Use `--device cpu` when CUDA is unavailable.
 
-```bash
---device cpu
-```
+The example accepts an English short utterance audio file. A mono 16 kHz WAV is preferred; other sample rates are resampled inside the script.
+
+## Output Meaning
 
-## 6. 成功运行后会输出什么
+The Streaming GOPT model forward pass returns:
 
-脚本会在终端打印一段 JSON，并在 `--output-json` 指定的位置写出同样内容。
+- `u1`: utterance-level accuracy.
+- `u2`: utterance-level completeness.
+- `u3`: utterance-level fluency.
+- `u4`: utterance-level prosodic score.
+- `u5`: utterance-level total score.
+- `p`: phone-level pronunciation score for each visible phone token.
+- `w1`: word-level accuracy.
+- `w2`: word-level stress.
+- `w3`: word-level total score.
+- `w4`: word-level ASR accuracy.
 
-典型输出结构如下：
+The example script reports the user-facing utterance scores as:
 
 ```json
 {
-  "status": "ok",
-  "audio_path": "/path/to/demo.wav",
-  "transcript": "she had your dark suit in greasy wash water all year",
   "utterance_scores": {
-    "accuracy": 3.91,
-    "completeness": 4.12,
-    "fluency": 3.66,
-    "prosodic": 3.58,
-    "total": 3.82
+    "accuracy": 8.4,
+    "completeness": 10.0,
+    "fluency": 8.3,
+    "prosodic": 7.9,
+    "total": 8.0
   },
-  "overall_score": 3.82
+  "overall_score": 8.0
 }
 ```
 
-其中：
+For `accuracy`, `completeness`, `fluency`, `prosodic`, and `total`, higher is better. These scores follow the SpeechOcean-style pronunciation scoring scale used during training.
 
-- `overall_score` 就是整体评分
-- `overall_score` 和 `utterance_scores.total` 是同一个值
-- 脚本会把最终分数裁到 `0-5` 区间，便于直接使用
+The word-level outputs have this meaning:
 
-## 7. 最短验证命令
+- `word_accuracy`: pronunciation accuracy for the word.
+- `word_stress`: stress score for the word.
+- `word_total`: overall word score.
+- `word_asr_accuracy`: whether the ASR-driven word matched the expected word.
 
-如果你只是想先确认“模型能不能在本地跑通”，可以直接拿 `Charsiu` 仓库自带的示例音频试一下：
+`word_asr_accuracy` is a 0/1-style score. In practice, it can be used as a word-read-correctly indicator: a value near `1` means the word was recognized/matched as read correctly, while a value near `0` means the word was not matched, not recognized correctly, or was not committed yet in the streaming prefix.
 
-```bash
-python "$BUNDLE_DIR/examples/infer_one_audio.py" \
-  --audio "$PWD/../third_party/charsiu_repo/local/SA1.WAV" \
-  --bundle-dir "$BUNDLE_DIR" \
-  --repo-root "$PWD" \
-  --charsiu-src-dir "$PWD/../third_party/charsiu_repo" \
-  --device cpu
-```
+中文提示：词级别的 `asr_accuracy` 评分可以当作“这个词是否读对”的辅助指标。它不替代发音分数本身，但很适合用来标记某个词在 ASR 视角下是否正确读出。
+
+## Test Metrics
+
+Best checkpoint epoch: `11`
+
+Held-out test metrics:
+
+- `phone_test_mse`: `0.049197`
+- `phone_test_pcc`: `0.397571`
+- `utt_test_pcc`: `[0.651909, 0.012690, 0.724998, 0.733279, 0.681940]`
+- `word_test_pcc`: `[0.404714, -0.003912, 0.412258, 0.417467]`
+
+The word PCC order is:
+
+`accuracy`, `stress`, `total`, `asr_accuracy`
 
-## 8. 说明
+## Notes
 
-- 这个脚本走的是：`Whisper -> Charsiu -> Streaming GOPT`
-- 输入 1 个音频，不需要你自己提供文本
-- 模型会先自动识别文本，再对齐音素，最后输出整体评分
-- 训练数据是 SpeechOcean762，所以更适合英语学习者短句朗读场景
+- The model was trained on SpeechOcean762-style English learner speech.
+- The pipeline does not require a reference transcript at inference time; it first obtains a transcript from Whisper, aligns phones with Charsiu, then scores with Streaming GOPT.
+- ASR errors can affect downstream word and phone alignment. Use `word_asr_accuracy` to identify words that the ASR-driven pipeline likely did or did not match correctly.

bundle_manifest.json

@@ -1,6 +1,6 @@
 {
   "repo_id": "faeea/custom-gopt-252-eval",
-  "description": "Bundle of the locally downloaded best validation Streaming GOPT checkpoint plus the Whisper and Charsiu models required by the evaluation pipeline.",
+  "description": "Bundle of the v6 ASR-confidence Streaming GOPT checkpoint plus the Whisper and Charsiu models required by the local evaluation pipeline.",
   "artifacts": [
     {
       "path": "streaming_gopt_best/best_audio_model.pth",
@@ -10,7 +10,7 @@
     {
       "path": "streaming_gopt_best/config.json",
       "type": "streaming_gopt_config",
-      "purpose": "Network shape and training arguments used to restore the model."
+      "purpose": "Network shape and training arguments used to restore the v6 ASR-confidence model."
     },
     {
       "path": "streaming_gopt_best/result.csv",
@@ -22,54 +22,81 @@
       "type": "test_metrics",
       "purpose": "Held-out test metrics for the best validation checkpoint."
     },
+    {
+      "path": "streaming_gopt_best/inference_assets.json",
+      "type": "inference_metadata",
+      "purpose": "Normalization stats, feature dimension, and phone-id mapping required for one-audio local inference."
+    },
     {
       "path": "whisper_best_model",
       "type": "transformers_whisper_model",
-      "purpose": "ASR model used to build streaming ASR-driven GOPT chunks."
+      "purpose": "ASR model used to build ASR-driven GOPT chunks."
     },
     {
       "path": "charsiu_en_w2v2_tiny_fc_10ms",
       "type": "charsiu_aligner_model",
-      "purpose": "Frame-level phone alignment model used by preprocessing."
-    },
-    {
-      "path": "examples/eval_streaming_gopt_test.py",
-      "type": "example_script",
-      "purpose": "Minimal evaluation script for val/test split using the bundled best GOPT checkpoint."
+      "purpose": "Frame-level phone alignment model used by preprocessing and inference."
     },
     {
       "path": "examples/infer_one_audio.py",
       "type": "example_script",
-      "purpose": "Minimal one-audio local inference script that prints the overall utterance score."
+      "purpose": "Minimal one-audio local inference script that prints utterance-level scores."
     },
     {
-      "path": "streaming_gopt_best/inference_assets.json",
-      "type": "inference_metadata",
-      "purpose": "Normalization stats and phone-id mapping required for one-audio local inference."
+      "path": "examples/eval_streaming_gopt_test.py",
+      "type": "example_script",
+      "purpose": "Minimal evaluation script for val/test split using the bundled best GOPT checkpoint."
     }
   ],
+  "model_outputs": {
+    "utterance": [
+      "accuracy",
+      "completeness",
+      "fluency",
+      "prosodic",
+      "total"
+    ],
+    "phone": [
+      "phone pronunciation score"
+    ],
+    "word": [
+      "accuracy",
+      "stress",
+      "total",
+      "asr_accuracy"
+    ],
+    "word_asr_accuracy_note": "The word-level ASR accuracy score can be used as an auxiliary indicator of whether the word was read correctly."
+  },
   "best_validation_summary": {
     "selection_metric": "phone_val_mse",
-    "best_epoch": 15,
-    "phone_val_mse": 0.04897475987672806,
-    "phone_val_pcc": 0.24482591936290432,
-    "utt_val_pcc_total": 0.6696266195414817,
-    "word_val_pcc_total": 0.24699408632096473
+    "best_epoch": 11,
+    "phone_val_mse": 0.04760764539241791,
+    "phone_val_pcc": 0.34831968338670993,
+    "utt_val_pcc_total": 0.661865583803676,
+    "word_val_pcc_total": 0.3418123225833971,
+    "word_val_pcc_asr_accuracy": 0.3633849619530143
   },
   "test_summary": {
-    "phone_test_mse": 0.04749840870499611,
-    "phone_test_pcc": 0.3332444625995981,
+    "phone_test_mse": 0.04919730871915817,
+    "phone_test_pcc": 0.3975705055993251,
     "utt_test_pcc": [
-      0.6184778675115561,
-      -0.005628494483717365,
-      0.7233702728305461,
-      0.7387418272039076,
-      0.6823243104620896
+      0.651909193610124,
+      0.012690043750859805,
+      0.7249977587357453,
+      0.7332794084754206,
+      0.681940037784353
     ],
     "word_test_pcc": [
-      0.3099214468824142,
-      0.022433912396827224,
-      0.321762854413528
+      0.4047142491023989,
+      -0.0039119825110344765,
+      0.4122583381095728,
+      0.41746734262039764
+    ],
+    "word_score_names": [
+      "accuracy",
+      "stress",
+      "total",
+      "asr_accuracy"
     ]
   },
   "code_dependencies": {

examples/infer_one_audio.py

@@ -296,7 +296,7 @@ def predict_scores(model, x, p, main_context_tokens, right_context_tokens):
             right_context_tokens=int(right_context_tokens),
         )
     values = torch.cat([u1, u2, u3, u4, u5], dim=1).squeeze(0).cpu().numpy() * 5.0
-    values = np.clip(values, 0.0, 5.0)
+    values = np.clip(values, 0.0, 10.0)
     return {name: float(value) for name, value in zip(SCORE_NAMES, values.tolist())}
 
 

streaming_gopt_best/best_audio_model.pth

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c86a67b3157d95635e5aebaa10e4a8e45eacf5b052201a2444a7ec61d26b8a20
-size 170812
+oid sha256:d3ab226af75d1dfd7dd71cbcf512a9441aa1200ba2079d34470f81159cd6b977
+size 190932

streaming_gopt_best/config.json

@@ -1,11 +1,11 @@
 {
-  "data_dir": "/DATA_2/guest/custom-gopt/data/streaming_asr_gopt_v4",
-  "input_dim": 46,
+  "data_dir": "/DATA_2/guest/custom-gopt/data/streaming_asr_gopt_v6_asrconf",
+  "input_dim": 47,
   "phn_num": 40,
-  "seq_len": 499,
+  "seq_len": 692,
   "args": {
-    "data_dir": "/DATA_2/guest/custom-gopt/data/streaming_asr_gopt_v4",
-    "exp_dir": "/DATA_2/guest/custom-gopt/exp/streaming-asr-gopt",
+    "data_dir": "/DATA_2/guest/custom-gopt/data/streaming_asr_gopt_v6_asrconf",
+    "exp_dir": "/DATA_2/guest/custom-gopt/exp/streaming-asr-gopt-v6-asrconf",
     "lr": 0.001,
     "n_epochs": 100,
     "depth": 3,
@@ -14,15 +14,16 @@
     "embed_dim": 24,
     "loss_w_phn": 1.0,
     "loss_w_word": 1.0,
+    "loss_w_word_asr": 1.0,
     "loss_w_utt": 1.0,
     "model": "streaming_gopt",
     "noise": 0.0,
-    "num_workers": 8,
+    "num_workers": 4,
     "device": "cuda",
     "main_context_tokens": "4,8,12,16",
     "right_context_tokens": "0,1,2,4",
-    "compile": true,
-    "tf32": true,
+    "compile": false,
+    "tf32": false,
     "seed": 1337,
     "resume": false,
     "main_context_token_choices": [

streaming_gopt_best/inference_assets.json

@@ -1,8 +1,8 @@
 {
   "sample_rate": 16000,
-  "train_norm_mean": 0.04211217910051346,
-  "train_norm_std": 0.23447780311107635,
-  "feat_dim": 46,
+  "train_norm_mean": 0.0722440779209137,
+  "train_norm_std": 0.25949764251708984,
+  "feat_dim": 47,
   "phn_num": 40,
   "phn_dict": {
     "W": 0,

streaming_gopt_best/test_metrics.json

@@ -1,29 +1,37 @@
 {
-  "phone_test_mse": 0.04749840870499611,
-  "phone_test_pcc": 0.3332444625995981,
+  "phone_test_mse": 0.04919730871915817,
+  "phone_test_pcc": 0.3975705055993251,
   "utt_test_mse": [
-    0.060593076050281525,
-    0.010257643647491932,
-    0.037500832229852676,
-    0.03365592285990715,
-    0.05141453817486763
+    0.051985688507556915,
+    0.0016903950599953532,
+    0.03446277603507042,
+    0.03239584341645241,
+    0.04794545844197273
   ],
   "utt_test_pcc": [
-    0.6184778675115561,
-    -0.005628494483717365,
-    0.7233702728305461,
-    0.7387418272039076,
-    0.6823243104620896
+    0.651909193610124,
+    0.012690043750859805,
+    0.7249977587357453,
+    0.7332794084754206,
+    0.681940037784353
   ],
   "word_test_mse": [
-    0.05728999897837639,
-    0.0032619887497276068,
-    0.04410763829946518
+    0.06279397755861282,
+    0.0037991644348949194,
+    0.04641091078519821,
+    0.14399507641792297
   ],
   "word_test_pcc": [
-    0.3099214468824142,
-    0.022433912396827224,
-    0.321762854413528
+    0.4047142491023989,
+    -0.0039119825110344765,
+    0.4122583381095728,
+    0.41746734262039764
   ],
-  "best_epoch": 15
+  "word_score_names": [
+    "accuracy",
+    "stress",
+    "total",
+    "asr_accuracy"
+  ],
+  "best_epoch": 11
 }