docs: honest metrics correction (apposition 0.87 not 0.90; disclose taigi self-match eval bug and PTT noise; show full 0.52 all-register macro)

README.md

@@ -47,18 +47,25 @@ In a clinical pipeline built in Taiwan, it handles the "understanding" step: Bre
 
 ## Results
 
-Task: jargon top-1. Given a Taiwanese clinical surface term, retrieve its standard concept from a combined pool. Slang, abbreviation, and apposition are each held out separately, self-matches excluded, scores macro-averaged. All four models use the same method and the same pool.
+Task: jargon top-1. Given a Taiwanese clinical surface term, retrieve its standard concept from a shared pool, self-matches excluded, held out from training. The score below is macro-averaged over the three registers whose evaluation is currently clean — slang, abbreviation, and apposition. Two further registers (Taigi and noisy PTT text) are held out of this number for the evaluation reasons stated below, not for model reasons. All four models use the same method and the same pool.
 
-| Model | On-prem | slang | abbrev | apposition | jargon top-1 (macro) |
+| Model | On-prem | slang | abbrev | apposition | top-1 (3 clean registers) |
 |---|---|---|---|---|---|
-| **IlhaEmbed** | yes, 37 MB CPU | **0.84** | **0.82** | **0.90** | **0.85** |
+| **IlhaEmbed** | yes, 37 MB CPU | **0.84** | **0.83** | **0.87** | **0.85** |
 | jina-embeddings-v2-base-zh | no | 0.06 | 0.14 | 0.45 | 0.22 |
 | ckip-base | yes | 0.00 | 0.01 | 0.36 | 0.12 |
-| bge-small-zh | yes | 0.00 | 0.00 | 0.33 | 0.11 |
+| bge-small-zh | yes | 0.00 | 0.00 | 0.31 | 0.10 |
 
 On these local terms the general and general-Chinese models mostly return nothing; this model finds the right concept. It is not meant to top general leaderboards such as MTEB. It does one thing, Taiwanese clinical vocabulary, and it is small enough to run on the ward.
 
-The method is reproducible. The raw evaluation pairs carry third-party copyright and are not redistributed. A separate Han-Taigi to standard-concept benchmark is still in progress, and will be added once its concept mapping is clean.
+The method is reproducible. The raw evaluation pairs carry third-party copyright and are not redistributed.
+
+**Two registers are held out of that 0.85, and the reason is an evaluation limit, not the model:**
+
+- **Taigi (romanized Han-Taigi → standard term):** the current retrieval set has a self-match bug — the query term also sits in the pool as another entry's canonical, so it matches itself (cosine 1.0) and never reaches the Taigi gold answer. The resulting near-zero measures the bug, not the model. A corrected Taigi benchmark is in progress; its real number will be published here.
+- **PTT (noisy clinical web text):** the source pairs are mislabeled, so the set is not yet a valid benchmark.
+
+Including those two broken registers, the all-register macro is **0.52**. That figure is pulled down by the two eval bugs above, so it under-states the model rather than measuring it — but it is shown so nothing is hidden: 0.85 on the three registers we can measure cleanly, 0.52 on everything including the two we cannot yet.
 
 ## Usage
 
@@ -129,18 +136,25 @@ Apache-2.0. Base model: IBM Granite (Apache-2.0). Training-pair sources are list
 
 ## 成效
 
-測的是 jargon top-1：給一個台灣臨床的表面詞，從概念池裡找出對應的標準概念。行話、縮寫、同位語三類各自留作測試，排除自我匹配，取各類平均。四個模型用同一套方法、同一個池。
+測的是 jargon top-1：給一個台灣臨床的表面詞，從共用概念池裡找出對應的標準概念，排除自我匹配，且留作測試。下面的分數是**目前量法乾淨的三個語域**（行話 slang、縮寫 abbrev、同位語 apposition）的平均。另外兩個語域（台語 Taigi、含雜訊的 PTT 文字）因為下述**評測**原因、而非模型原因，不計入這個數字。四個模型用同一套方法、同一個池。
 
-| 模型 | 在地端 | slang | abbrev | apposition | jargon top-1（macro） |
+| 模型 | 在地端 | slang | abbrev | apposition | top-1（三個乾淨語域） |
 |---|---|---|---|---|---|
-| **IlhaEmbed** | 是，37 MB CPU | **0.84** | **0.82** | **0.90** | **0.85** |
+| **IlhaEmbed** | 是，37 MB CPU | **0.84** | **0.83** | **0.87** | **0.85** |
 | jina-embeddings-v2-base-zh | 否 | 0.06 | 0.14 | 0.45 | 0.22 |
 | ckip-base | 是 | 0.00 | 0.01 | 0.36 | 0.12 |
-| bge-small-zh | 是 | 0.00 | 0.00 | 0.33 | 0.11 |
+| bge-small-zh | 是 | 0.00 | 0.00 | 0.31 | 0.10 |
 
 在這些在地詞彙上，通用模型和一般中文模型多半讀不出來，這個模型能對到正確概念。它不是要在通用排行榜（MTEB）上比高下，只把台灣臨床詞彙這一件事做好，而且小到能在病房的機器上跑。
 
-這套方法可以重現。原始的評估配對有第三方著作權，不隨模型散佈。另外一個「漢字台語對到標準概念」的測試還在做，等對應整理乾淨再補上。
+這套方法可以重現。原始的評估配對有第三方著作權，不隨模型散佈。
+
+**有兩個語域沒有計進這個 0.85，原因是評測的限制、不是模型的表現：**
+
+- **台語（漢字台語 → 標準詞）：** 目前的檢索集有一個自我匹配的 bug —— query 詞本身也在池子裡當別人的標準詞，於是它匹配到自己（cosine 1.0），永遠到不了台語的正確答案。量出來的接近 0 是在量那個 bug，不是量模型。修正版台語測試正在做，真實數字會補在這裡。
+- **PTT（含雜訊的臨床網路文字）：** 來源配對本身標錯，這個集還不是有效的 benchmark。
+
+把這兩個壞掉的語域算進去，全語域平均是 **0.52**。這個數字被上面兩個評測 bug 拉低，所以它其實低估了模型、而不是在量模型 —— 但還是把它列出來，不藏任何東西：能乾淨量的三個語域是 0.85，把還沒法量的兩個也算進去是 0.52。
 
 ## 用法