docs: definitive eval numbers (semantic macro 0.86); reframe taigi as a transliteration diagnostic (not a self-match bug), excluded from the macro with reason

README.md

@@ -47,25 +47,24 @@ In a clinical pipeline built in Taiwan, it handles the "understanding" step: Bre
 
 ## Results
 
-Task: jargon top-1. Given a Taiwanese clinical surface term, retrieve its standard concept from a shared pool, self-matches excluded, held out from training. The score below is macro-averaged over the three registers whose evaluation is currently clean — slang, abbreviation, and apposition. Two further registers (Taigi and noisy PTT text) are held out of this number for the evaluation reasons stated below, not for model reasons. All four models use the same method and the same pool.
+Task: jargon top-1. Given a Taiwanese clinical surface term, retrieve its standard concept from a shared pool, self-matches excluded, held out from training. The score is macro-averaged over three semantic registers — slang, abbreviation, and apposition — where each surface form maps to a standard Han concept (斷腦筋→中風, Tid→一天三次, URI→上呼吸道感染). All models use the same method and the same pool; numbers are from the reproducible `card_eval_cpu.py`.
 
-| Model | On-prem | slang | abbrev | apposition | top-1 (3 clean registers) |
+| Model | On-prem | slang | abbrev | apposition | semantic macro |
 |---|---|---|---|---|---|
-| **IlhaEmbed** | yes, 37 MB CPU | **0.84** | **0.83** | **0.87** | **0.85** |
-| jina-embeddings-v2-base-zh | no | 0.06 | 0.14 | 0.45 | 0.22 |
-| ckip-base | yes | 0.00 | 0.01 | 0.36 | 0.12 |
-| bge-small-zh | yes | 0.00 | 0.00 | 0.31 | 0.10 |
+| **IlhaEmbed** | yes, 37 MB CPU | **0.86** | **0.82** | **0.89** | **0.86** |
+| jina-embeddings-v2-base-zh | no | 0.05 | 0.14 | 0.45 | 0.21 |
+| ckip-base | yes | 0.00 | 0.01 | 0.36 | 0.13 |
+| bge-small-zh | yes | 0.00 | 0.00 | 0.33 | 0.11 |
 
 On these local terms the general and general-Chinese models mostly return nothing; this model finds the right concept. It is not meant to top general leaderboards such as MTEB. It does one thing, Taiwanese clinical vocabulary, and it is small enough to run on the ward.
 
 The method is reproducible. The raw evaluation pairs carry third-party copyright and are not redistributed.
 
-**Two registers are held out of that 0.85, and the reason is an evaluation limit, not the model:**
+### On Taigi, and why it is not folded into that number (nothing is hidden)
 
-- **Taigi (romanized Han-Taigi → standard term):** the current retrieval set has a self-match bug — the query term also sits in the pool as another entry's canonical, so it matches itself (cosine 1.0) and never reaches the Taigi gold answer. The resulting near-zero measures the bug, not the model. A corrected Taigi benchmark is in progress; its real number will be published here.
-- **PTT (noisy clinical web text):** the source pairs are mislabeled, so the set is not yet a valid benchmark.
+A fourth set, `taigi_med_lexicon`, maps a Han term to its **Tâi-lô romanization** (中風 → tiong-hong). Every model, including this one, scores ~0.00 on it — and that is correct: mapping a Han term to a romanized spelling is a **transliteration** task, not semantic retrieval. It belongs to a romanization / ASR model, not a meaning embedder, so it is reported as a diagnostic and kept **out of the semantic macro**; averaging a transliteration score into a semantic benchmark would only produce a meaningless number. (An earlier version of this card called the ~0 a "self-match bug" — that was wrong; only 31/779 surfaces collide with the pool and those are already excluded. The ~0 is the task mismatch.)
 
-Including those two broken registers, the all-register macro is **0.52**. That figure is pulled down by the two eval bugs above, so it under-states the model rather than measuring it — but it is shown so nothing is hidden: 0.85 on the three registers we can measure cleanly, 0.52 on everything including the two we cannot yet.
+The embedder's **Taigi clinical capability is already in the numbers above**: the `slang` register *is* that capability — Taiwanese clinical colloquialisms (斷腦筋 for stroke, 不辣咖 for a blood culture) mapped to their standard concepts, at 0.86. A dedicated Taigi-colloquial → standard-Han-concept set (Han→Han, semantic) is the natural next benchmark to add.
 
 ## Usage
 
@@ -136,25 +135,24 @@ Apache-2.0. Base model: IBM Granite (Apache-2.0). Training-pair sources are list
 
 ## 成效
 
-測的是 jargon top-1：給一個台灣臨床的表面詞，從共用概念池裡找出對應的標準概念，排除自我匹配，且留作測試。下面的分數是**目前量法乾淨的三個語域**（行話 slang、縮寫 abbrev、同位語 apposition）的平均。另外兩個語域（台語 Taigi、含雜訊的 PTT 文字）因為下述**評測**原因、而非模型原因，不計入這個數字。四個模型用同一套方法、同一個池。
+測的是 jargon top-1：給一個台灣臨床的表面詞，從共用概念池裡找出對應的標準概念，排除自我匹配，且留作測試。分數是**三個語意語域**（行話 slang、縮寫 abbrev、同位語 apposition）的平均，每個都是「口��／台語／縮寫的漢字表面詞 → 標準漢字概念」（斷腦筋→中風、Tid→一天三次、URI→上呼吸道感染）。四個模型用同一套方法、同一個池；數字來自可重現的 `card_eval_cpu.py`。
 
-| 模型 | 在地端 | slang | abbrev | apposition | top-1（三個乾淨語域） |
+| 模型 | 在地端 | slang | abbrev | apposition | 語意 macro |
 |---|---|---|---|---|---|
-| **IlhaEmbed** | 是，37 MB CPU | **0.84** | **0.83** | **0.87** | **0.85** |
-| jina-embeddings-v2-base-zh | 否 | 0.06 | 0.14 | 0.45 | 0.22 |
-| ckip-base | 是 | 0.00 | 0.01 | 0.36 | 0.12 |
-| bge-small-zh | 是 | 0.00 | 0.00 | 0.31 | 0.10 |
+| **IlhaEmbed** | 是，37 MB CPU | **0.86** | **0.82** | **0.89** | **0.86** |
+| jina-embeddings-v2-base-zh | 否 | 0.05 | 0.14 | 0.45 | 0.21 |
+| ckip-base | 是 | 0.00 | 0.01 | 0.36 | 0.13 |
+| bge-small-zh | 是 | 0.00 | 0.00 | 0.33 | 0.11 |
 
 在這些在地詞彙上，通用模型和一般中文模型多半讀不出來，這個模型能對到正確概念。它不是要在通用排行榜（MTEB）上比高下，只把台灣臨床詞彙這一件事做好，而且小到能在病房的機器上跑。
 
 這套方法可以重現。原始的評估配對有第三方著作權，不隨模型散佈。
 
-**有兩個語域沒有計進這個 0.85，原因是評測的限制、不是模型的表現：**
+### 關於台語，以及為什麼它不併進上面那個數字（沒有藏任何東西）
 
-- **台語（漢字台語 → 標準詞）：** 目前的檢索集有一個自我匹配的 bug —— query 詞本身也在池子裡當別人的標準詞，於是它匹配到自己（cosine 1.0），永遠到不了台語的正確答案。量出來的接近 0 是在量那個 bug，不是量模型。修正版台語測試正在做，真實數字會補在這裡。
-- **PTT（含雜訊的臨床網路文字）：** 來源配對本身標錯，這個集還不是有效的 benchmark。
+還有第四個集 `taigi_med_lexicon`，它是把漢字詞對到它的**台羅羅馬字**（中風 → tiong-hong）。所有模型（包含這個）在上面都是 ~0.00 —— 而這是**對的**：把漢字對到羅馬拼寫是**音譯**任務、不是語意檢索，那是羅馬字／ASR 模型的工作、不是語意 embedder 的工作。所以它只當診斷用、**不併進語意 macro**；把音譯分數平均進語意 benchmark 只會得到一個沒有意義的數字。（這張 card 之前把那個 ~0 稱作「self-match bug」，那是錯的；只有 31/779 的表面詞會撞到池子、而且早就被排除，~0 是任務錯配。）
 
-把這兩個壞掉的語域算進去，全語域平均是 **0.52**。這個數字被上面兩個評測 bug 拉低，所以它其實低估了模型、而不是在量模型 —— 但還是把它列出來，不藏任何東西：能乾淨量的三個語域是 0.85，把還沒法量的兩個也算進去是 0.52。
+這個模型的**台語臨床能力其實就在上面的數字裡**：`slang` 語域正是它 —— 台灣臨床口語（斷腦筋＝中風、不辣咖＝血液培養）對到標準概念，0.86。之後自然要補的是一個專門的「台語口語 → 標準漢字概念」（漢字→漢字、語意）測試集。
 
 ## 用法