Add confusion matrix, probability and F1 charts, and held-out examples to model card
Browse files- README.md +45 -0
- figures/f1_compare.png +0 -0
- figures/prob_hist.png +0 -0
README.md
CHANGED
|
@@ -30,6 +30,51 @@ documented in [`codedlang2026-finetune.md`](https://github.com/nopparatbudd-comm
|
|
| 30 |
This repo's checkpoint is a single deployable model trained the same way, released for
|
| 31 |
reproducibility and demoing, not for citing as the research result.
|
| 32 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
## Limitations
|
| 34 |
|
| 35 |
- Community-labeled dataset (not expert-annotated); community-level knowledge, not
|
|
|
|
| 30 |
This repo's checkpoint is a single deployable model trained the same way, released for
|
| 31 |
reproducibility and demoing, not for citing as the research result.
|
| 32 |
|
| 33 |
+
### Confusion matrix (held-out test, threshold 0.05)
|
| 34 |
+
|
| 35 |
+
| | predicted plain | predicted coded |
|
| 36 |
+
|---|---|---|
|
| 37 |
+
| **actually plain** (668) | 661 | 7 |
|
| 38 |
+
| **actually coded** (95) | 14 | 81 |
|
| 39 |
+
|
| 40 |
+
### How confident is the model?
|
| 41 |
+
|
| 42 |
+

|
| 43 |
+
|
| 44 |
+
Almost every review gets a probability very close to 0 or 1 (note the log scale). The 14 missed
|
| 45 |
+
coded reviews sit at the bottom-left, scored near 0, so moving the threshold would not recover
|
| 46 |
+
them. This is also why the threshold chosen on the dev split ended up at 0.05, the lowest value
|
| 47 |
+
searched. Calibration has not been checked.
|
| 48 |
+
|
| 49 |
+
### Compared with the paper's few-shot LLMs
|
| 50 |
+
|
| 51 |
+

|
| 52 |
+
|
| 53 |
+
Orange bars are this recipe; blue bars are copied from Table 2 of Wan et al. (2026). The
|
| 54 |
+
conditions are **not** the same: the LLMs were prompted few-shot without training, and the paper
|
| 55 |
+
does not state which subset they were scored on, while our model was trained on 80% of the same
|
| 56 |
+
dataset. In the 5-fold study the same recipe caught 92% of reviews whose codes appeared in
|
| 57 |
+
training but only 51% (39 of 77, Wilson 95% CI 0.40–0.62) of reviews whose codes never did, so a
|
| 58 |
+
large part of the gap over the LLMs comes from memorising known codes.
|
| 59 |
+
|
| 60 |
+
### Examples from the held-out test
|
| 61 |
+
|
| 62 |
+
Reviews between 8 and 60 characters, drawn with a fixed seed from each outcome group. English
|
| 63 |
+
glosses are the dataset's own `translated_review` field (machine translation, kept as is). The
|
| 64 |
+
dataset contains profanity disguised as homophones; it is shown unedited.
|
| 65 |
+
|
| 66 |
+
| Outcome | Review (original) | English (from dataset) | Code type | P(coded) |
|
| 67 |
+
|---|---|---|---|---|
|
| 68 |
+
| ✅ coded, caught | 这家店真的难吃,还坑钱,肉都是酸的,服务员态度极差。转发了推广优惠的朋友圈,商家又不承认。醉惹 | This shop is really unpalatable, it's pitted money, the meat is sour, and the waiter's attitude is extremely poor. Reposted the circle of friends that promoted the offer, but the merchant did not recognize it. Drunk | ambiguous_homophone | 0.995 |
|
| 69 |
+
| ✅ coded, caught | 住在索尔万,只有一句话:谁nmb……过来high!驻唱帅哥,新鲜啤酒!热辣妹纸!当然!主要是妹纸😍 | Living in Solvan, there is only one sentence: who nmb... come over high! Singing handsome guy, fresh beer! Hot babes paper! of course! Mainly sister paper | non_lexical_homophone, phonetic | 0.999 |
|
| 70 |
+
| ✅ coded, caught | 垃圾公司,网络不稳定还尼玛卡 | Junk company, the network is unstable and Nimaca | non_lexical_homophone | 1.000 |
|
| 71 |
+
| ✅ plain, correctly passed | 肉太老了!CP值低 | The meat is too old! CP value is low | — | 0.000 |
|
| 72 |
+
| ✅ plain, correctly passed | 牛肉美味 份量大 值得一再前往 | Beef is delicious Worth visiting again and again | — | 0.000 |
|
| 73 |
+
| ❌ coded, missed | 每小時八美金滴佛心停車場 | Eight dollars per hour drop Buddha heart parking lot | ambiguous_homophone | 0.000 |
|
| 74 |
+
| ❌ coded, missed | 所有的外卖都好像只是用微波炉加热了!那奶茶全是粉!感觉在吃shi! | All the takeaways seem to be just heated in a microwave oven! That milk tea is all powder! Feeling eating shi! | phonetic | 0.000 |
|
| 75 |
+
| ❌ plain, wrongly flagged | 感动感动,在ny吃到了好吃的马来菜,强推海南鸡饭。 | Impressed by the movement, I ate a delicious Malay dish in ny and pushed Hainanese chicken rice. | — | 0.994 |
|
| 76 |
+
| ❌ plain, wrongly flagged | 一流的辣鸡汉堡 一流的服务 | Great spicy chicken burger Great service | — | 0.997 |
|
| 77 |
+
|
| 78 |
## Limitations
|
| 79 |
|
| 80 |
- Community-labeled dataset (not expert-annotated); community-level knowledge, not
|
figures/f1_compare.png
ADDED
|
figures/prob_hist.png
ADDED
|