nopp23 commited on
Commit
6e6698a
·
verified ·
1 Parent(s): a928fce

Add confusion matrix, probability and F1 charts, and held-out examples to model card

Browse files
Files changed (3) hide show
  1. README.md +45 -0
  2. figures/f1_compare.png +0 -0
  3. figures/prob_hist.png +0 -0
README.md CHANGED
@@ -30,6 +30,51 @@ documented in [`codedlang2026-finetune.md`](https://github.com/nopparatbudd-comm
30
  This repo's checkpoint is a single deployable model trained the same way, released for
31
  reproducibility and demoing, not for citing as the research result.
32
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
33
  ## Limitations
34
 
35
  - Community-labeled dataset (not expert-annotated); community-level knowledge, not
 
30
  This repo's checkpoint is a single deployable model trained the same way, released for
31
  reproducibility and demoing, not for citing as the research result.
32
 
33
+ ### Confusion matrix (held-out test, threshold 0.05)
34
+
35
+ | | predicted plain | predicted coded |
36
+ |---|---|---|
37
+ | **actually plain** (668) | 661 | 7 |
38
+ | **actually coded** (95) | 14 | 81 |
39
+
40
+ ### How confident is the model?
41
+
42
+ ![Predicted probability by true label](figures/prob_hist.png)
43
+
44
+ Almost every review gets a probability very close to 0 or 1 (note the log scale). The 14 missed
45
+ coded reviews sit at the bottom-left, scored near 0, so moving the threshold would not recover
46
+ them. This is also why the threshold chosen on the dev split ended up at 0.05, the lowest value
47
+ searched. Calibration has not been checked.
48
+
49
+ ### Compared with the paper's few-shot LLMs
50
+
51
+ ![F1 comparison](figures/f1_compare.png)
52
+
53
+ Orange bars are this recipe; blue bars are copied from Table 2 of Wan et al. (2026). The
54
+ conditions are **not** the same: the LLMs were prompted few-shot without training, and the paper
55
+ does not state which subset they were scored on, while our model was trained on 80% of the same
56
+ dataset. In the 5-fold study the same recipe caught 92% of reviews whose codes appeared in
57
+ training but only 51% (39 of 77, Wilson 95% CI 0.40–0.62) of reviews whose codes never did, so a
58
+ large part of the gap over the LLMs comes from memorising known codes.
59
+
60
+ ### Examples from the held-out test
61
+
62
+ Reviews between 8 and 60 characters, drawn with a fixed seed from each outcome group. English
63
+ glosses are the dataset's own `translated_review` field (machine translation, kept as is). The
64
+ dataset contains profanity disguised as homophones; it is shown unedited.
65
+
66
+ | Outcome | Review (original) | English (from dataset) | Code type | P(coded) |
67
+ |---|---|---|---|---|
68
+ | ✅ coded, caught | 这家店真的难吃,还坑钱,肉都是酸的,服务员态度极差。转发了推广优惠的朋友圈,商家又不承认。醉惹 | This shop is really unpalatable, it's pitted money, the meat is sour, and the waiter's attitude is extremely poor. Reposted the circle of friends that promoted the offer, but the merchant did not recognize it. Drunk | ambiguous_homophone | 0.995 |
69
+ | ✅ coded, caught | 住在索尔万,只有一句话:谁nmb……过来high!驻唱帅哥,新鲜啤酒!热辣妹纸!当然!主要是妹纸😍 | Living in Solvan, there is only one sentence: who nmb... come over high! Singing handsome guy, fresh beer! Hot babes paper! of course! Mainly sister paper | non_lexical_homophone, phonetic | 0.999 |
70
+ | ✅ coded, caught | 垃圾公司,网络不稳定还尼玛卡 | Junk company, the network is unstable and Nimaca | non_lexical_homophone | 1.000 |
71
+ | ✅ plain, correctly passed | 肉太老了!CP值低 | The meat is too old! CP value is low | — | 0.000 |
72
+ | ✅ plain, correctly passed | 牛肉美味 份量大 值得一再前往 | Beef is delicious Worth visiting again and again | — | 0.000 |
73
+ | ❌ coded, missed | 每小時八美金滴佛心停車場 | Eight dollars per hour drop Buddha heart parking lot | ambiguous_homophone | 0.000 |
74
+ | ❌ coded, missed | 所有的外卖都好像只是用微波炉加热了!那奶茶全是粉!感觉在吃shi! | All the takeaways seem to be just heated in a microwave oven! That milk tea is all powder! Feeling eating shi! | phonetic | 0.000 |
75
+ | ❌ plain, wrongly flagged | 感动感动,在ny吃到了好吃的马来菜,强推海南鸡饭。 | Impressed by the movement, I ate a delicious Malay dish in ny and pushed Hainanese chicken rice. | — | 0.994 |
76
+ | ❌ plain, wrongly flagged | 一流的辣鸡汉堡 一流的服务 | Great spicy chicken burger Great service | — | 0.997 |
77
+
78
  ## Limitations
79
 
80
  - Community-labeled dataset (not expert-annotated); community-level knowledge, not
figures/f1_compare.png ADDED
figures/prob_hist.png ADDED