Blame for README.md

---
license: cc-by-4.0
language: zh
tags:
- text-classification
- coded-language
base_model: hfl/chinese-roberta-wwm-ext
---

# CodedLang coded-language detector (hfl/chinese-roberta-wwm-ext fine-tune)

Binary classifier: does this Chinese Google-Maps-style review contain coded/evasive language
(homophone substitution, orthographic tricks, cross-lingual phonetic spelling, cipher, etc.)?

Fine-tuned on **6208** reviews from the CodedLang dataset
([Wan, Li & Huang, ACL 2026](https://doi.org/10.18653/v1/2026.acl-long.426),
[RuyuanWan/CodedLang](https://huggingface.co/datasets/RuyuanWan/CodedLang), CC BY 4.0).

**Try it in your browser:** [nopp23/codedlang-detector-demo](https://huggingface.co/spaces/nopp23/codedlang-detector-demo) (runs locally via Transformers.js, nothing is sent to a server). The demo loads `onnx/model_quantized.onnx`, a per-channel 8-bit copy; on the held-out test it scored F1 0.891 vs 0.885 for the full model, with 1 of 763 reviews decided differently. `onnx/model.onnx` is the full-precision export and matches PyTorch exactly on the same split.

## Held-out test metrics (this checkpoint, 763 reviews never seen during training)

- precision: 0.92
- recall: 0.853
- f1: 0.885
- decision threshold: 0.05

**This is a single train/dev/test split, not the headline number.** The thesis's main result
(pooled out-of-fold F1 0.894 across a 5-fold cross-validation, exceeding the paper's best
few-shot LLM baseline of 0.78) comes from a *different* study with 5 separately-trained models,
documented in [`codedlang2026-finetune.md`](https://github.com/nopparatbudd-commits/Research/blob/main/07-Tools/poc/thai-concealed-topic/replication/codedlang2026-finetune.md).
This repo's checkpoint is a single deployable model trained the same way, released for
reproducibility and demoing, not for citing as the research result.

### Confusion matrix (held-out test, threshold 0.05)

| | predicted plain | predicted coded |
|---|---|---|
| **actually plain** (668) | 661 | 7 |
| **actually coded** (95) | 14 | 81 |

### How confident is the model?

![Predicted probability by true label](figures/prob_hist.png)

Almost every review gets a probability very close to 0 or 1 (note the log scale). The 14 missed
coded reviews sit at the bottom-left, scored near 0, so moving the threshold would not recover
them. This is also why the threshold chosen on the dev split ended up at 0.05, the lowest value
searched. Calibration has not been checked.

### Compared with the paper's few-shot LLMs

![F1 comparison](figures/f1_compare.png)

Orange bars are this recipe; blue bars are copied from Table 2 of Wan et al. (2026). The
conditions are **not** the same: the LLMs were prompted few-shot without training, and the paper
does not state which subset they were scored on, while our model was trained on 80% of the same
dataset. In the 5-fold study the same recipe caught 92% of reviews whose codes appeared in
training but only 51% (39 of 77, Wilson 95% CI 0.40–0.62) of reviews whose codes never did, so a
large part of the gap over the LLMs comes from memorising known codes.

### Examples from the held-out test

Reviews between 8 and 60 characters, drawn with a fixed seed from each outcome group. English
glosses are the dataset's own `translated_review` field (machine translation, kept as is). The
dataset contains profanity disguised as homophones; it is shown unedited.

| Outcome | Review (original) | English (from dataset) | Code type | P(coded) |
|---|---|---|---|---|
| ✅ coded, caught | 这家店真的难吃，还坑钱，肉都是酸的，服务员态度极差。转发了推广优惠的朋友圈，商家又不承认。醉惹 | This shop is really unpalatable, it's pitted money, the meat is sour, and the waiter's attitude is extremely poor. Reposted the circle of friends that promoted the offer, but the merchant did not recognize it. Drunk | ambiguous_homophone | 0.995 |
| ✅ coded, caught | 住在索尔万，只有一句话：谁nmb……过来high！驻唱帅哥，新鲜啤酒！热辣妹纸！当然！主要是妹纸😍 | Living in Solvan, there is only one sentence: who nmb... come over high! Singing handsome guy, fresh beer! Hot babes paper! of course! Mainly sister paper | non_lexical_homophone, phonetic | 0.999 |
| ✅ coded, caught | 垃圾公司，网络不稳定还尼玛卡 | Junk company, the network is unstable and Nimaca | non_lexical_homophone | 1.000 |
| ✅ plain, correctly passed | 肉太老了！CP值低 | The meat is too old! CP value is low | — | 0.000 |
| ✅ plain, correctly passed | 牛肉美味 份量大 值得一再前往 | Beef is delicious Worth visiting again and again | — | 0.000 |
| ❌ coded, missed | 每小時八美金滴佛心停車場 | Eight dollars per hour drop Buddha heart parking lot | ambiguous_homophone | 0.000 |
| ❌ coded, missed | 所有的外卖都好像只是用微波炉加热了！那奶茶全是粉！感觉在吃shi！ | All the takeaways seem to be just heated in a microwave oven! That milk tea is all powder! Feeling eating shi! | phonetic | 0.000 |
| ❌ plain, wrongly flagged | 感动感动,在ny吃到了好吃的马来菜，强推海南鸡饭。 | Impressed by the movement, I ate a delicious Malay dish in ny and pushed Hainanese chicken rice. | — | 0.994 |
| ❌ plain, wrongly flagged | 一流的辣鸡汉堡 一流的服务 | Great spicy chicken burger Great service | — | 0.997 |

## Limitations

- Community-labeled dataset (not expert-annotated); community-level knowledge, not
  conversation-specific codes.
- English/Chinese Google Maps reviews, not spoken dialogue or Thai text.
- Hyperparameters mostly copied from the 5-fold study, except batch size (4, not 16) and
  max sequence length (256, not 512) — reduced to fit this machine's GPU memory. 92% of
  reviews are under 256 characters, so truncation loss should be small, but this was not
  independently re-tuned for the new split.
- Single seed; variance across seeds not measured.

## Usage

```python
from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("nopp23/codedlang-coded-language-detector")
model = AutoModelForSequenceClassification.from_pretrained("nopp23/codedlang-coded-language-detector")
inputs = tok("your review text here", return_tensors="pt", truncation=True, max_length=512)
with torch.no_grad():
    prob = torch.softmax(model(**inputs).logits, -1)[0, 1].item()
print("coded-language probability:", prob)
```