RiskChainBench: A Benchmark for Obfuscated Platform Message Restoration and Evidence-Grounded Web Investigation
Paper • 2609.16900 • Published • 45
shareAI
non-profit
AI & ML interests
None defined yet.
Recent Activity
leonliuzx
submitted a
paper to Daily Papers 4 days ago
leonliuzx
authored a
paper 5 days ago
Post
233
🇹🇼 Releasing https://huggingface.co/lianghsun/tw-tokenizer-v1 — a tokenizer trained from scratch for Traditional Chinese (Taiwan).
**46% better Chinese compression than Qwen3.8-27B with 81% of its vocab (201K vs 248K), and English essentially untouched (4.657 vs 4.674 chars/token).**
The gain isn't from the regex — it's the corpus. Qwen carries **27,364 Simplified-only multi-char tokens**, 11% of its vocab, dead weight for Traditional Chinese. Train on pure Traditional and that waste never appears.
Recent work is skeptical that compression predicts quality (Lotz et al. 2025 measured ρ = −0.59), so we validated two levels deeper:
**Segmentation** — boundary hit rate against jieba: **85.6%** vs Qwen's 77.8%. Single-character tokens: **17.6%** vs 41.7%.
**Downstream** — trained a 270M model from scratch with each tokenizer, compared bits-per-character (the only metric fair across tokenizers). At equal compute: **4.434 vs 4.591**, a 3.4% win — with 13% fewer parameters. Same token budget means our model saw 440M characters vs 308M: **43% more data for the same compute**.
Also: 6-char cap on pure-CJK tokens (long tokens obscure orthographic info — Haslett, CL 2025), NFC not NFKC, 1,024 reserved tokens.
Known limits (weak Tâi-lô support, small-scale downstream validation, vocab sweep hadn't flattened) are in the card.
👉 https://huggingface.co/lianghsun/tw-tokenizer-v1
**46% better Chinese compression than Qwen3.8-27B with 81% of its vocab (201K vs 248K), and English essentially untouched (4.657 vs 4.674 chars/token).**
The gain isn't from the regex — it's the corpus. Qwen carries **27,364 Simplified-only multi-char tokens**, 11% of its vocab, dead weight for Traditional Chinese. Train on pure Traditional and that waste never appears.
Recent work is skeptical that compression predicts quality (Lotz et al. 2025 measured ρ = −0.59), so we validated two levels deeper:
**Segmentation** — boundary hit rate against jieba: **85.6%** vs Qwen's 77.8%. Single-character tokens: **17.6%** vs 41.7%.
專業素養、特質或經公告審查優勝
ours: ['專業素養', '、', '特質', '或經', '公告', '審查', '優勝']
Qwen: ['專業', '素', '養', ...] ← 「素養」split mid-word**Downstream** — trained a 270M model from scratch with each tokenizer, compared bits-per-character (the only metric fair across tokenizers). At equal compute: **4.434 vs 4.591**, a 3.4% win — with 13% fewer parameters. Same token budget means our model saw 440M characters vs 308M: **43% more data for the same compute**.
Also: 6-char cap on pure-CJK tokens (long tokens obscure orthographic info — Haslett, CL 2025), NFC not NFKC, 1,024 reserved tokens.
Known limits (weak Tâi-lô support, small-scale downstream validation, vocab sweep hadn't flattened) are in the card.
👉 https://huggingface.co/lianghsun/tw-tokenizer-v1
Lie24
authored a
paper about 1 month ago
WDong
authored a
paper about 1 month ago
WDong
submitted a
paper to Daily Papers about 1 month ago
sci-m-wang
authored 2
papers about 1 month ago
sci-m-wang
submitted 2
papers to Daily Papers about 1 month ago
sci-m-wang
authored a
paper about 2 months ago
sci-m-wang
submitted a
paper to Daily Papers about 2 months ago
Lie24
authored a
paper 4 months ago
Lie24
authored a
paper 5 months ago
chenzongchao
authored a
paper 6 months ago
OrlandoHugBot
authored a
paper 7 months ago
chenzongchao
authored a
paper 7 months ago
Evanwu50020
authored a
paper 7 months ago
Evanwu50020
submitted a
paper to Daily Papers 7 months ago
Baicai003
updated a
dataset 9 months ago