org card: both benchmarks, the paper, and the box basis

README.md

@@ -7,11 +7,15 @@ sdk: static
 pinned: true
 ---
 
+<p align="center">
+  <img src="https://huggingface.co/datasets/OpenTWBench/tw-exam-bench/resolve/main/assets/banner.png" alt="OpenTWBench" width="100%">
+</p>
+
 # OpenTWBench
 
 **An open evaluation suite for Taiwan-domain knowledge in large language models.**
 
-🔗 **[opentwbench.ai](https://opentwbench.ai)** — leaderboard, subject map, and how the data is built
+📄 **[arXiv:2610.19848](https://arxiv.org/abs/2610.19848)** · 🏆 **[opentwbench.ai](https://opentwbench.ai)** — 108 models, per-subject scores, raw predictions · 📚 **[Collection](https://huggingface.co/collections/OpenTWBench/opentwbench-6ad04707cb0fd0973d14e067)**
 
 ---
 
@@ -20,39 +24,101 @@ Chinese, about mainland institutions. A model can score well on them while knowi
 nothing about the Labor Standards Act, the Household Registration Act, or how a
 Taiwanese pharmacist is licensed. These datasets exist to make that gap measurable.
 
-The questions come from Taiwan's **national examinations**, published as open data by
-the **Ministry of Examination (考選部)** — professionally written, officially
-answer-keyed, and covering essentially every regulated profession in the country.
+## The two benchmarks
 
-## The datasets
+| | Questions | Source | Licence |
+|---|---|---|---|
+| [**`tw-exam-bench`**](https://huggingface.co/datasets/OpenTWBench/tw-exam-bench) | **255,854** across 148 subjects | Fifteen years of Taiwan's national examinations, published as open data by the Ministry of Examination (考選部) | Apache-2.0 |
+| [**`tw-formosa-bench`**](https://huggingface.co/datasets/OpenTWBench/tw-formosa-bench) | **1,696**, difficulty-curated | [Taiwan.md](https://github.com/frank890417/taiwan-md), a community-curated corpus of everyday and contemporary Taiwan | CC BY-SA 4.0 |
 
-One dataset per **學科 (academic subject)**, named `tw-<subject>-bench`, in the
-**Twinkle Eval MCQ** format (`question`, `A`–`D`, `answer`) with provenance columns
-alongside. Subjects are the unit, not the profession sitting the exam:
-神經疾病物理治療學 and 骨科疾病物理治療學 are separate benchmarks, because a model can
-know one and not the other.
+They measure different things on purpose. `tw-exam-bench` is the professional
+knowledge Taiwan licenses: 神經疾病物理治療學 and 骨科疾病物理治療學 are separate
+configs, because a model can know one and not the other. `tw-formosa-bench` is the
+Taiwan those examinations never ask about — 蔣渭水's《臨床講義》, who designed 衛武營,
+the MRT rules, the night markets — curated so that a frozen seven-model pool scored
+at the passing margin rather than at the ceiling.
 
 ```python
 from datasets import load_dataset
-ds = load_dataset("OpenTWBench/tw-medicine-bench", split="test")
+
+ds = load_dataset("OpenTWBench/tw-exam-bench", "dentistry", split="test")
+ds = load_dataset("OpenTWBench/tw-exam-bench", "field_medicine", split="test")
+ds = load_dataset("OpenTWBench/tw-exam-bench", "all", split="test")
+ds = load_dataset("OpenTWBench/tw-formosa-bench", "all", split="test")
 ```
 
+Every exam item keeps its `paper_id` and `q_no`, so any score traces back to the
+original PDF; every Formosa item keeps the `source_title` and `source_path` of the
+Taiwan.md passage it was written from.
+
+## What the leaderboard reports
+
+Every score on [opentwbench.ai](https://opentwbench.ai) is a **`\boxed{}` score**: the
+model is told to put the answer letter in `\boxed{}`, and a response that does not is
+counted wrong, not pattern-matched into one. The lenient score — what pattern matching
+would have added — sits beside it in the same row, so *what a model knows* and *what
+format it writes in* are two columns rather than one.
+
+This matters more than it sounds. On identical questions and models, switching
+extraction from pattern matching to `\boxed{}` moves accuracy from 20.8% to 93.9%.
+A board that mixes the two is not one measurement.
+
 ## Reading the scores honestly
 
 - These are **public past papers**, very likely in pretraining corpora. Use the suite
   for *relative* comparison between models, not to claim a model "passes" an exam.
-- **Check the parse rate before believing a low score** — a model that knows the
-  answer but writes it in an unexpected shape is a formatting failure.
-- **Shuffle the options.** The source papers carry an answer-position skew.
+- **Check the `\boxed{}` rate before believing a low score.** A model that knows the
+  answer but will not write it in the requested shape is a formatting failure, and the
+  board shows that rate next to every number.
+- **Shuffle the options.** The source papers carry an answer-position skew, and one
+  placeholder character in a system prompt once moved a 675B model from last place to
+  first on the same questions.
+- Answer keys agree across 31,312 duplicated question groups at **99.91%**; all 27
+  disagreements are accounted for item by item. That figure is the noise floor of
+  anything built on this source — differences finer than it are measuring the
+  apparatus, not the model.
 
 Taiwan **law** is covered separately and in depth by
 [`lianghsun/tw-legal-benchmark-v2`](https://huggingface.co/datasets/lianghsun/tw-legal-benchmark-v2);
 questions already published there are excluded from this org, item by item.
 
+## Getting your model on the leaderboard
+
+**We can run it for you.** Publish the model on Hugging Face — gated or private is
+fine — grant read access to **OpenTWBench**, and open an issue here with the repo id.
+We serve it, run the full suite, and publish the result tagged **Official** alongside
+the raw prediction files.
+
+You can also run the suite yourself and submit the numbers; those appear tagged
+**Self-reported**. Every leaderboard row carries its tag, so a reader can tell who
+produced the score. See [opentwbench.ai](https://opentwbench.ai) for both routes.
+
+The suite is not limited to what we build. Any benchmark that measures Taiwan in
+Traditional Chinese belongs here, whatever its source. Contributed datasets keep their
+own home and their own maintainers; we list them, tagged **Community**, after review.
+Send us a link from [opentwbench.ai](https://opentwbench.ai/#contribute).
+
 ## Contributing
 
 A wrong answer key is worse than a missing question. Open a discussion on the dataset
 it affects, quoting the `paper_id` and `q_no`, and it can be traced to the original PDF.
 
-*Questions and answer keys are official publications of 考選部, released as open data.
-Packaging and metadata: Apache-2.0.*
+## Citation
+
+```bibtex
+@misc{opentwbench2026,
+  title        = {OpenTWBench (Open Taiwan Benchmark): Taiwan-Domain Knowledge
+                  Evaluation for Large Language Models, and How the Measurement
+                  Apparatus Shapes the Score},
+  author       = {Huang, Liang-Hsun and Chen, Min-Yi and Chuang, Chao-Chun and
+                  Liu, Yu-Wei and Wu, Che-Yu and Lee, Po-Feng},
+  year         = {2026},
+  eprint       = {2610.19848},
+  archivePrefix= {arXiv},
+  primaryClass = {cs.CL}
+}
+```
+
+*Examination questions and answer keys are official publications of 考選部, released as
+open data; packaging and metadata are Apache-2.0. `tw-formosa-bench` is CC BY-SA 4.0,
+inherited from Taiwan.md.*