| --- |
| library_name: transformers |
| tags: |
| - qualitative-coding |
| - cpu |
| - research-artifact |
| extra_gated_prompt: >- |
| These checkpoints were trained on a qualitative-coding corpus whose |
| redistribution licence and participant-consent basis are not documented in |
| the retained research artifact (see "Data provenance" below). Access is |
| granted for non-commercial research use and reproduction of the accompanying |
| paper only. You are responsible for confirming that your own use is lawful in |
| your jurisdiction. Do not redistribute the weights or attempt to reconstruct |
| the underlying source corpus. |
| extra_gated_fields: |
| Name: text |
| Affiliation: text |
| Intended use: text |
| I will use these weights for non-commercial research only: checkbox |
| I will not redistribute the weights or reconstruct the source corpus: checkbox |
| extra_gated_button_content: Request access |
| --- |
| |
|
|
| # Matched-Audit CPU Checkpoints |
|
|
| This repository contains every unique epoch-8 checkpoint from the |
| paper's five-seed matched training and domain-composition experiments. |
|
|
| ## Layout |
|
|
| - `matched/tiny_hr/seed_*`: human benchmark codes retained; technical reviews |
| use machine codes. |
| - `matched/tiny_mr/seed_*`: benchmark codes replaced by machine codes; the same |
| technical reviews are used. |
| - `domain/single_hr/seed_*`: benchmark-only human-code condition at matched |
| update count. |
| - `domain/single_mr/seed_*`: benchmark-only machine-code condition at matched |
| update count. |
|
|
| Seeds are 13, 42, 71, 101, and 137. Every folder is directly loadable with the |
| Transformers `subfolder=` argument. The `mixed_hr` and `mixed_mr` checkpoint |
| copies from the domain run are not uploaded twice: their hashes exactly match |
| the corresponding `matched/tiny_hr` and `matched/tiny_mr` files. |
|
|
| The checkpoints are research artifacts, not 20 independently recommended |
| deployments. The human study used the seed-42 epoch-8 matched checkpoints. |
| Automatic reference agreement should not be interpreted as qualitative |
| validity. |
|
|
| Source-data permissions are still under review. |
| Access through this gated repository does not authorize onward redistribution of its weights or training examples. |
|
|
| ## Warning: the `tiny_mr` checkpoints degenerate |
| |
| The machine-code (`tiny_mr`) checkpoints repeat one of their own content words |
| in roughly 35% of benchmark outputs and 23% of app-review outputs, on every |
| seed ("Clarity and Clarity", "Motion for amputation of motion"), while scoring |
| close to `tiny_hr` on reference agreement. They are published as evidence for |
| that finding, **not** as usable coders. |
|
|
| ## Data provenance and release status |
|
|
| These weights derive from a 999-pair English open-coding benchmark: 600 pairs |
| from social-science work across three university faculties (interviews and |
| reviews, consensus-coded by three to five coders) and 399 SemEval-2014 Task 4 |
| review excerpts, plus 1,990 machine-coded ICLR peer-review excerpts. |
|
|
| The supplied artifact records only the passage and its label. It does **not** |
| record the original language, coder identities, adjudication trace, consent |
| basis, or redistribution licence. Public availability of source text does not |
| by itself establish permission to redistribute a compiled corpus or weights |
| trained on it. Access is therefore gated, and the source passages are **not** |
| released. |
|
|
| ## Memorisation |
|
|
| These are small sequence-to-sequence models fitted to a small label set, and |
| they reproduce training annotations verbatim at a substantial rate. At the |
| matched epoch-8 checkpoint, **28.9% of human-code (HR) outputs across 1,000 |
| evaluated items are exact strings from the human training labels**, drawing on |
| 28 distinct labels. Treat generated codes as potentially reproducing the |
| original coders' annotations rather than as novel interpretations. Long source |
| passages are not recoverable from a model of this size; the annotation set is |
| partially recoverable, which is why access is gated. |
|
|
| ## Intended use and limits |
|
|
| First-pass, editable code suggestions for one pre-segmented English passage, |
| returning one code of at most six words. These models do not segment |
| transcripts, assign multiple codes, build codebooks, or construct themes, and |
| are not a substitute for a researcher. In a blinded five-expert evaluation, |
| 47.5% of Tiny-HR suggestions were rated usable or better, against 85.0% for a |
| task-adapted Qwen2.5-7B. |
|
|
| ## Citation |
|
|
| Accompanying paper: *AI-Assisted Qualitative Coding on a CPU* (under review). |
| Code, prompts, analysis scripts and hashes accompany the submission. |
|
|
