# The reader's data

What `lab.reader` trains, checks and scores the tone reader with, each file at its key on the `tone-reader-data`
volume. It holds no third-party text:

- **49 files that are ours**, as they are: rc1's and rc2's runs (weights, results, each export and 16-bit
  file), the five r2 teachers' labels, the move pool, the label sets whose lines are all ours, and rc2's support-call
  and contrast sets: their committee labels and each line's call as it was written.
- **8 stand-ins**, in `rebuild/standins/` at their files' keys: files that mix our lines with others'. Each
  mirrors its file line for line in the same JSON style. Our rows are as they are; each value from a third-party
  source is a hole that says how to make it again from the source at its pin (`holes` in the manifest).
- **The 14 open sets are not here**: they are rebuilt from their sources' archives.

`rebuild/manifest.json` holds every file's sha256, the sources' pins, and the sha256 of each archive file the open
sets are read from.

## Rebuilding

`lab.reader.pipeline.pull` copies this folder to the volume. `lab.reader.pipeline.fetch` downloads each source at
its pin, rebuilds the open sets and the stand-ins' files, and writes a file only if its sha256 is the manifest's;
it also checks the files `pull` copied. Both run as pipecat-tone's
[`src/lab/README.md`](https://github.com/pipecat-ai/pipecat-tone/blob/main/src/lab/README.md#the-reader) says.

## Sources

| Source | At | Licence | Credit |
|---|---|---|---|
| natcs | [amazon-science/dstc11-track2-intent-induction@969b95a](https://github.com/amazon-science/dstc11-track2-intent-induction/tree/969b95a0d7365fbc6cd0e05989f1be6b44e6680c) | CDLA-Permissive-2.0 | NatCS (Gung et al., 2023; Amazon, DSTC11 Track 2), CDLA-Permissive-2.0 |
| taskmaster1 | [google-research-datasets/Taskmaster@d92cb6a](https://github.com/google-research-datasets/Taskmaster/tree/d92cb6af3005f1dc09c39e75e7daf4a04905e00b) | CC-BY-4.0 | Taskmaster-1 (Byrne et al., 2019; Google), CC BY 4.0 |
| taskmaster2 | [google-research-datasets/Taskmaster@d92cb6a](https://github.com/google-research-datasets/Taskmaster/tree/d92cb6af3005f1dc09c39e75e7daf4a04905e00b) | CC-BY-4.0 | Taskmaster-2 (Byrne et al., 2020; Google), CC BY 4.0 |
| taskmaster3 | [google-research-datasets/Taskmaster@d92cb6a](https://github.com/google-research-datasets/Taskmaster/tree/d92cb6af3005f1dc09c39e75e7daf4a04905e00b) | CC-BY-4.0 | Taskmaster-3 (Byrne et al., 2020; Google), CC BY 4.0 |
| abcd | [asappresearch/abcd@6b8700c](https://github.com/asappresearch/abcd/tree/6b8700ce67c6b37b062dd7a60abc76d7ef832a97) | MIT | ABCD, the Action-Based Conversations Dataset (Chen et al., 2021; ASAPP), MIT |
| multiwoz22 | [budzianowski/multiwoz@fe0c8e6](https://github.com/budzianowski/multiwoz/tree/fe0c8e65cfcd8462bd33c86e35f21addc84ca82b) | MIT | MultiWOZ 2.2 (Zang et al., 2020; Budzianowski et al., 2018), MIT |
| star | [RasaHQ/STAR@3058975](https://github.com/RasaHQ/STAR/tree/3058975092cdd09798e4a3a9e4ae135755824545) | MIT | STAR (Mosig et al., 2020; Rasa), MIT |
| ccpem | [google-research-datasets/ccpe@2c9cd30](https://github.com/google-research-datasets/ccpe/tree/2c9cd30f33f3a154b5a27d015333679262ff36f5) | CC-BY-4.0 | CCPE-M (Radlinski et al., 2019; Google), CC BY 4.0 |
| harpervalley | [cricketclub/gridspace-stanford-harper-valley@0bd721e](https://github.com/cricketclub/gridspace-stanford-harper-valley/tree/0bd721e877c4a85d8c13ff837e68661ea6200a98) | CC-BY-4.0 | Harper Valley Bank (Wu et al., 2020; Gridspace and Stanford), CC BY 4.0 |
| ami | [ami_public_manual_1.6.2.zip](https://groups.inf.ed.ac.uk/ami/AMICorpusAnnotations/ami_public_manual_1.6.2.zip) | CC-BY-4.0 | AMI Meeting Corpus manual annotations (Carletta et al., 2005; the AMI Consortium), CC BY 4.0 |
| dipco | [Zenodo 8122551](https://zenodo.org/records/8122551) | CDLA-Permissive-1.0 | DiPCo, the Dinner Party Corpus (Van Segbroeck et al., 2020; Amazon), CDLA-Permissive-1.0 |
| dynasent_r2 | [cgpotts/dynasent@707b620](https://github.com/cgpotts/dynasent/tree/707b6208978bfe7ff251a0c3263c9f0552c27b48) | CC-BY-4.0 | DynaSent round 2 (Potts et al., 2021), CC BY 4.0 |
| isarcasmeval | [iabufarha/iSarcasmEval@dfc708b](https://github.com/iabufarha/iSarcasmEval/tree/dfc708b53bde1bb571abfb5692f63231c2232195) | MIT | iSarcasmEval, SemEval-2022 Task 6 (Abu Farha et al., 2022), MIT |
| topicalchat_test | [alexa/Topical-Chat@7c93922](https://github.com/alexa/Topical-Chat/tree/7c939229cbcf6f55f6977b341a5a2f2fe982d53f) | CDLA-Sharing-1.0 | Topical-Chat (Gopalakrishnan et al., 2019; Amazon), CDLA-Sharing-1.0; its test splits only, for evaluation |
| goemotions | [google-research-datasets/go_emotions@add4922](https://huggingface.co/datasets/google-research-datasets/go_emotions/tree/add492243ff905527e67aeb8b80c082af02207c3) | Apache-2.0 | GoEmotions (Demszky et al., 2020; Google), Apache 2.0 |
| civil_comments | [google/civil_comments@f2970eb](https://huggingface.co/datasets/google/civil_comments/tree/f2970eb3a55777454c94069077cc8d9b5866312d) | CC0-1.0 | Civil Comments (Borkan et al., 2019; Jigsaw), CC0 1.0 |
| gutenberg | [willwade/Gutenberg-dialog-en@f3a0a1d](https://huggingface.co/datasets/willwade/Gutenberg-dialog-en/tree/f3a0a1df18275cd9f61f7d161ad6eed3ce35f88c) | MIT; the novels are public domain (Project Gutenberg) | The Gutenberg Dialogue Dataset (Csaky and Recski, 2021), English, from public-domain Project Gutenberg books, MIT |

Topical-Chat is share-alike: its test splits are read for evaluation only and never trained on. The rest of this
folder is ours.