| # Dataset attribution |
|
|
| Dataset: **SQuAD (Stanford Question Answering Dataset), version 1.1**, hosted on the Hugging Face Hub as `rajpurkar/squad`. |
|
|
| - Dataset: https://huggingface.co/datasets/rajpurkar/squad |
| - Project: https://rajpurkar.github.io/SQuAD-explorer/ |
| - Authors: Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. |
| - Paper: *SQuAD: 100,000+ Questions for Machine Comprehension of Text* (2016), https://arxiv.org/abs/1606.05250 |
| - Passage source: Wikipedia contributors. The article title is retained for each passage. |
| - Licence for the dataset excerpts: **Creative Commons Attribution-ShareAlike 4.0 International**, https://creativecommons.org/licenses/by-sa/4.0/ |
| - Split: `train`. |
| - Pinned revision: `7b6d24c440a36b6815f21b70d25016731768db1f`. |
|
|
| ## Selection and changes |
|
|
| The notebook streams this pinned revision and scans at most the first 6,000 rows. It selects the first 30 distinct contexts satisfying these teaching rules: |
|
|
| - At most three contexts per article title. |
| - At most 200 tokens for the title plus context, measured using the lab model tokenizer. |
| - Article display title at most 35 characters; the topic “Genocide” is excluded from this introductory sample. |
|
|
| Surrounding whitespace is stripped from contexts. Underscores in display titles become spaces. The passage text, original questions, and answer text are otherwise unchanged. The original SQuAD row ID remains in `squad_id`; local P01-P30 IDs link passages to the retrieval tables. |
|
|
| The provided sample has 30 passages across 10 article titles. It is a curated, bounded teaching sample, not a random or representative benchmark. The historical text is supplied for retrieval exercises and is not a source of live information. |
|
|
| Six evaluation questions come unchanged from the sampled SQuAD rows. The live Bitcoin price question and “When did it happen?” are instructor-written challenges. The notebook embeds only titles and contexts, keeping questions and answer spans separately for evaluation. |
|
|
| Dataset excerpts loaded or displayed by this app remain under CC BY-SA 4.0. Preserve this attribution when redistributing them. |
|
|
| ## Week 6 reuse |
| The document Q&A lesson reuses these same 30 passages unchanged. Seven evaluation cases use sampled SQuAD questions. The Montana question explicitly names the state. Three additional instructor-written cases test missing live information, a missing private detail, and ambiguity. Only source titles and passages enter the embedding index. |
|
|
| ## Direct Hub loading |
| The notebook and app now fetch the pinned dataset directly using datasets.load_dataset(..., streaming=True). No local passage JSON is required. Preserve attribution when redistributing loaded excerpts or outputs. |
| |
