Dataset attribution
Dataset: SQuAD (Stanford Question Answering Dataset), version 1.1, hosted on the Hugging Face Hub as rajpurkar/squad.
- Dataset: https://huggingface.co/datasets/rajpurkar/squad
- Project: https://rajpurkar.github.io/SQuAD-explorer/
- Authors: Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.
- Paper: SQuAD: 100,000+ Questions for Machine Comprehension of Text (2016), https://arxiv.org/abs/1606.05250
- Passage source: Wikipedia contributors. The article title is retained for each passage.
- Licence for the dataset excerpts: Creative Commons Attribution-ShareAlike 4.0 International, https://creativecommons.org/licenses/by-sa/4.0/
- Split:
train. - Pinned revision:
7b6d24c440a36b6815f21b70d25016731768db1f.
Selection and changes
The notebook streams this pinned revision and scans at most the first 6,000 rows. It selects the first 30 distinct contexts satisfying these teaching rules:
- At most three contexts per article title.
- At most 200 tokens for the title plus context, measured using the lab model tokenizer.
- Article display title at most 35 characters; the topic “Genocide” is excluded from this introductory sample.
Surrounding whitespace is stripped from contexts. Underscores in display titles become spaces. The passage text, original questions, and answer text are otherwise unchanged. The original SQuAD row ID remains in squad_id; local P01-P30 IDs link passages to the retrieval tables.
The provided sample has 30 passages across 10 article titles. It is a curated, bounded teaching sample, not a random or representative benchmark. The historical text is supplied for retrieval exercises and is not a source of live information.
Six evaluation questions come unchanged from the sampled SQuAD rows. The live Bitcoin price question and “When did it happen?” are instructor-written challenges. The notebook embeds only titles and contexts, keeping questions and answer spans separately for evaluation.
Dataset excerpts loaded or displayed by this app remain under CC BY-SA 4.0. Preserve this attribution when redistributing them.
Week 6 reuse
The document Q&A lesson reuses these same 30 passages unchanged. Seven evaluation cases use sampled SQuAD questions. The Montana question explicitly names the state. Three additional instructor-written cases test missing live information, a missing private detail, and ambiguity. Only source titles and passages enter the embedding index.
Direct Hub loading
The notebook and app now fetch the pinned dataset directly using datasets.load_dataset(..., streaming=True). No local passage JSON is required. Preserve attribution when redistributing loaded excerpts or outputs.