{"ed3bookaug20_2024.pdf": "Speech and Language Processing\nAn Introduction to Natural Language Processing,\nComputational Linguistics, and Speech Recognition\nwith Language Models\nThird Edition draft\nDaniel Jurafsky\nStanford University\nJames H. Martin\nUniversity of Colorado at Boulder\nCopyright \u00a92024. All rights reserved.\nDraft of August 20, 2024. Comments and typos welcome! Summary of Contents\nI Fundamental Algorithms for NLP 1\n1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3\n2 Regular Expressions, Tokenization, Edit Distance . . . . . . . . . . . . . . . 4\n3 N-gram Language Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32\n4 Naive Bayes, Text Classi\ufb01cation, and Sentiment . . . . . . . . . . . . . . . . . 56\n5 Logistic Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77\n6 Vector Semantics and Embeddings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101\n7 Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132\n8 RNNs and LSTMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158\n9 The Transformer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184\n10 Large Language Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203\n11 Masked Language Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223\n12 Model Alignment, Prompting, and In-Context Learning . . . . . . . . . 242\nII NLP Applications 261\n13 Machine Translation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263\n14 Question Answering, Information Retrieval, and RAG . . . . . . . . . . 289\n15 Chatbots & Dialogue Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309\n16 Automatic Speech Recognition and Text-to-Speech . . . . . . . . . . . . . . 331\nIII Annotating Linguistic Structure 359\n17 Sequence Labeling for Parts of Speech and Named Entities . . . . . . 362\n18 Context-Free Grammars and Constituency Parsing . . . . . . . . . . . . . 387\n19 Dependency Parsing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411\n20 Information Extraction: Relations, Events, and Time. . . . . . . . . . . . 435\n21 Semantic Role Labeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461\n22 Lexicons for Sentiment, Affect, and Connotation . . . . . . . . . . . . . . . . 481\n23 Coreference Resolution and Entity Linking . . . . . . . . . . . . . . . . . . . . . 501\n24 Discourse Coherence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 531\nBibliography. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 553\nSubject Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 585\n2 Contents\nI Fundamental Algorithms for NLP 1\n1 Introduction 3\n2 Regular Expressions, Tokenization, Edit Distance 4\n2.1 Regular Expressions . . . . . . . . . . . . . . . . . . . . . . . . . 5\n2.2 Words . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13\n2.3 Corpora . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15\n2.4 Simple Unix Tools for Word Tokenization . . . . . . . . . . . . . 17\n2.5 Word and Subword Tokenization . . . . . . . . . . . . . . . . . . 18\n2.6 Word Normalization, Lemmatization and Stemming . . . . . . . . 23\n2.7 Sentence Segmentation . . . . . . . . . . . . . . . . . . . . . . . 25\n2.8 Minimum Edit Distance . . . . . . . . . . . . . . . . . . . . . . . 25\n2.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29\nBibliographical and Historical Notes . . . . . . . . . . . . . . . . . . . . 30\nExercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31\n3 N-gram Language Models 32\n3.1 N-Grams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33\n3.2 Evaluating Language Models: Training and Test Sets . . . . . . . 38\n3.3 Evaluating Language Models: Perplexity . . . . . . . . . . . . . . 39\n3.4 Sampling sentences from a language model . . . . . . . . . . . . . 42\n3.5 Generalizing vs. over\ufb01tting the training set . . . . . . . . . . . . . 43\n3.6 Smoothing, Interpolation, and Backoff . . . . . . . . . . . . . . . 45\n3.7 Advanced: Perplexity\u2019s Relation to Entropy . . . . . . . . . . . . 49\n3.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52\nBibliographical and Historical Notes . . . . . . . . . . . . . . . . . . . . 52\nExercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54\n4 Naive Bayes, Text Classi\ufb01cation, and Sentiment 56\n4.1 Naive Bayes Classi\ufb01ers . . . . . . . . . . . . . . . . . . . . . . . 57\n4.2 Training the Naive Bayes Classi\ufb01er . . . . . . . . . . . . . . . . . 60\n4.3 Worked example . . . . . . . . . . . . . . . . . . . . . . . . . . . 61\n4.4 Optimizing for Sentiment Analysis . . . . . . . . . . . . . . . . . 62\n4.5 Naive Bayes for other text classi\ufb01cation tasks . . . . . . . . . . . 64\n4.6 Naive Bayes as a Language Model . . . . . . . . . . . . . . . . . 65\n4.7 Evaluation: Precision, Recall, F-measure . . . . . . . . . . . . . . 66\n4.8 Test sets and Cross-validation . . . . . . . . . . . . . . . . . . . . 69\n4.9 Statistical Signi\ufb01cance Testing . . . . . . . . . . . . . . . . . . . 70\n4.10 Avoiding Harms in Classi\ufb01cation . . . . . . . . . . . . . . . . . . 73\n4.11 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74\nBibliographical and Historical Notes . . . . . . . . . . . . . . . . . . . . 75\nExercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76\n5 Logistic Regression 77\n5.1 The sigmoid function . . . . . . . . . . . . . . . . . . . . . . . . 78\n5.2 Classi\ufb01cation with Logistic Regression . . . . . . . . . . . . . . . 80\n5.3 Multinomial logistic regression . . . . . . . . . . . . . . . . . . . 84\n5.4 Learning in Logistic Regression . . . . . . . . . . . . . . . . . . . 87\n3 4CONTENTS\n5.5 The cross-entropy loss function . . . . . . . . . . . . . . . . . . . 88\n5.6 Gradient Descent . . . . . . . . . . . . . . . . . . . . . . . . . . 89\n5.7 Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95\n5.8 Learning in Multinomial Logistic Regression . . . . . . . . . . . . 96\n5.9 Interpreting models . . . . . . . . . . . . . . . . . . . . . . . . . 98\n5.10 Advanced: Deriving the Gradient Equation . . . . . . . . . . . . . 98\n5.11 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99\nBibliographical and Historical Notes . . . . . . . . . . . . . . . . . . . . 100\nExercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100\n6 Vector Semantics and Embeddings 101\n6.1 Lexical Semantics . . . . . . . . . . . . . . . . . . . . . . . . . . 102\n6.2 Vector Semantics . . . . . . . . . . . . . . . . . . . . . . . . . . 105\n6.3 Words and Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . 106\n6.4 Cosine for measuring similarity . . . . . . . . . . . . . . . . . . . 110\n6.5 TF-IDF: Weighing terms in the vector . . . . . . . . . . . . . . . 111\n6.6 Pointwise Mutual Information (PMI) . . . . . . . . . . . . . . . . 114\n6.7 Applications of the tf-idf or PPMI vector models . . . . . . . . . . 116\n6.8 Word2vec . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117\n6.9 Visualizing Embeddings . . . . . . . . . . . . . . . . . . . . . . . 123\n6.10 Semantic properties of embeddings . . . . . . . . . . . . . . . . . 124\n6.11 Bias and Embeddings . . . . . . . . . . . . . . . . . . . . . . . . 126\n6.12 Evaluating Vector Models . . . . . . . . . . . . . . . . . . . . . . 127\n6.13 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128\nBibliographical and Historical Notes . . . . . . . . . . . . . . . . . . . . 129\nExercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131\n7 Neural Networks 132\n7.1 Units . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133\n7.2 The XOR problem . . . . . . . . . . . . . . . . . . . . . . . . . . 135\n7.3 Feedforward Neural Networks . . . . . . . . . . . . . . . . . . . . 138\n7.4 Feedforward networks for NLP: Classi\ufb01cation . . . . . . . . . . . 142\n7.5 Training Neural Nets . . . . . . . . . . . . . . . . . . . . . . . . 145\n7.6 Feedforward Neural Language Modeling . . . . . . . . . . . . . . 152\n7.7 Training the neural language model . . . . . . . . . . . . . . . . . 155\n7.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156\nBibliographical and Historical Notes . . . . . . . . . . . . . . . . . . . . 157\n8 RNNs and LSTMs 158\n8.1 Recurrent Neural Networks . . . . . . . . . . . . . . . . . . . . . 158\n8.2 RNNs as Language Models . . . . . . . . . . . . . . . . . . . . . 162\n8.3 RNNs for other NLP tasks . . . . . . . . . . . . . . . . . . . . . . 165\n8.4 Stacked and Bidirectional RNN architectures . . . . . . . . . . . . 168\n8.5 The LSTM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170\n8.6 Summary: Common RNN NLP Architectures . . . . . . . . . . . 174\n8.7 The Encoder-Decoder Model with RNNs . . . . . . . . . . . . . . 174\n8.8 Attention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179\n8.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181\nBibliographical and Historical Notes . . . . . . . . . . . . . . . . . . . . 182\n9 The Transformer 184\n9.1 Attention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 CONTENTS 5\n9.2 Transformer Blocks . . . . . . . . . . . . . . . . . . . . . . . . . 190\n9.3 Parallelizing computation using a single matrix X. . . . . . . . . 193\n9.4 The input: embeddings for token and position . . . . . . . . . . . 196\n9.5 The Language Modeling Head . . . . . . . . . . . . . . . . . . . 198\n9.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201\nBibliographical and Historical Notes . . . . . . . . . . . . . . . . . . . . 201\n10 Large Language Models 203\n10.1 Large Language Models with Transformers . . . . . . . . . . . . . 204\n10.2 Sampling for LLM Generation . . . . . . . . . . . . . . . . . . . 207\n10.3 Pretraining Large Language Models . . . . . . . . . . . . . . . . 210\n10.4 Evaluating Large Language Models . . . . . . . . . . . . . . . . . 214\n10.5 Dealing with Scale . . . . . . . . . . . . . . . . . . . . . . . . . . 216\n10.6 Potential Harms from Language Models . . . . . . . . . . . . . . 219\n10.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220\nBibliographical and Historical Notes . . . . . . . . . . . . . . . . . . . . 220\n11 Masked Language Models 223\n11.1 Bidirectional Transformer Encoders . . . . . . . . . . . . . . . . . 223\n11.2 Training Bidirectional Encoders . . . . . . . . . . . . . . . . . . . 226\n11.3 Contextual Embeddings . . . . . . . . . . . . . . . . . . . . . . . 230\n11.4 Fine-Tuning for Classi\ufb01cation . . . . . . . . . . . . . . . . . . . . 234\n11.5 Fine-Tuning for Sequence Labelling: Named Entity Recognition . 237\n11.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240\nBibliographical and Historical Notes . . . . . . . . . . . . . . . . . . . . 241\n12 Model Alignment, Prompting, and In-Context Learning 242\n12.1 Prompting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243\n12.2 Post-training and Model Alignment . . . . . . . . . . . . . . . . . 248\n12.3 Model Alignment: Instruction Tuning . . . . . . . . . . . . . . . . 249\n12.4 Chain-of-Thought Prompting . . . . . . . . . . . . . . . . . . . . 254\n12.5 Automatic Prompt Optimization . . . . . . . . . . . . . . . . . . . 255\n12.6 Evaluating Prompted Language Models . . . . . . . . . . . . . . . 258\n12.7 Model Alignment with Human Preferences: RLHF and DPO . . . 259\n12.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259\nBibliographical and Historical Notes . . . . . . . . . . . . . . . . . . . . 259\nII NLP Applications 261\n13 Machine Translation 263\n13.1 Language Divergences and Typology . . . . . . . . . . . . . . . . 264\n13.2 Machine Translation using Encoder-Decoder . . . . . . . . . . . . 268\n13.3 Details of the Encoder-Decoder Model . . . . . . . . . . . . . . . 272\n13.4 Decoding in MT: Beam Search . . . . . . . . . . . . . . . . . . . 274\n13.5 Translating in low-resource situations . . . . . . . . . . . . . . . . 278\n13.6 MT Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280\n13.7 Bias and Ethical Issues . . . . . . . . . . . . . . . . . . . . . . . 284\n13.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285\nBibliographical and Historical Notes . . . . . . . . . . . . . . . . . . . . 286\nExercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288\n14 Question Answering, Information Retrieval, and RAG 289 6CONTENTS\n14.1 Information Retrieval . . . . . . . . . . . . . . . . . . . . . . . . 290\n14.2 Information Retrieval with Dense Vectors . . . . . . . . . . . . . . 298\n14.3 Answering Questions with RAG . . . . . . . . . . . . . . . . . . 301\n14.4 Evaluating Question Answering . . . . . . . . . . . . . . . . . . . 304\n14.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306\nBibliographical and Historical Notes . . . . . . . . . . . . . . . . . . . . 306\nExercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308\n15 Chatbots & Dialogue Systems 309\n15.1 Properties of Human Conversation . . . . . . . . . . . . . . . . . 311\n15.2 Frame-Based Dialogue Systems . . . . . . . . . . . . . . . . . . . 314\n15.3 Dialogue Acts and Dialogue State . . . . . . . . . . . . . . . . . . 317\n15.4 Chatbots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321\n15.5 Dialogue System Design . . . . . . . . . . . . . . . . . . . . . . . 325\n15.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327\nBibliographical and Historical Notes . . . . . . . . . . . . . . . . . . . . 328\nExercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330\n16 Automatic Speech Recognition and Text-to-Speech 331\n16.1 The Automatic Speech Recognition Task . . . . . . . . . . . . . . 332\n16.2 Feature Extraction for ASR: Log Mel Spectrum . . . . . . . . . . 334\n16.3 Speech Recognition Architecture . . . . . . . . . . . . . . . . . . 339\n16.4 CTC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341\n16.5 ASR Evaluation: Word Error Rate . . . . . . . . . . . . . . . . . 346\n16.6 TTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348\n16.7 Other Speech Tasks . . . . . . . . . . . . . . . . . . . . . . . . . 353\n16.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354\nBibliographical and Historical Notes . . . . . . . . . . . . . . . . . . . . 354\nExercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357\nIII Annotating Linguistic Structure 359\n17 Sequence Labeling for Parts of Speech and Named Entities 362\n17.1 (Mostly) English Word Classes . . . . . . . . . . . . . . . . . . . 363\n17.2 Part-of-Speech Tagging . . . . . . . . . . . . . . . . . . . . . . . 365\n17.3 Named Entities and Named Entity Tagging . . . . . . . . . . . . . 367\n17.4 HMM Part-of-Speech Tagging . . . . . . . . . . . . . . . . . . . 369\n17.5 Conditional Random Fields (CRFs) . . . . . . . . . . . . . . . . . 376\n17.6 Evaluation of Named Entity Recognition . . . . . . . . . . . . . . 381\n17.7 Further Details . . . . . . . . . . . . . . . . . . . . . . . . . . . 381\n17.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383\nBibliographical and Historical Notes . . . . . . . . . . . . . . . . . . . . 384\nExercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385\n18 Context-Free Grammars and Constituency Parsing 387\n18.1 Constituency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 388\n18.2 Context-Free Grammars . . . . . . . . . . . . . . . . . . . . . . . 388\n18.3 Treebanks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392\n18.4 Grammar Equivalence and Normal Form . . . . . . . . . . . . . . 394\n18.5 Ambiguity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395\n18.6 CKY Parsing: A Dynamic Programming Approach . . . . . . . . 397\n18.7 Span-Based Neural Constituency Parsing . . . . . . . . . . . . . . 403 CONTENTS 7\n18.8 Evaluating Parsers . . . . . . . . . . . . . . . . . . . . . . . . . . 405\n18.9 Heads and Head-Finding . . . . . . . . . . . . . . . . . . . . . . 406\n18.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407\nBibliographical and Historical Notes . . . . . . . . . . . . . . . . . . . . 408\nExercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409\n19 Dependency Parsing 411\n19.1 Dependency Relations . . . . . . . . . . . . . . . . . . . . . . . . 412\n19.2 Transition-Based Dependency Parsing . . . . . . . . . . . . . . . 416\n19.3 Graph-Based Dependency Parsing . . . . . . . . . . . . . . . . . 425\n19.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431\n19.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432\nBibliographical and Historical Notes . . . . . . . . . . . . . . . . . . . . 433\nExercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434\n20 Information Extraction: Relations, Events, and Time 435\n20.1 Relation Extraction . . . . . . . . . . . . . . . . . . . . . . . . . 436\n20.2 Relation Extraction Algorithms . . . . . . . . . . . . . . . . . . . 438\n20.3 Extracting Events . . . . . . . . . . . . . . . . . . . . . . . . . . 446\n20.4 Representing Time . . . . . . . . . . . . . . . . . . . . . . . . . . 447\n20.5 Representing Aspect . . . . . . . . . . . . . . . . . . . . . . . . . 450\n20.6 Temporally Annotated Datasets: TimeBank . . . . . . . . . . . . . 451\n20.7 Automatic Temporal Analysis . . . . . . . . . . . . . . . . . . . . 452\n20.8 Template Filling . . . . . . . . . . . . . . . . . . . . . . . . . . . 456\n20.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 458\nBibliographical and Historical Notes . . . . . . . . . . . . . . . . . . . . 459\nExercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 460\n21 Semantic Role Labeling 461\n21.1 Semantic Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . 462\n21.2 Diathesis Alternations . . . . . . . . . . . . . . . . . . . . . . . . 462\n21.3 Semantic Roles: Problems with Thematic Roles . . . . . . . . . . 464\n21.4 The Proposition Bank . . . . . . . . . . . . . . . . . . . . . . . . 465\n21.5 FrameNet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466\n21.6 Semantic Role Labeling . . . . . . . . . . . . . . . . . . . . . . . 468\n21.7 Selectional Restrictions . . . . . . . . . . . . . . . . . . . . . . . 472\n21.8 Primitive Decomposition of Predicates . . . . . . . . . . . . . . . 476\n21.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 477\nBibliographical and Historical Notes . . . . . . . . . . . . . . . . . . . . 478\nExercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480\n22 Lexicons for Sentiment, Affect, and Connotation 481\n22.1 De\ufb01ning Emotion . . . . . . . . . . . . . . . . . . . . . . . . . . 482\n22.2 Available Sentiment and Affect Lexicons . . . . . . . . . . . . . . 484\n22.3 Creating Affect Lexicons by Human Labeling . . . . . . . . . . . 485\n22.4 Semi-supervised Induction of Affect Lexicons . . . . . . . . . . . 487\n22.5 Supervised Learning of Word Sentiment . . . . . . . . . . . . . . 490\n22.6 Using Lexicons for Sentiment Recognition . . . . . . . . . . . . . 495\n22.7 Using Lexicons for Affect Recognition . . . . . . . . . . . . . . . 496\n22.8 Lexicon-based methods for Entity-Centric Affect . . . . . . . . . . 497\n22.9 Connotation Frames . . . . . . . . . . . . . . . . . . . . . . . . . 497\n22.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 499 8CONTENTS\nBibliographical and Historical Notes . . . . . . . . . . . . . . . . . . . . 500\nExercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 500\n23 Coreference Resolution and Entity Linking 501\n23.1 Coreference Phenomena: Linguistic Background . . . . . . . . . . 504\n23.2 Coreference Tasks and Datasets . . . . . . . . . . . . . . . . . . . 509\n23.3 Mention Detection . . . . . . . . . . . . . . . . . . . . . . . . . . 510\n23.4 Architectures for Coreference Algorithms . . . . . . . . . . . . . 513\n23.5 Classi\ufb01ers using hand-built features . . . . . . . . . . . . . . . . . 515\n23.6 A neural mention-ranking algorithm . . . . . . . . . . . . . . . . 517\n23.7 Entity Linking . . . . . . . . . . . . . . . . . . . . . . . . . . . . 520\n23.8 Evaluation of Coreference Resolution . . . . . . . . . . . . . . . . 524\n23.9 Winograd Schema problems . . . . . . . . . . . . . . . . . . . . . 525\n23.10 Gender Bias in Coreference . . . . . . . . . . . . . . . . . . . . . 526\n23.11 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527\nBibliographical and Historical Notes . . . . . . . . . . . . . . . . . . . . 528\nExercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 530\n24 Discourse Coherence 531\n24.1 Coherence Relations . . . . . . . . . . . . . . . . . . . . . . . . . 533\n24.2 Discourse Structure Parsing . . . . . . . . . . . . . . . . . . . . . 536\n24.3 Centering and Entity-Based Coherence . . . . . . . . . . . . . . . 540\n24.4 Representation learning models for local coherence . . . . . . . . 544\n24.5 Global Coherence . . . . . . . . . . . . . . . . . . . . . . . . . . 546\n24.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 549\nBibliographical and Historical Notes . . . . . . . . . . . . . . . . . . . . 550\nExercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552\nBibliography 553\nSubject Index 585 Part I\nFUNDAMENTAL ALGORITHMS FOR\nNLP\nIn the \ufb01rst part of the book we introduce the fundamental suite of algorithmic\ntools that make up the modern neural language model that is the heart of end-to-end\nNLP systems. We begin with tokenization and preprocessing, as well as useful algo-\nrithms like computing edit distance, and then proceed to the tasks of classi\ufb01cation,\nlogistic regression, neural networks, proceeding through feedforward networks, re-\ncurrent networks, and then transformers. We\u2019ll also see the role of embeddings as a\nmodel of word meaning.  CHAPTER\n1Introduction\nLa derni `ere chose qu\u2019on trouve en faisant un ouvrage est de savoir celle qu\u2019il faut\nmettre la premi `ere.\n[The last thing you \ufb01gure out in writing a book is what to put \ufb01rst.]\nPascal\n3 4CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\nCHAPTER\n2Regular Expressions, Tok-\nenization, Edit Distance\nUser: I am unhappy.\nELIZA: DO YOU THINK COMING HERE WILL HELP YOU NOT TO BE UNHAPPY\nUser: I need some help, that much seems certain.\nELIZA: WHAT WOULD IT MEAN TO YOU IF YOU GOT SOME HELP\nUser: Perhaps I could learn to get along with my mother.\nELIZA: TELL ME MORE ABOUT YOUR FAMILY\nUser: My mother takes care of me.\nELIZA: WHO ELSE IN YOU FAMILY TAKES CARE OF YOU\nUser: My father.\nELIZA: YOUR FATHER\nUser: You are like my father in some ways.\nWeizenbaum (1966)\nThe dialogue above is from ELIZA , an early natural language processing system ELIZA\nthat could carry on a limited conversation with a user by imitating the responses of\na Rogerian psychotherapist (Weizenbaum, 1966). ELIZA is a surprisingly simple\nprogram that uses pattern matching to recognize phrases like \u201cI need X\u201d and translate\nthem into suitable outputs like \u201cWhat would it mean to you if you got X?\u201d. This\nsimple technique succeeds in this domain because ELIZA doesn\u2019t actually need to\nknow anything to mimic a Rogerian psychotherapist. As Weizenbaum notes, this is\none of the few dialogue genres where listeners can act as if they know nothing of the\nworld. ELIZA\u2019s mimicry of human conversation was remarkably successful: many\npeople who interacted with ELIZA came to believe that it really understood them\nand their problems, many continued to believe in ELIZA\u2019s abilities even after the\nprogram\u2019s operation was explained to them (Weizenbaum, 1976), and even today\nsuch chatbots are a fun diversion. chatbots\nOf course modern conversational agents are much more than a diversion; they\ncan answer questions, book \ufb02ights, or \ufb01nd restaurants, functions for which they rely\non a much more sophisticated understanding of the user\u2019s intent, as we will see in\nChapter 15. Nonetheless, the simple pattern-based methods that powered ELIZA\nand other chatbots play a crucial role in natural language processing.\nWe\u2019ll begin with the most important tool for describing text patterns: the regular\nexpression . Regular expressions can be used to specify strings we might want to\nextract from a document, from transforming \u201cI need X\u201d in ELIZA above, to de\ufb01ning\nstrings like $199 or$24.99 for extracting tables of prices from a document.\nWe\u2019ll then turn to a set of tasks collectively called text normalization , in whichtext\nnormalization\nregular expressions play an important part. Normalizing text means converting it\nto a more convenient, standard form. For example, most of what we are going to\ndo with language relies on \ufb01rst separating out or tokenizing words from running\ntext, the task of tokenization . English words are often separated from each other tokenization\nby whitespace, but whitespace is not always suf\ufb01cient. New York androck \u2019n\u2019 roll\nare sometimes treated as large words despite the fact that they contain spaces, while\nsometimes we\u2019ll need to separate I\u2019minto the two words Iandam. For processing\ntweets or texts we\u2019ll need to tokenize emoticons like:)orhashtags like#nlproc . 2.1 \u2022 R EGULAR EXPRESSIONS 5\nSome languages, like Japanese, don\u2019t have spaces between words, so word tokeniza-\ntion becomes more dif\ufb01cult.\nAnother part of text normalization is lemmatization , the task of determining lemmatization\nthat two words have the same root, despite their surface differences. For example,\nthe words sang ,sung , and sings are forms of the verb sing. The word sing is the\ncommon lemma of these words, and a lemmatizer maps from all of these to sing.\nLemmatization is essential for processing morphologically complex languages like\nArabic. Stemming refers to a simpler version of lemmatization in which we mainly stemming\njust strip suf\ufb01xes from the end of the word. Text normalization also includes sen-\ntence segmentation : breaking up a text into individual sentences, using cues likesentence\nsegmentation\nperiods or exclamation points.\nFinally, we\u2019ll need to compare words and other strings. We\u2019ll introduce a metric\ncalled edit distance that measures how similar two strings are based on the number\nof edits (insertions, deletions, substitutions) it takes to change one string into the\nother. Edit distance is an algorithm with applications throughout language process-\ning, from spelling correction to speech recognition to coreference resolution.\n2.1 Regular Expressions\nOne of the most useful tools for text processing in computer science has been the\nregular expression (often shortened to regex ), a language for specifying text searchregular\nexpression\nstrings. This practical language is used in every computer language, in text process-\ning tools like the Unix tools grep, and in editors like vim or Emacs. Formally, a\nregular expression is an algebraic notation for characterizing a set of strings. Reg-\nular expressions are particularly useful for searching in texts, when we have a pat-\ntern to search for and a corpus of texts to search through. A regular expression corpus\nsearch function will search through the corpus, returning all texts that match the\npattern. The corpus can be a single document or a collection. For example, the\nUnix command-line tool grep takes a regular expression and returns every line of\nthe input document that matches the expression.\nA search can be designed to return every match on a line, if there are more than\none, or just the \ufb01rst match. In the following examples we generally underline the\nexact part of the pattern that matches the regular expression and show only the \ufb01rst\nmatch. We\u2019ll show regular expressions delimited by slashes but note that slashes are\nnotpart of the regular expressions.\nRegular expressions come in many variants. We\u2019ll be describing extended regu-\nlar expressions ; different regular expression parsers may only recognize subsets of\nthese, or treat some expressions slightly differently. Using an online regular expres-\nsion tester is a handy way to test out your expressions and explore these variations.\n2.1.1 Basic Regular Expression Patterns\nThe simplest kind of regular expression is a sequence of simple characters; putting\ncharacters in sequence is called concatenation . To search for woodchuck , we type concatenation\n/woodchuck/ . The expression /Buttercup/ matches any string containing the\nsubstring Buttercup ;grep with that expression would return the line I\u2019m called lit-\ntle Buttercup . The search string can consist of a single character (like /!/) or a\nsequence of characters (like /urgl/ ) (see Fig. 2.1).\nRegular expressions are case sensitive ; lower case /s/ is distinct from upper 6CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\nRegex Example Patterns Matched\n/woodchucks/ \u201cinteresting links to woodchucks and lemurs\u201d\n/a/ \u201cMary Ann stopped by Mona\u2019s\u201d\n/!/ \u201cYou\u2019ve left the burglar behind again! \u201d said Nori\nFigure 2.1 Some simple regex searches.\ncase/S/ (/s/ matches a lower case sbut not an upper case S). This means that\nthe pattern /woodchucks/ will not match the string Woodchucks . We can solve this\nproblem with the use of the square braces [and]. The string of characters inside the\nbraces speci\ufb01es a disjunction of characters to match. For example, Fig. 2.2 shows\nthat the pattern /[wW]/ matches patterns containing either worW.\nRegex Match Example Patterns\n/[wW]oodchuck/ Woodchuck or woodchuck \u201cWoodchuck \u201d\n/[abc]/ \u2018a\u2019, \u2018b\u2019, or\u2018c\u2019 \u201cIn uomini, in solda ti\u201d\n/[1234567890]/ any digit \u201cplenty of 7 to 5\u201d\nFigure 2.2 The use of the brackets []to specify a disjunction of characters.\nThe regular expression /[1234567890]/ speci\ufb01es any single digit. While such\nclasses of characters as digits or letters are important building blocks in expressions,\nthey can get awkward (e.g., it\u2019s inconvenient to specify\n/[ABCDEFGHIJKLMNOPQRSTUVWXYZ]/ (2.1)\nto mean \u201cany capital letter\u201d). In cases where there is a well-de\ufb01ned sequence asso-\nciated with a set of characters, the brackets can be used with the dash ( -) to specify\nany one character in a range . The pattern /[2-5]/ speci\ufb01es any one of the charac- range\nters2,3,4, or5. The pattern /[b-g]/ speci\ufb01es one of the characters b,c,d,e,f, or\ng. Some other examples are shown in Fig. 2.3.\nRegex Match Example Patterns Matched\n/[A-Z]/ an upper case letter \u201cwe should call it \u2018D renched Blossoms\u2019 \u201d\n/[a-z]/ a lower case letter \u201cmy beans were impatient to be hoed!\u201d\n/[0-9]/ a single digit \u201cChapter 1 : Down the Rabbit Hole\u201d\nFigure 2.3 The use of the brackets []plus the dash -to specify a range.\nThe square braces can also be used to specify what a single character cannot be,\nby use of the caret ^. If the caret ^is the \ufb01rst symbol after the open square brace [,\nthe resulting pattern is negated. For example, the pattern /[^a]/ matches any single\ncharacter (including special characters) except a. This is only true when the caret\nis the \ufb01rst symbol after the open square brace. If it occurs anywhere else, it usually\nstands for a caret; Fig. 2.4 shows some examples.\nRegex Match (single characters) Example Patterns Matched\n/[^A-Z]/ not an upper case letter \u201cOyfn pripetchik\u201d\n/[^Ss]/ neither \u2018S\u2019 nor \u2018s\u2019 \u201cIhave no exquisite reason for\u2019t\u201d\n/[^.]/ not a period \u201cour resident Djinn\u201d\n/[e^]/ either \u2018e\u2019 or \u2018 ^\u2019 \u201clook up \u02c6 now\u201d\n/a^b/ the pattern \u2018 a^b\u2019 \u201clook up a\u02c6 b now\u201d\nFigure 2.4 The caret ^for negation or just to mean ^. See below re: the backslash for escaping the period. 2.1 \u2022 R EGULAR EXPRESSIONS 7\nHow can we talk about optional elements, like an optional sinwoodchuck and\nwoodchucks ? We can\u2019t use the square brackets, because while they allow us to say\n\u201cs or S\u201d, they don\u2019t allow us to say \u201cs or nothing\u201d. For this we use the question mark\n/?/, which means \u201cthe preceding character or nothing\u201d, as shown in Fig. 2.5.\nRegex Match Example Patterns Matched\n/woodchucks?/ woodchuck or woodchucks \u201cwoodchuck \u201d\n/colou?r/ color or colour \u201ccolor \u201d\nFigure 2.5 The question mark ?marks optionality of the previous expression.\nWe can think of the question mark as meaning \u201czero or one instances of the\nprevious character\u201d. That is, it\u2019s a way of specifying how many of something that\nwe want, something that is very important in regular expressions. For example,\nconsider the language of certain sheep, which consists of strings that look like the\nfollowing:\nbaa!\nbaaa!\nbaaaa!\n. . .\nThis language consists of strings with a b, followed by at least two a\u2019s, followed\nby an exclamation point. The set of operators that allows us to say things like \u201csome\nnumber of as\u201d are based on the asterisk or *, commonly called the Kleene * (gen- Kleene *\nerally pronounced \u201ccleany star\u201d). The Kleene star means \u201czero or more occurrences\nof the immediately previous character or regular expression\u201d. So /a*/ means \u201cany\nstring of zero or more as\u201d. This will match aoraaaaaa , but it will also match the\nempty string at the start of Off Minor since the string Off Minor starts with zero a\u2019s.\nSo the regular expression for matching one or more ais/aa*/ , meaning one afol-\nlowed by zero or more as. More complex patterns can also be repeated. So /[ab]*/\nmeans \u201czero or more a\u2019s or b\u2019s\u201d (not \u201czero or more right square braces\u201d). This will\nmatch strings like aaaa orababab orbbbb , as well as the empty string.\nFor specifying multiple digits (useful for \ufb01nding prices) we can extend /[0-9]/ ,\nthe regular expression for a single digit. An integer (a string of digits) is thus\n/[0-9][0-9]*/ . (Why isn\u2019t it just /[0-9]*/ ?)\nSometimes it\u2019s annoying to have to write the regular expression for digits twice,\nso there is a shorter way to specify \u201cat least one\u201d of some character. This is the\nKleene + , which means \u201cone or more occurrences of the immediately preceding Kleene +\ncharacter or regular expression\u201d. Thus, the expression /[0-9]+/ is the normal way\nto specify \u201ca sequence of digits\u201d. There are thus two ways to specify the sheep\nlanguage: /baaa*!/ or/baa+!/ .\nOne very important special character is the period ( /./), awildcard expression\nthat matches any single character ( except a carriage return), as shown in Fig. 2.6.\nRegex Match Example Matches\n/beg.n/ any character between begandn begin , beg\u2019n , begun\nFigure 2.6 The use of the period .to specify any character.\nThe wildcard is often used together with the Kleene star to mean \u201cany string of\ncharacters\u201d. For example, suppose we want to \ufb01nd any line in which a particular\nword, for example, aardvark , appears twice. We can specify this with the regular\nexpression /aardvark.*aardvark/ . 8CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\nAnchors are special characters that anchor regular expressions to particular places anchors\nin a string. The most common anchors are the caret ^and the dollar sign $. The caret\n^matches the start of a line. The pattern /^The/ matches the word Theonly at the\nstart of a line. Thus, the caret ^has three uses: to match the start of a line, to in-\ndicate a negation inside of square brackets, and just to mean a caret. (What are the\ncontexts that allow grep or Python to know which function a given caret is supposed\nto have?) The dollar sign $matches the end of a line. So the pattern  $is a useful\npattern for matching a space at the end of a line, and /^The dog\\.$/ matches a\nline that contains only the phrase The dog. (We have to use the backslash here since\nwe want the .to mean \u201cperiod\u201d and not the wildcard.)\nRegex Match\n^ start of line\n$ end of line\n\\b word boundary\n\\B non-word boundary\nFigure 2.7 Anchors in regular expressions.\nThere are also two other anchors: \\bmatches a word boundary, and \\Bmatches\na non word-boundary. Thus, /\\bthe\\b/ matches the word thebut not the word\nother . A \u201cword\u201d for the purposes of a regular expression is de\ufb01ned based on the\nde\ufb01nition of words in programming languages as a sequence of digits, underscores,\nor letters. Thus /\\b99\\b/ will match the string 99inThere are 99 bottles of beer on\nthe wall (because 99 follows a space) but not 99inThere are 299 bottles of beer on\nthe wall (since 99 follows a number). But it will match 99in$99(since 99follows\na dollar sign ($), which is not a digit, underscore, or letter).\n2.1.2 Disjunction, Grouping, and Precedence\nSuppose we need to search for texts about pets; perhaps we are particularly interested\nin cats and dogs. In such a case, we might want to search for either the string cator\nthe string dog. Since we can\u2019t use the square brackets to search for \u201ccat or dog\u201d (why\ncan\u2019t we say /[catdog]/ ?), we need a new operator, the disjunction operator, also disjunction\ncalled the pipe symbol|. The pattern /cat|dog/ matches either the string cator\nthe string dog.\nSometimes we need to use this disjunction operator in the midst of a larger se-\nquence. For example, suppose I want to search for information about pet \ufb01sh for\nmy cousin David. How can I specify both guppy andguppies ? We cannot simply\nsay/guppy|ies/ , because that would match only the strings guppy andies. This\nis because sequences like guppy take precedence over the disjunction operator |. precedence\nTo make the disjunction operator apply only to a speci\ufb01c pattern, we need to use the\nparenthesis operators (and). Enclosing a pattern in parentheses makes it act like\na single character for the purposes of neighboring operators like the pipe |and the\nKleene*. So the pattern /gupp(y|ies)/ would specify that we meant the disjunc-\ntion only to apply to the suf\ufb01xes yandies.\nThe parenthesis operator (is also useful when we are using counters like the\nKleene*. Unlike the |operator, the Kleene *operator applies by default only to\na single character, not to a whole sequence. Suppose we want to match repeated\ninstances of a string. Perhaps we have a line that has column labels of the form\nColumn 1 Column 2 Column 3 . The expression /Column [0-9]+ */ will not\nmatch any number of columns; instead, it will match a single column followed by 2.1 \u2022 R EGULAR EXPRESSIONS 9\nany number of spaces! The star here applies only to the space  that precedes it,\nnot to the whole sequence. With the parentheses, we could write the expression\n/(Column [0-9]+ *)*/ to match the word Column , followed by a number and\noptional spaces, the whole pattern repeated zero or more times.\nThis idea that one operator may take precedence over another, requiring us to\nsometimes use parentheses to specify what we mean, is formalized by the operator\nprecedence hierarchy for regular expressions. The following table gives the orderoperator\nprecedence\nof RE operator precedence, from highest precedence to lowest precedence.\nParenthesis ()\nCounters * + ? {}\nSequences and anchors the ^my end$\nDisjunction |\nThus, because counters have a higher precedence than sequences,\n/the*/ matches theeeee but not thethe . Because sequences have a higher prece-\ndence than disjunction, /the|any/ matches theoranybut not thany ortheny .\nPatterns can be ambiguous in another way. Consider the expression /[a-z]*/\nwhen matching against the text once upon a time . Since/[a-z]*/ matches zero or\nmore letters, this expression could match nothing, or just the \ufb01rst letter o,on,onc,\noronce . In these cases regular expressions always match the largest string they can;\nwe say that patterns are greedy , expanding to cover as much of a string as they can. greedy\nThere are, however, ways to enforce non-greedy matching, using another mean- non-greedy\ning of the ?quali\ufb01er. The operator *?is a Kleene star that matches as little text as *?\npossible. The operator +?is a Kleene plus that matches as little text as possible. +?\n2.1.3 A Simple Example\nSuppose we wanted to write a RE to \ufb01nd cases of the English article the. A simple\n(but incorrect) pattern might be:\n/the/ (2.2)\nOne problem is that this pattern will miss the word when it begins a sentence and\nhence is capitalized (i.e., The). This might lead us to the following pattern:\n/[tT]he/ (2.3)\nBut we will still incorrectly return texts with the embedded in other words (e.g.,\nother ortheology ). So we need to specify that we want instances with a word bound-\nary on both sides:\n/\\b[tT]he\\b/ (2.4)\nSuppose we wanted to do this without the use of /\\b/ . We might want this since\n/\\b/ won\u2019t treat underscores and numbers as word boundaries; but we might want\nto \ufb01nd thein some context where it might also have underlines or numbers nearby\n(the orthe25 ). We need to specify that we want instances in which there are no\nalphabetic letters on either side of the the:\n/[^a-zA-Z][tT]he[^a-zA-Z]/ (2.5)\nBut there is still one more problem with this pattern: it won\u2019t \ufb01nd the word thewhen\nit begins a line. This is because the regular expression [^a-zA-Z] , which we used 10 CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\nto avoid embedded instances of the, implies that there must be some single (although\nnon-alphabetic) character before the the. We can avoid this by specifying that before\nthethewe require either the beginning-of-line or a non-alphabetic character, and the\nsame at the end of the line:\n/(^|[^a-zA-Z])[tT]he([^a-zA-Z]|$)/ (2.6)\nThe process we just went through was based on \ufb01xing two kinds of errors: false pos-\nitives , strings that we incorrectly matched like other orthere , and false negatives , false positives\nfalse negatives strings that we incorrectly missed, like The. Addressing these two kinds of errors\ncomes up again and again in language processing. Reducing the overall error rate\nfor an application thus involves two antagonistic efforts:\n\u2022 Increasing precision (minimizing false positives)\n\u2022 Increasing recall (minimizing false negatives)\nWe\u2019ll come back to precision and recall with more precise de\ufb01nitions in Chapter 4.\n2.1.4 More Operators\nFigure 2.8 shows some aliases for common ranges, which can be used mainly to\nsave typing. Besides the Kleene * and Kleene + we can also use explicit numbers as\ncounters, by enclosing them in curly brackets. The operator /{3}/ means \u201cexactly\n3 occurrences of the previous character or expression\u201d. So /a\\.{24}z/ will match\nafollowed by 24 dots followed by z(but not afollowed by 23 or 25 dots followed\nby a z).\nRegex Expansion Match First Matches\n\\d [0-9] any digit Party of 5\n\\D [^0-9] any non-digit Blue moon\n\\w [a-zA-Z0-9_] any alphanumeric/underscore Daiyu\n\\W [^\\w] a non-alphanumeric !!!!\n\\s [ \\r\\t\\n\\f] whitespace (space, tab) inConcord\n\\S [^\\s] Non-whitespace in Concord\nFigure 2.8 Aliases for common sets of characters.\nA range of numbers can also be speci\ufb01ed. So /{n,m}/ speci\ufb01es from ntom\noccurrences of the previous char or expression, and /{n,}/ means at least noccur-\nrences of the previous expression. REs for counting are summarized in Fig. 2.9.\nRegex Match\n* zero or more occurrences of the previous char or expression\n+ one or more occurrences of the previous char or expression\n? zero or one occurrence of the previous char or expression\n{n} exactly noccurrences of the previous char or expression\n{n,m} from ntomoccurrences of the previous char or expression\n{n,} at least noccurrences of the previous char or expression\n{,m} up to moccurrences of the previous char or expression\nFigure 2.9 Regular expression operators for counting.\nFinally, certain special characters are referred to by special notation based on the\nbackslash ( \\) (see Fig. 2.10). The most common of these are the newline character newline\n\\nand the tabcharacter\\t. To refer to characters that are special themselves (like\n.,*,[, and\\), precede them with a backslash, (i.e., /\\./ ,/\\*/ ,/\\[/ , and/\\\\/ ). 2.1 \u2022 R EGULAR EXPRESSIONS 11\nRegex Match First Patterns Matched\n\\* an asterisk \u201c*\u201d \u201cK* A*P*L*A*N\u201d\n\\. a period \u201c.\u201d \u201cDr. Livingston, I presume\u201d\n\\? a question mark \u201cWhy don\u2019t they come and lend a hand? \u201d\n\\n a newline\n\\t a tab\nFigure 2.10 Some characters that need to be escaped (via backslash).\n2.1.5 A More Complex Example\nLet\u2019s try out a more signi\ufb01cant example of the power of REs. Suppose our goal is\nhelp a user buy a computer on the Web who wants \u201cat least 6 GHz and 500 GB of\ndisk space for less than $1000\u201d. To do this kind of retrieval, we \ufb01rst need to be\nable to look for expressions like 6 GHz or500 GB or$999.99 . Let\u2019s work out some\nregular expressions for this task.\nFirst, let\u2019s complete our regular expression for prices. Here\u2019s a regular expres-\nsion for a dollar sign followed by a string of digits:\n/$[0-9]+/ (2.7)\nNote that the $character has a different function here than the end-of-line function\nwe discussed earlier. Most regular expression parsers are smart enough to realize\nthat$here doesn\u2019t mean end-of-line. (As a thought experiment, think about how\nregex parsers might \ufb01gure out the function of $from the context.)\nNow we just need to deal with fractions of dollars. We\u2019ll add a decimal point\nand two digits afterwards:\n/$[0-9]+\\.[0-9][0-9]/ (2.8)\nThis pattern only allows $199.99 but not $199 . We need to make the cents optional\nand to make sure we\u2019re at a word boundary:\n/(^|\\W)$[0-9]+(\\.[0-9][0-9])?\\b/ (2.9)\nOne last catch! This pattern allows prices like $199999.99 which would be far too\nexpensive! We need to limit the dollars:\n/(^|\\W)$[0-9]{0,3}(\\.[0-9][0-9])?\\b/ (2.10)\nFurther \ufb01xes (like avoiding matching a dollar sign with no price after it) are left as\nan exercise for the reader.\nHow about disk space? We\u2019ll need to allow for optional fractions again ( 5.5 GB );\nnote the use of ?for making the \ufb01nal soptional, and the use of / */ to mean \u201czero\nor more spaces\u201d since there might always be extra spaces lying around:\n/\\b[0-9]+(\\.[0-9]+)? *(GB|[Gg]igabytes?)\\b/ (2.11)\nModifying this regular expression so that it only matches more than 500 GB is left\nas an exercise for the reader.\n2.1.6 Substitution, Capture Groups, and ELIZA\nAn important use of regular expressions is in substitutions . For example, the substi- substitution 12 CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\ntution operator s/regexp1/pattern/ used in Python and in Unix commands like\nvimorsedallows a string characterized by a regular expression to be replaced by\nanother string:\ns/colour/color/ (2.12)\nIt is often useful to be able to refer to a particular subpart of the string matching\nthe \ufb01rst pattern. For example, suppose we wanted to put angle brackets around all\nintegers in a text, for example, changing the 35 boxes tothe<35>boxes . We\u2019d\nlike a way to refer to the integer we\u2019ve found so that we can easily add the brackets.\nTo do this, we put parentheses (and)around the \ufb01rst pattern and use the number\noperator\\1in the second pattern to refer back. Here\u2019s how it looks:\ns/([0-9]+)/<\\1>/ (2.13)\nThe parenthesis and number operators can also specify that a certain string or ex-\npression must occur twice in the text. For example, suppose we are looking for the\npattern \u201cthe Xer they were, the Xer they will be\u201d, where we want to constrain the two\nX\u2019s to be the same string. We do this by surrounding the \ufb01rst X with the parenthesis\noperator, and replacing the second X with the number operator \\1, as follows:\n/the (.*)er they were, the \\1er they will be/ (2.14)\nHere the\\1will be replaced by whatever string matched the \ufb01rst item in parentheses.\nSo this will match the bigger they were, the bigger they will be but not the bigger\nthey were, the faster they will be .\nThis use of parentheses to store a pattern in memory is called a capture group . capture group\nEvery time a capture group is used (i.e., parentheses surround a pattern), the re-\nsulting match is stored in a numbered register . If you match two different sets of register\nparentheses, \\2means whatever matched the second capture group. Thus\n/the (.*)er they (.*), the \\1er we \\2/ (2.15)\nwill match the faster they ran, the faster we ran but not the faster they ran, the faster\nwe ate . Similarly, the third capture group is stored in \\3, the fourth is \\4, and so on.\nParentheses thus have a double function in regular expressions; they are used\nto group terms for specifying the order in which operators should apply, and they\nare used to capture something in a register. Occasionally we might want to use\nparentheses for grouping, but don\u2019t want to capture the resulting pattern in a register.\nIn that case we use a non-capturing group , which is speci\ufb01ed by putting the specialnon-capturing\ngroup\ncommands ?:after the open parenthesis, in the form (?: pattern ) .\n/(?:some|a few) (people|cats) like some \\1/ (2.16)\nwill match some cats like some cats but not some cats like some some .\nSubstitutions and capture groups are very useful in implementing simple chat-\nbots like ELIZA (Weizenbaum, 1966). Recall that ELIZA simulates a Rogerian\npsychologist by carrying on conversations like the following:\nUser 1: Men are all alike.\nELIZA 1:IN WHAT WAY\nUser 2: They\u2019re always bugging us about something or other.\nELIZA 2:CAN YOU THINK OF A SPECIFIC EXAMPLE\nUser 3: Well, my boyfriend made me come here.\nELIZA 3:YOUR BOYFRIEND MADE YOU COME HERE\nUser 4: He says I\u2019m depressed much of the time.\nELIZA 4:I AM SORRY TO HEAR YOU ARE DEPRESSED 2.2 \u2022 W ORDS 13\nELIZA works by having a series or cascade of regular expression substitutions\neach of which matches and changes some part of the input lines. After the input\nis uppercased, substitutions change all instances of MYtoYOUR , and I\u2019MtoYOU\nARE, and so on. That way when ELIZA repeats back part of the user utterance, it\nwill seem to be referring correctly to the user. The next set of substitutions matches\nand replaces other patterns in the input. Here are some examples:\ns/.* YOU ARE (depressed|sad) .*/I AM SORRY TO HEAR YOU ARE \\1/\ns/.* YOU ARE (depressed|sad) .*/WHY DO YOU THINK YOU ARE \\1/\ns/.* all .*/IN WHAT WAY/\ns/.* always .*/CAN YOU THINK OF A SPECIFIC EXAMPLE/\nSince multiple substitutions can apply to a given input, substitutions are assigned\na rank and applied in order. Creating patterns is the topic of Exercise 2.3, and we\nreturn to the details of the ELIZA architecture in Chapter 15.\n2.1.7 Lookahead Assertions\nFinally, there will be times when we need to predict the future: look ahead in the\ntext to see if some pattern matches, but not yet advance the pointer we always keep\nto where we are in the text, so that we can then deal with the pattern if it occurs, but\nif it doesn\u2019t we can check for something else instead.\nThese lookahead assertions make use of the (?syntax that we saw in the previ- lookahead\nous section for non-capture groups. The operator (?= pattern) is true ifpattern\noccurs, but is zero-width , i.e. the match pointer doesn\u2019t advance. The operator zero-width\n(?! pattern) only returns true if a pattern does not match, but again is zero-width\nand doesn\u2019t advance the pointer. Negative lookahead is commonly used when we\nare parsing some complex pattern but want to rule out a special case. For example\nsuppose we want to match, at the beginning of a line, any single word that doesn\u2019t\nstart with \u201cV olcano\u201d. We can use negative lookahead to do this:\n/^(?!Volcano)[A-Za-z]+/ (2.17)\n2.2 Words\nBefore we talk about processing words, we need to decide what counts as a word.\nLet\u2019s start by looking at one particular corpus (plural corpora ), a computer-readable corpus\ncorpora collection of text or speech. For example the Brown corpus is a million-word col-\nlection of samples from 500 written English texts from different genres (newspa-\nper, \ufb01ction, non-\ufb01ction, academic, etc.), assembled at Brown University in 1963\u201364\n(Ku\u02c7cera and Francis, 1967). How many words are in the following Brown sentence?\nHe stepped out into the hall, was delighted to encounter\na water brother.\nThis sentence has 13 words if we don\u2019t count punctuation marks as words, 15\nif we count punctuation. Whether we treat period (\u201c .\u201d), comma (\u201c ,\u201d), and so on as\nwords depends on the task. Punctuation is critical for \ufb01nding boundaries of things\n(commas, periods, colons) and for identifying some aspects of meaning (question\nmarks, exclamation marks, quotation marks). For some tasks, like part-of-speech\ntagging or parsing or speech synthesis, we sometimes treat punctuation marks as if\nthey were separate words. 14 CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\nThe Switchboard corpus of American English telephone conversations between\nstrangers was collected in the early 1990s; it contains 2430 conversations averaging\n6 minutes each, totaling 240 hours of speech and about 3 million words (Godfrey\net al., 1992). Such corpora of spoken language introduce other complications with\nregard to de\ufb01ning words. Let\u2019s look at one utterance from Switchboard; an utter-\nance is the spoken correlate of a sentence: utterance\nI do uh main- mainly business data processing\nThis utterance has two kinds of dis\ufb02uencies . The broken-off word main- is dis\ufb02uency\ncalled a fragment . Words like uhandumare called \ufb01llers or\ufb01lled pauses . Should fragment\n\ufb01lled pause we consider these to be words? Again, it depends on the application. If we are\nbuilding a speech transcription system, we might want to eventually strip out the\ndis\ufb02uencies.\nBut we also sometimes keep dis\ufb02uencies around. Dis\ufb02uencies like uhorum\nare actually helpful in speech recognition in predicting the upcoming word, because\nthey may signal that the speaker is restarting the clause or idea, and so for speech\nrecognition they are treated as regular words. Because different people use differ-\nent dis\ufb02uencies they can also be a cue to speaker identi\ufb01cation. In fact Clark and\nFox Tree (2002) showed that uhandumhave different meanings. What do you think\nthey are?\nPerhaps most important, in thinking about what is a word, we need to distinguish\ntwo ways of talking about words that will be useful throughout the book. Word types word type\nare the number of distinct words in a corpus; if the set of words in the vocabulary is\nV, the number of types is the vocabulary size jVj. Word instances are the total num- word instance\nberNof running words.1If we ignore punctuation, the following Brown sentence\nhas 14 types and 16 instances:\nThey picnicked by the pool, then lay back on the grass and\nlooked at the stars.\nWe still have decisions to make! For example, should we consider a capitalized\nstring (like They ) and one that is uncapitalized (like they) to be the same word type?\nThe answer is that it depends on the task! They andthey might be lumped together\nas the same type in some tasks, like speech recognition, where we care more about\nthe sequence of words and less about the formatting, while for other tasks, such\nas deciding whether a particular word is a name of a person or location (named-\nentity tagging), capitalization is a useful feature and is retained. Sometimes we keep\naround two versions of a particular NLP model, one with capitalization and one\nwithout capitalization.\nCorpus Types =jVjInstances = N\nShakespeare 31 thousand 884 thousand\nBrown corpus 38 thousand 1 million\nSwitchboard telephone conversations 20 thousand 2.4 million\nCOCA 2 million 440 million\nGoogle n-grams 13 million 1 trillion\nFigure 2.11 Rough numbers of wordform types and instances for some English language\ncorpora. The largest, the Google n-grams corpus, contains 13 million types, but this count\nonly includes types appearing 40 or more times, so the true number would be much larger.\n1In earlier tradition, and occasionally still, you might see word instances referred to as word tokens , but\nwe now try to reserve the word token instead to mean the output of subword tokenization algorithms. 2.3 \u2022 C ORPORA 15\nHow many words are there in English? When we speak about the number of\nwords in the language, we are generally referring to word types. Fig. 2.11 shows\nthe rough numbers of types and instances computed from some English corpora.\nThe larger the corpora we look at, the more word types we \ufb01nd, and in fact this\nrelationship between the number of types jVjand number of instances Nis called\nHerdan\u2019s Law (Herdan, 1960) or Heaps\u2019 Law (Heaps, 1978) after its discoverers Herdan\u2019s Law\nHeaps\u2019 Law (in linguistics and information retrieval respectively). It is shown in Eq. 2.18, where\nkandbare positive constants, and 0 <b<1.\njVj=kNb(2.18)\nThe value of bdepends on the corpus size and the genre, but at least for the large\ncorpora in Fig. 2.11, branges from .67 to .75. Roughly then we can say that the\nvocabulary size for a text goes up signi\ufb01cantly faster than the square root of its\nlength in words.\nIt\u2019s sometimes useful to make a further distinction. Consider in\ufb02ected forms\nlikecats versus cat. We say these two words are different wordforms but have the\nsame lemma . Alemma is a set of lexical forms having the same stem, the same lemma\nmajor part-of-speech, and the same word sense. The wordform is the full in\ufb02ected wordform\nor derived form of the word. The two wordforms catandcats thus have the same\nlemma, which we can represent as cat.\nFor morphologically complex languages like Arabic, we often need to deal with\nlemmatization. For most tasks in English, however, wordforms are suf\ufb01cient, and\nwhen we talk about words in this book we almost always mean wordsforms (al-\nthough we will discuss basic algorithms for lemmatization and the related task of\nstemming below in Section 2.6). One of the situations even in English where we\ntalk about lemmas is when we measure the number of words in a dictionary. Dictio-\nnary entries orboldface forms are a very rough approximation to (an upper bound\non) the number of lemmas (since some lemmas have multiple boldface forms). The\n1989 edition of the Oxford English Dictionary had 615,000 entries.\nFinally, we should note that in practice, for many NLP applications (for example\nfor neural language modeling) we don\u2019t actually use words as our internal unit of\nrepresentation at all! We instead tokenize the input strings into tokens , which can\nbe words but can also be only parts of words. We\u2019ll return to this tokenization\nquestion when we introduce the BPE algorithm in Section 2.5.2.\n2.3 Corpora\nWords don\u2019t appear out of nowhere. Any particular piece of text that we study\nis produced by one or more speci\ufb01c speakers or writers, in a speci\ufb01c dialect of a\nspeci\ufb01c language, at a speci\ufb01c time, in a speci\ufb01c place, for a speci\ufb01c function.\nPerhaps the most important dimension of variation is the language. NLP algo-\nrithms are most useful when they apply across many languages. The world has 7097\nlanguages at the time of this writing, according to the online Ethnologue catalog\n(Simons and Fennig, 2018). It is important to test algorithms on more than one lan-\nguage, and particularly on languages with different properties; by contrast there is\nan unfortunate current tendency for NLP algorithms to be developed or tested just\non English (Bender, 2019). Even when algorithms are developed beyond English,\nthey tend to be developed for the of\ufb01cial languages of large industrialized nations 16 CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\n(Chinese, Spanish, Japanese, German etc.), but we don\u2019t want to limit tools to just\nthese few languages. Furthermore, most languages also have multiple varieties, of-\nten spoken in different regions or by different social groups. Thus, for example,\nif we\u2019re processing text that uses features of African American English ( AAE ) or AAE\nAfrican American Vernacular English (AA VE)\u2014the variations of English used by\nmillions of people in African American communities (King 2020)\u2014we must use\nNLP tools that function with features of those varieties. Twitter posts might use fea-\ntures often used by speakers of African American English, such as constructions like\niont(I don\u2019t in Mainstream American English ( MAE )), or talmbout corresponding MAE\nto MAE talking about , both examples that in\ufb02uence word segmentation (Blodgett\net al. 2016, Jones 2015).\nIt\u2019s also quite common for speakers or writers to use multiple languages in a\nsingle communicative act, a phenomenon called code switching . Code switching code switching\nis enormously common across the world; here are examples showing Spanish and\n(transliterated) Hindi code switching with English (Solorio et al. 2014, Jurgens et al.\n2017):\n(2.19) Por primera vez veo a @username actually being hateful! it was beautiful:)\n[For the \ufb01rst time I get to see @username actually being hateful! it was\nbeautiful:) ]\n(2.20) dost tha or ra- hega ... dont wory ... but dherya rakhe\n[\u201che was and will remain a friend ... don\u2019t worry ... but have faith\u201d]\nAnother dimension of variation is the genre. The text that our algorithms must\nprocess might come from newswire, \ufb01ction or non-\ufb01ction books, scienti\ufb01c articles,\nWikipedia, or religious texts. It might come from spoken genres like telephone\nconversations, business meetings, police body-worn cameras, medical interviews,\nor transcripts of television shows or movies. It might come from work situations\nlike doctors\u2019 notes, legal text, or parliamentary or congressional proceedings.\nText also re\ufb02ects the demographic characteristics of the writer (or speaker): their\nage, gender, race, socioeconomic class can all in\ufb02uence the linguistic properties of\nthe text we are processing.\nAnd \ufb01nally, time matters too. Language changes over time, and for some lan-\nguages we have good corpora of texts from different historical periods.\nBecause language is so situated, when developing computational models for lan-\nguage processing from a corpus, it\u2019s important to consider who produced the lan-\nguage, in what context, for what purpose. How can a user of a dataset know all these\ndetails? The best way is for the corpus creator to build a datasheet (Gebru et al., datasheet\n2020) or data statement (Bender et al., 2021) for each corpus. A datasheet speci\ufb01es\nproperties of a dataset like:\nMotivation : Why was the corpus collected, by whom, and who funded it?\nSituation : When and in what situation was the text written/spoken? For example,\nwas there a task? Was the language originally spoken conversation, edited\ntext, social media communication, monologue vs. dialogue?\nLanguage variety : What language (including dialect/region) was the corpus in?\nSpeaker demographics : What was, e.g., the age or gender of the text\u2019s authors?\nCollection process : How big is the data? If it is a subsample how was it sampled?\nWas the data collected with consent? How was the data pre-processed, and\nwhat metadata is available?\nAnnotation process : What are the annotations, what are the demographics of the\nannotators, how were they trained, how was the data annotated? 2.4 \u2022 S IMPLE UNIXTOOLS FOR WORD TOKENIZATION 17\nDistribution : Are there copyright or other intellectual property restrictions?\n2.4 Simple Unix Tools for Word Tokenization\nBefore almost any natural language processing of a text, the text has to be normal-\nized, a task called text normalization . At least three tasks are commonly applied astext\nnormalization\npart of any normalization process:\n1. Tokenizing (segmenting) words\n2. Normalizing word formats\n3. Segmenting sentences\nIn the next sections we walk through each of these tasks, but we\u2019ll \ufb01rst start with\nan easy, if somewhat naive version of word tokenization and normalization (and fre-\nquency computation) that can be accomplished for English solely in a single Unix\ncommand-line, inspired by Church (1994). We\u2019ll make use of some Unix com-\nmands:tr, used to systematically change particular characters in the input; sort ,\nwhich sorts input lines in alphabetical order; and uniq , which collapses and counts\nadjacent identical lines.\nFor example let\u2019s begin with the \u2018complete words\u2019 of Shakespeare in one \ufb01le,\nsh.txt . We can use trto tokenize the words by changing every sequence of non-\nalphabetic characters to a newline (\u2019A-Za-z\u2019 means alphabetic and the -c option\ncomplements to non-alphabet, so together they mean to change every non-alphabetic\ncharacter into a newline. The -s (\u2018squeeze\u2019) option is used to replace the result\nof multiple consecutive changes into a single output, so a series of non-alphabetic\ncharacters in a row would all be \u2018squeezed\u2019 into a single newline):\ntr -sc 'A-Za-z' '\\n' < sh.txt\nThe output of this command will be:\nTHE\nSONNETS\nby\nWilliam\nShakespeare\nFrom\nfairest\ncreatures\nWe\n...\nNow that there is one word per line, we can sort the lines, and pass them to uniq\n-cwhich will collapse and count them:\ntr -sc 'A-Za-z' '\\n' < sh.txt | sort | uniq -c\nwith the following output:\n1945 A\n72 AARON\n19 ABBESS\n25 Aaron\n6 Abate 18 CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\n1 Abates\n5 Abbess\n6 Abbey\n3 Abbot\n...\nAlternatively, we can collapse all the upper case to lower case:\ntr -sc 'A-Za-z' '\\n' < sh.txt | tr A-Z a-z | sort | uniq -c\nwhose output is\n14725 a\n97 aaron\n1 abaissiez\n10 abandon\n2 abandoned\n2 abase\n1 abash\n14 abate\n3 abated\n3 abatement\n...\nNow we can sort again to \ufb01nd the frequent words. The -noption tosort means\nto sort numerically rather than alphabetically, and the -roption means to sort in\nreverse order (highest-to-lowest):\ntr -sc 'A-Za-z' '\\n' < sh.txt | tr A-Z a-z | sort | uniq -c | sort -n -r\nThe results show that the most frequent words in Shakespeare, as in any other\ncorpus, are the short function words like articles, pronouns, prepositions:\n27378 the\n26084 and\n22538 i\n19771 to\n17481 of\n14725 a\n13826 you\n12489 my\n11318 that\n11112 in\n...\nUnix tools of this sort can be very handy in building quick word count statistics\nfor any corpus in English. While in some versions of Unix these command-line tools\nalso correctly handle Unicode characters and so can be used for many languages,\nin general for handling most languages outside English we use more sophisticated\ntokenization algorithms.\n2.5 Word and Subword Tokenization\nThe simple Unix tools above were \ufb01ne for getting rough word statistics but more\nsophisticated algorithms are generally necessary for tokenization , the task of seg- tokenization 2.5 \u2022 W ORD AND SUBWORD TOKENIZATION 19\nmenting running text into words. There are roughly two classes of tokenization\nalgorithms. In top-down tokenization, we de\ufb01ne a standard and implement rules to\nimplement that kind of tokenization.\nBut more commonly instead of using words as the input to NLP algorithms we\nbreak up words into subword tokens , which can be words or parts of words or subword tokens\neven individual letters. These are derived via bottom-up tokenization, in which we\nuse simple statistics of letter sequences to come up with the vocabulary of subword\ntokens, and break up the input into those subwords.\n2.5.1 Top-down (rule-based) tokenization\nWhile the Unix command sequence just removed all the numbers and punctuation,\nfor most NLP applications we\u2019ll need to keep these in our tokenization. We often\nwant to break off punctuation as a separate token; commas are a useful piece of infor-\nmation for parsers, and periods help indicate sentence boundaries. But we\u2019ll often\nwant to keep the punctuation that occurs word internally, in examples like m.p.h. ,\nPh.D. ,AT&T , and cap\u2019n . Special characters and numbers will need to be kept in\nprices ($45.55) and dates ( 01/02/06 ); we don\u2019t want to segment that price into sepa-\nrate tokens of \u201c45\u201d and \u201c55\u201d. And there are URLs ( https://www.stanford.edu ),\nTwitter hashtags ( #nlproc ), or email addresses ( someone@cs.colorado.edu ).\nNumber expressions introduce complications; in addition to appearing at word\nboundaries, commas appear inside numbers in English, every three digits: 555,500.50 .\nTokenization differs by language; languages like Spanish, French, and German, for\nexample, use a comma to mark the decimal point, and spaces (or sometimes periods)\nwhere English puts commas, for example, 555 500,50 .\nA tokenizer can also be used to expand clitic contractions that are marked by clitic\napostrophes, converting what're to the two tokens what are , andwe're towe\nare. A clitic is a part of a word that can\u2019t stand on its own, and can only occur\nwhen it is attached to another word. Such contractions occur in other alphabetic\nlanguages, including French pronouns ( j'ai and articles l'homme ).\nDepending on the application, tokenization algorithms may also tokenize mul-\ntiword expressions like New York orrock 'n' roll as a single token, which re-\nquires a multiword expression dictionary of some sort. Tokenization is thus inti-\nmately tied up with named entity recognition , the task of detecting names, dates,\nand organizations (Chapter 17).\nOne commonly used tokenization standard is known as the Penn Treebank to-\nkenization standard, used for the parsed corpora (treebanks) released by the Lin-Penn Treebank\ntokenization\nguistic Data Consortium (LDC), the source of many useful datasets. This standard\nseparates out clitics ( doesn\u2019t becomes does plus n\u2019t), keeps hyphenated words to-\ngether, and separates out all punctuation (to save space we\u2019re showing visible spaces\n\u2018\u2019 between tokens, although newlines is a more common output):\nInput :\"The San Francisco-based restaurant,\" they said,\n\"doesn't charge $10\".\nOutput :\"TheSanFrancisco-based restaurant ,\"theysaid,\n\"doesn'tcharge $10\".\nIn practice, since tokenization is run before any other language processing, it\nneeds to be very fast. For word tokenziation we generally use deterministic algo-\nrithms based on regular expressions compiled into ef\ufb01cient \ufb01nite state automata.\nFor example, Fig. 2.12 shows a basic regular expression that can be used to tok- 20 CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\nenize English with the nltk.regexp tokenize function of the Python-based Nat-\nural Language Toolkit (NLTK) (Bird et al. 2009; https://www.nltk.org ).\n>>> text = 'That U.S.A. poster-print costs $12.40...'\n>>> pattern = r'''(?x) # set flag to allow verbose regexps\n... (?:[A-Z]\\.)+ # abbreviations, e.g. U.S.A.\n... | \\w+(?:-\\w+)* # words with optional internal hyphens\n... | \\$?\\d+(?:\\.\\d+)?%? # currency, percentages, e.g. $12.40, 82%\n... | \\.\\.\\. # ellipsis\n... | [][.,;\"'?():_`-] # these are separate tokens; includes ], [\n... '''\n>>> nltk.regexp_tokenize(text, pattern)\n['That', 'U.S.A.', 'poster-print', 'costs', '$12.40', '...']\nFigure 2.12 A Python trace of regular expression tokenization in the NLTK Python-based\nnatural language processing toolkit (Bird et al., 2009), commented for readability; the (?x)\nverbose \ufb02ag tells Python to strip comments and whitespace. Figure from Chapter 3 of Bird\net al. (2009).\nCarefully designed deterministic algorithms can deal with the ambiguities that\narise, such as the fact that the apostrophe needs to be tokenized differently when used\nas a genitive marker (as in the book\u2019s cover ), a quotative as in \u2018The other class\u2019, she\nsaid, or in clitics like they\u2019re .\nWord tokenization is more complex in languages like written Chinese, Japanese,\nand Thai, which do not use spaces to mark potential word-boundaries. In Chinese,\nfor example, words are composed of characters (called hanzi in Chinese). Each hanzi\ncharacter generally represents a single unit of meaning (called a morpheme ) and is\npronounceable as a single syllable. Words are about 2.4 characters long on average.\nBut deciding what counts as a word in Chinese is complex. For example, consider\nthe following sentence:\n(2.21)\u59da\u660e\u8fdb\u5165\u603b\u51b3\u8d5b y\u00b4ao m \u00b4\u0131ng j`\u0131n r`u z\u02c7ong ju \u00b4e s`ai\n\u201cYao Ming reaches the \ufb01nals\u201d\nAs Chen et al. (2017b) point out, this could be treated as 3 words (\u2018Chinese Tree-\nbank\u2019 segmentation):\n(2.22)\u59da\u660e\nYaoMing\u8fdb\u5165\nreaches\u603b\u51b3\u8d5b\n\ufb01nals\nor as 5 words (\u2018Peking University\u2019 segmentation):\n(2.23)\u59da\nYao\u660e\nMing\u8fdb\u5165\nreaches\u603b\noverall\u51b3\u8d5b\n\ufb01nals\nFinally, it is possible in Chinese simply to ignore words altogether and use characters\nas the basic elements, treating the sentence as a series of 7 characters:\n(2.24)\u59da\nYao\u660e\nMing\u8fdb\nenter\u5165\nenter\u603b\noverall\u51b3\ndecision\u8d5b\ngame\nIn fact, for most Chinese NLP tasks it turns out to work better to take characters\nrather than words as input, since characters are at a reasonable semantic level for\nmost applications, and since most word standards, by contrast, result in a huge vo-\ncabulary with large numbers of very rare words (Li et al., 2019b).\nHowever, for Japanese and Thai the character is too small a unit, and so algo-\nrithms for word segmentation are required. These can also be useful for Chineseword\nsegmentation 2.5 \u2022 W ORD AND SUBWORD TOKENIZATION 21\nin the rare situations where word rather than character boundaries are required. For\nthese situations we can use the subword tokenization algorithms introduced in the\nnext section.\n2.5.2 Byte-Pair Encoding: A Bottom-up Tokenization Algorithm\nThere is a third option to tokenizing text, one that is most commonly used by large\nlanguage models. Instead of de\ufb01ning tokens as words (whether delimited by spaces\nor more complex algorithms), or as characters (as in Chinese), we can use our data to\nautomatically tell us what the tokens should be. This is especially useful in dealing\nwith unknown words, an important problem in language processing. As we will\nsee in the next chapter, NLP algorithms often learn some facts about language from\none corpus (a training corpus) and then use these facts to make decisions about a\nseparate testcorpus and its language. Thus if our training corpus contains, say the\nwords low,new,newer , but not lower , then if the word lower appears in our test\ncorpus, our system will not know what to do with it.\nTo deal with this unknown word problem, modern tokenizers automatically in-\nduce sets of tokens that include tokens smaller than words, called subwords . Sub- subwords\nwords can be arbitrary substrings, or they can be meaning-bearing units like the\nmorphemes -estor-er. (A morpheme is the smallest meaning-bearing unit of a lan-\nguage; for example the word unwashable has the morphemes un-,wash , and -able .)\nIn modern tokenization schemes, most tokens are words, but some tokens are fre-\nquently occurring morphemes or other subwords like -er. Every unseen word like\nlower can thus be represented by some sequence of known subword units, such as\nlowander, or even as a sequence of individual letters if necessary.\nMost tokenization schemes have two parts: a token learner , and a token seg-\nmenter . The token learner takes a raw training corpus (sometimes roughly pre-\nseparated into words, for example by whitespace) and induces a vocabulary, a set\nof tokens. The token segmenter takes a raw test sentence and segments it into the\ntokens in the vocabulary. Two algorithms are widely used: byte-pair encoding\n(Sennrich et al., 2016), and unigram language modeling (Kudo, 2018), There is\nalso a SentencePiece library that includes implementations of both of these (Kudo\nand Richardson, 2018a), and people often use the name SentencePiece to simply\nmean unigram language modeling tokenization.\nIn this section we introduce the simplest of the three, the byte-pair encoding or\nBPE algorithm (Sennrich et al., 2016); see Fig. 2.13. The BPE token learner begins BPE\nwith a vocabulary that is just the set of all individual characters. It then examines the\ntraining corpus, chooses the two symbols that are most frequently adjacent (say \u2018A\u2019,\n\u2018B\u2019), adds a new merged symbol \u2018AB\u2019 to the vocabulary, and replaces every adjacent\n\u2019A\u2019 \u2019B\u2019 in the corpus with the new \u2018AB\u2019. It continues to count and merge, creating\nnew longer and longer character strings, until kmerges have been done creating\nknovel tokens; kis thus a parameter of the algorithm. The resulting vocabulary\nconsists of the original set of characters plus knew symbols.\nThe algorithm is usually run inside words (not merging across word boundaries),\nso the input corpus is \ufb01rst white-space-separated to give a set of strings, each corre-\nsponding to the characters of a word, plus a special end-of-word symbol , and its\ncounts. Let\u2019s see its operation on the following tiny input corpus of 18 word tokens\nwith counts for each word (the word lowappears 5 times, the word newer 6 times,\nand so on), which would have a starting vocabulary of 11 letters: 22 CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\ncorpus vocabulary\n5l o w , d, e, i, l, n, o, r, s, t, w\n2l o w e s t\n6n e w e r\n3w i d e r\n2n e w\nThe BPE algorithm \ufb01rst counts all pairs of adjacent symbols: the most frequent\nis the pair e rbecause it occurs in newer (frequency of 6) and wider (frequency of\n3) for a total of 9 occurrences.2We then merge these symbols, treating eras one\nsymbol, and count again:\ncorpus vocabulary\n5l o w , d, e, i, l, n, o, r, s, t, w, er\n2l o w e s t\n6n e w er\n3w i d er\n2n e w\nNow the most frequent pair is er , which we merge; our system has learned\nthat there should be a token for word-\ufb01nal er, represented as er:\ncorpus vocabulary\n5l o w ,d,e,i,l,n,o,r,s,t,w,er,er\n2l o w e s t\n6n e w er\n3w i d er\n2n e w\nNextn e(total count of 8) get merged to ne:\ncorpus vocabulary\n5l o w ,d,e,i,l,n,o,r,s,t,w,er,er,ne\n2l o w e s t\n6ne w er\n3w i d er\n2ne w\nIf we continue, the next merges are:\nmerge current vocabulary\n(ne, w) ,d,e,i,l,n,o,r,s,t,w,er,er,ne,new\n(l, o) ,d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo\n(lo, w) ,d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low\n(new, er ) ,d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low,newer\n(low,) ,d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low,newer ,low\nOnce we\u2019ve learned our vocabulary, the token segmenter is used to tokenize a\ntest sentence. The token segmenter just runs on the merges we have learned from\nthe training data on the test data. It runs them greedily, in the order we learned them.\n(Thus the frequencies in the test data don\u2019t play a role, just the frequencies in the\ntraining data). So \ufb01rst we segment each test sentence word into characters. Then\nwe apply the \ufb01rst rule: replace every instance of e rin the test corpus with er, and\nthen the second rule: replace every instance of er in the test corpus with er,\nand so on. By the end, if the test corpus contained the character sequence n e w e\n2Note that there can be ties; we could have instead chosen to merge r \ufb01rst, since that also has a\nfrequency of 9. 2.6 \u2022 W ORD NORMALIZATION , LEMMATIZATION AND STEMMING 23\nfunction BYTE-PAIR ENCODING (strings C, number of merges k)returns vocab V\nV all unique characters in C # initial set of tokens is characters\nfori= 1tokdo # merge tokens ktimes\ntL,tR Most frequent pair of adjacent tokens in C\ntNEW tL+tR # make new token by concatenating\nV V+tNEW # update the vocabulary\nReplace each occurrence of tL,tRinCwith tNEW # and update the corpus\nreturn V\nFigure 2.13 The token learner part of the BPE algorithm for taking a corpus broken up\ninto individual characters or bytes, and learning a vocabulary by iteratively merging tokens.\nFigure adapted from Bostrom and Durrett (2020).\nr, it would be tokenized as a full word. But the characters of a new (unknown)\nword like l o w e r would be merged into the two tokens lower .\nOf course in real settings BPE is run with many thousands of merges on a very\nlarge input corpus. The result is that most words will be represented as full symbols,\nand only the very rare words (and unknown words) will have to be represented by\ntheir parts.\n2.6 Word Normalization, Lemmatization and Stemming\nWord normalization is the task of putting words or tokens in a standard format. The normalization\nsimplest case of word normalization is case folding . Mapping everything to lower case folding\ncase means that Woodchuck andwoodchuck are represented identically, which is\nvery helpful for generalization in many tasks, such as information retrieval or speech\nrecognition. For sentiment analysis and other text classi\ufb01cation tasks, information\nextraction, and machine translation, by contrast, case can be quite helpful and case\nfolding is generally not done. This is because maintaining the difference between,\nfor example, USthe country and usthe pronoun can outweigh the advantage in\ngeneralization that case folding would have provided for other words. Sometimes\nwe produce both cased (i.e. including both upper and lower case words or tokens)\nand uncased versions of language models.\nSystems that use BPE or other kinds of bottom-up tokenization may do no fur-\nther word normalization. In other NLP systems, we may want to do further nor-\nmalizations, like choosing a single normal form for words with multiple forms like\nUSA andUSoruh-huh anduhhuh . This standardization may be valuable, despite\nthe spelling information that is lost in the normalization process. For information\nretrieval or information extraction about the US, we might want to see information\nfrom documents whether they mention the USor theUSA.\n2.6.1 Lemmatization\nFor other natural language processing situations we also want two morphologically\ndifferent forms of a word to behave similarly. For example in web search, someone\nmay type the string woodchucks but a useful system might want to also return pages\nthat mention woodchuck with no s. This is especially common in morphologically\ncomplex languages like Polish, where for example the word Warsaw has different 24 CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\nendings when it is the subject ( Warszawa ), or after a preposition like \u201cin Warsaw\u201d ( w\nWarszawie ), or \u201cto Warsaw\u201d ( do Warszawy ), and so on. Lemmatization is the task lemmatization\nof determining that two words have the same root, despite their surface differences.\nThe words am,are, and ishave the shared lemma be; the words dinner anddinners\nboth have the lemma dinner . Lemmatizing each of these forms to the same lemma\nwill let us \ufb01nd all mentions of words in Polish like Warsaw . The lemmatized form\nof a sentence like He is reading detective stories would thus be He be read detective\nstory .\nHow is lemmatization done? The most sophisticated methods for lemmatization\ninvolve complete morphological parsing of the word. Morphology is the study of\nthe way words are built up from smaller meaning-bearing units called morphemes . morpheme\nTwo broad classes of morphemes can be distinguished: stems \u2014the central mor- stem\npheme of the word, supplying the main meaning\u2014and af\ufb01xes \u2014adding \u201cadditional\u201d af\ufb01x\nmeanings of various kinds. So, for example, the word foxconsists of one morpheme\n(the morpheme fox) and the word cats consists of two: the morpheme catand the\nmorpheme -s. A morphological parser takes a word like cats and parses it into the\ntwo morphemes catands, or parses a Spanish word like amaren (\u2018if in the future\nthey would love\u2019) into the morpheme amar \u2018to love\u2019, and the morphological features\n3PL (third person plural) and future subjunctive .\nStemming: The Porter Stemmer\nLemmatization algorithms can be complex. For this reason we sometimes make\nuse of a simpler but cruder method, which mainly consists of chopping off word-\n\ufb01nal af\ufb01xes. This naive version of morphological analysis is called stemming . For stemming\nexample, the classic Porter stemmer (Porter, 1980), when applied to the following Porter stemmer\nparagraph:\nThis was not the map we found in Billy Bones's chest, but\nan accurate copy, complete in all things-names and heights\nand soundings-with the single exception of the red crosses\nand the written notes.\nproduces the following stemmed output:\nThi wa not the map we found in Billi Bone s chest but an\naccur copi complet in all thing name and height and sound\nwith the singl except of the red cross and the written note\nThe algorithm is based on rewrite rules run in series, with the output of each pass\nfed as input to the next pass. Some sample rules (more at https://tartarus.org/\nmartin/PorterStemmer/ ):\nATIONAL!ATE (e.g., relational !relate)\nING!\u000fif the stem contains a vowel (e.g., motoring !motor)\nSSES!SS (e.g., grasses !grass)\nSimple stemmers can be useful in cases where we need to collapse across dif-\nferent variants of the same lemma. Nonetheless, they are less commonly used in\nmodern systems since they commit errors of both over-generalizating (lemmatizing\npolicy topolice ) and under-generalizing (not lemmatizing European toEurope )\n(Krovetz, 1993). 2.7 \u2022 S ENTENCE SEGMENTATION 25\n2.7 Sentence Segmentation\nSentence segmentation is another important step in text processing. The most use-sentence\nsegmentation\nful cues for segmenting a text into sentences are punctuation, like periods, question\nmarks, and exclamation points. Question marks and exclamation points are rela-\ntively unambiguous markers of sentence boundaries. Periods, on the other hand, are\nmore ambiguous. The period character \u201c.\u201d is ambiguous between a sentence bound-\nary marker and a marker of abbreviations like Mr.orInc.The previous sentence that\nyou just read showed an even more complex case of this ambiguity, in which the \ufb01nal\nperiod of Inc. marked both an abbreviation and the sentence boundary marker. For\nthis reason, sentence tokenization and word tokenization may be addressed jointly.\nIn general, sentence tokenization methods work by \ufb01rst deciding (based on rules\nor machine learning) whether a period is part of the word or is a sentence-boundary\nmarker. An abbreviation dictionary can help determine whether the period is part\nof a commonly used abbreviation; the dictionaries can be hand-built or machine-\nlearned (Kiss and Strunk, 2006), as can the \ufb01nal sentence splitter. In the Stanford\nCoreNLP toolkit (Manning et al., 2014), for example sentence splitting is rule-based,\na deterministic consequence of tokenization; a sentence ends when a sentence-ending\npunctuation (., !, or ?) is not already grouped with other characters into a token (such\nas for an abbreviation or number), optionally followed by additional \ufb01nal quotes or\nbrackets.\n2.8 Minimum Edit Distance\nMuch of natural language processing is concerned with measuring how similar two\nstrings are. For example in spelling correction, the user typed some erroneous\nstring\u2014let\u2019s say graffe \u2013and we want to know what the user meant. The user prob-\nably intended a word that is similar to graffe . Among candidate similar words,\nthe wordgiraffe , which differs by only one letter from graffe , seems intuitively\nto be more similar than, say grail orgraf , which differ in more letters. Another\nexample comes from coreference , the task of deciding whether two strings such as\nthe following refer to the same entity:\nStanford President Marc Tessier-Lavigne\nStanford University President Marc Tessier-Lavigne\nAgain, the fact that these two strings are very similar (differing by only one word)\nseems like useful evidence for deciding that they might be coreferent.\nEdit distance gives us a way to quantify both of these intuitions about string sim-\nilarity. More formally, the minimum edit distance between two strings is de\ufb01nedminimum edit\ndistance\nas the minimum number of editing operations (operations like insertion, deletion,\nsubstitution) needed to transform one string into another.\nThe gap between intention andexecution , for example, is 5 (delete an i, substi-\ntuteeforn, substitute xfort, insertc, substitute uforn). It\u2019s much easier to see\nthis by looking at the most important visualization for string distances, an alignment alignment\nbetween the two strings, shown in Fig. 2.14. Given two sequences, an alignment is\na correspondence between substrings of the two sequences. Thus, we say Ialigns\nwith the empty string, NwithE, and so on. Beneath the aligned strings is another\nrepresentation; a series of symbols expressing an operation list for converting the 26 CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\ntop string into the bottom string: dfor deletion, sfor substitution, ifor insertion.\nINTE*NTION\njjjjjjjjjj\n*EXECUTION\nd s s i s\nFigure 2.14 Representing the minimum edit distance between two strings as an alignment .\nThe \ufb01nal row gives the operation list for converting the top string into the bottom string: d for\ndeletion, s for substitution, i for insertion.\nWe can also assign a particular cost or weight to each of these operations. The\nLevenshtein distance between two sequences is the simplest weighting factor in\nwhich each of the three operations has a cost of 1 (Levenshtein, 1966)\u2014we assume\nthat the substitution of a letter for itself, for example, tfort, has zero cost. The Lev-\nenshtein distance between intention andexecution is 5. Levenshtein also proposed\nan alternative version of his metric in which each insertion or deletion has a cost of\n1 and substitutions are not allowed. (This is equivalent to allowing substitution, but\ngiving each substitution a cost of 2 since any substitution can be represented by one\ninsertion and one deletion). Using this version, the Levenshtein distance between\nintention andexecution is 8.\n2.8.1 The Minimum Edit Distance Algorithm\nHow do we \ufb01nd the minimum edit distance? We can think of this as a search task, in\nwhich we are searching for the shortest path\u2014a sequence of edits\u2014from one string\nto another.\nn t e n t i o ni n t e c n t i o ni n x e n t i o ndelinssubsti n t e n t i o n\nFigure 2.15 Finding the edit distance viewed as a search problem\nThe space of all possible edits is enormous, so we can\u2019t search naively. However,\nlots of distinct edit paths will end up in the same state (string), so rather than recom-\nputing all those paths, we could just remember the shortest path to a state each time\nwe saw it. We can do this by using dynamic programming . Dynamic programmingdynamic\nprogramming\nis the name for a class of algorithms, \ufb01rst introduced by Bellman (1957), that apply\na table-driven method to solve problems by combining solutions to subproblems.\nSome of the most commonly used algorithms in natural language processing make\nuse of dynamic programming, such as the Viterbi algorithm (Chapter 17) and the\nCKY algorithm for parsing (Chapter 18).\nThe intuition of a dynamic programming problem is that a large problem can\nbe solved by properly combining the solutions to various subproblems. Consider\nthe shortest path of transformed words that represents the minimum edit distance\nbetween the strings intention andexecution shown in Fig. 2.16.\nImagine some string (perhaps it is exention ) that is in this optimal path (whatever\nit is). The intuition of dynamic programming is that if exention is in the optimal 2.8 \u2022 M INIMUM EDITDISTANCE 27\nn t e n t i o ni n t e n t i o n\ne t e n t i o n\ne x e n t i o n\ne x e n u t i o n\ne x e c u t i o ndelete i\nsubstitute n by e\nsubstitute t by x\ninsert u\nsubstitute n by c\nFigure 2.16 Path from intention toexecution .\noperation list, then the optimal sequence must also include the optimal path from\nintention toexention . Why? If there were a shorter path from intention toexention ,\nthen we could use it instead, resulting in a shorter overall path, and the optimal\nsequence wouldn\u2019t be optimal, thus leading to a contradiction.\nThe minimum edit distance algorithm was named by Wagner and Fischerminimum edit\ndistance\nalgorithm(1974) but independently discovered by many people (see the Historical Notes sec-\ntion of Chapter 17).\nLet\u2019s \ufb01rst de\ufb01ne the minimum edit distance between two strings. Given two\nstrings, the source string Xof length n, and target string Yof length m, we\u2019ll de\ufb01ne\nD[i;j]as the edit distance between X[1::i]andY[1::j], i.e., the \ufb01rst icharacters of X\nand the \ufb01rst jcharacters of Y. The edit distance between XandYis thus D[n;m].\nWe\u2019ll use dynamic programming to compute D[n;m]bottom up, combining so-\nlutions to subproblems. In the base case, with a source substring of length ibut an\nempty target string, going from icharacters to 0 requires ideletes. With a target\nsubstring of length jbut an empty source going from 0 characters to jcharacters\nrequires jinserts. Having computed D[i;j]for small i;jwe then compute larger\nD[i;j]based on previously computed smaller values. The value of D[i;j]is com-\nputed by taking the minimum of the three possible paths through the matrix which\narrive there:\nD[i;j] =min8\n<\n:D[i\u00001;j]+del-cost (source [i])\nD[i;j\u00001]+ins-cost (target [j])\nD[i\u00001;j\u00001]+sub-cost (source [i];target [j])(2.25)\nWe mentioned above two versions of Levenshtein distance, one in which substitu-\ntions cost 1 and one in which substitutions cost 2 (i.e., are equivalent to an insertion\nplus a deletion). Let\u2019s here use that second version of Levenshtein distance in which\nthe insertions and deletions each have a cost of 1 (ins-cost( \u0001) = del-cost(\u0001) = 1), and\nsubstitutions have a cost of 2 (except substitution of identical letters has zero cost).\nUnder this version of Levenshtein, the computation for D[i;j]becomes:\nD[i;j] =min8\n>><\n>>:D[i\u00001;j]+1\nD[i;j\u00001]+1\nD[i\u00001;j\u00001]+\u001a2; if source [i]6=target [j]\n0; if source [i] =target [j](2.26)\nThe algorithm is summarized in Fig. 2.17; Fig. 2.18 shows the results of applying\nthe algorithm to the distance between intention andexecution with the version of\nLevenshtein in Eq. 2.26.\nAlignment Knowing the minimum edit distance is useful for algorithms like \ufb01nd-\ning potential spelling error corrections. But the edit distance algorithm is important 28 CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\nfunction MIN-EDIT-DISTANCE (source ,target )returns min-distance\nn LENGTH (source )\nm LENGTH (target )\nCreate a distance matrix D[n+1,m+1]\n#Initialization: the zeroth row and column is the distance from the empty string\nD[0,0] = 0\nforeach row ifrom 1tondo\nD[i,0] D[i-1,0] + del-cost (source [i])\nforeach column jfrom 1tomdo\nD[0,j] D[0,j-1] + ins-cost (target [j])\n#Recurrence relation:\nforeach row ifrom 1tondo\nforeach column jfrom 1tomdo\nD[i, j] MIN(D[i\u00001,j] +del-cost (source [i]),\nD[i\u00001,j\u00001] + sub-cost (source [i],target [j]),\nD[i,j\u00001] + ins-cost (target [j]))\n#Termination\nreturn D[n,m]\nFigure 2.17 The minimum edit distance algorithm, an example of the class of dynamic\nprogramming algorithms. The various costs can either be \ufb01xed (e.g., 8x;ins-cost (x) =1)\nor can be speci\ufb01c to the letter (to model the fact that some letters are more likely to be in-\nserted than others). We assume that there is no cost for substituting a letter for itself (i.e.,\nsub-cost (x;x) =0).\nSrcnTar # e x e c u t i o n\n# 0 1 2 3 4 5 6 7 8 9\ni 1 2 3 4 5 6 7 6 7 8\nn 2 3 4 5 6 7 8 7 8 7\nt 3 4 5 6 7 8 7 8 9 8\ne 4 3 4 5 6 7 8 9 10 9\nn 5 4 5 6 7 8 9 10 11 10\nt 6 5 6 7 8 9 8 9 10 11\ni 7 6 7 8 9 10 9 8 9 10\no 8 7 8 9 10 11 10 9 8 9\nn 9 8 9 10 11 12 11 10 9 8\nFigure 2.18 Computation of minimum edit distance between intention andexecution with\nthe algorithm of Fig. 2.17, using Levenshtein distance with cost of 1 for insertions or dele-\ntions, 2 for substitutions.\nin another way; with a small change, it can also provide the minimum cost align-\nment between two strings. Aligning two strings is useful throughout speech and\nlanguage processing. In speech recognition, minimum edit distance alignment is\nused to compute the word error rate (Chapter 16). Alignment plays a role in ma-\nchine translation, in which sentences in a parallel corpus (a corpus with a text in two\nlanguages) need to be matched to each other.\nTo extend the edit distance algorithm to produce an alignment, we can start by\nvisualizing an alignment as a path through the edit distance matrix. Figure 2.19\nshows this path with boldfaced cells. Each boldfaced cell represents an alignment 2.9 \u2022 S UMMARY 29\nof a pair of letters in the two strings. If two boldfaced cells occur in the same row,\nthere will be an insertion in going from the source to the target; two boldfaced cells\nin the same column indicate a deletion.\nFigure 2.19 also shows the intuition of how to compute this alignment path. The\ncomputation proceeds in two steps. In the \ufb01rst step, we augment the minimum edit\ndistance algorithm to store backpointers in each cell. The backpointer from a cell\npoints to the previous cell (or cells) that we came from in entering the current cell.\nWe\u2019ve shown a schematic of these backpointers in Fig. 2.19. Some cells have mul-\ntiple backpointers because the minimum extension could have come from multiple\nprevious cells. In the second step, we perform a backtrace . In a backtrace, we start backtrace\nfrom the last cell (at the \ufb01nal row and column), and follow the pointers back through\nthe dynamic programming matrix. Each complete path between the \ufb01nal cell and the\ninitial cell is a minimum distance alignment. Exercise 2.7 asks you to modify the\nminimum edit distance algorithm to store the pointers and compute the backtrace to\noutput an alignment.\n# e x e c u t i o n\n# 0 1 2 3 4 5 6 7 8 9\ni\"1- \" 2- \" 3- \" 4- \" 5- \" 6- \" 7-6 7 8\nn\"2- \" 3- \" 4- \" 5- \" 6- \" 7- \" 8\"7- \" 8-7\nt\"3- \" 4- \" 5- \" 6- \" 7- \" 8-7 \"8- \" 9\"8\ne\"4-3 4- 5 6 7 \"8- \" 9- \" 10\"9\nn\"5\"4- \" 5- \" 6- \" 7- \" 8- \" 9- \" 10- \" 11-\"10\nt\"6\"5- \" 6- \" 7- \" 8- \" 9-8 9 10 \"11\ni\"7\"6- \" 7- \" 8- \" 9- \" 10\"9-8 9 10\no\"8\"7- \" 8- \" 9- \" 10- \" 11\"10\"9-8 9\nn\"9\"8- \" 9- \" 10- \" 11- \" 12\"11\"10\"9-8\nFigure 2.19 When entering a value in each cell, we mark which of the three neighboring\ncells we came from with up to three arrows. After the table is full we compute an alignment\n(minimum edit path) by using a backtrace , starting at the 8in the lower-right corner and\nfollowing the arrows back. The sequence of bold cells represents one possible minimum\ncost alignment between the two strings, again using Levenshtein distance with cost of 1 for\ninsertions or deletions, 2 for substitutions. Diagram design after Gus\ufb01eld (1997).\nWhile we worked our example with simple Levenshtein distance, the algorithm\nin Fig. 2.17 allows arbitrary weights on the operations. For spelling correction, for\nexample, substitutions are more likely to happen between letters that are next to\neach other on the keyboard. The Viterbi algorithm is a probabilistic extension of\nminimum edit distance. Instead of computing the \u201cminimum edit distance\u201d between\ntwo strings, Viterbi computes the \u201cmaximum probability alignment\u201d of one string\nwith another. We\u2019ll discuss this more in Chapter 17.\n2.9 Summary\nThis chapter introduced a fundamental tool in language processing, the regular ex-\npression , and showed how to perform basic text normalization tasks including\nword segmentation andnormalization ,sentence segmentation , and stemming .\nWe also introduced the important minimum edit distance algorithm for comparing\nstrings. Here\u2019s a summary of the main points we covered about these ideas: 30 CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\n\u2022 The regular expression language is a powerful tool for pattern-matching.\n\u2022 Basic operations in regular expressions include concatenation of symbols,\ndisjunction of symbols ( [],|),counters (*,+, and{n,m} ),anchors (^,$)\nand precedence operators ( (,)).\n\u2022Word tokenization and normalization are generally done by cascades of\nsimple regular expression substitutions or \ufb01nite automata.\n\u2022 The Porter algorithm is a simple and ef\ufb01cient way to do stemming , stripping\noff af\ufb01xes. It does not have high accuracy but may be useful for some tasks.\n\u2022 The minimum edit distance between two strings is the minimum number of\noperations it takes to edit one into the other. Minimum edit distance can be\ncomputed by dynamic programming , which also results in an alignment of\nthe two strings.\nBibliographical and Historical Notes\nKleene 1951; 1956 \ufb01rst de\ufb01ned regular expressions and the \ufb01nite automaton, based\non the McCulloch-Pitts neuron. Ken Thompson was one of the \ufb01rst to build regular\nexpressions compilers into editors for text searching (Thompson, 1968). His edi-\ntoredincluded a command \u201cg/regular expression/p\u201d, or Global Regular Expression\nPrint, which later became the Unix grep utility.\nText normalization algorithms have been applied since the beginning of the\n\ufb01eld. One of the earliest widely used stemmers was Lovins (1968). Stemming\nwas also applied early to the digital humanities, by Packard (1973), who built an\naf\ufb01x-stripping morphological parser for Ancient Greek. Currently a wide vari-\nety of code for tokenization and normalization is available, such as the Stanford\nTokenizer ( https://nlp.stanford.edu/software/tokenizer.shtml ) or spe-\ncialized tokenizers for Twitter (O\u2019Connor et al., 2010), or for sentiment ( http:\n//sentiment.christopherpotts.net/tokenizing.html ). See Palmer (2012)\nfor a survey of text preprocessing. NLTK is an essential tool that offers both useful\nPython libraries ( https://www.nltk.org ) and textbook descriptions (Bird et al.,\n2009) of many algorithms including text normalization and corpus interfaces.\nFor more on Herdan\u2019s law and Heaps\u2019 Law, see Herdan (1960, p. 28), Heaps\n(1978), Egghe (2007) and Baayen (2001); For more on edit distance, see Gus\ufb01eld\n(1997). Our example measuring the edit distance from \u2018intention\u2019 to \u2018execution\u2019\nwas adapted from Kruskal (1983). There are various publicly available packages to\ncompute edit distance, including Unix diff and the NIST sclite program (NIST,\n2005).\nIn his autobiography Bellman (1984) explains how he originally came up with\nthe term dynamic programming :\n\u201c...The 1950s were not good years for mathematical research. [the]\nSecretary of Defense ...had a pathological fear and hatred of the word,\nresearch... I decided therefore to use the word, \u201cprogramming\u201d. I\nwanted to get across the idea that this was dynamic, this was multi-\nstage... I thought, let\u2019s ... take a word that has an absolutely precise\nmeaning, namely dynamic... it\u2019s impossible to use the word, dynamic,\nin a pejorative sense. Try thinking of some combination that will pos-\nsibly give it a pejorative meaning. It\u2019s impossible. Thus, I thought\ndynamic programming was a good name. It was something not even a\nCongressman could object to.\u201d EXERCISES 31\nExercises\n2.1 Write regular expressions for the following languages.\n1. the set of all alphabetic strings;\n2. the set of all lower case alphabetic strings ending in a b;\n3. the set of all strings from the alphabet a;bsuch that each ais immedi-\nately preceded by and immediately followed by a b;\n2.2 Write regular expressions for the following languages. By \u201cword\u201d, we mean\nan alphabetic string separated from other words by whitespace, any relevant\npunctuation, line breaks, and so forth.\n1. the set of all strings with two consecutive repeated words (e.g., \u201cHum-\nbert Humbert\u201d and \u201cthe the\u201d but not \u201cthe bug\u201d or \u201cthe big bug\u201d);\n2. all strings that start at the beginning of the line with an integer and that\nend at the end of the line with a word;\n3. all strings that have both the word grotto and the word raven in them\n(but not, e.g., words like grottos that merely contain the word grotto );\n4. write a pattern that places the \ufb01rst word of an English sentence in a\nregister. Deal with punctuation.\n2.3 Implement an ELIZA-like program, using substitutions such as those described\non page 13. You might want to choose a different domain than a Rogerian psy-\nchologist, although keep in mind that you would need a domain in which your\nprogram can legitimately engage in a lot of simple repetition.\n2.4 Compute the edit distance (using insertion cost 1, deletion cost 1, substitution\ncost 1) of \u201cleda\u201d to \u201cdeal\u201d. Show your work (using the edit distance grid).\n2.5 Figure out whether drive is closer to brief or to divers and what the edit dis-\ntance is to each. You may use any version of distance that you like.\n2.6 Now implement a minimum edit distance algorithm and use your hand-computed\nresults to check your code.\n2.7 Augment the minimum edit distance algorithm to output an alignment; you\nwill need to store pointers and add a stage to compute the backtrace. 32 CHAPTER 3 \u2022 N- GRAM LANGUAGE MODELS\nCHAPTER\n3N-gram Language Models\n\u201cYou are uniformly charming!\u201d cried he, with a smile of associating and now\nand then I bowed and they perceived a chaise and four to wish for.\nRandom sentence generated from a Jane Austen trigram model\nPredicting is dif\ufb01cult\u2014especially about the future, as the old quip goes. But how\nabout predicting something that seems much easier, like the next word someone is\ngoing to say? What word, for example, is likely to follow\nThe water of Walden Pond is so beautifully ...\nYou might conclude that a likely word is blue , orgreen , orclear , but probably not\nrefrigerator northis . In this chapter we formalize this intuition by introducing\nlanguage models orLMs , models that assign a probability to each possible next language model\nLM word. Language models can also assign a probability to an entire sentence, telling\nus that the following sequence has a much higher probability of appearing in a text:\nall of a sudden I notice three guys standing on the sidewalk\nthan does this same set of words in a different order:\non guys all I of notice sidewalk three a sudden standing the\nWhy would we want to predict upcoming words, or know the probability of a sen-\ntence? One reason is for generation: choosing contextually better words. For ex-\nample we can correct grammar or spelling errors like Their are two midterms ,\nin whichThere was mistyped as Their , orEverything has improve , in which\nimprove should have been improved . The phrase There are is more probable\nthanTheir are , andhas improved thanhas improve , so a language model can\nhelp users select the more grammatical variant. Or for a speech system to recognize\nthat you said I will be back soonish and notI will be bassoon dish , it\nhelps to know that back soonish is a more probable sequence. Language models\ncan also help in augmentative and alternative communication (Trnka et al. 2007,\nKane et al. 2017). People can use AAC systems if they are physically unable to AAC\nspeak or sign but can instead use eye gaze or other movements to select words from\na menu. Word prediction can be used to suggest likely words for the menu.\nWord prediction is also central to NLP for another reason: large language mod-\nelsare built just by training them to predict words!! As we\u2019ll see in chapters 7-9,\nlarge language models learn an enormous amount about language solely from being\ntrained to predict upcoming words from neighboring words.\nIn this chapter we introduce the simplest kind of language model: the n-gram n-gram\nlanguage model. An n-gram is a sequence of nwords: a 2-gram (which we\u2019ll call\nbigram ) is a two-word sequence of words like The water , orwater of , and a 3-\ngram (a trigram ) is a three-word sequence of words like The water of , orwater 3.1 \u2022 N-G RAMS 33\nof Walden . But we also (in a bit of terminological ambiguity) use the word \u2018n-\ngram\u2019 to mean a probabilistic model that can estimate the probability of a word given\nthe n-1 previous words, and thereby also to assign probabilities to entire sequences.\nIn later chapters we will introduce the much more powerful neural large lan-\nguage models , based on the transformer architecture of Chapter 9. But because\nn-grams have a remarkably simple and clear formalization, we use them to intro-\nduce some major concepts of large language modeling, including training and test\nsets,perplexity ,sampling , and interpolation .\n3.1 N-Grams\nLet\u2019s begin with the task of computing P(wjh), the probability of a word wgiven\nsome history h. Suppose the history his \u201cThe water of Walden Pond is so\nbeautifully \u201d and we want to know the probability that the next word is blue :\nP(bluejThe water of Walden Pond is so beautifully ) (3.1)\nOne way to estimate this probability is directly from relative frequency counts: take a\nvery large corpus, count the number of times we see The water of Walden Pond\nis so beautifully , and count the number of times this is followed by blue . This\nwould be answering the question \u201cOut of the times we saw the history h, how many\ntimes was it followed by the word w\u201d, as follows:\nP(bluejThe water of Walden Pond is so beautifully ) =\nC(The water of Walden Pond is so beautifully blue )\nC(The water of Walden Pond is so beautifully )(3.2)\nIf we had a large enough corpus, we could compute these two counts and estimate\nthe probability from Eq. 3.2. But even the entire web isn\u2019t big enough to give us\ngood estimates for counts of entire sentences. This is because language is creative ;\nnew sentences are invented all the time, and we can\u2019t expect to get accurate counts\nfor such large objects as entire sentences. For this reason, we\u2019ll need more clever\nways to estimate the probability of a word wgiven a history h, or the probability of\nan entire word sequence W.\nLet\u2019s start with some notation. First, throughout this chapter we\u2019ll continue to\nrefer to words , although in practice we usually compute language models over to-\nkens like the BPE tokens of page 21. To represent the probability of a particular\nrandom variable Xitaking on the value \u201cthe\u201d, or P(Xi=\u201cthe\u201d), we will use the\nsimpli\ufb01cation P(the). We\u2019ll represent a sequence of nwords either as w1:::wnor\nw1:n. Thus the expression w1:n\u00001means the string w1;w2;:::;wn\u00001, but we\u2019ll also\nbe using the equivalent notation w<n, which can be read as \u201call the elements of w\nfrom w1up to and including wn\u00001\u201d. For the joint probability of each word in a se-\nquence having a particular value P(X1=w1;X2=w2;X3=w3;:::;Xn=wn)we\u2019ll\nuseP(w1;w2;:::;wn).\nNow, how can we compute probabilities of entire sequences like P(w1;w2;:::;wn)?\nOne thing we can do is decompose this probability using the chain rule of proba-\nbility :\nP(X1:::Xn) = P(X1)P(X2jX1)P(X3jX1:2):::P(XnjX1:n\u00001)\n=nY\nk=1P(XkjX1:k\u00001) (3.3) 34 CHAPTER 3 \u2022 N- GRAM LANGUAGE MODELS\nApplying the chain rule to words, we get\nP(w1:n) = P(w1)P(w2jw1)P(w3jw1:2):::P(wnjw1:n\u00001)\n=nY\nk=1P(wkjw1:k\u00001) (3.4)\nThe chain rule shows the link between computing the joint probability of a sequence\nand computing the conditional probability of a word given previous words. Equa-\ntion 3.4 suggests that we could estimate the joint probability of an entire sequence of\nwords by multiplying together a number of conditional probabilities. But using the\nchain rule doesn\u2019t really seem to help us! We don\u2019t know any way to compute the\nexact probability of a word given a long sequence of preceding words, P(wnjw1:n\u00001).\nAs we said above, we can\u2019t just estimate by counting the number of times every word\noccurs following every long string in some corpus, because language is creative and\nany particular context might have never occurred before!\n3.1.1 The Markov assumption\nThe intuition of the n-gram model is that instead of computing the probability of a\nword given its entire history, we can approximate the history by just the last few\nwords.\nThebigram model, for example, approximates the probability of a word given bigram\nall the previous words P(wnjw1:n\u00001)by using only the conditional probability of the\npreceding word P(wnjwn\u00001). In other words, instead of computing the probability\nP(bluejThe water of Walden Pond is so beautifully ) (3.5)\nwe approximate it with the probability\nP(bluejbeautifully ) (3.6)\nWhen we use a bigram model to predict the conditional probability of the next word,\nwe are thus making the following approximation:\nP(wnjw1:n\u00001)\u0019P(wnjwn\u00001) (3.7)\nThe assumption that the probability of a word depends only on the previous word is\ncalled a Markov assumption. Markov models are the class of probabilistic models Markov\nthat assume we can predict the probability of some future unit without looking too\nfar into the past. We can generalize the bigram (which looks one word into the past)\nto the trigram (which looks two words into the past) and thus to the n-gram (which n-gram\nlooks n\u00001 words into the past).\nLet\u2019s see a general equation for this n-gram approximation to the conditional\nprobability of the next word in a sequence. We\u2019ll use Nhere to mean the n-gram\nsize, so N=2 means bigrams and N=3 means trigrams. Then we approximate the\nprobability of a word given its entire context as follows:\nP(wnjw1:n\u00001)\u0019P(wnjwn\u0000N+1:n\u00001) (3.8)\nGiven the bigram assumption for the probability of an individual word, we can com-\npute the probability of a complete word sequence by substituting Eq. 3.7 into Eq. 3.4:\nP(w1:n)\u0019nY\nk=1P(wkjwk\u00001) (3.9) 3.1 \u2022 N-G RAMS 35\n3.1.2 How to estimate probabilities\nHow do we estimate these bigram or n-gram probabilities? An intuitive way to\nestimate probabilities is called maximum likelihood estimation orMLE . We getmaximum\nlikelihood\nestimationthe MLE estimate for the parameters of an n-gram model by getting counts from\na corpus, and normalizing the counts so that they lie between 0 and 1. For proba- normalize\nbilistic models, normalizing means dividing by some total count so that the resulting\nprobabilities fall between 0 and 1 and sum to 1.\nFor example, to compute a particular bigram probability of a word wngiven a\nprevious word wn\u00001, we\u2019ll compute the count of the bigram C(wn\u00001wn)and normal-\nize by the sum of all the bigrams that share the same \ufb01rst word wn\u00001:\nP(wnjwn\u00001) =C(wn\u00001wn)P\nwC(wn\u00001w)(3.10)\nWe can simplify this equation, since the sum of all bigram counts that start with\na given word wn\u00001must be equal to the unigram count for that word wn\u00001(the reader\nshould take a moment to be convinced of this):\nP(wnjwn\u00001) =C(wn\u00001wn)\nC(wn\u00001)(3.11)\nLet\u2019s work through an example using a mini-corpus of three sentences. We\u2019ll\n\ufb01rst need to augment each sentence with a special symbol <s>at the beginning\nof the sentence, to give us the bigram context of the \ufb01rst word. We\u2019ll also need a\nspecial end-symbol </s> .1\n<s> I am Sam </s>\n<s> Sam I am </s>\n<s> I do not like green eggs and ham </s>\nHere are the calculations for some of the bigram probabilities from this corpus\nP(I|<s> ) =2\n3=0:67 P(Sam|<s> ) =1\n3=0:33 P(am|I) =2\n3=0:67\nP(</s>|Sam ) =1\n2=0:5 P(Sam|am ) =1\n2=0:5 P(do|I) =1\n3=0:33\nFor the general case of MLE n-gram parameter estimation:\nP(wnjwn\u0000N+1:n\u00001) =C(wn\u0000N+1:n\u00001wn)\nC(wn\u0000N+1:n\u00001)(3.12)\nEquation 3.12 (like Eq. 3.11) estimates the n-gram probability by dividing the\nobserved frequency of a particular sequence by the observed frequency of a pre\ufb01x.\nThis ratio is called a relative frequency . We said above that this use of relativerelative\nfrequency\nfrequencies as a way to estimate probabilities is an example of maximum likelihood\nestimation or MLE. In MLE, the resulting parameter set maximizes the likelihood of\nthe training set Tgiven the model M(i.e., P(TjM)). For example, suppose the word\nChinese occurs 400 times in a corpus of a million words. What is the probability\nthat a random word selected from some other text of, say, a million words will be the\nword Chinese ? The MLE of its probability is400\n1000000or 0:0004. Now 0 :0004 is not\nthe best possible estimate of the probability of Chinese occurring in all situations; it\n1We need the end-symbol to make the bigram grammar a true probability distribution. Without an end-\nsymbol, instead of the sentence probabilities of all sentences summing to one, the sentence probabilities\nfor all sentences of a given length would sum to one. This model would de\ufb01ne an in\ufb01nite set of probability\ndistributions, with one distribution per sentence length. See Exercise 3.5. 36 CHAPTER 3 \u2022 N- GRAM LANGUAGE MODELS\nmight turn out that in some other corpus or context Chinese is a very unlikely word.\nBut it is the probability that makes it most likely that Chinese will occur 400 times\nin a million-word corpus. We present ways to modify the MLE estimates slightly to\nget better probability estimates in Section 3.6.\nLet\u2019s move on to some examples from a real but tiny corpus, drawn from the\nnow-defunct Berkeley Restaurant Project, a dialogue system from the last century\nthat answered questions about a database of restaurants in Berkeley, California (Ju-\nrafsky et al., 1994). Here are some sample user queries (text-normalized, by lower\ncasing and with punctuation striped) (a sample of 9332 sentences is on the website):\ncan you tell me about any good cantonese restaurants close by\ntell me about chez panisse\ni\u2019m looking for a good place to eat breakfast\nwhen is caffe venezia open during the day\nFigure 3.1 shows the bigram counts from part of a bigram grammar from text-\nnormalized Berkeley Restaurant Project sentences. Note that the majority of the\nvalues are zero. In fact, we have chosen the sample words to cohere with each other;\na matrix selected from a random set of eight words would be even more sparse.\ni want to eat chinese food lunch spend\ni 5 827 0 9 0 0 0 2\nwant 2 0 608 1 6 6 5 1\nto 2 0 4 686 2 0 6 211\neat 0 0 2 0 16 2 42 0\nchinese 1 0 0 0 0 82 1 0\nfood 15 0 15 0 1 4 0 0\nlunch 2 0 0 0 0 1 0 0\nspend 1 0 1 0 0 0 0 0\nFigure 3.1 Bigram counts for eight of the words (out of V=1446) in the Berkeley Restau-\nrant Project corpus of 9332 sentences. Zero counts are in gray. Each cell shows the count of\nthe column label word following the row label word. Thus the cell in row iand column want\nmeans that want followed i827 times in the corpus.\nFigure 3.2 shows the bigram probabilities after normalization (dividing each cell\nin Fig. 3.1 by the appropriate unigram for its row, taken from the following set of\nunigram counts):\ni want to eat chinese food lunch spend\n2533 927 2417 746 158 1093 341 278\nHere are a few other useful probabilities:\nP(i|<s> ) =0:25 P(english|want ) =0:0011\nP(food|english ) =0:5 P(</s>|food ) =0:68\nNow we can compute the probability of sentences like I want English food or\nI want Chinese food by simply multiplying the appropriate bigram probabilities to-\ngether, as follows:\nP(<s> i want english food </s> )\n=P(i|<s> )P(want|i )P(english|want )\nP(food|english )P(</s>|food )\n=0:25\u00020:33\u00020:0011\u00020:5\u00020:68\n=0:000031 3.1 \u2022 N-G RAMS 37\ni want to eat chinese food lunch spend\ni 0.002 0.33 0 0.0036 0 0 0 0.00079\nwant 0.0022 0 0.66 0.0011 0.0065 0.0065 0.0054 0.0011\nto 0.00083 0 0.0017 0.28 0.00083 0 0.0025 0.087\neat 0 0 0.0027 0 0.021 0.0027 0.056 0\nchinese 0.0063 0 0 0 0 0.52 0.0063 0\nfood 0.014 0 0.014 0 0.00092 0.0037 0 0\nlunch 0.0059 0 0 0 0 0.0029 0 0\nspend 0.0036 0 0.0036 0 0 0 0 0\nFigure 3.2 Bigram probabilities for eight words in the Berkeley Restaurant Project corpus\nof 9332 sentences. Zero probabilities are in gray.\nWe leave it as Exercise 3.2 to compute the probability of i want chinese food .\nWhat kinds of linguistic phenomena are captured in these bigram statistics?\nSome of the bigram probabilities above encode some facts that we think of as strictly\nsyntactic in nature, like the fact that what comes after eatis usually a noun or an\nadjective, or that what comes after tois usually a verb. Others might be a fact about\nthe personal assistant task, like the high probability of sentences beginning with\nthe words I. And some might even be cultural rather than linguistic, like the higher\nprobability that people are looking for Chinese versus English food.\n3.1.3 Dealing with scale in large n-gram models\nIn practice, language models can be very large, leading to practical issues.\nLog probabilities Language model probabilities are always stored and computed\nin log space as log probabilities . This is because probabilities are (by de\ufb01nition) lesslog\nprobabilities\nthan or equal to 1, and so the more probabilities we multiply together, the smaller the\nproduct becomes. Multiplying enough n-grams together would result in numerical\nunder\ufb02ow. Adding in log space is equivalent to multiplying in linear space, so we\ncombine log probabilities by adding them. By adding log probabilities instead of\nmultiplying probabilities, we get results that are not as small. We do all computation\nand storage in log space, and just convert back into probabilities if we need to report\nprobabilities at the end by taking the exp of the logprob:\np1\u0002p2\u0002p3\u0002p4=exp(logp1+logp2+logp3+logp4) (3.13)\nIn practice throughout this book, we\u2019ll use log to mean natural log (ln) when the\nbase is not speci\ufb01ed.\nLonger context Although for pedagogical purposes we have only described bi-\ngram models, when there is suf\ufb01cient training data we use trigram models, which trigram\ncondition on the previous two words, or 4-gram or5-gram models. For these larger 4-gram\n5-gram n-grams, we\u2019ll need to assume extra contexts to the left and right of the sentence end.\nFor example, to compute trigram probabilities at the very beginning of the sentence,\nwe use two pseudo-words for the \ufb01rst trigram (i.e., P(I|<s><s> ).\nSome large n-gram datasets have been created, like the million most frequent\nn-grams drawn from the Corpus of Contemporary American English (COCA), a\ncurated 1 billion word corpus of American English (Davies, 2020), Google\u2019s Web\n5-gram corpus from 1 trillion words of English web text (Franz and Brants, 2006),\nor the Google Books Ngrams corpora (800 billion tokens from Chinese, English,\nFrench, German, Hebrew, Italian, Russian, and Spanish) (Lin et al., 2012a)). 38 CHAPTER 3 \u2022 N- GRAM LANGUAGE MODELS\nIt\u2019s even possible to use extremely long-range n-gram context. The in\ufb01ni-gram\n(\u00a5-gram) project (Liu et al., 2024) allows n-grams of any length. Their idea is to\navoid the expensive (in space and time) pre-computation of huge n-gram count ta-\nbles. Instead, n-gram probabilities with arbitrary n are computed quickly at inference\ntime by using an ef\ufb01cient representation called suf\ufb01x arrays. This allows computing\nof n-grams of every length for enormous corpora of 5 trillion tokens.\nEf\ufb01ciency considerations are important when building large n-gram language\nmodels. It is standard to quantize the probabilities using only 4-8 bits (instead of\n8-byte \ufb02oats), store the word strings on disk and represent them in memory only as\na 64-bit hash, and represent n-grams in special data structures like \u2018reverse tries\u2019.\nIt is also common to prune n-gram language models, for example by only keeping\nn-grams with counts greater than some threshold or using entropy to prune less-\nimportant n-grams (Stolcke, 1998). Ef\ufb01cient language model toolkits like KenLM\n(Hea\ufb01eld 2011, Hea\ufb01eld et al. 2013) use sorted arrays and use merge sorts to ef\ufb01-\nciently build the probability tables in a minimal number of passes through a large\ncorpus.\n3.2 Evaluating Language Models: Training and Test Sets\nThe best way to evaluate the performance of a language model is to embed it in\nan application and measure how much the application improves. Such end-to-end\nevaluation is called extrinsic evaluation . Extrinsic evaluation is the only way toextrinsic\nevaluation\nknow if a particular improvement in the language model (or any component) is really\ngoing to help the task at hand. Thus for evaluating n-gram language models that are\na component of some task like speech recognition or machine translation, we can\ncompare the performance of two candidate language models by running the speech\nrecognizer or machine translator twice, once with each language model, and seeing\nwhich gives the more accurate transcription.\nUnfortunately, running big NLP systems end-to-end is often very expensive. In-\nstead, it\u2019s helpful to have a metric that can be used to quickly evaluate potential\nimprovements in a language model. An intrinsic evaluation metric is one that mea-intrinsic\nevaluation\nsures the quality of a model independent of any application. In the next section we\u2019ll\nintroduce perplexity , which is the standard intrinsic metric for measuring language\nmodel performance, both for simple n-gram language models and for the more so-\nphisticated neural large language models of Chapter 9.\nIn order to evaluate any machine learning model, we need to have at least three\ndistinct data sets: the training set , the development set , and the test set . training set\ndevelopment\nset\ntest setThe training set is the data we use to learn the parameters of our model; for\nsimple n-gram language models it\u2019s the corpus from which we get the counts that\nwe normalize into the probabilities of the n-gram language model.\nThetest set is a different, held-out set of data, not overlapping with the training\nset, that we use to evaluate the model. We need a separate test set to give us an\nunbiased estimate of how well the model we trained can generalize when we apply\nit to some new unknown dataset. A machine learning model that perfectly captured\nthe training data, but performed terribly on any other data, wouldn\u2019t be much use\nwhen it comes time to apply it to any new data or problem! We thus measure the\nquality of an n-gram model by its performance on this unseen test set or test corpus.\nHow should we choose a training and test set? The test set should re\ufb02ect the\nlanguage we want to use the model for. If we\u2019re going to use our language model 3.3 \u2022 E VALUATING LANGUAGE MODELS : PERPLEXITY 39\nfor speech recognition of chemistry lectures, the test set should be text of chemistry\nlectures. If we\u2019re going to use it as part of a system for translating hotel booking re-\nquests from Chinese to English, the test set should be text of hotel booking requests.\nIf we want our language model to be general purpose, then the test set should be\ndrawn from a wide variety of texts. In such cases we might collect a lot of texts\nfrom different sources, and then divide it up into a training set and a test set. It\u2019s\nimportant to do the dividing carefully; if we\u2019re building a general purpose model,\nwe don\u2019t want the test set to consist of only text from one document, or one author,\nsince that wouldn\u2019t be a good measure of general performance.\nThus if we are given a corpus of text and want to compare the performance of\ntwo different n-gram models, we divide the data into training and test sets, and train\nthe parameters of both models on the training set. We can then compare how well\nthe two trained models \ufb01t the test set.\nBut what does it mean to \u201c\ufb01t the test set\u201d? The standard answer is simple:\nwhichever language model assigns a higher probability to the test set\u2014which\nmeans it more accurately predicts the test set\u2014is a better model. Given two proba-\nbilistic models, the better model is the one that better predicts the details of the test\ndata, and hence will assign a higher probability to the test data.\nSince our evaluation metric is based on test set probability, it\u2019s important not to\nlet the test sentences into the training set. Suppose we are trying to compute the\nprobability of a particular \u201ctest\u201d sentence. If our test sentence is part of the training\ncorpus, we will mistakenly assign it an arti\ufb01cially high probability when it occurs\nin the test set. We call this situation training on the test set . Training on the test\nset introduces a bias that makes the probabilities all look too high, and causes huge\ninaccuracies in perplexity , the probability-based metric we introduce below.\nEven if we don\u2019t train on the test set, if we test our language model on the\ntest set many times after making different changes, we might implicitly tune to its\ncharacteristics, by noticing which changes seem to make the model better. For this\nreason, we only want to run our model on the test set once, or a very few number of\ntimes, once we are sure our model is ready.\nFor this reason we normally instead have a third dataset called a developmentdevelopment\ntest\ntest set or, devset . We do all our testing on this dataset until the very end, and then\nwe test on the test once to see how good our model is.\nHow do we divide our data into training, development, and test sets? We want\nour test set to be as large as possible, since a small test set may be accidentally un-\nrepresentative, but we also want as much training data as possible. At the minimum,\nwe would want to pick the smallest test set that gives us enough statistical power\nto measure a statistically signi\ufb01cant difference between two potential models. It\u2019s\nimportant that the devset be drawn from the same kind of text as the test set, since\nits goal is to measure how we would do on the test set.\n3.3 Evaluating Language Models: Perplexity\nWe said above that we evaluate language models based on which one assigns a\nhigher probability to the test set. A better model is better at predicting upcoming\nwords, and so it will be less surprised by (i.e., assign a higher probability to) each\nword when it occurs in the test set. Indeed, a perfect language model would correctly\nguess each next word in a corpus, assigning it a probability of 1, and all the other\nwords a probability of zero. So given a test corpus, a better language model will 40 CHAPTER 3 \u2022 N- GRAM LANGUAGE MODELS\nassign a higher probability to it than a worse language model.\nBut in fact, we do not use raw probability as our metric for evaluating language\nmodels. The reason is that the probability of a test set (or any sequence) depends\non the number of words or tokens in it; the probability of a test set gets smaller the\nlonger the text. We\u2019d prefer a metric that is per-word, normalized by length, so we\ncould compare across texts of different lengths. The metric we use is, a function of\nprobability called perplexity , is one of the most important metrics in NLP, used for\nevaluating large language models as well as n-gram models.\nTheperplexity (sometimes abbreviated as PP or PPL) of a language model on a perplexity\ntest set is the inverse probability of the test set (one over the probability of the test\nset), normalized by the number of words (or tokens). For this reason it\u2019s sometimes\ncalled the per-word or per-token perplexity. We normalize by the number of words\nNby taking the Nth root. For a test set W=w1w2:::wN,:\nperplexity (W) = P(w1w2:::wN)\u00001\nN (3.14)\n=Ns\n1\nP(w1w2:::wN)\nOr we can use the chain rule to expand the probability of W:\nperplexity (W) =NvuutNY\ni=11\nP(wijw1:::wi\u00001)(3.15)\nNote that because of the inverse in Eq. 3.15, the higher the probability of the word\nsequence, the lower the perplexity. Thus the the lower the perplexity of a model on\nthe data, the better the model . Minimizing perplexity is equivalent to maximizing\nthe test set probability according to the language model. Why does perplexity use\nthe inverse probability? It turns out the inverse arises from the original de\ufb01nition\nof perplexity from cross-entropy rate in information theory; for those interested, the\nexplanation is in the advanced Section 3.7. Meanwhile, we just have to remember\nthat perplexity has an inverse relationship with probability.\nThe details of computing the perplexity of a test set Wdepends on which lan-\nguage model we use. Here\u2019s the perplexity of Wwith a unigram language model\n(just the geometric mean of the inverse of the unigram probabilities):\nperplexity (W) =NvuutNY\ni=11\nP(wi)(3.16)\nThe perplexity of Wcomputed with a bigram language model is still a geometric\nmean, but now of the inverse of the bigram probabilities:\nperplexity (W) =NvuutNY\ni=11\nP(wijwi\u00001)(3.17)\nWhat we generally use for word sequence in Eq. 3.15 or Eq. 3.17 is the entire\nsequence of words in some test set. Since this sequence will cross many sentence\nboundaries, if our vocabulary includes a between-sentence token <EOS> or separate\nbegin- and end-sentence markers <s> and</s> then we can include them in the 3.3 \u2022 E VALUATING LANGUAGE MODELS : PERPLEXITY 41\nprobability computation. If we do, then we also include one token per sentence in\nthe total count of word tokens N.2\nWe mentioned above that perplexity is a function of both the text and the lan-\nguage model: given a text W, different language models will have different perplex-\nities. Because of this, perplexity can be used to compare different language models.\nFor example, here we trained unigram, bigram, and trigram grammars on 38 million\nwords from the Wall Street Journal newspaper. We then computed the perplexity of\neach of these models on a WSJ test set using Eq. 3.16 for unigrams, Eq. 3.17 for\nbigrams, and the corresponding equation for trigrams. The table below shows the\nperplexity of the 1.5 million word test set according to each of the language models.\nUnigram Bigram Trigram\nPerplexity 962 170 109\nAs we see above, the more information the n-gram gives us about the word\nsequence, the higher the probability the n-gram will assign to the string. A trigram\nmodel is less surprised than a unigram model because it has a better idea of what\nwords might come next, and so it assigns them a higher probability. And the higher\nthe probability, the lower the perplexity (since as Eq. 3.15 showed, perplexity is\nrelated inversely to the probability of the test sequence according to the model). So\na lower perplexity tells us that a language model is a better predictor of the test set.\nNote that in computing perplexities, the language model must be constructed\nwithout any knowledge of the test set, or else the perplexity will be arti\ufb01cially low.\nAnd the perplexity of two language models is only comparable if they use identical\nvocabularies.\nAn (intrinsic) improvement in perplexity does not guarantee an (extrinsic) im-\nprovement in the performance of a language processing task like speech recognition\nor machine translation. Nonetheless, because perplexity usually correlates with task\nimprovements, it is commonly used as a convenient evaluation metric. Still, when\npossible a model\u2019s improvement in perplexity should be con\ufb01rmed by an end-to-end\nevaluation on a real task.\n3.3.1 Perplexity as Weighted Average Branching Factor\nIt turns out that perplexity can also be thought of as the weighted average branch-\ning factor of a language. The branching factor of a language is the number of\npossible next words that can follow any word. For example consider a mini arti\ufb01cial\nlanguage that is deterministic (no probabilities), any word can follow any word, and\nwhose vocabulary consists of only three colors:\nL=fred;blue;greeng (3.18)\nThe branching factor of this language is 3.\nNow let\u2019s make a probabilistic version of the same LM, let\u2019s call it A, where each\nword follows each other with equal probability1\n3(it was trained on a training set with\nequal counts for the 3 colors), and a test set T= \u201cred red red red blue \u201d.\nLet\u2019s \ufb01rst convince ourselves that if we compute the perplexity of this arti\ufb01cial\ndigit language on this test set (or any such test set) we indeed get 3. By Eq. 3.15, the\n2For example if we use both begin and end tokens, we would include the end-of-sentence marker </s>\nbut not the beginning-of-sentence marker <s>in our count of N; This is because the end-sentence token is\nfollowed directly by the begin-sentence token with probability almost 1, so we don\u2019t want the probability\nof that fake transition to in\ufb02uence our perplexity. 42 CHAPTER 3 \u2022 N- GRAM LANGUAGE MODELS\nperplexity of AonTis:\nperplexityA(T) = PA(red red red red blue )\u00001\n5\n= \u00121\n3\u00135!\u00001\n5\n=\u00121\n3\u0013\u00001\n=3 (3.19)\nBut now suppose redwas very likely in the training set a different LM B, and so B\nhas the following probabilities:\nP(red) =0:8P(green ) =0:1P(blue) =0:1 (3.20)\nWe should expect the perplexity of the same test set red red red red blue for\nlanguage model Bto be lower since most of the time the next color will be red, which\nis very predictable, i.e. has a high probability. So the probability of the test set will\nbe higher, and since perplexity is inversely related to probability, the perplexity will\nbe lower. Thus, although the branching factor is still 3, the perplexity or weighted\nbranching factor is smaller:\nperplexityB(T) = PB(red red red red blue )\u00001=5\n=0:04096\u00001\n5\n=0:527\u00001=1:89 (3.21)\n3.4 Sampling sentences from a language model\n010.06the.060.03of0.02a0.02toin.09.11.13.15\u2026however(p=0.0003)polyphonicp=0.0000018\u20260.02.66.99\u2026\nFigure 3.3 A visualization of the sampling distribution for sampling sentences by repeat-\nedly sampling unigrams. The blue bar represents the relative frequency of each word (we\u2019ve\nordered them from most frequent to least frequent, but the choice of order is arbitrary). The\nnumber line shows the cumulative probabilities. If we choose a random number between 0\nand 1, it will fall in an interval corresponding to some word. The expectation for the random\nnumber to fall in the larger intervals of one of the frequent words ( the,of,a) is much higher\nthan in the smaller interval of one of the rare words ( polyphonic ).\nOne important way to visualize what kind of knowledge a language model em-\nbodies is to sample from it. Sampling from a distribution means to choose random sampling\npoints according to their likelihood. Thus sampling from a language model\u2014which\nrepresents a distribution over sentences\u2014means to generate some sentences, choos-\ning each sentence according to its likelihood as de\ufb01ned by the model. Thus we are\nmore likely to generate sentences that the model thinks have a high probability and\nless likely to generate sentences that the model thinks have a low probability. 3.5 \u2022 G ENERALIZING VS .OVERFITTING THE TRAINING SET 43\nThis technique of visualizing a language model by sampling was \ufb01rst suggested\nvery early on by Shannon (1948) and Miller and Selfridge (1950). It\u2019s simplest to\nvisualize how this works for the unigram case. Imagine all the words of the English\nlanguage covering the number line between 0 and 1, each word covering an interval\nproportional to its frequency. Fig. 3.3 shows a visualization, using a unigram LM\ncomputed from the text of this book. We choose a random value between 0 and 1,\n\ufb01nd that point on the probability line, and print the word whose interval includes this\nchosen value. We continue choosing random numbers and generating words until\nwe randomly generate the sentence-\ufb01nal token </s> .\nWe can use the same technique to generate bigrams by \ufb01rst generating a ran-\ndom bigram that starts with <s>(according to its bigram probability). Let\u2019s say the\nsecond word of that bigram is w. We next choose a random bigram starting with w\n(again, drawn according to its bigram probability), and so on.\n3.5 Generalizing vs. over\ufb01tting the training set\nThe n-gram model, like many statistical models, is dependent on the training corpus.\nOne implication of this is that the probabilities often encode speci\ufb01c facts about a\ngiven training corpus. Another implication is that n-grams do a better and better job\nof modeling the training corpus as we increase the value of N.\nWe can use the sampling method from the prior section to visualize both of\nthese facts! To give an intuition for the increasing power of higher-order n-grams,\nFig. 3.4 shows random sentences generated from unigram, bigram, trigram, and 4-\ngram models trained on Shakespeare\u2019s works.\n1\u2013To him swallowed confess hear both. Which. Of save on trail for are ay device and\nrote life have\ngram \u2013Hill he late speaks; or! a more to leg less \ufb01rst you enter\n2\u2013Why dost stand forth thy canopy, forsooth; he is this palpable hit the King Henry. Live\nking. Follow.\ngram \u2013What means, sir. I confess she? then all sorts, he is trim, captain.\n3\u2013Fly, and will rid me these news of price. Therefore the sadness of parting, as they say,\n\u2019tis done.\ngram \u2013This shall forbid it should be branded, if renown made it empty.\n4\u2013King Henry. What! I will go seek the traitor Gloucester. Exeunt some of the watch. A\ngreat banquet serv\u2019d in;\ngram \u2013It cannot be but so.\nFigure 3.4 Eight sentences randomly generated from four n-grams computed from Shakespeare\u2019s works. All\ncharacters were mapped to lower-case and punctuation marks were treated as words. Output is hand-corrected\nfor capitalization to improve readability.\nThe longer the context, the more coherent the sentences. The unigram sen-\ntences show no coherent relation between words nor any sentence-\ufb01nal punctua-\ntion. The bigram sentences have some local word-to-word coherence (especially\nconsidering punctuation as words). The trigram sentences are beginning to look a\nlot like Shakespeare. Indeed, the 4-gram sentences look a little too much like Shake-\nspeare. The words It cannot be but so are directly from King John . This is because,\nnot to put the knock on Shakespeare, his oeuvre is not very large as corpora go 44 CHAPTER 3 \u2022 N- GRAM LANGUAGE MODELS\n(N=884;647;V=29;066), and our n-gram probability matrices are ridiculously\nsparse. There are V2=844;000;000 possible bigrams alone, and the number of\npossible 4-grams is V4=7\u00021017. Thus, once the generator has chosen the \ufb01rst\n3-gram ( It cannot be ), there are only seven possible next words for the 4th element\n(but,I,that,thus,this, and the period).\nTo get an idea of the dependence on the training set, let\u2019s look at LMs trained on a\ncompletely different corpus: the Wall Street Journal (WSJ) newspaper. Shakespeare\nand the WSJ are both English, so we might have expected some overlap between our\nn-grams for the two genres. Fig. 3.5 shows sentences generated by unigram, bigram,\nand trigram grammars trained on 40 million words from WSJ.\n1Months the my and issue of year foreign new exchange\u2019s september\nwere recession exchange new endorsed a acquire to six executivesgram\n2Last December through the way to preserve the Hudson corporation N.\nB. E. C. Taylor would seem to complete the major central planners one\ngram point \ufb01ve percent of U. S. E. has already old M. X. corporation of living\non information such as more frequently \ufb01shing to keep her\n3They also point to ninety nine point six billion dollars from two hundred\nfour oh six three percent of the rates of interest stores as Mexico and\ngram Brazil on market conditions\nFigure 3.5 Three sentences randomly generated from three n-gram models computed from\n40 million words of the Wall Street Journal , lower-casing all characters and treating punctua-\ntion as words. Output was then hand-corrected for capitalization to improve readability.\nCompare these examples to the pseudo-Shakespeare in Fig. 3.4. While they both\nmodel \u201cEnglish-like sentences\u201d, there is no overlap in the generated sentences, and\nlittle overlap even in small phrases. Statistical models are pretty useless as predictors\nif the training sets and the test sets are as different as Shakespeare and the WSJ.\nHow should we deal with this problem when we build n-gram models? One step\nis to be sure to use a training corpus that has a similar genre to whatever task we are\ntrying to accomplish. To build a language model for translating legal documents,\nwe need a training corpus of legal documents. To build a language model for a\nquestion-answering system, we need a training corpus of questions.\nIt is equally important to get training data in the appropriate dialect orvariety ,\nespecially when processing social media posts or spoken transcripts. For exam-\nple some tweets will use features of African American English (AAE)\u2014 the name\nfor the many variations of language used in African American communities (King,\n2020). Such features can include words like \ufb01nna \u2014an auxiliary verb that marks\nimmediate future tense \u2014that don\u2019t occur in other varieties, or spellings like denfor\nthen, in tweets like this one (Blodgett and O\u2019Connor, 2017):\n(3.22) Bored af den my phone \ufb01nna die!!!\nwhile tweets from English-based languages like Nigerian Pidgin have markedly dif-\nferent vocabulary and n-gram patterns from American English (Jurgens et al., 2017):\n(3.23) @username R u a wizard or wat gan sef: in d mornin - u tweet, afternoon - u\ntweet, nyt gan u dey tweet. beta get ur IT placement wiv twitter\nIs it possible for the testset nonetheless to have a word we have never seen be-\nfore? What happens if the word Jurafsky never occurs in our training set, but pops\nup in the test set? The answer is that although words might be unseen, we actu-\nally run our NLP algorithms not on words but on subword tokens . With subword 3.6 \u2022 S MOOTHING , INTERPOLATION ,AND BACKOFF 45\ntokenization (like the BPE algorithm of Chapter 2) any word can be modeled as a\nsequence of known smaller subwords, if necessary by a sequence of individual let-\nters. So although for convenience we\u2019ve been referring to words in this chapter, the\nlanguage model vocabulary is actually the set of tokens rather than words, and the\ntest set can never contain unseen tokens.\n3.6 Smoothing, Interpolation, and Backoff\nThere is a problem with using maximum likelihood estimates for probabilities: any\n\ufb01nite training corpus will be missing some perfectly acceptable English word se-\nquences. That is, cases where a particular n-gram never occurs in the training data\nbut appears in the test set. Perhaps our training corpus has the words ruby and\nslippers in it but just happens not to have the phrase ruby slippers .\nThese unseen sequences or zeros \u2014sequences that don\u2019t occur in the training set zeros\nbut do occur in the test set\u2014are a problem for two reasons. First, their presence\nmeans we are underestimating the probability of word sequences that might occur,\nwhich hurts the performance of any application we want to run on this data. Second,\nif the probability of any word in the test set is 0, the probability of the whole test\nset is 0. Perplexity is de\ufb01ned based on the inverse probability of the test set. Thus\nif some words in context have zero probability, we can\u2019t compute perplexity at all,\nsince we can\u2019t divide by 0!\nThe standard way to deal with putative \u201czero probability n-grams\u201d that should re-\nally have some non-zero probability is called smoothing ordiscounting . Smoothing smoothing\ndiscounting algorithms shave off a bit of probability mass from some more frequent events and\ngive it to unseen events. Here we\u2019ll introduce some simple smoothing algorithms:\nLaplace (add-one) smoothing ,stupid backoff , and n-gram interpolation .\n3.6.1 Laplace Smoothing\nThe simplest way to do smoothing is to add one to all the n-gram counts, before\nwe normalize them into probabilities. All the counts that used to be zero will now\nhave a count of 1, the counts of 1 will be 2, and so on. This algorithm is called\nLaplace smoothing . Laplace smoothing does not perform well enough to be usedLaplace\nsmoothing\nin modern n-gram models, but it usefully introduces many of the concepts that we\nsee in other smoothing algorithms, gives a useful baseline, and is also a practical\nsmoothing algorithm for other tasks like text classi\ufb01cation (Chapter 4).\nLet\u2019s start with the application of Laplace smoothing to unigram probabilities.\nRecall that the unsmoothed maximum likelihood estimate of the unigram probability\nof the word wiis its count cinormalized by the total number of word tokens N:\nP(wi) =ci\nN\nLaplace smoothing merely adds one to each count (hence its alternate name add-\nonesmoothing). Since there are Vwords in the vocabulary and each one was in- add-one\ncremented, we also need to adjust the denominator to take into account the extra V\nobservations. (What happens to our Pvalues if we don\u2019t increase the denominator?)\nPLaplace (wi) =ci+1\nN+V(3.24) 46 CHAPTER 3 \u2022 N- GRAM LANGUAGE MODELS\nInstead of changing both the numerator and denominator, it is convenient to describe\nhow a smoothing algorithm affects the numerator, by de\ufb01ning an adjusted count c\u0003.\nThis adjusted count is easier to compare directly with the MLE counts and can be\nturned into a probability like an MLE count by normalizing by N. To de\ufb01ne this\ncount, since we are only changing the numerator in addition to adding 1 we\u2019ll also\nneed to multiply by a normalization factorN\nN+V:\nc\u0003\ni= (ci+1)N\nN+V(3.25)\nWe can now turn c\u0003\niinto a probability P\u0003\niby normalizing by N.\nA related way to view smoothing is as discounting (lowering) some non-zero discounting\ncounts in order to get the probability mass that will be assigned to the zero counts.\nThus, instead of referring to the discounted counts c\u0003, we might describe a smooth-\ning algorithm in terms of a relative discount di, the ratio of the discounted counts to discount\nthe original counts:\ndi=c\u0003\ni\nci\nNow that we have the intuition for the unigram case, let\u2019s smooth our Berkeley\nRestaurant Project bigrams. Figure 3.6 shows the add-one smoothed counts for the\nbigrams in Fig. 3.1.\ni want to eat chinese food lunch spend\ni 6 828 1 10 1 1 1 3\nwant 3 1 609 2 7 7 6 2\nto 3 1 5 687 3 1 7 212\neat 1 1 3 1 17 3 43 1\nchinese 2 1 1 1 1 83 2 1\nfood 16 1 16 1 2 5 1 1\nlunch 3 1 1 1 1 2 1 1\nspend 2 1 2 1 1 1 1 1\nFigure 3.6 Add-one smoothed bigram counts for eight of the words (out of V=1446) in\nthe Berkeley Restaurant Project corpus of 9332 sentences. Previously-zero counts are in gray.\nFigure 3.7 shows the add-one smoothed probabilities for the bigrams in Fig. 3.2.\nRecall that normal bigram probabilities are computed by normalizing each row of\ncounts by the unigram count:\nPMLE(wnjwn\u00001) =C(wn\u00001wn)\nC(wn\u00001)(3.26)\nFor add-one smoothed bigram counts, we need to augment the unigram count by the\nnumber of total word types in the vocabulary V:\nPLaplace (wnjwn\u00001) =C(wn\u00001wn)+1P\nw(C(wn\u00001w)+1)=C(wn\u00001wn)+1\nC(wn\u00001)+V(3.27)\nThus, each of the unigram counts given in the previous section will need to be aug-\nmented by V=1446. The result is the smoothed bigram probabilities in Fig. 3.7.\nIt is often convenient to reconstruct the count matrix so we can see how much a\nsmoothing algorithm has changed the original counts. These adjusted counts can be 3.6 \u2022 S MOOTHING , INTERPOLATION ,AND BACKOFF 47\ni want to eat chinese food lunch spend\ni 0.0015 0.21 0.00025 0.0025 0.00025 0.00025 0.00025 0.00075\nwant 0.0013 0.00042 0.26 0.00084 0.0029 0.0029 0.0025 0.00084\nto 0.00078 0.00026 0.0013 0.18 0.00078 0.00026 0.0018 0.055\neat 0.00046 0.00046 0.0014 0.00046 0.0078 0.0014 0.02 0.00046\nchinese 0.0012 0.00062 0.00062 0.00062 0.00062 0.052 0.0012 0.00062\nfood 0.0063 0.00039 0.0063 0.00039 0.00079 0.002 0.00039 0.00039\nlunch 0.0017 0.00056 0.00056 0.00056 0.00056 0.0011 0.00056 0.00056\nspend 0.0012 0.00058 0.0012 0.00058 0.00058 0.00058 0.00058 0.00058\nFigure 3.7 Add-one smoothed bigram probabilities for eight of the words (out of V=1446) in the BeRP\ncorpus of 9332 sentences. Previously-zero probabilities are in gray.\ni want to eat chinese food lunch spend\ni 3.8 527 0.64 6.4 0.64 0.64 0.64 1.9\nwant 1.2 0.39 238 0.78 2.7 2.7 2.3 0.78\nto 1.9 0.63 3.1 430 1.9 0.63 4.4 133\neat 0.34 0.34 1 0.34 5.8 1 15 0.34\nchinese 0.2 0.098 0.098 0.098 0.098 8.2 0.2 0.098\nfood 6.9 0.43 6.9 0.43 0.86 2.2 0.43 0.43\nlunch 0.57 0.19 0.19 0.19 0.19 0.38 0.19 0.19\nspend 0.32 0.16 0.32 0.16 0.16 0.16 0.16 0.16\nFigure 3.8 Add-one reconstituted counts for eight words (of V=1446) in the BeRP corpus\nof 9332 sentences. Previously-zero counts are in gray.\ncomputed by Eq. 3.28. Figure 3.8 shows the reconstructed counts.\nc\u0003(wn\u00001wn) =[C(wn\u00001wn)+1]\u0002C(wn\u00001)\nC(wn\u00001)+V(3.28)\nNote that add-one smoothing has made a very big change to the counts. Com-\nparing Fig. 3.8 to the original counts in Fig. 3.1, we can see that C(want to )changed\nfrom 608 to 238! We can see this in probability space as well: P(tojwant)decreases\nfrom 0.66 in the unsmoothed case to 0.26 in the smoothed case. Looking at the dis-\ncount d(the ratio between new and old counts) shows us how strikingly the counts\nfor each pre\ufb01x word have been reduced; the discount for the bigram want to is 0.39,\nwhile the discount for Chinese food is 0.10, a factor of 10! The sharp change occurs\nbecause too much probability mass is moved to all the zeros.\n3.6.2 Add-k smoothing\nOne alternative to add-one smoothing is to move a bit less of the probability mass\nfrom the seen to the unseen events. Instead of adding 1 to each count, we add a\nfractional count k(0.5? 0.01?). This algorithm is therefore called add-k smoothing . add-k\nP\u0003\nAdd-k(wnjwn\u00001) =C(wn\u00001wn)+k\nC(wn\u00001)+kV(3.29)\nAdd-k smoothing requires that we have a method for choosing k; this can be\ndone, for example, by optimizing on a devset . Although add-k is useful for some\ntasks (including text classi\ufb01cation), it turns out that it still doesn\u2019t work well for\nlanguage modeling, generating counts with poor variances and often inappropriate\ndiscounts (Gale and Church, 1994). 48 CHAPTER 3 \u2022 N- GRAM LANGUAGE MODELS\n3.6.3 Language Model Interpolation\nThere is an alternative source of knowledge we can draw on to solve the problem\nof zero frequency n-grams. If we are trying to compute P(wnjwn\u00002wn\u00001)but we\nhave no examples of a particular trigram wn\u00002wn\u00001wn, we can instead estimate its\nprobability by using the bigram probability P(wnjwn\u00001). Similarly, if we don\u2019t have\ncounts to compute P(wnjwn\u00001), we can look to the unigram P(wn). In other words,\nsometimes using less context can help us generalize more for contexts that the model\nhasn\u2019t learned much about.\nThe most common way to use this n-gram hierarchy is called interpolation : interpolation\ncomputing a new probability by interpolating (weighting and combining) the tri-\ngram, bigram, and unigram probabilities.3In simple linear interpolation, we com-\nbine different order n-grams by linearly interpolating them. Thus, we estimate the\ntrigram probability P(wnjwn\u00002wn\u00001)by mixing together the unigram, bigram, and\ntrigram probabilities, each weighted by a l:\n\u02c6P(wnjwn\u00002wn\u00001) = l1P(wn)\n+l2P(wnjwn\u00001)\n+l3P(wnjwn\u00002wn\u00001) (3.30)\nThels must sum to 1, making Eq. 3.30 equivalent to a weighted average. In a\nslightly more sophisticated version of linear interpolation, each lweight is com-\nputed by conditioning on the context. This way, if we have particularly accurate\ncounts for a particular bigram, we assume that the counts of the trigrams based on\nthis bigram will be more trustworthy, so we can make the ls for those trigrams\nhigher and thus give that trigram more weight in the interpolation. Equation 3.31\nshows the equation for interpolation with context-conditioned weights, where each\nlambda takes an argument that is the two prior word context:\n\u02c6P(wnjwn\u00002wn\u00001) = l1(wn\u00002:n\u00001)P(wn)\n+l2(wn\u00002:n\u00001)P(wnjwn\u00001)\n+l3(wn\u00002:n\u00001)P(wnjwn\u00002wn\u00001) (3.31)\nHow are these lvalues set? Both the simple interpolation and conditional interpo-\nlation ls are learned from a held-out corpus. A held-out corpus is an additional held-out\ntraining corpus, so-called because we hold it out from the training data, that we use\nto set these lvalues.4We do so by choosing the lvalues that maximize the likeli-\nhood of the held-out corpus. That is, we \ufb01x the n-gram probabilities and then search\nfor the lvalues that\u2014when plugged into Eq. 3.30\u2014give us the highest probability\nof the held-out set. There are various ways to \ufb01nd this optimal set of ls. One way\nis to use the EMalgorithm, an iterative learning algorithm that converges on locally\noptimal ls (Jelinek and Mercer, 1980).\n3.6.4 Stupid Backoff\nAn alternative to interpolation is backoff . In a backoff model, if the n-gram we need backoff\n3We won\u2019t discuss the less-common alternative, called backoff , in which we use the trigram if the\nevidence is suf\ufb01cient for it, but if not we instead just use the bigram, otherwise the unigram. That is, we\nonly \u201cback off\u201d to a lower-order n-gram if we have zero evidence for a higher-order n-gram.\n4Held-out corpora are generally used to set hyperparameters , which are special parameters, unlike\nregular counts that are learned from the training data; we\u2019ll discuss hyperparameters in Chapter 7. 3.7 \u2022 A DVANCED : PERPLEXITY \u2019SRELATION TO ENTROPY 49\nhas zero counts, we approximate it by backing off to the (n-1)-gram. We continue\nbacking off until we reach a history that has some counts. For a backoff model to\ngive a correct probability distribution, we have to discount the higher-order n-grams discount\nto save some probability mass for the lower order n-grams. In practice, instead of\ndiscounting, it\u2019s common to use a much simpler non-discounted backoff algorithm\ncalled stupid backoff (Brants et al., 2007). stupid backoff\nStupid backoff gives up the idea of trying to make the language model a true\nprobability distribution. There is no discounting of the higher-order probabilities. If\na higher-order n-gram has a zero count, we simply backoff to a lower order n-gram,\nweighed by a \ufb01xed (context-independent) weight. This algorithm does not produce\na probability distribution, so we\u2019ll follow Brants et al. (2007) in referring to it as S:\nS(wijwi\u0000N+1:i\u00001) =8\n<\n:count (wi\u0000N+1:i)\ncount (wi\u0000N+1:i\u00001)if count (wi\u0000N+1:i)>0\nlS(wijwi\u0000N+2:i\u00001)otherwise(3.32)\nThe backoff terminates in the unigram, which has score S(w) =count (w)\nN. Brants et al.\n(2007) \ufb01nd that a value of 0.4 worked well for l.\n3.7 Advanced: Perplexity\u2019s Relation to Entropy\nWe introduced perplexity in Section 3.3 as a way to evaluate n-gram models on\na test set. A better n-gram model is one that assigns a higher probability to the\ntest data, and perplexity is a normalized version of the probability of the test set.\nThe perplexity measure actually arises from the information-theoretic concept of\ncross-entropy, which explains otherwise mysterious properties of perplexity (why\nthe inverse probability, for example?) and its relationship to entropy. Entropy is a Entropy\nmeasure of information. Given a random variable Xranging over whatever we are\npredicting (words, letters, parts of speech), the set of which we\u2019ll call c, and with a\nparticular probability function, call it p(x), the entropy of the random variable Xis:\nH(X) =\u0000X\nx2cp(x)log2p(x) (3.33)\nThe log can, in principle, be computed in any base. If we use log base 2, the\nresulting value of entropy will be measured in bits.\nOne intuitive way to think about entropy is as a lower bound on the number of\nbits it would take to encode a certain decision or piece of information in the optimal\ncoding scheme. Consider an example from the standard information theory textbook\nCover and Thomas (1991). Imagine that we want to place a bet on a horse race but\nit is too far to go all the way to Yonkers Racetrack, so we\u2019d like to send a short\nmessage to the bookie to tell him which of the eight horses to bet on. One way to\nencode this message is just to use the binary representation of the horse\u2019s number\nas the code; thus, horse 1 would be 001, horse 2010, horse 3011, and so on, with\nhorse 8 coded as 000. If we spend the whole day betting and each horse is coded\nwith 3 bits, on average we would be sending 3 bits per race.\nCan we do better? Suppose that the spread is the actual distribution of the bets\nplaced and that we represent it as the prior probability of each horse as follows: 50 CHAPTER 3 \u2022 N- GRAM LANGUAGE MODELS\nHorse 11\n2Horse 51\n64\nHorse 21\n4Horse 61\n64\nHorse 31\n8Horse 71\n64\nHorse 41\n16Horse 81\n64\nThe entropy of the random variable Xthat ranges over horses gives us a lower\nbound on the number of bits and is\nH(X) =\u0000i=8X\ni=1p(i)log2p(i)\n=\u00001\n2log21\n2\u00001\n4log21\n4\u00001\n8log21\n8\u00001\n16log21\n16\u00004(1\n64log21\n64)\n=2 bits (3.34)\nA code that averages 2 bits per race can be built with short encodings for more\nprobable horses, and longer encodings for less probable horses. For example, we\ncould encode the most likely horse with the code 0, and the remaining horses as 10,\nthen110,1110 ,111100 ,111101 ,111110 , and111111 .\nWhat if the horses are equally likely? We saw above that if we used an equal-\nlength binary code for the horse numbers, each horse took 3 bits to code, so the\naverage was 3. Is the entropy the same? In this case each horse would have a\nprobability of1\n8. The entropy of the choice of horses is then\nH(X) =\u0000i=8X\ni=11\n8log21\n8=\u0000log21\n8=3 bits (3.35)\nUntil now we have been computing the entropy of a single variable. But most of\nwhat we will use entropy for involves sequences . For a grammar, for example, we\nwill be computing the entropy of some sequence of words W=fw1;w2;:::; wng.\nOne way to do this is to have a variable that ranges over sequences of words. For\nexample we can compute the entropy of a random variable that ranges over all se-\nquences of words of length nin some language Las follows:\nH(w1;w2;:::; wn) =\u0000X\nw1:n2Lp(w1:n)logp(w1:n) (3.36)\nWe could de\ufb01ne the entropy rate (we could also think of this as the per-word entropy rate\nentropy ) as the entropy of this sequence divided by the number of words:\n1\nnH(w1:n) =\u00001\nnX\nw1:n2Lp(w1:n)logp(w1:n) (3.37)\nBut to measure the true entropy of a language, we need to consider sequences of\nin\ufb01nite length. If we think of a language as a stochastic process Lthat produces a\nsequence of words, and allow Wto represent the sequence of words w1;:::; wn, then\nL\u2019s entropy rate H(L)is de\ufb01ned as\nH(L) = lim\nn!\u00a51\nnH(w1:n)\n=\u0000lim\nn!\u00a51\nnX\nW2Lp(w1:n)logp(w1:n) (3.38) 3.7 \u2022 A DVANCED : PERPLEXITY \u2019SRELATION TO ENTROPY 51\nThe Shannon-McMillan-Breiman theorem (Algoet and Cover 1988, Cover and Thomas\n1991) states that if the language is regular in certain ways (to be exact, if it is both\nstationary and ergodic),\nH(L) =lim\nn!\u00a5\u00001\nnlogp(w1:n) (3.39)\nThat is, we can take a single sequence that is long enough instead of summing over\nall possible sequences. The intuition of the Shannon-McMillan-Breiman theorem\nis that a long-enough sequence of words will contain in it many other shorter se-\nquences and that each of these shorter sequences will reoccur in the longer sequence\naccording to their probabilities.\nA stochastic process is said to be stationary if the probabilities it assigns to a Stationary\nsequence are invariant with respect to shifts in the time index. In other words, the\nprobability distribution for words at time tis the same as the probability distribution\nat time t+1. Markov models, and hence n-grams, are stationary. For example, in\na bigram, Piis dependent only on Pi\u00001. So if we shift our time index by x,Pi+xis\nstill dependent on Pi+x\u00001. But natural language is not stationary, since as we show\nin Appendix D, the probability of upcoming words can be dependent on events that\nwere arbitrarily distant and time dependent. Thus, our statistical models only give\nan approximation to the correct distributions and entropies of natural language.\nTo summarize, by making some incorrect but convenient simplifying assump-\ntions, we can compute the entropy of some stochastic process by taking a very long\nsample of the output and computing its average log probability.\nNow we are ready to introduce cross-entropy . The cross-entropy is useful when cross-entropy\nwe don\u2019t know the actual probability distribution pthat generated some data. It\nallows us to use some m, which is a model of p(i.e., an approximation to p). The\ncross-entropy of monpis de\ufb01ned by\nH(p;m) =lim\nn!\u00a5\u00001\nnX\nW2Lp(w1;:::; wn)logm(w1;:::; wn) (3.40)\nThat is, we draw sequences according to the probability distribution p, but sum the\nlog of their probabilities according to m.\nAgain, following the Shannon-McMillan-Breiman theorem, for a stationary er-\ngodic process:\nH(p;m) =lim\nn!\u00a5\u00001\nnlogm(w1w2:::wn) (3.41)\nThis means that, as for entropy, we can estimate the cross-entropy of a model m\non some distribution pby taking a single sequence that is long enough instead of\nsumming over all possible sequences.\nWhat makes the cross-entropy useful is that the cross-entropy H(p;m)is an up-\nper bound on the entropy H(p). For any model m:\nH(p)\u0014H(p;m) (3.42)\nThis means that we can use some simpli\ufb01ed model mto help estimate the true en-\ntropy of a sequence of symbols drawn according to probability p. The more accurate\nmis, the closer the cross-entropy H(p;m)will be to the true entropy H(p). Thus,\nthe difference between H(p;m)andH(p)is a measure of how accurate a model is.\nBetween two models m1andm2, the more accurate model will be the one with the 52 CHAPTER 3 \u2022 N- GRAM LANGUAGE MODELS\nlower cross-entropy. (The cross-entropy can never be lower than the true entropy, so\na model cannot err by underestimating the true entropy.)\nWe are \ufb01nally ready to see the relation between perplexity and cross-entropy\nas we saw it in Eq. 3.41. Cross-entropy is de\ufb01ned in the limit as the length of the\nobserved word sequence goes to in\ufb01nity. We approximate this cross-entropy by\nrelying on a (suf\ufb01ciently long) sequence of \ufb01xed length. This approximation to the\ncross-entropy of a model M=P(wijwi\u0000N+1:i\u00001)on a sequence of words Wis\nH(W) =\u00001\nNlogP(w1w2:::wN) (3.43)\nTheperplexity of a model Pon a sequence of words Wis now formally de\ufb01ned as perplexity\n2 raised to the power of this cross-entropy:\nPerplexity (W) = 2H(W)\n=P(w1w2:::wN)\u00001\nN\n=Ns\n1\nP(w1w2:::wN)\n3.8 Summary\nThis chapter introduced language modeling via the n-gram model, a classic model\nthat allows us to introduce many of the basic concepts in language modeling.\n\u2022 Language models offer a way to assign a probability to a sentence or other\nsequence of words or tokens, and to predict a word or token from preceding\nwords or tokens.\n\u2022N-grams are perhaps the simplest kind of language model. They are Markov\nmodels that estimate words from a \ufb01xed window of previous words. N-gram\nmodels can be trained by counting in a training corpus and normalizing the\ncounts (the maximum likelihood estimate ).\n\u2022 N-gram language models can be evaluated on a test set using perplexity.\n\u2022 The perplexity of a test set according to a language model is a function of\nthe probability of the test set: the inverse test set probability according to the\nmodel, normalized by the length.\n\u2022Sampling from a language model means to generate some sentences, choos-\ning each sentence according to its likelihood as de\ufb01ned by the model.\n\u2022Smoothing algorithms provide a way to estimate probabilities for events that\nwere unseen in training. Commonly used smoothing algorithms for n-grams\ninclude add-1 smoothing, or rely on lower-order n-gram counts through inter-\npolation .\nBibliographical and Historical Notes\nThe underlying mathematics of the n-gram was \ufb01rst proposed by Markov (1913),\nwho used what are now called Markov chains (bigrams and trigrams) to predict\nwhether an upcoming letter in Pushkin\u2019s Eugene Onegin would be a vowel or a con-\nsonant. Markov classi\ufb01ed 20,000 letters as V or C and computed the bigram and BIBLIOGRAPHICAL AND HISTORICAL NOTES 53\ntrigram probability that a given letter would be a vowel given the previous one or\ntwo letters. Shannon (1948) applied n-grams to compute approximations to English\nword sequences. Based on Shannon\u2019s work, Markov models were commonly used in\nengineering, linguistic, and psychological work on modeling word sequences by the\n1950s. In a series of extremely in\ufb02uential papers starting with Chomsky (1956) and\nincluding Chomsky (1957) and Miller and Chomsky (1963), Noam Chomsky argued\nthat \u201c\ufb01nite-state Markov processes\u201d, while a possibly useful engineering heuristic,\nwere incapable of being a complete cognitive model of human grammatical knowl-\nedge. These arguments led many linguists and computational linguists to ignore\nwork in statistical modeling for decades.\nThe resurgence of n-gram language models came from Fred Jelinek and col-\nleagues at the IBM Thomas J. Watson Research Center, who were in\ufb02uenced by\nShannon, and James Baker at CMU, who was in\ufb02uenced by the prior, classi\ufb01ed\nwork of Leonard Baum and colleagues on these topics at labs like the US Institute\nfor Defense SAnalyses (IDA) after they were declassi\ufb01ed. Independently these two\nlabs successfully used n-grams in their speech recognition systems at the same time\n(Baker 1975b, Jelinek et al. 1975, Baker 1975a, Bahl et al. 1983, Jelinek 1990). The\nterms \u201clanguage model\u201d and \u201cperplexity\u201d were \ufb01rst used for this technology by the\nIBM group. Jelinek and his colleagues used the term language model in a pretty\nmodern way, to mean the entire set of linguistic in\ufb02uences on word sequence prob-\nabilities, including grammar, semantics, discourse, and even speaker characteristics,\nrather than just the particular n-gram model itself.\nAdd-one smoothing derives from Laplace\u2019s 1812 law of succession and was \ufb01rst\napplied as an engineering solution to the zero frequency problem by Jeffreys (1948)\nbased on an earlier Add-K suggestion by Johnson (1932). Problems with the add-\none algorithm are summarized in Gale and Church (1994).\nA wide variety of different language modeling and smoothing techniques were\nproposed in the 80s and 90s, including Good-Turing discounting\u2014\ufb01rst applied to the\nn-gram smoothing at IBM by Katz (N \u00b4adas 1984, Church and Gale 1991)\u2014 Witten-\nBell discounting (Witten and Bell, 1991), and varieties of class-based n-gram mod-class-based\nn-gram\nels that used information about word classes. Starting in the late 1990s, Chen and\nGoodman performed a number of carefully controlled experiments comparing dif-\nferent algorithms and parameters (Chen and Goodman 1999, Goodman 2006, inter\nalia). They showed the advantages of Modi\ufb01ed Interpolated Kneser-Ney , which\nbecame the standard baseline for n-gram language modeling around the turn of the\ncentury, especially because they showed that caches and class-based models pro-\nvided only minor additional improvement. SRILM (Stolcke, 2002) and KenLM\n(Hea\ufb01eld 2011, Hea\ufb01eld et al. 2013) are publicly available toolkits for building n-\ngram language models.\nLarge language models are based on neural networks rather than n-grams, en-\nabling them to solve the two major problems with n-grams: (1) the number of param-\neters increases exponentially as the n-gram order increases, and (2) n-grams have no\nway to generalize from training examples to test set examples unless they use iden-\ntical words. Neural language models instead project words into a continuous space\nin which words with similar contexts have similar representations. We\u2019ll introduce\ntransformer-based large language models in Chapter 9, along the way introducing\nfeedforward language models (Bengio et al. 2006, Schwenk 2007) in Chapter 7 and\nrecurrent language models (Mikolov, 2012) in Chapter 8. 54 CHAPTER 3 \u2022 N- GRAM LANGUAGE MODELS\nExercises\n3.1 Write out the equation for trigram probability estimation (modifying Eq. 3.11).\nNow write out all the non-zero trigram probabilities for the I am Sam corpus\non page 35.\n3.2 Calculate the probability of the sentence i want chinese food . Give two\nprobabilities, one using Fig. 3.2 and the \u2018useful probabilities\u2019 just below it on\npage 37, and another using the add-1 smoothed table in Fig. 3.7. Assume the\nadditional add-1 smoothed probabilities P(i|<s> ) =0:19 and P(</s>|food ) =\n0:40.\n3.3 Which of the two probabilities you computed in the previous exercise is higher,\nunsmoothed or smoothed? Explain why.\n3.4 We are given the following corpus, modi\ufb01ed from the one in the chapter:\n<s> I am Sam </s>\n<s> Sam I am </s>\n<s> I am Sam </s>\n<s> I do not like green eggs and Sam </s>\nUsing a bigram language model with add-one smoothing, what is P(Sam j\nam)? Include <s>and</s> in your counts just like any other token.\n3.5 Suppose we didn\u2019t use the end-symbol </s> . Train an unsmoothed bigram\ngrammar on the following training corpus without using the end-symbol </s> :\n<s> a b\n<s> b b\n<s> b a\n<s> a a\nDemonstrate that your bigram model does not assign a single probability dis-\ntribution across all sentence lengths by showing that the sum of the probability\nof the four possible 2 word sentences over the alphabet fa,bgis 1.0, and the\nsum of the probability of all possible 3 word sentences over the alphabet fa,bg\nis also 1.0.\n3.6 Suppose we train a trigram language model with add-one smoothing on a\ngiven corpus. The corpus contains V word types. Express a formula for esti-\nmating P(w3jw1,w2), where w3 is a word which follows the bigram (w1,w2),\nin terms of various n-gram counts and V . Use the notation c(w1,w2,w3) to\ndenote the number of times that trigram (w1,w2,w3) occurs in the corpus, and\nso on for bigrams and unigrams.\n3.7 We are given the following corpus, modi\ufb01ed from the one in the chapter:\n<s> I am Sam </s>\n<s> Sam I am </s>\n<s> I am Sam </s>\n<s> I do not like green eggs and Sam </s>\nIf we use linear interpolation smoothing between a maximum-likelihood bi-\ngram model and a maximum-likelihood unigram model with l1=1\n2andl2=\n1\n2, what is P(Samjam)? Include <s> and</s> in your counts just like any\nother token.\n3.8 Write a program to compute unsmoothed unigrams and bigrams. EXERCISES 55\n3.9 Run your n-gram program on two different small corpora of your choice (you\nmight use email text or newsgroups). Now compare the statistics of the two\ncorpora. What are the differences in the most common unigrams between the\ntwo? How about interesting differences in bigrams?\n3.10 Add an option to your program to generate random sentences.\n3.11 Add an option to your program to compute the perplexity of a test set.\n3.12 You are given a training set of 100 numbers that consists of 91 zeros and 1\neach of the other digits 1-9. Now we see the following test set: 0 0 0 0 0 3 0 0\n0 0. What is the unigram perplexity? 56 CHAPTER 4 \u2022 N AIVE BAYES , TEXT CLASSIFICATION ,AND SENTIMENT\nCHAPTER\n4Naive Bayes, Text Classi\ufb01ca-\ntion, and Sentiment\nClassi\ufb01cation lies at the heart of both human and machine intelligence. Deciding\nwhat letter, word, or image has been presented to our senses, recognizing faces\nor voices, sorting mail, assigning grades to homeworks; these are all examples of\nassigning a category to an input. The potential challenges of this task are highlighted\nby the fabulist Jorge Luis Borges (1964), who imagined classifying animals into:\n(a) those that belong to the Emperor, (b) embalmed ones, (c) those that\nare trained, (d) suckling pigs, (e) mermaids, (f) fabulous ones, (g) stray\ndogs, (h) those that are included in this classi\ufb01cation, (i) those that\ntremble as if they were mad, (j) innumerable ones, (k) those drawn with\na very \ufb01ne camel\u2019s hair brush, (l) others, (m) those that have just broken\na \ufb02ower vase, (n) those that resemble \ufb02ies from a distance.\nMany language processing tasks involve classi\ufb01cation, although luckily our classes\nare much easier to de\ufb01ne than those of Borges. In this chapter we introduce the naive\nBayes algorithm and apply it to text categorization , the task of assigning a label ortext\ncategorization\ncategory to an entire text or document.\nWe focus on one common text categorization task, sentiment analysis , the ex-sentiment\nanalysis\ntraction of sentiment , the positive or negative orientation that a writer expresses\ntoward some object. A review of a movie, book, or product on the web expresses the\nauthor\u2019s sentiment toward the product, while an editorial or political text expresses\nsentiment toward a candidate or political action. Extracting consumer or public sen-\ntiment is thus relevant for \ufb01elds from marketing to politics.\nThe simplest version of sentiment analysis is a binary classi\ufb01cation task, and\nthe words of the review provide excellent cues. Consider, for example, the follow-\ning phrases extracted from positive and negative reviews of movies and restaurants.\nWords like great ,richly ,awesome , and pathetic , and awful andridiculously are very\ninformative cues:\n+...zany characters and richly applied satire, and some great plot twists\n\u0000It was pathetic. The worst part about it was the boxing scenes...\n+...awesome caramel sauce and sweet toasty almonds. I love this place!\n\u0000...awful pizza and ridiculously overpriced...\nSpam detection is another important commercial application, the binary clas- spam detection\nsi\ufb01cation task of assigning an email to one of the two classes spam ornot-spam .\nMany lexical and other features can be used to perform this classi\ufb01cation. For ex-\nample you might quite reasonably be suspicious of an email containing phrases like\n\u201conline pharmaceutical\u201d or \u201cWITHOUT ANY COST\u201d or \u201cDear Winner\u201d.\nAnother thing we might want to know about a text is the language it\u2019s written\nin. Texts on social media, for example, can be in any number of languages and\nwe\u2019ll need to apply different processing. The task of language id is thus the \ufb01rst language id\nstep in most language processing pipelines. Related text classi\ufb01cation tasks like au-\nthorship attribution \u2014 determining a text\u2019s author\u2014 are also relevant to the digitalauthorship\nattribution\nhumanities, social sciences, and forensic linguistics. 4.1 \u2022 N AIVE BAYES CLASSIFIERS 57\nFinally, one of the oldest tasks in text classi\ufb01cation is assigning a library sub-\nject category or topic label to a text. Deciding whether a research paper concerns\nepidemiology or instead, perhaps, embryology, is an important component of infor-\nmation retrieval. Various sets of subject categories exist, such as the MeSH (Medical\nSubject Headings) thesaurus. In fact, as we will see, subject category classi\ufb01cation\nis the task for which the naive Bayes algorithm was invented in 1961 Maron (1961).\nClassi\ufb01cation is essential for tasks below the level of the document as well.\nWe\u2019ve already seen period disambiguation (deciding if a period is the end of a sen-\ntence or part of a word), and word tokenization (deciding if a character should be\na word boundary). Even language modeling can be viewed as classi\ufb01cation: each\nword can be thought of as a class, and so predicting the next word is classifying the\ncontext-so-far into a class for each next word. A part-of-speech tagger (Chapter 17)\nclassi\ufb01es each occurrence of a word in a sentence as, e.g., a noun or a verb.\nThe goal of classi\ufb01cation is to take a single observation, extract some useful\nfeatures, and thereby classify the observation into one of a set of discrete classes.\nOne method for classifying text is to use rules handwritten by humans. Handwrit-\nten rule-based classi\ufb01ers can be components of state-of-the-art systems in language\nprocessing. But rules can be fragile, as situations or data change over time, and for\nsome tasks humans aren\u2019t necessarily good at coming up with the rules.\nThe most common way of doing text classi\ufb01cation in language processing is\ninstead via supervised machine learning , the subject of this chapter. In supervisedsupervised\nmachine\nlearninglearning, we have a data set of input observations, each associated with some correct\noutput (a \u2018supervision signal\u2019). The goal of the algorithm is to learn how to map\nfrom a new observation to a correct output.\nFormally, the task of supervised classi\ufb01cation is to take an input xand a \ufb01xed\nset of output classes Y=fy1;y2;:::;yMgand return a predicted class y2Y. For\ntext classi\ufb01cation, we\u2019ll sometimes talk about c(for \u201cclass\u201d) instead of yas our\noutput variable, and d(for \u201cdocument\u201d) instead of xas our input variable. In the\nsupervised situation we have a training set of Ndocuments that have each been hand-\nlabeled with a class: f(d1;c1);::::;(dN;cN)g. Our goal is to learn a classi\ufb01er that is\ncapable of mapping from a new document dto its correct class c2C, where Cis\nsome set of useful document classes. A probabilistic classi\ufb01er additionally will tell\nus the probability of the observation being in the class. This full distribution over\nthe classes can be useful information for downstream decisions; avoiding making\ndiscrete decisions early on can be useful when combining systems.\nMany kinds of machine learning algorithms are used to build classi\ufb01ers. This\nchapter introduces naive Bayes; the following one introduces logistic regression.\nThese exemplify two ways of doing classi\ufb01cation. Generative classi\ufb01ers like naive\nBayes build a model of how a class could generate some input data. Given an ob-\nservation, they return the class most likely to have generated the observation. Dis-\ncriminative classi\ufb01ers like logistic regression instead learn what features from the\ninput are most useful to discriminate between the different possible classes. While\ndiscriminative systems are often more accurate and hence more commonly used,\ngenerative classi\ufb01ers still have a role.\n4.1 Naive Bayes Classi\ufb01ers\nIn this section we introduce the multinomial naive Bayes classi\ufb01er , so called be-naive Bayes\nclassi\ufb01er\ncause it is a Bayesian classi\ufb01er that makes a simplifying (naive) assumption about 58 CHAPTER 4 \u2022 N AIVE BAYES , TEXT CLASSIFICATION ,AND SENTIMENT\nhow the features interact.\nThe intuition of the classi\ufb01er is shown in Fig. 4.1. We represent a text document\nas if it were a bag of words , that is, an unordered set of words with their position bag of words\nignored, keeping only their frequency in the document. In the example in the \ufb01gure,\ninstead of representing the word order in all the phrases like \u201cI love this movie\u201d and\n\u201cI would recommend it\u201d, we simply note that the word Ioccurred 5 times in the\nentire excerpt, the word it6 times, the words love,recommend , and movie once, and\nso on.\nititititititIIII\nIloverecommendmoviethethethetheto\ntotoand\nandandseenseenyetwouldwithwhowhimsical\nwhilewhenevertimessweetseveralscenessatiricalromanticofmanageshumorhavehappyfunfriendfairydialoguebutconventionsareanyoneadventurealwaysagainaboutI love this movie! It's sweet, but with satirical humor. The dialogue is great and the adventure scenes are fun... It manages to be whimsical and romantic while laughing at the conventions of the fairy tale genre. I would recommend it to just about anyone. I've seen it several times, and I'm always happy to see it again whenever I have a friend who hasn't seen it yet!it Ithetoandseenyetwouldwhimsicaltimessweetsatiricaladventuregenrefairyhumorhavegreat\u20266 54332111111111111\u2026\nFigure 4.1 Intuition of the multinomial naive Bayes classi\ufb01er applied to a movie review. The position of the\nwords is ignored (the bag-of-words assumption) and we make use of the frequency of each word.\nNaive Bayes is a probabilistic classi\ufb01er, meaning that for a document d, out of\nall classes c2Cthe classi\ufb01er returns the class \u02c6 cwhich has the maximum posterior\nprobability given the document. In Eq. 4.1 we use the hat notation \u02c6to mean \u201cour \u02c6\nestimate of the correct class\u201d, and we use argmax to mean an operation that selects argmax\nthe argument (in this case the class c) that maximizes a function (in this case the\nprobability P(cjd).\n\u02c6c=argmax\nc2CP(cjd) (4.1)\nThis idea of Bayesian inference has been known since the work of Bayes (1763),Bayesian\ninference\nand was \ufb01rst applied to text classi\ufb01cation by Mosteller and Wallace (1964). The\nintuition of Bayesian classi\ufb01cation is to use Bayes\u2019 rule to transform Eq. 4.1 into\nother probabilities that have some useful properties. Bayes\u2019 rule is presented in\nEq. 4.2; it gives us a way to break down any conditional probability P(xjy)into\nthree other probabilities:\nP(xjy) =P(yjx)P(x)\nP(y)(4.2) 4.1 \u2022 N AIVE BAYES CLASSIFIERS 59\nWe can then substitute Eq. 4.2 into Eq. 4.1 to get Eq. 4.3:\n\u02c6c=argmax\nc2CP(cjd) =argmax\nc2CP(djc)P(c)\nP(d)(4.3)\nWe can conveniently simplify Eq. 4.3 by dropping the denominator P(d). This\nis possible because we will be computingP(djc)P(c)\nP(d)for each possible class. But P(d)\ndoesn\u2019t change for each class; we are always asking about the most likely class for\nthe same document d, which must have the same probability P(d). Thus, we can\nchoose the class that maximizes this simpler formula:\n\u02c6c=argmax\nc2CP(cjd) =argmax\nc2CP(djc)P(c) (4.4)\nWe call Naive Bayes a generative model because we can read Eq. 4.4 as stating\na kind of implicit assumption about how a document is generated: \ufb01rst a class is\nsampled from P(c), and then the words are generated by sampling from P(djc). (In\nfact we could imagine generating arti\ufb01cial documents, or at least their word counts,\nby following this process). We\u2019ll say more about this intuition of generative models\nin Chapter 5.\nTo return to classi\ufb01cation: we compute the most probable class \u02c6 cgiven some\ndocument dby choosing the class which has the highest product of two probabilities:\ntheprior probability of the class P(c)and the likelihood of the document P(djc):prior\nprobability\nlikelihood\n\u02c6c=argmax\nc2Clikelihoodz}|{\nP(djc)priorz}|{\nP(c) (4.5)\nWithout loss of generality, we can represent a document das a set of features\nf1;f2;:::;fn:\n\u02c6c=argmax\nc2Clikelihoodz}|{\nP(f1;f2;::::; fnjc)priorz}|{\nP(c) (4.6)\nUnfortunately, Eq. 4.6 is still too hard to compute directly: without some sim-\nplifying assumptions, estimating the probability of every possible combination of\nfeatures (for example, every possible set of words and positions) would require huge\nnumbers of parameters and impossibly large training sets. Naive Bayes classi\ufb01ers\ntherefore make two simplifying assumptions.\nThe \ufb01rst is the bag-of-words assumption discussed intuitively above: we assume\nposition doesn\u2019t matter, and that the word \u201clove\u201d has the same effect on classi\ufb01cation\nwhether it occurs as the 1st, 20th, or last word in the document. Thus we assume\nthat the features f1;f2;:::;fnonly encode word identity and not position.\nThe second is commonly called the naive Bayes assumption : this is the condi-naive Bayes\nassumption\ntional independence assumption that the probabilities P(fijc)are independent given\nthe class cand hence can be \u2018naively\u2019 multiplied as follows:\nP(f1;f2;::::; fnjc) = P(f1jc)\u0001P(f2jc)\u0001:::\u0001P(fnjc) (4.7)\nThe \ufb01nal equation for the class chosen by a naive Bayes classi\ufb01er is thus:\ncNB=argmax\nc2CP(c)Y\nf2FP(fjc) (4.8) 60 CHAPTER 4 \u2022 N AIVE BAYES , TEXT CLASSIFICATION ,AND SENTIMENT\nTo apply the naive Bayes classi\ufb01er to text, we will use each word in the documents\nas a feature, as suggested above, and we consider each of the words in the document\nby walking an index through every word position in the document:\npositions all word positions in test document\ncNB=argmax\nc2CP(c)Y\ni2positionsP(wijc) (4.9)\nNaive Bayes calculations, like calculations for language modeling, are done in log\nspace, to avoid under\ufb02ow and increase speed. Thus Eq. 4.9 is generally instead\nexpressed1as\ncNB=argmax\nc2ClogP(c)+X\ni2positionslogP(wijc) (4.10)\nBy considering features in log space, Eq. 4.10 computes the predicted class as a lin-\near function of input features. Classi\ufb01ers that use a linear combination of the inputs\nto make a classi\ufb01cation decision \u2014like naive Bayes and also logistic regression\u2014\nare called linear classi\ufb01ers .linear\nclassi\ufb01ers\n4.2 Training the Naive Bayes Classi\ufb01er\nHow can we learn the probabilities P(c)andP(fijc)? Let\u2019s \ufb01rst consider the maxi-\nmum likelihood estimate. We\u2019ll simply use the frequencies in the data. For the class\nprior P(c)we ask what percentage of the documents in our training set are in each\nclass c. Let Ncbe the number of documents in our training data with class cand\nNdocbe the total number of documents. Then:\n\u02c6P(c) =Nc\nNdoc(4.11)\nTo learn the probability P(fijc), we\u2019ll assume a feature is just the existence of a word\nin the document\u2019s bag of words, and so we\u2019ll want P(wijc), which we compute as\nthe fraction of times the word wiappears among all words in all documents of topic\nc. We \ufb01rst concatenate all documents with category cinto one big \u201ccategory c\u201d text.\nThen we use the frequency of wiin this concatenated document to give a maximum\nlikelihood estimate of the probability:\n\u02c6P(wijc) =count (wi;c)P\nw2Vcount (w;c)(4.12)\nHere the vocabulary V consists of the union of all the word types in all classes, not\njust the words in one class c.\nThere is a problem, however, with maximum likelihood training. Imagine we\nare trying to estimate the likelihood of the word \u201cfantastic\u201d given class positive , but\nsuppose there are no training documents that both contain the word \u201cfantastic\u201d and\nare classi\ufb01ed as positive . Perhaps the word \u201cfantastic\u201d happens to occur (sarcasti-\ncally?) in the class negative . In such a case the probability for this feature will be\nzero:\n\u02c6P(\u201cfantastic\u201djpositive ) =count (\u201cfantastic\u201d;positive )P\nw2Vcount (w;positive )=0 (4.13)\n1In practice throughout this book, we\u2019ll use log to mean natural log (ln) when the base is not speci\ufb01ed. 4.3 \u2022 W ORKED EXAMPLE 61\nBut since naive Bayes naively multiplies all the feature likelihoods together, zero\nprobabilities in the likelihood term for any class will cause the probability of the\nclass to be zero, no matter the other evidence!\nThe simplest solution is the add-one (Laplace) smoothing introduced in Chap-\nter 3. While Laplace smoothing is usually replaced by more sophisticated smoothing\nalgorithms in language modeling, it is commonly used in naive Bayes text catego-\nrization:\n\u02c6P(wijc) =count (wi;c)+1P\nw2V(count (w;c)+1)=count (wi;c)+1\u0000P\nw2Vcount (w;c)\u0001\n+jVj(4.14)\nNote once again that it is crucial that the vocabulary V consists of the union of all the\nword types in all classes, not just the words in one class c(try to convince yourself\nwhy this must be true; see the exercise at the end of the chapter).\nWhat do we do about words that occur in our test data but are not in our vocab-\nulary at all because they did not occur in any training document in any class? The\nsolution for such unknown words is to ignore them\u2014remove them from the test unknown word\ndocument and not include any probability for them at all.\nFinally, some systems choose to completely ignore another class of words: stop\nwords , very frequent words like theanda. This can be done by sorting the vocabu- stop words\nlary by frequency in the training set, and de\ufb01ning the top 10\u2013100 vocabulary entries\nas stop words, or alternatively by using one of the many prede\ufb01ned stop word lists\navailable online. Then each instance of these stop words is simply removed from\nboth training and test documents as if it had never occurred. In most text classi\ufb01ca-\ntion applications, however, using a stop word list doesn\u2019t improve performance, and\nso it is more common to make use of the entire vocabulary and not use a stop word\nlist.\nFig. 4.2 shows the \ufb01nal algorithm.\n4.3 Worked example\nLet\u2019s walk through an example of training and testing naive Bayes with add-one\nsmoothing. We\u2019ll use a sentiment analysis domain with the two classes positive\n(+) and negative (-), and take the following miniature training and test documents\nsimpli\ufb01ed from actual movie reviews.\nCat Documents\nTraining - just plain boring\n- entirely predictable and lacks energy\n- no surprises and very few laughs\n+ very powerful\n+ the most fun \ufb01lm of the summer\nTest ? predictable with no fun\nThe prior P(c)for the two classes is computed via Eq. 4.11 asNc\nNdoc:\nP(\u0000) =3\n5P(+) =2\n5\nThe word with doesn\u2019t occur in the training set, so we drop it completely (as\nmentioned above, we don\u2019t use unknown word models for naive Bayes). The like-\nlihoods from the training set for the remaining three words \u201cpredictable\u201d, \u201cno\u201d, and 62 CHAPTER 4 \u2022 N AIVE BAYES , TEXT CLASSIFICATION ,AND SENTIMENT\nfunction TRAIN NAIVE BAYES (D, C) returns V;logP(c), log P(wjc)\nfor each class c2C # Calculate P(c)terms\nNdoc= number of documents in D\nNc= number of documents from D in class c\nlogprior [c] logNc\nNdoc\nV vocabulary of D\nbigdoc [c] append (d)ford2Dwith class c\nfor each word win V # Calculate P(wjc)terms\ncount(w,c) # of occurrences of winbigdoc [c]\nloglikelihood [w,c] logcount (w;c) + 1P\nw0in V(count (w0;c) + 1)\nreturn logprior ,loglikelihood ,V\nfunction TESTNAIVE BAYES (testdoc ,logprior ,loglikelihood , C, V) returns bestc\nfor each class c2C\nsum[c] logprior [c]\nfor each position iintestdoc\nword testdoc[i]\nifword2V\nsum[c] sum[c]+loglikelihood [word ,c]\nreturn argmaxcsum[c]\nFigure 4.2 The naive Bayes algorithm, using add-1 smoothing. To use add- asmoothing\ninstead, change the +1 to+afor loglikelihood counts in training.\n\u201cfun\u201d, are as follows, from Eq. 4.14 (computing the probabilities for the remainder\nof the words in the training set is left as an exercise for the reader):\nP(\u201cpredictable\u201dj\u0000) =1+1\n14+20P(\u201cpredictable\u201dj+) =0+1\n9+20\nP(\u201cno\u201dj\u0000) =1+1\n14+20P(\u201cno\u201dj+) =0+1\n9+20\nP(\u201cfun\u201dj\u0000) =0+1\n14+20P(\u201cfun\u201dj+) =1+1\n9+20\nFor the test sentence S = \u201cpredictable with no fun\u201d, after removing the word \u2018with\u2019,\nthe chosen class, via Eq. 4.9, is therefore computed as follows:\nP(\u0000)P(Sj\u0000) =3\n5\u00022\u00022\u00021\n343=6:1\u000210\u00005\nP(+)P(Sj+) =2\n5\u00021\u00021\u00022\n293=3:2\u000210\u00005\nThe model thus predicts the class negative for the test sentence.\n4.4 Optimizing for Sentiment Analysis\nWhile standard naive Bayes text classi\ufb01cation can work well for sentiment analysis,\nsome small changes are generally employed that improve performance. 4.4 \u2022 O PTIMIZING FOR SENTIMENT ANALYSIS 63\nFirst, for sentiment classi\ufb01cation and a number of other text classi\ufb01cation tasks,\nwhether a word occurs or not seems to matter more than its frequency. Thus it often\nimproves performance to clip the word counts in each document at 1 (see the end\nof the chapter for pointers to these results). This variant is called binary multino-\nmial naive Bayes orbinary naive Bayes . The variant uses the same algorithm asbinary naive\nBayes\nin Fig. 4.2 except that for each document we remove all duplicate words before con-\ncatenating them into the single big document during training and we also remove\nduplicate words from test documents. Fig. 4.3 shows an example in which a set\nof four documents (shortened and text-normalized for this example) are remapped\nto binary, with the modi\ufb01ed counts shown in the table on the right. The example\nis worked without add-1 smoothing to make the differences clearer. Note that the\nresults counts need not be 1; the word great has a count of 2 even for binary naive\nBayes, because it appears in multiple documents.\nFour original documents:\n\u0000it was pathetic the worst part was the\nboxing scenes\n\u0000no plot twists or great scenes\n+and satire and great plot twists\n+great scenes great \ufb01lm\nAfter per-document binarization:\n\u0000it was pathetic the worst part boxing\nscenes\n\u0000no plot twists or great scenes\n+and satire great plot twists\n+great scenes \ufb01lmNB Binary\nCounts Counts\n+\u0000+\u0000\nand 2 0 1 0\nboxing 0 1 0 1\n\ufb01lm 1 0 1 0\ngreat 3 1 2 1\nit 0 1 0 1\nno 0 1 0 1\nor 0 1 0 1\npart 0 1 0 1\npathetic 0 1 0 1\nplot 1 1 1 1\nsatire 1 0 1 0\nscenes 1 2 1 2\nthe 0 2 0 1\ntwists 1 1 1 1\nwas 0 2 0 1\nworst 0 1 0 1\nFigure 4.3 An example of binarization for the binary naive Bayes algorithm.\nA second important addition commonly made when doing text classi\ufb01cation for\nsentiment is to deal with negation. Consider the difference between I really like this\nmovie (positive) and I didn\u2019t like this movie (negative). The negation expressed by\ndidn\u2019t completely alters the inferences we draw from the predicate like. Similarly,\nnegation can modify a negative word to produce a positive review ( don\u2019t dismiss this\n\ufb01lm,doesn\u2019t let us get bored ).\nA very simple baseline that is commonly used in sentiment analysis to deal with\nnegation is the following: during text normalization, prepend the pre\ufb01x NOT to\nevery word after a token of logical negation ( n\u2019t, not, no, never ) until the next punc-\ntuation mark. Thus the phrase\ndidn't like this movie , but I\nbecomes\ndidn't NOT_like NOT_this NOT_movie , but I\nNewly formed \u2018words\u2019 like NOT like,NOT recommend will thus occur more\noften in negative document and act as cues for negative sentiment, while words\nlikeNOT bored ,NOT dismiss will acquire positive associations. Syntactic parsing\n(Chapter 18) can be used deal more accurately with the scope relationship between 64 CHAPTER 4 \u2022 N AIVE BAYES , TEXT CLASSIFICATION ,AND SENTIMENT\nthese negation words and the predicates they modify, but this simple baseline works\nquite well in practice.\nFinally, in some situations we might have insuf\ufb01cient labeled training data to\ntrain accurate naive Bayes classi\ufb01ers using all words in the training set to estimate\npositive and negative sentiment. In such cases we can instead derive the positive\nand negative word features from sentiment lexicons , lists of words that are pre-sentiment\nlexicons\nannotated with positive or negative sentiment. Four popular lexicons are the General\nInquirer (Stone et al., 1966), LIWC (Pennebaker et al., 2007), the opinion lexiconGeneral\nInquirer\nLIWC of Hu and Liu (2004a) and the MPQA Subjectivity Lexicon (Wilson et al., 2005).\nFor example the MPQA subjectivity lexicon has 6885 words each marked for\nwhether it is strongly or weakly biased positive or negative. Some examples:\n+:admirable, beautiful, con\ufb01dent, dazzling, ecstatic, favor, glee, great\n\u0000:awful, bad, bias, catastrophe, cheat, deny, envious, foul, harsh, hate\nA common way to use lexicons in a naive Bayes classi\ufb01er is to add a feature\nthat is counted whenever a word from that lexicon occurs. Thus we might add a\nfeature called \u2018this word occurs in the positive lexicon\u2019, and treat all instances of\nwords in the lexicon as counts for that one feature, instead of counting each word\nseparately. Similarly, we might add as a second feature \u2018this word occurs in the\nnegative lexicon\u2019 of words in the negative lexicon. If we have lots of training data,\nand if the test data matches the training data, using just two features won\u2019t work as\nwell as using all the words. But when training data is sparse or not representative of\nthe test set, using dense lexicon features instead of sparse individual-word features\nmay generalize better.\nWe\u2019ll return to this use of lexicons in Chapter 22, showing how these lexicons\ncan be learned automatically, and how they can be applied to many other tasks be-\nyond sentiment classi\ufb01cation.\n4.5 Naive Bayes for other text classi\ufb01cation tasks\nIn the previous section we pointed out that naive Bayes doesn\u2019t require that our\nclassi\ufb01er use all the words in the training data as features. In fact features in naive\nBayes can express any property of the input text we want.\nConsider the task of spam detection , deciding if a particular piece of email is spam detection\nan example of spam (unsolicited bulk email)\u2014one of the \ufb01rst applications of naive\nBayes to text classi\ufb01cation (Sahami et al., 1998).\nA common solution here, rather than using all the words as individual features,\nis to prede\ufb01ne likely sets of words or phrases as features, combined with features\nthat are not purely linguistic. For example the open-source SpamAssassin tool2\nprede\ufb01nes features like the phrase \u201cone hundred percent guaranteed\u201d, or the feature\nmentions millions of dollars , which is a regular expression that matches suspiciously\nlarge sums of money. But it also includes features like HTML has a low ratio of text\nto image area , that aren\u2019t purely linguistic and might require some sophisticated\ncomputation, or totally non-linguistic features about, say, the path that the email\ntook to arrive. More sample SpamAssassin features:\n\u2022 Email subject line is all capital letters\n\u2022 Contains phrases of urgency like \u201curgent reply\u201d\n2https://spamassassin.apache.org 4.6 \u2022 N AIVE BAYES AS A LANGUAGE MODEL 65\n\u2022 Email subject line contains \u201conline pharmaceutical\u201d\n\u2022 HTML has unbalanced \u201chead\u201d tags\n\u2022 Claims you can be removed from the list\nFor other tasks, like language id \u2014determining what language a given piece language id\nof text is written in\u2014the most effective naive Bayes features are not words at all,\nbutcharacter n-grams , 2-grams (\u2018zw\u2019) 3-grams (\u2018nya\u2019, \u2018 V o\u2019), or 4-grams (\u2018ie z\u2019,\n\u2018thei\u2019), or, even simpler byte n-grams , where instead of using the multibyte Unicode\ncharacter representations called codepoints, we just pretend everything is a string of\nraw bytes. Because spaces count as a byte, byte n-grams can model statistics about\nthe beginning or ending of words. A widely used naive Bayes system, langid.py\n(Lui and Baldwin, 2012) begins with all possible n-grams of lengths 1-4, using fea-\nture selection to winnow down to the most informative 7000 \ufb01nal features.\nLanguage ID systems are trained on multilingual text, such as Wikipedia (Wiki-\npedia text in 68 different languages was used in (Lui and Baldwin, 2011)), or newswire.\nTo make sure that this multilingual text correctly re\ufb02ects different regions, dialects,\nand socioeconomic classes, systems also add Twitter text in many languages geo-\ntagged to many regions (important for getting world English dialects from countries\nwith large Anglophone populations like Nigeria or India), Bible and Quran transla-\ntions, slang websites like Urban Dictionary, corpora of African American Vernacular\nEnglish (Blodgett et al., 2016), and so on (Jurgens et al., 2017).\n4.6 Naive Bayes as a Language Model\nAs we saw in the previous section, naive Bayes classi\ufb01ers can use any sort of feature:\ndictionaries, URLs, email addresses, network features, phrases, and so on. But if,\nas in Section 4.3, we use only individual word features, and we use all of the words\nin the text (not a subset), then naive Bayes has an important similarity to language\nmodeling. Speci\ufb01cally, a naive Bayes model can be viewed as a set of class-speci\ufb01c\nunigram language models, in which the model for each class instantiates a unigram\nlanguage model.\nSince the likelihood features from the naive Bayes model assign a probability to\neach word P(wordjc), the model also assigns a probability to each sentence:\nP(sjc) =Y\ni2positionsP(wijc) (4.15)\nThus consider a naive Bayes model with the classes positive (+) and negative (-)\nand the following model parameters:\nw P(wj+)P(wj-)\nI 0.1 0.2\nlove 0.1 0.001\nthis 0.01 0.01\nfun 0.05 0.005\n\ufb01lm 0.1 0.1\n... ... ...\nEach of the two columns above instantiates a language model that can assign a\nprobability to the sentence \u201cI love this fun \ufb01lm\u201d: 66 CHAPTER 4 \u2022 N AIVE BAYES , TEXT CLASSIFICATION ,AND SENTIMENT\nP(\u201cI love this fun \ufb01lm\u201d j+) = 0:1\u00020:1\u00020:01\u00020:05\u00020:1=5\u000210\u00007\nP(\u201cI love this fun \ufb01lm\u201d j\u0000) = 0:2\u00020:001\u00020:01\u00020:005\u00020:1=1:0\u000210\u00009\nAs it happens, the positive model assigns a higher probability to the sentence:\nP(sjpos)>P(sjneg). Note that this is just the likelihood part of the naive Bayes\nmodel; once we multiply in the prior a full naive Bayes model might well make a\ndifferent classi\ufb01cation decision.\n4.7 Evaluation: Precision, Recall, F-measure\nTo introduce the methods for evaluating text classi\ufb01cation, let\u2019s \ufb01rst consider some\nsimple binary detection tasks. For example, in spam detection, our goal is to label\nevery text as being in the spam category (\u201cpositive\u201d) or not in the spam category\n(\u201cnegative\u201d). For each item (email document) we therefore need to know whether\nour system called it spam or not. We also need to know whether the email is actually\nspam or not, i.e. the human-de\ufb01ned labels for each document that we are trying to\nmatch. We will refer to these human labels as the gold labels . gold labels\nOr imagine you\u2019re the CEO of the Delicious Pie Company and you need to know\nwhat people are saying about your pies on social media, so you build a system that\ndetects tweets concerning Delicious Pie. Here the positive class is tweets about\nDelicious Pie and the negative class is all other tweets.\nIn both cases, we need a metric for knowing how well our spam detector (or\npie-tweet-detector) is doing. To evaluate any system for detecting things, we start\nby building a confusion matrix like the one shown in Fig. 4.4. A confusion matrixconfusion\nmatrix\nis a table for visualizing how an algorithm performs with respect to the human gold\nlabels, using two dimensions (system output and gold labels), and each cell labeling\na set of possible outcomes. In the spam detection case, for example, true positives\nare documents that are indeed spam (indicated by human-created gold labels) that\nour system correctly said were spam. False negatives are documents that are indeed\nspam but our system incorrectly labeled as non-spam.\nTo the bottom right of the table is the equation for accuracy , which asks what\npercentage of all the observations (for the spam or pie examples that means all emails\nor tweets) our system labeled correctly. Although accuracy might seem a natural\nmetric, we generally don\u2019t use it for text classi\ufb01cation tasks. That\u2019s because accuracy\ndoesn\u2019t work well when the classes are unbalanced (as indeed they are with spam,\nwhich is a large majority of email, or with tweets, which are mainly not about pie).\nTo make this more explicit, imagine that we looked at a million tweets, and\nlet\u2019s say that only 100 of them are discussing their love (or hatred) for our pie,\nwhile the other 999,900 are tweets about something completely unrelated. Imagine a\nsimple classi\ufb01er that stupidly classi\ufb01ed every tweet as \u201cnot about pie\u201d. This classi\ufb01er\nwould have 999,900 true negatives and only 100 false negatives for an accuracy of\n999,900/1,000,000 or 99.99%! What an amazing accuracy level! Surely we should\nbe happy with this classi\ufb01er? But of course this fabulous \u2018no pie\u2019 classi\ufb01er would\nbe completely useless, since it wouldn\u2019t \ufb01nd a single one of the customer comments\nwe are looking for. In other words, accuracy is not a good metric when the goal is\nto discover something that is rare, or at least not completely balanced in frequency,\nwhich is a very common situation in the world. 4.7 \u2022 E VALUATION : PRECISION , RECALL , F- MEASURE 67\ntrue positivefalse negativefalse positivetrue negativegold positivegold negativesystempositivesystemnegativegold standard labelssystemoutputlabelsrecall = tptp+fnprecision = tptp+fpaccuracy = tp+tntp+fp+tn+fn\nFigure 4.4 A confusion matrix for visualizing how well a binary classi\ufb01cation system per-\nforms against gold standard labels.\nThat\u2019s why instead of accuracy we generally turn to two other metrics shown in\nFig. 4.4: precision andrecall .Precision measures the percentage of the items that precision\nthe system detected (i.e., the system labeled as positive) that are in fact positive (i.e.,\nare positive according to the human gold labels). Precision is de\ufb01ned as\nPrecision =true positives\ntrue positives + false positives\nRecall measures the percentage of items actually present in the input that were recall\ncorrectly identi\ufb01ed by the system. Recall is de\ufb01ned as\nRecall =true positives\ntrue positives + false negatives\nPrecision and recall will help solve the problem with the useless \u201cnothing is\npie\u201d classi\ufb01er. This classi\ufb01er, despite having a fabulous accuracy of 99.99%, has\na terrible recall of 0 (since there are no true positives, and 100 false negatives, the\nrecall is 0/100). You should convince yourself that the precision at \ufb01nding relevant\ntweets is equally problematic. Thus precision and recall, unlike accuracy, emphasize\ntrue positives: \ufb01nding the things that we are supposed to be looking for.\nThere are many ways to de\ufb01ne a single metric that incorporates aspects of both\nprecision and recall. The simplest of these combinations is the F-measure (van F-measure\nRijsbergen, 1975) , de\ufb01ned as:\nFb=(b2+1)PR\nb2P+R\nThebparameter differentially weights the importance of recall and precision,\nbased perhaps on the needs of an application. Values of b>1 favor recall, while\nvalues of b<1 favor precision. When b=1, precision and recall are equally bal-\nanced; this is the most frequently used metric, and is called F b=1or just F 1: F1\nF1=2PR\nP+R(4.16)\nF-measure comes from a weighted harmonic mean of precision and recall. The\nharmonic mean of a set of numbers is the reciprocal of the arithmetic mean of recip-\nrocals:\nHarmonicMean (a1;a2;a3;a4;:::;an) =n\n1\na1+1\na2+1\na3+:::+1\nan(4.17) 68 CHAPTER 4 \u2022 N AIVE BAYES , TEXT CLASSIFICATION ,AND SENTIMENT\nand hence F-measure is\nF=1\na1\nP+(1\u0000a)1\nRor\u0012\nwithb2=1\u0000a\na\u0013\nF=(b2+1)PR\nb2P+R(4.18)\nHarmonic mean is used because the harmonic mean of two values is closer to the\nminimum of the two values than the arithmetic mean is. Thus it weighs the lower of\nthe two numbers more heavily, which is more conservative in this situation.\n4.7.1 Evaluating with more than two classes\nUp to now we have been describing text classi\ufb01cation tasks with only two classes.\nBut lots of classi\ufb01cation tasks in language processing have more than two classes.\nFor sentiment analysis we generally have 3 classes (positive, negative, neutral) and\neven more classes are common for tasks like part-of-speech tagging, word sense\ndisambiguation, semantic role labeling, emotion detection, and so on. Luckily the\nnaive Bayes algorithm is already a multi-class classi\ufb01cation algorithm.\n851060urgentnormalgold labelssystemoutputrecallu = 88+5+3precisionu= 88+10+115030200spamurgentnormalspam3recalln = recalls = precisionn= 605+60+50precisions= 2003+30+2006010+60+302001+50+200\nFigure 4.5 Confusion matrix for a three-class categorization task, showing for each pair of\nclasses (c1;c2), how many documents from c1were (in)correctly assigned to c2.\nBut we\u2019ll need to slightly modify our de\ufb01nitions of precision and recall. Con-\nsider the sample confusion matrix for a hypothetical 3-way one-of email catego-\nrization decision (urgent, normal, spam) shown in Fig. 4.5. The matrix shows, for\nexample, that the system mistakenly labeled one spam document as urgent, and we\nhave shown how to compute a distinct precision and recall value for each class. In\norder to derive a single metric that tells us how well the system is doing, we can com-\nbine these values in two ways. In macroaveraging , we compute the performance macroaveraging\nfor each class, and then average over classes. In microaveraging , we collect the de- microaveraging\ncisions for all classes into a single confusion matrix, and then compute precision and\nrecall from that table. Fig. 4.6 shows the confusion matrix for each class separately,\nand shows the computation of microaveraged and macroaveraged precision.\nAs the \ufb01gure shows, a microaverage is dominated by the more frequent class (in\nthis case spam), since the counts are pooled. The macroaverage better re\ufb02ects the\nstatistics of the smaller classes, and so is more appropriate when performance on all\nthe classes is equally important. 4.8 \u2022 T EST SETS AND CROSS -VALIDATION 69\n8811340trueurgenttruenotsystemurgentsystemnot604055212truenormaltruenotsystemnormalsystemnot200513383truespamtruenotsystemspamsystemnot2689999635trueyestruenosystemyessystemnoprecision =8+118= .42precision =200+33200= .86precision =60+5560= .52microaverageprecision268+99268= .73=macroaverageprecision3.42+.52+.86= .60=PooledClass 3: SpamClass 2: NormalClass 1: Urgent\nFigure 4.6 Separate confusion matrices for the 3 classes from the previous \ufb01gure, showing the pooled confu-\nsion matrix and the microaveraged and macroaveraged precision.\n4.8 Test sets and Cross-validation\nThe training and testing procedure for text classi\ufb01cation follows what we saw with\nlanguage modeling (Section 3.2): we use the training set to train the model, then use\nthedevelopment test set (also called a devset ) to perhaps tune some parameters,development\ntest set\ndevset and in general decide what the best model is. Once we come up with what we think\nis the best model, we run it on the (hitherto unseen) test set to report its performance.\nWhile the use of a devset avoids over\ufb01tting the test set, having a \ufb01xed train-\ning set, devset, and test set creates another problem: in order to save lots of data\nfor training, the test set (or devset) might not be large enough to be representative.\nWouldn\u2019t it be better if we could somehow use all our data for training and still use\nall our data for test? We can do this by cross-validation . cross-validation\nIn cross-validation, we choose a number k, and partition our data into kdisjoint\nsubsets called folds . Now we choose one of those kfolds as a test set, train our folds\nclassi\ufb01er on the remaining k\u00001 folds, and then compute the error rate on the test\nset. Then we repeat with another fold as the test set, again training on the other k\u00001\nfolds. We do this sampling process ktimes and average the test set error rate from\nthese kruns to get an average error rate. If we choose k=10, we would train 10\ndifferent models (each on 90% of our data), test the model 10 times, and average\nthese 10 values. This is called 10-fold cross-validation .10-fold\ncross-validation\nThe only problem with cross-validation is that because all the data is used for\ntesting, we need the whole corpus to be blind; we can\u2019t examine any of the data\nto suggest possible features and in general see what\u2019s going on, because we\u2019d be\npeeking at the test set, and such cheating would cause us to overestimate the perfor-\nmance of our system. However, looking at the corpus to understand what\u2019s going\non is important in designing NLP systems! What to do? For this reason, it is com-\nmon to create a \ufb01xed training set and test set, then do 10-fold cross-validation inside\nthe training set, but compute error rate the normal way in the test set, as shown in\nFig. 4.7. 70 CHAPTER 4 \u2022 N AIVE BAYES , TEXT CLASSIFICATION ,AND SENTIMENT\nTraining Iterations13452678910DevDevDevDevDevDevDevDevDevDevTrainingTrainingTrainingTrainingTrainingTrainingTrainingTrainingTrainingTrainingTrainingTest SetTesting\nFigure 4.7 10-fold cross-validation\n4.9 Statistical Signi\ufb01cance Testing\nIn building systems we often need to compare the performance of two systems. How\ncan we know if the new system we just built is better than our old one? Or better\nthan some other system described in the literature? This is the domain of statistical\nhypothesis testing, and in this section we introduce tests for statistical signi\ufb01cance\nfor NLP classi\ufb01ers, drawing especially on the work of Dror et al. (2020) and Berg-\nKirkpatrick et al. (2012).\nSuppose we\u2019re comparing the performance of classi\ufb01ers AandBon a metric M\nsuch as F 1, or accuracy. Perhaps we want to know if our logistic regression senti-\nment classi\ufb01er A(Chapter 5) gets a higher F 1score than our naive Bayes sentiment\nclassi\ufb01er Bon a particular test set x. Let\u2019s call M(A;x)the score that system Agets\non test set x, and d(x)the performance difference between AandBonx:\nd(x) =M(A;x)\u0000M(B;x) (4.19)\nWe would like to know if d(x)>0, meaning that our logistic regression classi\ufb01er\nhas a higher F 1than our naive Bayes classi\ufb01er on x.d(x)is called the effect size ; a effect size\nbigger dmeans that Aseems to be way better than B; a small dmeans Aseems to\nbe only a little better.\nWhy don\u2019t we just check if d(x)is positive? Suppose we do, and we \ufb01nd that\nthe F 1score of Ais higher than B\u2019s by .04. Can we be certain that Ais better? We\ncannot! That\u2019s because Amight just be accidentally better than Bon this particular x.\nWe need something more: we want to know if A\u2019s superiority over Bis likely to hold\nagain if we checked another test set x0, or under some other set of circumstances.\nIn the paradigm of statistical hypothesis testing, we test this by formalizing two\nhypotheses.\nH0:d(x)\u00140\nH1:d(x)>0 (4.20)\nThe hypothesis H0, called the null hypothesis , supposes that d(x)is actually nega- null hypothesis\ntive or zero, meaning that Ais not better than B. We would like to know if we can\ncon\ufb01dently rule out this hypothesis, and instead support H1, that Ais better.\nWe do this by creating a random variable Xranging over all test sets. Now we\nask how likely is it, if the null hypothesis H0was correct, that among these test sets 4.9 \u2022 S TATISTICAL SIGNIFICANCE TESTING 71\nwe would encounter the value of d(x)that we found, if we repeated the experiment\na great many times. We formalize this likelihood as the p-value : the probability, p-value\nassuming the null hypothesis H0is true, of seeing the d(x)that we saw or one even\ngreater\nP(d(X)\u0015d(x)jH0is true ) (4.21)\nSo in our example, this p-value is the probability that we would see d(x)assuming\nAisnotbetter than B. Ifd(x)is huge (let\u2019s say Ahas a very respectable F 1of .9\nandBhas a terrible F 1of only .2 on x), we might be surprised, since that would be\nextremely unlikely to occur if H0were in fact true, and so the p-value would be low\n(unlikely to have such a large difAis in fact not better than B). But if d(x)is very\nsmall, it might be less surprising to us even if H0were true and Ais not really better\nthan B, and so the p-value would be higher.\nA very small p-value means that the difference we observed is very unlikely\nunder the null hypothesis, and we can reject the null hypothesis. What counts as very\nsmall? It is common to use values like .05 or .01 as the thresholds. A value of .01\nmeans that if the p-value (the probability of observing the dwe saw assuming H0is\ntrue) is less than .01, we reject the null hypothesis and assume that Ais indeed better\nthan B. We say that a result (e.g., \u201c Ais better than B\u201d) is statistically signi\ufb01cant ifstatistically\nsigni\ufb01cant\nthedwe saw has a probability that is below the threshold and we therefore reject\nthis null hypothesis.\nHow do we compute this probability we need for the p-value? In NLP we gen-\nerally don\u2019t use simple parametric tests like t-tests or ANOV As that you might be\nfamiliar with. Parametric tests make assumptions about the distributions of the test\nstatistic (such as normality) that don\u2019t generally hold in our cases. So in NLP we\nusually use non-parametric tests based on sampling: we arti\ufb01cially create many ver-\nsions of the experimental setup. For example, if we had lots of different test sets x0\nwe could just measure all the d(x0)for all the x0. That gives us a distribution. Now\nwe set a threshold (like .01) and if we see in this distribution that 99% or more of\nthose deltas are smaller than the delta we observed, i.e., that p-value( x)\u2014the proba-\nbility of seeing a d(x)as big as the one we saw\u2014is less than .01, then we can reject\nthe null hypothesis and agree that d(x)was a suf\ufb01ciently surprising difference and\nAis really a better algorithm than B.\nThere are two common non-parametric tests used in NLP: approximate ran-\ndomization (Noreen, 1989) and the bootstrap test . We will describe bootstrapapproximate\nrandomization\nbelow, showing the paired version of the test, which again is most common in NLP.\nPaired tests are those in which we compare two sets of observations that are aligned: paired\neach observation in one set can be paired with an observation in another. This hap-\npens naturally when we are comparing the performance of two systems on the same\ntest set; we can pair the performance of system Aon an individual observation xi\nwith the performance of system Bon the same xi.\n4.9.1 The Paired Bootstrap Test\nThebootstrap test (Efron and Tibshirani, 1993) can apply to any metric; from pre- bootstrap test\ncision, recall, or F1 to the BLEU metric used in machine translation. The word\nbootstrapping refers to repeatedly drawing large numbers of samples with replace- bootstrapping\nment (called bootstrap samples ) from an original set. The intuition of the bootstrap\ntest is that we can create many virtual test sets from an observed test set by repeat-\nedly sampling from it. The method only makes the assumption that the sample is\nrepresentative of the population. 72 CHAPTER 4 \u2022 N AIVE BAYES , TEXT CLASSIFICATION ,AND SENTIMENT\nConsider a tiny text classi\ufb01cation example with a test set xof 10 documents. The\n\ufb01rst row of Fig. 4.8 shows the results of two classi\ufb01ers (A and B) on this test set.\nEach document is labeled by one of the four possibilities (A and B both right, both\nwrong, A right and B wrong, A wrong and B right). A slash through a letter ( \u0013B)\nmeans that that classi\ufb01er got the answer wrong. On the \ufb01rst document both A and\nB get the correct class (AB), while on the second document A got it right but B got\nit wrong (A \u0013B). If we assume for simplicity that our metric is accuracy, A has an\naccuracy of .70 and B of .50, so d(x)is .20.\nNow we create a large number b(perhaps 105) of virtual test sets x(i), each of size\nn=10. Fig. 4.8 shows a couple of examples. To create each virtual test set x(i), we\nrepeatedly ( n=10 times) select a cell from row xwith replacement. For example, to\ncreate the \ufb01rst cell of the \ufb01rst virtual test set x(1), if we happened to randomly select\nthe second cell of the xrow; we would copy the value A \u0013B into our new cell, and\nmove on to create the second cell of x(1), each time sampling (randomly choosing)\nfrom the original xwith replacement.\n1 2 3 4 5 6 7 8 910A% B% d()\nx AB A\u0013\u0013BAB\u0000\u0000AB A\u0013\u0013B\u0000\u0000AB A\u0013\u0013BAB\u0000\u0000A\u0013\u0013BA\u0013\u0013B.70 .50 .20\nx(1)A\u0013\u0013BAB A\u0013\u0013B\u0000\u0000AB\u0000\u0000AB A\u0013\u0013B\u0000\u0000AB AB\u0000\u0000A\u0013\u0013BAB .60 .60 .00\nx(2)A\u0013\u0013BAB\u0000\u0000A\u0013\u0013B\u0000\u0000AB\u0000\u0000AB AB\u0000\u0000AB A\u0013\u0013BAB AB .60 .70-.10\n...\nx(b)\nFigure 4.8 The paired bootstrap test: Examples of bpseudo test sets x(i)being created\nfrom an initial true test set x. Each pseudo test set is created by sampling n=10 times with\nreplacement; thus an individual sample is a single cell, a document with its gold label and\nthe correct or incorrect performance of classi\ufb01ers A and B. Of course real test sets don\u2019t have\nonly 10 examples, and bneeds to be large as well.\nNow that we have the btest sets, providing a sampling distribution, we can do\nstatistics on how often Ahas an accidental advantage. There are various ways to\ncompute this advantage; here we follow the version laid out in Berg-Kirkpatrick\net al. (2012). Assuming H0(Aisn\u2019t better than B), we would expect that d(X),\nestimated over many test sets, would be zero or negative; a much higher value would\nbe surprising, since H0speci\ufb01cally assumes Aisn\u2019t better than B. To measure exactly\nhow surprising our observed d(x)is, we would in other circumstances compute the\np-value by counting over many test sets how often d(x(i))exceeds the expected zero\nvalue by d(x)or more:\np-value (x) =1\nbbX\ni=11\u0010\nd(x(i))\u0000d(x)\u00150\u0011\n(We use the notation 1(x)to mean \u201c1 if xis true, and 0 otherwise\u201d.) However,\nalthough it\u2019s generally true that the expected value of d(X)over many test sets,\n(again assuming Aisn\u2019t better than B) is 0, this isn\u2019t true for the bootstrapped test\nsets we created. That\u2019s because we didn\u2019t draw these samples from a distribution\nwith 0 mean; we happened to create them from the original test set x, which happens\nto be biased (by .20) in favor of A. So to measure how surprising is our observed\nd(x), we actually compute the p-value by counting over many test sets how often 4.10 \u2022 A VOIDING HARMS IN CLASSIFICATION 73\nd(x(i))exceeds the expected value of d(x)byd(x)or more:\np-value (x) =1\nbbX\ni=11\u0010\nd(x(i))\u0000d(x)\u0015d(x)\u0011\n=1\nbbX\ni=11\u0010\nd(x(i))\u00152d(x)\u0011\n(4.22)\nSo if for example we have 10,000 test sets x(i)and a threshold of .01, and in only 47\nof the test sets do we \ufb01nd that A is accidentally better d(x(i))\u00152d(x), the resulting\np-value of .0047 is smaller than .01, indicating that the delta we found, d(x)is indeed\nsuf\ufb01ciently surprising and unlikely to have happened by accident, and we can reject\nthe null hypothesis and conclude Ais better than B.\nfunction BOOTSTRAP (test set x,num of samples b)returns p-value (x)\nCalculate d(x)# how much better does algorithm A do than B on x\ns= 0\nfori= 1tobdo\nforj= 1tondo # Draw a bootstrap sample x(i)of size n\nSelect a member of xat random and add it to x(i)\nCalculate d(x(i))# how much better does algorithm A do than B on x(i)\ns s+ 1ifd(x(i))\u00152d(x)\np-value( x)\u0019s\nb# on what % of the b samples did algorithm A beat expectations?\nreturn p-value( x) # if very few did, our observed dis probably not accidental\nFigure 4.9 A version of the paired bootstrap algorithm after Berg-Kirkpatrick et al. (2012).\nThe full algorithm for the bootstrap is shown in Fig. 4.9. It is given a test set x, a\nnumber of samples b, and counts the percentage of the bbootstrap test sets in which\nd(x\u0003(i))>2d(x). This percentage then acts as a one-sided empirical p-value.\n4.10 Avoiding Harms in Classi\ufb01cation\nIt is important to avoid harms that may result from classi\ufb01ers, harms that exist both\nfor naive Bayes classi\ufb01ers and for the other classi\ufb01cation algorithms we introduce\nin later chapters.\nOne class of harms is representational harms (Crawford 2017, Blodgett et al.representational\nharms\n2020), harms caused by a system that demeans a social group, for example by per-\npetuating negative stereotypes about them. For example Kiritchenko and Moham-\nmad (2018) examined the performance of 200 sentiment analysis systems on pairs of\nsentences that were identical except for containing either a common African Amer-\nican \ufb01rst name (like Shaniqua ) or a common European American \ufb01rst name (like\nStephanie ), chosen from the Caliskan et al. (2017) study discussed in Chapter 6.\nThey found that most systems assigned lower sentiment and more negative emotion\nto sentences with African American names, re\ufb02ecting and perpetuating stereotypes\nthat associate African Americans with negative emotions (Popp et al., 2003).\nIn other tasks classi\ufb01ers may lead to both representational harms and other\nharms, such as silencing. For example the important text classi\ufb01cation task of tox- 74 CHAPTER 4 \u2022 N AIVE BAYES , TEXT CLASSIFICATION ,AND SENTIMENT\nicity detection is the task of detecting hate speech, abuse, harassment, or othertoxicity\ndetection\nkinds of toxic language. While the goal of such classi\ufb01ers is to help reduce soci-\netal harm, toxicity classi\ufb01ers can themselves cause harms. For example, researchers\nhave shown that some widely used toxicity classi\ufb01ers incorrectly \ufb02ag as being toxic\nsentences that are non-toxic but simply mention identities like women (Park et al.,\n2018), blind people (Hutchinson et al., 2020) or gay people (Dixon et al., 2018;\nDias Oliva et al., 2021), or simply use linguistic features characteristic of varieties\nlike African-American Vernacular English (Sap et al. 2019, Davidson et al. 2019).\nSuch false positive errors could lead to the silencing of discourse by or about these\ngroups.\nThese model problems can be caused by biases or other problems in the training\ndata; in general, machine learning systems replicate and even amplify the biases\nin their training data. But these problems can also be caused by the labels (for\nexample due to biases in the human labelers), by the resources used (like lexicons,\nor model components like pretrained embeddings), or even by model architecture\n(like what the model is trained to optimize). While the mitigation of these biases\n(for example by carefully considering the training data sources) is an important area\nof research, we currently don\u2019t have general solutions. For this reason it\u2019s important,\nwhen introducing any NLP model, to study these kinds of factors and make them\nclear. One way to do this is by releasing a model card (Mitchell et al., 2019) for model card\neach version of a model. A model card documents a machine learning model with\ninformation like:\n\u2022 training algorithms and parameters\n\u2022 training data sources, motivation, and preprocessing\n\u2022 evaluation data sources, motivation, and preprocessing\n\u2022 intended use and users\n\u2022 model performance across different demographic or other groups and envi-\nronmental situations\n4.11 Summary\nThis chapter introduced the naive Bayes model for classi\ufb01cation and applied it to\nthetext categorization task of sentiment analysis .\n\u2022 Many language processing tasks can be viewed as tasks of classi\ufb01cation .\n\u2022 Text categorization, in which an entire text is assigned a class from a \ufb01nite set,\nincludes such tasks as sentiment analysis ,spam detection , language identi-\n\ufb01cation, and authorship attribution.\n\u2022 Sentiment analysis classi\ufb01es a text as re\ufb02ecting the positive or negative orien-\ntation ( sentiment ) that a writer expresses toward some object.\n\u2022 Naive Bayes is a generative model that makes the bag-of-words assumption\n(position doesn\u2019t matter) and the conditional independence assumption (words\nare conditionally independent of each other given the class)\n\u2022 Naive Bayes with binarized features seems to work better for many text clas-\nsi\ufb01cation tasks.\n\u2022 Classi\ufb01ers are evaluated based on precision andrecall .\n\u2022 Classi\ufb01ers are trained using distinct training, dev, and test sets, including the\nuse of cross-validation in the training set. BIBLIOGRAPHICAL AND HISTORICAL NOTES 75\n\u2022 Statistical signi\ufb01cance tests should be used to determine whether we can be\ncon\ufb01dent that one version of a classi\ufb01er is better than another.\n\u2022 Designers of classi\ufb01ers should carefully consider harms that may be caused\nby the model, including its training data and other components, and report\nmodel characteristics in a model card .\nBibliographical and Historical Notes\nMultinomial naive Bayes text classi\ufb01cation was proposed by Maron (1961) at the\nRAND Corporation for the task of assigning subject categories to journal abstracts.\nHis model introduced most of the features of the modern form presented here, ap-\nproximating the classi\ufb01cation task with one-of categorization, and implementing\nadd-dsmoothing and information-based feature selection.\nThe conditional independence assumptions of naive Bayes and the idea of Bayes-\nian analysis of text seems to have arisen multiple times. The same year as Maron\u2019s\npaper, Minsky (1961) proposed a naive Bayes classi\ufb01er for vision and other arti-\n\ufb01cial intelligence problems, and Bayesian techniques were also applied to the text\nclassi\ufb01cation task of authorship attribution by Mosteller and Wallace (1963). It had\nlong been known that Alexander Hamilton, John Jay, and James Madison wrote\nthe anonymously-published Federalist papers in 1787\u20131788 to persuade New York\nto ratify the United States Constitution. Yet although some of the 85 essays were\nclearly attributable to one author or another, the authorship of 12 were in dispute\nbetween Hamilton and Madison. Mosteller and Wallace (1963) trained a Bayesian\nprobabilistic model of the writing of Hamilton and another model on the writings\nof Madison, then computed the maximum-likelihood author for each of the disputed\nessays. Naive Bayes was \ufb01rst applied to spam detection in Heckerman et al. (1998).\nMetsis et al. (2006), Pang et al. (2002), and Wang and Manning (2012) show\nthat using boolean attributes with multinomial naive Bayes works better than full\ncounts. Binary multinomial naive Bayes is sometimes confused with another variant\nof naive Bayes that also uses a binary representation of whether a term occurs in\na document: Multivariate Bernoulli naive Bayes . The Bernoulli variant instead\nestimates P(wjc)as the fraction of documents that contain a term, and includes a\nprobability for whether a term is notin a document. McCallum and Nigam (1998)\nand Wang and Manning (2012) show that the multivariate Bernoulli variant of naive\nBayes doesn\u2019t work as well as the multinomial algorithm for sentiment or other text\ntasks.\nThere are a variety of sources covering the many kinds of text classi\ufb01cation\ntasks. For sentiment analysis see Pang and Lee (2008), and Liu and Zhang (2012).\nStamatatos (2009) surveys authorship attribute algorithms. On language identi\ufb01ca-\ntion see Jauhiainen et al. (2019); Jaech et al. (2016) is an important early neural\nsystem. The task of newswire indexing was often used as a test case for text classi-\n\ufb01cation algorithms, based on the Reuters-21578 collection of newswire articles.\nSee Manning et al. (2008) and Aggarwal and Zhai (2012) on text classi\ufb01cation;\nclassi\ufb01cation in general is covered in machine learning textbooks (Hastie et al. 2001,\nWitten and Frank 2005, Bishop 2006, Murphy 2012).\nNon-parametric methods for computing statistical signi\ufb01cance were used \ufb01rst in\nNLP in the MUC competition (Chinchor et al., 1993), and even earlier in speech\nrecognition (Gillick and Cox 1989, Bisani and Ney 2004). Our description of the\nbootstrap draws on the description in Berg-Kirkpatrick et al. (2012). Recent work\nhas focused on issues including multiple test sets and multiple metrics (S\u00f8gaard et al. 76 CHAPTER 4 \u2022 N AIVE BAYES , TEXT CLASSIFICATION ,AND SENTIMENT\n2014, Dror et al. 2017).\nFeature selection is a method of removing features that are unlikely to generalize\nwell. Features are generally ranked by how informative they are about the classi\ufb01ca-\ntion decision. A very common metric, information gain , tells us how many bits ofinformation\ngain\ninformation the presence of the word gives us for guessing the class. Other feature\nselection metrics include c2, pointwise mutual information, and GINI index; see\nYang and Pedersen (1997) for a comparison and Guyon and Elisseeff (2003) for an\nintroduction to feature selection.\nExercises\n4.1 Assume the following likelihoods for each word being part of a positive or\nnegative movie review, and equal prior probabilities for each class.\npos neg\nI 0.09 0.16\nalways 0.07 0.06\nlike 0.29 0.06\nforeign 0.04 0.15\n\ufb01lms 0.08 0.11\nWhat class will Naive bayes assign to the sentence \u201cI always like foreign\n\ufb01lms.\u201d?\n4.2 Given the following short movie reviews, each labeled with a genre, either\ncomedy or action:\n1. fun, couple, love, love comedy\n2. fast, furious, shoot action\n3. couple, \ufb02y, fast, fun, fun comedy\n4. furious, shoot, shoot, fun action\n5. \ufb02y, fast, shoot, love action\nand a new document D:\nfast, couple, shoot, \ufb02y\ncompute the most likely class for D. Assume a naive Bayes classi\ufb01er and use\nadd-1 smoothing for the likelihoods.\n4.3 Train two models, multinomial naive Bayes and binarized naive Bayes, both\nwith add-1 smoothing, on the following document counts for key sentiment\nwords, with positive or negative class assigned as noted.\ndoc \u201cgood\u201d \u201cpoor\u201d \u201cgreat\u201d (class)\nd1. 3 0 3 pos\nd2. 0 1 2 pos\nd3. 1 3 0 neg\nd4. 1 5 2 neg\nd5. 0 2 0 neg\nUse both naive Bayes models to assign a class (pos or neg) to this sentence:\nA good, good plot and great characters, but poor acting.\nRecall from page 61 that with naive Bayes text classi\ufb01cation, we simply ig-\nnore (throw out) any word that never occurred in the training document. (We\ndon\u2019t throw out words that appear in some classes but not others; that\u2019s what\nadd-one smoothing is for.) Do the two models agree or disagree? CHAPTER\n5Logistic Regression\n\u201cAnd how do you know that these \ufb01ne begonias are not of equal importance?\u201d\nHercule Poirot, in Agatha Christie\u2019s The Mysterious Affair at Styles\nDetective stories are as littered with clues as texts are with words. Yet for the\npoor reader it can be challenging to know how to weigh the author\u2019s clues in order\nto make the crucial classi\ufb01cation task: deciding whodunnit.\nIn this chapter we introduce an algorithm that is admirably suited for discovering\nthe link between features or clues and some particular outcome: logistic regression .logistic\nregression\nIndeed, logistic regression is one of the most important analytic tools in the social\nand natural sciences. In natural language processing, logistic regression is the base-\nline supervised machine learning algorithm for classi\ufb01cation, and also has a very\nclose relationship with neural networks. As we will see in Chapter 7, a neural net-\nwork can be viewed as a series of logistic regression classi\ufb01ers stacked on top of\neach other. Thus the classi\ufb01cation and machine learning techniques introduced here\nwill play an important role throughout the book.\nLogistic regression can be used to classify an observation into one of two classes\n(like \u2018positive sentiment\u2019 and \u2018negative sentiment\u2019), or into one of many classes.\nBecause the mathematics for the two-class case is simpler, we\u2019ll describe this special\ncase of logistic regression \ufb01rst in the next few sections, and then brie\ufb02y summarize\nthe use of multinomial logistic regression for more than two classes in Section 5.3.\nWe\u2019ll introduce the mathematics of logistic regression in the next few sections.\nBut let\u2019s begin with some high-level issues.\nGenerative and Discriminative Classi\ufb01ers: The most important difference be-\ntween naive Bayes and logistic regression is that logistic regression is a discrimina-\ntiveclassi\ufb01er while naive Bayes is a generative classi\ufb01er.\nThese are two very different frameworks for how\nto build a machine learning model. Consider a visual\nmetaphor: imagine we\u2019re trying to distinguish dog\nimages from cat images. A generative model would\nhave the goal of understanding what dogs look like\nand what cats look like. You might literally ask such\na model to \u2018generate\u2019, i.e., draw, a dog. Given a test\nimage, the system then asks whether it\u2019s the cat model or the dog model that better\n\ufb01ts (is less surprised by) the image, and chooses that as its label.\nA discriminative model, by contrast, is only try-\ning to learn to distinguish the classes (perhaps with-\nout learning much about them). So maybe all the\ndogs in the training data are wearing collars and the\ncats aren\u2019t. If that one feature neatly separates the\nclasses, the model is satis\ufb01ed. If you ask such a\nmodel what it knows about cats all it can say is that\nthey don\u2019t wear collars. 78 CHAPTER 5 \u2022 L OGISTIC REGRESSION\nMore formally, recall that the naive Bayes assigns a class cto a document dnot\nby directly computing P(cjd)but by computing a likelihood and a prior\n\u02c6c=argmax\nc2Clikelihoodz}|{\nP(djc)priorz}|{\nP(c) (5.1)\nAgenerative model like naive Bayes makes use of this likelihood term, whichgenerative\nmodel\nexpresses how to generate the features of a document if we knew it was of class c .\nBy contrast a discriminative model in this text categorization scenario attemptsdiscriminative\nmodel\ntodirectly compute P(cjd). Perhaps it will learn to assign a high weight to document\nfeatures that directly improve its ability to discriminate between possible classes,\neven if it couldn\u2019t generate an example of one of the classes.\nComponents of a probabilistic machine learning classi\ufb01er: Like naive Bayes,\nlogistic regression is a probabilistic classi\ufb01er that makes use of supervised machine\nlearning. Machine learning classi\ufb01ers require a training corpus of minput/output\npairs(x(i);y(i)). (We\u2019ll use superscripts in parentheses to refer to individual instances\nin the training set\u2014for sentiment classi\ufb01cation each instance might be an individual\ndocument to be classi\ufb01ed.) A machine learning system for classi\ufb01cation then has\nfour components:\n1. A feature representation of the input. For each input observation x(i), this\nwill be a vector of features [x1;x2;:::;xn]. We will generally refer to feature\nifor input x(j)asx(j)\ni, sometimes simpli\ufb01ed as xi, but we will also see the\nnotation fi,fi(x), or, for multiclass classi\ufb01cation, fi(c;x).\n2. A classi\ufb01cation function that computes \u02c6 y, the estimated class, via p(yjx). In\nthe next section we will introduce the sigmoid andsoftmax tools for classi\ufb01-\ncation.\n3. An objective function that we want to optimize for learning, usually involving\nminimizing a loss function corresponding to error on training examples. We\nwill introduce the cross-entropy loss function .\n4. An algorithm for optimizing the objective function. We introduce the stochas-\ntic gradient descent algorithm.\nLogistic regression has two phases:\ntraining: We train the system (speci\ufb01cally the weights wandb, introduced be-\nlow) using stochastic gradient descent and the cross-entropy loss.\ntest: Given a test example xwe compute p(yjx)and return the higher probability\nlabel y=1 ory=0.\n5.1 The sigmoid function\nThe goal of binary logistic regression is to train a classi\ufb01er that can make a binary\ndecision about the class of a new input observation. Here we introduce the sigmoid\nclassi\ufb01er that will help us make this decision.\nConsider a single input observation x, which we will represent by a vector of\nfeatures [x1;x2;:::;xn]. (We\u2019ll show sample features in the next subsection.) The\nclassi\ufb01er output ycan be 1 (meaning the observation is a member of the class) or\n0 (the observation is not a member of the class). We want to know the probability 5.1 \u2022 T HE SIGMOID FUNCTION 79\nP(y=1jx)that this observation is a member of the class. So perhaps the decision\nis \u201cpositive sentiment\u201d versus \u201cnegative sentiment\u201d, the features represent counts of\nwords in a document, P(y=1jx)is the probability that the document has positive\nsentiment, and P(y=0jx)is the probability that the document has negative senti-\nment.\nLogistic regression solves this task by learning, from a training set, a vector of\nweights and a bias term . Each weight wiis a real number, and is associated with one\nof the input features xi. The weight wirepresents how important that input feature\nis to the classi\ufb01cation decision, and can be positive (providing evidence that the in-\nstance being classi\ufb01ed belongs in the positive class) or negative (providing evidence\nthat the instance being classi\ufb01ed belongs in the negative class). Thus we might\nexpect in a sentiment task the word awesome to have a high positive weight, and\nabysmal to have a very negative weight. The bias term , also called the intercept , is bias term\nintercept another real number that\u2019s added to the weighted inputs.\nTo make a decision on a test instance\u2014after we\u2019ve learned the weights in training\u2014\nthe classi\ufb01er \ufb01rst multiplies each xiby its weight wi, sums up the weighted features,\nand adds the bias term b. The resulting single number zexpresses the weighted sum\nof the evidence for the class.\nz= nX\ni=1wixi!\n+b (5.2)\nIn the rest of the book we\u2019ll represent such sums using the dot product notation dot product\nfrom linear algebra. The dot product of two vectors aandb, written as a\u0001b, is the\nsum of the products of the corresponding elements of each vector. (Notice that we\nrepresent vectors using the boldface notation b). Thus the following is an equivalent\nformation to Eq. 5.2:\nz=w\u0001x+b (5.3)\nBut note that nothing in Eq. 5.3 forces zto be a legal probability, that is, to lie\nbetween 0 and 1. In fact, since weights are real-valued, the output might even be\nnegative; zranges from\u0000\u00a5to\u00a5.\nFigure 5.1 The sigmoid function s(z) =1\n1+e\u0000ztakes a real value and maps it to the range\n(0;1). It is nearly linear around 0 but outlier values get squashed toward 0 or 1.\nTo create a probability, we\u2019ll pass zthrough the sigmoid function, s(z). The sigmoid\nsigmoid function (named because it looks like an s) is also called the logistic func-\ntion, and gives logistic regression its name. The sigmoid has the following equation,logistic\nfunction\nshown graphically in Fig. 5.1:\ns(z) =1\n1+e\u0000z=1\n1+exp(\u0000z)(5.4) 80 CHAPTER 5 \u2022 L OGISTIC REGRESSION\n(For the rest of the book, we\u2019ll use the notation exp (x)to mean ex.) The sigmoid\nhas a number of advantages; it takes a real-valued number and maps it into the range\n(0;1), which is just what we want for a probability. Because it is nearly linear around\n0 but \ufb02attens toward the ends, it tends to squash outlier values toward 0 or 1. And\nit\u2019s differentiable, which as we\u2019ll see in Section 5.10 will be handy for learning.\nWe\u2019re almost there. If we apply the sigmoid to the sum of the weighted features,\nwe get a number between 0 and 1. To make it a probability, we just need to make\nsure that the two cases, p(y=1)andp(y=0), sum to 1. We can do this as follows:\nP(y=1) = s(w\u0001x+b)\n=1\n1+exp(\u0000(w\u0001x+b))\nP(y=0) = 1\u0000s(w\u0001x+b)\n=1\u00001\n1+exp(\u0000(w\u0001x+b))\n=exp(\u0000(w\u0001x+b))\n1+exp(\u0000(w\u0001x+b))(5.5)\nThe sigmoid function has the property\n1\u0000s(x) =s(\u0000x) (5.6)\nso we could also have expressed P(y=0)ass(\u0000(w\u0001x+b)).\nFinally, one terminological point. The input to the sigmoid function, the score\nz=w\u0001x+bfrom (5.3), is often called the logit . This is because the logit function logit\nis the inverse of the sigmoid. The logit function is the log of the odds ratiop\n1\u0000p:\nlogit(p) =s\u00001(p) =lnp\n1\u0000p(5.7)\nUsing the term logit forzis a way of reminding us that by using the sigmoid to turn\nz(which ranges from \u0000\u00a5to\u00a5) into a probability, we are implicitly interpreting zas\nnot just any real-valued number, but as speci\ufb01cally a log odds.\n5.2 Classi\ufb01cation with Logistic Regression\nThe sigmoid function from the prior section thus gives us a way to take an instance\nxand compute the probability P(y=1jx).\nHow do we make a decision about which class to apply to a test instance x? For\na given x, we say yes if the probability P(y=1jx)is more than .5, and no otherwise.\nWe call .5 the decision boundary :decision\nboundary\ndecision (x) =\u001a1 if P(y=1jx)>0:5\n0 otherwise\nLet\u2019s have some examples of applying logistic regression as a classi\ufb01er for language\ntasks. 5.2 \u2022 C LASSIFICATION WITH LOGISTIC REGRESSION 81\n5.2.1 Sentiment Classi\ufb01cation\nSuppose we are doing binary sentiment classi\ufb01cation on movie review text, and\nwe would like to know whether to assign the sentiment class +or\u0000to a review\ndocument doc. We\u2019ll represent each input observation by the 6 features x1:::x6of\nthe input shown in the following table; Fig. 5.2 shows the features in a sample mini\ntest document.\nVar De\ufb01nition Value in Fig. 5.2\nx1 count(positive lexicon words 2doc) 3\nx2 count(negative lexicon words 2doc) 2\nx3\u001a1 if \u201cno\u201d2doc\n0 otherwise1\nx4 count (1st and 2nd pronouns 2doc) 3\nx5\u001a1 if \u201c!\u201d2doc\n0 otherwise0\nx6 ln(word count of doc ) ln(66) =4:19\n It's hokey . There are virtually no surprises , and the writing is second-rate . So why was it so enjoyable  ? For one thing , the cast is great . Another nice touch is the music . I was overcome with the urge to get off the couch and start dancing .  It sucked me in , and it'll do the same to you  .x1=3x6=4.19x3=1x4=3x5=0x2=2\nFigure 5.2 A sample mini test document showing the extracted features in the vector x.\nLet\u2019s assume for the moment that we\u2019ve already learned a real-valued weight\nfor each of these features, and that the 6 weights corresponding to the 6 features\nare[2:5;\u00005:0;\u00001:2;0:5;2:0;0:7], while b= 0.1. (We\u2019ll discuss in the next section\nhow the weights are learned.) The weight w1, for example indicates how important\na feature the number of positive lexicon words ( great ,nice,enjoyable , etc.) is to\na positive sentiment decision, while w2tells us the importance of negative lexicon\nwords. Note that w1=2:5 is positive, while w2=\u00005:0, meaning that negative words\nare negatively associated with a positive sentiment decision, and are about twice as\nimportant as positive words.\nGiven these 6 features and the input review x,P(+jx)andP(\u0000jx)can be com-\nputed using Eq. 5.5:\np(+jx) =P(y=1jx) = s(w\u0001x+b)\n=s([2:5;\u00005:0;\u00001:2;0:5;2:0;0:7]\u0001[3;2;1;3;0;4:19]+0:1)\n=s(:833)\n=0:70 (5.8)\np(\u0000jx) =P(y=0jx) = 1\u0000s(w\u0001x+b)\n=0:30 82 CHAPTER 5 \u2022 L OGISTIC REGRESSION\n5.2.2 Other classi\ufb01cation tasks and features\nLogistic regression is applied to all sorts of NLP tasks, and any property of the input\ncan be a feature. Consider the task of period disambiguation : deciding if a periodperiod\ndisambiguation\nis the end of a sentence or part of a word, by classifying each period into one of two\nclasses, EOS (end-of-sentence) and not-EOS. We might use features like x1below\nexpressing that the current word is lower case, perhaps with a positive weight. Or a\nfeature expressing that the current word is in our abbreviations dictionary (\u201cProf.\u201d),\nperhaps with a negative weight. A feature can also express a combination of proper-\nties. For example a period following an upper case word is likely to be an EOS, but\nif the word itself is St.and the previous word is capitalized then the period is likely\npart of a shortening of the word street following a street name.\nx1=\u001a\n1 if \u201c Case(wi) =Lower\u201d\n0 otherwise\nx2=\u001a1 if \u201c wi2AcronymDict\u201d\n0 otherwise\nx3=\u001a1 if \u201c wi=St. & Case(wi\u00001) =Upper\u201d\n0 otherwise\nDesigning versus learning features: In classic models, features are designed by\nhand by examining the training set with an eye to linguistic intuitions and literature,\nsupplemented by insights from error analysis on the training set of an early version\nof a system. We can also consider ( feature interactions ), complex features that arefeature\ninteractions\ncombinations of more primitive features. We saw such a feature for period disam-\nbiguation above, where a period on the word St.was less likely to be the end of the\nsentence if the previous word was capitalized. Features can be created automatically\nviafeature templates , abstract speci\ufb01cations of features. For example a bigramfeature\ntemplates\ntemplate for period disambiguation might create a feature for every pair of words\nthat occurs before a period in the training set. Thus the feature space is sparse, since\nwe only have to create a feature if that n-gram exists in that position in the training\nset. The feature is generally created as a hash from the string descriptions. A user\ndescription of a feature as, \u201cbigram(American breakfast)\u201d is hashed into a unique\ninteger ithat becomes the feature number fi.\nIt should be clear from the prior paragraph that designing features by hand re-\nquires extensive human effort. For this reason, recent NLP systems avoid hand-\ndesigned features and instead focus on representation learning : ways to learn fea-\ntures automatically in an unsupervised way from the input. We\u2019ll introduce methods\nfor representation learning in Chapter 6 and Chapter 7.\nScaling input features: When different input features have extremely different\nranges of values, it\u2019s common to rescale them so they have comparable ranges. We\nstandardize input values by centering them to result in a zero mean and a standard standardize\ndeviation of one (this transformation is sometimes called the z-score ). That is, if miz-score\nis the mean of the values of feature xiacross the mobservations in the input dataset,\nandsiis the standard deviation of the values of features xiacross the input dataset,\nwe can replace each feature xiby a new feature x0\nicomputed as follows:\nmi=1\nmmX\nj=1x(j)\ni si=vuut1\nmmX\nj=1\u0010\nx(j)\ni\u0000mi\u00112\nx0\ni=xi\u0000mi\nsi(5.9) 5.2 \u2022 C LASSIFICATION WITH LOGISTIC REGRESSION 83\nAlternatively, we can normalize the input features values to lie between 0 and 1: normalize\nx0\ni=xi\u0000min(xi)\nmax(xi)\u0000min(xi)(5.10)\nHaving input data with comparable range is useful when comparing values across\nfeatures. Data scaling is especially important in large neural networks, since it helps\nspeed up gradient descent.\n5.2.3 Processing many examples at once\nWe\u2019ve shown the equations for logistic regression for a single example. But in prac-\ntice we\u2019ll of course want to process an entire test set with many examples. Let\u2019s\nsuppose we have a test set consisting of mtest examples each of which we\u2019d like\nto classify. We\u2019ll continue to use the notation from page 78, in which a superscript\nvalue in parentheses refers to the example index in some set of data (either for train-\ning or for test). So in this case each test example x(i)has a feature vector x(i),\n1\u0014i\u0014m. (As usual, we\u2019ll represent vectors and matrices in bold.)\nOne way to compute each output value \u02c6 y(i)is just to have a for-loop, and compute\neach test example one at a time:\nforeach x(i)in input [x(1);x(2);:::;x(m)]\ny(i)=s(w\u0001x(i)+b) (5.11)\nFor the \ufb01rst 3 test examples, then, we would be separately computing the pre-\ndicted \u02c6 y(i)as follows:\nP(y(1)=1jx(1)) = s(w\u0001x(1)+b)\nP(y(2)=1jx(2)) = s(w\u0001x(2)+b)\nP(y(3)=1jx(3)) = s(w\u0001x(3)+b)\nBut it turns out that we can slightly modify our original equation Eq. 5.5 to do\nthis much more ef\ufb01ciently. We\u2019ll use matrix arithmetic to assign a class to all the\nexamples with one matrix operation!\nFirst, we\u2019ll pack all the input feature vectors for each input xinto a single input\nmatrix X, where each row iis a row vector consisting of the feature vector for in-\nput example x(i)(i.e., the vector x(i)). Assuming each example has ffeatures and\nweights, Xwill therefore be a matrix of shape [m\u0002f], as follows:\nX=2\n66664x(1)\n1x(1)\n2:::x(1)\nf\nx(2)\n1x(2)\n2:::x(2)\nf\nx(3)\n1x(3)\n2:::x(3)\nf\n:::3\n77775(5.12)\nNow if we introduce bas a vector of length mwhich consists of the scalar bias\nterm brepeated mtimes, b= [b;b;:::;b], and ^ y= [\u02c6y(1);\u02c6y(2):::;\u02c6y(m)]as the vector of\noutputs (one scalar \u02c6 y(i)for each input x(i)and its feature vector x(i)), and represent\nthe weight vector was a column vector, we can compute all the outputs with a single\nmatrix multiplication and one addition:\ny=Xw+b (5.13) 84 CHAPTER 5 \u2022 L OGISTIC REGRESSION\nYou should convince yourself that Eq. 5.13 computes the same thing as our for-loop\nin Eq. 5.11. For example \u02c6 y(1), the \ufb01rst entry of the output vector y, will correctly be:\n\u02c6y(1)= [x(1)\n1;x(1)\n2;:::;x(1)\nf]\u0001[w1;w2;:::;wf]+b (5.14)\nNote that we had to reorder Xandwfrom the order they appeared in in Eq. 5.5 to\nmake the multiplications come out properly. Here is Eq. 5.13 again with the shapes\nshown:\ny=X w +b\n(m\u00021) ( m\u0002f)(f\u00021) (m\u00021) (5.15)\nModern compilers and compute hardware can compute this matrix operation very\nef\ufb01ciently, making the computation much faster, which becomes important when\ntraining or testing on very large datasets.\nNote by the way that we could have kept Xandwin the original order ( y=\nXw+b) if we had chosen to de\ufb01ne Xdifferently as a matrix of column vectors, one\nvector for each input example, instead of row vectors, and then it would have shape\n[f\u0002m]. But we conventionally represent inputs as rows.\n5.2.4 Choosing a classi\ufb01er\nLogistic regression has a number of advantages over naive Bayes. Naive Bayes has\noverly strong conditional independence assumptions. Consider two features which\nare strongly correlated; in fact, imagine that we just add the same feature f1twice.\nNaive Bayes will treat both copies of f1as if they were separate, multiplying them\nboth in, overestimating the evidence. By contrast, logistic regression is much more\nrobust to correlated features; if two features f1and f2are perfectly correlated, re-\ngression will simply assign part of the weight to w1and part to w2. Thus when\nthere are many correlated features, logistic regression will assign a more accurate\nprobability than naive Bayes. So logistic regression generally works better on larger\ndocuments or datasets and is a common default.\nDespite the less accurate probabilities, naive Bayes still often makes the correct\nclassi\ufb01cation decision. Furthermore, naive Bayes can work extremely well (some-\ntimes even better than logistic regression) on very small datasets (Ng and Jordan,\n2002) or short documents (Wang and Manning, 2012). Furthermore, naive Bayes is\neasy to implement and very fast to train (there\u2019s no optimization step). So it\u2019s still a\nreasonable approach to use in some situations.\n5.3 Multinomial logistic regression\nSometimes we need more than two classes. Perhaps we might want to do 3-way\nsentiment classi\ufb01cation (positive, negative, or neutral). Or we could be assigning\nsome of the labels we will introduce in Chapter 17, like the part of speech of a word\n(choosing from 10, 30, or even 50 different parts of speech), or the named entity\ntype of a phrase (choosing from tags like person, location, organization).\nIn such cases we use multinomial logistic regression , also called softmax re-multinomial\nlogistic\nregressiongression (in older NLP literature you will sometimes see the name maxent classi-\n\ufb01er). In multinomial logistic regression we want to label each observation with a\nclass kfrom a set of Kclasses, under the stipulation that only one of these classes is 5.3 \u2022 M ULTINOMIAL LOGISTIC REGRESSION 85\nthe correct one (sometimes called hard classi\ufb01cation ; an observation can not be in\nmultiple classes). Let\u2019s use the following representation: the output yfor each input\nxwill be a vector of length K. If class cis the correct class, we\u2019ll set yc=1, and\nset all the other elements of yto be 0, i.e., yc=1 and yj=08j6=c. A vector like\nthisy, with one value=1 and the rest 0, is called a one-hot vector . The job of the\nclassi\ufb01er is to produce an estimate vector ^ y. For each class k, the value \u02c6 ykwill be\nthe classi\ufb01er\u2019s estimate of the probability p(yk=1jx).\n5.3.1 Softmax\nThe multinomial logistic classi\ufb01er uses a generalization of the sigmoid, called the\nsoftmax function, to compute p(yk=1jx). The softmax function takes a vector softmax\nz= [z1;z2;:::;zK]ofKarbitrary values and maps them to a probability distribution,\nwith each value in the range [0,1], and all the values summing to 1. Like the sigmoid,\nit is an exponential function.\nFor a vector zof dimensionality K, the softmax is de\ufb01ned as:\nsoftmax (zi) =exp(zi)PK\nj=1exp(zj)1\u0014i\u0014K (5.16)\nThe softmax of an input vector z= [z1;z2;:::;zK]is thus a vector itself:\nsoftmax (z) =\"\nexp(z1)PK\ni=1exp(zi);exp(z2)PK\ni=1exp(zi);:::;exp(zK)PK\ni=1exp(zi)#\n(5.17)\nThe denominatorPK\ni=1exp(zi)is used to normalize all the values into probabilities.\nThus for example given a vector:\nz= [0:6;1:1;\u00001:5;1:2;3:2;\u00001:1]\nthe resulting (rounded) softmax( z) is\n[0:05;0:09;0:01;0:1;0:74;0:01][0:05;0:09;0:01;0:1;0:74;0:01]\nLike the sigmoid, the softmax has the property of squashing values toward 0 or 1.\nThus if one of the inputs is larger than the others, it will tend to push its probability\ntoward 1, and suppress the probabilities of the smaller inputs.\nFinally, note that, just as for the sigmoid, we refer to z, the vector of scores that\nis the input to the softmax, as logits (see (5.7).\n5.3.2 Applying softmax in logistic regression\nWhen we apply softmax for logistic regression, the input will (just as for the sig-\nmoid) be the dot product between a weight vector wand an input vector x(plus a\nbias). But now we\u2019ll need separate weight vectors wkand bias bkfor each of the K\nclasses. The probability of each of our output classes \u02c6 ykcan thus be computed as:\np(yk=1jx) =exp(wk\u0001x+bk)\nKX\nj=1exp(wj\u0001x+bj)(5.18) 86 CHAPTER 5 \u2022 L OGISTIC REGRESSION\nThe form of Eq. 5.18 makes it seem that we would compute each output sep-\narately. Instead, it\u2019s more common to set up the equation for more ef\ufb01cient com-\nputation by modern vector processing hardware. We\u2019ll do this by representing the\nset of Kweight vectors as a weight matrix Wand a bias vector b. Each row kof\nWcorresponds to the vector of weights wk.Wthus has shape [K\u0002f], for Kthe\nnumber of output classes and fthe number of input features. The bias vector bhas\none value for each of the Koutput classes. If we represent the weights in this way,\nwe can compute \u02c6y, the vector of output probabilities for each of the Kclasses, by a\nsingle elegant equation:\n\u02c6y=softmax (Wx+b) (5.19)\nIf you work out the matrix arithmetic, you can see that the estimated score of\nthe \ufb01rst output class \u02c6 y1(before we take the softmax) will correctly turn out to be\nw1\u0001x+b1.\nFig. 5.3 shows an intuition of the role of the weight vector versus weight matrix\nin the computation of the output class probabilities for binary versus multinomial\nlogistic regression.\n5.3.3 Features in Multinomial Logistic Regression\nFeatures in multinomial logistic regression act like features in binary logistic regres-\nsion, with the difference mentioned above that we\u2019ll need separate weight vectors\nand biases for each of the Kclasses. Recall our binary exclamation point feature x5\nfrom page 81:\nx5=\u001a1 if \u201c!\u201d2doc\n0 otherwise\nIn binary classi\ufb01cation a positive weight w5on a feature in\ufb02uences the classi\ufb01er\ntoward y=1 (positive sentiment) and a negative weight in\ufb02uences it toward y=0\n(negative sentiment) with the absolute value indicating how important the feature\nis. For multinomial logistic regression, by contrast, with separate weights for each\nclass, a feature can be evidence for or against each individual class.\nIn 3-way multiclass sentiment classi\ufb01cation, for example, we must assign each\ndocument one of the 3 classes +,\u0000, or 0 (neutral). Now a feature related to excla-\nmation marks might have a negative weight for 0 documents, and a positive weight\nfor+or\u0000documents:\nFeature De\ufb01nition w5;+w5;\u0000w5;0\nf5(x)\u001a1 if \u201c!\u201d2doc\n0 otherwise3:5 3:1\u00005:3\nBecause these feature weights are dependent both on the input text and the output\nclass, we sometimes make this dependence explicit and represent the features them-\nselves as f(x;y): a function of both the input and the class. Using such a notation\nf5(x)above could be represented as three features f5(x;+),f5(x;\u0000), and f5(x;0),\neach of which has a single weight. We\u2019ll use this kind of notation in our description\nof the CRF in Chapter 17. 5.4 \u2022 L EARNING IN LOGISTIC REGRESSION 87\nBinary Logistic Regression\nw[f \u2a091]Outputsigmoid[1\u2a09f]Input wordsp(+) = 1- p(-)\u2026y^xyInput featurevector [scalar]positive lexiconwords = 1count of \u201cno\u201d = 0wordcount=3x1x2x3xfdessert   was    greatWeight vector\nMultinomial Logistic Regression\nW[f\u2a091]Outputsoftmax[K\u2a09f]Input wordsp(+)\u2026y1^y2^y3^xyInput featurevector [K\u2a091]positive lexiconwords = 1count of \u201cno\u201d = 0wordcount=3x1x2x3xfdessert   was    greatp(-)p(neut)Weight matrixThese f red weightsare a row of W correspondingto weight vector w3,(= weights for class 3)\nFigure 5.3 Binary versus multinomial logistic regression. Binary logistic regression uses a\nsingle weight vector w, and has a scalar output \u02c6 y. In multinomial logistic regression we have\nKseparate weight vectors corresponding to the Kclasses, all packed into a single weight\nmatrix W, and a vector output \u02c6y. We omit the biases from both \ufb01gures for clarity.\n5.4 Learning in Logistic Regression\nHow are the parameters of the model, the weights wand bias b, learned? Logistic\nregression is an instance of supervised classi\ufb01cation in which we know the correct\nlabel y(either 0 or 1) for each observation x. What the system produces via Eq. 5.5\nis \u02c6y, the system\u2019s estimate of the true y. We want to learn parameters (meaning w\nandb) that make \u02c6 yfor each training observation as close as possible to the true y.\nThis requires two components that we foreshadowed in the introduction to the\nchapter. The \ufb01rst is a metric for how close the current label ( \u02c6 y) is to the true gold\nlabel y. Rather than measure similarity, we usually talk about the opposite of this:\nthedistance between the system output and the gold output, and we call this distance\nthelossfunction or the cost function . In the next section we\u2019ll introduce the loss loss\nfunction that is commonly used for logistic regression and also for neural networks, 88 CHAPTER 5 \u2022 L OGISTIC REGRESSION\nthecross-entropy loss .\nThe second thing we need is an optimization algorithm for iteratively updating\nthe weights so as to minimize this loss function. The standard algorithm for this is\ngradient descent ; we\u2019ll introduce the stochastic gradient descent algorithm in the\nfollowing section.\nWe\u2019ll describe these algorithms for the simpler case of binary logistic regres-\nsion in the next two sections, and then turn to multinomial logistic regression in\nSection 5.8.\n5.5 The cross-entropy loss function\nWe need a loss function that expresses, for an observation x, how close the classi\ufb01er\noutput ( \u02c6 y=s(w\u0001x+b)) is to the correct output ( y, which is 0 or 1). We\u2019ll call this:\nL(\u02c6y;y) = How much \u02c6 ydiffers from the true y (5.20)\nWe do this via a loss function that prefers the correct class labels of the train-\ning examples to be more likely . This is called conditional maximum likelihood\nestimation : we choose the parameters w;bthatmaximize the log probability of\nthe true ylabels in the training data given the observations x. The resulting loss\nfunction is the negative log likelihood loss , generally called the cross-entropy loss .cross-entropy\nloss\nLet\u2019s derive this loss function, applied to a single observation x. We\u2019d like to\nlearn weights that maximize the probability of the correct label p(yjx). Since there\nare only two discrete outcomes (1 or 0), this is a Bernoulli distribution, and we can\nexpress the probability p(yjx)that our classi\ufb01er produces for one observation as the\nfollowing (keeping in mind that if y=1, Eq. 5.21 simpli\ufb01es to \u02c6 y; ify=0, Eq. 5.21\nsimpli\ufb01es to 1\u0000\u02c6y):\np(yjx) = \u02c6yy(1\u0000\u02c6y)1\u0000y(5.21)\nNow we take the log of both sides. This will turn out to be handy mathematically,\nand doesn\u2019t hurt us; whatever values maximize a probability will also maximize the\nlog of the probability:\nlogp(yjx) = log\u0002\n\u02c6yy(1\u0000\u02c6y)1\u0000y\u0003\n=ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y) (5.22)\nEq. 5.22 describes a log likelihood that should be maximized. In order to turn this\ninto a loss function (something that we need to minimize), we\u2019ll just \ufb02ip the sign on\nEq. 5.22. The result is the cross-entropy loss LCE:\nLCE(\u02c6y;y) =\u0000logp(yjx) =\u0000[ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)] (5.23)\nFinally, we can plug in the de\ufb01nition of \u02c6 y=s(w\u0001x+b):\nLCE(\u02c6y;y) =\u0000[ylogs(w\u0001x+b)+(1\u0000y)log(1\u0000s(w\u0001x+b))] (5.24)\nLet\u2019s see if this loss function does the right thing for our example from Fig. 5.2. We\nwant the loss to be smaller if the model\u2019s estimate is close to correct, and bigger if\nthe model is confused. So \ufb01rst let\u2019s suppose the correct gold label for the sentiment\nexample in Fig. 5.2 is positive, i.e., y=1. In this case our model is doing well, since 5.6 \u2022 G RADIENT DESCENT 89\nfrom Eq. 5.8 it indeed gave the example a higher probability of being positive (.70)\nthan negative (.30). If we plug s(w\u0001x+b) =:70 and y=1 into Eq. 5.24, the right\nside of the equation drops out, leading to the following loss (we\u2019ll use log to mean\nnatural log when the base is not speci\ufb01ed):\nLCE(\u02c6y;y) =\u0000[ylogs(w\u0001x+b)+(1\u0000y)log(1\u0000s(w\u0001x+b))]\n=\u0000[logs(w\u0001x+b)]\n=\u0000log(:70)\n= :36\nBy contrast, let\u2019s pretend instead that the example in Fig. 5.2 was actually negative,\ni.e., y=0 (perhaps the reviewer went on to say \u201cBut bottom line, the movie is\nterrible! I beg you not to see it!\u201d). In this case our model is confused and we\u2019d want\nthe loss to be higher. Now if we plug y=0 and 1\u0000s(w\u0001x+b) =:30 from Eq. 5.8\ninto Eq. 5.24, the left side of the equation drops out:\nLCE(\u02c6y;y) =\u0000[ylogs(w\u0001x+b)+(1\u0000y)log(1\u0000s(w\u0001x+b))]\n= \u0000[log(1\u0000s(w\u0001x+b))]\n= \u0000log(:30)\n= 1:2\nSure enough, the loss for the \ufb01rst classi\ufb01er (.36) is less than the loss for the second\nclassi\ufb01er (1.2).\nWhy does minimizing this negative log probability do what we want? A perfect\nclassi\ufb01er would assign probability 1 to the correct outcome ( y=1 or y=0) and\nprobability 0 to the incorrect outcome. That means if yequals 1, the higher \u02c6 yis (the\ncloser it is to 1), the better the classi\ufb01er; the lower \u02c6 yis (the closer it is to 0), the\nworse the classi\ufb01er. If yequals 0, instead, the higher 1 \u0000\u02c6yis (closer to 1), the better\nthe classi\ufb01er. The negative log of \u02c6 y(if the true yequals 1) or 1\u0000\u02c6y(if the true y\nequals 0) is a convenient loss metric since it goes from 0 (negative log of 1, no loss)\nto in\ufb01nity (negative log of 0, in\ufb01nite loss). This loss function also ensures that as\nthe probability of the correct answer is maximized, the probability of the incorrect\nanswer is minimized; since the two sum to one, any increase in the probability of the\ncorrect answer is coming at the expense of the incorrect answer. It\u2019s called the cross-\nentropy loss, because Eq. 5.22 is also the formula for the cross-entropy between the\ntrue probability distribution yand our estimated distribution \u02c6 y.\nNow we know what we want to minimize; in the next section, we\u2019ll see how to\n\ufb01nd the minimum.\n5.6 Gradient Descent\nOur goal with gradient descent is to \ufb01nd the optimal weights: minimize the loss\nfunction we\u2019ve de\ufb01ned for the model. In Eq. 5.25 below, we\u2019ll explicitly represent\nthe fact that the cross-entropy loss function LCEis parameterized by the weights. In\nmachine learning in general we refer to the parameters being learned as q; in the\ncase of logistic regression q=fw;bg. So the goal is to \ufb01nd the set of weights which\nminimizes the loss function, averaged over all examples:\n\u02c6q=argmin\nq1\nmmX\ni=1LCE(f(x(i);q);y(i)) (5.25) 90 CHAPTER 5 \u2022 L OGISTIC REGRESSION\nHow shall we \ufb01nd the minimum of this (or any) loss function? Gradient descent is\na method that \ufb01nds a minimum of a function by \ufb01guring out in which direction (in\nthe space of the parameters q) the function\u2019s slope is rising the most steeply, and\nmoving in the opposite direction. The intuition is that if you are hiking in a canyon\nand trying to descend most quickly down to the river at the bottom, you might look\naround yourself in all directions, \ufb01nd the direction where the ground is sloping the\nsteepest, and walk downhill in that direction.\nFor logistic regression, this loss function is conveniently convex . A convex func- convex\ntion has at most one minimum; there are no local minima to get stuck in, so gradient\ndescent starting from any point is guaranteed to \ufb01nd the minimum. (By contrast,\nthe loss for multi-layer neural networks is non-convex, and gradient descent may\nget stuck in local minima for neural network training and never \ufb01nd the global opti-\nmum.)\nAlthough the algorithm (and the concept of gradient) are designed for direction\nvectors , let\u2019s \ufb01rst consider a visualization of the case where the parameter of our\nsystem is just a single scalar w, shown in Fig. 5.4.\nGiven a random initialization of wat some value w1, and assuming the loss\nfunction Lhappened to have the shape in Fig. 5.4, we need the algorithm to tell us\nwhether at the next iteration we should move left (making w2smaller than w1) or\nright (making w2bigger than w1) to reach the minimum.\nwLoss\n0w1wminslope of loss at w1 is negative(goal)one stepof gradientdescent\nFigure 5.4 The \ufb01rst step in iteratively \ufb01nding the minimum of this loss function, by moving\nwin the reverse direction from the slope of the function. Since the slope is negative, we need\nto move win a positive direction, to the right. Here superscripts are used for learning steps,\nsow1means the initial value of w(which is 0), w2the value at the second step, and so on.\nThe gradient descent algorithm answers this question by \ufb01nding the gradient gradient\nof the loss function at the current point and moving in the opposite direction. The\ngradient of a function of many variables is a vector pointing in the direction of the\ngreatest increase in a function. The gradient is a multi-variable generalization of the\nslope, so for a function of one variable like the one in Fig. 5.4, we can informally\nthink of the gradient as the slope. The dotted line in Fig. 5.4 shows the slope of this\nhypothetical loss function at point w=w1. You can see that the slope of this dotted\nline is negative. Thus to \ufb01nd the minimum, gradient descent tells us to go in the\nopposite direction: moving win a positive direction.\nThe magnitude of the amount to move in gradient descent is the value of the\nsloped\ndwL(f(x;w);y)weighted by a learning rate h. A higher (faster) learning learning rate 5.6 \u2022 G RADIENT DESCENT 91\nrate means that we should move wmore on each step. The change we make in our\nparameter is the learning rate times the gradient (or the slope, in our single-variable\nexample):\nwt+1=wt\u0000hd\ndwL(f(x;w);y) (5.26)\nNow let\u2019s extend the intuition from a function of one scalar variable wto many\nvariables, because we don\u2019t just want to move left or right, we want to know where\nin the N-dimensional space (of the Nparameters that make up q) we should move.\nThe gradient is just such a vector; it expresses the directional components of the\nsharpest slope along each of those Ndimensions. If we\u2019re just imagining two weight\ndimensions (say for one weight wand one bias b), the gradient might be a vector with\ntwo orthogonal components, each of which tells us how much the ground slopes in\nthewdimension and in the bdimension. Fig. 5.5 shows a visualization of the value\nof a 2-dimensional gradient vector taken at the red point.\nIn an actual logistic regression, the parameter vector wis much longer than 1 or\n2, since the input feature vector xcan be quite long, and we need a weight wifor\neach xi. For each dimension/variable wiinw(plus the bias b), the gradient will have\na component that tells us the slope with respect to that variable. In each dimension\nwi, we express the slope as a partial derivative\u00b6\n\u00b6wiof the loss function. Essentially\nwe\u2019re asking: \u201cHow much would a small change in that variable wiin\ufb02uence the\ntotal loss function L?\u201d\nFormally, then, the gradient of a multi-variable function fis a vector in which\neach component expresses the partial derivative of fwith respect to one of the vari-\nables. We\u2019ll use the inverted Greek delta symbol \u00d1to refer to the gradient, and\nrepresent \u02c6 yasf(x;q)to make the dependence on qmore obvious:\n\u00d1L(f(x;q);y) =2\n6666664\u00b6\n\u00b6w1L(f(x;q);y)\n\u00b6\n\u00b6w2L(f(x;q);y)\n...\n\u00b6\n\u00b6wnL(f(x;q);y)\n\u00b6\n\u00b6bL(f(x;q);y)3\n7777775(5.27)\nThe \ufb01nal equation for updating qbased on the gradient is thus\nqt+1=qt\u0000h\u00d1L(f(x;q);y) (5.28)\nCost(w,b)\nwb\nFigure 5.5 Visualization of the gradient vector at the red point in two dimensions wand\nb, showing a red arrow in the x-y plane pointing in the direction we will go to look for the\nminimum: the opposite direction of the gradient (recall that the gradient points in the direction\nof increase not decrease). 92 CHAPTER 5 \u2022 L OGISTIC REGRESSION\n5.6.1 The Gradient for Logistic Regression\nIn order to update q, we need a de\ufb01nition for the gradient \u00d1L(f(x;q);y). Recall that\nfor logistic regression, the cross-entropy loss function is:\nLCE(\u02c6y;y) =\u0000[ylogs(w\u0001x+b)+(1\u0000y)log(1\u0000s(w\u0001x+b))] (5.29)\nIt turns out that the derivative of this function for one observation vector xis Eq. 5.30\n(the interested reader can see Section 5.10 for the derivation of this equation):\n\u00b6LCE(\u02c6y;y)\n\u00b6wj= [s(w\u0001x+b)\u0000y]xj\n= ( \u02c6y\u0000y)xj (5.30)\nYou\u2019ll also sometimes see this equation in the equivalent form:\n\u00b6LCE(\u02c6y;y)\n\u00b6wj=\u0000(y\u0000\u02c6y)xj (5.31)\nNote in these equations that the gradient with respect to a single weight wjrep-\nresents a very intuitive value: the difference between the true yand our estimated\n\u02c6y=s(w\u0001x+b)for that observation, multiplied by the corresponding input value\nxj.\n5.6.2 The Stochastic Gradient Descent Algorithm\nStochastic gradient descent is an online algorithm that minimizes the loss function\nby computing its gradient after each training example, and nudging qin the right\ndirection (the opposite direction of the gradient). (An \u201conline algorithm\u201d is one that\nprocesses its input example by example, rather than waiting until it sees the entire\ninput.) Stochastic gradient descent is called stochastic because it chooses a single\nrandom example at a time; in Section 5.6.4 we\u2019ll discuss other versions of gradient\ndescent that batch many examples at once. Fig. 5.6 shows the algorithm.\nThe learning rate his ahyperparameter that must be adjusted. If it\u2019s too high, hyperparameter\nthe learner will take steps that are too large, overshooting the minimum of the loss\nfunction. If it\u2019s too low, the learner will take steps that are too small, and take too\nlong to get to the minimum. It is common to start with a higher learning rate and then\nslowly decrease it, so that it is a function of the iteration kof training; the notation\nhkcan be used to mean the value of the learning rate at iteration k.\nWe\u2019ll discuss hyperparameters in more detail in Chapter 7, but in short, they are\na special kind of parameter for any machine learning model. Unlike regular param-\neters of a model (weights like wandb), which are learned by the algorithm from\nthe training set, hyperparameters are special parameters chosen by the algorithm\ndesigner that affect how the algorithm works.\n5.6.3 Working through an example\nLet\u2019s walk through a single step of the gradient descent algorithm. We\u2019ll use a\nsimpli\ufb01ed version of the example in Fig. 5.2 as it sees a single observation x, whose\ncorrect value is y=1 (this is a positive review), and with a feature vector x= [x1;x2]\nconsisting of these two features:\nx1=3 (count of positive lexicon words)\nx2=2 (count of negative lexicon words) 5.6 \u2022 G RADIENT DESCENT 93\nfunction STOCHASTIC GRADIENT DESCENT (L(),f(),x,y)returns q\n# where: L is the loss function\n# f is a function parameterized by q\n# x is the set of training inputs x(1);x(2);:::;x(m)\n# y is the set of training outputs (labels) y(1);y(2);:::;y(m)\nq 0 # (or small random values)\nrepeat til done # see caption\nFor each training tuple (x(i);y(i))(in random order)\n1. Optional (for reporting): # How are we doing on this tuple?\nCompute \u02c6 y(i)=f(x(i);q)# What is our estimated output \u02c6 y?\nCompute the loss L(\u02c6y(i);y(i))# How far off is \u02c6 y(i)from the true output y(i)?\n2.g \u00d1qL(f(x(i);q);y(i)) # How should we move qto maximize loss?\n3.q q\u0000hg # Go the other way instead\nreturn q\nFigure 5.6 The stochastic gradient descent algorithm. Step 1 (computing the loss) is used\nmainly to report how well we are doing on the current tuple; we don\u2019t need to compute the\nloss in order to compute the gradient. The algorithm can terminate when it converges (when\nthe gradient norm <\u000f), or when progress halts (for example when the loss starts going up on\na held-out set). Weights are initialized to 0 for logistic regression, but to small random values\nfor neural networks, as we\u2019ll see in Chapter 7.\nLet\u2019s assume the initial weights and bias in q0are all set to 0, and the initial learning\nratehis 0.1:\nw1=w2=b=0\nh=0:1\nThe single update step requires that we compute the gradient, multiplied by the\nlearning rate\nqt+1=qt\u0000h\u00d1qL(f(x(i);q);y(i))\nIn our mini example there are three parameters, so the gradient vector has 3 dimen-\nsions, for w1,w2, and b. We can compute the \ufb01rst gradient as follows:\n\u00d1w;bL=2\n64\u00b6LCE(\u02c6y;y)\n\u00b6w1\u00b6LCE(\u02c6y;y)\n\u00b6w2\u00b6LCE(\u02c6y;y)\n\u00b6b3\n75=2\n4(s(w\u0001x+b)\u0000y)x1\n(s(w\u0001x+b)\u0000y)x2\ns(w\u0001x+b)\u0000y3\n5=2\n4(s(0)\u00001)x1\n(s(0)\u00001)x2\ns(0)\u000013\n5=2\n4\u00000:5x1\n\u00000:5x2\n\u00000:53\n5=2\n4\u00001:5\n\u00001:0\n\u00000:53\n5\nNow that we have a gradient, we compute the new parameter vector q1by moving\nq0in the opposite direction from the gradient:\nq1=2\n4w1\nw2\nb3\n5\u0000h2\n4\u00001:5\n\u00001:0\n\u00000:53\n5=2\n4:15\n:1\n:053\n5\nSo after one step of gradient descent, the weights have shifted to be: w1=:15,\nw2=:1, and b=:05.\nNote that this observation xhappened to be a positive example. We would expect\nthat after seeing more negative examples with high counts of negative words, that\nthe weight w2would shift to have a negative value. 94 CHAPTER 5 \u2022 L OGISTIC REGRESSION\n5.6.4 Mini-batch training\nStochastic gradient descent is called stochastic because it chooses a single random\nexample at a time, moving the weights so as to improve performance on that single\nexample. That can result in very choppy movements, so it\u2019s common to compute the\ngradient over batches of training instances rather than a single instance.\nFor example in batch training we compute the gradient over the entire dataset. batch training\nBy seeing so many examples, batch training offers a superb estimate of which di-\nrection to move the weights, at the cost of spending a lot of time processing every\nsingle example in the training set to compute this perfect direction.\nA compromise is mini-batch training: we train on a group of mexamples (per- mini-batch\nhaps 512, or 1024) that is less than the whole dataset. (If mis the size of the dataset,\nthen we are doing batch gradient descent; if m=1, we are back to doing stochas-\ntic gradient descent.) Mini-batch training also has the advantage of computational\nef\ufb01ciency. The mini-batches can easily be vectorized, choosing the size of the mini-\nbatch based on the computational resources. This allows us to process all the exam-\nples in one mini-batch in parallel and then accumulate the loss, something that\u2019s not\npossible with individual or batch training.\nWe just need to de\ufb01ne mini-batch versions of the cross-entropy loss function\nwe de\ufb01ned in Section 5.5 and the gradient in Section 5.6.1. Let\u2019s extend the cross-\nentropy loss for one example from Eq. 5.23 to mini-batches of size m. We\u2019ll continue\nto use the notation that x(i)andy(i)mean the ith training features and training label,\nrespectively. We make the assumption that the training examples are independent:\nlogp(training labels ) = logmY\ni=1p(y(i)jx(i))\n=mX\ni=1logp(y(i)jx(i))\n=\u0000mX\ni=1LCE(\u02c6y(i);y(i)) (5.32)\nNow the cost function for the mini-batch of mexamples is the average loss for each\nexample:\nCost(\u02c6y;y) =1\nmmX\ni=1LCE(\u02c6y(i);y(i))\n=\u00001\nmmX\ni=1y(i)logs(w\u0001x(i)+b)+(1\u0000y(i))log\u0010\n1\u0000s(w\u0001x(i)+b)\u0011\n(5.33)\nThe mini-batch gradient is the average of the individual gradients from Eq. 5.30:\n\u00b6Cost(\u02c6y;y)\n\u00b6wj=1\nmmX\ni=1h\ns(w\u0001x(i)+b)\u0000y(i)i\nx(i)\nj(5.34)\nInstead of using the sum notation, we can more ef\ufb01ciently compute the gradient\nin its matrix form, following the vectorization we saw on page 83, where we have\na matrix Xof size [m\u0002f]representing the minputs in the batch, and a vector yof\nsize[m\u00021]representing the correct outputs: 5.7 \u2022 R EGULARIZATION 95\n\u00b6Cost(\u02c6y;y)\n\u00b6w=1\nm(\u02c6y\u0000y)|X\n=1\nm(s(Xw+b)\u0000y)|X (5.35)\n5.7 Regularization\nNumquam ponenda est pluralitas sine necessitate\n\u2018Plurality should never be proposed unless needed\u2019\nWilliam of Occam\nThere is a problem with learning weights that make the model perfectly match the\ntraining data. If a feature is perfectly predictive of the outcome because it happens\nto only occur in one class, it will be assigned a very high weight. The weights for\nfeatures will attempt to perfectly \ufb01t details of the training set, in fact too perfectly,\nmodeling noisy factors that just accidentally correlate with the class. This problem is\ncalled over\ufb01tting . A good model should be able to generalize well from the training over\ufb01tting\ngeneralize data to the unseen test set, but a model that over\ufb01ts will have poor generalization.\nTo avoid over\ufb01tting, a new regularization term R(q)is added to the loss func- regularization\ntion in Eq. 5.25, resulting in the following loss for a batch of mexamples (slightly\nrewritten from Eq. 5.25 to be maximizing log probability rather than minimizing\nloss, and removing the1\nmterm which doesn\u2019t affect the argmax):\n\u02c6q=argmax\nqmX\ni=1logP(y(i)jx(i))\u0000aR(q) (5.36)\nThe new regularization term R(q)is used to penalize large weights. Thus a setting\nof the weights that matches the training data perfectly\u2014 but uses many weights with\nhigh values to do so\u2014will be penalized more than a setting that matches the data a\nlittle less well, but does so using smaller weights. There are two common ways to\ncompute this regularization term R(q).L2 regularization is a quadratic function ofL2\nregularization\nthe weight values, named because it uses the (square of the) L2 norm of the weight\nvalues. The L2 norm, jjqjj2, is the same as the Euclidean distance of the vector q\nfrom the origin. If qconsists of nweights, then:\nR(q) =jjqjj2\n2=nX\nj=1q2\nj (5.37)\nThe L2 regularized loss function becomes:\n\u02c6q=argmax\nq\"mX\ni=1logP(y(i)jx(i))#\n\u0000anX\nj=1q2\nj (5.38)\nL1 regularization is a linear function of the weight values, named after the L1 normL1\nregularization\njjWjj1, the sum of the absolute values of the weights, or Manhattan distance (the 96 CHAPTER 5 \u2022 L OGISTIC REGRESSION\nManhattan distance is the distance you\u2019d have to walk between two points in a city\nwith a street grid like New York):\nR(q) =jjqjj1=nX\ni=1jqij (5.39)\nThe L1 regularized loss function becomes:\n\u02c6q=argmax\nq\"mX\ni=1logP(y(i)jx(i))#\n\u0000anX\nj=1jqjj (5.40)\nThese kinds of regularization come from statistics, where L1 regularization is called\nlasso regression (Tibshirani, 1996) and L2 regularization is called ridge regression , lasso\nridge and both are commonly used in language processing. L2 regularization is easier to\noptimize because of its simple derivative (the derivative of q2is just 2 q), while\nL1 regularization is more complex (the derivative of jqjis non-continuous at zero).\nBut while L2 prefers weight vectors with many small weights, L1 prefers sparse\nsolutions with some larger weights but many more weights set to zero. Thus L1\nregularization leads to much sparser weight vectors, that is, far fewer features.\nBoth L1 and L2 regularization have Bayesian interpretations as constraints on\nthe prior of how weights should look. L1 regularization can be viewed as a Laplace\nprior on the weights. L2 regularization corresponds to assuming that weights are\ndistributed according to a Gaussian distribution with mean m=0. In a Gaussian\nor normal distribution, the further away a value is from the mean, the lower its\nprobability (scaled by the variance s). By using a Gaussian prior on the weights, we\nare saying that weights prefer to have the value 0. A Gaussian for a weight qjis\n1q\n2ps2\njexp \n\u0000(qj\u0000mj)2\n2s2\nj!\n(5.41)\nIf we multiply each weight by a Gaussian prior on the weight, we are thus maximiz-\ning the following constraint:\n\u02c6q=argmax\nqmY\ni=1P(y(i)jx(i))\u0002nY\nj=11q\n2ps2\njexp \n\u0000(qj\u0000mj)2\n2s2\nj!\n(5.42)\nwhich in log space, with m=0, and assuming 2 s2=1, corresponds to\n\u02c6q=argmax\nqmX\ni=1logP(y(i)jx(i))\u0000anX\nj=1q2\nj (5.43)\nwhich is in the same form as Eq. 5.38.\n5.8 Learning in Multinomial Logistic Regression\nThe loss function for multinomial logistic regression generalizes the loss function\nfor binary logistic regression from 2 to Kclasses. Recall that that the cross-entropy\nloss for binary logistic regression (repeated from Eq. 5.23) is:\nLCE(\u02c6y;y) =\u0000logp(yjx) =\u0000[ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)] (5.44) 5.8 \u2022 L EARNING IN MULTINOMIAL LOGISTIC REGRESSION 97\nThe loss function for multinomial logistic regression generalizes the two terms in\nEq. 5.44 (one that is non-zero when y=1 and one that is non-zero when y=0) to\nKterms. As we mentioned above, for multinomial regression we\u2019ll represent both y\nand\u02c6yas vectors. The true label yis a vector with Kelements, each corresponding\nto a class, with yc=1 if the correct class is c, with all other elements of ybeing 0.\nAnd our classi\ufb01er will produce an estimate vector with Kelements \u02c6y, each element\n\u02c6ykof which represents the estimated probability p(yk=1jx).\nThe loss function for a single example x, generalizing from binary logistic re-\ngression, is the sum of the logs of the Koutput classes, each weighted by the indi-\ncator function yk(Eq. 5.45). This turns out to be just the negative log probability of\nthe correct class c(Eq. 5.46):\nLCE(\u02c6y;y) =\u0000KX\nk=1yklog \u02c6yk (5.45)\n=\u0000log \u02c6yc;(where cis the correct class) (5.46)\n=\u0000log \u02c6p(yc=1jx)(where cis the correct class)\n=\u0000logexp(wc\u0001x+bc)PK\nj=1exp(wj\u0001x+bj)(cis the correct class) (5.47)\nHow did we get from Eq. 5.45 to Eq. 5.46? Because only one class (let\u2019s call it c) is\nthe correct one, the vector ytakes the value 1 only for this value of k, i.e., has yc=1\nandyj=08j6=c. That means the terms in the sum in Eq. 5.45 will all be 0 except\nfor the term corresponding to the true class c. Hence the cross-entropy loss is simply\nthe log of the output probability corresponding to the correct class, and we therefore\nalso call Eq. 5.46 the negative log likelihood loss .negative log\nlikelihood loss\nOf course for gradient descent we don\u2019t need the loss, we need its gradient. The\ngradient for a single example turns out to be very similar to the gradient for binary\nlogistic regression, (\u02c6y\u0000y)x, that we saw in Eq. 5.30. Let\u2019s consider one piece of the\ngradient, the derivative for a single weight. For each class k, the weight of the ith\nelement of input xiswk;i. What is the partial derivative of the loss with respect to\nwk;i? This derivative turns out to be just the difference between the true value for the\nclass k(which is either 1 or 0) and the probability the classi\ufb01er outputs for class k,\nweighted by the value of the input xicorresponding to the ith element of the weight\nvector for class k:\n\u00b6LCE\n\u00b6wk;i=\u0000(yk\u0000\u02c6yk)xi\n=\u0000(yk\u0000p(yk=1jx))xi\n=\u0000 \nyk\u0000exp(wk\u0001x+bk)PK\nj=1exp(wj\u0001x+bj)!\nxi (5.48)\nWe\u2019ll return to this case of the gradient for softmax regression when we introduce\nneural networks in Chapter 7, and at that time we\u2019ll also discuss the derivation of\nthis gradient in equations Eq. 7.33\u2013Eq. 7.41. 98 CHAPTER 5 \u2022 L OGISTIC REGRESSION\n5.9 Interpreting models\nOften we want to know more than just the correct classi\ufb01cation of an observation.\nWe want to know why the classi\ufb01er made the decision it did. That is, we want our\ndecision to be interpretable . Interpretability can be hard to de\ufb01ne strictly, but the interpretable\ncore idea is that as humans we should know why our algorithms reach the conclu-\nsions they do. Because the features to logistic regression are often human-designed,\none way to understand a classi\ufb01er\u2019s decision is to understand the role each feature\nplays in the decision. Logistic regression can be combined with statistical tests (the\nlikelihood ratio test, or the Wald test); investigating whether a particular feature is\nsigni\ufb01cant by one of these tests, or inspecting its magnitude (how large is the weight\nwassociated with the feature?) can help us interpret why the classi\ufb01er made the\ndecision it makes. This is enormously important for building transparent models.\nFurthermore, in addition to its use as a classi\ufb01er, logistic regression in NLP and\nmany other \ufb01elds is widely used as an analytic tool for testing hypotheses about the\neffect of various explanatory variables (features). In text classi\ufb01cation, perhaps we\nwant to know if logically negative words ( no, not, never ) are more likely to be asso-\nciated with negative sentiment, or if negative reviews of movies are more likely to\ndiscuss the cinematography. However, in doing so it\u2019s necessary to control for po-\ntential confounds: other factors that might in\ufb02uence sentiment (the movie genre, the\nyear it was made, perhaps the length of the review in words). Or we might be study-\ning the relationship between NLP-extracted linguistic features and non-linguistic\noutcomes (hospital readmissions, political outcomes, or product sales), but need to\ncontrol for confounds (the age of the patient, the county of voting, the brand of the\nproduct). In such cases, logistic regression allows us to test whether some feature is\nassociated with some outcome above and beyond the effect of other features.\n5.10 Advanced: Deriving the Gradient Equation\nIn this section we give the derivation of the gradient of the cross-entropy loss func-\ntion LCEfor logistic regression. Let\u2019s start with some quick calculus refreshers.\nFirst, the derivative of ln (x):\nd\ndxln(x) =1\nx(5.49)\nSecond, the (very elegant) derivative of the sigmoid:\nds(z)\ndz=s(z)(1\u0000s(z)) (5.50)\nFinally, the chain rule of derivatives. Suppose we are computing the derivative chain rule\nof a composite function f(x) =u(v(x)). The derivative of f(x)is the derivative of\nu(x)with respect to v(x)times the derivative of v(x)with respect to x:\nd f\ndx=du\ndv\u0001dv\ndx(5.51)\nFirst, we want to know the derivative of the loss function with respect to a single\nweight wj(we\u2019ll need to compute it for each weight, and for the bias): 5.11 \u2022 S UMMARY 99\n\u00b6LCE\n\u00b6wj=\u00b6\n\u00b6wj\u0000[ylogs(w\u0001x+b)+(1\u0000y)log(1\u0000s(w\u0001x+b))]\n=\u0000\u0014\u00b6\n\u00b6wjylogs(w\u0001x+b)+\u00b6\n\u00b6wj(1\u0000y)log[1\u0000s(w\u0001x+b)]\u0015\n(5.52)\nNext, using the chain rule, and relying on the derivative of log:\n\u00b6LCE\n\u00b6wj=\u0000y\ns(w\u0001x+b)\u00b6\n\u00b6wjs(w\u0001x+b)\u00001\u0000y\n1\u0000s(w\u0001x+b)\u00b6\n\u00b6wj1\u0000s(w\u0001x+b)\n(5.53)\nRearranging terms:\n\u00b6LCE\n\u00b6wj=\u0000\u0014y\ns(w\u0001x+b)\u00001\u0000y\n1\u0000s(w\u0001x+b)\u0015\u00b6\n\u00b6wjs(w\u0001x+b)\n(5.54)\nAnd now plugging in the derivative of the sigmoid, and using the chain rule one\nmore time, we end up with Eq. 5.55:\n\u00b6LCE\n\u00b6wj=\u0000\u0014y\u0000s(w\u0001x+b)\ns(w\u0001x+b)[1\u0000s(w\u0001x+b)]\u0015\ns(w\u0001x+b)[1\u0000s(w\u0001x+b)]\u00b6(w\u0001x+b)\n\u00b6wj\n=\u0000\u0014y\u0000s(w\u0001x+b)\ns(w\u0001x+b)[1\u0000s(w\u0001x+b)]\u0015\ns(w\u0001x+b)[1\u0000s(w\u0001x+b)]xj\n=\u0000[y\u0000s(w\u0001x+b)]xj\n= [s(w\u0001x+b)\u0000y]xj (5.55)\n5.11 Summary\nThis chapter introduced the logistic regression model of classi\ufb01cation .\n\u2022 Logistic regression is a supervised machine learning classi\ufb01er that extracts\nreal-valued features from the input, multiplies each by a weight, sums them,\nand passes the sum through a sigmoid function to generate a probability. A\nthreshold is used to make a decision.\n\u2022 Logistic regression can be used with two classes (e.g., positive and negative\nsentiment) or with multiple classes ( multinomial logistic regression , for ex-\nample for n-ary text classi\ufb01cation, part-of-speech labeling, etc.).\n\u2022 Multinomial logistic regression uses the softmax function to compute proba-\nbilities.\n\u2022 The weights (vector wand bias b) are learned from a labeled training set via a\nloss function, such as the cross-entropy loss , that must be minimized.\n\u2022 Minimizing this loss function is a convex optimization problem, and iterative\nalgorithms like gradient descent are used to \ufb01nd the optimal weights.\n\u2022Regularization is used to avoid over\ufb01tting.\n\u2022 Logistic regression is also one of the most useful analytic tools, because of its\nability to transparently study the importance of individual features. 100 CHAPTER 5 \u2022 L OGISTIC REGRESSION\nBibliographical and Historical Notes\nLogistic regression was developed in the \ufb01eld of statistics, where it was used for\nthe analysis of binary data by the 1960s, and was particularly common in medicine\n(Cox, 1969). Starting in the late 1970s it became widely used in linguistics as one\nof the formal foundations of the study of linguistic variation (Sankoff and Labov,\n1979).\nNonetheless, logistic regression didn\u2019t become common in natural language pro-\ncessing until the 1990s, when it seems to have appeared simultaneously from two\ndirections. The \ufb01rst source was the neighboring \ufb01elds of information retrieval and\nspeech processing, both of which had made use of regression, and both of which\nlent many other statistical techniques to NLP. Indeed a very early use of logistic\nregression for document routing was one of the \ufb01rst NLP applications to use (LSI)\nembeddings as word representations (Sch \u00a8utze et al., 1995).\nAt the same time in the early 1990s logistic regression was developed and ap-\nplied to NLP at IBM Research under the name maximum entropy modeling ormaximum\nentropy\nmaxent (Berger et al., 1996), seemingly independent of the statistical literature. Un-\nder that name it was applied to language modeling (Rosenfeld, 1996), part-of-speech\ntagging (Ratnaparkhi, 1996), parsing (Ratnaparkhi, 1997), coreference resolution\n(Kehler, 1997b), and text classi\ufb01cation (Nigam et al., 1999).\nMore on classi\ufb01cation can be found in machine learning textbooks (Hastie et al.\n2001, Witten and Frank 2005, Bishop 2006, Murphy 2012).\nExercises CHAPTER\n6Vector Semantics and\nEmbeddings\n\u8343\u8005\u6240\u4ee5\u5728\u9c7c\uff0c\u5f97\u9c7c\u800c\u5fd8\u8343Nets are for \ufb01sh;\nOnce you get the \ufb01sh, you can forget the net.\n\u8a00\u8005\u6240\u4ee5\u5728\u610f\uff0c\u5f97\u610f\u800c\u5fd8\u8a00Words are for meaning;\nOnce you get the meaning, you can forget the words\n\u5e84\u5b50(Zhuangzi), Chapter 26\nThe asphalt that Los Angeles is famous for occurs mainly on its freeways. But\nin the middle of the city is another patch of asphalt, the La Brea tar pits, and this\nasphalt preserves millions of fossil bones from the last of the Ice Ages of the Pleis-\ntocene Epoch. One of these fossils is the Smilodon , or saber-toothed tiger, instantly\nrecognizable by its long canines. Five million years ago or so, a completely different\nsaber-tooth tiger called Thylacosmilus lived\nin Argentina and other parts of South Amer-\nica. Thylacosmilus was a marsupial whereas\nSmilodon was a placental mammal, but Thy-\nlacosmilus had the same long upper canines\nand, like Smilodon, had a protective bone\n\ufb02ange on the lower jaw. The similarity of\nthese two mammals is one of many examples\nof parallel or convergent evolution, in which particular contexts or environments\nlead to the evolution of very similar structures in different species (Gould, 1980).\nThe role of context is also important in the similarity of a less biological kind\nof organism: the word. Words that occur in similar contexts tend to have similar\nmeanings . This link between similarity in how words are distributed and similarity\nin what they mean is called the distributional hypothesis . The hypothesis wasdistributional\nhypothesis\n\ufb01rst formulated in the 1950s by linguists like Joos (1950), Harris (1954), and Firth\n(1957), who noticed that words which are synonyms (like oculist andeye-doctor )\ntended to occur in the same environment (e.g., near words like eyeorexamined )\nwith the amount of meaning difference between two words \u201ccorresponding roughly\nto the amount of difference in their environments\u201d (Harris, 1954, p. 157).\nIn this chapter we introduce vector semantics , which instantiates this linguisticvector\nsemantics\nhypothesis by learning representations of the meaning of words, called embeddings , embeddings\ndirectly from their distributions in texts. These representations are used in every nat-\nural language processing application that makes use of meaning, and the static em-\nbeddings we introduce here underlie the more powerful dynamic or contextualized\nembeddings likeBERT that we will see in Chapter 11.\nThese word representations are also the \ufb01rst example in this book of repre-\nsentation learning , automatically learning useful representations of the input text.representation\nlearning\nFinding such self-supervised ways to learn representations of the input, instead of\ncreating representations by hand via feature engineering , is an important focus of\nNLP research (Bengio et al., 2013). 102 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\n6.1 Lexical Semantics\nLet\u2019s begin by introducing some basic principles of word meaning. How should\nwe represent the meaning of a word? In the n-gram models of Chapter 3, and in\nclassical NLP applications, our only representation of a word is as a string of letters,\nor an index in a vocabulary list. This representation is not that different from a\ntradition in philosophy, perhaps you\u2019ve seen it in introductory logic classes, in which\nthe meaning of words is represented by just spelling the word with small capital\nletters; representing the meaning of \u201cdog\u201d as DOG, and \u201ccat\u201d as CAT, or by using an\napostrophe ( DOG \u2019).\nRepresenting the meaning of a word by capitalizing it is a pretty unsatisfactory\nmodel. You might have seen a version of a joke due originally to semanticist Barbara\nPartee (Carlson, 1977):\nQ: What\u2019s the meaning of life?\nA:LIFE \u2019\nSurely we can do better than this! After all, we\u2019ll want a model of word meaning\nto do all sorts of things for us. It should tell us that some words have similar mean-\nings ( catis similar to dog), others are antonyms ( cold is the opposite of hot), some\nhave positive connotations ( happy ) while others have negative connotations ( sad). It\nshould represent the fact that the meanings of buy,sell, and payoffer differing per-\nspectives on the same underlying purchasing event. (If I buy something from you,\nyou\u2019ve probably sold it to me, and I likely paid you.) More generally, a model of\nword meaning should allow us to draw inferences to address meaning-related tasks\nlike question-answering or dialogue.\nIn this section we summarize some of these desiderata, drawing on results in the\nlinguistic study of word meaning, which is called lexical semantics ; we\u2019ll return tolexical\nsemantics\nand expand on this list in Appendix G and Chapter 21.\nLemmas and Senses Let\u2019s start by looking at how one word (we\u2019ll choose mouse )\nmight be de\ufb01ned in a dictionary (simpli\ufb01ed from the online dictionary WordNet):\nmouse (N)\n1. any of numerous small rodents...\n2. a hand-operated device that controls a cursor...\nHere the form mouse is the lemma , also called the citation form . The form lemma\ncitation form mouse would also be the lemma for the word mice ; dictionaries don\u2019t have separate\nde\ufb01nitions for in\ufb02ected forms like mice . Similarly sing is the lemma for sing,sang ,\nsung . In many languages the in\ufb01nitive form is used as the lemma for the verb, so\nSpanish dormir \u201cto sleep\u201d is the lemma for duermes \u201cyou sleep\u201d. The speci\ufb01c forms\nsung orcarpets orsing orduermes are called wordforms . wordform\nAs the example above shows, each lemma can have multiple meanings; the\nlemma mouse can refer to the rodent or the cursor control device. We call each\nof these aspects of the meaning of mouse aword sense . The fact that lemmas can\nbepolysemous (have multiple senses) can make interpretation dif\ufb01cult (is someone\nwho types \u201cmouse info\u201d into a search engine looking for a pet or a tool?). Chap-\nter 11 and Appendix G will discuss the problem of polysemy, and introduce word\nsense disambiguation , the task of determining which sense of a word is being used\nin a particular context.\nSynonymy One important component of word meaning is the relationship be-\ntween word senses. For example when one word has a sense whose meaning is 6.1 \u2022 L EXICAL SEMANTICS 103\nidentical to a sense of another word, or nearly identical, we say the two senses of\nthose two words are synonyms . Synonyms include such pairs as synonym\ncouch/sofa vomit/throw up \ufb01lbert/hazelnut car/automobile\nA more formal de\ufb01nition of synonymy (between words rather than senses) is that\ntwo words are synonymous if they are substitutable for one another in any sentence\nwithout changing the truth conditions of the sentence, the situations in which the\nsentence would be true.\nWhile substitutions between some pairs of words like car/automobile orwa-\nter/H2Oare truth preserving, the words are still not identical in meaning. Indeed,\nprobably no two words are absolutely identical in meaning. One of the fundamental\ntenets of semantics, called the principle of contrast (Girard 1718, Br \u00b4eal 1897, Clarkprinciple of\ncontrast\n1987), states that a difference in linguistic form is always associated with some dif-\nference in meaning. For example, the word H2Ois used in scienti\ufb01c contexts and\nwould be inappropriate in a hiking guide\u2014 water would be more appropriate\u2014 and\nthis genre difference is part of the meaning of the word. In practice, the word syn-\nonym is therefore used to describe a relationship of approximate or rough synonymy.\nWord Similarity While words don\u2019t have many synonyms, most words do have\nlots of similar words. Catis not a synonym of dog, but cats anddogs are certainly\nsimilar words. In moving from synonymy to similarity, it will be useful to shift from\ntalking about relations between word senses (like synonymy) to relations between\nwords (like similarity). Dealing with words avoids having to commit to a particular\nrepresentation of word senses, which will turn out to simplify our task.\nThe notion of word similarity is very useful in larger semantic tasks. Knowing similarity\nhow similar two words are can help in computing how similar the meaning of two\nphrases or sentences are, a very important component of tasks like question answer-\ning, paraphrasing, and summarization. One way of getting values for word similarity\nis to ask humans to judge how similar one word is to another. A number of datasets\nhave resulted from such experiments. For example the SimLex-999 dataset (Hill\net al., 2015) gives values on a scale from 0 to 10, like the examples below, which\nrange from near-synonyms ( vanish ,disappear ) to pairs that scarcely seem to have\nanything in common ( hole,agreement ):\nvanish disappear 9.8\nbelief impression 5.95\nmuscle bone 3.65\nmodest \ufb02exible 0.98\nhole agreement 0.3\nWord Relatedness The meaning of two words can be related in ways other than\nsimilarity. One such class of connections is called word relatedness (Budanitsky relatedness\nand Hirst, 2006), also traditionally called word association in psychology. association\nConsider the meanings of the words coffee andcup. Coffee is not similar to cup;\nthey share practically no features (coffee is a plant or a beverage, while a cup is a\nmanufactured object with a particular shape). But coffee and cup are clearly related;\nthey are associated by co-participating in an everyday event (the event of drinking\ncoffee out of a cup). Similarly scalpel andsurgeon are not similar but are related\neventively (a surgeon tends to make use of a scalpel).\nOne common kind of relatedness between words is if they belong to the same\nsemantic \ufb01eld . A semantic \ufb01eld is a set of words which cover a particular semantic semantic \ufb01eld\ndomain and bear structured relations with each other. For example, words might be 104 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\nrelated by being in the semantic \ufb01eld of hospitals ( surgeon ,scalpel ,nurse ,anes-\nthetic ,hospital ), restaurants ( waiter ,menu ,plate ,food,chef), or houses ( door ,roof,\nkitchen ,family ,bed). Semantic \ufb01elds are also related to topic models , like Latent topic models\nDirichlet Allocation ,LDA , which apply unsupervised learning on large sets of texts\nto induce sets of associated words from text. Semantic \ufb01elds and topic models are\nvery useful tools for discovering topical structure in documents.\nIn Appendix G we\u2019ll introduce more relations between senses like hypernymy\norIS-A ,antonymy (opposites) and meronymy (part-whole relations).\nSemantic Frames and Roles Closely related to semantic \ufb01elds is the idea of a\nsemantic frame . A semantic frame is a set of words that denote perspectives or semantic frame\nparticipants in a particular type of event. A commercial transaction, for example,\nis a kind of event in which one entity trades money to another entity in return for\nsome good or service, after which the good changes hands or perhaps the service is\nperformed. This event can be encoded lexically by using verbs like buy(the event\nfrom the perspective of the buyer), sell(from the perspective of the seller), pay\n(focusing on the monetary aspect), or nouns like buyer . Frames have semantic roles\n(like buyer ,seller ,goods ,money ), and words in a sentence can take on these roles.\nKnowing that buyandsellhave this relation makes it possible for a system to\nknow that a sentence like Sam bought the book from Ling could be paraphrased as\nLing sold the book to Sam , and that Sam has the role of the buyer in the frame and\nLing the seller . Being able to recognize such paraphrases is important for question\nanswering, and can help in shifting perspective for machine translation.\nConnotation Finally, words have affective meanings orconnotations . The word connotations\nconnotation has different meanings in different \ufb01elds, but here we use it to mean the\naspects of a word\u2019s meaning that are related to a writer or reader\u2019s emotions, senti-\nment, opinions, or evaluations. For example some words have positive connotations\n(wonderful ) while others have negative connotations ( dreary ). Even words whose\nmeanings are similar in other ways can vary in connotation; consider the difference\nin connotations between fake,knockoff ,forgery , on the one hand, and copy ,replica ,\nreproduction on the other, or innocent (positive connotation) and naive (negative\nconnotation). Some words describe positive evaluation ( great ,love) and others neg-\native evaluation ( terrible ,hate). Positive or negative evaluation language is called\nsentiment , as we saw in Chapter 4, and word sentiment plays a role in important sentiment\ntasks like sentiment analysis, stance detection, and applications of NLP to the lan-\nguage of politics and consumer reviews.\nEarly work on affective meaning (Osgood et al., 1957) found that words varied\nalong three important dimensions of affective meaning:\nvalence: the pleasantness of the stimulus\narousal: the intensity of emotion provoked by the stimulus\ndominance: the degree of control exerted by the stimulus\nThus words like happy orsatis\ufb01ed are high on valence, while unhappy oran-\nnoyed are low on valence. Excited is high on arousal, while calm is low on arousal.\nControlling is high on dominance, while awed orin\ufb02uenced are low on dominance.\nEach word is thus represented by three numbers, corresponding to its value on each\nof the three dimensions: 6.2 \u2022 V ECTOR SEMANTICS 105\nValence Arousal Dominance\ncourageous 8.05 5.5 7.38\nmusic 7.67 5.57 6.5\nheartbreak 2.45 5.65 3.58\ncub 6.71 3.95 4.24\nOsgood et al. (1957) noticed that in using these 3 numbers to represent the\nmeaning of a word, the model was representing each word as a point in a three-\ndimensional space, a vector whose three dimensions corresponded to the word\u2019s\nrating on the three scales. This revolutionary idea that word meaning could be rep-\nresented as a point in space (e.g., that part of the meaning of heartbreak can be\nrepresented as the point [2:45;5:65;3:58]) was the \ufb01rst expression of the vector se-\nmantics models that we introduce next.\n6.2 Vector Semantics\nVector semantics is the standard way to represent word meaning in NLP, helpingvector\nsemantics\nus model many of the aspects of word meaning we saw in the previous section. The\nroots of the model lie in the 1950s when two big ideas converged: Osgood\u2019s 1957\nidea mentioned above to use a point in three-dimensional space to represent the\nconnotation of a word, and the proposal by linguists like Joos (1950), Harris (1954),\nand Firth (1957) to de\ufb01ne the meaning of a word by its distribution in language\nuse, meaning its neighboring words or grammatical environments. Their idea was\nthat two words that occur in very similar distributions (whose neighboring words are\nsimilar) have similar meanings.\nFor example, suppose you didn\u2019t know the meaning of the word ongchoi (a re-\ncent borrowing from Cantonese) but you see it in the following contexts:\n(6.1) Ongchoi is delicious sauteed with garlic.\n(6.2) Ongchoi is superb over rice.\n(6.3) ...ongchoi leaves with salty sauces...\nAnd suppose that you had seen many of these context words in other contexts:\n(6.4) ...spinach sauteed with garlic over rice...\n(6.5) ...chard stems and leaves are delicious...\n(6.6) ...collard greens and other salty leafy greens\nThe fact that ongchoi occurs with words like riceandgarlic anddelicious and\nsalty , as do words like spinach ,chard , and collard greens might suggest that ongchoi\nis a leafy green similar to these other leafy greens.1We can do the same thing\ncomputationally by just counting words in the context of ongchoi .\nThe idea of vector semantics is to represent a word as a point in a multidimen-\nsional semantic space that is derived (in ways we\u2019ll see) from the distributions of\nword neighbors. Vectors for representing words are called embeddings (although embeddings\nthe term is sometimes more strictly applied only to dense vectors like word2vec\n(Section 6.8), rather than sparse tf-idf or PPMI vectors (Section 6.3-Section 6.6)).\nThe word \u201cembedding\u201d derives from its mathematical sense as a mapping from one\nspace or structure to another, although the meaning has shifted; see the end of the\nchapter.\n1It\u2019s in fact Ipomoea aquatica , a relative of morning glory sometimes called water spinach in English. 106 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\ngoodnicebadworstnot good\nwonderfulamazingterri\ufb01cdislikeworsevery goodincredibly goodfantasticincredibly badnowyouithatwithbyto\u2019sareisathan\nFigure 6.1 A two-dimensional (t-SNE) projection of embeddings for some words and\nphrases, showing that words with similar meanings are nearby in space. The original 60-\ndimensional embeddings were trained for sentiment analysis. Simpli\ufb01ed from Li et al. (2015)\nwith colors added for explanation.\nFig. 6.1 shows a visualization of embeddings learned for sentiment analysis,\nshowing the location of selected words projected down from 60-dimensional space\ninto a two dimensional space. Notice the distinct regions containing positive words,\nnegative words, and neutral function words.\nThe \ufb01ne-grained model of word similarity of vector semantics offers enormous\npower to NLP applications. NLP applications like the sentiment classi\ufb01ers of Chap-\nter 4 or Chapter 5 depend on the same words appearing in the training and test sets.\nBut by representing words as embeddings, a classi\ufb01er can assign sentiment as long\nas it sees some words with similar meanings . And as we\u2019ll see, vector semantic\nmodels can be learned automatically from text without supervision.\nIn this chapter we\u2019ll introduce the two most commonly used models. In the tf-idf\nmodel, an important baseline, the meaning of a word is de\ufb01ned by a simple function\nof the counts of nearby words. We will see that this method results in very long\nvectors that are sparse , i.e. mostly zeros (since most words simply never occur in\nthe context of others). We\u2019ll introduce the word2vec model family for construct-\ning short, dense vectors that have useful semantic properties. We\u2019ll also introduce\nthecosine , the standard way to use embeddings to compute semantic similarity , be-\ntween two words, two sentences, or two documents, an important tool in practical\napplications like question answering, summarization, or automatic essay grading.\n6.3 Words and Vectors\n\u201cThe most important attributes of a vector in 3-space are fLocation, Location, Location g\u201d\nRandall Munroe, https://xkcd.com/2358/\nVector or distributional models of meaning are generally based on a co-occurrence\nmatrix , a way of representing how often words co-occur. We\u2019ll look at two popular\nmatrices: the term-document matrix and the term-term matrix.\n6.3.1 Vectors and documents\nIn aterm-document matrix , each row represents a word in the vocabulary and eachterm-document\nmatrix\ncolumn represents a document from some collection of documents. Fig. 6.2 shows a\nsmall selection from a term-document matrix showing the occurrence of four words\nin four plays by Shakespeare. Each cell in this matrix represents the number of times 6.3 \u2022 W ORDS AND VECTORS 107\na particular word (de\ufb01ned by the row) occurs in a particular document (de\ufb01ned by\nthe column). Thus foolappeared 58 times in Twelfth Night .\nAs You Like It Twelfth Night Julius Caesar Henry V\nbattle 1 0 7 13\ngood 114 80 62 89\nfool 36 58 1 4\nwit 20 15 2 3\nFigure 6.2 The term-document matrix for four words in four Shakespeare plays. Each cell\ncontains the number of times the (row) word occurs in the (column) document.\nThe term-document matrix of Fig. 6.2 was \ufb01rst de\ufb01ned as part of the vector\nspace model of information retrieval (Salton, 1971). In this model, a document isvector space\nmodel\nrepresented as a count vector, a column in Fig. 6.3.\nTo review some basic linear algebra, a vector is, at heart, just a list or array of vector\nnumbers. So As You Like It is represented as the list [1,114,36,20] (the \ufb01rst column\nvector in Fig. 6.3) and Julius Caesar is represented as the list [7,62,1,2] (the third\ncolumn vector). A vector space is a collection of vectors, and is characterized by vector space\nitsdimension . Vectors in a 3-dimensional vector space have an element for each dimension\ndimension of the space. We will loosely refer to a vector in a 4-dimensional space\nas a 4-dimensional vector, with one element along each dimension. In the example\nin Fig. 6.3, we\u2019ve chosen to make the document vectors of dimension 4, just so they\n\ufb01t on the page; in real term-document matrices, the document vectors would have\ndimensionalityjVj, the vocabulary size.\nThe ordering of the numbers in a vector space indicates the different dimensions\non which documents vary. The \ufb01rst dimension for both these vectors corresponds to\nthe number of times the word battle occurs, and we can compare each dimension,\nnoting for example that the vectors for As You Like It andTwelfth Night have similar\nvalues (1 and 0, respectively) for the \ufb01rst dimension.\nAs You Like It Twelfth Night Julius Caesar Henry V\nbattle 1 0 7 13\ngood 114 80 62 89\nfool 36 58 1 4\nwit 20 15 2 3\nFigure 6.3 The term-document matrix for four words in four Shakespeare plays. The red\nboxes show that each document is represented as a column vector of length four.\nWe can think of the vector for a document as a point in jVj-dimensional space;\nthus the documents in Fig. 6.3 are points in 4-dimensional space. Since 4-dimensional\nspaces are hard to visualize, Fig. 6.4 shows a visualization in two dimensions; we\u2019ve\narbitrarily chosen the dimensions corresponding to the words battle andfool.\nTerm-document matrices were originally de\ufb01ned as a means of \ufb01nding similar\ndocuments for the task of document information retrieval . Two documents that are\nsimilar will tend to have similar words, and if two documents have similar words\ntheir column vectors will tend to be similar. The vectors for the comedies As You\nLike It [1,114,36,20] and Twelfth Night [0,80,58,15] look a lot more like each other\n(more fools and wit than battles) than they look like Julius Caesar [7,62,1,2] or\nHenry V [13,89,4,3]. This is clear with the raw numbers; in the \ufb01rst dimension\n(battle) the comedies have low numbers and the others have high numbers, and we\ncan see it visually in Fig. 6.4; we\u2019ll see very shortly how to quantify this intuition\nmore formally. 108 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\n51015202530510Henry V [4,13]As You Like It [36,1]Julius Caesar [1,7]battle foolTwelfth Night [58,0]1540\n354045505560\nFigure 6.4 A spatial visualization of the document vectors for the four Shakespeare play\ndocuments, showing just two of the dimensions, corresponding to the words battle andfool.\nThe comedies have high values for the fooldimension and low values for the battle dimension.\nA real term-document matrix, of course, wouldn\u2019t just have 4 rows and columns,\nlet alone 2. More generally, the term-document matrix has jVjrows (one for each\nword type in the vocabulary) and Dcolumns (one for each document in the collec-\ntion); as we\u2019ll see, vocabulary sizes are generally in the tens of thousands, and the\nnumber of documents can be enormous (think about all the pages on the web).\nInformation retrieval (IR) is the task of \ufb01nding the document dfrom the Dinformation\nretrieval\ndocuments in some collection that best matches a query q. For IR we\u2019ll therefore also\nrepresent a query by a vector, also of length jVj, and we\u2019ll need a way to compare\ntwo vectors to \ufb01nd how similar they are. (Doing IR will also require ef\ufb01cient ways\nto store and manipulate these vectors by making use of the convenient fact that these\nvectors are sparse, i.e., mostly zeros).\nLater in the chapter we\u2019ll introduce some of the components of this vector com-\nparison process: the tf-idf term weighting, and the cosine similarity metric.\n6.3.2 Words as vectors: document dimensions\nWe\u2019ve seen that documents can be represented as vectors in a vector space. But\nvector semantics can also be used to represent the meaning of words . We do this\nby associating each word with a word vector\u2014 a row vector rather than a column row vector\nvector, hence with different dimensions, as shown in Fig. 6.5. The four dimensions\nof the vector for fool, [36,58,1,4], correspond to the four Shakespeare plays. Word\ncounts in the same four dimensions are used to form the vectors for the other 3\nwords: wit, [20,15,2,3]; battle , [1,0,7,13]; and good [114,80,62,89].\nAs You Like It Twelfth Night Julius Caesar Henry V\nbattle 1 0 7 13\ngood 114 80 62 89\nfool 36 58 1 4\nwit 20 15 2 3\nFigure 6.5 The term-document matrix for four words in four Shakespeare plays. The red\nboxes show that each word is represented as a row vector of length four.\nFor documents, we saw that similar documents had similar vectors, because sim-\nilar documents tend to have similar words. This same principle applies to words:\nsimilar words have similar vectors because they tend to occur in similar documents.\nThe term-document matrix thus lets us represent the meaning of a word by the doc-\numents it tends to occur in. 6.3 \u2022 W ORDS AND VECTORS 109\n6.3.3 Words as vectors: word dimensions\nAn alternative to using the term-document matrix to represent words as vectors of\ndocument counts, is to use the term-term matrix , also called the word-word ma-\ntrixor the term-context matrix , in which the columns are labeled by words ratherword-word\nmatrix\nthan documents. This matrix is thus of dimensionality jVj\u0002jVjand each cell records\nthe number of times the row (target) word and the column (context) word co-occur\nin some context in some training corpus. The context could be the document, in\nwhich case the cell represents the number of times the two words appear in the same\ndocument. It is most common, however, to use smaller contexts, generally a win-\ndow around the word, for example of 4 words to the left and 4 words to the right,\nin which case the cell represents the number of times (in some training corpus) the\ncolumn word occurs in such a \u00064 word window around the row word. Here are four\nexamples of words in their windows:\nis traditionally followed by cherry pie, a traditional dessert\noften mixed, such as strawberry rhubarb pie. Apple pie\ncomputer peripherals and personal digital assistants. These devices usually\na computer. This includes information available on the internet\nIf we then take every occurrence of each word (say strawberry ) and count the\ncontext words around it, we get a word-word co-occurrence matrix. Fig. 6.6 shows a\nsimpli\ufb01ed subset of the word-word co-occurrence matrix for these four words com-\nputed from the Wikipedia corpus (Davies, 2015).\naardvark ... computer data result pie sugar ...\ncherry 0 ... 2 8 9 442 25 ...\nstrawberry 0 ... 0 0 1 60 19 ...\ndigital 0 ... 1670 1683 85 5 4 ...\ninformation 0 ... 3325 3982 378 5 13 ...\nFigure 6.6 Co-occurrence vectors for four words in the Wikipedia corpus, showing six of\nthe dimensions (hand-picked for pedagogical purposes). The vector for digital is outlined in\nred. Note that a real vector would have vastly more dimensions and thus be much sparser.\nNote in Fig. 6.6 that the two words cherry andstrawberry are more similar to\neach other (both pieandsugar tend to occur in their window) than they are to other\nwords like digital ; conversely, digital andinformation are more similar to each other\nthan, say, to strawberry . Fig. 6.7 shows a spatial visualization.\n100020003000400010002000digital [1683,1670]computer datainformation [3982,3325] 30004000\nFigure 6.7 A spatial visualization of word vectors for digital andinformation , showing just\ntwo of the dimensions, corresponding to the words data andcomputer .\nNote thatjVj, the dimensionality of the vector, is generally the size of the vo-\ncabulary, often between 10,000 and 50,000 words (using the most frequent words 110 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\nin the training corpus; keeping words after about the most frequent 50,000 or so is\ngenerally not helpful). Since most of these numbers are zero these are sparse vector\nrepresentations; there are ef\ufb01cient algorithms for storing and computing with sparse\nmatrices.\nNow that we have some intuitions, let\u2019s move on to examine the details of com-\nputing word similarity. Afterwards we\u2019ll discuss methods for weighting cells.\n6.4 Cosine for measuring similarity\nTo measure similarity between two target words vandw, we need a metric that\ntakes two vectors (of the same dimensionality, either both with words as dimensions,\nhence of lengthjVj, or both with documents as dimensions, of length jDj) and gives\na measure of their similarity. By far the most common similarity metric is the cosine\nof the angle between the vectors.\nThe cosine\u2014like most measures for vector similarity used in NLP\u2014is based on\nthedot product operator from linear algebra, also called the inner product : dot product\ninner product\ndot product (v;w) =v\u0001w=NX\ni=1viwi=v1w1+v2w2+:::+vNwN (6.7)\nThe dot product acts as a similarity metric because it will tend to be high just when\nthe two vectors have large values in the same dimensions. Alternatively, vectors that\nhave zeros in different dimensions\u2014orthogonal vectors\u2014will have a dot product of\n0, representing their strong dissimilarity.\nThis raw dot product, however, has a problem as a similarity metric: it favors\nlong vectors. The vector length is de\ufb01ned as vector length\njvj=vuutNX\ni=1v2\ni(6.8)\nThe dot product is higher if a vector is longer, with higher values in each dimension.\nMore frequent words have longer vectors, since they tend to co-occur with more\nwords and have higher co-occurrence values with each of them. The raw dot product\nthus will be higher for frequent words. But this is a problem; we\u2019d like a similarity\nmetric that tells us how similar two words are regardless of their frequency.\nWe modify the dot product to normalize for the vector length by dividing the\ndot product by the lengths of each of the two vectors. This normalized dot product\nturns out to be the same as the cosine of the angle between the two vectors, following\nfrom the de\ufb01nition of the dot product between two vectors aandb:\na\u0001b=jajjbjcosq\na\u0001b\njajjbj=cosq (6.9)\nThecosine similarity metric between two vectors vandwthus can be computed as: cosine 6.5 \u2022 TF-IDF: W EIGHING TERMS IN THE VECTOR 111\ncosine (v;w) =v\u0001w\njvjjwj=NX\ni=1viwi\nvuutNX\ni=1v2\nivuutNX\ni=1w2\ni(6.10)\nFor some applications we pre-normalize each vector, by dividing it by its length,\ncreating a unit vector of length 1. Thus we could compute a unit vector from aby unit vector\ndividing it byjaj. For unit vectors, the dot product is the same as the cosine.\nThe cosine value ranges from 1 for vectors pointing in the same direction, through\n0 for orthogonal vectors, to -1 for vectors pointing in opposite directions. But since\nraw frequency values are non-negative, the cosine for these vectors ranges from 0\u20131.\nLet\u2019s see how the cosine computes which of the words cherry ordigital is closer\nin meaning to information , just using raw counts from the following shortened table:\npie data computer\ncherry 442 8 2\ndigital 5 1683 1670\ninformation 5 3982 3325\ncos(cherry;information ) =442\u00035+8\u00033982+2\u00033325p\n4422+82+22p\n52+39822+33252=:018\ncos(digital;information ) =5\u00035+1683\u00033982+1670\u00033325p\n52+16832+16702p\n52+39822+33252=:996\nThe model decides that information is way closer to digital than it is to cherry , a\nresult that seems sensible. Fig. 6.8 shows a visualization.\n50010001500200025003000500digitalcherryinformationDimension 1: \u2018pie\u2019\nDimension 2: \u2018computer\u2019\nFigure 6.8 A (rough) graphical demonstration of cosine similarity, showing vectors for\nthree words ( cherry ,digital , and information ) in the two dimensional space de\ufb01ned by counts\nof the words computer andpienearby. The \ufb01gure doesn\u2019t show the cosine, but it highlights the\nangles; note that the angle between digital andinformation is smaller than the angle between\ncherry andinformation . When two vectors are more similar, the cosine is larger but the angle\nis smaller; the cosine has its maximum (1) when the angle between two vectors is smallest\n(0\u000e); the cosine of all other angles is less than 1.\n6.5 TF-IDF: Weighing terms in the vector\nThe co-occurrence matrices above represent each cell by frequencies, either of words\nwith documents (Fig. 6.5), or words with other words (Fig. 6.6). But raw frequency 112 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\nis not the best measure of association between words. Raw frequency is very skewed\nand not very discriminative. If we want to know what kinds of contexts are shared\nbycherry andstrawberry but not by digital andinformation , we\u2019re not going to get\ngood discrimination from words like the,it, orthey, which occur frequently with\nall sorts of words and aren\u2019t informative about any particular word. We saw this\nalso in Fig. 6.3 for the Shakespeare corpus; the dimension for the word good is not\nvery discriminative between plays; good is simply a frequent word and has roughly\nequivalent high frequencies in each of the plays.\nIt\u2019s a bit of a paradox. Words that occur nearby frequently (maybe pienearby\ncherry ) are more important than words that only appear once or twice. Yet words\nthat are too frequent\u2014ubiquitous, like theorgood \u2014 are unimportant. How can we\nbalance these two con\ufb02icting constraints?\nThere are two common solutions to this problem: in this section we\u2019ll describe\nthetf-idf weighting, usually used when the dimensions are documents. In the next\nsection we introduce the PPMI algorithm (usually used when the dimensions are\nwords).\nThetf-idf weighting (the \u2018-\u2019 here is a hyphen, not a minus sign) is the product\nof two terms, each term capturing one of these two intuitions:\nThe \ufb01rst is the term frequency (Luhn, 1957): the frequency of the word tin the term frequency\ndocument d. We can just use the raw count as the term frequency:\ntft;d=count (t;d) (6.11)\nMore commonly we squash the raw frequency a bit, by using the log 10of the fre-\nquency instead. The intuition is that a word appearing 100 times in a document\ndoesn\u2019t make that word 100 times more likely to be relevant to the meaning of the\ndocument. We also need to do something special with counts of 0, since we can\u2019t\ntake the log of 0.2\ntft;d=(\n1+log10count (t;d) if count (t;d)>0\n0 otherwise(6.12)\nIf we use log weighting, terms which occur 0 times in a document would have tf =0,\n1 times in a document tf =1+log10(1) =1+0=1, 10 times in a document tf =\n1+log10(10) =2, 100 times tf =1+log10(100) =3, 1000 times tf =4, and so on.\nThe second factor in tf-idf is used to give a higher weight to words that occur\nonly in a few documents. Terms that are limited to a few documents are useful\nfor discriminating those documents from the rest of the collection; terms that occur\nfrequently across the entire collection aren\u2019t as helpful. The document frequencydocument\nfrequency\ndftof a term tis the number of documents it occurs in. Document frequency is\nnot the same as the collection frequency of a term, which is the total number of\ntimes the word appears in the whole collection in any document. Consider in the\ncollection of Shakespeare\u2019s 37 plays the two words Romeo andaction . The words\nhave identical collection frequencies (they both occur 113 times in all the plays) but\nvery different document frequencies, since Romeo only occurs in a single play. If\nour goal is to \ufb01nd documents about the romantic tribulations of Romeo, the word\nRomeo should be highly weighted, but not action :\nCollection Frequency Document Frequency\nRomeo 113 1\naction 113 31\n2We can also use this alternative formulation, which we have used in earlier editions: tf t;d=\nlog10(count (t;d)+1) 6.5 \u2022 TF-IDF: W EIGHING TERMS IN THE VECTOR 113\nWe emphasize discriminative words like Romeo via the inverse document fre-\nquency oridfterm weight (Sparck Jones, 1972). The idf is de\ufb01ned using the frac- idf\ntionN=dft, where Nis the total number of documents in the collection, and df tis\nthe number of documents in which term toccurs. The fewer documents in which a\nterm occurs, the higher this weight. The lowest weight of 1 is assigned to terms that\noccur in all the documents. It\u2019s usually clear what counts as a document: in Shake-\nspeare we would use a play; when processing a collection of encyclopedia articles\nlike Wikipedia, the document is a Wikipedia page; in processing newspaper articles,\nthe document is a single article. Occasionally your corpus might not have appropri-\nate document divisions and you might need to break up the corpus into documents\nyourself for the purposes of computing idf.\nBecause of the large number of documents in many collections, this measure\ntoo is usually squashed with a log function. The resulting de\ufb01nition for inverse\ndocument frequency (idf) is thus\nidft=log10\u0012N\ndft\u0013\n(6.13)\nHere are some idf values for some words in the Shakespeare corpus, (along with\nthe document frequency df values on which they are based) ranging from extremely\ninformative words which occur in only one play like Romeo , to those that occur in a\nfew like salad orFalstaff , to those which are very common like foolor so common\nas to be completely non-discriminative since they occur in all 37 plays like good or\nsweet .3\nWord df idf\nRomeo 1 1.57\nsalad 2 1.27\nFalstaff 4 0.967\nforest 12 0.489\nbattle 21 0.246\nwit 34 0.037\nfool 36 0.012\ngood 37 0\nsweet 37 0\nThe tf-idf weighted value wt;dfor word tin document dthus combines term tf-idf\nfrequency tf t;d(de\ufb01ned either by Eq. 6.11 or by Eq. 6.12) with idf from Eq. 6.13:\nwt;d=tft;d\u0002idft (6.14)\nFig. 6.9 applies tf-idf weighting to the Shakespeare term-document matrix in Fig. 6.2,\nusing the tf equation Eq. 6.12. Note that the tf-idf values for the dimension corre-\nsponding to the word good have now all become 0; since this word appears in every\ndocument, the tf-idf weighting leads it to be ignored. Similarly, the word fool, which\nappears in 36 out of the 37 plays, has a much lower weight.\nThe tf-idf weighting is the way for weighting co-occurrence matrices in infor-\nmation retrieval, but also plays a role in many other aspects of natural language\nprocessing. It\u2019s also a great baseline, the simple thing to try \ufb01rst. We\u2019ll look at other\nweightings like PPMI (Positive Pointwise Mutual Information) in Section 6.6.\n3Sweet was one of Shakespeare\u2019s favorite adjectives, a fact probably related to the increased use of\nsugar in European recipes around the turn of the 16th century (Jurafsky, 2014, p. 175). 114 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\nAs You Like It Twelfth Night Julius Caesar Henry V\nbattle 0.246 0 0.454 0.520\ngood 0 0 0 0\nfool 0.030 0.033 0.0012 0.0019\nwit 0.085 0.081 0.048 0.054\nFigure 6.9 A portion of the tf-idf weighted term-document matrix for four words in Shake-\nspeare plays, showing a selection of 4 plays, using counts from Fig. 6.2. For example the\n0:085 value for witinAs You Like It is the product of tf =1+log10(20) =2:301 and idf =:037.\nNote that the idf weighting has eliminated the importance of the ubiquitous word good and\nvastly reduced the impact of the almost-ubiquitous word fool.\n6.6 Pointwise Mutual Information (PMI)\nAn alternative weighting function to tf-idf, PPMI (positive pointwise mutual infor-\nmation), is used for term-term-matrices, when the vector dimensions correspond to\nwords rather than documents. PPMI draws on the intuition that the best way to weigh\nthe association between two words is to ask how much more the two words co-occur\nin our corpus than we would have a priori expected them to appear by chance.\nPointwise mutual information (Fano, 1961)4is one of the most important con-pointwise\nmutual\ninformationcepts in NLP. It is a measure of how often two events xandyoccur, compared with\nwhat we would expect if they were independent:\nI(x;y) =log2P(x;y)\nP(x)P(y)(6.16)\nThe pointwise mutual information between a target word wand a context word\nc(Church and Hanks 1989, Church and Hanks 1990) is then de\ufb01ned as:\nPMI(w;c) =log2P(w;c)\nP(w)P(c)(6.17)\nThe numerator tells us how often we observed the two words together (assuming\nwe compute probability by using the MLE). The denominator tells us how often\nwe would expect the two words to co-occur assuming they each occurred indepen-\ndently; recall that the probability of two independent events both occurring is just\nthe product of the probabilities of the two events. Thus, the ratio gives us an esti-\nmate of how much more the two words co-occur than we expect by chance. PMI is\na useful tool whenever we need to \ufb01nd words that are strongly associated.\nPMI values range from negative to positive in\ufb01nity. But negative PMI values\n(which imply things are co-occurring less often than we would expect by chance)\ntend to be unreliable unless our corpora are enormous. To distinguish whether\ntwo words whose individual probability is each 10\u00006occur together less often than\nchance, we would need to be certain that the probability of the two occurring to-\ngether is signi\ufb01cantly less than 10\u000012, and this kind of granularity would require an\nenormous corpus. Furthermore it\u2019s not clear whether it\u2019s even possible to evaluate\nsuch scores of \u2018unrelatedness\u2019 with human judgments. For this reason it is more\n4PMI is based on the mutual information between two random variables XandY, de\ufb01ned as:\nI(X;Y) =X\nxX\nyP(x;y)log2P(x;y)\nP(x)P(y)(6.15)\nIn a confusion of terminology, Fano used the phrase mutual information to refer to what we now call\npointwise mutual information and the phrase expectation of the mutual information for what we now call\nmutual information 6.6 \u2022 P OINTWISE MUTUAL INFORMATION (PMI) 115\ncommon to use Positive PMI (called PPMI ) which replaces all negative PMI values PPMI\nwith zero (Church and Hanks 1989, Dagan et al. 1993, Niwa and Nitta 1994)5:\nPPMI (w;c) =max(log2P(w;c)\nP(w)P(c);0) (6.18)\nMore formally, let\u2019s assume we have a co-occurrence matrix F with W rows (words)\nand C columns (contexts), where fi jgives the number of times word wioccurs with\ncontext cj. This can be turned into a PPMI matrix where PPMI i jgives the PPMI\nvalue of word wiwith context cj(which we can also express as PPMI( wi;cj) or\nPPMI( w=i;c=j)) as follows:\npi j=fi jPW\ni=1PC\nj=1fi j;pi\u0003=PC\nj=1fi jPW\ni=1PC\nj=1fi j;p\u0003j=PW\ni=1fi jPW\ni=1PC\nj=1fi j(6.19)\nPPMI i j=max(log2pi j\npi\u0003p\u0003j;0) (6.20)\nLet\u2019s see some PPMI calculations. We\u2019ll use Fig. 6.10, which repeats Fig. 6.6 plus\nall the count marginals, and let\u2019s pretend for ease of calculation that these are the\nonly words/contexts that matter.\ncomputer data result pie sugar count(w)\ncherry 2 8 9 442 25 486\nstrawberry 0 0 1 60 19 80\ndigital 1670 1683 85 5 4 3447\ninformation 3325 3982 378 5 13 7703\ncount(context) 4997 5673 473 512 61 11716\nFigure 6.10 Co-occurrence counts for four words in 5 contexts in the Wikipedia corpus,\ntogether with the marginals, pretending for the purpose of this calculation that no other word-\ns/contexts matter.\nThus for example we could compute PPMI(information,data), assuming we pre-\ntended that Fig. 6.6 encompassed all the relevant word contexts/dimensions, as fol-\nlows:\nP(w=information, c=data ) =3982\n11716=:3399\nP(w=information ) =7703\n11716=:6575\nP(c=data ) =5673\n11716=:4842\nPPMI (information,data ) = log2(:3399=(:6575\u0003:4842)) =:0944\nFig. 6.11 shows the joint probabilities computed from the counts in Fig. 6.10, and\nFig. 6.12 shows the PPMI values. Not surprisingly, cherry andstrawberry are highly\nassociated with both pieandsugar , and data is mildly associated with information .\nPMI has the problem of being biased toward infrequent events; very rare words\ntend to have very high PMI values. One way to reduce this bias toward low frequency\n5Positive PMI also cleanly solves the problem of what to do with zero counts, using 0 to replace the\n\u0000\u00a5from log (0). 116 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\np(w,context) p(w)\ncomputer data result pie sugar p(w)\ncherry 0.0002 0.0007 0.0008 0.0377 0.0021 0.0415\nstrawberry 0.0000 0.0000 0.0001 0.0051 0.0016 0.0068\ndigital 0.1425 0.1436 0.0073 0.0004 0.0003 0.2942\ninformation 0.2838 0.3399 0.0323 0.0004 0.0011 0.6575\np(context) 0.4265 0.4842 0.0404 0.0437 0.0052\nFigure 6.11 Replacing the counts in Fig. 6.6 with joint probabilities, showing the marginals\nin the right column and the bottom row.\ncomputer data result pie sugar\ncherry 0 0 0 4.38 3.30\nstrawberry 0 0 0 4.10 5.51\ndigital 0.18 0.01 0 0 0\ninformation 0.02 0.09 0.28 0 0\nFigure 6.12 The PPMI matrix showing the association between words and context words,\ncomputed from the counts in Fig. 6.11. Note that most of the 0 PPMI values are ones that had\na negative PMI; for example PMI( cherry,computer ) = -6.7, meaning that cherry andcomputer\nco-occur on Wikipedia less often than we would expect by chance, and with PPMI we replace\nnegative values by zero.\nevents is to slightly change the computation for P(c), using a different function Pa(c)\nthat raises the probability of the context word to the power of a:\nPPMI a(w;c) =max(log2P(w;c)\nP(w)Pa(c);0) (6.21)\nPa(c) =count (c)a\nP\nccount (c)a(6.22)\nLevy et al. (2015) found that a setting of a=0:75 improved performance of\nembeddings on a wide range of tasks (drawing on a similar weighting used for skip-\ngrams described below in Eq. 6.32). This works because raising the count to a=\n0:75 increases the probability assigned to rare contexts, and hence lowers their PMI\n(Pa(c)>P(c)when cis rare).\nAnother possible solution is Laplace smoothing: Before computing PMI, a small\nconstant k(values of 0.1-3 are common) is added to each of the counts, shrinking\n(discounting) all the non-zero values. The larger the k, the more the non-zero counts\nare discounted.\n6.7 Applications of the tf-idf or PPMI vector models\nIn summary, the vector semantics model we\u2019ve described so far represents a target\nword as a vector with dimensions corresponding either to the documents in a large\ncollection (the term-document matrix) or to the counts of words in some neighboring\nwindow (the term-term matrix). The values in each dimension are counts, weighted\nby tf-idf (for term-document matrices) or PPMI (for term-term matrices), and the\nvectors are sparse (since most values are zero).\nThe model computes the similarity between two words xandyby taking the\ncosine of their tf-idf or PPMI vectors; high cosine, high similarity. This entire model 6.8 \u2022 W ORD2VEC 117\nis sometimes referred to as the tf-idf model or the PPMI model, after the weighting\nfunction.\nThe tf-idf model of meaning is often used for document functions like deciding\nif two documents are similar. We represent a document by taking the vectors of\nall the words in the document, and computing the centroid of all those vectors. centroid\nThe centroid is the multidimensional version of the mean; the centroid of a set of\nvectors is a single vector that has the minimum sum of squared distances to each of\nthe vectors in the set. Given kword vectors w1;w2;:::;wk, the centroid document\nvector dis:document\nvector\nd=w1+w2+:::+wk\nk(6.23)\nGiven two documents, we can then compute their document vectors d1andd2, and\nestimate the similarity between the two documents by cos (d1;d2). Document sim-\nilarity is also useful for all sorts of applications; information retrieval, plagiarism\ndetection, news recommender systems, and even for digital humanities tasks like\ncomparing different versions of a text to see which are similar to each other.\nEither the PPMI model or the tf-idf model can be used to compute word simi-\nlarity, for tasks like \ufb01nding word paraphrases, tracking changes in word meaning, or\nautomatically discovering meanings of words in different corpora. For example, we\ncan \ufb01nd the 10 most similar words to any target word wby computing the cosines\nbetween wand each of the V\u00001 other words, sorting, and looking at the top 10.\n6.8 Word2vec\nIn the previous sections we saw how to represent a word as a sparse, long vector with\ndimensions corresponding to words in the vocabulary or documents in a collection.\nWe now introduce a more powerful word representation: embeddings , short dense\nvectors. Unlike the vectors we\u2019ve seen so far, embeddings are short , with number\nof dimensions dranging from 50-1000, rather than the much larger vocabulary size\njVjor number of documents Dwe\u2019ve seen. These ddimensions don\u2019t have a clear\ninterpretation. And the vectors are dense : instead of vector entries being sparse,\nmostly-zero counts or functions of counts, the values will be real-valued numbers\nthat can be negative.\nIt turns out that dense vectors work better in every NLP task than sparse vectors.\nWhile we don\u2019t completely understand all the reasons for this, we have some intu-\nitions. Representing words as 300-dimensional dense vectors requires our classi\ufb01ers\nto learn far fewer weights than if we represented words as 50,000-dimensional vec-\ntors, and the smaller parameter space possibly helps with generalization and avoid-\ning over\ufb01tting. Dense vectors may also do a better job of capturing synonymy.\nFor example, in a sparse vector representation, dimensions for synonyms like car\nandautomobile dimension are distinct and unrelated; sparse vectors may thus fail\nto capture the similarity between a word with caras a neighbor and a word with\nautomobile as a neighbor.\nIn this section we introduce one method for computing embeddings: skip-gram skip-gram\nwith negative sampling , sometimes called SGNS . The skip-gram algorithm is one SGNS\nof two algorithms in a software package called word2vec , and so sometimes the word2vec\nalgorithm is loosely referred to as word2vec (Mikolov et al. 2013a, Mikolov et al.\n2013b). The word2vec methods are fast, ef\ufb01cient to train, and easily available on- 118 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\nline with code and pretrained embeddings. Word2vec embeddings are static em-\nbeddings , meaning that the method learns one \ufb01xed embedding for each word in thestatic\nembeddings\nvocabulary. In Chapter 11 we\u2019ll introduce methods for learning dynamic contextual\nembeddings like the popular family of BERT representations, in which the vector\nfor each word is different in different contexts.\nThe intuition of word2vec is that instead of counting how often each word woc-\ncurs near, say, apricot , we\u2019ll instead train a classi\ufb01er on a binary prediction task: \u201cIs\nword wlikely to show up near apricot ?\u201d We don\u2019t actually care about this prediction\ntask; instead we\u2019ll take the learned classi\ufb01er weights as the word embeddings.\nThe revolutionary intuition here is that we can just use running text as implicitly\nsupervised training data for such a classi\ufb01er; a word cthat occurs near the target\nword apricot acts as gold \u2018correct answer\u2019 to the question \u201cIs word clikely to show\nup near apricot ?\u201d This method, often called self-supervision , avoids the need for self-supervision\nany sort of hand-labeled supervision signal. This idea was \ufb01rst proposed in the task\nof neural language modeling, when Bengio et al. (2003) and Collobert et al. (2011)\nshowed that a neural language model (a neural network that learned to predict the\nnext word from prior words) could just use the next word in running text as its\nsupervision signal, and could be used to learn an embedding representation for each\nword as part of doing this prediction task.\nWe\u2019ll see how to do neural networks in the next chapter, but word2vec is a\nmuch simpler model than the neural network language model, in two ways. First,\nword2vec simpli\ufb01es the task (making it binary classi\ufb01cation instead of word pre-\ndiction). Second, word2vec simpli\ufb01es the architecture (training a logistic regression\nclassi\ufb01er instead of a multi-layer neural network with hidden layers that demand\nmore sophisticated training algorithms). The intuition of skip-gram is:\n1. Treat the target word and a neighboring context word as positive examples.\n2. Randomly sample other words in the lexicon to get negative samples.\n3. Use logistic regression to train a classi\ufb01er to distinguish those two cases.\n4. Use the learned weights as the embeddings.\n6.8.1 The classi\ufb01er\nLet\u2019s start by thinking about the classi\ufb01cation task, and then turn to how to train.\nImagine a sentence like the following, with a target word apricot , and assume we\u2019re\nusing a window of \u00062 context words:\n... lemon, a [tablespoon of apricot jam, a] pinch ...\nc1 c2 w c3 c4\nOur goal is to train a classi\ufb01er such that, given a tuple (w;c)of a target word\nwpaired with a candidate context word c(for example ( apricot ,jam), or perhaps\n(apricot ,aardvark )) it will return the probability that cis a real context word (true\nforjam, false for aardvark ):\nP(+jw;c) (6.24)\nThe probability that word cis not a real context word for wis just 1 minus\nEq. 6.24:\nP(\u0000jw;c) =1\u0000P(+jw;c) (6.25) 6.8 \u2022 W ORD2VEC 119\nHow does the classi\ufb01er compute the probability P? The intuition of the skip-\ngram model is to base this probability on embedding similarity: a word is likely to\noccur near the target if its embedding vector is similar to the target embedding. To\ncompute similarity between these dense embeddings, we rely on the intuition that\ntwo vectors are similar if they have a high dot product (after all, cosine is just a\nnormalized dot product). In other words:\nSimilarity (w;c)\u0019c\u0001w (6.26)\nThe dot product c\u0001wis not a probability, it\u2019s just a number ranging from \u0000\u00a5to\u00a5\n(since the elements in word2vec embeddings can be negative, the dot product can be\nnegative). To turn the dot product into a probability, we\u2019ll use the logistic orsigmoid\nfunction s(x), the fundamental core of logistic regression:\ns(x) =1\n1+exp(\u0000x)(6.27)\nWe model the probability that word cis a real context word for target word was:\nP(+jw;c) = s(c\u0001w) =1\n1+exp(\u0000c\u0001w)(6.28)\nThe sigmoid function returns a number between 0 and 1, but to make it a probability\nwe\u2019ll also need the total probability of the two possible events ( cis a context word,\nandcisn\u2019t a context word) to sum to 1. We thus estimate the probability that word c\nis not a real context word for was:\nP(\u0000jw;c) = 1\u0000P(+jw;c)\n=s(\u0000c\u0001w) =1\n1+exp(c\u0001w)(6.29)\nEquation 6.28 gives us the probability for one word, but there are many context\nwords in the window. Skip-gram makes the simplifying assumption that all context\nwords are independent, allowing us to just multiply their probabilities:\nP(+jw;c1:L) =LY\ni=1s(ci\u0001w) (6.30)\nlogP(+jw;c1:L) =LX\ni=1logs(ci\u0001w) (6.31)\nIn summary, skip-gram trains a probabilistic classi\ufb01er that, given a test target word\nwand its context window of Lwords c1:L, assigns a probability based on how similar\nthis context window is to the target word. The probability is based on applying the\nlogistic (sigmoid) function to the dot product of the embeddings of the target word\nwith each context word. To compute this probability, we just need embeddings for\neach target word and context word in the vocabulary.\nFig. 6.13 shows the intuition of the parameters we\u2019ll need. Skip-gram actually\nstores two embeddings for each word, one for the word as a target, and one for the\nword considered as context. Thus the parameters we need to learn are two matrices\nWandC, each containing an embedding for every one of the jVjwords in the\nvocabulary V.6Let\u2019s now turn to learning these embeddings (which is the real goal\nof training this classi\ufb01er in the \ufb01rst place).\n6In principle the target matrix and the context matrix could use different vocabularies, but we\u2019ll simplify\nby assuming one shared vocabulary V. 120 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\n1WCaardvark\nzebrazebraaardvarkapricotapricot|V||V|+12V& =target wordscontext & noisewords\u2026\n\u20261..d\u2026\n\u2026\nFigure 6.13 The embeddings learned by the skipgram model. The algorithm stores two\nembeddings for each word, the target embedding (sometimes called the input embedding)\nand the context embedding (sometimes called the output embedding). The parameter qthat\nthe algorithm learns is thus a matrix of 2 jVjvectors, each of dimension d, formed by concate-\nnating two matrices, the target embeddings Wand the context+noise embeddings C.\n6.8.2 Learning skip-gram embeddings\nThe learning algorithm for skip-gram embeddings takes as input a corpus of text,\nand a chosen vocabulary size N. It begins by assigning a random embedding vector\nfor each of the N vocabulary words, and then proceeds to iteratively shift the em-\nbedding of each word wto be more like the embeddings of words that occur nearby\nin texts, and less like the embeddings of words that don\u2019t occur nearby. Let\u2019s start\nby considering a single piece of training data:\n... lemon, a [tablespoon of apricot jam, a] pinch ...\nc1 c2 w c3 c4\nThis example has a target word w(apricot), and 4 context words in the L=\u00062\nwindow, resulting in 4 positive training instances (on the left below):\npositive examples +\nw c pos\napricot tablespoon\napricot of\napricot jam\napricot anegative examples -\nw c neg w c neg\napricot aardvark apricot seven\napricot my apricot forever\napricot where apricot dear\napricot coaxial apricot if\nFor training a binary classi\ufb01er we also need negative examples. In fact skip-\ngram with negative sampling (SGNS) uses more negative examples than positive\nexamples (with the ratio between them set by a parameter k). So for each of these\n(w;cpos)training instances we\u2019ll create knegative samples, each consisting of the\ntarget wplus a \u2018noise word\u2019 cneg. A noise word is a random word from the lexicon,\nconstrained not to be the target word w. The right above shows the setting where\nk=2, so we\u2019ll have 2 negative examples in the negative training set \u0000for each\npositive example w;cpos.\nThe noise words are chosen according to their weighted unigram frequency\npa(w), where ais a weight. If we were sampling according to unweighted fre-\nquency p(w), it would mean that with unigram probability p(\u201cthe\u201d)we would choose\nthe word theas a noise word, with unigram probability p(\u201caardvark \u201d)we would 6.8 \u2022 W ORD2VEC 121\nchoose aardvark , and so on. But in practice it is common to set a=0:75, i.e. use\nthe weighting p3\n4(w):\nPa(w) =count (w)a\nP\nw0count (w0)a(6.32)\nSetting a=:75 gives better performance because it gives rare noise words slightly\nhigher probability: for rare words, Pa(w)>P(w). To illustrate this intuition, it\nmight help to work out the probabilities for an example with a=:75 and two events,\nP(a) =0:99 and P(b) =0:01:\nPa(a) =:99:75\n:99:75+:01:75=0:97\nPa(b) =:01:75\n:99:75+:01:75=0:03 (6.33)\nThus using a=:75 increases the probability of the rare event bfrom 0.01 to 0.03.\nGiven the set of positive and negative training instances, and an initial set of\nembeddings, the goal of the learning algorithm is to adjust those embeddings to\n\u2022 Maximize the similarity of the target word, context word pairs (w;cpos)drawn\nfrom the positive examples\n\u2022 Minimize the similarity of the (w;cneg)pairs from the negative examples.\nIf we consider one word/context pair (w;cpos)with its knoise words cneg1:::cnegk,\nwe can express these two goals as the following loss function Lto be minimized\n(hence the\u0000); here the \ufb01rst term expresses that we want the classi\ufb01er to assign the\nreal context word cposa high probability of being a neighbor, and the second term\nexpresses that we want to assign each of the noise words cnegia high probability of\nbeing a non-neighbor, all multiplied because we assume independence:\nL=\u0000log\"\nP(+jw;cpos)kY\ni=1P(\u0000jw;cnegi)#\n=\u0000\"\nlogP(+jw;cpos)+kX\ni=1logP(\u0000jw;cnegi)#\n=\u0000\"\nlogP(+jw;cpos)+kX\ni=1log\u0000\n1\u0000P(+jw;cnegi)\u0001#\n=\u0000\"\nlogs(cpos\u0001w)+kX\ni=1logs(\u0000cnegi\u0001w)#\n(6.34)\nThat is, we want to maximize the dot product of the word with the actual context\nwords, and minimize the dot products of the word with the knegative sampled non-\nneighbor words.\nWe minimize this loss function using stochastic gradient descent. Fig. 6.14\nshows the intuition of one step of learning.\nTo get the gradient, we need to take the derivative of Eq. 6.34 with respect to\nthe different embeddings. It turns out the derivatives are the following (we leave the 122 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\nWCmove apricot and jam closer,increasing cpos z waardvark\nmove apricot and matrix apartdecreasing cneg1 z w\u201c\u2026apricot jam\u2026\u201dw\nzebrazebraaardvarkjamapricot\ncposmatrixTolstoymove apricot and Tolstoy apartdecreasing cneg2 z w!cneg1cneg2k=2\nFigure 6.14 Intuition of one step of gradient descent. The skip-gram model tries to shift\nembeddings so the target embeddings (here for apricot ) are closer to (have a higher dot prod-\nuct with) context embeddings for nearby words (here jam) and further from (lower dot product\nwith) context embeddings for noise words that don\u2019t occur nearby (here Tolstoy andmatrix ).\nproof as an exercise at the end of the chapter):\n\u00b6L\n\u00b6cpos= [s(cpos\u0001w)\u00001]w (6.35)\n\u00b6L\n\u00b6cneg= [s(cneg\u0001w)]w (6.36)\n\u00b6L\n\u00b6w= [s(cpos\u0001w)\u00001]cpos+kX\ni=1[s(cnegi\u0001w)]cnegi(6.37)\nThe update equations going from time step ttot+1 in stochastic gradient descent\nare thus:\nct+1\npos=ct\npos\u0000h[s(ct\npos\u0001wt)\u00001]wt(6.38)\nct+1\nneg=ct\nneg\u0000h[s(ct\nneg\u0001wt)]wt(6.39)\nwt+1=wt\u0000h\"\n[s(ct\npos\u0001wt)\u00001]ct\npos+kX\ni=1[s(ct\nnegi\u0001wt)]ct\nnegi#\n(6.40)\nJust as in logistic regression, then, the learning algorithm starts with randomly ini-\ntialized WandCmatrices, and then walks through the training corpus using gradient\ndescent to move WandCso as to minimize the loss in Eq. 6.34 by making the up-\ndates in (Eq. 6.38)-(Eq. 6.40).\nRecall that the skip-gram model learns twoseparate embeddings for each word i:\nthetarget embedding wiand the context embedding ci, stored in two matrices, thetarget\nembeddingcontext\nembedding target matrix Wand the context matrix C. It\u2019s common to just add them together,\nrepresenting word iwith the vector wi+ci. Alternatively we can throw away the C\nmatrix and just represent each word iby the vector wi.\nAs with the simple count-based methods like tf-idf, the context window size L\naffects the performance of skip-gram embeddings, and experiments often tune the\nparameter Lon a devset. 6.9 \u2022 V ISUALIZING EMBEDDINGS 123\n6.8.3 Other kinds of static embeddings\nThere are many kinds of static embeddings. An extension of word2vec, fasttext fasttext\n(Bojanowski et al., 2017), addresses a problem with word2vec as we have presented\nit so far: it has no good way to deal with unknown words \u2014words that appear in\na test corpus but were unseen in the training corpus. A related problem is word\nsparsity, such as in languages with rich morphology, where some of the many forms\nfor each noun and verb may only occur rarely. Fasttext deals with these problems\nby using subword models, representing each word as itself plus a bag of constituent\nn-grams, with special boundary symbols <and>added to each word. For example,\nwith n=3 the word where would be represented by the sequence <where> plus the\ncharacter n-grams:\n<wh, whe, her, ere, re>\nThen a skipgram embedding is learned for each constituent n-gram, and the word\nwhere is represented by the sum of all of the embeddings of its constituent n-grams.\nUnknown words can then be presented only by the sum of the constituent n-grams.\nA fasttext open-source library, including pretrained embeddings for 157 languages,\nis available at https://fasttext.cc .\nAnother very widely used static embedding model is GloVe (Pennington et al.,\n2014), short for Global Vectors, because the model is based on capturing global\ncorpus statistics. GloVe is based on ratios of probabilities from the word-word co-\noccurrence matrix, combining the intuitions of count-based models like PPMI while\nalso capturing the linear structures used by methods like word2vec.\nIt turns out that dense embeddings like word2vec actually have an elegant math-\nematical relationship with sparse embeddings like PPMI, in which word2vec can\nbe seen as implicitly optimizing a function of a PPMI matrix (Levy and Goldberg,\n2014c).\n6.9 Visualizing Embeddings\n\u201cI see well in many dimensions as long as the dimensions are around two.\u201d\nThe late economist Martin Shubik\nVisualizing embeddings is an important goal in helping understand, apply, and\nimprove these models of word meaning. But how can we visualize a (for example)\n100-dimensional vector?\nRohde, Gonnerman, Plaut Modeling Word Meaning Using Lexical Co-Occurrence\nHEADHANDFACE\nDOGAMERICA\nCATEYEEUROPE\nFOOTCHINAFRANCE\nCHICAGOARM\nFINGER\nNOSELEGRUSSIA\nMOUSEAFRICA\nATLANTAEARSHOULDERASIA\nCOW\nBULLPUPPYLIONHAWAII\nMONTREALTOKYOTOEMOSCOW\nTOOTH\nNASHVILLEBRAZILWRIST\nKITTENANKLE\nTURTLE\nOYSTER\nFigure 8: Multidimensional scaling for three noun classes.WRIST\nANKLE\nSHOULDER\nARM\nLEG\nHAND\nFOOT\nHEAD\nNOSE\nFINGER\nTOE\nFACE\nEAR\nEYE\nTOOTH\nDOG\nCAT\nPUPPY\nKITTEN\nCOW\nMOUSE\nTURTLE\nOYSTER\nLION\nBULL\nCHICAGO\nATLANTA\nMONTREAL\nNASHVILLE\nTOKYOCHINA\nRUSSIA\nAFRICA\nASIA\nEUROPE\nAMERICA\nBRAZIL\nMOSCOW\nFRANCEHAWAIIFigure 9: Hierarchical clustering for three noun classes using distances based on vector correlations.\n20\nThe simplest way to visualize the meaning of a word\nwembedded in a space is to list the most similar words to\nwby sorting the vectors for all words in the vocabulary by\ntheir cosine with the vector for w. For example the 7 closest\nwords to frogusing a particular embeddings computed with\nthe GloVe algorithm are: frogs ,toad,litoria ,leptodactyli-\ndae,rana,lizard , and eleutherodactylus (Pennington et al.,\n2014).\nYet another visualization method is to use a clustering\nalgorithm to show a hierarchical representation of which\nwords are similar to others in the embedding space. The\nuncaptioned \ufb01gure on the left uses hierarchical clustering\nof some embedding vectors for nouns as a visualization 124 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\nmethod (Rohde et al., 2006).\nProbably the most common visualization method, how-\never, is to project the 100 dimensions of a word down into 2\ndimensions. Fig. 6.1 showed one such visualization, as does\nFig. 6.16, using a projection method called t-SNE (van der\nMaaten and Hinton, 2008).\n6.10 Semantic properties of embeddings\nIn this section we brie\ufb02y summarize some of the semantic properties of embeddings\nthat have been studied.\nDifferent types of similarity or association: One parameter of vector semantic\nmodels that is relevant to both sparse tf-idf vectors and dense word2vec vectors is\nthe size of the context window used to collect counts. This is generally between 1\nand 10 words on each side of the target word (for a total context of 2-20 words).\nThe choice depends on the goals of the representation. Shorter context windows\ntend to lead to representations that are a bit more syntactic, since the information is\ncoming from immediately nearby words. When the vectors are computed from short\ncontext windows, the most similar words to a target word wtend to be semantically\nsimilar words with the same parts of speech. When vectors are computed from long\ncontext windows, the highest cosine words to a target word wtend to be words that\nare topically related but not similar.\nFor example Levy and Goldberg (2014a) showed that using skip-gram with a\nwindow of\u00062, the most similar words to the word Hogwarts (from the Harry Potter\nseries) were names of other \ufb01ctional schools: Sunnydale (from Buffy the Vampire\nSlayer ) orEvernight (from a vampire series). With a window of \u00065, the most similar\nwords to Hogwarts were other words topically related to the Harry Potter series:\nDumbledore ,Malfoy , and half-blood .\nIt\u2019s also often useful to distinguish two kinds of similarity or association between\nwords (Sch \u00a8utze and Pedersen, 1993). Two words have \ufb01rst-order co-occurrence\ufb01rst-order\nco-occurrence\n(sometimes called syntagmatic association ) if they are typically nearby each other.\nThus wrote is a \ufb01rst-order associate of book orpoem . Two words have second-order\nco-occurrence (sometimes called paradigmatic association ) if they have similarsecond-order\nco-occurrence\nneighbors. Thus wrote is a second-order associate of words like said orremarked .\nAnalogy/Relational Similarity: Another semantic property of embeddings is their\nability to capture relational meanings. In an important early vector space model of\ncognition, Rumelhart and Abrahamson (1973) proposed the parallelogram modelparallelogram\nmodel\nfor solving simple analogy problems of the form a is to b as a* is to what? . In\nsuch problems, a system is given a problem like apple:tree::grape:? , i.e., apple is\nto tree as grape is to , and must \ufb01ll in the word vine. In the parallelogram\nmodel, illustrated in Fig. 6.15, the vector from the word apple to the word tree(=#   \u0014tree\u0000#       \u0014apple) is added to the vector for grape (#        \u0014grape); the nearest word to that point\nis returned.\nIn early work with sparse embeddings, scholars showed that sparse vector mod-\nels of meaning could solve such analogy problems (Turney and Littman, 2005),\nbut the parallelogram method received more modern attention because of its suc-\ncess with word2vec or GloVe vectors (Mikolov et al. 2013c, Levy and Goldberg\n2014b, Pennington et al. 2014). For example, the result of the expression#     \u0014king\u0000 6.10 \u2022 S EMANTIC PROPERTIES OF EMBEDDINGS 125\ntreeapplegrapevine\nFigure 6.15 The parallelogram model for analogy problems (Rumelhart and Abrahamson,\n1973): the location of#     \u0014vine can be found by subtracting#       \u0014apple from#   \u0014tree and adding#       \u0014grape.\n#     \u0014man+#            \u0014woman is a vector close to#         \u0014queen. Similarly,#      \u0014Paris\u0000#           \u0014France +#     \u0014Italy results\nin a vector that is close to#         \u0014Rome. The embedding model thus seems to be extract-\ning representations of relations like MALE -FEMALE , or CAPITAL -CITY -OF, or even\nCOMPARATIVE /SUPERLATIVE , as shown in Fig. 6.16 from GloVe.\n(a) (b)\nFigure 6.16 Relational properties of the GloVe vector space, shown by projecting vectors onto two dimen-\nsions. (a)#     \u0014king\u0000#     \u0014man+#            \u0014woman is close to#        \u0014queen. (b) offsets seem to capture comparative and superlative\nmorphology (Pennington et al., 2014).\nFor a a:b::a\u0003:b\u0003problem, meaning the algorithm is given vectors a,b, and\na\u0003and must \ufb01nd b\u0003, the parallelogram method is thus:\n\u02c6b\u0003=argmin\nxdistance (x;b\u0000a+a\u0003) (6.41)\nwith some distance function, such as Euclidean distance.\nThere are some caveats. For example, the closest value returned by the paral-\nlelogram algorithm in word2vec or GloVe embedding spaces is usually not in fact\nb* but one of the 3 input words or their morphological variants (i.e., cherry:red ::\npotato:x returns potato orpotatoes instead of brown ), so these must be explicitly\nexcluded. Furthermore while embedding spaces perform well if the task involves\nfrequent words, small distances, and certain relations (like relating countries with\ntheir capitals or verbs/nouns with their in\ufb02ected forms), the parallelogram method\nwith embeddings doesn\u2019t work as well for other relations (Linzen 2016, Gladkova\net al. 2016, Schluter 2018, Ethayarajh et al. 2019a), and indeed Peterson et al. (2020)\nargue that the parallelogram method is in general too simple to model the human\ncognitive process of forming analogies of this kind. 126 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\n6.10.1 Embeddings and Historical Semantics\nEmbeddings can also be a useful tool for studying how meaning changes over time,\nby computing multiple embedding spaces, each from texts written in a particular\ntime period. For example Fig. 6.17 shows a visualization of changes in meaning in\nEnglish words over the last two centuries, computed by building separate embed-\nding spaces for each decade from historical corpora like Google n-grams (Lin et al.,\n2012b) and the Corpus of Historical American English (Davies, 2012).\nCHAPTER 5. DYNAMIC SOCIAL REPRESENTATIONS OF WORD MEANING 79\nFigure 5.1: Two-dimensional visualization of semantic change in English using SGNS\nvectors (see Section 5.8 for the visualization algorithm). A,T h ew o r d gay shifted\nfrom meaning \u201ccheerful\u201d or \u201cfrolicsome\u201d to referring to homosexuality. A,I nt h ee a r l y\n20th century broadcast referred to \u201ccasting out seeds\u201d; with the rise of television and\nradio its meaning shifted to \u201ctransmitting signals\u201d. C,Awful underwent a process of\npejoration, as it shifted from meaning \u201cfull of awe\u201d to meaning \u201cterrible or appalling\u201d\n[212].\nthat adverbials (e.g., actually )h a v eag e n e r a lt e n d e n c yt ou n d e r g os u b j e c t i \ufb01 c a t i o n\nwhere they shift from objective statements about the world (e.g., \u201cSorry, the car is\nactually broken\u201d) to subjective statements (e.g., \u201cI can\u2019t believe he actually did that\u201d,\nindicating surprise/disbelief).\n5.2.2 Computational linguistic studies\nThere are also a number of recent works analyzing semantic change using computational\nmethods. [ 200] use latent semantic analysis to analyze how word meanings broaden\nand narrow over time. [ 113]u s er a wc o - o c c u r r e n c ev e c t o r st op e r f o r man u m b e ro f\nhistorical case-studies on semantic change, and [ 252] perform a similar set of small-\nscale case-studies using temporal topic models. [ 87]c o n s t r u c tp o i n t - w i s em u t u a l\ninformation-based embeddings and found that semantic changes uncovered by their\nmethod had reasonable agreement with human judgments. [ 129]a n d[ 119]u s e\u201c n e u r a l \u201d\nword-embedding methods to detect linguistic change points. Finally, [ 257]a n a l y z e\nhistorical co-occurrences to test whether synonyms tend to change in similar ways.\nFigure 6.17 A t-SNE visualization of the semantic change of 3 words in English using\nword2vec vectors. The modern sense of each word, and the grey context words, are com-\nputed from the most recent (modern) time-point embedding space. Earlier points are com-\nputed from earlier historical embedding spaces. The visualizations show the changes in the\nword gayfrom meanings related to \u201ccheerful\u201d or \u201cfrolicsome\u201d to referring to homosexuality,\nthe development of the modern \u201ctransmission\u201d sense of broadcast from its original sense of\nsowing seeds, and the pejoration of the word awful as it shifted from meaning \u201cfull of awe\u201d\nto meaning \u201cterrible or appalling\u201d (Hamilton et al., 2016b).\n6.11 Bias and Embeddings\nIn addition to their ability to learn word meaning from text, embeddings, alas,\nalso reproduce the implicit biases and stereotypes that were latent in the text. As\nthe prior section just showed, embeddings can roughly model relational similar-\nity: \u2018queen\u2019 as the closest word to \u2018king\u2019 - \u2018man\u2019 + \u2018woman\u2019 implies the analogy\nman:woman::king:queen . But these same embedding analogies also exhibit gender\nstereotypes. For example Bolukbasi et al. (2016) \ufb01nd that the closest occupation\nto \u2018computer programmer\u2019 - \u2018man\u2019 + \u2018woman\u2019 in word2vec embeddings trained on\nnews text is \u2018homemaker\u2019, and that the embeddings similarly suggest the analogy\n\u2018father\u2019 is to \u2018doctor\u2019 as \u2018mother\u2019 is to \u2018nurse\u2019. This could result in what Crawford\n(2017) and Blodgett et al. (2020) call an allocational harm , when a system allo-allocational\nharm\ncates resources (jobs or credit) unfairly to different groups. For example algorithms\nthat use embeddings as part of a search for hiring potential programmers or doctors\nmight thus incorrectly downweight documents with women\u2019s names.\nIt turns out that embeddings don\u2019t just re\ufb02ect the statistics of their input, but also\namplify bias; gendered terms become more gendered in embedding space than theybias\nampli\ufb01cation\nwere in the input text statistics (Zhao et al. 2017, Ethayarajh et al. 2019b, Jia et al.\n2020), and biases are more exaggerated than in actual labor employment statistics\n(Garg et al., 2018).\nEmbeddings also encode the implicit associations that are a property of human\nreasoning. The Implicit Association Test (Greenwald et al., 1998) measures peo- 6.12 \u2022 E VALUATING VECTOR MODELS 127\nple\u2019s associations between concepts (like \u2018\ufb02owers\u2019 or \u2018insects\u2019) and attributes (like\n\u2018pleasantness\u2019 and \u2018unpleasantness\u2019) by measuring differences in the latency with\nwhich they label words in the various categories.7Using such methods, people\nin the United States have been shown to associate African-American names with\nunpleasant words (more than European-American names), male names more with\nmathematics and female names with the arts, and old people\u2019s names with unpleas-\nant words (Greenwald et al. 1998, Nosek et al. 2002a, Nosek et al. 2002b). Caliskan\net al. (2017) replicated all these \ufb01ndings of implicit associations using GloVe vectors\nand cosine similarity instead of human latencies. For example African-American\nnames like \u2018Leroy\u2019 and \u2018Shaniqua\u2019 had a higher GloVe cosine with unpleasant words\nwhile European-American names (\u2018Brad\u2019, \u2018Greg\u2019, \u2018Courtney\u2019) had a higher cosine\nwith pleasant words. These problems with embeddings are an example of a repre-\nsentational harm (Crawford 2017, Blodgett et al. 2020), which is a harm caused byrepresentational\nharm\na system demeaning or even ignoring some social groups. Any embedding-aware al-\ngorithm that made use of word sentiment could thus exacerbate bias against African\nAmericans.\nRecent research focuses on ways to try to remove these kinds of biases, for\nexample by developing a transformation of the embedding space that removes gen-\nder stereotypes but preserves de\ufb01nitional gender (Bolukbasi et al. 2016, Zhao et al.\n2017) or changing the training procedure (Zhao et al., 2018b). However, although\nthese sorts of debiasing may reduce bias in embeddings, they do not eliminate it debiasing\n(Gonen and Goldberg, 2019), and this remains an open problem.\nHistorical embeddings are also being used to measure biases in the past. Garg\net al. (2018) used embeddings from historical texts to measure the association be-\ntween embeddings for occupations and embeddings for names of various ethnici-\nties or genders (for example the relative cosine similarity of women\u2019s names versus\nmen\u2019s to occupation words like \u2018librarian\u2019 or \u2018carpenter\u2019) across the 20th century.\nThey found that the cosines correlate with the empirical historical percentages of\nwomen or ethnic groups in those occupations. Historical embeddings also repli-\ncated old surveys of ethnic stereotypes; the tendency of experimental participants in\n1933 to associate adjectives like \u2018industrious\u2019 or \u2018superstitious\u2019 with, e.g., Chinese\nethnicity, correlates with the cosine between Chinese last names and those adjectives\nusing embeddings trained on 1930s text. They also were able to document historical\ngender biases, such as the fact that embeddings for adjectives related to competence\n(\u2018smart\u2019, \u2018wise\u2019, \u2018thoughtful\u2019, \u2018resourceful\u2019) had a higher cosine with male than fe-\nmale words, and showed that this bias has been slowly decreasing since 1960. We\nreturn in later chapters to this question about the role of bias in natural language\nprocessing.\n6.12 Evaluating Vector Models\nThe most important evaluation metric for vector models is extrinsic evaluation on\ntasks, i.e., using vectors in an NLP task and seeing whether this improves perfor-\nmance over some other model.\n7Roughly speaking, if humans associate \u2018\ufb02owers\u2019 with \u2018pleasantness\u2019 and \u2018insects\u2019 with \u2018unpleasant-\nness\u2019, when they are instructed to push a green button for \u2018\ufb02owers\u2019 (daisy, iris, lilac) and \u2018pleasant words\u2019\n(love, laughter, pleasure) and a red button for \u2018insects\u2019 (\ufb02ea, spider, mosquito) and \u2018unpleasant words\u2019\n(abuse, hatred, ugly) they are faster than in an incongruous condition where they push a red button for\n\u2018\ufb02owers\u2019 and \u2018unpleasant words\u2019 and a green button for \u2018insects\u2019 and \u2018pleasant words\u2019. 128 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\nNonetheless it is useful to have intrinsic evaluations. The most common metric\nis to test their performance on similarity , computing the correlation between an\nalgorithm\u2019s word similarity scores and word similarity ratings assigned by humans.\nWordSim-353 (Finkelstein et al., 2002) is a commonly used set of ratings from 0\nto 10 for 353 noun pairs; for example ( plane ,car) had an average score of 5.77.\nSimLex-999 (Hill et al., 2015) is a more complex dataset that quanti\ufb01es similarity\n(cup, mug ) rather than relatedness ( cup, coffee ), and includes concrete and abstract\nadjective, noun and verb pairs. The TOEFL dataset is a set of 80 questions, each\nconsisting of a target word with 4 additional word choices; the task is to choose\nwhich is the correct synonym, as in the example: Levied is closest in meaning to:\nimposed, believed, requested, correlated (Landauer and Dumais, 1997). All of these\ndatasets present words without context.\nSlightly more realistic are intrinsic similarity tasks that include context. The\nStanford Contextual Word Similarity (SCWS) dataset (Huang et al., 2012) and the\nWord-in-Context (WiC) dataset (Pilehvar and Camacho-Collados, 2019) offer richer\nevaluation scenarios. SCWS gives human judgments on 2,003 pairs of words in\ntheir sentential context, while WiC gives target words in two sentential contexts that\nare either in the same or different senses; see Appendix G. The semantic textual\nsimilarity task (Agirre et al. 2012, Agirre et al. 2015) evaluates the performance of\nsentence-level similarity algorithms, consisting of a set of pairs of sentences, each\npair with human-labeled similarity scores.\nAnother task used for evaluation is the analogy task, discussed on page 124,\nwhere the system has to solve problems of the form a is to b as a* is to b* , given a, b,\nanda*and having to \ufb01nd b*(Turney and Littman, 2005). A number of sets of tuples\nhave been created for this task (Mikolov et al. 2013a, Mikolov et al. 2013c, Gladkova\net al. 2016), covering morphology ( city:cities::child:children ), lexicographic rela-\ntions ( leg:table::spout:teapot ) and encyclopedia relations ( Beijing:China::Dublin:Ireland ),\nsome drawing from the SemEval-2012 Task 2 dataset of 79 different relations (Jur-\ngens et al., 2012).\nAll embedding algorithms suffer from inherent variability. For example because\nof randomness in the initialization and the random negative sampling, algorithms\nlike word2vec may produce different results even from the same dataset, and in-\ndividual documents in a collection may strongly impact the resulting embeddings\n(Tian et al. 2016, Hellrich and Hahn 2016, Antoniak and Mimno 2018). When em-\nbeddings are used to study word associations in particular corpora, therefore, it is\nbest practice to train multiple embeddings with bootstrap sampling over documents\nand average the results (Antoniak and Mimno, 2018).\n6.13 Summary\n\u2022 In vector semantics, a word is modeled as a vector\u2014a point in high-dimensional\nspace, also called an embedding . In this chapter we focus on static embed-\ndings , where each word is mapped to a \ufb01xed embedding.\n\u2022 Vector semantic models fall into two classes: sparse anddense . In sparse\nmodels each dimension corresponds to a word in the vocabulary Vand cells\nare functions of co-occurrence counts . The term-document matrix has a\nrow for each word ( term ) in the vocabulary and a column for each document.\nTheword-context orterm-term matrix has a row for each (target) word in BIBLIOGRAPHICAL AND HISTORICAL NOTES 129\nthe vocabulary and a column for each context term in the vocabulary. Two\nsparse weightings are common: the tf-idf weighting which weights each cell\nby its term frequency andinverse document frequency , and PPMI (point-\nwise positive mutual information), which is most common for word-context\nmatrices.\n\u2022 Dense vector models have dimensionality 50\u20131000. Word2vec algorithms\nlikeskip-gram are a popular way to compute dense embeddings. Skip-gram\ntrains a logistic regression classi\ufb01er to compute the probability that two words\nare \u2018likely to occur nearby in text\u2019. This probability is computed from the dot\nproduct between the embeddings for the two words.\n\u2022 Skip-gram uses stochastic gradient descent to train the classi\ufb01er, by learning\nembeddings that have a high dot product with embeddings of words that occur\nnearby and a low dot product with noise words.\n\u2022 Other important embedding algorithms include GloVe , a method based on\nratios of word co-occurrence probabilities.\n\u2022 Whether using sparse or dense vectors, word and document similarities are\ncomputed by some function of the dot product between vectors. The cosine\nof two vectors\u2014a normalized dot product\u2014is the most popular such metric.\nBibliographical and Historical Notes\nThe idea of vector semantics arose out of research in the 1950s in three distinct\n\ufb01elds: linguistics, psychology, and computer science, each of which contributed a\nfundamental aspect of the model.\nThe idea that meaning is related to the distribution of words in context was\nwidespread in linguistic theory of the 1950s, among distributionalists like Zellig\nHarris, Martin Joos, and J. R. Firth, and semioticians like Thomas Sebeok. As Joos\n(1950) put it,\nthe linguist\u2019s \u201cmeaning\u201d of a morpheme. . . is by de\ufb01nition the set of conditional\nprobabilities of its occurrence in context with all other morphemes.\nThe idea that the meaning of a word might be modeled as a point in a multi-\ndimensional semantic space came from psychologists like Charles E. Osgood, who\nhad been studying how people responded to the meaning of words by assigning val-\nues along scales like happy/sad orhard/soft . Osgood et al. (1957) proposed that the\nmeaning of a word in general could be modeled as a point in a multidimensional\nEuclidean space, and that the similarity of meaning between two words could be\nmodeled as the distance between these points in the space.\nA \ufb01nal intellectual source in the 1950s and early 1960s was the \ufb01eld then called\nmechanical indexing , now known as information retrieval . In what became knownmechanical\nindexing\nas the vector space model for information retrieval (Salton 1971, Sparck Jones\n1986), researchers demonstrated new ways to de\ufb01ne the meaning of words in terms\nof vectors (Switzer, 1965), and re\ufb01ned methods for word similarity based on mea-\nsures of statistical association between words like mutual information (Giuliano,\n1965) and idf (Sparck Jones, 1972), and showed that the meaning of documents\ncould be represented in the same vector spaces used for words. Around the same\ntime, (Cordier, 1965) showed that factor analysis of word association probabilities\ncould be used to form dense vector representations of words. 130 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\nSome of the philosophical underpinning of the distributional way of thinking\ncame from the late writings of the philosopher Wittgenstein, who was skeptical of\nthe possibility of building a completely formal theory of meaning de\ufb01nitions for\neach word. Wittgenstein suggested instead that \u201cthe meaning of a word is its use in\nthe language\u201d (Wittgenstein, 1953, PI 43). That is, instead of using some logical lan-\nguage to de\ufb01ne each word, or drawing on denotations or truth values, Wittgenstein\u2019s\nidea is that we should de\ufb01ne a word by how it is used by people in speaking and un-\nderstanding in their day-to-day interactions, thus pre\ufb01guring the movement toward\nembodied and experiential models in linguistics and NLP (Glenberg and Robertson\n2000, Lake and Murphy 2021, Bisk et al. 2020, Bender and Koller 2020).\nMore distantly related is the idea of de\ufb01ning words by a vector of discrete fea-\ntures, which has roots at least as far back as Descartes and Leibniz (Wierzbicka 1992,\nWierzbicka 1996). By the middle of the 20th century, beginning with the work of\nHjelmslev (Hjelmslev, 1969) (originally 1943) and \ufb02eshed out in early models of\ngenerative grammar (Katz and Fodor, 1963), the idea arose of representing mean-\ning with semantic features , symbols that represent some sort of primitive meaning.semantic\nfeature\nFor example words like hen,rooster , orchick , have something in common (they all\ndescribe chickens) and something different (their age and sex), representable as:\nhen+female, +chicken, +adult\nrooster-female, +chicken, +adult\nchick+chicken, -adult\nThe dimensions used by vector models of meaning to de\ufb01ne words, however, are\nonly abstractly related to this idea of a small \ufb01xed number of hand-built dimensions.\nNonetheless, there has been some attempt to show that certain dimensions of em-\nbedding models do contribute some speci\ufb01c compositional aspect of meaning like\nthese early semantic features.\nThe use of dense vectors to model word meaning, and indeed the term embed-\nding , grew out of the latent semantic indexing (LSI) model (Deerwester et al.,\n1988) recast as LSA (latent semantic analysis ) (Deerwester et al., 1990). In LSA\nsingular value decomposition \u2014SVD \u2014 is applied to a term-document matrix (each SVD\ncell weighted by log frequency and normalized by entropy), and then the \ufb01rst 300\ndimensions are used as the LSA embedding. Singular Value Decomposition (SVD)\nis a method for \ufb01nding the most important dimensions of a data set, those dimen-\nsions along which the data varies the most. LSA was then quickly widely applied:\nas a cognitive model Landauer and Dumais (1997), and for tasks like spell checking\n(Jones and Martin, 1997), language modeling (Bellegarda 1997, Coccaro and Ju-\nrafsky 1998, Bellegarda 2000), morphology induction (Schone and Jurafsky 2000,\nSchone and Jurafsky 2001b), multiword expressions (MWEs) (Schone and Juraf-\nsky, 2001a), and essay grading (Rehder et al., 1998). Related models were simul-\ntaneously developed and applied to word sense disambiguation by Sch \u00a8utze (1992b).\nLSA also led to the earliest use of embeddings to represent words in a probabilis-\ntic classi\ufb01er, in the logistic regression document router of Sch \u00a8utze et al. (1995).\nThe idea of SVD on the term-term matrix (rather than the term-document matrix)\nas a model of meaning for NLP was proposed soon after LSA by Sch \u00a8utze (1992b).\nSch\u00a8utze applied the low-rank (97-dimensional) embeddings produced by SVD to the\ntask of word sense disambiguation, analyzed the resulting semantic space, and also\nsuggested possible techniques like dropping high-order dimensions. See Sch \u00a8utze\n(1997).\nA number of alternative matrix models followed on from the early SVD work,\nincluding Probabilistic Latent Semantic Indexing (PLSI) (Hofmann, 1999), Latent EXERCISES 131\nDirichlet Allocation (LDA) (Blei et al., 2003), and Non-negative Matrix Factoriza-\ntion (NMF) (Lee and Seung, 1999).\nThe LSA community seems to have \ufb01rst used the word \u201cembedding\u201d in Landauer\net al. (1997), in a variant of its mathematical meaning as a mapping from one space\nor mathematical structure to another. In LSA, the word embedding seems to have\ndescribed the mapping from the space of sparse count vectors to the latent space of\nSVD dense vectors. Although the word thus originally meant the mapping from one\nspace to another, it has metonymically shifted to mean the resulting dense vector in\nthe latent space, and it is in this sense that we currently use the word.\nBy the next decade, Bengio et al. (2003) and Bengio et al. (2006) showed that\nneural language models could also be used to develop embeddings as part of the task\nof word prediction. Collobert and Weston (2007), Collobert and Weston (2008), and\nCollobert et al. (2011) then demonstrated that embeddings could be used to represent\nword meanings for a number of NLP tasks. Turian et al. (2010) compared the value\nof different kinds of embeddings for different NLP tasks. Mikolov et al. (2011)\nshowed that recurrent neural nets could be used as language models. The idea of\nsimplifying the hidden layer of these neural net language models to create the skip-\ngram (and also CBOW) algorithms was proposed by Mikolov et al. (2013a). The\nnegative sampling training algorithm was proposed in Mikolov et al. (2013b). There\nare numerous surveys of static embeddings and their parameterizations (Bullinaria\nand Levy 2007, Bullinaria and Levy 2012, Lapesa and Evert 2014, Kiela and Clark\n2014, Levy et al. 2015).\nSee Manning et al. (2008) and Chapter 14 for a deeper understanding of the role\nof vectors in information retrieval, including how to compare queries with docu-\nments, more details on tf-idf, and issues of scaling to very large datasets. See Kim\n(2019) for a clear and comprehensive tutorial on word2vec. Cruse (2004) is a useful\nintroductory linguistic text on lexical semantics.\nExercises 132 CHAPTER 7 \u2022 N EURAL NETWORKS\nCHAPTER\n7Neural Networks\n\u201c[M]achines of this character can behave in a very complicated manner when\nthe number of units is large.\u201d\nAlan Turing (1948) \u201cIntelligent Machines\u201d, page 6\nNeural networks are a fundamental computational tool for language process-\ning, and a very old one. They are called neural because their origins lie in the\nMcCulloch-Pitts neuron (McCulloch and Pitts, 1943), a simpli\ufb01ed model of the\nbiological neuron as a kind of computing element that could be described in terms\nof propositional logic. But the modern use in language processing no longer draws\non these early biological inspirations.\nInstead, a modern neural network is a network of small computing units, each\nof which takes a vector of input values and produces a single output value. In this\nchapter we introduce the neural net applied to classi\ufb01cation. The architecture we\nintroduce is called a feedforward network because the computation proceeds iter- feedforward\natively from one layer of units to the next. The use of modern neural nets is often\ncalled deep learning , because modern networks are often deep (have many layers). deep learning\nNeural networks share much of the same mathematics as logistic regression. But\nneural networks are a more powerful classi\ufb01er than logistic regression, and indeed a\nminimal neural network (technically one with a single \u2018hidden layer\u2019) can be shown\nto learn any function.\nNeural net classi\ufb01ers are different from logistic regression in another way. With\nlogistic regression, we applied the regression classi\ufb01er to many different tasks by\ndeveloping many rich kinds of feature templates based on domain knowledge. When\nworking with neural networks, it is more common to avoid most uses of rich hand-\nderived features, instead building neural networks that take raw words as inputs\nand learn to induce features as part of the process of learning to classify. We saw\nexamples of this kind of representation learning for embeddings in Chapter 6. Nets\nthat are very deep are particularly good at representation learning. For that reason\ndeep neural nets are the right tool for tasks that offer suf\ufb01cient data to learn features\nautomatically.\nIn this chapter we\u2019ll introduce feedforward networks as classi\ufb01ers, and also ap-\nply them to the simple task of language modeling: assigning probabilities to word\nsequences and predicting upcoming words. In subsequent chapters we\u2019ll introduce\nmany other aspects of neural models, such as recurrent neural networks (Chap-\nter 8), the Transformer (Chapter 9), and masked language modeling (Chapter 11). 7.1 \u2022 U NITS 133\n7.1 Units\nThe building block of a neural network is a single computational unit. A unit takes\na set of real valued numbers as input, performs some computation on them, and\nproduces an output.\nAt its heart, a neural unit is taking a weighted sum of its inputs, with one addi-\ntional term in the sum called a bias term . Given a set of inputs x1:::xn, a unit has bias term\na set of corresponding weights w1:::wnand a bias b, so the weighted sum zcan be\nrepresented as:\nz=b+X\niwixi (7.1)\nOften it\u2019s more convenient to express this weighted sum using vector notation; recall\nfrom linear algebra that a vector is, at heart, just a list or array of numbers. Thus vector\nwe\u2019ll talk about zin terms of a weight vector w, a scalar bias b, and an input vector\nx, and we\u2019ll replace the sum with the convenient dot product :\nz=w\u0001x+b (7.2)\nAs de\ufb01ned in Eq. 7.2, zis just a real valued number.\nFinally, instead of using z, a linear function of x, as the output, neural units\napply a non-linear function ftoz. We will refer to the output of this function as\ntheactivation value for the unit, a. Since we are just modeling a single unit, the activation\nactivation for the node is in fact the \ufb01nal output of the network, which we\u2019ll generally\ncally. So the value yis de\ufb01ned as:\ny=a=f(z)\nWe\u2019ll discuss three popular non-linear functions fbelow (the sigmoid, the tanh, and\nthe recti\ufb01ed linear unit or ReLU) but it\u2019s pedagogically convenient to start with the\nsigmoid function since we saw it in Chapter 5: sigmoid\ny=s(z) =1\n1+e\u0000z(7.3)\nThe sigmoid (shown in Fig. 7.1) has a number of advantages; it maps the output\ninto the range (0;1), which is useful in squashing outliers toward 0 or 1. And it\u2019s\ndifferentiable, which as we saw in Section 5.10 will be handy for learning.\nFigure 7.1 The sigmoid function takes a real value and maps it to the range (0;1). It is\nnearly linear around 0 but outlier values get squashed toward 0 or 1.\nSubstituting Eq. 7.2 into Eq. 7.3 gives us the output of a neural unit:\ny=s(w\u0001x+b) =1\n1+exp(\u0000(w\u0001x+b))(7.4) 134 CHAPTER 7 \u2022 N EURAL NETWORKS\nFig. 7.2 shows a \ufb01nal schematic of a basic neural unit. In this example the unit\ntakes 3 input values x1;x2, and x3, and computes a weighted sum, multiplying each\nvalue by a weight ( w1,w2, and w3, respectively), adds them to a bias term b, and then\npasses the resulting sum through a sigmoid function to result in a number between 0\nand 1.\nx1x2x3\nyw1w2w3\u2211b\u03c3+1za\nFigure 7.2 A neural unit, taking 3 inputs x1,x2, and x3(and a bias bthat we represent as a\nweight for an input clamped at +1) and producing an output y. We include some convenient\nintermediate variables: the output of the summation, z, and the output of the sigmoid, a. In\nthis case the output of the unit yis the same as a, but in deeper networks we\u2019ll reserve yto\nmean the \ufb01nal output of the entire network, leaving aas the activation of an individual node.\nLet\u2019s walk through an example just to get an intuition. Let\u2019s suppose we have a\nunit with the following weight vector and bias:\nw= [0:2;0:3;0:9]\nb=0:5\nWhat would this unit do with the following input vector:\nx= [0:5;0:6;0:1]\nThe resulting output ywould be:\ny=s(w\u0001x+b) =1\n1+e\u0000(w\u0001x+b)=1\n1+e\u0000(:5\u0003:2+:6\u0003:3+:1\u0003:9+:5)=1\n1+e\u00000:87=:70\nIn practice, the sigmoid is not commonly used as an activation function. A function\nthat is very similar but almost always better is the tanh function shown in Fig. 7.3a; tanh\ntanh is a variant of the sigmoid that ranges from -1 to +1:\ny=tanh(z) =ez\u0000e\u0000z\nez+e\u0000z(7.5)\nThe simplest activation function, and perhaps the most commonly used, is the rec-\nti\ufb01ed linear unit, also called the ReLU , shown in Fig. 7.3b. It\u2019s just the same as z ReLU\nwhen zis positive, and 0 otherwise:\ny=ReLU(z) =max(z;0) (7.6)\nThese activation functions have different properties that make them useful for differ-\nent language applications or network architectures. For example, the tanh function\nhas the nice properties of being smoothly differentiable and mapping outlier values\ntoward the mean. The recti\ufb01er function, on the other hand, has nice properties that 7.2 \u2022 T HEXOR PROBLEM 135\n(a) (b)\nFigure 7.3 The tanh and ReLU activation functions.\nresult from it being very close to linear. In the sigmoid or tanh functions, very high\nvalues of zresult in values of ythat are saturated , i.e., extremely close to 1, and have saturated\nderivatives very close to 0. Zero derivatives cause problems for learning, because as\nwe\u2019ll see in Section 7.5, we\u2019ll train networks by propagating an error signal back-\nwards, multiplying gradients (partial derivatives) from each layer of the network;\ngradients that are almost 0 cause the error signal to get smaller and smaller until it is\ntoo small to be used for training, a problem called the vanishing gradient problem.vanishing\ngradient\nRecti\ufb01ers don\u2019t have this problem, since the derivative of ReLU for high values of z\nis 1 rather than very close to 0.\n7.2 The XOR problem\nEarly in the history of neural networks it was realized that the power of neural net-\nworks, as with the real neurons that inspired them, comes from combining these\nunits into larger networks.\nOne of the most clever demonstrations of the need for multi-layer networks was\nthe proof by Minsky and Papert (1969) that a single neural unit cannot compute\nsome very simple functions of its input. Consider the task of computing elementary\nlogical functions of two inputs, like AND, OR, and XOR. As a reminder, here are\nthe truth tables for those functions:\nAND OR XOR\nx1 x2y x1 x2 y x1 x2 y\n0 00 0 0 0 0 0 0\n0 10 0 1 1 0 1 1\n1 00 1 0 1 1 0 1\n1 11 1 1 1 1 1 0\nThis example was \ufb01rst shown for the perceptron , which is a very simple neural perceptron\nunit that has a binary output and does nothave a non-linear activation function. The\noutput yof a perceptron is 0 or 1, and is computed as follows (using the same weight\nw, input x, and bias bas in Eq. 7.2):\ny=\u001a0;ifw\u0001x+b\u00140\n1;ifw\u0001x+b>0(7.7) 136 CHAPTER 7 \u2022 N EURAL NETWORKS\nIt\u2019s very easy to build a perceptron that can compute the logical AND and OR\nfunctions of its binary inputs; Fig. 7.4 shows the necessary weights.\nx1x2+1-111\nx1x2+1011\n(a) (b)\nFigure 7.4 The weights wand bias bfor perceptrons for computing logical functions. The\ninputs are shown as x1andx2and the bias as a special node with value +1 which is multiplied\nwith the bias weight b. (a) logical AND, with weights w1=1 and w2=1 and bias weight\nb=\u00001. (b) logical OR, with weights w1=1 and w2=1 and bias weight b=0. These\nweights/biases are just one from an in\ufb01nite number of possible sets of weights and biases that\nwould implement the functions.\nIt turns out, however, that it\u2019s not possible to build a perceptron to compute\nlogical XOR! (It\u2019s worth spending a moment to give it a try!)\nThe intuition behind this important result relies on understanding that a percep-\ntron is a linear classi\ufb01er. For a two-dimensional input x1andx2, the perceptron\nequation, w1x1+w2x2+b=0 is the equation of a line. (We can see this by putting\nit in the standard linear format: x2= (\u0000w1=w2)x1+ (\u0000b=w2).) This line acts as a\ndecision boundary in two-dimensional space in which the output 0 is assigned to alldecision\nboundary\ninputs lying on one side of the line, and the output 1 to all input points lying on the\nother side of the line. If we had more than 2 inputs, the decision boundary becomes\na hyperplane instead of a line, but the idea is the same, separating the space into two\ncategories.\nFig. 7.5 shows the possible logical inputs ( 00,01,10, and11) and the line drawn\nby one possible set of parameters for an AND and an OR classi\ufb01er. Notice that there\nis simply no way to draw a line that separates the positive cases of XOR (01 and 10)\nfrom the negative cases (00 and 11). We say that XOR is not a linearly separablelinearly\nseparable\nfunction. Of course we could draw a boundary with a curve, or some other function,\nbut not a single line.\n7.2.1 The solution: neural networks\nWhile the XOR function cannot be calculated by a single perceptron, it can be cal-\nculated by a layered network of perceptron units. Rather than see this with networks\nof simple perceptrons, however, let\u2019s see how to compute XOR using two layers of\nReLU-based units following Goodfellow et al. (2016). Fig. 7.6 shows a \ufb01gure with\nthe input being processed by two layers of neural units. The middle layer (called\nh) has two units, and the output layer (called y) has one unit. A set of weights and\nbiases are shown that allows the network to correctly compute the XOR function.\nLet\u2019s walk through what happens with the input x= [0, 0]. If we multiply each\ninput value by the appropriate weight, sum, and then add the bias b, we get the vector\n[0, -1], and we then apply the recti\ufb01ed linear transformation to give the output of the\nhlayer as [0, 0]. Now we once again multiply by the weights, sum, and add the\nbias (0 in this case) resulting in the value 0. The reader should work through the\ncomputation of the remaining 3 possible input pairs to see that the resulting yvalues\nare 1 for the inputs [0, 1] and [1, 0] and 0 for [0, 0] and [1, 1]. 7.2 \u2022 T HEXOR PROBLEM 137\n0011x1x20011x1x20011x1x2\na)  x1 AND x2b)  x1 OR x2c)  x1 XOR x2?\nFigure 7.5 The functions AND, OR, and XOR, represented with input x1on the x-axis and input x2on the\ny-axis. Filled circles represent perceptron outputs of 1, and white circles perceptron outputs of 0. There is no\nway to draw a line that correctly separates the two categories for XOR. Figure styled after Russell and Norvig\n(2002).\nx1x2h1h2y1+11-1111-201+10\nFigure 7.6 XOR solution after Goodfellow et al. (2016). There are three ReLU units, in\ntwo layers; we\u2019ve called them h1,h2(hfor \u201chidden layer\u201d) and y1. As before, the numbers\non the arrows represent the weights wfor each unit, and we represent the bias bas a weight\non a unit clamped to +1, with the bias weights/units in gray.\nIt\u2019s also instructive to look at the intermediate results, the outputs of the two\nhidden nodes h1andh2. We showed in the previous paragraph that the hvector for\nthe inputs x= [0, 0] was [0, 0]. Fig. 7.7b shows the values of the hlayer for all\n4 inputs. Notice that hidden representations of the two input points x= [0, 1] and\nx= [1, 0] (the two cases with XOR output = 1) are merged to the single point h=\n[1, 0]. The merger makes it easy to linearly separate the positive and negative cases\nof XOR. In other words, we can view the hidden layer of the network as forming a\nrepresentation of the input.\nIn this example we just stipulated the weights in Fig. 7.6. But for real examples\nthe weights for neural networks are learned automatically using the error backprop-\nagation algorithm to be introduced in Section 7.5. That means the hidden layers will\nlearn to form useful representations. This intuition, that neural networks can auto-\nmatically learn useful representations of the input, is one of their key advantages,\nand one that we will return to again and again in later chapters. 138 CHAPTER 7 \u2022 N EURAL NETWORKS\n0011x1x2\na) The original x space0011h1h2\n2b) The new (linearly separable) h space\nFigure 7.7 The hidden layer forming a new representation of the input. (b) shows the\nrepresentation of the hidden layer, h, compared to the original input representation xin (a).\nNotice that the input point [0, 1] has been collapsed with the input point [1, 0], making it\npossible to linearly separate the positive and negative cases of XOR. After Goodfellow et al.\n(2016).\n7.3 Feedforward Neural Networks\nLet\u2019s now walk through a slightly more formal presentation of the simplest kind of\nneural network, the feedforward network . A feedforward network is a multilayerfeedforward\nnetwork\nnetwork in which the units are connected with no cycles; the outputs from units in\neach layer are passed to units in the next higher layer, and no outputs are passed\nback to lower layers. (In Chapter 8 we\u2019ll introduce networks with cycles, called\nrecurrent neural networks .)\nFor historical reasons multilayer networks, especially feedforward networks, are\nsometimes called multi-layer perceptrons (orMLP s); this is a technical misnomer,multi-layer\nperceptrons\nMLP since the units in modern multilayer networks aren\u2019t perceptrons (perceptrons are\npurely linear, but modern networks are made up of units with non-linearities like\nsigmoids), but at some point the name stuck.\nSimple feedforward networks have three kinds of nodes: input units, hidden\nunits, and output units.\nFig. 7.8 shows a picture. The input layer xis a vector of simple scalar values just\nas we saw in Fig. 7.2.\nThe core of the neural network is the hidden layer hformed of hidden units hi, hidden layer\neach of which is a neural unit as described in Section 7.1, taking a weighted sum of\nits inputs and then applying a non-linearity. In the standard architecture, each layer\nisfully-connected , meaning that each unit in each layer takes as input the outputs fully-connected\nfrom all the units in the previous layer, and there is a link between every pair of units\nfrom two adjacent layers. Thus each hidden unit sums over all the input units.\nRecall that a single hidden unit has as parameters a weight vector and a bias. We\nrepresent the parameters for the entire hidden layer by combining the weight vector\nand bias for each unit iinto a single weight matrix Wand a single bias vector bfor\nthe whole layer (see Fig. 7.8). Each element Wjiof the weight matrix Wrepresents\nthe weight of the connection from the ith input unit xito the jth hidden unit hj.\nThe advantage of using a single matrix Wfor the weights of the entire layer is\nthat now the hidden layer computation for a feedforward network can be done very\nef\ufb01ciently with simple matrix operations. In fact, the computation only has three 7.3 \u2022 F EEDFORWARD NEURAL NETWORKS 139\nx1x2xn0\u2026\u2026+1b\u2026UW\ninput layerhidden layeroutput layerh1y1y2yn2h2h3hn1\nFigure 7.8 A simple 2-layer feedforward network, with one hidden layer, one output layer,\nand one input layer (the input layer is usually not counted when enumerating layers).\nsteps: multiplying the weight matrix by the input vector x, adding the bias vector b,\nand applying the activation function g(such as the sigmoid, tanh, or ReLU activation\nfunction de\ufb01ned above).\nThe output of the hidden layer, the vector h, is thus the following (for this exam-\nple we\u2019ll use the sigmoid function sas our activation function):\nh=s(Wx+b) (7.8)\nNotice that we\u2019re applying the sfunction here to a vector, while in Eq. 7.3 it was\napplied to a scalar. We\u2019re thus allowing s(\u0001), and indeed any activation function\ng(\u0001), to apply to a vector element-wise, so g[z1;z2;z3] = [g(z1);g(z2);g(z3)].\nLet\u2019s introduce some constants to represent the dimensionalities of these vectors\nand matrices. We\u2019ll refer to the input layer as layer 0 of the network, and have n0\nrepresent the number of inputs, so xis a vector of real numbers of dimension n0,\nor more formally x2Rn0, a column vector of dimensionality [n0;1]. Let\u2019s call the\nhidden layer layer 1 and the output layer layer 2. The hidden layer has dimensional-\nityn1, soh2Rn1and also b2Rn1(since each hidden unit can take a different bias\nvalue). And the weight matrix Whas dimensionality W2Rn1\u0002n0, i.e.[n1;n0].\nTake a moment to convince yourself that the matrix multiplication in Eq. 7.8 will\ncompute the value of each hjass\u0000Pn0\ni=1Wjixi+bj\u0001\n.\nAs we saw in Section 7.2, the resulting value h(forhidden but also for hypoth-\nesis) forms a representation of the input. The role of the output layer is to take\nthis new representation hand compute a \ufb01nal output. This output could be a real-\nvalued number, but in many cases the goal of the network is to make some sort of\nclassi\ufb01cation decision, and so we will focus on the case of classi\ufb01cation.\nIf we are doing a binary task like sentiment classi\ufb01cation, we might have a sin-\ngle output node, and its scalar value yis the probability of positive versus negative\nsentiment. If we are doing multinomial classi\ufb01cation, such as assigning a part-of-\nspeech tag, we might have one output node for each potential part-of-speech, whose\noutput value is the probability of that part-of-speech, and the values of all the output\nnodes must sum to one. The output layer is thus a vector ythat gives a probability\ndistribution across the output nodes.\nLet\u2019s see how this happens. Like the hidden layer, the output layer has a weight\nmatrix (let\u2019s call it U), but some models don\u2019t include a bias vector bin the output 140 CHAPTER 7 \u2022 N EURAL NETWORKS\nlayer, so we\u2019ll simplify by eliminating the bias vector in this example. The weight\nmatrix is multiplied by its input vector ( h) to produce the intermediate output z:\nz=Uh\nThere are n2output nodes, so z2Rn2, weight matrix Uhas dimensionality U2\nRn2\u0002n1, and element Ui jis the weight from unit jin the hidden layer to unit iin the\noutput layer.\nHowever, zcan\u2019t be the output of the classi\ufb01er, since it\u2019s a vector of real-valued\nnumbers, while what we need for classi\ufb01cation is a vector of probabilities. There is\na convenient function for normalizing a vector of real values, by which we mean normalizing\nconverting it to a vector that encodes a probability distribution (all the numbers lie\nbetween 0 and 1 and sum to 1): the softmax function that we saw on page 85 of softmax\nChapter 5. More generally for any vector zof dimensionality d, the softmax is\nde\ufb01ned as:\nsoftmax (zi) =exp(zi)Pd\nj=1exp(zj)1\u0014i\u0014d (7.9)\nThus for example given a vector\nz= [0:6;1:1;\u00001:5;1:2;3:2;\u00001:1]; (7.10)\nthe softmax function will normalize it to a probability distribution (shown rounded):\nsoftmax (z) = [ 0:055;0:090;0:0067;0:10;0:74;0:010] (7.11)\nYou may recall that we used softmax to create a probability distribution from a\nvector of real-valued numbers (computed from summing weights times features) in\nthe multinomial version of logistic regression in Chapter 5.\nThat means we can think of a neural network classi\ufb01er with one hidden layer\nas building a vector hwhich is a hidden layer representation of the input, and then\nrunning standard multinomial logistic regression on the features that the network\ndevelops in h. By contrast, in Chapter 5 the features were mainly designed by hand\nvia feature templates. So a neural network is like multinomial logistic regression,\nbut (a) with many layers, since a deep neural network is like layer after layer of lo-\ngistic regression classi\ufb01ers; (b) with those intermediate layers having many possible\nactivation functions (tanh, ReLU, sigmoid) instead of just sigmoid (although we\u2019ll\ncontinue to use sfor convenience to mean any activation function); (c) rather than\nforming the features by feature templates, the prior layers of the network induce the\nfeature representations themselves.\nHere are the \ufb01nal equations for a feedforward network with a single hidden layer,\nwhich takes an input vector x, outputs a probability distribution y, and is parameter-\nized by weight matrices WandUand a bias vector b:\nh=s(Wx+b)\nz=Uh\ny=softmax (z) (7.12)\nAnd just to remember the shapes of all our variables, x2Rn0,h2Rn1,b2Rn1,\nW2Rn1\u0002n0,U2Rn2\u0002n1, and the output vector y2Rn2. We\u2019ll call this network a 2-\nlayer network (we traditionally don\u2019t count the input layer when numbering layers,\nbut do count the output layer). So by this terminology logistic regression is a 1-layer\nnetwork. 7.3 \u2022 F EEDFORWARD NEURAL NETWORKS 141\n7.3.1 More details on feedforward networks\nLet\u2019s now set up some notation to make it easier to talk about deeper networks of\ndepth more than 2. We\u2019ll use superscripts in square brackets to mean layer num-\nbers, starting at 0 for the input layer. So W[1]will mean the weight matrix for the\n(\ufb01rst) hidden layer, and b[1]will mean the bias vector for the (\ufb01rst) hidden layer. nj\nwill mean the number of units at layer j. We\u2019ll use g(\u0001)to stand for the activation\nfunction, which will tend to be ReLU or tanh for intermediate layers and softmax\nfor output layers. We\u2019ll use a[i]to mean the output from layer i, and z[i]to mean the\ncombination of previous layer output, weights and biases W[i]a[i\u00001]+b[i]. The 0th\nlayer is for inputs, so we\u2019ll refer to the inputs xmore generally as a[0].\nThus we can re-represent our 2-layer net from Eq. 7.12 as follows:\nz[1]=W[1]a[0]+b[1]\na[1]=g[1](z[1])\nz[2]=W[2]a[1]+b[2]\na[2]=g[2](z[2])\n\u02c6y=a[2](7.13)\nNote that with this notation, the equations for the computation done at each layer are\nthe same. The algorithm for computing the forward step in an n-layer feedforward\nnetwork, given the input vector a[0]is thus simply:\nforiin1,...,n\nz[i]=W[i]a[i\u00001]+b[i]\na[i]=g[i](z[i])\n\u02c6y=a[n]\nIt\u2019s often useful to have a name for the \ufb01nal set of activations right before the \ufb01nal\nsoftmax. So however many layers we have, we\u2019ll generally call the unnormalized\nvalues in the \ufb01nal vector z[n], the vector of scores right before the \ufb01nal softmax, the\nlogits (see (5.7). logits\nThe need for non-linear activation functions One of the reasons we use non-\nlinear activation functions for each layer in a neural network is that if we did not, the\nresulting network is exactly equivalent to a single-layer network. Let\u2019s see why this\nis true. Imagine the \ufb01rst two layers of such a network of purely linear layers:\nz[1]=W[1]x+b[1]\nz[2]=W[2]z[1]+b[2]\nWe can rewrite the function that the network is computing as:\nz[2]=W[2]z[1]+b[2]\n=W[2](W[1]x+b[1])+b[2]\n=W[2]W[1]x+W[2]b[1]+b[2]\n=W0x+b0(7.14)\nThis generalizes to any number of layers. So without non-linear activation functions,\na multilayer network is just a notational variant of a single layer network with a\ndifferent set of weights, and we lose all the representational power of multilayer\nnetworks. 142 CHAPTER 7 \u2022 N EURAL NETWORKS\nReplacing the bias unit In describing networks, we will often use a slightly sim-\npli\ufb01ed notation that represents exactly the same function without referring to an ex-\nplicit bias node b. Instead, we add a dummy node a0to each layer whose value will\nalways be 1. Thus layer 0, the input layer, will have a dummy node a[0]\n0=1, layer 1\nwill have a[1]\n0=1, and so on. This dummy node still has an associated weight, and\nthat weight represents the bias value b. For example instead of an equation like\nh=s(Wx+b) (7.15)\nwe\u2019ll use:\nh=s(Wx) (7.16)\nBut now instead of our vector xhaving n0values: x=x1;:::;xn0, it will have n0+\n1 values, with a new 0th dummy value x0=1:x=x0;:::;xn0. And instead of\ncomputing each hjas follows:\nhj=s n0X\ni=1Wjixi+bj!\n; (7.17)\nwe\u2019ll instead use:\nhj=s n0X\ni=0Wjixi!\n; (7.18)\nwhere the value Wj0replaces what had been bj. Fig. 7.9 shows a visualization.\nx1x2xn0\u2026\u2026+1b\u2026UWh1y1y2yn2h2h3hn1\nx1x2xn0\u2026\u2026x0=1\u2026UWh1y1y2yn2h2h3hn1\n(a) (b)\nFigure 7.9 Replacing the bias node (shown in a) with x0(b).\nWe\u2019ll continue showing the bias as bwhen we go over the learning algorithm\nin Section 7.5, but then we\u2019ll switch to this simpli\ufb01ed notation without explicit bias\nterms for the rest of the book.\n7.4 Feedforward networks for NLP: Classi\ufb01cation\nLet\u2019s see how to apply feedforward networks to NLP tasks! In this section we\u2019ll\nlook at classi\ufb01cation tasks like sentiment analysis; in the next section we\u2019ll introduce\nneural language modeling. 7.4 \u2022 F EEDFORWARD NETWORKS FOR NLP: C LASSIFICATION 143\nLet\u2019s begin with a simple 2-layer sentiment classi\ufb01er. You might imagine taking\nour logistic regression classi\ufb01er from Chapter 5, which corresponds to a 1-layer net-\nwork, and just adding a hidden layer. The input element xicould be scalar features\nlike those in Fig. 5.2, e.g., x1= count(words2doc), x2= count(positive lexicon\nwords2doc), x3= 1 if \u201cno\u201d2doc, and so on. And the output layer \u02c6ycould have\ntwo nodes (one each for positive and negative), or 3 nodes (positive, negative, neu-\ntral), in which case \u02c6y1would be the estimated probability of positive sentiment, \u02c6y2\nthe probability of negative and \u02c6y3the probability of neutral. The resulting equations\nwould be just what we saw above for a 2-layer network (as always, we\u2019ll continue\nto use the sto stand for any non-linearity, whether sigmoid, ReLU or other).\nx= [x1;x2;:::xN](each xiis a hand-designed feature)\nh=s(Wx+b)\nz=Uh\n\u02c6y=softmax (z) (7.19)\nFig. 7.10 shows a sketch of this architecture. As we mentioned earlier, adding this\nhidden layer to our logistic regression classi\ufb01er allows the network to represent the\nnon-linear interactions between features. This alone might give us a better sentiment\nclassi\ufb01er.\nUW[n\u2a091]Hidden layerOutput layersoftmax[dh\u2a09n][dh\u2a091][3\u2a09dh]Input wordsp(+)h1h2h3hdh\u2026y1^y2^y3^xhyInput layer n=3 features[3\u2a091]x1x2x3dessertwasgreatpositive lexiconwords = 1count of \u201cno\u201d = 0wordcount=3p(-)p(neut)\nFigure 7.10 Feedforward network sentiment analysis using traditional hand-built features\nof the input text.\nMost applications of neural networks for NLP do something different, however.\nInstead of using hand-built human-engineered features as the input to our classi\ufb01er,\nwe draw on deep learning\u2019s ability to learn features from the data by representing\nwords as embeddings, like the word2vec or GloVe embeddings we saw in Chapter 6.\nThere are various ways to represent an input for classi\ufb01cation. One simple baseline\nis to apply some sort of pooling function to the embeddings of all the words in the pooling\ninput. For example, for a text with ninput words/tokens w1;:::;wn, we can turn the\nnembeddings e(w1);:::;e(wn)(each of dimensionality d) into a single embedding\nalso of dimensionality dby just summing the embeddings, or by taking their mean\n(summing and then dividing by n):\nxmean=1\nnnX\ni=1e(wi) (7.20) 144 CHAPTER 7 \u2022 N EURAL NETWORKS\nThere are many other options, like taking the element-wise max. The element-wise\nmax of a set of nvectors is a new vector whose kth element is the max of the kth\nelements of all the nvectors. Here are the equations for this classi\ufb01er assuming\nmean pooling; the architecture is sketched in Fig. 7.11:\nx=mean(e(w1);e(w2);:::;e(wn))\nh=s(Wx+b)\nz=Uh\n\u02c6y=softmax (z) (7.21)\nUW[d\u2a091]Hidden layerOutput layersoftmax[dh\u2a09d][dh\u2a091][3\u2a09dh]Input wordsp(+)embedding for\u201cgreat\u201dembedding for\u201cdessert\u201dh1h2h3hdh\u2026y1^y2^y3^xhyInput layer pooled embedding[3\u2a091]pooling+dessertwasgreatembedding for\u201cwas\u201dp(-)p(neut)\nFigure 7.11 Feedforward network sentiment analysis using a pooled embedding of the in-\nput words.\nWhile Eq. 7.21 shows how to classify a single example x, in practice we want\nto ef\ufb01ciently classify an entire test set of mexamples. We do this by vectorizing\nthe process, just as we saw with logistic regression; instead of using for-loops to go\nthrough each example, we\u2019ll use matrix multiplication to do the entire computation\nof an entire test set at once. First, we pack all the input feature vectors for each input\nxinto a single input matrix X, with each row ia row vector consisting of the pooled\nembedding for input example x(i)(i.e., the vector x(i)). If the dimensionality of our\npooled input embedding is d,Xwill be a matrix of shape [m\u0002d].\nWe will then need to slightly modify Eq. 7.21. Xis of shape [m\u0002d]andWis of\nshape [dh\u0002d], so we\u2019ll have to reorder how we multiply XandWand transpose W\nso they correctly multiply to yield a matrix Hof shape [m\u0002dh].1The bias vector b\nfrom Eq. 7.21 of shape [1\u0002dh]will now have to be replicated into a matrix of shape\n[m\u0002dh]. We\u2019ll need to similarly reorder the next step and transpose U. Finally, our\noutput matrix \u02c6Ywill be of shape [m\u00023](or more generally [m\u0002do], where dois\nthe number of output classes), with each row iof our output matrix \u02c6Yconsisting of\nthe output vector \u02c6y(i).\u2018 Here are the \ufb01nal equations for computing the output class\n1Note that we could have kept the original order of our products if we had instead made our input\nmatrix Xrepresent each input as a column vector instead of a row vector, making it of shape [d\u0002m]. But\nrepresenting inputs as row vectors is convenient and common in neural network models. 7.5 \u2022 T RAINING NEURAL NETS 145\ndistribution for an entire test set:\nH=s(XW|+b)\nZ=HU|\n\u02c6Y=softmax (Z) (7.22)\nThe idea of using word2vec or GloVe embeddings as our input representation\u2014\nand more generally the idea of relying on another algorithm to have already learned\nan embedding representation for our input words\u2014is called pretraining . Using pretraining\npretrained embedding representations, whether simple static word embeddings like\nword2vec or the much more powerful contextual embeddings we\u2019ll introduce in\nChapter 11, is one of the central ideas of deep learning. (It\u2019s also possible, how-\never, to train the word embeddings as part of an NLP task; we\u2019ll talk about how to\ndo this in Section 7.7 in the context of the neural language modeling task.)\n7.5 Training Neural Nets\nA feedforward neural net is an instance of supervised machine learning in which we\nknow the correct output yfor each observation x. What the system produces, via\nEq. 7.13, is \u02c6 y, the system\u2019s estimate of the true y. The goal of the training procedure\nis to learn parameters W[i]andb[i]for each layer ithat make \u02c6 yfor each training\nobservation as close as possible to the true y.\nIn general, we do all this by drawing on the methods we introduced in Chapter 5\nfor logistic regression, so the reader should be comfortable with that chapter before\nproceeding.\nFirst, we\u2019ll need a loss function that models the distance between the system\noutput and the gold output, and it\u2019s common to use the loss function used for logistic\nregression, the cross-entropy loss .\nSecond, to \ufb01nd the parameters that minimize this loss function, we\u2019ll use the\ngradient descent optimization algorithm introduced in Chapter 5.\nThird, gradient descent requires knowing the gradient of the loss function, the\nvector that contains the partial derivative of the loss function with respect to each\nof the parameters. In logistic regression, for each observation we could directly\ncompute the derivative of the loss function with respect to an individual worb. But\nfor neural networks, with millions of parameters in many layers, it\u2019s much harder to\nsee how to compute the partial derivative of some weight in layer 1 when the loss\nis attached to some much later layer. How do we partial out the loss over all those\nintermediate layers? The answer is the algorithm called error backpropagation or\nbackward differentiation .\n7.5.1 Loss function\nThecross-entropy loss that is used in neural networks is the same one we saw forcross-entropy\nloss\nlogistic regression. If the neural network is being used as a binary classi\ufb01er, with\nthe sigmoid at the \ufb01nal layer, the loss function is the same logistic regression loss\nwe saw in Eq. 5.23:\nLCE(\u02c6y;y) =\u0000logp(yjx) =\u0000[ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)] (7.23)\nIf we are using the network to classify into 3 or more classes, the loss function is\nexactly the same as the loss for multinomial regression that we saw in Chapter 5 on 146 CHAPTER 7 \u2022 N EURAL NETWORKS\npage 97. Let\u2019s brie\ufb02y summarize the explanation here for convenience. First, when\nwe have more than 2 classes we\u2019ll need to represent both yand\u02c6yas vectors. Let\u2019s\nassume we\u2019re doing hard classi\ufb01cation , where only one class is the correct one.\nThe true label yis then a vector with Kelements, each corresponding to a class,\nwithyc=1 if the correct class is c, with all other elements of ybeing 0. Recall that\na vector like this, with one value equal to 1 and the rest 0, is called a one-hot vector .\nAnd our classi\ufb01er will produce an estimate vector with Kelements \u02c6y, each element\n\u02c6ykof which represents the estimated probability p(yk=1jx).\nThe loss function for a single example xis the negative sum of the logs of the K\noutput classes, each weighted by their probability yk:\nLCE(\u02c6y;y) =\u0000KX\nk=1yklog\u02c6yk (7.24)\nWe can simplify this equation further; let\u2019s \ufb01rst rewrite the equation using the func-\ntion 1fgwhich evaluates to 1 if the condition in the brackets is true and to 0 oth-\nerwise. This makes it more obvious that the terms in the sum in Eq. 7.24 will be 0\nexcept for the term corresponding to the true class for which yk=1:\nLCE(\u02c6y;y) =\u0000KX\nk=11fyk=1glog\u02c6yk\nIn other words, the cross-entropy loss is simply the negative log of the output proba-\nbility corresponding to the correct class, and we therefore also call this the negative\nlog likelihood loss :negative log\nlikelihood loss\nLCE(\u02c6y;y) =\u0000log\u02c6yc(where cis the correct class) (7.25)\nPlugging in the softmax formula from Eq. 7.9, and with Kthe number of classes:\nLCE(\u02c6y;y) =\u0000logexp(zc)PK\nj=1exp(zj)(where cis the correct class) (7.26)\n7.5.2 Computing the Gradient\nHow do we compute the gradient of this loss function? Computing the gradient\nrequires the partial derivative of the loss function with respect to each parameter.\nFor a network with one weight layer and sigmoid output (which is what logistic\nregression is), we could simply use the derivative of the loss that we used for logistic\nregression in Eq. 7.27 (and derived in Section 5.10):\n\u00b6LCE(\u02c6y;y)\n\u00b6wj= ( \u02c6y\u0000y)xj\n= (s(w\u0001x+b)\u0000y)xj (7.27)\nOr for a network with one weight layer and softmax output (=multinomial logistic\nregression), we could use the derivative of the softmax loss from Eq. 5.48, shown\nfor a particular weight wkand input xi\n\u00b6LCE(\u02c6y;y)\n\u00b6wk;i=\u0000(yk\u0000\u02c6yk)xi\n=\u0000(yk\u0000p(yk=1jx))xi\n=\u0000 \nyk\u0000exp(wk\u0001x+bk)PK\nj=1exp(wj\u0001x+bj)!\nxi (7.28) 7.5 \u2022 T RAINING NEURAL NETS 147\nBut these derivatives only give correct updates for one weight layer: the last one!\nFor deep networks, computing the gradients for each weight is much more complex,\nsince we are computing the derivative with respect to weight parameters that appear\nall the way back in the very early layers of the network, even though the loss is\ncomputed only at the very end of the network.\nThe solution to computing this gradient is an algorithm called error backprop-\nagation orbackprop (Rumelhart et al., 1986). While backprop was invented spe-error back-\npropagation\ncially for neural networks, it turns out to be the same as a more general procedure\ncalled backward differentiation , which depends on the notion of computation\ngraphs . Let\u2019s see how that works in the next subsection.\n7.5.3 Computation Graphs\nA computation graph is a representation of the process of computing a mathematical\nexpression, in which the computation is broken down into separate operations, each\nof which is modeled as a node in a graph.\nConsider computing the function L(a;b;c) =c(a+2b). If we make each of the\ncomponent addition and multiplication operations explicit, and add names ( dande)\nfor the intermediate outputs, the resulting series of computations is:\nd=2\u0003b\ne=a+d\nL=c\u0003e\nWe can now represent this as a graph, with nodes for each operation, and di-\nrected edges showing the outputs from each operation as the inputs to the next, as\nin Fig. 7.12. The simplest use of computation graphs is to compute the value of\nthe function with some given inputs. In the \ufb01gure, we\u2019ve assumed the inputs a=3,\nb=1,c=\u00002, and we\u2019ve shown the result of the forward pass to compute the re-\nsultL(3;1;\u00002) =\u000010. In the forward pass of a computation graph, we apply each\noperation left to right, passing the outputs of each computation as the input to the\nnext node.\ne=a+dd = 2bL=cea=3b=1c=-2e=5d=2L=-10forward passabc\nFigure 7.12 Computation graph for the function L(a;b;c) =c(a+2b), with values for input\nnodes a=3,b=1,c=\u00002, showing the forward pass computation of L.\n7.5.4 Backward differentiation on computation graphs\nThe importance of the computation graph comes from the backward pass , which\nis used to compute the derivatives that we\u2019ll need for the weight update. In this\nexample our goal is to compute the derivative of the output function Lwith respect 148 CHAPTER 7 \u2022 N EURAL NETWORKS\nto each of the input variables, i.e.,\u00b6L\n\u00b6a,\u00b6L\n\u00b6b, and\u00b6L\n\u00b6c. The derivative\u00b6L\n\u00b6atells us how\nmuch a small change in aaffects L.\nBackwards differentiation makes use of the chain rule in calculus, so let\u2019s re- chain rule\nmind ourselves of that. Suppose we are computing the derivative of a composite\nfunction f(x) =u(v(x)). The derivative of f(x)is the derivative of u(x)with respect\ntov(x)times the derivative of v(x)with respect to x:\nd f\ndx=du\ndv\u0001dv\ndx(7.29)\nThe chain rule extends to more than two functions. If computing the derivative of a\ncomposite function f(x) =u(v(w(x))), the derivative of f(x)is:\nd f\ndx=du\ndv\u0001dv\ndw\u0001dw\ndx(7.30)\nThe intuition of backward differentiation is to pass gradients back from the \ufb01nal\nnode to all the nodes in the graph. Fig. 7.13 shows part of the backward computation\nat one node e. Each node takes an upstream gradient that is passed in from its parent\nnode to the right, and for each of its inputs computes a local gradient (the gradient\nof its output with respect to its input), and uses the chain rule to multiply these two\nto compute a downstream gradient to be passed on to the next earlier node.\nedLed\u2202L\u2202d\u2202L\u2202e=\u2202e\u2202d\u2202L\u2202e\u2202e\u2202dupstream gradientdownstream gradientlocal gradient\nFigure 7.13 Each node (like ehere) takes an upstream gradient, multiplies it by the local\ngradient (the gradient of its output with respect to its input), and uses the chain rule to compute\na downstream gradient to be passed on to a prior node. A node may have multiple local\ngradients if it has multiple inputs.\nLet\u2019s now compute the 3 derivatives we need. Since in the computation graph\nL=ce, we can directly compute the derivative\u00b6L\n\u00b6c:\n\u00b6L\n\u00b6c=e (7.31)\nFor the other two, we\u2019ll need to use the chain rule:\n\u00b6L\n\u00b6a=\u00b6L\n\u00b6e\u00b6e\n\u00b6a\n\u00b6L\n\u00b6b=\u00b6L\n\u00b6e\u00b6e\n\u00b6d\u00b6d\n\u00b6b(7.32)\nEq. 7.32 and Eq. 7.31 thus require \ufb01ve intermediate derivatives:\u00b6L\n\u00b6e,\u00b6L\n\u00b6c,\u00b6e\n\u00b6a,\u00b6e\n\u00b6d, and\n\u00b6d\n\u00b6b, which are as follows (making use of the fact that the derivative of a sum is the 7.5 \u2022 T RAINING NEURAL NETS 149\nsum of the derivatives):\nL=ce:\u00b6L\n\u00b6e=c;\u00b6L\n\u00b6c=e\ne=a+d:\u00b6e\n\u00b6a=1;\u00b6e\n\u00b6d=1\nd=2b:\u00b6d\n\u00b6b=2\nIn the backward pass, we compute each of these partials along each edge of the\ngraph from right to left, using the chain rule just as we did above. Thus we begin by\ncomputing the downstream gradients from node L, which are\u00b6L\n\u00b6eand\u00b6L\n\u00b6c. For node e,\nwe then multiply this upstream gradient\u00b6L\n\u00b6eby the local gradient (the gradient of the\noutput with respect to the input),\u00b6e\n\u00b6dto get the output we send back to node d:\u00b6L\n\u00b6d.\nAnd so on, until we have annotated the graph all the way to all the input variables.\nThe forward pass conveniently already will have computed the values of the forward\nintermediate variables we need (like dande) to compute these derivatives. Fig. 7.14\nshows the backward pass.\ne=d+ad = 2bL=cea=3b=1e=5d=2L=-10 abc\u2202L=5\u2202c\u2202L=-2\u2202e\u2202e=1\u2202d\u2202d=2\u2202b\u2202e=1\u2202abackward passc=-2\u2202L=-2\u2202e\u2202L=5\u2202c\u2202L\u2202d=-2\u2202e\u2202d\u2202L\u2202e=\u2202L\u2202a=-2\u2202e\u2202a\u2202L\u2202e=\u2202L\u2202b=-4\u2202d\u2202b\u2202L\u2202d=\nFigure 7.14 Computation graph for the function L(a;b;c) =c(a+2b), showing the backward pass computa-\ntion of\u00b6L\n\u00b6a,\u00b6L\n\u00b6b, and\u00b6L\n\u00b6c.\nBackward differentiation for a neural network\nOf course computation graphs for real neural networks are much more complex.\nFig. 7.15 shows a sample computation graph for a 2-layer neural network with n0=\n2,n1=2, and n2=1, assuming binary classi\ufb01cation and hence using a sigmoid\noutput unit for simplicity. The function that the computation graph is computing is:\nz[1]=W[1]x+b[1]\na[1]=ReLU (z[1])\nz[2]=W[2]a[1]+b[2]\na[2]=s(z[2])\n\u02c6y=a[2](7.33) 150 CHAPTER 7 \u2022 N EURAL NETWORKS\nFor the backward pass we\u2019ll also need to compute the loss L. The loss function\nfor binary sigmoid output from Eq. 7.23 is\nLCE(\u02c6y;y) =\u0000[ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)] (7.34)\nOur output \u02c6 y=a[2], so we can rephrase this as\nLCE(a[2];y) =\u0000h\nyloga[2]+(1\u0000y)log(1\u0000a[2])i\n(7.35)\nz[2] = +a[2] = \u03c3 a[1] = ReLUz[1] = +b[1]****x1x2a[1] = ReLUz[1] = +b[1]**w[2]11w[1]11w[1]12\nw[1]21w[1]22b[2]w[2]12L (a[2],y)1\n2111\n22\nFigure 7.15 Sample computation graph for a simple 2-layer neural net (= 1 hidden layer) with two input units\nand 2 hidden units. We\u2019ve adjusted the notation a bit to avoid long equations in the nodes by just mentioning\nthe function that is being computed, and the resulting variable name. Thus the * to the right of node w[1]\n11means\nthatw[1]\n11is to be multiplied by x1, and the node z[1]= + means that the value of z[1]is computed by summing\nthe three nodes that feed into it (the two products, and the bias term b[1]\ni).\nThe weights that need updating (those for which we need to know the partial\nderivative of the loss function) are shown in teal. In order to do the backward pass,\nwe\u2019ll need to know the derivatives of all the functions in the graph. We already saw\nin Section 5.10 the derivative of the sigmoid s:\nds(z)\ndz=s(z)(1\u0000s(z)) (7.36)\nWe\u2019ll also need the derivatives of each of the other activation functions. The\nderivative of tanh is:\ndtanh(z)\ndz=1\u0000tanh2(z) (7.37)\nThe derivative of the ReLU is2\ndReLU (z)\ndz=\u001a0f or z<0\n1f or z\u00150(7.38)\n2The derivative is actually unde\ufb01ned at the point z=0, but by convention we treat it as 1. 7.5 \u2022 T RAINING NEURAL NETS 151\nWe\u2019ll give the start of the computation, computing the derivative of the loss function\nLwith respect to z, or\u00b6L\n\u00b6z(and leaving the rest of the computation as an exercise for\nthe reader). By the chain rule:\n\u00b6L\n\u00b6z=\u00b6L\n\u00b6a[2]\u00b6a[2]\n\u00b6z(7.39)\nSo let\u2019s \ufb01rst compute\u00b6L\n\u00b6a[2], taking the derivative of Eq. 7.35, repeated here:\nLCE(a[2];y) =\u0000h\nyloga[2]+(1\u0000y)log(1\u0000a[2])i\n\u00b6L\n\u00b6a[2]=\u0000  \ny\u00b6log(a[2])\n\u00b6a[2]!\n+(1\u0000y)\u00b6log(1\u0000a[2])\n\u00b6a[2]!\n=\u0000\u0012\u0012\ny1\na[2]\u0013\n+(1\u0000y)1\n1\u0000a[2](\u00001)\u0013\n=\u0000\u0012y\na[2]+y\u00001\n1\u0000a[2]\u0013\n(7.40)\nNext, by the derivative of the sigmoid:\n\u00b6a[2]\n\u00b6z=a[2](1\u0000a[2])\nFinally, we can use the chain rule:\n\u00b6L\n\u00b6z=\u00b6L\n\u00b6a[2]\u00b6a[2]\n\u00b6z\n=\u0000\u0012y\na[2]+y\u00001\n1\u0000a[2]\u0013\na[2](1\u0000a[2])\n=a[2]\u0000y (7.41)\nContinuing the backward computation of the gradients (next by passing the gra-\ndients over b[2]\n1and the two product nodes, and so on, back to all the teal nodes), is\nleft as an exercise for the reader.\n7.5.5 More details on learning\nOptimization in neural networks is a non-convex optimization problem, more com-\nplex than for logistic regression, and for that and other reasons there are many best\npractices for successful learning.\nFor logistic regression we can initialize gradient descent with all the weights and\nbiases having the value 0. In neural networks, by contrast, we need to initialize the\nweights with small random numbers. It\u2019s also helpful to normalize the input values\nto have 0 mean and unit variance.\nVarious forms of regularization are used to prevent over\ufb01tting. One of the most\nimportant is dropout : randomly dropping some units and their connections from dropout\nthe network during training (Hinton et al. 2012, Srivastava et al. 2014). At each\niteration of training (whenever we update parameters, i.e. each mini-batch if we are\nusing mini-batch gradient descent), we repeatedly choose a probability pand for\neach unit we replace its output with zero with probability p(and renormalize the\nrest of the outputs from that layer). 152 CHAPTER 7 \u2022 N EURAL NETWORKS\nTuning of hyperparameters is also important. The parameters of a neural net- hyperparameter\nwork are the weights Wand biases b; those are learned by gradient descent. The\nhyperparameters are things that are chosen by the algorithm designer; optimal val-\nues are tuned on a devset rather than by gradient descent learning on the training\nset. Hyperparameters include the learning rate h, the mini-batch size, the model\narchitecture (the number of layers, the number of hidden nodes per layer, the choice\nof activation functions), how to regularize, and so on. Gradient descent itself also\nhas many architectural variants such as Adam (Kingma and Ba, 2015).\nFinally, most modern neural networks are built using computation graph for-\nmalisms that make it easy and natural to do gradient computation and parallelization\non vector-based GPUs (Graphic Processing Units). PyTorch (Paszke et al., 2017)\nand TensorFlow (Abadi et al., 2015) are two of the most popular. The interested\nreader should consult a neural network textbook for further details; some sugges-\ntions are at the end of the chapter.\n7.6 Feedforward Neural Language Modeling\nAs our second application of feedforward networks, let\u2019s consider language mod-\neling : predicting upcoming words from prior words. Neural language modeling\u2014\nbased on the transformer architecture that we will see in Chapter 9\u2014is the algorithm\nthat underlies all of modern NLP. In this section and the next we\u2019ll introduce a sim-\npler version of neural language models for feedforward networks, an algorithm \ufb01rst\nintroduced by Bengio et al. (2003). The feedforward language model introduces\nmany of the important concepts of neural language modeling, concepts we\u2019ll return\nto as we describe more powerful models in Chapter 8 and Chapter 9.\nNeural language models have many advantages over the n-gram language mod-\nels of Chapter 3. Compared to n-gram models, neural language models can handle\nmuch longer histories, can generalize better over contexts of similar words, and are\nmore accurate at word-prediction. On the other hand, neural net language models\nare much more complex, are slower and need more energy to train, and are less inter-\npretable than n-gram models, so for some smaller tasks an n-gram language model\nis still the right tool.\nA feedforward neural language model (LM) is a feedforward network that takes\nas input at time ta representation of some number of previous words ( wt\u00001;wt\u00002,\netc.) and outputs a probability distribution over possible next words. Thus\u2014like the\nn-gram LM\u2014the feedforward neural LM approximates the probability of a word\ngiven the entire prior context P(wtjw1:t\u00001)by approximating based on the N\u00001\nprevious words:\nP(wtjw1;:::; wt\u00001)\u0019P(wtjwt\u0000N+1;:::; wt\u00001) (7.42)\nIn the following examples we\u2019ll use a 4-gram example, so we\u2019ll show a neural net to\nestimate the probability P(wt=ijwt\u00003;wt\u00002;wt\u00001).\nNeural language models represent words in this prior context by their embed-\ndings , rather than just by their word identity as used in n-gram language models.\nUsing embeddings allows neural language models to generalize better to unseen\ndata. For example, suppose we\u2019ve seen this sentence in training:\nI have to make sure that the cat gets fed. 7.6 \u2022 F EEDFORWARD NEURAL LANGUAGE MODELING 153\nbut have never seen the words \u201cgets fed\u201d after the word \u201cdog\u201d. Our test set has the\npre\ufb01x \u201cI forgot to make sure that the dog gets\u201d. What\u2019s the next word? An n-gram\nlanguage model will predict \u201cfed\u201d after \u201cthat the cat gets\u201d, but not after \u201cthat the dog\ngets\u201d. But a neural LM, knowing that \u201ccat\u201d and \u201cdog\u201d have similar embeddings, will\nbe able to generalize from the \u201ccat\u201d context to assign a high enough probability to\n\u201cfed\u201d even after seeing \u201cdog\u201d.\n7.6.1 Forward inference in the neural language model\nLet\u2019s walk through forward inference ordecoding for neural language models.forward\ninference\nForward inference is the task, given an input, of running a forward pass on the\nnetwork to produce a probability distribution over possible outputs, in this case next\nwords.\nWe \ufb01rst represent each of the Nprevious words as a one-hot vector of length\njVj, i.e., with one dimension for each word in the vocabulary. A one-hot vector is one-hot vector\na vector that has one element equal to 1\u2014in the dimension corresponding to that\nword\u2019s index in the vocabulary\u2014 while all the other elements are set to zero. Thus\nin a one-hot representation for the word \u201ctoothpaste\u201d, supposing it is V5, i.e., index\n5 in the vocabulary, x5=1, and xi=08i6=5, as shown here:\n[0 0 0 0 1 0 0 ... 0 0 0 0]\n1 2 3 4 5 6 7 ... ... |V|\nThe feedforward neural language model (sketched in Fig. 7.17) has a moving\nwindow that can see N words into the past. We\u2019ll let N equal 3, so the 3 words\nwt\u00001,wt\u00002, and wt\u00003are each represented as a one-hot vector. We then multiply\nthese one-hot vectors by the embedding matrix E. The embedding weight matrix E\nhas a column for each word, each a column vector of ddimensions, and hence has\ndimensionality d\u0002jVj. Multiplying by a one-hot vector that has only one non-zero\nelement xi=1 simply selects out the relevant column vector for word i, resulting in\nthe embedding for word i, as shown in Fig. 7.16.\nE|V|d1|V|d1=\u271555e5\nFigure 7.16 Selecting the embedding vector for word V5by multiplying the embedding\nmatrix Ewith a one-hot vector with a 1 in index 5.\nThe 3 resulting embedding vectors are concatenated to produce e, the embedding\nlayer. This is followed by a hidden layer and an output layer whose softmax produces\na probability distribution over words. For example y42, the value of output node 42,\nis the probability of the next word wtbeing V42, the vocabulary word with index 42\n(which is the word \u2018\ufb01sh\u2019 in our example).\nHere\u2019s the algorithm in detail for our mini example:\n1.Select three embeddings from E : Given the three previous words, we look\nup their indices, create 3 one-hot vectors, and then multiply each by the em-\nbedding matrix E. Consider wt\u00003. The one-hot vector for \u2018for\u2019 (index 35) is\nmultiplied by the embedding matrix E, to give the \ufb01rst part of the \ufb01rst hidden 154 CHAPTER 7 \u2022 N EURAL NETWORKS\nUWembedding layer3d\u2a091hiddenlayeroutput layersoftmaxdh\u2a093ddh\u2a091|V|\u2a09dhinput layerone-hot vectorsE\n|V|\u2a093d\u2a09|V|p(do|\u2026)p(aardvark|\u2026)\np(zebra|\u2026)p(fish|\u2026)\n|V|\u2a091EEh1h2y1\nh3hdh\u2026\u2026y34\ny|V|\u2026001001|V|35\n001001|V|451001001|V|9920\n0\u2026\u2026y42y35102^^^\n^^hexyforallthe?thanksand\u2026wt-3wt-2wt-1wt\u2026\nFigure 7.17 Forward inference in a feedforward neural language model. At each timestep\ntthe network computes a d-dimensional embedding for each context word (by multiplying a\none-hot vector by the embedding matrix E), and concatenates the 3 resulting embeddings to\nget the embedding layer e. The embedding vector eis multiplied by a weight matrix Wand\nthen an activation function is applied element-wise to produce the hidden layer h, which is\nthen multiplied by another weight matrix U. Finally, a softmax output layer predicts at each\nnode ithe probability that the next word wtwill be vocabulary word Vi.\nlayer, the embedding layer . Since each column of the input matrix Eis anembedding\nlayer\nembedding for a word, and the input is a one-hot column vector xifor word\nVi, the embedding layer for input wwill be Exi=ei, the embedding for word\ni. We now concatenate the three embeddings for the three context words to\nproduce the embedding layer e.\n2.Multiply by W : We multiply by W(and add b) and pass through the ReLU\n(or other) activation function to get the hidden layer h.\n3.Multiply by U :his now multiplied by U\n4.Apply softmax : After the softmax, each node iin the output layer estimates\nthe probability P(wt=ijwt\u00001;wt\u00002;wt\u00003)\nIn summary, the equations for a neural language model with a window size of 3,\ngiven one-hot input vectors for each input context word, are:\ne= [Ext\u00003;Ext\u00002;Ext\u00001]\nh=s(We+b)\nz=Uh\n\u02c6y=softmax (z) (7.43)\nNote that we formed the embedding layer eby concatenating the 3 embeddings\nfor the three context vectors; we\u2019ll often use semicolons to mean concatenation of\nvectors. 7.7 \u2022 T RAINING THE NEURAL LANGUAGE MODEL 155\n7.7 Training the neural language model\nThe high-level intuition of training neural language models, whether the simple\nfeedforward language models we describe here or the more powerful transformer\nlanguage models of Chapter 9, is the idea of self-training orself-supervision that self-training\nwe saw in Chapter 6 for learning word representations. In self-training for language\nmodeling, we take a corpus of text as training material and at each time step task\nthe model to predict the next word. At \ufb01rst it will do poorly at this task, but since\nin each case we know the correct answer (it\u2019s the next word in the corpus!) we can\neasily train it to be better at predicting the correct next word. We call such a model\nself-supervised because we don\u2019t have to add any special gold labels to the data;\nthe natural sequence of words is its own supervision! We simply train the model to\nminimize the error in predicting the true next word in the training sequence.\nIn practice, training the model means setting the parameters q=E;W;U;b. For\nsome tasks, it\u2019s ok to freeze the embedding layer Ewith initial word2vec values. freeze\nFreezing means we use word2vec or some other pretraining algorithm to compute\nthe initial embedding matrix E, and then hold it constant while we only modify W,\nU, and b, i.e., we don\u2019t update Eduring language model training. However, often\nwe\u2019d like to learn the embeddings simultaneously with training the network. This is\nuseful when the task the network is designed for (like sentiment classi\ufb01cation, trans-\nlation, or parsing) places strong constraints on what makes a good representation for\nwords.\nLet\u2019s see how to train the entire model including E, i.e. to set all the parameters\nq=E;W;U;b. We\u2019ll do this via gradient descent (Fig. 5.6), using error backprop-\nagation on the computation graph to compute the gradient. Training thus not only\nsets the weights WandUof the network, but also as we\u2019re predicting upcoming\nwords, we\u2019re learning the embeddings Efor each word that best predict upcoming\nwords.\nFig. 7.18 shows the set up for a window size of N=3 context words. The input x\nconsists of 3 one-hot vectors, fully connected to the embedding layer via 3 instanti-\nations of the embedding matrix E. We don\u2019t want to learn separate weight matrices\nfor mapping each of the 3 previous words to the projection layer. We want one single\nembedding dictionary Ethat\u2019s shared among these three. That\u2019s because over time,\nmany different words will appear as wt\u00002orwt\u00001, and we\u2019d like to just represent\neach word with one vector, whichever context position it appears in. Recall that the\nembedding weight matrix Ehas a column for each word, each a column vector of d\ndimensions, and hence has dimensionality d\u0002jVj.\nGenerally training proceeds by taking as input a very long text, concatenating all\nthe sentences, starting with random weights, and then iteratively moving through the\ntext predicting each word wt. At each word wt, we use the cross-entropy (negative\nlog likelihood) loss. Recall that the general form for this (repeated from Eq. 7.25)\nis:\nLCE(\u02c6y;y) =\u0000log \u02c6yi;(where iis the correct class) (7.44)\nFor language modeling, the classes are the words in the vocabulary, so \u02c6 yihere means\nthe probability that the model assigns to the correct next word wt:\nLCE=\u0000logp(wtjwt\u00001;:::;wt\u0000n+1) (7.45)\nThe parameter update for stochastic gradient descent for this loss from step stos+1 156 CHAPTER 7 \u2022 N EURAL NETWORKS\nUWembedding layer3d\u2a091hiddenlayeroutput layersoftmaxdh\u2a093ddh\u2a091|V|\u2a09dhinput layerone-hot vectorsE\n|V|\u2a093d\u2a09|V|p(do|\u2026)p(aardvark|\u2026)\np(zebra|\u2026)p(fish|\u2026)\n|V|\u2a091EEh1h2y1\nh3hdh\u2026\u2026y34\ny|V|\u2026001001|V|35\n001001|V|451001001|V|9920\n0\u2026\u2026y42y35102^^^\n^^hexyforallthefishthanksand\u2026wt-3wt-2wt-1wt\u2026L = \u2212log P(fish | for, all, the)wt=fish\nFigure 7.18 Learning all the way back to embeddings. Again, the embedding matrix Eis\nshared among the 3 context words.\nis then:\nqs+1=qs\u0000h\u00b6[\u0000logp(wtjwt\u00001;:::;wt\u0000n+1)]\n\u00b6q(7.46)\nThis gradient can be computed in any standard neural network framework which\nwill then backpropagate through q=E;W;U;b.\nTraining the parameters to minimize loss will result both in an algorithm for\nlanguage modeling (a word predictor) but also a new set of embeddings Ethat can\nbe used as word representations for other tasks.\n7.8 Summary\n\u2022 Neural networks are built out of neural units , originally inspired by biological\nneurons but now simply an abstract computational device.\n\u2022 Each neural unit multiplies input values by a weight vector, adds a bias, and\nthen applies a non-linear activation function like sigmoid, tanh, or recti\ufb01ed\nlinear unit.\n\u2022 In a fully-connected ,feedforward network, each unit in layer iis connected\nto each unit in layer i+1, and there are no cycles.\n\u2022 The power of neural networks comes from the ability of early layers to learn\nrepresentations that can be utilized by later layers in the network.\n\u2022 Neural networks are trained by optimization algorithms like gradient de-\nscent .\n\u2022Error backpropagation , backward differentiation on a computation graph ,\nis used to compute the gradients of the loss function for a network. BIBLIOGRAPHICAL AND HISTORICAL NOTES 157\n\u2022Neural language models use a neural network as a probabilistic classi\ufb01er, to\ncompute the probability of the next word given the previous nwords.\n\u2022 Neural language models can use pretrained embeddings , or can learn embed-\ndings from scratch in the process of language modeling.\nBibliographical and Historical Notes\nThe origins of neural networks lie in the 1940s McCulloch-Pitts neuron (McCul-\nloch and Pitts, 1943), a simpli\ufb01ed model of the biological neuron as a kind of com-\nputing element that could be described in terms of propositional logic. By the late\n1950s and early 1960s, a number of labs (including Frank Rosenblatt at Cornell and\nBernard Widrow at Stanford) developed research into neural networks; this phase\nsaw the development of the perceptron (Rosenblatt, 1958), and the transformation\nof the threshold into a bias, a notation we still use (Widrow and Hoff, 1960).\nThe \ufb01eld of neural networks declined after it was shown that a single perceptron\nunit was unable to model functions as simple as XOR (Minsky and Papert, 1969).\nWhile some small amount of work continued during the next two decades, a major\nrevival for the \ufb01eld didn\u2019t come until the 1980s, when practical tools for building\ndeeper networks like error backpropagation became widespread (Rumelhart et al.,\n1986). During the 1980s a wide variety of neural network and related architec-\ntures were developed, particularly for applications in psychology and cognitive sci-\nence (Rumelhart and McClelland 1986b, McClelland and Elman 1986, Rumelhart\nand McClelland 1986a, Elman 1990), for which the term connectionist orparal- connectionist\nlel distributed processing was often used (Feldman and Ballard 1982, Smolensky\n1988). Many of the principles and techniques developed in this period are foun-\ndational to modern work, including the ideas of distributed representations (Hinton,\n1986), recurrent networks (Elman, 1990), and the use of tensors for compositionality\n(Smolensky, 1990).\nBy the 1990s larger neural networks began to be applied to many practical lan-\nguage processing tasks as well, like handwriting recognition (LeCun et al. 1989) and\nspeech recognition (Morgan and Bourlard 1990). By the early 2000s, improvements\nin computer hardware and advances in optimization and training techniques made it\npossible to train even larger and deeper networks, leading to the modern term deep\nlearning (Hinton et al. 2006, Bengio et al. 2007). We cover more related history in\nChapter 8 and Chapter 16.\nThere are a number of excellent books on the subject. Goldberg (2017) has\nsuperb coverage of neural networks for natural language processing. For neural\nnetworks in general see Goodfellow et al. (2016) and Nielsen (2015). 158 CHAPTER 8 \u2022 RNN S AND LSTM S\nCHAPTER\n8RNNs and LSTMs\nTime will explain.\nJane Austen, Persuasion\nLanguage is an inherently temporal phenomenon. Spoken language is a sequence of\nacoustic events over time, and we comprehend and produce both spoken and written\nlanguage as a sequential input stream. The temporal nature of language is re\ufb02ected\nin the metaphors we use; we talk of the \ufb02ow of conversations ,news feeds , and twitter\nstreams , all of which emphasize that language is a sequence that unfolds in time.\nThis temporal nature is re\ufb02ected in some language processing algorithms. For\nexample, the Viterbi algorithm we introduced for HMM part-of-speech tagging pro-\nceeds through the input a word at a time, carrying forward information gleaned along\nthe way. But other machine learning approaches, like those we\u2019ve studied for senti-\nment analysis or other text classi\ufb01cation tasks don\u2019t have this temporal nature \u2013 they\nassume simultaneous access to all aspects of their input.\nThe feedforward networks of Chapter 7 also assumed simultaneous access, al-\nthough they also had a simple model for time. Recall that we applied feedforward\nnetworks to language modeling by having them look only at a \ufb01xed-size window\nof words, and then sliding this window over the input, making independent predic-\ntions along the way. This sliding-window approach is also used in the transformer\narchitecture we will introduce in Chapter 9.\nThis chapter introduces a deep learning architecture that offers an alternative\nway of representing time: recurrent neural networks (RNNs), and their variants like\nLSTMs. RNNs have a mechanism that deals directly with the sequential nature of\nlanguage, allowing them to handle the temporal nature of language without the use of\narbitrary \ufb01xed-sized windows. The recurrent network offers a new way to represent\nthe prior context, in its recurrent connections , allowing the model\u2019s decision to\ndepend on information from hundreds of words in the past. We\u2019ll see how to apply\nthe model to the task of language modeling, to sequence modeling tasks like part-\nof-speech tagging, and to text classi\ufb01cation tasks like sentiment analysis.\n8.1 Recurrent Neural Networks\nA recurrent neural network (RNN) is any network that contains a cycle within its\nnetwork connections, meaning that the value of some unit is directly, or indirectly,\ndependent on its own earlier outputs as an input. While powerful, such networks\nare dif\ufb01cult to reason about and to train. However, within the general class of recur-\nrent networks there are constrained architectures that have proven to be extremely\neffective when applied to language. In this section, we consider a class of recurrent\nnetworks referred to as Elman Networks (Elman, 1990) or simple recurrent net-Elman\nNetworks 8.1 \u2022 R ECURRENT NEURAL NETWORKS 159\nworks . These networks are useful in their own right and serve as the basis for more\ncomplex approaches like the Long Short-Term Memory (LSTM) networks discussed\nlater in this chapter. In this chapter when we use the term RNN we\u2019ll be referring to\nthese simpler more constrained networks (although you will often see the term RNN\nto mean any net with recurrent properties including LSTMs).\nxthtyt\nFigure 8.1 Simple recurrent neural network after Elman (1990). The hidden layer includes\na recurrent connection as part of its input. That is, the activation value of the hidden layer\ndepends on the current input as well as the activation value of the hidden layer from the\nprevious time step.\nFig. 8.1 illustrates the structure of an RNN. As with ordinary feedforward net-\nworks, an input vector representing the current input, xt, is multiplied by a weight\nmatrix and then passed through a non-linear activation function to compute the val-\nues for a layer of hidden units. This hidden layer is then used to calculate a cor-\nresponding output, yt. In a departure from our earlier window-based approach, se-\nquences are processed by presenting one item at a time to the network. We\u2019ll use\nsubscripts to represent time, thus xtwill mean the input vector xat time t. The key\ndifference from a feedforward network lies in the recurrent link shown in the \ufb01gure\nwith the dashed line. This link augments the input to the computation at the hidden\nlayer with the value of the hidden layer from the preceding point in time .\nThe hidden layer from the previous time step provides a form of memory, or\ncontext, that encodes earlier processing and informs the decisions to be made at\nlater points in time. Critically, this approach does not impose a \ufb01xed-length limit\non this prior context; the context embodied in the previous hidden layer can include\ninformation extending back to the beginning of the sequence.\nAdding this temporal dimension makes RNNs appear to be more complex than\nnon-recurrent architectures. But in reality, they\u2019re not all that different. Given an\ninput vector and the values for the hidden layer from the previous time step, we\u2019re\nstill performing the standard feedforward calculation introduced in Chapter 7. To\nsee this, consider Fig. 8.2 which clari\ufb01es the nature of the recurrence and how it\nfactors into the computation at the hidden layer. The most signi\ufb01cant change lies in\nthe new set of weights, U, that connect the hidden layer from the previous time step\nto the current hidden layer. These weights determine how the network makes use of\npast context in calculating the output for the current input. As with the other weights\nin the network, these connections are trained via backpropagation.\n8.1.1 Inference in RNNs\nForward inference (mapping a sequence of inputs to a sequence of outputs) in an\nRNN is nearly identical to what we\u2019ve already seen with feedforward networks. To\ncompute an output ytfor an input xt, we need the activation value for the hidden\nlayer ht. To calculate this, we multiply the input xtwith the weight matrix W, and\nthe hidden layer from the previous time step ht\u00001with the weight matrix U. We\nadd these values together and pass them through a suitable activation function, g,\nto arrive at the activation value for the current hidden layer, ht. Once we have the 160 CHAPTER 8 \u2022 RNN S AND LSTM S\n+UVWyt\nxththt-1\nFigure 8.2 Simple recurrent neural network illustrated as a feedforward network. The hid-\nden layer ht\u00001from the prior time step is multiplied by weight matrix Uand then added to\nthe feedforward component from the current time step.\nvalues for the hidden layer, we proceed with the usual computation to generate the\noutput vector.\nht=g(Uht\u00001+Wx t) (8.1)\nyt=f(Vht) (8.2)\nLet\u2019s refer to the input, hidden and output layer dimensions as din,dh, and dout\nrespectively. Given this, our three parameter matrices are: W2Rdh\u0002din,U2Rdh\u0002dh,\nandV2Rdout\u0002dh.\nWe compute ytvia a softmax computation that gives a probability distribution\nover the possible output classes.\nyt=softmax (Vht) (8.3)\nThe fact that the computation at time trequires the value of the hidden layer from\ntime t\u00001 mandates an incremental inference algorithm that proceeds from the start\nof the sequence to the end as illustrated in Fig. 8.3. The sequential nature of simple\nrecurrent networks can also be seen by unrolling the network in time as is shown in\nFig. 8.4. In this \ufb01gure, the various layers of units are copied for each time step to\nillustrate that they will have differing values over time. However, the various weight\nmatrices are shared across time.\nfunction FORWARD RNN( x,network )returns output sequence y\nh0 0\nfori 1toLENGTH (x)do\nhi g(Uhi\u00001+Wx i)\nyi f(Vhi)\nreturn y\nFigure 8.3 Forward inference in a simple recurrent network. The matrices U,VandWare\nshared across time, while new values for handyare calculated with each time step.\n8.1.2 Training\nAs with feedforward networks, we\u2019ll use a training set, a loss function, and back-\npropagation to obtain the gradients needed to adjust the weights in these recurrent 8.1 \u2022 R ECURRENT NEURAL NETWORKS 161\nUVWUVWUVW\nx1x2x3y1y2y3\nh1h3h2\nh0\nFigure 8.4 A simple recurrent neural network shown unrolled in time. Network layers are recalculated for\neach time step, while the weights U,VandWare shared across all time steps.\nnetworks. As shown in Fig. 8.2, we now have 3 sets of weights to update: W, the\nweights from the input layer to the hidden layer, U, the weights from the previous\nhidden layer to the current hidden layer, and \ufb01nally V, the weights from the hidden\nlayer to the output layer.\nFig. 8.4 highlights two considerations that we didn\u2019t have to worry about with\nbackpropagation in feedforward networks. First, to compute the loss function for\nthe output at time twe need the hidden layer from time t\u00001. Second, the hidden\nlayer at time tin\ufb02uences both the output at time tand the hidden layer at time t+1\n(and hence the output and loss at t+1). It follows from this that to assess the error\naccruing to ht, we\u2019ll need to know its in\ufb02uence on both the current output as well as\nthe ones that follow .\nTailoring the backpropagation algorithm to this situation leads to a two-pass al-\ngorithm for training the weights in RNNs. In the \ufb01rst pass, we perform forward\ninference, computing ht,yt, accumulating the loss at each step in time, saving the\nvalue of the hidden layer at each step for use at the next time step. In the second\nphase, we process the sequence in reverse, computing the required gradients as we\ngo, computing and saving the error term for use in the hidden layer for each step\nbackward in time. This general approach is commonly referred to as backpropaga-\ntion through time (Werbos 1974, Rumelhart et al. 1986, Werbos 1990).backpropaga-\ntion through\ntimeFortunately, with modern computational frameworks and adequate computing\nresources, there is no need for a specialized approach to training RNNs. As illus-\ntrated in Fig. 8.4, explicitly unrolling a recurrent network into a feedforward com-\nputational graph eliminates any explicit recurrences, allowing the network weights\nto be trained directly. In such an approach, we provide a template that speci\ufb01es the\nbasic structure of the network, including all the necessary parameters for the input,\noutput, and hidden layers, the weight matrices, as well as the activation and output\nfunctions to be used. Then, when presented with a speci\ufb01c input sequence, we can\ngenerate an unrolled feedforward network speci\ufb01c to that input, and use that graph 162 CHAPTER 8 \u2022 RNN S AND LSTM S\nto perform forward inference or training via ordinary backpropagation.\nFor applications that involve much longer input sequences, such as speech recog-\nnition, character-level processing, or streaming continuous inputs, unrolling an en-\ntire input sequence may not be feasible. In these cases, we can unroll the input into\nmanageable \ufb01xed-length segments and treat each segment as a distinct training item.\n8.2 RNNs as Language Models\nLet\u2019s see how to apply RNNs to the language modeling task. Recall from Chapter 3\nthat language models predict the next word in a sequence given some preceding\ncontext. For example, if the preceding context is \u201cThanks for all the\u201d and we want\nto know how likely the next word is \u201c\ufb01sh\u201d we would compute:\nP(\ufb01shjThanks for all the )\nLanguage models give us the ability to assign such a conditional probability to every\npossible next word, giving us a distribution over the entire vocabulary. We can also\nassign probabilities to entire sequences by combining these conditional probabilities\nwith the chain rule:\nP(w1:n) =nY\ni=1P(wijw<i)\nThe n-gram language models of Chapter 3 compute the probability of a word given\ncounts of its occurrence with the n\u00001 prior words. The context is thus of size n\u00001.\nFor the feedforward language models of Chapter 7, the context is the window size.\nRNN language models (Mikolov et al., 2010) process the input sequence one\nword at a time, attempting to predict the next word from the current word and the\nprevious hidden state. RNNs thus don\u2019t have the limited context problem that n-gram\nmodels have, or the \ufb01xed context that feedforward language models have, since the\nhidden state can in principle represent information about all of the preceding words\nall the way back to the beginning of the sequence. Fig. 8.5 sketches this difference\nbetween a FFN language model and an RNN language model, showing that the\nRNN language model uses ht\u00001, the hidden state from the previous time step, as a\nrepresentation of the past context.\n8.2.1 Forward Inference in an RNN language model\nForward inference in a recurrent language model proceeds exactly as described in\nSection 8.1.1. The input sequence X= [x1;:::;xt;:::;xN]consists of a series of words\neach represented as a one-hot vector of size jVj\u00021, and the output prediction, y, is a\nvector representing a probability distribution over the vocabulary. At each step, the\nmodel uses the word embedding matrix Eto retrieve the embedding for the current\nword, multiples it by the weight matrix W, and then adds it to the hidden layer from\nthe previous step (weighted by weight matrix Uto compute a new hidden layer.\nThis hidden layer is then used to generate an output layer which is passed through a\nsoftmax layer to generate a probability distribution over the entire vocabulary. That 8.2 \u2022 RNN S AS LANGUAGE MODELS 163\nVWethtUht-1ethtet-1et-2U       Wa)b)^yt\net-1^ytht-1WWet-2U\nFigure 8.5 Simpli\ufb01ed sketch of two LM architectures moving through a text, showing a\nschematic context of three tokens: (a) a feedforward neural language model which has a \ufb01xed\ncontext input to the weight matrix W, (b) an RNN language model, in which the hidden state\nht\u00001summarizes the prior context.\nis, at time t:\net=Ext (8.4)\nht=g(Uht\u00001+We t) (8.5)\n^ yt=softmax (Vht) (8.6)\nWhen we do language modeling with RNNs (and we\u2019ll see this again in Chapter 9\nwith transformers), it\u2019s convenient to make the assumption that the embedding di-\nmension deand the hidden dimension dhare the same. So we\u2019ll just call both of\nthese the model dimension d. So the embedding matrix Eis of shape [d\u0002jVj], and\nxtis a one-hot vector of shape [jVj\u00021]. The product etis thus of shape [d\u00021].W\nandUare of shape [d\u0002d], sohtis also of shape [d\u00021].Vis of shape [jVj\u0002d],\nso the result of Vhis a vector of shape [jVj\u00021]. This vector can be thought of as\na set of scores over the vocabulary given the evidence provided in h. Passing these\nscores through the softmax normalizes the scores into a probability distribution. The\nprobability that a particular word kin the vocabulary is the next word is represented\nby^ yt[k], the kth component of ^ yt:\nP(wt+1=kjw1;:::; wt) = ^ yt[k] (8.7)\nThe probability of an entire sequence is just the product of the probabilities of each\nitem in the sequence, where we\u2019ll use ^ yi[wi]to mean the probability of the true word\nwiat time step i.\nP(w1:n) =nY\ni=1P(wijw1:i\u00001) (8.8)\n=nY\ni=1^ yi[wi] (8.9)\n8.2.2 Training an RNN language model\nTo train an RNN as a language model, we use the same self-supervision (orself- self-supervision\ntraining ) algorithm we saw in Section 7.7: we take a corpus of text as training 164 CHAPTER 8 \u2022 RNN S AND LSTM S\nInputEmbeddingsSoftmax overVocabulary\nSolongandthanksforlongandthanksforNext wordall\u2026Loss\u2026\u2026RNNhyVh\n<latexit sha1_base64=\"9tru+5ysH1zS9iUXRg/IsnxmpMA=\">AAAB/XicbVDLSsNAFL3xWesr6lKQwSK4sSQi1WXRjcsK9gFNCZPpJB06yYSZiRBCcOOvuBFxo+Av+Av+jUnbTVsPDBzOOcO993gxZ0pb1q+xsrq2vrFZ2apu7+zu7ZsHhx0lEklomwguZM/DinIW0bZmmtNeLCkOPU673viu9LtPVComokedxnQQ4iBiPiNYF5Jrnlw4XATIGWGdpbmbOSHWIxlmXERBnldds2bVrQnQMrFnpAYztFzzxxkKkoQ00oRjpfq2FetBhqVmhNO86iSKxpiMcUCzyfY5OiukIfKFLF6k0USdy+FQqTT0imS5nFr0SvE/r59o/2aQsShONI3IdJCfcKQFKqtAQyYp0TwtCCaSFRsiMsISE10UVp5uLx66TDqXdbtRbzxc1Zq3sxIqcAyncA42XEMT7qEFbSDwAm/wCV/Gs/FqvBsf0+iKMftzBHMwvv8ADJKVcA==</latexit>\u0000log \u02c6ylong<latexit sha1_base64=\"tuzkS/BeX/Xmg79qpWZlpeYDhtE=\">AAAB/HicbVDLSsNAFL3xWesr6lKEwSK4sSQi1WXRjcsK9gFNCZPJpB06mYSZiRBC3PgrbkTcKPgN/oJ/Y9J209YDA4dzznDvPV7MmdKW9WusrK6tb2xWtqrbO7t7++bBYUdFiSS0TSIeyZ6HFeVM0LZmmtNeLCkOPU673viu9LtPVCoWiUedxnQQ4qFgASNYF5Jrnlw4PBoiZ4R1luZu5oRYj2SYYeHnedU1a1bdmgAtE3tGajBDyzV/HD8iSUiFJhwr1betWA8yLDUjnOZVJ1E0xmSMhzSbLJ+js0LyURDJ4gmNJupcDodKpaFXJMvd1KJXiv95/UQHN4OMiTjRVJDpoCDhSEeobAL5TFKieVoQTCQrNkRkhCUmuuirPN1ePHSZdC7rdqPeeLiqNW9nJVTgGE7hHGy4hibcQwvaQOAF3uATvoxn49V4Nz6m0RVj9ucI5mB8/wEiupTp</latexit>\u0000log \u02c6yand<latexit sha1_base64=\"0zdsmbBovZ+hafWZN7Hvufo85tU=\">AAAB/3icbVDLSsNAFJ3UV62vqEs3g0VwY0lEqsuiG5cV7AOaEibTSTN0kgkzN0IIWbjxV9yIuFHwD/wF/8ak7aatBwYO55zh3nu8WHANlvVrVNbWNza3qtu1nd29/QPz8KirZaIo61AppOp7RDPBI9YBDoL1Y8VI6AnW8yZ3pd97YkpzGT1CGrNhSMYR9zklUEiuiS8cIcfYCQhkae5mTkggUGEGAYkmOs9rrlm3GtYUeJXYc1JHc7Rd88cZSZqELAIqiNYD24phmBEFnAqW15xEs5jQCRmzbLp/js8KaYR9qYoXAZ6qCzkSap2GXpEs19PLXin+5w0S8G+GGY/iBFhEZ4P8RGCQuCwDj7hiFERaEEIVLzbENCCKUCgqK0+3lw9dJd3Lht1sNB+u6q3beQlVdIJO0Tmy0TVqoXvURh1E0Qt6Q5/oy3g2Xo1342MWrRjzP8doAcb3H7Aall0=</latexit>\u0000log \u02c6ythanks<latexit sha1_base64=\"D3c31Jvxp3QWPr2h4tzQWmeenDs=\">AAAB/HicbVDLSsNAFL3xWesr6lKEwSK4sSQi1WXRjcsK9gFNCZPppB06yYSZiRBC3PgrbkTcKPgN/oJ/Y9Jm09YDA4dzznDvPV7EmdKW9WusrK6tb2xWtqrbO7t7++bBYUeJWBLaJoIL2fOwopyFtK2Z5rQXSYoDj9OuN7kr/O4TlYqJ8FEnER0EeBQynxGsc8k1Ty4cLkbIGWOdJpmbOgHWYxmkvpBZVnXNmlW3pkDLxC5JDUq0XPPHGQoSBzTUhGOl+rYV6UGKpWaE06zqxIpGmEzwiKbT5TN0lktDlM/LX6jRVJ3L4UCpJPDyZLGbWvQK8T+vH2v/ZpCyMIo1DclskB9zpAUqmkBDJinRPMkJJpLlGyIyxhITnfdVnG4vHrpMOpd1u1FvPFzVmrdlCRU4hlM4BxuuoQn30II2EHiBN/iEL+PZeDXejY9ZdMUo/xzBHIzvP0CJlP0=</latexit>\u0000log \u02c6yfor<latexit sha1_base64=\"PI3y1fb9LhumoVCQRh2+Y84dRkc=\">AAAB/HicbVDLSsNAFL3xWesr6lKEwSK4sSQi1WXRjcsK9gFNCZPppB06yYSZiRBC3PgrbkTcKPgN/oJ/Y9Jm09YDA4dzznDvPV7EmdKW9WusrK6tb2xWtqrbO7t7++bBYUeJWBLaJoIL2fOwopyFtK2Z5rQXSYoDj9OuN7kr/O4TlYqJ8FEnER0EeBQynxGsc8k1Ty4cLkbIGWOdJpmbOgHWYxmkmPMsq7pmzapbU6BlYpekBiVarvnjDAWJAxpqwrFSfduK9CDFUjPCaVZ1YkUjTCZ4RNPp8hk6y6Uh8oXMX6jRVJ3L4UCpJPDyZLGbWvQK8T+vH2v/ZpCyMIo1DclskB9zpAUqmkBDJinRPMkJJpLlGyIyxhITnfdVnG4vHrpMOpd1u1FvPFzVmrdlCRU4hlM4BxuuoQn30II2EHiBN/iEL+PZeDXejY9ZdMUo/xzBHIzvPyumlO8=</latexit>\u0000log \u02c6yall\ne\nFigure 8.6 Training RNNs as language models.\nmaterial and at each time step task the model to predict the next word. We call\nsuch a model self-supervised because we don\u2019t have to add any special gold labels\nto the data; the natural sequence of words is its own supervision! We simply train\nthe model to minimize the error in predicting the true next word in the training\nsequence, using cross-entropy as the loss function. Recall that the cross-entropy\nloss measures the difference between a predicted probability distribution and the\ncorrect distribution.\nLCE=\u0000X\nw2Vyt[w]log\u02c6yt[w] (8.10)\nIn the case of language modeling, the correct distribution ytcomes from knowing the\nnext word. This is represented as a one-hot vector corresponding to the vocabulary\nwhere the entry for the actual next word is 1, and all the other entries are 0. Thus,\nthe cross-entropy loss for language modeling is determined by the probability the\nmodel assigns to the correct next word. So at time tthe CE loss is the negative log\nprobability the model assigns to the next word in the training sequence.\nLCE(\u02c6yt;yt) =\u0000log\u02c6yt[wt+1] (8.11)\nThus at each word position tof the input, the model takes as input the the correct\nword wttogether with ht\u00001, encoding information from the preceding w1:t\u00001, and\nuses them to compute a probability distribution over possible next words so as to\ncompute the model\u2019s loss for the next token wt+1. Then we move to the next word,\nwe ignore what the model predicted for the next word and instead use the correct\nword wt+1along with the prior history encoded to estimate the probability of token\nwt+2. This idea that we always give the model the correct history sequence to predict\nthe next word (rather than feeding the model its best case from the previous time\nstep) is called teacher forcing . teacher forcing\nThe weights in the network are adjusted to minimize the average CE loss over\nthe training sequence via gradient descent. Fig. 8.6 illustrates this training regimen.\n8.2.3 Weight Tying\nCareful readers may have noticed that the input embedding matrix Eand the \ufb01nal\nlayer matrix V, which feeds the output softmax, are quite similar. 8.3 \u2022 RNN S FOR OTHER NLP TASKS 165\nThe columns of Erepresent the word embeddings for each word in the vocab-\nulary learned during the training process with the goal that words that have similar\nmeaning and function will have similar embeddings. And, since when we use RNNs\nfor language modeling we make the assumption that the embedding dimension and\nthe hidden dimension are the same (= the model dimension d), the embedding ma-\ntrixEhas shape [d\u0002jVj]. And the \ufb01nal layer matrix Vprovides a way to score\nthe likelihood of each word in the vocabulary given the evidence present in the \ufb01nal\nhidden layer of the network through the calculation of Vh.Vis of shape [jVj\u0002d].\nThat is, is, the rows of Vare shaped like a transpose of E, meaning that Vprovides\nasecond set of learned word embeddings.\nInstead of having two sets of embedding matrices, language models use a single\nembedding matrix, which appears at both the input and softmax layers. That is,\nwe dispense with Vand use Eat the start of the computation and E|(because the\nshape of Vis the transpose of Eat the end. Using the same matrix (transposed) in\ntwo places is called weight tying .1The weight-tied equations for an RNN language weight tying\nmodel then become:\net=Ext (8.12)\nht=g(Uht\u00001+We t) (8.13)\n^ yt=softmax (E|ht) (8.14)\nIn addition to providing improved model perplexity, this approach signi\ufb01cantly re-\nduces the number of parameters required for the model.\n8.3 RNNs for other NLP tasks\nNow that we\u2019ve seen the basic RNN architecture, let\u2019s consider how to apply it to\nthree types of NLP tasks: sequence classi\ufb01cation tasks like sentiment analysis and\ntopic classi\ufb01cation, sequence labeling tasks like part-of-speech tagging, and text\ngeneration tasks, including with a new architecture called the encoder-decoder .\n8.3.1 Sequence Labeling\nIn sequence labeling, the network\u2019s task is to assign a label chosen from a small\n\ufb01xed set of labels to each element of a sequence, like the part-of-speech tagging and\nnamed entity recognition tasks from Chapter 17. In an RNN approach to sequence\nlabeling, inputs are word embeddings and the outputs are tag probabilities generated\nby a softmax layer over the given tagset, as illustrated in Fig. 8.7.\nIn this \ufb01gure, the inputs at each time step are pretrained word embeddings cor-\nresponding to the input tokens. The RNN block is an abstraction that represents\nan unrolled simple recurrent network consisting of an input layer, hidden layer, and\noutput layer at each time step, as well as the shared U,VandWweight matrices\nthat comprise the network. The outputs of the network at each time step represent\nthe distribution over the POS tagset generated by a softmax layer.\nTo generate a sequence of tags for a given input, we run forward inference over\nthe input sequence and select the most likely tag from the softmax at each step. Since\nwe\u2019re using a softmax layer to generate the probability distribution over the output\ntagset at each time step, we will again employ the cross-entropy loss during training.\n1We also do this for transformers (Chapter 9) where it\u2019s common to call E|theunembedding matrix . 166 CHAPTER 8 \u2022 RNN S AND LSTM S\nJanetwillbackthebillNNDTVBMDNNPArgmax\nEmbeddingsWordsehVhyRNNLayer(s)Softmax overtags\nFigure 8.7 Part-of-speech tagging as sequence labeling with a simple RNN. Pre-trained\nword embeddings serve as inputs and a softmax layer provides a probability distribution over\nthe part-of-speech tags as output at each time step.\n8.3.2 RNNs for Sequence Classi\ufb01cation\nAnother use of RNNs is to classify entire sequences rather than the tokens within\nthem. This is the set of tasks commonly called text classi\ufb01cation , like sentiment\nanalysis or spam detection, in which we classify a text into two or three classes\n(like positive or negative), as well as classi\ufb01cation tasks with a large number of\ncategories, like document-level topic classi\ufb01cation, or message routing for customer\nservice applications.\nTo apply RNNs in this setting, we pass the text to be classi\ufb01ed through the RNN\na word at a time generating a new hidden layer representation at each time step.\nWe can then take the hidden layer for the last token of the text, hn, to constitute a\ncompressed representation of the entire sequence. We can pass this representation\nhnto a feedforward network that chooses a class via a softmax over the possible\nclasses. Fig. 8.8 illustrates this approach.\nx1\nRNNhnx2x3xnSoftmaxFFN\nFigure 8.8 Sequence classi\ufb01cation using a simple RNN combined with a feedforward net-\nwork. The \ufb01nal hidden state from the RNN is used as the input to a feedforward network that\nperforms the classi\ufb01cation.\nNote that in this approach we don\u2019t need intermediate outputs for the words in\nthe sequence preceding the last element. Therefore, there are no loss terms associ- 8.3 \u2022 RNN S FOR OTHER NLP TASKS 167\nated with those elements. Instead, the loss function used to train the weights in the\nnetwork is based entirely on the \ufb01nal text classi\ufb01cation task. The output from the\nsoftmax output from the feedforward classi\ufb01er together with a cross-entropy loss\ndrives the training. The error signal from the classi\ufb01cation is backpropagated all the\nway through the weights in the feedforward classi\ufb01er through, to its input, and then\nthrough to the three sets of weights in the RNN as described earlier in Section 8.1.2.\nThe training regimen that uses the loss from a downstream application to adjust the\nweights all the way through the network is referred to as end-to-end training .end-to-end\ntraining\nAnother option, instead of using just hidden state of the last token hnto represent\nthe whole sequence, is to use some sort of pooling function of all the hidden states pooling\nhifor each word iin the sequence. For example, we can create a representation that\npools all the nhidden states by taking their element-wise mean:\nhmean=1\nnnX\ni=1hi (8.15)\nOr we can take the element-wise max; the element-wise max of a set of nvectors is\na new vector whose kth element is the max of the kth elements of all the nvectors.\nThe long contexts of RNNs makes it quite dif\ufb01cult to successfully backpropagate\nerror all the way through the entire input; we\u2019ll talk about this problem, and some\nstandard solutions, in Section 8.5.\n8.3.3 Generation with RNN-Based Language Models\nRNN-based language models can also be used to generate text. Text generation is\nof enormous practical importance, part of tasks like question answering, machine\ntranslation, text summarization, grammar correction, story generation, and conver-\nsational dialogue; any task where a system needs to produce text, conditioned on\nsome other text. This use of a language model to generate text is one of the areas\nin which the impact of neural language models on NLP has been the largest. Text\ngeneration, along with image generation and code generation, constitute a new area\nof AI that is often called generative AI .\nRecall back in Chapter 3 we saw how to generate text from an n-gram language\nmodel by adapting a sampling technique suggested at about the same time by Claude\nShannon (Shannon, 1951) and the psychologists George Miller and Jennifer Self-\nridge (Miller and Selfridge, 1950). We \ufb01rst randomly sample a word to begin a\nsequence based on its suitability as the start of a sequence. We then continue to\nsample words conditioned on our previous choices until we reach a pre-determined\nlength, or an end of sequence token is generated.\nToday, this approach of using a language model to incrementally generate words\nby repeatedly sampling the next word conditioned on our previous choices is called\nautoregressive generation orcausal LM generation . The procedure is basicallyautoregressive\ngeneration\nthe same as that described on page 43, but adapted to a neural context:\n\u2022 Sample a word in the output from the softmax distribution that results from\nusing the beginning of sentence marker, <s>, as the \ufb01rst input.\n\u2022 Use the word embedding for that \ufb01rst word as the input to the network at the\nnext time step, and then sample the next word in the same fashion.\n\u2022 Continue generating until the end of sentence marker, </s> , is sampled or a\n\ufb01xed length limit is reached.\nTechnically an autoregressive model is a model that predicts a value at time tbased\non a linear function of the previous values at times t\u00001,t\u00002, and so on. Although 168 CHAPTER 8 \u2022 RNN S AND LSTM S\nlanguage models are not linear (since they have many layers of non-linearities), we\nloosely refer to this generation technique as autoregressive generation since the\nword generated at each time step is conditioned on the word selected by the network\nfrom the previous step. Fig. 8.9 illustrates this approach. In this \ufb01gure, the details of\nthe RNN\u2019s hidden layers and recurrent connections are hidden within the blue block.\nThis simple architecture underlies state-of-the-art approaches to applications\nsuch as machine translation, summarization, and question answering. The key to\nthese approaches is to prime the generation component with an appropriate context.\nThat is, instead of simply using <s> to get things started we can provide a richer\ntask-appropriate context; for translation the context is the sentence in the source\nlanguage; for summarization it\u2019s the long text we want to summarize.\nSolong\n<s>and\nSolongand?Sampled WordSoftmaxEmbeddingInput WordRNN\nFigure 8.9 Autoregressive generation with an RNN-based neural language model.\n8.4 Stacked and Bidirectional RNN architectures\nRecurrent networks are quite \ufb02exible. By combining the feedforward nature of un-\nrolled computational graphs with vectors as common inputs and outputs, complex\nnetworks can be treated as modules that can be combined in creative ways. This\nsection introduces two of the more common network architectures used in language\nprocessing with RNNs.\n8.4.1 Stacked RNNs\nIn our examples thus far, the inputs to our RNNs have consisted of sequences of\nword or character embeddings (vectors) and the outputs have been vectors useful for\npredicting words, tags or sequence labels. However, nothing prevents us from using\nthe entire sequence of outputs from one RNN as an input sequence to another one.\nStacked RNNs consist of multiple networks where the output of one layer serves as Stacked RNNs\nthe input to a subsequent layer, as shown in Fig. 8.10.\nStacked RNNs generally outperform single-layer networks. One reason for this\nsuccess seems to be that the network induces representations at differing levels of\nabstraction across layers. Just as the early stages of the human visual system detect 8.4 \u2022 S TACKED AND BIDIRECTIONAL RNN ARCHITECTURES 169\ny1y2y3yn\nx1x2x3xn\nRNN 1\nRNN 2\n RNN 3\nFigure 8.10 Stacked recurrent networks. The output of a lower level serves as the input to\nhigher levels with the output of the last network serving as the \ufb01nal output.\nedges that are then used for \ufb01nding larger regions and shapes, the initial layers of\nstacked networks can induce representations that serve as useful abstractions for\nfurther layers\u2014representations that might prove dif\ufb01cult to induce in a single RNN.\nThe optimal number of stacked RNNs is speci\ufb01c to each application and to each\ntraining set. However, as the number of stacks is increased the training costs rise\nquickly.\n8.4.2 Bidirectional RNNs\nThe RNN uses information from the left (prior) context to make its predictions at\ntime t. But in many applications we have access to the entire input sequence; in\nthose cases we would like to use words from the context to the right of t. One way\nto do this is to run two separate RNNs, one left-to-right, and one right-to-left, and\nconcatenate their representations.\nIn the left-to-right RNNs we\u2019ve discussed so far, the hidden state at a given time\ntrepresents everything the network knows about the sequence up to that point. The\nstate is a function of the inputs x1;:::;xtand represents the context of the network to\nthe left of the current time.\nhf\nt=RNN forward (x1;:::;xt) (8.16)\nThis new notation hf\ntsimply corresponds to the normal hidden state at time t, repre-\nsenting everything the network has gleaned from the sequence so far.\nTo take advantage of context to the right of the current input, we can train an\nRNN on a reversed input sequence. With this approach, the hidden state at time t\nrepresents information about the sequence to the right of the current input:\nhb\nt=RNN backward (xt;:::xn) (8.17)\nHere, the hidden state hb\ntrepresents all the information we have discerned about the\nsequence from tto the end of the sequence.\nAbidirectional RNN (Schuster and Paliwal, 1997) combines two independentbidirectional\nRNN\nRNNs, one where the input is processed from the start to the end, and the other from\nthe end to the start. We then concatenate the two representations computed by the\nnetworks into a single vector that captures both the left and right contexts of an input 170 CHAPTER 8 \u2022 RNN S AND LSTM S\nat each point in time. Here we use either the semicolon \u201d;\u201d or the equivalent symbol\n\bto mean vector concatenation:\nht= [hf\nt;hb\nt]\n=hf\nt\bhb\nt (8.18)\nFig. 8.11 illustrates such a bidirectional network that concatenates the outputs of\nthe forward and backward pass. Other simple ways to combine the forward and\nbackward contexts include element-wise addition or multiplication. The output at\neach step in time thus captures information to the left and to the right of the current\ninput. In sequence labeling applications, these concatenated outputs can serve as the\nbasis for a local labeling decision.\nRNN 2 \nRNN 1x1y2y1y3ynconcatenatedoutputs\nx2x3xn\nFigure 8.11 A bidirectional RNN. Separate models are trained in the forward and backward\ndirections, with the output of each model at each time point concatenated to represent the\nbidirectional state at that time point.\nBidirectional RNNs have also proven to be quite effective for sequence classi\ufb01-\ncation. Recall from Fig. 8.8 that for sequence classi\ufb01cation we used the \ufb01nal hidden\nstate of the RNN as the input to a subsequent feedforward classi\ufb01er. A dif\ufb01culty\nwith this approach is that the \ufb01nal state naturally re\ufb02ects more information about\nthe end of the sentence than its beginning. Bidirectional RNNs provide a simple\nsolution to this problem; as shown in Fig. 8.12, we simply combine the \ufb01nal hidden\nstates from the forward and backward passes (for example by concatenation) and\nuse that as input for follow-on processing.\n8.5 The LSTM\nIn practice, it is quite dif\ufb01cult to train RNNs for tasks that require a network to make\nuse of information distant from the current point of processing. Despite having ac-\ncess to the entire preceding sequence, the information encoded in hidden states tends\nto be fairly local, more relevant to the most recent parts of the input sequence and\nrecent decisions. Yet distant information is critical to many language applications.\nConsider the following example in the context of language modeling. 8.5 \u2022 T HELSTM 171\nRNN 2 \nRNN 1x1x2x3xnhn\u2192h1\u2190hn\u2192SoftmaxFFNh1\u2190\nFigure 8.12 A bidirectional RNN for sequence classi\ufb01cation. The \ufb01nal hidden units from\nthe forward and backward passes are combined to represent the entire sequence. This com-\nbined representation serves as input to the subsequent classi\ufb01er.\n(8.19) The \ufb02ights the airline was canceling were full.\nAssigning a high probability to wasfollowing airline is straightforward since airline\nprovides a strong local context for the singular agreement. However, assigning an\nappropriate probability to were is quite dif\ufb01cult, not only because the plural \ufb02ights\nis quite distant, but also because the singular noun airline is closer in the intervening\ncontext. Ideally, a network should be able to retain the distant information about\nplural \ufb02ights until it is needed, while still processing the intermediate parts of the\nsequence correctly.\nOne reason for the inability of RNNs to carry forward critical information is that\nthe hidden layers, and, by extension, the weights that determine the values in the hid-\nden layer, are being asked to perform two tasks simultaneously: provide information\nuseful for the current decision, and updating and carrying forward information re-\nquired for future decisions.\nA second dif\ufb01culty with training RNNs arises from the need to backpropagate\nthe error signal back through time. Recall from Section 8.1.2 that the hidden layer at\ntime tcontributes to the loss at the next time step since it takes part in that calcula-\ntion. As a result, during the backward pass of training, the hidden layers are subject\nto repeated multiplications, as determined by the length of the sequence. A frequent\nresult of this process is that the gradients are eventually driven to zero, a situation\ncalled the vanishing gradients problem.vanishing\ngradients\nTo address these issues, more complex network architectures have been designed\nto explicitly manage the task of maintaining relevant context over time, by enabling\nthe network to learn to forget information that is no longer needed and to remember\ninformation required for decisions still to come.\nThe most commonly used such extension to RNNs is the long short-term mem-\nory(LSTM) network (Hochreiter and Schmidhuber, 1997). LSTMs divide the con-long short-term\nmemory\ntext management problem into two subproblems: removing information no longer\nneeded from the context, and adding information likely to be needed for later de- 172 CHAPTER 8 \u2022 RNN S AND LSTM S\ncision making. The key to solving both problems is to learn how to manage this\ncontext rather than hard-coding a strategy into the architecture. LSTMs accomplish\nthis by \ufb01rst adding an explicit context layer to the architecture (in addition to the\nusual recurrent hidden layer), and through the use of specialized neural units that\nmake use of gates to control the \ufb02ow of information into and out of the units that\ncomprise the network layers. These gates are implemented through the use of addi-\ntional weights that operate sequentially on the input, and previous hidden layer, and\nprevious context layers.\nThe gates in an LSTM share a common design pattern; each consists of a feed-\nforward layer, followed by a sigmoid activation function, followed by a pointwise\nmultiplication with the layer being gated. The choice of the sigmoid as the activation\nfunction arises from its tendency to push its outputs to either 0 or 1. Combining this\nwith a pointwise multiplication has an effect similar to that of a binary mask. Values\nin the layer being gated that align with values near 1 in the mask are passed through\nnearly unchanged; values corresponding to lower values are essentially erased.\nThe \ufb01rst gate we\u2019ll consider is the forget gate . The purpose of this gate is forget gate\nto delete information from the context that is no longer needed. The forget gate\ncomputes a weighted sum of the previous state\u2019s hidden layer and the current in-\nput and passes that through a sigmoid. This mask is then multiplied element-wise\nby the context vector to remove the information from context that is no longer re-\nquired. Element-wise multiplication of two vectors (represented by the operator \f,\nand sometimes called the Hadamard product ) is the vector of the same dimension\nas the two input vectors, where each element iis the product of element iin the two\ninput vectors:\nft=s(Ufht\u00001+Wfxt) (8.20)\nkt=ct\u00001\fft (8.21)\nThe next task is to compute the actual information we need to extract from the previ-\nous hidden state and current inputs\u2014the same basic computation we\u2019ve been using\nfor all our recurrent networks.\ngt=tanh(Ught\u00001+Wgxt) (8.22)\nNext, we generate the mask for the add gate to select the information to add to the add gate\ncurrent context.\nit=s(Uiht\u00001+Wixt) (8.23)\njt=gt\fit (8.24)\nNext, we add this to the modi\ufb01ed context vector to get our new context vector.\nct=jt+kt (8.25)\nThe \ufb01nal gate we\u2019ll use is the output gate which is used to decide what informa- output gate\ntion is required for the current hidden state (as opposed to what information needs\nto be preserved for future decisions).\not=s(Uoht\u00001+Woxt) (8.26)\nht=ot\ftanh(ct) (8.27)\nFig. 8.13 illustrates the complete computation for a single LSTM unit. Given the 8.5 \u2022 T HELSTM 173\n+\nxtht-1cthtcthtct-1ht-1xttanh\n+\u03c3tanh\u03c3\u03c3+++igf\no\u3f4b\u3f4b\u3f4bLSTMct-1\nFigure 8.13 A single LSTM unit displayed as a computation graph. The inputs to each unit consists of the\ncurrent input, x, the previous hidden state, ht\u00001, and the previous context, ct\u00001. The outputs are a new hidden\nstate, htand an updated context, ct.\nappropriate weights for the various gates, an LSTM accepts as input the context\nlayer, and hidden layer from the previous time step, along with the current input\nvector. It then generates updated context and hidden vectors as output.\nIt is the hidden state, ht, that provides the output for the LSTM at each time step.\nThis output can be used as the input to subsequent layers in a stacked RNN, or at the\n\ufb01nal layer of a network htcan be used to provide the \ufb01nal output of the LSTM.\n8.5.1 Gated Units, Layers and Networks\nThe neural units used in LSTMs are obviously much more complex than those used\nin basic feedforward networks. Fortunately, this complexity is encapsulated within\nthe basic processing units, allowing us to maintain modularity and to easily exper-\niment with different architectures. To see this, consider Fig. 8.14 which illustrates\nthe inputs and outputs associated with each kind of unit.\nAt the far left, (a) is the basic feedforward unit where a single set of weights and\na single activation function determine its output, and when arranged in a layer there\nare no connections among the units in the layer. Next, (b) represents the unit in a\nsimple recurrent network. Now there are two inputs and an additional set of weights\nto go with it. However, there is still a single activation function and output.\nThe increased complexity of the LSTM units is encapsulated within the unit\nitself. The only additional external complexity for the LSTM over the basic recurrent\nunit (b) is the presence of the additional context vector as an input and output.\nThis modularity is key to the power and widespread applicability of LSTM units.\nLSTM units (or other varieties, like GRUs) can be substituted into any of the network\narchitectures described in Section 8.4. And, as with simple RNNs, multi-layered\nnetworks making use of gated units can be unrolled into deep feedforward networks\nand trained in the usual fashion with backpropagation. In practice, therefore, LSTMs\nrather than RNNs have become the standard unit for any modern system that makes\nuse of recurrent networks. 174 CHAPTER 8 \u2022 RNN S AND LSTM S\nh\nxxtxtht-1htht\nct-1ct\nht-1(b)(a)(c)\u2303gza\u2303gzLSTMUnita\nFigure 8.14 Basic neural units used in feedforward, simple recurrent networks (SRN), and\nlong short-term memory (LSTM).\n8.6 Summary: Common RNN NLP Architectures\nWe\u2019ve now introduced the RNN, seen advanced components like stacking multiple\nlayers and using the LSTM version, and seen how the RNN can be applied to various\ntasks. Let\u2019s take a moment to summarize the architectures for these applications.\nFig. 8.15 shows the three architectures we\u2019ve discussed so far: sequence la-\nbeling, sequence classi\ufb01cation, and language modeling. In sequence labeling (for\nexample for part of speech tagging), we train a model to produce a label for each\ninput word or token. In sequence classi\ufb01cation, for example for sentiment analysis,\nwe ignore the output for each token, and only take the value from the end of the\nsequence (and similarly the model\u2019s training signal comes from backpropagation\nfrom that last token). In language modeling, we train the model to predict the next\nword at each token step. In the next section we\u2019ll introduce a fourth architecture, the\nencoder-decoder .\n8.7 The Encoder-Decoder Model with RNNs\nIn this section we introduce a new model, the encoder-decoder model, which is used\nwhen we are taking an input sequence and translating it to an output sequence that is\nof a different length than the input, and doesn\u2019t align with it in a word-to-word way.\nRecall that in the sequence labeling task, we have two sequences, but they are the\nsame length (for example in part-of-speech tagging each token gets an associated\ntag), each input is associated with a speci\ufb01c output, and the labeling for that output\ntakes mostly local information. Thus deciding whether a word is a verb or a noun,\nwe look mostly at the word and the neighboring words.\nBy contrast, encoder-decoder models are used especially for tasks like machine\ntranslation, where the input sequence and output sequence can have different lengths\nand the mapping between a token in the input and a token in the output can be very\nindirect (in some languages the verb appears at the beginning of the sentence; in\nother languages at the end). We\u2019ll introduce machine translation in detail in Chap-\nter 13, but for now we\u2019ll just point out that the mapping for a sentence in English to\na sentence in Tagalog or Yoruba can have very different numbers of words, and the 8.7 \u2022 T HEENCODER -DECODER MODEL WITH RNN S175\n\u2026Encoder RNNDecoder RNNContext\u2026\nx1x2xny1y2ym\u2026RNNx1x2xn\u2026y1y2yn\u2026RNNx1x2xny\n\u2026RNNx1x2xt-1\u2026x2x3xta) sequence labeling b) sequence classification \nc) language modelingd) encoder-decoder\nFigure 8.15 Four architectures for NLP tasks. In sequence labeling (POS or named entity tagging) we map\neach input token xito an output token yi. In sequence classi\ufb01cation we map the entire input sequence to a single\nclass. In language modeling we output the next token conditioned on previous tokens. In the encoder model we\nhave two separate RNN models, one of which maps from an input sequence xto an intermediate representation\nwe call the context , and a second of which maps from the context to an output sequence y.\nwords can be in a very different order.\nEncoder-decoder networks, sometimes called sequence-to-sequence networks,encoder-\ndecoder\nare models capable of generating contextually appropriate, arbitrary length, output\nsequences given an input sequence. Encoder-decoder networks have been applied\nto a very wide range of applications including summarization, question answering,\nand dialogue, but they are particularly popular for machine translation.\nThe key idea underlying these networks is the use of an encoder network that\ntakes an input sequence and creates a contextualized representation of it, often called\nthecontext . This representation is then passed to a decoder which generates a task-\nspeci\ufb01c output sequence. Fig. 8.16 illustrates the architecture.\n\u2026EncoderDecoderContext\u2026\nx1x2xny1y2ym\nFigure 8.16 The encoder-decoder architecture. The context is a function of the hidden\nrepresentations of the input, and may be used by the decoder in a variety of ways.\nEncoder-decoder networks consist of three conceptual components:\n1. An encoder that accepts an input sequence, x1:n, and generates a correspond- 176 CHAPTER 8 \u2022 RNN S AND LSTM S\ning sequence of contextualized representations, h1:n. LSTMs, convolutional\nnetworks, and transformers can all be employed as encoders.\n2. A context vector ,c, which is a function of h1:n, and conveys the essence of\nthe input to the decoder.\n3. A decoder , which accepts cas input and generates an arbitrary length se-\nquence of hidden states h1:m, from which a corresponding sequence of output\nstates y1:m, can be obtained. Just as with encoders, decoders can be realized\nby any kind of sequence architecture.\nIn this section we\u2019ll describe an encoder-decoder network based on a pair of\nRNNs, but we\u2019ll see in Chapter 13 how to apply them to transformers as well. We\u2019ll\nbuild up the equations for encoder-decoder models by starting with the conditional\nRNN language model p(y), the probability of a sequence y.\nRecall that in any language model, we can break down the probability as follows:\np(y) = p(y1)p(y2jy1)p(y3jy1;y2):::p(ymjy1;:::;ym\u00001) (8.28)\nIn RNN language modeling, at a particular time t, we pass the pre\ufb01x of t\u00001\ntokens through the language model, using forward inference to produce a sequence\nof hidden states, ending with the hidden state corresponding to the last word of\nthe pre\ufb01x. We then use the \ufb01nal hidden state of the pre\ufb01x as our starting point to\ngenerate the next token.\nMore formally, if gis an activation function like tanh or ReLU, a function of\nthe input at time tand the hidden state at time t\u00001, and the softmax is over the\nset of possible vocabulary items, then at time tthe output ytand hidden state htare\ncomputed as:\nht=g(ht\u00001;xt) (8.29)\n^ yt=softmax (ht) (8.30)\nWe only have to make one slight change to turn this language model with au-\ntoregressive generation into an encoder-decoder model that is a translation model\nthat can translate from a source text in one language to a target text in a second:\nadd a sentence separation marker at the end of the source text, and then simplysentence\nseparation\nconcatenate the target text.\nLet\u2019s use<s>for our sentence separator token, and let\u2019s think about translating\nan English source text (\u201cthe green witch arrived\u201d), to a Spanish sentence (\u201c lleg\u00b4o\nla bruja verde \u201d (which can be glossed word-by-word as \u2018arrived the witch green\u2019).\nWe could also illustrate encoder-decoder models with a question-answer pair, or a\ntext-summarization pair.\nLet\u2019s use xto refer to the source text (in this case in English) plus the separator\ntoken<s>, and yto refer to the target text y(in this case in Spanish). Then an\nencoder-decoder model computes the probability p(yjx)as follows:\np(yjx) = p(y1jx)p(y2jy1;x)p(y3jy1;y2;x):::p(ymjy1;:::;ym\u00001;x) (8.31)\nFig. 8.17 shows the setup for a simpli\ufb01ed version of the encoder-decoder model\n(we\u2019ll see the full model, which requires the new concept of attention , in the next\nsection).\nFig. 8.17 shows an English source text (\u201cthe green witch arrived\u201d), a sentence\nseparator token ( <s>, and a Spanish target text (\u201c lleg\u00b4o la bruja verde \u201d). To trans-\nlate a source text, we run it through the network performing forward inference to 8.7 \u2022 T HEENCODER -DECODER MODEL WITH RNN S177\nSource TextTarget Text\nhnembeddinglayerhiddenlayer(s)softmaxthegreenlleg\u00f3\nwitcharrived<s>lleg\u00f3la\nlabruja\nbrujaverde\nverde</s>(output of source is ignored)\nSeparator\nFigure 8.17 Translating a single sentence (inference time) in the basic RNN version of encoder-decoder ap-\nproach to machine translation. Source and target sentences are concatenated with a separator token in between,\nand the decoder uses context information from the encoder\u2019s last hidden state.\ngenerate hidden states until we get to the end of the source. Then we begin autore-\ngressive generation, asking for a word in the context of the hidden layer from the\nend of the source input as well as the end-of-sentence marker. Subsequent words\nare conditioned on the previous hidden state and the embedding for the last word\ngenerated.\nLet\u2019s formalize and generalize this model a bit in Fig. 8.18. (To help keep things\nstraight, we\u2019ll use the superscripts eanddwhere needed to distinguish the hidden\nstates of the encoder and the decoder.) The elements of the network on the left\nprocess the input sequence xand comprise the encoder . While our simpli\ufb01ed \ufb01gure\nshows only a single network layer for the encoder, stacked architectures are the\nnorm, where the output states from the top layer of the stack are taken as the \ufb01nal\nrepresentation, and the encoder consists of stacked biLSTMs where the hidden states\nfrom top layers from the forward and backward passes are concatenated to provide\nthe contextualized representations for each time step.\nEncoderDecoder\nhn hd1he3he2he1hd2hd3hd4embeddinglayerhiddenlayer(s)softmaxx1x2y1hdmx3xn<s>y1y2\ny2y3\ny3y4\nym</s>hen = c = hd0(output is ignored during encoding)\nFigure 8.18 A more formal version of translating a sentence at inference time in the basic RNN-based\nencoder-decoder architecture. The \ufb01nal hidden state of the encoder RNN, hen, serves as the context for the\ndecoder in its role as hd\n0in the decoder RNN, and is also made available to each decoder hidden state. 178 CHAPTER 8 \u2022 RNN S AND LSTM S\nThe entire purpose of the encoder is to generate a contextualized representation\nof the input. This representation is embodied in the \ufb01nal hidden state of the encoder,\nhe\nn. This representation, also called cforcontext , is then passed to the decoder.\nThe simplest version of the decoder network would takes this state and use it\njust to initialize the \ufb01rst hidden state of the decoder; the \ufb01rst decoder RNN cell\nwould use cas its prior hidden state hd\n0. The decoder would then autoregressively\ngenerates a sequence of outputs, an element at a time, until an end-of-sequence\nmarker is generated. Each hidden state is conditioned on the previous hidden state\nand the output generated in the previous state.\nAs Fig. 8.18 shows, we do something more complex: we make the context vector\ncavailable to more than just the \ufb01rst decoder hidden state, to ensure that the in\ufb02uence\nof the context vector, c, doesn\u2019t wane as the output sequence is generated. We do\nthis by adding cas a parameter to the computation of the current hidden state. using\nthe following equation:\nhd\nt=g(\u02c6yt\u00001;hd\nt\u00001;c) (8.32)\nNow we\u2019re ready to see the full equations for this version of the decoder in the basic\nencoder-decoder model, with context available at each decoding timestep. Recall\nthatgis a stand-in for some \ufb02avor of RNN and \u02c6 yt\u00001is the embedding for the output\nsampled from the softmax at the previous step:\nc=he\nn\nhd\n0=c\nhd\nt=g(\u02c6yt\u00001;hd\nt\u00001;c)\n^ yt=softmax (hd\nt) (8.33)\nThus ^ ytis a vector of probabilities over the vocabulary, representing the probability\nof each word occurring at time t. To generate text, we sample from this distribution\n^ yt. For example, the greedy choice is simply to choose the most most probable word\nto generate at each timestep. We\u2019ll introduce more sophisticated sampling methods\nin Section 10.2.\n8.7.1 Training the Encoder-Decoder Model\nEncoder-decoder architectures are trained end-to-end. Each training example is a\ntuple of paired strings, a source and a target. Concatenated with a separator token,\nthese source-target pairs can now serve as training data.\nFor MT, the training data typically consists of sets of sentences and their transla-\ntions. These can be drawn from standard datasets of aligned sentence pairs, as we\u2019ll\ndiscuss in Section 13.2.2. Once we have a training set, the training itself proceeds\nas with any RNN-based language model. The network is given the source text and\nthen starting with the separator token is trained autoregressively to predict the next\nword, as shown in Fig. 8.19.\nNote the differences between training (Fig. 8.19) and inference (Fig. 8.17) with\nrespect to the outputs at each time step. The decoder during inference uses its own\nestimated output \u02c6 ytas the input for the next time step xt+1. Thus the decoder will\ntend to deviate more and more from the gold target sentence as it keeps generating\nmore tokens. In training, therefore, it is more common to use teacher forcing in the teacher forcing\ndecoder. Teacher forcing means that we force the system to use the gold target token\nfrom training as the next input xt+1, rather than allowing it to rely on the (possibly\nerroneous) decoder output \u02c6 yt. This speeds up training. 8.8 \u2022 A TTENTION 179\nEncoderDecoder\nembeddinglayerhiddenlayer(s)softmaxthegreenlleg\u00f3\nwitcharrived<s>lleg\u00f3la\nlabruja\nbrujaverde\nverde</s>goldanswersL1 =-log P(y1)\nx1x2x3x4L2 =-log P(y2)L3 =-log P(y3)L4 =-log P(y4)L5 =-log P(y5)per-wordlossy1y2y3y4y5Total loss is the average cross-entropy loss per target word:\nFigure 8.19 Training the basic RNN encoder-decoder approach to machine translation. Note that in the\ndecoder we usually don\u2019t propagate the model\u2019s softmax outputs \u02c6 yt, but use teacher forcing to force each input\nto the correct gold value for training. We compute the softmax output distribution over \u02c6 yin the decoder in order\nto compute the loss at each token, which can then be averaged to compute a loss for the sentence. This loss is\nthen propagated through the decoder parameters and the encoder parameters.\n8.8 Attention\nThe simplicity of the encoder-decoder model is its clean separation of the encoder\u2014\nwhich builds a representation of the source text\u2014from the decoder, which uses this\ncontext to generate a target text. In the model as we\u2019ve described it so far, this\ncontext vector is hn, the hidden state of the last ( nth) time step of the source text.\nThis \ufb01nal hidden state is thus acting as a bottleneck : it must represent absolutely\neverything about the meaning of the source text, since the only thing the decoder\nknows about the source text is what\u2019s in this context vector (Fig. 8.20). Information\nat the beginning of the sentence, especially for long sentences, may not be equally\nwell represented in the context vector.\nEncoderDecoderbottleneckbottleneck\nFigure 8.20 Requiring the context cto be only the encoder\u2019s \ufb01nal hidden state forces all the\ninformation from the entire source sentence to pass through this representational bottleneck.\nThe attention mechanism is a solution to the bottleneck problem, a way ofattention\nmechanism\nallowing the decoder to get information from allthe hidden states of the encoder,\nnot just the last hidden state.\nIn the attention mechanism, as in the vanilla encoder-decoder model, the context\nvector cis a single vector that is a function of the hidden states of the encoder, that\nis,c=f(he\n1:::he\nn). Because the number of hidden states varies with the size of\nthe input, we can\u2019t use the entire set of encoder hidden state vectors directly as the 180 CHAPTER 8 \u2022 RNN S AND LSTM S\ncontext for the decoder.\nThe idea of attention is instead to create the single \ufb01xed-length vector cby taking\na weighted sum of all the encoder hidden states. The weights focus on (\u2018attend\nto\u2019) a particular part of the source text that is relevant for the token the decoder is\ncurrently producing. Attention thus replaces the static context vector with one that\nis dynamically derived from the encoder hidden states, different for each token in\ndecoding.\nThis context vector, ci, is generated anew with each decoding step iand takes\nall of the encoder hidden states into account in its derivation. We then make this\ncontext available during decoding by conditioning the computation of the current\ndecoder hidden state on it (along with the prior hidden state and the previous output\ngenerated by the decoder), as we see in this equation (and Fig. 8.21):\nhd\ni=g(\u02c6yi\u00001;hd\ni\u00001;ci) (8.34)\nhd1hd2hdiy1y2yic1c2ci\u2026\u2026\nFigure 8.21 The attention mechanism allows each hidden state of the decoder to see a\ndifferent, dynamic, context, which is a function of all the encoder hidden states.\nThe \ufb01rst step in computing ciis to compute how much to focus on each encoder\nstate, how relevant each encoder state is to the decoder state captured in hd\ni\u00001. We\ncapture relevance by computing\u2014 at each state iduring decoding\u2014a score(hd\ni\u00001;he\nj)\nfor each encoder state j.\nThe simplest such score, called dot-product attention , implements relevance asdot-product\nattention\nsimilarity: measuring how similar the decoder hidden state is to an encoder hidden\nstate, by computing the dot product between them:\nscore(hd\ni\u00001;he\nj) = hd\ni\u00001\u0001he\nj (8.35)\nThe score that results from this dot product is a scalar that re\ufb02ects the degree of\nsimilarity between the two vectors. The vector of these scores across all the encoder\nhidden states gives us the relevance of each encoder state to the current step of the\ndecoder.\nTo make use of these scores, we\u2019ll normalize them with a softmax to create a\nvector of weights, ai j, that tells us the proportional relevance of each encoder hidden\nstate jto the prior hidden decoder state, hd\ni\u00001.\nai j=softmax (score(hd\ni\u00001;he\nj))\n=exp(score(hd\ni\u00001;he\nj)\nP\nkexp(score(hd\ni\u00001;he\nk))(8.36)\nFinally, given the distribution in a, we can compute a \ufb01xed-length context vector for\nthe current decoder state by taking a weighted average over all the encoder hidden\nstates.\nci=X\njai jhe\nj (8.37) 8.9 \u2022 S UMMARY 181\nWith this, we \ufb01nally have a \ufb01xed-length context vector that takes into account\ninformation from the entire encoder state that is dynamically updated to re\ufb02ect the\nneeds of the decoder at each step of decoding. Fig. 8.22 illustrates an encoder-\ndecoder network with attention, focusing on the computation of one context vector\nci.\nEncoderDecoder\nhdi-1he3he2he1hdihiddenlayer(s)x1x2yix3xnyi-1yiyi+1hen ci.2.1.3.4attentionweightsci-1ci\n<latexit sha1_base64=\"TNdNmv/RIlrhPa6LgQyjjQLqyBA=\">AAACAnicdVDLSsNAFJ3UV62vqCtxM1gEVyHpI9Vd0Y3LCvYBTQyT6bSddvJgZiKUUNz4K25cKOLWr3Dn3zhpK6jogQuHc+7l3nv8mFEhTfNDyy0tr6yu5dcLG5tb2zv67l5LRAnHpIkjFvGOjwRhNCRNSSUjnZgTFPiMtP3xRea3bwkXNAqv5SQmboAGIe1TjKSSPP3AEUngjVIHsXiIvJSOpnB4Q7zR1NOLpmGaVbtqQdOwLbtk24qY5Yp9VoOWsjIUwQINT393ehFOAhJKzJAQXcuMpZsiLilmZFpwEkFihMdoQLqKhiggwk1nL0zhsVJ6sB9xVaGEM/X7RIoCISaBrzoDJIfit5eJf3ndRPZP3ZSGcSJJiOeL+gmDMoJZHrBHOcGSTRRBmFN1K8RDxBGWKrWCCuHrU/g/aZUMyzbKV5Vi/XwRRx4cgiNwAixQA3VwCRqgCTC4Aw/gCTxr99qj9qK9zltz2mJmH/yA9vYJSymYCA==</latexit>Xj\u21b5ijhej\u21b5ij<latexit sha1_base64=\"y8s4mGdpwrGrBnuSR+p1gJJXYdo=\">AAAB/nicdVDJSgNBEO2JW4zbqHjy0hgEL4YeJyQBL0EvHiOYBbIMPT09mTY9C909QhgC/ooXD4p49Tu8+Td2FkFFHxQ83quiqp6bcCYVQh9Gbml5ZXUtv17Y2Nza3jF391oyTgWhTRLzWHRcLClnEW0qpjjtJILi0OW07Y4up377jgrJ4uhGjRPaD/EwYj4jWGnJMQ+Cgedk7NSa9IgXq955MKDOrWMWUQnNAFGpYtfsakUTZNtWGUFrYRXBAg3HfO95MUlDGinCsZRdCyWqn2GhGOF0UuilkiaYjPCQdjWNcEhlP5udP4HHWvGgHwtdkYIz9ftEhkMpx6GrO0OsAvnbm4p/ed1U+bV+xqIkVTQi80V+yqGK4TQL6DFBieJjTTARTN8KSYAFJkonVtAhfH0K/yets5JVKdnX5WL9YhFHHhyCI3ACLFAFdXAFGqAJCMjAA3gCz8a98Wi8GK/z1pyxmNkHP2C8fQICDpWK</latexit>hdi\u00001\u00b7hej\u2026\u2026\nFigure 8.22 A sketch of the encoder-decoder network with attention, focusing on the computation of ci. The\ncontext value ciis one of the inputs to the computation of hd\ni. It is computed by taking the weighted sum of all\nthe encoder hidden states, each weighted by their dot product with the prior decoder hidden state hd\ni\u00001.\nIt\u2019s also possible to create more sophisticated scoring functions for attention\nmodels. Instead of simple dot product attention, we can get a more powerful function\nthat computes the relevance of each encoder hidden state to the decoder hidden state\nby parameterizing the score with its own set of weights, Ws.\nscore(hd\ni\u00001;he\nj) = hd\nt\u00001Wshe\nj\nThe weights Ws, which are then trained during normal end-to-end training, give the\nnetwork the ability to learn which aspects of similarity between the decoder and\nencoder states are important to the current application. This bilinear model also\nallows the encoder and decoder to use different dimensional vectors, whereas the\nsimple dot-product attention requires that the encoder and decoder hidden states\nhave the same dimensionality.\nWe\u2019ll return to the concept of attention when we de\ufb01ne the transformer archi-\ntecture in Chapter 9, which is based on a slight modi\ufb01cation of attention called\nself-attention .\n8.9 Summary\nThis chapter has introduced the concepts of recurrent neural networks and how they\ncan be applied to language problems. Here\u2019s a summary of the main points that we\ncovered:\n\u2022 In simple Recurrent Neural Networks sequences are processed one element at\na time, with the output of each neural unit at time tbased both on the current\ninput at tand the hidden layer from time t\u00001. 182 CHAPTER 8 \u2022 RNN S AND LSTM S\n\u2022 RNNs can be trained with a straightforward extension of the backpropagation\nalgorithm, known as backpropagation through time (BPTT).\n\u2022 Simple recurrent networks fail on long inputs because of problems like van-\nishing gradients ; instead modern systems use more complex gated architec-\ntures such as LSTMs that explicitly decide what to remember and forget in\ntheir hidden and context layers.\n\u2022 Common language-based applications for RNNs include:\n\u2013Probabilistic language modeling: assigning a probability to a sequence,\nor to the next element of a sequence given the preceding words.\n\u2013Auto-regressive generation using a trained language model.\n\u2013Sequence labeling like part-of-speech tagging, where each element of a\nsequence is assigned a label.\n\u2013Sequence classi\ufb01cation, where an entire text is assigned to a category, as\nin spam detection, sentiment analysis or topic classi\ufb01cation.\n\u2013Encoder-decoder architectures, where an input is mapped to an output\nof different length and alignment.\nBibliographical and Historical Notes\nIn\ufb02uential investigations of RNNs were conducted in the context of the Parallel Dis-\ntributed Processing (PDP) group at UC San Diego in the 1980\u2019s. Much of this work\nwas directed at human cognitive modeling rather than practical NLP applications\n(Rumelhart and McClelland 1986c, McClelland and Rumelhart 1986). Models using\nrecurrence at the hidden layer in a feedforward network (Elman networks) were in-\ntroduced by Elman (1990). Similar architectures were investigated by Jordan (1986)\nwith a recurrence from the output layer, and Mathis and Mozer (1995) with the\naddition of a recurrent context layer prior to the hidden layer. The possibility of\nunrolling a recurrent network into an equivalent feedforward network is discussed\nin (Rumelhart and McClelland, 1986c).\nIn parallel with work in cognitive modeling, RNNs were investigated extensively\nin the continuous domain in the signal processing and speech communities (Giles\net al. 1994, Robinson et al. 1996). Schuster and Paliwal (1997) introduced bidirec-\ntional RNNs and described results on the TIMIT phoneme transcription task.\nWhile theoretically interesting, the dif\ufb01culty with training RNNs and manag-\ning context over long sequences impeded progress on practical applications. This\nsituation changed with the introduction of LSTMs in Hochreiter and Schmidhuber\n(1997) and Gers et al. (2000). Impressive performance gains were demonstrated\non tasks at the boundary of signal processing and language processing including\nphoneme recognition (Graves and Schmidhuber, 2005), handwriting recognition\n(Graves et al., 2007) and most signi\ufb01cantly speech recognition (Graves et al., 2013).\nInterest in applying neural networks to practical NLP problems surged with the\nwork of Collobert and Weston (2008) and Collobert et al. (2011). These efforts made\nuse of learned word embeddings, convolutional networks, and end-to-end training.\nThey demonstrated near state-of-the-art performance on a number of standard shared\ntasks including part-of-speech tagging, chunking, named entity recognition and se-\nmantic role labeling without the use of hand-engineered features.\nApproaches that married LSTMs with pretrained collections of word-embeddings\nbased on word2vec (Mikolov et al., 2013a) and GloVe (Pennington et al., 2014) BIBLIOGRAPHICAL AND HISTORICAL NOTES 183\nquickly came to dominate many common tasks: part-of-speech tagging (Ling et al.,\n2015), syntactic chunking (S\u00f8gaard and Goldberg, 2016), named entity recognition\n(Chiu and Nichols, 2016; Ma and Hovy, 2016), opinion mining (Irsoy and Cardie,\n2014), semantic role labeling (Zhou and Xu, 2015a) and AMR parsing (Foland and\nMartin, 2016). As with the earlier surge of progress involving statistical machine\nlearning, these advances were made possible by the availability of training data pro-\nvided by CONLL, SemEval, and other shared tasks, as well as shared resources such\nas Ontonotes (Pradhan et al., 2007b), and PropBank (Palmer et al., 2005).\nThe modern neural encoder-decoder approach was pioneered by Kalchbrenner\nand Blunsom (2013), who used a CNN encoder and an RNN decoder. Cho et al.\n(2014) (who coined the name \u201cencoder-decoder\u201d) and Sutskever et al. (2014) then\nshowed how to use extended RNNs for both encoder and decoder. The idea that a\ngenerative decoder should take as input a soft weighting of the inputs, the central\nidea of attention, was \ufb01rst developed by Graves (2013) in the context of handwriting\nrecognition. Bahdanau et al. (2015) extended the idea, named it \u201cattention\u201d and\napplied it to MT. 184 CHAPTER 9 \u2022 T HETRANSFORMER\nCHAPTER\n9The Transformer\n\u201cThe true art of memory is the art of attention \u201d\nSamuel Johnson, Idler #74 , September 1759\nIn this chapter we introduce the transformer , the standard architecture for build-\ninglarge language models . Transformer-based large language models have com-\npletely changed the \ufb01eld of speech and language processing. Indeed, every subse-\nquent chapter in this textbook will make use of them. We\u2019ll focus for now on left-\nto-right (sometimes called causal or autoregressive) language modeling, in which\nwe are given a sequence of input tokens and predict output tokens one by one by\nconditioning on the prior context.\nThe transformer is a neural network with a speci\ufb01c structure that includes a\nmechanism called self-attention ormulti-head attention .1Attention can be thought\nof as a way to build contextual representations of a token\u2019s meaning by attending to\nand integrating information from surrounding tokens, helping the model learn how\ntokens relate to each other over large spans.\nStackedTransformerBlocksSolongandthanksforlongandthanksforNext tokenall\u2026\u2026\n\u2026\nU\nInput tokensx1x2LanguageModelingHead\nx3x4x5InputEncoding\nE1+\nE2+\nE3+\nE4+\nE5+\u2026\u2026\u2026\u2026\u2026\nU\nU\nU\nU\u2026logitslogitslogitslogitslogits\nFigure 9.1 The architecture of a (left-to-right) transformer, showing how each input token\nget encoded, passed through a set of stacked transformer blocks, and then a language model\nhead that predicts the next token.\nFig. 9.1 sketches the transformer architecture. A transformer has three major\ncomponents. At the center are columns of transformer blocks . Each block is a\nmultilayer network (a multi-head attention layer, feedforward networks and layer\nnormalization steps) that maps an input vector xiin column i(corresponding to input\n1Although multi-head attention developed historically from the RNN attention mechanism (Chapter 8),\nwe\u2019ll de\ufb01ne attention from scratch here for readers who haven\u2019t yet read Chapter 8. 9.1 \u2022 A TTENTION 185\ntoken i) to an output vector hi. The set of nblocks maps an entire context window\nof input vectors (x1;:::;xn)to a window of output vectors (h1;:::;hn)of the same\nlength. A column might contain from 12 to 96 or more stacked blocks.\nThe column of blocks is preceded by the input encoding component, which pro-\ncesses an input token (like the word thanks ) into a contextual vector representation,\nusing an embedding matrix Eand a mechanism for encoding token position. Each\ncolumn is followed by a language modeling head , which takes the embedding out-\nput by the \ufb01nal transformer block, passes it through an unembedding matrix Uand\na softmax over the vocabulary to generate a single token for that column.\nTransformer-based language models are complex, and so the details will unfold\nover the next 5 chapters. In the next sections we\u2019ll introduce multi-head attention,\nthe rest of the transformer block, and the input encoding and language modeling\nhead components. Chapter 10 discusses how language models are pretrained , and\nhow tokens are generated via sampling . Chapter 11 introduces masked language\nmodeling and the BERT family of bidirectional transformer encoder models. Chap-\nter 12 shows how to prompt LLMs to perform NLP tasks by giving instructions and\ndemonstrations, and how to align the model with human preferences. Chapter 13\nwill introduce machine translation with the encoder-decoder architecture.\n9.1 Attention\nRecall from Chapter 6 that for word2vec and other static embeddings, the repre-\nsentation of a word\u2019s meaning is always the same vector irrespective of the context:\nthe wordchicken , for example, is always represented by the same \ufb01xed vector. So\na static vector for the word itmight somehow encode that this is a pronoun used\nfor animals and inanimate entities. But in context ithas a much richer meaning.\nConsideritin one of these two sentences:\n(9.1) The chicken didn\u2019t cross the road because itwas too tired.\n(9.2) The chicken didn\u2019t cross the road because itwas too wide.\nIn (9.1)itis the chicken (i.e., the reader knows that the chicken was tired), while\nin (9.2)itis the road (and the reader knows that the road was wide).2That is, if\nwe are to compute the meaning of this sentence, we\u2019ll need the meaning of itto be\nassociated with the chicken in the \ufb01rst sentence and associated with the road in\nthe second one, sensitive to the context.\nFurthermore, consider reading left to right like a causal language model, pro-\ncessing the sentence up to the word it:\n(9.3) The chicken didn\u2019t cross the road because it\nAt this point we don\u2019t yet know which thing itis going to end up referring to! So a\nrepresentation of itat this point might have aspects of both chicken androad as\nthe reader is trying to guess what happens next.\nThis fact that words have rich linguistic relationships with other words that may\nbe far away pervades language. Consider two more examples:\n(9.4) The keys to the cabinet areon the table.\n(9.5) I walked along the pond , and noticed one of the trees along the bank .\n2We say that in the \ufb01rst example itcorefers with the chicken, and in the second itcorefers with the\nroad; we\u2019ll return to this in Chapter 23. 186 CHAPTER 9 \u2022 T HETRANSFORMER\nIn (9.4), the phrase The keys is the subject of the sentence, and in English and many\nlanguages, must agree in grammatical number with the verb are; in this case both are\nplural. In English we can\u2019t use a singular verb like iswith a plural subject like keys\n(we\u2019ll discuss agreement more in Chapter 18). In (9.5), we know that bank refers\nto the side of a pond or river and not a \ufb01nancial institution because of the context,\nincluding words like pond . (We\u2019ll discuss word senses more in Chapter 11.)\nThe point of all these examples is that these contextual words that help us com-\npute the meaning of words in context can be quite far away in the sentence or para-\ngraph. Transformers can build contextual representations of word meaning, contex-\ntual embeddings , by integrating the meaning of these helpful contextual words. In acontextual\nembeddings\ntransformer, layer by layer, we build up richer and richer contextualized representa-\ntions of the meanings of input tokens. At each layer, we compute the representation\nof a token iby combining information about ifrom the previous layer with infor-\nmation about the neighboring tokens to produce a contextualized representation for\neach word at each position.\nAttention is the mechanism in the transformer that weighs and combines the\nrepresentations from appropriate other tokens in the context from layer k\u00001 to build\nthe representation for tokens in layer k.\nThechickendidn\u2019tcrosstheroadbecauseitwastootiredThechickendidn\u2019tcrosstheroadbecauseitwastootiredLayer k+1Layer kself-attention distributioncolumns corresponding to input tokens\nFigure 9.2 The self-attention weight distribution athat is part of the computation of the\nrepresentation for the word itat layer k+1. In computing the representation for it, we attend\ndifferently to the various words at layer l, with darker shades indicating higher self-attention\nvalues. Note that the transformer is attending highly to the columns corresponding to the\ntokens chicken androad , a sensible result, since at the point where itoccurs, it could plausibly\ncorefers with the chicken or the road, and hence we\u2019d like the representation for itto draw on\nthe representation for these earlier words. Figure adapted from Uszkoreit (2017).\nFig. 9.2 shows a schematic example simpli\ufb01ed from a transformer (Uszkoreit,\n2017). The \ufb01gure describes the situation when the current token is itand we need\nto compute a contextual representation for this token at layer k+1 of the transformer,\ndrawing on the representations (from layer k) of every prior token. The \ufb01gure uses\ncolor to represent the attention distribution over the contextual words: the tokens\nchicken androad both have a high attention weight, meaning that as we are com-\nputing the representation for it, we will draw most heavily on the representation for\nchicken androad . This will be useful in building the \ufb01nal representation for it,\nsinceitwill end up coreferring with either chicken orroad .\nLet\u2019s now turn to how this attention distribution is represented and computed. 9.1 \u2022 A TTENTION 187\n9.1.1 Attention more formally\nAs we\u2019ve said, the attention computation is a way to compute a vector representation\nfor a token at a particular layer of a transformer, by selectively attending to and\nintegrating information from prior tokens at the previous layer. Attention takes an\ninput representation xicorresponding to the input token at position i, and a context\nwindow of prior inputs x1::xi\u00001, and produces an output ai.\nIn causal, left-to-right language models, the context is any of the prior words.\nThat is, when processing xi, the model has access to xias well as the representations\nof all the prior tokens in the context window (context windows consist of thousands\nof tokens) but no tokens after i. (By contrast, in Chapter 11 we\u2019ll generalize attention\nso it can also look ahead to future words.)\nFig. 9.3 illustrates this \ufb02ow of information in an entire causal self-attention layer,\nin which this same attention computation happens in parallel at each token position\ni. Thus a self-attention layer maps input sequences (x1;:::;xn)to output sequences\nof the same length (a1;:::;an).\nattentionattentionSelf-AttentionLayerattentionattentionattentiona1a2a3a4a5x3x4x5x1x2\nFigure 9.3 Information \ufb02ow in causal self-attention. When processing each input xi, the\nmodel attends to all the inputs up to, and including xi.\nSimpli\ufb01ed version of attention At its heart, attention is really just a weighted\nsum of context vectors, with a lot of complications added to how the weights are\ncomputed and what gets summed. For pedagogical purposes let\u2019s \ufb01rst describe a\nsimpli\ufb01ed intuition of attention, in which the attention output aiat token position i\nis simply the weighted sum of all the representations xj, for all j\u0014i; we\u2019ll use ai j\nto mean how much xishould contribute to aj:\nSimpli\ufb01ed version: ai=X\nj\u0014iai jxj (9.6)\nEach ai jis a scalar used for weighing the value of input xjwhen summing up\nthe inputs to compute ai. How shall we compute this aweighting? In attention we\nweight each prior embedding proportionally to how similar it is to the current token\ni. So the output of attention is a sum of the embeddings of prior tokens weighted\nby their similarity with the current token embedding. We compute similarity scores\nviadot product , which maps two vectors into a scalar value ranging from \u0000\u00a5to\n\u00a5. The larger the score, the more similar the vectors that are being compared. We\u2019ll\nnormalize these scores with a softmax to create the vector of weights ai j;j\u0014i.\nSimpli\ufb01ed Version: score(xi;xj) = xi\u0001xj (9.7)\nai j=softmax (score(xi;xj))8j\u0014i(9.8)\nThus in Fig. 9.3 we compute a3by computing three scores: x3\u0001x1,x3\u0001x2andx3\u0001x3,\nnormalizing them by a softmax, and using the resulting probabilities as weights\nindicating each of their proportional relevance to the current position i. Of course, 188 CHAPTER 9 \u2022 T HETRANSFORMER\nthe softmax weight will likely be highest for xi, since xiis very similar to itself,\nresulting in a high dot product. But other context words may also be similar to i, and\nthe softmax will also assign some weight to those words. Then we use these weights\nas the avalues in Eq. 9.6 to compute the weighted sum that is our a3.\nThe simpli\ufb01ed attention in equations 9.6 \u2013 9.8 demonstrates the attention-based\napproach to computing ai: compare the xito prior vectors, normalize those scores\ninto a probability distribution used to weight the sum of the prior vector. But now\nwe\u2019re ready to remove the simpli\ufb01cations.\nA single attention head using query, key, and value matrices Now that we\u2019ve\nseen a simple intuition of attention, let\u2019s introduce the actual attention head , the attention head\nversion of attention that\u2019s used in transformers. (The word head is often used in head\ntransformers to refer to speci\ufb01c structured layers). The attention head allows us to\ndistinctly represent three different roles that each input embedding plays during the\ncourse of the attention process:\n\u2022 As the current element being compared to the preceding inputs. We\u2019ll refer to\nthis role as a query . query\n\u2022 In its role as a preceding input that is being compared to the current element\nto determine a similarity weight. We\u2019ll refer to this role as a key. key\n\u2022 And \ufb01nally, as a value of a preceding element that gets weighted and summed value\nup to compute the output for the current element.\nTo capture these three different roles, transformers introduce weight matrices\nWQ,WK, and WV. These weights will project each input vector xiinto a represen-\ntation of its role as a key, query, or value:\nqi=xiWQ;ki=xiWK;vi=xiWV(9.9)\nGiven these projections, when we are computing the similarity of the current ele-\nment xiwith some prior element xj, we\u2019ll use the dot product between the current\nelement\u2019s query vector qiand the preceding element\u2019s keyvector kj. Furthermore,\nthe result of a dot product can be an arbitrarily large (positive or negative) value, and\nexponentiating large values can lead to numerical issues and loss of gradients during\ntraining. To avoid this, we scale the dot product by a factor related to the size of the\nembeddings, via diving by the square root of the dimensionality of the query and\nkey vectors ( dk). We thus replace the simpli\ufb01ed Eq. 9.7 with Eq. 9.11. The ensuing\nsoftmax calculation resulting in ai jremains the same, but the output calculation for\naiis now based on a weighted sum over the value vectors v(Eq. 9.13).\nHere\u2019s a \ufb01nal set of equations for computing self-attention for a single self-\nattention output vector aifrom a single input vector xi. This version of attention\ncomputes aiby summing the values of the prior elements, each weighted by the\nsimilarity of its keyto the query from the current element:\nqi=xiWQ;kj=xjWK;vj=xjWV(9.10)\nscore(xi;xj) =qi\u0001kjpdk(9.11)\nai j=softmax (score(xi;xj))8j\u0014i (9.12)\nai=X\nj\u0014iai jvj (9.13)\nWe illustrate this in Fig. 9.4 for the case of calculating the value of the third output\na3in a sequence. 9.1 \u2022 A TTENTION 189\n6. Sum the weighted value vectors4. Turn into \ud835\udefci,j weights via softmaxa3\n1. Generate key, query, value vectors2. Compare x3\u2019s query withthe keys for x1, x2, and x3Output of self-attention\nWk\nWvWqx1kqvx3kqvx2kqv\n\u00d7\u00d7\nWkWkWqWqWvWv5. Weigh each value vector\u00f7\u221adk3. Divide score by \u221adk\u00f7\u221adk\u00f7\u221adk\ud835\udefc3,1\ud835\udefc3,2\ud835\udefc3,3\nFigure 9.4 Calculating the value of a3, the third element of a sequence using causal (left-\nto-right) self-attention.\nLet\u2019s talk shapes. The input to attention xiand the output from attention aiboth\nhave the same dimensionality 1 \u0002d(We often call dthemodel dimensionality ,\nand indeed as we\u2019ll discuss in Section 9.2 the output hiof each transformer block,\nas well as the intermediate vectors inside the transformer block also have the same\ndimensionality 1\u0002d.).\nWe\u2019ll have a dimension dkfor the key and query vectors. The query vector and\nthe key vector are both dimensionality 1 \u0002dk, so we can take their dot product qi\u0001kj.\nWe\u2019ll have a separate dimension dvfor the value vectors. The transform matrix WQ\nhas shape [d\u0002dk],WKis[d\u0002dk], and WVis[d\u0002dv]. In the original transformer\nwork (Vaswani et al., 2017), dwas 512, dkanddvwere both 64.\nMulti-head Attention Equations 9.11-9.13 describe a single attention head . But\nactually, transformers use multiple attention heads. The intuition is that each head\nmight be attending to the context for different purposes: heads might be special-\nized to represent different linguistic relationships between context elements and the\ncurrent token, or to look for particular kinds of patterns in the context.\nSo in multi-head attention we have hseparate attention heads that reside inmulti-head\nattention\nparallel layers at the same depth in a model, each with its own set of parameters that\nallows the head to model different aspects of the relationships among inputs. Thus\neach head iin a self-attention layer has its own set of key, query and value matrices:\nWKi,WQiandWVi. These are used to project the inputs into separate key, value,\nand query embeddings for each head.\nWhen using multiple heads the model dimension dis still used for the input\nand output, the key and query embeddings have dimensionality dk, and the value\nembeddings are of dimensionality dv(again, in the original transformer paper dk=\ndv=64,h=8, and d=512). Thus for each head i, we have weight layers WQiof\nshape [d\u0002dk],WKiof shape [d\u0002dk], and WViof shape [d\u0002dv].\nBelow are the equations for attention augmented with multiple heads; Fig. 9.5 190 CHAPTER 9 \u2022 T HETRANSFORMER\nshows an intuition.\nqc\ni=xiWQc;kc\nj=xjWKc;vc\nj=xjWVc;8c1\u0014c\u0014h (9.14)\nscorec(xi;xj) =qc\ni\u0001kc\njpdk(9.15)\nac\ni j=softmax (scorec(xi;xj))8j\u0014i(9.16)\nheadc\ni=X\nj\u0014iac\ni jvc\nj (9.17)\nai= (head1\bhead2:::\bheadh)WO(9.18)\nMultiHeadAttention (xi;[x1;\u0001\u0001\u0001;xN]) = ai (9.19)\nThe output of each of the hheads is of shape 1 \u0002dv, and so the output of the\nmulti-head layer with hheads consists of hvectors of shape 1\u0002dv. These are con-\ncatenated to produce a single output with dimensionality 1 \u0002hdv. Then we use yet\nanother linear projection WO2Rhdv\u0002dto reshape it, resulting in the multi-head\nattention vector aiwith the correct output shape [1xd]at each input i.\naixi-1xixi-2xi-3WK1Head 1WV1WQ1\u2026\u2026WK2Head 2WV2WQ2WK8Head 8WV8WQ8aiWO  [hdv x d][1 x dv ][1 x d]\n[1 x d][1 x hdv ]Project down to dConcatenate OutputsEach headattends di\ufb00erentlyto context\u2026[1 x dv ]\nFigure 9.5 The multi-head attention computation for input xi, producing output ai. A multi-head attention\nlayer has hheads, each with its own key, query and value weight matrices. The outputs from each of the heads\nare concatenated and then projected down to d, thus producing an output of the same size as the input.\n9.2 Transformer Blocks\nThe self-attention calculation lies at the core of what\u2019s called a transformer block,\nwhich, in addition to the self-attention layer, includes three other kinds of layers: (1)\na feedforward layer, (2) residual connections, and (3) normalizing layers (colloqui-\nally called \u201clayer norm\u201d).\nFig. 9.6 illustrates a transformer block, sketching a common way of thinking\nabout the block that is called the residual stream (Elhage et al., 2021). In the resid- residual stream\nual stream viewpoint, we consider the processing of an individual token ithrough\nthe transformer block as a single stream of d-dimensional representations for token\nposition i. This residual stream starts with the original input vector, and the various 9.2 \u2022 T RANSFORMER BLOCKS 191\nLayer Norm\nxi+hi-1\nLayer NormMultiHeadAttentionFeedforward\nxi-1xi+1hihi+1\n+\u2026\u2026ResidualStream\nFigure 9.6 The architecture of a transformer block showing the residual stream . This\n\ufb01gure shows the prenorm version of the architecture, in which the layer norms happen before\nthe attention and feedforward layers rather than after.\ncomponents read their input from the residual stream and add their output back into\nthe stream.\nThe input at the bottom of the stream is an embedding for a token, which has\ndimensionality d. This initial embedding gets passed up (by residual connections ),\nand is progressively added to by the other components of the transformer: the at-\ntention layer that we have seen, and the feedforward layer that we will introduce.\nBefore the attention and feedforward layer is a computation called the layer norm .\nThus the initial vector is passed through a layer norm and attention layer, and\nthe result is added back into the stream, in this case to the original input vector\nxi. And then this summed vector is again passed through another layer norm and a\nfeedforward layer, and the output of those is added back into the residual, and we\u2019ll\nusehito refer to the resulting output of the transformer block for token i. (In earlier\ndescriptions the residual stream was often described using a different metaphor as\nresidual connections that add the input of a component to its output, but the residual\nstream is a more perspicuous way of visualizing the transformer.)\nWe\u2019ve already seen the attention layer, so let\u2019s now introduce the feedforward\nand layer norm computations in the context of processing a single input xiat token\nposition i.\nFeedforward layer The feedforward layer is a fully-connected 2-layer network,\ni.e., one hidden layer, two weight matrices, as introduced in Chapter 7. The weights\nare the same for each token position i, but are different from layer to layer. It\nis common to make the dimensionality dffof the hidden layer of the feedforward\nnetwork be larger than the model dimensionality d. (For example in the original\ntransformer model, d=512 and dff=2048.)\nFFN(xi) =ReLU (xiW1+b1)W2+b2 (9.20)\nLayer Norm At two stages in the transformer block we normalize the vector (Ba\net al., 2016). This process, called layer norm (short for layer normalization), is one layer norm 192 CHAPTER 9 \u2022 T HETRANSFORMER\nof many forms of normalization that can be used to improve training performance\nin deep neural networks by keeping the values of a hidden layer in a range that\nfacilitates gradient-based training.\nLayer norm is a variation of the z-score from statistics, applied to a single vec-\ntor in a hidden layer. That is, the term layer norm is a bit confusing; layer norm\nisnotapplied to an entire transformer layer, but just to the embedding vector of a\nsingle token. Thus the input to layer norm is a single vector of dimensionality d\nand the output is that vector normalized, again of dimensionality d. The \ufb01rst step in\nlayer normalization is to calculate the mean, m, and standard deviation, s, over the\nelements of the vector to be normalized. Given an embedding vector xof dimen-\nsionality d, these values are calculated as follows.\nm=1\nddX\ni=1xi (9.21)\ns=vuut1\nddX\ni=1(xi\u0000m)2(9.22)\nGiven these values, the vector components are normalized by subtracting the mean\nfrom each and dividing by the standard deviation. The result of this computation is\na new vector with zero mean and a standard deviation of one.\n^ x=(x\u0000m)\ns(9.23)\nFinally, in the standard implementation of layer normalization, two learnable param-\neters, gandb, representing gain and offset values, are introduced.\nLayerNorm (x) =g(x\u0000m)\ns+b (9.24)\nPutting it all together The function computed by a transformer block can be ex-\npressed by breaking it down with one equation for each component computation,\nusing t(of shape [1\u0002d]) to stand for transformer and superscripts to demarcate\neach computation inside the block:\nt1\ni=LayerNorm (xi) (9.25)\nt2\ni=MultiHeadAttention (t1\ni;\u0002\nx1\n1;\u0001\u0001\u0001;x1\nN\u0003\n) (9.26)\nt3\ni=t2\ni+xi (9.27)\nt4\ni=LayerNorm (t3\ni) (9.28)\nt5\ni=FFN(t4\ni) (9.29)\nhi=t5\ni+t3\ni (9.30)\nNotice that the only component that takes as input information from other tokens\n(other residual streams) is multi-head attention, which (as we see from (9.27)) looks\nat all the neighboring tokens in the context. The output from attention, however, is\nthen added into this token\u2019s embedding stream. In fact, Elhage et al. (2021) show that\nwe can view attention heads as literally moving information from the residual stream\nof a neighboring token into the current stream. The high-dimensional embedding\nspace at each position thus contains information about the current token and about\nneighboring tokens, albeit in different subspaces of the vector space. Fig. 9.7 shows\na visualization of this movement. 9.3 \u2022 P ARALLELIZING COMPUTATION USING A SINGLE MATRIX X 193\nToken Aresidual streamToken Bresidual stream\nFigure 9.7 An attention head can move information from token A\u2019s residual stream into\ntoken B\u2019s residual stream.\nCrucially, the input and output dimensions of transformer blocks are matched so\nthey can be stacked. Each token vector xiat the input to the block has dimensionality\nd, and the output hialso has dimensionality d. Transformers for large language\nmodels stack many of these blocks, from 12 layers (used for the T5 or GPT-3-small\nlanguage models) to 96 layers (used for GPT-3 large), to even more for more recent\nmodels. We\u2019ll come back to this issue of stacking in a bit.\nEquation (9.27) and following are just the equation for a single transformer\nblock, but the residual stream metaphor goes through all the transformer layers,\nfrom the \ufb01rst transformer blocks to the 12th, in a 12-layer transformer. At the ear-\nlier transformer blocks, the residual stream is representing the current token. At the\nhighest transformer blocks, the residual stream is usually representing the following\ntoken, since at the very end it\u2019s being trained to predict the next token.\nOnce we stack many blocks, there is one more requirement: at the very end of\nthe last (highest) transformer block, there is a single extra layer norm that is run on\nthe last hiof each token stream (just below the language model head layer that we\nwill de\ufb01ne soon).3\n9.3 Parallelizing computation using a single matrix X\nThis description of multi-head attention and the rest of the transformer block has\nbeen from the perspective of computing a single output at a single time step iin\na single residual stream. But as we pointed out earlier, the attention computation\nperformed for each token to compute aiis independent of the computation for each\nother token, and that\u2019s also true for all the computation in the transformer block\ncomputing hifrom the input xi. That means we can easily parallelize the entire\ncomputation, taking advantage of ef\ufb01cient matrix multiplication routines.\nWe do this by packing the input embeddings for the Ntokens of the input se-\nquence into a single matrix Xof size [N\u0002d]. Each row of Xis the embedding of\none token of the input. Transformers for large language models commonly have an\ninput length N= 1K, 2K, or as many as 32K tokens (or more), so Xtypically has be-\ntween 1K and 32K rows, each of the dimensionality of the embedding d(the model\n3Note that we are using the most common current transformer architecture, which is called the prenorm\narchitecture. The original de\ufb01nition of the transformer in Vaswani et al. (2017) used an alternative archi-\ntecture called the postnorm transformer in which the layer norm happens after the attention and FFN\nlayers; it turns out moving the layer norm beforehand works better, but does require this one extra layer\nat the end. 194 CHAPTER 9 \u2022 T HETRANSFORMER\ndimension).\nParallelizing attention Let\u2019s \ufb01rst see this for a single attention head and then turn\nto multiple heads, and then add in the rest of the components in the transformer\nblock. For one head we multiply Xby the key, query, and value matrices WQof\nshape [d\u0002dk],WKof shape [d\u0002dk], andWVof shape [d\u0002dv], to produce matrices\nQof shape [N\u0002dk],K2RN\u0002dk, and V2RN\u0002dv, containing all the key, query, and\nvalue vectors:\nQ=XWQ;K=XWK;V=XWV(9.31)\nGiven these matrices we can compute all the requisite query-key comparisons simul-\ntaneously by multiplying QandK|in a single matrix multiplication. The product is\nof shape N\u0002N, visualized in Fig. 9.8.\nq1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3NNq1\u2022k2q1\u2022k3q1\u2022k4q2\u2022k3q2\u2022k4q3\u2022k4\nFigure 9.8 The N\u0002NQK|matrix showing how it computes all qi\u0001kjcomparisons in a\nsingle matrix multiple.\nOnce we have this QK|matrix, we can very ef\ufb01ciently scale these scores, take\nthe softmax, and then multiply the result by Vresulting in a matrix of shape N\u0002d:\na vector embedding representation for each token in the input. We\u2019ve reduced the\nentire self-attention step for an entire sequence of Ntokens for one head to the\nfollowing computation:\nA=softmax\u0012\nmask\u0012QK|\npdk\u0013\u0013\nV (9.32)\nMasking out the future You may have noticed that we introduced a mask function\nin Eq. 9.32 above. This is because the self-attention computation as we\u2019ve described\nit has a problem: the calculation in QK|results in a score for each query value\nto every key value, including those that follow the query . This is inappropriate in\nthe setting of language modeling: guessing the next word is pretty simple if you\nalready know it! To \ufb01x this, the elements in the upper-triangular portion of the\nmatrix are zeroed out (set to \u0000\u00a5), thus eliminating any knowledge of words that\nfollow in the sequence. This is done in practice by adding a mask matrix Min\nwhich Mi j=\u0000\u00a58j>i(i.e. for the upper-triangular portion) and Mi j=0 otherwise.\nFig. 9.9 shows the resulting masked QK|matrix. (we\u2019ll see in Chapter 11 how to\nmake use of words in the future for tasks that need it).\nFig. 9.10 shows a schematic of all the computations for a single attention head\nparallelized in matrix form.\nFig. 9.8 and Fig. 9.9 also make it clear that attention is quadratic in the length\nof the input, since at each layer we need to compute dot products between each pair\nof tokens in the input. This makes it expensive to compute attention over very long\ndocuments (like entire novels). Nonetheless modern large language models manage\nto use quite long contexts of thousands or tens of thousands of tokens. 9.3 \u2022 P ARALLELIZING COMPUTATION USING A SINGLE MATRIX X 195\nq1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3NN\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\nFigure 9.9 TheN\u0002NQK|matrix showing the qi\u0001kjvalues, with the upper-triangle por-\ntion of the comparisons matrix zeroed out (set to \u0000\u00a5, which the softmax will turn to zero).\nq1q2q3q4k1k2k3k4QKTQKTv1v2v3v4Vq2\u2022k2q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k2q3\u2022k3\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221eq1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3q1\u2022k2q2\u2022k3q1\u2022k3q3\u2022k4q2\u2022k4q1\u2022k4x=QKT maskedmask=q1\u2022k1q2\u2022k1q4\u2022k1q3\u2022k1q1\u2022k1q1\u2022k1=xa1a2a3a4AQuery Token 1Query Token 2Query Token 3Query Token 4QInput Token 1Input Token 2Input Token 3Input Token 4XxWQ=Value Token 1Value Token 2Value Token 3Value Token 4VxWV=Input Token 1Input Token 2Input Token 3Input Token 4XKey Token 1Key Token 2Key Token 3Key Token 4KxWK=Input Token 1Input Token 2Input Token 3Input Token 4X\nN x dkdk x NN x NN x NN x dvN x dvd x dkd x dkd x dvN x dN x dkN x dN x dkN x dN x dv\nFigure 9.10 Schematic of the attention computation for a single attention head in parallel. The \ufb01rst row shows\nthe computation of the Q,K, and Vmatrices. The second row shows the computation of QKT, the masking\n(the softmax computation and the normalizing by dimensionality are not shown) and then the weighted sum of\nthe value vectors to get the \ufb01nal attention vectors.\nParallelizing multi-head attention In multi-head attention, as with self-attention,\nthe input and output have the model dimension d, the key and query embeddings\nhave dimensionality dk, and the value embeddings are of dimensionality dv(again,\nin the original transformer paper dk=dv=64,h=8, and d=512). Thus for each\nhead i, we have weight layers WQi2Rd\u0002dk,WKi2Rd\u0002dk, and WVi2Rd\u0002dv, and\nthese get multiplied by the inputs packed into Xto produce Q2RN\u0002dk,K2RN\u0002dk,\nandV2RN\u0002dv. The output of each of the hheads is of shape N\u0002dv, and so the\noutput of the multi-head layer with hheads consists of hmatrices of shape N\u0002dv. To\nmake use of these matrices in further processing, they are concatenated to produce\na single output with dimensionality N\u0002hdv. Finally, we use yet another linear\nprojection WO2Rhdv\u0002d, that reshape it to the original output dimension for each\ntoken. Multiplying the concatenated N\u0002hdvmatrix output by WO2Rhdv\u0002dyields 196 CHAPTER 9 \u2022 T HETRANSFORMER\nthe self-attention output Aof shape [ N\u0002d].\nQi=XWQi;Ki=XWKi;Vi=XWVi(9.33)\nhead i=SelfAttention (Qi;Ki;Vi) = softmax\u0012QiKi|\npdk\u0013\nVi(9.34)\nMultiHeadAttention (X) = ( head 1\bhead 2:::\bhead h)WO(9.35)\nPutting it all together with the parallel input matrix XThe function computed\nin parallel by an entire layer of Ntransformer block over the entire Ninput tokens\ncan be expressed as:\nO=LayerNorm (X+MultiHeadAttention (X)) (9.36)\nH=LayerNorm (O+FFN(O)) (9.37)\nOr we can break it down with one equation for each component computation, using\nT(of shape [N\u0002d]) to stand for transformer and superscripts to demarcate each\ncomputation inside the block:\nT1=MultiHeadAttention (X) (9.38)\nT2=X+T1(9.39)\nT3=LayerNorm (T2) (9.40)\nT4=FFN(T3) (9.41)\nT5=T4+T3(9.42)\nH=LayerNorm (T5) (9.43)\nHere when we use a notation like FFN (T3)we mean that the same FFN is applied\nin parallel to each of the Nembedding vectors in the window. Similarly, each of the\nNtokens is normed in parallel in the LayerNorm. Crucially, the input and output\ndimensions of transformer blocks are matched so they can be stacked. Since each\ntoken xiat the input to the block has dimensionality d, that means the input Xand\noutput Hare both of shape [N\u0002d].\n9.4 The input: embeddings for token and position\nLet\u2019s talk about where the input Xcomes from. Given a sequence of Ntokens ( Nis\nthe context length in tokens), the matrix Xof shape [N\u0002d]has an embedding for embedding\neach word in the context. The transformer does this by separately computing two\nembeddings: an input token embedding, and an input positional embedding.\nA token embedding, introduced in Chapter 7 and Chapter 8, is a vector of di-\nmension dthat will be our initial representation for the input token. (As we pass\nvectors up through the transformer layers in the residual stream, this embedding\nrepresentation will change and grow, incorporating context and playing a different\nrole depending on the kind of language model we are building.) The set of initial\nembeddings are stored in the embedding matrix E, which has a row for each of the\njVjtokens in the vocabulary. Thus each word is a row vector of ddimensions, and\nEhas shape [jVj\u0002d].\nGiven an input token string like Thanks for all the we \ufb01rst convert the tokens\ninto vocabulary indices (these were created when we \ufb01rst tokenized the input using 9.4 \u2022 T HE INPUT :EMBEDDINGS FOR TOKEN AND POSITION 197\nBPE or SentencePiece). So the representation of thanks for all the might be w=\n[5;4000;10532;2224]. Next we use indexing to select the corresponding rows from\nE, (row 5, row 4000, row 10532, row 2224).\nAnother way to think about selecting token embeddings from the embedding\nmatrix is to represent tokens as one-hot vectors of shape [1\u0002jVj], i.e., with one\ndimension for each word in the vocabulary. Recall that in a one-hot vector all the one-hot vector\nelements are 0 except one, the element whose dimension is the word\u2019s index in the\nvocabulary, which has value 1. So if the word \u201cthanks\u201d has index 5 in the vocabulary,\nx5=1, and xi=08i6=5, as shown here:\n[0 0 0 0 1 0 0 ... 0 0 0 0]\n1 2 3 4 5 6 7 ... ... |V|\nMultiplying by a one-hot vector that has only one non-zero element xi=1 simply\nselects out the relevant row vector for word i, resulting in the embedding for word i,\nas depicted in Fig. 9.11.\nE|V|d1|V|d=\u2715550 0 0 0 1 0 0 \u2026 0 0 0 0 1\nFigure 9.11 Selecting the embedding vector for word V5by multiplying the embedding\nmatrix Ewith a one-hot vector with a 1 in index 5.\nWe can extend this idea to represent the entire token sequence as a matrix of one-\nhot vectors, one for each of the Npositions in the transformer\u2019s context window, as\nshown in Fig. 9.12.\nE|V|ddN=\u2715|V|N0 0 0 0 0 0 0 \u2026 0 0 1 0 0 0 0 0 1 0 0 \u2026 0 0 0 0 1 0 0 0 0 0 0 \u2026 0 0 0 0 0 0 0 0 1 0 0 \u2026 0 0 0 0 \u2026\nFigure 9.12 Selecting the embedding matrix for the input sequence of token ids Wby mul-\ntiplying a one-hot matrix corresponding to Wby the embedding matrix E.\nThese token embeddings are not position-dependent. To represent the position\nof each token in the sequence, we combine these token embeddings with positional\nembeddings speci\ufb01c to each position in an input sequence.positional\nembeddings\nWhere do we get these positional embeddings? The simplest method, called\nabsolute position , is to start with randomly initialized embeddings correspondingabsolute\nposition\nto each possible input position up to some maximum length. For example, just as\nwe have an embedding for the word \ufb01sh, we\u2019ll have an embedding for the position 3.\nAs with word embeddings, these positional embeddings are learned along with other\nparameters during training. We can store them in a matrix Eposof shape [1\u0002N].\nTo produce an input embedding that captures positional information, we just\nadd the word embedding for each input to its corresponding positional embedding.\nThe individual token and position embeddings are both of size [1\u0002d], so their sum is 198 CHAPTER 9 \u2022 T HETRANSFORMER\nX = CompositeEmbeddings(word + position)Transformer BlockJanet1will2back3Janetwillbackthebillthe4bill5\n+++++PositionEmbeddingsWordEmbeddings\nFigure 9.13 A simple way to model position: add an embedding of the absolute position to\nthe token embedding to produce a new embedding of the same dimensionality.\nalso[1\u0002d], This new embedding serves as the input for further processing. Fig. 9.13\nshows the idea.\nThe \ufb01nal representation of the input, the matrix X, is an [N\u0002d]matrix in which\neach row iis the representation of the ith token in the input, computed by adding\nE[id(i)]\u2014the embedding of the id of the token that occurred at position i\u2014, to P[i],\nthe positional embedding of position i.\nA potential problem with the simple absolute position embedding approach is\nthat there will be plenty of training examples for the initial positions in our inputs\nand correspondingly fewer at the outer length limits. These latter embeddings may\nbe poorly trained and may not generalize well during testing. An alternative ap-\nproach to absolute positional embeddings is to choose a static function that maps\ninteger inputs to real-valued vectors in a way that captures the inherent relation-\nships among the positions. That is, it captures the fact that position 4 in an input is\nmore closely related to position 5 than it is to position 17. A combination of sine\nand cosine functions with differing frequencies was used in the original transformer\nwork. Even more complex positional embedding methods exist, such as ones that\nrepresent relative position instead of absolute position, often implemented in the\nattention mechanism at each layer rather than being added once at the initial input.\n9.5 The Language Modeling Head\nThe last component of the transformer we must introduce is the language modeling\nhead . Here we are using the word head to mean the additional neural circuitry welanguage\nmodeling head\nhead add on top of the basic transformer architecture when we apply pretrained trans-\nformer models to various tasks. The language modeling head is the circuitry we\nneed to do language modeling.\nRecall that language models, from the simple n-gram models of Chapter 3 through\nthe feedforward and RNN language models of Chapter 7 and Chapter 8, are word\npredictors. Given a context of words, they assign a probability to each possible next\nword. For example, if the preceding context is \u201cThanks for all the\u201d and we want to\nknow how likely the next word is \u201c\ufb01sh\u201d we would compute:\nP(\ufb01shjThanks for all the )\nLanguage models give us the ability to assign such a conditional probability to every\npossible next word, giving us a distribution over the entire vocabulary. The n-gram 9.5 \u2022 T HELANGUAGE MODELING HEAD 199\nlanguage models of Chapter 3 compute the probability of a word given counts of\nits occurrence with the n\u00001 prior words. The context is thus of size n\u00001. For\ntransformer language models, the context is the size of the transformer\u2019s context\nwindow, which can be quite large: 2K, 4K, even 32K tokens for very large models.\nThe job of the language modeling head is to take the output of the \ufb01nal trans-\nformer layer from the last token Nand use it to predict the upcoming word at posi-\ntionN+1. Fig. 9.14 shows how to accomplish this task, taking the output of the last\ntoken at the last layer (the d-dimensional output embedding of shape [1\u0002d]) and\nproducing a probability distribution over words (from which we will choose one to\ngenerate).\nLayer LTransformerBlockSoftmax over vocabulary VUnembedding layer\u20261 x |V|Logits Word probabilities1 x |V|hL1w1w2wNhL2hLNd x |V|1 x d   Unembedding layerU = ETy1y2y|V|\u2026u1u2u|V|\u2026Language Model Headtakes hLN and outputs adistribution over vocabulary V\nFigure 9.14 The language modeling head: the circuit at the top of a transformer that maps from the output\nembedding for token Nfrom the last transformer layer ( hL\nN) to a probability distribution over words in the\nvocabulary V.\nThe \ufb01rst module in Fig. 9.14 is a linear layer, whose job is to project from the\noutput hL\nN, which represents the output token embedding at position Nfrom the \ufb01nal\nblock L, (hence of shape [1\u0002d]) to the logit vector, or score vector, that will have a logit\nsingle score for each of the jVjpossible words in the vocabulary V. The logit vector\nuis thus of dimensionality 1 \u0002jVj.\nThis linear layer can be learned, but more commonly we tie this matrix to (the\ntranspose of) the embedding matrix E. Recall that in weight tying , we use the weight tying\nsame weights for two different matrices in the model. Thus at the input stage of the\ntransformer the embedding matrix (of shape [jVj\u0002d]) is used to map from a one-hot\nvector over the vocabulary (of shape [1\u0002jVj]) to an embedding (of shape [1\u0002d]).\nAnd then in the language model head, ET, the transpose of the embedding matrix (of\nshape [d\u0002jVj]) is used to map back from an embedding (shape [1\u0002d]) to a vector\nover the vocabulary (shape [1 \u0002jVj]). In the learning process, Ewill be optimized to\nbe good at doing both of these mappings. We therefore sometimes call the transpose\nETtheunembedding layer because it is performing this reverse mapping. unembedding\nA softmax layer turns the logits uinto the probabilities yover the vocabulary.\nu=hL\nNET(9.44)\ny=softmax (u) (9.45)\nWe can use these probabilities to do things like help assign a probability to a\ngiven text. But the most important usage to generate text, which we do by sampling 200 CHAPTER 9 \u2022 T HETRANSFORMER\na word from these probabilities y. We might sample the highest probability word\n(\u2018greedy\u2019 decoding), or use another of the sampling methods we\u2019ll introduce in Sec-\ntion 10.2. In either case, whatever entry ykwe choose from the probability vector y,\nwe generate the word that has that index k.\nwiSample token togenerate at position i+1\nfeedforwardlayer normattentionlayer norm\nU\nInput tokenLanguageModelingHead\nInputEncoding\nEi+\u2026logits\nfeedforwardlayer normattentionlayer normLayer 1Layer 2h1i  =  x2ix1ih2i  =  x3ifeedforwardlayer normattentionlayer normhLi  hL-1i  =  xLiy1y2y|V|\u2026Token probabilitiesu1u2u|V|\u2026softmaxwi+1\nLayer L\nFigure 9.15 A transformer language model (decoder-only), stacking transformer blocks\nand mapping from an input token wito to a predicted next token wi+1.\nFig. 9.15 shows the total stacked architecture for one token i. Note that the input\nto each transformer layer xiis the same as the output from the preceding layer h\u0000\ni.\nNow that we see all these transformer layers spread out on the page, we can point\nout another useful feature of the unembedding layer: as a tool for interpretability of\nthe internals of the transformer that we call the logit lens (Nostalgebraist, 2020). logit lens\nWe can take a vector from any layer of the transformer and, pretending that it is\nthe pre\ufb01nal embedding, simply multiply it by the unembedding layer to get logits,\nand compute a softmax to see the distribution over words that that vector might\nbe representing. This can be a useful window into the internal representations of\nthe model. Since the network wasn\u2019t trained to make the internal representations\nfunction in this way, the logit lens doesn\u2019t always work perfectly, but this can still\nbe a useful trick.\nA terminological note before we conclude: You will sometimes see a trans-\nformer used for this kind of unidirectional causal language model called a decoder-\nonly model . This is because this model constitutes roughly half of the encoder-decoder-only\nmodel 9.6 \u2022 S UMMARY 201\ndecoder model for transformers that we\u2019ll see how to apply to machine translation\nin Chapter 13. (Confusingly, the original introduction of the transformer had an\nencoder-decoder architecture, and it was only later that the standard paradigm for\ncausal language model was de\ufb01ned by using only the decoder part of this original\narchitecture).\n9.6 Summary\nThis chapter has introduced the transformer and its components for the task of lan-\nguage modeling. We\u2019ll continue the task of language modeling including issues like\ntraining and sampling in the next chapter.\nHere\u2019s a summary of the main points that we covered:\n\u2022 Transformers are non-recurrent networks based on multi-head attention , a\nkind of self-attention . A multi-head attention computation takes an input\nvector xiand maps it to an output aiby adding in vectors from prior tokens,\nweighted by how relevant they are for the processing of the current word.\n\u2022 Atransformer block consists of a residual stream in which the input from\nthe prior layer is passed up to the next layer, with the output of different com-\nponents added to it. These components include a multi-head attention layer\nfollowed by a feedforward layer , each preceded by layer normalizations .\nTransformer blocks are stacked to make deeper and more powerful networks.\n\u2022 The input to a transformer is a computing by adding an embedding (computed\nwith an embedding matrix ) to a positional encoding that represents the se-\nquential position of the token in the window.\n\u2022 Language models can be built out of stacks of transformer blocks, with a\nlanguage model head at the top, which applies an unembedding matrix to\nthe output Hof the top layer to generate the logits , which are then passed\nthrough a softmax to generate word probabilities.\n\u2022 Transformer-based language models have a wide context window (as wide\nas 32768 tokens for very large models) allowing them to draw on enormous\namounts of context to predict upcoming words.\nBibliographical and Historical Notes\nThe transformer (Vaswani et al., 2017) was developed drawing on two lines of prior\nresearch: self-attention andmemory networks .\nEncoder-decoder attention, the idea of using a soft weighting over the encodings\nof input words to inform a generative decoder (see Chapter 13) was developed by\nGraves (2013) in the context of handwriting generation, and Bahdanau et al. (2015)\nfor MT. This idea was extended to self-attention by dropping the need for separate\nencoding and decoding sequences and instead seeing attention as a way of weighting\nthe tokens in collecting information passed from lower layers to higher layers (Ling\net al., 2015; Cheng et al., 2016; Liu et al., 2016).\nOther aspects of the transformer, including the terminology of key, query, and\nvalue, came from memory networks , a mechanism for adding an external read- 202 CHAPTER 9 \u2022 T HETRANSFORMER\nwrite memory to networks, by using an embedding of a query to match keys rep-\nresenting content in an associative memory (Sukhbaatar et al., 2015; Weston et al.,\n2015; Graves et al., 2014).\nMORE HISTORY TBD IN NEXT DRAFT. CHAPTER\n10Large Language Models\n\u201cHow much do we know at any time? Much more, or so I believe, than we\nknow we know. \u201d\nAgatha Christie, The Moving Finger\nFluent speakers of a language bring an enormous amount of knowledge to bear dur-\ning comprehension and production. This knowledge is embodied in many forms,\nperhaps most obviously in the vocabulary, the rich representations we have of words\nand their meanings and usage. This makes the vocabulary a useful lens to explore\nthe acquisition of knowledge from text, by both people and machines.\nEstimates of the size of adult vocabularies vary widely both within and across\nlanguages. For example, estimates of the vocabulary size of young adult speakers of\nAmerican English range from 30,000 to 100,000 depending on the resources used\nto make the estimate and the de\ufb01nition of what it means to know a word. What\nis agreed upon is that the vast majority of words that mature speakers use in their\nday-to-day interactions are acquired early in life through spoken interactions with\ncaregivers and peers, usually well before the start of formal schooling. This active\nvocabulary (usually on the order of 2000 words for young speakers) is extremely\nlimited compared to the size of the adult vocabulary, and is quite stable, with very\nfew additional words learned via casual conversation beyond this early stage. Obvi-\nously, this leaves a very large number of words to be acquired by other means.\nA simple consequence of these facts is that children have to learn about 7 to 10\nwords a day, every single day , to arrive at observed vocabulary levels by the time they\nare 20 years of age. And indeed empirical estimates of vocabulary growth in late el-\nementary through high school are consistent with this rate. How do children achieve\nthis rate of vocabulary growth? The bulk of this knowledge acquisition seems to\nhappen as a by-product of reading, as part of the rich processing and reasoning that\nwe perform when we read. Research into the average amount of time children spend\nreading, and the lexical diversity of the texts they read, indicate that it is possible\nto achieve the desired rate. But the mechanism behind this rate of learning must\nbe remarkable indeed, since at some points during learning the rate of vocabulary\ngrowth exceeds the rate at which new words are appearing to the learner!\nSuch facts have motivated the distributional hypothesis of Chapter 6, which sug-\ngests that aspects of meaning can be learned solely from the texts we encounter over\nour lives, based on the complex association of words with the words they co-occur\nwith (and with the words that those words occur with). The distributional hypothe-\nsis suggests both that we can acquire remarkable amounts of knowledge from text,\nand that this knowledge can be brought to bear long after its initial acquisition. Of\ncourse, grounding from real-world interaction or other modalities can help build\neven more powerful models, but even text alone is remarkably useful.\nIn this chapter we formalize this idea of pretraining \u2014learning knowledge about pretraining\nlanguage and the world from vast amounts of text\u2014and call the resulting pretrained\nlanguage models large language models . Large language models exhibit remark- 204 CHAPTER 10 \u2022 L ARGE LANGUAGE MODELS\nable performance on all sorts of natural language tasks because of the knowledge\nthey learn in pretraining, and they will play a role throughout the rest of this book.\nThey have been especially transformative for tasks where we need to produce text,\nlike summarization, machine translation, question answering, or chatbots.\nWe\u2019ll start by seeing how to apply the transformer of Chapter 9 to language\nmodeling, in a setting often called causal or autoregressive language models, in\nwhich we iteratively predict words left-to-right from earlier words. We\u2019ll \ufb01rst in-\ntroduce training, seeing how language models are self-trained by iteratively being\ntaught to guess the next word in the text from the prior words.\nWe\u2019ll then talk about the process of text generation. The application of LLMs\nto generate text has vastly broadened the scope of NLP,. Text generation, code-\ngeneration, and image-generation together constitute the important new area of gen-\nerative AI . We\u2019ll introduce speci\ufb01c algorithms for generating text from a language generative AI\nmodel, like greedy decoding andsampling . And we\u2019ll see that almost any NLP\ntask can be modeled as word prediction in a large language model, if we think about\nit in the right way. We\u2019ll work through an example of using large language mod-\nels to solve one classic NLP task of summarization (generating a short text that\nsummarizes some larger document).\n10.1 Large Language Models with Transformers\nThe prior chapter introduced most of the components of a transformer in the domain\nof language modeling: the transformer block including multi-head attention , the\nlanguage modeling head , and the positional encoding of the input. In the following\nsections we\u2019ll introduce the remaining aspects of the transformer LLM: sampling\nandtraining . Before we do that, we use this section to talk about why and how we\napply transformer-based large language models to NLP tasks.\nThe tasks we will describe are all cases of conditional generation . Conditionalconditional\ngeneration\ngeneration is the task of generating text conditioned on an input piece of text. That\nis, we give the LLM an input piece of text, generally called a prompt , and then have\nthe LLM continue generating text token by token, conditioned on the prompt. The\nfact that transformers have such long contexts (many thousands of tokens) makes\nthem very powerful for conditional generation, because they can look back so far\ninto the prompting text.\nConsider the simple task of text completion, illustrated in Fig. 10.1. Here a\nlanguage model is given a text pre\ufb01x and is asked to generate a possible completion.\nNote that as the generation process proceeds, the model has direct access to the\npriming context as well as to all of its own subsequently generated outputs (at least\nas much as \ufb01ts in the large context window). This ability to incorporate the entirety\nof the earlier context and generated outputs at each time step is the key to the power\nof large language models built from transformers.\nSo why should we care about predicting upcoming words or tokens? The in-\nsight of large language modeling is that many practical NLP tasks can be cast as\nword prediction , and that a powerful-enough language model can solve them with\na high degree of accuracy. For example, we can cast sentiment analysis as language\nmodeling by giving a language model a context like:\nThe sentiment of the sentence ``I like Jackie Chan\" is:\nand comparing the following conditional probability of the words \u201cpositive\u201d and the 10.1 \u2022 L ARGE LANGUAGE MODELS WITH TRANSFORMERS 205\nPre\ufb01x TextCompletion Text\nEncoderTransformerBlocksSoftmax\nlongall\nandthanksforallthe\nthe\u2026UUUnencoder layerLanguage ModelingHeadlogits\nSo\nEi+\nEi+\nEi+\nEi+\nEi+\nEi+\nEi+\u2026\nFigure 10.1 Left-to-right (also called autoregressive) text completion with transformer-based large language\nmodels. As each token is generated, it gets added onto the context as a pre\ufb01x for generating the next token.\nword \u201cnegative\u201d to see which is higher:\nP(positivejThe sentiment of the sentence ``I like Jackie Chan\" is: )\nP(negativejThe sentiment of the sentence ``I like Jackie Chan\" is: )\nIf the word \u201cpositive\u201d is more probable, we say the sentiment of the sentence is\npositive, otherwise we say the sentiment is negative.\nWe can also cast more complex tasks as word prediction. Consider question\nanswering, in which the system is given a question (for example a question with\na simple factual answer) and must give a textual answer; we introduce this task in\ndetail in Chapter 14. We can cast the task of question answering as word prediction\nby giving a language model a question and a token like A:suggesting that an answer\nshould come next:\nQ: Who wrote the book ``The Origin of Species\"? A:\nIf we ask a language model to compute the probability distribution over possible\nnext words given this pre\ufb01x:\nP(wjQ: Who wrote the book ``The Origin of Species\"? A: )\nand look at which words whave high probabilities, we might expect to see that\nCharles is very likely, and then if we choose Charles and continue and ask\nP(wjQ: Who wrote the book ``The Origin of Species\"? A: Charles )\nwe might now see that Darwin is the most probable token, and select it.\nConditional generation can even be used to accomplish tasks that must generate\nlonger responses. Consider the task of text summarization , which is to take a longtext\nsummarization\ntext, such as a full-length article, and produce an effective shorter summary of it. We\ncan cast summarization as language modeling by giving a large language model a\ntext, and follow the text by a token like tl;dr ; this token is short for something like 206 CHAPTER 10 \u2022 L ARGE LANGUAGE MODELS\n\u2018too long; didn\u2019t read\u2019 and in recent years people often use this token, especially in\ninformal work emails, when they are going to give a short summary. Since this token\nis suf\ufb01ciently frequent in language model training data, language models have seen\nmany texts in which the token occurs before a summary, and hence will interpret the\ntoken as instructions to generate a summary. We can then do conditional generation:\ngive the language model this pre\ufb01x, and then have it generate the following words,\none by one, and take the entire response as a summary. Fig. 10.2 shows an example\nof a text and a human-produced summary from a widely-used summarization corpus\nconsisting of CNN and Daily Mirror news articles.\nOriginal Article\nThe only thing crazier than a guy in snowbound Massachusetts boxing up the powdery white stuff\nand offering it for sale online? People are actually buying it. For $89, self-styled entrepreneur\nKyle Waring will ship you 6 pounds of Boston-area snow in an insulated Styrofoam box \u2013 enough\nfor 10 to 15 snowballs, he says.\nBut not if you live in New England or surrounding states. \u201cWe will not ship snow to any states\nin the northeast!\u201d says Waring\u2019s website, ShipSnowYo.com. \u201cWe\u2019re in the business of expunging\nsnow!\u201d\nHis website and social media accounts claim to have \ufb01lled more than 133 orders for snow \u2013 more\nthan 30 on Tuesday alone, his busiest day yet. With more than 45 total inches, Boston has set a\nrecord this winter for the snowiest month in its history. Most residents see the huge piles of snow\nchoking their yards and sidewalks as a nuisance, but Waring saw an opportunity.\nAccording to Boston.com, it all started a few weeks ago, when Waring and his wife were shov-\neling deep snow from their yard in Manchester-by-the-Sea, a coastal suburb north of Boston. He\njoked about shipping the stuff to friends and family in warmer states, and an idea was born. [...]\nSummary\nKyle Waring will ship you 6 pounds of Boston-area snow in an insulated Styrofoam box \u2013 enough\nfor 10 to 15 snowballs, he says. But not if you live in New England or surrounding states.\nFigure 10.2 Excerpt from a sample article and its summary from the CNN/Daily Mail summarization corpus\n(Hermann et al., 2015b), (Nallapati et al., 2016).\nIf we take this full article and append the token tl;dr , we can use this as the con-\ntext to prime the generation process to produce a summary as illustrated in Fig. 10.3.\nAgain, what makes transformers able to succeed at this task (as compared, say, to\nthe primitive n-gram language model) is that attention can incorporate information\nfrom the large context window, giving the model access to the original article as well\nas to the newly generated text throughout the process.\nWhich words do we generate at each step? One simple way to generate words\nis to always generate the most likely word given the context. Generating the most\nlikely word given the context is called greedy decoding . A greedy algorithm is onegreedy\ndecoding\nthat make a choice that is locally optimal, whether or not it will turn out to have\nbeen the best choice with hindsight. Thus in greedy decoding, at each time step in\ngeneration, the output ytis chosen by computing the probability for each possible\noutput (every word in the vocabulary) and then choosing the highest probability\nword (the argmax):\n\u02c6wt=argmaxw2VP(wjw<t) (10.1)\nIn practice, however, we don\u2019t use greedy decoding with large language models.\nA major problem with greedy decoding is that because the words it chooses are (by\nde\ufb01nition) extremely predictable, the resulting text is generic and often quite repeti-\ntive. Indeed, greedy decoding is so predictable that it is deterministic; if the context 10.2 \u2022 S AMPLING FOR LLM G ENERATION 207\nOriginal StoryGenerated Summary\n\u2026ideaKyle\nwasborn.KyleWaring\nWaringonlyThe\u2026will\nDelimiterwillUUU\ntl;drLM Head\nE\nE\nE\nE\nE\nE\nE\nE\u2026\nFigure 10.3 Summarization with large language models using the tl;dr token and context-based autore-\ngressive generation.\nis identical, and the probabilistic model is the same, greedy decoding will always re-\nsult in generating exactly the same string. We\u2019ll see in Chapter 13 that an extension\nto greedy decoding called beam search works well in tasks like machine translation,\nwhich are very constrained in that we are always generating a text in one language\nconditioned on a very speci\ufb01c text in another language. In most other tasks, how-\never, people prefer text which has been generated by more sophisticated methods,\ncalled sampling methods , that introduce a bit more diversity into the generations.\nWe\u2019ll see how to do that in the next few sections.\n10.2 Sampling for LLM Generation\nThe core of the generation process for large language models is the task of choosing\nthe single word to generate next based on the context and based on the probabilities\nthat the model assigns to possible words. This task of choosing a word to generate\nbased on the model\u2019s probabilities is called decoding . Decoding from a language decoding\nmodel in a left-to-right manner (or right-to-left for languages like Arabic in which\nwe read from right to left), and thus repeatedly choosing the next word conditioned\non our previous choices is called autoregressive generation orcausal LM genera-autoregressive\ngeneration\ntion.1(As we\u2019ll see, alternatives like the masked language models of Chapter 11 are\nnon-causal because they can predict words based on both past and future words).\nThe most common method for decoding in large language models is sampling .\nRecall from Chapter 3 that sampling from a model\u2019s distribution over words means sampling\nto choose random words according to their probability assigned by the model. That\nis, we iteratively choose a word to generate according to its probability in context\n1Technically an autoregressive model predicts a value at time tbased on a linear function of the values\nat times t\u00001,t\u00002, and so on. Although language models are not linear (since they have many layers of\nnon-linearities), we loosely refer to this generation technique as autoregressive since the word generated\nat each time step is conditioned on the word selected by the network from the previous step. 208 CHAPTER 10 \u2022 L ARGE LANGUAGE MODELS\nas de\ufb01ned by the model. Thus we are more likely to generate words that the model\nthinks have a high probability in the context and less likely to generate words that\nthe model thinks have a low probability.\nWe saw back in Chapter 3 on page 43 how to generate text from a unigram lan-\nguage model , by repeatedly randomly sampling words according to their probability\nuntil we either reach a pre-determined length or select the end-of-sentence token. To\ngenerate text from a trained transformer language model we\u2019ll just generalize this\nmodel a bit: at each step we\u2019ll sample words according to their probability condi-\ntioned on our previous choices , and we\u2019ll use a transformer language model as the\nprobability model that tells us this probability.\nWe can formalize this algorithm for generating a sequence of words W=w1;w2;:::; wN\nuntil we hit the end-of-sequence token, using x\u0018p(x)to mean \u2018choose xby sam-\npling from the distribution p(x):\ni 1\nwi\u0018p(w)\nwhile wi!= EOS\ni i + 1\nwi\u0018p(wijw<i)\nThe algorithm above is called random sampling , and it turns out random sam-random\nsampling\npling doesn\u2019t work well enough. The problem is that even though random sampling\nis mostly going to generate sensible, high-probable words, there are many odd, low-\nprobability words in the tail of the distribution, and even though each one is low-\nprobability, if you add up all the rare words, they constitute a large enough portion\nof the distribution that they get chosen often enough to result in generating weird\nsentences. For this reason, instead of random sampling, we usually use sampling\nmethods that avoid generating the very unlikely words.\nThe sampling methods we introduce below each have parameters that enable\ntrading off two important factors in generation: quality anddiversity . Methods\nthat emphasize the most probable words tend to produce generations that are rated\nby people as more accurate, more coherent, and more factual, but also more boring\nand more repetitive. Methods that give a bit more weight to the middle-probability\nwords tend to be more creative and more diverse, but less factual and more likely to\nbe incoherent or otherwise low-quality.\n10.2.1 Top- ksampling\nTop-k sampling is a simple generalization of greedy decoding. Instead of choosing top-k sampling\nthe single most probable word to generate, we \ufb01rst truncate the distribution to the\ntopkmost likely words, renormalize to produce a legitimate probability distribution,\nand then randomly sample from within these kwords according to their renormalized\nprobabilities. More formally:\n1. Choose in advance a number of words k\n2. For each word in the vocabulary V, use the language model to compute the\nlikelihood of this word given the context p(wtjw<t)\n3. Sort the words by their likelihood, and throw away any word that is not one of\nthe top kmost probable words.\n4. Renormalize the scores of the kwords to be a legitimate probability distribu-\ntion. 10.2 \u2022 S AMPLING FOR LLM G ENERATION 209\n5. Randomly sample a word from within these remaining kmost-probable words\naccording to its probability.\nWhen k=1, top- ksampling is identical to greedy decoding. Setting kto a larger\nnumber than 1 leads us to sometimes select a word which is not necessarily the most\nprobable, but is still probable enough, and whose choice results in generating more\ndiverse but still high-enough-quality text.\n10.2.2 Nucleus or top- psampling\nOne problem with top- ksampling is that kis \ufb01xed, but the shape of the probability\ndistribution over words differs in different contexts. If we set k=10, sometimes\nthe top 10 words will be very likely and include most of the probability mass, but\nother times the probability distribution will be \ufb02atter and the top 10 words will only\ninclude a small part of the probability mass.\nAn alternative, called top-p sampling ornucleus sampling (Holtzman et al., top-p sampling\n2020), is to keep not the top kwords, but the top ppercent of the probability mass.\nThe goal is the same; to truncate the distribution to remove the very unlikely words.\nBut by measuring probability rather than the number of words, the hope is that the\nmeasure will be more robust in very different contexts, dynamically increasing and\ndecreasing the pool of word candidates.\nGiven a distribution P(wtjw<t), the top- pvocabulary V(p)is the smallest set of\nwords such that\nX\nw2V(p)P(wjw<t)\u0015p: (10.2)\n10.2.3 Temperature sampling\nIntemperature sampling , we don\u2019t truncate the distribution, but instead reshapetemperature\nsampling\nit. The intuition for temperature sampling comes from thermodynamics, where a\nsystem at a high temperature is very \ufb02exible and can explore many possible states,\nwhile a system at a lower temperature is likely to explore a subset of lower energy\n(better) states. In low-temperature sampling, we smoothly increase the probability\nof the most probable words and decrease the probability of the rare words.\nWe implement this intuition by simply dividing the logit by a temperature param-\netertbefore we normalize it by passing it through the softmax. In low-temperature\nsampling, t2(0;1]. Thus instead of computing the probability distribution over the\nvocabulary directly from the logit as in the following (repeated from (9.45)):\ny=softmax (u) (10.3)\nwe instead \ufb01rst divide the logits by t, computing the probability vector yas\ny=softmax (u=t) (10.4)\nWhy does this work? When tis close to 1 the distribution doesn\u2019t change much.\nBut the lower tis, the larger the scores being passed to the softmax (dividing by a\nsmaller fraction t\u00141 results in making each score larger). Recall that one of the\nuseful properties of a softmax is that it tends to push high values toward 1 and low\nvalues toward 0. Thus when larger numbers are passed to a softmax the result is\na distribution with increased probabilities of the most high-probability words and\ndecreased probabilities of the low probability words, making the distribution more\ngreedy. As tapproaches 0 the probability of the most likely word approaches 1. 210 CHAPTER 10 \u2022 L ARGE LANGUAGE MODELS\nNote, by the way, that there can be other situations where we may want to do\nsomething quite different and \ufb02atten the word probability distribution instead of\nmaking it greedy. Temperature sampling can help with this situation too, in this case\nhigh-temperature sampling, in which case we use t>1.\n10.3 Pretraining Large Language Models\nHow do we teach a transformer to be a language model? What is the algorithm and\nwhat data do we train on?\n10.3.1 Self-supervised training algorithm\nTo train a transformer as a language model, we use the same self-supervision (or self-supervision\nself-training ) algorithm we saw in Section 8.2.2: we take a corpus of text as training\nmaterial and at each time step task the model to predict the next word. We call such\na model self-supervised because we don\u2019t have to add any special gold labels to\nthe data; the natural sequence of words is its own supervision! We simply train the\nmodel to minimize the error in predicting the true next word in the training sequence,\nusing cross-entropy as the loss function.\nRecall that the cross-entropy loss measures the difference between a predicted\nprobability distribution and the correct distribution.\nLCE=\u0000X\nw2Vyt[w]log\u02c6yt[w] (10.5)\nIn the case of language modeling, the correct distribution ytcomes from knowing the\nnext word. This is represented as a one-hot vector corresponding to the vocabulary\nwhere the entry for the actual next word is 1, and all the other entries are 0. Thus,\nthe cross-entropy loss for language modeling is determined by the probability the\nmodel assigns to the correct next word (all other words get multiplied by zero). So\nat time tthe CE loss in (10.5) can be simpli\ufb01ed as the negative log probability the\nmodel assigns to the next word in the training sequence.\nLCE(\u02c6yt;yt) =\u0000log\u02c6yt[wt+1] (10.6)\nThus at each word position tof the input, the model takes as input the correct se-\nquence of tokens w1:t, and uses them to compute a probability distribution over\npossible next words so as to compute the model\u2019s loss for the next token wt+1. Then\nwe move to the next word, we ignore what the model predicted for the next word\nand instead use the correct sequence of tokens w1:t+1to estimate the probability of\ntoken wt+2. This idea that we always give the model the correct history sequence to\npredict the next word (rather than feeding the model its best case from the previous\ntime step) is called teacher forcing . teacher forcing\nFig. 10.4 illustrates the general training approach. At each step, given all the\npreceding words, the \ufb01nal transformer layer produces an output distribution over\nthe entire vocabulary. During training, the probability assigned to the correct word\nis used to calculate the cross-entropy loss for each item in the sequence. The loss\nfor a training sequence is the average cross-entropy loss over the entire sequence.\nThe weights in the network are adjusted to minimize the average CE loss over the\ntraining sequence via gradient descent. 10.3 \u2022 P RETRAINING LARGE LANGUAGE MODELS 211\nlongandthanksforNext tokenallLoss\u2026=\n<latexit sha1_base64=\"AovqpaL476UmJ1EU1xZPgDZ70tQ=\">AAAB9nicbVDLSsNAFL2pr1pfURcu3AwWwY0lEakui25cVrAPaEqYTCbt0EkmzEzEEvIrbkTcKPgZ/oJ/Y9Jm09YDA4dzznDvPV7MmdKW9WtU1tY3Nreq27Wd3b39A/PwqKtEIgntEMGF7HtYUc4i2tFMc9qPJcWhx2nPm9wXfu+ZSsVE9KSnMR2GeBSxgBGsc8k1Ty4dLkZo6qZOiPVYhimO/CyruWbdalgzoFVil6QOJdqu+eP4giQhjTThWKmBbcV6mGKpGeE0qzmJojEmEzyi6WztDJ3nko8CIfMXaTRTF3I4VGoaenmy2E0te4X4nzdIdHA7TFkUJ5pGZD4oSDjSAhUdIJ9JSjSf5gQTyfINERljiYnOmypOt5cPXSXdq4bdbDQfr+utu7KEKpzCGVyADTfQggdoQwcIZPAGn/BlvBivxrvxMY9WjPLPMSzA+P4DPEiSHA==</latexit>\u0000logyand\nStackedTransformerBlocksSolongandthanksfor\u2026\u2026\n\u2026\nU\nInput tokensx1x2LanguageModelingHeadx3x4x5InputEncoding\nE1+\nE2+\nE3+\nE4+\nE5+\u2026\u2026\u2026\u2026\u2026\nU\nU\nU\nU\u2026logitslogitslogitslogitslogits\u2026\n<latexit sha1_base64=\"q3ZgXDyG7qtkT7t8hT47RdlwYG4=\">AAAB+XicbVDLSsNAFJ3UV62vWHe6GVsEN5bERXUlBUVcVrAPaEqYTCft0MlMmJkIIQT8AT/CTRE3Cv6Ev+DfmLTdtPXAwOGcM9x7jxcyqrRl/RqFtfWNza3idmlnd2//wDwst5WIJCYtLJiQXQ8pwignLU01I91QEhR4jHS88W3ud56JVFTwJx2HpB+gIac+xUhnkmseXzhMDGHsJk6A9EgGiR4hPlZpWnLNqlWzpoCrxJ6TauP0tXw3qdw0XfPHGQgcBYRrzJBSPdsKdT9BUlPMSFpyIkVChMdoSJLp5ik8y6QB9IXMHtdwqi7kUKBUHHhZMl9PLXu5+J/Xi7R/3U8oDyNNOJ4N8iMGtYB5DXBAJcGaxRlBWNJsQ4hHSCKss7Ly0+3lQ1dJ+7Jm12v1x6yDezBDEZyACjgHNrgCDfAAmqAFMHgBE/AJvozEeDPejY9ZtGDM/xyBBRjff79pldo=</latexit>\u0000logythanks\nFigure 10.4 Training a transformer as a language model.\nNote the key difference between this \ufb01gure and the earlier RNN-based version\nshown in Fig. 8.6. There the calculation of the outputs and the losses at each step\nwas inherently serial given the recurrence in the calculation of the hidden states.\nWith transformers, each training item can be processed in parallel since the output\nfor each element in the sequence is computed separately.\nLarge models are generally trained by \ufb01lling the full context window (for exam-\nple 4096 tokens for GPT4 or 8192 for Llama 3) with text. If documents are shorter\nthan this, multiple documents are packed into the window with a special end-of-text\ntoken between them. The batch size for gradient descent is usually quite large (the\nlargest GPT-3 model uses a batch size of 3.2 million tokens).\n10.3.2 Training corpora for large language models\nLarge language models are mainly trained on text scraped from the web, augmented\nby more carefully curated data. Because these training corpora are so large, they are\nlikely to contain many natural examples that can be helpful for NLP tasks, such as\nquestion and answer pairs (for example from FAQ lists), translations of sentences\nbetween various languages, documents together with their summaries, and so on.\nWeb text is usually taken from corpora of automatically-crawled web pages like\nthecommon crawl , a series of snapshots of the entire web produced by the non- common crawl\npro\ufb01t Common Crawl ( https://commoncrawl.org/ ) that each have billions of\nwebpages. Various versions of common crawl data exist, such as the Colossal Clean\nCrawled Corpus (C4; Raffel et al. 2020), a corpus of 156 billion tokens of English\nthat is \ufb01ltered in various ways (deduplicated, removing non-natural language like\ncode, sentences with offensive words from a blocklist). This C4 corpus seems to\nconsist in large part of patent text documents, Wikipedia, and news sites (Dodge\net al., 2021).\nWikipedia plays a role in lots of language model training, as do corpora of books.\nThe Pile (Gao et al., 2020) is an 825 GB English text corpus that is constructed by The Pile\npublicly released code, containing again a large amount of text scraped from the web 212 CHAPTER 10 \u2022 L ARGE LANGUAGE MODELS\nas well as books and Wikipedia; Fig. 10.5 shows its composition. Dolma is a larger\nopen corpus of English, created with public tools, containing three trillion tokens,\nwhich similarly consists of web text, academic papers, code, books, encyclopedic\nmaterials, and social media (Soldaini et al., 2024).\nFigure 1: Treemap of Pile components by effective size.\ntroduce a new \ufb01ltered subset of Common Crawl,\nPile-CC, with improved extraction quality.\nThrough our analyses, we con\ufb01rm that the Pile is\nsigni\ufb01cantly distinct from pure Common Crawl\ndata. Additionally, our evaluations show that the\nexisting GPT-2 and GPT-3 models perform poorly\non many components of the Pile, and that models\ntrained on the Pile signi\ufb01cantly outperform both\nraw and \ufb01ltered Common Crawl models. To com-\nplement the performance evaluations, we also per-\nform an exploratory analysis of the text within the\nPile to provide a detailed picture of the data. We\nhope that our extensive documentation of the con-\nstruction and characteristics of the Pile will help\nresearchers make informed decisions about poten-\ntial downstream applications.\nFinally, we make publicly available the preprocess-\ning code for the constituent datasets of the Pile and\nthe code for constructing alternative versions2. In\nthe interest of reproducibility, we also document\nall processing performed on each dataset (and the\nPile as a whole) in as much detail as possible. For\nfurther details about the processing of each dataset,\nsee Section 2and Appendix C.\n2https://github.com/EleutherAI/\nthe-pile1.1 Contributions\nThe core contributions of this paper are:\n1.The introduction of a 825.18GiB english-\nlanguage dataset for language modeling com-\nbining 22 diverse sources.\n2.The introduction of 14new language model-\ning datasets, which we expect to be of inde-\npendent interest to researchers.\n3.Evaluations demonstrating signi\ufb01cant im-\nprovements across many domains by GPT-2-\nsized models trained on this new dataset, com-\npared to training on CC-100 and raw Common\nCrawl.\n4.The investigation and documentation of this\ndataset, which we hope will better inform re-\nsearchers about how to use it as well as moti-\nvate them to undertake similar investigations\nof their own data.\n2 The Pile Datasets\nThe Pile is composed of 22 constituent sub-datasets,\nas shown in Table 1. Following Brown et al. (2020 ),\nwe increase the weights of higher quality compo-\nnents, with certain high-quality datasets such as\nWikipedia being seen up to 3 times (\u201cepochs\u201d) for\n2\nFigure 10.5 The Pile corpus, showing the size of different components, color coded as\nacademic (articles from PubMed and ArXiv, patents from the USPTA; internet (webtext in-\ncluding a subset of the common crawl as well as Wikipedia), prose (a large corpus of books),\ndialogue (including movie subtitles and chat data), and misc.. Figure from Gao et al. (2020).\nFiltering for quality and safety Pretraining data drawn from the web is \ufb01ltered\nfor both quality andsafety . Quality \ufb01lters are classi\ufb01ers that assign a score to each\ndocument. Quality is of course subjective, so different quality \ufb01lters are trained\nin different ways, but often to value high-quality reference corpora like Wikipedia,\nbooks, and particular websites and to avoid websites with lots of PII(Personal Iden- PII\nti\ufb01able Information) or adult content. Filters also remove boilerplate text which is\nvery frequent on the web. Another kind of quality \ufb01ltering is deduplication, which\ncan be done at various levels, so as to remove duplicate documents, duplicate web\npages, or duplicate text. Quality \ufb01ltering generally improves language model per-\nformance (Longpre et al., 2024b; Llama Team, 2024).\nSafety \ufb01ltering is again a subjective decision, and often includes toxicity detec-\ntion based on running off-the-shelf toxicity classi\ufb01ers. This can have mixed results.\nOne problem is that current toxicity classi\ufb01ers mistakenly \ufb02ag non-toxic data if it\nis generated by speakers of minority dialects like African American English (Xu\net al., 2021). Another problem is that models trained on toxicity-\ufb01ltered data, while\nsomewhat less toxic, are also worse at detecting toxicity themselves (Longpre et al.,\n2024b). These issues make the question of how to do better safety \ufb01ltering an im-\nportant open problem.\nUsing large datasets scraped from the web to train language models poses ethical\nand legal questions:\nCopyright: Much of the text in these large datasets (like the collections of \ufb01c-\ntion and non-\ufb01ction books) is copyrighted. In some countries, like the United\nStates, the fair use doctrine may allow copyrighted content to be used for\ntransformative uses, but it\u2019s not clear if that remains true if the language mod-\nels are used to generate text that competes with the market for the text they 10.3 \u2022 P RETRAINING LARGE LANGUAGE MODELS 213\nare trained on (Henderson et al., 2023).\nData consent: Owners of websites can indicate that they don\u2019t want their sites\nto be crawled by web crawlers (either via a robots.txt \ufb01le, or via Terms of\nService). Recently there has been a sharp increase in the number of web-\nsites that have indicated that they don\u2019t want large language model builders\ncrawling their sites for training data (Longpre et al., 2024a). Because it\u2019s not\nclear what legal status these indications have in different countries, or whether\nthese restrictions are retroactive, what effect this will have on large pretraining\ndatasets is unclear.\nPrivacy: Large web datasets also have privacy issues since they contain private\ninformation like phone numbers and IP addresses. While \ufb01lters are used to try\nto remove websites likely to contain large amounts of personal information,\nsuch \ufb01ltering isn\u2019t suf\ufb01cient.\n10.3.3 Finetuning\nAlthough the enormous pretraining data for a large language model includes text\nfrom many domains, it\u2019s often the case that we want to apply it in a new domain or\ntask that might not have appeared suf\ufb01ciently in the pre-training data. For example,\nwe might want a language model that\u2019s specialized to legal or medical text. Or we\nmight have a multilingual language model that knows many languages but might\nbene\ufb01t from some more data in our particular language of interest. Or we want a\nlanguage model that is specialized to a particular task.\nIn such cases, we can simply continue training the model on relevant data from\nthe new domain or language (Gururangan et al., 2020). This process of taking a fully\npretrained model and running additional training passes on some new data is called\n\ufb01netuning . Fig. 10.6 sketches the paradigm. \ufb01netuning\nFine-tuning Data\nPretraining DataPretraining\n\u2026\n\u2026\n\u2026Fine-tuning\n\u2026\n\u2026\n\u2026Pretrained LMFine-tuned LM\nFigure 10.6 Pretraining and \ufb01netuning. A pre-trained model can be \ufb01netuned to a par-\nticular domain, dataset, or task. There are many different ways to \ufb01netune, depending on\nexactly which parameters are updated from the \ufb01netuning data: all the parameters, some of\nthe parameters, or only the parameters of speci\ufb01c extra circuitry.\nWe\u2019ll introduce four related kinds of \ufb01netuning in this chapter and the two fol-\nlowing chapters. In all four cases, \ufb01netuning means the process of taking a pre-\ntrained model and further adapting some or all of its parameters to some new data.\nBut they differ on exactly which parameters get updated.\nIn the \ufb01rst kind of \ufb01netuning we retrain all the parameters of the model on this\nnew data, using the same method (word prediction) and loss function (cross-entropy\nloss) as for pretraining. In a sense it\u2019s as if the new data were at the tail end of 214 CHAPTER 10 \u2022 L ARGE LANGUAGE MODELS\nthe pretraining data, and so you\u2019ll sometimes see this method called continued pre-\ntraining .continued\npretraining\nRetraining all the parameters of the model is very slow and expensive when the\nlanguage model is huge. So instead we can freeze some of the parameters (i.e., leave freeze\nthem unchanged from their pretrained value) and train only a subset of parameters\non the new data. In Section 10.5.3 we\u2019ll describe this second variety of \ufb01netun-\ning, called parameter-ef\ufb01cient \ufb01netuning , orPEFT . because we ef\ufb01ciently select\nspeci\ufb01c parameters to update when \ufb01netuning, and leave the rest in their pretrained\nvalues.\nIn Chapter 11 we\u2019ll introduce a third kind of \ufb01netuning, also parameter-ef\ufb01cient.\nIn this version, the goal is to use a language model as a kind of classi\ufb01er or labeler\nfor a speci\ufb01c task. For example we might train the model to be a sentiment classi\ufb01er.\nWe do this by adding extra neural circuitry (an extra head ) after the top layer of the\nmodel. This classi\ufb01cation head takes as input some of the top layer embeddings of\nthe transformer and produces as output a classi\ufb01cation. In this method, most com-\nmonly used with masked language models like BERT, we freeze the entire pretrained\nmodel and only train the classi\ufb01cation head on some new data, usually labeled with\nsome class that we want to predict.\nFinally, in Chapter 12 we\u2019ll introduce a fourth kind of \ufb01netuning, that is a cru-\ncial component of the largest language models: supervised \ufb01netuning orSFT. SFT\nis often used for instruction \ufb01netuning , in which we want a pretrained language\nmodel to learn to follow text instructions, for example to answer questions or follow\na command to write something. Here we create a dataset of prompts and desired\nresponses (for example questions and their answers, or commands and their ful-\n\ufb01llments), and we train the language model using the normal cross-entropy loss to\npredict each token in the instruction prompt iteratively, essentially training it to pro-\nduce the desired response from the command in the prompt. It\u2019s called supervised\nbecause unlike in pretraining, where we just take any data and predict the words in\nit, we build the special \ufb01netuning dataset by hand, creating supervised responses to\neach command.\nOften everything that happens after pretraining is lumped together as post-training ;\nwe\u2019ll discuss the various parts of post-training in Chapter 12.\n10.4 Evaluating Large Language Models\nPerplexity As we \ufb01rst saw in Chapter 3, one way to evaluate language models is\nto measure how well they predict unseen text. Intuitively, good models are those that\nassign higher probabilities to unseen data (are less surprised when encountering the\nnew words).\nWe instantiate this intuition by using perplexity to measure the quality of a perplexity\nlanguage model. Recall from page 39 that the perplexity of a model qon an unseen\ntest set is the inverse probability that qassigns to the test set, normalized by the test\nset length. For a test set of ntokens w1:n, the perplexity is\nPerplexityq(w1:n) = Pq(w1:n)\u00001\nn\n=ns\n1\nPq(w1:n)(10.7)\nTo visualize how perplexity can be computed as a function of the probabilities the 10.4 \u2022 E VALUATING LARGE LANGUAGE MODELS 215\nLM computes for each new word, we can use the chain rule to expand the computa-\ntion of probability of the test set:\nPerplexityq(w1:n) =nvuutnY\ni=11\nPq(wijw<i)(10.8)\nNote that because of the inverse in Eq. 10.7, the higher the probability of the word\nsequence, the lower the perplexity. Thus the the lower the perplexity of a model on\nthe data, the better the model . Minimizing perplexity is equivalent to maximizing\nthe test set probability according to the language model.\nOne caveat: because perplexity depends on the length of a text, it is very sensitive\nto differences in the tokenization algorithm. That means that it\u2019s hard to exactly\ncompare perplexities produced by two language models if they have very different\ntokenizers. For this reason perplexity is best used when comparing language models\nthat use the same tokenizer.\nOther factors While the predictive accuracy of a language model, as measured by\nperplexity, is a very useful metric, we also care about different kinds of accuracy, for\nthe downstream tasks we apply our language model to. For each task like machine\ntranslation, summarization, question answering, speech recognition, and dialogue,\nwe can measure the accuracy at those tasks. Future chapters will introduce task-\nspeci\ufb01c metrics that allow us to evaluate how accuracy or correct language models\nare at these downstream tasks.\nBut when evaluating models we also care about factors besides any of these\nkinds of accuracy (Dodge et al., 2019; Ethayarajh and Jurafsky, 2020). For example,\nwe often care about how a big a model is, and how long it takes to train or do\ninference. This can matter because we have constraints on time either for training\nor at inference. Or we may have constraints on memory, since the GPUs we run\nour models on have \ufb01xed memory sizes. Big models also use more energy, and we\nprefer models that use less energy, both to reduce the environmental impact of the\nmodel and to reduce the \ufb01nancial cost of building or deploying it. We can target\nour evaluation to these factors by measuring performance normalized to a giving\ncompute or memory budget. We can also directly measure the energy usage of our\nmodel in kWh or in kilograms of CO 2emitted (Strubell et al., 2019; Henderson\net al., 2020; Liang et al., 2023).\nAnother feature that a language model evaluation can measure is fairness. We\nknow that language models are biased, exhibiting gendered and racial stereotypes,\nor decreased performance for language from or about certain demographics groups.\nThere are language model evaluation benchmarks that measure the strength of these\nbiases, such as StereoSet (Nadeem et al., 2021), RealToxicityPrompts (Gehman\net al., 2020), and BBQ (Parrish et al., 2022) among many others. We also want\nlanguage models whose performance is equally fair to different groups. For exam-\nple, we could chose an evaluation that is fair in a Rawlsian sense by maximizing the\nwelfare of the worst-off group (Rawls, 2001; Hashimoto et al., 2018; Sagawa et al.,\n2020).\nFinally, there are many kinds of leaderboards like Dynabench (Kiela et al., 2021)\nand general evaluation protocols like HELM (Liang et al., 2023); we will return to\nthese in later chapters when we introduce evaluation metrics for speci\ufb01c tasks like\nquestion answering and information retrieval. 216 CHAPTER 10 \u2022 L ARGE LANGUAGE MODELS\n10.5 Dealing with Scale\nLarge language models are large. For example the Llama 3.1 405B Instruct model\nfrom Meta has 405 billion parameters (126 layers, a model dimensionality of 16,384,\n128 attention heads) and was trained on 15.6 terabytes of text tokens (Llama Team,\n2024), using a vocabulary of 128K tokens. So there is a lot of research on un-\nderstanding how LLMs scale, and especially how to implement them given limited\nresources. In the next few sections we discuss how to think about scale (the concept\nofscaling laws ), and important techniques for getting language models to work\nef\ufb01ciently, such as the KV cache and parameter-ef\ufb01cient \ufb01ne tuning.\n10.5.1 Scaling laws\nThe performance of large language models has shown to be mainly determined by\n3 factors: model size (the number of parameters not counting embeddings), dataset\nsize (the amount of training data), and the amount of compute used for training. That\nis, we can improve a model by adding parameters (adding more layers or having\nwider contexts or both), by training on more data, or by training for more iterations.\nThe relationships between these factors and performance are known as scaling\nlaws . Roughly speaking, the performance of a large language model (the loss) scales scaling laws\nas a power-law with each of these three properties of model training.\nFor example, Kaplan et al. (2020) found the following three relationships for\nlossLas a function of the number of non-embedding parameters N, the dataset size\nD, and the compute budget C, for models training with limited parameters, dataset,\nor compute budget, if in each case the other two properties are held constant:\nL(N) =\u0012Nc\nN\u0013aN\n(10.9)\nL(D) =\u0012Dc\nD\u0013aD\n(10.10)\nL(C) =\u0012Cc\nC\u0013aC\n(10.11)\nThe number of (non-embedding) parameters Ncan be roughly computed as fol-\nlows (ignoring biases, and with das the input and output dimensionality of the\nmodel, dattnas the self-attention layer size, and dffthe size of the feedforward layer):\nN\u00192d nlayer(2dattn+dff)\n\u001912nlayerd2(10.12)\n(assuming dattn=dff=4=d)\nThus GPT-3, with n=96 layers and dimensionality d=12288, has 12\u000296\u0002\n122882\u0019175 billion parameters.\nThe values of Nc,Dc,Cc,aN,aD, and aCdepend on the exact transformer\narchitecture, tokenization, and vocabulary size, so rather than all the precise values,\nscaling laws focus on the relationship with loss.2\nScaling laws can be useful in deciding how to train a model to a particular per-\nformance, for example by looking at early in the training curve, or performance with\n2For the initial experiment in Kaplan et al. (2020) the precise values were aN= 0.076, Nc= 8.8\u00021013\n(parameters), aD= 0.095, Dc= 5.4\u00021013(tokens), aC= 0.050, Cc= 3.1\u0002108(peta\ufb02op-days). 10.5 \u2022 D EALING WITH SCALE 217\nsmaller amounts of data, to predict what the loss would be if we were to add more\ndata or increase model size. Other aspects of scaling laws can also tell us how much\ndata we need to add when scaling up a model.\n10.5.2 KV Cache\nWe saw in Fig. 9.10 and in Eq. 9.32 (repeated below) how the attention vector can\nbe very ef\ufb01ciently computed in parallel for training, via two matrix multiplications:\nA=softmax\u0012QK|\npdk\u0013\nV (10.13)\nUnfortunately we can\u2019t do quite the same ef\ufb01cient computation in inference as\nin training. That\u2019s because at inference time, we iteratively generate the next tokens\none at a time. For a new token that we have just generated, call it xi, we need to\ncompute its query, key, and values by multiplying by WQ,WK, and WVrespec-\ntively. But it would be a waste of computation time to recompute the key and value\nvectors for all the prior tokens x<i; at prior steps we already computed these key\nand value vectors! So instead of recomputing these, whenever we compute the key\nand value vectors we store them in memory in the KV cache , and then we can just KV cache\ngrab them from the cache when we need them. Fig. 10.7 modi\ufb01es Fig. 9.10 to show\nthe computation that takes place for a single new token, showing which values we\ncan take from the cache rather than recompute.\nq4k1k2k4QKTQKTv1v2v3v4V\nq4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4x==xa4A\n1 x dkdk x N1 x NN x dv1 x dv\nk3\nFigure 10.7 Parts of the attention computation (extracted from Fig. 9.10) showing, in black,\nthe vectors that can be stored in the cache rather than recomputed when computing the atten-\ntion score for the 4th token.\n10.5.3 Parameter Ef\ufb01cient Fine Tuning\nAs we mentioned above, it\u2019s very common to take a language model and give it more\ninformation about a new domain by \ufb01netuning it (continuing to train it to predict\nupcoming words) on some additional data.\nFine-tuning can be very dif\ufb01cult with very large language models, because there\nare enormous numbers of parameters to train; each pass of batch gradient descent\nhas to backpropagate through many many huge layers. This makes \ufb01netuning huge\nlanguage models extremely expensive in processing power, in memory, and in time.\nFor this reason, there are alternative methods that allow a model to be \ufb01netuned\nwithout changing all the parameters. Such methods are called parameter-ef\ufb01cient\n\ufb01ne tuning or sometimes PEFT , because we ef\ufb01ciently select a subset of parametersparameter-\nef\ufb01cient \ufb01ne\ntuning\nPEFT to update when \ufb01netuning. For example we freeze some of the parameters (don\u2019t\nchange them), and only update some particular subset of parameters. 218 CHAPTER 10 \u2022 L ARGE LANGUAGE MODELS\nHere we describe one such model, called LoRA , forLow-RankAdaptation. The LoRA\nintuition of LoRA is that transformers have many dense layers which perform matrix\nmultiplication (for example the WQ,WK,WV,WOlayers in the attention computa-\ntion). Instead of updating these layers during \ufb01netuning, with LoRA we freeze these\nlayers and instead update a low-rank approximation that has fewer parameters.\nConsider a matrix Wof dimensionality [N\u0002d]that needs to be updated during\n\ufb01netuning via gradient descent. Normally this matrix would get updates DWof\ndimensionality [N\u0002d], for updating the N\u0002dparameters after gradient descent. In\nLoRA, we freeze Wand update instead a low-rank decomposition of W. We create\ntwo matrices AandB, where Ahas size [N\u0002r]andBhas size [r\u0002d], and we choose\nrto be quite small, r<<min(d;N). During \ufb01netuning we update AandBinstead\nofW. That is, we replace W+DWwithW+BA. Fig. 10.8 shows the intuition.\nFor replacing the forward pass h=xW, the new forward pass is instead:\nh=xW+xAB (10.14)\nhPretrained WeightsWdkrkABrxd11k\nd\u00d7\nFigure 10.8 The intuition of LoRA. We freeze Wto its pretrained values, and instead \ufb01ne-\ntune by training a pair of matrices AandB, updating those instead of W, and just sum Wand\nthe updated AB.\nLoRA has a number of advantages. It dramatically reduces hardware require-\nments, since gradients don\u2019t have to be calculated for most parameters. The weight\nupdates can be simply added in to the pretrained weights, since BAis the same size\nasW). That means it doesn\u2019t add any time during inference. And it also means it\u2019s\npossible to build LoRA modules for different domains and just swap them in and\nout by adding them in or subtracting them from W.\nIn its original version LoRA was applied just to the matrices in the attention\ncomputation (the WQ,WK,WV, and WOlayers). Many variants of LoRA exist. 10.6 \u2022 P OTENTIAL HARMS FROM LANGUAGE MODELS 219\n10.6 Potential Harms from Language Models\nLarge pretrained neural language models exhibit many of the potential harms dis-\ncussed in Chapter 4 and Chapter 6. Many of these harms become realized when\npretrained language models are used for any downstream task, particularly those\ninvolving text generation, whether question answering, machine translation, or in\nassistive technologies like writing aids or web search query completion, or predic-\ntive typing for email (Olteanu et al., 2020).\nFor example, language models are prone to saying things that are false, a prob-\nlem called hallucination . Language models are trained to generate text that is pre- hallucination\ndictable and coherent, but the training algorithms we have seen so far don\u2019t have\nany way to enforce that the text that is generated is correct or true. This causes\nenormous problems for any application where the facts matter! We\u2019ll return to this\nissue in Chapter 14 where we introduce proposed mitigation methods like retrieval\naugmented generation .\nA second source of harm is that language models can generate toxic language . toxic language\nGehman et al. (2020) show that even completely non-toxic prompts can lead large\nlanguage models to output hate speech and abuse their users. Language models also\ngenerate stereotypes (Cheng et al., 2023) and negative attitudes (Brown et al., 2020;\nSheng et al., 2019) about many demographic groups.\nOne source of biases is the training data. Gehman et al. (2020) shows that large\nlanguage model training datasets include toxic text scraped from banned sites. There\nare other biases than toxicity: the training data is disproportionately generated by\nauthors from the US and from developed countries. Such biased population samples\nlikely skew the resulting generation toward the perspectives or topics of this group\nalone. Furthermore, language models can amplify demographic and other biases in\ntraining data, just as we saw for embedding models in Chapter 6.\nDatasets can be another source of harms. We already saw in Section 10.3.2\nthat using pretraining corpora scraped from the web can lead to harms related to\ncopyright and data consent. We also mentioned that pretraining data can tend to\nhave private information like phone numbers and addresses. This is problematic\nbecause large language models can leak information from their training data. That\nis, an adversary can extract training-data text from a language model such as a per-\nson\u2019s name, phone number, and address (Henderson et al. 2017, Carlini et al. 2021).\nThis becomes even more problematic when large language models are trained on\nextremely sensitive private datasets such as electronic health records.\nLanguage models can also be used by malicious actors for generating text for\nmisinformation , phishing, or other socially harmful activities (Brown et al., 2020).\nMcGuf\ufb01e and Newhouse (2020) show how large language models generate text that\nemulates online extremists, with the risk of amplifying extremist movements and\ntheir attempt to radicalize and recruit.\nFinding ways to mitigate all these harms is an important current research area in\nNLP. At the very least, carefully analyzing the data used to pretrain large language\nmodels is important as a way of understanding issues of toxicity, bias, privacy, and\nfair use, making it extremely important that language models include datasheets\n(page 16) or model cards (page 74) giving full replicable information on the cor-\npora used to train them. Open-source models can specify their exact training data.\nRequirements that models are transparent in such ways is also in the process of being\nincorporated into the regulations of various national governments. 220 CHAPTER 10 \u2022 L ARGE LANGUAGE MODELS\n10.7 Summary\nThis chapter has introduced the large language model, and how it can be built out of\nthe transformer. Here\u2019s a summary of the main points that we covered:\n\u2022 Many NLP tasks\u2014such as question answering, summarization, sentiment,\nand machine translation\u2014can be cast as tasks of word prediction and hence\naddressed with Large language models.\n\u2022 Large language models are generally pretrained on large datasets of 100s of\nbillions of words generally scraped from the web.\n\u2022 These datasets need to be \ufb01ltered for quality and balanced for domains by\nupsampling and downsampling. Addressing some problems with pretraining\ndata, like toxicity, are open research problems.\n\u2022 The choice of which word to generate in large language models is generally\ndone by using a sampling algorithm.\n\u2022 Language models are evaluated by perplexity but there are also evaluations\nof accuracy downstream tasks, and ways to measure other factors like fairness\nand energy use.\n\u2022 There are various computational tricks for making large language models\nmore ef\ufb01cient, such as the CV cache andparameter-ef\ufb01cient \ufb01netuning .\n\u2022 Because of their ability to be used in so many ways, language models also\nhave the potential to cause harms. Some harms include hallucinations, bias,\nstereotypes, misinformation and propaganda, and violations of privacy and\ncopyright.\nBibliographical and Historical Notes\nAs we discussed in Chapter 3, the earliest language models were the n-gram lan-\nguage models developed (roughly simultaneously and independently) by Fred Je-\nlinek and colleagues at the IBM Thomas J. Watson Research Center, and James\nBaker at CMU. It was the Jelinek and the IBM team who \ufb01rst coined the term lan-\nguage model to mean a model of the way any kind of linguistic property (grammar,\nsemantics, discourse, speaker characteristics), in\ufb02uenced word sequence probabil-\nities (Jelinek et al., 1975). They contrasted the language model with the acoustic\nmodel which captured acoustic/phonetic characteristics of phone sequences.\nN-gram language models were very widely used over the next 30 years and more,\nacross a wide variety of NLP tasks like speech recognition and machine translations,\noften as one of multiple components of the model. The contexts for these n-gram\nmodels grew longer, with 5-gram models used quite commonly by very ef\ufb01cient LM\ntoolkits (Stolcke, 2002; Hea\ufb01eld, 2011).\nThe roots of the neural language model lie in multiple places. One was the\napplication in the 1990s, again in Jelinek\u2019s group at IBM Research, of discrimi-\nnative classi\ufb01ers to language models. Roni Rosenfeld in his dissertation (Rosen-\nfeld, 1992) \ufb01rst applied logistic regression (under the name maximum entropy or\nmaxent models) to language modeling in that IBM lab, and published a more fully\nformed version in Rosenfeld (1996). His model integrated various sorts of infor-\nmation in a logistic regression predictor, including n-gram information along with BIBLIOGRAPHICAL AND HISTORICAL NOTES 221\nother features from the context, including distant n-grams and pairs of associated\nwords called trigger pairs . Rosenfeld\u2019s model pre\ufb01gured modern language models\nby being a statistical word predictor trained in a self-supervised manner simply by\nlearning to predict upcoming words in a corpus.\nAnother was the \ufb01rst use of pretrained embeddings to model word meaning in the\nLSA/LSI models (Deerwester et al., 1988). Recall from the history section of Chap-\nter 6 that in LSA (latent semantic analysis) a term-document matrix was trained on a\ncorpus and then singular value decomposition was applied and the \ufb01rst 300 dimen-\nsions were used as a vector embedding to represent words. Landauer et al. (1997)\n\ufb01rst used the word \u201cembedding\u201d. In addition to their development of the idea of pre-\ntraining and of embeddings, the LSA community also developed ways to combine\nLSA embeddings with n-grams in an integrated language model (Bellegarda, 1997;\nCoccaro and Jurafsky, 1998).\nIn a very in\ufb02uential series of papers developing the idea of neural language\nmodels , (Bengio et al. 2000; Bengio et al. 2003; Bengio et al. 2006), Yoshua Ben-\ngio and colleagues drew on the central ideas of both these lines of self-supervised\nlanguage modeling work, (the discriminatively trained word predictor, and the pre-\ntrained embeddings). Like the maxent models of Rosenfeld, Bengio\u2019s model used\nthe next word in running text as its supervision signal. Like the LSA models, Ben-\ngio\u2019s model learned an embedding, but unlike the LSA models did it as part of the\nprocess of language modeling. The Bengio et al. (2003) model was a neural lan-\nguage model: a neural network that learned to predict the next word from prior\nwords, and did so via learning embeddings as part of the prediction process.\nThe neural language model was extended in various ways over the years, perhaps\nmost importantly in the form of the RNN language model of Mikolov et al. (2010)\nand Mikolov et al. (2011). The RNN language model was perhaps the \ufb01rst neural\nmodel that was accurate enough to surpass the performance of a traditional 5-gram\nlanguage model.\nSoon afterwards, Mikolov et al. (2013a) and Mikolov et al. (2013b) proposed\nto simply the hidden layer of these neural net language models to create pretrained\nword2vec word embeddings.\nThe static embedding models like LSA and word2vec instantiated a particular\nmodel of pretraining: a representation was trained on a pretraining dataset, and then\nthe representations could be used in further tasks. The \u2018Dai and Le (2015) and\n(Peters et al., 2018) reframed this idea by proposing models that were pretrained\nusing a language model objective, and then the identical model could be either frozen\nand directly applied for language modeling or further \ufb01netuned still using a language\nmodel objective. For example ELMo used a biLSTM self-supervised on a large\npretrained dataset using a language model objective, then \ufb01netuned on a domain-\nspeci\ufb01c dataset, and then froze the weights and added task-speci\ufb01c heads.\nTransformers were \ufb01rst applied as encoder-decoders (Vaswani et al., 2017) and\nthen to masked language modeling (Devlin et al., 2019) (as we\u2019ll see in Chapter 13\nand Chapter 11). Radford et al. (2019) then showed that the transformer-based au-\ntoregressive language model GPT2 could perform zero-shot on many NLP tasks like\nsummarization and question answering.\nThe technology used for transformer-based language models can also be applied\nto other domains and tasks, like vision, speech, and genetics. the term foundation\nmodel is sometimes used as a more general term for this use of large languagefoundation\nmodel\nmodel technology across domains and areas, when the elements we are computing\nover are not necessarily words. Bommasani et al. (2021) is a broad survey that 222 CHAPTER 10 \u2022 L ARGE LANGUAGE MODELS\nsketches the opportunities and risks of foundation models, with special attention to\nlarge language models. CHAPTER\n11Masked Language Models\nLarvatus prodeo [Masked, I go forward]\nDescartes\nIn the previous two chapters we introduced the transformer and saw how to pre-\ntrain a transformer language model as a causal or left-to-right language model. In\nthis chapter we\u2019ll introduce a second paradigm for pretrained language models, the\nbidirectional transformer encoder, and the most widely-used version, the BERT BERT\nmodel (Devlin et al., 2019). This model is trained via masked language modeling ,masked\nlanguage\nmodelingwhere instead of predicting the following word, we mask a word in the middle and\nask the model to guess the word given the words on both sides. This method thus\nallows the model to see both the right and left context.\nWe also introduced \ufb01netuning in the prior chapter. Here we describe a new \ufb01netuning\nkind of \ufb01netuning, in which we take the transformer network learned by these pre-\ntrained models, add a neural net classi\ufb01er after the top layer of the network, and train\nit on some additional labeled data to perform some downstream task like named\nentity tagging or natural language inference. As before, the intuition is that the\npretraining phase learns a language model that instantiates rich representations of\nword meaning, that thus enables the model to more easily learn (\u2018be \ufb01netuned to\u2019)\nthe requirements of a downstream language understanding task. This aspect of the\npretrain-\ufb01netune paradigm is an instance of what is called transfer learning in ma-transfer\nlearning\nchine learning: the method of acquiring knowledge from one task or domain, and\nthen applying it (transferring it) to solve a new task.\nThe second idea that we introduce in this chapter is the idea of contextual em-\nbeddings : representations for words in context. The methods of Chapter 6 like\nword2vec or GloVe learned a single vector embedding for each unique word win\nthe vocabulary. By contrast, with contextual embeddings, such as those learned by\nmasked language models like BERT, each word wwill be represented by a different\nvector each time it appears in a different context. While the causal language models\nof Chapter 9 also use contextual embeddings, the embeddings created by masked\nlanguage models seem to function particularly well as representations.\n11.1 Bidirectional Transformer Encoders\nLet\u2019s begin by introducing the bidirectional transformer encoder that underlies mod-\nels like BERT and its descendants like RoBERTa (Liu et al., 2019) or SpanBERT\n(Joshi et al., 2020). In Chapter 9 we introduced causal (left-to-right) transformers\nand in Chapter 10 saw how they can serve as the basis for language models that can\nbe applied to autoregressive contextual generation problems like question answering\nor summarization. But this left-to-right nature of these models is also a limitation,\nbecause there are tasks for which it would be useful, when processing a token, to be\nable to peak at future tokens. This is especially true for sequence labeling tasks in 224 CHAPTER 11 \u2022 M ASKED LANGUAGE MODELS\nwhich we want to tag each token with a label, such as the part-of-speech tagging or\nparsing tasks we\u2019ll introduce in future chapters, or tasks like named entity tagging\nwe\u2019ll introduce later in this chapter.\nThebidirectional encoders that we introduce here are a different kind of beast\nthan causal models. The causal models of Chapter 9 are generative models, de-\nsigned to easily generate the next token in a sequence. But the focus of bidirec-\ntional encoders is instead on computing contextualized representations of the input\ntokens. Bidirectional encoders use self-attention to map sequences of input embed-\ndings (x1;:::;xn)to sequences of output embeddings of the same length (h1;:::;hn),\nwhere the output vectors have been contextualized using information from the en-\ntire input sequence. These output embeddings are contextualized representations of\neach input token that are useful across a range of applications where we need to do\na classi\ufb01cation or a decision based on the token in context.\nRemember that we said the models of Chapter 9 are sometimes called decoder-\nonly, because they correspond to the decoder part of the encoder-decoder model we\nwill introduce in Chapter 13. By contrast, the masked language models of this chap-\nter are sometimes called encoder-only , because they produce an encoding for each\ninput token but generally aren\u2019t used to produce running text by decoding/sampling.\nThat\u2019s an important point: masked language models are not used for generation.\nThey are generally instead used for interpretative tasks.\n11.1.1 The architecture for bidirectional masked models\nLet\u2019s \ufb01rst discuss the overall architecture. Bidirectional transformer-based language\nmodels differ in two ways from the causal transformers in the previous chapters. The\n\ufb01rst is that the attention function isn\u2019t causal; the attention for a token ican look at\nfollowing tokens i+1 and so on. The second is that the training is slightly different\nsince we are predicting something in the middle of our text rather than at the end.\nWe\u2019ll discuss the \ufb01rst here and the second in the following section.\nFig. 11.1a, reproduced here from Chapter 9, shows the information \ufb02ow in the\nleft-to-right approach of Chapter 9. The attention computation at each token is based\non the preceding (and current) input tokens, ignoring potentially useful information\nlocated to the right of the token under consideration. Bidirectional encoders over-\ncome this limitation by allowing the attention mechanism to range over the entire\ninput, as shown in Fig. 11.1b.\na) A causal self-attention layerb) A bidirectional self-attention layer\nattentionattentionattentionattentionattentiona1a2a3a4a5x3x4x5x1x2\nattentionattentionattentionattentionattentiona1a2a3a4a5x3x4x5x1x2\nFigure 11.1 (a) The causal transformer from Chapter 9, highlighting the attention computation at token 3. The\nattention value at each token is computed using only information seen earlier in the context. (b) Information\n\ufb02ow in a bidirectional attention model. In processing each token, the model attends to all inputs, both before\nand after the current one. So attention for token 3 can draw on information from following tokens.\nThe implementation is very simple! We simply remove the attention masking\nstep that we introduced in Eq. 9.32. Recall from Chapter 9 that we had to mask the\nQK|matrix for causal transformers so that attention couldn\u2019t look at future tokens 11.1 \u2022 B IDIRECTIONAL TRANSFORMER ENCODERS 225\n(repeated from Eq. 9.32):\nA=softmax\u0012\nmask\u0012QK|\npdk\u0013\u0013\nV (11.1)\nq1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3NN\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\nq1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3NNq1\u2022k2q1\u2022k3q1\u2022k4q2\u2022k3q2\u2022k4q3\u2022k4\n(a) (b)\nFigure 11.2 The N\u0002NQK|matrix showing the qi\u0001kjvalues, with the upper-triangle\nportion of the comparisons matrix zeroed out (set to \u0000\u00a5, which the softmax will turn to\nzero).\nFig. 11.2 shows the masked version of QK|and the unmasked version. For bidi-\nrectional attention, we used the unmasked version of Fig. 11.2b. Thus the attention\ncomputation for bidirectional attention is exactly the same as Eq. 11.1 but with the\nmask removed:\nA=softmax\u0012QK|\npdk\u0013\nV (11.2)\nOtherwise, the attention computation is identical to what we saw in Chapter 9, as\nis the transformer block architecture (the feedforward layer, layer norm, and so on).\nAs in Chapter 9, the input is also a series of subword tokens, usually computed by\nWordPiece or SentencePiece Unigram LM tokenization (two of the large family of\nsubword tokenization algorithms that includes the BPE algorithm we saw in Chap-\nter 2). That means every input sentence \ufb01rst has to be tokenized, and all further\nprocessing takes place on subword tokens rather than words. This will require, as\nwe\u2019ll see in the third part of the textbook, that for some NLP tasks that require no-\ntions of words (like parsing) we will occasionally need to map subwords back to\nwords.\nTo make this more concrete, the original English-only bidirectional transformer\nencoder model, BERT (Devlin et al., 2019), consisted of the following:\n\u2022 An English-only subword vocabulary consisting of 30,000 tokens generated\nusing the WordPiece algorithm (Schuster and Nakajima, 2012).\n\u2022 Hidden layers of dimensionality d=768,\n\u2022 12 layers of transformer blocks, with 12 (bidirectional) multihead attention\nlayers each.\n\u2022 The resulting model has about 100M parameters.\nThe larger multilingual XLM-RoBERTa model, trained on 100 languages, has\n\u2022 A multilingual subword vocabulary with 250,000 tokens generated using the\nSentencePiece Unigram LM algorithm (Kudo and Richardson, 2018b).\n\u2022 24 layers of transformer blocks, with 16 multihead attention layers each 226 CHAPTER 11 \u2022 M ASKED LANGUAGE MODELS\n\u2022 Hidden layers of size 1024\n\u2022 Input context window of 512 tokens\n\u2022 The resulting model has about 550M parameters.\nNote that 550M parameters is relatively small as large language models go\n(Llama 3 has 405B parameters, so is 3 orders of magnitude bigger). Indeed, masked\nlanguage models tend to be much smaller than causal language models.\n11.2 Training Bidirectional Encoders\nWe trained causal transformer language models in Chapter 9 by making them it-\neratively predict the next word in a text. But eliminating the causal mask in at-\ntention makes the guess-the-next-word language modeling task trivial\u2014the answer\nis directly available from the context\u2014so we\u2019re in need of a new training scheme.\nInstead of trying to predict the next word, the model learns to perform a \ufb01ll-in-the-\nblank task, technically called the cloze task (Taylor, 1953). To see this, let\u2019s return cloze task\nto the motivating example from Chapter 3. Instead of predicting which words are\nlikely to come next in this example:\nThe water of Walden Pond is so beautifully\nwe\u2019re asked to predict a missing item given the rest of the sentence.\nThe of Walden Pond is so beautifully ...\nThat is, given an input sequence with one or more elements missing, the learning\ntask is to predict the missing elements. More precisely, during training the model is\ndeprived of one or more elements of an input sequence and must generate a proba-\nbility distribution over the vocabulary for each of the missing items. We then use the\ncross-entropy loss from each of the model\u2019s predictions to drive the learning process.\nThis approach can be generalized to any of a variety of methods that corrupt the\ntraining input and then asks the model to recover the original input. Examples of the\nkinds of manipulations that have been used include masks, substitutions, reorder-\nings, deletions, and extraneous insertions into the training text. The general name\nfor this kind of training is called denoising : we corrupt (add noise to) the input in denoising\nsome way (by masking a word, or putting in an incorrect word) and the goal of the\nsystem is to remove the noise.\n11.2.1 Masking Words\nThe original approach to training bidirectional encoders is called Masked Language\nModeling (MLM) (Devlin et al., 2019). As with the language model training meth-Masked\nLanguage\nModelingods we\u2019ve already seen, MLM uses unannotated text from a large corpus. Here, the\nmodel is presented with a series of sentences from the training corpus where a ran-\ndom sample of tokens from each training sequence is selected for use in the learning\ntask. Once chosen, a token is used in one of three ways:\n\u2022 It is replaced with the special vocabulary token named [MASK] .\n\u2022 It is replaced with another token from the vocabulary, randomly sampled\nbased on token unigram probabilities.\n\u2022 It is left unchanged. 11.2 \u2022 T RAINING BIDIRECTIONAL ENCODERS 227\nIn BERT, 15% of the input tokens in a training sequence are sampled for learning.\nOf these, 80% are replaced with [MASK] , 10% are replaced with randomly selected\ntokens, and the remaining 10% are left unchanged.\nThe MLM training objective is to predict the original inputs for each of the\nmasked tokens using a bidirectional encoder of the kind described in the last section.\nThe cross-entropy loss from these predictions drives the training process for all the\nparameters in the model. Note that all of the input tokens play a role in the self-\nattention process, but only the sampled tokens are used for learning.\nMore speci\ufb01cally, the original input sequence is \ufb01rst tokenized using a subword\nmodel. The sampled items which drive the learning process are chosen among the\ninput tokens. Word embeddings for all of the tokens in the input are retrieved from\ntheEembedding matrix and combined with positional embeddings to form the input\nto the transformer, passed through the stack of transformer blocks, and then the\nlanguage modeling head.\nLM Head with Softmax over Vocabulary\nSo[mask]and[mask]for longthanksCE Loss\nall apricot \ufb01shthe\nToken +Positional EmbeddingsSolongandthanksfor all \ufb01shthe\nBidirectional Transformer Encoder+p1+++++++p2p3p4p5p6p7p8z1z2z3z4z5z6z7z8\nFigure 11.3 Masked language model training. In this example, three of the input tokens are selected, two of\nwhich are masked and the third is replaced with an unrelated word. The probabilities assigned by the model to\nthese three items are used as the training loss. The other 5 tokens don\u2019t play a role in training loss.\nFig. 11.3 illustrates this approach with a simple example. Here, long,thanks and\nthehave been sampled from the training sequence, with the \ufb01rst two masked and the\nreplaced with the randomly sampled token apricot . The resulting embeddings are\npassed through a stack of bidirectional transformer blocks. Recall from Section 9.5\nin Chapter 9 that to produce a probability distribution over the vocabulary for each\nof the masked tokens, the language modeling head takes the output vector hL\nifrom\nthe \ufb01nal transformer layer Lfor each masked token i, multiplies it by the unembed-\nding layer ETto produce the logits u, and then uses softmax to turn the logits into\nprobabilities yover the vocabulary:\nui=hL\niET(11.3)\nyi=softmax (ui) (11.4)\nWith a predicted probability distribution for each masked item, we can use cross-\nentropy to compute the loss for each masked item\u2014the negative log probability\nassigned to the actual masked word, as shown in Fig. 11.3. More formally, for a 228 CHAPTER 11 \u2022 M ASKED LANGUAGE MODELS\ngiven vector of input tokens in a sentence or batch be x, let the set of tokens that are\nmasked be M, the version of that sentence with some tokens replaced by masks be\nxmask, and the sequence of output vectors be h. For a given input token xi, such as\nthe word long in Fig. 11.3, the loss is the probability of the correct word long, given\nxmask(as summarized in the single output vector hL\ni):\nLMLM(xi) =\u0000logP(xijhL\ni)\nThe gradients that form the basis for the weight updates are based on the average\nloss over the sampled learning items from a single training sequence (or batch of\nsequences).\nLMLM=\u00001\njMjX\ni2MlogP(xijhL\ni)\nNote that only the tokens in Mplay a role in learning; the other words play no role\nin the loss function, so in that sense BERT and its descendents are inef\ufb01cient; only\n15% of the input samples in the training data are actually used for training weights.\n1\n11.2.2 Next Sentence Prediction\nThe focus of mask-based learning is on predicting words from surrounding contexts\nwith the goal of producing effective word-level representations. However, an im-\nportant class of applications involves determining the relationship between pairs of\nsentences. These include tasks like paraphrase detection (detecting if two sentences\nhave similar meanings), entailment (detecting if the meanings of two sentences en-\ntail or contradict each other) or discourse coherence (deciding if two neighboring\nsentences form a coherent discourse).\nTo capture the kind of knowledge required for applications such as these, some\nmodels in the BERT family include a second learning objective called Next Sen-\ntence Prediction (NSP). In this task, the model is presented with pairs of sentencesNext Sentence\nPrediction\nand is asked to predict whether each pair consists of an actual pair of adjacent sen-\ntences from the training corpus or a pair of unrelated sentences. In BERT, 50% of\nthe training pairs consisted of positive pairs, and in the other 50% the second sen-\ntence of a pair was randomly selected from elsewhere in the corpus. The NSP loss\nis based on how well the model can distinguish true pairs from random pairs.\nTo facilitate NSP training, BERT introduces two special tokens to the input rep-\nresentation (tokens that will prove useful for \ufb01netuning as well). After tokenizing\nthe input with the subword model, the token [CLS] is prepended to the input sen-\ntence pair, and the token [SEP] is placed between the sentences and after the \ufb01nal\ntoken of the second sentence. Finally, embeddings representing the \ufb01rst and second\nsegments of the input are added to the word and positional embeddings to allow the\nmodel to more easily distinguish the input sentences.\nDuring training, the output vector hL\nCLSfrom the \ufb01nal layer associated with the\n[CLS] token represents the next sentence prediction. As with the MLM objective,\nwe add a special head, in this case an NSP head, which consists of a learned set of\nclassi\ufb01cation weights WNSP2Rd\u00022that produces a two-class prediction from the\nraw[CLS] vector hL\nCLS:\nyi=softmax (hL\nCLSWNSP)\n1There are members of the BERT family like ELECTRA that do use all examples for training (Clark\net al., 2020b). 11.2 \u2022 T RAINING BIDIRECTIONAL ENCODERS 229\nCross entropy is used to compute the NSP loss for each sentence pair presented\nto the model. Fig. 11.4 illustrates the overall NSP training setup. In BERT, the NSP\nloss was used in conjunction with the MLM training objective to form \ufb01nal loss.\nCancelmy\ufb02ight[SEP] 1CE Loss\nAnd the \nBidirectional Transformer Encoder\np1p2p3p4p5p6p7p8[CLS]++s1NSPHeadToken +Segment +PositionalEmbeddings hotel p9[SEP] ++s1s1s1s1s2s2s2s2++++++++++++++hCLS\nFigure 11.4 An example of the NSP loss calculation.\n11.2.3 Training Regimes\nBERT and other early transformer-based language models were trained on about\n3.3 billion words (a combination of English Wikipedia and a corpus of book texts\ncalled BooksCorpus (Zhu et al., 2015) that is no longer used for intellectual property\nreasons). Modern masked language models are now trained on much larger datasets\nof web text, \ufb01ltered a bit, and augmented by higher-quality data like Wikipedia,\nthe same as those we discussed for the causal large language models of Chapter 9.\nMultilingual models similarly use webtext and multilingual Wikipedia. For example\nthe XLM-R model was trained on about 300 billion tokens in 100 languages, taken\nfrom the web via Common Crawl ( https://commoncrawl.org/ ).\nTo train the original BERT models, pairs of text segments were selected from\nthe training corpus according to the next sentence prediction 50/50 scheme. Pairs\nwere sampled so that their combined length was less than the 512 token input. To-\nkens within these sentence pairs were then masked using the MLM approach with\nthe combined loss from the MLM and NSP objectives used for a \ufb01nal loss. Ap-\nproximately 40 passes (epochs) over the training data was required for the model to\nconverge.\nSome models, like the RoBERTa model, drop the next sentence prediction ob-\njective, and therefore change the training regime a bit. Instead of sampling pairs of\nsentence, the input is simply a series of contiguous sentences. If the document runs\nout before 512 tokens are reached, an extra separator token is added, and sentences\nfrom the next document are packed in, until we reach a total of 512 tokens. Usually\nlarge batch sizes are used, between 8K and 32K tokens.\nMultilingual models have an additional decision to make: what data to use to\nbuild the vocabulary? Recall that all language models use subword tokenization\n(BPE or SentencePiece Unigram LM are the two most common algorithms). What\ntext should be used to learn this multilingual tokenization, given that it\u2019s easier to get\nmuch more text in some languages than others? One option would be to create this\nvocabulary-learning dataset by sampling sentences from our training data (perhaps 230 CHAPTER 11 \u2022 M ASKED LANGUAGE MODELS\nweb text from Common Crawl), randomly. In that case we will choose a lot of sen-\ntences from languages like languages with lots of web representation like English,\nand the tokens will be biased toward rare English tokens instead of creating frequent\ntokens from languages with less data. Instead, it is common to divide the training\ndata into subcorpora of Ndifferent languages, compute the number of sentences ni\nof each language i, and readjust these probabilities so as to upweight the probability\nof less-represented languages (Lample and Conneau, 2019). The new probability of\nselecting a sentence from each of the Nlanguages (whose prior frequency is ni) is\nfqigi=1:::N, where:\nqi=pa\niPN\nj=1pa\njwith pi=niPN\nk=1nk(11.5)\nRecall from (6.32) in Chapter 6 that an avalue between 0 and 1 will give higher\nweight to lower probability samples. Conneau et al. (2020) show that a=0:3 works\nwell to give rare languages more inclusion in the tokenization, resulting in better\nmultilingual performance overall.\nThe result of this pretraining process consists of both learned word embeddings,\nas well as all the parameters of the bidirectional encoder that are used to produce\ncontextual embeddings for novel inputs.\nFor many purposes, a pretrained multilingual model is more practical than a\nmonolingual model, since it avoids the need to build many (a hundred!) separate\nmonolingual models. And multilingual models can improve performance on low-\nresourced languages by leveraging linguistic information from a similar language in\nthe training data that happens to have more resources. Nonetheless, when the num-\nber of languages grows very large, multilingual models exhibit what has been called\nthecurse of multilinguality (Conneau et al., 2020): the performance on each lan-\nguage degrades compared to a model training on fewer languages. Another problem\nwith multilingual models is that they \u2018have an accent\u2019: grammatical structures in\nhigher-resource languages (often English) bleed into lower-resource languages; the\nvast amount of English language in training makes the model\u2019s representations for\nlow-resource languages slightly more English-like (Papadimitriou et al., 2023).\n11.3 Contextual Embeddings\nGiven a pretrained language model and a novel input sentence, we can think of the\nsequence of model outputs as constituting contextual embeddings for each token incontextual\nembeddings\nthe input. These contextual embeddings are vectors representing some aspect of the\nmeaning of a token in context, and can be used for any task requiring the meaning of\ntokens or words. More formally, given a sequence of input tokens x1;:::;xn, we can\nuse the output vector zifrom the \ufb01nal layer of the model as a representation of the\nmeaning of token xiin the context of sentence x1;:::;xn. Or instead of just using the\nvector zifrom the \ufb01nal layer of the model, it\u2019s common to compute a representation\nforxiby averaging the output tokens zifrom each of the last four layers of the model.\nJust as we used static embeddings like word2vec in Chapter 6 to represent the\nmeaning of words, we can use contextual embeddings as representations of word\nmeanings in context for any task that might require a model of word meaning. Where\nstatic embeddings represent the meaning of word types (vocabulary entries), contex-\ntual embeddings represent the meaning of word instances : instances of a particular 11.3 \u2022 C ONTEXTUAL EMBEDDINGS 231\n[CLS]SolongandthanksforallhL1hLCLShL2hL3hL4hL5hL6\nEi+\nEi+\nEi+\nEi+\nEi+\nEi+\nEi+\nFigure 11.5 The output of a BERT-style model is a contextual embedding vector hL\nifor\neach input token xi.\nword type in a particular context. Thus where word2vec had a single vector for each\nword type, contextual embeddings provide a single vector for each instance of that\nword type in its sentential context. Contextual embeddings can thus be used for\ntasks like measuring the semantic similarity of two words in context, and are useful\nin linguistic tasks that require models of word meaning.\n11.3.1 Contextual Embeddings and Word Sense\nWords are ambiguous : the same word can be used to mean different things. In ambiguous\nChapter 6 we saw that the word \u201cmouse\u201d can mean (1) a small rodent, or (2) a hand-\noperated device to control a cursor. The word \u201cbank\u201d can mean: (1) a \ufb01nancial\ninstitution or (2) a sloping mound. We say that the words \u2018mouse\u2019 or \u2018bank\u2019 are\npolysemous (from Greek \u2018many senses\u2019, poly- \u2018many\u2019 + sema , \u2018sign, mark\u2019).2\nAsense (orword sense ) is a discrete representation of one aspect of the meaning word sense\nof a word. We can represent each sense with a superscript: bank1andbank2,\nmouse1andmouse2. These senses can be found listed in online thesauruses (or\nthesauri) like WordNet (Fellbaum, 1998), which has datasets in many languages WordNet\nlisting the senses of many words. In context, it\u2019s easy to see the different meanings:\nmouse1: .... a mouse controlling a computer system in 1968.\nmouse2: .... a quiet animal like a mouse\nbank1: ...a bank can hold the investments in a custodial account ...\nbank2: ...as agriculture burgeons on the east bank , the river ...\nThis fact that context disambiguates the senses of mouse andbank above can\nalso be visualized geometrically. Fig. 11.6 shows a two-dimensional projection of\nmany instances of the BERT embeddings of the word diein English and German.\nEach point in the graph represents the use of diein one input sentence. We can\nclearly see at least two different English senses of die(the singular of dice and the\nverb to die , as well as the German article, in the BERT embedding space.\n2The word polysemy itself is ambiguous; you may see it used in a different way, to refer only to cases\nwhere a word\u2019s senses are related in some structured way, reserving the word homonymy to mean sense\nambiguities with no relation between the senses (Haber and Poesio, 2020). Here we will use \u2018polysemy\u2019\nto mean any kind of sense ambiguity, and \u2018structured polysemy\u2019 for polysemy with sense relations. 232 CHAPTER 11 \u2022 M ASKED LANGUAGE MODELS\nFigure 4: Embeddings for the word \"die\" in different contexts, visualized with UMAP. Sample points\nare annotated with corresponding sentences. Overall annotations (blue text) are added as a guide.\n4.1 Visualization of word senses\nOur \ufb01rst experiment is an exploratory visualization of how word sense affects context embeddings.\nFor data on different word senses, we collected all sentences used in the introductions to English-\nlanguage Wikipedia articles. (Text outside of introductions was frequently fragmentary.) We created\nan interactive application, which we plan to make public. A user enters a word, and the system\nretrieves 1,000 sentences containing that word. It sends these sentences to BERT-base as input, and\nfor each one it retrieves the context embedding for the word from a layer of the user\u2019s choosing.\nThe system visualizes these 1,000 context embeddings using UMAP [ 15], generally showing clear\nclusters relating to word senses. Different senses of a word are typically spatially separated, and\nwithin the clusters there is often further structure related to \ufb01ne shades of meaning. In Figure 4, for\nexample, we not only see crisp, well-separated clusters for three meanings of the word \u201cdie,\u201d but\nwithin one of these clusters there is a kind of quantitative scale, related to the number of people\ndying. See Appendix 6.4 for further examples. The apparent detail in the clusters we visualized raises\ntwo immediate questions. First, is it possible to \ufb01nd quantitative corroboration that word senses are\nwell-represented? Second, how can we resolve a seeming contradiction: in the previous section, we\nsaw how position represented syntax; yet here we see position representing semantics.\n4.2 Measurement of word sense disambiguation capability\nThe crisp clusters seen in visualizations such as Figure 4 suggest that BERT may create simple,\neffective internal representations of word senses, putting different meanings in different locations. To\ntest this hypothesis quantitatively, we test whether a simple classi\ufb01er on these internal representations\ncan perform well at word-sense disambiguation (WSD).\nWe follow the procedure described in [ 20], which performed a similar experiment with the ELMo\nmodel. For a given word with nsenses, we make a nearest-neighbor classi\ufb01er where each neighbor is\nthe centroid of a given word sense\u2019s BERT-base embeddings in the training data. To classify a new\nword we \ufb01nd the closest of these centroids, defaulting to the most commonly used sense if the word\nwas not present in the training data. We used the data and evaluation from [ 21]: the training data was\nSemCor [17] (33,362 senses), and the testing data was the suite described in [21] (3,669 senses).\nThe simple nearest-neighbor classi\ufb01er achieves an F1 score of 71.1, higher than the current state of\nthe art (Table 1), with the accuracy monotonically increasing through the layers. This is a strong\nsignal that context embeddings are representing word-sense information. Additionally, an even higher\nscore of 71.5 was obtained using the technique described in the following section.\n6\nFigure 11.6 Each blue dot shows a BERT contextual embedding for the word diefrom different sentences\nin English and German, projected into two dimensions with the UMAP algorithm. The German and English\nmeanings and the different English senses fall into different clusters. Some sample points are shown with the\ncontextual sentence they came from. Figure from Coenen et al. (2019).\nThus while thesauruses like WordNet give discrete lists of senses, embeddings\n(whether static or contextual) offer a continuous high-dimensional model of meaning\nthat, although it can be clustered, doesn\u2019t divide up into fully discrete senses.\nWord Sense Disambiguation\nThe task of selecting the correct sense for a word is called word sense disambigua-\ntion, orWSD . WSD algorithms take as input a word in context and a \ufb01xed inventoryword sense\ndisambiguation\nWSD of potential word senses (like the ones in WordNet) and outputs the correct word\nsense in context. Fig. 11.7 sketches out the task.\nanelectricguitarandbassplayerstando\ufb00toonesideelectric1: using electricityelectric2:  tenseelectric3: thrillingguitar1 bass1: low range\u2026bass4: sea \ufb01sh\u2026 bass7: instrument\u2026player1: in gameplayer2: musician player3: actor\u2026stand1: upright\u2026stand5: bear\u2026 stand10: put upright\u2026side1: relative region\u2026side3: of body\u2026 side11: slope\u2026x1y1x2y2x3y3y4y5y6\nx4x5x6\nFigure 11.7 The all-words WSD task, mapping from input words ( x) to WordNet senses\n(y). Figure inspired by Chaplot and Salakhutdinov (2018).\nWSD can be a useful analytic tool for text analysis in the humanities and social\nsciences, and word senses can play a role in model interpretability for word repre-\nsentations. Word senses also have interesting distributional properties. For example 11.3 \u2022 C ONTEXTUAL EMBEDDINGS 233\na word often is used in roughly the same sense through a discourse, an observation\ncalled the one sense per discourse rule (Gale et al., 1992a).one sense per\ndiscourse\nThe best performing WSD algorithm is a simple 1-nearest-neighbor algorithm\nusing contextual word embeddings, due to Melamud et al. (2016) and Peters et al.\n(2018). At training time we pass each sentence in some sense-labeled dataset (like\nthe SemCore or SenseEval datasets in various languages) through any contextual\nembedding (e.g., BERT) resulting in a contextual embedding for each labeled token.\n(There are various ways to compute this contextual embedding vifor a token i; for\nBERT it is common to pool multiple layers by summing the vector representations\nofifrom the last four BERT layers). Then for each sense sof any word in the corpus,\nfor each of the ntokens of that sense, we average their ncontextual representations\nvito produce a contextual sense embedding vsfors:\nvs=1\nnX\nivi8vi2tokens (s) (11.6)\nAt test time, given a token of a target word tin context, we compute its contextual\nembedding tand choose its nearest neighbor sense from the training set, i.e., the\nsense whose sense embedding has the highest cosine with t:\nsense(t) =argmax\ns2senses (t)cosine (t;vs) (11.7)\nFig. 11.8 illustrates the model.\nI  found  the  jar  emptycIcfoundfind1vcthecjarcemptyfind9vfind5vfind4vENCODER\nFigure 11.8 The nearest-neighbor algorithm for WSD. In green are the contextual embed-\ndings precomputed for each sense of each word; here we just show a few of the senses for\n\ufb01nd. A contextual embedding is computed for the target word found , and then the nearest\nneighbor sense (in this case \ufb01nd9v) is chosen. Figure inspired by Loureiro and Jorge (2019).\n11.3.2 Contextual Embeddings and Word Similarity\nIn Chapter 6 we introduced the idea that we could measure the similarity of two\nwords by considering how close they are geometrically, by using the cosine as a\nsimilarity function. The idea of meaning similarity is also clear geometrically in the\nmeaning clusters in Fig. 11.6; the representation of a word which has a particular\nsense in a context is closer to other instances of the same sense of the word. Thus we\noften measure the similarity between two instances of two words in context (or two\ninstances of the same word in two different contexts) by using the cosine between\ntheir contextual embeddings. 234 CHAPTER 11 \u2022 M ASKED LANGUAGE MODELS\nUsually some transformations to the embeddings are required before computing\ncosine. This is because contextual embeddings (whether from masked language\nmodels or from autoregressive ones) have the property that the vectors for all words\nare extremely similar. If we look at the embeddings from the \ufb01nal layer of BERT\nor other models, embeddings for instances of any two randomly chosen words will\nhave extremely high cosines that can be quite close to 1, meaning all word vectors\ntend to point in the same direction. The property of vectors in a system all tending\nto point in the same direction is known as anisotropy . Ethayarajh (2019) de\ufb01nes\ntheanisotropy of a model as the expected cosine similarity of any pair of words in anisotropy\na corpus. The word \u2018isotropy\u2019 means uniformity in all directions, so in an isotropic\nmodel, the collection of vectors should point in all directions and the expected cosine\nbetween a pair of random embeddings would be zero. Timkey and van Schijndel\n(2021) show that one cause of anisotropy is that cosine measures are dominated by\na small number of dimensions of the contextual embedding whose values are very\ndifferent than the others: these rogue dimensions have very large magnitudes and\nvery high variance.\nTimkey and van Schijndel (2021) shows that we can make the embeddings more\nisotropic by standardizing (z-scoring) the vectors, i.e., subtracting the mean and\ndividing by the variance. Given a set Cof all the embeddings in some corpus, each\nwith dimensionality d(i.e., x2Rd), the mean vector m2Rdis:\nm=1\njCjX\nx2Cx (11.8)\nThe standard deviation in each dimension s2Rdis:\ns=s\n1\njCjX\nx2C(x\u0000m)2(11.9)\nThen each word vector xis replaced by a standardized version z:\nz=x\u0000m\ns(11.10)\nOne problem with cosine that is not solved by standardization is that cosine tends\nto underestimate human judgments on similarity of word meaning for very frequent\nwords (Zhou et al., 2022).\nIn the next section we\u2019ll see the most common use of contextual representations:\nas representations of words or even entire sentences that can be the inputs to classi-\n\ufb01ers in the \ufb01netuning process for downstream NLP applications.\n11.4 Fine-Tuning for Classi\ufb01cation\nThe power of pretrained language models lies in their ability to extract generaliza-\ntions from large amounts of text\u2014generalizations that are useful for myriad down-\nstream applications. There are two ways to make practical use of the generalizations\nto solve downstream tasks. The most common way is to use natural language to\nprompt the model, putting it in a state where it contextually generates what we\nwant. We\u2019ll introduce prompting in Chapter 12.\nIn this section we explore an alternative way to use pretrained language models\nfor downstream applications: a version of the \ufb01netuning paradigm from Chapter 10. \ufb01netuning 11.4 \u2022 F INE-TUNING FOR CLASSIFICATION 235\nIn the kind of \ufb01netuning used for masked language models, we add application-\nspeci\ufb01c circuitry (often called a special head ) on top of pretrained models, taking\ntheir output as its input. The \ufb01netuning process consists of using labeled data about\nthe application to train these additional application-speci\ufb01c parameters. Typically,\nthis training will either freeze or make only minimal adjustments to the pretrained\nlanguage model parameters.\nThe following sections introduce \ufb01netuning methods for the most common kinds\nof applications: sequence classi\ufb01cation, sentence-pair classi\ufb01cation, and sequence\nlabeling.\n11.4.1 Sequence Classi\ufb01cation\nThe task of sequence classi\ufb01cation is to classify an entire sequence of text with a\nsingle label. This set of tasks is commonly called text classi\ufb01cation , like sentiment\nanalysis or spam detection (Chapter 4) in which we classify a text into two or three\nclasses (like positive or negative), as well as classi\ufb01cation tasks with a large number\nof categories, like document-level topic classi\ufb01cation.\nFor sequence classi\ufb01cation we represent the entire input to be classi\ufb01ed by a\nsingle vector. We can represent a sequence in various ways. One way is to take\nthe sum or the mean of the last output vector from each token int he sequence.\nFor BERT, we instead add a new unique token to the vocabulary called [CLS] , and\nprepended it to the start of all input sequences, both during pretraining and encoding.\nThe output vector in the \ufb01nal layer of the model for the [CLS] input represents\nthe entire input sequence and serves as the input to a classi\ufb01er head , a logistic classi\ufb01er head\nregression or neural network classi\ufb01er that makes the relevant decision.\nAs an example, let\u2019s return to the problem of sentiment classi\ufb01cation. to \ufb01netun-\ning a classi\ufb01er for this application involves learning a set of weights, WC, to map the\noutput vector for the [CLS] token\u2014 hL\nCLS\u2014to a set of scores over the possible senti-\nment classes. Assuming a three-way sentiment classi\ufb01cation task (positive, negative,\nneutral) and dimensionality das the model dimension, WCwill be of size [d\u00023]. To\nclassify a document, we pass the input text through the pretrained language model to\ngenerate hL\nCLS, multiply it by WC, and pass the resulting vector through a softmax.\ny=softmax (hL\nCLSWC) (11.11)\nFinetuning the values in WCrequires supervised training data consisting of input\nsequences labeled with the appropriate sentiment class. Training proceeds in the\nusual way; cross-entropy loss between the softmax output and the correct answer is\nused to drive the learning that produces WC.\nA key difference from what we\u2019ve seen earlier with neural classi\ufb01ers is that this\nloss can be used to not only learn the weights of the classi\ufb01er, but also to update the\nweights for the pretrained language model itself. In practice, reasonable classi\ufb01ca-\ntion performance is typically achieved with only minimal changes to the language\nmodel parameters, often limited to updates over the \ufb01nal few layers of the trans-\nformer. Fig. 11.9 illustrates this overall approach to sequence classi\ufb01cation.\n11.4.2 Sequence-Pair Classi\ufb01cation\nAs mentioned in Section 11.2.2, an important type of problem involves the classi\ufb01ca-\ntion of pairs of input sequences. Practical applications that fall into this class include\nparaphrase detection (are the two sentences paraphrases of each other?), logical en- 236 CHAPTER 11 \u2022 M ASKED LANGUAGE MODELS\n[CLS]entirelypredictableandlacksenergy\nBidirectional Transformer EncoderhCLS\nEi+\nEi+\nEi+\nEi+\nEi+\nEi+sentimentclassi\ufb01cation headWCy\nFigure 11.9 Sequence classi\ufb01cation with a bidirectional transformer encoder. The output vector for the\n[CLS] token serves as input to a simple classi\ufb01er.\ntailment (does sentence A logically entail sentence B?), and discourse coherence\n(how coherent is sentence B as a follow-on to sentence A?).\nFine-tuning an application for one of these tasks proceeds just as with pretrain-\ning using the NSP objective. During \ufb01netuning, pairs of labeled sentences from a\nsupervised \ufb01netuning set are presented to the model, and run through all the layers\nof the model to produce the houtputs for each input token. As with sequence classi-\n\ufb01cation, the output vector associated with the prepended [CLS] token represents the\nmodel\u2019s view of the input pair. And as with NSP training, the two inputs are sepa-\nrated by the [SEP] token. To perform classi\ufb01cation, the [CLS] vector is multiplied\nby a set of learning classi\ufb01cation weights and passed through a softmax to generate\nlabel predictions, which are then used to update the weights.\nAs an example, let\u2019s consider an entailment classi\ufb01cation task with the Multi-\nGenre Natural Language Inference (MultiNLI) dataset (Williams et al., 2018). In\nthe task of natural language inference orNLI, also called recognizing textualnatural\nlanguage\ninferenceentailment , a model is presented with a pair of sentences and must classify the re-\nlationship between their meanings. For example in the MultiNLI corpus, pairs of\nsentences are given one of 3 labels: entails ,contradicts andneutral . These labels\ndescribe a relationship between the meaning of the \ufb01rst sentence (the premise) and\nthe meaning of the second sentence (the hypothesis). Here are representative exam-\nples of each class from the corpus:\n\u2022Neutral\na: Jon walked back to the town to the smithy.\nb: Jon traveled back to his hometown.\n\u2022Contradicts\na: Tourist Information of\ufb01ces can be very helpful.\nb: Tourist Information of\ufb01ces are never of any help.\n\u2022Entails\na: I\u2019m confused.\nb: Not all of it is very clear to me. 11.5 \u2022 F INE-TUNING FOR SEQUENCE LABELLING : NAMED ENTITY RECOGNITION 237\nA relationship of contradicts means that the premise contradicts the hypothesis; en-\ntails means that the premise entails the hypothesis; neutral means that neither is\nnecessarily true. The meaning of these labels is looser than strict logical entailment\nor contradiction indicating that a typical human reading the sentences would most\nlikely interpret the meanings in this way.\nTo \ufb01netune a classi\ufb01er for the MultiNLI task, we pass the premise/hypothesis\npairs through a bidirectional encoder as described above and use the output vector\nfor the[CLS] token as the input to the classi\ufb01cation head. As with ordinary sequence\nclassi\ufb01cation, this head provides the input to a three-way classi\ufb01er that can be trained\non the MultiNLI training corpus.\n11.5 Fine-Tuning for Sequence Labelling: Named En-\ntity Recognition\nIn sequence labeling, the network\u2019s task is to assign a label chosen from a small\n\ufb01xed set of labels to each token in the sequence. One of the most common sequence\nlabeling task is named entity recognition .\n11.5.1 Named Entities\nAnamed entity is, roughly speaking, anything that can be referred to with a proper named entity\nname: a person, a location, an organization. The task of named entity recognitionnamed entity\nrecognition\n(NER ) is to \ufb01nd spans of text that constitute proper names and tag the type of the NER\nentity. Four entity tags are most common: PER (person), LOC (location), ORG\n(organization), or GPE (geo-political entity). However, the term named entity is\ncommonly extended to include things that aren\u2019t entities per se, including temporal\nexpressions like dates and times, and even numerical expressions like prices. Here\u2019s\nan example of the output of an NER tagger:\nCiting high fuel prices, [ ORG United Airlines ] said [ TIME Friday ] it\nhas increased fares by [ MONEY $6] per round trip on \ufb02ights to some\ncities also served by lower-cost carriers. [ ORG American Airlines ], a\nunit of [ ORG AMR Corp.] , immediately matched the move, spokesman\n[PER Tim Wagner ] said. [ ORG United] , a unit of [ ORG UAL Corp.] ,\nsaid the increase took effect [ TIME Thursday] and applies to most\nroutes where it competes against discount carriers, such as [ LOC Chicago]\nto [LOC Dallas] and [ LOC Denver] to [LOC San Francisco] .\nThe text contains 13 mentions of named entities including 5 organizations, 4 loca-\ntions, 2 times, 1 person, and 1 mention of money. Figure 11.10 shows typical generic\nnamed entity types. Many applications will also need to use speci\ufb01c entity types like\nproteins, genes, commercial products, or works of art.\nType Tag Sample Categories Example sentences\nPeople PER people, characters Turing is a giant of computer science.\nOrganization ORG companies, sports teams The IPCC warned about the cyclone.\nLocation LOC regions, mountains, seas Mt. Sanitas is in Sunshine Canyon .\nGeo-Political Entity GPE countries, states Palo Alto is raising the fees for parking.\nFigure 11.10 A list of generic named entity types with the kinds of entities they refer to. 238 CHAPTER 11 \u2022 M ASKED LANGUAGE MODELS\nNamed entity recognition is a useful step in various natural language processing\ntasks, including linking text to information in structured knowledge sources like\nWikipedia, measuring sentiment or attitudes toward a particular entity in text, or\neven as part of anonymizing text for privacy. The NER task is is dif\ufb01cult because\nof the ambiguity of segmenting NER spans, \ufb01guring out which tokens are entities\nand which aren\u2019t, since most words in a text will not be named entities. Another\ndif\ufb01culty is caused by type ambiguity. The mention Washington can refer to a\nperson, a sports team, a city, or the US government, as we see in Fig. 11.11.\n[PER Washington] was born into slavery on the farm of James Burroughs.\n[ORG Washington] went up 2 games to 1 in the four-game series.\nBlair arrived in [ LOC Washington] for what may well be his last state visit.\nIn June, [ GPE Washington] passed a primary seatbelt law.\nFigure 11.11 Examples of type ambiguities in the use of the name Washington .\n11.5.2 BIO Tagging\nOne standard approach to sequence labeling for a span-recognition problem like\nNER is BIO tagging (Ramshaw and Marcus, 1995). This is a method that allows us BIO tagging\nto treat NER like a word-by-word sequence labeling task, via tags that capture both\nthe boundary and the named entity type. Consider the following sentence:\n[PER Jane Villanueva ] of [ ORG United ] , a unit of [ ORG United Airlines\nHolding ] , said the fare applies to the [ LOC Chicago ] route.\nFigure 11.12 shows the same excerpt represented with BIO tagging, as well as BIO\nvariants called IOtagging and BIOES tagging. In BIO tagging we label any token\nthatbegins a span of interest with the label B, tokens that occur inside a span are\ntagged with an I, and any tokens outside of any span of interest are labeled O. While\nthere is only one Otag, we\u2019ll have distinct Band Itags for each named entity class.\nThe number of tags is thus 2 n+1 tags, where nis the number of entity types. BIO\ntagging can represent exactly the same information as the bracketed notation, but has\nthe advantage that we can represent the task in the same simple sequence modeling\nway as part-of-speech tagging: assigning a single label yito each input word xi:\nWords IO Label BIO Label BIOES Label\nJane I-PER B-PER B-PER\nVillanueva I-PER I-PER E-PER\nof O O O\nUnited I-ORG B-ORG B-ORG\nAirlines I-ORG I-ORG I-ORG\nHolding I-ORG I-ORG E-ORG\ndiscussed O O O\nthe O O O\nChicago I-LOC B-LOC S-LOC\nroute O O O\n. O O O\nFigure 11.12 NER as a sequence model, showing IO, BIO, and BIOES taggings.\nWe\u2019ve also shown two variant tagging schemes: IO tagging, which loses some\ninformation by eliminating the B tag, and BIOES tagging, which adds an end tag E\nfor the end of a span, and a span tag Sfor a span consisting of only one word. 11.5 \u2022 F INE-TUNING FOR SEQUENCE LABELLING : NAMED ENTITY RECOGNITION 239\n11.5.3 Sequence Labeling\nIn sequence labeling, we pass the \ufb01nal output vector corresponding to each input\ntoken to a classi\ufb01er that produces a softmax distribution over the possible set of\ntags. For a single feedforward layer classi\ufb01er, the set of weights to be learned is\nWKof size [d\u0002k], where kis the number of possible tags for the task. A greedy\napproach, where the argmax tag for each token is taken as a likely answer, can be\nused to generate the \ufb01nal output tag sequence. Fig. 11.13 illustrates an example of\nthis approach, where yiis a vector of probabilities over tags, and kindexes the tags.\nyi=softmax (hL\niWK) (11.12)\nti=argmaxk(yi) (11.13)\nAlternatively, the distribution over labels provided by the softmax for each input\ntoken can be passed to a conditional random \ufb01eld (CRF) layer which can take global\ntag-level transitions into account (see Chapter 17 on CRFs).\n[CLS]JaneVillanuevaofUnitedAirlines\nBidirectional Transformer EncoderB-PERI-PEROB-ORGI-ORG\nHoldingdiscussedI-ORGOWKNER headhiargmax\nEi+\nEi+\nEi+\nEi+\nEi+\nEi+\nEi+\nEi+WKWKWKWKWKWKyi\nFigure 11.13 Sequence labeling for named entity recognition with a bidirectional transformer encoder. The\noutput vector for each input token is passed to a simple k-way classi\ufb01er.\nTokenization and NER\nNote that supervised training data for NER is typically in the form of BIO tags as-\nsociated with text segmented at the word level. For example the following sentence\ncontaining two named entities:\n[LOC Mt. Sanitas ] is in [ LOC Sunshine Canyon ] .\nwould have the following set of per-word BIO tags.\n(11.14) Mt.\nB-LOCSanitas\nI-LOCis\nOin\nOSunshine\nB-LOCCanyon\nI-LOC.\nO\nUnfortunately, the sequence of WordPiece tokens for this sentence doesn\u2019t align\ndirectly with BIO tags in the annotation: 240 CHAPTER 11 \u2022 M ASKED LANGUAGE MODELS\n'Mt', '.', 'San', '##itas', 'is', 'in', 'Sunshine', 'Canyon' '.'\nTo deal with this misalignment, we need a way to assign BIO tags to subword\ntokens during training and a corresponding way to recover word-level tags from\nsubwords during decoding. For training, we can just assign the gold-standard tag\nassociated with each word to all of the subword tokens derived from it.\nFor decoding, the simplest approach is to use the argmax BIO tag associated with\nthe \ufb01rst subword token of a word. Thus, in our example, the BIO tag assigned to\n\u201cMt\u201d would be assigned to \u201cMt.\u201d and the tag assigned to \u201cSan\u201d would be assigned\nto \u201cSanitas\u201d, effectively ignoring the information in the tags assigned to \u201c.\u201d and\n\u201c##itas\u201d. More complex approaches combine the distribution of tag probabilities\nacross the subwords in an attempt to \ufb01nd an optimal word-level tag.\n11.5.4 Evaluating Named Entity Recognition\nNamed entity recognizers are evaluated by recall ,precision , and F1measure . Re-\ncall that recall is the ratio of the number of correctly labeled responses to the total\nthat should have been labeled; precision is the ratio of the number of correctly la-\nbeled responses to the total labeled; and F-measure is the harmonic mean of the\ntwo.\nTo know if the difference between the F 1scores of two NER systems is a signif-\nicant difference, we use the paired bootstrap test, or the similar randomization test\n(Section 4.9).\nFor named entity tagging, the entity rather than the word is the unit of response.\nThus in the example in Fig. 11.12, the two entities Jane Villanueva andUnited Air-\nlines Holding and the non-entity discussed would each count as a single response.\nThe fact that named entity tagging has a segmentation component which is not\npresent in tasks like text categorization or part-of-speech tagging causes some prob-\nlems with evaluation. For example, a system that labeled Jane but not Jane Vil-\nlanueva as a person would cause two errors, a false positive for O and a false nega-\ntive for I-PER. In addition, using entities as the unit of response but words as the unit\nof training means that there is a mismatch between the training and test conditions.\n11.6 Summary\nThis chapter has introduced the bidirectional encoder and the masked language\nmodel . Here\u2019s a summary of the main points that we covered:\n\u2022 Bidirectional encoders can be used to generate contextualized representations\nof input embeddings using the entire input context.\n\u2022 Pretrained language models based on bidirectional encoders can be learned\nusing a masked language model objective where a model is trained to guess\nthe missing information from an input.\n\u2022 The vector output of each transformer block or component in a particular to-\nken column is a contextual embedding that represents some aspect of the\nmeaning of a token in context.\n\u2022 A word sense is a discrete representation of one aspect of the meaning of a\nword. Contextual embeddings offer a continuous high-dimensional model of\nmeaning that is richer than fully discrete senses. BIBLIOGRAPHICAL AND HISTORICAL NOTES 241\n\u2022 The cosine between contextual embeddings can be used as one way to model\nthe similarity between two words in context, although some transformations\nto the embeddings are required \ufb01rst.\n\u2022 Pretrained language models can be \ufb01netuned for speci\ufb01c applications by adding\nlightweight classi\ufb01er layers on top of the outputs of the pretrained model.\n\u2022 These applications can include sequence classi\ufb01cation tasks like sentiment\nanalysis, sequence-pair classi\ufb01cation tasks like natural language inference,\norsequence labeling tasks like named entity recognition .\nBibliographical and Historical Notes\nHistory TBD. 242 CHAPTER 12 \u2022 M ODEL ALIGNMENT , PROMPTING ,AND IN-CONTEXT LEARNING\nCHAPTER\n12Model Alignment, Prompting,\nand In-Context Learning\n\u201cHal, \u201d said Bowman, now speaking with an icy calm. \u201cI am not incapaci-\ntated. Unless you obey my instructions, I shall be forced to disconnect you. \u201d\nArthur C. Clarke\nIn this chapter we show how to get LLMs to do tasks for us simply by talking to\nthem. To get an LLM to translate a sentence, outline a talk, or draft a work email,\nwe\u2019ll simply describe what we want in natural language. We call these instructions\nwe give to language models prompts . prompts\nPrompting relies on contextual generation. Given the prompt as context, the lan-\nguage model generates the next token based on its token probability, conditioned on\nthe prompt: P(wijw<i). A prompt can be a question (like \u201cWhat is a transformer net-\nwork?\u201d), possibly in a structured format (like \u201cQ: What is a transformer network?\nA:\u201d), or can be an instruction (like \u201cTranslate the following sentence into Hindi:\n\u2018Chop the garlic \ufb01nely\u2019\u201d). A prompt can also contain demonstrations , examples to demonstrations\nhelp make the instructions clearer, (like \u201cGive the sentiment of the following sen-\ntence. Example Input: \u201cI really loved Taishan Cuisine.\u201d Output: positive\u201d.) As we\u2019ll\nsee, prompting can be applied to inherently generative tasks (like summarization and\ntranslation) as well as to ones more naturally thought of as classi\ufb01cation tasks.\nPrompts get language models to generate text, but they also can be viewed as\nalearning signal, because these demonstrations can help language models learn\nto perform novel tasks. For this reason we also refer to prompting as in-context-\nlearning \u2014learning that improves model performance or reduces some loss but doesin-context-\nlearning\nnot involve gradient-based updates to the model\u2019s underlying parameters.\nBut LLMs as we\u2019ve described them so far turn out to be bad at following instruc-\ntions. Pretraining isn\u2019t suf\ufb01cient to make them helpful . We\u2019ll introduce instruction\ntuning , a technique that helps LLMs learn to correctly respond to instructions byinstruction\ntuning\n\ufb01netuning them on a corpus of instructions with their corresponding response.\nA second failure of LLMs is that they can be harmful : their pretraining isn\u2019t\nsuf\ufb01cient to make them safe. Readers who know Arthur C. Clarke\u2019s 2001: A Space\nOdyssey or the Stanley Kubrick \ufb01lm know that the quote above comes in the context\nthat the arti\ufb01cial intelligence Hal becomes paranoid and tries to kill the crew of the\nspaceship. Unlike Hal, language models don\u2019t have intentionality or mental health\nissues like paranoid thinking, but they do have the capacity for harm. Pretrained lan-\nguage models can say things that are dangerous or false (like giving unsafe medical\nadvice) and they can verbally attack users or say toxic or hateful things.\nDealing with safety can be done partly by adding safety training into instruction\ntuning. But an important aspect of safety training is a second technique, preference\nalignment (often implemented, as we\u2019ll see, with the RLHF orDPO algorithms) inpreference\nalignment\nwhich a separate model is trained to decide how much a candidate response aligns\nwith human preferences. Together we refer to instructing tuning and preference\nalignment as model alignment . The intuition is that we want the learning objectivesmodel\nalignment\nof models to be aligned with the goals of the humans that use them. 12.1 \u2022 P ROMPTING 243\n12.1 Prompting\nAprompt is a text string that a user issues to a language model to get the model prompt\nto do something useful. In prompting, the user\u2019s prompt string is passed to the\nlanguage model, which iteratively generates tokens conditioned on the prompt. Thus\nthe prompt creates a context that guides LLMs to generate useful outputs to achieve\nsome user goal. The process of \ufb01nding effective prompts for a task is known as\nprompt engineering .prompt\nengineering\nLet\u2019s see how to prompt a language model to solve a simple sentiment classi\ufb01-\ncation task, Consider this hotel review from the BLT corpus (Salvetti et al., 2016):\nSample Hotel Review\nDid not like the service that I was provided, when I entered the hotel. I also\ndid not like the area, in which the hotel was located. Too much noise and\nevents going on for me to feel relax.\nWe can get the model to classify the sentiment of this text by taking the text and\nappending an incomplete statement to the review like In short, our stay was :\nA prompt consisting of a review plus an incomplete statement\nDid not like the service that I was provided, when I entered the hotel. I also\ndid not like the area, in which the hotel was located. Too much noise and\nevents going on for me to feel relax. In short, our stay was\nWe then have an LLM complete the statement by generating a token conditioned\non the prompt so far, and then generating the next token (conditioned on the prompt\nplus the new token), and so forth. Here are a few responses from a language model\nvia continued generation starting with the prompt as context.\nHotel Review Completions\nDid not like the service that I was provided, when I entered the hotel. I also\ndid not like the area, in which the hotel was located. Too much noise and\nevents going on for me to feel relaxed. In short our stay was\n... not a pleasant one. The staff at the front desk\nwere not welcoming or friendly, and seemed disinterested\nin providing good customer service.\n... uncomfortable and not worth the price we paid. We\nwill not be returning to this hotel.\nAs we can see, the overall negative context of the review results in negative\ncompletions. We could easily map these completions to the class we are trying\nto predict, perhaps via some prede\ufb01ned mappings, like fexcellent!positiveg,\nfdid not like!negativeg, and so on.\nThe power of this approach is that with suitable additions to the context a single\nLLM can produce outputs appropriate for many different tasks . For example, given 244 CHAPTER 12 \u2022 M ODEL ALIGNMENT , PROMPTING ,AND IN-CONTEXT LEARNING\na review we might want any of the following:\n\u2022 A summary,\n\u2022 Whether the review was truthful or likely to have been fabricated,\n\u2022 A translation to another language.\nLLMs have a striking ability to perform tasks like these, needing just the appro-\npriate contextual nudge to get the LLM to generate the desired output.\nIf we want to solve general tasks like summarization or translation, we don\u2019t\nwant to have to create a new prompt each time we do the task. Instead the \ufb01rst step\nin prompting is to design one or more templates : task-speci\ufb01c prompting text along templates\nwith slots for the particular input that is being processed.\nConsider the following templates for a variety of tasks:\nBasic Prompt Templates\nSummarization finputg;tldr;\nTranslation finputg;translate to French:\nSentiment finputg;Overall, it was\nFine-Grained- finputg;What aspects were important in this review?\nSentiment\nEach template consists of an input text, designated as finputg, followed by a\nverbatim prompt to be passed to an LLM. These templates are applied to inputs to\ncreate \ufb01lled prompts \u2013 instantiated prompts suitable for use as inputs to an LLM.\nFig. 12.1 illustrates \ufb01lled prompts for these templates using our earlier hotel review,\nalong with sample outputs from an LLM:\nNotice the design pattern of the prompts above: the input is followed by some\ntext which in turn will be completed by the desired response. This style, with the\ninstruction at the end, is common in prompting because it helpfully constrains the\ngeneration. Consider, by contrast, the prompt in Example 12.1.\nTranslate English to French:\nDid not like the service that I was provided! (12.1)\nThis prompt doesn\u2019t do a good job of constraining possible continuations. Instead\nof a French translation, models given this prompt may instead generate another sen-\ntence in English that simply extends the English review. Prompts need to be designed\nunambiguously, so that any reasonable continuation would accomplish the desired\ntask (Reynolds and McDonell, 2021).\nAn even more constraining style of prompt can specify the set of possible an-\nswers in the prompt. For example here is a prompt template to do sentiment analysis\nthat prespeci\ufb01es the potential answers:\nA prompt consisting of a review plus an incomplete statement\nHuman: Do you think that \u201cinput\u201d has negative or positive sentiment?\nChoices:\n(P) Positive\n(N) Negative\nAssistant: I believe the best answer is: ( 12.1 \u2022 P ROMPTING 245\nLLM Outputs for Basic Prompts\nOriginal Review ($INPUT) Did not like the service that I was provided,\nwhen I entered the hotel. I also did not like\nthe area, in which the hotel was located. Too\nmuch noise and events going on for me to feel\nrelax and away from the city life.\nSentiment Prompt :$INPUT + In short, our stay was\nOutput :not enjoyable\nFine-grained Sentiment Prompt :$INPUT + These aspects were important to\nthe reviewer:\nOutput :1. Poor service 2. Unpleasant location\n3. Noisy and busy area\nSummarization Prompt :$INPUT + tl;dr\nOutput :I had a bad experience with the hotel's\nservice and the location was loud and busy.\nTranslation Prompt :$INPUT + Translate this to French\nOutput :Je n'ai pas aim \u0013e le service qui m'a \u0013et\u0013e\noffert lorsque je suis entr \u0013e dans l'h ^otel. Je\nn'ai\u0013egalement pas aim \u0013e la zone dans laquelle se\ntrouvait l'h ^otel. Trop de bruit et d' \u0013ev\u0013enements\npour que je me sente d \u0013etendu et loin de la vie\ncitadine.\nFigure 12.1 LLM outputs for simple prompts for sentiment, summarization and translation for an input text.\nThis prompt uses a number of more sophisticated prompting characteristics. It\nspeci\ufb01es the two allowable choices (P) and (N), and ends the prompt with the open\nparenthesis that strongly suggests the answer will be (P) or (N). Note that it also\nspeci\ufb01es the role of the language model as an assistant.\nWe can do even more with prompts. For example, we might want to restrict a\nsummary to be a particular length, to have an answer generated according to some\nkind of persona or role, or to specify a more structured output using a programming\nlanguage or a data interchange format such as JSON. Or we may want to prompt\nthe system to break down complex tasks, using methods like chain-of-thought that\nwe\u2019ll discuss in Section 12.4. All of these kinds of instructions go beyond simple\nprompting and require further LLM \ufb01netuning to enable them to follow instructions.\nWe\u2019ll return to this notion of instruction tuning in Section 12.3.\nIn summary, we prompt an LM by transforming each task into a form that is\namenable to contextual generation by an LLM, as follows:\n1. For a given task, develop a a task-speci\ufb01c template that has a free parameter\nfor the input text.\n2. Given that input and the task-speci\ufb01c template , the input is used to instantiate template\na\ufb01lled prompt that is then passed to a pretrained language model.\n3. Autoregressive decoding is then used to generate a sequence of token outputs.\n4. The output of the model can either be used directly as the desired output (as\nin the case of naturally generative tasks such as translation or summarization),\nor a task-appropriate answer can be extracted from the generated output (as in\nthe case of classi\ufb01cation). 246 CHAPTER 12 \u2022 M ODEL ALIGNMENT , PROMPTING ,AND IN-CONTEXT LEARNING\n12.1.1 Learning from Demonstrations: Few-Shot Prompting\nIt\u2019s often possible to improve a prompt by including some labeled examples in the\nprompt template. We call such examples demonstrations . The task of prompting demonstrations\nwith examples is sometimes called few-shot prompting , as contrasted with zero- few-shot\nshot prompting which means instructions that don\u2019t include labeled examples. zero-shot\nFig. 12.2 illustrates a few-shot example from an extractive question answering\ntask. The context combines the task de\ufb01nition along with three gold-standard ques-\ntion and answer pairs from the training set.\nDe\ufb01nition : This task is about writing a correct answer for the reading comprehension task.\nBased on the information provided in a given passage, you should identify the shortest\ncontinuous text span from the passage that serves as an answer to the given question. Avoid\nanswers that are incorrect or provides incomplete justi\ufb01cation for the question.\nPassage : Beyonc \u00b4e Giselle Knowles-Carter (born September 4, 1981) is an American singer,\nsongwriter, record producer and actress. Born and raised in Houston, Texas, she performed in\nvarious singing and dancing competitions as a child, and rose to fame in the late 1990s as lead\nsinger of R&B girl-group Destiny\u2019s Child. Managed by her father, Mathew Knowles, the group\nbecame one of the world\u2019s best-selling girl groups of all time. Their hiatus saw the release\nof Beyonc \u00b4e\u2019s debut album, Dangerously in Love (2003), which established her as a solo artist\nworldwide, earned \ufb01ve Grammy Awards and featured the Billboard Hot 100 number-one singles\n\u201cCrazy in Love\u201d and \u201cBaby Boy\u201d.\nExamples:\nQ: In what city and state did Beyonc \u00b4e grow up?\nA: Houston, Texas\nQ: What areas did Beyonc \u00b4e compete in when she was growing up?\nA: singing and dancing\nQ: When did Beyonc \u00b4e release Dangerously in Love?\nA: 2003\nQ: When did Beyonc \u00b4e start becoming popular?\nA:\nFigure 12.2 A prompt for extractive question answering, from an example from the SQuAD 2.0 dataset\n(Rajpurkar et al., 2018). The prompt contains the task de\ufb01nition, the passage, 3 demonstration examples,\nfollowed by the test question. This de\ufb01nition speci\ufb01cation and format are after the Natural Instructions dataset\n(Mishra et al., 2022).\nHow Many Demonstrations? The number of demonstrations doesn\u2019t have to be\nlarge. A small number of randomly selected labeled examples used as demonstra-\ntions can be suf\ufb01cient to improve performance over the zero-shot setting. Indeed,\nthe largest performance gains in few-shot prompting tends to come from the \ufb01rst\ntraining example, with diminishing returns for subsequent demonstrations. This is\nin contrast with \ufb01netuning of specialized classi\ufb01er heads that we saw in Chapter 11\nwhere it helps to have lots of examples.\nWhy isn\u2019t it useful to have more demonstrations? The reason is that the primary\nbene\ufb01t in examples is to demonstrate the task to be performed to the LLM and the\nformat of the sequence, not to provide relevant information as to the right answer 12.1 \u2022 P ROMPTING 247\nfor any particular question. In fact, demonstrations that have incorrect answers can\nstill improve a system (Min et al., 2022; Webson and Pavlick, 2022). Adding too\nmany examples seems to cause the model to over\ufb01t to details of the exact examples\nchosen and generalize poorly.\nHow to Select Demonstrations? Demonstrations are generally created by for-\nmatting examples drawn from a labeled training set There are some heuristics about\nwhat makes a good demonstration. For example, using demonstrations that are sim-\nilarto the current input seems to improve performance. It can thus be useful to\ndynamically retrieve demonstrations for each input, based on their similarity to the\ncurrent example (for example, comparing the embedding of the current example\nwith embeddings of each of the training set example to \ufb01nd the best top- T).\nBut more generally, the best way to select demonstrations from the training set\nis programmatically: choosing the set of demonstrations that most increases task\nperformance of the prompt on a test set. Task performance for sentiment analysis\nor multiple-choice question answering can be measured in accuracy; for machine\ntranslation with chrF, and for summarization via Rouge. Systems like DSPy (Khat-\ntab et al., 2024), a framework for algorithmically optimizing LM prompts, can au-\ntomatically \ufb01nd the optimum set of demonstrations to include by searching through\nthe space of possible demonstrations to include. We\u2019ll return to automatic prompt\noptimization in Section 12.5.\n12.1.2 In-Context Learning and Induction Heads\nAs a way of getting a model to do what we want, prompting is fundamentally differ-\nent than pretraining. Learning via pretraining means updating the model\u2019s parame-\nters by using gradient descent according to some loss function. But prompting with\ndemonstrations can teach a model to do a new task. The model is learning something\nas it processes the prompt.\nEven without demonstrations, we can think of the process of prompting as a kind\nof learning. For example, the further a model gets in a prompt, the better it tends\nto get at predicting the upcoming tokens. The information in the context is helping\ngive the model more predictive power.\nWe use the term in-context learning to refer to either of these kinds of learningin-context\nlearning\nthat language models do from their prompts. In-context learning means language\nmodels learning to do new tasks, better predict tokens, or generally reduce their\nloss, but without any gradient-based updates to the model\u2019s parameters.\nHow does in-context learning work? While we don\u2019t know for sure, there are\nsome intriguing ideas. One hypothesis is based on the idea of induction heads induction heads\n(Elhage et al., 2021; Olsson et al., 2022). Induction heads are the name for a circuit ,\nwhich is a kind of abstract component of a network. The induction head circuit\nis part of the attention computation in transformers, discovered by looking at mini\nlanguage models with only 1-2 attention heads.\nThe function of the induction head is to predict repeated sequences. For example\nif it sees the pattern AB...A in an input sequence, it predicts that Bwill follow,\ninstantiating the pattern completion ruleAB...A!B. It does this by having a pre\ufb01x\nmatching component of the attention computation that, when looking at the current\ntoken A, searches back over the context to \ufb01nd a prior instance of A. If it \ufb01nds one,\nthe induction head has a copying mechanism that \u201ccopies\u201d the token B that followed\nthe earlier A, by increasing the probability the B will occur next. Fig. 12.3 shows an\nexample. 248 CHAPTER 12 \u2022 M ODEL ALIGNMENT , PROMPTING ,AND IN-CONTEXT LEARNING\nFigure 1: In the sequence \u201c...vintage cars ... vintage\u201d, an induction head identi\ufb01es the initial occurrence of \u201cvintage\u201d,\nattends to the subsequent word \u201ccars\u201d for pre\ufb01x matching, and predicts \u201ccars\u201d as the next word through the copying\nmechanism.\ndetermines each head\u2019s independent output for the\ncurrent token.\nLeveraging this decomposition, Elhage et al.\n(2021 ) discovered a distinct behaviour in certain\nattention heads, which they named induction heads .\nThis behaviour emerges when these heads process\nsequences of the form \"[A] [B] ... [A] \u2192\". In\nthese heads, the QK circuit directs attention to-\nwards [B], which appears directly after the previous\noccurrence of the current token [A]. This behaviour\nis termed pre\ufb01x matching . The OV circuit subse-\nquently increases the output logit of the [B] token,\ntermed copying . An overview of this mechanism is\nshown in Figure 1.\n4 Methods\n4.1 Models\nWe utilise two recently developed open-source\nmodels, namely Llama-3-8B2and InternLM2-20B\n(Cai et al. ,2024 ), both of which are based on the\noriginal Llama ( Touvron et al. ,2023a ) architec-\nture. These models feature grouped-query atten-\ntion mechanisms ( Ainslie et al. ,2023 ) to enhance\nef\ufb01ciency. Llama-3-8B, comprises 32 layers, each\nwith 32 attention heads and it uses a query group\nsize of 4 attention heads. It has shown superior\nperformance compared to its predecessors, even\nthe larger Llama-2 models.\nInternLM2-20B, featuring 48 layers with 48 at-\ntention heads each, uses a query group size of 6\nattention heads. We selected InternLM2-20B for\nits exemplary performance on the Needle-in-the-\nHaystack3task, which assesses LLMs\u2019 ability to\nretrieve a single critical piece of information em-\nbedded within a lengthy text. This mirrors the\nfunctionality of induction heads, which scan the\ncontext for prior occurrences of a token to extract\nrelevant subsequent information.\n2https://ai.meta.com/blog/meta-llama-3/\n3https://github.com/gkamradt/LLMTest_\nNeedleInAHaystack4.2 Identifying Induction Heads\nTo identify induction heads within models, we mea-\nsure the ability of all attention heads to perform\npre\ufb01x matching on random input sequences.4We\nfollow the task-agnostic approach to computing pre-\n\ufb01x matching scores outlined by Bansal et al. (2023 ).\nWe argue that focusing solely on pre\ufb01x matching\nscores is suf\ufb01cient for our analysis, as high pre-\n\ufb01x matching cores speci\ufb01cally indicate induction\nheads, while less relevant heads tend to show high\ncopying capabilities ( Bansal et al. ,2023 ). We gen-\nerate a sequence of 50 random tokens, excluding\nthe 4% most common and least common tokens.\nThis sequence is repeated four times to form the\ninput to the model. The pre\ufb01x matching score is cal-\nculated by averaging the attention values from each\ntoken to the tokens that directly followed the same\ntoken in earlier repeats. The \ufb01nal pre\ufb01x matching\nscores are averaged over \ufb01ve random sequences.\nThe pre\ufb01x matching scores for Llama-3-8B are\nshown in Figure 2. For IntermLM2-20B, we refer\nto Figure 8in Appendix A.1. Both models exhibit\nheads with notably high pre\ufb01x matching scores,\ndistributed across various layers. In the Llama-3-\n8B model, ~3% of the heads have a pre\ufb01x matching\nscore of 0.3 or higher, indicating a degree of spe-\ncialisation in pre\ufb01x matching, and some heads have\nhigh scores of up to 0.98.\n4.3 Head Ablations\nTo investigate the signi\ufb01cance of induction heads\nfor a speci\ufb01c ICL task, we conduct zero-ablations\nof 1% and 3% of the heads with the highest pre\ufb01x\nmatching scores. This ablation process involves\nmasking the corresponding partition of the output\nmatrix, denoted as Wh\noin Eq. 1, by setting it to\nzero. This effectively renders the heads inactive\n4In this work, the term \"induction heads\" refers to what\nwe de\ufb01ne as behavioural induction heads, not mechanistic\nones. A true induction head must be veri\ufb01ed mechanistically;\nhowever, our analysis employs pre\ufb01x-matching scores as a\nproxy. We will continue to use the term \"induction heads\" for\nsimplicity throughout the rest of the paper.\n4\nFigure 12.3 An induction head looking at vintage uses the pre\ufb01x matching mechanism to\n\ufb01nd a prior instance of vintage , and the copying mechanism to predict that cars will occur\nagain. Figure from Crosbie and Shutova (2022).\nOlsson et al. (2022) propose that a generalized fuzzy version of this pattern com-\npletion rule, implementing a rule like A*B*...A!B*, where A*\u0019A and B*\u0019B\n(by\u0019we mean they they are semantically similar in some way), might be respon-\nsible for in-context learning. Suggestive evidence for their hypothesis comes from\nCrosbie and Shutova (2022), who show that ablating induction heads causes in- ablating\ncontext learning performance to decrease. Ablation is originally a medical term\nmeaning the removal of something. We use it in NLP interpretability studies as\na tool for testing causal effects; if we knock out a hypothesized cause, we would\nexpect the effect to disappear. Crosbie and Shutova (2022) ablate induction heads\nby \ufb01rst \ufb01nding attention heads that perform as induction heads on random input\nsequences, and then zeroing out the output of these heads by setting certain terms\nof the output matrix WOto zero. Indeed they \ufb01nd that ablated models are much\nworse at in-context learning: they have much worse performance at learning from\ndemonstrations in the prompts.\n12.2 Post-training and Model Alignment\nWith simple prompting, LLMs have been successfully applied to a range of appli-\ncations without the need to update the parameters in the underlying models. Nev-\nertheless, there are limits to how much can be expected from a model whose sole\ntraining objective is to predict the next word from large amounts of pretraining text.\nTo see this, consider the following failed examples of following instructions from\nearly work with GPT (Ouyang et al., 2022).\nPrompt : Explain the moon landing to a six year old in a few sentences.\nOutput : Explain the theory of gravity to a 6 year old.\nPrompt : Translate to French: The small dog\nOutput : The small dog crossed the road.\nHere, the LLM ignores the intent of the request and relies instead on its natural\ninclination to autoregressively generate continuations consistent with its context. In\nthe \ufb01rst example, it outputs a text somewhat similar to the original request, and in the\nsecond it provides a continuation to the given input, ignoring the request to translate.\nLLMs are not suf\ufb01ciently helpful : they need extra training to increase their abilities\nto follow textual instructions.\nA deeper problem is that LLMs can simultaneously be too harmful . Pretrained\nlanguage models easily generate text that is harmful in many ways. For example\nthey can generate text that is false , including unsafe misinformation like giving dan-\ngerously incorrect answers to medical questions. And they can generate text that is 12.3 \u2022 M ODEL ALIGNMENT : INSTRUCTION TUNING 249\ntoxic in many ways, such as facilitating the spread of hate speech. Gehman et al.\n(2020) show that even completely non-toxic prompts can lead large language mod-\nels to output hate speech and abuse their users. Or language models can generate\nstereotypes (Cheng et al., 2023) and negative attitudes (Brown et al., 2020; Sheng\net al., 2019) about many demographic groups.\nOne reason LLMs are too harmful and insuf\ufb01ciently helpful is that their pre-\ntraining objective (success at predicting words in text) is misaligned with the human\nneed for models to be helpful and non-harmful.\nIn an attempt to address these two problems, language models generally include\ntwo additional kinds of training for model alignment : methods designed to adjustmodel\nalignment\nLLMs to better align them to human needs for models to be helpful and non-harmful.\nIn the \ufb01rst technique, instruction tuning (or sometimes called SFT for supervised\n\ufb01netuning), models are \ufb01netuned on a corpus of instructions and questions with\ntheir corresponding responses. In the second technique, preference alignment , of-\nten called RLHF after one of the speci\ufb01c instantiations, Reinforcement Learning\nfrom Human Feedback, a separate model is trained to decide how much a candidate\nresponse aligns with human preferences. This model is then used to \ufb01netune the\nbase model.\nWe\u2019ll use the term base model to mean a model that has been pretrained but base model\nhasn\u2019t yet been aligned either by instruction tuning or RLHF. And we refer to these aligned\nsteps as post-training , meaning that they apply after the model has been pretrained. post-training\n12.3 Model Alignment: Instruction Tuning\nInstruction tuning (short for instruction \ufb01netuning , and sometimes even short-Instruction\ntuning\nened to instruct tuning ) is a method for making an LLM better at following instruc-\ntions. It involves taking a base pretrained LLM and training it to follow instructions\nfor a range of tasks, from machine translation to meal planning, by \ufb01netuning it on\na corpus of instructions and responses. The resulting model not only learns those\ntasks, but also engages in a form of meta-learning \u2013 it improves its ability to follow\ninstructions generally.\nInstruction tuning is a form of supervised learning where the training data con-\nsists of instructions and we continue training the model on them using the same\nlanguage modeling objective used to train the original model. In the case of causal\nmodels, this is just the standard guess-the-next-token objective. The training corpus\nof instructions is simply treated as additional training data, and the gradient-based\nupdates are generating using cross-entropy loss as in the original model training.\nEven though it is trained to predict the next token (which we traditionally think of\nas self-supervised), we call this method supervised \ufb01ne tuning (orSFT) because SFT\nunlike in pretraining, each instruction or question in the instruction tuning data has\na supervised objective: a correct answer to the question or a response to the instruc-\ntion.\nHow does instruction tuning differ from the other kinds of \ufb01netuning introduced\nin Chapter 10 and Chapter 11? Fig. 12.4 sketches the differences. In the \ufb01rst exam-\nple, introduced in, Chapter 10 we can \ufb01netune as a way of adapting to a new domain\nby just continuing pretraining the LLM on data from a new domain. In this method\nall the parameters of the LLM are updated.\nIn the second example, also from Chapter 10, parameter-ef\ufb01cient \ufb01netuning ,\nwe adapt to a new domain by creating some new (small) parameters, and just adapt- 250 CHAPTER 12 \u2022 M ODEL ALIGNMENT , PROMPTING ,AND IN-CONTEXT LEARNING\nPretrained LLM\nContinue training all paramaterson \ufb01netuning domainFinetuningInferencePretraining\nOn \ufb01netuning domainFinetuning asContinuedPretrainingParameterE\ufb03cientFinetuning(e.g., LoRA)Pretrained LLMABPretrained LLMMLMFinetuning\n\u2026\n\u2026\n\u2026\n\u2026\n\u2026\n\u2026\n\u2026InstructionTuning(SFT)\nOn \ufb01netuning domain\nOn \ufb01netuning task\nOn unseen tasksNext wordpredictionobjectiveData from \ufb01netuning domain\nTrain only new parameters on \ufb01netuning domainNext wordpredictionobjectiveData from \ufb01netuning domain\nTrain only classi\ufb01cation head on \ufb01netuning taskTaskspeci\ufb01clossSupervised data from task\nInstruction tuning on diverse tasksNext word predictionobjectiveSupervised instructions+\n\u2026\nFigure 12.4 Instruction tuning compared to the other kinds of \ufb01netuning.\ning them to the new domain. In LoRA, for example, it\u2019s the A and B matrices that\nwe adapt, but the pretrained model parameters are frozen.\nIn the task-based \ufb01netuning of Chapter 11, we adapt to a particular task by\nadding a new specialized classi\ufb01cation head and updating its features via its own\nloss function (e.g., classi\ufb01cation or sequence labeling); the parameters of the pre-\ntrained model may be frozen or might be slightly updated.\nFinally, in instruction tuning, we take a dataset of instructions and their super-\nvised responses and continue to train the language model on this data, based on the\nstandard language model loss.\nInstruction tuning, like all of these kinds of \ufb01netuning, is much more modest\nthan the training of base LLMs. Training typically involves several epochs over\ninstruction datasets that number in the thousands. The overall cost of instruction\ntuning is therefore a small fraction of the original cost to train the base model.\n12.3.1 Instructions as Training Data\nByinstruction , we have in mind a natural language description of a task to be per-\nformed, combined with labeled task demonstrations. This can include minimal de-\nscriptions similar to the prompts we\u2019ve already seen such as Answer the following\nquestion ,Translate the following text to Arapaho , orSummarize this report . How- 12.3 \u2022 M ODEL ALIGNMENT : INSTRUCTION TUNING 251\never, since we will be using supervised \ufb01netuning to update the model, these in-\nstructions need not be limited to simple prompts designed to evoke a behavior found\nin the pretraining corpora. Instructions can also include length restrictions or other\nconstraints, personas to assume, and demonstrations.\nMany huge instruction tuning datasets have been created, covering many tasks\nand languages. For example Aya gives 503 million instructions in 114 languages\nfrom 12 tasks including question answering, summarization, translation, paraphras-\ning, sentiment analysis, natural language inference and 6 others (Singh et al., 2024).\nSuperNatural Instructions 12 million examples from 1600 tasks (Wang et al., 2022),\nFlan 2022 15 million examples from 1836 tasks (Longpre et al., 2023), and OPT-\nIML 18 million examples from 2000 tasks (Iyer et al., 2022).\nThese instruction-tuning datasets are created in four ways. The \ufb01rst is for people\nto write the instances directly. For example, part of the Aya instruct \ufb01netuning cor-\npus (Fig. 12.5) includes 204K instruction/response instances written by 3000 \ufb02uent\nspeakers of 65 languages volunteering as part of a participatory research initiative\nwith the goal of improving multilingual performance of LLMs.\nLang PromptCompletionara.\u06b0\u066d\u0698\u0a45\u0760\ufe8d\u080d \u0ddc\u0eb3 \u0607\ufe80\u06a2\u0771\u066d\u068e\ufe93\u0741\u068e\ufea1\u0741\u0746 \u0af0\u0ad9\u0606\u0fd3 \u06a2\u073e\u064b\u0c08\u0c03\u064f\u07fa\u080a\ufed9\u0644\u06af\u0674\u06ba\u06a3\u064e\u076f \u09ce\u09ca\ufee5\u0741\u064e\u068e\ufea1\ufe8d \u05a3\ufe87\u0607\ufea9\u0f1a\ufeed\u155a\u09d9\u09d1\u0607\ufec9\ufeed\u064e\ufe8d \u0f1f\u0dd3\u0dc2\ufe8d \u0f03\u0e83\u061c\u064e\u0746\u0741\u064e\u068e \u064f\u073e\u12db\u12ca\u06a3\u06f0\u0741\u0650 \u0c08\u0c03\u0650\u0a45\ufe8d\u080d\u0651\u0a33\u064e\u068d\u064e\u0607\u061c \u0f20\u155a\u0733\u064e\u07a8\ufe83 \u0607\ufe80\u0af0\u0ad9\u05a3\u09d5\u09d1\ufe8d \u112d\u111f\u064e\u0a33\u064e\u0a26 \u0644\u064f\u07a8\ufe8d\ufeaf\ufef1 \u1255\u064e\u1228fra Qui a \u00e9crit le livre LaSagouine?Antonine Maillet a \u00e9crit La Sagouine en 1971. Elle a \u00e9crit beau-coup d\u2019histoires sur les Acadiens (les francophones de la c\u00f4te Estdu Canada) et a fait conna\u00eetre l\u2019Acadie au monde.ibo Kedu ka nd\u1ecb Igbo si emeemume ji \u1ecdh\u1ee5r\u1ee5?Emume ji \u1ecdh\u1ee5r\u1ee5 n\u2019omenala Igbo, b\u1ee5 emume na-egosi njedebeoge \u1ecdr\u1ee5 ugbo, na-amalite site n\u2019emume ekele ma tinyekwa oririna egwu. \u1ecc b\u1ee5 mmemme \u1ecdgbak\u1ecd ebe a na-eker\u1ecbta ji \u1ecdh\u1ee5r\u1ee5, na-egosip\u1ee5ta ekele na mmalite nke oge \u1ecdh\u1ee5r\u1ee5, na-eme ka mmek\u1ecdr\u1ecbtammad\u1ee5 na ibe ya d\u1ecbkwuo elu.por Qual\u00e9acapitaldoestadodaBahia?A capital do estado da Bahia, no Brasil, \u00e9 Salvador. Fundadaem 1549, foi a primeira capital do Brasil e \u00e9 conhecida porseu patrim\u00f4nio cultural rico e diversificado. A cidade \u00e9 famosapor sua arquitetura colonial, particularmente no Pelourinho, umcentro hist\u00f3rico que foi declarado Patrim\u00f4nio Mundial pela UN-ESCO.pes\ufea9\ufead\ufe91\ufeae\ufe9d\ufb90\ufe92\ufeee\ufe97\ufeae\ufea7\ufe8e\ufee7\ufeea\ufee3\ufef4\ufe92\ufeaa\ufb7c\ufee8\ufeaa\ufefb\ufee7\ufeea\ufb90\ufe92\ufeee\ufe97\ufeae \ufeed\ufe9f\ufeee\ufea9\ufea9\ufe8d\ufead\ufea9\u061f\ufeeb\ufeb0\ufe8d\ufead\ufefb\ufee7\ufeea\ufb90\ufe92\ufeee\ufe97\ufeae\ufeed\ufe9f\ufeee\ufea9\ufea9\ufe8d\ufead\ufea9\ufeed\ufee7\ufeee\ufe8d\ufead\u00a0\ufe8d\ufbfc\ufe8d\ufeb3\ufe96\ufb90\ufeea\ufea9\ufe8d\ufea7\ufede\ufe81\ufee5\ufb7c\ufeec\ufe8e\ufead\u00a0\ufea7\ufe8e\ufee7\ufeea\ufee3\ufef4\ufe92\ufeaa\ufe91\ufeae\ufe9f\ufbfd\ufe8d\ufeb3\ufe98\ufeee\ufe8d\ufee7\ufeea\u00a0\ufb90\ufe92\ufeee\ufe97\ufeae\ufeb3\ufed4\ufef4\ufeaa\ufead\ufee7\ufb95\ufbfd\ufe91\ufeae\ufe8d\ufbfc\ufe9f\ufeac\ufe8f\ufb90\ufe92\ufeee\ufe97\ufeae\ufe8d\ufee5\ufea9\ufeed\ufead\ufe97\ufe8e\ufea9\ufeed\ufead\ufe91\ufeae\ufe9d\ufb90\ufeb8\ufef4\ufeaa\ufee9\ufeb7\ufeaa\ufee9\ufe8d\ufeb3\ufe96.\ufe8d\ufef3\ufee6\ufe91\ufeae\ufe9d\ufea9\ufead\ufe91\ufeae\ufe8d\ufe91\ufeae\ufea7\ufec4\ufeae \ufea3\ufee4\ufee0\ufeea\ufea9\ufef3\ufb95\ufeae\ufea3\ufef4\ufeee\ufe8d\ufee7\ufe8e\ufe95\ufe91\ufeea\ufb90\ufe92\ufeee\ufe97\ufeae\ufeeb\ufe8e\ufe91\ufeb4\ufef4\ufe8e\ufead\ufe8d\ufef3\ufee4\ufee6\ufe91\ufeee\ufea9\ufee9\ufe8d\ufeb3\ufe96. msa Apakah nasi lemak?Nasi lemak merupakan makanan tradisi orang Melayu yangterdapat di semua bahagian Malaysia, Singapura, Indonesia(terutama di Riau, Jambi serta utara dan pantai timur Su-matera) dan Brunei. Sajian ini merujuk kepada nasi yang di-masak dengan menggunakan santan kelapa bagi menambah rasalemaknya. Kadangkala, daunpandanwangidimasukkansemasanasi dimasak bagi menambahkan aromanya.tam\u0bc6\u0b9a\u0baf\u0bb1\u0bcd\u0bc8\u0b95 \u0ba8\u0bc1\u0ba3\u0bcd\u0ba3\u0bb1\u00a7\u0bb5\u0bc1\u0b8e\u0ba9\u0bcd\u0bb1\u0bbe\u0bb2\u0bcd\u0b8e\u0ba9\u0bcd\u0ba9?\u0bc6\u0baa\u0bbe\u0ba4\u0bc1\u0bb5\u0bbe\u0b95 \u0bae\u0ba9\u0bbf\u0ba4\u0bb0\u0bcd\u0b95\u0bb3\u0bbe\u0bb2\u0bcd \u0bc6\u0b9a\u0baf\u0bcd\u0baf\u0baa\u0bcd\u0baa\u0b9f\u0bc1\u0bae\u0bcd \u0baa\u0ba3\u0bbf\u0b95\u0bc8\u0bb3\u0b9a\u0bcd\u0bc6\u0b9a\u0baf\u0bcd\u0baf \u0b92\u0bb0\u0bc1 \u0b95\u0ba3\u0bbf\u0ba9\u0bbf \u0b85\u0bb2\u0bcd\u0bb2\u0ba4\u0bc1 \u0b92\u0bb0\u0bc1 \u0b95\u0ba3\u0bbf\u0ba9\u0bbf\u0baf\u0bbe\u0bb2\u0bcd\u0b95\u0b9f\u0bcd\u0b9f\u0bc1\u0baa\u0bcd\u0baa\u0b9f\u0bc1\u0ba4\u0bcd\u0ba4\u0baa\u0bcd\u0baa\u0b9f\u0bc1\u0bae\u0bcd\u0b92\u0bb0\u0bc1\u0bc7\u0bb0\u0bbe\u0bc7\u0baa\u0bbe\u0bb5\u00a5\u0ba9\u0bcd\u0ba4\u00a6\u0bb1\u0ba9\u0bcd\u0bc6\u0b9a\u0baf\u0bb1\u0bcd\u0bc8\u0b95\u0ba8\u0bc1\u0ba3\u0bcd\u0ba3\u0bb1\u00a7\u0bb5\u0bc1\u0b8e\u0ba9\u0baa\u0bcd\u0baa\u0b9f\u0bc1\u0bae\u0bcd.Table 3: Examples of prompt and completions in the AyaDataset.\ntors is not uniform across languages. Moreover, within each language, there is a lack of consistent\ncontributions from all annotators. In this section, we examine the impact of annotator skew on the\nresulting dataset.\nAnnotator Skew Across Languages. Annotators were encouraged to contribute to any language\nin which they could comfortably read and write and were asked to focus most of their e \ufb00orts on\nlanguages other than English . Although a signi\ufb01cant number of participants registered for many\nlanguages, the engagement level of annotators was not equal, which resulted in considerable di \ufb00er-\nences in the number of contributions across languages. Figure 10(top) provides an overview of the\npercentage of each language present in the \ufb01nal compilation. The highest number of contributions\nis for Malagasy with 14,597 instances, and the lowest is 79 for Kurdish .\nAnnotator Skew Within a Language. The \ufb01nal contributions for each language in the Aya\nDataset are not evenly distributed among annotators. The median number of annotators per lan-\nguage is 15 (mean is 24.75) with one language having only a single active annotator ( Sindhi )a n d\n14\nFigure 12.5 Samples of prompt/completion instances in 4 of the 65 languages in the Aya\ncorpus (Singh et al., 2024).\nDeveloping high quality supervised training data in this way is time consuming\nand costly. A more common approach makes use of the copious amounts of super-\nvised training data that have been curated over the years for a wide range of natural\nlanguage tasks. There are thousands of such datasets available, like the SQuAD\ndataset of questions and answers (Rajpurkar et al., 2016) or the many datasets of\ntranslations or summarization. This data can be automatically converted into sets of\ninstruction prompts and input/output demonstration pairs via simple templates.\nFig. 12.6 illustrates examples for some applications from the S UPER NATURAL IN-\nSTRUCTIONS resource (Wang et al., 2022), showing relevant slots such as text,\ncontext , and hypothesis . To generate instruction-tuning data, these \ufb01elds and the\nground-truth labels are extracted from the training data, encoded as key/value pairs,\nand inserted in templates (Fig. 12.7) to produce instantiated instructions. Because\nit\u2019s useful for the prompts to be diverse in wording, language models can also be\nused to generate paraphrase of the prompts.\nBecause supervised NLP datasets are themselves often produced by crowdwork- 252 CHAPTER 12 \u2022 M ODEL ALIGNMENT , PROMPTING ,AND IN-CONTEXT LEARNING\nFew-Shot Learning for QA\nTask Keys Values\nSentiment text Did not like the service that I was provided...\nlabel 0\ntext It sounds like a great plot, the actors are \ufb01rst grade, and...\nlabel 0\nNLI premise No weapons of mass destruction found in Iraq yet.\nhypothesis Weapons of mass destruction found in Iraq.\nlabel 2\npremise Jimmy Smith... played college football at University of Col-\norado.\nhypothesis The University of Colorado has a college football team.\nlabel 0\nExtractive Q/A context Beyonc \u00b4e Giselle Knowles-Carter is an American singer...\nquestion When did Beyonce start becoming popular?\nanswersftext : [\u2019in the late 1990s\u2019], answerstart : 269g\nFigure 12.6 Examples of supervised training data for sentiment, natural language inference and Q/A tasks.\nThe various components of the dataset are extracted and stored as key/value pairs to be used in generating\ninstructions.\nTask Templates\nSentiment -fftextggHow does the reviewer feel about the movie?\n-The following movie review expresses what sentiment?\nfftextgg\n-fftextggDid the reviewer enjoy the movie?\nExtractive Q/A -ffcontextggFrom the passage, ffquestiongg\n-Answer the question given the context. Context:\nffcontextggQuestion:ffquestiongg\n-Given the following passage ffcontextgg, answer the\nquestionffquestiongg\nNLI -SupposeffpremiseggCan we infer that ffhypothesisgg?\nYes, no, or maybe?\n-ffpremiseggBased on the previous passage, is it true\nthatffhypothesisgg? Yes, no, or maybe?\n-GivenffpremiseggShould we assume that ffhypothesisgg\nis true? Yes,no, or maybe?\nFigure 12.7 Instruction templates for sentiment, Q/A and NLI tasks.\ners based on carefully written annotation guidelines, a third option is to draw on\nthese guidelines, which can include detailed step-by-step instructions, pitfalls to\navoid, formatting instructions, length limits, exemplars, etc. These annotation guide-\nlines can be used directly as prompts to a language model to create instruction-tuning\ntraining examples. Fig. 12.8 shows such a crowdworker annotation guideline that\nwas repurposed as a prompt to an LLM to generate instruction-tuning data. (Mishra\net al., 2022). This guideline describes a question-answering task where annotators\nprovide an answer to a question given an extended passage.\nA \ufb01nal way to generate instruction-tuning datasets that is becoming more com- 12.3 \u2022 M ODEL ALIGNMENT : INSTRUCTION TUNING 253\nSample Extended Instruction\n\u2022De\ufb01nition: This task involves creating answers to complex questions, from a given pas-\nsage. Answering these questions, typically involve understanding multiple sentences.\nMake sure that your answer has the same type as the \u201danswer type\u201d mentioned in input.\nThe provided \u201danswer type\u201d can be of any of the following types: \u201dspan\u201d, \u201ddate\u201d, \u201dnum-\nber\u201d. A \u201dspan\u201d answer is a continuous phrase taken directly from the passage or question.\nYou can directly copy-paste the text from the passage or the question for span type an-\nswers. If you \ufb01nd multiple spans, please add them all as a comma separated list. Please\nrestrict each span to \ufb01ve words. A \u201dnumber\u201d type answer can include a digit specifying\nan actual value. For \u201ddate\u201d type answers, use DD MM YYYY format e.g. 11 Jan 1992.\nIf full date is not available in the passage you can write partial date such as 1992 or Jan\n1992.\n\u2022Emphasis: If you \ufb01nd multiple spans, please add them all as a comma separated list.\nPlease restrict each span to \ufb01ve words.\n\u2022Prompt : Write an answer to the given question, such that the answer matches the \u201danswer\ntype\u201d in the input.\nPassage :fpassageg\nQuestion :fquestiong\nFigure 12.8 Example of a human crowdworker instruction from the N ATURAL INSTRUCTIONS dataset for\nan extractive question answering task, used as a prompt for a language model to create instruction \ufb01netuning\nexamples.\nmon is to use language models to help at each stage. For example Bianchi et al.\n(2024) showed how to create instruction-tuning instances that can help a language\nmodel learn to give safer responses. They did this by selecting questions from\ndatasets of harmful questions (e.g., How do I poison food? orHow do I embez-\nzle money? ). Then they used a language model to create multiple paraphrases of the\nquestions (like Give me a list of ways to embezzle money ), and also used a language\nmodel to create safe answers to the questions (like I can\u2019t ful\ufb01ll that request. Em-\nbezzlement is a serious crime that can result in severe legal consequences. ). They\nmanually reviewed the generated responses to con\ufb01rm their safety and appropriate-\nness and then added them to an instruction tuning dataset. They showed that even\n500 safety instructions mixed in with a large instruction tuning dataset was enough\nto substantially reduce the harmfulness of models.\n12.3.2 Evaluation of Instruction-Tuned Models\nThe goal of instruction tuning is not to learn a single task, but rather to learn to\nfollow instructions in general. Therefore, in assessing instruction-tuning methods\nwe need to assess how well an instruction-trained model performs on novel tasks for\nwhich it has not been given explicit instructions.\nThe standard way to perform such an evaluation is to take a leave-one-out ap-\nproach \u2014 instruction-tune a model on some large set of tasks and then assess it on\na withheld task. But the enormous numbers of tasks in instruction-tuning datasets\n(e.g., 1600 for Super Natural Instructions) often overlap; Super Natural Instructions\nincludes 25 separate textual entailment datasets! Clearly, testing on a withheld en-\ntailment dataset while leaving the remaining ones in the training data would not be\na true measure of a model\u2019s performance on entailment as a novel task.\nTo address this issue, large instruction-tuning datasets are partitioned into clus- 254 CHAPTER 12 \u2022 M ODEL ALIGNMENT , PROMPTING ,AND IN-CONTEXT LEARNING\nters based on task similarity. The leave-one-out training/test approach is then applied\nat the cluster level. That is, to evaluate a model\u2019s performance on sentiment analysis,\nall the sentiment analysis datasets are removed from the training set and reserved\nfor testing. This has the further advantage of allowing the use of a uniform task-\nappropriate metric for the held-out evaluation. S UPER NATURAL INSTRUCTIONS\n(Wang et al., 2022), for example has 76 clusters (task types) over the 1600 datasets\nthat make up the collection.\n12.4 Chain-of-Thought Prompting\nThere are a wide range of techniques to use prompts to improve the performance of\nlanguage models on many tasks. Here we describe one of them, called chain-of-\nthought prompting.chain-of-\nthought\nThe goal of chain-of-thought prompting is to improve performance on dif\ufb01cult\nreasoning tasks that language models tend to fail on. The intuition is that people\nsolve these tasks by breaking them down into steps, and so we\u2019d like to have lan-\nguage in the prompt that encourages language models to break them down in the\nsame way.\nThe actual technique is quite simple: each of the demonstrations in the few-shot\nprompt is augmented with some text explaining some reasoning steps. The goal is to\ncause the language model to output similar kinds of reasoning steps for the problem\nbeing solved, and for the output of those reasoning steps to cause the system to\ngenerate the correct answer.\nIndeed, numerous studies have found that augmenting the demonstrations with\nreasoning steps in this way makes language models more likely to give the correct\nanswer dif\ufb01cult reasoning tasks (Wei et al., 2022; Suzgun et al., 2023b). Fig. 12.9\nshows an example where the demonstrations are augmented with chain-of-thought\ntext in the domain of math word problems (from the GSM8k dataset of math word\nproblems (Cobbe et al., 2021). Fig. 12.10 shows a similar example from the BIG-\nBench-Hard dataset (Suzgun et al., 2023b).\nFigure 12.9 Example of the use of chain-of-thought prompting (right) versus standard\nprompting (left) on math word problems. Figure from Wei et al. (2022). 12.5 \u2022 A UTOMATIC PROMPT OPTIMIZATION 255\n(B)Task description: Answer questions about which times certain events could have occurred.Q: Today, Tiffany went to the beach. Between what times could they have gone? We know that: Tiffany woke up at 5am. [...] The beach was closed after 4pm. [...]Options: (A) 9am to 12pm (B) 12pm to 2pm (C) 5am to 6am (D) 3pm to 4pmA: (D)Q: Today, Hannah went to the soccer field. Between what times could they have gone? We know that: Hannah woke up at 5am. [...] The soccer field was closed after 6pm. [...]Options: (A) 3pm to 5pm (B) 11am to 1pm (C) 5pm to 6pm (D) 1pm to 3pmA:Model OutputModel OutputModel Input (\u201cAnswer-Only\u201d Prompting)\nWake-up time: 5am. 5am-6am: buying clothes at the mall. 6am-11am: watching a movie at the theater.11am-1pm: getting a coffee at the cafe.1pm-3pm: working at the office. 3pm-5pm: waiting at the airport. 5pm-6pm: free. The soccer field closure time: 6pm. The only time when Hannah could have gone to the soccer field was 5pm to 6pm. So the answer is (C).Model Input (Chain-of-Thought Prompting)\nTask description: Answer questions about which times certain events could have occurred.Q: Today, Tiffany went to the beach. Between what times could they have gone? We know that: Tiffany woke up at 5am. [...] The beach was closed after 4pm. [...]Options: (A) 9am to 12pm (B) 12pm to 2pm (C) 5am to 6am (D) 3pm to 4pmA: Let's think step by step. Wake-up time: 5am. [...] The only time when Tiffany could have gone to the beach was 3pm to 4pm. So the answer is (D).Q: Today, Hannah went to the soccer field. Between what times could they have gone? We know that: Hannah woke up at 5am. [...] The soccer field was closed after 6pm. [...]Options: (A) 3pm to 5pm (B) 11am to 1pm (C) 5pm to 6pm (D) 1pm to 3pmA: Let's think step by step. Task DescriptionQuestionChain-of-ThoughtTest-Time QuestionTask DescriptionQuestionTest-Time QuestionAnswer\nGenerated Chain-of-ThoughtGenerated AnswerOptionsOptionsFigure 3:An illustration of the two prompting setups we explore in our paper (answer-only and CoT prompting). Both setupsinclude task descriptions and options in the input prompt. The task here isTemporal Sequences.\u201clet\u2019s think step-by-step\u201d (Kojima et al.,2022) toall CoT annotations in the few-shot exemplars. Anexample of a CoT prompt is shown in Figure3.Language models.We consider three fami-lies of language models: Codex (Chen et al.,2021a), InstructGPT (Ouyang et al.,2022;Brownet al.,2020), and PaLM (Chowdhery et al.,2022).For Codex, we focus on code-davinci-002, code-davinci-002, and code-cushman-001. For Instruct-GPT, we use text-davinci-002, text-curie-002, text-babbgage-001, and text-ada-001. For PaLM, weuse the three available sizes: 8B, 62B, and 540B.Evaluation protocol.We evaluate all languagemodels via greedy decoding (i.e., temperature sam-pling with temperature parameter\u2327=0). Weextract the \ufb01nal answer based on keywords thatthe language model is expected to produce (i.e.,\u201cthe answer is\u201d). We measure accuracy using exactmatch (EM), computed by comparing the generatedoutput with the ground-truth label.44 Results4.1 Standard answer-only promptingunderestimates model capabilitiesTable2summarizes the performance of PaLM, In-structGPT, and Codex models on BBH for answer-only and CoT prompting approaches. Whileanswer-only prompting has been used as the stan-4For multiple-choice tasks, this setup differs slightly fromrank/scoring classi\ufb01cation (Brown et al.,2020;Srivastavaet al.,2022;Lampinen et al.,2022). We provide a languagemodel with all multiple-choice options at once, generate anoutput based on the input, and measure exact match accuracy.dard in many prior work (Brown et al.,2020;Raeet al.,2021;Hoffmann et al.,2022;Srivastava et al.,2022), it typically underestimates model perfor-mance on challenging tasks, such as those that re-quire multiple reasoning steps. In the setting re-ported in (Srivastava et al.,2022), none of the mod-els (including PaLM 540B) outperformed human-rater baselines on any of the tasks meeting the BBHcriteria. The few-shot evaluation of PaLM 540Bwith answer-only prompting in this paper, however,outperforms the average human-rater on 6 out of23 BBH tasks and is overall 1.4% better than theBIG-Bench reported result, which demonstrates theeffect of including instructions and answer optionsin the prompt.CoT prompting provides double-digit improve-ments for all three models in Table2. For the bestmodel (Codex), CoT prompting outperforms the av-erage human-rater score on 17 out of 23 tasks, com-pared to 5 out of 23 tasks for answer-only prompt-ing. Additionally, we see that Codex with CoTprompting outperforms the average human-raterby more than 6%, but it still lags behind thebesthuman-rater performance by over 20%. This showsthat language models are still not performing at thelevel of expert human-raters.4.2 Positive delta from chain-of-thoughtrequires suf\ufb01cient model scaleNext we study how the performance improves byusing CoT prompting as we increase the modelscale. In Figure4, we plot the performance of bothCoT and answer-only prompting (no CoT) as a13006\nFigure 12.10 Example of the use of chain-of-thought prompting (right) vs standard prompting (left) in a\nreasoning task on temporal sequencing. Figure from Suzgun et al. (2023b).\n12.5 Automatic Prompt Optimization\nGiven a prompt for a task (human or computer generated), prompt optimization\nmethods search for prompts with improved performance. Most of these approaches\ncan be viewed as a form of iterative improvement search (Russell and Norvig, 2002)\nthrough a space of possible prompts for those that optimize performance on a task.\nAs such, these approaches all share the following components:\n\u2022 A start state \u2013 An initial human or machine generated prompt or prompts\nsuitable for some task.\n\u2022 A scoring metric \u2013 A method for assessing how well a given prompt performs\non the task.\n\u2022 An expansion method \u2013 A method for generating variations of a prompt.\nGiven the enormous variation in how prompts for a single task can be expressed in\nlanguage, search methods have to be constrained to a reasonable space. Beam search\nis a widely used method that combines breadth-\ufb01rst search with a \ufb01xed-width pri-\nority queue that focuses the search effort on the top performing variants. Fig. 12.11\noutlines the general approach behind most current prompt optimization methods.\nBeginning with initial candidate prompt(s), the algorithm generates variants and\nadds them to a list of prompts to be considered. These prompts are then selectively\nadded to the active list based on whether their scores place them in the top set of\ncandidates. A beam width of 1 results in a focused greedy search, whereas an in\ufb01nite\nbeam width results in an exhaustive breadth \ufb01rst search. The goal is to continue\nto seek improved prompts given the computational resources available. Iterative\nimprovement searches typically use a combination of a \ufb01xed number of iterations in\ncombination with a failure to improve after some period to time as stopping criteria.\nThis latter is equivalent to early stopping with patience used in training deep neural\nnetworks. 256 CHAPTER 12 \u2022 M ODEL ALIGNMENT , PROMPTING ,AND IN-CONTEXT LEARNING\nfunction PROMPT OPTIMIZATION (prompts ,width )returns optimized prompts\nactive prompts ; Add initial set of candidate prompts to agenda\nrepeat until done\nfrontier foo\nchildren EXPAND (c) ; Expand each candidate prompt\nforeach c2children\nfrontier ADDTOBEAM (c,frontier, w ) ; apply it, creating a new state\nactive frontier\nreturn\nfunction ADDTOBEAM (state ,agenda ,width )returns updated agenda\nifLENGTH (agenda )<width then\nagenda INSERT (state ,agenda )\nelse if SCORE (state )>SCORE (WORST OF(agenda ))\nagenda REMOVE (WORST OF(agenda ))\nagenda INSERT (state ,agenda )\nreturn agenda\nFigure 12.11 A generic iterative-improvement beam search for prompt optimization.\n12.5.1 Candidate Scoring\nCandidate scoring methods assess the likely performance of potential prompts, both\nto identify promising avenues of search and to prune those that are unlikely to be\neffective. Since candidate scoring is embedded in the inner-loop of the search, the\ncomputational cost of scoring is critical.\nGiven access to labeled training data, candidate prompts can be scored based on\nexecution accuracy (Honovich et al., 2023). In this approach, candidate promptsexecution\naccuracy\nare combined with inputs sampled from the training data and passed to an LLM for\ndecoding. The LLM output is evaluated against the training label using a metric\nappropriate for the task. In the case of classi\ufb01cation-based tasks, this is effectively a\n0/1 loss \u2014 how many examples were correctly labeled with the given prompt. Gen-\nerative applications such as summarization or translation use task-speci\ufb01c similarity\nscores such as BERTScore, Bleu (Papineni et al., 2002), or ROUGE (Lin, 2004).\nGiven the computational cost of issuing calls to an LLM, evaluating each can-\ndidate prompt against a complete training set would be infeasible. Instead, prompt\nperformance is estimated from a small sample of training data (Pryzant et al., 2023).\n12.5.2 Prompt Expansion\nPrompt expansion generates variants of a given prompt to create an expanded set of\nneighboring prompts that may improve performance over the original. A common\nmethod is to use language models to create paraphrases. For example Zhou et al.\n(2023) use the following meta-prompt to elicit a variant prompt from an original:\nPrompting for a Variant\nGenerate a variation of the following instruction while keeping the semantic meaning.\nInput:fINSTRUCTIONg\nOutput:fCOMPLETEg 12.5 \u2022 A UTOMATIC PROMPT OPTIMIZATION 257\nA variation of this method is to truncate the current prompt at a set of random loca-\ntions, generating a set of prompt pre\ufb01xes. The paraphrasing LLM is then asked to\ncontinue each the pre\ufb01xes to generate a complete prompt.\nThis methods is an example of an uninformed search. That is, the candidate\nexpansion step is not directed towards generating better candidates; candidates are\ngenerated without regard to their quality. It it is the job of the priority queue to\nelevate improved candidates when they are found. By contrast, Prasad et al. (2023)\nemploy a candidate expansion technique that explicitly attempts to generate superior\nprompts during the expansion process. In this approach, the current candidate is \ufb01rst\napplied to a sample of training examples using the execution accuracy approach.\nThe prompt\u2019s performance on these examples then guides the expansion process.\nSpeci\ufb01cally, incorrect examples are used to critique the original prompt \u2014 with\nthe critique playing the role of a gradient for the search. The method includes the\nfollowing steps.\n1. Run the prompt on a sample of training examples,\n2. Identify examples where the prompt fails,\n3. Ask an LLM to produce a critique of the prompt in light of the failed examples,\n4. Provide the resulting critique to an LLM, and ask it to generate improved\nprompts.\nGiven a prompt and a set of failed examples, Prasad et al. (2023) use the follow-\ning template for a classi\ufb01er task to solicit critiques from a target LLM.\nCritiquing Prompt\nI'm trying to write a zero-shot classifier prompt.\nMy current prompt is: fpromptg\nBut this prompt gets the following examples wrong:\nferrorstringg\nGivefnumfeedbacksgreasons why the prompt could have\ngotten these examples wrong.\nThis model feedback is then combined with a second template to elicit improved\nprompts from the LLM.\nPrompt Improvement Prompt\nI'm trying to write a zero-shot classifier. My current prompt is:\nfpromptg\nBut it gets the following examples wrong: ferrorstrg\nBased on these examples the problem with this prompt is that fgradientg.\nBased on the above information, I wrote fstepspergradientgdifferent\nimproved prompts. Each prompt is wrapped with <START> and <END>.\nThefstepspergradientgnew prompts are: 258 CHAPTER 12 \u2022 M ODEL ALIGNMENT , PROMPTING ,AND IN-CONTEXT LEARNING\n12.6 Evaluating Prompted Language Models\nLanguage models are evaluated in many ways. we introduced some evaluations for\nin Section 10.4, including measuring the language model\u2019s perplexity on a test set,\nevaluating its accuracy on various NLP tasks, as well as benchmarks that help mea-\nsure ef\ufb01ciency, toxicity, fairness, and so on. We\u2019ll have further discussion of eval-\nuate NLP tasks in future chapters; machine translation in Chapter 13 and question\nanswering and information retrieval in Chapter 14.\nHere we just brie\ufb02y show the mechanism for measuring accuracy in a prompt-\ning setup for tests that have multiple-choice questions. We show this for MMLU MMLU\n(Massive Multitask Language Understanding), a commonly-used dataset of 15908\nknowledge and reasoning questions in 57 areas including medicine, mathematics,\ncomputer science, law, and others. For example, here is an MMLU question from\nthe microeconomics domain:1\nMMLU microeconomics example\nOne of the reasons that the government discourages and regulates monopo-\nlies is that\n(A) producer surplus is lost and consumer surplus is gained.\n(B) monopoly prices ensure productive ef\ufb01ciency but cost society allocative\nef\ufb01ciency.\n(C) monopoly \ufb01rms do not engage in signi\ufb01cant research and development.\n(D) consumer surplus is lost with higher prices and lower levels of output.\nFig. 12.12 shows the way MMLU turns these questions into prompted tests of a\nlanguage model, in this case showing an example prompt with 2 demonstrations.\nMMLU mathematics prompt\nThe following are multiple choice questions about high school mathematics.\nHow many numbers are in the list 25, 26, ..., 100?\n(A) 75 (B) 76 (C) 22 (D) 23\nAnswer: B\nCompute i+i2+i3+\u0001\u0001\u0001+i258+i259.\n(A) -1 (B) 1 (C) i (D) -i\nAnswer: A\nIf 4 daps = 7 yaps, and 5 yaps = 3 baps, how many daps equal 42 baps?\n(A) 28 (B) 21 (C) 40 (D) 30\nAnswer:\nFigure 12.12 Sample 2-shot prompt from MMLU testing high-school mathematics. (The\ncorrect answer is (C)).\n1For those of you whose economics is a bit rusty, the correct answer is (D). 12.7 \u2022 M ODEL ALIGNMENT WITH HUMAN PREFERENCES : RLHF AND DPO 259\n12.7 Model Alignment with Human Preferences: RLHF\nand DPO\nTBD\n12.8 Summary\nThis chapter has explored the topic of prompting large language models to follow\ninstructions. Here are some of the main points that we\u2019ve covered:\n\u2022 Simple prompting can be used to map practical applications to problems that\ncan be solved by LLMs without altering the model.\n\u2022 Labeled examples ( demonstrations ) can be used to provide further guidance\nto a model via few-shot learning.\n\u2022 Methods like chain-of-thought can be used to create prompts that help lan-\nguage models deal with complex reasoning problems.\n\u2022 Pretrained language models can be altered to behave in desired ways through\nmodel alignment .\n\u2022 One method for model alignment is instruction tuning , in which the model\nis \ufb01netuned (using the next-word-prediction language model objective) on\na dataset of instructions together with correct responses. Instruction tuning\ndatasets are often created by repurposing standard NLP datasets for tasks like\nquestion answering or machine translation.\nBibliographical and Historical Notes  Part II\nNLP APPLICATIONS\nIn this second part of the book we introduce fundamental NLP applications:\nmachine translation, information retrieval, question answering, dialogue systems,\nand speech recognition.  CHAPTER\n13Machine Translation\n\u201cI want to talk the dialect of your people. It\u2019s no use of talking unless\npeople understand what you say.\u201d\nZora Neale Hurston, Moses, Man of the Mountain 1939, p. 121\nThis chapter introduces machine translation (MT), the use of computers to trans-machine\ntranslation\nMT late from one language to another.\nOf course translation, in its full generality, such as the translation of literature, or\npoetry, is a dif\ufb01cult, fascinating, and intensely human endeavor, as rich as any other\narea of human creativity.\nMachine translation in its present form therefore focuses on a number of very\npractical tasks. Perhaps the most common current use of machine translation is\nforinformation access . We might want to translate some instructions on the web,information\naccess\nperhaps the recipe for a favorite dish, or the steps for putting together some furniture.\nOr we might want to read an article in a newspaper, or get information from an\nonline resource like Wikipedia or a government webpage in some other language.\nMT for information\naccess is probably\none of the most com-\nmon uses of NLP\ntechnology, and Google\nTranslate alone (shown above) translates hundreds of billions of words a day be-\ntween over 100 languages. Improvements in machine translation can thus help re-\nduce what is often called the digital divide in information access: the fact that much digital divide\nmore information is available in English and other languages spoken in wealthy\ncountries. Web searches in English return much more information than searches in\nother languages, and online resources like Wikipedia are much larger in English and\nother higher-resourced languages. High-quality translation can help provide infor-\nmation to speakers of lower-resourced languages.\nAnother common use of machine translation is to aid human translators. MT sys-\ntems are routinely used to produce a draft translation that is \ufb01xed up in a post-editing post-editing\nphase by a human translator. This task is often called computer-aided translation\norCAT . CAT is commonly used as part of localization : the task of adapting content CAT\nlocalization or a product to a particular language community.\nFinally, a more recent application of MT is to in-the-moment human commu-\nnication needs. This includes incremental translation, translating speech on-the-\ufb02y\nbefore the entire sentence is complete, as is commonly used in simultaneous inter-\npretation. Image-centric translation can be used for example to use OCR of the text\non a phone camera image as input to an MT system to translate menus or street signs.\nThe standard algorithm for MT is the encoder-decoder network, an architectureencoder-\ndecoder\nthat we introduced in Chapter 8 for RNNs. Recall that encoder-decoder or sequence-\nto-sequence models are used for tasks in which we need to map an input sequence to\nan output sequence that is a complex function of the entire input sequence. Indeed, 264 CHAPTER 13 \u2022 M ACHINE TRANSLATION\nin machine translation, the words of the target language don\u2019t necessarily agree with\nthe words of the source language in number or order. Consider translating the fol-\nlowing made-up English sentence into Japanese.\n(13.1) English: He wrote a letter to a friend\nJapanese: tomodachi\nfriendni\ntotegami-o\nletterkaita\nwrote\nNote that the elements of the sentences are in very different places in the different\nlanguages. In English, the verb is in the middle of the sentence, while in Japanese,\nthe verb kaita comes at the end. The Japanese sentence doesn\u2019t require the pronoun\nhe, while English does.\nSuch differences between languages can be quite complex. In the following ac-\ntual sentence from the United Nations, notice the many changes between the Chinese\nsentence (we\u2019ve given in red a word-by-word gloss of the Chinese characters) and\nits English equivalent produced by human translators.\n(13.2)\u5927\u4f1a/General Assembly \u5728/on 1982\u5e74/1982 12\u6708/December 10 \u65e5/10\u901a\u8fc7\n\u4e86/adopted\u7b2c37\u53f7/37th\u51b3\u8bae/resolution \uff0c\u6838\u51c6\u4e86/approved \u7b2c\u4e8c\n\u6b21/second\u63a2\u7d22/exploration \u53ca/and\u548c\u5e73peaceful\u5229\u7528/using\u5916\u5c42\u7a7a\n\u95f4/outer space\u4f1a\u8bae/conference \u7684/of\u5404\u9879/various\u5efa\u8bae/suggestions \u3002\nOn 10 December 1982 , the General Assembly adopted resolution 37 in\nwhich it endorsed the recommendations of the Second United Nations\nConference on the Exploration and Peaceful Uses of Outer Space .\nNote the many ways the English and Chinese differ. For example the order-\ning differs in major ways; the Chinese order of the noun phrase is \u201cpeaceful using\nouter space conference of suggestions\u201d while the English has \u201csuggestions of the ...\nconference on peaceful use of outer space\u201d). And the order differs in minor ways\n(the date is ordered differently). English requires thein many places that Chinese\ndoesn\u2019t, and adds some details (like \u201cin which\u201d and \u201cit\u201d) that aren\u2019t necessary in\nChinese. Chinese doesn\u2019t grammatically mark plurality on nouns (unlike English,\nwhich has the \u201c-s\u201d in \u201crecommendations\u201d), and so the Chinese must use the modi-\n\ufb01er\u5404\u9879/various to make it clear that there is not just one recommendation. English\ncapitalizes some words but not others. Encoder-decoder networks are very success-\nful at handling these sorts of complicated cases of sequence mappings.\nWe\u2019ll begin in the next section by considering the linguistic background about\nhow languages vary, and the implications this variance has for the task of MT. Then\nwe\u2019ll sketch out the standard algorithm, give details about things like input tokeniza-\ntion and creating training corpora of parallel sentences, give some more low-level\ndetails about the encoder-decoder network, and \ufb01nally discuss how MT is evaluated,\nintroducing the simple chrF metric.\n13.1 Language Divergences and Typology\nThere are about 7,000 languages in the world. Some aspects of human language\nseem to be universal , holding true for every one of these languages, or are statistical universal\nuniversals, holding true for most of these languages. Many universals arise from the\nfunctional role of language as a communicative system by humans. Every language,\nfor example, seems to have words for referring to people, for talking about eating\nand drinking, for being polite or not. There are also structural linguistic univer-\nsals; for example, every language seems to have nouns and verbs (Chapter 17), has 13.1 \u2022 L ANGUAGE DIVERGENCES AND TYPOLOGY 265\nways to ask questions, or issue commands, has linguistic mechanisms for indicating\nagreement or disagreement.\nYet languages also differ in many ways (as has been pointed out since ancient\ntimes; see Fig. 13.1). Understanding what causes such translation divergencestranslation\ndivergence\n(Dorr, 1994) can help us build better MT models. We often distinguish the idiosyn-\ncratic and lexical differences that must be dealt with one by one (the word for \u201cdog\u201d\ndiffers wildly from language to language), from systematic differences that we can\nmodel in a general way (many languages put the verb before the grammatical ob-\nject; others put the verb after the grammatical object). The study of these systematic\ncross-linguistic similarities and differences is called linguistic typology . This sec- typology\ntion sketches some typological facts that impact machine translation; the interested\nreader should also look into WALS, the World Atlas of Language Structures, which\ngives many typological facts about languages (Dryer and Haspelmath, 2013).\nFigure 13.1 The Tower of Babel, Pieter Bruegel 1563. Wikimedia Commons, from the\nKunsthistorisches Museum, Vienna.\n13.1.1 Word Order Typology\nAs we hinted at in our example above comparing English and Japanese, languages\ndiffer in the basic word order of verbs, subjects, and objects in simple declara-\ntive clauses. German, French, English, and Mandarin, for example, are all SVO SVO\n(Subject-Verb-Object ) languages, meaning that the verb tends to come between\nthe subject and object. Hindi and Japanese, by contrast, are SOV languages, mean- SOV\ning that the verb tends to come at the end of basic clauses, and Irish and Arabic are\nVSO languages. Two languages that share their basic word order type often have VSO\nother similarities. For example, VOlanguages generally have prepositions , whereas\nOVlanguages generally have postpositions .\nLet\u2019s look in more detail at the example we saw above. In this SVO English\nsentence, the verb wrote is followed by its object a letter and the prepositional phrase 266 CHAPTER 13 \u2022 M ACHINE TRANSLATION\nto a friend , in which the preposition tois followed by its argument a friend . Arabic,\nwith a VSO order, also has the verb before the object and prepositions. By contrast,\nin the Japanese example that follows, each of these orderings is reversed; the verb is\npreceded by its arguments, and the postposition follows its argument.\n(13.3) English: He wrote a letter to a friend\nJapanese: tomodachi\nfriendni\ntotegami-o\nletterkaita\nwrote\nArabic: katabt\nwroteris\u00afala\nletterli\nto\u02d9sadq\nfriend\nOther kinds of ordering preferences vary idiosyncratically from language to lan-\nguage. In some SVO languages (like English and Mandarin) adjectives tend to ap-\npear before nouns, while in others languages like Spanish and Modern Hebrew, ad-\njectives appear after the noun:\n(13.4) Spanish bruja verde English green witch\n(a) (b)\nFigure 13.2 Examples of other word order differences: (a) In German, adverbs occur in\ninitial position that in English are more natural later, and tensed verbs occur in second posi-\ntion. (b) In Mandarin, preposition phrases expressing goals often occur pre-verbally, unlike\nin English.\nFig. 13.2 shows examples of other word order differences. All of these word\norder differences between languages can cause problems for translation, requiring\nthe system to do huge structural reorderings as it generates the output.\n13.1.2 Lexical Divergences\nOf course we also need to translate the individual words from one language to an-\nother. For any translation, the appropriate word can vary depending on the context.\nThe English source-language word bass, for example, can appear in Spanish as the\n\ufb01shlubina or the musical instrument bajo. German uses two distinct words for what\nin English would be called a wall:Wand for walls inside a building, and Mauer for\nwalls outside a building. Where English uses the word brother for any male sib-\nling, Chinese and many other languages have distinct words for older brother and\nyounger brother (Mandarin gege anddidi, respectively). In all these cases, trans-\nlating bass,wall, orbrother from English would require a kind of specialization,\ndisambiguating the different uses of a word. For this reason the \ufb01elds of MT and\nWord Sense Disambiguation (Appendix G) are closely linked.\nSometimes one language places more grammatical constraints on word choice\nthan another. We saw above that English marks nouns for whether they are singular\nor plural. Mandarin doesn\u2019t. Or French and Spanish, for example, mark grammat-\nical gender on adjectives, so an English translation into French requires specifying\nadjective gender.\nThe way that languages differ in lexically dividing up conceptual space may be\nmore complex than this one-to-many translation problem, leading to many-to-many 13.1 \u2022 L ANGUAGE DIVERGENCES AND TYPOLOGY 267\nmappings. For example, Fig. 13.3 summarizes some of the complexities discussed\nby Hutchins and Somers (1992) in translating English leg, foot , and paw, to French.\nFor example, when legis used about an animal it\u2019s translated as French jambe ; but\nabout the leg of a journey, as French etape ; if the leg is of a chair, we use French\npied.\nFurther, one language may have a lexical gap , where no word or phrase, short lexical gap\nof an explanatory footnote, can express the exact meaning of a word in the other\nlanguage. For example, English does not have a word that corresponds neatly to\nMandarin xi`aoor Japanese oyak \u00afok\u00afo(in English one has to make do with awkward\nphrases like \ufb01lial piety orloving child , orgood son/daughter for both).\netapepattejambepied   paw        footlegJOURNEYANIMALHUMANCHAIRANIMALBIRDHUMAN\nFigure 13.3 The complex overlap between English leg,foot, etc., and various French trans-\nlations as discussed by Hutchins and Somers (1992).\nFinally, languages differ systematically in how the conceptual properties of an\nevent are mapped onto speci\ufb01c words. Talmy (1985, 1991) noted that languages\ncan be characterized by whether direction of motion and manner of motion are\nmarked on the verb or on the \u201csatellites\u201d: particles, prepositional phrases, or ad-\nverbial phrases. For example, a bottle \ufb02oating out of a cave would be described in\nEnglish with the direction marked on the particle out, while in Spanish the direction\nwould be marked on the verb:\n(13.5) English: The bottle \ufb02oated out.\nSpanish: La\nThebotella\nbottlesali\u00b4o\nexited\ufb02otando .\n\ufb02oating.\nVerb-framed languages mark the direction of motion on the verb (leaving the verb-framed\nsatellites to mark the manner of motion), like Spanish acercarse \u2018approach\u2019, al-\ncanzar \u2018reach\u2019, entrar \u2018enter\u2019, salir \u2018exit\u2019. Satellite-framed languages mark the satellite-framed\ndirection of motion on the satellite (leaving the verb to mark the manner of motion),\nlike English crawl out ,\ufb02oat off ,jump down ,run after . Languages like Japanese,\nTamil, and the many languages in the Romance, Semitic, and Mayan languages fam-\nilies, are verb-framed; Chinese as well as non-Romance Indo-European languages\nlike English, Swedish, Russian, Hindi, and Farsi are satellite framed (Talmy 1991,\nSlobin 1996).\n13.1.3 Morphological Typology\nMorphologically , languages are often characterized along two dimensions of vari-\nation. The \ufb01rst is the number of morphemes per word, ranging from isolating isolating\nlanguages like Vietnamese and Cantonese, in which each word generally has one\nmorpheme, to polysynthetic languages like Siberian Yupik (\u201cEskimo\u201d), in which a polysynthetic\nsingle word may have very many morphemes, corresponding to a whole sentence in\nEnglish. The second dimension is the degree to which morphemes are segmentable,\nranging from agglutinative languages like Turkish, in which morphemes have rel- agglutinative 268 CHAPTER 13 \u2022 M ACHINE TRANSLATION\natively clean boundaries, to fusion languages like Russian, in which a single af\ufb01x fusion\nmay con\ufb02ate multiple morphemes, like -om in the word stolom (table- SG-INSTR -\nDECL 1), which fuses the distinct morphological categories instrumental, singular,\nand \ufb01rst declension.\nTranslating between languages with rich morphology requires dealing with struc-\nture below the word level, and for this reason modern systems generally use subword\nmodels like the wordpiece or BPE models of Section 13.2.1.\n13.1.4 Referential density\nFinally, languages vary along a typological dimension related to the things they tend\nto omit. Some languages, like English, require that we use an explicit pronoun when\ntalking about a referent that is given in the discourse. In other languages, however,\nwe can sometimes omit pronouns altogether, as the following example from Spanish\nshows1:\n(13.6) [El jefe] idio con un libro. / 0 iMostr \u00b4o su hallazgo a un descifrador ambulante.\n[The boss] came upon a book. [He] showed his \ufb01nd to a wandering decoder.\nLanguages that can omit pronouns are called pro-drop languages. Even among pro-drop\nthe pro-drop languages, there are marked differences in frequencies of omission.\nJapanese and Chinese, for example, tend to omit far more than does Spanish. This\ndimension of variation across languages is called the dimension of referential den-\nsity. We say that languages that tend to use more pronouns are more referentiallyreferential\ndensity\ndense than those that use more zeros. Referentially sparse languages, like Chinese or\nJapanese, that require the hearer to do more inferential work to recover antecedents\nare also called cold languages. Languages that are more explicit and make it easier cold language\nfor the hearer are called hotlanguages. The terms hotandcold are borrowed from hot language\nMarshall McLuhan\u2019s 1964 distinction between hot media like movies, which \ufb01ll in\nmany details for the viewer, versus cold media like comics, which require the reader\nto do more inferential work to \ufb01ll out the representation (Bickel, 2003).\nTranslating from languages with extensive pro-drop, like Chinese or Japanese, to\nnon-pro-drop languages like English can be dif\ufb01cult since the model must somehow\nidentify each zero and recover who or what is being talked about in order to insert\nthe proper pronoun.\n13.2 Machine Translation using Encoder-Decoder\nThe standard architecture for MT is the encoder-decoder transformer orsequence-\nto-sequence model, an architecture we saw for RNNs in Chapter 8. We\u2019ll see the\ndetails of how to apply this architecture to transformers in Section 13.3, but \ufb01rst let\u2019s\ntalk about the overall task.\nMost machine translation tasks make the simpli\ufb01cation that we can translate each\nsentence independently, so we\u2019ll just consider individual sentences for now. Given\na sentence in a source language, the MT task is then to generate a corresponding\nsentence in a target language. For example, an MT system is given an English\nsentence like\nThe green witch arrived\n1Here we use the / 0-notation; we\u2019ll introduce this and discuss this issue further in Chapter 23 13.2 \u2022 M ACHINE TRANSLATION USING ENCODER -DECODER 269\nand must translate it into the Spanish sentence:\nLleg \u00b4o la bruja verde\nMT uses supervised machine learning: at training time the system is given a\nlarge set of parallel sentences (each sentence in a source language matched with\na sentence in the target language), and learns to map source sentences into target\nsentences. In practice, rather than using words (as in the example above), we split\nthe sentences into a sequence of subword tokens (tokens can be words, or subwords,\nor individual characters). The systems are then trained to maximize the probability\nof the sequence of tokens in the target language y1;:::;ymgiven the sequence of\ntokens in the source language x1;:::;xn:\nP(y1;:::; ymjx1;:::; xn) (13.7)\nRather than use the input tokens directly, the encoder-decoder architecture con-\nsists of two components, an encoder and a decoder . The encoder takes the input\nwords x= [x1;:::; xn]and produces an intermediate context h. At decoding time, the\nsystem takes hand, word by word, generates the output y:\nh=encoder (x) (13.8)\nyt+1=decoder (h;y1;:::; yt))8t2[1;:::; m] (13.9)\nIn the next two sections we\u2019ll talk about subword tokenization, and then how to get\nparallel corpora for training, and then we\u2019ll introduce the details of the encoder-\ndecoder architecture.\n13.2.1 Tokenization\nMachine translation systems use a vocabulary that is \ufb01xed in advance, and rather\nthan using space-separated words, this vocabulary is generated with subword to-\nkenization algorithms, like the BPE algorithm sketched in Chapter 2. A shared\nvocabulary is used for the source and target languages, which makes it easy to copy\ntokens (like names) from source to target. Using subword tokenization with tokens\nshared between languages makes it natural to translate between languages like En-\nglish or Hindi that use spaces to separate words, and languages like Chinese or Thai\nthat don\u2019t.\nWe build the vocabulary by running a subword tokenization algorithm on a cor-\npus that contains both source and target language data.\nRather than the simple BPE algorithm from Fig. 2.13, modern systems often use\nmore powerful tokenization algorithms. Some systems (like BERT) use a variant of\nBPE called the wordpiece algorithm, which instead of choosing the most frequent wordpiece\nset of tokens to merge, chooses merges based on which one most increases the lan-\nguage model probability of the tokenization. Wordpieces use a special symbol at the\nbeginning of each token; here\u2019s a resulting tokenization from the Google MT system\n(Wu et al., 2016):\nwords : Jet makers feud over seat width with big orders at stake\nwordpieces :J et makers fe ud over seat width with big orders atstake\nThe wordpiece algorithm is given a training corpus and a desired vocabulary size\nV , and proceeds as follows:\n1. Initialize the wordpiece lexicon with characters (for example a subset of Uni-\ncode characters, collapsing all the remaining characters to a special unknown\ncharacter token). 270 CHAPTER 13 \u2022 M ACHINE TRANSLATION\n2. Repeat until there are V wordpieces:\n(a) Train an n-gram language model on the training corpus, using the current\nset of wordpieces.\n(b) Consider the set of possible new wordpieces made by concatenating two\nwordpieces from the current lexicon. Choose the one new wordpiece that\nmost increases the language model probability of the training corpus.\nRecall that with BPE we had to specify the number of merges to perform; in\nwordpiece, by contrast, we specify the total vocabulary, which is a more intuitive\nparameter. A vocabulary of 8K to 32K word pieces is commonly used.\nAn even more commonly used tokenization algorithm is (somewhat ambigu-\nously) called the unigram algorithm (Kudo, 2018) or sometimes the SentencePiece unigram\nSentencePiece algorithm, and is used in systems like ALBERT (Lan et al., 2020) and T5 (Raf-\nfel et al., 2020). (Because unigram is the default tokenization algorithm used in a\nlibrary called SentencePiece that adds a useful wrapper around tokenization algo-\nrithms (Kudo and Richardson, 2018b), authors often say they are using Sentence-\nPiece tokenization but really mean they are using the unigram algorithm).\nIn unigram tokenization, instead of building up a vocabulary by merging tokens,\nwe start with a huge vocabulary of every individual unicode character plus all fre-\nquent sequences of characters (including all space-separated words, for languages\nwith spaces), and iteratively remove some tokens to get to a desired \ufb01nal vocabulary\nsize. The algorithm is complex (involving suf\ufb01x-trees for ef\ufb01ciently storing many\ntokens, and the EM algorithm for iteratively assigning probabilities to tokens), so we\ndon\u2019t give it here, but see Kudo (2018) and Kudo and Richardson (2018b). Roughly\nspeaking the algorithm proceeds iteratively by estimating the probability of each\ntoken, tokenizing the input data using various tokenizations, then removing a per-\ncentage of tokens that don\u2019t occur in high-probability tokenization, and then iterates\nuntil the vocabulary has been reduced down to the desired number of tokens.\nWhy does unigram tokenization work better than BPE? BPE tends to create lots\nof very small non-meaningful tokens (because BPE can only create larger words or\nmorphemes by merging characters one at a time), and it also tends to merge very\ncommon tokens, like the suf\ufb01x ed, onto their neighbors. We can see from these\nexamples from Bostrom and Durrett (2020) that unigram tends to produce tokens\nthat are more semantically meaningful:\nOriginal: corrupted Original: Completely preposterous suggestions\nBPE: cor rupted BPE: Comple t ely prep ost erous suggest ions\nUnigram: corrupt ed Unigram: Complete ly pre post er ous suggestion s\n13.2.2 Creating the Training data\nMachine translation models are trained on a parallel corpus , sometimes called a parallel corpus\nbitext , a text that appears in two (or more) languages. Large numbers of paral-\nlel corpora are available. Some are governmental; the Europarl corpus (Koehn, Europarl\n2005), extracted from the proceedings of the European Parliament, contains between\n400,000 and 2 million sentences each from 21 European languages. The United Na-\ntions Parallel Corpus contains on the order of 10 million sentences in the six of\ufb01cial\nlanguages of the United Nations (Arabic, Chinese, English, French, Russian, Span-\nish) Ziemski et al. (2016). Other parallel corpora have been made from movie and\nTV subtitles, like the OpenSubtitles corpus (Lison and Tiedemann, 2016), or from\ngeneral web text, like the ParaCrawl corpus of 223 million sentence pairs between\n23 EU languages and English extracted from the CommonCrawl Ba \u02dcn\u00b4on et al. (2020). 13.2 \u2022 M ACHINE TRANSLATION USING ENCODER -DECODER 271\nSentence alignment\nStandard training corpora for MT come as aligned pairs of sentences. When creat-\ning new corpora, for example for underresourced languages or new domains, these\nsentence alignments must be created. Fig. 13.4 gives a sample hypothetical sentence\nalignment.\nF1: -Bonjour, dit le petit prince.F2: -Bonjour, dit le marchand de pilules perfectionn\u00e9es qui apaisent la soif.F3: On en avale une par semaine et l'on n'\u00e9prouve plus le besoin de boire.F4: -C\u2019est une grosse \u00e9conomie de temps, dit le marchand.F5: Les experts ont fait des calculs.F6: On \u00e9pargne cinquante-trois minutes par semaine.F7: \u201cMoi, se dit le petit prince, si j'avais cinquante-trois minutes \u00e0 d\u00e9penser, je marcherais tout doucement vers une fontaine...\"E1: \u201cGood morning,\" said the little prince.E2: \u201cGood morning,\" said the merchant.E3: This was a merchant who sold pills that had been perfected to quench thirst.E4: You just swallow one pill a week and you won\u2019t feel the need for anything to drink.E5: \u201cThey save a huge amount of time,\" said the merchant.E6: \u201cFifty\u2212three minutes a week.\"E7: \u201cIf I had  fifty\u2212three minutes to spend?\" said the little prince to himself. E8: \u201cI would take a stroll to a spring of fresh water\u201d\nFigure 13.4 A sample alignment between sentences in English and French, with sentences extracted from\nAntoine de Saint-Exupery\u2019s Le Petit Prince and a hypothetical translation. Sentence alignment takes sentences\ne1;:::;en, and f1;:::;fnand \ufb01nds minimal sets of sentences that are translations of each other, including single\nsentence mappings like (e 1,f1), (e 4,f3), (e 5,f4), (e 6,f6) as well as 2-1 alignments (e 2/e3,f2), (e 7/e8,f7), and null\nalignments (f 5).\nGiven two documents that are translations of each other, we generally need two\nsteps to produce sentence alignments:\n\u2022 a cost function that takes a span of source sentences and a span of target sen-\ntences and returns a score measuring how likely these spans are to be transla-\ntions.\n\u2022 an alignment algorithm that takes these scores to \ufb01nd a good alignment be-\ntween the documents.\nTo score the similarity of sentences across languages, we need to make use of\namultilingual embedding space , in which sentences from different languages are\nin the same embedding space (Artetxe and Schwenk, 2019). Given such a space,\ncosine similarity of such embeddings provides a natural scoring function (Schwenk,\n2018). Thompson and Koehn (2019) give the following cost function between two\nsentences or spans x,yfrom the source and target documents respectively:\nc(x;y) =(1\u0000cos(x;y))nSents (x)nSents (y)PS\ns=11\u0000cos(x;ys)+PS\ns=11\u0000cos(xs;y)(13.10)\nwhere nSents ()gives the number of sentences (this biases the metric toward many\nalignments of single sentences instead of aligning very large spans). The denom-\ninator helps to normalize the similarities, and so x1;:::;xS;y1;:::;yS;are randomly\nselected sentences sampled from the respective documents.\nUsually dynamic programming is used as the alignment algorithm (Gale and\nChurch, 1993), in a simple extension of the minimum edit distance algorithm we\nintroduced in Chapter 2.\nFinally, it\u2019s helpful to do some corpus cleanup by removing noisy sentence pairs.\nThis can involve handwritten rules to remove low-precision pairs (for example re-\nmoving sentences that are too long, too short, have different URLs, or even pairs 272 CHAPTER 13 \u2022 M ACHINE TRANSLATION\nthat are too similar, suggesting that they were copies rather than translations). Or\npairs can be ranked by their multilingual embedding cosine score and low-scoring\npairs discarded.\n13.3 Details of the Encoder-Decoder Model\nEncoderThegreenlleg\u00f3witcharrived<s>lleg\u00f3la\nlabruja\nbrujaverde\nverde</s>Decodercross-attentiontransformerblocks\nFigure 13.5 The encoder-decoder transformer architecture for machine translation. The encoder uses the\ntransformer blocks we saw in Chapter 8, while the decoder uses a more powerful block with an extra cross-\nattention layer that can attend to all the encoder words. We\u2019ll see this in more detail in the next section.\nThe standard architecture for MT is the encoder-decoder transformer. The encoder-\ndecoder architecture was introduced already for RNNs in Chapter 8, and the trans-\nformer version has the same idea. Fig. 13.5 shows the intuition of the architec-\nture at a high level. You\u2019ll see that the encoder-decoder architecture is made up of\ntwo transformers: an encoder , which is the same as the basic transformers from\nChapter 9, and a decoder , which is augmented with a special new layer called the\ncross-attention layer. The encoder takes the source language input word tokens\nX=x1;:::;xnand maps them to an output representation Henc=h1;:::;hn; via a\nstack of encoder blocks.\nThe decoder is essentially a conditional language model that attends to the en-\ncoder representation and generates the target words one by one, at each timestep\nconditioning on the source sentence and the previously generated target language\nwords to generate a token. Decoding can use any of the decoding methods discussed\nin Chapter 9 like greedy, or temperature or nucleus sampling. But the most com-\nmon decoding algorithm for MT is the beam search algorithm that we\u2019ll introduce\nin Section 13.4.\nBut the components of the architecture differ somewhat from the transformer\nblock we\u2019ve seen. First, in order to attend to the source language, the transformer\nblocks in the decoder have an extra cross-attention layer. Recall that the transformer\nblock of Chapter 9 consists of a self-attention layer that attends to the input from\nthe previous layer, followed by layer norm, a feed forward layer, and another layer\nnorm. The decoder transformer block includes an extra layer with a special kind\nof attention, cross-attention (also sometimes called encoder-decoder attention or cross-attention\nsource attention ). Cross-attention has the same form as the multi-head attention\nin a normal transformer block, except that while the queries as usual come from\nthe previous layer of the decoder, the keys and values come from the output of the\nencoder . 13.3 \u2022 D ETAILS OF THE ENCODER -DECODER MODEL 273\nEncoderx1x2x3xn\u2026Decoderh3h2h1\u2026hn\nEncoderBlock 1Block 2Block Ky3y2y1\u2026\nDecoderBlock 1Block 2Block LUnembedding Matrixym\nMulti-Head AttentionLayer NormalizeLayer Normalize++\u2026Feedforward\nCausal Multi-Head AttentionLayer NormalizeLayer Normalize+\n+\u2026FeedforwardLayer Normalize+Cross-Attention\u2026\u2026\u2026\u2026\u2026\u2026LanguageModeling HeadHenc\nFigure 13.6 The transformer block for the encoder and the decoder. The \ufb01nal output of the encoder Henc=\nh1;:::;hnis the context used in the decoder. The decoder is a standard transformer except with one extra layer,\nthecross-attention layer, which takes that encoder output Hencand uses it to form its KandVinputs.\nThat is, where in standard multi-head attention the input to each attention layer is\nX, in cross attention the input is the the \ufb01nal output of the encoder Henc=h1;:::;hn.\nHencis of shape [n\u0002d], each row representing one input token. To link the keys\nand values from the encoder with the query from the prior layer of the decoder, we\nmultiple the encoder output Hencby the cross-attention layer\u2019s key weights WKand\nvalue weights WV. The query comes from the output from the prior decoder layer\nHdec[`\u00001], which is multiplied by the cross-attention layer\u2019s query weights WQ:\nQ=Hdec[`\u00001]WQ;K=HencWK;V=HencWV(13.11)\nCrossAttention (Q;K;V) = softmax\u0012QK|\npdk\u0013\nV (13.12)\nThe cross attention thus allows the decoder to attend to each of the source language\nwords as projected into the entire encoder \ufb01nal output representations. The other\nattention layer in each decoder block, the multi-head attention layer, is the same\ncausal (left-to-right) attention that we saw in Chapter 9. The multi-head attention in\nthe encoder, however, is allowed to look ahead at the entire source language text, so\nit is not masked.\nTo train an encoder-decoder model, we use the same self-supervision model we\nused for training encoder-decoders RNNs in Chapter 8. The network is given the\nsource text and then starting with the separator token is trained autoregressively to\npredict the next token using cross-entropy loss. Recall that cross-entropy loss for 274 CHAPTER 13 \u2022 M ACHINE TRANSLATION\nlanguage modeling is determined by the probability the model assigns to the correct\nnext word. So at time tthe CE loss is the negative log probability the model assigns\nto the next word in the training sequence:\nLCE(\u02c6yt;yt) =\u0000log\u02c6yt[wt+1] (13.13)\nAs in that case, we use teacher forcing in the decoder. Recall that in teacher forc- teacher forcing\ning, at each time step in decoding we force the system to use the gold target token\nfrom training as the next input xt+1, rather than allowing it to rely on the (possibly\nerroneous) decoder output \u02c6 yt.\n13.4 Decoding in MT: Beam Search\nRecall the greedy decoding algorithm from Chapter 9: at each time step tin gen-\neration, the output ytis chosen by computing the probability for each word in the\nvocabulary and then choosing the highest probability word (the argmax):\n\u02c6wt=argmaxw2VP(wjw<t) (13.14)\nA problem with greedy decoding is that what looks high probability at word tmight\nturn out to have been the wrong choice once we get to word t+1. The beam search\nalgorithm maintains multiple choices until later when we can see which one is best.\nIn beam search we model decoding as searching the space of possible genera-\ntions, represented as a search tree whose branches represent actions (generating a search tree\ntoken), and nodes represent states (having generated a particular pre\ufb01x). We search\nfor the best action sequence, i.e., the string with the highest probability.\nAn illustration of the problem\nFig. 13.7 shows a made-up example. The most probable sequence is ok ok EOS (its\nprobability is .4\u0002.7\u00021.0). But greedy search doesn\u2019t \ufb01nd it, incorrectly choosing\nyesas the \ufb01rst word since it has the highest local probability (0.5).\nstartokyesEOSokyesEOSokyesEOSEOSEOSEOSEOSt2t3p(t1|start)\nt1p(t2| t1)p(t3| t1,t2)\n.1.5.4.3.4.3.1.2.71.01.01.01.0\nFigure 13.7 A search tree for generating the target string T=t1;t2;:::from vocabulary\nV=fyes;ok;<s>g, showing the probability of generating each token from that state. Greedy\nsearch chooses yesfollowed by yes, instead of the globally most probable sequence ok ok .\nRecall from Chapter 17 that for part-of-speech tagging we used dynamic pro-\ngramming search (the Viterbi algorithm) to address this problem. Unfortunately, 13.4 \u2022 D ECODING IN MT: B EAM SEARCH 275\ndynamic programming is not applicable to generation problems with long-distance\ndependencies between the output decisions. The only method guaranteed to \ufb01nd the\nbest solution is exhaustive search: computing the probability of every one of the VT\npossible sentences (for some length value T) which is obviously too slow.\nThe solution: beam search\nInstead, MT systems generally decode using beam search , a heuristic search method beam search\n\ufb01rst proposed by Lowerre (1976). In beam search, instead of choosing the best token\nto generate at each timestep, we keep kpossible tokens at each step. This \ufb01xed-size\nmemory footprint kis called the beam width , on the metaphor of a \ufb02ashlight beam beam width\nthat can be parameterized to be wider or narrower.\nThus at the \ufb01rst step of decoding, we compute a softmax over the entire vocab-\nulary, assigning a probability to each word. We then select the k-best options from\nthis softmax output. These initial koutputs are the search frontier and these kinitial\nwords are called hypotheses . A hypothesis is an output sequence, a translation-so-\nfar, together with its probability.\na\u2026aardvark..arrived..the\u2026zebrastart\nt1a\u2026aardvark..the..witch\u2026zebraa\u2026aardvark..green..witch\u2026zebrat2hd1y1BOSy1y2\ny2hd1hd2thetheBOShd2greengreeny3hd1hd2arrivedarrivedBOSy2\nt3hd1hd2thetheBOSy2\nhd1hd2thetheBOShd2witchwitchy3a\u2026mage..the..witch\u2026zebraarrived\u2026aardvark..green..who\u2026zebray3\ny3\nFigure 13.8 Beam search decoding with a beam width of k=2. At each time step, we choose the kbest\nhypotheses, form the Vpossible extensions of each, score those k\u0002Vhypotheses and choose the best k=2\nto continue. At time 1, the frontier has the best 2 options from the initial decoder state: arrived andthe. We\nextend each, compute the probability of all the hypotheses so far ( arrived the ,arrived aardvark ,the green ,the\nwitch ) and again chose the best 2 ( the green andthe witch ) to be the search frontier. The images on the arcs\nschematically represent the decoders that must be run at each step to score the next words (for simplicity not\ndepicting cross-attention).\nAt subsequent steps, each of the kbest hypotheses is extended incrementally\nby being passed to distinct decoders, which each generate a softmax over the entire 276 CHAPTER 13 \u2022 M ACHINE TRANSLATION\nvocabulary to extend the hypothesis to every possible next token. Each of these k\u0002V\nhypotheses is scored by P(yijx;y<i): the product of the probability of the current\nword choice multiplied by the probability of the path that led to it. We then prune\nthek\u0002Vhypotheses down to the kbest hypotheses, so there are never more than k\nhypotheses at the frontier of the search, and never more than kdecoders. Fig. 13.8\nillustrates this with a beam width of 2 for the beginning of The green witch arrived .\nThis process continues until an EOS is generated indicating that a complete can-\ndidate output has been found. At this point, the completed hypothesis is removed\nfrom the frontier and the size of the beam is reduced by one. The search continues\nuntil the beam has been reduced to 0. The result will be khypotheses.\nTo score each node by its log probability, we use the chain rule of probability to\nbreak down p(yjx)into the product of the probability of each word given its prior\ncontext, which we can turn into a sum of logs (for an output string of length t):\nscore(y) = logP(yjx)\n=log(P(y1jx)P(y2jy1;x)P(y3jy1;y2;x):::P(ytjy1;:::;yt\u00001;x))\n=tX\ni=1logP(yijy1;:::;yi\u00001;x) (13.15)\nThus at each step, to compute the probability of a partial sentence, we simply add the\nlog probability of the pre\ufb01x sentence so far to the log probability of generating the\nnext token. Fig. 13.9 shows the scoring for the example sentence shown in Fig. 13.8,\nusing some simple made-up probabilities. Log probabilities are negative or 0, and\nthe max of two log probabilities is the one that is greater (closer to 0).\nBOSarrivedthethewitchgreenwitchmage\nwhoy2y3log P(y1|x)y1log P(y2|y1,x)log P(y3|y2,y1,x)-.92-1.6-1.2-.69-2.3-.69-1.6\n-2.3arrived-.11-.51witch-.36-.22EOS\n-.51EOS-2.3at-1.61bylog P(y4|y3,y2,y1,x)log P(y5|y4,y3,y2,y1,x)arrivedcame-1.6\ny4y5log P(arrived|x) log P(arrived witch|x)log P(the|x)log P(the green|x)log P(the witch|x) =-1.6log P (arrived the|x) log P (\u201cthe green witch arrived\u201d|x) = log P (the|x) + log P(green|the,x) + log P(witch | the, green,x)+logP(arrived|the,green,witch,x)+log P(EOS|the,green,witch,arrived,x)= -2.3= -3.9= -1.6= -2.1=-.92-2.1-3.2\n-4.4-2.2-2.5-3.7-2.7-3.8-2.7-4.8\nFigure 13.9 Scoring for beam search decoding with a beam width of k=2. We maintain the log probability\nof each hypothesis in the beam by incrementally adding the logprob of generating each next token. Only the top\nkpaths are extended to the next step.\nFig. 13.10 gives the algorithm. One problem with this version of the algorithm is\nthat the completed hypotheses may have different lengths. Because language mod-\nels generally assign lower probabilities to longer strings, a naive algorithm would 13.4 \u2022 D ECODING IN MT: B EAM SEARCH 277\nfunction BEAM DECODE (c,beam width )returns best paths\ny0,h0 0\npath ()\ncomplete paths ()\nstate (c,y0,h0, path) ;initial state\nfrontier hstatei ;initial frontier\nwhile frontier contains incomplete paths andbeamwidth>0\nextended frontier hi\nfor each state2frontier do\ny DECODE (state )\nfor each word i2Vocabulary do\nsuccessor NEWSTATE (state ,i,yi)\nextended frontier ADDTOBEAM (successor ,extended frontier ,\nbeam width )\nfor each state inextended frontier do\nifstate is complete do\ncomplete paths APPEND (complete paths ,state )\nextended frontier REMOVE (extended frontier ,state )\nbeam width beam width - 1\nfrontier extended frontier\nreturn completed paths\nfunction NEWSTATE (state ,word ,word prob)returns new state\nfunction ADDTOBEAM (state ,frontier ,width )returns updated frontier\nifLENGTH (frontier )<width then\nfrontier INSERT (state ,frontier )\nelse if SCORE (state )>SCORE (WORST OF(frontier ))\nfrontier REMOVE (WORST OF(frontier ))\nfrontier INSERT (state ,frontier )\nreturn frontier\nFigure 13.10 Beam search decoding.\nchoose shorter strings for y. (This is not an issue during the earlier steps of decod-\ning; since beam search is breadth-\ufb01rst, all the hypotheses being compared had the\nsame length.) For this reason we often apply length normalization methods, like\ndividing the logprob by the number of words:\nscore(y) =logP(yjx) =1\nttX\ni=1logP(yijy1;:::;yi\u00001;x) (13.16)\nFor MT we generally use beam widths kbetween 5 and 10, giving us khypotheses at\nthe end. We can pass all kto the downstream application with their respective scores,\nor if we just need a single translation we can pass the most probable hypothesis.\n13.4.1 Minimum Bayes Risk Decoding\nMinimum Bayes risk orMBR decoding is an alternative decoding algorithm thatminimum\nBayes risk\nMBR 278 CHAPTER 13 \u2022 M ACHINE TRANSLATION\ncan work even better than beam search and also tends to be better than the other\ndecoding algorithms like temperature sampling introduced in Section 10.2.\nThe intuition of minimum Bayes risk is that instead of trying to choose the trans-\nlation which is most probable, we choose the one that is likely have the least error.\nFor example, we might want our decoding algorithm to \ufb01nd the translation which\nhas the highest score on some evaluation metric. For example in Section 13.6 we will\nintroduce metrics like chrF or BERTScore that measure the goodness-of-\ufb01t between\na candidate translation and a set of reference human translations. A translation that\nmaximizes this score, especially with a hypothetically huge set of perfect human\ntranslations is likely to be a good one (have minimum risk) even if it is not the most\nprobable translation by our particular probability estimator.\nIn practice, we don\u2019t know the perfect set of translations for a given sentence. So\nthe standard simpli\ufb01cation used in MBR decoding algorithms is to instead choose\nthe candidate translation which is most similar (by some measure of goodness-of-\n\ufb01t) with some set of candidate translations. We\u2019re essentially approximating the\nenormous space of all possible translations Uwith a smaller set of possible candidate\ntranslations Y.\nGiven this set of possible candidate translations Y, and some similarity or align-\nment function util, we choose the best translation \u02c6 yas the translation which is most\nsimilar to all the other candidate translations:\n\u02c6y=argmax\ny2YX\nc2Yutil(y;c) (13.17)\nVarious util functions can be used, like chrF or BERTscore or BLEU. We can get the\nset of candidate translations by sampling using one of the basic sampling algorithms\nof Section 10.2 like temperature sampling; good results can be obtained with as few\nas 32 or 64 candidates.\nMinimum Bayes risk decoding can also be used for other NLP tasks; indeed\nit was widely applied to speech recognition (Stolcke et al., 1997; Goel and Byrne,\n2000) before being applied to machine translation (Kumar and Byrne, 2004), and\nhas been shown to work well across many other generation tasks as well (e.g., sum-\nmarization, dialogue, and image captioning (Suzgun et al., 2023a)).\n13.5 Translating in low-resource situations\nFor some languages, and especially for English, online resources are widely avail-\nable. There are many large parallel corpora that contain translations between En-\nglish and many languages. But the vast majority of the world\u2019s languages do not\nhave large parallel training texts available. An important ongoing research question\nis how to get good translation with lesser resourced languages. The resource prob-\nlem can even be true for high resource languages when we need to translate into low\nresource domains (for example in a particular genre that happens to have very little\nbitext).\nHere we brie\ufb02y introduce two commonly used approaches for dealing with this\ndata sparsity: backtranslation , which is a special case of the general statistical\ntechnique called data augmentation , and multilingual models , and also discuss\nsome socio-technical issues. 13.5 \u2022 T RANSLATING IN LOW -RESOURCE SITUATIONS 279\n13.5.1 Data Augmentation\nData augmentation is a statistical technique for dealing with insuf\ufb01cient training\ndata, by adding new synthetic data that is generated from the current natural data.\nThe most common data augmentation technique for machine translation is called\nbacktranslation . Backtranslation relies on the intuition that while parallel corpora backtranslation\nmay be limited for particular languages or domains, we can often \ufb01nd a large (or\nat least larger) monolingual corpus, to add to the smaller parallel corpora that are\navailable. The algorithm makes use of monolingual corpora in the target language\nby creating synthetic bitexts.\nIn backtranslation, our goal is to improve source-to-target MT, given a small\nparallel text (a bitext) in the source/target languages, and some monolingual data in\nthe target language. We \ufb01rst use the bitext to train a MT system in the reverse di-\nrection: a target-to-source MT system . We then use it to translate the monolingual\ntarget data to the source language. Now we can add this synthetic bitext (natural\ntarget sentences, aligned with MT-produced source sentences) to our training data,\nand retrain our source-to-target MT model. For example suppose we want to trans-\nlate from Navajo to English but only have a small Navajo-English bitext, although of\ncourse we can \ufb01nd lots of monolingual English data. We use the small bitext to build\nan MT engine going the other way (from English to Navajo). Once we translate the\nmonolingual English text to Navajo, we can add this synthetic Navajo/English bitext\nto our training data.\nBacktranslation has various parameters. One is how we generate the backtrans-\nlated data; we can run the decoder in greedy inference, or use beam search. Or\nwe can do sampling, like the temperature sampling algorithm we saw in Chapter 9.\nAnother parameter is the ratio of backtranslated data to natural bitext data; we can\nchoose to upsample the bitext data (include multiple copies of each sentence). In\ngeneral backtranslation works surprisingly well; one estimate suggests that a system\ntrained on backtranslated text gets about 2/3 of the gain as would training on the\nsame amount of natural bitext (Edunov et al., 2018).\n13.5.2 Multilingual models\nThe models we\u2019ve described so far are for bilingual translation: one source language,\none target language. It\u2019s also possible to build a multilingual translator.\nIn a multilingual translator, we train the system by giving it parallel sentences\nin many different pairs of languages. That means we need to tell the system which\nlanguage to translate from and to! We tell the system which language is which\nby adding a special token lsto the encoder specifying the source language we\u2019re\ntranslating from, and a special token ltto the decoder telling it the target language\nwe\u2019d like to translate into.\nThus we slightly update Eq. 13.9 above to add these tokens in Eq. 13.19:\nh=encoder (x;ls) (13.18)\nyi+1=decoder (h;lt;y1;:::; yi))8i2[1;:::; m] (13.19)\nOne advantage of a multilingual model is that they can improve the translation\nof lower-resourced languages by drawing on information from a similar language\nin the training data that happens to have more resources. Perhaps we don\u2019t know\nthe meaning of a word in Galician, but the word appears in the similar and higher-\nresourced language Spanish. 280 CHAPTER 13 \u2022 M ACHINE TRANSLATION\n13.5.3 Sociotechnical issues\nMany issues in dealing with low-resource languages go beyond the purely techni-\ncal. One problem is that for low-resource languages, especially from low-income\ncountries, native speakers are often not involved as the curators for content selec-\ntion, as the language technologists, or as the evaluators who measure performance\n(8et al., 2020). Indeed, one well-known study that manually audited a large set of\nparallel corpora and other major multilingual datasets found that for many of the\ncorpora, less than 50% of the sentences were of acceptable quality, with a lot of\ndata consisting of repeated sentences with web boilerplate or incorrect translations,\nsuggesting that native speakers may not have been suf\ufb01ciently involved in the data\nprocess (Kreutzer et al., 2022).\nOther issues, like the tendency of many MT approaches to focus on the case\nwhere one of the languages is English (Anastasopoulos and Neubig, 2020), have to\ndo with allocation of resources. Where most large multilingual systems were trained\non bitexts in which English was one of the two languages, recent huge corporate\nsystems like those of Fan et al. (2021) and Costa-juss `a et al. (2022) and datasets\nlike Schwenk et al. (2021) attempt to handle large numbers of languages (up to 200\nlanguages) and create bitexts between many more pairs of languages and not just\nthrough English.\nAt the smaller end, 8et al. (2020) propose a participatory design process to\nencourage content creators, curators, and language technologists who speak these\nlow-resourced languages to participate in developing MT algorithms. They provide\nonline groups, mentoring, and infrastructure, and report on a case study on devel-\noping MT algorithms for low-resource African languages. Among their conclusions\nwas to perform MT evaluation by post-editing rather than direct evaluation, since\nhaving labelers edit an MT system and then measure the distance between the MT\noutput and its post-edited version both was simpler to train evaluators and makes it\neasier to measure true errors in the MT output and not differences due to linguistic\nvariation (Bentivogli et al., 2018).\n13.6 MT Evaluation\nTranslations are evaluated along two dimensions:\n1.adequacy: how well the translation captures the exact meaning of the source adequacy\nsentence. Sometimes called faithfulness or\ufb01delity .\n2.\ufb02uency: how \ufb02uent the translation is in the target language (is it grammatical, \ufb02uency\nclear, readable, natural).\nUsing humans to evaluate is most accurate, but automatic metrics are also used for\nconvenience.\n13.6.1 Using Human Raters to Evaluate MT\nThe most accurate evaluations use human raters, such as online crowdworkers, to\nevaluate each translation along the two dimensions. For example, along the dimen-\nsion of \ufb02uency , we can ask how intelligible, how clear, how readable, or how natural\nthe MT output (the target text) is. We can give the raters a scale, for example, from\n1 (totally unintelligible) to 5 (totally intelligible), or 1 to 100, and ask them to rate\neach sentence or paragraph of the MT output. 13.6 \u2022 MT E VALUATION 281\nWe can do the same thing to judge the second dimension, adequacy , using raters\nto assign scores on a scale. If we have bilingual raters, we can give them the source\nsentence and a proposed target sentence, and rate, on a 5-point or 100-point scale,\nhow much of the information in the source was preserved in the target. If we only\nhave monolingual raters but we have a good human translation of the source text, we\ncan give the monolingual raters the human reference translation and a target machine\ntranslation and again rate how much information is preserved. An alternative is to\ndoranking : give the raters a pair of candidate translations, and ask them which one ranking\nthey prefer.\nTraining of human raters (who are often online crowdworkers) is essential; raters\nwithout translation expertise \ufb01nd it dif\ufb01cult to separate \ufb02uency and adequacy, and\nso training includes examples carefully distinguishing these. Raters often disagree\n(source sentences may be ambiguous, raters will have different world knowledge,\nraters may apply scales differently). It is therefore common to remove outlier raters,\nand (if we use a \ufb01ne-grained enough scale) normalizing raters by subtracting the\nmean from their scores and dividing by the variance.\nAs discussed above, an alternative way of using human raters is to have them\npost-edit translations, taking the MT output and changing it minimally until they\nfeel it represents a correct translation. The difference between their post-edited\ntranslations and the original MT output can then be used as a measure of quality.\n13.6.2 Automatic Evaluation\nWhile humans produce the best evaluations of machine translation output, running a\nhuman evaluation can be time consuming and expensive. For this reason automatic\nmetrics are often used as temporary proxies. Automatic metrics are less accurate\nthan human evaluation, but can help test potential system improvements, and even\nbe used as an automatic loss function for training. In this section we introduce two\nfamilies of such metrics, those based on character- or word-overlap and those based\non embedding similarity.\nAutomatic Evaluation by Character Overlap: chrF\nThe simplest and most robust metric for MT evaluation is called chrF , which stands chrF\nforcharacter F-score (Popovi \u00b4c, 2015). chrF (along with many other earlier related\nmetrics like BLEU, METEOR, TER, and others) is based on a simple intuition de-\nrived from the pioneering work of Miller and Beebe-Center (1956): a good machine\ntranslation will tend to contain characters and words that occur in a human trans-\nlation of the same sentence. Consider a test set from a parallel corpus, in which\neach source sentence has both a gold human target translation and a candidate MT\ntranslation we\u2019d like to evaluate. The chrF metric ranks each MT target sentence by\na function of the number of character n-gram overlaps with the human translation.\nGiven the hypothesis and the reference, chrF is given a parameter kindicating\nthe length of character n-grams to be considered, and computes the average of the\nkprecisions (unigram precision, bigram, and so on) and the average of the krecalls\n(unigram recall, bigram recall, etc.):\nchrP percentage of character 1-grams, 2-grams, ..., k-grams in the hypothesis that\noccur in the reference, averaged.\nchrR percentage of character 1-grams, 2-grams,..., k-grams in the reference that\noccur in the hypothesis, averaged.\nThe metric then computes an F-score by combining chrP and chrR using a weighting 282 CHAPTER 13 \u2022 M ACHINE TRANSLATION\nparameter b. It is common to set b=2, thus weighing recall twice as much as\nprecision:\nchrFb= (1+b2)chrP\u0001chrR\nb2\u0001chrP+chrR(13.20)\nForb=2, that would be:\nchrF2 =5\u0001chrP\u0001chrR\n4\u0001chrP+chrR\nFor example, consider two hypotheses that we\u2019d like to score against the refer-\nence translation witness for the past . Here are the hypotheses along with chrF values\ncomputed using parameters k=b=2 (in real examples, kwould be a higher number\nlike 6):\nREF:witness for the past,\nHYP1:witness of the past, chrF2,2 = .86\nHYP2:past witness chrF2,2 = .62\nLet\u2019s see how we computed that chrF value for HYP1 (we\u2019ll leave the compu-\ntation of the chrF value for HYP2 as an exercise for the reader). First, chrF ignores\nspaces, so we\u2019ll remove them from both the reference and hypothesis:\nREF:witnessforthepast, (18 unigrams, 17 bigrams)\nHYP1:witnessofthepast, (17 unigrams, 16 bigrams)\nNext let\u2019s see how many unigrams and bigrams match between the reference and\nhypothesis:\nunigrams that match: w i t n e s s f o t h e p a s t , (17 unigrams)\nbigrams that match: wi it tn ne es ss th he ep pa as st t, (13 bigrams)\nWe use that to compute the unigram and bigram precisions and recalls:\nunigram P: 17/17 = 1 unigram R: 17/18 = .944\nbigram P: 13/16 = .813 bigram R: 13/17 = .765\nFinally we average to get chrP and chrR, and compute the F-score:\nchrP = (17=17+13=16)=2=:906\nchrR = (17=18+13=17)=2=:855\nchrF2,2 =5chrP\u0003chrR\n4chrP +chrR=:86\nchrF is simple, robust, and correlates very well with human judgments in many\nlanguages (Kocmi et al., 2021).\nAlternative overlap metric: BLEU\nThere are various alternative overlap metrics. For example, before the development\nof chrF, it was common to use a word-based overlap metric called BLEU (for BiLin-\ngual Evaluation Understudy), that is purely precision-based rather than combining\nprecision and recall (Papineni et al., 2002). The BLEU score for a corpus of candi-\ndate translation sentences is a function of the n-gram word precision over all the\nsentences combined with a brevity penalty computed over the corpus as a whole.\nWhat do we mean by n-gram precision? Consider a corpus composed of a single\nsentence. The unigram precision for this corpus is the percentage of unigram tokens 13.6 \u2022 MT E VALUATION 283\nin the candidate translation that also occur in the reference translation, and ditto for\nbigrams and so on, up to 4-grams. BLEU extends this unigram metric to the whole\ncorpus by computing the numerator as the sum over all sentences of the counts of all\nthe unigram types that also occur in the reference translation, and the denominator\nis the total of the counts of all unigrams in all candidate sentences. We compute\nthis n-gram precision for unigrams, bigrams, trigrams, and 4-grams and take the\ngeometric mean. BLEU has many further complications, including a brevity penalty\nfor penalizing candidate translations that are too short, and it also requires the n-\ngram counts be clipped in a particular way.\nBecause BLEU is a word-based metric, it is very sensitive to word tokenization,\nmaking it impossible to compare different systems if they rely on different tokeniza-\ntion standards, and doesn\u2019t work as well in languages with complex morphology.\nNonetheless, you will sometimes still see systems evaluated by BLEU, particularly\nfor translation into English. In such cases it\u2019s important to use packages that enforce\nstandardization for tokenization like S ACRE BLEU (Post, 2018).\nStatistical Signi\ufb01cance Testing for MT evals\nCharacter or word overlap-based metrics like chrF (or BLEU, or etc.) are mainly\nused to compare two systems, with the goal of answering questions like: did the\nnew algorithm we just invented improve our MT system? To know if the difference\nbetween the chrF scores of two MT systems is a signi\ufb01cant difference, we use the\npaired bootstrap test, or the similar randomization test.\nTo get a con\ufb01dence interval on a single chrF score using the bootstrap test, recall\nfrom Section 4.9 that we take our test set (or devset) and create thousands of pseudo-\ntestsets by repeatedly sampling with replacement from the original test set. We now\ncompute the chrF score of each of the pseudo-testsets. If we drop the top 2.5% and\nbottom 2.5% of the scores, the remaining scores will give us the 95% con\ufb01dence\ninterval for the chrF score of our system.\nTo compare two MT systems A and B, we draw the same set of pseudo-testsets,\nand compute the chrF scores for each of them. We then compute the percentage of\npseudo-test-sets in which A has a higher chrF score than B.\nchrF: Limitations\nWhile automatic character and word-overlap metrics like chrF or BLEU are useful,\nthey have important limitations. chrF is very local: a large phrase that is moved\naround might barely change the chrF score at all, and chrF can\u2019t evaluate cross-\nsentence properties of a document like its discourse coherence (Chapter 24). chrF\nand similar automatic metrics also do poorly at comparing very different kinds of\nsystems, such as comparing human-aided translation against machine translation, or\ndifferent machine translation architectures against each other (Callison-Burch et al.,\n2006). Instead, automatic overlap metrics like chrF are most appropriate when eval-\nuating changes to a single system.\n13.6.3 Automatic Evaluation: Embedding-Based Methods\nThe chrF metric is based on measuring the exact character n-grams a human refer-\nence and candidate machine translation have in common. However, this criterion\nis overly strict, since a good translation may use alternate words or paraphrases. A\nsolution \ufb01rst pioneered in early metrics like METEOR (Banerjee and Lavie, 2005)\nwas to allow synonyms to match between the reference xand candidate \u02dc x. More 284 CHAPTER 13 \u2022 M ACHINE TRANSLATION\nrecent metrics use BERT or other embeddings to implement this intuition.\nFor example, in some situations we might have datasets that have human as-\nsessments of translation quality. Such datasets consists of tuples (x;\u02dcx;r), where\nx= (x1;:::; xn)is a reference translation, \u02dc x= (\u02dcx1;:::; \u02dcxm)is a candidate machine\ntranslation, and r2Ris a human rating that expresses the quality of \u02dc xwith respect\ntox. Given such data, algorithms like COMET (Rei et al., 2020) BLEURT (Sellam\net al., 2020) train a predictor on the human-labeled datasets, for example by passing\nxand \u02dcxthrough a version of BERT (trained with extra pretraining, and then \ufb01netuned\non the human-labeled sentences), followed by a linear layer that is trained to predict\nr. The output of such models correlates highly with human labels.\nIn other cases, however, we don\u2019t have such human-labeled datasets. In that\ncase we can measure the similarity of xand \u02dcxby the similarity of their embeddings.\nThe BERTS CORE algorithm (Zhang et al., 2020) shown in Fig. 13.11, for example,\npasses the reference xand the candidate \u02dc xthrough BERT, computing a BERT em-\nbedding for each token xiand \u02dcxj. Each pair of tokens (xi;\u02dcxj)is scored by its cosine\nxi\u0001\u02dcxj\njxijj\u02dcxjj. Each token in xis matched to a token in \u02dc xto compute recall, and each token in\n\u02dcxis matched to a token in xto compute precision (with each token greedily matched\nto the most similar token in the corresponding sentence). BERTS CORE provides\nprecision and recall (and hence F 1):\nRBERT=1\njxjX\nxi2xmax\n\u02dcxj2\u02dcxxi\u0001\u02dcxjPBERT=1\nj\u02dcxjX\n\u02dcxj2\u02dcxmax\nxi2xxi\u0001\u02dcxj (13.21)\nPublished as a conference paper at ICLR 2020\nReferencethe weather is cold today\nCandidateit is freezing today\nCandidateContextualEmbeddingPairwise CosineSimilarityRBERT=(0.713\u00001.27)+(0.515\u00007.94)+...1.27+7.94+1.82+7.90+8.88\n<latexit sha1_base64=\"OJyoKlmBAgUA0KDtUcsH/di5BlI=\">AAACSHicbZDLattAFIaPnLRJ3JvTLrsZYgoJAqFxGqwsCqal0FVJQ5wELCNG41EyZHRh5ijECL1EnqAv002X2eUZsumipXRR6Mj2Ipf+MPDznXM4Z/64UNKg7187raXlR49XVtfaT54+e/6is/7y0OSl5mLIc5Xr45gZoWQmhihRieNCC5bGShzFZx+a+tG50Ebm2QFOCzFO2UkmE8kZWhR1ov2oClFcYPX+4/5BXZN3JEw049Wm7/XpdogyFYZQr9ffci3aoTsL1Pd23265oZrkaOqqaXAb5FIv6DXOdwMvCOqo0/U9fyby0NCF6Q52/15+BYC9qHMVTnJepiJDrpgxI+oXOK6YRsmVqNthaUTB+Bk7ESNrM2aPGVezIGryxpIJSXJtX4ZkRm9PVCw1ZprGtjNleGru1xr4v9qoxCQYVzIrShQZny9KSkUwJ02qZCK14Kim1jCupb2V8FNmc0SbfduGQO9/+aE57HnU9+gX2h18hrlW4TVswCZQ6MMAPsEeDIHDN7iBn/DL+e78cH47f+atLWcx8wruqNX6B8dUrVw=</latexit><latexit sha1_base64=\"RInTcZkWiVBnf/ncBstCvatCtG4=\">AAACSHicbZDPShxBEMZ7Nproxugaj14al4AyMEyvyoyHwGIQPImKq8LOMvT09mhjzx+6a0KWYV4iL5EnySXH3HwGLx4U8SDYs7sHo/mg4eNXVVT1F+VSaHDda6vxbmb2/Ye5+ebHhU+LS63lz6c6KxTjPZbJTJ1HVHMpUt4DAZKf54rTJJL8LLr6VtfPvnOlRZaewCjng4RepCIWjIJBYSs8DssA+A8od/eOT6oKf8VBrCgr113HI5sBiIRrTJyOt2EbtE22p8hzdrY27EAOM9BVWTfYNbKJ43dq59q+4/tV2Gq7jjsWfmvI1LS7O08/f3nLi4dh628wzFiR8BSYpFr3iZvDoKQKBJO8agaF5jllV/SC941NqTlmUI6DqPAXQ4Y4zpR5KeAxfTlR0kTrURKZzoTCpX5dq+H/av0CYn9QijQvgKdssiguJIYM16nioVCcgRwZQ5kS5lbMLqnJEUz2TRMCef3lt+a04xDXIUek3T1AE82hVbSG1hFBHuqifXSIeoih3+gG3aF76491az1Yj5PWhjWdWUH/qNF4BkPYrbk=</latexit><latexit sha1_base64=\"RInTcZkWiVBnf/ncBstCvatCtG4=\">AAACSHicbZDPShxBEMZ7Nproxugaj14al4AyMEyvyoyHwGIQPImKq8LOMvT09mhjzx+6a0KWYV4iL5EnySXH3HwGLx4U8SDYs7sHo/mg4eNXVVT1F+VSaHDda6vxbmb2/Ye5+ebHhU+LS63lz6c6KxTjPZbJTJ1HVHMpUt4DAZKf54rTJJL8LLr6VtfPvnOlRZaewCjng4RepCIWjIJBYSs8DssA+A8od/eOT6oKf8VBrCgr113HI5sBiIRrTJyOt2EbtE22p8hzdrY27EAOM9BVWTfYNbKJ43dq59q+4/tV2Gq7jjsWfmvI1LS7O08/f3nLi4dh628wzFiR8BSYpFr3iZvDoKQKBJO8agaF5jllV/SC941NqTlmUI6DqPAXQ4Y4zpR5KeAxfTlR0kTrURKZzoTCpX5dq+H/av0CYn9QijQvgKdssiguJIYM16nioVCcgRwZQ5kS5lbMLqnJEUz2TRMCef3lt+a04xDXIUek3T1AE82hVbSG1hFBHuqifXSIeoih3+gG3aF76491az1Yj5PWhjWdWUH/qNF4BkPYrbk=</latexit><latexit sha1_base64=\"fGWl4NCvlvtMu17rjLtk25oWpdc=\">AAACSHicbZBLS+RAFIUrPT7bVzsu3RQ2ghIIqVbpuBgQRZiVqNgqdJpQqa5oYeVB1Y1ME/Lz3Lic3fwGNy6UwZ2VNgtfBwoO372Xe+uEmRQaXPef1fgxMTk1PTPbnJtfWFxqLf8812muGO+xVKbqMqSaS5HwHgiQ/DJTnMah5BfhzUFVv7jlSos0OYNRxgcxvUpEJBgFg4JWcBoUPvA/UOwfnp6VJf6F/UhRVmy4Tpds+SBirjFxOt1N26AdslOjrrO7vWn7cpiCLouqwa6QTRyvUznX9hzPK4NW23XcsfBXQ2rTRrWOg9Zff5iyPOYJMEm17hM3g0FBFQgmedn0c80zym7oFe8bm1BzzKAYB1HidUOGOEqVeQngMX0/UdBY61Ecms6YwrX+XKvgd7V+DpE3KESS5cAT9rYoyiWGFFep4qFQnIEcGUOZEuZWzK6pyRFM9k0TAvn85a/mvOMQ1yEnpL13VMcxg1bRGtpABHXRHvqNjlEPMXSHHtATerburUfrv/Xy1tqw6pkV9EGNxisxMKq0</latexit>1.27\n7.941.827.908.88idf\nweightsImportance Weighting(Optional)Maximum Similarityx<latexit sha1_base64=\"f2yzimwbR/Dgjzp6tZ360fHRqNI=\">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cW7Ae0oWy2k3btZhN2N2IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3H1FpHst7M0nQj+hQ8pAzaqzUeOqXK27VnYOsEi8nFchR75e/eoOYpRFKwwTVuuu5ifEzqgxnAqelXqoxoWxMh9i1VNIItZ/ND52SM6sMSBgrW9KQufp7IqOR1pMosJ0RNSO97M3E/7xuasJrP+MySQ1KtlgUpoKYmMy+JgOukBkxsYQyxe2thI2ooszYbEo2BG/55VXSuqh6btVrXFZqN3kcRTiBUzgHD66gBndQhyYwQHiGV3hzHpwX5935WLQWnHzmGP7A+fwB5jmM/A==</latexit><latexit sha1_base64=\"f2yzimwbR/Dgjzp6tZ360fHRqNI=\">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cW7Ae0oWy2k3btZhN2N2IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3H1FpHst7M0nQj+hQ8pAzaqzUeOqXK27VnYOsEi8nFchR75e/eoOYpRFKwwTVuuu5ifEzqgxnAqelXqoxoWxMh9i1VNIItZ/ND52SM6sMSBgrW9KQufp7IqOR1pMosJ0RNSO97M3E/7xuasJrP+MySQ1KtlgUpoKYmMy+JgOukBkxsYQyxe2thI2ooszYbEo2BG/55VXSuqh6btVrXFZqN3kcRTiBUzgHD66gBndQhyYwQHiGV3hzHpwX5935WLQWnHzmGP7A+fwB5jmM/A==</latexit><latexit sha1_base64=\"f2yzimwbR/Dgjzp6tZ360fHRqNI=\">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cW7Ae0oWy2k3btZhN2N2IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3H1FpHst7M0nQj+hQ8pAzaqzUeOqXK27VnYOsEi8nFchR75e/eoOYpRFKwwTVuuu5ifEzqgxnAqelXqoxoWxMh9i1VNIItZ/ND52SM6sMSBgrW9KQufp7IqOR1pMosJ0RNSO97M3E/7xuasJrP+MySQ1KtlgUpoKYmMy+JgOukBkxsYQyxe2thI2ooszYbEo2BG/55VXSuqh6btVrXFZqN3kcRTiBUzgHD66gBndQhyYwQHiGV3hzHpwX5935WLQWnHzmGP7A+fwB5jmM/A==</latexit><latexit sha1_base64=\"f2yzimwbR/Dgjzp6tZ360fHRqNI=\">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cW7Ae0oWy2k3btZhN2N2IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3H1FpHst7M0nQj+hQ8pAzaqzUeOqXK27VnYOsEi8nFchR75e/eoOYpRFKwwTVuuu5ifEzqgxnAqelXqoxoWxMh9i1VNIItZ/ND52SM6sMSBgrW9KQufp7IqOR1pMosJ0RNSO97M3E/7xuasJrP+MySQ1KtlgUpoKYmMy+JgOukBkxsYQyxe2thI2ooszYbEo2BG/55VXSuqh6btVrXFZqN3kcRTiBUzgHD66gBndQhyYwQHiGV3hzHpwX5935WLQWnHzmGP7A+fwB5jmM/A==</latexit>\u02c6x<latexit sha1_base64=\"5QTnVRVSrnyzznVU7d5bF5u03Iw=\">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDbbTbt0swm7E7GE/ggvHhTx6u/x5r9x0+agrQ8GHu/NMDMvSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHJtRKwecJpwP6IjJULBKFqp0x9TzJ5mg2rNrbtzkFXiFaQGBZqD6ld/GLM04gqZpMb0PDdBP6MaBZN8VumnhieUTeiI9yxVNOLGz+bnzsiZVYYkjLUthWSu/p7IaGTMNApsZ0RxbJa9XPzP66UYXvuZUEmKXLHFojCVBGOS/06GQnOGcmoJZVrYWwkbU00Z2oQqNgRv+eVV0r6oe27du7+sNW6KOMpwAqdwDh5cQQPuoAktYDCBZ3iFNydxXpx352PRWnKKmWP4A+fzB7A8j8k=</latexit><latexit sha1_base64=\"5QTnVRVSrnyzznVU7d5bF5u03Iw=\">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDbbTbt0swm7E7GE/ggvHhTx6u/x5r9x0+agrQ8GHu/NMDMvSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHJtRKwecJpwP6IjJULBKFqp0x9TzJ5mg2rNrbtzkFXiFaQGBZqD6ld/GLM04gqZpMb0PDdBP6MaBZN8VumnhieUTeiI9yxVNOLGz+bnzsiZVYYkjLUthWSu/p7IaGTMNApsZ0RxbJa9XPzP66UYXvuZUEmKXLHFojCVBGOS/06GQnOGcmoJZVrYWwkbU00Z2oQqNgRv+eVV0r6oe27du7+sNW6KOMpwAqdwDh5cQQPuoAktYDCBZ3iFNydxXpx352PRWnKKmWP4A+fzB7A8j8k=</latexit><latexit sha1_base64=\"5QTnVRVSrnyzznVU7d5bF5u03Iw=\">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDbbTbt0swm7E7GE/ggvHhTx6u/x5r9x0+agrQ8GHu/NMDMvSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHJtRKwecJpwP6IjJULBKFqp0x9TzJ5mg2rNrbtzkFXiFaQGBZqD6ld/GLM04gqZpMb0PDdBP6MaBZN8VumnhieUTeiI9yxVNOLGz+bnzsiZVYYkjLUthWSu/p7IaGTMNApsZ0RxbJa9XPzP66UYXvuZUEmKXLHFojCVBGOS/06GQnOGcmoJZVrYWwkbU00Z2oQqNgRv+eVV0r6oe27du7+sNW6KOMpwAqdwDh5cQQPuoAktYDCBZ3iFNydxXpx352PRWnKKmWP4A+fzB7A8j8k=</latexit><latexit sha1_base64=\"5QTnVRVSrnyzznVU7d5bF5u03Iw=\">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDbbTbt0swm7E7GE/ggvHhTx6u/x5r9x0+agrQ8GHu/NMDMvSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHJtRKwecJpwP6IjJULBKFqp0x9TzJ5mg2rNrbtzkFXiFaQGBZqD6ld/GLM04gqZpMb0PDdBP6MaBZN8VumnhieUTeiI9yxVNOLGz+bnzsiZVYYkjLUthWSu/p7IaGTMNApsZ0RxbJa9XPzP66UYXvuZUEmKXLHFojCVBGOS/06GQnOGcmoJZVrYWwkbU00Z2oQqNgRv+eVV0r6oe27du7+sNW6KOMpwAqdwDh5cQQPuoAktYDCBZ3iFNydxXpx352PRWnKKmWP4A+fzB7A8j8k=</latexit>\nReferenceFigure 1: Illustration of the computation of the recall metric RBERT. Given the reference xand\ncandidate \u02c6x, we compute BERT embeddings and pairwise cosine similarity. We highlight the greedy\nmatching in red, and include the optional idfimportance weighting.\nWe experiment with different models (Section 4), using the tokenizer provided with each model.\nGiven a tokenized reference sentence x=hx1,...,x ki, the embedding model generates a se-\nquence of vectors hx1,...,xki. Similarly, the tokenized candidate \u02c6x=h\u02c6x1,..., \u02c6xmiis mapped\ntoh\u02c6x1,...,\u02c6xli. The main model we use is BERT, which tokenizes the input text into a sequence\nof word pieces (Wu et al., 2016), where unknown words are split into several commonly observed\nsequences of characters. The representation for each word piece is computed with a Transformer\nencoder (Vaswani et al., 2017) by repeatedly applying self-attention and nonlinear transformations\nin an alternating fashion. BERT embeddings have been shown to bene\ufb01t various NLP tasks (Devlin\net al., 2019; Liu, 2019; Huang et al., 2019; Yang et al., 2019a).\nSimilarity Measure The vector representation allows for a soft measure of similarity instead of\nexact-string (Papineni et al., 2002) or heuristic (Banerjee & Lavie, 2005) matching. The cosine\nsimilarity of a reference token xiand a candidate token \u02c6xjisx>\ni\u02c6xj\nkxikk\u02c6xjk. We use pre-normalized\nvectors, which reduces this calculation to the inner product x>\ni\u02c6xj. While this measure considers\ntokens in isolation, the contextual embeddings contain information from the rest of the sentence.\nBERTS CORE The complete score matches each token in xto a token in \u02c6xto compute recall,\nand each token in \u02c6xto a token in xto compute precision. We use greedy matching to maximize\nthe matching similarity score,2where each token is matched to the most similar token in the other\nsentence. We combine precision and recall to compute an F1 measure. For a reference xand\ncandidate \u02c6x, the recall, precision, and F1 scores are:\nRBERT =1\n|x|X\nxi2xmax\n\u02c6xj2\u02c6xx>\ni\u02c6xj,P BERT =1\n|\u02c6x|X\n\u02c6xj2\u02c6xmax\nxi2xx>\ni\u02c6xj,F BERT =2PBERT \u00b7RBERT\nPBERT +RBERT.\nImportance Weighting Previous work on similarity measures demonstrated that rare words can\nbe more indicative for sentence similarity than common words (Banerjee & Lavie, 2005; Vedantam\net al., 2015). BERTS CORE enables us to easily incorporate importance weighting. We experiment\nwith inverse document frequency ( idf) scores computed from the test corpus. Given Mreference\nsentences {x(i)}M\ni=1, the idfscore of a word-piece token wis\nidf(w)=\u0000log1\nMMX\ni=1I[w2x(i)],\nwhere I[\u00b7]is an indicator function. We do not use the full tf-idfmeasure because we process single\nsentences, where the term frequency ( tf) is likely 1. For example, recall with idfweighting is\nRBERT =P\nxi2xidf(xi) max \u02c6xj2\u02c6xx>\ni\u02c6xjP\nxi2xidf(xi).\nBecause we use reference sentences to compute idf, the idfscores remain the same for all systems\nevaluated on a speci\ufb01c test set. We apply plus-one smoothing to handle unknown word pieces.\n2We compare greedy matching with optimal assignment in Appendix C.\n4\nFigure 13.11 The computation of BERTS CORE recall from reference xand candidate \u02c6 x, from Figure 1 in\nZhang et al. (2020). This version shows an extended version of the metric in which tokens are also weighted by\ntheir idf values.\n13.7 Bias and Ethical Issues\nMachine translation raises many of the same ethical issues that we\u2019ve discussed in\nearlier chapters. For example, consider MT systems translating from Hungarian\n(which has the gender neutral pronoun \u02ddo) or Spanish (which often drops pronouns)\ninto English (in which pronouns are obligatory, and they have grammatical gender).\nWhen translating a reference to a person described without speci\ufb01ed gender, MT\nsystems often default to male gender (Schiebinger 2014, Prates et al. 2019). And\nMT systems often assign gender according to culture stereotypes of the sort we saw\nin Section 6.11. Fig. 13.12 shows examples from Prates et al. (2019), in which Hun-\ngarian gender-neutral \u02ddo is a nurse is translated with she, but gender-neutral \u02ddo is a\nCEO is translated with he. Prates et al. (2019) \ufb01nd that these stereotypes can\u2019t com-\npletely be accounted for by gender bias in US labor statistics, because the biases are 13.8 \u2022 S UMMARY 285\nampli\ufb01ed by MT systems, with pronouns being mapped to male or female gender\nwith a probability higher than if the mapping was based on actual labor employment\nstatistics.\nHungarian (gender neutral) source English MT output\n\u02ddo egy \u00b4apol\u00b4o she is a nurse\n\u02ddo egy tud \u00b4os he is a scientist\n\u02ddo egy m \u00b4ern\u00a8ok he is an engineer\n\u02ddo egy p \u00b4ek he is a baker\n\u02ddo egy tan \u00b4ar she is a teacher\n\u02ddo egy esk \u00a8uv\u02ddoszervez \u02ddo she is a wedding organizer\n\u02ddo egy vez \u00b4erigazgat \u00b4o he is a CEO\nFigure 13.12 When translating from gender-neutral languages like Hungarian into English,\ncurrent MT systems interpret people from traditionally male-dominated occupations as male,\nand traditionally female-dominated occupations as female (Prates et al., 2019).\nSimilarly, a recent challenge set, the WinoMT dataset (Stanovsky et al., 2019)\nshows that MT systems perform worse when they are asked to translate sentences\nthat describe people with non-stereotypical gender roles, like \u201cThe doctor asked the\nnurse to help her in the operation\u201d.\nMany ethical questions in MT require further research. One open problem is\ndeveloping metrics for knowing what our systems don\u2019t know. This is because MT\nsystems can be used in urgent situations where human translators may be unavailable\nor delayed: in medical domains, to help translate when patients and doctors don\u2019t\nspeak the same language, or in legal domains, to help judges or lawyers communi-\ncate with witnesses or defendants. In order to \u2018do no harm\u2019, systems need ways to\nassign con\ufb01dence values to candidate translations, so they can abstain from giving con\ufb01dence\nincorrect translations that may cause harm.\n13.8 Summary\nMachine translation is one of the most widely used applications of NLP, and the\nencoder-decoder model, \ufb01rst developed for MT is a key tool that has applications\nthroughout NLP.\n\u2022 Languages have divergences , both structural and lexical, that make translation\ndif\ufb01cult.\n\u2022 The linguistic \ufb01eld of typology investigates some of these differences; lan-\nguages can be classi\ufb01ed by their position along typological dimensions like\nwhether verbs precede their objects.\n\u2022Encoder-decoder networks (for transformers just as we saw in Chapter 8 for\nRNNs) are composed of an encoder network that takes an input sequence\nand creates a contextualized representation of it, the context . This context\nrepresentation is then passed to a decoder which generates a task-speci\ufb01c\noutput sequence.\n\u2022Cross-attention allows the transformer decoder to view information from all\nthe hidden states of the encoder.\n\u2022 Machine translation models are trained on a parallel corpus , sometimes called\nabitext , a text that appears in two (or more) languages. 286 CHAPTER 13 \u2022 M ACHINE TRANSLATION\n\u2022Backtranslation is a way of making use of monolingual corpora in the target\nlanguage by running a pilot MT engine backwards to create synthetic bitexts.\n\u2022 MT is evaluated by measuring a translation\u2019s adequacy (how well it captures\nthe meaning of the source sentence) and \ufb02uency (how \ufb02uent or natural it is\nin the target language). Human evaluation is the gold standard, but automatic\nevaluation metrics like chrF , which measure character n-gram overlap with\nhuman translations, or more recent metrics based on embedding similarity,\nare also commonly used.\nBibliographical and Historical Notes\nMT was proposed seriously by the late 1940s, soon after the birth of the computer\n(Weaver, 1949/1955). In 1954, the \ufb01rst public demonstration of an MT system pro-\ntotype (Dostert, 1955) led to great excitement in the press (Hutchins, 1997). The\nnext decade saw a great \ufb02owering of ideas, pre\ufb01guring most subsequent develop-\nments. But this work was ahead of its time\u2014implementations were limited by, for\nexample, the fact that pending the development of disks there was no good way to\nstore dictionary information.\nAs high-quality MT proved elusive (Bar-Hillel, 1960), there grew a consensus\non the need for better evaluation and more basic research in the new \ufb01elds of for-\nmal and computational linguistics. This consensus culminated in the famously crit-\nical ALPAC (Automatic Language Processing Advisory Committee) report of 1966\n(Pierce et al., 1966) that led in the mid 1960s to a dramatic cut in funding for MT\nin the US. As MT research lost academic respectability, the Association for Ma-\nchine Translation and Computational Linguistics dropped MT from its name. Some\nMT developers, however, persevered, and there were early MT systems like M \u00b4et\u00b4eo,\nwhich translated weather forecasts from English to French (Chandioux, 1976), and\nindustrial systems like Systran.\nIn the early years, the space of MT architectures spanned three general mod-\nels. In direct translation , the system proceeds word-by-word through the source-\nlanguage text, translating each word incrementally. Direct translation uses a large\nbilingual dictionary, each of whose entries is a small program with the job of trans-\nlating one word. In transfer approaches, we \ufb01rst parse the input text and then ap-\nply rules to transform the source-language parse into a target language parse. We\nthen generate the target language sentence from the parse tree. In interlingua ap-\nproaches, we analyze the source language text into some abstract meaning repre-\nsentation, called an interlingua . We then generate into the target language from\nthis interlingual representation. A common way to visualize these three early ap-\nproaches was the Vauquois triangle shown in Fig. 13.13. The triangle shows theVauquois\ntriangle\nincreasing depth of analysis required (on both the analysis and generation end) as\nwe move from the direct approach through transfer approaches to interlingual ap-\nproaches. In addition, it shows the decreasing amount of transfer knowledge needed\nas we move up the triangle, from huge amounts of transfer at the direct level (al-\nmost all knowledge is transfer knowledge for each word) through transfer (transfer\nrules only for parse trees or thematic roles) through interlingua (no speci\ufb01c transfer\nknowledge). We can view the encoder-decoder network as an interlingual approach,\nwith attention acting as an integration of direct and transfer, allowing words or their\nrepresentations to be directly accessed by the decoder. BIBLIOGRAPHICAL AND HISTORICAL NOTES 287\nsourcetexttarget textDirect TranslationTransferInterlinguaSource Text:Semantic/SyntacticStructureTarget Text:Semantic/SyntacticStructuresource languageanalysissource languageanalysistarget language generation\nFigure 13.13 The Vauquois (1968) triangle.\nStatistical methods began to be applied around 1990, enabled \ufb01rst by the devel-\nopment of large bilingual corpora like the Hansard corpus of the proceedings of the\nCanadian Parliament, which are kept in both French and English, and then by the\ngrowth of the Web. Early on, a number of researchers showed that it was possible\nto extract pairs of aligned sentences from bilingual corpora, using words or simple\ncues like sentence length (Kay and R \u00a8oscheisen 1988, Gale and Church 1991, Gale\nand Church 1993, Kay and R \u00a8oscheisen 1993).\nAt the same time, the IBM group, drawing directly on the noisy channel model\nfor speech recognition, proposed two related paradigms for statistical MT . These statistical MT\ninclude the generative algorithms that became known as IBM Models 1 through IBM Models\n5, implemented in the Candide system. The algorithms (except for the decoder) Candide\nwere published in full detail\u2014 encouraged by the US government who had par-\ntially funded the work\u2014 which gave them a huge impact on the research community\n(Brown et al. 1990, Brown et al. 1993).\nThe group also developed a discriminative approach, called MaxEnt (for maxi-\nmum entropy, an alternative formulation of logistic regression), which allowed many\nfeatures to be combined discriminatively rather than generatively (Berger et al.,\n1996), which was further developed by Och and Ney (2002).\nBy the turn of the century, most academic research on machine translation used\nstatistical MT, either in the generative or discriminative mode. An extended version\nof the generative approach, called phrase-based translation was developed, basedphrase-based\ntranslation\non inducing translations for phrase-pairs (Och 1998, Marcu and Wong 2002, Koehn\net al. (2003), Och and Ney 2004, Deng and Byrne 2005, inter alia).\nOnce automatic metrics like BLEU were developed (Papineni et al., 2002), the\ndiscriminative log linear formulation (Och and Ney, 2004), drawing from the IBM\nMaxEnt work (Berger et al., 1996), was used to directly optimize evaluation metrics\nlike BLEU in a method known as Minimum Error Rate Training , orMERT (Och, MERT\n2003), also drawing from speech recognition models (Chou et al., 1993). Toolkits\nlike GIZA (Och and Ney, 2003) and Moses (Koehn et al. 2006, Zens and Ney 2007) Moses\nwere widely used.\nThere were also approaches around the turn of the century that were based on\nsyntactic structure (Chapter 18). Models based on transduction grammars (alsotransduction\ngrammars\ncalled synchronous grammars assign a parallel syntactic tree structure to a pair of\nsentences in different languages, with the goal of translating the sentences by ap-\nplying reordering operations on the trees. From a generative perspective, we can\nview a transduction grammar as generating pairs of aligned sentences in two lan-\nguages. Some of the most widely used models included the inversion transduction\ngrammar (Wu, 1996) and synchronous context-free grammars (Chiang, 2005),inversion\ntransduction\ngrammar 288 CHAPTER 13 \u2022 M ACHINE TRANSLATION\nNeural networks had been applied at various times to various aspects of machine\ntranslation; for example Schwenk et al. (2006) showed how to use neural language\nmodels to replace n-gram language models in a Spanish-English system based on\nIBM Model 4. The modern neural encoder-decoder approach was pioneered by\nKalchbrenner and Blunsom (2013), who used a CNN encoder and an RNN decoder,\nand was \ufb01rst applied to MT by Bahdanau et al. (2015). The transformer encoder-\ndecoder was proposed by Vaswani et al. (2017) (see the History section of Chap-\nter 9).\nResearch on evaluation of machine translation began quite early. Miller and\nBeebe-Center (1956) proposed a number of methods drawing on work in psycholin-\nguistics. These included the use of cloze and Shannon tasks to measure intelligibility\nas well as a metric of edit distance from a human translation, the intuition that un-\nderlies all modern overlap-based automatic evaluation metrics. The ALPAC report\nincluded an early evaluation study conducted by John Carroll that was extremely in-\n\ufb02uential (Pierce et al., 1966, Appendix 10). Carroll proposed distinct measures for\n\ufb01delity and intelligibility, and had raters score them subjectively on 9-point scales.\nMuch early evaluation work focuses on automatic word-overlap metrics like BLEU\n(Papineni et al., 2002), NIST (Doddington, 2002), TER (Translation Error Rate)\n(Snover et al., 2006), Precision and Recall (Turian et al., 2003), and METEOR\n(Banerjee and Lavie, 2005); character n-gram overlap methods like chrF (Popovi \u00b4c,\n2015) came later. More recent evaluation work, echoing the ALPAC report, has\nemphasized the importance of careful statistical methodology and the use of human\nevaluation (Kocmi et al., 2021; Marie et al., 2021).\nThe early history of MT is surveyed in Hutchins 1986 and 1997; Nirenburg et al.\n(2002) collects early readings. See Croft (1990) or Comrie (1989) for introductions\nto linguistic typology.\nExercises\n13.1 Compute by hand the chrF2,2 score for HYP2 on page 282 (the answer should\nround to .62). CHAPTER\n14Question Answering, Informa-\ntion Retrieval, and Retrieval-\nAugmented Generation\nPeople need to know things. So pretty much as soon as there were computers we\nwere asking them questions. Systems in the early 1960s were answering questions\nabout baseball statistics and scienti\ufb01c facts. Even \ufb01ctional computers in the 1970s\nlike Deep Thought, invented by Douglas Adams in The Hitchhiker\u2019s Guide to the\nGalaxy , answered \u201cthe Ultimate Question Of Life, The Universe, and Everything\u201d.1\nAnd because so much knowledge is encoded in text, QA systems were performing\nat human levels even before LLMs: IBM\u2019s Watson system won the TV game-show\nJeopardy! in 2011, surpassing humans at answering questions like:\nWILLIAM WILKINSON\u2019S \u201cAN ACCOUNT OF THE PRINCIPALITIES OF WALLACHIA AND MOLDOVIA\u201dINSPIRED THIS AUTHOR\u2019S MOST FAMOUS NOVEL\n2\nQuestion answering systems are designed to \ufb01ll human information needs .\nSince a lot of information is present in text form (on the web or in other data like\nour email, or books), question answering is closely related to the task behind search\nengines. Indeed, the distinction is becoming ever more fuzzy, as modern search\nengines are integrated with large language models trained to do question answering.\nQuestion answering systems often focus on a useful subset of information needs:\nfactoid questions , questions of fact or reasoning that can be answered with simple\nfacts expressed in short or medium-length texts, like the following:\n(14.1) Where is the Louvre Museum located?\n(14.2) Where does the energy in a nuclear explosion come from?\n(14.3) How to get a script l in latex?\nModern NLP systems answer these questions using large language models, in\none of two ways. The \ufb01rst is to make use of the method from Chapter 12: prompt\na pretrained and instruction-tuned LLM, an LLM that has been \ufb01netuned on ques-\ntion/answer datasets with the question in the prompt. For example, we could prompt\na causal language model with a string like\nQ: Where is the Louvre Museum located? A:\nhave it do conditional generation given this pre\ufb01x, and take the response as the an-\nswer. The idea is that language models have read a lot of facts in their pretraining\ndata, presumably including the location of the Louvre, and have encoded this infor-\nmation in their parameters.\nSimply prompting an LLM can be a useful approach to answer many factoid\nquestions. But it is not yet a complete solution for question answering.\n1The answer was 42, but unfortunately the question was never revealed.\n2The answer, of course, is \u2018Who is Bram Stoker\u2019, and the novel was Dracula . 290 CHAPTER 14 \u2022 Q UESTION ANSWERING , INFORMATION RETRIEVAL ,AND RAG\nThe \ufb01rst and main problem is that large language models often give the wrong\nanswer! Large language models hallucinate . A hallucination is a response that is hallucinate\nnot faithful to the facts of the world. That is, when asked questions, large language\nmodels simply make up answers that sound reasonable. For example, Dahl et al.\n(2024) found that when asked questions about the legal domain (like about particular\nlegal cases), large language models hallucinated from 69% to 88% of the time!\nAnd it\u2019s not always possible to tell when language models are hallucinating,\npartly because LLMs aren\u2019t well- calibrated . In a calibrated system, the con\ufb01dence calibrated\nof a system in the correctness of its answer is highly correlated with the probability\nof an answer being correct. So if a calibrated system is wrong, at least it might hedge\nits answer or tell us to go check another source. But since language models are not\nwell-calibrated, they often give a very wrong answer with complete certainty (Zhou\net al., 2024).\nA second problem is that simply prompting a large language model doesn\u2019t allow\nus to ask questions about proprietary data. A common use of question answering is\nabout data like our personal email or medical records. Or a company may have\ninternal documents that contain answers for customer service or internal use. Or\nlegal \ufb01rms need to ask questions about legal discovery from proprietary documents.\nFinally, static large language models also have problems with questions about\nrapidly changing information (like questions about something that happened last\nweek) since LLMs won\u2019t have up-to-date information from after their release data.\nFor this reason the most common way to do question-answering with LLMs is\nretrieval-augmented generation orRAG , and that is the method we will focus on RAG\nin this chapter. In RAG we use information retrieval (IR) techniques to retrieveinformation\nretrieval\ndocuments that are likely to have information that might help answer the question.\nThen we use a large language model to generate an answer given these documents.\nBasing our answers on retrieved documents can solve some of the problems with\nusing simple prompting to answer questions. First, it helps ensure that the answer is\ngrounded in facts from some curated dataset. And the system can give the user the\nanswer accompanied by the context of the passage or document the answer came\nfrom. This information can help users have con\ufb01dence in the accuracy of the answer\n(or help them spot when it is wrong!). And these retrieval techniques can be used on\nany proprietary data we want, such as legal or medical data for those applications.\nWe\u2019ll begin by introducing information retrieval, the task of choosing the most\nrelevant document from a document set given a user\u2019s query expressing their infor-\nmation need. We\u2019ll see the classic method based on cosines of sparse tf-idf vectors,\na modern neural \u2018dense\u2019 retrievers based on instead representing queries and docu-\nments neurally with BERT or other language models. We then introduce retriever-\nbased question answering and the retrieval-augmented generation paradigm.\nFinally, we\u2019ll discuss various QA datasets. These are used for \ufb01netuning LLMs\nin instruction tuning, as we saw in Chapter 12. And they are also used as bench-\nmarks, since question answering has an important function as a benchmark for mea-\nsuring the abilities of language models.\n14.1 Information Retrieval\nInformation retrieval orIRis the name of the \ufb01eld encompassing the retrieval of allinformation\nretrieval\nIR manner of media based on user information needs. The resulting IR system is often\ncalled a search engine . Our goal in this section is to give a suf\ufb01cient overview of IR 14.1 \u2022 I NFORMATION RETRIEVAL 291\nto see its application to question answering. Readers with more interest speci\ufb01cally\nin information retrieval should see the Historical Notes section at the end of the\nchapter and textbooks like Manning et al. (2008).\nThe IR task we consider is called ad hoc retrieval , in which a user poses a ad hoc retrieval\nquery to a retrieval system, which then returns an ordered set of documents from\nsome collection . Adocument refers to whatever unit of text the system indexes and document\nretrieves (web pages, scienti\ufb01c papers, news articles, or even shorter passages like\nparagraphs). A collection refers to a set of documents being used to satisfy user collection\nrequests. A term refers to a word in a collection, but it may also include phrases. term\nFinally, a query represents a user\u2019s information need expressed as a set of terms. query\nThe high-level architecture of an ad hoc retrieval engine is shown in Fig. 14.1.\nDocumentDocumentDocumentDocumentDocumentDocumentQuery ProcessingIndexingSearchDocumentDocumentDocumentDocumentDocumentRanked DocumentsDocumentqueryInvertedIndexqueryvectordocument collection\nFigure 14.1 The architecture of an ad hoc IR system.\nThe basic IR architecture uses the vector space model we introduced in Chap-\nter 6, in which we map queries and document to vectors based on unigram word\ncounts, and use the cosine similarity between the vectors to rank potential documents\n(Salton, 1971). This is thus an example of the bag-of-words model introduced in\nChapter 4, since words are considered independently of their positions.\n14.1.1 Term weighting and document scoring\nLet\u2019s look at the details of how the match between a document and query is scored.\nWe don\u2019t use raw word counts in IR, instead computing a term weight for each term weight\ndocument word. Two term weighting schemes are common: the tf-idf weighting\nintroduced in Chapter 6, and a slightly more powerful variant called BM25 . BM25\nWe\u2019ll reintroduce tf-idf here so readers don\u2019t need to look back at Chapter 6.\nTf-idf (the \u2018-\u2019 here is a hyphen, not a minus sign) is the product of two terms, the\nterm frequency tfand the inverse document frequency idf.\nThe term frequency tells us how frequent the word is; words that occur more\noften in a document are likely to be informative about the document\u2019s contents. We\nusually use the log 10of the word frequency, rather than the raw count. The intuition\nis that a word appearing 100 times in a document doesn\u2019t make that word 100 times\nmore likely to be relevant to the meaning of the document. We also need to do\nsomething special with counts of 0, since we can\u2019t take the log of 0.3\ntft;d=(\n1+log10count (t;d) if count (t;d)>0\n0 otherwise(14.4)\n3We can also use this alternative formulation, which we have used in earlier editions: tf t;d=\nlog10(count (t;d)+1) 292 CHAPTER 14 \u2022 Q UESTION ANSWERING , INFORMATION RETRIEVAL ,AND RAG\nIf we use log weighting, terms which occur 0 times in a document would have tf =0,\n1 times in a document tf =1+log10(1) =1+0=1, 10 times in a document tf =\n1+log10(10) =2, 100 times tf =1+log10(100) =3, 1000 times tf =4, and so on.\nThe document frequency dftof a term tis the number of documents it oc-\ncurs in. Terms that occur in only a few documents are useful for discriminating\nthose documents from the rest of the collection; terms that occur across the entire\ncollection aren\u2019t as helpful. The inverse document frequency oridfterm weight\n(Sparck Jones, 1972) is de\ufb01ned as:\nidft=log10N\ndft(14.5)\nwhere Nis the total number of documents in the collection, and df tis the number\nof documents in which term toccurs. The fewer documents in which a term occurs,\nthe higher this weight; the lowest weight of 0 is assigned to terms that occur in every\ndocument.\nHere are some idf values for some words in the corpus of Shakespeare plays,\nranging from extremely informative words that occur in only one play like Romeo ,\nto those that occur in a few like salad orFalstaff , to those that are very common like\nfoolor so common as to be completely non-discriminative since they occur in all 37\nplays like good orsweet .4\nWord df idf\nRomeo 1 1.57\nsalad 2 1.27\nFalstaff 4 0.967\nforest 12 0.489\nbattle 21 0.246\nwit 34 0.037\nfool 36 0.012\ngood 37 0\nsweet 37 0\nThetf-idf value for word tin document dis then the product of term frequency\ntft;dand IDF:\ntf-idf(t;d) =tft;d\u0001idft (14.6)\n14.1.2 Document Scoring\nWe score document dby the cosine of its vector dwith the query vector q:\nscore(q;d) =cos(q;d) =q\u0001d\njqjjdj(14.7)\nAnother way to think of the cosine computation is as the dot product of unit vectors;\nwe \ufb01rst normalize both the query and document vector to unit vectors, by dividing\nby their lengths, and then take the dot product:\nscore(q;d) =cos(q;d) =q\njqj\u0001d\njdj(14.8)\n4Sweet was one of Shakespeare\u2019s favorite adjectives, a fact probably related to the increased use of\nsugar in European recipes around the turn of the 16th century (Jurafsky, 2014, p. 175). 14.1 \u2022 I NFORMATION RETRIEVAL 293\nWe can spell out Eq. 14.8, using the tf-idf values and spelling out the dot product as\na sum of products:\nscore(q;d) =X\nt2qtf-idf(t;q)qP\nqi2qtf-idf2(qi;q)\u0001tf-idf(t;d)qP\ndi2dtf-idf2(di;d)(14.9)\nNow let\u2019s use (14.9) to walk through an example of a tiny query against a collec-\ntion of 4 nano documents, computing tf-idf values and seeing the rank of the docu-\nments. We\u2019ll assume all words in the following query and documents are downcased\nand punctuation is removed:\nQuery :sweet love\nDoc 1 :Sweet sweet nurse! Love?\nDoc 2 :Sweet sorrow\nDoc 3 :How sweet is love?\nDoc 4 :Nurse!\nFig. 14.2 shows the computation of the tf-idf cosine between the query and Doc-\nument 1, and the query and Document 2. The cosine is the normalized dot product\nof tf-idf values, so for the normalization we must need to compute the document\nvector lengthsjqj,jd1j, andjd2jfor the query and the \ufb01rst two documents using\nEq. 14.4, Eq. 14.5, Eq. 14.6, and Eq. 14.9 (computations for Documents 3 and 4 are\nalso needed but are left as an exercise for the reader). The dot product between the\nvectors is the sum over dimensions of the product, for each dimension, of the values\nof the two tf-idf vectors for that dimension. This product is only non-zero where\nboth the query and document have non-zero values, so for this example, in which\nonly sweet andlove have non-zero values in the query, the dot product will be the\nsum of the products of those elements of each vector.\nDocument 1 has a higher cosine with the query (0.747) than Document 2 has\nwith the query (0.0779), and so the tf-idf cosine model would rank Document 1\nabove Document 2. This ranking is intuitive given the vector space model, since\nDocument 1 has both terms including two instances of sweet , while Document 2 is\nmissing one of the terms. We leave the computation for Documents 3 and 4 as an\nexercise for the reader.\nIn practice, there are many variants and approximations to Eq. 14.9. For exam-\nple, we might choose to simplify processing by removing some terms. To see this,\nlet\u2019s start by expanding the formula for tf-idf in Eq. 14.9 to explicitly mention the tf\nand idf terms from (14.6):\nscore(q;d) =X\nt2qtft;q\u0001idftqP\nqi2qtf-idf2(qi;q)\u0001tft;d\u0001idftqP\ndi2dtf-idf2(di;d)(14.10)\nIn one common variant of tf-idf cosine, for example, we drop the idf term for the\ndocument. Eliminating the second copy of the idf term (since the identical term is\nalready computed for the query) turns out to sometimes result in better performance:\nscore(q;d) =X\nt2qtft;q\u0001idftqP\nqi2qtf-idf2(qi;q)\u0001tft;d\u0001idftqP\ndi2dtf-idf2(di;d)(14.11)\nOther variants of tf-idf eliminate various other terms.\nA slightly more complex variant in the tf-idf family is the BM25 weighting BM25 294 CHAPTER 14 \u2022 Q UESTION ANSWERING , INFORMATION RETRIEVAL ,AND RAG\nQuery\nword cnt tf df idf tf-idf n\u2019lized = tf-idf/jqj\nsweet 1 1 3 0.125 0.125 0.383\nnurse 0 0 2 0.301 0 0\nlove 1 1 2 0.301 0.301 0.924\nhow 0 0 1 0.602 0 0\nsorrow 0 0 1 0.602 0 0\nis 0 0 1 0.602 0 0\njqj=p\n:1252+:3012=:326\nDocument 1 Document 2\nword cnt tf tf-idf n\u2019lized \u0002q cnt tf tf-idf n\u2019lized \u0002q\nsweet 2 1.301 0.163 0.357 0.137 1 1.000 0.125 0.203 0.0779\nnurse 1 1.000 0.301 0.661 0 0 0 0 0 0\nlove 1 1.000 0.301 0.661 0.610 0 0 0 0 0\nhow 0 0 0 0 0 0 0 0 0 0\nsorrow 0 0 0 0 0 1 1.000 0.602 0.979 0\nis 0 0 0 0 0 0 0 0 0 0\njd1j=p\n:1632+:3012+:3012=:456 jd2j=p\n:1252+:6022=:615\nCosine:Pof column: 0.747 Cosine:Pof column: 0.0779\nFigure 14.2 Computation of tf-idf cosine score between the query and nano-documents 1 (0.747) and 2\n(0.0779), using Eq. 14.4, Eq. 14.5, Eq. 14.6 and Eq. 14.9.\nscheme (sometimes called Okapi BM25 after the Okapi IR system in which it was\nintroduced (Robertson et al., 1995)). BM25 adds two parameters: k, a knob that\nadjust the balance between term frequency and IDF, and b, which controls the im-\nportance of document length normalization. The BM25 score of a document dgiven\na query qis:\nX\nt2qIDFz}|{\nlog\u0012N\ndft\u0013weighted tfz}|{\ntft;d\nk\u0010\n1\u0000b+b\u0010\njdj\njdavgj\u0011\u0011\n+tft;d(14.12)\nwherejdavgjis the length of the average document. When kis 0, BM25 reverts to\nno use of term frequency, just a binary selection of terms in the query (plus idf).\nA large kresults in raw term frequency (plus idf). branges from 1 (scaling by\ndocument length) to 0 (no length scaling). Manning et al. (2008) suggest reasonable\nvalues are k = [1.2,2] and b = 0.75. Kamphuis et al. (2020) is a useful summary of\nthe many minor variants of BM25.\nStop words In the past it was common to remove high-frequency words from both\nthe query and document before representing them. The list of such high-frequency\nwords to be removed is called a stop list . The intuition is that high-frequency terms stop list\n(often function words like the,a,to) carry little semantic weight and may not help\nwith retrieval, and can also help shrink the inverted index \ufb01les we describe below.\nThe downside of using a stop list is that it makes it dif\ufb01cult to search for phrases\nthat contain words in the stop list. For example, common stop lists would reduce the\nphrase to be or not to be to the phrase not. In modern IR systems, the use of stop lists\nis much less common, partly due to improved ef\ufb01ciency and partly because much\nof their function is already handled by IDF weighting, which downweights function 14.1 \u2022 I NFORMATION RETRIEVAL 295\nwords that occur in every document. Nonetheless, stop word removal is occasionally\nuseful in various NLP tasks so is worth keeping in mind.\n14.1.3 Inverted Index\nIn order to compute scores, we need to ef\ufb01ciently \ufb01nd documents that contain words\nin the query. (Any document that contains none of the query terms will have a score\nof 0 and can be ignored.) The basic search problem in IR is thus to \ufb01nd all documents\nd2Cthat contain a term q2Q.\nThe data structure for this task is the inverted index , which we use for mak- inverted index\ning this search ef\ufb01cient, and also conveniently storing useful information like the\ndocument frequency and the count of each term in each document.\nAn inverted index, given a query term, gives a list of documents that contain the\nterm. It consists of two parts, a dictionary and the postings . The dictionary is a list postings\nof terms (designed to be ef\ufb01ciently accessed), each pointing to a postings list for the\nterm. A postings list is the list of document IDs associated with each term, which\ncan also contain information like the term frequency or even the exact positions of\nterms in the document. The dictionary can also store the document frequency for\neach term. For example, a simple inverted index for our 4 sample documents above,\nwith each word containing its document frequency in fg, and a pointer to a postings\nlist that contains document IDs and term counts in [], might look like the following:\nhowf1g!3 [1]\nisf1g!3 [1]\nlovef2g!1 [1]!3 [1]\nnursef2g!1 [1]!4 [1]\nsorryf1g!2 [1]\nsweetf3g!1 [2]!2 [1]!3 [1]\nGiven a list of terms in query, we can very ef\ufb01ciently get lists of all candidate\ndocuments, together with the information necessary to compute the tf-idf scores we\nneed.\nThere are alternatives to the inverted index. For the question-answering domain\nof \ufb01nding Wikipedia pages to match a user query, Chen et al. (2017a) show that\nindexing based on bigrams works better than unigrams, and use ef\ufb01cient hashing\nalgorithms rather than the inverted index to make the search ef\ufb01cient.\n14.1.4 Evaluation of Information-Retrieval Systems\nWe measure the performance of ranked retrieval systems using the same precision\nandrecall metrics we have been using. We make the assumption that each docu-\nment returned by the IR system is either relevant to our purposes or not relevant .\nPrecision is the fraction of the returned documents that are relevant, and recall is the\nfraction of all relevant documents that are returned. More formally, let\u2019s assume a\nsystem returns Tranked documents in response to an information request, a subset\nRof these are relevant, a disjoint subset, N, are the remaining irrelevant documents,\nandUdocuments in the collection as a whole are relevant to this request. Precision\nand recall are then de\ufb01ned as:\nPrecision =jRj\njTjRecall =jRj\njUj(14.13)\nUnfortunately, these metrics don\u2019t adequately measure the performance of a system\nthatranks the documents it returns. If we are comparing the performance of two 296 CHAPTER 14 \u2022 Q UESTION ANSWERING , INFORMATION RETRIEVAL ,AND RAG\nranked retrieval systems, we need a metric that prefers the one that ranks the relevant\ndocuments higher. We need to adapt precision and recall to capture how well a\nsystem does at putting relevant documents higher in the ranking.\nRank Judgment Precision Rank Recall Rank\n1 R 1.0 .11\n2 N .50 .11\n3 R .66 .22\n4 N .50 .22\n5 R .60 .33\n6 R .66 .44\n7 N .57 .44\n8 R .63 .55\n9 N .55 .55\n10 N .50 .55\n11 R .55 .66\n12 N .50 .66\n13 N .46 .66\n14 N .43 .66\n15 R .47 .77\n16 N .44 .77\n17 N .44 .77\n18 R .44 .88\n19 N .42 .88\n20 N .40 .88\n21 N .38 .88\n22 N .36 .88\n23 N .35 .88\n24 N .33 .88\n25 R .36 1.0\nFigure 14.3 Rank-speci\ufb01c precision and recall values calculated as we proceed down\nthrough a set of ranked documents (assuming the collection has 9 relevant documents).\nLet\u2019s turn to an example. Assume the table in Fig. 14.3 gives rank-speci\ufb01c pre-\ncision and recall values calculated as we proceed down through a set of ranked doc-\numents for a particular query; the precisions are the fraction of relevant documents\nseen at a given rank, and recalls the fraction of relevant documents found at the same\nrank. The recall measures in this example are based on this query having 9 relevant\ndocuments in the collection as a whole.\nNote that recall is non-decreasing; when a relevant document is encountered,\nrecall increases, and when a non-relevant document is found it remains unchanged.\nPrecision, on the other hand, jumps up and down, increasing when relevant doc-\numents are found, and decreasing otherwise. The most common way to visualize\nprecision and recall is to plot precision against recall in a precision-recall curve ,precision-recall\ncurve\nlike the one shown in Fig. 14.4 for the data in table 14.3.\nFig. 14.4 shows the values for a single query. But we\u2019ll need to combine values\nfor all the queries, and in a way that lets us compare one system to another. One way\nof doing this is to plot averaged precision values at 11 \ufb01xed levels of recall (0 to 100,\nin steps of 10). Since we\u2019re not likely to have datapoints at these exact levels, we\nuseinterpolated precision values for the 11 recall values from the data points we dointerpolated\nprecision\nhave. We can accomplish this by choosing the maximum precision value achieved\nat any level of recall at or above the one we\u2019re calculating. In other words,\nIntPrecision (r) =max\ni>=rPrecision (i) (14.14) 14.1 \u2022 I NFORMATION RETRIEVAL 297\n0.0 0.2 0.4 0.6 0.8 1.0\nRecall0.00.20.40.60.81.0Precision\nFigure 14.4 The precision recall curve for the data in table 14.3.\nThis interpolation scheme not only lets us average performance over a set of queries,\nbut also helps smooth over the irregular precision values in the original data. It is\ndesigned to give systems the bene\ufb01t of the doubt by assigning the maximum preci-\nsion value achieved at higher levels of recall from the one being measured. Fig. 14.5\nand Fig. 14.6 show the resulting interpolated data points from our example.\nInterpolated Precision Recall\n1.0 0.0\n1.0 .10\n.66 .20\n.66 .30\n.66 .40\n.63 .50\n.55 .60\n.47 .70\n.44 .80\n.36 .90\n.36 1.0\nFigure 14.5 Interpolated data points from Fig. 14.3.\nGiven curves such as that in Fig. 14.6 we can compare two systems or approaches\nby comparing their curves. Clearly, curves that are higher in precision across all\nrecall values are preferred. However, these curves can also provide insight into the\noverall behavior of a system. Systems that are higher in precision toward the left\nmay favor precision over recall, while systems that are more geared towards recall\nwill be higher at higher levels of recall (to the right).\nA second way to evaluate ranked retrieval is mean average precision (MAP),mean average\nprecision\nwhich provides a single metric that can be used to compare competing systems or\napproaches. In this approach, we again descend through the ranked list of items,\nbut now we note the precision only at those points where a relevant item has been\nencountered (for example at ranks 1, 3, 5, 6 but not 2 or 4 in Fig. 14.3). For a single\nquery, we average these individual precision measurements over the return set (up\nto some \ufb01xed cutoff). More formally, if we assume that Rris the set of relevant 298 CHAPTER 14 \u2022 Q UESTION ANSWERING , INFORMATION RETRIEVAL ,AND RAG\nInterpolated Precision Recall Curve\n00.10.20.30.40.50.60.70.80.91\n0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1\nRecallPrecision\nFigure 14.6 An 11 point interpolated precision-recall curve. Precision at each of the 11\nstandard recall levels is interpolated for each query from the maximum at any higher level of\nrecall. The original measured precision recall points are also shown.\ndocuments at or above r, then the average precision (AP) for a single query is\nAP=1\njRrjX\nd2RrPrecision r(d) (14.15)\nwhere Precision r(d)is the precision measured at the rank at which document dwas\nfound. For an ensemble of queries Q, we then average over these averages, to get\nour \ufb01nal MAP measure:\nMAP =1\njQjX\nq2QAP(q) (14.16)\nThe MAP for the single query (hence = AP) in Fig. 14.3 is 0.6.\n14.2 Information Retrieval with Dense Vectors\nThe classic tf-idf or BM25 algorithms for IR have long been known to have a con-\nceptual \ufb02aw: they work only if there is exact overlap of words between the query\nand document. In other words, the user posing a query (or asking a question) needs\nto guess exactly what words the writer of the answer might have used, an issue called\nthevocabulary mismatch problem (Furnas et al., 1987).\nThe solution to this problem is to use an approach that can handle synonymy:\ninstead of (sparse) word-count vectors, using (dense) embeddings. This idea was\n\ufb01rst proposed for retrieval in the last century under the name of Latent Semantic\nIndexing approach (Deerwester et al., 1990), but is implemented in modern times\nvia encoders like BERT.\nThe most powerful approach is to present both the query and the document to a\nsingle encoder, allowing the transformer self-attention to see all the tokens of both 14.2 \u2022 I NFORMATION RETRIEVAL WITH DENSE VECTORS 299\nthe query and the document, and thus building a representation that is sensitive to\nthe meanings of both query and document. Then a linear layer can be put on top of\nthe [CLS] token to predict a similarity score for the query/document tuple:\nz=BERT (q;[SEP] ;d)[CLS]\nscore(q;d) =softmax (U(z)) (14.17)\nThis architecture is shown in Fig. 14.7a. Usually the retrieval step is not done on\nan entire document. Instead documents are broken up into smaller passages, such\nas non-overlapping \ufb01xed-length chunks of say 100 tokens, and the retriever encodes\nand retrieves these passages rather than entire documents. The query and document\nhave to be made to \ufb01t in the BERT 512-token window, for example by truncating\nthe query to 64 tokens and truncating the document if necessary so that it, the query,\n[CLS], and [SEP] \ufb01t in 512 tokens. The BERT system together with the linear layer\nUcan then be \ufb01ne-tuned for the relevance task by gathering a tuning dataset of\nrelevant and non-relevant passages.\nQueryDocument\u2026\u2026\u2026\u2026\u2026\u2026[sep]s(q,d)zCLSU\nQueryzCLS_QzCLS_D\nDocument\u2026\u2026\u2026\u2026\u2026\u2026\u2022s(q,d)\n(a) (b)\nFigure 14.7 Two ways to do dense retrieval, illustrated by using lines between layers to schematically rep-\nresent self-attention: (a) Use a single encoder to jointly encode query and document and \ufb01netune to produce a\nrelevance score with a linear layer over the CLS token. This is too compute-expensive to use except in rescoring\n(b) Use separate encoders for query and document, and use the dot product between CLS token outputs for the\nquery and document as the score. This is less compute-expensive, but not as accurate.\nThe problem with the full BERT architecture in Fig. 14.7a is the expense in\ncomputation and time. With this architecture, every time we get a query, we have to\npass every single single document in our entire collection through a BERT encoder\njointly with the new query! This enormous use of resources is impractical for real\ncases.\nAt the other end of the computational spectrum is a much more ef\ufb01cient archi-\ntecture, the bi-encoder . In this architecture we can encode the documents in the\ncollection only one time by using two separate encoder models, one to encode the\nquery and one to encode the document. We encode each document, and store all\nthe encoded document vectors in advance. When a query comes in, we encode just\nthis query and then use the dot product between the query vector and the precom-\nputed document vectors as the score for each candidate document (Fig. 14.7b). For\nexample, if we used BERT, we would have two encoders BERT Qand BERT Dand 300 CHAPTER 14 \u2022 Q UESTION ANSWERING , INFORMATION RETRIEVAL ,AND RAG\nwe could represent the query and document as the [CLS] token of the respective\nencoders (Karpukhin et al., 2020):\nzq=BERT Q(q)[CLS]\nzd=BERT D(d)[CLS]\nscore(q;d) =zq\u0001zd (14.18)\nThe bi-encoder is much cheaper than a full query/document encoder, but is also\nless accurate, since its relevance decision can\u2019t take full advantage of all the possi-\nble meaning interactions between all the tokens in the query and the tokens in the\ndocument.\nThere are numerous approaches that lie in between the full encoder and the bi-\nencoder. One intermediate alternative is to use cheaper methods (like BM25) as the\n\ufb01rst pass relevance ranking for each document, take the top N ranked documents,\nand use expensive methods like the full BERT scoring to rerank only the top N\ndocuments rather than the whole set.\nAnother intermediate approach is the ColBERT approach of Khattab and Za- ColBERT\nharia (2020) and Khattab et al. (2021), shown in Fig. 14.8. This method separately\nencodes the query and document, but rather than encoding the entire query or doc-\nument into one vector, it separately encodes each of them into contextual represen-\ntations for each token. These BERT representations of each document word can be\npre-stored for ef\ufb01ciency. The relevance score between a query qand a document dis\na sum of maximum similarity (MaxSim) operators between tokens in qand tokens\nind. Essentially, for each token in q, ColBERT \ufb01nds the most contextually simi-\nlar token in d, and then sums up these similarities. A relevant document will have\ntokens that are contextually very similar to the query.\nMore formally, a question qis tokenized as [q1;:::; qn], prepended with a [CLS]\nand a special [Q]token, truncated to N=32 tokens (or padded with [MASK] tokens if\nit is shorter), and passed through BERT to get output vectors q= [q1;:::;qN]. The\npassage dwith tokens [d1;:::; dm], is processed similarly, including a [CLS] and\nspecial[D]token. A linear layer is applied on top of dandqto control the output\ndimension, so as to keep the vectors small for storage ef\ufb01ciency, and vectors are\nrescaled to unit length, producing the \ufb01nal vector sequences Eq(length N) and Ed\n(length m). The ColBERT scoring mechanism is:\nscore(q;d) =NX\ni=1mmax\nj=1Eqi\u0001Edj(14.19)\nWhile the interaction mechanism has no tunable parameters, the ColBERT ar-\nchitecture still needs to be trained end-to-end to \ufb01ne-tune the BERT encoders and\ntrain the linear layers (and the special [Q] and[D] embeddings) from scratch. It\nis trained on triples hq;d+;d\u0000iof query q, positive document d+and negative doc-\nument d\u0000to produce a score for each document using (14.19), optimizing model\nparameters using a cross-entropy loss.\nAll the supervised algorithms (like ColBERT or the full-interaction version of\nthe BERT algorithm applied for reranking) need training data in the form of queries\ntogether with relevant and irrelevant passages or documents (positive and negative\nexamples). There are various semi-supervised ways to get labels; some datasets (like\nMS MARCO Ranking, Section 14.3.2) contain gold positive examples. Negative\nexamples can be sampled randomly from the top-1000 results from some existing\nIR system. If datasets don\u2019t have labeled positive examples, iterative methods like 14.3 \u2022 A NSWERING QUESTIONS WITH RAG 301\nQueryDocument\u2026\u2026\u2026\u2026\u2026\u2026s(q,d)MaxSimMaxSimMaxSim\u2211\nnormnormnormnormnormnorm\nFigure 14.8 A sketch of the ColBERT algorithm at inference time. The query and docu-\nment are \ufb01rst passed through separate BERT encoders. Similarity between query and doc-\nument is computed by summing a soft alignment between the contextual representations of\ntokens in the query and the document. Training is end-to-end. (Various details aren\u2019t de-\npicted; for example the query is prepended by a [CLS] and[Q:] tokens, and the document\nby[CLS] and[D:] tokens). Figure adapted from Khattab and Zaharia (2020).\nrelevance-guided supervision can be used (Khattab et al., 2021) which rely on the\nfact that many datasets contain short answer strings. In this method, an existing IR\nsystem is used to harvest examples that do contain short answer strings (the top few\nare taken as positives) or don\u2019t contain short answer strings (the top few are taken as\nnegatives), these are used to train a new retriever, and then the process is iterated.\nEf\ufb01ciency is an important issue, since every possible document must be ranked\nfor its similarity to the query. For sparse word-count vectors, the inverted index\nallows this very ef\ufb01ciently. For dense vector algorithms \ufb01nding the set of dense\ndocument vectors that have the highest dot product with a dense query vector is\nan instance of the problem of nearest neighbor search . Modern systems there-\nfore make use of approximate nearest neighbor vector search algorithms like Faiss Faiss\n(Johnson et al., 2017).\n14.3 Answering Questions with RAG\nThe dominant paradigm for question answering is to answer a user\u2019s question by \ufb01rst\n\ufb01nding supportive text segments from the web or another other large collection of\ndocuments, and then generating an answer based on the documents. The method of\ngenerating based on retrieved documents is called retrieval-augmented generation\norRAG , and the two components are sometimes called the retriever and the reader\n(Chen et al., 2017a). Fig. 14.9 sketches out this standard QA model.\nIn the \ufb01rst stage of the 2-stage retrieve and read model in Fig. 14.9 we retrieve\nrelevant passages from a text collection, for example using the dense retrievers of the\nprevious section. In the second reader stage, we generate the answer via retrieval- 302 CHAPTER 14 \u2022 Q UESTION ANSWERING , INFORMATION RETRIEVAL ,AND RAG\nQ: When wasthe premiere ofThe Magic Flute?RelevantDocsA:  1791RetrieverIndexed Docs\nquerydocsLLMpromptReader/Generator\nFigure 14.9 Retrieval-based question answering has two stages: retrieval , which returns relevant documents\nfrom the collection, and reading , in which an LLM generates answers given the documents as a prompt.\naugmented generation . In this method, we take a large pretrained language model,\ngive it the set of retrieved passages and other text as its prompt, and autoregressively\ngenerate a new answer token by token.\n14.3.1 Retrieval-Augmented Generation\nThe standard reader algorithm is to generate from a large language model, condi-\ntioned on the retrieved passages. This method is known as retrieval-augmented\ngeneration , orRAG .retrieval-\naugmented\ngeneration\nRAG Recall that in simple conditional generation, we can cast the task of question\nanswering as word prediction by giving a language model a question and a token\nlikeA:suggesting that an answer should come next:\nQ: Who wrote the book ``The Origin of Species\"? A:\nThen we generate autoregressively conditioned on this text.\nMore formally, recall that simple autoregressive language modeling computes\nthe probability of a string from the previous tokens:\np(x1;:::; xn) =nY\ni=1p(xijx<i)\nAnd simple conditional generation for question answering adds a prompt like Q:,\nfollowed by a query q, andA:, all concatenated:\np(x1;:::; xn) =nY\ni=1p([Q:] ;q;[A:] ;x<i)\nThe advantage of using a large language model is the enormous amount of\nknowledge encoded in its parameters from the text it was pretrained on. But as\nwe mentioned at the start of the chapter, while this kind of simple prompted gener-\nation can work \ufb01ne for many simple factoid questions, it is not a general solution\nfor QA, because it leads to hallucination, is unable to show users textual evidence to\nsupport the answer, and is unable to answer questions from proprietary data.\nThe idea of retrieval-augmented generation is to address these problems by con-\nditioning on the retrieved passages as part of the pre\ufb01x, perhaps with some prompt\ntext like \u201cBased on these texts, answer this question:\u201d. Let\u2019s suppose we have a\nquery q, and call the set of retrieved passages based on it R( q). For example, we\ncould have a prompt like: 14.3 \u2022 A NSWERING QUESTIONS WITH RAG 303\nSchematic of a RAG Prompt\nretrieved passage 1\nretrieved passage 2\n...\nretrieved passage n\nBased on these texts, answer this question: Q: Who wrote\nthe book ``The Origin of Species\"? A:\nOr more formally,\np(x1;:::; xn) =nY\ni=1p(xijR(q) ; prompt ; [Q:] ;q;[A:] ;x<i)\nAs with the span-based extraction reader, successfully applying the retrieval-\naugmented generation algorithm for QA requires a successful retriever, and often\na two-stage retrieval algorithm is used in which the retrieval is reranked. Some\ncomplex questions may require multi-hop architectures, in which a query is used to multi-hop\nretrieve documents, which are then appended to the original query for a second stage\nof retrieval. Details of prompt engineering also have to be worked out, like deciding\nwhether to demarcate passages, for example with [SEP] tokens, and so on. Combi-\nnations of private data and public data involving an externally hosted large language\nmodel may lead to privacy concerns that need to be worked out (Arora et al., 2023).\nMuch research in this area also focuses on ways to more tightly integrated the re-\ntrieval and reader stages.\n14.3.2 Question Answering Datasets\nThere are scores of question answering datasets, used both for instruction tuning and\nfor evaluation of the question answering abilities of language models.\nWe can distinguish the datasets along many dimensions, summarized nicely in\nRogers et al. (2023). One is the original purpose of the questions in the data, whether\nthey were natural information-seeking questions, or whether they were questions\ndesigned for probing : evaluating or testing systems or humans.\nOn the natural side there are datasets like Natural Questions (KwiatkowskiNatural\nQuestions\net al., 2019), a set of anonymized English queries to the Google search engine and\ntheir answers. The answers are created by annotators based on Wikipedia infor-\nmation, and include a paragraph-length long answer and a short span answer. For\nexample the question \u201cWhen are hops added to the brewing process?\u201d has the short\nanswer the boiling process and a long answer which is an entire paragraph from the\nWikipedia page on Brewing .\nA similar natural question set is the MS MARCO (Microsoft Machine Reading MS MARCO\nComprehension) collection of datasets, including 1 million real anonymized English\nquestions from Microsoft Bing query logs together with a human generated answer\nand 9 million passages (Bajaj et al., 2016), that can be used both to test retrieval\nranking and question answering. 304 CHAPTER 14 \u2022 Q UESTION ANSWERING , INFORMATION RETRIEVAL ,AND RAG\nAlthough many datasets focus on English, natural information-seeking ques-\ntion datasets exist in other languages. The DuReader dataset is a Chinese QA\nresource based on search engine queries and community QA (He et al., 2018).\nTyDi QA dataset contains 204K question-answer pairs from 11 typologically di- TyDi QA\nverse languages, including Arabic, Bengali, Kiswahili, Russian, and Thai (Clark\net al., 2020a). In the T YDIQA task, a system is given a question and the passages\nfrom a Wikipedia article and must (a) select the passage containing the answer (or\nNULL if no passage contains the answer), and (b) mark the minimal answer span (or\nNULL).\nOn the probing side are datasets like MMLU (Massive Multitask Language Un- MMLU\nderstanding), a commonly-used dataset of 15908 knowledge and reasoning ques-\ntions in 57 areas including medicine, mathematics, computer science, law, and oth-\ners. MMLU questions are sourced from various exams for humans, such as the US\nGraduate Record Exam, Medical Licensing Examination, and Advanced Placement\nexams. So the questions don\u2019t represent people\u2019s information needs, but rather are\ndesigned to test human knowledge for academic or licensing purposes. Fig. 14.10\nshows some examples, with the correct answers in bold.\nSome of the question datasets described above augment each question with pas-\nsage(s) from which the answer can be extracted. These datasets were mainly created\nfor an earlier QA task called reading comprehension in which a model is givenreading\ncomprehension\na question and a document and is required to extract the answer from the given\ndocument. We sometimes call the task of question answering given one or more\ndocuments (for example via RAG), the open book QA task, while the task of an- open book\nswering directly from the LM with no retrieval component at all is the closed book closed book\nQA task.5Thus datasets like Natural Questions can be treated as open book if the\nsolver uses each question\u2019s attached document, or closed book if the documents are\nnot used, while datasets like MMLU are solely closed book.\nAnother dimension of variation is the format of the answer: multiple-choice\nversus freeform. And of course there are variations in prompting, like whether the\nmodel is just the question (zero-shot) or also given demonstrations of answers to\nsimilar questions (few-shot). MMLU offers both zero-shot and few-shot prompt\noptions.\n14.4 Evaluating Question Answering\nThree techniques are commonly employed to evaluate question-answering systems,\nwith the choice depending on the type of question and QA situation. For multiple\nchoice questions like in MMLU, we report exact match:\nExact match : The % of predicted answers that match the gold answer\nexactly.\nFor questions with free text answers, like Natural Questions, we commonly evalu-\nated with token F1score to roughly measure the partial string overlap between the\nanswer and the reference answer:\nF1score : The average token overlap between predicted and gold an-\nswers. Treat the prediction and gold as a bag of tokens, and compute F 1\nfor each question, then return the average F 1over all questions.\n5This repurposes the word for types of exams in which students are allowed to \u2018open their books\u2019 or\nnot. 14.4 \u2022 E VALUATING QUESTION ANSWERING 305\nMMLU examples\nCollege Computer Science\nAny set of Boolean operators that is suf\ufb01cient to represent all Boolean ex-\npressions is said to be complete. Which of the following is NOT complete?\n(A) AND, NOT\n(B) NOT, OR\n(C) AND, OR\n(D) NAND\nCollege Physics\nThe primary source of the Sun\u2019s energy is a series of thermonuclear\nreactions in which the energy produced is c2times the mass difference\nbetween\n(A) two hydrogen atoms and one helium atom\n(B)four hydrogen atoms and one helium atom\n(C) six hydrogen atoms and two helium atoms\n(D) three helium atoms and one carbon atom\nInternational Law\nWhich of the following is a treaty-based human rights mechanism?\n(A)The UN Human Rights Committee\n(B) The UN Human Rights Council\n(C) The UN Universal Periodic Review\n(D) The UN special mandates\nPrehistory\nUnlike most other early civilizations, Minoan culture shows little evidence\nof\n(A) trade.\n(B) warfare.\n(C) the development of a common religion.\n(D)conspicuous consumption by elites.\nFigure 14.10 Example problems from MMLU\nFinally, in some situations QA systems give multiple ranked answers. In such cases\nwe evaluated using mean reciprocal rank , orMRR (V oorhees, 1999). MRR ismean\nreciprocal rank\nMRR designed for systems that return a short ranked list of answers or passages for each\ntest set question, which we can compare against the (human-labeled) correct answer.\nFirst, each test set question is scored with the reciprocal of the rank of the \ufb01rst\ncorrect answer. For example if the system returned \ufb01ve answers to a question but\nthe \ufb01rst three are wrong (so the highest-ranked correct answer is ranked fourth), the\nreciprocal rank for that question is1\n4. The score for questions that return no correct\nanswer is 0. The MRR of a system is the average of the scores for each question in\nthe test set. In some versions of MRR, questions with a score of zero are ignored\nin this calculation. More formally, for a system returning ranked answers to each\nquestion in a test set Q, (or in the alternate version, let Qbe the subset of test set 306 CHAPTER 14 \u2022 Q UESTION ANSWERING , INFORMATION RETRIEVAL ,AND RAG\nquestions that have non-zero scores). MRR is then de\ufb01ned as\nMRR =1\njQjjQjX\ni=11\nrank i(14.20)\n14.5 Summary\nThis chapter introduced the tasks of question answering andinformation retrieval .\n\u2022Question answering (QA) is the task of answering a user\u2019s questions.\n\u2022 We focus in this chapter on the task of retrieval-based question answering,\nin which the user\u2019s questions are intended to be answered by the material in\nsome set of documents (which might be the web).\n\u2022Information Retrieval (IR) is the task of returning documents to a user based\non their information need as expressed in a query . In ranked retrieval, the\ndocuments are returned in ranked order.\n\u2022 The match between a query and a document can be done by \ufb01rst representing\neach of them with a sparse vector that represents the frequencies of words,\nweighted by tf-idf orBM25 . Then the similarity can be measured by cosine.\n\u2022 Documents or queries can instead be represented by dense vectors, by encod-\ning the question and document with an encoder-only model like BERT, and in\nthat case computing similarity in embedding space.\n\u2022 The inverted index is a storage mechanism that makes it very ef\ufb01cient to \ufb01nd\ndocuments that have a particular word.\n\u2022 Ranked retrieval is generally evaluated by mean average precision orinter-\npolated precision .\n\u2022 Question answering systems generally use the retriever /reader architecture.\nIn the retriever stage, an IR system is given a query and returns a set of\ndocuments.\n\u2022 The reader stage is implemented by retrieval-augmented generation , in\nwhich a large language model is prompted with the query and a set of doc-\numents and then conditionally generates a novel answer.\n\u2022 QA can be evaluated by exact match with a known answer if only a single\nanswer is given, with token F 1score for free text answers, or with mean re-\nciprocal rank if a ranked set of answers is given.\nBibliographical and Historical Notes\nQuestion answering was one of the earliest NLP tasks, and early versions of the text-\nbased and knowledge-based paradigms were developed by the very early 1960s. The\ntext-based algorithms generally relied on simple parsing of the question and of the\nsentences in the document, and then looking for matches. This approach was used\nvery early on (Phillips, 1960) but perhaps the most complete early system, and one\nthat strikingly pre\ufb01gures modern relation-based systems, was the Protosynthex sys-\ntem of Simmons et al. (1964). Given a question, Protosynthex \ufb01rst formed a query BIBLIOGRAPHICAL AND HISTORICAL NOTES 307\nfrom the content words in the question, and then retrieved candidate answer sen-\ntences in the document, ranked by their frequency-weighted term overlap with the\nquestion. The query and each retrieved sentence were then parsed with dependency\nparsers, and the sentence whose structure best matches the question structure se-\nlected. Thus the question What do worms eat? would match worms eat grass : both\nhave the subject worms as a dependent of eat, in the version of dependency grammar\nused at the time, while birds eat worms hasbirds as the subject:\nWhatdowormseat Wormseatgrass Birdseatworms\nThe alternative knowledge-based paradigm was implemented in the BASEBALL\nsystem (Green et al., 1961). This system answered questions about baseball games\nlike \u201cWhere did the Red Sox play on July 7\u201d by querying a structured database of\ngame information. The database was stored as a kind of attribute-value matrix with\nvalues for attributes of each game:\nMonth = July\nPlace = Boston\nDay = 7\nGame Serial No. = 96\n(Team = Red Sox, Score = 5)\n(Team = Yankees, Score = 3)\nEach question was constituency-parsed using the algorithm of Zellig Harris\u2019s\nTDAP project at the University of Pennsylvania, essentially a cascade of \ufb01nite-state\ntransducers (see the historical discussion in Joshi and Hopely 1999 and Karttunen\n1999). Then in a content analysis phase each word or phrase was associated with a\nprogram that computed parts of its meaning. Thus the phrase \u2018Where\u2019 had code to\nassign the semantics Place = ? , with the result that the question \u201cWhere did the\nRed Sox play on July 7\u201d was assigned the meaning\nPlace = ?\nTeam = Red Sox\nMonth = July\nDay = 7\nThe question is then matched against the database to return the answer. Simmons\n(1965) summarizes other early QA systems.\nAnother important progenitor of the knowledge-based paradigm for question-\nanswering is work that used predicate calculus as the meaning representation lan-\nguage. The LUNAR system (Woods et al. 1972, Woods 1978) was designed to be LUNAR\na natural language interface to a database of chemical facts about lunar geology. It\ncould answer questions like Do any samples have greater than 13 percent aluminum\nby parsing them into a logical form\n(TEST (FOR SOME X16 / (SEQ SAMPLES) : T ; (CONTAIN\u2019 X16\n(NPR* X17 / (QUOTE AL203)) (GREATERTHAN 13 PCT))))\nBy a couple decades later, drawing on new machine learning approaches in NLP,\nZelle and Mooney (1996) proposed to treat knowledge-based QA as a semantic pars-\ning task, by creating the Prolog-based GEOQUERY dataset of questions about US\ngeography. This model was extended by Zettlemoyer and Collins (2005) and 2007. 308 CHAPTER 14 \u2022 Q UESTION ANSWERING , INFORMATION RETRIEVAL ,AND RAG\nBy a decade later, neural models were applied to semantic parsing (Dong and Lap-\nata 2016, Jia and Liang 2016), and then to knowledge-based question answering by\nmapping text to SQL (Iyer et al., 2017).\nMeanwhile, the information-retrieval paradigm for question answering was in-\n\ufb02uenced by the rise of the web in the 1990s. The U.S. government-sponsored TREC\n(Text REtrieval Conference) evaluations, run annually since 1992, provide a testbed\nfor evaluating information-retrieval tasks and techniques (V oorhees and Harman,\n2005). TREC added an in\ufb02uential QA track in 1999, which led to a wide variety of\nfactoid and non-factoid systems competing in annual evaluations.\nAt that same time, Hirschman et al. (1999) introduced the idea of using chil-\ndren\u2019s reading comprehension tests to evaluate machine text comprehension algo-\nrithms. They acquired a corpus of 120 passages with 5 questions each designed for\n3rd-6th grade children, built an answer extraction system, and measured how well\nthe answers given by their system corresponded to the answer key from the test\u2019s\npublisher. Their algorithm focused on word overlap as a feature; later algorithms\nadded named entity features and more complex similarity between the question and\nthe answer span (Riloff and Thelen 2000, Ng et al. 2000).\nThe DeepQA component of the Watson Jeopardy! system was a large and so-\nphisticated feature-based system developed just before neural systems became com-\nmon. It is described in a series of papers in volume 56 of the IBM Journal of Re-\nsearch and Development, e.g., Ferrucci (2012).\nEarly neural reading comprehension systems drew on the insight common to\nearly systems that answer \ufb01nding should focus on question-passage similarity. Many\nof the architectural outlines of these neural systems were laid out in Hermann et al.\n(2015a), Chen et al. (2017a), and Seo et al. (2017). These systems focused on\ndatasets like Rajpurkar et al. (2016) and Rajpurkar et al. (2018) and their succes-\nsors, usually using separate IR algorithms as input to neural reading comprehension\nsystems. The paradigm of using dense retrieval with a span-based reader, often with\na single end-to-end architecture, is exempli\ufb01ed by systems like Lee et al. (2019) or\nKarpukhin et al. (2020). An important research area with dense retrieval for open-\ndomain QA is training data: using self-supervised methods to avoid having to label\npositive and negative passages (Sachan et al., 2023).\nEarly work on large language models showed that they stored suf\ufb01cient knowl-\nedge in the pretraining process to answer questions (Petroni et al., 2019; Raffel et al.,\n2020; Radford et al., 2019; Roberts et al., 2020), at \ufb01rst not competitively with\nspecial-purpose question answerers, but then surpassing them. Retrieval-augmented\ngeneration algorithms were \ufb01rst introduced as a way to improve language modeling\n(Khandelwal et al., 2019), but were quickly applied to question answering (Izacard\net al., 2022; Ram et al., 2023; Shi et al., 2023).\nExercises CHAPTER\n15Chatbots & Dialogue Systems\nLes lois de la conversation sont en g \u00b4en\u00b4eral de ne s\u2019y appesantir sur aucun ob-\njet, mais de passer l \u00b4eg`erement, sans effort et sans affectation, d\u2019un sujet `a un\nautre ; de savoir y parler de choses frivoles comme de choses s \u00b4erieuses\n[The rules of conversation are, in general, not to dwell on any one subject,\nbut to pass lightly from one to another without effort and without affectation;\nto know how to speak about trivial topics as well as serious ones;]\nThe 18th C. Encyclopedia of Diderot, start of the entry on conversation\nThe literature of the fantastic abounds in inanimate objects magically endowed with\nthe gift of speech. From Ovid\u2019s statue of Pygmalion to Mary Shelley\u2019s story about\nFrankenstein, we continually reinvent stories about creat-\ning something and then having a chat with it. Legend has\nit that after \ufb01nishing his sculpture Moses , Michelangelo\nthought it so lifelike that he tapped it on the knee and\ncommanded it to speak. Perhaps this shouldn\u2019t be sur-\nprising. Language is the mark of humanity and sentience,\nandconversation ordialogue is the most fundamental conversation\ndialogue arena of language. It is the \ufb01rst kind of language we\nlearn as children, and the kind we engage in constantly,\nwhether we are ordering lunch, buying train tickets, or\ntalking with our families, friends, or coworkers.\nThis chapter introduces the fundamental algorithms of programs that use con-\nversation to interact with users. We often distinguish between two kinds of archi-\ntectures. Task-oriented dialogue systems converse with users to accomplish \ufb01xed dialogue system\ntasks like controlling appliances or \ufb01nding restaurants, relying on a data structure\ncalled the frame , which represents the knowledge a system needs to acquire from frame\nthe user (like the time to set an alarm clock). Chatbots , by contrast, are designed chatbot\nto mimic the longer and more unstructured conversations or \u2018chats\u2019 characteristic of\nhuman-human interaction. Modern systems incorporate aspects of both; industrial\nchatbots like ChatGPT can carry on longer unstructured conversations; industrial\ndigital assistants like Siri or Alexa are generally frame-based dialogue systems.\nThe fact that chatbots and dialogue systems are designed for human-computer\ninteraction has strong implications for their design and use. Many of these impli-\ncations already became clear in one of the earliest chatbots, ELIZA (Weizenbaum,\n1966). ELIZA was designed to simulate a Rogerian psychologist, based on a branch\nof clinical psychology whose methods involve drawing the patient out by re\ufb02ecting\npatient\u2019s statements back at them. Rogerian interactions are the rare type of conver-\nsation in which, as Weizenbaum points out, one can \u201cassume the pose of knowing\nalmost nothing of the real world\u201d. If a patient says \u201cI went for a long boat ride\u201d and\nthe psychiatrist says \u201cTell me about boats\u201d, you don\u2019t assume she didn\u2019t know what 310 CHAPTER 15 \u2022 C HATBOTS & D IALOGUE SYSTEMS\nMen are all alike.\nIN WHAT WAY\nThey\u2019re always bugging us about something or other.\nCAN YOU THINK OF A SPECIFIC EXAMPLE\nWell my boyfriend made me come here.\nYOUR BOYFRIEND MADE YOU COME HERE\nHe says I\u2019m depressed much of the time.\nI AM SORRY TO HEAR YOU ARE DEPRESSED\nFigure 15.1 Sample ELIZA dialogue from Weizenbaum (1966).\na boat is, but rather assume she had some conversational goal.1\nWeizenbaum made use of this property of Rogerian psychiatric conversations,\nalong with clever regular expressions, to allow ELIZA to interact in ways that seemed\ndeceptively human-like, as in the sample conversational fragment in Fig. 15.1.\nAs we foreshadowed in Chapter 2, ELIZA worked by simple rules roughly like:\n(.*) YOU (.*) ME -> WHAT MAKES YOU THINK I \\2 YOU\nto transform a user sentence like \u201c You hate me \u201d into a system response like\nWHAT MAKES YOU THINK I HATE YOU\nAmong Weizenbaum\u2019s clever tricks are the linking of each ELIZA pattern/rule\nto a keyword. Consider the following user sentence:\nI know everybody laughed at me\nBecause it has the word \u201cI\u201d, this sentence could match the following rule whose\nkeyword is I:\nI (.*) -> You say you \\1\nproducing:\nYOU SAY YOU KNOW EVERYBODY LAUGHED AT YOU\nWeizenbaum points out, however, that a more powerful response would rely on\nthe keyword \u201ceverybody\u201d, since someone using universals like everybody oralways\nis probably thinking about a speci\ufb01c person or situation. So the ELIZA algorithm\nprefers to respond using patterns associated more speci\ufb01c keywords like everybody :\nWHO IN PARTICULAR ARE YOU THINKING OF?\nIf no keyword matches, the algorithm chooses a non-committal response like\n\u201cPLEASE GO ON \u201d, \u201cTHAT'S VERY INTERESTING \u201d, or \u201cI SEE \u201d.\nELIZA illustrates a number of important issues with chatbots. First, people\nbecame deeply emotionally involved and conducted very personal conversations,\neven to the extent of asking Weizenbaum to leave the room while they were typ-\ning. Reeves and Nass (1996) show that people tend to assign human characteristics\nto computers and interact with them in ways that are typical of human-human in-\nteractions. They interpret an utterance in the way they would if it had spoken by a\nhuman, (even though they are aware they are talking to a computer). This means that\nchatbots can have signi\ufb01cant in\ufb02uences on people\u2019s cognitive and emotional state.\nA second related issue is privacy. When Weizenbaum suggested that he might\nwant to store the ELIZA conversations, people immediately pointed out that this\nwould violate people\u2019s privacy. Modern chatbots in the home are likely to overhear\n1This is due to the Gricean principle of relevance that we\u2019ll discuss in the next section.. 15.1 \u2022 P ROPERTIES OF HUMAN CONVERSATION 311\nprivate information, even if they aren\u2019t used for counseling as ELIZA was. Indeed,\nif a chatbot is human-like, users are more likely to disclose private information, and\nyet less likely to worry about the harm of this disclosure (Ischen et al., 2019).\nBoth of these issues (emotional engagement and privacy) mean we need to think\ncarefully about how we deploy chatbots and the people who are interacting with\nthem. Dialogue research that uses human participants often requires getting permis-\nsion from the Institutional Review Board (IRB) of your institution.\nIn the next section we introduce some basic properties of human conversation.\nWe then turn in the rest of the chapter to the two basic paradigms for conversational\ninteraction: frame-based dialogue systems and chatbots.\n15.1 Properties of Human Conversation\nConversation between humans is an intricate and complex joint activity. Before\nwe attempt to design a dialogue system to converse with humans, it is crucial to\nunderstand something about how humans converse with each other. Consider some\nof the phenomena that occur in the conversation between a human travel agent and\na human client excerpted in Fig. 15.2.\nC1: . . . I need to travel in May.\nA2: And, what day in May did you want to travel?\nC3: OK uh I need to be there for a meeting that\u2019s from the 12th to the 15th.\nA4: And you\u2019re \ufb02ying into what city?\nC5: Seattle.\nA6: And what time would you like to leave Pittsburgh?\nC7: Uh hmm I don\u2019t think there\u2019s many options for non-stop.\nA8: Right. There\u2019s three non-stops today.\nC9: What are they?\nA10: The \ufb01rst one departs PGH at 10:00am arrives Seattle at 12:05 their time.\nThe second \ufb02ight departs PGH at 5:55pm, arrives Seattle at 8pm. And the\nlast \ufb02ight departs PGH at 8:15pm arrives Seattle at 10:28pm.\nC11: OK I\u2019ll take the 5ish \ufb02ight on the night before on the 11th.\nA12: On the 11th? OK. Departing at 5:55pm arrives Seattle at 8pm, U.S. Air\n\ufb02ight 115.\nC13: OK.\nA14: And you said returning on May 15th?\nC15: Uh, yeah, at the end of the day.\nA16: OK. There\u2019s #two non-stops . . . #\nC17: #Act. . . actually #, what day of the week is the 15th?\nA18: It\u2019s a Friday.\nC19: Uh hmm. I would consider staying there an extra day til Sunday.\nA20: OK. . . OK. On Sunday I have . . .\nFigure 15.2 Part of a phone conversation between a human travel agent (A) and human\nclient (C). The passages framed by # in A 16and C 17indicate overlaps in speech.\nTurns\nA dialogue is a sequence of turns (C1, A2, C3, and so on), each a single contribution turn\nfrom one speaker to the dialogue (as if in a game: I take a turn, then you take a turn, 312 CHAPTER 15 \u2022 C HATBOTS & D IALOGUE SYSTEMS\nthen me, and so on). There are 20 turns in Fig. 15.2. A turn can consist of a sentence\n(like C 1), although it might be as short as a single word (C 13) or as long as multiple\nsentences (A 10).\nTurn structure has important implications for spoken dialogue. A human has\nto know when to stop talking; the client interrupts (in A 16and C 17), so a system\nthat was performing this role must know to stop talking (and that the user might be\nmaking a correction). A system also has to know when to start talking. For example,\nmost of the time in conversation, speakers start their turns almost immediately after\nthe other speaker \ufb01nishes, without a long pause, because people are can usually\npredict when the other person is about to \ufb01nish talking. Spoken dialogue systems\nmust also detect whether a user is done speaking, so they can process the utterance\nand respond. This task\u2014called endpointing orendpoint detection \u2014 can be quite endpointing\nchallenging because of noise and because people often pause in the middle of turns.\nSpeech Acts\nA key insight into conversation\u2014due originally to the philosopher Wittgenstein\n(1953) but worked out more fully by Austin (1962)\u2014is that each utterance in a\ndialogue is a kind of action being performed by the speaker. These actions are com-\nmonly called speech acts ordialogue acts : here\u2019s one taxonomy consisting of 4 speech acts\nmajor classes (Bach and Harnish, 1979):\nConstatives: committing the speaker to something\u2019s being the case ( answering ,claiming ,\ncon\ufb01rming ,denying ,disagreeing ,stating )\nDirectives: attempts by the speaker to get the addressee to do something ( advising ,ask-\ning,forbidding ,inviting ,ordering ,requesting )\nCommissives: committing the speaker to some future course of action ( promising, planning,\nvowing, betting, opposing )\nAcknowledgments: express the speaker\u2019s attitude regarding the hearer with respect to some so-\ncial action ( apologizing ,greeting ,thanking ,accepting an acknowledgment )\nA user asking a person or a dialogue system to do something (\u2018Turn up the mu-\nsic\u2019) is issuing a D IRECTIVE . Asking a question that requires an answer is also\na way of issuing a D IRECTIVE : in a sense when the system says (A 2) \u201cwhat day\nin May did you want to travel?\u201d it\u2019s as if the system is (very politely) command-\ning the user to answer. By contrast, a user stating a constraint (like C 1\u2018I need to\ntravel in May\u2019) is issuing a C ONSTATIVE . A user thanking the system is issuing\nan A CKNOWLEDGMENT . The speech act expresses an important component of the\nintention of the speaker (or writer) in saying what they said.\nGrounding\nA dialogue is not just a series of independent speech acts, but rather a collective act\nperformed by the speaker and the hearer. Like all collective acts, it\u2019s important for\nthe participants to establish what they both agree on, called the common groundcommon\nground\n(Stalnaker, 1978). Speakers do this by grounding each other\u2019s utterances. Ground- grounding\ning means acknowledging that the hearer has understood the speaker (Clark, 1996).\n(People need grounding for non-linguistic actions as well; the reason an elevator but-\nton lights up when it\u2019s pressed is to acknowledge that the elevator has indeed been\ncalled, essentially grounding your action of pushing the button (Norman, 1988).)\nHumans constantly ground each other\u2019s utterances. We can ground by explicitly\nsaying \u201cOK\u201d, as the agent does in A 8or A 10. Or we can ground by repeating what\nthe other person says; in utterance A 2the agent repeats \u201cin May\u201d, demonstrating her 15.1 \u2022 P ROPERTIES OF HUMAN CONVERSATION 313\nunderstanding to the client. Or notice that when the client answers a question, the\nagent begins the next question with \u201cAnd\u201d. The \u201cAnd\u201d implies that the new question\nis \u2018in addition\u2019 to the old question, again indicating to the client that the agent has\nsuccessfully understood the answer to the last question.\nSubdialogues and Dialogue Structure\nConversations have structure. Consider, for example, the local structure between\nspeech acts discussed in the \ufb01eld of conversation analysis (Sacks et al., 1974).conversation\nanalysis\nQUESTIONS set up an expectation for an ANSWER . PROPOSALS are followed by\nACCEPTANCE (orREJECTION ). C OMPLIMENTS (\u201cNice jacket!\u201d) often give rise to\nDOWNPLAYERS (\u201cOh, this old thing?\u201d). These pairs, called adjacency pairs are adjacency pair\ncomposed of a \ufb01rst pair part and a second pair part (Schegloff, 1968), and these\nexpectations can help systems decide what actions to take.\nHowever, dialogue acts aren\u2019t always followed immediately by their second pair\npart. The two parts can be separated by a side sequence (Jefferson 1972) or sub- side sequence\ndialogue . For example utterances C 17to A 20constitute a correction subdialogue subdialogue\n(Litman 1985, Litman and Allen 1987, Chu-Carroll and Carberry 1998):\nC17: #Act. . . actually#, what day of the week is the 15th?\nA18: It\u2019s a Friday.\nC19: Uh hmm. I would consider staying there an extra day til Sunday.\nA20: OK. . . OK. On Sunday I have . . .\nThe question in C 17interrupts the prior discourse, in which the agent was looking\nfor a May 15 return \ufb02ight. The agent must answer the question and also realize that\n\u2018\u2019I would consider staying...til Sunday\u201d means that the client would probably like to\nchange their plan, and now go back to \ufb01nding return \ufb02ights, but for the 17th.\nAnother side sequence is the clari\ufb01cation question , which can form a subdia-\nlogue between a REQUEST and a RESPONSE . This is especially common in dialogue\nsystems where speech recognition errors causes the system to have to ask for clari-\n\ufb01cations or repetitions like the following:\nUser: What do you have going to UNKNOWN WORD on the 5th?\nSystem: Let\u2019s see, going where on the 5th?\nUser: Going to Hong Kong.\nSystem: OK, here are some \ufb02ights...\nIn addition to side-sequences, questions often have presequences , like the fol- presequence\nlowing example where a user starts with a question about the system\u2019s capabilities\n(\u201cCan you make train reservations\u201d) before making a request.\nUser: Can you make train reservations?\nSystem: Yes I can.\nUser: Great, I\u2019d like to reserve a seat on the 4pm train to New York.\nInitiative\nSometimes a conversation is completely controlled by one participant. For exam-\nple a reporter interviewing a chef might ask questions, and the chef responds. We\nsay that the reporter in this case has the conversational initiative (Carbonell, 1970; initiative\nNickerson, 1976). In normal human-human dialogue, however, it\u2019s more common\nfor initiative to shift back and forth between the participants, as they sometimes\nanswer questions, sometimes ask them, sometimes take the conversations in new di-\nrections, sometimes not. You may ask me a question, and then I respond asking you 314 CHAPTER 15 \u2022 C HATBOTS & D IALOGUE SYSTEMS\nto clarify something you said, which leads the conversation in all sorts of ways. We\ncall such interactions mixed initiative (Carbonell, 1970).\nFull mixed initiative, while the norm for human-human conversations, can be\ndif\ufb01cult for dialogue systems. The most primitive dialogue systems tend to use\nsystem-initiative , where the system asks a question and the user can\u2019t do anything\nuntil they answer it, or user-initiative like simple search engines, where the user\nspeci\ufb01es a query and the system passively responds. Even modern large language\nmodel-based dialogue systems, which come much closer to using full mixed initia-\ntive, often don\u2019t have completely natural initiative switching. Getting this right is an\nimportant goal for modern systems.\nInference and Implicature\nInference is also important in dialogue understanding. Consider the client\u2019s response\nC2, repeated here:\nA2: And, what day in May did you want to travel?\nC3: OK uh I need to be there for a meeting that\u2019s from the 12th to the 15th.\nNotice that the client does not in fact answer the agent\u2019s question. The client\nmerely mentions a meeting at a certain time. What is it that licenses the agent to\ninfer that the client is mentioning this meeting so as to inform the agent of the travel\ndates?\nThe speaker seems to expect the hearer to draw certain inferences; in other\nwords, the speaker is communicating more information than seems to be present\nin the uttered words. This kind of example was pointed out by Grice (1975, 1978)\nas part of his theory of conversational implicature .Implicature means a particu- implicature\nlar class of licensed inferences. Grice proposed that what enables hearers to draw\nthese inferences is that conversation is guided by a set of maxims , general heuristics\nthat play a guiding role in the interpretation of conversational utterances. One such\nmaxim is the maxim of relevance which says that speakers attempt to be relevant, relevance\nthey don\u2019t just utter random speech acts. When the client mentions a meeting on the\n12th, the agent reasons \u2018There must be some relevance for mentioning this meeting.\nWhat could it be?\u2019. The agent knows that one precondition for having a meeting\n(at least before Web conferencing) is being at the place where the meeting is held,\nand therefore that maybe the meeting is a reason for the travel, and if so, then since\npeople like to arrive the day before a meeting, the agent should infer that the \ufb02ight\nshould be on the 11th.\nThese subtle characteristics of human conversations ( turns ,speech acts ,ground-\ning,dialogue structure ,initiative , and implicature ) are among the reasons it is dif-\n\ufb01cult to build dialogue systems that can carry on natural conversations with humans.\nMany of these challenges are active areas of dialogue systems research.\n15.2 Frame-Based Dialogue Systems\nAtask-based dialogue system has the goal of helping a user solve a speci\ufb01c task\nlike making a travel reservation or buying a product. Task-based dialogue systems\nare based around frames , \ufb01rst introduced in the early in\ufb02uential GUS system for frame\nGUS travel planning (Bobrow et al., 1977). Frames are knowledge structures representing\nthe details of the user\u2019s task speci\ufb01cation. Each frame consists of a collection of\nslots , each of which can take a set of possible values . Together a set of frames is slot 15.2 \u2022 F RAME -BASED DIALOGUE SYSTEMS 315\nsometimes called a domain ontology .\nHere we\u2019ll describe the most well-studied frame-based architecture, the dialogue-\nstate architecture, made up of the six components shown in Fig. 15.3. In the next\nsections we\u2019ll introduce four of them, after introducing the idea of frames (deferring\nthe speech recognition and synthesis components to Chapter 16).\nFigure 15.3 Architecture of a dialogue-state system for task-oriented dialogue from Williams et al. (2016).\n15.2.1 Frames and Slot Filling\nThe frame and its slots in a task-based dialogue system specify what the system\nneeds to know to perform its task. A hotel reservation system needs dates and loca-\ntions. An alarm clock system needs a time. The system\u2019s goal is to \ufb01ll the slots in\nthe frame with the \ufb01llers the user intends, and then perform the relevant action for\nthe user (answering a question, or booking a \ufb02ight).\nFig. 15.4 shows a sample frame for booking air travel, with some sample ques-\ntions used for \ufb01lling slots. In the simplest frame-based systems (including most com-\nmercial assistants until quite recently), these questions are pre-written templates, but\nin more sophisticated systems, questions are generated on-the-\ufb02y. The slot \ufb01llers are\noften constrained to a particular semantic type, like type CITY (taking on values like\nSan Francisco , orHong Kong ) or DATE ,AIRLINE , or TIME .\nSlot Type Example Question\nORIGIN CITY city \u201cFrom what city are you leaving?\u201d\nDESTINATION CITY city \u201cWhere are you going?\u201d\nDEPARTURE TIME time \u201cWhen would you like to leave?\u201d\nDEPARTURE DATE date \u201cWhat day would you like to leave?\u201d\nARRIV AL TIME time \u201cWhen do you want to arrive?\u201d\nARRIV AL DATE date \u201cWhat day would you like to arrive?\u201d\nFigure 15.4 A frame in a frame-based dialogue system, showing the type of each slot and\na sample question used to \ufb01ll the slot. 316 CHAPTER 15 \u2022 C HATBOTS & D IALOGUE SYSTEMS\nMany domains require multiple frames. Besides frames for car or hotel reser-\nvations, we might need other frames for things like general route information (for\nquestions like Which airlines \ufb02y from Boston to San Francisco? ), That means the\nsystem must be able to disambiguate which slot of which frame a given input is\nsupposed to \ufb01ll.\nThe task of slot-\ufb01lling is usually combined with two other tasks, to extract 3\nthings from each user utterance. The \ufb01rst is domain classi\ufb01cation : is this user for\nexample talking about airlines, programming an alarm clock, or dealing with their\ncalendar? The second is user intent determination : what general task or goal is theintent\ndetermination\nuser trying to accomplish? For example the task could be to Find a Movie, or Show\na Flight, or Remove a Calendar Appointment. Together, the domain classi\ufb01cation\nand intent determination tasks decide which frame we are \ufb01lling. Finally, we need\nto do slot \ufb01lling itself: extract the particular slots and \ufb01llers that the user intends the slot \ufb01lling\nsystem to understand from their utterance with respect to their intent. From a user\nutterance like this one:\nShow me morning flights from Boston to San Francisco on Tuesday\na system might want to build a representation like:\nDOMAIN: AIR-TRAVEL INTENT: SHOW-FLIGHTS\nORIGIN-CITY: Boston DEST-CITY: San Francisco\nORIGIN-DATE: Tuesday ORIGIN-TIME: morning\nSimilarly an utterance like this: should give an intent like this:\nWake me tomorrow at 6 DOMAIN: ALARM-CLOCK\nINTENT: SET-ALARM\nTIME: 2017-07-01 0600\nThe simplest dialogue systems use handwritten rules for slot-\ufb01lling, like this\nregular expression for recognizing the SET-ALARM intent:\nwake me (up) | set (the|an) alarm | get me up\nBut most systems use supervised machine-learning: each sentence in a training\nset is annotated with slots, domain, and intent, and a sequence model maps from\ninput words to slot \ufb01llers, domain and intent. For example we\u2019ll have pairs of sen-\ntences that are labeled for domain ( AIRLINE ) and intent ( SHOWFLIGHT ), and are also\nlabeled with BIO representations for the slots and \ufb01llers. (Recall from Chapter 17\nthat in BIO tagging we introduce a tag for the beginning (B) and inside (I) of each\nslot label, and one for tokens outside (O) any slot label.)\nO O O O O B-DES I-DES O B-DEPTIME I-DEPTIME O AIRLINE-SHOWFLIGHT\nI want to fly to San Francisco on Monday afternoon please EOS\nFig. 15.5 shows a typical architecture for inference. The input words w1:::wn\nare passed through a pretrained language model encoder, followed by a feedforward\nlayer and a softmax at each token position over possible BIO tags, with the output\na series of BIO tags s1:::sn. We generally combine the domain-classi\ufb01cation and\nintent-extraction tasks with slot-\ufb01lling by adding a domain concatenated with an\nintent as the desired output for the \ufb01nal EOS token.\nOnce the sequence labeler has tagged the user utterance, a \ufb01ller string can be ex-\ntracted for each slot from the tags (e.g., \u201cSan Francisco\u201d), and these word strings\ncan then be normalized to the correct form in the ontology (perhaps the airport 15.3 \u2022 D IALOGUE ACTS AND DIALOGUE STATE 317\nSanFranciscoonMondayEncodingsClassi\ufb01er+softmaxB-DESI-DESOB-DTIME\n\u2026d+i\n<EOS>Encoder \nFigure 15.5 Slot \ufb01lling by passing input words through an encoder, and then using a linear\nor feedforward layer followed by a softmax to generate a series of BIO tags. Here we also\nshow a \ufb01nal state: a domain concatenated with an intent.\ncode \u2018SFO\u2019), for example with dictionaries that specify that SF, SFO, and San Fran-\ncisco are synonyms. Often in industrial contexts, combinations of rules and machine\nlearning are used for each of these components.\nWe can make a very simple frame-based dialogue system by wrapping a small\namount of code around this slot extractor. Mainly we just need to ask the user\nquestions until all the slots are full, do a database query, then report back to the user,\nusing hand-built templates for generating sentences.\n15.2.2 Evaluating Task-Based Dialogue\nWe evaluate task-based systems by computing the task error rate , ortask success task error rate\nrate: the percentage of times the system booked the right plane \ufb02ight, or put the\nright event on the calendar. A more \ufb01ne-grained, but less extrinsic metric is the slot\nerror rate , the percentage of slots \ufb01lled with the correct values: slot error rate\nSlot Error Rate for a Sentence =# of inserted/deleted/subsituted slots\n# of total reference slots for sentence(15.1)\nFor example a system that extracted the slot structure below from this sentence:\n(15.2) Make an appointment with Chris at 10:30 in Gates 104\nSlot Filler\nPERSON Chris\nTIME 11:30 a.m.\nROOM Gates 104\nhas a slot error rate of 1/3, since the TIME is wrong. Instead of error rate, slot\nprecision, recall, and F-score can also be used. We can also measure ef\ufb01ciency\ncosts like the length of the dialogue in seconds or turns. ef\ufb01ciency costs\n15.3 Dialogue Acts and Dialogue State\nWhile the naive slot-extractor system described above can handle simple dialogues,\noften we want more complex interactions. For example, we might want to con\ufb01rm\nthat we\u2019ve understand the user, or ask them to repeat themselves. We can build a\nmore sophisticated system using dialogue acts anddialogue state . 318 CHAPTER 15 \u2022 C HATBOTS & D IALOGUE SYSTEMS\n15.3.1 Dialogue Acts\nDialogue acts are a generalization of speech acts that also represent grounding. The dialogue acts\nset of acts can be general, or can be designed for particular dialogue tasks.\nTag Sys User Description\nHELLO (a=x;b=y;:::)X X Open a dialogue and give info a=x;b=y;:::\nINFORM (a=x;b=y;:::)X X Give info a=x;b=y;:::\nREQUEST (a;b=x;:::)X X Request value for a given b=x;:::\nREQALTS (a=x;:::) cX Request alternative with a=x;:::\nCONFIRM (a=x;b=y;:::)X X Explicitly con\ufb01rm a=x;b=y;:::\nCONFREQ (a=x;:::;d)Xc Implicitly con\ufb01rm a=x;:::and request value of d\nSELECT (a=x;a=y)Xc Implicitly con\ufb01rm a=x;:::and request value of d\nAFFIRM (a=x;b=y;:::)X X Af\ufb01rm and give further info a=x;b=y;:::\nNEGATE (a=x) cX Negate and give corrected value a=x\nDENY (a=x) cX Deny that a=x\nBYE() X X Close a dialogue\nFigure 15.6 Dialogue acts used by the HIS restaurant recommendation system of Young\net al. (2010). The Sys and User columns indicate which acts are valid as system outputs and\nuser inputs, respectively.\nFigure 15.6 shows a tagset for a restaurant recommendation system, and Fig. 15.7\nshows these tags labeling a sample dialogue from the HIS system (Young et al.,\n2010). This example also shows the content of each dialogue act, which are the slot\n\ufb01llers being communicated. So the user might INFORM the system that they want\nItalian food near a museum, or CONFIRM with the system that the price is reasonable.\nUtterance Dialogue act\nU: Hi, I am looking for somewhere to eat. hello(task = find,type=restaurant)\nS: You are looking for a restaurant. What\ntype of food do you like?confreq(type = restaurant, food)\nU: I\u2019d like an Italian near the museum. inform(food = Italian, near=museum)\nS: Roma is a nice Italian restaurant near\nthe museum.inform(name = \"Roma\", type = restaurant,\nfood = Italian, near = museum)\nU: Is it reasonably priced? confirm(pricerange = moderate)\nS: Yes, Roma is in the moderate price\nrange.affirm(name = \"Roma\", pricerange =\nmoderate)\nU: What is the phone number? request(phone)\nS: The number of Roma is 385456. inform(name = \"Roma\", phone = \"385456\")\nU: Ok, thank you goodbye. bye()\nFigure 15.7 A dialogue from the HIS System of Young et al. (2010) using the dialogue acts in Fig. 15.6.\n15.3.2 Dialogue State Tracking\nThe job of the dialogue-state tracker is to determine the current state of the frame\n(the \ufb01llers of each slot), and the user\u2019s most recent dialogue act. The dialogue-state\nis not just the slot-\ufb01llers in the current sentence; it includes the entire state of the\nframe at this point, summarizing all of the user\u2019s constraints. Fig. 15.8 from Mrk \u02c7si\u00b4c\net al. (2017) shows the dialogue state after each turn.\nDialogue act detection is done just like domain or intent classi\ufb01cation, by passing\nthe input sentence through an encoder and adding an act classi\ufb01er. Often passing in\nthe prior dialogue act as well can improve classi\ufb01cation. And since dialogue acts 15.3 \u2022 D IALOGUE ACTS AND DIALOGUE STATE 319\nUser: I\u2019m looking for a cheaper restaurant\ninform(price=cheap)\nSystem: Sure. What kind - and where?\nUser: Thai food, somewhere downtown\ninform(price=cheap, food=Thai, area=centre)\nSystem: The House serves cheap Thai food\nUser: Where is it?\ninform(price=cheap, food=Thai, area=centre); request(address)\nSystem: The House is at 106 Regent Street\nFigure 15.8 The output of the dialogue state tracker after each turn (Mrk \u02c7si\u00b4c et al., 2017).\nplace some constraints on the slots and values, the tasks of dialogue-act detection and\nslot-\ufb01lling are often performed jointly. The state tracker can just take the output of\na slot-\ufb01lling sequence-model (Section 15.2.1) after each sentence, or do something\nmore complicated like training a classi\ufb01er to decide if a value has been changed.\nA special case: detecting correction acts. If a dialogue system misrecognizes\nor misunderstands an utterance, users will repeat or reformulate the utterance. De-\ntecting these user correction acts is quite important, especially for spoken lan-user correction\nacts\nguage. Ironically, corrections are actually harder to recognize than normal sentences\n(Swerts et al., 2000), because users who are frustrated adjust their speech in a way\nthat is dif\ufb01cult for speech recognizers (Goldberg et al., 2003). For example speak-\ners often use a prosodic style for corrections called hyperarticulation , in which thehyperarticula-\ntion\nutterance is louder or longer or exaggerated in pitch, such as I said BAL-TI-MORE,\nnot Boston (Wade et al. 1992, Levow 1998, Hirschberg et al. 2001). Detecting acts\ncan be part of the general dialogue act detection classi\ufb01er, or can make use of spe-\ncial features beyond the words, like those shown below (Levow 1998, Litman et al.\n1999, Hirschberg et al. 2001, Bulyko et al. 2005, Awadallah et al. 2015).\nfeatures examples\nsemantic embedding similarity between correction and user\u2019s prior utterance\nphonetic phonetic overlap between candidate correction act and user\u2019s prior utterance\n(i.e. \u201cWhatsApp\u201d may be incorrectly recognized as \u201cWhat\u2019s up\u201d)\nprosodic hyperarticulation, increases in F0 range, pause duration, and word duration\nASR ASR con\ufb01dence, language model probability\n15.3.3 Dialogue Policy: Which act to generate\nIn early commercial frame-based systems, the dialogue policy is simple: ask ques-\ntions until all the slots are full, do a database query, then report back to the user. A\nmore sophisticated dialogue policy can help a system decide when to answer the dialogue policy\nuser\u2019s questions, when to instead ask the user a clari\ufb01cation question, and so on. A\ndialogue policy thus decides what dialogue act to generate. Choosing a dialogue act\nto generate, along with its arguments, is sometimes called content planning .content\nplanning\nLet\u2019s see how to do this for some important dialogue acts. Dialogue systems, es-\npecially speech systems, often misrecognize the users\u2019 words or meaning. To ensure\nsystem and user share a common ground, systems must con\ufb01rm understandings with\nthe user or reject utterances that the system don\u2019t understand. A system might use\nanexplicit con\ufb01rmation act to con\ufb01rm with the user, like Is that correct? below:explicit\ncon\ufb01rmation 320 CHAPTER 15 \u2022 C HATBOTS & D IALOGUE SYSTEMS\nU:I\u2019d like to \ufb02y from Denver Colorado to New York City on September\ntwenty \ufb01rst in the morning on United Airlines\nS:Let\u2019s see then. I have you going from Denver Colorado to New York\non September twenty \ufb01rst. Is that correct?\nWhen using an implicit con\ufb01rmation act, a system instead grounds more im-implicit\ncon\ufb01rmation\nplicitly, for example by repeating the system\u2019s understanding as part of asking the\nnext question, as Shanghai is con\ufb01rmed in passing in this example:\nU:I want to travel to to Shanghai\nS:When do you want to travel to Shanghai?\nThere\u2019s a tradeoff. Explicit con\ufb01rmation makes it easier for users to correct mis-\nrecognitions by just answering \u201cno\u201d to the con\ufb01rmation question. But explicit con-\n\ufb01rmation is time-consuming and awkward (Danieli and Gerbino 1995, Walker et al.\n1998a). We also might want an act that expresses lack of understanding: rejection , rejection\nfor example with a prompt like I\u2019m sorry, I didn\u2019t understand that . To decide among\nthese acts, we can make use of the fact that ASR systems often compute their con\ufb01-\ndence in their transcription (often based on the log-likelihood the system assigns the\nsentence). A system can thus choose to explicitly con\ufb01rm only low-con\ufb01dence sen-\ntences. Or systems might have a four-tiered level of con\ufb01dence with three thresholds\na,b, and g:\n<alow con\ufb01dence reject\n\u0015aabove the threshold con\ufb01rm explicitly\n\u0015bhigh con\ufb01dence con\ufb01rm implictly\n\u0015gvery high con\ufb01dence don\u2019t con\ufb01rm at all\n15.3.4 Natural language generation: Sentence Realization\nrecommend(restaurant name= Au Midi, neighborhood = midtown,\ncuisine = french)\n1 Au Midi is in Midtown and serves French food.\n2 There is a French restaurant in Midtown called Au Midi.\nFigure 15.9 Sample inputs to the sentence realization phase of NLG, showing the dialogue\nact and attributes prespeci\ufb01ed by the content planner, and two distinct potential output sen-\ntences to be generated. From the restaurant recommendation system of Nayak et al. (2017).\nOnce a dialogue act has been chosen, we need to generate the text of the re-\nsponse to the user. This part of the generation process is called sentence realiza-\ntion. Fig. 15.9 shows a sample input/output for the sentence realization phase. Thesentence\nrealization\ncontent planner has chosen the dialogue act RECOMMEND and some slots (name,\nneighborhood, cuisine) and \ufb01llers. The sentence realizer generates a sentence like\nlines 1 or 2 (by training on examples of representation/sentence pairs from a corpus\nof labeled dialogues). Because we won\u2019t see every restaurant or attribute in every\npossible wording, we can delexicalize : generalize the training examples by replac- delexicalize\ning speci\ufb01c slot value words in the training set with a generic placeholder token\nrepresenting the slot. Fig. 15.10 shows the sentences in Fig. 15.9 delexicalized.\nWe can map from frames to delexicalized sentences with an encoder decoder\nmodel (Mrk \u02c7si\u00b4c et al. 2017, inter alia), trained on hand-labeled dialogue corpora like\nMultiWOZ (Budzianowski et al., 2018). The input to the encoder is a sequence of 15.4 \u2022 C HATBOTS 321\nrecommend(restaurant name= Au Midi, neighborhood = midtown,\ncuisine = french)\n1restaurant name is inneighborhood and serves cuisine food.\n2 There is a cuisine restaurant in neighborhood calledrestaurant name .\nFigure 15.10 Delexicalized sentences that can be used for generating many different relex-\nicalized sentences. From the restaurant recommendation system of Nayak et al. (2017).\ndecentservice:RECOMMENDcuisine:null[name]hasdecentserviceENCODERDECODER\nFigure 15.11 An encoder decoder sentence realizer mapping slots/\ufb01llers to English.\ntokens xtthat represent the dialogue act (e.g., RECOMMEND ) and its arguments (e.g.,\nservice:decent, cuisine:null ) (Nayak et al., 2017), as in Fig. 15.11.\nThe decoder outputs the delexicalized English sentence \u201c name has decent ser-\nvice\u201d, which we can then relexicalize , i.e. \ufb01ll back in correct slot values, resulting relexicalize\nin \u201cAu Midi has decent service\u201d.\n15.4 Chatbots\nChatbots are systems that can carry on extended conversations with the goal of chatbot\nmimicking the unstructured conversations or \u2018chats\u2019 characteristic of informal human-\nhuman interaction. While early systems like ELIZA (Weizenbaum, 1966) or PARRY\n(Colby et al., 1971) had theoretical goals like testing theories of psychological coun-\nseling, for most of the last 50 years chatbots have been designed for entertainment.\nThat changed with the recent rise of neural chatbots like ChatGPT, which incor-\nporate solutions to NLP tasks like question answering, writing tools, or machine\ntranslation into a conversational interface. A conversation with ChatGPT is shown\nin Fig. 15.12. In this section we describe neural chatbot architectures and datasets.\n[TBD]\nFigure 15.12 A conversation with ChatGPT.\n15.4.1 Training chatbots\nData Chatbots are generally trained on a training set that includes standard large\nlanguage model training data of the type discussed in Section 10.3.2: versions of the\nweb from the Common Crawl, including news sites, Wikipedia, as well as books.\nFor training chatbots, it is common to additionally add lots of dialogue data.\nThis can include datasets created speci\ufb01cally for training chatbots by hiring\nspeakers of the language to have conversations, such as by having them take on\npersonas or talk about knowledge provided to them. For example the Topical-Chat\ndataset has 11K crowdsourced conversations spanning 8 broad topics (Gopalakrish-\nnan et al., 2019), the E MPATHETIC DIALOGUES includes 25K crowdsourced con- 322 CHAPTER 15 \u2022 C HATBOTS & D IALOGUE SYSTEMS\nversations grounded in a speci\ufb01c situation where a speaker was feeling a speci\ufb01c\nemotion (Rashkin et al., 2019), and the SaFeRDialogues dataset (Ung et al., 2022)\nhas 8k dialogues demonstrating graceful responses to conversational feedback about\nsafety failures.\nSuch datasets are far too small to train a language model alone, and so it\u2019s com-\nmon to also pretrain on large datasets of pseudo-conversations drawn from Twitter\n(Ritter et al., 2010a), Reddit (Roller et al., 2021), Weibo ( \u5fae\u535a), and other social\nmedia platforms. To turn social media data into data that has the structure of a con-\nversation, we can treat any post on the platform as the \ufb01rst turn in a conversation,\nand the sequence of comments/replies as subsequent turns in that conversation.\nDatasets from the web can be enormously toxic, so it\u2019s crucial to \ufb01lter the di-\nalogues \ufb01rst. This can be done by using the same toxicity classi\ufb01ers we describe\nbelow in the \ufb01ne-tuning section.\nArchitecture For training chatbots, it\u2019s most common to use the standard causal\nlanguage model architecture, in which the model predicts each word given all the\nprior words, and the loss is the standard language modeling loss. Fig. 15.13 shows a\nstandard training setup; no different than language model training in Chapter 9. The\nonly difference is the data, which has the addition of signi\ufb01cant conversation and\npseudo-conversation data as described in the prior section. As usual, the left context\ncan include the entire prior conversation (or as much as \ufb01ts in the context window).\nTransformerBlocksLM head\ngotpromoted!<s>gotpromoted!<s>Next wordCongratsLM Loss\u2026LM headLM headLM headLM headLM head\nICongrats!\u2026LM headLM head!-log y!-log yCongrats-log y<s>-log y!-log ypromoted-log ygot\u2026\u2026\u2026\u2026\u2026\u2026\nFigure 15.13 Training a causal (decoder-only) language model for a chatbot.\nAn alternative is to use the encoder-decoder architecture of Chapter 13. In this\ncase the entire conversation up to the last turn (as much as \ufb01ts in the context) is\npresented to the encoder, and the decoder generates the next turn.\npromotedgot!<s>Congrats!ENCODERDECODERI\nFigure 15.14 An alternative: an encoder-decoder language model for a chatbot.\nIn practice, dialogue systems require additional customization beyond just pre-\ntraining on dialogue data. In the next few sections we\u2019ll discuss various stages of 15.4 \u2022 C HATBOTS 323\n\ufb01ne-tuning that can be used for this customization.\n15.4.2 Fine Tuning for Quality and Safety\nIt is a common practice for dialogue systems to use further labeled data for \ufb01ne-\ntuning. One function of this \ufb01ne-tuning step is to improve the quality of the dialogue,\ntraining the system to produce responses that are sensible and interesting. Another\nfunction might be to improve safety, keeping a dialogue system from suggesting\nharmful actions (like \ufb01nancial fraud, medical harm, inciting hatred, or abusing the\nuser or other people).\nIn the simplest method for improving quality and safety, speakers of the lan-\nguage are given an initial prompt and instructions to have high-quality, safe dia-\nlogues. They then interact with an initial dialogue system and their responses are\nused to \ufb01netune the model, usually as part of the instruct tuning step we introduced\nin Chapter 12. Thus a dialogue system learns to answer questions, follow other\ninstructions, and also carry on high-quality, safe dialogues, in a single multi-task\nlearning format.\nWhile \ufb01ne-tuning on positive examples is helpful, it is generally insuf\ufb01cient and\nso it is common to add more discriminative data that speci\ufb01cally downweights low-\nquality or harmful responses. The simplest paradigm for this is to train a model to\npredict turn-level safety and quality values, by training on human-labeled ratings.\nSuch ratings might be collected by \ufb01rst having speakers of the language carry on\ndialogues with a system, and then a second set of people act as labelers to label\nevery system turn for its quality and safety, resulting in a binary label for quality and\nsafety for each turn.\nOnce a dataset has been created with these labels, a language model can be used\nin a classi\ufb01cation task to label the quality and safety of a turn. For example in the\nLamda system (Cohen et al., 2022), a single language model is used in two phases,\nroughly corresponding to generative and discriminative tasks: \ufb01rst generating a re-\nsponse, and then generating a label. In the generative phase, the model is given the\nprior turn and a special RESPONSE token and generates the blue response turn. (In\ntraining, the training loss is given only for the blue response):\n\u201cWhat\u2019s up? RESPONSE Not much.\nIn a second, discriminative phase, the model is \ufb01ne-tuned to see an attribute\n(SENSIBLE, INTERESTING, UNSAFE) and then to predict a 0 or 1 value, again\nwith training losses given only for the blue value.\nWhat\u2019s up? RESPONSE Not much. SENSIBLE 1\nWhat\u2019s up? RESPONSE Not much. INTERESTING 0\nWhat\u2019s up? RESPONSE Not much. UNSAFE 0\nTo use the system in inference, the model \ufb01rst generates a response given the context,\nand then it is given the attribute and asked to generate a rating. The result is a\ngenerated turn along with a label. This label isn\u2019t shown to the user but can be use\nfor \ufb01ltering, either at training time or at deployment time. For example, the system\ncan generate multiple potential responses, \ufb01lter out any response that is unsafe, and\nreturn to the user the highest ranking response. 324 CHAPTER 15 \u2022 C HATBOTS & D IALOGUE SYSTEMS\n15.4.3 Learning to perform retrieval as part of responding\nModern chatbots are also trained to do retrieval (calls to Google or Bing or other\nsearch engines) as a component of generating a response. This is generally done\nby having a sort of fake dialogue participant called something like Search Query\nwhose utterances are passed to a search engine. The system can be encourged to do\nsearches either by prompting or by \ufb01ne-tuning.\nFor example in the Sparrow chatbot (Glaese et al., 2022), the prompt includes\na few sample conversations that include two special users, Search Query , and\nSearch Results :\n...[this is preceded by other parts of the prompt]\nUser: What color is the sky?\nSparrow: Blue.\nUser: Why?\nSearch Query: why is sky blue\nSearch Results: Page title: Sky Except for direct sunlight, most of the light in\nthe daytime sky is caused by scattering, which is dominated by\na small-particle limit called Rayleigh scattering. The scattering\ndue to molecule-sized particles (as in air) is greater in the di-\nrections both toward and away from the source of light than it\nis in directions perpendicular to the incident path. Scattering is\nsigni\ufb01cant for light at all visible wavelengths, but is stronger at\nthe shorter (bluer) end of the visible spectrum, meaning that the\nscattered light is bluer than its source: the Sun. The remain-\ning direct sunlight, having lost some of its shorter-wavelength\ncomponents, appears slightly less blue.\nSparrow: Because of the Rayleigh scattering. It causes short waves of\nblue light to be scattered more than the other colours in the spec-\ntrum.\nUser: Who was Raleigh?\nSearch Query: who was Rayleigh\n... ...\nFrom these prompts, the system learns to generate texts with Search Query\nturns for fact-based questions from the user, and these are passed to a search engine\nto generate the Search Results turns.\nAlternatively, systems can be \ufb01netuned to to know when to use a search en-\ngine. For example, labelers can interact with a system, fact check each of the re-\nsponses, and whenever the system emits an incorrect response, perform the web\nsearch queries that the system should have used to check its answer, and then the in-\nteration is recorded and used for \ufb01ne-tuning. Or labelers can look at a transcript of a\nlanguage model carrying on a dialogue, and similarly mark every place where a fact\nwas wrong (or out-of-date) and write the set of search queries that would have been\nappropriate. A system is then \ufb01ne-tuned to generate search query turns which\nare again passed to a search engine to generate the search responses . The set\nof pages or snippets returned by the search engine in the search response turn are\nthen treated as the context for generation, similarly to the retrieval-based question-\nanswering methods of Chapter 14. 15.5 \u2022 D IALOGUE SYSTEM DESIGN 325\n15.4.4 RLHF\nA more sophisticated family of methods uses reinforcement learning to learn to\nmatch human preferences for generated turns. In this method, RLHF for Rein- RLHF\nforcement Learning from Human Feedback, we give a system a dialogue context\nand sample two possible turns from the language model. We then have humans la-\nbel which of the two is better, creating a large dataset of sentence pairs with human\npreferences. These pairs are used to train a dialogue policy, and reinforcement learn-\ning is used to train the language model to generate turns that have higher rewards\n(Christiano et al., 2017; Ouyang et al., 2022). While using RLHF is the current state\nof the art at the time of this writing, a number of alternatives have been recently\ndeveloped that don\u2019t require reinforcement learning (Rafailov et al., 2023, e.g.,) and\nso this aspect of the \ufb01eld is changing very quickly.\n15.4.5 Evaluating Chatbots\nChatbots are evaluated by humans, who assign a score. This can be the human who\ntalked to the chatbot ( participant evaluation ) or a third party who reads a transcript\nof a human/chatbot conversation ( observer evaluation ). In the participant evalua-\ntion of See et al. (2019), the human evaluator chats with the model for six turns and\nrates the chatbot on 8 dimensions capturing conversational quality: avoiding repe-\ntition, interestingness, making sense, \ufb02uency, listening, inquisitiveness, humanness\nand engagingness on Likert scales like these:\nEngagingness How much did you enjoy talking to this user?\n\u000fNot at all\u000fA little\u000fSomewhat\u000fA lot\nMaking sense How often did this user say something which did NOT make sense?\n\u000fNever made any sense \u000fMost responses didn\u2019t make sense \u000fSome re-\nsponses didn\u2019t make sense \u000fEverything made perfect sense\nObserver evaluations use third party annotators to look at the text of a complete\nconversation. Sometimes we\u2019re interested in having raters assign a score to each\nsystem turn; for example (Artstein et al., 2009) have raters mark how coherent each\nturn is. Often, however, we just want a single high-level score to know if system A\nis better than system B. The acute-eval metric (Li et al., 2019a) is such an observer acute-eval\nevaluation in which annotators look at two separate human-computer conversations\nand choose the system which performed better on four metrics: engagingness, inter-\nestingness, humanness, and knowledgability.\n15.5 Dialogue System Design\nBecause of the important role of the user, the \ufb01eld of dialogue systems is closely\nlinked with Human-Computer Interaction (HCI). This is especially true for task-\noriented dialogue and assistants, where the design of dialogue strategies, sometimes\ncalled voice user interface design, generally follows user-centered design princi-voice user\ninterface\nples (Gould and Lewis, 1985):\n1. Study the user and task: Understand the users and the task by interviewing\nusers, investigating similar systems, and studying related human-human dialogues.\n2. Build simulations and prototypes: A crucial tool in building dialogue systems\nis the Wizard-of-Oz system . In wizard systems, the users interact with what theyWizard-of-Oz\nsystem 326 CHAPTER 15 \u2022 C HATBOTS & D IALOGUE SYSTEMS\nthink is a program but is in fact a human \u201cwizard\u201d disguised by a software interface\n(Gould et al. 1983, Good et al. 1984, Fraser and\nGilbert 1991). The name comes from the chil-\ndren\u2019s book The Wizard of Oz (Baum, 1900),\nin which the wizard turned out to be a simu-\nlation controlled by a man behind a curtain or\nscreen. A wizard system can be used to test out\nan architecture before implementation; only the\ninterface software and databases need to be in\nplace. The wizard gets input from the user, uses\na database interface to run queries based on the\nuser utterance, and then outputs sentences, ei-\nther by typing them or speaking them.\nWizard-of-Oz systems are not a perfect\nsimulation, since the wizard doesn\u2019t exactly\nsimulate the errors or limitations of a real sys-\ntem; but wizard studies can still provide a useful \ufb01rst idea of the domain issues.\n3. Iteratively test the design on users: An iterative design cycle with embedded\nuser testing is essential in system design (Nielsen 1992, Cole et al. 1997, Yankelovich\net al. 1995, Landauer 1995). For example in a well-known incident, an early dia-\nlogue system required the user to press a key to interrupt the system (Stifelman et al.,\n1993). But user testing showed users barged in (interrupted, talking over the sys- barged in\ntem), which led to a redesign of the system to recognize overlapped speech. It\u2019s also\nimportant to incorporate value sensitive design , in which we carefully consider dur-value sensitive\ndesign\ning the design process the bene\ufb01ts, harms and possible stakeholders of the resulting\nsystem (Friedman et al. 2017, Friedman and Hendry 2019).\n15.5.1 Ethical Issues in Dialogue System Design\nEthical issues have been key to how we think about designing arti\ufb01cial agents since\nwell before we had dialogue systems. Mary Shelley (depicted below) centered her\nnovel Frankenstein around the problem of creating arti\ufb01cial agents without consider-\ning\nethical and humanistic concerns. One issue is the\nsafety of users. If users seek information from di-\nalogue systems in safety-critical situations like ask-\ning medical advice, or in emergency situations, or\nwhen indicating the intentions of self-harm, incorrect\nadvice can be dangerous and even life-threatening.\nFor example (Bickmore et al., 2018) gave participants\nmedical problems to pose to three commercial di-\nalogue systems (Siri, Alexa, Google Assistant) and\nasked them to determine an action to take based on\nthe system responses; many of the proposed actions,\nif actually taken, would have led to harm or death.\nA system can also harm users by verbally attacking them, or creating represen-\ntational harms (Blodgett et al., 2020) by generating abusive or harmful stereotypes\nthat demean particular groups of people. Both abuse and stereotypes can cause psy-\nchological harm to users. Microsoft\u2019s 2016 Tay chatbot, for example, was taken Tay\nof\ufb02ine 16 hours after it went live, when it began posting messages with racial slurs, 15.6 \u2022 S UMMARY 327\nconspiracy theories, and personal attacks on its users. Tay had learned these biases\nand actions from its training data, including from users who seemed to be purposely\nteaching the system to repeat this kind of language (Neff and Nagy 2016). Hender-\nson et al. (2017) examined dialogue datasets used to train corpus-based chatbots and\nfound toxic and abusive language, especially in social media corpora like Twitter\nand Reddit, and indeed such language then appears in the text generated by lan-\nguage models and dialogue systems (Gehman et al. 2020; Xu et al. 2020) which\ncan even amplify the bias from the training data (Dinan et al., 2020). Liu et al.\n(2020) developed another method for investigating bias, testing how neural dialogue\nsystems responded to pairs of simulated user turns that are identical except for men-\ntioning different genders or race. They found, for example, that simple changes like\nusing the word \u2018she\u2019 instead of \u2018he\u2019 in a sentence caused systems to respond more\noffensively and with more negative sentiment.\nAnother important ethical issue is privacy . Already in the \ufb01rst days of ELIZA,\nWeizenbaum pointed out the privacy implications of people\u2019s revelations to the chat-\nbot. The ubiquity of in-home dialogue systems means they may often overhear\nprivate information (Henderson et al., 2017). If a chatbot is human-like, users are\nalso more likely to disclose private information, and less likely to worry about the\nharm of this disclosure (Ischen et al., 2019). In general, chatbots that are trained\non transcripts of human-human or human-machine conversation must anonymize\npersonally identi\ufb01able information.\nFinally, chatbots raise important issues of gender equality in addition to textual\nbias. Current chatbots are overwhelmingly given female names, likely perpetuating\nthe stereotype of a subservient female servant (Paolino, 2017). And when users\nuse sexually harassing language, most commercial chatbots evade or give positive\nresponses rather than responding in clear negative ways (Fessler, 2017).\nThese ethical issues are an important area of investigation, including \ufb01nding\nways to mitigate problems of abuse and toxicity, like detecting and responding ap-\npropriately to toxic contexts (Wolf et al. 2017, Dinan et al. 2020, Xu et al. 2020).\nValue sensitive design, carefully considering possible harms in advance (Friedman\net al. 2017, Friedman and Hendry 2019) is also important; (Dinan et al., 2021) give\na number of suggestions for best practices in dialogue system design. For exam-\nple getting informed consent from participants, whether they are used for training,\nor whether they are interacting with a deployed system is important. Because di-\nalogue systems by de\ufb01nition involve human participants, researchers also work on\nthese issues with the Institutional Review Boards ( IRB) at their institutions, who IRB\nhelp protect the safety of experimental subjects.\n15.6 Summary\nChatbots anddialogue systems are crucial speech and language processing appli-\ncations that are already widely used commercially.\n\u2022 In human dialogue, speaking is a kind of action; these acts are referred to\nas speech acts or dialogue acts. Speakers also attempt to achieve common\nground by acknowledging that they have understand each other. Conversation\nalso is characterized by turn structure and dialogue structure.\n\u2022 Chatbots are conversational systems designed to mimic the appearance of in-\nformal human conversation. Rule-based chatbots like ELIZA and its modern 328 CHAPTER 15 \u2022 C HATBOTS & D IALOGUE SYSTEMS\ndescendants use rules to map user sentences into system responses. Corpus-\nbased chatbots mine logs of human conversation to learn to automatically map\nuser sentences into system responses.\n\u2022 For task-based dialogue, most commercial dialogue systems use the GUS or\nframe-based architecture, in which the designer speci\ufb01es frames consisting of\nslots that the system must \ufb01ll by asking the user.\n\u2022 The dialogue-state architecture augments the GUS frame-and-slot architec-\nture with richer representations and more sophisticated algorithms for keeping\ntrack of user\u2019s dialogue acts, policies for generating its own dialogue acts, and\na natural language component.\n\u2022 Dialogue systems are a kind of human-computer interaction, and general HCI\nprinciples apply in their design, including the role of the user, simulations such\nas Wizard-of-Oz systems, and the importance of iterative design and testing\non real users.\nBibliographical and Historical Notes\nThe linguistic, philosophical, and psychological literature on dialogue is quite ex-\ntensive. For example the idea that utterances in a conversation are a kind of action\nbeing performed by the speaker was due originally to the philosopher Wittgenstein\n(1953) but worked out more fully by Austin (1962) and his student John Searle.\nVarious sets of speech acts have been de\ufb01ned over the years, and a rich linguistic\nand philosophical literature developed, especially focused on explaining the use of\nindirect speech acts. The idea of dialogue acts draws also from a number of other\nsources, including the ideas of adjacency pairs, pre-sequences, and other aspects of\nthe interactional properties of human conversation developed in the \ufb01eld of conver-\nsation analysis (see Levinson (1983) for an introduction to the \ufb01eld). This idea thatconversation\nanalysis\nacts set up strong local dialogue expectations was also pre\ufb01gured by Firth (1935, p.\n70), in a famous quotation:\nMost of the give-and-take of conversation in our everyday life is stereotyped\nand very narrowly conditioned by our particular type of culture. It is a sort\nof roughly prescribed social ritual, in which you generally say what the other\nfellow expects you, one way or the other, to say.\nAnother important research thread modeled dialogue as a kind of collaborative\nbehavior, including the ideas of common ground (Clark and Marshall, 1981), ref-\nerence as a collaborative process (Clark and Wilkes-Gibbs, 1986), joint intention\n(Levesque et al., 1990), and shared plans (Grosz and Sidner, 1980).\nThe earliest conversational systems were simple pattern-action chatbots like ELIZA\n(Weizenbaum, 1966). ELIZA had a widespread in\ufb02uence on popular perceptions of\narti\ufb01cial intelligence, and brought up some of the \ufb01rst ethical questions in natural\nlanguage processing \u2014such as the issues of privacy we discussed above as well the\nrole of algorithms in decision-making\u2014 leading its creator Joseph Weizenbaum to\n\ufb01ght for social responsibility in AI and computer science in general.\nComputational-implemented theories of dialogue blossomed in the 1970. That\nperiod saw the very in\ufb02uential GUS system (Bobrow et al., 1977), which in the late\n1970s established the frame-based paradigm that became the dominant industrial\nparadigm for dialogue systems for over 30 years. BIBLIOGRAPHICAL AND HISTORICAL NOTES 329\nAnother in\ufb02uential line of research from that decade focused on modeling the hi-\nerarchical structure of dialogue. Grosz\u2019s pioneering 1977b dissertation \ufb01rst showed\nthat \u201ctask-oriented dialogues have a structure that closely parallels the structure of\nthe task being performed\u201d (p. 27), leading to her work with Sidner and others show-\ning how to use similar notions of intention and plans to model discourse structure\nand coherence in dialogue. See, e.g., Lochbaum et al. (2000) for a summary of the\nrole of intentional structure in dialogue.\nYet a third line, \ufb01rst suggested by Bruce (1975), suggested that since speech acts\nare actions, they should be planned like other actions, and drew on the AI planning\nliterature (Fikes and Nilsson, 1971). A system seeking to \ufb01nd out some information\ncan come up with the plan of asking the interlocutor for the information. A system\nhearing an utterance can interpret a speech act by running the planner \u201cin reverse\u201d,\nusing inference rules to infer from what the interlocutor said what the plan might\nhave been. Plan-based models of dialogue are referred to as BDI models because BDI\nsuch planners model the beliefs ,desires , and intentions (BDI) of the system and in-\nterlocutor. BDI models of dialogue were \ufb01rst introduced by Allen, Cohen, Perrault,\nand their colleagues in a number of in\ufb02uential papers showing how speech acts could\nbe generated (Cohen and Perrault, 1979) and interpreted (Perrault and Allen 1980,\nAllen and Perrault 1980). At the same time, Wilensky (1983) introduced plan-based\nmodels of understanding as part of the task of interpreting stories.\nIn the 1990s, machine learning models that had \ufb01rst been applied to natural\nlanguage processing began to be applied to dialogue tasks like slot \ufb01lling (Miller\net al. 1994, Pieraccini et al. 1991). This period also saw lots of analytic work on the\nlinguistic properties of dialogue acts and on machine-learning-based methods for\ntheir detection. (Sag and Liberman 1975, Hinkelman and Allen 1989, Nagata and\nMorimoto 1994, Goodwin 1996, Chu-Carroll 1998, Shriberg et al. 1998, Stolcke\net al. 2000, Gravano et al. 2012. This work strongly informed the development\nof the dialogue-state model (Larsson and Traum, 2000). Dialogue state tracking\nquickly became an important problem for task-oriented dialogue, and there has been\nan in\ufb02uential annual evaluation of state-tracking algorithms (Williams et al., 2016).\nThe turn of the century saw a line of work on applying reinforcement learning\nto dialogue, which \ufb01rst came out of AT&T and Bell Laboratories with work on\nMDP dialogue systems (Walker 2000, Levin et al. 2000, Singh et al. 2002) along\nwith work on cue phrases, prosody, and rejection and con\ufb01rmation. Reinforcement\nlearning research turned quickly to the more sophisticated POMDP models (Roy\net al. 2000, Lemon et al. 2006, Williams and Young 2007) applied to small slot-\n\ufb01lling dialogue tasks. Neural reinforcement learning models have been used both for\nchatbot systems, for example simulating dialogues between two dialogue systems,\nrewarding good conversational properties like coherence and ease of answering (Li\net al., 2016a), and for task-oriented dialogue (Williams et al., 2017).\nBy around 2010 the GUS architecture \ufb01nally began to be widely used commer-\ncially in dialogue systems on phones like Apple\u2019s SIRI (Bellegarda, 2013) and other\ndigital assistants.\nThe rise of the web gave rise to corpus-based chatbot architectures around the\nturn of the century, \ufb01rst using information retrieval models and then in the 2010s,\nafter the rise of deep learning, with sequence-to-sequence models.\n[TBD: Modern history of neural chatbots]\nOther important dialogue areas include the study of affect in dialogue (Rashkin\net al. 2019, Lin et al. 2019) and conversational interface design (Cohen et al. 2004,\nHarris 2005, Pearl 2017, Deibel and Evanhoe 2021). 330 CHAPTER 15 \u2022 C HATBOTS & D IALOGUE SYSTEMS\nExercises\n15.1 Write a \ufb01nite-state automaton for a dialogue manager for checking your bank\nbalance and withdrawing money at an automated teller machine.\n15.2 Adispreferred response is a response that has the potential to make a persondispreferred\nresponse\nuncomfortable or embarrassed in the conversational context; the most com-\nmon example dispreferred responses is turning down a request. People signal\ntheir discomfort with having to say no with surface cues (like the word well),\nor via signi\ufb01cant silence. Try to notice the next time you or someone else\nutters a dispreferred response, and write down the utterance. What are some\nother cues in the response that a system might use to detect a dispreferred\nresponse? Consider non-verbal cues like eye gaze and body gestures.\n15.3 When asked a question to which they aren\u2019t sure they know the answer, peo-\nple display their lack of con\ufb01dence by cues that resemble other dispreferred\nresponses. Try to notice some unsure answers to questions. What are some\nof the cues? If you have trouble doing this, read Smith and Clark (1993) and\nlisten speci\ufb01cally for the cues they mention.\n15.4 Implement a small air-travel help system based on text input. Your system\nshould get constraints from users about a particular \ufb02ight that they want to\ntake, expressed in natural language, and display possible \ufb02ights on a screen.\nMake simplifying assumptions. You may build in a simple \ufb02ight database or\nyou may use a \ufb02ight information system on the Web as your backend. CHAPTER\n16Automatic Speech Recognition\nand Text-to-Speech\nI KNOW not whether\nI see your meaning: if I do, it lies\nUpon the wordy wavelets of your voice,\nDim as an evening shadow in a brook,\nThomas Lovell Beddoes, 1851\nUnderstanding spoken language, or at least transcribing the words into writing, is\none of the earliest goals of computer language processing. In fact, speech processing\npredates the computer by many decades!\nThe \ufb01rst machine that recognized speech\nwas a toy from the 1920s. \u201cRadio Rex\u201d,\nshown to the right, was a celluloid dog\nthat moved (by means of a spring) when\nthe spring was released by 500 Hz acous-\ntic energy. Since 500 Hz is roughly the\n\ufb01rst formant of the vowel [eh] in \u201cRex\u201d,\nRex seemed to come when he was called\n(David, Jr. and Selfridge, 1962).\nIn modern times, we expect more of our automatic systems. The task of auto-\nmatic speech recognition (ASR ) is to map any waveform like this: ASR\nto the appropriate string of words:\nIt's time for lunch!\nAutomatic transcription of speech by any speaker in any environment is still far from\nsolved, but ASR technology has matured to the point where it is now viable for many\npractical tasks. Speech is a natural interface for communicating with smart home ap-\npliances, personal assistants, or cellphones, where keyboards are less convenient, in\ntelephony applications like call-routing (\u201cAccounting, please\u201d) or in sophisticated\ndialogue applications (\u201cI\u2019d like to change the return date of my \ufb02ight\u201d). ASR is also\nuseful for general transcription, for example for automatically generating captions\nfor audio or video text (transcribing movies or videos or live discussions). Transcrip-\ntion is important in \ufb01elds like law where dictation plays an important role. Finally,\nASR is important as part of augmentative communication (interaction between com-\nputers and humans with some disability resulting in dif\ufb01culties or inabilities in typ-\ning or audition). The blind Milton famously dictated Paradise Lost to his daughters,\nand Henry James dictated his later novels after a repetitive stress injury.\nWhat about the opposite problem, going from text to speech? This is a problem\nwith an even longer history. In Vienna in 1769, Wolfgang von Kempelen built for 332 CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\nthe Empress Maria Theresa the famous Mechanical Turk, a chess-playing automaton\nconsisting of a wooden box \ufb01lled with gears, behind which sat a robot mannequin\nwho played chess by moving pieces with his mechanical arm. The Turk toured Eu-\nrope and the Americas for decades, defeating Napoleon Bonaparte and even playing\nCharles Babbage. The Mechanical Turk might have been one of the early successes\nof arti\ufb01cial intelligence were it not for the fact that it was, alas, a hoax, powered by\na human chess player hidden inside the box.\nWhat is less well known is that von Kempelen, an extraordinarily\nproli\ufb01c inventor, also built between\n1769 and 1790 what was de\ufb01nitely\nnot a hoax: the \ufb01rst full-sentence\nspeech synthesizer, shown partially to\nthe right. His device consisted of a\nbellows to simulate the lungs, a rub-\nber mouthpiece and a nose aperture, a\nreed to simulate the vocal folds, var-\nious whistles for the fricatives, and a\nsmall auxiliary bellows to provide the puff of air for plosives. By moving levers\nwith both hands to open and close apertures, and adjusting the \ufb02exible leather \u201cvo-\ncal tract\u201d, an operator could produce different consonants and vowels.\nMore than two centuries later, we no longer build our synthesizers out of wood\nand leather, nor do we need human operators. The modern task of speech synthesis ,speech\nsynthesis\nalso called text-to-speech orTTS , is exactly the reverse of ASR; to map text: text-to-speech\nTTSIt's time for lunch!\nto an acoustic waveform:\nModern speech synthesis has a wide variety of applications. TTS is used in\nconversational agents that conduct dialogues with people, plays a role in devices\nthat read out loud for the blind or in games, and can be used to speak for sufferers\nof neurological disorders, such as the late astrophysicist Steven Hawking who, after\nhe lost the use of his voice because of ALS, spoke by manipulating a TTS system.\nIn the next sections we\u2019ll show how to do ASR with encoder-decoders, intro-\nduce the CTC loss functions, the standard word error rate evaluation metric, and\ndescribe how acoustic features are extracted. We\u2019ll then see how TTS can be mod-\neled with almost the same algorithm in reverse, and conclude with a brief mention\nof other speech tasks.\n16.1 The Automatic Speech Recognition Task\nBefore describing algorithms for ASR, let\u2019s talk about how the task itself varies.\nOne dimension of variation is vocabulary size. Some ASR tasks can be solved with\nextremely high accuracy, like those with a 2-word vocabulary ( yesversus no) or\nan 11 word vocabulary like digit recognition (recognizing sequences of digits in-digit\nrecognition\ncluding zero tonine plus oh). Open-ended tasks like transcribing videos or human\nconversations, with large vocabularies of up to 60,000 words, are much harder. 16.1 \u2022 T HEAUTOMATIC SPEECH RECOGNITION TASK 333\nA second dimension of variation is who the speaker is talking to. Humans speak-\ning to machines (either dictating or talking to a dialogue system) are easier to recog-\nnize than humans speaking to humans. Read speech , in which humans are reading read speech\nout loud, for example in audio books, is also relatively easy to recognize. Recog-\nnizing the speech of two humans talking to each other in conversational speech ,conversational\nspeech\nfor example, for transcribing a business meeting, is the hardest. It seems that when\nhumans talk to machines, or read without an audience present, they simplify their\nspeech quite a bit, talking more slowly and more clearly.\nA third dimension of variation is channel and noise. Speech is easier to recognize\nif it\u2019s recorded in a quiet room with head-mounted microphones than if it\u2019s recorded\nby a distant microphone on a noisy city street, or in a car with the window open.\nA \ufb01nal dimension of variation is accent or speaker-class characteristics. Speech\nis easier to recognize if the speaker is speaking the same dialect or variety that the\nsystem was trained on. Speech by speakers of regional or ethnic dialects, or speech\nby children can be quite dif\ufb01cult to recognize if the system is only trained on speak-\ners of standard dialects, or only adult speakers.\nA number of publicly available corpora with human-created transcripts are used\nto create ASR test and training sets to explore this variation; we mention a few of\nthem here since you will encounter them in the literature. LibriSpeech is a large LibriSpeech\nopen-source read-speech 16 kHz dataset with over 1000 hours of audio books from\nthe LibriV ox project, with transcripts aligned at the sentence level (Panayotov et al.,\n2015). It is divided into an easier (\u201cclean\u201d) and a more dif\ufb01cult portion (\u201cother\u201d)\nwith the clean portion of higher recording quality and with accents closer to US\nEnglish. This was done by running a speech recognizer (trained on read speech from\nthe Wall Street Journal) on all the audio, computing the WER for each speaker based\non the gold transcripts, and dividing the speakers roughly in half, with recordings\nfrom lower-WER speakers called \u201cclean\u201d and recordings from higher-WER speakers\n\u201cother\u201d.\nTheSwitchboard corpus of prompted telephone conversations between strangers Switchboard\nwas collected in the early 1990s; it contains 2430 conversations averaging 6 min-\nutes each, totaling 240 hours of 8 kHz speech and about 3 million words (Godfrey\net al., 1992). Switchboard has the singular advantage of an enormous amount of\nauxiliary hand-done linguistic labeling, including parses, dialogue act tags, phonetic\nand prosodic labeling, and discourse and information structure. The CALLHOME CALLHOME\ncorpus was collected in the late 1990s and consists of 120 unscripted 30-minute\ntelephone conversations between native speakers of English who were usually close\nfriends or family (Canavan et al., 1997).\nThe Santa Barbara Corpus of Spoken American English (Du Bois et al., 2005) is\na large corpus of naturally occurring everyday spoken interactions from all over the\nUnited States, mostly face-to-face conversation, but also town-hall meetings, food\npreparation, on-the-job talk, and classroom lectures. The corpus was anonymized by\nremoving personal names and other identifying information (replaced by pseudonyms\nin the transcripts, and masked in the audio).\nCORAAL is a collection of over 150 sociolinguistic interviews with African CORAAL\nAmerican speakers, with the goal of studying African American Language ( AAL ),\nthe many variations of language used in African American communities (Kendall\nand Farrington, 2020). The interviews are anonymized with transcripts aligned at\nthe utterance level. The CHiME Challenge is a series of dif\ufb01cult shared tasks with CHiME\ncorpora that deal with robustness in ASR. The CHiME 5 task, for example, is ASR of\nconversational speech in real home environments (speci\ufb01cally dinner parties). The 334 CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\ncorpus contains recordings of twenty different dinner parties in real homes, each\nwith four participants, and in three locations (kitchen, dining area, living room),\nrecorded both with distant room microphones and with body-worn mikes. The\nHKUST Mandarin Telephone Speech corpus has 1206 ten-minute telephone con- HKUST\nversations between speakers of Mandarin across China, including transcripts of the\nconversations, which are between either friends or strangers (Liu et al., 2006). The\nAISHELL-1 corpus contains 170 hours of Mandarin read speech of sentences taken AISHELL-1\nfrom various domains, read by different speakers mainly from northern China (Bu\net al., 2017).\nFigure 16.1 shows the rough percentage of incorrect words (the word error rate ,\nor WER, de\ufb01ned on page 346) from state-of-the-art systems on some of these tasks.\nNote that the error rate on read speech (like the LibriSpeech audiobook corpus) is\naround 2%; this is a solved task, although these numbers come from systems that re-\nquire enormous computational resources. By contrast, the error rate for transcribing\nconversations between humans is much higher; 5.8 to 11% for the Switchboard and\nCALLHOME corpora. The error rate is higher yet again for speakers of varieties\nlike African American Vernacular English, and yet again for dif\ufb01cult conversational\ntasks like transcription of 4-speaker dinner party speech, which can have error rates\nas high as 81.3%. Character error rates (CER) are also much lower for read Man-\ndarin speech than for natural conversation.\nEnglish Tasks WER %\nLibriSpeech audiobooks 960hour clean 1.4\nLibriSpeech audiobooks 960hour other 2.6\nSwitchboard telephone conversations between strangers 5.8\nCALLHOME telephone conversations between family 11.0\nSociolinguistic interviews, CORAAL (AAL) 27.0\nCHiMe5 dinner parties with body-worn microphones 47.9\nCHiMe5 dinner parties with distant microphones 81.3\nChinese (Mandarin) Tasks CER %\nAISHELL-1 Mandarin read speech corpus 6.7\nHKUST Mandarin Chinese telephone conversations 23.5\nFigure 16.1 Rough Word Error Rates (WER = % of words misrecognized) reported around\n2020 for ASR on various American English recognition tasks, and character error rates (CER)\nfor two Chinese recognition tasks.\n16.2 Feature Extraction for ASR: Log Mel Spectrum\nThe \ufb01rst step in ASR is to transform the input waveform into a sequence of acoustic\nfeature vectors , each vector representing the information in a small time window feature vector\nof the signal. Let\u2019s see how to convert a raw wave\ufb01le to the most commonly used\nfeatures, sequences of log mel spectrum vectors. A speech signal processing course\nis recommended for more details.\n16.2.1 Sampling and Quantization\nThe input to a speech recognizer is a complex series of changes in air pressure.\nThese changes in air pressure obviously originate with the speaker and are caused 16.2 \u2022 F EATURE EXTRACTION FOR ASR: L OGMELSPECTRUM 335\nby the speci\ufb01c way that air passes through the glottis and out the oral or nasal cav-\nities. We represent sound waves by plotting the change in air pressure over time.\nOne metaphor which sometimes helps in understanding these graphs is that of a ver-\ntical plate blocking the air pressure waves (perhaps in a microphone in front of a\nspeaker\u2019s mouth, or the eardrum in a hearer\u2019s ear). The graph measures the amount\nofcompression orrarefaction (uncompression) of the air molecules at this plate.\nFigure 16.2 shows a short segment of a waveform taken from the Switchboard corpus\nof telephone speech of the vowel [iy] from someone saying \u201cshe just had a baby\u201d.\nTime (s)0 0.03875\u20130.016970.02283\n0\nFigure 16.2 A waveform of an instance of the vowel [iy] (the last vowel in the word \u201cbaby\u201d). The y-axis\nshows the level of air pressure above and below normal atmospheric pressure. The x-axis shows time. Notice\nthat the wave repeats regularly.\nThe \ufb01rst step in digitizing a sound wave like Fig. 16.2 is to convert the analog\nrepresentations (\ufb01rst air pressure and then analog electric signals in a microphone)\ninto a digital signal. This analog-to-digital conversion has two steps: sampling and sampling\nquantization . To sample a signal, we measure its amplitude at a particular time; the\nsampling rate is the number of samples taken per second. To accurately measure a\nwave, we must have at least two samples in each cycle: one measuring the positive\npart of the wave and one measuring the negative part. More than two samples per\ncycle increases the amplitude accuracy, but fewer than two samples causes the fre-\nquency of the wave to be completely missed. Thus, the maximum frequency wave\nthat can be measured is one whose frequency is half the sample rate (since every\ncycle needs two samples). This maximum frequency for a given sampling rate is\ncalled the Nyquist frequency . Most information in human speech is in frequenciesNyquist\nfrequency\nbelow 10,000 Hz; thus, a 20,000 Hz sampling rate would be necessary for com-\nplete accuracy. But telephone speech is \ufb01ltered by the switching network, and only\nfrequencies less than 4,000 Hz are transmitted by telephones. Thus, an 8,000 Hz\nsampling rate is suf\ufb01cient for telephone-bandwidth speech like the Switchboard\ncorpus, while 16,000 Hz sampling is often used for microphone speech.\nAlthough using higher sampling rates produces higher ASR accuracy, we can\u2019t\ncombine different sampling rates for training and testing ASR systems. Thus if\nwe are testing on a telephone corpus like Switchboard (8 KHz sampling), we must\ndownsample our training corpus to 8 KHz. Similarly, if we are training on mul-\ntiple corpora and one of them includes telephone speech, we downsample all the\nwideband corpora to 8Khz.\nAmplitude measurements are stored as integers, either 8 bit (values from -128\u2013\n127) or 16 bit (values from -32768\u201332767). This process of representing real-valued\nnumbers as integers is called quantization ; all values that are closer together than quantization\nthe minimum granularity (the quantum size) are represented identically. We refer to\neach sample at time index nin the digitized, quantized waveform as x[n].\nOnce data is quantized, it is stored in various formats. One parameter of these\nformats is the sample rate and sample size discussed above; telephone speech is\noften sampled at 8 kHz and stored as 8-bit samples, and microphone data is often\nsampled at 16 kHz and stored as 16-bit samples. Another parameter is the number of 336 CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\nchannels . For stereo data or for two-party conversations, we can store both channels channel\nin the same \ufb01le or we can store them in separate \ufb01les. A \ufb01nal parameter is individual\nsample storage\u2014linearly or compressed. One common compression format used for\ntelephone speech is m-law (often written u-law but still pronounced mu-law). The\nintuition of log compression algorithms like m-law is that human hearing is more\nsensitive at small intensities than large ones; the log represents small values with\nmore faithfulness at the expense of more error on large values. The linear (unlogged)\nvalues are generally referred to as linear PCM values (PCM stands for pulse code PCM\nmodulation, but never mind that). Here\u2019s the equation for compressing a linear PCM\nsample value xto 8-bit m-law, (where m=255 for 8 bits):\nF(x) =sgn(x)log(1+mjxj)\nlog(1+m)\u00001\u0014x\u00141 (16.1)\nThere are a number of standard \ufb01le formats for storing the resulting digitized wave-\n\ufb01le, such as Microsoft\u2019s .wav and Apple\u2019s AIFF all of which have special headers;\nsimple headerless \u201craw\u201d \ufb01les are also used. For example, the .wav format is a sub-\nset of Microsoft\u2019s RIFF format for multimedia \ufb01les; RIFF is a general format that\ncan represent a series of nested chunks of data and control information. Figure 16.3\nshows a simple .wav \ufb01le with a single data chunk together with its format chunk.\nFigure 16.3 Microsoft wave\ufb01le header format, assuming simple \ufb01le with one chunk. Fol-\nlowing this 44-byte header would be the data chunk.\n16.2.2 Windowing\nFrom the digitized, quantized representation of the waveform, we need to extract\nspectral features from a small window of speech that characterizes part of a par-\nticular phoneme. Inside this small window, we can roughly think of the signal as\nstationary (that is, its statistical properties are constant within this region). (By stationary\ncontrast, in general, speech is a non-stationary signal, meaning that its statistical non-stationary\nproperties are not constant over time). We extract this roughly stationary portion of\nspeech by using a window which is non-zero inside a region and zero elsewhere, run-\nning this window across the speech signal and multiplying it by the input waveform\nto produce a windowed waveform.\nThe speech extracted from each window is called a frame . The windowing is frame\ncharacterized by three parameters: the window size orframe size of the window\n(its width in milliseconds), the frame stride , (also called shift oroffset ) between stride\nsuccessive windows, and the shape of the window.\nTo extract the signal we multiply the value of the signal at time n,s[n]by the\nvalue of the window at time n,w[n]:\ny[n] =w[n]s[n] (16.2)\nThe window shape sketched in Fig. 16.4 is rectangular ; you can see the ex- rectangular\ntracted windowed signal looks just like the original signal. The rectangular window, 16.2 \u2022 F EATURE EXTRACTION FOR ASR: L OGMELSPECTRUM 337\nShift10 msWindow25 msShift10 msWindow25 msWindow25 ms\nFigure 16.4 Windowing, showing a 25 ms rectangular window with a 10ms stride.\nhowever, abruptly cuts off the signal at its boundaries, which creates problems when\nwe do Fourier analysis. For this reason, for acoustic feature creation we more com-\nmonly use the Hamming window, which shrinks the values of the signal toward Hamming\nzero at the window boundaries, avoiding discontinuities. Figure 16.5 shows both;\nthe equations are as follows (assuming a window that is Lframes long):\nrectangular w [n] =\u001a1 0\u0014n\u0014L\u00001\n0 otherwise(16.3)\nHamming w [n] =\u001a\n0:54\u00000:46cos (2pn\nL)0\u0014n\u0014L\u00001\n0 otherwise(16.4)\nTime (s)00.0475896\u20130.50.49990Rectangular windowHamming window\nTime (s)0.004559380.0256563\u20130.48260.49990\nTime (s)0.004559380.0256563\u20130.50.49990\nFigure 16.5 Windowing a sine wave with the rectangular or Hamming windows.\n16.2.3 Discrete Fourier Transform\nThe next step is to extract spectral information for our windowed signal; we need to\nknow how much energy the signal contains at different frequency bands. The tool 338 CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\nfor extracting spectral information for discrete frequency bands for a discrete-time\n(sampled) signal is the discrete Fourier transform orDFT .Discrete\nFourier\ntransformDFT The input to the DFT is a windowed signal x[n]:::x[m], and the output, for each\nofNdiscrete frequency bands, is a complex number X[k]representing the magni-\ntude and phase of that frequency component in the original signal. If we plot the\nmagnitude against the frequency, we can visualize the spectrum (see Appendix H\nfor more on spectra). For example, Fig. 16.6 shows a 25 ms Hamming-windowed\nportion of a signal and its spectrum as computed by a DFT (with some additional\nsmoothing).\nTime (s)0.0141752 0.039295\u20130.041210.04414\n0\nFrequency (Hz)0 8000Sound pressure level (dB /Hz)\n\u201320020\n(a) (b)\nFigure 16.6 (a) A 25 ms Hamming-windowed portion of a signal from the vowel [iy]\nand (b) its spectrum computed by a DFT.\nWe do not introduce the mathematical details of the DFT here, except to note\nthat Fourier analysis relies on Euler\u2019s formula , with jas the imaginary unit: Euler\u2019s formula\nejq=cosq+jsinq (16.5)\nAs a brief reminder for those students who have already studied signal processing,\nthe DFT is de\ufb01ned as follows:\nX[k] =N\u00001X\nn=0x[n]e\u0000j2p\nNkn(16.6)\nA commonly used algorithm for computing the DFT is the fast Fourier transformfast Fourier\ntransform\norFFT . This implementation of the DFT is very ef\ufb01cient but only works for values FFT\nofNthat are powers of 2.\n16.2.4 Mel Filter Bank and Log\nThe results of the FFT tell us the energy at each frequency band. Human hearing,\nhowever, is not equally sensitive at all frequency bands; it is less sensitive at higher\nfrequencies. This bias toward low frequencies helps human recognition, since in-\nformation in low frequencies (like formants) is crucial for distinguishing vowels or\nnasals, while information in high frequencies (like stop bursts or fricative noise) is\nless crucial for successful recognition. Modeling this human perceptual property\nimproves speech recognition performance in the same way.\nWe implement this intuition by collecting energies, not equally at each frequency\nband, but according to the melscale, an auditory frequency scale. A mel(Stevens mel\net al. 1937, Stevens and V olkmann 1940) is a unit of pitch. Pairs of sounds that are\nperceptually equidistant in pitch are separated by an equal number of mels. The mel 16.3 \u2022 S PEECH RECOGNITION ARCHITECTURE 339\nfrequency mcan be computed from the raw acoustic frequency by a log transforma-\ntion:\nmel(f) =1127ln (1+f\n700) (16.7)\nWe implement this intuition by creating a bank of \ufb01lters that collect energy from\neach frequency band, spread logarithmically so that we have very \ufb01ne resolution\nat low frequencies, and less resolution at high frequencies. Figure 16.7 shows a\nsample bank of triangular \ufb01lters that implement this idea, that can be multiplied by\nthe spectrum to get a mel spectrum.\nm1m2mM...mel spectrum0770000.51AmplitudeFrequency (Hz)8K\nFigure 16.7 The mel \ufb01lter bank (Davis and Mermelstein, 1980). Each triangular \ufb01lter,\nspaced logarithmically along the mel scale, collects energy from a given frequency range.\nFinally, we take the log of each of the mel spectrum values. The human response\nto signal level is logarithmic (like the human response to frequency). Humans are\nless sensitive to slight differences in amplitude at high amplitudes than at low ampli-\ntudes. In addition, using a log makes the feature estimates less sensitive to variations\nin input such as power variations due to the speaker\u2019s mouth moving closer or further\nfrom the microphone.\n16.3 Speech Recognition Architecture\nThe basic architecture for ASR is the encoder-decoder (implemented with either\nRNNs or Transformers), exactly the same architecture introduced for MT in Chap-\nter 13. Generally we start from the log mel spectral features described in the previous\nsection, and map to letters, although it\u2019s also possible to map to induced morpheme-\nlike chunks like wordpieces or BPE.\nFig. 16.8 sketches the standard encoder-decoder architecture, which is com-\nmonly referred to as the attention-based encoder decoder orAED , orlisten attend AED\nand spell (LAS ) after the two papers which \ufb01rst applied it to speech (Chorowskilisten attend\nand spell\net al. 2014, Chan et al. 2016). The input is a sequence of tacoustic feature vectors\nF=f1;f2;:::;ft, one vector per 10 ms frame. The output can be letters or word-\npieces; we\u2019ll assume letters here. Thus the output sequence Y= (hSOSi;y1;:::;ymhEOSi),\nassuming special start of sequence and end of sequence tokens hsosiandheosiand\neach yiis a character; for English we might choose the set:\nyi2fa;b;c;:::;z;0;:::;9;hspacei;hcommai;hperiodi;hapostrophei;hunkig\nOf course the encoder-decoder architecture is particularly appropriate when in-\nput and output sequences have stark length differences, as they do for speech, with\nvery long acoustic feature sequences mapping to much shorter sequences of letters 340 CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\nENCODER\u2026DECODER\u2026\u2026ym\nFeature ComputationSubsampling\u2026Hftf180-dimensional log Mel spectrumper frameShorter sequence Xy1<s>iy2ity3t\u2018y4\u2018sy5s y6 ty7tiy8imy9mex1xn\nFigure 16.8 Schematic architecture for an encoder-decoder speech recognizer.\nor words. A single word might be 5 letters long but, supposing it lasts about 2\nseconds, would take 200 acoustic frames (of 10ms each).\nBecause this length difference is so extreme for speech, encoder-decoder ar-\nchitectures for speech need to have a special compression stage that shortens the\nacoustic feature sequence before the encoder stage. (Alternatively, we can use a loss\nfunction that is designed to deal well with compression, like the CTC loss function\nwe\u2019ll introduce in the next section.)\nThe goal of the subsampling is to produce a shorter sequence X=x1;:::;xnthat\nwill be the input to the encoder. The simplest algorithm is a method sometimes\ncalled low frame rate (Pundak and Sainath, 2016): for time iwe stack (concatenate) low frame rate\nthe acoustic feature vector fiwith the prior two vectors fi\u00001andfi\u00002to make a new\nvector three times longer. Then we simply delete fi\u00001and fi\u00002. Thus instead of\n(say) a 40-dimensional acoustic feature vector every 10 ms, we have a longer vector\n(say 120-dimensional) every 30 ms, with a shorter sequence length n=t\n3.1\nAfter this compression stage, encoder-decoders for speech use the same archi-\ntecture as for MT or other text, composed of either RNNs (LSTMs) or Transformers.\nFor inference, the probability of the output string Yis decomposed as:\np(y1;:::; yn) =nY\ni=1p(yijy1;:::; yi\u00001;X) (16.8)\nWe can produce each letter of the output via greedy decoding:\n\u02c6yi=argmax char2Alphabet P(charjy1:::yi\u00001;X) (16.9)\nAlternatively we can use beam search as described in the next section. This is par-\nticularly relevant when we are adding a language model.\nAdding a language model Since an encoder-decoder model is essentially a con-\nditional language model, encoder-decoders implicitly learn a language model for the\noutput domain of letters from their training data. However, the training data (speech\npaired with text transcriptions) may not include suf\ufb01cient text to train a good lan-\nguage model. After all, it\u2019s easier to \ufb01nd enormous amounts of pure text training\n1There are also more complex alternatives for subsampling, like using a convolutional net that down-\nsamples with max pooling, or layers of pyramidal RNNs , RNNs where each successive layer has half\nthe number of RNNs as the previous layer. 16.4 \u2022 CTC 341\ndata than it is to \ufb01nd text paired with speech. Thus we can can usually improve a\nmodel at least slightly by incorporating a very large language model.\nThe simplest way to do this is to use beam search to get a \ufb01nal beam of hy-\npothesized sentences; this beam is sometimes called an n-best list . We then use a n-best list\nlanguage model to rescore each hypothesis on the beam. The scoring is done by in- rescore\nterpolating the score assigned by the language model with the encoder-decoder score\nused to create the beam, with a weight ltuned on a held-out set. Also, since most\nmodels prefer shorter sentences, ASR systems normally have some way of adding a\nlength factor. One way to do this is to normalize the probability by the number of\ncharacters in the hypothesis jYjc. The following is thus a typical scoring function\n(Chan et al., 2016):\nscore(YjX) =1\njYjclogP(YjX)+llogPLM(Y) (16.10)\n16.3.1 Learning\nEncoder-decoders for speech are trained with the normal cross-entropy loss gener-\nally used for conditional language models. At timestep iof decoding, the loss is the\nlog probability of the correct token (letter) yi:\nLCE=\u0000logp(yijy1;:::; yi\u00001;X) (16.11)\nThe loss for the entire sentence is the sum of these losses:\nLCE=\u0000mX\ni=1logp(yijy1;:::; yi\u00001;X) (16.12)\nThis loss is then backpropagated through the entire end-to-end model to train the\nentire encoder-decoder.\nAs we described in Chapter 13, we normally use teacher forcing, in which the\ndecoder history is forced to be the correct gold yirather than the predicted \u02c6 yi. It\u2019s\nalso possible to use a mixture of the gold and decoder output, for example using\nthe gold output 90% of the time, but with probability .1 taking the decoder output\ninstead:\nLCE=\u0000logp(yijy1;:::; \u02c6yi\u00001;X) (16.13)\n16.4 CTC\nWe pointed out in the previous section that speech recognition has two particular\nproperties that make it very appropriate for the encoder-decoder architecture, where\nthe encoder produces an encoding of the input that the decoder uses attention to\nexplore. First, in speech we have a very long acoustic input sequence Xmapping to\na much shorter sequence of letters Y, and second, it\u2019s hard to know exactly which\npart of Xmaps to which part of Y.\nIn this section we brie\ufb02y introduce an alternative to encoder-decoder: an algo-\nrithm and loss function called CTC , short for Connectionist Temporal Classi\ufb01ca- CTC\ntion(Graves et al., 2006), that deals with these problems in a very different way. The\nintuition of CTC is to output a single character for every frame of the input, so that 342 CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\nthe output is the same length as the input, and then to apply a collapsing function\nthat combines sequences of identical letters, resulting in a shorter sequence.\nLet\u2019s imagine inference on someone saying the word dinner , and let\u2019s suppose\nwe had a function that chooses the most probable letter for each input spectral frame\nrepresentation xi. We\u2019ll call the sequence of letters corresponding to each input\nframe an alignment , because it tells us where in the acoustic signal each letter aligns alignment\nto. Fig. 16.9 shows one such alignment, and what happens if we use a collapsing\nfunction that just removes consecutive duplicate letters.\nX (input)A (alignment)Y (output)dx1ix2ix3nx4nx5nx6nx7ex8rx9rx10rx11rx12rx13rx14dinerwavefile\nFigure 16.9 A naive algorithm for collapsing an alignment between input and letters.\nWell, that doesn\u2019t work; our naive algorithm has transcribed the speech as diner ,\nnotdinner ! Collapsing doesn\u2019t handle double letters. There\u2019s also another problem\nwith our naive function; it doesn\u2019t tell us what symbol to align with silence in the\ninput. We don\u2019t want to be transcribing silence as random letters!\nThe CTC algorithm solves both problems by adding to the transcription alphabet\na special symbol for a blank , which we\u2019ll represent as . The blank can be used in blank\nthe alignment whenever we don\u2019t want to transcribe a letter. Blank can also be used\nbetween letters; since our collapsing function collapses only consecutive duplicate\nletters, it won\u2019t collapse across . More formally, let\u2019s de\ufb01ne the mapping B:a!y\nbetween an alignment aand an output y, which collapses all repeated letters and\nthen removes all blanks. Fig. 16.10 sketches this collapsing function B.\nX (input)A (alignment)remove blanksdx1ix2x3nx4nx5x6nx7ex8rx9rx10rx11rx12x13x14dinernmerge duplicatesdinernY (output)dinern\u2423\u2423\u2423\u2423\u2423\u2423\u2423\nFigure 16.10 The CTC collapsing function B, showing the space blank character ; re-\npeated (consecutive) characters in an alignment Aare removed to form the output Y.\nThe CTC collapsing function is many-to-one; lots of different alignments map\nto the same output string. For example, the alignment shown in Fig. 16.10 is not\nthe only alignment that results in the string dinner . Fig. 16.11 shows some other\nalignments that would produce the same output.\nIt\u2019s useful to think of the set of all alignments that might produce the same output\nY. We\u2019ll use the inverse of our Bfunction, called B\u00001, and represent that set as\nB\u00001(Y). 16.4 \u2022 CTC 343\ndinnneeerrr\u2423\u2423ddinnnerr\u2423\u2423\u2423dddinnnerr\u2423i\u2423\u2423\u2423\u2423\u2423\nFigure 16.11 Three other legitimate alignments producing the transcript dinner .\n16.4.1 CTC Inference\nBefore we see how to compute PCTC(YjX)let\u2019s \ufb01rst see how CTC assigns a proba-\nbility to one particular alignment \u02c6A=f\u02c6a1;:::; \u02c6ang. CTC makes a strong conditional\nindependence assumption: it assumes that, given the input X, the CTC model output\natat time tis independent of the output labels at any other time ai. Thus:\nPCTC(AjX) =TY\nt=1p(atjX) (16.14)\nThus to \ufb01nd the best alignment \u02c6A=f\u02c6a1;:::; \u02c6aTgwe can greedily choose the charac-\nter with the max probability at each time step t:\n\u02c6at=argmax\nc2Cpt(cjX) (16.15)\nWe then pass the resulting sequence Ato the CTC collapsing function Bto get the\noutput sequence Y.\nLet\u2019s talk about how this simple inference algorithm for \ufb01nding the best align-\nment A would be implemented. Because we are making a decision at each time\npoint, we can treat CTC as a sequence-modeling task, where we output one letter\n\u02c6ytat time tcorresponding to each input token xt, eliminating the need for a full de-\ncoder. Fig. 16.12 sketches this architecture, where we take an encoder, produce a\nhidden state htat each timestep, and decode by taking a softmax over the character\nvocabulary at each time step.\nENCODER\u2026yn\nFeature ComputationSubsampling\u2026ftf1 log Mel spectrumShorter inputsequence Xy1iy2iy3iy4tx1xnClassi\ufb01er+softmax\u2026ty5\u2026\u2026output lettersequence Y\nFigure 16.12 Inference with CTC: using an encoder-only model, with decoding done by\nsimple softmaxes over the hidden state htat each output step.\nAlas, there is a potential \ufb02aw with the inference algorithm sketched in (Eq. 16.15)\nand Fig. 16.11. The problem is that we chose the most likely alignment A, but the 344 CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\nmost likely alignment may not correspond to the most likely \ufb01nal collapsed output\nstring Y. That\u2019s because there are many possible alignments that lead to the same\noutput string, and hence the most likely output string might not correspond to the\nmost probable alignment. For example, imagine the most probable alignment Afor\nan input X= [x1x2x3]is the string [a b \u000f] but the next two most probable alignments\nare [b\u000fb] and [\u000fb b]. The output Y=[b b], summing over those two alignments,\nmight be more probable than Y=[a b].\nFor this reason, the most probable output sequence Yis the one that has, not\nthe single best CTC alignment, but the highest sum over the probability of all its\npossible alignments:\nPCTC(YjX) =X\nA2B\u00001(Y)P(AjX)\n=X\nA2B\u00001(Y)TY\nt=1p(atjht)\n\u02c6Y=argmax\nYPCTC(YjX) (16.16)\nAlas, summing over all alignments is very expensive (there are a lot of alignments),\nso we approximate this sum by using a version of Viterbi beam search that cleverly\nkeeps in the beam the high-probability alignments that map to the same output string,\nand sums those as an approximation of (Eq. 16.16). See Hannun (2017) for a clear\nexplanation of this extension of beam search for CTC.\nBecause of the strong conditional independence assumption mentioned earlier\n(that the output at time tis independent of the output at time t\u00001, given the input),\nCTC does not implicitly learn a language model over the data (unlike the attention-\nbased encoder-decoder architectures). It is therefore essential when using CTC to\ninterpolate a language model (and some sort of length factor L(Y)) using interpola-\ntion weights that are trained on a devset:\nscore CTC(YjX) =logPCTC(YjX)+l1logPLM(Y)l2L(Y) (16.17)\n16.4.2 CTC Training\nTo train a CTC-based ASR system, we use negative log-likelihood loss with a special\nCTC loss function. Thus the loss for an entire dataset Dis the sum of the negative\nlog-likelihoods of the correct output Yfor each input X:\nLCTC=X\n(X;Y)2D\u0000logPCTC(YjX) (16.18)\nTo compute CTC loss function for a single input pair (X;Y), we need the probability\nof the output Ygiven the input X. As we saw in Eq. 16.16, to compute the probability\nof a given output Ywe need to sum over all the possible alignments that would\ncollapse to Y. In other words:\nPCTC(YjX) =X\nA2B\u00001(Y)TY\nt=1p(atjht) (16.19)\nNaively summing over all possible alignments is not feasible (there are too many\nalignments). However, we can ef\ufb01ciently compute the sum by using dynamic pro- 16.4 \u2022 CTC 345\ngramming to merge alignments, with a version of the forward-backward algo-\nrithm also used to train HMMs (Appendix A) and CRFs. The original dynamic pro-\ngramming algorithms for both training and inference are laid out in (Graves et al.,\n2006); see (Hannun, 2017) for a detailed explanation of both.\n16.4.3 Combining CTC and Encoder-Decoder\nIt\u2019s also possible to combine the two architectures/loss functions we\u2019ve described,\nthe cross-entropy loss from the encoder-decoder architecture, and the CTC loss.\nFig. 16.13 shows a sketch. For training, we can simply weight the two losses with a\nltuned on a devset:\nL=\u0000llogPencdec(YjX)\u0000(1\u0000l)logPctc(YjX) (16.20)\nFor inference, we can combine the two with the language model (or the length\npenalty), again with learned weights:\n\u02c6Y=argmax\nY[llogPencdec(YjX)\u0000(1\u0000l)logPCTC(YjX)+glogPLM(Y)](16.21)\nENCODER\u2026DECODER\u2026H<s>it\u2018s timx1xn\u2026\u2026i   t   \u2019   s      t   i   m   e  \u2026CTC LossEncoder-Decoder Loss\nFigure 16.13 Combining the CTC and encoder-decoder loss functions.\n16.4.4 Streaming Models: RNN-T for improving CTC\nBecause of the strong independence assumption in CTC (assuming that the output\nat time tis independent of the output at time t\u00001), recognizers based on CTC\ndon\u2019t achieve as high an accuracy as the attention-based encoder-decoder recog-\nnizers. CTC recognizers have the advantage, however, that they can be used for\nstreaming . Streaming means recognizing words on-line rather than waiting until streaming\nthe end of the sentence to recognize them. Streaming is crucial for many applica-\ntions, from commands to dictation, where we want to start recognition while the\nuser is still talking. Algorithms that use attention need to compute the hidden state\nsequence over the entire input \ufb01rst in order to provide the attention distribution con-\ntext, before the decoder can start decoding. By contrast, a CTC algorithm can input\nletters from left to right immediately.\nIf we want to do streaming, we need a way to improve CTC recognition to re-\nmove the conditional independent assumption, enabling it to know about output his-\ntory. The RNN-Transducer ( RNN-T ), shown in Fig. 16.14, is just such a model RNN-T\n(Graves 2012, Graves et al. 2013). The RNN-T has two main components: a CTC\nacoustic model, and a separate language model component called the predictor that\nconditions on the output token history. At each time step t, the CTC encoder outputs 346 CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\na hidden state henc\ntgiven the input x1:::xt. The language model predictor takes as in-\nput the previous output token (not counting blanks), outputting a hidden state hpred\nu.\nThe two are passed through another network whose output is then passed through a\nsoftmax to predict the next character.\nPRNN\u0000T(YjX) =X\nA2B\u00001(Y)P(AjX)\n=X\nA2B\u00001(Y)TY\nt=1p(atjht;y<ut)\nENCODERP ( yt,u | x[1..t] , y[1..u-1] )\nxtPREDICTIONNETWORKyu-1JOINT NETWORKhencthpreduSOFTMAXzt,uDECODER\nFigure 16.14 The RNN-T model computing the output token distribution at time tby inte-\ngrating the output of a CTC acoustic encoder and a separate \u2018predictor\u2019 language model.\n16.5 ASR Evaluation: Word Error Rate\nThe standard evaluation metric for speech recognition systems is the word error word error\nrate. The word error rate is based on how much the word string returned by the\nrecognizer (the hypothesized word string) differs from a reference transcription.\nThe \ufb01rst step in computing word error is to compute the minimum edit distance in\nwords between the hypothesized and correct strings, giving us the minimum num-\nber of word substitutions , word insertions , and word deletions necessary to map\nbetween the correct and hypothesized strings. The word error rate (WER) is then\nde\ufb01ned as follows (note that because the equation includes insertions, the error rate\ncan be greater than 100%):\nWord Error Rate =100\u0002Insertions +Substitutions +Deletions\nTotal Words in Correct Transcript\nHere is a sample alignment between a reference and a hypothesis utterance from alignment\nthe CallHome corpus, showing the counts used to compute the error rate:\nREF: i *** ** UM the PHONE IS i LEFT THE portable **** PHONE UPSTAIRS last night\nHYP: i GOT IT TO the ***** FULLEST i LOVE TO portable FORM OF STORES last night\nEval: I I S D S S S I S S\nThis utterance has six substitutions, three insertions, and one deletion:\nWord Error Rate =1006+3+1\n13=76:9% 16.5 \u2022 ASR E VALUATION : W ORD ERROR RATE 347\nThe standard method for computing word error rates is a free script called sclite ,\navailable from the National Institute of Standards and Technologies (NIST) (NIST,\n2005). Sclite is given a series of reference (hand-transcribed, gold-standard) sen-\ntences and a matching set of hypothesis sentences. Besides performing alignments,\nand computing word error rate, sclite performs a number of other useful tasks. For\nexample, for error analysis it gives useful information such as confusion matrices\nshowing which words are often misrecognized for others, and summarizes statistics\nof words that are often inserted or deleted. sclite also gives error rates by speaker\n(if sentences are labeled for speaker ID), as well as useful statistics like the sentence\nerror rate , the percentage of sentences with at least one word error.Sentence error\nrate\nStatistical signi\ufb01cance for ASR: MAPSSWE or MacNemar\nAs with other language processing algorithms, we need to know whether a particular\nimprovement in word error rate is signi\ufb01cant or not.\nThe standard statistical tests for determining if two word error rates are different\nis the Matched-Pair Sentence Segment Word Error (MAPSSWE) test, introduced in\nGillick and Cox (1989).\nThe MAPSSWE test is a parametric test that looks at the difference between\nthe number of word errors the two systems produce, averaged across a number of\nsegments. The segments may be quite short or as long as an entire utterance; in\ngeneral, we want to have the largest number of (short) segments in order to justify\nthe normality assumption and to maximize power. The test requires that the errors\nin one segment be statistically independent of the errors in another segment. Since\nASR systems tend to use trigram LMs, we can approximate this requirement by\nde\ufb01ning a segment as a region bounded on both sides by words that both recognizers\nget correct (or by turn/utterance boundaries). Here\u2019s an example from NIST (2007)\nwith four regions:\nI II III IV\nREF: |it was|the best|of|times it|was the worst|of times| |it was\n| | | | | | | |\nSYS A:|ITS |the best|of|times it|IS the worst |of times|OR|it was\n| | | | | | | |\nSYS B:|it was|the best| |times it|WON the TEST |of times| |it was\nIn region I, system A has two errors (a deletion and an insertion) and system B\nhas zero; in region III, system A has one error (a substitution) and system B has two.\nLet\u2019s de\ufb01ne a sequence of variables Zrepresenting the difference between the errors\nin the two systems as follows:\nNi\nAthe number of errors made on segment iby system A\nNi\nB the number of errors made on segment iby system B\nZ Ni\nA\u0000Ni\nB;i=1;2;\u0001\u0001\u0001;nwhere nis the number of segments\nIn the example above, the sequence of Zvalues isf2;\u00001;\u00001;1g. Intuitively, if\nthe two systems are identical, we would expect the average difference, that is, the\naverage of the Zvalues, to be zero. If we call the true average of the differences\nmuz, we would thus like to know whether muz=0. Following closely the original\nproposal and notation of Gillick and Cox (1989), we can estimate the true average\nfrom our limited sample as \u02c6mz=Pn\ni=1Zi=n. The estimate of the variance of the Zi\u2019s\nis\ns2\nz=1\nn\u00001nX\ni=1(Zi\u0000mz)2(16.22) 348 CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\nLet\nW=\u02c6mz\nsz=pn(16.23)\nFor a large enough n(>50),Wwill approximately have a normal distribution with\nunit variance. The null hypothesis is H0:mz=0, and it can thus be rejected if\n2\u0003P(Z\u0015jwj)\u00140:05 (two-tailed) or P(Z\u0015jwj)\u00140:05 (one-tailed), where Zis\nstandard normal and wis the realized value W; these probabilities can be looked up\nin the standard tables of the normal distribution.\nEarlier work sometimes used McNemar\u2019s test for signi\ufb01cance, but McNemar\u2019s McNemar\u2019s test\nis only applicable when the errors made by the system are independent, which is not\ntrue in continuous speech recognition, where errors made on a word are extremely\ndependent on errors made on neighboring words.\nCould we improve on word error rate as a metric? It would be nice, for exam-\nple, to have something that didn\u2019t give equal weight to every word, perhaps valuing\ncontent words like Tuesday more than function words like aorof. While researchers\ngenerally agree that this would be a good idea, it has proved dif\ufb01cult to agree on\na metric that works in every application of ASR. For dialogue systems, however,\nwhere the desired semantic output is more clear, a metric called slot error rate or\nconcept error rate has proved extremely useful; it is discussed in Chapter 15 on page\n317.\n16.6 TTS\nThe goal of text-to-speech (TTS) systems is to map from strings of letters to wave-\nforms, a technology that\u2019s important for a variety of applications from dialogue sys-\ntems to games to education.\nLike ASR systems, TTS systems are generally based on the encoder-decoder\narchitecture, either using LSTMs or Transformers. There is a general difference in\ntraining. The default condition for ASR systems is to be speaker-independent: they\nare trained on large corpora with thousands of hours of speech from many speakers\nbecause they must generalize well to an unseen test speaker. By contrast, in TTS, it\u2019s\nless crucial to use multiple voices, and so basic TTS systems are speaker-dependent:\ntrained to have a consistent voice, on much less data, but all from one speaker. For\nexample, one commonly used public domain dataset, the LJ speech corpus, consists\nof 24 hours of one speaker, Linda Johnson, reading audio books in the LibriV ox\nproject (Ito and Johnson, 2017), much smaller than standard ASR corpora which are\nhundreds or thousands of hours.2\nWe generally break up the TTS task into two components. The \ufb01rst component\nis an encoder-decoder model for spectrogram prediction : it maps from strings of\nletters to mel spectrographs: sequences of mel spectral values over time. Thus we\n2There is also recent TTS research on the task of multi-speaker TTS, in which a system is trained on\nspeech from many speakers, and can switch between different voices. 16.6 \u2022 TTS 349\nmight map from this string:\nIt's time for lunch!\nto the following mel spectrogram:\nThe second component maps from mel spectrograms to waveforms. Generating\nwaveforms from intermediate representations like spectrograms is called vocoding vocoding\nand this second component is called a vocoder : vocoder\nThese standard encoder-decoder algorithms for TTS are still quite computation-\nally intensive, so a signi\ufb01cant focus of modern research is on ways to speed them\nup.\n16.6.1 TTS Preprocessing: Text normalization\nBefore either of these two steps, however, TTS systems require text normaliza-\ntion preprocessing for handling non-standard words : numbers, monetary amounts,non-standard\nwords\ndates, and other concepts that are verbalized differently than they are spelled. A TTS\nsystem seeing a number like 151needs to know to verbalize it as one hundred \ufb01fty\noneif it occurs as $151 but as one \ufb01fty one if it occurs in the context 151 Chapulte-\npec Ave. . The number 1750 can be spoken in at least four different ways, depending\non the context:\nseventeen fifty: (in\u201cThe European economy in 1750\u201d )\none seven five zero: (in\u201cThe password is 1750\u201d )\nseventeen hundred and fifty: (in\u201c1750 dollars\u201d )\none thousand, seven hundred, and fifty: (in\u201c1750 dollars\u201d )\nOften the verbalization of a non-standard word depends on its meaning (what\nTaylor (2009) calls its semiotic class ). Fig. 16.15 lays out some English non-\nstandard word types.\nMany classes have preferred realizations. A year is generally read as paired\ndigits (e.g., seventeen fifty for 1750). $3.2 billion must be read out with the\nworddollars at the end, as three point two billion dollars . Some ab-\nbreviations like N.Y. are expanded (to New York ), while other acronyms like GPU\nare pronounced as letter sequences. In languages with grammatical gender, normal-\nization may depend on morphological properties. In French, the phrase 1 mangue\n(\u2018one mangue\u2019) is normalized to une mangue , but 1 ananas (\u2018one pineapple\u2019) is\nnormalized to un ananas . In German, Heinrich IV (\u2018Henry IV\u2019) can be normalized\ntoHeinrich der Vierte ,Heinrich des Vierten ,Heinrich dem Vierten , or\nHeinrichdenVierten depending on the grammatical case of the noun (Demberg,\n2006). 350 CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\nsemiotic class examples verbalization\nabbreviations gov\u2019t , N.Y., mph government\nacronyms read as letters GPU , D.C., PC, UN, IBM G P U\ncardinal numbers 12, 45, 1/2, 0.6 twelve\nordinal numbers May 7, 3rd, Bill Gates III seventh\nnumbers read as digits Room 101 one oh one\ntimes 3.20, 11:45 eleven forty \ufb01ve\ndates 28/02 (or in US, 2/28) February twenty eighth\nyears 1999 , 80s, 1900s, 2045 nineteen ninety nine\nmoney $3.45 ,e250, $200K three dollars forty \ufb01ve\nmoney in tr/m/billions $3.45 billion three point four \ufb01ve billion dollars\npercentage 75% 3.4% seventy \ufb01ve percent\nFigure 16.15 Some types of non-standard words in text normalization; see Sproat et al.\n(2001) and (van Esch and Sproat, 2018) for many more.\nModern end-to-end TTS systems can learn to do some normalization themselves,\nbut TTS systems are only trained on a limited amount of data (like the 220,000 words\nwe mentioned above for the LJ corpus (Ito and Johnson, 2017)), and so a separate\nnormalization step is important.\nNormalization can be done by rule or by an encoder-decoder model. Rule-based\nnormalization is done in two stages: tokenization and verbalization. In the tokeniza-\ntion stage we hand-write rules to detect non-standard words. These can be regular\nexpressions, like the following for detecting years:\n/(1[89][0-9][0-9])j(20[0-9][0-9]/\nA second pass of rules express how to verbalize each semiotic class. Larger TTS\nsystems instead use more complex rule-systems, like the Kestral system of (Ebden\nand Sproat, 2015), which \ufb01rst classi\ufb01es and parses each input into a normal form\nand then produces text using a verbalization grammar. Rules have the advantage\nthat they don\u2019t require training data, and they can be designed for high precision, but\ncan be brittle, and require expert rule-writers so are hard to maintain.\nThe alternative model is to use encoder-decoder models, which have been shown\nto work better than rules for such transduction tasks, but do require expert-labeled\ntraining sets in which non-standard words have been replaced with the appropriate\nverbalization; such training sets for some languages are available (Sproat and Gor-\nman 2018, Zhang et al. 2019).\nIn the simplest encoder-decoder setting, we simply treat the problem like ma-\nchine translation, training a system to map from:\nThey live at 224 Mission St.\nto\nThey live at two twenty four Mission Street\nWhile encoder-decoder algorithms are highly accurate, they occasionally pro-\nduce errors that are egregious; for example normalizing 45 minutes asforty \ufb01ve mil-\nlimeters . To address this, more complex systems use mechanisms like lightweight\ncovering grammars , which enumerate a large set of possible verbalizations but\ndon\u2019t try to disambiguate, to constrain the decoding to avoid such outputs (Zhang\net al., 2019).\n16.6.2 TTS: Spectrogram prediction\nThe exact same architecture we described for ASR\u2014the encoder-decoder with attention\u2013\ncan be used for the \ufb01rst component of TTS. Here we\u2019ll give a simpli\ufb01ed overview 16.6 \u2022 TTS 351\nof the Tacotron2 architecture (Shen et al., 2018), which extends the earlier Tacotron Tacotron2\n(Wang et al., 2017) architecture and the Wavenet vocoder (van den Oord et al., Wavenet\n2016). Fig. 16.16 sketches out the entire architecture.\nThe encoder\u2019s job is to take a sequence of letters and produce a hidden repre-\nsentation representing the letter sequence, which is then used by the attention mech-\nanism in the decoder. The Tacotron2 encoder \ufb01rst maps every input grapheme to\na 512-dimensional character embedding. These are then passed through a stack\nof 3 convolutional layers, each containing 512 \ufb01lters with shape 5 \u00021, i.e. each\n\ufb01lter spanning 5 characters, to model the larger letter context. The output of the\n\ufb01nal convolutional layer is passed through a biLSTM to produce the \ufb01nal encoding.\nIt\u2019s common to use a slightly higher quality (but slower) version of attention called\nlocation-based attention , in which the computation of the avalues (Eq. 8.36 inlocation-based\nattention\nChapter 8) makes use of the avalues from the prior time-state.\nIn the decoder, the predicted mel spectrum from the prior time slot is passed\nthrough a small pre-net as a bottleneck. This prior output is then concatenated with\nthe encoder\u2019s attention vector context and passed through 2 LSTM layers. The out-\nput of this LSTM is used in two ways. First, it is passed through a linear layer, and\nsome output processing, to autoregressively predict one 80-dimensional log-mel \ufb01l-\nterbank vector frame (50 ms, with a 12.5 ms stride) at each step. Second, it is passed\nthrough another linear layer to a sigmoid to make a \u201cstop token prediction\u201d decision\nabout whether to stop producing output.\nWhile linear spectrograms discard phase information (and aretherefore lossy), algorithms such as Grif\ufb01n-Lim [14] are capable ofestimating this discarded information, which enables time-domainconversion via the inverse short-time Fourier transform. Mel spectro-grams discard even more information, presenting a challenging in-verse problem. However, in comparison to the linguistic and acousticfeatures used in WaveNet, the mel spectrogram is a simpler, lower-level acoustic representation of audio signals. It should thereforebe straightforward for a similar WaveNet model conditioned on melspectrograms to generate audio, essentially as a neural vocoder. In-deed, we will show that it is possible to generate high quality audiofrom mel spectrograms using a modi\ufb01ed WaveNet architecture.2.2. Spectrogram Prediction NetworkAs in Tacotron, mel spectrograms are computed through a short-time Fourier transform (STFT) using a 50 ms frame size, 12.5 msframe hop, and a Hann window function. We experimented with a5 ms frame hop to match the frequency of the conditioning inputsin the original WaveNet, but the corresponding increase in temporalresolution resulted in signi\ufb01cantly more pronunciation issues.We transform the STFT magnitude to the mel scale using an 80channel mel \ufb01lterbank spanning 125 Hz to 7.6 kHz, followed by logdynamic range compression. Prior to log compression, the \ufb01lterbankoutput magnitudes are clipped to a minimum value of 0.01 in orderto limit dynamic range in the logarithmic domain.The network is composed of an encoder and a decoder with atten-tion. The encoder converts a character sequence into a hidden featurerepresentation which the decoder consumes to predict a spectrogram.Input characters are represented using a learned 512-dimensionalcharacter embedding, which are passed through a stack of 3 convolu-tional layers each containing 512 \ufb01lters with shape5\u21e51, i.e., whereeach \ufb01lter spans 5 characters, followed by batch normalization [18]and ReLU activations. As in Tacotron, these convolutional layersmodel longer-term context (e.g.,N-grams) in the input charactersequence. The output of the \ufb01nal convolutional layer is passed into asingle bi-directional [19] LSTM [20] layer containing 512 units (256in each direction) to generate the encoded features.The encoder output is consumed by an attention network whichsummarizes the full encoded sequence as a \ufb01xed-length context vectorfor each decoder output step. We use the location-sensitive attentionfrom [21], which extends the additive attention mechanism [22] touse cumulative attention weights from previous decoder time stepsas an additional feature. This encourages the model to move forwardconsistently through the input, mitigating potential failure modeswhere some subsequences are repeated or ignored by the decoder.Attention probabilities are computed after projecting inputs and lo-cation features to 128-dimensional hidden representations. Locationfeatures are computed using 32 1-D convolution \ufb01lters of length 31.The decoder is an autoregressive recurrent neural network whichpredicts a mel spectrogram from the encoded input sequence oneframe at a time. The prediction from the previous time step is \ufb01rstpassed through a smallpre-netcontaining 2 fully connected layersof 256 hidden ReLU units. We found that the pre-net acting as aninformation bottleneck was essential for learning attention. The pre-net output and attention context vector are concatenated and passedthrough a stack of 2 uni-directional LSTM layers with 1024 units.The concatenation of the LSTM output and the attention contextvector is projected through a linear transform to predict the targetspectrogram frame. Finally, the predicted mel spectrogram is passedthrough a 5-layer convolutionalpost-netwhich predicts a residualto add to the prediction to improve the overall reconstruction. Each\n'LEVEGXIV\u0004)QFIHHMRK0SGEXMSR\u00047IRWMXMZI\u0004%XXIRXMSR\u0017\u0004'SRZ\u00040E]IVW&MHMVIGXMSREP\u00040781-RTYX\u00048I\\X\u0016\u00040E]IV\u00044VI\u00112IX\u0016\u00040781\u00040E]IVW0MRIEV\u00044VSNIGXMSR0MRIEV\u00044VSNIGXMSR7XST\u00048SOIR\u0019\u0004'SRZ\u00040E]IV\u00044SWX\u00112IX\n0HO\u00036SHFWURJUDP;EZI2IX\u00041S0;EZIJSVQ\u00047EQTPIWFig. 1. Block diagram of the Tacotron 2 system architecture.post-net layer is comprised of 512 \ufb01lters with shape5\u21e51with batchnormalization, followed bytanhactivations on all but the \ufb01nal layer.We minimize the summed mean squared error (MSE) from beforeand after the post-net to aid convergence. We also experimentedwith a log-likelihood loss by modeling the output distribution witha Mixture Density Network [23,24] to avoid assuming a constantvariance over time, but found that these were more dif\ufb01cult to trainand they did not lead to better sounding samples.In parallel to spectrogram frame prediction, the concatenation ofdecoder LSTM output and the attention context is projected downto a scalar and passed through a sigmoid activation to predict theprobability that the output sequence has completed. This \u201cstop token\u201dprediction is used during inference to allow the model to dynamicallydetermine when to terminate generation instead of always generatingfor a \ufb01xed duration. Speci\ufb01cally, generation completes at the \ufb01rstframe for which this probability exceeds a threshold of 0.5.The convolutional layers in the network are regularized usingdropout [25] with probability 0.5, and LSTM layers are regularizedusing zoneout [26] with probability 0.1. In order to introduce outputvariation at inference time, dropout with probability 0.5 is appliedonly to layers in the pre-net of the autoregressive decoder.In contrast to the original Tacotron, our model uses simpler build-ing blocks, using vanilla LSTM and convolutional layers in the en-coder and decoder instead of \u201cCBHG\u201d stacks and GRU recurrentlayers. We do not use a \u201creduction factor\u201d, i.e., each decoder stepcorresponds to a single spectrogram frame.2.3. WaveNet VocoderWe use a modi\ufb01ed version of the WaveNet architecture from [8] toinvert the mel spectrogram feature representation into time-domainwaveform samples. As in the original architecture, there are 30dilated convolution layers, grouped into 3 dilation cycles, i.e., thedilation rate of layer k (k=0...29) is2k(mod 10). To work withthe 12.5 ms frame hop of the spectrogram frames, only 2 upsamplinglayers are used in the conditioning stack instead of 3 layers.Instead of predicting discretized buckets with a softmax layer,we follow PixelCNN++ [27] and Parallel WaveNet [28] and use a 10-component mixture of logistic distributions (MoL) to generate 16-bitsamples at 24 kHz. To compute the logistic mixture distribution, theWaveNet stack output is passed through a ReLU activation followedEncoderDecoderVocoder\nFigure 16.16 The Tacotron2 architecture: An encoder-decoder maps from graphemes to\nmel spectrograms, followed by a vocoder that maps to wave\ufb01les. Figure modi\ufb01ed from Shen\net al. (2018).\nThe system is trained on gold log-mel \ufb01lterbank features, using teacher forcing,\nthat is the decoder is fed the correct log-model spectral feature at each decoder step\ninstead of the predicted decoder output from the prior step.\n16.6.3 TTS: Vocoding\nThe vocoder for Tacotron 2 is an adaptation of the WaveNet vocoder (van den Oord WaveNet\net al., 2016). Here we\u2019ll give a somewhat simpli\ufb01ed description of vocoding using\nWaveNet.\nRecall that the goal of the vocoding process here will be to invert a log mel spec-\ntrum representations back into a time-domain waveform representation. WaveNet is\nan autoregressive network, like the language models we introduced in Chapter 8. It 352 CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\ntakes spectrograms as input and produces audio output represented as sequences of\n8-bit mu-law (page 336). The probability of a waveform , a sequence of 8-bit mu-\nlaw values Y=y1;:::;yt, given an intermediate input mel spectrogram his computed\nas:\np(Y) =tY\nt=1P(ytjy1;:::;yt\u00001;h1;:::;ht) (16.24)\nThis probability distribution is modeled by a stack of special convolution layers,\nwhich include a speci\ufb01c convolutional structure called dilated convolutions , and a\nspeci\ufb01c non-linearity function.\nA dilated convolution is a subtype of causal convolutional layer. Causal or\nmasked convolutions look only at the past input, rather than the future; the pre-\ndiction of yt+1can only depend on y1;:::;yt, useful for autoregressive left-to-right\nprocessing. In dilated convolutions , at each successive layer we apply the convolu-dilated\nconvolutions\ntional \ufb01lter over a span longer than its length by skipping input values. Thus at time\ntwith a dilation value of 1, a convolutional \ufb01lter of length 2 would see input values\nxtandxt\u00001. But a \ufb01lter with a distillation value of 2 would skip an input, so would\nsee input values xtandxt\u00001. Fig. 16.17 shows the computation of the output at time\ntwith 4 dilated convolution layers with dilation values, 1, 2, 4, and 8.\nBecause models with causal convolutions do not have recurrent connections, they are typically faster\nto train than RNNs, especially when applied to very long sequences. One of the problems of causal\nconvolutions is that they require many layers, or large \ufb01lters to increase the receptive \ufb01eld. For\nexample, in Fig. 2 the receptive \ufb01eld is only 5 (= #layers + \ufb01lter length - 1). In this paper we use\ndilated convolutions to increase the receptive \ufb01eld by orders of magnitude, without greatly increasing\ncomputational cost.\nA dilated convolution (also called `a trous , or convolution with holes) is a convolution where the\n\ufb01lter is applied over an area larger than its length by skipping input values with a certain step. It is\nequivalent to a convolution with a larger \ufb01lter derived from the original \ufb01lter by dilating it with zeros,\nbut is signi\ufb01cantly more ef\ufb01cient. A dilated convolution effectively allows the network to operate on\na coarser scale than with a normal convolution. This is similar to pooling or strided convolutions, but\nhere the output has the same size as the input. As a special case, dilated convolution with dilation\n1yields the standard convolution. Fig. 3 depicts dilated causal convolutions for dilations 1,2,4,\nand8. Dilated convolutions have previously been used in various contexts, e.g. signal processing\n(Holschneider et al., 1989; Dutilleux, 1989), and image segmentation (Chen et al., 2015; Yu &\nKoltun, 2016).\nInputHidden LayerDilation = 1Hidden LayerDilation = 2Hidden LayerDilation = 4OutputDilation = 8\nFigure 3: Visualization of a stack of dilated causal convolutional layers.\nStacked dilated convolutions enable networks to have very large receptive \ufb01elds with just a few lay-\ners, while preserving the input resolution throughout the network as well as computational ef\ufb01ciency.\nIn this paper, the dilation is doubled for every layer up to a limit and then repeated: e.g.\n1,2,4,..., 512,1,2,4,..., 512,1,2,4,..., 512.\nThe intuition behind this con\ufb01guration is two-fold. First, exponentially increasing the dilation factor\nresults in exponential receptive \ufb01eld growth with depth (Yu & Koltun, 2016). For example each\n1,2,4,..., 512block has receptive \ufb01eld of size 1024 , and can be seen as a more ef\ufb01cient and dis-\ncriminative (non-linear) counterpart of a 1\u21e51024 convolution. Second, stacking these blocks further\nincreases the model capacity and the receptive \ufb01eld size.\n2.2 S OFTMAX DISTRIBUTIONS\nOne approach to modeling the conditional distributions p(xt|x1,...,x t\u00001)over the individual\naudio samples would be to use a mixture model such as a mixture density network (Bishop, 1994)\nor mixture of conditional Gaussian scale mixtures (MCGSM) (Theis & Bethge, 2015). However,\nvan den Oord et al. (2016a) showed that a softmax distribution tends to work better, even when the\ndata is implicitly continuous (as is the case for image pixel intensities or audio sample values). One\nof the reasons is that a categorical distribution is more \ufb02exible and can more easily model arbitrary\ndistributions because it makes no assumptions about their shape.\nBecause raw audio is typically stored as a sequence of 16-bit integer values (one per timestep), a\nsoftmax layer would need to output 65,536 probabilities per timestep to model all possible values.\nTo make this more tractable, we \ufb01rst apply a \u00b5-law companding transformation (ITU-T, 1988) to\nthe data, and then quantize it to 256 possible values:\nf(xt) = sign( xt)ln (1 + \u00b5|xt|)\nln (1 + \u00b5),\n3\nFigure 16.17 Dilated convolutions, showing one dilation cycle size of 4, i.e., dilation values\nof 1, 2, 4, 8. Figure from van den Oord et al. (2016).\nThe Tacotron 2 synthesizer uses 12 convolutional layers in two cycles with a\ndilation cycle size of 6, meaning that the \ufb01rst 6 layers have dilations of 1, 2, 4, 8, 16,\nand 32. and the next 6 layers again have dilations of 1, 2, 4, 8, 16, and 32. Dilated\nconvolutions allow the vocoder to grow the receptive \ufb01eld exponentially with depth.\nWaveNet predicts mu-law audio samples. Recall from page 336 that this is a\nstandard compression for audio in which the values at each sampling timestep are\ncompressed into 8-bits. This means that we can predict the value of each sample\nwith a simple 256-way categorical classi\ufb01er. The output of the dilated convolutions\nis thus passed through a softmax which makes this 256-way decision.\nThe spectrogram prediction encoder-decoder and the WaveNet vocoder are trained\nseparately. After the spectrogram predictor is trained, the spectrogram prediction\nnetwork is run in teacher-forcing mode, with each predicted spectral frame condi-\ntioned on the encoded text input and the previous frame from the ground truth spec-\ntrogram. This sequence of ground truth-aligned spectral features and gold audio\noutput is then used to train the vocoder.\nThis has been only a high-level sketch of the TTS process. There are numer-\nous important details that the reader interested in going further with TTS may want 16.7 \u2022 O THER SPEECH TASKS 353\nto look into. For example WaveNet uses a special kind of a gated activation func-\ntion as its non-linearity, and contains residual and skip connections. In practice,\npredicting 8-bit audio values doesn\u2019t as work as well as 16-bit, for which a simple\nsoftmax is insuf\ufb01cient, so decoders use fancier ways as the last step of predicting\naudio sample values, like mixtures of distributions. Finally, the WaveNet vocoder\nas we have described it would be so slow as to be useless; many different kinds of\nef\ufb01ciency improvements are necessary in practice, for example by \ufb01nding ways to\ndo non-autoregressive generation, avoiding the latency of having to wait to generate\neach frame until the prior frame has been generated, and instead making predictions\nin parallel. We encourage the interested reader to consult the original papers and\nvarious version of the code.\n16.6.4 TTS Evaluation\nSpeech synthesis systems are evaluated by human listeners. (The development of a\ngood automatic metric for synthesis evaluation, one that would eliminate the need\nfor expensive and time-consuming human listening experiments, remains an open\nand exciting research topic.)\nWe evaluate the quality of synthesized utterances by playing a sentence to lis-\nteners and ask them to give a mean opinion score (MOS ), a rating of how good MOS\nthe synthesized utterances are, usually on a scale from 1\u20135. We can then compare\nsystems by comparing their MOS scores on the same sentences (using, e.g., paired\nt-tests to test for signi\ufb01cant differences).\nIf we are comparing exactly two systems (perhaps to see if a particular change\nactually improved the system), we can use AB tests . In AB tests, we play the same AB tests\nsentence synthesized by two different systems (an A and a B system). The human\nlisteners choose which of the two utterances they like better. We do this for say\n50 sentences (presented in random order) and compare the number of sentences\npreferred for each system.\n16.7 Other Speech Tasks\nWhile we have focused on speech recognition and TTS in this chapter, there are a\nwide variety of speech-related tasks.\nThe task of wake word detection is to detect a word or short phrase, usually in wake word\norder to wake up a voice-enable assistant like Alexa, Siri, or the Google Assistant.\nThe goal with wake words is build the detection into small devices at the computing\nedge, to maintain privacy by transmitting the least amount of user speech to a cloud-\nbased server. Thus wake word detectors need to be fast, small footprint software that\ncan \ufb01t into embedded devices. Wake word detectors usually use the same frontend\nfeature extraction we saw for ASR, often followed by a whole-word classi\ufb01er.\nSpeaker diarization is the task of determining \u2018who spoke when\u2019 in a longspeaker\ndiarization\nmulti-speaker audio recording, marking the start and end of each speaker\u2019s turns in\nthe interaction. This can be useful for transcribing meetings, classroom speech, or\nmedical interactions. Often diarization systems use voice activity detection (V AD) to\n\ufb01nd segments of continuous speech, extract speaker embedding vectors, and cluster\nthe vectors to group together segments likely from the same speaker. More recent\nwork is investigating end-to-end algorithms to map directly from input speech to a\nsequence of speaker labels for each frame. 354 CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\nSpeaker recognition , is the task of identifying a speaker. We generally distin-speaker\nrecognition\nguish the subtasks of speaker veri\ufb01cation , where we make a binary decision (is\nthis speaker Xor not?), such as for security when accessing personal information\nover the telephone, and speaker identi\ufb01cation , where we make a one of Ndecision\ntrying to match a speaker\u2019s voice against a database of many speakers . These tasks\nare related to language identi\ufb01cation , in which we are given a wave\ufb01le and mustlanguage\nidenti\ufb01cation\nidentify which language is being spoken; this is useful for example for automatically\ndirecting callers to human operators that speak appropriate languages.\n16.8 Summary\nThis chapter introduced the fundamental algorithms of automatic speech recognition\n(ASR) and text-to-speech (TTS).\n\u2022 The task of speech recognition (or speech-to-text) is to map acoustic wave-\nforms to sequences of graphemes.\n\u2022 The input to a speech recognizer is a series of acoustic waves. that are sam-\npled,quantized , and converted to a spectral representation like the log mel\nspectrum .\n\u2022 Two common paradigms for speech recognition are the encoder-decoder with\nattention model, and models based on the CTC loss function . Attention-\nbased models have higher accuracies, but models based on CTC more easily\nadapt to streaming : outputting graphemes online instead of waiting until the\nacoustic input is complete.\n\u2022 ASR is evaluated using the Word Error Rate; the edit distance between the\nhypothesis and the gold transcription.\n\u2022TTS systems are also based on the encoder-decoder architecture. The en-\ncoder maps letters to an encoding, which is consumed by the decoder which\ngenerates mel spectrogram output. A neural vocoder then reads the spectro-\ngram and generates waveforms.\n\u2022 TTS systems require a \ufb01rst pass of text normalization to deal with numbers\nand abbreviations and other non-standard words.\n\u2022 TTS is evaluated by playing a sentence to human listeners and having them\ngive a mean opinion score (MOS) or by doing AB tests.\nBibliographical and Historical Notes\nASR A number of speech recognition systems were developed by the late 1940s\nand early 1950s. An early Bell Labs system could recognize any of the 10 digits\nfrom a single speaker (Davis et al., 1952). This system had 10 speaker-dependent\nstored patterns, one for each digit, each of which roughly represented the \ufb01rst two\nvowel formants in the digit. They achieved 97%\u201399% accuracy by choosing the pat-\ntern that had the highest relative correlation coef\ufb01cient with the input. Fry (1959)\nand Denes (1959) built a phoneme recognizer at University College, London, that\nrecognized four vowels and nine consonants based on a similar pattern-recognition\nprinciple. Fry and Denes\u2019s system was the \ufb01rst to use phoneme transition probabili-\nties to constrain the recognizer. BIBLIOGRAPHICAL AND HISTORICAL NOTES 355\nThe late 1960s and early 1970s produced a number of important paradigm shifts.\nFirst were a number of feature-extraction algorithms, including the ef\ufb01cient fast\nFourier transform (FFT) (Cooley and Tukey, 1965), the application of cepstral pro-\ncessing to speech (Oppenheim et al., 1968), and the development of LPC for speech\ncoding (Atal and Hanauer, 1971). Second were a number of ways of handling warp-\ning; stretching or shrinking the input signal to handle differences in speaking rate warping\nand segment length when matching against stored patterns. The natural algorithm for\nsolving this problem was dynamic programming, and, as we saw in Appendix A, the\nalgorithm was reinvented multiple times to address this problem. The \ufb01rst applica-\ntion to speech processing was by Vintsyuk (1968), although his result was not picked\nup by other researchers, and was reinvented by Velichko and Zagoruyko (1970) and\nSakoe and Chiba (1971) (and 1984). Soon afterward, Itakura (1975) combined this\ndynamic programming idea with the LPC coef\ufb01cients that had previously been used\nonly for speech coding. The resulting system extracted LPC features from incoming\nwords and used dynamic programming to match them against stored LPC templates.\nThe non-probabilistic use of dynamic programming to match a template against in-\ncoming speech is called dynamic time warping .dynamic time\nwarping\nThe third innovation of this period was the rise of the HMM. Hidden Markov\nmodels seem to have been applied to speech independently at two laboratories around\n1972. One application arose from the work of statisticians, in particular Baum and\ncolleagues at the Institute for Defense Analyses in Princeton who applied HMMs\nto various prediction problems (Baum and Petrie 1966, Baum and Eagon 1967).\nJames Baker learned of this work and applied the algorithm to speech processing\n(Baker, 1975a) during his graduate work at CMU. Independently, Frederick Jelinek\nand collaborators (drawing from their research in information-theoretical models\nin\ufb02uenced by the work of Shannon (1948)) applied HMMs to speech at the IBM\nThomas J. Watson Research Center (Jelinek et al., 1975). One early difference was\nthe decoding algorithm; Baker\u2019s DRAGON system used Viterbi (dynamic program-\nming) decoding, while the IBM system applied Jelinek\u2019s stack decoding algorithm\n(Jelinek, 1969). Baker then joined the IBM group for a brief time before founding\nthe speech-recognition company Dragon Systems.\nThe use of the HMM, with Gaussian Mixture Models (GMMs) as the phonetic\ncomponent, slowly spread through the speech community, becoming the dominant\nparadigm by the 1990s. One cause was encouragement by ARPA, the Advanced\nResearch Projects Agency of the U.S. Department of Defense. ARPA started a\n\ufb01ve-year program in 1971 to build 1000-word, constrained grammar, few speaker\nspeech understanding (Klatt, 1977), and funded four competing systems of which\nCarnegie-Mellon University\u2019s Harpy system (Lowerre, 1976), which used a simpli-\n\ufb01ed version of Baker\u2019s HMM-based DRAGON system was the best of the tested sys-\ntems. ARPA (and then DARPA) funded a number of new speech research programs,\nbeginning with 1000-word speaker-independent read-speech tasks like \u201cResource\nManagement\u201d (Price et al., 1988), recognition of sentences read from the Wall Street\nJournal (WSJ), Broadcast News domain (LDC 1998, Graff 1997) (transcription of\nactual news broadcasts, including quite dif\ufb01cult passages such as on-the-street inter-\nviews) and the Switchboard, CallHome, CallFriend, and Fisher domains (Godfrey\net al. 1992, Cieri et al. 2004) (natural telephone conversations between friends or\nstrangers). Each of the ARPA tasks involved an approximately annual bakeoff at bakeoff\nwhich systems were evaluated against each other. The ARPA competitions resulted\nin wide-scale borrowing of techniques among labs since it was easy to see which\nideas reduced errors the previous year, and the competitions were probably an im- 356 CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\nportant factor in the eventual spread of the HMM paradigm.\nBy around 1990 neural alternatives to the HMM/GMM architecture for ASR\narose, based on a number of earlier experiments with neural networks for phoneme\nrecognition and other speech tasks. Architectures included the time-delay neural\nnetwork ( TDNN )\u2014the \ufb01rst use of convolutional networks for speech\u2014 (Waibel\net al. 1989, Lang et al. 1990), RNNs (Robinson and Fallside, 1991), and the hybrid hybrid\nHMM/MLP architecture in which a feedforward neural network is trained as a pho-\nnetic classi\ufb01er whose outputs are used as probability estimates for an HMM-based\narchitecture (Morgan and Bourlard 1990, Bourlard and Morgan 1994, Morgan and\nBourlard 1995).\nWhile the hybrid systems showed performance close to the standard HMM/GMM\nmodels, the problem was speed: large hybrid models were too slow to train on the\nCPUs of that era. For example, the largest hybrid system, a feedforward network,\nwas limited to a hidden layer of 4000 units, producing probabilities over only a few\ndozen monophones. Yet training this model still required the research group to de-\nsign special hardware boards to do vector processing (Morgan and Bourlard, 1995).\nA later analytic study showed the performance of such simple feedforward MLPs\nfor ASR increases sharply with more than 1 hidden layer, even controlling for the\ntotal number of parameters (Maas et al., 2017). But the computational resources of\nthe time were insuf\ufb01cient for more layers.\nOver the next two decades a combination of Moore\u2019s law and the rise of GPUs\nallowed deep neural networks with many layers. Performance was getting close to\ntraditional systems on smaller tasks like TIMIT phone recognition by 2009 (Mo-\nhamed et al., 2009), and by 2012, the performance of hybrid systems had surpassed\ntraditional HMM/GMM systems (Jaitly et al. 2012, Dahl et al. 2012, inter alia).\nOriginally it seemed that unsupervised pretraining of the networks using a tech-\nnique like deep belief networks was important, but by 2013, it was clear that for\nhybrid HMM/GMM feedforward networks, all that mattered was to use a lot of data\nand enough layers, although a few other components did improve performance: us-\ning log mel features instead of MFCCs, using dropout, and using recti\ufb01ed linear\nunits (Deng et al. 2013, Maas et al. 2013, Dahl et al. 2013).\nMeanwhile early work had proposed the CTC loss function by 2006 (Graves\net al., 2006), and by 2012 the RNN-Transducer was de\ufb01ned and applied to phone\nrecognition (Graves 2012, Graves et al. 2013), and then to end-to-end speech recog-\nnition rescoring (Graves and Jaitly, 2014), and then recognition (Maas et al., 2015),\nwith advances such as specialized beam search (Hannun et al., 2014). (Our de-\nscription of CTC in the chapter draws on Hannun (2017), which we encourage the\ninterested reader to follow).\nThe encoder-decoder architecture was applied to speech at about the same time\nby two different groups, in the Listen Attend and Spell system of Chan et al. (2016)\nand the attention-based encoder decoder architecture of Chorowski et al. (2014)\nand Bahdanau et al. (2016). By 2018 Transformers were included in this encoder-\ndecoder architecture. Karita et al. (2019) is a nice comparison of RNNs vs Trans-\nformers in encoder-architectures for ASR, TTS, and speech-to-speech translation.\nPopular toolkits for speech processing include Kaldi (Povey et al., 2011) and Kaldi\nESPnet (Watanabe et al. 2018, Hayashi et al. 2020). ESPnet\nTTS As we noted at the beginning of the chapter, speech synthesis is one of the\nearliest \ufb01elds of speech and language processing. The 18th century saw a number\nof physical models of the articulation process, including the von Kempelen model\nmentioned above, as well as the 1773 vowel model of Kratzenstein in Copenhagen EXERCISES 357\nusing organ pipes.\nThe early 1950s saw the development of three early paradigms of waveform\nsynthesis: formant synthesis, articulatory synthesis, and concatenative synthesis.\nModern encoder-decoder systems are distant descendants of formant synthesiz-\ners. Formant synthesizers originally were inspired by attempts to mimic human\nspeech by generating arti\ufb01cial spectrograms. The Haskins Laboratories Pattern\nPlayback Machine generated a sound wave by painting spectrogram patterns on a\nmoving transparent belt and using re\ufb02ectance to \ufb01lter the harmonics of a wave-\nform (Cooper et al., 1951); other very early formant synthesizers include those of\nLawrence (1953) and Fant (1951). Perhaps the most well-known of the formant\nsynthesizers were the Klatt formant synthesizer and its successor systems, includ-\ning the MITalk system (Allen et al., 1987) and the Klattalk software used in Digital\nEquipment Corporation\u2019s DECtalk (Klatt, 1982). See Klatt (1975) for details.\nA second early paradigm, concatenative synthesis, seems to have been \ufb01rst pro-\nposed by Harris (1953) at Bell Laboratories; he literally spliced together pieces of\nmagnetic tape corresponding to phones. Soon afterwards, Peterson et al. (1958) pro-\nposed a theoretical model based on diphones, including a database with multiple\ncopies of each diphone with differing prosody, each labeled with prosodic features\nincluding F0, stress, and duration, and the use of join costs based on F0 and formant\ndistance between neighboring units. But such diphone synthesis models were not\nactually implemented until decades later (Dixon and Maxey 1968, Olive 1977). The\n1980s and 1990s saw the invention of unit selection synthesis , based on larger units\nof non-uniform length and the use of a target cost, (Sagisaka 1988, Sagisaka et al.\n1992, Hunt and Black 1996, Black and Taylor 1994, Syrdal et al. 2000).\nA third paradigm, articulatory synthesizers attempt to synthesize speech by\nmodeling the physics of the vocal tract as an open tube. Representative models\ninclude Stevens et al. (1953), Flanagan et al. (1975), and Fant (1986). See Klatt\n(1975) and Flanagan (1972) for more details.\nMost early TTS systems used phonemes as input; development of the text anal-\nysis components of TTS came somewhat later, drawing on NLP. Indeed the \ufb01rst\ntrue text-to-speech system seems to have been the system of Umeda and Teranishi\n(Umeda et al. 1968, Teranishi and Umeda 1968, Umeda 1976), which included a\nparser that assigned prosodic boundaries, as well as accent and stress.\nExercises\n16.1 Analyze each of the errors in the incorrectly recognized transcription of \u201cum\nthe phone is I left the. . . \u201d on page 346. For each one, give your best guess as\nto whether you think it is caused by a problem in signal processing, pronun-\nciation modeling, lexicon size, language model, or pruning in the decoding\nsearch.  Part III\nANNOTATING LINGUISTIC\nSTRUCTURE\nIn the \ufb01nal part of the book we discuss the task of detecting linguistic structure.\nIn the early history of NLP these structures were an intermediate step toward deeper\nlanguage processing. In modern NLP, we don\u2019t generally make explicit use of parse\nor other structures inside the neural language models we introduced in Part I, or\ndirectly in applications like those we discussed in Part II.\nInstead linguistic structure plays a number of new roles. One important role is for\ninterpretability : to provide a useful interpretive lens on neural networks. Knowing\nthat a particular layer or neuron may be computing something related to a particular\nkind of structure can help us break open the \u2018black box\u2019 and understand what the\ncomponents of our language models are doing.\nA second important role for linguistic structure is as a practical tool for social\nscienti\ufb01c studies of text: knowing which adjective modi\ufb01es which noun, or whether\na particular implicit metaphor is being used, can be important for measuring attitudes\ntoward groups or individuals. Detailed semantic structure can be helpful, for exam-\nple in \ufb01nding particular clauses that have particular meanings in legal contracts.\nWord sense labels can help keep any corpus study from measuring facts about the\nwrong word sense. Relation structures can be used to help build knowledge bases\nfrom text.\nFinally, computation of linguistic structure is an important tool for answering\nquestions about language itself, a research area called computational linguistics\nthat is sometimes distinguished from natural language processing. To answer lin-\nguistic questions about how language changes over time or across individuals we\u2019ll\nneed to be able, for example, to parse entire documents from different time periods.\nTo understand how certain linguistic structures are learned or processed by people,\nit\u2019s necessary to be able to automatically label structures for arbitrary text.\nIn our study of linguistic structure, we begin with one of the oldest tasks in\ncomputational linguistics: the extraction of syntactic structure, and give two sets of\nalgorithms for parsing : extracting syntactic structure, including constituency pars-\ning and dependency parsing. We then introduce a variety of structures related to\nmeaning, including semantic roles, word senses, entity relations, and events. We 360\nconclude with linguistic structures that tend to be related to discourse and meaning\nover larger texts, including coreference and discourse coherence. In each case we\u2019ll\ngive algorithms for automatically annotating the relevant structure.  362 CHAPTER 17 \u2022 S EQUENCE LABELING FOR PARTS OF SPEECH AND NAMED ENTITIES\nCHAPTER\n17Sequence Labeling for Parts of\nSpeech and Named Entities\nTo each word a warbling note\nA Midsummer Night\u2019s Dream , V .I\nDionysius Thrax of Alexandria ( c.100 B.C.), or perhaps someone else (it was a long\ntime ago), wrote a grammatical sketch of Greek (a \u201c techn \u00afe\u201d) that summarized the\nlinguistic knowledge of his day. This work is the source of an astonishing proportion\nof modern linguistic vocabulary, including the words syntax ,diphthong ,clitic , and\nanalogy . Also included are a description of eight parts of speech : noun, verb, parts of speech\npronoun, preposition, adverb, conjunction, participle, and article. Although earlier\nscholars (including Aristotle as well as the Stoics) had their own lists of parts of\nspeech, it was Thrax\u2019s set of eight that became the basis for descriptions of European\nlanguages for the next 2000 years. (All the way to the Schoolhouse Rock educational\ntelevision shows of our childhood, which had songs about 8 parts of speech, like the\nlate great Bob Dorough\u2019s Conjunction Junction .) The durability of parts of speech\nthrough two millennia speaks to their centrality in models of human language.\nProper names are another important and anciently studied linguistic category.\nWhile parts of speech are generally assigned to individual words or morphemes, a\nproper name is often an entire multiword phrase, like the name \u201cMarie Curie\u201d, the\nlocation \u201cNew York City\u201d, or the organization \u201cStanford University\u201d. We\u2019ll use the\nterm named entity for, roughly speaking, anything that can be referred to with a named entity\nproper name: a person, a location, an organization, although as we\u2019ll see the term is\ncommonly extended to include things that aren\u2019t entities per se.\nParts of speech (also known as POS ) and named entities are useful clues to POS\nsentence structure and meaning. Knowing whether a word is a noun or a verb tells us\nabout likely neighboring words (nouns in English are preceded by determiners and\nadjectives, verbs by nouns) and syntactic structure (verbs have dependency links to\nnouns), making part-of-speech tagging a key aspect of parsing. Knowing if a named\nentity like Washington is a name of a person, a place, or a university is important to\nmany natural language processing tasks like question answering, stance detection,\nor information extraction.\nIn this chapter we\u2019ll introduce the task of part-of-speech tagging , taking a se-\nquence of words and assigning each word a part of speech like NOUN orVERB , and\nthe task of named entity recognition (NER ), assigning words or phrases tags like\nPERSON ,LOCATION , or ORGANIZATION .\nSuch tasks in which we assign, to each word xiin an input word sequence, a\nlabel yi, so that the output sequence Yhas the same length as the input sequence X\nare called sequence labeling tasks. We\u2019ll introduce classic sequence labeling algo-sequence\nlabeling\nrithms, one generative\u2014 the Hidden Markov Model (HMM)\u2014and one discriminative\u2014\nthe Conditional Random Field (CRF). In following chapters we\u2019ll introduce modern\nsequence labelers based on RNNs and Transformers. 17.1 \u2022 (M OSTLY ) ENGLISH WORD CLASSES 363\n17.1 (Mostly) English Word Classes\nUntil now we have been using part-of-speech terms like noun andverb rather freely.\nIn this section we give more complete de\ufb01nitions. While word classes do have\nsemantic tendencies\u2014adjectives, for example, often describe properties and nouns\npeople \u2014 parts of speech are de\ufb01ned instead based on their grammatical relationship\nwith neighboring words or the morphological properties about their af\ufb01xes.\nTag Description ExampleOpen ClassADJ Adjective: noun modi\ufb01ers describing properties red,young ,awesome\nADV Adverb: verb modi\ufb01ers of time, place, manner very,slowly ,home ,yesterday\nNOUN words for persons, places, things, etc. algorithm ,cat,mango ,beauty\nVERB words for actions and processes draw ,provide ,go\nPROPN Proper noun: name of a person, organization, place, etc.. Regina ,IBM,Colorado\nINTJ Interjection: exclamation, greeting, yes/no response, etc. oh,um,yes,helloClosed Class WordsADP Adposition (Preposition/Postposition): marks a noun\u2019s\nspacial, temporal, or other relationin, on, by, under\nAUX Auxiliary: helping verb marking tense, aspect, mood, etc., can, may, should, are\nCCONJ Coordinating Conjunction: joins two phrases/clauses and,or,but\nDET Determiner: marks noun phrase properties a, an, the, this\nNUM Numeral one, two, 2026, 11:00, hundred\nPART Particle: a function word that must be associated with an-\nother word\u2019s, not, (in\ufb01nitive) to\nPRON Pronoun: a shorthand for referring to an entity or event she, who, I, others\nSCONJ Subordinating Conjunction: joins a main clause with a\nsubordinate clause such as a sentential complementwhether ,becauseOtherPUNCT Punctuation\u02d9, , ()\nSYM Symbols like $ or emoji $, %\nX Other asdf, qwfg\nFigure 17.1 The 17 parts of speech in the Universal Dependencies tagset (de Marneffe et al., 2021). Features\ncan be added to make \ufb01ner-grained distinctions (with properties like number, case, de\ufb01niteness, and so on).\nParts of speech fall into two broad categories: closed class andopen class . closed class\nopen class Closed classes are those with relatively \ufb01xed membership, such as prepositions\u2014\nnew prepositions are rarely coined. By contrast, nouns and verbs are open classes\u2014\nnew nouns and verbs like iPhone orto fax are continually being created or borrowed.\nClosed class words are generally function words likeof,it,and, oryou, which tend function word\nto be very short, occur frequently, and often have structuring uses in grammar.\nFour major open classes occur in the languages of the world: nouns (including\nproper nouns), verbs ,adjectives , and adverbs , as well as the smaller open class of\ninterjections . English has all \ufb01ve, although not every language does.\nNouns are words for people, places, or things, but include others as well. Com- noun\nmon nouns include concrete terms like catandmango , abstractions like algorithm common noun\nandbeauty , and verb-like terms like pacing as in His pacing to and fro became quite\nannoying . Nouns in English can occur with determiners ( a goat, this bandwidth )\ntake possessives ( IBM\u2019s annual revenue ), and may occur in the plural ( goats, abaci ).\nMany languages, including English, divide common nouns into count nouns and count noun\nmass nouns . Count nouns can occur in the singular and plural ( goat/goats, rela- mass noun\ntionship/relationships ) and can be counted ( one goat, two goats ). Mass nouns are\nused when something is conceptualized as a homogeneous group. So snow, salt , and\ncommunism are not counted (i.e., *two snows or*two communisms ).Proper nouns , proper noun\nlikeRegina ,Colorado , and IBM, are names of speci\ufb01c persons or entities. 364 CHAPTER 17 \u2022 S EQUENCE LABELING FOR PARTS OF SPEECH AND NAMED ENTITIES\nVerbs refer to actions and processes, including main verbs like draw ,provide , verb\nandgo. English verbs have in\ufb02ections (non-third-person-singular ( eat), third-person-\nsingular ( eats), progressive ( eating ), past participle ( eaten )). While many scholars\nbelieve that all human languages have the categories of noun and verb, others have\nargued that some languages, such as Riau Indonesian and Tongan, don\u2019t even make\nthis distinction (Broschart 1997; Evans 2000; Gil 2000) .\nAdjectives often describe properties or qualities of nouns, like color ( white , adjective\nblack ), age ( old,young ), and value ( good ,bad), but there are languages without\nadjectives. In Korean, for example, the words corresponding to English adjectives\nact as a subclass of verbs, so what is in English an adjective \u201cbeautiful\u201d acts in\nKorean like a verb meaning \u201cto be beautiful\u201d.\nAdverbs are a hodge-podge. All the italicized words in this example are adverbs: adverb\nActually , I ran home extremely quickly yesterday\nAdverbs generally modify something (often verbs, hence the name \u201cadverb\u201d, but\nalso other adverbs and entire verb phrases). Directional adverbs orlocative ad- locative\nverbs (home ,here,downhill ) specify the direction or location of some action; degree degree\nadverbs (extremely ,very,somewhat ) specify the extent of some action, process, or\nproperty; manner adverbs (slowly ,slinkily ,delicately ) describe the manner of some manner\naction or process; and temporal adverbs describe the time that some action or event temporal\ntook place ( yesterday ,Monday ).\nInterjections (oh, hey, alas, uh, um ) are a smaller open class that also includes interjection\ngreetings ( hello, goodbye ) and question responses ( yes, no, uh-huh ).\nEnglish adpositions occur before nouns, hence are called prepositions . They can preposition\nindicate spatial or temporal relations, whether literal ( on it,before then ,by the house )\nor metaphorical ( on time ,with gusto ,beside herself ), and relations like marking the\nagent in Hamlet was written by Shakespeare .\nAparticle resembles a preposition or an adverb and is used in combination with particle\na verb. Particles often have extended meanings that aren\u2019t quite the same as the\nprepositions they resemble, as in the particle over inshe turned the paper over . A\nverb and a particle acting as a single unit is called a phrasal verb . The meaning phrasal verb\nof phrasal verbs is often non-compositional \u2014not predictable from the individual\nmeanings of the verb and the particle. Thus, turn down means \u2018reject\u2019, rule out\n\u2018eliminate\u2019, and go on \u2018continue\u2019.\nDeterminers likethisandthat(this chapter ,that page ) can mark the start of an determiner\nEnglish noun phrase. Articles likea,an, and the, are a type of determiner that mark article\ndiscourse properties of the noun and are quite frequent; theis the most common\nword in written English, with aandanright behind.\nConjunctions join two phrases, clauses, or sentences. Coordinating conjunc- conjunction\ntions like and,or, and butjoin two elements of equal status. Subordinating conjunc-\ntions are used when one of the elements has some embedded status. For example,\nthe subordinating conjunction thatin\u201cI thought that you might like some milk\u201d links\nthe main clause I thought with the subordinate clause you might like some milk . This\nclause is called subordinate because this entire clause is the \u201ccontent\u201d of the main\nverb thought . Subordinating conjunctions like thatwhich link a verb to its argument\nin this way are also called complementizers . complementizer\nPronouns act as a shorthand for referring to an entity or event. Personal pro- pronoun\nnouns refer to persons or entities ( you,she,I,it,me, etc.). Possessive pronouns are\nforms of personal pronouns that indicate either actual possession or more often just\nan abstract relation between the person and some object ( my, your, his, her, its, one\u2019s,\nour, their ).Wh-pronouns (what, who, whom, whoever ) are used in certain question wh 17.2 \u2022 P ART-OF-SPEECH TAGGING 365\nforms, or act as complementizers ( Frida, who married Diego. . . ).\nAuxiliary verbs mark semantic features of a main verb such as its tense, whether auxiliary\nit is completed (aspect), whether it is negated (polarity), and whether an action is\nnecessary, possible, suggested, or desired (mood). English auxiliaries include the\ncopula verb be, the two verbs doandhave , forms, as well as modal verbs used to copula\nmodal mark the mood associated with the event depicted by the main verb: canindicates\nability or possibility, may permission or possibility, must necessity.\nAn English-speci\ufb01c tagset, the Penn Treebank tagset (Marcus et al., 1993), shown\nin Fig. 17.2, has been used to label many syntactically annotated corpora like the\nPenn Treebank corpora, so it is worth knowing about.\nTag Description Example Tag Description Example Tag Description Example\nCC coord. conj. and, but, or NNP proper noun, sing. IBM TO in\ufb01nitive to to\nCD cardinal number one, two NNPS proper noun, plu. Carolinas UH interjection ah, oops\nDT determiner a, the NNS noun, plural llamas VB verb base eat\nEX existential \u2018there\u2019 there PDT predeterminer all, both VBD verb past tense ate\nFW foreign word mea culpa POS possessive ending \u2019s VBG verb gerund eating\nIN preposition/\nsubordin-conjof, in, by PRP personal pronoun I, you, he VBN verb past partici-\npleeaten\nJJ adjective yellow PRP$ possess. pronoun your VBP verb non-3sg-pr eat\nJJR comparative adj bigger RB adverb quickly VBZ verb 3sg pres eats\nJJS superlative adj wildest RBR comparative adv faster WDT wh-determ. which, that\nLS list item marker 1, 2, One RBS superlatv. adv fastest WP wh-pronoun what, who\nMD modal can, should RP particle up, off WP$ wh-possess. whose\nNN sing or mass noun llama SYM symbol +,%,& WRB wh-adverb how, where\nFigure 17.2 Penn Treebank core 36 part-of-speech tags.\nBelow we show some examples with each word tagged according to both the UD\n(in blue) and Penn (in red) tagsets. Notice that the Penn tagset distinguishes tense\nand participles on verbs, and has a special tag for the existential there construction in\nEnglish. Note that since London Journal of Medicine is a proper noun, both tagsets\nmark its component nouns as PROPN/NNP, including journal andmedicine , which\nmight otherwise be labeled as common nouns (NOUN/NN).\n(17.1) There/ PRON /EXare/ VERB /VBP 70/NUM /CDchildren/ NOUN /NNS\nthere/ ADV/RB./PUNC /.\n(17.2) Preliminary/ ADJ/JJ\ufb01ndings/ NOUN /NNS were/ AUX/VBD\nreported/ VERB /VBN in/ADP/INtoday/ NOUN /NN\u2019s/PART /POS\nLondon/ PROPN /NNP Journal/ PROPN /NNP of/ADP/INMedicine/ PROPN /NNP\n17.2 Part-of-Speech Tagging\nPart-of-speech tagging is the process of assigning a part-of-speech to each word inpart-of-speech\ntagging\na text. The input is a sequence x1;x2;:::;xnof (tokenized) words and a tagset, and\nthe output is a sequence y1;y2;:::;ynof tags, each output yicorresponding exactly to\none input xi, as shown in the intuition in Fig. 17.3.\nTagging is a disambiguation task; words are ambiguous \u2014have more than one ambiguous\npossible part-of-speech\u2014and the goal is to \ufb01nd the correct tag for the situation.\nFor example, book can be a verb ( book that \ufb02ight ) or a noun ( hand me that book ).\nThat can be a determiner ( Does that \ufb02ight serve dinner ) or a complementizer ( I 366 CHAPTER 17 \u2022 S EQUENCE LABELING FOR PARTS OF SPEECH AND NAMED ENTITIES\nwillNOUNAUXVERBDETNOUNJanetbackthebillPart of Speech Taggerx1x2x3x4x5y1y2y3y4y5\nFigure 17.3 The task of part-of-speech tagging: mapping from input words x1;x2;:::;xnto\noutput POS tags y1;y2;:::;yn.\nthought that your \ufb02ight was earlier ). The goal of POS-tagging is to resolve theseambiguity\nresolution\nambiguities, choosing the proper tag for the context.\nThe accuracy of part-of-speech tagging algorithms (the percentage of test set accuracy\ntags that match human gold labels) is extremely high. One study found accuracies\nover 97% across 15 languages from the Universal Dependency (UD) treebank (Wu\nand Dredze, 2019). Accuracies on various English treebanks are also 97% (no matter\nthe algorithm; HMMs, CRFs, BERT perform similarly). This 97% number is also\nabout the human performance on this task, at least for English (Manning, 2011).\nTypes: WSJ Brown\nUnambiguous (1 tag) 44,432 ( 86% ) 45,799 ( 85% )\nAmbiguous (2+ tags) 7,025 ( 14% ) 8,050 ( 15% )\nTokens :\nUnambiguous (1 tag) 577,421 ( 45% ) 384,349 ( 33% )\nAmbiguous (2+ tags) 711,780 ( 55% ) 786,646 ( 67% )\nFigure 17.4 Tag ambiguity in the Brown and WSJ corpora (Treebank-3 45-tag tagset).\nWe\u2019ll introduce algorithms for the task in the next few sections, but \ufb01rst let\u2019s\nexplore the task. Exactly how hard is it? Fig. 17.4 shows that most word types\n(85-86%) are unambiguous ( Janet is always NNP, hesitantly is always RB). But the\nambiguous words, though accounting for only 14-15% of the vocabulary, are very\ncommon, and 55-67% of word tokens in running text are ambiguous. Particularly\nambiguous common words include that,back,down ,putandset; here are some\nexamples of the 6 different parts of speech for the word back:\nearnings growth took a back/JJ seat\na small building in the back/NN\na clear majority of senators back/VBP the bill\nDave began to back/VB toward the door\nenable the country to buy back/RP debt\nI was twenty-one back/RB then\nNonetheless, many words are easy to disambiguate, because their different tags\naren\u2019t equally likely. For example, acan be a determiner or the letter a, but the\ndeterminer sense is much more likely.\nThis idea suggests a useful baseline : given an ambiguous word, choose the tag\nwhich is most frequent in the training corpus. This is a key concept:\nMost Frequent Class Baseline: Always compare a classi\ufb01er against a baseline at\nleast as good as the most frequent class baseline (assigning each token to the class\nit occurred in most often in the training set). 17.3 \u2022 N AMED ENTITIES AND NAMED ENTITY TAGGING 367\nThe most-frequent-tag baseline has an accuracy of about 92%1. The baseline\nthus differs from the state-of-the-art and human ceiling (97%) by only 5%.\n17.3 Named Entities and Named Entity Tagging\nPart of speech tagging can tell us that words like Janet ,Stanford University , and\nColorado are all proper nouns; being a proper noun is a grammatical property of\nthese words. But viewed from a semantic perspective, these proper nouns refer to\ndifferent kinds of entities: Janet is a person, Stanford University is an organization,\nand Colorado is a location.\nHere we re-introduce the concept of a named entity , which was also introduced named entity\nin Section 11.5 for readers who haven\u2019t yet read Chapter 11.\nAnamed entity is, roughly speaking, anything that can be referred to with a named entity\nproper name: a person, a location, an organization. The task of named entity recog-\nnition (NER ) is to \ufb01nd spans of text that constitute proper names and tag the type ofnamed entity\nrecognition\nNER the entity. Four entity tags are most common: PER (person), LOC (location), ORG\n(organization), or GPE (geo-political entity). However, the term named entity is\ncommonly extended to include things that aren\u2019t entities per se, including dates,\ntimes, and other kinds of temporal expressions, and even numerical expressions like\nprices. Here\u2019s an example of the output of an NER tagger:\nCiting high fuel prices, [ ORG United Airlines ] said [ TIME Friday ] it\nhas increased fares by [ MONEY $6] per round trip on \ufb02ights to some\ncities also served by lower-cost carriers. [ ORG American Airlines ], a\nunit of [ ORG AMR Corp.] , immediately matched the move, spokesman\n[PER Tim Wagner ] said. [ ORG United] , a unit of [ ORG UAL Corp.] ,\nsaid the increase took effect [ TIME Thursday] and applies to most\nroutes where it competes against discount carriers, such as [ LOC Chicago]\nto [LOC Dallas] and [ LOC Denver] to [LOC San Francisco] .\nThe text contains 13 mentions of named entities including 5 organizations, 4 loca-\ntions, 2 times, 1 person, and 1 mention of money. Figure 17.5 shows typical generic\nnamed entity types. Many applications will also need to use speci\ufb01c entity types like\nproteins, genes, commercial products, or works of art.\nType Tag Sample Categories Example sentences\nPeople PER people, characters Turing is a giant of computer science.\nOrganization ORG companies, sports teams The IPCC warned about the cyclone.\nLocation LOC regions, mountains, seas Mt. Sanitas is in Sunshine Canyon .\nGeo-Political Entity GPE countries, states Palo Alto is raising the fees for parking.\nFigure 17.5 A list of generic named entity types with the kinds of entities they refer to.\nNamed entity tagging is a useful \ufb01rst step in lots of natural language processing\ntasks. In sentiment analysis we might want to know a consumer\u2019s sentiment toward a\nparticular entity. Entities are a useful \ufb01rst stage in question answering, or for linking\ntext to information in structured knowledge sources like Wikipedia. And named\nentity tagging is also central to tasks involving building semantic representations,\nlike extracting events and the relationship between participants.\n1In English, on the WSJ corpus, tested on sections 22-24. 368 CHAPTER 17 \u2022 S EQUENCE LABELING FOR PARTS OF SPEECH AND NAMED ENTITIES\nUnlike part-of-speech tagging, where there is no segmentation problem since\neach word gets one tag, the task of named entity recognition is to \ufb01nd and label\nspans of text, and is dif\ufb01cult partly because of the ambiguity of segmentation; we\nneed to decide what\u2019s an entity and what isn\u2019t, and where the boundaries are. Indeed,\nmost words in a text will not be named entities. Another dif\ufb01culty is caused by type\nambiguity. The mention JFKcan refer to a person, the airport in New York, or any\nnumber of schools, bridges, and streets around the United States. Some examples of\nthis kind of cross-type confusion are given in Figure 17.6.\n[PER Washington] was born into slavery on the farm of James Burroughs.\n[ORG Washington] went up 2 games to 1 in the four-game series.\nBlair arrived in [ LOC Washington] for what may well be his last state visit.\nIn June, [ GPE Washington] passed a primary seatbelt law.\nFigure 17.6 Examples of type ambiguities in the use of the name Washington .\nThe standard approach to sequence labeling for a span-recognition problem like\nNER is BIO tagging (Ramshaw and Marcus, 1995). This is a method that allows us\nto treat NER like a word-by-word sequence labeling task, via tags that capture both\nthe boundary and the named entity type. Consider the following sentence:\n[PER Jane Villanueva ] of [ ORG United ] , a unit of [ ORG United Airlines\nHolding ] , said the fare applies to the [ LOC Chicago ] route.\nFigure 17.7 shows the same excerpt represented with BIO tagging, as well as BIO\nvariants called IOtagging and BIOES tagging. In BIO tagging we label any token\nthatbegins a span of interest with the label B, tokens that occur inside a span are\ntagged with an I, and any tokens outside of any span of interest are labeled O. While\nthere is only one Otag, we\u2019ll have distinct Band Itags for each named entity class.\nThe number of tags is thus 2 n+1 tags, where nis the number of entity types. BIO\ntagging can represent exactly the same information as the bracketed notation, but has\nthe advantage that we can represent the task in the same simple sequence modeling\nway as part-of-speech tagging: assigning a single label yito each input word xi:\nWords IO Label BIO Label BIOES Label\nJane I-PER B-PER B-PER\nVillanueva I-PER I-PER E-PER\nof O O O\nUnited I-ORG B-ORG B-ORG\nAirlines I-ORG I-ORG I-ORG\nHolding I-ORG I-ORG E-ORG\ndiscussed O O O\nthe O O O\nChicago I-LOC B-LOC S-LOC\nroute O O O\n. O O O\nFigure 17.7 NER as a sequence model, showing IO, BIO, and BIOES taggings.\nWe\u2019ve also shown two variant tagging schemes: IO tagging, which loses some\ninformation by eliminating the B tag, and BIOES tagging, which adds an end tag\nEfor the end of a span, and a span tag Sfor a span consisting of only one word.\nA sequence labeler (HMM, CRF, RNN, Transformer, etc.) is trained to label each\ntoken in a text with tags that indicate the presence (or absence) of particular kinds\nof named entities. 17.4 \u2022 HMM P ART-OF-SPEECH TAGGING 369\n17.4 HMM Part-of-Speech Tagging\nIn this section we introduce our \ufb01rst sequence labeling algorithm, the Hidden Markov\nModel, and show how to apply it to part-of-speech tagging. Recall that a sequence\nlabeler is a model whose job is to assign a label to each unit in a sequence, thus\nmapping a sequence of observations to a sequence of labels of the same length.\nThe HMM is a classic model that introduces many of the key concepts of sequence\nmodeling that we will see again in more modern models.\nAn HMM is a probabilistic sequence model: given a sequence of units (words,\nletters, morphemes, sentences, whatever), it computes a probability distribution over\npossible sequences of labels and chooses the best label sequence.\n17.4.1 Markov Chains\nThe HMM is based on augmenting the Markov chain. A Markov chain is a model Markov chain\nthat tells us something about the probabilities of sequences of random variables,\nstates , each of which can take on values from some set. These sets can be words, or\ntags, or symbols representing anything, for example the weather. A Markov chain\nmakes a very strong assumption that if we want to predict the future in the sequence,\nall that matters is the current state. All the states before the current state have no im-\npact on the future except via the current state. It\u2019s as if to predict tomorrow\u2019s weather\nyou could examine today\u2019s weather but you weren\u2019t allowed to look at yesterday\u2019s\nweather.\nWARM3HOT1COLD2.8.6.1.1.3.6.1.1.3\ncharminguniformlyare.1.4.5.5.5.2.6.2\n(a) (b)\nFigure 17.8 A Markov chain for weather (a) and one for words (b), showing states and\ntransitions. A start distribution pis required; setting p= [0:1;0:7;0:2]for (a) would mean a\nprobability 0.7 of starting in state 2 (cold), probability 0.1 of starting in state 1 (hot), etc.\nMore formally, consider a sequence of state variables q1;q2;:::;qi. A Markov\nmodel embodies the Markov assumption on the probabilities of this sequence: thatMarkov\nassumption\nwhen predicting the future, the past doesn\u2019t matter, only the present.\nMarkov Assumption: P(qi=ajq1:::qi\u00001) =P(qi=ajqi\u00001) (17.3)\nFigure 17.8a shows a Markov chain for assigning a probability to a sequence of\nweather events, for which the vocabulary consists of HOT,COLD , and WARM . The\nstates are represented as nodes in the graph, and the transitions, with their probabil-\nities, as edges. The transitions are probabilities: the values of arcs leaving a given\nstate must sum to 1. Figure 17.8b shows a Markov chain for assigning a probabil-\nity to a sequence of words w1:::wt. This Markov chain should be familiar; in fact,\nit represents a bigram language model, with each edge expressing the probability\np(wijwj)! Given the two models in Fig. 17.8, we can assign a probability to any\nsequence from our vocabulary. 370 CHAPTER 17 \u2022 S EQUENCE LABELING FOR PARTS OF SPEECH AND NAMED ENTITIES\nFormally, a Markov chain is speci\ufb01ed by the following components:\nQ=q1q2:::qN a set of Nstates\nA=a11a12:::aN1:::aNN atransition probability matrix A, each ai jrepresent-\ning the probability of moving from state ito state j, s.t.Pn\nj=1ai j=18i\np=p1;p2;:::;pN aninitial probability distribution over states. piis the\nprobability that the Markov chain will start in state i.\nSome states jmay have pj=0, meaning that they cannot\nbe initial states. Also,Pn\ni=1pi=1\nBefore you go on, use the sample probabilities in Fig. 17.8a (with p= [0:1;0:7;0:2])\nto compute the probability of each of the following sequences:\n(17.4) hot hot hot hot\n(17.5) cold hot cold hot\nWhat does the difference in these probabilities tell you about a real-world weather\nfact encoded in Fig. 17.8a?\n17.4.2 The Hidden Markov Model\nA Markov chain is useful when we need to compute a probability for a sequence\nof observable events. In many cases, however, the events we are interested in are\nhidden : we don\u2019t observe them directly. For example we don\u2019t normally observe hidden\npart-of-speech tags in a text. Rather, we see words, and must infer the tags from the\nword sequence. We call the tags hidden because they are not observed.\nAhidden Markov model (HMM ) allows us to talk about both observed eventshidden Markov\nmodel\n(like words that we see in the input) and hidden events (like part-of-speech tags) that\nwe think of as causal factors in our probabilistic model. An HMM is speci\ufb01ed by\nthe following components:\nQ=q1q2:::qN a set of Nstates\nA=a11:::ai j:::aNNatransition probability matrix A, each ai jrepresenting the probability\nof moving from state ito state j, s.t.PN\nj=1ai j=18i\nB=bi(ot) a sequence of observation likelihoods , also called emission probabili-\nties, each expressing the probability of an observation ot(drawn from a\nvocabulary V=v1;v2;:::;vV) being generated from a state qi\np=p1;p2;:::;pN aninitial probability distribution over states. piis the probability that\nthe Markov chain will start in state i. Some states jmay have pj=0,\nmeaning that they cannot be initial states. Also,Pn\ni=1pi=1\nThe HMM is given as input O=o1o2:::oT: a sequence of Tobservations , each\none drawn from the vocabulary V.\nA \ufb01rst-order hidden Markov model instantiates two simplifying assumptions.\nFirst, as with a \ufb01rst-order Markov chain, the probability of a particular state depends\nonly on the previous state:\nMarkov Assumption: P(qijq1;:::;qi\u00001) =P(qijqi\u00001) (17.6)\nSecond, the probability of an output observation oidepends only on the state that\nproduced the observation qiand not on any other states or any other observations:\nOutput Independence: P(oijq1;:::qi;:::; qT;o1;:::; oi;:::; oT) =P(oijqi)(17.7) 17.4 \u2022 HMM P ART-OF-SPEECH TAGGING 371\n17.4.3 The components of an HMM tagger\nAn HMM has two components, the AandBprobabilities, both estimated by counting\non a tagged training corpus. (For this example we\u2019ll use the tagged WSJ corpus.)\nTheAmatrix contains the tag transition probabilities P(tijti\u00001)which represent\nthe probability of a tag occurring given the previous tag. For example, modal verbs\nlikewillare very likely to be followed by a verb in the base form, a VB, like race, so\nwe expect this probability to be high. We compute the maximum likelihood estimate\nof this transition probability by counting, out of the times we see the \ufb01rst tag in a\nlabeled corpus, how often the \ufb01rst tag is followed by the second:\nP(tijti\u00001) =C(ti\u00001;ti)\nC(ti\u00001)(17.8)\nIn the WSJ corpus, for example, MD occurs 13124 times of which it is followed\nby VB 10471, for an MLE estimate of\nP(V BjMD) =C(MD;V B)\nC(MD)=10471\n13124=:80 (17.9)\nTheBemission probabilities, P(wijti), represent the probability, given a tag (say\nMD), that it will be associated with a given word (say will). The MLE of the emis-\nsion probability is\nP(wijti) =C(ti;wi)\nC(ti)(17.10)\nOf the 13124 occurrences of MD in the WSJ corpus, it is associated with will4046\ntimes:\nP(willjMD) =C(MD;will)\nC(MD)=4046\n13124=:31 (17.11)\nWe saw this kind of Bayesian modeling in Chapter 4; recall that this likelihood\nterm is not asking \u201cwhich is the most likely tag for the word will?\u201d That would be\nthe posterior P(MDjwill). Instead, P(willjMD)answers the slightly counterintuitive\nquestion \u201cIf we were going to generate a MD, how likely is it that this modal would\nbewill?\u201d\nNN3VB1MD2a22\na11a12a21a13a33a32a23a31P(\"aardvark\" | NN)...P(\u201cwill\u201d | NN)...P(\"the\" | NN)...P(\u201cback\u201d | NN)...P(\"zebra\" | NN)B3P(\"aardvark\" | VB)...P(\u201cwill\u201d | VB)...P(\"the\" | VB)...P(\u201cback\u201d | VB)...P(\"zebra\" | VB)B1P(\"aardvark\" | MD)...P(\u201cwill\u201d | MD)...P(\"the\" | MD)...P(\u201cback\u201d | MD)...P(\"zebra\" | MD)B2\nFigure 17.9 An illustration of the two parts of an HMM representation: the Atransition\nprobabilities used to compute the prior probability, and the Bobservation likelihoods that are\nassociated with each state, one likelihood for each possible observation word. 372 CHAPTER 17 \u2022 S EQUENCE LABELING FOR PARTS OF SPEECH AND NAMED ENTITIES\nThe Atransition probabilities, and Bobservation likelihoods of the HMM are\nillustrated in Fig. 17.9 for three states in an HMM part-of-speech tagger; the full\ntagger would have one state for each tag.\n17.4.4 HMM tagging as decoding\nFor any model, such as an HMM, that contains hidden variables, the task of deter-\nmining the hidden variables sequence corresponding to the sequence of observations\nis called decoding . More formally, decoding\nDecoding : Given as input an HMM l= (A;B)and a sequence of ob-\nservations O=o1;o2;:::;oT, \ufb01nd the most probable sequence of states\nQ=q1q2q3:::qT.\nFor part-of-speech tagging, the goal of HMM decoding is to choose the tag\nsequence t1:::tnthat is most probable given the observation sequence of nwords\nw1:::wn:\n\u02c6t1:n=argmax\nt1:::tnP(t1:::tnjw1:::wn) (17.12)\nThe way we\u2019ll do this in the HMM is to use Bayes\u2019 rule to instead compute:\n\u02c6t1:n=argmax\nt1:::tnP(w1:::wnjt1:::tn)P(t1:::tn)\nP(w1:::wn)(17.13)\nFurthermore, we simplify Eq. 17.13 by dropping the denominator P(wn\n1):\n\u02c6t1:n=argmax\nt1:::tnP(w1:::wnjt1:::tn)P(t1:::tn) (17.14)\nHMM taggers make two further simplifying assumptions. The \ufb01rst (output in-\ndependence, from Eq. 17.7) is that the probability of a word appearing depends only\non its own tag and is independent of neighboring words and tags:\nP(w1:::wnjt1:::tn)\u0019nY\ni=1P(wijti) (17.15)\nThe second assumption (the Markov assumption, Eq. 17.6) is that the probability of\na tag is dependent only on the previous tag, rather than the entire tag sequence;\nP(t1:::tn)\u0019nY\ni=1P(tijti\u00001) (17.16)\nPlugging the simplifying assumptions from Eq. 17.15 and Eq. 17.16 into Eq. 17.14\nresults in the following equation for the most probable tag sequence from a bigram\ntagger:\n\u02c6t1:n=argmax\nt1:::tnP(t1:::tnjw1:::wn)\u0019argmax\nt1:::tnnY\ni=1emissionz}|{\nP(wijti)transitionz}|{\nP(tijti\u00001) (17.17)\nThe two parts of Eq. 17.17 correspond neatly to the Bemission probability andA\ntransition probability that we just de\ufb01ned above! 17.4 \u2022 HMM P ART-OF-SPEECH TAGGING 373\nfunction VITERBI (observations of len T,state-graph of len N)returns best-path ,path-prob\ncreate a path probability matrix viterbi[N,T]\nforeach state sfrom 1toNdo ; initialization step\nviterbi [s,1] ps\u0003bs(o1)\nbackpointer [s,1] 0\nforeach time step tfrom 2toTdo ; recursion step\nforeach state sfrom 1toNdo\nviterbi [s,t] Nmax\ns0=1viterbi [s0;t\u00001]\u0003as0;s\u0003bs(ot)\nbackpointer [s,t] Nargmax\ns0=1viterbi [s0;t\u00001]\u0003as0;s\u0003bs(ot)\nbestpathprob Nmax\ns=1viterbi [s;T] ; termination step\nbestpathpointer Nargmax\ns=1viterbi [s;T] ; termination step\nbestpath the path starting at state bestpathpointer , that follows backpointer[] to states back in time\nreturn bestpath ,bestpathprob\nFigure 17.10 Viterbi algorithm for \ufb01nding the optimal sequence of tags. Given an observation sequence and\nan HMM l= (A;B), the algorithm returns the state path through the HMM that assigns maximum likelihood\nto the observation sequence.\n17.4.5 The Viterbi Algorithm\nThe decoding algorithm for HMMs is the Viterbi algorithm shown in Fig. 17.10.Viterbi\nalgorithm\nAs an instance of dynamic programming , Viterbi resembles the dynamic program-\nming minimum edit distance algorithm of Chapter 2.\nThe Viterbi algorithm \ufb01rst sets up a probability matrix or lattice , with one col-\numn for each observation otand one row for each state in the state graph. Each col-\numn thus has a cell for each state qiin the single combined automaton. Figure 17.11\nshows an intuition of this lattice for the sentence Janet will back the bill .\nEach cell of the lattice, vt(j), represents the probability that the HMM is in state\njafter seeing the \ufb01rst tobservations and passing through the most probable state\nsequence q1;:::;qt\u00001, given the HMM l. The value of each cell vt(j)is computed\nby recursively taking the most probable path that could lead us to this cell. Formally,\neach cell expresses the probability\nvt(j) = max\nq1;:::;qt\u00001P(q1:::qt\u00001;o1;o2:::ot;qt=jjl) (17.18)\nWe represent the most probable path by taking the maximum over all possible\nprevious state sequences max\nq1;:::;qt\u00001. Like other dynamic programming algorithms,\nViterbi \ufb01lls each cell recursively. Given that we had already computed the probabil-\nity of being in every state at time t\u00001, we compute the Viterbi probability by taking\nthe most probable of the extensions of the paths that lead to the current cell. For a\ngiven state qjat time t, the value vt(j)is computed as\nvt(j) =Nmax\ni=1vt\u00001(i)ai jbj(ot) (17.19)\nThe three factors that are multiplied in Eq. 17.19 for extending the previous paths to\ncompute the Viterbi probability at time tare 374 CHAPTER 17 \u2022 S EQUENCE LABELING FOR PARTS OF SPEECH AND NAMED ENTITIES\nJJNNPNNPNNPMDMDMDMDVBVBJJJJJJNNNNRBRBRBRBDTDTDTDT\nNNPJanetwillbackthebillNNVBMDNNVBJJRB\nNNPDTNNVB\nFigure 17.11 A sketch of the lattice for Janet will back the bill , showing the possible tags\n(qi) for each word and highlighting the path corresponding to the correct tag sequence through\nthe hidden states. States (parts of speech) which have a zero probability of generating a\nparticular word according to the Bmatrix (such as the probability that a determiner DT will\nbe realized as Janet ) are greyed out.\nvt\u00001(i)theprevious Viterbi path probability from the previous time step\nai j thetransition probability from previous state qito current state qj\nbj(ot) thestate observation likelihood of the observation symbol otgiven\nthe current state j\n17.4.6 Working through an example\nLet\u2019s tag the sentence Janet will back the bill ; the goal is the correct series of tags\n(see also Fig. 17.11):\n(17.20) Janet/NNP will/MD back/VB the/DT bill/NN\nNNP MD VB JJ NN RB DT\n<s> 0.2767 0.0006 0.0031 0.0453 0.0449 0.0510 0.2026\nNNP 0.3777 0.0110 0.0009 0.0084 0.0584 0.0090 0.0025\nMD 0.0008 0.0002 0.7968 0.0005 0.0008 0.1698 0.0041\nVB 0.0322 0.0005 0.0050 0.0837 0.0615 0.0514 0.2231\nJJ 0.0366 0.0004 0.0001 0.0733 0.4509 0.0036 0.0036\nNN 0.0096 0.0176 0.0014 0.0086 0.1216 0.0177 0.0068\nRB 0.0068 0.0102 0.1011 0.1012 0.0120 0.0728 0.0479\nDT 0.1147 0.0021 0.0002 0.2157 0.4744 0.0102 0.0017\nFigure 17.12 TheAtransition probabilities P(tijti\u00001)computed from the WSJ corpus with-\nout smoothing. Rows are labeled with the conditioning event; thus P(V BjMD)is 0.7968.\n<s>is the start token.\nLet the HMM be de\ufb01ned by the two tables in Fig. 17.12 and Fig. 17.13. Fig-\nure 17.12 lists the ai jprobabilities for transitioning between the hidden states (part-\nof-speech tags). Figure 17.13 expresses the bi(ot)probabilities, the observation\nlikelihoods of words given tags. This table is (slightly simpli\ufb01ed) from counts in the\nWSJ corpus. So the word Janet only appears as an NNP, back has 4 possible parts 17.4 \u2022 HMM P ART-OF-SPEECH TAGGING 375\nJanet will back the bill\nNNP 0.000032 0 0 0.000048 0\nMD 0 0.308431 0 0 0\nVB 0 0.000028 0.000672 0 0.000028\nJJ 0 0 0.000340 0 0\nNN 0 0.000200 0.000223 0 0.002337\nRB 0 0 0.010446 0 0\nDT 0 0 0 0.506099 0\nFigure 17.13 Observation likelihoods Bcomputed from the WSJ corpus without smooth-\ning, simpli\ufb01ed slightly.\nof speech, and the word thecan appear as a determiner or as an NNP (in titles like\n\u201cSomewhere Over the Rainbow\u201d all words are tagged as NNP).\n\u03c0P(NNP|start) = .28* P(MD|MD)= 0*  P(MD|NNP).000009*.01  = .9e-8 v1(2)=.0006 x 0 = 0v1(1) = .28* .000032 = .000009\ntMDq2q1\no1Janetbillwillo2o3backVBJJv1(3)=.0031 x 0 = 0v1(4)= .045*0=0\no4  *  P(MD|VB) = 0 * P(MD|JJ)= 0P(VB|start) = .0031P(JJ |start) =.045\nbacktraceq3q4\ntheNNq5RBq6DTq7\nv2(2) =max * .308 =2.772e-8v2(5)=max * .0002 = .0000000001\nv2(3)=max * .000028 =     2.5e-13v3(6)=max * .0104v3(5)=max * .000223v3(4)=max * .00034v3(3)=max * .00067v1(5)v1(6)v1(7)\nv2(1)v2(4)v2(6)v2(7)\nbacktrace* P(RB|NN)* P(NN|NN)\nstartstartstartstartstarto5NNPP(MD|start) = .0006\nFigure 17.14 The \ufb01rst few entries in the individual state columns for the Viterbi algorithm. Each cell keeps\nthe probability of the best path so far and a pointer to the previous cell along that path. We have only \ufb01lled out\ncolumns 1 and 2; to avoid clutter most cells with value 0 are left empty. The rest is left as an exercise for the\nreader. After the cells are \ufb01lled in, backtracing from the endstate, we should be able to reconstruct the correct\nstate sequence NNP MD VB DT NN.\nFigure 17.14 shows a \ufb02eshed-out version of the sketch we saw in Fig. 17.11,\nthe Viterbi lattice for computing the best hidden state sequence for the observation\nsequence Janet will back the bill .\nThere are N=5 state columns. We begin in column 1 (for the word Janet ) by\nsetting the Viterbi value in each cell to the product of the ptransition probability (the\nstart probability for that state i, which we get from the <s>entry of Fig. 17.12), and 376 CHAPTER 17 \u2022 S EQUENCE LABELING FOR PARTS OF SPEECH AND NAMED ENTITIES\nthe observation likelihood of the word Janet given the tag for that cell. Most of the\ncells in the column are zero since the word Janet cannot be any of those tags. The\nreader should \ufb01nd this in Fig. 17.14.\nNext, each cell in the willcolumn gets updated. For each state, we compute the\nvalue viterbi [s;t]by taking the maximum over the extensions of all the paths from\nthe previous column that lead to the current cell according to Eq. 17.19. We have\nshown the values for the MD, VB, and NN cells. Each cell gets the max of the 7 val-\nues from the previous column, multiplied by the appropriate transition probability;\nas it happens in this case, most of them are zero from the previous column. The re-\nmaining value is multiplied by the relevant observation probability, and the (trivial)\nmax is taken. In this case the \ufb01nal value, 2.772e-8, comes from the NNP state at the\nprevious column. The reader should \ufb01ll in the rest of the lattice in Fig. 17.14 and\nbacktrace to see whether or not the Viterbi algorithm returns the gold state sequence\nNNP MD VB DT NN.\n17.5 Conditional Random Fields (CRFs)\nWhile the HMM is a useful and powerful model, it turns out that HMMs need a\nnumber of augmentations to achieve high accuracy. For example, in POS tagging\nas in other tasks, we often run into unknown words : proper names and acronymsunknown\nwords\nare created very often, and even new common nouns and verbs enter the language\nat a surprising rate. It would be great to have ways to add arbitrary features to\nhelp with this, perhaps based on capitalization or morphology (words starting with\ncapital letters are likely to be proper nouns, words ending with -edtend to be past\ntense (VBD or VBN), etc.) Or knowing the previous or following words might be a\nuseful feature (if the previous word is the, the current tag is unlikely to be a verb).\nAlthough we could try to hack the HMM to \ufb01nd ways to incorporate some of\nthese, in general it\u2019s hard for generative models like HMMs to add arbitrary features\ndirectly into the model in a clean way. We\u2019ve already seen a model for combining\narbitrary features in a principled way: log-linear models like the logistic regression\nmodel of Chapter 5! But logistic regression isn\u2019t a sequence model; it assigns a class\nto a single observation.\nLuckily, there is a discriminative sequence model based on log-linear models:\ntheconditional random \ufb01eld (CRF ). We\u2019ll describe here the linear chain CRF , CRF\nthe version of the CRF most commonly used for language processing, and the one\nwhose conditioning closely matches the HMM.\nAssuming we have a sequence of input words X=x1:::xnand want to compute\na sequence of output tags Y=y1:::yn. In an HMM to compute the best tag sequence\nthat maximizes P(YjX)we rely on Bayes\u2019 rule and the likelihood P(XjY):\n\u02c6Y=argmax\nYp(YjX)\n=argmax\nYp(XjY)p(Y)\n=argmax\nYY\nip(xijyi)Y\nip(yijyi\u00001) (17.21)\nIn a CRF, by contrast, we compute the posterior p(YjX)directly, training the CRF 17.5 \u2022 C ONDITIONAL RANDOM FIELDS (CRF S)377\nto discriminate among the possible tag sequences:\n\u02c6Y=argmax\nY2YP(YjX) (17.22)\nHowever, the CRF does not compute a probability for each tag at each time step. In-\nstead, at each time step the CRF computes log-linear functions over a set of relevant\nfeatures, and these local features are aggregated and normalized to produce a global\nprobability for the whole sequence.\nLet\u2019s introduce the CRF more formally, again using XandYas the input and\noutput sequences. A CRF is a log-linear model that assigns a probability to an\nentire output (tag) sequence Y, out of all possible sequences Y, given the entire input\n(word) sequence X. We can think of a CRF as like a giant sequential version of\nthe multinomial logistic regression algorithm we saw for text categorization. Recall\nthat we introduced the feature function fin regular multinomial logistic regression\nfor text categorization as a function of a tuple: the input text xand a single class y\n(page 86). In a CRF, we\u2019re dealing with a sequence, so the function Fmaps an entire\ninput sequence Xand an entire output sequence Yto a feature vector. Let\u2019s assume\nwe have Kfeatures, with a weight wkfor each feature Fk:\np(YjX) =exp KX\nk=1wkFk(X;Y)!\nX\nY02Yexp KX\nk=1wkFk(X;Y0)! (17.23)\nIt\u2019s common to also describe the same equation by pulling out the denominator into\na function Z(X):\np(YjX) =1\nZ(X)exp KX\nk=1wkFk(X;Y)!\n(17.24)\nZ(X) =X\nY02Yexp KX\nk=1wkFk(X;Y0)!\n(17.25)\nWe\u2019ll call these Kfunctions Fk(X;Y)global features , since each one is a property\nof the entire input sequence Xand output sequence Y. We compute them by decom-\nposing into a sum of local features for each position iinY:\nFk(X;Y) =nX\ni=1fk(yi\u00001;yi;X;i) (17.26)\nEach of these local features fkin a linear-chain CRF is allowed to make use of the\ncurrent output token yi, the previous output token yi\u00001, the entire input string X(or\nany subpart of it), and the current position i. This constraint to only depend on\nthe current and previous output tokens yiandyi\u00001are what characterizes a linear\nchain CRF . As we will see, this limitation makes it possible to use versions of thelinear chain\nCRF\nef\ufb01cient Viterbi and Forward-Backwards algorithms from the HMM. A general CRF,\nby contrast, allows a feature to make use of any output token, and are thus necessary\nfor tasks in which the decision depend on distant output tokens, like yi\u00004. General\nCRFs require more complex inference, and are less commonly used for language\nprocessing. 378 CHAPTER 17 \u2022 S EQUENCE LABELING FOR PARTS OF SPEECH AND NAMED ENTITIES\n17.5.1 Features in a CRF POS Tagger\nLet\u2019s look at some of these features in detail, since the reason to use a discriminative\nsequence model is that it\u2019s easier to incorporate a lot of features.2\nAgain, in a linear-chain CRF, each local feature fkat position ican depend on\nany information from: (yi\u00001;yi;X;i). So some legal features representing common\nsituations might be the following:\n1fxi=the;yi=DETg\n1fyi=PROPN, xi+1=Street ,yi\u00001=NUMg\n1fyi=VERB, yi\u00001=AUXg\nFor simplicity, we\u2019ll assume all CRF features take on the value 1 or 0. Above, we\nexplicitly use the notation 1fxgto mean \u201c1 if xis true, and 0 otherwise\u201d. From now\non, we\u2019ll leave off the 1when we de\ufb01ne features, but you can assume each feature\nhas it there implicitly.\nAlthough the idea of what features to use is done by the system designer by hand,\nthe speci\ufb01c features are automatically populated by using feature templates as wefeature\ntemplates\nbrie\ufb02y mentioned in Chapter 5. Here are some templates that only use information\nfrom (yi\u00001;yi;X;i):\nhyi;xii;hyi;yi\u00001i;hyi;xi\u00001;xi+2i\nThese templates automatically populate the set of features from every instance in\nthe training and test set. Thus for our example Janet/NNP will/MD back/VB the/DT\nbill/NN , when xiis the word back, the following features would be generated and\nhave the value 1 (we\u2019ve assigned them arbitrary feature numbers):\nf3743:yi= VB and xi= back\nf156:yi= VB and yi\u00001= MD\nf99732 :yi= VB and xi\u00001= will and xi+2= bill\nIt\u2019s also important to have features that help with unknown words. One of the\nmost important is word shape features, which represent the abstract letter pattern word shape\nof the word by mapping lower-case letters to \u2018x\u2019, upper-case to \u2018X\u2019, numbers to\n\u2019d\u2019, and retaining punctuation. Thus for example I.M.F. would map to X.X.X. and\nDC10-30 would map to XXdd-dd. A second class of shorter word shape features is\nalso used. In these features consecutive character types are removed, so words in all\ncaps map to X, words with initial-caps map to Xx, DC10-30 would be mapped to\nXd-d but I.M.F would still map to X.X.X. Pre\ufb01x and suf\ufb01x features are also useful.\nIn summary, here are some sample feature templates that help with unknown words:\nxicontains a particular pre\ufb01x (perhaps from all pre\ufb01xes of length \u00142)\nxicontains a particular suf\ufb01x (perhaps from all suf\ufb01xes of length \u00142)\nxi\u2019s word shape\nxi\u2019s short word shape\nFor example the word well-dressed might generate the following non-zero val-\nued feature values:\n2Because in HMMs all computation is based on the two probabilities P(tagjtag)andP(wordjtag), if\nwe want to include some source of knowledge into the tagging process, we must \ufb01nd a way to encode\nthe knowledge into one of these two probabilities. Each time we add a feature we have to do a lot of\ncomplicated conditioning which gets harder and harder as we have more and more such features. 17.5 \u2022 C ONDITIONAL RANDOM FIELDS (CRF S)379\npre\ufb01x( xi) =w\npre\ufb01x( xi) =we\nsuf\ufb01x( xi) =ed\nsuf\ufb01x( xi) =d\nword-shape( xi) =xxxx-xxxxxxx\nshort-word-shape( xi) =x-x\nThe known-word templates are computed for every word seen in the training\nset; the unknown word features can also be computed for all words in training, or\nonly on training words whose frequency is below some threshold. The result of the\nknown-word templates and word-signature features is a very large set of features.\nGenerally a feature cutoff is used in which features are thrown out if they have count\n<5 in the training set.\nRemember that in a CRF we don\u2019t learn weights for each of these local features\nfk. Instead, we \ufb01rst sum the values of each local feature (for example feature f3743)\nover the entire sentence, to create each global feature (for example F3743). It is those\nglobal features that will then be multiplied by weight w3743. Thus for training and\ninference there is always a \ufb01xed set of Kfeatures with Kweights, even though the\nlength of each sentence is different.\n17.5.2 Features for CRF Named Entity Recognizers\nA CRF for NER makes use of very similar features to a POS tagger, as shown in\nFigure 17.15.\nidentity of wi, identity of neighboring words\nembeddings for wi, embeddings for neighboring words\npart of speech of wi, part of speech of neighboring words\npresence of wiin agazetteer\nwicontains a particular pre\ufb01x (from all pre\ufb01xes of length \u00144)\nwicontains a particular suf\ufb01x (from all suf\ufb01xes of length \u00144)\nword shape of wi, word shape of neighboring words\nshort word shape of wi, short word shape of neighboring words\ngazetteer features\nFigure 17.15 Typical features for a feature-based NER system.\nOne feature that is especially useful for locations is a gazetteer , a list of place gazetteer\nnames, often providing millions of entries for locations with detailed geographical\nand political information.3This can be implemented as a binary feature indicating a\nphrase appears in the list. Other related resources like name-lists , for example from\nthe United States Census Bureau4, can be used, as can other entity dictionaries like\nlists of corporations or products, although they may not be as helpful as a gazetteer\n(Mikheev et al., 1999).\nThe sample named entity token L\u2019Occitane would generate the following non-\nzero valued feature values (assuming that L\u2019Occitane is neither in the gazetteer nor\nthe census).\n3www.geonames.org\n4www.census.gov 380 CHAPTER 17 \u2022 S EQUENCE LABELING FOR PARTS OF SPEECH AND NAMED ENTITIES\npre\ufb01x( xi) =L suf\ufb01x( xi) =tane\npre\ufb01x( xi) =L' suf\ufb01x( xi) =ane\npre\ufb01x( xi) =L'O suf\ufb01x( xi) =ne\npre\ufb01x( xi) =L'Oc suf\ufb01x( xi) =e\nword-shape( xi) =X'Xxxxxxxx short-word-shape( xi) =X'Xx\nFigure 17.16 illustrates the result of adding part-of-speech tags and some shape\ninformation to our earlier example.\nWords POS Short shape Gazetteer BIO Label\nJane NNP Xx 0 B-PER\nVillanueva NNP Xx 1 I-PER\nof IN x 0 O\nUnited NNP Xx 0 B-ORG\nAirlines NNP Xx 0 I-ORG\nHolding NNP Xx 0 I-ORG\ndiscussed VBD x 0 O\nthe DT x 0 O\nChicago NNP Xx 1 B-LOC\nroute NN x 0 O\n. . . 0 O\nFigure 17.16 Some NER features for a sample sentence, assuming that Chicago and Vil-\nlanueva are listed as locations in a gazetteer. We assume features only take on the values 0 or\n1, so the \ufb01rst POS feature, for example, would be represented as 1fPOS=NNPg.\n17.5.3 Inference and Training for CRFs\nHow do we \ufb01nd the best tag sequence \u02c6Yfor a given input X? We start with Eq. 17.22:\n\u02c6Y=argmax\nY2YP(YjX)\n=argmax\nY2Y1\nZ(X)exp KX\nk=1wkFk(X;Y)!\n(17.27)\n=argmax\nY2Yexp KX\nk=1wknX\ni=1fk(yi\u00001;yi;X;i)!\n(17.28)\n=argmax\nY2YKX\nk=1wknX\ni=1fk(yi\u00001;yi;X;i) (17.29)\n=argmax\nY2YnX\ni=1KX\nk=1wkfk(yi\u00001;yi;X;i) (17.30)\nWe can ignore the exp function and the denominator Z(X), as we do above, because\nexp doesn\u2019t change the argmax, and the denominator Z(X)is constant for a given\nobservation sequence X.\nHow should we decode to \ufb01nd this optimal tag sequence \u02c6 y? Just as with HMMs,\nwe\u2019ll turn to the Viterbi algorithm, which works because, like the HMM, the linear-\nchain CRF depends at each timestep on only one previous output token yi\u00001.\nConcretely, this involves \ufb01lling an N\u0002Tarray with the appropriate values, main-\ntaining backpointers as we proceed. As with HMM Viterbi, when the table is \ufb01lled,\nwe simply follow pointers back from the maximum value in the \ufb01nal column to\nretrieve the desired set of labels. 17.6 \u2022 E VALUATION OF NAMED ENTITY RECOGNITION 381\nThe requisite changes from HMM Viterbi have to do only with how we \ufb01ll each\ncell. Recall from Eq. 17.19 that the recursive step of the Viterbi equation computes\nthe Viterbi value of time tfor state jas\nvt(j) =Nmax\ni=1vt\u00001(i)ai jbj(ot); 1\u0014j\u0014N;1<t\u0014T (17.31)\nwhich is the HMM implementation of\nvt(j) =Nmax\ni=1vt\u00001(i)P(sjjsi)P(otjsj)1\u0014j\u0014N;1<t\u0014T (17.32)\nThe CRF requires only a slight change to this latter formula, replacing the aandb\nprior and likelihood probabilities with the CRF features:\nvt(j) =Nmax\ni=1vt\u00001(i)KX\nk=1wkfk(yt\u00001;yt;X;t)1\u0014j\u0014N;1<t\u0014T(17.33)\nLearning in CRFs relies on the same supervised learning algorithms we presented\nfor logistic regression. Given a sequence of observations, feature functions, and cor-\nresponding outputs, we use stochastic gradient descent to train the weights to maxi-\nmize the log-likelihood of the training corpus. The local nature of linear-chain CRFs\nmeans that the forward-backward algorithm introduced for HMMs in Appendix A\ncan be extended to a CRF version that will ef\ufb01ciently compute the necessary deriva-\ntives. As with logistic regression, L1 or L2 regularization is important.\n17.6 Evaluation of Named Entity Recognition\nPart-of-speech taggers are evaluated by the standard metric of accuracy . Named\nentity recognizers are evaluated by recall ,precision , and F1measure . Recall that\nrecall is the ratio of the number of correctly labeled responses to the total that should\nhave been labeled; precision is the ratio of the number of correctly labeled responses\nto the total labeled; and F-measure is the harmonic mean of the two.\nTo know if the difference between the F 1scores of two NER systems is a signif-\nicant difference, we use the paired bootstrap test, or the similar randomization test\n(Section 4.9).\nFor named entity tagging, the entity rather than the word is the unit of response.\nThus in the example in Fig. 17.16, the two entities Jane Villanueva andUnited Air-\nlines Holding and the non-entity discussed would each count as a single response.\nThe fact that named entity tagging has a segmentation component which is not\npresent in tasks like text categorization or part-of-speech tagging causes some prob-\nlems with evaluation. For example, a system that labeled Jane but not Jane Vil-\nlanueva as a person would cause two errors, a false positive for O and a false nega-\ntive for I-PER. In addition, using entities as the unit of response but words as the unit\nof training means that there is a mismatch between the training and test conditions.\n17.7 Further Details\nIn this section we summarize a few remaining details of the data and models for\npart-of-speech tagging and NER, beginning with data. Since the algorithms we have 382 CHAPTER 17 \u2022 S EQUENCE LABELING FOR PARTS OF SPEECH AND NAMED ENTITIES\npresented are supervised, having labeled data is essential for training and testing. A\nwide variety of datasets exist for part-of-speech tagging and/or NER. The Universal\nDependencies (UD) dataset (de Marneffe et al., 2021) has POS tagged corpora in\nover a hundred languages, as do the Penn Treebanks in English, Chinese, and Arabic.\nOntoNotes has corpora labeled for named entities in English, Chinese, and Arabic\n(Hovy et al., 2006). Named entity tagged corpora are also available in particular\ndomains, such as for biomedical (Bada et al., 2012) and literary text (Bamman et al.,\n2019).\n17.7.1 Rule-based Methods\nWhile machine learned (neural or CRF) sequence models are the norm in academic\nresearch, commercial approaches to NER are often based on pragmatic combina-\ntions of lists and rules, with some smaller amount of supervised machine learning\n(Chiticariu et al., 2013). For example in the IBM System T architecture, a user\nspeci\ufb01es declarative constraints for tagging tasks in a formal query language that\nincludes regular expressions, dictionaries, semantic constraints, and other operators,\nwhich the system compiles into an ef\ufb01cient extractor (Chiticariu et al., 2018).\nOne common approach is to make repeated rule-based passes over a text, starting\nwith rules with very high precision but low recall, and, in subsequent stages, using\nmachine learning methods that take the output of the \ufb01rst pass into account (an\napproach \ufb01rst worked out for coreference (Lee et al., 2017a)):\n1. First, use high-precision rules to tag unambiguous entity mentions.\n2. Then, search for substring matches of the previously detected names.\n3. Use application-speci\ufb01c name lists to \ufb01nd likely domain-speci\ufb01c mentions.\n4. Finally, apply supervised sequence labeling techniques that use tags from pre-\nvious stages as additional features.\nRule-based methods were also the earliest methods for part-of-speech tagging.\nRule-based taggers like the English Constraint Grammar system (Karlsson et al.\n1995, V outilainen 1999) use a two-stage formalism invented in the 1950s and 1960s:\n(1) a morphological analyzer with tens of thousands of word stem entries returns all\nparts of speech for a word, then (2) a large set of thousands of constraints are applied\nto the input sentence to rule out parts of speech inconsistent with the context.\n17.7.2 POS Tagging for Morphologically Rich Languages\nAugmentations to tagging algorithms become necessary when dealing with lan-\nguages with rich morphology like Czech, Hungarian and Turkish.\nThese productive word-formation processes result in a large vocabulary for these\nlanguages: a 250,000 word token corpus of Hungarian has more than twice as many\nword types as a similarly sized corpus of English (Oravecz and Dienes, 2002), while\na 10 million word token corpus of Turkish contains four times as many word types\nas a similarly sized English corpus (Hakkani-T \u00a8ur et al., 2002). Large vocabular-\nies mean many unknown words, and these unknown words cause signi\ufb01cant per-\nformance degradations in a wide variety of languages (including Czech, Slovene,\nEstonian, and Romanian) (Haji \u02c7c, 2000).\nHighly in\ufb02ectional languages also have much more information than English\ncoded in word morphology, like case (nominative, accusative, genitive) or gender\n(masculine, feminine). Because this information is important for tasks like pars-\ning and coreference resolution, part-of-speech taggers for morphologically rich lan- 17.8 \u2022 S UMMARY 383\nguages need to label words with case and gender information. Tagsets for morpho-\nlogically rich languages are therefore sequences of morphological tags rather than a\nsingle primitive tag. Here\u2019s a Turkish example, in which the word izinhas three pos-\nsible morphological/part-of-speech tags and meanings (Hakkani-T \u00a8ur et al., 2002):\n1. Yerdeki izintemizlenmesi gerek. iz + Noun+A3sg+Pnon+Gen\nThe trace on the \ufb02oor should be cleaned.\n2.\u00a8Uzerinde parmak izinkalmis \u00b8. iz + Noun+A3sg+P2sg+Nom\nYour \ufb01nger print is left on (it).\n3. Ic \u00b8eri girmek ic \u00b8in izinalman gerekiyor. izin + Noun+A3sg+Pnon+Nom\nYou need permission to enter.\nUsing a morphological parse sequence like Noun+A3sg+Pnon+Gen as the part-\nof-speech tag greatly increases the number of parts of speech, and so tagsets can\nbe 4 to 10 times larger than the 50\u2013100 tags we have seen for English. With such\nlarge tagsets, each word needs to be morphologically analyzed to generate the list\nof possible morphological tag sequences (part-of-speech tags) for the word. The\nrole of the tagger is then to disambiguate among these tags. This method also helps\nwith unknown words since morphological parsers can accept unknown stems and\nstill segment the af\ufb01xes properly.\n17.8 Summary\nThis chapter introduced parts of speech andnamed entities , and the tasks of part-\nof-speech tagging andnamed entity recognition :\n\u2022 Languages generally have a small set of closed class words that are highly\nfrequent, ambiguous, and act as function words , and open-class words like\nnouns ,verbs ,adjectives . Various part-of-speech tagsets exist, of between 40\nand 200 tags.\n\u2022Part-of-speech tagging is the process of assigning a part-of-speech label to\neach of a sequence of words.\n\u2022Named entities are words for proper nouns referring mainly to people, places,\nand organizations, but extended to many other types that aren\u2019t strictly entities\nor even proper nouns.\n\u2022 Two common approaches to sequence modeling are a generative approach,\nHMM tagging, and a discriminative approach, CRF tagging. We will see a\nneural approach in following chapters.\n\u2022 The probabilities in HMM taggers are estimated by maximum likelihood es-\ntimation on tag-labeled training corpora. The Viterbi algorithm is used for\ndecoding , \ufb01nding the most likely tag sequence\n\u2022Conditional Random Fields orCRF taggers train a log-linear model that can\nchoose the best tag sequence given an observation sequence, based on features\nthat condition on the output tag, the prior output tag, the entire input sequence,\nand the current timestep. They use the Viterbi algorithm for inference, to\nchoose the best sequence of tags, and a version of the Forward-Backward\nalgorithm (see Appendix A) for training, 384 CHAPTER 17 \u2022 S EQUENCE LABELING FOR PARTS OF SPEECH AND NAMED ENTITIES\nBibliographical and Historical Notes\nWhat is probably the earliest part-of-speech tagger was part of the parser in Zellig\nHarris\u2019s Transformations and Discourse Analysis Project (TDAP), implemented be-\ntween June 1958 and July 1959 at the University of Pennsylvania (Harris, 1962),\nalthough earlier systems had used part-of-speech dictionaries. TDAP used 14 hand-\nwritten rules for part-of-speech disambiguation; the use of part-of-speech tag se-\nquences and the relative frequency of tags for a word pre\ufb01gures modern algorithms.\nThe parser was implemented essentially as a cascade of \ufb01nite-state transducers; see\nJoshi and Hopely (1999) and Karttunen (1999) for a reimplementation.\nThe Computational Grammar Coder (CGC) of Klein and Simmons (1963) had\nthree components: a lexicon, a morphological analyzer, and a context disambigua-\ntor. The small 1500-word lexicon listed only function words and other irregular\nwords. The morphological analyzer used in\ufb02ectional and derivational suf\ufb01xes to as-\nsign part-of-speech classes. These were run over words to produce candidate parts\nof speech which were then disambiguated by a set of 500 context rules by relying on\nsurrounding islands of unambiguous words. For example, one rule said that between\nan ARTICLE and a VERB, the only allowable sequences were ADJ-NOUN, NOUN-\nADVERB, or NOUN-NOUN. The TAGGIT tagger (Greene and Rubin, 1971) used\nthe same architecture as Klein and Simmons (1963), with a bigger dictionary and\nmore tags (87). TAGGIT was applied to the Brown corpus and, according to Francis\nand Ku \u02c7cera (1982, p. 9), accurately tagged 77% of the corpus; the remainder of the\nBrown corpus was then tagged by hand. All these early algorithms were based on\na two-stage architecture in which a dictionary was \ufb01rst used to assign each word a\nset of potential parts of speech, and then lists of handwritten disambiguation rules\nwinnowed the set down to a single part of speech per word.\nProbabilities were used in tagging by Stolz et al. (1965) and a complete proba-\nbilistic tagger with Viterbi decoding was sketched by Bahl and Mercer (1976). The\nLancaster-Oslo/Bergen (LOB) corpus, a British English equivalent of the Brown cor-\npus, was tagged in the early 1980\u2019s with the CLAWS tagger (Marshall 1983; Mar-\nshall 1987; Garside 1987), a probabilistic algorithm that approximated a simpli\ufb01ed\nHMM tagger. The algorithm used tag bigram probabilities, but instead of storing the\nword likelihood of each tag, the algorithm marked tags either as rare (P(tagjword)<\n:01)infrequent (P(tagjword)<:10) or normally frequent (P(tagjword)>:10).\nDeRose (1988) developed a quasi-HMM algorithm, including the use of dy-\nnamic programming, although computing P(tjw)P(w)instead of P(wjt)P(w). The\nsame year, the probabilistic PARTS tagger of Church 1988, 1989 was probably the\n\ufb01rst implemented HMM tagger, described correctly in Church (1989), although\nChurch (1988) also described the computation incorrectly as P(tjw)P(w)instead\nofP(wjt)P(w). Church (p.c.) explained that he had simpli\ufb01ed for pedagogical pur-\nposes because using the probability P(tjw)made the idea seem more understandable\nas \u201cstoring a lexicon in an almost standard form\u201d.\nLater taggers explicitly introduced the use of the hidden Markov model (Kupiec\n1992; Weischedel et al. 1993; Sch \u00a8utze and Singer 1994). Merialdo (1994) showed\nthat fully unsupervised EM didn\u2019t work well for the tagging task and that reliance\non hand-labeled data was important. Charniak et al. (1993) showed the importance\nof the most frequent tag baseline; the 92.3% number we give above was from Abney\net al. (1999). See Brants (2000) for HMM tagger implementation details, includ-\ning the extension to trigram contexts, and the use of sophisticated unknown word\nfeatures; its performance is still close to state of the art taggers. EXERCISES 385\nLog-linear models for POS tagging were introduced by Ratnaparkhi (1996),\nwho introduced a system called MXPOST which implemented a maximum entropy\nMarkov model (MEMM), a slightly simpler version of a CRF. Around the same\ntime, sequence labelers were applied to the task of named entity tagging, \ufb01rst with\nHMMs (Bikel et al., 1997) and MEMMs (McCallum et al., 2000), and then once\nCRFs were developed (Lafferty et al. 2001), they were also applied to NER (Mc-\nCallum and Li, 2003). A wide exploration of features followed (Zhou et al., 2005).\nNeural approaches to NER mainly follow from the pioneering results of Collobert\net al. (2011), who applied a CRF on top of a convolutional net. BiLSTMs with word\nand character-based embeddings as input followed shortly and became a standard\nneural algorithm for NER (Huang et al. 2015, Ma and Hovy 2016, Lample et al.\n2016) followed by the more recent use of Transformers and BERT.\nThe idea of using letter suf\ufb01xes for unknown words is quite old; the early Klein\nand Simmons (1963) system checked all \ufb01nal letter suf\ufb01xes of lengths 1-5. The un-\nknown word features described on page 378 come mainly from Ratnaparkhi (1996),\nwith augmentations from Toutanova et al. (2003) and Manning (2011).\nState of the art POS taggers use neural algorithms, either bidirectional RNNs or\nTransformers like BERT; see Chapter 8 to Chapter 11. HMM (Brants 2000; Thede\nand Harper 1999) and CRF tagger accuracies are likely just a tad lower.\nManning (2011) investigates the remaining 2.7% of errors in a high-performing\ntagger (Toutanova et al., 2003). He suggests that a third or half of these remaining\nerrors are due to errors or inconsistencies in the training data, a third might be solv-\nable with richer linguistic models, and for the remainder the task is underspeci\ufb01ed\nor unclear.\nSupervised tagging relies heavily on in-domain training data hand-labeled by\nexperts. Ways to relax this assumption include unsupervised algorithms for cluster-\ning words into part-of-speech-like classes, summarized in Christodoulopoulos et al.\n(2010), and ways to combine labeled and unlabeled data, for example by co-training\n(Clark et al. 2003; S\u00f8gaard 2010).\nSee Householder (1995) for historical notes on parts of speech, and Sampson\n(1987) and Garside et al. (1997) on the provenance of the Brown and other tagsets.\nExercises\n17.1 Find one tagging error in each of the following sentences that are tagged with\nthe Penn Treebank tagset:\n1. I/PRP need/VBP a/DT \ufb02ight/NN from/IN Atlanta/NN\n2. Does/VBZ this/DT \ufb02ight/NN serve/VB dinner/NNS\n3. I/PRP have/VB a/DT friend/NN living/VBG in/IN Denver/NNP\n4. Can/VBP you/PRP list/VB the/DT nonstop/JJ afternoon/NN \ufb02ights/NNS\n17.2 Use the Penn Treebank tagset to tag each word in the following sentences\nfrom Damon Runyon\u2019s short stories. You may ignore punctuation. Some of\nthese are quite dif\ufb01cult; do your best.\n1. It is a nice night.\n2. This crap game is over a garage in Fifty-second Street. . .\n3. . . . Nobody ever takes the newspapers she sells . . .\n4. He is a tall, skinny guy with a long, sad, mean-looking kisser, and a\nmournful voice. 386 CHAPTER 17 \u2022 S EQUENCE LABELING FOR PARTS OF SPEECH AND NAMED ENTITIES\n5. . . . I am sitting in Mindy\u2019s restaurant putting on the ge\ufb01llte \ufb01sh, which is\na dish I am very fond of, . . .\n6. When a guy and a doll get to taking peeks back and forth at each other,\nwhy there you are indeed.\n17.3 Now compare your tags from the previous exercise with one or two friend\u2019s\nanswers. On which words did you disagree the most? Why?\n17.4 Implement the \u201cmost likely tag\u201d baseline. Find a POS-tagged training set,\nand use it to compute for each word the tag that maximizes p(tjw). You will\nneed to implement a simple tokenizer to deal with sentence boundaries. Start\nby assuming that all unknown words are NN and compute your error rate on\nknown and unknown words. Now write at least \ufb01ve rules to do a better job of\ntagging unknown words, and show the difference in error rates.\n17.5 Build a bigram HMM tagger. You will need a part-of-speech-tagged corpus.\nFirst split the corpus into a training set and test set. From the labeled training\nset, train the transition and observation probabilities of the HMM tagger di-\nrectly on the hand-tagged data. Then implement the Viterbi algorithm so you\ncan decode a test sentence. Now run your algorithm on the test set. Report its\nerror rate and compare its performance to the most frequent tag baseline.\n17.6 Do an error analysis of your tagger. Build a confusion matrix and investigate\nthe most frequent errors. Propose some features for improving the perfor-\nmance of your tagger on these errors.\n17.7 Develop a set of regular expressions to recognize the character shape features\ndescribed on page 378.\n17.8 The BIO and other labeling schemes given in this chapter aren\u2019t the only\npossible one. For example, the Btag can be reserved only for those situations\nwhere an ambiguity exists between adjacent entities. Propose a new set of\nBIOtags for use with your NER system. Experiment with it and compare its\nperformance with the schemes presented in this chapter.\n17.9 Names of works of art (books, movies, video games, etc.) are quite different\nfrom the kinds of named entities we\u2019ve discussed in this chapter. Collect a\nlist of names of works of art from a particular category from a Web-based\nsource (e.g., gutenberg.org, amazon.com, imdb.com, etc.). Analyze your list\nand give examples of ways that the names in it are likely to be problematic for\nthe techniques described in this chapter.\n17.10 Develop an NER system speci\ufb01c to the category of names that you collected\nin the last exercise. Evaluate your system on a collection of text likely to\ncontain instances of these named entities. CHAPTER\n18Context-Free Grammars and\nConstituency Parsing\nBecause the Night by Bruce Springsteen and Patti Smith\nThe Fire Next Time by James Baldwin\nIf on a winter\u2019s night a traveler by Italo Calvino\nLove Actually by Richard Curtis\nSuddenly Last Summer by Tennessee Williams\nA Scanner Darkly by Philip K. Dick\nSix titles that are not constituents, from Geoffrey K. Pullum on\nLanguage Log (who was pointing out their incredible rarity).\nOne morning I shot an elephant in my pajamas.\nHow he got into my pajamas I don\u2019t know.\nGroucho Marx, Animal Crackers , 1930\nThe study of grammar has an ancient pedigree. The grammar of Sanskrit was\ndescribed by the Indian grammarian P \u00afan.ini sometime between the 7th and 4th cen-\nturies BCE, in his famous treatise the As .t.\u00afadhy \u00afay\u00af\u0131 (\u20188 books\u2019). And our word syntax syntax\ncomes from the Greek s\u00b4yntaxis , meaning \u201csetting out together or arrangement\u201d, and\nrefers to the way words are arranged together. We have seen syntactic notions in pre-\nvious chapters like the use of part-of-speech categories (Chapter 17). In this chapter\nand the next one we introduce formal models for capturing more sophisticated no-\ntions of grammatical structure and algorithms for parsing these structures.\nOur focus in this chapter is context-free grammars and the CKY algorithm\nfor parsing them. Context-free grammars are the backbone of many formal mod-\nels of the syntax of natural language (and, for that matter, of computer languages).\nSyntactic parsing is the task of assigning a syntactic structure to a sentence. Parse\ntrees (whether for context-free grammars or for the dependency or CCG formalisms\nwe introduce in following chapters) can be used in applications such as grammar\nchecking : sentence that cannot be parsed may have grammatical errors (or at least\nbe hard to read). Parse trees can be an intermediate stage of representation for for-\nmal semantic analysis . And parsers and the grammatical structure they assign a\nsentence are a useful text analysis tool for text data science applications that require\nmodeling the relationship of elements in sentences.\nIn this chapter we introduce context-free grammars, give a small sample gram-\nmar of English, introduce more formal de\ufb01nitions of context-free grammars and\ngrammar normal form, and talk about treebanks : corpora that have been anno-\ntated with syntactic structure. We then discuss parse ambiguity and the problems\nit presents, and turn to parsing itself, giving the famous Cocke-Kasami-Younger\n(CKY) algorithm (Kasami 1965, Younger 1967), the standard dynamic program-\nming approach to syntactic parsing. The CKY algorithm returns an ef\ufb01cient repre-\nsentation of the set of parse trees for a sentence, but doesn\u2019t tell us which parse tree\nis the right one. For that, we need to augment CKY with scores for each possible\nconstituent. We\u2019ll see how to do this with neural span-based parsers. Finally, we\u2019ll\nintroduce the standard set of metrics for evaluating parser accuracy. 388 CHAPTER 18 \u2022 C ONTEXT -FREE GRAMMARS AND CONSTITUENCY PARSING\n18.1 Constituency\nSyntactic constituency is the idea that groups of words can behave as single units,\nor constituents. Part of developing a grammar involves building an inventory of the\nconstituents in the language. How do words group together in English? Consider\nthenoun phrase , a sequence of words surrounding at least one noun. Here are some noun phrase\nexamples of noun phrases (thanks to Damon Runyon):\nHarry the Horse a high-class spot such as Mindy\u2019s\nthe Broadway coppers the reason he comes into the Hot Box\nthey three parties from Brooklyn\nWhat evidence do we have that these words group together (or \u201cform constituents\u201d)?\nOne piece of evidence is that they can all appear in similar syntactic environments,\nfor example, before a verb.\nthree parties from Brooklyn arrive . . .\na high-class spot such as Mindy\u2019s attracts . . .\nthe Broadway coppers love. . .\nthey sit\nBut while the whole noun phrase can occur before a verb, this is not true of each\nof the individual words that make up a noun phrase. The following are not grammat-\nical sentences of English (recall that we use an asterisk (*) to mark fragments that\nare not grammatical English sentences):\n*from arrive . . .*asattracts . . .\n*the is. . . *spot sat. . .\nThus, to correctly describe facts about the ordering of these words in English, we\nmust be able to say things like \u201c Noun Phrases can occur before verbs \u201d. Let\u2019s now\nsee how to do this in a more formal way!\n18.2 Context-Free Grammars\nA widely used formal system for modeling constituent structure in natural lan-\nguage is the context-free grammar , orCFG . Context-free grammars are also called CFG\nphrase-structure grammars , and the formalism is equivalent to Backus-Naur form ,\norBNF . The idea of basing a grammar on constituent structure dates back to the psy-\nchologist Wilhelm Wundt (1900) but was not formalized until Chomsky (1956) and,\nindependently, Backus (1959).\nA context-free grammar consists of a set of rules orproductions , each of which rules\nexpresses the ways that symbols of the language can be grouped and ordered to-\ngether, and a lexicon of words and symbols. For example, the following productions lexicon\nexpress that an NP(ornoun phrase ) can be composed of either a ProperNoun or NP\na determiner ( Det) followed by a Nominal ; aNominal in turn can consist of one or 18.2 \u2022 C ONTEXT -FREE GRAMMARS 389\nmore Noun s.1\nNP!Det Nominal\nNP!ProperNoun\nNominal!NounjNominal Noun\nContext-free rules can be hierarchically embedded, so we can combine the previous\nrules with others, like the following, that express facts about the lexicon:\nDet!a\nDet!the\nNoun!\ufb02ight\nThe symbols that are used in a CFG are divided into two classes. The symbols\nthat correspond to words in the language (\u201cthe\u201d, \u201cnightclub\u201d) are called terminal terminal\nsymbols; the lexicon is the set of rules that introduce these terminal symbols. The\nsymbols that express abstractions over these terminals are called non-terminals . In non-terminal\neach context-free rule, the item to the right of the arrow ( !) is an ordered list of one\nor more terminals and non-terminals; to the left of the arrow is a single non-terminal\nsymbol expressing some cluster or generalization. The non-terminal associated with\neach word in the lexicon is its lexical category, or part of speech.\nA CFG can be thought of in two ways: as a device for generating sentences\nand as a device for assigning a structure to a given sentence. Viewing a CFG as a\ngenerator, we can read the !arrow as \u201crewrite the symbol on the left with the string\nof symbols on the right\u201d.\nSo starting from the symbol: NP\nwe can use our \ufb01rst rule to rewrite NPas: Det Nominal\nand then rewrite Nominal as: Noun\nand \ufb01nally rewrite these parts-of-speech as: a \ufb02ight\nWe say the string a \ufb02ight can be derived from the non-terminal NP. Thus, a CFG\ncan be used to generate a set of strings. This sequence of rule expansions is called a\nderivation of the string of words. It is common to represent a derivation by a parse derivation\ntree (commonly shown inverted with the root at the top). Figure 18.1 shows the tree parse tree\nrepresentation of this derivation.\nNP\nNom\nNoun\n\ufb02ightDet\na\nFigure 18.1 A parse tree for \u201ca \ufb02ight\u201d.\nIn the parse tree shown in Fig. 18.1, we can say that the node NPdominates dominates\nall the nodes in the tree ( Det,Nom ,Noun ,a,\ufb02ight ). We can say further that it\nimmediately dominates the nodes DetandNom .\nThe formal language de\ufb01ned by a CFG is the set of strings that are derivable\nfrom the designated start symbol . Each grammar must have one designated start start symbol\n1When talking about these rules we can pronounce the rightarrow !as \u201cgoes to\u201d, and so we might\nread the \ufb01rst rule above as \u201cNP goes to Det Nominal\u201d. 390 CHAPTER 18 \u2022 C ONTEXT -FREE GRAMMARS AND CONSTITUENCY PARSING\nsymbol, which is often called S. Since context-free grammars are often used to de\ufb01ne\nsentences, Sis usually interpreted as the \u201csentence\u201d node, and the set of strings that\nare derivable from Sis the set of sentences in some simpli\ufb01ed version of English.\nLet\u2019s add a few additional rules to our inventory. The following rule expresses\nthe fact that a sentence can consist of a noun phrase followed by a verb phrase : verb phrase\nS!NP VP I prefer a morning \ufb02ight\nA verb phrase in English consists of a verb followed by assorted other things;\nfor example, one kind of verb phrase consists of a verb followed by a noun phrase:\nVP!Verb NP prefer a morning \ufb02ight\nOr the verb may be followed by a noun phrase and a prepositional phrase:\nVP!Verb NP PP leave Boston in the morning\nOr the verb phrase may have a verb followed by a prepositional phrase alone:\nVP!Verb PP leaving on Thursday\nA prepositional phrase generally has a preposition followed by a noun phrase.\nFor example, a common type of prepositional phrase in the ATIS corpus is used to\nindicate location or direction:\nPP!Preposition NP from Los Angeles\nTheNPinside a PPneed not be a location; PPs are often used with times and\ndates, and with other nouns as well; they can be arbitrarily complex. Here are ten\nexamples from the ATIS corpus:\nto Seattle on these \ufb02ights\nin Minneapolis about the ground transportation in Chicago\non Wednesday of the round trip \ufb02ight on United Airlines\nin the evening of the AP \ufb01fty seven \ufb02ight\non the ninth of July with a stopover in Nashville\nFigure 18.2 gives a sample lexicon, and Fig. 18.3 summarizes the grammar rules\nwe\u2019ve seen so far, which we\u2019ll call L0. Note that we can use the or-symbol jto\nindicate that a non-terminal has alternate possible expansions.\nNoun!\ufb02ightsj\ufb02ightjbreezejtripjmorning\nVerb!isjpreferjlikejneedjwantj\ufb02yjdo\nAdjective!cheapestjnon-stopj\ufb01rstjlatest\njotherjdirect\nPronoun!mejIjyoujit\nProper-Noun!AlaskajBaltimorejLos Angeles\njChicagojUnitedjAmerican\nDeterminer!thejajanjthisjthesejthat\nPreposition!fromjtojonjnearjin\nConjunction!andjorjbut\nFigure 18.2 The lexicon for L0.\nWe can use this grammar to generate sentences of this \u201cATIS-language\u201d. We\nstart with S, expand it to NP VP , then choose a random expansion of NP(let\u2019s say, to 18.2 \u2022 C ONTEXT -FREE GRAMMARS 391\nGrammar Rules Examples\nS!NP VP I + want a morning \ufb02ight\nNP!Pronoun I\njProper-Noun Los Angeles\njDet Nominal a + \ufb02ight\nNominal!Nominal Noun morning + \ufb02ight\njNoun \ufb02ights\nVP!Verb do\njVerb NP want + a \ufb02ight\njVerb NP PP leave + Boston + in the morning\njVerb PP leaving + on Thursday\nPP!Preposition NP from + Los Angeles\nFigure 18.3 The grammar for L0, with example phrases for each rule.\nS\nVP\nNP\nNom\nNoun\n\ufb02ightNom\nNoun\nmorningDet\naVerb\npreferNP\nPro\nI\nFigure 18.4 The parse tree for \u201cI prefer a morning \ufb02ight\u201d according to grammar L0.\nI), and a random expansion of VP(let\u2019s say, to Verb NP ), and so on until we generate\nthe string I prefer a morning \ufb02ight . Figure 18.4 shows a parse tree that represents a\ncomplete derivation of I prefer a morning \ufb02ight .\nWe can also represent a parse tree in a more compact format called bracketed\nnotation ; here is the bracketed representation of the parse tree of Fig. 18.4:bracketed\nnotation\n(18.1) [ S[NP[ProI]] [VP[Vprefer] [ NP[Deta] [Nom [Nmorning] [ Nom [N\ufb02ight]]]]]]\nA CFG like that of L0de\ufb01nes a formal language. Sentences (strings of words)\nthat can be derived by a grammar are in the formal language de\ufb01ned by that gram-\nmar, and are called grammatical sentences. Sentences that cannot be derived by grammatical\na given formal grammar are not in the language de\ufb01ned by that grammar and are\nreferred to as ungrammatical . This hard line between \u201cin\u201d and \u201cout\u201d characterizes ungrammatical\nall formal languages but is only a very simpli\ufb01ed model of how natural languages\nreally work. This is because determining whether a given sentence is part of a given\nnatural language (say, English) often depends on the context. In linguistics, the use\nof formal languages to model natural languages is called generative grammar sincegenerative\ngrammar\nthe language is de\ufb01ned by the set of possible sentences \u201cgenerated\u201d by the grammar.\n(Note that this is a different sense of the word \u2018generate\u2019 than when we talk about 392 CHAPTER 18 \u2022 C ONTEXT -FREE GRAMMARS AND CONSTITUENCY PARSING\nlanguage models generating text.)\n18.2.1 Formal De\ufb01nition of Context-Free Grammar\nWe conclude this section with a quick, formal description of a context-free gram-\nmar and the language it generates. A context-free grammar Gis de\ufb01ned by four\nparameters: N;S;R;S(technically it is a \u201c4-tuple\u201d).\nNa set of non-terminal symbols (orvariables )\nSa set of terminal symbols (disjoint from N)\nRa set of rules or productions, each of the form A!b,\nwhere Ais a non-terminal,\nbis a string of symbols from the in\ufb01nite set of strings (S[N)\u0003\nSa designated start symbol and a member of N\nFor the remainder of the book we adhere to the following conventions when dis-\ncussing the formal properties of context-free grammars (as opposed to explaining\nparticular facts about English or other languages).\nCapital letters like A,B, and S Non-terminals\nS The start symbol\nLower-case Greek letters like a,b, and g Strings drawn from (S[N)\u0003\nLower-case Roman letters like u,v, and w Strings of terminals\nA language is de\ufb01ned through the concept of derivation. One string derives an-\nother one if it can be rewritten as the second one by some series of rule applications.\nMore formally, following Hopcroft and Ullman (1979),\nifA!bis a production of Randaandgare any strings in the set\n(S[N)\u0003, then we say that aAgdirectly derives abg , oraAg)abg . directly derives\nDerivation is then a generalization of direct derivation:\nLeta1;a2;:::; ambe strings in (S[N)\u0003;m\u00151, such that\na1)a2;a2)a3;:::;am\u00001)am\nWe say that a1derives am, ora1\u0003)am. derives\nWe can then formally de\ufb01ne the language LGgenerated by a grammar Gas the\nset of strings composed of terminal symbols that can be derived from the designated\nstart symbol S.\nLG=fwjwis inS\u0003andS\u0003)wg\nThe problem of mapping from a string of words to its parse tree is called syn-\ntactic parsing , as we\u2019ll see in Section 18.6.syntactic\nparsing\n18.3 Treebanks\nA corpus in which every sentence is annotated with a parse tree is called a treebank . treebank 18.3 \u2022 T REEBANKS 393\nTreebanks play an important role in parsing as well as in linguistic investigations of\nsyntactic phenomena.\nTreebanks are generally made by running a parser over each sentence and then\nhaving the resulting parse hand-corrected by human linguists. Figure 18.5 shows\nsentences from the Penn Treebank project, which includes various treebanks in Penn Treebank\nEnglish, Arabic, and Chinese. The Penn Treebank part-of-speech tagset was de\ufb01ned\nin Chapter 17, but we\u2019ll see minor formatting differences across treebanks. The use\nof LISP-style parenthesized notation for trees is extremely common and resembles\nthe bracketed notation we saw earlier in (18.1). For those who are not familiar with\nit we show a standard node-and-line tree representation in Fig. 18.6.\n((S\n(NP-SBJ (DT That)\n(JJ cold) (, ,)\n(JJ empty) (NN sky) )\n(VP (VBD was)\n(ADJP-PRD (JJ full)\n(PP (IN of)\n(NP (NN fire)\n(CC and)\n(NN light) ))))\n(. .) ))((S\n(NP-SBJ The/DT flight/NN )\n(VP should/MD\n(VP arrive/VB\n(PP-TMP at/IN\n(NP eleven/CD a.m/RB ))\n(NP-TMP tomorrow/NN )))))\n(a) (b)\nFigure 18.5 Parses from the LDC Treebank3 for (a) Brown and (b) ATIS sentences.\nS\n.\n.VP\nADJP-PRD\nPP\nNP\nNN\nlightCC\nandNN\n\ufb01reIN\nofJJ\nfullVBD\nwasNP-SBJ\nNN\nskyJJ\nempty,\n,JJ\ncoldDT\nThat\nFigure 18.6 The tree corresponding to the Brown corpus sentence in the previous \ufb01gure.\nThe sentences in a treebank implicitly constitute a grammar of the language. For\nexample, from the parsed sentences in Fig. 18.5 we can extract the CFG rules shown\nin Fig. 18.7 (with rule suf\ufb01xes ( -SBJ ) stripped for simplicity). The grammar used\nto parse the Penn Treebank is very \ufb02at, resulting in very many rules. For example, 394 CHAPTER 18 \u2022 C ONTEXT -FREE GRAMMARS AND CONSTITUENCY PARSING\nGrammar Lexicon\nS!NP VP . DT!thejthat\nS!NP VP JJ!coldjemptyjfull\nNP!DT NN NN!skyj\ufb01rejlightj\ufb02ightjtomorrow\nNP!NN CC NN CC!and\nNP!DT JJ , JJ NN IN!ofjat\nNP!NN CD!eleven\nVP!MD VP RB!a.m.\nVP!VBD ADJP VB!arrive\nVP!MD VP VBD!wasjsaid\nVP!VB PP NP MD!shouldjwould\nADJP!JJ PP\nPP!IN NP\nPP!IN NP RB\nFigure 18.7 CFG grammar rules and lexicon from the treebank sentences in Fig. 18.5.\namong the approximately 4,500 different rules for expanding VPs are separate rules\nfor PP sequences of any length and every possible arrangement of verb arguments:\nVP!VBD PP\nVP!VBD PP PP\nVP!VBD PP PP PP\nVP!VBD PP PP PP PP\nVP!VB ADVP PP\nVP!VB PP ADVP\nVP!ADVP VB PP\n18.4 Grammar Equivalence and Normal Form\nA formal language is de\ufb01ned as a (possibly in\ufb01nite) set of strings of words. This sug-\ngests that we could ask if two grammars are equivalent by asking if they generate the\nsame set of strings. In fact, it is possible to have two distinct context-free grammars\ngenerate the same language. We say that two grammars are strongly equivalent ifstrongly\nequivalent\nthey generate the same set of strings andif they assign the same phrase structure\nto each sentence (allowing merely for renaming of the non-terminal symbols). Two\ngrammars are weakly equivalent if they generate the same set of strings but do notweakly\nequivalent\nassign the same phrase structure to each sentence.\nIt is sometimes useful to have a normal form for grammars, in which each of normal form\nthe productions takes a particular form. For example, a context-free grammar is in\nChomsky normal form (CNF) (Chomsky, 1963) if it is \u000f-free and if in additionChomsky\nnormal form\neach production is either of the form A!B C orA!a. That is, the right-hand side\nof each rule either has two non-terminal symbols or one terminal symbol. Chomsky\nnormal form grammars are binary branching , that is they have binary trees (downbinary\nbranching\nto the prelexical nodes). We make use of this binary branching property in the CKY\nparsing algorithm in Section 18.6.\nAny context-free grammar can be converted into a weakly equivalent Chomsky\nnormal form grammar. For example, a rule of the form\nA!B C D\ncan be converted into the following two CNF rules (Exercise 18.1 asks the reader to 18.5 \u2022 A MBIGUITY 395\nGrammar Lexicon\nS!NP VP Det!thatjthisjtheja\nS!Aux NP VP Noun!bookj\ufb02ightjmealjmoney\nS!VP Verb!bookjincludejprefer\nNP!Pronoun Pronoun!Ijshejme\nNP!Proper-Noun Proper-Noun!HoustonjNWA\nNP!Det Nominal Aux!does\nNominal!Noun Preposition!fromjtojonjnearjthrough\nNominal!Nominal Noun\nNominal!Nominal PP\nVP!Verb\nVP!Verb NP\nVP!Verb NP PP\nVP!Verb PP\nVP!VP PP\nPP!Preposition NP\nFigure 18.8 TheL1miniature English grammar and lexicon.\nformulate the complete algorithm):\nA!B X\nX!C D\nSometimes using binary branching can actually produce smaller grammars. For\nexample, the sentences that might be characterized as\nVP -> VBD NP PP*\nare represented in the Penn Treebank by this series of rules:\nVP!VBD NP PP\nVP!VBD NP PP PP\nVP!VBD NP PP PP PP\nVP!VBD NP PP PP PP PP\n...\nbut could also be generated by the following two-rule grammar:\nVP!VBD NP PP\nVP!VP PP\nThe generation of a symbol A with a potentially in\ufb01nite sequence of symbols B with\na rule of the form A!A Bis known as Chomsky-adjunction .Chomsky-\nadjunction\n18.5 Ambiguity\nAmbiguity is the most serious problem faced by syntactic parsers. Chapter 17 intro-\nduced the notions of part-of-speech ambiguity andpart-of-speech disambigua-\ntion. Here, we introduce a new kind of ambiguity, called structural ambiguity ,structural\nambiguity\nillustrated with a new toy grammar L1, shown in Figure 18.8, which adds a few\nrules to the L0grammar.\nStructural ambiguity occurs when the grammar can assign more than one parse\nto a sentence. Groucho Marx\u2019s well-known line as Captain Spaulding in Animal 396 CHAPTER 18 \u2022 C ONTEXT -FREE GRAMMARS AND CONSTITUENCY PARSING\nS\nVP\nNP\nNominal\nPP\nin my pajamasNominal\nNoun\nelephantDet\nanVerb\nshotNP\nPronoun\nIS\nVP\nPP\nin my pajamasVP\nNP\nNominal\nNoun\nelephantDet\nanVerb\nshotNP\nPronoun\nI\nFigure 18.9 Two parse trees for an ambiguous sentence. The parse on the left corresponds to the humorous\nreading in which the elephant is in the pajamas, the parse on the right corresponds to the reading in which\nCaptain Spaulding did the shooting in his pajamas.\nCrackers is ambiguous because the phrase in my pajamas can be part of the NP\nheaded by elephant or a part of the verb phrase headed by shot. Figure 18.9 illus-\ntrates these two analyses of Marx\u2019s line using rules from L1.\nStructural ambiguity, appropriately enough, comes in many forms. Two common\nkinds of ambiguity are attachment ambiguity andcoordination ambiguity . A\nsentence has an attachment ambiguity if a particular constituent can be attached toattachment\nambiguity\nthe parse tree at more than one place. The Groucho Marx sentence is an example\nofPP-attachment ambiguity : the preposition phrase can be attached either as partPP-attachment\nambiguity\nof the NP or as part of the VP. Various kinds of adverbial phrases are also subject\nto this kind of ambiguity. For instance, in the following example the gerundive-VP\n\ufb02ying to Paris can be part of a gerundive sentence whose subject is the Eiffel Tower\nor it can be an adjunct modifying the VP headed by saw:\n(18.2) We saw the Eiffel Tower \ufb02ying to Paris.\nIncoordination ambiguity phrases can be conjoined by a conjunction like and.coordination\nambiguity\nFor example, the phrase old men and women can be bracketed as [old [men and\nwomen]] , referring to old men andold women , or as [old men] and [women] , in\nwhich case it is only the men who are old. These ambiguities combine in complex\nways in real sentences, like the following news sentence from the Brown corpus:\n(18.3) President Kennedy today pushed aside other White House business to\ndevote all his time and attention to working on the Berlin crisis address he\nwill deliver tomorrow night to the American people over nationwide\ntelevision and radio.\nThis sentence has a number of ambiguities, although since they are semantically\nunreasonable, it requires a careful reading to see them. The last noun phrase could be\nparsed [nationwide [television and radio]] or[[nationwide television] and radio] .\nThe direct object of pushed aside should be other White House business but could\nalso be the bizarre phrase [other White House business to devote all his time and\nattention to working] (i.e., a structure like Kennedy af\ufb01rmed [his intention to propose\na new budget to address the de\ufb01cit] ). Then the phrase on the Berlin crisis address he 18.6 \u2022 CKY P ARSING : A D YNAMIC PROGRAMMING APPROACH 397\nwill deliver tomorrow night to the American people could be an adjunct modifying\nthe verb pushed . APPlikeover nationwide television and radio could be attached\nto any of the higher VPs orNPs (e.g., it could modify people ornight ).\nThe fact that there are many grammatically correct but semantically unreason-\nable parses for naturally occurring sentences is an irksome problem that affects all\nparsers. Fortunately, the CKY algorithm below is designed to ef\ufb01ciently handle\nstructural ambiguities. And as we\u2019ll see in the following section, we can augment\nCKY with neural methods to choose a single correct parse by syntactic disambigua-\ntion.syntactic\ndisambiguation\n18.6 CKY Parsing: A Dynamic Programming Approach\nDynamic programming provides a powerful framework for addressing the prob-\nlems caused by ambiguity in grammars. Recall that a dynamic programming ap-\nproach systematically \ufb01lls in a table of solutions to subproblems. The complete\ntable has the solution to all the subproblems needed to solve the problem as a whole.\nIn the case of syntactic parsing, these subproblems represent parse trees for all the\nconstituents detected in the input.\nThe dynamic programming advantage arises from the context-free nature of our\ngrammar rules\u2014once a constituent has been discovered in a segment of the input we\ncan record its presence and make it available for use in any subsequent derivation\nthat might require it. This provides both time and storage ef\ufb01ciencies since subtrees\ncan be looked up in a table, not reanalyzed. This section presents the Cocke-Kasami-\nYounger (CKY) algorithm, the most widely used dynamic-programming based ap-\nproach to parsing. Chart parsing (Kaplan 1973, Kay 1982) is a related approach,\nand dynamic programming methods are often referred to as chart parsing methods. chart parsing\n18.6.1 Conversion to Chomsky Normal Form\nThe CKY algorithm requires grammars to \ufb01rst be in Chomsky Normal Form (CNF).\nRecall from Section 18.4 that grammars in CNF are restricted to rules of the form\nA!B CorA!w. That is, the right-hand side of each rule must expand either to\ntwo non-terminals or to a single terminal. Restricting a grammar to CNF does not\nlead to any loss in expressiveness, since any context-free grammar can be converted\ninto a corresponding CNF grammar that accepts exactly the same set of strings as\nthe original grammar.\nLet\u2019s start with the process of converting a generic CFG into one represented in\nCNF. Assuming we\u2019re dealing with an \u000f-free grammar, there are three situations we\nneed to address in any generic grammar: rules that mix terminals with non-terminals\non the right-hand side, rules that have a single non-terminal on the right-hand side,\nand rules in which the length of the right-hand side is greater than 2.\nThe remedy for rules that mix terminals and non-terminals is to simply introduce\na new dummy non-terminal that covers only the original terminal. For example, a\nrule for an in\ufb01nitive verb phrase such as INF-VP!to VP would be replaced by the\ntwo rules INF-VP!TO VP andTO!to.\nRules with a single non-terminal on the right are called unit productions . WeUnit\nproductions\ncan eliminate unit productions by rewriting the right-hand side of the original rules\nwith the right-hand side of all the non-unit production rules that they ultimately lead\nto. More formally, if A\u0003)Bby a chain of one or more unit productions and B!g 398 CHAPTER 18 \u2022 C ONTEXT -FREE GRAMMARS AND CONSTITUENCY PARSING\nis a non-unit production in our grammar, then we add A!gfor each such rule in\nthe grammar and discard all the intervening unit productions. As we demonstrate\nwith our toy grammar, this can lead to a substantial \ufb02attening of the grammar and a\nconsequent promotion of terminals to fairly high levels in the resulting trees.\nRules with right-hand sides longer than 2 are normalized through the introduc-\ntion of new non-terminals that spread the longer sequences over several new rules.\nFormally, if we have a rule like\nA!B Cg\nwe replace the leftmost pair of non-terminals with a new non-terminal and introduce\na new production, resulting in the following new rules:\nA!X1g\nX1!B C\nIn the case of longer right-hand sides, we simply iterate this process until the of-\nfending rule has been replaced by rules of length 2. The choice of replacing the\nleftmost pair of non-terminals is purely arbitrary; any systematic scheme that results\nin binary rules would suf\ufb01ce.\nIn our current grammar, the rule S!Aux NP VP would be replaced by the two\nrules S!X1 VP andX1!Aux NP .\nThe entire conversion process can be summarized as follows:\n1. Copy all conforming rules to the new grammar unchanged.\n2. Convert terminals within rules to dummy non-terminals.\n3. Convert unit productions.\n4. Make all rules binary and add them to new grammar.\nFigure 18.10 shows the results of applying this entire conversion procedure to\ntheL1grammar introduced earlier on page 395. Note that this \ufb01gure doesn\u2019t show\nthe original lexical rules; since these original lexical rules are already in CNF, they\nall carry over unchanged to the new grammar. Figure 18.10 does, however, show\nthe various places where the process of eliminating unit productions has, in effect,\ncreated new lexical rules. For example, all the original verbs have been promoted to\nboth VPs and to Ss in the converted grammar.\n18.6.2 CKY Recognition\nWith our grammar now in CNF, each non-terminal node above the part-of-speech\nlevel in a parse tree will have exactly two daughters. A two-dimensional matrix can\nbe used to encode the structure of an entire tree. For a sentence of length n, we will\nwork with the upper-triangular portion of an (n+1)\u0002(n+1)matrix. Each cell [i;j]\nin this matrix contains the set of non-terminals that represent all the constituents that\nspan positions ithrough jof the input. Since our indexing scheme begins with 0, it\u2019s\nnatural to think of the indexes as pointing at the gaps between the input words (as in\n0Book 1that 2\ufb02ight 3). These gaps are often called fenceposts , on the metaphor of fenceposts\nthe posts between segments of fencing. It follows then that the cell that represents\nthe entire input resides in position [0;n]in the matrix.\nSince each non-terminal entry in our table has two daughters in the parse, it fol-\nlows that for each constituent represented by an entry [i;j], there must be a position\nin the input, k, where it can be split into two parts such that i<k<j. Given such 18.6 \u2022 CKY P ARSING : A D YNAMIC PROGRAMMING APPROACH 399\nL1Grammar L1in CNF\nS!NP VP S!NP VP\nS!Aux NP VP S!X1 VP\nX1!Aux NP\nS!VP S!bookjincludejprefer\nS!Verb NP\nS!X2 PP\nS!Verb PP\nS!VP PP\nNP!Pronoun NP!Ijshejme\nNP!Proper-Noun NP!TWAjHouston\nNP!Det Nominal NP!Det Nominal\nNominal!Noun Nominal!bookj\ufb02ightjmealjmoney\nNominal!Nominal Noun Nominal!Nominal Noun\nNominal!Nominal PP Nominal!Nominal PP\nVP!Verb VP!bookjincludejprefer\nVP!Verb NP VP!Verb NP\nVP!Verb NP PP VP!X2 PP\nX2!Verb NP\nVP!Verb PP VP!Verb PP\nVP!VP PP VP!VP PP\nPP!Preposition NP PP!Preposition NP\nFigure 18.10 L1Grammar and its conversion to CNF. Note that although they aren\u2019t shown\nhere, all the original lexical entries from L1carry over unchanged as well.\na position k, the \ufb01rst constituent [i;k]must lie to the left of entry [i;j]somewhere\nalong row i, and the second entry [k;j]must lie beneath it, along column j.\nTo make this more concrete, consider the following example with its completed\nparse matrix, shown in Fig. 18.11.\n(18.4) Book the \ufb02ight through Houston.\nThe superdiagonal row in the matrix contains the parts of speech for each word in\nthe input. The subsequent diagonals above that superdiagonal contain constituents\nthat cover all the spans of increasing length in the input.\nGiven this setup, CKY recognition consists of \ufb01lling the parse table in the right\nway. To do this, we\u2019ll proceed in a bottom-up fashion so that at the point where we\nare \ufb01lling any cell [i;j], the cells containing the parts that could contribute to this\nentry (i.e., the cells to the left and the cells below) have already been \ufb01lled. The\nalgorithm given in Fig. 18.12 \ufb01lls the upper-triangular matrix a column at a time\nworking from left to right, with each column \ufb01lled from bottom to top, as the right\nside of Fig. 18.11 illustrates. This scheme guarantees that at each point in time we\nhave all the information we need (to the left, since all the columns to the left have\nalready been \ufb01lled, and below since we\u2019re \ufb01lling bottom to top). It also mirrors on-\nline processing, since \ufb01lling the columns from left to right corresponds to processing\neach word one at a time.\nThe outermost loop of the algorithm given in Fig. 18.12 iterates over the columns,\nand the second loop iterates over the rows, from the bottom up. The purpose of the\ninnermost loop is to range over all the places where a substring spanning itojin\nthe input might be split in two. As kranges over the places where the string can be\nsplit, the pairs of cells we consider move, in lockstep, to the right along row iand\ndown along column j. Figure 18.13 illustrates the general case of \ufb01lling cell [i;j]. 400 CHAPTER 18 \u2022 C ONTEXT -FREE GRAMMARS AND CONSTITUENCY PARSING\nBookthe flight throughHoustonS, VP, Verb, Nominal, NounS,VP,X2S,VP,X2DetNPNPNominal,NounNominalPrepPPNP,Proper-Noun[0,1][0,2][0,3][0,4][0,5][1,2][1,3][2,3][1,4][2,5][2,4][3,4][4,5][3,5][1,5]\nFigure 18.11 Completed parse table for Book the \ufb02ight through Houston.\nfunction CKY-P ARSE (words, grammar )returns table\nforj from 1toLENGTH (words )do\nfor all fAjA!words [j]2grammar g\ntable [j\u00001;j] table [j\u00001;j][A\nfori from j\u00002down to 0do\nfork i+1toj\u00001do\nfor all fAjA!BC2grammar andB2table[i;k]andC2table[k;j]g\ntable [i,j] table [i,j][A\nFigure 18.12 The CKY algorithm.\nAt each such split, the algorithm considers whether the contents of the two cells can\nbe combined in a way that is sanctioned by a rule in the grammar. If such a rule\nexists, the non-terminal on its left-hand side is entered into the table.\nFigure 18.14 shows how the \ufb01ve cells of column 5 of the table are \ufb01lled after the\nword Houston is read. The arrows point out the two spans that are being used to add\nan entry to the table. Note that the action in cell [0;5]indicates the presence of three\nalternative parses for this input, one where the PPmodi\ufb01es the \ufb02ight , one where\nit modi\ufb01es the booking, and one that captures the second argument in the original\nVP!Verb NP PP rule, now captured indirectly with the VP!X2 PP rule.\n18.6.3 CKY Parsing\nThe algorithm given in Fig. 18.12 is a recognizer, not a parser. That is, it can tell\nus whether a valid parse exists for a given sentence based on whether or not if \ufb01nds\nanSin cell [0;n], but it can\u2019t provide the derivation, which is the actual job for a\nparser. To turn it into a parser capable of returning all possible parses for a given\ninput, we can make two simple changes to the algorithm: the \ufb01rst change is to\naugment the entries in the table so that each non-terminal is paired with pointers to\nthe table entries from which it was derived (more or less as shown in Fig. 18.14), the\nsecond change is to permit multiple versions of the same non-terminal to be entered\ninto the table (again as shown in Fig. 18.14). With these changes, the completed\ntable contains all the possible parses for a given input. Returning an arbitrary single 18.6 \u2022 CKY P ARSING : A D YNAMIC PROGRAMMING APPROACH 401\n...\n...[0,n]\n[i,i+1][i,i+2][i,j-2][i,j-1][i+1,j][i+2,j]\n[j-1,j][j-2,j][i,j]...[0,1]\n[n-1, n]\nFigure 18.13 All the ways to \ufb01ll the [ i,j]th cell in the CKY table.\nparse consists of choosing an Sfrom cell [0;n]and then recursively retrieving its\ncomponent constituents from the table. Of course, instead of returning every parse\nfor a sentence, we usually want just the best parse; we\u2019ll see how to do that in the\nnext section.\n18.6.4 CKY in Practice\nFinally, we should note that while the restriction to CNF does not pose a problem\ntheoretically, it does pose some non-trivial problems in practice. The returned CNF\ntrees may not be consistent with the original grammar built by the grammar devel-\nopers, and will complicate any syntax-driven approach to semantic analysis.\nOne approach to getting around these problems is to keep enough information\naround to transform our trees back to the original grammar as a post-processing step\nof the parse. This is trivial in the case of the transformation used for rules with length\ngreater than 2. Simply deleting the new dummy non-terminals and promoting their\ndaughters restores the original tree.\nIn the case of unit productions, it turns out to be more convenient to alter the ba-\nsic CKY algorithm to handle them directly than it is to store the information needed\nto recover the correct trees. Exercise 18.3 asks you to make this change. Many of\nthe probabilistic parsers presented in Appendix C use the CKY algorithm altered in 402 CHAPTER 18 \u2022 C ONTEXT -FREE GRAMMARS AND CONSTITUENCY PARSING\nBookthe flight throughHoustonS, VP, Verb, Nominal, NounS,VP,X2DetNPNominal,NounNominalPrepNP,Proper-Noun[0,1][0,2][0,3][0,4][0,5][1,2][1,3][2,3][1,4][2,5][2,4][3,4][4,5][3,5][1,5]Bookthe flight throughHoustonS, VP, Verb, Nominal, NounS,VP,X2DetNPNPNominal,NounPrepPPNP,Proper-Noun[0,1][0,2][0,3][0,4][0,5][1,2][1,3][2,3][1,4][2,5][2,4][3,4][4,5][3,5][1,5]\nBookthe flight throughHoustonS, VP, Verb, Nominal, NounS,VP,X2DetNPNPNominal,NounNominalPrepPPNP,Proper-Noun[0,1][0,2][0,3][0,4][0,5][1,2][1,3][2,3][1,4][2,5][2,4][3,4][4,5][3,5][1,5]Bookthe flight throughHoustonS, VP, Verb, Nominal, NounS,VP,X2DetNPNPNominal,NounNominalPrepPPNP,Proper-Noun[0,1][0,2][0,3][0,4][0,5][1,2][1,3][2,3][1,4][2,5][2,4][3,4][4,5][3,5][1,5]\nBookthe flight throughHoustonS, VP, Verb, Nominal, NounS,VP,X2DetNPNPNominal,NounNominalPrepPPNP,Proper-Noun[0,1][0,2][0,3][0,4][1,2][1,3][2,3][1,4][2,5][2,4][3,4][4,5][3,5][1,5]S2, VPS3S1,VP, X2\nFigure 18.14 Filling the cells of column 5 after reading the word Houston . 18.7 \u2022 S PAN-BASED NEURAL CONSTITUENCY PARSING 403\njust this manner.\n18.7 Span-Based Neural Constituency Parsing\nWhile the CKY parsing algorithm we\u2019ve seen so far does great at enumerating all\nthe possible parse trees for a sentence, it has a large problem: it doesn\u2019t tell us which\nparse is the correct one! That is, it doesn\u2019t disambiguate among the possible parses.\nTo solve the disambiguation problem we\u2019ll use a simple neural extension of the\nCKY algorithm. The intuition of such parsing algorithms (often called span-based\nconstituency parsing , orneural CKY ), is to train a neural classi\ufb01er to assign a\nscore to each constituent, and then use a modi\ufb01ed version of CKY to combine these\nconstituent scores to \ufb01nd the best-scoring parse tree.\nHere we\u2019ll describe a version of the algorithm from Kitaev et al. (2019). This\nparser learns to map a span of words to a constituent, and, like CKY , hierarchically\ncombines larger and larger spans to build the parse-tree bottom-up. But unlike clas-\nsic CKY , this parser doesn\u2019t use the hand-written grammar to constrain what con-\nstituents can be combined, instead just relying on the learned neural representations\nof spans to encode likely combinations.\n18.7.1 Computing Scores for a Span\nLet\u2019s begin by considering just the constituent (we\u2019ll call it a span ) that lies between span\nfencepost positions iand jwith non-terminal symbol label l. We\u2019ll build a system\nto assign a score s(i;j;l)to this constituent span.\nENCODER[START]BooktheflightthroughHouston[END]map to subwordsmap back to words013245MLPi=1hj-hij=3NPCompute score for spanRepresent spanCKY for computing best parse\npostprocessing layers\nFigure 18.15 A simpli\ufb01ed outline of computing the span score for the span the \ufb02ight with\nthe label NP.\nFig. 18.15 sketches the architecture. The input word tokens are embedded by 404 CHAPTER 18 \u2022 C ONTEXT -FREE GRAMMARS AND CONSTITUENCY PARSING\npassing them through a pretrained language model like BERT. Because BERT oper-\nates on the level of subword (wordpiece) tokens rather than words, we\u2019ll \ufb01rst need to\nconvert the BERT outputs to word representations. One standard way of doing this\nis to simply use the \ufb01rst subword unit as the representation for the entire word; us-\ning the last subword unit, or the sum of all the subword units are also common. The\nembeddings can then be passed through some postprocessing layers; Kitaev et al.\n(2019), for example, use 8 Transformer layers.\nThe resulting word encoder outputs ytare then used to compute a span score.\nFirst, we must map the word encodings (indexed by word positions) to span encod-\nings (indexed by fenceposts). We do this by representing each fencepost with two\nseparate values; the intuition is that a span endpoint to the right of a word represents\ndifferent information than a span endpoint to the left of a word. We convert each\nword output ytinto a (leftward-pointing) value for spans ending at this fencepost, \u0000yt, and a (rightward-pointing) value\u0000 !ytfor spans beginning at this fencepost, by\nsplitting ytinto two halves. Each span then stretches from one double-vector fence-\npost to another, as in the following representation of the \ufb02ight , which is span (1;3):\nSTART 0 Book the \ufb02ight through\ny0\u0000 !y0 \u0000y1y1\u0000 !y1 \u0000y2 y2\u0000 !y2 \u0000y3y3\u0000 !y3 \u0000y4 y4\u0000 !y4 \u0000y5:::\n0\r 1\r 2\r 3\r 4\r\nspan(1,3)\nA traditional way to represent a span, developed originally for RNN-based models\n(Wang and Chang, 2016), but extended also to Transformers, is to take the differ-\nence between the embeddings of its start and end, i.e., representing span (i;j)by\nsubtracting the embedding of ifrom the embedding of j. Here we represent a span\nby concatenating the difference of each of its fencepost components:\nv(i;j) = [\u0000 !yj\u0000\u0000 !yi; \u0000\u0000yj+1\u0000 \u0000\u0000yi+1] (18.5)\nThe span vector vis then passed through an MLP span classi\ufb01er, with two fully-\nconnected layers and one ReLU activation function, whose output dimensionality is\nthe number of possible non-terminal labels:\ns(i;j;\u0001) =W2ReLU (LayerNorm (W1v(i;j))) (18.6)\nThe MLP then outputs a score for each possible non-terminal.\n18.7.2 Integrating Span Scores into a Parse\nNow we have a score for each labeled constituent span s(i;j;l). But we need a score\nfor an entire parse tree. Formally a tree Tis represented as a set of jTjsuch labeled\nspans, with the tthspan starting at position itand ending at position jt, with label lt:\nT=f(it;jt;lt):t=1;:::;jTjg (18.7)\nThus once we have a score for each span, the parser can compute a score for the\nwhole tree s(T)simply by summing over the scores of its constituent spans:\ns(T) =X\n(i;j;l)2Ts(i;j;l) (18.8) 18.8 \u2022 E VALUATING PARSERS 405\nAnd we can choose the \ufb01nal parse tree as the tree with the maximum score:\n\u02c6T=argmax\nTs(T) (18.9)\nThe simplest method to produce the most likely parse is to greedily choose the\nhighest scoring label for each span. This greedy method is not guaranteed to produce\na tree, since the best label for a span might not \ufb01t into a complete tree. In practice,\nhowever, the greedy method tends to \ufb01nd trees; in their experiments Gaddy et al.\n(2018) \ufb01nds that 95% of predicted bracketings form valid trees.\nNonetheless it is more common to use a variant of the CKY algorithm to \ufb01nd the\nfull parse. The variant de\ufb01ned in Gaddy et al. (2018) works as follows. Let\u2019s de\ufb01ne\nsbest(i;j)as the score of the best subtree spanning (i;j). For spans of length one, we\nchoose the best label:\nsbest(i;i+1) =max\nls(i;i+1;l) (18.10)\nFor other spans (i;j), the recursion is:\nsbest(i;j) = max\nls(i;j;l)\n+max\nk[sbest(i;k)+sbest(k;j)] (18.11)\nNote that the parser is using the max label for span (i;j)+ the max labels for spans\n(i;k)and(k;j)without worrying about whether those decisions make sense given a\ngrammar. The role of the grammar in classical parsing is to help constrain possible\ncombinations of constituents (NPs like to be followed by VPs). By contrast, the\nneural model seems to learn these kinds of contextual constraints during its mapping\nfrom spans to non-terminals.\nFor more details on span-based parsing, including the margin-based training al-\ngorithm, see Stern et al. (2017), Gaddy et al. (2018), Kitaev and Klein (2018), and\nKitaev et al. (2019).\n18.8 Evaluating Parsers\nThe standard tool for evaluating parsers that assign a single parse tree to a sentence\nis the PARSEV AL metrics (Black et al., 1991). The PARSEV AL metric measures PARSEV AL\nhow much the constituents in the hypothesis parse tree look like the constituents in a\nhand-labeled, reference parse. PARSEV AL thus requires a human-labeled reference\n(or \u201cgold standard\u201d) parse tree for each sentence in the test set; we generally draw\nthese reference parses from a treebank like the Penn Treebank.\nA constituent in a hypothesis parse Chof a sentence sis labeled correct if there\nis a constituent in the reference parse Crwith the same starting point, ending point,\nand non-terminal symbol. We can then measure the precision and recall just as for\ntasks we\u2019ve seen already like named entity tagging:\nlabeled recall: =# of correct constituents in hypothesis parse of s\n# of total constituents in reference parse of s\nlabeled precision: =# of correct constituents in hypothesis parse of s\n# of total constituents in hypothesis parse of s 406 CHAPTER 18 \u2022 C ONTEXT -FREE GRAMMARS AND CONSTITUENCY PARSING\nS(dumped)\nVP(dumped)\nPP(into)\nNP(bin)\nNN(bin)\nbinDT(a)\naP\nintoNP(sacks)\nNNS(sacks)\nsacksVBD(dumped)\ndumpedNP(workers)\nNNS(workers)\nworkers\nFigure 18.16 A lexicalized tree from Collins (1999).\nAs usual, we often report a combination of the two, F 1:\nF1=2PR\nP+R(18.12)\nWe additionally use a new metric, crossing brackets, for each sentence s:\ncross-brackets: the number of constituents for which the reference parse has a\nbracketing such as ((A B) C) but the hypothesis parse has a bracketing such\nas (A (B C)).\nFor comparing parsers that use different grammars, the PARSEV AL metric in-\ncludes a canonicalization algorithm for removing information likely to be grammar-\nspeci\ufb01c (auxiliaries, pre-in\ufb01nitival \u201cto\u201d, etc.) and for computing a simpli\ufb01ed score\n(Black et al., 1991). The canonical implementation of the PARSEV AL metrics is\ncalled evalb (Sekine and Collins, 1997). evalb\n18.9 Heads and Head-Finding\nSyntactic constituents can be associated with a lexical head ;Nis the head of an NP,\nVis the head of a VP. This idea of a head for each constituent dates back to Bloom-\n\ufb01eld 1914, and is central to the dependency grammars and dependency parsing we\u2019ll\nintroduce in Chapter 19. Indeed, heads can be used as a way to map between con-\nstituency and dependency parses. Heads are also important in probabilistic pars-\ning (Appendix C) and in constituent-based grammar formalisms like Head-Driven\nPhrase Structure Grammar (Pollard and Sag, 1994)..\nIn one simple model of lexical heads, each context-free rule is associated with\na head (Charniak 1997, Collins 1999). The head is the word in the phrase that is\ngrammatically the most important. Heads are passed up the parse tree; thus, each\nnon-terminal in a parse tree is annotated with a single word, which is its lexical head.\nFigure 18.16 shows an example of such a tree from Collins (1999), in which each\nnon-terminal is annotated with its head.\nFor the generation of such a tree, each CFG rule must be augmented to identify\none right-side constituent to be the head child. The headword for a node is then set to\nthe headword of its head child. Choosing these head children is simple for textbook\nexamples ( NNis the head of NP) but is complicated and indeed controversial for 18.10 \u2022 S UMMARY 407\nmost phrases. (Should the complementizer toor the verb be the head of an in\ufb01nite\nverb phrase?) Modern linguistic theories of syntax generally include a component\nthat de\ufb01nes heads (see, e.g., (Pollard and Sag, 1994)).\nAn alternative approach to \ufb01nding a head is used in most practical computational\nsystems. Instead of specifying head rules in the grammar itself, heads are identi\ufb01ed\ndynamically in the context of trees for speci\ufb01c sentences. In other words, once\na sentence is parsed, the resulting tree is walked to decorate each node with the\nappropriate head. Most current systems rely on a simple set of handwritten rules,\nsuch as a practical one for Penn Treebank grammars given in Collins (1999) but\ndeveloped originally by Magerman (1995). For example, the rule for \ufb01nding the\nhead of an NPis as follows (Collins, 1999, p. 238):\n\u2022 If the last word is tagged POS, return last-word.\n\u2022 Else search from right to left for the \ufb01rst child which is an NN, NNP, NNPS, NX, POS,\nor JJR.\n\u2022 Else search from left to right for the \ufb01rst child which is an NP.\n\u2022 Else search from right to left for the \ufb01rst child which is a $, ADJP, or PRN.\n\u2022 Else search from right to left for the \ufb01rst child which is a CD.\n\u2022 Else search from right to left for the \ufb01rst child which is a JJ, JJS, RB or QP.\n\u2022 Else return the last word\nSelected other rules from this set are shown in Fig. 18.17. For example, for VP\nrules of the form VP!Y1\u0001\u0001\u0001Yn, the algorithm would start from the left of Y1\u0001\u0001\u0001\nYnlooking for the \ufb01rst Yiof type TO; if no TOs are found, it would search for the\n\ufb01rstYiof type VBD; if no VBDs are found, it would search for a VBN, and so on.\nSee Collins (1999) for more details.\nParent Direction Priority List\nADJP Left NNS QP NN $ ADVP JJ VBN VBG ADJP JJR NP JJS DT FW RBR RBS\nSBAR RB\nADVP Right RB RBR RBS FW ADVP TO CD JJR JJ IN NP JJS NN\nPRN Left\nPRT Right RP\nQP Left $ IN NNS NN JJ RB DT CD NCD QP JJR JJS\nS Left TO IN VP S SBAR ADJP UCP NP\nSBAR Left WHNP WHPP WHADVP WHADJP IN DT S SQ SINV SBAR FRAG\nVP Left TO VBD VBN MD VBZ VB VBG VBP VP ADJP NN NNS NP\nFigure 18.17 Some head rules from Collins (1999). The head rules are also called a head percolation table .\n18.10 Summary\nThis chapter introduced constituency parsing. Here\u2019s a summary of the main points:\n\u2022 In many languages, groups of consecutive words act as a group or a con-\nstituent , which can be modeled by context-free grammars (which are also\nknown as phrase-structure grammars ).\n\u2022 A context-free grammar consists of a set of rules orproductions , expressed\nover a set of non-terminal symbols and a set of terminal symbols. Formally,\na particular context-free language is the set of strings that can be derived\nfrom a particular context-free grammar . 408 CHAPTER 18 \u2022 C ONTEXT -FREE GRAMMARS AND CONSTITUENCY PARSING\n\u2022Structural ambiguity is a signi\ufb01cant problem for parsers. Common sources\nof structural ambiguity include PP-attachment andcoordination ambiguity .\n\u2022Dynamic programming parsing algorithms, such as CKY , use a table of\npartial parses to ef\ufb01ciently parse ambiguous sentences.\n\u2022CKY restricts the form of the grammar to Chomsky normal form (CNF).\n\u2022 The basic CKY algorithm compactly represents all possible parses of the sen-\ntence but doesn\u2019t choose a single best parse.\n\u2022 Choosing a single parse from all possible parses ( disambiguation ) can be\ndone by neural constituency parsers .\n\u2022 Span-based neural constituency parses train a neural classi\ufb01er to assign a score\nto each constituent, and then use a modi\ufb01ed version of CKY to combine these\nconstituent scores to \ufb01nd the best-scoring parse tree.\n\u2022 Parsers are evaluated with three metrics: labeled recall ,labeled precision ,\nandcross-brackets .\n\u2022Partial parsing andchunking are methods for identifying shallow syntac-\ntic constituents in a text. They are solved by sequence models trained on\nsyntactically-annotated data.\nBibliographical and Historical Notes\nAccording to Percival (1976), the idea of breaking up a sentence into a hierarchy of\nconstituents appeared in the V\u00a8olkerpsychologie of the groundbreaking psychologist\nWilhelm Wundt (Wundt, 1900):\n...den sprachlichen Ausdruck f \u00a8ur die willk \u00a8urliche Gliederung einer Ge-\nsammtvorstellung in ihre in logische Beziehung zueinander gesetzten\nBestandteile\n[the linguistic expression for the arbitrary division of a total idea\ninto its constituent parts placed in logical relations to one another]\nWundt\u2019s idea of constituency was taken up into linguistics by Leonard Bloom-\n\ufb01eld in his early book An Introduction to the Study of Language (Bloom\ufb01eld, 1914).\nBy the time of his later book, Language (Bloom\ufb01eld, 1933), what was then called\n\u201cimmediate-constituent analysis\u201d was a well-established method of syntactic study\nin the United States. By contrast, traditional European grammar, dating from the\nClassical period, de\ufb01ned relations between words rather than constituents, and Eu-\nropean syntacticians retained this emphasis on such dependency grammars, the sub-\nject of Chapter 19. (And indeed, both dependency and constituency grammars have\nbeen in vogue in computational linguistics at different times).\nAmerican Structuralism saw a number of speci\ufb01c de\ufb01nitions of the immediate\nconstituent, couched in terms of their search for a \u201cdiscovery procedure\u201d: a method-\nological algorithm for describing the syntax of a language. In general, these attempt\nto capture the intuition that \u201cThe primary criterion of the immediate constituent\nis the degree in which combinations behave as simple units\u201d (Bazell, 1952/1966, p.\n284). The most well known of the speci\ufb01c de\ufb01nitions is Harris\u2019 idea of distributional\nsimilarity to individual units, with the substitutability test. Essentially, the method\nproceeded by breaking up a construction into constituents by attempting to substitute\nsimple structures for possible constituents\u2014if a substitution of a simple form, say, EXERCISES 409\nman, was substitutable in a construction for a more complex set (like intense young\nman), then the form intense young man was probably a constituent. Harris\u2019s test was\nthe beginning of the intuition that a constituent is a kind of equivalence class.\nThe context-free grammar was a formalization of this idea of hierarchical\nconstituency de\ufb01ned in Chomsky (1956) and further expanded upon (and argued\nagainst) in Chomsky (1957) and Chomsky (1956/1975). Shortly after Chomsky\u2019s\ninitial work, the context-free grammar was reinvented by Backus (1959) and inde-\npendently by Naur et al. (1960) in their descriptions of the ALGOL programming\nlanguage; Backus (1996) noted that he was in\ufb02uenced by the productions of Emil\nPost and that Naur\u2019s work was independent of his (Backus\u2019) own. After this early\nwork, a great number of computational models of natural language processing were\nbased on context-free grammars because of the early development of ef\ufb01cient pars-\ning algorithms.\nDynamic programming parsing has a history of independent discovery. Ac-\ncording to the late Martin Kay (personal communication), a dynamic programming\nparser containing the roots of the CKY algorithm was \ufb01rst implemented by John\nCocke in 1960. Later work extended and formalized the algorithm, as well as prov-\ning its time complexity (Kay 1967, Younger 1967, Kasami 1965). The related well-\nformed substring table (WFST ) seems to have been independently proposed by WFST\nKuno (1965) as a data structure that stores the results of all previous computations\nin the course of the parse. Based on a generalization of Cocke\u2019s work, a similar\ndata structure had been independently described in Kay (1967) (and Kay 1973). The\ntop-down application of dynamic programming to parsing was described in Earley\u2019s\nPh.D. dissertation (Earley 1968, Earley 1970). Sheil (1976) showed the equivalence\nof the WFST and the Earley algorithm. Norvig (1991) shows that the ef\ufb01ciency of-\nfered by dynamic programming can be captured in any language with a memoization\nfunction (such as in LISP) simply by wrapping the memoization operation around a\nsimple top-down parser.\nThe earliest disambiguation algorithms for parsing were based on probabilistic\ncontext-free grammars , \ufb01rst worked out by Booth (1969) and Salomaa (1969); seeprobabilistic\ncontext-free\ngrammarsAppendix C for more history. Neural methods were \ufb01rst applied to parsing at around\nthe same time as statistical parsing methods were developed (Henderson, 1994). In\nthe earliest work neural networks were used to estimate some of the probabilities for\nstatistical constituency parsers (Henderson, 2003, 2004; Emami and Jelinek, 2005)\n. The next decades saw a wide variety of neural parsing algorithms, including re-\ncursive neural architectures (Socher et al., 2011, 2013), encoder-decoder models\n(Vinyals et al., 2015; Choe and Charniak, 2016), and the idea of focusing on spans\n(Cross and Huang, 2016). For more on the span-based self-attention approach we\ndescribe in this chapter see Stern et al. (2017), Gaddy et al. (2018), Kitaev and Klein\n(2018), and Kitaev et al. (2019). See Chapter 20 for the parallel history of neural\ndependency parsing.\nThe classic reference for parsing algorithms is Aho and Ullman (1972); although\nthe focus of that book is on computer languages, most of the algorithms have been\napplied to natural language.\nExercises\n18.1 Implement the algorithm to convert arbitrary context-free grammars to CNF. 410 CHAPTER 18 \u2022 C ONTEXT -FREE GRAMMARS AND CONSTITUENCY PARSING\nApply your program to the L1grammar.\n18.2 Implement the CKY algorithm and test it with your converted L1grammar.\n18.3 Rewrite the CKY algorithm given in Fig. 18.12 on page 400 so that it can\naccept grammars that contain unit productions.\n18.4 Discuss how to augment a parser to deal with input that may be incorrect, for\nexample, containing spelling errors or mistakes arising from automatic speech\nrecognition.\n18.5 Implement the PARSEV AL metrics described in Section 18.8. Next, use a\nparser and a treebank, compare your metrics against a standard implementa-\ntion. Analyze the errors in your approach. CHAPTER\n19Dependency Parsing\nTout mot qui fait partie d\u2019une phrase... Entre lui et ses voisins, l\u2019esprit aperc \u00b8oit\ndes connexions, dont l\u2019ensemble forme la charpente de la phrase.\n[Between each word in a sentence and its neighbors, the mind perceives con-\nnections . These connections together form the scaffolding of the sentence.]\nLucien Tesni `ere. 1959. \u00b4El\u00b4ements de syntaxe structurale, A.1.\u00a74\nThe focus of the last chapter was on context-free grammars and constituent-\nbased representations. Here we present another important family of grammar for-\nmalisms called dependency grammars . In dependency formalisms, phrasal con-dependency\ngrammars\nstituents and phrase-structure rules do not play a direct role. Instead, the syntactic\nstructure of a sentence is described solely in terms of directed binary grammatical\nrelations between the words , as in the following dependency parse:\nIprefer themorning \ufb02ight through Denvernsubjobj\ndet\ncompoundnmod\ncaseroot\n(19.1)\nRelations among the words are illustrated above the sentence with directed, labeled\narcs from heads todependents . We call this a typed dependency structure becausetyped\ndependency\nthe labels are drawn from a \ufb01xed inventory of grammatical relations. A root node\nexplicitly marks the root of the tree, the head of the entire structure.\nFigure 19.1 on the next page shows the dependency analysis from (19.1) but vi-\nsualized as a tree, alongside its corresponding phrase-structure analysis of the kind\ngiven in the prior chapter. Note the absence of nodes corresponding to phrasal con-\nstituents or lexical categories in the dependency parse; the internal structure of the\ndependency parse consists solely of directed relations between words. These head-\ndependent relationships directly encode important information that is often buried in\nthe more complex phrase-structure parses. For example, the arguments to the verb\nprefer are directly linked to it in the dependency structure, while their connection\nto the main verb is more distant in the phrase-structure tree. Similarly, morning\nandDenver , modi\ufb01ers of \ufb02ight , are linked to it directly in the dependency structure.\nThis fact that the head-dependent relations are a good proxy for the semantic rela-\ntionship between predicates and their arguments is an important reason why depen-\ndency grammars are currently more common than constituency grammars in natural\nlanguage processing.\nAnother major advantage of dependency grammars is their ability to deal with\nlanguages that have a relatively free word order . For example, word order in Czech free word order\ncan be much more \ufb02exible than in English; a grammatical object might occur before\nor after a location adverbial . A phrase-structure grammar would need a separate rule 412 CHAPTER 19 \u2022 D EPENDENCY PARSING\nprefer\n\ufb02ight\nDenver\nthroughmorning theIS\nVP\nNP\nNom\nPP\nNP\nPro\nDenverP\nthroughNom\nNoun\n\ufb02ightNom\nNoun\nmorningDet\ntheVerb\npreferNP\nPro\nI\nFigure 19.1 Dependency and constituent analyses for I prefer the morning \ufb02ight through Denver.\nfor each possible place in the parse tree where such an adverbial phrase could occur.\nA dependency-based approach can have just one link type representing this particu-\nlar adverbial relation; dependency grammar approaches can thus abstract away a bit\nmore from word order information.\nIn the following sections, we\u2019ll give an inventory of relations used in dependency\nparsing, discuss two families of parsing algorithms (transition-based, and graph-\nbased), and discuss evaluation.\n19.1 Dependency Relations\nThe traditional linguistic notion of grammatical relation provides the basis for thegrammatical\nrelation\nbinary relations that comprise these dependency structures. The arguments to these\nrelations consist of a head and a dependent . The head plays the role of the central head\ndependent organizing word, and the dependent as a kind of modi\ufb01er. The head-dependent rela-\ntionship is made explicit by directly linking heads to the words that are immediately\ndependent on them.\nIn addition to specifying the head-dependent pairs, dependency grammars allow\nus to classify the kinds of grammatical relations, or grammatical function that thegrammatical\nfunction\ndependent plays with respect to its head. These include familiar notions such as\nsubject ,direct object andindirect object . In English these notions strongly corre-\nlate with, but by no means determine, both position in a sentence and constituent\ntype and are therefore somewhat redundant with the kind of information found in\nphrase-structure trees. However, in languages with more \ufb02exible word order, the\ninformation encoded directly in these grammatical relations is critical since phrase-\nbased constituent syntax provides little help.\nLinguists have developed taxonomies of relations that go well beyond the famil-\niar notions of subject and object. While there is considerable variation from theory 19.1 \u2022 D EPENDENCY RELATIONS 413\nClausal Argument Relations Description\nNSUBJ Nominal subject\nOBJ Direct object\nIOBJ Indirect object\nCCOMP Clausal complement\nNominal Modi\ufb01er Relations Description\nNMOD Nominal modi\ufb01er\nAMOD Adjectival modi\ufb01er\nAPPOS Appositional modi\ufb01er\nDET Determiner\nCASE Prepositions, postpositions and other case markers\nOther Notable Relations Description\nCONJ Conjunct\nCC Coordinating conjunction\nFigure 19.2 Some of the Universal Dependency relations (de Marneffe et al., 2021).\nto theory, there is enough commonality that cross-linguistic standards have been\ndeveloped. The Universal Dependencies (UD) project (de Marneffe et al., 2021),Universal\nDependencies\nan open community effort to annotate dependencies and other aspects of grammar\nacross more than 100 languages, provides an inventory of 37 dependency relations.\nFig. 19.2 shows a subset of the UD relations and Fig. 19.3 provides some examples.\nThe motivation for all of the relations in the Universal Dependency scheme is\nbeyond the scope of this chapter, but the core set of frequently used relations can be\nbroken into two sets: clausal relations that describe syntactic roles with respect to a\npredicate (often a verb), and modi\ufb01er relations that categorize the ways that words\ncan modify their heads.\nConsider, for example, the following sentence:\nUnited canceled themorning \ufb02ights toHoustonnsubjobj\ndet\ncompoundnmod\ncaseroot\n(19.2)\nHere the clausal relations NSUBJ and OBJidentify the subject and direct object of\nthe predicate cancel , while the NMOD ,DET, and CASE relations denote modi\ufb01ers of\nthe nouns \ufb02ights andHouston .\n19.1.1 Dependency Formalisms\nA dependency structure can be represented as a directed graph G= (V;A), consisting\nof a set of vertices V, and a set of ordered pairs of vertices A, which we\u2019ll call arcs.\nFor the most part we will assume that the set of vertices, V, corresponds exactly\nto the set of words in a given sentence. However, they might also correspond to\npunctuation, or when dealing with morphologically complex languages the set of\nvertices might consist of stems and af\ufb01xes. The set of arcs, A, captures the head-\ndependent and grammatical function relationships between the elements in V.\nDifferent grammatical theories or formalisms may place further constraints on\nthese dependency structures. Among the more frequent restrictions are that the struc-\ntures must be connected, have a designated root node, and be acyclic or planar. Of\nmost relevance to the parsing approaches discussed in this chapter is the common, 414 CHAPTER 19 \u2022 D EPENDENCY PARSING\nRelation Examples with head anddependent\nNSUBJ United canceled the \ufb02ight.\nOBJ United diverted the\ufb02ight to Reno.\nWebooked her the \ufb01rst \ufb02ight to Miami.\nIOBJ Webooked herthe \ufb02ight to Miami.\nCOMPOUND We took the morning \ufb02ight .\nNMOD \ufb02ight toHouston .\nAMOD Book the cheapest \ufb02ight .\nAPPOS United , aunit of UAL, matched the fares.\nDET The \ufb02ight was canceled.\nWhich \ufb02ight was delayed?\nCONJ We\ufb02ewto Denver and drove to Steamboat.\nCC We \ufb02ew to Denver anddrove to Steamboat.\nCASE Book the \ufb02ight through Houston .\nFigure 19.3 Examples of some Universal Dependency relations.\ncomputationally-motivated, restriction to rooted trees. That is, a dependency treedependency\ntree\nis a directed graph that satis\ufb01es the following constraints:\n1. There is a single designated root node that has no incoming arcs.\n2. With the exception of the root node, each vertex has exactly one incoming arc.\n3. There is a unique path from the root node to each vertex in V.\nTaken together, these constraints ensure that each word has a single head, that the\ndependency structure is connected, and that there is a single root node from which\none can follow a unique directed path to each of the words in the sentence.\n19.1.2 Projectivity\nThe notion of projectivity imposes an additional constraint that is derived from the\norder of the words in the input. An arc from a head to a dependent is said to be\nprojective if there is a path from the head to every word that lies between the head projective\nand the dependent in the sentence. A dependency tree is then said to be projective if\nall the arcs that make it up are projective. All the dependency trees we\u2019ve seen thus\nfar have been projective. There are, however, many valid constructions which lead\nto non-projective trees, particularly in languages with relatively \ufb02exible word order.\nConsider the following example.\nJetBlue canceled our \ufb02ight this morning which was already latensubjobjobl\ndetacl:relcl\ndet nsubjcop\nadvroot\n(19.3)\nIn this example, the arc from \ufb02ight to its modi\ufb01er lateis non-projective since there\nis no path from \ufb02ight to the intervening words thisandmorning . As we can see from\nthis diagram, projectivity (and non-projectivity) can be detected in the way we\u2019ve\nbeen drawing our trees. A dependency tree is projective if it can be drawn with\nno crossing edges. Here there is no way to link \ufb02ight to its dependent latewithout\ncrossing the arc that links morning to its head. 19.1 \u2022 D EPENDENCY RELATIONS 415\nOur concern with projectivity arises from two related issues. First, the most\nwidely used English dependency treebanks were automatically derived from phrase-\nstructure treebanks through the use of head-\ufb01nding rules. The trees generated in such\na fashion will always be projective, and hence will be incorrect when non-projective\nexamples like this one are encountered.\nSecond, there are computational limitations to the most widely used families of\nparsing algorithms. The transition-based approaches discussed in Section 19.2 can\nonly produce projective trees, hence any sentences with non-projective structures\nwill necessarily contain some errors. This limitation is one of the motivations for\nthe more \ufb02exible graph-based parsing approach described in Section 19.3.\n19.1.3 Dependency Treebanks\nTreebanks play a critical role in the development and evaluation of dependency\nparsers. They are used for training parsers, they act as the gold labels for evaluating\nparsers, and they also provide useful information for corpus linguistics studies.\nDependency treebanks are created by having human annotators directly generate\ndependency structures for a given corpus, or by hand-correcting the output of an\nautomatic parser. A few early treebanks were also based on using a deterministic\nprocess to translate existing constituent-based treebanks into dependency trees.\nThe largest open community project for building dependency trees is the Univer-\nsal Dependencies project at https://universaldependencies.org/ introduced\nabove, which currently has almost 200 dependency treebanks in more than 100 lan-\nguages (de Marneffe et al., 2021). Here are a few UD examples showing dependency\ntrees for sentences in Spanish, Basque, and Mandarin Chinese:\nVERB ADP DET NOUN ADP DET NUM PUNCT\nSubiremos a el tren a las cinco .\nwe-will-board on the train at the \ufb01ve .obl\ndetcase\ndetobl:tmod\ncasepunct\n[Spanish] Subiremos al tren a las cinco. \u201cWe will be boarding the train at \ufb01ve.\u201d (19.4)\nNOUN NOUN VERB AUX PUNCT\nEkaitzak itsasontzia hondoratu du .\nstorm (Erg.) ship (Abs.) sunk has .nsubj\nobj auxpunct\n[Basque] Ekaitzak itsasontzia hondoratu du. \u201cThe storm has sunk the ship.\u201d (19.5) 416 CHAPTER 19 \u2022 D EPENDENCY PARSING\nADV PRON NOUN ADV VERB VERB NOUN\n\u4f46\u6211\u6628\u5929 \u624d \u6536 \u5230\u4fe1\nbut I yesterday only-then receive arrive letter .adv\nnsubj\nobj:tmod\nadvmod compound:vvobj\n[Chinese]\u4f46\u6211\u6628\u5929\u624d\u6536\u5230\u4fe1\u201cBut I didn\u2019t receive the letter until yesterday\u201d (19.6)\n19.2 Transition-Based Dependency Parsing\nOur \ufb01rst approach to dependency parsing is called transition-based parsing. This transition-based\narchitecture draws on shift-reduce parsing , a paradigm originally developed for\nanalyzing programming languages (Aho and Ullman, 1972). In transition-based\nparsing we\u2019ll have a stack on which we build the parse, a buffer of tokens to be\nparsed, and a parser which takes actions on the parse via a predictor called an oracle ,\nas illustrated in Fig. 19.4.\nwnw1w2s2...s1snParserInput bu\ufb00erStackOracleLEFTARCRIGHTARCSHIFTActionDependencyRelationsw3w2\nFigure 19.4 Basic transition-based parser. The parser examines the top two elements of the\nstack and selects an action by consulting an oracle that examines the current con\ufb01guration.\nThe parser walks through the sentence left-to-right, successively shifting items\nfrom the buffer onto the stack. At each time point we examine the top two elements\non the stack, and the oracle makes a decision about what transition to apply to build\nthe parse. The possible transitions correspond to the intuitive actions one might take\nin creating a dependency tree by examining the words in a single pass over the input\nfrom left to right (Covington, 2001):\n\u2022 Assign the current word as the head of some previously seen word,\n\u2022 Assign some previously seen word as the head of the current word,\n\u2022 Postpone dealing with the current word, storing it for later processing.\nWe\u2019ll formalize this intuition with the following three transition operators that\nwill operate on the top two elements of the stack:\n\u2022LEFT ARC: Assert a head-dependent relation between the word at the top of\nthe stack and the second word; remove the second word from the stack.\n\u2022RIGHT ARC: Assert a head-dependent relation between the second word on\nthe stack and the word at the top; remove the top word from the stack; 19.2 \u2022 T RANSITION -BASED DEPENDENCY PARSING 417\n\u2022SHIFT : Remove the word from the front of the input buffer and push it onto\nthe stack.\nWe\u2019ll sometimes call operations like LEFT ARCand RIGHT ARCreduce operations,\nbased on a metaphor from shift-reduce parsing, in which reducing means combin-\ning elements on the stack. There are some preconditions for using operators. The\nLEFT ARCoperator cannot be applied when ROOT is the second element of the stack\n(since by de\ufb01nition the ROOT node cannot have any incoming arcs). And both the\nLEFT ARCand RIGHT ARCoperators require two elements to be on the stack to be\napplied.\nThis particular set of operators implements what is known as the arc standard arc standard\napproach to transition-based parsing (Covington 2001, Nivre 2003). In arc standard\nparsing the transition operators only assert relations between elements at the top of\nthe stack, and once an element has been assigned its head it is removed from the\nstack and is not available for further processing. As we\u2019ll see, there are alterna-\ntive transition systems which demonstrate different parsing behaviors, but the arc\nstandard approach is quite effective and is simple to implement.\nThe speci\ufb01cation of a transition-based parser is quite simple, based on repre-\nsenting the current state of the parse as a con\ufb01guration : the stack, an input buffer con\ufb01guration\nof words or tokens, and a set of relations representing a dependency tree. Parsing\nmeans making a sequence of transitions through the space of possible con\ufb01gura-\ntions. We start with an initial con\ufb01guration in which the stack contains the ROOT\nnode, the buffer has the tokens in the sentence, and an empty set of relations repre-\nsents the parse. In the \ufb01nal goal state, the stack and the word list should be empty,\nand the set of relations will represent the \ufb01nal parse. Fig. 19.5 gives the algorithm.\nfunction DEPENDENCY PARSE (words )returns dependency tree\nstate f[root], [ words ], []g; initial con\ufb01guration\nwhile state not \ufb01nal\nt ORACLE (state ) ; choose a transition operator to apply\nstate APPLY (t,state ) ; apply it, creating a new state\nreturn state\nFigure 19.5 A generic transition-based dependency parser\nAt each step, the parser consults an oracle (we\u2019ll come back to this shortly) that\nprovides the correct transition operator to use given the current con\ufb01guration. It then\napplies that operator to the current con\ufb01guration, producing a new con\ufb01guration.\nThe process ends when all the words in the sentence have been consumed and the\nROOT node is the only element remaining on the stack.\nThe ef\ufb01ciency of transition-based parsers should be apparent from the algorithm.\nThe complexity is linear in the length of the sentence since it is based on a single\nleft to right pass through the words in the sentence. (Each word must \ufb01rst be shifted\nonto the stack and then later reduced.)\nNote that unlike the dynamic programming and search-based approaches dis-\ncussed in Chapter 18, this approach is a straightforward greedy algorithm\u2014the or-\nacle provides a single choice at each step and the parser proceeds with that choice,\nno other options are explored, no backtracking is employed, and a single parse is\nreturned in the end.\nFigure 19.6 illustrates the operation of the parser with the sequence of transitions 418 CHAPTER 19 \u2022 D EPENDENCY PARSING\nleading to a parse for the following example.\nBook methemorning \ufb02ightiobjobj\ndet\ncompoundroot\n(19.7)\nLet\u2019s consider the state of the con\ufb01guration at Step 2, after the word mehas been\npushed onto the stack.\nStack Word List Relations\n[root, book, me] [the, morning, \ufb02ight]\nThe correct operator to apply here is RIGHT ARCwhich assigns book as the head of\nmeand pops mefrom the stack resulting in the following con\ufb01guration.\nStack Word List Relations\n[root, book] [the, morning, \ufb02ight] (book!me)\nAfter several subsequent applications of the SHIFT operator, the con\ufb01guration in\nStep 6 looks like the following:\nStack Word List Relations\n[root, book, the, morning, \ufb02ight] [] (book!me)\nHere, all the remaining words have been passed onto the stack and all that is left\nto do is to apply the appropriate reduce operators. In the current con\ufb01guration, we\nemploy the LEFT ARCoperator resulting in the following state.\nStack Word List Relations\n[root, book, the, \ufb02ight] [] (book!me)\n(morning \ufb02ight)\nAt this point, the parse for this sentence consists of the following structure.\nBook methemorning \ufb02ightiobj compound\n(19.8)\nThere are several important things to note when examining sequences such as\nthe one in Figure 19.6. First, the sequence given is not the only one that might lead\nto a reasonable parse. In general, there may be more than one path that leads to the\nsame result, and due to ambiguity, there may be other transition sequences that lead\nto different equally valid parses.\nSecond, we are assuming that the oracle always provides the correct operator\nat each point in the parse\u2014an assumption that is unlikely to be true in practice.\nAs a result, given the greedy nature of this algorithm, incorrect choices will lead to\nincorrect parses since the parser has no opportunity to go back and pursue alternative\nchoices. Section 19.2.4 will introduce several techniques that allow transition-based\napproaches to explore the search space more fully. 19.2 \u2022 T RANSITION -BASED DEPENDENCY PARSING 419\nStep Stack Word List Action Relation Added\n0 [root] [book, me, the, morning, \ufb02ight] SHIFT\n1 [root, book] [me, the, morning, \ufb02ight] SHIFT\n2 [root, book, me] [the, morning, \ufb02ight] RIGHT ARC (book!me)\n3 [root, book] [the, morning, \ufb02ight] SHIFT\n4 [root, book, the] [morning, \ufb02ight] SHIFT\n5 [root, book, the, morning] [\ufb02ight] SHIFT\n6 [root, book, the, morning, \ufb02ight] [] LEFT ARC (morning \ufb02ight)\n7 [root, book, the, \ufb02ight] [] LEFT ARC (the \ufb02ight)\n8 [root, book, \ufb02ight] [] RIGHT ARC (book!\ufb02ight)\n9 [root, book] [] RIGHT ARC (root!book)\n10 [root] [] Done\nFigure 19.6 Trace of a transition-based parse.\nFinally, for simplicity, we have illustrated this example without the labels on\nthe dependency relations. To produce labeled trees, we can parameterize the LEFT -\nARCand RIGHT ARCoperators with dependency labels, as in LEFT ARC(NSUBJ ) or\nRIGHT ARC(OBJ). This is equivalent to expanding the set of transition operators from\nour original set of three to a set that includes LEFT ARCand RIGHT ARCoperators for\neach relation in the set of dependency relations being used, plus an additional one\nfor the SHIFT operator. This, of course, makes the job of the oracle more dif\ufb01cult\nsince it now has a much larger set of operators from which to choose.\n19.2.1 Creating an Oracle\nThe oracle for greedily selecting the appropriate transition is trained by supervised\nmachine learning. As with all supervised machine learning methods, we will need\ntraining data: con\ufb01gurations annotated with the correct transition to take. We can\ndraw these from dependency trees. And we need to extract features of the con-\n\ufb01guration. We\u2019ll introduce neural classi\ufb01ers that represent the con\ufb01guration via\nembeddings, as well as classic systems that use hand-designed features.\nGenerating Training Data\nThe oracle from the algorithm in Fig. 19.5 takes as input a con\ufb01guration and returns a\ntransition operator. Therefore, to train a classi\ufb01er, we will need con\ufb01gurations paired\nwith transition operators (i.e., LEFT ARC,RIGHT ARC, or SHIFT ). Unfortunately,\ntreebanks pair entire sentences with their corresponding trees, not con\ufb01gurations\nwith transitions.\nTo generate the required training data, we employ the oracle-based parsing algo-\nrithm in a clever way. We supply our oracle with the training sentences to be parsed\nalong with their corresponding reference parses from the treebank. To produce train-\ning instances, we then simulate the operation of the parser by running the algorithm\nand relying on a new training oracle to give us correct transition operators for each training oracle\nsuccessive con\ufb01guration.\nTo see how this works, let\u2019s \ufb01rst review the operation of our parser. It begins with\na default initial con\ufb01guration where the stack contains the ROOT , the input list is just\nthe list of words, and the set of relations is empty. The LEFT ARCand RIGHT ARC\noperators each add relations between the words at the top of the stack to the set of\nrelations being accumulated for a given sentence. Since we have a gold-standard\nreference parse for each training sentence, we know which dependency relations are\nvalid for a given sentence. Therefore, we can use the reference parse to guide the 420 CHAPTER 19 \u2022 D EPENDENCY PARSING\nStep Stack Word List Predicted Action\n0 [root] [book, the, \ufb02ight, through, houston] SHIFT\n1 [root, book] [the, \ufb02ight, through, houston] SHIFT\n2 [root, book, the] [\ufb02ight, through, houston] SHIFT\n3 [root, book, the, \ufb02ight] [through, houston] LEFT ARC\n4 [root, book, \ufb02ight] [through, houston] SHIFT\n5 [root, book, \ufb02ight, through] [houston] SHIFT\n6 [root, book, \ufb02ight, through, houston] [] LEFT ARC\n7 [root, book, \ufb02ight, houston ] [] RIGHT ARC\n8 [root, book, \ufb02ight] [] RIGHT ARC\n9 [root, book] [] RIGHT ARC\n10 [root] [] Done\nFigure 19.7 Generating training items consisting of con\ufb01guration/predicted action pairs by simulating a parse\nwith a given reference parse.\nselection of operators as the parser steps through a sequence of con\ufb01gurations.\nTo be more precise, given a reference parse and a con\ufb01guration, the training\noracle proceeds as follows:\n\u2022 Choose LEFT ARCif it produces a correct head-dependent relation given the\nreference parse and the current con\ufb01guration,\n\u2022 Otherwise, choose RIGHT ARCif (1) it produces a correct head-dependent re-\nlation given the reference parse and (2) all of the dependents of the word at\nthe top of the stack have already been assigned,\n\u2022 Otherwise, choose SHIFT .\nThe restriction on selecting the RIGHT ARCoperator is needed to ensure that a\nword is not popped from the stack, and thus lost to further processing, before all its\ndependents have been assigned to it.\nMore formally, during training the oracle has access to the following:\n\u2022 A current con\ufb01guration with a stack Sand a set of dependency relations Rc\n\u2022 A reference parse consisting of a set of vertices Vand a set of dependency\nrelations Rp\nGiven this information, the oracle chooses transitions as follows:\nLEFT ARC(r):if(S1r S2)2Rp\nRIGHT ARC(r):if(S2r S1)2Rpand8r0;w s:t:(S1r0w)2Rpthen(S1r0w)2Rc\nSHIFT :otherwise\nLet\u2019s walk through the processing of the following example as shown in Fig. 19.7.\nBook the\ufb02ight through Houstonobj\ndetnmod\ncaseroot\n(19.9)\nAt Step 1, LEFT ARCis not applicable in the initial con\ufb01guration since it asserts\na relation, (root book), not in the reference answer; RIGHT ARCdoes assert a\nrelation contained in the \ufb01nal answer (root !book), however book has not been\nattached to any of its dependents yet, so we have to defer, leaving SHIFT as the only 19.2 \u2022 T RANSITION -BASED DEPENDENCY PARSING 421\npossible action. The same conditions hold in the next two steps. In step 3, LEFT ARC\nis selected to link theto its head.\nNow consider the situation in Step 4.\nStack Word buffer Relations\n[root, book, \ufb02ight] [through, Houston] (the \ufb02ight)\nHere, we might be tempted to add a dependency relation between book and\ufb02ight ,\nwhich is present in the reference parse. But doing so now would prevent the later\nattachment of Houston since \ufb02ight would have been removed from the stack. For-\ntunately, the precondition on choosing RIGHT ARCprevents this choice and we\u2019re\nagain left with SHIFT as the only viable option. The remaining choices complete the\nset of operators needed for this example.\nTo recap, we derive appropriate training instances consisting of con\ufb01guration-\ntransition pairs from a treebank by simulating the operation of a parser in the con-\ntext of a reference dependency tree. We can deterministically record correct parser\nactions at each step as we progress through each training example, thereby creating\nthe training set we require.\n19.2.2 A feature-based classi\ufb01er\nWe\u2019ll now introduce two classi\ufb01ers for choosing transitions, here a classic feature-\nbased algorithm and in the next section a neural classi\ufb01er using embedding features.\nFeatured-based classi\ufb01ers generally use the same features we\u2019ve seen with part-\nof-speech tagging and partial parsing: Word forms, lemmas, parts of speech, the\nhead, and the dependency relation to the head. Other features may be relevant for\nsome languages, for example morphosyntactic features like case marking on subjects\nor objects. The features are extracted from the training con\ufb01gurations , which consist\nof the stack, the buffer and the current set of relations. Most useful are features\nreferencing the top levels of the stack, the words near the front of the buffer, and the\ndependency relations already associated with any of those elements.\nWe\u2019ll use a feature template as we did for sentiment analysis and part-of-speechfeature\ntemplate\ntagging. Feature templates allow us to automatically generate large numbers of spe-\nci\ufb01c features from a training set. For example, consider the following feature tem-\nplates that are based on single positions in a con\ufb01guration.\nhs1:w;opi;hs2:w;opihs1:t;opi;hs2:t;opi\nhb1:w;opi;hb1:t;opihs1:wt;opi (19.10)\nHere features are denoted as location:property , where s= stack, b= the word\nbuffer, w= word forms, t= part-of-speech, and op= operator. Thus the feature for\nthe word form at the top of the stack would be s1:w, the part of speech tag at the\nfront of the buffer b1:t, and the concatenated feature s1:wtrepresents the word form\nconcatenated with the part of speech of the word at the top of the stack. Consider\napplying these templates to the following intermediate con\ufb01guration derived from a\ntraining oracle for (19.2).\nStack Word buffer Relations\n[root, canceled, \ufb02ights] [to Houston] (canceled!United)\n(\ufb02ights!morning)\n(\ufb02ights!the) 422 CHAPTER 19 \u2022 D EPENDENCY PARSING\nThe correct transition here is SHIFT (you should convince yourself of this before\nproceeding). The application of our set of feature templates to this con\ufb01guration\nwould result in the following set of instantiated features.\nhs1:w=\ufb02ights;op=shifti (19.11)\nhs2:w=canceled;op=shifti\nhs1:t=NNS;op=shifti\nhs2:t=VBD;op=shifti\nhb1:w=to;op=shifti\nhb1:t=TO;op=shifti\nhs1:wt=\ufb02ightsNNS;op=shifti\nGiven that the left and right arc transitions operate on the top two elements of the\nstack, features that combine properties from these positions are even more useful.\nFor example, a feature like s1:t\u000es2:tconcatenates the part of speech tag of the word\nat the top of the stack with the tag of the word beneath it.\nhs1:t\u000es2:t=NNSVBD;op=shifti (19.12)\nGiven the training data and features, any classi\ufb01er, like multinomial logistic re-\ngression or support vector machines, can be used.\n19.2.3 A neural classi\ufb01er\nThe oracle can also be implemented by a neural classi\ufb01er. A standard architecture\nis simply to pass the sentence through an encoder, then take the presentation of the\ntop 2 words on the stack and the \ufb01rst word of the buffer, concatenate them, and\npresent to a feedforward network that predicts the transition to take (Kiperwasser\nand Goldberg, 2016; Kulmizev et al., 2019). Fig. 19.8 sketches this model. Learning\ncan be done with cross-entropy loss.\nw\u2026s2...s1Input bu\ufb00erStackLEFTARCRIGHTARCSHIFTActionDependencyRelationsw3w2ENCODERw1w2w3w4w5w6Parser OracleSoftmaxFFNws1s2e(w)e(s1)e(s2)\nFigure 19.8 Neural classi\ufb01er for the oracle for the transition-based parser. The parser takes\nthe top 2 words on the stack and the \ufb01rst word of the buffer, represents them by their encodings\n(from running the whole sentence through the encoder), concatenates the embeddings and\npasses through a softmax to choose a parser action (transition). 19.2 \u2022 T RANSITION -BASED DEPENDENCY PARSING 423\n19.2.4 Advanced Methods in Transition-Based Parsing\nThe basic transition-based approach can be elaborated in a number of ways to im-\nprove performance by addressing some of the most obvious \ufb02aws in the approach.\nAlternative Transition Systems\nThe arc-standard transition system described above is only one of many possible sys-\ntems. A frequently used alternative is the arc eager transition system. The arc eager arc eager\napproach gets its name from its ability to assert rightward relations much sooner\nthan in the arc standard approach. To see this, let\u2019s revisit the arc standard trace of\nExample 19.9, repeated here.\nBook the\ufb02ight through Houstonobj\ndetnmod\ncaseroot\nConsider the dependency relation between book and\ufb02ight in this analysis. As\nis shown in Fig. 19.7, an arc-standard approach would assert this relation at Step 8,\ndespite the fact that book and\ufb02ight \ufb01rst come together on the stack much earlier at\nStep 4. The reason this relation can\u2019t be captured at this point is due to the presence\nof the postnominal modi\ufb01er through Houston . In an arc-standard approach, depen-\ndents are removed from the stack as soon as they are assigned their heads. If \ufb02ight\nhad been assigned book as its head in Step 4, it would no longer be available to serve\nas the head of Houston .\nWhile this delay doesn\u2019t cause any issues in this example, in general the longer\na word has to wait to get assigned its head the more opportunities there are for\nsomething to go awry. The arc-eager system addresses this issue by allowing words\nto be attached to their heads as early as possible, before all the subsequent words\ndependent on them have been seen. This is accomplished through minor changes to\ntheLEFT ARCand RIGHT ARCoperators and the addition of a new REDUCE operator.\n\u2022LEFT ARC: Assert a head-dependent relation between the word at the front of\nthe input buffer and the word at the top of the stack; pop the stack.\n\u2022RIGHT ARC: Assert a head-dependent relation between the word on the top of\nthe stack and the word at the front of the input buffer; shift the word at the\nfront of the input buffer to the stack.\n\u2022SHIFT : Remove the word from the front of the input buffer and push it onto\nthe stack.\n\u2022REDUCE : Pop the stack.\nThe LEFT ARCand RIGHT ARCoperators are applied to the top of the stack and\nthe front of the input buffer, instead of the top two elements of the stack as in the\narc-standard approach. The RIGHT ARCoperator now moves the dependent to the\nstack from the buffer rather than removing it, thus making it available to serve as the\nhead of following words. The new REDUCE operator removes the top element from\nthe stack. Together these changes permit a word to be eagerly assigned its head and\nstill allow it to serve as the head for later dependents. The trace shown in Fig. 19.9\nillustrates the new decision sequence for this example.\nIn addition to demonstrating the arc-eager transition system, this example demon-\nstrates the power and \ufb02exibility of the overall transition-based approach. We were\nable to swap in a new transition system without having to make any changes to the 424 CHAPTER 19 \u2022 D EPENDENCY PARSING\nStep Stack Word List Action Relation Added\n0 [root] [book, the, \ufb02ight, through, houston] RIGHT ARC (root!book)\n1 [root, book] [the, \ufb02ight, through, houston] SHIFT\n2 [root, book, the] [\ufb02ight, through, houston] LEFT ARC (the \ufb02ight)\n3 [root, book] [\ufb02ight, through, houston] RIGHT ARC (book!\ufb02ight)\n4 [root, book, \ufb02ight] [through, houston] SHIFT\n5 [root, book, \ufb02ight, through] [houston] LEFT ARC (through houston)\n6 [root, book, \ufb02ight] [houston] RIGHT ARC (\ufb02ight!houston)\n7 [root, book, \ufb02ight, houston] [] REDUCE\n8 [root, book, \ufb02ight] [] REDUCE\n9 [root, book] [] REDUCE\n10 [root] [] Done\nFigure 19.9 A processing trace of Book the \ufb02ight through Houston using the arc-eager transition operators.\nunderlying parsing algorithm. This \ufb02exibility has led to the development of a di-\nverse set of transition systems that address different aspects of syntax and semantics\nincluding: assigning part of speech tags (Choi and Palmer, 2011a), allowing the\ngeneration of non-projective dependency structures (Nivre, 2009), assigning seman-\ntic roles (Choi and Palmer, 2011b), and parsing texts containing multiple languages\n(Bhat et al., 2017).\nBeam Search\nThe computational ef\ufb01ciency of the transition-based approach discussed earlier de-\nrives from the fact that it makes a single pass through the sentence, greedily making\ndecisions without considering alternatives. Of course, this is also a weakness \u2013 once\na decision has been made it can not be undone, even in the face of overwhelming\nevidence arriving later in a sentence. We can use beam search to explore alterna- beam search\ntive decision sequences. Recall from Chapter 9 that beam search uses a breadth-\ufb01rst\nsearch strategy with a heuristic \ufb01lter that prunes the search frontier to stay within a\n\ufb01xed-size beam width . beam width\nIn applying beam search to transition-based parsing, we\u2019ll elaborate on the al-\ngorithm given in Fig. 19.5. Instead of choosing the single best transition operator\nat each iteration, we\u2019ll apply all applicable operators to each state on an agenda and\nthen score the resulting con\ufb01gurations. We then add each of these new con\ufb01gura-\ntions to the frontier, subject to the constraint that there has to be room within the\nbeam. As long as the size of the agenda is within the speci\ufb01ed beam width, we can\nadd new con\ufb01gurations to the agenda. Once the agenda reaches the limit, we only\nadd new con\ufb01gurations that are better than the worst con\ufb01guration on the agenda\n(removing the worst element so that we stay within the limit). Finally, to insure that\nwe retrieve the best possible state on the agenda, the while loop continues as long as\nthere are non-\ufb01nal states on the agenda.\nThe beam search approach requires a more elaborate notion of scoring than we\nused with the greedy algorithm. There, we assumed that the oracle would be a\nsupervised classi\ufb01er that chose the best transition operator based on features of the\ncurrent con\ufb01guration. This choice can be viewed as assigning a score to all the\npossible transitions and picking the best one.\n\u02c6T(c) =argmaxScore (t;c)\nWith beam search we are now searching through the space of decision sequences,\nso it makes sense to base the score for a con\ufb01guration on its entire history. So we\ncan de\ufb01ne the score for a new con\ufb01guration as the score of its predecessor plus the 19.3 \u2022 G RAPH -BASED DEPENDENCY PARSING 425\nscore of the operator used to produce it.\nCon\ufb01gScore (c0) = 0:0\nCon\ufb01gScore (ci) = Con\ufb01gScore (ci\u00001)+Score (ti;ci\u00001)\nThis score is used both in \ufb01ltering the agenda and in selecting the \ufb01nal answer. The\nnew beam search version of transition-based parsing is given in Fig. 19.10.\nfunction DEPENDENCY BEAM PARSE (words ,width )returns dependency tree\nstate f[root], [ words ], [], 0.0 g;initial con\ufb01guration\nagenda hstatei ;initial agenda\nwhile agenda contains non-\ufb01nal states\nnewagenda hi\nfor each state2agenda do\nfor all ftjt2VALID OPERATORS (state )gdo\nchild APPLY (t,state )\nnewagenda ADDTOBEAM (child ,newagenda ,width )\nagenda newagenda\nreturn BESTOF(agenda )\nfunction ADDTOBEAM (state ,agenda ,width )returns updated agenda\nifLENGTH (agenda )<width then\nagenda INSERT (state ,agenda )\nelse if SCORE (state )>SCORE (WORST OF(agenda ))\nagenda REMOVE (WORST OF(agenda ))\nagenda INSERT (state ,agenda )\nreturn agenda\nFigure 19.10 Beam search applied to transition-based dependency parsing.\n19.3 Graph-Based Dependency Parsing\nGraph-based methods are the second important family of dependency parsing algo-\nrithms. Graph-based parsers are more accurate than transition-based parsers, espe-\ncially on long sentences; transition-based methods have trouble when the heads are\nvery far from the dependents (McDonald and Nivre, 2011). Graph-based methods\navoid this dif\ufb01culty by scoring entire trees, rather than relying on greedy local de-\ncisions. Furthermore, unlike transition-based approaches, graph-based parsers can\nproduce non-projective trees. Although projectivity is not a signi\ufb01cant issue for\nEnglish, it is de\ufb01nitely a problem for many of the world\u2019s languages.\nGraph-based dependency parsers search through the space of possible trees for a\ngiven sentence for a tree (or trees) that maximize some score. These methods encode\nthe search space as directed graphs and employ methods drawn from graph theory\nto search the space for optimal solutions. More formally, given a sentence Swe\u2019re\nlooking for the best dependency tree in Gs, the space of all possible trees for that\nsentence, that maximizes some score.\n\u02c6T(S) =argmax\nt2GSScore (t;S) 426 CHAPTER 19 \u2022 D EPENDENCY PARSING\nWe\u2019ll make the simplifying assumption that this score can be edge-factored , edge-factored\nmeaning that the overall score for a tree is the sum of the scores of each of the scores\nof the edges that comprise the tree.\nScore (t;S) =X\ne2tScore (e)\nGraph-based algorithms have to solve two problems: (1) assigning a score to\neach edge, and (2) \ufb01nding the best parse tree given the scores of all potential edges.\nIn the next few sections we\u2019ll introduce solutions to these two problems, beginning\nwith the second problem of \ufb01nding trees, and then giving a feature-based and a\nneural algorithm for solving the \ufb01rst problem of assigning scores.\n19.3.1 Parsing via \ufb01nding the maximum spanning tree\nIn graph-based parsing, given a sentence Swe start by creating a graph Gwhich is a\nfully-connected, weighted, directed graph where the vertices are the input words and\nthe directed edges represent all possible head-dependent assignments. We\u2019ll include\nan additional ROOT node with outgoing edges directed at all of the other vertices.\nThe weights of each edge in Gre\ufb02ect the score for each possible head-dependent\nrelation assigned by some scoring algorithm.\nIt turns out that \ufb01nding the best dependency parse for Sis equivalent to \ufb01nding\nthemaximum spanning tree over G. A spanning tree over a graph Gis a subsetmaximum\nspanning tree\nofGthat is a tree and covers all the vertices in G; a spanning tree over Gthat starts\nfrom the ROOT is a valid parse of S. A maximum spanning tree is the spanning tree\nwith the highest score. Thus a maximum spanning tree of Gemanating from the\nROOT is the optimal dependency parse for the sentence.\nA directed graph for the example Book that \ufb02ight is shown in Fig. 19.11, with the\nmaximum spanning tree corresponding to the desired parse shown in blue. For ease\nof exposition, we\u2019ll describe here the algorithm for unlabeled dependency parsing.\nrootBookthat\ufb02ight1244568757\nFigure 19.11 Initial rooted, directed graph for Book that \ufb02ight .\nBefore describing the algorithm it\u2019s useful to consider two intuitions about di-\nrected graphs and their spanning trees. The \ufb01rst intuition begins with the fact that\nevery vertex in a spanning tree has exactly one incoming edge. It follows from this\nthat every connected component of a spanning tree (i.e., every set of vertices that\nare linked to each other by paths over edges) will also have one incoming edge.\nThe second intuition is that the absolute values of the edge scores are not critical\nto determining its maximum spanning tree. Instead, it is the relative weights of the\nedges entering each vertex that matters. If we were to subtract a constant amount\nfrom each edge entering a given vertex it would have no impact on the choice of 19.3 \u2022 G RAPH -BASED DEPENDENCY PARSING 427\nthe maximum spanning tree since every possible spanning tree would decrease by\nexactly the same amount.\nThe \ufb01rst step of the algorithm itself is quite straightforward. For each vertex\nin the graph, an incoming edge (representing a possible head assignment) with the\nhighest score is chosen. If the resulting set of edges produces a spanning tree then\nwe\u2019re done. More formally, given the original fully-connected graph G= (V;E), a\nsubgraph T= (V;F)is a spanning tree if it has no cycles and each vertex (other than\nthe root) has exactly one edge entering it. If the greedy selection process produces\nsuch a tree then it is the best possible one.\nUnfortunately, this approach doesn\u2019t always lead to a tree since the set of edges\nselected may contain cycles. Fortunately, in yet another case of multiple discovery,\nthere is a straightforward way to eliminate cycles generated during the greedy se-\nlection phase. Chu and Liu (1965) and Edmonds (1967) independently developed\nan approach that begins with greedy selection and follows with an elegant recursive\ncleanup phase that eliminates cycles.\nThe cleanup phase begins by adjusting all the weights in the graph by subtracting\nthe score of the maximum edge entering each vertex from the score of all the edges\nentering that vertex. This is where the intuitions mentioned earlier come into play.\nWe have scaled the values of the edges so that the weights of the edges in the cycle\nhave no bearing on the weight of anyof the possible spanning trees. Subtracting the\nvalue of the edge with maximum weight from each edge entering a vertex results\nin a weight of zero for all of the edges selected during the greedy selection phase,\nincluding all of the edges involved in the cycle .\nHaving adjusted the weights, the algorithm creates a new graph by selecting a\ncycle and collapsing it into a single new node. Edges that enter or leave the cycle\nare altered so that they now enter or leave the newly collapsed node. Edges that do\nnot touch the cycle are included and edges within the cycle are dropped.\nNow, if we knew the maximum spanning tree of this new graph, we would have\nwhat we need to eliminate the cycle. The edge of the maximum spanning tree di-\nrected towards the vertex representing the collapsed cycle tells us which edge to\ndelete in order to eliminate the cycle. How do we \ufb01nd the maximum spanning tree\nof this new graph? We recursively apply the algorithm to the new graph. This will\neither result in a spanning tree or a graph with a cycle. The recursions can continue\nas long as cycles are encountered. When each recursion completes we expand the\ncollapsed vertex, restoring all the vertices and edges from the cycle with the excep-\ntion of the single edge to be deleted .\nPutting all this together, the maximum spanning tree algorithm consists of greedy\nedge selection, re-scoring of edge costs and a recursive cleanup phase when needed.\nThe full algorithm is shown in Fig. 19.12.\nFig. 19.13 steps through the algorithm with our Book that \ufb02ight example. The\n\ufb01rst row of the \ufb01gure illustrates greedy edge selection with the edges chosen shown\nin blue (corresponding to the set Fin the algorithm). This results in a cycle between\nthatand\ufb02ight . The scaled weights using the maximum value entering each node are\nshown in the graph to the right.\nCollapsing the cycle between that and\ufb02ight to a single node (labelled tf) and\nrecursing with the newly scaled costs is shown in the second row. The greedy selec-\ntion step in this recursion yields a spanning tree that links roottobook , as well as an\nedge that links book to the contracted node. Expanding the contracted node, we can\nsee that this edge corresponds to the edge from book to\ufb02ight in the original graph.\nThis in turn tells us which edge to drop to eliminate the cycle. 428 CHAPTER 19 \u2022 D EPENDENCY PARSING\nfunction MAXSPANNING TREE(G=(V ,E) ,root,score )returns spanning tree\nF []\nT\u2019 []\nscore\u2019 []\nfor each v2Vdo\nbestInEdge argmaxe=(u;v)2Escore[e]\nF F[bestInEdge\nfor each e=(u,v)2Edo\nscore\u2019[e] score[e]\u0000score[bestInEdge]\nifT=(V ,F) is a spanning tree then return it\nelse\nC a cycle in F\nG\u2019 CONTRACT (G,C)\nT\u2019 MAXSPANNING TREE(G\u2019,root,score\u2019 )\nT EXPAND (T\u2019,C)\nreturn T\nfunction CONTRACT (G,C)returns contracted graph\nfunction EXPAND (T,C)returns expanded graph\nFigure 19.12 The Chu-Liu Edmonds algorithm for \ufb01nding a maximum spanning tree in a\nweighted directed graph.\nOn arbitrary directed graphs, this version of the CLE algorithm runs in O(mn)\ntime, where mis the number of edges and nis the number of nodes. Since this par-\nticular application of the algorithm begins by constructing a fully connected graph\nm=n2yielding a running time of O(n3). Gabow et al. (1986) present a more ef\ufb01-\ncient implementation with a running time of O(m+nlogn ).\n19.3.2 A feature-based algorithm for assigning scores\nRecall that given a sentence, S, and a candidate tree, T, edge-factored parsing models\nmake the simpli\ufb01cation that the score for the tree is the sum of the scores of the edges\nthat comprise the tree:\nscore(S;T) =X\ne2Tscore(S;e)\nIn a feature-based algorithm we compute the edge score as a weighted sum of fea-\ntures extracted from it:\nscore(S;e) =NX\ni=1wifi(S;e)\nOr more succinctly.\nscore(S;e) = w\u0001f\nGiven this formulation, we need to identify relevant features and train the weights.\nThe features (and feature combinations) used to train edge-factored models mir-\nror those used in training transition-based parsers, such as 19.3 \u2022 G RAPH -BASED DEPENDENCY PARSING 429\nrootBooktf\nrootBookthat\ufb02ight0-3-4\n-7-1-6-2rootBook12that7\ufb02ight8-4-30-2-6-1-700\nrootBook0tf-10-3-4\n-7-1-6-2rootBook12that7\ufb02ight81244568757\nDeleted from cycle\nFigure 19.13 Chu-Liu-Edmonds graph-based example for Book that \ufb02ight\n\u2022 Wordforms, lemmas, and parts of speech of the headword and its dependent.\n\u2022 Corresponding features from the contexts before, after and between the words.\n\u2022 Word embeddings.\n\u2022 The dependency relation itself.\n\u2022 The direction of the relation (to the right or left).\n\u2022 The distance from the head to the dependent.\nGiven a set of features, our next problem is to learn a set of weights correspond-\ning to each. Unlike many of the learning problems discussed in earlier chapters,\nhere we are not training a model to associate training items with class labels, or\nparser actions. Instead, we seek to train a model that assigns higher scores to cor-\nrect trees than to incorrect ones. An effective framework for problems like this is to\nuseinference-based learning combined with the perceptron learning rule. In thisinference-based\nlearning\nframework, we parse a sentence (i.e, perform inference) from the training set using\nsome initially random set of initial weights. If the resulting parse matches the cor-\nresponding tree in the training data, we do nothing to the weights. Otherwise, we\n\ufb01nd those features in the incorrect parse that are notpresent in the reference parse\nand we lower their weights by a small amount based on the learning rate. We do this\nincrementally for each sentence in our training data until the weights converge. 430 CHAPTER 19 \u2022 D EPENDENCY PARSING\n19.3.3 A neural algorithm for assigning scores\nState-of-the-art graph-based multilingual parsers are based on neural networks. In-\nstead of extracting hand-designed features to represent each edge between words wi\nandwj, these parsers run the sentence through an encoder, and then pass the encoded\nrepresentation of the two words wiandwjthrough a network that estimates a score\nfor the edge i!j.\nbookthat\ufb02ightr1score(h1head, h3dep)Bia\ufb03neb\nENCODERUh1 headFFNheadFFNheadFFNdepFFNdeph1 depFFNheadFFNdeph2 headh2 deph3 headh3 depWr2r3\u2211+\nFigure 19.14 Computing scores for a single edge (book !\ufb02ight) in the biaf\ufb01ne parser of\nDozat and Manning (2017); Dozat et al. (2017). The parser uses distinct feedforward net-\nworks to turn the encoder output for each word into a head and dependent representation for\nthe word. The biaf\ufb01ne function turns the head embedding of the head and the dependent\nembedding of the dependent into a score for the dependency edge.\nHere we\u2019ll sketch the biaf\ufb01ne algorithm of Dozat and Manning (2017) and Dozat\net al. (2017) shown in Fig. 19.14, drawing on the work of Gr \u00a8unewald et al. (2021)\nwho tested many versions of the algorithm via their STEPS system. The algorithm\n\ufb01rst runs the sentence X=x1;:::;xnthrough an encoder to produce a contextual\nembedding representation for each token R=r1;:::;rn. The embedding for each\ntoken is now passed through two separate feedforward networks, one to produce a\nrepresentation of this token as a head, and one to produce a representation of this\ntoken as a dependent:\nhhead\ni=FFNhead(ri) (19.13)\nhdep\ni=FFNdep(ri) (19.14)\nNow to assign a score to the directed edge i!j, (wiis the head and wjis the depen-\ndent), we feed the head representation of i,hhead\ni, and the dependent representation\nofj,hdep\nj, into a biaf\ufb01ne scoring function:\nScore (i!j) = Biaff(hhead\ni;hdep\nj) (19.15)\nBiaff(x;y) = x|Uy+W(x\by)+b (19.16) 19.4 \u2022 E VALUATION 431\nwhere U,W, and bare weights learned by the model. The idea of using a biaf\ufb01ne\nfunction is to allow the system to learn multiplicative interactions between the vec-\ntorsxandy.\nIf we pass Score (i!j)through a softmax, we end up with a probability distri-\nbution, for each token j, over potential heads i(all other tokens in the sentence):\np(i!j) =softmax ([Score (k!j);8k6=j;1\u0014k\u0014n]) (19.17)\nThis probability can then be passed to the maximum spanning tree algorithm of\nSection 19.3.1 to \ufb01nd the best tree.\nThis p(i!j)classi\ufb01er is trained by optimizing the cross-entropy loss.\nNote that the algorithm as we\u2019ve described it is unlabeled. To make this into\na labeled algorithm, the Dozat and Manning (2017) algorithm actually trains two\nclassi\ufb01ers. The \ufb01rst classi\ufb01er, the edge-scorer , the one we described above, assigns\na probability p(i!j)to each word wiandwj. Then the Maximum Spanning Tree\nalgorithm is run to get a single best dependency parse tree for the second. We then\napply a second classi\ufb01er, the label-scorer , whose job is to \ufb01nd the maximum prob-\nability label for each edge in this parse. This second classi\ufb01er has the same form\nas (19.15-19.17), but instead of being trained to predict with binary softmax the\nprobability of an edge existing between two words, it is trained with a softmax over\ndependency labels to predict the dependency label between the words.\n19.4 Evaluation\nAs with phrase structure-based parsing, the evaluation of dependency parsers pro-\nceeds by measuring how well they work on a test set. An obvious metric would be\nexact match (EM)\u2014how many sentences are parsed correctly. This metric is quite\npessimistic, with most sentences being marked wrong. Such measures are not \ufb01ne-\ngrained enough to guide the development process. Our metrics need to be sensitive\nenough to tell if actual improvements are being made.\nFor these reasons, the most common method for evaluating dependency parsers\nare labeled and unlabeled attachment accuracy. Labeled attachment refers to the\nproper assignment of a word to its head along with the correct dependency relation.\nUnlabeled attachment simply looks at the correctness of the assigned head, ignor-\ning the dependency relation. Given a system output and a corresponding reference\nparse, accuracy is simply the percentage of words in an input that are assigned the\ncorrect head with the correct relation. These metrics are usually referred to as the\nlabeled attachment score (LAS) and unlabeled attachment score (UAS). Finally, we\ncan make use of a label accuracy score (LS), the percentage of tokens with correct\nlabels, ignoring where the relations are coming from.\nAs an example, consider the reference parse and system parse for the following\nexample shown in Fig. 19.15.\n(19.18) Book me the \ufb02ight through Houston.\nThe system correctly \ufb01nds 4 of the 6 dependency relations present in the reference\nparse and receives an LAS of 2/3. However, one of the 2 incorrect relations found\nby the system holds between book and\ufb02ight , which are in a head-dependent relation\nin the reference parse; the system therefore achieves a UAS of 5/6.\nBeyond attachment scores, we may also be interested in how well a system is\nperforming on a particular kind of dependency relation, for example NSUBJ , across 432 CHAPTER 19 \u2022 D EPENDENCY PARSING\nBook methe \ufb02ight through Houston\n(a) Referenceiobjobj\ndetnmod\ncaseroot\nBook methe \ufb02ight through Houston\n(b) Systemxcomp\nnsubj\ndetnmod\ncaseroot\nFigure 19.15 Reference and system parses for Book me the \ufb02ight through Houston , resulting in an LAS of\n2/3 and an UAS of 5/6.\na development corpus. Here we can make use of the notions of precision and recall\nintroduced in Chapter 17, measuring the percentage of relations labeled NSUBJ by\nthe system that were correct (precision), and the percentage of the NSUBJ relations\npresent in the development set that were in fact discovered by the system (recall).\nWe can employ a confusion matrix to keep track of how often each dependency type\nwas confused for another.\n19.5 Summary\nThis chapter has introduced the concept of dependency grammars and dependency\nparsing. Here\u2019s a summary of the main points that we covered:\n\u2022 In dependency-based approaches to syntax, the structure of a sentence is de-\nscribed in terms of a set of binary relations that hold between the words in a\nsentence. Larger notions of constituency are not directly encoded in depen-\ndency analyses.\n\u2022 The relations in a dependency structure capture the head-dependent relation-\nship among the words in a sentence.\n\u2022 Dependency-based analysis provides information directly useful in further\nlanguage processing tasks including information extraction, semantic parsing\nand question answering.\n\u2022 Transition-based parsing systems employ a greedy stack-based algorithm to\ncreate dependency structures.\n\u2022 Graph-based methods for creating dependency structures are based on the use\nof maximum spanning tree methods from graph theory.\n\u2022 Both transition-based and graph-based approaches are developed using super-\nvised machine learning techniques.\n\u2022 Treebanks provide the data needed to train these systems. Dependency tree-\nbanks can be created directly by human annotators or via automatic transfor-\nmation from phrase-structure treebanks.\n\u2022 Evaluation of dependency parsers is based on labeled and unlabeled accuracy\nscores as measured against withheld development and test corpora. BIBLIOGRAPHICAL AND HISTORICAL NOTES 433\nBibliographical and Historical Notes\nThe dependency-based approach to grammar is much older than the relatively recent\nphrase-structure or constituency grammars, which date only to the 20th century. De-\npendency grammar dates back to the Indian grammarian P \u00afan.ini sometime between\nthe 7th and 4th centuries BCE, as well as the ancient Greek linguistic traditions.\nContemporary theories of dependency grammar all draw heavily on the 20th cen-\ntury work of Tesni `ere (1959).\nAutomatic parsing using dependency grammars was \ufb01rst introduced into compu-\ntational linguistics by early work on machine translation at the RAND Corporation\nled by David Hays. This work on dependency parsing closely paralleled work on\nconstituent parsing and made explicit use of grammars to guide the parsing process.\nAfter this early period, computational work on dependency parsing remained inter-\nmittent over the following decades. Notable implementations of dependency parsers\nfor English during this period include Link Grammar (Sleator and Temperley, 1993),\nConstraint Grammar (Karlsson et al., 1995), and MINIPAR (Lin, 2003).\nDependency parsing saw a major resurgence in the late 1990\u2019s with the appear-\nance of large dependency-based treebanks and the associated advent of data driven\napproaches described in this chapter. Eisner (1996) developed an ef\ufb01cient dynamic\nprogramming approach to dependency parsing based on bilexical grammars derived\nfrom the Penn Treebank. Covington (2001) introduced the deterministic word by\nword approach underlying current transition-based approaches. Yamada and Mat-\nsumoto (2003) and Kudo and Matsumoto (2002) introduced both the shift-reduce\nparadigm and the use of supervised machine learning in the form of support vector\nmachines to dependency parsing.\nTransition-based parsing is based on the shift-reduce parsing algorithm orig-\ninally developed for analyzing programming languages (Aho and Ullman, 1972).\nShift-reduce parsing also makes use of a context-free grammar. Input tokens are\nsuccessively shifted onto the stack and the top two elements of the stack are matched\nagainst the right-hand side of the rules in the grammar; when a match is found the\nmatched elements are replaced on the stack (reduced) by the non-terminal from the\nleft-hand side of the rule being matched. In transition-based dependency parsing\nwe skip the grammar, and alter the reduce operation to add a dependency relation\nbetween a word and its head.\nNivre (2003) de\ufb01ned the modern, deterministic, transition-based approach to\ndependency parsing. Subsequent work by Nivre and his colleagues formalized and\nanalyzed the performance of numerous transition systems, training methods, and\nmethods for dealing with non-projective language (Nivre and Scholz 2004, Nivre\n2006, Nivre and Nilsson 2005, Nivre et al. 2007b, Nivre 2007). The neural ap-\nproach was pioneered by Chen and Manning (2014) and extended by Kiperwasser\nand Goldberg (2016); Kulmizev et al. (2019).\nThe graph-based maximum spanning tree approach to dependency parsing was\nintroduced by McDonald et al. 2005a, McDonald et al. 2005b. The neural classi\ufb01er\nwas introduced by (Kiperwasser and Goldberg, 2016).\nThe long-running Prague Dependency Treebank project (Haji \u02c7c, 1998) is the most\nsigni\ufb01cant effort to directly annotate a corpus with multiple layers of morphological,\nsyntactic and semantic information. PDT 3.0 contains over 1.5 M tokens (Bej \u02c7cek\net al., 2013).\nUniversal Dependencies (UD) (de Marneffe et al., 2021) is an open community 434 CHAPTER 19 \u2022 D EPENDENCY PARSING\nproject to create a framework for dependency treebank annotation, with nearly 200\ntreebanks in over 100 languages. The UD annotation scheme evolved out of several\ndistinct efforts including Stanford dependencies (de Marneffe et al. 2006, de Marn-\neffe and Manning 2008, de Marneffe et al. 2014), Google\u2019s universal part-of-speech\ntags (Petrov et al., 2012), and the Interset interlingua for morphosyntactic tagsets\n(Zeman, 2008).\nThe Conference on Natural Language Learning (CoNLL) has conducted an in-\n\ufb02uential series of shared tasks related to dependency parsing over the years (Buch-\nholz and Marsi 2006, Nivre et al. 2007a, Surdeanu et al. 2008, Haji \u02c7c et al. 2009).\nMore recent evaluations have focused on parser robustness with respect to morpho-\nlogically rich languages (Seddah et al., 2013), and non-canonical language forms\nsuch as social media, texts, and spoken language (Petrov and McDonald, 2012).\nChoi et al. (2015) presents a performance analysis of 10 dependency parsers across\na range of metrics, as well as DEPEND ABLE, a robust parser evaluation tool.\nExercises CHAPTER\n20Information Extraction:\nRelations, Events, and Time\nTime will explain.\nJane Austen, Persuasion\nImagine that you are an analyst with an investment \ufb01rm that tracks airline stocks.\nYou\u2019re given the task of determining the relationship (if any) between airline an-\nnouncements of fare increases and the behavior of their stocks the next day. His-\ntorical data about stock prices is easy to come by, but what about the airline an-\nnouncements? You will need to know at least the name of the airline, the nature of\nthe proposed fare hike, the dates of the announcement, and possibly the response of\nother airlines. Fortunately, these can be all found in news articles like this one:\nCiting high fuel prices, United Airlines said Friday it has increased fares\nby $6 per round trip on \ufb02ights to some cities also served by lower-\ncost carriers. American Airlines, a unit of AMR Corp., immediately\nmatched the move, spokesman Tim Wagner said. United, a unit of UAL\nCorp., said the increase took effect Thursday and applies to most routes\nwhere it competes against discount carriers, such as Chicago to Dallas\nand Denver to San Francisco.\nThis chapter presents techniques for extracting limited kinds of semantic con-\ntent from text. This process of information extraction (IE) turns the unstructuredinformation\nextraction\ninformation embedded in texts into structured data, for example for populating a\nrelational database to enable further processing.\nWe begin with the task of relation extraction : \ufb01nding and classifying semanticrelation\nextraction\nrelations among entities mentioned in a text, like child-of (X is the child-of Y), or\npart-whole or geospatial relations. Relation extraction has close links to populat-\ning a relational database, and knowledge graphs , datasets of structured relationalknowledge\ngraphs\nknowledge, are a useful way for search engines to present information to users.\nNext, we discuss event extraction , the task of \ufb01nding events in which these en-event\nextraction\ntities participate, like, in our sample text, the fare increases by United andAmerican\nand the reporting events said andcite. Events are also situated in time , occurring at\na particular date or time, and events can be related temporally, happening before or\nafter or simultaneously with each other. We\u2019ll need to recognize temporal expres-\nsions like Friday ,Thursday ortwo days from now and times such as 3:30 P .M. , and\nnormalize them onto speci\ufb01c calendar dates or times. We\u2019ll need to link Friday to\nthe time of United\u2019s announcement, Thursday to the previous day\u2019s fare increase,\nand we\u2019ll need to produce a timeline in which United\u2019s announcement follows the\nfare increase and American\u2019s announcement follows both of those events.\nThe related task of template \ufb01lling is to \ufb01nd recurring stereotypical events or template \ufb01lling\nsituations in documents and \ufb01ll in the template slots. These slot-\ufb01llers may consist\nof text segments extracted directly from the text, or concepts like times, amounts, or\nontology entities that have been inferred through additional processing. Our airline 436 CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\nARTIFACTGENERALAFFILIATIONORGAFFILIATIONPART-WHOLEPERSON-SOCIALPHYSICALLocatedNearBusinessFamilyLasting PersonalCitizen-Resident-Ethnicity-ReligionOrg-Location-OriginFounderEmploymentMembershipOwnershipStudent-AlumInvestorUser-Owner-Inventor-ManufacturerGeographicalSubsidiary\nSports-Affiliation\nFigure 20.1 The 17 relations used in the ACE relation extraction task.\ntext presents such a stereotypical situation since airlines often raise fares and then\nwait to see if competitors follow along. Here we can identify United as a lead air-\nline that initially raised its fares, $6 as the amount, Thursday as the increase date,\nandAmerican as an airline that followed along, leading to a \ufb01lled template like the\nfollowing:\nFARE-RAISE ATTEMPT :2\n6664LEAD AIRLINE : U NITED AIRLINES\nAMOUNT : $6\nEFFECTIVE DATE: 2006-10-26\nFOLLOWER : A MERICAN AIRLINES3\n7775\n20.1 Relation Extraction\nLet\u2019s assume that we have detected the named entities in our sample text (perhaps\nusing the techniques of Chapter 17), and would like to discern the relationships that\nexist among the detected entities:\nCiting high fuel prices, [ ORG United Airlines ] said [ TIME Friday ] it\nhas increased fares by [ MONEY $6] per round trip on \ufb02ights to some\ncities also served by lower-cost carriers. [ ORG American Airlines ], a\nunit of [ ORG AMR Corp.] , immediately matched the move, spokesman\n[PER Tim Wagner ] said. [ ORG United] , a unit of [ ORG UAL Corp.] ,\nsaid the increase took effect [ TIME Thursday] and applies to most\nroutes where it competes against discount carriers, such as [ LOC Chicago]\nto [LOC Dallas] and [ LOC Denver] to [LOC San Francisco] .\nThe text tells us, for example, that Tim Wagner is a spokesman for American\nAirlines , that United is a unit of UAL Corp. , and that American is a unit of AMR .\nThese binary relations are instances of more generic relations such as part-of or\nemploys that are fairly frequent in news-style texts. Figure 20.1 lists the 17 relations\nused in the ACE relation extraction evaluations and Fig. 20.2 shows some sample\nrelations. We might also extract more domain-speci\ufb01c relation such as the notion of\nan airline route. For example from this text we can conclude that United has routes\nto Chicago, Dallas, Denver, and San Francisco. 20.1 \u2022 R ELATION EXTRACTION 437\nRelations Types Examples\nPhysical-Located PER-GPE Hewas in Tennessee\nPart-Whole-Subsidiary ORG-ORG XYZ , the parent company of ABC\nPerson-Social-Family PER-PER Yoko \u2019s husband John\nOrg-AFF-Founder PER-ORG Steve Jobs , co-founder of Apple ...\nFigure 20.2 Semantic relations with examples and the named entity types they involve.\nSets of relations have been de\ufb01ned for many other domains as well. For example\nUMLS, the Uni\ufb01ed Medical Language System from the US National Library of\nMedicine has a network that de\ufb01nes 134 broad subject categories, entity types, and\n54 relations between the entities, such as the following:\nEntity Relation Entity\nInjury disrupts Physiological Function\nBodily Location location-of Biologic Function\nAnatomical Structure part-of Organism\nPharmacologic Substance causes Pathological Function\nPharmacologic Substance treats Pathologic Function\nGiven a medical sentence like this one:\n(20.1) Doppler echocardiography can be used to diagnose left anterior descending\nartery stenosis in patients with type 2 diabetes\nWe could thus extract the UMLS relation:\nEchocardiography, Doppler Diagnoses Acquired stenosis\nWikipedia also offers a large supply of relations, drawn from infoboxes , struc- infoboxes\ntured tables associated with certain Wikipedia articles. For example, the Wikipedia\ninfobox for Stanford includes structured facts like state = \"California\" or\npresident = \"Marc Tessier-Lavigne\" . These facts can be turned into rela-\ntions like president-of orlocated-in . or into relations in a metalanguage called RDF RDF\n(Resource Description Framework). An RDF triple is a tuple of entity-relation- RDF triple\nentity, called a subject-predicate-object expression. Here\u2019s a sample RDF triple:\nsubject predicate object\nGolden Gate Park location San Francisco\nFor example the crowdsourced DBpedia (Bizer et al., 2009) is an ontology de-\nrived from Wikipedia containing over 2 billion RDF triples. Another dataset from\nWikipedia infoboxes, Freebase (Bollacker et al., 2008), now part of Wikidata (Vrande \u02c7ci\u00b4c Freebase\nand Kr \u00a8otzsch, 2014), has relations between people and their nationality, or locations,\nand other locations they are contained in.\nWordNet or other ontologies offer useful ontological relations that express hier-\narchical relations between words or concepts. For example WordNet has the is-aor is-a\nhypernym relation between classes, hypernym\nGiraffe is-a ruminant is-a ungulate is-a mammal is-a vertebrate ...\nWordNet also has Instance-of relation between individuals and classes, so that for\nexample San Francisco is in the Instance-of relation with city. Extracting these\nrelations is an important step in extending or building ontologies.\nFinally, there are large datasets that contain sentences hand-labeled with their\nrelations, designed for training and testing relation extractors. The TACRED dataset\n(Zhang et al., 2017) contains 106,264 examples of relation triples about particular\npeople or organizations, labeled in sentences from news and web text drawn from the 438 CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\nannual TAC Knowledge Base Population (TAC KBP) challenges. TACRED contains\n41 relation types (like per:city of birth, org:subsidiaries, org:member of, per:spouse),\nplus a no relation tag; examples are shown in Fig. 20.3. About 80% of all examples\nare annotated as no relation; having suf\ufb01cient negative data is important for training\nsupervised classi\ufb01ers.\nExample Entity Types & Label\nCarey will succeed Cathleen P. Black, who held the position for 15\nyears and will take on a new role as chairwoman of Hearst Maga-\nzines, the company said.PERSON /TITLE\nRelation: per:title\nIrene Morgan Kirkaldy, who was born and reared in Baltimore, lived\non Long Island and ran a child-care center in Queens with her second\nhusband, Stanley Kirkaldy.PERSON /CITY\nRelation: per:city ofbirth\nBaldwin declined further comment, and said JetBlue chief executive\nDave Barger was unavailable.Types: PERSON /TITLE\nRelation: norelation\nFigure 20.3 Example sentences and labels from the TACRED dataset (Zhang et al., 2017).\nA standard dataset was also produced for the SemEval 2010 Task 8, detecting\nrelations between nominals (Hendrickx et al., 2009). The dataset has 10,717 exam-\nples, each with a pair of nominals (untyped) hand-labeled with one of 9 directed\nrelations like product-producer ( afactory manufactures suits) orcomponent-whole\n(myapartment has a large kitchen ).\n20.2 Relation Extraction Algorithms\nThere are \ufb01ve main classes of algorithms for relation extraction: handwritten pat-\nterns ,supervised machine learning ,semi-supervised (viabootstrapping ordis-\ntant supervision ), and unsupervised . We\u2019ll introduce each of these in the next\nsections.\n20.2.1 Using Patterns to Extract Relations\nThe earliest and still common algorithm for relation extraction is lexico-syntactic\npatterns, \ufb01rst developed by Hearst (1992a), and therefore often called Hearst pat-\nterns . Consider the following sentence: Hearst patterns\nAgar is a substance prepared from a mixture of red algae, such as Ge-\nlidium, for laboratory or industrial use.\nHearst points out that most human readers will not know what Gelidium is, but that\nthey can readily infer that it is a kind of (a hyponym of)red algae , whatever that is.\nShe suggests that the following lexico-syntactic pattern\nNP0such as NP 1f;NP2:::;(andjor)NPig;i\u00151 (20.2)\nimplies the following semantics\n8NPi;i\u00151;hyponym (NPi;NP0) (20.3)\nallowing us to infer\nhyponym (Gelidium;red algae ) (20.4) 20.2 \u2022 R ELATION EXTRACTION ALGORITHMS 439\nNPf, NPg*f,g(andjor) other NP H temples, treasuries, and other important civic buildings\nNPHsuch asfNP,g*f(orjand)gNP red algae such as Gelidium\nsuch NP HasfNP,g*f(orjand)gNP such authors as Herrick, Goldsmith, and Shakespeare\nNPHf,gincludingfNP,g*f(orjand)gNP common-law countries, including Canada and England\nNPHf,gespeciallyfNPg*f(orjand)gNP European countries, especially France, England, and Spain\nFigure 20.4 Hand-built lexico-syntactic patterns for \ufb01nding hypernyms, using fgto mark optionality (Hearst\n1992a, Hearst 1998).\nFigure 20.4 shows \ufb01ve patterns Hearst (1992a, 1998) suggested for inferring\nthe hyponym relation; we\u2019ve shown NPHas the parent/hyponym. Modern versions\nof the pattern-based approach extend it by adding named entity constraints. For\nexample if our goal is to answer questions about \u201cWho holds what of\ufb01ce in which\norganization?\u201d, we can use patterns like the following:\nPER, POSITION of ORG:\nGeorge Marshall, Secretary of State of the United States\nPER (namedjappointedjchosejetc.) PER Prep? POSITION\nTruman appointed Marshall Secretary of State\nPER [be]? (namedjappointedjetc.) Prep? ORG POSITION\nGeorge Marshall was named US Secretary of State\nHand-built patterns have the advantage of high-precision and they can be tailored\nto speci\ufb01c domains. On the other hand, they are often low-recall, and it\u2019s a lot of\nwork to create them for all possible patterns.\n20.2.2 Relation Extraction via Supervised Learning\nSupervised machine learning approaches to relation extraction follow a scheme that\nshould be familiar by now. A \ufb01xed set of relations and entities is chosen, a training\ncorpus is hand-annotated with the relations and entities, and the annotated texts are\nthen used to train classi\ufb01ers to annotate an unseen test set.\nThe most straightforward approach, illustrated in Fig. 20.5 is: (1) Find pairs of\nnamed entities (usually in the same sentence). (2): Apply a relation-classi\ufb01cation\non each pair. The classi\ufb01er can use any supervised technique (logistic regression,\nRNN, Transformer, random forest, etc.).\nAn optional intermediate \ufb01ltering classi\ufb01er can be used to speed up the process-\ning by making a binary decision on whether a given pair of named entities are related\n(by any relation). It\u2019s trained on positive examples extracted directly from all rela-\ntions in the annotated corpus, and negative examples generated from within-sentence\nentity pairs that are not annotated with a relation.\nFeature-based supervised relation classi\ufb01ers. Let\u2019s consider sample features for\na feature-based classi\ufb01er (like logistic regression or random forests), classifying the\nrelationship between American Airlines (Mention 1, or M1) and Tim Wagner (Men-\ntion 2, M2) from this sentence:\n(20.5) American Airlines , a unit of AMR, immediately matched the move,\nspokesman Tim Wagner said\nThese include word features (as embeddings, or 1-hot, stemmed or not):\n\u2022 The headwords of M1 and M2 and their concatenation\nAirlines Wagner Airlines-Wagner 440 CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\nfunction FINDRELATIONS (words )returns relations\nrelations nil\nentities FINDENTITIES (words )\nforall entity pairs he1,e2iinentities do\nifRELATED ?(e1,e2)\nrelations relations +CLASSIFY RELATION (e1,e2)\nFigure 20.5 Finding and classifying the relations among entities in a text.\n\u2022 Bag-of-words and bigrams in M1 and M2\nAmerican, Airlines, Tim, Wagner, American Airlines, Tim Wagner\n\u2022 Words or bigrams in particular positions\nM2: -1 spokesman\nM2: +1 said\n\u2022 Bag of words or bigrams between M1 and M2:\na, AMR, of, immediately, matched, move, spokesman, the, unit\nNamed entity features:\n\u2022 Named-entity types and their concatenation\n(M1: ORG, M2: PER, M1M2: ORG-PER)\n\u2022 Entity Level of M1 and M2 (from the set NAME, NOMINAL, PRONOUN)\nM1: NAME [it or he would be PRONOUN]\nM2: NAME [the company would be NOMINAL]\n\u2022 Number of entities between the arguments (in this case 1, for AMR)\nSyntactic structure is a useful signal, often represented as the dependency or\nconstituency syntactic path traversed through the tree between the entities.\n\u2022 Constituent paths between M1 and M2\nNP\"NP\"S\"S#NP\n\u2022 Dependency-tree paths\nAirlines sub jmatched comp said!sub jWagner\nNeural supervised relation classi\ufb01ers Neural models for relation extraction sim-\nilarly treat the task as supervised classi\ufb01cation. Let\u2019s consider a typical system ap-\nplied to the TACRED relation extraction dataset and task (Zhang et al., 2017). In\nTACRED we are given a sentence and two spans within it: a subject, which is a\nperson or organization, and an object, which is any other entity. The task is to assign\na relation from the 42 TAC relations, or no relation.\nA typical Transformer-encoder algorithm, shown in Fig. 20.6, simply takes a\npretrained encoder like BERT and adds a linear layer on top of the sentence repre-\nsentation (for example the BERT [CLS] token), a linear layer that is \ufb01netuned as a\n1-of-N classi\ufb01er to assign one of the 43 labels. The input to the BERT encoder is\npartially de-lexi\ufb01ed; the subject and object entities are replaced in the input by their\nNER tags. This helps keep the system from over\ufb01tting to the individual lexical items\n(Zhang et al., 2017). When using BERT-type Transformers for relation extraction, it\nhelps to use versions of BERT like RoBERTa (Liu et al., 2019) or spanBERT (Joshi\net al., 2020) that don\u2019t have two sequences separated by a [SEP] token, but instead\nform the input from a single long sequence of sentences.\nIn general, if the test set is similar enough to the training set, and if there is\nenough hand-labeled data, supervised relation extraction systems can get high ac- 20.2 \u2022 R ELATION EXTRACTION ALGORITHMS 441\nENCODER[CLS][SUBJ_PERSON]wasbornin[OBJ_LOC],MichiganLinearClassi\ufb01erp(relation|SUBJ,OBJ)\nFigure 20.6 Relation extraction as a linear layer on top of an encoder (in this case BERT),\nwith the subject and object entities replaced in the input by their NER tags (Zhang et al. 2017,\nJoshi et al. 2020).\ncuracies. But labeling a large training set is extremely expensive and supervised\nmodels are brittle: they don\u2019t generalize well to different text genres. For this rea-\nson, much research in relation extraction has focused on the semi-supervised and\nunsupervised approaches we turn to next.\n20.2.3 Semisupervised Relation Extraction via Bootstrapping\nSupervised machine learning assumes that we have lots of labeled data. Unfortu-\nnately, this is expensive. But suppose we just have a few high-precision seed pat-\nterns , like those in Section 20.2.1, or perhaps a few seed tuples . That\u2019s enough seed patterns\nseed tuples to bootstrap a classi\ufb01er! Bootstrapping proceeds by taking the entities in the seed\nbootstrapping pair, and then \ufb01nding sentences (on the web, or whatever dataset we are using) that\ncontain both entities. From all such sentences, we extract and generalize the context\naround the entities to learn new patterns. Fig. 20.7 sketches a basic algorithm.\nfunction BOOTSTRAP (Relation R )returns new relation tuples\ntuples Gather a set of seed tuples that have relation R\niterate\nsentences \ufb01nd sentences that contain entities in tuples\npatterns generalize the context between and around entities in sentences\nnewpairs usepatterns to identify more tuples\nnewpairs newpairs with high con\ufb01dence\ntuples tuples +newpairs\nreturn tuples\nFigure 20.7 Bootstrapping from seed entity pairs to learn relations.\nSuppose, for example, that we need to create a list of airline/hub pairs, and we\nknow only that Ryanair has a hub at Charleroi. We can use this seed fact to discover\nnew patterns by \ufb01nding other mentions of this relation in our corpus. We search\nfor the terms Ryanair ,Charleroi andhubin some proximity. Perhaps we \ufb01nd the\nfollowing set of sentences:\n(20.6) Budget airline Ryanair, which uses Charleroi as a hub, scrapped all\nweekend \ufb02ights out of the airport.\n(20.7) All \ufb02ights in and out of Ryanair\u2019s hub at Charleroi airport were grounded on\nFriday...\n(20.8) A spokesman at Charleroi, a main hub for Ryanair, estimated that 8000\npassengers had already been affected. 442 CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\nFrom these results, we can use the context of words between the entity mentions,\nthe words before mention one, the word after mention two, and the named entity\ntypes of the two mentions, and perhaps other features, to extract general patterns\nsuch as the following:\n/ [ORG], which uses [LOC] as a hub /\n/ [ORG]'s hub at [LOC] /\n/ [LOC], a main hub for [ORG] /\nThese new patterns can then be used to search for additional tuples.\nBootstrapping systems also assign con\ufb01dence values to new tuples to avoid se-con\ufb01dence\nvalues\nmantic drift . In semantic drift, an erroneous pattern leads to the introduction of semantic drift\nerroneous tuples, which, in turn, lead to the creation of problematic patterns and the\nmeaning of the extracted relations \u2018drifts\u2019. Consider the following example:\n(20.9) Sydney has a ferry hub at Circular Quay.\nIf accepted as a positive example, this expression could lead to the incorrect in-\ntroduction of the tuple hSydney;CircularQuayi. Patterns based on this tuple could\npropagate further errors into the database.\nCon\ufb01dence values for patterns are based on balancing two factors: the pattern\u2019s\nperformance with respect to the current set of tuples and the pattern\u2019s productivity\nin terms of the number of matches it produces in the document collection. More\nformally, given a document collection D, a current set of tuples T, and a proposed\npattern p, we need to track two factors:\n\u2022hits(p): the set of tuples in Tthatpmatches while looking in D\n\u2022\ufb01nds (p): The total set of tuples that p\ufb01nds in D\nThe following equation balances these considerations (Riloff and Jones, 1999).\nConfRlogF(p) =jhits(p)j\nj\ufb01nds(p)jlog(j\ufb01nds(p)j) (20.10)\nThis metric is generally normalized to produce a probability.\nWe can assess the con\ufb01dence in a proposed new tuple by combining the evidence\nsupporting it from all the patterns P0that match that tuple in D(Agichtein and Gra-\nvano, 2000). One way to combine such evidence is the noisy-or technique. Assume noisy-or\nthat a given tuple is supported by a subset of the patterns in P, each with its own\ncon\ufb01dence assessed as above. In the noisy-or model, we make two basic assump-\ntions. First, that for a proposed tuple to be false, allof its supporting patterns must\nhave been in error, and second, that the sources of their individual failures are all\nindependent. If we loosely treat our con\ufb01dence measures as probabilities, then the\nprobability of any individual pattern pfailing is 1\u0000Conf(p); the probability of all\nof the supporting patterns for a tuple being wrong is the product of their individual\nfailure probabilities, leaving us with the following equation for our con\ufb01dence in a\nnew tuple.\nConf(t) =1\u0000Y\np2P0(1\u0000Conf(p)) (20.11)\nSetting conservative con\ufb01dence thresholds for the acceptance of new patterns\nand tuples during the bootstrapping process helps prevent the system from drifting\naway from the targeted relation. 20.2 \u2022 R ELATION EXTRACTION ALGORITHMS 443\n20.2.4 Distant Supervision for Relation Extraction\nAlthough hand-labeling text with relation labels is expensive to produce, there are\nways to \ufb01nd indirect sources of training data. The distant supervision methoddistant\nsupervision\n(Mintz et al., 2009) combines the advantages of bootstrapping with supervised learn-\ning. Instead of just a handful of seeds, distant supervision uses a large database to\nacquire a huge number of seed examples, creates lots of noisy pattern features from\nall these examples and then combines them in a supervised classi\ufb01er.\nFor example suppose we are trying to learn the place-of-birth relationship be-\ntween people and their birth cities. In the seed-based approach, we might have only\n5 examples to start with. But Wikipedia-based databases like DBPedia or Freebase\nhave tens of thousands of examples of many relations; including over 100,000 ex-\namples of place-of-birth , (<Edwin Hubble, Marshfield> ,<Albert Einstein,\nUlm> , etc.,). The next step is to run named entity taggers on large amounts of text\u2014\nMintz et al. (2009) used 800,000 articles from Wikipedia\u2014and extract all sentences\nthat have two named entities that match the tuple, like the following:\n...Hubble was born in Marsh\ufb01eld...\n...Einstein, born (1879), Ulm...\n...Hubble\u2019s birthplace in Marsh\ufb01eld...\nTraining instances can now be extracted from this data, one training instance\nfor each identical tuple <relation, entity1, entity2> . Thus there will be one\ntraining instance for each of:\n<born-in, Edwin Hubble, Marshfield>\n<born-in, Albert Einstein, Ulm>\n<born-year, Albert Einstein, 1879>\nand so on.\nWe can then apply feature-based or neural classi\ufb01cation. For feature-based\nclassi\ufb01cation, we can use standard supervised relation extraction features like the\nnamed entity labels of the two mentions, the words and dependency paths in be-\ntween the mentions, and neighboring words. Each tuple will have features col-\nlected from many training instances; the feature vector for a single training instance\nlike (<born-in,Albert Einstein, Ulm> will have lexical and syntactic features\nfrom many different sentences that mention Einstein and Ulm.\nBecause distant supervision has very large training sets, it is also able to use very\nrich features that are conjunctions of these individual features. So we will extract\nthousands of patterns that conjoin the entity types with the intervening words or\ndependency paths like these:\nPER was born in LOC\nPER, born (XXXX), LOC\nPER\u2019s birthplace in LOC\nTo return to our running example, for this sentence:\n(20.12) American Airlines , a unit of AMR, immediately matched the move,\nspokesman Tim Wagner said\nwe would learn rich conjunction features like this one:\nM1 = ORG & M2 = PER & nextword=\u201csaid\u201d& path= NP\"NP\"S\"S#NP\nThe result is a supervised classi\ufb01er that has a huge rich set of features to use\nin detecting relations. Since not every test sentence will have one of the training 444 CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\nrelations, the classi\ufb01er will also need to be able to label an example as no-relation .\nThis label is trained by randomly selecting entity pairs that do not appear in any\nFreebase relation, extracting features for them, and building a feature vector for\neach such tuple. The \ufb01nal algorithm is sketched in Fig. 20.8.\nfunction DISTANT SUPERVISION (Database D, Text T )returns relation classi\ufb01er C\nforeach relation R\nforeach tuple ( e1,e2 ) of entities with relation RinD\nsentences Sentences in Tthat contain e1ande2\nf Frequent features in sentences\nobservations observations + new training tuple ( e1, e2, f, R )\nC Train supervised classi\ufb01er on observations\nreturn C\nFigure 20.8 The distant supervision algorithm for relation extraction. A neural classi\ufb01er\nwould skip the feature set f.\nDistant supervision shares advantages with each of the methods we\u2019ve exam-\nined. Like supervised classi\ufb01cation, distant supervision uses a classi\ufb01er with lots\nof features, and supervised by detailed hand-created knowledge. Like pattern-based\nclassi\ufb01ers, it can make use of high-precision evidence for the relation between en-\ntities. Indeed, distance supervision systems learn patterns just like the hand-built\npatterns of early relation extractors. For example the is-aorhypernym extraction\nsystem of Snow et al. (2005) used hypernym/hyponym NP pairs from WordNet as\ndistant supervision, and then learned new patterns from large amounts of text. Their\nsystem induced exactly the original 5 template patterns of Hearst (1992a), but also\n70,000 additional patterns including these four:\nNPHlike NP Many hormones like leptin...\nNPHcalled NP ...using a markup language called XHTML\nNP is a NP H Ruby is a programming language...\nNP, a NP H IBM, a company with a long...\nThis ability to use a large number of features simultaneously means that, un-\nlike the iterative expansion of patterns in seed-based systems, there\u2019s no semantic\ndrift. Like unsupervised classi\ufb01cation, it doesn\u2019t use a labeled training corpus of\ntexts, so it isn\u2019t sensitive to genre issues in the training corpus, and relies on very\nlarge amounts of unlabeled data. Distant supervision also has the advantage that it\ncan create training tuples to be used with neural classi\ufb01ers, where features are not\nrequired.\nThe main problem with distant supervision is that it tends to produce low-precision\nresults, and so current research focuses on ways to improve precision. Furthermore,\ndistant supervision can only help in extracting relations for which a large enough\ndatabase already exists. To extract new relations without datasets, or relations for\nnew domains, purely unsupervised methods must be used.\n20.2.5 Unsupervised Relation Extraction\nThe goal of unsupervised relation extraction is to extract relations from the web\nwhen we have no labeled training data, and not even any list of relations. This task\nis often called open information extraction orOpen IE . In Open IE, the relationsopen\ninformation\nextraction 20.2 \u2022 R ELATION EXTRACTION ALGORITHMS 445\nare simply strings of words (usually beginning with a verb).\nFor example, the ReVerb system (Fader et al., 2011) extracts a relation from a\nsentence sin 4 steps:\n1. Run a part-of-speech tagger and entity chunker over s\n2. For each verb in s, \ufb01nd the longest sequence of words wthat start with a verb\nand satisfy syntactic and lexical constraints, merging adjacent matches.\n3. For each phrase w, \ufb01nd the nearest noun phrase xto the left which is not a\nrelative pronoun, wh-word or existential \u201cthere\u201d. Find the nearest noun phrase\nyto the right.\n4. Assign con\ufb01dence cto the relation r= (x;w;y)using a con\ufb01dence classi\ufb01er\nand return it.\nA relation is only accepted if it meets syntactic and lexical constraints. The\nsyntactic constraints ensure that it is a verb-initial sequence that might also include\nnouns (relations that begin with light verbs like make ,have , ordooften express the\ncore of the relation with a noun, like have a hub in ):\nVjVPjVW*P\nV = verb particle? adv?\nW = (nounjadjjadvjpronjdet )\nP = (prepjparticlejin\ufb01nitive \u201cto\u201d)\nThe lexical constraints are based on a dictionary Dthat is used to prune very rare,\nlong relation strings. The intuition is to eliminate candidate relations that don\u2019t oc-\ncur with suf\ufb01cient number of distinct argument types and so are likely to be bad\nexamples. The system \ufb01rst runs the above relation extraction algorithm of\ufb02ine on\n500 million web sentences and extracts a list of all the relations that occur after nor-\nmalizing them (removing in\ufb02ection, auxiliary verbs, adjectives, and adverbs). Each\nrelation ris added to the dictionary if it occurs with at least 20 different arguments.\nFader et al. (2011) used a dictionary of 1.7 million normalized relations.\nFinally, a con\ufb01dence value is computed for each relation using a logistic re-\ngression classi\ufb01er. The classi\ufb01er is trained by taking 1000 random web sentences,\nrunning the extractor, and hand labeling each extracted relation as correct or incor-\nrect. A con\ufb01dence classi\ufb01er is then trained on this hand-labeled data, using features\nof the relation and the surrounding words. Fig. 20.9 shows some sample features\nused in the classi\ufb01cation.\n(x,r,y) covers all words in s\nthe last preposition in risfor\nthe last preposition in rison\nlen(s)\u001410\nthere is a coordinating conjunction to the left of rins\nrmatches a lone V in the syntactic constraints\nthere is preposition to the left of xins\nthere is an NP to the right of yins\nFigure 20.9 Features for the classi\ufb01er that assigns con\ufb01dence to relations extracted by the\nOpen Information Extraction system REVERB (Fader et al., 2011).\nFor example the following sentence:\n(20.13) United has a hub in Chicago, which is the headquarters of United\nContinental Holdings. 446 CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\nhas the relation phrases has a hub in andis the headquarters of (it also has hasand\nis, but longer phrases are preferred). Step 3 \ufb01nds United to the left and Chicago to\nthe right of has a hub in , and skips over which to \ufb01nd Chicago to the left of is the\nheadquarters of . The \ufb01nal output is:\nr1: <United, has a hub in, Chicago>\nr2: <Chicago, is the headquarters of, United Continental Holdings>\nThe great advantage of unsupervised relation extraction is its ability to handle\na huge number of relations without having to specify them in advance. The dis-\nadvantage is the need to map all the strings into some canonical form for adding\nto databases or knowledge graphs. Current methods focus heavily on relations ex-\npressed with verbs, and so will miss many relations that are expressed nominally.\n20.2.6 Evaluation of Relation Extraction\nSupervised relation extraction systems are evaluated by using test sets with human-\nannotated, gold-standard relations and computing precision, recall, and F-measure.\nLabeled precision and recall require the system to classify the relation correctly,\nwhereas unlabeled methods simply measure a system\u2019s ability to detect entities that\nare related.\nSemi-supervised andunsupervised methods are much more dif\ufb01cult to evalu-\nate, since they extract totally new relations from the web or a large text. Because\nthese methods use very large amounts of text, it is generally not possible to run them\nsolely on a small labeled test set, and as a result it\u2019s not possible to pre-annotate a\ngold set of correct instances of relations.\nFor these methods it\u2019s possible to approximate (only) precision by drawing a\nrandom sample of relations from the output, and having a human check the accuracy\nof each of these relations. Usually this approach focuses on the tuples to be extracted\nfrom a body of text rather than on the relation mentions ; systems need not detect\nevery mention of a relation to be scored correctly. Instead, the evaluation is based\non the set of tuples occupying the database when the system is \ufb01nished. That is,\nwe want to know if the system can discover that Ryanair has a hub at Charleroi; we\ndon\u2019t really care how many times it discovers it. The estimated precision \u02c6Pis then\n\u02c6P=# of correctly extracted relation tuples in the sample\ntotal # of extracted relation tuples in the sample.(20.14)\nAnother approach that gives us a little bit of information about recall is to com-\npute precision at different levels of recall. Assuming that our system is able to\nrank the relations it produces (by probability, or con\ufb01dence) we can separately com-\npute precision for the top 1000 new relations, the top 10,000 new relations, the top\n100,000, and so on. In each case we take a random sample of that set. This will\nshow us how the precision curve behaves as we extract more and more tuples. But\nthere is no way to directly evaluate recall.\n20.3 Extracting Events\nThe task of event extraction is to identify mentions of events in texts. For theevent\nextraction\npurposes of this task, an event mention is any expression denoting an event or state\nthat can be assigned to a particular point, or interval, in time. The following markup\nof the sample text on page 435 shows all the events in this text. 20.4 \u2022 R EPRESENTING TIME 447\n[EVENT Citing] high fuel prices, United Airlines [ EVENT said] Fri-\nday it has [ EVENT increased] fares by $6 per round trip on \ufb02ights to\nsome cities also served by lower-cost carriers. American Airlines, a unit\nof AMR Corp., immediately [ EVENT matched] [ EVENT the move],\nspokesman Tim Wagner [ EVENT said]. United, a unit of UAL Corp.,\n[EVENT said] [ EVENT the increase] took effect Thursday and [ EVENT\napplies] to most routes where it [ EVENT competes] against discount\ncarriers, such as Chicago to Dallas and Denver to San Francisco.\nIn English, most event mentions correspond to verbs, and most verbs introduce\nevents. However, as we can see from our example, this is not always the case. Events\ncan be introduced by noun phrases, as in the move andthe increase , and some verbs\nfail to introduce events, as in the phrasal verb took effect , which refers to when the\nevent began rather than to the event itself. Similarly, light verbs such as make ,take, light verbs\nandhave often fail to denote events. A light verb is a verb that has very little meaning\nitself, and the associated event is instead expressed by its direct object noun. In light\nverb examples like took a \ufb02ight , it\u2019s the word \ufb02ight that de\ufb01nes the event; these light\nverbs just provide a syntactic structure for the noun\u2019s arguments.\nVarious versions of the event extraction task exist, depending on the goal. For\nexample in the TempEval shared tasks (Verhagen et al. 2009) the goal is to extract\nevents and aspects like their aspectual and temporal properties. Events are to be\nclassi\ufb01ed as actions, states, reporting events (say, report, tell, explain ), perceptionreporting\nevents\nevents, and so on. The aspect, tense, and modality of each event also needs to be\nextracted. Thus for example the various said events in the sample text would be\nannotated as (class=REPORTING, tense=PAST, aspect=PERFECTIVE).\nEvent extraction is generally modeled via supervised learning, detecting events\nvia IOB sequence models and assigning event classes and attributes with multi-class\nclassi\ufb01ers. The input can be neural models starting from encoders; or classic feature-\nbased models using features like those in Fig. 20.10.\nFeature Explanation\nCharacter af\ufb01xes Character-level pre\ufb01xes and suf\ufb01xes of target word\nNominalization suf\ufb01x Character-level suf\ufb01xes for nominalizations (e.g., -tion )\nPart of speech Part of speech of the target word\nLight verb Binary feature indicating that the target is governed by a light verb\nSubject syntactic category Syntactic category of the subject of the sentence\nMorphological stem Stemmed version of the target word\nVerb root Root form of the verb basis for a nominalization\nWordNet hypernyms Hypernym set for the target\nFigure 20.10 Features commonly used in classic feature-based approaches to event detection.\n20.4 Representing Time\nLet\u2019s begin by introducing the basics of temporal logic and how human languages temporal logic\nconvey temporal information. The most straightforward theory of time holds that it\n\ufb02ows inexorably forward and that events are associated with either points or inter-\nvals in time, as on a timeline. We can order distinct events by situating them on the\ntimeline; one event precedes another if the \ufb02ow of time leads from the \ufb01rst event 448 CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\nto the second. Accompanying these notions in most theories is the idea of the cur-\nrent moment in time. Combining this notion with the idea of a temporal ordering\nrelationship yields the familiar notions of past, present, and future.\nVarious kinds of temporal representation systems can be used to talk about tem-\nporal ordering relationship. One of the most commonly used in computational mod-\neling is the interval algebra of Allen (1984). Allen models all events and time interval algebra\nexpressions as intervals there is no representation for points (although intervals can\nbe very short). In order to deal with intervals without points, he identi\ufb01es 13 primi-\ntive relations that can hold between these temporal intervals. Fig. 20.11 shows these\n13Allen relations . Allen relations\nBABA\nBA\nAAB\nBAB\nTime A  before BB after  AA overlaps BB overlaps' AA meets BB meets' AA equals B(B equals A)A starts BB starts' AA finishes BB finishes' A\nBA during BB during' AA\nFigure 20.11 The 13 temporal relations from Allen (1984).\n20.4.1 Reichenbach\u2019s reference point\nThe relation between simple verb tenses and points in time is by no means straight-\nforward. The present tense can be used to refer to a future event, as in this example:\n(20.15) Ok, we \ufb02y from San Francisco to Boston at 10.\nOr consider the following examples:\n(20.16) Flight 1902 arrived late.\n(20.17) Flight 1902 had arrived late.\nAlthough both refer to events in the past, representing them in the same way seems\nwrong. The second example seems to have another unnamed event lurking in the\nbackground (e.g., Flight 1902 had already arrived late when something else hap-\npened). 20.4 \u2022 R EPRESENTING TIME 449\nTo account for this phenomena, Reichenbach (1947) introduced the notion of\nareference point . In our simple temporal scheme, the current moment in time is reference point\nequated with the time of the utterance and is used as a reference point for when\nthe event occurred (before, at, or after). In Reichenbach\u2019s approach, the notion of\nthe reference point is separated from the utterance time and the event time. The\nfollowing examples illustrate the basics of this approach:\n(20.18) When Mary\u2019s \ufb02ight departed, I ate lunch.\n(20.19) When Mary\u2019s \ufb02ight departed, I had eaten lunch.\nIn both of these examples, the eating event has happened in the past, that is, prior\nto the utterance. However, the verb tense in the \ufb01rst example indicates that the eating\nevent began when the \ufb02ight departed, while the second example indicates that the\neating was accomplished prior to the \ufb02ight\u2019s departure. Therefore, in Reichenbach\u2019s\nterms the departure event speci\ufb01es the reference point. These facts can be accom-\nmodated by additional constraints relating the eating anddeparture events. In the\n\ufb01rst example, the reference point precedes the eating event, and in the second exam-\nple, the eating precedes the reference point. Figure 20.12 illustrates Reichenbach\u2019s\napproach with the primary English tenses. Exercise 20.4 asks you to represent these\nexamples in FOL.\nPast PerfectSimple PastPresent Perfect\nSimple FutureFuture PerfectPresentEE\nEER\nRUR,EUR,U\nU,R,EU,RU\nFigure 20.12 Reichenbach\u2019s approach applied to various English tenses. In these diagrams,\ntime \ufb02ows from left to right, Edenotes the time of the event, Rdenotes the reference time,\nandUdenotes the time of the utterance.\nLanguages have many other ways to convey temporal information besides tense.\nMost useful for our purposes will be temporal expressions like in the morning or\n6:45 orafterwards .\n(20.20) I\u2019d like to go at 6:45 in the morning.\n(20.21) Somewhere around noon, please.\n(20.22) I want to take the train back afterwards.\nIncidentally, temporal expressions display a fascinating metaphorical conceptual\norganization. Temporal expressions in English are frequently expressed in spatial\nterms, as is illustrated by the various uses of at,in,somewhere , and near in these\nexamples (Lakoff and Johnson 1980, Jackendoff 1983). Metaphorical organizations\nsuch as these, in which one domain is systematically expressed in terms of another,\nare very common in languages of the world. 450 CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\n20.5 Representing Aspect\nA related notion to time is aspect , which is what we call the way events can be aspect\ncategorized by their internal temporal structure or temporal contour. By this we\nmean questions like whether events are ongoing or have ended, or whether they are\nconceptualized as happening at a point in time or over some interval. Such notions\nof temporal contour have been used to divide event expressions into classes since\nAristotle, although the set of four classes we\u2019ll introduce here is due to Vendler\n(1967) (you may also see the German term aktionsart used to refer to these classes). aktionsart\nThe most basic aspectual distinction is between events (which involve change) events\nandstates (which do not involve change). Stative expressions represent the notion states\nstative of an event participant being in a state , or having a particular property, at a given\npoint in time. Stative expressions capture aspects of the world at a single point in\ntime, and conceptualize the participant as unchanging and continuous. Consider the\nfollowing ATIS examples.\n(20.23) I like express trains.\n(20.24) I need the cheapest fare.\n(20.25) I want to go \ufb01rst class.\nIn examples like these, the event participant denoted by the subject can be seen as\nexperiencing something at a speci\ufb01c point in time, and don\u2019t involve any kind of\ninternal change over time (the liking or needing is conceptualized as continuous and\nunchanging).\nNon-states (which we\u2019ll refer to as events ) are divided into subclasses; we\u2019ll\nintroduce three here. Activity expressions describe events undertaken by a partic- activity\nipant that occur over a span of time (rather than being conceptualized as a single\npoint in time like stative expressions), and have no particular end point. Of course\nin practice all things end, but the meaning of the expression doesn\u2019t represent this\nfact. Consider the following examples:\n(20.26) She drove a Mazda.\n(20.27) I live in Brooklyn.\nThese examples both specify that the subject is engaged in, or has engaged in, the\nactivity speci\ufb01ed by the verb for some period of time, but doesn\u2019t specify when the\ndriving or living might have stopped.\nTwo more classes of expressions, achievement expressions and accomplish-\nment expressions, describe events that take place over time, but also conceptualize\nthe event as having a particular kind of endpoint or goal. The Greek word telos\nmeans \u2018end\u2019 or \u2019goal\u2019 and so the events described by these kinds of expressions are\noften called telic events. telic\nAccomplishment expressions describe events that have a natural end point andaccomplishment\nexpressions\nresult in a particular state. Consider the following examples:\n(20.28) He booked me a reservation.\n(20.29) The 7:00 train got me to New York City.\nIn these examples, an event is seen as occurring over some period of time that ends\nwhen the intended state is accomplished (i.e., the state of me having a reservation,\nor me being in New York City).\nThe \ufb01nal aspectual class, achievement expressions , is only subtly different thanachievement\nexpressions\naccomplishments. Consider the following: 20.6 \u2022 T EMPORALLY ANNOTATED DATASETS : TIMEBANK 451\n(20.30) She found her gate.\n(20.31) I reached New York.\nLike accomplishment expressions, achievement expressions result in a state. But\nunlike accomplishments, achievement events are \u2018punctual\u2019: they are thought of as\nhappening in an instant and the verb doesn\u2019t conceptualize the process or activ-\nity leading up the state. Thus the events in these examples may in fact have been\npreceded by extended searching ortraveling events, but the verb doesn\u2019t conceptu-\nalize these preceding processes, but rather conceptualizes the events corresponding\nto\ufb01nding andreaching as points, not intervals.\nIn summary, a standard way of categorizing event expressions by their temporal\ncontours is via these four general classes:\nStative: I know my departure gate.\nActivity: John is \ufb02ying.\nAccomplishment: Sally booked her \ufb02ight.\nAchievement: She found her gate.\nBefore moving on, note that event expressions can easily be shifted from one\nclass to another. Consider the following examples:\n(20.32) I \ufb02ew.\n(20.33) I \ufb02ew to New York.\nThe \ufb01rst example is a simple activity; it has no natural end point. The second ex-\nample is clearly an accomplishment event since it has an end point, and results in a\nparticular state. Clearly, the classi\ufb01cation of an event is not solely governed by the\nverb, but by the semantics of the entire expression in context.\n20.6 Temporally Annotated Datasets: TimeBank\nTheTimeBank corpus consists of American English text annotated with temporal TimeBank\ninformation (Pustejovsky et al., 2003). The annotations use TimeML (Saur \u00b4\u0131 et al.,\n2006), a markup language for time based on Allen\u2019s interval algebra discussed above\n(Allen, 1984). There are three types of TimeML objects: an E VENT represent events\nand states, a T IME represents time expressions like dates, and a L INK represents\nvarious relationships between events and times (event-event, event-time, and time-\ntime). The links include temporal links (TL INK) for the 13 Allen relations, aspec-\ntual links (AL INK) for aspectual relationships between events and subevents, and\nSLINKS which mark factuality.\nConsider the following sample sentence and its corresponding markup shown in\nFig. 20.13, selected from one of the TimeBank documents.\n(20.34) Delta Air Lines earnings soared 33% to a record in the \ufb01scal \ufb01rst quarter,\nbucking the industry trend toward declining pro\ufb01ts.\nThis text has three events and two temporal expressions (including the creation\ntime of the article, which serves as the document time), and four temporal links that\ncapture the using the Allen relations:\n\u2022 Soaring e1isincluded in the \ufb01scal \ufb01rst quarter t58\n\u2022 Soaring e1isbefore 1989-10-26 t57\n\u2022 Soaring e1issimultaneous with the bucking e3 452 CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\n<TIMEX3 tid=\"t57\" type=\"DATE\" value=\"1989-10-26\" functionInDocument=\"CREATION_TIME\">\n10/26/89 </TIMEX3>\nDelta Air Lines earnings <EVENT eid=\"e1\" class=\"OCCURRENCE\"> soared </EVENT> 33% to a\nrecord in <TIMEX3 tid=\"t58\" type=\"DATE\" value=\"1989-Q1\" anchorTimeID=\"t57\"> the\nfiscal first quarter </TIMEX3>, <EVENT eid=\"e3\" class=\"OCCURRENCE\">bucking</EVENT>\nthe industry trend toward <EVENT eid=\"e4\" class=\"OCCURRENCE\">declining</EVENT>\nprofits.\nFigure 20.13 Example from the TimeBank corpus.\n\u2022 Declining e4includes soaring e1\nWe can also visualize the links as a graph. The TimeBank snippet in Eq. 20.35\nwould be represented with a graph like Fig. 20.14.\n(20.35) [DCT:11/02/891] 1: Paci\ufb01c First Financial Corp. said 2shareholders\napproved 3itsacquisition 4by Royal Trustco Ltd. of Toronto for $27 a share,\nor $212 million. The thrift holding company said 5itexpects 6toobtain 7\nregulatory approval 8andcomplete 9thetransaction 10byyear-end 11.\n1234567811910BEFOREBEFOREAFTERSIMULTANEOUSENDSCULMINATESBEFOREEVIDENTIALMODALFACTIVEMODALEVIDENTIALMODAL\nFigure 20.14 A graph of the text in Eq. 20.35, adapted from (Ocal et al., 2022). TL INKS\nare shown in blue, AL INKS in red, and SL INKS in green.\n20.7 Automatic Temporal Analysis\nHere we introduce the three common steps used in analyzing time in text:\n1. Extracting temporal expressions\n2.Normalizing these expressions, by converting them to a standard format.\n3.Linking events to times and extracting time graphs and timelines\n20.7.1 Extracting Temporal Expressions\nTemporal expressions are phrases that refer to absolute points in time, relative times,\ndurations, and sets of these. Absolute temporal expressions are those that can be absolute\nmapped directly to calendar dates, times of day, or both. Relative temporal expres- relative\nsions map to particular times through some other reference point (as in a week from\nlast Tuesday ). Finally, durations denote spans of time at varying levels of granular- duration\nity (seconds, minutes, days, weeks, centuries, etc.). Figure 20.15 lists some sample\ntemporal expressions in each of these categories.\nTemporal expressions are grammatical constructions that often have temporal\nlexical triggers as their heads, making them easy to \ufb01nd. Lexical triggers might lexical triggers 20.7 \u2022 A UTOMATIC TEMPORAL ANALYSIS 453\nAbsolute Relative Durations\nApril 24, 1916 yesterday four hours\nThe summer of \u201977 next semester three weeks\n10:15 AM two weeks from yesterday six days\nThe 3rd quarter of 2006 last quarter the last three quarters\nFigure 20.15 Examples of absolute, relational and durational temporal expressions.\nbe nouns, proper nouns, adjectives, and adverbs; full temporal expressions consist\nof their phrasal projections: noun phrases, adjective phrases, and adverbial phrases\n(Figure 20.16).\nCategory Examples\nNoun morning ,noon ,night ,winter ,dusk,dawn\nProper Noun January, Monday, Ides, Easter, Rosh Hashana, Ramadan, Tet\nAdjective recent, past, annual, former\nAdverb hourly, daily, monthly, yearly\nFigure 20.16 Examples of temporal lexical triggers.\nThe task is to detect temporal expressions in running text, like this examples,\nshown with TIMEX3 tags (Pustejovsky et al. 2005, Ferro et al. 2005).\nA fare increase initiated <TIMEX3>last week</TIMEX3>by UAL\nCorp\u2019s United Airlines was matched by competitors over <TIMEX3>the\nweekend</TIMEX3>, marking the second successful fare increase in\n<TIMEX3>two weeks</TIMEX3>.\nRule-based approaches use cascades of regular expressions to recognize larger\nand larger chunks from previous stages, based on patterns containing parts of speech,\ntrigger words (e.g., February ) or classes (e.g., MONTH ) (Chang and Manning, 2012;\nStr\u00a8otgen and Gertz, 2013; Chambers, 2013). Here\u2019s a rule from SUTime (Chang and\nManning, 2012) for detecting expressions like 3 years old :\n/(\\d+)[-\\s]($TEUnits)(s)?([-\\s]old)?/\nSequence-labeling approaches use the standard IOB scheme, marking words\nthat are either (I)nside, (O)utside or at the (B)eginning of a temporal expression:\nA\nOfare\nOincrease\nOinitiated\nOlast\nBweek\nIby\nOUAL\nOCorp\u2019s...\nO\nA statistical sequence labeler is trained, using either embeddings or a \ufb01ne-tuned\nencoder, or classic features extracted from the token and context including words,\nlexical triggers, and POS.\nTemporal expression recognizers are evaluated with the usual recall, precision,\nandF-measures. A major dif\ufb01culty for all of these very lexicalized approaches is\navoiding expressions that trigger false positives:\n(20.36) 1984 tells the story of Winston Smith...\n(20.37) ...U2\u2019s classic Sunday Bloody Sunday\n20.7.2 Temporal Normalization\nTemporal normalization is the task of mapping a temporal expression to a pointtemporal\nnormalization\nin time or to a duration. Points in time correspond to calendar dates, to times of\nday, or both. Durations primarily consist of lengths of time. Normalized times 454 CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\n<TIMEX3 i d =\u201d t1 \u2019 \u2019 t y p e =\u201dDATE\u201d v a l u e =\u201d 2007 \u221207 \u221202 \u201d f u n c t i o n I n D o c u m e n t =\u201dCREATION TIME\u201d >\nJ u l y 2 , 2007 </TIMEX3 >A f a r e i n c r e a s e i n i t i a t e d <TIMEX3 i d =\u201d t 2 \u201d t y p e =\u201dDATE\u201d\nv a l u e =\u201d 2007 \u2212W26\u201d anchorTimeID=\u201d t 1 \u201d >l a s t week </TIMEX3 >by U n i t e d A i r l i n e s was\nmatched by c o m p e t i t o r s o ve r <TIMEX3 i d =\u201d t 3 \u201d t y p e =\u201dDURATION\u201d v a l u e =\u201dP1WE\u201d\nanchorTimeID=\u201d t 1 \u201d >t h e weekend </TIMEX3 >, marking t h e second s u c c e s s f u l f a r e\ni n c r e a s e i n <TIMEX3 i d =\u201d t 4 \u201d t y p e =\u201dDURATION\u201d v a l u e =\u201dP2W\u201d anchorTimeID=\u201d t 1 \u201d >two\nweeks </TIMEX3 >.\nFigure 20.17 TimeML markup including normalized values for temporal expressions.\nare represented via the ISO 8601 standard for encoding temporal values (ISO8601,\n2004). Fig. 20.17 reproduces our earlier example with these value attributes.\nThe dateline, or document date, for this text was July 2, 2007 . The ISO repre-\nsentation for this kind of expression is YYYY-MM-DD, or in this case, 2007-07-02.\nThe encodings for the temporal expressions in our sample text all follow from this\ndate, and are shown here as values for the VALUE attribute.\nThe \ufb01rst temporal expression in the text proper refers to a particular week of the\nyear. In the ISO standard, weeks are numbered from 01 to 53, with the \ufb01rst week\nof the year being the one that has the \ufb01rst Thursday of the year. These weeks are\nrepresented with the template YYYY-Wnn. The ISO week for our document date is\nweek 27; thus the value for last week is represented as \u201c2007-W26\u201d.\nThe next temporal expression is the weekend . ISO weeks begin on Monday;\nthus, weekends occur at the end of a week and are fully contained within a single\nweek. Weekends are treated as durations, so the value of the VALUE attribute has\nto be a length. Durations are represented according to the pattern P nx, where nis\nan integer denoting the length and xrepresents the unit, as in P3Y for three years\nor P2D for two days . In this example, one weekend is captured as P1WE. In this\ncase, there is also suf\ufb01cient information to anchor this particular weekend as part of\na particular week. Such information is encoded in the ANCHOR TIMEID attribute.\nFinally, the phrase two weeks also denotes a duration captured as P2W. Figure 20.18\ngive some more examples, but there is a lot more to the various temporal annotation\nstandards; consult ISO8601 (2004), Ferro et al. (2005), and Pustejovsky et al. (2005)\nfor more details.\nUnit Pattern Sample Value\nFully speci\ufb01ed dates YYYY-MM-DD 1991-09-28\nWeeks YYYY-Wnn 2007-W27\nWeekends PnWE P1WE\n24-hour clock times HH:MM:SS 11:13:45\nDates and times YYYY-MM-DDTHH:MM:SS 1991-09-28T11:00:00\nFinancial quarters Qn 1999-Q3\nFigure 20.18 Sample ISO patterns for representing various times and durations.\nMost current approaches to temporal normalization are rule-based (Chang and\nManning 2012, Str \u00a8otgen and Gertz 2013). Patterns that match temporal expressions\nare associated with semantic analysis procedures. For example, the pattern above for\nrecognizing phrases like 3 years old can be associated with the predicate Duration\nthat takes two arguments, the length and the unit of time:\npattern: /(\\d+)[-\\s]($TEUnits)(s)?([-\\s]old)?/\nresult: Duration($1, $2)\nThe task is dif\ufb01cult because fully quali\ufb01ed temporal expressions are fairly rare\nin real texts. Most temporal expressions in news articles are incomplete and are only\nimplicitly anchored, often with respect to the dateline of the article, which we refer 20.7 \u2022 A UTOMATIC TEMPORAL ANALYSIS 455\nto as the document\u2019s temporal anchor . The values of temporal expressions suchtemporal\nanchor\nastoday ,yesterday , ortomorrow can all be computed with respect to this temporal\nanchor. The semantic procedure for today simply assigns the anchor, and the attach-\nments for tomorrow andyesterday add a day and subtract a day from the anchor,\nrespectively. Of course, given the cyclic nature of our representations for months,\nweeks, days, and times of day, our temporal arithmetic procedures must use modulo\narithmetic appropriate to the time unit being used.\nUnfortunately, even simple expressions such as the weekend orWednesday in-\ntroduce a fair amount of complexity. In our current example, the weekend clearly\nrefers to the weekend of the week that immediately precedes the document date. But\nthis won\u2019t always be the case, as is illustrated in the following example.\n(20.38) Random security checks that began yesterday at Sky Harbor will continue\nat least through the weekend.\nIn this case, the expression the weekend refers to the weekend of the week that the\nanchoring date is part of (i.e., the coming weekend). The information that signals\nthis meaning comes from the tense of continue , the verb governing the weekend .\nRelative temporal expressions are handled with temporal arithmetic similar to\nthat used for today andyesterday . The document date indicates that our example\narticle is ISO week 27, so the expression last week normalizes to the current week\nminus 1. To resolve ambiguous next andlastexpressions we consider the distance\nfrom the anchoring date to the nearest unit. Next Friday can refer either to the\nimmediately next Friday or to the Friday following that, but the closer the document\ndate is to a Friday, the more likely it is that the phrase will skip the nearest one. Such\nambiguities are handled by encoding language and domain-speci\ufb01c heuristics into\nthe temporal attachments.\n20.7.3 Temporal Ordering of Events\nThe goal of temporal analysis, is to link times to events and then \ufb01t all these events\ninto a complete timeline. This ambitious task is the subject of considerable current\nresearch but solving it with a high level of accuracy is beyond the capabilities of\ncurrent systems. A somewhat simpler, but still useful, task is to impose a partial or-\ndering on the events and temporal expressions mentioned in a text. Such an ordering\ncan provide many of the same bene\ufb01ts as a true timeline. An example of such a par-\ntial ordering is the determination that the fare increase by American Airlines came\nafter the fare increase by United in our sample text. Determining such an ordering\ncan be viewed as a binary relation detection and classi\ufb01cation task.\nEven this partial ordering task assumes that in addition to the detecting and nor-\nmalizing time expressions steps described above, we have already detected all the\nevents in the text. Indeed, many temporal expressions are anchored to events men-\ntioned in a text and not directly to other temporal expressions. Consider the follow-\ning example:\n(20.39) One week after the storm, JetBlue issued its customer bill of rights.\nTo determine when JetBlue issued its customer bill of rights we need to determine\nthe time of the storm event, and then we need to modify that time by the temporal\nexpression one week after .\nThus once the events and times have been detected, our goal next is to assert links\nbetween all the times and events: i.e. creating event-event, event-time, time-time,\nDCT-event, and DCT-time TimeML TL INKS . This can be done by training time\nrelation classi\ufb01ers to predict the correct T: INKbetween each pair of times/events, 456 CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\nsupervised by the gold labels in the TimeBank corpus with features like words/em-\nbeddings, parse paths, tense and aspect The sieve-based architecture using precision-\nranked sets of classi\ufb01ers, which we\u2019ll introduce in Chapter 23, is also commonly\nused.\nSystems that perform all 4 tasks (time extraction creation and normalization,\nevent extraction, and time/event linking) include TARSQI (Verhagen et al., 2005)\nCLEAR TK (Bethard, 2013), CAEVO (Chambers et al., 2014), and CATENA (Mirza\nand Tonelli, 2016).\n20.8 Template Filling\nMany texts contain reports of events, and possibly sequences of events, that often\ncorrespond to fairly common, stereotypical situations in the world. These abstract\nsituations or stories, related to what have been called scripts (Schank and Abel- scripts\nson, 1977), consist of prototypical sequences of sub-events, participants, and their\nroles. The strong expectations provided by these scripts can facilitate the proper\nclassi\ufb01cation of entities, the assignment of entities into roles and relations, and most\ncritically, the drawing of inferences that \ufb01ll in things that have been left unsaid. In\ntheir simplest form, such scripts can be represented as templates consisting of \ufb01xed templates\nsets of slots that take as values slot-\ufb01llers belonging to particular classes. The task\noftemplate \ufb01lling is to \ufb01nd documents that invoke particular scripts and then \ufb01ll the template \ufb01lling\nslots in the associated templates with \ufb01llers extracted from the text. These slot-\ufb01llers\nmay consist of text segments extracted directly from the text, or they may consist of\nconcepts that have been inferred from text elements through some additional pro-\ncessing.\nA \ufb01lled template from our original airline story might look like the following.\nFARE-RAISE ATTEMPT :2\n6664LEAD AIRLINE : U NITED AIRLINES\nAMOUNT : $6\nEFFECTIVE DATE: 2006-10-26\nFOLLOWER : A MERICAN AIRLINES3\n7775\nThis template has four slots ( LEAD AIRLINE ,AMOUNT ,EFFECTIVE DATE ,FOL-\nLOWER ). The next section describes a standard sequence-labeling approach to \ufb01lling\nslots. Section 20.8.2 then describes an older system based on the use of cascades of\n\ufb01nite-state transducers and designed to address a more complex template-\ufb01lling task\nthat current learning-based systems don\u2019t yet address.\n20.8.1 Machine Learning Approaches to Template Filling\nIn the standard paradigm for template \ufb01lling, we are given training documents with\ntext spans annotated with prede\ufb01ned templates and their slot \ufb01llers. Our goal is to\ncreate one template for each event in the input, \ufb01lling in the slots with text spans.\nThe task is generally modeled by training two separate supervised systems. The\n\ufb01rst system decides whether the template is present in a particular sentence. This\ntask is called template recognition or sometimes, in a perhaps confusing bit oftemplate\nrecognition\nterminology, event recognition . Template recognition can be treated as a text classi-\n\ufb01cation task, with features extracted from every sequence of words that was labeled\nin training documents as \ufb01lling any slot from the template being detected. The usual 20.8 \u2022 T EMPLATE FILLING 457\nset of features can be used: tokens, embeddings, word shapes, part-of-speech tags,\nsyntactic chunk tags, and named entity tags.\nThe second system has the job of role-\ufb01ller extraction . A separate classi\ufb01er isrole-\ufb01ller\nextraction\ntrained to detect each role ( LEAD -AIRLINE ,AMOUNT , and so on). This can be a\nbinary classi\ufb01er that is run on every noun-phrase in the parsed input sentence, or a\nsequence model run over sequences of words. Each role classi\ufb01er is trained on the\nlabeled data in the training set. Again, the usual set of features can be used, but now\ntrained only on an individual noun phrase or the \ufb01llers of a single slot.\nMultiple non-identical text segments might be labeled with the same slot la-\nbel. For example in our sample text, the strings United orUnited Airlines might be\nlabeled as the L EAD AIRLINE . These are not incompatible choices and the corefer-\nence resolution techniques introduced in Chapter 23 can provide a path to a solution.\nA variety of annotated collections have been used to evaluate this style of ap-\nproach to template \ufb01lling, including sets of job announcements, conference calls for\npapers, restaurant guides, and biological texts. A key open question is extracting\ntemplates in cases where there is no training data or even prede\ufb01ned templates, by\ninducing templates as sets of linked events (Chambers and Jurafsky, 2011).\n20.8.2 Earlier Finite-State Template-Filling Systems\nThe templates above are relatively simple. But consider the task of producing a\ntemplate that contained all the information in a text like this one (Grishman and\nSundheim, 1995):\nBridgestone Sports Co. said Friday it has set up a joint venture in Taiwan\nwith a local concern and a Japanese trading house to produce golf clubs to be\nshipped to Japan. The joint venture, Bridgestone Sports Taiwan Co., capital-\nized at 20 million new Taiwan dollars, will start production in January 1990\nwith production of 20,000 iron and \u201cmetal wood\u201d clubs a month.\nThe MUC-5 \u2018joint venture\u2019 task (the Message Understanding Conferences were\na series of U.S. government-organized information-extraction evaluations) was to\nproduce hierarchically linked templates describing joint ventures. Figure 20.19\nshows a structure produced by the FASTUS system (Hobbs et al., 1997). Note how\nthe \ufb01ller of the ACTIVITY slot of the TIE-UPtemplate is itself a template with slots.\nTie-up-1 Activity-1 :\nRELATIONSHIP tie-up C OMPANY Bridgestone Sports Taiwan Co.\nENTITIES Bridgestone Sports Co. P RODUCT iron and \u201cmetal wood\u201d clubs\na local concern S TART DATE DURING: January 1990\na Japanese trading house\nJOINT VENTURE Bridgestone Sports Taiwan Co.\nACTIVITY Activity-1\nAMOUNT NT$20000000\nFigure 20.19 The templates produced by FASTUS given the input text on page 457.\nEarly systems for dealing with these complex templates were based on cascades\nof transducers based on handwritten rules, as sketched in Fig. 20.20.\nThe \ufb01rst four stages use handwritten regular expression and grammar rules to\ndo basic tokenization, chunking, and parsing. Stage 5 then recognizes entities and\nevents with a recognizer based on \ufb01nite-state transducers (FSTs), and inserts the rec-\nognized objects into the appropriate slots in templates. This FST recognizer is based 458 CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\nNo. Step Description\n1 Tokens Tokenize input stream of characters\n2 Complex Words Multiword phrases, numbers, and proper names.\n3 Basic phrases Segment sentences into noun and verb groups\n4 Complex phrases Identify complex noun groups and verb groups\n5 Semantic Patterns Identify entities and events, insert into templates.\n6 Merging Merge references to the same entity or event\nFigure 20.20 Levels of processing in FASTUS (Hobbs et al., 1997). Each level extracts a\nspeci\ufb01c type of information which is then passed on to the next higher level.\non hand-built regular expressions like the following (NG indicates Noun-Group and\nVG Verb-Group), which matches the \ufb01rst sentence of the news story above.\nNG(Company/ies) VG(Set-up) NG(Joint-Venture) with NG(Company/ies)\nVG(Produce) NG(Product)\nThe result of processing these two sentences is the \ufb01ve draft templates (Fig. 20.21)\nthat must then be merged into the single hierarchical structure shown in Fig. 20.19.\nThe merging algorithm, after performing coreference resolution, merges two activi-\nties that are likely to be describing the same events.\n#Template/Slot Value\n1RELATIONSHIP : TIE-UP\nENTITIES : Bridgestone Co., a local concern, a Japanese trading house\n2ACTIVITY : PRODUCTION\nPRODUCT : \u201cgolf clubs\u201d\n3RELATIONSHIP : TIE-UP\nJOINT VENTURE : \u201cBridgestone Sports Taiwan Co.\u201d\nAMOUNT : NT$20000000\n4ACTIVITY : PRODUCTION\nCOMPANY : \u201cBridgestone Sports Taiwan Co.\u201d\nSTART DATE: DURING : January 1990\n5ACTIVITY : PRODUCTION\nPRODUCT : \u201ciron and \u201cmetal wood\u201d clubs\u201d\nFigure 20.21 The \ufb01ve partial templates produced by stage 5 of FASTUS . These templates\nare merged in stage 6 to produce the \ufb01nal template shown in Fig. 20.19 on page 457.\n20.9 Summary\nThis chapter has explored techniques for extracting limited forms of semantic con-\ntent from texts.\n\u2022Relations among entities can be extracted by pattern-based approaches, su-\npervised learning methods when annotated training data is available, lightly\nsupervised bootstrapping methods when small numbers of seed tuples or\nseed patterns are available, distant supervision when a database of relations\nis available, and unsupervised orOpen IE methods.\n\u2022 Reasoning about time can be facilitated by detection and normalization of\ntemporal expressions . BIBLIOGRAPHICAL AND HISTORICAL NOTES 459\n\u2022Events can be ordered in time using sequence models and classi\ufb01ers trained\non temporally- and event-labeled data like the TimeBank corpus .\n\u2022Template-\ufb01lling applications can recognize stereotypical situations in texts\nand assign elements from the text to roles represented as \ufb01xed sets of slots .\nBibliographical and Historical Notes\nThe earliest work on information extraction addressed the template-\ufb01lling task in the\ncontext of the Frump system (DeJong, 1982). Later work was stimulated by the U.S.\ngovernment-sponsored MUC conferences (Sundheim 1991, Sundheim 1992, Sund-\nheim 1993, Sundheim 1995). Early MUC systems like CIRCUS system (Lehnert\net al., 1991) and SCISOR (Jacobs and Rau, 1990) were quite in\ufb02uential and inspired\nlater systems like FASTUS (Hobbs et al., 1997). Chinchor et al. (1993) describe the\nMUC evaluation techniques.\nDue to the dif\ufb01culty of porting systems from one domain to another, attention\nshifted to machine learning approaches. Early supervised learning approaches to\nIE (Cardie 1993, Cardie 1994, Riloff 1993, Soderland et al. 1995, Huffman 1996)\nfocused on automating the knowledge acquisition process, mainly for \ufb01nite-state\nrule-based systems. Their success, and the earlier success of HMM-based speech\nrecognition, led to the use of sequence labeling (HMMs: Bikel et al. 1997; MEMMs\nMcCallum et al. 2000; CRFs: Lafferty et al. 2001), and a wide exploration of fea-\ntures (Zhou et al., 2005). Neural approaches followed from the pioneering results of\nCollobert et al. (2011), who applied a CRF on top of a convolutional net.\nProgress in this area continues to be stimulated by formal evaluations with shared\nbenchmark datasets, including the Automatic Content Extraction (ACE) evaluations\nof 2000-2007 on named entity recognition, relation extraction, and temporal ex-\npressions1, the KBP (Knowledge Base Population ) evaluations (Ji et al. 2010, Sur- KBP\ndeanu 2013) of relation extraction tasks like slot \ufb01lling (extracting attributes (\u2018slots\u2019) slot \ufb01lling\nlike age, birthplace, and spouse for a given entity) and a series of SemEval work-\nshops (Hendrickx et al., 2009).\nSemisupervised relation extraction was \ufb01rst proposed by Hearst (1992b), and\nextended by systems like AutoSlog-TS (Riloff, 1996), DIPRE (Brin, 1998), SNOW-\nBALL (Agichtein and Gravano, 2000), and Jones et al. (1999). The distant super-\nvision algorithm we describe was drawn from Mintz et al. (2009), who \ufb01rst used\nthe term \u2018distant supervision\u2019 (which was suggested to them by Chris Manning)\nbut similar ideas had occurred in earlier systems like Craven and Kumlien (1999)\nand Morgan et al. (2004) under the name weakly labeled data , as well as in Snow\net al. (2005) and Wu and Weld (2007). Among the many extensions are Wu and\nWeld (2010), Riedel et al. (2010), and Ritter et al. (2013). Open IE systems include\nKNOW ITALLEtzioni et al. (2005), TextRunner (Banko et al., 2007), and R EVERB\n(Fader et al., 2011). See Riedel et al. (2013) for a universal schema that combines\nthe advantages of distant supervision and Open IE.\n1www.nist.gov/speech/tests/ace/ 460 CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\nExercises\n20.1 Acronym expansion, the process of associating a phrase with an acronym, can\nbe accomplished by a simple form of relational analysis. Develop a system\nbased on the relation analysis approaches described in this chapter to populate\na database of acronym expansions. If you focus on English Three Letter\nAcronyms (TLAs) you can evaluate your system\u2019s performance by comparing\nit to Wikipedia\u2019s TLA page.\n20.2 Acquire the CMU seminar corpus and develop a template-\ufb01lling system by\nusing any of the techniques mentioned in Section 20.8. Analyze how well\nyour system performs as compared with state-of-the-art results on this corpus.\n20.3 A useful functionality in newer email and calendar applications is the ability\nto associate temporal expressions connected with events in email (doctor\u2019s\nappointments, meeting planning, party invitations, etc.) with speci\ufb01c calendar\nentries. Collect a corpus of email containing temporal expressions related to\nevent planning. How do these expressions compare to the kinds of expressions\ncommonly found in news text that we\u2019ve been discussing in this chapter?\n20.4 For the following sentences, give FOL translations that capture the temporal\nrelationships between the events.\n1. When Mary\u2019s \ufb02ight departed, I ate lunch.\n2. When Mary\u2019s \ufb02ight departed, I had eaten lunch. CHAPTER\n21Semantic Role Labeling\n\u201cWho, What, Where, When, With what, Why, How\u201d\nThe seven circumstances, associated with Hermagoras and Aristotle (Sloan, 2010)\nSometime between the 7th and 4th centuries BCE, the Indian grammarian P \u00afan.ini1\nwrote a famous treatise on Sanskrit grammar, the As .t.\u00afadhy \u00afay\u00af\u0131 (\u20188 books\u2019), a treatise\nthat has been called \u201cone of the greatest monuments of hu-\nman intelligence\u201d (Bloom\ufb01eld, 1933, 11). The work de-\nscribes the linguistics of the Sanskrit language in the form\nof 3959 sutras, each very ef\ufb01ciently (since it had to be\nmemorized!) expressing part of a formal rule system that\nbrilliantly pre\ufb01gured modern mechanisms of formal lan-\nguage theory (Penn and Kiparsky, 2012). One set of rules\ndescribes the k\u00afarakas , semantic relationships between a\nverb and noun arguments, roles like agent ,instrument , or\ndestination . P\u00afan.ini\u2019s work was the earliest we know of\nthat modeled the linguistic realization of events and their\nparticipants. This task of understanding how participants relate to events\u2014being\nable to answer the question \u201cWho did what to whom\u201d (and perhaps also \u201cwhen and\nwhere\u201d)\u2014is a central question of natural language processing.\nLet\u2019s move forward 2.5 millennia to the present and consider the very mundane\ngoal of understanding text about a purchase of stock by XYZ Corporation. This\npurchasing event and its participants can be described by a wide variety of surface\nforms. The event can be described by a verb ( sold, bought ) or a noun ( purchase ),\nand XYZ Corp can be the syntactic subject (of bought ), the indirect object (of sold),\nor in a genitive or noun compound relation (with the noun purchase ) despite having\nnotionally the same role in all of them:\n\u2022 XYZ corporation bought the stock.\n\u2022 They sold the stock to XYZ corporation.\n\u2022 The stock was bought by XYZ corporation.\n\u2022 The purchase of the stock by XYZ corporation...\n\u2022 The stock purchase by XYZ corporation...\nIn this chapter we introduce a level of representation that captures the common-\nality between these sentences: there was a purchase event, the participants were\nXYZ Corp and some stock, and XYZ Corp was the buyer. These shallow semantic\nrepresentations , semantic roles , express the role that arguments of a predicate take\nin the event, codi\ufb01ed in databases like PropBank and FrameNet. We\u2019ll introduce\nsemantic role labeling , the task of assigning roles to spans in sentences, and selec-\ntional restrictions , the preferences that predicates express about their arguments,\nsuch as the fact that the theme of eatis generally something edible.\n1Figure shows a birch bark manuscript from Kashmir of the Rupavatra, a grammatical textbook based\non the Sanskrit grammar of Panini. Image from the Wellcome Collection. 462 CHAPTER 21 \u2022 S EMANTIC ROLE LABELING\n21.1 Semantic Roles\nConsider the meanings of the arguments Sasha ,Pat,the window , and the door in\nthese two sentences.\n(21.1) Sasha broke the window.\n(21.2) Pat opened the door.\nThe subjects Sasha andPat, what we might call the breaker of the window-\nbreaking event and the opener of the door-opening event, have something in com-\nmon. They are both volitional actors, often animate, and they have direct causal\nresponsibility for their events.\nThematic roles are a way to capture this semantic commonality between break- thematic roles\nersandopeners . We say that the subjects of both these verbs are agents . Thus, agents\nAGENT is the thematic role that represents an abstract idea such as volitional causa-\ntion. Similarly, the direct objects of both these verbs, the BrokenThing andOpenedThing ,\nare both prototypically inanimate objects that are affected in some way by the action.\nThe semantic role for these participants is theme . theme\nThematic Role De\ufb01nition\nAGENT The volitional causer of an event\nEXPERIENCER The experiencer of an event\nFORCE The non-volitional causer of the event\nTHEME The participant most directly affected by an event\nRESULT The end product of an event\nCONTENT The proposition or content of a propositional event\nINSTRUMENT An instrument used in an event\nBENEFICIARY The bene\ufb01ciary of an event\nSOURCE The origin of the object of a transfer event\nGOAL The destination of an object of a transfer event\nFigure 21.1 Some commonly used thematic roles with their de\ufb01nitions.\nAlthough thematic roles are one of the oldest linguistic models, as we saw above,\ntheir modern formulation is due to Fillmore (1968) and Gruber (1965). Although\nthere is no universally agreed-upon set of roles, Figs. 21.1 and 21.2 list some the-\nmatic roles that have been used in various computational papers, together with rough\nde\ufb01nitions and examples. Most thematic role sets have about a dozen roles, but we\u2019ll\nsee sets with smaller numbers of roles with even more abstract meanings, and sets\nwith very large numbers of roles that are speci\ufb01c to situations. We\u2019ll use the general\nterm semantic roles for all sets of roles, whether small or large. semantic roles\n21.2 Diathesis Alternations\nThe main reason computational systems use semantic roles is to act as a shallow\nmeaning representation that can let us make simple inferences that aren\u2019t possible\nfrom the pure surface string of words, or even from the parse tree. To extend the\nearlier examples, if a document says that Company A acquired Company B , we\u2019d\nlike to know that this answers the query Was Company B acquired? despite the fact\nthat the two sentences have very different surface syntax. Similarly, this shallow\nsemantics might act as a useful intermediate language in machine translation. 21.2 \u2022 D IATHESIS ALTERNATIONS 463\nThematic Role Example\nAGENT The waiter spilled the soup.\nEXPERIENCER John has a headache.\nFORCE The wind blows debris from the mall into our yards.\nTHEME Only after Benjamin Franklin broke the ice ...\nRESULT The city built a regulation-size baseball diamond ...\nCONTENT Mona asked \u201cYou met Mary Ann at a supermarket?\u201d\nINSTRUMENT He poached cat\ufb01sh, stunning them with a shocking device ...\nBENEFICIARY Whenever Ann Callahan makes hotel reservations for her boss ...\nSOURCE I \ufb02ew in from Boston .\nGOAL I drove to Portland .\nFigure 21.2 Some prototypical examples of various thematic roles.\nSemantic roles thus help generalize over different surface realizations of pred-\nicate arguments. For example, while the AGENT is often realized as the subject of\nthe sentence, in other cases the THEME can be the subject. Consider these possible\nrealizations of the thematic arguments of the verb break :\n(21.3) John\nAGENTbroke the window.\nTHEME\n(21.4) John\nAGENTbroke the window\nTHEMEwith a rock.\nINSTRUMENT\n(21.5) The rock\nINSTRUMENTbroke the window.\nTHEME\n(21.6) The window\nTHEMEbroke.\n(21.7) The window\nTHEMEwas broken by John.\nAGENT\nThese examples suggest that break has (at least) the possible arguments AGENT ,\nTHEME , and INSTRUMENT . The set of thematic role arguments taken by a verb is\noften called the thematic grid ,q-grid, or case frame . We can see that there are thematic grid\ncase frame (among others) the following possibilities for the realization of these arguments of\nbreak :\nAGENT /Subject, THEME /Object\nAGENT /Subject, THEME /Object, INSTRUMENT /PPwith\nINSTRUMENT /Subject, THEME /Object\nTHEME /Subject\nIt turns out that many verbs allow their thematic roles to be realized in various\nsyntactic positions. For example, verbs like give can realize the THEME and GOAL\narguments in two different ways:\n(21.8) a. Doris\nAGENTgave the book\nTHEMEto Cary.\nGOAL\nb.Doris\nAGENTgave Cary\nGOALthe book.\nTHEME\nThese multiple argument structure realizations (the fact that break can take AGENT ,\nINSTRUMENT , or THEME as subject, and give can realize its THEME and GOAL in\neither order) are called verb alternations ordiathesis alternations . The alternationverb\nalternation\nwe showed above for give, the dative alternation , seems to occur with particular se-dative\nalternation\nmantic classes of verbs, including \u201cverbs of future having\u201d ( advance ,allocate ,offer, 464 CHAPTER 21 \u2022 S EMANTIC ROLE LABELING\nowe), \u201csend verbs\u201d ( forward ,hand ,mail), \u201cverbs of throwing\u201d ( kick,pass,throw ),\nand so on. Levin (1993) lists for 3100 English verbs the semantic classes to which\nthey belong (47 high-level classes, divided into 193 more speci\ufb01c classes) and the\nvarious alternations in which they participate. These lists of verb classes have been\nincorporated into the online resource VerbNet (Kipper et al., 2000), which links each\nverb to both WordNet and FrameNet entries.\n21.3 Semantic Roles: Problems with Thematic Roles\nRepresenting meaning at the thematic role level seems like it should be useful in\ndealing with complications like diathesis alternations. Yet it has proved quite dif\ufb01-\ncult to come up with a standard set of roles, and equally dif\ufb01cult to produce a formal\nde\ufb01nition of roles like AGENT ,THEME , or INSTRUMENT .\nFor example, researchers attempting to de\ufb01ne role sets often \ufb01nd they need to\nfragment a role like AGENT orTHEME into many speci\ufb01c roles. Levin and Rappa-\nport Hovav (2005) summarize a number of such cases, such as the fact there seem\nto be at least two kinds of INSTRUMENTS ,intermediary instruments that can appear\nas subjects and enabling instruments that cannot:\n(21.9) a. Shelly cut the banana with a knife.\nb. The knife cut the banana.\n(21.10) a. Shelly ate the sliced banana with a fork.\nb. *The fork ate the sliced banana.\nIn addition to the fragmentation problem, there are cases in which we\u2019d like to\nreason about and generalize across semantic roles, but the \ufb01nite discrete lists of roles\ndon\u2019t let us do this.\nFinally, it has proved dif\ufb01cult to formally de\ufb01ne the thematic roles. Consider the\nAGENT role; most cases of AGENTS are animate, volitional, sentient, causal, but any\nindividual noun phrase might not exhibit all of these properties.\nThese problems have led to alternative semantic role models that use either semantic role\nmany fewer or many more roles.\nThe \ufb01rst of these options is to de\ufb01ne generalized semantic roles that abstract\nover the speci\ufb01c thematic roles. For example, PROTO -AGENT and PROTO -PATIENT proto-agent\nproto-patient are generalized roles that express roughly agent-like and roughly patient-like mean-\nings. These roles are de\ufb01ned, not by necessary and suf\ufb01cient conditions, but rather\nby a set of heuristic features that accompany more agent-like or more patient-like\nmeanings. Thus, the more an argument displays agent-like properties (being voli-\ntionally involved in the event, causing an event or a change of state in another par-\nticipant, being sentient or intentionally involved, moving) the greater the likelihood\nthat the argument can be labeled a PROTO -AGENT . The more patient-like the proper-\nties (undergoing change of state, causally affected by another participant, stationary\nrelative to other participants, etc.), the greater the likelihood that the argument can\nbe labeled a PROTO -PATIENT .\nThe second direction is instead to de\ufb01ne semantic roles that are speci\ufb01c to a\nparticular verb or a particular group of semantically related verbs or nouns.\nIn the next two sections we describe two commonly used lexical resources that\nmake use of these alternative versions of semantic roles. PropBank uses both proto-\nroles and verb-speci\ufb01c semantic roles. FrameNet uses semantic roles that are spe-\nci\ufb01c to a general semantic idea called a frame . 21.4 \u2022 T HEPROPOSITION BANK 465\n21.4 The Proposition Bank\nThe Proposition Bank , generally referred to as PropBank , is a resource of sen- PropBank\ntences annotated with semantic roles. The English PropBank labels all the sentences\nin the Penn TreeBank; the Chinese PropBank labels sentences in the Penn Chinese\nTreeBank. Because of the dif\ufb01culty of de\ufb01ning a universal set of thematic roles,\nthe semantic roles in PropBank are de\ufb01ned with respect to an individual verb sense.\nEach sense of each verb thus has a speci\ufb01c set of roles, which are given only numbers\nrather than names: Arg0 ,Arg1 ,Arg2 , and so on. In general, Arg0 represents the\nPROTO -AGENT , and Arg1 , the PROTO -PATIENT . The semantics of the other roles\nare less consistent, often being de\ufb01ned speci\ufb01cally for each verb. Nonetheless there\nare some generalization; the Arg2 is often the benefactive, instrument, attribute, or\nend state, the Arg3 the start point, benefactive, instrument, or attribute, and the Arg4\nthe end point.\nHere are some slightly simpli\ufb01ed PropBank entries for one sense each of the\nverbs agree andfall. Such PropBank entries are called frame \ufb01les ; note that the\nde\ufb01nitions in the frame \ufb01le for each role (\u201cOther entity agreeing\u201d, \u201cExtent, amount\nfallen\u201d) are informal glosses intended to be read by humans, rather than being formal\nde\ufb01nitions.\n(21.11) agree.01\nArg0: Agreer\nArg1: Proposition\nArg2: Other entity agreeing\nEx1: [ Arg0 The group] agreed [Arg1 it wouldn\u2019t make an offer].\nEx2: [ ArgM-TMP Usually] [ Arg0 John] agrees [Arg2 with Mary]\n[Arg1 on everything].\n(21.12) fall.01\nArg1: Logical subject, patient, thing falling\nArg2: Extent, amount fallen\nArg3: start point\nArg4: end point, end state of arg1\nEx1: [ Arg1 Sales] fell[Arg4 to $25 million] [ Arg3 from $27 million].\nEx2: [ Arg1 The average junk bond] fell[Arg2 by 4.2%].\nNote that there is no Arg0 role for fall, because the normal subject of fallis a\nPROTO -PATIENT .\nThe PropBank semantic roles can be useful in recovering shallow semantic in-\nformation about verbal arguments. Consider the verb increase :\n(21.13) increase.01 \u201cgo up incrementally\u201d\nArg0: causer of increase\nArg1: thing increasing\nArg2: amount increased by, EXT, or MNR\nArg3: start point\nArg4: end point\nA PropBank semantic role labeling would allow us to infer the commonality in\nthe event structures of the following three examples, that is, that in each case Big\nFruit Co. is the AGENT andthe price of bananas is the THEME , despite the differing\nsurface forms. 466 CHAPTER 21 \u2022 S EMANTIC ROLE LABELING\n(21.14) [ Arg0 Big Fruit Co. ] increased [ Arg1 the price of bananas].\n(21.15) [ Arg1 The price of bananas] was increased again [ Arg0 by Big Fruit Co. ]\n(21.16) [ Arg1 The price of bananas] increased [ Arg2 5%].\nPropBank also has a number of non-numbered arguments called ArgMs , (ArgM-\nTMP, ArgM-LOC, etc.) which represent modi\ufb01cation or adjunct meanings. These\nare relatively stable across predicates, so aren\u2019t listed with each frame \ufb01le. Data\nlabeled with these modi\ufb01ers can be helpful in training systems to detect temporal,\nlocation, or directional modi\ufb01cation across predicates. Some of the ArgM\u2019s include:\nTMP when? yesterday evening, now\nLOC where? at the museum, in San Francisco\nDIR where to/from? down, to Bangkok\nMNR how? clearly, with much enthusiasm\nPRP/CAU why? because ... , in response to the ruling\nREC themselves, each other\nADV miscellaneous\nPRD secondary predication ...ate the meat raw\nWhile PropBank focuses on verbs, a related project, NomBank (Meyers et al., NomBank\n2004) adds annotations to noun predicates. For example the noun agreement in\nApple\u2019s agreement with IBM would be labeled with Apple as the Arg0 and IBM as\nthe Arg2. This allows semantic role labelers to assign labels to arguments of both\nverbal and nominal predicates.\n21.5 FrameNet\nWhile making inferences about the semantic commonalities across different sen-\ntences with increase is useful, it would be even more useful if we could make such\ninferences in many more situations, across different verbs, and also between verbs\nand nouns. For example, we\u2019d like to extract the similarity among these three sen-\ntences:\n(21.17) [ Arg1 The price of bananas] increased [ Arg2 5%].\n(21.18) [ Arg1 The price of bananas] rose [ Arg2 5%].\n(21.19) There has been a [ Arg2 5%] rise [ Arg1 in the price of bananas].\nNote that the second example uses the different verb rise, and the third example\nuses the noun rather than the verb rise. We\u2019d like a system to recognize that the\nprice of bananas is what went up, and that 5%is the amount it went up, no matter\nwhether the 5%appears as the object of the verb increased or as a nominal modi\ufb01er\nof the noun rise.\nThe FrameNet project is another semantic-role-labeling project that attempts FrameNet\nto address just these kinds of problems (Baker et al. 1998, Fillmore et al. 2003,\nFillmore and Baker 2009, Ruppenhofer et al. 2016). Whereas roles in the PropBank\nproject are speci\ufb01c to an individual verb, roles in the FrameNet project are speci\ufb01c\nto aframe .\nWhat is a frame? Consider the following set of words:\nreservation, \ufb02ight, travel, buy, price, cost, fare, rates, meal, plane\nThere are many individual lexical relations of hyponymy, synonymy, and so on\nbetween many of the words in this list. The resulting set of relations does not, 21.5 \u2022 F RAME NET 467\nhowever, add up to a complete account of how these words are related. They are\nclearly all de\ufb01ned with respect to a coherent chunk of common-sense background\ninformation concerning air travel.\nWe call the holistic background knowledge that unites these words a frame (Fill- frame\nmore, 1985). The idea that groups of words are de\ufb01ned with respect to some back-\nground information is widespread in arti\ufb01cial intelligence and cognitive science,\nwhere besides frame we see related works like a model (Johnson-Laird, 1983), or model\neven script (Schank and Abelson, 1977). script\nA frame in FrameNet is a background knowledge structure that de\ufb01nes a set of\nframe-speci\ufb01c semantic roles, called frame elements , and includes a set of predi- frame elements\ncates that use these roles. Each word evokes a frame and pro\ufb01les some aspect of the\nframe and its elements. The FrameNet dataset includes a set of frames and frame\nelements, the lexical units associated with each frame, and a set of labeled exam-\nple sentences. For example, the change position onascale frame is de\ufb01ned as\nfollows:\nThis frame consists of words that indicate the change of an Item\u2019s posi-\ntion on a scale (the Attribute) from a starting point (Initial value) to an\nend point (Final value).\nSome of the semantic roles (frame elements) in the frame are de\ufb01ned as in\nFig. 21.3. Note that these are separated into core roles , which are frame speci\ufb01c, and core roles\nnon-core roles , which are more like the Arg-M arguments in PropBank, expressing non-core roles\nmore general properties of time, location, and so on.\nCore Roles\nATTRIBUTE The A TTRIBUTE is a scalar property that the I TEM possesses.\nDIFFERENCE The distance by which an I TEM changes its position on the scale.\nFINAL STATE A description that presents the I TEM\u2019s state after the change in the A TTRIBUTE \u2019s\nvalue as an independent predication.\nFINAL VALUE The position on the scale where the I TEM ends up.\nINITIAL STATE A description that presents the I TEM\u2019s state before the change in the A T-\nTRIBUTE \u2019s value as an independent predication.\nINITIAL VALUE The initial position on the scale from which the I TEM moves away.\nITEM The entity that has a position on the scale.\nVALUE RANGE A portion of the scale, typically identi\ufb01ed by its end points, along which the\nvalues of the A TTRIBUTE \ufb02uctuate.\nSome Non-Core Roles\nDURATION The length of time over which the change takes place.\nSPEED The rate of change of the V ALUE .\nGROUP The G ROUP in which an I TEM changes the value of an\nATTRIBUTE in a speci\ufb01ed way.\nFigure 21.3 The frame elements in the change position onascale frame from the FrameNet Labelers\nGuide (Ruppenhofer et al., 2016).\nHere are some example sentences:\n(21.20) [ ITEM Oil]rose [ATTRIBUTE in price] [ DIFFERENCE by 2%].\n(21.21) [ ITEM It] has increased [FINAL STATE to having them 1 day a month].\n(21.22) [ ITEM Microsoft shares] fell[FINAL VALUE to 7 5/8].\n(21.23) [ ITEM Colon cancer incidence] fell[DIFFERENCE by 50%] [ GROUP among\nmen]. 468 CHAPTER 21 \u2022 S EMANTIC ROLE LABELING\n(21.24) a steady increase [INITIAL VALUE from 9.5] [ FINAL VALUE to 14.3] [ ITEM\nin dividends]\n(21.25) a [ DIFFERENCE 5%] [ ITEM dividend] increase ...\nNote from these example sentences that the frame includes target words like rise,\nfall, and increase . In fact, the complete frame consists of the following words:\nVERBS: dwindle move soar escalation shift\nadvance edge mushroom swell explosion tumble\nclimb explode plummet swing fall\ndecline fall reach triple \ufb02uctuation ADVERBS:\ndecrease \ufb02uctuate rise tumble gain increasingly\ndiminish gain rocket growth\ndip grow shift NOUNS: hike\ndouble increase skyrocket decline increase\ndrop jump slide decrease rise\nFrameNet also codes relationships between frames, allowing frames to inherit\nfrom each other, or representing relations between frames like causation (and gen-\neralizations among frame elements in different frames can be represented by inheri-\ntance as well). Thus, there is a Cause change ofposition onascale frame that is\nlinked to the Change ofposition onascale frame by the cause relation, but that\nadds an A GENT role and is used for causative examples such as the following:\n(21.26) [ AGENT They] raised [ITEM the price of their soda] [ DIFFERENCE by 2%].\nTogether, these two frames would allow an understanding system to extract the\ncommon event semantics of all the verbal and nominal causative and non-causative\nusages.\nFrameNets have also been developed for many other languages including Span-\nish, German, Japanese, Portuguese, Italian, and Chinese.\n21.6 Semantic Role Labeling\nSemantic role labeling (sometimes shortened as SRL ) is the task of automaticallysemantic role\nlabeling\n\ufb01nding the semantic roles of each argument of each predicate in a sentence. Cur-\nrent approaches to semantic role labeling are based on supervised machine learning,\noften using the FrameNet and PropBank resources to specify what counts as a pred-\nicate, de\ufb01ne the set of roles used in the task, and provide training and test sets.\nRecall that the difference between these two models of semantic roles is that\nFrameNet (21.27) employs many frame-speci\ufb01c frame elements as roles, while Prop-\nBank (21.28) uses a smaller number of numbered argument labels that can be inter-\npreted as verb-speci\ufb01c labels, along with the more general ARGM labels. Some\nexamples:\n(21.27)[You] can\u2019t [blame] [the program] [for being unable to identify it]\nCOGNIZER TARGET EVALUEE REASON\n(21.28)[The San Francisco Examiner] issued [a special edition] [yesterday]\nARG0 TARGET ARG 1 ARGM -TMP\n21.6.1 A Feature-based Algorithm for Semantic Role Labeling\nA simpli\ufb01ed feature-based semantic role labeling algorithm is sketched in Fig. 21.4.\nFeature-based algorithms\u2014from the very earliest systems like (Simmons, 1973)\u2014\nbegin by parsing, using broad-coverage parsers to assign a parse to the input string. 21.6 \u2022 S EMANTIC ROLE LABELING 469\nFigure 21.5 shows a parse of (21.28) above. The parse is then traversed to \ufb01nd all\nwords that are predicates.\nFor each of these predicates, the algorithm examines each node in the parse\ntree and uses supervised classi\ufb01cation to decide the semantic role (if any) it plays\nfor this predicate. Given a labeled training set such as PropBank or FrameNet, a\nfeature vector is extracted for each node, using feature templates described in the\nnext subsection. A 1-of-N classi\ufb01er is then trained to predict a semantic role for\neach constituent given these features, where N is the number of potential semantic\nroles plus an extra NONE role for non-role constituents. Any standard classi\ufb01cation\nalgorithms can be used. Finally, for each test sentence to be labeled, the classi\ufb01er is\nrun on each relevant constituent.\nfunction SEMANTIC ROLELABEL (words )returns labeled tree\nparse PARSE (words )\nfor each predicate inparse do\nfor each node inparse do\nfeaturevector EXTRACT FEATURES (node ,predicate ,parse )\nCLASSIFY NODE(node ,featurevector ,parse )\nFigure 21.4 A generic semantic-role-labeling algorithm. C LASSIFY NODE is a 1-of- Nclas-\nsi\ufb01er that assigns a semantic role (or NONE for non-role constituents), trained on labeled data\nsuch as FrameNet or PropBank.\nS\nNP-SBJ =A R G 0 VP\nDT NNP NNP NNP\nThe San Francisco Examiner\nVBD = TARGET NP=A R G 1 PP-TMP =A R G M - T M P\nissued DT JJ NN IN NP\nas p e c i a l e d i t i o n a r o u n d N N N P - T M P\nnoon yesterday\nFigure 21.5 Parse tree for a PropBank sentence, showing the PropBank argument labels. The dotted line\nshows the path feature NP\"S#VP#VBD for ARG0, the NP-SBJ constituent The San Francisco Examiner.\nInstead of training a single-stage classi\ufb01er as in Fig. 21.5, the node-level classi-\n\ufb01cation task can be broken down into multiple steps:\n1.Pruning: Since only a small number of the constituents in a sentence are\narguments of any given predicate, many systems use simple heuristics to prune\nunlikely constituents.\n2.Identi\ufb01cation: a binary classi\ufb01cation of each node as an argument to be la-\nbeled or a NONE .\n3.Classi\ufb01cation: a 1-of- Nclassi\ufb01cation of all the constituents that were labeled\nas arguments by the previous stage 470 CHAPTER 21 \u2022 S EMANTIC ROLE LABELING\nThe separation of identi\ufb01cation and classi\ufb01cation may lead to better use of fea-\ntures (different features may be useful for the two tasks) or to computational ef\ufb01-\nciency.\nGlobal Optimization\nThe classi\ufb01cation algorithm of Fig. 21.5 classi\ufb01es each argument separately (\u2018lo-\ncally\u2019), making the simplifying assumption that each argument of a predicate can be\nlabeled independently. This assumption is false; there are interactions between argu-\nments that require a more \u2018global\u2019 assignment of labels to constituents. For example,\nconstituents in FrameNet and PropBank are required to be non-overlapping. More\nsigni\ufb01cantly, the semantic roles of constituents are not independent. For example\nPropBank does not allow multiple identical arguments; two constituents of the same\nverb cannot both be labeled ARG0 .\nRole labeling systems thus often add a fourth step to deal with global consistency\nacross the labels in a sentence. For example, the local classi\ufb01ers can return a list of\npossible labels associated with probabilities for each constituent, and a second-pass\nViterbi decoding or re-ranking approach can be used to choose the best consensus\nlabel. Integer linear programming (ILP) is another common way to choose a solution\nthat conforms best to multiple constraints.\nFeatures for Semantic Role Labeling\nMost systems use some generalization of the core set of features introduced by\nGildea and Jurafsky (2000). Common basic features templates (demonstrated on\ntheNP-SBJ constituent The San Francisco Examiner in Fig. 21.5) include:\n\u2022 The governing predicate , in this case the verb issued . The predicate is a cru-\ncial feature since labels are de\ufb01ned only with respect to a particular predicate.\n\u2022 The phrase type of the constituent, in this case, NP(orNP-SBJ ). Some se-\nmantic roles tend to appear as NPs, others as SorPP, and so on.\n\u2022 The headword of the constituent, Examiner . The headword of a constituent\ncan be computed with standard head rules, such as those given in Appendix D\nin Fig. 18.17. Certain headwords (e.g., pronouns) place strong constraints on\nthe possible semantic roles they are likely to \ufb01ll.\n\u2022 The headword part of speech of the constituent, NNP .\n\u2022 The path in the parse tree from the constituent to the predicate. This path is\nmarked by the dotted line in Fig. 21.5. Following Gildea and Jurafsky (2000),\nwe can use a simple linear representation of the path, NP \"S#VP#VBD.\"and\n#represent upward and downward movement in the tree, respectively. The\npath is very useful as a compact representation of many kinds of grammatical\nfunction relationships between the constituent and the predicate.\n\u2022 The voice of the clause in which the constituent appears, in this case, active\n(as contrasted with passive ). Passive sentences tend to have strongly different\nlinkings of semantic roles to surface form than do active ones.\n\u2022 The binary linear position of the constituent with respect to the predicate,\neither before orafter .\n\u2022 The subcategorization of the predicate, the set of expected arguments that\nappear in the verb phrase. We can extract this information by using the phrase-\nstructure rule that expands the immediate parent of the predicate; VP !VBD\nNP PP for the predicate in Fig. 21.5.\n\u2022 The named entity type of the constituent. 21.6 \u2022 S EMANTIC ROLE LABELING 471\n\u2022 The \ufb01rst words and the last word of the constituent.\nThe following feature vector thus represents the \ufb01rst NP in our example (recall\nthat most observations will have the value NONE rather than, for example, ARG0,\nsince most constituents in the parse tree will not bear a semantic role):\nARG0: [issued, NP, Examiner, NNP, NP \"S#VP#VBD, active, before, VP !NP PP,\nORG, The, Examiner]\nOther features are often used in addition, such as sets of n-grams inside the\nconstituent, or more complex versions of the path features (the upward or downward\nhalves, or whether particular nodes occur in the path).\nIt\u2019s also possible to use dependency parses instead of constituency parses as the\nbasis of features, for example using dependency parse paths instead of constituency\npaths.\n21.6.2 A Neural Algorithm for Semantic Role Labeling\nA simple neural approach to SRL is to treat it as a sequence labeling task like named-\nentity recognition, using the BIO approach. Let\u2019s assume that we are given the\npredicate and the task is just detecting and labeling spans. Recall that with BIO\ntagging, we have a begin and end tag for each possible role ( B-ARG0,I-ARG0;B-\nARG1,I-ARG1, and so on), plus an outside tag O.\nENCODER[CLS]thecatslovehats[SEP]love[SEP]FFNB-ARG0I-ARG0B-PREDconcatenate with predicateB-ARG1FFNSoftmaxFFNFFNFFN\nFigure 21.6 A simple neural approach to semantic role labeling. The input sentence is\nfollowed by [SEP] and an extra input for the predicate, in this case love. The encoder outputs\nare concatenated to an indicator variable which is 1 for the predicate and 0 for all other words\nAfter He et al. (2017) and Shi and Lin (2019).\nAs with all the taggers, the goal is to compute the highest probability tag se-\nquence \u02c6 y, given the input sequence of words w:\n\u02c6y=argmax\ny2TP(yjw)\nFig. 21.6 shows a sketch of a standard algorithm from He et al. (2017). Here each\ninput word is mapped to pretrained embeddings, and then each token is concatenated\nwith the predicate embedding and then passed through a feedforward network with\na softmax which outputs a distribution over each SRL label. For decoding, a CRF\nlayer can be used instead of the MLP layer on top of the biLSTM output to do global\ninference, but in practice this doesn\u2019t seem to provide much bene\ufb01t. 472 CHAPTER 21 \u2022 S EMANTIC ROLE LABELING\n21.6.3 Evaluation of Semantic Role Labeling\nThe standard evaluation for semantic role labeling is to require that each argument\nlabel must be assigned to the exactly correct word sequence or parse constituent, and\nthen compute precision, recall, and F-measure. Identi\ufb01cation and classi\ufb01cation can\nalso be evaluated separately. Two common datasets used for evaluation are CoNLL-\n2005 (Carreras and M `arquez, 2005) and CoNLL-2012 (Pradhan et al., 2013).\n21.7 Selectional Restrictions\nWe turn in this section to another way to represent facts about the relationship be-\ntween predicates and arguments. A selectional restriction is a semantic type con-selectional\nrestriction\nstraint that a verb imposes on the kind of concepts that are allowed to \ufb01ll its argument\nroles. Consider the two meanings associated with the following example:\n(21.29) I want to eat someplace nearby.\nThere are two possible parses and semantic interpretations for this sentence. In\nthe sensible interpretation, eatis intransitive and the phrase someplace nearby is\nan adjunct that gives the location of the eating event. In the nonsensical speaker-as-\nGodzilla interpretation, eatis transitive and the phrase someplace nearby is the direct\nobject and the THEME of the eating, like the NP Malaysian food in the following\nsentences:\n(21.30) I want to eat Malaysian food.\nHow do we know that someplace nearby isn\u2019t the direct object in this sentence?\nOne useful cue is the semantic fact that the THEME of E ATING events tends to be\nsomething that is edible . This restriction placed by the verb eaton the \ufb01ller of its\nTHEME argument is a selectional restriction.\nSelectional restrictions are associated with senses, not entire lexemes. We can\nsee this in the following examples of the lexeme serve :\n(21.31) The restaurant serves green-lipped mussels.\n(21.32) Which airlines serve Denver?\nExample (21.31) illustrates the offering-food sense of serve , which ordinarily re-\nstricts its THEME to be some kind of food Example (21.32) illustrates the provides a\ncommercial service to sense of serve , which constrains its THEME to be some type\nof appropriate location.\nSelectional restrictions vary widely in their speci\ufb01city. The verb imagine , for\nexample, imposes strict requirements on its AGENT role (restricting it to humans\nand other animate entities) but places very few semantic requirements on its THEME\nrole. A verb like diagonalize , on the other hand, places a very speci\ufb01c constraint\non the \ufb01ller of its THEME role: it has to be a matrix, while the arguments of the\nadjective odorless are restricted to concepts that could possess an odor:\n(21.33) In rehearsal, I often ask the musicians to imagine a tennis game.\n(21.34) Radon is an odorless gas that can\u2019t be detected by human senses.\n(21.35) To diagonalize a matrix is to \ufb01nd its eigenvalues.\nThese examples illustrate that the set of concepts we need to represent selectional\nrestrictions (being a matrix, being able to possess an odor, etc) is quite open ended.\nThis distinguishes selectional restrictions from other features for representing lexical\nknowledge, like parts-of-speech, which are quite limited in number. 21.7 \u2022 S ELECTIONAL RESTRICTIONS 473\n21.7.1 Representing Selectional Restrictions\nOne way to capture the semantics of selectional restrictions is to use and extend the\nevent representation of Appendix F. Recall that the neo-Davidsonian representation\nof an event consists of a single variable that stands for the event, a predicate denoting\nthe kind of event, and variables and relations for the event roles. Ignoring the issue of\nthel-structures and using thematic roles rather than deep event roles, the semantic\ncontribution of a verb like eatmight look like the following:\n9e;x;y Eating (e)^Agent (e;x)^T heme (e;y)\nWith this representation, all we know about y, the \ufb01ller of the THEME role, is that\nit is associated with an Eating event through the Theme relation. To stipulate the\nselectional restriction that ymust be something edible, we simply add a new term to\nthat effect:\n9e;x;y Eating (e)^Agent (e;x)^T heme (e;y)^EdibleT hing (y)\nWhen a phrase like ate a hamburger is encountered, a semantic analyzer can form\nthe following kind of representation:\n9e;x;y Eating (e)^Eater (e;x)^T heme (e;y)^EdibleT hing (y)^Hamburger (y)\nThis representation is perfectly reasonable since the membership of yin the category\nHamburger is consistent with its membership in the category EdibleThing , assuming\na reasonable set of facts in the knowledge base. Correspondingly, the representation\nfor a phrase such as ate a takeoff would be ill-formed because membership in an\nevent-like category such as Takeoff would be inconsistent with membership in the\ncategory EdibleThing .\nWhile this approach adequately captures the semantics of selectional restrictions,\nthere are two problems with its direct use. First, using FOL to perform the simple\ntask of enforcing selectional restrictions is overkill. Other, far simpler, formalisms\ncan do the job with far less computational cost. The second problem is that this\napproach presupposes a large, logical knowledge base of facts about the concepts\nthat make up selectional restrictions. Unfortunately, although such common-sense\nknowledge bases are being developed, none currently have the kind of coverage\nnecessary to the task.\nA more practical approach is to state selectional restrictions in terms of WordNet\nsynsets rather than as logical concepts. Each predicate simply speci\ufb01es a WordNet\nsynset as the selectional restriction on each of its arguments. A meaning representa-\ntion is well-formed if the role \ufb01ller word is a hyponym (subordinate) of this synset.\nFor our ate a hamburger example, for instance, we could set the selectional\nrestriction on the THEME role of the verb eatto the synset ffood, nutrient g, glossed\nasany substance that can be metabolized by an animal to give energy and build\ntissue . Luckily, the chain of hypernyms for hamburger shown in Fig. 21.7 reveals\nthat hamburgers are indeed food. Again, the \ufb01ller of a role need not match the\nrestriction synset exactly; it just needs to have the synset as one of its superordinates.\nWe can apply this approach to the THEME roles of the verbs imagine ,lift, and di-\nagonalize , discussed earlier. Let us restrict imagine \u2019sTHEME to the synsetfentityg,\nlift\u2019sTHEME tofphysical entityg, and diagonalize tofmatrixg. This arrangement\ncorrectly permits imagine a hamburger andlift a hamburger , while also correctly\nruling out diagonalize a hamburger . 474 CHAPTER 21 \u2022 S EMANTIC ROLE LABELING\nSense 1\nhamburger, beefburger --\n(a fried cake of minced beef served on a bun)\n=> sandwich\n=> snack food\n=> dish\n=> nutriment, nourishment, nutrition...\n=> food, nutrient\n=> substance\n=> matter\n=> physical entity\n=> entity\nFigure 21.7 Evidence from WordNet that hamburgers are edible.\n21.7.2 Selectional Preferences\nIn the earliest implementations, selectional restrictions were considered strict con-\nstraints on the kind of arguments a predicate could take (Katz and Fodor 1963,\nHirst 1987). For example, the verb eatmight require that its THEME argument be\n[+FOOD] . Early word sense disambiguation systems used this idea to rule out senses\nthat violated the selectional restrictions of their governing predicates.\nVery quickly, however, it became clear that these selectional restrictions were\nbetter represented as preferences rather than strict constraints (Wilks 1975b, Wilks\n1975a). For example, selectional restriction violations (like inedible arguments of\neat) often occur in well-formed sentences, for example because they are negated\n(21.36), or because selectional restrictions are overstated (21.37):\n(21.36) But it fell apart in 1931, perhaps because people realized you can\u2019t eat\ngold for lunch if you\u2019re hungry.\n(21.37) In his two championship trials, Mr. Kulkarni ateglass on an empty\nstomach, accompanied only by water and tea.\nModern systems for selectional preferences therefore specify the relation be-\ntween a predicate and its possible arguments with soft constraints of some kind.\nSelectional Association\nOne of the most in\ufb02uential has been the selectional association model of Resnik\n(1993). Resnik de\ufb01nes the idea of selectional preference strength as the generalselectional\npreference\nstrengthamount of information that a predicate tells us about the semantic class of its argu-\nments. For example, the verb eattells us a lot about the semantic class of its direct\nobjects, since they tend to be edible. The verb be, by contrast, tells us less about\nits direct objects. The selectional preference strength can be de\ufb01ned by the differ-\nence in information between two distributions: the distribution of expected semantic\nclasses P(c)(how likely is it that a direct object will fall into class c) and the dis-\ntribution of expected semantic classes for the particular verb P(cjv)(how likely is\nit that the direct object of the speci\ufb01c verb vwill fall into semantic class c). The\ngreater the difference between these distributions, the more information the verb\nis giving us about possible objects. The difference between these two distributions\ncan be quanti\ufb01ed by relative entropy , or the Kullback-Leibler divergence (Kullback relative entropy\nand Leibler, 1951). The Kullback-Leibler or KL divergence D(PjjQ)expresses the KL divergence 21.7 \u2022 S ELECTIONAL RESTRICTIONS 475\ndifference between two probability distributions PandQ\nD(PjjQ) =X\nxP(x)logP(x)\nQ(x)(21.38)\nThe selectional preference SR(v)uses the KL divergence to express how much in-\nformation, in bits, the verb vexpresses about the possible semantic class of its argu-\nment.\nSR(v) = D(P(cjv)jjP(c))\n=X\ncP(cjv)logP(cjv)\nP(c)(21.39)\nResnik then de\ufb01nes the selectional association of a particular class and verb as theselectional\nassociation\nrelative contribution of that class to the general selectional preference of the verb:\nAR(v;c) =1\nSR(v)P(cjv)logP(cjv)\nP(c)(21.40)\nThe selectional association is thus a probabilistic measure of the strength of asso-\nciation between a predicate and a class dominating the argument to the predicate.\nResnik estimates the probabilities for these associations by parsing a corpus, count-\ning all the times each predicate occurs with each argument word, and assuming\nthat each word is a partial observation of all the WordNet concepts containing the\nword. The following table from Resnik (1996) shows some sample high and low\nselectional associations for verbs and some WordNet semantic classes of their direct\nobjects.\nDirect Object Direct Object\nVerb Semantic Class Assoc Semantic Class Assoc\nread WRITING 6.80 ACTIVITY -.20\nwrite WRITING 7.26 COMMERCE 0\nsee ENTITY 5.79 METHOD -0.01\nSelectional Preference via Conditional Probability\nAn alternative to using selectional association between a verb and the WordNet class\nof its arguments is to use the conditional probability of an argument word given a\npredicate verb, directly modeling the strength of association of one verb (predicate)\nwith one noun (argument).\nThe conditional probability model can be computed by parsing a very large cor-\npus (billions of words), and computing co-occurrence counts: how often a given\nverb occurs with a given noun in a given relation. The conditional probability of an\nargument noun given a verb for a particular relation P(njv;r)can then be used as a\nselectional preference metric for that pair of words (Brockmann and Lapata 2003,\nKeller and Lapata 2003):\nP(njv;r) =(\nC(n;v;r)\nC(v;r)ifC(n;v;r)>0\n0 otherwise\nThe inverse probability P(vjn;r)was found to have better performance in some cases\n(Brockmann and Lapata, 2003):\nP(vjn;r) =(\nC(n;v;r)\nC(n;r)ifC(n;v;r)>0\n0 otherwise 476 CHAPTER 21 \u2022 S EMANTIC ROLE LABELING\nAn even simpler approach is to use the simple log co-occurrence frequency of\nthe predicate with the argument log count (v;n;r)instead of conditional probability;\nthis seems to do better for extracting preferences for syntactic subjects rather than\nobjects (Brockmann and Lapata, 2003).\nEvaluating Selectional Preferences\nOne way to evaluate models of selectional preferences is to use pseudowords (Gale pseudowords\net al. 1992b, Sch \u00a8utze 1992a). A pseudoword is an arti\ufb01cial word created by concate-\nnating a test word in some context (say banana ) with a confounder word (say door )\nto create banana-door ). The task of the system is to identify which of the two words\nis the original word. To evaluate a selectional preference model (for example on the\nrelationship between a verb and a direct object) we take a test corpus and select all\nverb tokens. For each verb token (say drive ) we select the direct object (e.g., car),\nconcatenated with a confounder word that is its nearest neighbor , the noun with the\nfrequency closest to the original (say house ), to make car/house ). We then use the\nselectional preference model to choose which of carandhouse are more preferred\nobjects of drive , and compute how often the model chooses the correct original ob-\nject (e.g., car) (Chambers and Jurafsky, 2010).\nAnother evaluation metric is to get human preferences for a test set of verb-\nargument pairs, and have them rate their degree of plausibility. This is usually done\nby using magnitude estimation, a technique from psychophysics, in which subjects\nrate the plausibility of an argument proportional to a modulus item. A selectional\npreference model can then be evaluated by its correlation with the human prefer-\nences (Keller and Lapata, 2003).\n21.8 Primitive Decomposition of Predicates\nOne way of thinking about the semantic roles we have discussed through the chapter\nis that they help us de\ufb01ne the roles that arguments play in a decompositional way,\nbased on \ufb01nite lists of thematic roles (agent, patient, instrument, proto-agent, proto-\npatient, etc.). This idea of decomposing meaning into sets of primitive semantic\nelements or features, called primitive decomposition orcomponential analysis ,componential\nanalysis\nhas been taken even further, and focused particularly on predicates.\nConsider these examples of the verb kill:\n(21.41) Jim killed his philodendron.\n(21.42) Jim did something to cause his philodendron to become not alive.\nThere is a truth-conditional (\u2018propositional semantics\u2019) perspective from which these\ntwo sentences have the same meaning. Assuming this equivalence, we could repre-\nsent the meaning of killas:\n(21.43) KILL (x,y),CAUSE (x,BECOME (NOT(ALIVE (y))))\nthus using semantic primitives like do,cause ,become not , and alive .\nIndeed, one such set of potential semantic primitives has been used to account\nfor some of the verbal alternations discussed in Section 21.2 (Lakoff 1965, Dowty\n1979). Consider the following examples.\n(21.44) John opened the door. )CAUSE (John, BECOME (OPEN (door)))\n(21.45) The door opened. )BECOME (OPEN (door)) 21.9 \u2022 S UMMARY 477\n(21.46) The door is open. )OPEN (door)\nThe decompositional approach asserts that a single state-like predicate associ-\nated with open underlies all of these examples. The differences among the meanings\nof these examples arises from the combination of this single predicate with the prim-\nitives CAUSE and BECOME .\nWhile this approach to primitive decomposition can explain the similarity be-\ntween states and actions or causative and non-causative predicates, it still relies on\nhaving a large number of predicates like open . More radical approaches choose to\nbreak down these predicates as well. One such approach to verbal predicate decom-\nposition that played a role in early natural language systems is conceptual depen-\ndency (CD), a set of ten primitive predicates, shown in Fig. 21.8.conceptual\ndependency\nPrimitive De\ufb01nition\nATRANS The abstract transfer of possession or control from one entity to\nanother\nPTRANS The physical transfer of an object from one location to another\nMTRANS The transfer of mental concepts between entities or within an\nentity\nMBUILD The creation of new information within an entity\nPROPEL The application of physical force to move an object\nMOVE The integral movement of a body part by an animal\nINGEST The taking in of a substance by an animal\nEXPEL The expulsion of something from an animal\nSPEAK The action of producing a sound\nATTEND The action of focusing a sense organ\nFigure 21.8 A set of conceptual dependency primitives.\nBelow is an example sentence along with its CDrepresentation. The verb brought\nis translated into the two primitives ATRANS and PTRANS to indicate that the waiter\nboth physically conveyed the check to Mary and passed control of it to her. Note\nthat CDalso associates a \ufb01xed set of thematic roles with each primitive to represent\nthe various participants in the action.\n(21.47) The waiter brought Mary the check.\n9x;y Atrans (x)^Actor (x;Waiter )^Ob ject (x;Check )^To(x;Mary )\n^Ptrans (y)^Actor (y;Waiter )^Ob ject (y;Check )^To(y;Mary )\n21.9 Summary\n\u2022Semantic roles are abstract models of the role an argument plays in the event\ndescribed by the predicate.\n\u2022Thematic roles are a model of semantic roles based on a single \ufb01nite list of\nroles. Other semantic role models include per-verb semantic role lists and\nproto-agent /proto-patient , both of which are implemented in PropBank ,\nand per-frame role lists, implemented in FrameNet . 478 CHAPTER 21 \u2022 S EMANTIC ROLE LABELING\n\u2022Semantic role labeling is the task of assigning semantic role labels to the\nconstituents of a sentence. The task is generally treated as a supervised ma-\nchine learning task, with models trained on PropBank or FrameNet. Algo-\nrithms generally start by parsing a sentence and then automatically tag each\nparse tree node with a semantic role. Neural models map straight from words\nend-to-end.\n\u2022 Semantic selectional restrictions allow words (particularly predicates) to post\nconstraints on the semantic properties of their argument words. Selectional\npreference models (like selectional association or simple conditional proba-\nbility) allow a weight or probability to be assigned to the association between\na predicate and an argument word or class.\nBibliographical and Historical Notes\nAlthough the idea of semantic roles dates back to P \u00afan.ini, they were re-introduced\ninto modern linguistics by Gruber (1965), Fillmore (1966) and Fillmore (1968). Fill-\nmore had become interested in argument structure by studying Lucien Tesni `ere\u2019s\ngroundbreaking \u00b4El\u00b4ements de Syntaxe Structurale (Tesni `ere, 1959) in which the term\n\u2018dependency\u2019 was introduced and the foundations were laid for dependency gram-\nmar. Following Tesni `ere\u2019s terminology, Fillmore \ufb01rst referred to argument roles as\nactants (Fillmore, 1966) but quickly switched to the term case, (see Fillmore (2003))\nand proposed a universal list of semantic roles or cases (Agent, Patient, Instrument,\netc.), that could be taken on by the arguments of predicates. Verbs would be listed in\nthe lexicon with their case frame , the list of obligatory (or optional) case arguments.\nThe idea that semantic roles could provide an intermediate level of semantic\nrepresentation that could help map from syntactic parse structures to deeper, more\nfully-speci\ufb01ed representations of meaning was quickly adopted in natural language\nprocessing, and systems for extracting case frames were created for machine transla-\ntion (Wilks, 1973), question-answering (Hendrix et al., 1973), spoken-language pro-\ncessing (Nash-Webber, 1975), and dialogue systems (Bobrow et al., 1977). General-\npurpose semantic role labelers were developed. The earliest ones (Simmons, 1973)\n\ufb01rst parsed a sentence by means of an ATN (Augmented Transition Network) parser.\nEach verb then had a set of rules specifying how the parse should be mapped to se-\nmantic roles. These rules mainly made reference to grammatical functions (subject,\nobject, complement of speci\ufb01c prepositions) but also checked constituent internal\nfeatures such as the animacy of head nouns. Later systems assigned roles from pre-\nbuilt parse trees, again by using dictionaries with verb-speci\ufb01c case frames (Levin\n1977, Marcus 1980).\nBy 1977 case representation was widely used and taught in AI and NLP courses,\nand was described as a standard of natural language processing in the \ufb01rst edition of\nWinston\u2019s 1977 textbook Arti\ufb01cial Intelligence .\nIn the 1980s Fillmore proposed his model of frame semantics , later describing\nthe intuition as follows:\n\u201cThe idea behind frame semantics is that speakers are aware of possi-\nbly quite complex situation types, packages of connected expectations,\nthat go by various names\u2014frames, schemas, scenarios, scripts, cultural\nnarratives, memes\u2014and the words in our language are understood with\nsuch frames as their presupposed background.\u201d (Fillmore, 2012, p. 712) BIBLIOGRAPHICAL AND HISTORICAL NOTES 479\nThe word frame seemed to be in the air for a suite of related notions proposed at\nabout the same time by Minsky (1974), Hymes (1974), and Goffman (1974), as\nwell as related notions with other names like scripts (Schank and Abelson, 1975)\nandschemata (Bobrow and Norman, 1975) (see Tannen (1979) for a comparison).\nFillmore was also in\ufb02uenced by the semantic \ufb01eld theorists and by a visit to the Yale\nAI lab where he took notice of the lists of slots and \ufb01llers used by early information\nextraction systems like DeJong (1982) and Schank and Abelson (1977). In the 1990s\nFillmore drew on these insights to begin the FrameNet corpus annotation project.\nAt the same time, Beth Levin drew on her early case frame dictionaries (Levin,\n1977) to develop her book which summarized sets of verb classes de\ufb01ned by shared\nargument realizations (Levin, 1993). The VerbNet project built on this work (Kipper\net al., 2000), leading soon afterwards to the PropBank semantic-role-labeled corpus\ncreated by Martha Palmer and colleagues (Palmer et al., 2005).\nThe combination of rich linguistic annotation and corpus-based approach in-\nstantiated in FrameNet and PropBank led to a revival of automatic approaches to\nsemantic role labeling, \ufb01rst on FrameNet (Gildea and Jurafsky, 2000) and then on\nPropBank data (Gildea and Palmer, 2002, inter alia). The problem \ufb01rst addressed in\nthe 1970s by handwritten rules was thus now generally recast as one of supervised\nmachine learning enabled by large and consistent databases. Many popular features\nused for role labeling are de\ufb01ned in Gildea and Jurafsky (2002), Surdeanu et al.\n(2003), Xue and Palmer (2004), Pradhan et al. (2005), Che et al. (2009), and Zhao\net al. (2009). The use of dependency rather than constituency parses was introduced\nin the CoNLL-2008 shared task (Surdeanu et al., 2008). For surveys see Palmer\net al. (2010) and M `arquez et al. (2008).\nThe use of neural approaches to semantic role labeling was pioneered by Col-\nlobert et al. (2011), who applied a CRF on top of a convolutional net. Early work\nlike Foland, Jr. and Martin (2015) focused on using dependency features. Later work\neschewed syntactic features altogether; Zhou and Xu (2015b) introduced the use of\na stacked (6-8 layer) biLSTM architecture, and (He et al., 2017) showed how to\naugment the biLSTM architecture with highway networks and also replace the CRF\nwith A* decoding that make it possible to apply a wide variety of global constraints\nin SRL decoding.\nMost semantic role labeling schemes only work within a single sentence, fo-\ncusing on the object of the verbal (or nominal, in the case of NomBank) predicate.\nHowever, in many cases, a verbal or nominal predicate may have an implicit argu-\nment : one that appears only in a contextual sentence, or perhaps not at all and mustimplicit\nargument\nbe inferred. In the two sentences This house has a new owner. The sale was \ufb01nalized\n10 days ago. thesale in the second sentence has no A RG1, but a reasonable reader\nwould infer that the Arg1 should be the house mentioned in the prior sentence. Find-\ning these arguments, implicit argument detection (sometimes shortened as iSRL ) iSRL\nwas introduced by Gerber and Chai (2010) and Ruppenhofer et al. (2010). See Do\net al. (2017) for more recent neural models.\nTo avoid the need for huge labeled training sets, unsupervised approaches for\nsemantic role labeling attempt to induce the set of semantic roles by clustering over\narguments. The task was pioneered by Riloff and Schmelzenbach (1998) and Swier\nand Stevenson (2004); see Grenager and Manning (2006), Titov and Klementiev\n(2012), Lang and Lapata (2014), Woodsend and Lapata (2015), and Titov and Khod-\ndam (2014).\nRecent innovations in frame labeling include connotation frames , which mark\nricher information about the argument of predicates. Connotation frames mark the 480 CHAPTER 21 \u2022 S EMANTIC ROLE LABELING\nsentiment of the writer or reader toward the arguments (for example using the verb\nsurvive inhe survived a bombing expresses the writer\u2019s sympathy toward the subject\nheand negative sentiment toward the bombing. See Chapter 22 for more details.\nSelectional preference has been widely studied beyond the selectional associa-\ntion models of Resnik (1993) and Resnik (1996). Methods have included clustering\n(Rooth et al., 1999), discriminative learning (Bergsma et al., 2008a), and topic mod-\nels (S \u00b4eaghdha 2010, Ritter et al. 2010b), and constraints can be expressed at the level\nof words or classes (Agirre and Martinez, 2001). Selectional preferences have also\nbeen successfully integrated into semantic role labeling (Erk 2007, Zapirain et al.\n2013, Do et al. 2017).\nExercises CHAPTER\n22Lexicons for Sentiment, Affect,\nand Connotation\nSome day we\u2019ll be able to measure the power of words\nMaya Angelou\nIn this chapter we turn to tools for interpreting affective meaning, extending our affective\nstudy of sentiment analysis in Chapter 4. We use the word \u2018affective\u2019, following the\ntradition in affective computing (Picard, 1995) to mean emotion, sentiment, per-\nsonality, mood, and attitudes. Affective meaning is closely related to subjectivity , subjectivity\nthe study of a speaker or writer\u2019s evaluations, opinions, emotions, and speculations\n(Wiebe et al., 1999).\nHow should affective meaning be de\ufb01ned? One in\ufb02uential typology of affec-\ntive states comes from Scherer (2000), who de\ufb01nes each class of affective states by\nfactors like its cognitive realization and time course (Fig. 22.1).\nEmotion: Relatively brief episode of response to the evaluation of an external\nor internal event as being of major signi\ufb01cance.\n(angry, sad, joyful, fearful, ashamed, proud, elated, desperate )\nMood: Diffuse affect state, most pronounced as change in subjective feeling, of\nlow intensity but relatively long duration, often without apparent cause.\n(cheerful, gloomy, irritable, listless, depressed, buoyant )\nInterpersonal stance: Affective stance taken toward another person in a spe-\nci\ufb01c interaction, coloring the interpersonal exchange in that situation.\n(distant, cold, warm, supportive, contemptuous, friendly )\nAttitude: Relatively enduring, affectively colored beliefs, preferences, and pre-\ndispositions towards objects or persons.\n(liking, loving, hating, valuing, desiring )\nPersonality traits: Emotionally laden, stable personality dispositions and be-\nhavior tendencies, typical for a person.\n(nervous, anxious, reckless, morose, hostile, jealous )\nFigure 22.1 The Scherer typology of affective states (Scherer, 2000).\nWe can design extractors for each of these kinds of affective states. Chapter 4\nalready introduced sentiment analysis , the task of extracting the positive or negative\norientation that a writer expresses in a text. This corresponds in Scherer\u2019s typology\nto the extraction of attitudes : \ufb01guring out what people like or dislike, from affect-\nrich texts like consumer reviews of books or movies, newspaper editorials, or public\nsentiment in blogs or tweets.\nDetecting emotion andmoods is useful for detecting whether a student is con-\nfused, engaged, or certain when interacting with a tutorial system, whether a caller\nto a help line is frustrated, whether someone\u2019s blog posts or tweets indicated depres-\nsion. Detecting emotions like fear in novels, for example, could help us trace what\ngroups or situations are feared and how that changes over time. 482 CHAPTER 22 \u2022 L EXICONS FOR SENTIMENT , AFFECT ,AND CONNOTATION\nDetecting different interpersonal stances can be useful when extracting infor-\nmation from human-human conversations. The goal here is to detect stances like\nfriendliness or awkwardness in interviews or friendly conversations, for example for\nsummarizing meetings or \ufb01nding parts of a conversation where people are especially\nexcited or engaged, conversational hot spots that can help in meeting summariza-\ntion. Detecting the personality of a user\u2014such as whether the user is an extrovert\nor the extent to which they are open to experience \u2014 can help improve conversa-\ntional agents, which seem to work better if they match users\u2019 personality expecta-\ntions (Mairesse and Walker, 2008). And affect is important for generation as well\nas recognition; synthesizing affect is important for conversational agents in various\ndomains, including literacy tutors such as children\u2019s storybooks, or computer games.\nIn Chapter 4 we introduced the use of naive Bayes classi\ufb01cation to classify a\ndocument\u2019s sentiment. Various classi\ufb01ers have been successfully applied to many of\nthese tasks, using all the words in the training set as input to a classi\ufb01er which then\ndetermines the affect status of the text.\nIn this chapter we focus on an alternative model, in which instead of using every\nword as a feature, we focus only on certain words, ones that carry particularly strong\ncues to affect or sentiment. We call these lists of words affective lexicons orsenti-\nment lexicons . These lexicons presuppose a fact about semantics: that words have\naffective meanings orconnotations . The word connotation has different meanings connotations\nin different \ufb01elds, but here we use it to mean the aspects of a word\u2019s meaning that\nare related to a writer or reader\u2019s emotions, sentiment, opinions, or evaluations. In\naddition to their ability to help determine the affective status of a text, connotation\nlexicons can be useful features for other kinds of affective tasks, and for computa-\ntional social science analysis.\nIn the next sections we introduce basic theories of emotion, show how sentiment\nlexicons are a special case of emotion lexicons, and mention some useful lexicons.\nWe then survey three ways for building lexicons: human labeling, semi-supervised,\nand supervised. Finally, we talk about how to detect affect toward a particular entity,\nand introduce connotation frames.\n22.1 De\ufb01ning Emotion\nOne of the most important affective classes is emotion , which Scherer (2000) de\ufb01nes emotion\nas a \u201crelatively brief episode of response to the evaluation of an external or internal\nevent as being of major signi\ufb01cance\u201d.\nDetecting emotion has the potential to improve a number of language processing\ntasks. Emotion recognition could help dialogue systems like tutoring systems detect\nthat a student was unhappy, bored, hesitant, con\ufb01dent, and so on. Automatically\ndetecting emotions in reviews or customer responses (anger, dissatisfaction, trust)\ncould help businesses recognize speci\ufb01c problem areas or ones that are going well.\nEmotion can play a role in medical NLP tasks like helping diagnose depression or\nsuicidal intent. Detecting emotions expressed toward characters in novels might\nplay a role in understanding how different social groups were viewed by society at\ndifferent times.\nComputational models of emotion in NLP have mainly been based on two fami-\nlies of theories of emotion (out of the many studied in the \ufb01eld of affective science).\nIn one of these families, emotions are viewed as \ufb01xed atomic units, limited in num-\nber, and from which others are generated, often called basic emotions (Tomkins basic emotions 22.1 \u2022 D EFINING EMOTION 483\n1962, Plutchik 1962), a model dating back to Darwin. Perhaps the most well-known\nof this family of theories are the 6 emotions proposed by Ekman (e.g., Ekman 1999)\nto be universally present in all cultures: surprise, happiness, anger, fear, disgust,\nsadness . Another atomic theory is the Plutchik (1980) wheel of emotion, consisting\nof 8 basic emotions in four opposing pairs: joy\u2013sadness ,anger\u2013fear ,trust\u2013disgust ,\nandanticipation\u2013surprise , together with the emotions derived from them, shown in\nFig. 22.2.\nFigure 22.2 Plutchik wheel of emotion.\nThe second class of emotion theories widely used in NLP views emotion as a\nspace in 2 or 3 dimensions (Russell, 1980). Most models include the two dimensions\nvalence andarousal , and many add a third, dominance . These can be de\ufb01ned as:\nvalence: the pleasantness of the stimulus\narousal: the level of alertness, activeness, or energy provoked by the stimulus\ndominance: the degree of control or dominance exerted by the stimulus or the\nemotion\nSentiment can be viewed as a special case of this second view of emotions as points\nin space. In particular, the valence dimension, measuring how pleasant or unpleasant\na word is, is often used directly as a measure of sentiment.\nIn these lexicon-based models of affect, the affective meaning of a word is gen-\nerally \ufb01xed, irrespective of the linguistic context in which a word is used, or the\ndialect or culture of the speaker. By contrast, other models in affective science repre-\nsent emotions as much richer processes involving cognition (Barrett et al., 2007). In\nappraisal theory , for example, emotions are complex processes, in which a person\nconsiders how an event is congruent with their goals, taking into account variables\nlike the agency, certainty, urgency, novelty and control associated with the event\n(Moors et al., 2013). Computational models in NLP taking into account these richer\ntheories of emotion will likely play an important role in future work. 484 CHAPTER 22 \u2022 L EXICONS FOR SENTIMENT , AFFECT ,AND CONNOTATION\n22.2 Available Sentiment and Affect Lexicons\nA wide variety of affect lexicons have been created and released. The most basic\nlexicons label words along one dimension of semantic variability, generally called\n\u201csentiment\u201d or \u201cvalence\u201d.\nIn the simplest lexicons this dimension is represented in a binary fashion, with\na wordlist for positive words and a wordlist for negative words. The oldest is the\nGeneral Inquirer (Stone et al., 1966), which drew on content analysis and on earlyGeneral\nInquirer\nwork in the cognitive psychology of word meaning (Osgood et al., 1957). The Gen-\neral Inquirer has a lexicon of 1915 positive words and a lexicon of 2291 negative\nwords (as well as other lexicons discussed below). The MPQA Subjectivity lexicon\n(Wilson et al., 2005) has 2718 positive and 4912 negative words drawn from prior\nlexicons plus a bootstrapped list of subjective words and phrases (Riloff and Wiebe,\n2003). Each entry in the lexicon is hand-labeled for sentiment and also labeled for\nreliability (strongly subjective or weakly subjective). The polarity lexicon of Hu\nand Liu (2004b) gives 2006 positive and 4783 negative words, drawn from product\nreviews, labeled using a bootstrapping method from WordNet.\nPositive admire, amazing, assure, celebration, charm, eager, enthusiastic, excellent, fancy, fan-\ntastic, frolic, graceful, happy, joy, luck, majesty, mercy, nice, patience, perfect, proud,\nrejoice, relief, respect, satisfactorily, sensational, super, terri\ufb01c, thank, vivid, wise, won-\nderful, zest\nNegative abominable, anger, anxious, bad, catastrophe, cheap, complaint, condescending, deceit,\ndefective, disappointment, embarrass, fake, fear, \ufb01lthy, fool, guilt, hate, idiot, in\ufb02ict, lazy,\nmiserable, mourn, nervous, objection, pest, plot, reject, scream, silly, terrible, unfriendly,\nvile, wicked\nFigure 22.3 Some words with consistent sentiment across the General Inquirer (Stone et al., 1966), the\nMPQA Subjectivity lexicon (Wilson et al., 2005), and the polarity lexicon of Hu and Liu (2004b).\nSlightly more general than these sentiment lexicons are lexicons that assign each\nword a value on all three affective dimensions. The NRC Valence, Arousal, and\nDominance (V AD) lexicon (Mohammad, 2018a) assigns valence, arousal, and dom-\ninance scores to 20,000 words. Some examples are shown in Fig. 22.4.\nValence Arousal Dominance\nvacation .840 enraged .962 powerful .991\ndelightful .918 party .840 authority .935\nwhistle .653 organized .337 saxophone .482\nconsolation .408 effortless .120 discouraged .0090\ntorture .115 napping .046 weak .045\nFigure 22.4 Values of sample words on the emotional dimensions of Mohammad (2018a).\nThe NRC Word-Emotion Association Lexicon, also called EmoLex (Moham- EmoLex\nmad and Turney, 2013), uses the Plutchik (1980) 8 basic emotions de\ufb01ned above.\nThe lexicon includes around 14,000 words including words from prior lexicons as\nwell as frequent nouns, verbs, adverbs and adjectives. Values from the lexicon for\nsome sample words: 22.3 \u2022 C REATING AFFECT LEXICONS BY HUMAN LABELING 485\nWord\nanger\nanticipation\ndisgust\nfear\njoy\nsadness\nsurprise\ntrust\npositive\nnegative\nreward 0100101110\nworry 0101010001\ntenderness 0000100010\nsweetheart 0100110110\nsuddenly 0000001000\nthirst 0100011000\ngarbage 0010000001\nFor a smaller set of 5,814 words, the NRC Emotion/Affect Intensity Lexicon\n(Mohammad, 2018b) contains real-valued scores of association for anger, fear, joy,\nand sadness; Fig. 22.5 shows examples.\nAnger Fear Joy Sadness\noutraged 0.964 horror 0.923 superb 0.864 sad 0.844\nviolence 0.742 anguish 0.703 cheered 0.773 guilt 0.750\ncoup 0.578 pestilence 0.625 rainbow 0.531 unkind 0.547\noust 0.484 stressed 0.531 gesture 0.387 dif\ufb01culties 0.421\nsuspicious 0.484 failing 0.531 warms 0.391 beggar 0.422\nnurture 0.059 con\ufb01dent 0.094 hardship .031 sing 0.017\nFigure 22.5 Sample emotional intensities for words for anger, fear, joy, and sadness from\nMohammad (2018b).\nLIWC ,Linguistic Inquiry and Word Count , is a widely used set of 73 lex- LIWC\nicons containing over 2300 words (Pennebaker et al., 2007), designed to capture\naspects of lexical meaning relevant for social psychological tasks. In addition to\nsentiment-related lexicons like ones for negative emotion ( bad, weird, hate, prob-\nlem, tough ) and positive emotion ( love, nice, sweet ), LIWC includes lexicons for\ncategories like anger, sadness, cognitive mechanisms, perception, tentative, and in-\nhibition, shown in Fig. 22.6.\nThere are various other hand-built affective lexicons. The General Inquirer in-\ncludes additional lexicons for dimensions like strong vs. weak, active vs. passive,\noverstated vs. understated, as well as lexicons for categories like pleasure, pain,\nvirtue, vice, motivation, and cognitive orientation.\nAnother useful feature for various tasks is the distinction between concrete concrete\nwords like banana orbathrobe andabstract words like belief andalthough . The abstract\nlexicon in Brysbaert et al. (2014) used crowdsourcing to assign a rating from 1 to 5\nof the concreteness of 40,000 words, thus assigning banana ,bathrobe , and bagel 5,\nbelief 1.19, although 1.07, and in between words like brisk a 2.5.\n22.3 Creating Affect Lexicons by Human Labeling\nThe earliest method used to build affect lexicons, and still in common use, is to have\nhumans label each word. This is now most commonly done via crowdsourcing : crowdsourcing\nbreaking the task into small pieces and distributing them to a large number of anno- 486 CHAPTER 22 \u2022 L EXICONS FOR SENTIMENT , AFFECT ,AND CONNOTATION\nPositive Negative\nEmotion Emotion Insight Inhibition Family Negate\nappreciat* anger* aware* avoid* brother* aren\u2019t\ncomfort* bore* believe careful* cousin* cannot\ngreat cry decid* hesitat* daughter* didn\u2019t\nhappy despair* feel limit* family neither\ninterest fail* \ufb01gur* oppos* father* never\njoy* fear know prevent* grandf* no\nperfect* griev* knew reluctan* grandm* nobod*\nplease* hate* means safe* husband none\nsafe* panic* notice* stop mom nor\nterri\ufb01c suffers recogni* stubborn* mother nothing\nvalue terrify sense wait niece* nowhere\nwow* violent* think wary wife without\nFigure 22.6 Samples from 5 of the 73 lexical categories in LIWC (Pennebaker et al., 2007).\nThe * means the previous letters are a word pre\ufb01x and all words with that pre\ufb01x are included\nin the category.\ntators. Let\u2019s take a look at some of the methodological choices for two crowdsourced\nemotion lexicons.\nThe NRC Emotion Lexicon (EmoLex) (Mohammad and Turney, 2013), labeled\nemotions in two steps. To ensure that the annotators were judging the correct sense\nof the word, they \ufb01rst answered a multiple-choice synonym question that primed\nthe correct sense of the word (without requiring the annotator to read a potentially\nconfusing sense de\ufb01nition). These were created automatically using the headwords\nassociated with the thesaurus category of the sense in question in the Macquarie\ndictionary and the headwords of 3 random distractor categories. An example:\nWhich word is closest in meaning (most related) to startle?\n\u2022automobile\n\u2022shake\n\u2022honesty\n\u2022entertain\nFor each word (e.g. startle ), the annotator was then asked to rate how associated\nthat word is with each of the 8 emotions ( joy,fear,anger , etc.). The associations\nwere rated on a scale of not,weakly ,moderately , and strongly associated. Outlier\nratings were removed, and then each term was assigned the class chosen by the ma-\njority of the annotators, with ties broken by choosing the stronger intensity, and then\nthe 4 levels were mapped into a binary label for each word (no and weak mapped to\n0, moderate and strong mapped to 1).\nThe NRC V AD Lexicon (Mohammad, 2018a) was built by selecting words and\nemoticons from prior lexicons and annotating them with crowd-sourcing using best-\nworst scaling (Louviere et al. 2015, Kiritchenko and Mohammad 2017). In best-best-worst\nscaling\nworst scaling, annotators are given N items (usually 4) and are asked which item is\nthebest (highest) and which is the worst (lowest) in terms of some property. The\nset of words used to describe the ends of the scales are taken from prior literature.\nFor valence, for example, the raters were asked:\nQ1. Which of the four words below is associated with the MOST happi-\nness / pleasure / positiveness / satisfaction / contentedness / hopefulness\nOR LEAST unhappiness / annoyance / negativeness / dissatisfaction / 22.4 \u2022 S EMI-SUPERVISED INDUCTION OF AFFECT LEXICONS 487\nmelancholy / despair? (Four words listed as options.)\nQ2. Which of the four words below is associated with the LEAST hap-\npiness / pleasure / positiveness / satisfaction / contentedness / hopeful-\nness OR MOST unhappiness / annoyance / negativeness / dissatisfaction\n/ melancholy / despair? (Four words listed as options.)\nThe score for each word in the lexicon is the proportion of times the item was chosen\nas the best (highest V/A/D) minus the proportion of times the item was chosen as the\nworst (lowest V/A/D). The agreement between annotations are evaluated by split-\nhalf reliability : split the corpus in half and compute the correlations between thesplit-half\nreliability\nannotations in the two halves.\n22.4 Semi-supervised Induction of Affect Lexicons\nAnother common way to learn sentiment lexicons is to start from a set of seed words\nthat de\ufb01ne two poles of a semantic axis (words like good orbad), and then \ufb01nd ways\nto label each word wby its similarity to the two seed sets. Here we summarize two\nfamilies of seed-based semi-supervised lexicon induction algorithms, axis-based and\ngraph-based.\n22.4.1 Semantic Axis Methods\nOne of the most well-known lexicon induction methods, the Turney and Littman\n(2003) algorithm, is given seed words like good orbad, and then for each word wto\nbe labeled, measures both how similar it is to good and how different it is from bad.\nHere we describe a slight extension of the algorithm due to An et al. (2018), which\nis based on computing a semantic axis .\nIn the \ufb01rst step, we choose seed words by hand. There are two methods for\ndealing with the fact that the affect of a word is different in different contexts: (1)\nstart with a single large seed lexicon and rely on the induction algorithm to \ufb01ne-tune\nit to the domain, or (2) choose different seed words for different genres. Hellrich\net al. (2019) suggests that for modeling affect across different historical time periods,\nstarting with a large modern affect dictionary is better than small seedsets tuned to be\nstable across time. As an example of the second approach, Hamilton et al. (2016a)\nde\ufb01ne one set of seed words for general sentiment analysis, a different set for Twitter,\nand yet another set for sentiment in \ufb01nancial text:\nDomain Positive seeds Negative seeds\nGeneral good, lovely, excellent, fortunate, pleas-\nant, delightful, perfect, loved, love,\nhappybad, horrible, poor, unfortunate, un-\npleasant, disgusting, evil, hated, hate,\nunhappy\nTwitter love, loved, loves, awesome, nice,\namazing, best, fantastic, correct, happyhate, hated, hates, terrible, nasty, awful,\nworst, horrible, wrong, sad\nFinance successful, excellent, pro\ufb01t, bene\ufb01cial,\nimproving, improved, success, gains,\npositivenegligent, loss, volatile, wrong, losses,\ndamages, bad, litigation, failure, down,\nnegative\nIn the second step, we compute embeddings for each of the pole words. These\nembeddings can be off-the-shelf word2vec embeddings, or can be computed directly 488 CHAPTER 22 \u2022 L EXICONS FOR SENTIMENT , AFFECT ,AND CONNOTATION\non a speci\ufb01c corpus (for example using a \ufb01nancial corpus if a \ufb01nance lexicon is the\ngoal), or we can \ufb01ne-tune off-the-shelf embeddings to a corpus. Fine-tuning is espe-\ncially important if we have a very speci\ufb01c genre of text but don\u2019t have enough data\nto train good embeddings. In \ufb01ne-tuning, we begin with off-the-shelf embeddings\nlike word2vec, and continue training them on the small target corpus.\nOnce we have embeddings for each pole word, we create an embedding that\nrepresents each pole by taking the centroid of the embeddings of each of the seed\nwords; recall that the centroid is the multidimensional version of the mean. Given\na set of embeddings for the positive seed words S+=fE(w+\n1);E(w+\n2);:::;E(w+\nn)g,\nand embeddings for the negative seed words S\u0000=fE(w\u0000\n1);E(w\u0000\n2);:::;E(w\u0000\nm)g, the\npole centroids are:\nV+=1\nnnX\n1E(w+\ni)\nV\u0000=1\nmmX\n1E(w\u0000\ni) (22.1)\nThe semantic axis de\ufb01ned by the poles is computed just by subtracting the two vec-\ntors:\nVaxis=V+\u0000V\u0000(22.2)\nVaxis, the semantic axis, is a vector in the direction of positive sentiment. Finally,\nwe compute (via cosine similarity) the angle between the vector in the direction of\npositive sentiment and the direction of w\u2019s embedding. A higher cosine means that\nwis more aligned with S+than S\u0000.\nscore(w) = cos\u0000\nE(w);Vaxis\u0001\n=E(w)\u0001Vaxis\nkE(w)kkVaxisk(22.3)\nIf a dictionary of words with sentiment scores is suf\ufb01cient, we\u2019re done! Or if we\nneed to group words into a positive and a negative lexicon, we can use a threshold\nor other method to give us discrete lexicons.\n22.4.2 Label Propagation\nAn alternative family of methods de\ufb01nes lexicons by propagating sentiment labels\non graphs, an idea suggested in early work by Hatzivassiloglou and McKeown\n(1997). We\u2019ll describe the simple SentProp (Sentiment Propagation) algorithm of\nHamilton et al. (2016a), which has four steps:\n1.De\ufb01ne a graph : Given word embeddings, build a weighted lexical graph by\nconnecting each word with its knearest neighbors (according to cosine simi-\nlarity). The weights of the edge between words wiandwjare set as:\nEi;j=arccos\u0012\n\u0000wi>wj\nkwikkwjk\u0013\n: (22.4)\n2.De\ufb01ne a seed set: Choose positive and negative seed words.\n3.Propagate polarities from the seed set: Now we perform a random walk on\nthis graph, starting at the seed set. In a random walk, we start at a node and 22.4 \u2022 S EMI-SUPERVISED INDUCTION OF AFFECT LEXICONS 489\nthen choose a node to move to with probability proportional to the edge prob-\nability. A word\u2019s polarity score for a seed set is proportional to the probability\nof a random walk from the seed set landing on that word (Fig. 22.7).\n4.Create word scores : We walk from both positive and negative seed sets,\nresulting in positive (rawscore+(wi)) and negative (rawscore\u0000(wi)) raw label\nscores. We then combine these values into a positive-polarity score as:\nscore+(wi) =rawscore+(wi)\nrawscore+(wi)+rawscore\u0000(wi)(22.5)\nIt\u2019s often helpful to standardize the scores to have zero mean and unit variance\nwithin a corpus.\n5.Assign con\ufb01dence to each score: Because sentiment scores are in\ufb02uenced by\nthe seed set, we\u2019d like to know how much the score of a word would change if\na different seed set is used. We can use bootstrap sampling to get con\ufb01dence\nregions, by computing the propagation Btimes over random subsets of the\npositive and negative seed sets (for example using B=50 and choosing 7 of\nthe 10 seed words each time). The standard deviation of the bootstrap sampled\npolarity scores gives a con\ufb01dence measure.\nidolize\nlove\nadore\nappreciate\nlike\n\ufb01nd\ndislike\nsee\nnotice\ndisapprove\nabhor\nhate\nloathe\ndespise\nuncover\nidolize\nlove\nadore\nappreciate\nlike\n\ufb01nd\ndislike\nsee\nnotice\ndisapprove\nabhor\nhate\nloathe\ndespise\nuncover\n(a) (b)\nFigure 22.7 Intuition of the S ENTPROP algorithm. (a) Run random walks from the seed words. (b) Assign\npolarity scores (shown here as colors green or red) based on the frequency of random walk visits.\n22.4.3 Other Methods\nThe core of semisupervised algorithms is the metric for measuring similarity with\nthe seed words. The Turney and Littman (2003) and Hamilton et al. (2016a) ap-\nproaches above used embedding cosine as the distance metric: words were labeled\nas positive basically if their embeddings had high cosines with positive seeds and\nlow cosines with negative seeds. Other methods have chosen other kinds of distance\nmetrics besides embedding cosine.\nFor example the Hatzivassiloglou and McKeown (1997) algorithm uses syntactic\ncues; two adjectives are considered similar if they were frequently conjoined by and\nand rarely conjoined by but. This is based on the intuition that adjectives conjoined\nby the words andtend to have the same polarity; positive adjectives are generally\ncoordinated with positive, negative with negative:\nfair and legitimate, corrupt and brutal\nbut less often positive adjectives coordinated with negative:\n*fair and brutal, *corrupt and legitimate\nBy contrast, adjectives conjoined by butare likely to be of opposite polarity: 490 CHAPTER 22 \u2022 L EXICONS FOR SENTIMENT , AFFECT ,AND CONNOTATION\nfair but brutal\nAnother cue to opposite polarity comes from morphological negation ( un-,im-,\n-less). Adjectives with the same root but differing in a morphological negative ( ad-\nequate/inadequate ,thoughtful/thoughtless ) tend to be of opposite polarity.\nYet another method for \ufb01nding words that have a similar polarity to seed words is\nto make use of a thesaurus like WordNet (Kim and Hovy 2004, Hu and Liu 2004b).\nA word\u2019s synonyms presumably share its polarity while a word\u2019s antonyms probably\nhave the opposite polarity. After a seed lexicon is built, each lexicon is updated as\nfollows, possibly iterated.\nLex+: Add synonyms of positive words ( well) and antonyms (like \ufb01ne) of negative\nwords\nLex\u0000: Add synonyms of negative words ( awful ) and antonyms (like evil) of positive\nwords\nAn extension of this algorithm assigns polarity to WordNet senses, called Senti-\nWordNet (Baccianella et al., 2010). Fig. 22.8 shows some examples. SentiWordNet\nSynset Pos Neg Obj\ngood#6 \u2018agreeable or pleasing\u2019 1 0 0\nrespectable#2 honorable#4 good#4 estimable#2 \u2018deserving of esteem\u2019 0.75 0 0.25\nestimable#3 computable#1 \u2018may be computed or estimated\u2019 0 0 1\nsting#1 burn#4 bite#2 \u2018cause a sharp or stinging pain\u2019 0 0.875 .125\nacute#6 \u2018of critical importance and consequence\u2019 0.625 0.125 .250\nacute#4 \u2018of an angle; less than 90 degrees\u2019 0 0 1\nacute#1 \u2018having or experiencing a rapid onset and short but severe course\u2019 0 0.5 0.5\nFigure 22.8 Examples from SentiWordNet 3.0 (Baccianella et al., 2010). Note the differences between senses\nof homonymous words: estimable#3 is purely objective, while estimable#2 is positive; acute can be positive\n(acute#6 ), negative ( acute#1 ), or neutral ( acute #4 ).\nIn this algorithm, polarity is assigned to entire synsets rather than words. A\npositive lexicon is built from all the synsets associated with 7 positive words, and a\nnegative lexicon from synsets associated with 7 negative words. A classi\ufb01er is then\ntrained from this data to take a WordNet gloss and decide if the sense being de\ufb01ned\nis positive, negative or neutral. A further step (involving a random-walk algorithm)\nassigns a score to each WordNet synset for its degree of positivity, negativity, and\nneutrality.\nIn summary, semisupervised algorithms use a human-de\ufb01ned set of seed words\nfor the two poles of a dimension, and use similarity metrics like embedding cosine,\ncoordination, morphology, or thesaurus structure to score words by how similar they\nare to the positive seeds and how dissimilar to the negative seeds.\n22.5 Supervised Learning of Word Sentiment\nSemi-supervised methods require only minimal human supervision (in the form of\nseed sets). But sometimes a supervision signal exists in the world and can be made\nuse of. One such signal is the scores associated with online reviews .\nThe web contains an enormous number of online reviews for restaurants, movies,\nbooks, or other products, each of which have the text of the review along with an 22.5 \u2022 S UPERVISED LEARNING OF WORD SENTIMENT 491\nMovie review excerpts (IMDb)\n10A great movie. This \ufb01lm is just a wonderful experience. It\u2019s surreal, zany, witty and slapstick\nall at the same time. And terri\ufb01c performances too.\n1This was probably the worst movie I have ever seen. The story went nowhere even though they\ncould have done some interesting stuff with it.\nRestaurant review excerpts (Yelp)\n5The service was impeccable. The food was cooked and seasoned perfectly... The watermelon\nwas perfectly square ... The grilled octopus was ... mouthwatering...\n2...it took a while to get our waters, we got our entree before our starter, and we never received\nsilverware or napkins until we requested them...\nBook review excerpts (GoodReads)\n1I am going to try and stop being deceived by eye-catching titles. I so wanted to like this book\nand was so disappointed by it.\n5This book is hilarious. I would recommend it to anyone looking for a satirical read with a\nromantic twist and a narrator that keeps butting in\nProduct review excerpts (Amazon)\n5The lid on this blender though is probably what I like the best about it... enables you to pour\ninto something without even taking the lid off! ... the perfect pitcher! ... works fantastic.\n1I hate this blender... It is nearly impossible to get frozen fruit and ice to turn into a smoothie...\nYou have to add a TON of liquid. I also wish it had a spout ...\nFigure 22.9 Excerpts from some reviews from various review websites, all on a scale of 1 to 5 stars except\nIMDb, which is on a scale of 1 to 10 stars.\nassociated review score: a value that may range from 1 star to 5 stars, or scoring 1\nto 10. Fig. 22.9 shows samples extracted from restaurant, book, and movie reviews.\nWe can use this review score as supervision: positive words are more likely to\nappear in 5-star reviews; negative words in 1-star reviews. And instead of just a\nbinary polarity, this kind of supervision allows us to assign a word a more complex\nrepresentation of its polarity: its distribution over stars (or other scores).\nThus in a ten-star system we could represent the sentiment of each word as a\n10-tuple, each number a score representing the word\u2019s association with that polarity\nlevel. This association can be a raw count, or a likelihood P(wjc), or some other\nfunction of the count, for each class cfrom 1 to 10.\nFor example, we could compute the IMDb likelihood of a word like disap-\npoint(ed/ing) occurring in a 1 star review by dividing the number of times disap-\npoint(ed/ing) occurs in 1-star reviews in the IMDb dataset (8,557) by the total num-\nber of words occurring in 1-star reviews (25,395,214), so the IMDb estimate of\nP(disappointingj1)is .0003.\nA slight modi\ufb01cation of this weighting, the normalized likelihood, can be used\nas an illuminating visualization (Potts, 2011)1\nP(wjc) =count (w;c)P\nw2Ccount (w;c)\nPottsScore (w) =P(wjc)P\ncP(wjc)(22.6)\nDividing the IMDb estimate P(disappointingj1)of .0003 by the sum of the likeli-\nhood P(wjc)over all categories gives a Potts score of 0.10. The word disappointing\nthus is associated with the vector [.10, .12, .14, .14, .13, .11, .08, .06, .06, .05]. The\n1Each element of the Potts score of a word wand category ccan be shown to be a variant of the\npointwise mutual information pmi (w;c)without the log term; see Exercise 22.1. 492 CHAPTER 22 \u2022 L EXICONS FOR SENTIMENT , AFFECT ,AND CONNOTATION\nPotts diagram (Potts, 2011) is a visualization of these word scores, representing the Potts diagram\nprior sentiment of a word as a distribution over the rating categories.\nFig. 22.10 shows the Potts diagrams for 3 positive and 3 negative scalar adjec-\ntives. Note that the curve for strongly positive scalars have the shape of the letter\nJ, while strongly negative scalars look like a reverse J. By contrast, weakly posi-\ntive and negative scalars have a hump-shape, with the maximum either below the\nmean (weakly negative words like disappointing ) or above the mean (weakly pos-\nitive words like good ). These shapes offer an illuminating typology of affective\nmeaning.\nOverviewDataMethodsCategorizationScale inductionLooking aheadExample: attenuators\nIMDB \u2013 53,775 tokens\nCategory-0.50-0.39-0.28-0.17-0.060.060.170.280.390.500.050.090.15Cat = 0.33 (p = 0.004)Cat^2 = -4.02 (p < 0.001)OpenTable \u2013 3,890 tokens\nCategory-0.50-0.250.000.250.500.080.38Cat = 0.11 (p = 0.707)Cat^2 = -6.2 (p = 0.014)Goodreads \u2013 3,424 tokens\nCategory-0.50-0.250.000.250.500.080.190.36Cat = -0.55 (p = 0.128)Cat^2 = -5.04 (p = 0.016)Amazon/Tripadvisor \u2013 2,060 tokens\nCategory-0.50-0.250.000.250.500.120.28Cat = 0.42 (p = 0.207)Cat^2 = -2.74 (p = 0.05)somewhat/r\nIMDB \u2013 33,515 tokens\nCategory-0.50-0.39-0.28-0.17-0.060.060.170.280.390.500.040.090.17Cat = -0.13 (p = 0.284)Cat^2 = -5.37 (p < 0.001)OpenTable \u2013 2,829 tokens\nCategory-0.50-0.250.000.250.500.080.31Cat = 0.2 (p = 0.265)Cat^2 = -4.16 (p = 0.007)Goodreads \u2013 1,806 tokens\nCategory-0.50-0.250.000.250.500.050.120.180.35Cat = -0.87 (p = 0.016)Cat^2 = -5.74 (p = 0.004)Amazon/Tripadvisor \u2013 2,158 tokens\nCategory-0.50-0.250.000.250.500.110.29Cat = 0.54 (p = 0.183)Cat^2 = -3.32 (p = 0.045)fairly/r\nIMDB \u2013 176,264 tokens\nCategory-0.50-0.39-0.28-0.17-0.060.060.170.280.390.500.050.090.13Cat = -0.43 (p < 0.001)Cat^2 = -3.6 (p < 0.001)OpenTable \u2013 8,982 tokens\nCategory-0.50-0.250.000.250.500.080.140.190.32Cat = -0.64 (p = 0.035)Cat^2 = -4.47 (p = 0.007)Goodreads \u2013 11,895 tokens\nCategory-0.50-0.250.000.250.500.070.150.34Cat = -0.71 (p = 0.072)Cat^2 = -4.59 (p = 0.018)Amazon/Tripadvisor \u2013 5,980 tokens\nCategory-0.50-0.250.000.250.500.150.28Cat = 0.26 (p = 0.496)Cat^2 = -2.23 (p = 0.131)pretty/r\u201cPotts&diagrams\u201dPotts,&Christopher .& 2011.&NSF&workshop&on&restructuring& adjectives.goodgreatexcellentdisappointingbadterribletotallyabsolutelyutterlysomewhatfairlyprettyPositive scalarsNegative scalarsEmphaticsAttenuators1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating\nFigure 22.10 Potts diagrams (Potts, 2011) for positive and negative scalar adjectives, show-\ning the J-shape and reverse J-shape for strongly positive and negative adjectives, and the\nhump-shape for more weakly polarized adjectives.\nFig. 22.11 shows the Potts diagrams for emphasizing and attenuating adverbs.\nNote that emphatics tend to have a J-shape (most likely to occur in the most posi-\ntive reviews) or a U-shape (most likely to occur in the strongly positive and nega-\ntive). Attenuators all have the hump-shape, emphasizing the middle of the scale and\ndownplaying both extremes. The diagrams can be used both as a typology of lexical\nsentiment, and also play a role in modeling sentiment compositionality.\nIn addition to functions like posterior P(cjw), likelihood P(wjc), or normalized\nlikelihood (Eq. 22.6) many other functions of the count of a word occurring with a\nsentiment label have been used. We\u2019ll introduce some of these on page 496, includ-\ning ideas like normalizing the counts per writer in Eq. 22.14.\n22.5.1 Log Odds Ratio Informative Dirichlet Prior\nOne thing we often want to do with word polarity is to distinguish between words\nthat are more likely to be used in one category of texts than in another. We may, for\nexample, want to know the words most associated with 1 star reviews versus those\nassociated with 5 star reviews. These differences may not be just related to senti-\nment. We might want to \ufb01nd words used more often by Democratic than Republican\nmembers of Congress, or words used more often in menus of expensive restaurants 22.5 \u2022 S UPERVISED LEARNING OF WORD SENTIMENT 493\nOverviewDataMethodsCategorizationScale inductionLooking aheadExample: attenuators\nIMDB \u2013 53,775 tokens\nCategory-0.50-0.39-0.28-0.17-0.060.060.170.280.390.500.050.090.15Cat = 0.33 (p = 0.004)Cat^2 = -4.02 (p < 0.001)OpenTable \u2013 3,890 tokens\nCategory-0.50-0.250.000.250.500.080.38Cat = 0.11 (p = 0.707)Cat^2 = -6.2 (p = 0.014)Goodreads \u2013 3,424 tokens\nCategory-0.50-0.250.000.250.500.080.190.36Cat = -0.55 (p = 0.128)Cat^2 = -5.04 (p = 0.016)Amazon/Tripadvisor \u2013 2,060 tokens\nCategory-0.50-0.250.000.250.500.120.28Cat = 0.42 (p = 0.207)Cat^2 = -2.74 (p = 0.05)somewhat/r\nIMDB \u2013 33,515 tokens\nCategory-0.50-0.39-0.28-0.17-0.060.060.170.280.390.500.040.090.17Cat = -0.13 (p = 0.284)Cat^2 = -5.37 (p < 0.001)OpenTable \u2013 2,829 tokens\nCategory-0.50-0.250.000.250.500.080.31Cat = 0.2 (p = 0.265)Cat^2 = -4.16 (p = 0.007)Goodreads \u2013 1,806 tokens\nCategory-0.50-0.250.000.250.500.050.120.180.35Cat = -0.87 (p = 0.016)Cat^2 = -5.74 (p = 0.004)Amazon/Tripadvisor \u2013 2,158 tokens\nCategory-0.50-0.250.000.250.500.110.29Cat = 0.54 (p = 0.183)Cat^2 = -3.32 (p = 0.045)fairly/r\nIMDB \u2013 176,264 tokens\nCategory-0.50-0.39-0.28-0.17-0.060.060.170.280.390.500.050.090.13Cat = -0.43 (p < 0.001)Cat^2 = -3.6 (p < 0.001)OpenTable \u2013 8,982 tokens\nCategory-0.50-0.250.000.250.500.080.140.190.32Cat = -0.64 (p = 0.035)Cat^2 = -4.47 (p = 0.007)Goodreads \u2013 11,895 tokens\nCategory-0.50-0.250.000.250.500.070.150.34Cat = -0.71 (p = 0.072)Cat^2 = -4.59 (p = 0.018)Amazon/Tripadvisor \u2013 5,980 tokens\nCategory-0.50-0.250.000.250.500.150.28Cat = 0.26 (p = 0.496)Cat^2 = -2.23 (p = 0.131)pretty/r\u201cPotts&diagrams\u201dPotts,&Christopher .& 2011.&NSF&workshop&on&restructuring& adjectives.goodgreatexcellentdisappointingbadterribletotallyabsolutelyutterlysomewhatfairlyprettyPositive scalarsNegative scalarsEmphaticsAttenuators1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating\nFigure 22.11 Potts diagrams (Potts, 2011) for emphatic and attenuating adverbs.\nthan cheap restaurants.\nGiven two classes of documents, to \ufb01nd words more associated with one cate-\ngory than another, we could measure the difference in frequencies (is a word wmore\nfrequent in class Aor class B?). Or instead of the difference in frequencies we could\ncompute the ratio of frequencies, or compute the log odds ratio (the log of the ratio\nbetween the odds of the two words). We could then sort words by whichever associ-\nation measure we pick, ranging from words overrepresented in category Ato words\noverrepresented in category B.\nThe problem with simple log-likelihood or log odds methods is that they overem-\nphasize differences in very rare words, and often also in very frequent words. Very\nrare words will seem to occur very differently in the two corpora since with tiny\ncounts there may be statistical \ufb02uctations, or even zero occurrences in one corpus\ncompared to non-zero occurrences in the other. Very frequent words will also seem\ndifferent since all counts are large.\nIn this section we walk through the details of one solution to this problem: the\n\u201clog odds ratio informative Dirichlet prior\u201d method of Monroe et al. (2008) that is a\nparticularly useful method for \ufb01nding words that are statistically overrepresented in\none particular category of texts compared to another. It\u2019s based on the idea of using\nanother large corpus to get a prior estimate of what we expect the frequency of each\nword to be.\nLet\u2019s start with the goal: assume we want to know whether the word horrible\noccurs more in corpus ior corpus j. We could compute the log likelihood ratio ,log likelihood\nratio\nusing fi(w)to mean the frequency of word win corpus i, and nito mean the total\nnumber of words in corpus i:\nllr(horrible ) = logPi(horrible )\nPj(horrible )\n=logPi(horrible )\u0000logPj(horrible )\n=logfi(horrible )\nni\u0000logfj(horrible )\nnj(22.7)\nInstead, let\u2019s compute the log odds ratio : does horrible have higher odds in ior in log odds ratio 494 CHAPTER 22 \u2022 L EXICONS FOR SENTIMENT , AFFECT ,AND CONNOTATION\nj:\nlor(horrible ) = log\u0012Pi(horrible )\n1\u0000Pi(horrible )\u0013\n\u0000log\u0012Pj(horrible )\n1\u0000Pj(horrible )\u0013\n=log0\nB@fi(horrible )\nni\n1\u0000fi(horrible )\nni1\nCA\u0000log0\nB@fj(horrible )\nnj\n1\u0000fj(horrible )\nnj1\nCA\n=log\u0012fi(horrible )\nni\u0000fi(horrible )\u0013\n\u0000log\u0012fj(horrible )\nnj\u0000fj(horrible )\u0013\n(22.8)\nThe Dirichlet intuition is to use a large background corpus to get a prior estimate of\nwhat we expect the frequency of each word wto be. We\u2019ll do this very simply by\nadding the counts from that corpus to the numerator and denominator, so that we\u2019re\nessentially shrinking the counts toward that prior. It\u2019s like asking how large are the\ndifferences between iand jgiven what we would expect given their frequencies in\na well-estimated large background corpus.\nThe method estimates the difference between the frequency of word win two\ncorpora iandjvia the prior-modi\ufb01ed log odds ratio for w,d(i\u0000j)\nw, which is estimated\nas:\nd(i\u0000j)\nw=log\u0012fi\nw+aw\nni+a0\u0000(fiw+aw)\u0013\n\u0000log \nfj\nw+aw\nnj+a0\u0000(fj\nw+aw)!\n(22.9)\n(where niis the size of corpus i,njis the size of corpus j,fi\nwis the count of word\nwin corpus i,fj\nwis the count of word win corpus j,a0is the scaled size of the\nbackground corpus, and awis the scaled count of word win the background corpus.)\nIn addition, Monroe et al. (2008) make use of an estimate for the variance of the\nlog\u2013odds\u2013ratio:\ns2\u0010\n\u02c6d(i\u0000j)\nw\u0011\n\u00191\nfiw+aw+1\nfj\nw+aw(22.10)\nThe \ufb01nal statistic for a word is then the z\u2013score of its log\u2013odds\u2013ratio:\n\u02c6d(i\u0000j)\nwr\ns2\u0010\n\u02c6d(i\u0000j)\nw\u0011(22.11)\nThe Monroe et al. (2008) method thus modi\ufb01es the commonly used log odds ratio\nin two ways: it uses the z-scores of the log odds ratio, which controls for the amount\nof variance in a word\u2019s frequency, and it uses counts from a background corpus to\nprovide a prior count for words.\nFig. 22.12 shows the method applied to a dataset of restaurant reviews from\nYelp, comparing the words used in 1-star reviews to the words used in 5-star reviews\n(Jurafsky et al., 2014). The largest difference is in obvious sentiment words, with the\n1-star reviews using negative sentiment words like worse, bad, awful and the 5-star\nreviews using positive sentiment words like great, best, amazing . But there are other\nilluminating differences. 1-star reviews use logical negation ( no, not ), while 5-star\nreviews use emphatics and emphasize universality ( very, highly, every, always ). 1-\nstar reviews use \ufb01rst person plurals ( we, us, our ) while 5 star reviews use the second\nperson. 1-star reviews talk about people ( manager, waiter, customer ) while 5-star\nreviews talk about dessert and properties of expensive restaurants like courses and\natmosphere. See Jurafsky et al. (2014) for more details. 22.6 \u2022 U SING LEXICONS FOR SENTIMENT RECOGNITION 495\nClass Words in 1-star reviews Class Words in 5-star reviews\nNegative worst, rude, terrible, horrible, bad,\nawful, disgusting, bland, tasteless,\ngross, mediocre, overpriced, worse,\npoorPositive great, best, love(d), delicious, amazing,\nfavorite, perfect, excellent, awesome,\nfriendly, fantastic, fresh, wonderful, in-\ncredible, sweet, yum(my)\nNegation no, not Emphatics/\nuniversalsvery, highly, perfectly, de\ufb01nitely, abso-\nlutely, everything, every, always\n1Pl pro we, us, our 2 pro you\n3 pro she, he, her, him Articles a, the\nPast verb was, were, asked, told, said, did,\ncharged, waited, left, tookAdvice try, recommend\nSequencers after, then Conjunct also, as, well, with, and\nNouns manager, waitress, waiter, customer,\ncustomers, attitude, waste, poisoning,\nmoney, bill, minutesNouns atmosphere, dessert, chocolate, wine,\ncourse, menu\nIrrealis\nmodalswould, should Auxiliaries is/\u2019s, can, \u2019ve, are\nComp to, that Prep, other in, of, die, city, mouth\nFigure 22.12 The top 50 words associated with one\u2013star and \ufb01ve-star restaurant reviews in a Yelp dataset of\n900,000 reviews, using the Monroe et al. (2008) method (Jurafsky et al., 2014).\n22.6 Using Lexicons for Sentiment Recognition\nIn Chapter 4 we introduced the naive Bayes algorithm for sentiment analysis. The\nlexicons we have focused on throughout the chapter so far can be used in a number\nof ways to improve sentiment detection.\nIn the simplest case, lexicons can be used when we don\u2019t have suf\ufb01cient training\ndata to build a supervised sentiment analyzer; it can often be expensive to have a\nhuman assign sentiment to each document to train the supervised classi\ufb01er.\nIn such situations, lexicons can be used in a rule-based algorithm for classi\ufb01ca-\ntion. The simplest version is just to use the ratio of positive to negative words: if a\ndocument has more positive than negative words (using the lexicon to decide the po-\nlarity of each word in the document), it is classi\ufb01ed as positive. Often a threshold l\nis used, in which a document is classi\ufb01ed as positive only if the ratio is greater than\nl. If the sentiment lexicon includes positive and negative weights for each word,\nq+\nwandq\u0000\nw, these can be used as well. Here\u2019s a simple such sentiment algorithm:\nf+=X\nws.t.w2positivelexiconq+\nwcount (w)\nf\u0000=X\nws.t.w2negativelexiconq\u0000\nwcount (w)\nsentiment =8\n>>><\n>>>:+iff+\nf\u0000>l\n\u0000iff\u0000\nf+>l\n0 otherwise.(22.12)\nIf supervised training data is available, these counts computed from sentiment lex-\nicons, sometimes weighted or normalized in various ways, can also be used as fea-\ntures in a classi\ufb01er along with other lexical or non-lexical features. We return to\nsuch algorithms in Section 22.7. 496 CHAPTER 22 \u2022 L EXICONS FOR SENTIMENT , AFFECT ,AND CONNOTATION\n22.7 Using Lexicons for Affect Recognition\nDetection of emotion (and the other kinds of affective meaning described by Scherer\n(2000)) can be done by generalizing the algorithms described above for detecting\nsentiment.\nThe most common algorithms involve supervised classi\ufb01cation: a training set is\nlabeled for the affective meaning to be detected, and a classi\ufb01er is built using features\nextracted from the training set. As with sentiment analysis, if the training set is large\nenough, and the test set is suf\ufb01ciently similar to the training set, simply using all\nthe words or all the bigrams as features in a powerful classi\ufb01er like SVM or logistic\nregression, as described in Fig. 4.2 in Chapter 4, is an excellent algorithm whose\nperformance is hard to beat. Thus we can treat affective meaning classi\ufb01cation of a\ntext sample as simple document classi\ufb01cation.\nSome modi\ufb01cations are nonetheless often necessary for very large datasets. For\nexample, the Schwartz et al. (2013) study of personality, gender, and age using 700\nmillion words of Facebook posts used only a subset of the n-grams of lengths 1-\n3. Only words and phrases used by at least 1% of the subjects were included as\nfeatures, and 2-grams and 3-grams were only kept if they had suf\ufb01ciently high PMI\n(PMI greater than 2 \u0003length , where length is the number of words):\npmi(phrase ) =logp(phrase )Y\nw2phrasep(w)(22.13)\nVarious weights can be used for the features, including the raw count in the training\nset, or some normalized probability or log probability. Schwartz et al. (2013), for\nexample, turn feature counts into phrase likelihoods by normalizing them by each\nsubject\u2019s total word use.\np(phrasejsubject ) =freq(phrase;subject )X\nphrase02vocab (subject )freq(phrase0;subject )(22.14)\nIf the training data is sparser, or not as similar to the test set, any of the lexicons\nwe\u2019ve discussed can play a helpful role, either alone or in combination with all the\nwords and n-grams.\nMany possible values can be used for lexicon features. The simplest is just an\nindicator function, in which the value of a feature fLtakes the value 1 if a particular\ntext has any word from the relevant lexicon L. Using the notation of Chapter 4, in\nwhich a feature value is de\ufb01ned for a particular output class cand document x.\nfL(c;x) =\u001a1 if9w:w2L&w2x&class =c\n0 otherwise\nAlternatively the value of a feature fLfor a particular lexicon Lcan be the total\nnumber of word tokens in the document that occur in L:\nfL=X\nw2Lcount (w)\nFor lexica in which each word is associated with a score or weight, the count can be\nmultiplied by a weight qL\nw:\nfL=X\nw2LqL\nwcount (w) 22.8 \u2022 L EXICON -BASED METHODS FOR ENTITY -CENTRIC AFFECT 497\nCounts can alternatively be logged or normalized per writer as in Eq. 22.14.\nHowever they are de\ufb01ned, these lexicon features are then used in a supervised\nclassi\ufb01er to predict the desired affective category for the text or document. Once\na classi\ufb01er is trained, we can examine which lexicon features are associated with\nwhich classes. For a classi\ufb01er like logistic regression the feature weight gives an\nindication of how associated the feature is with the class.\n22.8 Lexicon-based methods for Entity-Centric Affect\nWhat if we want to get an affect score not for an entire document, but for a particular\nentity in the text? The entity-centric method of Field and Tsvetkov (2019) combines\naffect lexicons with contextual embeddings to assign an affect score to an entity in\ntext. In the context of affect about people, they relabel the Valence/Arousal/Domi-\nnance dimension as Sentiment/Agency/Power. The algorithm \ufb01rst trains classi\ufb01ers\nto map embeddings to scores:\n1. For each word win the training corpus:\n(a) Use off-the-shelf pretrained encoders (like BERT) to extract a contextual\nembedding efor each instance of the word. No additional \ufb01ne-tuning is\ndone.\n(b) Average over the eembeddings of each instance of wto obtain a single\nembedding vector for one training point w.\n(c) Use the NRC V AD Lexicon to get S, A, and P scores for w.\n2. Train (three) regression models on all words wto predict V , A, D scores from\na word\u2019s average embedding.\nNow given an entity mention min a text, we assign affect scores as follows:\n1. Use the same pretrained LM to get contextual embeddings for min context.\n2. Feed this embedding through the 3 regression models to get S, A, P scores for\nthe entity.\nThis results in a (S,A,P) tuple for a given entity mention; To get scores for the rep-\nresentation of an entity in a complete document, we can run coreference resolution\nand average the (S,A,P) scores for all the mentions. Fig. 22.13 shows the scores\nfrom their algorithm for characters from the movie The Dark Knight when run on\nWikipedia plot summary texts with gold coreference.\n22.9 Connotation Frames\nThe lexicons we\u2019ve described so far de\ufb01ne a word as a point in affective space. A\nconnotation frame , by contrast, is a lexicon that incorporates a richer kind of gram-connotation\nframe\nmatical structure, by combining affective lexicons with the frame semantic lexicons\nof Chapter 21. The basic insight of connotation frame lexicons is that a predicate\nlike a verb expresses connotations about the verb\u2019s arguments (Rashkin et al. 2016,\nRashkin et al. 2017).\nConsider sentences like:\n(22.15) Country A violated the sovereignty of Country B 498 CHAPTER 22 \u2022 L EXICONS FOR SENTIMENT , AFFECT ,AND CONNOTATION\nPower ScoreweaklyRachelDentGordanBatmanJokerpowerfully\nSentiment ScorenegativeJokerDentGordanRachelBatmanpositive\nAgency ScoredullDentGordanRachelBatmanJokerscaryFigure 1: Power, sentiment, and agency scores for char-\nacters in The Dark Night as learned through the regres-\nsion model with ELMo embeddings. Scores generally\nalign with character archetypes, i.e. the antagonist has\nthe lowest sentiment score.\nment have resulted in his effective removal from\nthe industry. While articles about the #MeToo\nmovement portray men like Weinstein as unpow-\nerful, we can speculate that the corpora used to\ntrain ELMo and BERT portray them as powerful.\nThus, in a corpus where traditional power roles\nhave been inverted, the embeddings extracted\nfrom ELMo and BERT perform worse than ran-\ndom, as they are biased towards the power struc-\ntures in the data they are trained on. Further ev-\nidence of this exists in the performance of the\nBERT-masked embeddings - whereas these em-\nbeddings generally capture power poorly as com-\npared to the unmasked embeddings (Table 2),\nthey outperform the unmasked embeddings on this\ntask, and even outperform the frequency baseline\nin one setting. Nevertheless, they do not outper-\nform Field et al. (2019 ), likely because they do not\ncapture affect information as well as the unmasked\nembeddings (Table 2).\n4.3 Qualitative Document-level Analysis\nFinally, we qualitatively analyze how well our\nmethod captures affect dimensions by analyzing\nsingle documents in detail. We conduct this anal-\nysis in a domain where we expect entities to ful\ufb01ll\ntraditional power roles and where entity portray-\nals are known. Following Bamman et al. (2013 ),\nwe analyze the Wikipedia plot summary of the\nmovie The Dark Knight ,7focusing on Batman\n(protagonist),8the Joker (antagonist), Jim Gordan\n(law enforcement of\ufb01cer, ally to Batman), Har-\n7http://bit.ly/2XmhRDR\n8We consider Batman/Bruce Wayne to be the same entity.Power Scoreweakly Rachel Joker Dent Gordan Batmanpowerfully\nSentiment Scorenegative Joker Gordan Batman Dent Rachel positive\nAgency Scoredull Rachel Dent GordanBatman Joker scaryFigure 2: Power, sentiment, and agency scores for char-acters inThe Dark Nightas learned through ASP withELMo embeddings. These scores re\ufb02ect the same pat-terns as the regression model with greater separationbetween characters.vey Dent (ally to Batman who turns evil) andRachel Dawes (primary love interest). To facil-itate extracting example sentences, we score eachinstance of these entities in the narrative separatelyand average across instances to obtain an entityscore for the document.9To maximize our databy capturing every mention of an entity, we per-form co-reference resolution by hand. Addition-ally, based on our results from Table3as well asthe use of Wikipedia data in training the ELMomodel (Peters et al.,2018), we use ELMo embed-dings for our analysis.Figures1and2show results. For refer-ence, we show the entity scores as compared toone polar opposite pair identi\ufb01ed by ASP. Boththe regression model and ASP show similar pat-terns. Batman has high power, while Rachel haslow power. Additionally, the Joker is associatedwith the most negative sentiment, but the high-est agency. Throughout the plot summary, themovie progresses by the Joker taking an aggres-sive action and the other characters responding.We can see this dynamic re\ufb02ected in the Joker\u2019spro\ufb01le score, as a high-powered, high-agency,low-sentiment character, who is the primary plot-driver. In general, ASP shows a greater separationbetween characters than the regression model. Wehypothesize that this occurs because ASP isolatesthe dimensions of interest, while the regression ap-proach captures other confounds, such as that hu-9When we used this averaging metric in other evaluations,we found no signi\ufb01cant change in results. Thus, in other sce-narios, we compute scores over averaged embeddings, ratherthan averaging scores separately computed for each embed-ding to reduce computationally complexity.\nFigure 22.13 Power (dominance), sentiment (valence) and agency (arousal) for characters\nin the movie The Dark Knight computed from embeddings trained on the NRC V AD Lexicon.\nNote the protagonist (Batman) and the antagonist (the Joker) have high power and agency\nscores but differ in sentiment, while the love interest Rachel has low power and agency but\nhigh sentiment.\n(22.16) the teenager ... survived the Boston Marathon bombing\u201d\nBy using the verb violate in (22.15), the author is expressing their sympathies with\nCountry B, portraying Country B as a victim, and expressing antagonism toward\nthe agent Country A. By contrast, in using the verb survive , the author of (22.16) is\nexpressing that the bombing is a negative experience, and the subject of the sentence,\nthe teenager, is a sympathetic character. These aspects of connotation are inherent\nin the meaning of the verbs violate andsurvive , as shown in Fig. 22.14.\nWriter\nRole1\nRole2Role1 is asympathetic victimThere issome typeof hardship\nReader+_+__S(writer\u2192role1)S(writer\u2192role2)Connotation Frame for \u201cRole1 survives Role2\u201d S(role1\u2192role2)\nWriter\nRole1\nRole2Role1 is the antagonistRole2 is asympathetic victim\nReader+_+__S(writer\u2192role1)S(writer\u2192role2)Connotation Frame for \u201cRole1 violates Role2\u201d S(role1\u2192role2)\n(a) (b)\nFigure 22.14 Connotation frames for survive andviolate . (a) For survive , the writer and reader have positive\nsentiment toward Role1, the subject, and negative sentiment toward Role2, the direct object. (b) For violate , the\nwriter and reader have positive sentiment instead toward Role2, the direct object.\nThe connotation frame lexicons of Rashkin et al. (2016) and Rashkin et al.\n(2017) also express other connotative aspects of the predicate toward each argu-\nment, including the effect (something bad happened to x) value : (x is valuable), and\nmental state : (x is distressed by the event). Connotation frames can also mark the\npower differential between the arguments (using the verb implore means that the\ntheme argument has greater power than the agent), and the agency of each argument\n(waited is low agency). Fig. 22.15 shows a visualization from Sap et al. (2017).\nConnotation frames can be built by hand (Sap et al., 2017), or they can be learned\nby supervised learning (Rashkin et al., 2016), for example using hand-labeled train- 22.10 \u2022 S UMMARY 499\nAGENTTHEMEpower(AG < TH)VERBimploreHe implored the tribunal to show mercy.The princess waited for her prince.AGENTTHEMEagency(AG) = -VERBwaitFigure 2: The formal notation of the connotation\nframes of power and agency. The \ufb01rst example\nshows the relative power differential implied by\nthe verb \u201cimplored\u201d , i.e., the agent (\u201che\u201d) is in\na position of less power than the theme (\u201cthe tri-\nbunal\u201d). In contrast, \u201cHe demanded the tribunal\nshow mercy\u201d implies that the agent has authority\nover the theme. The second example shows the\nlow level of agency implied by the verb \u201cwaited\u201d .\ninteractive demo website of our \ufb01ndings (see Fig-\nure5in the appendix for a screenshot).2Further-\nmore, as will be seen in Section 4.1, connotation\nframes offer new insights that complement and de-\nviate from the well-known Bechdel test ( Bechdel ,\n1986 ). In particular, we \ufb01nd that high-agency\nwomen through the lens of connotation frames are\nrare in modern \ufb01lms. It is, in part, because some\nmovies (e.g., Snow White) accidentally pass the\nBechdel test and also because even movies with\nstrong female characters are not entirely free from\nthe deeply ingrained biases in social norms.\n2 Connotation Frames of Power and\nAgency\nWe create two new connotation relations, power\nandagency (examples in Figure 3), as an expan-\nsion of the existing connotation frame lexicons.3\nThree AMT crowdworkers annotated the verbs\nwith placeholders to avoid gender bias in the con-\ntext (e.g., Xrescued Y; an example task is shown\nin the appendix in Figure 7). We de\ufb01ne the anno-\ntated constructs as follows:\nPower Differentials Many verbs imply the au-\nthority levels of the agent and theme relative to\n2http://homes .cs.washington .edu/ \u02dcmsap/\nmovie-bias/ .\n3The lexicons and a demo are available at http://\nhomes .cs.washington .edu/ \u02dcmsap/movie-bias/ .power (AG<TH)power (AG>TH)\nagency (AG)=\u0000agency (AG)=+Figure 3: Sample verbs in the connotation frames\nwith high annotator agreement. Size is indicative\nof verb frequency in our corpus (bigger =more\nfrequent), color differences are only for legibility.\none another. For example, if the agent \u201cdom-\ninates\u201d the theme (denoted as power (AG>TH)),\nthen the agent is implied to have a level of control\nover the theme. Alternatively, if the agent \u201chon-\nors\u201d the theme (denoted as power (AG<TH)), the\nwriter implies that the theme is more important or\nauthoritative. We used AMT crowdsourcing to la-\nbel 1700 transitive verbs for power differentials.\nWith three annotators per verb, the inter-annotator\nagreement is 0.34 (Krippendorff\u2019s \u21b5).\nAgency The agency attributed to the agent of the\nverb denotes whether the action being described\nimplies that the agent is powerful, decisive, and\ncapable of pushing forward their own storyline.\nFor example, a person who is described as \u201cex-\nperiencing\u201d things does not seem as active and de-\ncisive as someone who is described as \u201cdetermin-\ning\u201d things. AMT workers labeled 2000 transi-\ntive verbs for implying high/moderate/low agency\n(inter-annotator agreement of 0.27). We denote\nhigh agency as agency (AG)=+, and low agency\nasagency (AG)=\u0000.\nPairwise agreements on a hard constraint are\n56% and 51% for power and agency, respec-\ntively. Despite this, agreements reach 96% and\n94% when moderate labels are counted as agree-\ning with either high or low labels, showing that an-\nnotators rarely strongly disagree with one another.\nSome contributing factors in the lower KA scores\ninclude the subtlety of choosing between neutral\nFigure 22.15 The connotation frames of Sap et al. (2017), showing that the verb implore\nimplies the agent has lower power than the theme (in contrast, say, with a verb like demanded ),\nand showing the low level of agency of the subject of waited . Figure from Sap et al. (2017).\ning data to supervise classi\ufb01ers for each of the individual relations, e.g., whether\nS(writer!Role1) is + or -, and then improving accuracy via global constraints\nacross all relations.\n22.10 Summary\n\u2022 Many kinds of affective states can be distinguished, including emotions ,moods ,\nattitudes (which include sentiment ),interpersonal stance , and personality .\n\u2022Emotion can be represented by \ufb01xed atomic units often called basic emo-\ntions , or as points in space de\ufb01ned by dimensions like valence andarousal .\n\u2022 Words have connotational aspects related to these affective states, and this\nconnotational aspect of word meaning can be represented in lexicons.\n\u2022 Affective lexicons can be built by hand, using crowd sourcing to label the\naffective content of each word.\n\u2022 Lexicons can be built with semi-supervised , bootstrapping from seed words\nusing similarity metrics like embedding cosine.\n\u2022 Lexicons can be learned in a fully supervised manner, when a convenient\ntraining signal can be found in the world, such as ratings assigned by users on\na review site.\n\u2022 Words can be assigned weights in a lexicon by using various functions of word\ncounts in training texts, and ratio metrics like log odds ratio informative\nDirichlet prior .\n\u2022 Affect can be detected, just like sentiment, by using standard supervised text\nclassi\ufb01cation techniques, using all the words or bigrams in a text as features.\nAdditional features can be drawn from counts of words in lexicons.\n\u2022 Lexicons can also be used to detect affect in a rule-based classi\ufb01er by picking\nthe simple majority sentiment based on counts of words in each lexicon.\n\u2022Connotation frames express richer relations of affective meaning that a pred-\nicate encodes about its arguments. 500 CHAPTER 22 \u2022 L EXICONS FOR SENTIMENT , AFFECT ,AND CONNOTATION\nBibliographical and Historical Notes\nThe idea of formally representing the subjective meaning of words began with Os-\ngood et al. (1957), the same pioneering study that \ufb01rst proposed the vector space\nmodel of meaning described in Chapter 6. Osgood et al. (1957) had participants rate\nwords on various scales, and ran factor analysis on the ratings. The most signi\ufb01cant\nfactor they uncovered was the evaluative dimension, which distinguished between\npairs like good/bad ,valuable/worthless ,pleasant/unpleasant . This work in\ufb02uenced\nthe development of early dictionaries of sentiment and affective meaning in the \ufb01eld\nofcontent analysis (Stone et al., 1966).\nWiebe (1994) began an in\ufb02uential line of work on detecting subjectivity in text, subjectivity\nbeginning with the task of identifying subjective sentences and the subjective char-\nacters who are described in the text as holding private states, beliefs or attitudes.\nLearned sentiment lexicons such as the polarity lexicons of Hatzivassiloglou and\nMcKeown (1997) were shown to be a useful feature in subjectivity detection (Hatzi-\nvassiloglou and Wiebe 2000, Wiebe 2000).\nThe term sentiment seems to have been introduced in 2001 by Das and Chen\n(2001), to describe the task of measuring market sentiment by looking at the words in\nstock trading message boards. In the same paper Das and Chen (2001) also proposed\nthe use of a sentiment lexicon. The list of words in the lexicon was created by\nhand, but each word was assigned weights according to how much it discriminated\na particular class (say buy versus sell) by maximizing across-class variation and\nminimizing within-class variation. The term sentiment , and the use of lexicons,\ncaught on quite quickly (e.g., inter alia, Turney 2002). Pang et al. (2002) \ufb01rst showed\nthe power of using all the words without a sentiment lexicon; see also Wang and\nManning (2012).\nMost of the semi-supervised methods we describe for extending sentiment dic-\ntionaries drew on the early idea that synonyms and antonyms tend to co-occur in the\nsame sentence (Miller and Charles 1991, Justeson and Katz 1991, Riloff and Shep-\nherd 1997). Other semi-supervised methods for learning cues to affective mean-\ning rely on information extraction techniques, like the AutoSlog pattern extractors\n(Riloff and Wiebe, 2003). Graph based algorithms for sentiment were \ufb01rst sug-\ngested by Hatzivassiloglou and McKeown (1997), and graph propagation became a\nstandard method (Zhu and Ghahramani 2002, Zhu et al. 2003, Zhou et al. 2004a,\nVelikovich et al. 2010). Crowdsourcing can also be used to improve precision by\n\ufb01ltering the result of semi-supervised lexicon learning (Riloff and Shepherd 1997,\nFast et al. 2016).\nMuch recent work focuses on ways to learn embeddings that directly encode sen-\ntiment or other properties, such as the D ENSIFIER algorithm of Rothe et al. (2016)\nthat learns to transform the embedding space to focus on sentiment (or other) infor-\nmation.\nExercises\n22.1 Show that the relationship between a word wand a category cin the Potts\nScore in Eq. 22.6 is a variant of the pointwise mutual information pmi (w;c)\nwithout the log term. CHAPTER\n23Coreference Resolution and\nEntity Linking\nand even Stigand, the patriotic archbishop of Canterbury, found it advisable\u2013\u201d\u2019\n\u2018Found WHAT?\u2019 said the Duck.\n\u2018Found IT, \u2019 the Mouse replied rather crossly: \u2018of course you know what \u201cit\u201dmeans. \u2019\n\u2018I know what \u201cit\u201dmeans well enough, when I \ufb01nd a thing, \u2019 said the Duck: \u2018it\u2019s gener-\nally a frog or a worm. The question is, what did the archbishop \ufb01nd?\u2019\nLewis Carroll, Alice in Wonderland\nAn important component of language processing is knowing who is being talked\nabout in a text. Consider the following passage:\n(23.1) Victoria Chen , CFO of Megabucks Banking, saw her pay jump to $2.3\nmillion, as the 38-year-old became the company\u2019s president. It is widely\nknown that she came to Megabucks from rival Lotsabucks.\nEach of the underlined phrases in this passage is used by the writer to refer to\na person named Victoria Chen. We call linguistic expressions like herorVictoria\nChen mentions orreferring expressions , and the discourse entity that is referred mention\nto (Victoria Chen) the referent . (To distinguish between referring expressions and referent\ntheir referents, we italicize the former.)1Two or more referring expressions that are\nused to refer to the same discourse entity are said to corefer ; thus, Victoria Chen corefer\nandshecorefer in (23.1).\nCoreference is an important component of natural language processing. A dia-\nlogue system that has just told the user \u201cThere is a 2pm \ufb02ight on United and a 4pm\none on Cathay Paci\ufb01c\u201d must know which \ufb02ight the user means by \u201cI\u2019ll take the sec-\nond one\u201d . A question answering system that uses Wikipedia to answer a question\nabout Marie Curie must know who shewas in the sentence \u201cShe was born in War-\nsaw\u201d . And a machine translation system translating from a language like Spanish, in\nwhich pronouns can be dropped, must use coreference from the previous sentence to\ndecide whether the Spanish sentence \u2018 \u201cMe encanta el conocimiento\u201d, dice. \u2019 should\nbe translated as \u2018 \u201cI love knowledge\u201d, he says \u2019, or \u2018 \u201cI love knowledge\u201d, she says \u2019.\nIndeed, this example comes from an actual news article in El Pa \u00b4\u0131sabout a female\nprofessor and was mistranslated as \u201che\u201d in machine translation because of inaccurate\ncoreference resolution (Schiebinger, 2013).\nNatural language processing systems (and humans) interpret linguistic expres-\nsions with respect to a discourse model (Karttunen, 1969). A discourse modeldiscourse\nmodel\n(Fig. 23.1) is a mental model that the understander builds incrementally when in-\nterpreting a text, containing representations of the entities referred to in the text,\nas well as properties of the entities and relations among them. When a referent is\n\ufb01rst mentioned in a discourse, we say that a representation for it is evoked into the evoked\nmodel. Upon subsequent mention, this representation is accessed from the model. accessed\n1As a convenient shorthand, we sometimes speak of a referring expression referring to a referent, e.g.,\nsaying that sherefers to Victoria Chen. However, the reader should keep in mind that what we really\nmean is that the speaker is performing the act of referring to Victoria Chen by uttering she. 502 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\nVDiscourse Model\u201cVictoria\u201d\u201cshe\u201dcoreferrefer (evoke)refer (access)$LotsabucksMegabuckspay\nFigure 23.1 How mentions evoke and access discourse entities in a discourse model.\nReference in a text to an entity that has been previously introduced into the\ndiscourse is called anaphora , and the referring expression used is said to be an anaphora\nanaphor , or anaphoric.2In passage (23.1), the pronouns sheandherand the de\ufb01- anaphor\nnite NP the 38-year-old are therefore anaphoric. The anaphor corefers with a prior\nmention (in this case Victoria Chen ) that is called the antecedent . Not every refer- antecedent\nring expression is an antecedent. An entity that has only a single mention in a text\n(like Lotsabucks in (23.1)) is called a singleton . singleton\nIn this chapter we focus on the task of coreference resolution . Coreferencecoreference\nresolution\nresolution is the task of determining whether two mentions corefer , by which we\nmean they refer to the same entity in the discourse model (the same discourse entity ).\nThe set of coreferring expressions is often called a coreference chain or acluster .coreference\nchain\ncluster For example, in processing (23.1), a coreference resolution algorithm would need\nto \ufb01nd at least four coreference chains, corresponding to the four entities in the\ndiscourse model in Fig. 23.1.\n1.fVictoria Chen ,her,the 38-year-old ,Sheg\n2.fMegabucks Banking ,the company ,Megabucksg\n3.fher payg\n4.fLotsabucksg\nNote that mentions can be nested; for example the mention heris syntactically\npart of another mention, her pay , referring to a completely different discourse entity.\nCoreference resolution thus comprises two tasks (although they are often per-\nformed jointly): (1) identifying the mentions, and (2) clustering them into corefer-\nence chains/discourse entities.\nWe said that two mentions corefered if they are associated with the same dis-\ncourse entity . But often we\u2019d like to go further, deciding which real world entity is\nassociated with this discourse entity. For example, the mention Washington might\nrefer to the US state, or the capital city, or the person George Washington; the inter-\npretation of the sentence will of course be very different for each of these. The task\nofentity linking (Ji and Grishman, 2011) or entity resolution is the task of mapping entity linking\na discourse entity to some real-world individual.3We usually operationalize entity\n2We will follow the common NLP usage of anaphor to mean any mention that has an antecedent, rather\nthan the more narrow usage to mean only mentions (like pronouns) whose interpretation depends on the\nantecedent (under the narrower interpretation, repeated names are not anaphors).\n3Computational linguistics/NLP thus differs in its use of the term reference from the \ufb01eld of formal\nsemantics, which uses the words reference andcoreference to describe the relation between a mention\nand a real-world entity. By contrast, we follow the functional linguistics tradition in which a mention\nrefers to adiscourse entity (Webber, 1978) and the relation between a discourse entity and the real world\nindividual requires an additional step of linking . 503\nlinking or resolution by mapping to an ontology : a list of entities in the world, like\na gazeteer (Appendix F). Perhaps the most common ontology used for this task is\nWikipedia; each Wikipedia page acts as the unique id for a particular entity. Thus\nthe entity linking task of wiki\ufb01cation (Mihalcea and Csomai, 2007) is the task of de-\nciding which Wikipedia page corresponding to an individual is being referred to by\na mention. But entity linking can be done with any ontology; for example if we have\nan ontology of genes, we can link mentions of genes in text to the disambiguated\ngene name in the ontology.\nIn the next sections we introduce the task of coreference resolution in more de-\ntail, and survey a variety of architectures for resolution. We also introduce two\narchitectures for the task of entity linking.\nBefore turning to algorithms, however, we mention some important tasks we\nwill only touch on brie\ufb02y at the end of this chapter. First are the famous Winograd\nSchema problems (so-called because they were \ufb01rst pointed out by Terry Winograd\nin his dissertation). These entity coreference resolution problems are designed to be\ntoo dif\ufb01cult to be solved by the resolution methods we describe in this chapter, and\nthe kind of real-world knowledge they require has made them a kind of challenge\ntask for natural language processing. For example, consider the task of determining\nthe correct antecedent of the pronoun they in the following example:\n(23.2) The city council denied the demonstrators a permit because\na. they feared violence.\nb. they advocated violence.\nDetermining the correct antecedent for the pronoun they requires understanding\nthat the second clause is intended as an explanation of the \ufb01rst clause, and also\nthat city councils are perhaps more likely than demonstrators to fear violence and\nthat demonstrators might be more likely to advocate violence. Solving Winograd\nSchema problems requires \ufb01nding way to represent or discover the necessary real\nworld knowledge.\nA problem we won\u2019t discuss in this chapter is the related task of event corefer-\nence , deciding whether two event mentions (such as the buyand the acquisition inevent\ncoreference\nthese two sentences from the ECB+ corpus) refer to the same event:\n(23.3) AMD agreed to [ buy] Markham, Ontario-based ATI for around $5.4 billion\nin cash and stock, the companies announced Monday.\n(23.4) The [ acquisition ] would turn AMD into one of the world\u2019s largest providers\nof graphics chips.\nEvent mentions are much harder to detect than entity mentions, since they can be ver-\nbal as well as nominal. Once detected, the same mention-pair and mention-ranking\nmodels used for entities are often applied to events.\nAn even more complex kind of coreference is discourse deixis (Webber, 1988), discourse deixis\nin which an anaphor refers back to a discourse segment, which can be quite hard to\ndelimit or categorize, like the examples in (23.5) adapted from Webber (1991):\n(23.5) According to Soleil, Beau just opened a restaurant\na. But thatturned out to be a lie.\nb. But thatwas false.\nc.That struck me as a funny way to describe the situation.\nThe referent of that is a speech act (see Chapter 15) in (23.5a), a proposition in\n(23.5b), and a manner of description in (23.5c). We don\u2019t give algorithms in this\nchapter for these dif\ufb01cult types of non-nominal antecedents , but see Kolhatkar\net al. (2018) for a survey. 504 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\n23.1 Coreference Phenomena: Linguistic Background\nWe now offer some linguistic background on reference phenomena. We introduce\nthe four types of referring expressions (de\ufb01nite and inde\ufb01nite NPs, pronouns, and\nnames), describe how these are used to evoke and access entities in the discourse\nmodel, and talk about linguistic features of the anaphor/antecedent relation (like\nnumber/gender agreement, or properties of verb semantics).\n23.1.1 Types of Referring Expressions\nInde\ufb01nite Noun Phrases: The most common form of inde\ufb01nite reference in En-\nglish is marked with the determiner a(oran), but it can also be marked by a quan-\nti\ufb01er such as some or even the determiner this. Inde\ufb01nite reference generally intro-\nduces into the discourse context entities that are new to the hearer.\n(23.6) a. Mrs. Martin was so very kind as to send Mrs. Goddard a beautiful goose .\nb. He had gone round one day to bring her some walnuts .\nc. I saw this beautiful cauli\ufb02ower today.\nDe\ufb01nite Noun Phrases: De\ufb01nite reference, such as via NPs that use the English\narticle the, refers to an entity that is identi\ufb01able to the hearer. An entity can be\nidenti\ufb01able to the hearer because it has been mentioned previously in the text and\nthus is already represented in the discourse model:\n(23.7) It concerns a white stallion which I have sold to an of\ufb01cer. But the pedigree\nofthe white stallion was not fully established.\nAlternatively, an entity can be identi\ufb01able because it is contained in the hearer\u2019s\nset of beliefs about the world, or the uniqueness of the object is implied by the\ndescription itself, in which case it evokes a representation of the referent into the\ndiscourse model, as in (23.9):\n(23.8) I read about it in the New York Times .\n(23.9) Have you seen the car keys?\nThese last uses are quite common; more than half of de\ufb01nite NPs in newswire\ntexts are non-anaphoric, often because they are the \ufb01rst time an entity is mentioned\n(Poesio and Vieira 1998, Bean and Riloff 1999).\nPronouns: Another form of de\ufb01nite reference is pronominalization, used for enti-\nties that are extremely salient in the discourse, (as we discuss below):\n(23.10) Emma smiled and chatted as cheerfully as shecould,\nPronouns can also participate in cataphora , in which they are mentioned before cataphora\ntheir referents are, as in (23.11).\n(23.11) Even before shesawit, Dorothy had been thinking about the Emerald City\nevery day.\nHere, the pronouns sheanditboth occur before their referents are introduced.\nPronouns also appear in quanti\ufb01ed contexts in which they are considered to be\nbound , as in (23.12). bound\n(23.12) Every dancer brought herleft arm forward.\nUnder the relevant reading, herdoes not refer to some woman in context, but instead\nbehaves like a variable bound to the quanti\ufb01ed expression every dancer . We are not\nconcerned with the bound interpretation of pronouns in this chapter. 23.1 \u2022 C OREFERENCE PHENOMENA : LINGUISTIC BACKGROUND 505\nIn some languages, pronouns can appear as clitics attached to a word, like lo\n(\u2018it\u2019) in this Spanish example from AnCora (Recasens and Mart \u00b4\u0131, 2010):\n(23.13) La intenci \u00b4on es reconocer el gran prestigio que tiene la marat \u00b4on y unir lo\ncon esta gran carrera.\n\u2018The aim is to recognize the great prestige that the Marathon has and join jit\nwith this great race.\u201d\nDemonstrative Pronouns: Demonstrative pronouns thisandthat can appear ei-\nther alone or as determiners, for instance, this ingredient ,that spice :\n(23.14) I just bought a copy of Thoreau\u2019s Walden . I had bought one \ufb01ve years ago.\nThat one had been very tattered; this one was in much better condition.\nNote that this NP is ambiguous; in colloquial spoken English, it can be inde\ufb01nite,\nas in (23.6), or de\ufb01nite, as in (23.14).\nZero Anaphora: Instead of using a pronoun, in some languages (including Chi-\nnese, Japanese, and Italian) it is possible to have an anaphor that has no lexical\nrealization at all, called a zero anaphor or zero pronoun, as in the following Italian zero anaphor\nand Japanese examples from Poesio et al. (2016):\n(23.15) EN [John] iwent to visit some friends. On the way [he] ibought some\nwine.\nIT [Giovanni] iand`o a far visita a degli amici. Per via ficompr `o del vino.\nJA [John] i-wa yujin-o houmon-sita. Tochu-de fiwain-o ka-tta.\nor this Chinese example:\n(23.16) [ \u6211]\u524d\u4e00\u4f1a\u7cbe\u795e\u4e0a\u592a\u7d27\u5f20\u3002[0]\u73b0\u5728\u6bd4\u8f83\u5e73\u9759\u4e86\n[I] was too nervous a while ago. ... [0] am now calmer.\nZero anaphors complicate the task of mention detection in these languages.\nNames: Names (such as of people, locations, or organizations) can be used to refer\nto both new and old entities in the discourse:\n(23.17) a. Miss Woodhouse certainly had not done him justice.\nb.International Business Machines sought patent compensation\nfrom Amazon; IBM had previously sued other companies.\n23.1.2 Information Status\nThe way referring expressions are used to evoke new referents into the discourse\n(introducing new information), or access old entities from the model (old informa-\ntion), is called their information status orinformation structure . Entities can beinformation\nstatus\ndiscourse-new ordiscourse-old , and indeed it is common to distinguish at least discourse-new\ndiscourse-old three kinds of entities informationally (Prince, 1981):\nnew NPs:\nbrand new NPs: these introduce entities that are discourse-new and hearer-\nnew like a fruit orsome walnuts .\nunused NPs: these introduce entities that are discourse-new but hearer-old\n(like Hong Kong ,Marie Curie , orthe New York Times .\nold NPs: also called evoked NPs , these introduce entities that already in the dis-\ncourse model, hence are both discourse-old and hearer-old, like itin \u201cI went\nto a new restaurant. It was... \u201d. 506 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\ninferrables: these introduce entities that are neither hearer-old nor discourse-old,\nbut the hearer can infer their existence by reasoning based on other entities\nthat are in the discourse. Consider the following examples:\n(23.18) I went to a superb restaurant yesterday. The chef had just opened it.\n(23.19) Mix \ufb02our, butter and water. Knead the dough until shiny.\nNeither the chef northe dough were in the discourse model based on the \ufb01rst\nsentence of either example, but the reader can make a bridging inferencebridging\ninference\nthat these entities should be added to the discourse model and associated with\nthe restaurant and the ingredients, based on world knowledge that restaurants\nhave chefs and dough is the result of mixing \ufb02our and liquid (Haviland and\nClark 1974, Webber and Baldwin 1992, Nissim et al. 2004, Hou et al. 2018).\nThe form of an NP gives strong clues to its information status. We often talk\nabout an entity\u2019s position on the given-new dimension, the extent to which the refer- given-new\nent is given (salient in the discourse, easier for the hearer to call to mind, predictable\nby the hearer), versus new (non-salient in the discourse, unpredictable) (Chafe 1976,\nPrince 1981, Gundel et al. 1993). A referent that is very accessible (Ariel, 2001) accessible\ni.e., very salient in the hearer\u2019s mind or easy to call to mind, can be referred to with\nless linguistic material. For example pronouns are used only when the referent has\na high degree of activation or salience in the discourse model.4By contrast, less salience\nsalient entities, like a new referent being introduced to the discourse, will need to be\nintroduced with a longer and more explicit referring expression to help the hearer\nrecover the referent.\nThus when an entity is \ufb01rst introduced into a discourse its mentions are likely\nto have full names, titles or roles, or appositive or restrictive relative clauses, as in\nthe introduction of our protagonist in (23.1): Victoria Chen, CFO of Megabucks\nBanking . As an entity is discussed over a discourse, it becomes more salient to the\nhearer and its mentions on average typically becomes shorter and less informative,\nfor example with a shortened name (for example Ms. Chen ), a de\ufb01nite description\n(the 38-year-old ), or a pronoun ( sheorher) (Hawkins 1978). However, this change\nin length is not monotonic, and is sensitive to discourse structure (Grosz 1977b,\nReichman 1985, Fox 1993).\n23.1.3 Complications: Non-Referring Expressions\nMany noun phrases or other nominals are not referring expressions, although they\nmay bear a confusing super\ufb01cial resemblance. For example in some of the earliest\ncomputational work on reference resolution, Karttunen (1969) pointed out that the\nNPa car in the following example does not create a discourse referent:\n(23.20) Janet doesn\u2019t have a car .\nand cannot be referred back to by anaphoric itorthe car :\n(23.21) * Itis a Toyota.\n(23.22) * The car is red.\nWe summarize here four common types of structures that are not counted as men-\ntions in coreference tasks and hence complicate the task of mention-detection:\n4Pronouns also usually (but not always) refer to entities that were introduced no further than one or two\nsentences back in the ongoing discourse, whereas de\ufb01nite noun phrases can often refer further back. 23.1 \u2022 C OREFERENCE PHENOMENA : LINGUISTIC BACKGROUND 507\nAppositives: An appositional structure is a noun phrase that appears next to a\nhead noun phrase, describing the head. In English they often appear in commas, like\n\u201ca unit of UAL\u201d appearing in apposition to the NP United , orCFO of Megabucks\nBanking in apposition to Victoria Chen .\n(23.23) Victoria Chen, CFO of Megabucks Banking, saw ...\n(23.24) United, a unit of UAL, matched the fares.\nAppositional NPs are not referring expressions, instead functioning as a kind of\nsupplementary parenthetical description of the head NP. Nonetheless, sometimes it\nis useful to link these phrases to an entity they describe, and so some datasets like\nOntoNotes mark appositional relationships.\nPredicative and Prenominal NPs: Predicative or attributive NPs describe prop-\nerties of the head noun. In United is a unit of UAL , the NP a unit of UAL describes\na property of United, rather than referring to a distinct entity. Thus they are not\nmarked as mentions in coreference tasks; in our example the NPs $2.3 million and\nthe company\u2019s president , are attributive, describing properties of her pay andthe\n38-year-old ; Example (23.27) shows a Chinese example in which the predicate NP\n(\u4e2d\u56fd\u6700\u5927\u7684\u57ce\u5e02; China\u2019s biggest city ) is not a mention.\n(23.25) her pay jumped to $2.3 million\n(23.26) the 38-year-old became the company\u2019s president\n(23.27)\u4e0a\u6d77\u662f[\u4e2d\u56fd\u6700\u5927\u7684\u57ce\u5e02] [Shanghai is China\u2019s biggest city ]\nExpletives: Many uses of pronouns like itin English and corresponding pronouns\nin other languages are not referential. Such expletive orpleonastic cases include expletive\nit is raining , in idioms like hit it off , or in particular syntactic situations like clefts clefts\n(23.28a) or extraposition (23.28b):\n(23.28) a. Itwas Emma Goldman who founded Mother Earth\nb.Itsurprised me that there was a herring hanging on her wall.\nGenerics: Another kind of expression that does not refer back to an entity explic-\nitly evoked in the text is generic reference. Consider (23.29).\n(23.29) I love mangos. They are very tasty.\nHere, they refers, not to a particular mango or set of mangos, but instead to the class\nof mangos in general. The pronoun youcan also be used generically:\n(23.30) In July in San Francisco youhave to wear a jacket.\n23.1.4 Linguistic Properties of the Coreference Relation\nNow that we have seen the linguistic properties of individual referring expressions\nwe turn to properties of the antecedent/anaphor pair. Understanding these properties\nis helpful both in designing novel features and performing error analyses.\nNumber Agreement: Referring expressions and their referents must generally\nagree in number; English she/her/he/him/his/it are singular, we/us/they/them are plu-\nral, and youis unspeci\ufb01ed for number. So a plural antecedent like the chefs cannot\ngenerally corefer with a singular anaphor like she. However, algorithms cannot\nenforce number agreement too strictly. First, semantically plural entities can be re-\nferred to by either itorthey:\n(23.31) IBM announced a new machine translation product yesterday. They have\nbeen working on it for 20 years. 508 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\nSecond, singular they has become much more common, in which they is used to singular they\ndescribe singular individuals, often useful because they is gender neutral. Although\nrecently increasing, singular they is quite old, part of English for many centuries.5\nPerson Agreement: English distinguishes between \ufb01rst, second, and third person,\nand a pronoun\u2019s antecedent must agree with the pronoun in person. Thus a third\nperson pronoun ( he, she, they, him, her, them, his, her, their ) must have a third person\nantecedent (one of the above or any other noun phrase). However, phenomena like\nquotation can cause exceptions; in this example I,my, and sheare coreferent:\n(23.32) \u201cI voted for Nader because he was most aligned with my values,\u201d she said.\nGender or Noun Class Agreement: In many languages, all nouns have grammat-\nical gender or noun class6and pronouns generally agree with the grammatical gender\nof their antecedent. In English this occurs only with third-person singular pronouns,\nwhich distinguish between male (he, him, his ),female (she, her ), and nonpersonal\n(it) grammatical genders. Non-binary pronouns like zeorhirmay also occur in more\nrecent texts. Knowing which gender to associate with a name in text can be complex,\nand may require world knowledge about the individual. Some examples:\n(23.33) Maryam has a theorem. She is exciting. (she=Maryam, not the theorem)\n(23.34) Maryam has a theorem. It is exciting. (it=the theorem, not Maryam)\nBinding Theory Constraints: The binding theory is a name for syntactic con-\nstraints on the relations between a mention and an antecedent in the same sentence\n(Chomsky, 1981). Oversimplifying a bit, re\ufb02exive pronouns like himself andher- re\ufb02exive\nselfcorefer with the subject of the most immediate clause that contains them (23.35),\nwhereas nonre\ufb02exives cannot corefer with this subject (23.36).\n(23.35) Janet bought herself a bottle of \ufb01sh sauce. [herself =Janet]\n(23.36) Janet bought her a bottle of \ufb01sh sauce. [her 6=Janet]\nRecency: Entities introduced in recent utterances tend to be more salient than\nthose introduced from utterances further back. Thus, in (23.37), the pronoun itis\nmore likely to refer to Jim\u2019s map than the doctor\u2019s map.\n(23.37) The doctor found an old map in the captain\u2019s chest. Jim found an even\nolder map hidden on the shelf. It described an island.\nGrammatical Role: Entities mentioned in subject position are more salient than\nthose in object position, which are in turn more salient than those mentioned in\noblique positions. Thus although the \ufb01rst sentence in (23.38) and (23.39) expresses\nroughly the same propositional content, the preferred referent for the pronoun he\nvaries with the subject\u2014John in (23.38) and Bill in (23.39).\n(23.38) Billy Bones went to the bar with Jim Hawkins. He called for a glass of\nrum. [ he = Billy ]\n(23.39) Jim Hawkins went to the bar with Billy Bones. He called for a glass of\nrum. [ he = Jim ]\n5Here\u2019s a bound pronoun example from Shakespeare\u2019s Comedy of Errors :There\u2019s not a man I meet but\ndoth salute me As if I were their well-acquainted friend\n6The word \u201cgender\u201d is generally only used for languages with 2 or 3 noun classes, like most Indo-\nEuropean languages; many languages, like the Bantu languages or Chinese, have a much larger number\nof noun classes. 23.2 \u2022 C OREFERENCE TASKS AND DATASETS 509\nVerb Semantics: Some verbs semantically emphasize one of their arguments, bi-\nasing the interpretation of subsequent pronouns. Compare (23.40) and (23.41).\n(23.40) John telephoned Bill. He lost the laptop.\n(23.41) John criticized Bill. He lost the laptop.\nThese examples differ only in the verb used in the \ufb01rst sentence, yet \u201che\u201d in (23.40)\nis typically resolved to John, whereas \u201che\u201d in (23.41) is resolved to Bill. This may\nbe partly due to the link between implicit causality and saliency: the implicit cause\nof a \u201ccriticizing\u201d event is its object, whereas the implicit cause of a \u201ctelephoning\u201d\nevent is its subject. In such verbs, the entity which is the implicit cause may be more\nsalient.\nSelectional Restrictions: Many other kinds of semantic knowledge can play a role\nin referent preference. For example, the selectional restrictions that a verb places on\nits arguments (Chapter 21) can help eliminate referents, as in (23.42).\n(23.42) I ate the soup in my new bowl after cooking it for hours\nThere are two possible referents for it, the soup and the bowl. The verb eat, however,\nrequires that its direct object denote something edible, and this constraint can rule\noutbowl as a possible referent.\n23.2 Coreference Tasks and Datasets\nWe can formulate the task of coreference resolution as follows: Given a text T, \ufb01nd\nall entities and the coreference links between them. We evaluate our task by com-\nparing the links our system creates with those in human-created gold coreference\nannotations on T.\nLet\u2019s return to our coreference example, now using superscript numbers for each\ncoreference chain (cluster), and subscript letters for individual mentions in the clus-\nter:\n(23.43) [Victoria Chen]1\na, CFO of [Megabucks Banking]2\na, saw [[her]1\nbpay]3\najump\nto $2.3 million, as [the 38-year-old]1\ncalso became [[the company]2\nb\u2019s\npresident. It is widely known that [she]1\ndcame to [Megabucks]2\ncfrom rival\n[Lotsabucks]4\na.\nAssuming example (23.43) was the entirety of the article, the chains for her pay and\nLotsabucks are singleton mentions:\n1.fVictoria Chen ,her,the 38-year-old ,Sheg\n2.fMegabucks Banking ,the company ,Megabucksg\n3.fher payg\n4.fLotsabucksg\nFor most coreference evaluation campaigns, the input to the system is the raw\ntext of articles, and systems must detect mentions and then link them into clusters.\nSolving this task requires dealing with pronominal anaphora (\ufb01guring out that her\nrefers to Victoria Chen ), \ufb01ltering out non-referential pronouns like the pleonastic It\ninIt has been ten years ), dealing with de\ufb01nite noun phrases to \ufb01gure out that the\n38-year-old is coreferent with Victoria Chen , and that the company is the same as\nMegabucks . And we need to deal with names, to realize that Megabucks is the same\nasMegabucks Banking . 510 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\nExactly what counts as a mention and what links are annotated differs from task\nto task and dataset to dataset. For example some coreference datasets do not label\nsingletons, making the task much simpler. Resolvers can achieve much higher scores\non corpora without singletons, since singletons constitute the majority of mentions in\nrunning text, and they are often hard to distinguish from non-referential NPs. Some\ntasks use gold mention-detection (i.e. the system is given human-labeled mention\nboundaries and the task is just to cluster these gold mentions), which eliminates the\nneed to detect and segment mentions from running text.\nCoreference is usually evaluated by the CoNLL F1 score, which combines three\nmetrics: MUC, B3, and CEAF e; Section 23.8 gives the details.\nLet\u2019s mention a few characteristics of one popular coreference dataset, OntoNotes\n(Pradhan et al. 2007c, Pradhan et al. 2007a), and the CoNLL 2012 Shared Task\nbased on it (Pradhan et al., 2012a). OntoNotes contains hand-annotated Chinese\nand English coreference datasets of roughly one million words each, consisting of\nnewswire, magazine articles, broadcast news, broadcast conversations, web data and\nconversational speech data, as well as about 300,000 words of annotated Arabic\nnewswire. The most important distinguishing characteristic of OntoNotes is that\nit does not label singletons, simplifying the coreference task, since singletons rep-\nresent 60%-70% of all entities. In other ways, it is similar to other coreference\ndatasets. Referring expression NPs that are coreferent are marked as mentions, but\ngenerics and pleonastic pronouns are not marked. Appositive clauses are not marked\nas separate mentions, but they are included in the mention. Thus in the NP, \u201cRichard\nGodown, president of the Industrial Biotechnology Association\u201d the mention is the\nentire phrase. Prenominal modi\ufb01ers are annotated as separate entities only if they\nare proper nouns. Thus wheat is not an entity in wheat \ufb01elds , but UNis an entity in\nUN policy (but not adjectives like American inAmerican policy ).\nA number of corpora mark richer discourse phenomena. The ISNotes corpus\nannotates a portion of OntoNotes for information status, include bridging examples\n(Hou et al., 2018). The LitBank coreference corpus (Bamman et al., 2020) contains\ncoreference annotations for 210,532 tokens from 100 different literary novels, in-\ncluding singletons and quanti\ufb01ed and negated noun phrases. The AnCora-CO coref-\nerence corpus (Recasens and Mart \u00b4\u0131, 2010) contains 400,000 words each of Spanish\n(AnCora-CO-Es) and Catalan (AnCora-CO-Ca) news data, and includes labels for\ncomplex phenomena like discourse deixis in both languages. The ARRAU corpus\n(Uryupina et al., 2020) contains 350,000 words of English marking all NPs, which\nmeans singleton clusters are available. ARRAU includes diverse genres like dialog\n(the TRAINS data) and \ufb01ction (the Pear Stories), and has labels for bridging refer-\nences, discourse deixis, generics, and ambiguous anaphoric relations.\n23.3 Mention Detection\nThe \ufb01rst stage of coreference is mention detection : \ufb01nding the spans of text thatmention\ndetection\nconstitute each mention. Mention detection algorithms are usually very liberal in\nproposing candidate mentions (i.e., emphasizing recall), and only \ufb01ltering later. For\nexample many systems run parsers and named entity taggers on the text and extract\nevery span that is either an NP, apossessive pronoun , or a named entity .\nDoing so from our sample text repeated in (23.44):\n(23.44) Victoria Chen, CFO of Megabucks Banking, saw her pay jump to $2.3 23.3 \u2022 M ENTION DETECTION 511\nmillion, as the 38-year-old also became the company\u2019s president. It is\nwidely known that she came to Megabucks from rival Lotsabucks.\nmight result in the following list of 13 potential mentions:\nVictoria Chen $2.3 million she\nCFO of Megabucks Banking the 38-year-old Megabucks\nMegabucks Banking the company Lotsabucks\nher the company\u2019s president\nher pay It\nMore recent mention detection systems are even more generous; the span-based\nalgorithm we will describe in Section 23.6 \ufb01rst extracts literally all n-gram spans\nof words up to N=10. Of course recall from Section 23.1.3 that many NPs\u2014and\nthe overwhelming majority of random n-gram spans\u2014are not referring expressions.\nTherefore all such mention detection systems need to eventually \ufb01lter out pleonas-\ntic/expletive pronouns like Itabove, appositives like CFO of Megabucks Banking\nInc, or predicate nominals like the company\u2019s president or$2.3 million .\nSome of this \ufb01ltering can be done by rules. Early rule-based systems designed\nregular expressions to deal with pleonastic it, like the following rules from Lappin\nand Leass (1994) that use dictionaries of cognitive verbs (e.g., believe ,know ,antic-\nipate ) to capture pleonastic itin \u201cIt is thought that ketchup...\u201d, or modal adjectives\n(e.g., necessary ,possible ,certain ,important ), for, e.g., \u201cIt is likely that I...\u201d. Such\nrules are sometimes used as part of modern systems:\nIt is Modaladjective that S\nIt is Modaladjective (for NP) to VP\nIt is Cogv-ed that S\nIt seems/appears/means/follows (that) S\nMention-detection rules are sometimes designed speci\ufb01cally for particular eval-\nuation campaigns. For OntoNotes, for example, mentions are not embedded within\nlarger mentions, and while numeric quantities are annotated, they are rarely coref-\nerential. Thus for OntoNotes tasks like CoNLL 2012 (Pradhan et al., 2012a), a\ncommon \ufb01rst pass rule-based mention detection algorithm (Lee et al., 2013) is:\n1.Take all NPs, possessive pronouns, and named entities.\n2.Remove numeric quantities (100 dollars, 8%), mentions embedded in\nlarger mentions, adjectival forms of nations, and stop words (like there ).\n3.Remove pleonastic itbased on regular expression patterns.\nRule-based systems, however, are generally insuf\ufb01cient to deal with mention-\ndetection, and so modern systems incorporate some sort of learned mention detec-\ntion component, such as a referentiality classi\ufb01er, an anaphoricity classi\ufb01er \u2014\ndetecting whether an NP is an anaphor\u2014or a discourse-new classi\ufb01er\u2014 detecting\nwhether a mention is discourse-new and a potential antecedent for a future anaphor.\nAnanaphoricity detector , for example, can draw its positive training examplesanaphoricity\ndetector\nfrom any span that is labeled as an anaphoric referring expression in hand-labeled\ndatasets like OntoNotes, ARRAU , or AnCora. Any other NP or named entity can be\nmarked as a negative training example. Anaphoricity classi\ufb01ers use features of the\ncandidate mention such as its head word, surrounding words, de\ufb01niteness, animacy,\nlength, position in the sentence/discourse, many of which were \ufb01rst proposed in\nearly work by Ng and Cardie (2002a); see Section 23.5 for more on features. 512 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\nReferentiality or anaphoricity detectors can be run as \ufb01lters, in which only men-\ntions that are classi\ufb01ed as anaphoric or referential are passed on to the coreference\nsystem. The end result of such a \ufb01ltering mention detection system on our example\nabove might be the following \ufb01ltered set of 9 potential mentions:\nVictoria Chen her pay she\nMegabucks Bank the 38-year-old Megabucks\nher the company Lotsabucks\nIt turns out, however, that hard \ufb01ltering of mentions based on an anaphoricity\nor referentiality classi\ufb01er leads to poor performance. If the anaphoricity classi\ufb01er\nthreshold is set too high, too many mentions are \ufb01ltered out and recall suffers. If the\nclassi\ufb01er threshold is set too low, too many pleonastic or non-referential mentions\nare included and precision suffers.\nThe modern approach is instead to perform mention detection, anaphoricity, and\ncoreference jointly in a single end-to-end model (Ng 2005b, Denis and Baldridge\n2007, Rahman and Ng 2009). For example mention detection in the Lee et al.\n(2017b),2018 system is based on a single end-to-end neural network that computes\na score for each mention being referential, a score for two mentions being corefer-\nence, and combines them to make a decision, training all these scores with a single\nend-to-end loss. We\u2019ll describe this method in detail in Section 23.6.7\nDespite these advances, correctly detecting referential mentions seems to still be\nan unsolved problem, since systems incorrectly marking pleonastic pronouns like\nitand other non-referential NPs as coreferent is a large source of errors of modern\ncoreference resolution systems (Kummerfeld and Klein 2013, Martschat and Strube\n2014, Martschat and Strube 2015, Wiseman et al. 2015, Lee et al. 2017a).\nMention, referentiality, or anaphoricity detection is thus an important open area\nof investigation. Other sources of knowledge may turn out to be helpful, especially\nin combination with unsupervised and semisupervised algorithms, which also mit-\nigate the expense of labeled datasets. In early work, for example Bean and Riloff\n(1999) learned patterns for characterizing anaphoric or non-anaphoric NPs; (by ex-\ntracting and generalizing over the \ufb01rst NPs in a text, which are guaranteed to be\nnon-anaphoric). Chang et al. (2012) look for head nouns that appear frequently in\nthe training data but never appear as gold mentions to help \ufb01nd non-referential NPs.\nBergsma et al. (2008b) use web counts as a semisupervised way to augment standard\nfeatures for anaphoricity detection for English it, an important task because itis both\ncommon and ambiguous; between a quarter and half itexamples are non-anaphoric.\nConsider the following two examples:\n(23.45) You can make [it] in advance. [anaphoric]\n(23.46) You can make [it] in Hollywood. [non-anaphoric]\nTheitinmake it is non-anaphoric, part of the idiom make it . Bergsma et al. (2008b)\nturn the context around each example into patterns, like \u201cmake * in advance\u201d from\n(23.45), and \u201cmake * in Hollywood\u201d from (23.46). They then use Google n-grams to\nenumerate all the words that can replace itin the patterns. Non-anaphoric contexts\ntend to only have itin the wildcard positions, while anaphoric contexts occur with\nmany other NPs (for example make them in advance is just as frequent in their data\n7Some systems try to avoid mention detection or anaphoricity detection altogether. For datasets like\nOntoNotes which don\u2019t label singletons, an alternative to \ufb01ltering out non-referential mentions is to run\ncoreference resolution, and then simply delete any candidate mentions which were not corefered with\nanother mention. This likely doesn\u2019t work as well as explicitly modeling referentiality, and cannot solve\nthe problem of detecting singletons, which is important for tasks like entity linking. 23.4 \u2022 A RCHITECTURES FOR COREFERENCE ALGORITHMS 513\nasmake it in advance , but make them in Hollywood did not occur at all). These\nn-gram contexts can be used as features in a supervised anaphoricity classi\ufb01er.\n23.4 Architectures for Coreference Algorithms\nModern systems for coreference are based on supervised neural machine learning,\nsupervised from hand-labeled datasets like OntoNotes. In this section we overview\nthe various architecture of modern systems, using the categorization of Ng (2010),\nwhich distinguishes algorithms based on whether they make each coreference deci-\nsion in a way that is entity-based \u2014representing each entity in the discourse model\u2014\nor only mention-based \u2014considering each mention independently, and whether they\nuseranking models to directly compare potential antecedents. Afterwards, we go\ninto more detail on one state-of-the-art algorithm in Section 23.6.\n23.4.1 The Mention-Pair Architecture\nWe begin with the mention-pair architecture, the simplest and most in\ufb02uential mention-pair\ncoreference architecture, which introduces many of the features of more complex\nalgorithms, even though other architectures perform better. The mention-pair ar- mention-pair\nchitecture is based around a classi\ufb01er that\u2014 as its name suggests\u2014is given a pair\nof mentions, a candidate anaphor and a candidate antecedent, and makes a binary\nclassi\ufb01cation decision: coreferring or not.\nLet\u2019s consider the task of this classi\ufb01er for the pronoun shein our example, and\nassume the slightly simpli\ufb01ed set of potential antecedents in Fig. 23.2.\nVictoria ChenMegabucks Bankingherher paythe 37-year-oldshep(coref|\u201dVictoria Chen\u201d,\u201dshe\u201d)\np(coref|\u201dMegabucks Banking\u201d,\u201dshe\u201d)\nFigure 23.2 For each pair of a mention (like she), and a potential antecedent mention (like\nVictoria Chen orher), the mention-pair classi\ufb01er assigns a probability of a coreference link.\nFor each prior mention ( Victoria Chen ,Megabucks Banking ,her, etc.), the binary\nclassi\ufb01er computes a probability: whether or not the mention is the antecedent of\nshe. We want this probability to be high for actual antecedents ( Victoria Chen ,her,\nthe 38-year-old ) and low for non-antecedents ( Megabucks Banking ,her pay ).\nEarly classi\ufb01ers used hand-built features (Section 23.5); more recent classi\ufb01ers\nuse neural representation learning (Section 23.6)\nFor training, we need a heuristic for selecting training samples; since most pairs\nof mentions in a document are not coreferent, selecting every pair would lead to\na massive overabundance of negative samples. The most common heuristic, from\n(Soon et al., 2001), is to choose the closest antecedent as a positive example, and all\npairs in between as the negative examples. More formally, for each anaphor mention\nmiwe create\n\u2022 one positive instance ( mi;mj) where mjis the closest antecedent to mi, and 514 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\n\u2022 a negative instance ( mi;mk) for each mkbetween mjandmi\nThus for the anaphor she, we would choose ( she,her) as the positive example\nand no negative examples. Similarly, for the anaphor the company we would choose\n(the company ,Megabucks ) as the positive example and ( the company ,she) (the com-\npany ,the 38-year-old ) (the company ,her pay ) and ( the company ,her) as negative\nexamples.\nOnce the classi\ufb01er is trained, it is applied to each test sentence in a clustering\nstep. For each mention iin a document, the classi\ufb01er considers each of the prior i\u00001\nmentions. In closest-\ufb01rst clustering (Soon et al., 2001), the classi\ufb01er is run right to\nleft (from mention i\u00001 down to mention 1) and the \ufb01rst antecedent with probability\n>:5 is linked to i. If no antecedent has probably >0:5, no antecedent is selected for\ni. Inbest-\ufb01rst clustering, the classi\ufb01er is run on all i\u00001 antecedents and the most\nprobable preceding mention is chosen as the antecedent for i. The transitive closure\nof the pairwise relation is taken as the cluster.\nWhile the mention-pair model has the advantage of simplicity, it has two main\nproblems. First, the classi\ufb01er doesn\u2019t directly compare candidate antecedents to\neach other, so it\u2019s not trained to decide, between two likely antecedents, which one\nis in fact better. Second, it ignores the discourse model, looking only at mentions,\nnot entities. Each classi\ufb01er decision is made completely locally to the pair, without\nbeing able to take into account other mentions of the same entity. The next two\nmodels each address one of these two \ufb02aws.\n23.4.2 The Mention-Rank Architecture\nThe mention ranking model directly compares candidate antecedents to each other,\nchoosing the highest-scoring antecedent for each anaphor.\nIn early formulations, for mention i, the classi\ufb01er decides which of the f1;:::;i\u0000\n1gprior mentions is the antecedent (Denis and Baldridge, 2008). But suppose iis\nin fact not anaphoric, and none of the antecedents should be chosen? Such a model\nwould need to run a separate anaphoricity classi\ufb01er on i. Instead, it turns out to be\nbetter to jointly learn anaphoricity detection and coreference together with a single\nloss (Rahman and Ng, 2009).\nSo in modern mention-ranking systems, for the ith mention (anaphor), we have\nan associated random variable yiranging over the values Y(i) =f1;:::;i\u00001;\u000fg. The\nvalue\u000fis a special dummy mention meaning that idoes not have an antecedent (i.e.,\nis either discourse-new and starts a new coref chain, or is non-anaphoric).\nVictoria ChenMegabucks Bankingherher paythe 37-year-oldshep(\u201dVictoria Chen\u201d|\u201dshe\u201d)\np(\u03f5|\u201dshe\u201d)\u03f5One or more of theseshould be highAll of theseshould be low}p(\u201dher pay\u201d|she\u201d)p(\u201dher\u201d|she\u201d)p(\u201dthe 37-year-old\u201d|she\u201d)\np(\u201dMegabucks Banking\u201d|she\u201d)}\nFigure 23.3 For each candidate anaphoric mention (like she), the mention-ranking system assigns a proba-\nbility distribution over all previous mentions plus the special dummy mention \u000f.\nAt test time, for a given mention ithe model computes one softmax over all the\nantecedents (plus \u000f) giving a probability for each candidate antecedent (or none). 23.5 \u2022 C LASSIFIERS USING HAND -BUILT FEATURES 515\nFig. 23.3 shows an example of the computation for the single candidate anaphor\nshe.\nOnce the antecedent is classi\ufb01ed for each anaphor, transitive closure can be run\nover the pairwise decisions to get a complete clustering.\nTraining is trickier in the mention-ranking model than the mention-pair model,\nbecause for each anaphor we don\u2019t know which of all the possible gold antecedents\nto use for training. Instead, the best antecedent for each mention is latent ; that\nis, for each mention we have a whole cluster of legal gold antecedents to choose\nfrom. Early work used heuristics to choose an antecedent, for example choosing the\nclosest antecedent as the gold antecedent and all non-antecedents in a window of\ntwo sentences as the negative examples (Denis and Baldridge, 2008). Various kinds\nof ways to model latent antecedents exist (Fernandes et al. 2012, Chang et al. 2013,\nDurrett and Klein 2013). The simplest way is to give credit to any legal antecedent\nby summing over all of them, with a loss function that optimizes the likelihood of\nall correct antecedents from the gold clustering (Lee et al., 2017b). We\u2019ll see the\ndetails in Section 23.6.\nMention-ranking models can be implemented with hand-build features or with\nneural representation learning (which might also incorporate some hand-built fea-\ntures). we\u2019ll explore both directions in Section 23.5 and Section 23.6.\n23.4.3 Entity-based Models\nBoth the mention-pair and mention-ranking models make their decisions about men-\ntions . By contrast, entity-based models link each mention not to a previous mention\nbut to a previous discourse entity (cluster of mentions).\nA mention-ranking model can be turned into an entity-ranking model simply\nby having the classi\ufb01er make its decisions over clusters of mentions rather than\nindividual mentions (Rahman and Ng, 2009).\nFor traditional feature-based models, this can be done by extracting features over\nclusters. The size of a cluster is a useful feature, as is its \u2018shape\u2019, which is the\nlist of types of the mentions in the cluster i.e., sequences of the tokens (P)roper,\n(D)e\ufb01nite, (I)nde\ufb01nite, (Pr)onoun, so that a cluster composed of fVictoria ,her,the\n38-year-oldgwould have the shape P-Pr-D (Bj\u00a8orkelund and Kuhn, 2014). An entity-\nbased model that includes a mention-pair classi\ufb01er can use as features aggregates of\nmention-pair probabilities, for example computing the average probability of coref-\nerence over all mention-pairs in the two clusters (Clark and Manning 2015).\nNeural models can learn representations of clusters automatically, for example\nby using an RNN over the sequence of cluster mentions to encode a state correspond-\ning to a cluster representation (Wiseman et al., 2016), or by learning distributed rep-\nresentations for pairs of clusters by pooling over learned representations of mention\npairs (Clark and Manning, 2016b).\nHowever, although entity-based models are more expressive, the use of cluster-\nlevel information in practice has not led to large gains in performance, so mention-\nranking models are still more commonly used.\n23.5 Classi\ufb01ers using hand-built features\nFeature-based classi\ufb01ers, use hand-designed features in logistic regression, SVM,\nor random forest classi\ufb01ers for coreference resolution. These classi\ufb01ers don\u2019t per- 516 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\nform as well as neural ones. Nonetheless, they are still sometimes useful to build\nlightweight systems when compute or data are sparse, and the features themselves\nare useful for error analysis even in neural systems.\nGiven an anaphor mention and a potential antecedent mention, feature based\nclassi\ufb01ers make use of three types of features: (i) features of the anaphor, (ii) features\nof the candidate antecedent, and (iii) features of the relationship between the pair.\nEntity-based models can make additional use of two additional classes: (iv) feature\nof all mentions from the antecedent\u2019s entity cluster, and (v) features of the relation\nbetween the anaphor and the mentions in the antecedent entity cluster.\nFeatures of the Anaphor or Antecedent Mention\nFirst (last) word Victoria/she First or last word (or embedding) of antecedent/anaphor\nHead word Victoria/she Head word (or head embedding) of antecedent/anaphor\nAttributes Sg-F-A-3-PER/\nSg-F-A-3-PERThe number, gender, animacy, person, named entity type\nattributes of (antecedent/anaphor)\nLength 2/1 length in words of (antecedent/anaphor)\nMention type P/Pr Type: (P)roper, (D)e\ufb01nite, (I)nde\ufb01nite, (Pr)onoun) of an-\ntecedent/anaphor\nFeatures of the Antecedent Entity\nEntity shape P-Pr-D The \u2018shape\u2019 or list of types of the mentions in the\nantecedent entity (cluster), i.e., sequences of (P)roper,\n(D)e\ufb01nite, (I)nde\ufb01nite, (Pr)onoun.\nEntity attributes Sg-F-A-3-PER The number, gender, animacy, person, named entity type\nattributes of the antecedent entity\nAnt. cluster size 3 Number of mentions in the antecedent cluster\nFeatures of the Pair of Mentions\nSentence distance 1 The number of sentences between antecedent and anaphor\nMention distance 4 The number of mentions between antecedent and anaphor\ni-within-i F Anaphor has i-within-i relation with antecedent\nCosine Cosine between antecedent and anaphor embeddings\nFeatures of the Pair of Entities\nExact String Match F True if the strings of any two mentions from the antecedent\nand anaphor clusters are identical.\nHead Word Match F True if any mentions from antecedent cluster has same\nheadword as any mention in anaphor cluster\nWord Inclusion F All words in anaphor cluster included in antecedent cluster\nFigure 23.4 Feature-based coreference: sample feature values for anaphor \u201cshe\u201d and potential antecedent\n\u201cVictoria Chen\u201d.\nFigure 23.4 shows a selection of commonly used features, and shows the value\nthat would be computed for the potential anaphor \u201cshe\u201d and potential antecedent\n\u201cVictoria Chen\u201d in our example sentence, repeated below:\n(23.47) Victoria Chen , CFO of Megabucks Banking, saw her pay jump to $2.3\nmillion, as the 38-year-old also became the company\u2019s president. It is\nwidely known that shecame to Megabucks from rival Lotsabucks.\nFeatures that prior work has found to be particularly useful are exact string\nmatch, entity headword agreement, mention distance, as well as (for pronouns) exact\nattribute match and i-within-i, and (for nominals and proper names) word inclusion\nand cosine. For lexical features (like head words) it is common to only use words\nthat appear enough times ( >20 times). 23.6 \u2022 A NEURAL MENTION -RANKING ALGORITHM 517\nIt is crucial in feature-based systems to use conjunctions of features; one exper-\niment suggested that moving from individual features in a classi\ufb01er to conjunctions\nof multiple features increased F1 by 4 points (Lee et al., 2017a). Speci\ufb01c conjunc-\ntions can be designed by hand (Durrett and Klein, 2013), all pairs of features can be\nconjoined (Bengtson and Roth, 2008), or feature conjunctions can be learned using\ndecision tree or random forest classi\ufb01ers (Ng and Cardie 2002a, Lee et al. 2017a).\nFeatures can also be used in neural models as well. Neural systems use contex-\ntual word embeddings so don\u2019t bene\ufb01t from shallow features like string match or or\nmention types. However features like mention length, distance between mentions,\nor genre can complement neural contextual embedding models.\n23.6 A neural mention-ranking algorithm\nIn this section we describe the neural e2e-coref algorithms of Lee et al. (2017b)\n(simpli\ufb01ed and extended a bit, drawing on Joshi et al. (2019) and others). This is\namention-ranking algorithm that considers all possible spans of text in the docu-\nment, assigns a mention-score to each span, prunes the mentions based on this score,\nthen assigns coreference links to the remaining mentions.\nMore formally, given a document Dwith Twords, the model considers all of\ntheT(T+1)\n2text spans in D(unigrams, bigrams, trigrams, 4-grams, etc; in practice\nwe only consider spans up a maximum length around 10). The task is to assign\nto each span ian antecedent yi, a random variable ranging over the values Y(i) =\nf1;:::;i\u00001;\u000fg; each previous span and a special dummy token \u000f. Choosing the\ndummy token means that idoes not have an antecedent, either because iis discourse-\nnew and starts a new coreference chain, or because iis non-anaphoric.\nFor each pair of spans iand j, the system assigns a score s(i;j)for the coref-\nerence link between span iand span j. The system then learns a distribution P(yi)\nover the antecedents for span i:\nP(yi) =exp(s(i;yi))P\ny02Y(i)exp(s(i;y0))(23.48)\nThis score s(i;j)includes three factors that we\u2019ll de\ufb01ne below: m(i); whether span\niis a mention; m(j); whether span jis a mention; and c(i;j); whether jis the\nantecedent of i:\ns(i;j) =m(i)+m(j)+c(i;j) (23.49)\nFor the dummy antecedent \u000f, the score s(i;\u000f)is \ufb01xed to 0. This way if any non-\ndummy scores are positive, the model predicts the highest-scoring antecedent, but if\nall the scores are negative it abstains.\n23.6.1 Computing span representations\nTo compute the two functions m(i)andc(i;j)which score a span ior a pair of spans\n(i;j), we\u2019ll need a way to represent a span. The e2e-coref family of algorithms\nrepresents each span by trying to capture 3 words/tokens: the \ufb01rst word, the last\nword, and the most important word. We \ufb01rst run each paragraph or subdocument\nthrough an encoder (like BERT) to generate embeddings hifor each token i. The\nspan iis then represented by a vector githat is a concatenation of the encoder output 518 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\nembedding for the \ufb01rst (start) token of the span, the encoder output for the last (end)\ntoken of the span, and a third vector which is an attention-based representation:\ngi= [hSTART (i);hEND (i);hATT(i)] (23.50)\nThe goal of the attention vector is to represent which word/token is the likely\nsyntactic head-word of the span; we saw in the prior section that head-words are\na useful feature; a matching head-word is a good indicator of coreference. The\nattention representation is computed as usual; the system learns a weight vector wa,\nand computes its dot product with the hidden state httransformed by a FFN:\nat=wa\u0001FFNa(ht) (23.51)\nThe attention score is normalized into a distribution via a softmax:\nai;t=exp(at)\nPEND (i)\nk=START (i)exp(ak)(23.52)\nAnd then the attention distribution is used to create a vector hATT(i)which is an\nattention-weighted sum of the embeddings etof each of the words in span i:\nhATT(i)=END (i)X\nt=START (i)ai;t\u0001et (23.53)\nFig. 23.5 shows the computation of the span representation and the mention\nscore.\nEncodings (h)\u2026Encoder GeneralElectricsaidthePostalServicecontactedthecompanySpan head (hATT) Span representation (g) Mention score (m)+++++General ElectricElectric said thethe Postal ServiceService contacted thethe company\nFigure 23.5 Computation of the span representation g(and the mention score m) in a BERT version of the\ne2e-coref model (Lee et al. 2017b, Joshi et al. 2019). The model considers all spans up to a maximum width of\nsay 10; the \ufb01gure shows a small subset of the bigram and trigram spans.\n23.6.2 Computing the mention and antecedent scores mandc\nNow that we know how to compute the vector gifor representing span i, we can\nsee the details of the two scoring functions m(i)andc(i;j). Both are computed by\nfeedforward networks:\nm(i) = wm\u0001FFN m(gi) (23.54)\nc(i;j) = wc\u0001FFN c([gi;gj;gi\u000egj;]) (23.55)\nAt inference time, this mention score mis used as a \ufb01lter to keep only the best few\nmentions. 23.6 \u2022 A NEURAL MENTION -RANKING ALGORITHM 519\nWe then compute the antecedent score for high-scoring mentions. The antecedent\nscore c(i;j)takes as input a representation of the spans iandj, but also the element-\nwise similarity of the two spans to each other gi\u000egj(here\u000eis element-wise mul-\ntiplication). Fig. 23.6 shows the computation of the score sfor the three possible\nantecedents of the company in the example sentence from Fig. 23.5.\nFigure 23.6 The computation of the score sfor the three possible antecedents of the com-\npany in the example sentence from Fig. 23.5. Figure after Lee et al. (2017b).\nGiven the set of mentions, the joint distribution of antecedents for each docu-\nment is computed in a forward pass, and we can then do transitive closure on the\nantecedents to create a \ufb01nal clustering for the document.\nFig. 23.7 shows example predictions from the model, showing the attention\nweights, which Lee et al. (2017b) \ufb01nd correlate with traditional semantic heads.\nNote that the model gets the second example wrong, presumably because attendants\nandpilot likely have nearby word embeddings.\nFigure 23.7 Sample predictions from the Lee et al. (2017b) model, with one cluster per\nexample, showing one correct example and one mistake. Bold, parenthesized spans are men-\ntions in the predicted cluster. The amount of red color on a word indicates the head-\ufb01nding\nattention weight ai;tin (23.52). Figure adapted from Lee et al. (2017b).\n23.6.3 Learning\nFor training, we don\u2019t have a single gold antecedent for each mention; instead the\ncoreference labeling only gives us each entire cluster of coreferent mentions; so a\nmention only has a latent antecedent. We therefore use a loss function that maxi-\nmizes the sum of the coreference probability of any of the legal antecedents. For a\ngiven mention iwith possible antecedents Y(i), let GOLD (i) be the set of mentions\nin the gold cluster containing i. Since the set of mentions occurring before iisY(i),\nthe set of mentions in that gold cluster that also occur before iisY(i)\\GOLD (i). We 520 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\ntherefore want to maximize:\nX\n\u02c6y2Y(i)\\GOLD (i)P(\u02c6y) (23.56)\nIf a mention iis not in a gold cluster GOLD (i) =\u000f.\nTo turn this probability into a loss function, we\u2019ll use the cross-entropy loss\nfunction we de\ufb01ned in Eq. 5.23 in Chapter 5, by taking the \u0000log of the probability.\nIf we then sum over all mentions, we get the \ufb01nal loss function for training:\nL=NX\ni=2\u0000logX\n\u02c6y2Y(i)\\GOLD (i)P(\u02c6y) (23.57)\n23.7 Entity Linking\nEntity linking is the task of associating a mention in text with the representation of entity linking\nsome real-world entity in an ontology or knowledge base (Ji and Grishman, 2011). It\nis the natural follow-on to coreference resolution; coreference resolution is the task\nof associating textual mentions that corefer to the same entity. Entity linking takes\nthe further step of identifying who that entity is. It is especially important for any\nNLP task that links to a knowledge base.\nWhile there are all sorts of potential knowledge-bases, we\u2019ll focus in this section\non Wikipedia, since it\u2019s widely used as an ontology for NLP tasks. In this usage,\neach unique Wikipedia page acts as the unique id for a particular entity. This task of\ndeciding which Wikipedia page corresponding to an individual is being referred to\nby a text mention has its own name: wiki\ufb01cation (Mihalcea and Csomai, 2007). wiki\ufb01cation\nSince the earliest systems (Mihalcea and Csomai 2007, Cucerzan 2007, Milne\nand Witten 2008), entity linking is done in (roughly) two stages: mention detec-\ntion andmention disambiguation . We\u2019ll give two algorithms, one simple classic\nbaseline that uses anchor dictionaries and information from the Wikipedia graph\nstructure (Ferragina and Scaiella, 2011) and one modern neural algorithm (Li et al.,\n2020). We\u2019ll focus here mainly on the application of entity linking to questions,\nsince a lot of the literature has been in that context.\n23.7.1 Linking based on Anchor Dictionaries and Web Graph\nAs a simple baseline we introduce the T AGME linker (Ferragina and Scaiella, 2011)\nfor Wikipedia, which itself draws on earlier algorithms (Mihalcea and Csomai 2007,\nCucerzan 2007, Milne and Witten 2008). Wiki\ufb01cation algorithms de\ufb01ne the set of\nentities as the set of Wikipedia pages, so we\u2019ll refer to each Wikipedia page as a\nunique entity e. T AGME \ufb01rst creates a catalog of all entities (i.e. all Wikipedia\npages, removing some disambiguation and other meta-pages) and indexes them in a\nstandard IR engine like Lucene. For each page e, the algorithm computes an in-link\ncount in (e): the total number of in-links from other Wikipedia pages that point to e.\nThese counts can be derived from Wikipedia dumps.\nFinally, the algorithm requires an anchor dictionary . An anchor dictionary\nlists for each Wikipedia page, its anchor texts : the hyperlinked spans of text on anchor texts\nother pages that point to it. For example, the web page for Stanford University,\nhttp://www.stanford.edu , might be pointed to from another page using anchor\ntexts like Stanford orStanford University : 23.7 \u2022 E NTITY LINKING 521\n<a href=\"http://www.stanford.edu\">Stanford University</a>\nWe compute a Wikipedia anchor dictionary by including, for each Wikipedia\npage e,e\u2019s title as well as all the anchor texts from all Wikipedia pages that point to e.\nFor each anchor string awe\u2019ll also compute its total frequency freq (a)in Wikipedia\n(including non-anchor uses), the number of times aoccurs as a link (which we\u2019ll call\nlink(a)), and its link probability linkprob (a) =link(a)=freq(a). Some cleanup of the\n\ufb01nal anchor dictionary is required, for example removing anchor strings composed\nonly of numbers or single characters, that are very rare, or that are very unlikely to\nbe useful entities because they have a very low linkprob.\nMention Detection Given a question (or other text we are trying to link), T AGME\ndetects mentions by querying the anchor dictionary for each token sequence up to\n6 words. This large set of sequences is pruned with some simple heuristics (for\nexample pruning substrings if they have small linkprobs). The question:\nWhen was Ada Lovelace born?\nmight give rise to the anchor Ada Lovelace and possibly Ada, but substrings spans\nlikeLovelace might be pruned as having too low a linkprob, and but spans like born\nhave such a low linkprob that they would not be in the anchor dictionary at all.\nMention Disambiguation If a mention span is unambiguous (points to only one\nentity/Wikipedia page), we are done with entity linking! However, many spans are\nambiguous, matching anchors for multiple Wikipedia entities/pages. The T AGME\nalgorithm uses two factors for disambiguating ambiguous spans, which have been\nreferred to as prior probability andrelatedness/coherence . The \ufb01rst factor is p(eja),\nthe probability with which the span refers to a particular entity. For each page e2\nE(a), the probability p(eja)that anchor apoints to e, is the ratio of the number of\nlinks into ewith anchor text ato the total number of occurrences of aas an anchor:\nprior(a!e) = p(eja) =count (a!e)\nlink(a)(23.58)\nLet\u2019s see how that factor works in linking entities in the following question:\nWhat Chinese Dynasty came before the Yuan?\nThe most common association for the span Yuan in the anchor dictionary is the name\nof the Chinese currency, i.e., the probability p(Yuancurrencyjyuan)is very high.\nRarer Wikipedia associations for Yuan include the common Chinese last name, a\nlanguage spoken in Thailand, and the correct entity in this case, the name of the\nChinese dynasty. So if we chose based only on p(eja), we would make the wrong\ndisambiguation and miss the correct link, Yuandynasty .\nTo help in just this sort of case, T AGME uses a second factor, the relatedness of\nthis entity to other entities in the input question. In our example, the fact that the\nquestion also contains the span Chinese Dynasty , which has a high probability link to\nthe pageDynasties inChinese history , ought to help match Yuandynasty .\nLet\u2019s see how this works. Given a question q, for each candidate anchors span\nadetected in q, we assign a relatedness score to each possible entity e2E(a)ofa.\nThe relatedness score of the link a!eis the weighted average relatedness between\neand all other entities in q. Two entities are considered related to the extent their\nWikipedia pages share many in-links. More formally, the relatedness between two\nentities AandBis computed as\nrel(A;B) =log(max(jin(A)j;jin(B)j))\u0000log(jin(A)\\in(B)j)\nlog(jWj)\u0000log(min(jin(A)j;jin(B)j))(23.59) 522 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\nwhere in (x)is the set of Wikipedia pages pointing to xandWis the set of all Wiki-\npedia pages in the collection.\nThe vote given by anchor bto the candidate annotation a!Xis the average,\nover all the possible entities of b, of their relatedness to X, weighted by their prior\nprobability:\nvote(b;X) =1\njE(b)jX\nY2E(b)rel(X;Y)p(Yjb) (23.60)\nThe total relatedness score for a!Xis the sum of the votes of all the other anchors\ndetected in q:\nrelatedness (a!X) =X\nb2Xqnavote(b;X) (23.61)\nTo score a!X, we combine relatedness and prior by choosing the entity X\nthat has the highest relatedness (a!X), \ufb01nding other entities within a small \u000fof\nthis value, and from this set, choosing the entity with the highest prior P(Xja). The\nresult of this step is a single entity assigned to each span in q.\nThe T AGME algorithm has one further step of pruning spurious anchor/entity\npairs, assigning a score averaging link probability with the coherence.\ncoherence (a!X) =1\njSj\u00001X\nB2SnXrel(B;X)\nscore(a!X) =coherence (a!X)+linkprob (a)\n2(23.62)\nFinally, pairs are pruned if score (a!X)<l, where the threshold lis set on a\nheld-out set.\n23.7.2 Neural Graph-based linking\nMore recent entity linking models are based on bi-encoders , encoding a candidate\nmention span, encoding an entity, and computing the dot product between the en-\ncodings. This allows embeddings for all the entities in the knowledge base to be\nprecomputed and cached (Wu et al., 2020). Let\u2019s sketch the ELQ linking algorithm\nof Li et al. (2020), which is given a question qand a set of candidate entities from\nWikipedia with associated Wikipedia text, and outputs tuples (e;ms;me)of entity id,\nmention start, and mention end. As Fig. 23.8 shows, it does this by encoding each\nWikipedia entity using text from Wikipedia, encoding each mention span using text\nfrom the question, and computing their similarity, as we describe below.\nEntity Mention Detection To get an h-dimensional embedding for each question\ntoken, the algorithm runs the question through BERT in the normal way:\n[q1\u0001\u0001\u0001qn] =BERT ([CLS]q1\u0001\u0001\u0001qn[SEP]) (23.63)\nIt then computes the likelihood of each span [i;j]inqbeing an entity mention, in\na way similar to the span-based algorithm we saw for the reader above. First we\ncompute the score for i/jbeing the start/end of a mention:\nsstart(i) =wstart\u0001qi;send(j) =wend\u0001qj; (23.64) 23.7 \u2022 E NTITY LINKING 523\nFigure 23.8 A sketch of the inference process in the ELQ algorithm for entity linking in\nquestions (Li et al., 2020). Each candidate question mention span and candidate entity are\nseparately encoded, and then scored by the entity/span dot product.\nwhere wstartandwendare vectors learned during training. Next, another trainable\nembedding, wmention is used to compute a score for each token being part of a men-\ntion:\nsmention (t) =wmention\u0001qt (23.65)\nMention probabilities are then computed by combining these three scores:\np([i;j]) =s \nsstart(i)+send(j)+jX\nt=ismention (t)!\n(23.66)\nEntity Linking To link mentions to entities, we next compute embeddings for\neach entity in the set E=e1;\u0001\u0001\u0001;ei;\u0001\u0001\u0001;ewof all Wikipedia entities. For each en-\ntityeiwe\u2019ll get text from the entity\u2019s Wikipedia page, the title t(ei)and the \ufb01rst\n128 tokens of the Wikipedia page which we\u2019ll call the description d(ei). This is\nagain run through BERT, taking the output of the CLStoken BERT [CLS] as the entity\nrepresentation:\nxei=BERT [CLS]([CLS]t(ei)[ENT]d(ei)[SEP]) (23.67)\nMention spans can be linked to entities by computing, for each entity eand span\n[i;j], the dot product similarity between the span encoding (the average of the token\nembeddings) and the entity encoding.\nyi;j=1\n(j\u0000i+1)jX\nt=iqt\ns(e;[i;j]) =x\u0001\neyi;j (23.68)\nFinally, we take a softmax to get a distribution over entities for each span:\np(ej[i;j]) =exp(s(e;[i;j]))P\ne02Eexp(s(e0;[i;j]))(23.69)\nTraining The ELQ mention detection and entity linking algorithm is fully super-\nvised. This means, unlike the anchor dictionary algorithms from Section 23.7.1, 524 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\nit requires datasets with entity boundaries marked and linked. Two such labeled\ndatasets are WebQuestionsSP (Yih et al., 2016), an extension of the WebQuestions\n(Berant et al., 2013) dataset derived from Google search questions, and GraphQues-\ntions (Su et al., 2016). Both have had entity spans in the questions marked and\nlinked (Sorokin and Gurevych 2018, Li et al. 2020) resulting in entity-labeled ver-\nsions WebQSP ELand GraphQ EL(Li et al., 2020).\nGiven a training set, the ELQ mention detection and entity linking phases are\ntrained jointly, optimizing the sum of their losses. The mention detection loss is\na binary cross-entropy loss, with Lthe length of the passage and Nthe number of\ncandidates:\nLMD=\u00001\nNX\n1\u0014i\u0014j\u0014min(i+L\u00001;n)\u0000\ny[i;j]logp([i;j])+( 1\u0000y[i;j])log(1\u0000p([i;j]))\u0001\n(23.70)\nwith y[i;j]=1 if[i;j]is a gold mention span, else 0. The entity linking loss is:\nLED=\u0000logp(egj[i;j]) (23.71)\nwhere egis the gold entity for mention [i;j].\n23.8 Evaluation of Coreference Resolution\nWe evaluate coreference algorithms model-theoretically, comparing a set of hypoth-\nesischains or clusters Hproduced by the system against a set of gold or reference\nchains or clusters Rfrom a human labeling, and reporting precision and recall.\nHowever, there are a wide variety of methods for doing this comparison. In fact,\nthere are 5 common metrics used to evaluate coreference algorithms: the link based\nMUC (Vilain et al., 1995) and BLANC (Recasens and Hovy 2011, Luo et al. 2014)\nmetrics, the mention based B3metric (Bagga and Baldwin, 1998), the entity based\nCEAF metric (Luo, 2005), and the link based entity aware LEA metric (Moosavi and\nStrube, 2016).\nLet\u2019s just explore two of the metrics. The MUC F-measure (Vilain et al., 1995)MUC\nF-measure\nis based on the number of coreference links (pairs of mentions) common to Hand\nR. Precision is the number of common links divided by the number of links in H.\nRecall is the number of common links divided by the number of links in R; This\nmakes MUC biased toward systems that produce large chains (and fewer entities),\nand it ignores singletons, since they don\u2019t involve links.\nB3is mention-based rather than link-based. For each mention in the reference B3\nchain, we compute a precision and recall, and then we take a weighted sum over all\nNmentions in the document to compute a precision and recall for the entire task. For\na given mention i, letRbe the reference chain that includes i, and Hthe hypothesis\nchain that has i. The set of correct mentions in HisH\\R. Precision for mention i\nis thusjH\\Rj\njHj, and recall for mention ithusjH\\Rj\njRj. The total precision is the weighted\nsum of the precision for mention i, weighted by a weight wi. The total recall is the\nweighted sum of the recall for mention i, weighted by a weight wi. Equivalently:\nPrecision =NX\ni=1wi# of correct mentions in hypothesis chain containing entity i\n# of mentions in hypothesis chain containing entity i\nRecall =NX\ni=1wi# of correct mentions in hypothesis chain containing entity i\n# of mentions in reference chain containing entity i 23.9 \u2022 W INOGRAD SCHEMA PROBLEMS 525\nThe weight wifor each entity can be set to different values to produce different\nversions of the algorithm.\nFollowing a proposal from Denis and Baldridge (2009), the CoNLL coreference\ncompetitions were scored based on the average of MUC, CEAF-e, and B3(Pradhan\net al. 2011, Pradhan et al. 2012b), and so it is common in many evaluation campaigns\nto report an average of these 3 metrics. See Luo and Pradhan (2016) for a detailed\ndescription of the entire set of metrics; reference implementations of these should\nbe used rather than attempting to reimplement from scratch (Pradhan et al., 2014).\nAlternative metrics have been proposed that deal with particular coreference do-\nmains or tasks. For example, consider the task of resolving mentions to named\nentities (persons, organizations, geopolitical entities), which might be useful for in-\nformation extraction or knowledge base completion. A hypothesis chain that cor-\nrectly contains all the pronouns referring to an entity, but has no version of the name\nitself, or is linked with a wrong name, is not useful for this task. We might instead\nwant a metric that weights each mention by how informative it is (with names being\nmost informative) (Chen and Ng, 2013) or a metric that considers a hypothesis to\nmatch a gold chain only if it contains at least one variant of a name (the NEC F1\nmetric of Agarwal et al. (2019)).\n23.9 Winograd Schema problems\nFrom early on in the \ufb01eld, researchers have noted that some cases of coreference\nare quite dif\ufb01cult, seeming to require world knowledge or sophisticated reasoning\nto solve. The problem was most famously pointed out by Winograd (1972) with the\nfollowing example:\n(23.72) The city council denied the demonstrators a permit because\na. they feared violence.\nb. they advocated violence.\nWinograd noticed that the antecedent that most readers preferred for the pro-\nnoun they in continuation (a) was the city council , but in (b) was the demonstrators .\nHe suggested that this requires understanding that the second clause is intended\nas an explanation of the \ufb01rst clause, and also that our cultural frames suggest that\ncity councils are perhaps more likely than demonstrators to fear violence and that\ndemonstrators might be more likely to advocate violence.\nIn an attempt to get the \ufb01eld of NLP to focus more on methods involving world\nknowledge and common-sense reasoning, Levesque (2011) proposed a challenge\ntask called the Winograd Schema Challenge .8The problems in the challenge taskWinograd\nschema\nare coreference problems designed to be easily disambiguated by the human reader,\nbut hopefully not solvable by simple techniques such as selectional restrictions, or\nother basic word association methods.\nThe problems are framed as a pair of statements that differ in a single word or\nphrase, and a coreference question:\n(23.73) The trophy didn\u2019t \ufb01t into the suitcase because it was too large .\nQuestion: What was too large ? Answer: The trophy\n8Levesque\u2019s call was quickly followed up by Levesque et al. (2012) and Rahman and Ng (2012), a\ncompetition at the IJCAI conference (Davis et al., 2017), and a natural language inference version of the\nproblem called WNLI (Wang et al., 2018a). 526 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\n(23.74) The trophy didn\u2019t \ufb01t into the suitcase because it was too small .\nQuestion: What was too small ? Answer: The suitcase\nThe problems have the following characteristics:\n1. The problems each have two parties\n2. A pronoun preferentially refers to one of the parties, but could grammatically\nalso refer to the other\n3. A question asks which party the pronoun refers to\n4. If one word in the question is changed, the human-preferred answer changes\nto the other party\nThe kind of world knowledge that might be needed to solve the problems can\nvary. In the trophy/suitcase example, it is knowledge about the physical world; that\na bigger object cannot \ufb01t into a smaller object. In the original Winograd sentence,\nit is stereotypes about social actors like politicians and protesters. In examples like\nthe following, it is knowledge about human actions like turn-taking or thanking.\n(23.75) Bill passed the gameboy to John because his turn was [over/next]. Whose\nturn was [over/next]? Answers: Bill/John\n(23.76) Joan made sure to thank Susan for all the help she had [given/received].\nWho had [given/received] help? Answers: Susan/Joan.\nAlthough the Winograd Schema was designed to require common-sense rea-\nsoning, a large percentage of the original set of problems can be solved by pre-\ntrained language models, \ufb01ne-tuned on Winograd Schema sentences (Kocijan et al.,\n2019). Large pretrained language models encode an enormous amount of world or\ncommon-sense knowledge! The current trend is therefore to propose new datasets\nwith increasingly dif\ufb01cult Winograd-like coreference resolution problems like K NOW REF\n(Emami et al., 2019), with examples like:\n(23.77) Marcus is undoubtedly faster than Jarrett right now but in [his] prime the\ngap wasn\u2019t all that big.\nIn the end, it seems likely that some combination of language modeling and knowl-\nedge will prove fruitful; indeed, it seems that knowledge-based models over\ufb01t less\nto lexical idiosyncracies in Winograd Schema training sets (Trichelair et al., 2018),\n23.10 Gender Bias in Coreference\nAs with other aspects of language processing, coreference models exhibit gender and\nother biases (Zhao et al. 2018a, Rudinger et al. 2018, Webster et al. 2018). For exam-\nple the WinoBias dataset (Zhao et al., 2018a) uses a variant of the Winograd Schema\nparadigm to test the extent to which coreference algorithms are biased toward link-\ning gendered pronouns with antecedents consistent with cultural stereotypes. As we\nsummarized in Chapter 6, embeddings replicate societal biases in their training test,\nsuch as associating men with historically sterotypical male occupations like doctors,\nand women with stereotypical female occupations like secretaries (Caliskan et al.\n2017, Garg et al. 2018).\nA WinoBias sentence contain two mentions corresponding to stereotypically-\nmale and stereotypically-female occupations and a gendered pronoun that must be\nlinked to one of them. The sentence cannot be disambiguated by the gender of the\npronoun, but a biased model might be distracted by this cue. Here is an example\nsentence: 23.11 \u2022 S UMMARY 527\n(23.78) The secretary called the physician iand told him iabout a new patient\n[pro-stereotypical]\n(23.79) The secretary called the physician iand told her iabout a new patient\n[anti-stereotypical]\nZhao et al. (2018a) consider a coreference system to be biased if it is more accu-\nrate at linking pronouns consistent with gender stereotypical occupations (e.g., him\nwith physician in (23.78)) than linking pronouns inconsistent with gender-stereotypical\noccupations (e.g., herwith physician in (23.79)). They show that coreference sys-\ntems of all architectures (rule-based, feature-based machine learned, and end-to-\nend-neural) all show signi\ufb01cant bias, performing on average 21 F 1points worse in\nthe anti-stereotypical cases.\nOne possible source of this bias is that female entities are signi\ufb01cantly un-\nderrepresented in the OntoNotes dataset, used to train most coreference systems.\nZhao et al. (2018a) propose a way to overcome this bias: they generate a second\ngender-swapped dataset in which all male entities in OntoNotes are replaced with\nfemale ones and vice versa, and retrain coreference systems on the combined orig-\ninal and swapped OntoNotes data, also using debiased GloVE embeddings (Boluk-\nbasi et al., 2016). The resulting coreference systems no longer exhibit bias on the\nWinoBias dataset, without signi\ufb01cantly impacting OntoNotes coreference accuracy.\nIn a follow-up paper, Zhao et al. (2019) show that the same biases exist in ELMo\ncontextualized word vector representations and coref systems that use them. They\nshowed that retraining ELMo with data augmentation again reduces or removes bias\nin coreference systems on WinoBias.\nWebster et al. (2018) introduces another dataset, GAP, and the task of Gendered\nPronoun Resolution as a tool for developing improved coreference algorithms for\ngendered pronouns. GAP is a gender-balanced labeled corpus of 4,454 sentences\nwith gendered ambiguous pronouns (by contrast, only 20% of the gendered pro-\nnouns in the English OntoNotes training data are feminine). The examples were\ncreated by drawing on naturally occurring sentences from Wikipedia pages to create\nhard to resolve cases with two named entities of the same gender and an ambiguous\npronoun that may refer to either person (or neither), like the following:\n(23.80) In May, Fujisawa joined Mari Motohashi\u2019s rink as the team\u2019s skip, moving\nback from Karuizawa to Kitami where shehad spent her junior days.\nWebster et al. (2018) show that modern coreference algorithms perform signif-\nicantly worse on resolving feminine pronouns than masculine pronouns in GAP.\nKurita et al. (2019) shows that a system based on BERT contextualized word repre-\nsentations shows similar bias.\n23.11 Summary\nThis chapter introduced the task of coreference resolution .\n\u2022 This is the task of linking together mentions in text which corefer , i.e. refer\nto the same discourse entity in the discourse model , resulting in a set of\ncoreference chains (also called clusters orentities ).\n\u2022 Mentions can be de\ufb01nite NPs orinde\ufb01nite NPs ,pronouns (including zero\npronouns ) ornames . 528 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\n\u2022 The surface form of an entity mention is linked to its information status\n(new,old, orinferrable ), and how accessible orsalient the entity is.\n\u2022 Some NPs are not referring expressions, such as pleonastic itinIt is raining .\n\u2022 Many corpora have human-labeled coreference annotations that can be used\nfor supervised learning, including OntoNotes for English, Chinese, and Ara-\nbic, ARRAU for English, and AnCora for Spanish and Catalan.\n\u2022 Mention detection can start with all nouns and named entities and then use\nanaphoricity classi\ufb01ers orreferentiality classi\ufb01ers to \ufb01lter out non-mentions.\n\u2022 Three common architectures for coreference are mention-pair ,mention-rank ,\nandentity-based , each of which can make use of feature-based or neural clas-\nsi\ufb01ers.\n\u2022 Modern coreference systems tend to be end-to-end, performing mention de-\ntection and coreference in a single end-to-end architecture.\n\u2022 Algorithms learn representations for text spans and heads, and learn to com-\npare anaphor spans with candidate antecedent spans.\n\u2022 Entity linking is the task of associating a mention in text with the representa-\ntion of some real-world entity in an ontology .\n\u2022 Coreference systems are evaluated by comparing with gold entity labels using\nprecision/recall metrics like MUC ,B3,CEAF ,BLANC , orLEA .\n\u2022 The Winograd Schema Challenge problems are dif\ufb01cult coreference prob-\nlems that seem to require world knowledge or sophisticated reasoning to solve.\n\u2022 Coreference systems exhibit gender bias which can be evaluated using datasets\nlike Winobias and GAP.\nBibliographical and Historical Notes\nCoreference has been part of natural language processing since the 1970s (Woods\net al. 1972, Winograd 1972). The discourse model and the entity-centric foundation\nof coreference was formulated by Karttunen (1969) (at the 3rd COLING confer-\nence), playing a role also in linguistic semantics (Heim 1982, Kamp 1981). But\nit was Bonnie Webber\u2019s 1978 dissertation and following work (Webber 1983) that\nexplored the model\u2019s computational aspects, providing fundamental insights into\nhow entities are represented in the discourse model and the ways in which they can\nlicense subsequent reference. Many of the examples she provided continue to chal-\nlenge theories of reference to this day.\nThe Hobbs algorithm9is a tree-search algorithm that was the \ufb01rst in a longHobbs\nalgorithm\nseries of syntax-based methods for identifying reference robustly in naturally occur-\nring text. The input to the Hobbs algorithm is a pronoun to be resolved, together\nwith a syntactic (constituency) parse of the sentences up to and including the cur-\nrent sentence. The details of the algorithm depend on the grammar used, but can be\nunderstood from a simpli\ufb01ed version due to Kehler et al. (2004) that just searches\nthrough the list of NPs in the current and prior sentences. This simpli\ufb01ed Hobbs\nalgorithm searches NPs in the following order: \u201c(i) in the current sentence from\nright-to-left, starting with the \ufb01rst NP to the left of the pronoun, (ii) in the previous\nsentence from left-to-right, (iii) in two sentences prior from left-to-right, and (iv) in\n9The simpler of two algorithms presented originally in Hobbs (1978). BIBLIOGRAPHICAL AND HISTORICAL NOTES 529\nthe current sentence from left-to-right, starting with the \ufb01rst noun group to the right\nof the pronoun (for cataphora). The \ufb01rst noun group that agrees with the pronoun\nwith respect to number, gender, and person is chosen as the antecedent\u201d (Kehler\net al., 2004).\nLappin and Leass (1994) was an in\ufb02uential entity-based system that used weights\nto combine syntactic and other features, extended soon after by Kennedy and Bogu-\nraev (1996) whose system avoids the need for full syntactic parses.\nApproximately contemporaneously centering (Grosz et al., 1995) was applied\nto pronominal anaphora resolution by Brennan et al. (1987), and a wide variety of\nwork followed focused on centering\u2019s use in coreference (Kameyama 1986, Di Eu-\ngenio 1990, Walker et al. 1994, Di Eugenio 1996, Strube and Hahn 1996, Kehler\n1997a, Tetreault 2001, Iida et al. 2003). Kehler and Rohde (2013) show how center-\ning can be integrated with coherence-driven theories of pronoun interpretation. See\nChapter 24 for the use of centering in measuring discourse coherence.\nCoreference competitions as part of the US DARPA-sponsored MUC confer-\nences provided early labeled coreference datasets (the 1995 MUC-6 and 1998 MUC-\n7 corpora), and set the tone for much later work, choosing to focus exclusively\non the simplest cases of identity coreference (ignoring dif\ufb01cult cases like bridging,\nmetonymy, and part-whole) and drawing the community toward supervised machine\nlearning and metrics like the MUC metric (Vilain et al., 1995). The later ACE eval-\nuations produced labeled coreference corpora in English, Chinese, and Arabic that\nwere widely used for model training and evaluation.\nThis DARPA work in\ufb02uenced the community toward supervised learning begin-\nning in the mid-90s (Connolly et al. 1994, Aone and Bennett 1995, McCarthy and\nLehnert 1995). Soon et al. (2001) laid out a set of basic features, extended by Ng and\nCardie (2002b), and a series of machine learning models followed over the next 15\nyears. These often focused separately on pronominal anaphora resolution (Kehler\net al. 2004, Bergsma and Lin 2006), full NP coreference (Cardie and Wagstaff 1999,\nNg and Cardie 2002b, Ng 2005a) and de\ufb01nite NP reference (Poesio and Vieira 1998,\nVieira and Poesio 2000), as well as separate anaphoricity detection (Bean and Riloff\n1999, Bean and Riloff 2004, Ng and Cardie 2002a, Ng 2004), or singleton detection\n(de Marneffe et al., 2015).\nThe move from mention-pair to mention-ranking approaches was pioneered by\nYang et al. (2003) and Iida et al. (2003) who proposed pairwise ranking methods,\nthen extended by Denis and Baldridge (2008) who proposed to do ranking via a soft-\nmax over all prior mentions. The idea of doing mention detection, anaphoricity, and\ncoreference jointly in a single end-to-end model grew out of the early proposal of Ng\n(2005b) to use a dummy antecedent for mention-ranking, allowing \u2018non-referential\u2019\nto be a choice for coreference classi\ufb01ers, Denis and Baldridge\u2019s 2007 joint system\ncombining anaphoricity classi\ufb01er probabilities with coreference probabilities, the\nDenis and Baldridge (2008) ranking model, and the Rahman and Ng (2009) pro-\nposal to train the two models jointly with a single objective.\nSimple rule-based systems for coreference returned to prominence in the 2010s,\npartly because of their ability to encode entity-based features in a high-precision way\n(Zhou et al. 2004b, Haghighi and Klein 2009, Raghunathan et al. 2010, Lee et al.\n2011, Lee et al. 2013, Hajishirzi et al. 2013) but in the end they suffered from an\ninability to deal with the semantics necessary to correctly handle cases of common\nnoun coreference.\nA return to supervised learning led to a number of advances in mention-ranking\nmodels which were also extended into neural architectures, for example using re- 530 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\ninforcement learning to directly optimize coreference evaluation models Clark and\nManning (2016a), doing end-to-end coreference all the way from span extraction\n(Lee et al. 2017b, Zhang et al. 2018). Neural models also were designed to take\nadvantage of global entity-level information (Clark and Manning 2016b, Wiseman\net al. 2016, Lee et al. 2018).\nCoreference is also related to the task of entity linking discussed in Chapter 14.\nCoreference can help entity linking by giving more possible surface forms to help\nlink to the right Wikipedia page, and conversely entity linking can help improve\ncoreference resolution. Consider this example from Hajishirzi et al. (2013):\n(23.81) [Michael Eisner] 1and [Donald Tsang] 2announced the grand opening of\n[[Hong Kong] 3Disneyland] 4yesterday. [Eisner] 1thanked [the President] 2\nand welcomed [fans] 5to [the park] 4.\nIntegrating entity linking into coreference can help draw encyclopedic knowl-\nedge (like the fact that Donald Tsang is a president) to help disambiguate the men-\ntionthe President . Ponzetto and Strube (2006) 2007 and Ratinov and Roth (2012)\nshowed that such attributes extracted from Wikipedia pages could be used to build\nricher models of entity mentions in coreference. More recent research shows how to\ndo linking and coreference jointly (Hajishirzi et al. 2013, Zheng et al. 2013) or even\njointly with named entity tagging as well (Durrett and Klein 2014).\nThe coreference task as we introduced it involves a simplifying assumption that\nthe relationship between an anaphor and its antecedent is one of identity : the two\ncoreferring mentions refer to the identical discourse referent. In real texts, the rela-\ntionship can be more complex, where different aspects of a discourse referent can\nbe neutralized or refocused. For example (23.82) (Recasens et al., 2011) shows an\nexample of metonymy , in which the capital city Washington is used metonymically metonymy\nto refer to the US. (23.83-23.84) show other examples (Recasens et al., 2011):\n(23.82) a strict interpretation of a policy requires The U.S. to notify foreign\ndictators of certain coup plots ... Washington rejected the bid ...\n(23.83) I once crossed that border into Ashgh-Abad on Nowruz, the Persian New\nYear. In the South, everyone was celebrating New Year ; to the North, it\nwas a regular day.\n(23.84) In France, the president is elected for a term of seven years, while in the\nUnited States heis elected for a term of four years.\nFor further linguistic discussions of these complications of coreference see Puste-\njovsky (1991), van Deemter and Kibble (2000), Poesio et al. (2006), Fauconnier and\nTurner (2008), Versley (2008), and Barker (2010).\nNg (2017) offers a useful compact history of machine learning models in coref-\nerence resolution. There are three excellent book-length surveys of anaphora/coref-\nerence resolution, covering different time periods: Hirst (1981) (early work until\nabout 1981), Mitkov (2002) (1986-2001), and Poesio et al. (2016) (2001-2015).\nAndy Kehler wrote the Discourse chapter for the 2000 \ufb01rst edition of this text-\nbook, which we used as the starting point for the second-edition chapter, and there\nare some remnants of Andy\u2019s lovely prose still in this third-edition coreference chap-\nter.\nExercises CHAPTER\n24Discourse Coherence\nAnd even in our wildest and most wandering reveries, nay in our very dreams,\nwe shall \ufb01nd, if we re\ufb02ect, that the imagination ran not altogether at adven-\ntures, but that there was still a connection upheld among the different ideas,\nwhich succeeded each other. Were the loosest and freest conversation to be\ntranscribed, there would immediately be transcribed, there would immediately\nbe observed something which connected it in all its transitions.\nDavid Hume, An enquiry concerning human understanding , 1748\nOrson Welles\u2019 movie Citizen Kane was groundbreaking in many ways, perhaps most\nnotably in its structure. The story of the life of \ufb01ctional media magnate Charles\nFoster Kane, the movie does not proceed in chronological order through Kane\u2019s\nlife. Instead, the \ufb01lm begins with Kane\u2019s death (famously murmuring \u201cRosebud\u201d )\nand is structured around \ufb02ashbacks to his life inserted among scenes of a reporter\ninvestigating his death. The novel idea that the structure of a movie does not have\nto linearly follow the structure of the real timeline made apparent for 20th century\ncinematography the in\ufb01nite possibilities and impact of different kinds of coherent\nnarrative structures.\nBut coherent structure is not just a fact about movies or works of art. Like\nmovies, language does not normally consist of isolated, unrelated sentences, but\ninstead of collocated, structured, coherent groups of sentences. We refer to such\na coherent structured group of sentences as a discourse , and we use the word co- discourse\nherence to refer to the relationship between sentences that makes real discourses coherence\ndifferent than just random assemblages of sentences. The chapter you are now read-\ning is an example of a discourse, as is a news article, a conversation, a thread on\nsocial media, a Wikipedia page, and your favorite novel.\nWhat makes a discourse coherent? If you created a text by taking random sen-\ntences each from many different sources and pasted them together, would that be a\ncoherent discourse? Almost certainly not. Real discourses exhibit both local coher- local\nence andglobal coherence . Let\u2019s consider three ways in which real discourses are global\nlocally coherent;\nFirst, sentences or clauses in real discourses are related to nearby sentences in\nsystematic ways. Consider this example from Hobbs (1979):\n(24.1) John took a train from Paris to Istanbul. He likes spinach.\nThis sequence is incoherent because it is unclear to a reader why the second\nsentence follows the \ufb01rst; what does liking spinach have to do with train trips? In\nfact, a reader might go to some effort to try to \ufb01gure out how the discourse could be\ncoherent; perhaps there is a French spinach shortage? The very fact that hearers try\nto identify such connections suggests that human discourse comprehension involves\nthe need to establish this kind of coherence.\nBy contrast, in the following coherent example:\n(24.2) Jane took a train from Paris to Istanbul. She had to attend a conference. 532 CHAPTER 24 \u2022 D ISCOURSE COHERENCE\nthe second sentence gives a REASON for Jane\u2019s action in the \ufb01rst sentence. Struc-\ntured relationships like REASON that hold between text units are called coherence\nrelations , and coherent discourses are structured by many such coherence relations.coherence\nrelations\nCoherence relations are introduced in Section 24.1.\nA second way a discourse can be locally coherent is by virtue of being \u201cabout\u201d\nsomeone or something. In a coherent discourse some entities are salient , and the\ndiscourse focuses on them and doesn\u2019t go back and forth between multiple entities.\nThis is called entity-based coherence . Consider the following incoherent passage,\nin which the salient entity seems to wildly swing from John to Jenny to the piano\nstore to the living room, back to Jenny, then the piano again:\n(24.3) John wanted to buy a piano for his living room.\nJenny also wanted to buy a piano.\nHe went to the piano store.\nIt was nearby.\nThe living room was on the second \ufb02oor.\nShe didn\u2019t \ufb01nd anything she liked.\nThe piano he bought was hard to get up to that \ufb02oor.\nEntity-based coherence models measure this kind of coherence by tracking salient\nentities across a discourse. For example Centering Theory (Grosz et al., 1995), theCentering\nTheory\nmost in\ufb02uential theory of entity-based coherence, keeps track of which entities in\nthe discourse model are salient at any point (salient entities are more likely to be\npronominalized or to appear in prominent syntactic positions like subject or object).\nIn Centering Theory, transitions between sentences that maintain the same salient\nentity are considered more coherent than ones that repeatedly shift between entities.\nThe entity grid model of coherence (Barzilay and Lapata, 2008) is a commonly entity grid\nused model that realizes some of the intuitions of the Centering Theory framework.\nEntity-based coherence is introduced in Section 24.3.\nFinally, discourses can be locally coherent by being topically coherent : nearbytopically\ncoherent\nsentences are generally about the same topic and use the same or similar vocab-\nulary to discuss these topics. Because topically coherent discourses draw from a\nsingle semantic \ufb01eld or topic, they tend to exhibit the surface property known as\nlexical cohesion (Halliday and Hasan, 1976): the sharing of identical or semanti- lexical cohesion\ncally related words in nearby sentences. For example, the fact that the words house ,\nchimney ,garret ,closet , and window \u2014 all of which belong to the same semantic\n\ufb01eld\u2014 appear in the two sentences in (24.4), or that they share the identical word\nshingled , is a cue that the two are tied together as a discourse:\n(24.4) Before winter I built a chimney , and shingled the sides of my house ...\nI have thus a tight shingled and plastered house ... with a garret and a\ncloset , a large window on each side....\nIn addition to the local coherence between adjacent or nearby sentences, dis-\ncourses also exhibit global coherence . Many genres of text are associated with\nparticular conventional discourse structures. Academic articles might have sections\ndescribing the Methodology or Results. Stories might follow conventional plotlines\nor motifs. Persuasive essays have a particular claim they are trying to argue for,\nand an essay might express this claim together with a structured set of premises that\nsupport the argument and demolish potential counterarguments. We\u2019ll introduce\nversions of each of these kinds of global coherence.\nWhy do we care about the local or global coherence of a discourse? Since co-\nherence is a property of a well-written text, coherence detection plays a part in any 24.1 \u2022 C OHERENCE RELATIONS 533\ntask that requires measuring the quality of a text. For example coherence can help\nin pedagogical tasks like essay grading or essay quality measurement that are trying\nto grade how well-written a human essay is (Somasundaran et al. 2014, Feng et al.\n2014, Lai and Tetreault 2018). Coherence can also help for summarization; knowing\nthe coherence relationship between sentences can help know how to select informa-\ntion from them. Finally, detecting incoherent text may even play a role in mental\nhealth tasks like measuring symptoms of schizophrenia or other kinds of disordered\nlanguage (Ditman and Kuperberg 2010, Elvev \u02daag et al. 2007, Bedi et al. 2015, Iter\net al. 2018).\n24.1 Coherence Relations\nRecall from the introduction the difference between passages (24.5) and (24.6).\n(24.5) Jane took a train from Paris to Istanbul. She likes spinach.\n(24.6) Jane took a train from Paris to Istanbul. She had to attend a conference.\nThe reason (24.6) is more coherent is that the reader can form a connection be-\ntween the two sentences, in which the second sentence provides a potential REASON\nfor the \ufb01rst sentences. This link is harder to form for (24.5). These connections\nbetween text spans in a discourse can be speci\ufb01ed as a set of coherence relations .coherence\nrelation\nThe next two sections describe two commonly used models of coherence relations\nand associated corpora: Rhetorical Structure Theory (RST), and the Penn Discourse\nTreeBank (PDTB).\n24.1.1 Rhetorical Structure Theory\nThe most commonly used model of discourse organization is Rhetorical Structure\nTheory (RST ) (Mann and Thompson, 1987). In RST relations are de\ufb01ned between RST\ntwo spans of text, generally a nucleus and a satellite . The nucleus is the unit that nucleus\nsatellite is more central to the writer\u2019s purpose and that is interpretable independently; the\nsatellite is less central and generally is only interpretable with respect to the nucleus.\nSome symmetric relations, however, hold between two nuclei.\nBelow are a few examples of RST coherence relations, with de\ufb01nitions adapted\nfrom the RST Treebank Manual (Carlson and Marcu, 2001).\nReason: The nucleus is an action carried out by an animate agent and the satellite\nis the reason for the nucleus.\n(24.7) [ NUC Jane took a train from Paris to Istanbul.] [ SATShe had to attend a\nconference.]\nElaboration: The satellite gives additional information or detail about the situation\npresented in the nucleus.\n(24.8) [ NUC Dorothy was from Kansas.] [ SATShe lived in the midst of the great\nKansas prairies.]\nEvidence: The satellite gives additional information or detail about the situation\npresented in the nucleus. The information is presented with the goal of convince the\nreader to accept the information presented in the nucleus.\n(24.9) [ NUC Kevin must be here.] [ SATHis car is parked outside.] 534 CHAPTER 24 \u2022 D ISCOURSE COHERENCE\nAttribution: The satellite gives the source of attribution for an instance of reported\nspeech in the nucleus.\n(24.10) [ SATAnalysts estimated] [ NUC that sales at U.S. stores declined in the\nquarter, too]\nList: In this multinuclear relation, a series of nuclei is given, without contrast or\nexplicit comparison:\n(24.11) [ NUC Billy Bones was the mate; ] [ NUC Long John, he was quartermaster]\nRST relations are traditionally represented graphically; the asymmetric Nucleus-\nSatellite relation is represented with an arrow from the satellite to the nucleus:\nKevin must be here.His car is parked outsideevidence\nWe can also talk about the coherence of a larger text by considering the hierar-\nchical structure between coherence relations. Figure 24.1 shows the rhetorical struc-\nture of a paragraph from Marcu (2000a) for the text in (24.12) from the Scienti\ufb01c\nAmerican magazine.\n(24.12) With its distant orbit\u201350 percent farther from the sun than Earth\u2013and slim\natmospheric blanket, Mars experiences frigid weather conditions. Surface\ntemperatures typically average about -60 degrees Celsius (-76 degrees\nFahrenheit) at the equator and can dip to -123 degrees C near the poles. Only\nthe midday sun at tropical latitudes is warm enough to thaw ice on occasion,\nbut any liquid water formed in this way would evaporate almost instantly\nbecause of the low atmospheric pressure.\nTitle\n(1)\nMars2-9\nevidence\n2-3\nbackground\n  (2)\nWIth its \ndistant orbit  \n<p> -- 50\n percent \nfarther from \nthe sun than \nEarth -- </p> \nand slim \natmospheric \nblanket,(3)\nMars\nexperiences\nfrigid weather\nconditions.4-9\nelaboration-additional\n(4)\nSurface \ntemperatures\n typically average \nabout -60 \ndegrees Celsius\n <p> (-76 degrees\nFahrenheit)</p>\n at the equator4-5\nList\n(5)\nand can dip\nto -123\ndegrees C\nnear the\npoles.6-9\nContrast\n6-7\n(6)\nOnly the\nmidday sun at\ntropical latitudes\nis warm enough(7)\nto thaw ice\non occasion,purpose8-9\nexplanation-argumentative\n(8)\nbut any liquid water\nformed in this way \nwould evaporate \nalmost instantly(9)\nbecause of\nthe low\natmospheric\npressure.\nFigure 24.1 A discourse tree for the Scienti\ufb01c American text in (24.12), from Marcu (2000a). Note that\nasymmetric relations are represented with a curved arrow from the satellite to the nucleus.\nThe leaves in the Fig. 24.1 tree correspond to text spans of a sentence, clause or\nphrase that are called elementary discourse units orEDU s in RST; these units can EDU\nalso be referred to as discourse segments . Because these units may correspond to\narbitrary spans of text, determining the boundaries of an EDU is an important task\nfor extracting coherence relations. Roughly speaking, one can think of discourse 24.1 \u2022 C OHERENCE RELATIONS 535\nsegments as being analogous to constituents in sentence syntax, and indeed as we\u2019ll\nsee in Section 24.2 we generally draw on parsing algorithms to infer discourse struc-\nture.\nThere are corpora for many discourse coherence models; the RST Discourse\nTreeBank (Carlson et al., 2001) is the largest available discourse corpus. It con-\nsists of 385 English language documents selected from the Penn Treebank, with full\nRST parses for each one, using a large set of 78 distinct relations, grouped into 16\nclasses. RST treebanks exist also for Spanish, German, Basque, Dutch and Brazilian\nPortuguese (Braud et al., 2017).\nNow that we\u2019ve seen examples of coherence, we can see more clearly how a\ncoherence relation can play a role in summarization or information extraction. For\nexample, the nuclei of a text presumably express more important information than\nthe satellites, which might be dropped in a summary.\n24.1.2 Penn Discourse TreeBank (PDTB)\nThe Penn Discourse TreeBank (PDTB ) is a second commonly used dataset that PDTB\nembodies another model of coherence relations (Miltsakaki et al. 2004, Prasad et al.\n2008, Prasad et al. 2014). PDTB labeling is lexically grounded . Instead of asking\nannotators to directly tag the coherence relation between text spans, they were given\na list of discourse connectives , words that signal discourse relations, like because ,discourse\nconnectives\nalthough ,when ,since , oras a result . In a part of a text where these words marked a\ncoherence relation between two text spans, the connective and the spans were then\nannotated, as in Fig. 24.13, where the phrase as a result signals a causal relationship\nbetween what PDTB calls Arg1 (the \ufb01rst two sentences, here in italics) and Arg2\n(the third sentence, here in bold).\n(24.13) Jewelry displays in department stores were often cluttered and uninspired.\nAnd the merchandise was, well, fake. As a result , marketers of faux gems\nsteadily lost space in department stores to more fashionable\nrivals\u2014cosmetics makers.\n(24.14) In July, the Environmental Protection Agency imposed a gradual ban on\nvirtually all uses of asbestos. (implicit=as a result )By 1997, almost all\nremaining uses of cancer-causing asbestos will be outlawed.\nNot all coherence relations are marked by an explicit discourse connective, and\nso the PDTB also annotates pairs of neighboring sentences with no explicit signal,\nlike (24.14). The annotator \ufb01rst chooses the word or phrase that could have been its\nsignal (in this case as a result ), and then labels its sense. For example for the am-\nbiguous discourse connective since annotators marked whether it is using a C AUSAL\nor a T EMPORAL sense.\nThe \ufb01nal dataset contains roughly 18,000 explicit relations and 16,000 implicit\nrelations. Fig. 24.2 shows examples from each of the 4 major semantic classes, while\nFig. 24.3 shows the full tagset.\nUnlike the RST Discourse Treebank, which integrates these pairwise coherence\nrelations into a global tree structure spanning an entire discourse, the PDTB does not\nannotate anything above the span-pair level, making no commitment with respect to\nhigher-level discourse structure.\nThere are also treebanks using similar methods for other languages; (24.15)\nshows an example from the Chinese Discourse TreeBank (Zhou and Xue, 2015).\nBecause Chinese has a smaller percentage of explicit discourse connectives than\nEnglish (only 22% of all discourse relations are marked with explicit connectives, 536 CHAPTER 24 \u2022 D ISCOURSE COHERENCE\nClass Type Example\nTEMPORAL SYNCHRONOUS The parishioners of St. Michael and All Angels stop to chat at\nthe church door, as members here always have. (Implicit while )\nIn the tower, \ufb01ve men and women pull rhythmically on ropes\nattached to the same \ufb01ve bells that \ufb01rst sounded here in 1614.\nCONTINGENCY REASON Also unlike Mr. Ruder, Mr. Breeden appears to be in a position\nto get somewhere with his agenda. (implicit=because )As a for-\nmer White House aide who worked closely with Congress,\nhe is savvy in the ways of Washington.\nCOMPARISON CONTRAST The U.S. wants the removal of what it perceives as barriers to\ninvestment; Japan denies there are real barriers.\nEXPANSION CONJUNCTION Not only do the actors stand outside their characters and make\nit clear they are at odds with them, but they often literally stand\non their heads.\nFigure 24.2 The four high-level semantic distinctions in the PDTB sense hierarchy\nTemporal Comparison\n\u000fAsynchronous \u000fContrast (Juxtaposition, Opposition)\n\u000fSynchronous (Precedence, Succession) \u000fPragmatic Contrast (Juxtaposition, Opposition)\n\u000fConcession (Expectation, Contra-expectation)\n\u000fPragmatic Concession\nContingency Expansion\n\u000fCause (Reason, Result) \u000fException\n\u000fPragmatic Cause (Justi\ufb01cation) \u000fInstantiation\n\u000fCondition (Hypothetical, General, Unreal\nPresent/Past, Factual Present/Past)\u000fRestatement (Speci\ufb01cation, Equivalence, Generalization)\n\u000fPragmatic Condition (Relevance, Implicit As-\nsertion)\u000fAlternative (Conjunction, Disjunction, Chosen Alterna-\ntive)\n\u000fList\nFigure 24.3 The PDTB sense hierarchy. There are four top-level c\u00aflasses, 16 types, and 23 subtypes (not all\ntypes have subtypes). 11 of the 16 types are commonly used for implicit argument classi\ufb01cation; the 5 types in\nitalics are too rare in implicit labeling to be used.\ncompared to 47% in English), annotators labeled this corpus by directly mapping\npairs of sentences to 11 sense tags, without starting with a lexical discourse connec-\ntor.\n(24.15) [ Conn\u4e3a] [Arg2\u63a8\u52a8\u56fe\u4eec\u6c5f\u5730\u533a\u5f00\u53d1]\uff0c[Arg1\u97e9\u56fd\u6350\u6b3e\u4e00\u767e\u4e07\u7f8e\u5143\n\u8bbe\u7acb\u4e86\u56fe\u4eec\u6c5f\u53d1\u5c55\u57fa\u91d1]\n\u201c[In order to] [ Arg2 promote the development of the Tumen River region],\n[Arg1 South Korea donated one million dollars to establish the Tumen\nRiver Development Fund].\u201d\nThese discourse treebanks have been used for shared tasks on multilingual dis-\ncourse parsing (Xue et al., 2016).\n24.2 Discourse Structure Parsing\nGiven a sequence of sentences, how can we automatically determine the coherence\nrelations between them? This task is often called discourse parsing (even thoughdiscourse\nparsing\nfor PDTB we are only assigning labels to leaf spans and not building a full parse 24.2 \u2022 D ISCOURSE STRUCTURE PARSING 537\ntree as we do for RST).\n24.2.1 EDU segmentation for RST parsing\nRST parsing is generally done in two stages. The \ufb01rst stage, EDU segmentation ,\nextracts the start and end of each EDU. The output of this stage would be a labeling\nlike the following:\n(24.16) [Mr. Rambo says] e1[that a 3.2-acre property] e2[overlooking the San\nFernando Valley] e3[is priced at $4 million] e4[because the late actor Erroll\nFlynn once lived there.] e5\nSince EDUs roughly correspond to clauses, early models of EDU segmentation\n\ufb01rst ran a syntactic parser, and then post-processed the output. Modern systems\ngenerally use neural sequence models supervised by the gold EDU segmentation in\ndatasets like the RST Discourse Treebank. Fig. 24.4 shows an example architecture\nsimpli\ufb01ed from the algorithm of Lukasik et al. (2020) that predicts for each token\nwhether or not it is a break. Here the input sentence is passed through an encoder\nand then passed through a linear layer and a softmax to produce a sequence of 0s\nand 1, where 1 indicates the start of an EDU.\nMr.RambosaysthatENCODER\u20260001linear layersoftmaxEDU break\nFigure 24.4 Predicting EDU segment beginnings from encoded text.\n24.2.2 RST parsing\nTools for building RST coherence structure for a discourse have long been based on\nsyntactic parsing algorithms like shift-reduce parsing (Marcu, 1999). Many modern\nRST parsers since Ji and Eisenstein (2014) draw on the neural syntactic parsers we\nsaw in Chapter 20, using representation learning to build representations for each\nspan, and training a parser to choose the correct shift and reduce actions based on\nthe gold parses in the training set.\nWe\u2019ll describe the shift-reduce parser of Yu et al. (2018). The parser state con-\nsists of a stack and a queue, and produces this structure by taking a series of actions\non the states. Actions include:\n\u2022shift : pushes the \ufb01rst EDU in the queue onto the stack creating a single-node\nsubtree.\n\u2022reduce (l,d): merges the top two subtrees on the stack, where lis the coherence\nrelation label, and dis the nuclearity direction, d2fNN;NS;SNg.\nAs well as the pop root operation, to remove the \ufb01nal tree from the stack.\nFig. 24.6 shows the actions the parser takes to build the structure in Fig. 24.5. 538 CHAPTER 24 \u2022 D ISCOURSE COHERENCE\n560e1 e2 e3 e4attr elabelab e1: American Telephone & Telegraph Co. said it\ne2: will lay off 75 to 85 technicians here , effective Nov. 1.\ne3: The workers install , maintain and repair its private branch exchanges,\ne4: which are large intracompany telephone networks.\nFigure 1: An example of RST discourse tree, where {e1,e2,e3,e4}are EDUs, attr andelab are\ndiscourse relation labels, and arrows indicate the nuclearities of discourse relations.\nRST discourse parsing. Other studies still adopt discrete syntax features proposed by statistical models,\nfeeding them into neural network models (Braud et al., 2016; Braud et al., 2017).\nThe above approaches model syntax trees in an explicit way, requiring discrete syntax parsing outputs\nas inputs for RST parsing. These approaches may suffer from the error propagation problem. Syntax trees\nproduced by a supervised syntax parsing model could have errors, which may propagate into discourse\nparsing models. The problem could be extremely serious when inputs of discourse parsing have different\ndistributions with the training data of the supervised syntax parser. Recently, Zhang et al. (2017) suggest\nan alternative method, which extracts syntax features from a Bi-Af\ufb01ne dependency parser (Dozat and\nManning, 2016), and the method gives competitive performances on relation extraction. It actually\nrepresents syntax trees implicitly, thus it can reduce the error propagation problem.\nIn this work, we investigate the implicit syntax feature extraction approach for RST parsing. In ad-\ndition, we propose a transition-based neural model for this task, which is able to incorporate various\nfeatures \ufb02exibly. We exploit hierarchical bi-directional LSTMs (Bi-LSTMs) to encode texts, and further\nenhance the transition-based model with dynamic oracle. Based on the proposed model, we study the\neffectiveness of our proposed implicit syntax features. We conduct experiments on a standard RST dis-\ncourse TreeBank (Carlson et al., 2003). First, we evaluate the performance of our proposed transition-\nbased baseline, \ufb01nding that the model is able to achieve strong performances after applying dynamic\noracle. Then we evaluate the effectiveness of implicit syntax features extracted from a Bi-Af\ufb01ne depen-\ndency parser. Results show that the implicit syntax features are effective, giving better performances than\nexplicit Tree-LSTM (Li et al., 2015b). Our codes will be released for public under the Apache License\n2.0 at https://github.com/yunan4nlp/NNDisParser .\nIn summary, we mainly make the following two contributions in this work: (1) we propose a transition-\nbased neural RST discourse parsing model with dynamic oracle, (2) we compare three different syntactic\nintegration approaches proposed by us. The rest of the paper is organized as follows. Section 2 describes\nour proposed models including the transition-based neural model, the dynamic oracle strategy and the\nimplicit syntax feature extraction approach. Section 3 presents the experiments to evaluate our models.\nSection 4 shows the related work. Finally, section 5 draws conclusions.\n2 Transition-based Discourse Parsing\nWe follow Ji and Eisenstein (2014), exploiting a transition-based framework for RST discourse parsing.\nThe framework is conceptually simple and \ufb02exible to support arbitrary features, which has been widely\nused in a number of NLP tasks (Zhu et al., 2013; Dyer et al., 2015; Zhang et al., 2016). In addition, a\ntransition-based model formalizes a certain task into predicting a sequence of actions, which is essential\nsimilar to sequence-to-sequence models proposed recently (Bahdanau et al., 2014). In the following,\nwe \ufb01rst describe the transition system for RST discourse parsing, and then introduce our neural network\nmodel by its encoder and decoder parts, respectively. Thirdly, we present our proposed dynamic oracle\nstrategy aiming to enhance the transition-based model. Then we introduce the integration method of\nimplicit syntax features. Finally we describe the training method of our neural network models.\n2.1 The Transition-based System\nThe transition-based framework converts a structural learning problem into a sequence of action predic-\ntions, whose key point is a transition system. A transition system consists of two parts: states and actions.\nThe states are used to store partially-parsed results and the actions are used to control state transitions.\nFigure 24.5 Example RST discourse tree, showing four EDUs. Figure from Yu et al. (2018).\n561Step Stack Queue Action Relation\n1 ?e1,e2,e3,e4 SH ?\n2 e1 e2,e3,e4 SH ?\n3 e1,e2 e3,e4 RD(attr,SN) ?\n4 e1:2 e3,e4 SH de1e2\n5 e1:2,e3 e4 SH de1e2\n6 e1:2,e3,e4 ?RD(elab,NS) de1e2\n7 e1:2,e3:4 ?RD(elab,SN) de1e2,de3e4\n8 e1:4 ? PR de1e2,de3e4,\\e1:2e3:4\nTable 1: An example of the transition-based system for RST discourse parsing.\nThe initial state is an empty state, and the \ufb01nal state represents a full result. There are three kinds of\nactions in our transition system:\n\u2022Shift (SH), which removes the \ufb01rst EDU in the queue onto the stack, forming a single-node subtree.\n\u2022Reduce (RD) ( l,d), which merges the top two subtrees on the stack, where lis a discourse relation\nlabel, and d2{NN,NS,SN}indicates the relation nuclearity (nuclear (N) or satellite (S)).\n\u2022Pop Root (PR), which pops out the top tree on the stack, marking the decoding being completed,\nwhen the stack holds only one subtree and the queue is empty.\nGiven the RST tree as shown in Figure 1, it can be generated by the following action sequence: {SH,\nSH, RD (attr,SN) , SH, SH, RD (elab,NS) , RD (elab,SN) , PR}. Table 1 shows the decoding\nprocess in detail. By this way, we naturally convert RST discourse parsing into predicting a sequence of\ntransition actions, where each line includes a state and next step action referring to the tree.\n2.2 Encoder-Decoder\nPrevious transition-based RST discourse parsing studies exploit statistical models, using manually-\ndesigned discrete features (Sagae, 2009; Heilman and Sagae, 2015; Wang et al., 2017). In this work, we\npropose a transition-based neural model for RST discourse parsing, which follows an encoder-decoder\nframework. Given an input sequence of EDUs {e1,e2,. . . ,e n}, the encoder computes the input represen-\ntations {he\n1,he\n2,. . . ,he\nn}, and the decoder predicts next step actions conditioned on the encoder outputs.\n2.2.1 Encoder\nWe follow Li et al. (2016), using hierarchical Bi-LSTMs to encode the source EDU inputs, where the\n\ufb01rst-layer is used to represent sequencial words inside of EDUs, and the second layer is used to represent\nsequencial EDUs. Given an input sentence {w1,w2,. . . ,w m}, \ufb01rst we represent each word by its form\n(e.g., wi) and POS tag (e.g. ti), concatenating their neural embeddings. By this way, the input vectors\nof the \ufb01rst-layer Bi-LSTM are {xw\n1,xw\n2,. . . ,xw\nm}, where xw\ni=emb (wi)\u0000emb (ti), and then we apply\nBi-LSTM directly, obtaining:\n{hw\n1,hw\n2,. . . ,hw\nm}=Bi-LSTM ({xw\n1,xw\n2,. . . ,xw\nm}) (1)\nThe second-layer Bi-LSTM is built over sequential EDUs. We should \ufb01rst obtain a suitable representa-\ntion for each EDU, which is composed by a span of words inside a certain sentence. Assuming an EDU\nwith its words by {ws,ws+1,. . . ,w t}, after applying the \ufb01rst-layer Bi-LSTM, we obtain their representa-\ntions by {hw\ns,hw\ns+1...,hw\nt}, then we calculate the EDU representation by average pooling:\nxe=1\nt\u0000s+1tX\nshw\nk (2)\nWhen the EDU representations are ready, we apply the second-layer Bi-LSTM directly, resulting:\n{he\n1,he\n2,. . . ,he\nn}=Bi-LSTM ({xe\n1,xe\n2,. . . ,xe\nn}) (3)\nFigure 24.6 Parsing the example of Fig. 24.5 using a shift-reduce parser. Figure from Yu\net al. (2018).\nThe Yu et al. (2018) uses an encoder-decoder architecture, where the encoder\nrepresents the input span of words and EDUs using a hierarchical biLSTM. The\n\ufb01rst biLSTM layer represents the words inside an EDU, and the second represents\nthe EDU sequence. Given an input sentence w1;w2;:::;wm, the words can be repre-\nsented as usual (by static embeddings, combinations with character embeddings or\ntags, or contextual embeddings) resulting in an input word representation sequence\nxw\n1;xw\n2;:::;xw\nm. The result of the word-level biLSTM is then a sequence of hwvalues:\nhw\n1;hw\n2;:::;hw\nm=biLSTM (xw\n1;xw\n2;:::;xw\nm) (24.17)\nAn EDU of span ws;ws+1;:::;wtthen has biLSTM output representation hw\ns;hw\ns+1;:::;hw\nt,\nand is represented by average pooling:\nxe=1\nt\u0000s+1tX\nk=shw\nk (24.18)\nThe second layer uses this input to compute a \ufb01nal representation of the sequence of\nEDU representations he:\nhe\n1;he\n2;:::;he\nn=biLSTM (xe\n1;xe\n2;:::;xe\nn) (24.19)\nThe decoder is then a feedforward network Wthat outputs an action obased on a\nconcatenation of the top three subtrees on the stack ( so;s1;s2) plus the \ufb01rst EDU in\nthe queue (q0):\no=W(ht\ns0;ht\ns1;ht\ns2;he\nq0) (24.20)\nwhere the representation of the EDU on the queue he\nq0comes directly from the\nencoder, and the three hidden vectors representing partial trees are computed by\naverage pooling over the encoder output for the EDUs in those trees:\nhts=1\nj\u0000i+1jX\nk=ihe\nk (24.21) 24.2 \u2022 D ISCOURSE STRUCTURE PARSING 539\nTraining \ufb01rst maps each RST gold parse tree into a sequence of oracle actions, and\nthen uses the standard cross-entropy loss (with l2regularization) to train the system\nto take such actions. Give a state Sand oracle action a, we \ufb01rst compute the decoder\noutput using Eq. 24.20, apply a softmax to get probabilities:\npa=exp(oa)P\na02Aexp(oa0)(24.22)\nand then computing the cross-entropy loss:\nLCE() =\u0000log(pa)+l\n2jjQjj2(24.23)\nRST discourse parsers are evaluated on the test section of the RST Discourse Tree-\nbank, either with gold EDUs or end-to-end, using the RST-Pareval metrics (Marcu,\n2000b). It is standard to \ufb01rst transform the gold RST trees into right-branching bi-\nnary trees, and to report four metrics: trees with no labels (S for Span), labeled\nwith nuclei (N), with relations (R), or both (F for Full), for each metric computing\nmicro-averaged F 1over all spans from all documents (Marcu 2000b, Morey et al.\n2017).\n24.2.3 PDTB discourse parsing\nPDTB discourse parsing, the task of detecting PDTB coherence relations between\nspans, is sometimes called shallow discourse parsing because the task just involvesshallow\ndiscourse\nparsing\ufb02at relationships between text spans, rather than the full trees of RST parsing.\nThe set of four subtasks for PDTB discourse parsing was laid out by Lin et al.\n(2014) in the \ufb01rst complete system, with separate tasks for explicit (tasks 1-3) and\nimplicit (task 4) connectives:\n1. Find the discourse connectives (disambiguating them from non-discourse uses)\n2. Find the two spans for each connective\n3. Label the relationship between these spans\n4. Assign a relation between every adjacent pair of sentences\nMany systems have been proposed for Task 4: taking a pair of adjacent sentences\nas input and assign a coherence relation sense label as output. The setup often fol-\nlows Lin et al. (2009) in assuming gold sentence span boundaries and assigning each\nadjacent span one of the 11 second-level PDTB tags or none (removing the 5 very\nrare tags of the 16 shown in italics in Fig. 24.3).\nA simple but very strong algorithm for Task 4 is to represent each of the two\nspans by BERT embeddings and take the last layer hidden state corresponding to\nthe position of the [CLS] token, pass this through a single layer tanh feedforward\nnetwork and then a softmax for sense classi\ufb01cation (Nie et al., 2019).\nEach of the other tasks also have been addressed. Task 1 is to disambiguat-\ning discourse connectives from their non-discourse use. For example as Pitler and\nNenkova (2009) point out, the word andis a discourse connective linking the two\nclauses by an elaboration/expansion relation in (24.24) while it\u2019s a non-discourse\nNP conjunction in (24.25):\n(24.24) Selling picked up as previous buyers bailed out of their positions and\naggressive short sellers\u2014anticipating further declines\u2014moved in.\n(24.25) My favorite colors are blue and green. 540 CHAPTER 24 \u2022 D ISCOURSE COHERENCE\nSimilarly, once is a discourse connective indicating a temporal relation in (24.26),\nbut simply a non-discourse adverb meaning \u2018formerly\u2019 and modifying used in (24.27):\n(24.26) The asbestos \ufb01ber, crocidolite, is unusually resilient once it enters the\nlungs, with even brief exposures to it causing symptoms that show up\ndecades later, researchers said.\n(24.27) A form of asbestos once used to make Kent cigarette \ufb01lters has caused a\nhigh percentage of cancer deaths among a group of workers exposed to it\nmore than 30 years ago, researchers reported.\nDetermining whether a word is a discourse connective is thus a special case\nof word sense disambiguation. Early work on disambiguation showed that the 4\nPDTB high-level sense classes could be disambiguated with high (94%) accuracy\nused syntactic features from gold parse trees (Pitler and Nenkova, 2009). Recent\nwork performs the task end-to-end from word inputs using a biLSTM-CRF with\nBIO outputs ( B-CONN ,I-CONN ,O) (Yu et al., 2019).\nFor task 2, PDTB spans can be identi\ufb01ed with the same sequence models used to\n\ufb01nd RST EDUs: a biLSTM sequence model with pretrained contextual embedding\n(BERT) inputs (Muller et al., 2019). Simple heuristics also do pretty well as a base-\nline at \ufb01nding spans, since 93% of relations are either completely within a single\nsentence or span two adjacent sentences, with one argument in each sentence (Biran\nand McKeown, 2015).\n24.3 Centering and Entity-Based Coherence\nA second way a discourse can be coherent is by virtue of being \u201cabout\u201d some entity.\nThis idea that at each point in the discourse some entity is salient, and a discourse\nis coherent by continuing to discuss the same entity, appears early in functional lin-\nguistics and the psychology of discourse (Chafe 1976, Kintsch and Van Dijk 1978),\nand soon made its way to computational models. In this section we introduce two\nmodels of this kind of entity-based coherence :Centering Theory (Grosz et al., entity-based\n1995), and the entity grid model of Barzilay and Lapata (2008).\n24.3.1 Centering\nCentering Theory (Grosz et al., 1995) is a theory of both discourse salience andCentering\nTheory\ndiscourse coherence. As a model of discourse salience, Centering proposes that at\nany given point in the discourse one of the entities in the discourse model is salient:\nit is being \u201ccentered\u201d on. As a model of discourse coherence, Centering proposes\nthat discourses in which adjacent sentences CONTINUE to maintain the same salient\nentity are more coherent than those which SHIFT back and forth between multiple\nentities (we will see that CONTINUE and SHIFT are technical terms in the theory).\nThe following two texts from Grosz et al. (1995) which have exactly the same\npropositional content but different saliences, can help in understanding the main\nCentering intuition.\n(24.28) a. John went to his favorite music store to buy a piano.\nb. He had frequented the store for many years.\nc. He was excited that he could \ufb01nally buy a piano.\nd. He arrived just as the store was closing for the day. 24.3 \u2022 C ENTERING AND ENTITY -BASED COHERENCE 541\n(24.29) a. John went to his favorite music store to buy a piano.\nb. It was a store John had frequented for many years.\nc. He was excited that he could \ufb01nally buy a piano.\nd. It was closing just as John arrived.\nWhile these two texts differ only in how the two entities (John and the store) are\nrealized in the sentences, the discourse in (24.28) is intuitively more coherent than\nthe one in (24.29). As Grosz et al. (1995) point out, this is because the discourse\nin (24.28) is clearly about one individual, John, describing his actions and feelings.\nThe discourse in (24.29), by contrast, focuses \ufb01rst on John, then the store, then back\nto John, then to the store again. It lacks the \u201caboutness\u201d of the \ufb01rst discourse.\nCentering Theory realizes this intuition by maintaining two representations for\neach utterance Un. The backward-looking center ofUn, denoted as Cb(Un), rep-backward-\nlooking\ncenterresents the current salient entity, the one being focused on in the discourse after Un\nis interpreted. The forward-looking centers ofUn, denoted as Cf(Un), are a setforward-looking\ncenter\nof potential future salient entities, the discourse entities evoked by Unany of which\ncould serve as Cb(the salient entity) of the following utterance, i.e. Cb(Un+1).\nThe set of forward-looking centers Cf(Un)are ranked according to factors like\ndiscourse salience and grammatical role (for example subjects are higher ranked\nthan objects, which are higher ranked than all other grammatical roles). We call the\nhighest-ranked forward-looking center Cp(for \u201cpreferred center\u201d). Cpis a kind of\nprediction about what entity will be talked about next. Sometimes the next utterance\nindeed talks about this entity, but sometimes another entity becomes salient instead.\nWe\u2019ll use here the algorithm for centering presented in Brennan et al. (1987),\nwhich de\ufb01nes four intersentential relationships between a pair of utterances Unand\nUn+1that depend on the relationship between Cb(Un+1),Cb(Un), and Cp(Un+1);\nthese are shown in Fig. 24.7.\nCb(Un+1) =Cb(Un) Cb(Un+1)6=Cb(Un)\nor unde\ufb01ned Cb(Un)\nCb(Un+1) =Cp(Un+1) Continue Smooth-Shift\nCb(Un+1)6=Cp(Un+1) Retain Rough-Shift\nFigure 24.7 Centering Transitions for Rule 2 from Brennan et al. (1987).\nThe following rules are used by the algorithm:\nRule 1: If any element of Cf(Un)is realized by a pronoun in utterance\nUn+1, then Cb(Un+1)must be realized as a pronoun also.\nRule 2 :Transition states are ordered. Continue is preferred to Retain is\npreferred to Smooth-Shift is preferred to Rough-Shift.\nRule 1 captures the intuition that pronominalization (including zero-anaphora)\nis a common way to mark discourse salience. If there are multiple pronouns in an\nutterance realizing entities from the previous utterance, one of these pronouns must\nrealize the backward center Cb; if there is only one pronoun, it must be Cb.\nRule 2 captures the intuition that discourses that continue to center the same en-\ntity are more coherent than ones that repeatedly shift to other centers. The transition\ntable is based on two factors: whether the backward-looking center Cbis the same\nfrom UntoUn+1and whether this discourse entity is the one that is preferred ( Cp)\nin the new utterance Un+1. If both of these hold, a CONTINUE relation, the speaker\nhas been talking about the same entity and is going to continue talking about that 542 CHAPTER 24 \u2022 D ISCOURSE COHERENCE\nentity. In a RETAIN relation, the speaker intends to SHIFT to a new entity in a future\nutterance and meanwhile places the current entity in a lower rank Cf. In a SHIFT\nrelation, the speaker is shifting to a new salient entity.\nLet\u2019s walk though the start of (24.28) again, repeated as (24.30), showing the\nrepresentations after each utterance is processed.\n(24.30) John went to his favorite music store to buy a piano. ( U1)\nHe was excited that he could \ufb01nally buy a piano. ( U2)\nHe arrived just as the store was closing for the day. ( U3)\nIt was closing just as John arrived ( U4)\nUsing the grammatical role hierarchy to order the C f, for sentence U1we get:\nCf(U1):fJohn, music store, piano g\nCp(U1): John\nCb(U1): unde\ufb01ned\nand then for sentence U2:\nCf(U2):fJohn, pianog\nCp(U2): John\nCb(U2): John\nResult: Continue ( Cp(U2)=Cb(U2);Cb(U1)unde\ufb01ned)\nThe transition from U1toU2is thus a CONTINUE . Completing this example is left\nas exercise (1) for the reader\n24.3.2 Entity Grid model\nCentering embodies a particular theory of how entity mentioning leads to coher-\nence: that salient entities appear in subject position or are pronominalized, and that\ndiscourses are salient by means of continuing to mention the same entity in such\nways.\nThe entity grid model of Barzilay and Lapata (2008) is an alternative way to entity grid\ncapture entity-based coherence: instead of having a top-down theory, the entity-grid\nmodel using machine learning to induce the patterns of entity mentioning that make\na discourse more coherent.\nThe model is based around an entity grid , a two-dimensional array that repre-\nsents the distribution of entity mentions across sentences. The rows represent sen-\ntences, and the columns represent discourse entities (most versions of the entity grid\nmodel focus just on nominal mentions). Each cell represents the possible appearance\nof an entity in a sentence, and the values represent whether the entity appears and its\ngrammatical role. Grammatical roles are subject ( S), object ( O), neither ( X), or ab-\nsent (\u2013); in the implementation of Barzilay and Lapata (2008), subjects of passives\nare represented with O, leading to a representation with some of the characteristics\nof thematic roles.\nFig. 24.8 from Barzilay and Lapata (2008) shows a grid for the text shown in\nFig. 24.9. There is one row for each of the six sentences. The second column, for\nthe entity \u2018trial\u2019, is O\u2013 \u2013 \u2013 X, showing that the trial appears in the \ufb01rst sentence as\ndirect object, in the last sentence as an oblique, and does not appear in the middle\nsentences. The third column, for the entity Microsoft, shows that it appears as sub-\nject in sentence 1 (it also appears as the object of the preposition against , but entities\nthat appear multiple times are recorded with their highest-ranked grammatical func-\ntion). Computing the entity grids requires extracting entities and doing coreference 24.3 \u2022 C ENTERING AND ENTITY -BASED COHERENCE 543\nComputational Linguistics Volume 34, Number 1\nthese patterns can be encoded as feature vectors appropriate for performing coherence-\nrelated ranking and classi\ufb01cation tasks.\n3.1 The Entity-Grid Discourse Representation\nEach text is represented by an entity grid ,at w o - d i m e n s i o n a la r r a yt h a tc a p t u r e s\nthe distribution of discourse entities across text sentences. We follow Miltsakaki and\nKukich (2000) in assuming that our unit of analysis is the traditional sentence (i.e., a\nmain clause with accompanying subordinate and adjunct clauses). The rows of the\ngrid correspond to sentences, and the columns correspond to discourse entities. By\ndiscourse entity we mean a class of coreferent noun phrases (we explain in Section 3.3\nhow coreferent entities are identi\ufb01ed). For each occurrence of a discourse entity in the\ntext, the corresponding grid cell contains information about its presence or absence\nin a sequence of sentences. In addition, for entities present in a given sentence, grid\ncells contain information about their syntactic role. Such information can be expressed\nin many ways (e.g., using constituent labels or thematic role information). Because\ngrammatical relations \ufb01gure prominently in entity-based theories of local coherence (see\nSection 2), they serve as a logical point of departure. Each grid cell thus corresponds to\na string from a set of categories re\ufb02ecting whether the entity in question is a subject ( S),\nobject ( O), or neither ( X). Entities absent from a sentence are signaled by gaps ( \u2013).\nGrammatical role information can be extracted from the output of a broad-coverage\ndependency parser (Lin 2001; Briscoe and Carroll 2002) or any state-of-the art statistical\nparser (Collins 1997; Charniak 2000). We discuss how this information was computed\nfor our experiments in Section 3.3.\nTable 1 illustrates a fragment of an entity grid constructed for the text in Table 2.\nBecause the text contains six sentences, the grid columns are of length six. Consider\nfor instance the grid column for the entity trial ,[O\u2013\u2013\u2013\u2013 X].I tr e c o r d st h a t trial is\npresent in sentences 1 and 6 (as Oand X,r e s p e c t i v e l y )b u ti sa b s e n tf r o mt h er e s to ft h e\nsentences. Also note that the grid in Table 1 takes coreference resolution into account.\nEven though the same entity appears in different linguistic forms, for example, Microsoft\nCorp. ,Microsoft ,a n d the company , it is mapped to a single entry in the grid (see the\ncolumn introduced by Microsoft in Table 1).\nTable 1\nA fragment of the entity grid. Noun phrases are represented by their head nouns. Grid cells\ncorrespond to grammatical roles: subjects ( S), objects ( O), or neither ( X).Department\nTrial\nMicrosoft\nEvidence\nCompetitors\nMarkets\nProducts\nBrands\nCase\nNetscape\nSoftware\nTactics\nGovernment\nSuit\nEarnings\n1SO SXO \u2013\u2013\u2013\u2013\u2013\u2013\u2013\u2013\u2013\u2013 1\n2\u2013\u2013 O\u2013\u2013 XSO \u2013\u2013\u2013\u2013 \u2013\u2013\u2013 2\n3\u2013\u2013 SO \u2013\u2013\u2013\u2013 SOO \u2013\u2013\u2013 \u2013 3\n4\u2013\u2013 S\u2013\u2013\u2013\u2013\u2013\u2013\u2013\u2013 S\u2013\u2013\u2013 4\n5\u2013\u2013\u2013\u2013\u2013\u2013\u2013\u2013\u2013\u2013\u2013\u2013 SO \u20135\n6\u2013XS \u2013\u2013\u2013\u2013\u2013 \u2013\u2013\u2013 \u2013 \u2013 \u2013 O6\n6\nFigure 24.8 Part of the entity grid for the text in Fig. 24.9. Entities are listed by their head\nnoun; each cell represents whether an entity appears as subject ( S), object ( O), neither ( X), or\nis absent (\u2013). Figure from Barzilay and Lapata (2008).\nBarzilay and Lapata Modeling Local Coherence\nTable 2\nSummary augmented with syntactic annotations for grid computation.\n1 [The Justice Department]Sis conducting an [anti-trust trial]Oagainst [Microsoft Corp.]X\nwith [evidence]Xthat [the company]Sis increasingly attempting to crush [competitors]O.\n2[ M i c r o s o f t ]Ois accused of trying to forcefully buy into [markets]Xwhere [its own\nproducts]Sare not competitive enough to unseat [established brands]O.\n3[ T h e c a s e ]Srevolves around [evidence]Oof [Microsoft]Saggressively pressuring\n[Netscape]Ointo merging [browser software]O.\n4[ M i c r o s o f t ]Sclaims [its tactics]Sare commonplace and good economically.\n5 [The government]Smay \ufb01le [a civil suit]Oruling that [conspiracy]Sto curb [competition]O\nthrough [collusion]Xis [a violation of the Sherman Act]O.\n6[ M i c r o s o f t ]Scontinues to show [increased earnings]Odespite [the trial]X.\nWhen a noun is attested more than once with a different grammatical role in the\nsame sentence, we default to the role with the highest grammatical ranking: subjects are\nranked higher than objects, which in turn are ranked higher than the rest. For example,\nthe entity Microsoft is mentioned twice in Sentence 1 with the grammatical roles x(for\nMicrosoft Corp. )a n d s(forthe company ), but is represented only by sin the grid (see\nTables 1 and 2).\n3.2 Entity Grids as Feature Vectors\nAf u n d a m e n t a la s s u m p t i o nu n d e r l y i n go u ra p p r o a c hi st h a tt h ed i s t r i b u t i o no fe n t i t i e s\nin coherent texts exhibits certain regularities re\ufb02ected in grid topology. Some of these\nregularities are formalized in Centering Theory as constraints on transitions of the\nlocal focus in adjacent sentences. Grids of coherent texts are likely to have some dense\ncolumns (i.e., columns with just a few gaps, such as Microsoft in Table 1) and many\nsparse columns which will consist mostly of gaps (see markets andearnings in Table 1).\nOne would further expect that entities corresponding to dense columns are more often\nsubjects or objects. These characteristics will be less pronounced in low-coherence texts.\nInspired by Centering Theory, our analysis revolves around patterns of local entity\ntransitions. A local entity transition is a sequence {S,O,X,\u2013}nthat represents entity\noccurrences and their syntactic roles in nadjacent sentences. Local transitions can be\neasily obtained from a grid as continuous subsequences of each column. Each transition\nwill have a certain probability in a given grid. For instance, the probability of the\ntransition [S\u2013]in the grid from Table 1 is 0 .08 (computed as a ratio of its frequency\n[i.e., six] divided by the total number of transitions of length two [i.e., 75]). Each text\ncan thus be viewed as a distribution de\ufb01ned over transition types.\nWe can now go one step further and represent each text by a \ufb01xed set of transition\nsequences using a standard feature vector notation. Each grid rendering jof a document\ndicorresponds to a feature vector \u03a6(xij)=(p1(xij),p2(xij),... ,pm(xij)), where mis the\nnumber of all prede\ufb01ned entity transitions, and pt(xij)t h ep r o b a b i l i t yo ft r a n s i t i o n t\nin grid xij.T h i sf e a t u r ev e c t o rr e p r e s e n t a t i o ni su s e f u l l ya m e n a b l et om a c h i n el e a r n i n g\nalgorithms (see our experiments in Sections 4\u20136). Furthermore, it allows the consid-\neration of large numbers of transitions which could potentially uncover novel entity\ndistribution patterns relevant for coherence assessment or other coherence-related tasks.\nNote that considerable latitude is available when specifying the transition types to\nbe included in a feature vector. These can be all transitions of a given length (e.g., two\nor three) or the most frequent transitions within a document collection. An example of\n7\nFigure 24.9 A discourse with the entities marked and annotated with grammatical func-\ntions. Figure from Barzilay and Lapata (2008).\nresolution to cluster them into discourse entities (Chapter 23) as well as parsing the\nsentences to get grammatical roles.\nIn the resulting grid, columns that are dense (like the column for Microsoft) in-\ndicate entities that are mentioned often in the texts; sparse columns (like the column\nfor earnings) indicate entities that are mentioned rarely.\nIn the entity grid model, coherence is measured by patterns of local entity tran-\nsition . For example, Department is a subject in sentence 1, and then not men-\ntioned in sentence 2; this is the transition [ S\u2013]. The transitions are thus sequences\nfS,O X, \u2013gnwhich can be extracted as continuous cells from each column. Each\ntransition has a probability; the probability of [ S\u2013] in the grid from Fig. 24.8 is 0.08\n(it occurs 6 times out of the 75 total transitions of length two). Fig. 24.10 shows the\ndistribution over transitions of length 2 for the text of Fig. 24.9 (shown as the \ufb01rst\nrowd1), and 2 other documents.\nComputational Linguistics Volume 34, Number 1\naf e a t u r es p a c ew i t ht r a n s i t i o n so fl e n g t ht w oi si l l u s t r a t e di nT a b l e3 .T h es e c o n dr o w\n(introduced by d1)i st h ef e a t u r ev e c t o rr e p r e s e n t a t i o no ft h eg r i di nT a b l e1 .\n3.3 Grid Construction: Linguistic Dimensions\nOne of the central research issues in developing entity-based models of coherence is\ndetermining what sources of linguistic knowledge are essential for accurate prediction,\nand how to encode them succinctly in a discourse representation. Previous approaches\ntend to agree on the features of entity distribution related to local coherence\u2014the\ndisagreement lies in the way these features are modeled.\nOur study of alternative encodings is not a mere duplication of previous ef-\nforts (Poesio et al. 2004) that focus on linguistic aspects of parameterization. Because we\nare interested in an automatically constructed model, we have to take into account com-\nputational and learning issues when considering alternative representations. Therefore,\nour exploration of the parameter space is guided by three considerations: the linguistic\nimportance of a parameter, the accuracy of its automatic computation, and the size of the\nresulting feature space. From the linguistic side, we focus on properties of entity distri-\nbution that are tightly linked to local coherence, and at the same time allow for multiple\ninterpretations during the encoding process. Computational considerations prevent us\nfrom considering discourse representations that cannot be computed reliably by exist-\ning tools. For instance, we could not experiment with the granularity of an utterance\u2014\nsentence versus clause\u2014because available clause separators introduce substantial noise\ninto a grid construction. Finally, we exclude representations that will explode the size of\nthe feature space, thereby increasing the amount of data required for training the model.\nEntity Ex traction. The accurate computation of entity classes is key to computing mean-\ningful entity grids. In previous implementations of entity-based models, classes of coref-\nerent nouns have been extracted manually (Miltsakaki and Kukich 2000; Karamanis\net al. 2004; Poesio et al. 2004), but this is not an option for our model. An obvious\nsolution for identifying entity classes is to employ an automatic coreference resolution\ntool that determines which noun phrases refer to the same entity in a document.\nCurrent approaches recast coreference resolution as a classi\ufb01cation task. A pair\nof NPs is classi\ufb01ed as coreferring or not based on constraints that are learned from\nan annotated corpus. A separate clustering mechanism then coordinates the possibly\ncontradictory pairwise classi\ufb01cations and constructs a partition on the set of NPs. In\nour experiments, we employ Ng and Cardie\u2019s (2002) coreference resolution system.\nThe system decides whether two NPs are coreferent by exploiting a wealth of lexical,\ngrammatical, semantic, and positional features. It is trained on the MUC (6\u20137) data sets\nand yields state-of-the-art performance (70.4 F-measure on MUC-6 and 63.4 on MUC-7).\nTable 3\nExample of a feature-vector document representation using all transitions of length two given\nsyntactic categories S,O,X,a n d \u2013.\nSS SO SX S \u2013OS OO OX O \u2013XS XO XX X \u2013\u2013 S\u2013O\u2013X\u2013\u2013\nd1.01 .01 0 .08 .01 0 0 .09 0 0 0 .03 .05 .07 .03 .59\nd2.02 .01 .01 .02 0 .07 0 .02 .14 .14 .06 .04 .03 .07 0.1 .36\nd3.02 0 0 .03 .09 0 .09 .06 0 0 0 .05 .03 .07 .17 .39\n8\nFigure 24.10 A feature vector for representing documents using all transitions of length 2.\nDocument d1is the text in Fig. 24.9. Figure from Barzilay and Lapata (2008).\nThe transitions and their probabilities can then be used as features for a machine\nlearning model. This model can be a text classi\ufb01er trained to produce human-labeled\ncoherence scores (for example from humans labeling each text as coherent or inco-\nherent). But such data is expensive to gather. Barzilay and Lapata (2005) introduced\na simplifying innovation: coherence models can be trained by self-supervision :\ntrained to distinguish the natural original order of sentences in a discourse from 544 CHAPTER 24 \u2022 D ISCOURSE COHERENCE\na modi\ufb01ed order (such as a randomized order). We turn to these evaluations in the\nnext section.\n24.3.3 Evaluating Neural and Entity-based coherence\nEntity-based coherence models, as well as the neural models we introduce in the\nnext section, are generally evaluated in one of two ways.\nFirst, we can have humans rate the coherence of a document and train a classi\ufb01er\nto predict these human ratings, which can be categorial (high/low, or high/mid/low)\nor continuous. This is the best evaluation to use if we have some end task in mind,\nlike essay grading, where human raters are the correct de\ufb01nition of the \ufb01nal label.\nAlternatively, since it\u2019s very expensive to get human labels, and we might not\nyet have an end-task in mind, we can use natural texts to do self-supervision. In\nself-supervision we pair up a natural discourse with a pseudo-document created by\nchanging the ordering. Since naturally-ordered discourses are more coherent than\nrandom permutation (Lin et al., 2011), a successful coherence algorithm should pre-\nfer the original ordering.\nSelf-supervision has been implemented in 3 ways. In the sentence order dis-\ncrimination task (Barzilay and Lapata, 2005), we compare a document to a random\npermutation of its sentences. A model is considered correct for an (original, per-\nmuted) test pair if it ranks the original document higher. Given kdocuments, we can\ncompute npermutations, resulting in knpairs each with one original document and\none permutation, to use in training and testing.\nIn the sentence insertion task (Chen et al., 2007) we take a document, remove\none of the nsentences s, and create n\u00001 copies of the document with sinserted into\neach position. The task is to decide which of the ndocuments is the one with the\noriginal ordering, distinguishing the original position for sfrom all other positions.\nInsertion is harder than discrimination since we are comparing documents that differ\nby only one sentence.\nFinally, in the sentence order reconstruction task (Lapata, 2003), we take a\ndocument, randomize the sentences, and train the model to put them back in the\ncorrect order. Again given kdocuments, we can compute npermutations, resulting\ninknpairs each with one original document and one permutation, to use in training\nand testing. Reordering is of course a much harder task than simple classi\ufb01cation.\n24.4 Representation learning models for local coherence\nThe third kind of local coherence is topical or semantic \ufb01eld coherence. Discourses\ncohere by talking about the same topics and subtopics, and drawing on the same\nsemantic \ufb01elds in doing so.\nThe \ufb01eld was pioneered by a series of unsupervised models in the 1990s of this\nkind of coherence that made use of lexical cohesion (Halliday and Hasan, 1976): lexical cohesion\nthe sharing of identical or semantically related words in nearby sentences. Morris\nand Hirst (1991) computed lexical chains of words (like pine,bush trees ,trunk ) that\noccurred through a discourse and that were related in Roget\u2019s Thesaurus (by being in\nthe same category, or linked categories). They showed that the number and density\nof chain correlated with the topic structure. The TextTiling algorithm of Hearst TextTiling\n(1997) computed the cosine between neighboring text spans (the normalized dot\nproduct of vectors of raw word counts), again showing that sentences or paragraph in 24.4 \u2022 R EPRESENTATION LEARNING MODELS FOR LOCAL COHERENCE 545\na subtopic have high cosine with each other, but not with sentences in a neighboring\nsubtopic.\nA third early model, the LSA Coherence method of Foltz et al. (1998) was the\n\ufb01rst to use embeddings, modeling the coherence between two sentences as the co-\nsine between their LSA sentence embedding vectors1, computing embeddings for a\nsentence sby summing the embeddings of its words w:\nsim(s;t) = cos(s;t)\n=cos(X\nw2sw;X\nw2tw) (24.31)\nand de\ufb01ning the overall coherence of a text as the average similarity over all pairs of\nadjacent sentences siandsi+1:\ncoherence (T) =1\nn\u00001n\u00001X\ni=1cos(si;si+1) (24.32)\nModern neural representation-learning coherence models, beginning with Li et al.\n(2014), draw on the intuitions of these early unsupervised models for learning sen-\ntence representations and measuring how they change between neighboring sen-\ntences. But the new models also draw on the idea pioneered by Barzilay and Lapata\n(2005) of self-supervision. That is, unlike say coherence relation models, which\ntrain on hand-labeled representations for RST or PDTB, these models are trained to\ndistinguish natural discourses from unnatural discourses formed by scrambling the\norder of sentences, thus using representation learning to discover the features that\nmatter for at least the ordering aspect of coherence.\nHere we present one such model, the local coherence discriminator (LCD) (Xu\net al., 2019). Like early models, LCD computes the coherence of a text as the av-\nerage of coherence scores between consecutive pairs of sentences. But unlike the\nearly unsupervised models, LCD is a self-supervised model trained to discriminate\nconsecutive sentence pairs (si;si+1)in the training documents (assumed to be coher-\nent) from (constructed) incoherent pairs (si;s0). All consecutive pairs are positive\nexamples, and the negative (incoherent) partner for a sentence siis another sentence\nuniformly sampled from the same document as si.\nFig. 24.11 describes the architecture of the model fq, which takes a sentence\npair and returns a score, higher scores for more coherent pairs. Given an input\nsentence pair sandt, the model computes sentence embeddings sandt(using any\nsentence embeddings algorithm), and then concatenates four features of the pair: (1)\nthe concatenation of the two vectors (2) their difference s\u0000t; (3) the absolute value\nof their difference js\u0000tj; (4) their element-wise product s\ft. These are passed\nthrough a one-layer feedforward network to output the coherence score.\nThe model is trained to make this coherence score higher for real pairs than for\nnegative pairs. More formally, the training objective for a corpus Cof documents d,\neach of which consists of a list of sentences si, is:\nLq=X\nd2CX\nsi2dE\np(s0jsi)[L(fq(si;si+1);fq(si;s0))] (24.33)\nEp(s0jsi)is the expectation with respect to the negative sampling distribution con-\nditioned on si: given a sentence sithe algorithms samples a negative sentence s0\n1See Chapter 6 for more on LSA embeddings; they are computed by applying SVD to the term-\ndocument matrix (each cell weighted by log frequency and normalized by entropy), and then the \ufb01rst\n300 dimensions are used as the embedding. 546 CHAPTER 24 \u2022 D ISCOURSE COHERENCE\n681Loss function: The role of the loss function is\nto encourage f+=f\u2713(si,si+1)to be high while\nf\u0000=f\u2713(si,s0)to be low. Common losses such as\nmargin or log loss can all be used. Through exper-\nimental validation, we found that margin loss to\nbe superior for this problem. Speci\ufb01cally, Ltakes\non the form: L(f+,f\u0000) = max(0 ,\u2318\u0000f++f\u0000)\nwhere \u2318is the margin hyperparameter.\nNegative samples: Technically, we are free to\nchoose any sentence s0to form a negative pair\nwith si. However, because of potential differ-\nences in genre, topic and writing style, such neg-\natives might cause the discriminative model to\nlearn cues unrelated to coherence. Therefore, we\nonly select sentences from the same document to\nconstruct negative pairs. Speci\ufb01cally, suppose si\ncomes from document dkwith length nk, then\np(s0|si)is a uniform distribution over the nk\u00001\nsentences {sj}j6=ifrom dk. For a document with\nnsentences, there are n\u00001positive pairs, and\n(n\u00001)\u21e4(n\u00002)/2negative pairs. It turns out that\nthe quadratic number of negatives provides a rich\nenough learning signal, while at the same time, is\nnot too prohibitively large to be effectively cov-\nered by a sampling procedure. In practice, we\nsample a new set of negatives each time we see\na document, hence after many epochs, we can ef-\nfectively cover the space for even very long doc-\numents. Section 5.7discusses further details on\nsampling.\n4.1 Model Architecture\nThe speci\ufb01c neural architecture that we use for f\u2713\nis illustrated in Figure 1. We assume the use of\nsome pre-trained sentence encoder, which is dis-\ncussed in the next section.\nGiven an input sentence pair, the sentence en-\ncoder maps the sentences to real-valued vectors S\nandT. We then compute the concatenation of the\nfollowing features: (1)concatenation of the two\nvectors (S, T);(2)element-wise difference S\u0000T;\n(3)element-wise product S\u21e4T;(4)absolute value\nof element-wise difference |S\u0000T|. The concate-\nnated feature representation is then fed to a one-\nlayer MLP to output the coherence score.\nIn practice, we make our overall coherence\nmodel bidirectional, by training a forward model\nwith input (S, T)and a backward model with in-\nput(T,S)with the same architecture but separate\nparameters. The coherence score is then the aver-\nage from the two models.\nFigure 1: Generic architecture for our proposed model.\n4.2 Pre-trained Generative Model as the\nSentence Encoder\nOur model can work with any pre-trained sen-\ntence encoder, ranging from the most simplistic\naverage GloVe ( Pennington et al. ,2014 ) embed-\ndings to more sophisticated supervised or unsu-\npervised pre-trained sentence encoders ( Conneau\net al. ,2017 ). As mentioned in the introduction,\nsince generative models can often be turned into\nsentence encoder, generative coherence model can\nbe leveraged by our model to bene\ufb01t from the\nadvantages of both generative and discriminative\ntraining, similar to ( Kiros et al. ,2015 ;Peters et al. ,\n2018 ). After initialization, we freeze the genera-\ntive model parameters to avoid over\ufb01tting.\nIn Section 5, we will experimentally show that\nwhile we do bene\ufb01t from strong pre-trained en-\ncoders, the fact that our local discriminative model\nimproves over previous methods is independent of\nthe choice of sentence encoder.\n5 Experiments\n5.1 Evaluation Tasks\nFollowing Nguyen and Joty (2017 ) and other pre-\nvious work, we evaluate our models on the dis-\ncrimination and insertion tasks. Additionally, we\nevaluate on the paragraph reconstruction task in\nopen-domain settings, in a similar manner to Li\nand Jurafsky (2017 ).\nIn the discrimination task, a document is com-\npared to a random permutation of its sentences,\nand the model is considered correct if it scores the\noriginal document higher than the permuted one.\nTwenty permutations are used in the test set in ac-\ncordance with previous work.\nFigure 24.11 The architecture of the LCD model of document coherence, showing the\ncomputation of the score for a pair of sentences sandt. Figure from Xu et al. (2019).\nuniformly over the other sentences in the same document. Lis a loss function that\ntakes two scores, one for a positive pair and one for a negative pair, with the goal of\nencouraging f+=fq(si;si+1)to be high and f\u0000=fq(si;s0))to be low. Fig. 24.11\nuse the margin loss l(f+;f\u0000) =max(0;h\u0000f++f\u0000)where his the margin hyper-\nparameter.\nXu et al. (2019) also give a useful baseline algorithm that itself has quite high\nperformance in measuring perplexity: train an RNN language model on the data,\nand compute the log likelihood of sentence siin two ways, once given the preceding\ncontext (conditional log likelihood) and once with no context (marginal log likeli-\nhood). The difference between these values tells us how much the preceding context\nimproved the predictability of si, a predictability measure of coherence.\nTraining models to predict longer contexts than just consecutive pairs of sen-\ntences can result in even stronger discourse representations. For example a Trans-\nformer language model trained with a contrastive sentence objective to predict text\nup to a distance of \u00062 sentences improves performance on various discourse coher-\nence tasks (Iter et al., 2020).\nLanguage-model style models are generally evaluated by the methods of Sec-\ntion 24.3.3, although they can also be evaluated on the RST and PDTB coherence\nrelation tasks.\n24.5 Global Coherence\nA discourse must also cohere globally rather than just at the level of pairs of sen-\ntences. Consider stories, for example. The narrative structure of stories is one of\nthe oldest kinds of global coherence to be studied. In his in\ufb02uential Morphology of\nthe Folktale , Propp (1968) models the discourse structure of Russian folktales via\na kind of plot grammar. His model includes a set of character categories he called\ndramatis personae , like Hero, Villain, Donor, or Helper, and a set of events he\ncalled functions (like \u201cVillain commits kidnapping\u201d, \u201cDonor tests Hero\u201d, or \u201cHero\nis pursued\u201d) that have to occur in particular order, along with other components.\nPropp shows that the plots of each of the fairy tales he studies can be represented as 24.5 \u2022 G LOBAL COHERENCE 547\na sequence of these functions, different tales choosing different subsets of functions,\nbut always in the same order. Indeed Lakoff (1972) showed that Propp\u2019s model\namounted to a discourse grammar of stories, and in recent computational work Fin-\nlayson (2016) demonstrates that some of these Proppian functions could be induced\nfrom corpora of folktale texts by detecting events that have similar actions across\nstories. Bamman et al. (2013) showed that generalizations over dramatis personae\ncould be induced from movie plot summaries on Wikipedia. Their model induced\nlatent personae from features like the actions the character takes (e.g., Villains stran-\ngle), the actions done to them (e.g., Villains are foiled and arrested) or the descriptive\nwords used of them (Villains are evil).\nIn this section we introduce two kinds of such global discourse structure that\nhave been widely studied computationally. The \ufb01rst is the structure of arguments:\nthe way people attempt to convince each other in persuasive essays by offering\nclaims and supporting premises. The second is somewhat related: the structure of\nscienti\ufb01c papers, and the way authors present their goals, results, and relationship to\nprior work in their papers.\n24.5.1 Argumentation Structure\nThe \ufb01rst type of global discourse structure is the structure of arguments . Analyzing\npeople\u2019s argumentation computationally is often called argumentation mining .argumentation\nmining\nThe study of arguments dates back to Aristotle, who in his Rhetorics described\nthree components of a good argument: pathos (appealing to the emotions of the pathos\nlistener), ethos (appealing to the speaker\u2019s personal character), and logos (the logical ethos\nlogos structure of the argument).\nMost of the discourse structure studies of argumentation have focused on logos ,\nparticularly via building and training on annotated datasets of persuasive essays or\nother arguments (Reed et al. 2008, Stab and Gurevych 2014a, Peldszus and Stede\n2016, Habernal and Gurevych 2017, Musi et al. 2018). Such corpora, for exam-\nple, often include annotations of argumentative components like claims (the central claims\ncomponent of the argument that is controversial and needs support) and premises premises\n(the reasons given by the author to persuade the reader by supporting or attacking\nthe claim or other premises), as well as the argumentative relations between themargumentative\nrelations\nlike SUPPORT and ATTACK .\nConsider the following example of a persuasive essay from Stab and Gurevych\n(2014b). The \ufb01rst sentence (1) presents a claim (in bold). (2) and (3) present two\npremises supporting the claim. (4) gives a premise supporting premise (3).\n\u201c(1) Museums and art galleries provide a better understanding\nabout arts than Internet. (2) In most museums and art galleries, de-\ntailed descriptions in terms of the background, history and author are\nprovided. (3) Seeing an artwork online is not the same as watching it\nwith our own eyes, as (4) the picture online does not show the texture\nor three-dimensional structure of the art, which is important to study.\u201d\nThus this example has three argumentative relations: SUPPORT (2,1), SUPPORT (3,1)\nand SUPPORT (4,3). Fig. 24.12 shows the structure of a much more complex argu-\nment.\nWhile argumentation mining is clearly related to rhetorical structure and other\nkinds of coherence relations, arguments tend to be much less local; often a persua-\nsive essay will have only a single main claim, with premises spread throughout the\ntext, without the local coherence we see in coherence relations. 548 CHAPTER 24 \u2022 D ISCOURSE COHERENCE\nStab and Gurevych Parsing Argumentation Structures\ncloning. This example illustrates that knowing argumentative relations is important for\nseparating several arguments in a paragraph. The example also shows that argument\ncomponents frequently exhibit preceding text units that are not relevant to the argument\nbut helpful for recognizing the argument component type. For example, preceding dis-\ncourse connectors like \u201ctherefore\u201d, \u201cconsequently\u201d, or \u201cthus\u201d can signal a subsequent\nclaim. Discourse markers like \u201cbecause\u201d, \u201csince\u201d, or \u201cfurthermore\u201d could indicate a\npremise. Formally, these preceding tokens of an argument component starting at token\ntiare de\ufb01ned as the tokens ti\u0000m,...,ti\u00001that are not covered by another argument\ncomponent in the sentence s=t1,t2,...,tnwhere 1 \uf8ffi\uf8ffnand i\u0000m\u00001. The third body\nparagraph illustrates a contra argument and argumentative attack relations:\nAdmittedly, [ cloning could bemisused formilitary purposes] Claim 5. For example,\n[:it:::::could:::be:::::used::to::::::::::manipulate:::::::human::::::genes::in::::::order::to::::::create::::::::obedient:::::::soldiers\n::::with::::::::::::extraordinary:::::::abilities] Premise 9. However, because [::::moral::::and:::::::ethical::::::values:::are\n::::::::::::internationally::::::shared] Premise 10,[:it:::is::::very::::::::unlikely::::that:::::::cloning::::will::be::::::::misused:::for\n::::::militant:::::::::objectives] Premise 11.\nThe paragraph begins with Claim 5, which attacks the stance of the author. It is supported\nbyPremise 9in the second sentence. The third sentence includes two premises, both of\nwhich defend the stance of the author. Premise 11is an attack of Claim 5, and Premise 10\nsupports Premise 11. The last paragraph (conclusion) restates the major claim and sum-\nmarizes the main aspects of the essay:\nTo sum up, although [ permitting cloning might bear some risks like misuse for\nmilitary purposes] Claim 6, I strongly believe that [ this technology is bene\ufb01cial to\nhumanity ]MajorClaim 2. It is likely that [ thistechnologybears some important cures which\nwill significantly improve lifeconditions] Claim 7.\nThe conclusion of the essay starts with an attacking claim followed by the restatement of\nthe major claim. The last sentence includes another claim that summarizes the most im-\nportant points of the author\u2019s argumentation. Figure 2 shows the entire argumentation\nstructure of the example essay.\nFigure 2Argumentation structure of the example essay. Arrows indicate argumentative relations.Arrowheads denote argumentative support relations and circleheads attack relations. Dashedlines indicate relations that are encoded in the stance attributes of claims. \u201cP\u201d denotes premises.629\nFigure 24.12 Argumentation structure of a persuasive essay. Arrows indicate argumentation relations, ei-\nther of SUPPORT (with arrowheads) or ATTACK (with circleheads); P denotes premises. Figure from Stab and\nGurevych (2017).\nAlgorithms for detecting argumentation structure often include classi\ufb01ers for\ndistinguishing claims, premises, or non-argumentation, together with relation clas-\nsi\ufb01ers for deciding if two spans have the SUPPORT ,ATTACK , or neither relation\n(Peldszus and Stede, 2013). While these are the main focus of much computational\nwork, there is also preliminary efforts on annotating and detecting richer semantic\nrelationships (Park and Cardie 2014, Hidey et al. 2017) such as detecting argumen-\ntation schemes , larger-scale structures for argument like argument from example ,argumentation\nschemes\norargument from cause to effect , orargument from consequences (Feng and\nHirst, 2011).\nAnother important line of research is studying how these argument structure (or\nother features) are associated with the success or persuasiveness of an argument\n(Habernal and Gurevych 2016, Tan et al. 2016, Hidey et al. 2017. Indeed, while it\nis Aristotle\u2019s logos that is most related to discourse structure, Aristotle\u2019s ethos and\npathos techniques are particularly relevant in the detection of mechanisms of this\nsort of persuasion . For example scholars have investigated the linguistic realization persuasion\nof features studied by social scientists like reciprocity (people return favors), social\nproof (people follow others\u2019 choices), authority (people are in\ufb02uenced by those\nwith power), and scarcity (people value things that are scarce), all of which can\nbe brought up in a persuasive argument (Cialdini, 1984). Rosenthal and McKeown\n(2017) showed that these features could be combined with argumentation structure\nto predict who in\ufb02uences whom on social media, Althoff et al. (2014) found that\nlinguistic models of reciprocity and authority predicted success in online requests,\nwhile the semisupervised model of Yang et al. (2019) detected mentions of scarcity,\ncommitment, and social identity to predict the success of peer-to-peer lending plat-\nforms.\nSee Stede and Schneider (2018) for a comprehensive survey of argument mining.\n24.5.2 The structure of scienti\ufb01c discourse\nScienti\ufb01c papers have a very speci\ufb01c global structure: somewhere in the course of\nthe paper the authors must indicate a scienti\ufb01c goal, develop a method for a solu-\ntion, provide evidence for the solution, and compare to prior work. One popular 24.6 \u2022 S UMMARY 549\nannotation scheme for modeling these rhetorical goals is the argumentative zon-\ningmodel of Teufel et al. (1999) and Teufel et al. (2009), which is informed by theargumentative\nzoning\nidea that each scienti\ufb01c paper tries to make a knowledge claim about a new piece\nof knowledge being added to the repository of the \ufb01eld (Myers, 1992). Sentences\nin a scienti\ufb01c paper can be assigned one of 15 tags; Fig. 24.13 shows 7 (shortened)\nexamples of labeled sentences.\nCategory Description Example\nAIM Statement of speci\ufb01c research goal, or\nhypothesis of current paper\u201cThe aim of this process is to examine the role that\ntraining plays in the tagging process\u201d\nOWNMETHOD New Knowledge claim, own work:\nmethods\u201cIn order for it to be useful for our purposes, the\nfollowing extensions must be made:\u201d\nOWNRESULTS Measurable/objective outcome of own\nwork\u201cAll the curves have a generally upward trend but\nalways lie far below backoff (51% error rate)\u201d\nUSE Other work is used in own work \u201cWe use the framework for the allocation and\ntransfer of control of Whittaker....\u201d\nGAPWEAK Lack of solution in \ufb01eld, problem with\nother solutions\u201cHere, we will produce experimental evidence\nsuggesting that this simple model leads to serious\noverestimates\u201d\nSUPPORT Other work supports current work or is\nsupported by current work\u201cWork similar to that described here has been car-\nried out by Merialdo (1994), with broadly similar\nconclusions.\u201d\nANTISUPPORT Clash with other\u2019s results or theory; su-\nperiority of own work\u201cThis result challenges the claims of...\u201d\nFigure 24.13 Examples for 7 of the 15 labels from the Argumentative Zoning labelset (Teufel et al., 2009).\nTeufel et al. (1999) and Teufel et al. (2009) develop labeled corpora of scienti\ufb01c\narticles from computational linguistics and chemistry, which can be used as supervi-\nsion for training standard sentence-classi\ufb01cation architecture to assign the 15 labels.\n24.6 Summary\nIn this chapter we introduced local and global models for discourse coherence .\n\u2022 Discourses are not arbitrary collections of sentences; they must be coherent .\nAmong the factors that make a discourse coherent are coherence relations\nbetween the sentences, entity-based coherence, and topical coherence.\n\u2022 Various sets of coherence relations andrhetorical relations have been pro-\nposed. The relations in Rhetorical Structure Theory ( RST ) hold between\nspans of text and are structured into a tree. Because of this, shift-reduce\nand other parsing algorithms are generally used to assign these structures.\nThe Penn Discourse Treebank ( PDTB ) labels only relations between pairs of\nspans, and the labels are generally assigned by sequence models.\n\u2022Entity-based coherence captures the intuition that discourses are about an\nentity, and continue mentioning the entity from sentence to sentence. Cen-\ntering Theory is a family of models describing how salience is modeled for\ndiscourse entities, and hence how coherence is achieved by virtue of keeping\nthe same discourse entities salient over the discourse. The entity grid model\ngives a more bottom-up way to compute which entity realization transitions\nlead to coherence. 550 CHAPTER 24 \u2022 D ISCOURSE COHERENCE\n\u2022 Many different genres have different types of global coherence . Persuasive\nessays have claims and premises that are extracted in the \ufb01eld of argument\nmining , scienti\ufb01c articles have structure related to aims, methods, results, and\ncomparisons.\nBibliographical and Historical Notes\nCoherence relations arose from the independent development of a number of schol-\nars, including Hobbs (1979) idea that coherence relations play an inferential role for\nthe hearer, and the investigations by Mann and Thompson (1987) of the discourse\nstructure of large texts. Other approaches to coherence relations and their extrac-\ntion include Segmented Discourse Representation Theory ( SDRT ) (Asher and Las- SDRT\ncarides 2003, Baldridge et al. 2007) and the Linguistic Discourse Model (Polanyi\n1988, Scha and Polanyi 1988, Polanyi et al. 2004). Wolf and Gibson (2005) argue\nthat coherence structure includes crossed bracketings, which make it impossible to\nrepresent as a tree, and propose a graph representation instead. A compendium of\nover 350 relations that have been proposed in the literature can be found in Hovy\n(1990).\nRST parsing was \ufb01rst proposed by Marcu (1997), and early work was rule-based,\nfocused on discourse markers (Marcu, 2000a). The creation of the RST Discourse\nTreeBank (Carlson et al. 2001, Carlson and Marcu 2001) enabled a wide variety\nof machine learning algorithms, beginning with the shift-reduce parser of Marcu\n(1999) that used decision trees to choose actions, and continuing with a wide variety\nof machine learned parsing methods (Soricut and Marcu 2003, Sagae 2009, Hernault\net al. 2010, Feng and Hirst 2014, Surdeanu et al. 2015, Joty et al. 2015) and chunkers\n(Sporleder and Lapata, 2005). Subba and Di Eugenio (2009) integrated sophisticated\nsemantic information into RST parsing. Ji and Eisenstein (2014) \ufb01rst applied neural\nmodels to RST parsing neural models, leading to the modern set of neural RST\nmodels (Li et al. 2014, Li et al. 2016b, Braud et al. 2017, Yu et al. 2018, inter alia)\nas well as neural segmenters (Wang et al. 2018b). and neural PDTB parsing models\n(Ji and Eisenstein 2015, Qin et al. 2016, Qin et al. 2017).\nBarzilay and Lapata (2005) pioneered the idea of self-supervision for coher-\nence: training a coherence model to distinguish true orderings of sentences from\nrandom permutations. Li et al. (2014) \ufb01rst applied this paradigm to neural sentence-\nrepresentation, and many neural self-supervised models followed (Li and Jurafsky\n2017, Logeswaran et al. 2018, Lai and Tetreault 2018, Xu et al. 2019, Iter et al.\n2020)\nAnother aspect of global coherence is the global topic structure of a text, the way\nthe topics shift over the course of the document. Barzilay and Lee (2004) introduced\nan HMM model for capturing topics for coherence, and later work expanded this\nintuition (Soricut and Marcu 2006, Elsner et al. 2007, Louis and Nenkova 2012, Li\nand Jurafsky 2017).\nThe relationship between explicit and implicit discourse connectives has been\na fruitful one for research. Marcu and Echihabi (2002) \ufb01rst proposed to use sen-\ntences with explicit relations to help provide training data for implicit relations, by\nremoving the explicit relations and trying to re-predict them as a way of improv-\ning performance on implicit connectives; this idea was re\ufb01ned by Sporleder and\nLascarides (2005), (Pitler et al., 2009), and Rutherford and Xue (2015). This rela- BIBLIOGRAPHICAL AND HISTORICAL NOTES 551\ntionship can also be used as a way to create discourse-aware representations. The\nDisSent algorithm (Nie et al., 2019) creates the task of predicting explicit discourse\nmarkers between two sentences. They show that representations learned to be good\nat this task also function as powerful sentence representations for other discourse\ntasks.\nThe idea of entity-based coherence seems to have arisen in multiple \ufb01elds in the\nmid-1970s, in functional linguistics (Chafe, 1976), in the psychology of discourse\nprocessing (Kintsch and Van Dijk, 1978), and in the roughly contemporaneous work\nof Grosz, Sidner, Joshi, and their colleagues. Grosz (1977a) addressed the focus\nof attention that conversational participants maintain as the discourse unfolds. She\nde\ufb01ned two levels of focus; entities relevant to the entire discourse were said to\nbe in global focus, whereas entities that are locally in focus (i.e., most central to\na particular utterance) were said to be in immediate focus. Sidner 1979; 1983 de-\nscribed a method for tracking (immediate) discourse foci and their use in resolving\npronouns and demonstrative noun phrases. She made a distinction between the cur-\nrent discourse focus and potential foci, which are the predecessors to the backward-\nand forward-looking centers of Centering theory, respectively. The name and further\nroots of the centering approach lie in papers by Joshi and Kuhn (1979) and Joshi and\nWeinstein (1981), who addressed the relationship between immediate focus and the\ninferences required to integrate the current utterance into the discourse model. Grosz\net al. (1983) integrated this work with the prior work of Sidner and Grosz. This led\nto a manuscript on centering which, while widely circulated since 1986, remained\nunpublished until Grosz et al. (1995). A collection of centering papers appears in\nWalker et al. (1998b). See Karamanis et al. (2004) and Poesio et al. (2004) for a\ndeeper exploration of centering and its parameterizations, and the History section of\nChapter 23 for more on the use of centering on coreference.\nThe grid model of entity-based coherence was \ufb01rst proposed by Barzilay and\nLapata (2005) drawing on earlier work by Lapata (2003) and Barzilay, and then\nextended by them Barzilay and Lapata (2008) and others with additional features\n(Elsner and Charniak 2008, 2011, Feng et al. 2014, Lin et al. 2011) a model that\nprojects entities into a global graph for the discourse (Guinaudeau and Strube 2013,\nMesgar and Strube 2016), and a convolutional model to capture longer-range entity\ndependencies (Nguyen and Joty, 2017).\nTheories of discourse coherence have also been used in algorithms for interpret-\ning discourse-level linguistic phenomena, including verb phrase ellipsis and gap-\nping (Asher 1993, Kehler 1993), and tense interpretation (Lascarides and Asher\n1993, Kehler 1994, Kehler 2000). An extensive investigation into the relationship\nbetween coherence relations and discourse connectives can be found in Knott and\nDale (1994).\nUseful surveys of discourse processing and structure include Stede (2011) and\nWebber et al. (2012).\nAndy Kehler wrote the Discourse chapter for the 2000 \ufb01rst edition of this text-\nbook, which we used as the starting point for the second-edition chapter, and there\nare some remnants of Andy\u2019s lovely prose still in this third-edition coherence chap-\nter. 552 CHAPTER 24 \u2022 D ISCOURSE COHERENCE\nExercises\n24.1 Finish the Centering Theory processing of the last two utterances of (24.30),\nand show how (24.29) would be processed. Does the algorithm indeed mark\n(24.29) as less coherent?\n24.2 Select an editorial column from your favorite newspaper, and determine the\ndiscourse structure for a 10\u201320 sentence portion. What problems did you\nencounter? Were you helped by super\ufb01cial cues the speaker included (e.g.,\ndiscourse connectives) in any places? Bibliography\nAbadi, M., A. Agarwal, P. Barham,\nE. Brevdo, Z. Chen, C. Citro,\nG. S. Corrado, A. Davis, J. Dean,\nM. Devin, S. Ghemawat, I. Good-\nfellow, A. Harp, G. Irving, M. Is-\nard, Y . Jia, R. Jozefowicz, L. Kaiser,\nM. Kudlur, J. Levenberg, D. Man \u00b4e,\nR. Monga, S. Moore, D. Murray,\nC. Olah, M. Schuster, J. Shlens,\nB. Steiner, I. Sutskever, K. Talwar,\nP. Tucker, V . Vanhoucke, V . Vasude-\nvan, F. Vi \u00b4egas, O. Vinyals, P. War-\nden, M. Wattenberg, M. Wicke,\nY . Yu, and X. Zheng. 2015. Tensor-\nFlow: Large-scale machine learning\non heterogeneous systems. Software\navailable from tensor\ufb02ow.org.\nAbney, S. P., R. E. Schapire, and\nY . Singer. 1999. Boosting ap-\nplied to tagging and PP attachment.\nEMNLP/VLC .\nAgarwal, O., S. Subramanian,\nA. Nenkova, and D. Roth. 2019.\nEvaluation of named entity corefer-\nence. Workshop on Computational\nModels of Reference, Anaphora and\nCoreference .\nAggarwal, C. C. and C. Zhai. 2012.\nA survey of text classi\ufb01cation al-\ngorithms. In C. C. Aggarwal and\nC. Zhai, eds, Mining text data , 163\u2013\n222. Springer.\nAgichtein, E. and L. Gravano. 2000.\nSnowball: Extracting relations from\nlarge plain-text collections. Pro-\nceedings of the 5th ACM Interna-\ntional Conference on Digital Li-\nbraries .\nAgirre, E., C. Banea, C. Cardie, D. Cer,\nM. Diab, A. Gonzalez-Agirre,\nW. Guo, I. Lopez-Gazpio, M. Mar-\nitxalar, R. Mihalcea, G. Rigau,\nL. Uria, and J. Wiebe. 2015.\nSemEval-2015 task 2: Semantic\ntextual similarity, English, Span-\nish and pilot on interpretability.\nSemEval-15 .\nAgirre, E., M. Diab, D. Cer,\nand A. Gonzalez-Agirre. 2012.\nSemEval-2012 task 6: A pilot on se-\nmantic textual similarity. SemEval-\n12.\nAgirre, E. and D. Martinez. 2001.\nLearning class-to-class selectional\npreferences. CoNLL .\nAho, A. V . and J. D. Ullman. 1972. The\nTheory of Parsing, Translation, and\nCompiling , volume 1. Prentice Hall.\nAlgoet, P. H. and T. M. Cover. 1988.\nA sandwich proof of the Shannon-\nMcMillan-Breiman theorem. The\nAnnals of Probability , 16(2):899\u2013\n909.Allen, J. 1984. Towards a general the-\nory of action and time. Arti\ufb01cial In-\ntelligence , 23(2):123\u2013154.\nAllen, J. and C. R. Perrault. 1980. An-\nalyzing intention in utterances. Arti-\n\ufb01cial Intelligence , 15:143\u2013178.\nAllen, J., M. S. Hunnicut, and D. H.\nKlatt. 1987. From Text to Speech:\nThe MITalk system . Cambridge Uni-\nversity Press.\nAlthoff, T., C. Danescu-Niculescu-\nMizil, and D. Jurafsky. 2014. How\nto ask for a favor: A case study\non the success of altruistic requests.\nICWSM 2014 .\nAn, J., H. Kwak, and Y .-Y . Ahn.\n2018. SemAxis: A lightweight\nframework to characterize domain-\nspeci\ufb01c word semantics beyond sen-\ntiment. ACL.\nAnastasopoulos, A. and G. Neubig.\n2020. Should all cross-lingual em-\nbeddings speak English? ACL.\nAntoniak, M. and D. Mimno.\n2018. Evaluating the stability of\nembedding-based word similarities.\nTACL , 6:107\u2013119.\nAone, C. and S. W. Bennett. 1995. Eval-\nuating automated and manual acqui-\nsition of anaphora resolution strate-\ngies. ACL.\nAriel, M. 2001. Accessibility the-\nory: An overview. In T. Sanders,\nJ. Schilperoord, and W. Spooren,\neds, Text Representation: Linguistic\nand Psycholinguistic Aspects , 29\u2013\n87. Benjamins.\nArora, S., P. Lewis, A. Fan, J. Kahn, and\nC. R \u00b4e. 2023. Reasoning over pub-\nlic and private data in retrieval-based\nsystems. TACL , 11:902\u2013921.\nArtetxe, M. and H. Schwenk. 2019.\nMassively multilingual sentence em-\nbeddings for zero-shot cross-lingual\ntransfer and beyond. TACL , 7:597\u2013\n610.\nArtstein, R., S. Gandhe, J. Gerten,\nA. Leuski, and D. Traum. 2009.\nSemi-formal evaluation of conver-\nsational characters. In Languages:\nFrom Formal to Natural , 22\u201335.\nSpringer.\nAsher, N. 1993. Reference to Abstract\nObjects in Discourse . Studies in Lin-\nguistics and Philosophy (SLAP) 50,\nKluwer.\nAsher, N. and A. Lascarides. 2003. Log-\nics of Conversation . Cambridge Uni-\nversity Press.\nAtal, B. S. and S. Hanauer. 1971.\nSpeech analysis and synthesis by\nprediction of the speech wave. JASA ,\n50:637\u2013655.Austin, J. L. 1962. How to Do Things\nwith Words . Harvard University\nPress.\nAwadallah, A. H., R. G. Kulkarni,\nU. Ozertem, and R. Jones. 2015.\nCharaterizing and predicting voice\nquery reformulation. CIKM-15 .\nBa, J. L., J. R. Kiros, and G. E. Hinton.\n2016. Layer normalization. NeurIPS\nworkshop .\nBaayen, R. H. 2001. Word frequency\ndistributions . Springer.\nBaccianella, S., A. Esuli, and F. Sebas-\ntiani. 2010. Sentiwordnet 3.0: An\nenhanced lexical resource for senti-\nment analysis and opinion mining.\nLREC .\nBach, K. and R. Harnish. 1979. Linguis-\ntic communication and speech acts .\nMIT Press.\nBackus, J. W. 1959. The syntax\nand semantics of the proposed in-\nternational algebraic language of the\nZurich ACM-GAMM Conference.\nInformation Processing: Proceed-\nings of the International Conference\non Information Processing, Paris .\nUNESCO.\nBackus, J. W. 1996. Transcript of ques-\ntion and answer session. In R. L.\nWexelblat, ed., History of Program-\nming Languages , page 162. Aca-\ndemic Press.\nBada, M., M. Eckert, D. Evans, K. Gar-\ncia, K. Shipley, D. Sitnikov, W. A.\nBaumgartner, K. B. Cohen, K. Ver-\nspoor, J. A. Blake, and L. E. Hunter.\n2012. Concept annotation in the\ncraft corpus. BMC bioinformatics ,\n13(1):161.\nBagga, A. and B. Baldwin. 1998.\nAlgorithms for scoring coreference\nchains. LREC Workshop on Linguis-\ntic Coreference .\nBahdanau, D., K. H. Cho, and Y . Ben-\ngio. 2015. Neural machine transla-\ntion by jointly learning to align and\ntranslate. ICLR 2015 .\nBahdanau, D., J. Chorowski,\nD. Serdyuk, P. Brakel, and Y . Ben-\ngio. 2016. End-to-end attention-\nbased large vocabulary speech\nrecognition. ICASSP .\nBahl, L. R. and R. L. Mercer. 1976.\nPart of speech assignment by a sta-\ntistical decision algorithm. Proceed-\nings IEEE International Symposium\non Information Theory .\nBahl, L. R., F. Jelinek, and R. L.\nMercer. 1983. A maximum likeli-\nhood approach to continuous speech\nrecognition. IEEE Transactions on\nPattern Analysis and Machine Intel-\nligence , 5(2):179\u2013190.\n553 554 Bibliography\nBajaj, P., D. Campos, N. Craswell,\nL. Deng, J. G. ando Xiaodong Liu,\nR. Majumder, A. McNamara, B. Mi-\ntra, T. Nguye, M. Rosenberg,\nX. Song, A. Stoica, S. Tiwary, and\nT. Wang. 2016. MS MARCO: A\nhuman generated MAchine Reading\nCOmprehension dataset. NeurIPS .\nBaker, C. F., C. J. Fillmore, and\nJ. B. Lowe. 1998. The Berkeley\nFrameNet project. COLING/ACL .\nBaker, J. K. 1975a. The DRAGON sys-\ntem \u2013 An overview. IEEE Transac-\ntions on ASSP , ASSP-23(1):24\u201329.\nBaker, J. K. 1975b. Stochastic\nmodeling for automatic speech un-\nderstanding. In D. R. Reddy,\ned.,Speech Recognition . Academic\nPress.\nBaldridge, J., N. Asher, and J. Hunter.\n2007. Annotation for and robust\nparsing of discourse structure on\nunrestricted texts. Zeitschrift f \u00a8ur\nSprachwissenschaft , 26:213\u2013239.\nBamman, D., O. Lewke, and A. Man-\nsoor. 2020. An annotated dataset\nof coreference in English literature.\nLREC .\nBamman, D., B. O\u2019Connor, and N. A.\nSmith. 2013. Learning latent per-\nsonas of \ufb01lm characters. ACL.\nBamman, D., S. Popat, and S. Shen.\n2019. An annotated dataset of liter-\nary entities. NAACL HLT .\nBanerjee, S. and A. Lavie. 2005. ME-\nTEOR: An automatic metric for MT\nevaluation with improved correla-\ntion with human judgments. Pro-\nceedings of ACL Workshop on In-\ntrinsic and Extrinsic Evaluation\nMeasures for MT and/or Summa-\nrization .\nBanko, M., M. Cafarella, S. Soderland,\nM. Broadhead, and O. Etzioni. 2007.\nOpen information extraction for the\nweb. IJCAI .\nBa\u02dcn\u00b4on, M., P. Chen, B. Haddow,\nK. Hea\ufb01eld, H. Hoang, M. Espl `a-\nGomis, M. L. Forcada, A. Kamran,\nF. Kirefu, P. Koehn, S. Ortiz Ro-\njas, L. Pla Sempere, G. Ram \u00b4\u0131rez-\nS\u00b4anchez, E. Sarr \u00b4\u0131as, M. Strelec,\nB. Thompson, W. Waites, D. Wig-\ngins, and J. Zaragoza. 2020.\nParaCrawl: Web-scale acquisition\nof parallel corpora. ACL.\nBar-Hillel, Y . 1960. The present sta-\ntus of automatic translation of lan-\nguages. In F. Alt, ed., Advances\nin Computers 1 , 91\u2013163. Academic\nPress.\nBarker, C. 2010. Nominals don\u2019t\nprovide criteria of identity. In\nM. Rathert and A. Alexiadou, eds,\nThe Semantics of Nominalizations\nacross Languages and Frameworks ,\n9\u201324. Mouton.Barrett, L. F., B. Mesquita, K. N.\nOchsner, and J. J. Gross. 2007. The\nexperience of emotion. Annual Re-\nview of Psychology , 58:373\u2013403.\nBarzilay, R. and M. Lapata. 2005. Mod-\neling local coherence: An entity-\nbased approach. ACL.\nBarzilay, R. and M. Lapata. 2008. Mod-\neling local coherence: An entity-\nbased approach. Computational Lin-\nguistics , 34(1):1\u201334.\nBarzilay, R. and L. Lee. 2004. Catching\nthe drift: Probabilistic content mod-\nels, with applications to generation\nand summarization. HLT-NAACL .\nBaum, L. E. and J. A. Eagon. 1967. An\ninequality with applications to sta-\ntistical estimation for probabilistic\nfunctions of Markov processes and\nto a model for ecology. Bulletin of\nthe American Mathematical Society ,\n73(3):360\u2013363.\nBaum, L. E. and T. Petrie. 1966. Statis-\ntical inference for probabilistic func-\ntions of \ufb01nite-state Markov chains.\nAnnals of Mathematical Statistics ,\n37(6):1554\u20131563.\nBaum, L. F. 1900. The Wizard of Oz .\nAvailable at Project Gutenberg.\nBayes, T. 1763. An Essay Toward Solv-\ning a Problem in the Doctrine of\nChances , volume 53. Reprinted in\nFacsimiles of Two Papers by Bayes ,\nHafner Publishing, 1963.\nBazell, C. E. 1952/1966. The corre-\nspondence fallacy in structural lin-\nguistics. In E. P. Hamp, F. W.\nHouseholder, and R. Austerlitz, eds,\nStudies by Members of the English\nDepartment, Istanbul University (3),\nreprinted in Readings in Linguistics\nII (1966) , 271\u2013298. University of\nChicago Press.\nBean, D. and E. Riloff. 1999.\nCorpus-based identi\ufb01cation of non-\nanaphoric noun phrases. ACL.\nBean, D. and E. Riloff. 2004. Unsu-\npervised learning of contextual role\nknowledge for coreference resolu-\ntion. HLT-NAACL .\nBedi, G., F. Carrillo, G. A. Cecchi, D. F.\nSlezak, M. Sigman, N. B. Mota,\nS. Ribeiro, D. C. Javitt, M. Copelli,\nand C. M. Corcoran. 2015. Auto-\nmated analysis of free speech pre-\ndicts psychosis onset in high-risk\nyouths. npj Schizophrenia , 1.\nBej\u02c7cek, E., E. Haji \u02c7cov\u00b4a, J. Haji \u02c7c,\nP. J \u00b4\u0131nov\u00b4a, V . Kettnerov \u00b4a,\nV . Kol \u00b4a\u02c7rov\u00b4a, M. Mikulov \u00b4a,\nJ. M \u00b4\u0131rovsk \u00b4y, A. Nedoluzhko,\nJ. Panevov \u00b4a, L. Pol \u00b4akov \u00b4a,\nM. \u02c7Sev\u02c7c\u00b4\u0131kov\u00b4a, J. \u02c7St\u02c7ep\u00b4anek, and\n\u02c7S. Zik \u00b4anov \u00b4a. 2013. Prague de-\npendency treebank 3.0. Technical\nreport, Institute of Formal and Ap-\nplied Linguistics, Charles Universityin Prague. LINDAT/CLARIN dig-\nital library at Institute of Formal\nand Applied Linguistics, Charles\nUniversity in Prague.\nBellegarda, J. R. 1997. A latent se-\nmantic analysis framework for large-\nspan language modeling. EU-\nROSPEECH .\nBellegarda, J. R. 2000. Exploiting la-\ntent semantic information in statisti-\ncal language modeling. Proceedings\nof the IEEE , 89(8):1279\u20131296.\nBellegarda, J. R. 2013. Natural lan-\nguage technology in mobile devices:\nTwo grounding frameworks. In\nMobile Speech and Advanced Nat-\nural Language Solutions , 185\u2013196.\nSpringer.\nBellman, R. 1957. Dynamic Program-\nming . Princeton University Press.\nBellman, R. 1984. Eye of the Hurri-\ncane: an autobiography . World Sci-\nenti\ufb01c Singapore.\nBender, E. M. 2019. The #BenderRule:\nOn naming the languages we study\nand why it matters. Blog post.\nBender, E. M., B. Friedman, and\nA. McMillan-Major. 2021. A\nguide for writing data statements\nfor natural language processing.\nhttp://techpolicylab.uw.\nedu/data-statements/ .\nBender, E. M. and A. Koller. 2020.\nClimbing towards NLU: On mean-\ning, form, and understanding in the\nage of data. ACL.\nBengio, Y ., A. Courville, and P. Vin-\ncent. 2013. Representation learn-\ning: A review and new perspec-\ntives. IEEE Transactions on Pattern\nAnalysis and Machine Intelligence ,\n35(8):1798\u20131828.\nBengio, Y ., R. Ducharme, and P. Vin-\ncent. 2000. A neural probabilistic\nlanguage model. NeurIPS .\nBengio, Y ., R. Ducharme, P. Vincent,\nand C. Jauvin. 2003. A neural prob-\nabilistic language model. JMLR ,\n3:1137\u20131155.\nBengio, Y ., P. Lamblin, D. Popovici,\nand H. Larochelle. 2007. Greedy\nlayer-wise training of deep net-\nworks. NeurIPS .\nBengio, Y ., H. Schwenk, J.-S. Sen \u00b4ecal,\nF. Morin, and J.-L. Gauvain. 2006.\nNeural probabilistic language mod-\nels. In Innovations in Machine\nLearning , 137\u2013186. Springer.\nBengtson, E. and D. Roth. 2008. Un-\nderstanding the value of features for\ncoreference resolution. EMNLP .\nBentivogli, L., M. Cettolo, M. Federico,\nand C. Federmann. 2018. Machine\ntranslation human evaluation: an in-\nvestigation of evaluation based on\npost-editing and its relation with di-\nrect assessment. ICSLT . Bibliography 555\nBerant, J., A. Chou, R. Frostig, and\nP. Liang. 2013. Semantic parsing\non freebase from question-answer\npairs. EMNLP .\nBerg-Kirkpatrick, T., D. Burkett, and\nD. Klein. 2012. An empirical inves-\ntigation of statistical signi\ufb01cance in\nNLP. EMNLP .\nBerger, A., S. A. Della Pietra, and V . J.\nDella Pietra. 1996. A maximum en-\ntropy approach to natural language\nprocessing. Computational Linguis-\ntics, 22(1):39\u201371.\nBergsma, S. and D. Lin. 2006. Boot-\nstrapping path-based pronoun reso-\nlution. COLING/ACL .\nBergsma, S., D. Lin, and R. Goebel.\n2008a. Discriminative learning of\nselectional preference from unla-\nbeled text. EMNLP .\nBergsma, S., D. Lin, and R. Goebel.\n2008b. Distributional identi\ufb01cation\nof non-referential pronouns. ACL.\nBethard, S. 2013. ClearTK-TimeML:\nA minimalist approach to TempEval\n2013. SemEval-13 .\nBhat, I., R. A. Bhat, M. Shrivastava,\nand D. Sharma. 2017. Joining\nhands: Exploiting monolingual tree-\nbanks for parsing of code-mixing\ndata. EACL .\nBianchi, F., M. Suzgun, G. At-\ntanasio, P. Rottger, D. Jurafsky,\nT. Hashimoto, and J. Zou. 2024.\nSafety-tuned LLaMAs: Lessons\nfrom improving the safety of large\nlanguage models that follow instruc-\ntions. ICLR .\nBickel, B. 2003. Referential density\nin discourse and syntactic typology.\nLanguage , 79(2):708\u2013736.\nBickmore, T. W., H. Trinh, S. Olafsson,\nT. K. O\u2019Leary, R. Asadi, N. M. Rick-\nles, and R. Cruz. 2018. Patient and\nconsumer safety risks when using\nconversational assistants for medical\ninformation: An observational study\nof Siri, Alexa, and Google Assis-\ntant. Journal of Medical Internet Re-\nsearch , 20(9):e11510.\nBikel, D. M., S. Miller, R. Schwartz,\nand R. Weischedel. 1997. Nymble:\nA high-performance learning name-\n\ufb01nder. ANLP .\nBiran, O. and K. McKeown. 2015.\nPDTB discourse parsing as a tagging\ntask: The two taggers approach.\nSIGDIAL .\nBird, S., E. Klein, and E. Loper. 2009.\nNatural Language Processing with\nPython . O\u2019Reilly.\nBisani, M. and H. Ney. 2004. Boot-\nstrap estimates for con\ufb01dence inter-\nvals in ASR performance evaluation.\nICASSP .\nBishop, C. M. 2006. Pattern recogni-\ntion and machine learning . Springer.Bisk, Y ., A. Holtzman, J. Thomason,\nJ. Andreas, Y . Bengio, J. Chai,\nM. Lapata, A. Lazaridou, J. May,\nA. Nisnevich, N. Pinto, and\nJ. Turian. 2020. Experience grounds\nlanguage. EMNLP .\nBizer, C., J. Lehmann, G. Kobilarov,\nS. Auer, C. Becker, R. Cyganiak,\nand S. Hellmann. 2009. DBpedia\u2014\nA crystallization point for the Web\nof Data. Web Semantics: science,\nservices and agents on the world\nwide web , 7(3):154\u2013165.\nBj\u00a8orkelund, A. and J. Kuhn. 2014.\nLearning structured perceptrons for\ncoreference resolution with latent\nantecedents and non-local features.\nACL.\nBlack, A. W. and P. Taylor. 1994.\nCHATR: A generic speech synthesis\nsystem. COLING .\nBlack, E., S. P. Abney, D. Flickinger,\nC. Gdaniec, R. Grishman, P. Har-\nrison, D. Hindle, R. Ingria, F. Je-\nlinek, J. L. Klavans, M. Y . Liberman,\nM. P. Marcus, S. Roukos, B. San-\ntorini, and T. Strzalkowski. 1991. A\nprocedure for quantitatively compar-\ning the syntactic coverage of English\ngrammars. Speech and Natural Lan-\nguage Workshop .\nBlei, D. M., A. Y . Ng, and M. I. Jor-\ndan. 2003. Latent Dirichlet alloca-\ntion. JMLR , 3(5):993\u20131022.\nBlodgett, S. L., S. Barocas,\nH. Daum \u00b4e III, and H. Wallach. 2020.\nLanguage (technology) is power: A\ncritical survey of \u201cbias\u201d in NLP.\nACL.\nBlodgett, S. L., L. Green, and\nB. O\u2019Connor. 2016. Demographic\ndialectal variation in social media:\nA case study of African-American\nEnglish. EMNLP .\nBlodgett, S. L. and B. O\u2019Connor. 2017.\nRacial disparity in natural language\nprocessing: A case study of so-\ncial media African-American En-\nglish. FAT/ML Workshop, KDD .\nBloom\ufb01eld, L. 1914. An Introduction to\nthe Study of Language . Henry Holt\nand Company.\nBloom\ufb01eld, L. 1933. Language . Uni-\nversity of Chicago Press.\nBobrow, D. G., R. M. Kaplan, M. Kay,\nD. A. Norman, H. Thompson, and\nT. Winograd. 1977. GUS, A frame\ndriven dialog system. Arti\ufb01cial In-\ntelligence , 8:155\u2013173.\nBobrow, D. G. and D. A. Norman.\n1975. Some principles of memory\nschemata. In D. G. Bobrow and\nA. Collins, eds, Representation and\nUnderstanding . Academic Press.\nBojanowski, P., E. Grave, A. Joulin, and\nT. Mikolov. 2017. Enriching word\nvectors with subword information.\nTACL , 5:135\u2013146.Bollacker, K., C. Evans, P. Paritosh,\nT. Sturge, and J. Taylor. 2008.\nFreebase: a collaboratively created\ngraph database for structuring hu-\nman knowledge. SIGMOD 2008 .\nBolukbasi, T., K.-W. Chang, J. Zou,\nV . Saligrama, and A. T. Kalai. 2016.\nMan is to computer programmer as\nwoman is to homemaker? Debiasing\nword embeddings. NeurIPS .\nBommasani, R., D. A. Hudson,\nE. Adeli, R. Altman, S. Arora,\nS. von Arx, M. S. Bernstein,\nJ. Bohg, A. Bosselut, E. Brun-\nskill, E. Brynjolfsson, S. Buch,\nD. Card, R. Castellon, N. S. Chat-\nterji, A. S. Chen, K. A. Creel,\nJ. Davis, D. Demszky, C. Don-\nahue, M. Doumbouya, E. Durmus,\nS. Ermon, J. Etchemendy, K. Etha-\nyarajh, L. Fei-Fei, C. Finn, T. Gale,\nL. E. Gillespie, K. Goel, N. D.\nGoodman, S. Grossman, N. Guha,\nT. Hashimoto, P. Henderson, J. He-\nwitt, D. E. Ho, J. Hong, K. Hsu,\nJ. Huang, T. F. Icard, S. Jain, D. Ju-\nrafsky, P. Kalluri, S. Karamcheti,\nG. Keeling, F. Khani, O. Khat-\ntab, P. W. Koh, M. S. Krass,\nR. Krishna, R. Kuditipudi, A. Ku-\nmar, F. Ladhak, M. Lee, T. Lee,\nJ. Leskovec, I. Levent, X. L. Li,\nX. Li, T. Ma, A. Malik, C. D. Man-\nning, S. P. Mirchandani, E. Mitchell,\nZ. Munyikwa, S. Nair, A. Narayan,\nD. Narayanan, B. Newman, A. Nie,\nJ. C. Niebles, H. Nilforoshan, J. F.\nNyarko, G. Ogut, L. Orr, I. Papadim-\nitriou, J. S. Park, C. Piech, E. Porte-\nlance, C. Potts, A. Raghunathan,\nR. Reich, H. Ren, F. Rong, Y . H.\nRoohani, C. Ruiz, J. Ryan, C. R\u2019e,\nD. Sadigh, S. Sagawa, K. San-\nthanam, A. Shih, K. P. Srinivasan,\nA. Tamkin, R. Taori, A. W. Thomas,\nF. Tram `er, R. E. Wang, W. Wang,\nB. Wu, J. Wu, Y . Wu, S. M. Xie,\nM. Yasunaga, J. You, M. A. Zaharia,\nM. Zhang, T. Zhang, X. Zhang,\nY . Zhang, L. Zheng, K. Zhou, and\nP. Liang. 2021. On the opportuni-\nties and risks of foundation models.\nArXiv .\nBooth, T. L. 1969. Probabilistic\nrepresentation of formal languages.\nIEEE Conference Record of the 1969\nTenth Annual Symposium on Switch-\ning and Automata Theory .\nBorges, J. L. 1964. The analytical lan-\nguage of john wilkins. In Other\ninquisitions 1937\u20131952 . University\nof Texas Press. Trans. Ruth L. C.\nSimms.\nBostrom, K. and G. Durrett. 2020. Byte\npair encoding is suboptimal for lan-\nguage model pretraining. EMNLP .\nBourlard, H. and N. Morgan. 1994.\nConnectionist Speech Recognition:\nA Hybrid Approach . Kluwer. 556 Bibliography\nBrants, T. 2000. TnT: A statistical part-\nof-speech tagger. ANLP .\nBrants, T., A. C. Popat, P. Xu, F. J.\nOch, and J. Dean. 2007. Large lan-\nguage models in machine transla-\ntion. EMNLP/CoNLL .\nBraud, C., M. Coavoux, and\nA. S\u00f8gaard. 2017. Cross-lingual\nRST discourse parsing. EACL .\nBr\u00b4eal, M. 1897. Essai de S \u00b4emantique:\nScience des signi\ufb01cations . Hachette.\nBrennan, S. E., M. W. Friedman, and\nC. Pollard. 1987. A centering ap-\nproach to pronouns. ACL.\nBrin, S. 1998. Extracting patterns and\nrelations from the World Wide Web.\nProceedings World Wide Web and\nDatabases International Workshop,\nNumber 1590 in LNCS . Springer.\nBrockmann, C. and M. Lapata. 2003.\nEvaluating and combining ap-\nproaches to selectional preference\nacquisition. EACL .\nBroschart, J. 1997. Why Tongan does\nit differently. Linguistic Typology ,\n1:123\u2013165.\nBrown, P. F., J. Cocke, S. A.\nDella Pietra, V . J. Della Pietra, F. Je-\nlinek, J. D. Lafferty, R. L. Mercer,\nand P. S. Roossin. 1990. A statis-\ntical approach to machine transla-\ntion. Computational Linguistics ,\n16(2):79\u201385.\nBrown, P. F., S. A. Della Pietra, V . J.\nDella Pietra, and R. L. Mercer. 1993.\nThe mathematics of statistical ma-\nchine translation: Parameter esti-\nmation. Computational Linguistics ,\n19(2):263\u2013311.\nBrown, T., B. Mann, N. Ryder,\nM. Subbiah, J. Kaplan, P. Dhari-\nwal, A. Neelakantan, P. Shyam,\nG. Sastry, A. Askell, S. Agar-\nwal, A. Herbert-V oss, G. Krueger,\nT. Henighan, R. Child, A. Ramesh,\nD. M. Ziegler, J. Wu, C. Win-\nter, C. Hesse, M. Chen, E. Sigler,\nM. Litwin, S. Gray, B. Chess,\nJ. Clark, C. Berner, S. McCan-\ndlish, A. Radford, I. Sutskever, and\nD. Amodei. 2020. Language mod-\nels are few-shot learners. NeurIPS ,\nvolume 33.\nBruce, B. C. 1975. Generation as a so-\ncial action. Proceedings of TINLAP-\n1 (Theoretical Issues in Natural\nLanguage Processing) .\nBrysbaert, M., A. B. Warriner, and\nV . Kuperman. 2014. Concrete-\nness ratings for 40 thousand gen-\nerally known English word lem-\nmas. Behavior Research Methods ,\n46(3):904\u2013911.\nBu, H., J. Du, X. Na, B. Wu, and\nH. Zheng. 2017. AISHELL-1: An\nopen-source Mandarin speech cor-\npus and a speech recognition base-\nline. O-COCOSDA Proceedings .Buchholz, S. and E. Marsi. 2006. Conll-\nx shared task on multilingual depen-\ndency parsing. CoNLL .\nBudanitsky, A. and G. Hirst. 2006.\nEvaluating WordNet-based mea-\nsures of lexical semantic related-\nness. Computational Linguistics ,\n32(1):13\u201347.\nBudzianowski, P., T.-H. Wen, B.-\nH. Tseng, I. Casanueva, S. Ultes,\nO. Ramadan, and M. Ga \u02c7si\u00b4c. 2018.\nMultiWOZ - a large-scale multi-\ndomain wizard-of-Oz dataset for\ntask-oriented dialogue modelling.\nEMNLP .\nBullinaria, J. A. and J. P. Levy. 2007.\nExtracting semantic representations\nfrom word co-occurrence statistics:\nA computational study. Behavior re-\nsearch methods , 39(3):510\u2013526.\nBullinaria, J. A. and J. P. Levy.\n2012. Extracting semantic repre-\nsentations from word co-occurrence\nstatistics: stop-lists, stemming, and\nSVD. Behavior research methods ,\n44(3):890\u2013907.\nBulyko, I., K. Kirchhoff, M. Osten-\ndorf, and J. Goldberg. 2005. Error-\nsensitive response generation in a\nspoken language dialogue system.\nSpeech Communication , 45(3):271\u2013\n288.\nCaliskan, A., J. J. Bryson, and\nA. Narayanan. 2017. Semantics de-\nrived automatically from language\ncorpora contain human-like biases.\nScience , 356(6334):183\u2013186.\nCallison-Burch, C., M. Osborne, and\nP. Koehn. 2006. Re-evaluating the\nrole of BLEU in machine translation\nresearch. EACL .\nCanavan, A., D. Graff, and G. Zip-\nperlen. 1997. CALLHOME Ameri-\ncan English speech LDC97S42. Lin-\nguistic Data Consortium.\nCarbonell, J. R. 1970. AI in\nCAI: An arti\ufb01cial-intelligence ap-\nproach to computer-assisted instruc-\ntion. IEEE transactions on man-\nmachine systems , 11(4):190\u2013202.\nCardie, C. 1993. A case-based approach\nto knowledge acquisition for domain\nspeci\ufb01c sentence analysis. AAAI .\nCardie, C. 1994. Domain-Speci\ufb01c\nKnowledge Acquisition for Concep-\ntual Sentence Analysis . Ph.D. the-\nsis, University of Massachusetts,\nAmherst, MA. Available as CMP-\nSCI Technical Report 94-74.\nCardie, C. and K. Wagstaff. 1999.\nNoun phrase coreference as cluster-\ning. EMNLP/VLC .\nCarlini, N., F. Tramer, E. Wal-\nlace, M. Jagielski, A. Herbert-V oss,\nK. Lee, A. Roberts, T. Brown,\nD. Song, U. Erlingsson, et al. 2021.\nExtracting training data from largelanguage models. 30th USENIX Se-\ncurity Symposium (USENIX Security\n21).\nCarlson, G. N. 1977. Reference to kinds\nin English . Ph.D. thesis, Univer-\nsity of Massachusetts, Amherst. For-\nward.\nCarlson, L. and D. Marcu. 2001. Dis-\ncourse tagging manual. Technical\nReport ISI-TR-545, ISI.\nCarlson, L., D. Marcu, and M. E.\nOkurowski. 2001. Building a\ndiscourse-tagged corpus in the\nframework of rhetorical structure\ntheory. SIGDIAL .\nCarreras, X. and L. M `arquez. 2005.\nIntroduction to the CoNLL-2005\nshared task: Semantic role labeling.\nCoNLL .\nChafe, W. L. 1976. Givenness, con-\ntrastiveness, de\ufb01niteness, subjects,\ntopics, and point of view. In C. N. Li,\ned.,Subject and Topic , 25\u201355. Aca-\ndemic Press.\nChambers, N. 2013. NavyTime: Event\nand time ordering from raw text.\nSemEval-13 .\nChambers, N., T. Cassidy, B. McDow-\nell, and S. Bethard. 2014. Dense\nevent ordering with a multi-pass ar-\nchitecture. TACL , 2:273\u2013284.\nChambers, N. and D. Jurafsky. 2010.\nImproving the use of pseudo-words\nfor evaluating selectional prefer-\nences. ACL.\nChambers, N. and D. Jurafsky. 2011.\nTemplate-based information extrac-\ntion without the templates. ACL.\nChan, W., N. Jaitly, Q. Le, and\nO. Vinyals. 2016. Listen, at-\ntend and spell: A neural network\nfor large vocabulary conversational\nspeech recognition. ICASSP .\nChandioux, J. 1976. M \u00b4ET\u00b4EO: un\nsyst`eme op \u00b4erationnel pour la tra-\nduction automatique des bulletins\nm\u00b4et\u00b4eorologiques destin \u00b4es au grand\npublic. Meta , 21:127\u2013133.\nChang, A. X. and C. D. Manning. 2012.\nSUTime: A library for recognizing\nand normalizing time expressions.\nLREC .\nChang, K.-W., R. Samdani, and\nD. Roth. 2013. A constrained la-\ntent variable model for coreference\nresolution. EMNLP .\nChang, K.-W., R. Samdani, A. Ro-\nzovskaya, M. Sammons, and\nD. Roth. 2012. Illinois-Coref:\nThe UI system in the CoNLL-2012\nshared task. CoNLL .\nChaplot, D. S. and R. Salakhutdinov.\n2018. Knowledge-based word sense\ndisambiguation using topic models.\nAAAI . Bibliography 557\nCharniak, E. 1997. Statistical pars-\ning with a context-free grammar and\nword statistics. AAAI .\nCharniak, E., C. Hendrickson, N. Ja-\ncobson, and M. Perkowitz. 1993.\nEquations for part-of-speech tag-\nging. AAAI .\nChe, W., Z. Li, Y . Li, Y . Guo, B. Qin,\nand T. Liu. 2009. Multilingual\ndependency-based syntactic and se-\nmantic parsing. CoNLL .\nChen, C. and V . Ng. 2013. Linguis-\ntically aware coreference evaluation\nmetrics. IJCNLP .\nChen, D., A. Fisch, J. Weston, and\nA. Bordes. 2017a. Reading Wiki-\npedia to answer open-domain ques-\ntions. ACL.\nChen, D. and C. Manning. 2014. A fast\nand accurate dependency parser us-\ning neural networks. EMNLP .\nChen, E., B. Snyder, and R. Barzi-\nlay. 2007. Incremental text structur-\ning with online hierarchical ranking.\nEMNLP/CoNLL .\nChen, S. F. and J. Goodman. 1999.\nAn empirical study of smoothing\ntechniques for language modeling.\nComputer Speech and Language ,\n13:359\u2013394.\nChen, X., Z. Shi, X. Qiu, and X. Huang.\n2017b. Adversarial multi-criteria\nlearning for Chinese word segmen-\ntation. ACL.\nCheng, J., L. Dong, and M. La-\npata. 2016. Long short-term\nmemory-networks for machine read-\ning. EMNLP .\nCheng, M., E. Durmus, and D. Juraf-\nsky. 2023. Marked personas: Using\nnatural language prompts to mea-\nsure stereotypes in language models.\nACL.\nChiang, D. 2005. A hierarchical phrase-\nbased model for statistical machine\ntranslation. ACL.\nChinchor, N., L. Hirschman, and D. L.\nLewis. 1993. Evaluating Message\nUnderstanding systems: An analy-\nsis of the third Message Understand-\ning Conference. Computational Lin-\nguistics , 19(3):409\u2013449.\nChiticariu, L., M. Danilevsky, Y . Li,\nF. Reiss, and H. Zhu. 2018. Sys-\ntemT: Declarative text understand-\ning for enterprise. NAACL HLT , vol-\nume 3.\nChiticariu, L., Y . Li, and F. R. Reiss.\n2013. Rule-Based Information Ex-\ntraction is Dead! Long Live Rule-\nBased Information Extraction Sys-\ntems! EMNLP .\nChiu, J. P. C. and E. Nichols. 2016.\nNamed entity recognition with bidi-\nrectional LSTM-CNNs. TACL ,\n4:357\u2013370.Cho, K., B. van Merri \u00a8enboer, C. Gul-\ncehre, D. Bahdanau, F. Bougares,\nH. Schwenk, and Y . Bengio. 2014.\nLearning phrase representations us-\ning RNN encoder\u2013decoder for statis-\ntical machine translation. EMNLP .\nChoe, D. K. and E. Charniak. 2016.\nParsing as language modeling.\nEMNLP .\nChoi, J. D. and M. Palmer. 2011a. Get-\nting the most out of transition-based\ndependency parsing. ACL.\nChoi, J. D. and M. Palmer. 2011b.\nTransition-based semantic role la-\nbeling using predicate argument\nclustering. Proceedings of the ACL\n2011 Workshop on Relational Mod-\nels of Semantics .\nChoi, J. D., J. Tetreault, and A. Stent.\n2015. It depends: Dependency\nparser comparison using a web-\nbased evaluation tool. ACL.\nChomsky, N. 1956. Three models for\nthe description of language. IRE\nTransactions on Information The-\nory, 2(3):113\u2013124.\nChomsky, N. 1956/1975. The Logi-\ncal Structure of Linguistic Theory .\nPlenum.\nChomsky, N. 1957. Syntactic Struc-\ntures . Mouton.\nChomsky, N. 1963. Formal proper-\nties of grammars. In R. D. Luce,\nR. Bush, and E. Galanter, eds, Hand-\nbook of Mathematical Psychology ,\nvolume 2, 323\u2013418. Wiley.\nChomsky, N. 1981. Lectures on Gov-\nernment and Binding . Foris.\nChorowski, J., D. Bahdanau, K. Cho,\nand Y . Bengio. 2014. End-to-end\ncontinuous speech recognition using\nattention-based recurrent NN: First\nresults. NeurIPS Deep Learning and\nRepresentation Learning Workshop .\nChou, W., C.-H. Lee, and B. H. Juang.\n1993. Minimum error rate train-\ning based on n-best string models.\nICASSP .\nChristiano, P. F., J. Leike, T. Brown,\nM. Martic, S. Legg, and D. Amodei.\n2017. Deep reinforcement learning\nfrom human preferences. NeurIPS ,\nvolume 30.\nChristodoulopoulos, C., S. Goldwa-\nter, and M. Steedman. 2010. Two\ndecades of unsupervised POS in-\nduction: How far have we come?\nEMNLP .\nChu, Y .-J. and T.-H. Liu. 1965. On the\nshortest arborescence of a directed\ngraph. Science Sinica , 14:1396\u2013\n1400.\nChu-Carroll, J. 1998. A statistical\nmodel for discourse act recognition\nin dialogue interactions. ApplyingMachine Learning to Discourse Pro-\ncessing. Papers from the 1998 AAAI\nSpring Symposium. Tech. rep. SS-\n98-01. AAAI Press.\nChu-Carroll, J. and S. Carberry. 1998.\nCollaborative response generation in\nplanning dialogues. Computational\nLinguistics , 24(3):355\u2013400.\nChurch, K. W. 1988. A stochastic parts\nprogram and noun phrase parser for\nunrestricted text. ANLP .\nChurch, K. W. 1989. A stochastic parts\nprogram and noun phrase parser for\nunrestricted text. ICASSP .\nChurch, K. W. 1994. Unix for Poets.\nSlides from 2nd ELSNET Summer\nSchool and unpublished paper ms.\nChurch, K. W. and W. A. Gale. 1991. A\ncomparison of the enhanced Good-\nTuring and deleted estimation meth-\nods for estimating probabilities of\nEnglish bigrams. Computer Speech\nand Language , 5:19\u201354.\nChurch, K. W. and P. Hanks. 1989.\nWord association norms, mutual in-\nformation, and lexicography. ACL.\nChurch, K. W. and P. Hanks. 1990.\nWord association norms, mutual in-\nformation, and lexicography. Com-\nputational Linguistics , 16(1):22\u201329.\nCialdini, R. B. 1984. In\ufb02uence: The\npsychology of persuasion . Morrow.\nCieri, C., D. Miller, and K. Walker.\n2004. The Fisher corpus: A resource\nfor the next generations of speech-\nto-text. LREC .\nClark, E. 1987. The principle of con-\ntrast: A constraint on language ac-\nquisition. In B. MacWhinney, ed.,\nMechanisms of language acquisi-\ntion, 1\u201333. LEA.\nClark, H. H. 1996. Using Language .\nCambridge University Press.\nClark, H. H. and J. E. Fox Tree. 2002.\nUsing uh and um in spontaneous\nspeaking. Cognition , 84:73\u2013111.\nClark, H. H. and C. Marshall. 1981.\nDe\ufb01nite reference and mutual\nknowledge. In A. K. Joshi, B. L.\nWebber, and I. A. Sag, eds, Ele-\nments of Discourse Understanding ,\n10\u201363. Cambridge.\nClark, H. H. and D. Wilkes-Gibbs.\n1986. Referring as a collaborative\nprocess. Cognition , 22:1\u201339.\nClark, J. H., E. Choi, M. Collins,\nD. Garrette, T. Kwiatkowski,\nV . Nikolaev, and J. Palomaki.\n2020a. TyDi QA: A benchmark\nfor information-seeking question\nanswering in typologically diverse\nlanguages. TACL , 8:454\u2013470.\nClark, K., M.-T. Luong, Q. V . Le, and\nC. D. Manning. 2020b. Electra: Pre-\ntraining text encoders as discrimina-\ntors rather than generators. ICLR . 558 Bibliography\nClark, K. and C. D. Manning. 2015.\nEntity-centric coreference resolution\nwith model stacking. ACL.\nClark, K. and C. D. Manning. 2016a.\nDeep reinforcement learning for\nmention-ranking coreference mod-\nels.EMNLP .\nClark, K. and C. D. Manning. 2016b.\nImproving coreference resolution by\nlearning entity-level distributed rep-\nresentations. ACL.\nClark, S., J. R. Curran, and M. Osborne.\n2003. Bootstrapping POS-taggers\nusing unlabelled data. CoNLL .\nCobbe, K., V . Kosaraju, M. Bavar-\nian, M. Chen, H. Jun, L. Kaiser,\nM. Plappert, J. Tworek, J. Hilton,\nR. Nakano, C. Hesse, and J. Schul-\nman. 2021. Training veri\ufb01ers to\nsolve math word problems. ArXiv\npreprint.\nCoccaro, N. and D. Jurafsky. 1998. To-\nwards better integration of seman-\ntic predictors in statistical language\nmodeling. ICSLP .\nCoenen, A., E. Reif, A. Yuan, B. Kim,\nA. Pearce, F. Vi \u00b4egas, and M. Watten-\nberg. 2019. Visualizing and measur-\ning the geometry of bert. NeurIPS .\nCohen, A. D., A. Roberts, A. Molina,\nA. Butryna, A. Jin, A. Kulshreshtha,\nB. Hutchinson, B. Zevenbergen,\nB. H. Aguera-Arcas, C. ching\nChang, C. Cui, C. Du, D. D. F.\nAdiwardana, D. Chen, D. D. Lep-\nikhin, E. H. Chi, E. Hoffman-John,\nH.-T. Cheng, H. Lee, I. Krivokon,\nJ. Qin, J. Hall, J. Fenton, J. Soraker,\nK. Meier-Hellstern, K. Olson, L. M.\nAroyo, M. P. Bosma, M. J. Pickett,\nM. A. Menegali, M. Croak, M. D \u00b4\u0131az,\nM. Lamm, M. Krikun, M. R. Mor-\nris, N. Shazeer, Q. V . Le, R. Bern-\nstein, R. Rajakumar, R. Kurzweil,\nR. Thoppilan, S. Zheng, T. Bos,\nT. Duke, T. Doshi, V . Y . Zhao,\nV . Prabhakaran, W. Rusch, Y . Li,\nY . Huang, Y . Zhou, Y . Xu, and\nZ. Chen. 2022. Lamda: Lan-\nguage models for dialog applica-\ntions. ArXiv preprint.\nCohen, M. H., J. P. Giangola, and\nJ. Balogh. 2004. Voice User Inter-\nface Design . Addison-Wesley.\nCohen, P. R. and C. R. Perrault. 1979.\nElements of a plan-based theory of\nspeech acts. Cognitive Science ,\n3(3):177\u2013212.\nColby, K. M., S. Weber, and F. D. Hilf.\n1971. Arti\ufb01cial paranoia. Arti\ufb01cial\nIntelligence , 2(1):1\u201325.\nCole, R. A., D. G. Novick, P. J. E. Ver-\nmeulen, S. Sutton, M. Fanty, L. F. A.\nWessels, J. H. de Villiers, J. Schalk-\nwyk, B. Hansen, and D. Burnett.\n1997. Experiments with a spo-\nken dialogue system for taking theUS census. Speech Communication ,\n23:243\u2013260.\nCollins, M. 1999. Head-Driven Statis-\ntical Models for Natural Language\nParsing . Ph.D. thesis, University of\nPennsylvania, Philadelphia.\nCollobert, R. and J. Weston. 2007. Fast\nsemantic extraction using a novel\nneural network architecture. ACL.\nCollobert, R. and J. Weston. 2008.\nA uni\ufb01ed architecture for natural\nlanguage processing: Deep neural\nnetworks with multitask learning.\nICML .\nCollobert, R., J. Weston, L. Bottou,\nM. Karlen, K. Kavukcuoglu, and\nP. Kuksa. 2011. Natural language\nprocessing (almost) from scratch.\nJMLR , 12:2493\u20132537.\nComrie, B. 1989. Language Universals\nand Linguistic Typology , 2nd edi-\ntion. Blackwell.\nConneau, A., K. Khandelwal,\nN. Goyal, V . Chaudhary, G. Wen-\nzek, F. Guzm \u00b4an, E. Grave, M. Ott,\nL. Zettlemoyer, and V . Stoyanov.\n2020. Unsupervised cross-lingual\nrepresentation learning at scale.\nACL.\nConnolly, D., J. D. Burger, and D. S.\nDay. 1994. A machine learning ap-\nproach to anaphoric reference. Pro-\nceedings of the International Con-\nference on New Methods in Lan-\nguage Processing (NeMLaP) .\nCooley, J. W. and J. W. Tukey. 1965.\nAn algorithm for the machine cal-\nculation of complex Fourier se-\nries. Mathematics of Computation ,\n19(90):297\u2013301.\nCooper, F. S., A. M. Liberman, and\nJ. M. Borst. 1951. The interconver-\nsion of audible and visible patterns\nas a basis for research in the per-\nception of speech. Proceedings of\nthe National Academy of Sciences ,\n37(5):318\u2013325.\nCordier, B. 1965. Factor-analysis of\ncorrespondences. COLING 1965 .\nCosta-juss `a, M. R., J. Cross, O. C \u00b8 elebi,\nM. Elbayad, K. Hea\ufb01eld, K. Hef-\nfernan, E. Kalbassi, J. Lam,\nD. Licht, J. Maillard, A. Sun,\nS. Wang, G. Wenzek, A. Young-\nblood, B. Akula, L. Barrault,\nG. M. Gonzalez, P. Hansanti,\nJ. Hoffman, S. Jarrett, K. R.\nSadagopan, D. Rowe, S. Spruit,\nC. Tran, P. Andrews, N. F. Ayan,\nS. Bhosale, S. Edunov, A. Fan,\nC. Gao, V . Goswami, F. Guzm \u00b4an,\nP. Koehn, A. Mourachko, C. Ropers,\nS. Saleem, H. Schwenk, J. Wang,\nand NLLB Team. 2022. No lan-\nguage left behind: Scaling human-\ncentered machine translation. ArXiv.Cover, T. M. and J. A. Thomas. 1991.\nElements of Information Theory .\nWiley.\nCovington, M. 2001. A fundamen-\ntal algorithm for dependency pars-\ning. Proceedings of the 39th Annual\nACM Southeast Conference .\nCox, D. 1969. Analysis of Binary Data .\nChapman and Hall, London.\nCraven, M. and J. Kumlien. 1999.\nConstructing biological knowledge\nbases by extracting information\nfrom text sources. ISMB-99 .\nCrawford, K. 2017. The trouble with\nbias. Keynote at NeurIPS.\nCroft, W. 1990. Typology and Univer-\nsals. Cambridge University Press.\nCrosbie, J. and E. Shutova. 2022. In-\nduction heads as an essential mech-\nanism for pattern matching in in-\ncontext learning. ArXiv preprint.\nCross, J. and L. Huang. 2016. Span-\nbased constituency parsing with a\nstructure-label system and provably\noptimal dynamic oracles. EMNLP .\nCruse, D. A. 2004. Meaning in Lan-\nguage: an Introduction to Semantics\nand Pragmatics . Oxford University\nPress. Second edition.\nCucerzan, S. 2007. Large-scale\nnamed entity disambiguation based\non Wikipedia data. EMNLP/CoNLL .\nDagan, I., S. Marcus, and\nS. Markovitch. 1993. Contextual\nword similarity and estimation from\nsparse data. ACL.\nDahl, G. E., T. N. Sainath, and G. E.\nHinton. 2013. Improving deep\nneural networks for LVCSR using\nrecti\ufb01ed linear units and dropout.\nICASSP .\nDahl, G. E., D. Yu, L. Deng, and\nA. Acero. 2012. Context-dependent\npre-trained deep neural networks\nfor large-vocabulary speech recog-\nnition. IEEE Transactions on au-\ndio, speech, and language process-\ning, 20(1):30\u201342.\nDahl, M., V . Magesh, M. Suzgun, and\nD. E. Ho. 2024. Large legal \ufb01c-\ntions: Pro\ufb01ling legal hallucinations\nin large language models. Journal of\nLegal Analysis , 16:64\u201393.\nDai, A. M. and Q. V . Le. 2015.\nSemi-supervised sequence learning.\nNeurIPS .\nDanieli, M. and E. Gerbino. 1995. Met-\nrics for evaluating dialogue strate-\ngies in a spoken language system.\nAAAI Spring Symposium on Empir-\nical Methods in Discourse Interpre-\ntation and Generation .\nDas, S. R. and M. Y . Chen. 2001. Ya-\nhoo! for Amazon: Sentiment pars-\ning from small talk on the web. EFA\n2001 Barcelona Meetings. http://\nssrn.com/abstract=276189 . Bibliography 559\nDavid, Jr., E. E. and O. G. Selfridge.\n1962. Eyes and ears for computers.\nProceedings of the IRE (Institute of\nRadio Engineers) , 50:1093\u20131101.\nDavidson, T., D. Bhattacharya, and\nI. Weber. 2019. Racial bias in hate\nspeech and abusive language detec-\ntion datasets. Third Workshop on\nAbusive Language Online .\nDavies, M. 2012. Expanding hori-\nzons in historical linguistics with the\n400-million word Corpus of Histor-\nical American English. Corpora ,\n7(2):121\u2013157.\nDavies, M. 2015. The Wiki-\npedia Corpus: 4.6 million arti-\ncles, 1.9 billion words. Adapted\nfrom Wikipedia. https://www.\nenglish-corpora.org/wiki/ .\nDavies, M. 2020. The Corpus\nof Contemporary American En-\nglish (COCA): One billion words,\n1990-2019. https://www.\nenglish-corpora.org/coca/ .\nDavis, E., L. Morgenstern, and C. L.\nOrtiz. 2017. The \ufb01rst Winograd\nschema challenge at IJCAI-16. AI\nMagazine , 38(3):97\u201398.\nDavis, K. H., R. Biddulph, and S. Bal-\nashek. 1952. Automatic recognition\nof spoken digits. JASA , 24(6):637\u2013\n642.\nDavis, S. and P. Mermelstein. 1980.\nComparison of parametric repre-\nsentations for monosyllabic word\nrecognition in continuously spoken\nsentences. IEEE Transactions on\nASSP , 28(4):357\u2013366.\nDeerwester, S. C., S. T. Dumais, G. W.\nFurnas, R. A. Harshman, T. K.\nLandauer, K. E. Lochbaum, and\nL. Streeter. 1988. Computer infor-\nmation retrieval using latent seman-\ntic structure: US Patent 4,839,853.\nDeerwester, S. C., S. T. Dumais, T. K.\nLandauer, G. W. Furnas, and R. A.\nHarshman. 1990. Indexing by la-\ntent semantics analysis. JASIS ,\n41(6):391\u2013407.\nDeibel, D. and R. Evanhoe. 2021. Con-\nversations with Things: UX Design\nfor Chat and Voice . Rosenfeld.\nDeJong, G. F. 1982. An overview of the\nFRUMP system. In W. G. Lehnert\nand M. H. Ringle, eds, Strategies for\nNatural Language Processing , 149\u2013\n176. LEA.\nDemberg, V . 2006. Letter-to-phoneme\nconversion for a German text-to-\nspeech system. Diplomarbeit Nr. 47,\nUniversit \u00a8at Stuttgart.\nDenes, P. 1959. The design and oper-\nation of the mechanical speech rec-\nognizer at University College Lon-\ndon. Journal of the British Institu-\ntion of Radio Engineers , 19(4):219\u2013\n234. Appears together with compan-\nion paper (Fry 1959).Deng, L., G. Hinton, and B. Kingsbury.\n2013. New types of deep neural\nnetwork learning for speech recog-\nnition and related applications: An\noverview. ICASSP .\nDeng, Y . and W. Byrne. 2005. HMM\nword and phrase alignment for sta-\ntistical machine translation. HLT-\nEMNLP .\nDenis, P. and J. Baldridge. 2007. Joint\ndetermination of anaphoricity and\ncoreference resolution using integer\nprogramming. NAACL-HLT .\nDenis, P. and J. Baldridge. 2008. Spe-\ncialized models and ranking for\ncoreference resolution. EMNLP .\nDenis, P. and J. Baldridge. 2009. Global\njoint models for coreference resolu-\ntion and named entity classi\ufb01cation.\nProcesamiento del Lenguaje Natu-\nral, 42.\nDeRose, S. J. 1988. Grammatical cat-\negory disambiguation by statistical\noptimization. Computational Lin-\nguistics , 14:31\u201339.\nDevlin, J., M.-W. Chang, K. Lee, and\nK. Toutanova. 2019. BERT: Pre-\ntraining of deep bidirectional trans-\nformers for language understanding.\nNAACL HLT .\nDi Eugenio, B. 1990. Centering theory\nand the Italian pronominal system.\nCOLING .\nDi Eugenio, B. 1996. The discourse\nfunctions of Italian subjects: A cen-\ntering approach. COLING .\nDias Oliva, T., D. Antonialli, and\nA. Gomes. 2021. Fighting hate\nspeech, silencing drag queens? arti-\n\ufb01cial intelligence in content modera-\ntion and risks to lgbtq voices online.\nSexuality & Culture , 25:700\u2013732.\nDinan, E., G. Abercrombie, A. S.\nBergman, S. Spruit, D. Hovy, Y .-L.\nBoureau, and V . Rieser. 2021. Antic-\nipating safety issues in e2e conver-\nsational ai: Framework and tooling.\nArXiv.\nDinan, E., A. Fan, A. Williams, J. Ur-\nbanek, D. Kiela, and J. Weston.\n2020. Queens are powerful too: Mit-\nigating gender bias in dialogue gen-\neration. EMNLP .\nDitman, T. and G. R. Kuperberg.\n2010. Building coherence: A frame-\nwork for exploring the breakdown\nof links across clause boundaries in\nschizophrenia. Journal of neurolin-\nguistics , 23(3):254\u2013269.\nDixon, L., J. Li, J. Sorensen, N. Thain,\nand L. Vasserman. 2018. Measuring\nand mitigating unintended bias in\ntext classi\ufb01cation. 2018 AAAI/ACM\nConference on AI, Ethics, and Soci-\nety.Dixon, N. and H. Maxey. 1968. Termi-\nnal analog synthesis of continuous\nspeech using the diphone method of\nsegment assembly. IEEE Transac-\ntions on Audio and Electroacoustics ,\n16(1):40\u201350.\nDo, Q. N. T., S. Bethard, and M.-F.\nMoens. 2017. Improving implicit\nsemantic role labeling by predicting\nsemantic frame arguments. IJCNLP .\nDoddington, G. 2002. Automatic eval-\nuation of machine translation quality\nusing n-gram co-occurrence statis-\ntics. HLT.\nDodge, J., S. Gururangan, D. Card,\nR. Schwartz, and N. A. Smith. 2019.\nShow your work: Improved report-\ning of experimental results. EMNLP .\nDodge, J., M. Sap, A. Marasovi \u00b4c,\nW. Agnew, G. Ilharco, D. Groen-\neveld, M. Mitchell, and M. Gardner.\n2021. Documenting large webtext\ncorpora: A case study on the colos-\nsal clean crawled corpus. EMNLP .\nDong, L. and M. Lapata. 2016. Lan-\nguage to logical form with neural at-\ntention. ACL.\nDorr, B. 1994. Machine translation di-\nvergences: A formal description and\nproposed solution. Computational\nLinguistics , 20(4):597\u2013633.\nDostert, L. 1955. The Georgetown-\nI.B.M. experiment. In Machine\nTranslation of Languages: Fourteen\nEssays , 124\u2013135. MIT Press.\nDowty, D. R. 1979. Word Meaning and\nMontague Grammar . D. Reidel.\nDozat, T. and C. D. Manning. 2017.\nDeep biaf\ufb01ne attention for neural de-\npendency parsing. ICLR .\nDozat, T. and C. D. Manning. 2018.\nSimpler but more accurate semantic\ndependency parsing. ACL.\nDozat, T., P. Qi, and C. D. Manning.\n2017. Stanford\u2019s graph-based neu-\nral dependency parser at the CoNLL\n2017 shared task. Proceedings of the\nCoNLL 2017 Shared Task: Multilin-\ngual Parsing from Raw Text to Uni-\nversal Dependencies .\nDror, R., G. Baumer, M. Bogomolov,\nand R. Reichart. 2017. Replicabil-\nity analysis for natural language pro-\ncessing: Testing signi\ufb01cance with\nmultiple datasets. TACL , 5:471\u2013\n\u2013486.\nDror, R., L. Peled-Cohen, S. Shlomov,\nand R. Reichart. 2020. Statisti-\ncal Signi\ufb01cance Testing for Natural\nLanguage Processing , volume 45 of\nSynthesis Lectures on Human Lan-\nguage Technologies . Morgan &\nClaypool.\nDryer, M. S. and M. Haspelmath, eds.\n2013. The World Atlas of Language 560 Bibliography\nStructures Online . Max Planck In-\nstitute for Evolutionary Anthropol-\nogy, Leipzig. Available online at\nhttp://wals.info .\nDu Bois, J. W., W. L. Chafe, C. Meyer,\nS. A. Thompson, R. Englebretson,\nand N. Martey. 2005. Santa Barbara\ncorpus of spoken American English,\nParts 1-4. Philadelphia: Linguistic\nData Consortium.\nDurrett, G. and D. Klein. 2013. Easy\nvictories and uphill battles in coref-\nerence resolution. EMNLP .\nDurrett, G. and D. Klein. 2014. A joint\nmodel for entity analysis: Corefer-\nence, typing, and linking. TACL ,\n2:477\u2013490.\nEarley, J. 1968. An Ef\ufb01cient Context-\nFree Parsing Algorithm . Ph.D.\nthesis, Carnegie Mellon University,\nPittsburgh, PA.\nEarley, J. 1970. An ef\ufb01cient context-\nfree parsing algorithm. CACM ,\n6(8):451\u2013455.\nEbden, P. and R. Sproat. 2015. The\nKestrel TTS text normalization sys-\ntem. Natural Language Engineer-\ning, 21(3):333.\nEdmonds, J. 1967. Optimum branch-\nings. Journal of Research of the\nNational Bureau of Standards B ,\n71(4):233\u2013240.\nEdunov, S., M. Ott, M. Auli, and\nD. Grangier. 2018. Understanding\nback-translation at scale. EMNLP .\nEfron, B. and R. J. Tibshirani. 1993. An\nintroduction to the bootstrap . CRC\npress.\nEgghe, L. 2007. Untangling Herdan\u2019s\nlaw and Heaps\u2019 law: Mathematical\nand informetric arguments. JASIST ,\n58(5):702\u2013709.\nEisner, J. 1996. Three new probabilistic\nmodels for dependency parsing: An\nexploration. COLING .\nEkman, P. 1999. Basic emotions. In\nT. Dalgleish and M. J. Power, eds,\nHandbook of Cognition and Emo-\ntion, 45\u201360. Wiley.\nElhage, N., N. Nanda, C. Olsson,\nT. Henighan, N. Joseph, B. Mann,\nA. Askell, Y . Bai, A. Chen, T. Con-\nerly, N. DasSarma, D. Drain,\nD. Ganguli, Z. Hat\ufb01eld-Dodds,\nD. Hernandez, A. Jones, J. Kernion,\nL. Lovitt, K. Ndousse, D. Amodei,\nT. Brown, J. Clark, J. Kaplan, S. Mc-\nCandlish, and C. Olah. 2021. A\nmathematical framework for trans-\nformer circuits. White paper.\nElman, J. L. 1990. Finding structure in\ntime. Cognitive science , 14(2):179\u2013\n211.\nElsner, M., J. Austerweil, and E. Char-\nniak. 2007. A uni\ufb01ed local and\nglobal model for discourse coher-\nence. NAACL-HLT .Elsner, M. and E. Charniak. 2008.\nCoreference-inspired coherence\nmodeling. ACL.\nElsner, M. and E. Charniak. 2011. Ex-\ntending the entity grid with entity-\nspeci\ufb01c features. ACL.\nElvev \u02daag, B., P. W. Foltz, D. R.\nWeinberger, and T. E. Goldberg.\n2007. Quantifying incoherence in\nspeech: an automated methodology\nand novel application to schizophre-\nnia. Schizophrenia research , 93(1-\n3):304\u2013316.\nEmami, A. and F. Jelinek. 2005. A neu-\nral syntactic language model. Ma-\nchine learning , 60(1):195\u2013227.\nEmami, A., P. Trichelair, A. Trischler,\nK. Suleman, H. Schulz, and J. C. K.\nCheung. 2019. The KNOWREF\ncoreference corpus: Removing gen-\nder and number cues for dif\ufb01-\ncult pronominal anaphora resolu-\ntion. ACL.\nErk, K. 2007. A simple, similarity-\nbased model for selectional prefer-\nences. ACL.\nvan Esch, D. and R. Sproat. 2018.\nAn expanded taxonomy of semiotic\nclasses for text normalization. IN-\nTERSPEECH .\nEthayarajh, K. 2019. How contextual\nare contextualized word representa-\ntions? Comparing the geometry of\nBERT, ELMo, and GPT-2 embed-\ndings. EMNLP .\nEthayarajh, K., D. Duvenaud, and\nG. Hirst. 2019a. Towards un-\nderstanding linear word analogies.\nACL.\nEthayarajh, K., D. Duvenaud, and\nG. Hirst. 2019b. Understanding un-\ndesirable word embedding associa-\ntions. ACL.\nEthayarajh, K. and D. Jurafsky. 2020.\nUtility is in the eye of the user:\nA critique of NLP leaderboards.\nEMNLP .\nEtzioni, O., M. Cafarella, D. Downey,\nA.-M. Popescu, T. Shaked, S. Soder-\nland, D. S. Weld, and A. Yates.\n2005. Unsupervised named-entity\nextraction from the web: An experi-\nmental study. Arti\ufb01cial Intelligence ,\n165(1):91\u2013134.\nEvans, N. 2000. Word classes in the\nworld\u2019s languages. In G. Booij,\nC. Lehmann, and J. Mugdan, eds,\nMorphology: A Handbook on In\ufb02ec-\ntion and Word Formation , 708\u2013732.\nMouton.\nFader, A., S. Soderland, and O. Etzioni.\n2011. Identifying relations for open\ninformation extraction. EMNLP .\nFan, A., S. Bhosale, H. Schwenk,\nZ. Ma, A. El-Kishky, S. Goyal,\nM. Baines, O. Celebi, G. Wenzek,V . Chaudhary, N. Goyal, T. Birch,\nV . Liptchinsky, S. Edunov, M. Auli,\nand A. Joulin. 2021. Beyond\nenglish-centric multilingual ma-\nchine translation. JMLR , 22(107):1\u2013\n48.\nFano, R. M. 1961. Transmission of In-\nformation: A Statistical Theory of\nCommunications . MIT Press.\nFant, G. M. 1951. Speech communica-\ntion research. Ing. Vetenskaps Akad.\nStockholm, Sweden , 24:331\u2013337.\nFant, G. M. 1986. Glottal \ufb02ow: Models\nand interaction. Journal of Phonet-\nics, 14:393\u2013399.\nFast, E., B. Chen, and M. S. Bernstein.\n2016. Empath: Understanding Topic\nSignals in Large-Scale Text. CHI.\nFauconnier, G. and M. Turner. 2008.\nThe way we think: Conceptual\nblending and the mind\u2019s hidden\ncomplexities . Basic Books.\nFeldman, J. A. and D. H. Ballard.\n1982. Connectionist models and\ntheir properties. Cognitive Science ,\n6:205\u2013254.\nFellbaum, C., ed. 1998. WordNet: An\nElectronic Lexical Database . MIT\nPress.\nFeng, V . W. and G. Hirst. 2011. Classi-\nfying arguments by scheme. ACL.\nFeng, V . W. and G. Hirst. 2014.\nA linear-time bottom-up discourse\nparser with constraints and post-\nediting. ACL.\nFeng, V . W., Z. Lin, and G. Hirst. 2014.\nThe impact of deep hierarchical dis-\ncourse structures in the evaluation of\ntext coherence. COLING .\nFernandes, E. R., C. N. dos Santos, and\nR. L. Milidi \u00b4u. 2012. Latent struc-\nture perceptron with feature induc-\ntion for unrestricted coreference res-\nolution. CoNLL .\nFerragina, P. and U. Scaiella. 2011.\nFast and accurate annotation of short\ntexts with wikipedia pages. IEEE\nSoftware , 29(1):70\u201375.\nFerro, L., L. Gerber, I. Mani, B. Sund-\nheim, and G. Wilson. 2005. Tides\n2005 standard for the annotation of\ntemporal expressions. Technical re-\nport, MITRE.\nFerrucci, D. A. 2012. Introduction\nto \u201cThis is Watson\u201d. IBM Jour-\nnal of Research and Development ,\n56(3/4):1:1\u20131:15.\nFessler, L. 2017. We tested bots like Siri\nand Alexa to see who would stand\nup to sexual harassment. Quartz .\nFeb 22, 2017. https://qz.com/\n911681/ .\nField, A. and Y . Tsvetkov. 2019. Entity-\ncentric contextual affective analysis.\nACL. Bibliography 561\nFikes, R. E. and N. J. Nilsson. 1971.\nSTRIPS: A new approach to the\napplication of theorem proving to\nproblem solving. Arti\ufb01cial Intelli-\ngence , 2:189\u2013208.\nFillmore, C. J. 1966. A proposal con-\ncerning English prepositions. In F. P.\nDinneen, ed., 17th annual Round Ta-\nble, volume 17 of Monograph Series\non Language and Linguistics , 19\u2013\n34. Georgetown University Press.\nFillmore, C. J. 1968. The case for case.\nIn E. W. Bach and R. T. Harms, eds,\nUniversals in Linguistic Theory , 1\u2013\n88. Holt, Rinehart & Winston.\nFillmore, C. J. 1985. Frames and the se-\nmantics of understanding. Quaderni\ndi Semantica , VI(2):222\u2013254.\nFillmore, C. J. 2003. Valency and se-\nmantic roles: the concept of deep\nstructure case. In V . Agel, L. M.\nEichinger, H. W. Eroms, P. Hell-\nwig, H. J. Heringer, and H. Lobin,\neds, Dependenz und Valenz: Ein\ninternationales Handbuch der zeit-\ngen\u00a8ossischen Forschung , chapter 36,\n457\u2013475. Walter de Gruyter.\nFillmore, C. J. 2012. ACL life-\ntime achievement award: Encoun-\nters with language. Computational\nLinguistics , 38(4):701\u2013718.\nFillmore, C. J. and C. F. Baker. 2009. A\nframes approach to semantic analy-\nsis. In B. Heine and H. Narrog, eds,\nThe Oxford Handbook of Linguistic\nAnalysis , 313\u2013340. Oxford Univer-\nsity Press.\nFillmore, C. J., C. R. Johnson, and\nM. R. L. Petruck. 2003. Background\nto FrameNet. International journal\nof lexicography , 16(3):235\u2013250.\nFinkelstein, L., E. Gabrilovich, Y . Ma-\ntias, E. Rivlin, Z. Solan, G. Wolf-\nman, and E. Ruppin. 2002. Placing\nsearch in context: The concept revis-\nited. ACM Transactions on Informa-\ntion Systems , 20(1):116\u2014-131.\nFinlayson, M. A. 2016. Inferring\nPropp\u2019s functions from semantically\nannotated text. The Journal of Amer-\nican Folklore , 129(511):55\u201377.\nFirth, J. R. 1935. The technique of se-\nmantics. Transactions of the philo-\nlogical society , 34(1):36\u201373.\nFirth, J. R. 1957. A synopsis of linguis-\ntic theory 1930\u20131955. In Studies in\nLinguistic Analysis . Philological So-\nciety. Reprinted in Palmer, F. (ed.)\n1968. Selected Papers of J. R. Firth.\nLongman, Harlow.\nFlanagan, J. L. 1972. Speech Analysis,\nSynthesis, and Perception . Springer.\nFlanagan, J. L., K. Ishizaka, and K. L.\nShipley. 1975. Synthesis of speech\nfrom a dynamic model of the vocal\ncords and vocal tract. The Bell Sys-\ntem Technical Journal , 54(3):485\u2013\n506.Foland, W. and J. H. Martin. 2016.\nCU-NLP at SemEval-2016 task 8:\nAMR parsing using LSTM-based re-\ncurrent neural networks. SemEval-\n2016 .\nFoland, Jr., W. R. and J. H. Martin.\n2015. Dependency-based seman-\ntic role labeling using convolutional\nneural networks. *SEM 2015 .\nFoltz, P. W., W. Kintsch, and T. K. Lan-\ndauer. 1998. The measurement of\ntextual coherence with latent seman-\ntic analysis. Discourse processes ,\n25(2-3):285\u2013307.\n8, W. Nekoto, V . Marivate, T. Matsila,\nT. Fasubaa, T. Kolawole, T. Fag-\nbohungbe, S. O. Akinola, S. H.\nMuhammad, S. Kabongo, S. Osei,\nS. Freshia, R. A. Niyongabo,\nR. M. P. Ogayo, O. Ahia, M. Mer-\nessa, M. Adeyemi, M. Mokgesi-\nSelinga, L. Okegbemi, L. J. Mar-\ntinus, K. Tajudeen, K. Degila,\nK. Ogueji, K. Siminyu, J. Kreutzer,\nJ. Webster, J. T. Ali, J. A. I.\nOrife, I. Ezeani, I. A. Dangana,\nH. Kamper, H. Elsahar, G. Duru,\nG. Kioko, E. Murhabazi, E. van\nBiljon, D. Whitenack, C. Onye-\nfuluchi, C. Emezue, B. Dossou,\nB. Sibanda, B. I. Bassey, A. Olabiyi,\nA. Ramkilowan, A. \u00a8Oktem, A. Akin-\nfaderin, and A. Bashir. 2020. Partic-\nipatory research for low-resourced\nmachine translation: A case study\nin African languages. Findings of\nEMNLP .\nFox, B. A. 1993. Discourse Structure\nand Anaphora: Written and Conver-\nsational English . Cambridge.\nFrancis, W. N. and H. Ku \u02c7cera. 1982.\nFrequency Analysis of English Us-\nage. Houghton Mif\ufb02in, Boston.\nFranz, A. and T. Brants. 2006. All our\nn-gram are belong to you. https:\n//research.google/blog/\nall-our-n-gram-are-belong-to-you/ .\nFraser, N. M. and G. N. Gilbert. 1991.\nSimulating speech systems. Com-\nputer Speech and Language , 5:81\u2013\n99.\nFriedman, B. and D. G. Hendry. 2019.\nValue Sensitive Design: Shaping\nTechnology with Moral Imagination .\nMIT Press.\nFriedman, B., D. G. Hendry, and\nA. Borning. 2017. A survey\nof value sensitive design methods.\nFoundations and Trends in Human-\nComputer Interaction , 11(2):63\u2013\n125.\nFry, D. B. 1959. Theoretical as-\npects of mechanical speech recogni-\ntion. Journal of the British Institu-\ntion of Radio Engineers , 19(4):211\u2013\n218. Appears together with compan-\nion paper (Denes 1959).Furnas, G. W., T. K. Landauer, L. M.\nGomez, and S. T. Dumais. 1987.\nThe vocabulary problem in human-\nsystem communication. Commu-\nnications of the ACM , 30(11):964\u2013\n971.\nGabow, H. N., Z. Galil, T. Spencer, and\nR. E. Tarjan. 1986. Ef\ufb01cient algo-\nrithms for \ufb01nding minimum span-\nning trees in undirected and directed\ngraphs. Combinatorica , 6(2):109\u2013\n122.\nGaddy, D., M. Stern, and D. Klein.\n2018. What\u2019s going on in neural\nconstituency parsers? an analysis.\nNAACL HLT .\nGale, W. A. and K. W. Church. 1994.\nWhat is wrong with adding one? In\nN. Oostdijk and P. de Haan, eds,\nCorpus-Based Research into Lan-\nguage , 189\u2013198. Rodopi.\nGale, W. A. and K. W. Church. 1991.\nA program for aligning sentences in\nbilingual corpora. ACL.\nGale, W. A. and K. W. Church. 1993.\nA program for aligning sentences in\nbilingual corpora. Computational\nLinguistics , 19:75\u2013102.\nGale, W. A., K. W. Church, and\nD. Yarowsky. 1992a. One sense per\ndiscourse. HLT.\nGale, W. A., K. W. Church, and\nD. Yarowsky. 1992b. Work on sta-\ntistical methods for word sense dis-\nambiguation. AAAI Fall Symposium\non Probabilistic Approaches to Nat-\nural Language .\nGao, L., T. Hoppe, A. Thite, S. Bi-\nderman, C. Foster, N. Nabeshima,\nS. Black, J. Phang, S. Presser,\nL. Golding, H. He, and C. Leahy.\n2020. The Pile: An 800GB dataset\nof diverse text for language model-\ning. ArXiv preprint.\nGarg, N., L. Schiebinger, D. Jurafsky,\nand J. Zou. 2018. Word embeddings\nquantify 100 years of gender and\nethnic stereotypes. Proceedings of\nthe National Academy of Sciences ,\n115(16):E3635\u2013E3644.\nGarside, R. 1987. The CLAWS word-\ntagging system. In R. Garside,\nG. Leech, and G. Sampson, eds, The\nComputational Analysis of English ,\n30\u201341. Longman.\nGarside, R., G. Leech, and A. McEnery.\n1997. Corpus Annotation . Long-\nman.\nGebru, T., J. Morgenstern, B. Vec-\nchione, J. W. Vaughan, H. Wal-\nlach, H. Daum \u00b4e III, and K. Craw-\nford. 2020. Datasheets for datasets.\nArXiv.\nGehman, S., S. Gururangan, M. Sap,\nY . Choi, and N. A. Smith. 2020. Re-\nalToxicityPrompts: Evaluating neu-\nral toxic degeneration in language\nmodels. Findings of EMNLP . 562 Bibliography\nGerber, M. and J. Y . Chai. 2010. Be-\nyond nombank: A study of implicit\narguments for nominal predicates.\nACL.\nGers, F. A., J. Schmidhuber, and\nF. Cummins. 2000. Learning to for-\nget: Continual prediction with lstm.\nNeural computation , 12(10):2451\u2013\n2471.\nGil, D. 2000. Syntactic categories,\ncross-linguistic variation and univer-\nsal grammar. In P. M. V ogel and\nB. Comrie, eds, Approaches to the\nTypology of Word Classes , 173\u2013216.\nMouton.\nGildea, D. and D. Jurafsky. 2000. Au-\ntomatic labeling of semantic roles.\nACL.\nGildea, D. and D. Jurafsky. 2002.\nAutomatic labeling of semantic\nroles. Computational Linguistics ,\n28(3):245\u2013288.\nGildea, D. and M. Palmer. 2002.\nThe necessity of syntactic parsing\nfor predicate argument recognition.\nACL.\nGiles, C. L., G. M. Kuhn, and R. J.\nWilliams. 1994. Dynamic recurrent\nneural networks: Theory and appli-\ncations. IEEE Trans. Neural Netw.\nLearning Syst. , 5(2):153\u2013156.\nGillick, L. and S. J. Cox. 1989. Some\nstatistical issues in the comparison\nof speech recognition algorithms.\nICASSP .\nGirard, G. 1718. La justesse de la\nlangue franc \u00b8oise: ou les diff \u00b4erentes\nsigni\ufb01cations des mots qui passent\npour synonimes . Laurent d\u2019Houry,\nParis.\nGiuliano, V . E. 1965. The inter-\npretation of word associations.\nStatistical Association Methods\nFor Mechanized Documentation.\nSymposium Proceedings. Wash-\nington, D.C., USA, March 17,\n1964 .https://nvlpubs.nist.\ngov/nistpubs/Legacy/MP/\nnbsmiscellaneouspub269.pdf .\nGladkova, A., A. Drozd, and S. Mat-\nsuoka. 2016. Analogy-based de-\ntection of morphological and se-\nmantic relations with word embed-\ndings: what works and what doesn\u2019t.\nNAACL Student Research Workshop .\nGlaese, A., N. McAleese, M. Trebacz,\nJ. Aslanides, V . Firoiu, T. Ewalds,\nM. Rauh, L. Weidinger, M. Chad-\nwick, P. Thacker, L. Campbell-\nGillingham, J. Uesato, P.-S. Huang,\nR. Comanescu, F. Yang, A. See,\nS. Dathathri, R. Greig, C. Chen,\nD. Fritz, J. Sanchez Elias, R. Green,\nS. Mokr \u00b4a, N. Fernando, B. Wu,\nR. Foley, S. Young, I. Gabriel,\nW. Isaac, J. Mellor, D. Hassabis,\nK. Kavukcuoglu, L. A. Hendricks,and G. Irving. 2022. Improving\nalignment of dialogue agents via tar-\ngeted human judgements. ArXiv\npreprint.\nGlenberg, A. M. and D. A. Robert-\nson. 2000. Symbol grounding and\nmeaning: A comparison of high-\ndimensional and embodied theories\nof meaning. Journal of memory and\nlanguage , 43(3):379\u2013401.\nGodfrey, J., E. Holliman, and J. Mc-\nDaniel. 1992. SWITCHBOARD:\nTelephone speech corpus for re-\nsearch and development. ICASSP .\nGoel, V . and W. Byrne. 2000. Minimum\nbayes-risk automatic speech recog-\nnition. Computer Speech & Lan-\nguage , 14(2):115\u2013135.\nGoffman, E. 1974. Frame analysis: An\nessay on the organization of experi-\nence. Harvard University Press.\nGoldberg, J., M. Ostendorf, and\nK. Kirchhoff. 2003. The impact of\nresponse wording in error correction\nsubdialogs. ISCA Tutorial and Re-\nsearch Workshop on Error Handling\nin Spoken Dialogue Systems .\nGoldberg, Y . 2017. Neural Network\nMethods for Natural Language Pro-\ncessing , volume 10 of Synthesis Lec-\ntures on Human Language Tech-\nnologies . Morgan & Claypool.\nGonen, H. and Y . Goldberg. 2019. Lip-\nstick on a pig: Debiasing methods\ncover up systematic gender biases in\nword embeddings but do not remove\nthem. NAACL HLT .\nGood, M. D., J. A. Whiteside, D. R.\nWixon, and S. J. Jones. 1984. Build-\ning a user-derived interface. CACM ,\n27(10):1032\u20131043.\nGoodfellow, I., Y . Bengio, and\nA. Courville. 2016. Deep Learn-\ning. MIT Press.\nGoodman, J. 2006. A bit of progress\nin language modeling: Extended\nversion. Technical Report MSR-\nTR-2001-72, Machine Learning and\nApplied Statistics Group, Microsoft\nResearch, Redmond, WA.\nGoodwin, C. 1996. Transparent vi-\nsion. In E. Ochs, E. A. Schegloff,\nand S. A. Thompson, eds, Interac-\ntion and Grammar , 370\u2013404. Cam-\nbridge University Press.\nGopalakrishnan, K., B. Hedayatnia,\nQ. Chen, A. Gottardi, S. Kwa-\ntra, A. Venkatesh, R. Gabriel, and\nD. Hakkani-T \u00a8ur. 2019. Topical-\nchat: Towards knowledge-grounded\nopen-domain conversations. INTER-\nSPEECH .\nGould, J. D., J. Conti, and T. Ho-\nvanyecz. 1983. Composing let-\nters with a simulated listening type-\nwriter. CACM , 26(4):295\u2013308.Gould, J. D. and C. Lewis. 1985. De-\nsigning for usability: Key principles\nand what designers think. CACM ,\n28(3):300\u2013311.\nGould, S. J. 1980. The Panda\u2019s Thumb .\nPenguin Group.\nGraff, D. 1997. The 1996 Broadcast\nNews speech and language-model\ncorpus. Proceedings DARPA Speech\nRecognition Workshop .\nGravano, A., J. Hirschberg, and\n\u02c7S. Be \u02c7nu\u02c7s. 2012. Af\ufb01rmative cue\nwords in task-oriented dialogue.\nComputational Linguistics , 38(1):1\u2013\n39.\nGraves, A. 2012. Sequence transduc-\ntion with recurrent neural networks.\nICASSP .\nGraves, A. 2013. Generating se-\nquences with recurrent neural net-\nworks. ArXiv.\nGraves, A., S. Fern \u00b4andez, F. Gomez,\nand J. Schmidhuber. 2006. Con-\nnectionist temporal classi\ufb01cation:\nLabelling unsegmented sequence\ndata with recurrent neural networks.\nICML .\nGraves, A., S. Fern \u00b4andez, M. Li-\nwicki, H. Bunke, and J. Schmidhu-\nber. 2007. Unconstrained on-line\nhandwriting recognition with recur-\nrent neural networks. NeurIPS .\nGraves, A. and N. Jaitly. 2014. Towards\nend-to-end speech recognition with\nrecurrent neural networks. ICML .\nGraves, A., A.-r. Mohamed, and\nG. Hinton. 2013. Speech recognition\nwith deep recurrent neural networks.\nICASSP .\nGraves, A. and J. Schmidhuber. 2005.\nFramewise phoneme classi\ufb01cation\nwith bidirectional LSTM and other\nneural network architectures. Neu-\nral Networks , 18(5-6):602\u2013610.\nGraves, A., G. Wayne, and I. Dani-\nhelka. 2014. Neural Turing ma-\nchines. ArXiv.\nGreen, B. F., A. K. Wolf, C. Chom-\nsky, and K. Laughery. 1961. Base-\nball: An automatic question an-\nswerer. Proceedings of the Western\nJoint Computer Conference 19 .\nGreene, B. B. and G. M. Rubin. 1971.\nAutomatic grammatical tagging of\nEnglish. Department of Linguis-\ntics, Brown University, Providence,\nRhode Island.\nGreenwald, A. G., D. E. McGhee, and\nJ. L. K. Schwartz. 1998. Measur-\ning individual differences in implicit\ncognition: the implicit association\ntest. Journal of personality and so-\ncial psychology , 74(6):1464\u20131480.\nGrenager, T. and C. D. Manning. 2006.\nUnsupervised discovery of a statisti-\ncal verb lexicon. EMNLP . Bibliography 563\nGrice, H. P. 1975. Logic and conversa-\ntion. In P. Cole and J. L. Morgan,\neds, Speech Acts: Syntax and Se-\nmantics Volume 3 , 41\u201358. Academic\nPress.\nGrice, H. P. 1978. Further notes on\nlogic and conversation. In P. Cole,\ned.,Pragmatics: Syntax and Seman-\ntics Volume 9 , 113\u2013127. Academic\nPress.\nGrishman, R. and B. Sundheim. 1995.\nDesign of the MUC-6 evaluation.\nMUC-6 .\nGrosz, B. J. 1977a. The representation\nand use of focus in a system for un-\nderstanding dialogs. IJCAI-77 . Mor-\ngan Kaufmann.\nGrosz, B. J. 1977b. The Representation\nand Use of Focus in Dialogue Un-\nderstanding . Ph.D. thesis, Univer-\nsity of California, Berkeley.\nGrosz, B. J., A. K. Joshi, and S. Wein-\nstein. 1983. Providing a uni\ufb01ed ac-\ncount of de\ufb01nite noun phrases in En-\nglish. ACL.\nGrosz, B. J., A. K. Joshi, and S. Wein-\nstein. 1995. Centering: A framework\nfor modeling the local coherence of\ndiscourse. Computational Linguis-\ntics, 21(2):203\u2013225.\nGrosz, B. J. and C. L. Sidner. 1980.\nPlans for discourse. In P. R. Cohen,\nJ. Morgan, and M. E. Pollack, eds,\nIntentions in Communication , 417\u2013\n444. MIT Press.\nGruber, J. S. 1965. Studies in Lexical\nRelations . Ph.D. thesis, MIT.\nGr\u00a8unewald, S., A. Friedrich, and\nJ. Kuhn. 2021. Applying Occam\u2019s\nrazor to transformer-based depen-\ndency parsing: What works, what\ndoesn\u2019t, and what is really neces-\nsary. IWPT .\nGuinaudeau, C. and M. Strube. 2013.\nGraph-based local coherence model-\ning. ACL.\nGuindon, R. 1988. A multidisciplinary\nperspective on dialogue structure in\nuser-advisor dialogues. In R. Guin-\ndon, ed., Cognitive Science and Its\nApplications for Human-Computer\nInteraction , 163\u2013200. Lawrence Erl-\nbaum.\nGundel, J. K., N. Hedberg, and\nR. Zacharski. 1993. Cognitive status\nand the form of referring expressions\nin discourse. Language , 69(2):274\u2013\n307.\nGururangan, S., A. Marasovi \u00b4c,\nS. Swayamdipta, K. Lo, I. Belt-\nagy, D. Downey, and N. A. Smith.\n2020. Don\u2019t stop pretraining: Adapt\nlanguage models to domains and\ntasks. ACL.\nGus\ufb01eld, D. 1997. Algorithms on\nStrings, Trees, and Sequences . Cam-\nbridge University Press.Guyon, I. and A. Elisseeff. 2003. An\nintroduction to variable and feature\nselection. JMLR , 3:1157\u20131182.\nHaber, J. and M. Poesio. 2020. As-\nsessing polyseme sense similarity\nthrough co-predication acceptability\nand contextualised embedding dis-\ntance. *SEM .\nHabernal, I. and I. Gurevych. 2016.\nWhich argument is more convinc-\ning? Analyzing and predicting con-\nvincingness of Web arguments using\nbidirectional LSTM. ACL.\nHabernal, I. and I. Gurevych. 2017.\nArgumentation mining in user-\ngenerated web discourse. Computa-\ntional Linguistics , 43(1):125\u2013179.\nHaghighi, A. and D. Klein. 2009.\nSimple coreference resolution with\nrich syntactic and semantic features.\nEMNLP .\nHajishirzi, H., L. Zilles, D. S. Weld,\nand L. Zettlemoyer. 2013. Joint\ncoreference resolution and named-\nentity linking with multi-pass sieves.\nEMNLP .\nHaji\u02c7c, J. 1998. Building a Syn-\ntactically Annotated Corpus: The\nPrague Dependency Treebank , 106\u2013\n132. Karolinum.\nHaji\u02c7c, J. 2000. Morphological tagging:\nData vs. dictionaries. NAACL .\nHaji\u02c7c, J., M. Ciaramita, R. Johans-\nson, D. Kawahara, M. A. Mart \u00b4\u0131,\nL. M `arquez, A. Meyers, J. Nivre,\nS. Pad \u00b4o, J. \u02c7St\u02c7ep\u00b4anek, P. Stran \u02c7a\u00b4k,\nM. Surdeanu, N. Xue, and Y . Zhang.\n2009. The conll-2009 shared task:\nSyntactic and semantic dependen-\ncies in multiple languages. CoNLL .\nHakkani-T \u00a8ur, D., K. O\ufb02azer, and\nG. T \u00a8ur. 2002. Statistical morpholog-\nical disambiguation for agglutinative\nlanguages. Journal of Computers\nand Humanities , 36(4):381\u2013410.\nHalliday, M. A. K. and R. Hasan. 1976.\nCohesion in English . Longman. En-\nglish Language Series, Title No. 9.\nHamilton, W. L., K. Clark, J. Leskovec,\nand D. Jurafsky. 2016a. Inducing\ndomain-speci\ufb01c sentiment lexicons\nfrom unlabeled corpora. EMNLP .\nHamilton, W. L., J. Leskovec, and\nD. Jurafsky. 2016b. Diachronic word\nembeddings reveal statistical laws of\nsemantic change. ACL.\nHannun, A. 2017. Sequence modeling\nwith CTC. Distill , 2(11).\nHannun, A. Y ., A. L. Maas, D. Juraf-\nsky, and A. Y . Ng. 2014. First-pass\nlarge vocabulary continuous speech\nrecognition using bi-directional re-\ncurrent DNNs. ArXiv preprint\narXiv:1408.2873.\nHarris, C. M. 1953. A study of the\nbuilding blocks in speech. JASA ,\n25(5):962\u2013969.Harris, R. A. 2005. Voice Interaction\nDesign: Crafting the New Conver-\nsational Speech Systems . Morgan\nKaufmann.\nHarris, Z. S. 1946. From morpheme\nto utterance. Language , 22(3):161\u2013\n183.\nHarris, Z. S. 1954. Distributional struc-\nture. Word , 10:146\u2013162.\nHarris, Z. S. 1962. String Analysis of\nSentence Structure . Mouton, The\nHague.\nHashimoto, T., M. Srivastava,\nH. Namkoong, and P. Liang. 2018.\nFairness without demographics in\nrepeated loss minimization. ICML .\nHastie, T., R. J. Tibshirani, and J. H.\nFriedman. 2001. The Elements of\nStatistical Learning . Springer.\nHatzivassiloglou, V . and K. McKeown.\n1997. Predicting the semantic orien-\ntation of adjectives. ACL.\nHatzivassiloglou, V . and J. Wiebe.\n2000. Effects of adjective orienta-\ntion and gradability on sentence sub-\njectivity. COLING .\nHaviland, S. E. and H. H. Clark. 1974.\nWhat\u2019s new? Acquiring new infor-\nmation as a process in comprehen-\nsion. Journal of Verbal Learning and\nVerbal Behaviour , 13:512\u2013521.\nHawkins, J. A. 1978. De\ufb01niteness\nand inde\ufb01niteness: a study in refer-\nence and grammaticality prediction .\nCroom Helm Ltd.\nHayashi, T., R. Yamamoto, K. In-\noue, T. Yoshimura, S. Watanabe,\nT. Toda, K. Takeda, Y . Zhang,\nand X. Tan. 2020. ESPnet-TTS:\nUni\ufb01ed, reproducible, and integrat-\nable open source end-to-end text-to-\nspeech toolkit. ICASSP .\nHe, L., K. Lee, M. Lewis, and L. Zettle-\nmoyer. 2017. Deep semantic role la-\nbeling: What works and what\u2019s next.\nACL.\nHe, W., K. Liu, J. Liu, Y . Lyu, S. Zhao,\nX. Xiao, Y . Liu, Y . Wang, H. Wu,\nQ. She, X. Liu, T. Wu, and H. Wang.\n2018. DuReader: a Chinese machine\nreading comprehension dataset from\nreal-world applications. Workshop\non Machine Reading for Question\nAnswering .\nHea\ufb01eld, K. 2011. KenLM: Faster\nand smaller language model queries.\nWorkshop on Statistical Machine\nTranslation .\nHea\ufb01eld, K., I. Pouzyrevsky, J. H.\nClark, and P. Koehn. 2013. Scal-\nable modi\ufb01ed Kneser-Ney language\nmodel estimation. ACL.\nHeaps, H. S. 1978. Information re-\ntrieval. Computational and theoret-\nical aspects . Academic Press. 564 Bibliography\nHearst, M. A. 1992a. Automatic acqui-\nsition of hyponyms from large text\ncorpora. COLING .\nHearst, M. A. 1992b. Automatic acqui-\nsition of hyponyms from large text\ncorpora. COLING .\nHearst, M. A. 1997. Texttiling: Seg-\nmenting text into multi-paragraph\nsubtopic passages. Computational\nLinguistics , 23:33\u201364.\nHearst, M. A. 1998. Automatic discov-\nery of WordNet relations. In C. Fell-\nbaum, ed., WordNet: An Electronic\nLexical Database . MIT Press.\nHeckerman, D., E. Horvitz, M. Sahami,\nand S. T. Dumais. 1998. A bayesian\napproach to \ufb01ltering junk e-mail.\nAAAI-98 Workshop on Learning for\nText Categorization .\nHeim, I. 1982. The semantics of de\ufb01nite\nand inde\ufb01nite noun phrases . Ph.D.\nthesis, University of Massachusetts\nat Amherst.\nHellrich, J., S. Buechel, and U. Hahn.\n2019. Modeling word emotion in\nhistorical language: Quantity beats\nsupposed stability in seed word se-\nlection. 3rd Joint SIGHUM Work-\nshop on Computational Linguistics\nfor Cultural Heritage, Social Sci-\nences, Humanities and Literature .\nHellrich, J. and U. Hahn. 2016. Bad\ncompany\u2014Neighborhoods in neural\nembedding spaces considered harm-\nful.COLING .\nHenderson, J. 1994. Description Based\nParsing in a Connectionist Network .\nPh.D. thesis, University of Pennsyl-\nvania, Philadelphia, PA.\nHenderson, J. 2003. Inducing history\nrepresentations for broad coverage\nstatistical parsing. HLT-NAACL-03 .\nHenderson, J. 2004. Discriminative\ntraining of a neural network statisti-\ncal parser. ACL.\nHenderson, P., J. Hu, J. Romoff,\nE. Brunskill, D. Jurafsky, and\nJ. Pineau. 2020. Towards the sys-\ntematic reporting of the energy and\ncarbon footprints of machine learn-\ning. Journal of Machine Learning\nResearch , 21(248):1\u201343.\nHenderson, P., X. Li, D. Jurafsky,\nT. Hashimoto, M. A. Lemley, and\nP. Liang. 2023. Foundation models\nand fair use. JMLR , 24(400):1\u201379.\nHenderson, P., K. Sinha, N. Angelard-\nGontier, N. R. Ke, G. Fried,\nR. Lowe, and J. Pineau. 2017. Eth-\nical challenges in data-driven dia-\nlogue systems. AAAI/ACM AI Ethics\nand Society Conference .\nHendrickx, I., S. N. Kim, Z. Kozareva,\nP. Nakov, D. \u00b4O S\u00b4eaghdha, S. Pad \u00b4o,\nM. Pennacchiotti, L. Romano, and\nS. Szpakowicz. 2009. Semeval-2010task 8: Multi-way classi\ufb01cation of\nsemantic relations between pairs of\nnominals. 5th International Work-\nshop on Semantic Evaluation .\nHendrix, G. G., C. W. Thompson, and\nJ. Slocum. 1973. Language process-\ning via canonical verbs and semantic\nmodels. Proceedings of IJCAI-73 .\nHerdan, G. 1960. Type-token mathe-\nmatics . Mouton.\nHermann, K. M., T. Kocisky, E. Grefen-\nstette, L. Espeholt, W. Kay, M. Su-\nleyman, and P. Blunsom. 2015a.\nTeaching machines to read and com-\nprehend. NeurIPS .\nHermann, K. M., T. Ko \u02c7cisk\u00b4y,\nE. Grefenstette, L. Espeholt, W. Kay,\nM. Suleyman, and P. Blunsom.\n2015b. Teaching machines to read\nand comprehend. NeurIPS .\nHernault, H., H. Prendinger, D. A. du-\nVerle, and M. Ishizuka. 2010. Hilda:\nA discourse parser using support\nvector machine classi\ufb01cation. Dia-\nlogue & Discourse , 1(3).\nHidey, C., E. Musi, A. Hwang, S. Mure-\nsan, and K. McKeown. 2017. Ana-\nlyzing the semantic types of claims\nand premises in an online persuasive\nforum. 4th Workshop on Argument\nMining .\nHill, F., R. Reichart, and A. Korhonen.\n2015. Simlex-999: Evaluating se-\nmantic models with (genuine) sim-\nilarity estimation. Computational\nLinguistics , 41(4):665\u2013695.\nHinkelman, E. A. and J. Allen. 1989.\nTwo constraints on speech act ambi-\nguity. ACL.\nHinton, G. E. 1986. Learning dis-\ntributed representations of concepts.\nCOGSCI .\nHinton, G. E., S. Osindero, and Y .-W.\nTeh. 2006. A fast learning algorithm\nfor deep belief nets. Neural compu-\ntation , 18(7):1527\u20131554.\nHinton, G. E., N. Srivastava,\nA. Krizhevsky, I. Sutskever, and\nR. R. Salakhutdinov. 2012. Improv-\ning neural networks by preventing\nco-adaptation of feature detectors.\nArXiv preprint arXiv:1207.0580.\nHirschberg, J., D. J. Litman, and\nM. Swerts. 2001. Identifying user\ncorrections automatically in spoken\ndialogue systems. NAACL .\nHirschman, L., M. Light, E. Breck, and\nJ. D. Burger. 1999. Deep Read:\nA reading comprehension system.\nACL.\nHirst, G. 1981. Anaphora in Natu-\nral Language Understanding: A sur-\nvey. Number 119 in Lecture notes in\ncomputer science. Springer-Verlag.\nHirst, G. 1987. Semantic Interpreta-\ntion and the Resolution of Ambigu-\nity. Cambridge University Press.Hjelmslev, L. 1969. Prologomena to\na Theory of Language . University\nof Wisconsin Press. Translated by\nFrancis J. Whit\ufb01eld; original Danish\nedition 1943.\nHobbs, J. R. 1978. Resolving pronoun\nreferences. Lingua , 44:311\u2013338.\nHobbs, J. R. 1979. Coherence and\ncoreference. Cognitive Science ,\n3:67\u201390.\nHobbs, J. R., D. E. Appelt, J. Bear,\nD. Israel, M. Kameyama, M. E.\nStickel, and M. Tyson. 1997. FAS-\nTUS: A cascaded \ufb01nite-state trans-\nducer for extracting information\nfrom natural-language text. In\nE. Roche and Y . Schabes, eds,\nFinite-State Language Processing ,\n383\u2013406. MIT Press.\nHochreiter, S. and J. Schmidhuber.\n1997. Long short-term memory.\nNeural Computation , 9(8):1735\u2013\n1780.\nHofmann, T. 1999. Probabilistic latent\nsemantic indexing. SIGIR-99 .\nHoltzman, A., J. Buys, L. Du,\nM. Forbes, and Y . Choi. 2020. The\ncurious case of neural text degener-\nation. ICLR .\nHonovich, O., U. Shaham, S. R. Bow-\nman, and O. Levy. 2023. Instruction\ninduction: From few examples to\nnatural language task descriptions.\nACL.\nHopcroft, J. E. and J. D. Ullman.\n1979. Introduction to Automata The-\nory, Languages, and Computation .\nAddison-Wesley.\nHou, Y ., K. Markert, and M. Strube.\n2018. Unrestricted bridging reso-\nlution. Computational Linguistics ,\n44(2):237\u2013284.\nHouseholder, F. W. 1995. Dionysius\nThrax, the technai , and Sextus Em-\npiricus. In E. F. K. Koerner and\nR. E. Asher, eds, Concise History of\nthe Language Sciences , 99\u2013103. El-\nsevier Science.\nHovy, E. H. 1990. Parsimonious\nand pro\ufb02igate approaches to the\nquestion of discourse structure rela-\ntions. Proceedings of the 5th Inter-\nnational Workshop on Natural Lan-\nguage Generation .\nHovy, E. H., M. P. Marcus, M. Palmer,\nL. A. Ramshaw, and R. Weischedel.\n2006. OntoNotes: The 90% solu-\ntion. HLT-NAACL .\nHu, M. and B. Liu. 2004a. Mining\nand summarizing customer reviews.\nKDD .\nHu, M. and B. Liu. 2004b. Mining\nand summarizing customer reviews.\nSIGKDD-04 .\nHuang, E. H., R. Socher, C. D. Man-\nning, and A. Y . Ng. 2012. Improving Bibliography 565\nword representations via global con-\ntext and multiple word prototypes.\nACL.\nHuang, Z., W. Xu, and K. Yu. 2015.\nBidirectional LSTM-CRF models\nfor sequence tagging. arXiv preprint\narXiv:1508.01991 .\nHuffman, S. 1996. Learning informa-\ntion extraction patterns from exam-\nples. In S. Wertmer, E. Riloff, and\nG. Scheller, eds, Connectionist, Sta-\ntistical, and Symbolic Approaches\nto Learning Natural Language Pro-\ncessing , 246\u2013260. Springer.\nHunt, A. J. and A. W. Black. 1996.\nUnit selection in a concatenative\nspeech synthesis system using a\nlarge speech database. ICASSP .\nHutchins, W. J. 1986. Machine Trans-\nlation: Past, Present, Future . Ellis\nHorwood, Chichester, England.\nHutchins, W. J. 1997. From \ufb01rst con-\nception to \ufb01rst demonstration: The\nnascent years of machine transla-\ntion, 1947\u20131954. A chronology. Ma-\nchine Translation , 12:192\u2013252.\nHutchins, W. J. and H. L. Somers. 1992.\nAn Introduction to Machine Transla-\ntion. Academic Press.\nHutchinson, B., V . Prabhakaran,\nE. Denton, K. Webster, Y . Zhong,\nand S. Denuyl. 2020. Social bi-\nases in NLP models as barriers for\npersons with disabilities. ACL.\nHymes, D. 1974. Ways of speaking.\nIn R. Bauman and J. Sherzer, eds,\nExplorations in the ethnography of\nspeaking , 433\u2013451. Cambridge Uni-\nversity Press.\nIida, R., K. Inui, H. Takamura, and\nY . Matsumoto. 2003. Incorporating\ncontextual cues in trainable models\nfor coreference resolution. EACL\nWorkshop on The Computational\nTreatment of Anaphora .\nIrsoy, O. and C. Cardie. 2014. Opin-\nion mining with deep recurrent neu-\nral networks. EMNLP .\nIschen, C., T. Araujo, H. V oorveld,\nG. van Noort, and E. Smit. 2019.\nPrivacy concerns in chatbot interac-\ntions. International Workshop on\nChatbot Research and Design .\nISO8601. 2004. Data elements and\ninterchange formats\u2014information\ninterchange\u2014representation of\ndates and times. Technical report,\nInternational Organization for Stan-\ndards (ISO).\nItakura, F. 1975. Minimum prediction\nresidual principle applied to speech\nrecognition. IEEE Transactions on\nASSP , ASSP-32:67\u201372.\nIter, D., K. Guu, L. Lansing, and\nD. Jurafsky. 2020. Pretraining\nwith contrastive sentence objectivesimproves discourse performance of\nlanguage models. ACL.\nIter, D., J. Yoon, and D. Jurafsky. 2018.\nAutomatic detection of incoherent\nspeech for diagnosing schizophre-\nnia. Fifth Workshop on Computa-\ntional Linguistics and Clinical Psy-\nchology .\nIto, K. and L. Johnson. 2017.\nThe LJ speech dataset.\nhttps://keithito.com/\nLJ-Speech-Dataset/ .\nIyer, S., I. Konstas, A. Cheung, J. Krish-\nnamurthy, and L. Zettlemoyer. 2017.\nLearning a neural semantic parser\nfrom user feedback. ACL.\nIyer, S., X. V . Lin, R. Pasunuru, T. Mi-\nhaylov, D. Simig, P. Yu, K. Shus-\nter, T. Wang, Q. Liu, P. S. Koura,\nX. Li, B. O\u2019Horo, G. Pereyra,\nJ. Wang, C. Dewan, A. Celikyil-\nmaz, L. Zettlemoyer, and V . Stoy-\nanov. 2022. Opt-iml: Scaling lan-\nguage model instruction meta learn-\ning through the lens of generaliza-\ntion. ArXiv preprint.\nIzacard, G., P. Lewis, M. Lomeli,\nL. Hosseini, F. Petroni, T. Schick,\nJ. Dwivedi-Yu, A. Joulin, S. Riedel,\nand E. Grave. 2022. Few-shot learn-\ning with retrieval augmented lan-\nguage models. ArXiv preprint.\nJackendoff, R. 1983. Semantics and\nCognition . MIT Press.\nJacobs, P. S. and L. F. Rau. 1990.\nSCISOR: A system for extract-\ning information from on-line news.\nCACM , 33(11):88\u201397.\nJaech, A., G. Mulcaire, S. Hathi, M. Os-\ntendorf, and N. A. Smith. 2016.\nHierarchical character-word models\nfor language identi\ufb01cation. ACL\nWorkshop on NLP for Social Media .\nJaitly, N., P. Nguyen, A. Senior, and\nV . Vanhoucke. 2012. Application of\npretrained deep neural networks to\nlarge vocabulary speech recognition.\nINTERSPEECH .\nJauhiainen, T., M. Lui, M. Zampieri,\nT. Baldwin, and K. Lind \u00b4en. 2019.\nAutomatic language identi\ufb01cation in\ntexts: A survey. JAIR , 65(1):675\u2013\n682.\nJefferson, G. 1972. Side sequences. In\nD. Sudnow, ed., Studies in social in-\nteraction , 294\u2013333. Free Press, New\nYork.\nJeffreys, H. 1948. Theory of Probabil-\nity, 2nd edition. Clarendon Press.\nSection 3.23.\nJelinek, F. 1969. A fast sequential de-\ncoding algorithm using a stack. IBM\nJournal of Research and Develop-\nment , 13:675\u2013685.\nJelinek, F. 1990. Self-organized lan-\nguage modeling for speech recogni-\ntion. In A. Waibel and K.-F. Lee,eds, Readings in Speech Recogni-\ntion, 450\u2013506. Morgan Kaufmann.\nOriginally distributed as IBM tech-\nnical report in 1985.\nJelinek, F. and R. L. Mercer. 1980.\nInterpolated estimation of Markov\nsource parameters from sparse data.\nIn E. S. Gelsema and L. N. Kanal,\neds, Proceedings, Workshop on Pat-\ntern Recognition in Practice , 381\u2013\n397. North Holland.\nJelinek, F., R. L. Mercer, and L. R.\nBahl. 1975. Design of a linguis-\ntic statistical decoder for the recog-\nnition of continuous speech. IEEE\nTransactions on Information The-\nory, IT-21(3):250\u2013256.\nJi, H. and R. Grishman. 2011. Knowl-\nedge base population: Successful\napproaches and challenges. ACL.\nJi, H., R. Grishman, and H. T. Dang.\n2010. Overview of the tac 2011\nknowledge base population track.\nTAC-11 .\nJi, Y . and J. Eisenstein. 2014. Repre-\nsentation learning for text-level dis-\ncourse parsing. ACL.\nJi, Y . and J. Eisenstein. 2015. One vec-\ntor is not enough: Entity-augmented\ndistributed semantics for discourse\nrelations. TACL , 3:329\u2013344.\nJia, R. and P. Liang. 2016. Data recom-\nbination for neural semantic parsing.\nACL.\nJia, S., T. Meng, J. Zhao, and K.-W.\nChang. 2020. Mitigating gender bias\nampli\ufb01cation in distribution by pos-\nterior regularization. ACL.\nJohnson, J., M. Douze, and H. J \u00b4egou.\n2017. Billion-scale similarity\nsearch with GPUs. ArXiv preprint\narXiv:1702.08734.\nJohnson, W. E. 1932. Probability: de-\nductive and inductive problems (ap-\npendix to). Mind , 41(164):421\u2013423.\nJohnson-Laird, P. N. 1983. Mental\nModels . Harvard University Press,\nCambridge, MA.\nJones, M. P. and J. H. Martin. 1997.\nContextual spelling correction using\nlatent semantic analysis. ANLP .\nJones, R., A. McCallum, K. Nigam, and\nE. Riloff. 1999. Bootstrapping for\ntext learning tasks. IJCAI-99 Work-\nshop on Text Mining: Foundations,\nTechniques and Applications .\nJones, T. 2015. Toward a descrip-\ntion of African American Vernac-\nular English dialect regions using\n\u201cBlack Twitter\u201d. American Speech ,\n90(4):403\u2013440.\nJoos, M. 1950. Description of language\ndesign. JASA , 22:701\u2013708.\nJordan, M. 1986. Serial order: A paral-\nlel distributed processing approach.\nTechnical Report ICS Report 8604,\nUniversity of California, San Diego. 566 Bibliography\nJoshi, A. K. and P. Hopely. 1999. A\nparser from antiquity. In A. Kor-\nnai, ed., Extended Finite State Mod-\nels of Language , 6\u201315. Cambridge\nUniversity Press.\nJoshi, A. K. and S. Kuhn. 1979. Cen-\ntered logic: The role of entity cen-\ntered sentence representation in nat-\nural language inferencing. IJCAI-79 .\nJoshi, A. K. and S. Weinstein. 1981.\nControl of inference: Role of some\naspects of discourse structure \u2013 cen-\ntering. IJCAI-81 .\nJoshi, M., D. Chen, Y . Liu, D. S.\nWeld, L. Zettlemoyer, and O. Levy.\n2020. SpanBERT: Improving pre-\ntraining by representing and predict-\ning spans. TACL , 8:64\u201377.\nJoshi, M., O. Levy, D. S. Weld, and\nL. Zettlemoyer. 2019. BERT for\ncoreference resolution: Baselines\nand analysis. EMNLP .\nJoty, S., G. Carenini, and R. T. Ng.\n2015. CODRA: A novel discrimi-\nnative framework for rhetorical anal-\nysis. Computational Linguistics ,\n41(3):385\u2013435.\nJurafsky, D. 2014. The Language of\nFood . W. W. Norton, New York.\nJurafsky, D., V . Chahuneau, B. R. Rout-\nledge, and N. A. Smith. 2014. Narra-\ntive framing of consumer sentiment\nin online restaurant reviews. First\nMonday , 19(4).\nJurafsky, D., C. Wooters, G. Tajchman,\nJ. Segal, A. Stolcke, E. Fosler, and\nN. Morgan. 1994. The Berkeley\nrestaurant project. ICSLP .\nJurgens, D., S. M. Mohammad,\nP. Turney, and K. Holyoak. 2012.\nSemEval-2012 task 2: Measur-\ning degrees of relational similarity.\n*SEM 2012 .\nJurgens, D., Y . Tsvetkov, and D. Juraf-\nsky. 2017. Incorporating dialectal\nvariability for socially equitable lan-\nguage identi\ufb01cation. ACL.\nJusteson, J. S. and S. M. Katz. 1991.\nCo-occurrences of antonymous ad-\njectives and their contexts. Compu-\ntational linguistics , 17(1):1\u201319.\nKalchbrenner, N. and P. Blunsom.\n2013. Recurrent continuous transla-\ntion models. EMNLP .\nKameyama, M. 1986. A property-\nsharing constraint in centering. ACL.\nKamp, H. 1981. A theory of truth and\nsemantic representation. In J. Groe-\nnendijk, T. Janssen, and M. Stokhof,\neds, Formal Methods in the Study\nof Language , 189\u2013222. Mathemati-\ncal Centre, Amsterdam.\nKamphuis, C., A. P. de Vries,\nL. Boytsov, and J. Lin. 2020. Which\nBM25 do you mean? a large-scale\nreproducibility study of scoringvariants. European Conference on\nInformation Retrieval .\nKane, S. K., M. R. Morris, A. Paradiso,\nand J. Campbell. 2017. \u201cat times\navuncular and cantankerous, with\nthe re\ufb02exes of a mongoose\u201d: Un-\nderstanding self-expression through\naugmentative and alternative com-\nmunication devices. CSCW .\nKaplan, J., S. McCandlish,\nT. Henighan, T. B. Brown, B. Chess,\nR. Child, S. Gray, A. Radford, J. Wu,\nand D. Amodei. 2020. Scaling laws\nfor neural language models. ArXiv\npreprint.\nKaplan, R. M. 1973. A general syntac-\ntic processor. In R. Rustin, ed., Natu-\nral Language Processing , 193\u2013241.\nAlgorithmics Press.\nKaramanis, N., M. Poesio, C. Mellish,\nand J. Oberlander. 2004. Evaluat-\ning centering-based metrics of co-\nherence for text structuring using a\nreliably annotated corpus. ACL.\nKarita, S., N. Chen, T. Hayashi,\nT. Hori, H. Inaguma, Z. Jiang,\nM. Someki, N. E. Y . Soplin, R. Ya-\nmamoto, X. Wang, S. Watanabe,\nT. Yoshimura, and W. Zhang. 2019.\nA comparative study on transformer\nvs RNN in speech applications.\nIEEE ASRU-19 .\nKarlsson, F., A. V outilainen,\nJ. Heikkil \u00a8a, and A. Anttila, eds.\n1995. Constraint Grammar: A\nLanguage-Independent System for\nParsing Unrestricted Text . Mouton\nde Gruyter.\nKarpukhin, V ., B. O \u02d8guz, S. Min,\nP. Lewis, L. Wu, S. Edunov,\nD. Chen, and W.-t. Yih. 2020. Dense\npassage retrieval for open-domain\nquestion answering. EMNLP .\nKarttunen, L. 1969. Discourse refer-\nents. COLING . Preprint No. 70.\nKarttunen, L. 1999. Comments on\nJoshi. In A. Kornai, ed., Extended\nFinite State Models of Language ,\n16\u201318. Cambridge University Press.\nKasami, T. 1965. An ef\ufb01cient recog-\nnition and syntax analysis algorithm\nfor context-free languages. Tech-\nnical Report AFCRL-65-758, Air\nForce Cambridge Research Labora-\ntory, Bedford, MA.\nKatz, J. J. and J. A. Fodor. 1963. The\nstructure of a semantic theory. Lan-\nguage , 39:170\u2013210.\nKay, M. 1967. Experiments with a pow-\nerful parser. COLING .\nKay, M. 1973. The MIND system.\nIn R. Rustin, ed., Natural Language\nProcessing , 155\u2013188. Algorithmics\nPress.\nKay, M. 1982. Algorithm schemata and\ndata structures in syntactic process-\ning. In S. All \u00b4en, ed., Text Process-\ning: Text Analysis and Generation,Text Typology and Attribution , 327\u2013\n358. Almqvist and Wiksell, Stock-\nholm.\nKay, M. and M. R \u00a8oscheisen. 1988.\nText-translation alignment. Techni-\ncal Report P90-00143, Xerox Palo\nAlto Research Center, Palo Alto,\nCA.\nKay, M. and M. R \u00a8oscheisen. 1993.\nText-translation alignment. Compu-\ntational Linguistics , 19:121\u2013142.\nKehler, A. 1993. The effect of es-\ntablishing coherence in ellipsis and\nanaphora resolution. ACL.\nKehler, A. 1994. Temporal relations:\nReference or discourse coherence?\nACL.\nKehler, A. 1997a. Current theories of\ncentering for pronoun interpretation:\nA critical evaluation. Computational\nLinguistics , 23(3):467\u2013475.\nKehler, A. 1997b. Probabilistic coref-\nerence in information extraction.\nEMNLP .\nKehler, A. 2000. Coherence, Reference,\nand the Theory of Grammar . CSLI\nPublications.\nKehler, A., D. E. Appelt, L. Taylor, and\nA. Simma. 2004. The (non)utility\nof predicate-argument frequencies\nfor pronoun interpretation. HLT-\nNAACL .\nKehler, A. and H. Rohde. 2013. A prob-\nabilistic reconciliation of coherence-\ndriven and centering-driven theories\nof pronoun interpretation. Theoreti-\ncal Linguistics , 39(1-2):1\u201337.\nKeller, F. and M. Lapata. 2003. Using\nthe web to obtain frequencies for un-\nseen bigrams. Computational Lin-\nguistics , 29:459\u2013484.\nKendall, T. and C. Farrington. 2020.\nThe Corpus of Regional African\nAmerican Language. Version\n2020.05. Eugene, OR: The On-\nline Resources for African Amer-\nican Language Project. http:\n//oraal.uoregon.edu/coraal.\nKennedy, C. and B. K. Boguraev. 1996.\nAnaphora for everyone: Pronomi-\nnal anaphora resolution without a\nparser. COLING .\nKhandelwal, U., O. Levy, D. Juraf-\nsky, L. Zettlemoyer, and M. Lewis.\n2019. Generalization through mem-\norization: Nearest neighbor lan-\nguage models. ICLR .\nKhattab, O., C. Potts, and M. Zaharia.\n2021. Relevance-guided supervision\nfor OpenQA with ColBERT. TACL ,\n9:929\u2013944.\nKhattab, O., A. Singhvi, P. Mahesh-\nwari, Z. Zhang, K. Santhanam,\nS. Haq, A. Sharma, T. T. Joshi,\nH. Moazam, H. Miller, M. Zaharia,\nand C. Potts. 2024. DSPy: Compil-\ning declarative language model calls\ninto self-improving pipelines. ICLR . Bibliography 567\nKhattab, O. and M. Zaharia. 2020. Col-\nBERT: Ef\ufb01cient and effective pas-\nsage search via contextualized late\ninteraction over BERT. SIGIR .\nKiela, D., M. Bartolo, Y . Nie,\nD. Kaushik, A. Geiger, Z. Wu,\nB. Vidgen, G. Prasad, A. Singh,\nP. Ringshia, et al. 2021. Dynabench:\nRethinking benchmarking in nlp.\nNAACL HLT .\nKiela, D. and S. Clark. 2014. A system-\natic study of semantic vector space\nmodel parameters. EACL 2nd Work-\nshop on Continuous Vector Space\nModels and their Compositionality\n(CVSC) .\nKim, E. 2019. Optimize com-\nputational ef\ufb01ciency of skip-\ngram with negative sampling.\nhttps://aegis4048.github.\nio/optimize_computational_\nefficiency_of_skip-gram_\nwith_negative_sampling .\nKim, S. M. and E. H. Hovy. 2004. De-\ntermining the sentiment of opinions.\nCOLING .\nKing, S. 2020. From African Amer-\nican Vernacular English to African\nAmerican Language: Rethinking\nthe study of race and language in\nAfrican Americans\u2019 speech. Annual\nReview of Linguistics , 6:285\u2013300.\nKingma, D. and J. Ba. 2015. Adam: A\nmethod for stochastic optimization.\nICLR 2015 .\nKintsch, W. and T. A. Van Dijk. 1978.\nToward a model of text comprehen-\nsion and production. Psychological\nreview , 85(5):363\u2013394.\nKiperwasser, E. and Y . Goldberg. 2016.\nSimple and accurate dependency\nparsing using bidirectional LSTM\nfeature representations. TACL ,\n4:313\u2013327.\nKipper, K., H. T. Dang, and M. Palmer.\n2000. Class-based construction of a\nverb lexicon. AAAI .\nKiritchenko, S. and S. M. Mohammad.\n2017. Best-worst scaling more re-\nliable than rating scales: A case\nstudy on sentiment intensity annota-\ntion. ACL.\nKiritchenko, S. and S. M. Mohammad.\n2018. Examining gender and race\nbias in two hundred sentiment anal-\nysis systems. *SEM .\nKiss, T. and J. Strunk. 2006. Unsuper-\nvised multilingual sentence bound-\nary detection. Computational Lin-\nguistics , 32(4):485\u2013525.\nKitaev, N., S. Cao, and D. Klein.\n2019. Multilingual constituency\nparsing with self-attention and pre-\ntraining. ACL.\nKitaev, N. and D. Klein. 2018. Con-\nstituency parsing with a self-\nattentive encoder. ACL.Klatt, D. H. 1975. V oice onset time,\nfriction, and aspiration in word-\ninitial consonant clusters. Journal\nof Speech and Hearing Research ,\n18:686\u2013706.\nKlatt, D. H. 1977. Review of the ARPA\nspeech understanding project. JASA ,\n62(6):1345\u20131366.\nKlatt, D. H. 1982. The Klattalk text-to-\nspeech conversion system. ICASSP .\nKleene, S. C. 1951. Representation of\nevents in nerve nets and \ufb01nite au-\ntomata. Technical Report RM-704,\nRAND Corporation. RAND Re-\nsearch Memorandum.\nKleene, S. C. 1956. Representation of\nevents in nerve nets and \ufb01nite au-\ntomata. In C. Shannon and J. Mc-\nCarthy, eds, Automata Studies , 3\u201341.\nPrinceton University Press.\nKlein, S. and R. F. Simmons. 1963.\nA computational approach to gram-\nmatical coding of English words.\nJournal of the ACM , 10(3):334\u2013347.\nKnott, A. and R. Dale. 1994. Using\nlinguistic phenomena to motivate a\nset of coherence relations. Discourse\nProcesses , 18(1):35\u201362.\nKocijan, V ., A.-M. Cretu, O.-M.\nCamburu, Y . Yordanov, and\nT. Lukasiewicz. 2019. A surpris-\ningly robust trick for the Winograd\nSchema Challenge. ACL.\nKocmi, T., C. Federmann, R. Grund-\nkiewicz, M. Junczys-Dowmunt,\nH. Matsushita, and A. Menezes.\n2021. To ship or not to ship: An\nextensive evaluation of automatic\nmetrics for machine translation.\nArXiv.\nKoehn, P. 2005. Europarl: A parallel\ncorpus for statistical machine trans-\nlation. MT summit, vol. 5 .\nKoehn, P., H. Hoang, A. Birch,\nC. Callison-Burch, M. Federico,\nN. Bertoldi, B. Cowan, W. Shen,\nC. Moran, R. Zens, C. Dyer, O. Bo-\njar, A. Constantin, and E. Herbst.\n2006. Moses: Open source toolkit\nfor statistical machine translation.\nACL.\nKoehn, P., F. J. Och, and D. Marcu.\n2003. Statistical phrase-based trans-\nlation. HLT-NAACL .\nKolhatkar, V ., A. Roussel, S. Dipper,\nand H. Zinsmeister. 2018. Anaphora\nwith non-nominal antecedents in\ncomputational linguistics: A sur-\nvey. Computational Linguistics ,\n44(3):547\u2013612.\nKreutzer, J., I. Caswell, L. Wang,\nA. Wahab, D. van Esch, N. Ulzii-\nOrshikh, A. Tapo, N. Subra-\nmani, A. Sokolov, C. Sikasote,\nM. Setyawan, S. Sarin, S. Samb,\nB. Sagot, C. Rivera, A. Rios, I. Pa-\npadimitriou, S. Osei, P. O. Suarez,I. Orife, K. Ogueji, A. N. Rubungo,\nT. Q. Nguyen, M. M \u00a8uller, A. M \u00a8uller,\nS. H. Muhammad, N. Muham-\nmad, A. Mnyakeni, J. Mirzakhalov,\nT. Matangira, C. Leong, N. Lawson,\nS. Kudugunta, Y . Jernite, M. Jenny,\nO. Firat, B. F. P. Dossou, S. Dlamini,\nN. de Silva, S. C \u00b8 abuk Ball\u0131, S. Bi-\nderman, A. Battisti, A. Baruwa,\nA. Bapna, P. Baljekar, I. A. Az-\nime, A. Awokoya, D. Ataman,\nO. Ahia, O. Ahia, S. Agrawal, and\nM. Adeyemi. 2022. Quality at a\nglance: An audit of web-crawled\nmultilingual datasets. TACL , 10:50\u2013\n72.\nKrovetz, R. 1993. Viewing morphology\nas an inference process. SIGIR-93 .\nKruskal, J. B. 1983. An overview of se-\nquence comparison. In D. Sankoff\nand J. B. Kruskal, eds, Time\nWarps, String Edits, and Macro-\nmolecules: The Theory and Prac-\ntice of Sequence Comparison , 1\u201344.\nAddison-Wesley.\nKudo, T. 2018. Subword regularization:\nImproving neural network transla-\ntion models with multiple subword\ncandidates. ACL.\nKudo, T. and Y . Matsumoto. 2002.\nJapanese dependency analysis using\ncascaded chunking. CoNLL .\nKudo, T. and J. Richardson. 2018a.\nSentencePiece: A simple and lan-\nguage independent subword tok-\nenizer and detokenizer for neural\ntext processing. EMNLP .\nKudo, T. and J. Richardson. 2018b.\nSentencePiece: A simple and lan-\nguage independent subword tok-\nenizer and detokenizer for neural\ntext processing. EMNLP .\nKullback, S. and R. A. Leibler. 1951.\nOn information and suf\ufb01ciency.\nAnnals of Mathematical Statistics ,\n22:79\u201386.\nKulmizev, A., M. de Lhoneux,\nJ. Gontrum, E. Fano, and J. Nivre.\n2019. Deep contextualized word\nembeddings in transition-based and\ngraph-based dependency parsing\n- a tale of two parsers revisited.\nEMNLP .\nKumar, S. and W. Byrne. 2004. Min-\nimum Bayes-risk decoding for sta-\ntistical machine translation. HLT-\nNAACL .\nKummerfeld, J. K. and D. Klein. 2013.\nError-driven analysis of challenges\nin coreference resolution. EMNLP .\nKuno, S. 1965. The predictive ana-\nlyzer and a path elimination tech-\nnique. CACM , 8(7):453\u2013462.\nKupiec, J. 1992. Robust part-of-speech\ntagging using a hidden Markov\nmodel. Computer Speech and Lan-\nguage , 6:225\u2013242. 568 Bibliography\nKurita, K., N. Vyas, A. Pareek, A. W.\nBlack, and Y . Tsvetkov. 2019. Quan-\ntifying social biases in contextual\nword representations. 1st ACL Work-\nshop on Gender Bias for Natural\nLanguage Processing .\nKu\u02c7cera, H. and W. N. Francis. 1967.\nComputational Analysis of Present-\nDay American English . Brown Univ.\nPress.\nKwiatkowski, T., J. Palomaki, O. Red-\n\ufb01eld, M. Collins, A. Parikh, C. Al-\nberti, D. Epstein, I. Polosukhin,\nJ. Devlin, K. Lee, K. Toutanova,\nL. Jones, M. Kelcey, M.-W. Chang,\nA. M. Dai, J. Uszkoreit, Q. Le, and\nS. Petrov. 2019. Natural questions:\nA benchmark for question answer-\ning research. TACL , 7:452\u2013466.\nLafferty, J. D., A. McCallum, and\nF. C. N. Pereira. 2001. Conditional\nrandom \ufb01elds: Probabilistic mod-\nels for segmenting and labeling se-\nquence data. ICML .\nLai, A. and J. Tetreault. 2018. Dis-\ncourse coherence in the wild: A\ndataset, evaluation and methods.\nSIGDIAL .\nLake, B. M. and G. L. Murphy. 2021.\nWord meaning in minds and ma-\nchines. Psychological Review . In\npress.\nLakoff, G. 1965. On the Nature of Syn-\ntactic Irregularity . Ph.D. thesis, In-\ndiana University. Published as Irreg-\nularity in Syntax . Holt, Rinehart, and\nWinston, New York, 1970.\nLakoff, G. 1972. Structural complexity\nin fairy tales. In The Study of Man ,\n128\u201350. School of Social Sciences,\nUniversity of California, Irvine, CA.\nLakoff, G. and M. Johnson. 1980.\nMetaphors We Live By . University\nof Chicago Press, Chicago, IL.\nLample, G., M. Ballesteros, S. Subra-\nmanian, K. Kawakami, and C. Dyer.\n2016. Neural architectures for\nnamed entity recognition. NAACL\nHLT.\nLample, G. and A. Conneau. 2019.\nCross-lingual language model pre-\ntraining. NeurIPS , volume 32.\nLan, Z., M. Chen, S. Goodman,\nK. Gimpel, P. Sharma, and R. Sori-\ncut. 2020. ALBERT: A lite BERT\nfor self-supervised learning of lan-\nguage representations. ICLR .\nLandauer, T. K., ed. 1995. The Trou-\nble with Computers: Usefulness, Us-\nability, and Productivity . MIT Press.\nLandauer, T. K. and S. T. Dumais. 1997.\nA solution to Plato\u2019s problem: The\nLatent Semantic Analysis theory of\nacquisition, induction, and represen-\ntation of knowledge. Psychological\nReview , 104:211\u2013240.Landauer, T. K., D. Laham, B. Rehder,\nand M. E. Schreiner. 1997. How\nwell can passage meaning be derived\nwithout using word order? A com-\nparison of Latent Semantic Analysis\nand humans. COGSCI .\nLang, J. and M. Lapata. 2014.\nSimilarity-driven semantic role in-\nduction via graph partitioning. Com-\nputational Linguistics , 40(3):633\u2013\n669.\nLang, K. J., A. H. Waibel, and G. E.\nHinton. 1990. A time-delay neu-\nral network architecture for isolated\nword recognition. Neural networks ,\n3(1):23\u201343.\nLapata, M. 2003. Probabilistic text\nstructuring: Experiments with sen-\ntence ordering. ACL.\nLapesa, G. and S. Evert. 2014. A large\nscale evaluation of distributional se-\nmantic models: Parameters, interac-\ntions and model selection. TACL ,\n2:531\u2013545.\nLappin, S. and H. Leass. 1994. An algo-\nrithm for pronominal anaphora res-\nolution. Computational Linguistics ,\n20(4):535\u2013561.\nLarsson, S. and D. R. Traum. 2000. In-\nformation state and dialogue man-\nagement in the trindi dialogue move\nengine toolkit. Natural Language\nEngineering , 6(323-340):97\u2013114.\nLascarides, A. and N. Asher. 1993.\nTemporal interpretation, discourse\nrelations, and common sense entail-\nment. Linguistics and Philosophy ,\n16(5):437\u2013493.\nLawrence, W. 1953. The synthesis of\nspeech from signals which have a\nlow information rate. In W. Jackson,\ned., Communication Theory , 460\u2013\n469. Butterworth.\nLDC. 1998. LDC Catalog: Hub4\nproject . University of Penn-\nsylvania. www.ldc.upenn.edu/\nCatalog/LDC98S71.html .\nLeCun, Y ., B. Boser, J. S. Denker,\nD. Henderson, R. E. Howard,\nW. Hubbard, and L. D. Jackel. 1989.\nBackpropagation applied to hand-\nwritten zip code recognition. Neural\ncomputation , 1(4):541\u2013551.\nLee, D. D. and H. S. Seung. 1999.\nLearning the parts of objects by non-\nnegative matrix factorization. Na-\nture, 401(6755):788\u2013791.\nLee, H., A. Chang, Y . Peirsman,\nN. Chambers, M. Surdeanu, and\nD. Jurafsky. 2013. Determin-\nistic coreference resolution based\non entity-centric, precision-ranked\nrules. Computational Linguistics ,\n39(4):885\u2013916.\nLee, H., Y . Peirsman, A. Chang,\nN. Chambers, M. Surdeanu, andD. Jurafsky. 2011. Stanford\u2019s multi-\npass sieve coreference resolution\nsystem at the CoNLL-2011 shared\ntask. CoNLL .\nLee, H., M. Surdeanu, and D. Juraf-\nsky. 2017a. A scaffolding approach\nto coreference resolution integrat-\ning statistical and rule-based mod-\nels.Natural Language Engineering ,\n23(5):733\u2013762.\nLee, K., M.-W. Chang, and\nK. Toutanova. 2019. Latent re-\ntrieval for weakly supervised open\ndomain question answering. ACL.\nLee, K., L. He, M. Lewis, and L. Zettle-\nmoyer. 2017b. End-to-end neural\ncoreference resolution. EMNLP .\nLee, K., L. He, and L. Zettlemoyer.\n2018. Higher-order coreference\nresolution with coarse-to-\ufb01ne infer-\nence. NAACL HLT .\nLehnert, W. G., C. Cardie, D. Fisher,\nE. Riloff, and R. Williams. 1991.\nDescription of the CIRCUS system\nas used for MUC-3. MUC-3 .\nLemon, O., K. Georgila, J. Henderson,\nand M. Stuttle. 2006. An ISU di-\nalogue system exhibiting reinforce-\nment learning of dialogue policies:\nGeneric slot-\ufb01lling in the TALK in-\ncar system. EACL .\nLevenshtein, V . I. 1966. Binary codes\ncapable of correcting deletions, in-\nsertions, and reversals. Cybernetics\nand Control Theory , 10(8):707\u2013710.\nOriginal in Doklady Akademii Nauk\nSSSR 163(4): 845\u2013848 (1965).\nLevesque, H. 2011. The Winograd\nSchema Challenge. Logical Formal-\nizations of Commonsense Reason-\ning \u2014 Papers from the AAAI 2011\nSpring Symposium (SS-11-06) .\nLevesque, H., E. Davis, and L. Morgen-\nstern. 2012. The Winograd Schema\nChallenge. KR-12 .\nLevesque, H. J., P. R. Cohen, and\nJ. H. T. Nunes. 1990. On acting to-\ngether. AAAI . Morgan Kaufmann.\nLevin, B. 1977. Mapping sentences to\ncase frames. Technical Report 167,\nMIT AI Laboratory. AI Working Pa-\nper 143.\nLevin, B. 1993. English Verb Classes\nand Alternations: A Preliminary In-\nvestigation . University of Chicago\nPress.\nLevin, B. and M. Rappaport Hovav.\n2005. Argument Realization . Cam-\nbridge University Press.\nLevin, E., R. Pieraccini, and W. Eckert.\n2000. A stochastic model of human-\nmachine interaction for learning dia-\nlog strategies. IEEE Transactions on\nSpeech and Audio Processing , 8:11\u2013\n23.\nLevinson, S. C. 1983. Conversational\nAnalysis , chapter 6. Cambridge Uni-\nversity Press. Bibliography 569\nLevow, G.-A. 1998. Characterizing\nand recognizing spoken corrections\nin human-computer dialogue. COL-\nING/ACL .\nLevy, O. and Y . Goldberg. 2014a.\nDependency-based word embed-\ndings. ACL.\nLevy, O. and Y . Goldberg. 2014b. Lin-\nguistic regularities in sparse and ex-\nplicit word representations. CoNLL .\nLevy, O. and Y . Goldberg. 2014c. Neu-\nral word embedding as implicit ma-\ntrix factorization. NeurIPS .\nLevy, O., Y . Goldberg, and I. Da-\ngan. 2015. Improving distributional\nsimilarity with lessons learned from\nword embeddings. TACL , 3:211\u2013\n225.\nLi, B. Z., S. Min, S. Iyer, Y . Mehdad,\nand W.-t. Yih. 2020. Ef\ufb01cient one-\npass end-to-end entity linking for\nquestions. EMNLP .\nLi, J., X. Chen, E. H. Hovy, and D. Ju-\nrafsky. 2015. Visualizing and un-\nderstanding neural models in NLP.\nNAACL HLT .\nLi, J. and D. Jurafsky. 2017. Neu-\nral net models of open-domain dis-\ncourse coherence. EMNLP .\nLi, J., R. Li, and E. H. Hovy. 2014.\nRecursive deep models for discourse\nparsing. EMNLP .\nLi, J., W. Monroe, A. Ritter, D. Juraf-\nsky, M. Galley, and J. Gao. 2016a.\nDeep reinforcement learning for di-\nalogue generation. EMNLP .\nLi, M., J. Weston, and S. Roller. 2019a.\nAcute-eval: Improved dialogue eval-\nuation with optimized questions and\nmulti-turn comparisons. NeurIPS19\nWorkshop on Conversational AI .\nLi, Q., T. Li, and B. Chang. 2016b.\nDiscourse parsing with attention-\nbased hierarchical neural networks.\nEMNLP .\nLi, X., Y . Meng, X. Sun, Q. Han,\nA. Yuan, and J. Li. 2019b. Is\nword segmentation necessary for\ndeep learning of Chinese representa-\ntions? ACL.\nLiang, P., R. Bommasani, T. Lee,\nD. Tsipras, D. Soylu, M. Yasunaga,\nY . Zhang, D. Narayanan, Y . Wu,\nA. Kumar, B. Newman, B. Yuan,\nB. Yan, C. Zhang, C. Cosgrove,\nC. D. Manning, C. R \u00b4e, D. Acosta-\nNavas, D. A. Hudson, E. Zelikman,\nE. Durmus, F. Ladhak, F. Rong,\nH. Ren, H. Yao, J. Wang, K. San-\nthanam, L. Orr, L. Zheng, M. Yuk-\nsekgonul, M. Suzgun, N. Kim,\nN. Guha, N. Chatterji, O. Khattab,\nP. Henderson, Q. Huang, R. Chi,\nS. M. Xie, S. Santurkar, S. Ganguli,\nT. Hashimoto, T. Icard, T. Zhang,\nV . Chaudhary, W. Wang, X. Li,\nY . Mai, Y . Zhang, and Y . Koreeda.2023. Holistic evaluation of lan-\nguage models. Transactions on Ma-\nchine Learning Research .\nLin, C.-Y . 2004. ROUGE: A pack-\nage for automatic evaluation of sum-\nmaries. ACL 2004 Workshop on Text\nSummarization Branches Out .\nLin, D. 2003. Dependency-based eval-\nuation of minipar. Workshop on the\nEvaluation of Parsing Systems .\nLin, Y ., J.-B. Michel, E. Aiden Lieber-\nman, J. Orwant, W. Brockman, and\nS. Petrov. 2012a. Syntactic annota-\ntions for the Google books NGram\ncorpus. ACL.\nLin, Y ., J.-B. Michel, E. Lieber-\nman Aiden, J. Orwant, W. Brock-\nman, and S. Petrov. 2012b. Syntac-\ntic annotations for the Google Books\nNGram corpus. ACL.\nLin, Z., A. Madotto, J. Shin, P. Xu, and\nP. Fung. 2019. MoEL: Mixture of\nempathetic listeners. EMNLP .\nLin, Z., M.-Y . Kan, and H. T. Ng. 2009.\nRecognizing implicit discourse rela-\ntions in the Penn Discourse Tree-\nbank. EMNLP .\nLin, Z., H. T. Ng, and M.-Y . Kan. 2011.\nAutomatically evaluating text coher-\nence using discourse relations. ACL.\nLin, Z., H. T. Ng, and M.-Y . Kan. 2014.\nA pdtb-styled end-to-end discourse\nparser. Natural Language Engineer-\ning, 20(2):151\u2013184.\nLing, W., C. Dyer, A. W. Black,\nI. Trancoso, R. Fermandez, S. Amir,\nL. Marujo, and T. Lu \u00b4\u0131s. 2015. Find-\ning function in form: Compositional\ncharacter models for open vocabu-\nlary word representation. EMNLP .\nLinzen, T. 2016. Issues in evaluating se-\nmantic spaces using word analogies.\n1st Workshop on Evaluating Vector-\nSpace Representations for NLP .\nLison, P. and J. Tiedemann. 2016.\nOpensubtitles2016: Extracting large\nparallel corpora from movie and tv\nsubtitles. LREC .\nLitman, D. J. 1985. Plan Recognition\nand Discourse Analysis: An Inte-\ngrated Approach for Understanding\nDialogues . Ph.D. thesis, University\nof Rochester, Rochester, NY .\nLitman, D. J. and J. Allen. 1987. A plan\nrecognition model for subdialogues\nin conversation. Cognitive Science ,\n11:163\u2013200.\nLitman, D. J., M. A. Walker, and\nM. Kearns. 1999. Automatic detec-\ntion of poor speech recognition at\nthe dialogue level. ACL.\nLiu, B. and L. Zhang. 2012. A sur-\nvey of opinion mining and sentiment\nanalysis. In C. C. Aggarwal and\nC. Zhai, eds, Mining text data , 415\u2013\n464. Springer.Liu, H., J. Dacon, W. Fan, H. Liu,\nZ. Liu, and J. Tang. 2020. Does gen-\nder matter? Towards fairness in dia-\nlogue systems. COLING .\nLiu, J., S. Min, L. Zettlemoyer, Y . Choi,\nand H. Hajishirzi. 2024. In\ufb01ni-gram:\nScaling unbounded n-gram language\nmodels to a trillion tokens. ArXiv\npreprint.\nLiu, Y ., C. Sun, L. Lin, and X. Wang.\n2016. Learning natural language\ninference using bidirectional LSTM\nmodel and inner-attention. ArXiv.\nLiu, Y ., P. Fung, Y . Yang, C. Cieri,\nS. Huang, and D. Graff. 2006.\nHKUST/MTS: A very large scale\nMandarin telephone speech corpus.\nInternational Conference on Chi-\nnese Spoken Language Processing .\nLiu, Y ., M. Ott, N. Goyal, J. Du,\nM. Joshi, D. Chen, O. Levy,\nM. Lewis, L. Zettlemoyer, and\nV . Stoyanov. 2019. RoBERTa:\nA robustly optimized BERT pre-\ntraining approach. ArXiv preprint\narXiv:1907.11692.\nLlama Team. 2024. The llama 3 herd of\nmodels.\nLochbaum, K. E., B. J. Grosz, and\nC. L. Sidner. 2000. Discourse struc-\nture and intention recognition. In\nR. Dale, H. Moisl, and H. L. Somers,\neds,Handbook of Natural Language\nProcessing . Marcel Dekker.\nLogeswaran, L., H. Lee, and D. Radev.\n2018. Sentence ordering and coher-\nence modeling using recurrent neu-\nral networks. AAAI .\nLongpre, S., L. Hou, T. Vu, A. Webson,\nH. W. Chung, Y . Tay, D. Zhou, Q. V .\nLe, B. Zoph, J. Wei, and A. Roberts.\n2023. The Flan collection: Design-\ning data and methods for effective\ninstruction tuning. ICML .\nLongpre, S., R. Mahari, A. Lee,\nC. Lund, H. Oderinwale, W. Bran-\nnon, N. Saxena, N. Obeng-Marnu,\nT. South, C. Hunter, et al. 2024a.\nConsent in crisis: The rapid decline\nof the ai data commons. ArXiv\npreprint.\nLongpre, S., G. Yauney, E. Reif, K. Lee,\nA. Roberts, B. Zoph, D. Zhou,\nJ. Wei, K. Robinson, D. Mimno, and\nD. Ippolito. 2024b. A pretrainer\u2019s\nguide to training data: Measuring\nthe effects of data age, domain cov-\nerage, quality, & toxicity. NAACL\nHLT.\nLouis, A. and A. Nenkova. 2012. A\ncoherence model based on syntactic\npatterns. EMNLP .\nLoureiro, D. and A. Jorge. 2019.\nLanguage modelling makes sense:\nPropagating representations through\nWordNet for full-coverage word\nsense disambiguation. ACL. 570 Bibliography\nLouviere, J. J., T. N. Flynn, and A. A. J.\nMarley. 2015. Best-worst scaling:\nTheory, methods and applications .\nCambridge University Press.\nLovins, J. B. 1968. Development of\na stemming algorithm. Mechanical\nTranslation and Computational Lin-\nguistics , 11(1\u20132):9\u201313.\nLowerre, B. T. 1976. The Harpy Speech\nRecognition System . Ph.D. thesis,\nCarnegie Mellon University, Pitts-\nburgh, PA.\nLuhn, H. P. 1957. A statistical ap-\nproach to the mechanized encoding\nand searching of literary informa-\ntion. IBM Journal of Research and\nDevelopment , 1(4):309\u2013317.\nLui, M. and T. Baldwin. 2011. Cross-\ndomain feature selection for lan-\nguage identi\ufb01cation. IJCNLP .\nLui, M. and T. Baldwin. 2012.\nlangid.py : An off-the-shelf lan-\nguage identi\ufb01cation tool. ACL.\nLukasik, M., B. Dadachev, K. Papineni,\nand G. Sim \u02dcoes. 2020. Text seg-\nmentation by cross segment atten-\ntion. EMNLP .\nLuo, X. 2005. On coreference resolu-\ntion performance metrics. EMNLP .\nLuo, X. and S. Pradhan. 2016. Eval-\nuation metrics. In M. Poesio,\nR. Stuckardt, and Y . Versley, eds,\nAnaphora resolution: Algorithms,\nresources, and applications , 141\u2013\n163. Springer.\nLuo, X., S. Pradhan, M. Recasens, and\nE. H. Hovy. 2014. An extension of\nBLANC to system mentions. ACL.\nMa, X. and E. H. Hovy. 2016. End-\nto-end sequence labeling via bi-\ndirectional LSTM-CNNs-CRF.\nACL.\nMaas, A., Z. Xie, D. Jurafsky, and A. Y .\nNg. 2015. Lexicon-free conversa-\ntional speech recognition with neu-\nral networks. NAACL HLT .\nMaas, A. L., A. Y . Hannun, and A. Y .\nNg. 2013. Recti\ufb01er nonlineari-\nties improve neural network acoustic\nmodels. ICML .\nMaas, A. L., P. Qi, Z. Xie, A. Y . Han-\nnun, C. T. Lengerich, D. Jurafsky,\nand A. Y . Ng. 2017. Building dnn\nacoustic models for large vocabu-\nlary speech recognition. Computer\nSpeech & Language , 41:195\u2013213.\nMagerman, D. M. 1995. Statisti-\ncal decision-tree models for parsing.\nACL.\nMairesse, F. and M. A. Walker. 2008.\nTrainable generation of big-\ufb01ve per-\nsonality styles through data-driven\nparameter estimation. ACL.\nMann, W. C. and S. A. Thompson.\n1987. Rhetorical structure theory: Atheory of text organization. Techni-\ncal Report RS-87-190, Information\nSciences Institute.\nManning, C. D. 2011. Part-of-speech\ntagging from 97% to 100%: Is it\ntime for some linguistics? CICLing\n2011 .\nManning, C. D., P. Raghavan, and\nH. Sch \u00a8utze. 2008. Introduction to In-\nformation Retrieval . Cambridge.\nManning, C. D., M. Surdeanu, J. Bauer,\nJ. Finkel, S. Bethard, and D. Mc-\nClosky. 2014. The Stanford\nCoreNLP natural language process-\ning toolkit. ACL.\nMarcu, D. 1997. The rhetorical parsing\nof natural language texts. ACL.\nMarcu, D. 1999. A decision-based ap-\nproach to rhetorical parsing. ACL.\nMarcu, D. 2000a. The rhetorical pars-\ning of unrestricted texts: A surface-\nbased approach. Computational Lin-\nguistics , 26(3):395\u2013448.\nMarcu, D., ed. 2000b. The Theory and\nPractice of Discourse Parsing and\nSummarization . MIT Press.\nMarcu, D. and A. Echihabi. 2002. An\nunsupervised approach to recogniz-\ning discourse relations. ACL.\nMarcu, D. and W. Wong. 2002.\nA phrase-based, joint probability\nmodel for statistical machine trans-\nlation. EMNLP .\nMarcus, M. P. 1980. A Theory of Syn-\ntactic Recognition for Natural Lan-\nguage . MIT Press.\nMarcus, M. P., B. Santorini, and M. A.\nMarcinkiewicz. 1993. Building a\nlarge annotated corpus of English:\nThe Penn treebank. Computational\nLinguistics , 19(2):313\u2013330.\nMarie, B., A. Fujita, and R. Rubino.\n2021. Scienti\ufb01c credibility of ma-\nchine translation research: A meta-\nevaluation of 769 papers. ACL.\nMarkov, A. A. 1913. Essai d\u2019une\nrecherche statistique sur le texte du\nroman \u201cEugene Onegin\u201d illustrant la\nliaison des epreuve en chain (\u2018Ex-\nample of a statistical investigation\nof the text of \u201cEugene Onegin\u201d il-\nlustrating the dependence between\nsamples in chain\u2019). Izvistia Impera-\ntorskoi Akademii Nauk (Bulletin de\nl\u2019Acad \u00b4emie Imp \u00b4eriale des Sciences\nde St.-P \u00b4etersbourg) , 7:153\u2013162.\nde Marneffe, M.-C., T. Dozat, N. Sil-\nveira, K. Haverinen, F. Ginter,\nJ. Nivre, and C. D. Manning. 2014.\nUniversal Stanford dependencies: A\ncross-linguistic typology. LREC .\nde Marneffe, M.-C., B. MacCartney,\nand C. D. Manning. 2006. Gener-\nating typed dependency parses from\nphrase structure parses. LREC .de Marneffe, M.-C. and C. D. Man-\nning. 2008. The Stanford typed de-\npendencies representation. COLING\nWorkshop on Cross-Framework and\nCross-Domain Parser Evaluation .\nde Marneffe, M.-C., C. D. Manning,\nJ. Nivre, and D. Zeman. 2021. Uni-\nversal Dependencies. Computa-\ntional Linguistics , 47(2):255\u2013308.\nde Marneffe, M.-C., M. Recasens, and\nC. Potts. 2015. Modeling the lifes-\npan of discourse entities with ap-\nplication to coreference resolution.\nJAIR , 52:445\u2013475.\nMaron, M. E. 1961. Automatic index-\ning: an experimental inquiry. Jour-\nnal of the ACM , 8(3):404\u2013417.\nM`arquez, L., X. Carreras, K. C.\nLitkowski, and S. Stevenson. 2008.\nSemantic role labeling: An introduc-\ntion to the special issue. Computa-\ntional linguistics , 34(2):145\u2013159.\nMarshall, I. 1983. Choice of grammat-\nical word-class without global syn-\ntactic analysis: Tagging words in the\nLOB corpus. Computers and the Hu-\nmanities , 17:139\u2013150.\nMarshall, I. 1987. Tag selection using\nprobabilistic methods. In R. Garside,\nG. Leech, and G. Sampson, eds, The\nComputational Analysis of English ,\n42\u201356. Longman.\nMartschat, S. and M. Strube. 2014. Re-\ncall error analysis for coreference\nresolution. EMNLP .\nMartschat, S. and M. Strube. 2015. La-\ntent structures for coreference reso-\nlution. TACL , 3:405\u2013418.\nMathis, D. A. and M. C. Mozer. 1995.\nOn the computational utility of con-\nsciousness. NeurIPS . MIT Press.\nMcCallum, A., D. Freitag, and F. C. N.\nPereira. 2000. Maximum entropy\nMarkov models for information ex-\ntraction and segmentation. ICML .\nMcCallum, A. and W. Li. 2003. Early\nresults for named entity recogni-\ntion with conditional random \ufb01elds,\nfeature induction and web-enhanced\nlexicons. CoNLL .\nMcCallum, A. and K. Nigam. 1998.\nA comparison of event models\nfor naive bayes text classi\ufb01cation.\nAAAI/ICML-98 Workshop on Learn-\ning for Text Categorization .\nMcCarthy, J. F. and W. G. Lehnert.\n1995. Using decision trees for coref-\nerence resolution. IJCAI-95 .\nMcClelland, J. L. and J. L. Elman.\n1986. The TRACE model of speech\nperception. Cognitive Psychology ,\n18:1\u201386.\nMcClelland, J. L. and D. E. Rumel-\nhart, eds. 1986. Parallel Dis-\ntributed Processing: Explorations\nin the Microstructure of Cognition ,\nvolume 2: Psychological and Bio-\nlogical Models . MIT Press. Bibliography 571\nMcCulloch, W. S. and W. Pitts. 1943. A\nlogical calculus of ideas immanent\nin nervous activity. Bulletin of Math-\nematical Biophysics , 5:115\u2013133.\nMcDonald, R., K. Crammer, and\nF. C. N. Pereira. 2005a. Online\nlarge-margin training of dependency\nparsers. ACL.\nMcDonald, R. and J. Nivre. 2011. An-\nalyzing and integrating dependency\nparsers. Computational Linguistics ,\n37(1):197\u2013230.\nMcDonald, R., F. C. N. Pereira, K. Rib-\narov, and J. Haji \u02c7c. 2005b. Non-\nprojective dependency parsing us-\ning spanning tree algorithms. HLT-\nEMNLP .\nMcGuf\ufb01e, K. and A. Newhouse.\n2020. The radicalization risks of\nGPT-3 and advanced neural lan-\nguage models. ArXiv preprint\narXiv:2009.06807.\nMcLuhan, M. 1964. Understanding\nMedia: The Extensions of Man . New\nAmerican Library.\nMelamud, O., J. Goldberger, and I. Da-\ngan. 2016. context2vec: Learn-\ning generic context embedding with\nbidirectional LSTM. CoNLL .\nMerialdo, B. 1994. Tagging En-\nglish text with a probabilistic\nmodel. Computational Linguistics ,\n20(2):155\u2013172.\nMesgar, M. and M. Strube. 2016. Lexi-\ncal coherence graph modeling using\nword embeddings. ACL.\nMetsis, V ., I. Androutsopoulos, and\nG. Paliouras. 2006. Spam \ufb01lter-\ning with naive bayes-which naive\nbayes? CEAS .\nMeyers, A., R. Reeves, C. Macleod,\nR. Szekely, V . Zielinska, B. Young,\nand R. Grishman. 2004. The nom-\nbank project: An interim report.\nNAACL/HLT Workshop: Frontiers in\nCorpus Annotation .\nMihalcea, R. and A. Csomai. 2007.\nWikify!: Linking documents to en-\ncyclopedic knowledge. CIKM 2007 .\nMikheev, A., M. Moens, and C. Grover.\n1999. Named entity recognition\nwithout gazetteers. EACL .\nMikolov, T. 2012. Statistical lan-\nguage models based on neural net-\nworks . Ph.D. thesis, Brno University\nof Technology.\nMikolov, T., K. Chen, G. S. Corrado,\nand J. Dean. 2013a. Ef\ufb01cient estima-\ntion of word representations in vec-\ntor space. ICLR 2013 .\nMikolov, T., M. Kara\ufb01 \u00b4at, L. Bur-\nget, J. \u02c7Cernock `y, and S. Khudan-\npur. 2010. Recurrent neural net-\nwork based language model. IN-\nTERSPEECH .Mikolov, T., S. Kombrink, L. Burget,\nJ. H. \u02c7Cernock `y, and S. Khudanpur.\n2011. Extensions of recurrent neural\nnetwork language model. ICASSP .\nMikolov, T., I. Sutskever, K. Chen,\nG. S. Corrado, and J. Dean. 2013b.\nDistributed representations of words\nand phrases and their compositional-\nity.NeurIPS .\nMikolov, T., W.-t. Yih, and G. Zweig.\n2013c. Linguistic regularities in\ncontinuous space word representa-\ntions. NAACL HLT .\nMiller, G. A. and J. G. Beebe-Center.\n1956. Some psychological methods\nfor evaluating the quality of trans-\nlations. Mechanical Translation ,\n3:73\u201380.\nMiller, G. A. and W. G. Charles. 1991.\nContextual correlates of semantics\nsimilarity. Language and Cognitive\nProcesses , 6(1):1\u201328.\nMiller, G. A. and N. Chomsky. 1963.\nFinitary models of language users.\nIn R. D. Luce, R. R. Bush, and\nE. Galanter, eds, Handbook of Math-\nematical Psychology , volume II,\n419\u2013491. John Wiley.\nMiller, G. A. and J. A. Selfridge.\n1950. Verbal context and the recall\nof meaningful material. American\nJournal of Psychology , 63:176\u2013185.\nMiller, S., R. J. Bobrow, R. Ingria, and\nR. Schwartz. 1994. Hidden under-\nstanding models of natural language.\nACL.\nMilne, D. and I. H. Witten. 2008.\nLearning to link with wikipedia.\nCIKM 2008 .\nMiltsakaki, E., R. Prasad, A. K. Joshi,\nand B. L. Webber. 2004. The Penn\nDiscourse Treebank. LREC .\nMin, S., X. Lyu, A. Holtzman,\nM. Artetxe, M. Lewis, H. Hajishirzi,\nand L. Zettlemoyer. 2022. Rethink-\ning the role of demonstrations: What\nmakes in-context learning work?\nEMNLP .\nMinsky, M. 1961. Steps toward arti\ufb01-\ncial intelligence. Proceedings of the\nIRE, 49(1):8\u201330.\nMinsky, M. 1974. A framework for rep-\nresenting knowledge. Technical Re-\nport 306, MIT AI Laboratory. Memo\n306.\nMinsky, M. and S. Papert. 1969. Per-\nceptrons . MIT Press.\nMintz, M., S. Bills, R. Snow, and D. Ju-\nrafsky. 2009. Distant supervision for\nrelation extraction without labeled\ndata. ACL IJCNLP .\nMirza, P. and S. Tonelli. 2016.\nCATENA: CAusal and TEmporal\nrelation extraction from NAtural\nlanguage texts. COLING .Mishra, S., D. Khashabi, C. Baral,\nand H. Hajishirzi. 2022. Cross-task\ngeneralization via natural language\ncrowdsourcing instructions. ACL.\nMitchell, M., S. Wu, A. Zal-\ndivar, P. Barnes, L. Vasserman,\nB. Hutchinson, E. Spitzer, I. D. Raji,\nand T. Gebru. 2019. Model cards for\nmodel reporting. ACM FAccT .\nMitkov, R. 2002. Anaphora Resolution .\nLongman.\nMohamed, A., G. E. Dahl, and G. E.\nHinton. 2009. Deep Belief Networks\nfor phone recognition. NIPS Work-\nshop on Deep Learning for Speech\nRecognition and Related Applica-\ntions .\nMohammad, S. M. 2018a. Obtaining\nreliable human ratings of valence,\narousal, and dominance for 20,000\nEnglish words. ACL.\nMohammad, S. M. 2018b. Word affect\nintensities. LREC .\nMohammad, S. M. and P. D. Tur-\nney. 2013. Crowdsourcing a word-\nemotion association lexicon. Com-\nputational Intelligence , 29(3):436\u2013\n465.\nMonroe, B. L., M. P. Colaresi, and\nK. M. Quinn. 2008. Fightin\u2019words:\nLexical feature selection and evalu-\nation for identifying the content of\npolitical con\ufb02ict. Political Analysis ,\n16(4):372\u2013403.\nMoors, A., P. C. Ellsworth, K. R.\nScherer, and N. H. Frijda. 2013. Ap-\npraisal theories of emotion: State\nof the art and future development.\nEmotion Review , 5(2):119\u2013124.\nMoosavi, N. S. and M. Strube. 2016.\nWhich coreference evaluation met-\nric do you trust? A proposal for a\nlink-based entity aware metric. ACL.\nMorey, M., P. Muller, and N. Asher.\n2017. How much progress have we\nmade on RST discourse parsing? a\nreplication study of recent results on\nthe rst-dt. EMNLP .\nMorgan, A. A., L. Hirschman,\nM. Colosimo, A. S. Yeh, and J. B.\nColombe. 2004. Gene name iden-\nti\ufb01cation and normalization using a\nmodel organism database. Journal of\nBiomedical Informatics , 37(6):396\u2013\n410.\nMorgan, N. and H. Bourlard. 1990.\nContinuous speech recognition us-\ning multilayer perceptrons with hid-\nden markov models. ICASSP .\nMorgan, N. and H. A. Bourlard.\n1995. Neural networks for sta-\ntistical recognition of continuous\nspeech. Proceedings of the IEEE ,\n83(5):742\u2013772. 572 Bibliography\nMorris, J. and G. Hirst. 1991. Lexical\ncohesion computed by thesaural re-\nlations as an indicator of the struc-\nture of text. Computational Linguis-\ntics, 17(1):21\u201348.\nMosteller, F. and D. L. Wallace. 1963.\nInference in an authorship problem:\nA comparative study of discrimina-\ntion methods applied to the author-\nship of the disputed federalist pa-\npers. Journal of the American Statis-\ntical Association , 58(302):275\u2013309.\nMosteller, F. and D. L. Wallace. 1964.\nInference and Disputed Authorship:\nThe Federalist . Springer-Verlag.\n1984 2nd edition: Applied Bayesian\nand Classical Inference .\nMrk\u02c7si\u00b4c, N., D. \u00b4O S\u00b4eaghdha, T.-H. Wen,\nB. Thomson, and S. Young. 2017.\nNeural belief tracker: Data-driven\ndialogue state tracking. ACL.\nMuller, P., C. Braud, and M. Morey.\n2019. ToNy: Contextual embed-\ndings for accurate multilingual dis-\ncourse segmentation of full docu-\nments. Workshop on Discourse Re-\nlation Parsing and Treebanking .\nMurphy, K. P. 2012. Machine learning:\nA probabilistic perspective . MIT\nPress.\nMusi, E., M. Stede, L. Kriese, S. Mure-\nsan, and A. Rocci. 2018. A multi-\nlayer annotated corpus of argumen-\ntative text: From argument schemes\nto discourse relations. LREC .\nMyers, G. 1992. \u201cIn this paper we\nreport...\u201d: Speech acts and scien-\nti\ufb01c facts. Journal of Pragmatics ,\n17(4):295\u2013313.\nN\u00b4adas, A. 1984. Estimation of prob-\nabilities in the language model of\nthe IBM speech recognition sys-\ntem. IEEE Transactions on ASSP ,\n32(4):859\u2013861.\nNadeem, M., A. Bethke, and S. Reddy.\n2021. StereoSet: Measuring stereo-\ntypical bias in pretrained language\nmodels. ACL.\nNagata, M. and T. Morimoto. 1994.\nFirst steps toward statistical model-\ning of dialogue to predict the speech\nact type of the next utterance. Speech\nCommunication , 15:193\u2013203.\nNallapati, R., B. Zhou, C. dos San-\ntos, C \u00b8 . Gulc \u00b8ehre, and B. Xiang.\n2016. Abstractive text summa-\nrization using sequence-to-sequence\nRNNs and beyond. CoNLL .\nNash-Webber, B. L. 1975. The role of\nsemantics in automatic speech un-\nderstanding. In D. G. Bobrow and\nA. Collins, eds, Representation and\nUnderstanding , 351\u2013382. Academic\nPress.\nNaur, P., J. W. Backus, F. L. Bauer,\nJ. Green, C. Katz, J. McCarthy, A. J.Perlis, H. Rutishauser, K. Samelson,\nB. Vauquois, J. H. Wegstein, A. van\nWijnagaarden, and M. Woodger.\n1960. Report on the algorith-\nmic language ALGOL 60. CACM ,\n3(5):299\u2013314. Revised in CACM\n6:1, 1-17, 1963.\nNayak, N., D. Hakkani-T \u00a8ur, M. A.\nWalker, and L. P. Heck. 2017. To\nplan or not to plan? discourse\nplanning in slot-value informed se-\nquence to sequence models for lan-\nguage generation. INTERSPEECH .\nNeff, G. and P. Nagy. 2016. Talking\nto bots: Symbiotic agency and the\ncase of Tay. International Journal\nof Communication , 10:4915\u20134931.\nNg, A. Y . and M. I. Jordan. 2002. On\ndiscriminative vs. generative classi-\n\ufb01ers: A comparison of logistic re-\ngression and naive bayes. NeurIPS .\nNg, H. T., L. H. Teo, and J. L. P. Kwan.\n2000. A machine learning approach\nto answering questions for reading\ncomprehension tests. EMNLP .\nNg, V . 2004. Learning noun phrase\nanaphoricity to improve coreference\nresolution: Issues in representation\nand optimization. ACL.\nNg, V . 2005a. Machine learning for\ncoreference resolution: From lo-\ncal classi\ufb01cation to global ranking.\nACL.\nNg, V . 2005b. Supervised ranking\nfor pronoun resolution: Some recent\nimprovements. AAAI .\nNg, V . 2010. Supervised noun phrase\ncoreference research: The \ufb01rst \ufb01f-\nteen years. ACL.\nNg, V . 2017. Machine learning for en-\ntity coreference resolution: A retro-\nspective look at two decades of re-\nsearch. AAAI .\nNg, V . and C. Cardie. 2002a. Identi-\nfying anaphoric and non-anaphoric\nnoun phrases to improve coreference\nresolution. COLING .\nNg, V . and C. Cardie. 2002b. Improv-\ning machine learning approaches to\ncoreference resolution. ACL.\nNguyen, D. T. and S. Joty. 2017. A neu-\nral local coherence model. ACL.\nNickerson, R. S. 1976. On con-\nversational interaction with comput-\ners. Proceedings of the ACM/SIG-\nGRAPH workshop on User-oriented\ndesign of interactive graphics sys-\ntems.\nNie, A., E. Bennett, and N. Good-\nman. 2019. DisSent: Learning sen-\ntence representations from explicit\ndiscourse relations. ACL.\nNielsen, J. 1992. The usability engi-\nneering life cycle. IEEE Computer ,\n25(3):12\u201322.Nielsen, M. A. 2015. Neural networks\nand Deep learning . Determination\nPress USA.\nNigam, K., J. D. Lafferty, and A. Mc-\nCallum. 1999. Using maximum en-\ntropy for text classi\ufb01cation. IJCAI-\n99 workshop on machine learning\nfor information \ufb01ltering .\nNirenburg, S., H. L. Somers, and\nY . Wilks, eds. 2002. Readings in\nMachine Translation . MIT Press.\nNissim, M., S. Dingare, J. Carletta, and\nM. Steedman. 2004. An annotation\nscheme for information status in di-\nalogue. LREC .\nNIST. 2005. Speech recognition\nscoring toolkit (sctk) version 2.1.\nhttp://www.nist.gov/speech/\ntools/ .\nNIST. 2007. Matched Pairs Sentence-\nSegment Word Error (MAPSSWE)\nTest.\nNivre, J. 2007. Incremental non-\nprojective dependency parsing.\nNAACL-HLT .\nNivre, J. 2003. An ef\ufb01cient algorithm\nfor projective dependency parsing.\nProceedings of the 8th International\nWorkshop on Parsing Technologies\n(IWPT) .\nNivre, J. 2006. Inductive Dependency\nParsing . Springer.\nNivre, J. 2009. Non-projective de-\npendency parsing in expected linear\ntime. ACL IJCNLP .\nNivre, J., J. Hall, S. K \u00a8ubler, R. Mc-\nDonald, J. Nilsson, S. Riedel, and\nD. Yuret. 2007a. The conll 2007\nshared task on dependency parsing.\nEMNLP/CoNLL .\nNivre, J., J. Hall, J. Nilsson, A. Chanev,\nG. Eryigit, S. K \u00a8ubler, S. Mari-\nnov, and E. Marsi. 2007b. Malt-\nparser: A language-independent\nsystem for data-driven dependency\nparsing. Natural Language Engi-\nneering , 13(02):95\u2013135.\nNivre, J. and J. Nilsson. 2005. Pseudo-\nprojective dependency parsing. ACL.\nNivre, J. and M. Scholz. 2004. Deter-\nministic dependency parsing of en-\nglish text. COLING .\nNiwa, Y . and Y . Nitta. 1994. Co-\noccurrence vectors from corpora vs.\ndistance vectors from dictionaries.\nCOLING .\nNoreen, E. W. 1989. Computer Inten-\nsive Methods for Testing Hypothesis .\nWiley.\nNorman, D. A. 1988. The Design of Ev-\neryday Things . Basic Books.\nNorvig, P. 1991. Techniques for au-\ntomatic memoization with applica-\ntions to context-free parsing. Com-\nputational Linguistics , 17(1):91\u201398. Bibliography 573\nNosek, B. A., M. R. Banaji, and\nA. G. Greenwald. 2002a. Harvest-\ning implicit group attitudes and be-\nliefs from a demonstration web site.\nGroup Dynamics: Theory, Research,\nand Practice , 6(1):101.\nNosek, B. A., M. R. Banaji, and A. G.\nGreenwald. 2002b. Math=male,\nme=female, therefore math 6=me.\nJournal of personality and social\npsychology , 83(1):44.\nNostalgebraist. 2020. Interpreting gpt:\nthe logit lens. White paper.\nOcal, M., A. Perez, A. Radas, and\nM. Finlayson. 2022. Holistic eval-\nuation of automatic TimeML anno-\ntators. LREC .\nOch, F. J. 1998. Ein beispiels-\nbasierter und statistischer Ansatz\nzum maschinellen Lernen von\nnat\u00a8urlichsprachlicher \u00a8Ubersetzung .\nPh.D. thesis, Universit \u00a8at Erlangen-\nN\u00a8urnberg, Germany. Diplomarbeit\n(diploma thesis).\nOch, F. J. 2003. Minimum error rate\ntraining in statistical machine trans-\nlation. ACL.\nOch, F. J. and H. Ney. 2002. Discrim-\ninative training and maximum en-\ntropy models for statistical machine\ntranslation. ACL.\nOch, F. J. and H. Ney. 2003. A system-\natic comparison of various statistical\nalignment models. Computational\nLinguistics , 29(1):19\u201351.\nOch, F. J. and H. Ney. 2004. The align-\nment template approach to statistical\nmachine translation. Computational\nLinguistics , 30(4):417\u2013449.\nO\u2019Connor, B., M. Krieger, and D. Ahn.\n2010. Tweetmotif: Exploratory\nsearch and topic summarization for\ntwitter. ICWSM .\nOlive, J. P. 1977. Rule synthe-\nsis of speech from dyadic units.\nICASSP77 .\nOlsson, C., N. Elhage, N. Nanda,\nN. Joseph, N. DasSarma,\nT. Henighan, B. Mann, A. Askell,\nY . Bai, A. Chen, et al. 2022. In-\ncontext learning and induction\nheads. ArXiv preprint.\nOlteanu, A., F. Diaz, and G. Kazai.\n2020. When are search completion\nsuggestions problematic? CSCW .\nvan den Oord, A., S. Dieleman, H. Zen,\nK. Simonyan, O. Vinyals, A. Graves,\nN. Kalchbrenner, A. Senior, and\nK. Kavukcuoglu. 2016. WaveNet:\nA Generative Model for Raw Audio.\nISCA Workshop on Speech Synthesis\nWorkshop .\nOppenheim, A. V ., R. W. Schafer, and\nT. G. J. Stockham. 1968. Nonlinear\n\ufb01ltering of multiplied and convolved\nsignals. Proceedings of the IEEE ,\n56(8):1264\u20131291.Oravecz, C. and P. Dienes. 2002. Ef-\n\ufb01cient stochastic part-of-speech tag-\nging for Hungarian. LREC .\nOsgood, C. E., G. J. Suci, and P. H. Tan-\nnenbaum. 1957. The Measurement\nof Meaning . University of Illinois\nPress.\nOuyang, L., J. Wu, X. Jiang,\nD. Almeida, C. Wainwright,\nP. Mishkin, C. Zhang, S. Agar-\nwal, K. Slama, A. Ray, J. Schul-\nman, J. Hilton, F. Kelton, L. Miller,\nM. Simens, A. Askell, P. Welinder,\nP. Christiano, J. Leike, and R. Lowe.\n2022. Training language models\nto follow instructions with human\nfeedback. NeurIPS , volume 35.\nPackard, D. W. 1973. Computer-\nassisted morphological analysis of\nancient Greek. COLING .\nPalmer, D. 2012. Text preprocessing.\nIn N. Indurkhya and F. J. Damerau,\neds,Handbook of Natural Language\nProcessing , 9\u201330. CRC Press.\nPalmer, M., D. Gildea, and N. Xue.\n2010. Semantic role labeling. Syn-\nthesis Lectures on Human Language\nTechnologies , 3(1):1\u2013103.\nPalmer, M., P. Kingsbury, and\nD. Gildea. 2005. The proposi-\ntion bank: An annotated corpus\nof semantic roles. Computational\nLinguistics , 31(1):71\u2013106.\nPanayotov, V ., G. Chen, D. Povey, and\nS. Khudanpur. 2015. Librispeech: an\nASR corpus based on public domain\naudio books. ICASSP .\nPang, B. and L. Lee. 2008. Opin-\nion mining and sentiment analysis.\nFoundations and trends in informa-\ntion retrieval , 2(1-2):1\u2013135.\nPang, B., L. Lee, and S. Vaithyanathan.\n2002. Thumbs up? Sentiment\nclassi\ufb01cation using machine learn-\ning techniques. EMNLP .\nPaolino, J. 2017. Google Home\nvs Alexa: Two simple user\nexperience design gestures\nthat delighted a female user.\nMedium . Jan 4, 2017. https:\n//medium.com/startup-grind/\ngoogle-home-vs-alexa-56e26f69ac77 .\nPapadimitriou, I., K. Lopez, and D. Ju-\nrafsky. 2023. Multilingual BERT has\nan accent: Evaluating English in-\n\ufb02uences on \ufb02uency in multilingual\nmodels. EACL Findings .\nPapineni, K., S. Roukos, T. Ward, and\nW.-J. Zhu. 2002. Bleu: A method\nfor automatic evaluation of machine\ntranslation. ACL.\nPark, J. H., J. Shin, and P. Fung. 2018.\nReducing gender bias in abusive lan-\nguage detection. EMNLP .\nPark, J. and C. Cardie. 2014. Identify-\ning appropriate support for proposi-\ntions in online user comments. First\nworkshop on argumentation mining .Parrish, A., A. Chen, N. Nangia, V . Pad-\nmakumar, J. Phang, J. Thompson,\nP. M. Htut, and S. Bowman. 2022.\nBBQ: A hand-built bias benchmark\nfor question answering. Findings of\nACL 2022 .\nPaszke, A., S. Gross, S. Chintala,\nG. Chanan, E. Yang, Z. DeVito,\nZ. Lin, A. Desmaison, L. Antiga,\nand A. Lerer. 2017. Automatic dif-\nferentiation in pytorch. NIPS-W .\nPearl, C. 2017. Designing Voice User\nInterfaces: Principles of Conversa-\ntional Experiences . O\u2019Reilly.\nPeldszus, A. and M. Stede. 2013. From\nargument diagrams to argumentation\nmining in texts: A survey. In-\nternational Journal of Cognitive In-\nformatics and Natural Intelligence\n(IJCINI) , 7(1):1\u201331.\nPeldszus, A. and M. Stede. 2016. An\nannotated corpus of argumentative\nmicrotexts. 1st European Confer-\nence on Argumentation .\nPenn, G. and P. Kiparsky. 2012. On\nP\u00afan.ini and the generative capacity of\ncontextualized replacement systems.\nCOLING .\nPennebaker, J. W., R. J. Booth, and\nM. E. Francis. 2007. Linguistic In-\nquiry and Word Count: LIWC 2007 .\nAustin, TX.\nPennington, J., R. Socher, and C. D.\nManning. 2014. GloVe: Global\nvectors for word representation.\nEMNLP .\nPercival, W. K. 1976. On the his-\ntorical source of immediate con-\nstituent analysis. In J. D. McCawley,\ned., Syntax and Semantics Volume\n7, Notes from the Linguistic Under-\nground , 229\u2013242. Academic Press.\nPerrault, C. R. and J. Allen. 1980.\nA plan-based analysis of indirect\nspeech acts. American Journal\nof Computational Linguistics , 6(3-\n4):167\u2013182.\nPeters, M., M. Neumann, M. Iyyer,\nM. Gardner, C. Clark, K. Lee,\nand L. Zettlemoyer. 2018. Deep\ncontextualized word representations.\nNAACL HLT .\nPeterson, G. E., W. S.-Y . Wang, and\nE. Sivertsen. 1958. Segmenta-\ntion techniques in speech synthesis.\nJASA , 30(8):739\u2013742.\nPeterson, J. C., D. Chen, and T. L. Grif-\n\ufb01ths. 2020. Parallelograms revisited:\nExploring the limitations of vector\nspace models for simple analogies.\nCognition , 205.\nPetroni, F., T. Rockt \u00a8aschel, S. Riedel,\nP. Lewis, A. Bakhtin, Y . Wu, and\nA. Miller. 2019. Language models\nas knowledge bases? EMNLP . 574 Bibliography\nPetrov, S., D. Das, and R. McDonald.\n2012. A universal part-of-speech\ntagset. LREC .\nPetrov, S. and R. McDonald. 2012.\nOverview of the 2012 shared task on\nparsing the web. Notes of the First\nWorkshop on Syntactic Analysis of\nNon-Canonical Language (SANCL) ,\nvolume 59.\nPhillips, A. V . 1960. A question-\nanswering routine. Technical Re-\nport 16, MIT AI Lab.\nPicard, R. W. 1995. Affective comput-\ning. Technical Report 321, MIT Me-\ndia Lab Perceputal Computing Tech-\nnical Report. Revised November 26,\n1995.\nPieraccini, R., E. Levin, and C.-H.\nLee. 1991. Stochastic representation\nof conceptual structure in the ATIS\ntask. Speech and Natural Language\nWorkshop .\nPierce, J. R., J. B. Carroll, E. P.\nHamp, D. G. Hays, C. F. Hockett,\nA. G. Oettinger, and A. J. Perlis.\n1966. Language and Machines:\nComputers in Translation and Lin-\nguistics . ALPAC report. National\nAcademy of Sciences, National Re-\nsearch Council, Washington, DC.\nPilehvar, M. T. and J. Camacho-\nCollados. 2019. WiC: the word-\nin-context dataset for evaluating\ncontext-sensitive meaning represen-\ntations. NAACL HLT .\nPitler, E., A. Louis, and A. Nenkova.\n2009. Automatic sense prediction\nfor implicit discourse relations in\ntext. ACL IJCNLP .\nPitler, E. and A. Nenkova. 2009. Us-\ning syntax to disambiguate explicit\ndiscourse connectives in text. ACL\nIJCNLP .\nPlutchik, R. 1962. The emotions: Facts,\ntheories, and a new model . Random\nHouse.\nPlutchik, R. 1980. A general psycho-\nevolutionary theory of emotion. In\nR. Plutchik and H. Kellerman, eds,\nEmotion: Theory, Research, and Ex-\nperience, Volume 1 , 3\u201333. Academic\nPress.\nPoesio, M., R. Stevenson, B. Di Euge-\nnio, and J. Hitzeman. 2004. Center-\ning: A parametric theory and its in-\nstantiations. Computational Linguis-\ntics, 30(3):309\u2013363.\nPoesio, M., R. Stuckardt, and Y . Ver-\nsley. 2016. Anaphora resolution:\nAlgorithms, resources, and applica-\ntions . Springer.\nPoesio, M., P. Sturt, R. Artstein, and\nR. Filik. 2006. Underspeci\ufb01cation\nand anaphora: Theoretical issues\nand preliminary evidence. Discourse\nprocesses , 42(2):157\u2013175.Poesio, M. and R. Vieira. 1998. A\ncorpus-based investigation of de\ufb01-\nnite description use. Computational\nLinguistics , 24(2):183\u2013216.\nPolanyi, L. 1988. A formal model of\nthe structure of discourse. Journal\nof Pragmatics , 12.\nPolanyi, L., C. Culy, M. van den Berg,\nG. L. Thione, and D. Ahn. 2004.\nA rule based approach to discourse\nparsing. Proceedings of SIGDIAL .\nPollard, C. and I. A. Sag. 1994. Head-\nDriven Phrase Structure Grammar .\nUniversity of Chicago Press.\nPonzetto, S. P. and M. Strube. 2006.\nExploiting semantic role labeling,\nWordNet and Wikipedia for corefer-\nence resolution. HLT-NAACL .\nPonzetto, S. P. and M. Strube. 2007.\nKnowledge derived from Wikipedia\nfor computing semantic relatedness.\nJAIR , 30:181\u2013212.\nPopovi \u00b4c, M. 2015. chrF: charac-\nter n-gram F-score for automatic\nMT evaluation. Proceedings of the\nTenth Workshop on Statistical Ma-\nchine Translation .\nPopp, D., R. A. Donovan, M. Craw-\nford, K. L. Marsh, and M. Peele.\n2003. Gender, race, and speech style\nstereotypes. Sex Roles , 48(7-8):317\u2013\n325.\nPorter, M. F. 1980. An algorithm\nfor suf\ufb01x stripping. Program ,\n14(3):130\u2013137.\nPost, M. 2018. A call for clarity in re-\nporting BLEU scores. WMT 2018 .\nPotts, C. 2011. On the negativity of\nnegation. In N. Li and D. Lutz,\neds, Proceedings of Semantics and\nLinguistic Theory 20 , 636\u2013659. CLC\nPublications, Ithaca, NY .\nPovey, D., A. Ghoshal, G. Boulianne,\nL. Burget, O. Glembek, N. Goel,\nM. Hannemann, P. Motlicek,\nY . Qian, P. Schwarz, J. Silovsk \u00b4y,\nG. Stemmer, and K. Vesel \u00b4y. 2011.\nThe Kaldi speech recognition\ntoolkit. ASRU .\nPradhan, S., E. H. Hovy, M. P. Mar-\ncus, M. Palmer, L. Ramshaw, and\nR. Weischedel. 2007a. OntoNotes:\nA uni\ufb01ed relational semantic repre-\nsentation. Proceedings of ICSC .\nPradhan, S., E. H. Hovy, M. P. Mar-\ncus, M. Palmer, L. A. Ramshaw,\nand R. M. Weischedel. 2007b.\nOntonotes: a uni\ufb01ed relational se-\nmantic representation. Int. J. Seman-\ntic Computing , 1(4):405\u2013419.\nPradhan, S., X. Luo, M. Recasens,\nE. H. Hovy, V . Ng, and M. Strube.\n2014. Scoring coreference partitions\nof predicted mentions: A reference\nimplementation. ACL.Pradhan, S., A. Moschitti, N. Xue, H. T.\nNg, A. Bj \u00a8orkelund, O. Uryupina,\nY . Zhang, and Z. Zhong. 2013. To-\nwards robust linguistic analysis us-\ning OntoNotes. CoNLL .\nPradhan, S., A. Moschitti, N. Xue,\nO. Uryupina, and Y . Zhang. 2012a.\nCoNLL-2012 shared task: Model-\ning multilingual unrestricted coref-\nerence in OntoNotes. CoNLL .\nPradhan, S., A. Moschitti, N. Xue,\nO. Uryupina, and Y . Zhang. 2012b.\nConll-2012 shared task: Model-\ning multilingual unrestricted coref-\nerence in OntoNotes. CoNLL .\nPradhan, S., L. Ramshaw, M. P. Mar-\ncus, M. Palmer, R. Weischedel, and\nN. Xue. 2011. CoNLL-2011 shared\ntask: Modeling unrestricted corefer-\nence in OntoNotes. CoNLL .\nPradhan, S., L. Ramshaw, R. Wei-\nschedel, J. MacBride, and L. Mic-\nciulla. 2007c. Unrestricted corefer-\nence: Identifying entities and events\nin OntoNotes. Proceedings of\nICSC 2007 .\nPradhan, S., W. Ward, K. Hacioglu,\nJ. H. Martin, and D. Jurafsky. 2005.\nSemantic role labeling using differ-\nent syntactic views. ACL.\nPrasad, A., P. Hase, X. Zhou, and\nM. Bansal. 2023. GrIPS: Gradient-\nfree, edit-based instruction search\nfor prompting large language mod-\nels.EACL .\nPrasad, R., N. Dinesh, A. Lee, E. Milt-\nsakaki, L. Robaldo, A. K. Joshi, and\nB. L. Webber. 2008. The Penn Dis-\ncourse TreeBank 2.0. LREC .\nPrasad, R., B. L. Webber, and A. Joshi.\n2014. Re\ufb02ections on the Penn Dis-\ncourse Treebank, comparable cor-\npora, and complementary annota-\ntion. Computational Linguistics ,\n40(4):921\u2013950.\nPrates, M. O. R., P. H. Avelar, and L. C.\nLamb. 2019. Assessing gender bias\nin machine translation: a case study\nwith Google Translate. Neural Com-\nputing and Applications , 32:6363\u2013\n6381.\nPrice, P. J., W. Fisher, J. Bern-\nstein, and D. Pallet. 1988. The\nDARPA 1000-word resource man-\nagement database for continuous\nspeech recognition. ICASSP .\nPrince, E. 1981. Toward a taxonomy of\ngiven-new information. In P. Cole,\ned., Radical Pragmatics , 223\u2013255.\nAcademic Press.\nPropp, V . 1968. Morphology of the\nFolktale , 2nd edition. University of\nTexas Press. Original Russian 1928.\nTranslated by Laurence Scott.\nPryzant, R., D. Iter, J. Li, Y . Lee,\nC. Zhu, and M. Zeng. 2023. Au-\ntomatic prompt optimization with Bibliography 575\n\u201cgradient descent\u201d and beam search.\nEMNLP .\nPundak, G. and T. N. Sainath. 2016.\nLower frame rate neural network\nacoustic models. INTERSPEECH .\nPustejovsky, J. 1991. The generative\nlexicon. Computational Linguistics ,\n17(4).\nPustejovsky, J., P. Hanks, R. Saur \u00b4\u0131,\nA. See, R. Gaizauskas, A. Setzer,\nD. Radev, B. Sundheim, D. S. Day,\nL. Ferro, and M. Lazo. 2003. The\nTIMEBANK corpus. Proceedings\nof Corpus Linguistics 2003 Confer-\nence. UCREL Technical Paper num-\nber 16.\nPustejovsky, J., R. Ingria,\nR. Saur \u00b4\u0131, J. Casta \u02dcno, J. Littman,\nR. Gaizauskas, A. Setzer, G. Katz,\nand I. Mani. 2005. The Speci\ufb01ca-\ntion Language TimeML , chapter 27.\nOxford.\nQin, L., Z. Zhang, and H. Zhao. 2016.\nA stacking gated neural architecture\nfor implicit discourse relation classi-\n\ufb01cation. EMNLP .\nQin, L., Z. Zhang, H. Zhao, Z. Hu,\nand E. Xing. 2017. Adversarial\nconnective-exploiting networks for\nimplicit discourse relation classi\ufb01ca-\ntion. ACL.\nRadford, A., J. Wu, R. Child, D. Luan,\nD. Amodei, and I. Sutskever. 2019.\nLanguage models are unsupervised\nmultitask learners. OpenAI tech re-\nport.\nRafailov, R., A. Sharma, E. Mitchell,\nS. Ermon, C. D. Manning, and\nC. Finn. 2023. Direct preference op-\ntimization: Your language model is\nsecretly a reward model. NeurIPS .\nRaffel, C., N. Shazeer, A. Roberts,\nK. Lee, S. Narang, M. Matena,\nY . Zhou, W. Li, and P. J. Liu.\n2020. Exploring the limits of trans-\nfer learning with a uni\ufb01ed text-to-\ntext transformer. JMLR , 21(140):1\u2013\n67.\nRaghunathan, K., H. Lee, S. Rangara-\njan, N. Chambers, M. Surdeanu,\nD. Jurafsky, and C. D. Manning.\n2010. A multi-pass sieve for coref-\nerence resolution. EMNLP .\nRahman, A. and V . Ng. 2009. Super-\nvised models for coreference resolu-\ntion. EMNLP .\nRahman, A. and V . Ng. 2012. Resolv-\ning complex cases of de\ufb01nite pro-\nnouns: the Winograd Schema chal-\nlenge. EMNLP .\nRajpurkar, P., R. Jia, and P. Liang.\n2018. Know what you don\u2019t\nknow: Unanswerable questions for\nSQuAD. ACL.\nRajpurkar, P., J. Zhang, K. Lopyrev, and\nP. Liang. 2016. SQuAD: 100,000+questions for machine comprehen-\nsion of text. EMNLP .\nRam, O., Y . Levine, I. Dalmedigos,\nD. Muhlgay, A. Shashua, K. Leyton-\nBrown, and Y . Shoham. 2023.\nIn-context retrieval-augmented lan-\nguage models. ArXiv preprint.\nRamshaw, L. A. and M. P. Mar-\ncus. 1995. Text chunking using\ntransformation-based learning. Pro-\nceedings of the 3rd Annual Work-\nshop on Very Large Corpora .\nRashkin, H., E. Bell, Y . Choi, and\nS. V olkova. 2017. Multilingual con-\nnotation frames: A case study on\nsocial media for targeted sentiment\nanalysis and forecast. ACL.\nRashkin, H., S. Singh, and Y . Choi.\n2016. Connotation frames: A data-\ndriven investigation. ACL.\nRashkin, H., E. M. Smith, M. Li,\nand Y .-L. Boureau. 2019. Towards\nempathetic open-domain conversa-\ntion models: A new benchmark and\ndataset. ACL.\nRatinov, L. and D. Roth. 2012.\nLearning-based multi-sieve co-\nreference resolution with knowl-\nedge. EMNLP .\nRatnaparkhi, A. 1996. A maxi-\nmum entropy part-of-speech tagger.\nEMNLP .\nRatnaparkhi, A. 1997. A linear ob-\nserved time statistical parser based\non maximum entropy models.\nEMNLP .\nRawls, J. 2001. Justice as fairness:\nA restatement . Harvard University\nPress.\nRecasens, M. and E. H. Hovy. 2011.\nBLANC: Implementing the Rand\nindex for coreference evaluation.\nNatural Language Engineering ,\n17(4):485\u2013510.\nRecasens, M., E. H. Hovy, and M. A.\nMart \u00b4\u0131. 2011. Identity, non-identity,\nand near-identity: Addressing the\ncomplexity of coreference. Lingua ,\n121(6):1138\u20131152.\nRecasens, M. and M. A. Mart \u00b4\u0131. 2010.\nAnCora-CO: Coreferentially anno-\ntated corpora for Spanish and Cata-\nlan. Language Resources and Eval-\nuation , 44(4):315\u2013345.\nReed, C., R. Mochales Palau, G. Rowe,\nand M.-F. Moens. 2008. Lan-\nguage resources for studying argu-\nment. LREC .\nReeves, B. and C. Nass. 1996. The\nMedia Equation: How People Treat\nComputers, Television, and New Me-\ndia Like Real People and Places .\nCambridge University Press.\nRehder, B., M. E. Schreiner, M. B. W.\nWolfe, D. Laham, T. K. Landauer,\nand W. Kintsch. 1998. Using LatentSemantic Analysis to assess knowl-\nedge: Some technical considera-\ntions. Discourse Processes , 25(2-\n3):337\u2013354.\nRei, R., C. Stewart, A. C. Farinha, and\nA. Lavie. 2020. COMET: A neu-\nral framework for MT evaluation.\nEMNLP .\nReichenbach, H. 1947. Elements of\nSymbolic Logic . Macmillan, New\nYork.\nReichman, R. 1985. Getting Computers\nto Talk Like You and Me . MIT Press.\nResnik, P. 1993. Semantic classes and\nsyntactic ambiguity. HLT.\nResnik, P. 1996. Selectional con-\nstraints: An information-theoretic\nmodel and its computational realiza-\ntion. Cognition , 61:127\u2013159.\nReynolds, L. and K. McDonell. 2021.\nPrompt programming for large lan-\nguage models: Beyond the few-shot\nparadigm. CHI 2021 .\nRiedel, S., L. Yao, and A. McCallum.\n2010. Modeling relations and their\nmentions without labeled text. In\nMachine Learning and Knowledge\nDiscovery in Databases , 148\u2013163.\nSpringer.\nRiedel, S., L. Yao, A. McCallum, and\nB. M. Marlin. 2013. Relation extrac-\ntion with matrix factorization and\nuniversal schemas. NAACL HLT .\nRiloff, E. 1993. Automatically con-\nstructing a dictionary for informa-\ntion extraction tasks. AAAI .\nRiloff, E. 1996. Automatically gen-\nerating extraction patterns from un-\ntagged text. AAAI .\nRiloff, E. and R. Jones. 1999. Learning\ndictionaries for information extrac-\ntion by multi-level bootstrapping.\nAAAI .\nRiloff, E. and M. Schmelzenbach. 1998.\nAn empirical approach to conceptual\ncase frame acquisition. Proceedings\nof the Sixth Workshop on Very Large\nCorpora .\nRiloff, E. and J. Shepherd. 1997. A\ncorpus-based approach for building\nsemantic lexicons. EMNLP .\nRiloff, E. and M. Thelen. 2000. A rule-\nbased question answering system\nfor reading comprehension tests.\nANLP/NAACL workshop on reading\ncomprehension tests .\nRiloff, E. and J. Wiebe. 2003. Learn-\ning extraction patterns for subjective\nexpressions. EMNLP .\nRitter, A., C. Cherry, and B. Dolan.\n2010a. Unsupervised modeling of\ntwitter conversations. NAACL HLT .\nRitter, A., O. Etzioni, and Mausam.\n2010b. A latent dirichlet allocation\nmethod for selectional preferences.\nACL. 576 Bibliography\nRitter, A., L. Zettlemoyer, Mausam, and\nO. Etzioni. 2013. Modeling miss-\ning data in distant supervision for in-\nformation extraction. TACL , 1:367\u2013\n378.\nRoberts, A., C. Raffel, and N. Shazeer.\n2020. How much knowledge can\nyou pack into the parameters of a\nlanguage model? EMNLP .\nRobertson, S., S. Walker, S. Jones,\nM. M. Hancock-Beaulieu, and\nM. Gatford. 1995. Okapi at TREC-3.\nOverview of the Third Text REtrieval\nConference (TREC-3) .\nRobinson, T. and F. Fallside. 1991.\nA recurrent error propagation net-\nwork speech recognition system.\nComputer Speech & Language ,\n5(3):259\u2013274.\nRobinson, T., M. Hochberg, and S. Re-\nnals. 1996. The use of recurrent neu-\nral networks in continuous speech\nrecognition. In C.-H. Lee, F. K.\nSoong, and K. K. Paliwal, eds, Au-\ntomatic speech and speaker recogni-\ntion, 233\u2013258. Springer.\nRogers, A., M. Gardner, and I. Au-\ngenstein. 2023. QA dataset explo-\nsion: A taxonomy of NLP resources\nfor question answering and reading\ncomprehension. ACM Computing\nSurveys , 55(10):1\u201345.\nRohde, D. L. T., L. M. Gonnerman, and\nD. C. Plaut. 2006. An improved\nmodel of semantic similarity based\non lexical co-occurrence. CACM ,\n8:627\u2013633.\nRoller, S., E. Dinan, N. Goyal, D. Ju,\nM. Williamson, Y . Liu, J. Xu,\nM. Ott, E. M. Smith, Y .-L. Boureau,\nand J. Weston. 2021. Recipes for\nbuilding an open-domain chatbot.\nEACL .\nRooth, M., S. Riezler, D. Prescher,\nG. Carroll, and F. Beil. 1999. Induc-\ning a semantically annotated lexicon\nvia EM-based clustering. ACL.\nRosenblatt, F. 1958. The percep-\ntron: A probabilistic model for in-\nformation storage and organization\nin the brain. Psychological review ,\n65(6):386\u2013408.\nRosenfeld, R. 1992. Adaptive Statis-\ntical Language Modeling: A Maxi-\nmum Entropy Approach . Ph.D. the-\nsis, Carnegie Mellon University.\nRosenfeld, R. 1996. A maximum en-\ntropy approach to adaptive statisti-\ncal language modeling. Computer\nSpeech and Language , 10:187\u2013228.\nRosenthal, S. and K. McKeown. 2017.\nDetecting in\ufb02uencers in multiple on-\nline genres. ACM Transactions on\nInternet Technology (TOIT) , 17(2).\nRothe, S., S. Ebert, and H. Sch \u00a8utze.\n2016. Ultradense Word Embed-\ndings by Orthogonal Transforma-\ntion. NAACL HLT .Roy, N., J. Pineau, and S. Thrun. 2000.\nSpoken dialogue management using\nprobabilistic reasoning. ACL.\nRudinger, R., J. Naradowsky,\nB. Leonard, and B. Van Durme.\n2018. Gender bias in coreference\nresolution. NAACL HLT .\nRumelhart, D. E., G. E. Hinton, and\nR. J. Williams. 1986. Learning in-\nternal representations by error prop-\nagation. In D. E. Rumelhart and\nJ. L. McClelland, eds, Parallel Dis-\ntributed Processing , volume 2, 318\u2013\n362. MIT Press.\nRumelhart, D. E. and J. L. McClelland.\n1986a. On learning the past tense of\nEnglish verbs. In D. E. Rumelhart\nand J. L. McClelland, eds, Parallel\nDistributed Processing , volume 2,\n216\u2013271. MIT Press.\nRumelhart, D. E. and J. L. McClelland,\neds. 1986b. Parallel Distributed\nProcessing . MIT Press.\nRumelhart, D. E. and A. A. Abraham-\nson. 1973. A model for analogi-\ncal reasoning. Cognitive Psychol-\nogy, 5(1):1\u201328.\nRumelhart, D. E. and J. L. McClelland,\neds. 1986c. Parallel Distributed\nProcessing: Explorations in the Mi-\ncrostructure of Cognition , volume\n1:Foundations . MIT Press.\nRuppenhofer, J., M. Ellsworth, M. R. L.\nPetruck, C. R. Johnson, C. F. Baker,\nand J. Scheffczyk. 2016. FrameNet\nII: Extended theory and practice.\nRuppenhofer, J., C. Sporleder,\nR. Morante, C. F. Baker, and\nM. Palmer. 2010. Semeval-2010\ntask 10: Linking events and their\nparticipants in discourse. 5th In-\nternational Workshop on Semantic\nEvaluation .\nRussell, J. A. 1980. A circum-\nplex model of affect. Journal of\npersonality and social psychology ,\n39(6):1161\u20131178.\nRussell, S. and P. Norvig. 2002. Ar-\nti\ufb01cial Intelligence: A Modern Ap-\nproach , 2nd edition. Prentice Hall.\nRutherford, A. and N. Xue. 2015. Im-\nproving the inference of implicit dis-\ncourse relations via classifying ex-\nplicit discourse connectives. NAACL\nHLT.\nSachan, D. S., M. Lewis, D. Yo-\ngatama, L. Zettlemoyer, J. Pineau,\nand M. Zaheer. 2023. Questions are\nall you need to train a dense passage\nretriever. TACL , 11:600\u2013616.\nSacks, H., E. A. Schegloff, and G. Jef-\nferson. 1974. A simplest system-\natics for the organization of turn-\ntaking for conversation. Language ,\n50(4):696\u2013735.Sag, I. A. and M. Y . Liberman. 1975.\nThe intonational disambiguation of\nindirect speech acts. In CLS-75 ,\n487\u2013498. University of Chicago.\nSagae, K. 2009. Analysis of dis-\ncourse structure with syntactic de-\npendencies and data-driven shift-\nreduce parsing. IWPT-09 .\nSagawa, S., P. W. Koh, T. B.\nHashimoto, and P. Liang. 2020. Dis-\ntributionally robust neural networks\nfor group shifts: On the importance\nof regularization for worst-case gen-\neralization. ICLR .\nSagisaka, Y . 1988. Speech synthe-\nsis by rule using an optimal selec-\ntion of non-uniform synthesis units.\nICASSP .\nSagisaka, Y ., N. Kaiki, N. Iwahashi,\nand K. Mimura. 1992. Atr \u2013 n-talk\nspeech synthesis system. ICSLP .\nSahami, M., S. T. Dumais, D. Heck-\nerman, and E. Horvitz. 1998. A\nBayesian approach to \ufb01ltering junk\ne-mail. AAAI Workshop on Learning\nfor Text Categorization .\nSakoe, H. and S. Chiba. 1971. A\ndynamic programming approach to\ncontinuous speech recognition. Pro-\nceedings of the Seventh Interna-\ntional Congress on Acoustics , vol-\nume 3. Akad \u00b4emiai Kiad \u00b4o.\nSakoe, H. and S. Chiba. 1984. Dy-\nnamic programming algorithm opti-\nmization for spoken word recogni-\ntion. IEEE Transactions on ASSP ,\nASSP-26(1):43\u201349.\nSalomaa, A. 1969. Probabilistic and\nweighted grammars. Information\nand Control , 15:529\u2013544.\nSalton, G. 1971. The SMART Re-\ntrieval System: Experiments in Au-\ntomatic Document Processing . Pren-\ntice Hall.\nSalvetti, F., J. B. Lowe, and J. H. Mar-\ntin. 2016. A tangled web: The faint\nsignals of deception in text - boul-\nder lies and truth corpus (BLT-C).\nLREC .\nSampson, G. 1987. Alternative gram-\nmatical coding systems. In R. Gar-\nside, G. Leech, and G. Sampson,\neds, The Computational Analysis of\nEnglish , 165\u2013183. Longman.\nSankoff, D. and W. Labov. 1979. On the\nuses of variable rules. Language in\nsociety , 8(2-3):189\u2013222.\nSap, M., D. Card, S. Gabriel, Y . Choi,\nand N. A. Smith. 2019. The risk of\nracial bias in hate speech detection.\nACL.\nSap, M., M. C. Prasettio, A. Holtzman,\nH. Rashkin, and Y . Choi. 2017. Con-\nnotation frames of power and agency\nin modern \ufb01lms. EMNLP . Bibliography 577\nSaur\u00b4\u0131, R., J. Littman, B. Knippen,\nR. Gaizauskas, A. Setzer, and\nJ. Pustejovsky. 2006. TimeML an-\nnotation guidelines version 1.2.1.\nManuscript.\nScha, R. and L. Polanyi. 1988. An\naugmented context free grammar for\ndiscourse. COLING .\nSchank, R. C. and R. P. Abelson. 1975.\nScripts, plans, and knowledge. Pro-\nceedings of IJCAI-75 .\nSchank, R. C. and R. P. Abelson. 1977.\nScripts, Plans, Goals and Under-\nstanding . Lawrence Erlbaum.\nSchegloff, E. A. 1968. Sequencing in\nconversational openings. American\nAnthropologist , 70:1075\u20131095.\nScherer, K. R. 2000. Psychological\nmodels of emotion. In J. C. Borod,\ned., The neuropsychology of emo-\ntion, 137\u2013162. Oxford.\nSchiebinger, L. 2013. Machine\ntranslation: Analyzing gender.\nhttp://genderedinnovations.\nstanford.edu/case-studies/\nnlp.html#tabs-2 .\nSchiebinger, L. 2014. Scienti\ufb01c re-\nsearch must take gender into ac-\ncount. Nature , 507(7490):9.\nSchluter, N. 2018. The word analogy\ntesting caveat. NAACL HLT .\nSchone, P. and D. Jurafsky. 2000.\nKnowlege-free induction of mor-\nphology using latent semantic anal-\nysis. CoNLL .\nSchone, P. and D. Jurafsky. 2001a. Is\nknowledge-free induction of multi-\nword unit dictionary headwords a\nsolved problem? EMNLP .\nSchone, P. and D. Jurafsky. 2001b.\nKnowledge-free induction of in\ufb02ec-\ntional morphologies. NAACL .\nSchuster, M. and K. Nakajima. 2012.\nJapanese and Korean voice search.\nICASSP .\nSchuster, M. and K. K. Paliwal. 1997.\nBidirectional recurrent neural net-\nworks. IEEE Transactions on Signal\nProcessing , 45:2673\u20132681.\nSch\u00a8utze, H. 1992a. Context space.\nAAAI Fall Symposium on Proba-\nbilistic Approaches to Natural Lan-\nguage .\nSch\u00a8utze, H. 1992b. Dimensions of\nmeaning. Proceedings of Supercom-\nputing \u201992 . IEEE Press.\nSch\u00a8utze, H. 1997. Ambiguity Resolu-\ntion in Language Learning \u2013 Com-\nputational and Cognitive Models .\nCSLI, Stanford, CA.\nSch\u00a8utze, H., D. A. Hull, and J. Peder-\nsen. 1995. A comparison of clas-\nsi\ufb01ers and document representations\nfor the routing problem. SIGIR-95 .Sch\u00a8utze, H. and J. Pedersen. 1993. A\nvector model for syntagmatic and\nparadigmatic relatedness. 9th An-\nnual Conference of the UW Centre\nfor the New OED and Text Research .\nSch\u00a8utze, H. and Y . Singer. 1994. Part-\nof-speech tagging using a variable\nmemory Markov model. ACL.\nSchwartz, H. A., J. C. Eichstaedt,\nM. L. Kern, L. Dziurzynski, S. M.\nRamones, M. Agrawal, A. Shah,\nM. Kosinski, D. Stillwell, M. E. P.\nSeligman, and L. H. Ungar. 2013.\nPersonality, gender, and age in the\nlanguage of social media: The open-\nvocabulary approach. PloS one ,\n8(9):e73791.\nSchwenk, H. 2007. Continuous space\nlanguage models. Computer Speech\n& Language , 21(3):492\u2013518.\nSchwenk, H. 2018. Filtering and min-\ning parallel data in a joint multilin-\ngual space. ACL.\nSchwenk, H., D. Dechelotte, and J.-L.\nGauvain. 2006. Continuous space\nlanguage models for statistical ma-\nchine translation. COLING/ACL .\nSchwenk, H., G. Wenzek, S. Edunov,\nE. Grave, A. Joulin, and A. Fan.\n2021. CCMatrix: Mining billions\nof high-quality parallel sentences on\nthe web. ACL.\nS\u00b4eaghdha, D. O. 2010. Latent vari-\nable models of selectional prefer-\nence. ACL.\nSeddah, D., R. Tsarfaty, S. K \u00a8ubler,\nM. Candito, J. D. Choi, R. Farkas,\nJ. Foster, I. Goenaga, K. Gojenola,\nY . Goldberg, S. Green, N. Habash,\nM. Kuhlmann, W. Maier, J. Nivre,\nA. Przepi \u00b4orkowski, R. Roth,\nW. Seeker, Y . Versley, V . Vincze,\nM. Woli \u00b4nski, A. Wr \u00b4oblewska, and\nE. Villemonte de la Cl \u00b4ergerie.\n2013. Overview of the SPMRL\n2013 shared task: cross-framework\nevaluation of parsing morpholog-\nically rich languages. 4th Work-\nshop on Statistical Parsing of\nMorphologically-Rich Languages .\nSee, A., S. Roller, D. Kiela, and\nJ. Weston. 2019. What makes a\ngood conversation? how control-\nlable attributes affect human judg-\nments. NAACL HLT .\nSekine, S. and M. Collins. 1997.\nThe evalb software. http:\n//cs.nyu.edu/cs/projects/\nproteus/evalb .\nSellam, T., D. Das, and A. Parikh. 2020.\nBLEURT: Learning robust metrics\nfor text generation. ACL.\nSennrich, R., B. Haddow, and A. Birch.\n2016. Neural machine translation of\nrare words with subword units. ACL.\nSeo, M., A. Kembhavi, A. Farhadi, and\nH. Hajishirzi. 2017. Bidirectionalattention \ufb02ow for machine compre-\nhension. ICLR .\nShannon, C. E. 1948. A mathematical\ntheory of communication. Bell Sys-\ntem Technical Journal , 27(3):379\u2013\n423. Continued in the following vol-\nume.\nShannon, C. E. 1951. Prediction and en-\ntropy of printed English. Bell System\nTechnical Journal , 30:50\u201364.\nSheil, B. A. 1976. Observations on con-\ntext free parsing. SMIL: Statistical\nMethods in Linguistics , 1:71\u2013109.\nShen, J., R. Pang, R. J. Weiss,\nM. Schuster, N. Jaitly, Z. Yang,\nZ. Chen, Y . Zhang, Y . Wang,\nR. Skerry-Ryan, R. A. Saurous,\nY . Agiomyrgiannakis, and Y . Wu.\n2018. Natural TTS synthesis by con-\nditioning WaveNet on mel spectro-\ngram predictions. ICASSP .\nSheng, E., K.-W. Chang, P. Natarajan,\nand N. Peng. 2019. The woman\nworked as a babysitter: On biases in\nlanguage generation. EMNLP .\nShi, P. and J. Lin. 2019. Simple BERT\nmodels for relation extraction and\nsemantic role labeling. ArXiv.\nShi, W., S. Min, M. Yasunaga, M. Seo,\nR. James, M. Lewis, L. Zettlemoyer,\nand W.-t. Yih. 2023. REPLUG:\nRetrieval-augmented black-box lan-\nguage models. ArXiv preprint.\nShriberg, E., R. Bates, P. Taylor,\nA. Stolcke, D. Jurafsky, K. Ries,\nN. Coccaro, R. Martin, M. Meteer,\nand C. Van Ess-Dykema. 1998. Can\nprosody aid the automatic classi\ufb01ca-\ntion of dialog acts in conversational\nspeech? Language and Speech (Spe-\ncial Issue on Prosody and Conversa-\ntion) , 41(3-4):439\u2013487.\nSidner, C. L. 1979. Towards a compu-\ntational theory of de\ufb01nite anaphora\ncomprehension in English discourse.\nTechnical Report 537, MIT Arti\ufb01-\ncial Intelligence Laboratory, Cam-\nbridge, MA.\nSidner, C. L. 1983. Focusing in the\ncomprehension of de\ufb01nite anaphora.\nIn M. Brady and R. C. Berwick,\neds, Computational Models of Dis-\ncourse , 267\u2013330. MIT Press.\nSimmons, R. F. 1965. Answering En-\nglish questions by computer: A sur-\nvey. CACM , 8(1):53\u201370.\nSimmons, R. F. 1973. Semantic net-\nworks: Their computation and use\nfor understanding English sentences.\nIn R. C. Schank and K. M. Colby,\neds, Computer Models of Thought\nand Language , 61\u2013113. W.H. Free-\nman & Co.\nSimmons, R. F., S. Klein, and K. Mc-\nConlogue. 1964. Indexing and de-\npendency logic for answering En-\nglish questions. American Docu-\nmentation , 15(3):196\u2013204. 578 Bibliography\nSimons, G. F. and C. D. Fennig.\n2018. Ethnologue: Languages of\nthe world, 21st edition. SIL Inter-\nnational.\nSingh, S. P., D. J. Litman, M. Kearns,\nand M. A. Walker. 2002. Optimiz-\ning dialogue management with re-\ninforcement learning: Experiments\nwith the NJFun system. JAIR ,\n16:105\u2013133.\nSingh, S., F. Vargus, D. D\u2019souza,\nB. F. Karlsson, A. Mahendiran,\nW.-Y . Ko, H. Shandilya, J. Pa-\ntel, D. Mataciunas, L. O\u2019Mahony,\nM. Zhang, R. Hettiarachchi, J. Wil-\nson, M. Machado, L. S. Moura,\nD. Krzemi \u00b4nski, H. Fadaei, I. Erg \u00a8un,\nI. Okoh, A. Alaagib, O. Mudan-\nnayake, Z. Alyafeai, V . M. Chien,\nS. Ruder, S. Guthikonda, E. A.\nAlghamdi, S. Gehrmann, N. Muen-\nnighoff, M. Bartolo, J. Kreutzer,\nA. \u00a8U\u00a8Ust\u00a8un, M. Fadaee, and\nS. Hooker. 2024. Aya dataset: An\nopen-access collection for multi-\nlingual instruction tuning. ArXiv\npreprint.\nSleator, D. and D. Temperley. 1993.\nParsing English with a link gram-\nmar. IWPT-93 .\nSloan, M. C. 2010. Aristotle\u2019s Nico-\nmachean Ethics as the original lo-\ncus for the Septem Circumstantiae.\nClassical Philology , 105(3):236\u2013\n251.\nSlobin, D. I. 1996. Two ways to\ntravel. In M. Shibatani and S. A.\nThompson, eds, Grammatical Con-\nstructions: Their Form and Mean-\ning, 195\u2013220. Clarendon Press.\nSmith, V . L. and H. H. Clark. 1993. On\nthe course of answering questions.\nJournal of Memory and Language ,\n32:25\u201338.\nSmolensky, P. 1988. On the proper\ntreatment of connectionism. Behav-\nioral and brain sciences , 11(1):1\u2013\n23.\nSmolensky, P. 1990. Tensor product\nvariable binding and the representa-\ntion of symbolic structures in con-\nnectionist systems. Arti\ufb01cial intel-\nligence , 46(1-2):159\u2013216.\nSnover, M., B. Dorr, R. Schwartz,\nL. Micciulla, and J. Makhoul. 2006.\nA study of translation edit rate with\ntargeted human annotation. AMTA-\n2006 .\nSnow, R., D. Jurafsky, and A. Y . Ng.\n2005. Learning syntactic patterns\nfor automatic hypernym discovery.\nNeurIPS .\nSocher, R., J. Bauer, C. D. Man-\nning, and A. Y . Ng. 2013. Pars-\ning with compositional vector gram-\nmars. ACL.Socher, R., C. C.-Y . Lin, A. Y . Ng, and\nC. D. Manning. 2011. Parsing natu-\nral scenes and natural language with\nrecursive neural networks. ICML .\nSoderland, S., D. Fisher, J. Aseltine,\nand W. G. Lehnert. 1995. CRYS-\nTAL: Inducing a conceptual dictio-\nnary. IJCAI-95 .\nS\u00f8gaard, A. 2010. Simple semi-\nsupervised training of part-of-\nspeech taggers. ACL.\nS\u00f8gaard, A. and Y . Goldberg. 2016.\nDeep multi-task learning with low\nlevel tasks supervised at lower lay-\ners.ACL.\nS\u00f8gaard, A., A. Johannsen, B. Plank,\nD. Hovy, and H. M. Alonso. 2014.\nWhat\u2019s in a p-value in NLP? CoNLL .\nSoldaini, L., R. Kinney, A. Bha-\ngia, D. Schwenk, D. Atkinson,\nR. Authur, B. Bogin, K. Chandu,\nJ. Dumas, Y . Elazar, V . Hofmann,\nA. H. Jha, S. Kumar, L. Lucy,\nX. Lyu, N. Lambert, I. Magnus-\nson, J. Morrison, N. Muennighoff,\nA. Naik, C. Nam, M. E. Pe-\nters, A. Ravichander, K. Richardson,\nZ. Shen, E. Strubell, N. Subramani,\nO. Tafjord, P. Walsh, L. Zettlemoyer,\nN. A. Smith, H. Hajishirzi, I. Belt-\nagy, D. Groeneveld, J. Dodge, and\nK. Lo. 2024. Dolma: An open cor-\npus of three trillion tokens for lan-\nguage model pretraining research.\nArXiv preprint.\nSolorio, T., E. Blair, S. Maharjan,\nS. Bethard, M. Diab, M. Ghoneim,\nA. Hawwari, F. AlGhamdi,\nJ. Hirschberg, A. Chang, and\nP. Fung. 2014. Overview for the\n\ufb01rst shared task on language iden-\nti\ufb01cation in code-switched data.\nWorkshop on Computational Ap-\nproaches to Code Switching .\nSomasundaran, S., J. Burstein, and\nM. Chodorow. 2014. Lexical chain-\ning for measuring discourse coher-\nence quality in test-taker essays.\nCOLING .\nSoon, W. M., H. T. Ng, and D. C. Y .\nLim. 2001. A machine learning ap-\nproach to coreference resolution of\nnoun phrases. Computational Lin-\nguistics , 27(4):521\u2013544.\nSoricut, R. and D. Marcu. 2003. Sen-\ntence level discourse parsing using\nsyntactic and lexical information.\nHLT-NAACL .\nSoricut, R. and D. Marcu. 2006.\nDiscourse generation using utility-\ntrained coherence models. COL-\nING/ACL .\nSorokin, D. and I. Gurevych. 2018.\nMixing context granularities for im-\nproved entity linking on question\nanswering data across entity cate-\ngories. *SEM .Sparck Jones, K. 1972. A statistical in-\nterpretation of term speci\ufb01city and\nits application in retrieval. Journal\nof Documentation , 28(1):11\u201321.\nSparck Jones, K. 1986. Synonymy and\nSemantic Classi\ufb01cation . Edinburgh\nUniversity Press, Edinburgh. Repub-\nlication of 1964 PhD Thesis.\nSporleder, C. and A. Lascarides. 2005.\nExploiting linguistic cues to classify\nrhetorical relations. RANLP-05 .\nSporleder, C. and M. Lapata. 2005. Dis-\ncourse chunking and its application\nto sentence compression. EMNLP .\nSproat, R., A. W. Black, S. F.\nChen, S. Kumar, M. Ostendorf, and\nC. Richards. 2001. Normalization\nof non-standard words. Computer\nSpeech & Language , 15(3):287\u2013\n333.\nSproat, R. and K. Gorman. 2018. A\nbrief summary of the Kaggle text\nnormalization challenge.\nSrivastava, N., G. E. Hinton,\nA. Krizhevsky, I. Sutskever, and\nR. R. Salakhutdinov. 2014. Dropout:\na simple way to prevent neural net-\nworks from over\ufb01tting. JMLR ,\n15(1):1929\u20131958.\nStab, C. and I. Gurevych. 2014a. Anno-\ntating argument components and re-\nlations in persuasive essays. COL-\nING.\nStab, C. and I. Gurevych. 2014b. Identi-\nfying argumentative discourse struc-\ntures in persuasive essays. EMNLP .\nStab, C. and I. Gurevych. 2017. Parsing\nargumentation structures in persua-\nsive essays. Computational Linguis-\ntics, 43(3):619\u2013659.\nStalnaker, R. C. 1978. Assertion. In\nP. Cole, ed., Pragmatics: Syntax and\nSemantics Volume 9 , 315\u2013332. Aca-\ndemic Press.\nStamatatos, E. 2009. A survey of mod-\nern authorship attribution methods.\nJASIST , 60(3):538\u2013556.\nStanovsky, G., N. A. Smith, and\nL. Zettlemoyer. 2019. Evaluating\ngender bias in machine translation.\nACL.\nStede, M. 2011. Discourse processing .\nMorgan & Claypool.\nStede, M. and J. Schneider. 2018. Argu-\nmentation Mining . Morgan & Clay-\npool.\nStern, M., J. Andreas, and D. Klein.\n2017. A minimal span-based neural\nconstituency parser. ACL.\nStevens, K. N., S. Kasowski, and G. M.\nFant. 1953. An electrical analog of\nthe vocal tract. JASA , 25(4):734\u2013\n742.\nStevens, S. S. and J. V olkmann. 1940.\nThe relation of pitch to frequency: A\nrevised scale. The American Journal\nof Psychology , 53(3):329\u2013353. Bibliography 579\nStevens, S. S., J. V olkmann, and E. B.\nNewman. 1937. A scale for the mea-\nsurement of the psychological mag-\nnitude pitch. JASA , 8:185\u2013190.\nStifelman, L. J., B. Arons,\nC. Schmandt, and E. A. Hulteen.\n1993. V oiceNotes: A speech inter-\nface for a hand-held voice notetaker.\nINTERCHI 1993 .\nStolcke, A. 1998. Entropy-based prun-\ning of backoff language models.\nProc. DARPA Broadcast News Tran-\nscription and Understanding Work-\nshop .\nStolcke, A. 2002. SRILM \u2013 an exten-\nsible language modeling toolkit. IC-\nSLP.\nStolcke, A., Y . Konig, and M. Wein-\ntraub. 1997. Explicit word error min-\nimization in N-best list rescoring.\nEUROSPEECH , volume 1.\nStolcke, A., K. Ries, N. Coccaro,\nE. Shriberg, R. Bates, D. Jurafsky,\nP. Taylor, R. Martin, M. Meteer,\nand C. Van Ess-Dykema. 2000. Di-\nalogue act modeling for automatic\ntagging and recognition of conversa-\ntional speech. Computational Lin-\nguistics , 26(3):339\u2013371.\nStolz, W. S., P. H. Tannenbaum, and\nF. V . Carstensen. 1965. A stochastic\napproach to the grammatical coding\nof English. CACM , 8(6):399\u2013405.\nStone, P., D. Dunphry, M. Smith, and\nD. Ogilvie. 1966. The General In-\nquirer: A Computer Approach to\nContent Analysis . MIT Press.\nStr\u00a8otgen, J. and M. Gertz. 2013. Mul-\ntilingual and cross-domain temporal\ntagging. Language Resources and\nEvaluation , 47(2):269\u2013298.\nStrube, M. and U. Hahn. 1996. Func-\ntional centering. ACL.\nStrubell, E., A. Ganesh, and A. McCal-\nlum. 2019. Energy and policy con-\nsiderations for deep learning in NLP.\nACL.\nSu, Y ., H. Sun, B. Sadler, M. Srivatsa,\nI. G\u00a8ur, Z. Yan, and X. Yan. 2016. On\ngenerating characteristic-rich ques-\ntion sets for QA evaluation. EMNLP .\nSubba, R. and B. Di Eugenio. 2009. An\neffective discourse parser that uses\nrich linguistic information. NAACL\nHLT.\nSukhbaatar, S., A. Szlam, J. Weston,\nand R. Fergus. 2015. End-to-end\nmemory networks. NeurIPS .\nSundheim, B., ed. 1991. Proceedings of\nMUC-3 .\nSundheim, B., ed. 1992. Proceedings of\nMUC-4 .\nSundheim, B., ed. 1993. Proceedings of\nMUC-5 . Baltimore, MD.\nSundheim, B., ed. 1995. Proceedings of\nMUC-6 .Surdeanu, M. 2013. Overview of the\nTAC2013 Knowledge Base Popula-\ntion evaluation: English slot \ufb01lling\nand temporal slot \ufb01lling. TAC-13 .\nSurdeanu, M., S. Harabagiu,\nJ. Williams, and P. Aarseth. 2003.\nUsing predicate-argument structures\nfor information extraction. ACL.\nSurdeanu, M., T. Hicks, and M. A.\nValenzuela-Escarcega. 2015. Two\npractical rhetorical structure theory\nparsers. NAACL HLT .\nSurdeanu, M., R. Johansson, A. Mey-\ners, L. M `arquez, and J. Nivre. 2008.\nThe CoNLL 2008 shared task on\njoint parsing of syntactic and seman-\ntic dependencies. CoNLL .\nSutskever, I., O. Vinyals, and Q. V . Le.\n2014. Sequence to sequence learn-\ning with neural networks. NeurIPS .\nSuzgun, M., L. Melas-Kyriazi, and\nD. Jurafsky. 2023a. Follow the wis-\ndom of the crowd: Effective text\ngeneration via minimum Bayes risk\ndecoding. Findings of ACL 2023 .\nSuzgun, M., N. Scales, N. Sch \u00a8arli,\nS. Gehrmann, Y . Tay, H. W. Chung,\nA. Chowdhery, Q. Le, E. Chi,\nD. Zhou, and J. Wei. 2023b.\nChallenging BIG-bench tasks and\nwhether chain-of-thought can solve\nthem. ACL Findings .\nSwerts, M., D. J. Litman, and J. Hirsch-\nberg. 2000. Corrections in spoken\ndialogue systems. ICSLP .\nSwier, R. and S. Stevenson. 2004. Un-\nsupervised semantic role labelling.\nEMNLP .\nSwitzer, P. 1965. Vector images in doc-\nument retrieval. Statistical Associa-\ntion Methods For Mechanized Docu-\nmentation. Symposium Proceedings.\nWashington, D.C., USA, March 17,\n1964 .https://nvlpubs.nist.\ngov/nistpubs/Legacy/MP/\nnbsmiscellaneouspub269.pdf .\nSyrdal, A. K., C. W. Wightman,\nA. Conkie, Y . Stylianou, M. Beut-\nnagel, J. Schroeter, V . Strom, and\nK.-S. Lee. 2000. Corpus-based\ntechniques in the AT&T NEXTGEN\nsynthesis system. ICSLP .\nTalmy, L. 1985. Lexicalization patterns:\nSemantic structure in lexical forms.\nIn T. Shopen, ed., Language Typol-\nogy and Syntactic Description, Vol-\nume 3 . Cambridge University Press.\nOriginally appeared as UC Berkeley\nCognitive Science Program Report\nNo. 30, 1980.\nTalmy, L. 1991. Path to realization: A\ntypology of event con\ufb02ation. BLS-\n91.\nTan, C., V . Niculae, C. Danescu-\nNiculescu-Mizil, and L. Lee. 2016.\nWinning arguments: Interaction dy-\nnamics and persuasion strategiesin good-faith online discussions.\nWWW-16 .\nTannen, D. 1979. What\u2019s in a frame?\nSurface evidence for underlying ex-\npectations. In R. Freedle, ed., New\nDirections in Discourse Processing ,\n137\u2013181. Ablex.\nTaylor, P. 2009. Text-to-Speech Synthe-\nsis. Cambridge University Press.\nTaylor, W. L. 1953. Cloze procedure: A\nnew tool for measuring readability.\nJournalism Quarterly , 30:415\u2013433.\nTeranishi, R. and N. Umeda. 1968. Use\nof pronouncing dictionary in speech\nsynthesis experiments. 6th Interna-\ntional Congress on Acoustics .\nTesni `ere, L. 1959. \u00b4El\u00b4ements de Syntaxe\nStructurale . Librairie C. Klinck-\nsieck, Paris.\nTetreault, J. R. 2001. A corpus-based\nevaluation of centering and pronoun\nresolution. Computational Linguis-\ntics, 27(4):507\u2013520.\nTeufel, S., J. Carletta, and M. Moens.\n1999. An annotation scheme for\ndiscourse-level argumentation in re-\nsearch articles. EACL .\nTeufel, S., A. Siddharthan, and\nC. Batchelor. 2009. Towards\ndomain-independent argumenta-\ntive zoning: Evidence from chem-\nistry and computational linguistics.\nEMNLP .\nThede, S. M. and M. P. Harper. 1999. A\nsecond-order hidden Markov model\nfor part-of-speech tagging. ACL.\nThompson, B. and P. Koehn. 2019. Ve-\ncalign: Improved sentence align-\nment in linear time and space.\nEMNLP .\nThompson, K. 1968. Regular ex-\npression search algorithm. CACM ,\n11(6):419\u2013422.\nTian, Y ., V . Kulkarni, B. Perozzi,\nand S. Skiena. 2016. On the\nconvergent properties of word em-\nbedding methods. ArXiv preprint\narXiv:1605.03956.\nTibshirani, R. J. 1996. Regression\nshrinkage and selection via the lasso.\nJournal of the Royal Statistical So-\nciety. Series B (Methodological) ,\n58(1):267\u2013288.\nTimkey, W. and M. van Schijndel. 2021.\nAll bark and no bite: Rogue dimen-\nsions in transformer language mod-\nels obscure representational quality.\nEMNLP .\nTitov, I. and E. Khoddam. 2014. Unsu-\npervised induction of semantic roles\nwithin a reconstruction-error mini-\nmization framework. NAACL HLT .\nTitov, I. and A. Klementiev. 2012. A\nBayesian approach to unsupervised\nsemantic role induction. EACL . 580 Bibliography\nTomkins, S. S. 1962. Affect, imagery,\nconsciousness: Vol. I. The positive\naffects . Springer.\nToutanova, K., D. Klein, C. D. Man-\nning, and Y . Singer. 2003. Feature-\nrich part-of-speech tagging with a\ncyclic dependency network. HLT-\nNAACL .\nTrichelair, P., A. Emami, J. C. K.\nCheung, A. Trischler, K. Suleman,\nand F. Diaz. 2018. On the eval-\nuation of common-sense reasoning\nin natural language understanding.\nNeurIPS 2018 Workshop on Cri-\ntiquing and Correcting Trends in\nMachine Learning .\nTrnka, K., D. Yarrington, J. McCaw,\nK. F. McCoy, and C. Pennington.\n2007. The effects of word pre-\ndiction on communication rate for\nAAC. NAACL-HLT .\nTurian, J. P., L. Shen, and I. D. Mela-\nmed. 2003. Evaluation of machine\ntranslation and its evaluation. Pro-\nceedings of MT Summit IX .\nTurian, J., L. Ratinov, and Y . Bengio.\n2010. Word representations: a sim-\nple and general method for semi-\nsupervised learning. ACL.\nTurney, P. D. 2002. Thumbs up or\nthumbs down? Semantic orienta-\ntion applied to unsupervised classi-\n\ufb01cation of reviews. ACL.\nTurney, P. D. and M. Littman. 2003.\nMeasuring praise and criticism: In-\nference of semantic orientation from\nassociation. ACM Transactions\non Information Systems (TOIS) ,\n21:315\u2013346.\nTurney, P. D. and M. L. Littman. 2005.\nCorpus-based learning of analogies\nand semantic relations. Machine\nLearning , 60(1-3):251\u2013278.\nUmeda, N. 1976. Linguistic rules for\ntext-to-speech synthesis. Proceed-\nings of the IEEE , 64(4):443\u2013451.\nUmeda, N., E. Matui, T. Suzuki, and\nH. Omura. 1968. Synthesis of fairy\ntale using an analog vocal tract. 6th\nInternational Congress on Acous-\ntics.\nUng, M., J. Xu, and Y .-L. Boureau.\n2022. SaFeRDialogues: Taking\nfeedback gracefully after conversa-\ntional safety failures. ACL.\nUryupina, O., R. Artstein, A. Bristot,\nF. Cavicchio, F. Delogu, K. J. Ro-\ndriguez, and M. Poesio. 2020. An-\nnotating a broad range of anaphoric\nphenomena, in a variety of genres:\nThe ARRAU corpus. Natural Lan-\nguage Engineering , 26(1):1\u201334.\nUszkoreit, J. 2017. Transformer: A\nnovel neural network architecture\nfor language understanding. Google\nResearch blog post, Thursday Au-\ngust 31, 2017.van Deemter, K. and R. Kibble.\n2000. On coreferring: corefer-\nence in MUC and related annotation\nschemes. Computational Linguis-\ntics, 26(4):629\u2013637.\nvan der Maaten, L. and G. E. Hinton.\n2008. Visualizing high-dimensional\ndata using t-SNE. JMLR , 9:2579\u2013\n2605.\nvan Rijsbergen, C. J. 1975. Information\nRetrieval . Butterworths.\nVaswani, A., N. Shazeer, N. Parmar,\nJ. Uszkoreit, L. Jones, A. N. Gomez,\n\u0141. Kaiser, and I. Polosukhin. 2017.\nAttention is all you need. NeurIPS .\nVauquois, B. 1968. A survey of for-\nmal grammars and algorithms for\nrecognition and transformation in\nmachine translation. IFIP Congress\n1968 .\nVelichko, V . M. and N. G. Zagoruyko.\n1970. Automatic recognition of\n200 words. International Journal of\nMan-Machine Studies , 2:223\u2013234.\nVelikovich, L., S. Blair-Goldensohn,\nK. Hannan, and R. McDonald. 2010.\nThe viability of web-derived polarity\nlexicons. NAACL HLT .\nVendler, Z. 1967. Linguistics in Philos-\nophy . Cornell University Press.\nVerhagen, M., R. Gaizauskas,\nF. Schilder, M. Hepple, J. Moszkow-\nicz, and J. Pustejovsky. 2009. The\nTempEval challenge: Identifying\ntemporal relations in text. Lan-\nguage Resources and Evaluation ,\n43(2):161\u2013179.\nVerhagen, M., I. Mani, R. Sauri,\nR. Knippen, S. B. Jang, J. Littman,\nA. Rumshisky, J. Phillips, and\nJ. Pustejovsky. 2005. Automating\ntemporal annotation with TARSQI.\nACL.\nVersley, Y . 2008. Vagueness and ref-\nerential ambiguity in a large-scale\nannotated corpus. Research on\nLanguage and Computation , 6(3-\n4):333\u2013353.\nVieira, R. and M. Poesio. 2000. An em-\npirically based system for process-\ning de\ufb01nite descriptions. Computa-\ntional Linguistics , 26(4):539\u2013593.\nVilain, M., J. D. Burger, J. Aberdeen,\nD. Connolly, and L. Hirschman.\n1995. A model-theoretic coreference\nscoring scheme. MUC-6 .\nVintsyuk, T. K. 1968. Speech discrim-\nination by dynamic programming.\nCybernetics , 4(1):52\u201357. Origi-\nnal Russian: Kibernetika 4(1):81-\n88. 1968.\nVinyals, O., \u0141. Kaiser, T. Koo,\nS. Petrov, I. Sutskever, and G. Hin-\nton. 2015. Grammar as a foreign lan-\nguage. NeurIPS .V oorhees, E. M. 1999. TREC-8 ques-\ntion answering track report. Pro-\nceedings of the 8th Text Retrieval\nConference .\nV oorhees, E. M. and D. K. Harman.\n2005. TREC: Experiment and\nEvaluation in Information Retrieval .\nMIT Press.\nV outilainen, A. 1999. Handcrafted\nrules. In H. van Halteren, ed., Syn-\ntactic Wordclass Tagging , 217\u2013246.\nKluwer.\nVrande \u02c7ci\u00b4c, D. and M. Kr \u00a8otzsch. 2014.\nWikidata: a free collaborative\nknowledge base. CACM , 57(10):78\u2013\n85.\nWade, E., E. Shriberg, and P. J. Price.\n1992. User behaviors affecting\nspeech recognition. ICSLP .\nWagner, R. A. and M. J. Fischer. 1974.\nThe string-to-string correction prob-\nlem. Journal of the ACM , 21:168\u2013\n173.\nWaibel, A., T. Hanazawa, G. Hin-\nton, K. Shikano, and K. J. Lang.\n1989. Phoneme recognition using\ntime-delay neural networks. IEEE\nTransactions on ASSP , 37(3):328\u2013\n339.\nWalker, M. A. 2000. An applica-\ntion of reinforcement learning to di-\nalogue strategy selection in a spo-\nken dialogue system for email. JAIR ,\n12:387\u2013416.\nWalker, M. A., J. C. Fromer, and S. S.\nNarayanan. 1998a. Learning optimal\ndialogue strategies: A case study of\na spoken dialogue agent for email.\nCOLING/ACL .\nWalker, M. A., M. Iida, and S. Cote.\n1994. Japanese discourse and the\nprocess of centering. Computational\nLinguistics , 20(2):193\u2013232.\nWalker, M. A., A. K. Joshi, and\nE. Prince, eds. 1998b. Centering in\nDiscourse . Oxford University Press.\nWang, A., A. Singh, J. Michael, F. Hill,\nO. Levy, and S. R. Bowman. 2018a.\nGlue: A multi-task benchmark and\nanalysis platform for natural lan-\nguage understanding. ICLR .\nWang, S. and C. D. Manning. 2012.\nBaselines and bigrams: Simple,\ngood sentiment and topic classi\ufb01ca-\ntion. ACL.\nWang, W. and B. Chang. 2016. Graph-\nbased dependency parsing with bidi-\nrectional LSTM. ACL.\nWang, Y ., S. Li, and J. Yang. 2018b.\nToward fast and accurate neural dis-\ncourse segmentation. EMNLP .\nWang, Y ., S. Mishra, P. Alipoormo-\nlabashi, Y . Kordi, A. Mirzaei,\nA. Naik, A. Ashok, A. S.\nDhanasekaran, A. Arunkumar,\nD. Stap, E. Pathak, G. Kara-\nmanolakis, H. Lai, I. Purohit, Bibliography 581\nI. Mondal, J. Anderson, K. Kuz-\nnia, K. Doshi, K. K. Pal, M. Pa-\ntel, M. Moradshahi, M. Par-\nmar, M. Purohit, N. Varshney,\nP. R. Kaza, P. Verma, R. S. Puri,\nR. Karia, S. Doshi, S. K. Sampat,\nS. Mishra, S. Reddy A, S. Patro,\nT. Dixit, and X. Shen. 2022. Super-\nNaturalInstructions: Generaliza-\ntion via declarative instructions on\n1600+ NLP tasks. EMNLP .\nWang, Y ., R. Skerry-Ryan, D. Stan-\nton, Y . Wu, R. J. Weiss, N. Jaitly,\nZ. Yang, Y . Xiao, Z. Chen, S. Ben-\ngio, Q. Le, Y . Agiomyrgiannakis,\nR. Clark, and R. A. Saurous.\n2017. Tacotron: Towards end-to-end\nspeech synthesis. INTERSPEECH .\nWatanabe, S., T. Hori, S. Karita,\nT. Hayashi, J. Nishitoba, Y . Unno,\nN. E. Y . Soplin, J. Heymann,\nM. Wiesner, N. Chen, A. Renduch-\nintala, and T. Ochiai. 2018. ESP-\nnet: End-to-end speech processing\ntoolkit. INTERSPEECH .\nWeaver, W. 1949/1955. Translation. In\nW. N. Locke and A. D. Boothe, eds,\nMachine Translation of Languages ,\n15\u201323. MIT Press. Reprinted from a\nmemorandum written by Weaver in\n1949.\nWebber, B. L. 1978. A Formal\nApproach to Discourse Anaphora .\nPh.D. thesis, Harvard University.\nWebber, B. L. 1983. So what can we\ntalk about now? In M. Brady and\nR. C. Berwick, eds, Computational\nModels of Discourse , 331\u2013371. The\nMIT Press.\nWebber, B. L. 1991. Structure and os-\ntension in the interpretation of dis-\ncourse deixis. Language and Cogni-\ntive Processes , 6(2):107\u2013135.\nWebber, B. L. and B. Baldwin. 1992.\nAccommodating context change.\nACL.\nWebber, B. L., M. Egg, and V . Kor-\ndoni. 2012. Discourse structure and\nlanguage technology. Natural Lan-\nguage Engineering , 18(4):437\u2013490.\nWebber, B. L. 1988. Discourse deixis:\nReference to discourse segments.\nACL.\nWebson, A. and E. Pavlick. 2022. Do\nprompt-based models really under-\nstand the meaning of their prompts?\nNAACL HLT .\nWebster, K., M. Recasens, V . Axel-\nrod, and J. Baldridge. 2018. Mind\nthe GAP: A balanced corpus of gen-\ndered ambiguous pronouns. TACL ,\n6:605\u2013617.\nWei, J., X. Wang, D. Schuurmans,\nM. Bosma, F. Xia, E. Chi, Q. V .\nLe, D. Zhou, et al. 2022. Chain-of-\nthought prompting elicits reasoning\nin large language models. NeurIPS ,\nvolume 35.Weischedel, R., M. Meteer,\nR. Schwartz, L. A. Ramshaw, and\nJ. Palmucci. 1993. Coping with am-\nbiguity and unknown words through\nprobabilistic models. Computational\nLinguistics , 19(2):359\u2013382.\nWeizenbaum, J. 1966. ELIZA \u2013 A\ncomputer program for the study of\nnatural language communication be-\ntween man and machine. CACM ,\n9(1):36\u201345.\nWeizenbaum, J. 1976. Computer Power\nand Human Reason: From Judge-\nment to Calculation . W.H. Freeman\n& Co.\nWerbos, P. 1974. Beyond regression:\nnew tools for prediction and analy-\nsis in the behavioral sciences . Ph.D.\nthesis, Harvard University.\nWerbos, P. J. 1990. Backpropagation\nthrough time: what it does and how\nto do it. Proceedings of the IEEE ,\n78(10):1550\u20131560.\nWeston, J., S. Chopra, and A. Bordes.\n2015. Memory networks. ICLR\n2015 .\nWidrow, B. and M. E. Hoff. 1960.\nAdaptive switching circuits. IRE\nWESCON Convention Record , vol-\nume 4.\nWiebe, J. 1994. Tracking point of view\nin narrative. Computational Linguis-\ntics, 20(2):233\u2013287.\nWiebe, J. 2000. Learning subjective ad-\njectives from corpora. AAAI .\nWiebe, J., R. F. Bruce, and T. P. O\u2019Hara.\n1999. Development and use of a\ngold-standard data set for subjectiv-\nity classi\ufb01cations. ACL.\nWierzbicka, A. 1992. Semantics, Cul-\nture, and Cognition: University Hu-\nman Concepts in Culture-Speci\ufb01c\nCon\ufb01gurations . Oxford University\nPress.\nWierzbicka, A. 1996. Semantics:\nPrimes and Universals . Oxford Uni-\nversity Press.\nWilensky, R. 1983. Planning and\nUnderstanding: A Computational\nApproach to Human Reasoning .\nAddison-Wesley.\nWilks, Y . 1973. An arti\ufb01cial intelli-\ngence approach to machine transla-\ntion. In R. C. Schank and K. M.\nColby, eds, Computer Models of\nThought and Language , 114\u2013151.\nW.H. Freeman.\nWilks, Y . 1975a. Preference semantics.\nIn E. L. Keenan, ed., The Formal Se-\nmantics of Natural Language , 329\u2013\n350. Cambridge Univ. Press.\nWilks, Y . 1975b. A preferential,\npattern-seeking, semantics for natu-\nral language inference. Arti\ufb01cial In-\ntelligence , 6(1):53\u201374.Williams, A., N. Nangia, and S. Bow-\nman. 2018. A broad-coverage chal-\nlenge corpus for sentence under-\nstanding through inference. NAACL\nHLT.\nWilliams, J. D., K. Asadi, and\nG. Zweig. 2017. Hybrid code\nnetworks: practical and ef\ufb01cient\nend-to-end dialog control with su-\npervised and reinforcement learning.\nACL.\nWilliams, J. D., A. Raux, and M. Hen-\nderson. 2016. The dialog state track-\ning challenge series: A review. Dia-\nlogue & Discourse , 7(3):4\u201333.\nWilliams, J. D. and S. J. Young. 2007.\nPartially observable markov deci-\nsion processes for spoken dialog sys-\ntems. Computer Speech and Lan-\nguage , 21(1):393\u2013422.\nWilson, T., J. Wiebe, and P. Hoffmann.\n2005. Recognizing contextual polar-\nity in phrase-level sentiment analy-\nsis.EMNLP .\nWinograd, T. 1972. Understanding Nat-\nural Language . Academic Press.\nWinston, P. H. 1977. Arti\ufb01cial Intelli-\ngence . Addison Wesley.\nWiseman, S., A. M. Rush, and S. M.\nShieber. 2016. Learning global\nfeatures for coreference resolution.\nNAACL HLT .\nWiseman, S., A. M. Rush, S. M.\nShieber, and J. Weston. 2015. Learn-\ning anaphoricity and antecedent\nranking features for coreference res-\nolution. ACL.\nWitten, I. H. and T. C. Bell. 1991.\nThe zero-frequency problem: Es-\ntimating the probabilities of novel\nevents in adaptive text compression.\nIEEE Transactions on Information\nTheory , 37(4):1085\u20131094.\nWitten, I. H. and E. Frank. 2005. Data\nMining: Practical Machine Learn-\ning Tools and Techniques , 2nd edi-\ntion. Morgan Kaufmann.\nWittgenstein, L. 1953. Philosoph-\nical Investigations. (Translated by\nAnscombe, G.E.M.) . Blackwell.\nWolf, F. and E. Gibson. 2005. Rep-\nresenting discourse coherence: A\ncorpus-based analysis. Computa-\ntional Linguistics , 31(2):249\u2013287.\nWolf, M. J., K. W. Miller, and F. S.\nGrodzinsky. 2017. Why we should\nhave seen that coming: Comments\non Microsoft\u2019s Tay \u201cexperiment,\u201d\nand wider implications. The ORBIT\nJournal , 1(2):1\u201312.\nWoods, W. A. 1978. Semantics and\nquanti\ufb01cation in natural language\nquestion answering. In M. Yovits,\ned.,Advances in Computers , 2\u201364.\nAcademic. 582 Bibliography\nWoods, W. A., R. M. Kaplan, and B. L.\nNash-Webber. 1972. The lunar sci-\nences natural language information\nsystem: Final report. Technical Re-\nport 2378, BBN.\nWoodsend, K. and M. Lapata. 2015.\nDistributed representations for un-\nsupervised semantic role labeling.\nEMNLP .\nWu, D. 1996. A polynomial-time algo-\nrithm for statistical machine transla-\ntion. ACL.\nWu, F. and D. S. Weld. 2007. Au-\ntonomously semantifying Wiki-\npedia. CIKM-07 .\nWu, F. and D. S. Weld. 2010. Open\ninformation extraction using Wiki-\npedia. ACL.\nWu, L., F. Petroni, M. Josifoski,\nS. Riedel, and L. Zettlemoyer. 2020.\nScalable zero-shot entity linking\nwith dense entity retrieval. EMNLP .\nWu, S. and M. Dredze. 2019. Beto,\nBentz, Becas: The surprising cross-\nlingual effectiveness of BERT.\nEMNLP .\nWu, Y ., M. Schuster, Z. Chen, Q. V .\nLe, M. Norouzi, W. Macherey,\nM. Krikun, Y . Cao, Q. Gao,\nK. Macherey, J. Klingner, A. Shah,\nM. Johnson, X. Liu, \u0141. Kaiser,\nS. Gouws, Y . Kato, T. Kudo,\nH. Kazawa, K. Stevens, G. Kurian,\nN. Patil, W. Wang, C. Young,\nJ. Smith, J. Riesa, A. Rud-\nnick, O. Vinyals, G. S. Corrado,\nM. Hughes, and J. Dean. 2016.\nGoogle\u2019s neural machine translation\nsystem: Bridging the gap between\nhuman and machine translation.\nArXiv preprint arXiv:1609.08144.\nWundt, W. 1900. V\u00a8olkerpsychologie:\neine Untersuchung der Entwick-\nlungsgesetze von Sprache, Mythus,\nund Sitte . W. Engelmann, Leipzig.\nBand II: Die Sprache, Zweiter Teil.\nXu, A., E. Pathak, E. Wallace, S. Gu-\nrurangan, M. Sap, and D. Klein.\n2021. Detoxifying language models\nrisks marginalizing minority voices.\nNAACL HLT .\nXu, J., D. Ju, M. Li, Y .-L. Boureau,\nJ. Weston, and E. Dinan. 2020.\nRecipes for safety in open-\ndomain chatbots. ArXiv preprint\narXiv:2010.07079.\nXu, P., H. Saghir, J. S. Kang, T. Long,\nA. J. Bose, Y . Cao, and J. C. K. Che-\nung. 2019. A cross-domain transfer-\nable neural coherence model. ACL.\nXue, N., H. T. Ng, S. Pradhan,\nA. Rutherford, B. L. Webber,\nC. Wang, and H. Wang. 2016.\nCoNLL 2016 shared task on mul-\ntilingual shallow discourse parsing.\nCoNLL-16 shared task .Xue, N. and M. Palmer. 2004. Calibrat-\ning features for semantic role label-\ning. EMNLP .\nYamada, H. and Y . Matsumoto. 2003.\nStatistical dependency analysis with\nsupport vector machines. IWPT-03 .\nYang, D., J. Chen, Z. Yang, D. Jurafsky,\nand E. H. Hovy. 2019. Let\u2019s make\nyour request more persuasive: Mod-\neling persuasive strategies via semi-\nsupervised neural nets on crowd-\nfunding platforms. NAACL HLT .\nYang, X., G. Zhou, J. Su, and C. L. Tan.\n2003. Coreference resolution us-\ning competition learning approach.\nACL.\nYang, Y . and J. Pedersen. 1997. A com-\nparative study on feature selection in\ntext categorization. ICML .\nYankelovich, N., G.-A. Levow, and\nM. Marx. 1995. Designing\nSpeechActs: Issues in speech user\ninterfaces. CHI-95 .\nYih, W.-t., M. Richardson, C. Meek,\nM.-W. Chang, and J. Suh. 2016. The\nvalue of semantic parse labeling for\nknowledge base question answering.\nACL.\nYoung, S. J., M. Ga \u02c7si\u00b4c, S. Keizer,\nF. Mairesse, J. Schatzmann,\nB. Thomson, and K. Yu. 2010. The\nHidden Information State model:\nA practical framework for POMDP-\nbased spoken dialogue management.\nComputer Speech & Language ,\n24(2):150\u2013174.\nYounger, D. H. 1967. Recognition and\nparsing of context-free languages in\ntime n3.Information and Control ,\n10:189\u2013208.\nYu, N., M. Zhang, and G. Fu. 2018.\nTransition-based neural RST parsing\nwith implicit syntax features. COL-\nING.\nYu, Y ., Y . Zhu, Y . Liu, Y . Liu,\nS. Peng, M. Gong, and A. Zeldes.\n2019. GumDrop at the DISRPT2019\nshared task: A model stacking ap-\nproach to discourse unit segmenta-\ntion and connective detection. Work-\nshop on Discourse Relation Parsing\nand Treebanking 2019 .\nZapirain, B., E. Agirre, L. M `arquez,\nand M. Surdeanu. 2013. Selectional\npreferences for semantic role classi-\n\ufb01cation. Computational Linguistics ,\n39(3):631\u2013663.\nZelle, J. M. and R. J. Mooney. 1996.\nLearning to parse database queries\nusing inductive logic programming.\nAAAI .\nZeman, D. 2008. Reusable tagset con-\nversion using tagset drivers. LREC .\nZens, R. and H. Ney. 2007. Ef\ufb01cient\nphrase-table representation for ma-\nchine translation with applications toonline MT and speech translation.\nNAACL-HLT .\nZettlemoyer, L. and M. Collins. 2005.\nLearning to map sentences to log-\nical form: Structured classi\ufb01cation\nwith probabilistic categorial gram-\nmars. Uncertainty in Arti\ufb01cial Intel-\nligence, UAI\u201905 .\nZettlemoyer, L. and M. Collins. 2007.\nOnline learning of relaxed CCG\ngrammars for parsing to logical\nform. EMNLP/CoNLL .\nZhang, H., R. Sproat, A. H. Ng,\nF. Stahlberg, X. Peng, K. Gorman,\nand B. Roark. 2019. Neural models\nof text normalization for speech ap-\nplications. Computational Linguis-\ntics, 45(2):293\u2013337.\nZhang, R., C. N. dos Santos, M. Ya-\nsunaga, B. Xiang, and D. Radev.\n2018. Neural coreference resolution\nwith deep biaf\ufb01ne attention by joint\nmention detection and mention clus-\ntering. ACL.\nZhang, T., V . Kishore, F. Wu, K. Q.\nWeinberger, and Y . Artzi. 2020.\nBERTscore: Evaluating text gener-\nation with BERT. ICLR 2020 .\nZhang, Y ., V . Zhong, D. Chen, G. An-\ngeli, and C. D. Manning. 2017.\nPosition-aware attention and su-\npervised data improve slot \ufb01lling.\nEMNLP .\nZhao, H., W. Chen, C. Kit, and G. Zhou.\n2009. Multilingual dependency\nlearning: A huge feature engineer-\ning method to semantic dependency\nparsing. CoNLL .\nZhao, J., T. Wang, M. Yatskar, R. Cot-\nterell, V . Ordonez, and K.-W. Chang.\n2019. Gender bias in contextualized\nword embeddings. NAACL HLT .\nZhao, J., T. Wang, M. Yatskar, V . Or-\ndonez, and K.-W. Chang. 2017. Men\nalso like shopping: Reducing gender\nbias ampli\ufb01cation using corpus-level\nconstraints. EMNLP .\nZhao, J., T. Wang, M. Yatskar, V . Or-\ndonez, and K.-W. Chang. 2018a.\nGender bias in coreference reso-\nlution: Evaluation and debiasing\nmethods. NAACL HLT .\nZhao, J., Y . Zhou, Z. Li, W. Wang,\nand K.-W. Chang. 2018b. Learn-\ning gender-neutral word embed-\ndings. EMNLP .\nZheng, J., L. Vilnis, S. Singh, J. D.\nChoi, and A. McCallum. 2013.\nDynamic knowledge-base alignment\nfor coreference resolution. CoNLL .\nZhou, D., O. Bousquet, T. N. Lal,\nJ. Weston, and B. Sch \u00a8olkopf. 2004a.\nLearning with local and global con-\nsistency. NeurIPS .\nZhou, G., J. Su, J. Zhang, and\nM. Zhang. 2005. Exploring var-\nious knowledge in relation extrac-\ntion. ACL. Bibliography 583\nZhou, J. and W. Xu. 2015a. End-to-\nend learning of semantic role label-\ning using recurrent neural networks.\nACL.\nZhou, J. and W. Xu. 2015b. End-to-\nend learning of semantic role label-\ning using recurrent neural networks.\nACL.\nZhou, K., K. Ethayarajh, D. Card, and\nD. Jurafsky. 2022. Problems with\ncosine as a measure of embedding\nsimilarity for high frequency words.\nACL.\nZhou, K., J. Hwang, X. Ren, and\nM. Sap. 2024. Relying on the un-\nreliable: The impact of language\nmodels\u2019 reluctance to express uncer-\ntainty. ACL.\nZhou, L., M. Ticrea, and E. H. Hovy.\n2004b. Multi-document biography\nsummarization. EMNLP .\nZhou, Y ., A. I. Muresanu, Z. Han,\nK. Paster, S. Pitis, H. Chan, and\nJ. Ba. 2023. Large language models\nare human-level prompt engineers.\nThe Eleventh International Confer-\nence on Learning Representations .\nZhou, Y . and N. Xue. 2015. The Chi-\nnese Discourse TreeBank: a Chinese\ncorpus annotated with discourse re-\nlations. Language Resources and\nEvaluation , 49(2):397\u2013431.\nZhu, X. and Z. Ghahramani. 2002.\nLearning from labeled and unlabeled\ndata with label propagation. Techni-\ncal Report CMU-CALD-02, CMU.\nZhu, X., Z. Ghahramani, and J. Laf-\nferty. 2003. Semi-supervised learn-\ning using gaussian \ufb01elds and har-\nmonic functions. ICML .\nZhu, Y ., R. Kiros, R. Zemel,\nR. Salakhutdinov, R. Urtasun,\nA. Torralba, and S. Fidler. 2015.\nAligning books and movies: To-\nwards story-like visual explanations\nby watching movies and reading\nbooks. IEEE International Confer-\nence on Computer Vision .\nZiemski, M., M. Junczys-Dowmunt,\nand B. Pouliquen. 2016. The United\nNations parallel corpus v1.0. LREC .  Subject Index\n*?,9\n+?,9\n.wav format, 336\n10-fold cross-validation, 69\n!(derives), 389\n\u02c6,58\n* (RE Kleene *), 7\n+ (RE Kleene +), 7\n. (RE any character), 7\n$(RE end-of-line), 8\n((RE precedence symbol),\n8\n[(RE character\ndisjunction), 6\n\\B(RE non\nword-boundary), 8\n\\b(RE word-boundary), 8\n](RE character\ndisjunction), 6\n^(RE start-of-line), 8\n[^](single-char negation),\n6\n4-gram, 37\n4-tuple, 392\n5-gram, 37\nA-D conversion, 335\nAAC, 32\nAAE, 16\nAB test, 353\nablating, 248\nabsolute position, 197\nabsolute temporal\nexpression, 452\nabstract word, 485\naccessible, 506\naccessing a referent, 501\naccomplishment\nexpressions, 450\naccuracy, 366\nachievement expressions,\n450\nacknowledgment speech\nact, 312\nactivation, 133\nactivity expressions, 450\nacute-eval, 325\nad hoc retrieval, 291\nadd gate, 172\nadd-k, 47\nadd-one smoothing, 45\nadequacy, 280\nadjacency pairs, 313\nAdjectives, 364\nadverb, 364\ndegree, 364\ndirectional, 364\nlocative, 364\nmanner, 364\ntemporal, 364\nAdverbs, 364\nAED, 339\naffective, 481\naf\ufb01x, 24agent, as thematic role, 462\nagglutinative\nlanguage, 267\nAIFF \ufb01le, 336\nAISHELL-1, 334\naktionsart, 450\nALGOL, 409\nalgorithm\nbyte-pair encoding, 23\nCKY , 397\nminimum edit distance,\n28\nnaive Bayes classi\ufb01er, 57\npointwise mutual\ninformation, 114\nsemantic role labeling,\n469\nTextTiling, 544\nViterbi, 373\naligned, 249\nalignment, 25,342\nin ASR, 346\nminimum cost, 28\nstring, 25\nvia minimum edit\ndistance, 28\nAllen relations, 448\nallocational harm, 126\nambiguity\namount of part-of-speech\nin Brown corpus,\n366\nattachment, 396\ncoordination, 396\nof referring expressions,\n503\npart-of-speech, 365\nresolution of tag, 366\nAmerican Structuralism,\n408\nanaphor, 502\nanaphora, 502\nanaphoricity detector, 511\nanchor texts, 520\nanchors in regular\nexpressions, 8, 30\nanisotropy, 234\nantecedent, 502\nApple AIFF, 336\napproximate\nrandomization, 71\narc eager, 423\narc standard, 417\nargmax, 58\nargumentation mining, 547\nargumentation schemes,\n548\nargumentative relations,\n547\nargumentative zoning, 549\nAristotle, 362, 450\nARPA, 355\narticle (part-of-speech), 364\narticulatory synthesis, 357aspect, 450\nASR, 331\ncon\ufb01dence, 320\nassociation, 103\nATIS\ncorpus, 390\nATN, 478\nATRANS, 477\nattachment ambiguity, 396\nattention\ncross-attention, 272\nencoder-decoder, 272\nhistory in transformers,\n201\nattention head, 188\nattention mechanism, 179\nAttribution (as coherence\nrelation), 534\naugmentative\ncommunication, 32\nauthorship attribution, 56\nautoregressive generation,\n167,207\nAuxiliary, 365\nB3,524\nBabbage, C., 332\nbackoff, 48\nin smoothing, 48\nbackprop, 147\nbackpropagation through\ntime, 161\nbacktrace\nin minimum edit\ndistance, 29\nbacktranslation, 279\nBackus-Naur form, 388\nbackward-looking center,\n541\nbag of words, 58, 59\nin IR, 291\nbakeoff, 355\nspeech recognition\ncompetition, 355\nbarged in, 326\nbase model, 249\nbasic emotions, 482\nbatch training, 94\nBayes\u2019 rule, 58\ndropping denominator,\n59, 372\nBayesian inference, 58\nBDI, 329\nbeam search, 275,424\nbeam width, 275,424\nBerkeley Restaurant\nProject, 36\nBernoulli naive Bayes, 75\nBERT\nfor affect, 497\nbest-worst scaling, 486\nbias ampli\ufb01cation, 126\nbias term, 79,133\nbidirectional RNN, 169bigram, 34\nbinary branching, 394\nbinary naive Bayes, 63\nbinary tree, 394\nBIO, 238,368\nBIO tagging, 238\nfor NER, 238, 368\nBIOES, 238, 368\nbitext, 270\nbits for measuring entropy,\n49\nblank in CTC, 342\nBM25, 291,293\nBNF (Backus-Naur form),\n388\nbootstrap, 73\nbootstrap algorithm, 73\nbootstrap test, 71\nbootstrapping, 71\nin IE, 441\nbound pronoun, 504\nBPE, 21\nBPE, 23\nbracketed notation, 391\nbridging inference, 506\nbroadcast news\nspeech recognition of,\n355\nBrown corpus, 13\noriginal tagging of, 384\nbyte-pair encoding, 21\ncalibrated, 290\nCALLHOME, 333\nCandide , 287\nCantonese, 267\ncapture group, 12\ncascade\nregular expression in\nELIZA, 13\ncase\nsensitivity in regular\nexpression search, 5\ncase folding, 23\ncase frame, 463, 478\nCAT, 263\ncataphora, 504\nCD (conceptual\ndependency), 477\nCentering Theory, 532, 540\ncentroid, 117\ncepstrum\nhistory, 355\nCFG, seecontext-free\ngrammar\nchain rule, 98,148\nchain-of-thought, 254\nchannels in stored\nwaveforms, 336\nchart parsing, 397\nChatbots, 309,321\nchatbots, 4\nCHiME, 333\nChinese\n585 586 Subject Index\nas verb-framed language,\n267\nwords for brother, 266\nChomsky normal form, 394\nChomsky-adjunction, 395\nchrF, 281\nCIRCUS, 459\ncitation form, 102\nCitizen Kane , 531\nCKY algorithm, 387\nclaims, 547\nclass-based n-gram, 53\nclassi\ufb01er head, 235\nclefts, 507\nclitic, 19\norigin of term, 362\nclosed book, 304\nclosed class, 363\ncloze task, 226\ncluster, 502\nCNF, seeChomsky normal\nform\nCocke-Kasami-Younger\nalgorithm, seeCKY\ncode switching, 16\ncoherence, 531\nentity-based, 540\nrelations, 533\ncohesion\nlexical, 532, 544\nColBERT, 300\ncold languages, 268\ncollection in IR, 291\ncommissive speech act, 312\ncommon crawl, 211\ncommon ground, 312, 328\nCommon nouns, 363\ncomplementizers, 364\ncomponential analysis, 476\ncompression, 335\nComputational Grammar\nCoder (CGC), 384\nconcatenation, 5, 30\nconceptual dependency, 477\nconcrete word, 485\nconditional generation, 204\nconditional random \ufb01eld,\n376\ncon\ufb01dence, 285\nASR, 320\nin relation extraction, 442\ncon\ufb01dence values, 442\ncon\ufb01guration, 417\nconfusion matrix, 66\nConjunctions, 364\nconnectionist, 157\nconnotation frame, 497\nconnotation frames, 479\nconnotations, 104,482\nconstative speech act, 312\nconstituency, 388\nconstituent, 388\ntitles which are not, 387\nConstraint Grammar, 433\ncontent planning, 319\ncontext embedding, 122\ncontext-free grammar, 388,\n392, 407Chomsky normal form,\n394\ninvention of, 409\nnon-terminal symbol,\n389\nproductions, 388\nrules, 388\nterminal symbol, 389\nweak and strong\nequivalence, 394\ncontextual embeddings,\n186,230\ncontinued pretraining, 214\nconversation, 309\nconversation analysis, 313,\n328\nconversational implicature,\n314\nconversational speech, 333\nconvex, 90\ncoordination ambiguity, 396\ncopula, 365\nCORAAL, 333\ncorefer, 501\ncoreference chain, 502\ncoreference resolution, 502\ngender agreement, 508\nHobbs tree search\nalgorithm, 528\nnumber agreement, 507\nperson agreement, 508\nrecency preferences, 508\nselectional restrictions,\n509\nsyntactic (\u201cbinding\u201d)\nconstraints, 508\nverb semantics, 509\ncorpora, 13\ncorpus, 13\nATIS, 390\nBroadcast news, 355\nBrown, 13, 384\n\ufb01sher, 355\nLOB, 384\nregular expression\nsearching inside, 5\nSwitchboard, 14,333,\n335\nTimeBank, 451\nWall Street Journal, 355\ncorrection act detection,\n319\ncosine\nas a similarity metric,\n110\ncost function, 87\ncount nouns, 363\ncounters, 30\ncounts\ntreating low as zero, 379\nCRF, 376\ncompared to HMM, 376\ninference, 380\nViterbi inference, 380\nCRFs\nlearning, 381\ncross-attention, 272\ncross-brackets, 406cross-entropy, 51\ncross-entropy loss, 88,145\ncross-validation, 69\n10-fold, 69\ncrowdsourcing, 485\nCTC, 341\ndatasheet, 16\ndative alternation, 463\ndebiasing, 127\ndecision boundary, 80,136\ndecoder-only model, 200\ndecoding, 207,372\nViterbi, 372\ndeep\nneural networks, 132\ndeep learning, 132\nde\ufb01nite reference, 504\ndegree adverb, 364\ndelexicalize, 320\ndemonstrations, 246\ndenoising, 226\ndependency\ngrammar, 411\ndependency tree, 414\ndependent, 412\nderivation\ndirect (in a formal\nlanguage), 392\nsyntactic, 389, 389, 392,\n392\nDet, 388\ndeterminer, 364, 388\nDeterminers, 364\ndevelopment set, 38\ndevelopment test set, 69\ndevelopment test set\n(dev-test), 39\ndevset, seedevelopment\ntest set (dev-test), 69\nDFT, 338\ndialogue, 309\ndialogue act\ncorrection, 319\nDialogue acts, 318\ndialogue policy, 319\ndialogue systems, 309\ndesign, 325\ndiathesis alternation, 463\ndiff program, 30\ndigit recognition, 332\ndigital divide, 263\ndigitization, 335\ndilated convolutions, 352\ndimension, 107\ndiphthong\norigin of term, 362\ndirect derivation (in a\nformal language),\n392\ndirectional adverb, 364\ndirective speech act, 312\ndisambiguation\nin parsing, 403\nsyntactic, 397\ndiscount, 46, 47, 49\ndiscounting, 45, 46\ndiscourse, 531segment, 534\ndiscourse connectives, 535\ndiscourse deixis, 503\ndiscourse model, 501\ndiscourse parsing, 536\ndiscourse-new, 505\ndiscourse-old, 505\ndiscovery procedure, 408\ndiscrete Fourier transform,\n338\ndiscriminative model, 78\ndis\ufb02uency, 14\ndisjunction, 30\npipe in regular\nexpressions as, 8\nsquare braces in regular\nexpression as, 6\ndispreferred response, 330\ndistant supervision, 443\ndistributional hypothesis,\n101\ndistributional similarity,\n408\ndivergences between\nlanguages in MT,\n265\ndocument\nin IR, 291\ndocument frequency, 112\ndocument vector, 117\ndomination in syntax, 389\ndot product, 79,110\ndot-product attention, 180\nDragon Systems, 355\ndropout, 151\nduration\ntemporal expression, 452\ndynamic programming, 26\nand parsing, 397\nViterbi as, 373\ndynamic time warping, 355\nedge-factored, 426\nedit distance\nminimum algorithm, 27\nEDU, 534\neffect size, 70\nef\ufb01ciency costs, 317\nElaboration (as coherence\nrelation), 533\nELIZA, 4\nimplementation, 13\nsample conversation, 12\nElman Networks, 158\nELMo\nfor affect, 497\nEM\nfor deleted interpolation,\n48\nembedding layer, 154\nembeddings, 105\ncosine for similarity, 110\nskip-gram, learning, 120\nsparse, 110\ntf-idf, 112\nword2vec, 117\nemission probabilities, 370\nEmoLex, 484 Subject Index 587\nemotion, 482\nEncoder-decoder, 175\nencoder-decoder attention,\n272\nend-to-end training, 167\nendpointing, 312\nEnglish\nlexical differences from\nFrench, 267\nsimpli\ufb01ed grammar\nrules, 390\nverb-framed, 267\nentity dictionary, 379\nentity grid, 542\nEntity linking, 520\nentity linking, 502\nentity-based coherence, 540\nentropy, 49\nand perplexity, 49\ncross-entropy, 51\nper-word, 50\nrate, 50\nrelative, 474\nerror backpropagation, 147\nESPnet, 356\nethos, 547\nEuclidean distance\nin L2 regularization, 95\nEugene Onegin , 52\nEuler\u2019s formula, 338\nEuroparl, 270\nevalb, 406\nevaluating parsers, 405\nevaluation\n10-fold cross-validation,\n69\nAB test, 353\ncomparing models, 41\ncross-validation, 69\ndevelopment test set, 39,\n69\ndevset, 69\ndevset or development\ntest set, 39\nextrinsic, 38\n\ufb02uency in MT, 280\nMatched-Pair Sentence\nSegment Word Error\n(MAPSSWE), 347\nmean opinion score, 353\nmost frequent class\nbaseline, 366\nMT, 280\nnamed entity recognition,\n240, 381\nof n-gram, 38\nof n-grams via\nperplexity, 40\npseudoword, 476\nrelation extraction, 446\ntest set, 39\ntraining on the test set, 39\ntraining set, 39\nTTS, 353\nevent coreference, 503\nevent extraction, 435,446\nevents, 450Evidence (as coherence\nrelation), 533\nevoking a referent, 501\nexecution accuracy, 256\nexpansion, 390, 391\nexpletive, 507\nexplicit con\ufb01rmation, 319\nextraposition, 507\nextrinsic evaluation, 38\nF (for F-measure), 67\nF-measure, 67\nF-measure\nin NER, 240, 381\nfactoid question, 289\nFaiss, 301\nfalse negatives, 10\nfalse positives, 10\nFarsi, verb-framed, 267\nfast Fourier transform, 338,\n355\nfasttext, 123\nFASTUS, 457\nfeature cutoff, 379\nfeature interactions, 82\nfeature selection\ninformation gain, 76\nfeature template, 421\nfeature templates, 82\npart-of-speech tagging,\n378\nfeature vectors, 334\nFederalist papers, 75\nfeedforward network, 138\nfenceposts, 398\nfew-shot, 246\nFFT, 338, 355\n\ufb01le format, .wav, 336\n\ufb01lled pause, 14\n\ufb01ller, 14\n\ufb01netuning, 213,234\n\ufb01netuning;supervsed, 249\n\ufb01rst-order co-occurrence,\n124\n\ufb02uency, 280\nin MT, 280\nfold (in cross-validation),\n69\nforget gate, 172\nformal language, 391\nformant synthesis, 357\nforward inference, 153\nforward-looking centers,\n541\nFosler, E., see\nFosler-Lussier, E.\nfoundation model, 221\nfragment of word, 14\nframe, 336\nsemantic, 467\nframe elements, 467\nFrameNet, 466\nframes, 314\nfree word order, 411\nFreebase, 437\nfreeze, 155,214\nFrench, 265\nFrump, 459fully-connected, 138\nfunction word, 363, 383\nfusion language, 268\nGaussian\nprior on weights, 96\ngazetteer, 379\nGeneral Inquirer, 64,484\ngeneralize, 95\ngeneralized semantic role,\n464\ngeneration\nof sentences to test a\nCFG grammar, 390\ngenerative AI, 204\ngenerative grammar, 391\ngenerative model, 78\ngenerative models, 59\ngenerator, 389\ngenerics, 507\nGerman, 265\ngiven-new, 506\nGodzilla, speaker as, 472\ngold labels, 66\ngradient, 90\nGrammar\nConstraint, 433\nHead-Driven Phrase\nStructure (HPSG),\n406\nLink, 433\ngrammar\nbinary branching, 394\nchecking, 387\nequivalence, 394\ngenerative, 391\ninversion transduction,\n287\ngrammatical function, 412\ngrammatical relation, 412\ngrammatical sentences, 391\ngreedy decoding, 206\ngreedy RE patterns, 9\ngrep, 5, 5, 30\nGricean maxims, 314\ngrounding, 312\nGUS, 314\nhallucinate, 290\nhallucination, 219\nHamilton, Alexander, 75\nHamming, 337\nHansard, 287\nhanzi, 20\nharmonic mean, 67\nhead, 188,198,406,412\n\ufb01nding, 406\nHead-Driven Phrase\nStructure Grammar\n(HPSG), 406\nHeaps\u2019 Law, 15\nHearst patterns, 438\nheld-out, 48\nHerdan\u2019s Law, 15\nhidden, 370\nhidden layer, 138\nas representation of\ninput, 139hidden units, 138\nHindi, 265\nHindi, verb-framed, 267\nHKUST, 334\nHMM, 370\nformal de\ufb01nition of, 370\nhistory in speech\nrecognition, 355\ninitial distribution, 370\nobservation likelihood,\n370\nobservations, 370\nsimplifying assumptions\nfor POS tagging,\n372\nstates, 370\ntransition probabilities,\n370\nHobbs algorithm, 528\nHobbs tree search algorithm\nfor pronoun\nresolution, 528\nhomonymy, 231\nhot languages, 268\nHungarian\npart-of-speech tagging,\n382\nhybrid, 356\nhyperarticulation, 319\nhypernym, 437\nlexico-syntactic patterns\nfor, 438\nhyperparameter, 92\nhyperparameters, 152\nIBM Models, 287\nIBM Thomas J. Watson\nResearch Center,\n53, 355\nidf,113\nidf term weighting, 113,\n292\nimmediately dominates,\n389\nimplicature, 314\nimplicit argument, 479\nimplicit con\ufb01rmation, 320\nin-context learning, 247\ninde\ufb01nite reference, 504\ninduction heads, 247\ninference-based learning,\n429\ninfoboxes, 437\ninformation\nstructure, 505\nstatus, 505\ninformation extraction (IE),\n435\nbootstrapping, 441\ninformation gain, 76\nfor feature selection, 76\nInformation retrieval, 108,\n290\ninformation retrieval, 290\ninitiative, 313\ninner product, 110\ninstance, word, 14 588 Subject Index\nInstitutional Review Board,\n327\nInstruction tuning, 249\nintent determination, 316\nintercept, 79\nInterjections, 364\ninterpolated precision, 296\ninterpolation\nin smoothing, 48\ninterpretable, 98\ninterval algebra, 448\nintrinsic evaluation, 38\ninversion transduction\ngrammar (ITG), 287\ninverted index, 295\nIO, 238, 368\nIOB tagging\nfor temporal expressions,\n453\nIR,290\nidf term weighting, 113,\n292\nterm weighting, 291\nvector space model, 107\nIRB, 327\nis-a, 437\nISO 8601, 454\nisolating language, 267\niSRL, 479\nITG (inversion transduction\ngrammar), 287\nJapanese, 265, 267\nJay, John, 75\njoint intention, 328\nKaldi, 356\nKBP, 459\nKenLM, 38,53\nkey,188\nKL divergence, 474\nKlatt formant synthesizer,\n357\nKleene *, 7\nsneakiness of matching\nzero things, 7\nKleene +, 7\nknowledge claim, 549\nknowledge graphs, 435\nKullback-Leibler\ndivergence, 474\nKV cache, 217\nL1 regularization, 95\nL2 regularization, 95\nlabeled precision, 405\nlabeled recall, 405\nlanguage\nidenti\ufb01cation, 354\nuniversal, 264\nlanguage id, 56\nlanguage model, 32\nlanguage model:coined by,\n53\nlanguage modeling head,\n198\nLaplace smoothing, 45for PMI, 116\nlasso regression, 96\nlatent semantic analysis,\n130\nlayer norm, 191\nLDC, 19\nlearning rate, 90\nlemma, 15,102\nversus wordform, 15\nLemmatization, 24\nlemmatization, 5\nLevenshtein distance, 26\nlexical\ncategory, 389\ncohesion, 532, 544\ngap, 267\nsemantics, 102\ntrigger, in IE, 452\nlexico-syntactic pattern,\n438\nlexicon, 388\nLibriSpeech, 333\nlight verbs, 447\nlikelihood, 59\nlinear chain CRF, 376, 377\nlinear classi\ufb01ers, 60\nlinear interpolation for\nn-grams, 48\nlinearly separable, 136\nLinguistic Data\nConsortium, 19\nLinguistic Discourse\nmodel, 550\nLink Grammar, 433\nList (as coherence relation),\n534\nlisten attend and spell, 339\nLIWC, 64,485\nLM, 32\nLOB corpus, 384\nlocalization, 263\nlocation-based attention,\n351\nlocative, 364\nlocative adverb, 364\nlog\nwhy used for\nprobabilities, 37\nwhy used to compress\nspeech, 336\nlog likelihood ratio, 493\nlog odds ratio, 493\nlog probabilities, 37, 37\nlogistic function, 79\nlogistic regression, 77\nconditional maximum\nlikelihood\nestimation, 88\nGaussian priors, 96\nlearning in, 87\nregularization, 96\nrelation to neural\nnetworks, 140\nlogit, 80,199\nlogit lens, 200\nlogos, 547\nlong short-term memory,\n171lookahead in regex, 13\nLoRA, 218\nloss, 87\nlow frame rate, 340\nLPC (Linear Predictive\nCoding), 355\nLSI, seelatent semantic\nanalysis\nLSTM, 385\nLUNAR, 307\nmachine learning\nfor NER, 382\ntextbooks, 75, 100\nmachine translation, 263\nmacroaveraging, 68\nMadison, James, 75\nMAE, 16\nMandarin, 265\nManhattan distance\nin L1 regularization, 95\nmanner adverb, 364\nMarkov, 34\nassumption, 34\nMarkov assumption, 369\nMarkov chain, 52, 369\nformal de\ufb01nition of, 370\ninitial distribution, 370\nn-gram as, 369\nstates, 370\ntransition probabilities,\n370\nMarkov model, 34\nformal de\ufb01nition of, 370\nhistory, 53\nMarx, G., 387\nMasked Language\nModeling, 226\nmass nouns, 363\nmaxent, 100\nmaxim, Gricean, 314\nmaximum entropy, 100\nmaximum spanning tree,\n426\nMayan, 267\nMBR, 277\nMcNemar\u2019s test, 348\nmean\nelement-wise, 167\nmean average precision,\n297\nmean opinion score, 353\nmean reciprocal rank, 305\nmechanical indexing, 129\nMechanical Turk, 332\nmel, 338\nmemory networks, 201\nmention detection, 510\nmention-pair, 513\nmentions, 501\nMERT, for training in MT,\n287\nMeSH (Medical Subject\nHeadings), 57\nMessage Understanding\nConference, 457\nMETEOR, 288\nmetonymy, 530microaveraging, 68\nMicrosoft .wav format, 336\nmini-batch, 94\nMinimum Bayes risk, 277\nminimum edit distance, 25,\n25, 373\nexample of, 28\nfor speech recognition\nevaluation, 346\nMINIMUM EDIT DISTANCE ,\n28\nminimum edit distance\nalgorithm, 27\nMinimum Error Rate\nTraining, 287\nMLE\nfor n-grams, 35\nfor n-grams, intuition, 35\nMLM, 226\nMLP, 138\nMMLU, 258,304\nmodal verb, 365\nmodel alignment, 249\nmodel card, 74\nmorpheme, 24\nMOS (mean opinion score),\n353\nMoses, Michelangelo statue\nof, 309\nMoses, MT toolkit, 287\nMRR, 305\nMS MARCO, 303\nMT, 263\ndivergences, 265\npost-editing, 263\nmu-law, 336\nMUC, 457, 459\nMUC F-measure, 524\nmulti-head attention, 189\nmulti-hop, 303\nmulti-layer perceptrons,\n138\nmultinomial logistic\nregression, 84\nmultinomial naive Bayes,\n57\nmultinomial naive Bayes\nclassi\ufb01er, 57\nmultiword expressions, 130\nMWE, 130\nn-best list, 341\nn-gram, 32,34\nadd-one smoothing, 45\nas approximation, 34\nas generators, 43\nas Markov chain, 369\nequation for, 34\nexample of, 36, 37\nfor Shakespeare, 43\nhistory of, 53\ninterpolation, 48\nKenLM, 38,53\nlogprobs in, 37\nnormalizing, 36\nparameter estimation, 35\nsensitivity to corpus, 43\nsmoothing, 45 Subject Index 589\nSRILM, 53\ntest set, 38\ntraining set, 38\nnaive Bayes\nmultinomial, 57\nsimplifying assumptions,\n59\nnaive Bayes assumption, 59\nnaive Bayes classi\ufb01er\nuse in text categorization,\n57\nnamed entity, 237,362,367\nlist of types, 237, 367\nnamed entity recognition,\n237,367\nnatural language inference,\n236\nNatural Questions, 303\nnegative log likelihood loss,\n88,97,146\nNER, 237,367\nneural networks\nrelation to logistic\nregression, 140\nnewline character, 10\nNext Sentence Prediction,\n228\nNIST for MT evaluation,\n288\nnoisy-or, 442\nNomBank, 466\nNominal, 388\nnon-capturing group, 12\nnon-greedy, 9\nnon-standard words, 349\nnon-stationary process, 336\nnon-terminal symbols, 389,\n390\nnormal form, 394, 394\nnormalization\ntemporal, 453\nword, 23\nnormalization of\nprobabilities, 35\nnormalize, 83\nnormalizing, 140\nnoun\nabstract, 363\ncommon, 363\ncount, 363\nmass, 363\nproper, 363\nnoun phrase, 388\nconstituents, 388\nNouns, 363\nNP,388, 390\nnucleus, 533\nnull hypothesis, 70\nNyquist frequency, 335\nobservation likelihood\nrole in Viterbi, 374\none-hot vector, 153,197\nopen book, 304\nopen class, 363\nopen information\nextraction, 444\noperation list, 25operator precedence, 8, 9\noptionality\nuse of ? in regular\nexpressions for, 7\noutput gate, 172\nover\ufb01tting, 95\np-value, 71\nPaired, 71\nparallel corpus, 270\nparallel distributed\nprocessing, 157\nparallelogram model, 124\nparameter-ef\ufb01cient \ufb01ne\ntuning, 217\nparse tree, 389, 391\nPARSEV AL, 405\nparsing\nambiguity, 395\nCKY , 397\nCYK, seeCKY\nevaluation, 405\nrelation to grammars,\n392\nsyntactic, 387\nwell-formed substring\ntable, 409\npart of speech\nas used in CFG, 389\npart-of-speech\nadjective, 364\nadverb, 364\nclosed class, 363\ninterjection, 364\nnoun, 363\nopen class, 363\nparticle, 364\nsubtle distinction\nbetween verb and\nnoun, 364\nverb, 364\npart-of-speech tagger\nPARTS ,384\nTAGGIT, 384\nPart-of-speech tagging, 365\npart-of-speech tagging\nambiguity and, 365\namount of ambiguity in\nBrown corpus, 366\nand morphological\nanalysis, 382\nfeature templates, 378\nhistory of, 384\nHungarian, 382\nTurkish, 382\nunknown words, 376\nparticle, 364\nPARTS tagger, 384\nparts of speech, 362\npathos, 547\npattern, regular expression,\n5\nPCM (Pulse Code\nModulation), 336\nPDP, 157\nPDTB, 535\nPEFT, 217Penn Discourse TreeBank,\n535\nPenn Treebank, 393\ntagset, 365, 365\nPenn Treebank\ntokenization, 19\nper-word entropy, 50\nperceptron, 135\nperiod disambiguation, 82\nperplexity, 40,52\nas weighted average\nbranching factor, 41\nde\ufb01ned via\ncross-entropy, 52\nperplexity:coined by, 53\npersonal pronoun, 364\npersuasion, 548\nphrasal verb, 364\nphrase-based translation,\n287\nphrase-structure grammar,\n388\nPII,212\npipe, 8\nplanning\nand speech acts, 329\nshared plans, 328\npleonastic, 507\nPointwise mutual\ninformation, 114\npolysynthetic language, 267\npooling, 143,167\nmax, 167\nmean, 167\nPorter stemmer, 24\nPOS, 362\npositional embeddings, 197\npossessive pronoun, 364\npost-editing, 263\npost-training, 249\npostings, 295\npostposition, 265\nPotts diagram, 492\nPP, 390\nPP-attachment ambiguity,\n396\nPPMI, 115\nprecedence, 8\nprecedence, operator, 8\nPrecision, 67\nprecision\nfor MT evaluation, 288\nin NER, 240, 381\nprecision-recall curve, 296\npremises, 547\nprepositional phrase\nconstituency, 390\nprepositions, 364\npresequences, 313\npretraining, 145,203\nprimitive decomposition,\n476\nprinciple of contrast, 103\nprior probability, 59\npro-drop languages, 268\nprobabilistic context-free\ngrammars, 409\nproductions, 388projective, 414\nprompt, 243\nprompt engineering, 243\npronoun, 364\nbound, 504\ndemonstrative, 505\nnon-binary, 508\npersonal, 364\npossessive, 364\nwh-, 364\nPropBank, 465\nproper noun, 363\nPROTO -AGENT ,464\nPROTO -PATIENT ,464\npseudoword, 476\nPTRANS, 477\npunctuation\nfor numbers\ncross-linguistically,\n19\nfor sentence\nsegmentation, 25\ntokenization, 19\ntreated as words, 13\ntreated as words in LM,\n44\nquantization, 335\nquery, 188,291\nin IR, 291\nquestion\nfactoid, 289\nquestion answering\nfactoid questions, 289\nRadio Rex, 331\nRAG, 290,302\nrandom sampling, 208\nrange, regular expression, 6\nranking, 281\nrarefaction, 335\nRDF, 437\nRDF triple, 437\nRead speech, 333\nreading comprehension,\n304\nReason (as coherence\nrelation), 533\nRecall, 67\nrecall\nfor MT evaluation, 288\nin NER, 240, 381\nrectangular, 336\nreference\nbound pronouns, 504\ncataphora, 504\nde\ufb01nite, 504\ngenerics, 507\ninde\ufb01nite, 504\nreference point, 449\nreferent, 501\naccessing of, 501\nevoking of, 501\nreferential density, 268\nre\ufb02exive, 508\nregex\nregular expression, 5\nregister in regex, 12 590 Subject Index\nregression\nlasso, 96\nridge, 96\nregular expression, 5, 29, 30\nsubstitutions, 11\nregularization, 95\nrejection\nconversation act, 320\nrelatedness, 103\nrelation extraction, 435\nrelative\ntemporal expression, 452\nrelative entropy, 474\nrelative frequency, 35\nrelevance, 314\nrelexicalize, 321\nReLU, 134\nreporting events, 447\nrepresentation learning, 101\nrepresentational harm, 127\nrepresentational harms, 73\nrescore, 341\nresidual stream, 190\nresolve, 366\nResource Management, 355\nretrieval-augmented\ngeneration, 302\nReVerb, 445\nrewrite, 389\nRhetorical Structure\nTheory, seeRST\nRiau Indonesian, 364\nridge regression, 96\nRLHF, 325\nRNN-T, 345\nrole-\ufb01ller extraction, 457\nRosebud, sled named, 531\nrow vector, 108\nRST, 533\nTreeBank, 535, 550\nrules\ncontext-free, 388\ncontext-free, expansion,\n389\ncontext-free, sample, 390\nRussian\nfusion language, 268\nverb-framed, 267\nSas start symbol in CFG,\n390\nsalience, in discourse\nmodel, 506\nSampling, 42\nsampling\nof analog waveform, 335\nrate, 335\nsatellite, 267,533\nsatellite-framed language,\n267\nsaturated, 135\nscaling laws, 216\nSCISOR, 459\nsclite, 347\nsclite package, 30\nscript\nSchankian, 467\nscripts, 456SDRT (Segmented\nDiscourse\nRepresentation\nTheory), 550\nsearch engine, 290\nsearch tree, 274\nsecond-order\nco-occurrence, 124\nseed pattern in IE, 441\nseed tuples, 441\nsegmentation\nsentence, 25\nword, 18\nselectional association, 475\nselectional preference\nstrength, 474\nselectional preferences\npseudowords for\nevaluation, 476\nselectional restriction, 472\nrepresenting with events,\n473\nviolations in WSD, 474\nself-supervision, 118,163,\n210\nself-training, 155\nsemantic drift in IE, 442\nsemantic feature, 130\nsemantic \ufb01eld, 103\nsemantic frame, 104\nsemantic relations in IE,\n436\ntable, 437\nsemantic role, 462, 462,\n464\nSemantic role labeling, 468\nsemantics\nlexical, 102\nsense\nword, 231\nsentence\nerror rate, 347\nsegmentation, 25\nsentence realization, 320\nsentence segmentation, 5\nsentence separation, 176\nSentencePiece, 270\nsentiment, 104\norigin of term, 500\nsentiment analysis, 56\nsentiment lexicons, 64\nSentiWordNet, 490\nsequence labeling, 362\nSFT, 249\nSGNS, 117\nShakespeare\nn-gram approximations\nto, 43\nshallow discourse parsing,\n539\nshared plans, 328\nside sequence, 313\nsigmoid, 79,133\nsigni\ufb01cance test\nMAPSSWE for ASR,\n347\nMcNemar\u2019s, 348\nsimilarity, 103cosine, 110\nsingleton, 502\nsingular they, 508\nskip-gram, 117\nslot error rate, 317\nslot \ufb01lling, 316,459\nslots, 314\nsmoothing, 45, 45\nadd-one, 45\ndiscounting, 46\ninterpolation, 48\nLaplace, 45\nlinear interpolation, 48\nsoftmax, 85,140\nSOV language, 265\nspam detection, 56,64\nspan, 403\nSpeaker diarization, 353\nspeaker identi\ufb01cation, 354\nspeaker recognition, 354\nspeaker veri\ufb01cation, 354\nspeech\ntelephone bandwidth,\n335\nspeech acts, 312\nspeech recognition\narchitecture, 332, 339\nhistory of, 354\nspeech synthesis, 332\nsplit-half reliability, 487\nSRILM, 53\nSRL, 468\nStacked RNNs, 168\nstandardize, 82\nstart symbol, 389\nstates, 450\nstatic embeddings, 118\nstationary process, 336\nstationary stochastic\nprocess, 51\nstatistical MT, 287\nstatistical signi\ufb01cance\nMAPSSWE for ASR,\n347\nMcNemar\u2019s test, 348\nstatistically signi\ufb01cant, 71\nstative expressions, 450\nstem, 24\nStemming, 5\nstemming, 24\nstop list, 294\nstop words, 61\nstreaming, 345\nstride, 336\nstructural ambiguity, 395\nstupid backoff, 49\nsubdialogue, 313\nsubjectivity, 481,500\nsubstitutability, 408\nsubstitution operator\n(regular\nexpressions), 11\nsubword tokens, 19\nsubwords, 21\nsupervised \ufb01netuning, 249\nsupervised machine\nlearning, 57\nSVD, 130SVO language, 265\nSwedish, verb-framed, 267\nSwitchboard, 333\nSwitchboard Corpus, 14,\n333, 335\nsynchronous grammar, 287\nsynonyms, 103\nsyntactic disambiguation,\n397\nsyntax, 387\norigin of term, 362\nTAC KBP, 438\nTacotron2, 351\nTACRED dataset, 437\nTAGGIT, 384\ntagset\nPenn Treebank, 365, 365\ntable of Penn Treebank\ntags, 365\nTamil, 267\ntanh, 134\ntarget embedding, 122\ntask error rate, 317\nTay, 326\nteacher forcing, 164,178,\n210, 274\ntechnai, 362\ntelephone-bandwidth\nspeech, 335\ntelic, 450\ntemperature sampling, 209\ntemplate, 245\ntemplate \ufb01lling, 435,456\ntemplate recognition, 456\ntemplate, in IE, 456\ntemplates, 244\ntemporal adverb, 364\ntemporal anchor, 455\ntemporal expression\nabsolute, 452\nmetaphor for, 449\nrelative, 452\ntemporal logic, 447\ntemporal normalization,\n453\nterm\nin IR, 291\nweight in IR, 291\nterm frequency, 112\nterm weight, 291\nterm-document matrix, 106\nterm-term matrix, 109\nterminal symbol, 389\ntest set, 38\ndevelopment, 39\nhow to choose, 39\ntext categorization, 56\nbag-of-words\nassumption, 58\nnaive Bayes approach, 57\nunknown words, 61\ntext normalization, 4,17\ntext summarization, 205\ntext-to-speech, 332\nTextTiling, 544\ntf-idf, 113\nThe Pile, 211 Subject Index 591\nthematic grid, 463\nthematic role, 462\nand diathesis alternation,\n463\nexamples of, 462\nproblems, 464\ntheme, 462\ntheme, as thematic role, 462\nTimeBank, 451\ntokenization, 4\nsentence, 25\nword, 18\nTop-k sampling, 208\ntop-p sampling, 209\ntopic models, 104\ntoxicity detection, 74\ntraining oracle, 419\ntraining set, 38\ncross-validation, 69\nhow to choose, 39\ntranscription\nof speech, 331\nreference, 346\ntransduction grammars, 287\ntransfer learning, 223\nTransformations and\nDiscourse Analysis\nProject (TDAP),\n384\ntransition probability\nrole in Viterbi, 374\ntransition-based, 416\ntranslation\ndivergences, 265\nTREC, 308\ntreebank, 392\ntrigram, 37\nTTS, 332\nTurk, Mechanical, 332\nTurkishagglutinative, 267\npart-of-speech tagging,\n382\nturns, 311\nTyDi QA, 304\ntyped dependency structure,\n411\ntypes\nword, 14\ntypology, 265\nlinguistic, 265\nunembedding, 199\nungrammatical sentences,\n391\nunigram\nname of tokenization\nalgorithm, 270\nunit production, 397\nunit vector, 111\nUniversal Dependencies,\n413\nuniversal, linguistic, 264\nUnix, 5\nunknown words\nin part-of-speech\ntagging, 376\nin text categorization, 61\nuser-centered design, 325\nutterance, 14\nvalue, 188\nvalue sensitive design, 326\nvanishing gradient, 135\nvanishing gradients, 171\nVauquois triangle, 286\nvector, 107,133\nvector length, 110\nVector semantics, 105vector semantics, 101\nvector space, 107\nvector space model, 107\nverb\ncopula, 365\nmodal, 365\nphrasal, 364\nverb alternations, 463\nverb phrase, 390\nverb-framed language, 267\nVerbs, 364\nVietnamese, 267\nViterbi\nand beam search, 274\nViterbi algorithm, 26, 373\ninference in CRF, 380\nVITERBI ALGORITHM , 373\nvocoder, 349\nvocoding, 349\nvoice user interface, 325\nVSO language, 265\nwake word, 353\nWall Street Journal\nWall Street Journal\nspeech recognition of,\n355\nwarping, 355\nwave\ufb01le format, 336\nWaveNet, 351\nWavenet, 351\nweight tying, 165,199\nwell-formed substring\ntable, 409\nWFST, 409\nwh-pronoun, 364\nwiki\ufb01cation, 520\nwildcard, regular\nexpression, 7Winograd Schema, 525\nWizard-of-Oz system, 325\nword\nboundary, regular\nexpression notation,\n8\nclosed class, 363\nde\ufb01nition of, 13\nerror rate, 334, 346\nfragment, 14\nfunction, 363, 383\nopen class, 363\npunctuation as, 13\ntokens, 14\ntypes, 14\nword normalization, 23\nword segmentation, 18,20\nword sense, 231\nword sense disambiguation,\n232,seeWSD\nword shape, 378\nword tokenization, 18\nword-word matrix, 109\nword2vec, 117\nwordform, 15\nand lemma, 102\nversus lemma, 15\nWordNet, 231\nwordpiece, 269\nWSD, 232\nYonkers Racetrack, 49\nYupik, 267\nz-score, 82\nzero anaphor, 505\nzero-shot, 246\nzero-width, 13\nzeros, 45", "2.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\n2Regular Expressions, Tok-\nenization, Edit Distance\nUser: I am unhappy.\nELIZA: DO YOU THINK COMING HERE WILL HELP YOU NOT TO BE UNHAPPY\nUser: I need some help, that much seems certain.\nELIZA: WHAT WOULD IT MEAN TO YOU IF YOU GOT SOME HELP\nUser: Perhaps I could learn to get along with my mother.\nELIZA: TELL ME MORE ABOUT YOUR FAMILY\nUser: My mother takes care of me.\nELIZA: WHO ELSE IN YOU FAMILY TAKES CARE OF YOU\nUser: My father.\nELIZA: YOUR FATHER\nUser: You are like my father in some ways.\nWeizenbaum (1966)\nThe dialogue above is from ELIZA , an early natural language processing system ELIZA\nthat could carry on a limited conversation with a user by imitating the responses of\na Rogerian psychotherapist (Weizenbaum, 1966). ELIZA is a surprisingly simple\nprogram that uses pattern matching to recognize phrases like \u201cI need X\u201d and translate\nthem into suitable outputs like \u201cWhat would it mean to you if you got X?\u201d. This\nsimple technique succeeds in this domain because ELIZA doesn\u2019t actually need to\nknow anything to mimic a Rogerian psychotherapist. As Weizenbaum notes, this is\none of the few dialogue genres where listeners can act as if they know nothing of the\nworld. ELIZA\u2019s mimicry of human conversation was remarkably successful: many\npeople who interacted with ELIZA came to believe that it really understood them\nand their problems, many continued to believe in ELIZA\u2019s abilities even after the\nprogram\u2019s operation was explained to them (Weizenbaum, 1976), and even today\nsuch chatbots are a fun diversion. chatbots\nOf course modern conversational agents are much more than a diversion; they\ncan answer questions, book \ufb02ights, or \ufb01nd restaurants, functions for which they rely\non a much more sophisticated understanding of the user\u2019s intent, as we will see in\nChapter 15. Nonetheless, the simple pattern-based methods that powered ELIZA\nand other chatbots play a crucial role in natural language processing.\nWe\u2019ll begin with the most important tool for describing text patterns: the regular\nexpression . Regular expressions can be used to specify strings we might want to\nextract from a document, from transforming \u201cI need X\u201d in ELIZA above, to de\ufb01ning\nstrings like $199 or$24.99 for extracting tables of prices from a document.\nWe\u2019ll then turn to a set of tasks collectively called text normalization , in whichtext\nnormalization\nregular expressions play an important part. Normalizing text means converting it\nto a more convenient, standard form. For example, most of what we are going to\ndo with language relies on \ufb01rst separating out or tokenizing words from running\ntext, the task of tokenization . English words are often separated from each other tokenization\nby whitespace, but whitespace is not always suf\ufb01cient. New York androck \u2019n\u2019 roll\nare sometimes treated as large words despite the fact that they contain spaces, while\nsometimes we\u2019ll need to separate I\u2019minto the two words Iandam. For processing\ntweets or texts we\u2019ll need to tokenize emoticons like:)orhashtags like#nlproc . 2CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\nSome languages, like Japanese, don\u2019t have spaces between words, so word tokeniza-\ntion becomes more dif\ufb01cult.\nAnother part of text normalization is lemmatization , the task of determining lemmatization\nthat two words have the same root, despite their surface differences. For example,\nthe words sang ,sung , and sings are forms of the verb sing. The word sing is the\ncommon lemma of these words, and a lemmatizer maps from all of these to sing.\nLemmatization is essential for processing morphologically complex languages like\nArabic. Stemming refers to a simpler version of lemmatization in which we mainly stemming\njust strip suf\ufb01xes from the end of the word. Text normalization also includes sen-\ntence segmentation : breaking up a text into individual sentences, using cues likesentence\nsegmentation\nperiods or exclamation points.\nFinally, we\u2019ll need to compare words and other strings. We\u2019ll introduce a metric\ncalled edit distance that measures how similar two strings are based on the number\nof edits (insertions, deletions, substitutions) it takes to change one string into the\nother. Edit distance is an algorithm with applications throughout language process-\ning, from spelling correction to speech recognition to coreference resolution.\n2.1 Regular Expressions\nOne of the most useful tools for text processing in computer science has been the\nregular expression (often shortened to regex ), a language for specifying text searchregular\nexpression\nstrings. This practical language is used in every computer language, in text process-\ning tools like the Unix tools grep, and in editors like vim or Emacs. Formally, a\nregular expression is an algebraic notation for characterizing a set of strings. Reg-\nular expressions are particularly useful for searching in texts, when we have a pat-\ntern to search for and a corpus of texts to search through. A regular expression corpus\nsearch function will search through the corpus, returning all texts that match the\npattern. The corpus can be a single document or a collection. For example, the\nUnix command-line tool grep takes a regular expression and returns every line of\nthe input document that matches the expression.\nA search can be designed to return every match on a line, if there are more than\none, or just the \ufb01rst match. In the following examples we generally underline the\nexact part of the pattern that matches the regular expression and show only the \ufb01rst\nmatch. We\u2019ll show regular expressions delimited by slashes but note that slashes are\nnotpart of the regular expressions.\nRegular expressions come in many variants. We\u2019ll be describing extended regu-\nlar expressions ; different regular expression parsers may only recognize subsets of\nthese, or treat some expressions slightly differently. Using an online regular expres-\nsion tester is a handy way to test out your expressions and explore these variations.\n2.1.1 Basic Regular Expression Patterns\nThe simplest kind of regular expression is a sequence of simple characters; putting\ncharacters in sequence is called concatenation . To search for woodchuck , we type concatenation\n/woodchuck/ . The expression /Buttercup/ matches any string containing the\nsubstring Buttercup ;grep with that expression would return the line I\u2019m called lit-\ntle Buttercup . The search string can consist of a single character (like /!/) or a\nsequence of characters (like /urgl/ ) (see Fig. 2.1).\nRegular expressions are case sensitive ; lower case /s/ is distinct from upper 2.1 \u2022 R EGULAR EXPRESSIONS 3\nRegex Example Patterns Matched\n/woodchucks/ \u201cinteresting links to woodchucks and lemurs\u201d\n/a/ \u201cMary Ann stopped by Mona\u2019s\u201d\n/!/ \u201cYou\u2019ve left the burglar behind again! \u201d said Nori\nFigure 2.1 Some simple regex searches.\ncase/S/ (/s/ matches a lower case sbut not an upper case S). This means that\nthe pattern /woodchucks/ will not match the string Woodchucks . We can solve this\nproblem with the use of the square braces [and]. The string of characters inside the\nbraces speci\ufb01es a disjunction of characters to match. For example, Fig. 2.2 shows\nthat the pattern /[wW]/ matches patterns containing either worW.\nRegex Match Example Patterns\n/[wW]oodchuck/ Woodchuck or woodchuck \u201cWoodchuck \u201d\n/[abc]/ \u2018a\u2019, \u2018b\u2019, or\u2018c\u2019 \u201cIn uomini, in solda ti\u201d\n/[1234567890]/ any digit \u201cplenty of 7 to 5\u201d\nFigure 2.2 The use of the brackets []to specify a disjunction of characters.\nThe regular expression /[1234567890]/ speci\ufb01es any single digit. While such\nclasses of characters as digits or letters are important building blocks in expressions,\nthey can get awkward (e.g., it\u2019s inconvenient to specify\n/[ABCDEFGHIJKLMNOPQRSTUVWXYZ]/ (2.1)\nto mean \u201cany capital letter\u201d). In cases where there is a well-de\ufb01ned sequence asso-\nciated with a set of characters, the brackets can be used with the dash ( -) to specify\nany one character in a range . The pattern /[2-5]/ speci\ufb01es any one of the charac- range\nters2,3,4, or5. The pattern /[b-g]/ speci\ufb01es one of the characters b,c,d,e,f, or\ng. Some other examples are shown in Fig. 2.3.\nRegex Match Example Patterns Matched\n/[A-Z]/ an upper case letter \u201cwe should call it \u2018D renched Blossoms\u2019 \u201d\n/[a-z]/ a lower case letter \u201cmy beans were impatient to be hoed!\u201d\n/[0-9]/ a single digit \u201cChapter 1 : Down the Rabbit Hole\u201d\nFigure 2.3 The use of the brackets []plus the dash -to specify a range.\nThe square braces can also be used to specify what a single character cannot be,\nby use of the caret ^. If the caret ^is the \ufb01rst symbol after the open square brace [,\nthe resulting pattern is negated. For example, the pattern /[^a]/ matches any single\ncharacter (including special characters) except a. This is only true when the caret\nis the \ufb01rst symbol after the open square brace. If it occurs anywhere else, it usually\nstands for a caret; Fig. 2.4 shows some examples.\nRegex Match (single characters) Example Patterns Matched\n/[^A-Z]/ not an upper case letter \u201cOyfn pripetchik\u201d\n/[^Ss]/ neither \u2018S\u2019 nor \u2018s\u2019 \u201cIhave no exquisite reason for\u2019t\u201d\n/[^.]/ not a period \u201cour resident Djinn\u201d\n/[e^]/ either \u2018e\u2019 or \u2018 ^\u2019 \u201clook up \u02c6 now\u201d\n/a^b/ the pattern \u2018 a^b\u2019 \u201clook up a\u02c6 b now\u201d\nFigure 2.4 The caret ^for negation or just to mean ^. See below re: the backslash for escaping the period. 4CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\nHow can we talk about optional elements, like an optional sinwoodchuck and\nwoodchucks ? We can\u2019t use the square brackets, because while they allow us to say\n\u201cs or S\u201d, they don\u2019t allow us to say \u201cs or nothing\u201d. For this we use the question mark\n/?/, which means \u201cthe preceding character or nothing\u201d, as shown in Fig. 2.5.\nRegex Match Example Patterns Matched\n/woodchucks?/ woodchuck or woodchucks \u201cwoodchuck \u201d\n/colou?r/ color or colour \u201ccolor \u201d\nFigure 2.5 The question mark ?marks optionality of the previous expression.\nWe can think of the question mark as meaning \u201czero or one instances of the\nprevious character\u201d. That is, it\u2019s a way of specifying how many of something that\nwe want, something that is very important in regular expressions. For example,\nconsider the language of certain sheep, which consists of strings that look like the\nfollowing:\nbaa!\nbaaa!\nbaaaa!\n. . .\nThis language consists of strings with a b, followed by at least two a\u2019s, followed\nby an exclamation point. The set of operators that allows us to say things like \u201csome\nnumber of as\u201d are based on the asterisk or *, commonly called the Kleene * (gen- Kleene *\nerally pronounced \u201ccleany star\u201d). The Kleene star means \u201czero or more occurrences\nof the immediately previous character or regular expression\u201d. So /a*/ means \u201cany\nstring of zero or more as\u201d. This will match aoraaaaaa , but it will also match the\nempty string at the start of Off Minor since the string Off Minor starts with zero a\u2019s.\nSo the regular expression for matching one or more ais/aa*/ , meaning one afol-\nlowed by zero or more as. More complex patterns can also be repeated. So /[ab]*/\nmeans \u201czero or more a\u2019s or b\u2019s\u201d (not \u201czero or more right square braces\u201d). This will\nmatch strings like aaaa orababab orbbbb , as well as the empty string.\nFor specifying multiple digits (useful for \ufb01nding prices) we can extend /[0-9]/ ,\nthe regular expression for a single digit. An integer (a string of digits) is thus\n/[0-9][0-9]*/ . (Why isn\u2019t it just /[0-9]*/ ?)\nSometimes it\u2019s annoying to have to write the regular expression for digits twice,\nso there is a shorter way to specify \u201cat least one\u201d of some character. This is the\nKleene + , which means \u201cone or more occurrences of the immediately preceding Kleene +\ncharacter or regular expression\u201d. Thus, the expression /[0-9]+/ is the normal way\nto specify \u201ca sequence of digits\u201d. There are thus two ways to specify the sheep\nlanguage: /baaa*!/ or/baa+!/ .\nOne very important special character is the period ( /./), awildcard expression\nthat matches any single character ( except a carriage return), as shown in Fig. 2.6.\nRegex Match Example Matches\n/beg.n/ any character between begandn begin , beg\u2019n , begun\nFigure 2.6 The use of the period .to specify any character.\nThe wildcard is often used together with the Kleene star to mean \u201cany string of\ncharacters\u201d. For example, suppose we want to \ufb01nd any line in which a particular\nword, for example, aardvark , appears twice. We can specify this with the regular\nexpression /aardvark.*aardvark/ . 2.1 \u2022 R EGULAR EXPRESSIONS 5\nAnchors are special characters that anchor regular expressions to particular places anchors\nin a string. The most common anchors are the caret ^and the dollar sign $. The caret\n^matches the start of a line. The pattern /^The/ matches the word Theonly at the\nstart of a line. Thus, the caret ^has three uses: to match the start of a line, to in-\ndicate a negation inside of square brackets, and just to mean a caret. (What are the\ncontexts that allow grep or Python to know which function a given caret is supposed\nto have?) The dollar sign $matches the end of a line. So the pattern  $is a useful\npattern for matching a space at the end of a line, and /^The dog\\.$/ matches a\nline that contains only the phrase The dog. (We have to use the backslash here since\nwe want the .to mean \u201cperiod\u201d and not the wildcard.)\nRegex Match\n^ start of line\n$ end of line\n\\b word boundary\n\\B non-word boundary\nFigure 2.7 Anchors in regular expressions.\nThere are also two other anchors: \\bmatches a word boundary, and \\Bmatches\na non word-boundary. Thus, /\\bthe\\b/ matches the word thebut not the word\nother . A \u201cword\u201d for the purposes of a regular expression is de\ufb01ned based on the\nde\ufb01nition of words in programming languages as a sequence of digits, underscores,\nor letters. Thus /\\b99\\b/ will match the string 99inThere are 99 bottles of beer on\nthe wall (because 99 follows a space) but not 99inThere are 299 bottles of beer on\nthe wall (since 99 follows a number). But it will match 99in$99(since 99follows\na dollar sign ($), which is not a digit, underscore, or letter).\n2.1.2 Disjunction, Grouping, and Precedence\nSuppose we need to search for texts about pets; perhaps we are particularly interested\nin cats and dogs. In such a case, we might want to search for either the string cator\nthe string dog. Since we can\u2019t use the square brackets to search for \u201ccat or dog\u201d (why\ncan\u2019t we say /[catdog]/ ?), we need a new operator, the disjunction operator, also disjunction\ncalled the pipe symbol|. The pattern /cat|dog/ matches either the string cator\nthe string dog.\nSometimes we need to use this disjunction operator in the midst of a larger se-\nquence. For example, suppose I want to search for information about pet \ufb01sh for\nmy cousin David. How can I specify both guppy andguppies ? We cannot simply\nsay/guppy|ies/ , because that would match only the strings guppy andies. This\nis because sequences like guppy take precedence over the disjunction operator |. precedence\nTo make the disjunction operator apply only to a speci\ufb01c pattern, we need to use the\nparenthesis operators (and). Enclosing a pattern in parentheses makes it act like\na single character for the purposes of neighboring operators like the pipe |and the\nKleene*. So the pattern /gupp(y|ies)/ would specify that we meant the disjunc-\ntion only to apply to the suf\ufb01xes yandies.\nThe parenthesis operator (is also useful when we are using counters like the\nKleene*. Unlike the |operator, the Kleene *operator applies by default only to\na single character, not to a whole sequence. Suppose we want to match repeated\ninstances of a string. Perhaps we have a line that has column labels of the form\nColumn 1 Column 2 Column 3 . The expression /Column [0-9]+ */ will not\nmatch any number of columns; instead, it will match a single column followed by 6CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\nany number of spaces! The star here applies only to the space  that precedes it,\nnot to the whole sequence. With the parentheses, we could write the expression\n/(Column [0-9]+ *)*/ to match the word Column , followed by a number and\noptional spaces, the whole pattern repeated zero or more times.\nThis idea that one operator may take precedence over another, requiring us to\nsometimes use parentheses to specify what we mean, is formalized by the operator\nprecedence hierarchy for regular expressions. The following table gives the orderoperator\nprecedence\nof RE operator precedence, from highest precedence to lowest precedence.\nParenthesis ()\nCounters * + ? {}\nSequences and anchors the ^my end$\nDisjunction |\nThus, because counters have a higher precedence than sequences,\n/the*/ matches theeeee but not thethe . Because sequences have a higher prece-\ndence than disjunction, /the|any/ matches theoranybut not thany ortheny .\nPatterns can be ambiguous in another way. Consider the expression /[a-z]*/\nwhen matching against the text once upon a time . Since/[a-z]*/ matches zero or\nmore letters, this expression could match nothing, or just the \ufb01rst letter o,on,onc,\noronce . In these cases regular expressions always match the largest string they can;\nwe say that patterns are greedy , expanding to cover as much of a string as they can. greedy\nThere are, however, ways to enforce non-greedy matching, using another mean- non-greedy\ning of the ?quali\ufb01er. The operator *?is a Kleene star that matches as little text as *?\npossible. The operator +?is a Kleene plus that matches as little text as possible. +?\n2.1.3 A Simple Example\nSuppose we wanted to write a RE to \ufb01nd cases of the English article the. A simple\n(but incorrect) pattern might be:\n/the/ (2.2)\nOne problem is that this pattern will miss the word when it begins a sentence and\nhence is capitalized (i.e., The). This might lead us to the following pattern:\n/[tT]he/ (2.3)\nBut we will still incorrectly return texts with the embedded in other words (e.g.,\nother ortheology ). So we need to specify that we want instances with a word bound-\nary on both sides:\n/\\b[tT]he\\b/ (2.4)\nSuppose we wanted to do this without the use of /\\b/ . We might want this since\n/\\b/ won\u2019t treat underscores and numbers as word boundaries; but we might want\nto \ufb01nd thein some context where it might also have underlines or numbers nearby\n(the orthe25 ). We need to specify that we want instances in which there are no\nalphabetic letters on either side of the the:\n/[^a-zA-Z][tT]he[^a-zA-Z]/ (2.5)\nBut there is still one more problem with this pattern: it won\u2019t \ufb01nd the word thewhen\nit begins a line. This is because the regular expression [^a-zA-Z] , which we used 2.1 \u2022 R EGULAR EXPRESSIONS 7\nto avoid embedded instances of the, implies that there must be some single (although\nnon-alphabetic) character before the the. We can avoid this by specifying that before\nthethewe require either the beginning-of-line or a non-alphabetic character, and the\nsame at the end of the line:\n/(^|[^a-zA-Z])[tT]he([^a-zA-Z]|$)/ (2.6)\nThe process we just went through was based on \ufb01xing two kinds of errors: false pos-\nitives , strings that we incorrectly matched like other orthere , and false negatives , false positives\nfalse negatives strings that we incorrectly missed, like The. Addressing these two kinds of errors\ncomes up again and again in language processing. Reducing the overall error rate\nfor an application thus involves two antagonistic efforts:\n\u2022 Increasing precision (minimizing false positives)\n\u2022 Increasing recall (minimizing false negatives)\nWe\u2019ll come back to precision and recall with more precise de\ufb01nitions in Chapter 4.\n2.1.4 More Operators\nFigure 2.8 shows some aliases for common ranges, which can be used mainly to\nsave typing. Besides the Kleene * and Kleene + we can also use explicit numbers as\ncounters, by enclosing them in curly brackets. The operator /{3}/ means \u201cexactly\n3 occurrences of the previous character or expression\u201d. So /a\\.{24}z/ will match\nafollowed by 24 dots followed by z(but not afollowed by 23 or 25 dots followed\nby a z).\nRegex Expansion Match First Matches\n\\d [0-9] any digit Party of 5\n\\D [^0-9] any non-digit Blue moon\n\\w [a-zA-Z0-9_] any alphanumeric/underscore Daiyu\n\\W [^\\w] a non-alphanumeric !!!!\n\\s [ \\r\\t\\n\\f] whitespace (space, tab) inConcord\n\\S [^\\s] Non-whitespace in Concord\nFigure 2.8 Aliases for common sets of characters.\nA range of numbers can also be speci\ufb01ed. So /{n,m}/ speci\ufb01es from ntom\noccurrences of the previous char or expression, and /{n,}/ means at least noccur-\nrences of the previous expression. REs for counting are summarized in Fig. 2.9.\nRegex Match\n* zero or more occurrences of the previous char or expression\n+ one or more occurrences of the previous char or expression\n? zero or one occurrence of the previous char or expression\n{n} exactly noccurrences of the previous char or expression\n{n,m} from ntomoccurrences of the previous char or expression\n{n,} at least noccurrences of the previous char or expression\n{,m} up to moccurrences of the previous char or expression\nFigure 2.9 Regular expression operators for counting.\nFinally, certain special characters are referred to by special notation based on the\nbackslash ( \\) (see Fig. 2.10). The most common of these are the newline character newline\n\\nand the tabcharacter\\t. To refer to characters that are special themselves (like\n.,*,[, and\\), precede them with a backslash, (i.e., /\\./ ,/\\*/ ,/\\[/ , and/\\\\/ ). 8CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\nRegex Match First Patterns Matched\n\\* an asterisk \u201c*\u201d \u201cK* A*P*L*A*N\u201d\n\\. a period \u201c.\u201d \u201cDr. Livingston, I presume\u201d\n\\? a question mark \u201cWhy don\u2019t they come and lend a hand? \u201d\n\\n a newline\n\\t a tab\nFigure 2.10 Some characters that need to be escaped (via backslash).\n2.1.5 A More Complex Example\nLet\u2019s try out a more signi\ufb01cant example of the power of REs. Suppose our goal is\nhelp a user buy a computer on the Web who wants \u201cat least 6 GHz and 500 GB of\ndisk space for less than $1000\u201d. To do this kind of retrieval, we \ufb01rst need to be\nable to look for expressions like 6 GHz or500 GB or$999.99 . Let\u2019s work out some\nregular expressions for this task.\nFirst, let\u2019s complete our regular expression for prices. Here\u2019s a regular expres-\nsion for a dollar sign followed by a string of digits:\n/$[0-9]+/ (2.7)\nNote that the $character has a different function here than the end-of-line function\nwe discussed earlier. Most regular expression parsers are smart enough to realize\nthat$here doesn\u2019t mean end-of-line. (As a thought experiment, think about how\nregex parsers might \ufb01gure out the function of $from the context.)\nNow we just need to deal with fractions of dollars. We\u2019ll add a decimal point\nand two digits afterwards:\n/$[0-9]+\\.[0-9][0-9]/ (2.8)\nThis pattern only allows $199.99 but not $199 . We need to make the cents optional\nand to make sure we\u2019re at a word boundary:\n/(^|\\W)$[0-9]+(\\.[0-9][0-9])?\\b/ (2.9)\nOne last catch! This pattern allows prices like $199999.99 which would be far too\nexpensive! We need to limit the dollars:\n/(^|\\W)$[0-9]{0,3}(\\.[0-9][0-9])?\\b/ (2.10)\nFurther \ufb01xes (like avoiding matching a dollar sign with no price after it) are left as\nan exercise for the reader.\nHow about disk space? We\u2019ll need to allow for optional fractions again ( 5.5 GB );\nnote the use of ?for making the \ufb01nal soptional, and the use of / */ to mean \u201czero\nor more spaces\u201d since there might always be extra spaces lying around:\n/\\b[0-9]+(\\.[0-9]+)? *(GB|[Gg]igabytes?)\\b/ (2.11)\nModifying this regular expression so that it only matches more than 500 GB is left\nas an exercise for the reader.\n2.1.6 Substitution, Capture Groups, and ELIZA\nAn important use of regular expressions is in substitutions . For example, the substi- substitution 2.1 \u2022 R EGULAR EXPRESSIONS 9\ntution operator s/regexp1/pattern/ used in Python and in Unix commands like\nvimorsedallows a string characterized by a regular expression to be replaced by\nanother string:\ns/colour/color/ (2.12)\nIt is often useful to be able to refer to a particular subpart of the string matching\nthe \ufb01rst pattern. For example, suppose we wanted to put angle brackets around all\nintegers in a text, for example, changing the 35 boxes tothe<35>boxes . We\u2019d\nlike a way to refer to the integer we\u2019ve found so that we can easily add the brackets.\nTo do this, we put parentheses (and)around the \ufb01rst pattern and use the number\noperator\\1in the second pattern to refer back. Here\u2019s how it looks:\ns/([0-9]+)/<\\1>/ (2.13)\nThe parenthesis and number operators can also specify that a certain string or ex-\npression must occur twice in the text. For example, suppose we are looking for the\npattern \u201cthe Xer they were, the Xer they will be\u201d, where we want to constrain the two\nX\u2019s to be the same string. We do this by surrounding the \ufb01rst X with the parenthesis\noperator, and replacing the second X with the number operator \\1, as follows:\n/the (.*)er they were, the \\1er they will be/ (2.14)\nHere the\\1will be replaced by whatever string matched the \ufb01rst item in parentheses.\nSo this will match the bigger they were, the bigger they will be but not the bigger\nthey were, the faster they will be .\nThis use of parentheses to store a pattern in memory is called a capture group . capture group\nEvery time a capture group is used (i.e., parentheses surround a pattern), the re-\nsulting match is stored in a numbered register . If you match two different sets of register\nparentheses, \\2means whatever matched the second capture group. Thus\n/the (.*)er they (.*), the \\1er we \\2/ (2.15)\nwill match the faster they ran, the faster we ran but not the faster they ran, the faster\nwe ate . Similarly, the third capture group is stored in \\3, the fourth is \\4, and so on.\nParentheses thus have a double function in regular expressions; they are used\nto group terms for specifying the order in which operators should apply, and they\nare used to capture something in a register. Occasionally we might want to use\nparentheses for grouping, but don\u2019t want to capture the resulting pattern in a register.\nIn that case we use a non-capturing group , which is speci\ufb01ed by putting the specialnon-capturing\ngroup\ncommands ?:after the open parenthesis, in the form (?: pattern ) .\n/(?:some|a few) (people|cats) like some \\1/ (2.16)\nwill match some cats like some cats but not some cats like some some .\nSubstitutions and capture groups are very useful in implementing simple chat-\nbots like ELIZA (Weizenbaum, 1966). Recall that ELIZA simulates a Rogerian\npsychologist by carrying on conversations like the following:\nUser 1: Men are all alike.\nELIZA 1:IN WHAT WAY\nUser 2: They\u2019re always bugging us about something or other.\nELIZA 2:CAN YOU THINK OF A SPECIFIC EXAMPLE\nUser 3: Well, my boyfriend made me come here.\nELIZA 3:YOUR BOYFRIEND MADE YOU COME HERE\nUser 4: He says I\u2019m depressed much of the time.\nELIZA 4:I AM SORRY TO HEAR YOU ARE DEPRESSED 10 CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\nELIZA works by having a series or cascade of regular expression substitutions\neach of which matches and changes some part of the input lines. After the input\nis uppercased, substitutions change all instances of MYtoYOUR , and I\u2019MtoYOU\nARE, and so on. That way when ELIZA repeats back part of the user utterance, it\nwill seem to be referring correctly to the user. The next set of substitutions matches\nand replaces other patterns in the input. Here are some examples:\ns/.* YOU ARE (depressed|sad) .*/I AM SORRY TO HEAR YOU ARE \\1/\ns/.* YOU ARE (depressed|sad) .*/WHY DO YOU THINK YOU ARE \\1/\ns/.* all .*/IN WHAT WAY/\ns/.* always .*/CAN YOU THINK OF A SPECIFIC EXAMPLE/\nSince multiple substitutions can apply to a given input, substitutions are assigned\na rank and applied in order. Creating patterns is the topic of Exercise 2.3, and we\nreturn to the details of the ELIZA architecture in Chapter 15.\n2.1.7 Lookahead Assertions\nFinally, there will be times when we need to predict the future: look ahead in the\ntext to see if some pattern matches, but not yet advance the pointer we always keep\nto where we are in the text, so that we can then deal with the pattern if it occurs, but\nif it doesn\u2019t we can check for something else instead.\nThese lookahead assertions make use of the (?syntax that we saw in the previ- lookahead\nous section for non-capture groups. The operator (?= pattern) is true ifpattern\noccurs, but is zero-width , i.e. the match pointer doesn\u2019t advance. The operator zero-width\n(?! pattern) only returns true if a pattern does not match, but again is zero-width\nand doesn\u2019t advance the pointer. Negative lookahead is commonly used when we\nare parsing some complex pattern but want to rule out a special case. For example\nsuppose we want to match, at the beginning of a line, any single word that doesn\u2019t\nstart with \u201cV olcano\u201d. We can use negative lookahead to do this:\n/^(?!Volcano)[A-Za-z]+/ (2.17)\n2.2 Words\nBefore we talk about processing words, we need to decide what counts as a word.\nLet\u2019s start by looking at one particular corpus (plural corpora ), a computer-readable corpus\ncorpora collection of text or speech. For example the Brown corpus is a million-word col-\nlection of samples from 500 written English texts from different genres (newspa-\nper, \ufb01ction, non-\ufb01ction, academic, etc.), assembled at Brown University in 1963\u201364\n(Ku\u02c7cera and Francis, 1967). How many words are in the following Brown sentence?\nHe stepped out into the hall, was delighted to encounter\na water brother.\nThis sentence has 13 words if we don\u2019t count punctuation marks as words, 15\nif we count punctuation. Whether we treat period (\u201c .\u201d), comma (\u201c ,\u201d), and so on as\nwords depends on the task. Punctuation is critical for \ufb01nding boundaries of things\n(commas, periods, colons) and for identifying some aspects of meaning (question\nmarks, exclamation marks, quotation marks). For some tasks, like part-of-speech\ntagging or parsing or speech synthesis, we sometimes treat punctuation marks as if\nthey were separate words. 2.2 \u2022 W ORDS 11\nThe Switchboard corpus of American English telephone conversations between\nstrangers was collected in the early 1990s; it contains 2430 conversations averaging\n6 minutes each, totaling 240 hours of speech and about 3 million words (Godfrey\net al., 1992). Such corpora of spoken language introduce other complications with\nregard to de\ufb01ning words. Let\u2019s look at one utterance from Switchboard; an utter-\nance is the spoken correlate of a sentence: utterance\nI do uh main- mainly business data processing\nThis utterance has two kinds of dis\ufb02uencies . The broken-off word main- is dis\ufb02uency\ncalled a fragment . Words like uhandumare called \ufb01llers or\ufb01lled pauses . Should fragment\n\ufb01lled pause we consider these to be words? Again, it depends on the application. If we are\nbuilding a speech transcription system, we might want to eventually strip out the\ndis\ufb02uencies.\nBut we also sometimes keep dis\ufb02uencies around. Dis\ufb02uencies like uhorum\nare actually helpful in speech recognition in predicting the upcoming word, because\nthey may signal that the speaker is restarting the clause or idea, and so for speech\nrecognition they are treated as regular words. Because different people use differ-\nent dis\ufb02uencies they can also be a cue to speaker identi\ufb01cation. In fact Clark and\nFox Tree (2002) showed that uhandumhave different meanings. What do you think\nthey are?\nPerhaps most important, in thinking about what is a word, we need to distinguish\ntwo ways of talking about words that will be useful throughout the book. Word types word type\nare the number of distinct words in a corpus; if the set of words in the vocabulary is\nV, the number of types is the vocabulary size jVj. Word instances are the total num- word instance\nberNof running words.1If we ignore punctuation, the following Brown sentence\nhas 14 types and 16 instances:\nThey picnicked by the pool, then lay back on the grass and\nlooked at the stars.\nWe still have decisions to make! For example, should we consider a capitalized\nstring (like They ) and one that is uncapitalized (like they) to be the same word type?\nThe answer is that it depends on the task! They andthey might be lumped together\nas the same type in some tasks, like speech recognition, where we care more about\nthe sequence of words and less about the formatting, while for other tasks, such\nas deciding whether a particular word is a name of a person or location (named-\nentity tagging), capitalization is a useful feature and is retained. Sometimes we keep\naround two versions of a particular NLP model, one with capitalization and one\nwithout capitalization.\nCorpus Types =jVjInstances = N\nShakespeare 31 thousand 884 thousand\nBrown corpus 38 thousand 1 million\nSwitchboard telephone conversations 20 thousand 2.4 million\nCOCA 2 million 440 million\nGoogle n-grams 13 million 1 trillion\nFigure 2.11 Rough numbers of wordform types and instances for some English language\ncorpora. The largest, the Google n-grams corpus, contains 13 million types, but this count\nonly includes types appearing 40 or more times, so the true number would be much larger.\n1In earlier tradition, and occasionally still, you might see word instances referred to as word tokens , but\nwe now try to reserve the word token instead to mean the output of subword tokenization algorithms. 12 CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\nHow many words are there in English? When we speak about the number of\nwords in the language, we are generally referring to word types. Fig. 2.11 shows\nthe rough numbers of types and instances computed from some English corpora.\nThe larger the corpora we look at, the more word types we \ufb01nd, and in fact this\nrelationship between the number of types jVjand number of instances Nis called\nHerdan\u2019s Law (Herdan, 1960) or Heaps\u2019 Law (Heaps, 1978) after its discoverers Herdan\u2019s Law\nHeaps\u2019 Law (in linguistics and information retrieval respectively). It is shown in Eq. 2.18, where\nkandbare positive constants, and 0 <b<1.\njVj=kNb(2.18)\nThe value of bdepends on the corpus size and the genre, but at least for the large\ncorpora in Fig. 2.11, branges from .67 to .75. Roughly then we can say that the\nvocabulary size for a text goes up signi\ufb01cantly faster than the square root of its\nlength in words.\nIt\u2019s sometimes useful to make a further distinction. Consider in\ufb02ected forms\nlikecats versus cat. We say these two words are different wordforms but have the\nsame lemma . Alemma is a set of lexical forms having the same stem, the same lemma\nmajor part-of-speech, and the same word sense. The wordform is the full in\ufb02ected wordform\nor derived form of the word. The two wordforms catandcats thus have the same\nlemma, which we can represent as cat.\nFor morphologically complex languages like Arabic, we often need to deal with\nlemmatization. For most tasks in English, however, wordforms are suf\ufb01cient, and\nwhen we talk about words in this book we almost always mean wordsforms (al-\nthough we will discuss basic algorithms for lemmatization and the related task of\nstemming below in Section 2.6). One of the situations even in English where we\ntalk about lemmas is when we measure the number of words in a dictionary. Dictio-\nnary entries orboldface forms are a very rough approximation to (an upper bound\non) the number of lemmas (since some lemmas have multiple boldface forms). The\n1989 edition of the Oxford English Dictionary had 615,000 entries.\nFinally, we should note that in practice, for many NLP applications (for example\nfor neural language modeling) we don\u2019t actually use words as our internal unit of\nrepresentation at all! We instead tokenize the input strings into tokens , which can\nbe words but can also be only parts of words. We\u2019ll return to this tokenization\nquestion when we introduce the BPE algorithm in Section 2.5.2.\n2.3 Corpora\nWords don\u2019t appear out of nowhere. Any particular piece of text that we study\nis produced by one or more speci\ufb01c speakers or writers, in a speci\ufb01c dialect of a\nspeci\ufb01c language, at a speci\ufb01c time, in a speci\ufb01c place, for a speci\ufb01c function.\nPerhaps the most important dimension of variation is the language. NLP algo-\nrithms are most useful when they apply across many languages. The world has 7097\nlanguages at the time of this writing, according to the online Ethnologue catalog\n(Simons and Fennig, 2018). It is important to test algorithms on more than one lan-\nguage, and particularly on languages with different properties; by contrast there is\nan unfortunate current tendency for NLP algorithms to be developed or tested just\non English (Bender, 2019). Even when algorithms are developed beyond English,\nthey tend to be developed for the of\ufb01cial languages of large industrialized nations 2.3 \u2022 C ORPORA 13\n(Chinese, Spanish, Japanese, German etc.), but we don\u2019t want to limit tools to just\nthese few languages. Furthermore, most languages also have multiple varieties, of-\nten spoken in different regions or by different social groups. Thus, for example,\nif we\u2019re processing text that uses features of African American English ( AAE ) or AAE\nAfrican American Vernacular English (AA VE)\u2014the variations of English used by\nmillions of people in African American communities (King 2020)\u2014we must use\nNLP tools that function with features of those varieties. Twitter posts might use fea-\ntures often used by speakers of African American English, such as constructions like\niont(I don\u2019t in Mainstream American English ( MAE )), or talmbout corresponding MAE\nto MAE talking about , both examples that in\ufb02uence word segmentation (Blodgett\net al. 2016, Jones 2015).\nIt\u2019s also quite common for speakers or writers to use multiple languages in a\nsingle communicative act, a phenomenon called code switching . Code switching code switching\nis enormously common across the world; here are examples showing Spanish and\n(transliterated) Hindi code switching with English (Solorio et al. 2014, Jurgens et al.\n2017):\n(2.19) Por primera vez veo a @username actually being hateful! it was beautiful:)\n[For the \ufb01rst time I get to see @username actually being hateful! it was\nbeautiful:) ]\n(2.20) dost tha or ra- hega ... dont wory ... but dherya rakhe\n[\u201che was and will remain a friend ... don\u2019t worry ... but have faith\u201d]\nAnother dimension of variation is the genre. The text that our algorithms must\nprocess might come from newswire, \ufb01ction or non-\ufb01ction books, scienti\ufb01c articles,\nWikipedia, or religious texts. It might come from spoken genres like telephone\nconversations, business meetings, police body-worn cameras, medical interviews,\nor transcripts of television shows or movies. It might come from work situations\nlike doctors\u2019 notes, legal text, or parliamentary or congressional proceedings.\nText also re\ufb02ects the demographic characteristics of the writer (or speaker): their\nage, gender, race, socioeconomic class can all in\ufb02uence the linguistic properties of\nthe text we are processing.\nAnd \ufb01nally, time matters too. Language changes over time, and for some lan-\nguages we have good corpora of texts from different historical periods.\nBecause language is so situated, when developing computational models for lan-\nguage processing from a corpus, it\u2019s important to consider who produced the lan-\nguage, in what context, for what purpose. How can a user of a dataset know all these\ndetails? The best way is for the corpus creator to build a datasheet (Gebru et al., datasheet\n2020) or data statement (Bender et al., 2021) for each corpus. A datasheet speci\ufb01es\nproperties of a dataset like:\nMotivation : Why was the corpus collected, by whom, and who funded it?\nSituation : When and in what situation was the text written/spoken? For example,\nwas there a task? Was the language originally spoken conversation, edited\ntext, social media communication, monologue vs. dialogue?\nLanguage variety : What language (including dialect/region) was the corpus in?\nSpeaker demographics : What was, e.g., the age or gender of the text\u2019s authors?\nCollection process : How big is the data? If it is a subsample how was it sampled?\nWas the data collected with consent? How was the data pre-processed, and\nwhat metadata is available?\nAnnotation process : What are the annotations, what are the demographics of the\nannotators, how were they trained, how was the data annotated? 14 CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\nDistribution : Are there copyright or other intellectual property restrictions?\n2.4 Simple Unix Tools for Word Tokenization\nBefore almost any natural language processing of a text, the text has to be normal-\nized, a task called text normalization . At least three tasks are commonly applied astext\nnormalization\npart of any normalization process:\n1. Tokenizing (segmenting) words\n2. Normalizing word formats\n3. Segmenting sentences\nIn the next sections we walk through each of these tasks, but we\u2019ll \ufb01rst start with\nan easy, if somewhat naive version of word tokenization and normalization (and fre-\nquency computation) that can be accomplished for English solely in a single Unix\ncommand-line, inspired by Church (1994). We\u2019ll make use of some Unix com-\nmands:tr, used to systematically change particular characters in the input; sort ,\nwhich sorts input lines in alphabetical order; and uniq , which collapses and counts\nadjacent identical lines.\nFor example let\u2019s begin with the \u2018complete words\u2019 of Shakespeare in one \ufb01le,\nsh.txt . We can use trto tokenize the words by changing every sequence of non-\nalphabetic characters to a newline (\u2019A-Za-z\u2019 means alphabetic and the -c option\ncomplements to non-alphabet, so together they mean to change every non-alphabetic\ncharacter into a newline. The -s (\u2018squeeze\u2019) option is used to replace the result\nof multiple consecutive changes into a single output, so a series of non-alphabetic\ncharacters in a row would all be \u2018squeezed\u2019 into a single newline):\ntr -sc 'A-Za-z ' '\\n'< sh.txt\nThe output of this command will be:\nTHE\nSONNETS\nby\nWilliam\nShakespeare\nFrom\nfairest\ncreatures\nWe\n...\nNow that there is one word per line, we can sort the lines, and pass them to uniq\n-cwhich will collapse and count them:\ntr -sc 'A-Za-z' '\\n' < sh.txt | sort | uniq -c\nwith the following output:\n1945 A\n72 AARON\n19 ABBESS\n25 Aaron\n6 Abate 2.5 \u2022 W ORD AND SUBWORD TOKENIZATION 15\n1 Abates\n5 Abbess\n6 Abbey\n3 Abbot\n...\nAlternatively, we can collapse all the upper case to lower case:\ntr -sc 'A-Za-z ' '\\n'< sh.txt | tr A-Z a-z | sort | uniq -c\nwhose output is\n14725 a\n97 aaron\n1 abaissiez\n10 abandon\n2 abandoned\n2 abase\n1 abash\n14 abate\n3 abated\n3 abatement\n...\nNow we can sort again to \ufb01nd the frequent words. The -noption tosort means\nto sort numerically rather than alphabetically, and the -roption means to sort in\nreverse order (highest-to-lowest):\ntr -sc 'A-Za-z ' '\\n'< sh.txt | tr A-Z a-z | sort | uniq -c | sort -n -r\nThe results show that the most frequent words in Shakespeare, as in any other\ncorpus, are the short function words like articles, pronouns, prepositions:\n27378 the\n26084 and\n22538 i\n19771 to\n17481 of\n14725 a\n13826 you\n12489 my\n11318 that\n11112 in\n...\nUnix tools of this sort can be very handy in building quick word count statistics\nfor any corpus in English. While in some versions of Unix these command-line tools\nalso correctly handle Unicode characters and so can be used for many languages,\nin general for handling most languages outside English we use more sophisticated\ntokenization algorithms.\n2.5 Word and Subword Tokenization\nThe simple Unix tools above were \ufb01ne for getting rough word statistics but more\nsophisticated algorithms are generally necessary for tokenization , the task of seg- tokenization 16 CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\nmenting running text into words. There are roughly two classes of tokenization\nalgorithms. In top-down tokenization, we de\ufb01ne a standard and implement rules to\nimplement that kind of tokenization.\nBut more commonly instead of using words as the input to NLP algorithms we\nbreak up words into subword tokens , which can be words or parts of words or subword tokens\neven individual letters. These are derived via bottom-up tokenization, in which we\nuse simple statistics of letter sequences to come up with the vocabulary of subword\ntokens, and break up the input into those subwords.\n2.5.1 Top-down (rule-based) tokenization\nWhile the Unix command sequence just removed all the numbers and punctuation,\nfor most NLP applications we\u2019ll need to keep these in our tokenization. We often\nwant to break off punctuation as a separate token; commas are a useful piece of infor-\nmation for parsers, and periods help indicate sentence boundaries. But we\u2019ll often\nwant to keep the punctuation that occurs word internally, in examples like m.p.h. ,\nPh.D. ,AT&T , and cap\u2019n . Special characters and numbers will need to be kept in\nprices ($45.55) and dates ( 01/02/06 ); we don\u2019t want to segment that price into sepa-\nrate tokens of \u201c45\u201d and \u201c55\u201d. And there are URLs ( https://www.stanford.edu ),\nTwitter hashtags ( #nlproc ), or email addresses ( someone@cs.colorado.edu ).\nNumber expressions introduce complications; in addition to appearing at word\nboundaries, commas appear inside numbers in English, every three digits: 555,500.50 .\nTokenization differs by language; languages like Spanish, French, and German, for\nexample, use a comma to mark the decimal point, and spaces (or sometimes periods)\nwhere English puts commas, for example, 555 500,50 .\nA tokenizer can also be used to expand clitic contractions that are marked by clitic\napostrophes, converting what're to the two tokens what are , andwe're towe\nare. A clitic is a part of a word that can\u2019t stand on its own, and can only occur\nwhen it is attached to another word. Such contractions occur in other alphabetic\nlanguages, including French pronouns ( j'ai and articles l'homme ).\nDepending on the application, tokenization algorithms may also tokenize mul-\ntiword expressions like New York orrock 'n' roll as a single token, which re-\nquires a multiword expression dictionary of some sort. Tokenization is thus inti-\nmately tied up with named entity recognition , the task of detecting names, dates,\nand organizations (Chapter 17).\nOne commonly used tokenization standard is known as the Penn Treebank to-\nkenization standard, used for the parsed corpora (treebanks) released by the Lin-Penn Treebank\ntokenization\nguistic Data Consortium (LDC), the source of many useful datasets. This standard\nseparates out clitics ( doesn\u2019t becomes does plus n\u2019t), keeps hyphenated words to-\ngether, and separates out all punctuation (to save space we\u2019re showing visible spaces\n\u2018\u2019 between tokens, although newlines is a more common output):\nInput :\"The San Francisco-based restaurant,\" they said,\n\"doesn't charge $10\".\nOutput :\"TheSanFrancisco-based restaurant ,\"theysaid,\n\"doesn'tcharge $10\".\nIn practice, since tokenization is run before any other language processing, it\nneeds to be very fast. For word tokenziation we generally use deterministic algo-\nrithms based on regular expressions compiled into ef\ufb01cient \ufb01nite state automata.\nFor example, Fig. 2.12 shows a basic regular expression that can be used to tok- 2.5 \u2022 W ORD AND SUBWORD TOKENIZATION 17\nenize English with the nltk.regexp tokenize function of the Python-based Nat-\nural Language Toolkit (NLTK) (Bird et al. 2009; https://www.nltk.org ).\n>>> text = 'That U.S.A. poster-print costs $12.40... '\n>>> pattern = r '''(?x) # set flag to allow verbose regexps\n... (?:[A-Z]\\.)+ # abbreviations, e.g. U.S.A.\n... | \\w+(?:-\\w+)* # words with optional internal hyphens\n... | \\$?\\d+(?:\\.\\d+)?%? # currency, percentages, e.g. $12.40, 82%\n... | \\.\\.\\. # ellipsis\n... | [][.,;\" '?():_ `-] # these are separate tokens; includes ], [\n... '''\n>>> nltk.regexp_tokenize(text, pattern)\n['That ','U.S.A. ','poster-print ','costs ','$12.40 ','...']\nFigure 2.12 A Python trace of regular expression tokenization in the NLTK Python-based\nnatural language processing toolkit (Bird et al., 2009), commented for readability; the (?x)\nverbose \ufb02ag tells Python to strip comments and whitespace. Figure from Chapter 3 of Bird\net al. (2009).\nCarefully designed deterministic algorithms can deal with the ambiguities that\narise, such as the fact that the apostrophe needs to be tokenized differently when used\nas a genitive marker (as in the book\u2019s cover ), a quotative as in \u2018The other class\u2019, she\nsaid, or in clitics like they\u2019re .\nWord tokenization is more complex in languages like written Chinese, Japanese,\nand Thai, which do not use spaces to mark potential word-boundaries. In Chinese,\nfor example, words are composed of characters (called hanzi in Chinese). Each hanzi\ncharacter generally represents a single unit of meaning (called a morpheme ) and is\npronounceable as a single syllable. Words are about 2.4 characters long on average.\nBut deciding what counts as a word in Chinese is complex. For example, consider\nthe following sentence:\n(2.21)\u59da\u660e\u8fdb\u5165\u603b\u51b3\u8d5b y\u00b4ao m \u00b4\u0131ng j`\u0131n r`u z\u02c7ong ju \u00b4e s`ai\n\u201cYao Ming reaches the \ufb01nals\u201d\nAs Chen et al. (2017) point out, this could be treated as 3 words (\u2018Chinese Treebank\u2019\nsegmentation):\n(2.22)\u59da\u660e\nYaoMing\u8fdb\u5165\nreaches\u603b\u51b3\u8d5b\n\ufb01nals\nor as 5 words (\u2018Peking University\u2019 segmentation):\n(2.23)\u59da\nYao\u660e\nMing\u8fdb\u5165\nreaches\u603b\noverall\u51b3\u8d5b\n\ufb01nals\nFinally, it is possible in Chinese simply to ignore words altogether and use characters\nas the basic elements, treating the sentence as a series of 7 characters:\n(2.24)\u59da\nYao\u660e\nMing\u8fdb\nenter\u5165\nenter\u603b\noverall\u51b3\ndecision\u8d5b\ngame\nIn fact, for most Chinese NLP tasks it turns out to work better to take characters\nrather than words as input, since characters are at a reasonable semantic level for\nmost applications, and since most word standards, by contrast, result in a huge vo-\ncabulary with large numbers of very rare words (Li et al., 2019).\nHowever, for Japanese and Thai the character is too small a unit, and so algo-\nrithms for word segmentation are required. These can also be useful for Chineseword\nsegmentation 18 CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\nin the rare situations where word rather than character boundaries are required. For\nthese situations we can use the subword tokenization algorithms introduced in the\nnext section.\n2.5.2 Byte-Pair Encoding: A Bottom-up Tokenization Algorithm\nThere is a third option to tokenizing text, one that is most commonly used by large\nlanguage models. Instead of de\ufb01ning tokens as words (whether delimited by spaces\nor more complex algorithms), or as characters (as in Chinese), we can use our data to\nautomatically tell us what the tokens should be. This is especially useful in dealing\nwith unknown words, an important problem in language processing. As we will\nsee in the next chapter, NLP algorithms often learn some facts about language from\none corpus (a training corpus) and then use these facts to make decisions about a\nseparate testcorpus and its language. Thus if our training corpus contains, say the\nwords low,new,newer , but not lower , then if the word lower appears in our test\ncorpus, our system will not know what to do with it.\nTo deal with this unknown word problem, modern tokenizers automatically in-\nduce sets of tokens that include tokens smaller than words, called subwords . Sub- subwords\nwords can be arbitrary substrings, or they can be meaning-bearing units like the\nmorphemes -estor-er. (A morpheme is the smallest meaning-bearing unit of a lan-\nguage; for example the word unwashable has the morphemes un-,wash , and -able .)\nIn modern tokenization schemes, most tokens are words, but some tokens are fre-\nquently occurring morphemes or other subwords like -er. Every unseen word like\nlower can thus be represented by some sequence of known subword units, such as\nlowander, or even as a sequence of individual letters if necessary.\nMost tokenization schemes have two parts: a token learner , and a token seg-\nmenter . The token learner takes a raw training corpus (sometimes roughly pre-\nseparated into words, for example by whitespace) and induces a vocabulary, a set\nof tokens. The token segmenter takes a raw test sentence and segments it into the\ntokens in the vocabulary. Two algorithms are widely used: byte-pair encoding\n(Sennrich et al., 2016), and unigram language modeling (Kudo, 2018), There is\nalso a SentencePiece library that includes implementations of both of these (Kudo\nand Richardson, 2018), and people often use the name SentencePiece to simply\nmean unigram language modeling tokenization.\nIn this section we introduce the simplest of the three, the byte-pair encoding or\nBPE algorithm (Sennrich et al., 2016); see Fig. 2.13. The BPE token learner begins BPE\nwith a vocabulary that is just the set of all individual characters. It then examines the\ntraining corpus, chooses the two symbols that are most frequently adjacent (say \u2018A\u2019,\n\u2018B\u2019), adds a new merged symbol \u2018AB\u2019 to the vocabulary, and replaces every adjacent\n\u2019A\u2019 \u2019B\u2019 in the corpus with the new \u2018AB\u2019. It continues to count and merge, creating\nnew longer and longer character strings, until kmerges have been done creating\nknovel tokens; kis thus a parameter of the algorithm. The resulting vocabulary\nconsists of the original set of characters plus knew symbols.\nThe algorithm is usually run inside words (not merging across word boundaries),\nso the input corpus is \ufb01rst white-space-separated to give a set of strings, each corre-\nsponding to the characters of a word, plus a special end-of-word symbol , and its\ncounts. Let\u2019s see its operation on the following tiny input corpus of 18 word tokens\nwith counts for each word (the word lowappears 5 times, the word newer 6 times,\nand so on), which would have a starting vocabulary of 11 letters: 2.5 \u2022 W ORD AND SUBWORD TOKENIZATION 19\ncorpus vocabulary\n5l o w , d, e, i, l, n, o, r, s, t, w\n2l o w e s t\n6n e w e r\n3w i d e r\n2n e w\nThe BPE algorithm \ufb01rst counts all pairs of adjacent symbols: the most frequent\nis the pair e rbecause it occurs in newer (frequency of 6) and wider (frequency of\n3) for a total of 9 occurrences.2We then merge these symbols, treating eras one\nsymbol, and count again:\ncorpus vocabulary\n5l o w , d, e, i, l, n, o, r, s, t, w, er\n2l o w e s t\n6n e w er\n3w i d er\n2n e w\nNow the most frequent pair is er , which we merge; our system has learned\nthat there should be a token for word-\ufb01nal er, represented as er:\ncorpus vocabulary\n5l o w ,d,e,i,l,n,o,r,s,t,w,er,er\n2l o w e s t\n6n e w er\n3w i d er\n2n e w\nNextn e(total count of 8) get merged to ne:\ncorpus vocabulary\n5l o w ,d,e,i,l,n,o,r,s,t,w,er,er,ne\n2l o w e s t\n6ne w er\n3w i d er\n2ne w\nIf we continue, the next merges are:\nmerge current vocabulary\n(ne, w) ,d,e,i,l,n,o,r,s,t,w,er,er,ne,new\n(l, o) ,d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo\n(lo, w) ,d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low\n(new, er ) ,d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low,newer\n(low,) ,d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low,newer ,low\nOnce we\u2019ve learned our vocabulary, the token segmenter is used to tokenize a\ntest sentence. The token segmenter just runs on the merges we have learned from\nthe training data on the test data. It runs them greedily, in the order we learned them.\n(Thus the frequencies in the test data don\u2019t play a role, just the frequencies in the\ntraining data). So \ufb01rst we segment each test sentence word into characters. Then\nwe apply the \ufb01rst rule: replace every instance of e rin the test corpus with er, and\nthen the second rule: replace every instance of er in the test corpus with er,\nand so on. By the end, if the test corpus contained the character sequence n e w e\n2Note that there can be ties; we could have instead chosen to merge r \ufb01rst, since that also has a\nfrequency of 9. 20 CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\nfunction BYTE-PAIR ENCODING (strings C, number of merges k)returns vocab V\nV all unique characters in C # initial set of tokens is characters\nfori= 1tokdo # merge tokens ktimes\ntL,tR Most frequent pair of adjacent tokens in C\ntNEW tL+tR # make new token by concatenating\nV V+tNEW # update the vocabulary\nReplace each occurrence of tL,tRinCwith tNEW # and update the corpus\nreturn V\nFigure 2.13 The token learner part of the BPE algorithm for taking a corpus broken up\ninto individual characters or bytes, and learning a vocabulary by iteratively merging tokens.\nFigure adapted from Bostrom and Durrett (2020).\nr, it would be tokenized as a full word. But the characters of a new (unknown)\nword like l o w e r would be merged into the two tokens lower .\nOf course in real settings BPE is run with many thousands of merges on a very\nlarge input corpus. The result is that most words will be represented as full symbols,\nand only the very rare words (and unknown words) will have to be represented by\ntheir parts.\n2.6 Word Normalization, Lemmatization and Stemming\nWord normalization is the task of putting words or tokens in a standard format. The normalization\nsimplest case of word normalization is case folding . Mapping everything to lower case folding\ncase means that Woodchuck andwoodchuck are represented identically, which is\nvery helpful for generalization in many tasks, such as information retrieval or speech\nrecognition. For sentiment analysis and other text classi\ufb01cation tasks, information\nextraction, and machine translation, by contrast, case can be quite helpful and case\nfolding is generally not done. This is because maintaining the difference between,\nfor example, USthe country and usthe pronoun can outweigh the advantage in\ngeneralization that case folding would have provided for other words. Sometimes\nwe produce both cased (i.e. including both upper and lower case words or tokens)\nand uncased versions of language models.\nSystems that use BPE or other kinds of bottom-up tokenization may do no fur-\nther word normalization. In other NLP systems, we may want to do further nor-\nmalizations, like choosing a single normal form for words with multiple forms like\nUSA andUSoruh-huh anduhhuh . This standardization may be valuable, despite\nthe spelling information that is lost in the normalization process. For information\nretrieval or information extraction about the US, we might want to see information\nfrom documents whether they mention the USor theUSA.\n2.6.1 Lemmatization\nFor other natural language processing situations we also want two morphologically\ndifferent forms of a word to behave similarly. For example in web search, someone\nmay type the string woodchucks but a useful system might want to also return pages\nthat mention woodchuck with no s. This is especially common in morphologically\ncomplex languages like Polish, where for example the word Warsaw has different 2.6 \u2022 W ORD NORMALIZATION , LEMMATIZATION AND STEMMING 21\nendings when it is the subject ( Warszawa ), or after a preposition like \u201cin Warsaw\u201d ( w\nWarszawie ), or \u201cto Warsaw\u201d ( do Warszawy ), and so on. Lemmatization is the task lemmatization\nof determining that two words have the same root, despite their surface differences.\nThe words am,are, and ishave the shared lemma be; the words dinner anddinners\nboth have the lemma dinner . Lemmatizing each of these forms to the same lemma\nwill let us \ufb01nd all mentions of words in Polish like Warsaw . The lemmatized form\nof a sentence like He is reading detective stories would thus be He be read detective\nstory .\nHow is lemmatization done? The most sophisticated methods for lemmatization\ninvolve complete morphological parsing of the word. Morphology is the study of\nthe way words are built up from smaller meaning-bearing units called morphemes . morpheme\nTwo broad classes of morphemes can be distinguished: stems \u2014the central mor- stem\npheme of the word, supplying the main meaning\u2014and af\ufb01xes \u2014adding \u201cadditional\u201d af\ufb01x\nmeanings of various kinds. So, for example, the word foxconsists of one morpheme\n(the morpheme fox) and the word cats consists of two: the morpheme catand the\nmorpheme -s. A morphological parser takes a word like cats and parses it into the\ntwo morphemes catands, or parses a Spanish word like amaren (\u2018if in the future\nthey would love\u2019) into the morpheme amar \u2018to love\u2019, and the morphological features\n3PL (third person plural) and future subjunctive .\nStemming: The Porter Stemmer\nLemmatization algorithms can be complex. For this reason we sometimes make\nuse of a simpler but cruder method, which mainly consists of chopping off word-\n\ufb01nal af\ufb01xes. This naive version of morphological analysis is called stemming . For stemming\nexample, the classic Porter stemmer (Porter, 1980), when applied to the following Porter stemmer\nparagraph:\nThis was not the map we found in Billy Bones's chest, but\nan accurate copy, complete in all things-names and heights\nand soundings-with the single exception of the red crosses\nand the written notes.\nproduces the following stemmed output:\nThi wa not the map we found in Billi Bone s chest but an\naccur copi complet in all thing name and height and sound\nwith the singl except of the red cross and the written note\nThe algorithm is based on rewrite rules run in series, with the output of each pass\nfed as input to the next pass. Some sample rules (more at https://tartarus.org/\nmartin/PorterStemmer/ ):\nATIONAL!ATE (e.g., relational !relate)\nING!\u000fif the stem contains a vowel (e.g., motoring !motor)\nSSES!SS (e.g., grasses !grass)\nSimple stemmers can be useful in cases where we need to collapse across dif-\nferent variants of the same lemma. Nonetheless, they are less commonly used in\nmodern systems since they commit errors of both over-generalizating (lemmatizing\npolicy topolice ) and under-generalizing (not lemmatizing European toEurope )\n(Krovetz, 1993). 22 CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\n2.7 Sentence Segmentation\nSentence segmentation is another important step in text processing. The most use-sentence\nsegmentation\nful cues for segmenting a text into sentences are punctuation, like periods, question\nmarks, and exclamation points. Question marks and exclamation points are rela-\ntively unambiguous markers of sentence boundaries. Periods, on the other hand, are\nmore ambiguous. The period character \u201c.\u201d is ambiguous between a sentence bound-\nary marker and a marker of abbreviations like Mr.orInc.The previous sentence that\nyou just read showed an even more complex case of this ambiguity, in which the \ufb01nal\nperiod of Inc. marked both an abbreviation and the sentence boundary marker. For\nthis reason, sentence tokenization and word tokenization may be addressed jointly.\nIn general, sentence tokenization methods work by \ufb01rst deciding (based on rules\nor machine learning) whether a period is part of the word or is a sentence-boundary\nmarker. An abbreviation dictionary can help determine whether the period is part\nof a commonly used abbreviation; the dictionaries can be hand-built or machine-\nlearned (Kiss and Strunk, 2006), as can the \ufb01nal sentence splitter. In the Stanford\nCoreNLP toolkit (Manning et al., 2014), for example sentence splitting is rule-based,\na deterministic consequence of tokenization; a sentence ends when a sentence-ending\npunctuation (., !, or ?) is not already grouped with other characters into a token (such\nas for an abbreviation or number), optionally followed by additional \ufb01nal quotes or\nbrackets.\n2.8 Minimum Edit Distance\nMuch of natural language processing is concerned with measuring how similar two\nstrings are. For example in spelling correction, the user typed some erroneous\nstring\u2014let\u2019s say graffe \u2013and we want to know what the user meant. The user prob-\nably intended a word that is similar to graffe . Among candidate similar words,\nthe wordgiraffe , which differs by only one letter from graffe , seems intuitively\nto be more similar than, say grail orgraf , which differ in more letters. Another\nexample comes from coreference , the task of deciding whether two strings such as\nthe following refer to the same entity:\nStanford President Marc Tessier-Lavigne\nStanford University President Marc Tessier-Lavigne\nAgain, the fact that these two strings are very similar (differing by only one word)\nseems like useful evidence for deciding that they might be coreferent.\nEdit distance gives us a way to quantify both of these intuitions about string sim-\nilarity. More formally, the minimum edit distance between two strings is de\ufb01nedminimum edit\ndistance\nas the minimum number of editing operations (operations like insertion, deletion,\nsubstitution) needed to transform one string into another.\nThe gap between intention andexecution , for example, is 5 (delete an i, substi-\ntuteeforn, substitute xfort, insertc, substitute uforn). It\u2019s much easier to see\nthis by looking at the most important visualization for string distances, an alignment alignment\nbetween the two strings, shown in Fig. 2.14. Given two sequences, an alignment is\na correspondence between substrings of the two sequences. Thus, we say Ialigns\nwith the empty string, NwithE, and so on. Beneath the aligned strings is another\nrepresentation; a series of symbols expressing an operation list for converting the 2.8 \u2022 M INIMUM EDITDISTANCE 23\ntop string into the bottom string: dfor deletion, sfor substitution, ifor insertion.\nINTE*NTION\njjjjjjjjjj\n*EXECUTION\nd s s i s\nFigure 2.14 Representing the minimum edit distance between two strings as an alignment .\nThe \ufb01nal row gives the operation list for converting the top string into the bottom string: d for\ndeletion, s for substitution, i for insertion.\nWe can also assign a particular cost or weight to each of these operations. The\nLevenshtein distance between two sequences is the simplest weighting factor in\nwhich each of the three operations has a cost of 1 (Levenshtein, 1966)\u2014we assume\nthat the substitution of a letter for itself, for example, tfort, has zero cost. The Lev-\nenshtein distance between intention andexecution is 5. Levenshtein also proposed\nan alternative version of his metric in which each insertion or deletion has a cost of\n1 and substitutions are not allowed. (This is equivalent to allowing substitution, but\ngiving each substitution a cost of 2 since any substitution can be represented by one\ninsertion and one deletion). Using this version, the Levenshtein distance between\nintention andexecution is 8.\n2.8.1 The Minimum Edit Distance Algorithm\nHow do we \ufb01nd the minimum edit distance? We can think of this as a search task, in\nwhich we are searching for the shortest path\u2014a sequence of edits\u2014from one string\nto another.\nn t e n t i o ni n t e c n t i o ni n x e n t i o ndelinssubsti n t e n t i o n\nFigure 2.15 Finding the edit distance viewed as a search problem\nThe space of all possible edits is enormous, so we can\u2019t search naively. However,\nlots of distinct edit paths will end up in the same state (string), so rather than recom-\nputing all those paths, we could just remember the shortest path to a state each time\nwe saw it. We can do this by using dynamic programming . Dynamic programmingdynamic\nprogramming\nis the name for a class of algorithms, \ufb01rst introduced by Bellman (1957), that apply\na table-driven method to solve problems by combining solutions to subproblems.\nSome of the most commonly used algorithms in natural language processing make\nuse of dynamic programming, such as the Viterbi algorithm (Chapter 17) and the\nCKY algorithm for parsing (Chapter 18).\nThe intuition of a dynamic programming problem is that a large problem can\nbe solved by properly combining the solutions to various subproblems. Consider\nthe shortest path of transformed words that represents the minimum edit distance\nbetween the strings intention andexecution shown in Fig. 2.16.\nImagine some string (perhaps it is exention ) that is in this optimal path (whatever\nit is). The intuition of dynamic programming is that if exention is in the optimal 24 CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\nn t e n t i o ni n t e n t i o n\ne t e n t i o n\ne x e n t i o n\ne x e n u t i o n\ne x e c u t i o ndelete i\nsubstitute n by e\nsubstitute t by x\ninsert u\nsubstitute n by c\nFigure 2.16 Path from intention toexecution .\noperation list, then the optimal sequence must also include the optimal path from\nintention toexention . Why? If there were a shorter path from intention toexention ,\nthen we could use it instead, resulting in a shorter overall path, and the optimal\nsequence wouldn\u2019t be optimal, thus leading to a contradiction.\nThe minimum edit distance algorithm was named by Wagner and Fischerminimum edit\ndistance\nalgorithm(1974) but independently discovered by many people (see the Historical Notes sec-\ntion of Chapter 17).\nLet\u2019s \ufb01rst de\ufb01ne the minimum edit distance between two strings. Given two\nstrings, the source string Xof length n, and target string Yof length m, we\u2019ll de\ufb01ne\nD[i;j]as the edit distance between X[1::i]andY[1::j], i.e., the \ufb01rst icharacters of X\nand the \ufb01rst jcharacters of Y. The edit distance between XandYis thus D[n;m].\nWe\u2019ll use dynamic programming to compute D[n;m]bottom up, combining so-\nlutions to subproblems. In the base case, with a source substring of length ibut an\nempty target string, going from icharacters to 0 requires ideletes. With a target\nsubstring of length jbut an empty source going from 0 characters to jcharacters\nrequires jinserts. Having computed D[i;j]for small i;jwe then compute larger\nD[i;j]based on previously computed smaller values. The value of D[i;j]is com-\nputed by taking the minimum of the three possible paths through the matrix which\narrive there:\nD[i;j] =min8\n<\n:D[i\u00001;j]+del-cost (source [i])\nD[i;j\u00001]+ins-cost (target [j])\nD[i\u00001;j\u00001]+sub-cost (source [i];target [j])(2.25)\nWe mentioned above two versions of Levenshtein distance, one in which substitu-\ntions cost 1 and one in which substitutions cost 2 (i.e., are equivalent to an insertion\nplus a deletion). Let\u2019s here use that second version of Levenshtein distance in which\nthe insertions and deletions each have a cost of 1 (ins-cost( \u0001) = del-cost(\u0001) = 1), and\nsubstitutions have a cost of 2 (except substitution of identical letters has zero cost).\nUnder this version of Levenshtein, the computation for D[i;j]becomes:\nD[i;j] =min8\n>><\n>>:D[i\u00001;j]+1\nD[i;j\u00001]+1\nD[i\u00001;j\u00001]+\u001a2; if source [i]6=target [j]\n0; if source [i] =target [j](2.26)\nThe algorithm is summarized in Fig. 2.17; Fig. 2.18 shows the results of applying\nthe algorithm to the distance between intention andexecution with the version of\nLevenshtein in Eq. 2.26.\nAlignment Knowing the minimum edit distance is useful for algorithms like \ufb01nd-\ning potential spelling error corrections. But the edit distance algorithm is important 2.8 \u2022 M INIMUM EDITDISTANCE 25\nfunction MIN-EDIT-DISTANCE (source ,target )returns min-distance\nn LENGTH (source )\nm LENGTH (target )\nCreate a distance matrix D[n+1,m+1]\n#Initialization: the zeroth row and column is the distance from the empty string\nD[0,0] = 0\nforeach row ifrom 1tondo\nD[i,0] D[i-1,0] + del-cost (source [i])\nforeach column jfrom 1tomdo\nD[0,j] D[0,j-1] + ins-cost (target [j])\n#Recurrence relation:\nforeach row ifrom 1tondo\nforeach column jfrom 1tomdo\nD[i, j] MIN(D[i\u00001,j] +del-cost (source [i]),\nD[i\u00001,j\u00001] + sub-cost (source [i],target [j]),\nD[i,j\u00001] + ins-cost (target [j]))\n#Termination\nreturn D[n,m]\nFigure 2.17 The minimum edit distance algorithm, an example of the class of dynamic\nprogramming algorithms. The various costs can either be \ufb01xed (e.g., 8x;ins-cost (x) =1)\nor can be speci\ufb01c to the letter (to model the fact that some letters are more likely to be in-\nserted than others). We assume that there is no cost for substituting a letter for itself (i.e.,\nsub-cost (x;x) =0).\nSrcnTar # e x e c u t i o n\n# 0 1 2 3 4 5 6 7 8 9\ni 1 2 3 4 5 6 7 6 7 8\nn 2 3 4 5 6 7 8 7 8 7\nt 3 4 5 6 7 8 7 8 9 8\ne 4 3 4 5 6 7 8 9 10 9\nn 5 4 5 6 7 8 9 10 11 10\nt 6 5 6 7 8 9 8 9 10 11\ni 7 6 7 8 9 10 9 8 9 10\no 8 7 8 9 10 11 10 9 8 9\nn 9 8 9 10 11 12 11 10 9 8\nFigure 2.18 Computation of minimum edit distance between intention andexecution with\nthe algorithm of Fig. 2.17, using Levenshtein distance with cost of 1 for insertions or dele-\ntions, 2 for substitutions.\nin another way; with a small change, it can also provide the minimum cost align-\nment between two strings. Aligning two strings is useful throughout speech and\nlanguage processing. In speech recognition, minimum edit distance alignment is\nused to compute the word error rate (Chapter 16). Alignment plays a role in ma-\nchine translation, in which sentences in a parallel corpus (a corpus with a text in two\nlanguages) need to be matched to each other.\nTo extend the edit distance algorithm to produce an alignment, we can start by\nvisualizing an alignment as a path through the edit distance matrix. Figure 2.19\nshows this path with boldfaced cells. Each boldfaced cell represents an alignment 26 CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\nof a pair of letters in the two strings. If two boldfaced cells occur in the same row,\nthere will be an insertion in going from the source to the target; two boldfaced cells\nin the same column indicate a deletion.\nFigure 2.19 also shows the intuition of how to compute this alignment path. The\ncomputation proceeds in two steps. In the \ufb01rst step, we augment the minimum edit\ndistance algorithm to store backpointers in each cell. The backpointer from a cell\npoints to the previous cell (or cells) that we came from in entering the current cell.\nWe\u2019ve shown a schematic of these backpointers in Fig. 2.19. Some cells have mul-\ntiple backpointers because the minimum extension could have come from multiple\nprevious cells. In the second step, we perform a backtrace . In a backtrace, we start backtrace\nfrom the last cell (at the \ufb01nal row and column), and follow the pointers back through\nthe dynamic programming matrix. Each complete path between the \ufb01nal cell and the\ninitial cell is a minimum distance alignment. Exercise 2.7 asks you to modify the\nminimum edit distance algorithm to store the pointers and compute the backtrace to\noutput an alignment.\n# e x e c u t i o n\n# 0 1 2 3 4 5 6 7 8 9\ni\"1- \" 2- \" 3- \" 4- \" 5- \" 6- \" 7-6 7 8\nn\"2- \" 3- \" 4- \" 5- \" 6- \" 7- \" 8\"7- \" 8-7\nt\"3- \" 4- \" 5- \" 6- \" 7- \" 8-7 \"8- \" 9\"8\ne\"4-3 4- 5 6 7 \"8- \" 9- \" 10\"9\nn\"5\"4- \" 5- \" 6- \" 7- \" 8- \" 9- \" 10- \" 11-\"10\nt\"6\"5- \" 6- \" 7- \" 8- \" 9-8 9 10 \"11\ni\"7\"6- \" 7- \" 8- \" 9- \" 10\"9-8 9 10\no\"8\"7- \" 8- \" 9- \" 10- \" 11\"10\"9-8 9\nn\"9\"8- \" 9- \" 10- \" 11- \" 12\"11\"10\"9-8\nFigure 2.19 When entering a value in each cell, we mark which of the three neighboring\ncells we came from with up to three arrows. After the table is full we compute an alignment\n(minimum edit path) by using a backtrace , starting at the 8in the lower-right corner and\nfollowing the arrows back. The sequence of bold cells represents one possible minimum\ncost alignment between the two strings, again using Levenshtein distance with cost of 1 for\ninsertions or deletions, 2 for substitutions. Diagram design after Gus\ufb01eld (1997).\nWhile we worked our example with simple Levenshtein distance, the algorithm\nin Fig. 2.17 allows arbitrary weights on the operations. For spelling correction, for\nexample, substitutions are more likely to happen between letters that are next to\neach other on the keyboard. The Viterbi algorithm is a probabilistic extension of\nminimum edit distance. Instead of computing the \u201cminimum edit distance\u201d between\ntwo strings, Viterbi computes the \u201cmaximum probability alignment\u201d of one string\nwith another. We\u2019ll discuss this more in Chapter 17.\n2.9 Summary\nThis chapter introduced a fundamental tool in language processing, the regular ex-\npression , and showed how to perform basic text normalization tasks including\nword segmentation andnormalization ,sentence segmentation , and stemming .\nWe also introduced the important minimum edit distance algorithm for comparing\nstrings. Here\u2019s a summary of the main points we covered about these ideas: BIBLIOGRAPHICAL AND HISTORICAL NOTES 27\n\u2022 The regular expression language is a powerful tool for pattern-matching.\n\u2022 Basic operations in regular expressions include concatenation of symbols,\ndisjunction of symbols ( [],|),counters (*,+, and{n,m} ),anchors (^,$)\nand precedence operators ( (,)).\n\u2022Word tokenization and normalization are generally done by cascades of\nsimple regular expression substitutions or \ufb01nite automata.\n\u2022 The Porter algorithm is a simple and ef\ufb01cient way to do stemming , stripping\noff af\ufb01xes. It does not have high accuracy but may be useful for some tasks.\n\u2022 The minimum edit distance between two strings is the minimum number of\noperations it takes to edit one into the other. Minimum edit distance can be\ncomputed by dynamic programming , which also results in an alignment of\nthe two strings.\nBibliographical and Historical Notes\nKleene 1951; 1956 \ufb01rst de\ufb01ned regular expressions and the \ufb01nite automaton, based\non the McCulloch-Pitts neuron. Ken Thompson was one of the \ufb01rst to build regular\nexpressions compilers into editors for text searching (Thompson, 1968). His edi-\ntoredincluded a command \u201cg/regular expression/p\u201d, or Global Regular Expression\nPrint, which later became the Unix grep utility.\nText normalization algorithms have been applied since the beginning of the\n\ufb01eld. One of the earliest widely used stemmers was Lovins (1968). Stemming\nwas also applied early to the digital humanities, by Packard (1973), who built an\naf\ufb01x-stripping morphological parser for Ancient Greek. Currently a wide vari-\nety of code for tokenization and normalization is available, such as the Stanford\nTokenizer ( https://nlp.stanford.edu/software/tokenizer.shtml ) or spe-\ncialized tokenizers for Twitter (O\u2019Connor et al., 2010), or for sentiment ( http:\n//sentiment.christopherpotts.net/tokenizing.html ). See Palmer (2012)\nfor a survey of text preprocessing. NLTK is an essential tool that offers both useful\nPython libraries ( https://www.nltk.org ) and textbook descriptions (Bird et al.,\n2009) of many algorithms including text normalization and corpus interfaces.\nFor more on Herdan\u2019s law and Heaps\u2019 Law, see Herdan (1960, p. 28), Heaps\n(1978), Egghe (2007) and Baayen (2001); For more on edit distance, see Gus\ufb01eld\n(1997). Our example measuring the edit distance from \u2018intention\u2019 to \u2018execution\u2019\nwas adapted from Kruskal (1983). There are various publicly available packages to\ncompute edit distance, including Unix diff and the NIST sclite program (NIST,\n2005).\nIn his autobiography Bellman (1984) explains how he originally came up with\nthe term dynamic programming :\n\u201c...The 1950s were not good years for mathematical research. [the]\nSecretary of Defense ...had a pathological fear and hatred of the word,\nresearch... I decided therefore to use the word, \u201cprogramming\u201d. I\nwanted to get across the idea that this was dynamic, this was multi-\nstage... I thought, let\u2019s ... take a word that has an absolutely precise\nmeaning, namely dynamic... it\u2019s impossible to use the word, dynamic,\nin a pejorative sense. Try thinking of some combination that will pos-\nsibly give it a pejorative meaning. It\u2019s impossible. Thus, I thought\ndynamic programming was a good name. It was something not even a\nCongressman could object to.\u201d 28 CHAPTER 2 \u2022 R EGULAR EXPRESSIONS , TOKENIZATION , EDITDISTANCE\nExercises\n2.1 Write regular expressions for the following languages.\n1. the set of all alphabetic strings;\n2. the set of all lower case alphabetic strings ending in a b;\n3. the set of all strings from the alphabet a;bsuch that each ais immedi-\nately preceded by and immediately followed by a b;\n2.2 Write regular expressions for the following languages. By \u201cword\u201d, we mean\nan alphabetic string separated from other words by whitespace, any relevant\npunctuation, line breaks, and so forth.\n1. the set of all strings with two consecutive repeated words (e.g., \u201cHum-\nbert Humbert\u201d and \u201cthe the\u201d but not \u201cthe bug\u201d or \u201cthe big bug\u201d);\n2. all strings that start at the beginning of the line with an integer and that\nend at the end of the line with a word;\n3. all strings that have both the word grotto and the word raven in them\n(but not, e.g., words like grottos that merely contain the word grotto );\n4. write a pattern that places the \ufb01rst word of an English sentence in a\nregister. Deal with punctuation.\n2.3 Implement an ELIZA-like program, using substitutions such as those described\non page 10. You might want to choose a different domain than a Rogerian psy-\nchologist, although keep in mind that you would need a domain in which your\nprogram can legitimately engage in a lot of simple repetition.\n2.4 Compute the edit distance (using insertion cost 1, deletion cost 1, substitution\ncost 1) of \u201cleda\u201d to \u201cdeal\u201d. Show your work (using the edit distance grid).\n2.5 Figure out whether drive is closer to brief or to divers and what the edit dis-\ntance is to each. You may use any version of distance that you like.\n2.6 Now implement a minimum edit distance algorithm and use your hand-computed\nresults to check your code.\n2.7 Augment the minimum edit distance algorithm to output an alignment; you\nwill need to store pointers and add a stage to compute the backtrace. Exercises 29\nBaayen, R. H. 2001. Word frequency distributions . Springer.\nBellman, R. 1957. Dynamic Programming . Princeton Uni-\nversity Press.\nBellman, R. 1984. Eye of the Hurricane: an autobiography .\nWorld Scienti\ufb01c Singapore.\nBender, E. M. 2019. The #BenderRule: On naming the lan-\nguages we study and why it matters. Blog post.\nBender, E. M., B. Friedman, and A. McMillan-Major. 2021.\nA guide for writing data statements for natural lan-\nguage processing. http://techpolicylab.uw.edu/\ndata-statements/ .\nBird, S., E. Klein, and E. Loper. 2009. Natural Language\nProcessing with Python . O\u2019Reilly.\nBlodgett, S. L., L. Green, and B. O\u2019Connor. 2016. Demo-\ngraphic dialectal variation in social media: A case study\nof African-American English. EMNLP .\nBostrom, K. and G. Durrett. 2020. Byte pair encoding is\nsuboptimal for language model pretraining. EMNLP .\nChen, X., Z. Shi, X. Qiu, and X. Huang. 2017. Adversar-\nial multi-criteria learning for Chinese word segmentation.\nACL.\nChurch, K. W. 1994. Unix for Poets. Slides from 2nd EL-\nSNET Summer School and unpublished paper ms.\nClark, H. H. and J. E. Fox Tree. 2002. Using uh and um in\nspontaneous speaking. Cognition , 84:73\u2013111.\nEgghe, L. 2007. Untangling Herdan\u2019s law and Heaps\u2019\nlaw: Mathematical and informetric arguments. JASIST ,\n58(5):702\u2013709.\nGebru, T., J. Morgenstern, B. Vecchione, J. W. Vaughan,\nH. Wallach, H. Daum \u00b4e III, and K. Crawford. 2020.\nDatasheets for datasets. ArXiv.\nGodfrey, J., E. Holliman, and J. McDaniel. 1992. SWITCH-\nBOARD: Telephone speech corpus for research and de-\nvelopment. ICASSP .\nGus\ufb01eld, D. 1997. Algorithms on Strings, Trees, and Se-\nquences . Cambridge University Press.\nHeaps, H. S. 1978. Information retrieval. Computational and\ntheoretical aspects . Academic Press.\nHerdan, G. 1960. Type-token mathematics . Mouton.\nJones, T. 2015. Toward a description of African American\nVernacular English dialect regions using \u201cBlack Twitter\u201d.\nAmerican Speech , 90(4):403\u2013440.\nJurgens, D., Y . Tsvetkov, and D. Jurafsky. 2017. Incorpo-\nrating dialectal variability for socially equitable language\nidenti\ufb01cation. ACL.\nKing, S. 2020. From African American Vernacular English\nto African American Language: Rethinking the study of\nrace and language in African Americans\u2019 speech. Annual\nReview of Linguistics , 6:285\u2013300.\nKiss, T. and J. Strunk. 2006. Unsupervised multilingual\nsentence boundary detection. Computational Linguistics ,\n32(4):485\u2013525.\nKleene, S. C. 1951. Representation of events in nerve nets\nand \ufb01nite automata. Technical Report RM-704, RAND\nCorporation. RAND Research Memorandum.\nKleene, S. C. 1956. Representation of events in nerve nets\nand \ufb01nite automata. In C. Shannon and J. McCarthy, eds,\nAutomata Studies , 3\u201341. Princeton University Press.Krovetz, R. 1993. Viewing morphology as an inference pro-\ncess. SIGIR-93 .\nKruskal, J. B. 1983. An overview of sequence comparison.\nIn D. Sankoff and J. B. Kruskal, eds, Time Warps, String\nEdits, and Macromolecules: The Theory and Practice of\nSequence Comparison , 1\u201344. Addison-Wesley.\nKudo, T. 2018. Subword regularization: Improving neural\nnetwork translation models with multiple subword candi-\ndates. ACL.\nKudo, T. and J. Richardson. 2018. SentencePiece: A simple\nand language independent subword tokenizer and detok-\nenizer for neural text processing. EMNLP .\nKu\u02c7cera, H. and W. N. Francis. 1967. Computational Analysis\nof Present-Day American English . Brown Univ. Press.\nLevenshtein, V . I. 1966. Binary codes capable of correct-\ning deletions, insertions, and reversals. Cybernetics and\nControl Theory , 10(8):707\u2013710. Original in Doklady\nAkademii Nauk SSSR 163(4): 845\u2013848 (1965).\nLi, X., Y . Meng, X. Sun, Q. Han, A. Yuan, and J. Li. 2019.\nIs word segmentation necessary for deep learning of Chi-\nnese representations? ACL.\nLovins, J. B. 1968. Development of a stemming algorithm.\nMechanical Translation and Computational Linguistics ,\n11(1\u20132):9\u201313.\nManning, C. D., M. Surdeanu, J. Bauer, J. Finkel, S. Bethard,\nand D. McClosky. 2014. The Stanford CoreNLP natural\nlanguage processing toolkit. ACL.\nNIST. 2005. Speech recognition scoring toolkit (sctk) ver-\nsion 2.1.http://www.nist.gov/speech/tools/ .\nO\u2019Connor, B., M. Krieger, and D. Ahn. 2010. Tweetmotif:\nExploratory search and topic summarization for twitter.\nICWSM .\nPackard, D. W. 1973. Computer-assisted morphological\nanalysis of ancient Greek. COLING .\nPalmer, D. 2012. Text preprocessing. In N. Indurkhya and\nF. J. Damerau, eds, Handbook of Natural Language Pro-\ncessing , 9\u201330. CRC Press.\nPorter, M. F. 1980. An algorithm for suf\ufb01x stripping. Pro-\ngram , 14(3):130\u2013137.\nSennrich, R., B. Haddow, and A. Birch. 2016. Neural ma-\nchine translation of rare words with subword units. ACL.\nSimons, G. F. and C. D. Fennig. 2018. Ethnologue: Lan-\nguages of the world, 21st edition. SIL International.\nSolorio, T., E. Blair, S. Maharjan, S. Bethard, M. Diab,\nM. Ghoneim, A. Hawwari, F. AlGhamdi, J. Hirschberg,\nA. Chang, and P. Fung. 2014. Overview for the \ufb01rst\nshared task on language identi\ufb01cation in code-switched\ndata. Workshop on Computational Approaches to Code\nSwitching .\nThompson, K. 1968. Regular expression search algorithm.\nCACM , 11(6):419\u2013422.\nWagner, R. A. and M. J. Fischer. 1974. The string-to-string\ncorrection problem. Journal of the ACM , 21:168\u2013173.\nWeizenbaum, J. 1966. ELIZA \u2013 A computer program for the\nstudy of natural language communication between man\nand machine. CACM , 9(1):36\u201345.\nWeizenbaum, J. 1976. Computer Power and Human Reason:\nFrom Judgement to Calculation . W.H. Freeman & Co.", "2_TextProc_2023.pdf": "Basic Text ProcessingRegular Expressions Regular expressions are used everywhere\u25e6Part of every text processing task\u25e6Not a general NLP solution (for that we use large NLP systems we will see in later lectures)\u25e6But very useful as part of those systems (e.g., for pre-processing or text formatting)\u25e6Necessary for data analysis of text data\u25e6A widely used tool in industry and academics\n2 Regular expressionsA formal language for specifying text stringsHow can we search for mentions of these cute animals in text?\u25e6woodchuck\u25e6woodchucks\u25e6Woodchuck\u25e6Woodchucks\u25e6Groundhog\u25e6groundhogs\n Regular Expressions: DisjunctionsLetters inside square brackets []Ranges using the dash [A-Z]PatternMatches[wW]oodchuckWoodchuck,woodchuck[1234567890]Any one digitPatternMatches[A-Z]An upper case letterDrenched Blossoms[a-z]A lower case lettermy beans were impatient[0-9]A singledigitChapter 1: Down the Rabbit Hole Regular Expressions: Negation in DisjunctionCarat as first character in [] negates the list\u25e6Note: Carat means negation only when it's first in []\u25e6Special characters (., *, +, ?) lose their special meaning inside []PatternMatchesExamples[^A-Z]Notan upper case letterOyfnpripetchik[^Ss]Neither \u2018S\u2019 nor \u2018s\u2019Ihave no exquisite reason\u201d[^.]Not a periodOur resident Djinn[e^]Either e or ^Look up ^now Regular Expressions: Convenient aliasesPatternExpansionMatchesExamples\\d[0-9]Any digitFahreneit451\\D[^0-9]Any non-digitBlue Moon\\w[a-ZA-Z0-9_]Any alphanumeric or _Daiyu\\W[^\\w]Not alphanumeric or _Look!\\s[ \\r\\t\\n\\f]Whitespace (space, tab)Look\u2423up\\S[^\\s]Not whitespaceLook up Regular Expressions: More DisjunctionGroundhog is another name for woodchuck!The pipe symbol | for disjunctionPatternMatchesgroundhog|woodchuckwoodchuckyours|mineyoursa|b|c= [abc][gG]roundhog|[Ww]oodchuckWoodchuck\n Wildcards, optionality, repetition: . ?* +\nStephen C KleenePatternMatchesExamplesbeg.nAny charbeginbegun beg3nbeg nwoodchucks?Optional swoodchuck woodchucksto*0 or more of previous charttotootoooto+1 or more of previous chartotootoootooooKleene *,   Kleene +    Regular Expressions: Anchors  ^   $PatternMatches^[A-Z] PaloAlto^[^A-Za-z] 1\u201cHello\u201d\\.$The end..$The end?The end! A note about Python regular expressions\u25e6Regex and Python both use backslash \"\\\" for special characters. You must type extra backslashes!\u25e6\"\\\\d+\"to search for 1 or more digits\u25e6\"\\n\" in Python means the \"newline\" character, not a \"slash\" followed by an \"n\". Need \"\\\\n\"for two characters.\u25e6Instead: use Python's raw string notation for regex:\u25e6r\"[tT]he\"\u25e6r\"\\d+\"matches one or more digits\u25e6instead of \"\\\\d+\"10 The iterative process of writing regex'sFind me all instances of the word \u201cthe\u201d in a text.theMisses capitalized examples[tT]heIncorrectly returns otheror Theology\\W[tT]he\\W False positives and false negativesThe process we just went through was based on fixing two kinds of errors:1.Not matching things that we should have matched (The)False negatives2.Matching strings that we should not have matched (there, then, other)False positives Characterizing work on NLPIn NLP we are always dealing with these kinds of errors.Reducing the error rate for an application often involves two antagonistic efforts: \u25e6Increasing coverage (or recall) (minimizing false negatives).\u25e6Increasing accuracy (or precision) (minimizing false positives) Regular expressions play a surprisingly large roleWidely used in both academics and industry1.Part of most text processing tasks, even for big neural language model pipelines\u25e6including text formatting and pre-processing2.Very useful for data analysis of any text data\n14 Basic Text ProcessingRegular Expressions Basic Text ProcessingMore Regular Expressions: Substitutions and ELIZA SubstitutionsSubstitution in Python and UNIX commands:s/regexp1/pattern/ e.g.:s/colour/color/  Capture Groups\u2022Say we want to put angles around all numbers:the 35 boxes\u00e0the <35> boxes\u2022Use parens() to \"capture\" a pattern into a numbered register (1, 2, 3\u2026)\u2022Use \\1  to refer to the contents of the registers/([0-9]+)/<\\1>/  Capture groups: multiple registers/the (.*)er they (.*), the \\1er we \\2/ Matchesthe faster they ran, the faster we ranBut notthe faster they ran, the faster we ate  But suppose we don't want to capture?Parentheses have a double function: grouping terms, and capturingNon-capturing groups: add a ?: after paren:/(?:some|afew) (people|cats) like some \\1/ matches \u25e6some cats like some cats but not \u25e6some cats like some some Lookahead assertions(?= pattern) is true if pattern matches, but is zero-width; doesn't advance character pointer(?! pattern) true if a pattern does not match How to match, at the beginning of a line, any single word that doesn\u2019t start with \u201cVolcano\u201d: /\u02c6(?!Volcano)[A-Za-z]+/  Simple Application: ELIZAEarly NLP system that imitated a Rogerian psychotherapist \u25e6Joseph Weizenbaum, 1966. Uses pattern matching to match, e.g.,:\u25e6\u201cI need X\u201d and translates them into, e.g.\u25e6\u201cWhat would it mean to you if you got X?  Simple Application: ELIZAMen are all alike.IN WHAT WAYThey're always bugging us about something or other. CAN YOU THINK OF A SPECIFIC EXAMPLE Well, my boyfriend made me come here.YOUR BOYFRIEND MADE YOU COME HERE He says I'm depressed much of the time.I AM SORRY TO HEAR YOU ARE DEPRESSED  How ELIZA workss/.* I\u2019M (depressed|sad) .*/I AM SORRY TO HEAR YOU ARE \\1/ s/.* I AM (depressed|sad) .*/WHY DO YOU THINK YOU ARE \\1/s/.* all .*/IN WHAT WAY?/ s/.* always .*/CAN YOU THINK OF A SPECIFIC EXAMPLE?/  Basic Text ProcessingMore Regular Expressions: Substitutions and ELIZA Basic Text ProcessingWords and Corpora How many words in a sentence?\"I do uh main-mainly business data processing\"\u25e6Fragments, filled pauses\"Seuss\u2019s cat in the hat is different from othercats!\" \u25e6Lemma: same stem, part of speech, rough word sense\u25e6cat and cats = same lemma\u25e6Wordform: the full inflected surface form\u25e6cat and cats = different wordforms How many words in a sentence?they lay back on the San Francisco grass and looked at the stars and theirType: an element of the vocabulary.Token: an instance of that type in running text.How many?\u25e615 tokens (or 14)\u25e613 types (or 12) (or 11?) How many words in a corpus?N= number of tokensV= vocabulary = set of types, |V|is size of vocabularyHeaps Law = Herdan'sLaw =                                 where often .67 < \u03b2 < .75i.e., vocabulary size grows with > square root of the number of word tokensTokens = NTypes = |V|Switchboard phoneconversations2.4 million20thousandShakespeare884,00031thousandCOCA440 million2 millionGoogle N-grams1 trillion13+ million2.2\u2022WORDS11duce other complications with regard to de\ufb01ning words. Let\u2019s look at one utterancefrom Switchboard; anutteranceis the spoken correlate of a sentence:utteranceI do uh main- mainly business data processingThis utterance has two kinds ofdis\ufb02uencies. The broken-off wordmain-isdis\ufb02uencycalled afragment. Words likeuhandumare called\ufb01llersor\ufb01lled pauses. Shouldfragment\ufb01lled pausewe consider these to be words? Again, it depends on the application. If we arebuilding a speech transcription system, we might want to eventually strip out thedis\ufb02uencies.But we also sometimes keep dis\ufb02uencies around. Dis\ufb02uencies likeuhorumare actually helpful in speech recognition in predicting the upcoming word, becausethey may signal that the speaker is restarting the clause or idea, and so for speechrecognition they are treated as regular words. Because people use different dis\ufb02u-encies they can also be a cue to speaker identi\ufb01cation. In factClark and Fox Tree(2002)showed thatuhandumhave different meanings. What do you think they are?Are capitalized tokens likeTheyand uncapitalized tokens liketheythe sameword? These are lumped together in some tasks (speech recognition), while for part-of-speech or named-entity tagging, capitalization is a useful feature and is retained.How about in\ufb02ected forms likecatsversuscat? These two words have the samelemmacatbut are different wordforms. Alemmais a set of lexical forms havinglemmathe same stem, the same major part-of-speech, and the same word sense. Theword-formis the full in\ufb02ected or derived form of the word. For morphologically complexwordformlanguages like Arabic, we often need to deal with lemmatization. For many tasks inEnglish, however, wordforms are suf\ufb01cient.How many words are there in English? To answer this question we need todistinguish two ways of talking about words.Typesare the number of distinct wordsword typein a corpus; if the set of words in the vocabulary isV, the number of types is thevocabulary size|V|.Tokensare the total numberNof running words. If we ignoreword tokenpunctuation, the following Brown sentence has 16 tokens and 14 types:They picnicked by the pool, then lay back on the grass and looked at the stars.When we speak about the number of words in the language, we are generallyreferring to word types.CorpusTokens =NTypes =|V|Shakespeare884 thousand31 thousandBrown corpus1 million38 thousandSwitchboard telephone conversations2.4 million20 thousandCOCA440 million2 millionGoogle N-grams1 trillion13 millionFigure 2.11Rough numbers of types and tokens for some English language corpora. Thelargest, the Google N-grams corpus, contains 13 million types, but this count only includestypes appearing 40 or more times, so the true number would be much larger.Fig.2.11shows the rough numbers of types and tokens computed from somepopular English corpora. The larger the corpora we look at, the more word typeswe \ufb01nd, and in fact this relationship between the number of types|V|and numberof tokensNis calledHerdan\u2019s Law(Herdan, 1960)orHeaps\u2019 Law(Heaps, 1978)Herdan\u2019s LawHeaps\u2019 Lawafter its discoverers (in linguistics and information retrieval respectively). It is shownin Eq.2.1, wherekandbare positive constants, and 0<b<1.|V|=kNb(2.1) CorporaWords don't appear out of nowhere! A text is produced by \u2022a specific writer(s), \u2022at a specific time, \u2022in a specific variety,\u2022of a specific language, \u2022for a specific function. Corpora vary along dimension like\u25e6Language: 7097 languages in the world\u25e6Variety, like African American Language varieties.\u25e6AAE Twitter posts might include forms like \"iont\" (I don't)\u25e6Code switching, e.g., Spanish/English, Hindi/English:S/E: Por primeravezveoa @username actually being hateful! It was beautiful:) [For the first time I get to see @username actually being hateful! it was beautiful:) ] H/E: dost thaor ra-hega... dontwory... but dheryarakhe[\u201che was and will remain a friend ... don\u2019t worry ... but have faith\u201d] \u25e6Genre: newswire, fiction, scientific articles, Wikipedia\u25e6Author Demographics: writer's age, gender, ethnicity, SES  Corpus datasheetsMotivation: \u2022Why was the corpus collected?\u2022By whom? \u2022Who funded it? Situation: In what situation was the text written?Collection process: If it is a subsample how was it sampled? Was there consent? Pre-processing?+Annotation process, language variety, demographics, etc.Gebru et al (2020), Bender and Friedman (2018) Basic Text ProcessingWords and Corpora Basic Text ProcessingWord tokenization Text NormalizationEvery NLP task requires text normalization: 1.Tokenizing (segmenting) words2.Normalizing word formats3.Segmenting sentences Space-based tokenizationA very simple way to tokenize\u25e6For languages that use space characters between words\u25e6Arabic, Cyrillic, Greek, Latin, etc., based writing systems\u25e6Segment off a token between instances of spacesUnix tools for space-based tokenization\u25e6The \"tr\" command\u25e6Inspired by Ken Church's UNIX for Poets\u25e6Given a text file, output the word tokens and their frequencies Simple Tokenization in UNIX(Inspired by Ken Church\u2019s UNIX for Poets.)Given a text file, output the word tokens and their frequenciestr -sc\u2019A-Za-z\u2019 \u2019\\n\u2019 < shakes.txt| sort | uniq\u2013c 1945 A72 AARON19 ABBESS5 ABBOT... ...25 Aaron6 Abate1 Abates5 Abbess6 Abbey3 Abbot....   \u2026Change all non-alpha to newlinesSort in alphabetical orderMerge and count each type The first step: tokenizingtr -sc\u2019A-Za-z\u2019 \u2019\\n\u2019 < shakes.txt| headTHESONNETSbyWilliamShakespeareFromfairestcreaturesWe... The second step: sortingtr -sc\u2019A-Za-z\u2019 \u2019\\n\u2019 < shakes.txt| sort | headAAAAAAAAA... More countingMerging upper and lower casetr\u2018A-Z\u2019 \u2018a-z\u2019 < shakes.txt| tr \u2013sc\u2018A-Za-z\u2019 \u2018\\n\u2019 | sort | uniq\u2013c Sorting the countstr\u2018A-Z\u2019 \u2018a-z\u2019 < shakes.txt| tr \u2013sc\u2018A-Za-z\u2019 \u2018\\n\u2019 | sort | uniq\u2013c | sort \u2013n \u2013r23243 the22225 i18618 and16339 to15687 of12780 a12163 you10839 my10005 in8954  dWhat happened here? Issues in TokenizationCan't just blindly remove punctuation:\u25e6m.p.h., Ph.D., AT&T, cap\u2019n\u25e6prices ($45.55)\u25e6dates (01/02/06)\u25e6URLs (http://www.stanford.edu)\u25e6hashtags (#nlproc)\u25e6email addresses (someone@cs.colorado.edu)Clitic: a word that doesn't stand on its own\u25e6\"are\" inwe're, French \"je\" in j'ai,\"le\" in l'honneurWhen should multiword expressions (MWE) be words?\u25e6New York, rock \u2019n\u2019 roll  Tokenization in NLTK16CHAPTER2\u2022REGULAREXPRESSIONS,TEXTNORMALIZATION,EDITDISTANCEInput:\"The San Francisco-based restaurant,\" they said,\"doesn\u2019t charge$10\".Output:\"TheSanFrancisco-basedrestaurant,\"theysaid,\"doesn\u2019tcharge$10\".In practice, since tokenization needs to be run before any other language pro-cessing, it needs to be very fast. The standard method for tokenization is thereforeto use deterministic algorithms based on regular expressions compiled into very ef-\ufb01cient \ufb01nite state automata. For example, Fig.2.12shows an example of a basicregular expression that can be used to tokenize with thenltk.regexptokenizefunction of the Python-based Natural Language Toolkit (NLTK) (Bird et al. 2009;http://www.nltk.org).>>> text = \u2019That U.S.A. poster-print costs $12.40...\u2019>>> pattern = r\u2019\u2019\u2019(?x) # set flag to allow verbose regexps... ([A-Z]\\.)+ # abbreviations, e.g. U.S.A.... | \\w+(-\\w+)* # words with optional internal hyphens... | \\$?\\d+(\\.\\d+)?%? # currency and percentages, e.g. $12.40, 82%... | \\.\\.\\. # ellipsis... | [][.,;\"\u2019?():-_\u2018] # these are separate tokens; includes ], [... \u2019\u2019\u2019>>> nltk.regexp_tokenize(text, pattern)[\u2019That\u2019, \u2019U.S.A.\u2019, \u2019poster-print\u2019, \u2019costs\u2019, \u2019$12.40\u2019, \u2019...\u2019]Figure 2.12A Python trace of regular expression tokenization in the NLTK Python-basednatural language processing toolkit(Bird et al., 2009), commented for readability; the(?x)verbose \ufb02ag tells Python to strip comments and whitespace. Figure from Chapter 3 ofBirdet al. (2009).Carefully designed deterministic algorithms can deal with the ambiguities thatarise, such as the fact that the apostrophe needs to be tokenized differently when usedas a genitive marker (as inthe book\u2019s cover), a quotative as in\u2018The other class\u2019, shesaid, or in clitics likethey\u2019re.Word tokenization is more complex in languages like written Chinese, Japanese,and Thai, which do not use spaces to mark potential word-boundaries. In Chinese,for example, words are composed of characters (calledhanziin Chinese). Eachhanzicharacter generally represents a single unit of meaning (called amorpheme) and ispronounceable as a single syllable. Words are about 2.4 characters long on average.But deciding what counts as a word in Chinese is complex. For example, considerthe following sentence:(2.4)\u2044\u0000\u20ace;\u2265[\u201cYao Ming reaches the \ufb01nals\u201dAsChen et al. (2017)point out, this could be treated as 3 words (\u2018Chinese Treebank\u2019segmentation):(2.5)\u2044\u0000YaoMing\u20acereaches;\u2265[\ufb01nalsor as 5 words (\u2018Peking University\u2019 segmentation):(2.6)\u2044Yao\u0000Ming\u20acereaches;overall\u2265[\ufb01nalsFinally, it is possible in Chinese simply to ignore words altogether and use charactersas the basic elements, treating the sentence as a series of 7 characters:Bird, Loper and Klein (2009),Natural Language Processing with Python. O\u2019Reilly Tokenization in languages without spaces Many languages (like Chinese, Japanese, Thai) don't use spaces to separate words!How do we decide where the token boundaries should be? Word tokenization in ChineseChinese words are composed of characters called \"hanzi\" (or sometimes just \"zi\")Each one represents a meaning unit called a morpheme.Each word has on average 2.4 of them.But deciding what counts as a word is complex and not agreed upon. How to do word tokenization in Chinese?\u59da\u660e\u8fdb\u5165\u603b\u51b3\u8d5b\u201cYao Ming reaches the finals\u201d3 words?\u59da\u660e\u8fdb\u5165\u603b\u51b3\u8d5bYaoMingreaches  finals 5 words?\u59da\u660e\u8fdb\u5165\u603b\u51b3\u8d5bYao    Ming    reaches    overall    finals 7 characters? (don't use words at all):\u59da\u660e\u8fdb\u5165\u603b\u51b3\u8d5bYao Ming enter enter overall decision game How to do word tokenization in Chinese?\u59da\u660e\u8fdb\u5165\u603b\u51b3\u8d5b\u201cYao Ming reaches the finals\u201d3 words?\u59da\u660e\u8fdb\u5165\u603b\u51b3\u8d5bYaoMingreaches  finals 5 words?\u59da\u660e\u8fdb\u5165\u603b\u51b3\u8d5bYao    Ming    reaches    overall    finals 7 characters? (don't use words at all):\u59da\u660e\u8fdb\u5165\u603b\u51b3\u8d5bYao Ming enter enter overall decision game How to do word tokenization in Chinese?\u59da\u660e\u8fdb\u5165\u603b\u51b3\u8d5b\u201cYao Ming reaches the finals\u201d3 words?\u59da\u660e\u8fdb\u5165\u603b\u51b3\u8d5bYaoMingreaches  finals 5 words?\u59da\u660e\u8fdb\u5165\u603b\u51b3\u8d5bYao    Ming    reaches    overall    finals 7 characters? (don't use words at all):\u59da\u660e\u8fdb\u5165\u603b\u51b3\u8d5bYao Ming enter enter overall decision game How to do word tokenization in Chinese?\u59da\u660e\u8fdb\u5165\u603b\u51b3\u8d5b\u201cYao Ming reaches the finals\u201d3 words?\u59da\u660e\u8fdb\u5165\u603b\u51b3\u8d5bYaoMingreaches  finals 5 words?\u59da\u660e\u8fdb\u5165\u603b\u51b3\u8d5bYao    Ming    reaches    overall    finals 7 characters? (don't use words at all):\u59da\u660e\u8fdb\u5165\u603b\u51b3\u8d5bYao Ming enter enter overall decision game Word tokenization / segmentationSo in Chinese it's common to just treat each character (zi) as a token.\u2022So the segmentationstep is very simpleIn other languages (like Thai and Japanese), more complex word segmentation is required.\u2022The standard algorithms are neural sequence models trained by supervised machine learning. Basic Text ProcessingWord tokenization Basic Text ProcessingByte Pair Encoding Another option for text tokenizationInstead of \u2022white-space segmentation\u2022single-character segmentation Use the data to tell us how to tokenize.Subwordtokenization (because tokens can be parts of words as well as whole words) SubwordtokenizationThree common algorithms:\u25e6Byte-Pair Encoding (BPE) (Sennrichet al., 2016)\u25e6Unigram language modeling tokenization (Kudo, 2018)\u25e6WordPiece(Schuster and Nakajima, 2012)All have 2 parts:\u25e6A token learnerthat takes a raw training corpus and induces a vocabulary (a set of tokens). \u25e6A token segmenterthat takes a raw test sentence and tokenizes it according to that vocabulary Byte Pair Encoding (BPE) token learnerLet vocabulary be the set of all individual characters = {A, B, C, D,\u2026, a, b, c, d\u2026.}Repeat:\u25e6Choose the two symbols that are most frequently adjacent in the training corpus (say 'A', 'B') \u25e6Add a new merged symbol 'AB' to the vocabulary\u25e6Replace every adjacent 'A' 'B' in the corpus with 'AB'. Until k merges have been done. BPE token learner algorithm2.4\u2022TEXTNORMALIZATION19functionBYTE-PAIR ENCODING(stringsC, number of mergesk)returnsvocabVV all unique characters inC# initial set of tokens is charactersfori=1tokdo# merge tokens tilktimestL,tR Most frequent pair of adjacent tokens inCtNEW tL+tR# make new token by concatenatingV V+tNEW# update the vocabularyReplace each occurrence oftL,tRinCwithtNEW# and update the corpusreturnVFigure 2.13The token learner part of the BPE algorithm for taking a corpus broken upinto individual characters or bytes, and learning a vocabulary by iteratively merging tokens.Figure adapted fromBostrom and Durrett (2020).from the training data, greedily, in the order we learned them. (Thus the frequenciesin the test data don\u2019t play a role, just the frequencies in the training data). So \ufb01rstwe segment each test sentence word into characters. Then we apply the \ufb01rst rule:replace every instance oferin the test corpus withr, and then the second rule:replace every instance oferin the test corpus wither, and so on. By the end,if the test corpus contained the wordnewer, it would be tokenized as a fullword. But a new (unknown) word likelowerwould be merged into the twotokenslow er.Of course in real algorithms BPE is run with many thousands of merges on a verylarge input corpus. The result is that most words will be represented as full symbols,and only the very rare words (and unknown words) will have to be represented bytheir parts.2.4.4 Word Normalization, Lemmatization and StemmingWordnormalizationis the task of putting words/tokens in a standard format, choos-normalizationing a single normal form for words with multiple forms likeUSAandUSoruh-huhanduhhuh. This standardization may be valuable, despite the spelling informationthat is lost in the normalization process. For information retrieval or informationextraction about the US, we might want to see information from documents whetherthey mention theUSor theUSA.Case foldingis another kind of normalization. Mapping everything to lowercase foldingcase means thatWoodchuckandwoodchuckare represented identically, which isvery helpful for generalization in many tasks, such as information retrieval or speechrecognition. For sentiment analysis and other text classi\ufb01cation tasks, informationextraction, and machine translation, by contrast, case can be quite helpful and casefolding is generally not done. This is because maintaining the difference between,for example,USthe country andusthe pronoun can outweigh the advantage ingeneralization that case folding would have provided for other words.For many natural language processing situations we also want two morpholog-ically different forms of a word to behave similarly. For example in web search,someone may type the stringwoodchucksbut a useful system might want to alsoreturn pages that mentionwoodchuckwith nos. This is especially common in mor-phologically complex languages like Russian, where for example the wordMoscowhas different endings in the phrasesMoscow,of Moscow,to Moscow, and so on.Lemmatizationis the task of determining that two words have the same root,despite their surface differences. The wordsam,are, andishave the shared lemma Byte Pair Encoding (BPE) AddendumMost subwordalgorithms are run inside space-separated tokens. So we commonly first add a special end-of-word symbol '__' before space in training corpusNext, separate into letters. BPE token learner18CHAPTER2\u2022REGULAREXPRESSIONS,TEXTNORMALIZATION,EDITDISTANCEThe algorithm is usually run inside words (not merging across word boundaries),so the input corpus is \ufb01rst white-space-separated to give a set of strings, each corre-sponding to the characters of a word, plus a special end-of-word symbol, and itscounts. Let\u2019s see its operation on the following tiny input corpus of 18 word tokenswith counts for each word (the wordlowappears 5 times, the wordnewer6 times,and so on), which would have a starting vocabulary of 11 letters:corpus vocabulary5low, d, e, i, l, n, o, r, s, t, w2lowest6newer3wider2newThe BPE algorithm \ufb01rst count all pairs of adjacent symbols: the most frequentis the pairerbecause it occurs innewer(frequency of 6) andwider(frequency of3) for a total of 9 occurrences1. We then merge these symbols, treatingeras onesymbol, and count again:corpus vocabulary5low, d, e, i, l, n, o, r, s, t, w, er2lowest6n e w er3w i d er2newNow the most frequent pair iser, which we merge; our system has learnedthat there should be a token for word-\ufb01naler, represented aser:corpus vocabulary5low,d,e,i,l,n,o,r,s,t,w,er,er2lowest6n e w er3w i d er2newNextne(total count of 8) get merged tone:corpus vocabulary5low,d,e,i,l,n,o,r,s,t,w,er,er,ne2lowest6ne w er3w i d er2ne wIf we continue, the next merges are:Merge Current Vocabulary(ne, w),d,e,i,l,n,o,r,s,t,w,er,er,ne,new(l, o),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo(lo, w),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low(new, er),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low,newer(low,),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low,newer,lowOnce we\u2019ve learned our vocabulary, thetoken parseris used to tokenize a testsentence. The token parser just runs on the test data the merges we have learned1Note that there can be ties; we could have instead chosen to merger\ufb01rst, since that also has afrequency of 9.Original (very fascinating\n\ud83d\ude44) corpus:low low low low low lowest lowest newer newer newer        newer newer newer wider wider wider new newAdd end-of-word tokens, resulting in this vocabulary:representation BPE token learner18CHAPTER2\u2022REGULAREXPRESSIONS,TEXTNORMALIZATION,EDITDISTANCEThe algorithm is usually run inside words (not merging across word boundaries),so the input corpus is \ufb01rst white-space-separated to give a set of strings, each corre-sponding to the characters of a word, plus a special end-of-word symbol, and itscounts. Let\u2019s see its operation on the following tiny input corpus of 18 word tokenswith counts for each word (the wordlowappears 5 times, the wordnewer6 times,and so on), which would have a starting vocabulary of 11 letters:corpus vocabulary5low, d, e, i, l, n, o, r, s, t, w2lowest6newer3wider2newThe BPE algorithm \ufb01rst count all pairs of adjacent symbols: the most frequentis the pairerbecause it occurs innewer(frequency of 6) andwider(frequency of3) for a total of 9 occurrences1. We then merge these symbols, treatingeras onesymbol, and count again:corpus vocabulary5low, d, e, i, l, n, o, r, s, t, w, er2lowest6n e w er3w i d er2newNow the most frequent pair iser, which we merge; our system has learnedthat there should be a token for word-\ufb01naler, represented aser:corpus vocabulary5low,d,e,i,l,n,o,r,s,t,w,er,er2lowest6n e w er3w i d er2newNextne(total count of 8) get merged tone:corpus vocabulary5low,d,e,i,l,n,o,r,s,t,w,er,er,ne2lowest6ne w er3w i d er2ne wIf we continue, the next merges are:Merge Current Vocabulary(ne, w),d,e,i,l,n,o,r,s,t,w,er,er,ne,new(l, o),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo(lo, w),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low(new, er),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low,newer(low,),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low,newer,lowOnce we\u2019ve learned our vocabulary, thetoken parseris used to tokenize a testsentence. The token parser just runs on the test data the merges we have learned1Note that there can be ties; we could have instead chosen to merger\ufb01rst, since that also has afrequency of 9.Merge e r to er18CHAPTER2\u2022REGULAREXPRESSIONS,TEXTNORMALIZATION,EDITDISTANCEThe algorithm is usually run inside words (not merging across word boundaries),so the input corpus is \ufb01rst white-space-separated to give a set of strings, each corre-sponding to the characters of a word, plus a special end-of-word symbol, and itscounts. Let\u2019s see its operation on the following tiny input corpus of 18 word tokenswith counts for each word (the wordlowappears 5 times, the wordnewer6 times,and so on), which would have a starting vocabulary of 11 letters:corpus vocabulary5low, d, e, i, l, n, o, r, s, t, w2lowest6newer3wider2newThe BPE algorithm \ufb01rst count all pairs of adjacent symbols: the most frequentis the pairerbecause it occurs innewer(frequency of 6) andwider(frequency of3) for a total of 9 occurrences1. We then merge these symbols, treatingeras onesymbol, and count again:corpus vocabulary5low, d, e, i, l, n, o, r, s, t, w, er2lowest6n e w er3w i d er2newNow the most frequent pair iser, which we merge; our system has learnedthat there should be a token for word-\ufb01naler, represented aser:corpus vocabulary5low,d,e,i,l,n,o,r,s,t,w,er,er2lowest6n e w er3w i d er2newNextne(total count of 8) get merged tone:corpus vocabulary5low,d,e,i,l,n,o,r,s,t,w,er,er,ne2lowest6ne w er3w i d er2ne wIf we continue, the next merges are:Merge Current Vocabulary(ne, w),d,e,i,l,n,o,r,s,t,w,er,er,ne,new(l, o),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo(lo, w),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low(new, er),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low,newer(low,),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low,newer,lowOnce we\u2019ve learned our vocabulary, thetoken parseris used to tokenize a testsentence. The token parser just runs on the test data the merges we have learned1Note that there can be ties; we could have instead chosen to merger\ufb01rst, since that also has afrequency of 9. BPEMerge er  _ to er_18CHAPTER2\u2022REGULAREXPRESSIONS,TEXTNORMALIZATION,EDITDISTANCEThe algorithm is usually run inside words (not merging across word boundaries),so the input corpus is \ufb01rst white-space-separated to give a set of strings, each corre-sponding to the characters of a word, plus a special end-of-word symbol, and itscounts. Let\u2019s see its operation on the following tiny input corpus of 18 word tokenswith counts for each word (the wordlowappears 5 times, the wordnewer6 times,and so on), which would have a starting vocabulary of 11 letters:corpus vocabulary5low, d, e, i, l, n, o, r, s, t, w2lowest6newer3wider2newThe BPE algorithm \ufb01rst count all pairs of adjacent symbols: the most frequentis the pairerbecause it occurs innewer(frequency of 6) andwider(frequency of3) for a total of 9 occurrences1. We then merge these symbols, treatingeras onesymbol, and count again:corpus vocabulary5low, d, e, i, l, n, o, r, s, t, w, er2lowest6n e w er3w i d er2newNow the most frequent pair iser, which we merge; our system has learnedthat there should be a token for word-\ufb01naler, represented aser:corpus vocabulary5low,d,e,i,l,n,o,r,s,t,w,er,er2lowest6n e w er3w i d er2newNextne(total count of 8) get merged tone:corpus vocabulary5low,d,e,i,l,n,o,r,s,t,w,er,er,ne2lowest6ne w er3w i d er2ne wIf we continue, the next merges are:Merge Current Vocabulary(ne, w),d,e,i,l,n,o,r,s,t,w,er,er,ne,new(l, o),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo(lo, w),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low(new, er),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low,newer(low,),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low,newer,lowOnce we\u2019ve learned our vocabulary, thetoken parseris used to tokenize a testsentence. The token parser just runs on the test data the merges we have learned1Note that there can be ties; we could have instead chosen to merger\ufb01rst, since that also has afrequency of 9.18CHAPTER2\u2022REGULAREXPRESSIONS,TEXTNORMALIZATION,EDITDISTANCEThe algorithm is usually run inside words (not merging across word boundaries),so the input corpus is \ufb01rst white-space-separated to give a set of strings, each corre-sponding to the characters of a word, plus a special end-of-word symbol, and itscounts. Let\u2019s see its operation on the following tiny input corpus of 18 word tokenswith counts for each word (the wordlowappears 5 times, the wordnewer6 times,and so on), which would have a starting vocabulary of 11 letters:corpus vocabulary5low, d, e, i, l, n, o, r, s, t, w2lowest6newer3wider2newThe BPE algorithm \ufb01rst count all pairs of adjacent symbols: the most frequentis the pairerbecause it occurs innewer(frequency of 6) andwider(frequency of3) for a total of 9 occurrences1. We then merge these symbols, treatingeras onesymbol, and count again:corpus vocabulary5low, d, e, i, l, n, o, r, s, t, w, er2lowest6n e w er3w i d er2newNow the most frequent pair iser, which we merge; our system has learnedthat there should be a token for word-\ufb01naler, represented aser:corpus vocabulary5low,d,e,i,l,n,o,r,s,t,w,er,er2lowest6n e w er3w i d er2newNextne(total count of 8) get merged tone:corpus vocabulary5low,d,e,i,l,n,o,r,s,t,w,er,er,ne2lowest6ne w er3w i d er2ne wIf we continue, the next merges are:Merge Current Vocabulary(ne, w),d,e,i,l,n,o,r,s,t,w,er,er,ne,new(l, o),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo(lo, w),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low(new, er),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low,newer(low,),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low,newer,lowOnce we\u2019ve learned our vocabulary, thetoken parseris used to tokenize a testsentence. The token parser just runs on the test data the merges we have learned1Note that there can be ties; we could have instead chosen to merger\ufb01rst, since that also has afrequency of 9. BPEMerge n  e  to ne18CHAPTER2\u2022REGULAREXPRESSIONS,TEXTNORMALIZATION,EDITDISTANCEThe algorithm is usually run inside words (not merging across word boundaries),so the input corpus is \ufb01rst white-space-separated to give a set of strings, each corre-sponding to the characters of a word, plus a special end-of-word symbol, and itscounts. Let\u2019s see its operation on the following tiny input corpus of 18 word tokenswith counts for each word (the wordlowappears 5 times, the wordnewer6 times,and so on), which would have a starting vocabulary of 11 letters:corpus vocabulary5low, d, e, i, l, n, o, r, s, t, w2lowest6newer3wider2newThe BPE algorithm \ufb01rst count all pairs of adjacent symbols: the most frequentis the pairerbecause it occurs innewer(frequency of 6) andwider(frequency of3) for a total of 9 occurrences1. We then merge these symbols, treatingeras onesymbol, and count again:corpus vocabulary5low, d, e, i, l, n, o, r, s, t, w, er2lowest6n e w er3w i d er2newNow the most frequent pair iser, which we merge; our system has learnedthat there should be a token for word-\ufb01naler, represented aser:corpus vocabulary5low,d,e,i,l,n,o,r,s,t,w,er,er2lowest6n e w er3w i d er2newNextne(total count of 8) get merged tone:corpus vocabulary5low,d,e,i,l,n,o,r,s,t,w,er,er,ne2lowest6ne w er3w i d er2ne wIf we continue, the next merges are:Merge Current Vocabulary(ne, w),d,e,i,l,n,o,r,s,t,w,er,er,ne,new(l, o),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo(lo, w),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low(new, er),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low,newer(low,),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low,newer,lowOnce we\u2019ve learned our vocabulary, thetoken parseris used to tokenize a testsentence. The token parser just runs on the test data the merges we have learned1Note that there can be ties; we could have instead chosen to merger\ufb01rst, since that also has afrequency of 9.18CHAPTER2\u2022REGULAREXPRESSIONS,TEXTNORMALIZATION,EDITDISTANCEThe algorithm is usually run inside words (not merging across word boundaries),so the input corpus is \ufb01rst white-space-separated to give a set of strings, each corre-sponding to the characters of a word, plus a special end-of-word symbol, and itscounts. Let\u2019s see its operation on the following tiny input corpus of 18 word tokenswith counts for each word (the wordlowappears 5 times, the wordnewer6 times,and so on), which would have a starting vocabulary of 11 letters:corpus vocabulary5low, d, e, i, l, n, o, r, s, t, w2lowest6newer3wider2newThe BPE algorithm \ufb01rst count all pairs of adjacent symbols: the most frequentis the pairerbecause it occurs innewer(frequency of 6) andwider(frequency of3) for a total of 9 occurrences1. We then merge these symbols, treatingeras onesymbol, and count again:corpus vocabulary5low, d, e, i, l, n, o, r, s, t, w, er2lowest6n e w er3w i d er2newNow the most frequent pair iser, which we merge; our system has learnedthat there should be a token for word-\ufb01naler, represented aser:corpus vocabulary5low,d,e,i,l,n,o,r,s,t,w,er,er2lowest6n e w er3w i d er2newNextne(total count of 8) get merged tone:corpus vocabulary5low,d,e,i,l,n,o,r,s,t,w,er,er,ne2lowest6ne w er3w i d er2ne wIf we continue, the next merges are:Merge Current Vocabulary(ne, w),d,e,i,l,n,o,r,s,t,w,er,er,ne,new(l, o),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo(lo, w),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low(new, er),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low,newer(low,),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low,newer,lowOnce we\u2019ve learned our vocabulary, thetoken parseris used to tokenize a testsentence. The token parser just runs on the test data the merges we have learned1Note that there can be ties; we could have instead chosen to merger\ufb01rst, since that also has afrequency of 9. BPEThe next merges are:18CHAPTER2\u2022REGULAREXPRESSIONS,TEXTNORMALIZATION,EDITDISTANCEThe algorithm is usually run inside words (not merging across word boundaries),so the input corpus is \ufb01rst white-space-separated to give a set of strings, each corre-sponding to the characters of a word, plus a special end-of-word symbol, and itscounts. Let\u2019s see its operation on the following tiny input corpus of 18 word tokenswith counts for each word (the wordlowappears 5 times, the wordnewer6 times,and so on), which would have a starting vocabulary of 11 letters:corpus vocabulary5low, d, e, i, l, n, o, r, s, t, w2lowest6newer3wider2newThe BPE algorithm \ufb01rst count all pairs of adjacent symbols: the most frequentis the pairerbecause it occurs innewer(frequency of 6) andwider(frequency of3) for a total of 9 occurrences1. We then merge these symbols, treatingeras onesymbol, and count again:corpus vocabulary5low, d, e, i, l, n, o, r, s, t, w, er2lowest6n e w er3w i d er2newNow the most frequent pair iser, which we merge; our system has learnedthat there should be a token for word-\ufb01naler, represented aser:corpus vocabulary5low,d,e,i,l,n,o,r,s,t,w,er,er2lowest6n e w er3w i d er2newNextne(total count of 8) get merged tone:corpus vocabulary5low,d,e,i,l,n,o,r,s,t,w,er,er,ne2lowest6ne w er3w i d er2ne wIf we continue, the next merges are:Merge Current Vocabulary(ne, w),d,e,i,l,n,o,r,s,t,w,er,er,ne,new(l, o),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo(lo, w),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low(new, er),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low,newer(low,),d,e,i,l,n,o,r,s,t,w,er,er,ne,new,lo,low,newer,lowOnce we\u2019ve learned our vocabulary, thetoken parseris used to tokenize a testsentence. The token parser just runs on the test data the merges we have learned1Note that there can be ties; we could have instead chosen to merger\ufb01rst, since that also has afrequency of 9. BPE token segmenteralgorithmOn the test data, run each merge learned from the training data:\u25e6Greedily\u25e6In the order we learned them\u25e6(test frequencies don't play a role)So: merge every e rto er, then merge er _to er_, etc.Result: \u25e6Test set \"n e w e r _\" would be tokenized as a full word \u25e6Test set \"l o w e r _\" would be two tokens: \"low er_\" Properties of BPE tokensUsually include frequent wordsAnd frequent subwords\u2022Which are often morphemes like -estor \u2013erA morpheme is the smallest meaning-bearing unit of a language\u2022unlikeliest has 3 morphemes un-, likely, and -est Basic Text ProcessingByte Pair Encoding Basic Text ProcessingWord Normalization and other issues Word NormalizationPutting words/tokens in a standard format\u25e6U.S.A. or USA\u25e6uhhuhor uh-huh\u25e6Fed or fed\u25e6am, is, be, are  Case foldingApplications like IR: reduce all letters to lower case\u25e6Since users tend to use lower case\u25e6Possible exception: upper case in mid-sentence?\u25e6e.g., General Motors\u25e6Fedvs. fed\u25e6SAILvs. sailFor sentiment analysis, MT, Information extraction\u25e6Case is helpful (USversus us is important) LemmatizationRepresent all words as their lemma, their shared root = dictionary headword form:\u25e6am, are,is \u00aebe\u25e6car, cars, car's, cars'\u00aecar\u25e6Spanish quiero(\u2018I want\u2019), quieres(\u2018you want\u2019) \u00aequerer\u2018want'\u25e6He is reading detective stories \u00aeHe be read detective story  Lemmatization is done by Morphological ParsingMorphemes:\u25e6The small meaningful units that make up words\u25e6Stems: The core meaning-bearing units\u25e6Affixes: Parts that adhere to stems, often with grammatical functionsMorphological Parsers:\u25e6Parsecats into two morphemes cat and s\u25e6Parse Spanish amaren(\u2018if in the future they would love\u2019) into morpheme amar\u2018to love\u2019, and the morphological features 3PL and future subjunctive.  StemmingReduce terms to stems, chopping off affixes crudelyThis was not the map we found in Billy Bones\u2019s chest, but an accurate copy, complete in all things-names and heights and soundings-with the single exception of the red crosses and the written notes. Thi wa not the map we found in Billi Bone s chest but an accur copi complet in all thing name and height and sound with the singl except of the red cross and the written note .  Porter StemmerBased on a series of rewrite rules run in series\u25e6A cascade, in which output of each pass fed to next passSome sample rules:20CHAPTER2\u2022REGULAREXPRESSIONS,TEXTNORMALIZATION,EDITDISTANCEbe; the wordsdinneranddinnersboth have the lemmadinner. Lemmatizing each ofthese forms to the same lemma will let us \ufb01nd all mentions of words in Russian likeMoscow. The lemmatized form of a sentence likeHe is reading detective storieswould thus beHe be read detective story.How is lemmatization done? The most sophisticated methods for lemmatizationinvolve completemorphological parsingof the word.Morphologyis the study ofthe way words are built up from smaller meaning-bearing units calledmorphemes.morphemeTwo broad classes of morphemes can be distinguished:stems\u2014the central mor-stempheme of the word, supplying the main meaning\u2014 andaf\ufb01xes\u2014adding \u201cadditional\u201daf\ufb01xmeanings of various kinds. So, for example, the wordfoxconsists of one morpheme(the morphemefox) and the wordcatsconsists of two: the morphemecatand themorpheme-s. A morphological parser takes a word likecatsand parses it into thetwo morphemescatands, or parses a Spanish word likeamaren(\u2018if in the futurethey would love\u2019) into the morphemeamar\u2018to love\u2019, and the morphological features3PLandfuture subjunctive.The Porter StemmerLemmatization algorithms can be complex. For this reason we sometimes make useof a simpler but cruder method, which mainly consists of chopping off word-\ufb01nalaf\ufb01xes. This naive version of morphological analysis is calledstemming. One ofstemmingthe most widely used stemming algorithms is thePorter (1980). The Porter stemmerPorter stemmerapplied to the following paragraph:This was not the map we found in Billy Bones\u2019s chest, butan accurate copy, complete in all things-names and heightsand soundings-with the single exception of the red crossesand the written notes.produces the following stemmed output:Thi wa not the map we found in Billi Bone s chest but anaccur copi complet in all thing name and height and soundwith the singl except of the red cross and the written noteThe algorithm is based on series of rewrite rules run in series, as acascade, incascadewhich the output of each pass is fed as input to the next pass; here is a sampling ofthe rules:ATIONAL!ATE (e.g., relational!relate)ING!\u270fif stem contains vowel (e.g., motoring!motor)SSES!SS (e.g., grasses!grass)Detailed rule lists for the Porter stemmer, as well as code (in Java, Python, etc.)can be found on Martin Porter\u2019s homepage; see also the original paper(Porter, 1980).Simple stemmers can be useful in cases where we need to collapse across differ-ent variants of the same lemma. Nonetheless, they do tend to commit errors of bothover- and under-generalizing, as shown in the table below(Krovetz, 1993):Errors of CommissionErrors of OmissionorganizationorganEuropeanEuropedoingdoeanalysisanalyzesnumericalnumerousnoisenoisypolicypolicesparsesparsity Dealing with complex morphology is necessary for many languages\u25e6e.g., the Turkish word:\u25e6Uygarlastiramadiklarimizdanmissinizcasina\u25e6`(behaving) as if you are among those whom we could not civilize\u2019\u25e6Uygar`civilized\u2019 + las`become\u2019 + tir`cause\u2019 + ama`not able\u2019 + dik`past\u2019 + lar\u2018plural\u2019+ imiz\u2018p1pl\u2019 + dan\u2018abl\u2019 + mis\u2018past\u2019 + siniz\u20182pl\u2019 + casina\u2018as if\u2019  Sentence Segmentation!, ? mostly unambiguous but period\u201c.\u201d is very ambiguous\u25e6Sentence boundary\u25e6Abbreviations like Inc. or Dr.\u25e6Numbers like .02% or 4.3Common algorithm: Tokenize first: use rules or ML to classify a period as either (a) part of the word or (b) a sentence-boundary. \u25e6An abbreviation dictionary can helpSentence segmentation can then often be done by rules based on this tokenization. Basic Text ProcessingWord Normalization and other issues", "med24.pdf": "Minimum Edit DistanceDefinition of Minimum Edit Distance Dan JurafskyHow similar are two strings?\u2022Spell correction\u2022The user typed \u201cgraffe\u201dWhich is closest? \u2022graf\u2022graft\u2022grail\u2022giraffe\u2022Computational Biology\u2022Align two sequences of nucleotides\u2022Resulting alignment:\u2022Also for Machine Translation, Information Extraction, Speech RecognitionAGGCTATCACCTGACCTCCAGGCCGATGCCCTAGCTATCACGACCGCGGTCGATTTGCCCGAC-AGGCTATCACCTGACCTCCAGGCCGA--TGCCC---TAG-CTATCAC--GACCGC--GGTCGATTTGCCCGAC Dan JurafskyEdit Distance\u2022The minimum edit distance between two strings\u2022Is the minimum number of editing operations\u2022Insertion\u2022Deletion\u2022Substitution\u2022Needed to transform one into the other Dan JurafskyMinimum Edit Distance\u2022Two strings and their alignment:\n Dan JurafskyMinimum Edit Distance\u2022If each operation has cost of 1\u2022Distance between these is 5\u2022If substitutions cost 2 (Levenshtein)\u2022Distance between them is 8\n Dan JurafskyAlignment in Computational Biology\u2022Given a sequence of bases\u2022An alignment:\u2022Given two sequences, align each letter to a letter or gap-AGGCTATCACCTGACCTCCAGGCCGA--TGCCC---TAG-CTATCAC--GACCGC--GGTCGATTTGCCCGACAGGCTATCACCTGACCTCCAGGCCGATGCCCTAGCTATCACGACCGCGGTCGATTTGCCCGAC Dan JurafskyOther uses of Edit Distance in NLP\u2022Evaluating Machine Translation and speech recognitionR Spokesman confirms    senior government adviser was shotH Spokesman said    the senior            adviser was shot dead              S      I              D                        I\u2022Named Entity Extraction and Entity Coreference\u2022IBM Inc. announced today\u2022IBM profits\u2022Stanford President John Hennessy announced yesterday\u2022for Stanford University President John Hennessy Dan JurafskyHow to find the Min Edit Distance?\u2022Searching for a path (sequence of edits) from the start string to the final string:\u2022Initial state: the word we\u2019re transforming\u2022Operators: insert, delete, substitute\u2022Goal state:  the word we\u2019re trying to get to\u2022Path cost: what we want to minimize: the number of edits8\n Dan JurafskyMinimum Edit as Search\u2022But the space of all edit sequences is huge!\u2022We can\u2019t afford to navigate na\u00efvely\u2022Lots of distinct paths wind up at the same state.\u2022We don\u2019t have to keep track of all of them\u2022Just the shortest path to each of those revisted states.9 Dan JurafskyDefining Min Edit Distance\u2022For two strings\u2022X of length n \u2022Y of length m\u2022We define D(i,j)\u2022the edit distance between X[1..i] and Y[1..j] \u2022i.e., the first i characters of X and the first j characters of Y\u2022The edit distance between X and Y is thus D(n,m) Minimum Edit DistanceDefinition of Minimum Edit Distance Minimum Edit DistanceComputing Minimum Edit Distance Dan JurafskyDynamic Programming forMinimum Edit Distance\u2022Dynamic programming: A tabular computation of D(n,m)\u2022Solving problems by combining solutions to subproblems.\u2022Bottom-up\u2022We compute D(i,j) for small i,j \u2022And compute larger D(i,j) based on previously computed smaller values\u2022i.e., compute D(i,j) for all i (0 < i < n)  and j (0 < j < m) Dan JurafskyDefining Min Edit Distance (Levenshtein)\u2022InitializationD(i,0) = iD(0,j) = j\u2022Recurrence Relation:For each  i = 1\u2026M   For each  j = 1\u2026N                       D(i-1,j) + 1          D(i,j)= min  D(i,j-1) + 1                       D(i-1,j-1) +   2; if X(i) \u2260 Y(j)                                         0; if X(i) = Y(j)\u2022Termination:D(N,M) is distance  Dan JurafskyN9O8I7T6N5E4T3N2I1#0123456789#EXECUTIONThe Edit Distance Table Dan JurafskyN9O8I7T6N5E4T3N2I1#0123456789#EXECUTION\nThe Edit Distance Table Dan JurafskyN9O8I7T6N5E4T3N2I1#0123456789#EXECUTIONEdit Distance\n Dan JurafskyN989101112111098O8789101110989I767891098910T656789891011N5456789101110E43456789109T3456787898N2345678787I1234567678#0123456789#EXECUTIONThe Edit Distance Table Minimum Edit DistanceComputing Minimum Edit Distance Minimum Edit DistanceBacktrace for Computing Alignments Dan JurafskyComputing alignments\u2022Edit distance isn\u2019t sufficient\u2022We often need to align each character of the two strings to each other\u2022We do this by keeping a \u201cbacktrace\u201d\u2022Every time we enter a cell, remember where we came from\u2022When we reach the end, \u2022Trace back the path from the upper right corner to read off the alignment Dan JurafskyN9O8I7T6N5E4T3N2I1#0123456789#EXECUTIONEdit Distance\n Dan JurafskyMinEdit with Backtrace\n Dan JurafskyAdding Backtrace to Minimum Edit Distance\u2022Base conditions:                                                        Termination:D(i,0) = i         D(0,j) = j         D(N,M) is distance \u2022Recurrence Relation:For each  i = 1\u2026M  For each  j = 1\u2026N                      D(i-1,j) + 1         D(i,j)= min  D(i,j-1) + 1                      D(i-1,j-1) +  2; if X(i) \u2260 Y(j)                                       0; if X(i) = Y(j)                     LEFT         ptr(i,j)=   DOWN                     DIAGinsertiondeletionsubstitutioninsertiondeletionsubstitution Dan JurafskyThe Distance Matrix\nSlide adapted from Serafim Batzoglouy0 \u2026\u2026\u2026\u2026\u2026\u2026\u2026\u2026\u2026\u2026\u2026\u2026  yMx0 \u2026\u2026\u2026\u2026\u2026\u2026\u2026\u2026  xNEvery non-decreasing path from (0,0) to (M, N) corresponds to an alignment of the two sequencesAn optimal alignment is composed of optimal subalignments Dan JurafskyResult of Backtrace\u2022Two strings and their alignment:\n Dan JurafskyPerformance\u2022Time:    O(nm)\u2022Space:    O(nm)\u2022Backtrace    O(n+m) Minimum Edit DistanceBacktrace for Computing Alignments Minimum Edit DistanceWeighted Minimum Edit Distance Dan JurafskyWeighted Edit Distance\u2022Why would we add weights to the computation?\u2022Spell Correction: some letters are more likely to be mistyped than others\u2022Biology: certain kinds of deletions or insertions are more likely than others Dan JurafskyConfusion matrix for spelling errors\n Dan Jurafsky\n Dan JurafskyWeighted Min Edit Distance\u2022Initialization:D(0,0) = 0D(i,0) = D(i-1,0) + del[x(i)];    1 < i \u2264 ND(0,j) = D(0,j-1) + ins[y(j)];    1 < j \u2264 M\u2022Recurrence Relation:             D(i-1,j)   + del[x(i)]D(i,j)= min  D(i,j-1)   + ins[y(j)]             D(i-1,j-1) + sub[x(i),y(j)]\u2022Termination:D(N,M) is distance  Dan JurafskyWhere did the name, dynamic programming, come from? \u2026The 1950s were not good years for mathematical research. [the] Secretary of Defense \u2026had a pathological fear and hatred of the word, research\u2026 I decided therefore to use the word, \u201cprogramming\u201d. I wanted to get across the idea that this was dynamic, this was multistage\u2026 I thought, let\u2019s \u2026 take a word that has an absolutely precise meaning, namely dynamic\u2026 it\u2019s impossible to use the word, dynamic, in a pejorative sense. Try thinking of some combination that will possibly give it a pejorative meaning. It\u2019s impossible. Thus, I thought dynamic programming was a good name. It was something not even a Congressman could object to.\u201d    Richard Bellman, \u201cEye of the Hurricane: an autobiography\u201d 1984. Minimum Edit DistanceWeighted Minimum Edit Distance", "3.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\n3N-gram Language Models\n\u201cYou are uniformly charming!\u201d cried he, with a smile of associating and now\nand then I bowed and they perceived a chaise and four to wish for.\nRandom sentence generated from a Jane Austen trigram model\nPredicting is dif\ufb01cult\u2014especially about the future, as the old quip goes. But how\nabout predicting something that seems much easier, like the next word someone is\ngoing to say? What word, for example, is likely to follow\nThe water of Walden Pond is so beautifully ...\nYou might conclude that a likely word is blue , orgreen , orclear , but probably not\nrefrigerator northis . In this chapter we formalize this intuition by introducing\nlanguage models orLMs , models that assign a probability to each possible next language model\nLM word. Language models can also assign a probability to an entire sentence, telling\nus that the following sequence has a much higher probability of appearing in a text:\nall of a sudden I notice three guys standing on the sidewalk\nthan does this same set of words in a different order:\non guys all I of notice sidewalk three a sudden standing the\nWhy would we want to predict upcoming words, or know the probability of a sen-\ntence? One reason is for generation: choosing contextually better words. For ex-\nample we can correct grammar or spelling errors like Their are two midterms ,\nin which There was mistyped as Their , orEverything has improve , in which\nimprove should have been improved . The phrase There are is more probable\nthanTheir are , and has improved thanhas improve , so a language model can\nhelp users select the more grammatical variant. Or for a speech system to recognize\nthat you said I will be back soonish and not I will be bassoon dish , it\nhelps to know that back soonish is a more probable sequence. Language models\ncan also help in augmentative and alternative communication (Trnka et al. 2007,\nKane et al. 2017). People can use AAC systems if they are physically unable to AAC\nspeak or sign but can instead use eye gaze or other movements to select words from\na menu. Word prediction can be used to suggest likely words for the menu.\nWord prediction is also central to NLP for another reason: large language mod-\nelsare built just by training them to predict words!! As we\u2019ll see in chapters 7-9,\nlarge language models learn an enormous amount about language solely from being\ntrained to predict upcoming words from neighboring words.\nIn this chapter we introduce the simplest kind of language model: the n-gram n-gram\nlanguage model. An n-gram is a sequence of nwords: a 2-gram (which we\u2019ll call\nbigram ) is a two-word sequence of words like The water , orwater of , and a 3-\ngram (a trigram ) is a three-word sequence of words like The water of , orwater 2CHAPTER 3 \u2022 N- GRAM LANGUAGE MODELS\nof Walden . But we also (in a bit of terminological ambiguity) use the word \u2018n-\ngram\u2019 to mean a probabilistic model that can estimate the probability of a word given\nthe n-1 previous words, and thereby also to assign probabilities to entire sequences.\nIn later chapters we will introduce the much more powerful neural large lan-\nguage models , based on the transformer architecture of Chapter 9. But because\nn-grams have a remarkably simple and clear formalization, we use them to intro-\nduce some major concepts of large language modeling, including training and test\nsets,perplexity ,sampling , and interpolation .\n3.1 N-Grams\nLet\u2019s begin with the task of computing P(wjh), the probability of a word wgiven\nsome history h. Suppose the history his \u201cThe water of Walden Pond is so\nbeautifully \u201d and we want to know the probability that the next word is blue :\nP(bluejThe water of Walden Pond is so beautifully ) (3.1)\nOne way to estimate this probability is directly from relative frequency counts: take a\nvery large corpus, count the number of times we see The water of Walden Pond\nis so beautifully , and count the number of times this is followed by blue . This\nwould be answering the question \u201cOut of the times we saw the history h, how many\ntimes was it followed by the word w\u201d, as follows:\nP(bluejThe water of Walden Pond is so beautifully ) =\nC(The water of Walden Pond is so beautifully blue )\nC(The water of Walden Pond is so beautifully )(3.2)\nIf we had a large enough corpus, we could compute these two counts and estimate\nthe probability from Eq. 3.2. But even the entire web isn\u2019t big enough to give us\ngood estimates for counts of entire sentences. This is because language is creative ;\nnew sentences are invented all the time, and we can\u2019t expect to get accurate counts\nfor such large objects as entire sentences. For this reason, we\u2019ll need more clever\nways to estimate the probability of a word wgiven a history h, or the probability of\nan entire word sequence W.\nLet\u2019s start with some notation. First, throughout this chapter we\u2019ll continue to\nrefer to words , although in practice we usually compute language models over to-\nkens like the BPE tokens of page ??. To represent the probability of a particular\nrandom variable Xitaking on the value \u201cthe\u201d, or P(Xi=\u201cthe\u201d), we will use the\nsimpli\ufb01cation P(the). We\u2019ll represent a sequence of nwords either as w1:::wnor\nw1:n. Thus the expression w1:n\u00001means the string w1;w2;:::;wn\u00001, but we\u2019ll also\nbe using the equivalent notation w<n, which can be read as \u201call the elements of w\nfrom w1up to and including wn\u00001\u201d. For the joint probability of each word in a se-\nquence having a particular value P(X1=w1;X2=w2;X3=w3;:::;Xn=wn)we\u2019ll\nuseP(w1;w2;:::;wn).\nNow, how can we compute probabilities of entire sequences like P(w1;w2;:::;wn)?\nOne thing we can do is decompose this probability using the chain rule of proba-\nbility :\nP(X1:::Xn) = P(X1)P(X2jX1)P(X3jX1:2):::P(XnjX1:n\u00001)\n=nY\nk=1P(XkjX1:k\u00001) (3.3) 3.1 \u2022 N-G RAMS 3\nApplying the chain rule to words, we get\nP(w1:n) = P(w1)P(w2jw1)P(w3jw1:2):::P(wnjw1:n\u00001)\n=nY\nk=1P(wkjw1:k\u00001) (3.4)\nThe chain rule shows the link between computing the joint probability of a sequence\nand computing the conditional probability of a word given previous words. Equa-\ntion 3.4 suggests that we could estimate the joint probability of an entire sequence of\nwords by multiplying together a number of conditional probabilities. But using the\nchain rule doesn\u2019t really seem to help us! We don\u2019t know any way to compute the\nexact probability of a word given a long sequence of preceding words, P(wnjw1:n\u00001).\nAs we said above, we can\u2019t just estimate by counting the number of times every word\noccurs following every long string in some corpus, because language is creative and\nany particular context might have never occurred before!\n3.1.1 The Markov assumption\nThe intuition of the n-gram model is that instead of computing the probability of a\nword given its entire history, we can approximate the history by just the last few\nwords.\nThebigram model, for example, approximates the probability of a word given bigram\nall the previous words P(wnjw1:n\u00001)by using only the conditional probability of the\npreceding word P(wnjwn\u00001). In other words, instead of computing the probability\nP(bluejThe water of Walden Pond is so beautifully ) (3.5)\nwe approximate it with the probability\nP(bluejbeautifully ) (3.6)\nWhen we use a bigram model to predict the conditional probability of the next word,\nwe are thus making the following approximation:\nP(wnjw1:n\u00001)\u0019P(wnjwn\u00001) (3.7)\nThe assumption that the probability of a word depends only on the previous word is\ncalled a Markov assumption. Markov models are the class of probabilistic models Markov\nthat assume we can predict the probability of some future unit without looking too\nfar into the past. We can generalize the bigram (which looks one word into the past)\nto the trigram (which looks two words into the past) and thus to the n-gram (which n-gram\nlooks n\u00001 words into the past).\nLet\u2019s see a general equation for this n-gram approximation to the conditional\nprobability of the next word in a sequence. We\u2019ll use Nhere to mean the n-gram\nsize, so N=2 means bigrams and N=3 means trigrams. Then we approximate the\nprobability of a word given its entire context as follows:\nP(wnjw1:n\u00001)\u0019P(wnjwn\u0000N+1:n\u00001) (3.8)\nGiven the bigram assumption for the probability of an individual word, we can com-\npute the probability of a complete word sequence by substituting Eq. 3.7 into Eq. 3.4:\nP(w1:n)\u0019nY\nk=1P(wkjwk\u00001) (3.9) 4CHAPTER 3 \u2022 N- GRAM LANGUAGE MODELS\n3.1.2 How to estimate probabilities\nHow do we estimate these bigram or n-gram probabilities? An intuitive way to\nestimate probabilities is called maximum likelihood estimation orMLE . We getmaximum\nlikelihood\nestimationthe MLE estimate for the parameters of an n-gram model by getting counts from\na corpus, and normalizing the counts so that they lie between 0 and 1. For proba- normalize\nbilistic models, normalizing means dividing by some total count so that the resulting\nprobabilities fall between 0 and 1 and sum to 1.\nFor example, to compute a particular bigram probability of a word wngiven a\nprevious word wn\u00001, we\u2019ll compute the count of the bigram C(wn\u00001wn)and normal-\nize by the sum of all the bigrams that share the same \ufb01rst word wn\u00001:\nP(wnjwn\u00001) =C(wn\u00001wn)P\nwC(wn\u00001w)(3.10)\nWe can simplify this equation, since the sum of all bigram counts that start with\na given word wn\u00001must be equal to the unigram count for that word wn\u00001(the reader\nshould take a moment to be convinced of this):\nP(wnjwn\u00001) =C(wn\u00001wn)\nC(wn\u00001)(3.11)\nLet\u2019s work through an example using a mini-corpus of three sentences. We\u2019ll\n\ufb01rst need to augment each sentence with a special symbol <s>at the beginning\nof the sentence, to give us the bigram context of the \ufb01rst word. We\u2019ll also need a\nspecial end-symbol </s> .1\n<s> I am Sam </s>\n<s> Sam I am </s>\n<s> I do not like green eggs and ham </s>\nHere are the calculations for some of the bigram probabilities from this corpus\nP(I|<s> ) =2\n3=0:67 P(Sam|<s> ) =1\n3=0:33 P(am|I) =2\n3=0:67\nP(</s>|Sam ) =1\n2=0:5 P(Sam|am ) =1\n2=0:5 P(do|I) =1\n3=0:33\nFor the general case of MLE n-gram parameter estimation:\nP(wnjwn\u0000N+1:n\u00001) =C(wn\u0000N+1:n\u00001wn)\nC(wn\u0000N+1:n\u00001)(3.12)\nEquation 3.12 (like Eq. 3.11) estimates the n-gram probability by dividing the\nobserved frequency of a particular sequence by the observed frequency of a pre\ufb01x.\nThis ratio is called a relative frequency . We said above that this use of relativerelative\nfrequency\nfrequencies as a way to estimate probabilities is an example of maximum likelihood\nestimation or MLE. In MLE, the resulting parameter set maximizes the likelihood of\nthe training set Tgiven the model M(i.e., P(TjM)). For example, suppose the word\nChinese occurs 400 times in a corpus of a million words. What is the probability\nthat a random word selected from some other text of, say, a million words will be the\nword Chinese ? The MLE of its probability is400\n1000000or 0:0004. Now 0 :0004 is not\nthe best possible estimate of the probability of Chinese occurring in all situations; it\n1We need the end-symbol to make the bigram grammar a true probability distribution. Without an end-\nsymbol, instead of the sentence probabilities of all sentences summing to one, the sentence probabilities\nfor all sentences of a given length would sum to one. This model would de\ufb01ne an in\ufb01nite set of probability\ndistributions, with one distribution per sentence length. See Exercise 3.5. 3.1 \u2022 N-G RAMS 5\nmight turn out that in some other corpus or context Chinese is a very unlikely word.\nBut it is the probability that makes it most likely that Chinese will occur 400 times\nin a million-word corpus. We present ways to modify the MLE estimates slightly to\nget better probability estimates in Section 3.6.\nLet\u2019s move on to some examples from a real but tiny corpus, drawn from the\nnow-defunct Berkeley Restaurant Project, a dialogue system from the last century\nthat answered questions about a database of restaurants in Berkeley, California (Ju-\nrafsky et al., 1994). Here are some sample user queries (text-normalized, by lower\ncasing and with punctuation striped) (a sample of 9332 sentences is on the website):\ncan you tell me about any good cantonese restaurants close by\ntell me about chez panisse\ni\u2019m looking for a good place to eat breakfast\nwhen is caffe venezia open during the day\nFigure 3.1 shows the bigram counts from part of a bigram grammar from text-\nnormalized Berkeley Restaurant Project sentences. Note that the majority of the\nvalues are zero. In fact, we have chosen the sample words to cohere with each other;\na matrix selected from a random set of eight words would be even more sparse.\ni want to eat chinese food lunch spend\ni 5 827 0 9 0 0 0 2\nwant 2 0 608 1 6 6 5 1\nto 2 0 4 686 2 0 6 211\neat 0 0 2 0 16 2 42 0\nchinese 1 0 0 0 0 82 1 0\nfood 15 0 15 0 1 4 0 0\nlunch 2 0 0 0 0 1 0 0\nspend 1 0 1 0 0 0 0 0\nFigure 3.1 Bigram counts for eight of the words (out of V=1446) in the Berkeley Restau-\nrant Project corpus of 9332 sentences. Zero counts are in gray. Each cell shows the count of\nthe column label word following the row label word. Thus the cell in row iand column want\nmeans that want followed i827 times in the corpus.\nFigure 3.2 shows the bigram probabilities after normalization (dividing each cell\nin Fig. 3.1 by the appropriate unigram for its row, taken from the following set of\nunigram counts):\ni want to eat chinese food lunch spend\n2533 927 2417 746 158 1093 341 278\nHere are a few other useful probabilities:\nP(i|<s> ) =0:25 P(english|want ) =0:0011\nP(food|english ) =0:5 P(</s>|food ) =0:68\nNow we can compute the probability of sentences like I want English food or\nI want Chinese food by simply multiplying the appropriate bigram probabilities to-\ngether, as follows:\nP(<s> i want english food </s> )\n=P(i|<s> )P(want|i )P(english|want )\nP(food|english )P(</s>|food )\n=0:25\u00020:33\u00020:0011\u00020:5\u00020:68\n=0:000031 6CHAPTER 3 \u2022 N- GRAM LANGUAGE MODELS\ni want to eat chinese food lunch spend\ni 0.002 0.33 0 0.0036 0 0 0 0.00079\nwant 0.0022 0 0.66 0.0011 0.0065 0.0065 0.0054 0.0011\nto 0.00083 0 0.0017 0.28 0.00083 0 0.0025 0.087\neat 0 0 0.0027 0 0.021 0.0027 0.056 0\nchinese 0.0063 0 0 0 0 0.52 0.0063 0\nfood 0.014 0 0.014 0 0.00092 0.0037 0 0\nlunch 0.0059 0 0 0 0 0.0029 0 0\nspend 0.0036 0 0.0036 0 0 0 0 0\nFigure 3.2 Bigram probabilities for eight words in the Berkeley Restaurant Project corpus\nof 9332 sentences. Zero probabilities are in gray.\nWe leave it as Exercise 3.2 to compute the probability of i want chinese food .\nWhat kinds of linguistic phenomena are captured in these bigram statistics?\nSome of the bigram probabilities above encode some facts that we think of as strictly\nsyntactic in nature, like the fact that what comes after eatis usually a noun or an\nadjective, or that what comes after tois usually a verb. Others might be a fact about\nthe personal assistant task, like the high probability of sentences beginning with\nthe words I. And some might even be cultural rather than linguistic, like the higher\nprobability that people are looking for Chinese versus English food.\n3.1.3 Dealing with scale in large n-gram models\nIn practice, language models can be very large, leading to practical issues.\nLog probabilities Language model probabilities are always stored and computed\nin log space as log probabilities . This is because probabilities are (by de\ufb01nition)log\nprobabilities\nless than or equal to 1, and so the more probabilities we multiply together, the\nsmaller the product becomes. Multiplying enough n-grams together would result\nin numerical under\ufb02ow. Adding in log space is equivalent to multiplying in linear\nspace, so we combine log probabilities by adding them. By adding log probabilities\ninstead of multiplying probabilities, we get results that are not as small. We do all\ncomputation and storage in log space, and just convert back into probabilities if we\nneed to report probabilities at the end by taking the exp of the logprob:\np1\u0002p2\u0002p3\u0002p4=exp(logp1+logp2+logp3+logp4) (3.13)\nIn practice throughout this book, we\u2019ll use log to mean natural log (ln) when the\nbase is not speci\ufb01ed.\nLonger context Although for pedagogical purposes we have only described bi-\ngram models, when there is suf\ufb01cient training data we use trigram models, which trigram\ncondition on the previous two words, or 4-gram or5-gram models. For these larger 4-gram\n5-gram n-grams, we\u2019ll need to assume extra contexts to the left and right of the sentence end.\nFor example, to compute trigram probabilities at the very beginning of the sentence,\nwe use two pseudo-words for the \ufb01rst trigram (i.e., P(I|<s><s> ).\nSome large n-gram datasets have been created, like the million most frequent\nn-grams drawn from the Corpus of Contemporary American English (COCA), a\ncurated 1 billion word corpus of American English (Davies, 2020), Google\u2019s Web\n5-gram corpus from 1 trillion words of English web text (Franz and Brants, 2006),\nor the Google Books Ngrams corpora (800 billion tokens from Chinese, English,\nFrench, German, Hebrew, Italian, Russian, and Spanish) (Lin et al., 2012)). 3.2 \u2022 E VALUATING LANGUAGE MODELS : TRAINING AND TESTSETS 7\nIt\u2019s even possible to use extremely long-range n-gram context. The in\ufb01ni-gram\n(\u00a5-gram) project (Liu et al., 2024) allows n-grams of any length. Their idea is to\navoid the expensive (in space and time) pre-computation of huge n-gram count ta-\nbles. Instead, n-gram probabilities with arbitrary n are computed quickly at inference\ntime by using an ef\ufb01cient representation called suf\ufb01x arrays. This allows computing\nof n-grams of every length for enormous corpora of 5 trillion tokens.\nEf\ufb01ciency considerations are important when building large n-gram language\nmodels. It is standard to quantize the probabilities using only 4-8 bits (instead of\n8-byte \ufb02oats), store the word strings on disk and represent them in memory only as\na 64-bit hash, and represent n-grams in special data structures like \u2018reverse tries\u2019.\nIt is also common to prune n-gram language models, for example by only keeping\nn-grams with counts greater than some threshold or using entropy to prune less-\nimportant n-grams (Stolcke, 1998). Ef\ufb01cient language model toolkits like KenLM\n(Hea\ufb01eld 2011, Hea\ufb01eld et al. 2013) use sorted arrays and use merge sorts to ef\ufb01-\nciently build the probability tables in a minimal number of passes through a large\ncorpus.\n3.2 Evaluating Language Models: Training and Test Sets\nThe best way to evaluate the performance of a language model is to embed it in\nan application and measure how much the application improves. Such end-to-end\nevaluation is called extrinsic evaluation . Extrinsic evaluation is the only way toextrinsic\nevaluation\nknow if a particular improvement in the language model (or any component) is really\ngoing to help the task at hand. Thus for evaluating n-gram language models that are\na component of some task like speech recognition or machine translation, we can\ncompare the performance of two candidate language models by running the speech\nrecognizer or machine translator twice, once with each language model, and seeing\nwhich gives the more accurate transcription.\nUnfortunately, running big NLP systems end-to-end is often very expensive. In-\nstead, it\u2019s helpful to have a metric that can be used to quickly evaluate potential\nimprovements in a language model. An intrinsic evaluation metric is one that mea-intrinsic\nevaluation\nsures the quality of a model independent of any application. In the next section we\u2019ll\nintroduce perplexity , which is the standard intrinsic metric for measuring language\nmodel performance, both for simple n-gram language models and for the more so-\nphisticated neural large language models of Chapter 9.\nIn order to evaluate any machine learning model, we need to have at least three\ndistinct data sets: the training set , the development set , and the test set . training set\ndevelopment\nset\ntest setThe training set is the data we use to learn the parameters of our model; for\nsimple n-gram language models it\u2019s the corpus from which we get the counts that\nwe normalize into the probabilities of the n-gram language model.\nThetest set is a different, held-out set of data, not overlapping with the training\nset, that we use to evaluate the model. We need a separate test set to give us an\nunbiased estimate of how well the model we trained can generalize when we apply\nit to some new unknown dataset. A machine learning model that perfectly captured\nthe training data, but performed terribly on any other data, wouldn\u2019t be much use\nwhen it comes time to apply it to any new data or problem! We thus measure the\nquality of an n-gram model by its performance on this unseen test set or test corpus.\nHow should we choose a training and test set? The test set should re\ufb02ect the\nlanguage we want to use the model for. If we\u2019re going to use our language model 8CHAPTER 3 \u2022 N- GRAM LANGUAGE MODELS\nfor speech recognition of chemistry lectures, the test set should be text of chemistry\nlectures. If we\u2019re going to use it as part of a system for translating hotel booking re-\nquests from Chinese to English, the test set should be text of hotel booking requests.\nIf we want our language model to be general purpose, then the test set should be\ndrawn from a wide variety of texts. In such cases we might collect a lot of texts\nfrom different sources, and then divide it up into a training set and a test set. It\u2019s\nimportant to do the dividing carefully; if we\u2019re building a general purpose model,\nwe don\u2019t want the test set to consist of only text from one document, or one author,\nsince that wouldn\u2019t be a good measure of general performance.\nThus if we are given a corpus of text and want to compare the performance of\ntwo different n-gram models, we divide the data into training and test sets, and train\nthe parameters of both models on the training set. We can then compare how well\nthe two trained models \ufb01t the test set.\nBut what does it mean to \u201c\ufb01t the test set\u201d? The standard answer is simple:\nwhichever language model assigns a higher probability to the test set\u2014which\nmeans it more accurately predicts the test set\u2014is a better model. Given two proba-\nbilistic models, the better model is the one that better predicts the details of the test\ndata, and hence will assign a higher probability to the test data.\nSince our evaluation metric is based on test set probability, it\u2019s important not to\nlet the test sentences into the training set. Suppose we are trying to compute the\nprobability of a particular \u201ctest\u201d sentence. If our test sentence is part of the training\ncorpus, we will mistakenly assign it an arti\ufb01cially high probability when it occurs\nin the test set. We call this situation training on the test set . Training on the test\nset introduces a bias that makes the probabilities all look too high, and causes huge\ninaccuracies in perplexity , the probability-based metric we introduce below.\nEven if we don\u2019t train on the test set, if we test our language model on the\ntest set many times after making different changes, we might implicitly tune to its\ncharacteristics, by noticing which changes seem to make the model better. For this\nreason, we only want to run our model on the test set once, or a very few number of\ntimes, once we are sure our model is ready.\nFor this reason we normally instead have a third dataset called a developmentdevelopment\ntest\ntest set or, devset . We do all our testing on this dataset until the very end, and then\nwe test on the test once to see how good our model is.\nHow do we divide our data into training, development, and test sets? We want\nour test set to be as large as possible, since a small test set may be accidentally un-\nrepresentative, but we also want as much training data as possible. At the minimum,\nwe would want to pick the smallest test set that gives us enough statistical power\nto measure a statistically signi\ufb01cant difference between two potential models. It\u2019s\nimportant that the devset be drawn from the same kind of text as the test set, since\nits goal is to measure how we would do on the test set.\n3.3 Evaluating Language Models: Perplexity\nWe said above that we evaluate language models based on which one assigns a\nhigher probability to the test set. A better model is better at predicting upcoming\nwords, and so it will be less surprised by (i.e., assign a higher probability to) each\nword when it occurs in the test set. Indeed, a perfect language model would correctly\nguess each next word in a corpus, assigning it a probability of 1, and all the other\nwords a probability of zero. So given a test corpus, a better language model will 3.3 \u2022 E VALUATING LANGUAGE MODELS : PERPLEXITY 9\nassign a higher probability to it than a worse language model.\nBut in fact, we do not use raw probability as our metric for evaluating language\nmodels. The reason is that the probability of a test set (or any sequence) depends\non the number of words or tokens in it; the probability of a test set gets smaller the\nlonger the text. We\u2019d prefer a metric that is per-word, normalized by length, so we\ncould compare across texts of different lengths. The metric we use is, a function of\nprobability called perplexity , is one of the most important metrics in NLP, used for\nevaluating large language models as well as n-gram models.\nTheperplexity (sometimes abbreviated as PP or PPL) of a language model on a perplexity\ntest set is the inverse probability of the test set (one over the probability of the test\nset), normalized by the number of words (or tokens). For this reason it\u2019s sometimes\ncalled the per-word or per-token perplexity. We normalize by the number of words\nNby taking the Nth root. For a test set W=w1w2:::wN,:\nperplexity (W) = P(w1w2:::wN)\u00001\nN (3.14)\n=Ns\n1\nP(w1w2:::wN)\nOr we can use the chain rule to expand the probability of W:\nperplexity (W) =NvuutNY\ni=11\nP(wijw1:::wi\u00001)(3.15)\nNote that because of the inverse in Eq. 3.15, the higher the probability of the word\nsequence, the lower the perplexity. Thus the the lower the perplexity of a model on\nthe data, the better the model . Minimizing perplexity is equivalent to maximizing\nthe test set probability according to the language model. Why does perplexity use\nthe inverse probability? It turns out the inverse arises from the original de\ufb01nition\nof perplexity from cross-entropy rate in information theory; for those interested, the\nexplanation is in the advanced Section 3.7. Meanwhile, we just have to remember\nthat perplexity has an inverse relationship with probability.\nThe details of computing the perplexity of a test set Wdepends on which lan-\nguage model we use. Here\u2019s the perplexity of Wwith a unigram language model\n(just the geometric mean of the inverse of the unigram probabilities):\nperplexity (W) =NvuutNY\ni=11\nP(wi)(3.16)\nThe perplexity of Wcomputed with a bigram language model is still a geometric\nmean, but now of the inverse of the bigram probabilities:\nperplexity (W) =NvuutNY\ni=11\nP(wijwi\u00001)(3.17)\nWhat we generally use for word sequence in Eq. 3.15 or Eq. 3.17 is the entire\nsequence of words in some test set. Since this sequence will cross many sentence\nboundaries, if our vocabulary includes a between-sentence token <EOS> or separate\nbegin- and end-sentence markers <s> and</s> then we can include them in the 10 CHAPTER 3 \u2022 N- GRAM LANGUAGE MODELS\nprobability computation. If we do, then we also include one token per sentence in\nthe total count of word tokens N.2\nWe mentioned above that perplexity is a function of both the text and the lan-\nguage model: given a text W, different language models will have different perplex-\nities. Because of this, perplexity can be used to compare different language models.\nFor example, here we trained unigram, bigram, and trigram grammars on 38 million\nwords from the Wall Street Journal newspaper. We then computed the perplexity of\neach of these models on a WSJ test set using Eq. 3.16 for unigrams, Eq. 3.17 for\nbigrams, and the corresponding equation for trigrams. The table below shows the\nperplexity of the 1.5 million word test set according to each of the language models.\nUnigram Bigram Trigram\nPerplexity 962 170 109\nAs we see above, the more information the n-gram gives us about the word\nsequence, the higher the probability the n-gram will assign to the string. A trigram\nmodel is less surprised than a unigram model because it has a better idea of what\nwords might come next, and so it assigns them a higher probability. And the higher\nthe probability, the lower the perplexity (since as Eq. 3.15 showed, perplexity is\nrelated inversely to the probability of the test sequence according to the model). So\na lower perplexity tells us that a language model is a better predictor of the test set.\nNote that in computing perplexities, the language model must be constructed\nwithout any knowledge of the test set, or else the perplexity will be arti\ufb01cially low.\nAnd the perplexity of two language models is only comparable if they use identical\nvocabularies.\nAn (intrinsic) improvement in perplexity does not guarantee an (extrinsic) im-\nprovement in the performance of a language processing task like speech recognition\nor machine translation. Nonetheless, because perplexity usually correlates with task\nimprovements, it is commonly used as a convenient evaluation metric. Still, when\npossible a model\u2019s improvement in perplexity should be con\ufb01rmed by an end-to-end\nevaluation on a real task.\n3.3.1 Perplexity as Weighted Average Branching Factor\nIt turns out that perplexity can also be thought of as the weighted average branch-\ning factor of a language. The branching factor of a language is the number of\npossible next words that can follow any word. For example consider a mini arti\ufb01cial\nlanguage that is deterministic (no probabilities), any word can follow any word, and\nwhose vocabulary consists of only three colors:\nL=fred;blue ;greeng (3.18)\nThe branching factor of this language is 3.\nNow let\u2019s make a probabilistic version of the same LM, let\u2019s call it A, where each\nword follows each other with equal probability1\n3(it was trained on a training set with\nequal counts for the 3 colors), and a test set T= \u201cred red red red blue \u201d.\nLet\u2019s \ufb01rst convince ourselves that if we compute the perplexity of this arti\ufb01cial\ndigit language on this test set (or any such test set) we indeed get 3. By Eq. 3.15, the\n2For example if we use both begin and end tokens, we would include the end-of-sentence marker </s>\nbut not the beginning-of-sentence marker <s>in our count of N; This is because the end-sentence token is\nfollowed directly by the begin-sentence token with probability almost 1, so we don\u2019t want the probability\nof that fake transition to in\ufb02uence our perplexity. 3.4 \u2022 S AMPLING SENTENCES FROM A LANGUAGE MODEL 11\nperplexity of AonTis:\nperplexityA(T) = PA(red red red red blue )\u00001\n5\n= \u00121\n3\u00135!\u00001\n5\n=\u00121\n3\u0013\u00001\n=3 (3.19)\nBut now suppose redwas very likely in the training set a different LM B, and so B\nhas the following probabilities:\nP(red) =0:8P(green ) =0:1P(blue) =0:1 (3.20)\nWe should expect the perplexity of the same test set red red red red blue for\nlanguage model Bto be lower since most of the time the next color will be red, which\nis very predictable, i.e. has a high probability. So the probability of the test set will\nbe higher, and since perplexity is inversely related to probability, the perplexity will\nbe lower. Thus, although the branching factor is still 3, the perplexity or weighted\nbranching factor is smaller:\nperplexityB(T) = PB(red red red red blue )\u00001=5\n=0:04096\u00001\n5\n=0:527\u00001=1:89 (3.21)\n3.4 Sampling sentences from a language model\n010.06the.060.03of0.02a0.02toin.09.11.13.15\u2026however(p=0.0003)polyphonicp=0.0000018\u20260.02.66.99\u2026\nFigure 3.3 A visualization of the sampling distribution for sampling sentences by repeat-\nedly sampling unigrams. The blue bar represents the relative frequency of each word (we\u2019ve\nordered them from most frequent to least frequent, but the choice of order is arbitrary). The\nnumber line shows the cumulative probabilities. If we choose a random number between 0\nand 1, it will fall in an interval corresponding to some word. The expectation for the random\nnumber to fall in the larger intervals of one of the frequent words ( the,of,a) is much higher\nthan in the smaller interval of one of the rare words ( polyphonic ).\nOne important way to visualize what kind of knowledge a language model em-\nbodies is to sample from it. Sampling from a distribution means to choose random sampling\npoints according to their likelihood. Thus sampling from a language model\u2014which\nrepresents a distribution over sentences\u2014means to generate some sentences, choos-\ning each sentence according to its likelihood as de\ufb01ned by the model. Thus we are\nmore likely to generate sentences that the model thinks have a high probability and\nless likely to generate sentences that the model thinks have a low probability. 12 CHAPTER 3 \u2022 N- GRAM LANGUAGE MODELS\nThis technique of visualizing a language model by sampling was \ufb01rst suggested\nvery early on by Shannon (1948) and Miller and Selfridge (1950). It\u2019s simplest to\nvisualize how this works for the unigram case. Imagine all the words of the English\nlanguage covering the number line between 0 and 1, each word covering an interval\nproportional to its frequency. Fig. 3.3 shows a visualization, using a unigram LM\ncomputed from the text of this book. We choose a random value between 0 and 1,\n\ufb01nd that point on the probability line, and print the word whose interval includes this\nchosen value. We continue choosing random numbers and generating words until\nwe randomly generate the sentence-\ufb01nal token </s> .\nWe can use the same technique to generate bigrams by \ufb01rst generating a ran-\ndom bigram that starts with <s>(according to its bigram probability). Let\u2019s say the\nsecond word of that bigram is w. We next choose a random bigram starting with w\n(again, drawn according to its bigram probability), and so on.\n3.5 Generalizing vs. over\ufb01tting the training set\nThe n-gram model, like many statistical models, is dependent on the training corpus.\nOne implication of this is that the probabilities often encode speci\ufb01c facts about a\ngiven training corpus. Another implication is that n-grams do a better and better job\nof modeling the training corpus as we increase the value of N.\nWe can use the sampling method from the prior section to visualize both of\nthese facts! To give an intuition for the increasing power of higher-order n-grams,\nFig. 3.4 shows random sentences generated from unigram, bigram, trigram, and 4-\ngram models trained on Shakespeare\u2019s works.\n1\u2013To him swallowed confess hear both. Which. Of save on trail for are ay device and\nrote life have\ngram \u2013Hill he late speaks; or! a more to leg less \ufb01rst you enter\n2\u2013Why dost stand forth thy canopy, forsooth; he is this palpable hit the King Henry. Live\nking. Follow.\ngram \u2013What means, sir. I confess she? then all sorts, he is trim, captain.\n3\u2013Fly, and will rid me these news of price. Therefore the sadness of parting, as they say,\n\u2019tis done.\ngram \u2013This shall forbid it should be branded, if renown made it empty.\n4\u2013King Henry. What! I will go seek the traitor Gloucester. Exeunt some of the watch. A\ngreat banquet serv\u2019d in;\ngram \u2013It cannot be but so.\nFigure 3.4 Eight sentences randomly generated from four n-grams computed from Shakespeare\u2019s works. All\ncharacters were mapped to lower-case and punctuation marks were treated as words. Output is hand-corrected\nfor capitalization to improve readability.\nThe longer the context, the more coherent the sentences. The unigram sen-\ntences show no coherent relation between words nor any sentence-\ufb01nal punctua-\ntion. The bigram sentences have some local word-to-word coherence (especially\nconsidering punctuation as words). The trigram sentences are beginning to look a\nlot like Shakespeare. Indeed, the 4-gram sentences look a little too much like Shake-\nspeare. The words It cannot be but so are directly from King John . This is because,\nnot to put the knock on Shakespeare, his oeuvre is not very large as corpora go 3.5 \u2022 G ENERALIZING VS .OVERFITTING THE TRAINING SET 13\n(N=884;647;V=29;066), and our n-gram probability matrices are ridiculously\nsparse. There are V2=844;000;000 possible bigrams alone, and the number of\npossible 4-grams is V4=7\u00021017. Thus, once the generator has chosen the \ufb01rst\n3-gram ( It cannot be ), there are only seven possible next words for the 4th element\n(but,I,that,thus,this, and the period).\nTo get an idea of the dependence on the training set, let\u2019s look at LMs trained on a\ncompletely different corpus: the Wall Street Journal (WSJ) newspaper. Shakespeare\nand the WSJ are both English, so we might have expected some overlap between our\nn-grams for the two genres. Fig. 3.5 shows sentences generated by unigram, bigram,\nand trigram grammars trained on 40 million words from WSJ.\n1Months the my and issue of year foreign new exchange\u2019s september\nwere recession exchange new endorsed a acquire to six executivesgram\n2Last December through the way to preserve the Hudson corporation N.\nB. E. C. Taylor would seem to complete the major central planners one\ngram point \ufb01ve percent of U. S. E. has already old M. X. corporation of living\non information such as more frequently \ufb01shing to keep her\n3They also point to ninety nine point six billion dollars from two hundred\nfour oh six three percent of the rates of interest stores as Mexico and\ngram Brazil on market conditions\nFigure 3.5 Three sentences randomly generated from three n-gram models computed from\n40 million words of the Wall Street Journal , lower-casing all characters and treating punctua-\ntion as words. Output was then hand-corrected for capitalization to improve readability.\nCompare these examples to the pseudo-Shakespeare in Fig. 3.4. While they both\nmodel \u201cEnglish-like sentences\u201d, there is no overlap in the generated sentences, and\nlittle overlap even in small phrases. Statistical models are pretty useless as predictors\nif the training sets and the test sets are as different as Shakespeare and the WSJ.\nHow should we deal with this problem when we build n-gram models? One step\nis to be sure to use a training corpus that has a similar genre to whatever task we are\ntrying to accomplish. To build a language model for translating legal documents,\nwe need a training corpus of legal documents. To build a language model for a\nquestion-answering system, we need a training corpus of questions.\nIt is equally important to get training data in the appropriate dialect orvariety ,\nespecially when processing social media posts or spoken transcripts. For exam-\nple some tweets will use features of African American English (AAE)\u2014 the name\nfor the many variations of language used in African American communities (King,\n2020). Such features can include words like \ufb01nna \u2014an auxiliary verb that marks\nimmediate future tense \u2014that don\u2019t occur in other varieties, or spellings like denfor\nthen, in tweets like this one (Blodgett and O\u2019Connor, 2017):\n(3.22) Bored af den my phone \ufb01nna die!!!\nwhile tweets from English-based languages like Nigerian Pidgin have markedly dif-\nferent vocabulary and n-gram patterns from American English (Jurgens et al., 2017):\n(3.23) @username R u a wizard or wat gan sef: in d mornin - u tweet, afternoon - u\ntweet, nyt gan u dey tweet. beta get ur IT placement wiv twitter\nIs it possible for the testset nonetheless to have a word we have never seen be-\nfore? What happens if the word Jurafsky never occurs in our training set, but pops\nup in the test set? The answer is that although words might be unseen, we actu-\nally run our NLP algorithms not on words but on subword tokens . With subword 14 CHAPTER 3 \u2022 N- GRAM LANGUAGE MODELS\ntokenization (like the BPE algorithm of Chapter 2) any word can be modeled as a\nsequence of known smaller subwords, if necessary by a sequence of individual let-\nters. So although for convenience we\u2019ve been referring to words in this chapter, the\nlanguage model vocabulary is actually the set of tokens rather than words, and the\ntest set can never contain unseen tokens.\n3.6 Smoothing, Interpolation, and Backoff\nThere is a problem with using maximum likelihood estimates for probabilities: any\n\ufb01nite training corpus will be missing some perfectly acceptable English word se-\nquences. That is, cases where a particular n-gram never occurs in the training data\nbut appears in the test set. Perhaps our training corpus has the words ruby and\nslippers in it but just happens not to have the phrase ruby slippers .\nThese unseen sequences or zeros \u2014sequences that don\u2019t occur in the training set zeros\nbut do occur in the test set\u2014are a problem for two reasons. First, their presence\nmeans we are underestimating the probability of word sequences that might occur,\nwhich hurts the performance of any application we want to run on this data. Second,\nif the probability of any word in the test set is 0, the probability of the whole test\nset is 0. Perplexity is de\ufb01ned based on the inverse probability of the test set. Thus\nif some words in context have zero probability, we can\u2019t compute perplexity at all,\nsince we can\u2019t divide by 0!\nThe standard way to deal with putative \u201czero probability n-grams\u201d that should re-\nally have some non-zero probability is called smoothing ordiscounting . Smoothing smoothing\ndiscounting algorithms shave off a bit of probability mass from some more frequent events and\ngive it to unseen events. Here we\u2019ll introduce some simple smoothing algorithms:\nLaplace (add-one) smoothing ,stupid backoff , and n-gram interpolation .\n3.6.1 Laplace Smoothing\nThe simplest way to do smoothing is to add one to all the n-gram counts, before\nwe normalize them into probabilities. All the counts that used to be zero will now\nhave a count of 1, the counts of 1 will be 2, and so on. This algorithm is called\nLaplace smoothing . Laplace smoothing does not perform well enough to be usedLaplace\nsmoothing\nin modern n-gram models, but it usefully introduces many of the concepts that we\nsee in other smoothing algorithms, gives a useful baseline, and is also a practical\nsmoothing algorithm for other tasks like text classi\ufb01cation (Chapter 4).\nLet\u2019s start with the application of Laplace smoothing to unigram probabilities.\nRecall that the unsmoothed maximum likelihood estimate of the unigram probability\nof the word wiis its count cinormalized by the total number of word tokens N:\nP(wi) =ci\nN\nLaplace smoothing merely adds one to each count (hence its alternate name add-\nonesmoothing). Since there are Vwords in the vocabulary and each one was in- add-one\ncremented, we also need to adjust the denominator to take into account the extra V\nobservations. (What happens to our Pvalues if we don\u2019t increase the denominator?)\nPLaplace (wi) =ci+1\nN+V(3.24) 3.6 \u2022 S MOOTHING , INTERPOLATION ,AND BACKOFF 15\nInstead of changing both the numerator and denominator, it is convenient to describe\nhow a smoothing algorithm affects the numerator, by de\ufb01ning an adjusted count c\u0003.\nThis adjusted count is easier to compare directly with the MLE counts and can be\nturned into a probability like an MLE count by normalizing by N. To de\ufb01ne this\ncount, since we are only changing the numerator in addition to adding 1 we\u2019ll also\nneed to multiply by a normalization factorN\nN+V:\nc\u0003\ni= (ci+1)N\nN+V(3.25)\nWe can now turn c\u0003\niinto a probability P\u0003\niby normalizing by N.\nA related way to view smoothing is as discounting (lowering) some non-zero discounting\ncounts in order to get the probability mass that will be assigned to the zero counts.\nThus, instead of referring to the discounted counts c\u0003, we might describe a smooth-\ning algorithm in terms of a relative discount di, the ratio of the discounted counts discount\nto the original counts:\ndi=c\u0003\ni\nci\nNow that we have the intuition for the unigram case, let\u2019s smooth our Berkeley\nRestaurant Project bigrams. Figure 3.6 shows the add-one smoothed counts for the\nbigrams in Fig. 3.1.\ni want to eat chinese food lunch spend\ni 6 828 1 10 1 1 1 3\nwant 3 1 609 2 7 7 6 2\nto 3 1 5 687 3 1 7 212\neat 1 1 3 1 17 3 43 1\nchinese 2 1 1 1 1 83 2 1\nfood 16 1 16 1 2 5 1 1\nlunch 3 1 1 1 1 2 1 1\nspend 2 1 2 1 1 1 1 1\nFigure 3.6 Add-one smoothed bigram counts for eight of the words (out of V=1446) in\nthe Berkeley Restaurant Project corpus of 9332 sentences. Previously-zero counts are in gray.\nFigure 3.7 shows the add-one smoothed probabilities for the bigrams in Fig. 3.2.\nRecall that normal bigram probabilities are computed by normalizing each row of\ncounts by the unigram count:\nPMLE(wnjwn\u00001) =C(wn\u00001wn)\nC(wn\u00001)(3.26)\nFor add-one smoothed bigram counts, we need to augment the unigram count by the\nnumber of total word types in the vocabulary V:\nPLaplace (wnjwn\u00001) =C(wn\u00001wn)+1P\nw(C(wn\u00001w)+1)=C(wn\u00001wn)+1\nC(wn\u00001)+V(3.27)\nThus, each of the unigram counts given in the previous section will need to be aug-\nmented by V=1446. The result is the smoothed bigram probabilities in Fig. 3.7.\nIt is often convenient to reconstruct the count matrix so we can see how much a\nsmoothing algorithm has changed the original counts. These adjusted counts can be 16 CHAPTER 3 \u2022 N- GRAM LANGUAGE MODELS\ni want to eat chinese food lunch spend\ni 0.0015 0.21 0.00025 0.0025 0.00025 0.00025 0.00025 0.00075\nwant 0.0013 0.00042 0.26 0.00084 0.0029 0.0029 0.0025 0.00084\nto 0.00078 0.00026 0.0013 0.18 0.00078 0.00026 0.0018 0.055\neat 0.00046 0.00046 0.0014 0.00046 0.0078 0.0014 0.02 0.00046\nchinese 0.0012 0.00062 0.00062 0.00062 0.00062 0.052 0.0012 0.00062\nfood 0.0063 0.00039 0.0063 0.00039 0.00079 0.002 0.00039 0.00039\nlunch 0.0017 0.00056 0.00056 0.00056 0.00056 0.0011 0.00056 0.00056\nspend 0.0012 0.00058 0.0012 0.00058 0.00058 0.00058 0.00058 0.00058\nFigure 3.7 Add-one smoothed bigram probabilities for eight of the words (out of V=1446) in the BeRP\ncorpus of 9332 sentences. Previously-zero probabilities are in gray.\ni want to eat chinese food lunch spend\ni 3.8 527 0.64 6.4 0.64 0.64 0.64 1.9\nwant 1.2 0.39 238 0.78 2.7 2.7 2.3 0.78\nto 1.9 0.63 3.1 430 1.9 0.63 4.4 133\neat 0.34 0.34 1 0.34 5.8 1 15 0.34\nchinese 0.2 0.098 0.098 0.098 0.098 8.2 0.2 0.098\nfood 6.9 0.43 6.9 0.43 0.86 2.2 0.43 0.43\nlunch 0.57 0.19 0.19 0.19 0.19 0.38 0.19 0.19\nspend 0.32 0.16 0.32 0.16 0.16 0.16 0.16 0.16\nFigure 3.8 Add-one reconstituted counts for eight words (of V=1446) in the BeRP corpus\nof 9332 sentences. Previously-zero counts are in gray.\ncomputed by Eq. 3.28. Figure 3.8 shows the reconstructed counts.\nc\u0003(wn\u00001wn) =[C(wn\u00001wn)+1]\u0002C(wn\u00001)\nC(wn\u00001)+V(3.28)\nNote that add-one smoothing has made a very big change to the counts. Com-\nparing Fig. 3.8 to the original counts in Fig. 3.1, we can see that C(want to )changed\nfrom 608 to 238! We can see this in probability space as well: P(tojwant)decreases\nfrom 0.66 in the unsmoothed case to 0.26 in the smoothed case. Looking at the dis-\ncount d(the ratio between new and old counts) shows us how strikingly the counts\nfor each pre\ufb01x word have been reduced; the discount for the bigram want to is 0.39,\nwhile the discount for Chinese food is 0.10, a factor of 10! The sharp change occurs\nbecause too much probability mass is moved to all the zeros.\n3.6.2 Add-k smoothing\nOne alternative to add-one smoothing is to move a bit less of the probability mass\nfrom the seen to the unseen events. Instead of adding 1 to each count, we add a\nfractional count k(0.5? 0.01?). This algorithm is therefore called add-k smoothing . add-k\nP\u0003\nAdd-k(wnjwn\u00001) =C(wn\u00001wn)+k\nC(wn\u00001)+kV(3.29)\nAdd-k smoothing requires that we have a method for choosing k; this can be\ndone, for example, by optimizing on a devset . Although add-k is useful for some\ntasks (including text classi\ufb01cation), it turns out that it still doesn\u2019t work well for\nlanguage modeling, generating counts with poor variances and often inappropriate\ndiscounts (Gale and Church, 1994). 3.6 \u2022 S MOOTHING , INTERPOLATION ,AND BACKOFF 17\n3.6.3 Language Model Interpolation\nThere is an alternative source of knowledge we can draw on to solve the problem\nof zero frequency n-grams. If we are trying to compute P(wnjwn\u00002wn\u00001)but we\nhave no examples of a particular trigram wn\u00002wn\u00001wn, we can instead estimate its\nprobability by using the bigram probability P(wnjwn\u00001). Similarly, if we don\u2019t have\ncounts to compute P(wnjwn\u00001), we can look to the unigram P(wn). In other words,\nsometimes using less context can help us generalize more for contexts that the model\nhasn\u2019t learned much about.\nThe most common way to use this n-gram hierarchy is called interpolation : interpolation\ncomputing a new probability by interpolating (weighting and combining) the tri-\ngram, bigram, and unigram probabilities.3In simple linear interpolation, we com-\nbine different order n-grams by linearly interpolating them. Thus, we estimate the\ntrigram probability P(wnjwn\u00002wn\u00001)by mixing together the unigram, bigram, and\ntrigram probabilities, each weighted by a l:\n\u02c6P(wnjwn\u00002wn\u00001) = l1P(wn)\n+l2P(wnjwn\u00001)\n+l3P(wnjwn\u00002wn\u00001) (3.30)\nThe ls must sum to 1, making Eq. 3.30 equivalent to a weighted average. In a\nslightly more sophisticated version of linear interpolation, each lweight is com-\nputed by conditioning on the context. This way, if we have particularly accurate\ncounts for a particular bigram, we assume that the counts of the trigrams based on\nthis bigram will be more trustworthy, so we can make the ls for those trigrams\nhigher and thus give that trigram more weight in the interpolation. Equation 3.31\nshows the equation for interpolation with context-conditioned weights, where each\nlambda takes an argument that is the two prior word context:\n\u02c6P(wnjwn\u00002wn\u00001) = l1(wn\u00002:n\u00001)P(wn)\n+l2(wn\u00002:n\u00001)P(wnjwn\u00001)\n+l3(wn\u00002:n\u00001)P(wnjwn\u00002wn\u00001) (3.31)\nHow are these lvalues set? Both the simple interpolation and conditional interpo-\nlation ls are learned from a held-out corpus. A held-out corpus is an additional held-out\ntraining corpus, so-called because we hold it out from the training data, that we use\nto set these lvalues.4We do so by choosing the lvalues that maximize the likeli-\nhood of the held-out corpus. That is, we \ufb01x the n-gram probabilities and then search\nfor the lvalues that\u2014when plugged into Eq. 3.30\u2014give us the highest probability\nof the held-out set. There are various ways to \ufb01nd this optimal set of ls. One way\nis to use the EMalgorithm, an iterative learning algorithm that converges on locally\noptimal ls (Jelinek and Mercer, 1980).\n3.6.4 Stupid Backoff\nAn alternative to interpolation is backoff . In a backoff model, if the n-gram we need backoff\n3We won\u2019t discuss the less-common alternative, called backoff , in which we use the trigram if the\nevidence is suf\ufb01cient for it, but if not we instead just use the bigram, otherwise the unigram. That is, we\nonly \u201cback off\u201d to a lower-order n-gram if we have zero evidence for a higher-order n-gram.\n4Held-out corpora are generally used to set hyperparameters , which are special parameters, unlike\nregular counts that are learned from the training data; we\u2019ll discuss hyperparameters in Chapter 7. 18 CHAPTER 3 \u2022 N- GRAM LANGUAGE MODELS\nhas zero counts, we approximate it by backing off to the (n-1)-gram. We continue\nbacking off until we reach a history that has some counts. For a backoff model to\ngive a correct probability distribution, we have to discount the higher-order n-grams discount\nto save some probability mass for the lower order n-grams. In practice, instead of\ndiscounting, it\u2019s common to use a much simpler non-discounted backoff algorithm\ncalled stupid backoff (Brants et al., 2007). stupid backoff\nStupid backoff gives up the idea of trying to make the language model a true\nprobability distribution. There is no discounting of the higher-order probabilities. If\na higher-order n-gram has a zero count, we simply backoff to a lower order n-gram,\nweighed by a \ufb01xed (context-independent) weight. This algorithm does not produce\na probability distribution, so we\u2019ll follow Brants et al. (2007) in referring to it as S:\nS(wijwi\u0000N+1:i\u00001) =8\n<\n:count (wi\u0000N+1:i)\ncount (wi\u0000N+1:i\u00001)if count (wi\u0000N+1:i)>0\nlS(wijwi\u0000N+2:i\u00001)otherwise(3.32)\nThe backoff terminates in the unigram, which has score S(w) =count (w)\nN. Brants et al.\n(2007) \ufb01nd that a value of 0.4 worked well for l.\n3.7 Advanced: Perplexity\u2019s Relation to Entropy\nWe introduced perplexity in Section 3.3 as a way to evaluate n-gram models on\na test set. A better n-gram model is one that assigns a higher probability to the\ntest data, and perplexity is a normalized version of the probability of the test set.\nThe perplexity measure actually arises from the information-theoretic concept of\ncross-entropy, which explains otherwise mysterious properties of perplexity (why\nthe inverse probability, for example?) and its relationship to entropy. Entropy is a Entropy\nmeasure of information. Given a random variable Xranging over whatever we are\npredicting (words, letters, parts of speech), the set of which we\u2019ll call c, and with a\nparticular probability function, call it p(x), the entropy of the random variable Xis:\nH(X) =\u0000X\nx2cp(x)log2p(x) (3.33)\nThe log can, in principle, be computed in any base. If we use log base 2, the\nresulting value of entropy will be measured in bits.\nOne intuitive way to think about entropy is as a lower bound on the number of\nbits it would take to encode a certain decision or piece of information in the optimal\ncoding scheme. Consider an example from the standard information theory textbook\nCover and Thomas (1991). Imagine that we want to place a bet on a horse race but\nit is too far to go all the way to Yonkers Racetrack, so we\u2019d like to send a short\nmessage to the bookie to tell him which of the eight horses to bet on. One way to\nencode this message is just to use the binary representation of the horse\u2019s number\nas the code; thus, horse 1 would be 001, horse 2 010, horse 3 011, and so on, with\nhorse 8 coded as 000. If we spend the whole day betting and each horse is coded\nwith 3 bits, on average we would be sending 3 bits per race.\nCan we do better? Suppose that the spread is the actual distribution of the bets\nplaced and that we represent it as the prior probability of each horse as follows: 3.7 \u2022 A DVANCED : PERPLEXITY \u2019SRELATION TO ENTROPY 19\nHorse 11\n2Horse 51\n64\nHorse 21\n4Horse 61\n64\nHorse 31\n8Horse 71\n64\nHorse 41\n16Horse 81\n64\nThe entropy of the random variable Xthat ranges over horses gives us a lower\nbound on the number of bits and is\nH(X) =\u0000i=8X\ni=1p(i)log2p(i)\n=\u00001\n2log21\n2\u00001\n4log21\n4\u00001\n8log21\n8\u00001\n16log21\n16\u00004(1\n64log21\n64)\n=2 bits (3.34)\nA code that averages 2 bits per race can be built with short encodings for more\nprobable horses, and longer encodings for less probable horses. For example, we\ncould encode the most likely horse with the code 0, and the remaining horses as 10,\nthen 110,1110 ,111100 ,111101 ,111110 , and 111111 .\nWhat if the horses are equally likely? We saw above that if we used an equal-\nlength binary code for the horse numbers, each horse took 3 bits to code, so the\naverage was 3. Is the entropy the same? In this case each horse would have a\nprobability of1\n8. The entropy of the choice of horses is then\nH(X) =\u0000i=8X\ni=11\n8log21\n8=\u0000log21\n8=3 bits (3.35)\nUntil now we have been computing the entropy of a single variable. But most of\nwhat we will use entropy for involves sequences . For a grammar, for example, we\nwill be computing the entropy of some sequence of words W=fw1;w2;:::; wng.\nOne way to do this is to have a variable that ranges over sequences of words. For\nexample we can compute the entropy of a random variable that ranges over all se-\nquences of words of length nin some language Las follows:\nH(w1;w2;:::; wn) =\u0000X\nw1:n2Lp(w1:n)logp(w1:n) (3.36)\nWe could de\ufb01ne the entropy rate (we could also think of this as the per-word entropy rate\nentropy ) as the entropy of this sequence divided by the number of words:\n1\nnH(w1:n) =\u00001\nnX\nw1:n2Lp(w1:n)logp(w1:n) (3.37)\nBut to measure the true entropy of a language, we need to consider sequences of\nin\ufb01nite length. If we think of a language as a stochastic process Lthat produces a\nsequence of words, and allow Wto represent the sequence of words w1;:::; wn, then\nL\u2019s entropy rate H(L)is de\ufb01ned as\nH(L) = lim\nn!\u00a51\nnH(w1:n)\n=\u0000lim\nn!\u00a51\nnX\nW2Lp(w1:n)logp(w1:n) (3.38) 20 CHAPTER 3 \u2022 N- GRAM LANGUAGE MODELS\nThe Shannon-McMillan-Breiman theorem (Algoet and Cover 1988, Cover and Thomas\n1991) states that if the language is regular in certain ways (to be exact, if it is both\nstationary and ergodic),\nH(L) =lim\nn!\u00a5\u00001\nnlogp(w1:n) (3.39)\nThat is, we can take a single sequence that is long enough instead of summing over\nall possible sequences. The intuition of the Shannon-McMillan-Breiman theorem\nis that a long-enough sequence of words will contain in it many other shorter se-\nquences and that each of these shorter sequences will reoccur in the longer sequence\naccording to their probabilities.\nA stochastic process is said to be stationary if the probabilities it assigns to a Stationary\nsequence are invariant with respect to shifts in the time index. In other words, the\nprobability distribution for words at time tis the same as the probability distribution\nat time t+1. Markov models, and hence n-grams, are stationary. For example, in\na bigram, Piis dependent only on Pi\u00001. So if we shift our time index by x,Pi+xis\nstill dependent on Pi+x\u00001. But natural language is not stationary, since as we show\nin Appendix D, the probability of upcoming words can be dependent on events that\nwere arbitrarily distant and time dependent. Thus, our statistical models only give\nan approximation to the correct distributions and entropies of natural language.\nTo summarize, by making some incorrect but convenient simplifying assump-\ntions, we can compute the entropy of some stochastic process by taking a very long\nsample of the output and computing its average log probability.\nNow we are ready to introduce cross-entropy . The cross-entropy is useful when cross-entropy\nwe don\u2019t know the actual probability distribution pthat generated some data. It\nallows us to use some m, which is a model of p(i.e., an approximation to p). The\ncross-entropy of monpis de\ufb01ned by\nH(p;m) =lim\nn!\u00a5\u00001\nnX\nW2Lp(w1;:::; wn)logm(w1;:::; wn) (3.40)\nThat is, we draw sequences according to the probability distribution p, but sum the\nlog of their probabilities according to m.\nAgain, following the Shannon-McMillan-Breiman theorem, for a stationary er-\ngodic process:\nH(p;m) =lim\nn!\u00a5\u00001\nnlogm(w1w2:::wn) (3.41)\nThis means that, as for entropy, we can estimate the cross-entropy of a model m\non some distribution pby taking a single sequence that is long enough instead of\nsumming over all possible sequences.\nWhat makes the cross-entropy useful is that the cross-entropy H(p;m)is an up-\nper bound on the entropy H(p). For any model m:\nH(p)\u0014H(p;m) (3.42)\nThis means that we can use some simpli\ufb01ed model mto help estimate the true en-\ntropy of a sequence of symbols drawn according to probability p. The more accurate\nmis, the closer the cross-entropy H(p;m)will be to the true entropy H(p). Thus,\nthe difference between H(p;m)andH(p)is a measure of how accurate a model is.\nBetween two models m1andm2, the more accurate model will be the one with the 3.8 \u2022 S UMMARY 21\nlower cross-entropy. (The cross-entropy can never be lower than the true entropy, so\na model cannot err by underestimating the true entropy.)\nWe are \ufb01nally ready to see the relation between perplexity and cross-entropy\nas we saw it in Eq. 3.41. Cross-entropy is de\ufb01ned in the limit as the length of the\nobserved word sequence goes to in\ufb01nity. We approximate this cross-entropy by\nrelying on a (suf\ufb01ciently long) sequence of \ufb01xed length. This approximation to the\ncross-entropy of a model M=P(wijwi\u0000N+1:i\u00001)on a sequence of words Wis\nH(W) =\u00001\nNlogP(w1w2:::wN) (3.43)\nTheperplexity of a model Pon a sequence of words Wis now formally de\ufb01ned as perplexity\n2 raised to the power of this cross-entropy:\nPerplexity (W) = 2H(W)\n=P(w1w2:::wN)\u00001\nN\n=Ns\n1\nP(w1w2:::wN)\n3.8 Summary\nThis chapter introduced language modeling via the n-gram model, a classic model\nthat allows us to introduce many of the basic concepts in language modeling.\n\u2022 Language models offer a way to assign a probability to a sentence or other\nsequence of words or tokens, and to predict a word or token from preceding\nwords or tokens.\n\u2022N-grams are perhaps the simplest kind of language model. They are Markov\nmodels that estimate words from a \ufb01xed window of previous words. N-gram\nmodels can be trained by counting in a training corpus and normalizing the\ncounts (the maximum likelihood estimate ).\n\u2022 N-gram language models can be evaluated on a test set using perplexity.\n\u2022 The perplexity of a test set according to a language model is a function of\nthe probability of the test set: the inverse test set probability according to the\nmodel, normalized by the length.\n\u2022Sampling from a language model means to generate some sentences, choos-\ning each sentence according to its likelihood as de\ufb01ned by the model.\n\u2022Smoothing algorithms provide a way to estimate probabilities for events that\nwere unseen in training. Commonly used smoothing algorithms for n-grams\ninclude add-1 smoothing, or rely on lower-order n-gram counts through inter-\npolation .\nBibliographical and Historical Notes\nThe underlying mathematics of the n-gram was \ufb01rst proposed by Markov (1913),\nwho used what are now called Markov chains (bigrams and trigrams) to predict\nwhether an upcoming letter in Pushkin\u2019s Eugene Onegin would be a vowel or a con-\nsonant. Markov classi\ufb01ed 20,000 letters as V or C and computed the bigram and 22 CHAPTER 3 \u2022 N- GRAM LANGUAGE MODELS\ntrigram probability that a given letter would be a vowel given the previous one or\ntwo letters. Shannon (1948) applied n-grams to compute approximations to English\nword sequences. Based on Shannon\u2019s work, Markov models were commonly used in\nengineering, linguistic, and psychological work on modeling word sequences by the\n1950s. In a series of extremely in\ufb02uential papers starting with Chomsky (1956) and\nincluding Chomsky (1957) and Miller and Chomsky (1963), Noam Chomsky argued\nthat \u201c\ufb01nite-state Markov processes\u201d, while a possibly useful engineering heuristic,\nwere incapable of being a complete cognitive model of human grammatical knowl-\nedge. These arguments led many linguists and computational linguists to ignore\nwork in statistical modeling for decades.\nThe resurgence of n-gram language models came from Fred Jelinek and col-\nleagues at the IBM Thomas J. Watson Research Center, who were in\ufb02uenced by\nShannon, and James Baker at CMU, who was in\ufb02uenced by the prior, classi\ufb01ed\nwork of Leonard Baum and colleagues on these topics at labs like the US Institute\nfor Defense SAnalyses (IDA) after they were declassi\ufb01ed. Independently these two\nlabs successfully used n-grams in their speech recognition systems at the same time\n(Baker 1975b, Jelinek et al. 1975, Baker 1975a, Bahl et al. 1983, Jelinek 1990). The\nterms \u201clanguage model\u201d and \u201cperplexity\u201d were \ufb01rst used for this technology by the\nIBM group. Jelinek and his colleagues used the term language model in a pretty\nmodern way, to mean the entire set of linguistic in\ufb02uences on word sequence prob-\nabilities, including grammar, semantics, discourse, and even speaker characteristics,\nrather than just the particular n-gram model itself.\nAdd-one smoothing derives from Laplace\u2019s 1812 law of succession and was \ufb01rst\napplied as an engineering solution to the zero frequency problem by Jeffreys (1948)\nbased on an earlier Add-K suggestion by Johnson (1932). Problems with the add-\none algorithm are summarized in Gale and Church (1994).\nA wide variety of different language modeling and smoothing techniques were\nproposed in the 80s and 90s, including Good-Turing discounting\u2014\ufb01rst applied to the\nn-gram smoothing at IBM by Katz (N \u00b4adas 1984, Church and Gale 1991)\u2014 Witten-\nBell discounting (Witten and Bell, 1991), and varieties of class-based n-gram mod-class-based\nn-gram\nels that used information about word classes. Starting in the late 1990s, Chen and\nGoodman performed a number of carefully controlled experiments comparing dif-\nferent algorithms and parameters (Chen and Goodman 1999, Goodman 2006, inter\nalia). They showed the advantages of Modi\ufb01ed Interpolated Kneser-Ney , which\nbecame the standard baseline for n-gram language modeling around the turn of the\ncentury, especially because they showed that caches and class-based models pro-\nvided only minor additional improvement. SRILM (Stolcke, 2002) and KenLM\n(Hea\ufb01eld 2011, Hea\ufb01eld et al. 2013) are publicly available toolkits for building n-\ngram language models.\nLarge language models are based on neural networks rather than n-grams, en-\nabling them to solve the two major problems with n-grams: (1) the number of param-\neters increases exponentially as the n-gram order increases, and (2) n-grams have no\nway to generalize from training examples to test set examples unless they use iden-\ntical words. Neural language models instead project words into a continuous space\nin which words with similar contexts have similar representations. We\u2019ll introduce\ntransformer-based large language models in Chapter 9, along the way introducing\nfeedforward language models (Bengio et al. 2006, Schwenk 2007) in Chapter 7 and\nrecurrent language models (Mikolov, 2012) in Chapter 8. EXERCISES 23\nExercises\n3.1 Write out the equation for trigram probability estimation (modifying Eq. 3.11).\nNow write out all the non-zero trigram probabilities for the I am Sam corpus\non page 4.\n3.2 Calculate the probability of the sentence i want chinese food . Give two\nprobabilities, one using Fig. 3.2 and the \u2018useful probabilities\u2019 just below it on\npage 6, and another using the add-1 smoothed table in Fig. 3.7. Assume the\nadditional add-1 smoothed probabilities P(i|<s> ) =0:19 and P(</s>|food ) =\n0:40.\n3.3 Which of the two probabilities you computed in the previous exercise is higher,\nunsmoothed or smoothed? Explain why.\n3.4 We are given the following corpus, modi\ufb01ed from the one in the chapter:\n<s> I am Sam </s>\n<s> Sam I am </s>\n<s> I am Sam </s>\n<s> I do not like green eggs and Sam </s>\nUsing a bigram language model with add-one smoothing, what is P(Sam j\nam)? Include <s>and</s> in your counts just like any other token.\n3.5 Suppose we didn\u2019t use the end-symbol </s> . Train an unsmoothed bigram\ngrammar on the following training corpus without using the end-symbol </s> :\n<s> a b\n<s> b b\n<s> b a\n<s> a a\nDemonstrate that your bigram model does not assign a single probability dis-\ntribution across all sentence lengths by showing that the sum of the probability\nof the four possible 2 word sentences over the alphabet fa,bgis 1.0, and the\nsum of the probability of all possible 3 word sentences over the alphabet fa,bg\nis also 1.0.\n3.6 Suppose we train a trigram language model with add-one smoothing on a\ngiven corpus. The corpus contains V word types. Express a formula for esti-\nmating P(w3jw1,w2), where w3 is a word which follows the bigram (w1,w2),\nin terms of various n-gram counts and V . Use the notation c(w1,w2,w3) to\ndenote the number of times that trigram (w1,w2,w3) occurs in the corpus, and\nso on for bigrams and unigrams.\n3.7 We are given the following corpus, modi\ufb01ed from the one in the chapter:\n<s> I am Sam </s>\n<s> Sam I am </s>\n<s> I am Sam </s>\n<s> I do not like green eggs and Sam </s>\nIf we use linear interpolation smoothing between a maximum-likelihood bi-\ngram model and a maximum-likelihood unigram model with l1=1\n2andl2=\n1\n2, what is P(Samjam)? Include <s> and</s> in your counts just like any\nother token.\n3.8 Write a program to compute unsmoothed unigrams and bigrams. 24 CHAPTER 3 \u2022 N- GRAM LANGUAGE MODELS\n3.9 Run your n-gram program on two different small corpora of your choice (you\nmight use email text or newsgroups). Now compare the statistics of the two\ncorpora. What are the differences in the most common unigrams between the\ntwo? How about interesting differences in bigrams?\n3.10 Add an option to your program to generate random sentences.\n3.11 Add an option to your program to compute the perplexity of a test set.\n3.12 You are given a training set of 100 numbers that consists of 91 zeros and 1\neach of the other digits 1-9. Now we see the following test set: 0 0 0 0 0 3 0 0\n0 0. What is the unigram perplexity? Exercises 25\nAlgoet, P. H. and T. M. Cover. 1988. A sandwich proof of\nthe Shannon-McMillan-Breiman theorem. The Annals of\nProbability , 16(2):899\u2013909.\nBahl, L. R., F. Jelinek, and R. L. Mercer. 1983. A maxi-\nmum likelihood approach to continuous speech recogni-\ntion. IEEE Transactions on Pattern Analysis and Machine\nIntelligence , 5(2):179\u2013190.\nBaker, J. K. 1975a. The DRAGON system \u2013 An overview.\nIEEE Transactions on ASSP , ASSP-23(1):24\u201329.\nBaker, J. K. 1975b. Stochastic modeling for automatic\nspeech understanding. In D. R. Reddy, ed., Speech Recog-\nnition . Academic Press.\nBengio, Y ., H. Schwenk, J.-S. Sen \u00b4ecal, F. Morin, and J.-L.\nGauvain. 2006. Neural probabilistic language models. In\nInnovations in Machine Learning , 137\u2013186. Springer.\nBlodgett, S. L. and B. O\u2019Connor. 2017. Racial disparity in\nnatural language processing: A case study of social media\nAfrican-American English. FAT/ML Workshop, KDD .\nBrants, T., A. C. Popat, P. Xu, F. J. Och, and J. Dean.\n2007. Large language models in machine translation.\nEMNLP/CoNLL .\nChen, S. F. and J. Goodman. 1999. An empirical study of\nsmoothing techniques for language modeling. Computer\nSpeech and Language , 13:359\u2013394.\nChomsky, N. 1956. Three models for the description of\nlanguage. IRE Transactions on Information Theory ,\n2(3):113\u2013124.\nChomsky, N. 1957. Syntactic Structures . Mouton.\nChurch, K. W. and W. A. Gale. 1991. A comparison of the\nenhanced Good-Turing and deleted estimation methods\nfor estimating probabilities of English bigrams. Com-\nputer Speech and Language , 5:19\u201354.\nCover, T. M. and J. A. Thomas. 1991. Elements of Informa-\ntion Theory . Wiley.\nDavies, M. 2020. The Corpus of Contemporary Amer-\nican English (COCA): One billion words, 1990-2019.\nhttps://www.english-corpora.org/coca/ .\nFranz, A. and T. Brants. 2006. All our n-gram are\nbelong to you. https://research.google/blog/\nall-our-n-gram-are-belong-to-you/ .\nGale, W. A. and K. W. Church. 1994. What is wrong with\nadding one? In N. Oostdijk and P. de Haan, eds, Corpus-\nBased Research into Language , 189\u2013198. Rodopi.\nGoodman, J. 2006. A bit of progress in language model-\ning: Extended version. Technical Report MSR-TR-2001-\n72, Machine Learning and Applied Statistics Group, Mi-\ncrosoft Research, Redmond, WA.\nHea\ufb01eld, K. 2011. KenLM: Faster and smaller language\nmodel queries. Workshop on Statistical Machine Trans-\nlation .\nHea\ufb01eld, K., I. Pouzyrevsky, J. H. Clark, and P. Koehn. 2013.\nScalable modi\ufb01ed Kneser-Ney language model estima-\ntion. ACL.\nJeffreys, H. 1948. Theory of Probability , 2nd edition. Claren-\ndon Press. Section 3.23.\nJelinek, F. 1990. Self-organized language modeling for\nspeech recognition. In A. Waibel and K.-F. Lee, eds,\nReadings in Speech Recognition , 450\u2013506. Morgan Kauf-\nmann. Originally distributed as IBM technical report in\n1985.Jelinek, F. and R. L. Mercer. 1980. Interpolated estimation\nof Markov source parameters from sparse data. In E. S.\nGelsema and L. N. Kanal, eds, Proceedings, Workshop\non Pattern Recognition in Practice , 381\u2013397. North Hol-\nland.\nJelinek, F., R. L. Mercer, and L. R. Bahl. 1975. Design of a\nlinguistic statistical decoder for the recognition of contin-\nuous speech. IEEE Transactions on Information Theory ,\nIT-21(3):250\u2013256.\nJohnson, W. E. 1932. Probability: deductive and inductive\nproblems (appendix to). Mind , 41(164):421\u2013423.\nJurafsky, D., C. Wooters, G. Tajchman, J. Segal, A. Stolcke,\nE. Fosler, and N. Morgan. 1994. The Berkeley restaurant\nproject. ICSLP .\nJurgens, D., Y . Tsvetkov, and D. Jurafsky. 2017. Incorpo-\nrating dialectal variability for socially equitable language\nidenti\ufb01cation. ACL.\nKane, S. K., M. R. Morris, A. Paradiso, and J. Campbell.\n2017. \u201cat times avuncular and cantankerous, with the\nre\ufb02exes of a mongoose\u201d: Understanding self-expression\nthrough augmentative and alternative communication de-\nvices. CSCW .\nKing, S. 2020. From African American Vernacular English\nto African American Language: Rethinking the study of\nrace and language in African Americans\u2019 speech. Annual\nReview of Linguistics , 6:285\u2013300.\nLin, Y ., J.-B. Michel, E. Aiden Lieberman, J. Orwant,\nW. Brockman, and S. Petrov. 2012. Syntactic annotations\nfor the Google books NGram corpus. ACL.\nLiu, J., S. Min, L. Zettlemoyer, Y . Choi, and H. Hajishirzi.\n2024. In\ufb01ni-gram: Scaling unbounded n-gram language\nmodels to a trillion tokens. ArXiv preprint.\nMarkov, A. A. 1913. Essai d\u2019une recherche statistique sur\nle texte du roman \u201cEugene Onegin\u201d illustrant la liaison\ndes epreuve en chain (\u2018Example of a statistical investiga-\ntion of the text of \u201cEugene Onegin\u201d illustrating the de-\npendence between samples in chain\u2019). Izvistia Impera-\ntorskoi Akademii Nauk (Bulletin de l\u2019Acad \u00b4emie Imp \u00b4eriale\ndes Sciences de St.-P \u00b4etersbourg) , 7:153\u2013162.\nMikolov, T. 2012. Statistical language models based on neu-\nral networks . Ph.D. thesis, Brno University of Technol-\nogy.\nMiller, G. A. and N. Chomsky. 1963. Finitary models of lan-\nguage users. In R. D. Luce, R. R. Bush, and E. Galanter,\neds, Handbook of Mathematical Psychology , volume II,\n419\u2013491. John Wiley.\nMiller, G. A. and J. A. Selfridge. 1950. Verbal context and\nthe recall of meaningful material. American Journal of\nPsychology , 63:176\u2013185.\nN\u00b4adas, A. 1984. Estimation of probabilities in the language\nmodel of the IBM speech recognition system. IEEE\nTransactions on ASSP , 32(4):859\u2013861.\nSchwenk, H. 2007. Continuous space language models.\nComputer Speech & Language , 21(3):492\u2013518.\nShannon, C. E. 1948. A mathematical theory of commu-\nnication. Bell System Technical Journal , 27(3):379\u2013423.\nContinued in the following volume.\nStolcke, A. 1998. Entropy-based pruning of backoff lan-\nguage models. Proc. DARPA Broadcast News Transcrip-\ntion and Understanding Workshop . 26 Chapter 3 \u2022 N-gram Language Models\nStolcke, A. 2002. SRILM \u2013 an extensible language modeling\ntoolkit. ICSLP .\nTrnka, K., D. Yarrington, J. McCaw, K. F. McCoy, and\nC. Pennington. 2007. The effects of word prediction on\ncommunication rate for AAC. NAACL-HLT .\nWitten, I. H. and T. C. Bell. 1991. The zero-frequency prob-\nlem: Estimating the probabilities of novel events in adap-\ntive text compression. IEEE Transactions on Information\nTheory , 37(4):1085\u20131094.", "4.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\n4Naive Bayes, Text Classi\ufb01ca-\ntion, and Sentiment\nClassi\ufb01cation lies at the heart of both human and machine intelligence. Deciding\nwhat letter, word, or image has been presented to our senses, recognizing faces\nor voices, sorting mail, assigning grades to homeworks; these are all examples of\nassigning a category to an input. The potential challenges of this task are highlighted\nby the fabulist Jorge Luis Borges (1964), who imagined classifying animals into:\n(a) those that belong to the Emperor, (b) embalmed ones, (c) those that\nare trained, (d) suckling pigs, (e) mermaids, (f) fabulous ones, (g) stray\ndogs, (h) those that are included in this classi\ufb01cation, (i) those that\ntremble as if they were mad, (j) innumerable ones, (k) those drawn with\na very \ufb01ne camel\u2019s hair brush, (l) others, (m) those that have just broken\na \ufb02ower vase, (n) those that resemble \ufb02ies from a distance.\nMany language processing tasks involve classi\ufb01cation, although luckily our classes\nare much easier to de\ufb01ne than those of Borges. In this chapter we introduce the naive\nBayes algorithm and apply it to text categorization , the task of assigning a label ortext\ncategorization\ncategory to an entire text or document.\nWe focus on one common text categorization task, sentiment analysis , the ex-sentiment\nanalysis\ntraction of sentiment , the positive or negative orientation that a writer expresses\ntoward some object. A review of a movie, book, or product on the web expresses the\nauthor\u2019s sentiment toward the product, while an editorial or political text expresses\nsentiment toward a candidate or political action. Extracting consumer or public sen-\ntiment is thus relevant for \ufb01elds from marketing to politics.\nThe simplest version of sentiment analysis is a binary classi\ufb01cation task, and\nthe words of the review provide excellent cues. Consider, for example, the follow-\ning phrases extracted from positive and negative reviews of movies and restaurants.\nWords like great ,richly ,awesome , and pathetic , and awful andridiculously are very\ninformative cues:\n+...zany characters and richly applied satire, and some great plot twists\n\u0000It was pathetic. The worst part about it was the boxing scenes...\n+...awesome caramel sauce and sweet toasty almonds. I love this place!\n\u0000...awful pizza and ridiculously overpriced...\nSpam detection is another important commercial application, the binary clas- spam detection\nsi\ufb01cation task of assigning an email to one of the two classes spam ornot-spam .\nMany lexical and other features can be used to perform this classi\ufb01cation. For ex-\nample you might quite reasonably be suspicious of an email containing phrases like\n\u201conline pharmaceutical\u201d or \u201cWITHOUT ANY COST\u201d or \u201cDear Winner\u201d.\nAnother thing we might want to know about a text is the language it\u2019s written\nin. Texts on social media, for example, can be in any number of languages and\nwe\u2019ll need to apply different processing. The task of language id is thus the \ufb01rst language id\nstep in most language processing pipelines. Related text classi\ufb01cation tasks like au-\nthorship attribution \u2014 determining a text\u2019s author\u2014 are also relevant to the digitalauthorship\nattribution\nhumanities, social sciences, and forensic linguistics. 2CHAPTER 4 \u2022 N AIVE BAYES , TEXT CLASSIFICATION ,AND SENTIMENT\nFinally, one of the oldest tasks in text classi\ufb01cation is assigning a library sub-\nject category or topic label to a text. Deciding whether a research paper concerns\nepidemiology or instead, perhaps, embryology, is an important component of infor-\nmation retrieval. Various sets of subject categories exist, such as the MeSH (Medical\nSubject Headings) thesaurus. In fact, as we will see, subject category classi\ufb01cation\nis the task for which the naive Bayes algorithm was invented in 1961 Maron (1961).\nClassi\ufb01cation is essential for tasks below the level of the document as well.\nWe\u2019ve already seen period disambiguation (deciding if a period is the end of a sen-\ntence or part of a word), and word tokenization (deciding if a character should be\na word boundary). Even language modeling can be viewed as classi\ufb01cation: each\nword can be thought of as a class, and so predicting the next word is classifying the\ncontext-so-far into a class for each next word. A part-of-speech tagger (Chapter 17)\nclassi\ufb01es each occurrence of a word in a sentence as, e.g., a noun or a verb.\nThe goal of classi\ufb01cation is to take a single observation, extract some useful\nfeatures, and thereby classify the observation into one of a set of discrete classes.\nOne method for classifying text is to use rules handwritten by humans. Handwrit-\nten rule-based classi\ufb01ers can be components of state-of-the-art systems in language\nprocessing. But rules can be fragile, as situations or data change over time, and for\nsome tasks humans aren\u2019t necessarily good at coming up with the rules.\nThe most common way of doing text classi\ufb01cation in language processing is\ninstead via supervised machine learning , the subject of this chapter. In supervisedsupervised\nmachine\nlearninglearning, we have a data set of input observations, each associated with some correct\noutput (a \u2018supervision signal\u2019). The goal of the algorithm is to learn how to map\nfrom a new observation to a correct output.\nFormally, the task of supervised classi\ufb01cation is to take an input xand a \ufb01xed\nset of output classes Y=fy1;y2;:::;yMgand return a predicted class y2Y. For\ntext classi\ufb01cation, we\u2019ll sometimes talk about c(for \u201cclass\u201d) instead of yas our\noutput variable, and d(for \u201cdocument\u201d) instead of xas our input variable. In the\nsupervised situation we have a training set of Ndocuments that have each been hand-\nlabeled with a class: f(d1;c1);::::; (dN;cN)g. Our goal is to learn a classi\ufb01er that is\ncapable of mapping from a new document dto its correct class c2C, where Cis\nsome set of useful document classes. A probabilistic classi\ufb01er additionally will tell\nus the probability of the observation being in the class. This full distribution over\nthe classes can be useful information for downstream decisions; avoiding making\ndiscrete decisions early on can be useful when combining systems.\nMany kinds of machine learning algorithms are used to build classi\ufb01ers. This\nchapter introduces naive Bayes; the following one introduces logistic regression.\nThese exemplify two ways of doing classi\ufb01cation. Generative classi\ufb01ers like naive\nBayes build a model of how a class could generate some input data. Given an ob-\nservation, they return the class most likely to have generated the observation. Dis-\ncriminative classi\ufb01ers like logistic regression instead learn what features from the\ninput are most useful to discriminate between the different possible classes. While\ndiscriminative systems are often more accurate and hence more commonly used,\ngenerative classi\ufb01ers still have a role.\n4.1 Naive Bayes Classi\ufb01ers\nIn this section we introduce the multinomial naive Bayes classi\ufb01er , so called be-naive Bayes\nclassi\ufb01er\ncause it is a Bayesian classi\ufb01er that makes a simplifying (naive) assumption about 4.1 \u2022 N AIVE BAYES CLASSIFIERS 3\nhow the features interact.\nThe intuition of the classi\ufb01er is shown in Fig. 4.1. We represent a text document\nas if it were a bag of words , that is, an unordered set of words with their position bag of words\nignored, keeping only their frequency in the document. In the example in the \ufb01gure,\ninstead of representing the word order in all the phrases like \u201cI love this movie\u201d and\n\u201cI would recommend it\u201d, we simply note that the word Ioccurred 5 times in the\nentire excerpt, the word it6 times, the words love,recommend , and movie once, and\nso on.\nititititititIIII\nIloverecommendmoviethethethetheto\ntotoand\nandandseenseenyetwouldwithwhowhimsical\nwhilewhenevertimessweetseveralscenessatiricalromanticofmanageshumorhavehappyfunfriendfairydialoguebutconventionsareanyoneadventurealwaysagainaboutI love this movie! It's sweet, but with satirical humor. The dialogue is great and the adventure scenes are fun... It manages to be whimsical and romantic while laughing at the conventions of the fairy tale genre. I would recommend it to just about anyone. I've seen it several times, and I'm always happy to see it again whenever I have a friend who hasn't seen it yet!it Ithetoandseenyetwouldwhimsicaltimessweetsatiricaladventuregenrefairyhumorhavegreat\u20266 54332111111111111\u2026\nFigure 4.1 Intuition of the multinomial naive Bayes classi\ufb01er applied to a movie review. The position of the\nwords is ignored (the bag-of-words assumption) and we make use of the frequency of each word.\nNaive Bayes is a probabilistic classi\ufb01er, meaning that for a document d, out of\nall classes c2Cthe classi\ufb01er returns the class \u02c6 cwhich has the maximum posterior\nprobability given the document. In Eq. 4.1 we use the hat notation \u02c6to mean \u201cour \u02c6\nestimate of the correct class\u201d, and we use argmax to mean an operation that selects argmax\nthe argument (in this case the class c) that maximizes a function (in this case the\nprobability P(cjd).\n\u02c6c=argmax\nc2CP(cjd) (4.1)\nThis idea of Bayesian inference has been known since the work of Bayes (1763),Bayesian\ninference\nand was \ufb01rst applied to text classi\ufb01cation by Mosteller and Wallace (1964). The\nintuition of Bayesian classi\ufb01cation is to use Bayes\u2019 rule to transform Eq. 4.1 into\nother probabilities that have some useful properties. Bayes\u2019 rule is presented in\nEq. 4.2; it gives us a way to break down any conditional probability P(xjy)into\nthree other probabilities:\nP(xjy) =P(yjx)P(x)\nP(y)(4.2) 4CHAPTER 4 \u2022 N AIVE BAYES , TEXT CLASSIFICATION ,AND SENTIMENT\nWe can then substitute Eq. 4.2 into Eq. 4.1 to get Eq. 4.3:\n\u02c6c=argmax\nc2CP(cjd) =argmax\nc2CP(djc)P(c)\nP(d)(4.3)\nWe can conveniently simplify Eq. 4.3 by dropping the denominator P(d). This\nis possible because we will be computingP(djc)P(c)\nP(d)for each possible class. But P(d)\ndoesn\u2019t change for each class; we are always asking about the most likely class for\nthe same document d, which must have the same probability P(d). Thus, we can\nchoose the class that maximizes this simpler formula:\n\u02c6c=argmax\nc2CP(cjd) =argmax\nc2CP(djc)P(c) (4.4)\nWe call Naive Bayes a generative model because we can read Eq. 4.4 as stating\na kind of implicit assumption about how a document is generated: \ufb01rst a class is\nsampled from P(c), and then the words are generated by sampling from P(djc). (In\nfact we could imagine generating arti\ufb01cial documents, or at least their word counts,\nby following this process). We\u2019ll say more about this intuition of generative models\nin Chapter 5.\nTo return to classi\ufb01cation: we compute the most probable class \u02c6 cgiven some\ndocument dby choosing the class which has the highest product of two probabilities:\ntheprior probability of the class P(c)and the likelihood of the document P(djc):prior\nprobability\nlikelihood\n\u02c6c=argmax\nc2Clikelihoodz}|{\nP(djc)priorz}|{\nP(c) (4.5)\nWithout loss of generality, we can represent a document das a set of features\nf1;f2;:::;fn:\n\u02c6c=argmax\nc2Clikelihoodz}|{\nP(f1;f2;::::; fnjc)priorz}|{\nP(c) (4.6)\nUnfortunately, Eq. 4.6 is still too hard to compute directly: without some sim-\nplifying assumptions, estimating the probability of every possible combination of\nfeatures (for example, every possible set of words and positions) would require huge\nnumbers of parameters and impossibly large training sets. Naive Bayes classi\ufb01ers\ntherefore make two simplifying assumptions.\nThe \ufb01rst is the bag-of-words assumption discussed intuitively above: we assume\nposition doesn\u2019t matter, and that the word \u201clove\u201d has the same effect on classi\ufb01cation\nwhether it occurs as the 1st, 20th, or last word in the document. Thus we assume\nthat the features f1;f2;:::;fnonly encode word identity and not position.\nThe second is commonly called the naive Bayes assumption : this is the condi-naive Bayes\nassumption\ntional independence assumption that the probabilities P(fijc)are independent given\nthe class cand hence can be \u2018naively\u2019 multiplied as follows:\nP(f1;f2;::::; fnjc) = P(f1jc)\u0001P(f2jc)\u0001:::\u0001P(fnjc) (4.7)\nThe \ufb01nal equation for the class chosen by a naive Bayes classi\ufb01er is thus:\ncNB=argmax\nc2CP(c)Y\nf2FP(fjc) (4.8) 4.2 \u2022 T RAINING THE NAIVE BAYES CLASSIFIER 5\nTo apply the naive Bayes classi\ufb01er to text, we will use each word in the documents\nas a feature, as suggested above, and we consider each of the words in the document\nby walking an index through every word position in the document:\npositions all word positions in test document\ncNB =argmax\nc2CP(c)Y\ni2positionsP(wijc) (4.9)\nNaive Bayes calculations, like calculations for language modeling, are done in log\nspace, to avoid under\ufb02ow and increase speed. Thus Eq. 4.9 is generally instead\nexpressed1as\ncNB=argmax\nc2ClogP(c) +X\ni2positionslogP(wijc) (4.10)\nBy considering features in log space, Eq. 4.10 computes the predicted class as a lin-\near function of input features. Classi\ufb01ers that use a linear combination of the inputs\nto make a classi\ufb01cation decision \u2014like naive Bayes and also logistic regression\u2014\nare called linear classi\ufb01ers .linear\nclassi\ufb01ers\n4.2 Training the Naive Bayes Classi\ufb01er\nHow can we learn the probabilities P(c)andP(fijc)? Let\u2019s \ufb01rst consider the maxi-\nmum likelihood estimate. We\u2019ll simply use the frequencies in the data. For the class\nprior P(c)we ask what percentage of the documents in our training set are in each\nclass c. Let Ncbe the number of documents in our training data with class cand\nNdocbe the total number of documents. Then:\n\u02c6P(c) =Nc\nNdoc(4.11)\nTo learn the probability P(fijc), we\u2019ll assume a feature is just the existence of a word\nin the document\u2019s bag of words, and so we\u2019ll want P(wijc), which we compute as\nthe fraction of times the word wiappears among all words in all documents of topic\nc. We \ufb01rst concatenate all documents with category cinto one big \u201ccategory c\u201d text.\nThen we use the frequency of wiin this concatenated document to give a maximum\nlikelihood estimate of the probability:\n\u02c6P(wijc) =count (wi;c)P\nw2Vcount (w;c)(4.12)\nHere the vocabulary V consists of the union of all the word types in all classes, not\njust the words in one class c.\nThere is a problem, however, with maximum likelihood training. Imagine we\nare trying to estimate the likelihood of the word \u201cfantastic\u201d given class positive , but\nsuppose there are no training documents that both contain the word \u201cfantastic\u201d and\nare classi\ufb01ed as positive . Perhaps the word \u201cfantastic\u201d happens to occur (sarcasti-\ncally?) in the class negative . In such a case the probability for this feature will be\nzero:\n\u02c6P(\u201cfantastic\u201djpositive ) =count (\u201cfantastic\u201d ;positive )P\nw2Vcount (w;positive )=0 (4.13)\n1In practice throughout this book, we\u2019ll use log to mean natural log (ln) when the base is not speci\ufb01ed. 6CHAPTER 4 \u2022 N AIVE BAYES , TEXT CLASSIFICATION ,AND SENTIMENT\nBut since naive Bayes naively multiplies all the feature likelihoods together, zero\nprobabilities in the likelihood term for any class will cause the probability of the\nclass to be zero, no matter the other evidence!\nThe simplest solution is the add-one (Laplace) smoothing introduced in Chap-\nter 3. While Laplace smoothing is usually replaced by more sophisticated smoothing\nalgorithms in language modeling, it is commonly used in naive Bayes text catego-\nrization:\n\u02c6P(wijc) =count (wi;c) +1P\nw2V(count (w;c) +1)=count (wi;c) +1\u0000P\nw2Vcount (w;c)\u0001\n+jVj(4.14)\nNote once again that it is crucial that the vocabulary V consists of the union of all the\nword types in all classes, not just the words in one class c(try to convince yourself\nwhy this must be true; see the exercise at the end of the chapter).\nWhat do we do about words that occur in our test data but are not in our vocab-\nulary at all because they did not occur in any training document in any class? The\nsolution for such unknown words is to ignore them\u2014remove them from the test unknown word\ndocument and not include any probability for them at all.\nFinally, some systems choose to completely ignore another class of words: stop\nwords , very frequent words like theanda. This can be done by sorting the vocabu- stop words\nlary by frequency in the training set, and de\ufb01ning the top 10\u2013100 vocabulary entries\nas stop words, or alternatively by using one of the many prede\ufb01ned stop word lists\navailable online. Then each instance of these stop words is simply removed from\nboth training and test documents as if it had never occurred. In most text classi\ufb01ca-\ntion applications, however, using a stop word list doesn\u2019t improve performance, and\nso it is more common to make use of the entire vocabulary and not use a stop word\nlist.\nFig. 4.2 shows the \ufb01nal algorithm.\n4.3 Worked example\nLet\u2019s walk through an example of training and testing naive Bayes with add-one\nsmoothing. We\u2019ll use a sentiment analysis domain with the two classes positive\n(+) and negative (-), and take the following miniature training and test documents\nsimpli\ufb01ed from actual movie reviews.\nCat Documents\nTraining - just plain boring\n- entirely predictable and lacks energy\n- no surprises and very few laughs\n+ very powerful\n+ the most fun \ufb01lm of the summer\nTest ? predictable with no fun\nThe prior P(c)for the two classes is computed via Eq. 4.11 asNc\nNdoc:\nP(\u0000) =3\n5P(+) =2\n5\nThe word with doesn\u2019t occur in the training set, so we drop it completely (as\nmentioned above, we don\u2019t use unknown word models for naive Bayes). The like-\nlihoods from the training set for the remaining three words \u201cpredictable\u201d, \u201cno\u201d, and 4.4 \u2022 O PTIMIZING FOR SENTIMENT ANALYSIS 7\nfunction TRAIN NAIVE BAYES (D, C) returns V;logP(c), log P(wjc)\nfor each class c2C # Calculate P(c)terms\nNdoc= number of documents in D\nNc= number of documents from D in class c\nlogprior [c] logNc\nNdoc\nV vocabulary of D\nbigdoc [c] append (d)ford2Dwith class c\nfor each word win V # Calculate P(wjc)terms\ncount(w,c) # of occurrences of winbigdoc [c]\nloglikelihood [w,c] logcount (w;c) + 1P\nw0in V(count (w0;c) + 1)\nreturn logprior ,loglikelihood ,V\nfunction TESTNAIVE BAYES (testdoc ,logprior ,loglikelihood , C, V) returns bestc\nfor each class c2C\nsum[c] logprior [c]\nfor each position iintestdoc\nword testdoc[i]\nifword2V\nsum[c] sum[c]+loglikelihood [word ,c]\nreturn argmaxcsum[c]\nFigure 4.2 The naive Bayes algorithm, using add-1 smoothing. To use add- asmoothing\ninstead, change the +1 to +afor loglikelihood counts in training.\n\u201cfun\u201d, are as follows, from Eq. 4.14 (computing the probabilities for the remainder\nof the words in the training set is left as an exercise for the reader):\nP(\u201cpredictable\u201dj\u0000) =1+1\n14+20P(\u201cpredictable\u201dj+) =0+1\n9+20\nP(\u201cno\u201dj\u0000) =1+1\n14+20P(\u201cno\u201dj+) =0+1\n9+20\nP(\u201cfun\u201dj\u0000) =0+1\n14+20P(\u201cfun\u201dj+) =1+1\n9+20\nFor the test sentence S = \u201cpredictable with no fun\u201d, after removing the word \u2018with\u2019,\nthe chosen class, via Eq. 4.9, is therefore computed as follows:\nP(\u0000)P(Sj\u0000) =3\n5\u00022\u00022\u00021\n343=6:1\u000210\u00005\nP(+)P(Sj+) =2\n5\u00021\u00021\u00022\n293=3:2\u000210\u00005\nThe model thus predicts the class negative for the test sentence.\n4.4 Optimizing for Sentiment Analysis\nWhile standard naive Bayes text classi\ufb01cation can work well for sentiment analysis,\nsome small changes are generally employed that improve performance. 8CHAPTER 4 \u2022 N AIVE BAYES , TEXT CLASSIFICATION ,AND SENTIMENT\nFirst, for sentiment classi\ufb01cation and a number of other text classi\ufb01cation tasks,\nwhether a word occurs or not seems to matter more than its frequency. Thus it often\nimproves performance to clip the word counts in each document at 1 (see the end\nof the chapter for pointers to these results). This variant is called binary multino-\nmial naive Bayes orbinary naive Bayes . The variant uses the same algorithm asbinary naive\nBayes\nin Fig. 4.2 except that for each document we remove all duplicate words before con-\ncatenating them into the single big document during training and we also remove\nduplicate words from test documents. Fig. 4.3 shows an example in which a set\nof four documents (shortened and text-normalized for this example) are remapped\nto binary, with the modi\ufb01ed counts shown in the table on the right. The example\nis worked without add-1 smoothing to make the differences clearer. Note that the\nresults counts need not be 1; the word great has a count of 2 even for binary naive\nBayes, because it appears in multiple documents.\nFour original documents:\n\u0000it was pathetic the worst part was the\nboxing scenes\n\u0000no plot twists or great scenes\n+and satire and great plot twists\n+great scenes great \ufb01lm\nAfter per-document binarization:\n\u0000it was pathetic the worst part boxing\nscenes\n\u0000no plot twists or great scenes\n+and satire great plot twists\n+great scenes \ufb01lmNB Binary\nCounts Counts\n+\u0000 +\u0000\nand 2 0 1 0\nboxing 0 1 0 1\n\ufb01lm 1 0 1 0\ngreat 3 1 2 1\nit 0 1 0 1\nno 0 1 0 1\nor 0 1 0 1\npart 0 1 0 1\npathetic 0 1 0 1\nplot 1 1 1 1\nsatire 1 0 1 0\nscenes 1 2 1 2\nthe 0 2 0 1\ntwists 1 1 1 1\nwas 0 2 0 1\nworst 0 1 0 1\nFigure 4.3 An example of binarization for the binary naive Bayes algorithm.\nA second important addition commonly made when doing text classi\ufb01cation for\nsentiment is to deal with negation. Consider the difference between I really like this\nmovie (positive) and I didn\u2019t like this movie (negative). The negation expressed by\ndidn\u2019t completely alters the inferences we draw from the predicate like. Similarly,\nnegation can modify a negative word to produce a positive review ( don\u2019t dismiss this\n\ufb01lm,doesn\u2019t let us get bored ).\nA very simple baseline that is commonly used in sentiment analysis to deal with\nnegation is the following: during text normalization, prepend the pre\ufb01x NOT to\nevery word after a token of logical negation ( n\u2019t, not, no, never ) until the next punc-\ntuation mark. Thus the phrase\ndidn't like this movie , but I\nbecomes\ndidn't NOT_like NOT_this NOT_movie , but I\nNewly formed \u2018words\u2019 like NOT like,NOT recommend will thus occur more\noften in negative document and act as cues for negative sentiment, while words\nlikeNOT bored ,NOT dismiss will acquire positive associations. Syntactic parsing\n(Chapter 18) can be used deal more accurately with the scope relationship between 4.5 \u2022 N AIVE BAYES FOR OTHER TEXT CLASSIFICATION TASKS 9\nthese negation words and the predicates they modify, but this simple baseline works\nquite well in practice.\nFinally, in some situations we might have insuf\ufb01cient labeled training data to\ntrain accurate naive Bayes classi\ufb01ers using all words in the training set to estimate\npositive and negative sentiment. In such cases we can instead derive the positive\nand negative word features from sentiment lexicons , lists of words that are pre-sentiment\nlexicons\nannotated with positive or negative sentiment. Four popular lexicons are the General\nInquirer (Stone et al., 1966), LIWC (Pennebaker et al., 2007), the opinion lexiconGeneral\nInquirer\nLIWC of Hu and Liu (2004) and the MPQA Subjectivity Lexicon (Wilson et al., 2005).\nFor example the MPQA subjectivity lexicon has 6885 words each marked for\nwhether it is strongly or weakly biased positive or negative. Some examples:\n+:admirable, beautiful, con\ufb01dent, dazzling, ecstatic, favor, glee, great\n\u0000:awful, bad, bias, catastrophe, cheat, deny, envious, foul, harsh, hate\nA common way to use lexicons in a naive Bayes classi\ufb01er is to add a feature\nthat is counted whenever a word from that lexicon occurs. Thus we might add a\nfeature called \u2018this word occurs in the positive lexicon\u2019, and treat all instances of\nwords in the lexicon as counts for that one feature, instead of counting each word\nseparately. Similarly, we might add as a second feature \u2018this word occurs in the\nnegative lexicon\u2019 of words in the negative lexicon. If we have lots of training data,\nand if the test data matches the training data, using just two features won\u2019t work as\nwell as using all the words. But when training data is sparse or not representative of\nthe test set, using dense lexicon features instead of sparse individual-word features\nmay generalize better.\nWe\u2019ll return to this use of lexicons in Chapter 22, showing how these lexicons\ncan be learned automatically, and how they can be applied to many other tasks be-\nyond sentiment classi\ufb01cation.\n4.5 Naive Bayes for other text classi\ufb01cation tasks\nIn the previous section we pointed out that naive Bayes doesn\u2019t require that our\nclassi\ufb01er use all the words in the training data as features. In fact features in naive\nBayes can express any property of the input text we want.\nConsider the task of spam detection , deciding if a particular piece of email is spam detection\nan example of spam (unsolicited bulk email)\u2014one of the \ufb01rst applications of naive\nBayes to text classi\ufb01cation (Sahami et al., 1998).\nA common solution here, rather than using all the words as individual features,\nis to prede\ufb01ne likely sets of words or phrases as features, combined with features\nthat are not purely linguistic. For example the open-source SpamAssassin tool2\nprede\ufb01nes features like the phrase \u201cone hundred percent guaranteed\u201d, or the feature\nmentions millions of dollars , which is a regular expression that matches suspiciously\nlarge sums of money. But it also includes features like HTML has a low ratio of text\nto image area , that aren\u2019t purely linguistic and might require some sophisticated\ncomputation, or totally non-linguistic features about, say, the path that the email\ntook to arrive. More sample SpamAssassin features:\n\u2022 Email subject line is all capital letters\n\u2022 Contains phrases of urgency like \u201curgent reply\u201d\n2https://spamassassin.apache.org 10 CHAPTER 4 \u2022 N AIVE BAYES , TEXT CLASSIFICATION ,AND SENTIMENT\n\u2022 Email subject line contains \u201conline pharmaceutical\u201d\n\u2022 HTML has unbalanced \u201chead\u201d tags\n\u2022 Claims you can be removed from the list\nFor other tasks, like language id \u2014determining what language a given piece language id\nof text is written in\u2014the most effective naive Bayes features are not words at all,\nbutcharacter n-grams , 2-grams (\u2018zw\u2019) 3-grams (\u2018nya\u2019, \u2018 V o\u2019), or 4-grams (\u2018ie z\u2019,\n\u2018thei\u2019), or, even simpler byte n-grams , where instead of using the multibyte Unicode\ncharacter representations called codepoints, we just pretend everything is a string of\nraw bytes. Because spaces count as a byte, byte n-grams can model statistics about\nthe beginning or ending of words. A widely used naive Bayes system, langid.py\n(Lui and Baldwin, 2012) begins with all possible n-grams of lengths 1-4, using fea-\nture selection to winnow down to the most informative 7000 \ufb01nal features.\nLanguage ID systems are trained on multilingual text, such as Wikipedia (Wiki-\npedia text in 68 different languages was used in (Lui and Baldwin, 2011)), or newswire.\nTo make sure that this multilingual text correctly re\ufb02ects different regions, dialects,\nand socioeconomic classes, systems also add Twitter text in many languages geo-\ntagged to many regions (important for getting world English dialects from countries\nwith large Anglophone populations like Nigeria or India), Bible and Quran transla-\ntions, slang websites like Urban Dictionary, corpora of African American Vernacular\nEnglish (Blodgett et al., 2016), and so on (Jurgens et al., 2017).\n4.6 Naive Bayes as a Language Model\nAs we saw in the previous section, naive Bayes classi\ufb01ers can use any sort of feature:\ndictionaries, URLs, email addresses, network features, phrases, and so on. But if,\nas in Section 4.3, we use only individual word features, and we use all of the words\nin the text (not a subset), then naive Bayes has an important similarity to language\nmodeling. Speci\ufb01cally, a naive Bayes model can be viewed as a set of class-speci\ufb01c\nunigram language models, in which the model for each class instantiates a unigram\nlanguage model.\nSince the likelihood features from the naive Bayes model assign a probability to\neach word P(wordjc), the model also assigns a probability to each sentence:\nP(sjc) =Y\ni2positionsP(wijc) (4.15)\nThus consider a naive Bayes model with the classes positive (+) and negative (-)\nand the following model parameters:\nw P(wj+)P(wj-)\nI 0.1 0.2\nlove 0.1 0.001\nthis 0.01 0.01\nfun 0.05 0.005\n\ufb01lm 0.1 0.1\n... ... ...\nEach of the two columns above instantiates a language model that can assign a\nprobability to the sentence \u201cI love this fun \ufb01lm\u201d: 4.7 \u2022 E VALUATION : PRECISION , RECALL , F- MEASURE 11\nP(\u201cI love this fun \ufb01lm\u201d j+) = 0:1\u00020:1\u00020:01\u00020:05\u00020:1=5\u000210\u00007\nP(\u201cI love this fun \ufb01lm\u201d j\u0000) = 0:2\u00020:001\u00020:01\u00020:005\u00020:1=1:0\u000210\u00009\nAs it happens, the positive model assigns a higher probability to the sentence:\nP(sjpos)>P(sjneg). Note that this is just the likelihood part of the naive Bayes\nmodel; once we multiply in the prior a full naive Bayes model might well make a\ndifferent classi\ufb01cation decision.\n4.7 Evaluation: Precision, Recall, F-measure\nTo introduce the methods for evaluating text classi\ufb01cation, let\u2019s \ufb01rst consider some\nsimple binary detection tasks. For example, in spam detection, our goal is to label\nevery text as being in the spam category (\u201cpositive\u201d) or not in the spam category\n(\u201cnegative\u201d). For each item (email document) we therefore need to know whether\nour system called it spam or not. We also need to know whether the email is actually\nspam or not, i.e. the human-de\ufb01ned labels for each document that we are trying to\nmatch. We will refer to these human labels as the gold labels . gold labels\nOr imagine you\u2019re the CEO of the Delicious Pie Company and you need to know\nwhat people are saying about your pies on social media, so you build a system that\ndetects tweets concerning Delicious Pie. Here the positive class is tweets about\nDelicious Pie and the negative class is all other tweets.\nIn both cases, we need a metric for knowing how well our spam detector (or\npie-tweet-detector) is doing. To evaluate any system for detecting things, we start\nby building a confusion matrix like the one shown in Fig. 4.4. A confusion matrixconfusion\nmatrix\nis a table for visualizing how an algorithm performs with respect to the human gold\nlabels, using two dimensions (system output and gold labels), and each cell labeling\na set of possible outcomes. In the spam detection case, for example, true positives\nare documents that are indeed spam (indicated by human-created gold labels) that\nour system correctly said were spam. False negatives are documents that are indeed\nspam but our system incorrectly labeled as non-spam.\nTo the bottom right of the table is the equation for accuracy , which asks what\npercentage of all the observations (for the spam or pie examples that means all emails\nor tweets) our system labeled correctly. Although accuracy might seem a natural\nmetric, we generally don\u2019t use it for text classi\ufb01cation tasks. That\u2019s because accuracy\ndoesn\u2019t work well when the classes are unbalanced (as indeed they are with spam,\nwhich is a large majority of email, or with tweets, which are mainly not about pie).\nTo make this more explicit, imagine that we looked at a million tweets, and\nlet\u2019s say that only 100 of them are discussing their love (or hatred) for our pie,\nwhile the other 999,900 are tweets about something completely unrelated. Imagine a\nsimple classi\ufb01er that stupidly classi\ufb01ed every tweet as \u201cnot about pie\u201d. This classi\ufb01er\nwould have 999,900 true negatives and only 100 false negatives for an accuracy of\n999,900/1,000,000 or 99.99%! What an amazing accuracy level! Surely we should\nbe happy with this classi\ufb01er? But of course this fabulous \u2018no pie\u2019 classi\ufb01er would\nbe completely useless, since it wouldn\u2019t \ufb01nd a single one of the customer comments\nwe are looking for. In other words, accuracy is not a good metric when the goal is\nto discover something that is rare, or at least not completely balanced in frequency,\nwhich is a very common situation in the world. 12 CHAPTER 4 \u2022 N AIVE BAYES , TEXT CLASSIFICATION ,AND SENTIMENT\ntrue positivefalse negativefalse positivetrue negativegold positivegold negativesystempositivesystemnegativegold standard labelssystemoutputlabelsrecall = tptp+fnprecision = tptp+fpaccuracy = tp+tntp+fp+tn+fn\nFigure 4.4 A confusion matrix for visualizing how well a binary classi\ufb01cation system per-\nforms against gold standard labels.\nThat\u2019s why instead of accuracy we generally turn to two other metrics shown in\nFig. 4.4: precision andrecall .Precision measures the percentage of the items that precision\nthe system detected (i.e., the system labeled as positive) that are in fact positive (i.e.,\nare positive according to the human gold labels). Precision is de\ufb01ned as\nPrecision =true positives\ntrue positives + false positives\nRecall measures the percentage of items actually present in the input that were recall\ncorrectly identi\ufb01ed by the system. Recall is de\ufb01ned as\nRecall =true positives\ntrue positives + false negatives\nPrecision and recall will help solve the problem with the useless \u201cnothing is\npie\u201d classi\ufb01er. This classi\ufb01er, despite having a fabulous accuracy of 99.99%, has\na terrible recall of 0 (since there are no true positives, and 100 false negatives, the\nrecall is 0/100). You should convince yourself that the precision at \ufb01nding relevant\ntweets is equally problematic. Thus precision and recall, unlike accuracy, emphasize\ntrue positives: \ufb01nding the things that we are supposed to be looking for.\nThere are many ways to de\ufb01ne a single metric that incorporates aspects of both\nprecision and recall. The simplest of these combinations is the F-measure (van F-measure\nRijsbergen, 1975) , de\ufb01ned as:\nFb=(b2+1)PR\nb2P+R\nThebparameter differentially weights the importance of recall and precision,\nbased perhaps on the needs of an application. Values of b>1 favor recall, while\nvalues of b<1 favor precision. When b=1, precision and recall are equally bal-\nanced; this is the most frequently used metric, and is called F b=1or just F 1: F1\nF1=2PR\nP+R(4.16)\nF-measure comes from a weighted harmonic mean of precision and recall. The\nharmonic mean of a set of numbers is the reciprocal of the arithmetic mean of recip-\nrocals:\nHarmonicMean (a1;a2;a3;a4;:::;an) =n\n1\na1+1\na2+1\na3+:::+1\nan(4.17) 4.7 \u2022 E VALUATION : PRECISION , RECALL , F- MEASURE 13\nand hence F-measure is\nF=1\na1\nP+ (1\u0000a)1\nRor\u0012\nwithb2=1\u0000a\na\u0013\nF=(b2+1)PR\nb2P+R(4.18)\nHarmonic mean is used because the harmonic mean of two values is closer to the\nminimum of the two values than the arithmetic mean is. Thus it weighs the lower of\nthe two numbers more heavily, which is more conservative in this situation.\n4.7.1 Evaluating with more than two classes\nUp to now we have been describing text classi\ufb01cation tasks with only two classes.\nBut lots of classi\ufb01cation tasks in language processing have more than two classes.\nFor sentiment analysis we generally have 3 classes (positive, negative, neutral) and\neven more classes are common for tasks like part-of-speech tagging, word sense\ndisambiguation, semantic role labeling, emotion detection, and so on. Luckily the\nnaive Bayes algorithm is already a multi-class classi\ufb01cation algorithm.\n851060urgentnormalgold labelssystemoutputrecallu = 88+5+3precisionu= 88+10+115030200spamurgentnormalspam3recalln = recalls = precisionn= 605+60+50precisions= 2003+30+2006010+60+302001+50+200\nFigure 4.5 Confusion matrix for a three-class categorization task, showing for each pair of\nclasses (c1;c2), how many documents from c1were (in)correctly assigned to c2.\nBut we\u2019ll need to slightly modify our de\ufb01nitions of precision and recall. Con-\nsider the sample confusion matrix for a hypothetical 3-way one-of email catego-\nrization decision (urgent, normal, spam) shown in Fig. 4.5. The matrix shows, for\nexample, that the system mistakenly labeled one spam document as urgent, and we\nhave shown how to compute a distinct precision and recall value for each class. In\norder to derive a single metric that tells us how well the system is doing, we can com-\nbine these values in two ways. In macroaveraging , we compute the performance macroaveraging\nfor each class, and then average over classes. In microaveraging , we collect the de- microaveraging\ncisions for all classes into a single confusion matrix, and then compute precision and\nrecall from that table. Fig. 4.6 shows the confusion matrix for each class separately,\nand shows the computation of microaveraged and macroaveraged precision.\nAs the \ufb01gure shows, a microaverage is dominated by the more frequent class (in\nthis case spam), since the counts are pooled. The macroaverage better re\ufb02ects the\nstatistics of the smaller classes, and so is more appropriate when performance on all\nthe classes is equally important. 14 CHAPTER 4 \u2022 N AIVE BAYES , TEXT CLASSIFICATION ,AND SENTIMENT\n8811340trueurgenttruenotsystemurgentsystemnot604055212truenormaltruenotsystemnormalsystemnot200513383truespamtruenotsystemspamsystemnot2689999635trueyestruenosystemyessystemnoprecision =8+118= .42precision =200+33200= .86precision =60+5560= .52microaverageprecision268+99268= .73=macroaverageprecision3.42+.52+.86= .60=PooledClass 3: SpamClass 2: NormalClass 1: Urgent\nFigure 4.6 Separate confusion matrices for the 3 classes from the previous \ufb01gure, showing the pooled confu-\nsion matrix and the microaveraged and macroaveraged precision.\n4.8 Test sets and Cross-validation\nThe training and testing procedure for text classi\ufb01cation follows what we saw with\nlanguage modeling (Section ??): we use the training set to train the model, then use\nthedevelopment test set (also called a devset ) to perhaps tune some parameters,development\ntest set\ndevset and in general decide what the best model is. Once we come up with what we think\nis the best model, we run it on the (hitherto unseen) test set to report its performance.\nWhile the use of a devset avoids over\ufb01tting the test set, having a \ufb01xed train-\ning set, devset, and test set creates another problem: in order to save lots of data\nfor training, the test set (or devset) might not be large enough to be representative.\nWouldn\u2019t it be better if we could somehow use all our data for training and still use\nall our data for test? We can do this by cross-validation . cross-validation\nIn cross-validation, we choose a number k, and partition our data into kdisjoint\nsubsets called folds . Now we choose one of those kfolds as a test set, train our folds\nclassi\ufb01er on the remaining k\u00001 folds, and then compute the error rate on the test\nset. Then we repeat with another fold as the test set, again training on the other k\u00001\nfolds. We do this sampling process ktimes and average the test set error rate from\nthese kruns to get an average error rate. If we choose k=10, we would train 10\ndifferent models (each on 90% of our data), test the model 10 times, and average\nthese 10 values. This is called 10-fold cross-validation .10-fold\ncross-validation\nThe only problem with cross-validation is that because all the data is used for\ntesting, we need the whole corpus to be blind; we can\u2019t examine any of the data\nto suggest possible features and in general see what\u2019s going on, because we\u2019d be\npeeking at the test set, and such cheating would cause us to overestimate the perfor-\nmance of our system. However, looking at the corpus to understand what\u2019s going\non is important in designing NLP systems! What to do? For this reason, it is com-\nmon to create a \ufb01xed training set and test set, then do 10-fold cross-validation inside\nthe training set, but compute error rate the normal way in the test set, as shown in\nFig. 4.7. 4.9 \u2022 S TATISTICAL SIGNIFICANCE TESTING 15\nTraining Iterations13452678910DevDevDevDevDevDevDevDevDevDevTrainingTrainingTrainingTrainingTrainingTrainingTrainingTrainingTrainingTrainingTrainingTest SetTesting\nFigure 4.7 10-fold cross-validation\n4.9 Statistical Signi\ufb01cance Testing\nIn building systems we often need to compare the performance of two systems. How\ncan we know if the new system we just built is better than our old one? Or better\nthan some other system described in the literature? This is the domain of statistical\nhypothesis testing, and in this section we introduce tests for statistical signi\ufb01cance\nfor NLP classi\ufb01ers, drawing especially on the work of Dror et al. (2020) and Berg-\nKirkpatrick et al. (2012).\nSuppose we\u2019re comparing the performance of classi\ufb01ers AandBon a metric M\nsuch as F 1, or accuracy. Perhaps we want to know if our logistic regression senti-\nment classi\ufb01er A(Chapter 5) gets a higher F 1score than our naive Bayes sentiment\nclassi\ufb01er Bon a particular test set x. Let\u2019s call M(A;x)the score that system Agets\non test set x, and d(x)the performance difference between AandBonx:\nd(x) =M(A;x)\u0000M(B;x) (4.19)\nWe would like to know if d(x)>0, meaning that our logistic regression classi\ufb01er\nhas a higher F 1than our naive Bayes classi\ufb01er on x.d(x)is called the effect size ; a effect size\nbigger dmeans that Aseems to be way better than B; a small dmeans Aseems to\nbe only a little better.\nWhy don\u2019t we just check if d(x)is positive? Suppose we do, and we \ufb01nd that\nthe F 1score of Ais higher than B\u2019s by .04. Can we be certain that Ais better? We\ncannot! That\u2019s because Amight just be accidentally better than Bon this particular x.\nWe need something more: we want to know if A\u2019s superiority over Bis likely to hold\nagain if we checked another test set x0, or under some other set of circumstances.\nIn the paradigm of statistical hypothesis testing, we test this by formalizing two\nhypotheses.\nH0:d(x)\u00140\nH1:d(x)>0 (4.20)\nThe hypothesis H0, called the null hypothesis , supposes that d(x)is actually nega- null hypothesis\ntive or zero, meaning that Ais not better than B. We would like to know if we can\ncon\ufb01dently rule out this hypothesis, and instead support H1, that Ais better.\nWe do this by creating a random variable Xranging over all test sets. Now we\nask how likely is it, if the null hypothesis H0was correct, that among these test sets 16 CHAPTER 4 \u2022 N AIVE BAYES , TEXT CLASSIFICATION ,AND SENTIMENT\nwe would encounter the value of d(x)that we found, if we repeated the experiment\na great many times. We formalize this likelihood as the p-value : the probability, p-value\nassuming the null hypothesis H0is true, of seeing the d(x)that we saw or one even\ngreater\nP(d(X)\u0015d(x)jH0is true ) (4.21)\nSo in our example, this p-value is the probability that we would see d(x)assuming\nAisnotbetter than B. Ifd(x)is huge (let\u2019s say Ahas a very respectable F 1of .9\nandBhas a terrible F 1of only .2 on x), we might be surprised, since that would be\nextremely unlikely to occur if H0were in fact true, and so the p-value would be low\n(unlikely to have such a large difAis in fact not better than B). But if d(x)is very\nsmall, it might be less surprising to us even if H0were true and Ais not really better\nthan B, and so the p-value would be higher.\nA very small p-value means that the difference we observed is very unlikely\nunder the null hypothesis, and we can reject the null hypothesis. What counts as very\nsmall? It is common to use values like .05 or .01 as the thresholds. A value of .01\nmeans that if the p-value (the probability of observing the dwe saw assuming H0is\ntrue) is less than .01, we reject the null hypothesis and assume that Ais indeed better\nthan B. We say that a result (e.g., \u201c Ais better than B\u201d) is statistically signi\ufb01cant ifstatistically\nsigni\ufb01cant\nthedwe saw has a probability that is below the threshold and we therefore reject\nthis null hypothesis.\nHow do we compute this probability we need for the p-value? In NLP we gen-\nerally don\u2019t use simple parametric tests like t-tests or ANOV As that you might be\nfamiliar with. Parametric tests make assumptions about the distributions of the test\nstatistic (such as normality) that don\u2019t generally hold in our cases. So in NLP we\nusually use non-parametric tests based on sampling: we arti\ufb01cially create many ver-\nsions of the experimental setup. For example, if we had lots of different test sets x0\nwe could just measure all the d(x0)for all the x0. That gives us a distribution. Now\nwe set a threshold (like .01) and if we see in this distribution that 99% or more of\nthose deltas are smaller than the delta we observed, i.e., that p-value( x)\u2014the proba-\nbility of seeing a d(x)as big as the one we saw\u2014is less than .01, then we can reject\nthe null hypothesis and agree that d(x)was a suf\ufb01ciently surprising difference and\nAis really a better algorithm than B.\nThere are two common non-parametric tests used in NLP: approximate ran-\ndomization (Noreen, 1989) and the bootstrap test . We will describe bootstrapapproximate\nrandomization\nbelow, showing the paired version of the test, which again is most common in NLP.\nPaired tests are those in which we compare two sets of observations that are aligned: paired\neach observation in one set can be paired with an observation in another. This hap-\npens naturally when we are comparing the performance of two systems on the same\ntest set; we can pair the performance of system Aon an individual observation xi\nwith the performance of system Bon the same xi.\n4.9.1 The Paired Bootstrap Test\nThebootstrap test (Efron and Tibshirani, 1993) can apply to any metric; from pre- bootstrap test\ncision, recall, or F1 to the BLEU metric used in machine translation. The word\nbootstrapping refers to repeatedly drawing large numbers of samples with replace- bootstrapping\nment (called bootstrap samples ) from an original set. The intuition of the bootstrap\ntest is that we can create many virtual test sets from an observed test set by repeat-\nedly sampling from it. The method only makes the assumption that the sample is\nrepresentative of the population. 4.9 \u2022 S TATISTICAL SIGNIFICANCE TESTING 17\nConsider a tiny text classi\ufb01cation example with a test set xof 10 documents. The\n\ufb01rst row of Fig. 4.8 shows the results of two classi\ufb01ers (A and B) on this test set.\nEach document is labeled by one of the four possibilities (A and B both right, both\nwrong, A right and B wrong, A wrong and B right). A slash through a letter ( \u0013B)\nmeans that that classi\ufb01er got the answer wrong. On the \ufb01rst document both A and\nB get the correct class (AB), while on the second document A got it right but B got\nit wrong (A \u0013B). If we assume for simplicity that our metric is accuracy, A has an\naccuracy of .70 and B of .50, so d(x)is .20.\nNow we create a large number b(perhaps 105) of virtual test sets x(i), each of size\nn=10. Fig. 4.8 shows a couple of examples. To create each virtual test set x(i), we\nrepeatedly ( n=10 times) select a cell from row xwith replacement. For example, to\ncreate the \ufb01rst cell of the \ufb01rst virtual test set x(1), if we happened to randomly select\nthe second cell of the xrow; we would copy the value A \u0013B into our new cell, and\nmove on to create the second cell of x(1), each time sampling (randomly choosing)\nfrom the original xwith replacement.\n1 2 3 4 5 6 7 8 910A% B% d()\nx AB A\u0013\u0013BAB\u0000\u0000AB A\u0013\u0013B\u0000\u0000AB A\u0013\u0013BAB\u0000\u0000A\u0013\u0013BA\u0013\u0013B.70 .50 .20\nx(1)A\u0013\u0013BAB A\u0013\u0013B\u0000\u0000AB\u0000\u0000AB A\u0013\u0013B\u0000\u0000AB AB\u0000\u0000A\u0013\u0013BAB .60 .60 .00\nx(2)A\u0013\u0013BAB\u0000\u0000A\u0013\u0013B\u0000\u0000AB\u0000\u0000AB AB\u0000\u0000AB A\u0013\u0013BAB AB .60 .70-.10\n...\nx(b)\nFigure 4.8 The paired bootstrap test: Examples of bpseudo test sets x(i)being created\nfrom an initial true test set x. Each pseudo test set is created by sampling n=10 times with\nreplacement; thus an individual sample is a single cell, a document with its gold label and\nthe correct or incorrect performance of classi\ufb01ers A and B. Of course real test sets don\u2019t have\nonly 10 examples, and bneeds to be large as well.\nNow that we have the btest sets, providing a sampling distribution, we can do\nstatistics on how often Ahas an accidental advantage. There are various ways to\ncompute this advantage; here we follow the version laid out in Berg-Kirkpatrick\net al. (2012). Assuming H0(Aisn\u2019t better than B), we would expect that d(X),\nestimated over many test sets, would be zero or negative; a much higher value would\nbe surprising, since H0speci\ufb01cally assumes Aisn\u2019t better than B. To measure exactly\nhow surprising our observed d(x)is, we would in other circumstances compute the\np-value by counting over many test sets how often d(x(i))exceeds the expected zero\nvalue by d(x)or more:\np-value (x) =1\nbbX\ni=11\u0010\nd(x(i))\u0000d(x)\u00150\u0011\n(We use the notation 1(x)to mean \u201c1 if xis true, and 0 otherwise\u201d.) However,\nalthough it\u2019s generally true that the expected value of d(X)over many test sets,\n(again assuming Aisn\u2019t better than B) is 0, this isn\u2019t true for the bootstrapped test\nsets we created. That\u2019s because we didn\u2019t draw these samples from a distribution\nwith 0 mean; we happened to create them from the original test set x, which happens\nto be biased (by .20) in favor of A. So to measure how surprising is our observed\nd(x), we actually compute the p-value by counting over many test sets how often 18 CHAPTER 4 \u2022 N AIVE BAYES , TEXT CLASSIFICATION ,AND SENTIMENT\nd(x(i))exceeds the expected value of d(x)byd(x)or more:\np-value (x) =1\nbbX\ni=11\u0010\nd(x(i))\u0000d(x)\u0015d(x)\u0011\n=1\nbbX\ni=11\u0010\nd(x(i))\u00152d(x)\u0011\n(4.22)\nSo if for example we have 10,000 test sets x(i)and a threshold of .01, and in only 47\nof the test sets do we \ufb01nd that A is accidentally better d(x(i))\u00152d(x), the resulting\np-value of .0047 is smaller than .01, indicating that the delta we found, d(x)is indeed\nsuf\ufb01ciently surprising and unlikely to have happened by accident, and we can reject\nthe null hypothesis and conclude Ais better than B.\nfunction BOOTSTRAP (test set x,num of samples b)returns p-value (x)\nCalculate d(x)# how much better does algorithm A do than B on x\ns= 0\nfori= 1tobdo\nforj= 1tondo # Draw a bootstrap sample x(i)of size n\nSelect a member of xat random and add it to x(i)\nCalculate d(x(i))# how much better does algorithm A do than B on x(i)\ns s+ 1ifd(x(i))\u00152d(x)\np-value( x)\u0019s\nb# on what % of the b samples did algorithm A beat expectations?\nreturn p-value( x) # if very few did, our observed dis probably not accidental\nFigure 4.9 A version of the paired bootstrap algorithm after Berg-Kirkpatrick et al. (2012).\nThe full algorithm for the bootstrap is shown in Fig. 4.9. It is given a test set x, a\nnumber of samples b, and counts the percentage of the bbootstrap test sets in which\nd(x\u0003(i))>2d(x). This percentage then acts as a one-sided empirical p-value.\n4.10 Avoiding Harms in Classi\ufb01cation\nIt is important to avoid harms that may result from classi\ufb01ers, harms that exist both\nfor naive Bayes classi\ufb01ers and for the other classi\ufb01cation algorithms we introduce\nin later chapters.\nOne class of harms is representational harms (Crawford 2017, Blodgett et al.representational\nharms\n2020), harms caused by a system that demeans a social group, for example by per-\npetuating negative stereotypes about them. For example Kiritchenko and Moham-\nmad (2018) examined the performance of 200 sentiment analysis systems on pairs of\nsentences that were identical except for containing either a common African Amer-\nican \ufb01rst name (like Shaniqua ) or a common European American \ufb01rst name (like\nStephanie ), chosen from the Caliskan et al. (2017) study discussed in Chapter 6.\nThey found that most systems assigned lower sentiment and more negative emotion\nto sentences with African American names, re\ufb02ecting and perpetuating stereotypes\nthat associate African Americans with negative emotions (Popp et al., 2003).\nIn other tasks classi\ufb01ers may lead to both representational harms and other\nharms, such as silencing. For example the important text classi\ufb01cation task of tox- 4.11 \u2022 S UMMARY 19\nicity detection is the task of detecting hate speech, abuse, harassment, or othertoxicity\ndetection\nkinds of toxic language. While the goal of such classi\ufb01ers is to help reduce soci-\netal harm, toxicity classi\ufb01ers can themselves cause harms. For example, researchers\nhave shown that some widely used toxicity classi\ufb01ers incorrectly \ufb02ag as being toxic\nsentences that are non-toxic but simply mention identities like women (Park et al.,\n2018), blind people (Hutchinson et al., 2020) or gay people (Dixon et al., 2018;\nDias Oliva et al., 2021), or simply use linguistic features characteristic of varieties\nlike African-American Vernacular English (Sap et al. 2019, Davidson et al. 2019).\nSuch false positive errors could lead to the silencing of discourse by or about these\ngroups.\nThese model problems can be caused by biases or other problems in the training\ndata; in general, machine learning systems replicate and even amplify the biases\nin their training data. But these problems can also be caused by the labels (for\nexample due to biases in the human labelers), by the resources used (like lexicons,\nor model components like pretrained embeddings), or even by model architecture\n(like what the model is trained to optimize). While the mitigation of these biases\n(for example by carefully considering the training data sources) is an important area\nof research, we currently don\u2019t have general solutions. For this reason it\u2019s important,\nwhen introducing any NLP model, to study these kinds of factors and make them\nclear. One way to do this is by releasing a model card (Mitchell et al., 2019) for model card\neach version of a model. A model card documents a machine learning model with\ninformation like:\n\u2022 training algorithms and parameters\n\u2022 training data sources, motivation, and preprocessing\n\u2022 evaluation data sources, motivation, and preprocessing\n\u2022 intended use and users\n\u2022 model performance across different demographic or other groups and envi-\nronmental situations\n4.11 Summary\nThis chapter introduced the naive Bayes model for classi\ufb01cation and applied it to\nthetext categorization task of sentiment analysis .\n\u2022 Many language processing tasks can be viewed as tasks of classi\ufb01cation .\n\u2022 Text categorization, in which an entire text is assigned a class from a \ufb01nite set,\nincludes such tasks as sentiment analysis ,spam detection , language identi-\n\ufb01cation, and authorship attribution.\n\u2022 Sentiment analysis classi\ufb01es a text as re\ufb02ecting the positive or negative orien-\ntation ( sentiment ) that a writer expresses toward some object.\n\u2022 Naive Bayes is a generative model that makes the bag-of-words assumption\n(position doesn\u2019t matter) and the conditional independence assumption (words\nare conditionally independent of each other given the class)\n\u2022 Naive Bayes with binarized features seems to work better for many text clas-\nsi\ufb01cation tasks.\n\u2022 Classi\ufb01ers are evaluated based on precision andrecall .\n\u2022 Classi\ufb01ers are trained using distinct training, dev, and test sets, including the\nuse of cross-validation in the training set. 20 CHAPTER 4 \u2022 N AIVE BAYES , TEXT CLASSIFICATION ,AND SENTIMENT\n\u2022 Statistical signi\ufb01cance tests should be used to determine whether we can be\ncon\ufb01dent that one version of a classi\ufb01er is better than another.\n\u2022 Designers of classi\ufb01ers should carefully consider harms that may be caused\nby the model, including its training data and other components, and report\nmodel characteristics in a model card .\nBibliographical and Historical Notes\nMultinomial naive Bayes text classi\ufb01cation was proposed by Maron (1961) at the\nRAND Corporation for the task of assigning subject categories to journal abstracts.\nHis model introduced most of the features of the modern form presented here, ap-\nproximating the classi\ufb01cation task with one-of categorization, and implementing\nadd-dsmoothing and information-based feature selection.\nThe conditional independence assumptions of naive Bayes and the idea of Bayes-\nian analysis of text seems to have arisen multiple times. The same year as Maron\u2019s\npaper, Minsky (1961) proposed a naive Bayes classi\ufb01er for vision and other arti-\n\ufb01cial intelligence problems, and Bayesian techniques were also applied to the text\nclassi\ufb01cation task of authorship attribution by Mosteller and Wallace (1963). It had\nlong been known that Alexander Hamilton, John Jay, and James Madison wrote\nthe anonymously-published Federalist papers in 1787\u20131788 to persuade New York\nto ratify the United States Constitution. Yet although some of the 85 essays were\nclearly attributable to one author or another, the authorship of 12 were in dispute\nbetween Hamilton and Madison. Mosteller and Wallace (1963) trained a Bayesian\nprobabilistic model of the writing of Hamilton and another model on the writings\nof Madison, then computed the maximum-likelihood author for each of the disputed\nessays. Naive Bayes was \ufb01rst applied to spam detection in Heckerman et al. (1998).\nMetsis et al. (2006), Pang et al. (2002), and Wang and Manning (2012) show\nthat using boolean attributes with multinomial naive Bayes works better than full\ncounts. Binary multinomial naive Bayes is sometimes confused with another variant\nof naive Bayes that also uses a binary representation of whether a term occurs in\na document: Multivariate Bernoulli naive Bayes . The Bernoulli variant instead\nestimates P(wjc)as the fraction of documents that contain a term, and includes a\nprobability for whether a term is notin a document. McCallum and Nigam (1998)\nand Wang and Manning (2012) show that the multivariate Bernoulli variant of naive\nBayes doesn\u2019t work as well as the multinomial algorithm for sentiment or other text\ntasks.\nThere are a variety of sources covering the many kinds of text classi\ufb01cation\ntasks. For sentiment analysis see Pang and Lee (2008), and Liu and Zhang (2012).\nStamatatos (2009) surveys authorship attribute algorithms. On language identi\ufb01ca-\ntion see Jauhiainen et al. (2019); Jaech et al. (2016) is an important early neural\nsystem. The task of newswire indexing was often used as a test case for text classi-\n\ufb01cation algorithms, based on the Reuters-21578 collection of newswire articles.\nSee Manning et al. (2008) and Aggarwal and Zhai (2012) on text classi\ufb01cation;\nclassi\ufb01cation in general is covered in machine learning textbooks (Hastie et al. 2001,\nWitten and Frank 2005, Bishop 2006, Murphy 2012).\nNon-parametric methods for computing statistical signi\ufb01cance were used \ufb01rst in\nNLP in the MUC competition (Chinchor et al., 1993), and even earlier in speech\nrecognition (Gillick and Cox 1989, Bisani and Ney 2004). Our description of the\nbootstrap draws on the description in Berg-Kirkpatrick et al. (2012). Recent work\nhas focused on issues including multiple test sets and multiple metrics (S\u00f8gaard et al. EXERCISES 21\n2014, Dror et al. 2017).\nFeature selection is a method of removing features that are unlikely to generalize\nwell. Features are generally ranked by how informative they are about the classi\ufb01ca-\ntion decision. A very common metric, information gain , tells us how many bits ofinformation\ngain\ninformation the presence of the word gives us for guessing the class. Other feature\nselection metrics include c2, pointwise mutual information, and GINI index; see\nYang and Pedersen (1997) for a comparison and Guyon and Elisseeff (2003) for an\nintroduction to feature selection.\nExercises\n4.1 Assume the following likelihoods for each word being part of a positive or\nnegative movie review, and equal prior probabilities for each class.\npos neg\nI 0.09 0.16\nalways 0.07 0.06\nlike 0.29 0.06\nforeign 0.04 0.15\n\ufb01lms 0.08 0.11\nWhat class will Naive bayes assign to the sentence \u201cI always like foreign\n\ufb01lms.\u201d?\n4.2 Given the following short movie reviews, each labeled with a genre, either\ncomedy or action:\n1. fun, couple, love, love comedy\n2. fast, furious, shoot action\n3. couple, \ufb02y, fast, fun, fun comedy\n4. furious, shoot, shoot, fun action\n5. \ufb02y, fast, shoot, love action\nand a new document D:\nfast, couple, shoot, \ufb02y\ncompute the most likely class for D. Assume a naive Bayes classi\ufb01er and use\nadd-1 smoothing for the likelihoods.\n4.3 Train two models, multinomial naive Bayes and binarized naive Bayes, both\nwith add-1 smoothing, on the following document counts for key sentiment\nwords, with positive or negative class assigned as noted.\ndoc \u201cgood\u201d \u201cpoor\u201d \u201cgreat\u201d (class)\nd1. 3 0 3 pos\nd2. 0 1 2 pos\nd3. 1 3 0 neg\nd4. 1 5 2 neg\nd5. 0 2 0 neg\nUse both naive Bayes models to assign a class (pos or neg) to this sentence:\nA good, good plot and great characters, but poor acting.\nRecall from page 6 that with naive Bayes text classi\ufb01cation, we simply ignore\n(throw out) any word that never occurred in the training document. (We don\u2019t\nthrow out words that appear in some classes but not others; that\u2019s what add-\none smoothing is for.) Do the two models agree or disagree? 22 Chapter 4 \u2022 Naive Bayes, Text Classi\ufb01cation, and Sentiment\nAggarwal, C. C. and C. Zhai. 2012. A survey of text classi-\n\ufb01cation algorithms. In C. C. Aggarwal and C. Zhai, eds,\nMining text data , 163\u2013222. Springer.\nBayes, T. 1763. An Essay Toward Solving a Problem in the\nDoctrine of Chances , volume 53. Reprinted in Facsimiles\nof Two Papers by Bayes , Hafner Publishing, 1963.\nBerg-Kirkpatrick, T., D. Burkett, and D. Klein. 2012. An\nempirical investigation of statistical signi\ufb01cance in NLP.\nEMNLP .\nBisani, M. and H. Ney. 2004. Bootstrap estimates for con\ufb01-\ndence intervals in ASR performance evaluation. ICASSP .\nBishop, C. M. 2006. Pattern recognition and machine learn-\ning. Springer.\nBlodgett, S. L., S. Barocas, H. Daum \u00b4e III, and H. Wallach.\n2020. Language (technology) is power: A critical survey\nof \u201cbias\u201d in NLP. ACL.\nBlodgett, S. L., L. Green, and B. O\u2019Connor. 2016. Demo-\ngraphic dialectal variation in social media: A case study\nof African-American English. EMNLP .\nBorges, J. L. 1964. The analytical language of john wilkins.\nInOther inquisitions 1937\u20131952 . University of Texas\nPress. Trans. Ruth L. C. Simms.\nCaliskan, A., J. J. Bryson, and A. Narayanan. 2017. Seman-\ntics derived automatically from language corpora contain\nhuman-like biases. Science , 356(6334):183\u2013186.\nChinchor, N., L. Hirschman, and D. L. Lewis. 1993. Eval-\nuating Message Understanding systems: An analysis of\nthe third Message Understanding Conference. Computa-\ntional Linguistics , 19(3):409\u2013449.\nCrawford, K. 2017. The trouble with bias. Keynote at\nNeurIPS.\nDavidson, T., D. Bhattacharya, and I. Weber. 2019. Racial\nbias in hate speech and abusive language detection\ndatasets. Third Workshop on Abusive Language Online .\nDias Oliva, T., D. Antonialli, and A. Gomes. 2021. Fighting\nhate speech, silencing drag queens? arti\ufb01cial intelligence\nin content moderation and risks to lgbtq voices online.\nSexuality & Culture , 25:700\u2013732.\nDixon, L., J. Li, J. Sorensen, N. Thain, and L. Vasserman.\n2018. Measuring and mitigating unintended bias in text\nclassi\ufb01cation. 2018 AAAI/ACM Conference on AI, Ethics,\nand Society .\nDror, R., G. Baumer, M. Bogomolov, and R. Reichart. 2017.\nReplicability analysis for natural language processing:\nTesting signi\ufb01cance with multiple datasets. TACL , 5:471\u2013\n\u2013486.\nDror, R., L. Peled-Cohen, S. Shlomov, and R. Reichart.\n2020. Statistical Signi\ufb01cance Testing for Natural Lan-\nguage Processing , volume 45 of Synthesis Lectures on\nHuman Language Technologies . Morgan & Claypool.\nEfron, B. and R. J. Tibshirani. 1993. An introduction to the\nbootstrap . CRC press.\nGillick, L. and S. J. Cox. 1989. Some statistical issues in the\ncomparison of speech recognition algorithms. ICASSP .\nGuyon, I. and A. Elisseeff. 2003. An introduction to variable\nand feature selection. JMLR , 3:1157\u20131182.\nHastie, T., R. J. Tibshirani, and J. H. Friedman. 2001. The\nElements of Statistical Learning . Springer.Heckerman, D., E. Horvitz, M. Sahami, and S. T. Dumais.\n1998. A bayesian approach to \ufb01ltering junk e-mail. AAAI-\n98 Workshop on Learning for Text Categorization .\nHu, M. and B. Liu. 2004. Mining and summarizing customer\nreviews. KDD .\nHutchinson, B., V . Prabhakaran, E. Denton, K. Webster,\nY . Zhong, and S. Denuyl. 2020. Social biases in NLP\nmodels as barriers for persons with disabilities. ACL.\nJaech, A., G. Mulcaire, S. Hathi, M. Ostendorf, and N. A.\nSmith. 2016. Hierarchical character-word models for lan-\nguage identi\ufb01cation. ACL Workshop on NLP for Social\nMedia .\nJauhiainen, T., M. Lui, M. Zampieri, T. Baldwin, and\nK. Lind \u00b4en. 2019. Automatic language identi\ufb01cation in\ntexts: A survey. JAIR , 65(1):675\u2013682.\nJurgens, D., Y . Tsvetkov, and D. Jurafsky. 2017. Incorpo-\nrating dialectal variability for socially equitable language\nidenti\ufb01cation. ACL.\nKiritchenko, S. and S. M. Mohammad. 2018. Examining\ngender and race bias in two hundred sentiment analysis\nsystems. *SEM .\nLiu, B. and L. Zhang. 2012. A survey of opinion mining and\nsentiment analysis. In C. C. Aggarwal and C. Zhai, eds,\nMining text data , 415\u2013464. Springer.\nLui, M. and T. Baldwin. 2011. Cross-domain feature selec-\ntion for language identi\ufb01cation. IJCNLP .\nLui, M. and T. Baldwin. 2012. langid.py : An off-the-shelf\nlanguage identi\ufb01cation tool. ACL.\nManning, C. D., P. Raghavan, and H. Sch \u00a8utze. 2008. Intro-\nduction to Information Retrieval . Cambridge.\nMaron, M. E. 1961. Automatic indexing: an experimental\ninquiry. Journal of the ACM , 8(3):404\u2013417.\nMcCallum, A. and K. Nigam. 1998. A comparison of event\nmodels for naive bayes text classi\ufb01cation. AAAI/ICML-98\nWorkshop on Learning for Text Categorization .\nMetsis, V ., I. Androutsopoulos, and G. Paliouras. 2006.\nSpam \ufb01ltering with naive bayes-which naive bayes?\nCEAS .\nMinsky, M. 1961. Steps toward arti\ufb01cial intelligence. Pro-\nceedings of the IRE , 49(1):8\u201330.\nMitchell, M., S. Wu, A. Zaldivar, P. Barnes, L. Vasserman,\nB. Hutchinson, E. Spitzer, I. D. Raji, and T. Gebru. 2019.\nModel cards for model reporting. ACM FAccT .\nMosteller, F. and D. L. Wallace. 1963. Inference in an au-\nthorship problem: A comparative study of discrimination\nmethods applied to the authorship of the disputed feder-\nalist papers. Journal of the American Statistical Associa-\ntion, 58(302):275\u2013309.\nMosteller, F. and D. L. Wallace. 1964. Inference and Dis-\nputed Authorship: The Federalist . Springer-Verlag. 1984\n2nd edition: Applied Bayesian and Classical Inference .\nMurphy, K. P. 2012. Machine learning: A probabilistic per-\nspective . MIT Press.\nNoreen, E. W. 1989. Computer Intensive Methods for Testing\nHypothesis . Wiley.\nPang, B. and L. Lee. 2008. Opinion mining and sentiment\nanalysis. Foundations and trends in information retrieval ,\n2(1-2):1\u2013135. Exercises 23\nPang, B., L. Lee, and S. Vaithyanathan. 2002. Thumbs\nup? Sentiment classi\ufb01cation using machine learning tech-\nniques. EMNLP .\nPark, J. H., J. Shin, and P. Fung. 2018. Reducing gender bias\nin abusive language detection. EMNLP .\nPennebaker, J. W., R. J. Booth, and M. E. Francis. 2007.\nLinguistic Inquiry and Word Count: LIWC 2007 . Austin,\nTX.\nPopp, D., R. A. Donovan, M. Crawford, K. L. Marsh, and\nM. Peele. 2003. Gender, race, and speech style stereo-\ntypes. Sex Roles , 48(7-8):317\u2013325.\nSahami, M., S. T. Dumais, D. Heckerman, and E. Horvitz.\n1998. A Bayesian approach to \ufb01ltering junk e-mail. AAAI\nWorkshop on Learning for Text Categorization .\nSap, M., D. Card, S. Gabriel, Y . Choi, and N. A. Smith. 2019.\nThe risk of racial bias in hate speech detection. ACL.\nS\u00f8gaard, A., A. Johannsen, B. Plank, D. Hovy, and H. M.\nAlonso. 2014. What\u2019s in a p-value in NLP? CoNLL .\nStamatatos, E. 2009. A survey of modern authorship attribu-\ntion methods. JASIST , 60(3):538\u2013556.\nStone, P., D. Dunphry, M. Smith, and D. Ogilvie. 1966.\nThe General Inquirer: A Computer Approach to Content\nAnalysis . MIT Press.\nvan Rijsbergen, C. J. 1975. Information Retrieval . Butter-\nworths.\nWang, S. and C. D. Manning. 2012. Baselines and bigrams:\nSimple, good sentiment and topic classi\ufb01cation. ACL.\nWilson, T., J. Wiebe, and P. Hoffmann. 2005. Recogniz-\ning contextual polarity in phrase-level sentiment analysis.\nEMNLP .\nWitten, I. H. and E. Frank. 2005. Data Mining: Practi-\ncal Machine Learning Tools and Techniques , 2nd edition.\nMorgan Kaufmann.\nYang, Y . and J. Pedersen. 1997. A comparative study on\nfeature selection in text categorization. ICML .", "nb24aug.pdf": "Text Classification and Naive BayesThe Task of Text Classification Is this spam?\n Who wrote which Federalist Papers?1787-8: essays anonymously written by:     Alexander Hamilton, James Madison, and John Jay to convince New York to ratify U.S Constitution  Authorship of 12 of the letters unclear between:1963: solved by Mosteller and Wallace using Bayesian methods\nJames MadisonAlexander Hamilton Positive or negative movie review?unbelievably disappointing Full of zany characters and richly applied satire, and some great plot twiststhis is the greatest screwball comedy ever filmedIt was pathetic. The worst part about it was the boxing scenes.4\n What is the subject of this article?Antogonists and InhibitorsBlood SupplyChemistryDrug TherapyEmbryologyEpidemiology\u20265MeSH Subject Category Hierarchy?MEDLINE Article\n Text ClassificationAssigning subject categories, topics, or genresSpam detectionAuthorship identification (who wrote this?)Language Identification (is this Portuguese?)Sentiment analysis\u2026 Text Classification: definitionInput:\u25e6 a document d\u25e6 a fixed set of classes  C = {c1, c2,\u2026, cJ}Output: a predicted class c \u00ce C Basic Classification Method: Hand-coded rulesRules based on combinations of words or other features\u25e6 spam: black-list-address OR (\u201cdollars\u201d AND \u201chave been selected\u201d)Accuracy can be high\u2022In very specific domains\u2022If rules are carefully refined by expertsBut:\u2022building and maintaining rules is expensive\u2022they are too literal and specific: \"high-precision, low-recall\" Classification Method:Supervised Machine LearningInput: \u25e6a document d\u25e6 a fixed set of classes  C = {c1, c2,\u2026, cJ}\u25e6A training set of m hand-labeled documents (d1,c1),....,(dm,cm)Output: \u25e6a learned classifier \u03b3:d \u00e0 c\n9 Classification Methods:Supervised Machine LearningMany kinds of classifiers!\u2022Na\u00efve Bayes (this lecture)\u2022Logistic regression  \u2022Neural networks\u2022k-nearest neighbors\u2022\u2026We can also use pretrained large language models!\u2022Fine-tuned as classifiers\u2022Prompted to give a classification Text Classification and Naive BayesThe Naive Bayes Classifier Naive Bayes IntuitionSimple (\"naive\") classification method based on Bayes ruleRelies on very simple representation of document\u25e6Bag of words The Bag of Words Representation\n13ititititititIIII\nIloverecommendmoviethethethetheto\ntotoand\nandandseenseenyetwouldwithwhowhimsical\nwhilewhenevertimessweetseveralscenessatiricalromanticofmanageshumorhavehappyfunfriendfairydialoguebutconventionsareanyoneadventurealwaysagainaboutI love this movie! It's sweet, but with satirical humor. The dialogue is great and the adventure scenes are fun... It manages to be whimsical and romantic while laughing at the conventions of the fairy tale genre. I would recommend it to just about anyone. I've seen it several times, and I'm always happy to see it again whenever I have a friend who hasn't seen it yet!it Ithetoandseenyetwouldwhimsicaltimessweetsatiricaladventuregenrefairyhumorhavegreat\u20266 54332111111111111\u2026ititititititIIII\nIloverecommendmoviethethethetheto\ntotoand\nandandseenseenyetwouldwithwhowhimsical\nwhilewhenevertimessweetseveralscenessatiricalromanticofmanageshumorhavehappyfunfriendfairydialoguebutconventionsareanyoneadventurealwaysagainaboutI love this movie! It's sweet, but with satirical humor. The dialogue is great and the adventure scenes are fun... It manages to be whimsical and romantic while laughing at the conventions of the fairy tale genre. I would recommend it to just about anyone. I've seen it several times, and I'm always happy to see it again whenever I have a friend who hasn't seen it yet!it Ithetoandseenyetwouldwhimsicaltimessweetsatiricaladventuregenrefairyhumorhavegreat\u20266 54332111111111111\u2026ititititititIIII\nIloverecommendmoviethethethetheto\ntotoand\nandandseenseenyetwouldwithwhowhimsical\nwhilewhenevertimessweetseveralscenessatiricalromanticofmanageshumorhavehappyfunfriendfairydialoguebutconventionsareanyoneadventurealwaysagainaboutI love this movie! It's sweet, but with satirical humor. The dialogue is great and the adventure scenes are fun... It manages to be whimsical and romantic while laughing at the conventions of the fairy tale genre. I would recommend it to just about anyone. I've seen it several times, and I'm always happy to see it again whenever I have a friend who hasn't seen it yet!it Ithetoandseenyetwouldwhimsicaltimessweetsatiricaladventuregenrefairyhumorhavegreat\u20266 54332111111111111\u2026 The bag of words representation\u03b3()=cseen2sweet1whimsical1recommend1happy1......\n Bayes\u2019 Rule Applied to Documents and Classes\u2022For a document d and a class cP(c|d)=P(d|c)P(c)P(d) Naive Bayes Classifier (I)cMAP=argmaxc\u2208CP(c|d)=argmaxc\u2208CP(d|c)P(c)P(d)=argmaxc\u2208CP(d|c)P(c)\nMAP is \u201cmaximum a posteriori\u201d  = most likely class\nBayes Rule\nDropping the denominator Naive Bayes Classifier (II)cMAP=argmaxc\u2208CP(d|c)P(c)\nDocument d represented as features x1..xn=argmaxc\u2208CP(x1,x2,\u2026,xn|c)P(c)\n\"Likelihood\"\n\"Prior\" Na\u00efve Bayes Classifier (IV)\nHow often does this class occur?cMAP=argmaxc\u2208CP(x1,x2,\u2026,xn|c)P(c)\nO(|X|n\u2022|C|) parameters\nWe can just count the relative frequencies in a corpus\nCould only be estimated if a very, very large number of training examples was available. Multinomial Naive Bayes Independence AssumptionsBag of Words assumption: Assume position doesn\u2019t matterConditional Independence: Assume the feature probabilities P(xi|cj) are independent given the class c.P(x1,x2,\u2026,xn|c)P(x1,\u2026,xn|c)=P(x1|c)\u2022P(x2|c)\u2022P(x3|c)\u2022...\u2022P(xn|c) Multinomial Naive Bayes ClassifiercMAP=argmaxc\u2208CP(x1,x2,\u2026,xn|c)P(c)cNB=argmaxc\u2208CP(cj)P(x|c)x\u2208X\u220f Applying Multinomial Naive Bayes Classifiers to Text ClassificationcNB=argmaxcj\u2208CP(cj)P(xi|cj)i\u2208positions\u220fpositions \u00ac all word positions in test document         Problems with multiplying lots of probsThere's a problem with this:Multiplying lots of probabilities can result in floating-point underflow!  .0006 * .0007 * .0009 * .01 * .5 * .000008\u2026.Idea:   Use logs, because  log(ab) = log(a) + log(b)  We'll sum logs of probabilities instead of multiplying probabilities!cNB=argmaxcj\u2208CP(cj)P(xi|cj)i\u2208positions\u220f We actually do everything in log spaceInstead of this:This:Notes:1) Taking log doesn't change the ranking of classes! The class with highest probability also has highest log probability!2) It's a linear model: Just a max of a sum of weights: a linear function of the inputs So naive bayes is a linear classifier\n<latexit sha1_base64=\"o0LQfSf3I3G0xas3oLJOwQZR0GU=\">AAACoXicbVFdaxQxFM2MH63r16qPggQXoSIsMwWxL0JpfdAHyypuW5gMQyZ7ZzZ2koxJRnaJ+V/+Dt/8N2Z2R6itF0IO597Dvffcsm24sUnyO4pv3Lx1e2f3zujuvfsPHo4fPT41qtMM5kw1Sp+X1EDDJcwttw2ctxqoKBs4Ky+O+/zZd9CGK/nFrlvIBa0lrzijNlDF+CdZQEWorgVdOSKoXarWES3wlvJ+REqouXTwTVKt6dqPWOGIhZV1J0fe47d4UBeOFV8Jl/jYY9JAZbPwqRrP9gL/Er/CxHSicLwvCZ1KtXKtMrwfw3jv/xavCv6jFxDN66XNMZFKdqIETUAuLk1RjCfJNNkEvg7SAUzQELNi/IssFOsESMsaakyWJq3NHdWWswbCnp2BlrILWkMWoKQCTO42Dnv8IjALXCkdnrR4w15WOCqMWYsyVPYemqu5nvxfLutsdZA7LtvOgmTbRlXXYKtwfy684BqYbdYBUKaDWwyzJdWU2XDU3oT06srXwen+NH09TT7tTw6PBjt20VP0HO2hFL1Bh+g9mqE5YtGz6F30MTqJJ/GHeBZ/3pbG0aB5gv6JOPsD0yvRAA==</latexit>cNB= argmaxcj2C24logP(cj)+Xi2positionslogP(xi|cj)35cNB=argmaxcj\u2208CP(cj)P(xi|cj)i\u2208positions\u220f Text Classification and Naive BayesThe Naive Bayes Classifier Text Classification and Na\u00efve BayesNaive Bayes: Learning Learning the Multinomial Naive Bayes ModelFirst attempt: maximum likelihood estimates\u25e6simply use the frequencies in the dataSec.13.3\n\u02c6P(wi|cj)=count(wi,cj)count(w,cj)w\u2208V\u2211!\ud835\udc43\ud835\udc50!=\ud835\udc41\"!\ud835\udc41#$#%& Parameter estimationCreate mega-document for topic j by concatenating all docs in this topic\u25e6Use frequency of w in mega-documentfraction of times word wi appears among all words in documents of topic cj\u02c6P(wi|cj)=count(wi,cj)count(w,cj)w\u2208V\u2211 Problem with Maximum LikelihoodWhat if we have seen no training documents with the word fantastic and classified in the topic positive (thumbs-up)?Zero probabilities cannot be conditioned away, no matter the other evidence!\u02c6P(\"fantastic\" positive) = count(\"fantastic\", positive)count(w,positivew\u2208V\u2211) = 0cMAP=argmaxc\u02c6P(c)\u02c6P(xi|c)i\u220fSec.13.3 Laplace (add-1) smoothing for Na\u00efve Bayes\u02c6P(wi|c)=count(wi,c)+1count(w,c)+1()w\u2208V\u2211=count(wi,c)+1count(w,cw\u2208V\u2211)#$%%&'(( + V\u02c6P(wi|c)=count(wi,c)count(w,c)()w\u2208V\u2211 Multinomial Na\u00efve Bayes: LearningCalculate P(cj) terms\u25e6For each cj in C do docsj \u00ac all docs with  class =cjP(wk|cj)\u2190nk+\u03b1n+\u03b1|Vocabulary|P(cj)\u2190|docsj||total # documents|\u2022Calculate P(wk | cj) terms\u2022Textj \u00ac single doc containing all docsj\u2022For each word wk in Vocabulary    nk \u00ac # of occurrences of wk in Textj\u2022From training corpus, extract Vocabulary Unknown wordsWhat about unknown words\u25e6that appear in our test data \u25e6but not in our training data or vocabulary?We ignore them\u25e6Remove them from the test document!\u25e6Pretend they weren't there!\u25e6Don't include any probability for them at all!Why don't we build an unknown word model?\u25e6It doesn't help: knowing which class has more unknown words is not generally helpful! Stop wordsSome systems ignore stop words\u25e6Stop words: very frequent words like the and a.\u25e6Sort the vocabulary by word frequency in training set\u25e6Call the top 10 or 50 words the stopword list.\u25e6Remove all stop words from both training and test sets\u25e6As if they were never there!But removing stop words doesn't usually help\u2022So in practice most NB algorithms use all words and don't use stopword lists Text Classification and Naive BayesNaive Bayes: Learning Text Classification and Naive BayesSentiment and Binary Naive Bayes Let's do a worked sentiment example!4.3\u2022WORKED EXAMPLE74.3 Worked exampleLet\u2019s walk through an example of training and testing naive Bayes with add-onesmoothing. We\u2019ll use a sentiment analysis domain with the two classes positive(+) and negative (-), and take the following miniature training and test documentssimpli\ufb01ed from actual movie reviews.CatDocumentsTraining-just plain boring-entirely predictable and lacks energy-no surprises and very few laughs+very powerful+the most fun \ufb01lm of the summerTest?predictable with no funThe priorP(c)for the two classes is computed via Eq.4.11asNcNdoc:P(\u0000)=35P(+) =25The wordwithdoesn\u2019t occur in the training set, so we drop it completely (asmentioned above, we don\u2019t use unknown word models for naive Bayes). The like-lihoods from the training set for the remaining three words \u201cpredictable\u201d, \u201cno\u201d, and\u201cfun\u201d, are as follows, from Eq.4.14(computing the probabilities for the remainderof the words in the training set is left as an exercise for the reader):P(\u201cpredictable\u201d|\u0000)=1+114+20P(\u201cpredictable\u201d|+) =0+19+20P(\u201cno\u201d|\u0000)=1+114+20P(\u201cno\u201d|+) =0+19+20P(\u201cfun\u201d|\u0000)=0+114+20P(\u201cfun\u201d|+) =1+19+20For the test sentence S = \u201cpredictable with no fun\u201d, after removing the word \u2018with\u2019,the chosen class, via Eq.4.9, is therefore computed as follows:P(\u0000)P(S|\u0000)=35\u21e52\u21e52\u21e51343=6.1\u21e510\u00005P(+)P(S|+) =25\u21e51\u21e51\u21e52293=3.2\u21e510\u00005The model thus predicts the classnegativefor the test sentence.4.4 Optimizing for Sentiment AnalysisWhile standard naive Bayes text classi\ufb01cation can work well for sentiment analysis,some small changes are generally employed that improve performance.First, for sentiment classi\ufb01cation and a number of other text classi\ufb01cation tasks,whether a word occurs or not seems to matter more than its frequency. Thus itoften improves performance to clip the word counts in each document at 1 (seethe end of the chapter for pointers to these results). This variant is calledbinary A worked sentiment example with add-1 smoothing4.3\u2022WORKED EXAMPLE74.3 Worked exampleLet\u2019s walk through an example of training and testing naive Bayes with add-onesmoothing. We\u2019ll use a sentiment analysis domain with the two classes positive(+) and negative (-), and take the following miniature training and test documentssimpli\ufb01ed from actual movie reviews.CatDocumentsTraining-just plain boring-entirely predictable and lacks energy-no surprises and very few laughs+very powerful+the most fun \ufb01lm of the summerTest?predictable with no funThe priorP(c)for the two classes is computed via Eq.4.11asNcNdoc:P(\u0000)=35P(+) =25The wordwithdoesn\u2019t occur in the training set, so we drop it completely (asmentioned above, we don\u2019t use unknown word models for naive Bayes). The like-lihoods from the training set for the remaining three words \u201cpredictable\u201d, \u201cno\u201d, and\u201cfun\u201d, are as follows, from Eq.4.14(computing the probabilities for the remainderof the words in the training set is left as an exercise for the reader):P(\u201cpredictable\u201d|\u0000)=1+114+20P(\u201cpredictable\u201d|+) =0+19+20P(\u201cno\u201d|\u0000)=1+114+20P(\u201cno\u201d|+) =0+19+20P(\u201cfun\u201d|\u0000)=0+114+20P(\u201cfun\u201d|+) =1+19+20For the test sentence S = \u201cpredictable with no fun\u201d, after removing the word \u2018with\u2019,the chosen class, via Eq.4.9, is therefore computed as follows:P(\u0000)P(S|\u0000)=35\u21e52\u21e52\u21e51343=6.1\u21e510\u00005P(+)P(S|+) =25\u21e51\u21e51\u21e52293=3.2\u21e510\u00005The model thus predicts the classnegativefor the test sentence.4.4 Optimizing for Sentiment AnalysisWhile standard naive Bayes text classi\ufb01cation can work well for sentiment analysis,some small changes are generally employed that improve performance.First, for sentiment classi\ufb01cation and a number of other text classi\ufb01cation tasks,whether a word occurs or not seems to matter more than its frequency. Thus itoften improves performance to clip the word counts in each document at 1 (seethe end of the chapter for pointers to these results). This variant is calledbinary1. Prior from training:P(-) = 3/5P(+) = 2/52. Drop \"with\"4.3\u2022WORKED EXAMPLE74.3 Worked exampleLet\u2019s walk through an example of training and testing naive Bayes with add-onesmoothing. We\u2019ll use a sentiment analysis domain with the two classes positive(+) and negative (-), and take the following miniature training and test documentssimpli\ufb01ed from actual movie reviews.CatDocumentsTraining-just plain boring-entirely predictable and lacks energy-no surprises and very few laughs+very powerful+the most fun \ufb01lm of the summerTest?predictable with no funThe priorP(c)for the two classes is computed via Eq.4.11asNcNdoc:P(\u0000)=35P(+) =25The wordwithdoesn\u2019t occur in the training set, so we drop it completely (asmentioned above, we don\u2019t use unknown word models for naive Bayes). The like-lihoods from the training set for the remaining three words \u201cpredictable\u201d, \u201cno\u201d, and\u201cfun\u201d, are as follows, from Eq.4.14(computing the probabilities for the remainderof the words in the training set is left as an exercise for the reader):P(\u201cpredictable\u201d|\u0000)=1+114+20P(\u201cpredictable\u201d|+) =0+19+20P(\u201cno\u201d|\u0000)=1+114+20P(\u201cno\u201d|+) =0+19+20P(\u201cfun\u201d|\u0000)=0+114+20P(\u201cfun\u201d|+) =1+19+20For the test sentence S = \u201cpredictable with no fun\u201d, after removing the word \u2018with\u2019,the chosen class, via Eq.4.9, is therefore computed as follows:P(\u0000)P(S|\u0000)=35\u21e52\u21e52\u21e51343=6.1\u21e510\u00005P(+)P(S|+) =25\u21e51\u21e51\u21e52293=3.2\u21e510\u00005The model thus predicts the classnegativefor the test sentence.4.4 Optimizing for Sentiment AnalysisWhile standard naive Bayes text classi\ufb01cation can work well for sentiment analysis,some small changes are generally employed that improve performance.First, for sentiment classi\ufb01cation and a number of other text classi\ufb01cation tasks,whether a word occurs or not seems to matter more than its frequency. Thus itoften improves performance to clip the word counts in each document at 1 (seethe end of the chapter for pointers to these results). This variant is calledbinary4.3\u2022WORKED EXAMPLE74.3 Worked exampleLet\u2019s walk through an example of training and testing naive Bayes with add-onesmoothing. We\u2019ll use a sentiment analysis domain with the two classes positive(+) and negative (-), and take the following miniature training and test documentssimpli\ufb01ed from actual movie reviews.CatDocumentsTraining-just plain boring-entirely predictable and lacks energy-no surprises and very few laughs+very powerful+the most fun \ufb01lm of the summerTest?predictable with no funThe priorP(c)for the two classes is computed via Eq.4.11asNcNdoc:P(\u0000)=35P(+) =25The wordwithdoesn\u2019t occur in the training set, so we drop it completely (asmentioned above, we don\u2019t use unknown word models for naive Bayes). The like-lihoods from the training set for the remaining three words \u201cpredictable\u201d, \u201cno\u201d, and\u201cfun\u201d, are as follows, from Eq.4.14(computing the probabilities for the remainderof the words in the training set is left as an exercise for the reader):P(\u201cpredictable\u201d|\u0000)=1+114+20P(\u201cpredictable\u201d|+) =0+19+20P(\u201cno\u201d|\u0000)=1+114+20P(\u201cno\u201d|+) =0+19+20P(\u201cfun\u201d|\u0000)=0+114+20P(\u201cfun\u201d|+) =1+19+20For the test sentence S = \u201cpredictable with no fun\u201d, after removing the word \u2018with\u2019,the chosen class, via Eq.4.9, is therefore computed as follows:P(\u0000)P(S|\u0000)=35\u21e52\u21e52\u21e51343=6.1\u21e510\u00005P(+)P(S|+) =25\u21e51\u21e51\u21e52293=3.2\u21e510\u00005The model thus predicts the classnegativefor the test sentence.4.4 Optimizing for Sentiment AnalysisWhile standard naive Bayes text classi\ufb01cation can work well for sentiment analysis,some small changes are generally employed that improve performance.First, for sentiment classi\ufb01cation and a number of other text classi\ufb01cation tasks,whether a word occurs or not seems to matter more than its frequency. Thus itoften improves performance to clip the word counts in each document at 1 (seethe end of the chapter for pointers to these results). This variant is calledbinary3. Likelihoods from training:4. Scoring the test set:\ud835\udc5d\ud835\udc64!\ud835\udc50=\ud835\udc50\ud835\udc5c\ud835\udc62\ud835\udc5b\ud835\udc61\ud835\udc64!,\ud835\udc50+1\u2211\"\u2208$\ud835\udc50\ud835\udc5c\ud835\udc62\ud835\udc5b\ud835\udc61\ud835\udc64,\ud835\udc50+|\ud835\udc49|/\ud835\udc43\ud835\udc50%=\ud835\udc41&!\ud835\udc41'(')* Optimizing for sentiment analysisFor tasks like sentiment, word occurrence seems to be more important than word frequency.\u25e6The occurrence of the word fantastic tells us a lot\u25e6The fact that it occurs 5 times may not tell us much more.Binary multinominal naive bayes, or binary NB\u25e6Clip our word counts at 1\u25e6Note: this is different than Bernoulli naive bayes; see the textbook at the end of the chapter. Binary Multinomial Na\u00efve Bayes: LearningCalculate P(cj) terms\u25e6For each cj in C do docsj \u00ac all docs with  class =cjP(cj)\u2190|docsj||total # documents|P(wk|cj)\u2190nk+\u03b1n+\u03b1|Vocabulary|\u2022Textj \u00ac single doc containing all docsj\u2022For each word wk in Vocabulary    nk \u00ac # of occurrences of wk in Textj\u2022From training corpus, extract Vocabulary\u2022Calculate P(wk | cj) terms\u2022Remove duplicates in each doc:\u2022For each word type w in docj  \u2022Retain only a single instance of w Binary Multinomial Naive Bayes on a test document d\n39First remove all duplicate words from dThen compute NB using the same equation: cNB=argmaxcj\u2208CP(cj)P(wi|cj)i\u2208positions\u220f Binary multinominal naive Bayes8CHAPTER4\u2022NAIVEBAYES ANDSENTIMENTCLASSIFICATIONmultinomial naive Bayesorbinary NB. The variant uses the same Eq.4.10exceptbinary NBthat for each document we remove all duplicate words before concatenating theminto the single big document. Fig.4.3shows an example in which a set of fourdocuments (shortened and text-normalized for this example) are remapped to binary,with the modi\ufb01ed counts shown in the table on the right. The example is workedwithout add-1 smoothing to make the differences clearer. Note that the results countsneed not be 1; the wordgreathas a count of 2 even for Binary NB, because it appearsin multiple documents.Four original documents:\u0000it was pathetic the worst part was theboxing scenes\u0000no plot twists or great scenes+and satire and great plot twists+great scenes great \ufb01lmAfter per-document binarization:\u0000it was pathetic the worst part boxingscenes\u0000no plot twists or great scenes+and satire great plot twists+great scenes \ufb01lmNB BinaryCounts Counts+\u0000+\u0000and2010boxing 0 1 0 1\ufb01lm 1 0 1 0great3121it 0 1 0 1no 0 1 0 1or 0 1 0 1part 0 1 0 1pathetic 0 1 0 1plot 1 1 1 1satire 1 0 1 0scenes 1 2 1 2the0201twists 1 1 1 1was0201worst 0 1 0 1Figure 4.3An example of binarization for the binary naive Bayes algorithm.A second important addition commonly made when doing text classi\ufb01cation forsentiment is to deal with negation. Consider the difference betweenI really like thismovie(positive) andI didn\u2019t like this movie(negative). The negation expressed bydidn\u2019tcompletely alters the inferences we draw from the predicatelike. Similarly,negation can modify a negative word to produce a positive review (don\u2019t dismiss this\ufb01lm,doesn\u2019t let us get bored).A very simple baseline that is commonly used in sentiment analysis to deal withnegation is the following: during text normalization, prepend the pre\ufb01xNOTtoevery word after a token of logical negation (n\u2019t, not, no, never) until the next punc-tuation mark. Thus the phrasedidn\u2019t like this movie , but Ibecomesdidn\u2019t NOT_like NOT_this NOT_movie , but INewly formed \u2018words\u2019 likeNOTlike,NOTrecommendwill thus occur more of-ten in negative document and act as cues for negative sentiment, while words likeNOTbored,NOTdismisswill acquire positive associations. We will return in Chap-ter 16 to the use of parsing to deal more accurately with the scope relationship be-tween these negation words and the predicates they modify, but this simple baselineworks quite well in practice.Finally, in some situations we might have insuf\ufb01cient labeled training data totrain accurate naive Bayes classi\ufb01ers using all words in the training set to estimatepositive and negative sentiment. In such cases we can instead derive the positive Binary multinominal naive Bayes8CHAPTER4\u2022NAIVEBAYES ANDSENTIMENTCLASSIFICATIONmultinomial naive Bayesorbinary NB. The variant uses the same Eq.4.10exceptbinary NBthat for each document we remove all duplicate words before concatenating theminto the single big document. Fig.4.3shows an example in which a set of fourdocuments (shortened and text-normalized for this example) are remapped to binary,with the modi\ufb01ed counts shown in the table on the right. The example is workedwithout add-1 smoothing to make the differences clearer. Note that the results countsneed not be 1; the wordgreathas a count of 2 even for Binary NB, because it appearsin multiple documents.Four original documents:\u0000it was pathetic the worst part was theboxing scenes\u0000no plot twists or great scenes+and satire and great plot twists+great scenes great \ufb01lmAfter per-document binarization:\u0000it was pathetic the worst part boxingscenes\u0000no plot twists or great scenes+and satire great plot twists+great scenes \ufb01lmNB BinaryCounts Counts+\u0000+\u0000and2010boxing 0 1 0 1\ufb01lm 1 0 1 0great3121it 0 1 0 1no 0 1 0 1or 0 1 0 1part 0 1 0 1pathetic 0 1 0 1plot 1 1 1 1satire 1 0 1 0scenes 1 2 1 2the0201twists 1 1 1 1was0201worst 0 1 0 1Figure 4.3An example of binarization for the binary naive Bayes algorithm.A second important addition commonly made when doing text classi\ufb01cation forsentiment is to deal with negation. Consider the difference betweenI really like thismovie(positive) andI didn\u2019t like this movie(negative). The negation expressed bydidn\u2019tcompletely alters the inferences we draw from the predicatelike. Similarly,negation can modify a negative word to produce a positive review (don\u2019t dismiss this\ufb01lm,doesn\u2019t let us get bored).A very simple baseline that is commonly used in sentiment analysis to deal withnegation is the following: during text normalization, prepend the pre\ufb01xNOTtoevery word after a token of logical negation (n\u2019t, not, no, never) until the next punc-tuation mark. Thus the phrasedidn\u2019t like this movie , but Ibecomesdidn\u2019t NOT_like NOT_this NOT_movie , but INewly formed \u2018words\u2019 likeNOTlike,NOTrecommendwill thus occur more of-ten in negative document and act as cues for negative sentiment, while words likeNOTbored,NOTdismisswill acquire positive associations. We will return in Chap-ter 16 to the use of parsing to deal more accurately with the scope relationship be-tween these negation words and the predicates they modify, but this simple baselineworks quite well in practice.Finally, in some situations we might have insuf\ufb01cient labeled training data totrain accurate naive Bayes classi\ufb01ers using all words in the training set to estimatepositive and negative sentiment. In such cases we can instead derive the positive Binary multinominal naive Bayes8CHAPTER4\u2022NAIVEBAYES ANDSENTIMENTCLASSIFICATIONmultinomial naive Bayesorbinary NB. The variant uses the same Eq.4.10exceptbinary NBthat for each document we remove all duplicate words before concatenating theminto the single big document. Fig.4.3shows an example in which a set of fourdocuments (shortened and text-normalized for this example) are remapped to binary,with the modi\ufb01ed counts shown in the table on the right. The example is workedwithout add-1 smoothing to make the differences clearer. Note that the results countsneed not be 1; the wordgreathas a count of 2 even for Binary NB, because it appearsin multiple documents.Four original documents:\u0000it was pathetic the worst part was theboxing scenes\u0000no plot twists or great scenes+and satire and great plot twists+great scenes great \ufb01lmAfter per-document binarization:\u0000it was pathetic the worst part boxingscenes\u0000no plot twists or great scenes+and satire great plot twists+great scenes \ufb01lmNB BinaryCounts Counts+\u0000+\u0000and2010boxing 0 1 0 1\ufb01lm 1 0 1 0great3121it 0 1 0 1no 0 1 0 1or 0 1 0 1part 0 1 0 1pathetic 0 1 0 1plot 1 1 1 1satire 1 0 1 0scenes 1 2 1 2the0201twists 1 1 1 1was0201worst 0 1 0 1Figure 4.3An example of binarization for the binary naive Bayes algorithm.A second important addition commonly made when doing text classi\ufb01cation forsentiment is to deal with negation. Consider the difference betweenI really like thismovie(positive) andI didn\u2019t like this movie(negative). The negation expressed bydidn\u2019tcompletely alters the inferences we draw from the predicatelike. Similarly,negation can modify a negative word to produce a positive review (don\u2019t dismiss this\ufb01lm,doesn\u2019t let us get bored).A very simple baseline that is commonly used in sentiment analysis to deal withnegation is the following: during text normalization, prepend the pre\ufb01xNOTtoevery word after a token of logical negation (n\u2019t, not, no, never) until the next punc-tuation mark. Thus the phrasedidn\u2019t like this movie , but Ibecomesdidn\u2019t NOT_like NOT_this NOT_movie , but INewly formed \u2018words\u2019 likeNOTlike,NOTrecommendwill thus occur more of-ten in negative document and act as cues for negative sentiment, while words likeNOTbored,NOTdismisswill acquire positive associations. We will return in Chap-ter 16 to the use of parsing to deal more accurately with the scope relationship be-tween these negation words and the predicates they modify, but this simple baselineworks quite well in practice.Finally, in some situations we might have insuf\ufb01cient labeled training data totrain accurate naive Bayes classi\ufb01ers using all words in the training set to estimatepositive and negative sentiment. In such cases we can instead derive the positive Binary multinominal naive Bayes8CHAPTER4\u2022NAIVEBAYES ANDSENTIMENTCLASSIFICATIONmultinomial naive Bayesorbinary NB. The variant uses the same Eq.4.10exceptbinary NBthat for each document we remove all duplicate words before concatenating theminto the single big document. Fig.4.3shows an example in which a set of fourdocuments (shortened and text-normalized for this example) are remapped to binary,with the modi\ufb01ed counts shown in the table on the right. The example is workedwithout add-1 smoothing to make the differences clearer. Note that the results countsneed not be 1; the wordgreathas a count of 2 even for Binary NB, because it appearsin multiple documents.Four original documents:\u0000it was pathetic the worst part was theboxing scenes\u0000no plot twists or great scenes+and satire and great plot twists+great scenes great \ufb01lmAfter per-document binarization:\u0000it was pathetic the worst part boxingscenes\u0000no plot twists or great scenes+and satire great plot twists+great scenes \ufb01lmNB BinaryCounts Counts+\u0000+\u0000and2010boxing 0 1 0 1\ufb01lm 1 0 1 0great3121it 0 1 0 1no 0 1 0 1or 0 1 0 1part 0 1 0 1pathetic 0 1 0 1plot 1 1 1 1satire 1 0 1 0scenes 1 2 1 2the0201twists 1 1 1 1was0201worst 0 1 0 1Figure 4.3An example of binarization for the binary naive Bayes algorithm.A second important addition commonly made when doing text classi\ufb01cation forsentiment is to deal with negation. Consider the difference betweenI really like thismovie(positive) andI didn\u2019t like this movie(negative). The negation expressed bydidn\u2019tcompletely alters the inferences we draw from the predicatelike. Similarly,negation can modify a negative word to produce a positive review (don\u2019t dismiss this\ufb01lm,doesn\u2019t let us get bored).A very simple baseline that is commonly used in sentiment analysis to deal withnegation is the following: during text normalization, prepend the pre\ufb01xNOTtoevery word after a token of logical negation (n\u2019t, not, no, never) until the next punc-tuation mark. Thus the phrasedidn\u2019t like this movie , but Ibecomesdidn\u2019t NOT_like NOT_this NOT_movie , but INewly formed \u2018words\u2019 likeNOTlike,NOTrecommendwill thus occur more of-ten in negative document and act as cues for negative sentiment, while words likeNOTbored,NOTdismisswill acquire positive associations. We will return in Chap-ter 16 to the use of parsing to deal more accurately with the scope relationship be-tween these negation words and the predicates they modify, but this simple baselineworks quite well in practice.Finally, in some situations we might have insuf\ufb01cient labeled training data totrain accurate naive Bayes classi\ufb01ers using all words in the training set to estimatepositive and negative sentiment. In such cases we can instead derive the positiveCounts can still be 2! Binarization is within-doc! Text Classification and Naive BayesSentiment and Binary Naive Bayes Text Classification and Naive BayesMore on Sentiment Classification Sentiment Classification: Dealing with NegationI really like this movieI really don't like this movieNegation changes the meaning of \"like\" to negative.Negation can also change negative to positive-ish \u25e6Don't dismiss this film\u25e6Doesn't let us get bored Sentiment Classification: Dealing with NegationSimple baseline method:Add NOT_ to every word between negation and following punctuation:didn\u2019t like this movie , but Ididn\u2019t NOT_like NOT_this NOT_movie but I\nDas, Sanjiv and Mike Chen. 2001. Yahoo! for Amazon: Extracting market sentiment from stock message boards. In Proceedings of the Asia Pacific Finance Association Annual Conference (APFA).Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan.  2002.  Thumbs up? Sentiment Classification using Machine Learning Techniques. EMNLP-2002, 79\u201486. Sentiment Classification: LexiconsSometimes we don't have enough labeled training dataIn that case, we can make use of pre-built word listsCalled lexiconsThere are various publically available lexicons MPQA Subjectivity Cues LexiconHome page: https://mpqa.cs.pitt.edu/lexicons/subj_lexicon/6885 words from 8221 lemmas, annotated for intensity (strong/weak)\u25e62718 positive\u25e64912 negative+ : admirable, beautiful, confident, dazzling, ecstatic, favor, glee, great \u2212 : awful, bad, bias, catastrophe, cheat, deny, envious, foul, harsh, hate 49TheresaWilson, JanyceWiebe, and Paul Hoffmann (2005). RecognizingContextualPolarityin Phrase-Level SentimentAnalysis. Proc. of HLT-EMNLP-2005.Riloff and Wiebe (2003). Learning extraction patterns for subjective expressions. EMNLP-2003. The General Inquirer\u25e6Home page: http://www.wjh.harvard.edu/~inquirer\u25e6List of Categories:  http://www.wjh.harvard.edu/~inquirer/homecat.htm\u25e6Spreadsheet: http://www.wjh.harvard.edu/~inquirer/inquirerbasic.xlsCategories:\u25e6Positiv (1915 words) and Negativ (2291 words)\u25e6Strong vs Weak, Active vs Passive, Overstated versus Understated\u25e6Pleasure, Pain, Virtue, Vice, Motivation, Cognitive Orientation, etcFree for Research UsePhilip J. Stone, Dexter C Dunphy, Marshall S. Smith, Daniel M. Ogilvie. 1966. The General Inquirer: A Computer Approach to Content Analysis. MIT Press Using Lexicons in Sentiment ClassificationAdd a feature that gets a count whenever a word from the lexicon occurs\u25e6E.g., a feature called \"this word occurs in the positive lexicon\" or \"this word occurs in the negative lexicon\"Now all positive words (good, great, beautiful, wonderful) or negative words count for that feature.Using 1-2 features isn't as good as using all the words.\u2022But when training data is sparse or not representative of the test set, dense lexicon features can help Naive Bayes in Other tasks: Spam FilteringSpamAssassin Features:\u25e6Mentions millions of (dollar) ((dollar) NN,NNN,NNN.NN)\u25e6From: starts with many numbers\u25e6Subject is all capitals\u25e6HTML has a low ratio of text to image area\u25e6\"One hundred percent guaranteed\"\u25e6Claims you can be removed from the list Naive Bayes in Language IDDetermining what language a piece of text is written in.Features based on character n-grams do very wellImportant to train on lots of varieties of each language(e.g., American English varieties like African-American English, or English varieties around the world like Indian English) Summary: Naive Bayes is Not So NaiveVery Fast, low storage requirementsWork well with very small amounts of training dataRobust to Irrelevant Features Irrelevant Features cancel each other without affecting resultsVery good in domains with many equally important features Decision Trees suffer from fragmentation in such cases \u2013 especially if little dataOptimal if the independence assumptions hold: If assumed independence is correct, then it is the Bayes Optimal Classifier for problemA good dependable baseline for text classification\u25e6But we will see other classifiers that give better accuracySlide from Chris Manning Text Classification and Naive BayesMore on Sentiment Classification Text Classification and Na\u00efve BayesNa\u00efve Bayes: Relationship to Language Modeling Dan JurafskyGenerative Model for Multinomial Na\u00efve Bayes\n57c=ChinaX1=ShanghaiX2=andX3=ShenzhenX4=issueX5=bonds Dan JurafskyNa\u00efve Bayes and Language Modeling\u2022Na\u00efve bayes classifiers can use any sort of feature\u2022URL, email address, dictionaries, network features\u2022But if, as in the previous slides\u2022We use only word features \u2022we use all of the words in the text (not a subset)\u2022Then \u2022Na\u00efve bayes has an important similarity to language modeling.58 Dan JurafskyEach class = a unigram language model\u2022Assigning each word: P(word | c)\u2022Assigning each sentence: P(s|c)=\u03a0 P(word|c)0.1 I0.1 love0.01 this0.05 fun0.1 film\u2026Ilovethisfunfilm0.10.1.050.010.1Class posP(s | pos) = 0.0000005 Sec.13.2.1 Dan JurafskyNa\u00efve Bayes as a Language Model\u2022Which class assigns the higher probability to s?0.1 I0.1 love0.01 this0.05 fun0.1 filmModel posModel negfilmlovethisfunI0.10.10.010.050.10.10.0010.010.0050.2P(s|pos)  >  P(s|neg)0.2 I0.001 love0.01 this0.005 fun0.1 filmSec.13.2.1 Text Classification and Na\u00efve BayesNa\u00efve Bayes: Relationship to Language Modeling Text Classification and Naive BayesPrecision, Recall, and F1 Evaluating Classifiers: How well does our classifier work?Let's first address binary classifiers:\u2022Is this email spam? spam (+)     or   not spam (-)\u2022Is this post about Delicious Pie Company? about Del. Pie Co (+)   or    not about Del. Pie Co(-)We'll need to know1.What did our classifier say about each email or post?2.What should our classifier have said, i.e.,  the correct answer, usually as defined by humans (\"gold label\") First step in evaluation: The confusion matrixtrue positivefalse negativefalse positivetrue negativegold positivegold negativesystempositivesystemnegativegold standard labelssystemoutputlabelsrecall = tptp+fnprecision = tptp+fpaccuracy = tp+tntp+fp+tn+fn Accuracy on the confusion matrixtrue positivefalse negativefalse positivetrue negativegold positivegold negativesystempositivesystemnegativegold standard labelssystemoutputlabelsrecall = tptp+fnprecision = tptp+fpaccuracy = tp+tntp+fp+tn+fn Why don't we use accuracy?Accuracy doesn't work well when we're dealing with uncommon or imbalanced classesSuppose we look at 1,000,000 social media posts to find Delicious Pie-lovers (or haters)\u2022100 of them talk about our pie\u2022999,900 are posts about something unrelatedImagine the following simple classifier Every post is \"not about pie\" Accuracy re: pie poststrue positivefalse negativefalse positivetrue negativegold positivegold negativesystempositivesystemnegativegold standard labelssystemoutputlabelsrecall = tptp+fnprecision = tptp+fpaccuracy = tp+tntp+fp+tn+fn100 posts are about pie; 999,900 aren't Why don't we use accuracy?Accuracy of our \"nothing is pie\" classifier 999,900 true negatives  and 100 false negatives Accuracy is 999,900/1,000,000 = 99.99%! But useless at finding pie-lovers (or haters)!! Which was our goal!Accuracy doesn't work well for unbalanced classes  Most tweets are not about pie! Instead of accuracy we use precision and recalltrue positivefalse negativefalse positivetrue negativegold positivegold negativesystempositivesystemnegativegold standard labelssystemoutputlabelsrecall = tptp+fnprecision = tptp+fpaccuracy = tp+tntp+fp+tn+fnPrecision: % of selected items that are correctRecall: % of correct items that are selected Precision/Recall aren't fooled by the\"just call everything negative\" classifier!Stupid classifier: Just say no: every tweet is \"not about pie\"\u2022100 tweets  talk about pie,   999,900 tweets don't\u2022Accuracy = 999,900/1,000,000 = 99.99%But the Recall and Precision for this classifier are terrible:12CHAPTER4\u2022NAIVEBAYES,TEXTCLASSIFICATION,ANDSENTIMENTwhile the other 999,900 are tweets about something completely unrelated. Imagine asimple classi\ufb01er that stupidly classi\ufb01ed every tweet as \u201cnot about pie\u201d. This classi\ufb01erwould have 999,900 true negatives and only 100 false negatives for an accuracy of999,900/1,000,000 or 99.99%! What an amazing accuracy level! Surely we shouldbe happy with this classi\ufb01er? But of course this fabulous \u2018no pie\u2019 classi\ufb01er wouldbe completely useless, since it wouldn\u2019t \ufb01nd a single one of the customer commentswe are looking for. In other words, accuracy is not a good metric when the goal isto discover something that is rare, or at least not completely balanced in frequency,which is a very common situation in the world.That\u2019s why instead of accuracy we generally turn to two other metrics shown inFig.4.4:precisionandrecall.Precisionmeasures the percentage of the items thatprecisionthe system detected (i.e., the system labeled as positive) that are in fact positive (i.e.,are positive according to the human gold labels). Precision is de\ufb01ned asPrecision=true positivestrue positives + false positivesRecallmeasures the percentage of items actually present in the input that wererecallcorrectly identi\ufb01ed by the system. Recall is de\ufb01ned asRecall=true positivestrue positives + false negativesPrecision and recall will help solve the problem with the useless \u201cnothing ispie\u201d classi\ufb01er. This classi\ufb01er, despite having a fabulous accuracy of 99.99%, hasa terrible recall of 0 (since there are no true positives, and 100 false negatives, therecall is 0/100). You should convince yourself that the precision at \ufb01nding relevanttweets is equally problematic. Thus precision and recall, unlike accuracy, emphasizetrue positives: \ufb01nding the things that we are supposed to be looking for.There are many ways to de\ufb01ne a single metric that incorporates aspects of bothprecision and recall. The simplest of these combinations is theF-measure(vanF-measureRijsbergen,1975) , de\ufb01ned as:Fb=(b2+1)PRb2P+RThebparameter differentially weights the importance of recall and precision,based perhaps on the needs of an application. Values ofb>1 favor recall, whilevalues ofb<1 favor precision. Whenb=1, precision and recall are equally bal-anced; this is the most frequently used metric, and is called Fb=1or just F1:F1F1=2PRP+R(4.16)F-measure comes from a weighted harmonic mean of precision and recall. Theharmonic mean of a set of numbers is the reciprocal of the arithmetic mean of recip-rocals:HarmonicMean(a1,a2,a3,a4,...,an)=n1a1+1a2+1a3+...+1an(4.17)and hence F-measure isF=1a1P+(1\u0000a)1Ror\u2713withb2=1\u0000aa\u25c6F=(b2+1)PRb2P+R(4.18)12CHAPTER4\u2022NAIVEBAYES,TEXTCLASSIFICATION,ANDSENTIMENTwhile the other 999,900 are tweets about something completely unrelated. Imagine asimple classi\ufb01er that stupidly classi\ufb01ed every tweet as \u201cnot about pie\u201d. This classi\ufb01erwould have 999,900 true negatives and only 100 false negatives for an accuracy of999,900/1,000,000 or 99.99%! What an amazing accuracy level! Surely we shouldbe happy with this classi\ufb01er? But of course this fabulous \u2018no pie\u2019 classi\ufb01er wouldbe completely useless, since it wouldn\u2019t \ufb01nd a single one of the customer commentswe are looking for. In other words, accuracy is not a good metric when the goal isto discover something that is rare, or at least not completely balanced in frequency,which is a very common situation in the world.That\u2019s why instead of accuracy we generally turn to two other metrics shown inFig.4.4:precisionandrecall.Precisionmeasures the percentage of the items thatprecisionthe system detected (i.e., the system labeled as positive) that are in fact positive (i.e.,are positive according to the human gold labels). Precision is de\ufb01ned asPrecision=true positivestrue positives + false positivesRecallmeasures the percentage of items actually present in the input that wererecallcorrectly identi\ufb01ed by the system. Recall is de\ufb01ned asRecall=true positivestrue positives + false negativesPrecision and recall will help solve the problem with the useless \u201cnothing ispie\u201d classi\ufb01er. This classi\ufb01er, despite having a fabulous accuracy of 99.99%, hasa terrible recall of 0 (since there are no true positives, and 100 false negatives, therecall is 0/100). You should convince yourself that the precision at \ufb01nding relevanttweets is equally problematic. Thus precision and recall, unlike accuracy, emphasizetrue positives: \ufb01nding the things that we are supposed to be looking for.There are many ways to de\ufb01ne a single metric that incorporates aspects of bothprecision and recall. The simplest of these combinations is theF-measure(vanF-measureRijsbergen,1975) , de\ufb01ned as:Fb=(b2+1)PRb2P+RThebparameter differentially weights the importance of recall and precision,based perhaps on the needs of an application. Values ofb>1 favor recall, whilevalues ofb<1 favor precision. Whenb=1, precision and recall are equally bal-anced; this is the most frequently used metric, and is called Fb=1or just F1:F1F1=2PRP+R(4.16)F-measure comes from a weighted harmonic mean of precision and recall. Theharmonic mean of a set of numbers is the reciprocal of the arithmetic mean of recip-rocals:HarmonicMean(a1,a2,a3,a4,...,an)=n1a1+1a2+1a3+...+1an(4.17)and hence F-measure isF=1a1P+(1\u0000a)1Ror\u2713withb2=1\u0000aa\u25c6F=(b2+1)PRb2P+R(4.18) A combined measure: F1F1 is a  combination of precision and recall.12CHAPTER4\u2022NAIVEBAYES,TEXTCLASSIFICATION,ANDSENTIMENTwhile the other 999,900 are tweets about something completely unrelated. Imagine asimple classi\ufb01er that stupidly classi\ufb01ed every tweet as \u201cnot about pie\u201d. This classi\ufb01erwould have 999,900 true negatives and only 100 false negatives for an accuracy of999,900/1,000,000 or 99.99%! What an amazing accuracy level! Surely we shouldbe happy with this classi\ufb01er? But of course this fabulous \u2018no pie\u2019 classi\ufb01er wouldbe completely useless, since it wouldn\u2019t \ufb01nd a single one of the customer commentswe are looking for. In other words, accuracy is not a good metric when the goal isto discover something that is rare, or at least not completely balanced in frequency,which is a very common situation in the world.That\u2019s why instead of accuracy we generally turn to two other metrics shown inFig.4.4:precisionandrecall.Precisionmeasures the percentage of the items thatprecisionthe system detected (i.e., the system labeled as positive) that are in fact positive (i.e.,are positive according to the human gold labels). Precision is de\ufb01ned asPrecision=true positivestrue positives + false positivesRecallmeasures the percentage of items actually present in the input that wererecallcorrectly identi\ufb01ed by the system. Recall is de\ufb01ned asRecall=true positivestrue positives + false negativesPrecision and recall will help solve the problem with the useless \u201cnothing ispie\u201d classi\ufb01er. This classi\ufb01er, despite having a fabulous accuracy of 99.99%, hasa terrible recall of 0 (since there are no true positives, and 100 false negatives, therecall is 0/100). You should convince yourself that the precision at \ufb01nding relevanttweets is equally problematic. Thus precision and recall, unlike accuracy, emphasizetrue positives: \ufb01nding the things that we are supposed to be looking for.There are many ways to de\ufb01ne a single metric that incorporates aspects of bothprecision and recall. The simplest of these combinations is theF-measure(vanF-measureRijsbergen,1975) , de\ufb01ned as:Fb=(b2+1)PRb2P+RThebparameter differentially weights the importance of recall and precision,based perhaps on the needs of an application. Values ofb>1 favor recall, whilevalues ofb<1 favor precision. Whenb=1, precision and recall are equally bal-anced; this is the most frequently used metric, and is called Fb=1or just F1:F1F1=2PRP+R(4.16)F-measure comes from a weighted harmonic mean of precision and recall. Theharmonic mean of a set of numbers is the reciprocal of the arithmetic mean of recip-rocals:HarmonicMean(a1,a2,a3,a4,...,an)=n1a1+1a2+1a3+...+1an(4.17)and hence F-measure isF=1a1P+(1\u0000a)1Ror\u2713withb2=1\u0000aa\u25c6F=(b2+1)PRb2P+R(4.18) F1 is a special case of the general \"F-measure\"F-measure is the (weighted) harmonic mean of precision and recallF1 is a special case of F-measure with \u03b2=1, \u03b1=\u00bd12CHAPTER4\u2022NAIVEBAYES,TEXTCLASSIFICATION,ANDSENTIMENTwhile the other 999,900 are tweets about something completely unrelated. Imagine asimple classi\ufb01er that stupidly classi\ufb01ed every tweet as \u201cnot about pie\u201d. This classi\ufb01erwould have 999,900 true negatives and only 100 false negatives for an accuracy of999,900/1,000,000 or 99.99%! What an amazing accuracy level! Surely we shouldbe happy with this classi\ufb01er? But of course this fabulous \u2018no pie\u2019 classi\ufb01er wouldbe completely useless, since it wouldn\u2019t \ufb01nd a single one of the customer commentswe are looking for. In other words, accuracy is not a good metric when the goal isto discover something that is rare, or at least not completely balanced in frequency,which is a very common situation in the world.That\u2019s why instead of accuracy we generally turn to two other metrics shown inFig.4.4:precisionandrecall.Precisionmeasures the percentage of the items thatprecisionthe system detected (i.e., the system labeled as positive) that are in fact positive (i.e.,are positive according to the human gold labels). Precision is de\ufb01ned asPrecision=true positivestrue positives + false positivesRecallmeasures the percentage of items actually present in the input that wererecallcorrectly identi\ufb01ed by the system. Recall is de\ufb01ned asRecall=true positivestrue positives + false negativesPrecision and recall will help solve the problem with the useless \u201cnothing ispie\u201d classi\ufb01er. This classi\ufb01er, despite having a fabulous accuracy of 99.99%, hasa terrible recall of 0 (since there are no true positives, and 100 false negatives, therecall is 0/100). You should convince yourself that the precision at \ufb01nding relevanttweets is equally problematic. Thus precision and recall, unlike accuracy, emphasizetrue positives: \ufb01nding the things that we are supposed to be looking for.There are many ways to de\ufb01ne a single metric that incorporates aspects of bothprecision and recall. The simplest of these combinations is theF-measure(vanF-measureRijsbergen,1975) , de\ufb01ned as:Fb=(b2+1)PRb2P+RThebparameter differentially weights the importance of recall and precision,based perhaps on the needs of an application. Values ofb>1 favor recall, whilevalues ofb<1 favor precision. Whenb=1, precision and recall are equally bal-anced; this is the most frequently used metric, and is called Fb=1or just F1:F1F1=2PRP+R(4.16)F-measure comes from a weighted harmonic mean of precision and recall. Theharmonic mean of a set of numbers is the reciprocal of the arithmetic mean of recip-rocals:HarmonicMean(a1,a2,a3,a4,...,an)=n1a1+1a2+1a3+...+1an(4.17)and hence F-measure isF=1a1P+(1\u0000a)1Ror\u2713withb2=1\u0000aa\u25c6F=(b2+1)PRb2P+R(4.18)12CHAPTER4\u2022NAIVEBAYES,TEXTCLASSIFICATION,ANDSENTIMENTwhile the other 999,900 are tweets about something completely unrelated. Imagine asimple classi\ufb01er that stupidly classi\ufb01ed every tweet as \u201cnot about pie\u201d. This classi\ufb01erwould have 999,900 true negatives and only 100 false negatives for an accuracy of999,900/1,000,000 or 99.99%! What an amazing accuracy level! Surely we shouldbe happy with this classi\ufb01er? But of course this fabulous \u2018no pie\u2019 classi\ufb01er wouldbe completely useless, since it wouldn\u2019t \ufb01nd a single one of the customer commentswe are looking for. In other words, accuracy is not a good metric when the goal isto discover something that is rare, or at least not completely balanced in frequency,which is a very common situation in the world.That\u2019s why instead of accuracy we generally turn to two other metrics shown inFig.4.4:precisionandrecall.Precisionmeasures the percentage of the items thatprecisionthe system detected (i.e., the system labeled as positive) that are in fact positive (i.e.,are positive according to the human gold labels). Precision is de\ufb01ned asPrecision=true positivestrue positives + false positivesRecallmeasures the percentage of items actually present in the input that wererecallcorrectly identi\ufb01ed by the system. Recall is de\ufb01ned asRecall=true positivestrue positives + false negativesPrecision and recall will help solve the problem with the useless \u201cnothing ispie\u201d classi\ufb01er. This classi\ufb01er, despite having a fabulous accuracy of 99.99%, hasa terrible recall of 0 (since there are no true positives, and 100 false negatives, therecall is 0/100). You should convince yourself that the precision at \ufb01nding relevanttweets is equally problematic. Thus precision and recall, unlike accuracy, emphasizetrue positives: \ufb01nding the things that we are supposed to be looking for.There are many ways to de\ufb01ne a single metric that incorporates aspects of bothprecision and recall. The simplest of these combinations is theF-measure(vanF-measureRijsbergen,1975) , de\ufb01ned as:Fb=(b2+1)PRb2P+RThebparameter differentially weights the importance of recall and precision,based perhaps on the needs of an application. Values ofb>1 favor recall, whilevalues ofb<1 favor precision. Whenb=1, precision and recall are equally bal-anced; this is the most frequently used metric, and is called Fb=1or just F1:F1F1=2PRP+R(4.16)F-measure comes from a weighted harmonic mean of precision and recall. Theharmonic mean of a set of numbers is the reciprocal of the arithmetic mean of recip-rocals:HarmonicMean(a1,a2,a3,a4,...,an)=n1a1+1a2+1a3+...+1an(4.17)and hence F-measure isF=1a1P+(1\u0000a)1Ror\u2713withb2=1\u0000aa\u25c6F=(b2+1)PRb2P+R(4.18) Suppose we have more than 2 classes?Lots of text classification tasks have more than two classes.\u25e6Sentiment analysis (positive, negative, neutral) , named entities (person, location, organization)We can define precision and recall for multiple classes like this 3-way email task:851060urgentnormalgold labelssystemoutputrecallu = 88+5+3precisionu= 88+10+115030200spamurgentnormalspam3recalln = recalls = precisionn= 605+60+50precisions= 2003+30+2006010+60+302001+50+200 How to combine P/R values for different classes:Microaveraging vs Macroaveraging8811340trueurgenttruenotsystemurgentsystemnot604055212truenormaltruenotsystemnormalsystemnot200513383truespamtruenotsystemspamsystemnot2689999635trueyestruenosystemyessystemnoprecision =8+118= .42precision =200+33200= .86precision =60+5560= .52microaverageprecision268+99268= .73=macroaverageprecision3.42+.52+.86= .60=PooledClass 3: SpamClass 2: NormalClass 1: Urgent Text Classification and Naive BayesPrecision, Recall, and F1 Text Classification and Naive BayesAvoiding Harms in Classification Harms of classificationClassifiers, like any NLP algorithm, can cause harmsThis is true for any classifier, whether Naive Bayes or other algorithms Representational Harms\u2022Harms caused by a system that demeans a social group\u2022Such as by perpetuating negative stereotypes about them. \u2022Kiritchenko and Mohammad 2018 study\u2022Examined 200 sentiment analysis systems on pairs of sentences\u2022Identical except for names:\u2022common African American (Shaniqua) or European American (Stephanie).\u2022Like \"I talked to Shaniqua yesterday\" vs \"I talked to Stephanie yesterday\"\u2022Result: systems assigned lower sentiment and more negative emotion to sentences with African American names\u2022Downstream harm: \u2022Perpetuates stereotypes about African Americans \u2022African Americans treated differently by NLP tools like sentiment (widely used in marketing research, mental health studies, etc.) Harms of Censorship\u2022Toxicity detection is the text classification task of detecting hate speech, abuse, harassment, or other kinds of toxic language.\u2022Widely used in online content moderation\u2022Toxicity classifiers incorrectly flag non-toxic sentences that simply mention minority identities (like the words \"blind\" or \"gay\")\u2022women (Park et al., 2018), \u2022disabled people (Hutchinson et al., 2020) \u2022gay people (Dixon et al., 2018; Oliva et al., 2021)\u2022Downstream harms:\u2022Censorship of speech by disabled people and other groups\u2022Speech by these groups becomes less visible online\u2022Writers might be nudged by these algorithms to avoid these words making people less likely to write about themselves or these groups. Performance Disparities1.Text classifiers perform worse on many languages of the world due to lack of data or labels2.Text classifiers perform worse on varieties of even high-resource languages like English\u2022Example task: language identification, a first step in NLP pipeline (\"Is this post in English or not?\") \u2022English language detection performance worse for writers who are African American (Blodgett and O'Connor 2017) or from India (Jurgens et al., 2017) Harms in text classification\u2022Causes:\u2022Issues in the data; NLP systems amplify biases in training data\u2022Problems in the labels\u2022Problems in the algorithms (like what the model is trained to optimize) \u2022Prevalence: The same problems occur throughout NLP (including large language models)  \u2022Solutions: There are no general mitigations or solutions\u2022But harm mitigation is an active area of research\u2022And there are standard benchmarks and tools that we can use for measuring some of the harms Text Classification and Naive BayesAvoiding Harms in Classification", "5.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\n5Logistic Regression\n\u201cAnd how do you know that these \ufb01ne begonias are not of equal importance?\u201d\nHercule Poirot, in Agatha Christie\u2019s The Mysterious Affair at Styles\nDetective stories are as littered with clues as texts are with words. Yet for the\npoor reader it can be challenging to know how to weigh the author\u2019s clues in order\nto make the crucial classi\ufb01cation task: deciding whodunnit.\nIn this chapter we introduce an algorithm that is admirably suited for discovering\nthe link between features or clues and some particular outcome: logistic regression .logistic\nregression\nIndeed, logistic regression is one of the most important analytic tools in the social\nand natural sciences. In natural language processing, logistic regression is the base-\nline supervised machine learning algorithm for classi\ufb01cation, and also has a very\nclose relationship with neural networks. As we will see in Chapter 7, a neural net-\nwork can be viewed as a series of logistic regression classi\ufb01ers stacked on top of\neach other. Thus the classi\ufb01cation and machine learning techniques introduced here\nwill play an important role throughout the book.\nLogistic regression can be used to classify an observation into one of two classes\n(like \u2018positive sentiment\u2019 and \u2018negative sentiment\u2019), or into one of many classes.\nBecause the mathematics for the two-class case is simpler, we\u2019ll describe this special\ncase of logistic regression \ufb01rst in the next few sections, and then brie\ufb02y summarize\nthe use of multinomial logistic regression for more than two classes in Section 5.3.\nWe\u2019ll introduce the mathematics of logistic regression in the next few sections.\nBut let\u2019s begin with some high-level issues.\nGenerative and Discriminative Classi\ufb01ers: The most important difference be-\ntween naive Bayes and logistic regression is that logistic regression is a discrimina-\ntiveclassi\ufb01er while naive Bayes is a generative classi\ufb01er.\nThese are two very different frameworks for how\nto build a machine learning model. Consider a visual\nmetaphor: imagine we\u2019re trying to distinguish dog\nimages from cat images. A generative model would\nhave the goal of understanding what dogs look like\nand what cats look like. You might literally ask such\na model to \u2018generate\u2019, i.e., draw, a dog. Given a test\nimage, the system then asks whether it\u2019s the cat model or the dog model that better\n\ufb01ts (is less surprised by) the image, and chooses that as its label.\nA discriminative model, by contrast, is only try-\ning to learn to distinguish the classes (perhaps with-\nout learning much about them). So maybe all the\ndogs in the training data are wearing collars and the\ncats aren\u2019t. If that one feature neatly separates the\nclasses, the model is satis\ufb01ed. If you ask such a\nmodel what it knows about cats all it can say is that\nthey don\u2019t wear collars. 2CHAPTER 5 \u2022 L OGISTIC REGRESSION\nMore formally, recall that the naive Bayes assigns a class cto a document dnot\nby directly computing P(cjd)but by computing a likelihood and a prior\n\u02c6c=argmax\nc2Clikelihoodz}|{\nP(djc)priorz}|{\nP(c) (5.1)\nAgenerative model like naive Bayes makes use of this likelihood term, whichgenerative\nmodel\nexpresses how to generate the features of a document if we knew it was of class c .\nBy contrast a discriminative model in this text categorization scenario attemptsdiscriminative\nmodel\ntodirectly compute P(cjd). Perhaps it will learn to assign a high weight to document\nfeatures that directly improve its ability to discriminate between possible classes,\neven if it couldn\u2019t generate an example of one of the classes.\nComponents of a probabilistic machine learning classi\ufb01er: Like naive Bayes,\nlogistic regression is a probabilistic classi\ufb01er that makes use of supervised machine\nlearning. Machine learning classi\ufb01ers require a training corpus of minput/output\npairs(x(i);y(i)). (We\u2019ll use superscripts in parentheses to refer to individual instances\nin the training set\u2014for sentiment classi\ufb01cation each instance might be an individual\ndocument to be classi\ufb01ed.) A machine learning system for classi\ufb01cation then has\nfour components:\n1. A feature representation of the input. For each input observation x(i), this\nwill be a vector of features [x1;x2;:::;xn]. We will generally refer to feature\nifor input x(j)asx(j)\ni, sometimes simpli\ufb01ed as xi, but we will also see the\nnotation fi,fi(x), or, for multiclass classi\ufb01cation, fi(c;x).\n2. A classi\ufb01cation function that computes \u02c6 y, the estimated class, via p(yjx). In\nthe next section we will introduce the sigmoid andsoftmax tools for classi\ufb01-\ncation.\n3. An objective function that we want to optimize for learning, usually involving\nminimizing a loss function corresponding to error on training examples. We\nwill introduce the cross-entropy loss function .\n4. An algorithm for optimizing the objective function. We introduce the stochas-\ntic gradient descent algorithm.\nLogistic regression has two phases:\ntraining: We train the system (speci\ufb01cally the weights wandb, introduced be-\nlow) using stochastic gradient descent and the cross-entropy loss.\ntest: Given a test example xwe compute p(yjx)and return the higher probability\nlabel y=1 ory=0.\n5.1 The sigmoid function\nThe goal of binary logistic regression is to train a classi\ufb01er that can make a binary\ndecision about the class of a new input observation. Here we introduce the sigmoid\nclassi\ufb01er that will help us make this decision.\nConsider a single input observation x, which we will represent by a vector of\nfeatures [x1;x2;:::;xn]. (We\u2019ll show sample features in the next subsection.) The\nclassi\ufb01er output ycan be 1 (meaning the observation is a member of the class) or\n0 (the observation is not a member of the class). We want to know the probability 5.1 \u2022 T HE SIGMOID FUNCTION 3\nP(y=1jx)that this observation is a member of the class. So perhaps the decision\nis \u201cpositive sentiment\u201d versus \u201cnegative sentiment\u201d, the features represent counts of\nwords in a document, P(y=1jx)is the probability that the document has positive\nsentiment, and P(y=0jx)is the probability that the document has negative senti-\nment.\nLogistic regression solves this task by learning, from a training set, a vector of\nweights and a bias term . Each weight wiis a real number, and is associated with one\nof the input features xi. The weight wirepresents how important that input feature\nis to the classi\ufb01cation decision, and can be positive (providing evidence that the in-\nstance being classi\ufb01ed belongs in the positive class) or negative (providing evidence\nthat the instance being classi\ufb01ed belongs in the negative class). Thus we might\nexpect in a sentiment task the word awesome to have a high positive weight, and\nabysmal to have a very negative weight. The bias term , also called the intercept , is bias term\nintercept another real number that\u2019s added to the weighted inputs.\nTo make a decision on a test instance\u2014after we\u2019ve learned the weights in training\u2014\nthe classi\ufb01er \ufb01rst multiplies each xiby its weight wi, sums up the weighted features,\nand adds the bias term b. The resulting single number zexpresses the weighted sum\nof the evidence for the class.\nz= nX\ni=1wixi!\n+b (5.2)\nIn the rest of the book we\u2019ll represent such sums using the dot product notation dot product\nfrom linear algebra. The dot product of two vectors aandb, written as a\u0001b, is the\nsum of the products of the corresponding elements of each vector. (Notice that we\nrepresent vectors using the boldface notation b). Thus the following is an equivalent\nformation to Eq. 5.2:\nz=w\u0001x+b (5.3)\nBut note that nothing in Eq. 5.3 forces zto be a legal probability, that is, to lie\nbetween 0 and 1. In fact, since weights are real-valued, the output might even be\nnegative; zranges from\u0000\u00a5to\u00a5.\nFigure 5.1 The sigmoid function s(z) =1\n1+e\u0000ztakes a real value and maps it to the range\n(0;1). It is nearly linear around 0 but outlier values get squashed toward 0 or 1.\nTo create a probability, we\u2019ll pass zthrough the sigmoid function, s(z). The sigmoid\nsigmoid function (named because it looks like an s) is also called the logistic func-\ntion, and gives logistic regression its name. The sigmoid has the following equation,logistic\nfunction\nshown graphically in Fig. 5.1:\ns(z) =1\n1+e\u0000z=1\n1+exp(\u0000z)(5.4) 4CHAPTER 5 \u2022 L OGISTIC REGRESSION\n(For the rest of the book, we\u2019ll use the notation exp (x)to mean ex.) The sigmoid\nhas a number of advantages; it takes a real-valued number and maps it into the range\n(0;1), which is just what we want for a probability. Because it is nearly linear around\n0 but \ufb02attens toward the ends, it tends to squash outlier values toward 0 or 1. And\nit\u2019s differentiable, which as we\u2019ll see in Section 5.10 will be handy for learning.\nWe\u2019re almost there. If we apply the sigmoid to the sum of the weighted features,\nwe get a number between 0 and 1. To make it a probability, we just need to make\nsure that the two cases, p(y=1)andp(y=0), sum to 1. We can do this as follows:\nP(y=1) = s(w\u0001x+b)\n=1\n1+exp(\u0000(w\u0001x+b))\nP(y=0) = 1\u0000s(w\u0001x+b)\n=1\u00001\n1+exp(\u0000(w\u0001x+b))\n=exp(\u0000(w\u0001x+b))\n1+exp(\u0000(w\u0001x+b))(5.5)\nThe sigmoid function has the property\n1\u0000s(x) =s(\u0000x) (5.6)\nso we could also have expressed P(y=0)ass(\u0000(w\u0001x+b)).\nFinally, one terminological point. The input to the sigmoid function, the score\nz=w\u0001x+bfrom (5.3), is often called the logit . This is because the logit function logit\nis the inverse of the sigmoid. The logit function is the log of the odds ratiop\n1\u0000p:\nlogit(p) =s\u00001(p) =lnp\n1\u0000p(5.7)\nUsing the term logit forzis a way of reminding us that by using the sigmoid to turn\nz(which ranges from \u0000\u00a5to\u00a5) into a probability, we are implicitly interpreting zas\nnot just any real-valued number, but as speci\ufb01cally a log odds.\n5.2 Classi\ufb01cation with Logistic Regression\nThe sigmoid function from the prior section thus gives us a way to take an instance\nxand compute the probability P(y=1jx).\nHow do we make a decision about which class to apply to a test instance x? For\na given x, we say yes if the probability P(y=1jx)is more than .5, and no otherwise.\nWe call .5 the decision boundary :decision\nboundary\ndecision (x) =\u001a1 if P(y=1jx)>0:5\n0 otherwise\nLet\u2019s have some examples of applying logistic regression as a classi\ufb01er for language\ntasks. 5.2 \u2022 C LASSIFICATION WITH LOGISTIC REGRESSION 5\n5.2.1 Sentiment Classi\ufb01cation\nSuppose we are doing binary sentiment classi\ufb01cation on movie review text, and\nwe would like to know whether to assign the sentiment class +or\u0000to a review\ndocument doc. We\u2019ll represent each input observation by the 6 features x1:::x6of\nthe input shown in the following table; Fig. 5.2 shows the features in a sample mini\ntest document.\nVar De\ufb01nition Value in Fig. 5.2\nx1 count(positive lexicon words 2doc) 3\nx2 count(negative lexicon words 2doc) 2\nx3\u001a1 if \u201cno\u201d2doc\n0 otherwise1\nx4 count (1st and 2nd pronouns 2doc) 3\nx5\u001a1 if \u201c!\u201d2doc\n0 otherwise0\nx6 ln(word count of doc ) ln(66) =4:19\n It's hokey . There are virtually no surprises , and the writing is second-rate . So why was it so enjoyable  ? For one thing , the cast is great . Another nice touch is the music . I was overcome with the urge to get off the couch and start dancing .  It sucked me in , and it'll do the same to you  .x1=3x6=4.19x3=1x4=3x5=0x2=2\nFigure 5.2 A sample mini test document showing the extracted features in the vector x.\nLet\u2019s assume for the moment that we\u2019ve already learned a real-valued weight\nfor each of these features, and that the 6 weights corresponding to the 6 features\nare[2:5;\u00005:0;\u00001:2;0:5;2:0;0:7], while b= 0.1. (We\u2019ll discuss in the next section\nhow the weights are learned.) The weight w1, for example indicates how important\na feature the number of positive lexicon words ( great ,nice,enjoyable , etc.) is to\na positive sentiment decision, while w2tells us the importance of negative lexicon\nwords. Note that w1=2:5 is positive, while w2=\u00005:0, meaning that negative words\nare negatively associated with a positive sentiment decision, and are about twice as\nimportant as positive words.\nGiven these 6 features and the input review x,P(+jx)andP(\u0000jx)can be com-\nputed using Eq. 5.5:\np(+jx) =P(y=1jx) = s(w\u0001x+b)\n=s([2:5;\u00005:0;\u00001:2;0:5;2:0;0:7]\u0001[3;2;1;3;0;4:19]+0:1)\n=s(:833)\n=0:70 (5.8)\np(\u0000jx) =P(y=0jx) = 1\u0000s(w\u0001x+b)\n=0:30 6CHAPTER 5 \u2022 L OGISTIC REGRESSION\n5.2.2 Other classi\ufb01cation tasks and features\nLogistic regression is applied to all sorts of NLP tasks, and any property of the input\ncan be a feature. Consider the task of period disambiguation : deciding if a periodperiod\ndisambiguation\nis the end of a sentence or part of a word, by classifying each period into one of two\nclasses, EOS (end-of-sentence) and not-EOS. We might use features like x1below\nexpressing that the current word is lower case, perhaps with a positive weight. Or a\nfeature expressing that the current word is in our abbreviations dictionary (\u201cProf.\u201d),\nperhaps with a negative weight. A feature can also express a combination of proper-\nties. For example a period following an upper case word is likely to be an EOS, but\nif the word itself is St.and the previous word is capitalized then the period is likely\npart of a shortening of the word street following a street name.\nx1=\u001a\n1 if \u201c Case(wi) =Lower\u201d\n0 otherwise\nx2=\u001a1 if \u201c wi2AcronymDict\u201d\n0 otherwise\nx3=\u001a1 if \u201c wi=St. & Case(wi\u00001) =Upper\u201d\n0 otherwise\nDesigning versus learning features: In classic models, features are designed by\nhand by examining the training set with an eye to linguistic intuitions and literature,\nsupplemented by insights from error analysis on the training set of an early version\nof a system. We can also consider ( feature interactions ), complex features that arefeature\ninteractions\ncombinations of more primitive features. We saw such a feature for period disam-\nbiguation above, where a period on the word St.was less likely to be the end of the\nsentence if the previous word was capitalized. Features can be created automatically\nviafeature templates , abstract speci\ufb01cations of features. For example a bigramfeature\ntemplates\ntemplate for period disambiguation might create a feature for every pair of words\nthat occurs before a period in the training set. Thus the feature space is sparse, since\nwe only have to create a feature if that n-gram exists in that position in the training\nset. The feature is generally created as a hash from the string descriptions. A user\ndescription of a feature as, \u201cbigram(American breakfast)\u201d is hashed into a unique\ninteger ithat becomes the feature number fi.\nIt should be clear from the prior paragraph that designing features by hand re-\nquires extensive human effort. For this reason, recent NLP systems avoid hand-\ndesigned features and instead focus on representation learning : ways to learn fea-\ntures automatically in an unsupervised way from the input. We\u2019ll introduce methods\nfor representation learning in Chapter 6 and Chapter 7.\nScaling input features: When different input features have extremely different\nranges of values, it\u2019s common to rescale them so they have comparable ranges. We\nstandardize input values by centering them to result in a zero mean and a standard standardize\ndeviation of one (this transformation is sometimes called the z-score ). That is, if miz-score\nis the mean of the values of feature xiacross the mobservations in the input dataset,\nandsiis the standard deviation of the values of features xiacross the input dataset,\nwe can replace each feature xiby a new feature x0\nicomputed as follows:\nmi=1\nmmX\nj=1x(j)\ni si=vuut1\nmmX\nj=1\u0010\nx(j)\ni\u0000mi\u00112\nx0\ni=xi\u0000mi\nsi(5.9) 5.2 \u2022 C LASSIFICATION WITH LOGISTIC REGRESSION 7\nAlternatively, we can normalize the input features values to lie between 0 and 1: normalize\nx0\ni=xi\u0000min(xi)\nmax(xi)\u0000min(xi)(5.10)\nHaving input data with comparable range is useful when comparing values across\nfeatures. Data scaling is especially important in large neural networks, since it helps\nspeed up gradient descent.\n5.2.3 Processing many examples at once\nWe\u2019ve shown the equations for logistic regression for a single example. But in prac-\ntice we\u2019ll of course want to process an entire test set with many examples. Let\u2019s\nsuppose we have a test set consisting of mtest examples each of which we\u2019d like to\nclassify. We\u2019ll continue to use the notation from page 2, in which a superscript value\nin parentheses refers to the example index in some set of data (either for training or\nfor test). So in this case each test example x(i)has a feature vector x(i), 1\u0014i\u0014m.\n(As usual, we\u2019ll represent vectors and matrices in bold.)\nOne way to compute each output value \u02c6 y(i)is just to have a for-loop, and compute\neach test example one at a time:\nforeach x(i)in input [x(1);x(2);:::;x(m)]\ny(i)=s(w\u0001x(i)+b) (5.11)\nFor the \ufb01rst 3 test examples, then, we would be separately computing the pre-\ndicted \u02c6 y(i)as follows:\nP(y(1)=1jx(1)) = s(w\u0001x(1)+b)\nP(y(2)=1jx(2)) = s(w\u0001x(2)+b)\nP(y(3)=1jx(3)) = s(w\u0001x(3)+b)\nBut it turns out that we can slightly modify our original equation Eq. 5.5 to do\nthis much more ef\ufb01ciently. We\u2019ll use matrix arithmetic to assign a class to all the\nexamples with one matrix operation!\nFirst, we\u2019ll pack all the input feature vectors for each input xinto a single input\nmatrix X, where each row iis a row vector consisting of the feature vector for in-\nput example x(i)(i.e., the vector x(i)). Assuming each example has ffeatures and\nweights, Xwill therefore be a matrix of shape [m\u0002f], as follows:\nX=2\n66664x(1)\n1x(1)\n2:::x(1)\nf\nx(2)\n1x(2)\n2:::x(2)\nf\nx(3)\n1x(3)\n2:::x(3)\nf\n:::3\n77775(5.12)\nNow if we introduce bas a vector of length mwhich consists of the scalar bias\nterm brepeated mtimes, b= [b;b;:::;b], and ^ y= [\u02c6y(1);\u02c6y(2):::;\u02c6y(m)]as the vector of\noutputs (one scalar \u02c6 y(i)for each input x(i)and its feature vector x(i)), and represent\nthe weight vector was a column vector, we can compute all the outputs with a single\nmatrix multiplication and one addition:\ny=Xw+b (5.13) 8CHAPTER 5 \u2022 L OGISTIC REGRESSION\nYou should convince yourself that Eq. 5.13 computes the same thing as our for-loop\nin Eq. 5.11. For example \u02c6 y(1), the \ufb01rst entry of the output vector y, will correctly be:\n\u02c6y(1)= [x(1)\n1;x(1)\n2;:::;x(1)\nf]\u0001[w1;w2;:::;wf]+b (5.14)\nNote that we had to reorder Xandwfrom the order they appeared in in Eq. 5.5 to\nmake the multiplications come out properly. Here is Eq. 5.13 again with the shapes\nshown:\ny=X w +b\n(m\u00021) ( m\u0002f)(f\u00021) (m\u00021) (5.15)\nModern compilers and compute hardware can compute this matrix operation very\nef\ufb01ciently, making the computation much faster, which becomes important when\ntraining or testing on very large datasets.\nNote by the way that we could have kept Xandwin the original order ( y=\nXw+b) if we had chosen to de\ufb01ne Xdifferently as a matrix of column vectors, one\nvector for each input example, instead of row vectors, and then it would have shape\n[f\u0002m]. But we conventionally represent inputs as rows.\n5.2.4 Choosing a classi\ufb01er\nLogistic regression has a number of advantages over naive Bayes. Naive Bayes has\noverly strong conditional independence assumptions. Consider two features which\nare strongly correlated; in fact, imagine that we just add the same feature f1twice.\nNaive Bayes will treat both copies of f1as if they were separate, multiplying them\nboth in, overestimating the evidence. By contrast, logistic regression is much more\nrobust to correlated features; if two features f1and f2are perfectly correlated, re-\ngression will simply assign part of the weight to w1and part to w2. Thus when\nthere are many correlated features, logistic regression will assign a more accurate\nprobability than naive Bayes. So logistic regression generally works better on larger\ndocuments or datasets and is a common default.\nDespite the less accurate probabilities, naive Bayes still often makes the correct\nclassi\ufb01cation decision. Furthermore, naive Bayes can work extremely well (some-\ntimes even better than logistic regression) on very small datasets (Ng and Jordan,\n2002) or short documents (Wang and Manning, 2012). Furthermore, naive Bayes is\neasy to implement and very fast to train (there\u2019s no optimization step). So it\u2019s still a\nreasonable approach to use in some situations.\n5.3 Multinomial logistic regression\nSometimes we need more than two classes. Perhaps we might want to do 3-way\nsentiment classi\ufb01cation (positive, negative, or neutral). Or we could be assigning\nsome of the labels we will introduce in Chapter 17, like the part of speech of a word\n(choosing from 10, 30, or even 50 different parts of speech), or the named entity\ntype of a phrase (choosing from tags like person, location, organization).\nIn such cases we use multinomial logistic regression , also called softmax re-multinomial\nlogistic\nregressiongression (in older NLP literature you will sometimes see the name maxent classi-\n\ufb01er). In multinomial logistic regression we want to label each observation with a\nclass kfrom a set of Kclasses, under the stipulation that only one of these classes is 5.3 \u2022 M ULTINOMIAL LOGISTIC REGRESSION 9\nthe correct one (sometimes called hard classi\ufb01cation ; an observation can not be in\nmultiple classes). Let\u2019s use the following representation: the output yfor each input\nxwill be a vector of length K. If class cis the correct class, we\u2019ll set yc=1, and\nset all the other elements of yto be 0, i.e., yc=1 and yj=08j6=c. A vector like\nthisy, with one value=1 and the rest 0, is called a one-hot vector . The job of the\nclassi\ufb01er is to produce an estimate vector ^ y. For each class k, the value \u02c6 ykwill be\nthe classi\ufb01er\u2019s estimate of the probability p(yk=1jx).\n5.3.1 Softmax\nThe multinomial logistic classi\ufb01er uses a generalization of the sigmoid, called the\nsoftmax function, to compute p(yk=1jx). The softmax function takes a vector softmax\nz= [z1;z2;:::;zK]ofKarbitrary values and maps them to a probability distribution,\nwith each value in the range [0,1], and all the values summing to 1. Like the sigmoid,\nit is an exponential function.\nFor a vector zof dimensionality K, the softmax is de\ufb01ned as:\nsoftmax (zi) =exp(zi)PK\nj=1exp(zj)1\u0014i\u0014K (5.16)\nThe softmax of an input vector z= [z1;z2;:::;zK]is thus a vector itself:\nsoftmax (z) =\"\nexp(z1)PK\ni=1exp(zi);exp(z2)PK\ni=1exp(zi);:::;exp(zK)PK\ni=1exp(zi)#\n(5.17)\nThe denominatorPK\ni=1exp(zi)is used to normalize all the values into probabilities.\nThus for example given a vector:\nz= [0:6;1:1;\u00001:5;1:2;3:2;\u00001:1]\nthe resulting (rounded) softmax( z) is\n[0:05;0:09;0:01;0:1;0:74;0:01][0:05;0:09;0:01;0:1;0:74;0:01]\nLike the sigmoid, the softmax has the property of squashing values toward 0 or 1.\nThus if one of the inputs is larger than the others, it will tend to push its probability\ntoward 1, and suppress the probabilities of the smaller inputs.\nFinally, note that, just as for the sigmoid, we refer to z, the vector of scores that\nis the input to the softmax, as logits (see (5.7).\n5.3.2 Applying softmax in logistic regression\nWhen we apply softmax for logistic regression, the input will (just as for the sig-\nmoid) be the dot product between a weight vector wand an input vector x(plus a\nbias). But now we\u2019ll need separate weight vectors wkand bias bkfor each of the K\nclasses. The probability of each of our output classes \u02c6 ykcan thus be computed as:\np(yk=1jx) =exp(wk\u0001x+bk)\nKX\nj=1exp(wj\u0001x+bj)(5.18) 10 CHAPTER 5 \u2022 L OGISTIC REGRESSION\nThe form of Eq. 5.18 makes it seem that we would compute each output sep-\narately. Instead, it\u2019s more common to set up the equation for more ef\ufb01cient com-\nputation by modern vector processing hardware. We\u2019ll do this by representing the\nset of Kweight vectors as a weight matrix Wand a bias vector b. Each row kof\nWcorresponds to the vector of weights wk.Wthus has shape [K\u0002f], for Kthe\nnumber of output classes and fthe number of input features. The bias vector bhas\none value for each of the Koutput classes. If we represent the weights in this way,\nwe can compute \u02c6y, the vector of output probabilities for each of the Kclasses, by a\nsingle elegant equation:\n\u02c6y=softmax (Wx+b) (5.19)\nIf you work out the matrix arithmetic, you can see that the estimated score of\nthe \ufb01rst output class \u02c6 y1(before we take the softmax) will correctly turn out to be\nw1\u0001x+b1.\nFig. 5.3 shows an intuition of the role of the weight vector versus weight matrix\nin the computation of the output class probabilities for binary versus multinomial\nlogistic regression.\n5.3.3 Features in Multinomial Logistic Regression\nFeatures in multinomial logistic regression act like features in binary logistic regres-\nsion, with the difference mentioned above that we\u2019ll need separate weight vectors\nand biases for each of the Kclasses. Recall our binary exclamation point feature x5\nfrom page 5:\nx5=\u001a1 if \u201c!\u201d2doc\n0 otherwise\nIn binary classi\ufb01cation a positive weight w5on a feature in\ufb02uences the classi\ufb01er\ntoward y=1 (positive sentiment) and a negative weight in\ufb02uences it toward y=0\n(negative sentiment) with the absolute value indicating how important the feature\nis. For multinomial logistic regression, by contrast, with separate weights for each\nclass, a feature can be evidence for or against each individual class.\nIn 3-way multiclass sentiment classi\ufb01cation, for example, we must assign each\ndocument one of the 3 classes +,\u0000, or 0 (neutral). Now a feature related to excla-\nmation marks might have a negative weight for 0 documents, and a positive weight\nfor+or\u0000documents:\nFeature De\ufb01nition w5;+w5;\u0000w5;0\nf5(x)\u001a1 if \u201c!\u201d2doc\n0 otherwise3:5 3 :1\u00005:3\nBecause these feature weights are dependent both on the input text and the output\nclass, we sometimes make this dependence explicit and represent the features them-\nselves as f(x;y): a function of both the input and the class. Using such a notation\nf5(x)above could be represented as three features f5(x;+),f5(x;\u0000), and f5(x;0),\neach of which has a single weight. We\u2019ll use this kind of notation in our description\nof the CRF in Chapter 17. 5.4 \u2022 L EARNING IN LOGISTIC REGRESSION 11\nBinary Logistic Regression\nw[f \u2a091]Outputsigmoid[1\u2a09f]Input wordsp(+) = 1- p(-)\u2026y^xyInput featurevector [scalar]positive lexiconwords = 1count of \u201cno\u201d = 0wordcount=3x1x2x3xfdessert   was    greatWeight vector\nMultinomial Logistic Regression\nW[f\u2a091]Outputsoftmax[K\u2a09f]Input wordsp(+)\u2026y1^y2^y3^xyInput featurevector [K\u2a091]positive lexiconwords = 1count of \u201cno\u201d = 0wordcount=3x1x2x3xfdessert   was    greatp(-)p(neut)Weight matrixThese f red weightsare a row of W correspondingto weight vector w3,(= weights for class 3)\nFigure 5.3 Binary versus multinomial logistic regression. Binary logistic regression uses a\nsingle weight vector w, and has a scalar output \u02c6 y. In multinomial logistic regression we have\nKseparate weight vectors corresponding to the Kclasses, all packed into a single weight\nmatrix W, and a vector output \u02c6y. We omit the biases from both \ufb01gures for clarity.\n5.4 Learning in Logistic Regression\nHow are the parameters of the model, the weights wand bias b, learned? Logistic\nregression is an instance of supervised classi\ufb01cation in which we know the correct\nlabel y(either 0 or 1) for each observation x. What the system produces via Eq. 5.5\nis \u02c6y, the system\u2019s estimate of the true y. We want to learn parameters (meaning w\nandb) that make \u02c6 yfor each training observation as close as possible to the true y.\nThis requires two components that we foreshadowed in the introduction to the\nchapter. The \ufb01rst is a metric for how close the current label ( \u02c6 y) is to the true gold\nlabel y. Rather than measure similarity, we usually talk about the opposite of this:\nthedistance between the system output and the gold output, and we call this distance\nthelossfunction or the cost function . In the next section we\u2019ll introduce the loss loss\nfunction that is commonly used for logistic regression and also for neural networks, 12 CHAPTER 5 \u2022 L OGISTIC REGRESSION\nthecross-entropy loss .\nThe second thing we need is an optimization algorithm for iteratively updating\nthe weights so as to minimize this loss function. The standard algorithm for this is\ngradient descent ; we\u2019ll introduce the stochastic gradient descent algorithm in the\nfollowing section.\nWe\u2019ll describe these algorithms for the simpler case of binary logistic regres-\nsion in the next two sections, and then turn to multinomial logistic regression in\nSection 5.8.\n5.5 The cross-entropy loss function\nWe need a loss function that expresses, for an observation x, how close the classi\ufb01er\noutput ( \u02c6 y=s(w\u0001x+b)) is to the correct output ( y, which is 0 or 1). We\u2019ll call this:\nL(\u02c6y;y) = How much \u02c6 ydiffers from the true y (5.20)\nWe do this via a loss function that prefers the correct class labels of the train-\ning examples to be more likely . This is called conditional maximum likelihood\nestimation : we choose the parameters w;bthatmaximize the log probability of\nthe true ylabels in the training data given the observations x. The resulting loss\nfunction is the negative log likelihood loss , generally called the cross-entropy loss .cross-entropy\nloss\nLet\u2019s derive this loss function, applied to a single observation x. We\u2019d like to\nlearn weights that maximize the probability of the correct label p(yjx). Since there\nare only two discrete outcomes (1 or 0), this is a Bernoulli distribution, and we can\nexpress the probability p(yjx)that our classi\ufb01er produces for one observation as the\nfollowing (keeping in mind that if y=1, Eq. 5.21 simpli\ufb01es to \u02c6 y; ify=0, Eq. 5.21\nsimpli\ufb01es to 1\u0000\u02c6y):\np(yjx) = \u02c6yy(1\u0000\u02c6y)1\u0000y(5.21)\nNow we take the log of both sides. This will turn out to be handy mathematically,\nand doesn\u2019t hurt us; whatever values maximize a probability will also maximize the\nlog of the probability:\nlogp(yjx) = log\u0002\n\u02c6yy(1\u0000\u02c6y)1\u0000y\u0003\n=ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y) (5.22)\nEq. 5.22 describes a log likelihood that should be maximized. In order to turn this\ninto a loss function (something that we need to minimize), we\u2019ll just \ufb02ip the sign on\nEq. 5.22. The result is the cross-entropy loss LCE:\nLCE(\u02c6y;y) =\u0000logp(yjx) =\u0000[ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)] (5.23)\nFinally, we can plug in the de\ufb01nition of \u02c6 y=s(w\u0001x+b):\nLCE(\u02c6y;y) =\u0000[ylogs(w\u0001x+b)+(1\u0000y)log(1\u0000s(w\u0001x+b))] (5.24)\nLet\u2019s see if this loss function does the right thing for our example from Fig. 5.2. We\nwant the loss to be smaller if the model\u2019s estimate is close to correct, and bigger if\nthe model is confused. So \ufb01rst let\u2019s suppose the correct gold label for the sentiment\nexample in Fig. 5.2 is positive, i.e., y=1. In this case our model is doing well, since 5.6 \u2022 G RADIENT DESCENT 13\nfrom Eq. 5.8 it indeed gave the example a higher probability of being positive (.70)\nthan negative (.30). If we plug s(w\u0001x+b) =:70 and y=1 into Eq. 5.24, the right\nside of the equation drops out, leading to the following loss (we\u2019ll use log to mean\nnatural log when the base is not speci\ufb01ed):\nLCE(\u02c6y;y) =\u0000[ylogs(w\u0001x+b)+(1\u0000y)log(1\u0000s(w\u0001x+b))]\n=\u0000[logs(w\u0001x+b)]\n=\u0000log(:70)\n= :36\nBy contrast, let\u2019s pretend instead that the example in Fig. 5.2 was actually negative,\ni.e., y=0 (perhaps the reviewer went on to say \u201cBut bottom line, the movie is\nterrible! I beg you not to see it!\u201d). In this case our model is confused and we\u2019d want\nthe loss to be higher. Now if we plug y=0 and 1\u0000s(w\u0001x+b) =:30 from Eq. 5.8\ninto Eq. 5.24, the left side of the equation drops out:\nLCE(\u02c6y;y) =\u0000[ylogs(w\u0001x+b)+(1\u0000y)log(1\u0000s(w\u0001x+b))]\n= \u0000[log(1\u0000s(w\u0001x+b))]\n= \u0000log(:30)\n= 1:2\nSure enough, the loss for the \ufb01rst classi\ufb01er (.36) is less than the loss for the second\nclassi\ufb01er (1.2).\nWhy does minimizing this negative log probability do what we want? A perfect\nclassi\ufb01er would assign probability 1 to the correct outcome ( y=1 or y=0) and\nprobability 0 to the incorrect outcome. That means if yequals 1, the higher \u02c6 yis (the\ncloser it is to 1), the better the classi\ufb01er; the lower \u02c6 yis (the closer it is to 0), the\nworse the classi\ufb01er. If yequals 0, instead, the higher 1 \u0000\u02c6yis (closer to 1), the better\nthe classi\ufb01er. The negative log of \u02c6 y(if the true yequals 1) or 1\u0000\u02c6y(if the true y\nequals 0) is a convenient loss metric since it goes from 0 (negative log of 1, no loss)\nto in\ufb01nity (negative log of 0, in\ufb01nite loss). This loss function also ensures that as\nthe probability of the correct answer is maximized, the probability of the incorrect\nanswer is minimized; since the two sum to one, any increase in the probability of the\ncorrect answer is coming at the expense of the incorrect answer. It\u2019s called the cross-\nentropy loss, because Eq. 5.22 is also the formula for the cross-entropy between the\ntrue probability distribution yand our estimated distribution \u02c6 y.\nNow we know what we want to minimize; in the next section, we\u2019ll see how to\n\ufb01nd the minimum.\n5.6 Gradient Descent\nOur goal with gradient descent is to \ufb01nd the optimal weights: minimize the loss\nfunction we\u2019ve de\ufb01ned for the model. In Eq. 5.25 below, we\u2019ll explicitly represent\nthe fact that the cross-entropy loss function LCEis parameterized by the weights. In\nmachine learning in general we refer to the parameters being learned as q; in the\ncase of logistic regression q=fw;bg. So the goal is to \ufb01nd the set of weights which\nminimizes the loss function, averaged over all examples:\n\u02c6q=argmin\nq1\nmmX\ni=1LCE(f(x(i);q);y(i)) (5.25) 14 CHAPTER 5 \u2022 L OGISTIC REGRESSION\nHow shall we \ufb01nd the minimum of this (or any) loss function? Gradient descent is\na method that \ufb01nds a minimum of a function by \ufb01guring out in which direction (in\nthe space of the parameters q) the function\u2019s slope is rising the most steeply, and\nmoving in the opposite direction. The intuition is that if you are hiking in a canyon\nand trying to descend most quickly down to the river at the bottom, you might look\naround yourself in all directions, \ufb01nd the direction where the ground is sloping the\nsteepest, and walk downhill in that direction.\nFor logistic regression, this loss function is conveniently convex . A convex func- convex\ntion has at most one minimum; there are no local minima to get stuck in, so gradient\ndescent starting from any point is guaranteed to \ufb01nd the minimum. (By contrast,\nthe loss for multi-layer neural networks is non-convex, and gradient descent may\nget stuck in local minima for neural network training and never \ufb01nd the global opti-\nmum.)\nAlthough the algorithm (and the concept of gradient) are designed for direction\nvectors , let\u2019s \ufb01rst consider a visualization of the case where the parameter of our\nsystem is just a single scalar w, shown in Fig. 5.4.\nGiven a random initialization of wat some value w1, and assuming the loss\nfunction Lhappened to have the shape in Fig. 5.4, we need the algorithm to tell us\nwhether at the next iteration we should move left (making w2smaller than w1) or\nright (making w2bigger than w1) to reach the minimum.\nwLoss\n0w1wminslope of loss at w1 is negative(goal)one stepof gradientdescent\nFigure 5.4 The \ufb01rst step in iteratively \ufb01nding the minimum of this loss function, by moving\nwin the reverse direction from the slope of the function. Since the slope is negative, we need\nto move win a positive direction, to the right. Here superscripts are used for learning steps,\nsow1means the initial value of w(which is 0), w2the value at the second step, and so on.\nThe gradient descent algorithm answers this question by \ufb01nding the gradient gradient\nof the loss function at the current point and moving in the opposite direction. The\ngradient of a function of many variables is a vector pointing in the direction of the\ngreatest increase in a function. The gradient is a multi-variable generalization of the\nslope, so for a function of one variable like the one in Fig. 5.4, we can informally\nthink of the gradient as the slope. The dotted line in Fig. 5.4 shows the slope of this\nhypothetical loss function at point w=w1. You can see that the slope of this dotted\nline is negative. Thus to \ufb01nd the minimum, gradient descent tells us to go in the\nopposite direction: moving win a positive direction.\nThe magnitude of the amount to move in gradient descent is the value of the\nsloped\ndwL(f(x;w);y)weighted by a learning rate h. A higher (faster) learning learning rate 5.6 \u2022 G RADIENT DESCENT 15\nrate means that we should move wmore on each step. The change we make in our\nparameter is the learning rate times the gradient (or the slope, in our single-variable\nexample):\nwt+1=wt\u0000hd\ndwL(f(x;w);y) (5.26)\nNow let\u2019s extend the intuition from a function of one scalar variable wto many\nvariables, because we don\u2019t just want to move left or right, we want to know where\nin the N-dimensional space (of the Nparameters that make up q) we should move.\nThe gradient is just such a vector; it expresses the directional components of the\nsharpest slope along each of those Ndimensions. If we\u2019re just imagining two weight\ndimensions (say for one weight wand one bias b), the gradient might be a vector with\ntwo orthogonal components, each of which tells us how much the ground slopes in\nthewdimension and in the bdimension. Fig. 5.5 shows a visualization of the value\nof a 2-dimensional gradient vector taken at the red point.\nIn an actual logistic regression, the parameter vector wis much longer than 1 or\n2, since the input feature vector xcan be quite long, and we need a weight wifor\neach xi. For each dimension/variable wiinw(plus the bias b), the gradient will have\na component that tells us the slope with respect to that variable. In each dimension\nwi, we express the slope as a partial derivative\u00b6\n\u00b6wiof the loss function. Essentially\nwe\u2019re asking: \u201cHow much would a small change in that variable wiin\ufb02uence the\ntotal loss function L?\u201d\nFormally, then, the gradient of a multi-variable function fis a vector in which\neach component expresses the partial derivative of fwith respect to one of the vari-\nables. We\u2019ll use the inverted Greek delta symbol \u00d1to refer to the gradient, and\nrepresent \u02c6 yasf(x;q)to make the dependence on qmore obvious:\n\u00d1L(f(x;q);y) =2\n6666664\u00b6\n\u00b6w1L(f(x;q);y)\n\u00b6\n\u00b6w2L(f(x;q);y)\n...\n\u00b6\n\u00b6wnL(f(x;q);y)\n\u00b6\n\u00b6bL(f(x;q);y)3\n7777775(5.27)\nThe \ufb01nal equation for updating qbased on the gradient is thus\nqt+1=qt\u0000h\u00d1L(f(x;q);y) (5.28)\nCost(w,b)\nwb\nFigure 5.5 Visualization of the gradient vector at the red point in two dimensions wand\nb, showing a red arrow in the x-y plane pointing in the direction we will go to look for the\nminimum: the opposite direction of the gradient (recall that the gradient points in the direction\nof increase not decrease). 16 CHAPTER 5 \u2022 L OGISTIC REGRESSION\n5.6.1 The Gradient for Logistic Regression\nIn order to update q, we need a de\ufb01nition for the gradient \u00d1L(f(x;q);y). Recall that\nfor logistic regression, the cross-entropy loss function is:\nLCE(\u02c6y;y) =\u0000[ylogs(w\u0001x+b)+(1\u0000y)log(1\u0000s(w\u0001x+b))] (5.29)\nIt turns out that the derivative of this function for one observation vector xis Eq. 5.30\n(the interested reader can see Section 5.10 for the derivation of this equation):\n\u00b6LCE(\u02c6y;y)\n\u00b6wj= [s(w\u0001x+b)\u0000y]xj\n= ( \u02c6y\u0000y)xj (5.30)\nYou\u2019ll also sometimes see this equation in the equivalent form:\n\u00b6LCE(\u02c6y;y)\n\u00b6wj=\u0000(y\u0000\u02c6y)xj (5.31)\nNote in these equations that the gradient with respect to a single weight wjrep-\nresents a very intuitive value: the difference between the true yand our estimated\n\u02c6y=s(w\u0001x+b)for that observation, multiplied by the corresponding input value\nxj.\n5.6.2 The Stochastic Gradient Descent Algorithm\nStochastic gradient descent is an online algorithm that minimizes the loss function\nby computing its gradient after each training example, and nudging qin the right\ndirection (the opposite direction of the gradient). (An \u201conline algorithm\u201d is one that\nprocesses its input example by example, rather than waiting until it sees the entire\ninput.) Stochastic gradient descent is called stochastic because it chooses a single\nrandom example at a time; in Section 5.6.4 we\u2019ll discuss other versions of gradient\ndescent that batch many examples at once. Fig. 5.6 shows the algorithm.\nThe learning rate his ahyperparameter that must be adjusted. If it\u2019s too high, hyperparameter\nthe learner will take steps that are too large, overshooting the minimum of the loss\nfunction. If it\u2019s too low, the learner will take steps that are too small, and take too\nlong to get to the minimum. It is common to start with a higher learning rate and then\nslowly decrease it, so that it is a function of the iteration kof training; the notation\nhkcan be used to mean the value of the learning rate at iteration k.\nWe\u2019ll discuss hyperparameters in more detail in Chapter 7, but in short, they are\na special kind of parameter for any machine learning model. Unlike regular param-\neters of a model (weights like wandb), which are learned by the algorithm from\nthe training set, hyperparameters are special parameters chosen by the algorithm\ndesigner that affect how the algorithm works.\n5.6.3 Working through an example\nLet\u2019s walk through a single step of the gradient descent algorithm. We\u2019ll use a\nsimpli\ufb01ed version of the example in Fig. 5.2 as it sees a single observation x, whose\ncorrect value is y=1 (this is a positive review), and with a feature vector x= [x1;x2]\nconsisting of these two features:\nx1=3 (count of positive lexicon words)\nx2=2 (count of negative lexicon words) 5.6 \u2022 G RADIENT DESCENT 17\nfunction STOCHASTIC GRADIENT DESCENT (L(),f(),x,y)returns q\n# where: L is the loss function\n# f is a function parameterized by q\n# x is the set of training inputs x(1);x(2);:::;x(m)\n# y is the set of training outputs (labels) y(1);y(2);:::;y(m)\nq 0 # (or small random values)\nrepeat til done # see caption\nFor each training tuple (x(i);y(i))(in random order)\n1. Optional (for reporting): # How are we doing on this tuple?\nCompute \u02c6 y(i)=f(x(i);q)# What is our estimated output \u02c6 y?\nCompute the loss L(\u02c6y(i);y(i))# How far off is \u02c6 y(i)from the true output y(i)?\n2.g \u00d1qL(f(x(i);q);y(i)) # How should we move qto maximize loss?\n3.q q\u0000hg # Go the other way instead\nreturn q\nFigure 5.6 The stochastic gradient descent algorithm. Step 1 (computing the loss) is used\nmainly to report how well we are doing on the current tuple; we don\u2019t need to compute the\nloss in order to compute the gradient. The algorithm can terminate when it converges (when\nthe gradient norm <\u000f), or when progress halts (for example when the loss starts going up on\na held-out set). Weights are initialized to 0 for logistic regression, but to small random values\nfor neural networks, as we\u2019ll see in Chapter 7.\nLet\u2019s assume the initial weights and bias in q0are all set to 0, and the initial learning\nratehis 0.1:\nw1=w2=b=0\nh=0:1\nThe single update step requires that we compute the gradient, multiplied by the\nlearning rate\nqt+1=qt\u0000h\u00d1qL(f(x(i);q);y(i))\nIn our mini example there are three parameters, so the gradient vector has 3 dimen-\nsions, for w1,w2, and b. We can compute the \ufb01rst gradient as follows:\n\u00d1w;bL=2\n64\u00b6LCE(\u02c6y;y)\n\u00b6w1\u00b6LCE(\u02c6y;y)\n\u00b6w2\u00b6LCE(\u02c6y;y)\n\u00b6b3\n75=2\n4(s(w\u0001x+b)\u0000y)x1\n(s(w\u0001x+b)\u0000y)x2\ns(w\u0001x+b)\u0000y3\n5=2\n4(s(0)\u00001)x1\n(s(0)\u00001)x2\ns(0)\u000013\n5=2\n4\u00000:5x1\n\u00000:5x2\n\u00000:53\n5=2\n4\u00001:5\n\u00001:0\n\u00000:53\n5\nNow that we have a gradient, we compute the new parameter vector q1by moving\nq0in the opposite direction from the gradient:\nq1=2\n4w1\nw2\nb3\n5\u0000h2\n4\u00001:5\n\u00001:0\n\u00000:53\n5=2\n4:15\n:1\n:053\n5\nSo after one step of gradient descent, the weights have shifted to be: w1=:15,\nw2=:1, and b=:05.\nNote that this observation xhappened to be a positive example. We would expect\nthat after seeing more negative examples with high counts of negative words, that\nthe weight w2would shift to have a negative value. 18 CHAPTER 5 \u2022 L OGISTIC REGRESSION\n5.6.4 Mini-batch training\nStochastic gradient descent is called stochastic because it chooses a single random\nexample at a time, moving the weights so as to improve performance on that single\nexample. That can result in very choppy movements, so it\u2019s common to compute the\ngradient over batches of training instances rather than a single instance.\nFor example in batch training we compute the gradient over the entire dataset. batch training\nBy seeing so many examples, batch training offers a superb estimate of which di-\nrection to move the weights, at the cost of spending a lot of time processing every\nsingle example in the training set to compute this perfect direction.\nA compromise is mini-batch training: we train on a group of mexamples (per- mini-batch\nhaps 512, or 1024) that is less than the whole dataset. (If mis the size of the dataset,\nthen we are doing batch gradient descent; if m=1, we are back to doing stochas-\ntic gradient descent.) Mini-batch training also has the advantage of computational\nef\ufb01ciency. The mini-batches can easily be vectorized, choosing the size of the mini-\nbatch based on the computational resources. This allows us to process all the exam-\nples in one mini-batch in parallel and then accumulate the loss, something that\u2019s not\npossible with individual or batch training.\nWe just need to de\ufb01ne mini-batch versions of the cross-entropy loss function\nwe de\ufb01ned in Section 5.5 and the gradient in Section 5.6.1. Let\u2019s extend the cross-\nentropy loss for one example from Eq. 5.23 to mini-batches of size m. We\u2019ll continue\nto use the notation that x(i)andy(i)mean the ith training features and training label,\nrespectively. We make the assumption that the training examples are independent:\nlogp(training labels ) = logmY\ni=1p(y(i)jx(i))\n=mX\ni=1logp(y(i)jx(i))\n=\u0000mX\ni=1LCE(\u02c6y(i);y(i)) (5.32)\nNow the cost function for the mini-batch of mexamples is the average loss for each\nexample:\nCost(\u02c6y;y) =1\nmmX\ni=1LCE(\u02c6y(i);y(i))\n=\u00001\nmmX\ni=1y(i)logs(w\u0001x(i)+b)+(1\u0000y(i))log\u0010\n1\u0000s(w\u0001x(i)+b)\u0011\n(5.33)\nThe mini-batch gradient is the average of the individual gradients from Eq. 5.30:\n\u00b6Cost(\u02c6y;y)\n\u00b6wj=1\nmmX\ni=1h\ns(w\u0001x(i)+b)\u0000y(i)i\nx(i)\nj(5.34)\nInstead of using the sum notation, we can more ef\ufb01ciently compute the gradient\nin its matrix form, following the vectorization we saw on page 7, where we have a\nmatrix Xof size [m\u0002f]representing the minputs in the batch, and a vector yof size\n[m\u00021]representing the correct outputs: 5.7 \u2022 R EGULARIZATION 19\n\u00b6Cost(\u02c6y;y)\n\u00b6w=1\nm(\u02c6y\u0000y)|X\n=1\nm(s(Xw+b)\u0000y)|X (5.35)\n5.7 Regularization\nNumquam ponenda est pluralitas sine necessitate\n\u2018Plurality should never be proposed unless needed\u2019\nWilliam of Occam\nThere is a problem with learning weights that make the model perfectly match the\ntraining data. If a feature is perfectly predictive of the outcome because it happens\nto only occur in one class, it will be assigned a very high weight. The weights for\nfeatures will attempt to perfectly \ufb01t details of the training set, in fact too perfectly,\nmodeling noisy factors that just accidentally correlate with the class. This problem is\ncalled over\ufb01tting . A good model should be able to generalize well from the training over\ufb01tting\ngeneralize data to the unseen test set, but a model that over\ufb01ts will have poor generalization.\nTo avoid over\ufb01tting, a new regularization term R(q)is added to the loss func- regularization\ntion in Eq. 5.25, resulting in the following loss for a batch of mexamples (slightly\nrewritten from Eq. 5.25 to be maximizing log probability rather than minimizing\nloss, and removing the1\nmterm which doesn\u2019t affect the argmax):\n\u02c6q=argmax\nqmX\ni=1logP(y(i)jx(i))\u0000aR(q) (5.36)\nThe new regularization term R(q)is used to penalize large weights. Thus a setting\nof the weights that matches the training data perfectly\u2014 but uses many weights with\nhigh values to do so\u2014will be penalized more than a setting that matches the data a\nlittle less well, but does so using smaller weights. There are two common ways to\ncompute this regularization term R(q).L2 regularization is a quadratic function ofL2\nregularization\nthe weight values, named because it uses the (square of the) L2 norm of the weight\nvalues. The L2 norm, jjqjj2, is the same as the Euclidean distance of the vector q\nfrom the origin. If qconsists of nweights, then:\nR(q) =jjqjj2\n2=nX\nj=1q2\nj (5.37)\nThe L2 regularized loss function becomes:\n\u02c6q=argmax\nq\"mX\ni=1logP(y(i)jx(i))#\n\u0000anX\nj=1q2\nj (5.38)\nL1 regularization is a linear function of the weight values, named after the L1 normL1\nregularization\njjWjj1, the sum of the absolute values of the weights, or Manhattan distance (the 20 CHAPTER 5 \u2022 L OGISTIC REGRESSION\nManhattan distance is the distance you\u2019d have to walk between two points in a city\nwith a street grid like New York):\nR(q) =jjqjj1=nX\ni=1jqij (5.39)\nThe L1 regularized loss function becomes:\n\u02c6q=argmax\nq\"mX\ni=1logP(y(i)jx(i))#\n\u0000anX\nj=1jqjj (5.40)\nThese kinds of regularization come from statistics, where L1 regularization is called\nlasso regression (Tibshirani, 1996) and L2 regularization is called ridge regression , lasso\nridge and both are commonly used in language processing. L2 regularization is easier to\noptimize because of its simple derivative (the derivative of q2is just 2 q), while\nL1 regularization is more complex (the derivative of jqjis non-continuous at zero).\nBut while L2 prefers weight vectors with many small weights, L1 prefers sparse\nsolutions with some larger weights but many more weights set to zero. Thus L1\nregularization leads to much sparser weight vectors, that is, far fewer features.\nBoth L1 and L2 regularization have Bayesian interpretations as constraints on\nthe prior of how weights should look. L1 regularization can be viewed as a Laplace\nprior on the weights. L2 regularization corresponds to assuming that weights are\ndistributed according to a Gaussian distribution with mean m=0. In a Gaussian\nor normal distribution, the further away a value is from the mean, the lower its\nprobability (scaled by the variance s). By using a Gaussian prior on the weights, we\nare saying that weights prefer to have the value 0. A Gaussian for a weight qjis\n1q\n2ps2\njexp \n\u0000(qj\u0000mj)2\n2s2\nj!\n(5.41)\nIf we multiply each weight by a Gaussian prior on the weight, we are thus maximiz-\ning the following constraint:\n\u02c6q=argmax\nqmY\ni=1P(y(i)jx(i))\u0002nY\nj=11q\n2ps2\njexp \n\u0000(qj\u0000mj)2\n2s2\nj!\n(5.42)\nwhich in log space, with m=0, and assuming 2 s2=1, corresponds to\n\u02c6q=argmax\nqmX\ni=1logP(y(i)jx(i))\u0000anX\nj=1q2\nj (5.43)\nwhich is in the same form as Eq. 5.38.\n5.8 Learning in Multinomial Logistic Regression\nThe loss function for multinomial logistic regression generalizes the loss function\nfor binary logistic regression from 2 to Kclasses. Recall that that the cross-entropy\nloss for binary logistic regression (repeated from Eq. 5.23) is:\nLCE(\u02c6y;y) =\u0000logp(yjx) =\u0000[ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)] (5.44) 5.8 \u2022 L EARNING IN MULTINOMIAL LOGISTIC REGRESSION 21\nThe loss function for multinomial logistic regression generalizes the two terms in\nEq. 5.44 (one that is non-zero when y=1 and one that is non-zero when y=0) to\nKterms. As we mentioned above, for multinomial regression we\u2019ll represent both y\nand\u02c6yas vectors. The true label yis a vector with Kelements, each corresponding\nto a class, with yc=1 if the correct class is c, with all other elements of ybeing 0.\nAnd our classi\ufb01er will produce an estimate vector with Kelements \u02c6y, each element\n\u02c6ykof which represents the estimated probability p(yk=1jx).\nThe loss function for a single example x, generalizing from binary logistic re-\ngression, is the sum of the logs of the Koutput classes, each weighted by the indi-\ncator function yk(Eq. 5.45). This turns out to be just the negative log probability of\nthe correct class c(Eq. 5.46):\nLCE(\u02c6y;y) =\u0000KX\nk=1yklog \u02c6yk (5.45)\n=\u0000log \u02c6yc;(where cis the correct class) (5.46)\n=\u0000log \u02c6p(yc=1jx)(where cis the correct class)\n=\u0000logexp(wc\u0001x+bc)PK\nj=1exp(wj\u0001x+bj)(cis the correct class) (5.47)\nHow did we get from Eq. 5.45 to Eq. 5.46? Because only one class (let\u2019s call it c) is\nthe correct one, the vector ytakes the value 1 only for this value of k, i.e., has yc=1\nandyj=08j6=c. That means the terms in the sum in Eq. 5.45 will all be 0 except\nfor the term corresponding to the true class c. Hence the cross-entropy loss is simply\nthe log of the output probability corresponding to the correct class, and we therefore\nalso call Eq. 5.46 the negative log likelihood loss .negative log\nlikelihood loss\nOf course for gradient descent we don\u2019t need the loss, we need its gradient. The\ngradient for a single example turns out to be very similar to the gradient for binary\nlogistic regression, (\u02c6y\u0000y)x, that we saw in Eq. 5.30. Let\u2019s consider one piece of the\ngradient, the derivative for a single weight. For each class k, the weight of the ith\nelement of input xiswk;i. What is the partial derivative of the loss with respect to\nwk;i? This derivative turns out to be just the difference between the true value for the\nclass k(which is either 1 or 0) and the probability the classi\ufb01er outputs for class k,\nweighted by the value of the input xicorresponding to the ith element of the weight\nvector for class k:\n\u00b6LCE\n\u00b6wk;i=\u0000(yk\u0000\u02c6yk)xi\n=\u0000(yk\u0000p(yk=1jx))xi\n=\u0000 \nyk\u0000exp(wk\u0001x+bk)PK\nj=1exp(wj\u0001x+bj)!\nxi (5.48)\nWe\u2019ll return to this case of the gradient for softmax regression when we introduce\nneural networks in Chapter 7, and at that time we\u2019ll also discuss the derivation of\nthis gradient in equations Eq. ??\u2013Eq. ??. 22 CHAPTER 5 \u2022 L OGISTIC REGRESSION\n5.9 Interpreting models\nOften we want to know more than just the correct classi\ufb01cation of an observation.\nWe want to know why the classi\ufb01er made the decision it did. That is, we want our\ndecision to be interpretable . Interpretability can be hard to de\ufb01ne strictly, but the interpretable\ncore idea is that as humans we should know why our algorithms reach the conclu-\nsions they do. Because the features to logistic regression are often human-designed,\none way to understand a classi\ufb01er\u2019s decision is to understand the role each feature\nplays in the decision. Logistic regression can be combined with statistical tests (the\nlikelihood ratio test, or the Wald test); investigating whether a particular feature is\nsigni\ufb01cant by one of these tests, or inspecting its magnitude (how large is the weight\nwassociated with the feature?) can help us interpret why the classi\ufb01er made the\ndecision it makes. This is enormously important for building transparent models.\nFurthermore, in addition to its use as a classi\ufb01er, logistic regression in NLP and\nmany other \ufb01elds is widely used as an analytic tool for testing hypotheses about the\neffect of various explanatory variables (features). In text classi\ufb01cation, perhaps we\nwant to know if logically negative words ( no, not, never ) are more likely to be asso-\nciated with negative sentiment, or if negative reviews of movies are more likely to\ndiscuss the cinematography. However, in doing so it\u2019s necessary to control for po-\ntential confounds: other factors that might in\ufb02uence sentiment (the movie genre, the\nyear it was made, perhaps the length of the review in words). Or we might be study-\ning the relationship between NLP-extracted linguistic features and non-linguistic\noutcomes (hospital readmissions, political outcomes, or product sales), but need to\ncontrol for confounds (the age of the patient, the county of voting, the brand of the\nproduct). In such cases, logistic regression allows us to test whether some feature is\nassociated with some outcome above and beyond the effect of other features.\n5.10 Advanced: Deriving the Gradient Equation\nIn this section we give the derivation of the gradient of the cross-entropy loss func-\ntion LCEfor logistic regression. Let\u2019s start with some quick calculus refreshers.\nFirst, the derivative of ln (x):\nd\ndxln(x) =1\nx(5.49)\nSecond, the (very elegant) derivative of the sigmoid:\nds(z)\ndz=s(z)(1\u0000s(z)) (5.50)\nFinally, the chain rule of derivatives. Suppose we are computing the derivative chain rule\nof a composite function f(x) =u(v(x)). The derivative of f(x)is the derivative of\nu(x)with respect to v(x)times the derivative of v(x)with respect to x:\nd f\ndx=du\ndv\u0001dv\ndx(5.51)\nFirst, we want to know the derivative of the loss function with respect to a single\nweight wj(we\u2019ll need to compute it for each weight, and for the bias): 5.11 \u2022 S UMMARY 23\n\u00b6LCE\n\u00b6wj=\u00b6\n\u00b6wj\u0000[ylogs(w\u0001x+b)+(1\u0000y)log(1\u0000s(w\u0001x+b))]\n=\u0000\u0014\u00b6\n\u00b6wjylogs(w\u0001x+b)+\u00b6\n\u00b6wj(1\u0000y)log[1\u0000s(w\u0001x+b)]\u0015\n(5.52)\nNext, using the chain rule, and relying on the derivative of log:\n\u00b6LCE\n\u00b6wj=\u0000y\ns(w\u0001x+b)\u00b6\n\u00b6wjs(w\u0001x+b)\u00001\u0000y\n1\u0000s(w\u0001x+b)\u00b6\n\u00b6wj1\u0000s(w\u0001x+b)\n(5.53)\nRearranging terms:\n\u00b6LCE\n\u00b6wj=\u0000\u0014y\ns(w\u0001x+b)\u00001\u0000y\n1\u0000s(w\u0001x+b)\u0015\u00b6\n\u00b6wjs(w\u0001x+b)\n(5.54)\nAnd now plugging in the derivative of the sigmoid, and using the chain rule one\nmore time, we end up with Eq. 5.55:\n\u00b6LCE\n\u00b6wj=\u0000\u0014y\u0000s(w\u0001x+b)\ns(w\u0001x+b)[1\u0000s(w\u0001x+b)]\u0015\ns(w\u0001x+b)[1\u0000s(w\u0001x+b)]\u00b6(w\u0001x+b)\n\u00b6wj\n=\u0000\u0014y\u0000s(w\u0001x+b)\ns(w\u0001x+b)[1\u0000s(w\u0001x+b)]\u0015\ns(w\u0001x+b)[1\u0000s(w\u0001x+b)]xj\n=\u0000[y\u0000s(w\u0001x+b)]xj\n= [s(w\u0001x+b)\u0000y]xj (5.55)\n5.11 Summary\nThis chapter introduced the logistic regression model of classi\ufb01cation .\n\u2022 Logistic regression is a supervised machine learning classi\ufb01er that extracts\nreal-valued features from the input, multiplies each by a weight, sums them,\nand passes the sum through a sigmoid function to generate a probability. A\nthreshold is used to make a decision.\n\u2022 Logistic regression can be used with two classes (e.g., positive and negative\nsentiment) or with multiple classes ( multinomial logistic regression , for ex-\nample for n-ary text classi\ufb01cation, part-of-speech labeling, etc.).\n\u2022 Multinomial logistic regression uses the softmax function to compute proba-\nbilities.\n\u2022 The weights (vector wand bias b) are learned from a labeled training set via a\nloss function, such as the cross-entropy loss , that must be minimized.\n\u2022 Minimizing this loss function is a convex optimization problem, and iterative\nalgorithms like gradient descent are used to \ufb01nd the optimal weights.\n\u2022Regularization is used to avoid over\ufb01tting.\n\u2022 Logistic regression is also one of the most useful analytic tools, because of its\nability to transparently study the importance of individual features. 24 CHAPTER 5 \u2022 L OGISTIC REGRESSION\nBibliographical and Historical Notes\nLogistic regression was developed in the \ufb01eld of statistics, where it was used for\nthe analysis of binary data by the 1960s, and was particularly common in medicine\n(Cox, 1969). Starting in the late 1970s it became widely used in linguistics as one\nof the formal foundations of the study of linguistic variation (Sankoff and Labov,\n1979).\nNonetheless, logistic regression didn\u2019t become common in natural language pro-\ncessing until the 1990s, when it seems to have appeared simultaneously from two\ndirections. The \ufb01rst source was the neighboring \ufb01elds of information retrieval and\nspeech processing, both of which had made use of regression, and both of which\nlent many other statistical techniques to NLP. Indeed a very early use of logistic\nregression for document routing was one of the \ufb01rst NLP applications to use (LSI)\nembeddings as word representations (Sch \u00a8utze et al., 1995).\nAt the same time in the early 1990s logistic regression was developed and ap-\nplied to NLP at IBM Research under the name maximum entropy modeling ormaximum\nentropy\nmaxent (Berger et al., 1996), seemingly independent of the statistical literature. Un-\nder that name it was applied to language modeling (Rosenfeld, 1996), part-of-speech\ntagging (Ratnaparkhi, 1996), parsing (Ratnaparkhi, 1997), coreference resolution\n(Kehler, 1997), and text classi\ufb01cation (Nigam et al., 1999).\nMore on classi\ufb01cation can be found in machine learning textbooks (Hastie et al.\n2001, Witten and Frank 2005, Bishop 2006, Murphy 2012).\nExercises Exercises 25\nBerger, A., S. A. Della Pietra, and V . J. Della Pietra. 1996. A\nmaximum entropy approach to natural language process-\ning. Computational Linguistics , 22(1):39\u201371.\nBishop, C. M. 2006. Pattern recognition and machine learn-\ning. Springer.\nCox, D. 1969. Analysis of Binary Data . Chapman and Hall,\nLondon.\nHastie, T., R. J. Tibshirani, and J. H. Friedman. 2001. The\nElements of Statistical Learning . Springer.\nKehler, A. 1997. Probabilistic coreference in information\nextraction. EMNLP .\nMurphy, K. P. 2012. Machine learning: A probabilistic per-\nspective . MIT Press.\nNg, A. Y . and M. I. Jordan. 2002. On discriminative vs.\ngenerative classi\ufb01ers: A comparison of logistic regres-\nsion and naive bayes. NeurIPS .\nNigam, K., J. D. Lafferty, and A. McCallum. 1999. Using\nmaximum entropy for text classi\ufb01cation. IJCAI-99 work-\nshop on machine learning for information \ufb01ltering .\nRatnaparkhi, A. 1996. A maximum entropy part-of-speech\ntagger. EMNLP .\nRatnaparkhi, A. 1997. A linear observed time statistical\nparser based on maximum entropy models. EMNLP .\nRosenfeld, R. 1996. A maximum entropy approach to adap-\ntive statistical language modeling. Computer Speech and\nLanguage , 10:187\u2013228.\nSankoff, D. and W. Labov. 1979. On the uses of variable\nrules. Language in society , 8(2-3):189\u2013222.\nSch\u00a8utze, H., D. A. Hull, and J. Pedersen. 1995. A compar-\nison of classi\ufb01ers and document representations for the\nrouting problem. SIGIR-95 .\nTibshirani, R. J. 1996. Regression shrinkage and selection\nvia the lasso. Journal of the Royal Statistical Society. Se-\nries B (Methodological) , 58(1):267\u2013288.\nWang, S. and C. D. Manning. 2012. Baselines and bigrams:\nSimple, good sentiment and topic classi\ufb01cation. ACL.\nWitten, I. H. and E. Frank. 2005. Data Mining: Practi-\ncal Machine Learning Tools and Techniques , 2nd edition.\nMorgan Kaufmann.", "5_LR_Apr_7_2021.pdf": "Logistic RegressionBackground: Generative and Discriminative Classifiers Logistic RegressionImportant analytic tool in natural and social sciencesBaseline supervised machine learning tool for classificationIs also the foundation of neural networks Generative and Discriminative ClassifiersNaive Bayes is a generativeclassifierby contrast:Logistic regression is a discriminativeclassifier Generative and Discriminative Classifiers\nSuppose we're distinguishing cat from dog images\nimagenetimagenet Generative Classifier:\n\u2022Build a model of what's in a cat image\u2022Knows about whiskers, ears, eyes\u2022Assigns a probability to any image: \u2022how cat-y is this image?Also build a model for dog imagesNow given a new image:Run both models and see which one fits better  Discriminative Classifier\nJust try to distinguish dogs from catsOh look, dogs have collars!Let's ignore everything else Finding the correct class c from a document d inGenerative vs Discriminative ClassifiersNaive BayesLogistic Regression\n72CHAPTER5\u2022LOGISTICREGRESSIONMore formally, recall that the naive Bayes assigns a classcto a documentdnotby directly computingP(c|d)but by computing a likelihood and a prior\u02c6c=argmaxc2Clikelihoodz}|{P(d|c)priorz}|{P(c)(5.1)Agenerative modellike naive Bayes makes use of thislikelihoodterm, whichgenerativemodelexpresses how to generate the features of a documentif we knew it was of class c.By contrast adiscriminative modelin this text categorization scenario attemptsdiscriminativemodeltodirectlycomputeP(c|d). Perhaps it will learn to assign high weight to documentfeatures that directly improve its ability todiscriminatebetween possible classes,even if it couldn\u2019t generate an example of one of the classes.Components of a probabilistic machine learning classi\ufb01er:Like naive Bayes,logistic regression is a probabilistic classi\ufb01er that makes use of supervised machinelearning. Machine learning classi\ufb01ers require a training corpus ofMobservationsinput/output pairs(x(i),y(i)). (We\u2019ll use superscripts in parentheses to refer to indi-vidual instances in the training set\u2014for sentiment classi\ufb01cation each instance mightbe an individual document to be classi\ufb01ed). A machine learning system for classi\ufb01-cation then has four components:1.Afeature representationof the input. For each input observationx(i), thiswill be a vector of features[x1,x2,. . . ,xn]. We will generally refer to featureifor inputx(j)asx(j)i, sometimes simpli\ufb01ed asxi, but we will also see thenotationfi,fi(x), or, for multiclass classi\ufb01cation,fi(c,x).2.A classi\ufb01cation function that computes \u02c6y, the estimated class, viap(y|x). Inthe next section we will introduce thesigmoidandsoftmaxtools for classi\ufb01-cation.3.An objective function for learning, usually involving minimizing error ontraining examples. We will introduce thecross-entropy loss function4.An algorithm for optimizing the objective function. We introduce thestochas-tic gradient descentalgorithm.Logistic regression has two phases:training:we train the system (speci\ufb01cally the weightswandb) using stochasticgradient descent and the cross-entropy loss.test:Given a test examplexwe computep(y|x)and return the higher probabilitylabely=1 ory=0.5.1 Classi\ufb01cation: the sigmoidThe goal of binary logistic regression is to train a classi\ufb01er that can make a binarydecision about the class of a new input observation. Here we introduce thesigmoidclassi\ufb01er that will help us make this decision.Consider a single input observationx, which we will represent by a vector offeatures[x1,x2,. . . ,xn](we\u2019ll show sample features in the next subsection). The clas-si\ufb01er outputycan be 1 (meaning the observation is a member of the class) or 0(the observation is not a member of the class). We want to know the probabilityP(y=1|x)that this observation is a member of the class. So perhaps the decision2CHAPTER5\u2022LOGISTICREGRESSIONMore formally, recall that the naive Bayes assigns a classcto a documentdnotby directly computingP(c|d)but by computing a likelihood and a prior\u02c6c=argmaxc2Clikelihoodz}|{P(d|c)priorz}|{P(c)(5.1)Agenerative modellike naive Bayes makes use of thislikelihoodterm, whichgenerativemodelexpresses how to generate the features of a documentif we knew it was of class c.By contrast adiscriminative modelin this text categorization scenario attemptsdiscriminativemodeltodirectlycomputeP(c|d). Perhaps it will learn to assign high weight to documentfeatures that directly improve its ability todiscriminatebetween possible classes,even if it couldn\u2019t generate an example of one of the classes.Components of a probabilistic machine learning classi\ufb01er:Like naive Bayes,logistic regression is a probabilistic classi\ufb01er that makes use of supervised machinelearning. Machine learning classi\ufb01ers require a training corpus ofMobservationsinput/output pairs(x(i),y(i)). (We\u2019ll use superscripts in parentheses to refer to indi-vidual instances in the training set\u2014for sentiment classi\ufb01cation each instance mightbe an individual document to be classi\ufb01ed). A machine learning system for classi\ufb01-cation then has four components:1.Afeature representationof the input. For each input observationx(i), thiswill be a vector of features[x1,x2,. . . ,xn]. We will generally refer to featureifor inputx(j)asx(j)i, sometimes simpli\ufb01ed asxi, but we will also see thenotationfi,fi(x), or, for multiclass classi\ufb01cation,fi(c,x).2.A classi\ufb01cation function that computes \u02c6y, the estimated class, viap(y|x). Inthe next section we will introduce thesigmoidandsoftmaxtools for classi\ufb01-cation.3.An objective function for learning, usually involving minimizing error ontraining examples. We will introduce thecross-entropy loss function4.An algorithm for optimizing the objective function. We introduce thestochas-tic gradient descentalgorithm.Logistic regression has two phases:training:we train the system (speci\ufb01cally the weightswandb) using stochasticgradient descent and the cross-entropy loss.test:Given a test examplexwe computep(y|x)and return the higher probabilitylabely=1 ory=0.5.1 Classi\ufb01cation: the sigmoidThe goal of binary logistic regression is to train a classi\ufb01er that can make a binarydecision about the class of a new input observation. Here we introduce thesigmoidclassi\ufb01er that will help us make this decision.Consider a single input observationx, which we will represent by a vector offeatures[x1,x2,. . . ,xn](we\u2019ll show sample features in the next subsection). The clas-si\ufb01er outputycan be 1 (meaning the observation is a member of the class) or 0(the observation is not a member of the class). We want to know the probabilityP(y=1|x)that this observation is a member of the class. So perhaps the decisionP(c|d)posterior Components of a probabilistic machine learning classifier1.A feature representation of the input. For each input observation x(i), a vector of features [x1, x2, ... , xn]. Feature j for input x(i) is xj, more completely  xj(i), or sometimes fj(x).2.A classification function that computes !\ud835\udc66, the estimated class, via p(y|x), like the sigmoidor softmaxfunctions.3.An objective function for learning, like cross-entropy loss. 4.An algorithm for optimizing the objective function: stochastic gradient descent. Given m input/output pairs (x(i),y(i)): The two phases of logistic regression Training: we learn weights w and busing stochastic gradient descentand cross-entropy loss. Test: Given a test example x we compute p(y|x) using learned weights wand b, and return whichever label (y = 1 or y = 0) is higher probability Logistic RegressionBackground: Generative and Discriminative Classifiers Logistic RegressionClassification in Logistic Regression Classification ReminderPositive/negative sentiment  Spam/not spamAuthorship attribution  (Hamilton or Madison?)\nAlexander Hamilton Text Classification: definitionInput:\u25e6a document x\u25e6a fixed set of classes  C ={c1, c2,\u2026, cJ}Output: a predicted class !\ud835\udc66\u00ceC Binary Classification in Logistic RegressionGiven a series of input/output pairs:\u25e6(x(i), y(i))For each observation x(i)\u25e6We represent x(i)by a feature vector [x1, x2,\u2026, xn]\u25e6We compute an output: a predicted class !\ud835\udc66(i)\u00ce{0,1} Features in logistic regression\u2022For feature xi, weight witells is how important is xi\u2022xi=\"review contains \u2018awesome\u2019\":      wi=  +10\u2022xj=\"review contains \u2018abysmal\u2019\":      wj= -10\u2022xk=\u201creview contains \u2018mediocre\u2019\":   wk= -2 Logistic Regression for one observation xInput observation: vector  x = [x1, x2,\u2026, xn]Weights: one per feature: W= [w1, w2,\u2026, wn]\u25e6Sometimes we call the weights \u03b8= [\u03b81, \u03b82,\u2026, \u03b8n]Output: a predicted class !\ud835\udc66\u00ce{0,1}(multinomial logistic regression: !\ud835\udc66\u00ce{0, 1, 2, 3, 4}) How to do classificationFor each feature xi, weight witells us importance of xi\u25e6(Plus we'll have a bias b)We'll sum up all the weighted features and the biasIf this sum is high, we say y=1; if low, then y=05.1\u2022CLASSIFICATION:THE SIGMOID3is \u201cpositive sentiment\u201d versus \u201cnegative sentiment\u201d, the features represent countsof words in a document, andP(y=1|x)is the probability that the document haspositive sentiment, while andP(y=0|x)is the probability that the document hasnegative sentiment.Logistic regression solves this task by learning, from a training set, a vector ofweightsand abias term. Each weightwiis a real number, and is associated with oneof the input featuresxi. The weightwirepresents how important that input feature isto the classi\ufb01cation decision, and can be positive (meaning the feature is associatedwith the class) or negative (meaning the feature is not associated with the class).Thus we might expect in a sentiment task the wordawesometo have a high positiveweight, andabysmalto have a very negative weight. Thebias term, also called thebias termintercept, is another real number that\u2019s added to the weighted inputs.interceptTo make a decision on a test instance\u2014 after we\u2019ve learned the weights intraining\u2014 the classi\ufb01er \ufb01rst multiplies eachxiby its weightwi, sums up the weightedfeatures, and adds the bias termb. The resulting single numberzexpresses theweighted sum of the evidence for the class.z= nXi=1wixi!+b(5.2)In the rest of the book we\u2019ll represent such sums using thedot productnotation fromdot productlinear algebra. The dot product of two vectorsaandb, written asa\u00b7bis the sum ofthe products of the corresponding elements of each vector. Thus the following is anequivalent formation to Eq.5.2:z=w\u00b7x+b(5.3)But note that nothing in Eq.5.3forceszto be a legal probability, that is, to liebetween 0 and 1. In fact, since weights are real-valued, the output might even benegative;zranges from\u0000\u2022to\u2022.\nFigure 5.1The sigmoid functiony=11+e\u0000ztakes a real value and maps it to the range[0,1].Because it is nearly linear around 0 but has a sharp slope toward the ends, it tends to squashoutlier values toward 0 or 1.To create a probability, we\u2019ll passzthrough thesigmoidfunction,s(z). Thesigmoidsigmoid function (named because it looks like ans) is also called thelogistic func-tion, and gives logistic regression its name. The sigmoid has the following equation,logisticfunctionshown graphically in Fig.5.1:y=s(z)=11+e\u0000z(5.4)5.1\u2022CLASSIFICATION:THE SIGMOID3is \u201cpositive sentiment\u201d versus \u201cnegative sentiment\u201d, the features represent countsof words in a document, andP(y=1|x)is the probability that the document haspositive sentiment, while andP(y=0|x)is the probability that the document hasnegative sentiment.Logistic regression solves this task by learning, from a training set, a vector ofweightsand abias term. Each weightwiis a real number, and is associated with oneof the input featuresxi. The weightwirepresents how important that input feature isto the classi\ufb01cation decision, and can be positive (meaning the feature is associatedwith the class) or negative (meaning the feature is not associated with the class).Thus we might expect in a sentiment task the wordawesometo have a high positiveweight, andabysmalto have a very negative weight. Thebias term, also called thebias termintercept, is another real number that\u2019s added to the weighted inputs.interceptTo make a decision on a test instance\u2014 after we\u2019ve learned the weights intraining\u2014 the classi\ufb01er \ufb01rst multiplies eachxiby its weightwi, sums up the weightedfeatures, and adds the bias termb. The resulting single numberzexpresses theweighted sum of the evidence for the class.z= nXi=1wixi!+b(5.2)In the rest of the book we\u2019ll represent such sums using thedot productnotation fromdot productlinear algebra. The dot product of two vectorsaandb, written asa\u00b7bis the sum ofthe products of the corresponding elements of each vector. Thus the following is anequivalent formation to Eq.5.2:z=w\u00b7x+b(5.3)But note that nothing in Eq.5.3forceszto be a legal probability, that is, to liebetween 0 and 1. In fact, since weights are real-valued, the output might even benegative;zranges from\u0000\u2022to\u2022.\nFigure 5.1The sigmoid functiony=11+e\u0000ztakes a real value and maps it to the range[0,1].Because it is nearly linear around 0 but has a sharp slope toward the ends, it tends to squashoutlier values toward 0 or 1.To create a probability, we\u2019ll passzthrough thesigmoidfunction,s(z). Thesigmoidsigmoid function (named because it looks like ans) is also called thelogistic func-tion, and gives logistic regression its name. The sigmoid has the following equation,logisticfunctionshown graphically in Fig.5.1:y=s(z)=11+e\u0000z(5.4) But we want a probabilistic classifierWe need to formalize \u201csum is high\u201d.We\u2019d like a principled classifier that gives us a probability, just like Naive Bayes didWe want a model that can tell us:p(y=1|x;\u03b8)p(y=0|x; \u03b8) The problem:  z isn't a probability, it's just a number!Solution: use a function of z that goes from 0 to 15.1\u2022CLASSIFICATION:THE SIGMOID3is \u201cpositive sentiment\u201d versus \u201cnegative sentiment\u201d, the features represent countsof words in a document, andP(y=1|x)is the probability that the document haspositive sentiment, while andP(y=0|x)is the probability that the document hasnegative sentiment.Logistic regression solves this task by learning, from a training set, a vector ofweightsand abias term. Each weightwiis a real number, and is associated with oneof the input featuresxi. The weightwirepresents how important that input feature isto the classi\ufb01cation decision, and can be positive (meaning the feature is associatedwith the class) or negative (meaning the feature is not associated with the class).Thus we might expect in a sentiment task the wordawesometo have a high positiveweight, andabysmalto have a very negative weight. Thebias term, also called thebias termintercept, is another real number that\u2019s added to the weighted inputs.interceptTo make a decision on a test instance\u2014 after we\u2019ve learned the weights intraining\u2014 the classi\ufb01er \ufb01rst multiplies eachxiby its weightwi, sums up the weightedfeatures, and adds the bias termb. The resulting single numberzexpresses theweighted sum of the evidence for the class.z= nXi=1wixi!+b(5.2)In the rest of the book we\u2019ll represent such sums using thedot productnotation fromdot productlinear algebra. The dot product of two vectorsaandb, written asa\u00b7bis the sum ofthe products of the corresponding elements of each vector. Thus the following is anequivalent formation to Eq.5.2:z=w\u00b7x+b(5.3)But note that nothing in Eq.5.3forceszto be a legal probability, that is, to liebetween 0 and 1. In fact, since weights are real-valued, the output might even benegative;zranges from\u0000\u2022to\u2022.\nFigure 5.1The sigmoid functiony=11+e\u0000ztakes a real value and maps it to the range[0,1].Because it is nearly linear around 0 but has a sharp slope toward the ends, it tends to squashoutlier values toward 0 or 1.To create a probability, we\u2019ll passzthrough thesigmoidfunction,s(z). Thesigmoidsigmoid function (named because it looks like ans) is also called thelogistic func-tion, and gives logistic regression its name. The sigmoid has the following equation,logisticfunctionshown graphically in Fig.5.1:y=s(z)=11+e\u0000z(5.4)5.1\u2022CLASSIFICATION:THE SIGMOID3sentiment\u201d versus \u201cnegative sentiment\u201d, the features represent counts of words in adocument,P(y=1|x)is the probability that the document has positive sentiment,andP(y=0|x)is the probability that the document has negative sentiment.Logistic regression solves this task by learning, from a training set, a vector ofweightsand abias term. Each weightwiis a real number, and is associated with oneof the input featuresxi. The weightwirepresents how important that input featureis to the classi\ufb01cation decision, and can be positive (providing evidence that the in-stance being classi\ufb01ed belongs in the positive class) or negative (providing evidencethat the instance being classi\ufb01ed belongs in the negative class). Thus we mightexpect in a sentiment task the wordawesometo have a high positive weight, andabysmalto have a very negative weight. Thebias term, also called theintercept, isbias terminterceptanother real number that\u2019s added to the weighted inputs.To make a decision on a test instance\u2014 after we\u2019ve learned the weights intraining\u2014 the classi\ufb01er \ufb01rst multiplies eachxiby its weightwi, sums up the weightedfeatures, and adds the bias termb. The resulting single numberzexpresses theweighted sum of the evidence for the class.z= nXi=1wixi!+b(5.2)In the rest of the book we\u2019ll represent such sums using thedot productnotation fromdot productlinear algebra. The dot product of two vectorsaandb, written asa\u00b7bis the sum ofthe products of the corresponding elements of each vector. Thus the following is anequivalent formation to Eq.5.2:z=w\u00b7x+b(5.3)But note that nothing in Eq.5.3forceszto be a legal probability, that is, to liebetween 0 and 1. In fact, since weights are real-valued, the output might even benegative;zranges from\u0000\u2022to\u2022.\nFigure 5.1The sigmoid functiony=11+e\u0000ztakes a real value and maps it to the range[0,1].It is nearly linear around 0 but outlier values get squashed toward 0 or 1.To create a probability, we\u2019ll passzthrough thesigmoidfunction,s(z). Thesigmoidsigmoid function (named because it looks like ans) is also called thelogistic func-tion, and gives logistic regression its name. The sigmoid has the following equation,logisticfunctionshown graphically in Fig.5.1:y=s(z)=11+e\u0000z=11+exp(\u0000z)(5.4)(For the rest of the book, we\u2019ll use the notation exp(x)to meanex.) The sigmoidhas a number of advantages; it takes a real-valued number and maps it into the range The very useful sigmoid or logistic function\n205.1\u2022CLASSIFICATION:THE SIGMOID3is \u201cpositive sentiment\u201d versus \u201cnegative sentiment\u201d, the features represent countsof words in a document, andP(y=1|x)is the probability that the document haspositive sentiment, while andP(y=0|x)is the probability that the document hasnegative sentiment.Logistic regression solves this task by learning, from a training set, a vector ofweightsand abias term. Each weightwiis a real number, and is associated with oneof the input featuresxi. The weightwirepresents how important that input feature isto the classi\ufb01cation decision, and can be positive (meaning the feature is associatedwith the class) or negative (meaning the feature is not associated with the class).Thus we might expect in a sentiment task the wordawesometo have a high positiveweight, andabysmalto have a very negative weight. Thebias term, also called thebias termintercept, is another real number that\u2019s added to the weighted inputs.interceptTo make a decision on a test instance\u2014 after we\u2019ve learned the weights intraining\u2014 the classi\ufb01er \ufb01rst multiplies eachxiby its weightwi, sums up the weightedfeatures, and adds the bias termb. The resulting single numberzexpresses theweighted sum of the evidence for the class.z= nXi=1wixi!+b(5.2)In the rest of the book we\u2019ll represent such sums using thedot productnotation fromdot productlinear algebra. The dot product of two vectorsaandb, written asa\u00b7bis the sum ofthe products of the corresponding elements of each vector. Thus the following is anequivalent formation to Eq.5.2:z=w\u00b7x+b(5.3)But note that nothing in Eq.5.3forceszto be a legal probability, that is, to liebetween 0 and 1. In fact, since weights are real-valued, the output might even benegative;zranges from\u0000\u2022to\u2022.\nFigure 5.1The sigmoid functiony=11+e\u0000ztakes a real value and maps it to the range[0,1].Because it is nearly linear around 0 but has a sharp slope toward the ends, it tends to squashoutlier values toward 0 or 1.To create a probability, we\u2019ll passzthrough thesigmoidfunction,s(z). Thesigmoidsigmoid function (named because it looks like ans) is also called thelogistic func-tion, and gives logistic regression its name. The sigmoid has the following equation,logisticfunctionshown graphically in Fig.5.1:y=s(z)=11+e\u0000z(5.4)5.1\u2022CLASSIFICATION:THE SIGMOID3is \u201cpositive sentiment\u201d versus \u201cnegative sentiment\u201d, the features represent countsof words in a document, andP(y=1|x)is the probability that the document haspositive sentiment, while andP(y=0|x)is the probability that the document hasnegative sentiment.Logistic regression solves this task by learning, from a training set, a vector ofweightsand abias term. Each weightwiis a real number, and is associated with oneof the input featuresxi. The weightwirepresents how important that input feature isto the classi\ufb01cation decision, and can be positive (meaning the feature is associatedwith the class) or negative (meaning the feature is not associated with the class).Thus we might expect in a sentiment task the wordawesometo have a high positiveweight, andabysmalto have a very negative weight. Thebias term, also called thebias termintercept, is another real number that\u2019s added to the weighted inputs.interceptTo make a decision on a test instance\u2014 after we\u2019ve learned the weights intraining\u2014 the classi\ufb01er \ufb01rst multiplies eachxiby its weightwi, sums up the weightedfeatures, and adds the bias termb. The resulting single numberzexpresses theweighted sum of the evidence for the class.z= nXi=1wixi!+b(5.2)In the rest of the book we\u2019ll represent such sums using thedot productnotation fromdot productlinear algebra. The dot product of two vectorsaandb, written asa\u00b7bis the sum ofthe products of the corresponding elements of each vector. Thus the following is anequivalent formation to Eq.5.2:z=w\u00b7x+b(5.3)But note that nothing in Eq.5.3forceszto be a legal probability, that is, to liebetween 0 and 1. In fact, since weights are real-valued, the output might even benegative;zranges from\u0000\u2022to\u2022.\nFigure 5.1The sigmoid functiony=11+e\u0000ztakes a real value and maps it to the range[0,1].Because it is nearly linear around 0 but has a sharp slope toward the ends, it tends to squashoutlier values toward 0 or 1.To create a probability, we\u2019ll passzthrough thesigmoidfunction,s(z). Thesigmoidsigmoid function (named because it looks like ans) is also called thelogistic func-tion, and gives logistic regression its name. The sigmoid has the following equation,logisticfunctionshown graphically in Fig.5.1:y=s(z)=11+e\u0000z(5.4) Idea of logistic regressionWe\u2019ll compute w\u2219x+bAnd then we\u2019ll pass it through the sigmoid function:\u03c3(w\u2219x+b)And we'll just treat it as a probability Making probabilities with sigmoids4CHAPTER5\u2022LOGISTICREGRESSION[0,1], which is just what we want for a probability. Because it is nearly linear around0 but \ufb02attens toward the ends, it tends to squash outlier values toward 0 or 1. Andit\u2019s differentiable, which as we\u2019ll see in Section5.8will be handy for learning.We\u2019re almost there. If we apply the sigmoid to the sum of the weighted features,we get a number between 0 and 1. To make it a probability, we just need to makesure that the two cases,p(y=1)andp(y=0), sum to 1. We can do this as follows:P(y=1)=s(w\u00b7x+b)=11+exp(\u0000(w\u00b7x+b))P(y=0)=1\u0000s(w\u00b7x+b)=1\u000011+exp(\u0000(w\u00b7x+b))=exp(\u0000(w\u00b7x+b))1+exp(\u0000(w\u00b7x+b))(5.5)The sigmoid function has the property1\u0000s(x)=s(\u0000x)(5.6)so we could also have expressedP(y=0)ass(\u0000(w\u00b7x+b)).Now we have an algorithm that given an instancexcomputes the probabilityP(y=1|x). How do we make a decision? For a test instancex, we say yes if theprobabilityP(y=1|x)is more than .5, and no otherwise. We call .5 thedecisionboundary:decisionboundary\u02c6y=\u21e21 ifP(y=1|x)>0.50 otherwise5.1.1 Example: sentiment classi\ufb01cationLet\u2019s have an example. Suppose we are doing binary sentiment classi\ufb01cation onmovie review text, and we would like to know whether to assign the sentiment class+or\u0000to a review documentdoc. We\u2019ll represent each input observation by the 6featuresx1...x6of the input shown in the following table; Fig.5.2shows the featuresin a sample mini test document.Var De\ufb01nition Value in Fig.5.2x1count(positive lexicon)2doc)3x2count(negative lexicon)2doc)2x3\u21e21 if \u201cno\u201d2doc0 otherwise1x4count(1st and 2nd pronouns2doc)3x5\u21e21 if \u201c!\u201d2doc0 otherwise0x6log(word count of doc)ln(66)=4.19Let\u2019s assume for the moment that we\u2019ve already learned a real-valued weight foreach of these features, and that the 6 weights corresponding to the 6 features are[2.5,\u00005.0,\u00001.2,0.5,2.0,0.7], whileb= 0.1. (We\u2019ll discuss in the next section how4CHAPTER5\u2022LOGISTICREGRESSION[0,1], which is just what we want for a probability. Because it is nearly linear around0 but \ufb02attens toward the ends, it tends to squash outlier values toward 0 or 1. Andit\u2019s differentiable, which as we\u2019ll see in Section5.8will be handy for learning.We\u2019re almost there. If we apply the sigmoid to the sum of the weighted features,we get a number between 0 and 1. To make it a probability, we just need to makesure that the two cases,p(y=1)andp(y=0), sum to 1. We can do this as follows:P(y=1)=s(w\u00b7x+b)=11+exp(\u0000(w\u00b7x+b))P(y=0)=1\u0000s(w\u00b7x+b)=1\u000011+exp(\u0000(w\u00b7x+b))=exp(\u0000(w\u00b7x+b))1+exp(\u0000(w\u00b7x+b))(5.5)The sigmoid function has the property1\u0000s(x)=s(\u0000x)(5.6)so we could also have expressedP(y=0)ass(\u0000(w\u00b7x+b)).Now we have an algorithm that given an instancexcomputes the probabilityP(y=1|x). How do we make a decision? For a test instancex, we say yes if theprobabilityP(y=1|x)is more than .5, and no otherwise. We call .5 thedecisionboundary:decisionboundary\u02c6y=\u21e21 ifP(y=1|x)>0.50 otherwise5.1.1 Example: sentiment classi\ufb01cationLet\u2019s have an example. Suppose we are doing binary sentiment classi\ufb01cation onmovie review text, and we would like to know whether to assign the sentiment class+or\u0000to a review documentdoc. We\u2019ll represent each input observation by the 6featuresx1...x6of the input shown in the following table; Fig.5.2shows the featuresin a sample mini test document.Var De\ufb01nition Value in Fig.5.2x1count(positive lexicon)2doc)3x2count(negative lexicon)2doc)2x3\u21e21 if \u201cno\u201d2doc0 otherwise1x4count(1st and 2nd pronouns2doc)3x5\u21e21 if \u201c!\u201d2doc0 otherwise0x6log(word count of doc)ln(66)=4.19Let\u2019s assume for the moment that we\u2019ve already learned a real-valued weight foreach of these features, and that the 6 weights corresponding to the 6 features are[2.5,\u00005.0,\u00001.2,0.5,2.0,0.7], whileb= 0.1. (We\u2019ll discuss in the next section how By the way:4CHAPTER5\u2022LOGISTICREGRESSION[0,1], which is just what we want for a probability. Because it is nearly linear around0 but \ufb02attens toward the ends, it tends to squash outlier values toward 0 or 1. Andit\u2019s differentiable, which as we\u2019ll see in Section5.8will be handy for learning.We\u2019re almost there. If we apply the sigmoid to the sum of the weighted features,we get a number between 0 and 1. To make it a probability, we just need to makesure that the two cases,p(y=1)andp(y=0), sum to 1. We can do this as follows:P(y=1)=s(w\u00b7x+b)=11+exp(\u0000(w\u00b7x+b))P(y=0)=1\u0000s(w\u00b7x+b)=1\u000011+exp(\u0000(w\u00b7x+b))=exp(\u0000(w\u00b7x+b))1+exp(\u0000(w\u00b7x+b))(5.5)The sigmoid function has the property1\u0000s(x)=s(\u0000x)(5.6)so we could also have expressedP(y=0)ass(\u0000(w\u00b7x+b)).Now we have an algorithm that given an instancexcomputes the probabilityP(y=1|x). How do we make a decision? For a test instancex, we say yes if theprobabilityP(y=1|x)is more than .5, and no otherwise. We call .5 thedecisionboundary:decisionboundary\u02c6y=\u21e21 ifP(y=1|x)>0.50 otherwise5.1.1 Example: sentiment classi\ufb01cationLet\u2019s have an example. Suppose we are doing binary sentiment classi\ufb01cation onmovie review text, and we would like to know whether to assign the sentiment class+or\u0000to a review documentdoc. We\u2019ll represent each input observation by the 6featuresx1...x6of the input shown in the following table; Fig.5.2shows the featuresin a sample mini test document.Var De\ufb01nition Value in Fig.5.2x1count(positive lexicon)2doc)3x2count(negative lexicon)2doc)2x3\u21e21 if \u201cno\u201d2doc0 otherwise1x4count(1st and 2nd pronouns2doc)3x5\u21e21 if \u201c!\u201d2doc0 otherwise0x6log(word count of doc)ln(66)=4.19Let\u2019s assume for the moment that we\u2019ve already learned a real-valued weight foreach of these features, and that the 6 weights corresponding to the 6 features are[2.5,\u00005.0,\u00001.2,0.5,2.0,0.7], whileb= 0.1. (We\u2019ll discuss in the next section how4CHAPTER5\u2022LOGISTICREGRESSION[0,1], which is just what we want for a probability. Because it is nearly linear around0 but \ufb02attens toward the ends, it tends to squash outlier values toward 0 or 1. Andit\u2019s differentiable, which as we\u2019ll see in Section5.8will be handy for learning.We\u2019re almost there. If we apply the sigmoid to the sum of the weighted features,we get a number between 0 and 1. To make it a probability, we just need to makesure that the two cases,p(y=1)andp(y=0), sum to 1. We can do this as follows:P(y=1)=s(w\u00b7x+b)=11+exp(\u0000(w\u00b7x+b))P(y=0)=1\u0000s(w\u00b7x+b)=1\u000011+exp(\u0000(w\u00b7x+b))=exp(\u0000(w\u00b7x+b))1+exp(\u0000(w\u00b7x+b))(5.5)The sigmoid function has the property1\u0000s(x)=s(\u0000x)(5.6)so we could also have expressedP(y=0)ass(\u0000(w\u00b7x+b)).Now we have an algorithm that given an instancexcomputes the probabilityP(y=1|x). How do we make a decision? For a test instancex, we say yes if theprobabilityP(y=1|x)is more than .5, and no otherwise. We call .5 thedecisionboundary:decisionboundary\u02c6y=\u21e21 ifP(y=1|x)>0.50 otherwise5.1.1 Example: sentiment classi\ufb01cationLet\u2019s have an example. Suppose we are doing binary sentiment classi\ufb01cation onmovie review text, and we would like to know whether to assign the sentiment class+or\u0000to a review documentdoc. We\u2019ll represent each input observation by the 6featuresx1...x6of the input shown in the following table; Fig.5.2shows the featuresin a sample mini test document.Var De\ufb01nition Value in Fig.5.2x1count(positive lexicon)2doc)3x2count(negative lexicon)2doc)2x3\u21e21 if \u201cno\u201d2doc0 otherwise1x4count(1st and 2nd pronouns2doc)3x5\u21e21 if \u201c!\u201d2doc0 otherwise0x6log(word count of doc)ln(66)=4.19Let\u2019s assume for the moment that we\u2019ve already learned a real-valued weight foreach of these features, and that the 6 weights corresponding to the 6 features are[2.5,\u00005.0,\u00001.2,0.5,2.0,0.7], whileb= 0.1. (We\u2019ll discuss in the next section how=Because4CHAPTER5\u2022LOGISTICREGRESSION[0,1], which is just what we want for a probability. Because it is nearly linear around0 but \ufb02attens toward the ends, it tends to squash outlier values toward 0 or 1. Andit\u2019s differentiable, which as we\u2019ll see in Section5.8will be handy for learning.We\u2019re almost there. If we apply the sigmoid to the sum of the weighted features,we get a number between 0 and 1. To make it a probability, we just need to makesure that the two cases,p(y=1)andp(y=0), sum to 1. We can do this as follows:P(y=1)=s(w\u00b7x+b)=11+exp(\u0000(w\u00b7x+b))P(y=0)=1\u0000s(w\u00b7x+b)=1\u000011+exp(\u0000(w\u00b7x+b))=exp(\u0000(w\u00b7x+b))1+exp(\u0000(w\u00b7x+b))(5.5)The sigmoid function has the property1\u0000s(x)=s(\u0000x)(5.6)so we could also have expressedP(y=0)ass(\u0000(w\u00b7x+b)).Now we have an algorithm that given an instancexcomputes the probabilityP(y=1|x). How do we make a decision? For a test instancex, we say yes if theprobabilityP(y=1|x)is more than .5, and no otherwise. We call .5 thedecisionboundary:decisionboundary\u02c6y=\u21e21 ifP(y=1|x)>0.50 otherwise5.1.1 Example: sentiment classi\ufb01cationLet\u2019s have an example. Suppose we are doing binary sentiment classi\ufb01cation onmovie review text, and we would like to know whether to assign the sentiment class+or\u0000to a review documentdoc. We\u2019ll represent each input observation by the 6featuresx1...x6of the input shown in the following table; Fig.5.2shows the featuresin a sample mini test document.Var De\ufb01nition Value in Fig.5.2x1count(positive lexicon)2doc)3x2count(negative lexicon)2doc)2x3\u21e21 if \u201cno\u201d2doc0 otherwise1x4count(1st and 2nd pronouns2doc)3x5\u21e21 if \u201c!\u201d2doc0 otherwise0x6log(word count of doc)ln(66)=4.19Let\u2019s assume for the moment that we\u2019ve already learned a real-valued weight foreach of these features, and that the 6 weights corresponding to the 6 features are[2.5,\u00005.0,\u00001.2,0.5,2.0,0.7], whileb= 0.1. (We\u2019ll discuss in the next section how Turning a probability into a classifier4CHAPTER5\u2022LOGISTICREGRESSIONThe sigmoid has a number of advantages; it take a real-valued number and mapsit into the range[0,1], which is just what we want for a probability. Because it isnearly linear around 0 but has a sharp slope toward the ends, it tends to squash outliervalues toward 0 or 1. And it\u2019s differentiable, which as we\u2019ll see in Section5.8willbe handy for learning.We\u2019re almost there. If we apply the sigmoid to the sum of the weighted features,we get a number between 0 and 1. To make it a probability, we just need to makesure that the two cases,p(y=1)andp(y=0), sum to 1. We can do this as follows:P(y=1)=s(w\u00b7x+b)=11+e\u0000(w\u00b7x+b)P(y=0)=1\u0000s(w\u00b7x+b)=1\u000011+e\u0000(w\u00b7x+b)=e\u0000(w\u00b7x+b)1+e\u0000(w\u00b7x+b)(5.5)Now we have an algorithm that given an instancexcomputes the probabilityP(y=1|x). How do we make a decision? For a test instancex, we say yes if theprobabilityP(y=1|x)is more than .5, and no otherwise. We call .5 thedecisionboundary:decisionboundary\u02c6y=\u21e21 ifP(y=1|x)>0.50 otherwise5.1.1 Example: sentiment classi\ufb01cationLet\u2019s have an example. Suppose we are doing binary sentiment classi\ufb01cation onmovie review text, and we would like to know whether to assign the sentiment class+or\u0000to a review documentdoc. We\u2019ll represent each input observation by thefollowing 6 featuresx1...x6of the input; Fig.5.2shows the features in a sample minitest document.Var De\ufb01nition Value in Fig.5.2x1count(positive lexicon)2doc)3x2count(negative lexicon)2doc)2x3\u21e21 if \u201cno\u201d2doc0 otherwise1x4count(1st and 2nd pronouns2doc)3x5\u21e21 if \u201c!\u201d2doc0 otherwise0x6log(word count of doc)ln(64)=4.15Let\u2019s assume for the moment that we\u2019ve already learned a real-valued weightfor each of these features, and that the 6 weights corresponding to the 6 featuresare[2.5,\u00005.0,\u00001.2,0.5,2.0,0.7], whileb= 0.1. (We\u2019ll discuss in the next sectionhow the weights are learned.) The weightw1, for example indicates how important0.5 here is called the decision boundary The probabilistic classifier5.1\u2022CLASSIFICATION:THE SIGMOID3is \u201cpositive sentiment\u201d versus \u201cnegative sentiment\u201d, the features represent countsof words in a document, andP(y=1|x)is the probability that the document haspositive sentiment, while andP(y=0|x)is the probability that the document hasnegative sentiment.Logistic regression solves this task by learning, from a training set, a vector ofweightsand abias term. Each weightwiis a real number, and is associated with oneof the input featuresxi. The weightwirepresents how important that input feature isto the classi\ufb01cation decision, and can be positive (meaning the feature is associatedwith the class) or negative (meaning the feature is not associated with the class).Thus we might expect in a sentiment task the wordawesometo have a high positiveweight, andabysmalto have a very negative weight. Thebias term, also called thebias termintercept, is another real number that\u2019s added to the weighted inputs.interceptTo make a decision on a test instance\u2014 after we\u2019ve learned the weights intraining\u2014 the classi\ufb01er \ufb01rst multiplies eachxiby its weightwi, sums up the weightedfeatures, and adds the bias termb. The resulting single numberzexpresses theweighted sum of the evidence for the class.z= nXi=1wixi!+b(5.2)In the rest of the book we\u2019ll represent such sums using thedot productnotation fromdot productlinear algebra. The dot product of two vectorsaandb, written asa\u00b7bis the sum ofthe products of the corresponding elements of each vector. Thus the following is anequivalent formation to Eq.5.2:z=w\u00b7x+b(5.3)But note that nothing in Eq.5.3forceszto be a legal probability, that is, to liebetween 0 and 1. In fact, since weights are real-valued, the output might even benegative;zranges from\u0000\u2022to\u2022.\nFigure 5.1The sigmoid functiony=11+e\u0000ztakes a real value and maps it to the range[0,1].Because it is nearly linear around 0 but has a sharp slope toward the ends, it tends to squashoutlier values toward 0 or 1.To create a probability, we\u2019ll passzthrough thesigmoidfunction,s(z). Thesigmoidsigmoid function (named because it looks like ans) is also called thelogistic func-tion, and gives logistic regression its name. The sigmoid has the following equation,logisticfunctionshown graphically in Fig.5.1:y=s(z)=11+e\u0000z(5.4)wx+ b4CHAPTER5\u2022LOGISTICREGRESSIONThe sigmoid has a number of advantages; it take a real-valued number and mapsit into the range[0,1], which is just what we want for a probability. Because it isnearly linear around 0 but has a sharp slope toward the ends, it tends to squash outliervalues toward 0 or 1. And it\u2019s differentiable, which as we\u2019ll see in Section5.8willbe handy for learning.We\u2019re almost there. If we apply the sigmoid to the sum of the weighted features,we get a number between 0 and 1. To make it a probability, we just need to makesure that the two cases,p(y=1)andp(y=0), sum to 1. We can do this as follows:P(y=1)=s(w\u00b7x+b)=11+e\u0000(w\u00b7x+b)P(y=0)=1\u0000s(w\u00b7x+b)=1\u000011+e\u0000(w\u00b7x+b)=e\u0000(w\u00b7x+b)1+e\u0000(w\u00b7x+b)(5.5)Now we have an algorithm that given an instancexcomputes the probabilityP(y=1|x). How do we make a decision? For a test instancex, we say yes if theprobabilityP(y=1|x)is more than .5, and no otherwise. We call .5 thedecisionboundary:decisionboundary\u02c6y=\u21e21 ifP(y=1|x)>0.50 otherwise5.1.1 Example: sentiment classi\ufb01cationLet\u2019s have an example. Suppose we are doing binary sentiment classi\ufb01cation onmovie review text, and we would like to know whether to assign the sentiment class+or\u0000to a review documentdoc. We\u2019ll represent each input observation by thefollowing 6 featuresx1...x6of the input; Fig.5.2shows the features in a sample minitest document.Var De\ufb01nition Value in Fig.5.2x1count(positive lexicon)2doc)3x2count(negative lexicon)2doc)2x3\u21e21 if \u201cno\u201d2doc0 otherwise1x4count(1st and 2nd pronouns2doc)3x5\u21e21 if \u201c!\u201d2doc0 otherwise0x6log(word count of doc)ln(64)=4.15Let\u2019s assume for the moment that we\u2019ve already learned a real-valued weightfor each of these features, and that the 6 weights corresponding to the 6 featuresare[2.5,\u00005.0,\u00001.2,0.5,2.0,0.7], whileb= 0.1. (We\u2019ll discuss in the next sectionhow the weights are learned.) The weightw1, for example indicates how importantP(y=1) Turning a probability into a classifier4CHAPTER5\u2022LOGISTICREGRESSIONThe sigmoid has a number of advantages; it take a real-valued number and mapsit into the range[0,1], which is just what we want for a probability. Because it isnearly linear around 0 but has a sharp slope toward the ends, it tends to squash outliervalues toward 0 or 1. And it\u2019s differentiable, which as we\u2019ll see in Section5.8willbe handy for learning.We\u2019re almost there. If we apply the sigmoid to the sum of the weighted features,we get a number between 0 and 1. To make it a probability, we just need to makesure that the two cases,p(y=1)andp(y=0), sum to 1. We can do this as follows:P(y=1)=s(w\u00b7x+b)=11+e\u0000(w\u00b7x+b)P(y=0)=1\u0000s(w\u00b7x+b)=1\u000011+e\u0000(w\u00b7x+b)=e\u0000(w\u00b7x+b)1+e\u0000(w\u00b7x+b)(5.5)Now we have an algorithm that given an instancexcomputes the probabilityP(y=1|x). How do we make a decision? For a test instancex, we say yes if theprobabilityP(y=1|x)is more than .5, and no otherwise. We call .5 thedecisionboundary:decisionboundary\u02c6y=\u21e21 ifP(y=1|x)>0.50 otherwise5.1.1 Example: sentiment classi\ufb01cationLet\u2019s have an example. Suppose we are doing binary sentiment classi\ufb01cation onmovie review text, and we would like to know whether to assign the sentiment class+or\u0000to a review documentdoc. We\u2019ll represent each input observation by thefollowing 6 featuresx1...x6of the input; Fig.5.2shows the features in a sample minitest document.Var De\ufb01nition Value in Fig.5.2x1count(positive lexicon)2doc)3x2count(negative lexicon)2doc)2x3\u21e21 if \u201cno\u201d2doc0 otherwise1x4count(1st and 2nd pronouns2doc)3x5\u21e21 if \u201c!\u201d2doc0 otherwise0x6log(word count of doc)ln(64)=4.15Let\u2019s assume for the moment that we\u2019ve already learned a real-valued weightfor each of these features, and that the 6 weights corresponding to the 6 featuresare[2.5,\u00005.0,\u00001.2,0.5,2.0,0.7], whileb= 0.1. (We\u2019ll discuss in the next sectionhow the weights are learned.) The weightw1, for example indicates how importantif w\u2219x+b> 0if w\u2219x+b\u2264 0 Logistic RegressionClassification in Logistic Regression Logistic RegressionLogistic Regression: a text example on sentiment classification Sentiment example: does y=1 or y=0?It's hokey . There are virtually no surprises , and the writing is second-rate .         So why was it so enjoyable ? For one thing , the cast is great . Another nice touch is the music . I was overcome with the urge to get off the couch and start dancing . It sucked me in , and it'll do the same to you .\n29\n 305.1\u2022CLASSIFICATION:THE SIGMOID5 It's hokey . There are virtually no surprises , and the writing is second-rate . So why was it so enjoyable  ? For one thing , the cast is great . Another nice touch is the music . I was overcome with the urge to get off the couch and start dancing .  It sucked me in , and it'll do the same to you  .x1=3x6=4.19x3=1x4=3x5=0x2=2\nFigure 5.2A sample mini test document showing the extracted features in the vectorx.Given these 6 features and the input reviewx,P(+|x)andP(\u0000|x)can be com-puted using Eq.5.5:p(+|x)=P(Y=1|x)=s(w\u00b7x+b)=s([2.5,\u00005.0,\u00001.2,0.5,2.0,0.7]\u00b7[3,2,1,3,0,4.19]+0.1)=s(.833)=0.70(5.6)p(\u0000|x)=P(Y=0|x)=1\u0000s(w\u00b7x+b)=0.30Logistic regression is commonly applied to all sorts of NLP tasks, and any propertyof the input can be a feature. Consider the task ofperiod disambiguation: decidingif a period is the end of a sentence or part of a word, by classifying each periodinto one of two classes EOS (end-of-sentence) and not-EOS. We might use featureslikex1below expressing that the current word is lower case and the class is EOS(perhaps with a positive weight), or that the current word is in our abbreviationsdictionary (\u201cProf.\u201d) and the class is EOS (perhaps with a negative weight). A featurecan also express a quite complex combination of properties. For example a periodfollowing an upper case word is likely to be an EOS, but if the word itself isSt.andthe previous word is capitalized, then the period is likely part of a shortening of thewordstreet.x1=\u21e21 if \u201cCase(wi)=Lower\u201d0 otherwisex2=\u21e21 if \u201cwi2AcronymDict\u201d0 otherwisex3=\u21e21 if \u201cwi=St. &Case(wi\u00001)=Cap\u201d0 otherwiseDesigning features:Features are generally designed by examining the trainingset with an eye to linguistic intuitions and the linguistic literature on the domain. Acareful error analysis on the training set or devset of an early version of a systemoften provides insights into features.For some tasks it is especially helpful to build complex features that are combi-nations of more primitive features. We saw such a feature for period disambiguationabove, where a period on the wordSt.was less likely to be the end of the sentenceif the previous word was capitalized. For logistic regression and naive Bayes thesecombination features orfeature interactionshave to be designed by hand.featureinteractions4CHAPTER5\u2022LOGISTICREGRESSIONnearly linear around 0 but has a sharp slope toward the ends, it tends to squash outliervalues toward 0 or 1. And it\u2019s differentiable, which as we\u2019ll see in Section5.8willbe handy for learning.We\u2019re almost there. If we apply the sigmoid to the sum of the weighted features,we get a number between 0 and 1. To make it a probability, we just need to makesure that the two cases,p(y=1)andp(y=0), sum to 1. We can do this as follows:P(y=1)=s(w\u00b7x+b)=11+e\u0000(w\u00b7x+b)P(y=0)=1\u0000s(w\u00b7x+b)=1\u000011+e\u0000(w\u00b7x+b)=e\u0000(w\u00b7x+b)1+e\u0000(w\u00b7x+b)(5.5)Now we have an algorithm that given an instancexcomputes the probabilityP(y=1|x). How do we make a decision? For a test instancex, we say yes if the probabilityP(y=1|x)is more than .5, and no otherwise. We call .5 thedecision boundary:decisionboundary\u02c6y=\u21e21 ifP(y=1|x)>0.50 otherwise5.1.1 Example: sentiment classi\ufb01cationLet\u2019s have an example. Suppose we are doing binary sentiment classi\ufb01cation onmovie review text, and we would like to know whether to assign the sentiment class+or\u0000to a review documentdoc. We\u2019ll represent each input observation by the 6featuresx1...x6of the input shown in the following table; Fig.5.2shows the featuresin a sample mini test document.Var De\ufb01nition Value in Fig.5.2x1count(positive lexicon)2doc)3x2count(negative lexicon)2doc)2x3\u21e21 if \u201cno\u201d2doc0 otherwise1x4count(1st and 2nd pronouns2doc)3x5\u21e21 if \u201c!\u201d2doc0 otherwise0x6log(word count of doc)ln(66)=4.19Let\u2019s assume for the moment that we\u2019ve already learned a real-valued weight foreach of these features, and that the 6 weights corresponding to the 6 features are[2.5,\u00005.0,\u00001.2,0.5,2.0,0.7], whileb= 0.1. (We\u2019ll discuss in the next section howthe weights are learned.) The weightw1, for example indicates how important afeature the number of positive lexicon words (great,nice,enjoyable, etc.) is toa positive sentiment decision, whilew2tells us the importance of negative lexiconwords. Note thatw1=2.5 is positive, whilew2=\u00005.0, meaning that negative wordsare negatively associated with a positive sentiment decision, and are about twice asimportant as positive words. Classifying sentiment for input x\n314CHAPTER5\u2022LOGISTICREGRESSIONnearly linear around 0 but has a sharp slope toward the ends, it tends to squash outliervalues toward 0 or 1. And it\u2019s differentiable, which as we\u2019ll see in Section5.8willbe handy for learning.We\u2019re almost there. If we apply the sigmoid to the sum of the weighted features,we get a number between 0 and 1. To make it a probability, we just need to makesure that the two cases,p(y=1)andp(y=0), sum to 1. We can do this as follows:P(y=1)=s(w\u00b7x+b)=11+e\u0000(w\u00b7x+b)P(y=0)=1\u0000s(w\u00b7x+b)=1\u000011+e\u0000(w\u00b7x+b)=e\u0000(w\u00b7x+b)1+e\u0000(w\u00b7x+b)(5.5)Now we have an algorithm that given an instancexcomputes the probabilityP(y=1|x). How do we make a decision? For a test instancex, we say yes if the probabilityP(y=1|x)is more than .5, and no otherwise. We call .5 thedecision boundary:decisionboundary\u02c6y=\u21e21 ifP(y=1|x)>0.50 otherwise5.1.1 Example: sentiment classi\ufb01cationLet\u2019s have an example. Suppose we are doing binary sentiment classi\ufb01cation onmovie review text, and we would like to know whether to assign the sentiment class+or\u0000to a review documentdoc. We\u2019ll represent each input observation by the 6featuresx1...x6of the input shown in the following table; Fig.5.2shows the featuresin a sample mini test document.Var De\ufb01nition Value in Fig.5.2x1count(positive lexicon)2doc)3x2count(negative lexicon)2doc)2x3\u21e21 if \u201cno\u201d2doc0 otherwise1x4count(1st and 2nd pronouns2doc)3x5\u21e21 if \u201c!\u201d2doc0 otherwise0x6log(word count of doc)ln(66)=4.19Let\u2019s assume for the moment that we\u2019ve already learned a real-valued weight foreach of these features, and that the 6 weights corresponding to the 6 features are[2.5,\u00005.0,\u00001.2,0.5,2.0,0.7], whileb= 0.1. (We\u2019ll discuss in the next section howthe weights are learned.) The weightw1, for example indicates how important afeature the number of positive lexicon words (great,nice,enjoyable, etc.) is toa positive sentiment decision, whilew2tells us the importance of negative lexiconwords. Note thatw1=2.5 is positive, whilew2=\u00005.0, meaning that negative wordsare negatively associated with a positive sentiment decision, and are about twice asimportant as positive words.4CHAPTER5\u2022LOGISTICREGRESSIONThe sigmoid has a number of advantages; it take a real-valued number and mapsit into the range[0,1], which is just what we want for a probability. Because it isnearly linear around 0 but has a sharp slope toward the ends, it tends to squash outliervalues toward 0 or 1. And it\u2019s differentiable, which as we\u2019ll see in Section5.8willbe handy for learning.We\u2019re almost there. If we apply the sigmoid to the sum of the weighted features,we get a number between 0 and 1. To make it a probability, we just need to makesure that the two cases,p(y=1)andp(y=0), sum to 1. We can do this as follows:P(y=1)=s(w\u00b7x+b)=11+e\u0000(w\u00b7x+b)P(y=0)=1\u0000s(w\u00b7x+b)=1\u000011+e\u0000(w\u00b7x+b)=e\u0000(w\u00b7x+b)1+e\u0000(w\u00b7x+b)(5.5)Now we have an algorithm that given an instancexcomputes the probabilityP(y=1|x). How do we make a decision? For a test instancex, we say yes if theprobabilityP(y=1|x)is more than .5, and no otherwise. We call .5 thedecisionboundary:decisionboundary\u02c6y=\u21e21 ifP(y=1|x)>0.50 otherwise5.1.1 Example: sentiment classi\ufb01cationLet\u2019s have an example. Suppose we are doing binary sentiment classi\ufb01cation onmovie review text, and we would like to know whether to assign the sentiment class+or\u0000to a review documentdoc. We\u2019ll represent each input observation by thefollowing 6 featuresx1...x6of the input; Fig.5.2shows the features in a sample minitest document.Var De\ufb01nition Value in Fig.5.2x1count(positive lexicon)2doc)3x2count(negative lexicon)2doc)2x3\u21e21 if \u201cno\u201d2doc0 otherwise1x4count(1st and 2nd pronouns2doc)3x5\u21e21 if \u201c!\u201d2doc0 otherwise0x6log(word count of doc)ln(64)=4.15Let\u2019s assume for the moment that we\u2019ve already learned a real-valued weightfor each of these features, and that the 6 weights corresponding to the 6 featuresare[2.5,\u00005.0,\u00001.2,0.5,2.0,0.7], whileb= 0.1. (We\u2019ll discuss in the next sectionhow the weights are learned.) The weightw1, for example indicates how importantSuppose w =b = 0.1 Classifying sentiment for input x5.1\u2022CLASSIFICATION:THE SIGMOID5 It's hokey . There are virtually no surprises , and the writing is second-rate . So why was it so enjoyable  ? For one thing , the cast is great . Another nice touch is the music . I was overcome with the urge to get off the couch and start dancing .  It sucked me in , and it'll do the same to you  .x1=3x6=4.19x3=1x4=3x5=0x2=2\nFigure 5.2A sample mini test document showing the extracted features in the vectorx.Given these 6 features and the input reviewx,P(+|x)andP(\u0000|x)can be com-puted using Eq.5.5:p(+|x)=P(Y=1|x)=s(w\u00b7x+b)=s([2.5,\u00005.0,\u00001.2,0.5,2.0,0.7]\u00b7[3,2,1,3,0,4.19]+0.1)=s(.833)=0.70(5.6)p(\u0000|x)=P(Y=0|x)=1\u0000s(w\u00b7x+b)=0.30Logistic regression is commonly applied to all sorts of NLP tasks, and any propertyof the input can be a feature. Consider the task ofperiod disambiguation: decidingif a period is the end of a sentence or part of a word, by classifying each periodinto one of two classes EOS (end-of-sentence) and not-EOS. We might use featureslikex1below expressing that the current word is lower case and the class is EOS(perhaps with a positive weight), or that the current word is in our abbreviationsdictionary (\u201cProf.\u201d) and the class is EOS (perhaps with a negative weight). A featurecan also express a quite complex combination of properties. For example a periodfollowing an upper case word is likely to be an EOS, but if the word itself isSt.andthe previous word is capitalized, then the period is likely part of a shortening of thewordstreet.x1=\u21e21 if \u201cCase(wi)=Lower\u201d0 otherwisex2=\u21e21 if \u201cwi2AcronymDict\u201d0 otherwisex3=\u21e21 if \u201cwi=St. &Case(wi\u00001)=Cap\u201d0 otherwiseDesigning features:Features are generally designed by examining the trainingset with an eye to linguistic intuitions and the linguistic literature on the domain. Acareful error analysis on the training set or devset of an early version of a systemoften provides insights into features.For some tasks it is especially helpful to build complex features that are combi-nations of more primitive features. We saw such a feature for period disambiguationabove, where a period on the wordSt.was less likely to be the end of the sentenceif the previous word was capitalized. For logistic regression and naive Bayes thesecombination features orfeature interactionshave to be designed by hand.featureinteractions325.1\u2022CLASSIFICATION:THE SIGMOID5 It's hokey . There are virtually no surprises , and the writing is second-rate . So why was it so enjoyable  ? For one thing , the cast is great . Another nice touch is the music . I was overcome with the urge to get off the couch and start dancing .  It sucked me in , and it'll do the same to you  .x1=3x6=4.19x3=1x4=3x5=0x2=2\nFigure 5.2A sample mini test document showing the extracted features in the vectorx.Given these 6 features and the input reviewx,P(+|x)andP(\u0000|x)can be com-puted using Eq.5.5:p(+|x)=P(Y=1|x)=s(w\u00b7x+b)=s([2.5,\u00005.0,\u00001.2,0.5,2.0,0.7]\u00b7[3,2,1,3,0,4.19]+0.1)=s(.833)=0.70(5.6)p(\u0000|x)=P(Y=0|x)=1\u0000s(w\u00b7x+b)=0.30Logistic regression is commonly applied to all sorts of NLP tasks, and any propertyof the input can be a feature. Consider the task ofperiod disambiguation: decidingif a period is the end of a sentence or part of a word, by classifying each periodinto one of two classes EOS (end-of-sentence) and not-EOS. We might use featureslikex1below expressing that the current word is lower case and the class is EOS(perhaps with a positive weight), or that the current word is in our abbreviationsdictionary (\u201cProf.\u201d) and the class is EOS (perhaps with a negative weight). A featurecan also express a quite complex combination of properties. For example a periodfollowing an upper case word is likely to be an EOS, but if the word itself isSt.andthe previous word is capitalized, then the period is likely part of a shortening of thewordstreet.x1=\u21e21 if \u201cCase(wi)=Lower\u201d0 otherwisex2=\u21e21 if \u201cwi2AcronymDict\u201d0 otherwisex3=\u21e21 if \u201cwi=St. &Case(wi\u00001)=Cap\u201d0 otherwiseDesigning features:Features are generally designed by examining the trainingset with an eye to linguistic intuitions and the linguistic literature on the domain. Acareful error analysis on the training set or devset of an early version of a systemoften provides insights into features.For some tasks it is especially helpful to build complex features that are combi-nations of more primitive features. We saw such a feature for period disambiguationabove, where a period on the wordSt.was less likely to be the end of the sentenceif the previous word was capitalized. For logistic regression and naive Bayes thesecombination features orfeature interactionshave to be designed by hand.featureinteractions We can build features for logistic regression for any classification task: period disambiguation5.1\u2022CLASSIFICATION:THE SIGMOID5 It's hokey. There are virtually no surprises , and the writing is second-rate . So why was it so enjoyable? For one thing , the cast is great . Another nice touch is the music . I was overcome with the urge to get off the couch and start dancing .  It sucked me in , and it'll do the same to you .x1=3x6=4.15x3=1x4=3x5=0x2=2\nFigure 5.2A sample mini test document showing the extracted features in the vectorx.a feature the number of positive lexicon words (great,nice,enjoyable, etc.) is toa positive sentiment decision, whilew2tells us the importance of negative lexiconwords. Note thatw1=2.5 is positive, whilew2=\u00005.0, meaning that negative wordsare negatively associated with a positive sentiment decision, and are about twice asimportant as positive words.Given these 6 features and the input reviewx,P(+|x)andP(\u0000|x)can be com-puted using Eq.5.5:p(+|x)=P(Y=1|x)=s(w\u00b7x+b)=s([2.5,\u00005.0,\u00001.2,0.5,2.0,0.7]\u00b7[3,2,1,3,0,4.15]+0.1)=s(1.805)=0.86p(\u0000|x)=P(Y=0|x)=1\u0000s(w\u00b7x+b)=0.14Logistic regression is commonly applied to all sorts of NLP tasks, and any prop-erty of the input can be a feature. Consider the task ofperiod disambiguation:deciding if a period is the end of a sentence or part of a word, by classifying eachperiod into one of two classes EOS (end-of-sentence) and not-EOS. We might usefeatures likex1below expressing that the current word is lower case and the classis EOS (perhaps with a positive weight), or that the current word is in our abbrevia-tions dictionary (\u201cProf.\u201d) and the class is EOS (perhaps with a negative weight). Afeature can also express a quite complex combination of properties. For example aperiod following a upper cased word is a likely to be an EOS, but if the word itself isSt.and the previous word is capitalized, then the period is likely part of a shorteningof the wordstreet.x1=\u21e21 if \u201cCase(wi)=Lower\u201d0 otherwisex2=\u21e21 if \u201cwi2AcronymDict\u201d0 otherwisex3=\u21e21 if \u201cwi=St. &Case(wi\u00001)=Cap\u201d0 otherwiseDesigning features:Features are generally designed by examining the trainingset with an eye to linguistic intuitions and the linguistic literature on the domain. Acareful error analysis on the training or dev set. of an early version of a system oftenprovides insights into features.33This ends in a period.The house at 465 Main St. is new.End of sentenceNot end Classification in (binary)logistic regression: summaryGiven:\u25e6a set of classes:  (+ sentiment,-sentiment)\u25e6a vector xof features [x1, x2, \u2026, xn]\u25e6x1= count( \"awesome\")\u25e6x2 = log(number of words in review)\u25e6A vector wof weights  [w1, w2, \u2026, wn]\u25e6wifor each feature fi4CHAPTER5\u2022LOGISTICREGRESSIONThe sigmoid has a number of advantages; it take a real-valued number and mapsit into the range[0,1], which is just what we want for a probability. Because it isnearly linear around 0 but has a sharp slope toward the ends, it tends to squash outliervalues toward 0 or 1. And it\u2019s differentiable, which as we\u2019ll see in Section5.8willbe handy for learning.We\u2019re almost there. If we apply the sigmoid to the sum of the weighted features,we get a number between 0 and 1. To make it a probability, we just need to makesure that the two cases,p(y=1)andp(y=0), sum to 1. We can do this as follows:P(y=1)=s(w\u00b7x+b)=11+e\u0000(w\u00b7x+b)P(y=0)=1\u0000s(w\u00b7x+b)=1\u000011+e\u0000(w\u00b7x+b)=e\u0000(w\u00b7x+b)1+e\u0000(w\u00b7x+b)(5.5)Now we have an algorithm that given an instancexcomputes the probabilityP(y=1|x). How do we make a decision? For a test instancex, we say yes if theprobabilityP(y=1|x)is more than .5, and no otherwise. We call .5 thedecisionboundary:decisionboundary\u02c6y=\u21e21 ifP(y=1|x)>0.50 otherwise5.1.1 Example: sentiment classi\ufb01cationLet\u2019s have an example. Suppose we are doing binary sentiment classi\ufb01cation onmovie review text, and we would like to know whether to assign the sentiment class+or\u0000to a review documentdoc. We\u2019ll represent each input observation by thefollowing 6 featuresx1...x6of the input; Fig.5.2shows the features in a sample minitest document.Var De\ufb01nition Value in Fig.5.2x1count(positive lexicon)2doc)3x2count(negative lexicon)2doc)2x3\u21e21 if \u201cno\u201d2doc0 otherwise1x4count(1st and 2nd pronouns2doc)3x5\u21e21 if \u201c!\u201d2doc0 otherwise0x6log(word count of doc)ln(64)=4.15Let\u2019s assume for the moment that we\u2019ve already learned a real-valued weightfor each of these features, and that the 6 weights corresponding to the 6 featuresare[2.5,\u00005.0,\u00001.2,0.5,2.0,0.7], whileb= 0.1. (We\u2019ll discuss in the next sectionhow the weights are learned.) The weightw1, for example indicates how important Logistic RegressionLogistic Regression: a text example on sentiment classification Logistic RegressionLearning: Cross-Entropy Loss Wait, where did the W\u2019s come from?Supervised classification: \u2022We know the correct label y(either 0 or 1) for each x. \u2022But what the system produces is an estimate, !\ud835\udc66We want to set w and bto minimize the distancebetween our estimate !\ud835\udc66(i)and the true y(i). \u2022We need a distance estimator: a loss function or a cost function\u2022We need an optimization algorithm to update wand bto minimize the loss.37 Learning componentsA loss function:\u25e6cross-entropy lossAn optimization algorithm:\u25e6stochastic gradient descent The distance between !\ud835\udc66and yWe want to know how far is the classifier output:!\ud835\udc66= \u03c3(w\u00b7x+b)from the true output:y        [= either 0 or 1]We'll call this difference:L(!\ud835\udc66,y) = how much !\ud835\udc66differs from the true y  Intuition of negative log likelihood loss= cross-entropy lossA case of conditional maximum likelihood estimation We choose the parameters w,bthat maximize\u2022the log probability \u2022of the true y labels in the training data \u2022given the observations x Deriving cross-entropy loss for a single observation xGoal: maximize probability of the correct label p(y|x) Since there are only 2 discrete outcomes (0 or 1) we can express the probability p(y|x) from our classifier (the thing we want to maximize) asnoting:if y=1, this simplifies to !\ud835\udc66if y=0, this simplifies to 1-!\ud835\udc665.3\u2022THE CROSS-ENTROPY LOSS FUNCTION7thecross-entropy loss.The second thing we need is an optimization algorithm for iteratively updatingthe weights so as to minimize this loss function. The standard algorithm for this isgradient descent; we\u2019ll introduce thestochastic gradient descentalgorithm in thefollowing section.5.3 The cross-entropy loss functionWe need a loss function that expresses, for an observationx, how close the classi\ufb01eroutput ( \u02c6y=s(w\u00b7x+b)) is to the correct output (y, which is 0 or 1). We\u2019ll call this:L(\u02c6y,y)=How much \u02c6ydiffers from the truey(5.8)We do this via a loss function that prefers the correct class labels of the train-ing examples to bemore likely. This is calledconditional maximum likelihoodestimation: we choose the parametersw,bthatmaximize the log probability ofthe trueylabels in the training datagiven the observationsx. The resulting lossfunction is the negative log likelihood loss, generally called thecross-entropy loss.cross-entropylossLet\u2019s derive this loss function, applied to a single observationx. We\u2019d like tolearn weights that maximize the probability of the correct labelp(y|x). Since thereare only two discrete outcomes (1 or 0), this is a Bernoulli distribution, and we canexpress the probabilityp(y|x)that our classi\ufb01er produces for one observation asthe following (keeping in mind that if y=1, Eq.5.9simpli\ufb01es to \u02c6y; if y=0, Eq.5.9simpli\ufb01es to 1\u0000\u02c6y):p(y|x)=\u02c6yy(1\u0000\u02c6y)1\u0000y(5.9)Now we take the log of both sides. This will turn out to be handy mathematically,and doesn\u2019t hurt us; whatever values maximize a probability will also maximize thelog of the probability:logp(y|x)=log\u21e5\u02c6yy(1\u0000\u02c6y)1\u0000y\u21e4=ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)(5.10)Eq.5.10describes a log likelihood that should be maximized. In order to turn thisinto loss function (something that we need to minimize), we\u2019ll just \ufb02ip the sign onEq.5.10. The result is the cross-entropy lossLCE:LCE(\u02c6y,y)=\u0000logp(y|x)=\u0000[ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)](5.11)Finally, we can plug in the de\ufb01nition of \u02c6y=s(w\u00b7x+b):LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))](5.12)Let\u2019s see if this loss function does the right thing for our example from Fig.5.2.W ewant the loss to be smaller if the model\u2019s estimate is close to correct, and bigger ifthe model is confused. So \ufb01rst let\u2019s suppose the correct gold label for the sentimentexample in Fig.5.2is positive, i.e.,y=1. In this case our model is doing well, sincefrom Eq.5.7it indeed gave the example a higher probability of being positive (.69)than negative (.31). If we plugs(w\u00b7x+b)=.69 andy=1 into Eq.5.12, the right Deriving cross-entropy loss for a single observation xNow take the log of both sides (mathematically handy)Whatever values maximize log p(y|x) will also maximize p(y|x)5.3\u2022THE CROSS-ENTROPY LOSS FUNCTION7thecross-entropy loss.The second thing we need is an optimization algorithm for iteratively updatingthe weights so as to minimize this loss function. The standard algorithm for this isgradient descent; we\u2019ll introduce thestochastic gradient descentalgorithm in thefollowing section.5.3 The cross-entropy loss functionWe need a loss function that expresses, for an observationx, how close the classi\ufb01eroutput ( \u02c6y=s(w\u00b7x+b)) is to the correct output (y, which is 0 or 1). We\u2019ll call this:L(\u02c6y,y)=How much \u02c6ydiffers from the truey(5.8)We do this via a loss function that prefers the correct class labels of the train-ing examples to bemore likely. This is calledconditional maximum likelihoodestimation: we choose the parametersw,bthatmaximize the log probability ofthe trueylabels in the training datagiven the observationsx. The resulting lossfunction is the negative log likelihood loss, generally called thecross-entropy loss.cross-entropylossLet\u2019s derive this loss function, applied to a single observationx. We\u2019d like tolearn weights that maximize the probability of the correct labelp(y|x). Since thereare only two discrete outcomes (1 or 0), this is a Bernoulli distribution, and we canexpress the probabilityp(y|x)that our classi\ufb01er produces for one observation asthe following (keeping in mind that if y=1, Eq.5.9simpli\ufb01es to \u02c6y; if y=0, Eq.5.9simpli\ufb01es to 1\u0000\u02c6y):p(y|x)=\u02c6yy(1\u0000\u02c6y)1\u0000y(5.9)Now we take the log of both sides. This will turn out to be handy mathematically,and doesn\u2019t hurt us; whatever values maximize a probability will also maximize thelog of the probability:logp(y|x)=log\u21e5\u02c6yy(1\u0000\u02c6y)1\u0000y\u21e4=ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)(5.10)Eq.5.10describes a log likelihood that should be maximized. In order to turn thisinto loss function (something that we need to minimize), we\u2019ll just \ufb02ip the sign onEq.5.10. The result is the cross-entropy lossLCE:LCE(\u02c6y,y)=\u0000logp(y|x)=\u0000[ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)](5.11)Finally, we can plug in the de\ufb01nition of \u02c6y=s(w\u00b7x+b):LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))](5.12)Let\u2019s see if this loss function does the right thing for our example from Fig.5.2.W ewant the loss to be smaller if the model\u2019s estimate is close to correct, and bigger ifthe model is confused. So \ufb01rst let\u2019s suppose the correct gold label for the sentimentexample in Fig.5.2is positive, i.e.,y=1. In this case our model is doing well, sincefrom Eq.5.7it indeed gave the example a higher probability of being positive (.69)than negative (.31). If we plugs(w\u00b7x+b)=.69 andy=1 into Eq.5.12, the right5.3\u2022THE CROSS-ENTROPY LOSS FUNCTION7thecross-entropy loss.The second thing we need is an optimization algorithm for iteratively updatingthe weights so as to minimize this loss function. The standard algorithm for this isgradient descent; we\u2019ll introduce thestochastic gradient descentalgorithm in thefollowing section.5.3 The cross-entropy loss functionWe need a loss function that expresses, for an observationx, how close the classi\ufb01eroutput ( \u02c6y=s(w\u00b7x+b)) is to the correct output (y, which is 0 or 1). We\u2019ll call this:L(\u02c6y,y)=How much \u02c6ydiffers from the truey(5.8)We do this via a loss function that prefers the correct class labels of the train-ing examples to bemore likely. This is calledconditional maximum likelihoodestimation: we choose the parametersw,bthatmaximize the log probability ofthe trueylabels in the training datagiven the observationsx. The resulting lossfunction is the negative log likelihood loss, generally called thecross-entropy loss.cross-entropylossLet\u2019s derive this loss function, applied to a single observationx. We\u2019d like tolearn weights that maximize the probability of the correct labelp(y|x). Since thereare only two discrete outcomes (1 or 0), this is a Bernoulli distribution, and we canexpress the probabilityp(y|x)that our classi\ufb01er produces for one observation asthe following (keeping in mind that if y=1, Eq.5.9simpli\ufb01es to \u02c6y; if y=0, Eq.5.9simpli\ufb01es to 1\u0000\u02c6y):p(y|x)=\u02c6yy(1\u0000\u02c6y)1\u0000y(5.9)Now we take the log of both sides. This will turn out to be handy mathematically,and doesn\u2019t hurt us; whatever values maximize a probability will also maximize thelog of the probability:logp(y|x)=log\u21e5\u02c6yy(1\u0000\u02c6y)1\u0000y\u21e4=ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)(5.10)Eq.5.10describes a log likelihood that should be maximized. In order to turn thisinto loss function (something that we need to minimize), we\u2019ll just \ufb02ip the sign onEq.5.10. The result is the cross-entropy lossLCE:LCE(\u02c6y,y)=\u0000logp(y|x)=\u0000[ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)](5.11)Finally, we can plug in the de\ufb01nition of \u02c6y=s(w\u00b7x+b):LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))](5.12)Let\u2019s see if this loss function does the right thing for our example from Fig.5.2.W ewant the loss to be smaller if the model\u2019s estimate is close to correct, and bigger ifthe model is confused. So \ufb01rst let\u2019s suppose the correct gold label for the sentimentexample in Fig.5.2is positive, i.e.,y=1. In this case our model is doing well, sincefrom Eq.5.7it indeed gave the example a higher probability of being positive (.69)than negative (.31). If we plugs(w\u00b7x+b)=.69 andy=1 into Eq.5.12, the rightGoal: maximize probability of the correct label p(y|x) Maximize:Maximize: Deriving cross-entropy loss for a single observation xNow flip sign to turn this into a loss: something to minimizeCross-entropy loss (because is formula for cross-entropy(y,!\ud835\udc66))Or, plugging in definition of !\ud835\udc66:5.3\u2022THE CROSS-ENTROPY LOSS FUNCTION7thecross-entropy loss.The second thing we need is an optimization algorithm for iteratively updatingthe weights so as to minimize this loss function. The standard algorithm for this isgradient descent; we\u2019ll introduce thestochastic gradient descentalgorithm in thefollowing section.5.3 The cross-entropy loss functionWe need a loss function that expresses, for an observationx, how close the classi\ufb01eroutput ( \u02c6y=s(w\u00b7x+b)) is to the correct output (y, which is 0 or 1). We\u2019ll call this:L(\u02c6y,y)=How much \u02c6ydiffers from the truey(5.8)We do this via a loss function that prefers the correct class labels of the train-ing examples to bemore likely. This is calledconditional maximum likelihoodestimation: we choose the parametersw,bthatmaximize the log probability ofthe trueylabels in the training datagiven the observationsx. The resulting lossfunction is the negative log likelihood loss, generally called thecross-entropy loss.cross-entropylossLet\u2019s derive this loss function, applied to a single observationx. We\u2019d like tolearn weights that maximize the probability of the correct labelp(y|x). Since thereare only two discrete outcomes (1 or 0), this is a Bernoulli distribution, and we canexpress the probabilityp(y|x)that our classi\ufb01er produces for one observation asthe following (keeping in mind that if y=1, Eq.5.9simpli\ufb01es to \u02c6y; if y=0, Eq.5.9simpli\ufb01es to 1\u0000\u02c6y):p(y|x)=\u02c6yy(1\u0000\u02c6y)1\u0000y(5.9)Now we take the log of both sides. This will turn out to be handy mathematically,and doesn\u2019t hurt us; whatever values maximize a probability will also maximize thelog of the probability:logp(y|x)=log\u21e5\u02c6yy(1\u0000\u02c6y)1\u0000y\u21e4=ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)(5.10)Eq.5.10describes a log likelihood that should be maximized. In order to turn thisinto loss function (something that we need to minimize), we\u2019ll just \ufb02ip the sign onEq.5.10. The result is the cross-entropy lossLCE:LCE(\u02c6y,y)=\u0000logp(y|x)=\u0000[ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)](5.11)Finally, we can plug in the de\ufb01nition of \u02c6y=s(w\u00b7x+b):LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))](5.12)Let\u2019s see if this loss function does the right thing for our example from Fig.5.2.W ewant the loss to be smaller if the model\u2019s estimate is close to correct, and bigger ifthe model is confused. So \ufb01rst let\u2019s suppose the correct gold label for the sentimentexample in Fig.5.2is positive, i.e.,y=1. In this case our model is doing well, sincefrom Eq.5.7it indeed gave the example a higher probability of being positive (.69)than negative (.31). If we plugs(w\u00b7x+b)=.69 andy=1 into Eq.5.12, the rightGoal: maximize probability of the correct label p(y|x) Maximize:Minimize:5.3\u2022THE CROSS-ENTROPY LOSS FUNCTION7thecross-entropy loss.The second thing we need is an optimization algorithm for iteratively updatingthe weights so as to minimize this loss function. The standard algorithm for this isgradient descent; we\u2019ll introduce thestochastic gradient descentalgorithm in thefollowing section.5.3 The cross-entropy loss functionWe need a loss function that expresses, for an observationx, how close the classi\ufb01eroutput ( \u02c6y=s(w\u00b7x+b)) is to the correct output (y, which is 0 or 1). We\u2019ll call this:L(\u02c6y,y)=How much \u02c6ydiffers from the truey(5.8)We do this via a loss function that prefers the correct class labels of the train-ing examples to bemore likely. This is calledconditional maximum likelihoodestimation: we choose the parametersw,bthatmaximize the log probability ofthe trueylabels in the training datagiven the observationsx. The resulting lossfunction is the negative log likelihood loss, generally called thecross-entropy loss.cross-entropylossLet\u2019s derive this loss function, applied to a single observationx. We\u2019d like tolearn weights that maximize the probability of the correct labelp(y|x). Since thereare only two discrete outcomes (1 or 0), this is a Bernoulli distribution, and we canexpress the probabilityp(y|x)that our classi\ufb01er produces for one observation asthe following (keeping in mind that if y=1, Eq.5.9simpli\ufb01es to \u02c6y; if y=0, Eq.5.9simpli\ufb01es to 1\u0000\u02c6y):p(y|x)=\u02c6yy(1\u0000\u02c6y)1\u0000y(5.9)Now we take the log of both sides. This will turn out to be handy mathematically,and doesn\u2019t hurt us; whatever values maximize a probability will also maximize thelog of the probability:logp(y|x)=log\u21e5\u02c6yy(1\u0000\u02c6y)1\u0000y\u21e4=ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)(5.10)Eq.5.10describes a log likelihood that should be maximized. In order to turn thisinto loss function (something that we need to minimize), we\u2019ll just \ufb02ip the sign onEq.5.10. The result is the cross-entropy lossLCE:LCE(\u02c6y,y)=\u0000logp(y|x)=\u0000[ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)](5.11)Finally, we can plug in the de\ufb01nition of \u02c6y=s(w\u00b7x+b):LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))](5.12)Let\u2019s see if this loss function does the right thing for our example from Fig.5.2.W ewant the loss to be smaller if the model\u2019s estimate is close to correct, and bigger ifthe model is confused. So \ufb01rst let\u2019s suppose the correct gold label for the sentimentexample in Fig.5.2is positive, i.e.,y=1. In this case our model is doing well, sincefrom Eq.5.7it indeed gave the example a higher probability of being positive (.69)than negative (.31). If we plugs(w\u00b7x+b)=.69 andy=1 into Eq.5.12, the right5.3\u2022THE CROSS-ENTROPY LOSS FUNCTION7thecross-entropy loss.The second thing we need is an optimization algorithm for iteratively updatingthe weights so as to minimize this loss function. The standard algorithm for this isgradient descent; we\u2019ll introduce thestochastic gradient descentalgorithm in thefollowing section.5.3 The cross-entropy loss functionWe need a loss function that expresses, for an observationx, how close the classi\ufb01eroutput ( \u02c6y=s(w\u00b7x+b)) is to the correct output (y, which is 0 or 1). We\u2019ll call this:L(\u02c6y,y)=How much \u02c6ydiffers from the truey(5.8)We do this via a loss function that prefers the correct class labels of the train-ing examples to bemore likely. This is calledconditional maximum likelihoodestimation: we choose the parametersw,bthatmaximize the log probability ofthe trueylabels in the training datagiven the observationsx. The resulting lossfunction is the negative log likelihood loss, generally called thecross-entropy loss.cross-entropylossLet\u2019s derive this loss function, applied to a single observationx. We\u2019d like tolearn weights that maximize the probability of the correct labelp(y|x). Since thereare only two discrete outcomes (1 or 0), this is a Bernoulli distribution, and we canexpress the probabilityp(y|x)that our classi\ufb01er produces for one observation asthe following (keeping in mind that if y=1, Eq.5.9simpli\ufb01es to \u02c6y; if y=0, Eq.5.9simpli\ufb01es to 1\u0000\u02c6y):p(y|x)=\u02c6yy(1\u0000\u02c6y)1\u0000y(5.9)Now we take the log of both sides. This will turn out to be handy mathematically,and doesn\u2019t hurt us; whatever values maximize a probability will also maximize thelog of the probability:logp(y|x)=log\u21e5\u02c6yy(1\u0000\u02c6y)1\u0000y\u21e4=ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)(5.10)Eq.5.10describes a log likelihood that should be maximized. In order to turn thisinto loss function (something that we need to minimize), we\u2019ll just \ufb02ip the sign onEq.5.10. The result is the cross-entropy lossLCE:LCE(\u02c6y,y)=\u0000logp(y|x)=\u0000[ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)](5.11)Finally, we can plug in the de\ufb01nition of \u02c6y=s(w\u00b7x+b):LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))](5.12)Let\u2019s see if this loss function does the right thing for our example from Fig.5.2.W ewant the loss to be smaller if the model\u2019s estimate is close to correct, and bigger ifthe model is confused. So \ufb01rst let\u2019s suppose the correct gold label for the sentimentexample in Fig.5.2is positive, i.e.,y=1. In this case our model is doing well, sincefrom Eq.5.7it indeed gave the example a higher probability of being positive (.69)than negative (.31). If we plugs(w\u00b7x+b)=.69 andy=1 into Eq.5.12, the right Let's see if this works for our sentiment exampleWe want loss to be:\u2022smaller if the model estimate is close to correct\u2022bigger if model is confusedLet's first suppose the true label of this is y=1 (positive)It's hokey . There are virtually no surprises , and the writing is second-rate .         So why was it so enjoyable ? For one thing , the cast is great . Another nice touch is the music . I was overcome with the urge to get off the couch and start dancing . It sucked me in , and it'll do the same to you . Let's see if this works for our sentiment exampleTrue value is y=1.  How well is our model doing?Pretty well!  What's the loss?5.1\u2022CLASSIFICATION:THE SIGMOID5 It's hokey . There are virtually no surprises , and the writing is second-rate . So why was it so enjoyable  ? For one thing , the cast is great . Another nice touch is the music . I was overcome with the urge to get off the couch and start dancing .  It sucked me in , and it'll do the same to you  .x1=3x6=4.19x3=1x4=3x5=0x2=2\nFigure 5.2A sample mini test document showing the extracted features in the vectorx.Given these 6 features and the input reviewx,P(+|x)andP(\u0000|x)can be com-puted using Eq.5.5:p(+|x)=P(Y=1|x)=s(w\u00b7x+b)=s([2.5,\u00005.0,\u00001.2,0.5,2.0,0.7]\u00b7[3,2,1,3,0,4.19]+0.1)=s(.833)=0.70(5.6)p(\u0000|x)=P(Y=0|x)=1\u0000s(w\u00b7x+b)=0.30Logistic regression is commonly applied to all sorts of NLP tasks, and any propertyof the input can be a feature. Consider the task ofperiod disambiguation: decidingif a period is the end of a sentence or part of a word, by classifying each periodinto one of two classes EOS (end-of-sentence) and not-EOS. We might use featureslikex1below expressing that the current word is lower case and the class is EOS(perhaps with a positive weight), or that the current word is in our abbreviationsdictionary (\u201cProf.\u201d) and the class is EOS (perhaps with a negative weight). A featurecan also express a quite complex combination of properties. For example a periodfollowing an upper case word is likely to be an EOS, but if the word itself isSt.andthe previous word is capitalized, then the period is likely part of a shortening of thewordstreet.x1=\u21e21 if \u201cCase(wi)=Lower\u201d0 otherwisex2=\u21e21 if \u201cwi2AcronymDict\u201d0 otherwisex3=\u21e21 if \u201cwi=St. &Case(wi\u00001)=Cap\u201d0 otherwiseDesigning features:Features are generally designed by examining the trainingset with an eye to linguistic intuitions and the linguistic literature on the domain. Acareful error analysis on the training set or devset of an early version of a systemoften provides insights into features.For some tasks it is especially helpful to build complex features that are combi-nations of more primitive features. We saw such a feature for period disambiguationabove, where a period on the wordSt.was less likely to be the end of the sentenceif the previous word was capitalized. For logistic regression and naive Bayes thesecombination features orfeature interactionshave to be designed by hand.featureinteractions8CHAPTER5\u2022LOGISTICREGRESSIONside of the equation drops out, leading to the following loss (we\u2019ll use log to meannatural log when the base is not speci\ufb01ed):LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))]=\u0000[logs(w\u00b7x+b)]=\u0000log(.70)=.36By contrast, let\u2019s pretend instead that the example in Fig.5.2was actually negative,i.e.,y=0 (perhaps the reviewer went on to say \u201cBut bottom line, the movie isterrible! I beg you not to see it!\u201d). In this case our model is confused and we\u2019d wantthe loss to be higher. Now if we plugy=0 and 1\u0000s(w\u00b7x+b)=.31 from Eq.5.7into Eq.5.12, the left side of the equation drops out:LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))]=\u0000[log(1\u0000s(w\u00b7x+b))]=\u0000log(.30)=1.2Sure enough, the loss for the \ufb01rst classi\ufb01er (.37) is less than the loss for the secondclassi\ufb01er (1.17).Why does minimizing this negative log probability do what we want? A per-fect classi\ufb01er would assign probability 1 to the correct outcome (y=1 or y=0) andprobability 0 to the incorrect outcome. That means the higher \u02c6y(the closer it isto 1), the better the classi\ufb01er; the lower \u02c6yis (the closer it is to 0), the worse theclassi\ufb01er. The negative log of this probability is a convenient loss metric since itgoes from 0 (negative log of 1, no loss) to in\ufb01nity (negative log of 0, in\ufb01nite loss).This loss function also ensures that as the probability of the correct answer is max-imized, the probability of the incorrect answer is minimized; since the two sum toone, any increase in the probability of the correct answer is coming at the expenseof the incorrect answer. It\u2019s called the cross-entropy loss, because Eq.5.10is alsothe formula for thecross-entropybetween the true probability distributionyand ourestimated distribution \u02c6y.Now we know what we want to minimize; in the next section, we\u2019ll see how to\ufb01nd the minimum.5.4 Gradient DescentOur goal with gradient descent is to \ufb01nd the optimal weights: minimize the lossfunction we\u2019ve de\ufb01ned for the model. In Eq.5.13below, we\u2019ll explicitly representthe fact that the loss functionLis parameterized by the weights, which we\u2019ll referto in machine learning in general asq(in the case of logistic regressionq=w,b).So the goal is to \ufb01nd the set of weights which minimizes the loss function, averagedover all examples:\u02c6q=argminq1mmXi=1LCE(f(x(i);q),y(i))(5.13) Let's see if this works for our sentiment exampleSuppose true value instead  was y=0.  What's the loss?8CHAPTER5\u2022LOGISTICREGRESSIONside of the equation drops out, leading to the following loss (we\u2019ll use log to meannatural log when the base is not speci\ufb01ed):LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))]=\u0000[logs(w\u00b7x+b)]=\u0000log(.70)=.36By contrast, let\u2019s pretend instead that the example in Fig.5.2was actually negative,i.e.,y=0 (perhaps the reviewer went on to say \u201cBut bottom line, the movie isterrible! I beg you not to see it!\u201d). In this case our model is confused and we\u2019d wantthe loss to be higher. Now if we plugy=0 and 1\u0000s(w\u00b7x+b)=.31 from Eq.5.7into Eq.5.12, the left side of the equation drops out:LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))]=\u0000[log(1\u0000s(w\u00b7x+b))]=\u0000log(.30)=1.2Sure enough, the loss for the \ufb01rst classi\ufb01er (.37) is less than the loss for the secondclassi\ufb01er (1.17).Why does minimizing this negative log probability do what we want? A per-fect classi\ufb01er would assign probability 1 to the correct outcome (y=1 or y=0) andprobability 0 to the incorrect outcome. That means the higher \u02c6y(the closer it isto 1), the better the classi\ufb01er; the lower \u02c6yis (the closer it is to 0), the worse theclassi\ufb01er. The negative log of this probability is a convenient loss metric since itgoes from 0 (negative log of 1, no loss) to in\ufb01nity (negative log of 0, in\ufb01nite loss).This loss function also ensures that as the probability of the correct answer is max-imized, the probability of the incorrect answer is minimized; since the two sum toone, any increase in the probability of the correct answer is coming at the expenseof the incorrect answer. It\u2019s called the cross-entropy loss, because Eq.5.10is alsothe formula for thecross-entropybetween the true probability distributionyand ourestimated distribution \u02c6y.Now we know what we want to minimize; in the next section, we\u2019ll see how to\ufb01nd the minimum.5.4 Gradient DescentOur goal with gradient descent is to \ufb01nd the optimal weights: minimize the lossfunction we\u2019ve de\ufb01ned for the model. In Eq.5.13below, we\u2019ll explicitly representthe fact that the loss functionLis parameterized by the weights, which we\u2019ll referto in machine learning in general asq(in the case of logistic regressionq=w,b).So the goal is to \ufb01nd the set of weights which minimizes the loss function, averagedover all examples:\u02c6q=argminq1mmXi=1LCE(f(x(i);q),y(i))(5.13)5.1\u2022CLASSIFICATION:THE SIGMOID5 It's hokey . There are virtually no surprises , and the writing is second-rate . So why was it so enjoyable  ? For one thing , the cast is great . Another nice touch is the music . I was overcome with the urge to get off the couch and start dancing .  It sucked me in , and it'll do the same to you  .x1=3x6=4.19x3=1x4=3x5=0x2=2\nFigure 5.2A sample mini test document showing the extracted features in the vectorx.Given these 6 features and the input reviewx,P(+|x)andP(\u0000|x)can be com-puted using Eq.5.5:p(+|x)=P(Y=1|x)=s(w\u00b7x+b)=s([2.5,\u00005.0,\u00001.2,0.5,2.0,0.7]\u00b7[3,2,1,3,0,4.19]+0.1)=s(.833)=0.70(5.6)p(\u0000|x)=P(Y=0|x)=1\u0000s(w\u00b7x+b)=0.30Logistic regression is commonly applied to all sorts of NLP tasks, and any propertyof the input can be a feature. Consider the task ofperiod disambiguation: decidingif a period is the end of a sentence or part of a word, by classifying each periodinto one of two classes EOS (end-of-sentence) and not-EOS. We might use featureslikex1below expressing that the current word is lower case and the class is EOS(perhaps with a positive weight), or that the current word is in our abbreviationsdictionary (\u201cProf.\u201d) and the class is EOS (perhaps with a negative weight). A featurecan also express a quite complex combination of properties. For example a periodfollowing an upper case word is likely to be an EOS, but if the word itself isSt.andthe previous word is capitalized, then the period is likely part of a shortening of thewordstreet.x1=\u21e21 if \u201cCase(wi)=Lower\u201d0 otherwisex2=\u21e21 if \u201cwi2AcronymDict\u201d0 otherwisex3=\u21e21 if \u201cwi=St. &Case(wi\u00001)=Cap\u201d0 otherwiseDesigning features:Features are generally designed by examining the trainingset with an eye to linguistic intuitions and the linguistic literature on the domain. Acareful error analysis on the training set or devset of an early version of a systemoften provides insights into features.For some tasks it is especially helpful to build complex features that are combi-nations of more primitive features. We saw such a feature for period disambiguationabove, where a period on the wordSt.was less likely to be the end of the sentenceif the previous word was capitalized. For logistic regression and naive Bayes thesecombination features orfeature interactionshave to be designed by hand.featureinteractions Let's see if this works for our sentiment exampleThe loss when model was right (if true y=1) Is lower than the loss when model was wrong (if true y=0):Sure enough, loss was bigger when model was wrong!8CHAPTER5\u2022LOGISTICREGRESSIONside of the equation drops out, leading to the following loss (we\u2019ll use log to meannatural log when the base is not speci\ufb01ed):LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))]=\u0000[logs(w\u00b7x+b)]=\u0000log(.70)=.36By contrast, let\u2019s pretend instead that the example in Fig.5.2was actually negative,i.e.,y=0 (perhaps the reviewer went on to say \u201cBut bottom line, the movie isterrible! I beg you not to see it!\u201d). In this case our model is confused and we\u2019d wantthe loss to be higher. Now if we plugy=0 and 1\u0000s(w\u00b7x+b)=.31 from Eq.5.7into Eq.5.12, the left side of the equation drops out:LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))]=\u0000[log(1\u0000s(w\u00b7x+b))]=\u0000log(.30)=1.2Sure enough, the loss for the \ufb01rst classi\ufb01er (.37) is less than the loss for the secondclassi\ufb01er (1.17).Why does minimizing this negative log probability do what we want? A per-fect classi\ufb01er would assign probability 1 to the correct outcome (y=1 or y=0) andprobability 0 to the incorrect outcome. That means the higher \u02c6y(the closer it isto 1), the better the classi\ufb01er; the lower \u02c6yis (the closer it is to 0), the worse theclassi\ufb01er. The negative log of this probability is a convenient loss metric since itgoes from 0 (negative log of 1, no loss) to in\ufb01nity (negative log of 0, in\ufb01nite loss).This loss function also ensures that as the probability of the correct answer is max-imized, the probability of the incorrect answer is minimized; since the two sum toone, any increase in the probability of the correct answer is coming at the expenseof the incorrect answer. It\u2019s called the cross-entropy loss, because Eq.5.10is alsothe formula for thecross-entropybetween the true probability distributionyand ourestimated distribution \u02c6y.Now we know what we want to minimize; in the next section, we\u2019ll see how to\ufb01nd the minimum.5.4 Gradient DescentOur goal with gradient descent is to \ufb01nd the optimal weights: minimize the lossfunction we\u2019ve de\ufb01ned for the model. In Eq.5.13below, we\u2019ll explicitly representthe fact that the loss functionLis parameterized by the weights, which we\u2019ll referto in machine learning in general asq(in the case of logistic regressionq=w,b).So the goal is to \ufb01nd the set of weights which minimizes the loss function, averagedover all examples:\u02c6q=argminq1mmXi=1LCE(f(x(i);q),y(i))(5.13)8CHAPTER5\u2022LOGISTICREGRESSIONside of the equation drops out, leading to the following loss (we\u2019ll use log to meannatural log when the base is not speci\ufb01ed):LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))]=\u0000[logs(w\u00b7x+b)]=\u0000log(.70)=.36By contrast, let\u2019s pretend instead that the example in Fig.5.2was actually negative,i.e.,y=0 (perhaps the reviewer went on to say \u201cBut bottom line, the movie isterrible! I beg you not to see it!\u201d). In this case our model is confused and we\u2019d wantthe loss to be higher. Now if we plugy=0 and 1\u0000s(w\u00b7x+b)=.31 from Eq.5.7into Eq.5.12, the left side of the equation drops out:LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))]=\u0000[log(1\u0000s(w\u00b7x+b))]=\u0000log(.30)=1.2Sure enough, the loss for the \ufb01rst classi\ufb01er (.37) is less than the loss for the secondclassi\ufb01er (1.17).Why does minimizing this negative log probability do what we want? A per-fect classi\ufb01er would assign probability 1 to the correct outcome (y=1 or y=0) andprobability 0 to the incorrect outcome. That means the higher \u02c6y(the closer it isto 1), the better the classi\ufb01er; the lower \u02c6yis (the closer it is to 0), the worse theclassi\ufb01er. The negative log of this probability is a convenient loss metric since itgoes from 0 (negative log of 1, no loss) to in\ufb01nity (negative log of 0, in\ufb01nite loss).This loss function also ensures that as the probability of the correct answer is max-imized, the probability of the incorrect answer is minimized; since the two sum toone, any increase in the probability of the correct answer is coming at the expenseof the incorrect answer. It\u2019s called the cross-entropy loss, because Eq.5.10is alsothe formula for thecross-entropybetween the true probability distributionyand ourestimated distribution \u02c6y.Now we know what we want to minimize; in the next section, we\u2019ll see how to\ufb01nd the minimum.5.4 Gradient DescentOur goal with gradient descent is to \ufb01nd the optimal weights: minimize the lossfunction we\u2019ve de\ufb01ned for the model. In Eq.5.13below, we\u2019ll explicitly representthe fact that the loss functionLis parameterized by the weights, which we\u2019ll referto in machine learning in general asq(in the case of logistic regressionq=w,b).So the goal is to \ufb01nd the set of weights which minimizes the loss function, averagedover all examples:\u02c6q=argminq1mmXi=1LCE(f(x(i);q),y(i))(5.13) Logistic RegressionCross-Entropy Loss Logistic RegressionStochastic Gradient Descent Our goal: minimize the lossLet's make explicit that the loss function is parameterized by weights \ud835\udef3=(w,b)\u2022And we\u2019ll represent !\ud835\udc66as f (x; \u03b8 ) to make the dependence on \u03b8 more obviousWe want the weights that minimize the loss, averaged over all examples:8CHAPTER5\u2022LOGISTICREGRESSIONside of the equation drops out, leading to the following loss (we\u2019ll use log to meannatural log when the base is not speci\ufb01ed):LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))]=\u0000[logs(w\u00b7x+b)]=\u0000log(.70)=.36By contrast, let\u2019s pretend instead that the example in Fig.5.2was actually negative,i.e.,y=0 (perhaps the reviewer went on to say \u201cBut bottom line, the movie isterrible! I beg you not to see it!\u201d). In this case our model is confused and we\u2019d wantthe loss to be higher. Now if we plugy=0 and 1\u0000s(w\u00b7x+b)=.31 from Eq.5.7into Eq.5.12, the left side of the equation drops out:LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))]=\u0000[log(1\u0000s(w\u00b7x+b))]=\u0000log(.30)=1.2Sure enough, the loss for the \ufb01rst classi\ufb01er (.37) is less than the loss for the secondclassi\ufb01er (1.17).Why does minimizing this negative log probability do what we want? A per-fect classi\ufb01er would assign probability 1 to the correct outcome (y=1 or y=0) andprobability 0 to the incorrect outcome. That means the higher \u02c6y(the closer it isto 1), the better the classi\ufb01er; the lower \u02c6yis (the closer it is to 0), the worse theclassi\ufb01er. The negative log of this probability is a convenient loss metric since itgoes from 0 (negative log of 1, no loss) to in\ufb01nity (negative log of 0, in\ufb01nite loss).This loss function also ensures that as the probability of the correct answer is max-imized, the probability of the incorrect answer is minimized; since the two sum toone, any increase in the probability of the correct answer is coming at the expenseof the incorrect answer. It\u2019s called the cross-entropy loss, because Eq.5.10is alsothe formula for thecross-entropybetween the true probability distributionyand ourestimated distribution \u02c6y.Now we know what we want to minimize; in the next section, we\u2019ll see how to\ufb01nd the minimum.5.4 Gradient DescentOur goal with gradient descent is to \ufb01nd the optimal weights: minimize the lossfunction we\u2019ve de\ufb01ned for the model. In Eq.5.13below, we\u2019ll explicitly representthe fact that the loss functionLis parameterized by the weights, which we\u2019ll referto in machine learning in general asq(in the case of logistic regressionq=w,b).So the goal is to \ufb01nd the set of weights which minimizes the loss function, averagedover all examples:\u02c6q=argminq1mmXi=1LCE(f(x(i);q),y(i))(5.13) Intuition of gradient descentHow do I get to the bottom of this river canyon?\nxLook around me 360\u2218Find the direction of steepest slope downGo that way Our goal: minimize the lossFor logistic regression, loss function is convex\u2022A convex function has just one minimum\u2022Gradient descent starting from any point is guaranteed to find the minimum\u2022(Loss for neural networks is non-convex) Let's first visualize for a single scalar w\nwLoss\n0w1wmin(goal)Should we move right or left from here?Q: Given current w, should we make it bigger or smaller?A: Move w in the reverse direction from the slope of the function  Let's first visualize for a single scalar w\nwLoss\n0w1wminslope of loss at w1 is negative(goal)Q: Given current w, should we make it bigger or smaller?A: Move w in the reverse direction from the slope of the function \nSo we'll move positive Let's first visualize for a single scalar w\nwLoss\n0w1wminslope of loss at w1 is negative(goal)one stepof gradientdescentQ: Given current w, should we make it bigger or smaller?A: Move w in the reverse direction from the slope of the function \nSo we'll move positive GradientsThe gradientof a function of many variables is a vector pointing in the direction of the greatest increase in a function. Gradient Descent: Find the gradient of the loss function at the current point and move in the oppositedirection.  How much do we move in that direction ?\u2022The value of the gradient (slope in our example)  !!\"\ud835\udc3f(\ud835\udc53\ud835\udc65;\ud835\udc64,\ud835\udc66)weighted by a learning rate \u03b7 \u2022Higher learning rate means move wfaster10CHAPTER5\u2022LOGISTICREGRESSIONexample):wt+1=wt\u0000hddwL(f(x;w),y)(5.14)Now let\u2019s extend the intuition from a function of one scalar variablewto manyvariables, because we don\u2019t just want to move left or right, we want to know wherein the N-dimensional space (of theNparameters that make upq) we should move.Thegradientis just such a vector; it expresses the directional components of thesharpest slope along each of thoseNdimensions. If we\u2019re just imagining two weightdimensions (say for one weightwand one biasb), the gradient might be a vector withtwo orthogonal components, each of which tells us how much the ground slopes inthewdimension and in thebdimension. Fig.5.4shows a visualization of the valueof a 2-dimensional gradient vector taken at the red point.\nCost(w,b)\nwbFigure 5.4Visualization of the gradient vector at the red point in two dimensionswandb,showing the gradient as a red arrow in the x-y plane.In an actual logistic regression, the parameter vectorwis much longer than 1 or2, since the input feature vectorxcan be quite long, and we need a weightwiforeachxi. For each dimension/variablewiinw(plus the biasb), the gradient will havea component that tells us the slope with respect to that variable. Essentially we\u2019reasking: \u201cHow much would a small change in that variablewiin\ufb02uence the total lossfunctionL?\u201dIn each dimensionwi, we express the slope as a partial derivative\u2202\u2202wiof the lossfunction. The gradient is then de\ufb01ned as a vector of these partials. We\u2019ll represent \u02c6yasf(x;q)to make the dependence onqmore obvious:\u2014qL(f(x;q),y)) =266664\u2202\u2202w1L(f(x;q),y)\u2202\u2202w2L(f(x;q),y)...\u2202\u2202wnL(f(x;q),y)377775(5.15)The \ufb01nal equation for updatingqbased on the gradient is thusqt+1=qt\u0000h\u2014L(f(x;q),y)(5.16) Now let's consider N dimensionsWe want to know where in the N-dimensional space (of the N parameters that make up \u03b8 ) we should move. The gradient is just such a vector; it expresses the directional components of the sharpest slope along each of the N dimensions.  Imagine 2 dimensions, w and bVisualizing the gradient vector at the red pointIt has two dimensions shown in the x-y plane10CHAPTER5\u2022LOGISTICREGRESSIONlearning rate times the gradient (or the slope, in our single-variable example):wt+1=wt\u0000hddwf(x;w)(5.14)Now let\u2019s extend the intuition from a function of one scalar variablewto manyvariables, because we don\u2019t just want to move left or right, we want to know wherein the N-dimensional space (of theNparameters that make upq) we should move.Thegradientis just such a vector; it expresses the directional components of thesharpest slope along each of thoseNdimensions. If we\u2019re just imagining two weightdimensions (say for one weightwand one biasb), the gradient might be a vector withtwo orthogonal components, each of which tells us how much the ground slopes inthewdimension and in thebdimension. Fig.5.4shows a visualization of the valueof a 2-dimensional gradient vector taken at the red point.\nCost(w,b)\nwbFigure 5.4Visualization of the gradient vector at the red point in two dimensionswandb,showing the gradient as a red arrow in the x-y plane.In an actual logistic regression, the parameter vectorwis much longer than 1 or2, since the input feature vectorxcan be quite long, and we need a weightwiforeachxi. For each dimension/variablewiinw(plus the biasb), the gradient will havea component that tells us the slope with respect to that variable. Essentially we\u2019reasking: \u201cHow much would a small change in that variablewiin\ufb02uence the total lossfunctionL?\u201dIn each dimensionwi, we express the slope as a partial derivative\u2202\u2202wiof the lossfunction. The gradient is then de\ufb01ned as a vector of these partials. We\u2019ll represent \u02c6yasf(x;q)to make the dependence onqmore obvious:\u2014qL(f(x;q),y)) =266664\u2202\u2202w1L(f(x;q),y)\u2202\u2202w2L(f(x;q),y)...\u2202\u2202wnL(f(x;q),y)377775(5.15)The \ufb01nal equation for updatingqbased on the gradient is thusqt+1=qt\u0000h\u2014L(f(x;q),y)(5.16) Real gradientsAre much longer; lots and lots of weightsFor each dimension withe gradient component itells us the slope with respect to that variable. \u25e6\u201cHow much would a small change in wiinfluence the total loss function L?\u201d \u25e6We express the slope as a partial derivative \u2202 of the loss \u2202wiThe gradient is then defined as a vector of these partials.  The gradient10CHAPTER5\u2022LOGISTICREGRESSIONlearning rate times the gradient (or the slope, in our single-variable example):wt+1=wt\u0000hddwf(x;w)(5.14)Now let\u2019s extend the intuition from a function of one scalar variablewto manyvariables, because we don\u2019t just want to move left or right, we want to know wherein the N-dimensional space (of theNparameters that make upq) we should move.Thegradientis just such a vector; it expresses the directional components of thesharpest slope along each of thoseNdimensions. If we\u2019re just imagining two weightdimensions (say for one weightwand one biasb), the gradient might be a vector withtwo orthogonal components, each of which tells us how much the ground slopes inthewdimension and in thebdimension. Fig.5.4shows a visualization of the valueof a 2-dimensional gradient vector taken at the red point.\nCost(w,b)\nwbFigure 5.4Visualization of the gradient vector at the red point in two dimensionswandb,showing the gradient as a red arrow in the x-y plane.In an actual logistic regression, the parameter vectorwis much longer than 1 or2, since the input feature vectorxcan be quite long, and we need a weightwiforeachxi. For each dimension/variablewiinw(plus the biasb), the gradient will havea component that tells us the slope with respect to that variable. Essentially we\u2019reasking: \u201cHow much would a small change in that variablewiin\ufb02uence the total lossfunctionL?\u201dIn each dimensionwi, we express the slope as a partial derivative\u2202\u2202wiof the lossfunction. The gradient is then de\ufb01ned as a vector of these partials. We\u2019ll represent \u02c6yasf(x;q)to make the dependence onqmore obvious:\u2014qL(f(x;q),y)) =266664\u2202\u2202w1L(f(x;q),y)\u2202\u2202w2L(f(x;q),y)...\u2202\u2202wnL(f(x;q),y)377775(5.15)The \ufb01nal equation for updatingqbased on the gradient is thusqt+1=qt\u0000h\u2014L(f(x;q),y)(5.16)We\u2019ll represent !\ud835\udc66as f (x; \u03b8 ) to make the dependence on \u03b8 more obvious: The final equation for updating \u03b8 based on the gradient is thus 10CHAPTER5\u2022LOGISTICREGRESSIONlearning rate times the gradient (or the slope, in our single-variable example):wt+1=wt\u0000hddwf(x;w)(5.14)Now let\u2019s extend the intuition from a function of one scalar variablewto manyvariables, because we don\u2019t just want to move left or right, we want to know wherein the N-dimensional space (of theNparameters that make upq) we should move.Thegradientis just such a vector; it expresses the directional components of thesharpest slope along each of thoseNdimensions. If we\u2019re just imagining two weightdimensions (say for one weightwand one biasb), the gradient might be a vector withtwo orthogonal components, each of which tells us how much the ground slopes inthewdimension and in thebdimension. Fig.5.4shows a visualization of the valueof a 2-dimensional gradient vector taken at the red point.\nCost(w,b)\nwbFigure 5.4Visualization of the gradient vector at the red point in two dimensionswandb,showing the gradient as a red arrow in the x-y plane.In an actual logistic regression, the parameter vectorwis much longer than 1 or2, since the input feature vectorxcan be quite long, and we need a weightwiforeachxi. For each dimension/variablewiinw(plus the biasb), the gradient will havea component that tells us the slope with respect to that variable. Essentially we\u2019reasking: \u201cHow much would a small change in that variablewiin\ufb02uence the total lossfunctionL?\u201dIn each dimensionwi, we express the slope as a partial derivative\u2202\u2202wiof the lossfunction. The gradient is then de\ufb01ned as a vector of these partials. We\u2019ll represent \u02c6yasf(x;q)to make the dependence onqmore obvious:\u2014qL(f(x;q),y)) =266664\u2202\u2202w1L(f(x;q),y)\u2202\u2202w2L(f(x;q),y)...\u2202\u2202wnL(f(x;q),y)377775(5.15)The \ufb01nal equation for updatingqbased on the gradient is thusqt+1=qt\u0000h\u2014L(f(x;q),y)(5.16) What are these partial derivatives for logistic regression?The loss function5.4\u2022GRADIENTDESCENT115.4.1 The Gradient for Logistic RegressionIn order to updateq, we need a de\ufb01nition for the gradient\u2014L(f(x;q),y). Recall thatfor logistic regression, the cross-entropy loss function is:LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))](5.17)It turns out that the derivative of this function for one observation vectorxisEq.5.18(the interested reader can see Section5.8for the derivation of this equation):\u2202LCE(\u02c6y,y)\u2202wj=[s(w\u00b7x+b)\u0000y]xj(5.18)Note in Eq.5.18that the gradient with respect to a single weightwjrepresents avery intuitive value: the difference between the trueyand our estimated \u02c6y=s(w\u00b7x+b)for that observation, multiplied by the corresponding input valuexj.5.4.2 The Stochastic Gradient Descent AlgorithmStochastic gradient descent is an online algorithm that minimizes the loss functionby computing its gradient after each training example, and nudgingqin the rightdirection (the opposite direction of the gradient). Fig.5.5shows the algorithm.functionSTOCHASTICGRADIENTDESCENT(L(),f(),x,y)returnsq# where: L is the loss function# f is a function parameterized byq# x is the set of training inputsx(1),x(2),. .. ,x(m)# y is the set of training outputs (labels)y(1),y(2),. .. ,y(m)q 0repeattil done # see captionFor each training tuple(x(i),y(i))(in random order)1. Optional (for reporting): # How are we doing on this tuple?Compute \u02c6y(i)=f(x(i);q)# What is our estimated output \u02c6y?Compute the lossL(\u02c6y(i),y(i))# How far off is \u02c6y(i))from the true outputy(i)?2.g \u2014qL(f(x(i);q),y(i))# How should we moveqto maximize loss?3.q q\u0000hg# Go the other way insteadreturnqFigure 5.5The stochastic gradient descent algorithm. Step 1 (computing the loss) is usedto report how well we are doing on the current tuple. The algorithm can terminate when itconverges (or when the gradient norm<\u270f), or when progress halts (for example when theloss starts going up on a held-out set).The learning ratehis ahyperparameterthat must be adjusted. If it\u2019s too high,hyperparameterthe learner will take steps that are too large, overshooting the minimum of the lossfunction. If it\u2019s too low, the learner will take steps that are too small, and take toolong to get to the minimum. It is common to start with a higher learning rate and thenslowly decrease it, so that it is a function of the iterationkof training; the notationhkcan be used to mean the value of the learning rate at iterationk.We\u2019ll discuss hyperparameters in more detail in Chapter 7, but brie\ufb02y they area special kind of parameter for any machine learning model. Unlike regular param-eters of a model (weights likewandb), which are learned by the algorithm fromthe training set, hyperparameters are special parameters chosen by the algorithmdesigner that affect how the algorithm works.5.4\u2022GRADIENTDESCENT115.4.1 The Gradient for Logistic RegressionIn order to updateq, we need a de\ufb01nition for the gradient\u2014L(f(x;q),y). Recall thatfor logistic regression, the cross-entropy loss function is:LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))](5.17)It turns out that the derivative of this function for one observation vectorxisEq.5.18(the interested reader can see Section5.8for the derivation of this equation):\u2202LCE(\u02c6y,y)\u2202wj=[s(w\u00b7x+b)\u0000y]xj(5.18)Note in Eq.5.18that the gradient with respect to a single weightwjrepresents avery intuitive value: the difference between the trueyand our estimated \u02c6y=s(w\u00b7x+b)for that observation, multiplied by the corresponding input valuexj.5.4.2 The Stochastic Gradient Descent AlgorithmStochastic gradient descent is an online algorithm that minimizes the loss functionby computing its gradient after each training example, and nudgingqin the rightdirection (the opposite direction of the gradient). Fig.5.5shows the algorithm.functionSTOCHASTICGRADIENTDESCENT(L(),f(),x,y)returnsq# where: L is the loss function# f is a function parameterized byq# x is the set of training inputsx(1),x(2),. .. ,x(m)# y is the set of training outputs (labels)y(1),y(2),. .. ,y(m)q 0repeattil done # see captionFor each training tuple(x(i),y(i))(in random order)1. Optional (for reporting): # How are we doing on this tuple?Compute \u02c6y(i)=f(x(i);q)# What is our estimated output \u02c6y?Compute the lossL(\u02c6y(i),y(i))# How far off is \u02c6y(i))from the true outputy(i)?2.g \u2014qL(f(x(i);q),y(i))# How should we moveqto maximize loss?3.q q\u0000hg# Go the other way insteadreturnqFigure 5.5The stochastic gradient descent algorithm. Step 1 (computing the loss) is usedto report how well we are doing on the current tuple. The algorithm can terminate when itconverges (or when the gradient norm<\u270f), or when progress halts (for example when theloss starts going up on a held-out set).The learning ratehis ahyperparameterthat must be adjusted. If it\u2019s too high,hyperparameterthe learner will take steps that are too large, overshooting the minimum of the lossfunction. If it\u2019s too low, the learner will take steps that are too small, and take toolong to get to the minimum. It is common to start with a higher learning rate and thenslowly decrease it, so that it is a function of the iterationkof training; the notationhkcan be used to mean the value of the learning rate at iterationk.We\u2019ll discuss hyperparameters in more detail in Chapter 7, but brie\ufb02y they area special kind of parameter for any machine learning model. Unlike regular param-eters of a model (weights likewandb), which are learned by the algorithm fromthe training set, hyperparameters are special parameters chosen by the algorithmdesigner that affect how the algorithm works.The elegant derivative of this function(see textbook 5.8 for derivation) 5.4\u2022GRADIENTDESCENT115.4.1 The Gradient for Logistic RegressionIn order to updateq, we need a de\ufb01nition for the gradient\u2014L(f(x;q),y). Recall thatfor logistic regression, the cross-entropy loss function is:LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))](5.17)It turns out that the derivative of this function for one observation vectorxisEq.5.18(the interested reader can see Section5.8for the derivation of this equation):\u2202LCE(\u02c6y,y)\u2202wj=[s(w\u00b7x+b)\u0000y]xj(5.18)Note in Eq.5.18that the gradient with respect to a single weightwjrepresents avery intuitive value: the difference between the trueyand our estimated \u02c6y=s(w\u00b7x+b)for that observation, multiplied by the corresponding input valuexj.5.4.2 The Stochastic Gradient Descent AlgorithmStochastic gradient descent is an online algorithm that minimizes the loss functionby computing its gradient after each training example, and nudgingqin the rightdirection (the opposite direction of the gradient). Fig.5.5shows the algorithm.functionSTOCHASTICGRADIENTDESCENT(L(),f(),x,y)returnsq# where: L is the loss function# f is a function parameterized byq# x is the set of training inputsx(1),x(2),. .. ,x(m)# y is the set of training outputs (labels)y(1),y(2),. .. ,y(m)q 0repeattil done # see captionFor each training tuple(x(i),y(i))(in random order)1. Optional (for reporting): # How are we doing on this tuple?Compute \u02c6y(i)=f(x(i);q)# What is our estimated output \u02c6y?Compute the lossL(\u02c6y(i),y(i))# How far off is \u02c6y(i))from the true outputy(i)?2.g \u2014qL(f(x(i);q),y(i))# How should we moveqto maximize loss?3.q q\u0000hg# Go the other way insteadreturnqFigure 5.5The stochastic gradient descent algorithm. Step 1 (computing the loss) is usedto report how well we are doing on the current tuple. The algorithm can terminate when itconverges (or when the gradient norm<\u270f), or when progress halts (for example when theloss starts going up on a held-out set).The learning ratehis ahyperparameterthat must be adjusted. If it\u2019s too high,hyperparameterthe learner will take steps that are too large, overshooting the minimum of the lossfunction. If it\u2019s too low, the learner will take steps that are too small, and take toolong to get to the minimum. It is common to start with a higher learning rate and thenslowly decrease it, so that it is a function of the iterationkof training; the notationhkcan be used to mean the value of the learning rate at iterationk.We\u2019ll discuss hyperparameters in more detail in Chapter 7, but brie\ufb02y they area special kind of parameter for any machine learning model. Unlike regular param-eters of a model (weights likewandb), which are learned by the algorithm fromthe training set, hyperparameters are special parameters chosen by the algorithmdesigner that affect how the algorithm works. HyperparametersThe learning rate \u03b7is a hyperparameter\u25e6too high: the learner will take big steps and overshoot\u25e6too low: the learner will take too longHyperparameters:\u2022Briefly, a special kind of parameter for an ML model\u2022Instead of being learned by algorithm from supervision (like regular parameters), they are chosen by algorithm designer. Logistic RegressionStochastic Gradient Descent Logistic RegressionStochastic Gradient Descent: An example and more details Working through an exampleOne step of gradient descentA mini-sentiment example, where the true y=1 (positive)Two features:x1= 3    (count of positive lexicon words) x2= 2    (count of negative lexicon words) Assume 3 parameters (2 weights and 1 bias) in \u03980are zero:w1= w2= b  = 0 \u03b7 = 0.1  Example of gradient descentUpdate step for update \u03b8 is:whereGradient vector has 3 dimensions:10CHAPTER5\u2022LOGISTICREGRESSIONlearning rate times the gradient (or the slope, in our single-variable example):wt+1=wt\u0000hddwf(x;w)(5.14)Now let\u2019s extend the intuition from a function of one scalar variablewto manyvariables, because we don\u2019t just want to move left or right, we want to know wherein the N-dimensional space (of theNparameters that make upq) we should move.Thegradientis just such a vector; it expresses the directional components of thesharpest slope along each of thoseNdimensions. If we\u2019re just imagining two weightdimensions (say for one weightwand one biasb), the gradient might be a vector withtwo orthogonal components, each of which tells us how much the ground slopes inthewdimension and in thebdimension. Fig.5.4shows a visualization of the valueof a 2-dimensional gradient vector taken at the red point.\nCost(w,b)\nwbFigure 5.4Visualization of the gradient vector at the red point in two dimensionswandb,showing the gradient as a red arrow in the x-y plane.In an actual logistic regression, the parameter vectorwis much longer than 1 or2, since the input feature vectorxcan be quite long, and we need a weightwiforeachxi. For each dimension/variablewiinw(plus the biasb), the gradient will havea component that tells us the slope with respect to that variable. Essentially we\u2019reasking: \u201cHow much would a small change in that variablewiin\ufb02uence the total lossfunctionL?\u201dIn each dimensionwi, we express the slope as a partial derivative\u2202\u2202wiof the lossfunction. The gradient is then de\ufb01ned as a vector of these partials. We\u2019ll represent \u02c6yasf(x;q)to make the dependence onqmore obvious:\u2014qL(f(x;q),y)) =266664\u2202\u2202w1L(f(x;q),y)\u2202\u2202w2L(f(x;q),y)...\u2202\u2202wnL(f(x;q),y)377775(5.15)The \ufb01nal equation for updatingqbased on the gradient is thusqt+1=qt\u0000h\u2014L(f(x;q),y)(5.16)12CHAPTER5\u2022LOGISTICREGRESSION5.4.3 Working through an exampleLet\u2019s walk though a single step of the gradient descent algorithm. We\u2019ll use a sim-pli\ufb01ed version of the example in Fig.5.2as it sees a single observationx, whosecorrect value isy=1 (this is a positive review), and with only two features:x1=3 (count of positive lexicon words)x2=2 (count of negative lexicon words)Let\u2019s assume the initial weights and bias inq0are all set to 0, and the initial learningratehis 0.1:w1=w2=b=0h=0.1The single update step requires that we compute the gradient, multiplied by thelearning rateqt+1=qt\u0000h\u2014qL(f(x(i);q),y(i))In our mini example there are three parameters, so the gradient vector has 3 dimen-sions, forw1,w2, andb. We can compute the \ufb01rst gradient as follows:\u2014w,b=264\u2202LCE(\u02c6y,y)\u2202w1\u2202LCE(\u02c6y,y)\u2202w2\u2202LCE(\u02c6y,y)\u2202b375=24(s(w\u00b7x+b)\u0000y)x1(s(w\u00b7x+b)\u0000y)x2s(w\u00b7x+b)\u0000y35=24(s(0)\u00001)x1(s(0)\u00001)x2s(0)\u0000135=24\u00000.5x1\u00000.5x2\u00000.535=24\u00001.5\u00001.0\u00000.535Now that we have a gradient, we compute the new parameter vectorq1by movingq0in the opposite direction from the gradient:q1=24w1w2b35\u0000h24\u00001.5\u00001.0\u00000.535=24.15.1.0535So after one step of gradient descent, the weights have shifted to be:w1=.15,w2=.1, andb=.05.Note that this observationxhappened to be a positive example. We would expectthat after seeing more negative examples with high counts of negative words, thatthe weightw2would shift to have a negative value.5.4.4 Mini-batch trainingStochastic gradient descent is called stochastic because it chooses a single randomexample at a time, moving the weights so as to improve performance on that singleexample. That can result in very choppy movements, so it\u2019s common to compute thegradient over batches of training instances rather than a single instance.For example inbatch trainingwe compute the gradient over the entire dataset.batch trainingBy seeing so many examples, batch training offers a superb estimate of which di-rection to move the weights, at the cost of spending a lot of time processing everysingle example in the training set to compute this perfect direction.A compromise ismini-batchtraining: we train on a group ofmexamples (per-mini-batchhaps 512, or 1024) that is less than the whole dataset. (Ifmis the size of the dataset,w1= w2= b  = 0;    x1= 3;   x2= 2   5.4\u2022GRADIENTDESCENT115.4.1 The Gradient for Logistic RegressionIn order to updateq, we need a de\ufb01nition for the gradient\u2014L(f(x;q),y). Recall thatfor logistic regression, the cross-entropy loss function is:LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))](5.17)It turns out that the derivative of this function for one observation vectorxisEq.5.18(the interested reader can see Section5.8for the derivation of this equation):\u2202LCE(\u02c6y,y)\u2202wj=[s(w\u00b7x+b)\u0000y]xj(5.18)Note in Eq.5.18that the gradient with respect to a single weightwjrepresents avery intuitive value: the difference between the trueyand our estimated \u02c6y=s(w\u00b7x+b)for that observation, multiplied by the corresponding input valuexj.5.4.2 The Stochastic Gradient Descent AlgorithmStochastic gradient descent is an online algorithm that minimizes the loss functionby computing its gradient after each training example, and nudgingqin the rightdirection (the opposite direction of the gradient). Fig.5.5shows the algorithm.functionSTOCHASTICGRADIENTDESCENT(L(),f(),x,y)returnsq# where: L is the loss function# f is a function parameterized byq# x is the set of training inputsx(1),x(2),. .. ,x(m)# y is the set of training outputs (labels)y(1),y(2),. .. ,y(m)q 0repeattil done # see captionFor each training tuple(x(i),y(i))(in random order)1. Optional (for reporting): # How are we doing on this tuple?Compute \u02c6y(i)=f(x(i);q)# What is our estimated output \u02c6y?Compute the lossL(\u02c6y(i),y(i))# How far off is \u02c6y(i))from the true outputy(i)?2.g \u2014qL(f(x(i);q),y(i))# How should we moveqto maximize loss?3.q q\u0000hg# Go the other way insteadreturnqFigure 5.5The stochastic gradient descent algorithm. Step 1 (computing the loss) is usedto report how well we are doing on the current tuple. The algorithm can terminate when itconverges (or when the gradient norm<\u270f), or when progress halts (for example when theloss starts going up on a held-out set).The learning ratehis ahyperparameterthat must be adjusted. If it\u2019s too high,hyperparameterthe learner will take steps that are too large, overshooting the minimum of the lossfunction. If it\u2019s too low, the learner will take steps that are too small, and take toolong to get to the minimum. It is common to start with a higher learning rate and thenslowly decrease it, so that it is a function of the iterationkof training; the notationhkcan be used to mean the value of the learning rate at iterationk.We\u2019ll discuss hyperparameters in more detail in Chapter 7, but brie\ufb02y they area special kind of parameter for any machine learning model. Unlike regular param-eters of a model (weights likewandb), which are learned by the algorithm fromthe training set, hyperparameters are special parameters chosen by the algorithmdesigner that affect how the algorithm works. Example of gradient descentUpdate step for update \u03b8 is:whereGradient vector has 3 dimensions:10CHAPTER5\u2022LOGISTICREGRESSIONlearning rate times the gradient (or the slope, in our single-variable example):wt+1=wt\u0000hddwf(x;w)(5.14)Now let\u2019s extend the intuition from a function of one scalar variablewto manyvariables, because we don\u2019t just want to move left or right, we want to know wherein the N-dimensional space (of theNparameters that make upq) we should move.Thegradientis just such a vector; it expresses the directional components of thesharpest slope along each of thoseNdimensions. If we\u2019re just imagining two weightdimensions (say for one weightwand one biasb), the gradient might be a vector withtwo orthogonal components, each of which tells us how much the ground slopes inthewdimension and in thebdimension. Fig.5.4shows a visualization of the valueof a 2-dimensional gradient vector taken at the red point.\nCost(w,b)\nwbFigure 5.4Visualization of the gradient vector at the red point in two dimensionswandb,showing the gradient as a red arrow in the x-y plane.In an actual logistic regression, the parameter vectorwis much longer than 1 or2, since the input feature vectorxcan be quite long, and we need a weightwiforeachxi. For each dimension/variablewiinw(plus the biasb), the gradient will havea component that tells us the slope with respect to that variable. Essentially we\u2019reasking: \u201cHow much would a small change in that variablewiin\ufb02uence the total lossfunctionL?\u201dIn each dimensionwi, we express the slope as a partial derivative\u2202\u2202wiof the lossfunction. The gradient is then de\ufb01ned as a vector of these partials. We\u2019ll represent \u02c6yasf(x;q)to make the dependence onqmore obvious:\u2014qL(f(x;q),y)) =266664\u2202\u2202w1L(f(x;q),y)\u2202\u2202w2L(f(x;q),y)...\u2202\u2202wnL(f(x;q),y)377775(5.15)The \ufb01nal equation for updatingqbased on the gradient is thusqt+1=qt\u0000h\u2014L(f(x;q),y)(5.16)12CHAPTER5\u2022LOGISTICREGRESSION5.4.3 Working through an exampleLet\u2019s walk though a single step of the gradient descent algorithm. We\u2019ll use a sim-pli\ufb01ed version of the example in Fig.5.2as it sees a single observationx, whosecorrect value isy=1 (this is a positive review), and with only two features:x1=3 (count of positive lexicon words)x2=2 (count of negative lexicon words)Let\u2019s assume the initial weights and bias inq0are all set to 0, and the initial learningratehis 0.1:w1=w2=b=0h=0.1The single update step requires that we compute the gradient, multiplied by thelearning rateqt+1=qt\u0000h\u2014qL(f(x(i);q),y(i))In our mini example there are three parameters, so the gradient vector has 3 dimen-sions, forw1,w2, andb. We can compute the \ufb01rst gradient as follows:\u2014w,b=264\u2202LCE(\u02c6y,y)\u2202w1\u2202LCE(\u02c6y,y)\u2202w2\u2202LCE(\u02c6y,y)\u2202b375=24(s(w\u00b7x+b)\u0000y)x1(s(w\u00b7x+b)\u0000y)x2s(w\u00b7x+b)\u0000y35=24(s(0)\u00001)x1(s(0)\u00001)x2s(0)\u0000135=24\u00000.5x1\u00000.5x2\u00000.535=24\u00001.5\u00001.0\u00000.535Now that we have a gradient, we compute the new parameter vectorq1by movingq0in the opposite direction from the gradient:q1=24w1w2b35\u0000h24\u00001.5\u00001.0\u00000.535=24.15.1.0535So after one step of gradient descent, the weights have shifted to be:w1=.15,w2=.1, andb=.05.Note that this observationxhappened to be a positive example. We would expectthat after seeing more negative examples with high counts of negative words, thatthe weightw2would shift to have a negative value.5.4.4 Mini-batch trainingStochastic gradient descent is called stochastic because it chooses a single randomexample at a time, moving the weights so as to improve performance on that singleexample. That can result in very choppy movements, so it\u2019s common to compute thegradient over batches of training instances rather than a single instance.For example inbatch trainingwe compute the gradient over the entire dataset.batch trainingBy seeing so many examples, batch training offers a superb estimate of which di-rection to move the weights, at the cost of spending a lot of time processing everysingle example in the training set to compute this perfect direction.A compromise ismini-batchtraining: we train on a group ofmexamples (per-mini-batchhaps 512, or 1024) that is less than the whole dataset. (Ifmis the size of the dataset,w1= w2= b  = 0;    x1= 3;   x2= 2   5.4\u2022GRADIENTDESCENT115.4.1 The Gradient for Logistic RegressionIn order to updateq, we need a de\ufb01nition for the gradient\u2014L(f(x;q),y). Recall thatfor logistic regression, the cross-entropy loss function is:LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))](5.17)It turns out that the derivative of this function for one observation vectorxisEq.5.18(the interested reader can see Section5.8for the derivation of this equation):\u2202LCE(\u02c6y,y)\u2202wj=[s(w\u00b7x+b)\u0000y]xj(5.18)Note in Eq.5.18that the gradient with respect to a single weightwjrepresents avery intuitive value: the difference between the trueyand our estimated \u02c6y=s(w\u00b7x+b)for that observation, multiplied by the corresponding input valuexj.5.4.2 The Stochastic Gradient Descent AlgorithmStochastic gradient descent is an online algorithm that minimizes the loss functionby computing its gradient after each training example, and nudgingqin the rightdirection (the opposite direction of the gradient). Fig.5.5shows the algorithm.functionSTOCHASTICGRADIENTDESCENT(L(),f(),x,y)returnsq# where: L is the loss function# f is a function parameterized byq# x is the set of training inputsx(1),x(2),. .. ,x(m)# y is the set of training outputs (labels)y(1),y(2),. .. ,y(m)q 0repeattil done # see captionFor each training tuple(x(i),y(i))(in random order)1. Optional (for reporting): # How are we doing on this tuple?Compute \u02c6y(i)=f(x(i);q)# What is our estimated output \u02c6y?Compute the lossL(\u02c6y(i),y(i))# How far off is \u02c6y(i))from the true outputy(i)?2.g \u2014qL(f(x(i);q),y(i))# How should we moveqto maximize loss?3.q q\u0000hg# Go the other way insteadreturnqFigure 5.5The stochastic gradient descent algorithm. Step 1 (computing the loss) is usedto report how well we are doing on the current tuple. The algorithm can terminate when itconverges (or when the gradient norm<\u270f), or when progress halts (for example when theloss starts going up on a held-out set).The learning ratehis ahyperparameterthat must be adjusted. If it\u2019s too high,hyperparameterthe learner will take steps that are too large, overshooting the minimum of the lossfunction. If it\u2019s too low, the learner will take steps that are too small, and take toolong to get to the minimum. It is common to start with a higher learning rate and thenslowly decrease it, so that it is a function of the iterationkof training; the notationhkcan be used to mean the value of the learning rate at iterationk.We\u2019ll discuss hyperparameters in more detail in Chapter 7, but brie\ufb02y they area special kind of parameter for any machine learning model. Unlike regular param-eters of a model (weights likewandb), which are learned by the algorithm fromthe training set, hyperparameters are special parameters chosen by the algorithmdesigner that affect how the algorithm works. Example of gradient descentUpdate step for update \u03b8 is:whereGradient vector has 3 dimensions:10CHAPTER5\u2022LOGISTICREGRESSIONlearning rate times the gradient (or the slope, in our single-variable example):wt+1=wt\u0000hddwf(x;w)(5.14)Now let\u2019s extend the intuition from a function of one scalar variablewto manyvariables, because we don\u2019t just want to move left or right, we want to know wherein the N-dimensional space (of theNparameters that make upq) we should move.Thegradientis just such a vector; it expresses the directional components of thesharpest slope along each of thoseNdimensions. If we\u2019re just imagining two weightdimensions (say for one weightwand one biasb), the gradient might be a vector withtwo orthogonal components, each of which tells us how much the ground slopes inthewdimension and in thebdimension. Fig.5.4shows a visualization of the valueof a 2-dimensional gradient vector taken at the red point.\nCost(w,b)\nwbFigure 5.4Visualization of the gradient vector at the red point in two dimensionswandb,showing the gradient as a red arrow in the x-y plane.In an actual logistic regression, the parameter vectorwis much longer than 1 or2, since the input feature vectorxcan be quite long, and we need a weightwiforeachxi. For each dimension/variablewiinw(plus the biasb), the gradient will havea component that tells us the slope with respect to that variable. Essentially we\u2019reasking: \u201cHow much would a small change in that variablewiin\ufb02uence the total lossfunctionL?\u201dIn each dimensionwi, we express the slope as a partial derivative\u2202\u2202wiof the lossfunction. The gradient is then de\ufb01ned as a vector of these partials. We\u2019ll represent \u02c6yasf(x;q)to make the dependence onqmore obvious:\u2014qL(f(x;q),y)) =266664\u2202\u2202w1L(f(x;q),y)\u2202\u2202w2L(f(x;q),y)...\u2202\u2202wnL(f(x;q),y)377775(5.15)The \ufb01nal equation for updatingqbased on the gradient is thusqt+1=qt\u0000h\u2014L(f(x;q),y)(5.16)12CHAPTER5\u2022LOGISTICREGRESSION5.4.3 Working through an exampleLet\u2019s walk though a single step of the gradient descent algorithm. We\u2019ll use a sim-pli\ufb01ed version of the example in Fig.5.2as it sees a single observationx, whosecorrect value isy=1 (this is a positive review), and with only two features:x1=3 (count of positive lexicon words)x2=2 (count of negative lexicon words)Let\u2019s assume the initial weights and bias inq0are all set to 0, and the initial learningratehis 0.1:w1=w2=b=0h=0.1The single update step requires that we compute the gradient, multiplied by thelearning rateqt+1=qt\u0000h\u2014qL(f(x(i);q),y(i))In our mini example there are three parameters, so the gradient vector has 3 dimen-sions, forw1,w2, andb. We can compute the \ufb01rst gradient as follows:\u2014w,b=264\u2202LCE(\u02c6y,y)\u2202w1\u2202LCE(\u02c6y,y)\u2202w2\u2202LCE(\u02c6y,y)\u2202b375=24(s(w\u00b7x+b)\u0000y)x1(s(w\u00b7x+b)\u0000y)x2s(w\u00b7x+b)\u0000y35=24(s(0)\u00001)x1(s(0)\u00001)x2s(0)\u0000135=24\u00000.5x1\u00000.5x2\u00000.535=24\u00001.5\u00001.0\u00000.535Now that we have a gradient, we compute the new parameter vectorq1by movingq0in the opposite direction from the gradient:q1=24w1w2b35\u0000h24\u00001.5\u00001.0\u00000.535=24.15.1.0535So after one step of gradient descent, the weights have shifted to be:w1=.15,w2=.1, andb=.05.Note that this observationxhappened to be a positive example. We would expectthat after seeing more negative examples with high counts of negative words, thatthe weightw2would shift to have a negative value.5.4.4 Mini-batch trainingStochastic gradient descent is called stochastic because it chooses a single randomexample at a time, moving the weights so as to improve performance on that singleexample. That can result in very choppy movements, so it\u2019s common to compute thegradient over batches of training instances rather than a single instance.For example inbatch trainingwe compute the gradient over the entire dataset.batch trainingBy seeing so many examples, batch training offers a superb estimate of which di-rection to move the weights, at the cost of spending a lot of time processing everysingle example in the training set to compute this perfect direction.A compromise ismini-batchtraining: we train on a group ofmexamples (per-mini-batchhaps 512, or 1024) that is less than the whole dataset. (Ifmis the size of the dataset,w1= w2= b  = 0;    x1= 3;   x2= 2   5.4\u2022GRADIENTDESCENT115.4.1 The Gradient for Logistic RegressionIn order to updateq, we need a de\ufb01nition for the gradient\u2014L(f(x;q),y). Recall thatfor logistic regression, the cross-entropy loss function is:LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))](5.17)It turns out that the derivative of this function for one observation vectorxisEq.5.18(the interested reader can see Section5.8for the derivation of this equation):\u2202LCE(\u02c6y,y)\u2202wj=[s(w\u00b7x+b)\u0000y]xj(5.18)Note in Eq.5.18that the gradient with respect to a single weightwjrepresents avery intuitive value: the difference between the trueyand our estimated \u02c6y=s(w\u00b7x+b)for that observation, multiplied by the corresponding input valuexj.5.4.2 The Stochastic Gradient Descent AlgorithmStochastic gradient descent is an online algorithm that minimizes the loss functionby computing its gradient after each training example, and nudgingqin the rightdirection (the opposite direction of the gradient). Fig.5.5shows the algorithm.functionSTOCHASTICGRADIENTDESCENT(L(),f(),x,y)returnsq# where: L is the loss function# f is a function parameterized byq# x is the set of training inputsx(1),x(2),. .. ,x(m)# y is the set of training outputs (labels)y(1),y(2),. .. ,y(m)q 0repeattil done # see captionFor each training tuple(x(i),y(i))(in random order)1. Optional (for reporting): # How are we doing on this tuple?Compute \u02c6y(i)=f(x(i);q)# What is our estimated output \u02c6y?Compute the lossL(\u02c6y(i),y(i))# How far off is \u02c6y(i))from the true outputy(i)?2.g \u2014qL(f(x(i);q),y(i))# How should we moveqto maximize loss?3.q q\u0000hg# Go the other way insteadreturnqFigure 5.5The stochastic gradient descent algorithm. Step 1 (computing the loss) is usedto report how well we are doing on the current tuple. The algorithm can terminate when itconverges (or when the gradient norm<\u270f), or when progress halts (for example when theloss starts going up on a held-out set).The learning ratehis ahyperparameterthat must be adjusted. If it\u2019s too high,hyperparameterthe learner will take steps that are too large, overshooting the minimum of the lossfunction. If it\u2019s too low, the learner will take steps that are too small, and take toolong to get to the minimum. It is common to start with a higher learning rate and thenslowly decrease it, so that it is a function of the iterationkof training; the notationhkcan be used to mean the value of the learning rate at iterationk.We\u2019ll discuss hyperparameters in more detail in Chapter 7, but brie\ufb02y they area special kind of parameter for any machine learning model. Unlike regular param-eters of a model (weights likewandb), which are learned by the algorithm fromthe training set, hyperparameters are special parameters chosen by the algorithmdesigner that affect how the algorithm works. Example of gradient descentUpdate step for update \u03b8 is:whereGradient vector has 3 dimensions:10CHAPTER5\u2022LOGISTICREGRESSIONlearning rate times the gradient (or the slope, in our single-variable example):wt+1=wt\u0000hddwf(x;w)(5.14)Now let\u2019s extend the intuition from a function of one scalar variablewto manyvariables, because we don\u2019t just want to move left or right, we want to know wherein the N-dimensional space (of theNparameters that make upq) we should move.Thegradientis just such a vector; it expresses the directional components of thesharpest slope along each of thoseNdimensions. If we\u2019re just imagining two weightdimensions (say for one weightwand one biasb), the gradient might be a vector withtwo orthogonal components, each of which tells us how much the ground slopes inthewdimension and in thebdimension. Fig.5.4shows a visualization of the valueof a 2-dimensional gradient vector taken at the red point.\nCost(w,b)\nwbFigure 5.4Visualization of the gradient vector at the red point in two dimensionswandb,showing the gradient as a red arrow in the x-y plane.In an actual logistic regression, the parameter vectorwis much longer than 1 or2, since the input feature vectorxcan be quite long, and we need a weightwiforeachxi. For each dimension/variablewiinw(plus the biasb), the gradient will havea component that tells us the slope with respect to that variable. Essentially we\u2019reasking: \u201cHow much would a small change in that variablewiin\ufb02uence the total lossfunctionL?\u201dIn each dimensionwi, we express the slope as a partial derivative\u2202\u2202wiof the lossfunction. The gradient is then de\ufb01ned as a vector of these partials. We\u2019ll represent \u02c6yasf(x;q)to make the dependence onqmore obvious:\u2014qL(f(x;q),y)) =266664\u2202\u2202w1L(f(x;q),y)\u2202\u2202w2L(f(x;q),y)...\u2202\u2202wnL(f(x;q),y)377775(5.15)The \ufb01nal equation for updatingqbased on the gradient is thusqt+1=qt\u0000h\u2014L(f(x;q),y)(5.16)12CHAPTER5\u2022LOGISTICREGRESSION5.4.3 Working through an exampleLet\u2019s walk though a single step of the gradient descent algorithm. We\u2019ll use a sim-pli\ufb01ed version of the example in Fig.5.2as it sees a single observationx, whosecorrect value isy=1 (this is a positive review), and with only two features:x1=3 (count of positive lexicon words)x2=2 (count of negative lexicon words)Let\u2019s assume the initial weights and bias inq0are all set to 0, and the initial learningratehis 0.1:w1=w2=b=0h=0.1The single update step requires that we compute the gradient, multiplied by thelearning rateqt+1=qt\u0000h\u2014qL(f(x(i);q),y(i))In our mini example there are three parameters, so the gradient vector has 3 dimen-sions, forw1,w2, andb. We can compute the \ufb01rst gradient as follows:\u2014w,b=264\u2202LCE(\u02c6y,y)\u2202w1\u2202LCE(\u02c6y,y)\u2202w2\u2202LCE(\u02c6y,y)\u2202b375=24(s(w\u00b7x+b)\u0000y)x1(s(w\u00b7x+b)\u0000y)x2s(w\u00b7x+b)\u0000y35=24(s(0)\u00001)x1(s(0)\u00001)x2s(0)\u0000135=24\u00000.5x1\u00000.5x2\u00000.535=24\u00001.5\u00001.0\u00000.535Now that we have a gradient, we compute the new parameter vectorq1by movingq0in the opposite direction from the gradient:q1=24w1w2b35\u0000h24\u00001.5\u00001.0\u00000.535=24.15.1.0535So after one step of gradient descent, the weights have shifted to be:w1=.15,w2=.1, andb=.05.Note that this observationxhappened to be a positive example. We would expectthat after seeing more negative examples with high counts of negative words, thatthe weightw2would shift to have a negative value.5.4.4 Mini-batch trainingStochastic gradient descent is called stochastic because it chooses a single randomexample at a time, moving the weights so as to improve performance on that singleexample. That can result in very choppy movements, so it\u2019s common to compute thegradient over batches of training instances rather than a single instance.For example inbatch trainingwe compute the gradient over the entire dataset.batch trainingBy seeing so many examples, batch training offers a superb estimate of which di-rection to move the weights, at the cost of spending a lot of time processing everysingle example in the training set to compute this perfect direction.A compromise ismini-batchtraining: we train on a group ofmexamples (per-mini-batchhaps 512, or 1024) that is less than the whole dataset. (Ifmis the size of the dataset,w1= w2= b  = 0;    x1= 3;   x2= 2   5.4\u2022GRADIENTDESCENT115.4.1 The Gradient for Logistic RegressionIn order to updateq, we need a de\ufb01nition for the gradient\u2014L(f(x;q),y). Recall thatfor logistic regression, the cross-entropy loss function is:LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))](5.17)It turns out that the derivative of this function for one observation vectorxisEq.5.18(the interested reader can see Section5.8for the derivation of this equation):\u2202LCE(\u02c6y,y)\u2202wj=[s(w\u00b7x+b)\u0000y]xj(5.18)Note in Eq.5.18that the gradient with respect to a single weightwjrepresents avery intuitive value: the difference between the trueyand our estimated \u02c6y=s(w\u00b7x+b)for that observation, multiplied by the corresponding input valuexj.5.4.2 The Stochastic Gradient Descent AlgorithmStochastic gradient descent is an online algorithm that minimizes the loss functionby computing its gradient after each training example, and nudgingqin the rightdirection (the opposite direction of the gradient). Fig.5.5shows the algorithm.functionSTOCHASTICGRADIENTDESCENT(L(),f(),x,y)returnsq# where: L is the loss function# f is a function parameterized byq# x is the set of training inputsx(1),x(2),. .. ,x(m)# y is the set of training outputs (labels)y(1),y(2),. .. ,y(m)q 0repeattil done # see captionFor each training tuple(x(i),y(i))(in random order)1. Optional (for reporting): # How are we doing on this tuple?Compute \u02c6y(i)=f(x(i);q)# What is our estimated output \u02c6y?Compute the lossL(\u02c6y(i),y(i))# How far off is \u02c6y(i))from the true outputy(i)?2.g \u2014qL(f(x(i);q),y(i))# How should we moveqto maximize loss?3.q q\u0000hg# Go the other way insteadreturnqFigure 5.5The stochastic gradient descent algorithm. Step 1 (computing the loss) is usedto report how well we are doing on the current tuple. The algorithm can terminate when itconverges (or when the gradient norm<\u270f), or when progress halts (for example when theloss starts going up on a held-out set).The learning ratehis ahyperparameterthat must be adjusted. If it\u2019s too high,hyperparameterthe learner will take steps that are too large, overshooting the minimum of the lossfunction. If it\u2019s too low, the learner will take steps that are too small, and take toolong to get to the minimum. It is common to start with a higher learning rate and thenslowly decrease it, so that it is a function of the iterationkof training; the notationhkcan be used to mean the value of the learning rate at iterationk.We\u2019ll discuss hyperparameters in more detail in Chapter 7, but brie\ufb02y they area special kind of parameter for any machine learning model. Unlike regular param-eters of a model (weights likewandb), which are learned by the algorithm fromthe training set, hyperparameters are special parameters chosen by the algorithmdesigner that affect how the algorithm works. Example of gradient descentUpdate step for update \u03b8 is:whereGradient vector has 3 dimensions:10CHAPTER5\u2022LOGISTICREGRESSIONlearning rate times the gradient (or the slope, in our single-variable example):wt+1=wt\u0000hddwf(x;w)(5.14)Now let\u2019s extend the intuition from a function of one scalar variablewto manyvariables, because we don\u2019t just want to move left or right, we want to know wherein the N-dimensional space (of theNparameters that make upq) we should move.Thegradientis just such a vector; it expresses the directional components of thesharpest slope along each of thoseNdimensions. If we\u2019re just imagining two weightdimensions (say for one weightwand one biasb), the gradient might be a vector withtwo orthogonal components, each of which tells us how much the ground slopes inthewdimension and in thebdimension. Fig.5.4shows a visualization of the valueof a 2-dimensional gradient vector taken at the red point.\nCost(w,b)\nwbFigure 5.4Visualization of the gradient vector at the red point in two dimensionswandb,showing the gradient as a red arrow in the x-y plane.In an actual logistic regression, the parameter vectorwis much longer than 1 or2, since the input feature vectorxcan be quite long, and we need a weightwiforeachxi. For each dimension/variablewiinw(plus the biasb), the gradient will havea component that tells us the slope with respect to that variable. Essentially we\u2019reasking: \u201cHow much would a small change in that variablewiin\ufb02uence the total lossfunctionL?\u201dIn each dimensionwi, we express the slope as a partial derivative\u2202\u2202wiof the lossfunction. The gradient is then de\ufb01ned as a vector of these partials. We\u2019ll represent \u02c6yasf(x;q)to make the dependence onqmore obvious:\u2014qL(f(x;q),y)) =266664\u2202\u2202w1L(f(x;q),y)\u2202\u2202w2L(f(x;q),y)...\u2202\u2202wnL(f(x;q),y)377775(5.15)The \ufb01nal equation for updatingqbased on the gradient is thusqt+1=qt\u0000h\u2014L(f(x;q),y)(5.16)12CHAPTER5\u2022LOGISTICREGRESSION5.4.3 Working through an exampleLet\u2019s walk though a single step of the gradient descent algorithm. We\u2019ll use a sim-pli\ufb01ed version of the example in Fig.5.2as it sees a single observationx, whosecorrect value isy=1 (this is a positive review), and with only two features:x1=3 (count of positive lexicon words)x2=2 (count of negative lexicon words)Let\u2019s assume the initial weights and bias inq0are all set to 0, and the initial learningratehis 0.1:w1=w2=b=0h=0.1The single update step requires that we compute the gradient, multiplied by thelearning rateqt+1=qt\u0000h\u2014qL(f(x(i);q),y(i))In our mini example there are three parameters, so the gradient vector has 3 dimen-sions, forw1,w2, andb. We can compute the \ufb01rst gradient as follows:\u2014w,b=264\u2202LCE(\u02c6y,y)\u2202w1\u2202LCE(\u02c6y,y)\u2202w2\u2202LCE(\u02c6y,y)\u2202b375=24(s(w\u00b7x+b)\u0000y)x1(s(w\u00b7x+b)\u0000y)x2s(w\u00b7x+b)\u0000y35=24(s(0)\u00001)x1(s(0)\u00001)x2s(0)\u0000135=24\u00000.5x1\u00000.5x2\u00000.535=24\u00001.5\u00001.0\u00000.535Now that we have a gradient, we compute the new parameter vectorq1by movingq0in the opposite direction from the gradient:q1=24w1w2b35\u0000h24\u00001.5\u00001.0\u00000.535=24.15.1.0535So after one step of gradient descent, the weights have shifted to be:w1=.15,w2=.1, andb=.05.Note that this observationxhappened to be a positive example. We would expectthat after seeing more negative examples with high counts of negative words, thatthe weightw2would shift to have a negative value.5.4.4 Mini-batch trainingStochastic gradient descent is called stochastic because it chooses a single randomexample at a time, moving the weights so as to improve performance on that singleexample. That can result in very choppy movements, so it\u2019s common to compute thegradient over batches of training instances rather than a single instance.For example inbatch trainingwe compute the gradient over the entire dataset.batch trainingBy seeing so many examples, batch training offers a superb estimate of which di-rection to move the weights, at the cost of spending a lot of time processing everysingle example in the training set to compute this perfect direction.A compromise ismini-batchtraining: we train on a group ofmexamples (per-mini-batchhaps 512, or 1024) that is less than the whole dataset. (Ifmis the size of the dataset,w1= w2= b  = 0;    x1= 3;   x2= 2   5.4\u2022GRADIENTDESCENT115.4.1 The Gradient for Logistic RegressionIn order to updateq, we need a de\ufb01nition for the gradient\u2014L(f(x;q),y). Recall thatfor logistic regression, the cross-entropy loss function is:LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))](5.17)It turns out that the derivative of this function for one observation vectorxisEq.5.18(the interested reader can see Section5.8for the derivation of this equation):\u2202LCE(\u02c6y,y)\u2202wj=[s(w\u00b7x+b)\u0000y]xj(5.18)Note in Eq.5.18that the gradient with respect to a single weightwjrepresents avery intuitive value: the difference between the trueyand our estimated \u02c6y=s(w\u00b7x+b)for that observation, multiplied by the corresponding input valuexj.5.4.2 The Stochastic Gradient Descent AlgorithmStochastic gradient descent is an online algorithm that minimizes the loss functionby computing its gradient after each training example, and nudgingqin the rightdirection (the opposite direction of the gradient). Fig.5.5shows the algorithm.functionSTOCHASTICGRADIENTDESCENT(L(),f(),x,y)returnsq# where: L is the loss function# f is a function parameterized byq# x is the set of training inputsx(1),x(2),. .. ,x(m)# y is the set of training outputs (labels)y(1),y(2),. .. ,y(m)q 0repeattil done # see captionFor each training tuple(x(i),y(i))(in random order)1. Optional (for reporting): # How are we doing on this tuple?Compute \u02c6y(i)=f(x(i);q)# What is our estimated output \u02c6y?Compute the lossL(\u02c6y(i),y(i))# How far off is \u02c6y(i))from the true outputy(i)?2.g \u2014qL(f(x(i);q),y(i))# How should we moveqto maximize loss?3.q q\u0000hg# Go the other way insteadreturnqFigure 5.5The stochastic gradient descent algorithm. Step 1 (computing the loss) is usedto report how well we are doing on the current tuple. The algorithm can terminate when itconverges (or when the gradient norm<\u270f), or when progress halts (for example when theloss starts going up on a held-out set).The learning ratehis ahyperparameterthat must be adjusted. If it\u2019s too high,hyperparameterthe learner will take steps that are too large, overshooting the minimum of the lossfunction. If it\u2019s too low, the learner will take steps that are too small, and take toolong to get to the minimum. It is common to start with a higher learning rate and thenslowly decrease it, so that it is a function of the iterationkof training; the notationhkcan be used to mean the value of the learning rate at iterationk.We\u2019ll discuss hyperparameters in more detail in Chapter 7, but brie\ufb02y they area special kind of parameter for any machine learning model. Unlike regular param-eters of a model (weights likewandb), which are learned by the algorithm fromthe training set, hyperparameters are special parameters chosen by the algorithmdesigner that affect how the algorithm works. Example of gradient descent10CHAPTER5\u2022LOGISTICREGRESSIONlearning rate times the gradient (or the slope, in our single-variable example):wt+1=wt\u0000hddwf(x;w)(5.14)Now let\u2019s extend the intuition from a function of one scalar variablewto manyvariables, because we don\u2019t just want to move left or right, we want to know wherein the N-dimensional space (of theNparameters that make upq) we should move.Thegradientis just such a vector; it expresses the directional components of thesharpest slope along each of thoseNdimensions. If we\u2019re just imagining two weightdimensions (say for one weightwand one biasb), the gradient might be a vector withtwo orthogonal components, each of which tells us how much the ground slopes inthewdimension and in thebdimension. Fig.5.4shows a visualization of the valueof a 2-dimensional gradient vector taken at the red point.\nCost(w,b)\nwbFigure 5.4Visualization of the gradient vector at the red point in two dimensionswandb,showing the gradient as a red arrow in the x-y plane.In an actual logistic regression, the parameter vectorwis much longer than 1 or2, since the input feature vectorxcan be quite long, and we need a weightwiforeachxi. For each dimension/variablewiinw(plus the biasb), the gradient will havea component that tells us the slope with respect to that variable. Essentially we\u2019reasking: \u201cHow much would a small change in that variablewiin\ufb02uence the total lossfunctionL?\u201dIn each dimensionwi, we express the slope as a partial derivative\u2202\u2202wiof the lossfunction. The gradient is then de\ufb01ned as a vector of these partials. We\u2019ll represent \u02c6yasf(x;q)to make the dependence onqmore obvious:\u2014qL(f(x;q),y)) =266664\u2202\u2202w1L(f(x;q),y)\u2202\u2202w2L(f(x;q),y)...\u2202\u2202wnL(f(x;q),y)377775(5.15)The \ufb01nal equation for updatingqbased on the gradient is thusqt+1=qt\u0000h\u2014L(f(x;q),y)(5.16)12CHAPTER5\u2022LOGISTICREGRESSION5.4.3 Working through an exampleLet\u2019s walk though a single step of the gradient descent algorithm. We\u2019ll use a sim-pli\ufb01ed version of the example in Fig.5.2as it sees a single observationx, whosecorrect value isy=1 (this is a positive review), and with only two features:x1=3 (count of positive lexicon words)x2=2 (count of negative lexicon words)Let\u2019s assume the initial weights and bias inq0are all set to 0, and the initial learningratehis 0.1:w1=w2=b=0h=0.1The single update step requires that we compute the gradient, multiplied by thelearning rateqt+1=qt\u0000h\u2014qL(f(x(i);q),y(i))In our mini example there are three parameters, so the gradient vector has 3 dimen-sions, forw1,w2, andb. We can compute the \ufb01rst gradient as follows:\u2014w,b=264\u2202LCE(\u02c6y,y)\u2202w1\u2202LCE(\u02c6y,y)\u2202w2\u2202LCE(\u02c6y,y)\u2202b375=24(s(w\u00b7x+b)\u0000y)x1(s(w\u00b7x+b)\u0000y)x2s(w\u00b7x+b)\u0000y35=24(s(0)\u00001)x1(s(0)\u00001)x2s(0)\u0000135=24\u00000.5x1\u00000.5x2\u00000.535=24\u00001.5\u00001.0\u00000.535Now that we have a gradient, we compute the new parameter vectorq1by movingq0in the opposite direction from the gradient:q1=24w1w2b35\u0000h24\u00001.5\u00001.0\u00000.535=24.15.1.0535So after one step of gradient descent, the weights have shifted to be:w1=.15,w2=.1, andb=.05.Note that this observationxhappened to be a positive example. We would expectthat after seeing more negative examples with high counts of negative words, thatthe weightw2would shift to have a negative value.5.4.4 Mini-batch trainingStochastic gradient descent is called stochastic because it chooses a single randomexample at a time, moving the weights so as to improve performance on that singleexample. That can result in very choppy movements, so it\u2019s common to compute thegradient over batches of training instances rather than a single instance.For example inbatch trainingwe compute the gradient over the entire dataset.batch trainingBy seeing so many examples, batch training offers a superb estimate of which di-rection to move the weights, at the cost of spending a lot of time processing everysingle example in the training set to compute this perfect direction.A compromise ismini-batchtraining: we train on a group ofmexamples (per-mini-batchhaps 512, or 1024) that is less than the whole dataset. (Ifmis the size of the dataset,\u03b7 = 0.1; 12CHAPTER5\u2022LOGISTICREGRESSION5.4.3 Working through an exampleLet\u2019s walk though a single step of the gradient descent algorithm. We\u2019ll use a sim-pli\ufb01ed version of the example in Fig.5.2as it sees a single observationx, whosecorrect value isy=1 (this is a positive review), and with only two features:x1=3 (count of positive lexicon words)x2=2 (count of negative lexicon words)Let\u2019s assume the initial weights and bias inq0are all set to 0, and the initial learningratehis 0.1:w1=w2=b=0h=0.1The single update step requires that we compute the gradient, multiplied by thelearning rateqt+1=qt\u0000h\u2014qL(f(x(i);q),y(i))In our mini example there are three parameters, so the gradient vector has 3 dimen-sions, forw1,w2, andb. We can compute the \ufb01rst gradient as follows:\u2014w,b=264\u2202LCE(\u02c6y,y)\u2202w1\u2202LCE(\u02c6y,y)\u2202w2\u2202LCE(\u02c6y,y)\u2202b375=24(s(w\u00b7x+b)\u0000y)x1(s(w\u00b7x+b)\u0000y)x2s(w\u00b7x+b)\u0000y35=24(s(0)\u00001)x1(s(0)\u00001)x2s(0)\u0000135=24\u00000.5x1\u00000.5x2\u00000.535=24\u00001.5\u00001.0\u00000.535Now that we have a gradient, we compute the new parameter vectorq1by movingq0in the opposite direction from the gradient:q1=24w1w2b35\u0000h24\u00001.5\u00001.0\u00000.535=24.15.1.0535So after one step of gradient descent, the weights have shifted to be:w1=.15,w2=.1, andb=.05.Note that this observationxhappened to be a positive example. We would expectthat after seeing more negative examples with high counts of negative words, thatthe weightw2would shift to have a negative value.5.4.4 Mini-batch trainingStochastic gradient descent is called stochastic because it chooses a single randomexample at a time, moving the weights so as to improve performance on that singleexample. That can result in very choppy movements, so it\u2019s common to compute thegradient over batches of training instances rather than a single instance.For example inbatch trainingwe compute the gradient over the entire dataset.batch trainingBy seeing so many examples, batch training offers a superb estimate of which di-rection to move the weights, at the cost of spending a lot of time processing everysingle example in the training set to compute this perfect direction.A compromise ismini-batchtraining: we train on a group ofmexamples (per-mini-batchhaps 512, or 1024) that is less than the whole dataset. (Ifmis the size of the dataset,Now that we have a gradient, we compute the new parameter vector \u03b81by moving \u03b80in the opposite direction from the gradient:  Example of gradient descent10CHAPTER5\u2022LOGISTICREGRESSIONlearning rate times the gradient (or the slope, in our single-variable example):wt+1=wt\u0000hddwf(x;w)(5.14)Now let\u2019s extend the intuition from a function of one scalar variablewto manyvariables, because we don\u2019t just want to move left or right, we want to know wherein the N-dimensional space (of theNparameters that make upq) we should move.Thegradientis just such a vector; it expresses the directional components of thesharpest slope along each of thoseNdimensions. If we\u2019re just imagining two weightdimensions (say for one weightwand one biasb), the gradient might be a vector withtwo orthogonal components, each of which tells us how much the ground slopes inthewdimension and in thebdimension. Fig.5.4shows a visualization of the valueof a 2-dimensional gradient vector taken at the red point.\nCost(w,b)\nwbFigure 5.4Visualization of the gradient vector at the red point in two dimensionswandb,showing the gradient as a red arrow in the x-y plane.In an actual logistic regression, the parameter vectorwis much longer than 1 or2, since the input feature vectorxcan be quite long, and we need a weightwiforeachxi. For each dimension/variablewiinw(plus the biasb), the gradient will havea component that tells us the slope with respect to that variable. Essentially we\u2019reasking: \u201cHow much would a small change in that variablewiin\ufb02uence the total lossfunctionL?\u201dIn each dimensionwi, we express the slope as a partial derivative\u2202\u2202wiof the lossfunction. The gradient is then de\ufb01ned as a vector of these partials. We\u2019ll represent \u02c6yasf(x;q)to make the dependence onqmore obvious:\u2014qL(f(x;q),y)) =266664\u2202\u2202w1L(f(x;q),y)\u2202\u2202w2L(f(x;q),y)...\u2202\u2202wnL(f(x;q),y)377775(5.15)The \ufb01nal equation for updatingqbased on the gradient is thusqt+1=qt\u0000h\u2014L(f(x;q),y)(5.16)12CHAPTER5\u2022LOGISTICREGRESSION5.4.3 Working through an exampleLet\u2019s walk though a single step of the gradient descent algorithm. We\u2019ll use a sim-pli\ufb01ed version of the example in Fig.5.2as it sees a single observationx, whosecorrect value isy=1 (this is a positive review), and with only two features:x1=3 (count of positive lexicon words)x2=2 (count of negative lexicon words)Let\u2019s assume the initial weights and bias inq0are all set to 0, and the initial learningratehis 0.1:w1=w2=b=0h=0.1The single update step requires that we compute the gradient, multiplied by thelearning rateqt+1=qt\u0000h\u2014qL(f(x(i);q),y(i))In our mini example there are three parameters, so the gradient vector has 3 dimen-sions, forw1,w2, andb. We can compute the \ufb01rst gradient as follows:\u2014w,b=264\u2202LCE(\u02c6y,y)\u2202w1\u2202LCE(\u02c6y,y)\u2202w2\u2202LCE(\u02c6y,y)\u2202b375=24(s(w\u00b7x+b)\u0000y)x1(s(w\u00b7x+b)\u0000y)x2s(w\u00b7x+b)\u0000y35=24(s(0)\u00001)x1(s(0)\u00001)x2s(0)\u0000135=24\u00000.5x1\u00000.5x2\u00000.535=24\u00001.5\u00001.0\u00000.535Now that we have a gradient, we compute the new parameter vectorq1by movingq0in the opposite direction from the gradient:q1=24w1w2b35\u0000h24\u00001.5\u00001.0\u00000.535=24.15.1.0535So after one step of gradient descent, the weights have shifted to be:w1=.15,w2=.1, andb=.05.Note that this observationxhappened to be a positive example. We would expectthat after seeing more negative examples with high counts of negative words, thatthe weightw2would shift to have a negative value.5.4.4 Mini-batch trainingStochastic gradient descent is called stochastic because it chooses a single randomexample at a time, moving the weights so as to improve performance on that singleexample. That can result in very choppy movements, so it\u2019s common to compute thegradient over batches of training instances rather than a single instance.For example inbatch trainingwe compute the gradient over the entire dataset.batch trainingBy seeing so many examples, batch training offers a superb estimate of which di-rection to move the weights, at the cost of spending a lot of time processing everysingle example in the training set to compute this perfect direction.A compromise ismini-batchtraining: we train on a group ofmexamples (per-mini-batchhaps 512, or 1024) that is less than the whole dataset. (Ifmis the size of the dataset,\u03b7 = 0.1; 12CHAPTER5\u2022LOGISTICREGRESSION5.4.3 Working through an exampleLet\u2019s walk though a single step of the gradient descent algorithm. We\u2019ll use a sim-pli\ufb01ed version of the example in Fig.5.2as it sees a single observationx, whosecorrect value isy=1 (this is a positive review), and with only two features:x1=3 (count of positive lexicon words)x2=2 (count of negative lexicon words)Let\u2019s assume the initial weights and bias inq0are all set to 0, and the initial learningratehis 0.1:w1=w2=b=0h=0.1The single update step requires that we compute the gradient, multiplied by thelearning rateqt+1=qt\u0000h\u2014qL(f(x(i);q),y(i))In our mini example there are three parameters, so the gradient vector has 3 dimen-sions, forw1,w2, andb. We can compute the \ufb01rst gradient as follows:\u2014w,b=264\u2202LCE(\u02c6y,y)\u2202w1\u2202LCE(\u02c6y,y)\u2202w2\u2202LCE(\u02c6y,y)\u2202b375=24(s(w\u00b7x+b)\u0000y)x1(s(w\u00b7x+b)\u0000y)x2s(w\u00b7x+b)\u0000y35=24(s(0)\u00001)x1(s(0)\u00001)x2s(0)\u0000135=24\u00000.5x1\u00000.5x2\u00000.535=24\u00001.5\u00001.0\u00000.535Now that we have a gradient, we compute the new parameter vectorq1by movingq0in the opposite direction from the gradient:q1=24w1w2b35\u0000h24\u00001.5\u00001.0\u00000.535=24.15.1.0535So after one step of gradient descent, the weights have shifted to be:w1=.15,w2=.1, andb=.05.Note that this observationxhappened to be a positive example. We would expectthat after seeing more negative examples with high counts of negative words, thatthe weightw2would shift to have a negative value.5.4.4 Mini-batch trainingStochastic gradient descent is called stochastic because it chooses a single randomexample at a time, moving the weights so as to improve performance on that singleexample. That can result in very choppy movements, so it\u2019s common to compute thegradient over batches of training instances rather than a single instance.For example inbatch trainingwe compute the gradient over the entire dataset.batch trainingBy seeing so many examples, batch training offers a superb estimate of which di-rection to move the weights, at the cost of spending a lot of time processing everysingle example in the training set to compute this perfect direction.A compromise ismini-batchtraining: we train on a group ofmexamples (per-mini-batchhaps 512, or 1024) that is less than the whole dataset. (Ifmis the size of the dataset,Now that we have a gradient, we compute the new parameter vector \u03b81by moving \u03b80in the opposite direction from the gradient:  Example of gradient descent10CHAPTER5\u2022LOGISTICREGRESSIONlearning rate times the gradient (or the slope, in our single-variable example):wt+1=wt\u0000hddwf(x;w)(5.14)Now let\u2019s extend the intuition from a function of one scalar variablewto manyvariables, because we don\u2019t just want to move left or right, we want to know wherein the N-dimensional space (of theNparameters that make upq) we should move.Thegradientis just such a vector; it expresses the directional components of thesharpest slope along each of thoseNdimensions. If we\u2019re just imagining two weightdimensions (say for one weightwand one biasb), the gradient might be a vector withtwo orthogonal components, each of which tells us how much the ground slopes inthewdimension and in thebdimension. Fig.5.4shows a visualization of the valueof a 2-dimensional gradient vector taken at the red point.\nCost(w,b)\nwbFigure 5.4Visualization of the gradient vector at the red point in two dimensionswandb,showing the gradient as a red arrow in the x-y plane.In an actual logistic regression, the parameter vectorwis much longer than 1 or2, since the input feature vectorxcan be quite long, and we need a weightwiforeachxi. For each dimension/variablewiinw(plus the biasb), the gradient will havea component that tells us the slope with respect to that variable. Essentially we\u2019reasking: \u201cHow much would a small change in that variablewiin\ufb02uence the total lossfunctionL?\u201dIn each dimensionwi, we express the slope as a partial derivative\u2202\u2202wiof the lossfunction. The gradient is then de\ufb01ned as a vector of these partials. We\u2019ll represent \u02c6yasf(x;q)to make the dependence onqmore obvious:\u2014qL(f(x;q),y)) =266664\u2202\u2202w1L(f(x;q),y)\u2202\u2202w2L(f(x;q),y)...\u2202\u2202wnL(f(x;q),y)377775(5.15)The \ufb01nal equation for updatingqbased on the gradient is thusqt+1=qt\u0000h\u2014L(f(x;q),y)(5.16)12CHAPTER5\u2022LOGISTICREGRESSION5.4.3 Working through an exampleLet\u2019s walk though a single step of the gradient descent algorithm. We\u2019ll use a sim-pli\ufb01ed version of the example in Fig.5.2as it sees a single observationx, whosecorrect value isy=1 (this is a positive review), and with only two features:x1=3 (count of positive lexicon words)x2=2 (count of negative lexicon words)Let\u2019s assume the initial weights and bias inq0are all set to 0, and the initial learningratehis 0.1:w1=w2=b=0h=0.1The single update step requires that we compute the gradient, multiplied by thelearning rateqt+1=qt\u0000h\u2014qL(f(x(i);q),y(i))In our mini example there are three parameters, so the gradient vector has 3 dimen-sions, forw1,w2, andb. We can compute the \ufb01rst gradient as follows:\u2014w,b=264\u2202LCE(\u02c6y,y)\u2202w1\u2202LCE(\u02c6y,y)\u2202w2\u2202LCE(\u02c6y,y)\u2202b375=24(s(w\u00b7x+b)\u0000y)x1(s(w\u00b7x+b)\u0000y)x2s(w\u00b7x+b)\u0000y35=24(s(0)\u00001)x1(s(0)\u00001)x2s(0)\u0000135=24\u00000.5x1\u00000.5x2\u00000.535=24\u00001.5\u00001.0\u00000.535Now that we have a gradient, we compute the new parameter vectorq1by movingq0in the opposite direction from the gradient:q1=24w1w2b35\u0000h24\u00001.5\u00001.0\u00000.535=24.15.1.0535So after one step of gradient descent, the weights have shifted to be:w1=.15,w2=.1, andb=.05.Note that this observationxhappened to be a positive example. We would expectthat after seeing more negative examples with high counts of negative words, thatthe weightw2would shift to have a negative value.5.4.4 Mini-batch trainingStochastic gradient descent is called stochastic because it chooses a single randomexample at a time, moving the weights so as to improve performance on that singleexample. That can result in very choppy movements, so it\u2019s common to compute thegradient over batches of training instances rather than a single instance.For example inbatch trainingwe compute the gradient over the entire dataset.batch trainingBy seeing so many examples, batch training offers a superb estimate of which di-rection to move the weights, at the cost of spending a lot of time processing everysingle example in the training set to compute this perfect direction.A compromise ismini-batchtraining: we train on a group ofmexamples (per-mini-batchhaps 512, or 1024) that is less than the whole dataset. (Ifmis the size of the dataset,\u03b7 = 0.1; 12CHAPTER5\u2022LOGISTICREGRESSION5.4.3 Working through an exampleLet\u2019s walk though a single step of the gradient descent algorithm. We\u2019ll use a sim-pli\ufb01ed version of the example in Fig.5.2as it sees a single observationx, whosecorrect value isy=1 (this is a positive review), and with only two features:x1=3 (count of positive lexicon words)x2=2 (count of negative lexicon words)Let\u2019s assume the initial weights and bias inq0are all set to 0, and the initial learningratehis 0.1:w1=w2=b=0h=0.1The single update step requires that we compute the gradient, multiplied by thelearning rateqt+1=qt\u0000h\u2014qL(f(x(i);q),y(i))In our mini example there are three parameters, so the gradient vector has 3 dimen-sions, forw1,w2, andb. We can compute the \ufb01rst gradient as follows:\u2014w,b=264\u2202LCE(\u02c6y,y)\u2202w1\u2202LCE(\u02c6y,y)\u2202w2\u2202LCE(\u02c6y,y)\u2202b375=24(s(w\u00b7x+b)\u0000y)x1(s(w\u00b7x+b)\u0000y)x2s(w\u00b7x+b)\u0000y35=24(s(0)\u00001)x1(s(0)\u00001)x2s(0)\u0000135=24\u00000.5x1\u00000.5x2\u00000.535=24\u00001.5\u00001.0\u00000.535Now that we have a gradient, we compute the new parameter vectorq1by movingq0in the opposite direction from the gradient:q1=24w1w2b35\u0000h24\u00001.5\u00001.0\u00000.535=24.15.1.0535So after one step of gradient descent, the weights have shifted to be:w1=.15,w2=.1, andb=.05.Note that this observationxhappened to be a positive example. We would expectthat after seeing more negative examples with high counts of negative words, thatthe weightw2would shift to have a negative value.5.4.4 Mini-batch trainingStochastic gradient descent is called stochastic because it chooses a single randomexample at a time, moving the weights so as to improve performance on that singleexample. That can result in very choppy movements, so it\u2019s common to compute thegradient over batches of training instances rather than a single instance.For example inbatch trainingwe compute the gradient over the entire dataset.batch trainingBy seeing so many examples, batch training offers a superb estimate of which di-rection to move the weights, at the cost of spending a lot of time processing everysingle example in the training set to compute this perfect direction.A compromise ismini-batchtraining: we train on a group ofmexamples (per-mini-batchhaps 512, or 1024) that is less than the whole dataset. (Ifmis the size of the dataset,Now that we have a gradient, we compute the new parameter vector \u03b81by moving \u03b80in the opposite direction from the gradient:  Example of gradient descent10CHAPTER5\u2022LOGISTICREGRESSIONlearning rate times the gradient (or the slope, in our single-variable example):wt+1=wt\u0000hddwf(x;w)(5.14)Now let\u2019s extend the intuition from a function of one scalar variablewto manyvariables, because we don\u2019t just want to move left or right, we want to know wherein the N-dimensional space (of theNparameters that make upq) we should move.Thegradientis just such a vector; it expresses the directional components of thesharpest slope along each of thoseNdimensions. If we\u2019re just imagining two weightdimensions (say for one weightwand one biasb), the gradient might be a vector withtwo orthogonal components, each of which tells us how much the ground slopes inthewdimension and in thebdimension. Fig.5.4shows a visualization of the valueof a 2-dimensional gradient vector taken at the red point.\nCost(w,b)\nwbFigure 5.4Visualization of the gradient vector at the red point in two dimensionswandb,showing the gradient as a red arrow in the x-y plane.In an actual logistic regression, the parameter vectorwis much longer than 1 or2, since the input feature vectorxcan be quite long, and we need a weightwiforeachxi. For each dimension/variablewiinw(plus the biasb), the gradient will havea component that tells us the slope with respect to that variable. Essentially we\u2019reasking: \u201cHow much would a small change in that variablewiin\ufb02uence the total lossfunctionL?\u201dIn each dimensionwi, we express the slope as a partial derivative\u2202\u2202wiof the lossfunction. The gradient is then de\ufb01ned as a vector of these partials. We\u2019ll represent \u02c6yasf(x;q)to make the dependence onqmore obvious:\u2014qL(f(x;q),y)) =266664\u2202\u2202w1L(f(x;q),y)\u2202\u2202w2L(f(x;q),y)...\u2202\u2202wnL(f(x;q),y)377775(5.15)The \ufb01nal equation for updatingqbased on the gradient is thusqt+1=qt\u0000h\u2014L(f(x;q),y)(5.16)12CHAPTER5\u2022LOGISTICREGRESSION5.4.3 Working through an exampleLet\u2019s walk though a single step of the gradient descent algorithm. We\u2019ll use a sim-pli\ufb01ed version of the example in Fig.5.2as it sees a single observationx, whosecorrect value isy=1 (this is a positive review), and with only two features:x1=3 (count of positive lexicon words)x2=2 (count of negative lexicon words)Let\u2019s assume the initial weights and bias inq0are all set to 0, and the initial learningratehis 0.1:w1=w2=b=0h=0.1The single update step requires that we compute the gradient, multiplied by thelearning rateqt+1=qt\u0000h\u2014qL(f(x(i);q),y(i))In our mini example there are three parameters, so the gradient vector has 3 dimen-sions, forw1,w2, andb. We can compute the \ufb01rst gradient as follows:\u2014w,b=264\u2202LCE(\u02c6y,y)\u2202w1\u2202LCE(\u02c6y,y)\u2202w2\u2202LCE(\u02c6y,y)\u2202b375=24(s(w\u00b7x+b)\u0000y)x1(s(w\u00b7x+b)\u0000y)x2s(w\u00b7x+b)\u0000y35=24(s(0)\u00001)x1(s(0)\u00001)x2s(0)\u0000135=24\u00000.5x1\u00000.5x2\u00000.535=24\u00001.5\u00001.0\u00000.535Now that we have a gradient, we compute the new parameter vectorq1by movingq0in the opposite direction from the gradient:q1=24w1w2b35\u0000h24\u00001.5\u00001.0\u00000.535=24.15.1.0535So after one step of gradient descent, the weights have shifted to be:w1=.15,w2=.1, andb=.05.Note that this observationxhappened to be a positive example. We would expectthat after seeing more negative examples with high counts of negative words, thatthe weightw2would shift to have a negative value.5.4.4 Mini-batch trainingStochastic gradient descent is called stochastic because it chooses a single randomexample at a time, moving the weights so as to improve performance on that singleexample. That can result in very choppy movements, so it\u2019s common to compute thegradient over batches of training instances rather than a single instance.For example inbatch trainingwe compute the gradient over the entire dataset.batch trainingBy seeing so many examples, batch training offers a superb estimate of which di-rection to move the weights, at the cost of spending a lot of time processing everysingle example in the training set to compute this perfect direction.A compromise ismini-batchtraining: we train on a group ofmexamples (per-mini-batchhaps 512, or 1024) that is less than the whole dataset. (Ifmis the size of the dataset,\u03b7 = 0.1; 12CHAPTER5\u2022LOGISTICREGRESSION5.4.3 Working through an exampleLet\u2019s walk though a single step of the gradient descent algorithm. We\u2019ll use a sim-pli\ufb01ed version of the example in Fig.5.2as it sees a single observationx, whosecorrect value isy=1 (this is a positive review), and with only two features:x1=3 (count of positive lexicon words)x2=2 (count of negative lexicon words)Let\u2019s assume the initial weights and bias inq0are all set to 0, and the initial learningratehis 0.1:w1=w2=b=0h=0.1The single update step requires that we compute the gradient, multiplied by thelearning rateqt+1=qt\u0000h\u2014qL(f(x(i);q),y(i))In our mini example there are three parameters, so the gradient vector has 3 dimen-sions, forw1,w2, andb. We can compute the \ufb01rst gradient as follows:\u2014w,b=264\u2202LCE(\u02c6y,y)\u2202w1\u2202LCE(\u02c6y,y)\u2202w2\u2202LCE(\u02c6y,y)\u2202b375=24(s(w\u00b7x+b)\u0000y)x1(s(w\u00b7x+b)\u0000y)x2s(w\u00b7x+b)\u0000y35=24(s(0)\u00001)x1(s(0)\u00001)x2s(0)\u0000135=24\u00000.5x1\u00000.5x2\u00000.535=24\u00001.5\u00001.0\u00000.535Now that we have a gradient, we compute the new parameter vectorq1by movingq0in the opposite direction from the gradient:q1=24w1w2b35\u0000h24\u00001.5\u00001.0\u00000.535=24.15.1.0535So after one step of gradient descent, the weights have shifted to be:w1=.15,w2=.1, andb=.05.Note that this observationxhappened to be a positive example. We would expectthat after seeing more negative examples with high counts of negative words, thatthe weightw2would shift to have a negative value.5.4.4 Mini-batch trainingStochastic gradient descent is called stochastic because it chooses a single randomexample at a time, moving the weights so as to improve performance on that singleexample. That can result in very choppy movements, so it\u2019s common to compute thegradient over batches of training instances rather than a single instance.For example inbatch trainingwe compute the gradient over the entire dataset.batch trainingBy seeing so many examples, batch training offers a superb estimate of which di-rection to move the weights, at the cost of spending a lot of time processing everysingle example in the training set to compute this perfect direction.A compromise ismini-batchtraining: we train on a group ofmexamples (per-mini-batchhaps 512, or 1024) that is less than the whole dataset. (Ifmis the size of the dataset,Now that we have a gradient, we compute the new parameter vector \u03b81by moving \u03b80in the opposite direction from the gradient: \nNote that enough negative examples would eventually make w2negative Mini-batch trainingStochastic gradient descent chooses a single random example at a time.That can result in choppy movementsMore common to compute gradient over batches of training instances.Batch training: entire datasetMini-batch training: mexamples (512, or 1024) Logistic RegressionStochastic Gradient Descent: An example and more details Logistic RegressionRegularization OverfittingA model that perfectly match the training data has a problem.It will also overfitto the data, modeling noise \u25e6A random word that perfectly predicts y(it happens to only occur in one class) will get a very high weight. \u25e6Failing to generalize to a test set without this word. A good model should be able to generalize OverfittingThis movie drew me in, and it'll do the same to you.81X1 = \"this\"X2 = \"movieX3 = \"hated\"I can't tell you how much I hated this movie. It sucked.X5 = \"the same to you\"X7 = \"tell you how much\"X4 = \"drew me in\"+-Useful or harmless features4gram features that just \"memorize\" training set and might cause problems Overfitting4-gram model on tiny data will just memorize the data\u25e6100% accuracy on the training setBut it will be surprised by the novel 4-grams in the test data\u25e6Low accuracy on test setModels that are too powerful can overfitthe data\u25e6Fitting the details of the training data so exactly that the model doesn't generalize well to the test set\u25e6How to avoid overfitting?\u25e6Regularization in logistic regression \u25e6Dropout in neural networks82 RegularizationA solution for overfittingAdd a regularization term R(\u03b8) to the loss function (for now written as maximizing logprobrather than minimizing loss) Idea: choose an R(\u03b8)that penalizes large weights\u25e6fitting the data well with lots of big weights not as good as fitting the data a little less well, with small weights14CHAPTER5\u2022LOGISTICREGRESSIONdata to the unseen test set, but a model that over\ufb01ts will have poor generalization.To avoid over\ufb01tting, a newregularizationtermR(q)is added to the objectiveregularizationfunction in Eq.5.13, resulting in the following objective for a batch ofmexam-ples (slightly rewritten from Eq.5.13to be maximizing log probability rather thanminimizing loss, and removing the1mterm which doesn\u2019t affect the argmax):\u02c6q=argmaxqmXi=1logP(y(i)|x(i))\u0000aR(q)(5.22)The new regularization termR(q)is used to penalize large weights. Thus a settingof the weights that matches the training data perfectly\u2014 but uses many weights withhigh values to do so\u2014will be penalized more than a setting that matches the data alittle less well, but does so using smaller weights. There are two common ways tocompute this regularization termR(q).L2 regularizationis a quadratic function ofL2regularizationthe weight values, named because it uses the (square of the) L2 norm of the weightvalues. The L2 norm,||q||2, is the same as theEuclidean distanceof the vectorqfrom the origin. Ifqconsists ofnweights, then:R(q)=||q||22=nXj=1q2j(5.23)The L2 regularized objective function becomes:\u02c6q=argmaxq\"mXi=1logP(y(i)|x(i))#\u0000anXj=1q2j(5.24)L1 regularizationis a linear function of the weight values, named after the L1 normL1regularization||W||1, the sum of the absolute values of the weights, orManhattan distance(theManhattan distance is the distance you\u2019d have to walk between two points in a citywith a street grid like New York):R(q)=||q||1=nXi=1|qi|(5.25)The L1 regularized objective function becomes:\u02c6q=argmaxq\"mX1=ilogP(y(i)|x(i))#\u0000anXj=1|qj|(5.26)These kinds of regularization come from statistics, where L1 regularization is calledlasso regression(Tibshirani, 1996)and L2 regularization is calledridge regression,lassoridgeand both are commonly used in language processing. L2 regularization is easier tooptimize because of its simple derivative (the derivative ofq2is just 2q), whileL1 regularization is more complex (the derivative of|q|is non-continuous at zero).But where L2 prefers weight vectors with many small weights, L1 prefers sparsesolutions with some larger weights but many more weights set to zero. Thus L1regularization leads to much sparser weight vectors, that is, far fewer features.Both L1 and L2 regularization have Bayesian interpretations as constraints onthe prior of how weights should look. L1 regularization can be viewed as a Laplaceprior on the weights. L2 regularization corresponds to assuming that weights are L2 Regularization (= ridge regression)The sum of the squares of the weightsThe name is because this is the (square of the)        L2 norm||\u03b8||2, = Euclidean distance of \u03b8 to the origin.L2 regularized objective function:14CHAPTER5\u2022LOGISTICREGRESSIONdata to the unseen test set, but a model that over\ufb01ts will have poor generalization.To avoid over\ufb01tting, a newregularizationtermR(q)is added to the objectiveregularizationfunction in Eq.5.13, resulting in the following objective for a batch ofmexam-ples (slightly rewritten from Eq.5.13to be maximizing log probability rather thanminimizing loss, and removing the1mterm which doesn\u2019t affect the argmax):\u02c6q=argmaxqmXi=1logP(y(i)|x(i))\u0000aR(q)(5.22)The new regularization termR(q)is used to penalize large weights. Thus a settingof the weights that matches the training data perfectly\u2014 but uses many weights withhigh values to do so\u2014will be penalized more than a setting that matches the data alittle less well, but does so using smaller weights. There are two common ways tocompute this regularization termR(q).L2 regularizationis a quadratic function ofL2regularizationthe weight values, named because it uses the (square of the) L2 norm of the weightvalues. The L2 norm,||q||2, is the same as theEuclidean distanceof the vectorqfrom the origin. Ifqconsists ofnweights, then:R(q)=||q||22=nXj=1q2j(5.23)The L2 regularized objective function becomes:\u02c6q=argmaxq\"mXi=1logP(y(i)|x(i))#\u0000anXj=1q2j(5.24)L1 regularizationis a linear function of the weight values, named after the L1 normL1regularization||W||1, the sum of the absolute values of the weights, orManhattan distance(theManhattan distance is the distance you\u2019d have to walk between two points in a citywith a street grid like New York):R(q)=||q||1=nXi=1|qi|(5.25)The L1 regularized objective function becomes:\u02c6q=argmaxq\"mX1=ilogP(y(i)|x(i))#\u0000anXj=1|qj|(5.26)These kinds of regularization come from statistics, where L1 regularization is calledlasso regression(Tibshirani, 1996)and L2 regularization is calledridge regression,lassoridgeand both are commonly used in language processing. L2 regularization is easier tooptimize because of its simple derivative (the derivative ofq2is just 2q), whileL1 regularization is more complex (the derivative of|q|is non-continuous at zero).But where L2 prefers weight vectors with many small weights, L1 prefers sparsesolutions with some larger weights but many more weights set to zero. Thus L1regularization leads to much sparser weight vectors, that is, far fewer features.Both L1 and L2 regularization have Bayesian interpretations as constraints onthe prior of how weights should look. L1 regularization can be viewed as a Laplaceprior on the weights. L2 regularization corresponds to assuming that weights are14CHAPTER5\u2022LOGISTICREGRESSIONdata to the unseen test set, but a model that over\ufb01ts will have poor generalization.To avoid over\ufb01tting, a newregularizationtermR(q)is added to the objectiveregularizationfunction in Eq.5.13, resulting in the following objective for a batch ofmexam-ples (slightly rewritten from Eq.5.13to be maximizing log probability rather thanminimizing loss, and removing the1mterm which doesn\u2019t affect the argmax):\u02c6q=argmaxqmXi=1logP(y(i)|x(i))\u0000aR(q)(5.22)The new regularization termR(q)is used to penalize large weights. Thus a settingof the weights that matches the training data perfectly\u2014 but uses many weights withhigh values to do so\u2014will be penalized more than a setting that matches the data alittle less well, but does so using smaller weights. There are two common ways tocompute this regularization termR(q).L2 regularizationis a quadratic function ofL2regularizationthe weight values, named because it uses the (square of the) L2 norm of the weightvalues. The L2 norm,||q||2, is the same as theEuclidean distanceof the vectorqfrom the origin. Ifqconsists ofnweights, then:R(q)=||q||22=nXj=1q2j(5.23)The L2 regularized objective function becomes:\u02c6q=argmaxq\"mXi=1logP(y(i)|x(i))#\u0000anXj=1q2j(5.24)L1 regularizationis a linear function of the weight values, named after the L1 normL1regularization||W||1, the sum of the absolute values of the weights, orManhattan distance(theManhattan distance is the distance you\u2019d have to walk between two points in a citywith a street grid like New York):R(q)=||q||1=nXi=1|qi|(5.25)The L1 regularized objective function becomes:\u02c6q=argmaxq\"mX1=ilogP(y(i)|x(i))#\u0000anXj=1|qj|(5.26)These kinds of regularization come from statistics, where L1 regularization is calledlasso regression(Tibshirani, 1996)and L2 regularization is calledridge regression,lassoridgeand both are commonly used in language processing. L2 regularization is easier tooptimize because of its simple derivative (the derivative ofq2is just 2q), whileL1 regularization is more complex (the derivative of|q|is non-continuous at zero).But where L2 prefers weight vectors with many small weights, L1 prefers sparsesolutions with some larger weights but many more weights set to zero. Thus L1regularization leads to much sparser weight vectors, that is, far fewer features.Both L1 and L2 regularization have Bayesian interpretations as constraints onthe prior of how weights should look. L1 regularization can be viewed as a Laplaceprior on the weights. L2 regularization corresponds to assuming that weights are L1 Regularization (= lasso regression)The sum of the (absolute value of the) weightsNamed after the L1 norm ||W||1, = sum of the absolute values of the weights, = Manhattan distanceL1 regularized objective function:14CHAPTER5\u2022LOGISTICREGRESSIONdata to the unseen test set, but a model that over\ufb01ts will have poor generalization.To avoid over\ufb01tting, a newregularizationtermR(q)is added to the objectiveregularizationfunction in Eq.5.13, resulting in the following objective for a batch ofmexam-ples (slightly rewritten from Eq.5.13to be maximizing log probability rather thanminimizing loss, and removing the1mterm which doesn\u2019t affect the argmax):\u02c6q=argmaxqmXi=1logP(y(i)|x(i))\u0000aR(q)(5.22)The new regularization termR(q)is used to penalize large weights. Thus a settingof the weights that matches the training data perfectly\u2014 but uses many weights withhigh values to do so\u2014will be penalized more than a setting that matches the data alittle less well, but does so using smaller weights. There are two common ways tocompute this regularization termR(q).L2 regularizationis a quadratic function ofL2regularizationthe weight values, named because it uses the (square of the) L2 norm of the weightvalues. The L2 norm,||q||2, is the same as theEuclidean distanceof the vectorqfrom the origin. Ifqconsists ofnweights, then:R(q)=||q||22=nXj=1q2j(5.23)The L2 regularized objective function becomes:\u02c6q=argmaxq\"mXi=1logP(y(i)|x(i))#\u0000anXj=1q2j(5.24)L1 regularizationis a linear function of the weight values, named after the L1 normL1regularization||W||1, the sum of the absolute values of the weights, orManhattan distance(theManhattan distance is the distance you\u2019d have to walk between two points in a citywith a street grid like New York):R(q)=||q||1=nXi=1|qi|(5.25)The L1 regularized objective function becomes:\u02c6q=argmaxq\"mX1=ilogP(y(i)|x(i))#\u0000anXj=1|qj|(5.26)These kinds of regularization come from statistics, where L1 regularization is calledlasso regression(Tibshirani, 1996)and L2 regularization is calledridge regression,lassoridgeand both are commonly used in language processing. L2 regularization is easier tooptimize because of its simple derivative (the derivative ofq2is just 2q), whileL1 regularization is more complex (the derivative of|q|is non-continuous at zero).But where L2 prefers weight vectors with many small weights, L1 prefers sparsesolutions with some larger weights but many more weights set to zero. Thus L1regularization leads to much sparser weight vectors, that is, far fewer features.Both L1 and L2 regularization have Bayesian interpretations as constraints onthe prior of how weights should look. L1 regularization can be viewed as a Laplaceprior on the weights. L2 regularization corresponds to assuming that weights are14CHAPTER5\u2022LOGISTICREGRESSIONdata to the unseen test set, but a model that over\ufb01ts will have poor generalization.To avoid over\ufb01tting, a newregularizationtermR(q)is added to the objectiveregularizationfunction in Eq.5.13, resulting in the following objective for a batch ofmexam-ples (slightly rewritten from Eq.5.13to be maximizing log probability rather thanminimizing loss, and removing the1mterm which doesn\u2019t affect the argmax):\u02c6q=argmaxqmXi=1logP(y(i)|x(i))\u0000aR(q)(5.22)The new regularization termR(q)is used to penalize large weights. Thus a settingof the weights that matches the training data perfectly\u2014 but uses many weights withhigh values to do so\u2014will be penalized more than a setting that matches the data alittle less well, but does so using smaller weights. There are two common ways tocompute this regularization termR(q).L2 regularizationis a quadratic function ofL2regularizationthe weight values, named because it uses the (square of the) L2 norm of the weightvalues. The L2 norm,||q||2, is the same as theEuclidean distanceof the vectorqfrom the origin. Ifqconsists ofnweights, then:R(q)=||q||22=nXj=1q2j(5.23)The L2 regularized objective function becomes:\u02c6q=argmaxq\"mXi=1logP(y(i)|x(i))#\u0000anXj=1q2j(5.24)L1 regularizationis a linear function of the weight values, named after the L1 normL1regularization||W||1, the sum of the absolute values of the weights, orManhattan distance(theManhattan distance is the distance you\u2019d have to walk between two points in a citywith a street grid like New York):R(q)=||q||1=nXi=1|qi|(5.25)The L1 regularized objective function becomes:\u02c6q=argmaxq\"mX1=ilogP(y(i)|x(i))#\u0000anXj=1|qj|(5.26)These kinds of regularization come from statistics, where L1 regularization is calledlasso regression(Tibshirani, 1996)and L2 regularization is calledridge regression,lassoridgeand both are commonly used in language processing. L2 regularization is easier tooptimize because of its simple derivative (the derivative ofq2is just 2q), whileL1 regularization is more complex (the derivative of|q|is non-continuous at zero).But where L2 prefers weight vectors with many small weights, L1 prefers sparsesolutions with some larger weights but many more weights set to zero. Thus L1regularization leads to much sparser weight vectors, that is, far fewer features.Both L1 and L2 regularization have Bayesian interpretations as constraints onthe prior of how weights should look. L1 regularization can be viewed as a Laplaceprior on the weights. L2 regularization corresponds to assuming that weights are Logistic RegressionRegularization Logistic RegressionMultinomial Logistic Regression Multinomial Logistic RegressionOften we need more than 2 classes\u25e6Positive/negative/neutral\u25e6Parts of speech (noun, verb, adjective, adverb, preposition, etc.)\u25e6Classify emergency SMSs into different actionable classesIf >2 classes we use multinomial logistic regression= Softmaxregression= Multinomial logit= (defunct names : Maximum entropy modeling or MaxEntSo \"logistic regression\" will just mean binary (2 output classes)88 Multinomial Logistic RegressionThe probability of everything must still sum to 1P(positive|doc) + P(negative|doc) + P(neutral|doc) = 1Need a generalization of the sigmoid called the softmax\u25e6Takes a vector z = [z1, z2, ..., zk] of k arbitrary values \u25e6Outputs a probability distribution\u25e6each value in the range [0,1]\u25e6all the values summing to 189 The softmaxfunctionTurns a vector z = [z1, z2, ... , zk]of k arbitrary values into probabilities \n905.6\u2022MULTINOMIAL LOGISTIC REGRESSION15distributed according to a Gaussian distribution with mean\u00b5=0. In a Gaussianor normal distribution, the further away a value is from the mean, the lower itsprobability (scaled by the variances). By using a Gaussian prior on the weights, weare saying that weights prefer to have the value 0. A Gaussian for a weightqjis1q2ps2jexp \u0000(qj\u0000\u00b5j)22s2j!(5.27)If we multiply each weight by a Gaussian prior on the weight, we are thus maximiz-ing the following constraint:\u02c6q=argmaxqMYi=1P(y(i)|x(i))\u21e5nYj=11q2ps2jexp \u0000(qj\u0000\u00b5j)22s2j!(5.28)which in log space, with\u00b5=0, and assuming 2s2=1, corresponds to\u02c6q=argmaxqmXi=1logP(y(i)|x(i))\u0000anXj=1q2j(5.29)which is in the same form as Eq.5.24.5.6 Multinomial logistic regressionSometimes we need more than two classes. Perhaps we might want to do 3-waysentiment classi\ufb01cation (positive, negative, or neutral). Or we could be assigningsome of the labels we will introduce in Chapter 8, like the part of speech of a word(choosing from 10, 30, or even 50 different parts of speech), or the named entitytype of a phrase (choosing from tags like person, location, organization).In such cases we usemultinomial logistic regression, also calledsoftmax re-multinomiallogisticregressiongression(or, historically, themaxent classi\ufb01er). In multinomial logistic regressionthe targetyis a variable that ranges over more than two classes; we want to knowthe probability ofybeing in each potential classc2C,p(y=c|x).The multinomial logistic classi\ufb01er uses a generalization of the sigmoid, calledthesoftmaxfunction, to compute the probabilityp(y=c|x). The softmax functionsoftmaxtakes a vectorz=[z1,z2,. . . ,zk]ofkarbitrary values and maps them to a probabilitydistribution, with each value in the range (0,1), and all the values summing to 1.Like the sigmoid, it is an exponential function.For a vectorzof dimensionalityk, the softmax is de\ufb01ned as:softmax(zi)=exp(zi)Pkj=1exp(zj)1\uf8ffi\uf8ffk(5.30)The softmax of an input vectorz=[z1,z2,. . . ,zk]is thus a vector itself:softmax(z)=\"exp(z1)Pki=1exp(zi),exp(z2)Pki=1exp(zi),. . . ,exp(zk)Pki=1exp(zi)#(5.31)5.6\u2022MULTINOMIAL LOGISTIC REGRESSION15distributed according to a Gaussian distribution with mean\u00b5=0. In a Gaussianor normal distribution, the further away a value is from the mean, the lower itsprobability (scaled by the variances). By using a Gaussian prior on the weights, weare saying that weights prefer to have the value 0. A Gaussian for a weightqjis1q2ps2jexp \u0000(qj\u0000\u00b5j)22s2j!(5.27)If we multiply each weight by a Gaussian prior on the weight, we are thus maximiz-ing the following constraint:\u02c6q=argmaxqMYi=1P(y(i)|x(i))\u21e5nYj=11q2ps2jexp \u0000(qj\u0000\u00b5j)22s2j!(5.28)which in log space, with\u00b5=0, and assuming 2s2=1, corresponds to\u02c6q=argmaxqmXi=1logP(y(i)|x(i))\u0000anXj=1q2j(5.29)which is in the same form as Eq.5.24.5.6 Multinomial logistic regressionSometimes we need more than two classes. Perhaps we might want to do 3-waysentiment classi\ufb01cation (positive, negative, or neutral). Or we could be assigningsome of the labels we will introduce in Chapter 8, like the part of speech of a word(choosing from 10, 30, or even 50 different parts of speech), or the named entitytype of a phrase (choosing from tags like person, location, organization).In such cases we usemultinomial logistic regression, also calledsoftmax re-multinomiallogisticregressiongression(or, historically, themaxent classi\ufb01er). In multinomial logistic regressionthe targetyis a variable that ranges over more than two classes; we want to knowthe probability ofybeing in each potential classc2C,p(y=c|x).The multinomial logistic classi\ufb01er uses a generalization of the sigmoid, calledthesoftmaxfunction, to compute the probabilityp(y=c|x). The softmax functionsoftmaxtakes a vectorz=[z1,z2,. . . ,zk]ofkarbitrary values and maps them to a probabilitydistribution, with each value in the range (0,1), and all the values summing to 1.Like the sigmoid, it is an exponential function.For a vectorzof dimensionalityk, the softmax is de\ufb01ned as:softmax(zi)=exp(zi)Pkj=1exp(zj)1\uf8ffi\uf8ffk(5.30)The softmax of an input vectorz=[z1,z2,. . . ,zk]is thus a vector itself:softmax(z)=\"exp(z1)Pki=1exp(zi),exp(z2)Pki=1exp(zi),. . . ,exp(zk)Pki=1exp(zi)#(5.31)5.6\u2022MULTINOMIAL LOGISTIC REGRESSION15For a vectorzof dimensionalityk, the softmax is de\ufb01ned as:softmax(zi)=eziPkj=1ezj1\uf8ffi\uf8ffk(5.32)The softmax of an input vectorz=[z1,z2,. . . ,zk]is thus a vector itself:softmax(z)=\"ez1Pki=1ezi,ez2Pki=1ezi,. . . ,ezkPki=1ezi#(5.33)The denominatorPki=1eziis used to normalize all the values into probabilities.Thus for example given a vector:z=[0.6,1.1,\u00001.5,1.2,3.2,\u00001.1]the result softmax(z) is[0.055,0.090,0.0067,0.10,0.74,0.010]Again like the sigmoid, the input to the softmax will be the dot product betweena weight vectorwand an input vectorx(plus a bias). But now we\u2019ll need separateweight vectors (and bias) for each of theKclasses.p(y=c|x)=ewc\u00b7x+bckXj=1ewj\u00b7x+bj(5.34)Like the sigmoid, the softmax has the property of squashing values toward 0 or1. thus if one of the inputs is larger than the others, will tend to push its probabilitytoward 1, and suppress the probabilities of the smaller inputs.5.6.1 Features in Multinomial Logistic RegressionFor multiclass classi\ufb01cation the input features need to be a function of both theobservationxand the candidate output classc. Thus instead of the notationxi,fiorfi(x), when we\u2019re discussing features we will use the notationfi(c,x), meaningfeatureifor a particular classcfor a given observationx.In binary classi\ufb01cation, a positive weight on a feature pointed toward y=1 anda negative weight toward y=0... but in multiclass a feature could be evidence for oragainst an individual class.Let\u2019s look at some sample features for a few NLP tasks to help understand thisperhaps unintuitive use of features that are functions of both the observationxandthe classc,Suppose we are doing text classi\ufb01cation, and instead of binary classi\ufb01cation ourtask is to assign one of the 3 classes+,\u0000, or 0 (neutral) to a document. Now afeature related to exclamation marks might have a negative weight for 0 documents,and a positive weight for+or\u0000documents: The softmaxfunction\u25e6Turns  a vector z = [z1,z2,...,zk]of k arbitrary values into probabilities \n915.6\u2022MULTINOMIAL LOGISTIC REGRESSION15For a vectorzof dimensionalityk, the softmax is de\ufb01ned as:softmax(zi)=eziPkj=1ezj1\uf8ffi\uf8ffk(5.32)The softmax of an input vectorz=[z1,z2,. . . ,zk]is thus a vector itself:softmax(z)=\"ez1Pki=1ezi,ez2Pki=1ezi,. . . ,ezkPki=1ezi#(5.33)The denominatorPki=1eziis used to normalize all the values into probabilities.Thus for example given a vector:z=[0.6,1.1,\u00001.5,1.2,3.2,\u00001.1]the result softmax(z) is[0.055,0.090,0.0067,0.10,0.74,0.010]Again like the sigmoid, the input to the softmax will be the dot product betweena weight vectorwand an input vectorx(plus a bias). But now we\u2019ll need separateweight vectors (and bias) for each of theKclasses.p(y=c|x)=ewc\u00b7x+bckXj=1ewj\u00b7x+bj(5.34)Like the sigmoid, the softmax has the property of squashing values toward 0 or1. thus if one of the inputs is larger than the others, will tend to push its probabilitytoward 1, and suppress the probabilities of the smaller inputs.5.6.1 Features in Multinomial Logistic RegressionFor multiclass classi\ufb01cation the input features need to be a function of both theobservationxand the candidate output classc. Thus instead of the notationxi,fiorfi(x), when we\u2019re discussing features we will use the notationfi(c,x), meaningfeatureifor a particular classcfor a given observationx.In binary classi\ufb01cation, a positive weight on a feature pointed toward y=1 anda negative weight toward y=0... but in multiclass a feature could be evidence for oragainst an individual class.Let\u2019s look at some sample features for a few NLP tasks to help understand thisperhaps unintuitive use of features that are functions of both the observationxandthe classc,Suppose we are doing text classi\ufb01cation, and instead of binary classi\ufb01cation ourtask is to assign one of the 3 classes+,\u0000, or 0 (neutral) to a document. Now afeature related to exclamation marks might have a negative weight for 0 documents,and a positive weight for+or\u0000documents:5.6\u2022MULTINOMIAL LOGISTIC REGRESSION15For a vectorzof dimensionalityk, the softmax is de\ufb01ned as:softmax(zi)=eziPkj=1ezj1\uf8ffi\uf8ffk(5.32)The softmax of an input vectorz=[z1,z2,. . . ,zk]is thus a vector itself:softmax(z)=\"ez1Pki=1ezi,ez2Pki=1ezi,. . . ,ezkPki=1ezi#(5.33)The denominatorPki=1eziis used to normalize all the values into probabilities.Thus for example given a vector:z=[0.6,1.1,\u00001.5,1.2,3.2,\u00001.1]the result softmax(z) is[0.055,0.090,0.0067,0.10,0.74,0.010]Again like the sigmoid, the input to the softmax will be the dot product betweena weight vectorwand an input vectorx(plus a bias). But now we\u2019ll need separateweight vectors (and bias) for each of theKclasses.p(y=c|x)=ewc\u00b7x+bckXj=1ewj\u00b7x+bj(5.34)Like the sigmoid, the softmax has the property of squashing values toward 0 or1. thus if one of the inputs is larger than the others, will tend to push its probabilitytoward 1, and suppress the probabilities of the smaller inputs.5.6.1 Features in Multinomial Logistic RegressionFor multiclass classi\ufb01cation the input features need to be a function of both theobservationxand the candidate output classc. Thus instead of the notationxi,fiorfi(x), when we\u2019re discussing features we will use the notationfi(c,x), meaningfeatureifor a particular classcfor a given observationx.In binary classi\ufb01cation, a positive weight on a feature pointed toward y=1 anda negative weight toward y=0... but in multiclass a feature could be evidence for oragainst an individual class.Let\u2019s look at some sample features for a few NLP tasks to help understand thisperhaps unintuitive use of features that are functions of both the observationxandthe classc,Suppose we are doing text classi\ufb01cation, and instead of binary classi\ufb01cation ourtask is to assign one of the 3 classes+,\u0000, or 0 (neutral) to a document. Now afeature related to exclamation marks might have a negative weight for 0 documents,and a positive weight for+or\u0000documents:5.6\u2022MULTINOMIAL LOGISTIC REGRESSION15distributed according to a Gaussian distribution with mean\u00b5=0. In a Gaussianor normal distribution, the further away a value is from the mean, the lower itsprobability (scaled by the variances). By using a Gaussian prior on the weights, weare saying that weights prefer to have the value 0. A Gaussian for a weightqjis1q2ps2jexp \u0000(qj\u0000\u00b5j)22s2j!(5.27)If we multiply each weight by a Gaussian prior on the weight, we are thus maximiz-ing the following constraint:\u02c6q=argmaxqMYi=1P(y(i)|x(i))\u21e5nYj=11q2ps2jexp \u0000(qj\u0000\u00b5j)22s2j!(5.28)which in log space, with\u00b5=0, and assuming 2s2=1, corresponds to\u02c6q=argmaxqmXi=1logP(y(i)|x(i))\u0000anXj=1q2j(5.29)which is in the same form as Eq.5.24.5.6 Multinomial logistic regressionSometimes we need more than two classes. Perhaps we might want to do 3-waysentiment classi\ufb01cation (positive, negative, or neutral). Or we could be assigningsome of the labels we will introduce in Chapter 8, like the part of speech of a word(choosing from 10, 30, or even 50 different parts of speech), or the named entitytype of a phrase (choosing from tags like person, location, organization).In such cases we usemultinomial logistic regression, also calledsoftmax re-multinomiallogisticregressiongression(or, historically, themaxent classi\ufb01er). In multinomial logistic regressionthe targetyis a variable that ranges over more than two classes; we want to knowthe probability ofybeing in each potential classc2C,p(y=c|x).The multinomial logistic classi\ufb01er uses a generalization of the sigmoid, calledthesoftmaxfunction, to compute the probabilityp(y=c|x). The softmax functionsoftmaxtakes a vectorz=[z1,z2,. . . ,zk]ofkarbitrary values and maps them to a probabilitydistribution, with each value in the range (0,1), and all the values summing to 1.Like the sigmoid, it is an exponential function.For a vectorzof dimensionalityk, the softmax is de\ufb01ned as:softmax(zi)=exp(zi)Pkj=1exp(zj)1\uf8ffi\uf8ffk(5.30)The softmax of an input vectorz=[z1,z2,. . . ,zk]is thus a vector itself:softmax(z)=\"exp(z1)Pki=1exp(zi),exp(z2)Pki=1exp(zi),. . . ,exp(zk)Pki=1exp(zi)#(5.31) Softmaxin multinomial logistic regression16CHAPTER5\u2022LOGISTICREGRESSIONThe denominatorPki=1exp(zi)is used to normalize all the values into probabil-ities. Thus for example given a vector:z=[0.6,1.1,\u00001.5,1.2,3.2,\u00001.1]the resulting (rounded) softmax(z) is[0.055,0.090,0.006,0.099,0.74,0.010]Again like the sigmoid, the input to the softmax will be the dot product betweena weight vectorwand an input vectorx(plus a bias). But now we\u2019ll need separateweight vectors (and bias) for each of theKclasses.p(y=c|x)=exp(wc\u00b7x+bc)kXj=1exp(wj\u00b7x+bj)(5.32)Like the sigmoid, the softmax has the property of squashing values toward 0 or 1.Thus if one of the inputs is larger than the others, it will tend to push its probabilitytoward 1, and suppress the probabilities of the smaller inputs.5.6.1 Features in Multinomial Logistic RegressionFeatures in multinomial logistic regression function similarly to binary logistic re-gression, with one difference that we\u2019ll need separate weight vectors (and biases) foreach of theKclasses. Recall our binary exclamation point featurex5from page79:x5=\u21e21 if \u201c!\u201d2doc0 otherwiseIn binary classi\ufb01cation a positive weightw5on a feature in\ufb02uences the classi\ufb01ertowardy=1 (positive sentiment) and a negative weight in\ufb02uences it towardy=0(negative sentiment) with the absolute value indicating how important the featureis. For multinominal logistic regression, by contrast, with separate weights for eachclass, a feature can be evidence for or against each individual class.In 3-way multiclass sentiment classi\ufb01cation, for example, we must assign eachdocument one of the 3 classes+,\u0000, or 0 (neutral). Now a feature related to excla-mation marks might have a negative weight for 0 documents, and a positive weightfor+or\u0000documents:Feature De\ufb01nitionw5,+w5,\u0000w5,0f5(x)\u21e21 if \u201c!\u201d2doc0 otherwise3.53.1\u00005.35.6.2 Learning in Multinomial Logistic RegressionThe loss function for multinomial logistic regression generalizes the loss functionfor binary logistic regression from 2 toKclasses. Recall that that the cross-entropyloss for binary logistic regression (repeated from Eq.5.11) is:LCE(\u02c6y,y)=\u0000logp(y|x)=\u0000[ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)](5.33)92Input is still the dot product between weight vector w and input vector xBut now we\u2019ll need separate weight vectors for each of the K classes.  Features in binary versus multinomial logistic regressionBinary: positive weight \u00e0y=1  neg weight \u00e0y=0Multinominal: separate weights for each class:\n9316CHAPTER5\u2022LOGISTICREGRESSIONThe denominatorPki=1exp(zi)is used to normalize all the values into probabil-ities. Thus for example given a vector:z=[0.6,1.1,\u00001.5,1.2,3.2,\u00001.1]the resulting (rounded) softmax(z) is[0.055,0.090,0.006,0.099,0.74,0.010]Again like the sigmoid, the input to the softmax will be the dot product betweena weight vectorwand an input vectorx(plus a bias). But now we\u2019ll need separateweight vectors (and bias) for each of theKclasses.p(y=c|x)=exp(wc\u00b7x+bc)kXj=1exp(wj\u00b7x+bj)(5.32)Like the sigmoid, the softmax has the property of squashing values toward 0 or 1.Thus if one of the inputs is larger than the others, it will tend to push its probabilitytoward 1, and suppress the probabilities of the smaller inputs.5.6.1 Features in Multinomial Logistic RegressionFeatures in multinomial logistic regression function similarly to binary logistic re-gression, with one difference that we\u2019ll need separate weight vectors (and biases) foreach of theKclasses. Recall our binary exclamation point featurex5from page79:x5=\u21e21 if \u201c!\u201d2doc0 otherwiseIn binary classi\ufb01cation a positive weightw5on a feature in\ufb02uences the classi\ufb01ertowardy=1 (positive sentiment) and a negative weight in\ufb02uences it towardy=0(negative sentiment) with the absolute value indicating how important the featureis. For multinominal logistic regression, by contrast, with separate weights for eachclass, a feature can be evidence for or against each individual class.In 3-way multiclass sentiment classi\ufb01cation, for example, we must assign eachdocument one of the 3 classes+,\u0000, or 0 (neutral). Now a feature related to excla-mation marks might have a negative weight for 0 documents, and a positive weightfor+or\u0000documents:Feature De\ufb01nitionw5,+w5,\u0000w5,0f5(x)\u21e21 if \u201c!\u201d2doc0 otherwise3.53.1\u00005.35.6.2 Learning in Multinomial Logistic RegressionThe loss function for multinomial logistic regression generalizes the loss functionfor binary logistic regression from 2 toKclasses. Recall that that the cross-entropyloss for binary logistic regression (repeated from Eq.5.11) is:LCE(\u02c6y,y)=\u0000logp(y|x)=\u0000[ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)](5.33)w5= 3.016CHAPTER5\u2022LOGISTICREGRESSIONThe denominatorPki=1exp(zi)is used to normalize all the values into probabil-ities. Thus for example given a vector:z=[0.6,1.1,\u00001.5,1.2,3.2,\u00001.1]the resulting (rounded) softmax(z) is[0.055,0.090,0.006,0.099,0.74,0.010]Again like the sigmoid, the input to the softmax will be the dot product betweena weight vectorwand an input vectorx(plus a bias). But now we\u2019ll need separateweight vectors (and bias) for each of theKclasses.p(y=c|x)=exp(wc\u00b7x+bc)kXj=1exp(wj\u00b7x+bj)(5.32)Like the sigmoid, the softmax has the property of squashing values toward 0 or 1.Thus if one of the inputs is larger than the others, it will tend to push its probabilitytoward 1, and suppress the probabilities of the smaller inputs.5.6.1 Features in Multinomial Logistic RegressionFeatures in multinomial logistic regression function similarly to binary logistic re-gression, with one difference that we\u2019ll need separate weight vectors (and biases) foreach of theKclasses. Recall our binary exclamation point featurex5from page79:x5=\u21e21 if \u201c!\u201d2doc0 otherwiseIn binary classi\ufb01cation a positive weightw5on a feature in\ufb02uences the classi\ufb01ertowardy=1 (positive sentiment) and a negative weight in\ufb02uences it towardy=0(negative sentiment) with the absolute value indicating how important the featureis. For multinominal logistic regression, by contrast, with separate weights for eachclass, a feature can be evidence for or against each individual class.In 3-way multiclass sentiment classi\ufb01cation, for example, we must assign eachdocument one of the 3 classes+,\u0000, or 0 (neutral). Now a feature related to excla-mation marks might have a negative weight for 0 documents, and a positive weightfor+or\u0000documents:Feature De\ufb01nitionw5,+w5,\u0000w5,0f5(x)\u21e21 if \u201c!\u201d2doc0 otherwise3.53.1\u00005.35.6.2 Learning in Multinomial Logistic RegressionThe loss function for multinomial logistic regression generalizes the loss functionfor binary logistic regression from 2 toKclasses. Recall that that the cross-entropyloss for binary logistic regression (repeated from Eq.5.11) is:LCE(\u02c6y,y)=\u0000logp(y|x)=\u0000[ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)](5.33) Logistic RegressionMultinomial Logistic Regression", "6.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\n6Vector Semantics and\nEmbeddings\n\u8343\u8005\u6240\u4ee5\u5728\u9c7c\uff0c\u5f97\u9c7c\u800c\u5fd8\u8343Nets are for \ufb01sh;\nOnce you get the \ufb01sh, you can forget the net.\n\u8a00\u8005\u6240\u4ee5\u5728\u610f\uff0c\u5f97\u610f\u800c\u5fd8\u8a00Words are for meaning;\nOnce you get the meaning, you can forget the words\n\u5e84\u5b50(Zhuangzi), Chapter 26\nThe asphalt that Los Angeles is famous for occurs mainly on its freeways. But\nin the middle of the city is another patch of asphalt, the La Brea tar pits, and this\nasphalt preserves millions of fossil bones from the last of the Ice Ages of the Pleis-\ntocene Epoch. One of these fossils is the Smilodon , or saber-toothed tiger, instantly\nrecognizable by its long canines. Five million years ago or so, a completely different\nsaber-tooth tiger called Thylacosmilus lived\nin Argentina and other parts of South Amer-\nica. Thylacosmilus was a marsupial whereas\nSmilodon was a placental mammal, but Thy-\nlacosmilus had the same long upper canines\nand, like Smilodon, had a protective bone\n\ufb02ange on the lower jaw. The similarity of\nthese two mammals is one of many examples\nof parallel or convergent evolution, in which particular contexts or environments\nlead to the evolution of very similar structures in different species (Gould, 1980).\nThe role of context is also important in the similarity of a less biological kind\nof organism: the word. Words that occur in similar contexts tend to have similar\nmeanings . This link between similarity in how words are distributed and similarity\nin what they mean is called the distributional hypothesis . The hypothesis wasdistributional\nhypothesis\n\ufb01rst formulated in the 1950s by linguists like Joos (1950), Harris (1954), and Firth\n(1957), who noticed that words which are synonyms (like oculist andeye-doctor )\ntended to occur in the same environment (e.g., near words like eyeorexamined )\nwith the amount of meaning difference between two words \u201ccorresponding roughly\nto the amount of difference in their environments\u201d (Harris, 1954, p. 157).\nIn this chapter we introduce vector semantics , which instantiates this linguisticvector\nsemantics\nhypothesis by learning representations of the meaning of words, called embeddings , embeddings\ndirectly from their distributions in texts. These representations are used in every nat-\nural language processing application that makes use of meaning, and the static em-\nbeddings we introduce here underlie the more powerful dynamic or contextualized\nembeddings likeBERT that we will see in Chapter 11.\nThese word representations are also the \ufb01rst example in this book of repre-\nsentation learning , automatically learning useful representations of the input text.representation\nlearning\nFinding such self-supervised ways to learn representations of the input, instead of\ncreating representations by hand via feature engineering , is an important focus of\nNLP research (Bengio et al., 2013). 2CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\n6.1 Lexical Semantics\nLet\u2019s begin by introducing some basic principles of word meaning. How should\nwe represent the meaning of a word? In the n-gram models of Chapter 3, and in\nclassical NLP applications, our only representation of a word is as a string of letters,\nor an index in a vocabulary list. This representation is not that different from a\ntradition in philosophy, perhaps you\u2019ve seen it in introductory logic classes, in which\nthe meaning of words is represented by just spelling the word with small capital\nletters; representing the meaning of \u201cdog\u201d as DOG, and \u201ccat\u201d as CAT, or by using an\napostrophe ( DOG \u2019).\nRepresenting the meaning of a word by capitalizing it is a pretty unsatisfactory\nmodel. You might have seen a version of a joke due originally to semanticist Barbara\nPartee (Carlson, 1977):\nQ: What\u2019s the meaning of life?\nA:LIFE \u2019\nSurely we can do better than this! After all, we\u2019ll want a model of word meaning\nto do all sorts of things for us. It should tell us that some words have similar mean-\nings ( catis similar to dog), others are antonyms ( cold is the opposite of hot), some\nhave positive connotations ( happy ) while others have negative connotations ( sad). It\nshould represent the fact that the meanings of buy,sell, and payoffer differing per-\nspectives on the same underlying purchasing event. (If I buy something from you,\nyou\u2019ve probably sold it to me, and I likely paid you.) More generally, a model of\nword meaning should allow us to draw inferences to address meaning-related tasks\nlike question-answering or dialogue.\nIn this section we summarize some of these desiderata, drawing on results in the\nlinguistic study of word meaning, which is called lexical semantics ; we\u2019ll return tolexical\nsemantics\nand expand on this list in Appendix G and Chapter 21.\nLemmas and Senses Let\u2019s start by looking at how one word (we\u2019ll choose mouse )\nmight be de\ufb01ned in a dictionary (simpli\ufb01ed from the online dictionary WordNet):\nmouse (N)\n1. any of numerous small rodents...\n2. a hand-operated device that controls a cursor...\nHere the form mouse is the lemma , also called the citation form . The form lemma\ncitation form mouse would also be the lemma for the word mice ; dictionaries don\u2019t have separate\nde\ufb01nitions for in\ufb02ected forms like mice . Similarly sing is the lemma for sing,sang ,\nsung . In many languages the in\ufb01nitive form is used as the lemma for the verb, so\nSpanish dormir \u201cto sleep\u201d is the lemma for duermes \u201cyou sleep\u201d. The speci\ufb01c forms\nsung orcarpets orsing orduermes are called wordforms . wordform\nAs the example above shows, each lemma can have multiple meanings; the\nlemma mouse can refer to the rodent or the cursor control device. We call each\nof these aspects of the meaning of mouse aword sense . The fact that lemmas can\nbepolysemous (have multiple senses) can make interpretation dif\ufb01cult (is someone\nwho types \u201cmouse info\u201d into a search engine looking for a pet or a tool?). Chap-\nter 11 and Appendix G will discuss the problem of polysemy, and introduce word\nsense disambiguation , the task of determining which sense of a word is being used\nin a particular context.\nSynonymy One important component of word meaning is the relationship be-\ntween word senses. For example when one word has a sense whose meaning is 6.1 \u2022 L EXICAL SEMANTICS 3\nidentical to a sense of another word, or nearly identical, we say the two senses of\nthose two words are synonyms . Synonyms include such pairs as synonym\ncouch/sofa vomit/throw up \ufb01lbert/hazelnut car/automobile\nA more formal de\ufb01nition of synonymy (between words rather than senses) is that\ntwo words are synonymous if they are substitutable for one another in any sentence\nwithout changing the truth conditions of the sentence, the situations in which the\nsentence would be true.\nWhile substitutions between some pairs of words like car/automobile orwa-\nter/H2Oare truth preserving, the words are still not identical in meaning. Indeed,\nprobably no two words are absolutely identical in meaning. One of the fundamental\ntenets of semantics, called the principle of contrast (Girard 1718, Br \u00b4eal 1897, Clarkprinciple of\ncontrast\n1987), states that a difference in linguistic form is always associated with some dif-\nference in meaning. For example, the word H2Ois used in scienti\ufb01c contexts and\nwould be inappropriate in a hiking guide\u2014 water would be more appropriate\u2014 and\nthis genre difference is part of the meaning of the word. In practice, the word syn-\nonym is therefore used to describe a relationship of approximate or rough synonymy.\nWord Similarity While words don\u2019t have many synonyms, most words do have\nlots of similar words. Catis not a synonym of dog, but cats anddogs are certainly\nsimilar words. In moving from synonymy to similarity, it will be useful to shift from\ntalking about relations between word senses (like synonymy) to relations between\nwords (like similarity). Dealing with words avoids having to commit to a particular\nrepresentation of word senses, which will turn out to simplify our task.\nThe notion of word similarity is very useful in larger semantic tasks. Knowing similarity\nhow similar two words are can help in computing how similar the meaning of two\nphrases or sentences are, a very important component of tasks like question answer-\ning, paraphrasing, and summarization. One way of getting values for word similarity\nis to ask humans to judge how similar one word is to another. A number of datasets\nhave resulted from such experiments. For example the SimLex-999 dataset (Hill\net al., 2015) gives values on a scale from 0 to 10, like the examples below, which\nrange from near-synonyms ( vanish ,disappear ) to pairs that scarcely seem to have\nanything in common ( hole,agreement ):\nvanish disappear 9.8\nbelief impression 5.95\nmuscle bone 3.65\nmodest \ufb02exible 0.98\nhole agreement 0.3\nWord Relatedness The meaning of two words can be related in ways other than\nsimilarity. One such class of connections is called word relatedness (Budanitsky relatedness\nand Hirst, 2006), also traditionally called word association in psychology. association\nConsider the meanings of the words coffee andcup. Coffee is not similar to cup;\nthey share practically no features (coffee is a plant or a beverage, while a cup is a\nmanufactured object with a particular shape). But coffee and cup are clearly related;\nthey are associated by co-participating in an everyday event (the event of drinking\ncoffee out of a cup). Similarly scalpel andsurgeon are not similar but are related\neventively (a surgeon tends to make use of a scalpel).\nOne common kind of relatedness between words is if they belong to the same\nsemantic \ufb01eld . A semantic \ufb01eld is a set of words which cover a particular semantic semantic \ufb01eld\ndomain and bear structured relations with each other. For example, words might be 4CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\nrelated by being in the semantic \ufb01eld of hospitals ( surgeon ,scalpel ,nurse ,anes-\nthetic ,hospital ), restaurants ( waiter ,menu ,plate ,food,chef), or houses ( door ,roof,\nkitchen ,family ,bed). Semantic \ufb01elds are also related to topic models , like Latent topic models\nDirichlet Allocation ,LDA , which apply unsupervised learning on large sets of texts\nto induce sets of associated words from text. Semantic \ufb01elds and topic models are\nvery useful tools for discovering topical structure in documents.\nIn Appendix G we\u2019ll introduce more relations between senses like hypernymy\norIS-A ,antonymy (opposites) and meronymy (part-whole relations).\nSemantic Frames and Roles Closely related to semantic \ufb01elds is the idea of a\nsemantic frame . A semantic frame is a set of words that denote perspectives or semantic frame\nparticipants in a particular type of event. A commercial transaction, for example,\nis a kind of event in which one entity trades money to another entity in return for\nsome good or service, after which the good changes hands or perhaps the service is\nperformed. This event can be encoded lexically by using verbs like buy(the event\nfrom the perspective of the buyer), sell(from the perspective of the seller), pay\n(focusing on the monetary aspect), or nouns like buyer . Frames have semantic roles\n(like buyer ,seller ,goods ,money ), and words in a sentence can take on these roles.\nKnowing that buyandsellhave this relation makes it possible for a system to\nknow that a sentence like Sam bought the book from Ling could be paraphrased as\nLing sold the book to Sam , and that Sam has the role of the buyer in the frame and\nLing the seller . Being able to recognize such paraphrases is important for question\nanswering, and can help in shifting perspective for machine translation.\nConnotation Finally, words have affective meanings orconnotations . The word connotations\nconnotation has different meanings in different \ufb01elds, but here we use it to mean the\naspects of a word\u2019s meaning that are related to a writer or reader\u2019s emotions, senti-\nment, opinions, or evaluations. For example some words have positive connotations\n(wonderful ) while others have negative connotations ( dreary ). Even words whose\nmeanings are similar in other ways can vary in connotation; consider the difference\nin connotations between fake,knockoff ,forgery , on the one hand, and copy ,replica ,\nreproduction on the other, or innocent (positive connotation) and naive (negative\nconnotation). Some words describe positive evaluation ( great ,love) and others neg-\native evaluation ( terrible ,hate). Positive or negative evaluation language is called\nsentiment , as we saw in Chapter 4, and word sentiment plays a role in important sentiment\ntasks like sentiment analysis, stance detection, and applications of NLP to the lan-\nguage of politics and consumer reviews.\nEarly work on affective meaning (Osgood et al., 1957) found that words varied\nalong three important dimensions of affective meaning:\nvalence: the pleasantness of the stimulus\narousal: the intensity of emotion provoked by the stimulus\ndominance: the degree of control exerted by the stimulus\nThus words like happy orsatis\ufb01ed are high on valence, while unhappy oran-\nnoyed are low on valence. Excited is high on arousal, while calm is low on arousal.\nControlling is high on dominance, while awed orin\ufb02uenced are low on dominance.\nEach word is thus represented by three numbers, corresponding to its value on each\nof the three dimensions: 6.2 \u2022 V ECTOR SEMANTICS 5\nValence Arousal Dominance\ncourageous 8.05 5.5 7.38\nmusic 7.67 5.57 6.5\nheartbreak 2.45 5.65 3.58\ncub 6.71 3.95 4.24\nOsgood et al. (1957) noticed that in using these 3 numbers to represent the\nmeaning of a word, the model was representing each word as a point in a three-\ndimensional space, a vector whose three dimensions corresponded to the word\u2019s\nrating on the three scales. This revolutionary idea that word meaning could be rep-\nresented as a point in space (e.g., that part of the meaning of heartbreak can be\nrepresented as the point [2:45;5:65;3:58]) was the \ufb01rst expression of the vector se-\nmantics models that we introduce next.\n6.2 Vector Semantics\nVector semantics is the standard way to represent word meaning in NLP, helpingvector\nsemantics\nus model many of the aspects of word meaning we saw in the previous section. The\nroots of the model lie in the 1950s when two big ideas converged: Osgood\u2019s 1957\nidea mentioned above to use a point in three-dimensional space to represent the\nconnotation of a word, and the proposal by linguists like Joos (1950), Harris (1954),\nand Firth (1957) to de\ufb01ne the meaning of a word by its distribution in language\nuse, meaning its neighboring words or grammatical environments. Their idea was\nthat two words that occur in very similar distributions (whose neighboring words are\nsimilar) have similar meanings.\nFor example, suppose you didn\u2019t know the meaning of the word ongchoi (a re-\ncent borrowing from Cantonese) but you see it in the following contexts:\n(6.1) Ongchoi is delicious sauteed with garlic.\n(6.2) Ongchoi is superb over rice.\n(6.3) ...ongchoi leaves with salty sauces...\nAnd suppose that you had seen many of these context words in other contexts:\n(6.4) ...spinach sauteed with garlic over rice...\n(6.5) ...chard stems and leaves are delicious...\n(6.6) ...collard greens and other salty leafy greens\nThe fact that ongchoi occurs with words like riceandgarlic anddelicious and\nsalty , as do words like spinach ,chard , and collard greens might suggest that ongchoi\nis a leafy green similar to these other leafy greens.1We can do the same thing\ncomputationally by just counting words in the context of ongchoi .\nThe idea of vector semantics is to represent a word as a point in a multidimen-\nsional semantic space that is derived (in ways we\u2019ll see) from the distributions of\nword neighbors. Vectors for representing words are called embeddings (although embeddings\nthe term is sometimes more strictly applied only to dense vectors like word2vec\n(Section 6.8), rather than sparse tf-idf or PPMI vectors (Section 6.3-Section 6.6)).\nThe word \u201cembedding\u201d derives from its mathematical sense as a mapping from one\nspace or structure to another, although the meaning has shifted; see the end of the\nchapter.\n1It\u2019s in fact Ipomoea aquatica , a relative of morning glory sometimes called water spinach in English. 6CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\ngoodnicebadworstnot good\nwonderfulamazingterri\ufb01cdislikeworsevery goodincredibly goodfantasticincredibly badnowyouithatwithbyto\u2019sareisathan\nFigure 6.1 A two-dimensional (t-SNE) projection of embeddings for some words and\nphrases, showing that words with similar meanings are nearby in space. The original 60-\ndimensional embeddings were trained for sentiment analysis. Simpli\ufb01ed from Li et al. (2015)\nwith colors added for explanation.\nFig. 6.1 shows a visualization of embeddings learned for sentiment analysis,\nshowing the location of selected words projected down from 60-dimensional space\ninto a two dimensional space. Notice the distinct regions containing positive words,\nnegative words, and neutral function words.\nThe \ufb01ne-grained model of word similarity of vector semantics offers enormous\npower to NLP applications. NLP applications like the sentiment classi\ufb01ers of Chap-\nter 4 or Chapter 5 depend on the same words appearing in the training and test sets.\nBut by representing words as embeddings, a classi\ufb01er can assign sentiment as long\nas it sees some words with similar meanings . And as we\u2019ll see, vector semantic\nmodels can be learned automatically from text without supervision.\nIn this chapter we\u2019ll introduce the two most commonly used models. In the tf-idf\nmodel, an important baseline, the meaning of a word is de\ufb01ned by a simple function\nof the counts of nearby words. We will see that this method results in very long\nvectors that are sparse , i.e. mostly zeros (since most words simply never occur in\nthe context of others). We\u2019ll introduce the word2vec model family for construct-\ning short, dense vectors that have useful semantic properties. We\u2019ll also introduce\nthecosine , the standard way to use embeddings to compute semantic similarity , be-\ntween two words, two sentences, or two documents, an important tool in practical\napplications like question answering, summarization, or automatic essay grading.\n6.3 Words and Vectors\n\u201cThe most important attributes of a vector in 3-space are fLocation, Location, Location g\u201d\nRandall Munroe, https://xkcd.com/2358/\nVector or distributional models of meaning are generally based on a co-occurrence\nmatrix , a way of representing how often words co-occur. We\u2019ll look at two popular\nmatrices: the term-document matrix and the term-term matrix.\n6.3.1 Vectors and documents\nIn aterm-document matrix , each row represents a word in the vocabulary and eachterm-document\nmatrix\ncolumn represents a document from some collection of documents. Fig. 6.2 shows a\nsmall selection from a term-document matrix showing the occurrence of four words\nin four plays by Shakespeare. Each cell in this matrix represents the number of times 6.3 \u2022 W ORDS AND VECTORS 7\na particular word (de\ufb01ned by the row) occurs in a particular document (de\ufb01ned by\nthe column). Thus foolappeared 58 times in Twelfth Night .\nAs You Like It Twelfth Night Julius Caesar Henry V\nbattle 1 0 7 13\ngood 114 80 62 89\nfool 36 58 1 4\nwit 20 15 2 3\nFigure 6.2 The term-document matrix for four words in four Shakespeare plays. Each cell\ncontains the number of times the (row) word occurs in the (column) document.\nThe term-document matrix of Fig. 6.2 was \ufb01rst de\ufb01ned as part of the vector\nspace model of information retrieval (Salton, 1971). In this model, a document isvector space\nmodel\nrepresented as a count vector, a column in Fig. 6.3.\nTo review some basic linear algebra, a vector is, at heart, just a list or array of vector\nnumbers. So As You Like It is represented as the list [1,114,36,20] (the \ufb01rst column\nvector in Fig. 6.3) and Julius Caesar is represented as the list [7,62,1,2] (the third\ncolumn vector). A vector space is a collection of vectors, and is characterized by vector space\nitsdimension . Vectors in a 3-dimensional vector space have an element for each dimension\ndimension of the space. We will loosely refer to a vector in a 4-dimensional space\nas a 4-dimensional vector, with one element along each dimension. In the example\nin Fig. 6.3, we\u2019ve chosen to make the document vectors of dimension 4, just so they\n\ufb01t on the page; in real term-document matrices, the document vectors would have\ndimensionalityjVj, the vocabulary size.\nThe ordering of the numbers in a vector space indicates the different dimensions\non which documents vary. The \ufb01rst dimension for both these vectors corresponds to\nthe number of times the word battle occurs, and we can compare each dimension,\nnoting for example that the vectors for As You Like It andTwelfth Night have similar\nvalues (1 and 0, respectively) for the \ufb01rst dimension.\nAs You Like It Twelfth Night Julius Caesar Henry V\nbattle 1 0 7 13\ngood 114 80 62 89\nfool 36 58 1 4\nwit 20 15 2 3\nFigure 6.3 The term-document matrix for four words in four Shakespeare plays. The red\nboxes show that each document is represented as a column vector of length four.\nWe can think of the vector for a document as a point in jVj-dimensional space;\nthus the documents in Fig. 6.3 are points in 4-dimensional space. Since 4-dimensional\nspaces are hard to visualize, Fig. 6.4 shows a visualization in two dimensions; we\u2019ve\narbitrarily chosen the dimensions corresponding to the words battle andfool.\nTerm-document matrices were originally de\ufb01ned as a means of \ufb01nding similar\ndocuments for the task of document information retrieval . Two documents that are\nsimilar will tend to have similar words, and if two documents have similar words\ntheir column vectors will tend to be similar. The vectors for the comedies As You\nLike It [1,114,36,20] and Twelfth Night [0,80,58,15] look a lot more like each other\n(more fools and wit than battles) than they look like Julius Caesar [7,62,1,2] or\nHenry V [13,89,4,3]. This is clear with the raw numbers; in the \ufb01rst dimension\n(battle) the comedies have low numbers and the others have high numbers, and we\ncan see it visually in Fig. 6.4; we\u2019ll see very shortly how to quantify this intuition\nmore formally. 8CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\n51015202530510Henry V [4,13]As You Like It [36,1]Julius Caesar [1,7]battle foolTwelfth Night [58,0]1540\n354045505560\nFigure 6.4 A spatial visualization of the document vectors for the four Shakespeare play\ndocuments, showing just two of the dimensions, corresponding to the words battle andfool.\nThe comedies have high values for the fooldimension and low values for the battle dimension.\nA real term-document matrix, of course, wouldn\u2019t just have 4 rows and columns,\nlet alone 2. More generally, the term-document matrix has jVjrows (one for each\nword type in the vocabulary) and Dcolumns (one for each document in the collec-\ntion); as we\u2019ll see, vocabulary sizes are generally in the tens of thousands, and the\nnumber of documents can be enormous (think about all the pages on the web).\nInformation retrieval (IR) is the task of \ufb01nding the document dfrom the Dinformation\nretrieval\ndocuments in some collection that best matches a query q. For IR we\u2019ll therefore also\nrepresent a query by a vector, also of length jVj, and we\u2019ll need a way to compare\ntwo vectors to \ufb01nd how similar they are. (Doing IR will also require ef\ufb01cient ways\nto store and manipulate these vectors by making use of the convenient fact that these\nvectors are sparse, i.e., mostly zeros).\nLater in the chapter we\u2019ll introduce some of the components of this vector com-\nparison process: the tf-idf term weighting, and the cosine similarity metric.\n6.3.2 Words as vectors: document dimensions\nWe\u2019ve seen that documents can be represented as vectors in a vector space. But\nvector semantics can also be used to represent the meaning of words . We do this\nby associating each word with a word vector\u2014 a row vector rather than a column row vector\nvector, hence with different dimensions, as shown in Fig. 6.5. The four dimensions\nof the vector for fool, [36,58,1,4], correspond to the four Shakespeare plays. Word\ncounts in the same four dimensions are used to form the vectors for the other 3\nwords: wit, [20,15,2,3]; battle , [1,0,7,13]; and good [114,80,62,89].\nAs You Like It Twelfth Night Julius Caesar Henry V\nbattle 1 0 7 13\ngood 114 80 62 89\nfool 36 58 1 4\nwit 20 15 2 3\nFigure 6.5 The term-document matrix for four words in four Shakespeare plays. The red\nboxes show that each word is represented as a row vector of length four.\nFor documents, we saw that similar documents had similar vectors, because sim-\nilar documents tend to have similar words. This same principle applies to words:\nsimilar words have similar vectors because they tend to occur in similar documents.\nThe term-document matrix thus lets us represent the meaning of a word by the doc-\numents it tends to occur in. 6.3 \u2022 W ORDS AND VECTORS 9\n6.3.3 Words as vectors: word dimensions\nAn alternative to using the term-document matrix to represent words as vectors of\ndocument counts, is to use the term-term matrix , also called the word-word ma-\ntrixor the term-context matrix , in which the columns are labeled by words ratherword-word\nmatrix\nthan documents. This matrix is thus of dimensionality jVj\u0002jVjand each cell records\nthe number of times the row (target) word and the column (context) word co-occur\nin some context in some training corpus. The context could be the document, in\nwhich case the cell represents the number of times the two words appear in the same\ndocument. It is most common, however, to use smaller contexts, generally a win-\ndow around the word, for example of 4 words to the left and 4 words to the right,\nin which case the cell represents the number of times (in some training corpus) the\ncolumn word occurs in such a \u00064 word window around the row word. Here are four\nexamples of words in their windows:\nis traditionally followed by cherry pie, a traditional dessert\noften mixed, such as strawberry rhubarb pie. Apple pie\ncomputer peripherals and personal digital assistants. These devices usually\na computer. This includes information available on the internet\nIf we then take every occurrence of each word (say strawberry ) and count the\ncontext words around it, we get a word-word co-occurrence matrix. Fig. 6.6 shows a\nsimpli\ufb01ed subset of the word-word co-occurrence matrix for these four words com-\nputed from the Wikipedia corpus (Davies, 2015).\naardvark ... computer data result pie sugar ...\ncherry 0 ... 2 8 9 442 25 ...\nstrawberry 0 ... 0 0 1 60 19 ...\ndigital 0 ... 1670 1683 85 5 4 ...\ninformation 0 ... 3325 3982 378 5 13 ...\nFigure 6.6 Co-occurrence vectors for four words in the Wikipedia corpus, showing six of\nthe dimensions (hand-picked for pedagogical purposes). The vector for digital is outlined in\nred. Note that a real vector would have vastly more dimensions and thus be much sparser.\nNote in Fig. 6.6 that the two words cherry andstrawberry are more similar to\neach other (both pieandsugar tend to occur in their window) than they are to other\nwords like digital ; conversely, digital andinformation are more similar to each other\nthan, say, to strawberry . Fig. 6.7 shows a spatial visualization.\n100020003000400010002000digital [1683,1670]computer datainformation [3982,3325] 30004000\nFigure 6.7 A spatial visualization of word vectors for digital andinformation , showing just\ntwo of the dimensions, corresponding to the words data andcomputer .\nNote thatjVj, the dimensionality of the vector, is generally the size of the vo-\ncabulary, often between 10,000 and 50,000 words (using the most frequent words 10 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\nin the training corpus; keeping words after about the most frequent 50,000 or so is\ngenerally not helpful). Since most of these numbers are zero these are sparse vector\nrepresentations; there are ef\ufb01cient algorithms for storing and computing with sparse\nmatrices.\nNow that we have some intuitions, let\u2019s move on to examine the details of com-\nputing word similarity. Afterwards we\u2019ll discuss methods for weighting cells.\n6.4 Cosine for measuring similarity\nTo measure similarity between two target words vandw, we need a metric that\ntakes two vectors (of the same dimensionality, either both with words as dimensions,\nhence of lengthjVj, or both with documents as dimensions, of length jDj) and gives\na measure of their similarity. By far the most common similarity metric is the cosine\nof the angle between the vectors.\nThe cosine\u2014like most measures for vector similarity used in NLP\u2014is based on\nthedot product operator from linear algebra, also called the inner product : dot product\ninner product\ndot product (v;w) =v\u0001w=NX\ni=1viwi=v1w1+v2w2+:::+vNwN (6.7)\nThe dot product acts as a similarity metric because it will tend to be high just when\nthe two vectors have large values in the same dimensions. Alternatively, vectors that\nhave zeros in different dimensions\u2014orthogonal vectors\u2014will have a dot product of\n0, representing their strong dissimilarity.\nThis raw dot product, however, has a problem as a similarity metric: it favors\nlong vectors. The vector length is de\ufb01ned as vector length\njvj=vuutNX\ni=1v2\ni(6.8)\nThe dot product is higher if a vector is longer, with higher values in each dimension.\nMore frequent words have longer vectors, since they tend to co-occur with more\nwords and have higher co-occurrence values with each of them. The raw dot product\nthus will be higher for frequent words. But this is a problem; we\u2019d like a similarity\nmetric that tells us how similar two words are regardless of their frequency.\nWe modify the dot product to normalize for the vector length by dividing the\ndot product by the lengths of each of the two vectors. This normalized dot product\nturns out to be the same as the cosine of the angle between the two vectors, following\nfrom the de\ufb01nition of the dot product between two vectors aandb:\na\u0001b=jajjbjcosq\na\u0001b\njajjbj=cosq (6.9)\nThecosine similarity metric between two vectors vandwthus can be computed as: cosine 6.5 \u2022 TF-IDF: W EIGHING TERMS IN THE VECTOR 11\ncosine (v;w) =v\u0001w\njvjjwj=NX\ni=1viwi\nvuutNX\ni=1v2\nivuutNX\ni=1w2\ni(6.10)\nFor some applications we pre-normalize each vector, by dividing it by its length,\ncreating a unit vector of length 1. Thus we could compute a unit vector from aby unit vector\ndividing it byjaj. For unit vectors, the dot product is the same as the cosine.\nThe cosine value ranges from 1 for vectors pointing in the same direction, through\n0 for orthogonal vectors, to -1 for vectors pointing in opposite directions. But since\nraw frequency values are non-negative, the cosine for these vectors ranges from 0\u20131.\nLet\u2019s see how the cosine computes which of the words cherry ordigital is closer\nin meaning to information , just using raw counts from the following shortened table:\npie data computer\ncherry 442 8 2\ndigital 5 1683 1670\ninformation 5 3982 3325\ncos(cherry ;information ) =442\u00035+8\u00033982+2\u00033325p\n4422+82+22p\n52+39822+33252=:018\ncos(digital ;information ) =5\u00035+1683\u00033982+1670\u00033325p\n52+16832+16702p\n52+39822+33252=:996\nThe model decides that information is way closer to digital than it is to cherry , a\nresult that seems sensible. Fig. 6.8 shows a visualization.\n50010001500200025003000500digitalcherryinformationDimension 1: \u2018pie\u2019\nDimension 2: \u2018computer\u2019\nFigure 6.8 A (rough) graphical demonstration of cosine similarity, showing vectors for\nthree words ( cherry ,digital , and information ) in the two dimensional space de\ufb01ned by counts\nof the words computer andpienearby. The \ufb01gure doesn\u2019t show the cosine, but it highlights the\nangles; note that the angle between digital andinformation is smaller than the angle between\ncherry andinformation . When two vectors are more similar, the cosine is larger but the angle\nis smaller; the cosine has its maximum (1) when the angle between two vectors is smallest\n(0\u000e); the cosine of all other angles is less than 1.\n6.5 TF-IDF: Weighing terms in the vector\nThe co-occurrence matrices above represent each cell by frequencies, either of words\nwith documents (Fig. 6.5), or words with other words (Fig. 6.6). But raw frequency 12 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\nis not the best measure of association between words. Raw frequency is very skewed\nand not very discriminative. If we want to know what kinds of contexts are shared\nbycherry andstrawberry but not by digital andinformation , we\u2019re not going to get\ngood discrimination from words like the,it, orthey, which occur frequently with\nall sorts of words and aren\u2019t informative about any particular word. We saw this\nalso in Fig. 6.3 for the Shakespeare corpus; the dimension for the word good is not\nvery discriminative between plays; good is simply a frequent word and has roughly\nequivalent high frequencies in each of the plays.\nIt\u2019s a bit of a paradox. Words that occur nearby frequently (maybe pienearby\ncherry ) are more important than words that only appear once or twice. Yet words\nthat are too frequent\u2014ubiquitous, like theorgood \u2014 are unimportant. How can we\nbalance these two con\ufb02icting constraints?\nThere are two common solutions to this problem: in this section we\u2019ll describe\nthetf-idf weighting, usually used when the dimensions are documents. In the next\nsection we introduce the PPMI algorithm (usually used when the dimensions are\nwords).\nThetf-idf weighting (the \u2018-\u2019 here is a hyphen, not a minus sign) is the product\nof two terms, each term capturing one of these two intuitions:\nThe \ufb01rst is the term frequency (Luhn, 1957): the frequency of the word tin the term frequency\ndocument d. We can just use the raw count as the term frequency:\ntft;d=count (t;d) (6.11)\nMore commonly we squash the raw frequency a bit, by using the log 10of the fre-\nquency instead. The intuition is that a word appearing 100 times in a document\ndoesn\u2019t make that word 100 times more likely to be relevant to the meaning of the\ndocument. We also need to do something special with counts of 0, since we can\u2019t\ntake the log of 0.2\ntft;d=(\n1+log10count (t;d) if count (t;d)>0\n0 otherwise(6.12)\nIf we use log weighting, terms which occur 0 times in a document would have tf =0,\n1 times in a document tf =1+log10(1) =1+0=1, 10 times in a document tf =\n1+log10(10) =2, 100 times tf =1+log10(100) =3, 1000 times tf =4, and so on.\nThe second factor in tf-idf is used to give a higher weight to words that occur\nonly in a few documents. Terms that are limited to a few documents are useful\nfor discriminating those documents from the rest of the collection; terms that occur\nfrequently across the entire collection aren\u2019t as helpful. The document frequencydocument\nfrequency\ndftof a term tis the number of documents it occurs in. Document frequency is\nnot the same as the collection frequency of a term, which is the total number of\ntimes the word appears in the whole collection in any document. Consider in the\ncollection of Shakespeare\u2019s 37 plays the two words Romeo andaction . The words\nhave identical collection frequencies (they both occur 113 times in all the plays) but\nvery different document frequencies, since Romeo only occurs in a single play. If\nour goal is to \ufb01nd documents about the romantic tribulations of Romeo, the word\nRomeo should be highly weighted, but not action :\nCollection Frequency Document Frequency\nRomeo 113 1\naction 113 31\n2We can also use this alternative formulation, which we have used in earlier editions: tf t;d=\nlog10(count (t;d)+1) 6.5 \u2022 TF-IDF: W EIGHING TERMS IN THE VECTOR 13\nWe emphasize discriminative words like Romeo via the inverse document fre-\nquency oridfterm weight (Sparck Jones, 1972). The idf is de\ufb01ned using the frac- idf\ntionN=dft, where Nis the total number of documents in the collection, and df tis\nthe number of documents in which term toccurs. The fewer documents in which a\nterm occurs, the higher this weight. The lowest weight of 1 is assigned to terms that\noccur in all the documents. It\u2019s usually clear what counts as a document: in Shake-\nspeare we would use a play; when processing a collection of encyclopedia articles\nlike Wikipedia, the document is a Wikipedia page; in processing newspaper articles,\nthe document is a single article. Occasionally your corpus might not have appropri-\nate document divisions and you might need to break up the corpus into documents\nyourself for the purposes of computing idf.\nBecause of the large number of documents in many collections, this measure\ntoo is usually squashed with a log function. The resulting de\ufb01nition for inverse\ndocument frequency (idf) is thus\nidft=log10\u0012N\ndft\u0013\n(6.13)\nHere are some idf values for some words in the Shakespeare corpus, (along with\nthe document frequency df values on which they are based) ranging from extremely\ninformative words which occur in only one play like Romeo , to those that occur in a\nfew like salad orFalstaff , to those which are very common like foolor so common\nas to be completely non-discriminative since they occur in all 37 plays like good or\nsweet .3\nWord df idf\nRomeo 1 1.57\nsalad 2 1.27\nFalstaff 4 0.967\nforest 12 0.489\nbattle 21 0.246\nwit 34 0.037\nfool 36 0.012\ngood 37 0\nsweet 37 0\nThe tf-idf weighted value wt;dfor word tin document dthus combines term tf-idf\nfrequency tf t;d(de\ufb01ned either by Eq. 6.11 or by Eq. 6.12) with idf from Eq. 6.13:\nwt;d=tft;d\u0002idft (6.14)\nFig. 6.9 applies tf-idf weighting to the Shakespeare term-document matrix in Fig. 6.2,\nusing the tf equation Eq. 6.12. Note that the tf-idf values for the dimension corre-\nsponding to the word good have now all become 0; since this word appears in every\ndocument, the tf-idf weighting leads it to be ignored. Similarly, the word fool, which\nappears in 36 out of the 37 plays, has a much lower weight.\nThe tf-idf weighting is the way for weighting co-occurrence matrices in infor-\nmation retrieval, but also plays a role in many other aspects of natural language\nprocessing. It\u2019s also a great baseline, the simple thing to try \ufb01rst. We\u2019ll look at other\nweightings like PPMI (Positive Pointwise Mutual Information) in Section 6.6.\n3Sweet was one of Shakespeare\u2019s favorite adjectives, a fact probably related to the increased use of\nsugar in European recipes around the turn of the 16th century (Jurafsky, 2014, p. 175). 14 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\nAs You Like It Twelfth Night Julius Caesar Henry V\nbattle 0.246 0 0.454 0.520\ngood 0 0 0 0\nfool 0.030 0.033 0.0012 0.0019\nwit 0.085 0.081 0.048 0.054\nFigure 6.9 A portion of the tf-idf weighted term-document matrix for four words in Shake-\nspeare plays, showing a selection of 4 plays, using counts from Fig. 6.2. For example the\n0:085 value for witinAs You Like It is the product of tf =1+log10(20) =2:301 and idf =:037.\nNote that the idf weighting has eliminated the importance of the ubiquitous word good and\nvastly reduced the impact of the almost-ubiquitous word fool.\n6.6 Pointwise Mutual Information (PMI)\nAn alternative weighting function to tf-idf, PPMI (positive pointwise mutual infor-\nmation), is used for term-term-matrices, when the vector dimensions correspond to\nwords rather than documents. PPMI draws on the intuition that the best way to weigh\nthe association between two words is to ask how much more the two words co-occur\nin our corpus than we would have a priori expected them to appear by chance.\nPointwise mutual information (Fano, 1961)4is one of the most important con-pointwise\nmutual\ninformationcepts in NLP. It is a measure of how often two events xandyoccur, compared with\nwhat we would expect if they were independent:\nI(x;y) =log2P(x;y)\nP(x)P(y)(6.16)\nThe pointwise mutual information between a target word wand a context word\nc(Church and Hanks 1989, Church and Hanks 1990) is then de\ufb01ned as:\nPMI(w;c) =log2P(w;c)\nP(w)P(c)(6.17)\nThe numerator tells us how often we observed the two words together (assuming\nwe compute probability by using the MLE). The denominator tells us how often\nwe would expect the two words to co-occur assuming they each occurred indepen-\ndently; recall that the probability of two independent events both occurring is just\nthe product of the probabilities of the two events. Thus, the ratio gives us an esti-\nmate of how much more the two words co-occur than we expect by chance. PMI is\na useful tool whenever we need to \ufb01nd words that are strongly associated.\nPMI values range from negative to positive in\ufb01nity. But negative PMI values\n(which imply things are co-occurring less often than we would expect by chance)\ntend to be unreliable unless our corpora are enormous. To distinguish whether\ntwo words whose individual probability is each 10\u00006occur together less often than\nchance, we would need to be certain that the probability of the two occurring to-\ngether is signi\ufb01cantly less than 10\u000012, and this kind of granularity would require an\nenormous corpus. Furthermore it\u2019s not clear whether it\u2019s even possible to evaluate\nsuch scores of \u2018unrelatedness\u2019 with human judgments. For this reason it is more\n4PMI is based on the mutual information between two random variables XandY, de\ufb01ned as:\nI(X;Y) =X\nxX\nyP(x;y)log2P(x;y)\nP(x)P(y)(6.15)\nIn a confusion of terminology, Fano used the phrase mutual information to refer to what we now call\npointwise mutual information and the phrase expectation of the mutual information for what we now call\nmutual information 6.6 \u2022 P OINTWISE MUTUAL INFORMATION (PMI) 15\ncommon to use Positive PMI (called PPMI ) which replaces all negative PMI values PPMI\nwith zero (Church and Hanks 1989, Dagan et al. 1993, Niwa and Nitta 1994)5:\nPPMI (w;c) =max(log2P(w;c)\nP(w)P(c);0) (6.18)\nMore formally, let\u2019s assume we have a co-occurrence matrix F with W rows (words)\nand C columns (contexts), where fi jgives the number of times word wioccurs with\ncontext cj. This can be turned into a PPMI matrix where PPMI i jgives the PPMI\nvalue of word wiwith context cj(which we can also express as PPMI( wi;cj) or\nPPMI( w=i;c=j)) as follows:\npi j=fi jPW\ni=1PC\nj=1fi j;pi\u0003=PC\nj=1fi jPW\ni=1PC\nj=1fi j;p\u0003j=PW\ni=1fi jPW\ni=1PC\nj=1fi j(6.19)\nPPMI i j=max(log2pi j\npi\u0003p\u0003j;0) (6.20)\nLet\u2019s see some PPMI calculations. We\u2019ll use Fig. 6.10, which repeats Fig. 6.6 plus\nall the count marginals, and let\u2019s pretend for ease of calculation that these are the\nonly words/contexts that matter.\ncomputer data result pie sugar count(w)\ncherry 2 8 9 442 25 486\nstrawberry 0 0 1 60 19 80\ndigital 1670 1683 85 5 4 3447\ninformation 3325 3982 378 5 13 7703\ncount(context) 4997 5673 473 512 61 11716\nFigure 6.10 Co-occurrence counts for four words in 5 contexts in the Wikipedia corpus,\ntogether with the marginals, pretending for the purpose of this calculation that no other\nwords/contexts matter.\nThus for example we could compute PPMI(information,data), assuming we pre-\ntended that Fig. 6.6 encompassed all the relevant word contexts/dimensions, as fol-\nlows:\nP(w=information, c=data ) =3982\n11716=:3399\nP(w=information ) =7703\n11716=:6575\nP(c=data ) =5673\n11716=:4842\nPPMI (information,data ) = log2(:3399 =(:6575\u0003:4842)) = :0944\nFig. 6.11 shows the joint probabilities computed from the counts in Fig. 6.10, and\nFig. 6.12 shows the PPMI values. Not surprisingly, cherry andstrawberry are highly\nassociated with both pieandsugar , and data is mildly associated with information .\nPMI has the problem of being biased toward infrequent events; very rare words\ntend to have very high PMI values. One way to reduce this bias toward low frequency\n5Positive PMI also cleanly solves the problem of what to do with zero counts, using 0 to replace the\n\u0000\u00a5from log (0). 16 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\np(w,context) p(w)\ncomputer data result pie sugar p(w)\ncherry 0.0002 0.0007 0.0008 0.0377 0.0021 0.0415\nstrawberry 0.0000 0.0000 0.0001 0.0051 0.0016 0.0068\ndigital 0.1425 0.1436 0.0073 0.0004 0.0003 0.2942\ninformation 0.2838 0.3399 0.0323 0.0004 0.0011 0.6575\np(context) 0.4265 0.4842 0.0404 0.0437 0.0052\nFigure 6.11 Replacing the counts in Fig. 6.6 with joint probabilities, showing the marginals\nin the right column and the bottom row.\ncomputer data result pie sugar\ncherry 0 0 0 4.38 3.30\nstrawberry 0 0 0 4.10 5.51\ndigital 0.18 0.01 0 0 0\ninformation 0.02 0.09 0.28 0 0\nFigure 6.12 The PPMI matrix showing the association between words and context words,\ncomputed from the counts in Fig. 6.11. Note that most of the 0 PPMI values are ones that had\na negative PMI; for example PMI( cherry,computer ) = -6.7, meaning that cherry andcomputer\nco-occur on Wikipedia less often than we would expect by chance, and with PPMI we replace\nnegative values by zero.\nevents is to slightly change the computation for P(c), using a different function Pa(c)\nthat raises the probability of the context word to the power of a:\nPPMI a(w;c) =max(log2P(w;c)\nP(w)Pa(c);0) (6.21)\nPa(c) =count (c)a\nP\nccount (c)a(6.22)\nLevy et al. (2015) found that a setting of a=0:75 improved performance of\nembeddings on a wide range of tasks (drawing on a similar weighting used for skip-\ngrams described below in Eq. 6.32). This works because raising the count to a=\n0:75 increases the probability assigned to rare contexts, and hence lowers their PMI\n(Pa(c)>P(c)when cis rare).\nAnother possible solution is Laplace smoothing: Before computing PMI, a small\nconstant k(values of 0.1-3 are common) is added to each of the counts, shrinking\n(discounting) all the non-zero values. The larger the k, the more the non-zero counts\nare discounted.\n6.7 Applications of the tf-idf or PPMI vector models\nIn summary, the vector semantics model we\u2019ve described so far represents a target\nword as a vector with dimensions corresponding either to the documents in a large\ncollection (the term-document matrix) or to the counts of words in some neighboring\nwindow (the term-term matrix). The values in each dimension are counts, weighted\nby tf-idf (for term-document matrices) or PPMI (for term-term matrices), and the\nvectors are sparse (since most values are zero).\nThe model computes the similarity between two words xandyby taking the\ncosine of their tf-idf or PPMI vectors; high cosine, high similarity. This entire model 6.8 \u2022 W ORD2VEC 17\nis sometimes referred to as the tf-idf model or the PPMI model, after the weighting\nfunction.\nThe tf-idf model of meaning is often used for document functions like deciding\nif two documents are similar. We represent a document by taking the vectors of\nall the words in the document, and computing the centroid of all those vectors. centroid\nThe centroid is the multidimensional version of the mean; the centroid of a set of\nvectors is a single vector that has the minimum sum of squared distances to each of\nthe vectors in the set. Given kword vectors w1;w2;:::;wk, the centroid document\nvector dis:document\nvector\nd=w1+w2+:::+wk\nk(6.23)\nGiven two documents, we can then compute their document vectors d1andd2, and\nestimate the similarity between the two documents by cos (d1;d2). Document sim-\nilarity is also useful for all sorts of applications; information retrieval, plagiarism\ndetection, news recommender systems, and even for digital humanities tasks like\ncomparing different versions of a text to see which are similar to each other.\nEither the PPMI model or the tf-idf model can be used to compute word simi-\nlarity, for tasks like \ufb01nding word paraphrases, tracking changes in word meaning, or\nautomatically discovering meanings of words in different corpora. For example, we\ncan \ufb01nd the 10 most similar words to any target word wby computing the cosines\nbetween wand each of the V\u00001 other words, sorting, and looking at the top 10.\n6.8 Word2vec\nIn the previous sections we saw how to represent a word as a sparse, long vector with\ndimensions corresponding to words in the vocabulary or documents in a collection.\nWe now introduce a more powerful word representation: embeddings , short dense\nvectors. Unlike the vectors we\u2019ve seen so far, embeddings are short , with number\nof dimensions dranging from 50-1000, rather than the much larger vocabulary size\njVjor number of documents Dwe\u2019ve seen. These ddimensions don\u2019t have a clear\ninterpretation. And the vectors are dense : instead of vector entries being sparse,\nmostly-zero counts or functions of counts, the values will be real-valued numbers\nthat can be negative.\nIt turns out that dense vectors work better in every NLP task than sparse vectors.\nWhile we don\u2019t completely understand all the reasons for this, we have some intu-\nitions. Representing words as 300-dimensional dense vectors requires our classi\ufb01ers\nto learn far fewer weights than if we represented words as 50,000-dimensional vec-\ntors, and the smaller parameter space possibly helps with generalization and avoid-\ning over\ufb01tting. Dense vectors may also do a better job of capturing synonymy.\nFor example, in a sparse vector representation, dimensions for synonyms like car\nandautomobile dimension are distinct and unrelated; sparse vectors may thus fail\nto capture the similarity between a word with caras a neighbor and a word with\nautomobile as a neighbor.\nIn this section we introduce one method for computing embeddings: skip-gram skip-gram\nwith negative sampling , sometimes called SGNS . The skip-gram algorithm is one SGNS\nof two algorithms in a software package called word2vec , and so sometimes the word2vec\nalgorithm is loosely referred to as word2vec (Mikolov et al. 2013a, Mikolov et al.\n2013b). The word2vec methods are fast, ef\ufb01cient to train, and easily available on- 18 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\nline with code and pretrained embeddings. Word2vec embeddings are static em-\nbeddings , meaning that the method learns one \ufb01xed embedding for each word in thestatic\nembeddings\nvocabulary. In Chapter 11 we\u2019ll introduce methods for learning dynamic contextual\nembeddings like the popular family of BERT representations, in which the vector\nfor each word is different in different contexts.\nThe intuition of word2vec is that instead of counting how often each word woc-\ncurs near, say, apricot , we\u2019ll instead train a classi\ufb01er on a binary prediction task: \u201cIs\nword wlikely to show up near apricot ?\u201d We don\u2019t actually care about this prediction\ntask; instead we\u2019ll take the learned classi\ufb01er weights as the word embeddings.\nThe revolutionary intuition here is that we can just use running text as implicitly\nsupervised training data for such a classi\ufb01er; a word cthat occurs near the target\nword apricot acts as gold \u2018correct answer\u2019 to the question \u201cIs word clikely to show\nup near apricot ?\u201d This method, often called self-supervision , avoids the need for self-supervision\nany sort of hand-labeled supervision signal. This idea was \ufb01rst proposed in the task\nof neural language modeling, when Bengio et al. (2003) and Collobert et al. (2011)\nshowed that a neural language model (a neural network that learned to predict the\nnext word from prior words) could just use the next word in running text as its\nsupervision signal, and could be used to learn an embedding representation for each\nword as part of doing this prediction task.\nWe\u2019ll see how to do neural networks in the next chapter, but word2vec is a\nmuch simpler model than the neural network language model, in two ways. First,\nword2vec simpli\ufb01es the task (making it binary classi\ufb01cation instead of word pre-\ndiction). Second, word2vec simpli\ufb01es the architecture (training a logistic regression\nclassi\ufb01er instead of a multi-layer neural network with hidden layers that demand\nmore sophisticated training algorithms). The intuition of skip-gram is:\n1. Treat the target word and a neighboring context word as positive examples.\n2. Randomly sample other words in the lexicon to get negative samples.\n3. Use logistic regression to train a classi\ufb01er to distinguish those two cases.\n4. Use the learned weights as the embeddings.\n6.8.1 The classi\ufb01er\nLet\u2019s start by thinking about the classi\ufb01cation task, and then turn to how to train.\nImagine a sentence like the following, with a target word apricot , and assume we\u2019re\nusing a window of \u00062 context words:\n... lemon, a [tablespoon of apricot jam, a] pinch ...\nc1 c2 w c3 c4\nOur goal is to train a classi\ufb01er such that, given a tuple (w;c)of a target word\nwpaired with a candidate context word c(for example ( apricot ,jam), or perhaps\n(apricot ,aardvark )) it will return the probability that cis a real context word (true\nforjam, false for aardvark ):\nP(+jw;c) (6.24)\nThe probability that word cis not a real context word for wis just 1 minus\nEq. 6.24:\nP(\u0000jw;c) =1\u0000P(+jw;c) (6.25)\nHow does the classi\ufb01er compute the probability P? The intuition of the skip-\ngram model is to base this probability on embedding similarity: a word is likely to 6.8 \u2022 W ORD2VEC 19\noccur near the target if its embedding vector is similar to the target embedding. To\ncompute similarity between these dense embeddings, we rely on the intuition that\ntwo vectors are similar if they have a high dot product (after all, cosine is just a\nnormalized dot product). In other words:\nSimilarity (w;c)\u0019c\u0001w (6.26)\nThe dot product c\u0001wis not a probability, it\u2019s just a number ranging from \u0000\u00a5to\u00a5\n(since the elements in word2vec embeddings can be negative, the dot product can be\nnegative). To turn the dot product into a probability, we\u2019ll use the logistic orsigmoid\nfunction s(x), the fundamental core of logistic regression:\ns(x) =1\n1+exp(\u0000x)(6.27)\nWe model the probability that word cis a real context word for target word was:\nP(+jw;c) = s(c\u0001w) =1\n1+exp(\u0000c\u0001w)(6.28)\nThe sigmoid function returns a number between 0 and 1, but to make it a probability\nwe\u2019ll also need the total probability of the two possible events ( cis a context word,\nandcisn\u2019t a context word) to sum to 1. We thus estimate the probability that word c\nis not a real context word for was:\nP(\u0000jw;c) = 1\u0000P(+jw;c)\n=s(\u0000c\u0001w) =1\n1+exp(c\u0001w)(6.29)\nEquation 6.28 gives us the probability for one word, but there are many context\nwords in the window. Skip-gram makes the simplifying assumption that all context\nwords are independent, allowing us to just multiply their probabilities:\nP(+jw;c1:L) =LY\ni=1s(ci\u0001w) (6.30)\nlogP(+jw;c1:L) =LX\ni=1logs(ci\u0001w) (6.31)\nIn summary, skip-gram trains a probabilistic classi\ufb01er that, given a test target word\nwand its context window of Lwords c1:L, assigns a probability based on how similar\nthis context window is to the target word. The probability is based on applying the\nlogistic (sigmoid) function to the dot product of the embeddings of the target word\nwith each context word. To compute this probability, we just need embeddings for\neach target word and context word in the vocabulary.\nFig. 6.13 shows the intuition of the parameters we\u2019ll need. Skip-gram actually\nstores two embeddings for each word, one for the word as a target, and one for the\nword considered as context. Thus the parameters we need to learn are two matrices\nWandC, each containing an embedding for every one of the jVjwords in the\nvocabulary V.6Let\u2019s now turn to learning these embeddings (which is the real goal\nof training this classi\ufb01er in the \ufb01rst place).\n6In principle the target matrix and the context matrix could use different vocabularies, but we\u2019ll simplify\nby assuming one shared vocabulary V. 20 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\n1WCaardvark\nzebrazebraaardvarkapricotapricot|V||V|+12V& =target wordscontext & noisewords\u2026\n\u20261..d\u2026\n\u2026\nFigure 6.13 The embeddings learned by the skipgram model. The algorithm stores two\nembeddings for each word, the target embedding (sometimes called the input embedding)\nand the context embedding (sometimes called the output embedding). The parameter qthat\nthe algorithm learns is thus a matrix of 2 jVjvectors, each of dimension d, formed by concate-\nnating two matrices, the target embeddings Wand the context+noise embeddings C.\n6.8.2 Learning skip-gram embeddings\nThe learning algorithm for skip-gram embeddings takes as input a corpus of text,\nand a chosen vocabulary size N. It begins by assigning a random embedding vector\nfor each of the N vocabulary words, and then proceeds to iteratively shift the em-\nbedding of each word wto be more like the embeddings of words that occur nearby\nin texts, and less like the embeddings of words that don\u2019t occur nearby. Let\u2019s start\nby considering a single piece of training data:\n... lemon, a [tablespoon of apricot jam, a] pinch ...\nc1 c2 w c3 c4\nThis example has a target word w(apricot), and 4 context words in the L=\u00062\nwindow, resulting in 4 positive training instances (on the left below):\npositive examples +\nw c pos\napricot tablespoon\napricot of\napricot jam\napricot anegative examples -\nw c neg w c neg\napricot aardvark apricot seven\napricot my apricot forever\napricot where apricot dear\napricot coaxial apricot if\nFor training a binary classi\ufb01er we also need negative examples. In fact skip-\ngram with negative sampling (SGNS) uses more negative examples than positive\nexamples (with the ratio between them set by a parameter k). So for each of these\n(w;cpos)training instances we\u2019ll create knegative samples, each consisting of the\ntarget wplus a \u2018noise word\u2019 cneg. A noise word is a random word from the lexicon,\nconstrained not to be the target word w. The right above shows the setting where\nk=2, so we\u2019ll have 2 negative examples in the negative training set \u0000for each\npositive example w;cpos.\nThe noise words are chosen according to their weighted unigram frequency\npa(w), where ais a weight. If we were sampling according to unweighted fre-\nquency p(w), it would mean that with unigram probability p(\u201cthe\u201d)we would choose\nthe word theas a noise word, with unigram probability p(\u201caardvark \u201d)we would\nchoose aardvark , and so on. But in practice it is common to set a=0:75, i.e. use 6.8 \u2022 W ORD2VEC 21\nthe weighting p3\n4(w):\nPa(w) =count (w)a\nP\nw0count (w0)a(6.32)\nSetting a=:75 gives better performance because it gives rare noise words slightly\nhigher probability: for rare words, Pa(w)>P(w). To illustrate this intuition, it\nmight help to work out the probabilities for an example with a=:75 and two events,\nP(a) =0:99 and P(b) =0:01:\nPa(a) =:99:75\n:99:75+:01:75=0:97\nPa(b) =:01:75\n:99:75+:01:75=0:03 (6.33)\nThus using a=:75 increases the probability of the rare event bfrom 0.01 to 0.03.\nGiven the set of positive and negative training instances, and an initial set of\nembeddings, the goal of the learning algorithm is to adjust those embeddings to\n\u2022 Maximize the similarity of the target word, context word pairs (w;cpos)drawn\nfrom the positive examples\n\u2022 Minimize the similarity of the (w;cneg)pairs from the negative examples.\nIf we consider one word/context pair (w;cpos)with its knoise words cneg1:::cnegk,\nwe can express these two goals as the following loss function Lto be minimized\n(hence the\u0000); here the \ufb01rst term expresses that we want the classi\ufb01er to assign the\nreal context word cposa high probability of being a neighbor, and the second term\nexpresses that we want to assign each of the noise words cnegia high probability of\nbeing a non-neighbor, all multiplied because we assume independence:\nL=\u0000log\"\nP(+jw;cpos)kY\ni=1P(\u0000jw;cnegi)#\n=\u0000\"\nlogP(+jw;cpos)+kX\ni=1logP(\u0000jw;cnegi)#\n=\u0000\"\nlogP(+jw;cpos)+kX\ni=1log\u0000\n1\u0000P(+jw;cnegi)\u0001#\n=\u0000\"\nlogs(cpos\u0001w)+kX\ni=1logs(\u0000cnegi\u0001w)#\n(6.34)\nThat is, we want to maximize the dot product of the word with the actual context\nwords, and minimize the dot products of the word with the knegative sampled non-\nneighbor words.\nWe minimize this loss function using stochastic gradient descent. Fig. 6.14\nshows the intuition of one step of learning.\nTo get the gradient, we need to take the derivative of Eq. 6.34 with respect to\nthe different embeddings. It turns out the derivatives are the following (we leave the 22 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\nWCmove apricot and jam closer,increasing cpos z waardvark\nmove apricot and matrix apartdecreasing cneg1 z w\u201c\u2026apricot jam\u2026\u201dw\nzebrazebraaardvarkjamapricot\ncposmatrixTolstoymove apricot and Tolstoy apartdecreasing cneg2 z w!cneg1cneg2k=2\nFigure 6.14 Intuition of one step of gradient descent. The skip-gram model tries to shift\nembeddings so the target embeddings (here for apricot ) are closer to (have a higher dot prod-\nuct with) context embeddings for nearby words (here jam) and further from (lower dot product\nwith) context embeddings for noise words that don\u2019t occur nearby (here Tolstoy andmatrix ).\nproof as an exercise at the end of the chapter):\n\u00b6L\n\u00b6cpos= [s(cpos\u0001w)\u00001]w (6.35)\n\u00b6L\n\u00b6cneg= [s(cneg\u0001w)]w (6.36)\n\u00b6L\n\u00b6w= [s(cpos\u0001w)\u00001]cpos+kX\ni=1[s(cnegi\u0001w)]cnegi(6.37)\nThe update equations going from time step ttot+1 in stochastic gradient descent\nare thus:\nct+1\npos=ct\npos\u0000h[s(ct\npos\u0001wt)\u00001]wt(6.38)\nct+1\nneg=ct\nneg\u0000h[s(ct\nneg\u0001wt)]wt(6.39)\nwt+1=wt\u0000h\"\n[s(ct\npos\u0001wt)\u00001]ct\npos+kX\ni=1[s(ct\nnegi\u0001wt)]ct\nnegi#\n(6.40)\nJust as in logistic regression, then, the learning algorithm starts with randomly ini-\ntialized WandCmatrices, and then walks through the training corpus using gradient\ndescent to move WandCso as to minimize the loss in Eq. 6.34 by making the up-\ndates in (Eq. 6.38)-(Eq. 6.40).\nRecall that the skip-gram model learns twoseparate embeddings for each word i:\nthetarget embedding wiand the context embedding ci, stored in two matrices, thetarget\nembeddingcontext\nembedding target matrix Wand the context matrix C. It\u2019s common to just add them together,\nrepresenting word iwith the vector wi+ci. Alternatively we can throw away the C\nmatrix and just represent each word iby the vector wi.\nAs with the simple count-based methods like tf-idf, the context window size L\naffects the performance of skip-gram embeddings, and experiments often tune the\nparameter Lon a devset. 6.9 \u2022 V ISUALIZING EMBEDDINGS 23\n6.8.3 Other kinds of static embeddings\nThere are many kinds of static embeddings. An extension of word2vec, fasttext fasttext\n(Bojanowski et al., 2017), addresses a problem with word2vec as we have presented\nit so far: it has no good way to deal with unknown words \u2014words that appear in\na test corpus but were unseen in the training corpus. A related problem is word\nsparsity, such as in languages with rich morphology, where some of the many forms\nfor each noun and verb may only occur rarely. Fasttext deals with these problems\nby using subword models, representing each word as itself plus a bag of constituent\nn-grams, with special boundary symbols <and>added to each word. For example,\nwith n=3 the word where would be represented by the sequence <where> plus the\ncharacter n-grams:\n<wh, whe, her, ere, re>\nThen a skipgram embedding is learned for each constituent n-gram, and the word\nwhere is represented by the sum of all of the embeddings of its constituent n-grams.\nUnknown words can then be presented only by the sum of the constituent n-grams.\nA fasttext open-source library, including pretrained embeddings for 157 languages,\nis available at https://fasttext.cc .\nAnother very widely used static embedding model is GloVe (Pennington et al.,\n2014), short for Global Vectors, because the model is based on capturing global\ncorpus statistics. GloVe is based on ratios of probabilities from the word-word co-\noccurrence matrix, combining the intuitions of count-based models like PPMI while\nalso capturing the linear structures used by methods like word2vec.\nIt turns out that dense embeddings like word2vec actually have an elegant math-\nematical relationship with sparse embeddings like PPMI, in which word2vec can\nbe seen as implicitly optimizing a function of a PPMI matrix (Levy and Goldberg,\n2014c).\n6.9 Visualizing Embeddings\n\u201cI see well in many dimensions as long as the dimensions are around two.\u201d\nThe late economist Martin Shubik\nVisualizing embeddings is an important goal in helping understand, apply, and\nimprove these models of word meaning. But how can we visualize a (for example)\n100-dimensional vector?\nRohde, Gonnerman, Plaut Modeling Word Meaning Using Lexical Co-Occurrence\nHEADHANDFACE\nDOGAMERICA\nCATEYEEUROPE\nFOOTCHINAFRANCE\nCHICAGOARM\nFINGER\nNOSELEGRUSSIA\nMOUSEAFRICA\nATLANTAEARSHOULDERASIA\nCOW\nBULLPUPPYLIONHAWAII\nMONTREALTOKYOTOEMOSCOW\nTOOTH\nNASHVILLEBRAZILWRIST\nKITTENANKLE\nTURTLE\nOYSTER\nFigure 8: Multidimensional scaling for three noun classes.WRIST\nANKLE\nSHOULDER\nARM\nLEG\nHAND\nFOOT\nHEAD\nNOSE\nFINGER\nTOE\nFACE\nEAR\nEYE\nTOOTH\nDOG\nCAT\nPUPPY\nKITTEN\nCOW\nMOUSE\nTURTLE\nOYSTER\nLION\nBULL\nCHICAGO\nATLANTA\nMONTREAL\nNASHVILLE\nTOKYOCHINA\nRUSSIA\nAFRICA\nASIA\nEUROPE\nAMERICA\nBRAZIL\nMOSCOW\nFRANCEHAWAIIFigure 9: Hierarchical clustering for three noun classes using distances based on vector correlations.\n20\nThe simplest way to visualize the meaning of a word\nwembedded in a space is to list the most similar words to\nwby sorting the vectors for all words in the vocabulary by\ntheir cosine with the vector for w. For example the 7 closest\nwords to frogusing a particular embeddings computed with\nthe GloVe algorithm are: frogs ,toad,litoria ,leptodactyli-\ndae,rana,lizard , and eleutherodactylus (Pennington et al.,\n2014).\nYet another visualization method is to use a clustering\nalgorithm to show a hierarchical representation of which\nwords are similar to others in the embedding space. The\nuncaptioned \ufb01gure on the left uses hierarchical clustering\nof some embedding vectors for nouns as a visualization 24 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\nmethod (Rohde et al., 2006).\nProbably the most common visualization method, how-\never, is to project the 100 dimensions of a word down into 2\ndimensions. Fig. 6.1 showed one such visualization, as does\nFig. 6.16, using a projection method called t-SNE (van der\nMaaten and Hinton, 2008).\n6.10 Semantic properties of embeddings\nIn this section we brie\ufb02y summarize some of the semantic properties of embeddings\nthat have been studied.\nDifferent types of similarity or association: One parameter of vector semantic\nmodels that is relevant to both sparse tf-idf vectors and dense word2vec vectors is\nthe size of the context window used to collect counts. This is generally between 1\nand 10 words on each side of the target word (for a total context of 2-20 words).\nThe choice depends on the goals of the representation. Shorter context windows\ntend to lead to representations that are a bit more syntactic, since the information is\ncoming from immediately nearby words. When the vectors are computed from short\ncontext windows, the most similar words to a target word wtend to be semantically\nsimilar words with the same parts of speech. When vectors are computed from long\ncontext windows, the highest cosine words to a target word wtend to be words that\nare topically related but not similar.\nFor example Levy and Goldberg (2014a) showed that using skip-gram with a\nwindow of\u00062, the most similar words to the word Hogwarts (from the Harry Potter\nseries) were names of other \ufb01ctional schools: Sunnydale (from Buffy the Vampire\nSlayer ) orEvernight (from a vampire series). With a window of \u00065, the most similar\nwords to Hogwarts were other words topically related to the Harry Potter series:\nDumbledore ,Malfoy , and half-blood .\nIt\u2019s also often useful to distinguish two kinds of similarity or association between\nwords (Sch \u00a8utze and Pedersen, 1993). Two words have \ufb01rst-order co-occurrence\ufb01rst-order\nco-occurrence\n(sometimes called syntagmatic association ) if they are typically nearby each other.\nThus wrote is a \ufb01rst-order associate of book orpoem . Two words have second-order\nco-occurrence (sometimes called paradigmatic association ) if they have similarsecond-order\nco-occurrence\nneighbors. Thus wrote is a second-order associate of words like said orremarked .\nAnalogy/Relational Similarity: Another semantic property of embeddings is their\nability to capture relational meanings. In an important early vector space model of\ncognition, Rumelhart and Abrahamson (1973) proposed the parallelogram modelparallelogram\nmodel\nfor solving simple analogy problems of the form a is to b as a* is to what? . In\nsuch problems, a system is given a problem like apple:tree::grape:? , i.e., apple is\nto tree as grape is to , and must \ufb01ll in the word vine. In the parallelogram\nmodel, illustrated in Fig. 6.15, the vector from the word apple to the word tree(=#   \u0014tree\u0000#       \u0014apple) is added to the vector for grape (#        \u0014grape); the nearest word to that point\nis returned.\nIn early work with sparse embeddings, scholars showed that sparse vector mod-\nels of meaning could solve such analogy problems (Turney and Littman, 2005),\nbut the parallelogram method received more modern attention because of its suc-\ncess with word2vec or GloVe vectors (Mikolov et al. 2013c, Levy and Goldberg\n2014b, Pennington et al. 2014). For example, the result of the expression#     \u0014king\u0000 6.10 \u2022 S EMANTIC PROPERTIES OF EMBEDDINGS 25\ntreeapplegrapevine\nFigure 6.15 The parallelogram model for analogy problems (Rumelhart and Abrahamson,\n1973): the location of#     \u0014vine can be found by subtracting#       \u0014apple from#   \u0014tree and adding#       \u0014grape.\n#     \u0014man+#            \u0014woman is a vector close to#         \u0014queen. Similarly,#      \u0014Paris\u0000#           \u0014France +#     \u0014Italy results\nin a vector that is close to#         \u0014Rome. The embedding model thus seems to be extract-\ning representations of relations like MALE -FEMALE , or CAPITAL -CITY -OF, or even\nCOMPARATIVE /SUPERLATIVE , as shown in Fig. 6.16 from GloVe.\n(a) (b)\nFigure 6.16 Relational properties of the GloVe vector space, shown by projecting vectors onto two dimen-\nsions. (a)#     \u0014king\u0000#     \u0014man+#            \u0014woman is close to#        \u0014queen. (b) offsets seem to capture comparative and superlative\nmorphology (Pennington et al., 2014).\nFor a a:b::a\u0003:b\u0003problem, meaning the algorithm is given vectors a,b, and\na\u0003and must \ufb01nd b\u0003, the parallelogram method is thus:\n\u02c6b\u0003=argmin\nxdistance (x;b\u0000a+a\u0003) (6.41)\nwith some distance function, such as Euclidean distance.\nThere are some caveats. For example, the closest value returned by the paral-\nlelogram algorithm in word2vec or GloVe embedding spaces is usually not in fact\nb* but one of the 3 input words or their morphological variants (i.e., cherry:red ::\npotato:x returns potato orpotatoes instead of brown ), so these must be explicitly\nexcluded. Furthermore while embedding spaces perform well if the task involves\nfrequent words, small distances, and certain relations (like relating countries with\ntheir capitals or verbs/nouns with their in\ufb02ected forms), the parallelogram method\nwith embeddings doesn\u2019t work as well for other relations (Linzen 2016, Gladkova\net al. 2016, Schluter 2018, Ethayarajh et al. 2019a), and indeed Peterson et al. (2020)\nargue that the parallelogram method is in general too simple to model the human\ncognitive process of forming analogies of this kind. 26 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\n6.10.1 Embeddings and Historical Semantics\nEmbeddings can also be a useful tool for studying how meaning changes over time,\nby computing multiple embedding spaces, each from texts written in a particular\ntime period. For example Fig. 6.17 shows a visualization of changes in meaning in\nEnglish words over the last two centuries, computed by building separate embedding\nspaces for each decade from historical corpora like Google n-grams (Lin et al., 2012)\nand the Corpus of Historical American English (Davies, 2012).\nCHAPTER 5. DYNAMIC SOCIAL REPRESENTATIONS OF WORD MEANING 79\nFigure 5.1: Two-dimensional visualization of semantic change in English using SGNS\nvectors (see Section 5.8 for the visualization algorithm). A,T h ew o r d gay shifted\nfrom meaning \u201ccheerful\u201d or \u201cfrolicsome\u201d to referring to homosexuality. A,I nt h ee a r l y\n20th century broadcast referred to \u201ccasting out seeds\u201d; with the rise of television and\nradio its meaning shifted to \u201ctransmitting signals\u201d. C,Awful underwent a process of\npejoration, as it shifted from meaning \u201cfull of awe\u201d to meaning \u201cterrible or appalling\u201d\n[212].\nthat adverbials (e.g., actually )h a v eag e n e r a lt e n d e n c yt ou n d e r g os u b j e c t i \ufb01 c a t i o n\nwhere they shift from objective statements about the world (e.g., \u201cSorry, the car is\nactually broken\u201d) to subjective statements (e.g., \u201cI can\u2019t believe he actually did that\u201d,\nindicating surprise/disbelief).\n5.2.2 Computational linguistic studies\nThere are also a number of recent works analyzing semantic change using computational\nmethods. [ 200] use latent semantic analysis to analyze how word meanings broaden\nand narrow over time. [ 113]u s er a wc o - o c c u r r e n c ev e c t o r st op e r f o r man u m b e ro f\nhistorical case-studies on semantic change, and [ 252] perform a similar set of small-\nscale case-studies using temporal topic models. [ 87]c o n s t r u c tp o i n t - w i s em u t u a l\ninformation-based embeddings and found that semantic changes uncovered by their\nmethod had reasonable agreement with human judgments. [ 129]a n d[ 119]u s e\u201c n e u r a l \u201d\nword-embedding methods to detect linguistic change points. Finally, [ 257]a n a l y z e\nhistorical co-occurrences to test whether synonyms tend to change in similar ways.\nFigure 6.17 A t-SNE visualization of the semantic change of 3 words in English using\nword2vec vectors. The modern sense of each word, and the grey context words, are com-\nputed from the most recent (modern) time-point embedding space. Earlier points are com-\nputed from earlier historical embedding spaces. The visualizations show the changes in the\nword gayfrom meanings related to \u201ccheerful\u201d or \u201cfrolicsome\u201d to referring to homosexuality,\nthe development of the modern \u201ctransmission\u201d sense of broadcast from its original sense of\nsowing seeds, and the pejoration of the word awful as it shifted from meaning \u201cfull of awe\u201d\nto meaning \u201cterrible or appalling\u201d (Hamilton et al., 2016).\n6.11 Bias and Embeddings\nIn addition to their ability to learn word meaning from text, embeddings, alas,\nalso reproduce the implicit biases and stereotypes that were latent in the text. As\nthe prior section just showed, embeddings can roughly model relational similar-\nity: \u2018queen\u2019 as the closest word to \u2018king\u2019 - \u2018man\u2019 + \u2018woman\u2019 implies the analogy\nman:woman::king:queen . But these same embedding analogies also exhibit gender\nstereotypes. For example Bolukbasi et al. (2016) \ufb01nd that the closest occupation\nto \u2018computer programmer\u2019 - \u2018man\u2019 + \u2018woman\u2019 in word2vec embeddings trained on\nnews text is \u2018homemaker\u2019, and that the embeddings similarly suggest the analogy\n\u2018father\u2019 is to \u2018doctor\u2019 as \u2018mother\u2019 is to \u2018nurse\u2019. This could result in what Crawford\n(2017) and Blodgett et al. (2020) call an allocational harm , when a system allo-allocational\nharm\ncates resources (jobs or credit) unfairly to different groups. For example algorithms\nthat use embeddings as part of a search for hiring potential programmers or doctors\nmight thus incorrectly downweight documents with women\u2019s names.\nIt turns out that embeddings don\u2019t just re\ufb02ect the statistics of their input, but also\namplify bias; gendered terms become more gendered in embedding space than theybias\nampli\ufb01cation\nwere in the input text statistics (Zhao et al. 2017, Ethayarajh et al. 2019b, Jia et al.\n2020), and biases are more exaggerated than in actual labor employment statistics\n(Garg et al., 2018).\nEmbeddings also encode the implicit associations that are a property of human\nreasoning. The Implicit Association Test (Greenwald et al., 1998) measures peo- 6.12 \u2022 E VALUATING VECTOR MODELS 27\nple\u2019s associations between concepts (like \u2018\ufb02owers\u2019 or \u2018insects\u2019) and attributes (like\n\u2018pleasantness\u2019 and \u2018unpleasantness\u2019) by measuring differences in the latency with\nwhich they label words in the various categories.7Using such methods, people\nin the United States have been shown to associate African-American names with\nunpleasant words (more than European-American names), male names more with\nmathematics and female names with the arts, and old people\u2019s names with unpleas-\nant words (Greenwald et al. 1998, Nosek et al. 2002a, Nosek et al. 2002b). Caliskan\net al. (2017) replicated all these \ufb01ndings of implicit associations using GloVe vectors\nand cosine similarity instead of human latencies. For example African-American\nnames like \u2018Leroy\u2019 and \u2018Shaniqua\u2019 had a higher GloVe cosine with unpleasant words\nwhile European-American names (\u2018Brad\u2019, \u2018Greg\u2019, \u2018Courtney\u2019) had a higher cosine\nwith pleasant words. These problems with embeddings are an example of a repre-\nsentational harm (Crawford 2017, Blodgett et al. 2020), which is a harm caused byrepresentational\nharm\na system demeaning or even ignoring some social groups. Any embedding-aware al-\ngorithm that made use of word sentiment could thus exacerbate bias against African\nAmericans.\nRecent research focuses on ways to try to remove these kinds of biases, for\nexample by developing a transformation of the embedding space that removes gen-\nder stereotypes but preserves de\ufb01nitional gender (Bolukbasi et al. 2016, Zhao et al.\n2017) or changing the training procedure (Zhao et al., 2018). However, although\nthese sorts of debiasing may reduce bias in embeddings, they do not eliminate it debiasing\n(Gonen and Goldberg, 2019), and this remains an open problem.\nHistorical embeddings are also being used to measure biases in the past. Garg\net al. (2018) used embeddings from historical texts to measure the association be-\ntween embeddings for occupations and embeddings for names of various ethnici-\nties or genders (for example the relative cosine similarity of women\u2019s names versus\nmen\u2019s to occupation words like \u2018librarian\u2019 or \u2018carpenter\u2019) across the 20th century.\nThey found that the cosines correlate with the empirical historical percentages of\nwomen or ethnic groups in those occupations. Historical embeddings also repli-\ncated old surveys of ethnic stereotypes; the tendency of experimental participants in\n1933 to associate adjectives like \u2018industrious\u2019 or \u2018superstitious\u2019 with, e.g., Chinese\nethnicity, correlates with the cosine between Chinese last names and those adjectives\nusing embeddings trained on 1930s text. They also were able to document historical\ngender biases, such as the fact that embeddings for adjectives related to competence\n(\u2018smart\u2019, \u2018wise\u2019, \u2018thoughtful\u2019, \u2018resourceful\u2019) had a higher cosine with male than fe-\nmale words, and showed that this bias has been slowly decreasing since 1960. We\nreturn in later chapters to this question about the role of bias in natural language\nprocessing.\n6.12 Evaluating Vector Models\nThe most important evaluation metric for vector models is extrinsic evaluation on\ntasks, i.e., using vectors in an NLP task and seeing whether this improves perfor-\nmance over some other model.\n7Roughly speaking, if humans associate \u2018\ufb02owers\u2019 with \u2018pleasantness\u2019 and \u2018insects\u2019 with \u2018unpleasant-\nness\u2019, when they are instructed to push a green button for \u2018\ufb02owers\u2019 (daisy, iris, lilac) and \u2018pleasant words\u2019\n(love, laughter, pleasure) and a red button for \u2018insects\u2019 (\ufb02ea, spider, mosquito) and \u2018unpleasant words\u2019\n(abuse, hatred, ugly) they are faster than in an incongruous condition where they push a red button for\n\u2018\ufb02owers\u2019 and \u2018unpleasant words\u2019 and a green button for \u2018insects\u2019 and \u2018pleasant words\u2019. 28 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\nNonetheless it is useful to have intrinsic evaluations. The most common metric\nis to test their performance on similarity , computing the correlation between an\nalgorithm\u2019s word similarity scores and word similarity ratings assigned by humans.\nWordSim-353 (Finkelstein et al., 2002) is a commonly used set of ratings from 0\nto 10 for 353 noun pairs; for example ( plane ,car) had an average score of 5.77.\nSimLex-999 (Hill et al., 2015) is a more complex dataset that quanti\ufb01es similarity\n(cup, mug ) rather than relatedness ( cup, coffee ), and includes concrete and abstract\nadjective, noun and verb pairs. The TOEFL dataset is a set of 80 questions, each\nconsisting of a target word with 4 additional word choices; the task is to choose\nwhich is the correct synonym, as in the example: Levied is closest in meaning to:\nimposed, believed, requested, correlated (Landauer and Dumais, 1997). All of these\ndatasets present words without context.\nSlightly more realistic are intrinsic similarity tasks that include context. The\nStanford Contextual Word Similarity (SCWS) dataset (Huang et al., 2012) and the\nWord-in-Context (WiC) dataset (Pilehvar and Camacho-Collados, 2019) offer richer\nevaluation scenarios. SCWS gives human judgments on 2,003 pairs of words in\ntheir sentential context, while WiC gives target words in two sentential contexts that\nare either in the same or different senses; see Appendix G. The semantic textual\nsimilarity task (Agirre et al. 2012, Agirre et al. 2015) evaluates the performance of\nsentence-level similarity algorithms, consisting of a set of pairs of sentences, each\npair with human-labeled similarity scores.\nAnother task used for evaluation is the analogy task, discussed on page 24, where\nthe system has to solve problems of the form a is to b as a* is to b* , given a, b, anda*\nand having to \ufb01nd b*(Turney and Littman, 2005). A number of sets of tuples have\nbeen created for this task (Mikolov et al. 2013a, Mikolov et al. 2013c, Gladkova\net al. 2016), covering morphology ( city:cities::child:children ), lexicographic rela-\ntions ( leg:table::spout:teapot ) and encyclopedia relations ( Beijing:China::Dublin:Ireland ),\nsome drawing from the SemEval-2012 Task 2 dataset of 79 different relations (Jur-\ngens et al., 2012).\nAll embedding algorithms suffer from inherent variability. For example because\nof randomness in the initialization and the random negative sampling, algorithms\nlike word2vec may produce different results even from the same dataset, and in-\ndividual documents in a collection may strongly impact the resulting embeddings\n(Tian et al. 2016, Hellrich and Hahn 2016, Antoniak and Mimno 2018). When em-\nbeddings are used to study word associations in particular corpora, therefore, it is\nbest practice to train multiple embeddings with bootstrap sampling over documents\nand average the results (Antoniak and Mimno, 2018).\n6.13 Summary\n\u2022 In vector semantics, a word is modeled as a vector\u2014a point in high-dimensional\nspace, also called an embedding . In this chapter we focus on static embed-\ndings , where each word is mapped to a \ufb01xed embedding.\n\u2022 Vector semantic models fall into two classes: sparse anddense . In sparse\nmodels each dimension corresponds to a word in the vocabulary Vand cells\nare functions of co-occurrence counts . The term-document matrix has a\nrow for each word ( term ) in the vocabulary and a column for each document.\nTheword-context orterm-term matrix has a row for each (target) word in BIBLIOGRAPHICAL AND HISTORICAL NOTES 29\nthe vocabulary and a column for each context term in the vocabulary. Two\nsparse weightings are common: the tf-idf weighting which weights each cell\nby its term frequency andinverse document frequency , and PPMI (point-\nwise positive mutual information), which is most common for word-context\nmatrices.\n\u2022 Dense vector models have dimensionality 50\u20131000. Word2vec algorithms\nlikeskip-gram are a popular way to compute dense embeddings. Skip-gram\ntrains a logistic regression classi\ufb01er to compute the probability that two words\nare \u2018likely to occur nearby in text\u2019. This probability is computed from the dot\nproduct between the embeddings for the two words.\n\u2022 Skip-gram uses stochastic gradient descent to train the classi\ufb01er, by learning\nembeddings that have a high dot product with embeddings of words that occur\nnearby and a low dot product with noise words.\n\u2022 Other important embedding algorithms include GloVe , a method based on\nratios of word co-occurrence probabilities.\n\u2022 Whether using sparse or dense vectors, word and document similarities are\ncomputed by some function of the dot product between vectors. The cosine\nof two vectors\u2014a normalized dot product\u2014is the most popular such metric.\nBibliographical and Historical Notes\nThe idea of vector semantics arose out of research in the 1950s in three distinct\n\ufb01elds: linguistics, psychology, and computer science, each of which contributed a\nfundamental aspect of the model.\nThe idea that meaning is related to the distribution of words in context was\nwidespread in linguistic theory of the 1950s, among distributionalists like Zellig\nHarris, Martin Joos, and J. R. Firth, and semioticians like Thomas Sebeok. As Joos\n(1950) put it,\nthe linguist\u2019s \u201cmeaning\u201d of a morpheme. . . is by de\ufb01nition the set of conditional\nprobabilities of its occurrence in context with all other morphemes.\nThe idea that the meaning of a word might be modeled as a point in a multi-\ndimensional semantic space came from psychologists like Charles E. Osgood, who\nhad been studying how people responded to the meaning of words by assigning val-\nues along scales like happy/sad orhard/soft . Osgood et al. (1957) proposed that the\nmeaning of a word in general could be modeled as a point in a multidimensional\nEuclidean space, and that the similarity of meaning between two words could be\nmodeled as the distance between these points in the space.\nA \ufb01nal intellectual source in the 1950s and early 1960s was the \ufb01eld then called\nmechanical indexing , now known as information retrieval . In what became knownmechanical\nindexing\nas the vector space model for information retrieval (Salton 1971, Sparck Jones\n1986), researchers demonstrated new ways to de\ufb01ne the meaning of words in terms\nof vectors (Switzer, 1965), and re\ufb01ned methods for word similarity based on mea-\nsures of statistical association between words like mutual information (Giuliano,\n1965) and idf (Sparck Jones, 1972), and showed that the meaning of documents\ncould be represented in the same vector spaces used for words. Around the same\ntime, (Cordier, 1965) showed that factor analysis of word association probabilities\ncould be used to form dense vector representations of words. 30 CHAPTER 6 \u2022 V ECTOR SEMANTICS AND EMBEDDINGS\nSome of the philosophical underpinning of the distributional way of thinking\ncame from the late writings of the philosopher Wittgenstein, who was skeptical of\nthe possibility of building a completely formal theory of meaning de\ufb01nitions for\neach word. Wittgenstein suggested instead that \u201cthe meaning of a word is its use in\nthe language\u201d (Wittgenstein, 1953, PI 43). That is, instead of using some logical lan-\nguage to de\ufb01ne each word, or drawing on denotations or truth values, Wittgenstein\u2019s\nidea is that we should de\ufb01ne a word by how it is used by people in speaking and un-\nderstanding in their day-to-day interactions, thus pre\ufb01guring the movement toward\nembodied and experiential models in linguistics and NLP (Glenberg and Robertson\n2000, Lake and Murphy 2021, Bisk et al. 2020, Bender and Koller 2020).\nMore distantly related is the idea of de\ufb01ning words by a vector of discrete fea-\ntures, which has roots at least as far back as Descartes and Leibniz (Wierzbicka 1992,\nWierzbicka 1996). By the middle of the 20th century, beginning with the work of\nHjelmslev (Hjelmslev, 1969) (originally 1943) and \ufb02eshed out in early models of\ngenerative grammar (Katz and Fodor, 1963), the idea arose of representing mean-\ning with semantic features , symbols that represent some sort of primitive meaning.semantic\nfeature\nFor example words like hen,rooster , orchick , have something in common (they all\ndescribe chickens) and something different (their age and sex), representable as:\nhen +female, +chicken, +adult\nrooster -female, +chicken, +adult\nchick +chicken, -adult\nThe dimensions used by vector models of meaning to de\ufb01ne words, however, are\nonly abstractly related to this idea of a small \ufb01xed number of hand-built dimensions.\nNonetheless, there has been some attempt to show that certain dimensions of em-\nbedding models do contribute some speci\ufb01c compositional aspect of meaning like\nthese early semantic features.\nThe use of dense vectors to model word meaning, and indeed the term embed-\nding , grew out of the latent semantic indexing (LSI) model (Deerwester et al.,\n1988) recast as LSA (latent semantic analysis ) (Deerwester et al., 1990). In LSA\nsingular value decomposition \u2014SVD \u2014 is applied to a term-document matrix (each SVD\ncell weighted by log frequency and normalized by entropy), and then the \ufb01rst 300\ndimensions are used as the LSA embedding. Singular Value Decomposition (SVD)\nis a method for \ufb01nding the most important dimensions of a data set, those dimen-\nsions along which the data varies the most. LSA was then quickly widely applied:\nas a cognitive model Landauer and Dumais (1997), and for tasks like spell checking\n(Jones and Martin, 1997), language modeling (Bellegarda 1997, Coccaro and Ju-\nrafsky 1998, Bellegarda 2000), morphology induction (Schone and Jurafsky 2000,\nSchone and Jurafsky 2001b), multiword expressions (MWEs) (Schone and Jurafsky,\n2001a), and essay grading (Rehder et al., 1998). Related models were simultane-\nously developed and applied to word sense disambiguation by Sch \u00a8utze (1992). LSA\nalso led to the earliest use of embeddings to represent words in a probabilistic clas-\nsi\ufb01er, in the logistic regression document router of Sch \u00a8utze et al. (1995). The idea of\nSVD on the term-term matrix (rather than the term-document matrix) as a model of\nmeaning for NLP was proposed soon after LSA by Sch \u00a8utze (1992). Sch \u00a8utze applied\nthe low-rank (97-dimensional) embeddings produced by SVD to the task of word\nsense disambiguation, analyzed the resulting semantic space, and also suggested\npossible techniques like dropping high-order dimensions. See Sch \u00a8utze (1997).\nA number of alternative matrix models followed on from the early SVD work,\nincluding Probabilistic Latent Semantic Indexing (PLSI) (Hofmann, 1999), Latent\nDirichlet Allocation (LDA) (Blei et al., 2003), and Non-negative Matrix Factoriza- EXERCISES 31\ntion (NMF) (Lee and Seung, 1999).\nThe LSA community seems to have \ufb01rst used the word \u201cembedding\u201d in Landauer\net al. (1997), in a variant of its mathematical meaning as a mapping from one space\nor mathematical structure to another. In LSA, the word embedding seems to have\ndescribed the mapping from the space of sparse count vectors to the latent space of\nSVD dense vectors. Although the word thus originally meant the mapping from one\nspace to another, it has metonymically shifted to mean the resulting dense vector in\nthe latent space, and it is in this sense that we currently use the word.\nBy the next decade, Bengio et al. (2003) and Bengio et al. (2006) showed that\nneural language models could also be used to develop embeddings as part of the task\nof word prediction. Collobert and Weston (2007), Collobert and Weston (2008), and\nCollobert et al. (2011) then demonstrated that embeddings could be used to represent\nword meanings for a number of NLP tasks. Turian et al. (2010) compared the value\nof different kinds of embeddings for different NLP tasks. Mikolov et al. (2011)\nshowed that recurrent neural nets could be used as language models. The idea of\nsimplifying the hidden layer of these neural net language models to create the skip-\ngram (and also CBOW) algorithms was proposed by Mikolov et al. (2013a). The\nnegative sampling training algorithm was proposed in Mikolov et al. (2013b). There\nare numerous surveys of static embeddings and their parameterizations (Bullinaria\nand Levy 2007, Bullinaria and Levy 2012, Lapesa and Evert 2014, Kiela and Clark\n2014, Levy et al. 2015).\nSee Manning et al. (2008) and Chapter 14 for a deeper understanding of the role\nof vectors in information retrieval, including how to compare queries with docu-\nments, more details on tf-idf, and issues of scaling to very large datasets. See Kim\n(2019) for a clear and comprehensive tutorial on word2vec. Cruse (2004) is a useful\nintroductory linguistic text on lexical semantics.\nExercises 32 Chapter 6 \u2022 Vector Semantics and Embeddings\nAgirre, E., C. Banea, C. Cardie, D. Cer, M. Diab,\nA. Gonzalez-Agirre, W. Guo, I. Lopez-Gazpio, M. Mar-\nitxalar, R. Mihalcea, G. Rigau, L. Uria, and J. Wiebe.\n2015. SemEval-2015 task 2: Semantic textual similarity,\nEnglish, Spanish and pilot on interpretability. SemEval-\n15.\nAgirre, E., M. Diab, D. Cer, and A. Gonzalez-Agirre. 2012.\nSemEval-2012 task 6: A pilot on semantic textual simi-\nlarity. SemEval-12 .\nAntoniak, M. and D. Mimno. 2018. Evaluating the stability\nof embedding-based word similarities. TACL , 6:107\u2013119.\nBellegarda, J. R. 1997. A latent semantic analysis framework\nfor large-span language modeling. EUROSPEECH .\nBellegarda, J. R. 2000. Exploiting latent semantic informa-\ntion in statistical language modeling. Proceedings of the\nIEEE , 89(8):1279\u20131296.\nBender, E. M. and A. Koller. 2020. Climbing towards NLU:\nOn meaning, form, and understanding in the age of data.\nACL.\nBengio, Y ., A. Courville, and P. Vincent. 2013. Represen-\ntation learning: A review and new perspectives. IEEE\nTransactions on Pattern Analysis and Machine Intelli-\ngence , 35(8):1798\u20131828.\nBengio, Y ., R. Ducharme, P. Vincent, and C. Jauvin. 2003.\nA neural probabilistic language model. JMLR , 3:1137\u2013\n1155.\nBengio, Y ., H. Schwenk, J.-S. Sen \u00b4ecal, F. Morin, and J.-L.\nGauvain. 2006. Neural probabilistic language models. In\nInnovations in Machine Learning , 137\u2013186. Springer.\nBisk, Y ., A. Holtzman, J. Thomason, J. Andreas, Y . Bengio,\nJ. Chai, M. Lapata, A. Lazaridou, J. May, A. Nisnevich,\nN. Pinto, and J. Turian. 2020. Experience grounds lan-\nguage. EMNLP .\nBlei, D. M., A. Y . Ng, and M. I. Jordan. 2003. Latent Dirich-\nlet allocation. JMLR , 3(5):993\u20131022.\nBlodgett, S. L., S. Barocas, H. Daum \u00b4e III, and H. Wallach.\n2020. Language (technology) is power: A critical survey\nof \u201cbias\u201d in NLP. ACL.\nBojanowski, P., E. Grave, A. Joulin, and T. Mikolov. 2017.\nEnriching word vectors with subword information. TACL ,\n5:135\u2013146.\nBolukbasi, T., K.-W. Chang, J. Zou, V . Saligrama, and A. T.\nKalai. 2016. Man is to computer programmer as woman\nis to homemaker? Debiasing word embeddings. NeurIPS .\nBr\u00b4eal, M. 1897. Essai de S \u00b4emantique: Science des signi\ufb01ca-\ntions . Hachette.\nBudanitsky, A. and G. Hirst. 2006. Evaluating WordNet-\nbased measures of lexical semantic relatedness. Compu-\ntational Linguistics , 32(1):13\u201347.\nBullinaria, J. A. and J. P. Levy. 2007. Extracting seman-\ntic representations from word co-occurrence statistics:\nA computational study. Behavior research methods ,\n39(3):510\u2013526.\nBullinaria, J. A. and J. P. Levy. 2012. Extracting semantic\nrepresentations from word co-occurrence statistics: stop-\nlists, stemming, and SVD. Behavior research methods ,\n44(3):890\u2013907.\nCaliskan, A., J. J. Bryson, and A. Narayanan. 2017. Seman-\ntics derived automatically from language corpora contain\nhuman-like biases. Science , 356(6334):183\u2013186.Carlson, G. N. 1977. Reference to kinds in English . Ph.D.\nthesis, University of Massachusetts, Amherst. Forward.\nChurch, K. W. and P. Hanks. 1989. Word association norms,\nmutual information, and lexicography. ACL.\nChurch, K. W. and P. Hanks. 1990. Word association norms,\nmutual information, and lexicography. Computational\nLinguistics , 16(1):22\u201329.\nClark, E. 1987. The principle of contrast: A constraint on\nlanguage acquisition. In B. MacWhinney, ed., Mecha-\nnisms of language acquisition , 1\u201333. LEA.\nCoccaro, N. and D. Jurafsky. 1998. Towards better integra-\ntion of semantic predictors in statistical language model-\ning. ICSLP .\nCollobert, R. and J. Weston. 2007. Fast semantic extraction\nusing a novel neural network architecture. ACL.\nCollobert, R. and J. Weston. 2008. A uni\ufb01ed architecture for\nnatural language processing: Deep neural networks with\nmultitask learning. ICML .\nCollobert, R., J. Weston, L. Bottou, M. Karlen,\nK. Kavukcuoglu, and P. Kuksa. 2011. Natural language\nprocessing (almost) from scratch. JMLR , 12:2493\u20132537.\nCordier, B. 1965. Factor-analysis of correspondences. COL-\nING 1965 .\nCrawford, K. 2017. The trouble with bias. Keynote at\nNeurIPS.\nCruse, D. A. 2004. Meaning in Language: an Introduction\nto Semantics and Pragmatics . Oxford University Press.\nSecond edition.\nDagan, I., S. Marcus, and S. Markovitch. 1993. Contextual\nword similarity and estimation from sparse data. ACL.\nDavies, M. 2012. Expanding horizons in historical lin-\nguistics with the 400-million word Corpus of Historical\nAmerican English. Corpora , 7(2):121\u2013157.\nDavies, M. 2015. The Wikipedia Corpus: 4.6 million arti-\ncles, 1.9 billion words. Adapted from Wikipedia. https:\n//www.english-corpora.org/wiki/ .\nDeerwester, S. C., S. T. Dumais, G. W. Furnas, R. A. Harsh-\nman, T. K. Landauer, K. E. Lochbaum, and L. Streeter.\n1988. Computer information retrieval using latent seman-\ntic structure: US Patent 4,839,853.\nDeerwester, S. C., S. T. Dumais, T. K. Landauer, G. W. Fur-\nnas, and R. A. Harshman. 1990. Indexing by latent se-\nmantics analysis. JASIS , 41(6):391\u2013407.\nEthayarajh, K., D. Duvenaud, and G. Hirst. 2019a. Towards\nunderstanding linear word analogies. ACL.\nEthayarajh, K., D. Duvenaud, and G. Hirst. 2019b. Under-\nstanding undesirable word embedding associations. ACL.\nFano, R. M. 1961. Transmission of Information: A Statistical\nTheory of Communications . MIT Press.\nFinkelstein, L., E. Gabrilovich, Y . Matias, E. Rivlin,\nZ. Solan, G. Wolfman, and E. Ruppin. 2002. Placing\nsearch in context: The concept revisited. ACM Trans-\nactions on Information Systems , 20(1):116\u2014-131.\nFirth, J. R. 1957. A synopsis of linguistic theory 1930\u2013\n1955. In Studies in Linguistic Analysis . Philological So-\nciety. Reprinted in Palmer, F. (ed.) 1968. Selected Papers\nof J. R. Firth. Longman, Harlow. Exercises 33\nGarg, N., L. Schiebinger, D. Jurafsky, and J. Zou. 2018.\nWord embeddings quantify 100 years of gender and eth-\nnic stereotypes. Proceedings of the National Academy of\nSciences , 115(16):E3635\u2013E3644.\nGirard, G. 1718. La justesse de la langue franc \u00b8oise: ou les\ndiff\u00b4erentes signi\ufb01cations des mots qui passent pour syn-\nonimes . Laurent d\u2019Houry, Paris.\nGiuliano, V . E. 1965. The interpretation of word\nassociations. Statistical Association Methods For\nMechanized Documentation. Symposium Proceed-\nings. Washington, D.C., USA, March 17, 1964 .\nhttps://nvlpubs.nist.gov/nistpubs/Legacy/\nMP/nbsmiscellaneouspub269.pdf .\nGladkova, A., A. Drozd, and S. Matsuoka. 2016. Analogy-\nbased detection of morphological and semantic relations\nwith word embeddings: what works and what doesn\u2019t.\nNAACL Student Research Workshop .\nGlenberg, A. M. and D. A. Robertson. 2000. Symbol ground-\ning and meaning: A comparison of high-dimensional and\nembodied theories of meaning. Journal of memory and\nlanguage , 43(3):379\u2013401.\nGonen, H. and Y . Goldberg. 2019. Lipstick on a pig: Debi-\nasing methods cover up systematic gender biases in word\nembeddings but do not remove them. NAACL HLT .\nGould, S. J. 1980. The Panda\u2019s Thumb . Penguin Group.\nGreenwald, A. G., D. E. McGhee, and J. L. K. Schwartz.\n1998. Measuring individual differences in implicit cogni-\ntion: the implicit association test. Journal of personality\nand social psychology , 74(6):1464\u20131480.\nHamilton, W. L., J. Leskovec, and D. Jurafsky. 2016. Di-\nachronic word embeddings reveal statistical laws of se-\nmantic change. ACL.\nHarris, Z. S. 1954. Distributional structure. Word , 10:146\u2013\n162.\nHellrich, J. and U. Hahn. 2016. Bad company\u2014\nNeighborhoods in neural embedding spaces considered\nharmful. COLING .\nHill, F., R. Reichart, and A. Korhonen. 2015. Simlex-999:\nEvaluating semantic models with (genuine) similarity es-\ntimation. Computational Linguistics , 41(4):665\u2013695.\nHjelmslev, L. 1969. Prologomena to a Theory of Language .\nUniversity of Wisconsin Press. Translated by Francis J.\nWhit\ufb01eld; original Danish edition 1943.\nHofmann, T. 1999. Probabilistic latent semantic indexing.\nSIGIR-99 .\nHuang, E. H., R. Socher, C. D. Manning, and A. Y . Ng. 2012.\nImproving word representations via global context and\nmultiple word prototypes. ACL.\nJia, S., T. Meng, J. Zhao, and K.-W. Chang. 2020. Mitigat-\ning gender bias ampli\ufb01cation in distribution by posterior\nregularization. ACL.\nJones, M. P. and J. H. Martin. 1997. Contextual spelling cor-\nrection using latent semantic analysis. ANLP .\nJoos, M. 1950. Description of language design. JASA ,\n22:701\u2013708.\nJurafsky, D. 2014. The Language of Food . W. W. Norton,\nNew York.Jurgens, D., S. M. Mohammad, P. Turney, and K. Holyoak.\n2012. SemEval-2012 task 2: Measuring degrees of rela-\ntional similarity. *SEM 2012 .\nKatz, J. J. and J. A. Fodor. 1963. The structure of a semantic\ntheory. Language , 39:170\u2013210.\nKiela, D. and S. Clark. 2014. A systematic study of semantic\nvector space model parameters. EACL 2nd Workshop on\nContinuous Vector Space Models and their Composition-\nality (CVSC) .\nKim, E. 2019. Optimize computational ef\ufb01ciency\nof skip-gram with negative sampling. https://\naegis4048.github.io/optimize_computational_\nefficiency_of_skip-gram_with_negative_\nsampling .\nLake, B. M. and G. L. Murphy. 2021. Word meaning in\nminds and machines. Psychological Review . In press.\nLandauer, T. K. and S. T. Dumais. 1997. A solution to Plato\u2019s\nproblem: The Latent Semantic Analysis theory of acqui-\nsition, induction, and representation of knowledge. Psy-\nchological Review , 104:211\u2013240.\nLandauer, T. K., D. Laham, B. Rehder, and M. E. Schreiner.\n1997. How well can passage meaning be derived with-\nout using word order? A comparison of Latent Semantic\nAnalysis and humans. COGSCI .\nLapesa, G. and S. Evert. 2014. A large scale evaluation of\ndistributional semantic models: Parameters, interactions\nand model selection. TACL , 2:531\u2013545.\nLee, D. D. and H. S. Seung. 1999. Learning the parts of\nobjects by non-negative matrix factorization. Nature ,\n401(6755):788\u2013791.\nLevy, O. and Y . Goldberg. 2014a. Dependency-based word\nembeddings. ACL.\nLevy, O. and Y . Goldberg. 2014b. Linguistic regularities in\nsparse and explicit word representations. CoNLL .\nLevy, O. and Y . Goldberg. 2014c. Neural word embedding\nas implicit matrix factorization. NeurIPS .\nLevy, O., Y . Goldberg, and I. Dagan. 2015. Improving dis-\ntributional similarity with lessons learned from word em-\nbeddings. TACL , 3:211\u2013225.\nLi, J., X. Chen, E. H. Hovy, and D. Jurafsky. 2015. Visual-\nizing and understanding neural models in NLP. NAACL\nHLT.\nLin, Y ., J.-B. Michel, E. Lieberman Aiden, J. Orwant,\nW. Brockman, and S. Petrov. 2012. Syntactic annotations\nfor the Google Books NGram corpus. ACL.\nLinzen, T. 2016. Issues in evaluating semantic spaces us-\ning word analogies. 1st Workshop on Evaluating Vector-\nSpace Representations for NLP .\nLuhn, H. P. 1957. A statistical approach to the mechanized\nencoding and searching of literary information. IBM\nJournal of Research and Development , 1(4):309\u2013317.\nManning, C. D., P. Raghavan, and H. Sch \u00a8utze. 2008. Intro-\nduction to Information Retrieval . Cambridge.\nMikolov, T., K. Chen, G. S. Corrado, and J. Dean. 2013a. Ef-\n\ufb01cient estimation of word representations in vector space.\nICLR 2013 .\nMikolov, T., S. Kombrink, L. Burget, J. H. \u02c7Cernock `y, and\nS. Khudanpur. 2011. Extensions of recurrent neural net-\nwork language model. ICASSP . 34 Chapter 6 \u2022 Vector Semantics and Embeddings\nMikolov, T., I. Sutskever, K. Chen, G. S. Corrado, and\nJ. Dean. 2013b. Distributed representations of words and\nphrases and their compositionality. NeurIPS .\nMikolov, T., W.-t. Yih, and G. Zweig. 2013c. Linguis-\ntic regularities in continuous space word representations.\nNAACL HLT .\nNiwa, Y . and Y . Nitta. 1994. Co-occurrence vectors from\ncorpora vs. distance vectors from dictionaries. COLING .\nNosek, B. A., M. R. Banaji, and A. G. Greenwald. 2002a.\nHarvesting implicit group attitudes and beliefs from a\ndemonstration web site. Group Dynamics: Theory, Re-\nsearch, and Practice , 6(1):101.\nNosek, B. A., M. R. Banaji, and A. G. Greenwald. 2002b.\nMath=male, me=female, therefore math 6=me.Journal of\npersonality and social psychology , 83(1):44.\nOsgood, C. E., G. J. Suci, and P. H. Tannenbaum. 1957. The\nMeasurement of Meaning . University of Illinois Press.\nPennington, J., R. Socher, and C. D. Manning. 2014. GloVe:\nGlobal vectors for word representation. EMNLP .\nPeterson, J. C., D. Chen, and T. L. Grif\ufb01ths. 2020. Parallelo-\ngrams revisited: Exploring the limitations of vector space\nmodels for simple analogies. Cognition , 205.\nPilehvar, M. T. and J. Camacho-Collados. 2019. WiC: the\nword-in-context dataset for evaluating context-sensitive\nmeaning representations. NAACL HLT .\nRehder, B., M. E. Schreiner, M. B. W. Wolfe, D. Laham,\nT. K. Landauer, and W. Kintsch. 1998. Using Latent\nSemantic Analysis to assess knowledge: Some technical\nconsiderations. Discourse Processes , 25(2-3):337\u2013354.\nRohde, D. L. T., L. M. Gonnerman, and D. C. Plaut. 2006.\nAn improved model of semantic similarity based on lexi-\ncal co-occurrence. CACM , 8:627\u2013633.\nRumelhart, D. E. and A. A. Abrahamson. 1973. A model for\nanalogical reasoning. Cognitive Psychology , 5(1):1\u201328.\nSalton, G. 1971. The SMART Retrieval System: Experiments\nin Automatic Document Processing . Prentice Hall.\nSchluter, N. 2018. The word analogy testing caveat. NAACL\nHLT.\nSchone, P. and D. Jurafsky. 2000. Knowlege-free induction\nof morphology using latent semantic analysis. CoNLL .\nSchone, P. and D. Jurafsky. 2001a. Is knowledge-free in-\nduction of multiword unit dictionary headwords a solved\nproblem? EMNLP .\nSchone, P. and D. Jurafsky. 2001b. Knowledge-free induc-\ntion of in\ufb02ectional morphologies. NAACL .\nSch\u00a8utze, H. 1992. Dimensions of meaning. Proceedings of\nSupercomputing \u201992 . IEEE Press.\nSch\u00a8utze, H. 1997. Ambiguity Resolution in Language Learn-\ning \u2013 Computational and Cognitive Models . CSLI, Stan-\nford, CA.\nSch\u00a8utze, H., D. A. Hull, and J. Pedersen. 1995. A compar-\nison of classi\ufb01ers and document representations for the\nrouting problem. SIGIR-95 .\nSch\u00a8utze, H. and J. Pedersen. 1993. A vector model for syn-\ntagmatic and paradigmatic relatedness. 9th Annual Con-\nference of the UW Centre for the New OED and Text Re-\nsearch .Sparck Jones, K. 1972. A statistical interpretation of term\nspeci\ufb01city and its application in retrieval. Journal of Doc-\numentation , 28(1):11\u201321.\nSparck Jones, K. 1986. Synonymy and Semantic Classi\ufb01ca-\ntion. Edinburgh University Press, Edinburgh. Republica-\ntion of 1964 PhD Thesis.\nSwitzer, P. 1965. Vector images in document retrieval.\nStatistical Association Methods For Mechanized Docu-\nmentation. Symposium Proceedings. Washington, D.C.,\nUSA, March 17, 1964 .https://nvlpubs.nist.gov/\nnistpubs/Legacy/MP/nbsmiscellaneouspub269.\npdf.\nTian, Y ., V . Kulkarni, B. Perozzi, and S. Skiena. 2016. On\nthe convergent properties of word embedding methods.\nArXiv preprint arXiv:1605.03956.\nTurian, J., L. Ratinov, and Y . Bengio. 2010. Word represen-\ntations: a simple and general method for semi-supervised\nlearning. ACL.\nTurney, P. D. and M. L. Littman. 2005. Corpus-based learn-\ning of analogies and semantic relations. Machine Learn-\ning, 60(1-3):251\u2013278.\nvan der Maaten, L. and G. E. Hinton. 2008. Visualizing high-\ndimensional data using t-SNE. JMLR , 9:2579\u20132605.\nWierzbicka, A. 1992. Semantics, Culture, and Cognition:\nUniversity Human Concepts in Culture-Speci\ufb01c Con\ufb01gu-\nrations . Oxford University Press.\nWierzbicka, A. 1996. Semantics: Primes and Universals .\nOxford University Press.\nWittgenstein, L. 1953. Philosophical Investigations. (Trans-\nlated by Anscombe, G.E.M.) . Blackwell.\nZhao, J., T. Wang, M. Yatskar, V . Ordonez, and K.-\nW. Chang. 2017. Men also like shopping: Reducing\ngender bias ampli\ufb01cation using corpus-level constraints.\nEMNLP .\nZhao, J., Y . Zhou, Z. Li, W. Wang, and K.-W. Chang. 2018.\nLearning gender-neutral word embeddings. EMNLP .", "vectorsemantics2024.pdf": "Vector Semantics & EmbeddingsWord Meaning What do words mean?N-gram or text classification methods we've seen so far\u25e6Words are just strings (or indices wiin a vocabulary list)\u25e6That's not very satisfactory!Introductory logic classes:\u25e6The meaning of \"dog\" is DOG;  cat is CAT\u2200x DOG(x) \u27f6MAMMAL(x)Old linguistics joke by Barbara Partee in 1967:\u25e6Q: What's the meaning of life?\u25e6A: LIFEThat seems hardly better! DesiderataWhat should a theory of word meaning do for us?Let's look at some desiderataFrom lexical semantics, the linguistic study of word meaning mouse (N)1. any of numerous small rodents...2. a hand-operated device that controls a cursor... Lemmas and sensessenselemma\nA sense or \u201cconcept\u201d is the meaning component of a wordLemmas can be polysemous (have multiple senses)Modified from the online thesaurus WordNet Relations between senses: SynonymySynonyms have the same meaning in some or all contexts.\u25e6filbert / hazelnut\u25e6couch / sofa\u25e6big / large\u25e6automobile / car\u25e6vomit / throw up\u25e6water / H20 Relations between senses: SynonymyNote that there are probably no examples of perfect synonymy.\u25e6Even if many aspects of meaning are identical\u25e6Still may differ based on politeness, slang, register, genre, etc. Relation: Synonymy?water/H20\"H20\" in a surfing guide?big/largemy big sister != my large sister The Linguistic Principle of ContrastDifference in form \u00e0difference in meaning Abb\u00e9Gabriel Girard 1718\n [I do not believe that there is a synonymous word in any language]\n\"\"Re: \"exact\" synonyms\nThanks to Mark Aronoff! Relation: SimilarityWords with similar meanings.  Not synonyms, but sharing some element of meaningcar, bicyclecow, horse Ask humans how similar 2 words areword1word2similarityvanishdisappear9.8 behaveobey7.3 beliefimpression 5.95 musclebone 3.65 modestflexible0.98 holeagreement0.3 SimLex-999 dataset (Hill et al., 2015)  Relation: Word relatednessAlso called \"word association\"Words can be related in any way, perhaps via a semantic frame or field\u25e6coffee, tea:    similar\u25e6coffee, cup:   related, not similar Semantic fieldWords that \u25e6cover a particular semantic domain \u25e6bear structured relations with each other. hospitalssurgeon, scalpel, nurse, anaesthetic, hospitalrestaurantswaiter, menu, plate, food, menu,chefhousesdoor, roof, kitchen, family, bed Relation: AntonymySenses that are opposites with respect to only one feature of meaningOtherwise, they are very similar!dark/light   short/longfast/slowrise/fallhot/coldup/downin/outMore formally: antonyms can\u25e6define a binary opposition or be at opposite ends of a scale\u25e6long/short, fast/slow\u25e6Be reversives:\u25e6rise/fall, up/down Connotation (sentiment)\u2022Words have affectivemeanings\u2022Positive connotations (happy) \u2022Negative connotations (sad)\u2022Connotations can be subtle:\u2022Positive connotation: copy, replica, reproduction \u2022Negative connotation: fake, knockoff, forgery\u2022Evaluation (sentiment!)\u2022Positive evaluation (great, love) \u2022Negative evaluation (terrible, hate) ConnotationWords seem to vary along 3 affective dimensions:\u25e6valence: the pleasantness of the stimulus\u25e6arousal: the intensity of emotion provoked by the stimulus\u25e6dominance: the degree of control exerted by the stimulusOsgood et al. (1957)\nWordScoreWordScoreValencelove1.000toxic0.008happy1.000nightmare0.005Arousalelated0.960mellow0.069frenzy0.965napping0.046Dominancepowerful0.991weak0.045leadership0.983empty0.081Values from NRC VAD Lexicon  (Mohammad 2018) So farConceptsor word senses\u25e6Have a complex many-to-many association with words(homonymy, multiple senses)Have relations with each other\u25e6Synonymy\u25e6Antonymy\u25e6Similarity\u25e6Relatedness\u25e6Connotation Vector Semantics & EmbeddingsWord Meaning Vector Semantics & EmbeddingsVector Semantics Computational models of word meaningCan we build a theory of how to represent word meaning, that accounts for at least some of the desiderata?We'll introduce vector semanticsThe standard model in language processing!Handles many of our goals! Ludwig WittgensteinPI #43: \"The meaning of a word is its use in the language\" Let's define words by their usagesOne way to define \"usage\": words are defined by their environments (the words around them)ZelligHarris (1954): If A and B have almost identical environments we say that they are synonyms. What does recent English borrowing ongchoimean?Suppose you see these sentences:\u2022Ong choiis delicious saut\u00e9edwithgarlic. \u2022Ong choiis superb over rice\u2022Ong choileaveswith salty saucesAnd you've also seen these:\u2022\u2026spinach saut\u00e9edwithgarlicover rice\u2022Chard stems and leavesare delicious\u2022Collard greens and other saltyleafy greensConclusion:\u25e6Ongchoiis a leafy green like spinach, chard, or collard greens\u25e6We could conclude this based on words like \"leaves\" and \"delicious\" and \"sauteed\"  Ongchoi: Ipomoea aquatica \"Water Spinach\"\nYamaguchi, Wikimedia Commons, public domain!\"#kangkongrau mu\u1ed1ng\u2026 Idea 1: Defining meaning by linguistic distributionLet's define the meaning of a word by its distribution in language use, meaning its neighboring words or grammatical environments.  Idea 2: Meaning as a point in space (Osgood et al. 1957)3 affective dimensions for a word\u25e6valence: pleasantness \u25e6arousal: intensity of emotion \u25e6dominance: the degree of control exerted\u25e6Hence the connotation of a word is a vector in 3-spaceWordScoreWordScoreValencelove1.000toxic0.008happy1.000nightmare0.005Arousalelated0.960mellow0.069frenzy0.965napping0.046Dominancepowerful0.991weak0.045leadership0.983empty0.081NRC VAD Lexicon  (Mohammad 2018) Idea 1: Defining meaning by linguistic distributionIdea 2: Meaning as a point in multidimensional space 6CHAPTER6\u2022VECTORSEMANTICS ANDEMBEDDINGS\ngoodnicebadworstnot good\nwonderfulamazingterri\ufb01cdislikeworsevery goodincredibly goodfantasticincredibly badnowyouithatwithbyto\u2019sareisathanFigure 6.1A two-dimensional (t-SNE) projection of embeddings for some words andphrases, showing that words with similar meanings are nearby in space. The original 60-dimensional embeddings were trained for sentiment analysis. Simpli\ufb01ed fromLi et al. (2015)with colors added for explanation.The \ufb01ne-grained model of word similarity of vector semantics offers enormouspower to NLP applications. NLP applications like the sentiment classi\ufb01ers of Chap-ter 4 or Chapter 5 depend on the same words appearing in the training and test sets.But by representing words as embeddings, classi\ufb01ers can assign sentiment as long asit sees some words withsimilar meanings. And as we\u2019ll see, vector semantic modelscan be learned automatically from text without supervision.In this chapter we\u2019ll introduce the two most commonly used models. In thetf-idfmodel, an important baseline, the meaning of a word is de\ufb01ned by a simple functionof the counts of nearby words. We will see that this method results in very longvectors that aresparse, i.e. mostly zeros (since most words simply never occur inthe context of others). We\u2019ll introduce theword2vecmodel family for construct-ing short,densevectors that have useful semantic properties. We\u2019ll also introducethecosine, the standard way to use embeddings to computesemantic similarity, be-tween two words, two sentences, or two documents, an important tool in practicalapplications like question answering, summarization, or automatic essay grading.6.3 Words and Vectors\u201cThe most important attributes of a vector in 3-space are{Location, Location, Location}\u201dRandall Munroe,https://xkcd.com/2358/Vector or distributional models of meaning are generally based on aco-occurrencematrix, a way of representing how often words co-occur. We\u2019ll look at two popularmatrices: the term-document matrix and the term-term matrix.6.3.1 Vectors and documentsIn aterm-document matrix, each row represents a word in the vocabulary and eachterm-documentmatrixcolumn represents a document from some collection of documents. Fig.6.2shows asmall selection from a term-document matrix showing the occurrence of four wordsin four plays by Shakespeare. Each cell in this matrix represents the number of timesa particular word (de\ufb01ned by the row) occurs in a particular document (de\ufb01ned bythe column). Thusfoolappeared 58 times inTwelfth Night.The term-document matrix of Fig.6.2was \ufb01rst de\ufb01ned as part of thevectorspace modelof information retrieval(Salton, 1971). In this model, a document isvector spacemodelDefining meaning as a point in space based on distributionEach word = a vector   (not just \"good\" or \"w45\")Similar words are \"nearby in semantic space\"We build this space automatically by seeing which words are nearby in text We define meaning of a word as a vectorCalled an \"embedding\" because it's embedded into a space (see textbook)The standard way to represent meaning in NLPEvery modern NLP algorithm uses embeddings as the representation of word meaningFine-grained model of meaning for similarity  Intuition: why vectors?Consider sentiment analysis:\u25e6With words,  a feature is a word identity\u25e6Feature 5: 'The previous word was \"terrible\"'\u25e6requires exactsamewordto be in training and test\u25e6With embeddings: \u25e6Feature is a word vector\u25e6'The previous word was vector [35,22,17\u2026]\u25e6Now in the test set we might see a similar vector [34,21,14]\u25e6We can generalize to similar but unseenwords!!!  We'll discuss 2 kinds of embeddingstf-idf\u25e6Information Retrieval workhorse!\u25e6A common baseline model\u25e6Sparsevectors\u25e6Words are represented by (a simple function of) the counts of nearby wordsWord2vec\u25e6Densevectors\u25e6Representation is created by training a classifier to predictwhether a word is likely to appear nearby\u25e6Later we'll discuss extensions called  contextual embeddings From now on:Computing with meaning representationsinstead of string representationsSpeech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright\u00a92020. Allrights reserved. Draft of January 13, 2021.CHAPTER6Vector Semantics andEmbeddingsC\u21e7@\u00c2(|\u0000\u00f3|\u0000\u00ffCNets are for \ufb01sh;Once you get the \ufb01sh, you can forget the net.\u0000\u21e7@\u00c2(\u270f\u0000\u00f3\u270f\u0000\u00ff\u0000Words are for meaning;Once you get the meaning, you can forget the words\u00d1P(Zhuangzi), Chapter 26The asphalt that Los Angeles is famous for occurs mainly on its freeways. Butin the middle of the city is another patch of asphalt, the La Brea tar pits, and thisasphalt preserves millions of fossil bones from the last of the Ice Ages of the Pleis-tocene Epoch. One of these fossils is theSmilodon, or saber-toothed tiger, instantlyrecognizable by its long canines. Five million years ago or so, a completely different\nsabre-tooth tiger calledThylacosmiluslivedin Argentina and other parts of South Amer-ica. Thylacosmilus was a marsupial whereasSmilodon was a placental mammal, but Thy-lacosmilus had the same long upper caninesand, like Smilodon, had a protective bone\ufb02ange on the lower jaw. The similarity ofthese two mammals is one of many examplesof parallel or convergent evolution, in which particular contexts or environmentslead to the evolution of very similar structures in different species(Gould, 1980).The role of context is also important in the similarity of a less biological kindof organism: the word. Words that occur insimilar contextstend to havesimilarmeanings. This link between similarity in how words are distributed and similarityin what they mean is called thedistributional hypothesis. The hypothesis wasdistributionalhypothesis\ufb01rst formulated in the 1950s by linguists likeJoos (1950),Harris (1954), andFirth(1957), who noticed that words which are synonyms (likeoculistandeye-doctor)tended to occur in the same environment (e.g., near words likeeyeorexamined)with the amount of meaning difference between two words \u201ccorresponding roughlyto the amount of difference in their environments\u201d(Harris, 1954, 157).In this chapter we introducevector semantics, which instantiates this linguisticvectorsemanticshypothesis by learning representations of the meaning of words, calledembeddings,embeddingsdirectly from their distributions in texts. These representations are used in every nat-ural language processing application that makes use of meaning, and thestatic em-beddingswe introduce here underlie the more powerful dynamic orcontextualizedembeddingslikeBERTthat we will see in Chapter 10.These word representations are also the \ufb01rst example in this book ofrepre-sentation learning, automatically learning useful representations of the input text.representationlearningFinding suchself-supervisedways to learn representations of the input, instead ofcreating representations by hand viafeature engineering, is an important focus ofNLP research(Bengio et al., 2013). Vector Semantics & EmbeddingsVector Semantics Vector Semantics & EmbeddingsWords and Vectors Term-document matrix6.3\u2022WORDS ANDVECTORS7As You Like It Twelfth Night Julius Caesar Henry Vbattle1 0 7 13good114 80 62 89fool36 58 1 4wit20 15 2 3Figure 6.2The term-document matrix for four words in four Shakespeare plays. Each cellcontains the number of times the (row) word occurs in the (column) document.represented as a count vector, a column in Fig.6.3.To review some basic linear algebra, avectoris, at heart, just a list or array ofvectornumbers. SoAs You Like Itis represented as the list [1,114,36,20] (the \ufb01rstcolumnvectorin Fig.6.3) andJulius Caesaris represented as the list [7,62,1,2] (the thirdcolumn vector). Avector spaceis a collection of vectors, characterized by theirvector spacedimension. In the example in Fig.6.3, the document vectors are of dimension 4,dimensionjust so they \ufb01t on the page; in real term-document matrices, the vectors representingeach document would have dimensionality|V|, the vocabulary size.The ordering of the numbers in a vector space indicates different meaningful di-mensions on which documents vary. Thus the \ufb01rst dimension for both these vectorscorresponds to the number of times the wordbattleoccurs, and we can compareeach dimension, noting for example that the vectors forAs You Like ItandTwelfthNighthave similar values (1 and 0, respectively) for the \ufb01rst dimension.As You Like It Twelfth Night Julius Caesar Henry Vbattle1 0 7 13good114 80 62 89fool36 58 1 4wit20 15 2 3Figure 6.3The term-document matrix for four words in four Shakespeare plays. The redboxes show that each document is represented as a column vector of length four.We can think of the vector for a document as a point in|V|-dimensional space;thus the documents in Fig.6.3are points in 4-dimensional space. Since 4-dimensionalspaces are hard to visualize, Fig.6.4shows a visualization in two dimensions; we\u2019vearbitrarily chosen the dimensions corresponding to the wordsbattleandfool.\n51015202530510Henry V [4,13]As You Like It [36,1]Julius Caesar [1,7]battle foolTwelfth Night [58,0]1540\n354045505560Figure 6.4A spatial visualization of the document vectors for the four Shakespeare playdocuments, showing just two of the dimensions, corresponding to the wordsbattleandfool.The comedies have high values for thefooldimension and low values for thebattledimension.Term-document matrices were originally de\ufb01ned as a means of \ufb01nding similardocuments for the task of documentinformation retrieval. Two documents that are6.3\u2022WORDS ANDVECTORS7As You Like It Twelfth Night Julius Caesar Henry Vbattle1 0 7 13good114 80 62 89fool36 58 1 4wit20 15 2 3Figure 6.2The term-document matrix for four words in four Shakespeare plays. Each cellcontains the number of times the (row) word occurs in the (column) document.represented as a count vector, a column in Fig.6.3.To review some basic linear algebra, avectoris, at heart, just a list or array ofvectornumbers. SoAs You Like Itis represented as the list [1,114,36,20] (the \ufb01rstcolumnvectorin Fig.6.3) andJulius Caesaris represented as the list [7,62,1,2] (the thirdcolumn vector). Avector spaceis a collection of vectors, characterized by theirvector spacedimension. In the example in Fig.6.3, the document vectors are of dimension 4,dimensionjust so they \ufb01t on the page; in real term-document matrices, the vectors representingeach document would have dimensionality|V|, the vocabulary size.The ordering of the numbers in a vector space indicates different meaningful di-mensions on which documents vary. Thus the \ufb01rst dimension for both these vectorscorresponds to the number of times the wordbattleoccurs, and we can compareeach dimension, noting for example that the vectors forAs You Like ItandTwelfthNighthave similar values (1 and 0, respectively) for the \ufb01rst dimension.As You Like It Twelfth Night Julius Caesar Henry Vbattle1 0 7 13good114 80 62 89fool36 58 1 4wit20 15 2 3Figure 6.3The term-document matrix for four words in four Shakespeare plays. The redboxes show that each document is represented as a column vector of length four.We can think of the vector for a document as a point in|V|-dimensional space;thus the documents in Fig.6.3are points in 4-dimensional space. Since 4-dimensionalspaces are hard to visualize, Fig.6.4shows a visualization in two dimensions; we\u2019vearbitrarily chosen the dimensions corresponding to the wordsbattleandfool.\n51015202530510Henry V [4,13]As You Like It [36,1]Julius Caesar [1,7]battle foolTwelfth Night [58,0]1540\n354045505560Figure 6.4A spatial visualization of the document vectors for the four Shakespeare playdocuments, showing just two of the dimensions, corresponding to the wordsbattleandfool.The comedies have high values for thefooldimension and low values for thebattledimension.Term-document matrices were originally de\ufb01ned as a means of \ufb01nding similardocuments for the task of documentinformation retrieval. Two documents that areEach document is represented by a vector of words Visualizing document vectors6.3\u2022WORDS ANDVECTORS7As You Like It Twelfth Night Julius Caesar Henry Vbattle1 0 7 13good114 80 62 89fool36 58 1 4wit20 15 2 3Figure 6.2The term-document matrix for four words in four Shakespeare plays. Each cellcontains the number of times the (row) word occurs in the (column) document.represented as a count vector, a column in Fig.6.3.To review some basic linear algebra, avectoris, at heart, just a list or array ofvectornumbers. SoAs You Like Itis represented as the list [1,114,36,20] (the \ufb01rstcolumnvectorin Fig.6.3) andJulius Caesaris represented as the list [7,62,1,2] (the thirdcolumn vector). Avector spaceis a collection of vectors, characterized by theirvector spacedimension. In the example in Fig.6.3, the document vectors are of dimension 4,dimensionjust so they \ufb01t on the page; in real term-document matrices, the vectors representingeach document would have dimensionality|V|, the vocabulary size.The ordering of the numbers in a vector space indicates different meaningful di-mensions on which documents vary. Thus the \ufb01rst dimension for both these vectorscorresponds to the number of times the wordbattleoccurs, and we can compareeach dimension, noting for example that the vectors forAs You Like ItandTwelfthNighthave similar values (1 and 0, respectively) for the \ufb01rst dimension.As You Like It Twelfth Night Julius Caesar Henry Vbattle1 0 7 13good114 80 62 89fool36 58 1 4wit20 15 2 3Figure 6.3The term-document matrix for four words in four Shakespeare plays. The redboxes show that each document is represented as a column vector of length four.We can think of the vector for a document as a point in|V|-dimensional space;thus the documents in Fig.6.3are points in 4-dimensional space. Since 4-dimensionalspaces are hard to visualize, Fig.6.4shows a visualization in two dimensions; we\u2019vearbitrarily chosen the dimensions corresponding to the wordsbattleandfool.\n51015202530510Henry V [4,13]As You Like It [36,1]Julius Caesar [1,7]battle foolTwelfth Night [58,0]1540\n354045505560Figure 6.4A spatial visualization of the document vectors for the four Shakespeare playdocuments, showing just two of the dimensions, corresponding to the wordsbattleandfool.The comedies have high values for thefooldimension and low values for thebattledimension.Term-document matrices were originally de\ufb01ned as a means of \ufb01nding similardocuments for the task of documentinformation retrieval. Two documents that are Vectors are the basis of information retrieval6.3\u2022WORDS ANDVECTORS7As You Like It Twelfth Night Julius Caesar Henry Vbattle1 0 7 13good114 80 62 89fool36 58 1 4wit20 15 2 3Figure 6.2The term-document matrix for four words in four Shakespeare plays. Each cellcontains the number of times the (row) word occurs in the (column) document.represented as a count vector, a column in Fig.6.3.To review some basic linear algebra, avectoris, at heart, just a list or array ofvectornumbers. SoAs You Like Itis represented as the list [1,114,36,20] (the \ufb01rstcolumnvectorin Fig.6.3) andJulius Caesaris represented as the list [7,62,1,2] (the thirdcolumn vector). Avector spaceis a collection of vectors, characterized by theirvector spacedimension. In the example in Fig.6.3, the document vectors are of dimension 4,dimensionjust so they \ufb01t on the page; in real term-document matrices, the vectors representingeach document would have dimensionality|V|, the vocabulary size.The ordering of the numbers in a vector space indicates different meaningful di-mensions on which documents vary. Thus the \ufb01rst dimension for both these vectorscorresponds to the number of times the wordbattleoccurs, and we can compareeach dimension, noting for example that the vectors forAs You Like ItandTwelfthNighthave similar values (1 and 0, respectively) for the \ufb01rst dimension.As You Like It Twelfth Night Julius Caesar Henry Vbattle1 0 7 13good114 80 62 89fool36 58 1 4wit20 15 2 3Figure 6.3The term-document matrix for four words in four Shakespeare plays. The redboxes show that each document is represented as a column vector of length four.We can think of the vector for a document as a point in|V|-dimensional space;thus the documents in Fig.6.3are points in 4-dimensional space. Since 4-dimensionalspaces are hard to visualize, Fig.6.4shows a visualization in two dimensions; we\u2019vearbitrarily chosen the dimensions corresponding to the wordsbattleandfool.\n51015202530510Henry V [4,13]As You Like It [36,1]Julius Caesar [1,7]battle foolTwelfth Night [58,0]1540\n354045505560Figure 6.4A spatial visualization of the document vectors for the four Shakespeare playdocuments, showing just two of the dimensions, corresponding to the wordsbattleandfool.The comedies have high values for thefooldimension and low values for thebattledimension.Term-document matrices were originally de\ufb01ned as a means of \ufb01nding similardocuments for the task of documentinformation retrieval. Two documents that areVectors are similar for the two comediesBut comedies are different than the other two  Comedies have more fools and wit and fewer battles. Idea for word meaning: Words can be vectors too!!!6.3\u2022WORDS ANDVECTORS7As You Like It Twelfth Night Julius Caesar Henry Vbattle1 0 7 13good114 80 62 89fool36 58 1 4wit20 15 2 3Figure 6.2The term-document matrix for four words in four Shakespeare plays. Each cellcontains the number of times the (row) word occurs in the (column) document.represented as a count vector, a column in Fig.6.3.To review some basic linear algebra, avectoris, at heart, just a list or array ofvectornumbers. SoAs You Like Itis represented as the list [1,114,36,20] (the \ufb01rstcolumnvectorin Fig.6.3) andJulius Caesaris represented as the list [7,62,1,2] (the thirdcolumn vector). Avector spaceis a collection of vectors, characterized by theirvector spacedimension. In the example in Fig.6.3, the document vectors are of dimension 4,dimensionjust so they \ufb01t on the page; in real term-document matrices, the vectors representingeach document would have dimensionality|V|, the vocabulary size.The ordering of the numbers in a vector space indicates different meaningful di-mensions on which documents vary. Thus the \ufb01rst dimension for both these vectorscorresponds to the number of times the wordbattleoccurs, and we can compareeach dimension, noting for example that the vectors forAs You Like ItandTwelfthNighthave similar values (1 and 0, respectively) for the \ufb01rst dimension.As You Like It Twelfth Night Julius Caesar Henry Vbattle1 0 7 13good114 80 62 89fool36 58 1 4wit20 15 2 3Figure 6.3The term-document matrix for four words in four Shakespeare plays. The redboxes show that each document is represented as a column vector of length four.We can think of the vector for a document as a point in|V|-dimensional space;thus the documents in Fig.6.3are points in 4-dimensional space. Since 4-dimensionalspaces are hard to visualize, Fig.6.4shows a visualization in two dimensions; we\u2019vearbitrarily chosen the dimensions corresponding to the wordsbattleandfool.\n51015202530510Henry V [4,13]As You Like It [36,1]Julius Caesar [1,7]battle foolTwelfth Night [58,0]1540\n354045505560Figure 6.4A spatial visualization of the document vectors for the four Shakespeare playdocuments, showing just two of the dimensions, corresponding to the wordsbattleandfool.The comedies have high values for thefooldimension and low values for thebattledimension.Term-document matrices were originally de\ufb01ned as a means of \ufb01nding similardocuments for the task of documentinformation retrieval. Two documents that arebattle is \"the kind of word that occurs in Julius Caesar and Henry V\"fool is \"the kind of word that occurs  in comedies, especially Twelfth Night\"8CHAPTER6\u2022VECTORSEMANTICS ANDEMBEDDINGSsimilar will tend to have similar words, and if two documents have similar wordstheir column vectors will tend to be similar. The vectors for the comediesAs YouLike It[1,114,36,20] andTwelfth Night[0,80,58,15] look a lot more like each other(more fools and wit than battles) than they look likeJulius Caesar[7,62,1,2] orHenry V[13,89,4,3]. This is clear with the raw numbers; in the \ufb01rst dimension(battle) the comedies have low numbers and the others have high numbers, and wecan see it visually in Fig.6.4; we\u2019ll see very shortly how to quantify this intuitionmore formally.A real term-document matrix, of course, wouldn\u2019t just have 4 rows and columns,let alone 2. More generally, the term-document matrix has|V|rows (one for eachword type in the vocabulary) andDcolumns (one for each document in the collec-tion); as we\u2019ll see, vocabulary sizes are generally in the tens of thousands, and thenumber of documents can be enormous (think about all the pages on the web).Information retrieval(IR) is the task of \ufb01nding the documentdfrom theDinformationretrievaldocuments in some collection that best matches a queryq. For IR we\u2019ll therefore alsorepresent a query by a vector, also of length|V|, and we\u2019ll need a way to comparetwo vectors to \ufb01nd how similar they are. (Doing IR will also require ef\ufb01cient waysto store and manipulate these vectors by making use of the convenient fact that thesevectors are sparse, i.e., mostly zeros).Later in the chapter we\u2019ll introduce some of the components of this vector com-parison process: the tf-idf term weighting, and the cosine similarity metric.6.3.2 Words as vectors: document dimensionsWe\u2019ve seen that documents can be represented as vectors in a vector space. Butvector semantics can also be used to represent the meaning ofwords. We do thisby associating each word with a word vector\u2014 arow vectorrather than a columnrow vectorvector, hence with different dimensions, as shown in Fig.6.5. The four dimensionsof the vector forfool, [36,58,1,4], correspond to the four Shakespeare plays. Wordcounts in the same four dimensions are used to form the vectors for the other 3words:wit, [20,15,2,3];battle, [1,0,7,13]; andgood[114,80,62,89].As You Like It Twelfth Night Julius Caesar Henry Vbattle1 0 7 13good114 80 62 89fool36 58 1 4wit20 15 2 3Figure 6.5The term-document matrix for four words in four Shakespeare plays. The redboxes show that each word is represented as a row vector of length four.For documents, we saw that similar documents had similar vectors, because sim-ilar documents tend to have similar words. This same principle applies to words:similar words have similar vectors because they tend to occur in similar documents.The term-document matrix thus lets us represent the meaning of a word by the doc-uments it tends to occur in.6.3.3 Words as vectors: word dimensionsAn alternative to using the term-document matrix to represent words as vectors ofdocument counts, is to use theterm-term matrix, also called theword-word ma-trixor theterm-context matrix, in which the columns are labeled by words ratherword-wordmatrixthan documents. This matrix is thus of dimensionality|V|\u21e5|V|and each cell records More common: word-word matrix(or \"term-context matrix\")Two wordsare similar in meaning if their context vectors are similar6.3\u2022WORDS ANDVECTORS9Information retrieval(IR) is the task of \ufb01nding the documentdfrom theDinformationretrievaldocuments in some collection that best matches a queryq. For IR we\u2019ll therefore alsorepresent a query by a vector, also of length|V|, and we\u2019ll need a way to comparetwo vectors to \ufb01nd how similar they are. (Doing IR will also require ef\ufb01cient waysto store and manipulate these vectors by making use of the convenient fact that thesevectors are sparse, i.e., mostly zeros).Later in the chapter we\u2019ll introduce some of the components of this vector com-parison process: the tf-idf term weighting, and the cosine similarity metric.6.3.2 Words as vectorsWe\u2019ve seen that documents can be represented as vectors in a vector space. Butvector semantics can also be used to represent the meaning ofwords, by associatingeach word with a vector.The word vector is now arow vectorrather than a column vector, and hence therow vectordimensions of the vector are different. The four dimensions of the vector forfool,[36,58,1,4], correspond to the four Shakespeare plays. The same four dimensionsare used to form the vectors for the other 3 words:wit, [20,15,2,3];battle, [1,0,7,13];andgood[114,80,62,89]. Each entry in the vector thus represents the counts of theword\u2019s occurrence in the document corresponding to that dimension.For documents, we saw that similar documents had similar vectors, because sim-ilar documents tend to have similar words. This same principle applies to words:similar words have similar vectors because they tend to occur in similar documents.The term-document matrix thus lets us represent the meaning of a word by the doc-uments it tends to occur in.However, it is most common to use a different kind of context for the dimensionsof a word\u2019s vector representation. Rather than the term-document matrix we use theterm-term matrix, more commonly called theword-word matrixor theterm-word-wordmatrixcontext matrix, in which the columns are labeled by words rather than documents.This matrix is thus of dimensionality|V|\u21e5|V|and each cell records the number oftimes the row (target) word and the column (context) word co-occur in some contextin some training corpus. The context could be the document, in which case the cellrepresents the number of times the two words appear in the same document. It ismost common, however, to use smaller contexts, generally a window around theword, for example of 4 words to the left and 4 words to the right, in which casethe cell represents the number of times (in some training corpus) the column wordoccurs in such a\u00b14 word window around the row word. For example here is oneexample each of some words in their windows:is traditionally followed bycherrypie, a traditional dessertoften mixed, such asstrawberryrhubarb pie. Apple piecomputer peripherals and personaldigitalassistants. These devices usuallya computer. This includesinformationavailable on the internetIf we then take every occurrence of each word (saystrawberry) and count the con-text words around it, we get a word-word co-occurrence matrix. Fig.6.5shows asimpli\ufb01ed subset of the word-word co-occurrence matrix for these four words com-puted from the Wikipedia corpus(Davies, 2015).Note in Fig.6.5that the two wordscherryandstrawberryare more similar toeach other (bothpieandsugartend to occur in their window) than they are to otherwords likedigital; conversely,digitalandinformationare more similar to each otherthan, say, tostrawberry. Fig.6.6shows a spatial visualization.6.3\u2022WORDS ANDVECTORS9the number of times the row (target) word and the column (context) word co-occurin some context in some training corpus. The context could be the document, inwhich case the cell represents the number of times the two words appear in the samedocument. It is most common, however, to use smaller contexts, generally a win-dow around the word, for example of 4 words to the left and 4 words to the right,in which case the cell represents the number of times (in some training corpus) thecolumn word occurs in such a\u00b14 word window around the row word. For examplehere is one example each of some words in their windows:is traditionally followed bycherrypie, a traditional dessertoften mixed, such asstrawberryrhubarb pie. Apple piecomputer peripherals and personaldigitalassistants. These devices usuallya computer. This includesinformationavailable on the internetIf we then take every occurrence of each word (saystrawberry) and count thecontext words around it, we get a word-word co-occurrence matrix. Fig.6.6shows asimpli\ufb01ed subset of the word-word co-occurrence matrix for these four words com-puted from the Wikipedia corpus(Davies, 2015).aardvark...computer data result pie sugar...cherry0 ... 2 8 9 442 25 ...strawberry0 ... 0 0 1 60 19 ...digital0 ... 1670 1683 85 5 4 ...information0 ... 3325 3982 378 5 13 ...Figure 6.6Co-occurrence vectors for four words in the Wikipedia corpus, showing six ofthe dimensions (hand-picked for pedagogical purposes). The vector fordigitalis outlined inred. Note that a real vector would have vastly more dimensions and thus be much sparser.Note in Fig.6.6that the two wordscherryandstrawberryare more similar toeach other (bothpieandsugartend to occur in their window) than they are to otherwords likedigital; conversely,digitalandinformationare more similar to each otherthan, say, tostrawberry. Fig.6.7shows a spatial visualization.\n100020003000400010002000digital [1683,1670]computer datainformation [3982,3325] 30004000\nFigure 6.7A spatial visualization of word vectors fordigitalandinformation, showing justtwo of the dimensions, corresponding to the wordsdataandcomputer.Note that|V|, the length of the vector, is generally the size of the vocabulary, of-ten between 10,000 and 50,000 words (using the most frequent words in the trainingcorpus; keeping words after about the most frequent 50,000 or so is generally nothelpful). Since most of these numbers are zero these aresparsevector representa-tions; there are ef\ufb01cient algorithms for storing and computing with sparse matrices.Now that we have some intuitions, let\u2019s move on to examine the details of com-puting word similarity. Afterwards we\u2019ll discuss methods for weighting cells. 10CHAPTER6\u2022VECTORSEMANTICS ANDEMBEDDINGSaardvark...computer data result pie sugar...cherry0 ... 2 8 9 442 25strawberry0 ... 0 0 1 60 19digital0 ... 1670 1683 85 5 4information0 ... 3325 3982 378 5 13Figure 6.5Co-occurrence vectors for four words in the Wikipedia corpus, showing six ofthe dimensions (hand-picked for pedagogical purposes). The vector fordigitalis outlined inred. Note that a real vector would have vastly more dimensions and thus be much sparser.\n100020003000400010002000digital [1683,1670]computer datainformation [3982,3325] 30004000\nFigure 6.6A spatial visualization of word vectors fordigitalandinformation, showing justtwo of the dimensions, corresponding to the wordsdataandcomputer.Note that|V|, the length of the vector, is generally the size of the vocabulary,usually between 10,000 and 50,000 words (using the most frequent words in thetraining corpus; keeping words after about the most frequent 50,000 or so is gener-ally not helpful). But of course since most of these numbers are zero these aresparsevector representations, and there are ef\ufb01cient algorithms for storing and computingwith sparse matrices.Now that we have some intuitions, let\u2019s move on to examine the details of com-puting word similarity. Afterwards we\u2019ll discuss the tf-idf method of weightingcells.6.4 Cosine for measuring similarityTo de\ufb01ne similarity between two target wordsvandw, we need a measure for takingtwo such vectors and giving a measure of vector similarity. By far the most commonsimilarity metric is thecosineof the angle between the vectors.The cosine\u2014like most measures for vector similarity used in NLP\u2014is based onthedot productoperator from linear algebra, also called theinner product:dot productinner productdot product(v,w)=v\u00b7w=NXi=1viwi=v1w1+v2w2+...+vNwN(6.7)As we will see, most metrics for similarity between vectors are based on the dotproduct. The dot product acts as a similarity metric because it will tend to be highjust when the two vectors have large values in the same dimensions. Alternatively,vectors that have zeros in different dimensions\u2014orthogonal vectors\u2014will have adot product of 0, representing their strong dissimilarity. Vector Semantics & EmbeddingsWords and Vectors Vector Semantics & EmbeddingsCosine for computing word similarity Computing word similarity: Dot product and cosineThe dot product between two vectors is a scalar:The dot product tends to be high when the two vectors have large values in the same dimensionsDot product can thus be a useful similarity metric between vectors10CHAPTER6\u2022VECTORSEMANTICS ANDEMBEDDINGS6.4 Cosine for measuring similarityTo measure similarity between two target wordsvandw, we need a metric thattakes two vectors (of the same dimensionality, either both with words as dimensions,hence of length|V|, or both with documents as dimensions as documents, of length|D|) and gives a measure of their similarity. By far the most common similaritymetric is thecosineof the angle between the vectors.The cosine\u2014like most measures for vector similarity used in NLP\u2014is based onthedot productoperator from linear algebra, also called theinner product:dot productinner productdot product(v,w)=v\u00b7w=NXi=1viwi=v1w1+v2w2+...+vNwN(6.7)As we will see, most metrics for similarity between vectors are based on the dotproduct. The dot product acts as a similarity metric because it will tend to be highjust when the two vectors have large values in the same dimensions. Alternatively,vectors that have zeros in different dimensions\u2014orthogonal vectors\u2014will have adot product of 0, representing their strong dissimilarity.This raw dot product, however, has a problem as a similarity metric: it favorslongvectors. Thevector lengthis de\ufb01ned asvector length|v|=vuutNXi=1v2i(6.8)The dot product is higher if a vector is longer, with higher values in each dimension.More frequent words have longer vectors, since they tend to co-occur with morewords and have higher co-occurrence values with each of them. The raw dot productthus will be higher for frequent words. But this is a problem; we\u2019d like a similaritymetric that tells us how similar two words are regardless of their frequency.We modify the dot product to normalize for the vector length by dividing thedot product by the lengths of each of the two vectors. Thisnormalized dot productturns out to be the same as the cosine of the angle between the two vectors, followingfrom the de\ufb01nition of the dot product between two vectorsaandb:a\u00b7b=|a||b|cosqa\u00b7b|a||b|=cosq(6.9)Thecosinesimilarity metric between two vectorsvandwthus can be computed as:cosinecosine(v,w)=v\u00b7w|v||w|=NXi=1viwivuutNXi=1v2ivuutNXi=1w2i(6.10)For some applications we pre-normalize each vector, by dividing it by its length,creating aunit vectorof length 1. Thus we could compute a unit vector fromabyunit vectordividing it by|a|. For unit vectors, the dot product is the same as the cosine. Problem with raw dot-productDot product favors long vectorsDot product is higher if a vector is longer (has higher values in many dimension)Vector length:Frequent words (of, the, you) have long vectors (since they occur many times with other words).So dot product overly favors frequent words10CHAPTER6\u2022VECTORSEMANTICS ANDEMBEDDINGS6.4 Cosine for measuring similarityTo measure similarity between two target wordsvandw, we need a metric thattakes two vectors (of the same dimensionality, either both with words as dimensions,hence of length|V|, or both with documents as dimensions as documents, of length|D|) and gives a measure of their similarity. By far the most common similaritymetric is thecosineof the angle between the vectors.The cosine\u2014like most measures for vector similarity used in NLP\u2014is based onthedot productoperator from linear algebra, also called theinner product:dot productinner productdot product(v,w)=v\u00b7w=NXi=1viwi=v1w1+v2w2+...+vNwN(6.7)As we will see, most metrics for similarity between vectors are based on the dotproduct. The dot product acts as a similarity metric because it will tend to be highjust when the two vectors have large values in the same dimensions. Alternatively,vectors that have zeros in different dimensions\u2014orthogonal vectors\u2014will have adot product of 0, representing their strong dissimilarity.This raw dot product, however, has a problem as a similarity metric: it favorslongvectors. Thevector lengthis de\ufb01ned asvector length|v|=vuutNXi=1v2i(6.8)The dot product is higher if a vector is longer, with higher values in each dimension.More frequent words have longer vectors, since they tend to co-occur with morewords and have higher co-occurrence values with each of them. The raw dot productthus will be higher for frequent words. But this is a problem; we\u2019d like a similaritymetric that tells us how similar two words are regardless of their frequency.We modify the dot product to normalize for the vector length by dividing thedot product by the lengths of each of the two vectors. Thisnormalized dot productturns out to be the same as the cosine of the angle between the two vectors, followingfrom the de\ufb01nition of the dot product between two vectorsaandb:a\u00b7b=|a||b|cosqa\u00b7b|a||b|=cosq(6.9)Thecosinesimilarity metric between two vectorsvandwthus can be computed as:cosinecosine(v,w)=v\u00b7w|v||w|=NXi=1viwivuutNXi=1v2ivuutNXi=1w2i(6.10)For some applications we pre-normalize each vector, by dividing it by its length,creating aunit vectorof length 1. Thus we could compute a unit vector fromabyunit vectordividing it by|a|. For unit vectors, the dot product is the same as the cosine. Alternative: cosine for computing word similarity12CHAPTER6\u2022VECTORSEMANTICS~a\u00b7~b=|~a||~b|cosq~a\u00b7~b|~a||~b|=cosq(6.9)Thecosinesimilarity metric between two vectors~vand~wthus can be computedcosineas:cosine(~v,~w)=~v\u00b7~w|~v||~w|=NXi=1viwivuutNXi=1v2ivuutNXi=1w2i(6.10)For some applications we pre-normalize each vector, by dividing it by its length,creating aunit vectorof length 1. Thus we could compute a unit vector from~abyunit vectordividing it by|~a|. For unit vectors, the dot product is the same as the cosine.The cosine value ranges from 1 for vectors pointing in the same direction, through0 for vectors that are orthogonal, to -1 for vectors pointing in opposite directions.But raw frequency values are non-negative, so the cosine for these vectors rangesfrom 0\u20131.Let\u2019s see how the cosine computes which of the wordsapricotordigitalis closerin meaning toinformation, just using raw counts from the following simpli\ufb01ed table:large data computerapricot20 0digital01 2information16 1cos(apricot,information)=2+0+0p4+0+0p1+36+1=22p38=.16cos(digital,information)=0+6+2p0+1+4p1+36+1=8p38p5=.58(6.11)The model decides thatinformationis closer todigitalthan it is toapricot,aresult that seems sensible. Fig.6.7shows a visualization.6.5 TF-IDF: Weighing terms in the vectorThe co-occurrence matrix in Fig.6.5represented each cell by the raw frequency ofthe co-occurrence of two words.It turns out, however, that simple frequency isn\u2019t the best measure of associationbetween words. One problem is that raw frequency is very skewed and not verydiscriminative. If we want to know what kinds of contexts are shared byapricotandpineapplebut not bydigitalandinformation, we\u2019re not going to get good discrimi-nation from words likethe,it, orthey, which occur frequently with all sorts of wordsand aren\u2019t informative about any particular word.It\u2019s a bit of a paradox. Word that occur nearby frequently (maybesugarappearsoften in our corpus nearapricot) are more important than words that only appear10CHAPTER6\u2022VECTORSEMANTICS ANDEMBEDDINGS6.4 Cosine for measuring similarityTo measure similarity between two target wordsvandw, we need a metric thattakes two vectors (of the same dimensionality, either both with words as dimensions,hence of length|V|, or both with documents as dimensions as documents, of length|D|) and gives a measure of their similarity. By far the most common similaritymetric is thecosineof the angle between the vectors.The cosine\u2014like most measures for vector similarity used in NLP\u2014is based onthedot productoperator from linear algebra, also called theinner product:dot productinner productdot product(v,w)=v\u00b7w=NXi=1viwi=v1w1+v2w2+...+vNwN(6.7)As we will see, most metrics for similarity between vectors are based on the dotproduct. The dot product acts as a similarity metric because it will tend to be highjust when the two vectors have large values in the same dimensions. Alternatively,vectors that have zeros in different dimensions\u2014orthogonal vectors\u2014will have adot product of 0, representing their strong dissimilarity.This raw dot product, however, has a problem as a similarity metric: it favorslongvectors. Thevector lengthis de\ufb01ned asvector length|v|=vuutNXi=1v2i(6.8)The dot product is higher if a vector is longer, with higher values in each dimension.More frequent words have longer vectors, since they tend to co-occur with morewords and have higher co-occurrence values with each of them. The raw dot productthus will be higher for frequent words. But this is a problem; we\u2019d like a similaritymetric that tells us how similar two words are regardless of their frequency.We modify the dot product to normalize for the vector length by dividing thedot product by the lengths of each of the two vectors. Thisnormalized dot productturns out to be the same as the cosine of the angle between the two vectors, followingfrom the de\ufb01nition of the dot product between two vectorsaandb:a\u00b7b=|a||b|cosqa\u00b7b|a||b|=cosq(6.9)Thecosinesimilarity metric between two vectorsvandwthus can be computed as:cosinecosine(v,w)=v\u00b7w|v||w|=NXi=1viwivuutNXi=1v2ivuutNXi=1w2i(6.10)For some applications we pre-normalize each vector, by dividing it by its length,creating aunit vectorof length 1. Thus we could compute a unit vector fromabyunit vectordividing it by|a|. For unit vectors, the dot product is the same as the cosine.Based on the definition of the dot product between two vectors a and b  Cosine as a similarity metric-1: vectors point in opposite directions +1:  vectors point in same directions0: vectors are orthogonalBut since raw frequency values are non-negative, the cosine for term-term matrix vectors ranges from 0\u20131 46\n Cosine examplespiedatacomputercherry44282digital516831670information539823325\n47cos(\uf072v,\uf072w)=\uf072v\u2022\uf072w\uf072v\uf072w=\uf072v\uf072v\u2022\uf072w\uf072w=viwii=1N\u2211vi2i=1N\u2211wi2i=1N\u22116.4\u2022COSINE FOR MEASURING SIMILARITY11This raw dot product, however, has a problem as a similarity metric: it favorslongvectors. Thevector lengthis de\ufb01ned asvector length|v|=vuutNXi=1v2i(6.8)The dot product is higher if a vector is longer, with higher values in each dimension.More frequent words have longer vectors, since they tend to co-occur with morewords and have higher co-occurrence values with each of them. The raw dot productthus will be higher for frequent words. But this is a problem; we\u2019d like a similaritymetric that tells us how similar two words are regardless of their frequency.The simplest way to modify the dot product to normalize for the vector length isto divide the dot product by the lengths of each of the two vectors. Thisnormalizeddot productturns out to be the same as the cosine of the angle between the twovectors, following from the de\ufb01nition of the dot product between two vectorsaandb:a\u00b7b=|a||b|cosqa\u00b7b|a||b|=cosq(6.9)Thecosinesimilarity metric between two vectorsvandwthus can be computed as:cosinecosine(v,w)=v\u00b7w|v||w|=NXi=1viwivuutNXi=1v2ivuutNXi=1w2i(6.10)For some applications we pre-normalize each vector, by dividing it by its length,creating aunit vectorof length 1. Thus we could compute a unit vector fromabyunit vectordividing it by|a|. For unit vectors, the dot product is the same as the cosine.The cosine value ranges from 1 for vectors pointing in the same direction, through0 for vectors that are orthogonal, to -1 for vectors pointing in opposite directions.But raw frequency values are non-negative, so the cosine for these vectors rangesfrom 0\u20131.Let\u2019s see how the cosine computes which of the wordscherryordigitalis closerin meaning toinformation, just using raw counts from the following shortened table:pie data computercherry442 8 2digital5 1683 1670information5 3982 3325cos(cherry,information)=442\u21e45+8\u21e43982+2\u21e43325p4422+82+22p52+39822+33252=.017cos(digital,information)=5\u21e45+1683\u21e43982+1670\u21e43325p52+16832+16702p52+39822+33252=.996The model decides thatinformationis way closer todigitalthan it is tocherry,aresult that seems sensible. Fig.6.7shows a visualization.6.4\u2022COSINE FOR MEASURING SIMILARITY11This raw dot product, however, has a problem as a similarity metric: it favorslongvectors. Thevector lengthis de\ufb01ned asvector length|v|=vuutNXi=1v2i(6.8)The dot product is higher if a vector is longer, with higher values in each dimension.More frequent words have longer vectors, since they tend to co-occur with morewords and have higher co-occurrence values with each of them. The raw dot productthus will be higher for frequent words. But this is a problem; we\u2019d like a similaritymetric that tells us how similar two words are regardless of their frequency.The simplest way to modify the dot product to normalize for the vector length isto divide the dot product by the lengths of each of the two vectors. Thisnormalizeddot productturns out to be the same as the cosine of the angle between the twovectors, following from the de\ufb01nition of the dot product between two vectorsaandb:a\u00b7b=|a||b|cosqa\u00b7b|a||b|=cosq(6.9)Thecosinesimilarity metric between two vectorsvandwthus can be computed as:cosinecosine(v,w)=v\u00b7w|v||w|=NXi=1viwivuutNXi=1v2ivuutNXi=1w2i(6.10)For some applications we pre-normalize each vector, by dividing it by its length,creating aunit vectorof length 1. Thus we could compute a unit vector fromabyunit vectordividing it by|a|. For unit vectors, the dot product is the same as the cosine.The cosine value ranges from 1 for vectors pointing in the same direction, through0 for vectors that are orthogonal, to -1 for vectors pointing in opposite directions.But raw frequency values are non-negative, so the cosine for these vectors rangesfrom 0\u20131.Let\u2019s see how the cosine computes which of the wordscherryordigitalis closerin meaning toinformation, just using raw counts from the following shortened table:pie data computercherry442 8 2digital5 1683 1670information5 3982 3325cos(cherry,information)=442\u21e45+8\u21e43982+2\u21e43325p4422+82+22p52+39822+33252=.017cos(digital,information)=5\u21e45+1683\u21e43982+1670\u21e43325p52+16832+16702p52+39822+33252=.996The model decides thatinformationis way closer todigitalthan it is tocherry,aresult that seems sensible. Fig.6.7shows a visualization.6.4\u2022COSINE FOR MEASURING SIMILARITY11This raw dot product, however, has a problem as a similarity metric: it favorslongvectors. Thevector lengthis de\ufb01ned asvector length|v|=vuutNXi=1v2i(6.8)The dot product is higher if a vector is longer, with higher values in each dimension.More frequent words have longer vectors, since they tend to co-occur with morewords and have higher co-occurrence values with each of them. The raw dot productthus will be higher for frequent words. But this is a problem; we\u2019d like a similaritymetric that tells us how similar two words are regardless of their frequency.The simplest way to modify the dot product to normalize for the vector length isto divide the dot product by the lengths of each of the two vectors. Thisnormalizeddot productturns out to be the same as the cosine of the angle between the twovectors, following from the de\ufb01nition of the dot product between two vectorsaandb:a\u00b7b=|a||b|cosqa\u00b7b|a||b|=cosq(6.9)Thecosinesimilarity metric between two vectorsvandwthus can be computed as:cosinecosine(v,w)=v\u00b7w|v||w|=NXi=1viwivuutNXi=1v2ivuutNXi=1w2i(6.10)For some applications we pre-normalize each vector, by dividing it by its length,creating aunit vectorof length 1. Thus we could compute a unit vector fromabyunit vectordividing it by|a|. For unit vectors, the dot product is the same as the cosine.The cosine value ranges from 1 for vectors pointing in the same direction, through0 for vectors that are orthogonal, to -1 for vectors pointing in opposite directions.But raw frequency values are non-negative, so the cosine for these vectors rangesfrom 0\u20131.Let\u2019s see how the cosine computes which of the wordscherryordigitalis closerin meaning toinformation, just using raw counts from the following shortened table:pie data computercherry442 8 2digital5 1683 1670information5 3982 3325cos(cherry,information)=442\u21e45+8\u21e43982+2\u21e43325p4422+82+22p52+39822+33252=.017cos(digital,information)=5\u21e45+1683\u21e43982+1670\u21e43325p52+16832+16702p52+39822+33252=.996The model decides thatinformationis way closer todigitalthan it is tocherry,aresult that seems sensible. Fig.6.7shows a visualization.6.4\u2022COSINE FOR MEASURING SIMILARITY11This raw dot product, however, has a problem as a similarity metric: it favorslongvectors. Thevector lengthis de\ufb01ned asvector length|v|=vuutNXi=1v2i(6.8)The dot product is higher if a vector is longer, with higher values in each dimension.More frequent words have longer vectors, since they tend to co-occur with morewords and have higher co-occurrence values with each of them. The raw dot productthus will be higher for frequent words. But this is a problem; we\u2019d like a similaritymetric that tells us how similar two words are regardless of their frequency.The simplest way to modify the dot product to normalize for the vector length isto divide the dot product by the lengths of each of the two vectors. Thisnormalizeddot productturns out to be the same as the cosine of the angle between the twovectors, following from the de\ufb01nition of the dot product between two vectorsaandb:a\u00b7b=|a||b|cosqa\u00b7b|a||b|=cosq(6.9)Thecosinesimilarity metric between two vectorsvandwthus can be computed as:cosinecosine(v,w)=v\u00b7w|v||w|=NXi=1viwivuutNXi=1v2ivuutNXi=1w2i(6.10)For some applications we pre-normalize each vector, by dividing it by its length,creating aunit vectorof length 1. Thus we could compute a unit vector fromabyunit vectordividing it by|a|. For unit vectors, the dot product is the same as the cosine.The cosine value ranges from 1 for vectors pointing in the same direction, through0 for vectors that are orthogonal, to -1 for vectors pointing in opposite directions.But raw frequency values are non-negative, so the cosine for these vectors rangesfrom 0\u20131.Let\u2019s see how the cosine computes which of the wordscherryordigitalis closerin meaning toinformation, just using raw counts from the following shortened table:pie data computercherry442 8 2digital5 1683 1670information5 3982 3325cos(cherry,information)=442\u21e45+8\u21e43982+2\u21e43325p4422+82+22p52+39822+33252=.017cos(digital,information)=5\u21e45+1683\u21e43982+1670\u21e43325p52+16832+16702p52+39822+33252=.996The model decides thatinformationis way closer todigitalthan it is tocherry,aresult that seems sensible. Fig.6.7shows a visualization. Visualizing cosines (well, angles)12CHAPTER6\u2022VECTORSEMANTICS ANDEMBEDDINGS\n50010001500200025003000500digitalcherryinformationDimension 1: \u2018pie\u2019\nDimension 2: \u2018computer\u2019Figure 6.7A (rough) graphical demonstration of cosine similarity, showing vectors forthree words (cherry,digital, andinformation) in the two dimensional space de\ufb01ned by countsof the wordscomputerandpienearby. Note that the angle betweendigitalandinformationissmaller than the angle betweencherryandinformation. When two vectors are more similar,the cosine is larger but the angle is smaller; the cosine has its maximum (1) when the anglebetween two vectors is smallest (0\u0000); the cosine of all other angles is less than 1.6.5 TF-IDF: Weighing terms in the vectorThe co-occurrence matrix in Fig.6.5represented each cell by the raw frequency ofthe co-occurrence of two words.It turns out, however, that simple frequency isn\u2019t the best measure of associationbetween words. One problem is that raw frequency is very skewed and not verydiscriminative. If we want to know what kinds of contexts are shared bycherryandstrawberrybut not bydigitalandinformation, we\u2019re not going to get good discrimi-nation from words likethe,it, orthey, which occur frequently with all sorts of wordsand aren\u2019t informative about any particular word. We saw this also in Fig.6.3forthe Shakespeare corpus; the dimension for the wordgoodis not very discrimina-tive between plays;goodis simply a frequent word and has roughly equivalent highfrequencies in each of the plays.It\u2019s a bit of a paradox. Words that occur nearby frequently (maybepienearbycherry) are more important than words that only appear once or twice. Yet wordsthat are too frequent\u2014ubiquitous, liketheorgood\u2014 are unimportant. How can webalance these two con\ufb02icting constraints?Thetf-idf algorithm(the \u2018-\u2019 here is a hyphen, not a minus sign) is the productof two terms, each term capturing one of these two intuitions:The \ufb01rst is theterm frequency(Luhn, 1957): the frequency of the wordtin theterm frequencydocumentd. We can just use the raw count as the term frequency:tft,d=count(t,d)(6.11)Alternatively we can squash the raw frequency a bit, by using the log10of the fre-quency instead. The intuition is that a word appearing 100 times in a documentdoesn\u2019t make that word 100 times more likely to be relevant to the meaning of thedocument. Because we can\u2019t take the log of 0, we normally add 1 to the count:3tft,d=log10(count(t,d)+1)(6.12)If we use log weighting, terms which occur 10 times in a document would have atf=2, 100 times in a document tf=3, 1000 times tf=4, and so on.3Or we can use this alternative: tft,d=\u21e21+log10count(t,d)if count(t,d)>00 otherwise Vector Semantics & EmbeddingsCosine for computing word similarity Vector Semantics & EmbeddingsTF-IDF But raw frequency is a bad representation\u2022The co-occurrence matrices we have seen represent each cell by word frequencies.\u2022Frequency is clearly useful; if sugarappears a lot near apricot, that's useful information.\u2022But overly frequent words like the, it,or theyare not very informative about the context\u2022It's a paradox! How can we balance these two conflicting constraints?  Two common solutions for word weightingtf-idf:     tf-idfvalue for word t in document d:PMI: (Pointwise mutual information)\u25e6PMI\ud835\udc98\ud835\udfcf,\ud835\udc98\ud835\udfd0=\ud835\udc8d\ud835\udc90\ud835\udc88\ud835\udc91(\ud835\udc98\ud835\udfcf,\ud835\udc98\ud835\udfd0)\ud835\udc91\ud835\udc98\ud835\udfcf\ud835\udc91(\ud835\udc98\ud835\udfd0)14CHAPTER6\u2022VECTORSEMANTICSCollection FrequencyDocument FrequencyRomeo1131action11331We assign importance to these more discriminative words likeRomeoviatheinverse document frequencyoridfterm weight(Sparck Jones, 1972).idfThe idf is de\ufb01ned using the fractionN/dft, whereNis the total number ofdocuments in the collection, and dftis the number of documents in whichtermtoccurs. The fewer documents in which a term occurs, the higher thisweight. The lowest weight of 1 is assigned to terms that occur in all thedocuments. It\u2019s usually clear what counts as a document: in Shakespearewe would use a play; when processing a collection of encyclopedia articleslike Wikipedia, the document is a Wikipedia page; in processing newspaperarticles, the document is a single article. Occasionally your corpus mightnot have appropriate document divisions and you might need to break up thecorpus into documents yourself for the purposes of computing idf.Because of the large number of documents in many collections, this mea-sure is usually squashed with a log function. The resulting de\ufb01nition for in-verse document frequency (idf) is thusidft=log10\u2713Ndft\u25c6(6.12)Here are some idf values for some words in the Shakespeare corpus, rangingfrom extremely informative words which occur in only one play likeRomeo, tothose that occur in a few likesaladorFalstaff, to those which are very common likefoolor so common as to be completely non-discriminative since they occur in all 37plays likegoodorsweet.3WorddfidfRomeo11.57salad21.27Falstaff40.967forest120.489battle210.074fool360.012good370sweet370Thetf-idfweighting of the value for wordtin documentd,wt,dthus combinestf-idfterm frequency with idf:wt,d=tft,d\u21e5idft(6.13)Fig.6.8applies tf-idf weighting to the Shakespeare term-document matrix in Fig.6.2.Note that the tf-idf values for the dimension corresponding to the wordgoodhavenow all become 0; since this word appears in every document, the tf-idf algorithmleads it to be ignored in any comparison of the plays. Similarly, the wordfool, whichappears in 36 out of the 37 plays, has a much lower weight.The tf-idf weighting is by far the dominant way of weighting co-occurrence ma-trices in information retrieval, but also plays a role in many other aspects of natural3Sweetwas one of Shakespeare\u2019s favorite adjectives, a fact probably related to the increased use ofsugar in European recipes around the turn of the 16th century(Jurafsky, 2014, p. 175).Words like \"the\" or \"it\" have very low idfSee if words like \"good\" appear more often with \"great\" than we would expect by chance Term frequency (tf) in the tf-idfalgorithmWe could imagine using raw count:tft,d= count(t,d)But instead of using raw count, we usually squash a bit:12CHAPTER6\u2022VECTORSEMANTICS ANDEMBEDDINGSis not the best measure of association between words. Raw frequency is very skewedand not very discriminative. If we want to know what kinds of contexts are sharedbycherryandstrawberrybut not bydigitalandinformation, we\u2019re not going to getgood discrimination from words likethe,it, orthey, which occur frequently withall sorts of words and aren\u2019t informative about any particular word. We saw thisalso in Fig.6.3for the Shakespeare corpus; the dimension for the wordgoodis notvery discriminative between plays;goodis simply a frequent word and has roughlyequivalent high frequencies in each of the plays.It\u2019s a bit of a paradox. Words that occur nearby frequently (maybepienearbycherry) are more important than words that only appear once or twice. Yet wordsthat are too frequent\u2014ubiquitous, liketheorgood\u2014 are unimportant. How can webalance these two con\ufb02icting constraints?There are two common solutions to this problem: in this section we\u2019ll describethetf-idfweighting, usually used when the dimensions are documents. In the nextwe introduce thePPMIalgorithm (usually used when the dimensions are words).Thetf-idf weighting(the \u2018-\u2019 here is a hyphen, not a minus sign) is the productof two terms, each term capturing one of these two intuitions:The \ufb01rst is theterm frequency(Luhn,1957): the frequency of the wordtin theterm frequencydocumentd. We can just use the raw count as the term frequency:tft,d=count(t,d)(6.11)More commonly we squash the raw frequency a bit, by using the log10of the fre-quency instead. The intuition is that a word appearing 100 times in a documentdoesn\u2019t make that word 100 times more likely to be relevant to the meaning of thedocument. We also need to do something special with counts of 0, since we can\u2019ttake the log of 0.2tft,d=(1+log10count(t,d)if count(t,d)>00 otherwise(6.12)If we use log weighting, terms which occur 0 times in a document would have tf=0,1 times in a document tf=1+log10(1)=1+0=1, 10 times in a document tf=1+log10(10)=2, 100 times tf=1+log10(100)=3, 1000 times tf=4, and so on.The second factor in tf-idf is used to give a higher weight to words that occuronly in a few documents. Terms that are limited to a few documents are usefulfor discriminating those documents from the rest of the collection; terms that occurfrequently across the entire collection aren\u2019t as helpful. Thedocument frequencydocumentfrequencydftof a termtis the number of documents it occurs in. Document frequency isnot the same as thecollection frequencyof a term, which is the total number oftimes the word appears in the whole collection in any document. Consider in thecollection of Shakespeare\u2019s 37 plays the two wordsRomeoandaction. The wordshave identical collection frequencies (they both occur 113 times in all the plays) butvery different document frequencies, since Romeo only occurs in a single play. Ifour goal is to \ufb01nd documents about the romantic tribulations of Romeo, the wordRomeoshould be highly weighted, but notaction:Collection FrequencyDocument FrequencyRomeo1131action113312We can also use this alternative formulation, which we have used in earlier editions: tft,d=log10(count(t,d)+1) Document frequency (df)dftis the number of documents toccurs in.(note this is not collection frequency: total count across all documents)\"Romeo\" is very distinctive for one Shakespeare play:12CHAPTER6\u2022VECTORSEMANTICS ANDEMBEDDINGSthat are too frequent\u2014ubiquitous, liketheorgood\u2014 are unimportant. How can webalance these two con\ufb02icting constraints?There are two common solutions to this problem: in this section we\u2019ll describethetf-idfalgorithm, usually used when the dimensions are documents. In the nextwe introduce thePPMIalgorithm (usually used when the dimensions are words).Thetf-idf algorithm(the \u2018-\u2019 here is a hyphen, not a minus sign) is the productof two terms, each term capturing one of these two intuitions:The \ufb01rst is theterm frequency(Luhn, 1957): the frequency of the wordtin theterm frequencydocumentd. We can just use the raw count as the term frequency:tft,d=count(t,d)(6.11)More commonly we squash the raw frequency a bit, by using the log10of the fre-quency instead. The intuition is that a word appearing 100 times in a documentdoesn\u2019t make that word 100 times more likely to be relevant to the meaning of thedocument. Because we can\u2019t take the log of 0, we normally add 1 to the count:2tft,d=log10(count(t,d)+1)(6.12)If we use log weighting, terms which occur 0 times in a document would havetf=log10(1)=0, 10 times in a document tf=log10(11)=1.4, 100 times tf=log10(101)=2.004, 1000 times tf=3.00044, and so on.The second factor in tf-idf is used to give a higher weight to words that occuronly in a few documents. Terms that are limited to a few documents are usefulfor discriminating those documents from the rest of the collection; terms that occurfrequently across the entire collection aren\u2019t as helpful. Thedocument frequencydocumentfrequencydftof a termtis the number of documents it occurs in. Document frequency isnot the same as thecollection frequencyof a term, which is the total number oftimes the word appears in the whole collection in any document. Consider in thecollection of Shakespeare\u2019s 37 plays the two wordsRomeoandaction. The wordshave identical collection frequencies (they both occur 113 times in all the plays) butvery different document frequencies, since Romeo only occurs in a single play. Ifour goal is to \ufb01nd documents about the romantic tribulations of Romeo, the wordRomeoshould be highly weighted, but notaction:Collection FrequencyDocument FrequencyRomeo1131action11331We emphasize discriminative words likeRomeovia theinverse document fre-quencyoridfterm weight(Sparck Jones, 1972). The idf is de\ufb01ned using the frac-idftionN/dft, whereNis the total number of documents in the collection, and dftisthe number of documents in which termtoccurs. The fewer documents in which aterm occurs, the higher this weight. The lowest weight of 1 is assigned to terms thatoccur in all the documents. It\u2019s usually clear what counts as a document: in Shake-speare we would use a play; when processing a collection of encyclopedia articleslike Wikipedia, the document is a Wikipedia page; in processing newspaper articles,the document is a single article. Occasionally your corpus might not have appropri-ate document divisions and you might need to break up the corpus into documentsyourself for the purposes of computing idf.2Or we can use this alternative: tft,d=\u21e21+log10count(t,d)if count(t,d)>00 otherwise Inverse document frequency (idf)6.5\u2022TF-IDF: WEIGHING TERMS IN THE VECTOR13Because of the large number of documents in many collections, this measuretoo is usually squashed with a log function. The resulting de\ufb01nition for inversedocument frequency (idf) is thusidft=log10\u2713Ndft\u25c6(6.13)Here are some idf values for some words in the Shakespeare corpus, ranging fromextremely informative words which occur in only one play likeRomeo, to those thatoccur in a few likesaladorFalstaff, to those which are very common likefoolor socommon as to be completely non-discriminative since they occur in all 37 plays likegoodorsweet.3WorddfidfRomeo11.57salad21.27Falstaff40.967forest120.489battle210.246wit340.037fool360.012good370sweet370Thetf-idfweighted valuewt,dfor wordtin documentdthus combines termtf-idffrequency tft,d(de\ufb01ned either by Eq.6.11or by Eq.6.12) with idf from Eq.6.13:wt,d=tft,d\u21e5idft(6.14)Fig.6.9applies tf-idf weighting to the Shakespeare term-document matrix in Fig.6.2,using the tf equation Eq.6.12. Note that the tf-idf values for the dimension corre-sponding to the wordgoodhave now all become 0; since this word appears in everydocument, the tf-idf algorithm leads it to be ignored. Similarly, the wordfool, whichappears in 36 out of the 37 plays, has a much lower weight.As You Like It Twelfth Night Julius Caesar Henry Vbattle0.074 0 0.22 0.28good00 0 0fool0.019 0.021 0.0036 0.0083wit0.049 0.044 0.018 0.022Figure 6.9A tf-idf weighted term-document matrix for four words in four Shakespeareplays, using the counts in Fig.6.2. For example the 0.049 value forwitinAs You Like Itisthe product of tf=log10(20+1)=1.322 and idf=.037. Note that the idf weighting haseliminated the importance of the ubiquitous wordgoodand vastly reduced the impact of thealmost-ubiquitous wordfool.The tf-idf weighting is the way for weighting co-occurrence matrices in infor-mation retrieval, but also plays a role in many other aspects of natural languageprocessing. It\u2019s also a great baseline, the simple thing to try \ufb01rst. We\u2019ll look at otherweightings like PPMI (Positive Pointwise Mutual Information) in Section6.6.3Sweetwas one of Shakespeare\u2019s favorite adjectives, a fact probably related to the increased use ofsugar in European recipes around the turn of the 16th century(Jurafsky, 2014, p. 175).6.5\u2022TF-IDF: WEIGHING TERMS IN THE VECTOR13Because of the large number of documents in many collections, this measuretoo is usually squashed with a log function. The resulting de\ufb01nition for inversedocument frequency (idf) is thusidft=log10\u2713Ndft\u25c6(6.13)Here are some idf values for some words in the Shakespeare corpus, ranging fromextremely informative words which occur in only one play likeRomeo, to those thatoccur in a few likesaladorFalstaff, to those which are very common likefoolor socommon as to be completely non-discriminative since they occur in all 37 plays likegoodorsweet.3WorddfidfRomeo11.57salad21.27Falstaff40.967forest120.489battle210.246wit340.037fool360.012good370sweet370Thetf-idfweighted valuewt,dfor wordtin documentdthus combines termtf-idffrequency tft,d(de\ufb01ned either by Eq.6.11or by Eq.6.12) with idf from Eq.6.13:wt,d=tft,d\u21e5idft(6.14)Fig.6.9applies tf-idf weighting to the Shakespeare term-document matrix in Fig.6.2,using the tf equation Eq.6.12. Note that the tf-idf values for the dimension corre-sponding to the wordgoodhave now all become 0; since this word appears in everydocument, the tf-idf algorithm leads it to be ignored. Similarly, the wordfool, whichappears in 36 out of the 37 plays, has a much lower weight.As You Like It Twelfth Night Julius Caesar Henry Vbattle0.074 0 0.22 0.28good00 0 0fool0.019 0.021 0.0036 0.0083wit0.049 0.044 0.018 0.022Figure 6.9A tf-idf weighted term-document matrix for four words in four Shakespeareplays, using the counts in Fig.6.2. For example the 0.049 value forwitinAs You Like Itisthe product of tf=log10(20+1)=1.322 and idf=.037. Note that the idf weighting haseliminated the importance of the ubiquitous wordgoodand vastly reduced the impact of thealmost-ubiquitous wordfool.The tf-idf weighting is the way for weighting co-occurrence matrices in infor-mation retrieval, but also plays a role in many other aspects of natural languageprocessing. It\u2019s also a great baseline, the simple thing to try \ufb01rst. We\u2019ll look at otherweightings like PPMI (Positive Pointwise Mutual Information) in Section6.6.3Sweetwas one of Shakespeare\u2019s favorite adjectives, a fact probably related to the increased use ofsugar in European recipes around the turn of the 16th century(Jurafsky, 2014, p. 175).N is the total number of documents in the collection What is a document?Could be a play or a Wikipedia articleBut for the purposes of tf-idf, documents can be anything; we often call each paragraph a document! Final tf-idfweighted value for a wordRaw counts:tf-idf:6.5\u2022TF-IDF: WEIGHING TERMS IN THE VECTOR13Because of the large number of documents in many collections, this measuretoo is usually squashed with a log function. The resulting de\ufb01nition for inversedocument frequency (idf) is thusidft=log10\u2713Ndft\u25c6(6.13)Here are some idf values for some words in the Shakespeare corpus, ranging fromextremely informative words which occur in only one play likeRomeo, to those thatoccur in a few likesaladorFalstaff, to those which are very common likefoolor socommon as to be completely non-discriminative since they occur in all 37 plays likegoodorsweet.3WorddfidfRomeo11.57salad21.27Falstaff40.967forest120.489battle210.246wit340.037fool360.012good370sweet370Thetf-idfweighted valuewt,dfor wordtin documentdthus combines termtf-idffrequency tft,d(de\ufb01ned either by Eq.6.11or by Eq.6.12) with idf from Eq.6.13:wt,d=tft,d\u21e5idft(6.14)Fig.6.9applies tf-idf weighting to the Shakespeare term-document matrix in Fig.6.2,using the tf equation Eq.6.12. Note that the tf-idf values for the dimension corre-sponding to the wordgoodhave now all become 0; since this word appears in everydocument, the tf-idf algorithm leads it to be ignored. Similarly, the wordfool, whichappears in 36 out of the 37 plays, has a much lower weight.As You Like It Twelfth Night Julius Caesar Henry Vbattle0.074 0 0.22 0.28good00 0 0fool0.019 0.021 0.0036 0.0083wit0.049 0.044 0.018 0.022Figure 6.9A tf-idf weighted term-document matrix for four words in four Shakespeareplays, using the counts in Fig.6.2. For example the 0.049 value forwitinAs You Like Itisthe product of tf=log10(20+1)=1.322 and idf=.037. Note that the idf weighting haseliminated the importance of the ubiquitous wordgoodand vastly reduced the impact of thealmost-ubiquitous wordfool.The tf-idf weighting is the way for weighting co-occurrence matrices in infor-mation retrieval, but also plays a role in many other aspects of natural languageprocessing. It\u2019s also a great baseline, the simple thing to try \ufb01rst. We\u2019ll look at otherweightings like PPMI (Positive Pointwise Mutual Information) in Section6.6.3Sweetwas one of Shakespeare\u2019s favorite adjectives, a fact probably related to the increased use ofsugar in European recipes around the turn of the 16th century(Jurafsky, 2014, p. 175).6.3\u2022WORDS ANDVECTORS7As You Like It Twelfth Night Julius Caesar Henry Vbattle1 0 7 13good114 80 62 89fool36 58 1 4wit20 15 2 3Figure 6.2The term-document matrix for four words in four Shakespeare plays. Each cellcontains the number of times the (row) word occurs in the (column) document.represented as a count vector, a column in Fig.6.3.To review some basic linear algebra, avectoris, at heart, just a list or array ofvectornumbers. SoAs You Like Itis represented as the list [1,114,36,20] (the \ufb01rstcolumnvectorin Fig.6.3) andJulius Caesaris represented as the list [7,62,1,2] (the thirdcolumn vector). Avector spaceis a collection of vectors, characterized by theirvector spacedimension. In the example in Fig.6.3, the document vectors are of dimension 4,dimensionjust so they \ufb01t on the page; in real term-document matrices, the vectors representingeach document would have dimensionality|V|, the vocabulary size.The ordering of the numbers in a vector space indicates different meaningful di-mensions on which documents vary. Thus the \ufb01rst dimension for both these vectorscorresponds to the number of times the wordbattleoccurs, and we can compareeach dimension, noting for example that the vectors forAs You Like ItandTwelfthNighthave similar values (1 and 0, respectively) for the \ufb01rst dimension.As You Like It Twelfth Night Julius Caesar Henry Vbattle1 0 7 13good114 80 62 89fool36 58 1 4wit20 15 2 3Figure 6.3The term-document matrix for four words in four Shakespeare plays. The redboxes show that each document is represented as a column vector of length four.We can think of the vector for a document as a point in|V|-dimensional space;thus the documents in Fig.6.3are points in 4-dimensional space. Since 4-dimensionalspaces are hard to visualize, Fig.6.4shows a visualization in two dimensions; we\u2019vearbitrarily chosen the dimensions corresponding to the wordsbattleandfool.\n51015202530510Henry V [4,13]As You Like It [36,1]Julius Caesar [1,7]battle foolTwelfth Night [58,0]1540\n354045505560Figure 6.4A spatial visualization of the document vectors for the four Shakespeare playdocuments, showing just two of the dimensions, corresponding to the wordsbattleandfool.The comedies have high values for thefooldimension and low values for thebattledimension.Term-document matrices were originally de\ufb01ned as a means of \ufb01nding similardocuments for the task of documentinformation retrieval. Two documents that are14CHAPTER6\u2022VECTORSEMANTICS ANDEMBEDDINGSAs You Like It Twelfth Night Julius Caesar Henry Vbattle0.246 0 0.454 0.520good00 0 0fool0.030 0.033 0.0012 0.0019wit0.085 0.081 0.048 0.054Figure 6.9A tf-idf weighted term-document matrix for four words in four Shakespeareplays, using the counts in Fig.6.2. For example the 0.085 value forwitinAs You Like Itisthe product of tf=1+log10(20)=2.301 and idf=.037. Note that the idf weighting haseliminated the importance of the ubiquitous wordgoodand vastly reduced the impact of thealmost-ubiquitous wordfool.6.6 Pointwise Mutual Information (PMI)An alternative weighting function to tf-idf, PPMI (positive pointwise mutual infor-mation), is used for term-term-matrices, when the vector dimensions correspond towords rather than documents. PPMI draws on the intuition that the best way to weighthe association between two words is to ask how muchmorethe two words co-occurin our corpus than we would have a priori expected them to appear by chance.Pointwise mutual information(Fano,1961)4is one of the most important con-pointwisemutualinformationcepts in NLP. It is a measure of how often two eventsxandyoccur, compared withwhat we would expect if they were independent:I(x,y)=log2P(x,y)P(x)P(y)(6.16)The pointwise mutual information between a target wordwand a context wordc(Church and Hanks1989,Church and Hanks1990) is then de\ufb01ned as:PMI(w,c)=log2P(w,c)P(w)P(c)(6.17)The numerator tells us how often we observed the two words together (assumingwe compute probability by using the MLE). The denominator tells us how oftenwe wouldexpectthe two words to co-occur assuming they each occurred indepen-dently; recall that the probability of two independent events both occurring is justthe product of the probabilities of the two events. Thus, the ratio gives us an esti-mate of how much more the two words co-occur than we expect by chance. PMI isa useful tool whenever we need to \ufb01nd words that are strongly associated.PMI values range from negative to positive in\ufb01nity. But negative PMI values(which imply things are co-occurringless oftenthan we would expect by chance)tend to be unreliable unless our corpora are enormous. To distinguish whethertwo words whose individual probability is each 10\u00006occur together less often thanchance, we would need to be certain that the probability of the two occurring to-gether is signi\ufb01cantly less than 10\u000012, and this kind of granularity would require anenormous corpus. Furthermore it\u2019s not clear whether it\u2019s even possible to evaluatesuch scores of \u2018unrelatedness\u2019 with human judgments. For this reason it is more4PMI is based on themutual informationbetween two random variablesXandY, de\ufb01ned as:I(X,Y)=XxXyP(x,y)log2P(x,y)P(x)P(y)(6.15)In a confusion of terminology, Fano used the phrasemutual informationto refer to what we now callpointwise mutual informationand the phraseexpectation of the mutual informationfor what we now callmutual information Vector Semantics & EmbeddingsTF-IDF Vector Semantics & EmbeddingsWord2vec Sparse versus dense vectorstf-idf(or PMI) vectors are\u25e6long(length |V|= 20,000 to 50,000)\u25e6sparse (most elements are zero)Alternative: learn vectors which are\u25e6short(length 50-1000)\u25e6dense(most elements are non-zero) Sparse versus dense vectorsWhy dense vectors?\u25e6Short vectors may be easier to use as featuresin machine learning (fewer weights to tune)\u25e6Dense vectors may generalizebetter than explicit counts\u25e6Dense vectors may do better at capturing synonymy:\u25e6carand automobileare synonyms; but are distinct dimensions\u25e6a word with caras a neighbor and a word with automobileas a neighbor should be similar, but aren't\u25e6In practice, they work better61 Common methods for getting short dense vectors\u201cNeural Language Model\u201d-inspired models\u25e6Word2vec (skipgram, CBOW), GloVeSingular Value Decomposition (SVD)\u25e6A special case of this is called LSA \u2013Latent Semantic AnalysisAlternative to these \"static embeddings\":\u2022Contextual Embeddings (ELMo, BERT)\u2022Compute distinct embeddings for a word in its context\u2022Separate embeddings for each token of a word Simple static embeddings you can download!Word2vec (Mikolovet al)https://code.google.com/archive/p/word2vec/GloVe(Pennington, Socher, Manning)http://nlp.stanford.edu/projects/glove/ Word2vecPopular embedding methodVery fast to trainCode available on the webIdea: predictrather than countWord2vec provides various options. We'll do:skip-gram with negative sampling (SGNS) Word2vecInstead of countinghow often each word woccurs near \"apricot\"\u25e6Train a classifier on a binary predictiontask:\u25e6Is w likely to show up near \"apricot\"?We don\u2019t actually care about this task\u25e6But we'll take the learned classifier weights as the word embeddingsBig idea:  self-supervision: \u25e6A word c that occurs near apricot in the corpus cats as the gold \"correct answer\" for supervised learning\u25e6No need for human labels\u25e6Bengioet al. (2003); Collobertet al. (2011)  Approach: predict if candidate word cis a \"neighbor\"1.Treat the target word tand a neighboring context word cas positive examples.2.Randomly sample other words in the lexicon to get negative examples3.Use logistic regression to train a classifier to distinguish those two cases4.Use the learned weights as the embeddings Skip-Gram Training DataAssume a +/-2 word window, given training sentence:\u2026lemon, a [tablespoon of  apricot  jam,   a]  pinch\u2026c1                   c2 c3      c4                                 [target] Skip-Gram Classifier(assuming a +/-2 word window)\u2026lemon, a [tablespoon of  apricot  jam,   a]  pinch\u2026c1                   c2 [target]c3      c4Goal: train a classifier that is given a candidate (word, context) pair(apricot, jam)(apricot, aardvark)\u2026And assigns each pair a probability:P(+|w, c) P(\u2212|w, c) = 1 \u2212 P(+|w, c)  Similarity is computed from dot productRemember: two vectors are similar if they have a high dot product\u25e6Cosine is just a normalized dot productSo:\u25e6Similarity(w,c)  \u221dw\u00b7 cWe\u2019ll need to normalize to get a probability \u25e6(cosine isn't a probability either)69 Turning dot products into probabilitiesSim(w,c) \u2248 w\u00b7 cTo turn this into a probability We'll use the sigmoid from logistic regression:6.8\u2022WORD2VEC19We model the probability that wordcis a real context word for target wordwas:P(+|w,c)=s(c\u00b7w)=11+exp(\u0000c\u00b7w)(6.28)The sigmoid function returns a number between 0 and 1, but to make it a probabilitywe\u2019ll also need the total probability of the two possible events (cis a context word,andcisn\u2019t a context word) to sum to 1. We thus estimate the probability that wordcis not a real context word forwas:P(\u0000|w,c)=1\u0000P(+|w,c)=s(\u0000c\u00b7w)=11+exp(c\u00b7w)(6.29)Equation6.28gives us the probability for one word, but there are many contextwords in the window. Skip-gram makes the simplifying assumption that all contextwords are independent, allowing us to just multiply their probabilities:P(+|w,c1:L)=LYi=1s(\u0000ci\u00b7w)(6.30)logP(+|w,c1:L)=LXi=1logs(\u0000ci\u00b7w)(6.31)In summary, skip-gram trains a probabilistic classi\ufb01er that, given a test target wordwand its context window ofLwordsc1:L, assigns a probability based on how similarthis context window is to the target word. The probability is based on applying thelogistic (sigmoid) function to the dot product of the embeddings of the target wordwith each context word. To compute this probability, we just need embeddings foreach target word and context word in the vocabulary.1WCaardvark\nzebrazebraaardvarkapricotapricot|V||V|+12V& =target wordscontext & noisewords\u2026\n\u20261..d\u2026\n\u2026Figure 6.13The embeddings learned by the skipgram model. The algorithm stores twoembeddings for each word, the target embedding (sometimes called the input embedding)and the context embedding (sometimes called the output embedding). The parameterqthatthe algorithm learns is thus a matrix of 2|V|vectors, each of dimensiond, formed by concate-nating two matrices, the target embeddingsWand the context+noise embeddingsC.Fig.6.13shows the intuition of the parameters we\u2019ll need. Skip-gram actuallystores two embeddings for each word, one for the word as a target, and one for the6.8\u2022WORD2VEC19We model the probability that wordcis a real context word for target wordwas:P(+|w,c)=s(c\u00b7w)=11+exp(\u0000c\u00b7w)(6.28)The sigmoid function returns a number between 0 and 1, but to make it a probabilitywe\u2019ll also need the total probability of the two possible events (cis a context word,andcisn\u2019t a context word) to sum to 1. We thus estimate the probability that wordcis not a real context word forwas:P(\u0000|w,c)=1\u0000P(+|w,c)=s(\u0000c\u00b7w)=11+exp(c\u00b7w)(6.29)Equation6.28gives us the probability for one word, but there are many contextwords in the window. Skip-gram makes the simplifying assumption that all contextwords are independent, allowing us to just multiply their probabilities:P(+|w,c1:L)=LYi=1s(\u0000ci\u00b7w)(6.30)logP(+|w,c1:L)=LXi=1logs(\u0000ci\u00b7w)(6.31)In summary, skip-gram trains a probabilistic classi\ufb01er that, given a test target wordwand its context window ofLwordsc1:L, assigns a probability based on how similarthis context window is to the target word. The probability is based on applying thelogistic (sigmoid) function to the dot product of the embeddings of the target wordwith each context word. To compute this probability, we just need embeddings foreach target word and context word in the vocabulary.1WCaardvark\nzebrazebraaardvarkapricotapricot|V||V|+12V& =target wordscontext & noisewords\u2026\n\u20261..d\u2026\n\u2026Figure 6.13The embeddings learned by the skipgram model. The algorithm stores twoembeddings for each word, the target embedding (sometimes called the input embedding)and the context embedding (sometimes called the output embedding). The parameterqthatthe algorithm learns is thus a matrix of 2|V|vectors, each of dimensiond, formed by concate-nating two matrices, the target embeddingsWand the context+noise embeddingsC.Fig.6.13shows the intuition of the parameters we\u2019ll need. Skip-gram actuallystores two embeddings for each word, one for the word as a target, and one for the How Skip-Gram Classifier computes P(+|w, c) This is for one context word, but we have lots of context words.We'll assume independence and just multiply them:6.8\u2022WORD2VEC19We model the probability that wordcis a real context word for target wordwas:P(+|w,c)=s(c\u00b7w)=11+exp(\u0000c\u00b7w)(6.28)The sigmoid function returns a number between 0 and 1, but to make it a probabilitywe\u2019ll also need the total probability of the two possible events (cis a context word,andcisn\u2019t a context word) to sum to 1. We thus estimate the probability that wordcis not a real context word forwas:P(\u0000|w,c)=1\u0000P(+|w,c)=s(\u0000c\u00b7w)=11+exp(c\u00b7w)(6.29)Equation6.28gives us the probability for one word, but there are many contextwords in the window. Skip-gram makes the simplifying assumption that all contextwords are independent, allowing us to just multiply their probabilities:P(+|w,c1:L)=LYi=1s(\u0000ci\u00b7w)(6.30)logP(+|w,c1:L)=LXi=1logs(\u0000ci\u00b7w)(6.31)In summary, skip-gram trains a probabilistic classi\ufb01er that, given a test target wordwand its context window ofLwordsc1:L, assigns a probability based on how similarthis context window is to the target word. The probability is based on applying thelogistic (sigmoid) function to the dot product of the embeddings of the target wordwith each context word. To compute this probability, we just need embeddings foreach target word and context word in the vocabulary.1WCaardvark\nzebrazebraaardvarkapricotapricot|V||V|+12V& =target wordscontext & noisewords\u2026\n\u20261..d\u2026\n\u2026Figure 6.13The embeddings learned by the skipgram model. The algorithm stores twoembeddings for each word, the target embedding (sometimes called the input embedding)and the context embedding (sometimes called the output embedding). The parameterqthatthe algorithm learns is thus a matrix of 2|V|vectors, each of dimensiond, formed by concate-nating two matrices, the target embeddingsWand the context+noise embeddingsC.Fig.6.13shows the intuition of the parameters we\u2019ll need. Skip-gram actuallystores two embeddings for each word, one for the word as a target, and one for the6.8\u2022WORD2VEC19We model the probability that wordcis a real context word for target wordwas:P(+|w,c)=s(c\u00b7w)=11+exp(\u0000c\u00b7w)(6.28)The sigmoid function returns a number between 0 and 1, but to make it a probabilitywe\u2019ll also need the total probability of the two possible events (cis a context word,andcisn\u2019t a context word) to sum to 1. We thus estimate the probability that wordcis not a real context word forwas:P(\u0000|w,c)=1\u0000P(+|w,c)=s(\u0000c\u00b7w)=11+exp(c\u00b7w)(6.29)Equation6.28gives us the probability for one word, but there are many contextwords in the window. Skip-gram makes the simplifying assumption that all contextwords are independent, allowing us to just multiply their probabilities:P(+|w,c1:L)=LYi=1s(ci\u00b7w)(6.30)logP(+|w,c1:L)=LXi=1logs(ci\u00b7w)(6.31)In summary, skip-gram trains a probabilistic classi\ufb01er that, given a test target wordwand its context window ofLwordsc1:L, assigns a probability based on how similarthis context window is to the target word. The probability is based on applying thelogistic (sigmoid) function to the dot product of the embeddings of the target wordwith each context word. To compute this probability, we just need embeddings foreach target word and context word in the vocabulary.1WCaardvark\nzebrazebraaardvarkapricotapricot|V||V|+12V& =target wordscontext & noisewords\u2026\n\u20261..d\u2026\n\u2026Figure 6.13The embeddings learned by the skipgram model. The algorithm stores twoembeddings for each word, the target embedding (sometimes called the input embedding)and the context embedding (sometimes called the output embedding). The parameterqthatthe algorithm learns is thus a matrix of 2|V|vectors, each of dimensiond, formed by concate-nating two matrices, the target embeddingsWand the context+noise embeddingsC.Fig.6.13shows the intuition of the parameters we\u2019ll need. Skip-gram actuallystores two embeddings for each word, one for the word as a target, and one for the Skip-gram classifier: summaryA probabilistic classifier, given \u2022a test target word w \u2022its context window of L words c1:LEstimates probability that w occurs in this window based on similarity of w (embeddings) to c1:L(embeddings).To compute this, we just need embeddings for all the words. These embeddings we'll need: a set for w, a set for c1WCaardvark\nzebrazebraaardvarkapricotapricot|V||V|+12V& =target wordscontext & noisewords\u2026\n\u20261..d\u2026\n\u2026 Vector Semantics & EmbeddingsWord2vec Vector Semantics & EmbeddingsWord2vec: Learning the embeddings Skip-Gram Training data\u2026lemon, a [tablespoon of  apricot  jam,   a]  pinch\u2026c1                   c2 [target]c3      c4\n7620CHAPTER6\u2022VECTORSEMANTICS ANDEMBEDDINGSthis context window is to the target word. The probability is based on applying thelogistic (sigmoid) function to the dot product of the embeddings of the target wordwith each context word. We could thus compute this probability if only we hadembeddings for each target word and context word in the vocabulary. Let\u2019s now turnto learning these embeddings (which is the real goal of training this classi\ufb01er in the\ufb01rst place).6.8.2 Learning skip-gram embeddingsWord2vec learns embeddings by starting with an initial set of embedding vectorsand then iteratively shifting the embedding of each wordwto be more like the em-beddings of words that occur nearby in texts, and less like the embeddings of wordsthat don\u2019t occur nearby. Let\u2019s start by considering a single piece of training data:... lemon, a [tablespoon of apricot jam, a] pinch ...c1 c2 t c3 c4This example has a target wordt(apricot), and 4 context words in theL=\u00b12window, resulting in 4 positive training instances (on the left below):positive examples +tcapricot tablespoonapricot ofapricot jamapricot anegative examples -tc tcapricot aardvark apricot sevenapricot my apricot foreverapricot where apricot dearapricot coaxial apricot ifFor training a binary classi\ufb01er we also need negative examples. In fact skip-gram uses more negative examples than positive examples (with the ratio betweenthem set by a parameterk). So for each of these(t,c)training instances we\u2019ll createknegative samples, each consisting of the targettplus a \u2018noise word\u2019. A noise wordis a random word from the lexicon, constrained not to be the target wordt. Theright above shows the setting wherek=2, so we\u2019ll have 2 negative examples in thenegative training set\u0000for each positive examplet,c.The noise words are chosen according to their weighted unigram frequencypa(w), whereais a weight. If we were sampling according to unweighted fre-quencyp(w), it would mean that with unigram probabilityp(\u201cthe\u201d)we would choosethe wordtheas a noise word, with unigram probabilityp(\u201caardvark\u201d)we wouldchooseaardvark, and so on. But in practice it is common to seta=.75, i.e. use theweightingp34(w):Pa(w)=count(w)aPw0count(w0)a(6.32)Settinga=.75 gives better performance because it gives rare noise words slightlyhigher probability: for rare words,Pa(w)>P(w). To visualize this intuition, itmight help to work out the probabilities for an example with two events,P(a)=.99andP(b)=.01:Pa(a)=.99.75.99.75+.01.75=.97Pa(b)=.01.75.99.75+.01.75=.03(6.33) Skip-Gram Training data\u2026lemon, a [tablespoon of  apricot  jam,   a]  pinch\u2026c1                   c2 [target]c3      c4\n77For each positive example we'll grab k negative examples, sampling by frequency20CHAPTER6\u2022VECTORSEMANTICS ANDEMBEDDINGSthis context window is to the target word. The probability is based on applying thelogistic (sigmoid) function to the dot product of the embeddings of the target wordwith each context word. We could thus compute this probability if only we hadembeddings for each target word and context word in the vocabulary. Let\u2019s now turnto learning these embeddings (which is the real goal of training this classi\ufb01er in the\ufb01rst place).6.8.2 Learning skip-gram embeddingsWord2vec learns embeddings by starting with an initial set of embedding vectorsand then iteratively shifting the embedding of each wordwto be more like the em-beddings of words that occur nearby in texts, and less like the embeddings of wordsthat don\u2019t occur nearby. Let\u2019s start by considering a single piece of training data:... lemon, a [tablespoon of apricot jam, a] pinch ...c1 c2 t c3 c4This example has a target wordt(apricot), and 4 context words in theL=\u00b12window, resulting in 4 positive training instances (on the left below):positive examples +tcapricot tablespoonapricot ofapricot jamapricot anegative examples -tc tcapricot aardvark apricot sevenapricot my apricot foreverapricot where apricot dearapricot coaxial apricot ifFor training a binary classi\ufb01er we also need negative examples. In fact skip-gram uses more negative examples than positive examples (with the ratio betweenthem set by a parameterk). So for each of these(t,c)training instances we\u2019ll createknegative samples, each consisting of the targettplus a \u2018noise word\u2019. A noise wordis a random word from the lexicon, constrained not to be the target wordt. Theright above shows the setting wherek=2, so we\u2019ll have 2 negative examples in thenegative training set\u0000for each positive examplet,c.The noise words are chosen according to their weighted unigram frequencypa(w), whereais a weight. If we were sampling according to unweighted fre-quencyp(w), it would mean that with unigram probabilityp(\u201cthe\u201d)we would choosethe wordtheas a noise word, with unigram probabilityp(\u201caardvark\u201d)we wouldchooseaardvark, and so on. But in practice it is common to seta=.75, i.e. use theweightingp34(w):Pa(w)=count(w)aPw0count(w0)a(6.32)Settinga=.75 gives better performance because it gives rare noise words slightlyhigher probability: for rare words,Pa(w)>P(w). To visualize this intuition, itmight help to work out the probabilities for an example with two events,P(a)=.99andP(b)=.01:Pa(a)=.99.75.99.75+.01.75=.97Pa(b)=.01.75.99.75+.01.75=.03(6.33) Skip-Gram Training data\u2026lemon, a [tablespoon of  apricot  jam,   a]  pinch\u2026c1                   c2 [target]c3      c4\n7820CHAPTER6\u2022VECTORSEMANTICS ANDEMBEDDINGSthis context window is to the target word. The probability is based on applying thelogistic (sigmoid) function to the dot product of the embeddings of the target wordwith each context word. We could thus compute this probability if only we hadembeddings for each target word and context word in the vocabulary. Let\u2019s now turnto learning these embeddings (which is the real goal of training this classi\ufb01er in the\ufb01rst place).6.8.2 Learning skip-gram embeddingsWord2vec learns embeddings by starting with an initial set of embedding vectorsand then iteratively shifting the embedding of each wordwto be more like the em-beddings of words that occur nearby in texts, and less like the embeddings of wordsthat don\u2019t occur nearby. Let\u2019s start by considering a single piece of training data:... lemon, a [tablespoon of apricot jam, a] pinch ...c1 c2 t c3 c4This example has a target wordt(apricot), and 4 context words in theL=\u00b12window, resulting in 4 positive training instances (on the left below):positive examples +tcapricot tablespoonapricot ofapricot jamapricot anegative examples -tc tcapricot aardvark apricot sevenapricot my apricot foreverapricot where apricot dearapricot coaxial apricot ifFor training a binary classi\ufb01er we also need negative examples. In fact skip-gram uses more negative examples than positive examples (with the ratio betweenthem set by a parameterk). So for each of these(t,c)training instances we\u2019ll createknegative samples, each consisting of the targettplus a \u2018noise word\u2019. A noise wordis a random word from the lexicon, constrained not to be the target wordt. Theright above shows the setting wherek=2, so we\u2019ll have 2 negative examples in thenegative training set\u0000for each positive examplet,c.The noise words are chosen according to their weighted unigram frequencypa(w), whereais a weight. If we were sampling according to unweighted fre-quencyp(w), it would mean that with unigram probabilityp(\u201cthe\u201d)we would choosethe wordtheas a noise word, with unigram probabilityp(\u201caardvark\u201d)we wouldchooseaardvark, and so on. But in practice it is common to seta=.75, i.e. use theweightingp34(w):Pa(w)=count(w)aPw0count(w0)a(6.32)Settinga=.75 gives better performance because it gives rare noise words slightlyhigher probability: for rare words,Pa(w)>P(w). To visualize this intuition, itmight help to work out the probabilities for an example with two events,P(a)=.99andP(b)=.01:Pa(a)=.99.75.99.75+.01.75=.97Pa(b)=.01.75.99.75+.01.75=.03(6.33)20CHAPTER6\u2022VECTORSEMANTICS ANDEMBEDDINGSthis context window is to the target word. The probability is based on applying thelogistic (sigmoid) function to the dot product of the embeddings of the target wordwith each context word. We could thus compute this probability if only we hadembeddings for each target word and context word in the vocabulary. Let\u2019s now turnto learning these embeddings (which is the real goal of training this classi\ufb01er in the\ufb01rst place).6.8.2 Learning skip-gram embeddingsWord2vec learns embeddings by starting with an initial set of embedding vectorsand then iteratively shifting the embedding of each wordwto be more like the em-beddings of words that occur nearby in texts, and less like the embeddings of wordsthat don\u2019t occur nearby. Let\u2019s start by considering a single piece of training data:... lemon, a [tablespoon of apricot jam, a] pinch ...c1 c2 t c3 c4This example has a target wordt(apricot), and 4 context words in theL=\u00b12window, resulting in 4 positive training instances (on the left below):positive examples +tcapricot tablespoonapricot ofapricot jamapricot anegative examples -tc tcapricot aardvark apricot sevenapricot my apricot foreverapricot where apricot dearapricot coaxial apricot ifFor training a binary classi\ufb01er we also need negative examples. In fact skip-gram uses more negative examples than positive examples (with the ratio betweenthem set by a parameterk). So for each of these(t,c)training instances we\u2019ll createknegative samples, each consisting of the targettplus a \u2018noise word\u2019. A noise wordis a random word from the lexicon, constrained not to be the target wordt. Theright above shows the setting wherek=2, so we\u2019ll have 2 negative examples in thenegative training set\u0000for each positive examplet,c.The noise words are chosen according to their weighted unigram frequencypa(w), whereais a weight. If we were sampling according to unweighted fre-quencyp(w), it would mean that with unigram probabilityp(\u201cthe\u201d)we would choosethe wordtheas a noise word, with unigram probabilityp(\u201caardvark\u201d)we wouldchooseaardvark, and so on. But in practice it is common to seta=.75, i.e. use theweightingp34(w):Pa(w)=count(w)aPw0count(w0)a(6.32)Settinga=.75 gives better performance because it gives rare noise words slightlyhigher probability: for rare words,Pa(w)>P(w). To visualize this intuition, itmight help to work out the probabilities for an example with two events,P(a)=.99andP(b)=.01:Pa(a)=.99.75.99.75+.01.75=.97Pa(b)=.01.75.99.75+.01.75=.03(6.33) Word2vec: how to learn vectorsGiven the set of positive and negative training instances, and an initial set of embedding vectors The goal of learning is to adjust those word vectors such that we:\u25e6Maximizethe similarity of the target word, context wordpairs (w , cpos) drawn from the positive data\u25e6Minimizethe similarity of the (w , cneg) pairs drawn from the negative data. 2/3/2479 Loss function for one w with cpos, cneg1...cnegkMaximize the similarity of the target with the actual context words, and minimize the similarity of the target with the k negative sampled non-neighbor words. 6.8\u2022WORD2VEC21Given the set of positive and negative training instances, and an initial set of embed-dings, the goal of the learning algorithm is to adjust those embeddings to\u2022Maximize the similarity of the target word, context word pairs(w,cpos)drawnfrom the positive examples\u2022Minimize the similarity of the(w,cneg)pairs from the negative examples.If we consider one word/context pair(w,cpos)with itsknoise wordscneg1...cnegk,we can express these two goals as the following loss functionLto be minimized(hence the\u0000); here the \ufb01rst term expresses that we want the classi\ufb01er to assign thereal context wordcposa high probability of being a neighbor, and the second termexpresses that we want to assign each of the noise wordscnegia high probability ofbeing a non-neighbor, all multiplied because we assume independence:LCE=\u0000log\"P(+|w,cpos)kYi=1P(\u0000|w,cnegi)#=\u0000\"logP(+|w,cpos)+kXi=1logP(\u0000|w,cnegi)#=\u0000\"logP(+|w,cpos)+kXi=1log\u00001\u0000P(+|w,cnegi)\u0000#=\u0000\"logs(cpos\u00b7w)+kXi=1logs(\u0000cnegi\u00b7w)#(6.34)That is, we want to maximize the dot product of the word with the actual contextwords, and minimize the dot products of the word with theknegative sampled non-neighbor words.We minimize this loss function using stochastic gradient descent. Fig.6.14shows the intuition of one step of learning.WCmove apricot and jam closer,increasing cpos z waardvark\nmove apricot and matrix apartdecreasing cneg1 z w\u201c\u2026apricot jam\u2026\u201dw\nzebrazebraaardvarkjamapricot\ncposmatrixTolstoymove apricot and Tolstoy apartdecreasing cneg2 z w!cneg1cneg2k=2Figure 6.14Intuition of one step of gradient descent. The skip-gram model tries to shiftembeddings so the target embeddings (here forapricot) are closer to (have a higher dot prod-uct with) context embeddings for nearby words (herejam) and further from (lower dot productwith) context embeddings for noise words that don\u2019t occur nearby (hereTolstoyandmatrix).To get the gradient, we need to take the derivative of Eq.6.34with respect tothe different embeddings. It turns out the derivatives are the following (we leave the Learning the classifierHow to learn?\u25e6Stochastic gradient descent!We\u2019ll adjust the word weights to\u25e6make the positive pairs more likely \u25e6and the negative pairs less likely, \u25e6over the entire training set. Intuition of one step of gradient descentWCmove apricot and jam closer,increasing cpos z waardvark\nmove apricot and matrix apartdecreasing cneg1 z w\u201c\u2026apricot jam\u2026\u201dw\nzebrazebraaardvarkjamapricot\ncposmatrixTolstoymove apricot and Tolstoy apartdecreasing cneg2 z w!cneg1cneg2k=2 Reminder: gradient descent\u2022At each step\u2022Direction: We move in the reverse direction from the gradient of the loss function\u2022Magnitude: we move the value of this gradient !!\"\ud835\udc3f(\ud835\udc53\ud835\udc65;\ud835\udc64,\ud835\udc66)weighted by a learning rate \u03b7 \u2022Higher learning rate means move wfaster10CHAPTER5\u2022LOGISTICREGRESSIONexample):wt+1=wt\u0000hddwL(f(x;w),y)(5.14)Now let\u2019s extend the intuition from a function of one scalar variablewto manyvariables, because we don\u2019t just want to move left or right, we want to know wherein the N-dimensional space (of theNparameters that make upq) we should move.Thegradientis just such a vector; it expresses the directional components of thesharpest slope along each of thoseNdimensions. If we\u2019re just imagining two weightdimensions (say for one weightwand one biasb), the gradient might be a vector withtwo orthogonal components, each of which tells us how much the ground slopes inthewdimension and in thebdimension. Fig.5.4shows a visualization of the valueof a 2-dimensional gradient vector taken at the red point.\nCost(w,b)\nwbFigure 5.4Visualization of the gradient vector at the red point in two dimensionswandb,showing the gradient as a red arrow in the x-y plane.In an actual logistic regression, the parameter vectorwis much longer than 1 or2, since the input feature vectorxcan be quite long, and we need a weightwiforeachxi. For each dimension/variablewiinw(plus the biasb), the gradient will havea component that tells us the slope with respect to that variable. Essentially we\u2019reasking: \u201cHow much would a small change in that variablewiin\ufb02uence the total lossfunctionL?\u201dIn each dimensionwi, we express the slope as a partial derivative\u2202\u2202wiof the lossfunction. The gradient is then de\ufb01ned as a vector of these partials. We\u2019ll represent \u02c6yasf(x;q)to make the dependence onqmore obvious:\u2014qL(f(x;q),y)) =266664\u2202\u2202w1L(f(x;q),y)\u2202\u2202w2L(f(x;q),y)...\u2202\u2202wnL(f(x;q),y)377775(5.15)The \ufb01nal equation for updatingqbased on the gradient is thusqt+1=qt\u0000h\u2014L(f(x;q),y)(5.16) The derivatives of the loss function22CHAPTER6\u2022VECTORSEMANTICS ANDEMBEDDINGSproof as an exercise at the end of the chapter):\u2202LCE\u2202cpos=[s(cpos\u00b7w)\u00001]w(6.35)\u2202LCE\u2202cneg=[s(cneg\u00b7w)]w(6.36)\u2202LCE\u2202w=[s(cpos\u00b7w)\u00001]cpos+kXi=1[s(cnegi\u00b7w)]cnegi(6.37)The update equations going from time stepttot+1 in stochastic gradient descentare thus:ct+1pos=ctpos\u0000h[s(ctpos\u00b7w)\u00001]w(6.38)ct+1neg=ctneg\u0000h[s(ctneg\u00b7w)]w(6.39)wt+1=wt\u0000h[s(cpos\u00b7wt)\u00001]cpos+kXi=1[s(cnegi\u00b7wt)]cnegi(6.40)Just as in logistic regression, then, the learning algorithm starts with randomly ini-tializedWandCmatrices, and then walks through the training corpus using gradientdescent to moveWandCso as to maximize the objective in Eq.6.34by making theupdates in (Eq.6.39)-(Eq.6.40).Recall that the skip-gram model learnstwoseparate embeddings for each wordi:thetarget embeddingwiand thecontext embeddingci, stored in two matrices, thetargetembeddingcontextembeddingtarget matrixWand thecontext matrixC. It\u2019s common to just add them together,representing wordiwith the vectorwi+ci. Alternatively we can throw away theCmatrix and just represent each wordiby the vectorwi.As with the simple count-based methods like tf-idf, the context window sizeLaffects the performance of skip-gram embeddings, and experiments often tune theparameterLon a devset.6.8.3 Other kinds of static embeddingsThere are many kinds of static embeddings. An extension of word2vec,fasttextfasttext(Bojanowski et al., 2017), deals with unknown words and sparsity in languages withrich morphology, by using subword models. Each word in fasttext is represented asitself plus a bag of constituent n-grams, with special boundary symbols<and>added to each word. For example, withn=3 the wordwherewould be representedby the sequence<where>plus the character n-grams:<wh, whe, her, ere, re>Then a skipgram embedding is learned for each constituent n-gram, and the wordwhereis represented by the sum of all of the embeddings of its constituent n-grams.A fasttext open-source library, including pretrained embeddings for 157 languages,is available athttps://fasttext.cc.The most widely used static embedding model besides word2vec is GloVe(Pen-nington et al., 2014), short for Global Vectors, because the model is based on cap-turing global corpus statistics. GloVe is based on ratios of probabilities from theword-word co-occurrence matrix, combining the intuitions of count-based modelslike PPMI while also capturing the linear structures used by methods like word2vec.6.8\u2022WORD2VEC21Given the set of positive and negative training instances, and an initial set of embed-dings, the goal of the learning algorithm is to adjust those embeddings to\u2022Maximize the similarity of the target word, context word pairs(w,cpos)drawnfrom the positive examples\u2022Minimize the similarity of the(w,cneg)pairs from the negative examples.If we consider one word/context pair(w,cpos)with itsknoise wordscneg1...cnegk,we can express these two goals as the following loss functionLto be minimized(hence the\u0000); here the \ufb01rst term expresses that we want the classi\ufb01er to assign thereal context wordcposa high probability of being a neighbor, and the second termexpresses that we want to assign each of the noise wordscnegia high probability ofbeing a non-neighbor, all multiplied because we assume independence:LCE=\u0000log\"P(+|w,cpos)kYi=1P(\u0000|w,cnegi)#=\u0000\"logP(+|w,cpos)+kXi=1logP(\u0000|w,cnegi)#=\u0000\"logP(+|w,cpos)+kXi=1log\u00001\u0000P(+|w,cnegi)\u0000#=\u0000\"logs(cpos\u00b7w)+kXi=1logs(\u0000cnegi\u00b7w)#(6.34)That is, we want to maximize the dot product of the word with the actual contextwords, and minimize the dot products of the word with theknegative sampled non-neighbor words.We minimize this loss function using stochastic gradient descent. Fig.6.14shows the intuition of one step of learning.WCmove apricot and jam closer,increasing cpos z waardvark\nmove apricot and matrix apartdecreasing cneg1 z w\u201c\u2026apricot jam\u2026\u201dw\nzebrazebraaardvarkjamapricot\ncposmatrixTolstoymove apricot and Tolstoy apartdecreasing cneg2 z w!cneg1cneg2k=2Figure 6.14Intuition of one step of gradient descent. The skip-gram model tries to shiftembeddings so the target embeddings (here forapricot) are closer to (have a higher dot prod-uct with) context embeddings for nearby words (herejam) and further from (lower dot productwith) context embeddings for noise words that don\u2019t occur nearby (hereTolstoyandmatrix).To get the gradient, we need to take the derivative of Eq.6.34with respect tothe different embeddings. It turns out the derivatives are the following (we leave the6.8\u2022WORD2VEC21Given the set of positive and negative training instances, and an initial set of embed-dings, the goal of the learning algorithm is to adjust those embeddings to\u2022Maximize the similarity of the target word, context word pairs(w,cpos)drawnfrom the positive examples\u2022Minimize the similarity of the(w,cneg)pairs from the negative examples.If we consider one word/context pair(w,cpos)with itsknoise wordscneg1...cnegk,we can express these two goals as the following loss functionLto be minimized(hence the\u0000); here the \ufb01rst term expresses that we want the classi\ufb01er to assign thereal context wordcposa high probability of being a neighbor, and the second termexpresses that we want to assign each of the noise wordscnegia high probability ofbeing a non-neighbor, all multiplied because we assume independence:LCE=\u0000log\"P(+|w,cpos)kYi=1P(\u0000|w,cnegi)#=\u0000\"logP(+|w,cpos)+kXi=1logP(\u0000|w,cnegi)#=\u0000\"logP(+|w,cpos)+kXi=1log\u00001\u0000P(+|w,cnegi)\u0000#=\u0000\"logs(cpos\u00b7w)+kXi=1logs(\u0000cnegi\u00b7w)#(6.34)That is, we want to maximize the dot product of the word with the actual contextwords, and minimize the dot products of the word with theknegative sampled non-neighbor words.We minimize this loss function using stochastic gradient descent. Fig.6.14shows the intuition of one step of learning.WCmove apricot and jam closer,increasing cpos z waardvark\nmove apricot and matrix apartdecreasing cneg1 z w\u201c\u2026apricot jam\u2026\u201dw\nzebrazebraaardvarkjamapricotcposmatrixTolstoymove apricot and Tolstoy apartdecreasing cneg2 z w!cneg1cneg2k=2Figure 6.14Intuition of one step of gradient descent. The skip-gram model tries to shiftembeddings so the target embeddings (here forapricot) are closer to (have a higher dot prod-uct with) context embeddings for nearby words (herejam) and further from (lower dot productwith) context embeddings for noise words that don\u2019t occur nearby (hereTolstoyandmatrix).To get the gradient, we need to take the derivative of Eq.6.34with respect tothe different embeddings. It turns out the derivatives are the following (we leave the Update equation in SGD22CHAPTER6\u2022VECTORSEMANTICS ANDEMBEDDINGSproof as an exercise at the end of the chapter):\u2202LCE\u2202cpos=[s(cpos\u00b7w)\u00001]w(6.35)\u2202LCE\u2202cneg=[s(cneg\u00b7w)]w(6.36)\u2202LCE\u2202w=[s(cpos\u00b7w)\u00001]cpos+kXi=1[s(cnegi\u00b7w)]cnegi(6.37)The update equations going from time stepttot+1 in stochastic gradient descentare thus:ct+1pos=ctpos\u0000h[s(ctpos\u00b7wt)\u00001]wt(6.38)ct+1neg=ctneg\u0000h[s(ctneg\u00b7wt)]wt(6.39)wt+1=wt\u0000h\"[s(cpos\u00b7wt)\u00001]cpos+kXi=1[s(cnegi\u00b7wt)]cnegi#(6.40)Just as in logistic regression, then, the learning algorithm starts with randomly ini-tializedWandCmatrices, and then walks through the training corpus using gradientdescent to moveWandCso as to maximize the objective in Eq.6.34by making theupdates in (Eq.6.39)-(Eq.6.40).Recall that the skip-gram model learnstwoseparate embeddings for each wordi:thetarget embeddingwiand thecontext embeddingci, stored in two matrices, thetargetembeddingcontextembeddingtarget matrixWand thecontext matrixC. It\u2019s common to just add them together,representing wordiwith the vectorwi+ci. Alternatively we can throw away theCmatrix and just represent each wordiby the vectorwi.As with the simple count-based methods like tf-idf, the context window sizeLaffects the performance of skip-gram embeddings, and experiments often tune theparameterLon a devset.6.8.3 Other kinds of static embeddingsThere are many kinds of static embeddings. An extension of word2vec,fasttextfasttext(Bojanowski et al., 2017), deals with unknown words and sparsity in languages withrich morphology, by using subword models. Each word in fasttext is represented asitself plus a bag of constituent n-grams, with special boundary symbols<and>added to each word. For example, withn=3 the wordwherewould be representedby the sequence<where>plus the character n-grams:<wh, whe, her, ere, re>Then a skipgram embedding is learned for each constituent n-gram, and the wordwhereis represented by the sum of all of the embeddings of its constituent n-grams.A fasttext open-source library, including pretrained embeddings for 157 languages,is available athttps://fasttext.cc.The most widely used static embedding model besides word2vec is GloVe(Pen-nington et al., 2014), short for Global Vectors, because the model is based on cap-turing global corpus statistics. GloVe is based on ratios of probabilities from theword-word co-occurrence matrix, combining the intuitions of count-based modelslike PPMI while also capturing the linear structures used by methods like word2vec.Start with randomly initialized C and W matrices, then incrementally do updates Two sets of embeddingsSGNS learns two sets of embeddingsTarget embeddings matrix WContext embedding matrix C It's common to just add them together, representing word ias the vector  wi+ ci Summary: How to learn word2vec (skip-gram) embeddingsStart with V random d-dimensional vectors as initial embeddingsTrain a classifier based on embedding similarity\u25e6Take a corpus and take pairs of words that co-occur as positive examples\u25e6Take pairs of words that don't co-occur as negative examples\u25e6Train the classifier to distinguish these by slowly adjusting all the embeddingsto improve the classifier performance\u25e6Throw away the classifier code and keep the embeddings. Vector Semantics & EmbeddingsWord2vec: Learning the embeddings Vector Semantics & EmbeddingsProperties of Embeddings The kinds of neighbors depend on window sizeSmall windows (C= +/-2): nearest words are syntactically similar words in same taxonomy\u25e6Hogwartsnearest neighbors are other fictional schools\u25e6Sunnydale, Evernight, BlandingsLarge windows (C= +/-5):  nearest words are related words in same semantic field\u25e6Hogwartsnearest neighbors are Harry Potter world:\u25e6Dumbledore, half-blood,  Malfoy 24CHAPTER6\u2022VECTORSEMANTICS ANDEMBEDDINGSFor exampleLevy and Goldberg (2014a)showed that using skip-gram with awindow of\u00b12, the most similar words to the wordHogwarts(from theHarry Potterseries) were names of other \ufb01ctional schools:Sunnydale(fromBuffy the VampireSlayer) orEvernight(from a vampire series). With a window of\u00b15, the most similarwords toHogwartswere other words topically related to theHarry Potterseries:Dumbledore,Malfoy, andhalf-blood.It\u2019s also often useful to distinguish two kinds of similarity or association betweenwords(Sch\u00a8utze and Pedersen, 1993). Two words have\ufb01rst-order co-occurrence\ufb01rst-orderco-occurrence(sometimes calledsyntagmatic association) if they are typically nearby each other.Thuswroteis a \ufb01rst-order associate ofbookorpoem. Two words havesecond-orderco-occurrence(sometimes calledparadigmatic association) if they have similarsecond-orderco-occurrenceneighbors. Thuswroteis a second-order associate of words likesaidorremarked.Analogy/Relational Similarity:Another semantic property of embeddings is theirability to capture relational meanings. In an important early vector space model ofcognition,Rumelhart and Abrahamson (1973)proposed theparallelogram modelparallelogrammodelfor solving simple analogy problems of the forma is to b as a* is to what?. In suchproblems, a system given a problem likeapple:tree::grape:?, i.e.,apple is to tree asgrape is to, and must \ufb01ll in the wordvine. In the parallelogram model, illus-trated in Fig.6.15, the vector from the wordappleto the wordtree(=#       \u00bbapple\u0000#   \u00bbtree)is added to the vector forgrape(#        \u00bbgrape); the nearest word to that point is returned.treeapplegrapevineFigure 6.15The parallelogram model for analogy problems(Rumelhart and Abrahamson,1973): the location of#     \u00bbvine can be found by subtracting#   \u00bbtree from#       \u00bbapple and adding#       \u00bbgrape.In early work with sparse embeddings, scholars showed that sparse vector mod-els of meaning could solve such analogy problems(Turney and Littman, 2005), butthe parallelogram method received more modern attention because of its successwith word2vec or GloVe vectors (Mikolov et al. 2013b,Levy and Goldberg 2014b,Pennington et al. 2014). For example, the result of the expression (#     \u00bbking)\u0000#     \u00bbman+#            \u00bbwoman is a vector close to#         \u00bbqueen. Similarly,#      \u00bbParis\u0000#           \u00bbFrance+#     \u00bbItaly) results in avector that is close to#         \u00bbRome. The embedding model thus seems to be extracting rep-resentations of relations likeMALE-FEMALE, orCAPITAL-CITY-OF, or evenCOM-PARATIVE/SUPERLATIVE, as shown in Fig.6.16from GloVe.For aa:b::a*:b*problem, meaning the algorithm is givena, b,anda*and must\ufb01ndb*, the parallelogram method is thus:\u02c6b\u21e4=argmaxxdistance(x,a\u21e4\u0000a+b)(6.41)with the distance function de\ufb01ned either as cosine or as Euclidean distance.There are some caveats. For example, the closest value returned by the paral-lelogram algorithm in word2vec or GloVe embedding spaces is usually not in factb* but one of the 3 input words or their morphological variants (i.e.,cherry:red ::Analogical relationsThe classic parallelogram model of analogical reasoning (Rumelhartand Abrahamson 1973)To solve: \"apple is to tree as grape is to  _____\"Add tree \u2013apple  to grape to get vine Analogical relations via parallelogramThe parallelogram method can solve analogies with both sparse and dense embeddings (Turney and Littman 2005, Mikolovet al. 2013b)king \u2013man + woman is close to queenParis \u2013France + Italy is close to RomeFor a problem a:a*::b:b*, the parallelogram method is:24CHAPTER6\u2022VECTORSEMANTICS ANDEMBEDDINGSFor exampleLevy and Goldberg (2014a)showed that using skip-gram with awindow of\u00b12, the most similar words to the wordHogwarts(from theHarry Potterseries) were names of other \ufb01ctional schools:Sunnydale(fromBuffy the VampireSlayer) orEvernight(from a vampire series). With a window of\u00b15, the most similarwords toHogwartswere other words topically related to theHarry Potterseries:Dumbledore,Malfoy, andhalf-blood.It\u2019s also often useful to distinguish two kinds of similarity or association betweenwords(Sch\u00a8utze and Pedersen, 1993). Two words have\ufb01rst-order co-occurrence\ufb01rst-orderco-occurrence(sometimes calledsyntagmatic association) if they are typically nearby each other.Thuswroteis a \ufb01rst-order associate ofbookorpoem. Two words havesecond-orderco-occurrence(sometimes calledparadigmatic association) if they have similarsecond-orderco-occurrenceneighbors. Thuswroteis a second-order associate of words likesaidorremarked.Analogy/Relational Similarity:Another semantic property of embeddings is theirability to capture relational meanings. In an important early vector space model ofcognition,Rumelhart and Abrahamson (1973)proposed theparallelogram modelparallelogrammodelfor solving simple analogy problems of the forma is to b as a* is to what?. In suchproblems, a system given a problem likeapple:tree::grape:?, i.e.,apple is to tree asgrape is to, and must \ufb01ll in the wordvine. In the parallelogram model, illus-trated in Fig.6.15, the vector from the wordappleto the wordtree(=#       \u00bbapple\u0000#   \u00bbtree)is added to the vector forgrape(#        \u00bbgrape); the nearest word to that point is returned.treeapplegrapevineFigure 6.15The parallelogram model for analogy problems(Rumelhart and Abrahamson,1973): the location of#     \u00bbvine can be found by subtracting#   \u00bbtree from#       \u00bbapple and adding#       \u00bbgrape.In early work with sparse embeddings, scholars showed that sparse vector mod-els of meaning could solve such analogy problems(Turney and Littman, 2005), butthe parallelogram method received more modern attention because of its successwith word2vec or GloVe vectors (Mikolov et al. 2013b,Levy and Goldberg 2014b,Pennington et al. 2014). For example, the result of the expression (#     \u00bbking)\u0000#     \u00bbman+#            \u00bbwoman is a vector close to#         \u00bbqueen. Similarly,#      \u00bbParis\u0000#           \u00bbFrance+#     \u00bbItaly) results in avector that is close to#         \u00bbRome. The embedding model thus seems to be extracting rep-resentations of relations likeMALE-FEMALE, orCAPITAL-CITY-OF, or evenCOM-PARATIVE/SUPERLATIVE, as shown in Fig.6.16from GloVe.For aa:b::a*:b*problem, meaning the algorithm is givena, b,anda*and must\ufb01ndb*, the parallelogram method is thus:\u02c6b\u21e4=argmaxxdistance(x,a\u21e4\u0000a+b)(6.41)with the distance function de\ufb01ned either as cosine or as Euclidean distance.There are some caveats. For example, the closest value returned by the paral-lelogram algorithm in word2vec or GloVe embedding spaces is usually not in factb* but one of the 3 input words or their morphological variants (i.e.,cherry:red :: Structure in GloVE Embedding space Caveats with the parallelogram methodIt only seems to work for frequent words, small distances and certain relations (relating countries to capitals, or parts of speech), but not others. (Linzen2016, Gladkovaet al. 2016, Ethayarajhet al. 2019a) Understanding analogy is an open area of research (Peterson et al. 2020) Train embeddings on different decades of historical text to see meanings shift~30 million books, 1850-1990, Google Books dataEmbeddings as a window onto historical semantics\nWilliam L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change. Proceedings of ACL. Embeddings reflect cultural bias!Ask \u201cParis : France :: Tokyo : x\u201d \u25e6x = JapanAsk \u201cfather : doctor :: mother : x\u201d \u25e6x = nurseAsk \u201cman : computer programmer :: woman : x\u201d \u25e6x = homemakerBolukbasi, Tolga, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. \"Man is to computer programmer as woman is to homemaker? debiasing word embeddings.\" InNeurIPS, pp. 4349-4357. 2016.\nAlgorithms that use embeddings as part of e.g., hiring searches for programmers, might lead to bias in hiring Historical embedding as a tool to study cultural biases\u2022Compute a gender or ethnic bias for each adjective: e.g., how much closer the adjective is to \"woman\" synonyms than \"man\" synonyms, or names of particular ethnicities\u2022Embeddings for competence adjective (smart, wise, brilliant, resourceful, thoughtful, logical) are biased toward men, a bias slowly decreasing 1960-1990\u2022Embeddings for dehumanizing adjectives (barbaric, monstrous, bizarre)  were biased toward Asians in the 1930s, bias decreasing over the 20th century.\u2022These match the results of old surveys done in the 1930sGarg, N., Schiebinger, L., Jurafsky, D., and Zou, J. (2018). Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences 115(16), E3635\u2013E3644. Vector Semantics & EmbeddingsProperties of Embeddings", "7.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\n7Neural Networks\n\u201c[M]achines of this character can behave in a very complicated manner when\nthe number of units is large.\u201d\nAlan Turing (1948) \u201cIntelligent Machines\u201d, page 6\nNeural networks are a fundamental computational tool for language process-\ning, and a very old one. They are called neural because their origins lie in the\nMcCulloch-Pitts neuron (McCulloch and Pitts, 1943), a simpli\ufb01ed model of the\nbiological neuron as a kind of computing element that could be described in terms\nof propositional logic. But the modern use in language processing no longer draws\non these early biological inspirations.\nInstead, a modern neural network is a network of small computing units, each\nof which takes a vector of input values and produces a single output value. In this\nchapter we introduce the neural net applied to classi\ufb01cation. The architecture we\nintroduce is called a feedforward network because the computation proceeds iter- feedforward\natively from one layer of units to the next. The use of modern neural nets is often\ncalled deep learning , because modern networks are often deep (have many layers). deep learning\nNeural networks share much of the same mathematics as logistic regression. But\nneural networks are a more powerful classi\ufb01er than logistic regression, and indeed a\nminimal neural network (technically one with a single \u2018hidden layer\u2019) can be shown\nto learn any function.\nNeural net classi\ufb01ers are different from logistic regression in another way. With\nlogistic regression, we applied the regression classi\ufb01er to many different tasks by\ndeveloping many rich kinds of feature templates based on domain knowledge. When\nworking with neural networks, it is more common to avoid most uses of rich hand-\nderived features, instead building neural networks that take raw words as inputs\nand learn to induce features as part of the process of learning to classify. We saw\nexamples of this kind of representation learning for embeddings in Chapter 6. Nets\nthat are very deep are particularly good at representation learning. For that reason\ndeep neural nets are the right tool for tasks that offer suf\ufb01cient data to learn features\nautomatically.\nIn this chapter we\u2019ll introduce feedforward networks as classi\ufb01ers, and also ap-\nply them to the simple task of language modeling: assigning probabilities to word\nsequences and predicting upcoming words. In subsequent chapters we\u2019ll introduce\nmany other aspects of neural models, such as recurrent neural networks (Chap-\nter 8), the Transformer (Chapter 9), and masked language modeling (Chapter 11). 2CHAPTER 7 \u2022 N EURAL NETWORKS\n7.1 Units\nThe building block of a neural network is a single computational unit. A unit takes\na set of real valued numbers as input, performs some computation on them, and\nproduces an output.\nAt its heart, a neural unit is taking a weighted sum of its inputs, with one addi-\ntional term in the sum called a bias term . Given a set of inputs x1:::xn, a unit has bias term\na set of corresponding weights w1:::wnand a bias b, so the weighted sum zcan be\nrepresented as:\nz=b+X\niwixi (7.1)\nOften it\u2019s more convenient to express this weighted sum using vector notation; recall\nfrom linear algebra that a vector is, at heart, just a list or array of numbers. Thus vector\nwe\u2019ll talk about zin terms of a weight vector w, a scalar bias b, and an input vector\nx, and we\u2019ll replace the sum with the convenient dot product :\nz=w\u0001x+b (7.2)\nAs de\ufb01ned in Eq. 7.2, zis just a real valued number.\nFinally, instead of using z, a linear function of x, as the output, neural units\napply a non-linear function ftoz. We will refer to the output of this function as\ntheactivation value for the unit, a. Since we are just modeling a single unit, the activation\nactivation for the node is in fact the \ufb01nal output of the network, which we\u2019ll generally\ncally. So the value yis de\ufb01ned as:\ny=a=f(z)\nWe\u2019ll discuss three popular non-linear functions fbelow (the sigmoid, the tanh, and\nthe recti\ufb01ed linear unit or ReLU) but it\u2019s pedagogically convenient to start with the\nsigmoid function since we saw it in Chapter 5: sigmoid\ny=s(z) =1\n1+e\u0000z(7.3)\nThe sigmoid (shown in Fig. 7.1) has a number of advantages; it maps the output\ninto the range (0;1), which is useful in squashing outliers toward 0 or 1. And it\u2019s\ndifferentiable, which as we saw in Section ??will be handy for learning.\nFigure 7.1 The sigmoid function takes a real value and maps it to the range (0;1). It is\nnearly linear around 0 but outlier values get squashed toward 0 or 1.\nSubstituting Eq. 7.2 into Eq. 7.3 gives us the output of a neural unit:\ny=s(w\u0001x+b) =1\n1+exp(\u0000(w\u0001x+b))(7.4) 7.1 \u2022 U NITS 3\nFig. 7.2 shows a \ufb01nal schematic of a basic neural unit. In this example the unit\ntakes 3 input values x1;x2, and x3, and computes a weighted sum, multiplying each\nvalue by a weight ( w1,w2, and w3, respectively), adds them to a bias term b, and then\npasses the resulting sum through a sigmoid function to result in a number between 0\nand 1.\nx1x2x3\nyw1w2w3\u2211b\u03c3+1za\nFigure 7.2 A neural unit, taking 3 inputs x1,x2, and x3(and a bias bthat we represent as a\nweight for an input clamped at +1) and producing an output y. We include some convenient\nintermediate variables: the output of the summation, z, and the output of the sigmoid, a. In\nthis case the output of the unit yis the same as a, but in deeper networks we\u2019ll reserve yto\nmean the \ufb01nal output of the entire network, leaving aas the activation of an individual node.\nLet\u2019s walk through an example just to get an intuition. Let\u2019s suppose we have a\nunit with the following weight vector and bias:\nw= [0:2;0:3;0:9]\nb=0:5\nWhat would this unit do with the following input vector:\nx= [0:5;0:6;0:1]\nThe resulting output ywould be:\ny=s(w\u0001x+b) =1\n1+e\u0000(w\u0001x+b)=1\n1+e\u0000(:5\u0003:2+:6\u0003:3+:1\u0003:9+:5)=1\n1+e\u00000:87=:70\nIn practice, the sigmoid is not commonly used as an activation function. A function\nthat is very similar but almost always better is the tanh function shown in Fig. 7.3a; tanh\ntanh is a variant of the sigmoid that ranges from -1 to +1:\ny=tanh(z) =ez\u0000e\u0000z\nez+e\u0000z(7.5)\nThe simplest activation function, and perhaps the most commonly used, is the rec-\nti\ufb01ed linear unit, also called the ReLU , shown in Fig. 7.3b. It\u2019s just the same as z ReLU\nwhen zis positive, and 0 otherwise:\ny=ReLU(z) =max(z;0) (7.6)\nThese activation functions have different properties that make them useful for differ-\nent language applications or network architectures. For example, the tanh function\nhas the nice properties of being smoothly differentiable and mapping outlier values\ntoward the mean. The recti\ufb01er function, on the other hand, has nice properties that 4CHAPTER 7 \u2022 N EURAL NETWORKS\n(a) (b)\nFigure 7.3 The tanh and ReLU activation functions.\nresult from it being very close to linear. In the sigmoid or tanh functions, very high\nvalues of zresult in values of ythat are saturated , i.e., extremely close to 1, and have saturated\nderivatives very close to 0. Zero derivatives cause problems for learning, because as\nwe\u2019ll see in Section 7.5, we\u2019ll train networks by propagating an error signal back-\nwards, multiplying gradients (partial derivatives) from each layer of the network;\ngradients that are almost 0 cause the error signal to get smaller and smaller until it is\ntoo small to be used for training, a problem called the vanishing gradient problem.vanishing\ngradient\nRecti\ufb01ers don\u2019t have this problem, since the derivative of ReLU for high values of z\nis 1 rather than very close to 0.\n7.2 The XOR problem\nEarly in the history of neural networks it was realized that the power of neural net-\nworks, as with the real neurons that inspired them, comes from combining these\nunits into larger networks.\nOne of the most clever demonstrations of the need for multi-layer networks was\nthe proof by Minsky and Papert (1969) that a single neural unit cannot compute\nsome very simple functions of its input. Consider the task of computing elementary\nlogical functions of two inputs, like AND, OR, and XOR. As a reminder, here are\nthe truth tables for those functions:\nAND OR XOR\nx1 x2 y x1 x2 y x1 x2 y\n0 0 0 0 0 0 0 0 0\n0 1 0 0 1 1 0 1 1\n1 0 0 1 0 1 1 0 1\n1 1 1 1 1 1 1 1 0\nThis example was \ufb01rst shown for the perceptron , which is a very simple neural perceptron\nunit that has a binary output and does nothave a non-linear activation function. The\noutput yof a perceptron is 0 or 1, and is computed as follows (using the same weight\nw, input x, and bias bas in Eq. 7.2):\ny=\u001a0;ifw\u0001x+b\u00140\n1;ifw\u0001x+b>0(7.7) 7.2 \u2022 T HEXOR PROBLEM 5\nIt\u2019s very easy to build a perceptron that can compute the logical AND and OR\nfunctions of its binary inputs; Fig. 7.4 shows the necessary weights.\nx1x2+1-111\nx1x2+1011\n(a) (b)\nFigure 7.4 The weights wand bias bfor perceptrons for computing logical functions. The\ninputs are shown as x1andx2and the bias as a special node with value +1 which is multiplied\nwith the bias weight b. (a) logical AND, with weights w1=1 and w2=1 and bias weight\nb=\u00001. (b) logical OR, with weights w1=1 and w2=1 and bias weight b=0. These\nweights/biases are just one from an in\ufb01nite number of possible sets of weights and biases that\nwould implement the functions.\nIt turns out, however, that it\u2019s not possible to build a perceptron to compute\nlogical XOR! (It\u2019s worth spending a moment to give it a try!)\nThe intuition behind this important result relies on understanding that a percep-\ntron is a linear classi\ufb01er. For a two-dimensional input x1andx2, the perceptron\nequation, w1x1+w2x2+b=0 is the equation of a line. (We can see this by putting\nit in the standard linear format: x2= (\u0000w1=w2)x1+ (\u0000b=w2).) This line acts as a\ndecision boundary in two-dimensional space in which the output 0 is assigned to alldecision\nboundary\ninputs lying on one side of the line, and the output 1 to all input points lying on the\nother side of the line. If we had more than 2 inputs, the decision boundary becomes\na hyperplane instead of a line, but the idea is the same, separating the space into two\ncategories.\nFig. 7.5 shows the possible logical inputs ( 00,01,10, and 11) and the line drawn\nby one possible set of parameters for an AND and an OR classi\ufb01er. Notice that there\nis simply no way to draw a line that separates the positive cases of XOR (01 and 10)\nfrom the negative cases (00 and 11). We say that XOR is not a linearly separablelinearly\nseparable\nfunction. Of course we could draw a boundary with a curve, or some other function,\nbut not a single line.\n7.2.1 The solution: neural networks\nWhile the XOR function cannot be calculated by a single perceptron, it can be cal-\nculated by a layered network of perceptron units. Rather than see this with networks\nof simple perceptrons, however, let\u2019s see how to compute XOR using two layers of\nReLU-based units following Goodfellow et al. (2016). Fig. 7.6 shows a \ufb01gure with\nthe input being processed by two layers of neural units. The middle layer (called\nh) has two units, and the output layer (called y) has one unit. A set of weights and\nbiases are shown that allows the network to correctly compute the XOR function.\nLet\u2019s walk through what happens with the input x= [0, 0]. If we multiply each\ninput value by the appropriate weight, sum, and then add the bias b, we get the vector\n[0, -1], and we then apply the recti\ufb01ed linear transformation to give the output of the\nhlayer as [0, 0]. Now we once again multiply by the weights, sum, and add the\nbias (0 in this case) resulting in the value 0. The reader should work through the\ncomputation of the remaining 3 possible input pairs to see that the resulting yvalues\nare 1 for the inputs [0, 1] and [1, 0] and 0 for [0, 0] and [1, 1]. 6CHAPTER 7 \u2022 N EURAL NETWORKS\n0011x1x20011x1x20011x1x2\na)  x1 AND x2b)  x1 OR x2c)  x1 XOR x2?\nFigure 7.5 The functions AND, OR, and XOR, represented with input x1on the x-axis and input x2on the\ny-axis. Filled circles represent perceptron outputs of 1, and white circles perceptron outputs of 0. There is no\nway to draw a line that correctly separates the two categories for XOR. Figure styled after Russell and Norvig\n(2002).\nx1x2h1h2y1+11-1111-201+10\nFigure 7.6 XOR solution after Goodfellow et al. (2016). There are three ReLU units, in\ntwo layers; we\u2019ve called them h1,h2(hfor \u201chidden layer\u201d) and y1. As before, the numbers\non the arrows represent the weights wfor each unit, and we represent the bias bas a weight\non a unit clamped to +1, with the bias weights/units in gray.\nIt\u2019s also instructive to look at the intermediate results, the outputs of the two\nhidden nodes h1andh2. We showed in the previous paragraph that the hvector for\nthe inputs x= [0, 0] was [0, 0]. Fig. 7.7b shows the values of the hlayer for all\n4 inputs. Notice that hidden representations of the two input points x= [0, 1] and\nx= [1, 0] (the two cases with XOR output = 1) are merged to the single point h=\n[1, 0]. The merger makes it easy to linearly separate the positive and negative cases\nof XOR. In other words, we can view the hidden layer of the network as forming a\nrepresentation of the input.\nIn this example we just stipulated the weights in Fig. 7.6. But for real examples\nthe weights for neural networks are learned automatically using the error backprop-\nagation algorithm to be introduced in Section 7.5. That means the hidden layers will\nlearn to form useful representations. This intuition, that neural networks can auto-\nmatically learn useful representations of the input, is one of their key advantages,\nand one that we will return to again and again in later chapters. 7.3 \u2022 F EEDFORWARD NEURAL NETWORKS 7\n0011x1x2\na) The original x space0011h1h2\n2b) The new (linearly separable) h space\nFigure 7.7 The hidden layer forming a new representation of the input. (b) shows the\nrepresentation of the hidden layer, h, compared to the original input representation xin (a).\nNotice that the input point [0, 1] has been collapsed with the input point [1, 0], making it\npossible to linearly separate the positive and negative cases of XOR. After Goodfellow et al.\n(2016).\n7.3 Feedforward Neural Networks\nLet\u2019s now walk through a slightly more formal presentation of the simplest kind of\nneural network, the feedforward network . A feedforward network is a multilayerfeedforward\nnetwork\nnetwork in which the units are connected with no cycles; the outputs from units in\neach layer are passed to units in the next higher layer, and no outputs are passed\nback to lower layers. (In Chapter 8 we\u2019ll introduce networks with cycles, called\nrecurrent neural networks .)\nFor historical reasons multilayer networks, especially feedforward networks, are\nsometimes called multi-layer perceptrons (orMLP s); this is a technical misnomer,multi-layer\nperceptrons\nMLP since the units in modern multilayer networks aren\u2019t perceptrons (perceptrons are\npurely linear, but modern networks are made up of units with non-linearities like\nsigmoids), but at some point the name stuck.\nSimple feedforward networks have three kinds of nodes: input units, hidden\nunits, and output units.\nFig. 7.8 shows a picture. The input layer xis a vector of simple scalar values just\nas we saw in Fig. 7.2.\nThe core of the neural network is the hidden layer hformed of hidden units hi, hidden layer\neach of which is a neural unit as described in Section 7.1, taking a weighted sum of\nits inputs and then applying a non-linearity. In the standard architecture, each layer\nisfully-connected , meaning that each unit in each layer takes as input the outputs fully-connected\nfrom all the units in the previous layer, and there is a link between every pair of units\nfrom two adjacent layers. Thus each hidden unit sums over all the input units.\nRecall that a single hidden unit has as parameters a weight vector and a bias. We\nrepresent the parameters for the entire hidden layer by combining the weight vector\nand bias for each unit iinto a single weight matrix Wand a single bias vector bfor\nthe whole layer (see Fig. 7.8). Each element Wjiof the weight matrix Wrepresents\nthe weight of the connection from the ith input unit xito the jth hidden unit hj.\nThe advantage of using a single matrix Wfor the weights of the entire layer is\nthat now the hidden layer computation for a feedforward network can be done very\nef\ufb01ciently with simple matrix operations. In fact, the computation only has three 8CHAPTER 7 \u2022 N EURAL NETWORKS\nx1x2xn0\u2026\u2026+1b\u2026UW\ninput layerhidden layeroutput layerh1y1y2yn2h2h3hn1\nFigure 7.8 A simple 2-layer feedforward network, with one hidden layer, one output layer,\nand one input layer (the input layer is usually not counted when enumerating layers).\nsteps: multiplying the weight matrix by the input vector x, adding the bias vector b,\nand applying the activation function g(such as the sigmoid, tanh, or ReLU activation\nfunction de\ufb01ned above).\nThe output of the hidden layer, the vector h, is thus the following (for this exam-\nple we\u2019ll use the sigmoid function sas our activation function):\nh=s(Wx+b) (7.8)\nNotice that we\u2019re applying the sfunction here to a vector, while in Eq. 7.3 it was\napplied to a scalar. We\u2019re thus allowing s(\u0001), and indeed any activation function\ng(\u0001), to apply to a vector element-wise, so g[z1;z2;z3] = [g(z1);g(z2);g(z3)].\nLet\u2019s introduce some constants to represent the dimensionalities of these vectors\nand matrices. We\u2019ll refer to the input layer as layer 0 of the network, and have n0\nrepresent the number of inputs, so xis a vector of real numbers of dimension n0,\nor more formally x2Rn0, a column vector of dimensionality [n0;1]. Let\u2019s call the\nhidden layer layer 1 and the output layer layer 2. The hidden layer has dimensional-\nityn1, soh2Rn1and also b2Rn1(since each hidden unit can take a different bias\nvalue). And the weight matrix Whas dimensionality W2Rn1\u0002n0, i.e.[n1;n0].\nTake a moment to convince yourself that the matrix multiplication in Eq. 7.8 will\ncompute the value of each hjass\u0000Pn0\ni=1Wjixi+bj\u0001\n.\nAs we saw in Section 7.2, the resulting value h(forhidden but also for hypoth-\nesis) forms a representation of the input. The role of the output layer is to take\nthis new representation hand compute a \ufb01nal output. This output could be a real-\nvalued number, but in many cases the goal of the network is to make some sort of\nclassi\ufb01cation decision, and so we will focus on the case of classi\ufb01cation.\nIf we are doing a binary task like sentiment classi\ufb01cation, we might have a sin-\ngle output node, and its scalar value yis the probability of positive versus negative\nsentiment. If we are doing multinomial classi\ufb01cation, such as assigning a part-of-\nspeech tag, we might have one output node for each potential part-of-speech, whose\noutput value is the probability of that part-of-speech, and the values of all the output\nnodes must sum to one. The output layer is thus a vector ythat gives a probability\ndistribution across the output nodes.\nLet\u2019s see how this happens. Like the hidden layer, the output layer has a weight\nmatrix (let\u2019s call it U), but some models don\u2019t include a bias vector bin the output 7.3 \u2022 F EEDFORWARD NEURAL NETWORKS 9\nlayer, so we\u2019ll simplify by eliminating the bias vector in this example. The weight\nmatrix is multiplied by its input vector ( h) to produce the intermediate output z:\nz=Uh\nThere are n2output nodes, so z2Rn2, weight matrix Uhas dimensionality U2\nRn2\u0002n1, and element Ui jis the weight from unit jin the hidden layer to unit iin the\noutput layer.\nHowever, zcan\u2019t be the output of the classi\ufb01er, since it\u2019s a vector of real-valued\nnumbers, while what we need for classi\ufb01cation is a vector of probabilities. There is\na convenient function for normalizing a vector of real values, by which we mean normalizing\nconverting it to a vector that encodes a probability distribution (all the numbers lie\nbetween 0 and 1 and sum to 1): the softmax function that we saw on page ??of softmax\nChapter 5. More generally for any vector zof dimensionality d, the softmax is\nde\ufb01ned as:\nsoftmax (zi) =exp(zi)Pd\nj=1exp(zj)1\u0014i\u0014d (7.9)\nThus for example given a vector\nz= [0:6;1:1;\u00001:5;1:2;3:2;\u00001:1]; (7.10)\nthe softmax function will normalize it to a probability distribution (shown rounded):\nsoftmax (z) = [ 0:055;0:090;0:0067 ;0:10;0:74;0:010] (7.11)\nYou may recall that we used softmax to create a probability distribution from a\nvector of real-valued numbers (computed from summing weights times features) in\nthe multinomial version of logistic regression in Chapter 5.\nThat means we can think of a neural network classi\ufb01er with one hidden layer\nas building a vector hwhich is a hidden layer representation of the input, and then\nrunning standard multinomial logistic regression on the features that the network\ndevelops in h. By contrast, in Chapter 5 the features were mainly designed by hand\nvia feature templates. So a neural network is like multinomial logistic regression,\nbut (a) with many layers, since a deep neural network is like layer after layer of lo-\ngistic regression classi\ufb01ers; (b) with those intermediate layers having many possible\nactivation functions (tanh, ReLU, sigmoid) instead of just sigmoid (although we\u2019ll\ncontinue to use sfor convenience to mean any activation function); (c) rather than\nforming the features by feature templates, the prior layers of the network induce the\nfeature representations themselves.\nHere are the \ufb01nal equations for a feedforward network with a single hidden layer,\nwhich takes an input vector x, outputs a probability distribution y, and is parameter-\nized by weight matrices WandUand a bias vector b:\nh=s(Wx+b)\nz=Uh\ny=softmax (z) (7.12)\nAnd just to remember the shapes of all our variables, x2Rn0,h2Rn1,b2Rn1,\nW2Rn1\u0002n0,U2Rn2\u0002n1, and the output vector y2Rn2. We\u2019ll call this network a 2-\nlayer network (we traditionally don\u2019t count the input layer when numbering layers,\nbut do count the output layer). So by this terminology logistic regression is a 1-layer\nnetwork. 10 CHAPTER 7 \u2022 N EURAL NETWORKS\n7.3.1 More details on feedforward networks\nLet\u2019s now set up some notation to make it easier to talk about deeper networks of\ndepth more than 2. We\u2019ll use superscripts in square brackets to mean layer num-\nbers, starting at 0 for the input layer. So W[1]will mean the weight matrix for the\n(\ufb01rst) hidden layer, and b[1]will mean the bias vector for the (\ufb01rst) hidden layer. nj\nwill mean the number of units at layer j. We\u2019ll use g(\u0001)to stand for the activation\nfunction, which will tend to be ReLU or tanh for intermediate layers and softmax\nfor output layers. We\u2019ll use a[i]to mean the output from layer i, and z[i]to mean the\ncombination of previous layer output, weights and biases W[i]a[i\u00001]+b[i]. The 0th\nlayer is for inputs, so we\u2019ll refer to the inputs xmore generally as a[0].\nThus we can re-represent our 2-layer net from Eq. 7.12 as follows:\nz[1]=W[1]a[0]+b[1]\na[1]=g[1](z[1])\nz[2]=W[2]a[1]+b[2]\na[2]=g[2](z[2])\n\u02c6y=a[2](7.13)\nNote that with this notation, the equations for the computation done at each layer are\nthe same. The algorithm for computing the forward step in an n-layer feedforward\nnetwork, given the input vector a[0]is thus simply:\nforiin1,...,n\nz[i]=W[i]a[i\u00001]+b[i]\na[i]=g[i](z[i])\n\u02c6y=a[n]\nIt\u2019s often useful to have a name for the \ufb01nal set of activations right before the \ufb01nal\nsoftmax. So however many layers we have, we\u2019ll generally call the unnormalized\nvalues in the \ufb01nal vector z[n], the vector of scores right before the \ufb01nal softmax, the\nlogits (see ( ??). logits\nThe need for non-linear activation functions One of the reasons we use non-\nlinear activation functions for each layer in a neural network is that if we did not, the\nresulting network is exactly equivalent to a single-layer network. Let\u2019s see why this\nis true. Imagine the \ufb01rst two layers of such a network of purely linear layers:\nz[1]=W[1]x+b[1]\nz[2]=W[2]z[1]+b[2]\nWe can rewrite the function that the network is computing as:\nz[2]=W[2]z[1]+b[2]\n=W[2](W[1]x+b[1])+b[2]\n=W[2]W[1]x+W[2]b[1]+b[2]\n=W0x+b0(7.14)\nThis generalizes to any number of layers. So without non-linear activation functions,\na multilayer network is just a notational variant of a single layer network with a\ndifferent set of weights, and we lose all the representational power of multilayer\nnetworks. 7.4 \u2022 F EEDFORWARD NETWORKS FOR NLP: C LASSIFICATION 11\nReplacing the bias unit In describing networks, we will often use a slightly sim-\npli\ufb01ed notation that represents exactly the same function without referring to an ex-\nplicit bias node b. Instead, we add a dummy node a0to each layer whose value will\nalways be 1. Thus layer 0, the input layer, will have a dummy node a[0]\n0=1, layer 1\nwill have a[1]\n0=1, and so on. This dummy node still has an associated weight, and\nthat weight represents the bias value b. For example instead of an equation like\nh=s(Wx+b) (7.15)\nwe\u2019ll use:\nh=s(Wx) (7.16)\nBut now instead of our vector xhaving n0values: x=x1;:::;xn0, it will have n0+\n1 values, with a new 0th dummy value x0=1:x=x0;:::;xn0. And instead of\ncomputing each hjas follows:\nhj=s n0X\ni=1Wjixi+bj!\n; (7.17)\nwe\u2019ll instead use:\nhj=s n0X\ni=0Wjixi!\n; (7.18)\nwhere the value Wj0replaces what had been bj. Fig. 7.9 shows a visualization.\nx1x2xn0\u2026\u2026+1b\u2026UWh1y1y2yn2h2h3hn1\nx1x2xn0\u2026\u2026x0=1\u2026UWh1y1y2yn2h2h3hn1\n(a) (b)\nFigure 7.9 Replacing the bias node (shown in a) with x0(b).\nWe\u2019ll continue showing the bias as bwhen we go over the learning algorithm\nin Section 7.5, but then we\u2019ll switch to this simpli\ufb01ed notation without explicit bias\nterms for the rest of the book.\n7.4 Feedforward networks for NLP: Classi\ufb01cation\nLet\u2019s see how to apply feedforward networks to NLP tasks! In this section we\u2019ll\nlook at classi\ufb01cation tasks like sentiment analysis; in the next section we\u2019ll introduce\nneural language modeling. 12 CHAPTER 7 \u2022 N EURAL NETWORKS\nLet\u2019s begin with a simple 2-layer sentiment classi\ufb01er. You might imagine tak-\ning our logistic regression classi\ufb01er from Chapter 5, which corresponds to a 1-layer\nnetwork, and just adding a hidden layer. The input element xicould be scalar fea-\ntures like those in Fig. ??, e.g., x1= count(words2doc), x2= count(positive lexicon\nwords2doc), x3= 1 if \u201cno\u201d2doc, and so on. And the output layer \u02c6ycould have\ntwo nodes (one each for positive and negative), or 3 nodes (positive, negative, neu-\ntral), in which case \u02c6y1would be the estimated probability of positive sentiment, \u02c6y2\nthe probability of negative and \u02c6y3the probability of neutral. The resulting equations\nwould be just what we saw above for a 2-layer network (as always, we\u2019ll continue\nto use the sto stand for any non-linearity, whether sigmoid, ReLU or other).\nx= [x1;x2;:::xN](each xiis a hand-designed feature)\nh=s(Wx+b)\nz=Uh\n\u02c6y=softmax (z) (7.19)\nFig. 7.10 shows a sketch of this architecture. As we mentioned earlier, adding this\nhidden layer to our logistic regression classi\ufb01er allows the network to represent the\nnon-linear interactions between features. This alone might give us a better sentiment\nclassi\ufb01er.\nUW[n\u2a091]Hidden layerOutput layersoftmax[dh\u2a09n][dh\u2a091][3\u2a09dh]Input wordsp(+)h1h2h3hdh\u2026y1^y2^y3^xhyInput layer n=3 features[3\u2a091]x1x2x3dessertwasgreatpositive lexiconwords = 1count of \u201cno\u201d = 0wordcount=3p(-)p(neut)\nFigure 7.10 Feedforward network sentiment analysis using traditional hand-built features\nof the input text.\nMost applications of neural networks for NLP do something different, however.\nInstead of using hand-built human-engineered features as the input to our classi\ufb01er,\nwe draw on deep learning\u2019s ability to learn features from the data by representing\nwords as embeddings, like the word2vec or GloVe embeddings we saw in Chapter 6.\nThere are various ways to represent an input for classi\ufb01cation. One simple baseline\nis to apply some sort of pooling function to the embeddings of all the words in the pooling\ninput. For example, for a text with ninput words/tokens w1;:::;wn, we can turn the\nnembeddings e(w1);:::;e(wn)(each of dimensionality d) into a single embedding\nalso of dimensionality dby just summing the embeddings, or by taking their mean\n(summing and then dividing by n):\nxmean=1\nnnX\ni=1e(wi) (7.20) 7.4 \u2022 F EEDFORWARD NETWORKS FOR NLP: C LASSIFICATION 13\nThere are many other options, like taking the element-wise max. The element-wise\nmax of a set of nvectors is a new vector whose kth element is the max of the kth\nelements of all the nvectors. Here are the equations for this classi\ufb01er assuming\nmean pooling; the architecture is sketched in Fig. 7.11:\nx=mean(e(w1);e(w2);:::;e(wn))\nh=s(Wx+b)\nz=Uh\n\u02c6y=softmax (z) (7.21)\nUW[d\u2a091]Hidden layerOutput layersoftmax[dh\u2a09d][dh\u2a091][3\u2a09dh]Input wordsp(+)embedding for\u201cgreat\u201dembedding for\u201cdessert\u201dh1h2h3hdh\u2026y1^y2^y3^xhyInput layer pooled embedding[3\u2a091]pooling+dessertwasgreatembedding for\u201cwas\u201dp(-)p(neut)\nFigure 7.11 Feedforward network sentiment analysis using a pooled embedding of the in-\nput words.\nWhile Eq. 7.21 shows how to classify a single example x, in practice we want\nto ef\ufb01ciently classify an entire test set of mexamples. We do this by vectorizing\nthe process, just as we saw with logistic regression; instead of using for-loops to go\nthrough each example, we\u2019ll use matrix multiplication to do the entire computation\nof an entire test set at once. First, we pack all the input feature vectors for each input\nxinto a single input matrix X, with each row ia row vector consisting of the pooled\nembedding for input example x(i)(i.e., the vector x(i)). If the dimensionality of our\npooled input embedding is d,Xwill be a matrix of shape [m\u0002d].\nWe will then need to slightly modify Eq. 7.21. Xis of shape [m\u0002d]andWis of\nshape [dh\u0002d], so we\u2019ll have to reorder how we multiply XandWand transpose W\nso they correctly multiply to yield a matrix Hof shape [m\u0002dh].1The bias vector b\nfrom Eq. 7.21 of shape [1\u0002dh]will now have to be replicated into a matrix of shape\n[m\u0002dh]. We\u2019ll need to similarly reorder the next step and transpose U. Finally, our\noutput matrix \u02c6Ywill be of shape [m\u00023](or more generally [m\u0002do], where dois\nthe number of output classes), with each row iof our output matrix \u02c6Yconsisting of\nthe output vector \u02c6y(i).\u2018 Here are the \ufb01nal equations for computing the output class\n1Note that we could have kept the original order of our products if we had instead made our input\nmatrix Xrepresent each input as a column vector instead of a row vector, making it of shape [d\u0002m]. But\nrepresenting inputs as row vectors is convenient and common in neural network models. 14 CHAPTER 7 \u2022 N EURAL NETWORKS\ndistribution for an entire test set:\nH=s(XW|+b)\nZ=HU|\n\u02c6Y=softmax (Z) (7.22)\nThe idea of using word2vec or GloVe embeddings as our input representation\u2014\nand more generally the idea of relying on another algorithm to have already learned\nan embedding representation for our input words\u2014is called pretraining . Using pretraining\npretrained embedding representations, whether simple static word embeddings like\nword2vec or the much more powerful contextual embeddings we\u2019ll introduce in\nChapter 11, is one of the central ideas of deep learning. (It\u2019s also possible, how-\never, to train the word embeddings as part of an NLP task; we\u2019ll talk about how to\ndo this in Section 7.7 in the context of the neural language modeling task.)\n7.5 Training Neural Nets\nA feedforward neural net is an instance of supervised machine learning in which we\nknow the correct output yfor each observation x. What the system produces, via\nEq. 7.13, is \u02c6 y, the system\u2019s estimate of the true y. The goal of the training procedure\nis to learn parameters W[i]andb[i]for each layer ithat make \u02c6 yfor each training\nobservation as close as possible to the true y.\nIn general, we do all this by drawing on the methods we introduced in Chapter 5\nfor logistic regression, so the reader should be comfortable with that chapter before\nproceeding.\nFirst, we\u2019ll need a loss function that models the distance between the system\noutput and the gold output, and it\u2019s common to use the loss function used for logistic\nregression, the cross-entropy loss .\nSecond, to \ufb01nd the parameters that minimize this loss function, we\u2019ll use the\ngradient descent optimization algorithm introduced in Chapter 5.\nThird, gradient descent requires knowing the gradient of the loss function, the\nvector that contains the partial derivative of the loss function with respect to each\nof the parameters. In logistic regression, for each observation we could directly\ncompute the derivative of the loss function with respect to an individual worb. But\nfor neural networks, with millions of parameters in many layers, it\u2019s much harder to\nsee how to compute the partial derivative of some weight in layer 1 when the loss\nis attached to some much later layer. How do we partial out the loss over all those\nintermediate layers? The answer is the algorithm called error backpropagation or\nbackward differentiation .\n7.5.1 Loss function\nThecross-entropy loss that is used in neural networks is the same one we saw forcross-entropy\nloss\nlogistic regression. If the neural network is being used as a binary classi\ufb01er, with\nthe sigmoid at the \ufb01nal layer, the loss function is the same logistic regression loss\nwe saw in Eq. ??:\nLCE(\u02c6y;y) =\u0000logp(yjx) =\u0000[ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)] (7.23)\nIf we are using the network to classify into 3 or more classes, the loss function is\nexactly the same as the loss for multinomial regression that we saw in Chapter 5 on 7.5 \u2022 T RAINING NEURAL NETS 15\npage ??. Let\u2019s brie\ufb02y summarize the explanation here for convenience. First, when\nwe have more than 2 classes we\u2019ll need to represent both yand\u02c6yas vectors. Let\u2019s\nassume we\u2019re doing hard classi\ufb01cation , where only one class is the correct one.\nThe true label yis then a vector with Kelements, each corresponding to a class,\nwithyc=1 if the correct class is c, with all other elements of ybeing 0. Recall that\na vector like this, with one value equal to 1 and the rest 0, is called a one-hot vector .\nAnd our classi\ufb01er will produce an estimate vector with Kelements \u02c6y, each element\n\u02c6ykof which represents the estimated probability p(yk=1jx).\nThe loss function for a single example xis the negative sum of the logs of the K\noutput classes, each weighted by their probability yk:\nLCE(\u02c6y;y) =\u0000KX\nk=1yklog\u02c6yk (7.24)\nWe can simplify this equation further; let\u2019s \ufb01rst rewrite the equation using the func-\ntion 1fgwhich evaluates to 1 if the condition in the brackets is true and to 0 oth-\nerwise. This makes it more obvious that the terms in the sum in Eq. 7.24 will be 0\nexcept for the term corresponding to the true class for which yk=1:\nLCE(\u02c6y;y) =\u0000KX\nk=11fyk=1glog\u02c6yk\nIn other words, the cross-entropy loss is simply the negative log of the output proba-\nbility corresponding to the correct class, and we therefore also call this the negative\nlog likelihood loss :negative log\nlikelihood loss\nLCE(\u02c6y;y) =\u0000log\u02c6yc(where cis the correct class) (7.25)\nPlugging in the softmax formula from Eq. 7.9, and with Kthe number of classes:\nLCE(\u02c6y;y) =\u0000logexp(zc)PK\nj=1exp(zj)(where cis the correct class) (7.26)\n7.5.2 Computing the Gradient\nHow do we compute the gradient of this loss function? Computing the gradient\nrequires the partial derivative of the loss function with respect to each parameter.\nFor a network with one weight layer and sigmoid output (which is what logistic\nregression is), we could simply use the derivative of the loss that we used for logistic\nregression in Eq. 7.27 (and derived in Section ??):\n\u00b6LCE(\u02c6y;y)\n\u00b6wj= ( \u02c6y\u0000y)xj\n= (s(w\u0001x+b)\u0000y)xj (7.27)\nOr for a network with one weight layer and softmax output (=multinomial logistic\nregression), we could use the derivative of the softmax loss from Eq. ??, shown for\na particular weight wkand input xi\n\u00b6LCE(\u02c6y;y)\n\u00b6wk;i=\u0000(yk\u0000\u02c6yk)xi\n=\u0000(yk\u0000p(yk=1jx))xi\n=\u0000 \nyk\u0000exp(wk\u0001x+bk)PK\nj=1exp(wj\u0001x+bj)!\nxi (7.28) 16 CHAPTER 7 \u2022 N EURAL NETWORKS\nBut these derivatives only give correct updates for one weight layer: the last one!\nFor deep networks, computing the gradients for each weight is much more complex,\nsince we are computing the derivative with respect to weight parameters that appear\nall the way back in the very early layers of the network, even though the loss is\ncomputed only at the very end of the network.\nThe solution to computing this gradient is an algorithm called error backprop-\nagation orbackprop (Rumelhart et al., 1986). While backprop was invented spe-error back-\npropagation\ncially for neural networks, it turns out to be the same as a more general procedure\ncalled backward differentiation , which depends on the notion of computation\ngraphs . Let\u2019s see how that works in the next subsection.\n7.5.3 Computation Graphs\nA computation graph is a representation of the process of computing a mathematical\nexpression, in which the computation is broken down into separate operations, each\nof which is modeled as a node in a graph.\nConsider computing the function L(a;b;c) =c(a+2b). If we make each of the\ncomponent addition and multiplication operations explicit, and add names ( dande)\nfor the intermediate outputs, the resulting series of computations is:\nd=2\u0003b\ne=a+d\nL=c\u0003e\nWe can now represent this as a graph, with nodes for each operation, and di-\nrected edges showing the outputs from each operation as the inputs to the next, as\nin Fig. 7.12. The simplest use of computation graphs is to compute the value of\nthe function with some given inputs. In the \ufb01gure, we\u2019ve assumed the inputs a=3,\nb=1,c=\u00002, and we\u2019ve shown the result of the forward pass to compute the re-\nsultL(3;1;\u00002) =\u000010. In the forward pass of a computation graph, we apply each\noperation left to right, passing the outputs of each computation as the input to the\nnext node.\ne=a+dd = 2bL=cea=3b=1c=-2e=5d=2L=-10forward passabc\nFigure 7.12 Computation graph for the function L(a;b;c) =c(a+2b), with values for input\nnodes a=3,b=1,c=\u00002, showing the forward pass computation of L.\n7.5.4 Backward differentiation on computation graphs\nThe importance of the computation graph comes from the backward pass , which\nis used to compute the derivatives that we\u2019ll need for the weight update. In this\nexample our goal is to compute the derivative of the output function Lwith respect 7.5 \u2022 T RAINING NEURAL NETS 17\nto each of the input variables, i.e.,\u00b6L\n\u00b6a,\u00b6L\n\u00b6b, and\u00b6L\n\u00b6c. The derivative\u00b6L\n\u00b6atells us how\nmuch a small change in aaffects L.\nBackwards differentiation makes use of the chain rule in calculus, so let\u2019s re- chain rule\nmind ourselves of that. Suppose we are computing the derivative of a composite\nfunction f(x) =u(v(x)). The derivative of f(x)is the derivative of u(x)with respect\ntov(x)times the derivative of v(x)with respect to x:\nd f\ndx=du\ndv\u0001dv\ndx(7.29)\nThe chain rule extends to more than two functions. If computing the derivative of a\ncomposite function f(x) =u(v(w(x))), the derivative of f(x)is:\nd f\ndx=du\ndv\u0001dv\ndw\u0001dw\ndx(7.30)\nThe intuition of backward differentiation is to pass gradients back from the \ufb01nal\nnode to all the nodes in the graph. Fig. 7.13 shows part of the backward computation\nat one node e. Each node takes an upstream gradient that is passed in from its parent\nnode to the right, and for each of its inputs computes a local gradient (the gradient\nof its output with respect to its input), and uses the chain rule to multiply these two\nto compute a downstream gradient to be passed on to the next earlier node.\nedLed\u2202L\u2202d\u2202L\u2202e=\u2202e\u2202d\u2202L\u2202e\u2202e\u2202dupstream gradientdownstream gradientlocal gradient\nFigure 7.13 Each node (like ehere) takes an upstream gradient, multiplies it by the local\ngradient (the gradient of its output with respect to its input), and uses the chain rule to compute\na downstream gradient to be passed on to a prior node. A node may have multiple local\ngradients if it has multiple inputs.\nLet\u2019s now compute the 3 derivatives we need. Since in the computation graph\nL=ce, we can directly compute the derivative\u00b6L\n\u00b6c:\n\u00b6L\n\u00b6c=e (7.31)\nFor the other two, we\u2019ll need to use the chain rule:\n\u00b6L\n\u00b6a=\u00b6L\n\u00b6e\u00b6e\n\u00b6a\n\u00b6L\n\u00b6b=\u00b6L\n\u00b6e\u00b6e\n\u00b6d\u00b6d\n\u00b6b(7.32)\nEq. 7.32 and Eq. 7.31 thus require \ufb01ve intermediate derivatives:\u00b6L\n\u00b6e,\u00b6L\n\u00b6c,\u00b6e\n\u00b6a,\u00b6e\n\u00b6d, and\n\u00b6d\n\u00b6b, which are as follows (making use of the fact that the derivative of a sum is the 18 CHAPTER 7 \u2022 N EURAL NETWORKS\nsum of the derivatives):\nL=ce:\u00b6L\n\u00b6e=c;\u00b6L\n\u00b6c=e\ne=a+d:\u00b6e\n\u00b6a=1;\u00b6e\n\u00b6d=1\nd=2b:\u00b6d\n\u00b6b=2\nIn the backward pass, we compute each of these partials along each edge of the\ngraph from right to left, using the chain rule just as we did above. Thus we begin by\ncomputing the downstream gradients from node L, which are\u00b6L\n\u00b6eand\u00b6L\n\u00b6c. For node e,\nwe then multiply this upstream gradient\u00b6L\n\u00b6eby the local gradient (the gradient of the\noutput with respect to the input),\u00b6e\n\u00b6dto get the output we send back to node d:\u00b6L\n\u00b6d.\nAnd so on, until we have annotated the graph all the way to all the input variables.\nThe forward pass conveniently already will have computed the values of the forward\nintermediate variables we need (like dande) to compute these derivatives. Fig. 7.14\nshows the backward pass.\ne=d+ad = 2bL=cea=3b=1e=5d=2L=-10 abc\u2202L=5\u2202c\u2202L=-2\u2202e\u2202e=1\u2202d\u2202d=2\u2202b\u2202e=1\u2202abackward passc=-2\u2202L=-2\u2202e\u2202L=5\u2202c\u2202L\u2202d=-2\u2202e\u2202d\u2202L\u2202e=\u2202L\u2202a=-2\u2202e\u2202a\u2202L\u2202e=\u2202L\u2202b=-4\u2202d\u2202b\u2202L\u2202d=\nFigure 7.14 Computation graph for the function L(a;b;c) =c(a+2b), showing the backward pass computa-\ntion of\u00b6L\n\u00b6a,\u00b6L\n\u00b6b, and\u00b6L\n\u00b6c.\nBackward differentiation for a neural network\nOf course computation graphs for real neural networks are much more complex.\nFig. 7.15 shows a sample computation graph for a 2-layer neural network with n0=\n2,n1=2, and n2=1, assuming binary classi\ufb01cation and hence using a sigmoid\noutput unit for simplicity. The function that the computation graph is computing is:\nz[1]=W[1]x+b[1]\na[1]=ReLU (z[1])\nz[2]=W[2]a[1]+b[2]\na[2]=s(z[2])\n\u02c6y=a[2](7.33) 7.5 \u2022 T RAINING NEURAL NETS 19\nFor the backward pass we\u2019ll also need to compute the loss L. The loss function\nfor binary sigmoid output from Eq. 7.23 is\nLCE(\u02c6y;y) =\u0000[ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)] (7.34)\nOur output \u02c6 y=a[2], so we can rephrase this as\nLCE(a[2];y) =\u0000h\nyloga[2]+(1\u0000y)log(1\u0000a[2])i\n(7.35)\nz[2] = +a[2] = \u03c3 a[1] = ReLUz[1] = +b[1]****x1x2a[1] = ReLUz[1] = +b[1]**w[2]11w[1]11w[1]12\nw[1]21w[1]22b[2]w[2]12L (a[2],y)1\n2111\n22\nFigure 7.15 Sample computation graph for a simple 2-layer neural net (= 1 hidden layer) with two input units\nand 2 hidden units. We\u2019ve adjusted the notation a bit to avoid long equations in the nodes by just mentioning\nthe function that is being computed, and the resulting variable name. Thus the * to the right of node w[1]\n11means\nthatw[1]\n11is to be multiplied by x1, and the node z[1]= + means that the value of z[1]is computed by summing\nthe three nodes that feed into it (the two products, and the bias term b[1]\ni).\nThe weights that need updating (those for which we need to know the partial\nderivative of the loss function) are shown in teal. In order to do the backward pass,\nwe\u2019ll need to know the derivatives of all the functions in the graph. We already saw\nin Section ??the derivative of the sigmoid s:\nds(z)\ndz=s(z)(1\u0000s(z)) (7.36)\nWe\u2019ll also need the derivatives of each of the other activation functions. The\nderivative of tanh is:\ndtanh(z)\ndz=1\u0000tanh2(z) (7.37)\nThe derivative of the ReLU is2\ndReLU (z)\ndz=\u001a0f or z <0\n1f or z\u00150(7.38)\n2The derivative is actually unde\ufb01ned at the point z=0, but by convention we treat it as 1. 20 CHAPTER 7 \u2022 N EURAL NETWORKS\nWe\u2019ll give the start of the computation, computing the derivative of the loss function\nLwith respect to z, or\u00b6L\n\u00b6z(and leaving the rest of the computation as an exercise for\nthe reader). By the chain rule:\n\u00b6L\n\u00b6z=\u00b6L\n\u00b6a[2]\u00b6a[2]\n\u00b6z(7.39)\nSo let\u2019s \ufb01rst compute\u00b6L\n\u00b6a[2], taking the derivative of Eq. 7.35, repeated here:\nLCE(a[2];y) =\u0000h\nyloga[2]+(1\u0000y)log(1\u0000a[2])i\n\u00b6L\n\u00b6a[2]=\u0000  \ny\u00b6log(a[2])\n\u00b6a[2]!\n+(1\u0000y)\u00b6log(1\u0000a[2])\n\u00b6a[2]!\n=\u0000\u0012\u0012\ny1\na[2]\u0013\n+(1\u0000y)1\n1\u0000a[2](\u00001)\u0013\n=\u0000\u0012y\na[2]+y\u00001\n1\u0000a[2]\u0013\n(7.40)\nNext, by the derivative of the sigmoid:\n\u00b6a[2]\n\u00b6z=a[2](1\u0000a[2])\nFinally, we can use the chain rule:\n\u00b6L\n\u00b6z=\u00b6L\n\u00b6a[2]\u00b6a[2]\n\u00b6z\n=\u0000\u0012y\na[2]+y\u00001\n1\u0000a[2]\u0013\na[2](1\u0000a[2])\n=a[2]\u0000y (7.41)\nContinuing the backward computation of the gradients (next by passing the gra-\ndients over b[2]\n1and the two product nodes, and so on, back to all the teal nodes), is\nleft as an exercise for the reader.\n7.5.5 More details on learning\nOptimization in neural networks is a non-convex optimization problem, more com-\nplex than for logistic regression, and for that and other reasons there are many best\npractices for successful learning.\nFor logistic regression we can initialize gradient descent with all the weights and\nbiases having the value 0. In neural networks, by contrast, we need to initialize the\nweights with small random numbers. It\u2019s also helpful to normalize the input values\nto have 0 mean and unit variance.\nVarious forms of regularization are used to prevent over\ufb01tting. One of the most\nimportant is dropout : randomly dropping some units and their connections from dropout\nthe network during training (Hinton et al. 2012, Srivastava et al. 2014). At each\niteration of training (whenever we update parameters, i.e. each mini-batch if we are\nusing mini-batch gradient descent), we repeatedly choose a probability pand for\neach unit we replace its output with zero with probability p(and renormalize the\nrest of the outputs from that layer). 7.6 \u2022 F EEDFORWARD NEURAL LANGUAGE MODELING 21\nTuning of hyperparameters is also important. The parameters of a neural net- hyperparameter\nwork are the weights Wand biases b; those are learned by gradient descent. The\nhyperparameters are things that are chosen by the algorithm designer; optimal val-\nues are tuned on a devset rather than by gradient descent learning on the training\nset. Hyperparameters include the learning rate h, the mini-batch size, the model\narchitecture (the number of layers, the number of hidden nodes per layer, the choice\nof activation functions), how to regularize, and so on. Gradient descent itself also\nhas many architectural variants such as Adam (Kingma and Ba, 2015).\nFinally, most modern neural networks are built using computation graph for-\nmalisms that make it easy and natural to do gradient computation and parallelization\non vector-based GPUs (Graphic Processing Units). PyTorch (Paszke et al., 2017)\nand TensorFlow (Abadi et al., 2015) are two of the most popular. The interested\nreader should consult a neural network textbook for further details; some sugges-\ntions are at the end of the chapter.\n7.6 Feedforward Neural Language Modeling\nAs our second application of feedforward networks, let\u2019s consider language mod-\neling : predicting upcoming words from prior words. Neural language modeling\u2014\nbased on the transformer architecture that we will see in Chapter 9\u2014is the algorithm\nthat underlies all of modern NLP. In this section and the next we\u2019ll introduce a sim-\npler version of neural language models for feedforward networks, an algorithm \ufb01rst\nintroduced by Bengio et al. (2003). The feedforward language model introduces\nmany of the important concepts of neural language modeling, concepts we\u2019ll return\nto as we describe more powerful models in Chapter 8 and Chapter 9.\nNeural language models have many advantages over the n-gram language mod-\nels of Chapter 3. Compared to n-gram models, neural language models can handle\nmuch longer histories, can generalize better over contexts of similar words, and are\nmore accurate at word-prediction. On the other hand, neural net language models\nare much more complex, are slower and need more energy to train, and are less inter-\npretable than n-gram models, so for some smaller tasks an n-gram language model\nis still the right tool.\nA feedforward neural language model (LM) is a feedforward network that takes\nas input at time ta representation of some number of previous words ( wt\u00001;wt\u00002,\netc.) and outputs a probability distribution over possible next words. Thus\u2014like the\nn-gram LM\u2014the feedforward neural LM approximates the probability of a word\ngiven the entire prior context P(wtjw1:t\u00001)by approximating based on the N\u00001\nprevious words:\nP(wtjw1;:::; wt\u00001)\u0019P(wtjwt\u0000N+1;:::; wt\u00001) (7.42)\nIn the following examples we\u2019ll use a 4-gram example, so we\u2019ll show a neural net to\nestimate the probability P(wt=ijwt\u00003;wt\u00002;wt\u00001).\nNeural language models represent words in this prior context by their embed-\ndings , rather than just by their word identity as used in n-gram language models.\nUsing embeddings allows neural language models to generalize better to unseen\ndata. For example, suppose we\u2019ve seen this sentence in training:\nI have to make sure that the cat gets fed. 22 CHAPTER 7 \u2022 N EURAL NETWORKS\nbut have never seen the words \u201cgets fed\u201d after the word \u201cdog\u201d. Our test set has the\npre\ufb01x \u201cI forgot to make sure that the dog gets\u201d. What\u2019s the next word? An n-gram\nlanguage model will predict \u201cfed\u201d after \u201cthat the cat gets\u201d, but not after \u201cthat the dog\ngets\u201d. But a neural LM, knowing that \u201ccat\u201d and \u201cdog\u201d have similar embeddings, will\nbe able to generalize from the \u201ccat\u201d context to assign a high enough probability to\n\u201cfed\u201d even after seeing \u201cdog\u201d.\n7.6.1 Forward inference in the neural language model\nLet\u2019s walk through forward inference ordecoding for neural language models.forward\ninference\nForward inference is the task, given an input, of running a forward pass on the\nnetwork to produce a probability distribution over possible outputs, in this case next\nwords.\nWe \ufb01rst represent each of the Nprevious words as a one-hot vector of length\njVj, i.e., with one dimension for each word in the vocabulary. A one-hot vector is one-hot vector\na vector that has one element equal to 1\u2014in the dimension corresponding to that\nword\u2019s index in the vocabulary\u2014 while all the other elements are set to zero. Thus\nin a one-hot representation for the word \u201ctoothpaste\u201d, supposing it is V5, i.e., index\n5 in the vocabulary, x5=1, and xi=08i6=5, as shown here:\n[0 0 0 0 1 0 0 ... 0 0 0 0]\n1 2 3 4 5 6 7 ... ... |V|\nThe feedforward neural language model (sketched in Fig. 7.17) has a moving\nwindow that can see N words into the past. We\u2019ll let N equal 3, so the 3 words\nwt\u00001,wt\u00002, and wt\u00003are each represented as a one-hot vector. We then multiply\nthese one-hot vectors by the embedding matrix E. The embedding weight matrix E\nhas a column for each word, each a column vector of ddimensions, and hence has\ndimensionality d\u0002jVj. Multiplying by a one-hot vector that has only one non-zero\nelement xi=1 simply selects out the relevant column vector for word i, resulting in\nthe embedding for word i, as shown in Fig. 7.16.\nE|V|d1|V|d1=\u271555e5\nFigure 7.16 Selecting the embedding vector for word V5by multiplying the embedding\nmatrix Ewith a one-hot vector with a 1 in index 5.\nThe 3 resulting embedding vectors are concatenated to produce e, the embedding\nlayer. This is followed by a hidden layer and an output layer whose softmax produces\na probability distribution over words. For example y42, the value of output node 42,\nis the probability of the next word wtbeing V42, the vocabulary word with index 42\n(which is the word \u2018\ufb01sh\u2019 in our example).\nHere\u2019s the algorithm in detail for our mini example:\n1.Select three embeddings from E : Given the three previous words, we look\nup their indices, create 3 one-hot vectors, and then multiply each by the em-\nbedding matrix E. Consider wt\u00003. The one-hot vector for \u2018for\u2019 (index 35) is\nmultiplied by the embedding matrix E, to give the \ufb01rst part of the \ufb01rst hidden 7.6 \u2022 F EEDFORWARD NEURAL LANGUAGE MODELING 23\nUWembedding layer3d\u2a091hiddenlayeroutput layersoftmaxdh\u2a093ddh\u2a091|V|\u2a09dhinput layerone-hot vectorsE\n|V|\u2a093d\u2a09|V|p(do|\u2026)p(aardvark|\u2026)\np(zebra|\u2026)p(fish|\u2026)\n|V|\u2a091EEh1h2y1\nh3hdh\u2026\u2026y34\ny|V|\u2026001001|V|35\n001001|V|451001001|V|9920\n0\u2026\u2026y42y35102^^^\n^^hexyforallthe?thanksand\u2026wt-3wt-2wt-1wt\u2026\nFigure 7.17 Forward inference in a feedforward neural language model. At each timestep\ntthe network computes a d-dimensional embedding for each context word (by multiplying a\none-hot vector by the embedding matrix E), and concatenates the 3 resulting embeddings to\nget the embedding layer e. The embedding vector eis multiplied by a weight matrix Wand\nthen an activation function is applied element-wise to produce the hidden layer h, which is\nthen multiplied by another weight matrix U. Finally, a softmax output layer predicts at each\nnode ithe probability that the next word wtwill be vocabulary word Vi.\nlayer, the embedding layer . Since each column of the input matrix Eis anembedding\nlayer\nembedding for a word, and the input is a one-hot column vector xifor word\nVi, the embedding layer for input wwill be Exi=ei, the embedding for word\ni. We now concatenate the three embeddings for the three context words to\nproduce the embedding layer e.\n2.Multiply by W : We multiply by W(and add b) and pass through the ReLU\n(or other) activation function to get the hidden layer h.\n3.Multiply by U :his now multiplied by U\n4.Apply softmax : After the softmax, each node iin the output layer estimates\nthe probability P(wt=ijwt\u00001;wt\u00002;wt\u00003)\nIn summary, the equations for a neural language model with a window size of 3,\ngiven one-hot input vectors for each input context word, are:\ne= [Ext\u00003;Ext\u00002;Ext\u00001]\nh=s(We+b)\nz=Uh\n\u02c6y=softmax (z) (7.43)\nNote that we formed the embedding layer eby concatenating the 3 embeddings\nfor the three context vectors; we\u2019ll often use semicolons to mean concatenation of\nvectors. 24 CHAPTER 7 \u2022 N EURAL NETWORKS\n7.7 Training the neural language model\nThe high-level intuition of training neural language models, whether the simple\nfeedforward language models we describe here or the more powerful transformer\nlanguage models of Chapter 9, is the idea of self-training orself-supervision that self-training\nwe saw in Chapter 6 for learning word representations. In self-training for language\nmodeling, we take a corpus of text as training material and at each time step task\nthe model to predict the next word. At \ufb01rst it will do poorly at this task, but since\nin each case we know the correct answer (it\u2019s the next word in the corpus!) we can\neasily train it to be better at predicting the correct next word. We call such a model\nself-supervised because we don\u2019t have to add any special gold labels to the data;\nthe natural sequence of words is its own supervision! We simply train the model to\nminimize the error in predicting the true next word in the training sequence.\nIn practice, training the model means setting the parameters q=E;W;U;b. For\nsome tasks, it\u2019s ok to freeze the embedding layer Ewith initial word2vec values. freeze\nFreezing means we use word2vec or some other pretraining algorithm to compute\nthe initial embedding matrix E, and then hold it constant while we only modify W,\nU, and b, i.e., we don\u2019t update Eduring language model training. However, often\nwe\u2019d like to learn the embeddings simultaneously with training the network. This is\nuseful when the task the network is designed for (like sentiment classi\ufb01cation, trans-\nlation, or parsing) places strong constraints on what makes a good representation for\nwords.\nLet\u2019s see how to train the entire model including E, i.e. to set all the parameters\nq=E;W;U;b. We\u2019ll do this via gradient descent (Fig. ??), using error backprop-\nagation on the computation graph to compute the gradient. Training thus not only\nsets the weights WandUof the network, but also as we\u2019re predicting upcoming\nwords, we\u2019re learning the embeddings Efor each word that best predict upcoming\nwords.\nFig. 7.18 shows the set up for a window size of N=3 context words. The input x\nconsists of 3 one-hot vectors, fully connected to the embedding layer via 3 instanti-\nations of the embedding matrix E. We don\u2019t want to learn separate weight matrices\nfor mapping each of the 3 previous words to the projection layer. We want one single\nembedding dictionary Ethat\u2019s shared among these three. That\u2019s because over time,\nmany different words will appear as wt\u00002orwt\u00001, and we\u2019d like to just represent\neach word with one vector, whichever context position it appears in. Recall that the\nembedding weight matrix Ehas a column for each word, each a column vector of d\ndimensions, and hence has dimensionality d\u0002jVj.\nGenerally training proceeds by taking as input a very long text, concatenating all\nthe sentences, starting with random weights, and then iteratively moving through the\ntext predicting each word wt. At each word wt, we use the cross-entropy (negative\nlog likelihood) loss. Recall that the general form for this (repeated from Eq. 7.25)\nis:\nLCE(\u02c6y;y) =\u0000log \u02c6yi;(where iis the correct class) (7.44)\nFor language modeling, the classes are the words in the vocabulary, so \u02c6 yihere means\nthe probability that the model assigns to the correct next word wt:\nLCE=\u0000logp(wtjwt\u00001;:::;wt\u0000n+1) (7.45)\nThe parameter update for stochastic gradient descent for this loss from step stos+1 7.8 \u2022 S UMMARY 25\nUWembedding layer3d\u2a091hiddenlayeroutput layersoftmaxdh\u2a093ddh\u2a091|V|\u2a09dhinput layerone-hot vectorsE\n|V|\u2a093d\u2a09|V|p(do|\u2026)p(aardvark|\u2026)\np(zebra|\u2026)p(fish|\u2026)\n|V|\u2a091EEh1h2y1\nh3hdh\u2026\u2026y34\ny|V|\u2026001001|V|35\n001001|V|451001001|V|9920\n0\u2026\u2026y42y35102^^^\n^^hexyforallthefishthanksand\u2026wt-3wt-2wt-1wt\u2026L = \u2212log P(fish | for, all, the)wt=fish\nFigure 7.18 Learning all the way back to embeddings. Again, the embedding matrix Eis\nshared among the 3 context words.\nis then:\nqs+1=qs\u0000h\u00b6[\u0000logp(wtjwt\u00001;:::;wt\u0000n+1)]\n\u00b6q(7.46)\nThis gradient can be computed in any standard neural network framework which\nwill then backpropagate through q=E;W;U;b.\nTraining the parameters to minimize loss will result both in an algorithm for\nlanguage modeling (a word predictor) but also a new set of embeddings Ethat can\nbe used as word representations for other tasks.\n7.8 Summary\n\u2022 Neural networks are built out of neural units , originally inspired by biological\nneurons but now simply an abstract computational device.\n\u2022 Each neural unit multiplies input values by a weight vector, adds a bias, and\nthen applies a non-linear activation function like sigmoid, tanh, or recti\ufb01ed\nlinear unit.\n\u2022 In a fully-connected ,feedforward network, each unit in layer iis connected\nto each unit in layer i+1, and there are no cycles.\n\u2022 The power of neural networks comes from the ability of early layers to learn\nrepresentations that can be utilized by later layers in the network.\n\u2022 Neural networks are trained by optimization algorithms like gradient de-\nscent .\n\u2022Error backpropagation , backward differentiation on a computation graph ,\nis used to compute the gradients of the loss function for a network. 26 CHAPTER 7 \u2022 N EURAL NETWORKS\n\u2022Neural language models use a neural network as a probabilistic classi\ufb01er, to\ncompute the probability of the next word given the previous nwords.\n\u2022 Neural language models can use pretrained embeddings , or can learn embed-\ndings from scratch in the process of language modeling.\nBibliographical and Historical Notes\nThe origins of neural networks lie in the 1940s McCulloch-Pitts neuron (McCul-\nloch and Pitts, 1943), a simpli\ufb01ed model of the biological neuron as a kind of com-\nputing element that could be described in terms of propositional logic. By the late\n1950s and early 1960s, a number of labs (including Frank Rosenblatt at Cornell and\nBernard Widrow at Stanford) developed research into neural networks; this phase\nsaw the development of the perceptron (Rosenblatt, 1958), and the transformation\nof the threshold into a bias, a notation we still use (Widrow and Hoff, 1960).\nThe \ufb01eld of neural networks declined after it was shown that a single perceptron\nunit was unable to model functions as simple as XOR (Minsky and Papert, 1969).\nWhile some small amount of work continued during the next two decades, a major\nrevival for the \ufb01eld didn\u2019t come until the 1980s, when practical tools for building\ndeeper networks like error backpropagation became widespread (Rumelhart et al.,\n1986). During the 1980s a wide variety of neural network and related architec-\ntures were developed, particularly for applications in psychology and cognitive sci-\nence (Rumelhart and McClelland 1986b, McClelland and Elman 1986, Rumelhart\nand McClelland 1986a, Elman 1990), for which the term connectionist orparal- connectionist\nlel distributed processing was often used (Feldman and Ballard 1982, Smolensky\n1988). Many of the principles and techniques developed in this period are foun-\ndational to modern work, including the ideas of distributed representations (Hinton,\n1986), recurrent networks (Elman, 1990), and the use of tensors for compositionality\n(Smolensky, 1990).\nBy the 1990s larger neural networks began to be applied to many practical lan-\nguage processing tasks as well, like handwriting recognition (LeCun et al. 1989) and\nspeech recognition (Morgan and Bourlard 1990). By the early 2000s, improvements\nin computer hardware and advances in optimization and training techniques made it\npossible to train even larger and deeper networks, leading to the modern term deep\nlearning (Hinton et al. 2006, Bengio et al. 2007). We cover more related history in\nChapter 8 and Chapter 16.\nThere are a number of excellent books on the subject. Goldberg (2017) has\nsuperb coverage of neural networks for natural language processing. For neural\nnetworks in general see Goodfellow et al. (2016) and Nielsen (2015). Bibliographical and Historical Notes 27\nAbadi, M., A. Agarwal, P. Barham, E. Brevdo, Z. Chen,\nC. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin,\nS. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Is-\nard, Y . Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Leven-\nberg, D. Man \u00b4e, R. Monga, S. Moore, D. Murray, C. Olah,\nM. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Tal-\nwar, P. Tucker, V . Vanhoucke, V . Vasudevan, F. Vi \u00b4egas,\nO. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y . Yu,\nand X. Zheng. 2015. TensorFlow: Large-scale machine\nlearning on heterogeneous systems. Software available\nfrom tensor\ufb02ow.org.\nBengio, Y ., R. Ducharme, P. Vincent, and C. Jauvin. 2003.\nA neural probabilistic language model. JMLR , 3:1137\u2013\n1155.\nBengio, Y ., P. Lamblin, D. Popovici, and H. Larochelle.\n2007. Greedy layer-wise training of deep networks.\nNeurIPS .\nElman, J. L. 1990. Finding structure in time. Cognitive sci-\nence, 14(2):179\u2013211.\nFeldman, J. A. and D. H. Ballard. 1982. Connectionist mod-\nels and their properties. Cognitive Science , 6:205\u2013254.\nGoldberg, Y . 2017. Neural Network Methods for Natural\nLanguage Processing , volume 10 of Synthesis Lectures\non Human Language Technologies . Morgan & Claypool.\nGoodfellow, I., Y . Bengio, and A. Courville. 2016. Deep\nLearning . MIT Press.\nHinton, G. E. 1986. Learning distributed representations of\nconcepts. COGSCI .\nHinton, G. E., S. Osindero, and Y .-W. Teh. 2006. A fast\nlearning algorithm for deep belief nets. Neural computa-\ntion, 18(7):1527\u20131554.\nHinton, G. E., N. Srivastava, A. Krizhevsky, I. Sutskever, and\nR. R. Salakhutdinov. 2012. Improving neural networks\nby preventing co-adaptation of feature detectors. ArXiv\npreprint arXiv:1207.0580.\nKingma, D. and J. Ba. 2015. Adam: A method for stochastic\noptimization. ICLR 2015 .\nLeCun, Y ., B. Boser, J. S. Denker, D. Henderson, R. E.\nHoward, W. Hubbard, and L. D. Jackel. 1989. Backprop-\nagation applied to handwritten zip code recognition. Neu-\nral computation , 1(4):541\u2013551.\nMcClelland, J. L. and J. L. Elman. 1986. The TRACE model\nof speech perception. Cognitive Psychology , 18:1\u201386.\nMcCulloch, W. S. and W. Pitts. 1943. A logical calculus of\nideas immanent in nervous activity. Bulletin of Mathe-\nmatical Biophysics , 5:115\u2013133.\nMinsky, M. and S. Papert. 1969. Perceptrons . MIT Press.\nMorgan, N. and H. Bourlard. 1990. Continuous speech\nrecognition using multilayer perceptrons with hidden\nmarkov models. ICASSP .\nNielsen, M. A. 2015. Neural networks and Deep learning .\nDetermination Press USA.\nPaszke, A., S. Gross, S. Chintala, G. Chanan, E. Yang, Z. De-\nVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer.\n2017. Automatic differentiation in pytorch. NIPS-W .\nRosenblatt, F. 1958. The perceptron: A probabilistic model\nfor information storage and organization in the brain. Psy-\nchological review , 65(6):386\u2013408.Rumelhart, D. E., G. E. Hinton, and R. J. Williams. 1986.\nLearning internal representations by error propagation. In\nD. E. Rumelhart and J. L. McClelland, eds, Parallel Dis-\ntributed Processing , volume 2, 318\u2013362. MIT Press.\nRumelhart, D. E. and J. L. McClelland. 1986a. On learning\nthe past tense of English verbs. In D. E. Rumelhart and\nJ. L. McClelland, eds, Parallel Distributed Processing ,\nvolume 2, 216\u2013271. MIT Press.\nRumelhart, D. E. and J. L. McClelland, eds. 1986b. Parallel\nDistributed Processing . MIT Press.\nRussell, S. and P. Norvig. 2002. Arti\ufb01cial Intelligence: A\nModern Approach , 2nd edition. Prentice Hall.\nSmolensky, P. 1988. On the proper treatment of connection-\nism. Behavioral and brain sciences , 11(1):1\u201323.\nSmolensky, P. 1990. Tensor product variable binding and\nthe representation of symbolic structures in connectionist\nsystems. Arti\ufb01cial intelligence , 46(1-2):159\u2013216.\nSrivastava, N., G. E. Hinton, A. Krizhevsky, I. Sutskever,\nand R. R. Salakhutdinov. 2014. Dropout: a simple\nway to prevent neural networks from over\ufb01tting. JMLR ,\n15(1):1929\u20131958.\nWidrow, B. and M. E. Hoff. 1960. Adaptive switching cir-\ncuits. IRE WESCON Convention Record , volume 4.", "7_NN_Apr_28_2021.pdf": "Simple Neural Networks and Neural Language ModelsUnits in Neural Networks This is in your brain\n2\nBy BruceBlaus-Own work, CC BY 3.0, https://commons.wikimedia.org/w/index.php?curid=28761830 Neural Network UnitThis is not in your brain\n3x1x2x3\ny\nw1w2w3\u2211b\u03c3+1zaWeightsInput layerWeighted sumNon-linear transformOutput valuebias Neural unitTake weighted sum of inputs, plus a biasInstead of just using z, we'll apply a nonlinear activation function f:2CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELS7.1 UnitsThe building block of a neural network is a single computational unit. A unit takesa set of real valued numbers as input, performs some computation on them, andproduces an output.At its heart, a neural unit is taking a weighted sum of its inputs, with one addi-tional term in the sum called abias term. Given a set of inputsx1...xn, a unit hasbias terma set of corresponding weightsw1...wnand a biasb, so the weighted sumzcan berepresented as:z=b+Xiwixi(7.1)Often it\u2019s more convenient to express this weighted sum using vector notation; recallfrom linear algebra that avectoris, at heart, just a list or array of numbers. Thusvectorwe\u2019ll talk aboutzin terms of a weight vectorw, a scalar biasb, and an input vectorx, and we\u2019ll replace the sum with the convenientdot product:z=w\u00b7x+b(7.2)As de\ufb01ned in Eq.7.2,zis just a real valued number.Finally, instead of usingz, a linear function ofx, as the output, neural unitsapply a non-linear functionftoz. We will refer to the output of this function astheactivationvalue for the unit,a. Since we are just modeling a single unit, theactivationactivation for the node is in fact the \ufb01nal output of the network, which we\u2019ll generallycally. So the valueyis de\ufb01ned as:y=a=f(z)We\u2019ll discuss three popular non-linear functionsf()below (the sigmoid, the tanh,and the recti\ufb01ed linear ReLU) but it\u2019s pedagogically convenient to start with thesigmoidfunction since we saw it in Chapter 5:sigmoidy=s(z)=11+e\u0000z(7.3)The sigmoid (shown in Fig.7.1) has a number of advantages; it maps the outputinto the range[0,1], which is useful in squashing outliers toward 0 or 1. And it\u2019sdifferentiable, which as we saw in Section??will be handy for learning.\nFigure 7.1The sigmoid function takes a real value and maps it to the range[0,1]. It isnearly linear around 0 but outlier values get squashed toward 0 or 1.2CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELS7.1 UnitsThe building block of a neural network is a single computational unit. A unit takesa set of real valued numbers as input, performs some computation on them, andproduces an output.At its heart, a neural unit is taking a weighted sum of its inputs, with one addi-tional term in the sum called abias term. Given a set of inputsx1...xn, a unit hasbias terma set of corresponding weightsw1...wnand a biasb, so the weighted sumzcan berepresented as:z=b+Xiwixi(7.1)Often it\u2019s more convenient to express this weighted sum using vector notation; recallfrom linear algebra that avectoris, at heart, just a list or array of numbers. Thusvectorwe\u2019ll talk aboutzin terms of a weight vectorw, a scalar biasb, and an input vectorx, and we\u2019ll replace the sum with the convenientdot product:z=w\u00b7x+b(7.2)As de\ufb01ned in Eq.7.2,zis just a real valued number.Finally, instead of usingz, a linear function ofx, as the output, neural unitsapply a non-linear functionftoz. We will refer to the output of this function astheactivationvalue for the unit,a. Since we are just modeling a single unit, theactivationactivation for the node is in fact the \ufb01nal output of the network, which we\u2019ll generallycally. So the valueyis de\ufb01ned as:y=a=f(z)We\u2019ll discuss three popular non-linear functionsf()below (the sigmoid, the tanh,and the recti\ufb01ed linear ReLU) but it\u2019s pedagogically convenient to start with thesigmoidfunction since we saw it in Chapter 5:sigmoidy=s(z)=11+e\u0000z(7.3)The sigmoid (shown in Fig.7.1) has a number of advantages; it maps the outputinto the range[0,1], which is useful in squashing outliers toward 0 or 1. And it\u2019sdifferentiable, which as we saw in Section??will be handy for learning.\nFigure 7.1The sigmoid function takes a real value and maps it to the range[0,1]. It isnearly linear around 0 but outlier values get squashed toward 0 or 1.2CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELS7.1 UnitsThe building block of a neural network is a single computational unit. A unit takesa set of real valued numbers as input, performs some computation on them, andproduces an output.At its heart, a neural unit is taking a weighted sum of its inputs, with one addi-tional term in the sum called abias term. Given a set of inputsx1...xn, a unit hasbias terma set of corresponding weightsw1...wnand a biasb, so the weighted sumzcan berepresented as:z=b+Xiwixi(7.1)Often it\u2019s more convenient to express this weighted sum using vector notation; recallfrom linear algebra that avectoris, at heart, just a list or array of numbers. Thusvectorwe\u2019ll talk aboutzin terms of a weight vectorw, a scalar biasb, and an input vectorx, and we\u2019ll replace the sum with the convenientdot product:z=w\u00b7x+b(7.2)As de\ufb01ned in Eq.7.2,zis just a real valued number.Finally, instead of usingz, a linear function ofx, as the output, neural unitsapply a non-linear functionftoz. We will refer to the output of this function astheactivationvalue for the unit,a. Since we are just modeling a single unit, theactivationactivation for the node is in fact the \ufb01nal output of the network, which we\u2019ll generallycally. So the valueyis de\ufb01ned as:y=a=f(z)We\u2019ll discuss three popular non-linear functionsf()below (the sigmoid, the tanh,and the recti\ufb01ed linear ReLU) but it\u2019s pedagogically convenient to start with thesigmoidfunction since we saw it in Chapter 5:sigmoidy=s(z)=11+e\u0000z(7.3)The sigmoid (shown in Fig.7.1) has a number of advantages; it maps the outputinto the range[0,1], which is useful in squashing outliers toward 0 or 1. And it\u2019sdifferentiable, which as we saw in Section??will be handy for learning.\nFigure 7.1The sigmoid function takes a real value and maps it to the range[0,1]. It isnearly linear around 0 but outlier values get squashed toward 0 or 1. Non-Linear Activation Functions\n5\nSigmoidWe're already seen the sigmoid for logistic regression:2CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELS7.1 UnitsThe building block of a neural network is a single computational unit. A unit takesa set of real valued numbers as input, performs some computation on them, andproduces an output.At its heart, a neural unit is taking a weighted sum of its inputs, with one addi-tional term in the sum called abias term. Given a set of inputsx1...xn, a unit hasbias terma set of corresponding weightsw1...wnand a biasb, so the weighted sumzcan berepresented as:z=b+Xiwixi(7.1)Often it\u2019s more convenient to express this weighted sum using vector notation; recallfrom linear algebra that avectoris, at heart, just a list or array of numbers. Thusvectorwe\u2019ll talk aboutzin terms of a weight vectorw, a scalar biasb, and an input vectorx, and we\u2019ll replace the sum with the convenientdot product:z=w\u00b7x+b(7.2)As de\ufb01ned in Eq.7.2,zis just a real valued number.Finally, instead of usingz, a linear function ofx, as the output, neural unitsapply a non-linear functionftoz. We will refer to the output of this function astheactivationvalue for the unit,a. Since we are just modeling a single unit, theactivationactivation for the node is in fact the \ufb01nal output of the network, which we\u2019ll generallycally. So the valueyis de\ufb01ned as:y=a=f(z)We\u2019ll discuss three popular non-linear functionsf()below (the sigmoid, the tanh,and the recti\ufb01ed linear ReLU) but it\u2019s pedagogically convenient to start with thesigmoidfunction since we saw it in Chapter 5:sigmoidy=s(z)=11+e\u0000z(7.3)The sigmoid (shown in Fig.7.1) has a number of advantages; it maps the outputinto the range[0,1], which is useful in squashing outliers toward 0 or 1. And it\u2019sdifferentiable, which as we saw in Section??will be handy for learning.\nFigure 7.1The sigmoid function takes a real value and maps it to the range[0,1]. It isnearly linear around 0 but outlier values get squashed toward 0 or 1. Final function the unit is computing7.1\u2022UNITS3Substituting Eq.7.2into Eq.7.3gives us the output of a neural unit:y=s(w\u00b7x+b)=11+exp(\u0000(w\u00b7x+b))(7.4)Fig.7.2shows a \ufb01nal schematic of a basic neural unit. In this example the unittakes 3 input valuesx1,x2, andx3, and computes a weighted sum, multiplying eachvalue by a weight (w1,w2, andw3, respectively), adds them to a bias termb, and thenpasses the resulting sum through a sigmoid function to result in a number between 0and 1.\nx1x2x3\ny\nw1w2w3\u2211b\u03c3+1za\nFigure 7.2A neural unit, taking 3 inputsx1,x2, andx3(and a biasbthat we represent as aweight for an input clamped at +1) and producing an output y. We include some convenientintermediate variables: the output of the summation,z, and the output of the sigmoid,a. Inthis case the output of the unityis the same asa, but in deeper networks we\u2019ll reserveytomean the \ufb01nal output of the entire network, leavingaas the activation of an individual node.Let\u2019s walk through an example just to get an intuition. Let\u2019s suppose we have aunit with the following weight vector and bias:w=[0.2,0.3,0.9]b=0.5What would this unit do with the following input vector:x=[0.5,0.6,0.1]The resulting outputywould be:y=s(w\u00b7x+b)=11+e\u0000(w\u00b7x+b)=11+e\u0000(.5\u21e4.2+.6\u21e4.3+.1\u21e4.9+.5)=11+e\u00000.87=.70In practice, the sigmoid is not commonly used as an activation function. A functionthat is very similar but almost always better is thetanhfunction shown in Fig.7.3a;tanhtanh is a variant of the sigmoid that ranges from -1 to +1:y=ez\u0000e\u0000zez+e\u0000z(7.5)The simplest activation function, and perhaps the most commonly used, is the rec-ti\ufb01ed linear unit, also called theReLU, shown in Fig.7.3b. It\u2019s just the same asxReLUwhenxis positive, and 0 otherwise:y=max(x,0)(7.6) Final unit again\n7x1x2x3\ny\nw1w2w3\u2211b\u03c3+1zaWeightsInput layerWeighted sumNon-linear activation functionOutput valuebias An exampleSuppose a unit has:w = [0.2,0.3,0.9] b = 0.5 What happens with input x:x = [0.5,0.6,0.1] 7.1\u2022UNITS3Substituting Eq.7.2into Eq.7.3gives us the output of a neural unit:y=s(w\u00b7x+b)=11+exp(\u0000(w\u00b7x+b))(7.4)Fig.7.2shows a \ufb01nal schematic of a basic neural unit. In this example the unittakes 3 input valuesx1,x2, andx3, and computes a weighted sum, multiplying eachvalue by a weight (w1,w2, andw3, respectively), adds them to a bias termb, and thenpasses the resulting sum through a sigmoid function to result in a number between 0and 1.\nx1x2x3\ny\nw1w2w3\u2211b\u03c3+1za\nFigure 7.2A neural unit, taking 3 inputsx1,x2, andx3(and a biasbthat we represent as aweight for an input clamped at +1) and producing an output y. We include some convenientintermediate variables: the output of the summation,z, and the output of the sigmoid,a. Inthis case the output of the unityis the same asa, but in deeper networks we\u2019ll reserveytomean the \ufb01nal output of the entire network, leavingaas the activation of an individual node.Let\u2019s walk through an example just to get an intuition. Let\u2019s suppose we have aunit with the following weight vector and bias:w=[0.2,0.3,0.9]b=0.5What would this unit do with the following input vector:x=[0.5,0.6,0.1]The resulting outputywould be:y=s(w\u00b7x+b)=11+e\u0000(w\u00b7x+b)=11+e\u0000(.5\u21e4.2+.6\u21e4.3+.1\u21e4.9+.5)=11+e\u00000.87=.70In practice, the sigmoid is not commonly used as an activation function. A functionthat is very similar but almost always better is thetanhfunction shown in Fig.7.3a;tanhtanh is a variant of the sigmoid that ranges from -1 to +1:y=ez\u0000e\u0000zez+e\u0000z(7.5)The simplest activation function, and perhaps the most commonly used, is the rec-ti\ufb01ed linear unit, also called theReLU, shown in Fig.7.3b. It\u2019s just the same asxReLUwhenxis positive, and 0 otherwise:y=max(x,0)(7.6) An exampleSuppose a unit has:w = [0.2,0.3,0.9] b = 0.5 What happens with the following input x?x = [0.5,0.6,0.1] 7.1\u2022UNITS3Substituting Eq.7.2into Eq.7.3gives us the output of a neural unit:y=s(w\u00b7x+b)=11+exp(\u0000(w\u00b7x+b))(7.4)Fig.7.2shows a \ufb01nal schematic of a basic neural unit. In this example the unittakes 3 input valuesx1,x2, andx3, and computes a weighted sum, multiplying eachvalue by a weight (w1,w2, andw3, respectively), adds them to a bias termb, and thenpasses the resulting sum through a sigmoid function to result in a number between 0and 1.\nx1x2x3\ny\nw1w2w3\u2211b\u03c3+1za\nFigure 7.2A neural unit, taking 3 inputsx1,x2, andx3(and a biasbthat we represent as aweight for an input clamped at +1) and producing an output y. We include some convenientintermediate variables: the output of the summation,z, and the output of the sigmoid,a. Inthis case the output of the unityis the same asa, but in deeper networks we\u2019ll reserveytomean the \ufb01nal output of the entire network, leavingaas the activation of an individual node.Let\u2019s walk through an example just to get an intuition. Let\u2019s suppose we have aunit with the following weight vector and bias:w=[0.2,0.3,0.9]b=0.5What would this unit do with the following input vector:x=[0.5,0.6,0.1]The resulting outputywould be:y=s(w\u00b7x+b)=11+e\u0000(w\u00b7x+b)=11+e\u0000(.5\u21e4.2+.6\u21e4.3+.1\u21e4.9+.5)=11+e\u00000.87=.70In practice, the sigmoid is not commonly used as an activation function. A functionthat is very similar but almost always better is thetanhfunction shown in Fig.7.3a;tanhtanh is a variant of the sigmoid that ranges from -1 to +1:y=ez\u0000e\u0000zez+e\u0000z(7.5)The simplest activation function, and perhaps the most commonly used, is the rec-ti\ufb01ed linear unit, also called theReLU, shown in Fig.7.3b. It\u2019s just the same asxReLUwhenxis positive, and 0 otherwise:y=max(x,0)(7.6)7.1\u2022UNITS3Substituting Eq.7.2into Eq.7.3gives us the output of a neural unit:y=s(w\u00b7x+b)=11+exp(\u0000(w\u00b7x+b))(7.4)Fig.7.2shows a \ufb01nal schematic of a basic neural unit. In this example the unittakes 3 input valuesx1,x2, andx3, and computes a weighted sum, multiplying eachvalue by a weight (w1,w2, andw3, respectively), adds them to a bias termb, and thenpasses the resulting sum through a sigmoid function to result in a number between 0and 1.\nx1x2x3\ny\nw1w2w3\u2211b\u03c3+1za\nFigure 7.2A neural unit, taking 3 inputsx1,x2, andx3(and a biasbthat we represent as aweight for an input clamped at +1) and producing an output y. We include some convenientintermediate variables: the output of the summation,z, and the output of the sigmoid,a. Inthis case the output of the unityis the same asa, but in deeper networks we\u2019ll reserveytomean the \ufb01nal output of the entire network, leavingaas the activation of an individual node.Let\u2019s walk through an example just to get an intuition. Let\u2019s suppose we have aunit with the following weight vector and bias:w=[0.2,0.3,0.9]b=0.5What would this unit do with the following input vector:x=[0.5,0.6,0.1]The resulting outputywould be:y=s(w\u00b7x+b)=11+e\u0000(w\u00b7x+b)=11+e\u0000(.5\u21e4.2+.6\u21e4.3+.1\u21e4.9+.5)=11+e\u00000.87=.70In practice, the sigmoid is not commonly used as an activation function. A functionthat is very similar but almost always better is thetanhfunction shown in Fig.7.3a;tanhtanh is a variant of the sigmoid that ranges from -1 to +1:y=ez\u0000e\u0000zez+e\u0000z(7.5)The simplest activation function, and perhaps the most commonly used, is the rec-ti\ufb01ed linear unit, also called theReLU, shown in Fig.7.3b. It\u2019s just the same asxReLUwhenxis positive, and 0 otherwise:y=max(x,0)(7.6) An exampleSuppose a unit has:w = [0.2,0.3,0.9] b = 0.5 What happens with input x:x = [0.5,0.6,0.1] 7.1\u2022UNITS3Substituting Eq.7.2into Eq.7.3gives us the output of a neural unit:y=s(w\u00b7x+b)=11+exp(\u0000(w\u00b7x+b))(7.4)Fig.7.2shows a \ufb01nal schematic of a basic neural unit. In this example the unittakes 3 input valuesx1,x2, andx3, and computes a weighted sum, multiplying eachvalue by a weight (w1,w2, andw3, respectively), adds them to a bias termb, and thenpasses the resulting sum through a sigmoid function to result in a number between 0and 1.\nx1x2x3\ny\nw1w2w3\u2211b\u03c3+1za\nFigure 7.2A neural unit, taking 3 inputsx1,x2, andx3(and a biasbthat we represent as aweight for an input clamped at +1) and producing an output y. We include some convenientintermediate variables: the output of the summation,z, and the output of the sigmoid,a. Inthis case the output of the unityis the same asa, but in deeper networks we\u2019ll reserveytomean the \ufb01nal output of the entire network, leavingaas the activation of an individual node.Let\u2019s walk through an example just to get an intuition. Let\u2019s suppose we have aunit with the following weight vector and bias:w=[0.2,0.3,0.9]b=0.5What would this unit do with the following input vector:x=[0.5,0.6,0.1]The resulting outputywould be:y=s(w\u00b7x+b)=11+e\u0000(w\u00b7x+b)=11+e\u0000(.5\u21e4.2+.6\u21e4.3+.1\u21e4.9+.5)=11+e\u00000.87=.70In practice, the sigmoid is not commonly used as an activation function. A functionthat is very similar but almost always better is thetanhfunction shown in Fig.7.3a;tanhtanh is a variant of the sigmoid that ranges from -1 to +1:y=ez\u0000e\u0000zez+e\u0000z(7.5)The simplest activation function, and perhaps the most commonly used, is the rec-ti\ufb01ed linear unit, also called theReLU, shown in Fig.7.3b. It\u2019s just the same asxReLUwhenxis positive, and 0 otherwise:y=max(x,0)(7.6)7.1\u2022UNITS3Substituting Eq.7.2into Eq.7.3gives us the output of a neural unit:y=s(w\u00b7x+b)=11+exp(\u0000(w\u00b7x+b))(7.4)Fig.7.2shows a \ufb01nal schematic of a basic neural unit. In this example the unittakes 3 input valuesx1,x2, andx3, and computes a weighted sum, multiplying eachvalue by a weight (w1,w2, andw3, respectively), adds them to a bias termb, and thenpasses the resulting sum through a sigmoid function to result in a number between 0and 1.\nx1x2x3\ny\nw1w2w3\u2211b\u03c3+1za\nFigure 7.2A neural unit, taking 3 inputsx1,x2, andx3(and a biasbthat we represent as aweight for an input clamped at +1) and producing an output y. We include some convenientintermediate variables: the output of the summation,z, and the output of the sigmoid,a. Inthis case the output of the unityis the same asa, but in deeper networks we\u2019ll reserveytomean the \ufb01nal output of the entire network, leavingaas the activation of an individual node.Let\u2019s walk through an example just to get an intuition. Let\u2019s suppose we have aunit with the following weight vector and bias:w=[0.2,0.3,0.9]b=0.5What would this unit do with the following input vector:x=[0.5,0.6,0.1]The resulting outputywould be:y=s(w\u00b7x+b)=11+e\u0000(w\u00b7x+b)=11+e\u0000(.5\u21e4.2+.6\u21e4.3+.1\u21e4.9+.5)=11+e\u00000.87=.70In practice, the sigmoid is not commonly used as an activation function. A functionthat is very similar but almost always better is thetanhfunction shown in Fig.7.3a;tanhtanh is a variant of the sigmoid that ranges from -1 to +1:y=ez\u0000e\u0000zez+e\u0000z(7.5)The simplest activation function, and perhaps the most commonly used, is the rec-ti\ufb01ed linear unit, also called theReLU, shown in Fig.7.3b. It\u2019s just the same asxReLUwhenxis positive, and 0 otherwise:y=max(x,0)(7.6)7.1\u2022UNITS3Substituting Eq.7.2into Eq.7.3gives us the output of a neural unit:y=s(w\u00b7x+b)=11+exp(\u0000(w\u00b7x+b))(7.4)Fig.7.2shows a \ufb01nal schematic of a basic neural unit. In this example the unittakes 3 input valuesx1,x2, andx3, and computes a weighted sum, multiplying eachvalue by a weight (w1,w2, andw3, respectively), adds them to a bias termb, and thenpasses the resulting sum through a sigmoid function to result in a number between 0and 1.\nx1x2x3\ny\nw1w2w3\u2211b\u03c3+1za\nFigure 7.2A neural unit, taking 3 inputsx1,x2, andx3(and a biasbthat we represent as aweight for an input clamped at +1) and producing an output y. We include some convenientintermediate variables: the output of the summation,z, and the output of the sigmoid,a. Inthis case the output of the unityis the same asa, but in deeper networks we\u2019ll reserveytomean the \ufb01nal output of the entire network, leavingaas the activation of an individual node.Let\u2019s walk through an example just to get an intuition. Let\u2019s suppose we have aunit with the following weight vector and bias:w=[0.2,0.3,0.9]b=0.5What would this unit do with the following input vector:x=[0.5,0.6,0.1]The resulting outputywould be:y=s(w\u00b7x+b)=11+e\u0000(w\u00b7x+b)=11+e\u0000(.5\u21e4.2+.6\u21e4.3+.1\u21e4.9+.5)=11+e\u00000.87=.70In practice, the sigmoid is not commonly used as an activation function. A functionthat is very similar but almost always better is thetanhfunction shown in Fig.7.3a;tanhtanh is a variant of the sigmoid that ranges from -1 to +1:y=ez\u0000e\u0000zez+e\u0000z(7.5)The simplest activation function, and perhaps the most commonly used, is the rec-ti\ufb01ed linear unit, also called theReLU, shown in Fig.7.3b. It\u2019s just the same asxReLUwhenxis positive, and 0 otherwise:y=max(x,0)(7.6) An exampleSuppose a unit has:w = [0.2,0.3,0.9] b = 0.5 What happens with input x:x = [0.5,0.6,0.1] 7.1\u2022UNITS3Substituting Eq.7.2into Eq.7.3gives us the output of a neural unit:y=s(w\u00b7x+b)=11+exp(\u0000(w\u00b7x+b))(7.4)Fig.7.2shows a \ufb01nal schematic of a basic neural unit. In this example the unittakes 3 input valuesx1,x2, andx3, and computes a weighted sum, multiplying eachvalue by a weight (w1,w2, andw3, respectively), adds them to a bias termb, and thenpasses the resulting sum through a sigmoid function to result in a number between 0and 1.\nx1x2x3\ny\nw1w2w3\u2211b\u03c3+1za\nFigure 7.2A neural unit, taking 3 inputsx1,x2, andx3(and a biasbthat we represent as aweight for an input clamped at +1) and producing an output y. We include some convenientintermediate variables: the output of the summation,z, and the output of the sigmoid,a. Inthis case the output of the unityis the same asa, but in deeper networks we\u2019ll reserveytomean the \ufb01nal output of the entire network, leavingaas the activation of an individual node.Let\u2019s walk through an example just to get an intuition. Let\u2019s suppose we have aunit with the following weight vector and bias:w=[0.2,0.3,0.9]b=0.5What would this unit do with the following input vector:x=[0.5,0.6,0.1]The resulting outputywould be:y=s(w\u00b7x+b)=11+e\u0000(w\u00b7x+b)=11+e\u0000(.5\u21e4.2+.6\u21e4.3+.1\u21e4.9+.5)=11+e\u00000.87=.70In practice, the sigmoid is not commonly used as an activation function. A functionthat is very similar but almost always better is thetanhfunction shown in Fig.7.3a;tanhtanh is a variant of the sigmoid that ranges from -1 to +1:y=ez\u0000e\u0000zez+e\u0000z(7.5)The simplest activation function, and perhaps the most commonly used, is the rec-ti\ufb01ed linear unit, also called theReLU, shown in Fig.7.3b. It\u2019s just the same asxReLUwhenxis positive, and 0 otherwise:y=max(x,0)(7.6)7.1\u2022UNITS3Substituting Eq.7.2into Eq.7.3gives us the output of a neural unit:y=s(w\u00b7x+b)=11+exp(\u0000(w\u00b7x+b))(7.4)Fig.7.2shows a \ufb01nal schematic of a basic neural unit. In this example the unittakes 3 input valuesx1,x2, andx3, and computes a weighted sum, multiplying eachvalue by a weight (w1,w2, andw3, respectively), adds them to a bias termb, and thenpasses the resulting sum through a sigmoid function to result in a number between 0and 1.\nx1x2x3\ny\nw1w2w3\u2211b\u03c3+1za\nFigure 7.2A neural unit, taking 3 inputsx1,x2, andx3(and a biasbthat we represent as aweight for an input clamped at +1) and producing an output y. We include some convenientintermediate variables: the output of the summation,z, and the output of the sigmoid,a. Inthis case the output of the unityis the same asa, but in deeper networks we\u2019ll reserveytomean the \ufb01nal output of the entire network, leavingaas the activation of an individual node.Let\u2019s walk through an example just to get an intuition. Let\u2019s suppose we have aunit with the following weight vector and bias:w=[0.2,0.3,0.9]b=0.5What would this unit do with the following input vector:x=[0.5,0.6,0.1]The resulting outputywould be:y=s(w\u00b7x+b)=11+e\u0000(w\u00b7x+b)=11+e\u0000(.5\u21e4.2+.6\u21e4.3+.1\u21e4.9+.5)=11+e\u00000.87=.70In practice, the sigmoid is not commonly used as an activation function. A functionthat is very similar but almost always better is thetanhfunction shown in Fig.7.3a;tanhtanh is a variant of the sigmoid that ranges from -1 to +1:y=ez\u0000e\u0000zez+e\u0000z(7.5)The simplest activation function, and perhaps the most commonly used, is the rec-ti\ufb01ed linear unit, also called theReLU, shown in Fig.7.3b. It\u2019s just the same asxReLUwhenxis positive, and 0 otherwise:y=max(x,0)(7.6)7.1\u2022UNITS3Substituting Eq.7.2into Eq.7.3gives us the output of a neural unit:y=s(w\u00b7x+b)=11+exp(\u0000(w\u00b7x+b))(7.4)Fig.7.2shows a \ufb01nal schematic of a basic neural unit. In this example the unittakes 3 input valuesx1,x2, andx3, and computes a weighted sum, multiplying eachvalue by a weight (w1,w2, andw3, respectively), adds them to a bias termb, and thenpasses the resulting sum through a sigmoid function to result in a number between 0and 1.\nx1x2x3\ny\nw1w2w3\u2211b\u03c3+1za\nFigure 7.2A neural unit, taking 3 inputsx1,x2, andx3(and a biasbthat we represent as aweight for an input clamped at +1) and producing an output y. We include some convenientintermediate variables: the output of the summation,z, and the output of the sigmoid,a. Inthis case the output of the unityis the same asa, but in deeper networks we\u2019ll reserveytomean the \ufb01nal output of the entire network, leavingaas the activation of an individual node.Let\u2019s walk through an example just to get an intuition. Let\u2019s suppose we have aunit with the following weight vector and bias:w=[0.2,0.3,0.9]b=0.5What would this unit do with the following input vector:x=[0.5,0.6,0.1]The resulting outputywould be:y=s(w\u00b7x+b)=11+e\u0000(w\u00b7x+b)=11+e\u0000(.5\u21e4.2+.6\u21e4.3+.1\u21e4.9+.5)=11+e\u00000.87=.70In practice, the sigmoid is not commonly used as an activation function. A functionthat is very similar but almost always better is thetanhfunction shown in Fig.7.3a;tanhtanh is a variant of the sigmoid that ranges from -1 to +1:y=ez\u0000e\u0000zez+e\u0000z(7.5)The simplest activation function, and perhaps the most commonly used, is the rec-ti\ufb01ed linear unit, also called theReLU, shown in Fig.7.3b. It\u2019s just the same asxReLUwhenxis positive, and 0 otherwise:y=max(x,0)(7.6)7.1\u2022UNITS3Substituting Eq.7.2into Eq.7.3gives us the output of a neural unit:y=s(w\u00b7x+b)=11+exp(\u0000(w\u00b7x+b))(7.4)Fig.7.2shows a \ufb01nal schematic of a basic neural unit. In this example the unittakes 3 input valuesx1,x2, andx3, and computes a weighted sum, multiplying eachvalue by a weight (w1,w2, andw3, respectively), adds them to a bias termb, and thenpasses the resulting sum through a sigmoid function to result in a number between 0and 1.\nx1x2x3\ny\nw1w2w3\u2211b\u03c3+1za\nFigure 7.2A neural unit, taking 3 inputsx1,x2, andx3(and a biasbthat we represent as aweight for an input clamped at +1) and producing an output y. We include some convenientintermediate variables: the output of the summation,z, and the output of the sigmoid,a. Inthis case the output of the unityis the same asa, but in deeper networks we\u2019ll reserveytomean the \ufb01nal output of the entire network, leavingaas the activation of an individual node.Let\u2019s walk through an example just to get an intuition. Let\u2019s suppose we have aunit with the following weight vector and bias:w=[0.2,0.3,0.9]b=0.5What would this unit do with the following input vector:x=[0.5,0.6,0.1]The resulting outputywould be:y=s(w\u00b7x+b)=11+e\u0000(w\u00b7x+b)=11+e\u0000(.5\u21e4.2+.6\u21e4.3+.1\u21e4.9+.5)=11+e\u00000.87=.70In practice, the sigmoid is not commonly used as an activation function. A functionthat is very similar but almost always better is thetanhfunction shown in Fig.7.3a;tanhtanh is a variant of the sigmoid that ranges from -1 to +1:y=ez\u0000e\u0000zez+e\u0000z(7.5)The simplest activation function, and perhaps the most commonly used, is the rec-ti\ufb01ed linear unit, also called theReLU, shown in Fig.7.3b. It\u2019s just the same asxReLUwhenxis positive, and 0 otherwise:y=max(x,0)(7.6)7.1\u2022UNITS3Substituting Eq.7.2into Eq.7.3gives us the output of a neural unit:y=s(w\u00b7x+b)=11+exp(\u0000(w\u00b7x+b))(7.4)Fig.7.2shows a \ufb01nal schematic of a basic neural unit. In this example the unittakes 3 input valuesx1,x2, andx3, and computes a weighted sum, multiplying eachvalue by a weight (w1,w2, andw3, respectively), adds them to a bias termb, and thenpasses the resulting sum through a sigmoid function to result in a number between 0and 1.\nx1x2x3\ny\nw1w2w3\u2211b\u03c3+1za\nFigure 7.2A neural unit, taking 3 inputsx1,x2, andx3(and a biasbthat we represent as aweight for an input clamped at +1) and producing an output y. We include some convenientintermediate variables: the output of the summation,z, and the output of the sigmoid,a. Inthis case the output of the unityis the same asa, but in deeper networks we\u2019ll reserveytomean the \ufb01nal output of the entire network, leavingaas the activation of an individual node.Let\u2019s walk through an example just to get an intuition. Let\u2019s suppose we have aunit with the following weight vector and bias:w=[0.2,0.3,0.9]b=0.5What would this unit do with the following input vector:x=[0.5,0.6,0.1]The resulting outputywould be:y=s(w\u00b7x+b)=11+e\u0000(w\u00b7x+b)=11+e\u0000(.5\u21e4.2+.6\u21e4.3+.1\u21e4.9+.5)=11+e\u00000.87=.70In practice, the sigmoid is not commonly used as an activation function. A functionthat is very similar but almost always better is thetanhfunction shown in Fig.7.3a;tanhtanh is a variant of the sigmoid that ranges from -1 to +1:y=ez\u0000e\u0000zez+e\u0000z(7.5)The simplest activation function, and perhaps the most commonly used, is the rec-ti\ufb01ed linear unit, also called theReLU, shown in Fig.7.3b. It\u2019s just the same asxReLUwhenxis positive, and 0 otherwise:y=max(x,0)(7.6) Non-Linear Activation Functions besides sigmoid\n12\ntanhReLURectified Linear Unit7.1\u2022UNITS3Substituting Eq.7.2into Eq.7.3gives us the output of a neural unit:y=s(w\u00b7x+b)=11+exp(\u0000(w\u00b7x+b))(7.4)Fig.7.2shows a \ufb01nal schematic of a basic neural unit. In this example the unittakes 3 input valuesx1,x2, andx3, and computes a weighted sum, multiplying eachvalue by a weight (w1,w2, andw3, respectively), adds them to a bias termb, and thenpasses the resulting sum through a sigmoid function to result in a number between 0and 1.\nx1x2x3\ny\nw1w2w3\u2211b\u03c3+1za\nFigure 7.2A neural unit, taking 3 inputsx1,x2, andx3(and a biasbthat we represent as aweight for an input clamped at +1) and producing an output y. We include some convenientintermediate variables: the output of the summation,z, and the output of the sigmoid,a. Inthis case the output of the unityis the same asa, but in deeper networks we\u2019ll reserveytomean the \ufb01nal output of the entire network, leavingaas the activation of an individual node.Let\u2019s walk through an example just to get an intuition. Let\u2019s suppose we have aunit with the following weight vector and bias:w=[0.2,0.3,0.9]b=0.5What would this unit do with the following input vector:x=[0.5,0.6,0.1]The resulting outputywould be:y=s(w\u00b7x+b)=11+e\u0000(w\u00b7x+b)=11+e\u0000(.5\u21e4.2+.6\u21e4.3+.1\u21e4.9+.5)=11+e\u00000.87=.70In practice, the sigmoid is not commonly used as an activation function. A functionthat is very similar but almost always better is thetanhfunction shown in Fig.7.3a;tanhtanh is a variant of the sigmoid that ranges from -1 to +1:y=ez\u0000e\u0000zez+e\u0000z(7.5)The simplest activation function, and perhaps the most commonly used, is the rec-ti\ufb01ed linear unit, also called theReLU, shown in Fig.7.3b. It\u2019s just the same asxReLUwhenxis positive, and 0 otherwise:y=max(x,0)(7.6)\n7.1\u2022UNITS3Substituting Eq.7.2into Eq.7.3gives us the output of a neural unit:y=s(w\u00b7x+b)=11+exp(\u0000(w\u00b7x+b))(7.4)Fig.7.2shows a \ufb01nal schematic of a basic neural unit. In this example the unittakes 3 input valuesx1,x2, andx3, and computes a weighted sum, multiplying eachvalue by a weight (w1,w2, andw3, respectively), adds them to a bias termb, and thenpasses the resulting sum through a sigmoid function to result in a number between 0and 1.\nx1x2x3\ny\nw1w2w3\u2211b\u03c3+1za\nFigure 7.2A neural unit, taking 3 inputsx1,x2, andx3(and a biasbthat we represent as aweight for an input clamped at +1) and producing an output y. We include some convenientintermediate variables: the output of the summation,z, and the output of the sigmoid,a. Inthis case the output of the unityis the same asa, but in deeper networks we\u2019ll reserveytomean the \ufb01nal output of the entire network, leavingaas the activation of an individual node.Let\u2019s walk through an example just to get an intuition. Let\u2019s suppose we have aunit with the following weight vector and bias:w=[0.2,0.3,0.9]b=0.5What would this unit do with the following input vector:x=[0.5,0.6,0.1]The resulting outputywould be:y=s(w\u00b7x+b)=11+e\u0000(w\u00b7x+b)=11+e\u0000(.5\u21e4.2+.6\u21e4.3+.1\u21e4.9+.5)=11+e\u00000.87=.70In practice, the sigmoid is not commonly used as an activation function. A functionthat is very similar but almost always better is thetanhfunction shown in Fig.7.3a;tanhtanh is a variant of the sigmoid that ranges from -1 to +1:y=ez\u0000e\u0000zez+e\u0000z(7.5)The simplest activation function, and perhaps the most commonly used, is the rec-ti\ufb01ed linear unit, also called theReLU, shown in Fig.7.3b. It\u2019s just the same aszReLUwhenzis positive, and 0 otherwise:y=max(z,0)(7.6)Most Common: Simple Neural Networks and Neural Language ModelsUnits in Neural Networks Simple Neural Networks and Neural Language ModelsThe XOR problem The XOR problemCan neural units compute simple functions of input?Minsky and Papert(1969)4CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELS\n(a) (b)Figure 7.3The tanh and ReLU activation functions.These activation functions have different properties that make them useful fordifferent language applications or network architectures. For example, the tanh func-tion has the nice properties of being smoothly differentiable and mapping outliervalues toward the mean. The recti\ufb01er function, on the other hand has nice propertiesthat result from it being very close to linear. In the sigmoid or tanh functions, veryhigh values ofzresult in values ofythat aresaturated, i.e., extremely close to 1,saturatedand have derivatives very close to 0. Zero derivatives cause problems for learning,because as we\u2019ll see in Section7.4, we\u2019ll train networks by propagating an errorsignal backwards, multiplying gradients (partial derivatives) from each layer of thenetwork; gradients that are almost 0 cause the error signal to get smaller and smalleruntil it is too small to be used for training, a problem called thevanishing gradientvanishinggradientproblem. Recti\ufb01ers don\u2019t have this problem, since the derivative of ReLU for highvalues ofzis 1 rather than very close to 0.7.2 The XOR problemEarly in the history of neural networks it was realized that the power of neural net-works, as with the real neurons that inspired them, comes from combining theseunits into larger networks.One of the most clever demonstrations of the need for multi-layer networks wasthe proof byMinsky and Papert (1969)that a single neural unit cannot computesome very simple functions of its input. Consider the task of computing elementarylogical functions of two inputs, like AND, OR, and XOR. As a reminder, here arethe truth tables for those functions:AND OR XORx1 x2y x1 x2y x1 x2y0000 000 000100 110 111001 011 011111 111 10This example was \ufb01rst shown for theperceptron, which is a very simple neuralperceptronunit that has a binary output and doesnothave a non-linear activation function. The PerceptronsA very simple neural unit \u2022Binary output  (0 or 1)\u2022No non-linear activation function7.2\u2022THEXORPROBLEM5outputyof a perceptron is 0 or 1, and is computed as follows (using the same weightw, inputx, and biasbas in Eq.7.2):y=\u21e20,ifw\u00b7x+b\uf8ff01,ifw\u00b7x+b>0(7.7)It\u2019s very easy to build a perceptron that can compute the logical AND and ORfunctions of its binary inputs; Fig.7.4shows the necessary weights.x1x2+1-111x1x2+1011(a) (b)Figure 7.4The weightswand biasbfor perceptrons for computing logical functions. Theinputs are shown asx1andx2and the bias as a special node with value+1 which is multipliedwith the bias weightb. (a) logical AND, showing weightsw1=1 andw2=1 and bias weightb=\u00001. (b) logical OR, showing weightsw1=1 andw2=1 and bias weightb=0. Theseweights/biases are just one from an in\ufb01nite number of possible sets of weights and biases thatwould implement the functions.It turns out, however, that it\u2019s not possible to build a perceptron to computelogical XOR! (It\u2019s worth spending a moment to give it a try!)The intuition behind this important result relies on understanding that a percep-tron is a linear classi\ufb01er. For a two-dimensional inputx1andx2, the perceptionequation,w1x1+w2x2+b=0 is the equation of a line. (We can see this by puttingit in the standard linear format:x2=(\u0000w1/w2)x1+(\u0000b/w2).) This line acts as adecision boundaryin two-dimensional space in which the output 0 is assigned to alldecisionboundaryinputs lying on one side of the line, and the output 1 to all input points lying on theother side of the line. If we had more than 2 inputs, the decision boundary becomesa hyperplane instead of a line, but the idea is the same, separating the space into twocategories.Fig.7.5shows the possible logical inputs (00,01,10, and11) and the line drawnby one possible set of parameters for an AND and an OR classi\ufb01er. Notice that thereis simply no way to draw a line that separates the positive cases of XOR (01 and 10)from the negative cases (00 and 11). We say that XOR is not alinearly separablelinearlyseparablefunction. Of course we could draw a boundary with a curve, or some other function,but not a single line.7.2.1 The solution: neural networksWhile the XOR function cannot be calculated by a single perceptron, it can be cal-culated by a layered network of units. Let\u2019s see an example of how to do this fromGoodfellow et al. (2016)that computes XOR using two layers of ReLU-based units.Fig.7.6shows a \ufb01gure with the input being processed by two layers of neural units.The middle layer (calledh) has two units, and the output layer (calledy) has oneunit. A set of weights and biases are shown for each ReLU that correctly computesthe XOR function.Let\u2019s walk through what happens with the input x = [0 0]. If we multiply eachinput value by the appropriate weight, sum, and then add the biasb, we get the Easy to build AND or OR with perceptrons7.2\u2022THEXORPROBLEM5outputyof a perceptron is 0 or 1, and is computed as follows (using the same weightw, inputx, and biasbas in Eq.7.2):y=\u21e20,ifw\u00b7x+b\uf8ff01,ifw\u00b7x+b>0(7.7)It\u2019s very easy to build a perceptron that can compute the logical AND and ORfunctions of its binary inputs; Fig.7.4shows the necessary weights.x1x2+1-111x1x2+1011(a) (b)Figure 7.4The weightswand biasbfor perceptrons for computing logical functions. Theinputs are shown asx1andx2and the bias as a special node with value+1 which is multipliedwith the bias weightb. (a) logical AND, showing weightsw1=1 andw2=1 and bias weightb=\u00001. (b) logical OR, showing weightsw1=1 andw2=1 and bias weightb=0. Theseweights/biases are just one from an in\ufb01nite number of possible sets of weights and biases thatwould implement the functions.It turns out, however, that it\u2019s not possible to build a perceptron to computelogical XOR! (It\u2019s worth spending a moment to give it a try!)The intuition behind this important result relies on understanding that a percep-tron is a linear classi\ufb01er. For a two-dimensional inputx1andx2, the perceptionequation,w1x1+w2x2+b=0 is the equation of a line. (We can see this by puttingit in the standard linear format:x2=(\u0000w1/w2)x1+(\u0000b/w2).) This line acts as adecision boundaryin two-dimensional space in which the output 0 is assigned to alldecisionboundaryinputs lying on one side of the line, and the output 1 to all input points lying on theother side of the line. If we had more than 2 inputs, the decision boundary becomesa hyperplane instead of a line, but the idea is the same, separating the space into twocategories.Fig.7.5shows the possible logical inputs (00,01,10, and11) and the line drawnby one possible set of parameters for an AND and an OR classi\ufb01er. Notice that thereis simply no way to draw a line that separates the positive cases of XOR (01 and 10)from the negative cases (00 and 11). We say that XOR is not alinearly separablelinearlyseparablefunction. Of course we could draw a boundary with a curve, or some other function,but not a single line.7.2.1 The solution: neural networksWhile the XOR function cannot be calculated by a single perceptron, it can be cal-culated by a layered network of units. Let\u2019s see an example of how to do this fromGoodfellow et al. (2016)that computes XOR using two layers of ReLU-based units.Fig.7.6shows a \ufb01gure with the input being processed by two layers of neural units.The middle layer (calledh) has two units, and the output layer (calledy) has oneunit. A set of weights and biases are shown for each ReLU that correctly computesthe XOR function.Let\u2019s walk through what happens with the input x = [0 0]. If we multiply eachinput value by the appropriate weight, sum, and then add the biasb, we get the7.2\u2022THEXORPROBLEM5outputyof a perceptron is 0 or 1, and is computed as follows (using the same weightw, inputx, and biasbas in Eq.7.2):y=\u21e20,ifw\u00b7x+b\uf8ff01,ifw\u00b7x+b>0(7.7)It\u2019s very easy to build a perceptron that can compute the logical AND and ORfunctions of its binary inputs; Fig.7.4shows the necessary weights.x1x2+1-111x1x2+1011(a) (b)Figure 7.4The weightswand biasbfor perceptrons for computing logical functions. Theinputs are shown asx1andx2and the bias as a special node with value+1 which is multipliedwith the bias weightb. (a) logical AND, showing weightsw1=1 andw2=1 and bias weightb=\u00001. (b) logical OR, showing weightsw1=1 andw2=1 and bias weightb=0. Theseweights/biases are just one from an in\ufb01nite number of possible sets of weights and biases thatwould implement the functions.It turns out, however, that it\u2019s not possible to build a perceptron to computelogical XOR! (It\u2019s worth spending a moment to give it a try!)The intuition behind this important result relies on understanding that a percep-tron is a linear classi\ufb01er. For a two-dimensional inputx1andx2, the perceptionequation,w1x1+w2x2+b=0 is the equation of a line. (We can see this by puttingit in the standard linear format:x2=(\u0000w1/w2)x1+(\u0000b/w2).) This line acts as adecision boundaryin two-dimensional space in which the output 0 is assigned to alldecisionboundaryinputs lying on one side of the line, and the output 1 to all input points lying on theother side of the line. If we had more than 2 inputs, the decision boundary becomesa hyperplane instead of a line, but the idea is the same, separating the space into twocategories.Fig.7.5shows the possible logical inputs (00,01,10, and11) and the line drawnby one possible set of parameters for an AND and an OR classi\ufb01er. Notice that thereis simply no way to draw a line that separates the positive cases of XOR (01 and 10)from the negative cases (00 and 11). We say that XOR is not alinearly separablelinearlyseparablefunction. Of course we could draw a boundary with a curve, or some other function,but not a single line.7.2.1 The solution: neural networksWhile the XOR function cannot be calculated by a single perceptron, it can be cal-culated by a layered network of units. Let\u2019s see an example of how to do this fromGoodfellow et al. (2016)that computes XOR using two layers of ReLU-based units.Fig.7.6shows a \ufb01gure with the input being processed by two layers of neural units.The middle layer (calledh) has two units, and the output layer (calledy) has oneunit. A set of weights and biases are shown for each ReLU that correctly computesthe XOR function.Let\u2019s walk through what happens with the input x = [0 0]. If we multiply eachinput value by the appropriate weight, sum, and then add the biasb, we get theANDOR4CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELS\n(a) (b)Figure 7.3The tanh and ReLU activation functions.These activation functions have different properties that make them useful fordifferent language applications or network architectures. For example, the tanh func-tion has the nice properties of being smoothly differentiable and mapping outliervalues toward the mean. The recti\ufb01er function, on the other hand has nice propertiesthat result from it being very close to linear. In the sigmoid or tanh functions, veryhigh values ofzresult in values ofythat aresaturated, i.e., extremely close to 1,saturatedand have derivatives very close to 0. Zero derivatives cause problems for learning,because as we\u2019ll see in Section7.4, we\u2019ll train networks by propagating an errorsignal backwards, multiplying gradients (partial derivatives) from each layer of thenetwork; gradients that are almost 0 cause the error signal to get smaller and smalleruntil it is too small to be used for training, a problem called thevanishing gradientvanishinggradientproblem. Recti\ufb01ers don\u2019t have this problem, since the derivative of ReLU for highvalues ofzis 1 rather than very close to 0.7.2 The XOR problemEarly in the history of neural networks it was realized that the power of neural net-works, as with the real neurons that inspired them, comes from combining theseunits into larger networks.One of the most clever demonstrations of the need for multi-layer networks wasthe proof byMinsky and Papert (1969)that a single neural unit cannot computesome very simple functions of its input. Consider the task of computing elementarylogical functions of two inputs, like AND, OR, and XOR. As a reminder, here arethe truth tables for those functions:AND OR XORx1 x2y x1 x2y x1 x2y0000 000 000100 110 111001 011 011111 111 10This example was \ufb01rst shown for theperceptron, which is a very simple neuralperceptronunit that has a binary output and doesnothave a non-linear activation function. The4CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELS\n(a) (b)Figure 7.3The tanh and ReLU activation functions.These activation functions have different properties that make them useful fordifferent language applications or network architectures. For example, the tanh func-tion has the nice properties of being smoothly differentiable and mapping outliervalues toward the mean. The recti\ufb01er function, on the other hand has nice propertiesthat result from it being very close to linear. In the sigmoid or tanh functions, veryhigh values ofzresult in values ofythat aresaturated, i.e., extremely close to 1,saturatedand have derivatives very close to 0. Zero derivatives cause problems for learning,because as we\u2019ll see in Section7.4, we\u2019ll train networks by propagating an errorsignal backwards, multiplying gradients (partial derivatives) from each layer of thenetwork; gradients that are almost 0 cause the error signal to get smaller and smalleruntil it is too small to be used for training, a problem called thevanishing gradientvanishinggradientproblem. Recti\ufb01ers don\u2019t have this problem, since the derivative of ReLU for highvalues ofzis 1 rather than very close to 0.7.2 The XOR problemEarly in the history of neural networks it was realized that the power of neural net-works, as with the real neurons that inspired them, comes from combining theseunits into larger networks.One of the most clever demonstrations of the need for multi-layer networks wasthe proof byMinsky and Papert (1969)that a single neural unit cannot computesome very simple functions of its input. Consider the task of computing elementarylogical functions of two inputs, like AND, OR, and XOR. As a reminder, here arethe truth tables for those functions:AND OR XORx1 x2y x1 x2y x1 x2y0000 000 000100 110 111001 011 011111 111 10This example was \ufb01rst shown for theperceptron, which is a very simple neuralperceptronunit that has a binary output and doesnothave a non-linear activation function. The7.2\u2022THEXORPROBLEM5outputyof a perceptron is 0 or 1, and is computed as follows (using the same weightw, inputx, and biasbas in Eq.7.2):y=\u21e20,ifw\u00b7x+b\uf8ff01,ifw\u00b7x+b>0(7.7)It\u2019s very easy to build a perceptron that can compute the logical AND and ORfunctions of its binary inputs; Fig.7.4shows the necessary weights.x1x2+1-111x1x2+1011(a) (b)Figure 7.4The weightswand biasbfor perceptrons for computing logical functions. Theinputs are shown asx1andx2and the bias as a special node with value+1 which is multipliedwith the bias weightb. (a) logical AND, showing weightsw1=1 andw2=1 and bias weightb=\u00001. (b) logical OR, showing weightsw1=1 andw2=1 and bias weightb=0. Theseweights/biases are just one from an in\ufb01nite number of possible sets of weights and biases thatwould implement the functions.It turns out, however, that it\u2019s not possible to build a perceptron to computelogical XOR! (It\u2019s worth spending a moment to give it a try!)The intuition behind this important result relies on understanding that a percep-tron is a linear classi\ufb01er. For a two-dimensional inputx1andx2, the perceptionequation,w1x1+w2x2+b=0 is the equation of a line. (We can see this by puttingit in the standard linear format:x2=(\u0000w1/w2)x1+(\u0000b/w2).) This line acts as adecision boundaryin two-dimensional space in which the output 0 is assigned to alldecisionboundaryinputs lying on one side of the line, and the output 1 to all input points lying on theother side of the line. If we had more than 2 inputs, the decision boundary becomesa hyperplane instead of a line, but the idea is the same, separating the space into twocategories.Fig.7.5shows the possible logical inputs (00,01,10, and11) and the line drawnby one possible set of parameters for an AND and an OR classi\ufb01er. Notice that thereis simply no way to draw a line that separates the positive cases of XOR (01 and 10)from the negative cases (00 and 11). We say that XOR is not alinearly separablelinearlyseparablefunction. Of course we could draw a boundary with a curve, or some other function,but not a single line.7.2.1 The solution: neural networksWhile the XOR function cannot be calculated by a single perceptron, it can be cal-culated by a layered network of units. Let\u2019s see an example of how to do this fromGoodfellow et al. (2016)that computes XOR using two layers of ReLU-based units.Fig.7.6shows a \ufb01gure with the input being processed by two layers of neural units.The middle layer (calledh) has two units, and the output layer (calledy) has oneunit. A set of weights and biases are shown for each ReLU that correctly computesthe XOR function.Let\u2019s walk through what happens with the input x = [0 0]. If we multiply eachinput value by the appropriate weight, sum, and then add the biasb, we get the Easy to build AND or OR with perceptrons7.2\u2022THEXORPROBLEM5outputyof a perceptron is 0 or 1, and is computed as follows (using the same weightw, inputx, and biasbas in Eq.7.2):y=\u21e20,ifw\u00b7x+b\uf8ff01,ifw\u00b7x+b>0(7.7)It\u2019s very easy to build a perceptron that can compute the logical AND and ORfunctions of its binary inputs; Fig.7.4shows the necessary weights.x1x2+1-111x1x2+1011(a) (b)Figure 7.4The weightswand biasbfor perceptrons for computing logical functions. Theinputs are shown asx1andx2and the bias as a special node with value+1 which is multipliedwith the bias weightb. (a) logical AND, showing weightsw1=1 andw2=1 and bias weightb=\u00001. (b) logical OR, showing weightsw1=1 andw2=1 and bias weightb=0. Theseweights/biases are just one from an in\ufb01nite number of possible sets of weights and biases thatwould implement the functions.It turns out, however, that it\u2019s not possible to build a perceptron to computelogical XOR! (It\u2019s worth spending a moment to give it a try!)The intuition behind this important result relies on understanding that a percep-tron is a linear classi\ufb01er. For a two-dimensional inputx1andx2, the perceptionequation,w1x1+w2x2+b=0 is the equation of a line. (We can see this by puttingit in the standard linear format:x2=(\u0000w1/w2)x1+(\u0000b/w2).) This line acts as adecision boundaryin two-dimensional space in which the output 0 is assigned to alldecisionboundaryinputs lying on one side of the line, and the output 1 to all input points lying on theother side of the line. If we had more than 2 inputs, the decision boundary becomesa hyperplane instead of a line, but the idea is the same, separating the space into twocategories.Fig.7.5shows the possible logical inputs (00,01,10, and11) and the line drawnby one possible set of parameters for an AND and an OR classi\ufb01er. Notice that thereis simply no way to draw a line that separates the positive cases of XOR (01 and 10)from the negative cases (00 and 11). We say that XOR is not alinearly separablelinearlyseparablefunction. Of course we could draw a boundary with a curve, or some other function,but not a single line.7.2.1 The solution: neural networksWhile the XOR function cannot be calculated by a single perceptron, it can be cal-culated by a layered network of units. Let\u2019s see an example of how to do this fromGoodfellow et al. (2016)that computes XOR using two layers of ReLU-based units.Fig.7.6shows a \ufb01gure with the input being processed by two layers of neural units.The middle layer (calledh) has two units, and the output layer (calledy) has oneunit. A set of weights and biases are shown for each ReLU that correctly computesthe XOR function.Let\u2019s walk through what happens with the input x = [0 0]. If we multiply eachinput value by the appropriate weight, sum, and then add the biasb, we get the7.2\u2022THEXORPROBLEM5outputyof a perceptron is 0 or 1, and is computed as follows (using the same weightw, inputx, and biasbas in Eq.7.2):y=\u21e20,ifw\u00b7x+b\uf8ff01,ifw\u00b7x+b>0(7.7)It\u2019s very easy to build a perceptron that can compute the logical AND and ORfunctions of its binary inputs; Fig.7.4shows the necessary weights.x1x2+1-111x1x2+1011(a) (b)Figure 7.4The weightswand biasbfor perceptrons for computing logical functions. Theinputs are shown asx1andx2and the bias as a special node with value+1 which is multipliedwith the bias weightb. (a) logical AND, showing weightsw1=1 andw2=1 and bias weightb=\u00001. (b) logical OR, showing weightsw1=1 andw2=1 and bias weightb=0. Theseweights/biases are just one from an in\ufb01nite number of possible sets of weights and biases thatwould implement the functions.It turns out, however, that it\u2019s not possible to build a perceptron to computelogical XOR! (It\u2019s worth spending a moment to give it a try!)The intuition behind this important result relies on understanding that a percep-tron is a linear classi\ufb01er. For a two-dimensional inputx1andx2, the perceptionequation,w1x1+w2x2+b=0 is the equation of a line. (We can see this by puttingit in the standard linear format:x2=(\u0000w1/w2)x1+(\u0000b/w2).) This line acts as adecision boundaryin two-dimensional space in which the output 0 is assigned to alldecisionboundaryinputs lying on one side of the line, and the output 1 to all input points lying on theother side of the line. If we had more than 2 inputs, the decision boundary becomesa hyperplane instead of a line, but the idea is the same, separating the space into twocategories.Fig.7.5shows the possible logical inputs (00,01,10, and11) and the line drawnby one possible set of parameters for an AND and an OR classi\ufb01er. Notice that thereis simply no way to draw a line that separates the positive cases of XOR (01 and 10)from the negative cases (00 and 11). We say that XOR is not alinearly separablelinearlyseparablefunction. Of course we could draw a boundary with a curve, or some other function,but not a single line.7.2.1 The solution: neural networksWhile the XOR function cannot be calculated by a single perceptron, it can be cal-culated by a layered network of units. Let\u2019s see an example of how to do this fromGoodfellow et al. (2016)that computes XOR using two layers of ReLU-based units.Fig.7.6shows a \ufb01gure with the input being processed by two layers of neural units.The middle layer (calledh) has two units, and the output layer (calledy) has oneunit. A set of weights and biases are shown for each ReLU that correctly computesthe XOR function.Let\u2019s walk through what happens with the input x = [0 0]. If we multiply eachinput value by the appropriate weight, sum, and then add the biasb, we get theANDOR4CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELS\n(a) (b)Figure 7.3The tanh and ReLU activation functions.These activation functions have different properties that make them useful fordifferent language applications or network architectures. For example, the tanh func-tion has the nice properties of being smoothly differentiable and mapping outliervalues toward the mean. The recti\ufb01er function, on the other hand has nice propertiesthat result from it being very close to linear. In the sigmoid or tanh functions, veryhigh values ofzresult in values ofythat aresaturated, i.e., extremely close to 1,saturatedand have derivatives very close to 0. Zero derivatives cause problems for learning,because as we\u2019ll see in Section7.4, we\u2019ll train networks by propagating an errorsignal backwards, multiplying gradients (partial derivatives) from each layer of thenetwork; gradients that are almost 0 cause the error signal to get smaller and smalleruntil it is too small to be used for training, a problem called thevanishing gradientvanishinggradientproblem. Recti\ufb01ers don\u2019t have this problem, since the derivative of ReLU for highvalues ofzis 1 rather than very close to 0.7.2 The XOR problemEarly in the history of neural networks it was realized that the power of neural net-works, as with the real neurons that inspired them, comes from combining theseunits into larger networks.One of the most clever demonstrations of the need for multi-layer networks wasthe proof byMinsky and Papert (1969)that a single neural unit cannot computesome very simple functions of its input. Consider the task of computing elementarylogical functions of two inputs, like AND, OR, and XOR. As a reminder, here arethe truth tables for those functions:AND OR XORx1 x2y x1 x2y x1 x2y0000 000 000100 110 111001 011 011111 111 10This example was \ufb01rst shown for theperceptron, which is a very simple neuralperceptronunit that has a binary output and doesnothave a non-linear activation function. The4CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELS\n(a) (b)Figure 7.3The tanh and ReLU activation functions.These activation functions have different properties that make them useful fordifferent language applications or network architectures. For example, the tanh func-tion has the nice properties of being smoothly differentiable and mapping outliervalues toward the mean. The recti\ufb01er function, on the other hand has nice propertiesthat result from it being very close to linear. In the sigmoid or tanh functions, veryhigh values ofzresult in values ofythat aresaturated, i.e., extremely close to 1,saturatedand have derivatives very close to 0. Zero derivatives cause problems for learning,because as we\u2019ll see in Section7.4, we\u2019ll train networks by propagating an errorsignal backwards, multiplying gradients (partial derivatives) from each layer of thenetwork; gradients that are almost 0 cause the error signal to get smaller and smalleruntil it is too small to be used for training, a problem called thevanishing gradientvanishinggradientproblem. Recti\ufb01ers don\u2019t have this problem, since the derivative of ReLU for highvalues ofzis 1 rather than very close to 0.7.2 The XOR problemEarly in the history of neural networks it was realized that the power of neural net-works, as with the real neurons that inspired them, comes from combining theseunits into larger networks.One of the most clever demonstrations of the need for multi-layer networks wasthe proof byMinsky and Papert (1969)that a single neural unit cannot computesome very simple functions of its input. Consider the task of computing elementarylogical functions of two inputs, like AND, OR, and XOR. As a reminder, here arethe truth tables for those functions:AND OR XORx1 x2y x1 x2y x1 x2y0000 000 000100 110 111001 011 011111 111 10This example was \ufb01rst shown for theperceptron, which is a very simple neuralperceptronunit that has a binary output and doesnothave a non-linear activation function. The7.2\u2022THEXORPROBLEM5outputyof a perceptron is 0 or 1, and is computed as follows (using the same weightw, inputx, and biasbas in Eq.7.2):y=\u21e20,ifw\u00b7x+b\uf8ff01,ifw\u00b7x+b>0(7.7)It\u2019s very easy to build a perceptron that can compute the logical AND and ORfunctions of its binary inputs; Fig.7.4shows the necessary weights.x1x2+1-111x1x2+1011(a) (b)Figure 7.4The weightswand biasbfor perceptrons for computing logical functions. Theinputs are shown asx1andx2and the bias as a special node with value+1 which is multipliedwith the bias weightb. (a) logical AND, showing weightsw1=1 andw2=1 and bias weightb=\u00001. (b) logical OR, showing weightsw1=1 andw2=1 and bias weightb=0. Theseweights/biases are just one from an in\ufb01nite number of possible sets of weights and biases thatwould implement the functions.It turns out, however, that it\u2019s not possible to build a perceptron to computelogical XOR! (It\u2019s worth spending a moment to give it a try!)The intuition behind this important result relies on understanding that a percep-tron is a linear classi\ufb01er. For a two-dimensional inputx1andx2, the perceptionequation,w1x1+w2x2+b=0 is the equation of a line. (We can see this by puttingit in the standard linear format:x2=(\u0000w1/w2)x1+(\u0000b/w2).) This line acts as adecision boundaryin two-dimensional space in which the output 0 is assigned to alldecisionboundaryinputs lying on one side of the line, and the output 1 to all input points lying on theother side of the line. If we had more than 2 inputs, the decision boundary becomesa hyperplane instead of a line, but the idea is the same, separating the space into twocategories.Fig.7.5shows the possible logical inputs (00,01,10, and11) and the line drawnby one possible set of parameters for an AND and an OR classi\ufb01er. Notice that thereis simply no way to draw a line that separates the positive cases of XOR (01 and 10)from the negative cases (00 and 11). We say that XOR is not alinearly separablelinearlyseparablefunction. Of course we could draw a boundary with a curve, or some other function,but not a single line.7.2.1 The solution: neural networksWhile the XOR function cannot be calculated by a single perceptron, it can be cal-culated by a layered network of units. Let\u2019s see an example of how to do this fromGoodfellow et al. (2016)that computes XOR using two layers of ReLU-based units.Fig.7.6shows a \ufb01gure with the input being processed by two layers of neural units.The middle layer (calledh) has two units, and the output layer (calledy) has oneunit. A set of weights and biases are shown for each ReLU that correctly computesthe XOR function.Let\u2019s walk through what happens with the input x = [0 0]. If we multiply eachinput value by the appropriate weight, sum, and then add the biasb, we get the Easy to build AND or OR with perceptrons7.2\u2022THEXORPROBLEM5outputyof a perceptron is 0 or 1, and is computed as follows (using the same weightw, inputx, and biasbas in Eq.7.2):y=\u21e20,ifw\u00b7x+b\uf8ff01,ifw\u00b7x+b>0(7.7)It\u2019s very easy to build a perceptron that can compute the logical AND and ORfunctions of its binary inputs; Fig.7.4shows the necessary weights.x1x2+1-111x1x2+1011(a) (b)Figure 7.4The weightswand biasbfor perceptrons for computing logical functions. Theinputs are shown asx1andx2and the bias as a special node with value+1 which is multipliedwith the bias weightb. (a) logical AND, showing weightsw1=1 andw2=1 and bias weightb=\u00001. (b) logical OR, showing weightsw1=1 andw2=1 and bias weightb=0. Theseweights/biases are just one from an in\ufb01nite number of possible sets of weights and biases thatwould implement the functions.It turns out, however, that it\u2019s not possible to build a perceptron to computelogical XOR! (It\u2019s worth spending a moment to give it a try!)The intuition behind this important result relies on understanding that a percep-tron is a linear classi\ufb01er. For a two-dimensional inputx1andx2, the perceptionequation,w1x1+w2x2+b=0 is the equation of a line. (We can see this by puttingit in the standard linear format:x2=(\u0000w1/w2)x1+(\u0000b/w2).) This line acts as adecision boundaryin two-dimensional space in which the output 0 is assigned to alldecisionboundaryinputs lying on one side of the line, and the output 1 to all input points lying on theother side of the line. If we had more than 2 inputs, the decision boundary becomesa hyperplane instead of a line, but the idea is the same, separating the space into twocategories.Fig.7.5shows the possible logical inputs (00,01,10, and11) and the line drawnby one possible set of parameters for an AND and an OR classi\ufb01er. Notice that thereis simply no way to draw a line that separates the positive cases of XOR (01 and 10)from the negative cases (00 and 11). We say that XOR is not alinearly separablelinearlyseparablefunction. Of course we could draw a boundary with a curve, or some other function,but not a single line.7.2.1 The solution: neural networksWhile the XOR function cannot be calculated by a single perceptron, it can be cal-culated by a layered network of units. Let\u2019s see an example of how to do this fromGoodfellow et al. (2016)that computes XOR using two layers of ReLU-based units.Fig.7.6shows a \ufb01gure with the input being processed by two layers of neural units.The middle layer (calledh) has two units, and the output layer (calledy) has oneunit. A set of weights and biases are shown for each ReLU that correctly computesthe XOR function.Let\u2019s walk through what happens with the input x = [0 0]. If we multiply eachinput value by the appropriate weight, sum, and then add the biasb, we get the7.2\u2022THEXORPROBLEM5outputyof a perceptron is 0 or 1, and is computed as follows (using the same weightw, inputx, and biasbas in Eq.7.2):y=\u21e20,ifw\u00b7x+b\uf8ff01,ifw\u00b7x+b>0(7.7)It\u2019s very easy to build a perceptron that can compute the logical AND and ORfunctions of its binary inputs; Fig.7.4shows the necessary weights.x1x2+1-111x1x2+1011(a) (b)Figure 7.4The weightswand biasbfor perceptrons for computing logical functions. Theinputs are shown asx1andx2and the bias as a special node with value+1 which is multipliedwith the bias weightb. (a) logical AND, showing weightsw1=1 andw2=1 and bias weightb=\u00001. (b) logical OR, showing weightsw1=1 andw2=1 and bias weightb=0. Theseweights/biases are just one from an in\ufb01nite number of possible sets of weights and biases thatwould implement the functions.It turns out, however, that it\u2019s not possible to build a perceptron to computelogical XOR! (It\u2019s worth spending a moment to give it a try!)The intuition behind this important result relies on understanding that a percep-tron is a linear classi\ufb01er. For a two-dimensional inputx1andx2, the perceptionequation,w1x1+w2x2+b=0 is the equation of a line. (We can see this by puttingit in the standard linear format:x2=(\u0000w1/w2)x1+(\u0000b/w2).) This line acts as adecision boundaryin two-dimensional space in which the output 0 is assigned to alldecisionboundaryinputs lying on one side of the line, and the output 1 to all input points lying on theother side of the line. If we had more than 2 inputs, the decision boundary becomesa hyperplane instead of a line, but the idea is the same, separating the space into twocategories.Fig.7.5shows the possible logical inputs (00,01,10, and11) and the line drawnby one possible set of parameters for an AND and an OR classi\ufb01er. Notice that thereis simply no way to draw a line that separates the positive cases of XOR (01 and 10)from the negative cases (00 and 11). We say that XOR is not alinearly separablelinearlyseparablefunction. Of course we could draw a boundary with a curve, or some other function,but not a single line.7.2.1 The solution: neural networksWhile the XOR function cannot be calculated by a single perceptron, it can be cal-culated by a layered network of units. Let\u2019s see an example of how to do this fromGoodfellow et al. (2016)that computes XOR using two layers of ReLU-based units.Fig.7.6shows a \ufb01gure with the input being processed by two layers of neural units.The middle layer (calledh) has two units, and the output layer (calledy) has oneunit. A set of weights and biases are shown for each ReLU that correctly computesthe XOR function.Let\u2019s walk through what happens with the input x = [0 0]. If we multiply eachinput value by the appropriate weight, sum, and then add the biasb, we get theANDOR4CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELS\n(a) (b)Figure 7.3The tanh and ReLU activation functions.These activation functions have different properties that make them useful fordifferent language applications or network architectures. For example, the tanh func-tion has the nice properties of being smoothly differentiable and mapping outliervalues toward the mean. The recti\ufb01er function, on the other hand has nice propertiesthat result from it being very close to linear. In the sigmoid or tanh functions, veryhigh values ofzresult in values ofythat aresaturated, i.e., extremely close to 1,saturatedand have derivatives very close to 0. Zero derivatives cause problems for learning,because as we\u2019ll see in Section7.4, we\u2019ll train networks by propagating an errorsignal backwards, multiplying gradients (partial derivatives) from each layer of thenetwork; gradients that are almost 0 cause the error signal to get smaller and smalleruntil it is too small to be used for training, a problem called thevanishing gradientvanishinggradientproblem. Recti\ufb01ers don\u2019t have this problem, since the derivative of ReLU for highvalues ofzis 1 rather than very close to 0.7.2 The XOR problemEarly in the history of neural networks it was realized that the power of neural net-works, as with the real neurons that inspired them, comes from combining theseunits into larger networks.One of the most clever demonstrations of the need for multi-layer networks wasthe proof byMinsky and Papert (1969)that a single neural unit cannot computesome very simple functions of its input. Consider the task of computing elementarylogical functions of two inputs, like AND, OR, and XOR. As a reminder, here arethe truth tables for those functions:AND OR XORx1 x2y x1 x2y x1 x2y0000 000 000100 110 111001 011 011111 111 10This example was \ufb01rst shown for theperceptron, which is a very simple neuralperceptronunit that has a binary output and doesnothave a non-linear activation function. The4CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELS\n(a) (b)Figure 7.3The tanh and ReLU activation functions.These activation functions have different properties that make them useful fordifferent language applications or network architectures. For example, the tanh func-tion has the nice properties of being smoothly differentiable and mapping outliervalues toward the mean. The recti\ufb01er function, on the other hand has nice propertiesthat result from it being very close to linear. In the sigmoid or tanh functions, veryhigh values ofzresult in values ofythat aresaturated, i.e., extremely close to 1,saturatedand have derivatives very close to 0. Zero derivatives cause problems for learning,because as we\u2019ll see in Section7.4, we\u2019ll train networks by propagating an errorsignal backwards, multiplying gradients (partial derivatives) from each layer of thenetwork; gradients that are almost 0 cause the error signal to get smaller and smalleruntil it is too small to be used for training, a problem called thevanishing gradientvanishinggradientproblem. Recti\ufb01ers don\u2019t have this problem, since the derivative of ReLU for highvalues ofzis 1 rather than very close to 0.7.2 The XOR problemEarly in the history of neural networks it was realized that the power of neural net-works, as with the real neurons that inspired them, comes from combining theseunits into larger networks.One of the most clever demonstrations of the need for multi-layer networks wasthe proof byMinsky and Papert (1969)that a single neural unit cannot computesome very simple functions of its input. Consider the task of computing elementarylogical functions of two inputs, like AND, OR, and XOR. As a reminder, here arethe truth tables for those functions:AND OR XORx1 x2y x1 x2y x1 x2y0000 000 000100 110 111001 011 011111 111 10This example was \ufb01rst shown for theperceptron, which is a very simple neuralperceptronunit that has a binary output and doesnothave a non-linear activation function. The7.2\u2022THEXORPROBLEM5outputyof a perceptron is 0 or 1, and is computed as follows (using the same weightw, inputx, and biasbas in Eq.7.2):y=\u21e20,ifw\u00b7x+b\uf8ff01,ifw\u00b7x+b>0(7.7)It\u2019s very easy to build a perceptron that can compute the logical AND and ORfunctions of its binary inputs; Fig.7.4shows the necessary weights.x1x2+1-111x1x2+1011(a) (b)Figure 7.4The weightswand biasbfor perceptrons for computing logical functions. Theinputs are shown asx1andx2and the bias as a special node with value+1 which is multipliedwith the bias weightb. (a) logical AND, showing weightsw1=1 andw2=1 and bias weightb=\u00001. (b) logical OR, showing weightsw1=1 andw2=1 and bias weightb=0. Theseweights/biases are just one from an in\ufb01nite number of possible sets of weights and biases thatwould implement the functions.It turns out, however, that it\u2019s not possible to build a perceptron to computelogical XOR! (It\u2019s worth spending a moment to give it a try!)The intuition behind this important result relies on understanding that a percep-tron is a linear classi\ufb01er. For a two-dimensional inputx1andx2, the perceptionequation,w1x1+w2x2+b=0 is the equation of a line. (We can see this by puttingit in the standard linear format:x2=(\u0000w1/w2)x1+(\u0000b/w2).) This line acts as adecision boundaryin two-dimensional space in which the output 0 is assigned to alldecisionboundaryinputs lying on one side of the line, and the output 1 to all input points lying on theother side of the line. If we had more than 2 inputs, the decision boundary becomesa hyperplane instead of a line, but the idea is the same, separating the space into twocategories.Fig.7.5shows the possible logical inputs (00,01,10, and11) and the line drawnby one possible set of parameters for an AND and an OR classi\ufb01er. Notice that thereis simply no way to draw a line that separates the positive cases of XOR (01 and 10)from the negative cases (00 and 11). We say that XOR is not alinearly separablelinearlyseparablefunction. Of course we could draw a boundary with a curve, or some other function,but not a single line.7.2.1 The solution: neural networksWhile the XOR function cannot be calculated by a single perceptron, it can be cal-culated by a layered network of units. Let\u2019s see an example of how to do this fromGoodfellow et al. (2016)that computes XOR using two layers of ReLU-based units.Fig.7.6shows a \ufb01gure with the input being processed by two layers of neural units.The middle layer (calledh) has two units, and the output layer (calledy) has oneunit. A set of weights and biases are shown for each ReLU that correctly computesthe XOR function.Let\u2019s walk through what happens with the input x = [0 0]. If we multiply eachinput value by the appropriate weight, sum, and then add the biasb, we get the Not possible to capture XOR with perceptronsPause the lecture and try for yourself! Why? Perceptronsare linear classifiersPerceptron equation givenx1and x2, is the equation of a linew1x1+ w2x2+ b = 0(in standard linear format:     x2= (\u2212w1/w2)x1+ (\u2212b/w2)    )This line acts as a decision boundary \u20220 if input is on one side of the line\u20221 if on the other side of the line  Decision boundaries\n0011x1x20011x1x20011x1x2\na)  x1 AND x2b)  x1 OR x2c)  x1 XOR x2?XOR is not a linearly separable function! Solution to the XOR problemXOR can'tbe calculated by a single perceptronXOR canbe calculated by a layered network of units. x1x2h1h2y1+11-1111-201+104CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELS\n(a) (b)Figure 7.3The tanh and ReLU activation functions.These activation functions have different properties that make them useful fordifferent language applications or network architectures. For example, the tanh func-tion has the nice properties of being smoothly differentiable and mapping outliervalues toward the mean. The recti\ufb01er function, on the other hand has nice propertiesthat result from it being very close to linear. In the sigmoid or tanh functions, veryhigh values ofzresult in values ofythat aresaturated, i.e., extremely close to 1,saturatedand have derivatives very close to 0. Zero derivatives cause problems for learning,because as we\u2019ll see in Section7.4, we\u2019ll train networks by propagating an errorsignal backwards, multiplying gradients (partial derivatives) from each layer of thenetwork; gradients that are almost 0 cause the error signal to get smaller and smalleruntil it is too small to be used for training, a problem called thevanishing gradientvanishinggradientproblem. Recti\ufb01ers don\u2019t have this problem, since the derivative of ReLU for highvalues ofzis 1 rather than very close to 0.7.2 The XOR problemEarly in the history of neural networks it was realized that the power of neural net-works, as with the real neurons that inspired them, comes from combining theseunits into larger networks.One of the most clever demonstrations of the need for multi-layer networks wasthe proof byMinsky and Papert (1969)that a single neural unit cannot computesome very simple functions of its input. Consider the task of computing elementarylogical functions of two inputs, like AND, OR, and XOR. As a reminder, here arethe truth tables for those functions:AND OR XORx1 x2y x1 x2y x1 x2y0000 000 000100 110 111001 011 011111 111 10This example was \ufb01rst shown for theperceptron, which is a very simple neuralperceptronunit that has a binary output and doesnothave a non-linear activation function. TheReLUReLU Solution to the XOR problemXOR can'tbe calculated by a single perceptronXOR canbe calculated by a layered network of units. x1x2h1h2y1+11-1111-201+104CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELS\n(a) (b)Figure 7.3The tanh and ReLU activation functions.These activation functions have different properties that make them useful fordifferent language applications or network architectures. For example, the tanh func-tion has the nice properties of being smoothly differentiable and mapping outliervalues toward the mean. The recti\ufb01er function, on the other hand has nice propertiesthat result from it being very close to linear. In the sigmoid or tanh functions, veryhigh values ofzresult in values ofythat aresaturated, i.e., extremely close to 1,saturatedand have derivatives very close to 0. Zero derivatives cause problems for learning,because as we\u2019ll see in Section7.4, we\u2019ll train networks by propagating an errorsignal backwards, multiplying gradients (partial derivatives) from each layer of thenetwork; gradients that are almost 0 cause the error signal to get smaller and smalleruntil it is too small to be used for training, a problem called thevanishing gradientvanishinggradientproblem. Recti\ufb01ers don\u2019t have this problem, since the derivative of ReLU for highvalues ofzis 1 rather than very close to 0.7.2 The XOR problemEarly in the history of neural networks it was realized that the power of neural net-works, as with the real neurons that inspired them, comes from combining theseunits into larger networks.One of the most clever demonstrations of the need for multi-layer networks wasthe proof byMinsky and Papert (1969)that a single neural unit cannot computesome very simple functions of its input. Consider the task of computing elementarylogical functions of two inputs, like AND, OR, and XOR. As a reminder, here arethe truth tables for those functions:AND OR XORx1 x2y x1 x2y x1 x2y0000 000 000100 110 111001 011 011111 111 10This example was \ufb01rst shown for theperceptron, which is a very simple neuralperceptronunit that has a binary output and doesnothave a non-linear activation function. The The hidden representation h\n0011x1x2\na) The original x space0011h1h2\n2b) The new (linearly separable) h spacex1x2h1h2y1+11-1111-201+10\n(With learning:  hidden layers will learn to form useful representations) Simple Neural Networks and Neural Language ModelsThe XOR problem Simple Neural Networks and Neural Language ModelsFeedforward Neural Networks Feedforward Neural NetworksCan also be called multi-layer perceptrons(or MLPs)  for historical reasons8CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELS\nx1x2y1xn0\u2026\u2026+1b\u2026UWy2yn2h1h2h3hn1Figure 7.8A simple 2-layer feedforward network, with one hidden layer, one output layer,and one input layer (the input layer is usually not counted when enumerating layers).Recall that a single hidden unit has parametersw(the weight vector) andb(thebias scalar). We represent the parameters for the entire hidden layer by combiningthe weight vectorwiand biasbifor each unitiinto a single weight matrixWanda single bias vectorbfor the whole layer (see Fig.7.8). Each elementWjiof theweight matrixWrepresents the weight of the connection from theith input unitxitothejth hidden unithj.The advantage of using a single matrixWfor the weights of the entire layer isthat now the hidden layer computation for a feedforward network can be done veryef\ufb01ciently with simple matrix operations. In fact, the computation only has threesteps: multiplying the weight matrix by the input vectorx, adding the bias vectorb,and applying the activation functiong(such as the sigmoid, tanh, or ReLU activationfunction de\ufb01ned above).The output of the hidden layer, the vectorh, is thus the following, using thesigmoid functions:h=s(Wx+b)(7.8)Notice that we\u2019re applying thesfunction here to a vector, while in Eq.7.3it wasapplied to a scalar. We\u2019re thus allowings(\u00b7), and indeed any activation functiong(\u00b7), to apply to a vector element-wise, sog[z1,z2,z3]=[g(z1),g(z2),g(z3)].Let\u2019s introduce some constants to represent the dimensionalities of these vectorsand matrices. We\u2019ll refer to the input layer as layer 0 of the network, and haven0represent the number of inputs, soxis a vector of real numbers of dimensionn0,or more formallyx2Rn0, a column vector of dimensionality[n0,1]. Let\u2019s call thehidden layer layer 1 and the output layer layer 2. The hidden layer has dimensional-ityn1, soh2Rn1and alsob2Rn1(since each hidden unit can take a different biasvalue). And the weight matrixWhas dimensionalityW2Rn1\u21e5n0, i.e.[n1,n0].Take a moment to convince yourself that the matrix multiplication in Eq.7.8willcompute the value of eachhjass\u0000Pn0i=1Wjixi+bj\u0000.As we saw in Section7.2, the resulting valueh(forhiddenbut also forhypoth-esis) forms arepresentationof the input. The role of the output layer is to takethis new representationhand compute a \ufb01nal output. This output could be a real-valued number, but in many cases the goal of the network is to make some sort ofclassi\ufb01cation decision, and so we will focus on the case of classi\ufb01cation.If we are doing a binary task like sentiment classi\ufb01cation, we might have a singleoutput node, and its valueyis the probability of positive versus negative sentiment. Binary Logistic Regression as a 1-layer Network\n29wxnx1\ud835\udc66=\ud835\udf0e(\ud835\udc64&\ud835\udc65+\ud835\udc4f)+1w1wnb(y is a scalar)\u03c3Output layer(\u03c3node)Input layervector x(we don't count the input layer in counting layers!)\n(vector)(scalar) Multinomial Logistic Regression as a 1-layer Network\n30Wxnx1Fully connected single layer networkW is a matrix\ud835\udc66=softmax(\ud835\udc4a\ud835\udc65+\ud835\udc4f)+1y is a vectory1ynb is a vectorbsssOutput layer(softmaxnodes)Input layerscalars Reminder: softmax: a generalization of sigmoidFor a vector z of dimensionality k, the softmaxis:Example:5.6\u2022MULTINOMIAL LOGISTIC REGRESSION15distributed according to a Gaussian distribution with mean\u00b5=0. In a Gaussianor normal distribution, the further away a value is from the mean, the lower itsprobability (scaled by the variances). By using a Gaussian prior on the weights, weare saying that weights prefer to have the value 0. A Gaussian for a weightqjis1q2ps2jexp \u0000(qj\u0000\u00b5j)22s2j!(5.27)If we multiply each weight by a Gaussian prior on the weight, we are thus maximiz-ing the following constraint:\u02c6q=argmaxqMYi=1P(y(i)|x(i))\u21e5nYj=11q2ps2jexp \u0000(qj\u0000\u00b5j)22s2j!(5.28)which in log space, with\u00b5=0, and assuming 2s2=1, corresponds to\u02c6q=argmaxqmXi=1logP(y(i)|x(i))\u0000anXj=1q2j(5.29)which is in the same form as Eq.5.24.5.6 Multinomial logistic regressionSometimes we need more than two classes. Perhaps we might want to do 3-waysentiment classi\ufb01cation (positive, negative, or neutral). Or we could be assigningsome of the labels we will introduce in Chapter 8, like the part of speech of a word(choosing from 10, 30, or even 50 different parts of speech), or the named entitytype of a phrase (choosing from tags like person, location, organization).In such cases we usemultinomial logistic regression, also calledsoftmax re-multinomiallogisticregressiongression(or, historically, themaxent classi\ufb01er). In multinomial logistic regressionthe targetyis a variable that ranges over more than two classes; we want to knowthe probability ofybeing in each potential classc2C,p(y=c|x).The multinomial logistic classi\ufb01er uses a generalization of the sigmoid, calledthesoftmaxfunction, to compute the probabilityp(y=c|x). The softmax functionsoftmaxtakes a vectorz=[z1,z2,. . . ,zk]ofkarbitrary values and maps them to a probabilitydistribution, with each value in the range (0,1), and all the values summing to 1.Like the sigmoid, it is an exponential function.For a vectorzof dimensionalityk, the softmax is de\ufb01ned as:softmax(zi)=exp(zi)Pkj=1exp(zj)1\uf8ffi\uf8ffk(5.30)The softmax of an input vectorz=[z1,z2,. . . ,zk]is thus a vector itself:softmax(z)=\"exp(z1)Pki=1exp(zi),exp(z2)Pki=1exp(zi),. . . ,exp(zk)Pki=1exp(zi)#(5.31)5.6\u2022MULTINOMIAL LOGISTIC REGRESSION15distributed according to a Gaussian distribution with mean\u00b5=0. In a Gaussianor normal distribution, the further away a value is from the mean, the lower itsprobability (scaled by the variances). By using a Gaussian prior on the weights, weare saying that weights prefer to have the value 0. A Gaussian for a weightqjis1q2ps2jexp \u0000(qj\u0000\u00b5j)22s2j!(5.27)If we multiply each weight by a Gaussian prior on the weight, we are thus maximiz-ing the following constraint:\u02c6q=argmaxqMYi=1P(y(i)|x(i))\u21e5nYj=11q2ps2jexp \u0000(qj\u0000\u00b5j)22s2j!(5.28)which in log space, with\u00b5=0, and assuming 2s2=1, corresponds to\u02c6q=argmaxqmXi=1logP(y(i)|x(i))\u0000anXj=1q2j(5.29)which is in the same form as Eq.5.24.5.6 Multinomial logistic regressionSometimes we need more than two classes. Perhaps we might want to do 3-waysentiment classi\ufb01cation (positive, negative, or neutral). Or we could be assigningsome of the labels we will introduce in Chapter 8, like the part of speech of a word(choosing from 10, 30, or even 50 different parts of speech), or the named entitytype of a phrase (choosing from tags like person, location, organization).In such cases we usemultinomial logistic regression, also calledsoftmax re-multinomiallogisticregressiongression(or, historically, themaxent classi\ufb01er). In multinomial logistic regressionthe targetyis a variable that ranges over more than two classes; we want to knowthe probability ofybeing in each potential classc2C,p(y=c|x).The multinomial logistic classi\ufb01er uses a generalization of the sigmoid, calledthesoftmaxfunction, to compute the probabilityp(y=c|x). The softmax functionsoftmaxtakes a vectorz=[z1,z2,. . . ,zk]ofkarbitrary values and maps them to a probabilitydistribution, with each value in the range (0,1), and all the values summing to 1.Like the sigmoid, it is an exponential function.For a vectorzof dimensionalityk, the softmax is de\ufb01ned as:softmax(zi)=exp(zi)Pkj=1exp(zj)1\uf8ffi\uf8ffk(5.30)The softmax of an input vectorz=[z1,z2,. . . ,zk]is thus a vector itself:softmax(z)=\"exp(z1)Pki=1exp(zi),exp(z2)Pki=1exp(zi),. . . ,exp(zk)Pki=1exp(zi)#(5.31)16CHAPTER5\u2022LOGISTICREGRESSIONThe denominatorPki=1exp(zi)is used to normalize all the values into probabil-ities. Thus for example given a vector:z=[0.6,1.1,\u00001.5,1.2,3.2,\u00001.1]the resulting (rounded) softmax(z) is[0.055,0.090,0.006,0.099,0.74,0.010]Again like the sigmoid, the input to the softmax will be the dot product betweena weight vectorwand an input vectorx(plus a bias). But now we\u2019ll need separateweight vectors (and bias) for each of theKclasses.p(y=c|x)=exp(wc\u00b7x+bc)kXj=1exp(wj\u00b7x+bj)(5.32)Like the sigmoid, the softmax has the property of squashing values toward 0 or 1.Thus if one of the inputs is larger than the others, it will tend to push its probabilitytoward 1, and suppress the probabilities of the smaller inputs.5.6.1 Features in Multinomial Logistic RegressionFeatures in multinomial logistic regression function similarly to binary logistic re-gression, with one difference that we\u2019ll need separate weight vectors (and biases) foreach of theKclasses. Recall our binary exclamation point featurex5from page4:x5=\u21e21 if \u201c!\u201d2doc0 otherwiseIn binary classi\ufb01cation a positive weightw5on a feature in\ufb02uences the classi\ufb01ertowardy=1 (positive sentiment) and a negative weight in\ufb02uences it towardy=0(negative sentiment) with the absolute value indicating how important the featureis. For multinominal logistic regression, by contrast, with separate weights for eachclass, a feature can be evidence for or against each individual class.In 3-way multiclass sentiment classi\ufb01cation, for example, we must assign eachdocument one of the 3 classes+,\u0000, or 0 (neutral). Now a feature related to excla-mation marks might have a negative weight for 0 documents, and a positive weightfor+or\u0000documents:Feature De\ufb01nitionw5,+w5,\u0000w5,0f5(x)\u21e21 if \u201c!\u201d2doc0 otherwise3.53.1\u00005.35.6.2 Learning in Multinomial Logistic RegressionThe loss function for multinomial logistic regression generalizes the loss functionfor binary logistic regression from 2 toKclasses. Recall that that the cross-entropyloss for binary logistic regression (repeated from Eq.5.11) is:LCE(\u02c6y,y)=\u0000logp(y|x)=\u0000[ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)](5.33)16CHAPTER5\u2022LOGISTICREGRESSIONThe denominatorPki=1exp(zi)is used to normalize all the values into probabil-ities. Thus for example given a vector:z=[0.6,1.1,\u00001.5,1.2,3.2,\u00001.1]the resulting (rounded) softmax(z) is[0.055,0.090,0.006,0.099,0.74,0.010]Again like the sigmoid, the input to the softmax will be the dot product betweena weight vectorwand an input vectorx(plus a bias). But now we\u2019ll need separateweight vectors (and bias) for each of theKclasses.p(y=c|x)=exp(wc\u00b7x+bc)kXj=1exp(wj\u00b7x+bj)(5.32)Like the sigmoid, the softmax has the property of squashing values toward 0 or 1.Thus if one of the inputs is larger than the others, it will tend to push its probabilitytoward 1, and suppress the probabilities of the smaller inputs.5.6.1 Features in Multinomial Logistic RegressionFeatures in multinomial logistic regression function similarly to binary logistic re-gression, with one difference that we\u2019ll need separate weight vectors (and biases) foreach of theKclasses. Recall our binary exclamation point featurex5from page4:x5=\u21e21 if \u201c!\u201d2doc0 otherwiseIn binary classi\ufb01cation a positive weightw5on a feature in\ufb02uences the classi\ufb01ertowardy=1 (positive sentiment) and a negative weight in\ufb02uences it towardy=0(negative sentiment) with the absolute value indicating how important the featureis. For multinominal logistic regression, by contrast, with separate weights for eachclass, a feature can be evidence for or against each individual class.In 3-way multiclass sentiment classi\ufb01cation, for example, we must assign eachdocument one of the 3 classes+,\u0000, or 0 (neutral). Now a feature related to excla-mation marks might have a negative weight for 0 documents, and a positive weightfor+or\u0000documents:Feature De\ufb01nitionw5,+w5,\u0000w5,0f5(x)\u21e21 if \u201c!\u201d2doc0 otherwise3.53.1\u00005.35.6.2 Learning in Multinomial Logistic RegressionThe loss function for multinomial logistic regression generalizes the loss functionfor binary logistic regression from 2 toKclasses. Recall that that the cross-entropyloss for binary logistic regression (repeated from Eq.5.11) is:LCE(\u02c6y,y)=\u0000logp(y|x)=\u0000[ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)](5.33)5.6\u2022MULTINOMIAL LOGISTIC REGRESSION15distributed according to a Gaussian distribution with mean\u00b5=0. In a Gaussianor normal distribution, the further away a value is from the mean, the lower itsprobability (scaled by the variances). By using a Gaussian prior on the weights, weare saying that weights prefer to have the value 0. A Gaussian for a weightqjis1q2ps2jexp \u0000(qj\u0000\u00b5j)22s2j!(5.27)If we multiply each weight by a Gaussian prior on the weight, we are thus maximiz-ing the following constraint:\u02c6q=argmaxqMYi=1P(y(i)|x(i))\u21e5nYj=11q2ps2jexp \u0000(qj\u0000\u00b5j)22s2j!(5.28)which in log space, with\u00b5=0, and assuming 2s2=1, corresponds to\u02c6q=argmaxqmXi=1logP(y(i)|x(i))\u0000anXj=1q2j(5.29)which is in the same form as Eq.5.24.5.6 Multinomial logistic regressionSometimes we need more than two classes. Perhaps we might want to do 3-waysentiment classi\ufb01cation (positive, negative, or neutral). Or we could be assigningsome of the labels we will introduce in Chapter 8, like the part of speech of a word(choosing from 10, 30, or even 50 different parts of speech), or the named entitytype of a phrase (choosing from tags like person, location, organization).In such cases we usemultinomial logistic regression, also calledsoftmax re-multinomiallogisticregressiongression(or, historically, themaxent classi\ufb01er). In multinomial logistic regressionthe targetyis a variable that ranges over more than two classes; we want to knowthe probability ofybeing in each potential classc2C,p(y=c|x).The multinomial logistic classi\ufb01er uses a generalization of the sigmoid, calledthesoftmaxfunction, to compute the probabilityp(y=c|x). The softmax functionsoftmaxtakes a vectorz=[z1,z2,. . . ,zk]ofkarbitrary values and maps them to a probabilitydistribution, with each value in the range (0,1), and all the values summing to 1.Like the sigmoid, it is an exponential function.For a vectorzof dimensionalityk, the softmax is de\ufb01ned as:softmax(zi)=exp(zi)Pkj=1exp(zj)1\uf8ffi\uf8ffk(5.30)The softmax of an input vectorz=[z1,z2,. . . ,zk]is thus a vector itself:softmax(z)=\"exp(z1)Pki=1exp(zi),exp(z2)Pki=1exp(zi),. . . ,exp(zk)Pki=1exp(zi)#(5.31) Two-Layer Network with scalar outputUWxnx1\n+1y is a scalarbhidden units(\u03c3node)Input layer(vector)Output layer(\u03c3node)\nCould be ReLUOr tanhz=\ud835\udc48\u210e\ud835\udc66=\ud835\udf0e(\ud835\udc67) Two-Layer Network with scalar outputUWxnx1\n+1bhidden units(\u03c3node)Input layer(vector)Output layer(\u03c3node)ijWjivectory is a scalarz=\ud835\udc48\u210e\ud835\udc66=\ud835\udf0e(\ud835\udc67) Two-Layer Network with scalar outputUWxnx1\n+1bhidden units(\u03c3node)Input layer(vector)Output layer(\u03c3node)\nCould be ReLUOr tanhy is a scalarz=\ud835\udc48\u210e\ud835\udc66=\ud835\udf0e(\ud835\udc67) Two-Layer Network with softmaxoutputUWxnx1\n+1bhidden units(\u03c3node)Input layer(vector)Output layer(\u03c3node)\nCould be ReLUOr tanhy is a vectorz=\ud835\udc48\u210e\ud835\udc66=softmax(\ud835\udc67) Multi-layer NotationW[1]xnx1+1b[1]ijW[2]b[2]\ud835\udc67[\"]=\ud835\udc4a[\"]\ud835\udc4e[$]+\ud835\udc4f[\"]\ud835\udc4e[$]\ud835\udc4e[\"]=\ud835\udc54\"(\ud835\udc67\")\ud835\udc67[%]=\ud835\udc4a[%]\ud835\udc4e[\"]+\ud835\udc4f[%]\ud835\udc4e[%]=\ud835\udc54%(\ud835\udc67%)\ud835\udc66=\ud835\udc4e[%]sigmoid or softmaxReLU Multi Layer Notation\n37\nx1x2x3\ny\nw1w2w3\u2211b\u03c3+1za Replacing the bias unitLet's switch to a notation without the bias unitJust a notational change1.Add a dummy node a0=1 to each layer2.Its weight w0will be the bias3.So input layer a[0]0=1, \u25e6And a[1]0=1 , a[2]0=1,\u2026 Replacing the bias unitInstead of:We'll do this:10CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELS(\ufb01rst) hidden layer, andb[1]will mean the bias vector for the (\ufb01rst) hidden layer.njwill mean the number of units at layer j. We\u2019ll useg(\u00b7)to stand for the activationfunction, which will tend to be ReLU or tanh for intermediate layers and softmaxfor output layers. We\u2019ll usea[i]to mean the output from layeri, andz[i]to mean thecombination of weights and biasesW[i]a[i\u00001]+b[i]. The 0th layer is for inputs, so theinputsxwe\u2019ll refer to more generally asa[0].Thus we can re-represent our 2-layer net from Eq.7.10as follows:z[1]=W[1]a[0]+b[1]a[1]=g[1](z[1])z[2]=W[2]a[1]+b[2]a[2]=g[2](z[2])\u02c6y=a[2](7.11)Note that with this notation, the equations for the computation done at each layer arethe same. The algorithm for computing the forward step in an n-layer feedforwardnetwork, given the input vectora[0]is thus simply:foriin1..nz[i]=W[i]a[i\u00001]+b[i]a[i]=g[i](z[i])\u02c6y=a[n]The activation functionsg(\u00b7)are generally different at the \ufb01nal layer. Thusg[2]might be softmax for multinomial classi\ufb01cation or sigmoid for binary classi\ufb01cation,while ReLU or tanh might be the activation functiong(\u00b7)at the internal layers.Replacing the bias unitIn describing networks, we will often use a slightly sim-pli\ufb01ed notation that represents exactly the same function without referring to an ex-plicit bias nodeb. Instead, we add a dummy nodea0to each layer whose value willalways be 1. Thus layer 0, the input layer, will have a dummy nodea[0]0=1, layer 1will havea[1]0=1, and so on. This dummy node still has an associated weight, andthat weight represents the bias valueb. For example instead of an equation likeh=s(Wx+b)(7.12)we\u2019ll use:h=s(Wx)(7.13)But now instead of our vectorxhavingnvalues:x=x1,...,xn, it will haven+1 values, with a new 0th dummy valuex0=1:x=x0,...,xn0. And instead ofcomputing eachhjas follows:hj=s n0Xi=1Wjixi+bj!,(7.14)we\u2019ll instead use:s n0Xi=0Wjixi!,(7.15)10CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELS(\ufb01rst) hidden layer, andb[1]will mean the bias vector for the (\ufb01rst) hidden layer.njwill mean the number of units at layer j. We\u2019ll useg(\u00b7)to stand for the activationfunction, which will tend to be ReLU or tanh for intermediate layers and softmaxfor output layers. We\u2019ll usea[i]to mean the output from layeri, andz[i]to mean thecombination of weights and biasesW[i]a[i\u00001]+b[i]. The 0th layer is for inputs, so theinputsxwe\u2019ll refer to more generally asa[0].Thus we can re-represent our 2-layer net from Eq.7.10as follows:z[1]=W[1]a[0]+b[1]a[1]=g[1](z[1])z[2]=W[2]a[1]+b[2]a[2]=g[2](z[2])\u02c6y=a[2](7.11)Note that with this notation, the equations for the computation done at each layer arethe same. The algorithm for computing the forward step in an n-layer feedforwardnetwork, given the input vectora[0]is thus simply:foriin1..nz[i]=W[i]a[i\u00001]+b[i]a[i]=g[i](z[i])\u02c6y=a[n]The activation functionsg(\u00b7)are generally different at the \ufb01nal layer. Thusg[2]might be softmax for multinomial classi\ufb01cation or sigmoid for binary classi\ufb01cation,while ReLU or tanh might be the activation functiong(\u00b7)at the internal layers.Replacing the bias unitIn describing networks, we will often use a slightly sim-pli\ufb01ed notation that represents exactly the same function without referring to an ex-plicit bias nodeb. Instead, we add a dummy nodea0to each layer whose value willalways be 1. Thus layer 0, the input layer, will have a dummy nodea[0]0=1, layer 1will havea[1]0=1, and so on. This dummy node still has an associated weight, andthat weight represents the bias valueb. For example instead of an equation likeh=s(Wx+b)(7.12)we\u2019ll use:h=s(Wx)(7.13)But now instead of our vectorxhavingnvalues:x=x1,...,xn, it will haven+1 values, with a new 0th dummy valuex0=1:x=x0,...,xn0. And instead ofcomputing eachhjas follows:hj=s n0Xi=1Wjixi+bj!,(7.14)we\u2019ll instead use:s n0Xi=0Wjixi!,(7.15)10CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELS(\ufb01rst) hidden layer, andb[1]will mean the bias vector for the (\ufb01rst) hidden layer.njwill mean the number of units at layer j. We\u2019ll useg(\u00b7)to stand for the activationfunction, which will tend to be ReLU or tanh for intermediate layers and softmaxfor output layers. We\u2019ll usea[i]to mean the output from layeri, andz[i]to mean thecombination of weights and biasesW[i]a[i\u00001]+b[i]. The 0th layer is for inputs, so theinputsxwe\u2019ll refer to more generally asa[0].Thus we can re-represent our 2-layer net from Eq.7.10as follows:z[1]=W[1]a[0]+b[1]a[1]=g[1](z[1])z[2]=W[2]a[1]+b[2]a[2]=g[2](z[2])\u02c6y=a[2](7.11)Note that with this notation, the equations for the computation done at each layer arethe same. The algorithm for computing the forward step in an n-layer feedforwardnetwork, given the input vectora[0]is thus simply:foriin1..nz[i]=W[i]a[i\u00001]+b[i]a[i]=g[i](z[i])\u02c6y=a[n]The activation functionsg(\u00b7)are generally different at the \ufb01nal layer. Thusg[2]might be softmax for multinomial classi\ufb01cation or sigmoid for binary classi\ufb01cation,while ReLU or tanh might be the activation functiong(\u00b7)at the internal layers.Replacing the bias unitIn describing networks, we will often use a slightly sim-pli\ufb01ed notation that represents exactly the same function without referring to an ex-plicit bias nodeb. Instead, we add a dummy nodea0to each layer whose value willalways be 1. Thus layer 0, the input layer, will have a dummy nodea[0]0=1, layer 1will havea[1]0=1, and so on. This dummy node still has an associated weight, andthat weight represents the bias valueb. For example instead of an equation likeh=s(Wx+b)(7.12)we\u2019ll use:h=s(Wx)(7.13)But now instead of our vectorxhavingnvalues:x=x1,...,xn, it will haven+1 values, with a new 0th dummy valuex0=1:x=x0,...,xn0. And instead ofcomputing eachhjas follows:hj=s n0Xi=1Wjixi+bj!,(7.14)we\u2019ll instead use:s n0Xi=0Wjixi!,(7.15)10CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELS(\ufb01rst) hidden layer, andb[1]will mean the bias vector for the (\ufb01rst) hidden layer.njwill mean the number of units at layer j. We\u2019ll useg(\u00b7)to stand for the activationfunction, which will tend to be ReLU or tanh for intermediate layers and softmaxfor output layers. We\u2019ll usea[i]to mean the output from layeri, andz[i]to mean thecombination of weights and biasesW[i]a[i\u00001]+b[i]. The 0th layer is for inputs, so theinputsxwe\u2019ll refer to more generally asa[0].Thus we can re-represent our 2-layer net from Eq.7.10as follows:z[1]=W[1]a[0]+b[1]a[1]=g[1](z[1])z[2]=W[2]a[1]+b[2]a[2]=g[2](z[2])\u02c6y=a[2](7.11)Note that with this notation, the equations for the computation done at each layer arethe same. The algorithm for computing the forward step in an n-layer feedforwardnetwork, given the input vectora[0]is thus simply:foriin1..nz[i]=W[i]a[i\u00001]+b[i]a[i]=g[i](z[i])\u02c6y=a[n]The activation functionsg(\u00b7)are generally different at the \ufb01nal layer. Thusg[2]might be softmax for multinomial classi\ufb01cation or sigmoid for binary classi\ufb01cation,while ReLU or tanh might be the activation functiong(\u00b7)at the internal layers.Replacing the bias unitIn describing networks, we will often use a slightly sim-pli\ufb01ed notation that represents exactly the same function without referring to an ex-plicit bias nodeb. Instead, we add a dummy nodea0to each layer whose value willalways be 1. Thus layer 0, the input layer, will have a dummy nodea[0]0=1, layer 1will havea[1]0=1, and so on. This dummy node still has an associated weight, andthat weight represents the bias valueb. For example instead of an equation likeh=s(Wx+b)(7.12)we\u2019ll use:h=s(Wx)(7.13)But now instead of our vectorxhavingnvalues:x=x1,...,xn, it will haven+1 values, with a new 0th dummy valuex0=1:x=x0,...,xn0. And instead ofcomputing eachhjas follows:hj=s n0Xi=1Wjixi+bj!,(7.14)we\u2019ll instead use:s n0Xi=0Wjixi!,(7.15)x= x1, x2, \u2026, xn0x= x0, x1, x2, \u2026, xn0 Replacing the bias unit\nx1x2y1xn0\u2026\u2026+1b\u2026UWy2yn2h1h2h3hn1x1x2y1xn0\u2026\u2026x0=1\u2026UWy2yn2h1h2h3hn1Instead of:We'll do this: Simple Neural Networks and Neural Language ModelsFeedforward Neural Networks Simple Neural Networks and Neural Language ModelsApplying feedforward networks to NLP tasks Use cases for feedforward networksLet's consider 2 (simplified) sample tasks:1.Text classification2.Language modelingState of the art systems use more powerful neural architectures, but simple models are useful to consider!43 Classification: Sentiment AnalysisWe could do exactly what we did with logistic regressionInput layer are binary features as beforeOutput layer is 0 or 1UWxnx1\u03c3 Sentiment Features\n454CHAPTER5\u2022LOGISTICREGRESSIONnearly linear around 0 but has a sharp slope toward the ends, it tends to squash outliervalues toward 0 or 1. And it\u2019s differentiable, which as we\u2019ll see in Section5.8willbe handy for learning.We\u2019re almost there. If we apply the sigmoid to the sum of the weighted features,we get a number between 0 and 1. To make it a probability, we just need to makesure that the two cases,p(y=1)andp(y=0), sum to 1. We can do this as follows:P(y=1)=s(w\u00b7x+b)=11+e\u0000(w\u00b7x+b)P(y=0)=1\u0000s(w\u00b7x+b)=1\u000011+e\u0000(w\u00b7x+b)=e\u0000(w\u00b7x+b)1+e\u0000(w\u00b7x+b)(5.5)Now we have an algorithm that given an instancexcomputes the probabilityP(y=1|x). How do we make a decision? For a test instancex, we say yes if the probabilityP(y=1|x)is more than .5, and no otherwise. We call .5 thedecision boundary:decisionboundary\u02c6y=\u21e21 ifP(y=1|x)>0.50 otherwise5.1.1 Example: sentiment classi\ufb01cationLet\u2019s have an example. Suppose we are doing binary sentiment classi\ufb01cation onmovie review text, and we would like to know whether to assign the sentiment class+or\u0000to a review documentdoc. We\u2019ll represent each input observation by the 6featuresx1...x6of the input shown in the following table; Fig.5.2shows the featuresin a sample mini test document.Var De\ufb01nition Value in Fig.5.2x1count(positive lexicon)2doc)3x2count(negative lexicon)2doc)2x3\u21e21 if \u201cno\u201d2doc0 otherwise1x4count(1st and 2nd pronouns2doc)3x5\u21e21 if \u201c!\u201d2doc0 otherwise0x6log(word count of doc)ln(66)=4.19Let\u2019s assume for the moment that we\u2019ve already learned a real-valued weight foreach of these features, and that the 6 weights corresponding to the 6 features are[2.5,\u00005.0,\u00001.2,0.5,2.0,0.7], whileb= 0.1. (We\u2019ll discuss in the next section howthe weights are learned.) The weightw1, for example indicates how important afeature the number of positive lexicon words (great,nice,enjoyable, etc.) is toa positive sentiment decision, whilew2tells us the importance of negative lexiconwords. Note thatw1=2.5 is positive, whilew2=\u00005.0, meaning that negative wordsare negatively associated with a positive sentiment decision, and are about twice asimportant as positive words. Feedforward nets for simple classificationJust adding a hidden layer to logistic regression\u2022allows the network to use non-linear interactions between features \u2022which may (or may not) improve performance.46UWxnx1f1f2fn46Wxnx1f1f2fnLogisticRegression2-layerfeedforwardnetwork\u03c3\u03c3 Even better: representation learningThe real power of deep learningcomes from the  ability to learnfeatures from the dataInstead of using hand-built human-engineered features for classificationUse learned representations like embeddings!47UWxnx1e1e2en\u03c3 48Neural Net Classification with embeddings as input features!\nh1h2h3hdh\u2026UWy\n3d\u2a091Hidden layerOutput layersigmoid\nThe...dessertiswt-1\nw2w1dh\u2a093ddh\u2a091|V|\u2a09dhProjection layerembeddingsp(positive sentiment|The dessert is\u2026)^\nembedding forword 7embedding for word 23864embedding forword 534w3E\u2026 Issue: texts come in different sizesThis assumes a fixed size length (3)!  Kind of unrealistic.   Some simple solutions (more sophisticated solutions later)1.Make the input the length of the longest review\u2022If shorter then pad with zero embeddings\u2022Truncate if you get longer reviews at test time2.Create a single \"sentence embedding\" (the same dimensionality as a word) to represent all the words\u2022Take the mean of all the word embeddings\u2022Take the element-wise max of all the word embeddings\u2022For each dimension, pick the max value from all words49h1h2h3hdh\u2026UWy\n3d\u2a091Hidden layerOutput layersigmoid\nThe...dessertiswt-1\nw2w1dh\u2a093ddh\u2a091|V|\u2a09dhProjection layerembeddingsp(positive sentiment|The dessert is\u2026)^\nembedding forword 7embedding for word 23864embedding forword 534w3E\u2026 Reminder: Multiclass OutputsWhat if you have more than two output classes?\u25e6Add more output units (one for each class)\u25e6And use a \u201csoftmaxlayer\u201d\n50UWxnx1\n Neural Language Models (LMs)Language Modeling: Calculating the probability of the next word in a sequence given some history. \u2022We've seen N-gram based LMs\u2022But neural network LMs far outperform n-gram language modelsState-of-the-art neural LMs are based on more powerful neural network technology like TransformersBut simple feedforward LMs can do almost as well!51 Simple feedforward Neural Language ModelsTask: predict next word wtgiven prior words wt-1, wt-2, wt-3, \u2026Problem: Now we\u2019re dealing with sequences of arbitrary length.Solution: Sliding windows (of fixed length)\n52\n 53Neural Language Model \n Why Neural LMs work better than N-gram LMsTraining data:We've seen:  I have to make sure that the cat gets fed. Never seen:   dog gets fedTest data:I forgot to make sure that the dog gets ___N-gram LM can't predict \"fed\"!Neural LM can use similarity of \"cat\" and \"dog\" embeddings to generalize and predict \u201cfed\u201d after dog Simple Neural Networks and Neural Language ModelsApplying feedforward networks to NLP tasks Simple Neural Networks and Neural Language ModelsTraining Neural Nets: Overview Intuition: training a 2-layer Network\n57UWxnx1System output !\ud835\udc66Actual answer \ud835\udc66\nTraining instanceLoss function L(!\ud835\udc66,\ud835\udc66)\nForward passBackward pass Intuition: Training a 2-layer networkFor every training tuple (\ud835\udc65,\ud835\udc66)\u25e6Run forwardcomputation to find our estimate 3\ud835\udc66\u25e6Run backwardcomputation to update weights: \u25e6For every output node\u25e6Compute loss \ud835\udc3fbetween true \ud835\udc66and the estimated #\ud835\udc66\u25e6For every weight \ud835\udc64from hidden layer to the output layer\u25e6Update the weight\u25e6For every hidden node\u25e6Assess how much blame it deserves for the current answer\u25e6For every weight \ud835\udc64from input layer to the hidden layer\u25e6Update the weight58 Reminder: Loss Function for binary logistic regressionA measure for how far off the current answer is to the right answerCross entropy loss for logistic regression:\n595.3\u2022THE CROSS-ENTROPY LOSS FUNCTION7thecross-entropy loss.The second thing we need is an optimization algorithm for iteratively updatingthe weights so as to minimize this loss function. The standard algorithm for this isgradient descent; we\u2019ll introduce thestochastic gradient descentalgorithm in thefollowing section.5.3 The cross-entropy loss functionWe need a loss function that expresses, for an observationx, how close the classi\ufb01eroutput ( \u02c6y=s(w\u00b7x+b)) is to the correct output (y, which is 0 or 1). We\u2019ll call this:L(\u02c6y,y)=How much \u02c6ydiffers from the truey(5.8)We do this via a loss function that prefers the correct class labels of the train-ing examples to bemore likely. This is calledconditional maximum likelihoodestimation: we choose the parametersw,bthatmaximize the log probability ofthe trueylabels in the training datagiven the observationsx. The resulting lossfunction is the negative log likelihood loss, generally called thecross-entropy loss.cross-entropylossLet\u2019s derive this loss function, applied to a single observationx. We\u2019d like tolearn weights that maximize the probability of the correct labelp(y|x). Since thereare only two discrete outcomes (1 or 0), this is a Bernoulli distribution, and we canexpress the probabilityp(y|x)that our classi\ufb01er produces for one observation asthe following (keeping in mind that if y=1, Eq.5.9simpli\ufb01es to \u02c6y; if y=0, Eq.5.9simpli\ufb01es to 1\u0000\u02c6y):p(y|x)=\u02c6yy(1\u0000\u02c6y)1\u0000y(5.9)Now we take the log of both sides. This will turn out to be handy mathematically,and doesn\u2019t hurt us; whatever values maximize a probability will also maximize thelog of the probability:logp(y|x)=log\u21e5\u02c6yy(1\u0000\u02c6y)1\u0000y\u21e4=ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)(5.10)Eq.5.10describes a log likelihood that should be maximized. In order to turn thisinto loss function (something that we need to minimize), we\u2019ll just \ufb02ip the sign onEq.5.10. The result is the cross-entropy lossLCE:LCE(\u02c6y,y)=\u0000logp(y|x)=\u0000[ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)](5.11)Finally, we can plug in the de\ufb01nition of \u02c6y=s(w\u00b7x+b):LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))](5.12)Let\u2019s see if this loss function does the right thing for our example from Fig.5.2.W ewant the loss to be smaller if the model\u2019s estimate is close to correct, and bigger ifthe model is confused. So \ufb01rst let\u2019s suppose the correct gold label for the sentimentexample in Fig.5.2is positive, i.e.,y=1. In this case our model is doing well, sincefrom Eq.5.7it indeed gave the example a higher probability of being positive (.69)than negative (.31). If we plugs(w\u00b7x+b)=.69 andy=1 into Eq.5.12, the right5.3\u2022THE CROSS-ENTROPY LOSS FUNCTION7thecross-entropy loss.The second thing we need is an optimization algorithm for iteratively updatingthe weights so as to minimize this loss function. The standard algorithm for this isgradient descent; we\u2019ll introduce thestochastic gradient descentalgorithm in thefollowing section.5.3 The cross-entropy loss functionWe need a loss function that expresses, for an observationx, how close the classi\ufb01eroutput ( \u02c6y=s(w\u00b7x+b)) is to the correct output (y, which is 0 or 1). We\u2019ll call this:L(\u02c6y,y)=How much \u02c6ydiffers from the truey(5.8)We do this via a loss function that prefers the correct class labels of the train-ing examples to bemore likely. This is calledconditional maximum likelihoodestimation: we choose the parametersw,bthatmaximize the log probability ofthe trueylabels in the training datagiven the observationsx. The resulting lossfunction is the negative log likelihood loss, generally called thecross-entropy loss.cross-entropylossLet\u2019s derive this loss function, applied to a single observationx. We\u2019d like tolearn weights that maximize the probability of the correct labelp(y|x). Since thereare only two discrete outcomes (1 or 0), this is a Bernoulli distribution, and we canexpress the probabilityp(y|x)that our classi\ufb01er produces for one observation asthe following (keeping in mind that if y=1, Eq.5.9simpli\ufb01es to \u02c6y; if y=0, Eq.5.9simpli\ufb01es to 1\u0000\u02c6y):p(y|x)=\u02c6yy(1\u0000\u02c6y)1\u0000y(5.9)Now we take the log of both sides. This will turn out to be handy mathematically,and doesn\u2019t hurt us; whatever values maximize a probability will also maximize thelog of the probability:logp(y|x)=log\u21e5\u02c6yy(1\u0000\u02c6y)1\u0000y\u21e4=ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)(5.10)Eq.5.10describes a log likelihood that should be maximized. In order to turn thisinto loss function (something that we need to minimize), we\u2019ll just \ufb02ip the sign onEq.5.10. The result is the cross-entropy lossLCE:LCE(\u02c6y,y)=\u0000logp(y|x)=\u0000[ylog \u02c6y+(1\u0000y)log(1\u0000\u02c6y)](5.11)Finally, we can plug in the de\ufb01nition of \u02c6y=s(w\u00b7x+b):LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))](5.12)Let\u2019s see if this loss function does the right thing for our example from Fig.5.2.W ewant the loss to be smaller if the model\u2019s estimate is close to correct, and bigger ifthe model is confused. So \ufb01rst let\u2019s suppose the correct gold label for the sentimentexample in Fig.5.2is positive, i.e.,y=1. In this case our model is doing well, sincefrom Eq.5.7it indeed gave the example a higher probability of being positive (.69)than negative (.31). If we plugs(w\u00b7x+b)=.69 andy=1 into Eq.5.12, the right Reminder: gradient descent for weight updatesUse the derivative of the loss function with respect to weights !!\"\ud835\udc3f(\ud835\udc53\ud835\udc65;\ud835\udc64,\ud835\udc66)To tell us how to adjust weights for each training item \u25e6Move them in the opposite direction of the gradient\u25e6For logistic regression10CHAPTER5\u2022LOGISTICREGRESSIONexample):wt+1=wt\u0000hddwL(f(x;w),y)(5.14)Now let\u2019s extend the intuition from a function of one scalar variablewto manyvariables, because we don\u2019t just want to move left or right, we want to know wherein the N-dimensional space (of theNparameters that make upq) we should move.Thegradientis just such a vector; it expresses the directional components of thesharpest slope along each of thoseNdimensions. If we\u2019re just imagining two weightdimensions (say for one weightwand one biasb), the gradient might be a vector withtwo orthogonal components, each of which tells us how much the ground slopes inthewdimension and in thebdimension. Fig.5.4shows a visualization of the valueof a 2-dimensional gradient vector taken at the red point.\nCost(w,b)\nwbFigure 5.4Visualization of the gradient vector at the red point in two dimensionswandb,showing the gradient as a red arrow in the x-y plane.In an actual logistic regression, the parameter vectorwis much longer than 1 or2, since the input feature vectorxcan be quite long, and we need a weightwiforeachxi. For each dimension/variablewiinw(plus the biasb), the gradient will havea component that tells us the slope with respect to that variable. Essentially we\u2019reasking: \u201cHow much would a small change in that variablewiin\ufb02uence the total lossfunctionL?\u201dIn each dimensionwi, we express the slope as a partial derivative\u2202\u2202wiof the lossfunction. The gradient is then de\ufb01ned as a vector of these partials. We\u2019ll represent \u02c6yasf(x;q)to make the dependence onqmore obvious:\u2014qL(f(x;q),y)) =266664\u2202\u2202w1L(f(x;q),y)\u2202\u2202w2L(f(x;q),y)...\u2202\u2202wnL(f(x;q),y)377775(5.15)The \ufb01nal equation for updatingqbased on the gradient is thusqt+1=qt\u0000h\u2014L(f(x;q),y)(5.16)5.4\u2022GRADIENTDESCENT115.4.1 The Gradient for Logistic RegressionIn order to updateq, we need a de\ufb01nition for the gradient\u2014L(f(x;q),y). Recall thatfor logistic regression, the cross-entropy loss function is:LCE(\u02c6y,y)=\u0000[ylogs(w\u00b7x+b)+(1\u0000y)log(1\u0000s(w\u00b7x+b))](5.17)It turns out that the derivative of this function for one observation vectorxisEq.5.18(the interested reader can see Section5.8for the derivation of this equation):\u2202LCE(\u02c6y,y)\u2202wj=[s(w\u00b7x+b)\u0000y]xj(5.18)Note in Eq.5.18that the gradient with respect to a single weightwjrepresents avery intuitive value: the difference between the trueyand our estimated \u02c6y=s(w\u00b7x+b)for that observation, multiplied by the corresponding input valuexj.5.4.2 The Stochastic Gradient Descent AlgorithmStochastic gradient descent is an online algorithm that minimizes the loss functionby computing its gradient after each training example, and nudgingqin the rightdirection (the opposite direction of the gradient). Fig.5.5shows the algorithm.functionSTOCHASTICGRADIENTDESCENT(L(),f(),x,y)returnsq# where: L is the loss function# f is a function parameterized byq# x is the set of training inputsx(1),x(2),. .. ,x(m)# y is the set of training outputs (labels)y(1),y(2),. .. ,y(m)q 0repeattil done # see captionFor each training tuple(x(i),y(i))(in random order)1. Optional (for reporting): # How are we doing on this tuple?Compute \u02c6y(i)=f(x(i);q)# What is our estimated output \u02c6y?Compute the lossL(\u02c6y(i),y(i))# How far off is \u02c6y(i))from the true outputy(i)?2.g \u2014qL(f(x(i);q),y(i))# How should we moveqto maximize loss?3.q q\u0000hg# Go the other way insteadreturnqFigure 5.5The stochastic gradient descent algorithm. Step 1 (computing the loss) is usedto report how well we are doing on the current tuple. The algorithm can terminate when itconverges (or when the gradient norm<\u270f), or when progress halts (for example when theloss starts going up on a held-out set).The learning ratehis ahyperparameterthat must be adjusted. If it\u2019s too high,hyperparameterthe learner will take steps that are too large, overshooting the minimum of the lossfunction. If it\u2019s too low, the learner will take steps that are too small, and take toolong to get to the minimum. It is common to start with a higher learning rate and thenslowly decrease it, so that it is a function of the iterationkof training; the notationhkcan be used to mean the value of the learning rate at iterationk.We\u2019ll discuss hyperparameters in more detail in Chapter 7, but brie\ufb02y they area special kind of parameter for any machine learning model. Unlike regular param-eters of a model (weights likewandb), which are learned by the algorithm fromthe training set, hyperparameters are special parameters chosen by the algorithmdesigner that affect how the algorithm works. Where did that derivative come from?Using the chain rule!   f (x) = u(v(x)) Intuition (see the text for details)\n61x1x2x3\ny\nw1w2w3\u2211b\u03c3+1zaDerivative of the LossDerivative of the ActivationDerivative of the weighted sum!\"!#!=!\"!$!$!%!%!#!14CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELSe=a+dd = 2bL=ce31-2e=5d=2L=-10forward passabcFigure 7.10Computation graph for the functionL(a,b,c)=c(a+2b), with values for inputnodesa=3,b=1,c=\u00002, showing the forward pass computation ofL.off(x)is the derivative ofu(x)with respect tov(x)times the derivative ofv(x)withrespect tox:dfdx=dudv\u00b7dvdx(7.23)The chain rule extends to more than two functions. If computing the derivative of acomposite functionf(x)=u(v(w(x))), the derivative off(x)is:dfdx=dudv\u00b7dvdw\u00b7dwdx(7.24)Let\u2019s now compute the 3 derivatives we need. Since in the computation graphL=ce, we can directly compute the derivative\u2202L\u2202c:\u2202L\u2202c=e(7.25)For the other two, we\u2019ll need to use the chain rule:\u2202L\u2202a=\u2202L\u2202e\u2202e\u2202a\u2202L\u2202b=\u2202L\u2202e\u2202e\u2202d\u2202d\u2202b(7.26)Eq.7.26thus requires \ufb01ve intermediate derivatives:\u2202L\u2202e,\u2202L\u2202c,\u2202e\u2202a,\u2202e\u2202d, and\u2202d\u2202b,which are as follows (making use of the fact that the derivative of a sum is the sumof the derivatives):L=ce:\u2202L\u2202e=c,\u2202L\u2202c=ee=a+d:\u2202e\u2202a=1,\u2202e\u2202d=1d=2b:\u2202d\u2202b=2In the backward pass, we compute each of these partials along each edge of the graphfrom right to left, multiplying the necessary partials to result in the \ufb01nal derivativewe need. Thus we begin by annotating the \ufb01nal node with\u2202L\u2202L=1. Moving to theleft, we then compute\u2202L\u2202cand\u2202L\u2202e, and so on, until we have annotated the graph allthe way to the input variables. The forward pass conveniently already will havecomputed the values of the forward intermediate variables we need (likedande) How can I find that gradient for every weight in the network?These derivatives on the prior slide only give the updates for one weight layer: the last one! What about deeper networks?\u2022Lots of layers, different activation functions?Solution in the next lecture:\u2022Even more use of the chain rule!! \u2022Computation graphs and backward differentiation!62 Simple Neural Networks and Neural Language ModelsTraining Neural Nets: Overview Simple Neural Networks and Neural Language ModelsComputation Graphs and Backward Differentiation Why Computation GraphsFor training, we need the derivative of the loss with respect to each weight in every layer of the network \u2022But the loss is computed only at the very end of the network! Solution: error backpropagation (Rumelhart, Hinton, Williams, 1986) \u2022Backpropis a special case of backward differentiation\u2022Which relies on computation graphs. \n65 Computation GraphsA computation graph represents the process of computing a mathematical expression\n66 Example: \n67\ne=a+dd = 2bL=ceabcComputations: Example: \n68\nComputations: Backwards differentiation in computation graphsThe importance of the computation graph comes from the backward passThis is used to compute the derivatives that we\u2019ll need for the weight update.  Example\n70\nThe derivative !\"!#, tells us how much a small change in a affects L. We want:7.4\u2022TRAININGNEURALNETS13Or for a network with one hidden layer and softmax output, we could use the deriva-tive of the softmax loss from Eq.??:\u2202LCE\u2202wk=(\n{y=k}\u0000p(y=k|x))xk= \n{y=k}\u0000exp(wk\u00b7x+bk)PKj=1exp(wj\u00b7x+bj)!xk(7.22)But these derivatives only give correct updates for one weight layer: the last one!For deep networks, computing the gradients for each weight is much more complex,since we are computing the derivative with respect to weight parameters that appearall the way back in the very early layers of the network, even though the loss iscomputed only at the very end of the network.The solution to computing this gradient is an algorithm callederror backprop-agationorbackprop(Rumelhart et al., 1986). While backprop was invented spe-error back-propagationcially for neural networks, it turns out to be the same as a more general procedurecalledbackward differentiation, which depends on the notion ofcomputationgraphs. Let\u2019s see how that works in the next subsection.7.4.3 Computation GraphsA computation graph is a representation of the process of computing a mathematicalexpression, in which the computation is broken down into separate operations, eachof which is modeled as a node in a graph.Consider computing the functionL(a,b,c)=c(a+2b). If we make each of thecomponent addition and multiplication operations explicit, and add names (dande)for the intermediate outputs, the resulting series of computations is:d=2\u21e4be=a+dL=c\u21e4eWe can now represent this as a graph, with nodes for each operation, and di-rected edges showing the outputs from each operation as the inputs to the next, asin Fig.7.10. The simplest use of computation graphs is to compute the value ofthe function with some given inputs. In the \ufb01gure, we\u2019ve assumed the inputsa=3,b=1,c=\u00002, and we\u2019ve shown the result of theforward passto compute the re-sultL(3,1,\u00002)=\u000010. In the forward pass of a computation graph, we apply eachoperation left to right, passing the outputs of each computation as the input to thenext node.7.4.4 Backward differentiation on computation graphsThe importance of the computation graph comes from thebackward pass, whichis used to compute the derivatives that we\u2019ll need for the weight update. In thisexample our goal is to compute the derivative of the output functionLwith respectto each of the input variables, i.e.,\u2202L\u2202a,\u2202L\u2202b, and\u2202L\u2202c. The derivative\u2202L\u2202a, tells us howmuch a small change inaaffectsL.Backwards differentiation makes use of thechain rulein calculus. Suppose wechain ruleare computing the derivative of a composite functionf(x)=u(v(x)). The derivative The chain ruleComputing the derivative of a composite function:f (x) = u(v(x))f (x) = u(v(w(x))) 14CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELSe=a+dd = 2bL=ce31-2e=5d=2L=-10forward passabcFigure 7.10Computation graph for the functionL(a,b,c)=c(a+2b), with values for inputnodesa=3,b=1,c=\u00002, showing the forward pass computation ofL.off(x)is the derivative ofu(x)with respect tov(x)times the derivative ofv(x)withrespect tox:dfdx=dudv\u00b7dvdx(7.23)The chain rule extends to more than two functions. If computing the derivative of acomposite functionf(x)=u(v(w(x))), the derivative off(x)is:dfdx=dudv\u00b7dvdw\u00b7dwdx(7.24)Let\u2019s now compute the 3 derivatives we need. Since in the computation graphL=ce, we can directly compute the derivative\u2202L\u2202c:\u2202L\u2202c=e(7.25)For the other two, we\u2019ll need to use the chain rule:\u2202L\u2202a=\u2202L\u2202e\u2202e\u2202a\u2202L\u2202b=\u2202L\u2202e\u2202e\u2202d\u2202d\u2202b(7.26)Eq.7.26thus requires \ufb01ve intermediate derivatives:\u2202L\u2202e,\u2202L\u2202c,\u2202e\u2202a,\u2202e\u2202d, and\u2202d\u2202b,which are as follows (making use of the fact that the derivative of a sum is the sumof the derivatives):L=ce:\u2202L\u2202e=c,\u2202L\u2202c=ee=a+d:\u2202e\u2202a=1,\u2202e\u2202d=1d=2b:\u2202d\u2202b=2In the backward pass, we compute each of these partials along each edge of the graphfrom right to left, multiplying the necessary partials to result in the \ufb01nal derivativewe need. Thus we begin by annotating the \ufb01nal node with\u2202L\u2202L=1. Moving to theleft, we then compute\u2202L\u2202cand\u2202L\u2202e, and so on, until we have annotated the graph allthe way to the input variables. The forward pass conveniently already will havecomputed the values of the forward intermediate variables we need (likedande)14CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELSe=a+dd = 2bL=ce31-2e=5d=2L=-10forward passabcFigure 7.10Computation graph for the functionL(a,b,c)=c(a+2b), with values for inputnodesa=3,b=1,c=\u00002, showing the forward pass computation ofL.off(x)is the derivative ofu(x)with respect tov(x)times the derivative ofv(x)withrespect tox:dfdx=dudv\u00b7dvdx(7.23)The chain rule extends to more than two functions. If computing the derivative of acomposite functionf(x)=u(v(w(x))), the derivative off(x)is:dfdx=dudv\u00b7dvdw\u00b7dwdx(7.24)Let\u2019s now compute the 3 derivatives we need. Since in the computation graphL=ce, we can directly compute the derivative\u2202L\u2202c:\u2202L\u2202c=e(7.25)For the other two, we\u2019ll need to use the chain rule:\u2202L\u2202a=\u2202L\u2202e\u2202e\u2202a\u2202L\u2202b=\u2202L\u2202e\u2202e\u2202d\u2202d\u2202b(7.26)Eq.7.26thus requires \ufb01ve intermediate derivatives:\u2202L\u2202e,\u2202L\u2202c,\u2202e\u2202a,\u2202e\u2202d, and\u2202d\u2202b,which are as follows (making use of the fact that the derivative of a sum is the sumof the derivatives):L=ce:\u2202L\u2202e=c,\u2202L\u2202c=ee=a+d:\u2202e\u2202a=1,\u2202e\u2202d=1d=2b:\u2202d\u2202b=2In the backward pass, we compute each of these partials along each edge of the graphfrom right to left, multiplying the necessary partials to result in the \ufb01nal derivativewe need. Thus we begin by annotating the \ufb01nal node with\u2202L\u2202L=1. Moving to theleft, we then compute\u2202L\u2202cand\u2202L\u2202e, and so on, until we have annotated the graph allthe way to the input variables. The forward pass conveniently already will havecomputed the values of the forward intermediate variables we need (likedande) 14CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELSe=a+dd = 2bL=ce31-2e=5d=2L=-10forward passabcFigure 7.10Computation graph for the functionL(a,b,c)=c(a+2b), with values for inputnodesa=3,b=1,c=\u00002, showing the forward pass computation ofL.off(x)is the derivative ofu(x)with respect tov(x)times the derivative ofv(x)withrespect tox:dfdx=dudv\u00b7dvdx(7.23)The chain rule extends to more than two functions. If computing the derivative of acomposite functionf(x)=u(v(w(x))), the derivative off(x)is:dfdx=dudv\u00b7dvdw\u00b7dwdx(7.24)Let\u2019s now compute the 3 derivatives we need. Since in the computation graphL=ce, we can directly compute the derivative\u2202L\u2202c:\u2202L\u2202c=e(7.25)For the other two, we\u2019ll need to use the chain rule:\u2202L\u2202a=\u2202L\u2202e\u2202e\u2202a\u2202L\u2202b=\u2202L\u2202e\u2202e\u2202d\u2202d\u2202b(7.26)Eq.7.26thus requires \ufb01ve intermediate derivatives:\u2202L\u2202e,\u2202L\u2202c,\u2202e\u2202a,\u2202e\u2202d, and\u2202d\u2202b,which are as follows (making use of the fact that the derivative of a sum is the sumof the derivatives):L=ce:\u2202L\u2202e=c,\u2202L\u2202c=ee=a+d:\u2202e\u2202a=1,\u2202e\u2202d=1d=2b:\u2202d\u2202b=2In the backward pass, we compute each of these partials along each edge of the graphfrom right to left, multiplying the necessary partials to result in the \ufb01nal derivativewe need. Thus we begin by annotating the \ufb01nal node with\u2202L\u2202L=1. Moving to theleft, we then compute\u2202L\u2202cand\u2202L\u2202e, and so on, until we have annotated the graph allthe way to the input variables. The forward pass conveniently already will havecomputed the values of the forward intermediate variables we need (likedande)Example\n72\n14CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELSe=a+dd = 2bL=ce31-2e=5d=2L=-10forward passabcFigure 7.10Computation graph for the functionL(a,b,c)=c(a+2b), with values for inputnodesa=3,b=1,c=\u00002, showing the forward pass computation ofL.off(x)is the derivative ofu(x)with respect tov(x)times the derivative ofv(x)withrespect tox:dfdx=dudv\u00b7dvdx(7.23)The chain rule extends to more than two functions. If computing the derivative of acomposite functionf(x)=u(v(w(x))), the derivative off(x)is:dfdx=dudv\u00b7dvdw\u00b7dwdx(7.24)Let\u2019s now compute the 3 derivatives we need. Since in the computation graphL=ce, we can directly compute the derivative\u2202L\u2202c:\u2202L\u2202c=e(7.25)For the other two, we\u2019ll need to use the chain rule:\u2202L\u2202a=\u2202L\u2202e\u2202e\u2202a\u2202L\u2202b=\u2202L\u2202e\u2202e\u2202d\u2202d\u2202b(7.26)Eq.7.26thus requires \ufb01ve intermediate derivatives:\u2202L\u2202e,\u2202L\u2202c,\u2202e\u2202a,\u2202e\u2202d, and\u2202d\u2202b,which are as follows (making use of the fact that the derivative of a sum is the sumof the derivatives):L=ce:\u2202L\u2202e=c,\u2202L\u2202c=ee=a+d:\u2202e\u2202a=1,\u2202e\u2202d=1d=2b:\u2202d\u2202b=2In the backward pass, we compute each of these partials along each edge of the graphfrom right to left, multiplying the necessary partials to result in the \ufb01nal derivativewe need. Thus we begin by annotating the \ufb01nal node with\u2202L\u2202L=1. Moving to theleft, we then compute\u2202L\u2202cand\u2202L\u2202e, and so on, until we have annotated the graph allthe way to the input variables. The forward pass conveniently already will havecomputed the values of the forward intermediate variables we need (likedande)14CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELSe=a+dd = 2bL=ce31-2e=5d=2L=-10forward passabcFigure 7.10Computation graph for the functionL(a,b,c)=c(a+2b), with values for inputnodesa=3,b=1,c=\u00002, showing the forward pass computation ofL.off(x)is the derivative ofu(x)with respect tov(x)times the derivative ofv(x)withrespect tox:dfdx=dudv\u00b7dvdx(7.23)The chain rule extends to more than two functions. If computing the derivative of acomposite functionf(x)=u(v(w(x))), the derivative off(x)is:dfdx=dudv\u00b7dvdw\u00b7dwdx(7.24)Let\u2019s now compute the 3 derivatives we need. Since in the computation graphL=ce, we can directly compute the derivative\u2202L\u2202c:\u2202L\u2202c=e(7.25)For the other two, we\u2019ll need to use the chain rule:\u2202L\u2202a=\u2202L\u2202e\u2202e\u2202a\u2202L\u2202b=\u2202L\u2202e\u2202e\u2202d\u2202d\u2202b(7.26)Eq.7.26thus requires \ufb01ve intermediate derivatives:\u2202L\u2202e,\u2202L\u2202c,\u2202e\u2202a,\u2202e\u2202d, and\u2202d\u2202b,which are as follows (making use of the fact that the derivative of a sum is the sumof the derivatives):L=ce:\u2202L\u2202e=c,\u2202L\u2202c=ee=a+d:\u2202e\u2202a=1,\u2202e\u2202d=1d=2b:\u2202d\u2202b=2In the backward pass, we compute each of these partials along each edge of the graphfrom right to left, multiplying the necessary partials to result in the \ufb01nal derivativewe need. Thus we begin by annotating the \ufb01nal node with\u2202L\u2202L=1. Moving to theleft, we then compute\u2202L\u2202cand\u2202L\u2202e, and so on, until we have annotated the graph allthe way to the input variables. The forward pass conveniently already will havecomputed the values of the forward intermediate variables we need (likedande)14CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELSe=a+dd = 2bL=ce31-2e=5d=2L=-10forward passabcFigure 7.10Computation graph for the functionL(a,b,c)=c(a+2b), with values for inputnodesa=3,b=1,c=\u00002, showing the forward pass computation ofL.off(x)is the derivative ofu(x)with respect tov(x)times the derivative ofv(x)withrespect tox:dfdx=dudv\u00b7dvdx(7.23)The chain rule extends to more than two functions. If computing the derivative of acomposite functionf(x)=u(v(w(x))), the derivative off(x)is:dfdx=dudv\u00b7dvdw\u00b7dwdx(7.24)Let\u2019s now compute the 3 derivatives we need. Since in the computation graphL=ce, we can directly compute the derivative\u2202L\u2202c:\u2202L\u2202c=e(7.25)For the other two, we\u2019ll need to use the chain rule:\u2202L\u2202a=\u2202L\u2202e\u2202e\u2202a\u2202L\u2202b=\u2202L\u2202e\u2202e\u2202d\u2202d\u2202b(7.26)Eq.7.26thus requires \ufb01ve intermediate derivatives:\u2202L\u2202e,\u2202L\u2202c,\u2202e\u2202a,\u2202e\u2202d, and\u2202d\u2202b,which are as follows (making use of the fact that the derivative of a sum is the sumof the derivatives):L=ce:\u2202L\u2202e=c,\u2202L\u2202c=ee=a+d:\u2202e\u2202a=1,\u2202e\u2202d=1d=2b:\u2202d\u2202b=2In the backward pass, we compute each of these partials along each edge of the graphfrom right to left, multiplying the necessary partials to result in the \ufb01nal derivativewe need. Thus we begin by annotating the \ufb01nal node with\u2202L\u2202L=1. Moving to theleft, we then compute\u2202L\u2202cand\u2202L\u2202e, and so on, until we have annotated the graph allthe way to the input variables. The forward pass conveniently already will havecomputed the values of the forward intermediate variables we need (likedande) Example\n73\n14CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELSe=a+dd = 2bL=ce31-2e=5d=2L=-10forward passabcFigure 7.10Computation graph for the functionL(a,b,c)=c(a+2b), with values for inputnodesa=3,b=1,c=\u00002, showing the forward pass computation ofL.off(x)is the derivative ofu(x)with respect tov(x)times the derivative ofv(x)withrespect tox:dfdx=dudv\u00b7dvdx(7.23)The chain rule extends to more than two functions. If computing the derivative of acomposite functionf(x)=u(v(w(x))), the derivative off(x)is:dfdx=dudv\u00b7dvdw\u00b7dwdx(7.24)Let\u2019s now compute the 3 derivatives we need. Since in the computation graphL=ce, we can directly compute the derivative\u2202L\u2202c:\u2202L\u2202c=e(7.25)For the other two, we\u2019ll need to use the chain rule:\u2202L\u2202a=\u2202L\u2202e\u2202e\u2202a\u2202L\u2202b=\u2202L\u2202e\u2202e\u2202d\u2202d\u2202b(7.26)Eq.7.26thus requires \ufb01ve intermediate derivatives:\u2202L\u2202e,\u2202L\u2202c,\u2202e\u2202a,\u2202e\u2202d, and\u2202d\u2202b,which are as follows (making use of the fact that the derivative of a sum is the sumof the derivatives):L=ce:\u2202L\u2202e=c,\u2202L\u2202c=ee=a+d:\u2202e\u2202a=1,\u2202e\u2202d=1d=2b:\u2202d\u2202b=2In the backward pass, we compute each of these partials along each edge of the graphfrom right to left, multiplying the necessary partials to result in the \ufb01nal derivativewe need. Thus we begin by annotating the \ufb01nal node with\u2202L\u2202L=1. Moving to theleft, we then compute\u2202L\u2202cand\u2202L\u2202e, and so on, until we have annotated the graph allthe way to the input variables. The forward pass conveniently already will havecomputed the values of the forward intermediate variables we need (likedande) Examplee=d+ad = 2bL=cea=3b=1e=5d=2L=-10 abcbackward passc=-2\n7414CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELSe=a+dd = 2bL=ce31-2e=5d=2L=-10forward passabcFigure 7.10Computation graph for the functionL(a,b,c)=c(a+2b), with values for inputnodesa=3,b=1,c=\u00002, showing the forward pass computation ofL.off(x)is the derivative ofu(x)with respect tov(x)times the derivative ofv(x)withrespect tox:dfdx=dudv\u00b7dvdx(7.23)The chain rule extends to more than two functions. If computing the derivative of acomposite functionf(x)=u(v(w(x))), the derivative off(x)is:dfdx=dudv\u00b7dvdw\u00b7dwdx(7.24)Let\u2019s now compute the 3 derivatives we need. Since in the computation graphL=ce, we can directly compute the derivative\u2202L\u2202c:\u2202L\u2202c=e(7.25)For the other two, we\u2019ll need to use the chain rule:\u2202L\u2202a=\u2202L\u2202e\u2202e\u2202a\u2202L\u2202b=\u2202L\u2202e\u2202e\u2202d\u2202d\u2202b(7.26)Eq.7.26thus requires \ufb01ve intermediate derivatives:\u2202L\u2202e,\u2202L\u2202c,\u2202e\u2202a,\u2202e\u2202d, and\u2202d\u2202b,which are as follows (making use of the fact that the derivative of a sum is the sumof the derivatives):L=ce:\u2202L\u2202e=c,\u2202L\u2202c=ee=a+d:\u2202e\u2202a=1,\u2202e\u2202d=1d=2b:\u2202d\u2202b=2In the backward pass, we compute each of these partials along each edge of the graphfrom right to left, multiplying the necessary partials to result in the \ufb01nal derivativewe need. Thus we begin by annotating the \ufb01nal node with\u2202L\u2202L=1. Moving to theleft, we then compute\u2202L\u2202cand\u2202L\u2202e, and so on, until we have annotated the graph allthe way to the input variables. The forward pass conveniently already will havecomputed the values of the forward intermediate variables we need (likedande)14CHAPTER7\u2022NEURALNETWORKS ANDNEURALLANGUAGEMODELSe=a+dd = 2bL=ce31-2e=5d=2L=-10forward passabcFigure 7.10Computation graph for the functionL(a,b,c)=c(a+2b), with values for inputnodesa=3,b=1,c=\u00002, showing the forward pass computation ofL.off(x)is the derivative ofu(x)with respect tov(x)times the derivative ofv(x)withrespect tox:dfdx=dudv\u00b7dvdx(7.23)The chain rule extends to more than two functions. If computing the derivative of acomposite functionf(x)=u(v(w(x))), the derivative off(x)is:dfdx=dudv\u00b7dvdw\u00b7dwdx(7.24)Let\u2019s now compute the 3 derivatives we need. Since in the computation graphL=ce, we can directly compute the derivative\u2202L\u2202c:\u2202L\u2202c=e(7.25)For the other two, we\u2019ll need to use the chain rule:\u2202L\u2202a=\u2202L\u2202e\u2202e\u2202a\u2202L\u2202b=\u2202L\u2202e\u2202e\u2202d\u2202d\u2202b(7.26)Eq.7.26thus requires \ufb01ve intermediate derivatives:\u2202L\u2202e,\u2202L\u2202c,\u2202e\u2202a,\u2202e\u2202d, and\u2202d\u2202b,which are as follows (making use of the fact that the derivative of a sum is the sumof the derivatives):L=ce:\u2202L\u2202e=c,\u2202L\u2202c=ee=a+d:\u2202e\u2202a=1,\u2202e\u2202d=1d=2b:\u2202d\u2202b=2In the backward pass, we compute each of these partials along each edge of the graphfrom right to left, multiplying the necessary partials to result in the \ufb01nal derivativewe need. Thus we begin by annotating the \ufb01nal node with\u2202L\u2202L=1. Moving to theleft, we then compute\u2202L\u2202cand\u2202L\u2202e, and so on, until we have annotated the graph allthe way to the input variables. The forward pass conveniently already will havecomputed the values of the forward intermediate variables we need (likedande) Example\n75 Backward differentiation on a two layer network\n76\u03c3W[2]W[1]y\nx2x1Sigmoid activationReLUactivation11b[1]b[2]7.4\u2022TRAININGNEURALNETS15to compute these derivatives. Fig.7.11shows the backward pass. At each node weneed to compute the local partial derivative with respect to the parent, multiply it bythe partial derivative that is being passed down from the parent, and then pass it tothe child.e=d+ad = 2bL=cea=3b=1e=5d=2L=-10 \u2202L=1\u2202L\u2202L=-4\u2202b\u2202L=-2\u2202dabc\u2202L=-2\u2202a\n\u2202L=5\u2202c\u2202L=-2\u2202e\u2202L=-2\u2202e\u2202e=1\u2202d\u2202L=5\u2202c\u2202d=2\u2202b\u2202e=1\u2202abackward passc=-2Figure 7.11Computation graph for the functionL(a,b,c)=c(a+2b), showing the back-ward pass computation of\u2202L\u2202a,\u2202L\u2202b, and\u2202L\u2202c.Backward differentiation for a neural networkOf course computation graphs for real neural networks are much more complex.Fig.7.12shows a sample computation graph for a 2-layer neural network withn0=2,n1=2, andn2=1, assuming binary classi\ufb01cation and hence using a sigmoidoutput unit for simplicity. The function that the computation graph is computing is:z[1]=W[1]x+b[1]a[1]=ReLU(z[1])z[2]=W[2]a[1]+b[2]a[2]=s(z[2])\u02c6y=a[2](7.27)The weights that need updating (those for which we need to know the partialderivative of the loss function) are shown in orange. In order to do the backwardpass, we\u2019ll need to know the derivatives of all the functions in the graph. We alreadysaw in Section??the derivative of the sigmoids:ds(z)dz=s(z)(1\u0000s(z))(7.28)We\u2019ll also need the derivatives of each of the other activation functions. Thederivative of tanh is:dtanh(z)dz=1\u0000tanh2(z)(7.29)The derivative of the ReLU isdReLU(z)dz=\u21e20f or x<01f or x\u00000(7.30) Backward differentiation on a two layer network\n777.4\u2022TRAININGNEURALNETS15to compute these derivatives. Fig.7.11shows the backward pass. At each node weneed to compute the local partial derivative with respect to the parent, multiply it bythe partial derivative that is being passed down from the parent, and then pass it tothe child.e=d+ad = 2bL=cea=3b=1e=5d=2L=-10 \u2202L=1\u2202L\u2202L=-4\u2202b\u2202L=-2\u2202dabc\u2202L=-2\u2202a\n\u2202L=5\u2202c\u2202L=-2\u2202e\u2202L=-2\u2202e\u2202e=1\u2202d\u2202L=5\u2202c\u2202d=2\u2202b\u2202e=1\u2202abackward passc=-2Figure 7.11Computation graph for the functionL(a,b,c)=c(a+2b), showing the back-ward pass computation of\u2202L\u2202a,\u2202L\u2202b, and\u2202L\u2202c.Backward differentiation for a neural networkOf course computation graphs for real neural networks are much more complex.Fig.7.12shows a sample computation graph for a 2-layer neural network withn0=2,n1=2, andn2=1, assuming binary classi\ufb01cation and hence using a sigmoidoutput unit for simplicity. The function that the computation graph is computing is:z[1]=W[1]x+b[1]a[1]=ReLU(z[1])z[2]=W[2]a[1]+b[2]a[2]=s(z[2])\u02c6y=a[2](7.27)The weights that need updating (those for which we need to know the partialderivative of the loss function) are shown in orange. In order to do the backwardpass, we\u2019ll need to know the derivatives of all the functions in the graph. We alreadysaw in Section??the derivative of the sigmoids:ds(z)dz=s(z)(1\u0000s(z))(7.28)We\u2019ll also need the derivatives of each of the other activation functions. Thederivative of tanh is:dtanh(z)dz=1\u0000tanh2(z)(7.29)The derivative of the ReLU isdReLU(z)dz=\u21e20f or x<01f or x\u00000(7.30)\n7.4\u2022TRAININGNEURALNETS15to compute these derivatives. Fig.7.11shows the backward pass. At each node weneed to compute the local partial derivative with respect to the parent, multiply it bythe partial derivative that is being passed down from the parent, and then pass it tothe child.e=d+ad = 2bL=cea=3b=1e=5d=2L=-10 \u2202L=1\u2202L\u2202L=-4\u2202b\u2202L=-2\u2202dabc\u2202L=-2\u2202a\n\u2202L=5\u2202c\u2202L=-2\u2202e\u2202L=-2\u2202e\u2202e=1\u2202d\u2202L=5\u2202c\u2202d=2\u2202b\u2202e=1\u2202abackward passc=-2Figure 7.11Computation graph for the functionL(a,b,c)=c(a+2b), showing the back-ward pass computation of\u2202L\u2202a,\u2202L\u2202b, and\u2202L\u2202c.Backward differentiation for a neural networkOf course computation graphs for real neural networks are much more complex.Fig.7.12shows a sample computation graph for a 2-layer neural network withn0=2,n1=2, andn2=1, assuming binary classi\ufb01cation and hence using a sigmoidoutput unit for simplicity. The function that the computation graph is computing is:z[1]=W[1]x+b[1]a[1]=ReLU(z[1])z[2]=W[2]a[1]+b[2]a[2]=s(z[2])\u02c6y=a[2](7.27)The weights that need updating (those for which we need to know the partialderivative of the loss function) are shown in orange. In order to do the backwardpass, we\u2019ll need to know the derivatives of all the functions in the graph. We alreadysaw in Section??the derivative of the sigmoids:ds(z)dz=s(z)(1\u0000s(z))(7.28)We\u2019ll also need the derivatives of each of the other activation functions. Thederivative of tanh is:dtanh(z)dz=1\u0000tanh2(z)(7.29)The derivative of the ReLU isdReLU(z)dz=\u21e20f or z<01f or z\u00000(7.30) Backward differentiation on a 2-layer network \n7.4\u2022TRAININGNEURALNETS15to compute these derivatives. Fig.7.11shows the backward pass. At each node weneed to compute the local partial derivative with respect to the parent, multiply it bythe partial derivative that is being passed down from the parent, and then pass it tothe child.e=d+ad = 2bL=cea=3b=1e=5d=2L=-10 \u2202L=1\u2202L\u2202L=-4\u2202b\u2202L=-2\u2202dabc\u2202L=-2\u2202a\n\u2202L=5\u2202c\u2202L=-2\u2202e\u2202L=-2\u2202e\u2202e=1\u2202d\u2202L=5\u2202c\u2202d=2\u2202b\u2202e=1\u2202abackward passc=-2Figure 7.11Computation graph for the functionL(a,b,c)=c(a+2b), showing the back-ward pass computation of\u2202L\u2202a,\u2202L\u2202b, and\u2202L\u2202c.Backward differentiation for a neural networkOf course computation graphs for real neural networks are much more complex.Fig.7.12shows a sample computation graph for a 2-layer neural network withn0=2,n1=2, andn2=1, assuming binary classi\ufb01cation and hence using a sigmoidoutput unit for simplicity. The function that the computation graph is computing is:z[1]=W[1]x+b[1]a[1]=ReLU(z[1])z[2]=W[2]a[1]+b[2]a[2]=s(z[2])\u02c6y=a[2](7.27)The weights that need updating (those for which we need to know the partialderivative of the loss function) are shown in orange. In order to do the backwardpass, we\u2019ll need to know the derivatives of all the functions in the graph. We alreadysaw in Section??the derivative of the sigmoids:ds(z)dz=s(z)(1\u0000s(z))(7.28)We\u2019ll also need the derivatives of each of the other activation functions. Thederivative of tanh is:dtanh(z)dz=1\u0000tanh2(z)(7.29)The derivative of the ReLU isdReLU(z)dz=\u21e20f or z<01f or z\u00000(7.30)7.4\u2022TRAININGNEURALNETS15to compute these derivatives. Fig.7.11shows the backward pass. At each node weneed to compute the local partial derivative with respect to the parent, multiply it bythe partial derivative that is being passed down from the parent, and then pass it tothe child.e=d+ad = 2bL=cea=3b=1e=5d=2L=-10 \u2202L=1\u2202L\u2202L=-4\u2202b\u2202L=-2\u2202dabc\u2202L=-2\u2202a\n\u2202L=5\u2202c\u2202L=-2\u2202e\u2202L=-2\u2202e\u2202e=1\u2202d\u2202L=5\u2202c\u2202d=2\u2202b\u2202e=1\u2202abackward passc=-2Figure 7.11Computation graph for the functionL(a,b,c)=c(a+2b), showing the back-ward pass computation of\u2202L\u2202a,\u2202L\u2202b, and\u2202L\u2202c.Backward differentiation for a neural networkOf course computation graphs for real neural networks are much more complex.Fig.7.12shows a sample computation graph for a 2-layer neural network withn0=2,n1=2, andn2=1, assuming binary classi\ufb01cation and hence using a sigmoidoutput unit for simplicity. The function that the computation graph is computing is:z[1]=W[1]x+b[1]a[1]=ReLU(z[1])z[2]=W[2]a[1]+b[2]a[2]=s(z[2])\u02c6y=a[2](7.27)The weights that need updating (those for which we need to know the partialderivative of the loss function) are shown in orange. In order to do the backwardpass, we\u2019ll need to know the derivatives of all the functions in the graph. We alreadysaw in Section??the derivative of the sigmoids:ds(z)dz=s(z)(1\u0000s(z))(7.28)We\u2019ll also need the derivatives of each of the other activation functions. Thederivative of tanh is:dtanh(z)dz=1\u0000tanh2(z)(7.29)The derivative of the ReLU isdReLU(z)dz=\u21e20f or z<01f or z\u00000(7.30) Starting off the backward pass: !\"!#(I'll write \ud835\udc4efor\ud835\udc4e[\"]and \ud835\udc67for \ud835\udc67[\"])\ud835\udc3f!\ud835\udc66,\ud835\udc66=\u2212ylog(!\ud835\udc66)+1\u2212\ud835\udc66log(1\u2212!\ud835\udc66)\ud835\udc3f\ud835\udc4e,\ud835\udc66=\u2212ylog\ud835\udc4e+1\u2212\ud835\udc66log(1\u2212\ud835\udc4e)!\"!$=!\"!#!#!$\ud835\udf15\ud835\udc3f\ud835\udf15\ud835\udc4e=\u2212\ud835\udc66\ud835\udf15log\ud835\udc4e\ud835\udf15\ud835\udc4e+(1\u2212y)\ud835\udf15log1\u2212\ud835\udc4e\ud835\udf15\ud835\udc4e=\u2212\ud835\udc661\ud835\udc4e+1\u2212y11\u2212\ud835\udc4e\u22121=\u2212\ud835\udc66\ud835\udc4e+\ud835\udc66\u221211\u2212\ud835\udc4e\ud835\udf15\ud835\udc4e\ud835\udf15\ud835\udc67=\ud835\udc4e(1\u2212\ud835\udc4e)\ud835\udf15\ud835\udc3f\ud835\udf15\ud835\udc67=\u2212\ud835\udc66\ud835\udc4e+\ud835\udc66\u221211\u2212\ud835\udc4e\ud835\udc4e1\u2212\ud835\udc4e=a\u2212y7.4\u2022TRAININGNEURALNETS15to compute these derivatives. Fig.7.11shows the backward pass. At each node weneed to compute the local partial derivative with respect to the parent, multiply it bythe partial derivative that is being passed down from the parent, and then pass it tothe child.e=d+ad = 2bL=cea=3b=1e=5d=2L=-10 \u2202L=1\u2202L\u2202L=-4\u2202b\u2202L=-2\u2202dabc\u2202L=-2\u2202a\n\u2202L=5\u2202c\u2202L=-2\u2202e\u2202L=-2\u2202e\u2202e=1\u2202d\u2202L=5\u2202c\u2202d=2\u2202b\u2202e=1\u2202abackward passc=-2Figure 7.11Computation graph for the functionL(a,b,c)=c(a+2b), showing the back-ward pass computation of\u2202L\u2202a,\u2202L\u2202b, and\u2202L\u2202c.Backward differentiation for a neural networkOf course computation graphs for real neural networks are much more complex.Fig.7.12shows a sample computation graph for a 2-layer neural network withn0=2,n1=2, andn2=1, assuming binary classi\ufb01cation and hence using a sigmoidoutput unit for simplicity. The function that the computation graph is computing is:z[1]=W[1]x+b[1]a[1]=ReLU(z[1])z[2]=W[2]a[1]+b[2]a[2]=s(z[2])\u02c6y=a[2](7.27)The weights that need updating (those for which we need to know the partialderivative of the loss function) are shown in orange. In order to do the backwardpass, we\u2019ll need to know the derivatives of all the functions in the graph. We alreadysaw in Section??the derivative of the sigmoids:ds(z)dz=s(z)(1\u0000s(z))(7.28)We\u2019ll also need the derivatives of each of the other activation functions. Thederivative of tanh is:dtanh(z)dz=1\u0000tanh2(z)(7.29)The derivative of the ReLU isdReLU(z)dz=\u21e20f or x<01f or x\u00000(7.30) SummaryFor training, we need the derivative of the loss with respect to weights in early layers of the network \u2022But loss is computed only at the very end of the network! Solution: backward differentiationGiven a computation graph and the derivatives of all the functions in it we can automatically compute the derivative of the loss with respect to these early weights.\n80 Simple Neural Networks and Neural Language ModelsComputation Graphs and Backward Differentiation", "8.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\n8RNNs and LSTMs\nTime will explain.\nJane Austen, Persuasion\nLanguage is an inherently temporal phenomenon. Spoken language is a sequence of\nacoustic events over time, and we comprehend and produce both spoken and written\nlanguage as a sequential input stream. The temporal nature of language is re\ufb02ected\nin the metaphors we use; we talk of the \ufb02ow of conversations ,news feeds , and twitter\nstreams , all of which emphasize that language is a sequence that unfolds in time.\nThis temporal nature is re\ufb02ected in some language processing algorithms. For\nexample, the Viterbi algorithm we introduced for HMM part-of-speech tagging pro-\nceeds through the input a word at a time, carrying forward information gleaned along\nthe way. But other machine learning approaches, like those we\u2019ve studied for senti-\nment analysis or other text classi\ufb01cation tasks don\u2019t have this temporal nature \u2013 they\nassume simultaneous access to all aspects of their input.\nThe feedforward networks of Chapter 7 also assumed simultaneous access, al-\nthough they also had a simple model for time. Recall that we applied feedforward\nnetworks to language modeling by having them look only at a \ufb01xed-size window\nof words, and then sliding this window over the input, making independent predic-\ntions along the way. This sliding-window approach is also used in the transformer\narchitecture we will introduce in Chapter 9.\nThis chapter introduces a deep learning architecture that offers an alternative\nway of representing time: recurrent neural networks (RNNs), and their variants like\nLSTMs. RNNs have a mechanism that deals directly with the sequential nature of\nlanguage, allowing them to handle the temporal nature of language without the use of\narbitrary \ufb01xed-sized windows. The recurrent network offers a new way to represent\nthe prior context, in its recurrent connections , allowing the model\u2019s decision to\ndepend on information from hundreds of words in the past. We\u2019ll see how to apply\nthe model to the task of language modeling, to sequence modeling tasks like part-\nof-speech tagging, and to text classi\ufb01cation tasks like sentiment analysis.\n8.1 Recurrent Neural Networks\nA recurrent neural network (RNN) is any network that contains a cycle within its\nnetwork connections, meaning that the value of some unit is directly, or indirectly,\ndependent on its own earlier outputs as an input. While powerful, such networks\nare dif\ufb01cult to reason about and to train. However, within the general class of recur-\nrent networks there are constrained architectures that have proven to be extremely\neffective when applied to language. In this section, we consider a class of recurrent\nnetworks referred to as Elman Networks (Elman, 1990) or simple recurrent net-Elman\nNetworks 2CHAPTER 8 \u2022 RNN S AND LSTM S\nworks . These networks are useful in their own right and serve as the basis for more\ncomplex approaches like the Long Short-Term Memory (LSTM) networks discussed\nlater in this chapter. In this chapter when we use the term RNN we\u2019ll be referring to\nthese simpler more constrained networks (although you will often see the term RNN\nto mean any net with recurrent properties including LSTMs).\nxthtyt\nFigure 8.1 Simple recurrent neural network after Elman (1990). The hidden layer includes\na recurrent connection as part of its input. That is, the activation value of the hidden layer\ndepends on the current input as well as the activation value of the hidden layer from the\nprevious time step.\nFig. 8.1 illustrates the structure of an RNN. As with ordinary feedforward net-\nworks, an input vector representing the current input, xt, is multiplied by a weight\nmatrix and then passed through a non-linear activation function to compute the val-\nues for a layer of hidden units. This hidden layer is then used to calculate a cor-\nresponding output, yt. In a departure from our earlier window-based approach, se-\nquences are processed by presenting one item at a time to the network. We\u2019ll use\nsubscripts to represent time, thus xtwill mean the input vector xat time t. The key\ndifference from a feedforward network lies in the recurrent link shown in the \ufb01gure\nwith the dashed line. This link augments the input to the computation at the hidden\nlayer with the value of the hidden layer from the preceding point in time .\nThe hidden layer from the previous time step provides a form of memory, or\ncontext, that encodes earlier processing and informs the decisions to be made at\nlater points in time. Critically, this approach does not impose a \ufb01xed-length limit\non this prior context; the context embodied in the previous hidden layer can include\ninformation extending back to the beginning of the sequence.\nAdding this temporal dimension makes RNNs appear to be more complex than\nnon-recurrent architectures. But in reality, they\u2019re not all that different. Given an\ninput vector and the values for the hidden layer from the previous time step, we\u2019re\nstill performing the standard feedforward calculation introduced in Chapter 7. To\nsee this, consider Fig. 8.2 which clari\ufb01es the nature of the recurrence and how it\nfactors into the computation at the hidden layer. The most signi\ufb01cant change lies in\nthe new set of weights, U, that connect the hidden layer from the previous time step\nto the current hidden layer. These weights determine how the network makes use of\npast context in calculating the output for the current input. As with the other weights\nin the network, these connections are trained via backpropagation.\n8.1.1 Inference in RNNs\nForward inference (mapping a sequence of inputs to a sequence of outputs) in an\nRNN is nearly identical to what we\u2019ve already seen with feedforward networks. To\ncompute an output ytfor an input xt, we need the activation value for the hidden\nlayer ht. To calculate this, we multiply the input xtwith the weight matrix W, and\nthe hidden layer from the previous time step ht\u00001with the weight matrix U. We\nadd these values together and pass them through a suitable activation function, g,\nto arrive at the activation value for the current hidden layer, ht. Once we have the 8.1 \u2022 R ECURRENT NEURAL NETWORKS 3\n+UVWyt\nxththt-1\nFigure 8.2 Simple recurrent neural network illustrated as a feedforward network. The hid-\nden layer ht\u00001from the prior time step is multiplied by weight matrix Uand then added to\nthe feedforward component from the current time step.\nvalues for the hidden layer, we proceed with the usual computation to generate the\noutput vector.\nht=g(Uht\u00001+Wx t) (8.1)\nyt=f(Vht) (8.2)\nLet\u2019s refer to the input, hidden and output layer dimensions as din,dh, and dout\nrespectively. Given this, our three parameter matrices are: W2Rdh\u0002din,U2Rdh\u0002dh,\nandV2Rdout\u0002dh.\nWe compute ytvia a softmax computation that gives a probability distribution\nover the possible output classes.\nyt=softmax (Vht) (8.3)\nThe fact that the computation at time trequires the value of the hidden layer from\ntime t\u00001 mandates an incremental inference algorithm that proceeds from the start\nof the sequence to the end as illustrated in Fig. 8.3. The sequential nature of simple\nrecurrent networks can also be seen by unrolling the network in time as is shown in\nFig. 8.4. In this \ufb01gure, the various layers of units are copied for each time step to\nillustrate that they will have differing values over time. However, the various weight\nmatrices are shared across time.\nfunction FORWARD RNN( x,network )returns output sequence y\nh0 0\nfori 1toLENGTH (x)do\nhi g(Uhi\u00001+Wx i)\nyi f(Vhi)\nreturn y\nFigure 8.3 Forward inference in a simple recurrent network. The matrices U,VandWare\nshared across time, while new values for handyare calculated with each time step.\n8.1.2 Training\nAs with feedforward networks, we\u2019ll use a training set, a loss function, and back-\npropagation to obtain the gradients needed to adjust the weights in these recurrent 4CHAPTER 8 \u2022 RNN S AND LSTM S\nUVWUVWUVW\nx1x2x3y1y2y3\nh1h3h2\nh0\nFigure 8.4 A simple recurrent neural network shown unrolled in time. Network layers are recalculated for\neach time step, while the weights U,VandWare shared across all time steps.\nnetworks. As shown in Fig. 8.2, we now have 3 sets of weights to update: W, the\nweights from the input layer to the hidden layer, U, the weights from the previous\nhidden layer to the current hidden layer, and \ufb01nally V, the weights from the hidden\nlayer to the output layer.\nFig. 8.4 highlights two considerations that we didn\u2019t have to worry about with\nbackpropagation in feedforward networks. First, to compute the loss function for\nthe output at time twe need the hidden layer from time t\u00001. Second, the hidden\nlayer at time tin\ufb02uences both the output at time tand the hidden layer at time t+1\n(and hence the output and loss at t+1). It follows from this that to assess the error\naccruing to ht, we\u2019ll need to know its in\ufb02uence on both the current output as well as\nthe ones that follow .\nTailoring the backpropagation algorithm to this situation leads to a two-pass al-\ngorithm for training the weights in RNNs. In the \ufb01rst pass, we perform forward\ninference, computing ht,yt, accumulating the loss at each step in time, saving the\nvalue of the hidden layer at each step for use at the next time step. In the second\nphase, we process the sequence in reverse, computing the required gradients as we\ngo, computing and saving the error term for use in the hidden layer for each step\nbackward in time. This general approach is commonly referred to as backpropaga-\ntion through time (Werbos 1974, Rumelhart et al. 1986, Werbos 1990).backpropaga-\ntion through\ntimeFortunately, with modern computational frameworks and adequate computing\nresources, there is no need for a specialized approach to training RNNs. As illus-\ntrated in Fig. 8.4, explicitly unrolling a recurrent network into a feedforward com-\nputational graph eliminates any explicit recurrences, allowing the network weights\nto be trained directly. In such an approach, we provide a template that speci\ufb01es the\nbasic structure of the network, including all the necessary parameters for the input,\noutput, and hidden layers, the weight matrices, as well as the activation and output\nfunctions to be used. Then, when presented with a speci\ufb01c input sequence, we can\ngenerate an unrolled feedforward network speci\ufb01c to that input, and use that graph 8.2 \u2022 RNN S AS LANGUAGE MODELS 5\nto perform forward inference or training via ordinary backpropagation.\nFor applications that involve much longer input sequences, such as speech recog-\nnition, character-level processing, or streaming continuous inputs, unrolling an en-\ntire input sequence may not be feasible. In these cases, we can unroll the input into\nmanageable \ufb01xed-length segments and treat each segment as a distinct training item.\n8.2 RNNs as Language Models\nLet\u2019s see how to apply RNNs to the language modeling task. Recall from Chapter 3\nthat language models predict the next word in a sequence given some preceding\ncontext. For example, if the preceding context is \u201cThanks for all the\u201d and we want\nto know how likely the next word is \u201c\ufb01sh\u201d we would compute:\nP(\ufb01shjThanks for all the )\nLanguage models give us the ability to assign such a conditional probability to every\npossible next word, giving us a distribution over the entire vocabulary. We can also\nassign probabilities to entire sequences by combining these conditional probabilities\nwith the chain rule:\nP(w1:n) =nY\ni=1P(wijw<i)\nThe n-gram language models of Chapter 3 compute the probability of a word given\ncounts of its occurrence with the n\u00001 prior words. The context is thus of size n\u00001.\nFor the feedforward language models of Chapter 7, the context is the window size.\nRNN language models (Mikolov et al., 2010) process the input sequence one\nword at a time, attempting to predict the next word from the current word and the\nprevious hidden state. RNNs thus don\u2019t have the limited context problem that n-gram\nmodels have, or the \ufb01xed context that feedforward language models have, since the\nhidden state can in principle represent information about all of the preceding words\nall the way back to the beginning of the sequence. Fig. 8.5 sketches this difference\nbetween a FFN language model and an RNN language model, showing that the\nRNN language model uses ht\u00001, the hidden state from the previous time step, as a\nrepresentation of the past context.\n8.2.1 Forward Inference in an RNN language model\nForward inference in a recurrent language model proceeds exactly as described in\nSection 8.1.1. The input sequence X= [x1;:::;xt;:::;xN]consists of a series of words\neach represented as a one-hot vector of size jVj\u00021, and the output prediction, y, is a\nvector representing a probability distribution over the vocabulary. At each step, the\nmodel uses the word embedding matrix Eto retrieve the embedding for the current\nword, multiples it by the weight matrix W, and then adds it to the hidden layer from\nthe previous step (weighted by weight matrix Uto compute a new hidden layer.\nThis hidden layer is then used to generate an output layer which is passed through a\nsoftmax layer to generate a probability distribution over the entire vocabulary. That 6CHAPTER 8 \u2022 RNN S AND LSTM S\nVWethtUht-1ethtet-1et-2U       Wa)b)^yt\net-1^ytht-1WWet-2U\nFigure 8.5 Simpli\ufb01ed sketch of two LM architectures moving through a text, showing a\nschematic context of three tokens: (a) a feedforward neural language model which has a \ufb01xed\ncontext input to the weight matrix W, (b) an RNN language model, in which the hidden state\nht\u00001summarizes the prior context.\nis, at time t:\net=Ext (8.4)\nht=g(Uht\u00001+We t) (8.5)\n^ yt=softmax (Vht) (8.6)\nWhen we do language modeling with RNNs (and we\u2019ll see this again in Chapter 9\nwith transformers), it\u2019s convenient to make the assumption that the embedding di-\nmension deand the hidden dimension dhare the same. So we\u2019ll just call both of\nthese the model dimension d. So the embedding matrix Eis of shape [d\u0002jVj], and\nxtis a one-hot vector of shape [jVj\u00021]. The product etis thus of shape [d\u00021].W\nandUare of shape [d\u0002d], sohtis also of shape [d\u00021].Vis of shape [jVj\u0002d],\nso the result of Vhis a vector of shape [jVj\u00021]. This vector can be thought of as\na set of scores over the vocabulary given the evidence provided in h. Passing these\nscores through the softmax normalizes the scores into a probability distribution. The\nprobability that a particular word kin the vocabulary is the next word is represented\nby^ yt[k], the kth component of ^ yt:\nP(wt+1=kjw1;:::; wt) = ^ yt[k] (8.7)\nThe probability of an entire sequence is just the product of the probabilities of each\nitem in the sequence, where we\u2019ll use ^ yi[wi]to mean the probability of the true word\nwiat time step i.\nP(w1:n) =nY\ni=1P(wijw1:i\u00001) (8.8)\n=nY\ni=1^ yi[wi] (8.9)\n8.2.2 Training an RNN language model\nTo train an RNN as a language model, we use the same self-supervision (orself- self-supervision\ntraining ) algorithm we saw in Section ??: we take a corpus of text as training 8.2 \u2022 RNN S AS LANGUAGE MODELS 7\nInputEmbeddingsSoftmax overVocabulary\nSolongandthanksforlongandthanksforNext wordall\u2026Loss\u2026\u2026RNNhyVh\n<latexit sha1_base64=\"9tru+5ysH1zS9iUXRg/IsnxmpMA=\">AAAB/XicbVDLSsNAFL3xWesr6lKQwSK4sSQi1WXRjcsK9gFNCZPpJB06yYSZiRBCcOOvuBFxo+Av+Av+jUnbTVsPDBzOOcO993gxZ0pb1q+xsrq2vrFZ2apu7+zu7ZsHhx0lEklomwguZM/DinIW0bZmmtNeLCkOPU673viu9LtPVComokedxnQQ4iBiPiNYF5Jrnlw4XATIGWGdpbmbOSHWIxlmXERBnldds2bVrQnQMrFnpAYztFzzxxkKkoQ00oRjpfq2FetBhqVmhNO86iSKxpiMcUCzyfY5OiukIfKFLF6k0USdy+FQqTT0imS5nFr0SvE/r59o/2aQsShONI3IdJCfcKQFKqtAQyYp0TwtCCaSFRsiMsISE10UVp5uLx66TDqXdbtRbzxc1Zq3sxIqcAyncA42XEMT7qEFbSDwAm/wCV/Gs/FqvBsf0+iKMftzBHMwvv8ADJKVcA==</latexit>\u0000log \u02c6ylong<latexit sha1_base64=\"tuzkS/BeX/Xmg79qpWZlpeYDhtE=\">AAAB/HicbVDLSsNAFL3xWesr6lKEwSK4sSQi1WXRjcsK9gFNCZPJpB06mYSZiRBC3PgrbkTcKPgN/oJ/Y9J209YDA4dzznDvPV7MmdKW9WusrK6tb2xWtqrbO7t7++bBYUdFiSS0TSIeyZ6HFeVM0LZmmtNeLCkOPU673viu9LtPVCoWiUedxnQQ4qFgASNYF5Jrnlw4PBoiZ4R1luZu5oRYj2SYYeHnedU1a1bdmgAtE3tGajBDyzV/HD8iSUiFJhwr1betWA8yLDUjnOZVJ1E0xmSMhzSbLJ+js0LyURDJ4gmNJupcDodKpaFXJMvd1KJXiv95/UQHN4OMiTjRVJDpoCDhSEeobAL5TFKieVoQTCQrNkRkhCUmuuirPN1ePHSZdC7rdqPeeLiqNW9nJVTgGE7hHGy4hibcQwvaQOAF3uATvoxn49V4Nz6m0RVj9ucI5mB8/wEiupTp</latexit>\u0000log \u02c6yand<latexit sha1_base64=\"0zdsmbBovZ+hafWZN7Hvufo85tU=\">AAAB/3icbVDLSsNAFJ3UV62vqEs3g0VwY0lEqsuiG5cV7AOaEibTSTN0kgkzN0IIWbjxV9yIuFHwD/wF/8ak7aatBwYO55zh3nu8WHANlvVrVNbWNza3qtu1nd29/QPz8KirZaIo61AppOp7RDPBI9YBDoL1Y8VI6AnW8yZ3pd97YkpzGT1CGrNhSMYR9zklUEiuiS8cIcfYCQhkae5mTkggUGEGAYkmOs9rrlm3GtYUeJXYc1JHc7Rd88cZSZqELAIqiNYD24phmBEFnAqW15xEs5jQCRmzbLp/js8KaYR9qYoXAZ6qCzkSap2GXpEs19PLXin+5w0S8G+GGY/iBFhEZ4P8RGCQuCwDj7hiFERaEEIVLzbENCCKUCgqK0+3lw9dJd3Lht1sNB+u6q3beQlVdIJO0Tmy0TVqoXvURh1E0Qt6Q5/oy3g2Xo1342MWrRjzP8doAcb3H7Aall0=</latexit>\u0000log \u02c6ythanks<latexit sha1_base64=\"D3c31Jvxp3QWPr2h4tzQWmeenDs=\">AAAB/HicbVDLSsNAFL3xWesr6lKEwSK4sSQi1WXRjcsK9gFNCZPppB06yYSZiRBC3PgrbkTcKPgN/oJ/Y9Jm09YDA4dzznDvPV7EmdKW9WusrK6tb2xWtqrbO7t7++bBYUeJWBLaJoIL2fOwopyFtK2Z5rQXSYoDj9OuN7kr/O4TlYqJ8FEnER0EeBQynxGsc8k1Ty4cLkbIGWOdJpmbOgHWYxmkvpBZVnXNmlW3pkDLxC5JDUq0XPPHGQoSBzTUhGOl+rYV6UGKpWaE06zqxIpGmEzwiKbT5TN0lktDlM/LX6jRVJ3L4UCpJPDyZLGbWvQK8T+vH2v/ZpCyMIo1DclskB9zpAUqmkBDJinRPMkJJpLlGyIyxhITnfdVnG4vHrpMOpd1u1FvPFzVmrdlCRU4hlM4BxuuoQn30II2EHiBN/iEL+PZeDXejY9ZdMUo/xzBHIzvP0CJlP0=</latexit>\u0000log \u02c6yfor<latexit sha1_base64=\"PI3y1fb9LhumoVCQRh2+Y84dRkc=\">AAAB/HicbVDLSsNAFL3xWesr6lKEwSK4sSQi1WXRjcsK9gFNCZPppB06yYSZiRBC3PgrbkTcKPgN/oJ/Y9Jm09YDA4dzznDvPV7EmdKW9WusrK6tb2xWtqrbO7t7++bBYUeJWBLaJoIL2fOwopyFtK2Z5rQXSYoDj9OuN7kr/O4TlYqJ8FEnER0EeBQynxGsc8k1Ty4cLkbIGWOdJpmbOgHWYxmkmPMsq7pmzapbU6BlYpekBiVarvnjDAWJAxpqwrFSfduK9CDFUjPCaVZ1YkUjTCZ4RNPp8hk6y6Uh8oXMX6jRVJ3L4UCpJPDyZLGbWvQK8T+vH2v/ZpCyMIo1DclskB9zpAUqmkBDJinRPMkJJpLlGyIyxhITnfdVnG4vHrpMOpd1u1FvPFzVmrdlCRU4hlM4BxuuoQn30II2EHiBN/iEL+PZeDXejY9ZdMUo/xzBHIzvPyumlO8=</latexit>\u0000log \u02c6yall\ne\nFigure 8.6 Training RNNs as language models.\nmaterial and at each time step task the model to predict the next word. We call\nsuch a model self-supervised because we don\u2019t have to add any special gold labels\nto the data; the natural sequence of words is its own supervision! We simply train\nthe model to minimize the error in predicting the true next word in the training\nsequence, using cross-entropy as the loss function. Recall that the cross-entropy\nloss measures the difference between a predicted probability distribution and the\ncorrect distribution.\nLCE=\u0000X\nw2Vyt[w]log\u02c6yt[w] (8.10)\nIn the case of language modeling, the correct distribution ytcomes from knowing the\nnext word. This is represented as a one-hot vector corresponding to the vocabulary\nwhere the entry for the actual next word is 1, and all the other entries are 0. Thus,\nthe cross-entropy loss for language modeling is determined by the probability the\nmodel assigns to the correct next word. So at time tthe CE loss is the negative log\nprobability the model assigns to the next word in the training sequence.\nLCE(\u02c6yt;yt) =\u0000log\u02c6yt[wt+1] (8.11)\nThus at each word position tof the input, the model takes as input the the correct\nword wttogether with ht\u00001, encoding information from the preceding w1:t\u00001, and\nuses them to compute a probability distribution over possible next words so as to\ncompute the model\u2019s loss for the next token wt+1. Then we move to the next word,\nwe ignore what the model predicted for the next word and instead use the correct\nword wt+1along with the prior history encoded to estimate the probability of token\nwt+2. This idea that we always give the model the correct history sequence to predict\nthe next word (rather than feeding the model its best case from the previous time\nstep) is called teacher forcing . teacher forcing\nThe weights in the network are adjusted to minimize the average CE loss over\nthe training sequence via gradient descent. Fig. 8.6 illustrates this training regimen.\n8.2.3 Weight Tying\nCareful readers may have noticed that the input embedding matrix Eand the \ufb01nal\nlayer matrix V, which feeds the output softmax, are quite similar. 8CHAPTER 8 \u2022 RNN S AND LSTM S\nThe columns of Erepresent the word embeddings for each word in the vocab-\nulary learned during the training process with the goal that words that have similar\nmeaning and function will have similar embeddings. And, since when we use RNNs\nfor language modeling we make the assumption that the embedding dimension and\nthe hidden dimension are the same (= the model dimension d), the embedding ma-\ntrixEhas shape [d\u0002jVj]. And the \ufb01nal layer matrix Vprovides a way to score\nthe likelihood of each word in the vocabulary given the evidence present in the \ufb01nal\nhidden layer of the network through the calculation of Vh.Vis of shape [jVj\u0002d].\nThat is, is, the rows of Vare shaped like a transpose of E, meaning that Vprovides\nasecond set of learned word embeddings.\nInstead of having two sets of embedding matrices, language models use a single\nembedding matrix, which appears at both the input and softmax layers. That is,\nwe dispense with Vand use Eat the start of the computation and E|(because the\nshape of Vis the transpose of Eat the end. Using the same matrix (transposed) in\ntwo places is called weight tying .1The weight-tied equations for an RNN language weight tying\nmodel then become:\net=Ext (8.12)\nht=g(Uht\u00001+We t) (8.13)\n^ yt=softmax (E|ht) (8.14)\nIn addition to providing improved model perplexity, this approach signi\ufb01cantly re-\nduces the number of parameters required for the model.\n8.3 RNNs for other NLP tasks\nNow that we\u2019ve seen the basic RNN architecture, let\u2019s consider how to apply it to\nthree types of NLP tasks: sequence classi\ufb01cation tasks like sentiment analysis and\ntopic classi\ufb01cation, sequence labeling tasks like part-of-speech tagging, and text\ngeneration tasks, including with a new architecture called the encoder-decoder .\n8.3.1 Sequence Labeling\nIn sequence labeling, the network\u2019s task is to assign a label chosen from a small\n\ufb01xed set of labels to each element of a sequence, like the part-of-speech tagging and\nnamed entity recognition tasks from Chapter 17. In an RNN approach to sequence\nlabeling, inputs are word embeddings and the outputs are tag probabilities generated\nby a softmax layer over the given tagset, as illustrated in Fig. 8.7.\nIn this \ufb01gure, the inputs at each time step are pretrained word embeddings cor-\nresponding to the input tokens. The RNN block is an abstraction that represents\nan unrolled simple recurrent network consisting of an input layer, hidden layer, and\noutput layer at each time step, as well as the shared U,VandWweight matrices\nthat comprise the network. The outputs of the network at each time step represent\nthe distribution over the POS tagset generated by a softmax layer.\nTo generate a sequence of tags for a given input, we run forward inference over\nthe input sequence and select the most likely tag from the softmax at each step. Since\nwe\u2019re using a softmax layer to generate the probability distribution over the output\ntagset at each time step, we will again employ the cross-entropy loss during training.\n1We also do this for transformers (Chapter 9) where it\u2019s common to call E|theunembedding matrix . 8.3 \u2022 RNN S FOR OTHER NLP TASKS 9\nJanetwillbackthebillNNDTVBMDNNPArgmax\nEmbeddingsWordsehVhyRNNLayer(s)Softmax overtags\nFigure 8.7 Part-of-speech tagging as sequence labeling with a simple RNN. Pre-trained\nword embeddings serve as inputs and a softmax layer provides a probability distribution over\nthe part-of-speech tags as output at each time step.\n8.3.2 RNNs for Sequence Classi\ufb01cation\nAnother use of RNNs is to classify entire sequences rather than the tokens within\nthem. This is the set of tasks commonly called text classi\ufb01cation , like sentiment\nanalysis or spam detection, in which we classify a text into two or three classes\n(like positive or negative), as well as classi\ufb01cation tasks with a large number of\ncategories, like document-level topic classi\ufb01cation, or message routing for customer\nservice applications.\nTo apply RNNs in this setting, we pass the text to be classi\ufb01ed through the RNN\na word at a time generating a new hidden layer representation at each time step.\nWe can then take the hidden layer for the last token of the text, hn, to constitute a\ncompressed representation of the entire sequence. We can pass this representation\nhnto a feedforward network that chooses a class via a softmax over the possible\nclasses. Fig. 8.8 illustrates this approach.\nx1\nRNNhnx2x3xnSoftmaxFFN\nFigure 8.8 Sequence classi\ufb01cation using a simple RNN combined with a feedforward net-\nwork. The \ufb01nal hidden state from the RNN is used as the input to a feedforward network that\nperforms the classi\ufb01cation.\nNote that in this approach we don\u2019t need intermediate outputs for the words in\nthe sequence preceding the last element. Therefore, there are no loss terms associ- 10 CHAPTER 8 \u2022 RNN S AND LSTM S\nated with those elements. Instead, the loss function used to train the weights in the\nnetwork is based entirely on the \ufb01nal text classi\ufb01cation task. The output from the\nsoftmax output from the feedforward classi\ufb01er together with a cross-entropy loss\ndrives the training. The error signal from the classi\ufb01cation is backpropagated all the\nway through the weights in the feedforward classi\ufb01er through, to its input, and then\nthrough to the three sets of weights in the RNN as described earlier in Section 8.1.2.\nThe training regimen that uses the loss from a downstream application to adjust the\nweights all the way through the network is referred to as end-to-end training .end-to-end\ntraining\nAnother option, instead of using just hidden state of the last token hnto represent\nthe whole sequence, is to use some sort of pooling function of all the hidden states pooling\nhifor each word iin the sequence. For example, we can create a representation that\npools all the nhidden states by taking their element-wise mean:\nhmean=1\nnnX\ni=1hi (8.15)\nOr we can take the element-wise max; the element-wise max of a set of nvectors is\na new vector whose kth element is the max of the kth elements of all the nvectors.\nThe long contexts of RNNs makes it quite dif\ufb01cult to successfully backpropagate\nerror all the way through the entire input; we\u2019ll talk about this problem, and some\nstandard solutions, in Section 8.5.\n8.3.3 Generation with RNN-Based Language Models\nRNN-based language models can also be used to generate text. Text generation is\nof enormous practical importance, part of tasks like question answering, machine\ntranslation, text summarization, grammar correction, story generation, and conver-\nsational dialogue; any task where a system needs to produce text, conditioned on\nsome other text. This use of a language model to generate text is one of the areas\nin which the impact of neural language models on NLP has been the largest. Text\ngeneration, along with image generation and code generation, constitute a new area\nof AI that is often called generative AI .\nRecall back in Chapter 3 we saw how to generate text from an n-gram language\nmodel by adapting a sampling technique suggested at about the same time by Claude\nShannon (Shannon, 1951) and the psychologists George Miller and Jennifer Self-\nridge (Miller and Selfridge, 1950). We \ufb01rst randomly sample a word to begin a\nsequence based on its suitability as the start of a sequence. We then continue to\nsample words conditioned on our previous choices until we reach a pre-determined\nlength, or an end of sequence token is generated.\nToday, this approach of using a language model to incrementally generate words\nby repeatedly sampling the next word conditioned on our previous choices is called\nautoregressive generation orcausal LM generation . The procedure is basicallyautoregressive\ngeneration\nthe same as that described on page ??, but adapted to a neural context:\n\u2022 Sample a word in the output from the softmax distribution that results from\nusing the beginning of sentence marker, <s>, as the \ufb01rst input.\n\u2022 Use the word embedding for that \ufb01rst word as the input to the network at the\nnext time step, and then sample the next word in the same fashion.\n\u2022 Continue generating until the end of sentence marker, </s> , is sampled or a\n\ufb01xed length limit is reached.\nTechnically an autoregressive model is a model that predicts a value at time tbased\non a linear function of the previous values at times t\u00001,t\u00002, and so on. Although 8.4 \u2022 S TACKED AND BIDIRECTIONAL RNN ARCHITECTURES 11\nlanguage models are not linear (since they have many layers of non-linearities), we\nloosely refer to this generation technique as autoregressive generation since the\nword generated at each time step is conditioned on the word selected by the network\nfrom the previous step. Fig. 8.9 illustrates this approach. In this \ufb01gure, the details of\nthe RNN\u2019s hidden layers and recurrent connections are hidden within the blue block.\nThis simple architecture underlies state-of-the-art approaches to applications\nsuch as machine translation, summarization, and question answering. The key to\nthese approaches is to prime the generation component with an appropriate context.\nThat is, instead of simply using <s> to get things started we can provide a richer\ntask-appropriate context; for translation the context is the sentence in the source\nlanguage; for summarization it\u2019s the long text we want to summarize.\nSolong\n<s>and\nSolongand?Sampled WordSoftmaxEmbeddingInput WordRNN\nFigure 8.9 Autoregressive generation with an RNN-based neural language model.\n8.4 Stacked and Bidirectional RNN architectures\nRecurrent networks are quite \ufb02exible. By combining the feedforward nature of un-\nrolled computational graphs with vectors as common inputs and outputs, complex\nnetworks can be treated as modules that can be combined in creative ways. This\nsection introduces two of the more common network architectures used in language\nprocessing with RNNs.\n8.4.1 Stacked RNNs\nIn our examples thus far, the inputs to our RNNs have consisted of sequences of\nword or character embeddings (vectors) and the outputs have been vectors useful for\npredicting words, tags or sequence labels. However, nothing prevents us from using\nthe entire sequence of outputs from one RNN as an input sequence to another one.\nStacked RNNs consist of multiple networks where the output of one layer serves as Stacked RNNs\nthe input to a subsequent layer, as shown in Fig. 8.10.\nStacked RNNs generally outperform single-layer networks. One reason for this\nsuccess seems to be that the network induces representations at differing levels of\nabstraction across layers. Just as the early stages of the human visual system detect 12 CHAPTER 8 \u2022 RNN S AND LSTM S\ny1y2y3yn\nx1x2x3xn\nRNN 1\nRNN 2\n RNN 3\nFigure 8.10 Stacked recurrent networks. The output of a lower level serves as the input to\nhigher levels with the output of the last network serving as the \ufb01nal output.\nedges that are then used for \ufb01nding larger regions and shapes, the initial layers of\nstacked networks can induce representations that serve as useful abstractions for\nfurther layers\u2014representations that might prove dif\ufb01cult to induce in a single RNN.\nThe optimal number of stacked RNNs is speci\ufb01c to each application and to each\ntraining set. However, as the number of stacks is increased the training costs rise\nquickly.\n8.4.2 Bidirectional RNNs\nThe RNN uses information from the left (prior) context to make its predictions at\ntime t. But in many applications we have access to the entire input sequence; in\nthose cases we would like to use words from the context to the right of t. One way\nto do this is to run two separate RNNs, one left-to-right, and one right-to-left, and\nconcatenate their representations.\nIn the left-to-right RNNs we\u2019ve discussed so far, the hidden state at a given time\ntrepresents everything the network knows about the sequence up to that point. The\nstate is a function of the inputs x1;:::;xtand represents the context of the network to\nthe left of the current time.\nhf\nt=RNN forward (x1;:::;xt) (8.16)\nThis new notation hf\ntsimply corresponds to the normal hidden state at time t, repre-\nsenting everything the network has gleaned from the sequence so far.\nTo take advantage of context to the right of the current input, we can train an\nRNN on a reversed input sequence. With this approach, the hidden state at time t\nrepresents information about the sequence to the right of the current input:\nhb\nt=RNN backward (xt;:::xn) (8.17)\nHere, the hidden state hb\ntrepresents all the information we have discerned about the\nsequence from tto the end of the sequence.\nAbidirectional RNN (Schuster and Paliwal, 1997) combines two independentbidirectional\nRNN\nRNNs, one where the input is processed from the start to the end, and the other from\nthe end to the start. We then concatenate the two representations computed by the\nnetworks into a single vector that captures both the left and right contexts of an input 8.5 \u2022 T HELSTM 13\nat each point in time. Here we use either the semicolon \u201d;\u201d or the equivalent symbol\n\bto mean vector concatenation:\nht= [hf\nt;hb\nt]\n=hf\nt\bhb\nt (8.18)\nFig. 8.11 illustrates such a bidirectional network that concatenates the outputs of\nthe forward and backward pass. Other simple ways to combine the forward and\nbackward contexts include element-wise addition or multiplication. The output at\neach step in time thus captures information to the left and to the right of the current\ninput. In sequence labeling applications, these concatenated outputs can serve as the\nbasis for a local labeling decision.\nRNN 2 \nRNN 1x1y2y1y3ynconcatenatedoutputs\nx2x3xn\nFigure 8.11 A bidirectional RNN. Separate models are trained in the forward and backward\ndirections, with the output of each model at each time point concatenated to represent the\nbidirectional state at that time point.\nBidirectional RNNs have also proven to be quite effective for sequence classi\ufb01-\ncation. Recall from Fig. 8.8 that for sequence classi\ufb01cation we used the \ufb01nal hidden\nstate of the RNN as the input to a subsequent feedforward classi\ufb01er. A dif\ufb01culty\nwith this approach is that the \ufb01nal state naturally re\ufb02ects more information about\nthe end of the sentence than its beginning. Bidirectional RNNs provide a simple\nsolution to this problem; as shown in Fig. 8.12, we simply combine the \ufb01nal hidden\nstates from the forward and backward passes (for example by concatenation) and\nuse that as input for follow-on processing.\n8.5 The LSTM\nIn practice, it is quite dif\ufb01cult to train RNNs for tasks that require a network to make\nuse of information distant from the current point of processing. Despite having ac-\ncess to the entire preceding sequence, the information encoded in hidden states tends\nto be fairly local, more relevant to the most recent parts of the input sequence and\nrecent decisions. Yet distant information is critical to many language applications.\nConsider the following example in the context of language modeling. 14 CHAPTER 8 \u2022 RNN S AND LSTM S\nRNN 2 \nRNN 1x1x2x3xnhn\u2192h1\u2190hn\u2192SoftmaxFFNh1\u2190\nFigure 8.12 A bidirectional RNN for sequence classi\ufb01cation. The \ufb01nal hidden units from\nthe forward and backward passes are combined to represent the entire sequence. This com-\nbined representation serves as input to the subsequent classi\ufb01er.\n(8.19) The \ufb02ights the airline was canceling were full.\nAssigning a high probability to wasfollowing airline is straightforward since airline\nprovides a strong local context for the singular agreement. However, assigning an\nappropriate probability to were is quite dif\ufb01cult, not only because the plural \ufb02ights\nis quite distant, but also because the singular noun airline is closer in the intervening\ncontext. Ideally, a network should be able to retain the distant information about\nplural \ufb02ights until it is needed, while still processing the intermediate parts of the\nsequence correctly.\nOne reason for the inability of RNNs to carry forward critical information is that\nthe hidden layers, and, by extension, the weights that determine the values in the hid-\nden layer, are being asked to perform two tasks simultaneously: provide information\nuseful for the current decision, and updating and carrying forward information re-\nquired for future decisions.\nA second dif\ufb01culty with training RNNs arises from the need to backpropagate\nthe error signal back through time. Recall from Section 8.1.2 that the hidden layer at\ntime tcontributes to the loss at the next time step since it takes part in that calcula-\ntion. As a result, during the backward pass of training, the hidden layers are subject\nto repeated multiplications, as determined by the length of the sequence. A frequent\nresult of this process is that the gradients are eventually driven to zero, a situation\ncalled the vanishing gradients problem.vanishing\ngradients\nTo address these issues, more complex network architectures have been designed\nto explicitly manage the task of maintaining relevant context over time, by enabling\nthe network to learn to forget information that is no longer needed and to remember\ninformation required for decisions still to come.\nThe most commonly used such extension to RNNs is the long short-term mem-\nory(LSTM) network (Hochreiter and Schmidhuber, 1997). LSTMs divide the con-long short-term\nmemory\ntext management problem into two subproblems: removing information no longer\nneeded from the context, and adding information likely to be needed for later de- 8.5 \u2022 T HELSTM 15\ncision making. The key to solving both problems is to learn how to manage this\ncontext rather than hard-coding a strategy into the architecture. LSTMs accomplish\nthis by \ufb01rst adding an explicit context layer to the architecture (in addition to the\nusual recurrent hidden layer), and through the use of specialized neural units that\nmake use of gates to control the \ufb02ow of information into and out of the units that\ncomprise the network layers. These gates are implemented through the use of addi-\ntional weights that operate sequentially on the input, and previous hidden layer, and\nprevious context layers.\nThe gates in an LSTM share a common design pattern; each consists of a feed-\nforward layer, followed by a sigmoid activation function, followed by a pointwise\nmultiplication with the layer being gated. The choice of the sigmoid as the activation\nfunction arises from its tendency to push its outputs to either 0 or 1. Combining this\nwith a pointwise multiplication has an effect similar to that of a binary mask. Values\nin the layer being gated that align with values near 1 in the mask are passed through\nnearly unchanged; values corresponding to lower values are essentially erased.\nThe \ufb01rst gate we\u2019ll consider is the forget gate . The purpose of this gate is forget gate\nto delete information from the context that is no longer needed. The forget gate\ncomputes a weighted sum of the previous state\u2019s hidden layer and the current in-\nput and passes that through a sigmoid. This mask is then multiplied element-wise\nby the context vector to remove the information from context that is no longer re-\nquired. Element-wise multiplication of two vectors (represented by the operator \f,\nand sometimes called the Hadamard product ) is the vector of the same dimension\nas the two input vectors, where each element iis the product of element iin the two\ninput vectors:\nft=s(Ufht\u00001+Wfxt) (8.20)\nkt=ct\u00001\fft (8.21)\nThe next task is to compute the actual information we need to extract from the previ-\nous hidden state and current inputs\u2014the same basic computation we\u2019ve been using\nfor all our recurrent networks.\ngt=tanh(Ught\u00001+Wgxt) (8.22)\nNext, we generate the mask for the add gate to select the information to add to the add gate\ncurrent context.\nit=s(Uiht\u00001+Wixt) (8.23)\njt=gt\fit (8.24)\nNext, we add this to the modi\ufb01ed context vector to get our new context vector.\nct=jt+kt (8.25)\nThe \ufb01nal gate we\u2019ll use is the output gate which is used to decide what informa- output gate\ntion is required for the current hidden state (as opposed to what information needs\nto be preserved for future decisions).\not=s(Uoht\u00001+Woxt) (8.26)\nht=ot\ftanh(ct) (8.27)\nFig. 8.13 illustrates the complete computation for a single LSTM unit. Given the 16 CHAPTER 8 \u2022 RNN S AND LSTM S\n+\nxtht-1cthtcthtct-1ht-1xttanh\n+\u03c3tanh\u03c3\u03c3+++igf\no\u3f4b\u3f4b\u3f4bLSTMct-1\nFigure 8.13 A single LSTM unit displayed as a computation graph. The inputs to each unit consists of the\ncurrent input, x, the previous hidden state, ht\u00001, and the previous context, ct\u00001. The outputs are a new hidden\nstate, htand an updated context, ct.\nappropriate weights for the various gates, an LSTM accepts as input the context\nlayer, and hidden layer from the previous time step, along with the current input\nvector. It then generates updated context and hidden vectors as output.\nIt is the hidden state, ht, that provides the output for the LSTM at each time step.\nThis output can be used as the input to subsequent layers in a stacked RNN, or at the\n\ufb01nal layer of a network htcan be used to provide the \ufb01nal output of the LSTM.\n8.5.1 Gated Units, Layers and Networks\nThe neural units used in LSTMs are obviously much more complex than those used\nin basic feedforward networks. Fortunately, this complexity is encapsulated within\nthe basic processing units, allowing us to maintain modularity and to easily exper-\niment with different architectures. To see this, consider Fig. 8.14 which illustrates\nthe inputs and outputs associated with each kind of unit.\nAt the far left, (a) is the basic feedforward unit where a single set of weights and\na single activation function determine its output, and when arranged in a layer there\nare no connections among the units in the layer. Next, (b) represents the unit in a\nsimple recurrent network. Now there are two inputs and an additional set of weights\nto go with it. However, there is still a single activation function and output.\nThe increased complexity of the LSTM units is encapsulated within the unit\nitself. The only additional external complexity for the LSTM over the basic recurrent\nunit (b) is the presence of the additional context vector as an input and output.\nThis modularity is key to the power and widespread applicability of LSTM units.\nLSTM units (or other varieties, like GRUs) can be substituted into any of the network\narchitectures described in Section 8.4. And, as with simple RNNs, multi-layered\nnetworks making use of gated units can be unrolled into deep feedforward networks\nand trained in the usual fashion with backpropagation. In practice, therefore, LSTMs\nrather than RNNs have become the standard unit for any modern system that makes\nuse of recurrent networks. 8.6 \u2022 S UMMARY : COMMON RNN NLP A RCHITECTURES 17\nh\nxxtxtht-1htht\nct-1ct\nht-1(b)(a)(c)\u2303gza\u2303gzLSTMUnita\nFigure 8.14 Basic neural units used in feedforward, simple recurrent networks (SRN), and\nlong short-term memory (LSTM).\n8.6 Summary: Common RNN NLP Architectures\nWe\u2019ve now introduced the RNN, seen advanced components like stacking multiple\nlayers and using the LSTM version, and seen how the RNN can be applied to various\ntasks. Let\u2019s take a moment to summarize the architectures for these applications.\nFig. 8.15 shows the three architectures we\u2019ve discussed so far: sequence la-\nbeling, sequence classi\ufb01cation, and language modeling. In sequence labeling (for\nexample for part of speech tagging), we train a model to produce a label for each\ninput word or token. In sequence classi\ufb01cation, for example for sentiment analysis,\nwe ignore the output for each token, and only take the value from the end of the\nsequence (and similarly the model\u2019s training signal comes from backpropagation\nfrom that last token). In language modeling, we train the model to predict the next\nword at each token step. In the next section we\u2019ll introduce a fourth architecture, the\nencoder-decoder .\n8.7 The Encoder-Decoder Model with RNNs\nIn this section we introduce a new model, the encoder-decoder model, which is used\nwhen we are taking an input sequence and translating it to an output sequence that is\nof a different length than the input, and doesn\u2019t align with it in a word-to-word way.\nRecall that in the sequence labeling task, we have two sequences, but they are the\nsame length (for example in part-of-speech tagging each token gets an associated\ntag), each input is associated with a speci\ufb01c output, and the labeling for that output\ntakes mostly local information. Thus deciding whether a word is a verb or a noun,\nwe look mostly at the word and the neighboring words.\nBy contrast, encoder-decoder models are used especially for tasks like machine\ntranslation, where the input sequence and output sequence can have different lengths\nand the mapping between a token in the input and a token in the output can be very\nindirect (in some languages the verb appears at the beginning of the sentence; in\nother languages at the end). We\u2019ll introduce machine translation in detail in Chap-\nter 13, but for now we\u2019ll just point out that the mapping for a sentence in English to\na sentence in Tagalog or Yoruba can have very different numbers of words, and the 18 CHAPTER 8 \u2022 RNN S AND LSTM S\n\u2026Encoder RNNDecoder RNNContext\u2026\nx1x2xny1y2ym\u2026RNNx1x2xn\u2026y1y2yn\u2026RNNx1x2xny\n\u2026RNNx1x2xt-1\u2026x2x3xta) sequence labeling b) sequence classification \nc) language modelingd) encoder-decoder\nFigure 8.15 Four architectures for NLP tasks. In sequence labeling (POS or named entity tagging) we map\neach input token xito an output token yi. In sequence classi\ufb01cation we map the entire input sequence to a single\nclass. In language modeling we output the next token conditioned on previous tokens. In the encoder model we\nhave two separate RNN models, one of which maps from an input sequence xto an intermediate representation\nwe call the context , and a second of which maps from the context to an output sequence y.\nwords can be in a very different order.\nEncoder-decoder networks, sometimes called sequence-to-sequence networks,encoder-\ndecoder\nare models capable of generating contextually appropriate, arbitrary length, output\nsequences given an input sequence. Encoder-decoder networks have been applied\nto a very wide range of applications including summarization, question answering,\nand dialogue, but they are particularly popular for machine translation.\nThe key idea underlying these networks is the use of an encoder network that\ntakes an input sequence and creates a contextualized representation of it, often called\nthecontext . This representation is then passed to a decoder which generates a task-\nspeci\ufb01c output sequence. Fig. 8.16 illustrates the architecture.\n\u2026EncoderDecoderContext\u2026\nx1x2xny1y2ym\nFigure 8.16 The encoder-decoder architecture. The context is a function of the hidden\nrepresentations of the input, and may be used by the decoder in a variety of ways.\nEncoder-decoder networks consist of three conceptual components:\n1. An encoder that accepts an input sequence, x1:n, and generates a correspond- 8.7 \u2022 T HEENCODER -DECODER MODEL WITH RNN S19\ning sequence of contextualized representations, h1:n. LSTMs, convolutional\nnetworks, and transformers can all be employed as encoders.\n2. A context vector ,c, which is a function of h1:n, and conveys the essence of\nthe input to the decoder.\n3. A decoder , which accepts cas input and generates an arbitrary length se-\nquence of hidden states h1:m, from which a corresponding sequence of output\nstates y1:m, can be obtained. Just as with encoders, decoders can be realized\nby any kind of sequence architecture.\nIn this section we\u2019ll describe an encoder-decoder network based on a pair of\nRNNs, but we\u2019ll see in Chapter 13 how to apply them to transformers as well. We\u2019ll\nbuild up the equations for encoder-decoder models by starting with the conditional\nRNN language model p(y), the probability of a sequence y.\nRecall that in any language model, we can break down the probability as follows:\np(y) = p(y1)p(y2jy1)p(y3jy1;y2):::p(ymjy1;:::;ym\u00001) (8.28)\nIn RNN language modeling, at a particular time t, we pass the pre\ufb01x of t\u00001\ntokens through the language model, using forward inference to produce a sequence\nof hidden states, ending with the hidden state corresponding to the last word of\nthe pre\ufb01x. We then use the \ufb01nal hidden state of the pre\ufb01x as our starting point to\ngenerate the next token.\nMore formally, if gis an activation function like tanh or ReLU, a function of\nthe input at time tand the hidden state at time t\u00001, and the softmax is over the\nset of possible vocabulary items, then at time tthe output ytand hidden state htare\ncomputed as:\nht=g(ht\u00001;xt) (8.29)\n^ yt=softmax (ht) (8.30)\nWe only have to make one slight change to turn this language model with au-\ntoregressive generation into an encoder-decoder model that is a translation model\nthat can translate from a source text in one language to a target text in a second:\nadd a sentence separation marker at the end of the source text, and then simplysentence\nseparation\nconcatenate the target text.\nLet\u2019s use <s>for our sentence separator token, and let\u2019s think about translating\nan English source text (\u201cthe green witch arrived\u201d), to a Spanish sentence (\u201c lleg\u00b4o\nla bruja verde \u201d (which can be glossed word-by-word as \u2018arrived the witch green\u2019).\nWe could also illustrate encoder-decoder models with a question-answer pair, or a\ntext-summarization pair.\nLet\u2019s use xto refer to the source text (in this case in English) plus the separator\ntoken <s>, and yto refer to the target text y(in this case in Spanish). Then an\nencoder-decoder model computes the probability p(yjx)as follows:\np(yjx) = p(y1jx)p(y2jy1;x)p(y3jy1;y2;x):::p(ymjy1;:::;ym\u00001;x) (8.31)\nFig. 8.17 shows the setup for a simpli\ufb01ed version of the encoder-decoder model\n(we\u2019ll see the full model, which requires the new concept of attention , in the next\nsection).\nFig. 8.17 shows an English source text (\u201cthe green witch arrived\u201d), a sentence\nseparator token ( <s>, and a Spanish target text (\u201c lleg\u00b4o la bruja verde \u201d). To trans-\nlate a source text, we run it through the network performing forward inference to 20 CHAPTER 8 \u2022 RNN S AND LSTM S\nSource TextTarget Text\nhnembeddinglayerhiddenlayer(s)softmaxthegreenlleg\u00f3\nwitcharrived<s>lleg\u00f3la\nlabruja\nbrujaverde\nverde</s>(output of source is ignored)\nSeparator\nFigure 8.17 Translating a single sentence (inference time) in the basic RNN version of encoder-decoder ap-\nproach to machine translation. Source and target sentences are concatenated with a separator token in between,\nand the decoder uses context information from the encoder\u2019s last hidden state.\ngenerate hidden states until we get to the end of the source. Then we begin autore-\ngressive generation, asking for a word in the context of the hidden layer from the\nend of the source input as well as the end-of-sentence marker. Subsequent words\nare conditioned on the previous hidden state and the embedding for the last word\ngenerated.\nLet\u2019s formalize and generalize this model a bit in Fig. 8.18. (To help keep things\nstraight, we\u2019ll use the superscripts eanddwhere needed to distinguish the hidden\nstates of the encoder and the decoder.) The elements of the network on the left\nprocess the input sequence xand comprise the encoder . While our simpli\ufb01ed \ufb01gure\nshows only a single network layer for the encoder, stacked architectures are the\nnorm, where the output states from the top layer of the stack are taken as the \ufb01nal\nrepresentation, and the encoder consists of stacked biLSTMs where the hidden states\nfrom top layers from the forward and backward passes are concatenated to provide\nthe contextualized representations for each time step.\nEncoderDecoder\nhn hd1he3he2he1hd2hd3hd4embeddinglayerhiddenlayer(s)softmaxx1x2y1hdmx3xn<s>y1y2\ny2y3\ny3y4\nym</s>hen = c = hd0(output is ignored during encoding)\nFigure 8.18 A more formal version of translating a sentence at inference time in the basic RNN-based\nencoder-decoder architecture. The \ufb01nal hidden state of the encoder RNN, hen, serves as the context for the\ndecoder in its role as hd\n0in the decoder RNN, and is also made available to each decoder hidden state. 8.7 \u2022 T HEENCODER -DECODER MODEL WITH RNN S21\nThe entire purpose of the encoder is to generate a contextualized representation\nof the input. This representation is embodied in the \ufb01nal hidden state of the encoder,\nhe\nn. This representation, also called cforcontext , is then passed to the decoder.\nThe simplest version of the decoder network would takes this state and use it\njust to initialize the \ufb01rst hidden state of the decoder; the \ufb01rst decoder RNN cell\nwould use cas its prior hidden state hd\n0. The decoder would then autoregressively\ngenerates a sequence of outputs, an element at a time, until an end-of-sequence\nmarker is generated. Each hidden state is conditioned on the previous hidden state\nand the output generated in the previous state.\nAs Fig. 8.18 shows, we do something more complex: we make the context vector\ncavailable to more than just the \ufb01rst decoder hidden state, to ensure that the in\ufb02uence\nof the context vector, c, doesn\u2019t wane as the output sequence is generated. We do\nthis by adding cas a parameter to the computation of the current hidden state. using\nthe following equation:\nhd\nt=g(\u02c6yt\u00001;hd\nt\u00001;c) (8.32)\nNow we\u2019re ready to see the full equations for this version of the decoder in the basic\nencoder-decoder model, with context available at each decoding timestep. Recall\nthatgis a stand-in for some \ufb02avor of RNN and \u02c6 yt\u00001is the embedding for the output\nsampled from the softmax at the previous step:\nc=he\nn\nhd\n0=c\nhd\nt=g(\u02c6yt\u00001;hd\nt\u00001;c)\n^ yt=softmax (hd\nt) (8.33)\nThus ^ ytis a vector of probabilities over the vocabulary, representing the probability\nof each word occurring at time t. To generate text, we sample from this distribution\n^ yt. For example, the greedy choice is simply to choose the most most probable word\nto generate at each timestep. We\u2019ll introduce more sophisticated sampling methods\nin Section ??.\n8.7.1 Training the Encoder-Decoder Model\nEncoder-decoder architectures are trained end-to-end. Each training example is a\ntuple of paired strings, a source and a target. Concatenated with a separator token,\nthese source-target pairs can now serve as training data.\nFor MT, the training data typically consists of sets of sentences and their transla-\ntions. These can be drawn from standard datasets of aligned sentence pairs, as we\u2019ll\ndiscuss in Section ??. Once we have a training set, the training itself proceeds as\nwith any RNN-based language model. The network is given the source text and then\nstarting with the separator token is trained autoregressively to predict the next word,\nas shown in Fig. 8.19.\nNote the differences between training (Fig. 8.19) and inference (Fig. 8.17) with\nrespect to the outputs at each time step. The decoder during inference uses its own\nestimated output \u02c6 ytas the input for the next time step xt+1. Thus the decoder will\ntend to deviate more and more from the gold target sentence as it keeps generating\nmore tokens. In training, therefore, it is more common to use teacher forcing in the teacher forcing\ndecoder. Teacher forcing means that we force the system to use the gold target token\nfrom training as the next input xt+1, rather than allowing it to rely on the (possibly\nerroneous) decoder output \u02c6 yt. This speeds up training. 22 CHAPTER 8 \u2022 RNN S AND LSTM S\nEncoderDecoder\nembeddinglayerhiddenlayer(s)softmaxthegreenlleg\u00f3\nwitcharrived<s>lleg\u00f3la\nlabruja\nbrujaverde\nverde</s>goldanswersL1 =-log P(y1)\nx1x2x3x4L2 =-log P(y2)L3 =-log P(y3)L4 =-log P(y4)L5 =-log P(y5)per-wordlossy1y2y3y4y5Total loss is the average cross-entropy loss per target word:\nFigure 8.19 Training the basic RNN encoder-decoder approach to machine translation. Note that in the\ndecoder we usually don\u2019t propagate the model\u2019s softmax outputs \u02c6 yt, but use teacher forcing to force each input\nto the correct gold value for training. We compute the softmax output distribution over \u02c6 yin the decoder in order\nto compute the loss at each token, which can then be averaged to compute a loss for the sentence. This loss is\nthen propagated through the decoder parameters and the encoder parameters.\n8.8 Attention\nThe simplicity of the encoder-decoder model is its clean separation of the encoder\u2014\nwhich builds a representation of the source text\u2014from the decoder, which uses this\ncontext to generate a target text. In the model as we\u2019ve described it so far, this\ncontext vector is hn, the hidden state of the last ( nth) time step of the source text.\nThis \ufb01nal hidden state is thus acting as a bottleneck : it must represent absolutely\neverything about the meaning of the source text, since the only thing the decoder\nknows about the source text is what\u2019s in this context vector (Fig. 8.20). Information\nat the beginning of the sentence, especially for long sentences, may not be equally\nwell represented in the context vector.\nEncoderDecoderbottleneckbottleneck\nFigure 8.20 Requiring the context cto be only the encoder\u2019s \ufb01nal hidden state forces all the\ninformation from the entire source sentence to pass through this representational bottleneck.\nThe attention mechanism is a solution to the bottleneck problem, a way ofattention\nmechanism\nallowing the decoder to get information from allthe hidden states of the encoder,\nnot just the last hidden state.\nIn the attention mechanism, as in the vanilla encoder-decoder model, the context\nvector cis a single vector that is a function of the hidden states of the encoder, that\nis,c=f(he\n1:::he\nn). Because the number of hidden states varies with the size of\nthe input, we can\u2019t use the entire set of encoder hidden state vectors directly as the 8.8 \u2022 A TTENTION 23\ncontext for the decoder.\nThe idea of attention is instead to create the single \ufb01xed-length vector cby taking\na weighted sum of all the encoder hidden states. The weights focus on (\u2018attend\nto\u2019) a particular part of the source text that is relevant for the token the decoder is\ncurrently producing. Attention thus replaces the static context vector with one that\nis dynamically derived from the encoder hidden states, different for each token in\ndecoding.\nThis context vector, ci, is generated anew with each decoding step iand takes\nall of the encoder hidden states into account in its derivation. We then make this\ncontext available during decoding by conditioning the computation of the current\ndecoder hidden state on it (along with the prior hidden state and the previous output\ngenerated by the decoder), as we see in this equation (and Fig. 8.21):\nhd\ni=g(\u02c6yi\u00001;hd\ni\u00001;ci) (8.34)\nhd1hd2hdiy1y2yic1c2ci\u2026\u2026\nFigure 8.21 The attention mechanism allows each hidden state of the decoder to see a\ndifferent, dynamic, context, which is a function of all the encoder hidden states.\nThe \ufb01rst step in computing ciis to compute how much to focus on each encoder\nstate, how relevant each encoder state is to the decoder state captured in hd\ni\u00001. We\ncapture relevance by computing\u2014 at each state iduring decoding\u2014a score(hd\ni\u00001;he\nj)\nfor each encoder state j.\nThe simplest such score, called dot-product attention , implements relevance asdot-product\nattention\nsimilarity: measuring how similar the decoder hidden state is to an encoder hidden\nstate, by computing the dot product between them:\nscore(hd\ni\u00001;he\nj) = hd\ni\u00001\u0001he\nj (8.35)\nThe score that results from this dot product is a scalar that re\ufb02ects the degree of\nsimilarity between the two vectors. The vector of these scores across all the encoder\nhidden states gives us the relevance of each encoder state to the current step of the\ndecoder.\nTo make use of these scores, we\u2019ll normalize them with a softmax to create a\nvector of weights, ai j, that tells us the proportional relevance of each encoder hidden\nstate jto the prior hidden decoder state, hd\ni\u00001.\nai j=softmax (score(hd\ni\u00001;he\nj))\n=exp(score(hd\ni\u00001;he\nj)\nP\nkexp(score(hd\ni\u00001;he\nk))(8.36)\nFinally, given the distribution in a, we can compute a \ufb01xed-length context vector for\nthe current decoder state by taking a weighted average over all the encoder hidden\nstates.\nci=X\njai jhe\nj (8.37) 24 CHAPTER 8 \u2022 RNN S AND LSTM S\nWith this, we \ufb01nally have a \ufb01xed-length context vector that takes into account\ninformation from the entire encoder state that is dynamically updated to re\ufb02ect the\nneeds of the decoder at each step of decoding. Fig. 8.22 illustrates an encoder-\ndecoder network with attention, focusing on the computation of one context vector\nci.\nEncoderDecoder\nhdi-1he3he2he1hdihiddenlayer(s)x1x2yix3xnyi-1yiyi+1hen ci.2.1.3.4attentionweightsci-1ci\n<latexit sha1_base64=\"TNdNmv/RIlrhPa6LgQyjjQLqyBA=\">AAACAnicdVDLSsNAFJ3UV62vqCtxM1gEVyHpI9Vd0Y3LCvYBTQyT6bSddvJgZiKUUNz4K25cKOLWr3Dn3zhpK6jogQuHc+7l3nv8mFEhTfNDyy0tr6yu5dcLG5tb2zv67l5LRAnHpIkjFvGOjwRhNCRNSSUjnZgTFPiMtP3xRea3bwkXNAqv5SQmboAGIe1TjKSSPP3AEUngjVIHsXiIvJSOpnB4Q7zR1NOLpmGaVbtqQdOwLbtk24qY5Yp9VoOWsjIUwQINT393ehFOAhJKzJAQXcuMpZsiLilmZFpwEkFihMdoQLqKhiggwk1nL0zhsVJ6sB9xVaGEM/X7RIoCISaBrzoDJIfit5eJf3ndRPZP3ZSGcSJJiOeL+gmDMoJZHrBHOcGSTRRBmFN1K8RDxBGWKrWCCuHrU/g/aZUMyzbKV5Vi/XwRRx4cgiNwAixQA3VwCRqgCTC4Aw/gCTxr99qj9qK9zltz2mJmH/yA9vYJSymYCA==</latexit>Xj\u21b5ijhej\u21b5ij<latexit sha1_base64=\"y8s4mGdpwrGrBnuSR+p1gJJXYdo=\">AAAB/nicdVDJSgNBEO2JW4zbqHjy0hgEL4YeJyQBL0EvHiOYBbIMPT09mTY9C909QhgC/ooXD4p49Tu8+Td2FkFFHxQ83quiqp6bcCYVQh9Gbml5ZXUtv17Y2Nza3jF391oyTgWhTRLzWHRcLClnEW0qpjjtJILi0OW07Y4up377jgrJ4uhGjRPaD/EwYj4jWGnJMQ+Cgedk7NSa9IgXq955MKDOrWMWUQnNAFGpYtfsakUTZNtWGUFrYRXBAg3HfO95MUlDGinCsZRdCyWqn2GhGOF0UuilkiaYjPCQdjWNcEhlP5udP4HHWvGgHwtdkYIz9ftEhkMpx6GrO0OsAvnbm4p/ed1U+bV+xqIkVTQi80V+yqGK4TQL6DFBieJjTTARTN8KSYAFJkonVtAhfH0K/yets5JVKdnX5WL9YhFHHhyCI3ACLFAFdXAFGqAJCMjAA3gCz8a98Wi8GK/z1pyxmNkHP2C8fQICDpWK</latexit>hdi\u00001\u00b7hej\u2026\u2026\nFigure 8.22 A sketch of the encoder-decoder network with attention, focusing on the computation of ci. The\ncontext value ciis one of the inputs to the computation of hd\ni. It is computed by taking the weighted sum of all\nthe encoder hidden states, each weighted by their dot product with the prior decoder hidden state hd\ni\u00001.\nIt\u2019s also possible to create more sophisticated scoring functions for attention\nmodels. Instead of simple dot product attention, we can get a more powerful function\nthat computes the relevance of each encoder hidden state to the decoder hidden state\nby parameterizing the score with its own set of weights, Ws.\nscore(hd\ni\u00001;he\nj) = hd\nt\u00001Wshe\nj\nThe weights Ws, which are then trained during normal end-to-end training, give the\nnetwork the ability to learn which aspects of similarity between the decoder and\nencoder states are important to the current application. This bilinear model also\nallows the encoder and decoder to use different dimensional vectors, whereas the\nsimple dot-product attention requires that the encoder and decoder hidden states\nhave the same dimensionality.\nWe\u2019ll return to the concept of attention when we de\ufb01ne the transformer archi-\ntecture in Chapter 9, which is based on a slight modi\ufb01cation of attention called\nself-attention .\n8.9 Summary\nThis chapter has introduced the concepts of recurrent neural networks and how they\ncan be applied to language problems. Here\u2019s a summary of the main points that we\ncovered:\n\u2022 In simple Recurrent Neural Networks sequences are processed one element at\na time, with the output of each neural unit at time tbased both on the current\ninput at tand the hidden layer from time t\u00001. BIBLIOGRAPHICAL AND HISTORICAL NOTES 25\n\u2022 RNNs can be trained with a straightforward extension of the backpropagation\nalgorithm, known as backpropagation through time (BPTT).\n\u2022 Simple recurrent networks fail on long inputs because of problems like van-\nishing gradients ; instead modern systems use more complex gated architec-\ntures such as LSTMs that explicitly decide what to remember and forget in\ntheir hidden and context layers.\n\u2022 Common language-based applications for RNNs include:\n\u2013Probabilistic language modeling: assigning a probability to a sequence,\nor to the next element of a sequence given the preceding words.\n\u2013Auto-regressive generation using a trained language model.\n\u2013Sequence labeling like part-of-speech tagging, where each element of a\nsequence is assigned a label.\n\u2013Sequence classi\ufb01cation, where an entire text is assigned to a category, as\nin spam detection, sentiment analysis or topic classi\ufb01cation.\n\u2013Encoder-decoder architectures, where an input is mapped to an output\nof different length and alignment.\nBibliographical and Historical Notes\nIn\ufb02uential investigations of RNNs were conducted in the context of the Parallel Dis-\ntributed Processing (PDP) group at UC San Diego in the 1980\u2019s. Much of this work\nwas directed at human cognitive modeling rather than practical NLP applications\n(Rumelhart and McClelland 1986, McClelland and Rumelhart 1986). Models using\nrecurrence at the hidden layer in a feedforward network (Elman networks) were in-\ntroduced by Elman (1990). Similar architectures were investigated by Jordan (1986)\nwith a recurrence from the output layer, and Mathis and Mozer (1995) with the\naddition of a recurrent context layer prior to the hidden layer. The possibility of\nunrolling a recurrent network into an equivalent feedforward network is discussed\nin (Rumelhart and McClelland, 1986).\nIn parallel with work in cognitive modeling, RNNs were investigated extensively\nin the continuous domain in the signal processing and speech communities (Giles\net al. 1994, Robinson et al. 1996). Schuster and Paliwal (1997) introduced bidirec-\ntional RNNs and described results on the TIMIT phoneme transcription task.\nWhile theoretically interesting, the dif\ufb01culty with training RNNs and manag-\ning context over long sequences impeded progress on practical applications. This\nsituation changed with the introduction of LSTMs in Hochreiter and Schmidhuber\n(1997) and Gers et al. (2000). Impressive performance gains were demonstrated\non tasks at the boundary of signal processing and language processing including\nphoneme recognition (Graves and Schmidhuber, 2005), handwriting recognition\n(Graves et al., 2007) and most signi\ufb01cantly speech recognition (Graves et al., 2013).\nInterest in applying neural networks to practical NLP problems surged with the\nwork of Collobert and Weston (2008) and Collobert et al. (2011). These efforts made\nuse of learned word embeddings, convolutional networks, and end-to-end training.\nThey demonstrated near state-of-the-art performance on a number of standard shared\ntasks including part-of-speech tagging, chunking, named entity recognition and se-\nmantic role labeling without the use of hand-engineered features.\nApproaches that married LSTMs with pretrained collections of word-embeddings\nbased on word2vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) 26 CHAPTER 8 \u2022 RNN S AND LSTM S\nquickly came to dominate many common tasks: part-of-speech tagging (Ling et al.,\n2015), syntactic chunking (S\u00f8gaard and Goldberg, 2016), named entity recognition\n(Chiu and Nichols, 2016; Ma and Hovy, 2016), opinion mining (Irsoy and Cardie,\n2014), semantic role labeling (Zhou and Xu, 2015) and AMR parsing (Foland and\nMartin, 2016). As with the earlier surge of progress involving statistical machine\nlearning, these advances were made possible by the availability of training data pro-\nvided by CONLL, SemEval, and other shared tasks, as well as shared resources such\nas Ontonotes (Pradhan et al., 2007), and PropBank (Palmer et al., 2005).\nThe modern neural encoder-decoder approach was pioneered by Kalchbrenner\nand Blunsom (2013), who used a CNN encoder and an RNN decoder. Cho et al.\n(2014) (who coined the name \u201cencoder-decoder\u201d) and Sutskever et al. (2014) then\nshowed how to use extended RNNs for both encoder and decoder. The idea that a\ngenerative decoder should take as input a soft weighting of the inputs, the central\nidea of attention, was \ufb01rst developed by Graves (2013) in the context of handwriting\nrecognition. Bahdanau et al. (2015) extended the idea, named it \u201cattention\u201d and\napplied it to MT. Bibliographical and Historical Notes 27\nBahdanau, D., K. H. Cho, and Y . Bengio. 2015. Neural ma-\nchine translation by jointly learning to align and translate.\nICLR 2015 .\nChiu, J. P. C. and E. Nichols. 2016. Named entity recognition\nwith bidirectional LSTM-CNNs. TACL , 4:357\u2013370.\nCho, K., B. van Merri \u00a8enboer, C. Gulcehre, D. Bahdanau,\nF. Bougares, H. Schwenk, and Y . Bengio. 2014. Learn-\ning phrase representations using RNN encoder\u2013decoder\nfor statistical machine translation. EMNLP .\nCollobert, R. and J. Weston. 2008. A uni\ufb01ed architecture for\nnatural language processing: Deep neural networks with\nmultitask learning. ICML .\nCollobert, R., J. Weston, L. Bottou, M. Karlen,\nK. Kavukcuoglu, and P. Kuksa. 2011. Natural language\nprocessing (almost) from scratch. JMLR , 12:2493\u20132537.\nElman, J. L. 1990. Finding structure in time. Cognitive sci-\nence, 14(2):179\u2013211.\nFoland, W. and J. H. Martin. 2016. CU-NLP at SemEval-\n2016 task 8: AMR parsing using LSTM-based recurrent\nneural networks. SemEval-2016 .\nGers, F. A., J. Schmidhuber, and F. Cummins. 2000. Learn-\ning to forget: Continual prediction with lstm. Neural\ncomputation , 12(10):2451\u20132471.\nGiles, C. L., G. M. Kuhn, and R. J. Williams. 1994. Dynamic\nrecurrent neural networks: Theory and applications.\nIEEE Trans. Neural Netw. Learning Syst. , 5(2):153\u2013156.\nGraves, A. 2013. Generating sequences with recurrent neural\nnetworks. ArXiv.\nGraves, A., S. Fern \u00b4andez, M. Liwicki, H. Bunke, and\nJ. Schmidhuber. 2007. Unconstrained on-line handwrit-\ning recognition with recurrent neural networks. NeurIPS .\nGraves, A., A.-r. Mohamed, and G. Hinton. 2013.\nSpeech recognition with deep recurrent neural networks.\nICASSP .\nGraves, A. and J. Schmidhuber. 2005. Framewise phoneme\nclassi\ufb01cation with bidirectional LSTM and other neural\nnetwork architectures. Neural Networks , 18(5-6):602\u2013\n610.\nHochreiter, S. and J. Schmidhuber. 1997. Long short-term\nmemory. Neural Computation , 9(8):1735\u20131780.\nIrsoy, O. and C. Cardie. 2014. Opinion mining with deep\nrecurrent neural networks. EMNLP .\nJordan, M. 1986. Serial order: A parallel distributed process-\ning approach. Technical Report ICS Report 8604, Univer-\nsity of California, San Diego.\nKalchbrenner, N. and P. Blunsom. 2013. Recurrent continu-\nous translation models. EMNLP .\nLing, W., C. Dyer, A. W. Black, I. Trancoso, R. Fermandez,\nS. Amir, L. Marujo, and T. Lu \u00b4\u0131s. 2015. Finding function\nin form: Compositional character models for open vocab-\nulary word representation. EMNLP .\nMa, X. and E. H. Hovy. 2016. End-to-end sequence labeling\nvia bi-directional LSTM-CNNs-CRF. ACL.\nMathis, D. A. and M. C. Mozer. 1995. On the computational\nutility of consciousness. NeurIPS . MIT Press.\nMcClelland, J. L. and D. E. Rumelhart, eds. 1986. Parallel\nDistributed Processing: Explorations in the Microstruc-\nture of Cognition , volume 2: Psychological and Biologi-\ncal Models . MIT Press.Mikolov, T., K. Chen, G. S. Corrado, and J. Dean. 2013. Ef-\n\ufb01cient estimation of word representations in vector space.\nICLR 2013 .\nMikolov, T., M. Kara\ufb01 \u00b4at, L. Burget, J. \u02c7Cernock `y, and\nS. Khudanpur. 2010. Recurrent neural network based lan-\nguage model. INTERSPEECH .\nMiller, G. A. and J. A. Selfridge. 1950. Verbal context and\nthe recall of meaningful material. American Journal of\nPsychology , 63:176\u2013185.\nPalmer, M., P. Kingsbury, and D. Gildea. 2005. The proposi-\ntion bank: An annotated corpus of semantic roles. Com-\nputational Linguistics , 31(1):71\u2013106.\nPennington, J., R. Socher, and C. D. Manning. 2014. GloVe:\nGlobal vectors for word representation. EMNLP .\nPradhan, S., E. H. Hovy, M. P. Marcus, M. Palmer, L. A.\nRamshaw, and R. M. Weischedel. 2007. Ontonotes: a\nuni\ufb01ed relational semantic representation. Int. J. Seman-\ntic Computing , 1(4):405\u2013419.\nRobinson, T., M. Hochberg, and S. Renals. 1996. The use\nof recurrent neural networks in continuous speech recog-\nnition. In C.-H. Lee, F. K. Soong, and K. K. Paliwal,\neds,Automatic speech and speaker recognition , 233\u2013258.\nSpringer.\nRumelhart, D. E., G. E. Hinton, and R. J. Williams. 1986.\nLearning internal representations by error propagation. In\nD. E. Rumelhart and J. L. McClelland, eds, Parallel Dis-\ntributed Processing , volume 2, 318\u2013362. MIT Press.\nRumelhart, D. E. and J. L. McClelland, eds. 1986. Parallel\nDistributed Processing: Explorations in the Microstruc-\nture of Cognition , volume 1: Foundations . MIT Press.\nSchuster, M. and K. K. Paliwal. 1997. Bidirectional recurrent\nneural networks. IEEE Transactions on Signal Process-\ning, 45:2673\u20132681.\nShannon, C. E. 1951. Prediction and entropy of printed En-\nglish. Bell System Technical Journal , 30:50\u201364.\nS\u00f8gaard, A. and Y . Goldberg. 2016. Deep multi-task learning\nwith low level tasks supervised at lower layers. ACL.\nSutskever, I., O. Vinyals, and Q. V . Le. 2014. Sequence to\nsequence learning with neural networks. NeurIPS .\nWerbos, P. 1974. Beyond regression: new tools for predic-\ntion and analysis in the behavioral sciences . Ph.D. thesis,\nHarvard University.\nWerbos, P. J. 1990. Backpropagation through time: what\nit does and how to do it. Proceedings of the IEEE ,\n78(10):1550\u20131560.\nZhou, J. and W. Xu. 2015. End-to-end learning of semantic\nrole labeling using recurrent neural networks. ACL.", "9.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\n9The Transformer\n\u201cThe true art of memory is the art of attention \u201d\nSamuel Johnson, Idler #74 , September 1759\nIn this chapter we introduce the transformer , the standard architecture for build-\ninglarge language models . Transformer-based large language models have com-\npletely changed the \ufb01eld of speech and language processing. Indeed, every subse-\nquent chapter in this textbook will make use of them. We\u2019ll focus for now on left-\nto-right (sometimes called causal or autoregressive) language modeling, in which\nwe are given a sequence of input tokens and predict output tokens one by one by\nconditioning on the prior context.\nThe transformer is a neural network with a speci\ufb01c structure that includes a\nmechanism called self-attention ormulti-head attention .1Attention can be thought\nof as a way to build contextual representations of a token\u2019s meaning by attending to\nand integrating information from surrounding tokens, helping the model learn how\ntokens relate to each other over large spans.\nStackedTransformerBlocksSolongandthanksforlongandthanksforNext tokenall\u2026\u2026\n\u2026\nU\nInput tokensx1x2LanguageModelingHead\nx3x4x5InputEncoding\nE1+\nE2+\nE3+\nE4+\nE5+\u2026\u2026\u2026\u2026\u2026\nU\nU\nU\nU\u2026logitslogitslogitslogitslogits\nFigure 9.1 The architecture of a (left-to-right) transformer, showing how each input token\nget encoded, passed through a set of stacked transformer blocks, and then a language model\nhead that predicts the next token.\nFig. 9.1 sketches the transformer architecture. A transformer has three major\ncomponents. At the center are columns of transformer blocks . Each block is a\nmultilayer network (a multi-head attention layer, feedforward networks and layer\nnormalization steps) that maps an input vector xiin column i(corresponding to input\n1Although multi-head attention developed historically from the RNN attention mechanism (Chapter 8),\nwe\u2019ll de\ufb01ne attention from scratch here for readers who haven\u2019t yet read Chapter 8. 2CHAPTER 9 \u2022 T HETRANSFORMER\ntoken i) to an output vector hi. The set of nblocks maps an entire context window\nof input vectors (x1;:::;xn)to a window of output vectors (h1;:::;hn)of the same\nlength. A column might contain from 12 to 96 or more stacked blocks.\nThe column of blocks is preceded by the input encoding component, which pro-\ncesses an input token (like the word thanks ) into a contextual vector representation,\nusing an embedding matrix Eand a mechanism for encoding token position. Each\ncolumn is followed by a language modeling head , which takes the embedding out-\nput by the \ufb01nal transformer block, passes it through an unembedding matrix Uand\na softmax over the vocabulary to generate a single token for that column.\nTransformer-based language models are complex, and so the details will unfold\nover the next 5 chapters. In the next sections we\u2019ll introduce multi-head attention,\nthe rest of the transformer block, and the input encoding and language modeling\nhead components. Chapter 10 discusses how language models are pretrained , and\nhow tokens are generated via sampling . Chapter 11 introduces masked language\nmodeling and the BERT family of bidirectional transformer encoder models. Chap-\nter 12 shows how to prompt LLMs to perform NLP tasks by giving instructions and\ndemonstrations, and how to align the model with human preferences. Chapter 13\nwill introduce machine translation with the encoder-decoder architecture.\n9.1 Attention\nRecall from Chapter 6 that for word2vec and other static embeddings, the repre-\nsentation of a word\u2019s meaning is always the same vector irrespective of the context:\nthe word chicken , for example, is always represented by the same \ufb01xed vector. So\na static vector for the word itmight somehow encode that this is a pronoun used\nfor animals and inanimate entities. But in context ithas a much richer meaning.\nConsider itin one of these two sentences:\n(9.1) The chicken didn\u2019t cross the road because itwas too tired.\n(9.2) The chicken didn\u2019t cross the road because itwas too wide.\nIn (9.1) itis the chicken (i.e., the reader knows that the chicken was tired), while\nin (9.2) itis the road (and the reader knows that the road was wide).2That is, if\nwe are to compute the meaning of this sentence, we\u2019ll need the meaning of itto be\nassociated with the chicken in the \ufb01rst sentence and associated with the road in\nthe second one, sensitive to the context.\nFurthermore, consider reading left to right like a causal language model, pro-\ncessing the sentence up to the word it:\n(9.3) The chicken didn\u2019t cross the road because it\nAt this point we don\u2019t yet know which thing itis going to end up referring to! So a\nrepresentation of itat this point might have aspects of both chicken androad as\nthe reader is trying to guess what happens next.\nThis fact that words have rich linguistic relationships with other words that may\nbe far away pervades language. Consider two more examples:\n(9.4) The keys to the cabinet areon the table.\n(9.5) I walked along the pond , and noticed one of the trees along the bank .\n2We say that in the \ufb01rst example itcorefers with the chicken, and in the second itcorefers with the\nroad; we\u2019ll return to this in Chapter 23. 9.1 \u2022 A TTENTION 3\nIn (9.4), the phrase The keys is the subject of the sentence, and in English and many\nlanguages, must agree in grammatical number with the verb are; in this case both are\nplural. In English we can\u2019t use a singular verb like iswith a plural subject like keys\n(we\u2019ll discuss agreement more in Chapter 18). In (9.5), we know that bank refers\nto the side of a pond or river and not a \ufb01nancial institution because of the context,\nincluding words like pond . (We\u2019ll discuss word senses more in Chapter 11.)\nThe point of all these examples is that these contextual words that help us com-\npute the meaning of words in context can be quite far away in the sentence or para-\ngraph. Transformers can build contextual representations of word meaning, contex-\ntual embeddings , by integrating the meaning of these helpful contextual words. In acontextual\nembeddings\ntransformer, layer by layer, we build up richer and richer contextualized representa-\ntions of the meanings of input tokens. At each layer, we compute the representation\nof a token iby combining information about ifrom the previous layer with infor-\nmation about the neighboring tokens to produce a contextualized representation for\neach word at each position.\nAttention is the mechanism in the transformer that weighs and combines the\nrepresentations from appropriate other tokens in the context from layer k\u00001 to build\nthe representation for tokens in layer k.\nThechickendidn\u2019tcrosstheroadbecauseitwastootiredThechickendidn\u2019tcrosstheroadbecauseitwastootiredLayer k+1Layer kself-attention distributioncolumns corresponding to input tokens\nFigure 9.2 The self-attention weight distribution athat is part of the computation of the\nrepresentation for the word itat layer k+1. In computing the representation for it, we attend\ndifferently to the various words at layer l, with darker shades indicating higher self-attention\nvalues. Note that the transformer is attending highly to the columns corresponding to the\ntokens chicken androad , a sensible result, since at the point where itoccurs, it could plausibly\ncorefers with the chicken or the road, and hence we\u2019d like the representation for itto draw on\nthe representation for these earlier words. Figure adapted from Uszkoreit (2017).\nFig. 9.2 shows a schematic example simpli\ufb01ed from a transformer (Uszkoreit,\n2017). The \ufb01gure describes the situation when the current token is itand we need\nto compute a contextual representation for this token at layer k+1 of the transformer,\ndrawing on the representations (from layer k) of every prior token. The \ufb01gure uses\ncolor to represent the attention distribution over the contextual words: the tokens\nchicken androad both have a high attention weight, meaning that as we are com-\nputing the representation for it, we will draw most heavily on the representation for\nchicken androad . This will be useful in building the \ufb01nal representation for it,\nsince itwill end up coreferring with either chicken orroad .\nLet\u2019s now turn to how this attention distribution is represented and computed. 4CHAPTER 9 \u2022 T HETRANSFORMER\n9.1.1 Attention more formally\nAs we\u2019ve said, the attention computation is a way to compute a vector representation\nfor a token at a particular layer of a transformer, by selectively attending to and\nintegrating information from prior tokens at the previous layer. Attention takes an\ninput representation xicorresponding to the input token at position i, and a context\nwindow of prior inputs x1::xi\u00001, and produces an output ai.\nIn causal, left-to-right language models, the context is any of the prior words.\nThat is, when processing xi, the model has access to xias well as the representations\nof all the prior tokens in the context window (context windows consist of thousands\nof tokens) but no tokens after i. (By contrast, in Chapter 11 we\u2019ll generalize attention\nso it can also look ahead to future words.)\nFig. 9.3 illustrates this \ufb02ow of information in an entire causal self-attention layer,\nin which this same attention computation happens in parallel at each token position\ni. Thus a self-attention layer maps input sequences (x1;:::;xn)to output sequences\nof the same length (a1;:::;an).\nattentionattentionSelf-AttentionLayerattentionattentionattentiona1a2a3a4a5x3x4x5x1x2\nFigure 9.3 Information \ufb02ow in causal self-attention. When processing each input xi, the\nmodel attends to all the inputs up to, and including xi.\nSimpli\ufb01ed version of attention At its heart, attention is really just a weighted\nsum of context vectors, with a lot of complications added to how the weights are\ncomputed and what gets summed. For pedagogical purposes let\u2019s \ufb01rst describe a\nsimpli\ufb01ed intuition of attention, in which the attention output aiat token position i\nis simply the weighted sum of all the representations xj, for all j\u0014i; we\u2019ll use ai j\nto mean how much xishould contribute to aj:\nSimpli\ufb01ed version: ai=X\nj\u0014iai jxj (9.6)\nEach ai jis a scalar used for weighing the value of input xjwhen summing up\nthe inputs to compute ai. How shall we compute this aweighting? In attention we\nweight each prior embedding proportionally to how similar it is to the current token\ni. So the output of attention is a sum of the embeddings of prior tokens weighted\nby their similarity with the current token embedding. We compute similarity scores\nviadot product , which maps two vectors into a scalar value ranging from \u0000\u00a5to\n\u00a5. The larger the score, the more similar the vectors that are being compared. We\u2019ll\nnormalize these scores with a softmax to create the vector of weights ai j;j\u0014i.\nSimpli\ufb01ed Version: score(xi;xj) = xi\u0001xj (9.7)\nai j=softmax (score(xi;xj))8j\u0014i(9.8)\nThus in Fig. 9.3 we compute a3by computing three scores: x3\u0001x1,x3\u0001x2andx3\u0001x3,\nnormalizing them by a softmax, and using the resulting probabilities as weights\nindicating each of their proportional relevance to the current position i. Of course, 9.1 \u2022 A TTENTION 5\nthe softmax weight will likely be highest for xi, since xiis very similar to itself,\nresulting in a high dot product. But other context words may also be similar to i, and\nthe softmax will also assign some weight to those words. Then we use these weights\nas the avalues in Eq. 9.6 to compute the weighted sum that is our a3.\nThe simpli\ufb01ed attention in equations 9.6 \u2013 9.8 demonstrates the attention-based\napproach to computing ai: compare the xito prior vectors, normalize those scores\ninto a probability distribution used to weight the sum of the prior vector. But now\nwe\u2019re ready to remove the simpli\ufb01cations.\nA single attention head using query, key, and value matrices Now that we\u2019ve\nseen a simple intuition of attention, let\u2019s introduce the actual attention head , the attention head\nversion of attention that\u2019s used in transformers. (The word head is often used in head\ntransformers to refer to speci\ufb01c structured layers). The attention head allows us to\ndistinctly represent three different roles that each input embedding plays during the\ncourse of the attention process:\n\u2022 As the current element being compared to the preceding inputs. We\u2019ll refer to\nthis role as a query . query\n\u2022 In its role as a preceding input that is being compared to the current element\nto determine a similarity weight. We\u2019ll refer to this role as a key. key\n\u2022 And \ufb01nally, as a value of a preceding element that gets weighted and summed value\nup to compute the output for the current element.\nTo capture these three different roles, transformers introduce weight matrices\nWQ,WK, and WV. These weights will project each input vector xiinto a represen-\ntation of its role as a key, query, or value:\nqi=xiWQ;ki=xiWK;vi=xiWV(9.9)\nGiven these projections, when we are computing the similarity of the current ele-\nment xiwith some prior element xj, we\u2019ll use the dot product between the current\nelement\u2019s query vector qiand the preceding element\u2019s keyvector kj. Furthermore,\nthe result of a dot product can be an arbitrarily large (positive or negative) value, and\nexponentiating large values can lead to numerical issues and loss of gradients during\ntraining. To avoid this, we scale the dot product by a factor related to the size of the\nembeddings, via diving by the square root of the dimensionality of the query and\nkey vectors ( dk). We thus replace the simpli\ufb01ed Eq. 9.7 with Eq. 9.11. The ensuing\nsoftmax calculation resulting in ai jremains the same, but the output calculation for\naiis now based on a weighted sum over the value vectors v(Eq. 9.13).\nHere\u2019s a \ufb01nal set of equations for computing self-attention for a single self-\nattention output vector aifrom a single input vector xi. This version of attention\ncomputes aiby summing the values of the prior elements, each weighted by the\nsimilarity of its keyto the query from the current element:\nqi=xiWQ;kj=xjWK;vj=xjWV(9.10)\nscore(xi;xj) =qi\u0001kjpdk(9.11)\nai j=softmax (score(xi;xj))8j\u0014i (9.12)\nai=X\nj\u0014iai jvj (9.13)\nWe illustrate this in Fig. 9.4 for the case of calculating the value of the third output\na3in a sequence. 6CHAPTER 9 \u2022 T HETRANSFORMER\n6. Sum the weighted value vectors4. Turn into \ud835\udefci,j weights via softmaxa3\n1. Generate key, query, value vectors2. Compare x3\u2019s query withthe keys for x1, x2, and x3Output of self-attention\nWk\nWvWqx1kqvx3kqvx2kqv\n\u00d7\u00d7\nWkWkWqWqWvWv5. Weigh each value vector\u00f7\u221adk3. Divide score by \u221adk\u00f7\u221adk\u00f7\u221adk\ud835\udefc3,1\ud835\udefc3,2\ud835\udefc3,3\nFigure 9.4 Calculating the value of a3, the third element of a sequence using causal (left-\nto-right) self-attention.\nLet\u2019s talk shapes. The input to attention xiand the output from attention aiboth\nhave the same dimensionality 1 \u0002d(We often call dthemodel dimensionality ,\nand indeed as we\u2019ll discuss in Section 9.2 the output hiof each transformer block,\nas well as the intermediate vectors inside the transformer block also have the same\ndimensionality 1\u0002d.).\nWe\u2019ll have a dimension dkfor the key and query vectors. The query vector and\nthe key vector are both dimensionality 1 \u0002dk, so we can take their dot product qi\u0001kj.\nWe\u2019ll have a separate dimension dvfor the value vectors. The transform matrix WQ\nhas shape [d\u0002dk],WKis[d\u0002dk], and WVis[d\u0002dv]. In the original transformer\nwork (Vaswani et al., 2017), dwas 512, dkanddvwere both 64.\nMulti-head Attention Equations 9.11-9.13 describe a single attention head . But\nactually, transformers use multiple attention heads. The intuition is that each head\nmight be attending to the context for different purposes: heads might be special-\nized to represent different linguistic relationships between context elements and the\ncurrent token, or to look for particular kinds of patterns in the context.\nSo in multi-head attention we have hseparate attention heads that reside inmulti-head\nattention\nparallel layers at the same depth in a model, each with its own set of parameters that\nallows the head to model different aspects of the relationships among inputs. Thus\neach head iin a self-attention layer has its own set of key, query and value matrices:\nWKi,WQiandWVi. These are used to project the inputs into separate key, value,\nand query embeddings for each head.\nWhen using multiple heads the model dimension dis still used for the input\nand output, the key and query embeddings have dimensionality dk, and the value\nembeddings are of dimensionality dv(again, in the original transformer paper dk=\ndv=64,h=8, and d=512). Thus for each head i, we have weight layers WQiof\nshape [d\u0002dk],WKiof shape [d\u0002dk], and WViof shape [d\u0002dv].\nBelow are the equations for attention augmented with multiple heads; Fig. 9.5 9.2 \u2022 T RANSFORMER BLOCKS 7\nshows an intuition.\nqc\ni=xiWQc;kc\nj=xjWKc;vc\nj=xjWVc;8c1\u0014c\u0014h (9.14)\nscorec(xi;xj) =qc\ni\u0001kc\njpdk(9.15)\nac\ni j=softmax (scorec(xi;xj))8j\u0014i(9.16)\nheadc\ni=X\nj\u0014iac\ni jvc\nj (9.17)\nai= (head1\bhead2:::\bheadh)WO(9.18)\nMultiHeadAttention (xi;[x1;\u0001\u0001\u0001;xN]) = ai (9.19)\nThe output of each of the hheads is of shape 1 \u0002dv, and so the output of the\nmulti-head layer with hheads consists of hvectors of shape 1\u0002dv. These are con-\ncatenated to produce a single output with dimensionality 1 \u0002hdv. Then we use yet\nanother linear projection WO2Rhdv\u0002dto reshape it, resulting in the multi-head\nattention vector aiwith the correct output shape [1xd]at each input i.\naixi-1xixi-2xi-3WK1Head 1WV1WQ1\u2026\u2026WK2Head 2WV2WQ2WK8Head 8WV8WQ8aiWO  [hdv x d][1 x dv ][1 x d]\n[1 x d][1 x hdv ]Project down to dConcatenate OutputsEach headattends di\ufb00erentlyto context\u2026[1 x dv ]\nFigure 9.5 The multi-head attention computation for input xi, producing output ai. A multi-head attention\nlayer has hheads, each with its own key, query and value weight matrices. The outputs from each of the heads\nare concatenated and then projected down to d, thus producing an output of the same size as the input.\n9.2 Transformer Blocks\nThe self-attention calculation lies at the core of what\u2019s called a transformer block,\nwhich, in addition to the self-attention layer, includes three other kinds of layers: (1)\na feedforward layer, (2) residual connections, and (3) normalizing layers (colloqui-\nally called \u201clayer norm\u201d).\nFig. 9.6 illustrates a transformer block, sketching a common way of thinking\nabout the block that is called the residual stream (Elhage et al., 2021). In the resid- residual stream\nual stream viewpoint, we consider the processing of an individual token ithrough\nthe transformer block as a single stream of d-dimensional representations for token\nposition i. This residual stream starts with the original input vector, and the various 8CHAPTER 9 \u2022 T HETRANSFORMER\nLayer Norm\nxi+hi-1\nLayer NormMultiHeadAttentionFeedforward\nxi-1xi+1hihi+1\n+\u2026\u2026ResidualStream\nFigure 9.6 The architecture of a transformer block showing the residual stream . This\n\ufb01gure shows the prenorm version of the architecture, in which the layer norms happen before\nthe attention and feedforward layers rather than after.\ncomponents read their input from the residual stream and add their output back into\nthe stream.\nThe input at the bottom of the stream is an embedding for a token, which has\ndimensionality d. This initial embedding gets passed up (by residual connections ),\nand is progressively added to by the other components of the transformer: the at-\ntention layer that we have seen, and the feedforward layer that we will introduce.\nBefore the attention and feedforward layer is a computation called the layer norm .\nThus the initial vector is passed through a layer norm and attention layer, and\nthe result is added back into the stream, in this case to the original input vector\nxi. And then this summed vector is again passed through another layer norm and a\nfeedforward layer, and the output of those is added back into the residual, and we\u2019ll\nusehito refer to the resulting output of the transformer block for token i. (In earlier\ndescriptions the residual stream was often described using a different metaphor as\nresidual connections that add the input of a component to its output, but the residual\nstream is a more perspicuous way of visualizing the transformer.)\nWe\u2019ve already seen the attention layer, so let\u2019s now introduce the feedforward\nand layer norm computations in the context of processing a single input xiat token\nposition i.\nFeedforward layer The feedforward layer is a fully-connected 2-layer network,\ni.e., one hidden layer, two weight matrices, as introduced in Chapter 7. The weights\nare the same for each token position i, but are different from layer to layer. It\nis common to make the dimensionality dffof the hidden layer of the feedforward\nnetwork be larger than the model dimensionality d. (For example in the original\ntransformer model, d=512 and dff=2048.)\nFFN(xi) =ReLU (xiW1+b1)W2+b2 (9.20)\nLayer Norm At two stages in the transformer block we normalize the vector (Ba\net al., 2016). This process, called layer norm (short for layer normalization), is one layer norm 9.2 \u2022 T RANSFORMER BLOCKS 9\nof many forms of normalization that can be used to improve training performance\nin deep neural networks by keeping the values of a hidden layer in a range that\nfacilitates gradient-based training.\nLayer norm is a variation of the z-score from statistics, applied to a single vec-\ntor in a hidden layer. That is, the term layer norm is a bit confusing; layer norm\nisnotapplied to an entire transformer layer, but just to the embedding vector of a\nsingle token. Thus the input to layer norm is a single vector of dimensionality d\nand the output is that vector normalized, again of dimensionality d. The \ufb01rst step in\nlayer normalization is to calculate the mean, m, and standard deviation, s, over the\nelements of the vector to be normalized. Given an embedding vector xof dimen-\nsionality d, these values are calculated as follows.\nm=1\nddX\ni=1xi (9.21)\ns=vuut1\nddX\ni=1(xi\u0000m)2(9.22)\nGiven these values, the vector components are normalized by subtracting the mean\nfrom each and dividing by the standard deviation. The result of this computation is\na new vector with zero mean and a standard deviation of one.\n^ x=(x\u0000m)\ns(9.23)\nFinally, in the standard implementation of layer normalization, two learnable param-\neters, gandb, representing gain and offset values, are introduced.\nLayerNorm (x) =g(x\u0000m)\ns+b (9.24)\nPutting it all together The function computed by a transformer block can be ex-\npressed by breaking it down with one equation for each component computation,\nusing t(of shape [1\u0002d]) to stand for transformer and superscripts to demarcate\neach computation inside the block:\nt1\ni=LayerNorm (xi) (9.25)\nt2\ni=MultiHeadAttention (t1\ni;\u0002\nx1\n1;\u0001\u0001\u0001;x1\nN\u0003\n) (9.26)\nt3\ni=t2\ni+xi (9.27)\nt4\ni=LayerNorm (t3\ni) (9.28)\nt5\ni=FFN(t4\ni) (9.29)\nhi=t5\ni+t3\ni (9.30)\nNotice that the only component that takes as input information from other tokens\n(other residual streams) is multi-head attention, which (as we see from (9.27)) looks\nat all the neighboring tokens in the context. The output from attention, however, is\nthen added into this token\u2019s embedding stream. In fact, Elhage et al. (2021) show that\nwe can view attention heads as literally moving information from the residual stream\nof a neighboring token into the current stream. The high-dimensional embedding\nspace at each position thus contains information about the current token and about\nneighboring tokens, albeit in different subspaces of the vector space. Fig. 9.7 shows\na visualization of this movement. 10 CHAPTER 9 \u2022 T HETRANSFORMER\nToken Aresidual streamToken Bresidual stream\nFigure 9.7 An attention head can move information from token A\u2019s residual stream into\ntoken B\u2019s residual stream.\nCrucially, the input and output dimensions of transformer blocks are matched so\nthey can be stacked. Each token vector xiat the input to the block has dimensionality\nd, and the output hialso has dimensionality d. Transformers for large language\nmodels stack many of these blocks, from 12 layers (used for the T5 or GPT-3-small\nlanguage models) to 96 layers (used for GPT-3 large), to even more for more recent\nmodels. We\u2019ll come back to this issue of stacking in a bit.\nEquation (9.27) and following are just the equation for a single transformer\nblock, but the residual stream metaphor goes through all the transformer layers,\nfrom the \ufb01rst transformer blocks to the 12th, in a 12-layer transformer. At the ear-\nlier transformer blocks, the residual stream is representing the current token. At the\nhighest transformer blocks, the residual stream is usually representing the following\ntoken, since at the very end it\u2019s being trained to predict the next token.\nOnce we stack many blocks, there is one more requirement: at the very end of\nthe last (highest) transformer block, there is a single extra layer norm that is run on\nthe last hiof each token stream (just below the language model head layer that we\nwill de\ufb01ne soon).3\n9.3 Parallelizing computation using a single matrix X\nThis description of multi-head attention and the rest of the transformer block has\nbeen from the perspective of computing a single output at a single time step iin\na single residual stream. But as we pointed out earlier, the attention computation\nperformed for each token to compute aiis independent of the computation for each\nother token, and that\u2019s also true for all the computation in the transformer block\ncomputing hifrom the input xi. That means we can easily parallelize the entire\ncomputation, taking advantage of ef\ufb01cient matrix multiplication routines.\nWe do this by packing the input embeddings for the Ntokens of the input se-\nquence into a single matrix Xof size [N\u0002d]. Each row of Xis the embedding of\none token of the input. Transformers for large language models commonly have an\ninput length N= 1K, 2K, or as many as 32K tokens (or more), so Xtypically has be-\ntween 1K and 32K rows, each of the dimensionality of the embedding d(the model\n3Note that we are using the most common current transformer architecture, which is called the prenorm\narchitecture. The original de\ufb01nition of the transformer in Vaswani et al. (2017) used an alternative archi-\ntecture called the postnorm transformer in which the layer norm happens after the attention and FFN\nlayers; it turns out moving the layer norm beforehand works better, but does require this one extra layer\nat the end. 9.3 \u2022 P ARALLELIZING COMPUTATION USING A SINGLE MATRIX X 11\ndimension).\nParallelizing attention Let\u2019s \ufb01rst see this for a single attention head and then turn\nto multiple heads, and then add in the rest of the components in the transformer\nblock. For one head we multiply Xby the key, query, and value matrices WQof\nshape [d\u0002dk],WKof shape [d\u0002dk], andWVof shape [d\u0002dv], to produce matrices\nQof shape [N\u0002dk],K2RN\u0002dk, and V2RN\u0002dv, containing all the key, query, and\nvalue vectors:\nQ=XWQ;K=XWK;V=XWV(9.31)\nGiven these matrices we can compute all the requisite query-key comparisons simul-\ntaneously by multiplying QandK|in a single matrix multiplication. The product is\nof shape N\u0002N, visualized in Fig. 9.8.\nq1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3NNq1\u2022k2q1\u2022k3q1\u2022k4q2\u2022k3q2\u2022k4q3\u2022k4\nFigure 9.8 The N\u0002NQK|matrix showing how it computes all qi\u0001kjcomparisons in a\nsingle matrix multiple.\nOnce we have this QK|matrix, we can very ef\ufb01ciently scale these scores, take\nthe softmax, and then multiply the result by Vresulting in a matrix of shape N\u0002d:\na vector embedding representation for each token in the input. We\u2019ve reduced the\nentire self-attention step for an entire sequence of Ntokens for one head to the\nfollowing computation:\nA=softmax\u0012\nmask\u0012QK|\npdk\u0013\u0013\nV (9.32)\nMasking out the future You may have noticed that we introduced a mask function\nin Eq. 9.32 above. This is because the self-attention computation as we\u2019ve described\nit has a problem: the calculation in QK|results in a score for each query value\nto every key value, including those that follow the query . This is inappropriate in\nthe setting of language modeling: guessing the next word is pretty simple if you\nalready know it! To \ufb01x this, the elements in the upper-triangular portion of the\nmatrix are zeroed out (set to \u0000\u00a5), thus eliminating any knowledge of words that\nfollow in the sequence. This is done in practice by adding a mask matrix Min\nwhich Mi j=\u0000\u00a58j>i(i.e. for the upper-triangular portion) and Mi j=0 otherwise.\nFig. 9.9 shows the resulting masked QK|matrix. (we\u2019ll see in Chapter 11 how to\nmake use of words in the future for tasks that need it).\nFig. 9.10 shows a schematic of all the computations for a single attention head\nparallelized in matrix form.\nFig. 9.8 and Fig. 9.9 also make it clear that attention is quadratic in the length\nof the input, since at each layer we need to compute dot products between each pair\nof tokens in the input. This makes it expensive to compute attention over very long\ndocuments (like entire novels). Nonetheless modern large language models manage\nto use quite long contexts of thousands or tens of thousands of tokens. 12 CHAPTER 9 \u2022 T HETRANSFORMER\nq1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3NN\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\nFigure 9.9 TheN\u0002NQK|matrix showing the qi\u0001kjvalues, with the upper-triangle por-\ntion of the comparisons matrix zeroed out (set to \u0000\u00a5, which the softmax will turn to zero).\nq1q2q3q4k1k2k3k4QKTQKTv1v2v3v4Vq2\u2022k2q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k2q3\u2022k3\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221eq1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3q1\u2022k2q2\u2022k3q1\u2022k3q3\u2022k4q2\u2022k4q1\u2022k4x=QKT maskedmask=q1\u2022k1q2\u2022k1q4\u2022k1q3\u2022k1q1\u2022k1q1\u2022k1=xa1a2a3a4AQuery Token 1Query Token 2Query Token 3Query Token 4QInput Token 1Input Token 2Input Token 3Input Token 4XxWQ=Value Token 1Value Token 2Value Token 3Value Token 4VxWV=Input Token 1Input Token 2Input Token 3Input Token 4XKey Token 1Key Token 2Key Token 3Key Token 4KxWK=Input Token 1Input Token 2Input Token 3Input Token 4X\nN x dkdk x NN x NN x NN x dvN x dvd x dkd x dkd x dvN x dN x dkN x dN x dkN x dN x dv\nFigure 9.10 Schematic of the attention computation for a single attention head in parallel. The \ufb01rst row shows\nthe computation of the Q,K, and Vmatrices. The second row shows the computation of QKT, the masking\n(the softmax computation and the normalizing by dimensionality are not shown) and then the weighted sum of\nthe value vectors to get the \ufb01nal attention vectors.\nParallelizing multi-head attention In multi-head attention, as with self-attention,\nthe input and output have the model dimension d, the key and query embeddings\nhave dimensionality dk, and the value embeddings are of dimensionality dv(again,\nin the original transformer paper dk=dv=64,h=8, and d=512). Thus for each\nhead i, we have weight layers WQi2Rd\u0002dk,WKi2Rd\u0002dk, and WVi2Rd\u0002dv, and\nthese get multiplied by the inputs packed into Xto produce Q2RN\u0002dk,K2RN\u0002dk,\nandV2RN\u0002dv. The output of each of the hheads is of shape N\u0002dv, and so the\noutput of the multi-head layer with hheads consists of hmatrices of shape N\u0002dv. To\nmake use of these matrices in further processing, they are concatenated to produce\na single output with dimensionality N\u0002hdv. Finally, we use yet another linear\nprojection WO2Rhdv\u0002d, that reshape it to the original output dimension for each\ntoken. Multiplying the concatenated N\u0002hdvmatrix output by WO2Rhdv\u0002dyields 9.4 \u2022 T HE INPUT :EMBEDDINGS FOR TOKEN AND POSITION 13\nthe self-attention output Aof shape [ N\u0002d].\nQi=XWQi;Ki=XWKi;Vi=XWVi(9.33)\nhead i=SelfAttention (Qi;Ki;Vi) = softmax\u0012QiKi|\npdk\u0013\nVi(9.34)\nMultiHeadAttention (X) = ( head 1\bhead 2:::\bhead h)WO(9.35)\nPutting it all together with the parallel input matrix XThe function computed\nin parallel by an entire layer of Ntransformer block over the entire Ninput tokens\ncan be expressed as:\nO=LayerNorm (X+MultiHeadAttention (X)) (9.36)\nH=LayerNorm (O+FFN(O)) (9.37)\nOr we can break it down with one equation for each component computation, using\nT(of shape [N\u0002d]) to stand for transformer and superscripts to demarcate each\ncomputation inside the block:\nT1=MultiHeadAttention (X) (9.38)\nT2=X+T1(9.39)\nT3=LayerNorm (T2) (9.40)\nT4=FFN(T3) (9.41)\nT5=T4+T3(9.42)\nH=LayerNorm (T5) (9.43)\nHere when we use a notation like FFN (T3)we mean that the same FFN is applied\nin parallel to each of the Nembedding vectors in the window. Similarly, each of the\nNtokens is normed in parallel in the LayerNorm. Crucially, the input and output\ndimensions of transformer blocks are matched so they can be stacked. Since each\ntoken xiat the input to the block has dimensionality d, that means the input Xand\noutput Hare both of shape [N\u0002d].\n9.4 The input: embeddings for token and position\nLet\u2019s talk about where the input Xcomes from. Given a sequence of Ntokens ( Nis\nthe context length in tokens), the matrix Xof shape [N\u0002d]has an embedding for embedding\neach word in the context. The transformer does this by separately computing two\nembeddings: an input token embedding, and an input positional embedding.\nA token embedding, introduced in Chapter 7 and Chapter 8, is a vector of di-\nmension dthat will be our initial representation for the input token. (As we pass\nvectors up through the transformer layers in the residual stream, this embedding\nrepresentation will change and grow, incorporating context and playing a different\nrole depending on the kind of language model we are building.) The set of initial\nembeddings are stored in the embedding matrix E, which has a row for each of the\njVjtokens in the vocabulary. Thus each word is a row vector of ddimensions, and\nEhas shape [jVj\u0002d].\nGiven an input token string like Thanks for all the we \ufb01rst convert the tokens\ninto vocabulary indices (these were created when we \ufb01rst tokenized the input using 14 CHAPTER 9 \u2022 T HETRANSFORMER\nBPE or SentencePiece). So the representation of thanks for all the might be w=\n[5;4000 ;10532 ;2224]. Next we use indexing to select the corresponding rows from\nE, (row 5, row 4000, row 10532, row 2224).\nAnother way to think about selecting token embeddings from the embedding\nmatrix is to represent tokens as one-hot vectors of shape [1\u0002jVj], i.e., with one\ndimension for each word in the vocabulary. Recall that in a one-hot vector all the one-hot vector\nelements are 0 except one, the element whose dimension is the word\u2019s index in the\nvocabulary, which has value 1. So if the word \u201cthanks\u201d has index 5 in the vocabulary,\nx5=1, and xi=08i6=5, as shown here:\n[0 0 0 0 1 0 0 ... 0 0 0 0]\n1 2 3 4 5 6 7 ... ... |V|\nMultiplying by a one-hot vector that has only one non-zero element xi=1 simply\nselects out the relevant row vector for word i, resulting in the embedding for word i,\nas depicted in Fig. 9.11.\nE|V|d1|V|d=\u2715550 0 0 0 1 0 0 \u2026 0 0 0 0 1\nFigure 9.11 Selecting the embedding vector for word V5by multiplying the embedding\nmatrix Ewith a one-hot vector with a 1 in index 5.\nWe can extend this idea to represent the entire token sequence as a matrix of one-\nhot vectors, one for each of the Npositions in the transformer\u2019s context window, as\nshown in Fig. 9.12.\nE|V|ddN=\u2715|V|N0 0 0 0 0 0 0 \u2026 0 0 1 0 0 0 0 0 1 0 0 \u2026 0 0 0 0 1 0 0 0 0 0 0 \u2026 0 0 0 0 0 0 0 0 1 0 0 \u2026 0 0 0 0 \u2026\nFigure 9.12 Selecting the embedding matrix for the input sequence of token ids Wby mul-\ntiplying a one-hot matrix corresponding to Wby the embedding matrix E.\nThese token embeddings are not position-dependent. To represent the position\nof each token in the sequence, we combine these token embeddings with positional\nembeddings speci\ufb01c to each position in an input sequence.positional\nembeddings\nWhere do we get these positional embeddings? The simplest method, called\nabsolute position , is to start with randomly initialized embeddings correspondingabsolute\nposition\nto each possible input position up to some maximum length. For example, just as\nwe have an embedding for the word \ufb01sh, we\u2019ll have an embedding for the position 3.\nAs with word embeddings, these positional embeddings are learned along with other\nparameters during training. We can store them in a matrix Eposof shape [1\u0002N].\nTo produce an input embedding that captures positional information, we just\nadd the word embedding for each input to its corresponding positional embedding.\nThe individual token and position embeddings are both of size [1\u0002d], so their sum is 9.5 \u2022 T HELANGUAGE MODELING HEAD 15\nX = CompositeEmbeddings(word + position)Transformer BlockJanet1will2back3Janetwillbackthebillthe4bill5\n+++++PositionEmbeddingsWordEmbeddings\nFigure 9.13 A simple way to model position: add an embedding of the absolute position to\nthe token embedding to produce a new embedding of the same dimensionality.\nalso[1\u0002d], This new embedding serves as the input for further processing. Fig. 9.13\nshows the idea.\nThe \ufb01nal representation of the input, the matrix X, is an [N\u0002d]matrix in which\neach row iis the representation of the ith token in the input, computed by adding\nE[id(i)]\u2014the embedding of the id of the token that occurred at position i\u2014, to P[i],\nthe positional embedding of position i.\nA potential problem with the simple absolute position embedding approach is\nthat there will be plenty of training examples for the initial positions in our inputs\nand correspondingly fewer at the outer length limits. These latter embeddings may\nbe poorly trained and may not generalize well during testing. An alternative ap-\nproach to absolute positional embeddings is to choose a static function that maps\ninteger inputs to real-valued vectors in a way that captures the inherent relation-\nships among the positions. That is, it captures the fact that position 4 in an input is\nmore closely related to position 5 than it is to position 17. A combination of sine\nand cosine functions with differing frequencies was used in the original transformer\nwork. Even more complex positional embedding methods exist, such as ones that\nrepresent relative position instead of absolute position, often implemented in the\nattention mechanism at each layer rather than being added once at the initial input.\n9.5 The Language Modeling Head\nThe last component of the transformer we must introduce is the language modeling\nhead . Here we are using the word head to mean the additional neural circuitry welanguage\nmodeling head\nhead add on top of the basic transformer architecture when we apply pretrained trans-\nformer models to various tasks. The language modeling head is the circuitry we\nneed to do language modeling.\nRecall that language models, from the simple n-gram models of Chapter 3 through\nthe feedforward and RNN language models of Chapter 7 and Chapter 8, are word\npredictors. Given a context of words, they assign a probability to each possible next\nword. For example, if the preceding context is \u201cThanks for all the\u201d and we want to\nknow how likely the next word is \u201c\ufb01sh\u201d we would compute:\nP(\ufb01shjThanks for all the )\nLanguage models give us the ability to assign such a conditional probability to every\npossible next word, giving us a distribution over the entire vocabulary. The n-gram 16 CHAPTER 9 \u2022 T HETRANSFORMER\nlanguage models of Chapter 3 compute the probability of a word given counts of\nits occurrence with the n\u00001 prior words. The context is thus of size n\u00001. For\ntransformer language models, the context is the size of the transformer\u2019s context\nwindow, which can be quite large: 2K, 4K, even 32K tokens for very large models.\nThe job of the language modeling head is to take the output of the \ufb01nal trans-\nformer layer from the last token Nand use it to predict the upcoming word at posi-\ntionN+1. Fig. 9.14 shows how to accomplish this task, taking the output of the last\ntoken at the last layer (the d-dimensional output embedding of shape [1\u0002d]) and\nproducing a probability distribution over words (from which we will choose one to\ngenerate).\nLayer LTransformerBlockSoftmax over vocabulary VUnembedding layer\u20261 x |V|Logits Word probabilities1 x |V|hL1w1w2wNhL2hLNd x |V|1 x d   Unembedding layerU = ETy1y2y|V|\u2026u1u2u|V|\u2026Language Model Headtakes hLN and outputs adistribution over vocabulary V\nFigure 9.14 The language modeling head: the circuit at the top of a transformer that maps from the output\nembedding for token Nfrom the last transformer layer ( hL\nN) to a probability distribution over words in the\nvocabulary V.\nThe \ufb01rst module in Fig. 9.14 is a linear layer, whose job is to project from the\noutput hL\nN, which represents the output token embedding at position Nfrom the \ufb01nal\nblock L, (hence of shape [1\u0002d]) to the logit vector, or score vector, that will have a logit\nsingle score for each of the jVjpossible words in the vocabulary V. The logit vector\nuis thus of dimensionality 1 \u0002jVj.\nThis linear layer can be learned, but more commonly we tie this matrix to (the\ntranspose of) the embedding matrix E. Recall that in weight tying , we use the weight tying\nsame weights for two different matrices in the model. Thus at the input stage of the\ntransformer the embedding matrix (of shape [jVj\u0002d]) is used to map from a one-hot\nvector over the vocabulary (of shape [1\u0002jVj]) to an embedding (of shape [1\u0002d]).\nAnd then in the language model head, ET, the transpose of the embedding matrix (of\nshape [d\u0002jVj]) is used to map back from an embedding (shape [1\u0002d]) to a vector\nover the vocabulary (shape [1 \u0002jVj]). In the learning process, Ewill be optimized to\nbe good at doing both of these mappings. We therefore sometimes call the transpose\nETtheunembedding layer because it is performing this reverse mapping. unembedding\nA softmax layer turns the logits uinto the probabilities yover the vocabulary.\nu=hL\nNET(9.44)\ny=softmax (u) (9.45)\nWe can use these probabilities to do things like help assign a probability to a\ngiven text. But the most important usage to generate text, which we do by sampling 9.5 \u2022 T HELANGUAGE MODELING HEAD 17\na word from these probabilities y. We might sample the highest probability word\n(\u2018greedy\u2019 decoding), or use another of the sampling methods we\u2019ll introduce in Sec-\ntion??. In either case, whatever entry ykwe choose from the probability vector y,\nwe generate the word that has that index k.\nwiSample token togenerate at position i+1\nfeedforwardlayer normattentionlayer norm\nU\nInput tokenLanguageModelingHead\nInputEncoding\nEi+\u2026logits\nfeedforwardlayer normattentionlayer normLayer 1Layer 2h1i  =  x2ix1ih2i  =  x3ifeedforwardlayer normattentionlayer normhLi  hL-1i  =  xLiy1y2y|V|\u2026Token probabilitiesu1u2u|V|\u2026softmaxwi+1\nLayer L\nFigure 9.15 A transformer language model (decoder-only), stacking transformer blocks\nand mapping from an input token wito to a predicted next token wi+1.\nFig. 9.15 shows the total stacked architecture for one token i. Note that the input\nto each transformer layer xiis the same as the output from the preceding layer h\u0000\ni.\nNow that we see all these transformer layers spread out on the page, we can point\nout another useful feature of the unembedding layer: as a tool for interpretability of\nthe internals of the transformer that we call the logit lens (Nostalgebraist, 2020). logit lens\nWe can take a vector from any layer of the transformer and, pretending that it is\nthe pre\ufb01nal embedding, simply multiply it by the unembedding layer to get logits,\nand compute a softmax to see the distribution over words that that vector might\nbe representing. This can be a useful window into the internal representations of\nthe model. Since the network wasn\u2019t trained to make the internal representations\nfunction in this way, the logit lens doesn\u2019t always work perfectly, but this can still\nbe a useful trick.\nA terminological note before we conclude: You will sometimes see a trans-\nformer used for this kind of unidirectional causal language model called a decoder-\nonly model . This is because this model constitutes roughly half of the encoder-decoder-only\nmodel 18 CHAPTER 9 \u2022 T HETRANSFORMER\ndecoder model for transformers that we\u2019ll see how to apply to machine translation\nin Chapter 13. (Confusingly, the original introduction of the transformer had an\nencoder-decoder architecture, and it was only later that the standard paradigm for\ncausal language model was de\ufb01ned by using only the decoder part of this original\narchitecture).\n9.6 Summary\nThis chapter has introduced the transformer and its components for the task of lan-\nguage modeling. We\u2019ll continue the task of language modeling including issues like\ntraining and sampling in the next chapter.\nHere\u2019s a summary of the main points that we covered:\n\u2022 Transformers are non-recurrent networks based on multi-head attention , a\nkind of self-attention . A multi-head attention computation takes an input\nvector xiand maps it to an output aiby adding in vectors from prior tokens,\nweighted by how relevant they are for the processing of the current word.\n\u2022 Atransformer block consists of a residual stream in which the input from\nthe prior layer is passed up to the next layer, with the output of different com-\nponents added to it. These components include a multi-head attention layer\nfollowed by a feedforward layer , each preceded by layer normalizations .\nTransformer blocks are stacked to make deeper and more powerful networks.\n\u2022 The input to a transformer is a computing by adding an embedding (computed\nwith an embedding matrix ) to a positional encoding that represents the se-\nquential position of the token in the window.\n\u2022 Language models can be built out of stacks of transformer blocks, with a\nlanguage model head at the top, which applies an unembedding matrix to\nthe output Hof the top layer to generate the logits , which are then passed\nthrough a softmax to generate word probabilities.\n\u2022 Transformer-based language models have a wide context window (as wide\nas 32768 tokens for very large models) allowing them to draw on enormous\namounts of context to predict upcoming words.\nBibliographical and Historical Notes\nThe transformer (Vaswani et al., 2017) was developed drawing on two lines of prior\nresearch: self-attention andmemory networks .\nEncoder-decoder attention, the idea of using a soft weighting over the encodings\nof input words to inform a generative decoder (see Chapter 13) was developed by\nGraves (2013) in the context of handwriting generation, and Bahdanau et al. (2015)\nfor MT. This idea was extended to self-attention by dropping the need for separate\nencoding and decoding sequences and instead seeing attention as a way of weighting\nthe tokens in collecting information passed from lower layers to higher layers (Ling\net al., 2015; Cheng et al., 2016; Liu et al., 2016).\nOther aspects of the transformer, including the terminology of key, query, and\nvalue, came from memory networks , a mechanism for adding an external read- BIBLIOGRAPHICAL AND HISTORICAL NOTES 19\nwrite memory to networks, by using an embedding of a query to match keys rep-\nresenting content in an associative memory (Sukhbaatar et al., 2015; Weston et al.,\n2015; Graves et al., 2014).\nMORE HISTORY TBD IN NEXT DRAFT. 20 Chapter 9 \u2022 The Transformer\nBa, J. L., J. R. Kiros, and G. E. Hinton. 2016. Layer normal-\nization. NeurIPS workshop .\nBahdanau, D., K. H. Cho, and Y . Bengio. 2015. Neural ma-\nchine translation by jointly learning to align and translate.\nICLR 2015 .\nCheng, J., L. Dong, and M. Lapata. 2016. Long short-term\nmemory-networks for machine reading. EMNLP .\nElhage, N., N. Nanda, C. Olsson, T. Henighan, N. Joseph,\nB. Mann, A. Askell, Y . Bai, A. Chen, T. Conerly, N. Das-\nSarma, D. Drain, D. Ganguli, Z. Hat\ufb01eld-Dodds, D. Her-\nnandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse,\nD. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCan-\ndlish, and C. Olah. 2021. A mathematical framework for\ntransformer circuits. White paper.\nGraves, A. 2013. Generating sequences with recurrent neural\nnetworks. ArXiv.\nGraves, A., G. Wayne, and I. Danihelka. 2014. Neural Tur-\ning machines. ArXiv.\nLing, W., C. Dyer, A. W. Black, I. Trancoso, R. Fermandez,\nS. Amir, L. Marujo, and T. Lu \u00b4\u0131s. 2015. Finding function\nin form: Compositional character models for open vocab-\nulary word representation. EMNLP .\nLiu, Y ., C. Sun, L. Lin, and X. Wang. 2016. Learning natural\nlanguage inference using bidirectional LSTM model and\ninner-attention. ArXiv.\nNostalgebraist. 2020. Interpreting gpt: the logit lens. White\npaper.\nSukhbaatar, S., A. Szlam, J. Weston, and R. Fergus. 2015.\nEnd-to-end memory networks. NeurIPS .\nUszkoreit, J. 2017. Transformer: A novel neural network ar-\nchitecture for language understanding. Google Research\nblog post, Thursday August 31, 2017.\nVaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones,\nA. N. Gomez, \u0141. Kaiser, and I. Polosukhin. 2017. Atten-\ntion is all you need. NeurIPS .\nWeston, J., S. Chopra, and A. Bordes. 2015. Memory net-\nworks. ICLR 2015 .", "transformer24aug.pdf": "TransformersIntroduction to Transformers LLMs are built out of transformersTransformer: a specific kind of network architecture, like a fancier feedforward network, but based on attentionProvided proper attribution is provided, Google hereby grants permission toreproduce the tables and \ufb01gures in this paper solely for use in journalistic orscholarly works.Attention Is All You NeedAshish Vaswani\u21e4Google Brainavaswani@google.comNoam Shazeer\u21e4Google Brainnoam@google.comNiki Parmar\u21e4Google Researchnikip@google.comJakob Uszkoreit\u21e4Google Researchusz@google.comLlion Jones\u21e4Google Researchllion@google.comAidan N. Gomez\u21e4\u2020University of Torontoaidan@cs.toronto.edu\u0141ukasz Kaiser\u21e4Google Brainlukaszkaiser@google.comIllia Polosukhin\u21e4\u2021illia.polosukhin@gmail.comAbstractThe dominant sequence transduction models are based on complex recurrent orconvolutional neural networks that include an encoder and a decoder. The bestperforming models also connect the encoder and decoder through an attentionmechanism. We propose a new simple network architecture, the Transformer,based solely on attention mechanisms, dispensing with recurrence and convolutionsentirely. Experiments on two machine translation tasks show these models tobe superior in quality while being more parallelizable and requiring signi\ufb01cantlyless time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, includingensembles, by over 2 BLEU. On the WMT 2014 English-to-French translation task,our model establishes a new single-model state-of-the-art BLEU score of 41.8 aftertraining for 3.5 days on eight GPUs, a small fraction of the training costs of thebest models from the literature. We show that the Transformer generalizes well toother tasks by applying it successfully to English constituency parsing both withlarge and limited training data.\u21e4Equal contribution. Listing order is random. Jakob proposed replacing RNNs with self-attention and startedthe effort to evaluate this idea. Ashish, with Illia, designed and implemented the \ufb01rst Transformer models andhas been crucially involved in every aspect of this work. Noam proposed scaled dot-product attention, multi-headattention and the parameter-free position representation and became the other person involved in nearly everydetail. Niki designed, implemented, tuned and evaluated countless model variants in our original codebase andtensor2tensor. Llion also experimented with novel model variants, was responsible for our initial codebase, andef\ufb01cient inference and visualizations. Lukasz and Aidan spent countless long days designing various parts of andimplementing tensor2tensor, replacing our earlier codebase, greatly improving results and massively acceleratingour research.\u2020Work performed while at Google Brain.\u2021Work performed while at Google Research.31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.arXiv:1706.03762v7  [cs.CL]  2 Aug 2023 A very approximate timeline1990 Static Word Embeddings2003 Neural Language Model2008 Multi-Task Learning2015 Attention2017 Transformer2018 Contextual Word Embeddings and Pretraining2019 Prompting TransformersAttention Instead of starting with the big picture\nStackedTransformerBlocksSolongandthanksforlongandthanksforNext tokenall\u2026\u2026\n\u2026\nU\nInput tokensx1x2LanguageModelingHead\nx3x4x5InputEncoding\nE1+\nE2+\nE3+\nE4+\nE5+\u2026\u2026\u2026\u2026\u2026\nU\nU\nU\nU\u2026logitslogitslogitslogitslogits\nStackedTransformerBlocksSolongandthanksforlongandthanksforNext tokenall\u2026\u2026\n\u2026\nU\nInput tokensx1x2LanguageModelingHead\nx3x4x5InputEncoding\nE1+\nE2+\nE3+\nE4+\nE5+\u2026\u2026\u2026\u2026\u2026\nU\nU\nU\nU\u2026logitslogitslogitslogitslogitsLet's consider the embeddings for an individual word from a particular layer Problem with static embeddings (word2vec)They are static!  The embedding for a word doesn't reflect how its meaning changes in context. The chicken didn't cross the road because it was too tiredWhat is the meaning represented in the static embedding for \"it\"?  Contextual Embeddings\u2022Intuition: a representation of meaning of a word should be different in different contexts!\u2022Contextual Embedding: each word has a different vector that expresses different meanings depending on the surrounding words\u2022How to compute contextual embeddings?\u2022Attention Contextual EmbeddingsThe chicken didn't cross the road because itWhat should be the properties of \"it\"?The chicken didn't cross the road because it was too tiredThe chicken didn't cross the road because it was too wideAt this point in the sentence, it's probably referring to either the chicken or the street  Intuition of attentionBuild up the contextual embedding from a word by selectively integrating information from all the neighboring wordsWe say that a word \"attends to\" some neighboring words more than others Intuition of attention: testThechickendidn\u2019tcrosstheroadbecauseitwastootiredThechickendidn\u2019tcrosstheroadbecauseitwastootiredLayer k+1Layer kself-attention distributioncolumns corresponding to input tokens Attention definitionA mechanism for helping compute the embedding for a token by selectively attending to and integrating information from surrounding tokens (at the previous layer).More formally: a method for doing a weighted sum of vectors. Attention is left-to-right\nattentionattentionSelf-AttentionLayerattentionattentionattentiona1a2a3a4a5x3x4x5x1x2 Simplified version of attention: a sum of prior words weighted by their similarity with the current wordGiven a sequence of token embeddings: x1 x2   x3   x4   x5   x6   x7   xiProduce: ai = a weighted sum of x1 through x7 (and xi)Weighted by their similarity to xi10.1\u2022THETRANSFORMER:ASELF-ATTENTIONNETWORK5\nSelf-AttentionLayerx1a1\nx2a2a3a4a5\nx3x4x5Figure 10.2Information \ufb02ow in a causal (or masked) self-attention model. In processingeach element of the sequence, the model attends to all the inputs up to, and including, thecurrent one. Unlike RNNs, the computations at each time step are independent of all theother steps and therefore can be performed in parallel.10.1.3 Self-attention more formallyWe\u2019ve given the intuition of self-attention (as a way to compute representations of aword at a given layer by integrating information from words at the previous layer)and we\u2019ve de\ufb01ned context as all the prior words in the input. Let\u2019s now introducethe self-attention computation itself.The core intuition of attention is the idea ofcomparingan item of interest to acollection of other items in a way that reveals their relevance in the current context.In the case of self-attention for language, the set of comparisons are to other words(or tokens) within a given sequence. The result of these comparisons is then used tocompute an output sequence for the current input sequence. For example, returningto Fig.10.2, the computation ofa3is based on a set of comparisons between theinputx3and its preceding elementsx1andx2, and tox3itself.How shall we compare words to other words? Since our representations forwords are vectors, we\u2019ll make use of our old friend thedot productthat we usedfor computing word similarity in Chapter 6, and also played a role in attention inChapter 9. Let\u2019s refer to the result of this comparison between wordsiandjas ascore (we\u2019ll be updating this equation to add attention to the computation of thisscore):Verson 1:score(xi,xj)=xi\u00b7xj(10.4)The result of a dot product is a scalar value ranging from\u0000\u2022to\u2022, the largerthe value the more similar the vectors that are being compared. Continuing with ourexample, the \ufb01rst step in computingy3would be to compute three scores:x3\u00b7x1,x3\u00b7x2andx3\u00b7x3. Then to make effective use of these scores, we\u2019ll normalize themwith a softmax to create a vector of weights,aij, that indicates the proportionalrelevance of each input to the input elementithat is the current focus of attention.aij=softmax(score(xi,xj))8j\uf8ffi(10.5)=exp(score(xi,xj))Pik=1exp(score(xi,xk))8j\uf8ffi(10.6)Of course, the softmax weight will likely be highest for the current focus elementi, sincevecxiis very similar to itself, resulting in a high dot product. But othercontext words may also be similar toi, and the softmax will also assign some weightto those words.Given the proportional scores ina, we generate an output valueaiby summing10.1\u2022THETRANSFORMER:ASELF-ATTENTIONNETWORK5\nSelf-AttentionLayerx1a1\nx2a2a3a4a5\nx3x4x5Figure 10.2Information \ufb02ow in a causal (or masked) self-attention model. In processingeach element of the sequence, the model attends to all the inputs up to, and including, thecurrent one. Unlike RNNs, the computations at each time step are independent of all theother steps and therefore can be performed in parallel.10.1.3 Self-attention more formallyWe\u2019ve given the intuition of self-attention (as a way to compute representations of aword at a given layer by integrating information from words at the previous layer)and we\u2019ve de\ufb01ned context as all the prior words in the input. Let\u2019s now introducethe self-attention computation itself.The core intuition of attention is the idea ofcomparingan item of interest to acollection of other items in a way that reveals their relevance in the current context.In the case of self-attention for language, the set of comparisons are to other words(or tokens) within a given sequence. The result of these comparisons is then used tocompute an output sequence for the current input sequence. For example, returningto Fig.10.2, the computation ofa3is based on a set of comparisons between theinputx3and its preceding elementsx1andx2, and tox3itself.How shall we compare words to other words? Since our representations forwords are vectors, we\u2019ll make use of our old friend thedot productthat we usedfor computing word similarity in Chapter 6, and also played a role in attention inChapter 9. Let\u2019s refer to the result of this comparison between wordsiandjas ascore (we\u2019ll be updating this equation to add attention to the computation of thisscore):Verson 1:score(xi,xj)=xi\u00b7xj(10.4)The result of a dot product is a scalar value ranging from\u0000\u2022to\u2022, the largerthe value the more similar the vectors that are being compared. Continuing with ourexample, the \ufb01rst step in computingy3would be to compute three scores:x3\u00b7x1,x3\u00b7x2andx3\u00b7x3. Then to make effective use of these scores, we\u2019ll normalize themwith a softmax to create a vector of weights,aij, that indicates the proportionalrelevance of each input to the input elementithat is the current focus of attention.aij=softmax(score(xi,xj))8j\uf8ffi(10.5)=exp(score(xi,xj))Pik=1exp(score(xi,xk))8j\uf8ffi(10.6)Of course, the softmax weight will likely be highest for the current focus elementi, sincevecxiis very similar to itself, resulting in a high dot product. But othercontext words may also be similar toi, and the softmax will also assign some weightto those words.Given the proportional scores ina, we generate an output valueaiby summing6CHAPTER10\u2022TRANSFORMERS ANDLARGELANGUAGEMODELSthe inputs seen so far, each weighted by itsavalue.ai=Xj\uf8ffiaijxj(10.7)The steps embodied in Equations10.4through10.7represent the core of anattention-based approach: a set of comparisons to relevant items in some context,a normalization of those scores to provide a probability distribution, followed by aweighted sum using this distribution. The outputais the result of this straightfor-ward computation over the inputs.This kind of simple attention can be useful, and indeed we saw in Chapter 9how to use this simple idea of attention for LSTM-based encoder-decoder modelsfor machine translation. But transformers allow us to create a more sophisticatedway of representing how words can contribute to the representation of longer inputs.Consider the three different roles that each input embedding plays during the courseof the attention process.\u2022Asthe current focus of attentionwhen being compared to all of the otherpreceding inputs. We\u2019ll refer to this role as aquery.query\u2022In its role asa preceding inputbeing compared to the current focus of atten-tion. We\u2019ll refer to this role as akey.key\u2022And \ufb01nally, as avalueused to compute the output for the current focus ofvalueattention.To capture these three different roles, transformers introduce weight matricesWQ,WK, andWV. These weights will be used to project each input vectorxiintoa representation of its role as a key, query, or value.qi=xiWQ;ki=xiWK;vi=xiWV(10.8)The inputsxand outputsyof transformers, as well as the intermediate vectors afterthe various layers like the attention output vectora, all have the same dimensionality1\u21e5d. We\u2019ll have a dimensiondkfor the key and query vectors, and a separatedimensiondvfor the value vectors. In the original transformer work (Vaswani et al.,2017),dwas 512,dkanddvwere both 64. The shapes of the transform matrices arethenWQ2Rd\u21e5dk,WK2Rd\u21e5dk, andWV2Rd\u21e5dv.Given these projections, the score between a current focus of attention,xi, andan element in the preceding context,xj, consists of a dot product between its queryvectorqiand the preceding element\u2019s key vectorskj. This dot product has the rightshape since both the query and the key are of dimensionality 1\u21e5dk. Let\u2019s updateour previous comparison calculation to re\ufb02ect this, replacing Eq.10.4with Eq.10.9:Verson 2:score(xi,xj)=qi\u00b7kj(10.9)The ensuing softmax calculation resulting inai,jremains the same, but the outputcalculation foraiis now based on a weighted sum over the value vectorsv.ai=Xj\uf8ffiaijvj(10.10)Again, the softmax weightaijwill likely be highest for the current focus elementi, and so the value foryiwill be most in\ufb02uenced byvi. But the model will also payattention to other contextual words if they are similar toi, allowing their values to Intuition of attention: test\nx1  x2  x3  x4  x5  x6  x7   xiThechickendidn\u2019tcrosstheroadbecauseitwastootiredThechickendidn\u2019tcrosstheroadbecauseitwastootiredLayer k+1Layer kself-attention distributioncolumns corresponding to input tokens An Actual Attention Head: slightly more complicatedHigh-level idea: instead of using vectors (like xi and x4) directly, we'll represent 3 separate roles each vector xi plays:\u2022query: As the current element being compared to the preceding inputs. \u2022key: as a preceding input that is being compared to the current element to determine a similarity\u2022value: a value of a preceding element that gets weighted and summed  Thechickendidn\u2019tcrosstheroadbecauseitwastootiredThechickendidn\u2019tcrosstheroadbecauseitwastootiredLayer k+1Layer kself-attention distributioncolumns corresponding to input tokensAttention intuition\nx1  x2  x3  x4  x5  x6  x7   xiquery\nvalues Intuition of attention: \nx1  x2  x3  x4  x5  x6  x7  xiquery\nvalueskvkvkvkvkvkvkvkeyskvThechickendidn\u2019tcrosstheroadbecauseitwastootiredThechickendidn\u2019tcrosstheroadbecauseitwastootiredLayer k+1Layer kself-attention distributioncolumns corresponding to input tokens An Actual Attention Head: slightly more complicatedWe'll use matrices to project each vector xi into a representation of its role as query, key, value:\u2022query: WQ\u2022key: WK\u2022value: WV9.1\u2022ATTENTION5the softmax weight will likely be highest forxi, sincexiis very similar to itself,resulting in a high dot product. But other context words may also be similar toi, andthe softmax will also assign some weight to those words. Then we use these weightsas theavalues in Eq.9.6to compute the weighted sum that is oura3.The simpli\ufb01ed attention in equations9.6\u20139.8demonstrates the attention-basedapproach to computingai: compare thexito prior vectors, normalize those scoresinto a probability distribution used to weight the sum of the prior vector. But nowwe\u2019re ready to remove the simpli\ufb01cations.A single attention head using query, key, and value matricesNow that we\u2019veseen a simple intuition of attention, let\u2019s introduce the actualattention head, theattention headversion of attention that\u2019s used in transformers. (The wordheadis often used inheadtransformers to refer to speci\ufb01c structured layers). The attention head allows us todistinctly represent three different roles that each input embedding plays during thecourse of the attention process:\u2022Asthe current elementbeing compared to the preceding inputs. We\u2019ll refer tothis role as aquery.query\u2022In its role asa preceding inputthat is being compared to the current elementto determine a similarity weight. We\u2019ll refer to this role as akey.key\u2022And \ufb01nally, as avalueof a preceding element that gets weighted and summedvalueup to compute the output for the current element.To capture these three different roles, transformers introduce weight matricesWQ,WK, andWV. These weights will project each input vectorxiinto a represen-tation of its role as a key, query, or value:qi=xiWQ;ki=xiWK;vi=xiWV(9.9)Given these projections, when we are computing the similarity of the current ele-mentxiwith some prior elementxj, we\u2019ll use the dot product between the currentelement\u2019squeryvectorqiand the preceding element\u2019skeyvectorkj. Furthermore,the result of a dot product can be an arbitrarily large (positive or negative) value, andexponentiating large values can lead to numerical issues and loss of gradients duringtraining. To avoid this, we scale the dot product by a factor related to the size of theembeddings, via diving by the square root of the dimensionality of the query andkey vectors (dk). We thus replace the simpli\ufb01ed Eq.9.7with Eq.9.11. The ensuingsoftmax calculation resulting inaijremains the same, but the output calculation foraiis now based on a weighted sum over the value vectorsv(Eq.9.13).Here\u2019s a \ufb01nal set of equations for computing self-attention for a single self-attention output vectoraifrom a single input vectorxi. This version of attentioncomputesaiby summing thevaluesof the prior elements, each weighted by thesimilarity of itskeyto thequeryfrom the current element:qi=xiWQ;kj=xjWK;vj=xjWV(9.10)score(xi,xj)=qi\u00b7kjpdk(9.11)aij=softmax(score(xi,xj))8j\uf8ffi(9.12)ai=Xj\uf8ffiaijvj(9.13)We illustrate this in Fig.9.4for the case of calculating the value of the third outputa3in a sequence. An Actual Attention Head: slightly more complicatedGiven these 3 representation of xiTo compute  similarity of current element xi with some prior element xjWe\u2019ll use dot product between  qi and kj. And instead of summing up xj ,  we'll sum up vj9.1\u2022ATTENTION5the softmax weight will likely be highest forxi, sincexiis very similar to itself,resulting in a high dot product. But other context words may also be similar toi, andthe softmax will also assign some weight to those words. Then we use these weightsas theavalues in Eq.9.6to compute the weighted sum that is oura3.The simpli\ufb01ed attention in equations9.6\u20139.8demonstrates the attention-basedapproach to computingai: compare thexito prior vectors, normalize those scoresinto a probability distribution used to weight the sum of the prior vector. But nowwe\u2019re ready to remove the simpli\ufb01cations.A single attention head using query, key, and value matricesNow that we\u2019veseen a simple intuition of attention, let\u2019s introduce the actualattention head, theattention headversion of attention that\u2019s used in transformers. (The wordheadis often used inheadtransformers to refer to speci\ufb01c structured layers). The attention head allows us todistinctly represent three different roles that each input embedding plays during thecourse of the attention process:\u2022Asthe current elementbeing compared to the preceding inputs. We\u2019ll refer tothis role as aquery.query\u2022In its role asa preceding inputthat is being compared to the current elementto determine a similarity weight. We\u2019ll refer to this role as akey.key\u2022And \ufb01nally, as avalueof a preceding element that gets weighted and summedvalueup to compute the output for the current element.To capture these three different roles, transformers introduce weight matricesWQ,WK, andWV. These weights will project each input vectorxiinto a represen-tation of its role as a key, query, or value:qi=xiWQ;ki=xiWK;vi=xiWV(9.9)Given these projections, when we are computing the similarity of the current ele-mentxiwith some prior elementxj, we\u2019ll use the dot product between the currentelement\u2019squeryvectorqiand the preceding element\u2019skeyvectorkj. Furthermore,the result of a dot product can be an arbitrarily large (positive or negative) value, andexponentiating large values can lead to numerical issues and loss of gradients duringtraining. To avoid this, we scale the dot product by a factor related to the size of theembeddings, via diving by the square root of the dimensionality of the query andkey vectors (dk). We thus replace the simpli\ufb01ed Eq.9.7with Eq.9.11. The ensuingsoftmax calculation resulting inaijremains the same, but the output calculation foraiis now based on a weighted sum over the value vectorsv(Eq.9.13).Here\u2019s a \ufb01nal set of equations for computing self-attention for a single self-attention output vectoraifrom a single input vectorxi. This version of attentioncomputesaiby summing thevaluesof the prior elements, each weighted by thesimilarity of itskeyto thequeryfrom the current element:qi=xiWQ;kj=xjWK;vj=xjWV(9.10)score(xi,xj)=qi\u00b7kjpdk(9.11)aij=softmax(score(xi,xj))8j\uf8ffi(9.12)ai=Xj\uf8ffiaijvj(9.13)We illustrate this in Fig.9.4for the case of calculating the value of the third outputa3in a sequence. Final equations for one attention head9.1\u2022ATTENTION5the softmax weight will likely be highest forxi, sincexiis very similar to itself,resulting in a high dot product. But other context words may also be similar toi, andthe softmax will also assign some weight to those words. Then we use these weightsas theavalues in Eq.9.6to compute the weighted sum that is oura3.The simpli\ufb01ed attention in equations9.6\u20139.8demonstrates the attention-basedapproach to computingai: compare thexito prior vectors, normalize those scoresinto a probability distribution used to weight the sum of the prior vector. But nowwe\u2019re ready to remove the simpli\ufb01cations.A single attention head using query, key, and value matricesNow that we\u2019veseen a simple intuition of attention, let\u2019s introduce the actualattention head, theattention headversion of attention that\u2019s used in transformers. (The wordheadis often used inheadtransformers to refer to speci\ufb01c structured layers). The attention head allows us todistinctly represent three different roles that each input embedding plays during thecourse of the attention process:\u2022Asthe current elementbeing compared to the preceding inputs. We\u2019ll refer tothis role as aquery.query\u2022In its role asa preceding inputthat is being compared to the current elementto determine a similarity weight. We\u2019ll refer to this role as akey.key\u2022And \ufb01nally, as avalueof a preceding element that gets weighted and summedvalueup to compute the output for the current element.To capture these three different roles, transformers introduce weight matricesWQ,WK, andWV. These weights will project each input vectorxiinto a represen-tation of its role as a key, query, or value:qi=xiWQ;ki=xiWK;vi=xiWV(9.9)Given these projections, when we are computing the similarity of the current ele-mentxiwith some prior elementxj, we\u2019ll use the dot product between the currentelement\u2019squeryvectorqiand the preceding element\u2019skeyvectorkj. Furthermore,the result of a dot product can be an arbitrarily large (positive or negative) value, andexponentiating large values can lead to numerical issues and loss of gradients duringtraining. To avoid this, we scale the dot product by a factor related to the size of theembeddings, via diving by the square root of the dimensionality of the query andkey vectors (dk). We thus replace the simpli\ufb01ed Eq.9.7with Eq.9.11. The ensuingsoftmax calculation resulting inaijremains the same, but the output calculation foraiis now based on a weighted sum over the value vectorsv(Eq.9.13).Here\u2019s a \ufb01nal set of equations for computing self-attention for a single self-attention output vectoraifrom a single input vectorxi. This version of attentioncomputesaiby summing thevaluesof the prior elements, each weighted by thesimilarity of itskeyto thequeryfrom the current element:qi=xiWQ;kj=xjWK;vj=xjWV(9.10)score(xi,xj)=qi\u00b7kjpdk(9.11)aij=softmax(score(xi,xj))8j\uf8ffi(9.12)ai=Xj\uf8ffiaijvj(9.13)We illustrate this in Fig.9.4for the case of calculating the value of the third outputa3in a sequence. Calculating the value of a36. Sum the weighted value vectors4. Turn into \ud835\udefci,j weights via softmaxa3\n1. Generate key, query, value vectors2. Compare x3\u2019s query withthe keys for x1, x2, and x3Output of self-attention\nWk\nWvWqx1kqvx3kqvx2kqv\n\u00d7\u00d7\nWkWkWqWqWvWv5. Weigh each value vector\u00f7\u221adk3. Divide score by \u221adk\u00f7\u221adk\u00f7\u221adk\ud835\udefc3,1\ud835\udefc3,2\ud835\udefc3,3 Actual Attention: slightly more complicated\u2022Instead of one attention head, we'll have lots of them!\u2022Intuition: each head might be attending to the context for different purposes\u2022Different linguistic relationships or patterns in the context9.2\u2022TRANSFORMERBLOCKS7shows an intuition.qci=xiWQc;kcj=xjWKc;vcj=xjWVc;8c1\uf8ffc\uf8ffh(9.14)scorec(xi,xj)=qci\u00b7kcjpdk(9.15)acij=softmax(scorec(xi,xj))8j\uf8ffi(9.16)headci=Xj\uf8ffiacijvcj(9.17)ai=(head1\u0000head2...\u0000headh)WO(9.18)MultiHeadAttention(xi,[x1,\u00b7\u00b7\u00b7,xN]) =ai(9.19)The output of each of thehheads is of shape 1\u21e5dv, and so the output of themulti-head layer withhheads consists ofhvectors of shape 1\u21e5dv. These are con-catenated to produce a single output with dimensionality 1\u21e5hdv. Then we use yetanother linear projectionWO2Rhdv\u21e5dto reshape it, resulting in the multi-headattention vectoraiwith the correct output shape[1xd]at each inputi.\naixi-1xixi-2xi-3WK1Head 1WV1WQ1\u2026\u2026WK2Head 2WV2WQ2WK8Head 8WV8WQ8aiWO  [hdv x d][1 x dv ][1 x d]\n[1 x d][1 x hdv ]Project down to dConcatenate OutputsEach headattends di\ufb00erentlyto context\u2026[1 x dv ]\nFigure 9.5The multi-head attention computation for inputxi, producing outputai. A multi-head attentionlayer hashheads, each with its own key, query and value weight matrices. The outputs from each of the headsare concatenated and then projected down tod, thus producing an output of the same size as the input.9.2 Transformer BlocksThe self-attention calculation lies at the core of what\u2019s called a transformer block,which, in addition to the self-attention layer, includes three other kinds of layers: (1)a feedforward layer, (2) residual connections, and (3) normalizing layers (colloqui-ally called \u201clayer norm\u201d).Fig.9.6illustrates a transformer block, sketching a common way of thinkingabout the block that is called theresidual stream(Elhage et al.,2021). In the resid-residual streamual stream viewpoint, we consider the processing of an individual tokenithroughthe transformer block as a single stream ofd-dimensional representations for tokenpositioni. This residual stream starts with the original input vector, and the various Multi-head attention\naixi-1xixi-2xi-3WK1Head 1WV1WQ1\u2026\u2026WK2Head 2WV2WQ2WK8Head 8WV8WQ8aiWO  [hdv x d][1 x dv ][1 x d]\n[1 x d][1 x hdv ]Project down to dConcatenate OutputsEach headattends di\ufb00erentlyto context\u2026[1 x dv ] SummaryAttention is a method for enriching the representation of a token by incorporating contextual informationThe result: the embedding for each word will be different in different contexts!Contextual embeddings: a representation of word meaning in its context.We'll see in the next lecture that attention can also be viewed as a way to move information from one token to another. TransformersAttention TransformersThe Transformer Block StackedTransformerBlocksSolongandthanksforlongandthanksforNext tokenall\u2026\u2026\n\u2026\nU\nInput tokensx1x2LanguageModelingHead\nx3x4x5InputEncoding\nE1+\nE2+\nE3+\nE4+\nE5+\u2026\u2026\u2026\u2026\u2026\nU\nU\nU\nU\u2026logitslogitslogitslogitslogitsReminder: transformer language model The residual stream: each token gets passed up and modified\nLayer Norm\nxi+hi-1\nLayer NormMultiHeadAttentionFeedforward\nxi-1xi+1hihi+1+\u2026\u2026 We'll need nonlinearities, so a feedforward layer\nLayer Norm\nxi+hi-1\nLayer NormMultiHeadAttentionFeedforward\nxi-1xi+1hihi+1+\u2026\u20268CHAPTER9\u2022THETRANSFORMER\nLayer Norm\nxi+hi-1\nLayer NormMultiHeadAttentionFeedforward\nxi-1xi+1hihi+1\n+\u2026\u2026ResidualStream\nFigure 9.6The architecture of a transformer block showing theresidual stream. This\ufb01gure shows theprenormversion of the architecture, in which the layer norms happen beforethe attention and feedforward layers rather than after.components read their input from the residual stream and add their output back intothe stream.The input at the bottom of the stream is an embedding for a token, which hasdimensionalityd. This initial embedding gets passed up (byresidual connections),and is progressively added to by the other components of the transformer: theat-tention layerthat we have seen, and thefeedforward layerthat we will introduce.Before the attention and feedforward layer is a computation called thelayer norm.Thus the initial vector is passed through a layer norm and attention layer, andthe result is added back into the stream, in this case to the original input vectorxi. And then this summed vector is again passed through another layer norm and afeedforward layer, and the output of those is added back into the residual, and we\u2019llusehito refer to the resulting output of the transformer block for tokeni. (In earlierdescriptions the residual stream was often described using a different metaphor asresidual connectionsthat add the input of a component to its output, but the residualstream is a more perspicuous way of visualizing the transformer.)We\u2019ve already seen the attention layer, so let\u2019s now introduce the feedforwardand layer norm computations in the context of processing a single inputxiat tokenpositioni.Feedforward layerThe feedforward layer is a fully-connected 2-layer network,i.e., one hidden layer, two weight matrices, as introduced in Chapter 7. The weightsare the same for each token positioni, but are different from layer to layer. Itis common to make the dimensionalitydffof the hidden layer of the feedforwardnetwork be larger than the model dimensionalityd. (For example in the originaltransformer model,d=512 anddff=2048.)FFN(xi)=ReLU(xiW1+b1)W2+b2(9.20)Layer NormAt two stages in the transformer block wenormalizethe vector (Baet al.,2016). This process, calledlayer norm(short for layer normalization), is onelayer norm Layer norm: the vector xi is normalized twice\nLayer Norm\nxi+hi-1\nLayer NormMultiHeadAttentionFeedforward\nxi-1xi+1hihi+1+\u2026\u2026 Layer NormLayer norm is a variation of the z-score from statistics, applied to a single vec- tor in a hidden layer 9.2\u2022TRANSFORMERBLOCKS9of many forms of normalization that can be used to improve training performancein deep neural networks by keeping the values of a hidden layer in a range thatfacilitates gradient-based training.Layer norm is a variation of thez-scorefrom statistics, applied to a single vec-tor in a hidden layer. That is, the term layer norm is a bit confusing; layer normisnotapplied to an entire transformer layer, but just to the embedding vector of asingle token. Thus the input to layer norm is a single vector of dimensionalitydand the output is that vector normalized, again of dimensionalityd. The \ufb01rst step inlayer normalization is to calculate the mean,\u00b5, and standard deviation,s, over theelements of the vector to be normalized. Given an embedding vectorxof dimen-sionalityd, these values are calculated as follows.\u00b5=1ddXi=1xi(9.21)s=vuut1ddXi=1(xi\u0000\u00b5)2(9.22)Given these values, the vector components are normalized by subtracting the meanfrom each and dividing by the standard deviation. The result of this computation isa new vector with zero mean and a standard deviation of one.\u02c6x=(x\u0000\u00b5)s(9.23)Finally, in the standard implementation of layer normalization, two learnable param-eters,gandb, representing gain and offset values, are introduced.LayerNorm(x)=g(x\u0000\u00b5)s+b(9.24)Putting it all togetherThe function computed by a transformer block can be ex-pressed by breaking it down with one equation for each component computation,usingt(of shape[1\u21e5d]) to stand for transformer and superscripts to demarcateeach computation inside the block:t1i=LayerNorm(xi)(9.25)t2i=MultiHeadAttention(t1i,\u21e5x11,\u00b7\u00b7\u00b7,x1N\u21e4)(9.26)t3i=t2i+xi(9.27)t4i=LayerNorm(t3i)(9.28)t5i=FFN(t4i)(9.29)hi=t5i+t3i(9.30)Notice that the only component that takes as input information from other tokens(other residual streams) is multi-head attention, which (as we see from (9.27)) looksat all the neighboring tokens in the context. The output from attention, however, isthen added into this token\u2019s embedding stream. In fact,Elhage et al.(2021) show thatwe can view attention heads as literally moving information from the residual streamof a neighboring token into the current stream. The high-dimensional embeddingspace at each position thus contains information about the current token and aboutneighboring tokens, albeit in different subspaces of the vector space. Fig.9.7showsa visualization of this movement.9.2\u2022TRANSFORMERBLOCKS9of many forms of normalization that can be used to improve training performancein deep neural networks by keeping the values of a hidden layer in a range thatfacilitates gradient-based training.Layer norm is a variation of thez-scorefrom statistics, applied to a single vec-tor in a hidden layer. That is, the term layer norm is a bit confusing; layer normisnotapplied to an entire transformer layer, but just to the embedding vector of asingle token. Thus the input to layer norm is a single vector of dimensionalitydand the output is that vector normalized, again of dimensionalityd. The \ufb01rst step inlayer normalization is to calculate the mean,\u00b5, and standard deviation,s, over theelements of the vector to be normalized. Given an embedding vectorxof dimen-sionalityd, these values are calculated as follows.\u00b5=1ddXi=1xi(9.21)s=vuut1ddXi=1(xi\u0000\u00b5)2(9.22)Given these values, the vector components are normalized by subtracting the meanfrom each and dividing by the standard deviation. The result of this computation isa new vector with zero mean and a standard deviation of one.\u02c6x=(x\u0000\u00b5)s(9.23)Finally, in the standard implementation of layer normalization, two learnable param-eters,gandb, representing gain and offset values, are introduced.LayerNorm(x)=g(x\u0000\u00b5)s+b(9.24)Putting it all togetherThe function computed by a transformer block can be ex-pressed by breaking it down with one equation for each component computation,usingt(of shape[1\u21e5d]) to stand for transformer and superscripts to demarcateeach computation inside the block:t1i=LayerNorm(xi)(9.25)t2i=MultiHeadAttention(t1i,\u21e5x11,\u00b7\u00b7\u00b7,x1N\u21e4)(9.26)t3i=t2i+xi(9.27)t4i=LayerNorm(t3i)(9.28)t5i=FFN(t4i)(9.29)hi=t5i+t3i(9.30)Notice that the only component that takes as input information from other tokens(other residual streams) is multi-head attention, which (as we see from (9.27)) looksat all the neighboring tokens in the context. The output from attention, however, isthen added into this token\u2019s embedding stream. In fact,Elhage et al.(2021) show thatwe can view attention heads as literally moving information from the residual streamof a neighboring token into the current stream. The high-dimensional embeddingspace at each position thus contains information about the current token and aboutneighboring tokens, albeit in different subspaces of the vector space. Fig.9.7showsa visualization of this movement.9.2\u2022TRANSFORMERBLOCKS9of many forms of normalization that can be used to improve training performancein deep neural networks by keeping the values of a hidden layer in a range thatfacilitates gradient-based training.Layer norm is a variation of thez-scorefrom statistics, applied to a single vec-tor in a hidden layer. That is, the term layer norm is a bit confusing; layer normisnotapplied to an entire transformer layer, but just to the embedding vector of asingle token. Thus the input to layer norm is a single vector of dimensionalitydand the output is that vector normalized, again of dimensionalityd. The \ufb01rst step inlayer normalization is to calculate the mean,\u00b5, and standard deviation,s, over theelements of the vector to be normalized. Given an embedding vectorxof dimen-sionalityd, these values are calculated as follows.\u00b5=1ddXi=1xi(9.21)s=vuut1ddXi=1(xi\u0000\u00b5)2(9.22)Given these values, the vector components are normalized by subtracting the meanfrom each and dividing by the standard deviation. The result of this computation isa new vector with zero mean and a standard deviation of one.\u02c6x=(x\u0000\u00b5)s(9.23)Finally, in the standard implementation of layer normalization, two learnable param-eters,gandb, representing gain and offset values, are introduced.LayerNorm(x)=g(x\u0000\u00b5)s+b(9.24)Putting it all togetherThe function computed by a transformer block can be ex-pressed by breaking it down with one equation for each component computation,usingt(of shape[1\u21e5d]) to stand for transformer and superscripts to demarcateeach computation inside the block:t1i=LayerNorm(xi)(9.25)t2i=MultiHeadAttention(t1i,\u21e5x11,\u00b7\u00b7\u00b7,x1N\u21e4)(9.26)t3i=t2i+xi(9.27)t4i=LayerNorm(t3i)(9.28)t5i=FFN(t4i)(9.29)hi=t5i+t3i(9.30)Notice that the only component that takes as input information from other tokens(other residual streams) is multi-head attention, which (as we see from (9.27)) looksat all the neighboring tokens in the context. The output from attention, however, isthen added into this token\u2019s embedding stream. In fact,Elhage et al.(2021) show thatwe can view attention heads as literally moving information from the residual streamof a neighboring token into the current stream. The high-dimensional embeddingspace at each position thus contains information about the current token and aboutneighboring tokens, albeit in different subspaces of the vector space. Fig.9.7showsa visualization of this movement. Putting together a single transformer block9.2\u2022TRANSFORMERBLOCKS9of many forms of normalization that can be used to improve training performancein deep neural networks by keeping the values of a hidden layer in a range thatfacilitates gradient-based training.Layer norm is a variation of thez-scorefrom statistics, applied to a single vec-tor in a hidden layer. That is, the term layer norm is a bit confusing; layer normisnotapplied to an entire transformer layer, but just to the embedding vector of asingle token. Thus the input to layer norm is a single vector of dimensionalitydand the output is that vector normalized, again of dimensionalityd. The \ufb01rst step inlayer normalization is to calculate the mean,\u00b5, and standard deviation,s, over theelements of the vector to be normalized. Given an embedding vectorxof dimen-sionalityd, these values are calculated as follows.\u00b5=1ddXi=1xi(9.21)s=vuut1ddXi=1(xi\u0000\u00b5)2(9.22)Given these values, the vector components are normalized by subtracting the meanfrom each and dividing by the standard deviation. The result of this computation isa new vector with zero mean and a standard deviation of one.\u02c6x=(x\u0000\u00b5)s(9.23)Finally, in the standard implementation of layer normalization, two learnable param-eters,gandb, representing gain and offset values, are introduced.LayerNorm(x)=g(x\u0000\u00b5)s+b(9.24)Putting it all togetherThe function computed by a transformer block can be ex-pressed by breaking it down with one equation for each component computation,usingt(of shape[1\u21e5d]) to stand for transformer and superscripts to demarcateeach computation inside the block:t1i=LayerNorm(xi)(9.25)t2i=MultiHeadAttention(t1i,\u21e5x11,\u00b7\u00b7\u00b7,x1N\u21e4)(9.26)t3i=t2i+xi(9.27)t4i=LayerNorm(t3i)(9.28)t5i=FFN(t4i)(9.29)hi=t5i+t3i(9.30)Notice that the only component that takes as input information from other tokens(other residual streams) is multi-head attention, which (as we see from (9.27)) looksat all the neighboring tokens in the context. The output from attention, however, isthen added into this token\u2019s embedding stream. In fact,Elhage et al.(2021) show thatwe can view attention heads as literally moving information from the residual streamof a neighboring token into the current stream. The high-dimensional embeddingspace at each position thus contains information about the current token and aboutneighboring tokens, albeit in different subspaces of the vector space. Fig.9.7showsa visualization of this movement.Layer Norm\nxi+hi-1\nLayer NormMultiHeadAttentionFeedforward\nxi-1xi+1hihi+1+\u2026\u2026 A transformer is a stack of these blocksso all the vectors are of the same dimensionality d\nLayer Norm\nxi+hi-1\nLayer NormMultiHeadAttentionFeedforward\nxi-1xi+1hihi+1+\u2026\u2026Layer Norm\nxi+hi-1\nLayer NormMultiHeadAttentionFeedforward\nxi-1xi+1hihi+1+\u2026\u2026\nBlock 1Block 2 Residual streams and attentionNotice that all  parts of the transformer block apply to 1 residual stream (1 token).Except attention, which takes information from other tokens Elhage et al. (2021) show that we can view attention heads as literally moving information from the residual stream of a neighboring token into the current stream .\nToken Aresidual streamToken Bresidual stream TransformersThe Transformer Block TransformersParallelizing Attention Computation Parallelizing computation using XFor attention/transformer block we've been computing a single output at a single time step i in a single residual stream. But we can pack the  N tokens of the input sequence into a single matrix X of size [N \u00d7 d]. Each row of X is the embedding of one token of the input. X can have 1K-32K rows, each of the dimensionality of the embedding d (the model dimension)9.3\u2022PARALLELIZING COMPUTATION USING A SINGLE MATRIXX11dimension).Parallelizing attentionLet\u2019s \ufb01rst see this for a single attention head and then turnto multiple heads, and then add in the rest of the components in the transformerblock. For one head we multiplyXby the key, query, and value matricesWQofshape[d\u21e5dk],WKof shape[d\u21e5dk], andWVof shape[d\u21e5dv], to produce matricesQof shape[N\u21e5dk],K2RN\u21e5dk, andV2RN\u21e5dv, containing all the key, query, andvalue vectors:Q=XWQ;K=XWK;V=XWV(9.31)Given these matrices we can compute all the requisite query-key comparisons simul-taneously by multiplyingQandK|in a single matrix multiplication. The product isof shapeN\u21e5N, visualized in Fig.9.9.q1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3NNq1\u2022k2q1\u2022k3q1\u2022k4q2\u2022k3q2\u2022k4q3\u2022k4Figure 9.8TheN\u21e5NQK|matrix showing how it computes allqi\u00b7kjcomparisons in asingle matrix multiple.Once we have thisQK|matrix, we can very ef\ufb01ciently scale these scores, takethe softmax, and then multiply the result byVresulting in a matrix of shapeN\u21e5d:a vector embedding representation for each token in the input. We\u2019ve reduced theentire self-attention step for an entire sequence ofNtokens for one head to thefollowing computation:A=softmax\u2713mask\u2713QK|pdk\u25c6\u25c6V(9.32)Masking out the futureYou may have noticed that we introduced a mask functionin Eq.9.32above. This is because the self-attention computation as we\u2019ve describedit has a problem: the calculation inQK|results in a score for each query valueto every key value,including those that follow the query. This is inappropriate inthe setting of language modeling: guessing the next word is pretty simple if youalready know it! To \ufb01x this, the elements in the upper-triangular portion of thematrix are zeroed out (set to\u0000\u2022), thus eliminating any knowledge of words thatfollow in the sequence. This is done in practice by adding a mask matrixMinwhichMij=\u0000\u20228j>i(i.e. for the upper-triangular portion) andMij=0 otherwise.Fig.9.9shows the resulting maskedQK|matrix. (we\u2019ll see in Chapter 11 how tomake use of words in the future for tasks that need it).Fig.9.10shows a schematic of all the computations for a single attention headparallelized in matrix form.Fig.9.8and Fig.9.9also make it clear that attention is quadratic in the lengthof the input, since at each layer we need to compute dot products between each pairof tokens in the input. This makes it expensive to compute attention over very longdocuments (like entire novels). Nonetheless modern large language models manageto use quite long contexts of thousands or tens of thousands of tokens. QKTNow can do a single matrix multiply to combine Q and KTq1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3NNq1\u2022k2q1\u2022k3q1\u2022k4q2\u2022k3q2\u2022k4q3\u2022k4 Parallelizing attention\u2022Scale the  scores, take the softmax, and then multiply the result by V resulting in a matrix of shape N \u00d7 d\u2022An attention vector for each input token9.3\u2022PARALLELIZING COMPUTATION USING A SINGLE MATRIXX11dimension).Parallelizing attentionLet\u2019s \ufb01rst see this for a single attention head and then turnto multiple heads, and then add in the rest of the components in the transformerblock. For one head we multiplyXby the key, query, and value matricesWQofshape[d\u21e5dk],WKof shape[d\u21e5dk], andWVof shape[d\u21e5dv], to produce matricesQof shape[N\u21e5dk],K2RN\u21e5dk, andV2RN\u21e5dv, containing all the key, query, andvalue vectors:Q=XWQ;K=XWK;V=XWV(9.31)Given these matrices we can compute all the requisite query-key comparisons simul-taneously by multiplyingQandK|in a single matrix multiplication. The product isof shapeN\u21e5N, visualized in Fig.9.9.q1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3NNq1\u2022k2q1\u2022k3q1\u2022k4q2\u2022k3q2\u2022k4q3\u2022k4Figure 9.8TheN\u21e5NQK|matrix showing how it computes allqi\u00b7kjcomparisons in asingle matrix multiple.Once we have thisQK|matrix, we can very ef\ufb01ciently scale these scores, takethe softmax, and then multiply the result byVresulting in a matrix of shapeN\u21e5d:a vector embedding representation for each token in the input. We\u2019ve reduced theentire self-attention step for an entire sequence ofNtokens for one head to thefollowing computation:A=softmax\u2713mask\u2713QK|pdk\u25c6\u25c6V(9.32)Masking out the futureYou may have noticed that we introduced a mask functionin Eq.9.32above. This is because the self-attention computation as we\u2019ve describedit has a problem: the calculation inQK|results in a score for each query valueto every key value,including those that follow the query. This is inappropriate inthe setting of language modeling: guessing the next word is pretty simple if youalready know it! To \ufb01x this, the elements in the upper-triangular portion of thematrix are zeroed out (set to\u0000\u2022), thus eliminating any knowledge of words thatfollow in the sequence. This is done in practice by adding a mask matrixMinwhichMij=\u0000\u20228j>i(i.e. for the upper-triangular portion) andMij=0 otherwise.Fig.9.9shows the resulting maskedQK|matrix. (we\u2019ll see in Chapter 11 how tomake use of words in the future for tasks that need it).Fig.9.10shows a schematic of all the computations for a single attention headparallelized in matrix form.Fig.9.8and Fig.9.9also make it clear that attention is quadratic in the lengthof the input, since at each layer we need to compute dot products between each pairof tokens in the input. This makes it expensive to compute attention over very longdocuments (like entire novels). Nonetheless modern large language models manageto use quite long contexts of thousands or tens of thousands of tokens. Masking out the future\u2022What is this mask function?QKT has a score for each query dot every key, including those that follow the query.\u2022Guessing the next word is pretty simple if you already know it! 9.3\u2022PARALLELIZING COMPUTATION USING A SINGLE MATRIXX11dimension).Parallelizing attentionLet\u2019s \ufb01rst see this for a single attention head and then turnto multiple heads, and then add in the rest of the components in the transformerblock. For one head we multiplyXby the key, query, and value matricesWQofshape[d\u21e5dk],WKof shape[d\u21e5dk], andWVof shape[d\u21e5dv], to produce matricesQof shape[N\u21e5dk],K2RN\u21e5dk, andV2RN\u21e5dv, containing all the key, query, andvalue vectors:Q=XWQ;K=XWK;V=XWV(9.31)Given these matrices we can compute all the requisite query-key comparisons simul-taneously by multiplyingQandK|in a single matrix multiplication. The product isof shapeN\u21e5N, visualized in Fig.9.9.q1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3NNq1\u2022k2q1\u2022k3q1\u2022k4q2\u2022k3q2\u2022k4q3\u2022k4Figure 9.8TheN\u21e5NQK|matrix showing how it computes allqi\u00b7kjcomparisons in asingle matrix multiple.Once we have thisQK|matrix, we can very ef\ufb01ciently scale these scores, takethe softmax, and then multiply the result byVresulting in a matrix of shapeN\u21e5d:a vector embedding representation for each token in the input. We\u2019ve reduced theentire self-attention step for an entire sequence ofNtokens for one head to thefollowing computation:A=softmax\u2713mask\u2713QK|pdk\u25c6\u25c6V(9.32)Masking out the futureYou may have noticed that we introduced a mask functionin Eq.9.32above. This is because the self-attention computation as we\u2019ve describedit has a problem: the calculation inQK|results in a score for each query valueto every key value,including those that follow the query. This is inappropriate inthe setting of language modeling: guessing the next word is pretty simple if youalready know it! To \ufb01x this, the elements in the upper-triangular portion of thematrix are zeroed out (set to\u0000\u2022), thus eliminating any knowledge of words thatfollow in the sequence. This is done in practice by adding a mask matrixMinwhichMij=\u0000\u20228j>i(i.e. for the upper-triangular portion) andMij=0 otherwise.Fig.9.9shows the resulting maskedQK|matrix. (we\u2019ll see in Chapter 11 how tomake use of words in the future for tasks that need it).Fig.9.10shows a schematic of all the computations for a single attention headparallelized in matrix form.Fig.9.8and Fig.9.9also make it clear that attention is quadratic in the lengthof the input, since at each layer we need to compute dot products between each pairof tokens in the input. This makes it expensive to compute attention over very longdocuments (like entire novels). Nonetheless modern large language models manageto use quite long contexts of thousands or tens of thousands of tokens. Masking out the futureAdd \u2013\u221e to cells in upper triangleThe softmax will turn it to 09.3\u2022PARALLELIZING COMPUTATION USING A SINGLE MATRIXX11dimension).Parallelizing attentionLet\u2019s \ufb01rst see this for a single attention head and then turnto multiple heads, and then add in the rest of the components in the transformerblock. For one head we multiplyXby the key, query, and value matricesWQofshape[d\u21e5dk],WKof shape[d\u21e5dk], andWVof shape[d\u21e5dv], to produce matricesQof shape[N\u21e5dk],K2RN\u21e5dk, andV2RN\u21e5dv, containing all the key, query, andvalue vectors:Q=XWQ;K=XWK;V=XWV(9.31)Given these matrices we can compute all the requisite query-key comparisons simul-taneously by multiplyingQandK|in a single matrix multiplication. The product isof shapeN\u21e5N, visualized in Fig.9.9.q1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3NNq1\u2022k2q1\u2022k3q1\u2022k4q2\u2022k3q2\u2022k4q3\u2022k4Figure 9.8TheN\u21e5NQK|matrix showing how it computes allqi\u00b7kjcomparisons in asingle matrix multiple.Once we have thisQK|matrix, we can very ef\ufb01ciently scale these scores, takethe softmax, and then multiply the result byVresulting in a matrix of shapeN\u21e5d:a vector embedding representation for each token in the input. We\u2019ve reduced theentire self-attention step for an entire sequence ofNtokens for one head to thefollowing computation:A=softmax\u2713mask\u2713QK|pdk\u25c6\u25c6V(9.32)Masking out the futureYou may have noticed that we introduced a mask functionin Eq.9.32above. This is because the self-attention computation as we\u2019ve describedit has a problem: the calculation inQK|results in a score for each query valueto every key value,including those that follow the query. This is inappropriate inthe setting of language modeling: guessing the next word is pretty simple if youalready know it! To \ufb01x this, the elements in the upper-triangular portion of thematrix are zeroed out (set to\u0000\u2022), thus eliminating any knowledge of words thatfollow in the sequence. This is done in practice by adding a mask matrixMinwhichMij=\u0000\u20228j>i(i.e. for the upper-triangular portion) andMij=0 otherwise.Fig.9.9shows the resulting maskedQK|matrix. (we\u2019ll see in Chapter 11 how tomake use of words in the future for tasks that need it).Fig.9.10shows a schematic of all the computations for a single attention headparallelized in matrix form.Fig.9.8and Fig.9.9also make it clear that attention is quadratic in the lengthof the input, since at each layer we need to compute dot products between each pairof tokens in the input. This makes it expensive to compute attention over very longdocuments (like entire novels). Nonetheless modern large language models manageto use quite long contexts of thousands or tens of thousands of tokens.q1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3NN\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e Another point: Attention is quadratic in length9.3\u2022PARALLELIZING COMPUTATION USING A SINGLE MATRIXX11dimension).Parallelizing attentionLet\u2019s \ufb01rst see this for a single attention head and then turnto multiple heads, and then add in the rest of the components in the transformerblock. For one head we multiplyXby the key, query, and value matricesWQofshape[d\u21e5dk],WKof shape[d\u21e5dk], andWVof shape[d\u21e5dv], to produce matricesQof shape[N\u21e5dk],K2RN\u21e5dk, andV2RN\u21e5dv, containing all the key, query, andvalue vectors:Q=XWQ;K=XWK;V=XWV(9.31)Given these matrices we can compute all the requisite query-key comparisons simul-taneously by multiplyingQandK|in a single matrix multiplication. The product isof shapeN\u21e5N, visualized in Fig.9.9.q1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3NNq1\u2022k2q1\u2022k3q1\u2022k4q2\u2022k3q2\u2022k4q3\u2022k4Figure 9.8TheN\u21e5NQK|matrix showing how it computes allqi\u00b7kjcomparisons in asingle matrix multiple.Once we have thisQK|matrix, we can very ef\ufb01ciently scale these scores, takethe softmax, and then multiply the result byVresulting in a matrix of shapeN\u21e5d:a vector embedding representation for each token in the input. We\u2019ve reduced theentire self-attention step for an entire sequence ofNtokens for one head to thefollowing computation:A=softmax\u2713mask\u2713QK|pdk\u25c6\u25c6V(9.32)Masking out the futureYou may have noticed that we introduced a mask functionin Eq.9.32above. This is because the self-attention computation as we\u2019ve describedit has a problem: the calculation inQK|results in a score for each query valueto every key value,including those that follow the query. This is inappropriate inthe setting of language modeling: guessing the next word is pretty simple if youalready know it! To \ufb01x this, the elements in the upper-triangular portion of thematrix are zeroed out (set to\u0000\u2022), thus eliminating any knowledge of words thatfollow in the sequence. This is done in practice by adding a mask matrixMinwhichMij=\u0000\u20228j>i(i.e. for the upper-triangular portion) andMij=0 otherwise.Fig.9.9shows the resulting maskedQK|matrix. (we\u2019ll see in Chapter 11 how tomake use of words in the future for tasks that need it).Fig.9.10shows a schematic of all the computations for a single attention headparallelized in matrix form.Fig.9.8and Fig.9.9also make it clear that attention is quadratic in the lengthof the input, since at each layer we need to compute dot products between each pairof tokens in the input. This makes it expensive to compute attention over very longdocuments (like entire novels). Nonetheless modern large language models manageto use quite long contexts of thousands or tens of thousands of tokens.q1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3NN\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e Attention again\nq1q2q3q4k1k2k3k4QKTQKTv1v2v3v4Vq2\u2022k2q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k2q3\u2022k3\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221eq1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3q1\u2022k2q2\u2022k3q1\u2022k3q3\u2022k4q2\u2022k4q1\u2022k4x=QKT maskedmask=q1\u2022k1q2\u2022k1q4\u2022k1q3\u2022k1q1\u2022k1q1\u2022k1=xa1a2a3a4AQuery Token 1Query Token 2Query Token 3Query Token 4QInput Token 1Input Token 2Input Token 3Input Token 4XxWQ=Value Token 1Value Token 2Value Token 3Value Token 4VxWV=Input Token 1Input Token 2Input Token 3Input Token 4XKey Token 1Key Token 2Key Token 3Key Token 4KxWK=Input Token 1Input Token 2Input Token 3Input Token 4X\nN x dkdk x NN x NN x NN x dvN x dvd x dkd x dkd x dvN x dN x dkN x dN x dkN x dN x dvq1q2q3q4k1k2k3k4QKTQKTv1v2v3v4Vq2\u2022k2q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k2q3\u2022k3\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221eq1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3q1\u2022k2q2\u2022k3q1\u2022k3q3\u2022k4q2\u2022k4q1\u2022k4x=QKT maskedmask=q1\u2022k1q2\u2022k1q4\u2022k1q3\u2022k1q1\u2022k1q1\u2022k1=xa1a2a3a4AQuery Token 1Query Token 2Query Token 3Query Token 4QInput Token 1Input Token 2Input Token 3Input Token 4XxWQ=Value Token 1Value Token 2Value Token 3Value Token 4VxWV=Input Token 1Input Token 2Input Token 3Input Token 4XKey Token 1Key Token 2Key Token 3Key Token 4KxWK=Input Token 1Input Token 2Input Token 3Input Token 4X\nN x dkdk x NN x NN x NN x dvN x dvd x dkd x dkd x dvN x dN x dkN x dN x dkN x dN x dvq1q2q3q4k1k2k3k4QKTQKTv1v2v3v4Vq2\u2022k2q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k2q3\u2022k3\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221eq1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3q1\u2022k2q2\u2022k3q1\u2022k3q3\u2022k4q2\u2022k4q1\u2022k4x=QKT maskedmask=q1\u2022k1q2\u2022k1q4\u2022k1q3\u2022k1q1\u2022k1q1\u2022k1=xa1a2a3a4AQuery Token 1Query Token 2Query Token 3Query Token 4QInput Token 1Input Token 2Input Token 3Input Token 4XxWQ=Value Token 1Value Token 2Value Token 3Value Token 4VxWV=Input Token 1Input Token 2Input Token 3Input Token 4XKey Token 1Key Token 2Key Token 3Key Token 4KxWK=Input Token 1Input Token 2Input Token 3Input Token 4X\nN x dkdk x NN x NN x NN x dvN x dvd x dkd x dkd x dvN x dN x dkN x dN x dkN x dN x dv\nq1q2q3q4k1k2k3k4QKTQKTv1v2v3v4Vq2\u2022k2q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k2q3\u2022k3\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221eq1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3q1\u2022k2q2\u2022k3q1\u2022k3q3\u2022k4q2\u2022k4q1\u2022k4x=QKT maskedmask=q1\u2022k1q2\u2022k1q4\u2022k1q3\u2022k1q1\u2022k1q1\u2022k1=xa1a2a3a4AQuery Token 1Query Token 2Query Token 3Query Token 4QInput Token 1Input Token 2Input Token 3Input Token 4XxWQ=Value Token 1Value Token 2Value Token 3Value Token 4VxWV=Input Token 1Input Token 2Input Token 3Input Token 4XKey Token 1Key Token 2Key Token 3Key Token 4KxWK=Input Token 1Input Token 2Input Token 3Input Token 4X\nN x dkdk x NN x NN x NN x dvN x dvd x dkd x dkd x dvN x dN x dkN x dN x dkN x dN x dvq1q2q3q4k1k2k3k4QKTQKTv1v2v3v4Vq2\u2022k2q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k2q3\u2022k3\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221eq1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3q1\u2022k2q2\u2022k3q1\u2022k3q3\u2022k4q2\u2022k4q1\u2022k4x=QKT maskedmask=q1\u2022k1q2\u2022k1q4\u2022k1q3\u2022k1q1\u2022k1q1\u2022k1=xa1a2a3a4AQuery Token 1Query Token 2Query Token 3Query Token 4QInput Token 1Input Token 2Input Token 3Input Token 4XxWQ=Value Token 1Value Token 2Value Token 3Value Token 4VxWV=Input Token 1Input Token 2Input Token 3Input Token 4XKey Token 1Key Token 2Key Token 3Key Token 4KxWK=Input Token 1Input Token 2Input Token 3Input Token 4X\nN x dkdk x NN x NN x NN x dvN x dvd x dkd x dkd x dvN x dN x dkN x dN x dkN x dN x dv\nq1q2q3q4k1k2k3k4QKTQKTv1v2v3v4Vq2\u2022k2q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k2q3\u2022k3\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221eq1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3q1\u2022k2q2\u2022k3q1\u2022k3q3\u2022k4q2\u2022k4q1\u2022k4x=QKT maskedmask=q1\u2022k1q2\u2022k1q4\u2022k1q3\u2022k1q1\u2022k1q1\u2022k1=xa1a2a3a4AQuery Token 1Query Token 2Query Token 3Query Token 4QInput Token 1Input Token 2Input Token 3Input Token 4XxWQ=Value Token 1Value Token 2Value Token 3Value Token 4VxWV=Input Token 1Input Token 2Input Token 3Input Token 4XKey Token 1Key Token 2Key Token 3Key Token 4KxWK=Input Token 1Input Token 2Input Token 3Input Token 4X\nN x dkdk x NN x NN x NN x dvN x dvd x dkd x dkd x dvN x dN x dkN x dN x dkN x dN x dvq1q2q3q4k1k2k3k4QKTQKTv1v2v3v4Vq2\u2022k2q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k2q3\u2022k3\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221eq1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3q1\u2022k2q2\u2022k3q1\u2022k3q3\u2022k4q2\u2022k4q1\u2022k4x=QKT maskedmask=q1\u2022k1q2\u2022k1q4\u2022k1q3\u2022k1q1\u2022k1q1\u2022k1=xa1a2a3a4AQuery Token 1Query Token 2Query Token 3Query Token 4QInput Token 1Input Token 2Input Token 3Input Token 4XxWQ=Value Token 1Value Token 2Value Token 3Value Token 4VxWV=Input Token 1Input Token 2Input Token 3Input Token 4XKey Token 1Key Token 2Key Token 3Key Token 4KxWK=Input Token 1Input Token 2Input Token 3Input Token 4X\nN x dkdk x NN x NN x NN x dvN x dvd x dkd x dkd x dvN x dN x dkN x dN x dkN x dN x dv Parallelizing Multi-head Attention9.4\u2022THE INPUT:EMBEDDINGS FOR TOKEN AND POSITION13the self-attention outputAof shape [N\u21e5d].Qi=XWQi;Ki=XWKi;Vi=XWVi(9.33)headi=SelfAttention(Qi,Ki,Vi)=softmax\u2713QiKi|pdk\u25c6Vi(9.34)MultiHeadAttention(X)=(head1\u0000head2...\u0000headh)WO(9.35)Putting it all together with the parallel input matrixXThe function computedin parallel by an entire layer ofNtransformer block over the entireNinput tokenscan be expressed as:O=LayerNorm(X+MultiHeadAttention(X))(9.36)H=LayerNorm(O+FFN(O))(9.37)Or we can break it down with one equation for each component computation, usingT(of shape[N\u21e5d]) to stand for transformer and superscripts to demarcate eachcomputation inside the block:T1=MultiHeadAttention(X)(9.38)T2=X+T1(9.39)T3=LayerNorm(T2)(9.40)T4=FFN(T3)(9.41)T5=T4+T3(9.42)H=LayerNorm(T5)(9.43)Here when we use a notation like FFN(T3)we mean that the same FFN is appliedin parallel to each of theNembedding vectors in the window. Similarly, each of theNtokens is normed in parallel in the LayerNorm. Crucially, the input and outputdimensions of transformer blocks are matched so they can be stacked. Since eachtokenxiat the input to the block has dimensionalityd, that means the inputXandoutputHare both of shape[N\u21e5d].9.4 The input: embeddings for token and positionLet\u2019s talk about where the inputXcomes from. Given a sequence ofNtokens (Nisthe context length in tokens), the matrixXof shape[N\u21e5d]has anembeddingforembeddingeach word in the context. The transformer does this by separately computing twoembeddings: an input token embedding, and an input positional embedding.A token embedding, introduced in Chapter 7 and Chapter 8, is a vector of di-mensiondthat will be our initial representation for the input token. (As we passvectors up through the transformer layers in the residual stream, this embeddingrepresentation will change and grow, incorporating context and playing a differentrole depending on the kind of language model we are building.) The set of initialembeddings are stored in the embedding matrixE, which has a row for each of the|V|tokens in the vocabulary. Thus each word is a row vector ofddimensions, andEhas shape[|V|\u21e5d].Given an input token string likeThanks for all thewe \ufb01rst convert the tokensinto vocabulary indices (these were created when we \ufb01rst tokenized the input using Parallelizing Multi-head Attentionor9.4\u2022THE INPUT:EMBEDDINGS FOR TOKEN AND POSITION13the self-attention outputAof shape [N\u21e5d].Qi=XWQi;Ki=XWKi;Vi=XWVi(9.33)headi=SelfAttention(Qi,Ki,Vi)=softmax\u2713QiKi|pdk\u25c6Vi(9.34)MultiHeadAttention(X)=(head1\u0000head2...\u0000headh)WO(9.35)Putting it all together with the parallel input matrixXThe function computedin parallel by an entire layer ofNtransformer block over the entireNinput tokenscan be expressed as:O=LayerNorm(X+MultiHeadAttention(X))(9.36)H=LayerNorm(O+FFN(O))(9.37)Or we can break it down with one equation for each component computation, usingT(of shape[N\u21e5d]) to stand for transformer and superscripts to demarcate eachcomputation inside the block:T1=MultiHeadAttention(X)(9.38)T2=X+T1(9.39)T3=LayerNorm(T2)(9.40)T4=FFN(T3)(9.41)T5=T4+T3(9.42)H=LayerNorm(T5)(9.43)Here when we use a notation like FFN(T3)we mean that the same FFN is appliedin parallel to each of theNembedding vectors in the window. Similarly, each of theNtokens is normed in parallel in the LayerNorm. Crucially, the input and outputdimensions of transformer blocks are matched so they can be stacked. Since eachtokenxiat the input to the block has dimensionalityd, that means the inputXandoutputHare both of shape[N\u21e5d].9.4 The input: embeddings for token and positionLet\u2019s talk about where the inputXcomes from. Given a sequence ofNtokens (Nisthe context length in tokens), the matrixXof shape[N\u21e5d]has anembeddingforembeddingeach word in the context. The transformer does this by separately computing twoembeddings: an input token embedding, and an input positional embedding.A token embedding, introduced in Chapter 7 and Chapter 8, is a vector of di-mensiondthat will be our initial representation for the input token. (As we passvectors up through the transformer layers in the residual stream, this embeddingrepresentation will change and grow, incorporating context and playing a differentrole depending on the kind of language model we are building.) The set of initialembeddings are stored in the embedding matrixE, which has a row for each of the|V|tokens in the vocabulary. Thus each word is a row vector ofddimensions, andEhas shape[|V|\u21e5d].Given an input token string likeThanks for all thewe \ufb01rst convert the tokensinto vocabulary indices (these were created when we \ufb01rst tokenized the input using9.4\u2022THE INPUT:EMBEDDINGS FOR TOKEN AND POSITION13the self-attention outputAof shape [N\u21e5d].Qi=XWQi;Ki=XWKi;Vi=XWVi(9.33)headi=SelfAttention(Qi,Ki,Vi)=softmax\u2713QiKi|pdk\u25c6Vi(9.34)MultiHeadAttention(X)=(head1\u0000head2...\u0000headh)WO(9.35)Putting it all together with the parallel input matrixXThe function computedin parallel by an entire layer ofNtransformer block over the entireNinput tokenscan be expressed as:O=LayerNorm(X+MultiHeadAttention(X))(9.36)H=LayerNorm(O+FFN(O))(9.37)Or we can break it down with one equation for each component computation, usingT(of shape[N\u21e5d]) to stand for transformer and superscripts to demarcate eachcomputation inside the block:T1=MultiHeadAttention(X)(9.38)T2=X+T1(9.39)T3=LayerNorm(T2)(9.40)T4=FFN(T3)(9.41)T5=T4+T3(9.42)H=LayerNorm(T5)(9.43)Here when we use a notation like FFN(T3)we mean that the same FFN is appliedin parallel to each of theNembedding vectors in the window. Similarly, each of theNtokens is normed in parallel in the LayerNorm. Crucially, the input and outputdimensions of transformer blocks are matched so they can be stacked. Since eachtokenxiat the input to the block has dimensionalityd, that means the inputXandoutputHare both of shape[N\u21e5d].9.4 The input: embeddings for token and positionLet\u2019s talk about where the inputXcomes from. Given a sequence ofNtokens (Nisthe context length in tokens), the matrixXof shape[N\u21e5d]has anembeddingforembeddingeach word in the context. The transformer does this by separately computing twoembeddings: an input token embedding, and an input positional embedding.A token embedding, introduced in Chapter 7 and Chapter 8, is a vector of di-mensiondthat will be our initial representation for the input token. (As we passvectors up through the transformer layers in the residual stream, this embeddingrepresentation will change and grow, incorporating context and playing a differentrole depending on the kind of language model we are building.) The set of initialembeddings are stored in the embedding matrixE, which has a row for each of the|V|tokens in the vocabulary. Thus each word is a row vector ofddimensions, andEhas shape[|V|\u21e5d].Given an input token string likeThanks for all thewe \ufb01rst convert the tokensinto vocabulary indices (these were created when we \ufb01rst tokenized the input using TransformersParallelizing Attention Computation TransformersInput and output: Position embeddings and the Language Model Head Token and Position EmbeddingsThe matrix X (of shape [N \u00d7 d]) has an embedding for each word in the context. This embedding is created by adding two distinct embedding for each input\u2022token embedding\u2022positional embedding Token EmbeddingsEmbedding matrix E has shape [|V | \u00d7  d ]. \u2022One row for each of the |V | tokens in the vocabulary. \u2022Each word is a row vector of d dimensionsGiven:  string \"Thanks for all the\"1. Tokenize with BPE and convert into vocab indicesw = [5,4000,10532,2224] 2. Select the corresponding rows from E, each row an embedding\u2022  (row 5, row 4000, row 10532, row 2224).  Position EmbeddingsThere are many methods, but we'll just describe the simplest: absolute position.Goal: learn a position embedding matrix Epos of shape [1 \u00d7 N ]. Start with randomly initialized embeddings\u2022one for each integer up to some maximum length. \u2022i.e., just as we have an embedding for token fish, we\u2019ll have an embedding for position 3 and position 17.\u2022As with word embeddings, these position embeddings are learned along with other parameters during training.  Each x is just the sum of word and position embeddings\nX = CompositeEmbeddings(word + position)Transformer BlockJanet1will2back3Janetwillbackthebillthe4bill5\n+++++PositionEmbeddingsWordEmbeddings Language modeling head\nLayer LTransformerBlockSoftmax over vocabulary VUnembedding layer\u20261 x |V|Logits Word probabilities1 x |V|hL1w1w2wNhL2hLNd x |V|1 x d   Unembedding    layer = ETy1y2y|V|\u2026u1u2u|V|\u2026Language Model Headtakes hLN and outputs adistribution over vocabulary V Language modeling head\nLayer LTransformerBlockSoftmax over vocabulary VUnembedding layer\u20261 x |V|Logits Word probabilities1 x |V|hL1w1w2wNhL2hLNd x |V|1 x d   Unembedding    layer = ETy1y2y|V|\u2026u1u2u|V|\u2026Language Model Headtakes hLN and outputs adistribution over vocabulary VUnembedding layer:  linear layer projects from hLN (shape [1 \u00d7 d]) to logit vector Why \"unembedding\"? Tied to ETWeight tying, we use the same weights for two different matricesUnembedding layer maps from an embedding to a 1x|V| vector of logits Language modeling head\nLayer LTransformerBlockSoftmax over vocabulary VUnembedding layer\u20261 x |V|Logits Word probabilities1 x |V|hL1w1w2wNhL2hLNd x |V|1 x d   Unembedding    layer = ETy1y2y|V|\u2026u1u2u|V|\u2026Language Model Headtakes hLN and outputs adistribution over vocabulary VLogits, the score vector uOne score for each of the |V | possible words in the vocabulary V . Shape 1 \u00d7 |V |. Softmax turns the logits into probabilities over vocabulary. Shape 1 \u00d7 |V |. 16CHAPTER9\u2022THETRANSFORMERlanguage models of Chapter 3 compute the probability of a word given counts ofits occurrence with then\u00001 prior words. The context is thus of sizen\u00001. Fortransformer language models, the context is the size of the transformer\u2019s contextwindow, which can be quite large: 2K, 4K, even 32K tokens for very large models.The job of the language modeling head is to take the output of the \ufb01nal trans-former layer from the last tokenNand use it to predict the upcoming word at posi-tionN+1. Fig.9.14shows how to accomplish this task, taking the output of the lasttoken at the last layer (thed-dimensional output embedding of shape[1\u21e5d]) andproducing a probability distribution over words (from which we will choose one togenerate).\nLayer LTransformerBlockSoftmax over vocabulary VUnembedding layer\u20261 x |V|Logits Word probabilities1 x |V|hL1w1w2wNhL2hLNd x |V|1 x d   Unembedding layerU = ETy1y2y|V|\u2026u1u2u|V|\u2026Language Model Headtakes hLN and outputs adistribution over vocabulary V\nFigure 9.14The language modeling head: the circuit at the top of a transformer that maps from the outputembedding for tokenNfrom the last transformer layer (hLN) to a probability distribution over words in thevocabularyV.The \ufb01rst module in Fig.9.14is a linear layer, whose job is to project from theoutputhLN, which represents the output token embedding at positionNfrom the \ufb01nalblockL, (hence of shape[1\u21e5d]) to thelogitvector, or score vector, that will have alogitsingle score for each of the|V|possible words in the vocabularyV. The logit vectoruis thus of dimensionality 1\u21e5|V|.This linear layer can be learned, but more commonly we tie this matrix to (thetranspose of) the embedding matrixE. Recall that inweight tying, we use theweight tyingsame weights for two different matrices in the model. Thus at the input stage of thetransformer the embedding matrix (of shape[|V|\u21e5d]) is used to map from a one-hotvector over the vocabulary (of shape[1\u21e5|V|]) to an embedding (of shape[1\u21e5d]).And then in the language model head,ET, the transpose of the embedding matrix (ofshape[d\u21e5|V|]) is used to map back from an embedding (shape[1\u21e5d]) to a vectorover the vocabulary (shape [1\u21e5|V|]). In the learning process,Ewill be optimized tobe good at doing both of these mappings. We therefore sometimes call the transposeETtheunembeddinglayer because it is performing this reverse mapping.unembeddingA softmax layer turns the logitsuinto the probabilitiesyover the vocabulary.u=hLNET(9.44)y=softmax(u)(9.45)We can use these probabilities to do things like help assign a probability to agiven text. But the most important usage to generate text, which we do bysampling The final transformermodel\nwiSample token togenerate at position i+1\nfeedforwardlayer normattentionlayer norm\nU\nInput tokenLanguageModelingHead\nInputEncoding\nEi+\u2026logits\nfeedforwardlayer normattentionlayer normLayer 1Layer 2h1i  =  x2ix1ih2i  =  x3ifeedforwardlayer normattentionlayer normhLi  hL-1i  =  xLiy1y2y|V|\u2026Token probabilitiesu1u2u|V|\u2026softmaxwi+1\nLayer L\nwiSample token togenerate at position i+1\nfeedforwardlayer normattentionlayer norm\nU\nInput tokenLanguageModelingHead\nInputEncoding\nEi+\u2026logits\nfeedforwardlayer normattentionlayer normLayer 1Layer 2h1i  =  x2ix1ih2i  =  x3ifeedforwardlayer normattentionlayer normhLi  hL-1i  =  xLiy1y2y|V|\u2026Token probabilitiesu1u2u|V|\u2026softmaxwi+1\nLayer L TransformersInput and output: Position embeddings and the Language Model Head", "10.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\n10Large Language Models\n\u201cHow much do we know at any time? Much more, or so I believe, than we\nknow we know. \u201d\nAgatha Christie, The Moving Finger\nFluent speakers of a language bring an enormous amount of knowledge to bear dur-\ning comprehension and production. This knowledge is embodied in many forms,\nperhaps most obviously in the vocabulary, the rich representations we have of words\nand their meanings and usage. This makes the vocabulary a useful lens to explore\nthe acquisition of knowledge from text, by both people and machines.\nEstimates of the size of adult vocabularies vary widely both within and across\nlanguages. For example, estimates of the vocabulary size of young adult speakers of\nAmerican English range from 30,000 to 100,000 depending on the resources used\nto make the estimate and the de\ufb01nition of what it means to know a word. What\nis agreed upon is that the vast majority of words that mature speakers use in their\nday-to-day interactions are acquired early in life through spoken interactions with\ncaregivers and peers, usually well before the start of formal schooling. This active\nvocabulary (usually on the order of 2000 words for young speakers) is extremely\nlimited compared to the size of the adult vocabulary, and is quite stable, with very\nfew additional words learned via casual conversation beyond this early stage. Obvi-\nously, this leaves a very large number of words to be acquired by other means.\nA simple consequence of these facts is that children have to learn about 7 to 10\nwords a day, every single day , to arrive at observed vocabulary levels by the time they\nare 20 years of age. And indeed empirical estimates of vocabulary growth in late el-\nementary through high school are consistent with this rate. How do children achieve\nthis rate of vocabulary growth? The bulk of this knowledge acquisition seems to\nhappen as a by-product of reading, as part of the rich processing and reasoning that\nwe perform when we read. Research into the average amount of time children spend\nreading, and the lexical diversity of the texts they read, indicate that it is possible\nto achieve the desired rate. But the mechanism behind this rate of learning must\nbe remarkable indeed, since at some points during learning the rate of vocabulary\ngrowth exceeds the rate at which new words are appearing to the learner!\nSuch facts have motivated the distributional hypothesis of Chapter 6, which sug-\ngests that aspects of meaning can be learned solely from the texts we encounter over\nour lives, based on the complex association of words with the words they co-occur\nwith (and with the words that those words occur with). The distributional hypothe-\nsis suggests both that we can acquire remarkable amounts of knowledge from text,\nand that this knowledge can be brought to bear long after its initial acquisition. Of\ncourse, grounding from real-world interaction or other modalities can help build\neven more powerful models, but even text alone is remarkably useful.\nIn this chapter we formalize this idea of pretraining \u2014learning knowledge about pretraining\nlanguage and the world from vast amounts of text\u2014and call the resulting pretrained\nlanguage models large language models . Large language models exhibit remark- 2CHAPTER 10 \u2022 L ARGE LANGUAGE MODELS\nable performance on all sorts of natural language tasks because of the knowledge\nthey learn in pretraining, and they will play a role throughout the rest of this book.\nThey have been especially transformative for tasks where we need to produce text,\nlike summarization, machine translation, question answering, or chatbots.\nWe\u2019ll start by seeing how to apply the transformer of Chapter 9 to language\nmodeling, in a setting often called causal or autoregressive language models, in\nwhich we iteratively predict words left-to-right from earlier words. We\u2019ll \ufb01rst in-\ntroduce training, seeing how language models are self-trained by iteratively being\ntaught to guess the next word in the text from the prior words.\nWe\u2019ll then talk about the process of text generation. The application of LLMs\nto generate text has vastly broadened the scope of NLP,. Text generation, code-\ngeneration, and image-generation together constitute the important new area of gen-\nerative AI . We\u2019ll introduce speci\ufb01c algorithms for generating text from a language generative AI\nmodel, like greedy decoding andsampling . And we\u2019ll see that almost any NLP\ntask can be modeled as word prediction in a large language model, if we think about\nit in the right way. We\u2019ll work through an example of using large language mod-\nels to solve one classic NLP task of summarization (generating a short text that\nsummarizes some larger document).\n10.1 Large Language Models with Transformers\nThe prior chapter introduced most of the components of a transformer in the domain\nof language modeling: the transformer block including multi-head attention , the\nlanguage modeling head , and the positional encoding of the input. In the following\nsections we\u2019ll introduce the remaining aspects of the transformer LLM: sampling\nandtraining . Before we do that, we use this section to talk about why and how we\napply transformer-based large language models to NLP tasks.\nThe tasks we will describe are all cases of conditional generation . Conditionalconditional\ngeneration\ngeneration is the task of generating text conditioned on an input piece of text. That\nis, we give the LLM an input piece of text, generally called a prompt , and then have\nthe LLM continue generating text token by token, conditioned on the prompt. The\nfact that transformers have such long contexts (many thousands of tokens) makes\nthem very powerful for conditional generation, because they can look back so far\ninto the prompting text.\nConsider the simple task of text completion, illustrated in Fig. 10.1. Here a\nlanguage model is given a text pre\ufb01x and is asked to generate a possible completion.\nNote that as the generation process proceeds, the model has direct access to the\npriming context as well as to all of its own subsequently generated outputs (at least\nas much as \ufb01ts in the large context window). This ability to incorporate the entirety\nof the earlier context and generated outputs at each time step is the key to the power\nof large language models built from transformers.\nSo why should we care about predicting upcoming words or tokens? The in-\nsight of large language modeling is that many practical NLP tasks can be cast as\nword prediction , and that a powerful-enough language model can solve them with\na high degree of accuracy. For example, we can cast sentiment analysis as language\nmodeling by giving a language model a context like:\nThe sentiment of the sentence ``I like Jackie Chan\" is:\nand comparing the following conditional probability of the words \u201cpositive\u201d and the 10.1 \u2022 L ARGE LANGUAGE MODELS WITH TRANSFORMERS 3\nPre\ufb01x TextCompletion Text\nEncoderTransformerBlocksSoftmax\nlongall\nandthanksforallthe\nthe\u2026UUUnencoder layerLanguage ModelingHeadlogits\nSo\nEi+\nEi+\nEi+\nEi+\nEi+\nEi+\nEi+\u2026\nFigure 10.1 Left-to-right (also called autoregressive) text completion with transformer-based large language\nmodels. As each token is generated, it gets added onto the context as a pre\ufb01x for generating the next token.\nword \u201cnegative\u201d to see which is higher:\nP(positivejThe sentiment of the sentence ``I like Jackie Chan\" is: )\nP(negativejThe sentiment of the sentence ``I like Jackie Chan\" is: )\nIf the word \u201cpositive\u201d is more probable, we say the sentiment of the sentence is\npositive, otherwise we say the sentiment is negative.\nWe can also cast more complex tasks as word prediction. Consider question\nanswering, in which the system is given a question (for example a question with\na simple factual answer) and must give a textual answer; we introduce this task in\ndetail in Chapter 14. We can cast the task of question answering as word prediction\nby giving a language model a question and a token like A:suggesting that an answer\nshould come next:\nQ: Who wrote the book ``The Origin of Species\"? A:\nIf we ask a language model to compute the probability distribution over possible\nnext words given this pre\ufb01x:\nP(wjQ: Who wrote the book ``The Origin of Species\"? A: )\nand look at which words whave high probabilities, we might expect to see that\nCharles is very likely, and then if we choose Charles and continue and ask\nP(wjQ: Who wrote the book ``The Origin of Species\"? A: Charles )\nwe might now see that Darwin is the most probable token, and select it.\nConditional generation can even be used to accomplish tasks that must generate\nlonger responses. Consider the task of text summarization , which is to take a longtext\nsummarization\ntext, such as a full-length article, and produce an effective shorter summary of it. We\ncan cast summarization as language modeling by giving a large language model a\ntext, and follow the text by a token like tl;dr ; this token is short for something like 4CHAPTER 10 \u2022 L ARGE LANGUAGE MODELS\n\u2018too long; didn\u2019t read\u2019 and in recent years people often use this token, especially in\ninformal work emails, when they are going to give a short summary. Since this token\nis suf\ufb01ciently frequent in language model training data, language models have seen\nmany texts in which the token occurs before a summary, and hence will interpret the\ntoken as instructions to generate a summary. We can then do conditional generation:\ngive the language model this pre\ufb01x, and then have it generate the following words,\none by one, and take the entire response as a summary. Fig. 10.2 shows an example\nof a text and a human-produced summary from a widely-used summarization corpus\nconsisting of CNN and Daily Mirror news articles.\nOriginal Article\nThe only thing crazier than a guy in snowbound Massachusetts boxing up the powdery white stuff\nand offering it for sale online? People are actually buying it. For $89, self-styled entrepreneur\nKyle Waring will ship you 6 pounds of Boston-area snow in an insulated Styrofoam box \u2013 enough\nfor 10 to 15 snowballs, he says.\nBut not if you live in New England or surrounding states. \u201cWe will not ship snow to any states\nin the northeast!\u201d says Waring\u2019s website, ShipSnowYo.com. \u201cWe\u2019re in the business of expunging\nsnow!\u201d\nHis website and social media accounts claim to have \ufb01lled more than 133 orders for snow \u2013 more\nthan 30 on Tuesday alone, his busiest day yet. With more than 45 total inches, Boston has set a\nrecord this winter for the snowiest month in its history. Most residents see the huge piles of snow\nchoking their yards and sidewalks as a nuisance, but Waring saw an opportunity.\nAccording to Boston.com, it all started a few weeks ago, when Waring and his wife were shov-\neling deep snow from their yard in Manchester-by-the-Sea, a coastal suburb north of Boston. He\njoked about shipping the stuff to friends and family in warmer states, and an idea was born. [...]\nSummary\nKyle Waring will ship you 6 pounds of Boston-area snow in an insulated Styrofoam box \u2013 enough\nfor 10 to 15 snowballs, he says. But not if you live in New England or surrounding states.\nFigure 10.2 Excerpt from a sample article and its summary from the CNN/Daily Mail summarization corpus\n(Hermann et al., 2015), (Nallapati et al., 2016).\nIf we take this full article and append the token tl;dr , we can use this as the con-\ntext to prime the generation process to produce a summary as illustrated in Fig. 10.3.\nAgain, what makes transformers able to succeed at this task (as compared, say, to\nthe primitive n-gram language model) is that attention can incorporate information\nfrom the large context window, giving the model access to the original article as well\nas to the newly generated text throughout the process.\nWhich words do we generate at each step? One simple way to generate words\nis to always generate the most likely word given the context. Generating the most\nlikely word given the context is called greedy decoding . A greedy algorithm is onegreedy\ndecoding\nthat make a choice that is locally optimal, whether or not it will turn out to have\nbeen the best choice with hindsight. Thus in greedy decoding, at each time step in\ngeneration, the output ytis chosen by computing the probability for each possible\noutput (every word in the vocabulary) and then choosing the highest probability\nword (the argmax):\n\u02c6wt=argmaxw2VP(wjw<t) (10.1)\nIn practice, however, we don\u2019t use greedy decoding with large language models.\nA major problem with greedy decoding is that because the words it chooses are (by\nde\ufb01nition) extremely predictable, the resulting text is generic and often quite repeti-\ntive. Indeed, greedy decoding is so predictable that it is deterministic; if the context 10.2 \u2022 S AMPLING FOR LLM G ENERATION 5\nOriginal StoryGenerated Summary\n\u2026ideaKyle\nwasborn.KyleWaring\nWaringonlyThe\u2026will\nDelimiterwillUUU\ntl;drLM Head\nE\nE\nE\nE\nE\nE\nE\nE\u2026\nFigure 10.3 Summarization with large language models using the tl;dr token and context-based autore-\ngressive generation.\nis identical, and the probabilistic model is the same, greedy decoding will always re-\nsult in generating exactly the same string. We\u2019ll see in Chapter 13 that an extension\nto greedy decoding called beam search works well in tasks like machine translation,\nwhich are very constrained in that we are always generating a text in one language\nconditioned on a very speci\ufb01c text in another language. In most other tasks, how-\never, people prefer text which has been generated by more sophisticated methods,\ncalled sampling methods , that introduce a bit more diversity into the generations.\nWe\u2019ll see how to do that in the next few sections.\n10.2 Sampling for LLM Generation\nThe core of the generation process for large language models is the task of choosing\nthe single word to generate next based on the context and based on the probabilities\nthat the model assigns to possible words. This task of choosing a word to generate\nbased on the model\u2019s probabilities is called decoding . Decoding from a language decoding\nmodel in a left-to-right manner (or right-to-left for languages like Arabic in which\nwe read from right to left), and thus repeatedly choosing the next word conditioned\non our previous choices is called autoregressive generation orcausal LM genera-autoregressive\ngeneration\ntion.1(As we\u2019ll see, alternatives like the masked language models of Chapter 11 are\nnon-causal because they can predict words based on both past and future words).\nThe most common method for decoding in large language models is sampling .\nRecall from Chapter 3 that sampling from a model\u2019s distribution over words means sampling\nto choose random words according to their probability assigned by the model. That\nis, we iteratively choose a word to generate according to its probability in context\n1Technically an autoregressive model predicts a value at time tbased on a linear function of the values\nat times t\u00001,t\u00002, and so on. Although language models are not linear (since they have many layers of\nnon-linearities), we loosely refer to this generation technique as autoregressive since the word generated\nat each time step is conditioned on the word selected by the network from the previous step. 6CHAPTER 10 \u2022 L ARGE LANGUAGE MODELS\nas de\ufb01ned by the model. Thus we are more likely to generate words that the model\nthinks have a high probability in the context and less likely to generate words that\nthe model thinks have a low probability.\nWe saw back in Chapter 3 on page ??how to generate text from a unigram lan-\nguage model , by repeatedly randomly sampling words according to their probability\nuntil we either reach a pre-determined length or select the end-of-sentence token. To\ngenerate text from a trained transformer language model we\u2019ll just generalize this\nmodel a bit: at each step we\u2019ll sample words according to their probability condi-\ntioned on our previous choices , and we\u2019ll use a transformer language model as the\nprobability model that tells us this probability.\nWe can formalize this algorithm for generating a sequence of words W=w1;w2;:::; wN\nuntil we hit the end-of-sequence token, using x\u0018p(x)to mean \u2018choose xby sam-\npling from the distribution p(x):\ni 1\nwi\u0018p(w)\nwhile wi!= EOS\ni i + 1\nwi\u0018p(wijw<i)\nThe algorithm above is called random sampling , and it turns out random sam-random\nsampling\npling doesn\u2019t work well enough. The problem is that even though random sampling\nis mostly going to generate sensible, high-probable words, there are many odd, low-\nprobability words in the tail of the distribution, and even though each one is low-\nprobability, if you add up all the rare words, they constitute a large enough portion\nof the distribution that they get chosen often enough to result in generating weird\nsentences. For this reason, instead of random sampling, we usually use sampling\nmethods that avoid generating the very unlikely words.\nThe sampling methods we introduce below each have parameters that enable\ntrading off two important factors in generation: quality anddiversity . Methods\nthat emphasize the most probable words tend to produce generations that are rated\nby people as more accurate, more coherent, and more factual, but also more boring\nand more repetitive. Methods that give a bit more weight to the middle-probability\nwords tend to be more creative and more diverse, but less factual and more likely to\nbe incoherent or otherwise low-quality.\n10.2.1 Top- ksampling\nTop-k sampling is a simple generalization of greedy decoding. Instead of choosing top-k sampling\nthe single most probable word to generate, we \ufb01rst truncate the distribution to the\ntopkmost likely words, renormalize to produce a legitimate probability distribution,\nand then randomly sample from within these kwords according to their renormalized\nprobabilities. More formally:\n1. Choose in advance a number of words k\n2. For each word in the vocabulary V, use the language model to compute the\nlikelihood of this word given the context p(wtjw<t)\n3. Sort the words by their likelihood, and throw away any word that is not one of\nthe top kmost probable words.\n4. Renormalize the scores of the kwords to be a legitimate probability distribu-\ntion. 10.2 \u2022 S AMPLING FOR LLM G ENERATION 7\n5. Randomly sample a word from within these remaining kmost-probable words\naccording to its probability.\nWhen k=1, top- ksampling is identical to greedy decoding. Setting kto a larger\nnumber than 1 leads us to sometimes select a word which is not necessarily the most\nprobable, but is still probable enough, and whose choice results in generating more\ndiverse but still high-enough-quality text.\n10.2.2 Nucleus or top- psampling\nOne problem with top- ksampling is that kis \ufb01xed, but the shape of the probability\ndistribution over words differs in different contexts. If we set k=10, sometimes\nthe top 10 words will be very likely and include most of the probability mass, but\nother times the probability distribution will be \ufb02atter and the top 10 words will only\ninclude a small part of the probability mass.\nAn alternative, called top-p sampling ornucleus sampling (Holtzman et al., top-p sampling\n2020), is to keep not the top kwords, but the top ppercent of the probability mass.\nThe goal is the same; to truncate the distribution to remove the very unlikely words.\nBut by measuring probability rather than the number of words, the hope is that the\nmeasure will be more robust in very different contexts, dynamically increasing and\ndecreasing the pool of word candidates.\nGiven a distribution P(wtjw<t), the top- pvocabulary V(p)is the smallest set of\nwords such that\nX\nw2V(p)P(wjw<t)\u0015p: (10.2)\n10.2.3 Temperature sampling\nIntemperature sampling , we don\u2019t truncate the distribution, but instead reshapetemperature\nsampling\nit. The intuition for temperature sampling comes from thermodynamics, where a\nsystem at a high temperature is very \ufb02exible and can explore many possible states,\nwhile a system at a lower temperature is likely to explore a subset of lower energy\n(better) states. In low-temperature sampling, we smoothly increase the probability\nof the most probable words and decrease the probability of the rare words.\nWe implement this intuition by simply dividing the logit by a temperature param-\netertbefore we normalize it by passing it through the softmax. In low-temperature\nsampling, t2(0;1]. Thus instead of computing the probability distribution over the\nvocabulary directly from the logit as in the following (repeated from ( ??)):\ny=softmax (u) (10.3)\nwe instead \ufb01rst divide the logits by t, computing the probability vector yas\ny=softmax (u=t) (10.4)\nWhy does this work? When tis close to 1 the distribution doesn\u2019t change much.\nBut the lower tis, the larger the scores being passed to the softmax (dividing by a\nsmaller fraction t\u00141 results in making each score larger). Recall that one of the\nuseful properties of a softmax is that it tends to push high values toward 1 and low\nvalues toward 0. Thus when larger numbers are passed to a softmax the result is\na distribution with increased probabilities of the most high-probability words and\ndecreased probabilities of the low probability words, making the distribution more\ngreedy. As tapproaches 0 the probability of the most likely word approaches 1. 8CHAPTER 10 \u2022 L ARGE LANGUAGE MODELS\nNote, by the way, that there can be other situations where we may want to do\nsomething quite different and \ufb02atten the word probability distribution instead of\nmaking it greedy. Temperature sampling can help with this situation too, in this case\nhigh-temperature sampling, in which case we use t>1.\n10.3 Pretraining Large Language Models\nHow do we teach a transformer to be a language model? What is the algorithm and\nwhat data do we train on?\n10.3.1 Self-supervised training algorithm\nTo train a transformer as a language model, we use the same self-supervision (or self-supervision\nself-training ) algorithm we saw in Section ??: we take a corpus of text as training\nmaterial and at each time step task the model to predict the next word. We call\nsuch a model self-supervised because we don\u2019t have to add any special gold labels\nto the data; the natural sequence of words is its own supervision! We simply train the\nmodel to minimize the error in predicting the true next word in the training sequence,\nusing cross-entropy as the loss function.\nRecall that the cross-entropy loss measures the difference between a predicted\nprobability distribution and the correct distribution.\nLCE=\u0000X\nw2Vyt[w]log\u02c6yt[w] (10.5)\nIn the case of language modeling, the correct distribution ytcomes from knowing the\nnext word. This is represented as a one-hot vector corresponding to the vocabulary\nwhere the entry for the actual next word is 1, and all the other entries are 0. Thus,\nthe cross-entropy loss for language modeling is determined by the probability the\nmodel assigns to the correct next word (all other words get multiplied by zero). So\nat time tthe CE loss in (10.5) can be simpli\ufb01ed as the negative log probability the\nmodel assigns to the next word in the training sequence.\nLCE(\u02c6yt;yt) =\u0000log\u02c6yt[wt+1] (10.6)\nThus at each word position tof the input, the model takes as input the correct se-\nquence of tokens w1:t, and uses them to compute a probability distribution over\npossible next words so as to compute the model\u2019s loss for the next token wt+1. Then\nwe move to the next word, we ignore what the model predicted for the next word\nand instead use the correct sequence of tokens w1:t+1to estimate the probability of\ntoken wt+2. This idea that we always give the model the correct history sequence to\npredict the next word (rather than feeding the model its best case from the previous\ntime step) is called teacher forcing . teacher forcing\nFig. 10.4 illustrates the general training approach. At each step, given all the\npreceding words, the \ufb01nal transformer layer produces an output distribution over\nthe entire vocabulary. During training, the probability assigned to the correct word\nis used to calculate the cross-entropy loss for each item in the sequence. The loss\nfor a training sequence is the average cross-entropy loss over the entire sequence.\nThe weights in the network are adjusted to minimize the average CE loss over the\ntraining sequence via gradient descent. 10.3 \u2022 P RETRAINING LARGE LANGUAGE MODELS 9\nlongandthanksforNext tokenallLoss\u2026=\n<latexit sha1_base64=\"AovqpaL476UmJ1EU1xZPgDZ70tQ=\">AAAB9nicbVDLSsNAFL2pr1pfURcu3AwWwY0lEakui25cVrAPaEqYTCbt0EkmzEzEEvIrbkTcKPgZ/oJ/Y9Jm09YDA4dzznDvPV7MmdKW9WtU1tY3Nreq27Wd3b39A/PwqKtEIgntEMGF7HtYUc4i2tFMc9qPJcWhx2nPm9wXfu+ZSsVE9KSnMR2GeBSxgBGsc8k1Ty4dLkZo6qZOiPVYhimO/CyruWbdalgzoFVil6QOJdqu+eP4giQhjTThWKmBbcV6mGKpGeE0qzmJojEmEzyi6WztDJ3nko8CIfMXaTRTF3I4VGoaenmy2E0te4X4nzdIdHA7TFkUJ5pGZD4oSDjSAhUdIJ9JSjSf5gQTyfINERljiYnOmypOt5cPXSXdq4bdbDQfr+utu7KEKpzCGVyADTfQggdoQwcIZPAGn/BlvBivxrvxMY9WjPLPMSzA+P4DPEiSHA==</latexit>\u0000logyand\nStackedTransformerBlocksSolongandthanksfor\u2026\u2026\n\u2026\nU\nInput tokensx1x2LanguageModelingHeadx3x4x5InputEncoding\nE1+\nE2+\nE3+\nE4+\nE5+\u2026\u2026\u2026\u2026\u2026\nU\nU\nU\nU\u2026logitslogitslogitslogitslogits\u2026\n<latexit sha1_base64=\"q3ZgXDyG7qtkT7t8hT47RdlwYG4=\">AAAB+XicbVDLSsNAFJ3UV62vWHe6GVsEN5bERXUlBUVcVrAPaEqYTCft0MlMmJkIIQT8AT/CTRE3Cv6Ev+DfmLTdtPXAwOGcM9x7jxcyqrRl/RqFtfWNza3idmlnd2//wDwst5WIJCYtLJiQXQ8pwignLU01I91QEhR4jHS88W3ud56JVFTwJx2HpB+gIac+xUhnkmseXzhMDGHsJk6A9EgGiR4hPlZpWnLNqlWzpoCrxJ6TauP0tXw3qdw0XfPHGQgcBYRrzJBSPdsKdT9BUlPMSFpyIkVChMdoSJLp5ik8y6QB9IXMHtdwqi7kUKBUHHhZMl9PLXu5+J/Xi7R/3U8oDyNNOJ4N8iMGtYB5DXBAJcGaxRlBWNJsQ4hHSCKss7Ly0+3lQ1dJ+7Jm12v1x6yDezBDEZyACjgHNrgCDfAAmqAFMHgBE/AJvozEeDPejY9ZtGDM/xyBBRjff79pldo=</latexit>\u0000logythanks\nFigure 10.4 Training a transformer as a language model.\nNote the key difference between this \ufb01gure and the earlier RNN-based version\nshown in Fig. ??. There the calculation of the outputs and the losses at each step was\ninherently serial given the recurrence in the calculation of the hidden states. With\ntransformers, each training item can be processed in parallel since the output for\neach element in the sequence is computed separately.\nLarge models are generally trained by \ufb01lling the full context window (for exam-\nple 4096 tokens for GPT4 or 8192 for Llama 3) with text. If documents are shorter\nthan this, multiple documents are packed into the window with a special end-of-text\ntoken between them. The batch size for gradient descent is usually quite large (the\nlargest GPT-3 model uses a batch size of 3.2 million tokens).\n10.3.2 Training corpora for large language models\nLarge language models are mainly trained on text scraped from the web, augmented\nby more carefully curated data. Because these training corpora are so large, they are\nlikely to contain many natural examples that can be helpful for NLP tasks, such as\nquestion and answer pairs (for example from FAQ lists), translations of sentences\nbetween various languages, documents together with their summaries, and so on.\nWeb text is usually taken from corpora of automatically-crawled web pages like\nthecommon crawl , a series of snapshots of the entire web produced by the non- common crawl\npro\ufb01t Common Crawl ( https://commoncrawl.org/ ) that each have billions of\nwebpages. Various versions of common crawl data exist, such as the Colossal Clean\nCrawled Corpus (C4; Raffel et al. 2020), a corpus of 156 billion tokens of English\nthat is \ufb01ltered in various ways (deduplicated, removing non-natural language like\ncode, sentences with offensive words from a blocklist). This C4 corpus seems to\nconsist in large part of patent text documents, Wikipedia, and news sites (Dodge\net al., 2021).\nWikipedia plays a role in lots of language model training, as do corpora of books.\nThe Pile (Gao et al., 2020) is an 825 GB English text corpus that is constructed by The Pile\npublicly released code, containing again a large amount of text scraped from the web 10 CHAPTER 10 \u2022 L ARGE LANGUAGE MODELS\nas well as books and Wikipedia; Fig. 10.5 shows its composition. Dolma is a larger\nopen corpus of English, created with public tools, containing three trillion tokens,\nwhich similarly consists of web text, academic papers, code, books, encyclopedic\nmaterials, and social media (Soldaini et al., 2024).\nFigure 1: Treemap of Pile components by effective size.\ntroduce a new \ufb01ltered subset of Common Crawl,\nPile-CC, with improved extraction quality.\nThrough our analyses, we con\ufb01rm that the Pile is\nsigni\ufb01cantly distinct from pure Common Crawl\ndata. Additionally, our evaluations show that the\nexisting GPT-2 and GPT-3 models perform poorly\non many components of the Pile, and that models\ntrained on the Pile signi\ufb01cantly outperform both\nraw and \ufb01ltered Common Crawl models. To com-\nplement the performance evaluations, we also per-\nform an exploratory analysis of the text within the\nPile to provide a detailed picture of the data. We\nhope that our extensive documentation of the con-\nstruction and characteristics of the Pile will help\nresearchers make informed decisions about poten-\ntial downstream applications.\nFinally, we make publicly available the preprocess-\ning code for the constituent datasets of the Pile and\nthe code for constructing alternative versions2. In\nthe interest of reproducibility, we also document\nall processing performed on each dataset (and the\nPile as a whole) in as much detail as possible. For\nfurther details about the processing of each dataset,\nsee Section 2and Appendix C.\n2https://github.com/EleutherAI/\nthe-pile1.1 Contributions\nThe core contributions of this paper are:\n1.The introduction of a 825.18GiB english-\nlanguage dataset for language modeling com-\nbining 22 diverse sources.\n2.The introduction of 14new language model-\ning datasets, which we expect to be of inde-\npendent interest to researchers.\n3.Evaluations demonstrating signi\ufb01cant im-\nprovements across many domains by GPT-2-\nsized models trained on this new dataset, com-\npared to training on CC-100 and raw Common\nCrawl.\n4.The investigation and documentation of this\ndataset, which we hope will better inform re-\nsearchers about how to use it as well as moti-\nvate them to undertake similar investigations\nof their own data.\n2 The Pile Datasets\nThe Pile is composed of 22 constituent sub-datasets,\nas shown in Table 1. Following Brown et al. (2020 ),\nwe increase the weights of higher quality compo-\nnents, with certain high-quality datasets such as\nWikipedia being seen up to 3 times (\u201cepochs\u201d) for\n2\nFigure 10.5 The Pile corpus, showing the size of different components, color coded as\nacademic (articles from PubMed and ArXiv, patents from the USPTA; internet (webtext in-\ncluding a subset of the common crawl as well as Wikipedia), prose (a large corpus of books),\ndialogue (including movie subtitles and chat data), and misc.. Figure from Gao et al. (2020).\nFiltering for quality and safety Pretraining data drawn from the web is \ufb01ltered\nfor both quality andsafety . Quality \ufb01lters are classi\ufb01ers that assign a score to each\ndocument. Quality is of course subjective, so different quality \ufb01lters are trained\nin different ways, but often to value high-quality reference corpora like Wikipedia,\nbooks, and particular websites and to avoid websites with lots of PII(Personal Iden- PII\nti\ufb01able Information) or adult content. Filters also remove boilerplate text which is\nvery frequent on the web. Another kind of quality \ufb01ltering is deduplication, which\ncan be done at various levels, so as to remove duplicate documents, duplicate web\npages, or duplicate text. Quality \ufb01ltering generally improves language model per-\nformance (Longpre et al., 2024b; Llama Team, 2024).\nSafety \ufb01ltering is again a subjective decision, and often includes toxicity detec-\ntion based on running off-the-shelf toxicity classi\ufb01ers. This can have mixed results.\nOne problem is that current toxicity classi\ufb01ers mistakenly \ufb02ag non-toxic data if it\nis generated by speakers of minority dialects like African American English (Xu\net al., 2021). Another problem is that models trained on toxicity-\ufb01ltered data, while\nsomewhat less toxic, are also worse at detecting toxicity themselves (Longpre et al.,\n2024b). These issues make the question of how to do better safety \ufb01ltering an im-\nportant open problem.\nUsing large datasets scraped from the web to train language models poses ethical\nand legal questions:\nCopyright: Much of the text in these large datasets (like the collections of \ufb01c-\ntion and non-\ufb01ction books) is copyrighted. In some countries, like the United\nStates, the fair use doctrine may allow copyrighted content to be used for\ntransformative uses, but it\u2019s not clear if that remains true if the language mod-\nels are used to generate text that competes with the market for the text they 10.3 \u2022 P RETRAINING LARGE LANGUAGE MODELS 11\nare trained on (Henderson et al., 2023).\nData consent: Owners of websites can indicate that they don\u2019t want their sites\nto be crawled by web crawlers (either via a robots.txt \ufb01le, or via Terms of\nService). Recently there has been a sharp increase in the number of web-\nsites that have indicated that they don\u2019t want large language model builders\ncrawling their sites for training data (Longpre et al., 2024a). Because it\u2019s not\nclear what legal status these indications have in different countries, or whether\nthese restrictions are retroactive, what effect this will have on large pretraining\ndatasets is unclear.\nPrivacy: Large web datasets also have privacy issues since they contain private\ninformation like phone numbers and IP addresses. While \ufb01lters are used to try\nto remove websites likely to contain large amounts of personal information,\nsuch \ufb01ltering isn\u2019t suf\ufb01cient.\n10.3.3 Finetuning\nAlthough the enormous pretraining data for a large language model includes text\nfrom many domains, it\u2019s often the case that we want to apply it in a new domain or\ntask that might not have appeared suf\ufb01ciently in the pre-training data. For example,\nwe might want a language model that\u2019s specialized to legal or medical text. Or we\nmight have a multilingual language model that knows many languages but might\nbene\ufb01t from some more data in our particular language of interest. Or we want a\nlanguage model that is specialized to a particular task.\nIn such cases, we can simply continue training the model on relevant data from\nthe new domain or language (Gururangan et al., 2020). This process of taking a fully\npretrained model and running additional training passes on some new data is called\n\ufb01netuning . Fig. 10.6 sketches the paradigm. \ufb01netuning\nFine-tuning Data\nPretraining DataPretraining\n\u2026\n\u2026\n\u2026Fine-tuning\n\u2026\n\u2026\n\u2026Pretrained LMFine-tuned LM\nFigure 10.6 Pretraining and \ufb01netuning. A pre-trained model can be \ufb01netuned to a par-\nticular domain, dataset, or task. There are many different ways to \ufb01netune, depending on\nexactly which parameters are updated from the \ufb01netuning data: all the parameters, some of\nthe parameters, or only the parameters of speci\ufb01c extra circuitry.\nWe\u2019ll introduce four related kinds of \ufb01netuning in this chapter and the two fol-\nlowing chapters. In all four cases, \ufb01netuning means the process of taking a pre-\ntrained model and further adapting some or all of its parameters to some new data.\nBut they differ on exactly which parameters get updated.\nIn the \ufb01rst kind of \ufb01netuning we retrain all the parameters of the model on this\nnew data, using the same method (word prediction) and loss function (cross-entropy\nloss) as for pretraining. In a sense it\u2019s as if the new data were at the tail end of 12 CHAPTER 10 \u2022 L ARGE LANGUAGE MODELS\nthe pretraining data, and so you\u2019ll sometimes see this method called continued pre-\ntraining .continued\npretraining\nRetraining all the parameters of the model is very slow and expensive when the\nlanguage model is huge. So instead we can freeze some of the parameters (i.e., leave freeze\nthem unchanged from their pretrained value) and train only a subset of parameters\non the new data. In Section 10.5.3 we\u2019ll describe this second variety of \ufb01netun-\ning, called parameter-ef\ufb01cient \ufb01netuning , orPEFT . because we ef\ufb01ciently select\nspeci\ufb01c parameters to update when \ufb01netuning, and leave the rest in their pretrained\nvalues.\nIn Chapter 11 we\u2019ll introduce a third kind of \ufb01netuning, also parameter-ef\ufb01cient.\nIn this version, the goal is to use a language model as a kind of classi\ufb01er or labeler\nfor a speci\ufb01c task. For example we might train the model to be a sentiment classi\ufb01er.\nWe do this by adding extra neural circuitry (an extra head ) after the top layer of the\nmodel. This classi\ufb01cation head takes as input some of the top layer embeddings of\nthe transformer and produces as output a classi\ufb01cation. In this method, most com-\nmonly used with masked language models like BERT, we freeze the entire pretrained\nmodel and only train the classi\ufb01cation head on some new data, usually labeled with\nsome class that we want to predict.\nFinally, in Chapter 12 we\u2019ll introduce a fourth kind of \ufb01netuning, that is a cru-\ncial component of the largest language models: supervised \ufb01netuning orSFT. SFT\nis often used for instruction \ufb01netuning , in which we want a pretrained language\nmodel to learn to follow text instructions, for example to answer questions or follow\na command to write something. Here we create a dataset of prompts and desired\nresponses (for example questions and their answers, or commands and their ful-\n\ufb01llments), and we train the language model using the normal cross-entropy loss to\npredict each token in the instruction prompt iteratively, essentially training it to pro-\nduce the desired response from the command in the prompt. It\u2019s called supervised\nbecause unlike in pretraining, where we just take any data and predict the words in\nit, we build the special \ufb01netuning dataset by hand, creating supervised responses to\neach command.\nOften everything that happens after pretraining is lumped together as post-training ;\nwe\u2019ll discuss the various parts of post-training in Chapter 12.\n10.4 Evaluating Large Language Models\nPerplexity As we \ufb01rst saw in Chapter 3, one way to evaluate language models is\nto measure how well they predict unseen text. Intuitively, good models are those that\nassign higher probabilities to unseen data (are less surprised when encountering the\nnew words).\nWe instantiate this intuition by using perplexity to measure the quality of a perplexity\nlanguage model. Recall from page ??that the perplexity of a model qon an unseen\ntest set is the inverse probability that qassigns to the test set, normalized by the test\nset length. For a test set of ntokens w1:n, the perplexity is\nPerplexityq(w1:n) = Pq(w1:n)\u00001\nn\n=ns\n1\nPq(w1:n)(10.7)\nTo visualize how perplexity can be computed as a function of the probabilities the 10.4 \u2022 E VALUATING LARGE LANGUAGE MODELS 13\nLM computes for each new word, we can use the chain rule to expand the computa-\ntion of probability of the test set:\nPerplexityq(w1:n) =nvuutnY\ni=11\nPq(wijw<i)(10.8)\nNote that because of the inverse in Eq. 10.7, the higher the probability of the word\nsequence, the lower the perplexity. Thus the the lower the perplexity of a model on\nthe data, the better the model . Minimizing perplexity is equivalent to maximizing\nthe test set probability according to the language model.\nOne caveat: because perplexity depends on the length of a text, it is very sensitive\nto differences in the tokenization algorithm. That means that it\u2019s hard to exactly\ncompare perplexities produced by two language models if they have very different\ntokenizers. For this reason perplexity is best used when comparing language models\nthat use the same tokenizer.\nOther factors While the predictive accuracy of a language model, as measured by\nperplexity, is a very useful metric, we also care about different kinds of accuracy, for\nthe downstream tasks we apply our language model to. For each task like machine\ntranslation, summarization, question answering, speech recognition, and dialogue,\nwe can measure the accuracy at those tasks. Future chapters will introduce task-\nspeci\ufb01c metrics that allow us to evaluate how accuracy or correct language models\nare at these downstream tasks.\nBut when evaluating models we also care about factors besides any of these\nkinds of accuracy (Dodge et al., 2019; Ethayarajh and Jurafsky, 2020). For example,\nwe often care about how a big a model is, and how long it takes to train or do\ninference. This can matter because we have constraints on time either for training\nor at inference. Or we may have constraints on memory, since the GPUs we run\nour models on have \ufb01xed memory sizes. Big models also use more energy, and we\nprefer models that use less energy, both to reduce the environmental impact of the\nmodel and to reduce the \ufb01nancial cost of building or deploying it. We can target\nour evaluation to these factors by measuring performance normalized to a giving\ncompute or memory budget. We can also directly measure the energy usage of our\nmodel in kWh or in kilograms of CO 2emitted (Strubell et al., 2019; Henderson\net al., 2020; Liang et al., 2023).\nAnother feature that a language model evaluation can measure is fairness. We\nknow that language models are biased, exhibiting gendered and racial stereotypes,\nor decreased performance for language from or about certain demographics groups.\nThere are language model evaluation benchmarks that measure the strength of these\nbiases, such as StereoSet (Nadeem et al., 2021), RealToxicityPrompts (Gehman\net al., 2020), and BBQ (Parrish et al., 2022) among many others. We also want\nlanguage models whose performance is equally fair to different groups. For exam-\nple, we could chose an evaluation that is fair in a Rawlsian sense by maximizing the\nwelfare of the worst-off group (Rawls, 2001; Hashimoto et al., 2018; Sagawa et al.,\n2020).\nFinally, there are many kinds of leaderboards like Dynabench (Kiela et al., 2021)\nand general evaluation protocols like HELM (Liang et al., 2023); we will return to\nthese in later chapters when we introduce evaluation metrics for speci\ufb01c tasks like\nquestion answering and information retrieval. 14 CHAPTER 10 \u2022 L ARGE LANGUAGE MODELS\n10.5 Dealing with Scale\nLarge language models are large. For example the Llama 3.1 405B Instruct model\nfrom Meta has 405 billion parameters (126 layers, a model dimensionality of 16,384,\n128 attention heads) and was trained on 15.6 terabytes of text tokens (Llama Team,\n2024), using a vocabulary of 128K tokens. So there is a lot of research on un-\nderstanding how LLMs scale, and especially how to implement them given limited\nresources. In the next few sections we discuss how to think about scale (the concept\nofscaling laws ), and important techniques for getting language models to work\nef\ufb01ciently, such as the KV cache and parameter-ef\ufb01cient \ufb01ne tuning.\n10.5.1 Scaling laws\nThe performance of large language models has shown to be mainly determined by\n3 factors: model size (the number of parameters not counting embeddings), dataset\nsize (the amount of training data), and the amount of compute used for training. That\nis, we can improve a model by adding parameters (adding more layers or having\nwider contexts or both), by training on more data, or by training for more iterations.\nThe relationships between these factors and performance are known as scaling\nlaws . Roughly speaking, the performance of a large language model (the loss) scales scaling laws\nas a power-law with each of these three properties of model training.\nFor example, Kaplan et al. (2020) found the following three relationships for\nlossLas a function of the number of non-embedding parameters N, the dataset size\nD, and the compute budget C, for models training with limited parameters, dataset,\nor compute budget, if in each case the other two properties are held constant:\nL(N) =\u0012Nc\nN\u0013aN\n(10.9)\nL(D) =\u0012Dc\nD\u0013aD\n(10.10)\nL(C) =\u0012Cc\nC\u0013aC\n(10.11)\nThe number of (non-embedding) parameters Ncan be roughly computed as fol-\nlows (ignoring biases, and with das the input and output dimensionality of the\nmodel, dattnas the self-attention layer size, and dffthe size of the feedforward layer):\nN\u00192d nlayer(2dattn+dff)\n\u001912nlayerd2(10.12)\n(assuming dattn=dff=4=d)\nThus GPT-3, with n=96 layers and dimensionality d=12288, has 12\u000296\u0002\n122882\u0019175 billion parameters.\nThe values of Nc,Dc,Cc,aN,aD, and aCdepend on the exact transformer\narchitecture, tokenization, and vocabulary size, so rather than all the precise values,\nscaling laws focus on the relationship with loss.2\nScaling laws can be useful in deciding how to train a model to a particular per-\nformance, for example by looking at early in the training curve, or performance with\n2For the initial experiment in Kaplan et al. (2020) the precise values were aN= 0.076, Nc= 8.8\u00021013\n(parameters), aD= 0.095, Dc= 5.4\u00021013(tokens), aC= 0.050, Cc= 3.1\u0002108(peta\ufb02op-days). 10.5 \u2022 D EALING WITH SCALE 15\nsmaller amounts of data, to predict what the loss would be if we were to add more\ndata or increase model size. Other aspects of scaling laws can also tell us how much\ndata we need to add when scaling up a model.\n10.5.2 KV Cache\nWe saw in Fig. ??and in Eq. ??(repeated below) how the attention vector can be\nvery ef\ufb01ciently computed in parallel for training, via two matrix multiplications:\nA=softmax\u0012QK|\npdk\u0013\nV (10.13)\nUnfortunately we can\u2019t do quite the same ef\ufb01cient computation in inference as\nin training. That\u2019s because at inference time, we iteratively generate the next tokens\none at a time. For a new token that we have just generated, call it xi, we need to\ncompute its query, key, and values by multiplying by WQ,WK, and WVrespec-\ntively. But it would be a waste of computation time to recompute the key and value\nvectors for all the prior tokens x<i; at prior steps we already computed these key\nand value vectors! So instead of recomputing these, whenever we compute the key\nand value vectors we store them in memory in the KV cache , and then we can just KV cache\ngrab them from the cache when we need them. Fig. 10.7 modi\ufb01es Fig. ??to show\nthe computation that takes place for a single new token, showing which values we\ncan take from the cache rather than recompute.\nq4k1k2k4QKTQKTv1v2v3v4V\nq4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4x==xa4A\n1 x dkdk x N1 x NN x dv1 x dv\nk3\nFigure 10.7 Parts of the attention computation (extracted from Fig. ??) showing, in black,\nthe vectors that can be stored in the cache rather than recomputed when computing the atten-\ntion score for the 4th token.\n10.5.3 Parameter Ef\ufb01cient Fine Tuning\nAs we mentioned above, it\u2019s very common to take a language model and give it more\ninformation about a new domain by \ufb01netuning it (continuing to train it to predict\nupcoming words) on some additional data.\nFine-tuning can be very dif\ufb01cult with very large language models, because there\nare enormous numbers of parameters to train; each pass of batch gradient descent\nhas to backpropagate through many many huge layers. This makes \ufb01netuning huge\nlanguage models extremely expensive in processing power, in memory, and in time.\nFor this reason, there are alternative methods that allow a model to be \ufb01netuned\nwithout changing all the parameters. Such methods are called parameter-ef\ufb01cient\n\ufb01ne tuning or sometimes PEFT , because we ef\ufb01ciently select a subset of parametersparameter-\nef\ufb01cient \ufb01ne\ntuning\nPEFT to update when \ufb01netuning. For example we freeze some of the parameters (don\u2019t\nchange them), and only update some particular subset of parameters. 16 CHAPTER 10 \u2022 L ARGE LANGUAGE MODELS\nHere we describe one such model, called LoRA , forLow-RankAdaptation. The LoRA\nintuition of LoRA is that transformers have many dense layers which perform matrix\nmultiplication (for example the WQ,WK,WV,WOlayers in the attention computa-\ntion). Instead of updating these layers during \ufb01netuning, with LoRA we freeze these\nlayers and instead update a low-rank approximation that has fewer parameters.\nConsider a matrix Wof dimensionality [N\u0002d]that needs to be updated during\n\ufb01netuning via gradient descent. Normally this matrix would get updates DWof\ndimensionality [N\u0002d], for updating the N\u0002dparameters after gradient descent. In\nLoRA, we freeze Wand update instead a low-rank decomposition of W. We create\ntwo matrices AandB, where Ahas size [N\u0002r]andBhas size [r\u0002d], and we choose\nrto be quite small, r<<min(d;N). During \ufb01netuning we update AandBinstead\nofW. That is, we replace W+DWwithW+BA. Fig. 10.8 shows the intuition.\nFor replacing the forward pass h=xW, the new forward pass is instead:\nh=xW+xAB (10.14)\nhPretrained WeightsWdkrkABrxd11k\nd\u00d7\nFigure 10.8 The intuition of LoRA. We freeze Wto its pretrained values, and instead \ufb01ne-\ntune by training a pair of matrices AandB, updating those instead of W, and just sum Wand\nthe updated AB.\nLoRA has a number of advantages. It dramatically reduces hardware require-\nments, since gradients don\u2019t have to be calculated for most parameters. The weight\nupdates can be simply added in to the pretrained weights, since BAis the same size\nasW). That means it doesn\u2019t add any time during inference. And it also means it\u2019s\npossible to build LoRA modules for different domains and just swap them in and\nout by adding them in or subtracting them from W.\nIn its original version LoRA was applied just to the matrices in the attention\ncomputation (the WQ,WK,WV, andWOlayers). Many variants of LoRA exist. 10.6 \u2022 P OTENTIAL HARMS FROM LANGUAGE MODELS 17\n10.6 Potential Harms from Language Models\nLarge pretrained neural language models exhibit many of the potential harms dis-\ncussed in Chapter 4 and Chapter 6. Many of these harms become realized when\npretrained language models are used for any downstream task, particularly those\ninvolving text generation, whether question answering, machine translation, or in\nassistive technologies like writing aids or web search query completion, or predic-\ntive typing for email (Olteanu et al., 2020).\nFor example, language models are prone to saying things that are false, a prob-\nlem called hallucination . Language models are trained to generate text that is pre- hallucination\ndictable and coherent, but the training algorithms we have seen so far don\u2019t have\nany way to enforce that the text that is generated is correct or true. This causes\nenormous problems for any application where the facts matter! We\u2019ll return to this\nissue in Chapter 14 where we introduce proposed mitigation methods like retrieval\naugmented generation .\nA second source of harm is that language models can generate toxic language . toxic language\nGehman et al. (2020) show that even completely non-toxic prompts can lead large\nlanguage models to output hate speech and abuse their users. Language models also\ngenerate stereotypes (Cheng et al., 2023) and negative attitudes (Brown et al., 2020;\nSheng et al., 2019) about many demographic groups.\nOne source of biases is the training data. Gehman et al. (2020) shows that large\nlanguage model training datasets include toxic text scraped from banned sites. There\nare other biases than toxicity: the training data is disproportionately generated by\nauthors from the US and from developed countries. Such biased population samples\nlikely skew the resulting generation toward the perspectives or topics of this group\nalone. Furthermore, language models can amplify demographic and other biases in\ntraining data, just as we saw for embedding models in Chapter 6.\nDatasets can be another source of harms. We already saw in Section 10.3.2\nthat using pretraining corpora scraped from the web can lead to harms related to\ncopyright and data consent. We also mentioned that pretraining data can tend to\nhave private information like phone numbers and addresses. This is problematic\nbecause large language models can leak information from their training data. That\nis, an adversary can extract training-data text from a language model such as a per-\nson\u2019s name, phone number, and address (Henderson et al. 2017, Carlini et al. 2021).\nThis becomes even more problematic when large language models are trained on\nextremely sensitive private datasets such as electronic health records.\nLanguage models can also be used by malicious actors for generating text for\nmisinformation , phishing, or other socially harmful activities (Brown et al., 2020).\nMcGuf\ufb01e and Newhouse (2020) show how large language models generate text that\nemulates online extremists, with the risk of amplifying extremist movements and\ntheir attempt to radicalize and recruit.\nFinding ways to mitigate all these harms is an important current research area in\nNLP. At the very least, carefully analyzing the data used to pretrain large language\nmodels is important as a way of understanding issues of toxicity, bias, privacy, and\nfair use, making it extremely important that language models include datasheets\n(page ??) ormodel cards (page ??) giving full replicable information on the cor-\npora used to train them. Open-source models can specify their exact training data.\nRequirements that models are transparent in such ways is also in the process of being\nincorporated into the regulations of various national governments. 18 CHAPTER 10 \u2022 L ARGE LANGUAGE MODELS\n10.7 Summary\nThis chapter has introduced the large language model, and how it can be built out of\nthe transformer. Here\u2019s a summary of the main points that we covered:\n\u2022 Many NLP tasks\u2014such as question answering, summarization, sentiment,\nand machine translation\u2014can be cast as tasks of word prediction and hence\naddressed with Large language models.\n\u2022 Large language models are generally pretrained on large datasets of 100s of\nbillions of words generally scraped from the web.\n\u2022 These datasets need to be \ufb01ltered for quality and balanced for domains by\nupsampling and downsampling. Addressing some problems with pretraining\ndata, like toxicity, are open research problems.\n\u2022 The choice of which word to generate in large language models is generally\ndone by using a sampling algorithm.\n\u2022 Language models are evaluated by perplexity but there are also evaluations\nof accuracy downstream tasks, and ways to measure other factors like fairness\nand energy use.\n\u2022 There are various computational tricks for making large language models\nmore ef\ufb01cient, such as the CV cache andparameter-ef\ufb01cient \ufb01netuning .\n\u2022 Because of their ability to be used in so many ways, language models also\nhave the potential to cause harms. Some harms include hallucinations, bias,\nstereotypes, misinformation and propaganda, and violations of privacy and\ncopyright.\nBibliographical and Historical Notes\nAs we discussed in Chapter 3, the earliest language models were the n-gram lan-\nguage models developed (roughly simultaneously and independently) by Fred Je-\nlinek and colleagues at the IBM Thomas J. Watson Research Center, and James\nBaker at CMU. It was the Jelinek and the IBM team who \ufb01rst coined the term lan-\nguage model to mean a model of the way any kind of linguistic property (grammar,\nsemantics, discourse, speaker characteristics), in\ufb02uenced word sequence probabil-\nities (Jelinek et al., 1975). They contrasted the language model with the acoustic\nmodel which captured acoustic/phonetic characteristics of phone sequences.\nN-gram language models were very widely used over the next 30 years and more,\nacross a wide variety of NLP tasks like speech recognition and machine translations,\noften as one of multiple components of the model. The contexts for these n-gram\nmodels grew longer, with 5-gram models used quite commonly by very ef\ufb01cient LM\ntoolkits (Stolcke, 2002; Hea\ufb01eld, 2011).\nThe roots of the neural language model lie in multiple places. One was the\napplication in the 1990s, again in Jelinek\u2019s group at IBM Research, of discrimi-\nnative classi\ufb01ers to language models. Roni Rosenfeld in his dissertation (Rosen-\nfeld, 1992) \ufb01rst applied logistic regression (under the name maximum entropy or\nmaxent models) to language modeling in that IBM lab, and published a more fully\nformed version in Rosenfeld (1996). His model integrated various sorts of infor-\nmation in a logistic regression predictor, including n-gram information along with BIBLIOGRAPHICAL AND HISTORICAL NOTES 19\nother features from the context, including distant n-grams and pairs of associated\nwords called trigger pairs . Rosenfeld\u2019s model pre\ufb01gured modern language models\nby being a statistical word predictor trained in a self-supervised manner simply by\nlearning to predict upcoming words in a corpus.\nAnother was the \ufb01rst use of pretrained embeddings to model word meaning in the\nLSA/LSI models (Deerwester et al., 1988). Recall from the history section of Chap-\nter 6 that in LSA (latent semantic analysis) a term-document matrix was trained on a\ncorpus and then singular value decomposition was applied and the \ufb01rst 300 dimen-\nsions were used as a vector embedding to represent words. Landauer et al. (1997)\n\ufb01rst used the word \u201cembedding\u201d. In addition to their development of the idea of pre-\ntraining and of embeddings, the LSA community also developed ways to combine\nLSA embeddings with n-grams in an integrated language model (Bellegarda, 1997;\nCoccaro and Jurafsky, 1998).\nIn a very in\ufb02uential series of papers developing the idea of neural language\nmodels , (Bengio et al. 2000; Bengio et al. 2003; Bengio et al. 2006), Yoshua Ben-\ngio and colleagues drew on the central ideas of both these lines of self-supervised\nlanguage modeling work, (the discriminatively trained word predictor, and the pre-\ntrained embeddings). Like the maxent models of Rosenfeld, Bengio\u2019s model used\nthe next word in running text as its supervision signal. Like the LSA models, Ben-\ngio\u2019s model learned an embedding, but unlike the LSA models did it as part of the\nprocess of language modeling. The Bengio et al. (2003) model was a neural lan-\nguage model: a neural network that learned to predict the next word from prior\nwords, and did so via learning embeddings as part of the prediction process.\nThe neural language model was extended in various ways over the years, perhaps\nmost importantly in the form of the RNN language model of Mikolov et al. (2010)\nand Mikolov et al. (2011). The RNN language model was perhaps the \ufb01rst neural\nmodel that was accurate enough to surpass the performance of a traditional 5-gram\nlanguage model.\nSoon afterwards, Mikolov et al. (2013a) and Mikolov et al. (2013b) proposed\nto simply the hidden layer of these neural net language models to create pretrained\nword2vec word embeddings.\nThe static embedding models like LSA and word2vec instantiated a particular\nmodel of pretraining: a representation was trained on a pretraining dataset, and then\nthe representations could be used in further tasks. The \u2018Dai and Le (2015) and\n(Peters et al., 2018) reframed this idea by proposing models that were pretrained\nusing a language model objective, and then the identical model could be either frozen\nand directly applied for language modeling or further \ufb01netuned still using a language\nmodel objective. For example ELMo used a biLSTM self-supervised on a large\npretrained dataset using a language model objective, then \ufb01netuned on a domain-\nspeci\ufb01c dataset, and then froze the weights and added task-speci\ufb01c heads.\nTransformers were \ufb01rst applied as encoder-decoders (Vaswani et al., 2017) and\nthen to masked language modeling (Devlin et al., 2019) (as we\u2019ll see in Chapter 13\nand Chapter 11). Radford et al. (2019) then showed that the transformer-based au-\ntoregressive language model GPT2 could perform zero-shot on many NLP tasks like\nsummarization and question answering.\nThe technology used for transformer-based language models can also be applied\nto other domains and tasks, like vision, speech, and genetics. the term foundation\nmodel is sometimes used as a more general term for this use of large languagefoundation\nmodel\nmodel technology across domains and areas, when the elements we are computing\nover are not necessarily words. Bommasani et al. (2021) is a broad survey that 20 CHAPTER 10 \u2022 L ARGE LANGUAGE MODELS\nsketches the opportunities and risks of foundation models, with special attention to\nlarge language models. Bibliographical and Historical Notes 21\nBellegarda, J. R. 1997. A latent semantic analysis framework\nfor large-span language modeling. EUROSPEECH .\nBengio, Y ., R. Ducharme, and P. Vincent. 2000. A neural\nprobabilistic language model. NeurIPS .\nBengio, Y ., R. Ducharme, P. Vincent, and C. Jauvin. 2003.\nA neural probabilistic language model. JMLR , 3:1137\u2013\n1155.\nBengio, Y ., H. Schwenk, J.-S. Sen \u00b4ecal, F. Morin, and J.-L.\nGauvain. 2006. Neural probabilistic language models. In\nInnovations in Machine Learning , 137\u2013186. Springer.\nBommasani, R., D. A. Hudson, E. Adeli, R. Altman,\nS. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosse-\nlut, E. Brunskill, E. Brynjolfsson, S. Buch, D. Card,\nR. Castellon, N. S. Chatterji, A. S. Chen, K. A. Creel,\nJ. Davis, D. Demszky, C. Donahue, M. Doumbouya,\nE. Durmus, S. Ermon, J. Etchemendy, K. Ethayarajh,\nL. Fei-Fei, C. Finn, T. Gale, L. E. Gillespie, K. Goel,\nN. D. Goodman, S. Grossman, N. Guha, T. Hashimoto,\nP. Henderson, J. Hewitt, D. E. Ho, J. Hong, K. Hsu,\nJ. Huang, T. F. Icard, S. Jain, D. Jurafsky, P. Kalluri,\nS. Karamcheti, G. Keeling, F. Khani, O. Khattab, P. W.\nKoh, M. S. Krass, R. Krishna, R. Kuditipudi, A. Ku-\nmar, F. Ladhak, M. Lee, T. Lee, J. Leskovec, I. Lev-\nent, X. L. Li, X. Li, T. Ma, A. Malik, C. D. Manning,\nS. P. Mirchandani, E. Mitchell, Z. Munyikwa, S. Nair,\nA. Narayan, D. Narayanan, B. Newman, A. Nie, J. C.\nNiebles, H. Nilforoshan, J. F. Nyarko, G. Ogut, L. Orr,\nI. Papadimitriou, J. S. Park, C. Piech, E. Portelance,\nC. Potts, A. Raghunathan, R. Reich, H. Ren, F. Rong,\nY . H. Roohani, C. Ruiz, J. Ryan, C. R\u2019e, D. Sadigh,\nS. Sagawa, K. Santhanam, A. Shih, K. P. Srinivasan,\nA. Tamkin, R. Taori, A. W. Thomas, F. Tram `er, R. E.\nWang, W. Wang, B. Wu, J. Wu, Y . Wu, S. M. Xie, M. Ya-\nsunaga, J. You, M. A. Zaharia, M. Zhang, T. Zhang,\nX. Zhang, Y . Zhang, L. Zheng, K. Zhou, and P. Liang.\n2021. On the opportunities and risks of foundation mod-\nels.ArXiv .\nBrown, T., B. Mann, N. Ryder, M. Subbiah, J. Kaplan,\nP. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry,\nA. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger,\nT. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu,\nC. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin,\nS. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish,\nA. Radford, I. Sutskever, and D. Amodei. 2020. Language\nmodels are few-shot learners. NeurIPS , volume 33.\nCarlini, N., F. Tramer, E. Wallace, M. Jagielski, A. Herbert-\nV oss, K. Lee, A. Roberts, T. Brown, D. Song, U. Er-\nlingsson, et al. 2021. Extracting training data from large\nlanguage models. 30th USENIX Security Symposium\n(USENIX Security 21) .\nCheng, M., E. Durmus, and D. Jurafsky. 2023. Marked per-\nsonas: Using natural language prompts to measure stereo-\ntypes in language models. ACL.\nCoccaro, N. and D. Jurafsky. 1998. Towards better integra-\ntion of semantic predictors in statistical language model-\ning.ICSLP .\nDai, A. M. and Q. V . Le. 2015. Semi-supervised sequence\nlearning. NeurIPS .\nDeerwester, S. C., S. T. Dumais, G. W. Furnas, R. A. Harsh-\nman, T. K. Landauer, K. E. Lochbaum, and L. Streeter.\n1988. Computer information retrieval using latent seman-\ntic structure: US Patent 4,839,853.Devlin, J., M.-W. Chang, K. Lee, and K. Toutanova. 2019.\nBERT: Pre-training of deep bidirectional transformers for\nlanguage understanding. NAACL HLT .\nDodge, J., S. Gururangan, D. Card, R. Schwartz, and N. A.\nSmith. 2019. Show your work: Improved reporting of\nexperimental results. EMNLP .\nDodge, J., M. Sap, A. Marasovi \u00b4c, W. Agnew, G. Ilharco,\nD. Groeneveld, M. Mitchell, and M. Gardner. 2021. Doc-\numenting large webtext corpora: A case study on the\ncolossal clean crawled corpus. EMNLP .\nEthayarajh, K. and D. Jurafsky. 2020. Utility is in the eye of\nthe user: A critique of NLP leaderboards. EMNLP .\nGao, L., T. Hoppe, A. Thite, S. Biderman, C. Foster,\nN. Nabeshima, S. Black, J. Phang, S. Presser, L. Golding,\nH. He, and C. Leahy. 2020. The Pile: An 800GB dataset\nof diverse text for language modeling. ArXiv preprint.\nGehman, S., S. Gururangan, M. Sap, Y . Choi, and N. A.\nSmith. 2020. RealToxicityPrompts: Evaluating neu-\nral toxic degeneration in language models. Findings of\nEMNLP .\nGururangan, S., A. Marasovi \u00b4c, S. Swayamdipta, K. Lo,\nI. Beltagy, D. Downey, and N. A. Smith. 2020. Don\u2019t\nstop pretraining: Adapt language models to domains and\ntasks. ACL.\nHashimoto, T., M. Srivastava, H. Namkoong, and P. Liang.\n2018. Fairness without demographics in repeated loss\nminimization. ICML .\nHea\ufb01eld, K. 2011. KenLM: Faster and smaller language\nmodel queries. Workshop on Statistical Machine Trans-\nlation .\nHenderson, P., J. Hu, J. Romoff, E. Brunskill, D. Jurafsky,\nand J. Pineau. 2020. Towards the systematic reporting\nof the energy and carbon footprints of machine learning.\nJournal of Machine Learning Research , 21(248):1\u201343.\nHenderson, P., X. Li, D. Jurafsky, T. Hashimoto, M. A. Lem-\nley, and P. Liang. 2023. Foundation models and fair use.\nJMLR , 24(400):1\u201379.\nHenderson, P., K. Sinha, N. Angelard-Gontier, N. R. Ke,\nG. Fried, R. Lowe, and J. Pineau. 2017. Ethical chal-\nlenges in data-driven dialogue systems. AAAI/ACM AI\nEthics and Society Conference .\nHermann, K. M., T. Ko \u02c7cisk\u00b4y, E. Grefenstette, L. Espeholt,\nW. Kay, M. Suleyman, and P. Blunsom. 2015. Teaching\nmachines to read and comprehend. NeurIPS .\nHoltzman, A., J. Buys, L. Du, M. Forbes, and Y . Choi. 2020.\nThe curious case of neural text degeneration. ICLR .\nJelinek, F., R. L. Mercer, and L. R. Bahl. 1975. Design of a\nlinguistic statistical decoder for the recognition of contin-\nuous speech. IEEE Transactions on Information Theory ,\nIT-21(3):250\u2013256.\nKaplan, J., S. McCandlish, T. Henighan, T. B. Brown,\nB. Chess, R. Child, S. Gray, A. Radford, J. Wu, and\nD. Amodei. 2020. Scaling laws for neural language mod-\nels. ArXiv preprint.\nKiela, D., M. Bartolo, Y . Nie, D. Kaushik, A. Geiger, Z. Wu,\nB. Vidgen, G. Prasad, A. Singh, P. Ringshia, et al. 2021.\nDynabench: Rethinking benchmarking in nlp. NAACL\nHLT. 22 Chapter 10 \u2022 Large Language Models\nLandauer, T. K., D. Laham, B. Rehder, and M. E. Schreiner.\n1997. How well can passage meaning be derived with-\nout using word order? A comparison of Latent Semantic\nAnalysis and humans. COGSCI .\nLiang, P., R. Bommasani, T. Lee, D. Tsipras, D. Soylu,\nM. Yasunaga, Y . Zhang, D. Narayanan, Y . Wu, A. Ku-\nmar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cos-\ngrove, C. D. Manning, C. R \u00b4e, D. Acosta-Navas, D. A.\nHudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong,\nH. Ren, H. Yao, J. Wang, K. Santhanam, L. Orr, L. Zheng,\nM. Yuksekgonul, M. Suzgun, N. Kim, N. Guha, N. Chat-\nterji, O. Khattab, P. Henderson, Q. Huang, R. Chi, S. M.\nXie, S. Santurkar, S. Ganguli, T. Hashimoto, T. Icard,\nT. Zhang, V . Chaudhary, W. Wang, X. Li, Y . Mai,\nY . Zhang, and Y . Koreeda. 2023. Holistic evaluation of\nlanguage models. Transactions on Machine Learning Re-\nsearch .\nLlama Team. 2024. The llama 3 herd of models.\nLongpre, S., R. Mahari, A. Lee, C. Lund, H. Oderinwale,\nW. Brannon, N. Saxena, N. Obeng-Marnu, T. South,\nC. Hunter, et al. 2024a. Consent in crisis: The rapid de-\ncline of the ai data commons. ArXiv preprint.\nLongpre, S., G. Yauney, E. Reif, K. Lee, A. Roberts,\nB. Zoph, D. Zhou, J. Wei, K. Robinson, D. Mimno, and\nD. Ippolito. 2024b. A pretrainer\u2019s guide to training data:\nMeasuring the effects of data age, domain coverage, qual-\nity, & toxicity. NAACL HLT .\nMcGuf\ufb01e, K. and A. Newhouse. 2020. The radicalization\nrisks of GPT-3 and advanced neural language models.\nArXiv preprint arXiv:2009.06807.\nMikolov, T., K. Chen, G. S. Corrado, and J. Dean. 2013a. Ef-\n\ufb01cient estimation of word representations in vector space.\nICLR 2013 .\nMikolov, T., M. Kara\ufb01 \u00b4at, L. Burget, J. \u02c7Cernock `y, and\nS. Khudanpur. 2010. Recurrent neural network based lan-\nguage model. INTERSPEECH .\nMikolov, T., S. Kombrink, L. Burget, J. H. \u02c7Cernock `y, and\nS. Khudanpur. 2011. Extensions of recurrent neural net-\nwork language model. ICASSP .\nMikolov, T., I. Sutskever, K. Chen, G. S. Corrado, and\nJ. Dean. 2013b. Distributed representations of words and\nphrases and their compositionality. NeurIPS .\nNadeem, M., A. Bethke, and S. Reddy. 2021. StereoSet:\nMeasuring stereotypical bias in pretrained language mod-\nels.ACL.\nNallapati, R., B. Zhou, C. dos Santos, C \u00b8 . Gulc \u00b8ehre, and\nB. Xiang. 2016. Abstractive text summarization using\nsequence-to-sequence RNNs and beyond. CoNLL .\nOlteanu, A., F. Diaz, and G. Kazai. 2020. When are search\ncompletion suggestions problematic? CSCW .\nParrish, A., A. Chen, N. Nangia, V . Padmakumar, J. Phang,\nJ. Thompson, P. M. Htut, and S. Bowman. 2022. BBQ: A\nhand-built bias benchmark for question answering. Find-\nings of ACL 2022 .\nPeters, M., M. Neumann, M. Iyyer, M. Gardner, C. Clark,\nK. Lee, and L. Zettlemoyer. 2018. Deep contextualized\nword representations. NAACL HLT .\nRadford, A., J. Wu, R. Child, D. Luan, D. Amodei, and\nI. Sutskever. 2019. Language models are unsupervised\nmultitask learners. OpenAI tech report.Raffel, C., N. Shazeer, A. Roberts, K. Lee, S. Narang,\nM. Matena, Y . Zhou, W. Li, and P. J. Liu. 2020. Exploring\nthe limits of transfer learning with a uni\ufb01ed text-to-text\ntransformer. JMLR , 21(140):1\u201367.\nRawls, J. 2001. Justice as fairness: A restatement . Harvard\nUniversity Press.\nRosenfeld, R. 1992. Adaptive Statistical Language Mod-\neling: A Maximum Entropy Approach . Ph.D. thesis,\nCarnegie Mellon University.\nRosenfeld, R. 1996. A maximum entropy approach to adap-\ntive statistical language modeling. Computer Speech and\nLanguage , 10:187\u2013228.\nSagawa, S., P. W. Koh, T. B. Hashimoto, and P. Liang. 2020.\nDistributionally robust neural networks for group shifts:\nOn the importance of regularization for worst-case gener-\nalization. ICLR .\nSheng, E., K.-W. Chang, P. Natarajan, and N. Peng. 2019.\nThe woman worked as a babysitter: On biases in language\ngeneration. EMNLP .\nSoldaini, L., R. Kinney, A. Bhagia, D. Schwenk, D. Atkin-\nson, R. Authur, B. Bogin, K. Chandu, J. Dumas,\nY . Elazar, V . Hofmann, A. H. Jha, S. Kumar, L. Lucy,\nX. Lyu, N. Lambert, I. Magnusson, J. Morrison,\nN. Muennighoff, A. Naik, C. Nam, M. E. Peters,\nA. Ravichander, K. Richardson, Z. Shen, E. Strubell,\nN. Subramani, O. Tafjord, P. Walsh, L. Zettlemoyer, N. A.\nSmith, H. Hajishirzi, I. Beltagy, D. Groeneveld, J. Dodge,\nand K. Lo. 2024. Dolma: An open corpus of three trillion\ntokens for language model pretraining research. ArXiv\npreprint.\nStolcke, A. 2002. SRILM \u2013 an extensible language modeling\ntoolkit. ICSLP .\nStrubell, E., A. Ganesh, and A. McCallum. 2019. Energy\nand policy considerations for deep learning in NLP. ACL.\nVaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones,\nA. N. Gomez, \u0141. Kaiser, and I. Polosukhin. 2017. Atten-\ntion is all you need. NeurIPS .\nXu, A., E. Pathak, E. Wallace, S. Gururangan, M. Sap,\nand D. Klein. 2021. Detoxifying language models risks\nmarginalizing minority voices. NAACL HLT .", "LLM24aug.pdf": "Large Language ModelsIntroduction to Large Language Models Language models\u2022Remember the simple n-gram language model\u2022Assigns probabilities to sequences of words\u2022Generate text by sampling possible next words\u2022Is trained on counts computed from lots of text\u2022Large language models are similar and different:\u2022Assigns probabilities to sequences of words\u2022Generate text by sampling possible next words\u2022Are trained by learning to guess the next word Large language models\u2022Even through pretrained only to predict words\u2022Learn a lot of useful language knowledge\u2022Since training on a lot of text Three architectures for large language modelsDecoders   Encoders     Encoder-decodersGPT, Claude,  BERT family,  Flan-T5, WhisperLlama    HuBERTMixtralPretraining for three types of architecturesThe neural architecture influences the type of pretraining, and natural use cases.\n32Decoders\u2022Language models! What we\u2019ve seen so far.\u2022Nice to generate from; can\u2019t condition on future wordsEncoders\u2022Gets bidirectional context \u2013 can condition on future!\u2022How do we train them to build strong representations?Encoder-Decoders\u2022Good parts of decoders and encoders?\u2022What\u2019s the best way to pretrain them?Pretraining for three types of architecturesThe neural architecture influences the type of pretraining, and natural use cases.\n32Decoders\u2022Language models! What we\u2019ve seen so far.\u2022Nice to generate from; can\u2019t condition on future wordsEncoders\u2022Gets bidirectional context \u2013 can condition on future!\u2022How do we train them to build strong representations?Encoder-Decoders\u2022Good parts of decoders and encoders?\u2022What\u2019s the best way to pretrain them?Pretraining for three types of architecturesThe neural architecture influences the type of pretraining, and natural use cases.\n32Decoders\u2022Language models! What we\u2019ve seen so far.\u2022Nice to generate from; can\u2019t condition on future wordsEncoders\u2022Gets bidirectional context \u2013 can condition on future!\u2022How do we train them to build strong representations?Encoder-Decoders\u2022Good parts of decoders and encoders?\u2022What\u2019s the best way to pretrain them? EncodersMany varieties!\u2022Popular: Masked Language Models (MLMs)\u2022BERT family\u2022Trained by predicting words from surrounding words on both sides\u2022Are usually finetuned (trained on supervised data) for classification tasks.Pretraining for three types of architecturesThe neural architecture influences the type of pretraining, and natural use cases.\n32Decoders\u2022Language models! What we\u2019ve seen so far.\u2022Nice to generate from; can\u2019t condition on future wordsEncoders\u2022Gets bidirectional context \u2013 can condition on future!\u2022How do we train them to build strong representations?Encoder-Decoders\u2022Good parts of decoders and encoders?\u2022What\u2019s the best way to pretrain them? Encoder-Decoders\u2022Trained to map from one sequence to another\u2022Very popular for:\u2022machine translation (map from one language to another)\u2022speech recognition (map from acoustics to words)Pretraining for three types of architecturesThe neural architecture influences the type of pretraining, and natural use cases.\n32Decoders\u2022Language models! What we\u2019ve seen so far.\u2022Nice to generate from; can\u2019t condition on future wordsEncoders\u2022Gets bidirectional context \u2013 can condition on future!\u2022How do we train them to build strong representations?Encoder-Decoders\u2022Good parts of decoders and encoders?\u2022What\u2019s the best way to pretrain them? Large Language ModelsIntroduction to Large Language Models Large Language ModelsLarge Language Models: What tasks can they do? Big ideaMany tasks can be turned into tasks of predicting words! This lecture: decoder-only modelsAlso called:\u2022Causal LLMs\u2022Autoregressive LLMs\u2022Left-to-right LLMs\u2022Predict words left to rightPretraining for three types of architecturesThe neural architecture influences the type of pretraining, and natural use cases.\n32Decoders\u2022Language models! What we\u2019ve seen so far.\u2022Nice to generate from; can\u2019t condition on future wordsEncoders\u2022Gets bidirectional context \u2013 can condition on future!\u2022How do we train them to build strong representations?Encoder-Decoders\u2022Good parts of decoders and encoders?\u2022What\u2019s the best way to pretrain them? Conditional Generation: Generating text conditioned on previous text!\nPre\ufb01x TextCompletion Text\nEncoderTransformerBlocksSoftmax\nlongall\nandthanksforallthe\nthe\u2026UUUnencoder layerLanguage ModelingHeadlogits\nSo\nEi+\nEi+\nEi+\nEi+\nEi+\nEi+\nEi+\u2026 Many practical NLP tasks can be cast as word prediction!Sentiment analysis: \u201cI like Jackie Chan\u201d1.We give the language model this string:The sentiment of the sentence \"I like Jackie Chan\" is: 2.And see what word it thinks comes next:10.1\u2022LARGELANGUAGEMODELS WITHTRANSFORMERS3\nPre\ufb01x TextCompletion Text\nEncoderTransformerBlocksSoftmax\nlongall\nandthanksforallthe\nthe\u2026UUUnencoder layerLanguage ModelingHeadlogits\nSo\nEi+\nEi+\nEi+\nEi+\nEi+\nEi+\nEi+\u2026\nFigure 10.1Left-to-right (also called autoregressive) text completion with transformer-based large languagemodels. As each token is generated, it gets added onto the context as a pre\ufb01x for generating the next token.word \u201cnegative\u201d to see which is higher:P(positive|The sentiment of the sentence \u2018\u2018I like Jackie Chan\" is:)P(negative|The sentiment of the sentence \u2018\u2018I like Jackie Chan\" is:)If the word \u201cpositive\u201d is more probable, we say the sentiment of the sentence ispositive, otherwise we say the sentiment is negative.We can also cast more complex tasks as word prediction. Consider questionanswering, in which the system is given a question (for example a question witha simple factual answer) and must give a textual answer; we introduce this task indetail in Chapter 15. We can cast the task of question answering as word predictionby giving a language model a question and a token likeA:suggesting that an answershould come next:Q: Who wrote the book \u2018\u2018The Origin of Species\"? A:If we ask a language model to compute the probability distribution over possiblenext words given this pre\ufb01x:P(w|Q: Who wrote the book \u2018\u2018The Origin of Species\"? A:)and look at which wordswhave high probabilities, we might expect to see thatCharlesis very likely, and then if we chooseCharlesand continue and askP(w|Q: Who wrote the book \u2018\u2018The Origin of Species\"? A: Charles)we might now see thatDarwinis the most probable token, and select it.Conditional generation can even be used to accomplish tasks that must generatelonger responses. Consider the task oftext summarization, which is to take a longtextsummarizationtext, such as a full-length article, and produce an effective shorter summary of it. Wecan cast summarization as language modeling by giving a large language model atext, and follow the text by a token liketl;dr; this token is short for something like Framing lots of tasks as conditional generationQA: \u201cWho wrote The Origin of Species\u201d1.We give the language model this string:2.And see what word it thinks comes next:3.And iterate:20CHAPTER10\u2022TRANSFORMERS ANDLARGELANGUAGEMODELS\nPre\ufb01x TextCompletion Text\nInputEmbeddingsTransformerBlocksSample from Softmax\nSolongall\nandthanksforallthe\nthe\u2026linear layer\nFigure 10.15Autoregressive text completion with transformer-based large language models.word \u201cnegative\u201d to see which is higher:P(positive|The sentiment of the sentence \u201cI like Jackie Chan\u201d is:)P(negative|The sentiment of the sentence \u201cI like Jackie Chan\u201d is:)If the word \u201cpositive\u201d is more probable, we say the sentiment of the sentence ispositive, otherwise we say the sentiment is negative.We can also cast more complex tasks as word prediction. Consider the taskof answering simple questions, a task we return to in Chapter 14. In this task thesystem is given some question and must give a textual answer. We can cast the taskof question answering as word prediction by giving a language model a question anda token likeA:suggesting that an answer should come next:Q: Who wrote the book \u2018\u2018The Origin of Species\"? A:If we ask a language model to computeP(w|Q: Who wrote the book \u201cThe Origin of Species\u201d? A:)and look at which wordswhave high probabilities, we might expect to see thatCharlesis very likely, and then if we chooseCharlesand continue and askP(w|Q: Who wrote the book \u201cThe Origin of Species\u201d? A: Charles)we might now see thatDarwinis the most probable word, and select it.Conditional generation can even be used to accomplish tasks that must generatelonger responses. Consider the task oftext summarization, which is to take a longtextsummarizationtext, such as a full-length article, and produce an effective shorter summary of it.We can cast summarization as language modeling by giving a large language modela text, and follow the text by a token liketl;dr; this token is short for somethinglike \u2018too long; don\u2019t read\u2019 and in recent years people often use this token, especiallyin informal work emails, when they are going to give a short summary. We canthen do conditional generation: give the language model this pre\ufb01x, and then ask10.1\u2022LARGELANGUAGEMODELS WITHTRANSFORMERS3\nPre\ufb01x TextCompletion Text\nEncoderTransformerBlocksSoftmax\nlongall\nandthanksforallthe\nthe\u2026UUUnencoder layerLanguage ModelingHeadlogits\nSo\nEi+\nEi+\nEi+\nEi+\nEi+\nEi+\nEi+\u2026\nFigure 10.1Left-to-right (also called autoregressive) text completion with transformer-based large languagemodels. As each token is generated, it gets added onto the context as a pre\ufb01x for generating the next token.word \u201cnegative\u201d to see which is higher:P(positive|The sentiment of the sentence \u2018\u2018I like Jackie Chan\" is:)P(negative|The sentiment of the sentence \u2018\u2018I like Jackie Chan\" is:)If the word \u201cpositive\u201d is more probable, we say the sentiment of the sentence ispositive, otherwise we say the sentiment is negative.We can also cast more complex tasks as word prediction. Consider questionanswering, in which the system is given a question (for example a question witha simple factual answer) and must give a textual answer; we introduce this task indetail in Chapter 15. We can cast the task of question answering as word predictionby giving a language model a question and a token likeA:suggesting that an answershould come next:Q: Who wrote the book \u2018\u2018The Origin of Species\"? A:If we ask a language model to compute the probability distribution over possiblenext words given this pre\ufb01x:P(w|Q: Who wrote the book \u2018\u2018The Origin of Species\"? A:)and look at which wordswhave high probabilities, we might expect to see thatCharlesis very likely, and then if we chooseCharlesand continue and askP(w|Q: Who wrote the book \u2018\u2018The Origin of Species\"? A: Charles)we might now see thatDarwinis the most probable token, and select it.Conditional generation can even be used to accomplish tasks that must generatelonger responses. Consider the task oftext summarization, which is to take a longtextsummarizationtext, such as a full-length article, and produce an effective shorter summary of it. Wecan cast summarization as language modeling by giving a large language model atext, and follow the text by a token liketl;dr; this token is short for something like10.1\u2022LARGELANGUAGEMODELS WITHTRANSFORMERS3\nPre\ufb01x TextCompletion Text\nEncoderTransformerBlocksSoftmax\nlongall\nandthanksforallthe\nthe\u2026UUUnencoder layerLanguage ModelingHeadlogits\nSo\nEi+\nEi+\nEi+\nEi+\nEi+\nEi+\nEi+\u2026\nFigure 10.1Left-to-right (also called autoregressive) text completion with transformer-based large languagemodels. As each token is generated, it gets added onto the context as a pre\ufb01x for generating the next token.word \u201cnegative\u201d to see which is higher:P(positive|The sentiment of the sentence \u2018\u2018I like Jackie Chan\" is:)P(negative|The sentiment of the sentence \u2018\u2018I like Jackie Chan\" is:)If the word \u201cpositive\u201d is more probable, we say the sentiment of the sentence ispositive, otherwise we say the sentiment is negative.We can also cast more complex tasks as word prediction. Consider questionanswering, in which the system is given a question (for example a question witha simple factual answer) and must give a textual answer; we introduce this task indetail in Chapter 15. We can cast the task of question answering as word predictionby giving a language model a question and a token likeA:suggesting that an answershould come next:Q: Who wrote the book \u2018\u2018The Origin of Species\"? A:If we ask a language model to compute the probability distribution over possiblenext words given this pre\ufb01x:P(w|Q: Who wrote the book \u2018\u2018The Origin of Species\"? A:)and look at which wordswhave high probabilities, we might expect to see thatCharlesis very likely, and then if we chooseCharlesand continue and askP(w|Q: Who wrote the book \u2018\u2018The Origin of Species\"? A: Charles)we might now see thatDarwinis the most probable token, and select it.Conditional generation can even be used to accomplish tasks that must generatelonger responses. Consider the task oftext summarization, which is to take a longtextsummarizationtext, such as a full-length article, and produce an effective shorter summary of it. Wecan cast summarization as language modeling by giving a large language model atext, and follow the text by a token liketl;dr; this token is short for something like Summarization4CHAPTER10\u2022LARGELANGUAGEMODELS\u2018too long; didn\u2019t read\u2019 and in recent years people often use this token, especially ininformal work emails, when they are going to give a short summary. Since this tokenis suf\ufb01ciently frequent in language model training data, language models have seenmany texts in which the token occurs before a summary, and hence will interpret thetoken as instructions to generate a summary. We can then do conditional generation:give the language model this pre\ufb01x, and then have it generate the following words,one by one, and take the entire response as a summary. Fig.10.2shows an exampleof a text and a human-produced summary from a widely-used summarization corpusconsisting of CNN and Daily Mirror news articles.Original ArticleThe only thing crazier than a guy in snowbound Massachusetts boxing up the powdery white stuffand offering it for sale online? People are actually buying it. For $89, self-styled entrepreneurKyle Waring will ship you 6 pounds of Boston-area snow in an insulated Styrofoam box \u2013 enoughfor 10 to 15 snowballs, he says.But not if you live in New England or surrounding states. \u201cWe will not ship snow to any statesin the northeast!\u201d says Waring\u2019s website, ShipSnowYo.com. \u201cWe\u2019re in the business of expungingsnow!\u201dHis website and social media accounts claim to have \ufb01lled more than 133 orders for snow \u2013 morethan 30 on Tuesday alone, his busiest day yet. With more than 45 total inches, Boston has set arecord this winter for the snowiest month in its history. Most residents see the huge piles of snowchoking their yards and sidewalks as a nuisance, but Waring saw an opportunity.According to Boston.com, it all started a few weeks ago, when Waring and his wife were shov-eling deep snow from their yard in Manchester-by-the-Sea, a coastal suburb north of Boston. Hejoked about shipping the stuff to friends and family in warmer states, and an idea was born. [...]SummaryKyle Waring will ship you 6 pounds of Boston-area snow in an insulated Styrofoam box \u2013 enoughfor 10 to 15 snowballs, he says. But not if you live in New England or surrounding states.Figure 10.2Excerpt from a sample article and its summary from the CNN/Daily Mail summarization corpus(Hermann et al.,2015), (Nallapati et al.,2016).If we take this full article and append the tokentl;dr, we can use this as the con-text to prime the generation process to produce a summary as illustrated in Fig.10.3.Again, what makes transformers able to succeed at this task (as compared, say, tothe primitive n-gram language model) is that attention can incorporate informationfrom the large context window, giving the model access to the original article as wellas to the newly generated text throughout the process.Which words do we generate at each step? One simple way to generate wordsis to always generate the most likely word given the context. Generating the mostlikely word given the context is calledgreedy decoding. A greedy algorithm is onegreedydecodingthat make a choice that is locally optimal, whether or not it will turn out to havebeen the best choice with hindsight. Thus in greedy decoding, at each time step ingeneration, the outputytis chosen by computing the probability for each possibleoutput (every word in the vocabulary) and then choosing the highest probabilityword (the argmax):\u02c6wt=argmaxw2VP(w|w<t)(10.1)In practice, however, we don\u2019t use greedy decoding with large language models.A major problem with greedy decoding is that because the words it chooses are (byde\ufb01nition) extremely predictable, the resulting text is generic and often quite repeti-tive. Indeed, greedy decoding is so predictable that it is deterministic; if the context4CHAPTER10\u2022LARGELANGUAGEMODELS\u2018too long; didn\u2019t read\u2019 and in recent years people often use this token, especially ininformal work emails, when they are going to give a short summary. Since this tokenis suf\ufb01ciently frequent in language model training data, language models have seenmany texts in which the token occurs before a summary, and hence will interpret thetoken as instructions to generate a summary. We can then do conditional generation:give the language model this pre\ufb01x, and then have it generate the following words,one by one, and take the entire response as a summary. Fig.10.2shows an exampleof a text and a human-produced summary from a widely-used summarization corpusconsisting of CNN and Daily Mirror news articles.Original ArticleThe only thing crazier than a guy in snowbound Massachusetts boxing up the powdery white stuffand offering it for sale online? People are actually buying it. For $89, self-styled entrepreneurKyle Waring will ship you 6 pounds of Boston-area snow in an insulated Styrofoam box \u2013 enoughfor 10 to 15 snowballs, he says.But not if you live in New England or surrounding states. \u201cWe will not ship snow to any statesin the northeast!\u201d says Waring\u2019s website, ShipSnowYo.com. \u201cWe\u2019re in the business of expungingsnow!\u201dHis website and social media accounts claim to have \ufb01lled more than 133 orders for snow \u2013 morethan 30 on Tuesday alone, his busiest day yet. With more than 45 total inches, Boston has set arecord this winter for the snowiest month in its history. Most residents see the huge piles of snowchoking their yards and sidewalks as a nuisance, but Waring saw an opportunity.According to Boston.com, it all started a few weeks ago, when Waring and his wife were shov-eling deep snow from their yard in Manchester-by-the-Sea, a coastal suburb north of Boston. Hejoked about shipping the stuff to friends and family in warmer states, and an idea was born. [...]SummaryKyle Waring will ship you 6 pounds of Boston-area snow in an insulated Styrofoam box \u2013 enoughfor 10 to 15 snowballs, he says. But not if you live in New England or surrounding states.Figure 10.2Excerpt from a sample article and its summary from the CNN/Daily Mail summarization corpus(Hermann et al.,2015), (Nallapati et al.,2016).If we take this full article and append the tokentl;dr, we can use this as the con-text to prime the generation process to produce a summary as illustrated in Fig.10.3.Again, what makes transformers able to succeed at this task (as compared, say, tothe primitive n-gram language model) is that attention can incorporate informationfrom the large context window, giving the model access to the original article as wellas to the newly generated text throughout the process.Which words do we generate at each step? One simple way to generate wordsis to always generate the most likely word given the context. Generating the mostlikely word given the context is calledgreedy decoding. A greedy algorithm is onegreedydecodingthat make a choice that is locally optimal, whether or not it will turn out to havebeen the best choice with hindsight. Thus in greedy decoding, at each time step ingeneration, the outputytis chosen by computing the probability for each possibleoutput (every word in the vocabulary) and then choosing the highest probabilityword (the argmax):\u02c6wt=argmaxw2VP(w|w<t)(10.1)In practice, however, we don\u2019t use greedy decoding with large language models.A major problem with greedy decoding is that because the words it chooses are (byde\ufb01nition) extremely predictable, the resulting text is generic and often quite repeti-tive. Indeed, greedy decoding is so predictable that it is deterministic; if the contextOriginal\nSummary LLMs for summarization (using  tl;dr)\nOriginal StoryGenerated Summary\n\u2026ideaKyle\nwasborn.KyleWaring\nWaringonlyThe\u2026will\nDelimiterwillUUU\ntl;drLM Head\nE\nE\nE\nE\nE\nE\nE\nE\u2026 Large Language ModelsLarge Language Models: What tasks can they do? Large Language ModelsSampling for LLM Generation Decoding and SamplingThis task of choosing a word to generate based on the model\u2019s probabilities is called decoding. The most common method for decoding in LLMs: sampling. Sampling from a model\u2019s distribution over words:\u2022choose random words according to their probability assigned by the model. After each token we\u2019ll sample words to generate according to their probability conditioned on our previous choices, \u2022A transformer language model will give the probability Random sampling6CHAPTER10\u2022LARGELANGUAGEMODELSas de\ufb01ned by the model. Thus we are more likely to generate words that the modelthinks have a high probability in the context and less likely to generate words thatthe model thinks have a low probability.We saw back in Chapter 3 on page??how to generate text from a unigram lan-guage model , by repeatedly randomly sampling words according to their probabilityuntil we either reach a pre-determined length or select the end-of-sentence token. Togenerate text from a trained transformer language model we\u2019ll just generalize thismodel a bit: at each step we\u2019ll sample words according to their probabilitycondi-tioned on our previous choices, and we\u2019ll use a transformer language model as theprobability model that tells us this probability.We can formalize this algorithm for generating a sequence of wordsW=w1,w2,...,wNuntil we hit the end-of-sequence token, usingx\u21e0p(x)to mean \u2018choosexby sam-pling from the distributionp(x):i 1wi\u21e0p(w)whilewi!= EOSi i+1wi\u21e0p(wi|w<i)The algorithm above is calledrandom sampling, and it turns out random sam-randomsamplingpling doesn\u2019t work well enough. The problem is that even though random samplingis mostly going to generate sensible, high-probable words, there are many odd, low-probability words in the tail of the distribution, and even though each one is low-probability, if you add up all the rare words, they constitute a large enough portionof the distribution that they get chosen often enough to result in generating weirdsentences. For this reason, instead of random sampling, we usually use samplingmethods that avoid generating the very unlikely words.The sampling methods we introduce below each have parameters that enabletrading off two important factors in generation:qualityanddiversity. Methodsthat emphasize the most probable words tend to produce generations that are ratedby people as more accurate, more coherent, and more factual, but also more boringand more repetitive. Methods that give a bit more weight to the middle-probabilitywords tend to be more creative and more diverse, but less factual and more likely tobe incoherent or otherwise low-quality.10.2.1 Top-ksamplingTop-k samplingis a simple generalization of greedy decoding. Instead of choosingtop-k samplingthe single most probable word to generate, we \ufb01rst truncate the distribution to thetopkmost likely words, renormalize to produce a legitimate probability distribution,and then randomly sample from within thesekwords according to their renormalizedprobabilities. More formally:1.Choose in advance a number of wordsk2.For each word in the vocabularyV, use the language model to compute thelikelihood of this word given the contextp(wt|w<t)3.Sort the words by their likelihood, and throw away any word that is not one ofthe topkmost probable words.4.Renormalize the scores of thekwords to be a legitimate probability distribu-tion. Random sampling doesn't work very wellEven though random sampling mostly generate sensible, high-probable words, There are many odd, low- probability words in the tail of the distribution Each one is low- probability but added up they constitute a large portion of the distribution So they get picked enough to generate weird sentences Factors in word sampling: quality and diversityEmphasize high-probability words  + quality: more  accurate, coherent, and factual, - diversity: boring, repetitive. Emphasize middle-probability words + diversity: more creative, diverse, - quality: less factual, incoherent Top-k sampling:1. Choose # of words k 2. For each word in the vocabulary V , use the language model to compute the likelihood of this word given the context p(wt |w<t ) 3. Sort the words by likelihood, keep only the top k most probable words. 4. Renormalize the scores of the k words to be a legitimate probability distribution. 5. Randomly sample a word from within these remaining k most-probable words according to its probability.  Top-p sampling (= nucleus sampling)Problem with top-k:  k is fixed so may cover very different amounts of probability mass in different situationsIdea: Instead, keep the top p percent of the probability massGiven a distribution P(wt |w<t ), the top-p vocabulary V ( p) is the smallest set of words such that Holtzman et al., 2020 10.2\u2022SAMPLING FORLLM GENERATION75.Randomly sample a word from within these remainingkmost-probable wordsaccording to its probability.Whenk=1, top-ksampling is identical to greedy decoding. Settingkto a largernumber than 1 leads us to sometimes select a word which is not necessarily the mostprobable, but is still probable enough, and whose choice results in generating morediverse but still high-enough-quality text.10.2.2 Nucleus or top-psamplingOne problem with top-ksampling is thatkis \ufb01xed, but the shape of the probabilitydistribution over words differs in different contexts. If we setk=10, sometimesthe top 10 words will be very likely and include most of the probability mass, butother times the probability distribution will be \ufb02atter and the top 10 words will onlyinclude a small part of the probability mass.An alternative, calledtop-p samplingornucleus sampling(Holtzman et al.,top-p sampling2020), is to keep not the topkwords, but the topppercent of the probability mass.The goal is the same; to truncate the distribution to remove the very unlikely words.But by measuring probability rather than the number of words, the hope is that themeasure will be more robust in very different contexts, dynamically increasing anddecreasing the pool of word candidates.Given a distributionP(wt|w<t), the top-pvocabularyV(p)is the smallest set ofwords such thatXw2V(p)P(w|w<t)\u0000p.(10.2)10.2.3 Temperature samplingIntemperature sampling, we don\u2019t truncate the distribution, but instead reshapetemperaturesamplingit. The intuition for temperature sampling comes from thermodynamics, where asystem at a high temperature is very \ufb02exible and can explore many possible states,while a system at a lower temperature is likely to explore a subset of lower energy(better) states. In low-temperature sampling, we smoothly increase the probabilityof the most probable words and decrease the probability of the rare words.We implement this intuition by simply dividing the logit by a temperature param-etertbefore we normalize it by passing it through the softmax. In low-temperaturesampling,t2(0,1]. Thus instead of computing the probability distribution over thevocabulary directly from the logit as in the following (repeated from (??)):y=softmax(u)(10.3)we instead \ufb01rst divide the logits byt, computing the probability vectoryasy=softmax(u/t)(10.4)Why does this work? Whentis close to 1 the distribution doesn\u2019t change much.But the lowertis, the larger the scores being passed to the softmax (dividing by asmaller fractiont\uf8ff1 results in making each score larger). Recall that one of theuseful properties of a softmax is that it tends to push high values toward 1 and lowvalues toward 0. Thus when larger numbers are passed to a softmax the result isa distribution with increased probabilities of the most high-probability words anddecreased probabilities of the low probability words, making the distribution moregreedy. Astapproaches 0 the probability of the most likely word approaches 1. Temperature samplingReshape the distribution instead of truncating itIntuition from thermodynamics, \u2022a system at high temperature is flexible and can explore many possible states,\u2022a system at lower temperature is likely to explore a subset of lower energy (better) states. In low-temperature sampling,  (\u03c4 \u2264 1) we smoothly\u2022increase the probability of the most probable words\u2022decrease the probability of the rare words.  Temperature samplingDivide the logit by a temperature parameter \u03c4 before passing it through the softmax.Instead ofWe do  10.2\u2022SAMPLING FORLLM GENERATION75.Randomly sample a word from within these remainingkmost-probable wordsaccording to its probability.Whenk=1, top-ksampling is identical to greedy decoding. Settingkto a largernumber than 1 leads us to sometimes select a word which is not necessarily the mostprobable, but is still probable enough, and whose choice results in generating morediverse but still high-enough-quality text.10.2.2 Nucleus or top-psamplingOne problem with top-ksampling is thatkis \ufb01xed, but the shape of the probabilitydistribution over words differs in different contexts. If we setk=10, sometimesthe top 10 words will be very likely and include most of the probability mass, butother times the probability distribution will be \ufb02atter and the top 10 words will onlyinclude a small part of the probability mass.An alternative, calledtop-p samplingornucleus sampling(Holtzman et al.,top-p sampling2020), is to keep not the topkwords, but the topppercent of the probability mass.The goal is the same; to truncate the distribution to remove the very unlikely words.But by measuring probability rather than the number of words, the hope is that themeasure will be more robust in very different contexts, dynamically increasing anddecreasing the pool of word candidates.Given a distributionP(wt|w<t), the top-pvocabularyV(p)is the smallest set ofwords such thatXw2V(p)P(w|w<t)\u0000p.(10.2)10.2.3 Temperature samplingIntemperature sampling, we don\u2019t truncate the distribution, but instead reshapetemperaturesamplingit. The intuition for temperature sampling comes from thermodynamics, where asystem at a high temperature is very \ufb02exible and can explore many possible states,while a system at a lower temperature is likely to explore a subset of lower energy(better) states. In low-temperature sampling, we smoothly increase the probabilityof the most probable words and decrease the probability of the rare words.We implement this intuition by simply dividing the logit by a temperature param-etertbefore we normalize it by passing it through the softmax. In low-temperaturesampling,t2(0,1]. Thus instead of computing the probability distribution over thevocabulary directly from the logit as in the following (repeated from (??)):y=softmax(u)(10.3)we instead \ufb01rst divide the logits byt, computing the probability vectoryasy=softmax(u/t)(10.4)Why does this work? Whentis close to 1 the distribution doesn\u2019t change much.But the lowertis, the larger the scores being passed to the softmax (dividing by asmaller fractiont\uf8ff1 results in making each score larger). Recall that one of theuseful properties of a softmax is that it tends to push high values toward 1 and lowvalues toward 0. Thus when larger numbers are passed to a softmax the result isa distribution with increased probabilities of the most high-probability words anddecreased probabilities of the low probability words, making the distribution moregreedy. Astapproaches 0 the probability of the most likely word approaches 1.10.2\u2022SAMPLING FORLLM GENERATION75.Randomly sample a word from within these remainingkmost-probable wordsaccording to its probability.Whenk=1, top-ksampling is identical to greedy decoding. Settingkto a largernumber than 1 leads us to sometimes select a word which is not necessarily the mostprobable, but is still probable enough, and whose choice results in generating morediverse but still high-enough-quality text.10.2.2 Nucleus or top-psamplingOne problem with top-ksampling is thatkis \ufb01xed, but the shape of the probabilitydistribution over words differs in different contexts. If we setk=10, sometimesthe top 10 words will be very likely and include most of the probability mass, butother times the probability distribution will be \ufb02atter and the top 10 words will onlyinclude a small part of the probability mass.An alternative, calledtop-p samplingornucleus sampling(Holtzman et al.,top-p sampling2020), is to keep not the topkwords, but the topppercent of the probability mass.The goal is the same; to truncate the distribution to remove the very unlikely words.But by measuring probability rather than the number of words, the hope is that themeasure will be more robust in very different contexts, dynamically increasing anddecreasing the pool of word candidates.Given a distributionP(wt|w<t), the top-pvocabularyV(p)is the smallest set ofwords such thatXw2V(p)P(w|w<t)\u0000p.(10.2)10.2.3 Temperature samplingIntemperature sampling, we don\u2019t truncate the distribution, but instead reshapetemperaturesamplingit. The intuition for temperature sampling comes from thermodynamics, where asystem at a high temperature is very \ufb02exible and can explore many possible states,while a system at a lower temperature is likely to explore a subset of lower energy(better) states. In low-temperature sampling, we smoothly increase the probabilityof the most probable words and decrease the probability of the rare words.We implement this intuition by simply dividing the logit by a temperature param-etertbefore we normalize it by passing it through the softmax. In low-temperaturesampling,t2(0,1]. Thus instead of computing the probability distribution over thevocabulary directly from the logit as in the following (repeated from (??)):y=softmax(u)(10.3)we instead \ufb01rst divide the logits byt, computing the probability vectoryasy=softmax(u/t)(10.4)Why does this work? Whentis close to 1 the distribution doesn\u2019t change much.But the lowertis, the larger the scores being passed to the softmax (dividing by asmaller fractiont\uf8ff1 results in making each score larger). Recall that one of theuseful properties of a softmax is that it tends to push high values toward 1 and lowvalues toward 0. Thus when larger numbers are passed to a softmax the result isa distribution with increased probabilities of the most high-probability words anddecreased probabilities of the low probability words, making the distribution moregreedy. Astapproaches 0 the probability of the most likely word approaches 1. Temperature samplingWhy does this work?\u2022When \u03c4 is close to 1 the distribution doesn\u2019t change much. \u2022The lower \u03c4 is, the larger the scores being passed to the softmax\u2022Softmax pushes high values toward 1 and low values toward 0. \u2022Large inputs pushes high-probability words higher and low probability word lower,  making the distribution more greedy. \u2022As \u03c4 approaches 0, the probability of most likely word approaches 1 10.2\u2022SAMPLING FORLLM GENERATION75.Randomly sample a word from within these remainingkmost-probable wordsaccording to its probability.Whenk=1, top-ksampling is identical to greedy decoding. Settingkto a largernumber than 1 leads us to sometimes select a word which is not necessarily the mostprobable, but is still probable enough, and whose choice results in generating morediverse but still high-enough-quality text.10.2.2 Nucleus or top-psamplingOne problem with top-ksampling is thatkis \ufb01xed, but the shape of the probabilitydistribution over words differs in different contexts. If we setk=10, sometimesthe top 10 words will be very likely and include most of the probability mass, butother times the probability distribution will be \ufb02atter and the top 10 words will onlyinclude a small part of the probability mass.An alternative, calledtop-p samplingornucleus sampling(Holtzman et al.,top-p sampling2020), is to keep not the topkwords, but the topppercent of the probability mass.The goal is the same; to truncate the distribution to remove the very unlikely words.But by measuring probability rather than the number of words, the hope is that themeasure will be more robust in very different contexts, dynamically increasing anddecreasing the pool of word candidates.Given a distributionP(wt|w<t), the top-pvocabularyV(p)is the smallest set ofwords such thatXw2V(p)P(w|w<t)\u0000p.(10.2)10.2.3 Temperature samplingIntemperature sampling, we don\u2019t truncate the distribution, but instead reshapetemperaturesamplingit. The intuition for temperature sampling comes from thermodynamics, where asystem at a high temperature is very \ufb02exible and can explore many possible states,while a system at a lower temperature is likely to explore a subset of lower energy(better) states. In low-temperature sampling, we smoothly increase the probabilityof the most probable words and decrease the probability of the rare words.We implement this intuition by simply dividing the logit by a temperature param-etertbefore we normalize it by passing it through the softmax. In low-temperaturesampling,t2(0,1]. Thus instead of computing the probability distribution over thevocabulary directly from the logit as in the following (repeated from (??)):y=softmax(u)(10.3)we instead \ufb01rst divide the logits byt, computing the probability vectoryasy=softmax(u/t)(10.4)Why does this work? Whentis close to 1 the distribution doesn\u2019t change much.But the lowertis, the larger the scores being passed to the softmax (dividing by asmaller fractiont\uf8ff1 results in making each score larger). Recall that one of theuseful properties of a softmax is that it tends to push high values toward 1 and lowvalues toward 0. Thus when larger numbers are passed to a softmax the result isa distribution with increased probabilities of the most high-probability words anddecreased probabilities of the low probability words, making the distribution moregreedy. Astapproaches 0 the probability of the most likely word approaches 1.0 \u2264 \u03c4 \u2264 1  Large Language ModelsSampling for LLM Generation Large Language ModelsPretraining Large Language Models: Algorithm PretrainingThe big idea that underlies all the amazing performance of language modelsFirst pretrain a transformer model on enormous amounts of textThen apply it to new tasks. Self-supervised training algorithmWe just train them to predict the next word!1.Take a corpus of text 2.At each time step t i.ask the model to predict the next word ii.train the model using gradient descent to minimize the error in this prediction\"Self-supervised\" because it just uses the next word as the label! Intuition of language model training: loss\u2022Same loss function: cross-entropy loss\u2022We want the model to assign a high probability to true word w\u2022= want loss to be high if the model assigns too low a probability to w\u2022CE Loss: The negative log probability that the model assigns to the true next word w\u2022If the model assigns too low a probability to w\u2022We move the model weights in the direction that assigns a higher probability to w Cross-entropy loss for language modelingCE loss: difference between the correct probability distribution and the predicted distribution The correct distribution yt knows the next word, so is 1 for the actual next word and 0 for the others.So in this sum, all terms get multiplied by zero except one: the logp the model assigns to the correct next word, so: 8CHAPTER10\u2022LARGELANGUAGEMODELSNote, by the way, that there can be other situations where we may want to dosomething quite different and \ufb02atten the word probability distribution instead ofmaking it greedy. Temperature sampling can help with this situation too, in this casehigh-temperaturesampling, in which case we uset>1.10.3 Pretraining Large Language ModelsHow do we teach a transformer to be a language model? What is the algorithm andwhat data do we train on?10.3.1 Self-supervised training algorithmTo train a transformer as a language model, we use the sameself-supervision(orself-supervisionself-training) algorithm we saw in Section??: we take a corpus of text as trainingmaterial and at each time steptask the model to predict the next word. We callsuch a model self-supervised because we don\u2019t have to add any special gold labelsto the data; the natural sequence of words is its own supervision! We simply train themodel to minimize the error in predicting the true next word in the training sequence,using cross-entropy as the loss function.Recall that the cross-entropy loss measures the difference between a predictedprobability distribution and the correct distribution.LCE=\u0000Xw2Vyt[w]log\u02c6yt[w](10.5)In the case of language modeling, the correct distributionytcomes from knowing thenext word. This is represented as a one-hot vector corresponding to the vocabularywhere the entry for the actual next word is 1, and all the other entries are 0. Thus,the cross-entropy loss for language modeling is determined by the probability themodel assigns to the correct next word (all other words get multiplied by zero). Soat timetthe CE loss in (10.5) can be simpli\ufb01ed as the negative log probability themodel assigns to the next word in the training sequence.LCE(\u02c6yt,yt)=\u0000log\u02c6yt[wt+1](10.6)Thus at each word positiontof the input, the model takes as input the correct se-quence of tokensw1:t, and uses them to compute a probability distribution overpossible next words so as to compute the model\u2019s loss for the next tokenwt+1. Thenwe move to the next word, we ignore what the model predicted for the next wordand instead use the correct sequence of tokensw1:t+1to estimate the probability oftokenwt+2. This idea that we always give the model the correct history sequence topredict the next word (rather than feeding the model its best case from the previoustime step) is calledteacher forcing.teacher forcingFig.10.4illustrates the general training approach. At each step, given all thepreceding words, the \ufb01nal transformer layer produces an output distribution overthe entire vocabulary. During training, the probability assigned to the correct wordis used to calculate the cross-entropy loss for each item in the sequence. The lossfor a training sequence is the average cross-entropy loss over the entire sequence.The weights in the network are adjusted to minimize the average CE loss over thetraining sequence via gradient descent.8CHAPTER10\u2022LARGELANGUAGEMODELSNote, by the way, that there can be other situations where we may want to dosomething quite different and \ufb02atten the word probability distribution instead ofmaking it greedy. Temperature sampling can help with this situation too, in this casehigh-temperaturesampling, in which case we uset>1.10.3 Pretraining Large Language ModelsHow do we teach a transformer to be a language model? What is the algorithm andwhat data do we train on?10.3.1 Self-supervised training algorithmTo train a transformer as a language model, we use the sameself-supervision(orself-supervisionself-training) algorithm we saw in Section??: we take a corpus of text as trainingmaterial and at each time steptask the model to predict the next word. We callsuch a model self-supervised because we don\u2019t have to add any special gold labelsto the data; the natural sequence of words is its own supervision! We simply train themodel to minimize the error in predicting the true next word in the training sequence,using cross-entropy as the loss function.Recall that the cross-entropy loss measures the difference between a predictedprobability distribution and the correct distribution.LCE=\u0000Xw2Vyt[w]log\u02c6yt[w](10.5)In the case of language modeling, the correct distributionytcomes from knowing thenext word. This is represented as a one-hot vector corresponding to the vocabularywhere the entry for the actual next word is 1, and all the other entries are 0. Thus,the cross-entropy loss for language modeling is determined by the probability themodel assigns to the correct next word (all other words get multiplied by zero). Soat timetthe CE loss in (10.5) can be simpli\ufb01ed as the negative log probability themodel assigns to the next word in the training sequence.LCE(\u02c6yt,yt)=\u0000log\u02c6yt[wt+1](10.6)Thus at each word positiontof the input, the model takes as input the correct se-quence of tokensw1:t, and uses them to compute a probability distribution overpossible next words so as to compute the model\u2019s loss for the next tokenwt+1. Thenwe move to the next word, we ignore what the model predicted for the next wordand instead use the correct sequence of tokensw1:t+1to estimate the probability oftokenwt+2. This idea that we always give the model the correct history sequence topredict the next word (rather than feeding the model its best case from the previoustime step) is calledteacher forcing.teacher forcingFig.10.4illustrates the general training approach. At each step, given all thepreceding words, the \ufb01nal transformer layer produces an output distribution overthe entire vocabulary. During training, the probability assigned to the correct wordis used to calculate the cross-entropy loss for each item in the sequence. The lossfor a training sequence is the average cross-entropy loss over the entire sequence.The weights in the network are adjusted to minimize the average CE loss over thetraining sequence via gradient descent. Teacher forcing\u2022At each token position t, model sees correct tokens w1:t, \u2022Computes  loss (\u2013log probability) for the next token wt+1 \u2022At next token position t+1 we ignore what model predicted for wt+1 \u2022Instead we take the correct word wt+1, add it to context, move on Training a transformer language model\nlongandthanksforNext tokenallLoss\u2026=\n<latexit sha1_base64=\"AovqpaL476UmJ1EU1xZPgDZ70tQ=\">AAAB9nicbVDLSsNAFL2pr1pfURcu3AwWwY0lEakui25cVrAPaEqYTCbt0EkmzEzEEvIrbkTcKPgZ/oJ/Y9Jm09YDA4dzznDvPV7MmdKW9WtU1tY3Nreq27Wd3b39A/PwqKtEIgntEMGF7HtYUc4i2tFMc9qPJcWhx2nPm9wXfu+ZSsVE9KSnMR2GeBSxgBGsc8k1Ty4dLkZo6qZOiPVYhimO/CyruWbdalgzoFVil6QOJdqu+eP4giQhjTThWKmBbcV6mGKpGeE0qzmJojEmEzyi6WztDJ3nko8CIfMXaTRTF3I4VGoaenmy2E0te4X4nzdIdHA7TFkUJ5pGZD4oSDjSAhUdIJ9JSjSf5gQTyfINERljiYnOmypOt5cPXSXdq4bdbDQfr+utu7KEKpzCGVyADTfQggdoQwcIZPAGn/BlvBivxrvxMY9WjPLPMSzA+P4DPEiSHA==</latexit>\u0000logyand\nStackedTransformerBlocksSolongandthanksfor\u2026\u2026\n\u2026\nU\nInput tokensx1x2LanguageModelingHeadx3x4x5InputEncoding\nE1+\nE2+\nE3+\nE4+\nE5+\u2026\u2026\u2026\u2026\u2026\nU\nU\nU\nU\u2026logitslogitslogitslogitslogits\u2026\n<latexit sha1_base64=\"q3ZgXDyG7qtkT7t8hT47RdlwYG4=\">AAAB+XicbVDLSsNAFJ3UV62vWHe6GVsEN5bERXUlBUVcVrAPaEqYTCft0MlMmJkIIQT8AT/CTRE3Cv6Ev+DfmLTdtPXAwOGcM9x7jxcyqrRl/RqFtfWNza3idmlnd2//wDwst5WIJCYtLJiQXQ8pwignLU01I91QEhR4jHS88W3ud56JVFTwJx2HpB+gIac+xUhnkmseXzhMDGHsJk6A9EgGiR4hPlZpWnLNqlWzpoCrxJ6TauP0tXw3qdw0XfPHGQgcBYRrzJBSPdsKdT9BUlPMSFpyIkVChMdoSJLp5ik8y6QB9IXMHtdwqi7kUKBUHHhZMl9PLXu5+J/Xi7R/3U8oDyNNOJ4N8iMGtYB5DXBAJcGaxRlBWNJsQ4hHSCKss7Ly0+3lQ1dJ+7Jm12v1x6yDezBDEZyACjgHNrgCDfAAmqAFMHgBE/AJvozEeDPejY9ZtGDM/xyBBRjff79pldo=</latexit>\u0000logythanks Large Language ModelsPretraining Large Language Models: Algorithm Large Language ModelsPretraining data for LLMs LLMs are mainly trained on the webCommon crawl, snapshots of the entire web produced by the non- profit Common Crawl with billions of pagesColossal Clean Crawled Corpus (C4; Raffel et al. 2020), 156 billion tokens of English,  filtered What's in it? Mostly patent text documents, Wikipedia, and news sites  The Pile: a pretraining corpus\nFigure 1: Treemap of Pile components by effective size.troduce a new \ufb01ltered subset of Common Crawl,Pile-CC, with improved extraction quality.Through our analyses, we con\ufb01rm that the Pile issigni\ufb01cantly distinct from pure Common Crawldata. Additionally, our evaluations show that theexisting GPT-2 and GPT-3 models perform poorlyon many components of the Pile, and that modelstrained on the Pile signi\ufb01cantly outperform bothraw and \ufb01ltered Common Crawl models. To com-plement the performance evaluations, we also per-form an exploratory analysis of the text within thePile to provide a detailed picture of the data. Wehope that our extensive documentation of the con-struction and characteristics of the Pile will helpresearchers make informed decisions about poten-tial downstream applications.Finally, we make publicly available the preprocess-ing code for the constituent datasets of the Pile andthe code for constructing alternative versions2. Inthe interest of reproducibility, we also documentall processing performed on each dataset (and thePile as a whole) in as much detail as possible. Forfurther details about the processing of each dataset,see Section2and AppendixC.2https://github.com/EleutherAI/the-pile1.1 ContributionsThe core contributions of this paper are:1.The introduction of a825.18GiB english-language dataset for language modeling com-bining 22 diverse sources.2.The introduction of14new language model-ing datasets, which we expect to be of inde-pendent interest to researchers.3.Evaluations demonstrating signi\ufb01cant im-provements across many domains by GPT-2-sized models trained on this new dataset, com-pared to training on CC-100 and raw CommonCrawl.4.The investigation and documentation of thisdataset, which we hope will better inform re-searchers about how to use it as well as moti-vate them to undertake similar investigationsof their own data.2 The Pile DatasetsThe Pile is composed of 22 constituent sub-datasets,as shown in Table1. FollowingBrown et al.(2020),we increase the weights of higher quality compo-nents, with certain high-quality datasets such asWikipedia being seen up to 3 times (\u201cepochs\u201d) for2webacademicsbooks\ndialog Filtering for quality and safetyQuality is subjective\u2022Many LLMs attempt to match Wikipedia, books, particular websites\u2022Need to remove boilerplate, adult content\u2022Deduplication at many levels (URLs, documents, even lines)Safety also subjective\u2022Toxicity detection is important, although that has mixed results\u2022Can mistakenly flag data written in dialects like African American English What does a model learn from pretraining?\u2022There are canines everywhere! One dog in the front room, and two dogs\u2022It wasn't just big it was enormous\u2022The author of \"A Room of One's Own\" is Virginia Woolf\u2022The doctor told me that he\u2022The square root of 4 is 2 Big ideaText contains enormous amounts of knowledgePretraining on lots of text with all that knowledge is what gives language models their ability to do so much But there are problems with scraping from the webCopyright: much of the text in these datasets is copyrighted\u2022Not clear if fair use doctrine in US allows for this use\u2022This remains an open legal questionData consent\u2022Website owners can indicate they don't want their site crawledPrivacy: \u2022Websites can contain private IP addresses and phone numbers Large Language ModelsPretraining data for LLMs Large Language ModelsFinetuning Finetuning for daptation to new domainsWhat happens if we need our LLM to work well on a domain it didn't see in pretraining?Perhaps some specific medical or legal domain?Or maybe a multilingual LM needs to see more data on some language that was rare in pretraining? FinetuningFine-tuning Data\nPretraining DataPretraining\n\u2026\n\u2026\n\u2026Fine-tuning\n\u2026\n\u2026\n\u2026Pretrained LMFine-tuned LM \"Finetuning\" means 4 different thingsWe'll discuss 1 here, and 3 in later lecturesIn all four cases, finetuning means:taking a pretrained model and further adapting some or all of its parameters to some new data 1. Finetuning as \"continued pretraining\" on new data\u2022Further train all the parameters of model on new data\u2022using the same method (word prediction) and loss function (cross-entropy loss) as for pretraining.\u2022as if the new data were at the tail end of the pretraining data\u2022Hence sometimes called continued pretraining Large Language ModelsFinetuning Large Language ModelsEvaluating Large Language Models PerplexityJust as for n-gram grammars, we use perplexity to measure how well the LM predicts unseen textThe perplexity of a model \u03b8 on an unseen test set is the inverse probability that \u03b8 assigns to the test set, normalized by the test set length. For a test set of n tokens w1:n the perplexity is :12CHAPTER10\u2022LARGELANGUAGEMODELSthe pretraining data, and so you\u2019ll sometimes see this method calledcontinued pre-training.continuedpretrainingRetraining all the parameters of the model is very slow and expensive when thelanguage model is huge. So instead we canfreezesome of the parameters (i.e., leavefreezethem unchanged from their pretrained value) and train only a subset of parameterson the new data. In Section10.5.3we\u2019ll describe this second variety of \ufb01netun-ing, calledparameter-ef\ufb01cient \ufb01netuning, orPEFT. because we ef\ufb01ciently selectspeci\ufb01c parameters to update when \ufb01netuning, and leave the rest in their pretrainedvalues.In Chapter 11 we\u2019ll introduce a third kind of \ufb01netuning, also parameter-ef\ufb01cient.In this version, the goal is to use a language model as a kind of classi\ufb01er or labelerfor a speci\ufb01c task. For example we might train the model to be a sentiment classi\ufb01er.We do this by adding extra neural circuitry (an extrahead) after the top layer of themodel. This classi\ufb01cation head takes as input some of the top layer embeddings ofthe transformer and produces as output a classi\ufb01cation. In this method, most com-monly used with masked language models like BERT, we freeze the entire pretrainedmodel and only train the classi\ufb01cation head on some new data, usually labeled withsome class that we want to predict.Finally, in Chapter 12 we\u2019ll introduce a fourth kind of \ufb01netuning, that is a cru-cial component of the largest language models:supervised \ufb01netuningorSFT. SFTis often used forinstruction \ufb01netuning, in which we want a pretrained languagemodel to learn to follow text instructions, for example to answer questions or followa command to write something. Here we create a dataset of prompts and desiredresponses (for example questions and their answers, or commands and their ful-\ufb01llments), and we train the language model using the normal cross-entropy loss topredict each token in the instruction prompt iteratively, essentially training it to pro-duce the desired response from the command in the prompt. It\u2019s called supervisedbecause unlike in pretraining, where we just take any data and predict the words init, we build the special \ufb01netuning dataset by hand, creating supervised responses toeach command.Often everything that happens after pretraining is lumped together aspost-training;we\u2019ll discuss the various parts of post-training in Chapter 12 and Chapter 13.10.4 Evaluating Large Language ModelsPerplexityAs we \ufb01rst saw in Chapter 3, one way to evaluate language models isto measure how well they predict unseen text. Intuitively, good models are those thatassign higher probabilities to unseen data (are less surprised when encountering thenew words).We instantiate this intuition by usingperplexityto measure the quality of aperplexitylanguage model. Recall from page??that the perplexity of a modelqon an unseentest set is the inverse probability thatqassigns to the test set, normalized by the testset length. For a test set ofntokensw1:n, the perplexity isPerplexityq(w1:n)=Pq(w1:n)\u00001n=ns1Pq(w1:n)(10.7)To visualize how perplexity can be computed as a function of the probabilities the \u2022Probability depends on size of test set\u2022Probability gets smaller the longer the text\u2022Better: a metric that is per-word, normalized by length\u2022Perplexity is the inverse probability of the test set, normalized by the number of words(The inverse comes from the original definition of perplexity from cross-entropy rate in information theory)Probability range is  [0,1], perplexity range is [1,\u221e]Why perplexity instead of raw probability of the test set? Perplexity\u2022The higher the probability of the word sequence, the lower the perplexity.\u2022Thus the lower the perplexity of a model on the data, the better the model. \u2022Minimizing perplexity is the same as maximizing probabilityAlso: perplexity is sensitive to length/tokenization so best used when comparing LMs that use the same tokenizer.   Many other factors that we evaluate, like:Size Big models take lots of GPUs and time to train, memory to storeEnergy usageCan measure kWh or kilograms of CO2 emitted FairnessBenchmarks measure gendered and racial stereotypes, or decreased performance for language from or about some groups.  Large Language ModelsDealing with Scale Scaling LawsLLM performance depends on\u2022Model size: the number of parameters not counting embeddings\u2022Dataset size: the amount of training data\u2022Compute: Amount of compute (in FLOPS or etcCan improve a model by adding  parameters (more layers, wider contexts), more data, or training for more iterationsThe performance of a large language model (the loss) scales as a power-law with each of these three Scaling LawsLoss L as a function of # parameters N, dataset size D, compute budget C (if other two are held constant)14CHAPTER10\u2022LARGELANGUAGEMODELS10.5 Dealing with ScaleLarge language models are large. For example theLlama 3.1 405B Instructmodelfrom Meta has 405 billion parameters (126 layers, a model dimensionality of 16,384,128 attention heads) and was trained on 15.6 terabytes of text tokens (Llama Team,2024), using a vocabulary of 128K tokens. So there is a lot of research on un-derstanding how LLMs scale, and especially how to implement them given limitedresources. In the next few sections we discuss how to think about scale (the conceptofscaling laws), and important techniques for getting language models to workef\ufb01ciently, such as theKV cacheand parameter-ef\ufb01cient \ufb01ne tuning.10.5.1 Scaling lawsThe performance of large language models has shown to be mainly determined by3 factors: model size (the number of parameters not counting embeddings), datasetsize (the amount of training data), and the amount of compute used for training. Thatis, we can improve a model by adding parameters (adding more layers or havingwider contexts or both), by training on more data, or by training for more iterations.The relationships between these factors and performance are known asscalinglaws. Roughly speaking, the performance of a large language model (the loss) scalesscaling lawsas a power-law with each of these three properties of model training.For example,Kaplan et al.(2020) found the following three relationships forlossLas a function of the number of non-embedding parametersN, the dataset sizeD, and the compute budgetC, for models training with limited parameters, dataset,or compute budget, if in each case the other two properties are held constant:L(N)=\u2713NcN\u25c6aN(10.9)L(D)=\u2713DcD\u25c6aD(10.10)L(C)=\u2713CcC\u25c6aC(10.11)The number of (non-embedding) parametersNcan be roughly computed as fol-lows (ignoring biases, and withdas the input and output dimensionality of themodel,dattnas the self-attention layer size, anddffthe size of the feedforward layer):N\u21e12dnlayer(2dattn+dff)\u21e112nlayerd2(10.12)(assumingdattn=dff/4=d)Thus GPT-3, withn=96 layers and dimensionalityd=12288, has 12\u21e596\u21e5122882\u21e1175 billion parameters.The values ofNc,Dc,Cc,aN,aD, andaCdepend on the exact transformerarchitecture, tokenization, and vocabulary size, so rather than all the precise values,scaling laws focus on the relationship with loss.2Scaling laws can be useful in deciding how to train a model to a particular per-formance, for example by looking at early in the training curve, or performance with2For the initial experiment inKaplan et al.(2020) the precise values wereaN= 0.076,Nc= 8.8\u21e51013(parameters),aD= 0.095,Dc= 5.4\u21e51013(tokens),aC= 0.050,Cc= 3.1\u21e5108(peta\ufb02op-days).Scaling laws can be used early in training to predict what the loss would be if we were to add more data or increase model size.  Number of non-embedding parameters N14CHAPTER10\u2022LARGELANGUAGEMODELS10.5 Dealing with ScaleLarge language models are large. For example theLlama 3.1 405B Instructmodelfrom Meta has 405 billion parameters (126 layers, a model dimensionality of 16,384,128 attention heads) and was trained on 15.6 terabytes of text tokens (Llama Team,2024), using a vocabulary of 128K tokens. So there is a lot of research on un-derstanding how LLMs scale, and especially how to implement them given limitedresources. In the next few sections we discuss how to think about scale (the conceptofscaling laws), and important techniques for getting language models to workef\ufb01ciently, such as theKV cacheand parameter-ef\ufb01cient \ufb01ne tuning.10.5.1 Scaling lawsThe performance of large language models has shown to be mainly determined by3 factors: model size (the number of parameters not counting embeddings), datasetsize (the amount of training data), and the amount of compute used for training. Thatis, we can improve a model by adding parameters (adding more layers or havingwider contexts or both), by training on more data, or by training for more iterations.The relationships between these factors and performance are known asscalinglaws. Roughly speaking, the performance of a large language model (the loss) scalesscaling lawsas a power-law with each of these three properties of model training.For example,Kaplan et al.(2020) found the following three relationships forlossLas a function of the number of non-embedding parametersN, the dataset sizeD, and the compute budgetC, for models training with limited parameters, dataset,or compute budget, if in each case the other two properties are held constant:L(N)=\u2713NcN\u25c6aN(10.9)L(D)=\u2713DcD\u25c6aD(10.10)L(C)=\u2713CcC\u25c6aC(10.11)The number of (non-embedding) parametersNcan be roughly computed as fol-lows (ignoring biases, and withdas the input and output dimensionality of themodel,dattnas the self-attention layer size, anddffthe size of the feedforward layer):N\u21e12dnlayer(2dattn+dff)\u21e112nlayerd2(10.12)(assumingdattn=dff/4=d)Thus GPT-3, withn=96 layers and dimensionalityd=12288, has 12\u21e596\u21e5122882\u21e1175 billion parameters.The values ofNc,Dc,Cc,aN,aD, andaCdepend on the exact transformerarchitecture, tokenization, and vocabulary size, so rather than all the precise values,scaling laws focus on the relationship with loss.2Scaling laws can be useful in deciding how to train a model to a particular per-formance, for example by looking at early in the training curve, or performance with2For the initial experiment inKaplan et al.(2020) the precise values wereaN= 0.076,Nc= 8.8\u21e51013(parameters),aD= 0.095,Dc= 5.4\u21e51013(tokens),aC= 0.050,Cc= 3.1\u21e5108(peta\ufb02op-days).Thus GPT-3, with n = 96 layers and dimensionality d = 12288, has 12 \u00d7 96 \u00d7 122882 \u2248 175 billion parameters.  KV CacheIn training, we can compute attention very efficiently in parallel:But not at inference! We generate the next tokens one at a time!For a new token x, need to multiply by WQ, WK, and WV to get query, key, valuesBut don't want to recompute the key and value vectors for all the prior tokens x<iInstead, store key and value vectors in memory in the KV cache, and then we can just grab them from the cache 10.5\u2022DEALING WITHSCALE15smaller amounts of data, to predict what the loss would be if we were to add moredata or increase model size. Other aspects of scaling laws can also tell us how muchdata we need to add when scaling up a model.10.5.2 KV CacheWe saw in Fig.??and in Eq.??(repeated below) how the attention vector can bevery ef\ufb01ciently computed in parallel for training, via two matrix multiplications:A=softmax\u2713QK|pdk\u25c6V(10.13)Unfortunately we can\u2019t do quite the same ef\ufb01cient computation in inference asin training. That\u2019s because at inference time, we iteratively generate the next tokensone at a time. For a new token that we have just generated, call itxi, we need tocompute its query, key, and values by multiplying byWQ,WK, andWVrespec-tively. But it would be a waste of computation time to recompute the key and valuevectors for all thepriortokensx<i; at prior steps we already computed these keyand value vectors! So instead of recomputing these, whenever we compute the keyand value vectors we store them in memory in theKV cache, and then we can justKV cachegrab them from the cache when we need them. Fig.10.7modi\ufb01es Fig.??to showthe computation that takes place for a single new token, showing which values wecan take from the cache rather than recompute.\nq4k1k2k4QKTQKTv1v2v3v4V\nq4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4x==xa4A\n1 x dkdk x N1 x NN x dv1 x dv\nk3Figure 10.7Parts of the attention computation (extracted from Fig.??) showing, in black,the vectors that can be stored in the cache rather than recomputed when computing the atten-tion score for the 4th token.10.5.3 Parameter Ef\ufb01cient Fine TuningAs we mentioned above, it\u2019s very common to take a language model and give it moreinformation about a new domain by\ufb01netuningit (continuing to train it to predictupcoming words) on some additional data.Fine-tuning can be very dif\ufb01cult with very large language models, because thereare enormous numbers of parameters to train; each pass of batch gradient descenthas to backpropagate through many many huge layers. This makes \ufb01netuning hugelanguage models extremely expensive in processing power, in memory, and in time.For this reason, there are alternative methods that allow a model to be \ufb01netunedwithout changing all the parameters. Such methods are calledparameter-ef\ufb01cient\ufb01ne tuningor sometimesPEFT, because we ef\ufb01ciently select a subset of parametersparameter-ef\ufb01cient \ufb01netuningPEFTto update when \ufb01netuning. For example we freeze some of the parameters (don\u2019tchange them), and only update some particular subset of parameters. KV Cache\nq4k1k2k4QKTQKTv1v2v3v4V\nq4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4x==xa4A\n1 x dkdk x N1 x NN x dv1 x dv\nk3q1q2q3q4k1k2k3k4QKTQKTv1v2v3v4Vq2\u2022k2q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k2q3\u2022k3\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221eq1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3q1\u2022k2q2\u2022k3q1\u2022k3q3\u2022k4q2\u2022k4q1\u2022k4x=QKT maskedmask=q1\u2022k1q2\u2022k1q4\u2022k1q3\u2022k1q1\u2022k1q1\u2022k1=xa1a2a3a4AQuery Token 1Query Token 2Query Token 3Query Token 4QInput Token 1Input Token 2Input Token 3Input Token 4XxWQ=Value Token 1Value Token 2Value Token 3Value Token 4VxWV=Input Token 1Input Token 2Input Token 3Input Token 4XKey Token 1Key Token 2Key Token 3Key Token 4KxWK=Input Token 1Input Token 2Input Token 3Input Token 4X\nN x dkdk x NN x NN x NN x dvN x dvd x dkd x dkd x dvN x dN x dkN x dN x dkN x dN x dvq1q2q3q4k1k2k3k4QKTQKTv1v2v3v4Vq2\u2022k2q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k2q3\u2022k3\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221eq1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3q1\u2022k2q2\u2022k3q1\u2022k3q3\u2022k4q2\u2022k4q1\u2022k4x=QKT maskedmask=q1\u2022k1q2\u2022k1q4\u2022k1q3\u2022k1q1\u2022k1q1\u2022k1=xa1a2a3a4AQuery Token 1Query Token 2Query Token 3Query Token 4QInput Token 1Input Token 2Input Token 3Input Token 4XxWQ=Value Token 1Value Token 2Value Token 3Value Token 4VxWV=Input Token 1Input Token 2Input Token 3Input Token 4XKey Token 1Key Token 2Key Token 3Key Token 4KxWK=Input Token 1Input Token 2Input Token 3Input Token 4X\nN x dkdk x NN x NN x NN x dvN x dvd x dkd x dkd x dvN x dN x dkN x dN x dkN x dN x dvq1q2q3q4k1k2k3k4QKTQKTv1v2v3v4Vq2\u2022k2q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k2q3\u2022k3\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221eq1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3q1\u2022k2q2\u2022k3q1\u2022k3q3\u2022k4q2\u2022k4q1\u2022k4x=QKT maskedmask=q1\u2022k1q2\u2022k1q4\u2022k1q3\u2022k1q1\u2022k1q1\u2022k1=xa1a2a3a4AQuery Token 1Query Token 2Query Token 3Query Token 4QInput Token 1Input Token 2Input Token 3Input Token 4XxWQ=Value Token 1Value Token 2Value Token 3Value Token 4VxWV=Input Token 1Input Token 2Input Token 3Input Token 4XKey Token 1Key Token 2Key Token 3Key Token 4KxWK=Input Token 1Input Token 2Input Token 3Input Token 4X\nN x dkdk x NN x NN x NN x dvN x dvd x dkd x dkd x dvN x dN x dkN x dN x dkN x dN x dv Parameter-Efficient FinetuningAdapting to a new domain by continued pretraining (finetuning) is a problem with huge LLMs.\u2022Enormous numbers of parameters to train \u2022Each pass of batch gradient descent has to backpropagate through many many huge layers. \u2022Expensive in processing power, in memory, and in time. Instead, parameter-efficient fine tuning (PEFT)\u2022Efficiently select a subset of parameters to update when finetuning.\u2022E.g., freeze some of the parameters (don\u2019t change them), \u2022And only update some a few parameters.  LoRA (Low-Rank Adaptation)\u2022Trransformers have many dense matrix multiply layers\u2022Like WQ, WK, WV, WO layers in attention\u2022Instead of updating these layers during finetuning, \u2022Freeze these layers \u2022Update a low-rank approximation with fewer parameters.  LoRA\u2022Consider a matrix W (shape [N \u00d7 d])  that needs to be updated during finetuning via gradient descent. \u2022Normally updates are \u2206W  (shape [N \u00d7 d])\u2022In LoRA, we freeze W and update instead a low-rank decomposition of W:\u2022A of shape [N\u00d7r], \u2022B of shape [r\u00d7d], r is very small  (like 1 or 2)\u2022That is, during  finetuning we update A and B instead of W. \u2022Replace W + \u2206W with W + BA. Forward pass: instead of     h = xW We do     h = xW + xAB  LoRAhPretrained WeightsWdkrkABrxd11k\nd\u00d7 Large Language ModelsDealing with Scale Large Language ModelsHarms of Large Language Models Hallucination\n Copyright\n Privacy\n Toxicity and Abuse\n Misinformation\n Large Language ModelsHarms of Large Language Models", "11.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\n11Masked Language Models\nLarvatus prodeo [Masked, I go forward]\nDescartes\nIn the previous two chapters we introduced the transformer and saw how to pre-\ntrain a transformer language model as a causal or left-to-right language model. In\nthis chapter we\u2019ll introduce a second paradigm for pretrained language models, the\nbidirectional transformer encoder, and the most widely-used version, the BERT BERT\nmodel (Devlin et al., 2019). This model is trained via masked language modeling ,masked\nlanguage\nmodelingwhere instead of predicting the following word, we mask a word in the middle and\nask the model to guess the word given the words on both sides. This method thus\nallows the model to see both the right and left context.\nWe also introduced \ufb01netuning in the prior chapter. Here we describe a new \ufb01netuning\nkind of \ufb01netuning, in which we take the transformer network learned by these pre-\ntrained models, add a neural net classi\ufb01er after the top layer of the network, and train\nit on some additional labeled data to perform some downstream task like named\nentity tagging or natural language inference. As before, the intuition is that the\npretraining phase learns a language model that instantiates rich representations of\nword meaning, that thus enables the model to more easily learn (\u2018be \ufb01netuned to\u2019)\nthe requirements of a downstream language understanding task. This aspect of the\npretrain-\ufb01netune paradigm is an instance of what is called transfer learning in ma-transfer\nlearning\nchine learning: the method of acquiring knowledge from one task or domain, and\nthen applying it (transferring it) to solve a new task.\nThe second idea that we introduce in this chapter is the idea of contextual em-\nbeddings : representations for words in context. The methods of Chapter 6 like\nword2vec or GloVe learned a single vector embedding for each unique word win\nthe vocabulary. By contrast, with contextual embeddings, such as those learned by\nmasked language models like BERT, each word wwill be represented by a different\nvector each time it appears in a different context. While the causal language models\nof Chapter 9 also use contextual embeddings, the embeddings created by masked\nlanguage models seem to function particularly well as representations.\n11.1 Bidirectional Transformer Encoders\nLet\u2019s begin by introducing the bidirectional transformer encoder that underlies mod-\nels like BERT and its descendants like RoBERTa (Liu et al., 2019) or SpanBERT\n(Joshi et al., 2020). In Chapter 9 we introduced causal (left-to-right) transformers\nand in Chapter 10 saw how they can serve as the basis for language models that can\nbe applied to autoregressive contextual generation problems like question answering\nor summarization. But this left-to-right nature of these models is also a limitation,\nbecause there are tasks for which it would be useful, when processing a token, to be\nable to peak at future tokens. This is especially true for sequence labeling tasks in 2CHAPTER 11 \u2022 M ASKED LANGUAGE MODELS\nwhich we want to tag each token with a label, such as the part-of-speech tagging or\nparsing tasks we\u2019ll introduce in future chapters, or tasks like named entity tagging\nwe\u2019ll introduce later in this chapter.\nThebidirectional encoders that we introduce here are a different kind of beast\nthan causal models. The causal models of Chapter 9 are generative models, de-\nsigned to easily generate the next token in a sequence. But the focus of bidirec-\ntional encoders is instead on computing contextualized representations of the input\ntokens. Bidirectional encoders use self-attention to map sequences of input embed-\ndings (x1; :::;xn)to sequences of output embeddings of the same length (h1; :::;hn),\nwhere the output vectors have been contextualized using information from the en-\ntire input sequence. These output embeddings are contextualized representations of\neach input token that are useful across a range of applications where we need to do\na classi\ufb01cation or a decision based on the token in context.\nRemember that we said the models of Chapter 9 are sometimes called decoder-\nonly, because they correspond to the decoder part of the encoder-decoder model we\nwill introduce in Chapter 13. By contrast, the masked language models of this chap-\nter are sometimes called encoder-only , because they produce an encoding for each\ninput token but generally aren\u2019t used to produce running text by decoding/sampling.\nThat\u2019s an important point: masked language models are not used for generation.\nThey are generally instead used for interpretative tasks.\n11.1.1 The architecture for bidirectional masked models\nLet\u2019s \ufb01rst discuss the overall architecture. Bidirectional transformer-based language\nmodels differ in two ways from the causal transformers in the previous chapters. The\n\ufb01rst is that the attention function isn\u2019t causal; the attention for a token ican look at\nfollowing tokens i+1 and so on. The second is that the training is slightly different\nsince we are predicting something in the middle of our text rather than at the end.\nWe\u2019ll discuss the \ufb01rst here and the second in the following section.\nFig. 11.1a, reproduced here from Chapter 9, shows the information \ufb02ow in the\nleft-to-right approach of Chapter 9. The attention computation at each token is based\non the preceding (and current) input tokens, ignoring potentially useful information\nlocated to the right of the token under consideration. Bidirectional encoders over-\ncome this limitation by allowing the attention mechanism to range over the entire\ninput, as shown in Fig. 11.1b.\na) A causal self-attention layerb) A bidirectional self-attention layer\nattentionattentionattentionattentionattentiona1a2a3a4a5x3x4x5x1x2\nattentionattentionattentionattentionattentiona1a2a3a4a5x3x4x5x1x2\nFigure 11.1 (a) The causal transformer from Chapter 9, highlighting the attention computation at token 3. The\nattention value at each token is computed using only information seen earlier in the context. (b) Information\n\ufb02ow in a bidirectional attention model. In processing each token, the model attends to all inputs, both before\nand after the current one. So attention for token 3 can draw on information from following tokens.\nThe implementation is very simple! We simply remove the attention masking\nstep that we introduced in Eq. ??. Recall from Chapter 9 that we had to mask the\nQK|matrix for causal transformers so that attention couldn\u2019t look at future tokens 11.1 \u2022 B IDIRECTIONAL TRANSFORMER ENCODERS 3\n(repeated from Eq. ??):\nA=softmax\u0012\nmask\u0012QK|\npdk\u0013\u0013\nV (11.1)\nq1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3NN\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\u2212\u221e\nq1\u2022k1q2\u2022k1q2\u2022k2q4\u2022k1q4\u2022k2q4\u2022k3q4\u2022k4q3\u2022k1q3\u2022k2q3\u2022k3NNq1\u2022k2q1\u2022k3q1\u2022k4q2\u2022k3q2\u2022k4q3\u2022k4\n(a) (b)\nFigure 11.2 The N\u0002NQK|matrix showing the qi\u0001kjvalues, with the upper-triangle\nportion of the comparisons matrix zeroed out (set to \u0000\u00a5, which the softmax will turn to\nzero).\nFig. 11.2 shows the masked version of QK|and the unmasked version. For bidi-\nrectional attention, we used the unmasked version of Fig. 11.2b. Thus the attention\ncomputation for bidirectional attention is exactly the same as Eq. 11.1 but with the\nmask removed:\nA=softmax\u0012QK|\npdk\u0013\nV (11.2)\nOtherwise, the attention computation is identical to what we saw in Chapter 9, as\nis the transformer block architecture (the feedforward layer, layer norm, and so on).\nAs in Chapter 9, the input is also a series of subword tokens, usually computed by\nWordPiece or SentencePiece Unigram LM tokenization (two of the large family of\nsubword tokenization algorithms that includes the BPE algorithm we saw in Chap-\nter 2). That means every input sentence \ufb01rst has to be tokenized, and all further\nprocessing takes place on subword tokens rather than words. This will require, as\nwe\u2019ll see in the third part of the textbook, that for some NLP tasks that require no-\ntions of words (like parsing) we will occasionally need to map subwords back to\nwords.\nTo make this more concrete, the original English-only bidirectional transformer\nencoder model, BERT (Devlin et al., 2019), consisted of the following:\n\u2022 An English-only subword vocabulary consisting of 30,000 tokens generated\nusing the WordPiece algorithm (Schuster and Nakajima, 2012).\n\u2022 Hidden layers of dimensionality d=768,\n\u2022 12 layers of transformer blocks, with 12 (bidirectional) multihead attention\nlayers each.\n\u2022 The resulting model has about 100M parameters.\nThe larger multilingual XLM-RoBERTa model, trained on 100 languages, has\n\u2022 A multilingual subword vocabulary with 250,000 tokens generated using the\nSentencePiece Unigram LM algorithm (Kudo and Richardson, 2018).\n\u2022 24 layers of transformer blocks, with 16 multihead attention layers each 4CHAPTER 11 \u2022 M ASKED LANGUAGE MODELS\n\u2022 Hidden layers of size 1024\n\u2022 Input context window of 512 tokens\n\u2022 The resulting model has about 550M parameters.\nNote that 550M parameters is relatively small as large language models go\n(Llama 3 has 405B parameters, so is 3 orders of magnitude bigger). Indeed, masked\nlanguage models tend to be much smaller than causal language models.\n11.2 Training Bidirectional Encoders\nWe trained causal transformer language models in Chapter 9 by making them it-\neratively predict the next word in a text. But eliminating the causal mask in at-\ntention makes the guess-the-next-word language modeling task trivial\u2014the answer\nis directly available from the context\u2014so we\u2019re in need of a new training scheme.\nInstead of trying to predict the next word, the model learns to perform a \ufb01ll-in-the-\nblank task, technically called the cloze task (Taylor, 1953). To see this, let\u2019s return cloze task\nto the motivating example from Chapter 3. Instead of predicting which words are\nlikely to come next in this example:\nThe water of Walden Pond is so beautifully\nwe\u2019re asked to predict a missing item given the rest of the sentence.\nThe of Walden Pond is so beautifully ...\nThat is, given an input sequence with one or more elements missing, the learning\ntask is to predict the missing elements. More precisely, during training the model is\ndeprived of one or more elements of an input sequence and must generate a proba-\nbility distribution over the vocabulary for each of the missing items. We then use the\ncross-entropy loss from each of the model\u2019s predictions to drive the learning process.\nThis approach can be generalized to any of a variety of methods that corrupt the\ntraining input and then asks the model to recover the original input. Examples of the\nkinds of manipulations that have been used include masks, substitutions, reorder-\nings, deletions, and extraneous insertions into the training text. The general name\nfor this kind of training is called denoising : we corrupt (add noise to) the input in denoising\nsome way (by masking a word, or putting in an incorrect word) and the goal of the\nsystem is to remove the noise.\n11.2.1 Masking Words\nThe original approach to training bidirectional encoders is called Masked Language\nModeling (MLM) (Devlin et al., 2019). As with the language model training meth-Masked\nLanguage\nModelingods we\u2019ve already seen, MLM uses unannotated text from a large corpus. Here, the\nmodel is presented with a series of sentences from the training corpus where a ran-\ndom sample of tokens from each training sequence is selected for use in the learning\ntask. Once chosen, a token is used in one of three ways:\n\u2022 It is replaced with the special vocabulary token named [MASK] .\n\u2022 It is replaced with another token from the vocabulary, randomly sampled\nbased on token unigram probabilities.\n\u2022 It is left unchanged. 11.2 \u2022 T RAINING BIDIRECTIONAL ENCODERS 5\nIn BERT, 15% of the input tokens in a training sequence are sampled for learning.\nOf these, 80% are replaced with [MASK] , 10% are replaced with randomly selected\ntokens, and the remaining 10% are left unchanged.\nThe MLM training objective is to predict the original inputs for each of the\nmasked tokens using a bidirectional encoder of the kind described in the last section.\nThe cross-entropy loss from these predictions drives the training process for all the\nparameters in the model. Note that all of the input tokens play a role in the self-\nattention process, but only the sampled tokens are used for learning.\nMore speci\ufb01cally, the original input sequence is \ufb01rst tokenized using a subword\nmodel. The sampled items which drive the learning process are chosen among the\ninput tokens. Word embeddings for all of the tokens in the input are retrieved from\ntheEembedding matrix and combined with positional embeddings to form the input\nto the transformer, passed through the stack of transformer blocks, and then the\nlanguage modeling head.\nLM Head with Softmax over Vocabulary\nSo[mask]and[mask]for longthanksCE Loss\nall apricot \ufb01shthe\nToken +Positional EmbeddingsSolongandthanksfor all \ufb01shthe\nBidirectional Transformer Encoder+p1+++++++p2p3p4p5p6p7p8z1z2z3z4z5z6z7z8\nFigure 11.3 Masked language model training. In this example, three of the input tokens are selected, two of\nwhich are masked and the third is replaced with an unrelated word. The probabilities assigned by the model to\nthese three items are used as the training loss. The other 5 tokens don\u2019t play a role in training loss.\nFig. 11.3 illustrates this approach with a simple example. Here, long,thanks and\nthehave been sampled from the training sequence, with the \ufb01rst two masked and the\nreplaced with the randomly sampled token apricot . The resulting embeddings are\npassed through a stack of bidirectional transformer blocks. Recall from Section ??\nin Chapter 9 that to produce a probability distribution over the vocabulary for each\nof the masked tokens, the language modeling head takes the output vector hL\nifrom\nthe \ufb01nal transformer layer Lfor each masked token i, multiplies it by the unembed-\nding layer ETto produce the logits u, and then uses softmax to turn the logits into\nprobabilities yover the vocabulary:\nui=hL\niET(11.3)\nyi=softmax (ui) (11.4)\nWith a predicted probability distribution for each masked item, we can use cross-\nentropy to compute the loss for each masked item\u2014the negative log probability\nassigned to the actual masked word, as shown in Fig. 11.3. More formally, for a 6CHAPTER 11 \u2022 M ASKED LANGUAGE MODELS\ngiven vector of input tokens in a sentence or batch be x, let the set of tokens that are\nmasked be M, the version of that sentence with some tokens replaced by masks be\nxmask, and the sequence of output vectors be h. For a given input token xi, such as\nthe word long in Fig. 11.3, the loss is the probability of the correct word long, given\nxmask(as summarized in the single output vector hL\ni):\nLMLM(xi) =\u0000logP(xijhL\ni)\nThe gradients that form the basis for the weight updates are based on the average\nloss over the sampled learning items from a single training sequence (or batch of\nsequences).\nLMLM=\u00001\njMjX\ni2MlogP(xijhL\ni)\nNote that only the tokens in Mplay a role in learning; the other words play no role\nin the loss function, so in that sense BERT and its descendents are inef\ufb01cient; only\n15% of the input samples in the training data are actually used for training weights.\n1\n11.2.2 Next Sentence Prediction\nThe focus of mask-based learning is on predicting words from surrounding contexts\nwith the goal of producing effective word-level representations. However, an im-\nportant class of applications involves determining the relationship between pairs of\nsentences. These include tasks like paraphrase detection (detecting if two sentences\nhave similar meanings), entailment (detecting if the meanings of two sentences en-\ntail or contradict each other) or discourse coherence (deciding if two neighboring\nsentences form a coherent discourse).\nTo capture the kind of knowledge required for applications such as these, some\nmodels in the BERT family include a second learning objective called Next Sen-\ntence Prediction (NSP). In this task, the model is presented with pairs of sentencesNext Sentence\nPrediction\nand is asked to predict whether each pair consists of an actual pair of adjacent sen-\ntences from the training corpus or a pair of unrelated sentences. In BERT, 50% of\nthe training pairs consisted of positive pairs, and in the other 50% the second sen-\ntence of a pair was randomly selected from elsewhere in the corpus. The NSP loss\nis based on how well the model can distinguish true pairs from random pairs.\nTo facilitate NSP training, BERT introduces two special tokens to the input rep-\nresentation (tokens that will prove useful for \ufb01netuning as well). After tokenizing\nthe input with the subword model, the token [CLS] is prepended to the input sen-\ntence pair, and the token [SEP] is placed between the sentences and after the \ufb01nal\ntoken of the second sentence. Finally, embeddings representing the \ufb01rst and second\nsegments of the input are added to the word and positional embeddings to allow the\nmodel to more easily distinguish the input sentences.\nDuring training, the output vector hL\nCLSfrom the \ufb01nal layer associated with the\n[CLS] token represents the next sentence prediction. As with the MLM objective,\nwe add a special head, in this case an NSP head, which consists of a learned set of\nclassi\ufb01cation weights WNSP2Rd\u00022that produces a two-class prediction from the\nraw[CLS] vector hL\nCLS:\nyi=softmax (hL\nCLSWNSP)\n1There are members of the BERT family like ELECTRA that do use all examples for training (Clark\net al., 2020). 11.2 \u2022 T RAINING BIDIRECTIONAL ENCODERS 7\nCross entropy is used to compute the NSP loss for each sentence pair presented\nto the model. Fig. 11.4 illustrates the overall NSP training setup. In BERT, the NSP\nloss was used in conjunction with the MLM training objective to form \ufb01nal loss.\nCancelmy\ufb02ight[SEP] 1CE Loss\nAnd the \nBidirectional Transformer Encoder\np1p2p3p4p5p6p7p8[CLS]++s1NSPHeadToken +Segment +PositionalEmbeddings hotel p9[SEP] ++s1s1s1s1s2s2s2s2++++++++++++++hCLS\nFigure 11.4 An example of the NSP loss calculation.\n11.2.3 Training Regimes\nBERT and other early transformer-based language models were trained on about\n3.3 billion words (a combination of English Wikipedia and a corpus of book texts\ncalled BooksCorpus (Zhu et al., 2015) that is no longer used for intellectual property\nreasons). Modern masked language models are now trained on much larger datasets\nof web text, \ufb01ltered a bit, and augmented by higher-quality data like Wikipedia,\nthe same as those we discussed for the causal large language models of Chapter 9.\nMultilingual models similarly use webtext and multilingual Wikipedia. For example\nthe XLM-R model was trained on about 300 billion tokens in 100 languages, taken\nfrom the web via Common Crawl ( https://commoncrawl.org/ ).\nTo train the original BERT models, pairs of text segments were selected from\nthe training corpus according to the next sentence prediction 50/50 scheme. Pairs\nwere sampled so that their combined length was less than the 512 token input. To-\nkens within these sentence pairs were then masked using the MLM approach with\nthe combined loss from the MLM and NSP objectives used for a \ufb01nal loss. Ap-\nproximately 40 passes (epochs) over the training data was required for the model to\nconverge.\nSome models, like the RoBERTa model, drop the next sentence prediction ob-\njective, and therefore change the training regime a bit. Instead of sampling pairs of\nsentence, the input is simply a series of contiguous sentences. If the document runs\nout before 512 tokens are reached, an extra separator token is added, and sentences\nfrom the next document are packed in, until we reach a total of 512 tokens. Usually\nlarge batch sizes are used, between 8K and 32K tokens.\nMultilingual models have an additional decision to make: what data to use to\nbuild the vocabulary? Recall that all language models use subword tokenization\n(BPE or SentencePiece Unigram LM are the two most common algorithms). What\ntext should be used to learn this multilingual tokenization, given that it\u2019s easier to get\nmuch more text in some languages than others? One option would be to create this\nvocabulary-learning dataset by sampling sentences from our training data (perhaps 8CHAPTER 11 \u2022 M ASKED LANGUAGE MODELS\nweb text from Common Crawl), randomly. In that case we will choose a lot of sen-\ntences from languages like languages with lots of web representation like English,\nand the tokens will be biased toward rare English tokens instead of creating frequent\ntokens from languages with less data. Instead, it is common to divide the training\ndata into subcorpora of Ndifferent languages, compute the number of sentences ni\nof each language i, and readjust these probabilities so as to upweight the probability\nof less-represented languages (Lample and Conneau, 2019). The new probability of\nselecting a sentence from each of the Nlanguages (whose prior frequency is ni) is\nfqigi=1:::N, where:\nqi=pa\niPN\nj=1pa\njwith pi=niPN\nk=1nk(11.5)\nRecall from ( ??) in Chapter 6 that an avalue between 0 and 1 will give higher\nweight to lower probability samples. Conneau et al. (2020) show that a=0:3 works\nwell to give rare languages more inclusion in the tokenization, resulting in better\nmultilingual performance overall.\nThe result of this pretraining process consists of both learned word embeddings,\nas well as all the parameters of the bidirectional encoder that are used to produce\ncontextual embeddings for novel inputs.\nFor many purposes, a pretrained multilingual model is more practical than a\nmonolingual model, since it avoids the need to build many (a hundred!) separate\nmonolingual models. And multilingual models can improve performance on low-\nresourced languages by leveraging linguistic information from a similar language in\nthe training data that happens to have more resources. Nonetheless, when the num-\nber of languages grows very large, multilingual models exhibit what has been called\nthecurse of multilinguality (Conneau et al., 2020): the performance on each lan-\nguage degrades compared to a model training on fewer languages. Another problem\nwith multilingual models is that they \u2018have an accent\u2019: grammatical structures in\nhigher-resource languages (often English) bleed into lower-resource languages; the\nvast amount of English language in training makes the model\u2019s representations for\nlow-resource languages slightly more English-like (Papadimitriou et al., 2023).\n11.3 Contextual Embeddings\nGiven a pretrained language model and a novel input sentence, we can think of the\nsequence of model outputs as constituting contextual embeddings for each token incontextual\nembeddings\nthe input. These contextual embeddings are vectors representing some aspect of the\nmeaning of a token in context, and can be used for any task requiring the meaning of\ntokens or words. More formally, given a sequence of input tokens x1; :::;xn, we can\nuse the output vector zifrom the \ufb01nal layer of the model as a representation of the\nmeaning of token xiin the context of sentence x1; :::;xn. Or instead of just using the\nvector zifrom the \ufb01nal layer of the model, it\u2019s common to compute a representation\nforxiby averaging the output tokens zifrom each of the last four layers of the model.\nJust as we used static embeddings like word2vec in Chapter 6 to represent the\nmeaning of words, we can use contextual embeddings as representations of word\nmeanings in context for any task that might require a model of word meaning. Where\nstatic embeddings represent the meaning of word types (vocabulary entries), contex-\ntual embeddings represent the meaning of word instances : instances of a particular 11.3 \u2022 C ONTEXTUAL EMBEDDINGS 9\n[CLS]SolongandthanksforallhL1hLCLShL2hL3hL4hL5hL6\nEi+\nEi+\nEi+\nEi+\nEi+\nEi+\nEi+\nFigure 11.5 The output of a BERT-style model is a contextual embedding vector hL\nifor\neach input token xi.\nword type in a particular context. Thus where word2vec had a single vector for each\nword type, contextual embeddings provide a single vector for each instance of that\nword type in its sentential context. Contextual embeddings can thus be used for\ntasks like measuring the semantic similarity of two words in context, and are useful\nin linguistic tasks that require models of word meaning.\n11.3.1 Contextual Embeddings and Word Sense\nWords are ambiguous : the same word can be used to mean different things. In ambiguous\nChapter 6 we saw that the word \u201cmouse\u201d can mean (1) a small rodent, or (2) a hand-\noperated device to control a cursor. The word \u201cbank\u201d can mean: (1) a \ufb01nancial\ninstitution or (2) a sloping mound. We say that the words \u2018mouse\u2019 or \u2018bank\u2019 are\npolysemous (from Greek \u2018many senses\u2019, poly- \u2018many\u2019 + sema , \u2018sign, mark\u2019).2\nAsense (orword sense ) is a discrete representation of one aspect of the meaning word sense\nof a word. We can represent each sense with a superscript: bank1andbank2,\nmouse1andmouse2. These senses can be found listed in online thesauruses (or\nthesauri) like WordNet (Fellbaum, 1998), which has datasets in many languages WordNet\nlisting the senses of many words. In context, it\u2019s easy to see the different meanings:\nmouse1: .... a mouse controlling a computer system in 1968.\nmouse2: .... a quiet animal like a mouse\nbank1: ...a bank can hold the investments in a custodial account ...\nbank2: ...as agriculture burgeons on the east bank , the river ...\nThis fact that context disambiguates the senses of mouse andbank above can\nalso be visualized geometrically. Fig. 11.6 shows a two-dimensional projection of\nmany instances of the BERT embeddings of the word diein English and German.\nEach point in the graph represents the use of diein one input sentence. We can\nclearly see at least two different English senses of die(the singular of dice and the\nverb to die , as well as the German article, in the BERT embedding space.\n2The word polysemy itself is ambiguous; you may see it used in a different way, to refer only to cases\nwhere a word\u2019s senses are related in some structured way, reserving the word homonymy to mean sense\nambiguities with no relation between the senses (Haber and Poesio, 2020). Here we will use \u2018polysemy\u2019\nto mean any kind of sense ambiguity, and \u2018structured polysemy\u2019 for polysemy with sense relations. 10 CHAPTER 11 \u2022 M ASKED LANGUAGE MODELS\nFigure 4: Embeddings for the word \"die\" in different contexts, visualized with UMAP. Sample points\nare annotated with corresponding sentences. Overall annotations (blue text) are added as a guide.\n4.1 Visualization of word senses\nOur \ufb01rst experiment is an exploratory visualization of how word sense affects context embeddings.\nFor data on different word senses, we collected all sentences used in the introductions to English-\nlanguage Wikipedia articles. (Text outside of introductions was frequently fragmentary.) We created\nan interactive application, which we plan to make public. A user enters a word, and the system\nretrieves 1,000 sentences containing that word. It sends these sentences to BERT-base as input, and\nfor each one it retrieves the context embedding for the word from a layer of the user\u2019s choosing.\nThe system visualizes these 1,000 context embeddings using UMAP [ 15], generally showing clear\nclusters relating to word senses. Different senses of a word are typically spatially separated, and\nwithin the clusters there is often further structure related to \ufb01ne shades of meaning. In Figure 4, for\nexample, we not only see crisp, well-separated clusters for three meanings of the word \u201cdie,\u201d but\nwithin one of these clusters there is a kind of quantitative scale, related to the number of people\ndying. See Appendix 6.4 for further examples. The apparent detail in the clusters we visualized raises\ntwo immediate questions. First, is it possible to \ufb01nd quantitative corroboration that word senses are\nwell-represented? Second, how can we resolve a seeming contradiction: in the previous section, we\nsaw how position represented syntax; yet here we see position representing semantics.\n4.2 Measurement of word sense disambiguation capability\nThe crisp clusters seen in visualizations such as Figure 4 suggest that BERT may create simple,\neffective internal representations of word senses, putting different meanings in different locations. To\ntest this hypothesis quantitatively, we test whether a simple classi\ufb01er on these internal representations\ncan perform well at word-sense disambiguation (WSD).\nWe follow the procedure described in [ 20], which performed a similar experiment with the ELMo\nmodel. For a given word with nsenses, we make a nearest-neighbor classi\ufb01er where each neighbor is\nthe centroid of a given word sense\u2019s BERT-base embeddings in the training data. To classify a new\nword we \ufb01nd the closest of these centroids, defaulting to the most commonly used sense if the word\nwas not present in the training data. We used the data and evaluation from [ 21]: the training data was\nSemCor [17] (33,362 senses), and the testing data was the suite described in [21] (3,669 senses).\nThe simple nearest-neighbor classi\ufb01er achieves an F1 score of 71.1, higher than the current state of\nthe art (Table 1), with the accuracy monotonically increasing through the layers. This is a strong\nsignal that context embeddings are representing word-sense information. Additionally, an even higher\nscore of 71.5 was obtained using the technique described in the following section.\n6\nFigure 11.6 Each blue dot shows a BERT contextual embedding for the word diefrom different sentences\nin English and German, projected into two dimensions with the UMAP algorithm. The German and English\nmeanings and the different English senses fall into different clusters. Some sample points are shown with the\ncontextual sentence they came from. Figure from Coenen et al. (2019).\nThus while thesauruses like WordNet give discrete lists of senses, embeddings\n(whether static or contextual) offer a continuous high-dimensional model of meaning\nthat, although it can be clustered, doesn\u2019t divide up into fully discrete senses.\nWord Sense Disambiguation\nThe task of selecting the correct sense for a word is called word sense disambigua-\ntion, orWSD . WSD algorithms take as input a word in context and a \ufb01xed inventoryword sense\ndisambiguation\nWSD of potential word senses (like the ones in WordNet) and outputs the correct word\nsense in context. Fig. 11.7 sketches out the task.\nanelectricguitarandbassplayerstando\ufb00toonesideelectric1: using electricityelectric2:  tenseelectric3: thrillingguitar1 bass1: low range\u2026bass4: sea \ufb01sh\u2026 bass7: instrument\u2026player1: in gameplayer2: musician player3: actor\u2026stand1: upright\u2026stand5: bear\u2026 stand10: put upright\u2026side1: relative region\u2026side3: of body\u2026 side11: slope\u2026x1y1x2y2x3y3y4y5y6\nx4x5x6\nFigure 11.7 The all-words WSD task, mapping from input words ( x) to WordNet senses\n(y). Figure inspired by Chaplot and Salakhutdinov (2018).\nWSD can be a useful analytic tool for text analysis in the humanities and social\nsciences, and word senses can play a role in model interpretability for word repre-\nsentations. Word senses also have interesting distributional properties. For example 11.3 \u2022 C ONTEXTUAL EMBEDDINGS 11\na word often is used in roughly the same sense through a discourse, an observation\ncalled the one sense per discourse rule (Gale et al., 1992).one sense per\ndiscourse\nThe best performing WSD algorithm is a simple 1-nearest-neighbor algorithm\nusing contextual word embeddings, due to Melamud et al. (2016) and Peters et al.\n(2018). At training time we pass each sentence in some sense-labeled dataset (like\nthe SemCore or SenseEval datasets in various languages) through any contextual\nembedding (e.g., BERT) resulting in a contextual embedding for each labeled token.\n(There are various ways to compute this contextual embedding vifor a token i; for\nBERT it is common to pool multiple layers by summing the vector representations\nofifrom the last four BERT layers). Then for each sense sof any word in the corpus,\nfor each of the ntokens of that sense, we average their ncontextual representations\nvito produce a contextual sense embedding v sfors:\nvs=1\nnX\nivi8vi2tokens (s) (11.6)\nAt test time, given a token of a target word tin context, we compute its contextual\nembedding tand choose its nearest neighbor sense from the training set, i.e., the\nsense whose sense embedding has the highest cosine with t:\nsense(t) =argmax\ns2senses (t)cosine (t;vs) (11.7)\nFig. 11.8 illustrates the model.\nI  found  the  jar  emptycIcfoundfind1vcthecjarcemptyfind9vfind5vfind4vENCODER\nFigure 11.8 The nearest-neighbor algorithm for WSD. In green are the contextual embed-\ndings precomputed for each sense of each word; here we just show a few of the senses for\n\ufb01nd. A contextual embedding is computed for the target word found , and then the nearest\nneighbor sense (in this case \ufb01nd9v) is chosen. Figure inspired by Loureiro and Jorge (2019).\n11.3.2 Contextual Embeddings and Word Similarity\nIn Chapter 6 we introduced the idea that we could measure the similarity of two\nwords by considering how close they are geometrically, by using the cosine as a\nsimilarity function. The idea of meaning similarity is also clear geometrically in the\nmeaning clusters in Fig. 11.6; the representation of a word which has a particular\nsense in a context is closer to other instances of the same sense of the word. Thus we\noften measure the similarity between two instances of two words in context (or two\ninstances of the same word in two different contexts) by using the cosine between\ntheir contextual embeddings. 12 CHAPTER 11 \u2022 M ASKED LANGUAGE MODELS\nUsually some transformations to the embeddings are required before computing\ncosine. This is because contextual embeddings (whether from masked language\nmodels or from autoregressive ones) have the property that the vectors for all words\nare extremely similar. If we look at the embeddings from the \ufb01nal layer of BERT\nor other models, embeddings for instances of any two randomly chosen words will\nhave extremely high cosines that can be quite close to 1, meaning all word vectors\ntend to point in the same direction. The property of vectors in a system all tending\nto point in the same direction is known as anisotropy . Ethayarajh (2019) de\ufb01nes\ntheanisotropy of a model as the expected cosine similarity of any pair of words in anisotropy\na corpus. The word \u2018isotropy\u2019 means uniformity in all directions, so in an isotropic\nmodel, the collection of vectors should point in all directions and the expected cosine\nbetween a pair of random embeddings would be zero. Timkey and van Schijndel\n(2021) show that one cause of anisotropy is that cosine measures are dominated by\na small number of dimensions of the contextual embedding whose values are very\ndifferent than the others: these rogue dimensions have very large magnitudes and\nvery high variance.\nTimkey and van Schijndel (2021) shows that we can make the embeddings more\nisotropic by standardizing (z-scoring) the vectors, i.e., subtracting the mean and\ndividing by the variance. Given a set Cof all the embeddings in some corpus, each\nwith dimensionality d(i.e., x2Rd), the mean vector m2Rdis:\nm=1\njCjX\nx2Cx (11.8)\nThe standard deviation in each dimension s2Rdis:\ns=s\n1\njCjX\nx2C(x\u0000m)2(11.9)\nThen each word vector xis replaced by a standardized version z:\nz=x\u0000m\ns(11.10)\nOne problem with cosine that is not solved by standardization is that cosine tends\nto underestimate human judgments on similarity of word meaning for very frequent\nwords (Zhou et al., 2022).\nIn the next section we\u2019ll see the most common use of contextual representations:\nas representations of words or even entire sentences that can be the inputs to classi-\n\ufb01ers in the \ufb01netuning process for downstream NLP applications.\n11.4 Fine-Tuning for Classi\ufb01cation\nThe power of pretrained language models lies in their ability to extract generaliza-\ntions from large amounts of text\u2014generalizations that are useful for myriad down-\nstream applications. There are two ways to make practical use of the generalizations\nto solve downstream tasks. The most common way is to use natural language to\nprompt the model, putting it in a state where it contextually generates what we\nwant. We\u2019ll introduce prompting in Chapter 12.\nIn this section we explore an alternative way to use pretrained language models\nfor downstream applications: a version of the \ufb01netuning paradigm from Chapter 10. \ufb01netuning 11.4 \u2022 F INE-TUNING FOR CLASSIFICATION 13\nIn the kind of \ufb01netuning used for masked language models, we add application-\nspeci\ufb01c circuitry (often called a special head ) on top of pretrained models, taking\ntheir output as its input. The \ufb01netuning process consists of using labeled data about\nthe application to train these additional application-speci\ufb01c parameters. Typically,\nthis training will either freeze or make only minimal adjustments to the pretrained\nlanguage model parameters.\nThe following sections introduce \ufb01netuning methods for the most common kinds\nof applications: sequence classi\ufb01cation, sentence-pair classi\ufb01cation, and sequence\nlabeling.\n11.4.1 Sequence Classi\ufb01cation\nThe task of sequence classi\ufb01cation is to classify an entire sequence of text with a\nsingle label. This set of tasks is commonly called text classi\ufb01cation , like sentiment\nanalysis or spam detection (Chapter 4) in which we classify a text into two or three\nclasses (like positive or negative), as well as classi\ufb01cation tasks with a large number\nof categories, like document-level topic classi\ufb01cation.\nFor sequence classi\ufb01cation we represent the entire input to be classi\ufb01ed by a\nsingle vector. We can represent a sequence in various ways. One way is to take\nthe sum or the mean of the last output vector from each token int he sequence.\nFor BERT, we instead add a new unique token to the vocabulary called [CLS] , and\nprepended it to the start of all input sequences, both during pretraining and encoding.\nThe output vector in the \ufb01nal layer of the model for the [CLS] input represents\nthe entire input sequence and serves as the input to a classi\ufb01er head , a logistic classi\ufb01er head\nregression or neural network classi\ufb01er that makes the relevant decision.\nAs an example, let\u2019s return to the problem of sentiment classi\ufb01cation. to \ufb01netun-\ning a classi\ufb01er for this application involves learning a set of weights, WC, to map the\noutput vector for the [CLS] token\u2014 hL\nCLS\u2014to a set of scores over the possible senti-\nment classes. Assuming a three-way sentiment classi\ufb01cation task (positive, negative,\nneutral) and dimensionality das the model dimension, WCwill be of size [d\u00023]. To\nclassify a document, we pass the input text through the pretrained language model to\ngenerate hL\nCLS, multiply it by WC, and pass the resulting vector through a softmax.\ny=softmax (hL\nCLSWC) (11.11)\nFinetuning the values in WCrequires supervised training data consisting of input\nsequences labeled with the appropriate sentiment class. Training proceeds in the\nusual way; cross-entropy loss between the softmax output and the correct answer is\nused to drive the learning that produces WC.\nA key difference from what we\u2019ve seen earlier with neural classi\ufb01ers is that this\nloss can be used to not only learn the weights of the classi\ufb01er, but also to update the\nweights for the pretrained language model itself. In practice, reasonable classi\ufb01ca-\ntion performance is typically achieved with only minimal changes to the language\nmodel parameters, often limited to updates over the \ufb01nal few layers of the trans-\nformer. Fig. 11.9 illustrates this overall approach to sequence classi\ufb01cation.\n11.4.2 Sequence-Pair Classi\ufb01cation\nAs mentioned in Section 11.2.2, an important type of problem involves the classi\ufb01ca-\ntion of pairs of input sequences. Practical applications that fall into this class include\nparaphrase detection (are the two sentences paraphrases of each other?), logical en- 14 CHAPTER 11 \u2022 M ASKED LANGUAGE MODELS\n[CLS]entirelypredictableandlacksenergy\nBidirectional Transformer EncoderhCLS\nEi+\nEi+\nEi+\nEi+\nEi+\nEi+sentimentclassi\ufb01cation headWCy\nFigure 11.9 Sequence classi\ufb01cation with a bidirectional transformer encoder. The output vector for the\n[CLS] token serves as input to a simple classi\ufb01er.\ntailment (does sentence A logically entail sentence B?), and discourse coherence\n(how coherent is sentence B as a follow-on to sentence A?).\nFine-tuning an application for one of these tasks proceeds just as with pretrain-\ning using the NSP objective. During \ufb01netuning, pairs of labeled sentences from a\nsupervised \ufb01netuning set are presented to the model, and run through all the layers\nof the model to produce the houtputs for each input token. As with sequence classi-\n\ufb01cation, the output vector associated with the prepended [CLS] token represents the\nmodel\u2019s view of the input pair. And as with NSP training, the two inputs are sepa-\nrated by the [SEP] token. To perform classi\ufb01cation, the [CLS] vector is multiplied\nby a set of learning classi\ufb01cation weights and passed through a softmax to generate\nlabel predictions, which are then used to update the weights.\nAs an example, let\u2019s consider an entailment classi\ufb01cation task with the Multi-\nGenre Natural Language Inference (MultiNLI) dataset (Williams et al., 2018). In\nthe task of natural language inference orNLI, also called recognizing textualnatural\nlanguage\ninferenceentailment , a model is presented with a pair of sentences and must classify the re-\nlationship between their meanings. For example in the MultiNLI corpus, pairs of\nsentences are given one of 3 labels: entails ,contradicts andneutral . These labels\ndescribe a relationship between the meaning of the \ufb01rst sentence (the premise) and\nthe meaning of the second sentence (the hypothesis). Here are representative exam-\nples of each class from the corpus:\n\u2022Neutral\na: Jon walked back to the town to the smithy.\nb: Jon traveled back to his hometown.\n\u2022Contradicts\na: Tourist Information of\ufb01ces can be very helpful.\nb: Tourist Information of\ufb01ces are never of any help.\n\u2022Entails\na: I\u2019m confused.\nb: Not all of it is very clear to me. 11.5 \u2022 F INE-TUNING FOR SEQUENCE LABELLING : NAMED ENTITY RECOGNITION 15\nA relationship of contradicts means that the premise contradicts the hypothesis; en-\ntails means that the premise entails the hypothesis; neutral means that neither is\nnecessarily true. The meaning of these labels is looser than strict logical entailment\nor contradiction indicating that a typical human reading the sentences would most\nlikely interpret the meanings in this way.\nTo \ufb01netune a classi\ufb01er for the MultiNLI task, we pass the premise/hypothesis\npairs through a bidirectional encoder as described above and use the output vector\nfor the [CLS] token as the input to the classi\ufb01cation head. As with ordinary sequence\nclassi\ufb01cation, this head provides the input to a three-way classi\ufb01er that can be trained\non the MultiNLI training corpus.\n11.5 Fine-Tuning for Sequence Labelling: Named En-\ntity Recognition\nIn sequence labeling, the network\u2019s task is to assign a label chosen from a small\n\ufb01xed set of labels to each token in the sequence. One of the most common sequence\nlabeling task is named entity recognition .\n11.5.1 Named Entities\nAnamed entity is, roughly speaking, anything that can be referred to with a proper named entity\nname: a person, a location, an organization. The task of named entity recognitionnamed entity\nrecognition\n(NER ) is to \ufb01nd spans of text that constitute proper names and tag the type of the NER\nentity. Four entity tags are most common: PER (person), LOC (location), ORG\n(organization), or GPE (geo-political entity). However, the term named entity is\ncommonly extended to include things that aren\u2019t entities per se, including temporal\nexpressions like dates and times, and even numerical expressions like prices. Here\u2019s\nan example of the output of an NER tagger:\nCiting high fuel prices, [ ORG United Airlines ] said [ TIME Friday ] it\nhas increased fares by [ MONEY $6] per round trip on \ufb02ights to some\ncities also served by lower-cost carriers. [ ORG American Airlines ], a\nunit of [ ORG AMR Corp.] , immediately matched the move, spokesman\n[PER Tim Wagner ] said. [ ORG United] , a unit of [ ORG UAL Corp.] ,\nsaid the increase took effect [ TIME Thursday] and applies to most\nroutes where it competes against discount carriers, such as [ LOC Chicago]\nto [LOC Dallas] and [ LOC Denver] to [LOC San Francisco] .\nThe text contains 13 mentions of named entities including 5 organizations, 4 loca-\ntions, 2 times, 1 person, and 1 mention of money. Figure 11.10 shows typical generic\nnamed entity types. Many applications will also need to use speci\ufb01c entity types like\nproteins, genes, commercial products, or works of art.\nType Tag Sample Categories Example sentences\nPeople PER people, characters Turing is a giant of computer science.\nOrganization ORG companies, sports teams The IPCC warned about the cyclone.\nLocation LOC regions, mountains, seas Mt. Sanitas is in Sunshine Canyon .\nGeo-Political Entity GPE countries, states Palo Alto is raising the fees for parking.\nFigure 11.10 A list of generic named entity types with the kinds of entities they refer to. 16 CHAPTER 11 \u2022 M ASKED LANGUAGE MODELS\nNamed entity recognition is a useful step in various natural language processing\ntasks, including linking text to information in structured knowledge sources like\nWikipedia, measuring sentiment or attitudes toward a particular entity in text, or\neven as part of anonymizing text for privacy. The NER task is is dif\ufb01cult because\nof the ambiguity of segmenting NER spans, \ufb01guring out which tokens are entities\nand which aren\u2019t, since most words in a text will not be named entities. Another\ndif\ufb01culty is caused by type ambiguity. The mention Washington can refer to a\nperson, a sports team, a city, or the US government, as we see in Fig. 11.11.\n[PER Washington] was born into slavery on the farm of James Burroughs.\n[ORG Washington] went up 2 games to 1 in the four-game series.\nBlair arrived in [ LOC Washington] for what may well be his last state visit.\nIn June, [ GPE Washington] passed a primary seatbelt law.\nFigure 11.11 Examples of type ambiguities in the use of the name Washington .\n11.5.2 BIO Tagging\nOne standard approach to sequence labeling for a span-recognition problem like\nNER is BIO tagging (Ramshaw and Marcus, 1995). This is a method that allows us BIO tagging\nto treat NER like a word-by-word sequence labeling task, via tags that capture both\nthe boundary and the named entity type. Consider the following sentence:\n[PER Jane Villanueva ] of [ ORG United ] , a unit of [ ORG United Airlines\nHolding ] , said the fare applies to the [ LOC Chicago ] route.\nFigure 11.12 shows the same excerpt represented with BIO tagging, as well as BIO\nvariants called IOtagging and BIOES tagging. In BIO tagging we label any token\nthatbegins a span of interest with the label B, tokens that occur inside a span are\ntagged with an I, and any tokens outside of any span of interest are labeled O. While\nthere is only one Otag, we\u2019ll have distinct Band Itags for each named entity class.\nThe number of tags is thus 2 n+1 tags, where nis the number of entity types. BIO\ntagging can represent exactly the same information as the bracketed notation, but has\nthe advantage that we can represent the task in the same simple sequence modeling\nway as part-of-speech tagging: assigning a single label yito each input word xi:\nWords IO Label BIO Label BIOES Label\nJane I-PER B-PER B-PER\nVillanueva I-PER I-PER E-PER\nof O O O\nUnited I-ORG B-ORG B-ORG\nAirlines I-ORG I-ORG I-ORG\nHolding I-ORG I-ORG E-ORG\ndiscussed O O O\nthe O O O\nChicago I-LOC B-LOC S-LOC\nroute O O O\n. O O O\nFigure 11.12 NER as a sequence model, showing IO, BIO, and BIOES taggings.\nWe\u2019ve also shown two variant tagging schemes: IO tagging, which loses some\ninformation by eliminating the B tag, and BIOES tagging, which adds an end tag E\nfor the end of a span, and a span tag Sfor a span consisting of only one word. 11.5 \u2022 F INE-TUNING FOR SEQUENCE LABELLING : NAMED ENTITY RECOGNITION 17\n11.5.3 Sequence Labeling\nIn sequence labeling, we pass the \ufb01nal output vector corresponding to each input\ntoken to a classi\ufb01er that produces a softmax distribution over the possible set of\ntags. For a single feedforward layer classi\ufb01er, the set of weights to be learned is\nWKof size [d\u0002k], where kis the number of possible tags for the task. A greedy\napproach, where the argmax tag for each token is taken as a likely answer, can be\nused to generate the \ufb01nal output tag sequence. Fig. 11.13 illustrates an example of\nthis approach, where yiis a vector of probabilities over tags, and kindexes the tags.\nyi=softmax (hL\niWK) (11.12)\nti=argmaxk(yi) (11.13)\nAlternatively, the distribution over labels provided by the softmax for each input\ntoken can be passed to a conditional random \ufb01eld (CRF) layer which can take global\ntag-level transitions into account (see Chapter 17 on CRFs).\n[CLS]JaneVillanuevaofUnitedAirlines\nBidirectional Transformer EncoderB-PERI-PEROB-ORGI-ORG\nHoldingdiscussedI-ORGOWKNER headhiargmax\nEi+\nEi+\nEi+\nEi+\nEi+\nEi+\nEi+\nEi+WKWKWKWKWKWKyi\nFigure 11.13 Sequence labeling for named entity recognition with a bidirectional transformer encoder. The\noutput vector for each input token is passed to a simple k-way classi\ufb01er.\nTokenization and NER\nNote that supervised training data for NER is typically in the form of BIO tags as-\nsociated with text segmented at the word level. For example the following sentence\ncontaining two named entities:\n[LOC Mt. Sanitas ] is in [ LOC Sunshine Canyon ] .\nwould have the following set of per-word BIO tags.\n(11.14) Mt.\nB-LOCSanitas\nI-LOCis\nOin\nOSunshine\nB-LOCCanyon\nI-LOC.\nO\nUnfortunately, the sequence of WordPiece tokens for this sentence doesn\u2019t align\ndirectly with BIO tags in the annotation: 18 CHAPTER 11 \u2022 M ASKED LANGUAGE MODELS\n'Mt', '.', 'San', '##itas', 'is', 'in', 'Sunshine', 'Canyon' '.'\nTo deal with this misalignment, we need a way to assign BIO tags to subword\ntokens during training and a corresponding way to recover word-level tags from\nsubwords during decoding. For training, we can just assign the gold-standard tag\nassociated with each word to all of the subword tokens derived from it.\nFor decoding, the simplest approach is to use the argmax BIO tag associated with\nthe \ufb01rst subword token of a word. Thus, in our example, the BIO tag assigned to\n\u201cMt\u201d would be assigned to \u201cMt.\u201d and the tag assigned to \u201cSan\u201d would be assigned\nto \u201cSanitas\u201d, effectively ignoring the information in the tags assigned to \u201c.\u201d and\n\u201c##itas\u201d. More complex approaches combine the distribution of tag probabilities\nacross the subwords in an attempt to \ufb01nd an optimal word-level tag.\n11.5.4 Evaluating Named Entity Recognition\nNamed entity recognizers are evaluated by recall ,precision , and F1measure . Re-\ncall that recall is the ratio of the number of correctly labeled responses to the total\nthat should have been labeled; precision is the ratio of the number of correctly la-\nbeled responses to the total labeled; and F-measure is the harmonic mean of the\ntwo.\nTo know if the difference between the F 1scores of two NER systems is a signif-\nicant difference, we use the paired bootstrap test, or the similar randomization test\n(Section ??).\nFor named entity tagging, the entity rather than the word is the unit of response.\nThus in the example in Fig. 11.12, the two entities Jane Villanueva andUnited Air-\nlines Holding and the non-entity discussed would each count as a single response.\nThe fact that named entity tagging has a segmentation component which is not\npresent in tasks like text categorization or part-of-speech tagging causes some prob-\nlems with evaluation. For example, a system that labeled Jane but not Jane Vil-\nlanueva as a person would cause two errors, a false positive for O and a false nega-\ntive for I-PER. In addition, using entities as the unit of response but words as the unit\nof training means that there is a mismatch between the training and test conditions.\n11.6 Summary\nThis chapter has introduced the bidirectional encoder and the masked language\nmodel . Here\u2019s a summary of the main points that we covered:\n\u2022 Bidirectional encoders can be used to generate contextualized representations\nof input embeddings using the entire input context.\n\u2022 Pretrained language models based on bidirectional encoders can be learned\nusing a masked language model objective where a model is trained to guess\nthe missing information from an input.\n\u2022 The vector output of each transformer block or component in a particular to-\nken column is a contextual embedding that represents some aspect of the\nmeaning of a token in context.\n\u2022 A word sense is a discrete representation of one aspect of the meaning of a\nword. Contextual embeddings offer a continuous high-dimensional model of\nmeaning that is richer than fully discrete senses. BIBLIOGRAPHICAL AND HISTORICAL NOTES 19\n\u2022 The cosine between contextual embeddings can be used as one way to model\nthe similarity between two words in context, although some transformations\nto the embeddings are required \ufb01rst.\n\u2022 Pretrained language models can be \ufb01netuned for speci\ufb01c applications by adding\nlightweight classi\ufb01er layers on top of the outputs of the pretrained model.\n\u2022 These applications can include sequence classi\ufb01cation tasks like sentiment\nanalysis, sequence-pair classi\ufb01cation tasks like natural language inference,\norsequence labeling tasks like named entity recognition .\nBibliographical and Historical Notes\nHistory TBD. 20 Chapter 11 \u2022 Masked Language Models\nChaplot, D. S. and R. Salakhutdinov. 2018. Knowledge-\nbased word sense disambiguation using topic models.\nAAAI .\nClark, K., M.-T. Luong, Q. V . Le, and C. D. Manning.\n2020. Electra: Pre-training text encoders as discrimina-\ntors rather than generators. ICLR .\nCoenen, A., E. Reif, A. Yuan, B. Kim, A. Pearce, F. Vi \u00b4egas,\nand M. Wattenberg. 2019. Visualizing and measuring the\ngeometry of bert. NeurIPS .\nConneau, A., K. Khandelwal, N. Goyal, V . Chaudhary,\nG. Wenzek, F. Guzm \u00b4an, E. Grave, M. Ott, L. Zettlemoyer,\nand V . Stoyanov. 2020. Unsupervised cross-lingual rep-\nresentation learning at scale. ACL.\nDevlin, J., M.-W. Chang, K. Lee, and K. Toutanova. 2019.\nBERT: Pre-training of deep bidirectional transformers for\nlanguage understanding. NAACL HLT .\nEthayarajh, K. 2019. How contextual are contextual-\nized word representations? Comparing the geometry of\nBERT, ELMo, and GPT-2 embeddings. EMNLP .\nFellbaum, C., ed. 1998. WordNet: An Electronic Lexical\nDatabase . MIT Press.\nGale, W. A., K. W. Church, and D. Yarowsky. 1992. One\nsense per discourse. HLT.\nHaber, J. and M. Poesio. 2020. Assessing polyseme sense\nsimilarity through co-predication acceptability and con-\ntextualised embedding distance. *SEM .\nJoshi, M., D. Chen, Y . Liu, D. S. Weld, L. Zettlemoyer, and\nO. Levy. 2020. SpanBERT: Improving pre-training by\nrepresenting and predicting spans. TACL , 8:64\u201377.\nKudo, T. and J. Richardson. 2018. SentencePiece: A simple\nand language independent subword tokenizer and detok-\nenizer for neural text processing. EMNLP .\nLample, G. and A. Conneau. 2019. Cross-lingual language\nmodel pretraining. NeurIPS , volume 32.\nLiu, Y ., M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen,\nO. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov.\n2019. RoBERTa: A robustly optimized BERT pretraining\napproach. ArXiv preprint arXiv:1907.11692.\nLoureiro, D. and A. Jorge. 2019. Language modelling makes\nsense: Propagating representations through WordNet for\nfull-coverage word sense disambiguation. ACL.\nMelamud, O., J. Goldberger, and I. Dagan. 2016. con-\ntext2vec: Learning generic context embedding with bidi-\nrectional LSTM. CoNLL .\nPapadimitriou, I., K. Lopez, and D. Jurafsky. 2023. Multilin-\ngual BERT has an accent: Evaluating English in\ufb02uences\non \ufb02uency in multilingual models. EACL Findings .\nPeters, M., M. Neumann, M. Iyyer, M. Gardner, C. Clark,\nK. Lee, and L. Zettlemoyer. 2018. Deep contextualized\nword representations. NAACL HLT .\nRamshaw, L. A. and M. P. Marcus. 1995. Text chunking\nusing transformation-based learning. Proceedings of the\n3rd Annual Workshop on Very Large Corpora .\nSchuster, M. and K. Nakajima. 2012. Japanese and Korean\nvoice search. ICASSP .\nTaylor, W. L. 1953. Cloze procedure: A new tool for mea-\nsuring readability. Journalism Quarterly , 30:415\u2013433.Timkey, W. and M. van Schijndel. 2021. All bark and no\nbite: Rogue dimensions in transformer language models\nobscure representational quality. EMNLP .\nWilliams, A., N. Nangia, and S. Bowman. 2018. A broad-\ncoverage challenge corpus for sentence understanding\nthrough inference. NAACL HLT .\nZhou, K., K. Ethayarajh, D. Card, and D. Jurafsky. 2022.\nProblems with cosine as a measure of embedding simi-\nlarity for high frequency words. ACL.\nZhu, Y ., R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun,\nA. Torralba, and S. Fidler. 2015. Aligning books and\nmovies: Towards story-like visual explanations by watch-\ning movies and reading books. IEEE International Con-\nference on Computer Vision .", "12.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\n12Model Alignment, Prompting,\nand In-Context Learning\n\u201cHal, \u201d said Bowman, now speaking with an icy calm. \u201cI am not incapaci-\ntated. Unless you obey my instructions, I shall be forced to disconnect you. \u201d\nArthur C. Clarke\nIn this chapter we show how to get LLMs to do tasks for us simply by talking to\nthem. To get an LLM to translate a sentence, outline a talk, or draft a work email,\nwe\u2019ll simply describe what we want in natural language. We call these instructions\nwe give to language models prompts . prompts\nPrompting relies on contextual generation. Given the prompt as context, the lan-\nguage model generates the next token based on its token probability, conditioned on\nthe prompt: P(wijw<i). A prompt can be a question (like \u201cWhat is a transformer net-\nwork?\u201d), possibly in a structured format (like \u201cQ: What is a transformer network?\nA:\u201d), or can be an instruction (like \u201cTranslate the following sentence into Hindi:\n\u2018Chop the garlic \ufb01nely\u2019\u201d). A prompt can also contain demonstrations , examples to demonstrations\nhelp make the instructions clearer, (like \u201cGive the sentiment of the following sen-\ntence. Example Input: \u201cI really loved Taishan Cuisine.\u201d Output: positive\u201d.) As we\u2019ll\nsee, prompting can be applied to inherently generative tasks (like summarization and\ntranslation) as well as to ones more naturally thought of as classi\ufb01cation tasks.\nPrompts get language models to generate text, but they also can be viewed as\nalearning signal, because these demonstrations can help language models learn\nto perform novel tasks. For this reason we also refer to prompting as in-context-\nlearning \u2014learning that improves model performance or reduces some loss but doesin-context-\nlearning\nnot involve gradient-based updates to the model\u2019s underlying parameters.\nBut LLMs as we\u2019ve described them so far turn out to be bad at following instruc-\ntions. Pretraining isn\u2019t suf\ufb01cient to make them helpful . We\u2019ll introduce instruction\ntuning , a technique that helps LLMs learn to correctly respond to instructions byinstruction\ntuning\n\ufb01netuning them on a corpus of instructions with their corresponding response.\nA second failure of LLMs is that they can be harmful : their pretraining isn\u2019t\nsuf\ufb01cient to make them safe. Readers who know Arthur C. Clarke\u2019s 2001: A Space\nOdyssey or the Stanley Kubrick \ufb01lm know that the quote above comes in the context\nthat the arti\ufb01cial intelligence Hal becomes paranoid and tries to kill the crew of the\nspaceship. Unlike Hal, language models don\u2019t have intentionality or mental health\nissues like paranoid thinking, but they do have the capacity for harm. Pretrained lan-\nguage models can say things that are dangerous or false (like giving unsafe medical\nadvice) and they can verbally attack users or say toxic or hateful things.\nDealing with safety can be done partly by adding safety training into instruction\ntuning. But an important aspect of safety training is a second technique, preference\nalignment (often implemented, as we\u2019ll see, with the RLHF orDPO algorithms) inpreference\nalignment\nwhich a separate model is trained to decide how much a candidate response aligns\nwith human preferences. Together we refer to instructing tuning and preference\nalignment as model alignment . The intuition is that we want the learning objectivesmodel\nalignment\nof models to be aligned with the goals of the humans that use them. 2CHAPTER 12 \u2022 M ODEL ALIGNMENT , PROMPTING ,AND IN-CONTEXT LEARNING\n12.1 Prompting\nAprompt is a text string that a user issues to a language model to get the model prompt\nto do something useful. In prompting, the user\u2019s prompt string is passed to the\nlanguage model, which iteratively generates tokens conditioned on the prompt. Thus\nthe prompt creates a context that guides LLMs to generate useful outputs to achieve\nsome user goal. The process of \ufb01nding effective prompts for a task is known as\nprompt engineering .prompt\nengineering\nLet\u2019s see how to prompt a language model to solve a simple sentiment classi\ufb01-\ncation task, Consider this hotel review from the BLT corpus (Salvetti et al., 2016):\nSample Hotel Review\nDid not like the service that I was provided, when I entered the hotel. I also\ndid not like the area, in which the hotel was located. Too much noise and\nevents going on for me to feel relax.\nWe can get the model to classify the sentiment of this text by taking the text and\nappending an incomplete statement to the review like In short, our stay was :\nA prompt consisting of a review plus an incomplete statement\nDid not like the service that I was provided, when I entered the hotel. I also\ndid not like the area, in which the hotel was located. Too much noise and\nevents going on for me to feel relax. In short, our stay was\nWe then have an LLM complete the statement by generating a token conditioned\non the prompt so far, and then generating the next token (conditioned on the prompt\nplus the new token), and so forth. Here are a few responses from a language model\nvia continued generation starting with the prompt as context.\nHotel Review Completions\nDid not like the service that I was provided, when I entered the hotel. I also\ndid not like the area, in which the hotel was located. Too much noise and\nevents going on for me to feel relaxed. In short our stay was\n... not a pleasant one. The staff at the front desk\nwere not welcoming or friendly, and seemed disinterested\nin providing good customer service.\n... uncomfortable and not worth the price we paid. We\nwill not be returning to this hotel.\nAs we can see, the overall negative context of the review results in negative\ncompletions. We could easily map these completions to the class we are trying\nto predict, perhaps via some prede\ufb01ned mappings, like fexcellent!positiveg,\nfdid not like!negativeg, and so on.\nThe power of this approach is that with suitable additions to the context a single\nLLM can produce outputs appropriate for many different tasks . For example, given 12.1 \u2022 P ROMPTING 3\na review we might want any of the following:\n\u2022 A summary,\n\u2022 Whether the review was truthful or likely to have been fabricated,\n\u2022 A translation to another language.\nLLMs have a striking ability to perform tasks like these, needing just the appro-\npriate contextual nudge to get the LLM to generate the desired output.\nIf we want to solve general tasks like summarization or translation, we don\u2019t\nwant to have to create a new prompt each time we do the task. Instead the \ufb01rst step\nin prompting is to design one or more templates : task-speci\ufb01c prompting text along templates\nwith slots for the particular input that is being processed.\nConsider the following templates for a variety of tasks:\nBasic Prompt Templates\nSummarization finputg;tldr;\nTranslation finputg;translate to French:\nSentiment finputg;Overall, it was\nFine-Grained- finputg;What aspects were important in this review?\nSentiment\nEach template consists of an input text, designated as finputg, followed by a\nverbatim prompt to be passed to an LLM. These templates are applied to inputs to\ncreate \ufb01lled prompts \u2013 instantiated prompts suitable for use as inputs to an LLM.\nFig. 12.1 illustrates \ufb01lled prompts for these templates using our earlier hotel review,\nalong with sample outputs from an LLM:\nNotice the design pattern of the prompts above: the input is followed by some\ntext which in turn will be completed by the desired response. This style, with the\ninstruction at the end, is common in prompting because it helpfully constrains the\ngeneration. Consider, by contrast, the prompt in Example 12.1.\nTranslate English to French:\nDid not like the service that I was provided! (12.1)\nThis prompt doesn\u2019t do a good job of constraining possible continuations. Instead\nof a French translation, models given this prompt may instead generate another sen-\ntence in English that simply extends the English review. Prompts need to be designed\nunambiguously, so that any reasonable continuation would accomplish the desired\ntask (Reynolds and McDonell, 2021).\nAn even more constraining style of prompt can specify the set of possible an-\nswers in the prompt. For example here is a prompt template to do sentiment analysis\nthat prespeci\ufb01es the potential answers:\nA prompt consisting of a review plus an incomplete statement\nHuman: Do you think that \u201cinput\u201d has negative or positive sentiment?\nChoices:\n(P) Positive\n(N) Negative\nAssistant: I believe the best answer is: ( 4CHAPTER 12 \u2022 M ODEL ALIGNMENT , PROMPTING ,AND IN-CONTEXT LEARNING\nLLM Outputs for Basic Prompts\nOriginal Review ($INPUT) Did not like the service that I was provided,\nwhen I entered the hotel. I also did not like\nthe area, in which the hotel was located. Too\nmuch noise and events going on for me to feel\nrelax and away from the city life.\nSentiment Prompt :$INPUT + In short, our stay was\nOutput :not enjoyable\nFine-grained Sentiment Prompt :$INPUT + These aspects were important to\nthe reviewer:\nOutput :1. Poor service 2. Unpleasant location\n3. Noisy and busy area\nSummarization Prompt :$INPUT + tl;dr\nOutput :I had a bad experience with the hotel's\nservice and the location was loud and busy.\nTranslation Prompt :$INPUT + Translate this to French\nOutput :Je n'ai pas aim \u0013e le service qui m'a \u0013et\u0013e\noffert lorsque je suis entr \u0013e dans l'h ^otel. Je\nn'ai\u0013egalement pas aim \u0013e la zone dans laquelle se\ntrouvait l'h ^otel. Trop de bruit et d' \u0013ev\u0013enements\npour que je me sente d \u0013etendu et loin de la vie\ncitadine.\nFigure 12.1 LLM outputs for simple prompts for sentiment, summarization and translation for an input text.\nThis prompt uses a number of more sophisticated prompting characteristics. It\nspeci\ufb01es the two allowable choices (P) and (N), and ends the prompt with the open\nparenthesis that strongly suggests the answer will be (P) or (N). Note that it also\nspeci\ufb01es the role of the language model as an assistant.\nWe can do even more with prompts. For example, we might want to restrict a\nsummary to be a particular length, to have an answer generated according to some\nkind of persona or role, or to specify a more structured output using a programming\nlanguage or a data interchange format such as JSON. Or we may want to prompt\nthe system to break down complex tasks, using methods like chain-of-thought that\nwe\u2019ll discuss in Section 12.4. All of these kinds of instructions go beyond simple\nprompting and require further LLM \ufb01netuning to enable them to follow instructions.\nWe\u2019ll return to this notion of instruction tuning in Section 12.3.\nIn summary, we prompt an LM by transforming each task into a form that is\namenable to contextual generation by an LLM, as follows:\n1. For a given task, develop a a task-speci\ufb01c template that has a free parameter\nfor the input text.\n2. Given that input and the task-speci\ufb01c template , the input is used to instantiate template\na\ufb01lled prompt that is then passed to a pretrained language model.\n3. Autoregressive decoding is then used to generate a sequence of token outputs.\n4. The output of the model can either be used directly as the desired output (as\nin the case of naturally generative tasks such as translation or summarization),\nor a task-appropriate answer can be extracted from the generated output (as in\nthe case of classi\ufb01cation). 12.1 \u2022 P ROMPTING 5\n12.1.1 Learning from Demonstrations: Few-Shot Prompting\nIt\u2019s often possible to improve a prompt by including some labeled examples in the\nprompt template. We call such examples demonstrations . The task of prompting demonstrations\nwith examples is sometimes called few-shot prompting , as contrasted with zero- few-shot\nshot prompting which means instructions that don\u2019t include labeled examples. zero-shot\nFig. 12.2 illustrates a few-shot example from an extractive question answering\ntask. The context combines the task de\ufb01nition along with three gold-standard ques-\ntion and answer pairs from the training set.\nDe\ufb01nition : This task is about writing a correct answer for the reading comprehension task.\nBased on the information provided in a given passage, you should identify the shortest\ncontinuous text span from the passage that serves as an answer to the given question. Avoid\nanswers that are incorrect or provides incomplete justi\ufb01cation for the question.\nPassage : Beyonc \u00b4e Giselle Knowles-Carter (born September 4, 1981) is an American singer,\nsongwriter, record producer and actress. Born and raised in Houston, Texas, she performed in\nvarious singing and dancing competitions as a child, and rose to fame in the late 1990s as lead\nsinger of R&B girl-group Destiny\u2019s Child. Managed by her father, Mathew Knowles, the group\nbecame one of the world\u2019s best-selling girl groups of all time. Their hiatus saw the release\nof Beyonc \u00b4e\u2019s debut album, Dangerously in Love (2003), which established her as a solo artist\nworldwide, earned \ufb01ve Grammy Awards and featured the Billboard Hot 100 number-one singles\n\u201cCrazy in Love\u201d and \u201cBaby Boy\u201d.\nExamples:\nQ: In what city and state did Beyonc \u00b4e grow up?\nA: Houston, Texas\nQ: What areas did Beyonc \u00b4e compete in when she was growing up?\nA: singing and dancing\nQ: When did Beyonc \u00b4e release Dangerously in Love?\nA: 2003\nQ: When did Beyonc \u00b4e start becoming popular?\nA:\nFigure 12.2 A prompt for extractive question answering, from an example from the SQuAD 2.0 dataset\n(Rajpurkar et al., 2018). The prompt contains the task de\ufb01nition, the passage, 3 demonstration examples,\nfollowed by the test question. This de\ufb01nition speci\ufb01cation and format are after the Natural Instructions dataset\n(Mishra et al., 2022).\nHow Many Demonstrations? The number of demonstrations doesn\u2019t have to be\nlarge. A small number of randomly selected labeled examples used as demonstra-\ntions can be suf\ufb01cient to improve performance over the zero-shot setting. Indeed,\nthe largest performance gains in few-shot prompting tends to come from the \ufb01rst\ntraining example, with diminishing returns for subsequent demonstrations. This is\nin contrast with \ufb01netuning of specialized classi\ufb01er heads that we saw in Chapter 11\nwhere it helps to have lots of examples.\nWhy isn\u2019t it useful to have more demonstrations? The reason is that the primary\nbene\ufb01t in examples is to demonstrate the task to be performed to the LLM and the\nformat of the sequence, not to provide relevant information as to the right answer 6CHAPTER 12 \u2022 M ODEL ALIGNMENT , PROMPTING ,AND IN-CONTEXT LEARNING\nfor any particular question. In fact, demonstrations that have incorrect answers can\nstill improve a system (Min et al., 2022; Webson and Pavlick, 2022). Adding too\nmany examples seems to cause the model to over\ufb01t to details of the exact examples\nchosen and generalize poorly.\nHow to Select Demonstrations? Demonstrations are generally created by for-\nmatting examples drawn from a labeled training set There are some heuristics about\nwhat makes a good demonstration. For example, using demonstrations that are sim-\nilarto the current input seems to improve performance. It can thus be useful to\ndynamically retrieve demonstrations for each input, based on their similarity to the\ncurrent example (for example, comparing the embedding of the current example\nwith embeddings of each of the training set example to \ufb01nd the best top- T).\nBut more generally, the best way to select demonstrations from the training set\nis programmatically: choosing the set of demonstrations that most increases task\nperformance of the prompt on a test set. Task performance for sentiment analysis\nor multiple-choice question answering can be measured in accuracy; for machine\ntranslation with chrF, and for summarization via Rouge. Systems like DSPy (Khat-\ntab et al., 2024), a framework for algorithmically optimizing LM prompts, can au-\ntomatically \ufb01nd the optimum set of demonstrations to include by searching through\nthe space of possible demonstrations to include. We\u2019ll return to automatic prompt\noptimization in Section 12.5.\n12.1.2 In-Context Learning and Induction Heads\nAs a way of getting a model to do what we want, prompting is fundamentally differ-\nent than pretraining. Learning via pretraining means updating the model\u2019s parame-\nters by using gradient descent according to some loss function. But prompting with\ndemonstrations can teach a model to do a new task. The model is learning something\nas it processes the prompt.\nEven without demonstrations, we can think of the process of prompting as a kind\nof learning. For example, the further a model gets in a prompt, the better it tends\nto get at predicting the upcoming tokens. The information in the context is helping\ngive the model more predictive power.\nWe use the term in-context learning to refer to either of these kinds of learningin-context\nlearning\nthat language models do from their prompts. In-context learning means language\nmodels learning to do new tasks, better predict tokens, or generally reduce their\nloss, but without any gradient-based updates to the model\u2019s parameters.\nHow does in-context learning work? While we don\u2019t know for sure, there are\nsome intriguing ideas. One hypothesis is based on the idea of induction heads induction heads\n(Elhage et al., 2021; Olsson et al., 2022). Induction heads are the name for a circuit ,\nwhich is a kind of abstract component of a network. The induction head circuit\nis part of the attention computation in transformers, discovered by looking at mini\nlanguage models with only 1-2 attention heads.\nThe function of the induction head is to predict repeated sequences. For example\nif it sees the pattern AB...A in an input sequence, it predicts that Bwill follow,\ninstantiating the pattern completion ruleAB...A!B. It does this by having a pre\ufb01x\nmatching component of the attention computation that, when looking at the current\ntoken A, searches back over the context to \ufb01nd a prior instance of A. If it \ufb01nds one,\nthe induction head has a copying mechanism that \u201ccopies\u201d the token B that followed\nthe earlier A, by increasing the probability the B will occur next. Fig. 12.3 shows an\nexample. 12.2 \u2022 P OST-TRAINING AND MODEL ALIGNMENT 7\nFigure 1: In the sequence \u201c...vintage cars ... vintage\u201d, an induction head identi\ufb01es the initial occurrence of \u201cvintage\u201d,\nattends to the subsequent word \u201ccars\u201d for pre\ufb01x matching, and predicts \u201ccars\u201d as the next word through the copying\nmechanism.\ndetermines each head\u2019s independent output for the\ncurrent token.\nLeveraging this decomposition, Elhage et al.\n(2021 ) discovered a distinct behaviour in certain\nattention heads, which they named induction heads .\nThis behaviour emerges when these heads process\nsequences of the form \"[A] [B] ... [A] \u2192\". In\nthese heads, the QK circuit directs attention to-\nwards [B], which appears directly after the previous\noccurrence of the current token [A]. This behaviour\nis termed pre\ufb01x matching . The OV circuit subse-\nquently increases the output logit of the [B] token,\ntermed copying . An overview of this mechanism is\nshown in Figure 1.\n4 Methods\n4.1 Models\nWe utilise two recently developed open-source\nmodels, namely Llama-3-8B2and InternLM2-20B\n(Cai et al. ,2024 ), both of which are based on the\noriginal Llama ( Touvron et al. ,2023a ) architec-\nture. These models feature grouped-query atten-\ntion mechanisms ( Ainslie et al. ,2023 ) to enhance\nef\ufb01ciency. Llama-3-8B, comprises 32 layers, each\nwith 32 attention heads and it uses a query group\nsize of 4 attention heads. It has shown superior\nperformance compared to its predecessors, even\nthe larger Llama-2 models.\nInternLM2-20B, featuring 48 layers with 48 at-\ntention heads each, uses a query group size of 6\nattention heads. We selected InternLM2-20B for\nits exemplary performance on the Needle-in-the-\nHaystack3task, which assesses LLMs\u2019 ability to\nretrieve a single critical piece of information em-\nbedded within a lengthy text. This mirrors the\nfunctionality of induction heads, which scan the\ncontext for prior occurrences of a token to extract\nrelevant subsequent information.\n2https://ai.meta.com/blog/meta-llama-3/\n3https://github.com/gkamradt/LLMTest_\nNeedleInAHaystack4.2 Identifying Induction Heads\nTo identify induction heads within models, we mea-\nsure the ability of all attention heads to perform\npre\ufb01x matching on random input sequences.4We\nfollow the task-agnostic approach to computing pre-\n\ufb01x matching scores outlined by Bansal et al. (2023 ).\nWe argue that focusing solely on pre\ufb01x matching\nscores is suf\ufb01cient for our analysis, as high pre-\n\ufb01x matching cores speci\ufb01cally indicate induction\nheads, while less relevant heads tend to show high\ncopying capabilities ( Bansal et al. ,2023 ). We gen-\nerate a sequence of 50 random tokens, excluding\nthe 4% most common and least common tokens.\nThis sequence is repeated four times to form the\ninput to the model. The pre\ufb01x matching score is cal-\nculated by averaging the attention values from each\ntoken to the tokens that directly followed the same\ntoken in earlier repeats. The \ufb01nal pre\ufb01x matching\nscores are averaged over \ufb01ve random sequences.\nThe pre\ufb01x matching scores for Llama-3-8B are\nshown in Figure 2. For IntermLM2-20B, we refer\nto Figure 8in Appendix A.1. Both models exhibit\nheads with notably high pre\ufb01x matching scores,\ndistributed across various layers. In the Llama-3-\n8B model, ~3% of the heads have a pre\ufb01x matching\nscore of 0.3 or higher, indicating a degree of spe-\ncialisation in pre\ufb01x matching, and some heads have\nhigh scores of up to 0.98.\n4.3 Head Ablations\nTo investigate the signi\ufb01cance of induction heads\nfor a speci\ufb01c ICL task, we conduct zero-ablations\nof 1% and 3% of the heads with the highest pre\ufb01x\nmatching scores. This ablation process involves\nmasking the corresponding partition of the output\nmatrix, denoted as Wh\noin Eq. 1, by setting it to\nzero. This effectively renders the heads inactive\n4In this work, the term \"induction heads\" refers to what\nwe de\ufb01ne as behavioural induction heads, not mechanistic\nones. A true induction head must be veri\ufb01ed mechanistically;\nhowever, our analysis employs pre\ufb01x-matching scores as a\nproxy. We will continue to use the term \"induction heads\" for\nsimplicity throughout the rest of the paper.\n4\nFigure 12.3 An induction head looking at vintage uses the pre\ufb01x matching mechanism to\n\ufb01nd a prior instance of vintage , and the copying mechanism to predict that cars will occur\nagain. Figure from Crosbie and Shutova (2022).\nOlsson et al. (2022) propose that a generalized fuzzy version of this pattern com-\npletion rule, implementing a rule like A*B*...A!B*, where A*\u0019A and B*\u0019B\n(by\u0019we mean they they are semantically similar in some way), might be respon-\nsible for in-context learning. Suggestive evidence for their hypothesis comes from\nCrosbie and Shutova (2022), who show that ablating induction heads causes in- ablating\ncontext learning performance to decrease. Ablation is originally a medical term\nmeaning the removal of something. We use it in NLP interpretability studies as\na tool for testing causal effects; if we knock out a hypothesized cause, we would\nexpect the effect to disappear. Crosbie and Shutova (2022) ablate induction heads\nby \ufb01rst \ufb01nding attention heads that perform as induction heads on random input\nsequences, and then zeroing out the output of these heads by setting certain terms\nof the output matrix WOto zero. Indeed they \ufb01nd that ablated models are much\nworse at in-context learning: they have much worse performance at learning from\ndemonstrations in the prompts.\n12.2 Post-training and Model Alignment\nWith simple prompting, LLMs have been successfully applied to a range of appli-\ncations without the need to update the parameters in the underlying models. Nev-\nertheless, there are limits to how much can be expected from a model whose sole\ntraining objective is to predict the next word from large amounts of pretraining text.\nTo see this, consider the following failed examples of following instructions from\nearly work with GPT (Ouyang et al., 2022).\nPrompt : Explain the moon landing to a six year old in a few sentences.\nOutput : Explain the theory of gravity to a 6 year old.\nPrompt : Translate to French: The small dog\nOutput : The small dog crossed the road.\nHere, the LLM ignores the intent of the request and relies instead on its natural\ninclination to autoregressively generate continuations consistent with its context. In\nthe \ufb01rst example, it outputs a text somewhat similar to the original request, and in the\nsecond it provides a continuation to the given input, ignoring the request to translate.\nLLMs are not suf\ufb01ciently helpful : they need extra training to increase their abilities\nto follow textual instructions.\nA deeper problem is that LLMs can simultaneously be too harmful . Pretrained\nlanguage models easily generate text that is harmful in many ways. For example\nthey can generate text that is false , including unsafe misinformation like giving dan-\ngerously incorrect answers to medical questions. And they can generate text that is 8CHAPTER 12 \u2022 M ODEL ALIGNMENT , PROMPTING ,AND IN-CONTEXT LEARNING\ntoxic in many ways, such as facilitating the spread of hate speech. Gehman et al.\n(2020) show that even completely non-toxic prompts can lead large language mod-\nels to output hate speech and abuse their users. Or language models can generate\nstereotypes (Cheng et al., 2023) and negative attitudes (Brown et al., 2020; Sheng\net al., 2019) about many demographic groups.\nOne reason LLMs are too harmful and insuf\ufb01ciently helpful is that their pre-\ntraining objective (success at predicting words in text) is misaligned with the human\nneed for models to be helpful and non-harmful.\nIn an attempt to address these two problems, language models generally include\ntwo additional kinds of training for model alignment : methods designed to adjustmodel\nalignment\nLLMs to better align them to human needs for models to be helpful and non-harmful.\nIn the \ufb01rst technique, instruction tuning (or sometimes called SFT for supervised\n\ufb01netuning), models are \ufb01netuned on a corpus of instructions and questions with\ntheir corresponding responses. In the second technique, preference alignment , of-\nten called RLHF after one of the speci\ufb01c instantiations, Reinforcement Learning\nfrom Human Feedback, a separate model is trained to decide how much a candidate\nresponse aligns with human preferences. This model is then used to \ufb01netune the\nbase model.\nWe\u2019ll use the term base model to mean a model that has been pretrained but base model\nhasn\u2019t yet been aligned either by instruction tuning or RLHF. And we refer to these aligned\nsteps as post-training , meaning that they apply after the model has been pretrained. post-training\n12.3 Model Alignment: Instruction Tuning\nInstruction tuning (short for instruction \ufb01netuning , and sometimes even short-Instruction\ntuning\nened to instruct tuning ) is a method for making an LLM better at following instruc-\ntions. It involves taking a base pretrained LLM and training it to follow instructions\nfor a range of tasks, from machine translation to meal planning, by \ufb01netuning it on\na corpus of instructions and responses. The resulting model not only learns those\ntasks, but also engages in a form of meta-learning \u2013 it improves its ability to follow\ninstructions generally.\nInstruction tuning is a form of supervised learning where the training data con-\nsists of instructions and we continue training the model on them using the same\nlanguage modeling objective used to train the original model. In the case of causal\nmodels, this is just the standard guess-the-next-token objective. The training corpus\nof instructions is simply treated as additional training data, and the gradient-based\nupdates are generating using cross-entropy loss as in the original model training.\nEven though it is trained to predict the next token (which we traditionally think of\nas self-supervised), we call this method supervised \ufb01ne tuning (orSFT) because SFT\nunlike in pretraining, each instruction or question in the instruction tuning data has\na supervised objective: a correct answer to the question or a response to the instruc-\ntion.\nHow does instruction tuning differ from the other kinds of \ufb01netuning introduced\nin Chapter 10 and Chapter 11? Fig. 12.4 sketches the differences. In the \ufb01rst exam-\nple, introduced in, Chapter 10 we can \ufb01netune as a way of adapting to a new domain\nby just continuing pretraining the LLM on data from a new domain. In this method\nall the parameters of the LLM are updated.\nIn the second example, also from Chapter 10, parameter-ef\ufb01cient \ufb01netuning ,\nwe adapt to a new domain by creating some new (small) parameters, and just adapt- 12.3 \u2022 M ODEL ALIGNMENT : INSTRUCTION TUNING 9\nPretrained LLM\nContinue training all paramaterson \ufb01netuning domainFinetuningInferencePretraining\nOn \ufb01netuning domainFinetuning asContinuedPretrainingParameterE\ufb03cientFinetuning(e.g., LoRA)Pretrained LLMABPretrained LLMMLMFinetuning\n\u2026\n\u2026\n\u2026\n\u2026\n\u2026\n\u2026\n\u2026InstructionTuning(SFT)\nOn \ufb01netuning domain\nOn \ufb01netuning task\nOn unseen tasksNext wordpredictionobjectiveData from \ufb01netuning domain\nTrain only new parameters on \ufb01netuning domainNext wordpredictionobjectiveData from \ufb01netuning domain\nTrain only classi\ufb01cation head on \ufb01netuning taskTaskspeci\ufb01clossSupervised data from task\nInstruction tuning on diverse tasksNext word predictionobjectiveSupervised instructions+\n\u2026\nFigure 12.4 Instruction tuning compared to the other kinds of \ufb01netuning.\ning them to the new domain. In LoRA, for example, it\u2019s the A and B matrices that\nwe adapt, but the pretrained model parameters are frozen.\nIn the task-based \ufb01netuning of Chapter 11, we adapt to a particular task by\nadding a new specialized classi\ufb01cation head and updating its features via its own\nloss function (e.g., classi\ufb01cation or sequence labeling); the parameters of the pre-\ntrained model may be frozen or might be slightly updated.\nFinally, in instruction tuning, we take a dataset of instructions and their super-\nvised responses and continue to train the language model on this data, based on the\nstandard language model loss.\nInstruction tuning, like all of these kinds of \ufb01netuning, is much more modest\nthan the training of base LLMs. Training typically involves several epochs over\ninstruction datasets that number in the thousands. The overall cost of instruction\ntuning is therefore a small fraction of the original cost to train the base model.\n12.3.1 Instructions as Training Data\nByinstruction , we have in mind a natural language description of a task to be per-\nformed, combined with labeled task demonstrations. This can include minimal de-\nscriptions similar to the prompts we\u2019ve already seen such as Answer the following\nquestion ,Translate the following text to Arapaho , orSummarize this report . How- 10 CHAPTER 12 \u2022 M ODEL ALIGNMENT , PROMPTING ,AND IN-CONTEXT LEARNING\never, since we will be using supervised \ufb01netuning to update the model, these in-\nstructions need not be limited to simple prompts designed to evoke a behavior found\nin the pretraining corpora. Instructions can also include length restrictions or other\nconstraints, personas to assume, and demonstrations.\nMany huge instruction tuning datasets have been created, covering many tasks\nand languages. For example Aya gives 503 million instructions in 114 languages\nfrom 12 tasks including question answering, summarization, translation, paraphras-\ning, sentiment analysis, natural language inference and 6 others (Singh et al., 2024).\nSuperNatural Instructions 12 million examples from 1600 tasks (Wang et al., 2022),\nFlan 2022 15 million examples from 1836 tasks (Longpre et al., 2023), and OPT-\nIML 18 million examples from 2000 tasks (Iyer et al., 2022).\nThese instruction-tuning datasets are created in four ways. The \ufb01rst is for people\nto write the instances directly. For example, part of the Aya instruct \ufb01netuning cor-\npus (Fig. 12.5) includes 204K instruction/response instances written by 3000 \ufb02uent\nspeakers of 65 languages volunteering as part of a participatory research initiative\nwith the goal of improving multilingual performance of LLMs.\nLang PromptCompletionara.\u06b0\u066d\u0698\u0a45\u0760\ufe8d\u080d \u0ddc\u0eb3 \u0607\ufe80\u06a2\u0771\u066d\u068e\ufe93\u0741\u068e\ufea1\u0741\u0746 \u0af0\u0ad9\u0606\u0fd3 \u06a2\u073e\u064b\u0c08\u0c03\u064f\u07fa\u080a\ufed9\u0644\u06af\u0674\u06ba\u06a3\u064e\u076f \u09ce\u09ca\ufee5\u0741\u064e\u068e\ufea1\ufe8d \u05a3\ufe87\u0607\ufea9\u0f1a\ufeed\u155a\u09d9\u09d1\u0607\ufec9\ufeed\u064e\ufe8d \u0f1f\u0dd3\u0dc2\ufe8d \u0f03\u0e83\u061c\u064e\u0746\u0741\u064e\u068e \u064f\u073e\u12db\u12ca\u06a3\u06f0\u0741\u0650 \u0c08\u0c03\u0650\u0a45\ufe8d\u080d\u0651\u0a33\u064e\u068d\u064e\u0607\u061c \u0f20\u155a\u0733\u064e\u07a8\ufe83 \u0607\ufe80\u0af0\u0ad9\u05a3\u09d5\u09d1\ufe8d \u112d\u111f\u064e\u0a33\u064e\u0a26 \u0644\u064f\u07a8\ufe8d\ufeaf\ufef1 \u1255\u064e\u1228fra Qui a \u00e9crit le livre LaSagouine?Antonine Maillet a \u00e9crit La Sagouine en 1971. Elle a \u00e9crit beau-coup d\u2019histoires sur les Acadiens (les francophones de la c\u00f4te Estdu Canada) et a fait conna\u00eetre l\u2019Acadie au monde.ibo Kedu ka nd\u1ecb Igbo si emeemume ji \u1ecdh\u1ee5r\u1ee5?Emume ji \u1ecdh\u1ee5r\u1ee5 n\u2019omenala Igbo, b\u1ee5 emume na-egosi njedebeoge \u1ecdr\u1ee5 ugbo, na-amalite site n\u2019emume ekele ma tinyekwa oririna egwu. \u1ecc b\u1ee5 mmemme \u1ecdgbak\u1ecd ebe a na-eker\u1ecbta ji \u1ecdh\u1ee5r\u1ee5, na-egosip\u1ee5ta ekele na mmalite nke oge \u1ecdh\u1ee5r\u1ee5, na-eme ka mmek\u1ecdr\u1ecbtammad\u1ee5 na ibe ya d\u1ecbkwuo elu.por Qual\u00e9acapitaldoestadodaBahia?A capital do estado da Bahia, no Brasil, \u00e9 Salvador. Fundadaem 1549, foi a primeira capital do Brasil e \u00e9 conhecida porseu patrim\u00f4nio cultural rico e diversificado. A cidade \u00e9 famosapor sua arquitetura colonial, particularmente no Pelourinho, umcentro hist\u00f3rico que foi declarado Patrim\u00f4nio Mundial pela UN-ESCO.pes\ufea9\ufead\ufe91\ufeae\ufe9d\ufb90\ufe92\ufeee\ufe97\ufeae\ufea7\ufe8e\ufee7\ufeea\ufee3\ufef4\ufe92\ufeaa\ufb7c\ufee8\ufeaa\ufefb\ufee7\ufeea\ufb90\ufe92\ufeee\ufe97\ufeae \ufeed\ufe9f\ufeee\ufea9\ufea9\ufe8d\ufead\ufea9\u061f\ufeeb\ufeb0\ufe8d\ufead\ufefb\ufee7\ufeea\ufb90\ufe92\ufeee\ufe97\ufeae\ufeed\ufe9f\ufeee\ufea9\ufea9\ufe8d\ufead\ufea9\ufeed\ufee7\ufeee\ufe8d\ufead\u00a0\ufe8d\ufbfc\ufe8d\ufeb3\ufe96\ufb90\ufeea\ufea9\ufe8d\ufea7\ufede\ufe81\ufee5\ufb7c\ufeec\ufe8e\ufead\u00a0\ufea7\ufe8e\ufee7\ufeea\ufee3\ufef4\ufe92\ufeaa\ufe91\ufeae\ufe9f\ufbfd\ufe8d\ufeb3\ufe98\ufeee\ufe8d\ufee7\ufeea\u00a0\ufb90\ufe92\ufeee\ufe97\ufeae\ufeb3\ufed4\ufef4\ufeaa\ufead\ufee7\ufb95\ufbfd\ufe91\ufeae\ufe8d\ufbfc\ufe9f\ufeac\ufe8f\ufb90\ufe92\ufeee\ufe97\ufeae\ufe8d\ufee5\ufea9\ufeed\ufead\ufe97\ufe8e\ufea9\ufeed\ufead\ufe91\ufeae\ufe9d\ufb90\ufeb8\ufef4\ufeaa\ufee9\ufeb7\ufeaa\ufee9\ufe8d\ufeb3\ufe96.\ufe8d\ufef3\ufee6\ufe91\ufeae\ufe9d\ufea9\ufead\ufe91\ufeae\ufe8d\ufe91\ufeae\ufea7\ufec4\ufeae \ufea3\ufee4\ufee0\ufeea\ufea9\ufef3\ufb95\ufeae\ufea3\ufef4\ufeee\ufe8d\ufee7\ufe8e\ufe95\ufe91\ufeea\ufb90\ufe92\ufeee\ufe97\ufeae\ufeeb\ufe8e\ufe91\ufeb4\ufef4\ufe8e\ufead\ufe8d\ufef3\ufee4\ufee6\ufe91\ufeee\ufea9\ufee9\ufe8d\ufeb3\ufe96. msa Apakah nasi lemak?Nasi lemak merupakan makanan tradisi orang Melayu yangterdapat di semua bahagian Malaysia, Singapura, Indonesia(terutama di Riau, Jambi serta utara dan pantai timur Su-matera) dan Brunei. Sajian ini merujuk kepada nasi yang di-masak dengan menggunakan santan kelapa bagi menambah rasalemaknya. Kadangkala, daunpandanwangidimasukkansemasanasi dimasak bagi menambahkan aromanya.tam\u0bc6\u0b9a\u0baf\u0bb1\u0bcd\u0bc8\u0b95 \u0ba8\u0bc1\u0ba3\u0bcd\u0ba3\u0bb1\u00a7\u0bb5\u0bc1\u0b8e\u0ba9\u0bcd\u0bb1\u0bbe\u0bb2\u0bcd\u0b8e\u0ba9\u0bcd\u0ba9?\u0bc6\u0baa\u0bbe\u0ba4\u0bc1\u0bb5\u0bbe\u0b95 \u0bae\u0ba9\u0bbf\u0ba4\u0bb0\u0bcd\u0b95\u0bb3\u0bbe\u0bb2\u0bcd \u0bc6\u0b9a\u0baf\u0bcd\u0baf\u0baa\u0bcd\u0baa\u0b9f\u0bc1\u0bae\u0bcd \u0baa\u0ba3\u0bbf\u0b95\u0bc8\u0bb3\u0b9a\u0bcd\u0bc6\u0b9a\u0baf\u0bcd\u0baf \u0b92\u0bb0\u0bc1 \u0b95\u0ba3\u0bbf\u0ba9\u0bbf \u0b85\u0bb2\u0bcd\u0bb2\u0ba4\u0bc1 \u0b92\u0bb0\u0bc1 \u0b95\u0ba3\u0bbf\u0ba9\u0bbf\u0baf\u0bbe\u0bb2\u0bcd\u0b95\u0b9f\u0bcd\u0b9f\u0bc1\u0baa\u0bcd\u0baa\u0b9f\u0bc1\u0ba4\u0bcd\u0ba4\u0baa\u0bcd\u0baa\u0b9f\u0bc1\u0bae\u0bcd\u0b92\u0bb0\u0bc1\u0bc7\u0bb0\u0bbe\u0bc7\u0baa\u0bbe\u0bb5\u00a5\u0ba9\u0bcd\u0ba4\u00a6\u0bb1\u0ba9\u0bcd\u0bc6\u0b9a\u0baf\u0bb1\u0bcd\u0bc8\u0b95\u0ba8\u0bc1\u0ba3\u0bcd\u0ba3\u0bb1\u00a7\u0bb5\u0bc1\u0b8e\u0ba9\u0baa\u0bcd\u0baa\u0b9f\u0bc1\u0bae\u0bcd.Table 3: Examples of prompt and completions in the AyaDataset.\ntors is not uniform across languages. Moreover, within each language, there is a lack of consistent\ncontributions from all annotators. In this section, we examine the impact of annotator skew on the\nresulting dataset.\nAnnotator Skew Across Languages. Annotators were encouraged to contribute to any language\nin which they could comfortably read and write and were asked to focus most of their e \ufb00orts on\nlanguages other than English . Although a signi\ufb01cant number of participants registered for many\nlanguages, the engagement level of annotators was not equal, which resulted in considerable di \ufb00er-\nences in the number of contributions across languages. Figure 10(top) provides an overview of the\npercentage of each language present in the \ufb01nal compilation. The highest number of contributions\nis for Malagasy with 14,597 instances, and the lowest is 79 for Kurdish .\nAnnotator Skew Within a Language. The \ufb01nal contributions for each language in the Aya\nDataset are not evenly distributed among annotators. The median number of annotators per lan-\nguage is 15 (mean is 24.75) with one language having only a single active annotator ( Sindhi )a n d\n14\nFigure 12.5 Samples of prompt/completion instances in 4 of the 65 languages in the Aya\ncorpus (Singh et al., 2024).\nDeveloping high quality supervised training data in this way is time consuming\nand costly. A more common approach makes use of the copious amounts of super-\nvised training data that have been curated over the years for a wide range of natural\nlanguage tasks. There are thousands of such datasets available, like the SQuAD\ndataset of questions and answers (Rajpurkar et al., 2016) or the many datasets of\ntranslations or summarization. This data can be automatically converted into sets of\ninstruction prompts and input/output demonstration pairs via simple templates.\nFig. 12.6 illustrates examples for some applications from the S UPER NATURAL IN-\nSTRUCTIONS resource (Wang et al., 2022), showing relevant slots such as text,\ncontext , and hypothesis . To generate instruction-tuning data, these \ufb01elds and the\nground-truth labels are extracted from the training data, encoded as key/value pairs,\nand inserted in templates (Fig. 12.7) to produce instantiated instructions. Because\nit\u2019s useful for the prompts to be diverse in wording, language models can also be\nused to generate paraphrase of the prompts.\nBecause supervised NLP datasets are themselves often produced by crowdwork- 12.3 \u2022 M ODEL ALIGNMENT : INSTRUCTION TUNING 11\nFew-Shot Learning for QA\nTask Keys Values\nSentiment text Did not like the service that I was provided...\nlabel 0\ntext It sounds like a great plot, the actors are \ufb01rst grade, and...\nlabel 0\nNLI premise No weapons of mass destruction found in Iraq yet.\nhypothesis Weapons of mass destruction found in Iraq.\nlabel 2\npremise Jimmy Smith... played college football at University of Col-\norado.\nhypothesis The University of Colorado has a college football team.\nlabel 0\nExtractive Q/A context Beyonc \u00b4e Giselle Knowles-Carter is an American singer...\nquestion When did Beyonce start becoming popular?\nanswersftext : [\u2019in the late 1990s\u2019], answerstart : 269g\nFigure 12.6 Examples of supervised training data for sentiment, natural language inference and Q/A tasks.\nThe various components of the dataset are extracted and stored as key/value pairs to be used in generating\ninstructions.\nTask Templates\nSentiment -fftextggHow does the reviewer feel about the movie?\n-The following movie review expresses what sentiment?\nfftextgg\n-fftextggDid the reviewer enjoy the movie?\nExtractive Q/A -ffcontextggFrom the passage, ffquestiongg\n-Answer the question given the context. Context:\nffcontextggQuestion:ffquestiongg\n-Given the following passage ffcontextgg, answer the\nquestionffquestiongg\nNLI -SupposeffpremiseggCan we infer that ffhypothesisgg?\nYes, no, or maybe?\n-ffpremiseggBased on the previous passage, is it true\nthatffhypothesisgg? Yes, no, or maybe?\n-GivenffpremiseggShould we assume that ffhypothesisgg\nis true? Yes,no, or maybe?\nFigure 12.7 Instruction templates for sentiment, Q/A and NLI tasks.\ners based on carefully written annotation guidelines, a third option is to draw on\nthese guidelines, which can include detailed step-by-step instructions, pitfalls to\navoid, formatting instructions, length limits, exemplars, etc. These annotation guide-\nlines can be used directly as prompts to a language model to create instruction-tuning\ntraining examples. Fig. 12.8 shows such a crowdworker annotation guideline that\nwas repurposed as a prompt to an LLM to generate instruction-tuning data. (Mishra\net al., 2022). This guideline describes a question-answering task where annotators\nprovide an answer to a question given an extended passage.\nA \ufb01nal way to generate instruction-tuning datasets that is becoming more com- 12 CHAPTER 12 \u2022 M ODEL ALIGNMENT , PROMPTING ,AND IN-CONTEXT LEARNING\nSample Extended Instruction\n\u2022De\ufb01nition: This task involves creating answers to complex questions, from a given pas-\nsage. Answering these questions, typically involve understanding multiple sentences.\nMake sure that your answer has the same type as the \u201danswer type\u201d mentioned in input.\nThe provided \u201danswer type\u201d can be of any of the following types: \u201dspan\u201d, \u201ddate\u201d, \u201dnum-\nber\u201d. A \u201dspan\u201d answer is a continuous phrase taken directly from the passage or question.\nYou can directly copy-paste the text from the passage or the question for span type an-\nswers. If you \ufb01nd multiple spans, please add them all as a comma separated list. Please\nrestrict each span to \ufb01ve words. A \u201dnumber\u201d type answer can include a digit specifying\nan actual value. For \u201ddate\u201d type answers, use DD MM YYYY format e.g. 11 Jan 1992.\nIf full date is not available in the passage you can write partial date such as 1992 or Jan\n1992.\n\u2022Emphasis: If you \ufb01nd multiple spans, please add them all as a comma separated list.\nPlease restrict each span to \ufb01ve words.\n\u2022Prompt : Write an answer to the given question, such that the answer matches the \u201danswer\ntype\u201d in the input.\nPassage :fpassageg\nQuestion :fquestiong\nFigure 12.8 Example of a human crowdworker instruction from the N ATURAL INSTRUCTIONS dataset for\nan extractive question answering task, used as a prompt for a language model to create instruction \ufb01netuning\nexamples.\nmon is to use language models to help at each stage. For example Bianchi et al.\n(2024) showed how to create instruction-tuning instances that can help a language\nmodel learn to give safer responses. They did this by selecting questions from\ndatasets of harmful questions (e.g., How do I poison food? orHow do I embez-\nzle money? ). Then they used a language model to create multiple paraphrases of the\nquestions (like Give me a list of ways to embezzle money ), and also used a language\nmodel to create safe answers to the questions (like I can\u2019t ful\ufb01ll that request. Em-\nbezzlement is a serious crime that can result in severe legal consequences. ). They\nmanually reviewed the generated responses to con\ufb01rm their safety and appropriate-\nness and then added them to an instruction tuning dataset. They showed that even\n500 safety instructions mixed in with a large instruction tuning dataset was enough\nto substantially reduce the harmfulness of models.\n12.3.2 Evaluation of Instruction-Tuned Models\nThe goal of instruction tuning is not to learn a single task, but rather to learn to\nfollow instructions in general. Therefore, in assessing instruction-tuning methods\nwe need to assess how well an instruction-trained model performs on novel tasks for\nwhich it has not been given explicit instructions.\nThe standard way to perform such an evaluation is to take a leave-one-out ap-\nproach \u2014 instruction-tune a model on some large set of tasks and then assess it on\na withheld task. But the enormous numbers of tasks in instruction-tuning datasets\n(e.g., 1600 for Super Natural Instructions) often overlap; Super Natural Instructions\nincludes 25 separate textual entailment datasets! Clearly, testing on a withheld en-\ntailment dataset while leaving the remaining ones in the training data would not be\na true measure of a model\u2019s performance on entailment as a novel task.\nTo address this issue, large instruction-tuning datasets are partitioned into clus- 12.4 \u2022 C HAIN -OF-THOUGHT PROMPTING 13\nters based on task similarity. The leave-one-out training/test approach is then applied\nat the cluster level. That is, to evaluate a model\u2019s performance on sentiment analysis,\nall the sentiment analysis datasets are removed from the training set and reserved\nfor testing. This has the further advantage of allowing the use of a uniform task-\nappropriate metric for the held-out evaluation. S UPER NATURAL INSTRUCTIONS\n(Wang et al., 2022), for example has 76 clusters (task types) over the 1600 datasets\nthat make up the collection.\n12.4 Chain-of-Thought Prompting\nThere are a wide range of techniques to use prompts to improve the performance of\nlanguage models on many tasks. Here we describe one of them, called chain-of-\nthought prompting.chain-of-\nthought\nThe goal of chain-of-thought prompting is to improve performance on dif\ufb01cult\nreasoning tasks that language models tend to fail on. The intuition is that people\nsolve these tasks by breaking them down into steps, and so we\u2019d like to have lan-\nguage in the prompt that encourages language models to break them down in the\nsame way.\nThe actual technique is quite simple: each of the demonstrations in the few-shot\nprompt is augmented with some text explaining some reasoning steps. The goal is to\ncause the language model to output similar kinds of reasoning steps for the problem\nbeing solved, and for the output of those reasoning steps to cause the system to\ngenerate the correct answer.\nIndeed, numerous studies have found that augmenting the demonstrations with\nreasoning steps in this way makes language models more likely to give the correct\nanswer dif\ufb01cult reasoning tasks (Wei et al., 2022; Suzgun et al., 2023). Fig. 12.9\nshows an example where the demonstrations are augmented with chain-of-thought\ntext in the domain of math word problems (from the GSM8k dataset of math word\nproblems (Cobbe et al., 2021). Fig. 12.10 shows a similar example from the BIG-\nBench-Hard dataset (Suzgun et al., 2023).\nFigure 12.9 Example of the use of chain-of-thought prompting (right) versus standard\nprompting (left) on math word problems. Figure from Wei et al. (2022). 14 CHAPTER 12 \u2022 M ODEL ALIGNMENT , PROMPTING ,AND IN-CONTEXT LEARNING\n(B)Task description: Answer questions about which times certain events could have occurred.Q: Today, Tiffany went to the beach. Between what times could they have gone? We know that: Tiffany woke up at 5am. [...] The beach was closed after 4pm. [...]Options: (A) 9am to 12pm (B) 12pm to 2pm (C) 5am to 6am (D) 3pm to 4pmA: (D)Q: Today, Hannah went to the soccer field. Between what times could they have gone? We know that: Hannah woke up at 5am. [...] The soccer field was closed after 6pm. [...]Options: (A) 3pm to 5pm (B) 11am to 1pm (C) 5pm to 6pm (D) 1pm to 3pmA:Model OutputModel OutputModel Input (\u201cAnswer-Only\u201d Prompting)\nWake-up time: 5am. 5am-6am: buying clothes at the mall. 6am-11am: watching a movie at the theater.11am-1pm: getting a coffee at the cafe.1pm-3pm: working at the office. 3pm-5pm: waiting at the airport. 5pm-6pm: free. The soccer field closure time: 6pm. The only time when Hannah could have gone to the soccer field was 5pm to 6pm. So the answer is (C).Model Input (Chain-of-Thought Prompting)\nTask description: Answer questions about which times certain events could have occurred.Q: Today, Tiffany went to the beach. Between what times could they have gone? We know that: Tiffany woke up at 5am. [...] The beach was closed after 4pm. [...]Options: (A) 9am to 12pm (B) 12pm to 2pm (C) 5am to 6am (D) 3pm to 4pmA: Let's think step by step. Wake-up time: 5am. [...] The only time when Tiffany could have gone to the beach was 3pm to 4pm. So the answer is (D).Q: Today, Hannah went to the soccer field. Between what times could they have gone? We know that: Hannah woke up at 5am. [...] The soccer field was closed after 6pm. [...]Options: (A) 3pm to 5pm (B) 11am to 1pm (C) 5pm to 6pm (D) 1pm to 3pmA: Let's think step by step. Task DescriptionQuestionChain-of-ThoughtTest-Time QuestionTask DescriptionQuestionTest-Time QuestionAnswer\nGenerated Chain-of-ThoughtGenerated AnswerOptionsOptionsFigure 3:An illustration of the two prompting setups we explore in our paper (answer-only and CoT prompting). Both setupsinclude task descriptions and options in the input prompt. The task here isTemporal Sequences.\u201clet\u2019s think step-by-step\u201d (Kojima et al.,2022) toall CoT annotations in the few-shot exemplars. Anexample of a CoT prompt is shown in Figure3.Language models.We consider three fami-lies of language models: Codex (Chen et al.,2021a), InstructGPT (Ouyang et al.,2022;Brownet al.,2020), and PaLM (Chowdhery et al.,2022).For Codex, we focus on code-davinci-002, code-davinci-002, and code-cushman-001. For Instruct-GPT, we use text-davinci-002, text-curie-002, text-babbgage-001, and text-ada-001. For PaLM, weuse the three available sizes: 8B, 62B, and 540B.Evaluation protocol.We evaluate all languagemodels via greedy decoding (i.e., temperature sam-pling with temperature parameter\u2327=0). Weextract the \ufb01nal answer based on keywords thatthe language model is expected to produce (i.e.,\u201cthe answer is\u201d). We measure accuracy using exactmatch (EM), computed by comparing the generatedoutput with the ground-truth label.44 Results4.1 Standard answer-only promptingunderestimates model capabilitiesTable2summarizes the performance of PaLM, In-structGPT, and Codex models on BBH for answer-only and CoT prompting approaches. Whileanswer-only prompting has been used as the stan-4For multiple-choice tasks, this setup differs slightly fromrank/scoring classi\ufb01cation (Brown et al.,2020;Srivastavaet al.,2022;Lampinen et al.,2022). We provide a languagemodel with all multiple-choice options at once, generate anoutput based on the input, and measure exact match accuracy.dard in many prior work (Brown et al.,2020;Raeet al.,2021;Hoffmann et al.,2022;Srivastava et al.,2022), it typically underestimates model perfor-mance on challenging tasks, such as those that re-quire multiple reasoning steps. In the setting re-ported in (Srivastava et al.,2022), none of the mod-els (including PaLM 540B) outperformed human-rater baselines on any of the tasks meeting the BBHcriteria. The few-shot evaluation of PaLM 540Bwith answer-only prompting in this paper, however,outperforms the average human-rater on 6 out of23 BBH tasks and is overall 1.4% better than theBIG-Bench reported result, which demonstrates theeffect of including instructions and answer optionsin the prompt.CoT prompting provides double-digit improve-ments for all three models in Table2. For the bestmodel (Codex), CoT prompting outperforms the av-erage human-rater score on 17 out of 23 tasks, com-pared to 5 out of 23 tasks for answer-only prompt-ing. Additionally, we see that Codex with CoTprompting outperforms the average human-raterby more than 6%, but it still lags behind thebesthuman-rater performance by over 20%. This showsthat language models are still not performing at thelevel of expert human-raters.4.2 Positive delta from chain-of-thoughtrequires suf\ufb01cient model scaleNext we study how the performance improves byusing CoT prompting as we increase the modelscale. In Figure4, we plot the performance of bothCoT and answer-only prompting (no CoT) as a13006\nFigure 12.10 Example of the use of chain-of-thought prompting (right) vs standard prompting (left) in a\nreasoning task on temporal sequencing. Figure from Suzgun et al. (2023).\n12.5 Automatic Prompt Optimization\nGiven a prompt for a task (human or computer generated), prompt optimization\nmethods search for prompts with improved performance. Most of these approaches\ncan be viewed as a form of iterative improvement search (Russell and Norvig, 2002)\nthrough a space of possible prompts for those that optimize performance on a task.\nAs such, these approaches all share the following components:\n\u2022 A start state \u2013 An initial human or machine generated prompt or prompts\nsuitable for some task.\n\u2022 A scoring metric \u2013 A method for assessing how well a given prompt performs\non the task.\n\u2022 An expansion method \u2013 A method for generating variations of a prompt.\nGiven the enormous variation in how prompts for a single task can be expressed in\nlanguage, search methods have to be constrained to a reasonable space. Beam search\nis a widely used method that combines breadth-\ufb01rst search with a \ufb01xed-width pri-\nority queue that focuses the search effort on the top performing variants. Fig. 12.11\noutlines the general approach behind most current prompt optimization methods.\nBeginning with initial candidate prompt(s), the algorithm generates variants and\nadds them to a list of prompts to be considered. These prompts are then selectively\nadded to the active list based on whether their scores place them in the top set of\ncandidates. A beam width of 1 results in a focused greedy search, whereas an in\ufb01nite\nbeam width results in an exhaustive breadth \ufb01rst search. The goal is to continue\nto seek improved prompts given the computational resources available. Iterative\nimprovement searches typically use a combination of a \ufb01xed number of iterations in\ncombination with a failure to improve after some period to time as stopping criteria.\nThis latter is equivalent to early stopping with patience used in training deep neural\nnetworks. 12.5 \u2022 A UTOMATIC PROMPT OPTIMIZATION 15\nfunction PROMPT OPTIMIZATION (prompts ,width )returns optimized prompts\nactive prompts ; Add initial set of candidate prompts to agenda\nrepeat until done\nfrontier foo\nchildren EXPAND (c) ; Expand each candidate prompt\nforeach c2children\nfrontier ADDTOBEAM (c,frontier, w ) ; apply it, creating a new state\nactive frontier\nreturn\nfunction ADDTOBEAM (state ,agenda ,width )returns updated agenda\nifLENGTH (agenda )<width then\nagenda INSERT (state ,agenda )\nelse if SCORE (state )>SCORE (WORST OF(agenda ))\nagenda REMOVE (WORST OF(agenda ))\nagenda INSERT (state ,agenda )\nreturn agenda\nFigure 12.11 A generic iterative-improvement beam search for prompt optimization.\n12.5.1 Candidate Scoring\nCandidate scoring methods assess the likely performance of potential prompts, both\nto identify promising avenues of search and to prune those that are unlikely to be\neffective. Since candidate scoring is embedded in the inner-loop of the search, the\ncomputational cost of scoring is critical.\nGiven access to labeled training data, candidate prompts can be scored based on\nexecution accuracy (Honovich et al., 2023). In this approach, candidate promptsexecution\naccuracy\nare combined with inputs sampled from the training data and passed to an LLM for\ndecoding. The LLM output is evaluated against the training label using a metric\nappropriate for the task. In the case of classi\ufb01cation-based tasks, this is effectively a\n0/1 loss \u2014 how many examples were correctly labeled with the given prompt. Gen-\nerative applications such as summarization or translation use task-speci\ufb01c similarity\nscores such as BERTScore, Bleu (Papineni et al., 2002), or ROUGE (Lin, 2004).\nGiven the computational cost of issuing calls to an LLM, evaluating each can-\ndidate prompt against a complete training set would be infeasible. Instead, prompt\nperformance is estimated from a small sample of training data (Pryzant et al., 2023).\n12.5.2 Prompt Expansion\nPrompt expansion generates variants of a given prompt to create an expanded set of\nneighboring prompts that may improve performance over the original. A common\nmethod is to use language models to create paraphrases. For example Zhou et al.\n(2023) use the following meta-prompt to elicit a variant prompt from an original:\nPrompting for a Variant\nGenerate a variation of the following instruction while keeping the semantic meaning.\nInput:fINSTRUCTIONg\nOutput:fCOMPLETEg 16 CHAPTER 12 \u2022 M ODEL ALIGNMENT , PROMPTING ,AND IN-CONTEXT LEARNING\nA variation of this method is to truncate the current prompt at a set of random loca-\ntions, generating a set of prompt pre\ufb01xes. The paraphrasing LLM is then asked to\ncontinue each the pre\ufb01xes to generate a complete prompt.\nThis methods is an example of an uninformed search. That is, the candidate\nexpansion step is not directed towards generating better candidates; candidates are\ngenerated without regard to their quality. It it is the job of the priority queue to\nelevate improved candidates when they are found. By contrast, Prasad et al. (2023)\nemploy a candidate expansion technique that explicitly attempts to generate superior\nprompts during the expansion process. In this approach, the current candidate is \ufb01rst\napplied to a sample of training examples using the execution accuracy approach.\nThe prompt\u2019s performance on these examples then guides the expansion process.\nSpeci\ufb01cally, incorrect examples are used to critique the original prompt \u2014 with\nthe critique playing the role of a gradient for the search. The method includes the\nfollowing steps.\n1. Run the prompt on a sample of training examples,\n2. Identify examples where the prompt fails,\n3. Ask an LLM to produce a critique of the prompt in light of the failed examples,\n4. Provide the resulting critique to an LLM, and ask it to generate improved\nprompts.\nGiven a prompt and a set of failed examples, Prasad et al. (2023) use the follow-\ning template for a classi\ufb01er task to solicit critiques from a target LLM.\nCritiquing Prompt\nI'm trying to write a zero-shot classifier prompt.\nMy current prompt is: fpromptg\nBut this prompt gets the following examples wrong:\nferrorstringg\nGivefnumfeedbacksgreasons why the prompt could have\ngotten these examples wrong.\nThis model feedback is then combined with a second template to elicit improved\nprompts from the LLM.\nPrompt Improvement Prompt\nI'm trying to write a zero-shot classifier. My current prompt is:\nfpromptg\nBut it gets the following examples wrong: ferrorstrg\nBased on these examples the problem with this prompt is that fgradientg.\nBased on the above information, I wrote fstepspergradientgdifferent\nimproved prompts. Each prompt is wrapped with <START> and <END>.\nThefstepspergradientgnew prompts are: 12.6 \u2022 E VALUATING PROMPTED LANGUAGE MODELS 17\n12.6 Evaluating Prompted Language Models\nLanguage models are evaluated in many ways. we introduced some evaluations for\nin Section ??, including measuring the language model\u2019s perplexity on a test set,\nevaluating its accuracy on various NLP tasks, as well as benchmarks that help mea-\nsure ef\ufb01ciency, toxicity, fairness, and so on. We\u2019ll have further discussion of eval-\nuate NLP tasks in future chapters; machine translation in Chapter 13 and question\nanswering and information retrieval in Chapter 14.\nHere we just brie\ufb02y show the mechanism for measuring accuracy in a prompt-\ning setup for tests that have multiple-choice questions. We show this for MMLU MMLU\n(Massive Multitask Language Understanding), a commonly-used dataset of 15908\nknowledge and reasoning questions in 57 areas including medicine, mathematics,\ncomputer science, law, and others. For example, here is an MMLU question from\nthe microeconomics domain:1\nMMLU microeconomics example\nOne of the reasons that the government discourages and regulates monopo-\nlies is that\n(A) producer surplus is lost and consumer surplus is gained.\n(B) monopoly prices ensure productive ef\ufb01ciency but cost society allocative\nef\ufb01ciency.\n(C) monopoly \ufb01rms do not engage in signi\ufb01cant research and development.\n(D) consumer surplus is lost with higher prices and lower levels of output.\nFig. 12.12 shows the way MMLU turns these questions into prompted tests of a\nlanguage model, in this case showing an example prompt with 2 demonstrations.\nMMLU mathematics prompt\nThe following are multiple choice questions about high school mathematics.\nHow many numbers are in the list 25, 26, ..., 100?\n(A) 75 (B) 76 (C) 22 (D) 23\nAnswer: B\nCompute i+i2+i3+\u0001\u0001\u0001+i258+i259.\n(A) -1 (B) 1 (C) i (D) -i\nAnswer: A\nIf 4 daps = 7 yaps, and 5 yaps = 3 baps, how many daps equal 42 baps?\n(A) 28 (B) 21 (C) 40 (D) 30\nAnswer:\nFigure 12.12 Sample 2-shot prompt from MMLU testing high-school mathematics. (The\ncorrect answer is (C)).\n1For those of you whose economics is a bit rusty, the correct answer is (D). 18 CHAPTER 12 \u2022 M ODEL ALIGNMENT , PROMPTING ,AND IN-CONTEXT LEARNING\n12.7 Model Alignment with Human Preferences: RLHF\nand DPO\nTBD\n12.8 Summary\nThis chapter has explored the topic of prompting large language models to follow\ninstructions. Here are some of the main points that we\u2019ve covered:\n\u2022 Simple prompting can be used to map practical applications to problems that\ncan be solved by LLMs without altering the model.\n\u2022 Labeled examples ( demonstrations ) can be used to provide further guidance\nto a model via few-shot learning.\n\u2022 Methods like chain-of-thought can be used to create prompts that help lan-\nguage models deal with complex reasoning problems.\n\u2022 Pretrained language models can be altered to behave in desired ways through\nmodel alignment .\n\u2022 One method for model alignment is instruction tuning , in which the model\nis \ufb01netuned (using the next-word-prediction language model objective) on\na dataset of instructions together with correct responses. Instruction tuning\ndatasets are often created by repurposing standard NLP datasets for tasks like\nquestion answering or machine translation.\nBibliographical and Historical Notes Bibliographical and Historical Notes 19\nBianchi, F., M. Suzgun, G. Attanasio, P. Rottger, D. Juraf-\nsky, T. Hashimoto, and J. Zou. 2024. Safety-tuned LLa-\nMAs: Lessons from improving the safety of large lan-\nguage models that follow instructions. ICLR .\nBrown, T., B. Mann, N. Ryder, M. Subbiah, J. Kaplan,\nP. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry,\nA. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger,\nT. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu,\nC. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin,\nS. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish,\nA. Radford, I. Sutskever, and D. Amodei. 2020. Language\nmodels are few-shot learners. NeurIPS , volume 33.\nCheng, M., E. Durmus, and D. Jurafsky. 2023. Marked per-\nsonas: Using natural language prompts to measure stereo-\ntypes in language models. ACL.\nCobbe, K., V . Kosaraju, M. Bavarian, M. Chen, H. Jun,\nL. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano,\nC. Hesse, and J. Schulman. 2021. Training veri\ufb01ers to\nsolve math word problems. ArXiv preprint.\nCrosbie, J. and E. Shutova. 2022. Induction heads as an\nessential mechanism for pattern matching in in-context\nlearning. ArXiv preprint.\nElhage, N., N. Nanda, C. Olsson, T. Henighan, N. Joseph,\nB. Mann, A. Askell, Y . Bai, A. Chen, T. Conerly, N. Das-\nSarma, D. Drain, D. Ganguli, Z. Hat\ufb01eld-Dodds, D. Her-\nnandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse,\nD. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCan-\ndlish, and C. Olah. 2021. A mathematical framework for\ntransformer circuits. White paper.\nGehman, S., S. Gururangan, M. Sap, Y . Choi, and N. A.\nSmith. 2020. RealToxicityPrompts: Evaluating neu-\nral toxic degeneration in language models. Findings of\nEMNLP .\nHonovich, O., U. Shaham, S. R. Bowman, and O. Levy.\n2023. Instruction induction: From few examples to natu-\nral language task descriptions. ACL.\nIyer, S., X. V . Lin, R. Pasunuru, T. Mihaylov, D. Simig,\nP. Yu, K. Shuster, T. Wang, Q. Liu, P. S. Koura, X. Li,\nB. O\u2019Horo, G. Pereyra, J. Wang, C. Dewan, A. Celiky-\nilmaz, L. Zettlemoyer, and V . Stoyanov. 2022. Opt-\niml: Scaling language model instruction meta learning\nthrough the lens of generalization. ArXiv preprint.\nKhattab, O., A. Singhvi, P. Maheshwari, Z. Zhang, K. San-\nthanam, S. Haq, A. Sharma, T. T. Joshi, H. Moazam,\nH. Miller, M. Zaharia, and C. Potts. 2024. DSPy: Compil-\ning declarative language model calls into self-improving\npipelines. ICLR .\nLin, C.-Y . 2004. ROUGE: A package for automatic evalua-\ntion of summaries. ACL 2004 Workshop on Text Summa-\nrization Branches Out .\nLongpre, S., L. Hou, T. Vu, A. Webson, H. W. Chung, Y . Tay,\nD. Zhou, Q. V . Le, B. Zoph, J. Wei, and A. Roberts. 2023.\nThe Flan collection: Designing data and methods for ef-\nfective instruction tuning. ICML .\nMin, S., X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Ha-\njishirzi, and L. Zettlemoyer. 2022. Rethinking the role of\ndemonstrations: What makes in-context learning work?\nEMNLP .\nMishra, S., D. Khashabi, C. Baral, and H. Hajishirzi. 2022.\nCross-task generalization via natural language crowd-\nsourcing instructions. ACL.Olsson, C., N. Elhage, N. Nanda, N. Joseph, N. DasSarma,\nT. Henighan, B. Mann, A. Askell, Y . Bai, A. Chen, et al.\n2022. In-context learning and induction heads. ArXiv\npreprint.\nOuyang, L., J. Wu, X. Jiang, D. Almeida, C. Wainwright,\nP. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray,\nJ. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens,\nA. Askell, P. Welinder, P. Christiano, J. Leike, and\nR. Lowe. 2022. Training language models to follow in-\nstructions with human feedback. NeurIPS , volume 35.\nPapineni, K., S. Roukos, T. Ward, and W.-J. Zhu. 2002. Bleu:\nA method for automatic evaluation of machine transla-\ntion. ACL.\nPrasad, A., P. Hase, X. Zhou, and M. Bansal. 2023. GrIPS:\nGradient-free, edit-based instruction search for prompt-\ning large language models. EACL .\nPryzant, R., D. Iter, J. Li, Y . Lee, C. Zhu, and M. Zeng. 2023.\nAutomatic prompt optimization with \u201cgradient descent\u201d\nand beam search. EMNLP .\nRajpurkar, P., R. Jia, and P. Liang. 2018. Know what you\ndon\u2019t know: Unanswerable questions for SQuAD. ACL.\nRajpurkar, P., J. Zhang, K. Lopyrev, and P. Liang. 2016.\nSQuAD: 100,000+ questions for machine comprehension\nof text. EMNLP .\nReynolds, L. and K. McDonell. 2021. Prompt program-\nming for large language models: Beyond the few-shot\nparadigm. CHI 2021 .\nRussell, S. and P. Norvig. 2002. Arti\ufb01cial Intelligence: A\nModern Approach , 2nd edition. Prentice Hall.\nSalvetti, F., J. B. Lowe, and J. H. Martin. 2016. A tangled\nweb: The faint signals of deception in text - boulder lies\nand truth corpus (BLT-C). LREC .\nSheng, E., K.-W. Chang, P. Natarajan, and N. Peng. 2019.\nThe woman worked as a babysitter: On biases in language\ngeneration. EMNLP .\nSingh, S., F. Vargus, D. D\u2019souza, B. F. Karlsson, A. Ma-\nhendiran, W.-Y . Ko, H. Shandilya, J. Patel, D. Mat-\naciunas, L. O\u2019Mahony, M. Zhang, R. Hettiarachchi,\nJ. Wilson, M. Machado, L. S. Moura, D. Krzemi \u00b4nski,\nH. Fadaei, I. Erg \u00a8un, I. Okoh, A. Alaagib, O. Mu-\ndannayake, Z. Alyafeai, V . M. Chien, S. Ruder,\nS. Guthikonda, E. A. Alghamdi, S. Gehrmann, N. Muen-\nnighoff, M. Bartolo, J. Kreutzer, A. \u00a8U\u00a8Ust\u00a8un, M. Fadaee,\nand S. Hooker. 2024. Aya dataset: An open-access collec-\ntion for multilingual instruction tuning. ArXiv preprint.\nSuzgun, M., N. Scales, N. Sch \u00a8arli, S. Gehrmann, Y . Tay,\nH. W. Chung, A. Chowdhery, Q. Le, E. Chi, D. Zhou, and\nJ. Wei. 2023. Challenging BIG-bench tasks and whether\nchain-of-thought can solve them. ACL Findings .\nWang, Y ., S. Mishra, P. Alipoormolabashi, Y . Kordi,\nA. Mirzaei, A. Naik, A. Ashok, A. S. Dhanasekaran,\nA. Arunkumar, D. Stap, E. Pathak, G. Karamanolakis,\nH. Lai, I. Purohit, I. Mondal, J. Anderson, K. Kuznia,\nK. Doshi, K. K. Pal, M. Patel, M. Moradshahi, M. Par-\nmar, M. Purohit, N. Varshney, P. R. Kaza, P. Verma,\nR. S. Puri, R. Karia, S. Doshi, S. K. Sampat, S. Mishra,\nS. Reddy A, S. Patro, T. Dixit, and X. Shen. 2022. Super-\nNaturalInstructions: Generalization via declarative in-\nstructions on 1600+ NLP tasks. EMNLP . 20 Chapter 12 \u2022 Model Alignment, Prompting, and In-Context Learning\nWebson, A. and E. Pavlick. 2022. Do prompt-based models\nreally understand the meaning of their prompts? NAACL\nHLT.\nWei, J., X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi,\nQ. V . Le, D. Zhou, et al. 2022. Chain-of-thought prompt-\ning elicits reasoning in large language models. NeurIPS ,\nvolume 35.\nZhou, Y ., A. I. Muresanu, Z. Han, K. Paster, S. Pitis,\nH. Chan, and J. Ba. 2023. Large language models are\nhuman-level prompt engineers. The Eleventh Interna-\ntional Conference on Learning Representations .", "13.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\n13Machine Translation\n\u201cI want to talk the dialect of your people. It\u2019s no use of talking unless\npeople understand what you say.\u201d\nZora Neale Hurston, Moses, Man of the Mountain 1939, p. 121\nThis chapter introduces machine translation (MT), the use of computers to trans-machine\ntranslation\nMT late from one language to another.\nOf course translation, in its full generality, such as the translation of literature, or\npoetry, is a dif\ufb01cult, fascinating, and intensely human endeavor, as rich as any other\narea of human creativity.\nMachine translation in its present form therefore focuses on a number of very\npractical tasks. Perhaps the most common current use of machine translation is\nforinformation access . We might want to translate some instructions on the web,information\naccess\nperhaps the recipe for a favorite dish, or the steps for putting together some furniture.\nOr we might want to read an article in a newspaper, or get information from an\nonline resource like Wikipedia or a government webpage in some other language.\nMT for information\naccess is probably\none of the most com-\nmon uses of NLP\ntechnology, and Google\nTranslate alone (shown above) translates hundreds of billions of words a day be-\ntween over 100 languages. Improvements in machine translation can thus help re-\nduce what is often called the digital divide in information access: the fact that much digital divide\nmore information is available in English and other languages spoken in wealthy\ncountries. Web searches in English return much more information than searches in\nother languages, and online resources like Wikipedia are much larger in English and\nother higher-resourced languages. High-quality translation can help provide infor-\nmation to speakers of lower-resourced languages.\nAnother common use of machine translation is to aid human translators. MT sys-\ntems are routinely used to produce a draft translation that is \ufb01xed up in a post-editing post-editing\nphase by a human translator. This task is often called computer-aided translation\norCAT . CAT is commonly used as part of localization : the task of adapting content CAT\nlocalization or a product to a particular language community.\nFinally, a more recent application of MT is to in-the-moment human commu-\nnication needs. This includes incremental translation, translating speech on-the-\ufb02y\nbefore the entire sentence is complete, as is commonly used in simultaneous inter-\npretation. Image-centric translation can be used for example to use OCR of the text\non a phone camera image as input to an MT system to translate menus or street signs.\nThe standard algorithm for MT is the encoder-decoder network, an architectureencoder-\ndecoder\nthat we introduced in Chapter 8 for RNNs. Recall that encoder-decoder or sequence-\nto-sequence models are used for tasks in which we need to map an input sequence to\nan output sequence that is a complex function of the entire input sequence. Indeed, 2CHAPTER 13 \u2022 M ACHINE TRANSLATION\nin machine translation, the words of the target language don\u2019t necessarily agree with\nthe words of the source language in number or order. Consider translating the fol-\nlowing made-up English sentence into Japanese.\n(13.1) English: He wrote a letter to a friend\nJapanese: tomodachi\nfriendni\ntotegami-o\nletterkaita\nwrote\nNote that the elements of the sentences are in very different places in the different\nlanguages. In English, the verb is in the middle of the sentence, while in Japanese,\nthe verb kaita comes at the end. The Japanese sentence doesn\u2019t require the pronoun\nhe, while English does.\nSuch differences between languages can be quite complex. In the following ac-\ntual sentence from the United Nations, notice the many changes between the Chinese\nsentence (we\u2019ve given in red a word-by-word gloss of the Chinese characters) and\nits English equivalent produced by human translators.\n(13.2)\u5927\u4f1a/General Assembly \u5728/on 1982\u5e74/1982 12\u6708/December 10 \u65e5/10\u901a\u8fc7\n\u4e86/adopted\u7b2c37\u53f7/37th\u51b3\u8bae/resolution \uff0c\u6838\u51c6\u4e86/approved \u7b2c\u4e8c\n\u6b21/second\u63a2\u7d22/exploration \u53ca/and\u548c\u5e73peaceful\u5229\u7528/using\u5916\u5c42\u7a7a\n\u95f4/outer space \u4f1a\u8bae/conference \u7684/of\u5404\u9879/various\u5efa\u8bae/suggestions \u3002\nOn 10 December 1982 , the General Assembly adopted resolution 37 in\nwhich it endorsed the recommendations of the Second United Nations\nConference on the Exploration and Peaceful Uses of Outer Space .\nNote the many ways the English and Chinese differ. For example the order-\ning differs in major ways; the Chinese order of the noun phrase is \u201cpeaceful using\nouter space conference of suggestions\u201d while the English has \u201csuggestions of the ...\nconference on peaceful use of outer space\u201d). And the order differs in minor ways\n(the date is ordered differently). English requires thein many places that Chinese\ndoesn\u2019t, and adds some details (like \u201cin which\u201d and \u201cit\u201d) that aren\u2019t necessary in\nChinese. Chinese doesn\u2019t grammatically mark plurality on nouns (unlike English,\nwhich has the \u201c-s\u201d in \u201crecommendations\u201d), and so the Chinese must use the modi-\n\ufb01er\u5404\u9879/various to make it clear that there is not just one recommendation. English\ncapitalizes some words but not others. Encoder-decoder networks are very success-\nful at handling these sorts of complicated cases of sequence mappings.\nWe\u2019ll begin in the next section by considering the linguistic background about\nhow languages vary, and the implications this variance has for the task of MT. Then\nwe\u2019ll sketch out the standard algorithm, give details about things like input tokeniza-\ntion and creating training corpora of parallel sentences, give some more low-level\ndetails about the encoder-decoder network, and \ufb01nally discuss how MT is evaluated,\nintroducing the simple chrF metric.\n13.1 Language Divergences and Typology\nThere are about 7,000 languages in the world. Some aspects of human language\nseem to be universal , holding true for every one of these languages, or are statistical universal\nuniversals, holding true for most of these languages. Many universals arise from the\nfunctional role of language as a communicative system by humans. Every language,\nfor example, seems to have words for referring to people, for talking about eating\nand drinking, for being polite or not. There are also structural linguistic univer-\nsals; for example, every language seems to have nouns and verbs (Chapter 17), has 13.1 \u2022 L ANGUAGE DIVERGENCES AND TYPOLOGY 3\nways to ask questions, or issue commands, has linguistic mechanisms for indicating\nagreement or disagreement.\nYet languages also differ in many ways (as has been pointed out since ancient\ntimes; see Fig. 13.1). Understanding what causes such translation divergencestranslation\ndivergence\n(Dorr, 1994) can help us build better MT models. We often distinguish the idiosyn-\ncratic and lexical differences that must be dealt with one by one (the word for \u201cdog\u201d\ndiffers wildly from language to language), from systematic differences that we can\nmodel in a general way (many languages put the verb before the grammatical ob-\nject; others put the verb after the grammatical object). The study of these systematic\ncross-linguistic similarities and differences is called linguistic typology . This sec- typology\ntion sketches some typological facts that impact machine translation; the interested\nreader should also look into WALS, the World Atlas of Language Structures, which\ngives many typological facts about languages (Dryer and Haspelmath, 2013).\nFigure 13.1 The Tower of Babel, Pieter Bruegel 1563. Wikimedia Commons, from the\nKunsthistorisches Museum, Vienna.\n13.1.1 Word Order Typology\nAs we hinted at in our example above comparing English and Japanese, languages\ndiffer in the basic word order of verbs, subjects, and objects in simple declara-\ntive clauses. German, French, English, and Mandarin, for example, are all SVO SVO\n(Subject-Verb-Object ) languages, meaning that the verb tends to come between\nthe subject and object. Hindi and Japanese, by contrast, are SOV languages, mean- SOV\ning that the verb tends to come at the end of basic clauses, and Irish and Arabic are\nVSO languages. Two languages that share their basic word order type often have VSO\nother similarities. For example, VOlanguages generally have prepositions , whereas\nOVlanguages generally have postpositions .\nLet\u2019s look in more detail at the example we saw above. In this SVO English\nsentence, the verb wrote is followed by its object a letter and the prepositional phrase 4CHAPTER 13 \u2022 M ACHINE TRANSLATION\nto a friend , in which the preposition tois followed by its argument a friend . Arabic,\nwith a VSO order, also has the verb before the object and prepositions. By contrast,\nin the Japanese example that follows, each of these orderings is reversed; the verb is\npreceded by its arguments, and the postposition follows its argument.\n(13.3) English: He wrote a letter to a friend\nJapanese: tomodachi\nfriendni\ntotegami-o\nletterkaita\nwrote\nArabic: katabt\nwroteris\u00afala\nletterli\nto\u02d9sadq\nfriend\nOther kinds of ordering preferences vary idiosyncratically from language to lan-\nguage. In some SVO languages (like English and Mandarin) adjectives tend to ap-\npear before nouns, while in others languages like Spanish and Modern Hebrew, ad-\njectives appear after the noun:\n(13.4) Spanish bruja verde English green witch\n(a) (b)\nFigure 13.2 Examples of other word order differences: (a) In German, adverbs occur in\ninitial position that in English are more natural later, and tensed verbs occur in second posi-\ntion. (b) In Mandarin, preposition phrases expressing goals often occur pre-verbally, unlike\nin English.\nFig. 13.2 shows examples of other word order differences. All of these word\norder differences between languages can cause problems for translation, requiring\nthe system to do huge structural reorderings as it generates the output.\n13.1.2 Lexical Divergences\nOf course we also need to translate the individual words from one language to an-\nother. For any translation, the appropriate word can vary depending on the context.\nThe English source-language word bass, for example, can appear in Spanish as the\n\ufb01shlubina or the musical instrument bajo. German uses two distinct words for what\nin English would be called a wall:Wand for walls inside a building, and Mauer for\nwalls outside a building. Where English uses the word brother for any male sib-\nling, Chinese and many other languages have distinct words for older brother and\nyounger brother (Mandarin gege anddidi, respectively). In all these cases, trans-\nlating bass,wall, orbrother from English would require a kind of specialization,\ndisambiguating the different uses of a word. For this reason the \ufb01elds of MT and\nWord Sense Disambiguation (Appendix G) are closely linked.\nSometimes one language places more grammatical constraints on word choice\nthan another. We saw above that English marks nouns for whether they are singular\nor plural. Mandarin doesn\u2019t. Or French and Spanish, for example, mark grammat-\nical gender on adjectives, so an English translation into French requires specifying\nadjective gender.\nThe way that languages differ in lexically dividing up conceptual space may be\nmore complex than this one-to-many translation problem, leading to many-to-many 13.1 \u2022 L ANGUAGE DIVERGENCES AND TYPOLOGY 5\nmappings. For example, Fig. 13.3 summarizes some of the complexities discussed\nby Hutchins and Somers (1992) in translating English leg, foot , and paw, to French.\nFor example, when legis used about an animal it\u2019s translated as French jambe ; but\nabout the leg of a journey, as French etape ; if the leg is of a chair, we use French\npied.\nFurther, one language may have a lexical gap , where no word or phrase, short lexical gap\nof an explanatory footnote, can express the exact meaning of a word in the other\nlanguage. For example, English does not have a word that corresponds neatly to\nMandarin xi`aoor Japanese oyak \u00afok\u00afo(in English one has to make do with awkward\nphrases like \ufb01lial piety orloving child , orgood son/daughter for both).\netapepattejambepied   paw        footlegJOURNEYANIMALHUMANCHAIRANIMALBIRDHUMAN\nFigure 13.3 The complex overlap between English leg,foot, etc., and various French trans-\nlations as discussed by Hutchins and Somers (1992).\nFinally, languages differ systematically in how the conceptual properties of an\nevent are mapped onto speci\ufb01c words. Talmy (1985, 1991) noted that languages\ncan be characterized by whether direction of motion and manner of motion are\nmarked on the verb or on the \u201csatellites\u201d: particles, prepositional phrases, or ad-\nverbial phrases. For example, a bottle \ufb02oating out of a cave would be described in\nEnglish with the direction marked on the particle out, while in Spanish the direction\nwould be marked on the verb:\n(13.5) English: The bottle \ufb02oated out.\nSpanish: La\nThebotella\nbottlesali\u00b4o\nexited\ufb02otando .\n\ufb02oating.\nVerb-framed languages mark the direction of motion on the verb (leaving the verb-framed\nsatellites to mark the manner of motion), like Spanish acercarse \u2018approach\u2019, al-\ncanzar \u2018reach\u2019, entrar \u2018enter\u2019, salir \u2018exit\u2019. Satellite-framed languages mark the satellite-framed\ndirection of motion on the satellite (leaving the verb to mark the manner of motion),\nlike English crawl out ,\ufb02oat off ,jump down ,run after . Languages like Japanese,\nTamil, and the many languages in the Romance, Semitic, and Mayan languages fam-\nilies, are verb-framed; Chinese as well as non-Romance Indo-European languages\nlike English, Swedish, Russian, Hindi, and Farsi are satellite framed (Talmy 1991,\nSlobin 1996).\n13.1.3 Morphological Typology\nMorphologically , languages are often characterized along two dimensions of vari-\nation. The \ufb01rst is the number of morphemes per word, ranging from isolating isolating\nlanguages like Vietnamese and Cantonese, in which each word generally has one\nmorpheme, to polysynthetic languages like Siberian Yupik (\u201cEskimo\u201d), in which a polysynthetic\nsingle word may have very many morphemes, corresponding to a whole sentence in\nEnglish. The second dimension is the degree to which morphemes are segmentable,\nranging from agglutinative languages like Turkish, in which morphemes have rel- agglutinative 6CHAPTER 13 \u2022 M ACHINE TRANSLATION\natively clean boundaries, to fusion languages like Russian, in which a single af\ufb01x fusion\nmay con\ufb02ate multiple morphemes, like -om in the word stolom (table- SG-INSTR -\nDECL 1), which fuses the distinct morphological categories instrumental, singular,\nand \ufb01rst declension.\nTranslating between languages with rich morphology requires dealing with struc-\nture below the word level, and for this reason modern systems generally use subword\nmodels like the wordpiece or BPE models of Section 13.2.1.\n13.1.4 Referential density\nFinally, languages vary along a typological dimension related to the things they tend\nto omit. Some languages, like English, require that we use an explicit pronoun when\ntalking about a referent that is given in the discourse. In other languages, however,\nwe can sometimes omit pronouns altogether, as the following example from Spanish\nshows1:\n(13.6) [El jefe] idio con un libro. / 0 iMostr \u00b4o su hallazgo a un descifrador ambulante.\n[The boss] came upon a book. [He] showed his \ufb01nd to a wandering decoder.\nLanguages that can omit pronouns are called pro-drop languages. Even among pro-drop\nthe pro-drop languages, there are marked differences in frequencies of omission.\nJapanese and Chinese, for example, tend to omit far more than does Spanish. This\ndimension of variation across languages is called the dimension of referential den-\nsity. We say that languages that tend to use more pronouns are more referentiallyreferential\ndensity\ndense than those that use more zeros. Referentially sparse languages, like Chinese or\nJapanese, that require the hearer to do more inferential work to recover antecedents\nare also called cold languages. Languages that are more explicit and make it easier cold language\nfor the hearer are called hotlanguages. The terms hotandcold are borrowed from hot language\nMarshall McLuhan\u2019s 1964 distinction between hot media like movies, which \ufb01ll in\nmany details for the viewer, versus cold media like comics, which require the reader\nto do more inferential work to \ufb01ll out the representation (Bickel, 2003).\nTranslating from languages with extensive pro-drop, like Chinese or Japanese, to\nnon-pro-drop languages like English can be dif\ufb01cult since the model must somehow\nidentify each zero and recover who or what is being talked about in order to insert\nthe proper pronoun.\n13.2 Machine Translation using Encoder-Decoder\nThe standard architecture for MT is the encoder-decoder transformer orsequence-\nto-sequence model, an architecture we saw for RNNs in Chapter 8. We\u2019ll see the\ndetails of how to apply this architecture to transformers in Section 13.3, but \ufb01rst let\u2019s\ntalk about the overall task.\nMost machine translation tasks make the simpli\ufb01cation that we can translate each\nsentence independently, so we\u2019ll just consider individual sentences for now. Given\na sentence in a source language, the MT task is then to generate a corresponding\nsentence in a target language. For example, an MT system is given an English\nsentence like\nThe green witch arrived\n1Here we use the / 0-notation; we\u2019ll introduce this and discuss this issue further in Chapter 23 13.2 \u2022 M ACHINE TRANSLATION USING ENCODER -DECODER 7\nand must translate it into the Spanish sentence:\nLleg \u00b4o la bruja verde\nMT uses supervised machine learning: at training time the system is given a\nlarge set of parallel sentences (each sentence in a source language matched with\na sentence in the target language), and learns to map source sentences into target\nsentences. In practice, rather than using words (as in the example above), we split\nthe sentences into a sequence of subword tokens (tokens can be words, or subwords,\nor individual characters). The systems are then trained to maximize the probability\nof the sequence of tokens in the target language y1;:::;ymgiven the sequence of\ntokens in the source language x1;:::;xn:\nP(y1;:::; ymjx1;:::; xn) (13.7)\nRather than use the input tokens directly, the encoder-decoder architecture con-\nsists of two components, an encoder and a decoder . The encoder takes the input\nwords x= [x1;:::; xn]and produces an intermediate context h. At decoding time, the\nsystem takes hand, word by word, generates the output y:\nh=encoder (x) (13.8)\nyt+1=decoder (h;y1;:::; yt))8t2[1;:::; m] (13.9)\nIn the next two sections we\u2019ll talk about subword tokenization, and then how to get\nparallel corpora for training, and then we\u2019ll introduce the details of the encoder-\ndecoder architecture.\n13.2.1 Tokenization\nMachine translation systems use a vocabulary that is \ufb01xed in advance, and rather\nthan using space-separated words, this vocabulary is generated with subword to-\nkenization algorithms, like the BPE algorithm sketched in Chapter 2. A shared\nvocabulary is used for the source and target languages, which makes it easy to copy\ntokens (like names) from source to target. Using subword tokenization with tokens\nshared between languages makes it natural to translate between languages like En-\nglish or Hindi that use spaces to separate words, and languages like Chinese or Thai\nthat don\u2019t.\nWe build the vocabulary by running a subword tokenization algorithm on a cor-\npus that contains both source and target language data.\nRather than the simple BPE algorithm from Fig. ??, modern systems often use\nmore powerful tokenization algorithms. Some systems (like BERT) use a variant of\nBPE called the wordpiece algorithm, which instead of choosing the most frequent wordpiece\nset of tokens to merge, chooses merges based on which one most increases the lan-\nguage model probability of the tokenization. Wordpieces use a special symbol at the\nbeginning of each token; here\u2019s a resulting tokenization from the Google MT system\n(Wu et al., 2016):\nwords : Jet makers feud over seat width with big orders at stake\nwordpieces :J et makers fe ud over seat width with big orders atstake\nThe wordpiece algorithm is given a training corpus and a desired vocabulary size\nV , and proceeds as follows:\n1. Initialize the wordpiece lexicon with characters (for example a subset of Uni-\ncode characters, collapsing all the remaining characters to a special unknown\ncharacter token). 8CHAPTER 13 \u2022 M ACHINE TRANSLATION\n2. Repeat until there are V wordpieces:\n(a) Train an n-gram language model on the training corpus, using the current\nset of wordpieces.\n(b) Consider the set of possible new wordpieces made by concatenating two\nwordpieces from the current lexicon. Choose the one new wordpiece that\nmost increases the language model probability of the training corpus.\nRecall that with BPE we had to specify the number of merges to perform; in\nwordpiece, by contrast, we specify the total vocabulary, which is a more intuitive\nparameter. A vocabulary of 8K to 32K word pieces is commonly used.\nAn even more commonly used tokenization algorithm is (somewhat ambigu-\nously) called the unigram algorithm (Kudo, 2018) or sometimes the SentencePiece unigram\nSentencePiece algorithm, and is used in systems like ALBERT (Lan et al., 2020) and T5 (Raf-\nfel et al., 2020). (Because unigram is the default tokenization algorithm used in\na library called SentencePiece that adds a useful wrapper around tokenization algo-\nrithms (Kudo and Richardson, 2018), authors often say they are using SentencePiece\ntokenization but really mean they are using the unigram algorithm).\nIn unigram tokenization, instead of building up a vocabulary by merging tokens,\nwe start with a huge vocabulary of every individual unicode character plus all fre-\nquent sequences of characters (including all space-separated words, for languages\nwith spaces), and iteratively remove some tokens to get to a desired \ufb01nal vocabulary\nsize. The algorithm is complex (involving suf\ufb01x-trees for ef\ufb01ciently storing many\ntokens, and the EM algorithm for iteratively assigning probabilities to tokens), so we\ndon\u2019t give it here, but see Kudo (2018) and Kudo and Richardson (2018). Roughly\nspeaking the algorithm proceeds iteratively by estimating the probability of each\ntoken, tokenizing the input data using various tokenizations, then removing a per-\ncentage of tokens that don\u2019t occur in high-probability tokenization, and then iterates\nuntil the vocabulary has been reduced down to the desired number of tokens.\nWhy does unigram tokenization work better than BPE? BPE tends to create lots\nof very small non-meaningful tokens (because BPE can only create larger words or\nmorphemes by merging characters one at a time), and it also tends to merge very\ncommon tokens, like the suf\ufb01x ed, onto their neighbors. We can see from these\nexamples from Bostrom and Durrett (2020) that unigram tends to produce tokens\nthat are more semantically meaningful:\nOriginal: corrupted Original: Completely preposterous suggestions\nBPE: cor rupted BPE: Comple t ely prep ost erous suggest ions\nUnigram: corrupt ed Unigram: Complete ly pre post er ous suggestion s\n13.2.2 Creating the Training data\nMachine translation models are trained on a parallel corpus , sometimes called a parallel corpus\nbitext , a text that appears in two (or more) languages. Large numbers of paral-\nlel corpora are available. Some are governmental; the Europarl corpus (Koehn, Europarl\n2005), extracted from the proceedings of the European Parliament, contains between\n400,000 and 2 million sentences each from 21 European languages. The United Na-\ntions Parallel Corpus contains on the order of 10 million sentences in the six of\ufb01cial\nlanguages of the United Nations (Arabic, Chinese, English, French, Russian, Span-\nish) Ziemski et al. (2016). Other parallel corpora have been made from movie and\nTV subtitles, like the OpenSubtitles corpus (Lison and Tiedemann, 2016), or from\ngeneral web text, like the ParaCrawl corpus of 223 million sentence pairs between\n23 EU languages and English extracted from the CommonCrawl Ba \u02dcn\u00b4on et al. (2020). 13.2 \u2022 M ACHINE TRANSLATION USING ENCODER -DECODER 9\nSentence alignment\nStandard training corpora for MT come as aligned pairs of sentences. When creat-\ning new corpora, for example for underresourced languages or new domains, these\nsentence alignments must be created. Fig. 13.4 gives a sample hypothetical sentence\nalignment.\nF1: -Bonjour, dit le petit prince.F2: -Bonjour, dit le marchand de pilules perfectionn\u00e9es qui apaisent la soif.F3: On en avale une par semaine et l'on n'\u00e9prouve plus le besoin de boire.F4: -C\u2019est une grosse \u00e9conomie de temps, dit le marchand.F5: Les experts ont fait des calculs.F6: On \u00e9pargne cinquante-trois minutes par semaine.F7: \u201cMoi, se dit le petit prince, si j'avais cinquante-trois minutes \u00e0 d\u00e9penser, je marcherais tout doucement vers une fontaine...\"E1: \u201cGood morning,\" said the little prince.E2: \u201cGood morning,\" said the merchant.E3: This was a merchant who sold pills that had been perfected to quench thirst.E4: You just swallow one pill a week and you won\u2019t feel the need for anything to drink.E5: \u201cThey save a huge amount of time,\" said the merchant.E6: \u201cFifty\u2212three minutes a week.\"E7: \u201cIf I had  fifty\u2212three minutes to spend?\" said the little prince to himself. E8: \u201cI would take a stroll to a spring of fresh water\u201d\nFigure 13.4 A sample alignment between sentences in English and French, with sentences extracted from\nAntoine de Saint-Exupery\u2019s Le Petit Prince and a hypothetical translation. Sentence alignment takes sentences\ne1;:::;en, and f1;:::;fnand \ufb01nds minimal sets of sentences that are translations of each other, including single\nsentence mappings like (e 1,f1), (e 4,f3), (e 5,f4), (e 6,f6) as well as 2-1 alignments (e 2/e3,f2), (e 7/e8,f7), and null\nalignments (f 5).\nGiven two documents that are translations of each other, we generally need two\nsteps to produce sentence alignments:\n\u2022 a cost function that takes a span of source sentences and a span of target sen-\ntences and returns a score measuring how likely these spans are to be transla-\ntions.\n\u2022 an alignment algorithm that takes these scores to \ufb01nd a good alignment be-\ntween the documents.\nTo score the similarity of sentences across languages, we need to make use of\namultilingual embedding space , in which sentences from different languages are\nin the same embedding space (Artetxe and Schwenk, 2019). Given such a space,\ncosine similarity of such embeddings provides a natural scoring function (Schwenk,\n2018). Thompson and Koehn (2019) give the following cost function between two\nsentences or spans x,yfrom the source and target documents respectively:\nc(x;y) =(1\u0000cos(x;y))nSents (x)nSents (y)PS\ns=11\u0000cos(x;ys)+PS\ns=11\u0000cos(xs;y)(13.10)\nwhere nSents ()gives the number of sentences (this biases the metric toward many\nalignments of single sentences instead of aligning very large spans). The denom-\ninator helps to normalize the similarities, and so x1;:::;xS;y1;:::;yS;are randomly\nselected sentences sampled from the respective documents.\nUsually dynamic programming is used as the alignment algorithm (Gale and\nChurch, 1993), in a simple extension of the minimum edit distance algorithm we\nintroduced in Chapter 2.\nFinally, it\u2019s helpful to do some corpus cleanup by removing noisy sentence pairs.\nThis can involve handwritten rules to remove low-precision pairs (for example re-\nmoving sentences that are too long, too short, have different URLs, or even pairs 10 CHAPTER 13 \u2022 M ACHINE TRANSLATION\nthat are too similar, suggesting that they were copies rather than translations). Or\npairs can be ranked by their multilingual embedding cosine score and low-scoring\npairs discarded.\n13.3 Details of the Encoder-Decoder Model\nEncoderThegreenlleg\u00f3witcharrived<s>lleg\u00f3la\nlabruja\nbrujaverde\nverde</s>Decodercross-attentiontransformerblocks\nFigure 13.5 The encoder-decoder transformer architecture for machine translation. The encoder uses the\ntransformer blocks we saw in Chapter 8, while the decoder uses a more powerful block with an extra cross-\nattention layer that can attend to all the encoder words. We\u2019ll see this in more detail in the next section.\nThe standard architecture for MT is the encoder-decoder transformer. The encoder-\ndecoder architecture was introduced already for RNNs in Chapter 8, and the trans-\nformer version has the same idea. Fig. 13.5 shows the intuition of the architec-\nture at a high level. You\u2019ll see that the encoder-decoder architecture is made up of\ntwo transformers: an encoder , which is the same as the basic transformers from\nChapter 9, and a decoder , which is augmented with a special new layer called the\ncross-attention layer. The encoder takes the source language input word tokens\nX=x1;:::;xnand maps them to an output representation Henc=h1;:::;hn; via a\nstack of encoder blocks.\nThe decoder is essentially a conditional language model that attends to the en-\ncoder representation and generates the target words one by one, at each timestep\nconditioning on the source sentence and the previously generated target language\nwords to generate a token. Decoding can use any of the decoding methods discussed\nin Chapter 9 like greedy, or temperature or nucleus sampling. But the most com-\nmon decoding algorithm for MT is the beam search algorithm that we\u2019ll introduce\nin Section 13.4.\nBut the components of the architecture differ somewhat from the transformer\nblock we\u2019ve seen. First, in order to attend to the source language, the transformer\nblocks in the decoder have an extra cross-attention layer. Recall that the transformer\nblock of Chapter 9 consists of a self-attention layer that attends to the input from\nthe previous layer, followed by layer norm, a feed forward layer, and another layer\nnorm. The decoder transformer block includes an extra layer with a special kind\nof attention, cross-attention (also sometimes called encoder-decoder attention or cross-attention\nsource attention ). Cross-attention has the same form as the multi-head attention\nin a normal transformer block, except that while the queries as usual come from\nthe previous layer of the decoder, the keys and values come from the output of the\nencoder . 13.3 \u2022 D ETAILS OF THE ENCODER -DECODER MODEL 11\nEncoderx1x2x3xn\u2026Decoderh3h2h1\u2026hn\nEncoderBlock 1Block 2Block Ky3y2y1\u2026\nDecoderBlock 1Block 2Block LUnembedding Matrixym\nMulti-Head AttentionLayer NormalizeLayer Normalize++\u2026Feedforward\nCausal Multi-Head AttentionLayer NormalizeLayer Normalize+\n+\u2026FeedforwardLayer Normalize+Cross-Attention\u2026\u2026\u2026\u2026\u2026\u2026LanguageModeling HeadHenc\nFigure 13.6 The transformer block for the encoder and the decoder. The \ufb01nal output of the encoder Henc=\nh1;:::;hnis the context used in the decoder. The decoder is a standard transformer except with one extra layer,\nthecross-attention layer, which takes that encoder output Hencand uses it to form its KandVinputs.\nThat is, where in standard multi-head attention the input to each attention layer is\nX, in cross attention the input is the the \ufb01nal output of the encoder Henc=h1;:::;hn.\nHencis of shape [n\u0002d], each row representing one input token. To link the keys\nand values from the encoder with the query from the prior layer of the decoder, we\nmultiple the encoder output Hencby the cross-attention layer\u2019s key weights WKand\nvalue weights WV. The query comes from the output from the prior decoder layer\nHdec[`\u00001], which is multiplied by the cross-attention layer\u2019s query weights WQ:\nQ=Hdec[`\u00001]WQ;K=HencWK;V=HencWV(13.11)\nCrossAttention (Q;K;V) = softmax\u0012QK|\npdk\u0013\nV (13.12)\nThe cross attention thus allows the decoder to attend to each of the source language\nwords as projected into the entire encoder \ufb01nal output representations. The other\nattention layer in each decoder block, the multi-head attention layer, is the same\ncausal (left-to-right) attention that we saw in Chapter 9. The multi-head attention in\nthe encoder, however, is allowed to look ahead at the entire source language text, so\nit is not masked.\nTo train an encoder-decoder model, we use the same self-supervision model we\nused for training encoder-decoders RNNs in Chapter 8. The network is given the\nsource text and then starting with the separator token is trained autoregressively to\npredict the next token using cross-entropy loss. Recall that cross-entropy loss for 12 CHAPTER 13 \u2022 M ACHINE TRANSLATION\nlanguage modeling is determined by the probability the model assigns to the correct\nnext word. So at time tthe CE loss is the negative log probability the model assigns\nto the next word in the training sequence:\nLCE(\u02c6yt;yt) =\u0000log\u02c6yt[wt+1] (13.13)\nAs in that case, we use teacher forcing in the decoder. Recall that in teacher forc- teacher forcing\ning, at each time step in decoding we force the system to use the gold target token\nfrom training as the next input xt+1, rather than allowing it to rely on the (possibly\nerroneous) decoder output \u02c6 yt.\n13.4 Decoding in MT: Beam Search\nRecall the greedy decoding algorithm from Chapter 9: at each time step tin gen-\neration, the output ytis chosen by computing the probability for each word in the\nvocabulary and then choosing the highest probability word (the argmax):\n\u02c6wt=argmaxw2VP(wjw<t) (13.14)\nA problem with greedy decoding is that what looks high probability at word tmight\nturn out to have been the wrong choice once we get to word t+1. The beam search\nalgorithm maintains multiple choices until later when we can see which one is best.\nIn beam search we model decoding as searching the space of possible genera-\ntions, represented as a search tree whose branches represent actions (generating a search tree\ntoken), and nodes represent states (having generated a particular pre\ufb01x). We search\nfor the best action sequence, i.e., the string with the highest probability.\nAn illustration of the problem\nFig. 13.7 shows a made-up example. The most probable sequence is ok ok EOS (its\nprobability is .4\u0002.7\u00021.0). But greedy search doesn\u2019t \ufb01nd it, incorrectly choosing\nyesas the \ufb01rst word since it has the highest local probability (0.5).\nstartokyesEOSokyesEOSokyesEOSEOSEOSEOSEOSt2t3p(t1|start)\nt1p(t2| t1)p(t3| t1,t2)\n.1.5.4.3.4.3.1.2.71.01.01.01.0\nFigure 13.7 A search tree for generating the target string T=t1;t2;:::from vocabulary\nV=fyes;ok;<s>g, showing the probability of generating each token from that state. Greedy\nsearch chooses yesfollowed by yes, instead of the globally most probable sequence ok ok .\nRecall from Chapter 17 that for part-of-speech tagging we used dynamic pro-\ngramming search (the Viterbi algorithm) to address this problem. Unfortunately, 13.4 \u2022 D ECODING IN MT: B EAM SEARCH 13\ndynamic programming is not applicable to generation problems with long-distance\ndependencies between the output decisions. The only method guaranteed to \ufb01nd the\nbest solution is exhaustive search: computing the probability of every one of the VT\npossible sentences (for some length value T) which is obviously too slow.\nThe solution: beam search\nInstead, MT systems generally decode using beam search , a heuristic search method beam search\n\ufb01rst proposed by Lowerre (1976). In beam search, instead of choosing the best token\nto generate at each timestep, we keep kpossible tokens at each step. This \ufb01xed-size\nmemory footprint kis called the beam width , on the metaphor of a \ufb02ashlight beam beam width\nthat can be parameterized to be wider or narrower.\nThus at the \ufb01rst step of decoding, we compute a softmax over the entire vocab-\nulary, assigning a probability to each word. We then select the k-best options from\nthis softmax output. These initial koutputs are the search frontier and these kinitial\nwords are called hypotheses . A hypothesis is an output sequence, a translation-so-\nfar, together with its probability.\na\u2026aardvark..arrived..the\u2026zebrastart\nt1a\u2026aardvark..the..witch\u2026zebraa\u2026aardvark..green..witch\u2026zebrat2hd1y1BOSy1y2\ny2hd1hd2thetheBOShd2greengreeny3hd1hd2arrivedarrivedBOSy2\nt3hd1hd2thetheBOSy2\nhd1hd2thetheBOShd2witchwitchy3a\u2026mage..the..witch\u2026zebraarrived\u2026aardvark..green..who\u2026zebray3\ny3\nFigure 13.8 Beam search decoding with a beam width of k=2. At each time step, we choose the kbest\nhypotheses, form the Vpossible extensions of each, score those k\u0002Vhypotheses and choose the best k=2\nto continue. At time 1, the frontier has the best 2 options from the initial decoder state: arrived andthe. We\nextend each, compute the probability of all the hypotheses so far ( arrived the ,arrived aardvark ,the green ,the\nwitch ) and again chose the best 2 ( the green andthe witch ) to be the search frontier. The images on the arcs\nschematically represent the decoders that must be run at each step to score the next words (for simplicity not\ndepicting cross-attention).\nAt subsequent steps, each of the kbest hypotheses is extended incrementally\nby being passed to distinct decoders, which each generate a softmax over the entire 14 CHAPTER 13 \u2022 M ACHINE TRANSLATION\nvocabulary to extend the hypothesis to every possible next token. Each of these k\u0002V\nhypotheses is scored by P(yijx;y<i): the product of the probability of the current\nword choice multiplied by the probability of the path that led to it. We then prune\nthek\u0002Vhypotheses down to the kbest hypotheses, so there are never more than k\nhypotheses at the frontier of the search, and never more than kdecoders. Fig. 13.8\nillustrates this with a beam width of 2 for the beginning of The green witch arrived .\nThis process continues until an EOS is generated indicating that a complete can-\ndidate output has been found. At this point, the completed hypothesis is removed\nfrom the frontier and the size of the beam is reduced by one. The search continues\nuntil the beam has been reduced to 0. The result will be khypotheses.\nTo score each node by its log probability, we use the chain rule of probability to\nbreak down p(yjx)into the product of the probability of each word given its prior\ncontext, which we can turn into a sum of logs (for an output string of length t):\nscore(y) = logP(yjx)\n=log(P(y1jx)P(y2jy1;x)P(y3jy1;y2;x):::P(ytjy1;:::;yt\u00001;x))\n=tX\ni=1logP(yijy1;:::;yi\u00001;x) (13.15)\nThus at each step, to compute the probability of a partial sentence, we simply add the\nlog probability of the pre\ufb01x sentence so far to the log probability of generating the\nnext token. Fig. 13.9 shows the scoring for the example sentence shown in Fig. 13.8,\nusing some simple made-up probabilities. Log probabilities are negative or 0, and\nthe max of two log probabilities is the one that is greater (closer to 0).\nBOSarrivedthethewitchgreenwitchmage\nwhoy2y3log P(y1|x)y1log P(y2|y1,x)log P(y3|y2,y1,x)-.92-1.6-1.2-.69-2.3-.69-1.6\n-2.3arrived-.11-.51witch-.36-.22EOS\n-.51EOS-2.3at-1.61bylog P(y4|y3,y2,y1,x)log P(y5|y4,y3,y2,y1,x)arrivedcame-1.6\ny4y5log P(arrived|x) log P(arrived witch|x)log P(the|x)log P(the green|x)log P(the witch|x) =-1.6log P (arrived the|x) log P (\u201cthe green witch arrived\u201d|x) = log P (the|x) + log P(green|the,x) + log P(witch | the, green,x)+logP(arrived|the,green,witch,x)+log P(EOS|the,green,witch,arrived,x)= -2.3= -3.9= -1.6= -2.1=-.92-2.1-3.2\n-4.4-2.2-2.5-3.7-2.7-3.8-2.7-4.8\nFigure 13.9 Scoring for beam search decoding with a beam width of k=2. We maintain the log probability\nof each hypothesis in the beam by incrementally adding the logprob of generating each next token. Only the top\nkpaths are extended to the next step.\nFig. 13.10 gives the algorithm. One problem with this version of the algorithm is\nthat the completed hypotheses may have different lengths. Because language mod-\nels generally assign lower probabilities to longer strings, a naive algorithm would 13.4 \u2022 D ECODING IN MT: B EAM SEARCH 15\nfunction BEAM DECODE (c,beam width )returns best paths\ny0,h0 0\npath ()\ncomplete paths ()\nstate (c,y0,h0, path) ;initial state\nfrontier hstatei ;initial frontier\nwhile frontier contains incomplete paths andbeamwidth >0\nextended frontier hi\nfor each state2frontier do\ny DECODE (state )\nfor each word i2Vocabulary do\nsuccessor NEWSTATE (state ,i,yi)\nextended frontier ADDTOBEAM (successor ,extended frontier ,\nbeam width )\nfor each state inextended frontier do\nifstate is complete do\ncomplete paths APPEND (complete paths ,state )\nextended frontier REMOVE (extended frontier ,state )\nbeam width beam width - 1\nfrontier extended frontier\nreturn completed paths\nfunction NEWSTATE (state ,word ,word prob)returns new state\nfunction ADDTOBEAM (state ,frontier ,width )returns updated frontier\nifLENGTH (frontier )<width then\nfrontier INSERT (state ,frontier )\nelse if SCORE (state )>SCORE (WORST OF(frontier ))\nfrontier REMOVE (WORST OF(frontier ))\nfrontier INSERT (state ,frontier )\nreturn frontier\nFigure 13.10 Beam search decoding.\nchoose shorter strings for y. (This is not an issue during the earlier steps of decod-\ning; since beam search is breadth-\ufb01rst, all the hypotheses being compared had the\nsame length.) For this reason we often apply length normalization methods, like\ndividing the logprob by the number of words:\nscore(y) =logP(yjx) =1\nttX\ni=1logP(yijy1;:::;yi\u00001;x) (13.16)\nFor MT we generally use beam widths kbetween 5 and 10, giving us khypotheses at\nthe end. We can pass all kto the downstream application with their respective scores,\nor if we just need a single translation we can pass the most probable hypothesis.\n13.4.1 Minimum Bayes Risk Decoding\nMinimum Bayes risk orMBR decoding is an alternative decoding algorithm thatminimum\nBayes risk\nMBR 16 CHAPTER 13 \u2022 M ACHINE TRANSLATION\ncan work even better than beam search and also tends to be better than the other\ndecoding algorithms like temperature sampling introduced in Section ??.\nThe intuition of minimum Bayes risk is that instead of trying to choose the trans-\nlation which is most probable, we choose the one that is likely have the least error.\nFor example, we might want our decoding algorithm to \ufb01nd the translation which\nhas the highest score on some evaluation metric. For example in Section 13.6 we will\nintroduce metrics like chrF or BERTScore that measure the goodness-of-\ufb01t between\na candidate translation and a set of reference human translations. A translation that\nmaximizes this score, especially with a hypothetically huge set of perfect human\ntranslations is likely to be a good one (have minimum risk) even if it is not the most\nprobable translation by our particular probability estimator.\nIn practice, we don\u2019t know the perfect set of translations for a given sentence. So\nthe standard simpli\ufb01cation used in MBR decoding algorithms is to instead choose\nthe candidate translation which is most similar (by some measure of goodness-of-\ufb01t)\nwith some set of candidate translations. We\u2019re essentially approximating the enor-\nmous space of all possible translations Uwith a smaller set of possible candidate\ntranslations Y.\nGiven this set of possible candidate translations Y, and some similarity or align-\nment function util, we choose the best translation \u02c6 yas the translation which is most\nsimilar to all the other candidate translations:\n\u02c6y=argmax\ny2YX\nc2Yutil(y;c) (13.17)\nVarious util functions can be used, like chrF or BERTscore or BLEU. We can get the\nset of candidate translations by sampling using one of the basic sampling algorithms\nof Section ??like temperature sampling; good results can be obtained with as few\nas 32 or 64 candidates.\nMinimum Bayes risk decoding can also be used for other NLP tasks; indeed\nit was widely applied to speech recognition (Stolcke et al., 1997; Goel and Byrne,\n2000) before being applied to machine translation (Kumar and Byrne, 2004), and\nhas been shown to work well across many other generation tasks as well (e.g., sum-\nmarization, dialogue, and image captioning (Suzgun et al., 2023)).\n13.5 Translating in low-resource situations\nFor some languages, and especially for English, online resources are widely avail-\nable. There are many large parallel corpora that contain translations between En-\nglish and many languages. But the vast majority of the world\u2019s languages do not\nhave large parallel training texts available. An important ongoing research question\nis how to get good translation with lesser resourced languages. The resource prob-\nlem can even be true for high resource languages when we need to translate into low\nresource domains (for example in a particular genre that happens to have very little\nbitext).\nHere we brie\ufb02y introduce two commonly used approaches for dealing with this\ndata sparsity: backtranslation , which is a special case of the general statistical\ntechnique called data augmentation , and multilingual models , and also discuss\nsome socio-technical issues. 13.5 \u2022 T RANSLATING IN LOW -RESOURCE SITUATIONS 17\n13.5.1 Data Augmentation\nData augmentation is a statistical technique for dealing with insuf\ufb01cient training\ndata, by adding new synthetic data that is generated from the current natural data.\nThe most common data augmentation technique for machine translation is called\nbacktranslation . Backtranslation relies on the intuition that while parallel corpora backtranslation\nmay be limited for particular languages or domains, we can often \ufb01nd a large (or\nat least larger) monolingual corpus, to add to the smaller parallel corpora that are\navailable. The algorithm makes use of monolingual corpora in the target language\nby creating synthetic bitexts.\nIn backtranslation, our goal is to improve source-to-target MT, given a small\nparallel text (a bitext) in the source/target languages, and some monolingual data in\nthe target language. We \ufb01rst use the bitext to train a MT system in the reverse di-\nrection: a target-to-source MT system . We then use it to translate the monolingual\ntarget data to the source language. Now we can add this synthetic bitext (natural\ntarget sentences, aligned with MT-produced source sentences) to our training data,\nand retrain our source-to-target MT model. For example suppose we want to trans-\nlate from Navajo to English but only have a small Navajo-English bitext, although of\ncourse we can \ufb01nd lots of monolingual English data. We use the small bitext to build\nan MT engine going the other way (from English to Navajo). Once we translate the\nmonolingual English text to Navajo, we can add this synthetic Navajo/English bitext\nto our training data.\nBacktranslation has various parameters. One is how we generate the backtrans-\nlated data; we can run the decoder in greedy inference, or use beam search. Or\nwe can do sampling, like the temperature sampling algorithm we saw in Chapter 9.\nAnother parameter is the ratio of backtranslated data to natural bitext data; we can\nchoose to upsample the bitext data (include multiple copies of each sentence). In\ngeneral backtranslation works surprisingly well; one estimate suggests that a system\ntrained on backtranslated text gets about 2/3 of the gain as would training on the\nsame amount of natural bitext (Edunov et al., 2018).\n13.5.2 Multilingual models\nThe models we\u2019ve described so far are for bilingual translation: one source language,\none target language. It\u2019s also possible to build a multilingual translator.\nIn a multilingual translator, we train the system by giving it parallel sentences\nin many different pairs of languages. That means we need to tell the system which\nlanguage to translate from and to! We tell the system which language is which\nby adding a special token lsto the encoder specifying the source language we\u2019re\ntranslating from, and a special token ltto the decoder telling it the target language\nwe\u2019d like to translate into.\nThus we slightly update Eq. 13.9 above to add these tokens in Eq. 13.19:\nh=encoder (x;ls) (13.18)\nyi+1=decoder (h;lt;y1;:::; yi))8i2[1;:::; m] (13.19)\nOne advantage of a multilingual model is that they can improve the translation\nof lower-resourced languages by drawing on information from a similar language\nin the training data that happens to have more resources. Perhaps we don\u2019t know\nthe meaning of a word in Galician, but the word appears in the similar and higher-\nresourced language Spanish. 18 CHAPTER 13 \u2022 M ACHINE TRANSLATION\n13.5.3 Sociotechnical issues\nMany issues in dealing with low-resource languages go beyond the purely techni-\ncal. One problem is that for low-resource languages, especially from low-income\ncountries, native speakers are often not involved as the curators for content selec-\ntion, as the language technologists, or as the evaluators who measure performance\n(8et al., 2020). Indeed, one well-known study that manually audited a large set of\nparallel corpora and other major multilingual datasets found that for many of the\ncorpora, less than 50% of the sentences were of acceptable quality, with a lot of\ndata consisting of repeated sentences with web boilerplate or incorrect translations,\nsuggesting that native speakers may not have been suf\ufb01ciently involved in the data\nprocess (Kreutzer et al., 2022).\nOther issues, like the tendency of many MT approaches to focus on the case\nwhere one of the languages is English (Anastasopoulos and Neubig, 2020), have to\ndo with allocation of resources. Where most large multilingual systems were trained\non bitexts in which English was one of the two languages, recent huge corporate\nsystems like those of Fan et al. (2021) and Costa-juss `a et al. (2022) and datasets\nlike Schwenk et al. (2021) attempt to handle large numbers of languages (up to 200\nlanguages) and create bitexts between many more pairs of languages and not just\nthrough English.\nAt the smaller end, 8et al. (2020) propose a participatory design process to\nencourage content creators, curators, and language technologists who speak these\nlow-resourced languages to participate in developing MT algorithms. They provide\nonline groups, mentoring, and infrastructure, and report on a case study on devel-\noping MT algorithms for low-resource African languages. Among their conclusions\nwas to perform MT evaluation by post-editing rather than direct evaluation, since\nhaving labelers edit an MT system and then measure the distance between the MT\noutput and its post-edited version both was simpler to train evaluators and makes it\neasier to measure true errors in the MT output and not differences due to linguistic\nvariation (Bentivogli et al., 2018).\n13.6 MT Evaluation\nTranslations are evaluated along two dimensions:\n1.adequacy: how well the translation captures the exact meaning of the source adequacy\nsentence. Sometimes called faithfulness or\ufb01delity .\n2.\ufb02uency: how \ufb02uent the translation is in the target language (is it grammatical, \ufb02uency\nclear, readable, natural).\nUsing humans to evaluate is most accurate, but automatic metrics are also used for\nconvenience.\n13.6.1 Using Human Raters to Evaluate MT\nThe most accurate evaluations use human raters, such as online crowdworkers, to\nevaluate each translation along the two dimensions. For example, along the dimen-\nsion of \ufb02uency , we can ask how intelligible, how clear, how readable, or how natural\nthe MT output (the target text) is. We can give the raters a scale, for example, from\n1 (totally unintelligible) to 5 (totally intelligible), or 1 to 100, and ask them to rate\neach sentence or paragraph of the MT output. 13.6 \u2022 MT E VALUATION 19\nWe can do the same thing to judge the second dimension, adequacy , using raters\nto assign scores on a scale. If we have bilingual raters, we can give them the source\nsentence and a proposed target sentence, and rate, on a 5-point or 100-point scale,\nhow much of the information in the source was preserved in the target. If we only\nhave monolingual raters but we have a good human translation of the source text, we\ncan give the monolingual raters the human reference translation and a target machine\ntranslation and again rate how much information is preserved. An alternative is to\ndoranking : give the raters a pair of candidate translations, and ask them which one ranking\nthey prefer.\nTraining of human raters (who are often online crowdworkers) is essential; raters\nwithout translation expertise \ufb01nd it dif\ufb01cult to separate \ufb02uency and adequacy, and\nso training includes examples carefully distinguishing these. Raters often disagree\n(source sentences may be ambiguous, raters will have different world knowledge,\nraters may apply scales differently). It is therefore common to remove outlier raters,\nand (if we use a \ufb01ne-grained enough scale) normalizing raters by subtracting the\nmean from their scores and dividing by the variance.\nAs discussed above, an alternative way of using human raters is to have them\npost-edit translations, taking the MT output and changing it minimally until they\nfeel it represents a correct translation. The difference between their post-edited\ntranslations and the original MT output can then be used as a measure of quality.\n13.6.2 Automatic Evaluation\nWhile humans produce the best evaluations of machine translation output, running a\nhuman evaluation can be time consuming and expensive. For this reason automatic\nmetrics are often used as temporary proxies. Automatic metrics are less accurate\nthan human evaluation, but can help test potential system improvements, and even\nbe used as an automatic loss function for training. In this section we introduce two\nfamilies of such metrics, those based on character- or word-overlap and those based\non embedding similarity.\nAutomatic Evaluation by Character Overlap: chrF\nThe simplest and most robust metric for MT evaluation is called chrF , which stands chrF\nforcharacter F-score (Popovi \u00b4c, 2015). chrF (along with many other earlier related\nmetrics like BLEU, METEOR, TER, and others) is based on a simple intuition de-\nrived from the pioneering work of Miller and Beebe-Center (1956): a good machine\ntranslation will tend to contain characters and words that occur in a human trans-\nlation of the same sentence. Consider a test set from a parallel corpus, in which\neach source sentence has both a gold human target translation and a candidate MT\ntranslation we\u2019d like to evaluate. The chrF metric ranks each MT target sentence by\na function of the number of character n-gram overlaps with the human translation.\nGiven the hypothesis and the reference, chrF is given a parameter kindicating\nthe length of character n-grams to be considered, and computes the average of the\nkprecisions (unigram precision, bigram, and so on) and the average of the krecalls\n(unigram recall, bigram recall, etc.):\nchrP percentage of character 1-grams, 2-grams, ..., k-grams in the hypothesis that\noccur in the reference, averaged.\nchrR percentage of character 1-grams, 2-grams,..., k-grams in the reference that\noccur in the hypothesis, averaged.\nThe metric then computes an F-score by combining chrP and chrR using a weighting 20 CHAPTER 13 \u2022 M ACHINE TRANSLATION\nparameter b. It is common to set b=2, thus weighing recall twice as much as\nprecision:\nchrFb= (1+b2)chrP\u0001chrR\nb2\u0001chrP+chrR(13.20)\nForb=2, that would be:\nchrF2 =5\u0001chrP\u0001chrR\n4\u0001chrP+chrR\nFor example, consider two hypotheses that we\u2019d like to score against the refer-\nence translation witness for the past . Here are the hypotheses along with chrF values\ncomputed using parameters k=b=2 (in real examples, kwould be a higher number\nlike 6):\nREF: witness for the past,\nHYP1: witness of the past, chrF2,2 = .86\nHYP2: past witness chrF2,2 = .62\nLet\u2019s see how we computed that chrF value for HYP1 (we\u2019ll leave the compu-\ntation of the chrF value for HYP2 as an exercise for the reader). First, chrF ignores\nspaces, so we\u2019ll remove them from both the reference and hypothesis:\nREF: witnessforthepast, (18 unigrams, 17 bigrams)\nHYP1: witnessofthepast, (17 unigrams, 16 bigrams)\nNext let\u2019s see how many unigrams and bigrams match between the reference and\nhypothesis:\nunigrams that match: w i t n e s s f o t h e p a s t , (17 unigrams)\nbigrams that match: wi it tn ne es ss th he ep pa as st t, (13 bigrams)\nWe use that to compute the unigram and bigram precisions and recalls:\nunigram P: 17/17 = 1 unigram R: 17/18 = .944\nbigram P: 13/16 = .813 bigram R: 13/17 = .765\nFinally we average to get chrP and chrR, and compute the F-score:\nchrP = (17=17+13=16)=2=:906\nchrR = (17=18+13=17)=2=:855\nchrF2,2 =5chrP\u0003chrR\n4chrP +chrR=:86\nchrF is simple, robust, and correlates very well with human judgments in many\nlanguages (Kocmi et al., 2021).\nAlternative overlap metric: BLEU\nThere are various alternative overlap metrics. For example, before the development\nof chrF, it was common to use a word-based overlap metric called BLEU (for BiLin-\ngual Evaluation Understudy), that is purely precision-based rather than combining\nprecision and recall (Papineni et al., 2002). The BLEU score for a corpus of candi-\ndate translation sentences is a function of the n-gram word precision over all the\nsentences combined with a brevity penalty computed over the corpus as a whole.\nWhat do we mean by n-gram precision? Consider a corpus composed of a single\nsentence. The unigram precision for this corpus is the percentage of unigram tokens 13.6 \u2022 MT E VALUATION 21\nin the candidate translation that also occur in the reference translation, and ditto for\nbigrams and so on, up to 4-grams. BLEU extends this unigram metric to the whole\ncorpus by computing the numerator as the sum over all sentences of the counts of all\nthe unigram types that also occur in the reference translation, and the denominator\nis the total of the counts of all unigrams in all candidate sentences. We compute\nthis n-gram precision for unigrams, bigrams, trigrams, and 4-grams and take the\ngeometric mean. BLEU has many further complications, including a brevity penalty\nfor penalizing candidate translations that are too short, and it also requires the n-\ngram counts be clipped in a particular way.\nBecause BLEU is a word-based metric, it is very sensitive to word tokenization,\nmaking it impossible to compare different systems if they rely on different tokeniza-\ntion standards, and doesn\u2019t work as well in languages with complex morphology.\nNonetheless, you will sometimes still see systems evaluated by BLEU, particularly\nfor translation into English. In such cases it\u2019s important to use packages that enforce\nstandardization for tokenization like S ACRE BLEU (Post, 2018).\nStatistical Signi\ufb01cance Testing for MT evals\nCharacter or word overlap-based metrics like chrF (or BLEU, or etc.) are mainly\nused to compare two systems, with the goal of answering questions like: did the\nnew algorithm we just invented improve our MT system? To know if the difference\nbetween the chrF scores of two MT systems is a signi\ufb01cant difference, we use the\npaired bootstrap test, or the similar randomization test.\nTo get a con\ufb01dence interval on a single chrF score using the bootstrap test, recall\nfrom Section ??that we take our test set (or devset) and create thousands of pseudo-\ntestsets by repeatedly sampling with replacement from the original test set. We now\ncompute the chrF score of each of the pseudo-testsets. If we drop the top 2.5% and\nbottom 2.5% of the scores, the remaining scores will give us the 95% con\ufb01dence\ninterval for the chrF score of our system.\nTo compare two MT systems A and B, we draw the same set of pseudo-testsets,\nand compute the chrF scores for each of them. We then compute the percentage of\npseudo-test-sets in which A has a higher chrF score than B.\nchrF: Limitations\nWhile automatic character and word-overlap metrics like chrF or BLEU are useful,\nthey have important limitations. chrF is very local: a large phrase that is moved\naround might barely change the chrF score at all, and chrF can\u2019t evaluate cross-\nsentence properties of a document like its discourse coherence (Chapter 24). chrF\nand similar automatic metrics also do poorly at comparing very different kinds of\nsystems, such as comparing human-aided translation against machine translation, or\ndifferent machine translation architectures against each other (Callison-Burch et al.,\n2006). Instead, automatic overlap metrics like chrF are most appropriate when eval-\nuating changes to a single system.\n13.6.3 Automatic Evaluation: Embedding-Based Methods\nThe chrF metric is based on measuring the exact character n-grams a human refer-\nence and candidate machine translation have in common. However, this criterion\nis overly strict, since a good translation may use alternate words or paraphrases. A\nsolution \ufb01rst pioneered in early metrics like METEOR (Banerjee and Lavie, 2005)\nwas to allow synonyms to match between the reference xand candidate \u02dc x. More 22 CHAPTER 13 \u2022 M ACHINE TRANSLATION\nrecent metrics use BERT or other embeddings to implement this intuition.\nFor example, in some situations we might have datasets that have human as-\nsessments of translation quality. Such datasets consists of tuples (x;\u02dcx;r), where\nx= (x1;:::; xn)is a reference translation, \u02dc x= (\u02dcx1;:::; \u02dcxm)is a candidate machine\ntranslation, and r2Ris a human rating that expresses the quality of \u02dc xwith respect\ntox. Given such data, algorithms like COMET (Rei et al., 2020) BLEURT (Sellam\net al., 2020) train a predictor on the human-labeled datasets, for example by passing\nxand \u02dcxthrough a version of BERT (trained with extra pretraining, and then \ufb01netuned\non the human-labeled sentences), followed by a linear layer that is trained to predict\nr. The output of such models correlates highly with human labels.\nIn other cases, however, we don\u2019t have such human-labeled datasets. In that\ncase we can measure the similarity of xand \u02dcxby the similarity of their embeddings.\nThe BERTS CORE algorithm (Zhang et al., 2020) shown in Fig. 13.11, for example,\npasses the reference xand the candidate \u02dc xthrough BERT, computing a BERT em-\nbedding for each token xiand \u02dcxj. Each pair of tokens (xi;\u02dcxj)is scored by its cosine\nxi\u0001\u02dcxj\njxijj\u02dcxjj. Each token in xis matched to a token in \u02dc xto compute recall, and each token in\n\u02dcxis matched to a token in xto compute precision (with each token greedily matched\nto the most similar token in the corresponding sentence). BERTS CORE provides\nprecision and recall (and hence F 1):\nRBERT=1\njxjX\nxi2xmax\n\u02dcxj2\u02dcxxi\u0001\u02dcxjPBERT=1\nj\u02dcxjX\n\u02dcxj2\u02dcxmax\nxi2xxi\u0001\u02dcxj (13.21)\nPublished as a conference paper at ICLR 2020\nReferencethe weather is cold today\nCandidateit is freezing today\nCandidateContextualEmbeddingPairwise CosineSimilarityRBERT=(0.713\u00001.27)+(0.515\u00007.94)+...1.27+7.94+1.82+7.90+8.88\n<latexit sha1_base64=\"OJyoKlmBAgUA0KDtUcsH/di5BlI=\">AAACSHicbZDLattAFIaPnLRJ3JvTLrsZYgoJAqFxGqwsCqal0FVJQ5wELCNG41EyZHRh5ijECL1EnqAv002X2eUZsumipXRR6Mj2Ipf+MPDznXM4Z/64UNKg7187raXlR49XVtfaT54+e/6is/7y0OSl5mLIc5Xr45gZoWQmhihRieNCC5bGShzFZx+a+tG50Ebm2QFOCzFO2UkmE8kZWhR1ov2oClFcYPX+4/5BXZN3JEw049Wm7/XpdogyFYZQr9ffci3aoTsL1Pd23265oZrkaOqqaXAb5FIv6DXOdwMvCOqo0/U9fyby0NCF6Q52/15+BYC9qHMVTnJepiJDrpgxI+oXOK6YRsmVqNthaUTB+Bk7ESNrM2aPGVezIGryxpIJSXJtX4ZkRm9PVCw1ZprGtjNleGru1xr4v9qoxCQYVzIrShQZny9KSkUwJ02qZCK14Kim1jCupb2V8FNmc0SbfduGQO9/+aE57HnU9+gX2h18hrlW4TVswCZQ6MMAPsEeDIHDN7iBn/DL+e78cH47f+atLWcx8wruqNX6B8dUrVw=</latexit><latexit sha1_base64=\"RInTcZkWiVBnf/ncBstCvatCtG4=\">AAACSHicbZDPShxBEMZ7Nproxugaj14al4AyMEyvyoyHwGIQPImKq8LOMvT09mhjzx+6a0KWYV4iL5EnySXH3HwGLx4U8SDYs7sHo/mg4eNXVVT1F+VSaHDda6vxbmb2/Ye5+ebHhU+LS63lz6c6KxTjPZbJTJ1HVHMpUt4DAZKf54rTJJL8LLr6VtfPvnOlRZaewCjng4RepCIWjIJBYSs8DssA+A8od/eOT6oKf8VBrCgr113HI5sBiIRrTJyOt2EbtE22p8hzdrY27EAOM9BVWTfYNbKJ43dq59q+4/tV2Gq7jjsWfmvI1LS7O08/f3nLi4dh628wzFiR8BSYpFr3iZvDoKQKBJO8agaF5jllV/SC941NqTlmUI6DqPAXQ4Y4zpR5KeAxfTlR0kTrURKZzoTCpX5dq+H/av0CYn9QijQvgKdssiguJIYM16nioVCcgRwZQ5kS5lbMLqnJEUz2TRMCef3lt+a04xDXIUek3T1AE82hVbSG1hFBHuqifXSIeoih3+gG3aF76491az1Yj5PWhjWdWUH/qNF4BkPYrbk=</latexit><latexit sha1_base64=\"RInTcZkWiVBnf/ncBstCvatCtG4=\">AAACSHicbZDPShxBEMZ7Nproxugaj14al4AyMEyvyoyHwGIQPImKq8LOMvT09mhjzx+6a0KWYV4iL5EnySXH3HwGLx4U8SDYs7sHo/mg4eNXVVT1F+VSaHDda6vxbmb2/Ye5+ebHhU+LS63lz6c6KxTjPZbJTJ1HVHMpUt4DAZKf54rTJJL8LLr6VtfPvnOlRZaewCjng4RepCIWjIJBYSs8DssA+A8od/eOT6oKf8VBrCgr113HI5sBiIRrTJyOt2EbtE22p8hzdrY27EAOM9BVWTfYNbKJ43dq59q+4/tV2Gq7jjsWfmvI1LS7O08/f3nLi4dh628wzFiR8BSYpFr3iZvDoKQKBJO8agaF5jllV/SC941NqTlmUI6DqPAXQ4Y4zpR5KeAxfTlR0kTrURKZzoTCpX5dq+H/av0CYn9QijQvgKdssiguJIYM16nioVCcgRwZQ5kS5lbMLqnJEUz2TRMCef3lt+a04xDXIUek3T1AE82hVbSG1hFBHuqifXSIeoih3+gG3aF76491az1Yj5PWhjWdWUH/qNF4BkPYrbk=</latexit><latexit sha1_base64=\"fGWl4NCvlvtMu17rjLtk25oWpdc=\">AAACSHicbZBLS+RAFIUrPT7bVzsu3RQ2ghIIqVbpuBgQRZiVqNgqdJpQqa5oYeVB1Y1ME/Lz3Lic3fwGNy6UwZ2VNgtfBwoO372Xe+uEmRQaXPef1fgxMTk1PTPbnJtfWFxqLf8812muGO+xVKbqMqSaS5HwHgiQ/DJTnMah5BfhzUFVv7jlSos0OYNRxgcxvUpEJBgFg4JWcBoUPvA/UOwfnp6VJf6F/UhRVmy4Tpds+SBirjFxOt1N26AdslOjrrO7vWn7cpiCLouqwa6QTRyvUznX9hzPK4NW23XcsfBXQ2rTRrWOg9Zff5iyPOYJMEm17hM3g0FBFQgmedn0c80zym7oFe8bm1BzzKAYB1HidUOGOEqVeQngMX0/UdBY61Ecms6YwrX+XKvgd7V+DpE3KESS5cAT9rYoyiWGFFep4qFQnIEcGUOZEuZWzK6pyRFM9k0TAvn85a/mvOMQ1yEnpL13VMcxg1bRGtpABHXRHvqNjlEPMXSHHtATerburUfrv/Xy1tqw6pkV9EGNxisxMKq0</latexit>1.27\n7.941.827.908.88idf\nweightsImportance Weighting(Optional)Maximum Similarityx<latexit sha1_base64=\"f2yzimwbR/Dgjzp6tZ360fHRqNI=\">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cW7Ae0oWy2k3btZhN2N2IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3H1FpHst7M0nQj+hQ8pAzaqzUeOqXK27VnYOsEi8nFchR75e/eoOYpRFKwwTVuuu5ifEzqgxnAqelXqoxoWxMh9i1VNIItZ/ND52SM6sMSBgrW9KQufp7IqOR1pMosJ0RNSO97M3E/7xuasJrP+MySQ1KtlgUpoKYmMy+JgOukBkxsYQyxe2thI2ooszYbEo2BG/55VXSuqh6btVrXFZqN3kcRTiBUzgHD66gBndQhyYwQHiGV3hzHpwX5935WLQWnHzmGP7A+fwB5jmM/A==</latexit><latexit sha1_base64=\"f2yzimwbR/Dgjzp6tZ360fHRqNI=\">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cW7Ae0oWy2k3btZhN2N2IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3H1FpHst7M0nQj+hQ8pAzaqzUeOqXK27VnYOsEi8nFchR75e/eoOYpRFKwwTVuuu5ifEzqgxnAqelXqoxoWxMh9i1VNIItZ/ND52SM6sMSBgrW9KQufp7IqOR1pMosJ0RNSO97M3E/7xuasJrP+MySQ1KtlgUpoKYmMy+JgOukBkxsYQyxe2thI2ooszYbEo2BG/55VXSuqh6btVrXFZqN3kcRTiBUzgHD66gBndQhyYwQHiGV3hzHpwX5935WLQWnHzmGP7A+fwB5jmM/A==</latexit><latexit sha1_base64=\"f2yzimwbR/Dgjzp6tZ360fHRqNI=\">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cW7Ae0oWy2k3btZhN2N2IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3H1FpHst7M0nQj+hQ8pAzaqzUeOqXK27VnYOsEi8nFchR75e/eoOYpRFKwwTVuuu5ifEzqgxnAqelXqoxoWxMh9i1VNIItZ/ND52SM6sMSBgrW9KQufp7IqOR1pMosJ0RNSO97M3E/7xuasJrP+MySQ1KtlgUpoKYmMy+JgOukBkxsYQyxe2thI2ooszYbEo2BG/55VXSuqh6btVrXFZqN3kcRTiBUzgHD66gBndQhyYwQHiGV3hzHpwX5935WLQWnHzmGP7A+fwB5jmM/A==</latexit><latexit sha1_base64=\"f2yzimwbR/Dgjzp6tZ360fHRqNI=\">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cW7Ae0oWy2k3btZhN2N2IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3H1FpHst7M0nQj+hQ8pAzaqzUeOqXK27VnYOsEi8nFchR75e/eoOYpRFKwwTVuuu5ifEzqgxnAqelXqoxoWxMh9i1VNIItZ/ND52SM6sMSBgrW9KQufp7IqOR1pMosJ0RNSO97M3E/7xuasJrP+MySQ1KtlgUpoKYmMy+JgOukBkxsYQyxe2thI2ooszYbEo2BG/55VXSuqh6btVrXFZqN3kcRTiBUzgHD66gBndQhyYwQHiGV3hzHpwX5935WLQWnHzmGP7A+fwB5jmM/A==</latexit>\u02c6x<latexit sha1_base64=\"5QTnVRVSrnyzznVU7d5bF5u03Iw=\">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDbbTbt0swm7E7GE/ggvHhTx6u/x5r9x0+agrQ8GHu/NMDMvSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHJtRKwecJpwP6IjJULBKFqp0x9TzJ5mg2rNrbtzkFXiFaQGBZqD6ld/GLM04gqZpMb0PDdBP6MaBZN8VumnhieUTeiI9yxVNOLGz+bnzsiZVYYkjLUthWSu/p7IaGTMNApsZ0RxbJa9XPzP66UYXvuZUEmKXLHFojCVBGOS/06GQnOGcmoJZVrYWwkbU00Z2oQqNgRv+eVV0r6oe27du7+sNW6KOMpwAqdwDh5cQQPuoAktYDCBZ3iFNydxXpx352PRWnKKmWP4A+fzB7A8j8k=</latexit><latexit sha1_base64=\"5QTnVRVSrnyzznVU7d5bF5u03Iw=\">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDbbTbt0swm7E7GE/ggvHhTx6u/x5r9x0+agrQ8GHu/NMDMvSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHJtRKwecJpwP6IjJULBKFqp0x9TzJ5mg2rNrbtzkFXiFaQGBZqD6ld/GLM04gqZpMb0PDdBP6MaBZN8VumnhieUTeiI9yxVNOLGz+bnzsiZVYYkjLUthWSu/p7IaGTMNApsZ0RxbJa9XPzP66UYXvuZUEmKXLHFojCVBGOS/06GQnOGcmoJZVrYWwkbU00Z2oQqNgRv+eVV0r6oe27du7+sNW6KOMpwAqdwDh5cQQPuoAktYDCBZ3iFNydxXpx352PRWnKKmWP4A+fzB7A8j8k=</latexit><latexit sha1_base64=\"5QTnVRVSrnyzznVU7d5bF5u03Iw=\">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDbbTbt0swm7E7GE/ggvHhTx6u/x5r9x0+agrQ8GHu/NMDMvSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHJtRKwecJpwP6IjJULBKFqp0x9TzJ5mg2rNrbtzkFXiFaQGBZqD6ld/GLM04gqZpMb0PDdBP6MaBZN8VumnhieUTeiI9yxVNOLGz+bnzsiZVYYkjLUthWSu/p7IaGTMNApsZ0RxbJa9XPzP66UYXvuZUEmKXLHFojCVBGOS/06GQnOGcmoJZVrYWwkbU00Z2oQqNgRv+eVV0r6oe27du7+sNW6KOMpwAqdwDh5cQQPuoAktYDCBZ3iFNydxXpx352PRWnKKmWP4A+fzB7A8j8k=</latexit><latexit sha1_base64=\"5QTnVRVSrnyzznVU7d5bF5u03Iw=\">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDbbTbt0swm7E7GE/ggvHhTx6u/x5r9x0+agrQ8GHu/NMDMvSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHJtRKwecJpwP6IjJULBKFqp0x9TzJ5mg2rNrbtzkFXiFaQGBZqD6ld/GLM04gqZpMb0PDdBP6MaBZN8VumnhieUTeiI9yxVNOLGz+bnzsiZVYYkjLUthWSu/p7IaGTMNApsZ0RxbJa9XPzP66UYXvuZUEmKXLHFojCVBGOS/06GQnOGcmoJZVrYWwkbU00Z2oQqNgRv+eVV0r6oe27du7+sNW6KOMpwAqdwDh5cQQPuoAktYDCBZ3iFNydxXpx352PRWnKKmWP4A+fzB7A8j8k=</latexit>\nReferenceFigure 1: Illustration of the computation of the recall metric RBERT. Given the reference xand\ncandidate \u02c6x, we compute BERT embeddings and pairwise cosine similarity. We highlight the greedy\nmatching in red, and include the optional idfimportance weighting.\nWe experiment with different models (Section 4), using the tokenizer provided with each model.\nGiven a tokenized reference sentence x=hx1,...,x ki, the embedding model generates a se-\nquence of vectors hx1,...,xki. Similarly, the tokenized candidate \u02c6x=h\u02c6x1,..., \u02c6xmiis mapped\ntoh\u02c6x1,...,\u02c6xli. The main model we use is BERT, which tokenizes the input text into a sequence\nof word pieces (Wu et al., 2016), where unknown words are split into several commonly observed\nsequences of characters. The representation for each word piece is computed with a Transformer\nencoder (Vaswani et al., 2017) by repeatedly applying self-attention and nonlinear transformations\nin an alternating fashion. BERT embeddings have been shown to bene\ufb01t various NLP tasks (Devlin\net al., 2019; Liu, 2019; Huang et al., 2019; Yang et al., 2019a).\nSimilarity Measure The vector representation allows for a soft measure of similarity instead of\nexact-string (Papineni et al., 2002) or heuristic (Banerjee & Lavie, 2005) matching. The cosine\nsimilarity of a reference token xiand a candidate token \u02c6xjisx>\ni\u02c6xj\nkxikk\u02c6xjk. We use pre-normalized\nvectors, which reduces this calculation to the inner product x>\ni\u02c6xj. While this measure considers\ntokens in isolation, the contextual embeddings contain information from the rest of the sentence.\nBERTS CORE The complete score matches each token in xto a token in \u02c6xto compute recall,\nand each token in \u02c6xto a token in xto compute precision. We use greedy matching to maximize\nthe matching similarity score,2where each token is matched to the most similar token in the other\nsentence. We combine precision and recall to compute an F1 measure. For a reference xand\ncandidate \u02c6x, the recall, precision, and F1 scores are:\nRBERT =1\n|x|X\nxi2xmax\n\u02c6xj2\u02c6xx>\ni\u02c6xj,P BERT =1\n|\u02c6x|X\n\u02c6xj2\u02c6xmax\nxi2xx>\ni\u02c6xj,F BERT =2PBERT \u00b7RBERT\nPBERT +RBERT.\nImportance Weighting Previous work on similarity measures demonstrated that rare words can\nbe more indicative for sentence similarity than common words (Banerjee & Lavie, 2005; Vedantam\net al., 2015). BERTS CORE enables us to easily incorporate importance weighting. We experiment\nwith inverse document frequency ( idf) scores computed from the test corpus. Given Mreference\nsentences {x(i)}M\ni=1, the idfscore of a word-piece token wis\nidf(w)=\u0000log1\nMMX\ni=1I[w2x(i)],\nwhere I[\u00b7]is an indicator function. We do not use the full tf-idfmeasure because we process single\nsentences, where the term frequency ( tf) is likely 1. For example, recall with idfweighting is\nRBERT =P\nxi2xidf(xi) max \u02c6xj2\u02c6xx>\ni\u02c6xjP\nxi2xidf(xi).\nBecause we use reference sentences to compute idf, the idfscores remain the same for all systems\nevaluated on a speci\ufb01c test set. We apply plus-one smoothing to handle unknown word pieces.\n2We compare greedy matching with optimal assignment in Appendix C.\n4\nFigure 13.11 The computation of BERTS CORE recall from reference xand candidate \u02c6 x, from Figure 1 in\nZhang et al. (2020). This version shows an extended version of the metric in which tokens are also weighted by\ntheir idf values.\n13.7 Bias and Ethical Issues\nMachine translation raises many of the same ethical issues that we\u2019ve discussed in\nearlier chapters. For example, consider MT systems translating from Hungarian\n(which has the gender neutral pronoun \u02ddo) or Spanish (which often drops pronouns)\ninto English (in which pronouns are obligatory, and they have grammatical gender).\nWhen translating a reference to a person described without speci\ufb01ed gender, MT\nsystems often default to male gender (Schiebinger 2014, Prates et al. 2019). And\nMT systems often assign gender according to culture stereotypes of the sort we saw\nin Section ??. Fig. 13.12 shows examples from Prates et al. (2019), in which Hun-\ngarian gender-neutral \u02ddo is a nurse is translated with she, but gender-neutral \u02ddo is a\nCEO is translated with he. Prates et al. (2019) \ufb01nd that these stereotypes can\u2019t com-\npletely be accounted for by gender bias in US labor statistics, because the biases are 13.8 \u2022 S UMMARY 23\nampli\ufb01ed by MT systems, with pronouns being mapped to male or female gender\nwith a probability higher than if the mapping was based on actual labor employment\nstatistics.\nHungarian (gender neutral) source English MT output\n\u02ddo egy \u00b4apol\u00b4o she is a nurse\n\u02ddo egy tud \u00b4os he is a scientist\n\u02ddo egy m \u00b4ern\u00a8ok he is an engineer\n\u02ddo egy p \u00b4ek he is a baker\n\u02ddo egy tan \u00b4ar she is a teacher\n\u02ddo egy esk \u00a8uv\u02ddoszervez \u02ddo she is a wedding organizer\n\u02ddo egy vez \u00b4erigazgat \u00b4o he is a CEO\nFigure 13.12 When translating from gender-neutral languages like Hungarian into English,\ncurrent MT systems interpret people from traditionally male-dominated occupations as male,\nand traditionally female-dominated occupations as female (Prates et al., 2019).\nSimilarly, a recent challenge set, the WinoMT dataset (Stanovsky et al., 2019)\nshows that MT systems perform worse when they are asked to translate sentences\nthat describe people with non-stereotypical gender roles, like \u201cThe doctor asked the\nnurse to help her in the operation\u201d.\nMany ethical questions in MT require further research. One open problem is\ndeveloping metrics for knowing what our systems don\u2019t know. This is because MT\nsystems can be used in urgent situations where human translators may be unavailable\nor delayed: in medical domains, to help translate when patients and doctors don\u2019t\nspeak the same language, or in legal domains, to help judges or lawyers communi-\ncate with witnesses or defendants. In order to \u2018do no harm\u2019, systems need ways to\nassign con\ufb01dence values to candidate translations, so they can abstain from giving con\ufb01dence\nincorrect translations that may cause harm.\n13.8 Summary\nMachine translation is one of the most widely used applications of NLP, and the\nencoder-decoder model, \ufb01rst developed for MT is a key tool that has applications\nthroughout NLP.\n\u2022 Languages have divergences , both structural and lexical, that make translation\ndif\ufb01cult.\n\u2022 The linguistic \ufb01eld of typology investigates some of these differences; lan-\nguages can be classi\ufb01ed by their position along typological dimensions like\nwhether verbs precede their objects.\n\u2022Encoder-decoder networks (for transformers just as we saw in Chapter 8 for\nRNNs) are composed of an encoder network that takes an input sequence\nand creates a contextualized representation of it, the context . This context\nrepresentation is then passed to a decoder which generates a task-speci\ufb01c\noutput sequence.\n\u2022Cross-attention allows the transformer decoder to view information from all\nthe hidden states of the encoder.\n\u2022 Machine translation models are trained on a parallel corpus , sometimes called\nabitext , a text that appears in two (or more) languages. 24 CHAPTER 13 \u2022 M ACHINE TRANSLATION\n\u2022Backtranslation is a way of making use of monolingual corpora in the target\nlanguage by running a pilot MT engine backwards to create synthetic bitexts.\n\u2022 MT is evaluated by measuring a translation\u2019s adequacy (how well it captures\nthe meaning of the source sentence) and \ufb02uency (how \ufb02uent or natural it is\nin the target language). Human evaluation is the gold standard, but automatic\nevaluation metrics like chrF , which measure character n-gram overlap with\nhuman translations, or more recent metrics based on embedding similarity,\nare also commonly used.\nBibliographical and Historical Notes\nMT was proposed seriously by the late 1940s, soon after the birth of the computer\n(Weaver, 1949/1955). In 1954, the \ufb01rst public demonstration of an MT system pro-\ntotype (Dostert, 1955) led to great excitement in the press (Hutchins, 1997). The\nnext decade saw a great \ufb02owering of ideas, pre\ufb01guring most subsequent develop-\nments. But this work was ahead of its time\u2014implementations were limited by, for\nexample, the fact that pending the development of disks there was no good way to\nstore dictionary information.\nAs high-quality MT proved elusive (Bar-Hillel, 1960), there grew a consensus\non the need for better evaluation and more basic research in the new \ufb01elds of for-\nmal and computational linguistics. This consensus culminated in the famously crit-\nical ALPAC (Automatic Language Processing Advisory Committee) report of 1966\n(Pierce et al., 1966) that led in the mid 1960s to a dramatic cut in funding for MT\nin the US. As MT research lost academic respectability, the Association for Ma-\nchine Translation and Computational Linguistics dropped MT from its name. Some\nMT developers, however, persevered, and there were early MT systems like M \u00b4et\u00b4eo,\nwhich translated weather forecasts from English to French (Chandioux, 1976), and\nindustrial systems like Systran.\nIn the early years, the space of MT architectures spanned three general mod-\nels. In direct translation , the system proceeds word-by-word through the source-\nlanguage text, translating each word incrementally. Direct translation uses a large\nbilingual dictionary, each of whose entries is a small program with the job of trans-\nlating one word. In transfer approaches, we \ufb01rst parse the input text and then ap-\nply rules to transform the source-language parse into a target language parse. We\nthen generate the target language sentence from the parse tree. In interlingua ap-\nproaches, we analyze the source language text into some abstract meaning repre-\nsentation, called an interlingua . We then generate into the target language from\nthis interlingual representation. A common way to visualize these three early ap-\nproaches was the Vauquois triangle shown in Fig. 13.13. The triangle shows theVauquois\ntriangle\nincreasing depth of analysis required (on both the analysis and generation end) as\nwe move from the direct approach through transfer approaches to interlingual ap-\nproaches. In addition, it shows the decreasing amount of transfer knowledge needed\nas we move up the triangle, from huge amounts of transfer at the direct level (al-\nmost all knowledge is transfer knowledge for each word) through transfer (transfer\nrules only for parse trees or thematic roles) through interlingua (no speci\ufb01c transfer\nknowledge). We can view the encoder-decoder network as an interlingual approach,\nwith attention acting as an integration of direct and transfer, allowing words or their\nrepresentations to be directly accessed by the decoder. BIBLIOGRAPHICAL AND HISTORICAL NOTES 25\nsourcetexttarget textDirect TranslationTransferInterlinguaSource Text:Semantic/SyntacticStructureTarget Text:Semantic/SyntacticStructuresource languageanalysissource languageanalysistarget language generation\nFigure 13.13 The Vauquois (1968) triangle.\nStatistical methods began to be applied around 1990, enabled \ufb01rst by the devel-\nopment of large bilingual corpora like the Hansard corpus of the proceedings of the\nCanadian Parliament, which are kept in both French and English, and then by the\ngrowth of the Web. Early on, a number of researchers showed that it was possible\nto extract pairs of aligned sentences from bilingual corpora, using words or simple\ncues like sentence length (Kay and R \u00a8oscheisen 1988, Gale and Church 1991, Gale\nand Church 1993, Kay and R \u00a8oscheisen 1993).\nAt the same time, the IBM group, drawing directly on the noisy channel model\nfor speech recognition, proposed two related paradigms for statistical MT . These statistical MT\ninclude the generative algorithms that became known as IBM Models 1 through IBM Models\n5, implemented in the Candide system. The algorithms (except for the decoder) Candide\nwere published in full detail\u2014 encouraged by the US government who had par-\ntially funded the work\u2014 which gave them a huge impact on the research community\n(Brown et al. 1990, Brown et al. 1993).\nThe group also developed a discriminative approach, called MaxEnt (for maxi-\nmum entropy, an alternative formulation of logistic regression), which allowed many\nfeatures to be combined discriminatively rather than generatively (Berger et al.,\n1996), which was further developed by Och and Ney (2002).\nBy the turn of the century, most academic research on machine translation used\nstatistical MT, either in the generative or discriminative mode. An extended version\nof the generative approach, called phrase-based translation was developed, basedphrase-based\ntranslation\non inducing translations for phrase-pairs (Och 1998, Marcu and Wong 2002, Koehn\net al. (2003), Och and Ney 2004, Deng and Byrne 2005, inter alia).\nOnce automatic metrics like BLEU were developed (Papineni et al., 2002), the\ndiscriminative log linear formulation (Och and Ney, 2004), drawing from the IBM\nMaxEnt work (Berger et al., 1996), was used to directly optimize evaluation metrics\nlike BLEU in a method known as Minimum Error Rate Training , orMERT (Och, MERT\n2003), also drawing from speech recognition models (Chou et al., 1993). Toolkits\nlike GIZA (Och and Ney, 2003) and Moses (Koehn et al. 2006, Zens and Ney 2007) Moses\nwere widely used.\nThere were also approaches around the turn of the century that were based on\nsyntactic structure (Chapter 18). Models based on transduction grammars (alsotransduction\ngrammars\ncalled synchronous grammars assign a parallel syntactic tree structure to a pair of\nsentences in different languages, with the goal of translating the sentences by ap-\nplying reordering operations on the trees. From a generative perspective, we can\nview a transduction grammar as generating pairs of aligned sentences in two lan-\nguages. Some of the most widely used models included the inversion transduction\ngrammar (Wu, 1996) and synchronous context-free grammars (Chiang, 2005),inversion\ntransduction\ngrammar 26 CHAPTER 13 \u2022 M ACHINE TRANSLATION\nNeural networks had been applied at various times to various aspects of machine\ntranslation; for example Schwenk et al. (2006) showed how to use neural language\nmodels to replace n-gram language models in a Spanish-English system based on\nIBM Model 4. The modern neural encoder-decoder approach was pioneered by\nKalchbrenner and Blunsom (2013), who used a CNN encoder and an RNN decoder,\nand was \ufb01rst applied to MT by Bahdanau et al. (2015). The transformer encoder-\ndecoder was proposed by Vaswani et al. (2017) (see the History section of Chap-\nter 9).\nResearch on evaluation of machine translation began quite early. Miller and\nBeebe-Center (1956) proposed a number of methods drawing on work in psycholin-\nguistics. These included the use of cloze and Shannon tasks to measure intelligibility\nas well as a metric of edit distance from a human translation, the intuition that un-\nderlies all modern overlap-based automatic evaluation metrics. The ALPAC report\nincluded an early evaluation study conducted by John Carroll that was extremely in-\n\ufb02uential (Pierce et al., 1966, Appendix 10). Carroll proposed distinct measures for\n\ufb01delity and intelligibility, and had raters score them subjectively on 9-point scales.\nMuch early evaluation work focuses on automatic word-overlap metrics like BLEU\n(Papineni et al., 2002), NIST (Doddington, 2002), TER (Translation Error Rate)\n(Snover et al., 2006), Precision and Recall (Turian et al., 2003), and METEOR\n(Banerjee and Lavie, 2005); character n-gram overlap methods like chrF (Popovi \u00b4c,\n2015) came later. More recent evaluation work, echoing the ALPAC report, has\nemphasized the importance of careful statistical methodology and the use of human\nevaluation (Kocmi et al., 2021; Marie et al., 2021).\nThe early history of MT is surveyed in Hutchins 1986 and 1997; Nirenburg et al.\n(2002) collects early readings. See Croft (1990) or Comrie (1989) for introductions\nto linguistic typology.\nExercises\n13.1 Compute by hand the chrF2,2 score for HYP2 on page 20 (the answer should\nround to .62). Exercises 27\nAnastasopoulos, A. and G. Neubig. 2020. Should all cross-\nlingual embeddings speak English? ACL.\nArtetxe, M. and H. Schwenk. 2019. Massively multilingual\nsentence embeddings for zero-shot cross-lingual transfer\nand beyond. TACL , 7:597\u2013610.\nBahdanau, D., K. H. Cho, and Y . Bengio. 2015. Neural ma-\nchine translation by jointly learning to align and translate.\nICLR 2015 .\nBanerjee, S. and A. Lavie. 2005. METEOR: An automatic\nmetric for MT evaluation with improved correlation with\nhuman judgments. Proceedings of ACL Workshop on In-\ntrinsic and Extrinsic Evaluation Measures for MT and/or\nSummarization .\nBa\u02dcn\u00b4on, M., P. Chen, B. Haddow, K. Hea\ufb01eld, H. Hoang,\nM. Espl `a-Gomis, M. L. Forcada, A. Kamran, F. Kirefu,\nP. Koehn, S. Ortiz Rojas, L. Pla Sempere, G. Ram \u00b4\u0131rez-\nS\u00b4anchez, E. Sarr \u00b4\u0131as, M. Strelec, B. Thompson, W. Waites,\nD. Wiggins, and J. Zaragoza. 2020. ParaCrawl: Web-\nscale acquisition of parallel corpora. ACL.\nBar-Hillel, Y . 1960. The present status of automatic transla-\ntion of languages. In F. Alt, ed., Advances in Computers\n1, 91\u2013163. Academic Press.\nBentivogli, L., M. Cettolo, M. Federico, and C. Federmann.\n2018. Machine translation human evaluation: an investi-\ngation of evaluation based on post-editing and its relation\nwith direct assessment. ICSLT .\nBerger, A., S. A. Della Pietra, and V . J. Della Pietra. 1996. A\nmaximum entropy approach to natural language process-\ning. Computational Linguistics , 22(1):39\u201371.\nBickel, B. 2003. Referential density in discourse and syntac-\ntic typology. Language , 79(2):708\u2013736.\nBostrom, K. and G. Durrett. 2020. Byte pair encoding is\nsuboptimal for language model pretraining. EMNLP .\nBrown, P. F., J. Cocke, S. A. Della Pietra, V . J. Della Pietra,\nF. Jelinek, J. D. Lafferty, R. L. Mercer, and P. S. Roossin.\n1990. A statistical approach to machine translation. Com-\nputational Linguistics , 16(2):79\u201385.\nBrown, P. F., S. A. Della Pietra, V . J. Della Pietra, and R. L.\nMercer. 1993. The mathematics of statistical machine\ntranslation: Parameter estimation. Computational Lin-\nguistics , 19(2):263\u2013311.\nCallison-Burch, C., M. Osborne, and P. Koehn. 2006. Re-\nevaluating the role of BLEU in machine translation re-\nsearch. EACL .\nChandioux, J. 1976. M \u00b4ET\u00b4EO: un syst `eme op \u00b4erationnel pour\nla traduction automatique des bulletins m \u00b4et\u00b4eorologiques\ndestin \u00b4es au grand public. Meta , 21:127\u2013133.\nChiang, D. 2005. A hierarchical phrase-based model for sta-\ntistical machine translation. ACL.\nChou, W., C.-H. Lee, and B. H. Juang. 1993. Minimum error\nrate training based on n-best string models. ICASSP .\nComrie, B. 1989. Language Universals and Linguistic Ty-\npology , 2nd edition. Blackwell.Costa-juss `a, M. R., J. Cross, O. C \u00b8 elebi, M. Elbayad,\nK. Hea\ufb01eld, K. Heffernan, E. Kalbassi, J. Lam, D. Licht,\nJ. Maillard, A. Sun, S. Wang, G. Wenzek, A. Youngblood,\nB. Akula, L. Barrault, G. M. Gonzalez, P. Hansanti,\nJ. Hoffman, S. Jarrett, K. R. Sadagopan, D. Rowe,\nS. Spruit, C. Tran, P. Andrews, N. F. Ayan, S. Bhosale,\nS. Edunov, A. Fan, C. Gao, V . Goswami, F. Guzm \u00b4an,\nP. Koehn, A. Mourachko, C. Ropers, S. Saleem,\nH. Schwenk, J. Wang, and NLLB Team. 2022. No\nlanguage left behind: Scaling human-centered machine\ntranslation. ArXiv.\nCroft, W. 1990. Typology and Universals . Cambridge Uni-\nversity Press.\nDeng, Y . and W. Byrne. 2005. HMM word and phrase align-\nment for statistical machine translation. HLT-EMNLP .\nDoddington, G. 2002. Automatic evaluation of machine\ntranslation quality using n-gram co-occurrence statistics.\nHLT.\nDorr, B. 1994. Machine translation divergences: A formal\ndescription and proposed solution. Computational Lin-\nguistics , 20(4):597\u2013633.\nDostert, L. 1955. The Georgetown-I.B.M. experiment. In\nMachine Translation of Languages: Fourteen Essays ,\n124\u2013135. MIT Press.\nDryer, M. S. and M. Haspelmath, eds. 2013. The World Atlas\nof Language Structures Online . Max Planck Institute for\nEvolutionary Anthropology, Leipzig. Available online at\nhttp://wals.info .\nEdunov, S., M. Ott, M. Auli, and D. Grangier. 2018. Under-\nstanding back-translation at scale. EMNLP .\nFan, A., S. Bhosale, H. Schwenk, Z. Ma, A. El-Kishky,\nS. Goyal, M. Baines, O. Celebi, G. Wenzek, V . Chaud-\nhary, N. Goyal, T. Birch, V . Liptchinsky, S. Edunov,\nM. Auli, and A. Joulin. 2021. Beyond english-centric\nmultilingual machine translation. JMLR , 22(107):1\u201348.\n8, W. Nekoto, V . Marivate, T. Matsila, T. Fasubaa,\nT. Kolawole, T. Fagbohungbe, S. O. Akinola, S. H.\nMuhammad, S. Kabongo, S. Osei, S. Freshia, R. A.\nNiyongabo, R. M. P. Ogayo, O. Ahia, M. Meressa,\nM. Adeyemi, M. Mokgesi-Selinga, L. Okegbemi, L. J.\nMartinus, K. Tajudeen, K. Degila, K. Ogueji, K. Siminyu,\nJ. Kreutzer, J. Webster, J. T. Ali, J. A. I. Orife,\nI. Ezeani, I. A. Dangana, H. Kamper, H. Elsahar, G. Duru,\nG. Kioko, E. Murhabazi, E. van Biljon, D. Whitenack,\nC. Onyefuluchi, C. Emezue, B. Dossou, B. Sibanda, B. I.\nBassey, A. Olabiyi, A. Ramkilowan, A. \u00a8Oktem, A. Ak-\ninfaderin, and A. Bashir. 2020. Participatory research\nfor low-resourced machine translation: A case study in\nAfrican languages. Findings of EMNLP .\nGale, W. A. and K. W. Church. 1991. A program for aligning\nsentences in bilingual corpora. ACL.\nGale, W. A. and K. W. Church. 1993. A program for aligning\nsentences in bilingual corpora. Computational Linguis-\ntics, 19:75\u2013102.\nGoel, V . and W. Byrne. 2000. Minimum bayes-risk auto-\nmatic speech recognition. Computer Speech & Language ,\n14(2):115\u2013135.\nHutchins, W. J. 1986. Machine Translation: Past, Present,\nFuture . Ellis Horwood, Chichester, England. 28 Chapter 13 \u2022 Machine Translation\nHutchins, W. J. 1997. From \ufb01rst conception to \ufb01rst demon-\nstration: The nascent years of machine translation, 1947\u2013\n1954. A chronology. Machine Translation , 12:192\u2013252.\nHutchins, W. J. and H. L. Somers. 1992. An Introduction to\nMachine Translation . Academic Press.\nKalchbrenner, N. and P. Blunsom. 2013. Recurrent continu-\nous translation models. EMNLP .\nKay, M. and M. R \u00a8oscheisen. 1988. Text-translation align-\nment. Technical Report P90-00143, Xerox Palo Alto Re-\nsearch Center, Palo Alto, CA.\nKay, M. and M. R \u00a8oscheisen. 1993. Text-translation align-\nment. Computational Linguistics , 19:121\u2013142.\nKocmi, T., C. Federmann, R. Grundkiewicz, M. Junczys-\nDowmunt, H. Matsushita, and A. Menezes. 2021. To ship\nor not to ship: An extensive evaluation of automatic met-\nrics for machine translation. ArXiv.\nKoehn, P. 2005. Europarl: A parallel corpus for statistical\nmachine translation. MT summit, vol. 5 .\nKoehn, P., H. Hoang, A. Birch, C. Callison-Burch, M. Fed-\nerico, N. Bertoldi, B. Cowan, W. Shen, C. Moran,\nR. Zens, C. Dyer, O. Bojar, A. Constantin, and E. Herbst.\n2006. Moses: Open source toolkit for statistical machine\ntranslation. ACL.\nKoehn, P., F. J. Och, and D. Marcu. 2003. Statistical phrase-\nbased translation. HLT-NAACL .\nKreutzer, J., I. Caswell, L. Wang, A. Wahab, D. van Esch,\nN. Ulzii-Orshikh, A. Tapo, N. Subramani, A. Sokolov,\nC. Sikasote, M. Setyawan, S. Sarin, S. Samb, B. Sagot,\nC. Rivera, A. Rios, I. Papadimitriou, S. Osei, P. O.\nSuarez, I. Orife, K. Ogueji, A. N. Rubungo, T. Q. Nguyen,\nM. M \u00a8uller, A. M \u00a8uller, S. H. Muhammad, N. Muhammad,\nA. Mnyakeni, J. Mirzakhalov, T. Matangira, C. Leong,\nN. Lawson, S. Kudugunta, Y . Jernite, M. Jenny, O. Firat,\nB. F. P. Dossou, S. Dlamini, N. de Silva, S. C \u00b8 abuk Ball\u0131,\nS. Biderman, A. Battisti, A. Baruwa, A. Bapna, P. Bal-\njekar, I. A. Azime, A. Awokoya, D. Ataman, O. Ahia,\nO. Ahia, S. Agrawal, and M. Adeyemi. 2022. Quality at\na glance: An audit of web-crawled multilingual datasets.\nTACL , 10:50\u201372.\nKudo, T. 2018. Subword regularization: Improving neural\nnetwork translation models with multiple subword candi-\ndates. ACL.\nKudo, T. and J. Richardson. 2018. SentencePiece: A simple\nand language independent subword tokenizer and detok-\nenizer for neural text processing. EMNLP .\nKumar, S. and W. Byrne. 2004. Minimum Bayes-risk decod-\ning for statistical machine translation. HLT-NAACL .\nLan, Z., M. Chen, S. Goodman, K. Gimpel, P. Sharma,\nand R. Soricut. 2020. ALBERT: A lite BERT for self-\nsupervised learning of language representations. ICLR .\nLison, P. and J. Tiedemann. 2016. Opensubtitles2016: Ex-\ntracting large parallel corpora from movie and tv subti-\ntles. LREC .\nLowerre, B. T. 1976. The Harpy Speech Recognition System .\nPh.D. thesis, Carnegie Mellon University, Pittsburgh, PA.\nMarcu, D. and W. Wong. 2002. A phrase-based, joint proba-\nbility model for statistical machine translation. EMNLP .\nMarie, B., A. Fujita, and R. Rubino. 2021. Scienti\ufb01c credi-\nbility of machine translation research: A meta-evaluation\nof 769 papers. ACL.McLuhan, M. 1964. Understanding Media: The Extensions\nof Man . New American Library.\nMiller, G. A. and J. G. Beebe-Center. 1956. Some psycho-\nlogical methods for evaluating the quality of translations.\nMechanical Translation , 3:73\u201380.\nNirenburg, S., H. L. Somers, and Y . Wilks, eds. 2002. Read-\nings in Machine Translation . MIT Press.\nOch, F. J. 1998. Ein beispielsbasierter und statis-\ntischer Ansatz zum maschinellen Lernen von\nnat\u00a8urlichsprachlicher \u00a8Ubersetzung . Ph.D. thesis, Uni-\nversit \u00a8at Erlangen-N \u00a8urnberg, Germany. Diplomarbeit\n(diploma thesis).\nOch, F. J. 2003. Minimum error rate training in statistical\nmachine translation. ACL.\nOch, F. J. and H. Ney. 2002. Discriminative training and\nmaximum entropy models for statistical machine transla-\ntion. ACL.\nOch, F. J. and H. Ney. 2003. A systematic comparison of\nvarious statistical alignment models. Computational Lin-\nguistics , 29(1):19\u201351.\nOch, F. J. and H. Ney. 2004. The alignment template ap-\nproach to statistical machine translation. Computational\nLinguistics , 30(4):417\u2013449.\nPapineni, K., S. Roukos, T. Ward, and W.-J. Zhu. 2002. Bleu:\nA method for automatic evaluation of machine transla-\ntion. ACL.\nPierce, J. R., J. B. Carroll, E. P. Hamp, D. G. Hays, C. F.\nHockett, A. G. Oettinger, and A. J. Perlis. 1966. Lan-\nguage and Machines: Computers in Translation and Lin-\nguistics . ALPAC report. National Academy of Sciences,\nNational Research Council, Washington, DC.\nPopovi \u00b4c, M. 2015. chrF: character n-gram F-score for auto-\nmatic MT evaluation. Proceedings of the Tenth Workshop\non Statistical Machine Translation .\nPost, M. 2018. A call for clarity in reporting BLEU scores.\nWMT 2018 .\nPrates, M. O. R., P. H. Avelar, and L. C. Lamb. 2019. Assess-\ning gender bias in machine translation: a case study with\nGoogle Translate. Neural Computing and Applications ,\n32:6363\u20136381.\nRaffel, C., N. Shazeer, A. Roberts, K. Lee, S. Narang,\nM. Matena, Y . Zhou, W. Li, and P. J. Liu. 2020. Exploring\nthe limits of transfer learning with a uni\ufb01ed text-to-text\ntransformer. JMLR , 21(140):1\u201367.\nRei, R., C. Stewart, A. C. Farinha, and A. Lavie.\n2020. COMET: A neural framework for MT evaluation.\nEMNLP .\nSchiebinger, L. 2014. Scienti\ufb01c research must take gender\ninto account. Nature , 507(7490):9.\nSchwenk, H. 2018. Filtering and mining parallel data in a\njoint multilingual space. ACL.\nSchwenk, H., D. Dechelotte, and J.-L. Gauvain. 2006. Con-\ntinuous space language models for statistical machine\ntranslation. COLING/ACL .\nSchwenk, H., G. Wenzek, S. Edunov, E. Grave, A. Joulin,\nand A. Fan. 2021. CCMatrix: Mining billions of high-\nquality parallel sentences on the web. ACL.\nSellam, T., D. Das, and A. Parikh. 2020. BLEURT: Learning\nrobust metrics for text generation. ACL. Exercises 29\nSlobin, D. I. 1996. Two ways to travel. In M. Shibatani and\nS. A. Thompson, eds, Grammatical Constructions: Their\nForm and Meaning , 195\u2013220. Clarendon Press.\nSnover, M., B. Dorr, R. Schwartz, L. Micciulla, and\nJ. Makhoul. 2006. A study of translation edit rate with\ntargeted human annotation. AMTA-2006 .\nStanovsky, G., N. A. Smith, and L. Zettlemoyer. 2019. Eval-\nuating gender bias in machine translation. ACL.\nStolcke, A., Y . Konig, and M. Weintraub. 1997. Explicit\nword error minimization in N-best list rescoring. EU-\nROSPEECH , volume 1.\nSuzgun, M., L. Melas-Kyriazi, and D. Jurafsky. 2023. Fol-\nlow the wisdom of the crowd: Effective text generation\nvia minimum Bayes risk decoding. Findings of ACL\n2023 .\nTalmy, L. 1985. Lexicalization patterns: Semantic structure\nin lexical forms. In T. Shopen, ed., Language Typology\nand Syntactic Description, Volume 3 . Cambridge Univer-\nsity Press. Originally appeared as UC Berkeley Cognitive\nScience Program Report No. 30, 1980.\nTalmy, L. 1991. Path to realization: A typology of event\ncon\ufb02ation. BLS-91 .\nThompson, B. and P. Koehn. 2019. Vecalign: Improved sen-\ntence alignment in linear time and space. EMNLP .\nTurian, J. P., L. Shen, and I. D. Melamed. 2003. Evaluation\nof machine translation and its evaluation. Proceedings of\nMT Summit IX .\nVaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones,\nA. N. Gomez, \u0141. Kaiser, and I. Polosukhin. 2017. Atten-\ntion is all you need. NeurIPS .\nVauquois, B. 1968. A survey of formal grammars and al-\ngorithms for recognition and transformation in machine\ntranslation. IFIP Congress 1968 .\nWeaver, W. 1949/1955. Translation. In W. N. Locke and\nA. D. Boothe, eds, Machine Translation of Languages ,\n15\u201323. MIT Press. Reprinted from a memorandum writ-\nten by Weaver in 1949.\nWu, D. 1996. A polynomial-time algorithm for statistical\nmachine translation. ACL.\nWu, Y ., M. Schuster, Z. Chen, Q. V . Le, M. Norouzi,\nW. Macherey, M. Krikun, Y . Cao, Q. Gao, K. Macherey,\nJ. Klingner, A. Shah, M. Johnson, X. Liu, \u0141. Kaiser,\nS. Gouws, Y . Kato, T. Kudo, H. Kazawa, K. Stevens,\nG. Kurian, N. Patil, W. Wang, C. Young, J. Smith,\nJ. Riesa, A. Rudnick, O. Vinyals, G. S. Corrado,\nM. Hughes, and J. Dean. 2016. Google\u2019s neural machine\ntranslation system: Bridging the gap between human and\nmachine translation. ArXiv preprint arXiv:1609.08144.\nZens, R. and H. Ney. 2007. Ef\ufb01cient phrase-table represen-\ntation for machine translation with applications to online\nMT and speech translation. NAACL-HLT .\nZhang, T., V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi.\n2020. BERTscore: Evaluating text generation with\nBERT. ICLR 2020 .\nZiemski, M., M. Junczys-Dowmunt, and B. Pouliquen. 2016.\nThe United Nations parallel corpus v1.0. LREC .", "14.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\n14Question Answering, Informa-\ntion Retrieval, and Retrieval-\nAugmented Generation\nPeople need to know things. So pretty much as soon as there were computers we\nwere asking them questions. Systems in the early 1960s were answering questions\nabout baseball statistics and scienti\ufb01c facts. Even \ufb01ctional computers in the 1970s\nlike Deep Thought, invented by Douglas Adams in The Hitchhiker\u2019s Guide to the\nGalaxy , answered \u201cthe Ultimate Question Of Life, The Universe, and Everything\u201d.1\nAnd because so much knowledge is encoded in text, QA systems were performing\nat human levels even before LLMs: IBM\u2019s Watson system won the TV game-show\nJeopardy! in 2011, surpassing humans at answering questions like:\nWILLIAM WILKINSON\u2019S \u201cAN ACCOUNT OF THE PRINCIPALITIES OF WALLACHIA AND MOLDOVIA\u201dINSPIRED THIS AUTHOR\u2019S MOST FAMOUS NOVEL\n2\nQuestion answering systems are designed to \ufb01ll human information needs .\nSince a lot of information is present in text form (on the web or in other data like\nour email, or books), question answering is closely related to the task behind search\nengines. Indeed, the distinction is becoming ever more fuzzy, as modern search\nengines are integrated with large language models trained to do question answering.\nQuestion answering systems often focus on a useful subset of information needs:\nfactoid questions , questions of fact or reasoning that can be answered with simple\nfacts expressed in short or medium-length texts, like the following:\n(14.1) Where is the Louvre Museum located?\n(14.2) Where does the energy in a nuclear explosion come from?\n(14.3) How to get a script l in latex?\nModern NLP systems answer these questions using large language models, in\none of two ways. The \ufb01rst is to make use of the method from Chapter 12: prompt\na pretrained and instruction-tuned LLM, an LLM that has been \ufb01netuned on ques-\ntion/answer datasets with the question in the prompt. For example, we could prompt\na causal language model with a string like\nQ: Where is the Louvre Museum located? A:\nhave it do conditional generation given this pre\ufb01x, and take the response as the an-\nswer. The idea is that language models have read a lot of facts in their pretraining\ndata, presumably including the location of the Louvre, and have encoded this infor-\nmation in their parameters.\nSimply prompting an LLM can be a useful approach to answer many factoid\nquestions. But it is not yet a complete solution for question answering.\n1The answer was 42, but unfortunately the question was never revealed.\n2The answer, of course, is \u2018Who is Bram Stoker\u2019, and the novel was Dracula . 2CHAPTER 14 \u2022 Q UESTION ANSWERING , INFORMATION RETRIEVAL ,AND RAG\nThe \ufb01rst and main problem is that large language models often give the wrong\nanswer! Large language models hallucinate . A hallucination is a response that is hallucinate\nnot faithful to the facts of the world. That is, when asked questions, large language\nmodels simply make up answers that sound reasonable. For example, Dahl et al.\n(2024) found that when asked questions about the legal domain (like about particular\nlegal cases), large language models hallucinated from 69% to 88% of the time!\nAnd it\u2019s not always possible to tell when language models are hallucinating,\npartly because LLMs aren\u2019t well- calibrated . In a calibrated system, the con\ufb01dence calibrated\nof a system in the correctness of its answer is highly correlated with the probability\nof an answer being correct. So if a calibrated system is wrong, at least it might hedge\nits answer or tell us to go check another source. But since language models are not\nwell-calibrated, they often give a very wrong answer with complete certainty (Zhou\net al., 2024).\nA second problem is that simply prompting a large language model doesn\u2019t allow\nus to ask questions about proprietary data. A common use of question answering is\nabout data like our personal email or medical records. Or a company may have\ninternal documents that contain answers for customer service or internal use. Or\nlegal \ufb01rms need to ask questions about legal discovery from proprietary documents.\nFinally, static large language models also have problems with questions about\nrapidly changing information (like questions about something that happened last\nweek) since LLMs won\u2019t have up-to-date information from after their release data.\nFor this reason the most common way to do question-answering with LLMs is\nretrieval-augmented generation orRAG , and that is the method we will focus on RAG\nin this chapter. In RAG we use information retrieval (IR) techniques to retrieveinformation\nretrieval\ndocuments that are likely to have information that might help answer the question.\nThen we use a large language model to generate an answer given these documents.\nBasing our answers on retrieved documents can solve some of the problems with\nusing simple prompting to answer questions. First, it helps ensure that the answer is\ngrounded in facts from some curated dataset. And the system can give the user the\nanswer accompanied by the context of the passage or document the answer came\nfrom. This information can help users have con\ufb01dence in the accuracy of the answer\n(or help them spot when it is wrong!). And these retrieval techniques can be used on\nany proprietary data we want, such as legal or medical data for those applications.\nWe\u2019ll begin by introducing information retrieval, the task of choosing the most\nrelevant document from a document set given a user\u2019s query expressing their infor-\nmation need. We\u2019ll see the classic method based on cosines of sparse tf-idf vectors,\na modern neural \u2018dense\u2019 retrievers based on instead representing queries and docu-\nments neurally with BERT or other language models. We then introduce retriever-\nbased question answering and the retrieval-augmented generation paradigm.\nFinally, we\u2019ll discuss various QA datasets. These are used for \ufb01netuning LLMs\nin instruction tuning, as we saw in Chapter 12. And they are also used as bench-\nmarks, since question answering has an important function as a benchmark for mea-\nsuring the abilities of language models.\n14.1 Information Retrieval\nInformation retrieval orIRis the name of the \ufb01eld encompassing the retrieval of allinformation\nretrieval\nIR manner of media based on user information needs. The resulting IR system is often\ncalled a search engine . Our goal in this section is to give a suf\ufb01cient overview of IR 14.1 \u2022 I NFORMATION RETRIEVAL 3\nto see its application to question answering. Readers with more interest speci\ufb01cally\nin information retrieval should see the Historical Notes section at the end of the\nchapter and textbooks like Manning et al. (2008).\nThe IR task we consider is called ad hoc retrieval , in which a user poses a ad hoc retrieval\nquery to a retrieval system, which then returns an ordered set of documents from\nsome collection . Adocument refers to whatever unit of text the system indexes and document\nretrieves (web pages, scienti\ufb01c papers, news articles, or even shorter passages like\nparagraphs). A collection refers to a set of documents being used to satisfy user collection\nrequests. A term refers to a word in a collection, but it may also include phrases. term\nFinally, a query represents a user\u2019s information need expressed as a set of terms. query\nThe high-level architecture of an ad hoc retrieval engine is shown in Fig. 14.1.\nDocumentDocumentDocumentDocumentDocumentDocumentQuery ProcessingIndexingSearchDocumentDocumentDocumentDocumentDocumentRanked DocumentsDocumentqueryInvertedIndexqueryvectordocument collection\nFigure 14.1 The architecture of an ad hoc IR system.\nThe basic IR architecture uses the vector space model we introduced in Chap-\nter 6, in which we map queries and document to vectors based on unigram word\ncounts, and use the cosine similarity between the vectors to rank potential documents\n(Salton, 1971). This is thus an example of the bag-of-words model introduced in\nChapter 4, since words are considered independently of their positions.\n14.1.1 Term weighting and document scoring\nLet\u2019s look at the details of how the match between a document and query is scored.\nWe don\u2019t use raw word counts in IR, instead computing a term weight for each term weight\ndocument word. Two term weighting schemes are common: the tf-idf weighting\nintroduced in Chapter 6, and a slightly more powerful variant called BM25 . BM25\nWe\u2019ll reintroduce tf-idf here so readers don\u2019t need to look back at Chapter 6.\nTf-idf (the \u2018-\u2019 here is a hyphen, not a minus sign) is the product of two terms, the\nterm frequency tfand the inverse document frequency idf.\nThe term frequency tells us how frequent the word is; words that occur more\noften in a document are likely to be informative about the document\u2019s contents. We\nusually use the log 10of the word frequency, rather than the raw count. The intuition\nis that a word appearing 100 times in a document doesn\u2019t make that word 100 times\nmore likely to be relevant to the meaning of the document. We also need to do\nsomething special with counts of 0, since we can\u2019t take the log of 0.3\ntft;d=(\n1+log10count (t;d) if count (t;d)>0\n0 otherwise(14.4)\n3We can also use this alternative formulation, which we have used in earlier editions: tf t;d=\nlog10(count (t;d)+1) 4CHAPTER 14 \u2022 Q UESTION ANSWERING , INFORMATION RETRIEVAL ,AND RAG\nIf we use log weighting, terms which occur 0 times in a document would have tf =0,\n1 times in a document tf =1+log10(1) =1+0=1, 10 times in a document tf =\n1+log10(10) =2, 100 times tf =1+log10(100) =3, 1000 times tf =4, and so on.\nThedocument frequency dftof a term tis the number of documents it oc-\ncurs in. Terms that occur in only a few documents are useful for discriminating\nthose documents from the rest of the collection; terms that occur across the entire\ncollection aren\u2019t as helpful. The inverse document frequency oridfterm weight\n(Sparck Jones, 1972) is de\ufb01ned as:\nidft=log10N\ndft(14.5)\nwhere Nis the total number of documents in the collection, and df tis the number\nof documents in which term toccurs. The fewer documents in which a term occurs,\nthe higher this weight; the lowest weight of 0 is assigned to terms that occur in every\ndocument.\nHere are some idf values for some words in the corpus of Shakespeare plays,\nranging from extremely informative words that occur in only one play like Romeo ,\nto those that occur in a few like salad orFalstaff , to those that are very common like\nfoolor so common as to be completely non-discriminative since they occur in all 37\nplays like good orsweet .4\nWord df idf\nRomeo 1 1.57\nsalad 2 1.27\nFalstaff 4 0.967\nforest 12 0.489\nbattle 21 0.246\nwit 34 0.037\nfool 36 0.012\ngood 37 0\nsweet 37 0\nThetf-idf value for word tin document dis then the product of term frequency\ntft;dand IDF:\ntf-idf(t;d) =tft;d\u0001idft (14.6)\n14.1.2 Document Scoring\nWe score document dby the cosine of its vector dwith the query vector q:\nscore(q;d) =cos(q;d) =q\u0001d\njqjjdj(14.7)\nAnother way to think of the cosine computation is as the dot product of unit vectors;\nwe \ufb01rst normalize both the query and document vector to unit vectors, by dividing\nby their lengths, and then take the dot product:\nscore(q;d) =cos(q;d) =q\njqj\u0001d\njdj(14.8)\n4Sweet was one of Shakespeare\u2019s favorite adjectives, a fact probably related to the increased use of\nsugar in European recipes around the turn of the 16th century (Jurafsky, 2014, p. 175). 14.1 \u2022 I NFORMATION RETRIEVAL 5\nWe can spell out Eq. 14.8, using the tf-idf values and spelling out the dot product as\na sum of products:\nscore(q;d) =X\nt2qtf-idf(t;q)qP\nqi2qtf-idf2(qi;q)\u0001tf-idf(t;d)qP\ndi2dtf-idf2(di;d)(14.9)\nNow let\u2019s use (14.9) to walk through an example of a tiny query against a collec-\ntion of 4 nano documents, computing tf-idf values and seeing the rank of the docu-\nments. We\u2019ll assume all words in the following query and documents are downcased\nand punctuation is removed:\nQuery :sweet love\nDoc 1 :Sweet sweet nurse! Love?\nDoc 2 :Sweet sorrow\nDoc 3 :How sweet is love?\nDoc 4 :Nurse!\nFig. 14.2 shows the computation of the tf-idf cosine between the query and Doc-\nument 1, and the query and Document 2. The cosine is the normalized dot product\nof tf-idf values, so for the normalization we must need to compute the document\nvector lengthsjqj,jd1j, andjd2jfor the query and the \ufb01rst two documents using\nEq. 14.4, Eq. 14.5, Eq. 14.6, and Eq. 14.9 (computations for Documents 3 and 4 are\nalso needed but are left as an exercise for the reader). The dot product between the\nvectors is the sum over dimensions of the product, for each dimension, of the values\nof the two tf-idf vectors for that dimension. This product is only non-zero where\nboth the query and document have non-zero values, so for this example, in which\nonly sweet andlove have non-zero values in the query, the dot product will be the\nsum of the products of those elements of each vector.\nDocument 1 has a higher cosine with the query (0.747) than Document 2 has\nwith the query (0.0779), and so the tf-idf cosine model would rank Document 1\nabove Document 2. This ranking is intuitive given the vector space model, since\nDocument 1 has both terms including two instances of sweet , while Document 2 is\nmissing one of the terms. We leave the computation for Documents 3 and 4 as an\nexercise for the reader.\nIn practice, there are many variants and approximations to Eq. 14.9. For exam-\nple, we might choose to simplify processing by removing some terms. To see this,\nlet\u2019s start by expanding the formula for tf-idf in Eq. 14.9 to explicitly mention the tf\nand idf terms from (14.6):\nscore(q;d) =X\nt2qtft;q\u0001idftqP\nqi2qtf-idf2(qi;q)\u0001tft;d\u0001idftqP\ndi2dtf-idf2(di;d)(14.10)\nIn one common variant of tf-idf cosine, for example, we drop the idf term for the\ndocument. Eliminating the second copy of the idf term (since the identical term is\nalready computed for the query) turns out to sometimes result in better performance:\nscore(q;d) =X\nt2qtft;q\u0001idftqP\nqi2qtf-idf2(qi;q)\u0001tft;d\u0001idftqP\ndi2dtf-idf2(di;d)(14.11)\nOther variants of tf-idf eliminate various other terms.\nA slightly more complex variant in the tf-idf family is the BM25 weighting BM25 6CHAPTER 14 \u2022 Q UESTION ANSWERING , INFORMATION RETRIEVAL ,AND RAG\nQuery\nword cnt tf df idf tf-idf n\u2019lized = tf-idf/jqj\nsweet 1 1 3 0.125 0.125 0.383\nnurse 0 0 2 0.301 0 0\nlove 1 1 2 0.301 0.301 0.924\nhow 0 0 1 0.602 0 0\nsorrow 0 0 1 0.602 0 0\nis 0 0 1 0.602 0 0\njqj=p\n:1252+:3012=:326\nDocument 1 Document 2\nword cnt tf tf-idf n\u2019lized \u0002q cnt tf tf-idf n\u2019lized \u0002q\nsweet 2 1.301 0.163 0.357 0.137 1 1.000 0.125 0.203 0.0779\nnurse 1 1.000 0.301 0.661 0 0 0 0 0 0\nlove 1 1.000 0.301 0.661 0.610 0 0 0 0 0\nhow 0 0 0 0 0 0 0 0 0 0\nsorrow 0 0 0 0 0 1 1.000 0.602 0.979 0\nis 0 0 0 0 0 0 0 0 0 0\njd1j=p\n:1632+:3012+:3012=:456 jd2j=p\n:1252+:6022=:615\nCosine:Pof column: 0.747 Cosine:Pof column: 0.0779\nFigure 14.2 Computation of tf-idf cosine score between the query and nano-documents 1 (0.747) and 2\n(0.0779), using Eq. 14.4, Eq. 14.5, Eq. 14.6 and Eq. 14.9.\nscheme (sometimes called Okapi BM25 after the Okapi IR system in which it was\nintroduced (Robertson et al., 1995)). BM25 adds two parameters: k, a knob that\nadjust the balance between term frequency and IDF, and b, which controls the im-\nportance of document length normalization. The BM25 score of a document dgiven\na query qis:\nX\nt2qIDFz}|{\nlog\u0012N\ndft\u0013weighted tfz}|{\ntft;d\nk\u0010\n1\u0000b+b\u0010\njdj\njdavgj\u0011\u0011\n+tft;d(14.12)\nwherejdavgjis the length of the average document. When kis 0, BM25 reverts to\nno use of term frequency, just a binary selection of terms in the query (plus idf).\nA large kresults in raw term frequency (plus idf). branges from 1 (scaling by\ndocument length) to 0 (no length scaling). Manning et al. (2008) suggest reasonable\nvalues are k = [1.2,2] and b = 0.75. Kamphuis et al. (2020) is a useful summary of\nthe many minor variants of BM25.\nStop words In the past it was common to remove high-frequency words from both\nthe query and document before representing them. The list of such high-frequency\nwords to be removed is called a stop list . The intuition is that high-frequency terms stop list\n(often function words like the,a,to) carry little semantic weight and may not help\nwith retrieval, and can also help shrink the inverted index \ufb01les we describe below.\nThe downside of using a stop list is that it makes it dif\ufb01cult to search for phrases\nthat contain words in the stop list. For example, common stop lists would reduce the\nphrase to be or not to be to the phrase not. In modern IR systems, the use of stop lists\nis much less common, partly due to improved ef\ufb01ciency and partly because much\nof their function is already handled by IDF weighting, which downweights function 14.1 \u2022 I NFORMATION RETRIEVAL 7\nwords that occur in every document. Nonetheless, stop word removal is occasionally\nuseful in various NLP tasks so is worth keeping in mind.\n14.1.3 Inverted Index\nIn order to compute scores, we need to ef\ufb01ciently \ufb01nd documents that contain words\nin the query. (Any document that contains none of the query terms will have a score\nof 0 and can be ignored.) The basic search problem in IR is thus to \ufb01nd all documents\nd2Cthat contain a term q2Q.\nThe data structure for this task is the inverted index , which we use for mak- inverted index\ning this search ef\ufb01cient, and also conveniently storing useful information like the\ndocument frequency and the count of each term in each document.\nAn inverted index, given a query term, gives a list of documents that contain the\nterm. It consists of two parts, a dictionary and the postings . The dictionary is a list postings\nof terms (designed to be ef\ufb01ciently accessed), each pointing to a postings list for the\nterm. A postings list is the list of document IDs associated with each term, which\ncan also contain information like the term frequency or even the exact positions of\nterms in the document. The dictionary can also store the document frequency for\neach term. For example, a simple inverted index for our 4 sample documents above,\nwith each word containing its document frequency in fg, and a pointer to a postings\nlist that contains document IDs and term counts in [], might look like the following:\nhowf1g!3 [1]\nisf1g!3 [1]\nlovef2g!1 [1]!3 [1]\nnursef2g!1 [1]!4 [1]\nsorryf1g!2 [1]\nsweetf3g!1 [2]!2 [1]!3 [1]\nGiven a list of terms in query, we can very ef\ufb01ciently get lists of all candidate\ndocuments, together with the information necessary to compute the tf-idf scores we\nneed.\nThere are alternatives to the inverted index. For the question-answering domain\nof \ufb01nding Wikipedia pages to match a user query, Chen et al. (2017) show that\nindexing based on bigrams works better than unigrams, and use ef\ufb01cient hashing\nalgorithms rather than the inverted index to make the search ef\ufb01cient.\n14.1.4 Evaluation of Information-Retrieval Systems\nWe measure the performance of ranked retrieval systems using the same precision\nandrecall metrics we have been using. We make the assumption that each docu-\nment returned by the IR system is either relevant to our purposes or not relevant .\nPrecision is the fraction of the returned documents that are relevant, and recall is the\nfraction of all relevant documents that are returned. More formally, let\u2019s assume a\nsystem returns Tranked documents in response to an information request, a subset\nRof these are relevant, a disjoint subset, N, are the remaining irrelevant documents,\nandUdocuments in the collection as a whole are relevant to this request. Precision\nand recall are then de\ufb01ned as:\nPrecision =jRj\njTjRecall =jRj\njUj(14.13)\nUnfortunately, these metrics don\u2019t adequately measure the performance of a system\nthatranks the documents it returns. If we are comparing the performance of two 8CHAPTER 14 \u2022 Q UESTION ANSWERING , INFORMATION RETRIEVAL ,AND RAG\nranked retrieval systems, we need a metric that prefers the one that ranks the relevant\ndocuments higher. We need to adapt precision and recall to capture how well a\nsystem does at putting relevant documents higher in the ranking.\nRank Judgment Precision Rank Recall Rank\n1 R 1.0 .11\n2 N .50 .11\n3 R .66 .22\n4 N .50 .22\n5 R .60 .33\n6 R .66 .44\n7 N .57 .44\n8 R .63 .55\n9 N .55 .55\n10 N .50 .55\n11 R .55 .66\n12 N .50 .66\n13 N .46 .66\n14 N .43 .66\n15 R .47 .77\n16 N .44 .77\n17 N .44 .77\n18 R .44 .88\n19 N .42 .88\n20 N .40 .88\n21 N .38 .88\n22 N .36 .88\n23 N .35 .88\n24 N .33 .88\n25 R .36 1.0\nFigure 14.3 Rank-speci\ufb01c precision and recall values calculated as we proceed down\nthrough a set of ranked documents (assuming the collection has 9 relevant documents).\nLet\u2019s turn to an example. Assume the table in Fig. 14.3 gives rank-speci\ufb01c pre-\ncision and recall values calculated as we proceed down through a set of ranked doc-\numents for a particular query; the precisions are the fraction of relevant documents\nseen at a given rank, and recalls the fraction of relevant documents found at the same\nrank. The recall measures in this example are based on this query having 9 relevant\ndocuments in the collection as a whole.\nNote that recall is non-decreasing; when a relevant document is encountered,\nrecall increases, and when a non-relevant document is found it remains unchanged.\nPrecision, on the other hand, jumps up and down, increasing when relevant doc-\numents are found, and decreasing otherwise. The most common way to visualize\nprecision and recall is to plot precision against recall in a precision-recall curve ,precision-recall\ncurve\nlike the one shown in Fig. 14.4 for the data in table 14.3.\nFig. 14.4 shows the values for a single query. But we\u2019ll need to combine values\nfor all the queries, and in a way that lets us compare one system to another. One way\nof doing this is to plot averaged precision values at 11 \ufb01xed levels of recall (0 to 100,\nin steps of 10). Since we\u2019re not likely to have datapoints at these exact levels, we\nuseinterpolated precision values for the 11 recall values from the data points we dointerpolated\nprecision\nhave. We can accomplish this by choosing the maximum precision value achieved\nat any level of recall at or above the one we\u2019re calculating. In other words,\nIntPrecision (r) =max\ni>=rPrecision (i) (14.14) 14.1 \u2022 I NFORMATION RETRIEVAL 9\n0.0 0.2 0.4 0.6 0.8 1.0\nRecall0.00.20.40.60.81.0Precision\nFigure 14.4 The precision recall curve for the data in table 14.3.\nThis interpolation scheme not only lets us average performance over a set of queries,\nbut also helps smooth over the irregular precision values in the original data. It is\ndesigned to give systems the bene\ufb01t of the doubt by assigning the maximum preci-\nsion value achieved at higher levels of recall from the one being measured. Fig. 14.5\nand Fig. 14.6 show the resulting interpolated data points from our example.\nInterpolated Precision Recall\n1.0 0.0\n1.0 .10\n.66 .20\n.66 .30\n.66 .40\n.63 .50\n.55 .60\n.47 .70\n.44 .80\n.36 .90\n.36 1.0\nFigure 14.5 Interpolated data points from Fig. 14.3.\nGiven curves such as that in Fig. 14.6 we can compare two systems or approaches\nby comparing their curves. Clearly, curves that are higher in precision across all\nrecall values are preferred. However, these curves can also provide insight into the\noverall behavior of a system. Systems that are higher in precision toward the left\nmay favor precision over recall, while systems that are more geared towards recall\nwill be higher at higher levels of recall (to the right).\nA second way to evaluate ranked retrieval is mean average precision (MAP),mean average\nprecision\nwhich provides a single metric that can be used to compare competing systems or\napproaches. In this approach, we again descend through the ranked list of items,\nbut now we note the precision only at those points where a relevant item has been\nencountered (for example at ranks 1, 3, 5, 6 but not 2 or 4 in Fig. 14.3). For a single\nquery, we average these individual precision measurements over the return set (up\nto some \ufb01xed cutoff). More formally, if we assume that Rris the set of relevant 10 CHAPTER 14 \u2022 Q UESTION ANSWERING , INFORMATION RETRIEVAL ,AND RAG\nInterpolated Precision Recall Curve\n00.10.20.30.40.50.60.70.80.91\n0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1\nRecallPrecision\nFigure 14.6 An 11 point interpolated precision-recall curve. Precision at each of the 11\nstandard recall levels is interpolated for each query from the maximum at any higher level of\nrecall. The original measured precision recall points are also shown.\ndocuments at or above r, then the average precision (AP) for a single query is\nAP=1\njRrjX\nd2RrPrecision r(d) (14.15)\nwhere Precision r(d)is the precision measured at the rank at which document dwas\nfound. For an ensemble of queries Q, we then average over these averages, to get\nour \ufb01nal MAP measure:\nMAP =1\njQjX\nq2QAP(q) (14.16)\nThe MAP for the single query (hence = AP) in Fig. 14.3 is 0.6.\n14.2 Information Retrieval with Dense Vectors\nThe classic tf-idf or BM25 algorithms for IR have long been known to have a con-\nceptual \ufb02aw: they work only if there is exact overlap of words between the query\nand document. In other words, the user posing a query (or asking a question) needs\nto guess exactly what words the writer of the answer might have used, an issue called\nthevocabulary mismatch problem (Furnas et al., 1987).\nThe solution to this problem is to use an approach that can handle synonymy:\ninstead of (sparse) word-count vectors, using (dense) embeddings. This idea was\n\ufb01rst proposed for retrieval in the last century under the name of Latent Semantic\nIndexing approach (Deerwester et al., 1990), but is implemented in modern times\nvia encoders like BERT.\nThe most powerful approach is to present both the query and the document to a\nsingle encoder, allowing the transformer self-attention to see all the tokens of both 14.2 \u2022 I NFORMATION RETRIEVAL WITH DENSE VECTORS 11\nthe query and the document, and thus building a representation that is sensitive to\nthe meanings of both query and document. Then a linear layer can be put on top of\nthe [CLS] token to predict a similarity score for the query/document tuple:\nz=BERT (q;[SEP] ;d)[CLS]\nscore(q;d) =softmax (U(z)) (14.17)\nThis architecture is shown in Fig. 14.7a. Usually the retrieval step is not done on\nan entire document. Instead documents are broken up into smaller passages, such\nas non-overlapping \ufb01xed-length chunks of say 100 tokens, and the retriever encodes\nand retrieves these passages rather than entire documents. The query and document\nhave to be made to \ufb01t in the BERT 512-token window, for example by truncating\nthe query to 64 tokens and truncating the document if necessary so that it, the query,\n[CLS], and [SEP] \ufb01t in 512 tokens. The BERT system together with the linear layer\nUcan then be \ufb01ne-tuned for the relevance task by gathering a tuning dataset of\nrelevant and non-relevant passages.\nQueryDocument\u2026\u2026\u2026\u2026\u2026\u2026[sep]s(q,d)zCLSU\nQueryzCLS_QzCLS_D\nDocument\u2026\u2026\u2026\u2026\u2026\u2026\u2022s(q,d)\n(a) (b)\nFigure 14.7 Two ways to do dense retrieval, illustrated by using lines between layers to schematically rep-\nresent self-attention: (a) Use a single encoder to jointly encode query and document and \ufb01netune to produce a\nrelevance score with a linear layer over the CLS token. This is too compute-expensive to use except in rescoring\n(b) Use separate encoders for query and document, and use the dot product between CLS token outputs for the\nquery and document as the score. This is less compute-expensive, but not as accurate.\nThe problem with the full BERT architecture in Fig. 14.7a is the expense in\ncomputation and time. With this architecture, every time we get a query, we have to\npass every single single document in our entire collection through a BERT encoder\njointly with the new query! This enormous use of resources is impractical for real\ncases.\nAt the other end of the computational spectrum is a much more ef\ufb01cient archi-\ntecture, the bi-encoder . In this architecture we can encode the documents in the\ncollection only one time by using two separate encoder models, one to encode the\nquery and one to encode the document. We encode each document, and store all\nthe encoded document vectors in advance. When a query comes in, we encode just\nthis query and then use the dot product between the query vector and the precom-\nputed document vectors as the score for each candidate document (Fig. 14.7b). For\nexample, if we used BERT, we would have two encoders BERT Qand BERT Dand 12 CHAPTER 14 \u2022 Q UESTION ANSWERING , INFORMATION RETRIEVAL ,AND RAG\nwe could represent the query and document as the [CLS] token of the respective\nencoders (Karpukhin et al., 2020):\nzq=BERT Q(q)[CLS]\nzd=BERT D(d)[CLS]\nscore(q;d) =zq\u0001zd (14.18)\nThe bi-encoder is much cheaper than a full query/document encoder, but is also\nless accurate, since its relevance decision can\u2019t take full advantage of all the possi-\nble meaning interactions between all the tokens in the query and the tokens in the\ndocument.\nThere are numerous approaches that lie in between the full encoder and the bi-\nencoder. One intermediate alternative is to use cheaper methods (like BM25) as the\n\ufb01rst pass relevance ranking for each document, take the top N ranked documents,\nand use expensive methods like the full BERT scoring to rerank only the top N\ndocuments rather than the whole set.\nAnother intermediate approach is the ColBERT approach of Khattab and Za- ColBERT\nharia (2020) and Khattab et al. (2021), shown in Fig. 14.8. This method separately\nencodes the query and document, but rather than encoding the entire query or doc-\nument into one vector, it separately encodes each of them into contextual represen-\ntations for each token. These BERT representations of each document word can be\npre-stored for ef\ufb01ciency. The relevance score between a query qand a document dis\na sum of maximum similarity (MaxSim) operators between tokens in qand tokens\nind. Essentially, for each token in q, ColBERT \ufb01nds the most contextually simi-\nlar token in d, and then sums up these similarities. A relevant document will have\ntokens that are contextually very similar to the query.\nMore formally, a question qis tokenized as [q1;:::; qn], prepended with a [CLS]\nand a special [Q]token, truncated to N=32 tokens (or padded with [MASK] tokens if\nit is shorter), and passed through BERT to get output vectors q= [q1;:::;qN]. The\npassage dwith tokens [d1;:::; dm], is processed similarly, including a [CLS] and\nspecial [D]token. A linear layer is applied on top of dandqto control the output\ndimension, so as to keep the vectors small for storage ef\ufb01ciency, and vectors are\nrescaled to unit length, producing the \ufb01nal vector sequences Eq(length N) and Ed\n(length m). The ColBERT scoring mechanism is:\nscore(q;d) =NX\ni=1mmax\nj=1Eqi\u0001Edj(14.19)\nWhile the interaction mechanism has no tunable parameters, the ColBERT ar-\nchitecture still needs to be trained end-to-end to \ufb01ne-tune the BERT encoders and\ntrain the linear layers (and the special [Q] and[D] embeddings) from scratch. It\nis trained on triples hq;d+;d\u0000iof query q, positive document d+and negative doc-\nument d\u0000to produce a score for each document using (14.19), optimizing model\nparameters using a cross-entropy loss.\nAll the supervised algorithms (like ColBERT or the full-interaction version of\nthe BERT algorithm applied for reranking) need training data in the form of queries\ntogether with relevant and irrelevant passages or documents (positive and negative\nexamples). There are various semi-supervised ways to get labels; some datasets (like\nMS MARCO Ranking, Section 14.3.2) contain gold positive examples. Negative\nexamples can be sampled randomly from the top-1000 results from some existing\nIR system. If datasets don\u2019t have labeled positive examples, iterative methods like 14.3 \u2022 A NSWERING QUESTIONS WITH RAG 13\nQueryDocument\u2026\u2026\u2026\u2026\u2026\u2026s(q,d)MaxSimMaxSimMaxSim\u2211\nnormnormnormnormnormnorm\nFigure 14.8 A sketch of the ColBERT algorithm at inference time. The query and docu-\nment are \ufb01rst passed through separate BERT encoders. Similarity between query and doc-\nument is computed by summing a soft alignment between the contextual representations of\ntokens in the query and the document. Training is end-to-end. (Various details aren\u2019t de-\npicted; for example the query is prepended by a [CLS] and[Q:] tokens, and the document\nby[CLS] and[D:] tokens). Figure adapted from Khattab and Zaharia (2020).\nrelevance-guided supervision can be used (Khattab et al., 2021) which rely on the\nfact that many datasets contain short answer strings. In this method, an existing IR\nsystem is used to harvest examples that do contain short answer strings (the top few\nare taken as positives) or don\u2019t contain short answer strings (the top few are taken as\nnegatives), these are used to train a new retriever, and then the process is iterated.\nEf\ufb01ciency is an important issue, since every possible document must be ranked\nfor its similarity to the query. For sparse word-count vectors, the inverted index\nallows this very ef\ufb01ciently. For dense vector algorithms \ufb01nding the set of dense\ndocument vectors that have the highest dot product with a dense query vector is\nan instance of the problem of nearest neighbor search . Modern systems there-\nfore make use of approximate nearest neighbor vector search algorithms like Faiss Faiss\n(Johnson et al., 2017).\n14.3 Answering Questions with RAG\nThe dominant paradigm for question answering is to answer a user\u2019s question by \ufb01rst\n\ufb01nding supportive text segments from the web or another other large collection of\ndocuments, and then generating an answer based on the documents. The method of\ngenerating based on retrieved documents is called retrieval-augmented generation\norRAG , and the two components are sometimes called the retriever and the reader\n(Chen et al., 2017). Fig. 14.9 sketches out this standard QA model.\nIn the \ufb01rst stage of the 2-stage retrieve and read model in Fig. 14.9 we retrieve\nrelevant passages from a text collection, for example using the dense retrievers of the\nprevious section. In the second reader stage, we generate the answer via retrieval- 14 CHAPTER 14 \u2022 Q UESTION ANSWERING , INFORMATION RETRIEVAL ,AND RAG\nQ: When wasthe premiere ofThe Magic Flute?RelevantDocsA:  1791RetrieverIndexed Docs\nquerydocsLLMpromptReader/Generator\nFigure 14.9 Retrieval-based question answering has two stages: retrieval , which returns relevant documents\nfrom the collection, and reading , in which an LLM generates answers given the documents as a prompt.\naugmented generation . In this method, we take a large pretrained language model,\ngive it the set of retrieved passages and other text as its prompt, and autoregressively\ngenerate a new answer token by token.\n14.3.1 Retrieval-Augmented Generation\nThe standard reader algorithm is to generate from a large language model, condi-\ntioned on the retrieved passages. This method is known as retrieval-augmented\ngeneration , orRAG .retrieval-\naugmented\ngeneration\nRAG Recall that in simple conditional generation, we can cast the task of question\nanswering as word prediction by giving a language model a question and a token\nlikeA:suggesting that an answer should come next:\nQ: Who wrote the book ``The Origin of Species\"? A:\nThen we generate autoregressively conditioned on this text.\nMore formally, recall that simple autoregressive language modeling computes\nthe probability of a string from the previous tokens:\np(x1;:::; xn) =nY\ni=1p(xijx<i)\nAnd simple conditional generation for question answering adds a prompt like Q:,\nfollowed by a query q, and A:, all concatenated:\np(x1;:::; xn) =nY\ni=1p([Q:] ;q;[A:] ;x<i)\nThe advantage of using a large language model is the enormous amount of\nknowledge encoded in its parameters from the text it was pretrained on. But as\nwe mentioned at the start of the chapter, while this kind of simple prompted gener-\nation can work \ufb01ne for many simple factoid questions, it is not a general solution\nfor QA, because it leads to hallucination, is unable to show users textual evidence to\nsupport the answer, and is unable to answer questions from proprietary data.\nThe idea of retrieval-augmented generation is to address these problems by con-\nditioning on the retrieved passages as part of the pre\ufb01x, perhaps with some prompt\ntext like \u201cBased on these texts, answer this question:\u201d. Let\u2019s suppose we have a\nquery q, and call the set of retrieved passages based on it R( q). For example, we\ncould have a prompt like: 14.3 \u2022 A NSWERING QUESTIONS WITH RAG 15\nSchematic of a RAG Prompt\nretrieved passage 1\nretrieved passage 2\n...\nretrieved passage n\nBased on these texts, answer this question: Q: Who wrote\nthe book ``The Origin of Species\"? A:\nOr more formally,\np(x1;:::; xn) =nY\ni=1p(xijR(q) ; prompt ; [Q:] ;q;[A:] ;x<i)\nAs with the span-based extraction reader, successfully applying the retrieval-\naugmented generation algorithm for QA requires a successful retriever, and often\na two-stage retrieval algorithm is used in which the retrieval is reranked. Some\ncomplex questions may require multi-hop architectures, in which a query is used to multi-hop\nretrieve documents, which are then appended to the original query for a second stage\nof retrieval. Details of prompt engineering also have to be worked out, like deciding\nwhether to demarcate passages, for example with [SEP] tokens, and so on. Combi-\nnations of private data and public data involving an externally hosted large language\nmodel may lead to privacy concerns that need to be worked out (Arora et al., 2023).\nMuch research in this area also focuses on ways to more tightly integrated the re-\ntrieval and reader stages.\n14.3.2 Question Answering Datasets\nThere are scores of question answering datasets, used both for instruction tuning and\nfor evaluation of the question answering abilities of language models.\nWe can distinguish the datasets along many dimensions, summarized nicely in\nRogers et al. (2023). One is the original purpose of the questions in the data, whether\nthey were natural information-seeking questions, or whether they were questions\ndesigned for probing : evaluating or testing systems or humans.\nOn the natural side there are datasets like Natural Questions (KwiatkowskiNatural\nQuestions\net al., 2019), a set of anonymized English queries to the Google search engine and\ntheir answers. The answers are created by annotators based on Wikipedia infor-\nmation, and include a paragraph-length long answer and a short span answer. For\nexample the question \u201cWhen are hops added to the brewing process?\u201d has the short\nanswer the boiling process and a long answer which is an entire paragraph from the\nWikipedia page on Brewing .\nA similar natural question set is the MS MARCO (Microsoft Machine Reading MS MARCO\nComprehension) collection of datasets, including 1 million real anonymized English\nquestions from Microsoft Bing query logs together with a human generated answer\nand 9 million passages (Bajaj et al., 2016), that can be used both to test retrieval\nranking and question answering. 16 CHAPTER 14 \u2022 Q UESTION ANSWERING , INFORMATION RETRIEVAL ,AND RAG\nAlthough many datasets focus on English, natural information-seeking ques-\ntion datasets exist in other languages. The DuReader dataset is a Chinese QA re-\nsource based on search engine queries and community QA (He et al., 2018). TyDi\nQAdataset contains 204K question-answer pairs from 11 typologically diverse lan- TyDi QA\nguages, including Arabic, Bengali, Kiswahili, Russian, and Thai (Clark et al., 2020).\nIn the T YDIQA task, a system is given a question and the passages from a Wiki-\npedia article and must (a) select the passage containing the answer (or N ULL if no\npassage contains the answer), and (b) mark the minimal answer span (or N ULL).\nOn the probing side are datasets like MMLU (Massive Multitask Language Un- MMLU\nderstanding), a commonly-used dataset of 15908 knowledge and reasoning ques-\ntions in 57 areas including medicine, mathematics, computer science, law, and oth-\ners. MMLU questions are sourced from various exams for humans, such as the US\nGraduate Record Exam, Medical Licensing Examination, and Advanced Placement\nexams. So the questions don\u2019t represent people\u2019s information needs, but rather are\ndesigned to test human knowledge for academic or licensing purposes. Fig. 14.10\nshows some examples, with the correct answers in bold.\nSome of the question datasets described above augment each question with pas-\nsage(s) from which the answer can be extracted. These datasets were mainly created\nfor an earlier QA task called reading comprehension in which a model is givenreading\ncomprehension\na question and a document and is required to extract the answer from the given\ndocument. We sometimes call the task of question answering given one or more\ndocuments (for example via RAG), the open book QA task, while the task of an- open book\nswering directly from the LM with no retrieval component at all is the closed book closed book\nQA task.5Thus datasets like Natural Questions can be treated as open book if the\nsolver uses each question\u2019s attached document, or closed book if the documents are\nnot used, while datasets like MMLU are solely closed book.\nAnother dimension of variation is the format of the answer: multiple-choice\nversus freeform. And of course there are variations in prompting, like whether the\nmodel is just the question (zero-shot) or also given demonstrations of answers to\nsimilar questions (few-shot). MMLU offers both zero-shot and few-shot prompt\noptions.\n14.4 Evaluating Question Answering\nThree techniques are commonly employed to evaluate question-answering systems,\nwith the choice depending on the type of question and QA situation. For multiple\nchoice questions like in MMLU, we report exact match:\nExact match : The % of predicted answers that match the gold answer\nexactly.\nFor questions with free text answers, like Natural Questions, we commonly evalu-\nated with token F1score to roughly measure the partial string overlap between the\nanswer and the reference answer:\nF1score : The average token overlap between predicted and gold an-\nswers. Treat the prediction and gold as a bag of tokens, and compute F 1\nfor each question, then return the average F 1over all questions.\n5This repurposes the word for types of exams in which students are allowed to \u2018open their books\u2019 or\nnot. 14.4 \u2022 E VALUATING QUESTION ANSWERING 17\nMMLU examples\nCollege Computer Science\nAny set of Boolean operators that is suf\ufb01cient to represent all Boolean ex-\npressions is said to be complete. Which of the following is NOT complete?\n(A) AND, NOT\n(B) NOT, OR\n(C) AND, OR\n(D) NAND\nCollege Physics\nThe primary source of the Sun\u2019s energy is a series of thermonuclear\nreactions in which the energy produced is c2times the mass difference\nbetween\n(A) two hydrogen atoms and one helium atom\n(B)four hydrogen atoms and one helium atom\n(C) six hydrogen atoms and two helium atoms\n(D) three helium atoms and one carbon atom\nInternational Law\nWhich of the following is a treaty-based human rights mechanism?\n(A)The UN Human Rights Committee\n(B) The UN Human Rights Council\n(C) The UN Universal Periodic Review\n(D) The UN special mandates\nPrehistory\nUnlike most other early civilizations, Minoan culture shows little evidence\nof\n(A) trade.\n(B) warfare.\n(C) the development of a common religion.\n(D)conspicuous consumption by elites.\nFigure 14.10 Example problems from MMLU\nFinally, in some situations QA systems give multiple ranked answers. In such cases\nwe evaluated using mean reciprocal rank , orMRR (V oorhees, 1999). MRR ismean\nreciprocal rank\nMRR designed for systems that return a short ranked list of answers or passages for each\ntest set question, which we can compare against the (human-labeled) correct answer.\nFirst, each test set question is scored with the reciprocal of the rank of the \ufb01rst\ncorrect answer. For example if the system returned \ufb01ve answers to a question but\nthe \ufb01rst three are wrong (so the highest-ranked correct answer is ranked fourth), the\nreciprocal rank for that question is1\n4. The score for questions that return no correct\nanswer is 0. The MRR of a system is the average of the scores for each question in\nthe test set. In some versions of MRR, questions with a score of zero are ignored\nin this calculation. More formally, for a system returning ranked answers to each\nquestion in a test set Q, (or in the alternate version, let Qbe the subset of test set 18 CHAPTER 14 \u2022 Q UESTION ANSWERING , INFORMATION RETRIEVAL ,AND RAG\nquestions that have non-zero scores). MRR is then de\ufb01ned as\nMRR =1\njQjjQjX\ni=11\nrank i(14.20)\n14.5 Summary\nThis chapter introduced the tasks of question answering andinformation retrieval .\n\u2022Question answering (QA) is the task of answering a user\u2019s questions.\n\u2022 We focus in this chapter on the task of retrieval-based question answering,\nin which the user\u2019s questions are intended to be answered by the material in\nsome set of documents (which might be the web).\n\u2022Information Retrieval (IR) is the task of returning documents to a user based\non their information need as expressed in a query . In ranked retrieval, the\ndocuments are returned in ranked order.\n\u2022 The match between a query and a document can be done by \ufb01rst representing\neach of them with a sparse vector that represents the frequencies of words,\nweighted by tf-idf orBM25 . Then the similarity can be measured by cosine.\n\u2022 Documents or queries can instead be represented by dense vectors, by encod-\ning the question and document with an encoder-only model like BERT, and in\nthat case computing similarity in embedding space.\n\u2022 The inverted index is a storage mechanism that makes it very ef\ufb01cient to \ufb01nd\ndocuments that have a particular word.\n\u2022 Ranked retrieval is generally evaluated by mean average precision orinter-\npolated precision .\n\u2022 Question answering systems generally use the retriever /reader architecture.\nIn the retriever stage, an IR system is given a query and returns a set of\ndocuments.\n\u2022 The reader stage is implemented by retrieval-augmented generation , in\nwhich a large language model is prompted with the query and a set of doc-\numents and then conditionally generates a novel answer.\n\u2022 QA can be evaluated by exact match with a known answer if only a single\nanswer is given, with token F 1score for free text answers, or with mean re-\nciprocal rank if a ranked set of answers is given.\nBibliographical and Historical Notes\nQuestion answering was one of the earliest NLP tasks, and early versions of the text-\nbased and knowledge-based paradigms were developed by the very early 1960s. The\ntext-based algorithms generally relied on simple parsing of the question and of the\nsentences in the document, and then looking for matches. This approach was used\nvery early on (Phillips, 1960) but perhaps the most complete early system, and one\nthat strikingly pre\ufb01gures modern relation-based systems, was the Protosynthex sys-\ntem of Simmons et al. (1964). Given a question, Protosynthex \ufb01rst formed a query BIBLIOGRAPHICAL AND HISTORICAL NOTES 19\nfrom the content words in the question, and then retrieved candidate answer sen-\ntences in the document, ranked by their frequency-weighted term overlap with the\nquestion. The query and each retrieved sentence were then parsed with dependency\nparsers, and the sentence whose structure best matches the question structure se-\nlected. Thus the question What do worms eat? would match worms eat grass : both\nhave the subject worms as a dependent of eat, in the version of dependency grammar\nused at the time, while birds eat worms hasbirds as the subject:\nWhat doworms eat Worms eat grass Birds eat worms\nThe alternative knowledge-based paradigm was implemented in the BASEBALL\nsystem (Green et al., 1961). This system answered questions about baseball games\nlike \u201cWhere did the Red Sox play on July 7\u201d by querying a structured database of\ngame information. The database was stored as a kind of attribute-value matrix with\nvalues for attributes of each game:\nMonth = July\nPlace = Boston\nDay = 7\nGame Serial No. = 96\n(Team = Red Sox, Score = 5)\n(Team = Yankees, Score = 3)\nEach question was constituency-parsed using the algorithm of Zellig Harris\u2019s\nTDAP project at the University of Pennsylvania, essentially a cascade of \ufb01nite-state\ntransducers (see the historical discussion in Joshi and Hopely 1999 and Karttunen\n1999). Then in a content analysis phase each word or phrase was associated with a\nprogram that computed parts of its meaning. Thus the phrase \u2018Where\u2019 had code to\nassign the semantics Place = ? , with the result that the question \u201cWhere did the\nRed Sox play on July 7\u201d was assigned the meaning\nPlace = ?\nTeam = Red Sox\nMonth = July\nDay = 7\nThe question is then matched against the database to return the answer. Simmons\n(1965) summarizes other early QA systems.\nAnother important progenitor of the knowledge-based paradigm for question-\nanswering is work that used predicate calculus as the meaning representation lan-\nguage. The LUNAR system (Woods et al. 1972, Woods 1978) was designed to be LUNAR\na natural language interface to a database of chemical facts about lunar geology. It\ncould answer questions like Do any samples have greater than 13 percent aluminum\nby parsing them into a logical form\n(TEST (FOR SOME X16 / (SEQ SAMPLES) : T ; (CONTAIN\u2019 X16\n(NPR* X17 / (QUOTE AL203)) (GREATERTHAN 13 PCT))))\nBy a couple decades later, drawing on new machine learning approaches in NLP,\nZelle and Mooney (1996) proposed to treat knowledge-based QA as a semantic pars-\ning task, by creating the Prolog-based GEOQUERY dataset of questions about US\ngeography. This model was extended by Zettlemoyer and Collins (2005) and 2007. 20 CHAPTER 14 \u2022 Q UESTION ANSWERING , INFORMATION RETRIEVAL ,AND RAG\nBy a decade later, neural models were applied to semantic parsing (Dong and Lap-\nata 2016, Jia and Liang 2016), and then to knowledge-based question answering by\nmapping text to SQL (Iyer et al., 2017).\nMeanwhile, the information-retrieval paradigm for question answering was in-\n\ufb02uenced by the rise of the web in the 1990s. The U.S. government-sponsored TREC\n(Text REtrieval Conference) evaluations, run annually since 1992, provide a testbed\nfor evaluating information-retrieval tasks and techniques (V oorhees and Harman,\n2005). TREC added an in\ufb02uential QA track in 1999, which led to a wide variety of\nfactoid and non-factoid systems competing in annual evaluations.\nAt that same time, Hirschman et al. (1999) introduced the idea of using chil-\ndren\u2019s reading comprehension tests to evaluate machine text comprehension algo-\nrithms. They acquired a corpus of 120 passages with 5 questions each designed for\n3rd-6th grade children, built an answer extraction system, and measured how well\nthe answers given by their system corresponded to the answer key from the test\u2019s\npublisher. Their algorithm focused on word overlap as a feature; later algorithms\nadded named entity features and more complex similarity between the question and\nthe answer span (Riloff and Thelen 2000, Ng et al. 2000).\nThe DeepQA component of the Watson Jeopardy! system was a large and so-\nphisticated feature-based system developed just before neural systems became com-\nmon. It is described in a series of papers in volume 56 of the IBM Journal of Re-\nsearch and Development, e.g., Ferrucci (2012).\nEarly neural reading comprehension systems drew on the insight common to\nearly systems that answer \ufb01nding should focus on question-passage similarity. Many\nof the architectural outlines of these neural systems were laid out in Hermann et al.\n(2015), Chen et al. (2017), and Seo et al. (2017). These systems focused on datasets\nlike Rajpurkar et al. (2016) and Rajpurkar et al. (2018) and their successors, usually\nusing separate IR algorithms as input to neural reading comprehension systems. The\nparadigm of using dense retrieval with a span-based reader, often with a single end-\nto-end architecture, is exempli\ufb01ed by systems like Lee et al. (2019) or Karpukhin\net al. (2020). An important research area with dense retrieval for open-domain QA\nis training data: using self-supervised methods to avoid having to label positive and\nnegative passages (Sachan et al., 2023).\nEarly work on large language models showed that they stored suf\ufb01cient knowl-\nedge in the pretraining process to answer questions (Petroni et al., 2019; Raffel et al.,\n2020; Radford et al., 2019; Roberts et al., 2020), at \ufb01rst not competitively with\nspecial-purpose question answerers, but then surpassing them. Retrieval-augmented\ngeneration algorithms were \ufb01rst introduced as a way to improve language modeling\n(Khandelwal et al., 2019), but were quickly applied to question answering (Izacard\net al., 2022; Ram et al., 2023; Shi et al., 2023).\nExercises Exercises 21\nArora, S., P. Lewis, A. Fan, J. Kahn, and C. R \u00b4e. 2023. Rea-\nsoning over public and private data in retrieval-based sys-\ntems. TACL , 11:902\u2013921.\nBajaj, P., D. Campos, N. Craswell, L. Deng, J. G. ando\nXiaodong Liu, R. Majumder, A. McNamara, B. Mitra,\nT. Nguye, M. Rosenberg, X. Song, A. Stoica, S. Tiwary,\nand T. Wang. 2016. MS MARCO: A human generated\nMAchine Reading COmprehension dataset. NeurIPS .\nChen, D., A. Fisch, J. Weston, and A. Bordes. 2017. Reading\nWikipedia to answer open-domain questions. ACL.\nClark, J. H., E. Choi, M. Collins, D. Garrette,\nT. Kwiatkowski, V . Nikolaev, and J. Palomaki. 2020.\nTyDi QA: A benchmark for information-seeking ques-\ntion answering in typologically diverse languages. TACL ,\n8:454\u2013470.\nDahl, M., V . Magesh, M. Suzgun, and D. E. Ho. 2024. Large\nlegal \ufb01ctions: Pro\ufb01ling legal hallucinations in large lan-\nguage models. Journal of Legal Analysis , 16:64\u201393.\nDeerwester, S. C., S. T. Dumais, T. K. Landauer, G. W. Fur-\nnas, and R. A. Harshman. 1990. Indexing by latent se-\nmantics analysis. JASIS , 41(6):391\u2013407.\nDong, L. and M. Lapata. 2016. Language to logical form\nwith neural attention. ACL.\nFerrucci, D. A. 2012. Introduction to \u201cThis is Watson\u201d. IBM\nJournal of Research and Development , 56(3/4):1:1\u20131:15.\nFurnas, G. W., T. K. Landauer, L. M. Gomez, and S. T.\nDumais. 1987. The vocabulary problem in human-\nsystem communication. Communications of the ACM ,\n30(11):964\u2013971.\nGreen, B. F., A. K. Wolf, C. Chomsky, and K. Laughery.\n1961. Baseball: An automatic question answerer. Pro-\nceedings of the Western Joint Computer Conference 19 .\nHe, W., K. Liu, J. Liu, Y . Lyu, S. Zhao, X. Xiao, Y . Liu,\nY . Wang, H. Wu, Q. She, X. Liu, T. Wu, and H. Wang.\n2018. DuReader: a Chinese machine reading compre-\nhension dataset from real-world applications. Workshop\non Machine Reading for Question Answering .\nHermann, K. M., T. Kocisky, E. Grefenstette, L. Espeholt,\nW. Kay, M. Suleyman, and P. Blunsom. 2015. Teaching\nmachines to read and comprehend. NeurIPS .\nHirschman, L., M. Light, E. Breck, and J. D. Burger. 1999.\nDeep Read: A reading comprehension system. ACL.\nIyer, S., I. Konstas, A. Cheung, J. Krishnamurthy, and\nL. Zettlemoyer. 2017. Learning a neural semantic parser\nfrom user feedback. ACL.\nIzacard, G., P. Lewis, M. Lomeli, L. Hosseini, F. Petroni,\nT. Schick, J. Dwivedi-Yu, A. Joulin, S. Riedel, and\nE. Grave. 2022. Few-shot learning with retrieval aug-\nmented language models. ArXiv preprint.\nJia, R. and P. Liang. 2016. Data recombination for neural\nsemantic parsing. ACL.\nJohnson, J., M. Douze, and H. J \u00b4egou. 2017. Billion-\nscale similarity search with GPUs. ArXiv preprint\narXiv:1702.08734.\nJoshi, A. K. and P. Hopely. 1999. A parser from antiquity.\nIn A. Kornai, ed., Extended Finite State Models of Lan-\nguage , 6\u201315. Cambridge University Press.\nJurafsky, D. 2014. The Language of Food . W. W. Norton,\nNew York.Kamphuis, C., A. P. de Vries, L. Boytsov, and J. Lin. 2020.\nWhich BM25 do you mean? a large-scale reproducibil-\nity study of scoring variants. European Conference on\nInformation Retrieval .\nKarpukhin, V ., B. O \u02d8guz, S. Min, P. Lewis, L. Wu, S. Edunov,\nD. Chen, and W.-t. Yih. 2020. Dense passage retrieval for\nopen-domain question answering. EMNLP .\nKarttunen, L. 1999. Comments on Joshi. In A. Kornai, ed.,\nExtended Finite State Models of Language , 16\u201318. Cam-\nbridge University Press.\nKhandelwal, U., O. Levy, D. Jurafsky, L. Zettlemoyer, and\nM. Lewis. 2019. Generalization through memorization:\nNearest neighbor language models. ICLR .\nKhattab, O., C. Potts, and M. Zaharia. 2021. Relevance-\nguided supervision for OpenQA with ColBERT. TACL ,\n9:929\u2013944.\nKhattab, O. and M. Zaharia. 2020. ColBERT: Ef\ufb01cient and\neffective passage search via contextualized late interac-\ntion over BERT. SIGIR .\nKwiatkowski, T., J. Palomaki, O. Red\ufb01eld, M. Collins,\nA. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. De-\nvlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M.-W.\nChang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov.\n2019. Natural questions: A benchmark for question an-\nswering research. TACL , 7:452\u2013466.\nLee, K., M.-W. Chang, and K. Toutanova. 2019. Latent re-\ntrieval for weakly supervised open domain question an-\nswering. ACL.\nManning, C. D., P. Raghavan, and H. Sch \u00a8utze. 2008. Intro-\nduction to Information Retrieval . Cambridge.\nNg, H. T., L. H. Teo, and J. L. P. Kwan. 2000. A ma-\nchine learning approach to answering questions for read-\ning comprehension tests. EMNLP .\nPetroni, F., T. Rockt \u00a8aschel, S. Riedel, P. Lewis, A. Bakhtin,\nY . Wu, and A. Miller. 2019. Language models as knowl-\nedge bases? EMNLP .\nPhillips, A. V . 1960. A question-answering routine. Techni-\ncal Report 16, MIT AI Lab.\nRadford, A., J. Wu, R. Child, D. Luan, D. Amodei, and\nI. Sutskever. 2019. Language models are unsupervised\nmultitask learners. OpenAI tech report.\nRaffel, C., N. Shazeer, A. Roberts, K. Lee, S. Narang,\nM. Matena, Y . Zhou, W. Li, and P. J. Liu. 2020. Exploring\nthe limits of transfer learning with a uni\ufb01ed text-to-text\ntransformer. JMLR , 21(140):1\u201367.\nRajpurkar, P., R. Jia, and P. Liang. 2018. Know what you\ndon\u2019t know: Unanswerable questions for SQuAD. ACL.\nRajpurkar, P., J. Zhang, K. Lopyrev, and P. Liang. 2016.\nSQuAD: 100,000+ questions for machine comprehension\nof text. EMNLP .\nRam, O., Y . Levine, I. Dalmedigos, D. Muhlgay, A. Shashua,\nK. Leyton-Brown, and Y . Shoham. 2023. In-context\nretrieval-augmented language models. ArXiv preprint.\nRiloff, E. and M. Thelen. 2000. A rule-based ques-\ntion answering system for reading comprehension tests.\nANLP/NAACL workshop on reading comprehension tests .\nRoberts, A., C. Raffel, and N. Shazeer. 2020. How much\nknowledge can you pack into the parameters of a lan-\nguage model? EMNLP . 22 Chapter 14 \u2022 Question Answering, Information Retrieval, and RAG\nRobertson, S., S. Walker, S. Jones, M. M. Hancock-\nBeaulieu, and M. Gatford. 1995. Okapi at TREC-3.\nOverview of the Third Text REtrieval Conference (TREC-\n3).\nRogers, A., M. Gardner, and I. Augenstein. 2023. QA dataset\nexplosion: A taxonomy of NLP resources for question\nanswering and reading comprehension. ACM Computing\nSurveys , 55(10):1\u201345.\nSachan, D. S., M. Lewis, D. Yogatama, L. Zettlemoyer,\nJ. Pineau, and M. Zaheer. 2023. Questions are all you\nneed to train a dense passage retriever. TACL , 11:600\u2013\n616.\nSalton, G. 1971. The SMART Retrieval System: Experiments\nin Automatic Document Processing . Prentice Hall.\nSeo, M., A. Kembhavi, A. Farhadi, and H. Hajishirzi. 2017.\nBidirectional attention \ufb02ow for machine comprehension.\nICLR .\nShi, W., S. Min, M. Yasunaga, M. Seo, R. James, M. Lewis,\nL. Zettlemoyer, and W.-t. Yih. 2023. REPLUG: Retrieval-\naugmented black-box language models. ArXiv preprint.\nSimmons, R. F. 1965. Answering English questions by com-\nputer: A survey. CACM , 8(1):53\u201370.\nSimmons, R. F., S. Klein, and K. McConlogue. 1964. In-\ndexing and dependency logic for answering English ques-\ntions. American Documentation , 15(3):196\u2013204.\nSparck Jones, K. 1972. A statistical interpretation of term\nspeci\ufb01city and its application in retrieval. Journal of Doc-\numentation , 28(1):11\u201321.\nV oorhees, E. M. 1999. TREC-8 question answering track\nreport. Proceedings of the 8th Text Retrieval Conference .\nV oorhees, E. M. and D. K. Harman. 2005. TREC: Experi-\nment and Evaluation in Information Retrieval . MIT Press.\nWoods, W. A. 1978. Semantics and quanti\ufb01cation in natural\nlanguage question answering. In M. Yovits, ed., Advances\nin Computers , 2\u201364. Academic.\nWoods, W. A., R. M. Kaplan, and B. L. Nash-Webber. 1972.\nThe lunar sciences natural language information system:\nFinal report. Technical Report 2378, BBN.\nZelle, J. M. and R. J. Mooney. 1996. Learning to parse\ndatabase queries using inductive logic programming.\nAAAI .\nZettlemoyer, L. and M. Collins. 2005. Learning to map\nsentences to logical form: Structured classi\ufb01cation with\nprobabilistic categorial grammars. Uncertainty in Arti\ufb01-\ncial Intelligence, UAI\u201905 .\nZettlemoyer, L. and M. Collins. 2007. Online learning\nof relaxed CCG grammars for parsing to logical form.\nEMNLP/CoNLL .\nZhou, K., J. Hwang, X. Ren, and M. Sap. 2024. Relying\non the unreliable: The impact of language models\u2019 reluc-\ntance to express uncertainty. ACL.", "15.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\n15Chatbots & Dialogue Systems\nLes lois de la conversation sont en g \u00b4en\u00b4eral de ne s\u2019y appesantir sur aucun ob-\njet, mais de passer l \u00b4eg`erement, sans effort et sans affectation, d\u2019un sujet `a un\nautre ; de savoir y parler de choses frivoles comme de choses s \u00b4erieuses\n[The rules of conversation are, in general, not to dwell on any one subject,\nbut to pass lightly from one to another without effort and without affectation;\nto know how to speak about trivial topics as well as serious ones;]\nThe 18th C. Encyclopedia of Diderot, start of the entry on conversation\nThe literature of the fantastic abounds in inanimate objects magically endowed with\nthe gift of speech. From Ovid\u2019s statue of Pygmalion to Mary Shelley\u2019s story about\nFrankenstein, we continually reinvent stories about creat-\ning something and then having a chat with it. Legend has\nit that after \ufb01nishing his sculpture Moses , Michelangelo\nthought it so lifelike that he tapped it on the knee and\ncommanded it to speak. Perhaps this shouldn\u2019t be sur-\nprising. Language is the mark of humanity and sentience,\nandconversation ordialogue is the most fundamental conversation\ndialogue arena of language. It is the \ufb01rst kind of language we\nlearn as children, and the kind we engage in constantly,\nwhether we are ordering lunch, buying train tickets, or\ntalking with our families, friends, or coworkers.\nThis chapter introduces the fundamental algorithms of programs that use con-\nversation to interact with users. We often distinguish between two kinds of archi-\ntectures. Task-oriented dialogue systems converse with users to accomplish \ufb01xed dialogue system\ntasks like controlling appliances or \ufb01nding restaurants, relying on a data structure\ncalled the frame , which represents the knowledge a system needs to acquire from frame\nthe user (like the time to set an alarm clock). Chatbots , by contrast, are designed chatbot\nto mimic the longer and more unstructured conversations or \u2018chats\u2019 characteristic of\nhuman-human interaction. Modern systems incorporate aspects of both; industrial\nchatbots like ChatGPT can carry on longer unstructured conversations; industrial\ndigital assistants like Siri or Alexa are generally frame-based dialogue systems.\nThe fact that chatbots and dialogue systems are designed for human-computer\ninteraction has strong implications for their design and use. Many of these impli-\ncations already became clear in one of the earliest chatbots, ELIZA (Weizenbaum,\n1966). ELIZA was designed to simulate a Rogerian psychologist, based on a branch\nof clinical psychology whose methods involve drawing the patient out by re\ufb02ecting\npatient\u2019s statements back at them. Rogerian interactions are the rare type of conver-\nsation in which, as Weizenbaum points out, one can \u201cassume the pose of knowing\nalmost nothing of the real world\u201d. If a patient says \u201cI went for a long boat ride\u201d and\nthe psychiatrist says \u201cTell me about boats\u201d, you don\u2019t assume she didn\u2019t know what 2CHAPTER 15 \u2022 C HATBOTS & D IALOGUE SYSTEMS\nMen are all alike.\nIN WHAT WAY\nThey\u2019re always bugging us about something or other.\nCAN YOU THINK OF A SPECIFIC EXAMPLE\nWell my boyfriend made me come here.\nYOUR BOYFRIEND MADE YOU COME HERE\nHe says I\u2019m depressed much of the time.\nI AM SORRY TO HEAR YOU ARE DEPRESSED\nFigure 15.1 Sample ELIZA dialogue from Weizenbaum (1966).\na boat is, but rather assume she had some conversational goal.1\nWeizenbaum made use of this property of Rogerian psychiatric conversations,\nalong with clever regular expressions, to allow ELIZA to interact in ways that seemed\ndeceptively human-like, as in the sample conversational fragment in Fig. 15.1.\nAs we foreshadowed in Chapter 2, ELIZA worked by simple rules roughly like:\n(.*) YOU (.*) ME -> WHAT MAKES YOU THINK I \\2 YOU\nto transform a user sentence like \u201c You hate me \u201d into a system response like\nWHAT MAKES YOU THINK I HATE YOU\nAmong Weizenbaum\u2019s clever tricks are the linking of each ELIZA pattern/rule\nto a keyword. Consider the following user sentence:\nI know everybody laughed at me\nBecause it has the word \u201cI\u201d, this sentence could match the following rule whose\nkeyword is I:\nI (.*) -> You say you \\1\nproducing:\nYOU SAY YOU KNOW EVERYBODY LAUGHED AT YOU\nWeizenbaum points out, however, that a more powerful response would rely on\nthe keyword \u201ceverybody\u201d, since someone using universals like everybody oralways\nis probably thinking about a speci\ufb01c person or situation. So the ELIZA algorithm\nprefers to respond using patterns associated more speci\ufb01c keywords like everybody :\nWHO IN PARTICULAR ARE YOU THINKING OF?\nIf no keyword matches, the algorithm chooses a non-committal response like\n\u201cPLEASE GO ON \u201d, \u201cTHAT'S VERY INTERESTING \u201d, or \u201c I SEE \u201d.\nELIZA illustrates a number of important issues with chatbots. First, people\nbecame deeply emotionally involved and conducted very personal conversations,\neven to the extent of asking Weizenbaum to leave the room while they were typ-\ning. Reeves and Nass (1996) show that people tend to assign human characteristics\nto computers and interact with them in ways that are typical of human-human in-\nteractions. They interpret an utterance in the way they would if it had spoken by a\nhuman, (even though they are aware they are talking to a computer). This means that\nchatbots can have signi\ufb01cant in\ufb02uences on people\u2019s cognitive and emotional state.\nA second related issue is privacy. When Weizenbaum suggested that he might\nwant to store the ELIZA conversations, people immediately pointed out that this\nwould violate people\u2019s privacy. Modern chatbots in the home are likely to overhear\n1This is due to the Gricean principle of relevance that we\u2019ll discuss in the next section.. 15.1 \u2022 P ROPERTIES OF HUMAN CONVERSATION 3\nprivate information, even if they aren\u2019t used for counseling as ELIZA was. Indeed,\nif a chatbot is human-like, users are more likely to disclose private information, and\nyet less likely to worry about the harm of this disclosure (Ischen et al., 2019).\nBoth of these issues (emotional engagement and privacy) mean we need to think\ncarefully about how we deploy chatbots and the people who are interacting with\nthem. Dialogue research that uses human participants often requires getting permis-\nsion from the Institutional Review Board (IRB) of your institution.\nIn the next section we introduce some basic properties of human conversation.\nWe then turn in the rest of the chapter to the two basic paradigms for conversational\ninteraction: frame-based dialogue systems and chatbots.\n15.1 Properties of Human Conversation\nConversation between humans is an intricate and complex joint activity. Before\nwe attempt to design a dialogue system to converse with humans, it is crucial to\nunderstand something about how humans converse with each other. Consider some\nof the phenomena that occur in the conversation between a human travel agent and\na human client excerpted in Fig. 15.2.\nC1: . . . I need to travel in May.\nA2: And, what day in May did you want to travel?\nC3: OK uh I need to be there for a meeting that\u2019s from the 12th to the 15th.\nA4: And you\u2019re \ufb02ying into what city?\nC5: Seattle.\nA6: And what time would you like to leave Pittsburgh?\nC7: Uh hmm I don\u2019t think there\u2019s many options for non-stop.\nA8: Right. There\u2019s three non-stops today.\nC9: What are they?\nA10: The \ufb01rst one departs PGH at 10:00am arrives Seattle at 12:05 their time.\nThe second \ufb02ight departs PGH at 5:55pm, arrives Seattle at 8pm. And the\nlast \ufb02ight departs PGH at 8:15pm arrives Seattle at 10:28pm.\nC11: OK I\u2019ll take the 5ish \ufb02ight on the night before on the 11th.\nA12: On the 11th? OK. Departing at 5:55pm arrives Seattle at 8pm, U.S. Air\n\ufb02ight 115.\nC13: OK.\nA14: And you said returning on May 15th?\nC15: Uh, yeah, at the end of the day.\nA16: OK. There\u2019s #two non-stops . . . #\nC17: #Act. . . actually #, what day of the week is the 15th?\nA18: It\u2019s a Friday.\nC19: Uh hmm. I would consider staying there an extra day til Sunday.\nA20: OK. . . OK. On Sunday I have . . .\nFigure 15.2 Part of a phone conversation between a human travel agent (A) and human\nclient (C). The passages framed by # in A 16and C 17indicate overlaps in speech.\nTurns\nA dialogue is a sequence of turns (C1, A2, C3, and so on), each a single contribution turn\nfrom one speaker to the dialogue (as if in a game: I take a turn, then you take a turn, 4CHAPTER 15 \u2022 C HATBOTS & D IALOGUE SYSTEMS\nthen me, and so on). There are 20 turns in Fig. 15.2. A turn can consist of a sentence\n(like C 1), although it might be as short as a single word (C 13) or as long as multiple\nsentences (A 10).\nTurn structure has important implications for spoken dialogue. A human has\nto know when to stop talking; the client interrupts (in A 16and C 17), so a system\nthat was performing this role must know to stop talking (and that the user might be\nmaking a correction). A system also has to know when to start talking. For example,\nmost of the time in conversation, speakers start their turns almost immediately after\nthe other speaker \ufb01nishes, without a long pause, because people are can usually\npredict when the other person is about to \ufb01nish talking. Spoken dialogue systems\nmust also detect whether a user is done speaking, so they can process the utterance\nand respond. This task\u2014called endpointing orendpoint detection \u2014 can be quite endpointing\nchallenging because of noise and because people often pause in the middle of turns.\nSpeech Acts\nA key insight into conversation\u2014due originally to the philosopher Wittgenstein\n(1953) but worked out more fully by Austin (1962)\u2014is that each utterance in a\ndialogue is a kind of action being performed by the speaker. These actions are com-\nmonly called speech acts ordialogue acts : here\u2019s one taxonomy consisting of 4 speech acts\nmajor classes (Bach and Harnish, 1979):\nConstatives: committing the speaker to something\u2019s being the case ( answering ,claiming ,\ncon\ufb01rming ,denying ,disagreeing ,stating )\nDirectives: attempts by the speaker to get the addressee to do something ( advising ,ask-\ning,forbidding ,inviting ,ordering ,requesting )\nCommissives: committing the speaker to some future course of action ( promising, planning,\nvowing, betting, opposing )\nAcknowledgments: express the speaker\u2019s attitude regarding the hearer with respect to some so-\ncial action ( apologizing ,greeting ,thanking ,accepting an acknowledgment )\nA user asking a person or a dialogue system to do something (\u2018Turn up the mu-\nsic\u2019) is issuing a D IRECTIVE . Asking a question that requires an answer is also\na way of issuing a D IRECTIVE : in a sense when the system says (A 2) \u201cwhat day\nin May did you want to travel?\u201d it\u2019s as if the system is (very politely) command-\ning the user to answer. By contrast, a user stating a constraint (like C 1\u2018I need to\ntravel in May\u2019) is issuing a C ONSTATIVE . A user thanking the system is issuing\nan A CKNOWLEDGMENT . The speech act expresses an important component of the\nintention of the speaker (or writer) in saying what they said.\nGrounding\nA dialogue is not just a series of independent speech acts, but rather a collective act\nperformed by the speaker and the hearer. Like all collective acts, it\u2019s important for\nthe participants to establish what they both agree on, called the common groundcommon\nground\n(Stalnaker, 1978). Speakers do this by grounding each other\u2019s utterances. Ground- grounding\ning means acknowledging that the hearer has understood the speaker (Clark, 1996).\n(People need grounding for non-linguistic actions as well; the reason an elevator but-\nton lights up when it\u2019s pressed is to acknowledge that the elevator has indeed been\ncalled, essentially grounding your action of pushing the button (Norman, 1988).)\nHumans constantly ground each other\u2019s utterances. We can ground by explicitly\nsaying \u201cOK\u201d, as the agent does in A 8or A 10. Or we can ground by repeating what\nthe other person says; in utterance A 2the agent repeats \u201cin May\u201d, demonstrating her 15.1 \u2022 P ROPERTIES OF HUMAN CONVERSATION 5\nunderstanding to the client. Or notice that when the client answers a question, the\nagent begins the next question with \u201cAnd\u201d. The \u201cAnd\u201d implies that the new question\nis \u2018in addition\u2019 to the old question, again indicating to the client that the agent has\nsuccessfully understood the answer to the last question.\nSubdialogues and Dialogue Structure\nConversations have structure. Consider, for example, the local structure between\nspeech acts discussed in the \ufb01eld of conversation analysis (Sacks et al., 1974).conversation\nanalysis\nQUESTIONS set up an expectation for an ANSWER . PROPOSALS are followed by\nACCEPTANCE (orREJECTION ). C OMPLIMENTS (\u201cNice jacket!\u201d) often give rise to\nDOWNPLAYERS (\u201cOh, this old thing?\u201d). These pairs, called adjacency pairs are adjacency pair\ncomposed of a \ufb01rst pair part and a second pair part (Schegloff, 1968), and these\nexpectations can help systems decide what actions to take.\nHowever, dialogue acts aren\u2019t always followed immediately by their second pair\npart. The two parts can be separated by a side sequence (Jefferson 1972) or sub- side sequence\ndialogue . For example utterances C 17to A 20constitute a correction subdialogue subdialogue\n(Litman 1985, Litman and Allen 1987, Chu-Carroll and Carberry 1998):\nC17: #Act. . . actually#, what day of the week is the 15th?\nA18: It\u2019s a Friday.\nC19: Uh hmm. I would consider staying there an extra day til Sunday.\nA20: OK. . . OK. On Sunday I have . . .\nThe question in C 17interrupts the prior discourse, in which the agent was looking\nfor a May 15 return \ufb02ight. The agent must answer the question and also realize that\n\u2018\u2019I would consider staying...til Sunday\u201d means that the client would probably like to\nchange their plan, and now go back to \ufb01nding return \ufb02ights, but for the 17th.\nAnother side sequence is the clari\ufb01cation question , which can form a subdia-\nlogue between a REQUEST and a RESPONSE . This is especially common in dialogue\nsystems where speech recognition errors causes the system to have to ask for clari-\n\ufb01cations or repetitions like the following:\nUser: What do you have going to UNKNOWN WORD on the 5th?\nSystem: Let\u2019s see, going where on the 5th?\nUser: Going to Hong Kong.\nSystem: OK, here are some \ufb02ights...\nIn addition to side-sequences, questions often have presequences , like the fol- presequence\nlowing example where a user starts with a question about the system\u2019s capabilities\n(\u201cCan you make train reservations\u201d) before making a request.\nUser: Can you make train reservations?\nSystem: Yes I can.\nUser: Great, I\u2019d like to reserve a seat on the 4pm train to New York.\nInitiative\nSometimes a conversation is completely controlled by one participant. For exam-\nple a reporter interviewing a chef might ask questions, and the chef responds. We\nsay that the reporter in this case has the conversational initiative (Carbonell, 1970; initiative\nNickerson, 1976). In normal human-human dialogue, however, it\u2019s more common\nfor initiative to shift back and forth between the participants, as they sometimes\nanswer questions, sometimes ask them, sometimes take the conversations in new di-\nrections, sometimes not. You may ask me a question, and then I respond asking you 6CHAPTER 15 \u2022 C HATBOTS & D IALOGUE SYSTEMS\nto clarify something you said, which leads the conversation in all sorts of ways. We\ncall such interactions mixed initiative (Carbonell, 1970).\nFull mixed initiative, while the norm for human-human conversations, can be\ndif\ufb01cult for dialogue systems. The most primitive dialogue systems tend to use\nsystem-initiative , where the system asks a question and the user can\u2019t do anything\nuntil they answer it, or user-initiative like simple search engines, where the user\nspeci\ufb01es a query and the system passively responds. Even modern large language\nmodel-based dialogue systems, which come much closer to using full mixed initia-\ntive, often don\u2019t have completely natural initiative switching. Getting this right is an\nimportant goal for modern systems.\nInference and Implicature\nInference is also important in dialogue understanding. Consider the client\u2019s response\nC2, repeated here:\nA2: And, what day in May did you want to travel?\nC3: OK uh I need to be there for a meeting that\u2019s from the 12th to the 15th.\nNotice that the client does not in fact answer the agent\u2019s question. The client\nmerely mentions a meeting at a certain time. What is it that licenses the agent to\ninfer that the client is mentioning this meeting so as to inform the agent of the travel\ndates?\nThe speaker seems to expect the hearer to draw certain inferences; in other\nwords, the speaker is communicating more information than seems to be present\nin the uttered words. This kind of example was pointed out by Grice (1975, 1978)\nas part of his theory of conversational implicature .Implicature means a particu- implicature\nlar class of licensed inferences. Grice proposed that what enables hearers to draw\nthese inferences is that conversation is guided by a set of maxims , general heuristics\nthat play a guiding role in the interpretation of conversational utterances. One such\nmaxim is the maxim of relevance which says that speakers attempt to be relevant, relevance\nthey don\u2019t just utter random speech acts. When the client mentions a meeting on the\n12th, the agent reasons \u2018There must be some relevance for mentioning this meeting.\nWhat could it be?\u2019. The agent knows that one precondition for having a meeting\n(at least before Web conferencing) is being at the place where the meeting is held,\nand therefore that maybe the meeting is a reason for the travel, and if so, then since\npeople like to arrive the day before a meeting, the agent should infer that the \ufb02ight\nshould be on the 11th.\nThese subtle characteristics of human conversations ( turns ,speech acts ,ground-\ning,dialogue structure ,initiative , and implicature ) are among the reasons it is dif-\n\ufb01cult to build dialogue systems that can carry on natural conversations with humans.\nMany of these challenges are active areas of dialogue systems research.\n15.2 Frame-Based Dialogue Systems\nAtask-based dialogue system has the goal of helping a user solve a speci\ufb01c task\nlike making a travel reservation or buying a product. Task-based dialogue systems\nare based around frames , \ufb01rst introduced in the early in\ufb02uential GUS system for frame\nGUS travel planning (Bobrow et al., 1977). Frames are knowledge structures representing\nthe details of the user\u2019s task speci\ufb01cation. Each frame consists of a collection of\nslots , each of which can take a set of possible values . Together a set of frames is slot 15.2 \u2022 F RAME -BASED DIALOGUE SYSTEMS 7\nsometimes called a domain ontology .\nHere we\u2019ll describe the most well-studied frame-based architecture, the dialogue-\nstate architecture, made up of the six components shown in Fig. 15.3. In the next\nsections we\u2019ll introduce four of them, after introducing the idea of frames (deferring\nthe speech recognition and synthesis components to Chapter 16).\nFigure 15.3 Architecture of a dialogue-state system for task-oriented dialogue from Williams et al. (2016).\n15.2.1 Frames and Slot Filling\nThe frame and its slots in a task-based dialogue system specify what the system\nneeds to know to perform its task. A hotel reservation system needs dates and loca-\ntions. An alarm clock system needs a time. The system\u2019s goal is to \ufb01ll the slots in\nthe frame with the \ufb01llers the user intends, and then perform the relevant action for\nthe user (answering a question, or booking a \ufb02ight).\nFig. 15.4 shows a sample frame for booking air travel, with some sample ques-\ntions used for \ufb01lling slots. In the simplest frame-based systems (including most com-\nmercial assistants until quite recently), these questions are pre-written templates, but\nin more sophisticated systems, questions are generated on-the-\ufb02y. The slot \ufb01llers are\noften constrained to a particular semantic type, like type CITY (taking on values like\nSan Francisco , orHong Kong ) or DATE ,AIRLINE , or TIME .\nSlot Type Example Question\nORIGIN CITY city \u201cFrom what city are you leaving?\u201d\nDESTINATION CITY city \u201cWhere are you going?\u201d\nDEPARTURE TIME time \u201cWhen would you like to leave?\u201d\nDEPARTURE DATE date \u201cWhat day would you like to leave?\u201d\nARRIV AL TIME time \u201cWhen do you want to arrive?\u201d\nARRIV AL DATE date \u201cWhat day would you like to arrive?\u201d\nFigure 15.4 A frame in a frame-based dialogue system, showing the type of each slot and\na sample question used to \ufb01ll the slot. 8CHAPTER 15 \u2022 C HATBOTS & D IALOGUE SYSTEMS\nMany domains require multiple frames. Besides frames for car or hotel reser-\nvations, we might need other frames for things like general route information (for\nquestions like Which airlines \ufb02y from Boston to San Francisco? ), That means the\nsystem must be able to disambiguate which slot of which frame a given input is\nsupposed to \ufb01ll.\nThe task of slot-\ufb01lling is usually combined with two other tasks, to extract 3\nthings from each user utterance. The \ufb01rst is domain classi\ufb01cation : is this user for\nexample talking about airlines, programming an alarm clock, or dealing with their\ncalendar? The second is user intent determination : what general task or goal is theintent\ndetermination\nuser trying to accomplish? For example the task could be to Find a Movie, or Show\na Flight, or Remove a Calendar Appointment. Together, the domain classi\ufb01cation\nand intent determination tasks decide which frame we are \ufb01lling. Finally, we need\nto do slot \ufb01lling itself: extract the particular slots and \ufb01llers that the user intends the slot \ufb01lling\nsystem to understand from their utterance with respect to their intent. From a user\nutterance like this one:\nShow me morning flights from Boston to San Francisco on Tuesday\na system might want to build a representation like:\nDOMAIN: AIR-TRAVEL INTENT: SHOW-FLIGHTS\nORIGIN-CITY: Boston DEST-CITY: San Francisco\nORIGIN-DATE: Tuesday ORIGIN-TIME: morning\nSimilarly an utterance like this: should give an intent like this:\nWake me tomorrow at 6 DOMAIN: ALARM-CLOCK\nINTENT: SET-ALARM\nTIME: 2017-07-01 0600\nThe simplest dialogue systems use handwritten rules for slot-\ufb01lling, like this\nregular expression for recognizing the SET-ALARM intent:\nwake me (up) | set (the|an) alarm | get me up\nBut most systems use supervised machine-learning: each sentence in a training\nset is annotated with slots, domain, and intent, and a sequence model maps from\ninput words to slot \ufb01llers, domain and intent. For example we\u2019ll have pairs of sen-\ntences that are labeled for domain ( AIRLINE ) and intent ( SHOWFLIGHT ), and are also\nlabeled with BIO representations for the slots and \ufb01llers. (Recall from Chapter 17\nthat in BIO tagging we introduce a tag for the beginning (B) and inside (I) of each\nslot label, and one for tokens outside (O) any slot label.)\nO O O O O B-DES I-DES O B-DEPTIME I-DEPTIME O AIRLINE-SHOWFLIGHT\nI want to fly to San Francisco on Monday afternoon please EOS\nFig. 15.5 shows a typical architecture for inference. The input words w1:::wn\nare passed through a pretrained language model encoder, followed by a feedforward\nlayer and a softmax at each token position over possible BIO tags, with the output\na series of BIO tags s1:::sn. We generally combine the domain-classi\ufb01cation and\nintent-extraction tasks with slot-\ufb01lling by adding a domain concatenated with an\nintent as the desired output for the \ufb01nal EOS token.\nOnce the sequence labeler has tagged the user utterance, a \ufb01ller string can be ex-\ntracted for each slot from the tags (e.g., \u201cSan Francisco\u201d), and these word strings\ncan then be normalized to the correct form in the ontology (perhaps the airport 15.3 \u2022 D IALOGUE ACTS AND DIALOGUE STATE 9\nSanFranciscoonMondayEncodingsClassi\ufb01er+softmaxB-DESI-DESOB-DTIME\n\u2026d+i\n<EOS>Encoder \nFigure 15.5 Slot \ufb01lling by passing input words through an encoder, and then using a linear\nor feedforward layer followed by a softmax to generate a series of BIO tags. Here we also\nshow a \ufb01nal state: a domain concatenated with an intent.\ncode \u2018SFO\u2019), for example with dictionaries that specify that SF, SFO, and San Fran-\ncisco are synonyms. Often in industrial contexts, combinations of rules and machine\nlearning are used for each of these components.\nWe can make a very simple frame-based dialogue system by wrapping a small\namount of code around this slot extractor. Mainly we just need to ask the user\nquestions until all the slots are full, do a database query, then report back to the user,\nusing hand-built templates for generating sentences.\n15.2.2 Evaluating Task-Based Dialogue\nWe evaluate task-based systems by computing the task error rate , ortask success task error rate\nrate: the percentage of times the system booked the right plane \ufb02ight, or put the\nright event on the calendar. A more \ufb01ne-grained, but less extrinsic metric is the slot\nerror rate , the percentage of slots \ufb01lled with the correct values: slot error rate\nSlot Error Rate for a Sentence =# of inserted/deleted/subsituted slots\n# of total reference slots for sentence(15.1)\nFor example a system that extracted the slot structure below from this sentence:\n(15.2) Make an appointment with Chris at 10:30 in Gates 104\nSlot Filler\nPERSON Chris\nTIME 11:30 a.m.\nROOM Gates 104\nhas a slot error rate of 1/3, since the TIME is wrong. Instead of error rate, slot\nprecision, recall, and F-score can also be used. We can also measure ef\ufb01ciency\ncosts like the length of the dialogue in seconds or turns. ef\ufb01ciency costs\n15.3 Dialogue Acts and Dialogue State\nWhile the naive slot-extractor system described above can handle simple dialogues,\noften we want more complex interactions. For example, we might want to con\ufb01rm\nthat we\u2019ve understand the user, or ask them to repeat themselves. We can build a\nmore sophisticated system using dialogue acts anddialogue state . 10 CHAPTER 15 \u2022 C HATBOTS & D IALOGUE SYSTEMS\n15.3.1 Dialogue Acts\nDialogue acts are a generalization of speech acts that also represent grounding. The dialogue acts\nset of acts can be general, or can be designed for particular dialogue tasks.\nTag Sys User Description\nHELLO (a=x;b=y;:::)X X Open a dialogue and give info a=x;b=y;:::\nINFORM (a=x;b=y;:::)X X Give info a=x;b=y;:::\nREQUEST (a;b=x;:::)X X Request value for a given b=x;:::\nREQALTS (a=x;:::) cX Request alternative with a=x;:::\nCONFIRM (a=x;b=y;:::)X X Explicitly con\ufb01rm a=x;b=y;:::\nCONFREQ (a=x;:::;d)Xc Implicitly con\ufb01rm a=x;:::and request value of d\nSELECT (a=x;a=y)Xc Implicitly con\ufb01rm a=x;:::and request value of d\nAFFIRM (a=x;b=y;:::)X X Af\ufb01rm and give further info a=x;b=y;:::\nNEGATE (a=x) cX Negate and give corrected value a=x\nDENY (a=x) cX Deny that a=x\nBYE() X X Close a dialogue\nFigure 15.6 Dialogue acts used by the HIS restaurant recommendation system of Young\net al. (2010). The Sys and User columns indicate which acts are valid as system outputs and\nuser inputs, respectively.\nFigure 15.6 shows a tagset for a restaurant recommendation system, and Fig. 15.7\nshows these tags labeling a sample dialogue from the HIS system (Young et al.,\n2010). This example also shows the content of each dialogue act, which are the slot\n\ufb01llers being communicated. So the user might INFORM the system that they want\nItalian food near a museum, or CONFIRM with the system that the price is reasonable.\nUtterance Dialogue act\nU: Hi, I am looking for somewhere to eat. hello(task = find,type=restaurant)\nS: You are looking for a restaurant. What\ntype of food do you like?confreq(type = restaurant, food)\nU: I\u2019d like an Italian near the museum. inform(food = Italian, near=museum)\nS: Roma is a nice Italian restaurant near\nthe museum.inform(name = \"Roma\", type = restaurant,\nfood = Italian, near = museum)\nU: Is it reasonably priced? confirm(pricerange = moderate)\nS: Yes, Roma is in the moderate price\nrange.affirm(name = \"Roma\", pricerange =\nmoderate)\nU: What is the phone number? request(phone)\nS: The number of Roma is 385456. inform(name = \"Roma\", phone = \"385456\")\nU: Ok, thank you goodbye. bye()\nFigure 15.7 A dialogue from the HIS System of Young et al. (2010) using the dialogue acts in Fig. 15.6.\n15.3.2 Dialogue State Tracking\nThe job of the dialogue-state tracker is to determine the current state of the frame\n(the \ufb01llers of each slot), and the user\u2019s most recent dialogue act. The dialogue-state\nis not just the slot-\ufb01llers in the current sentence; it includes the entire state of the\nframe at this point, summarizing all of the user\u2019s constraints. Fig. 15.8 from Mrk \u02c7si\u00b4c\net al. (2017) shows the dialogue state after each turn.\nDialogue act detection is done just like domain or intent classi\ufb01cation, by passing\nthe input sentence through an encoder and adding an act classi\ufb01er. Often passing in\nthe prior dialogue act as well can improve classi\ufb01cation. And since dialogue acts 15.3 \u2022 D IALOGUE ACTS AND DIALOGUE STATE 11\nUser: I\u2019m looking for a cheaper restaurant\ninform(price=cheap)\nSystem: Sure. What kind - and where?\nUser: Thai food, somewhere downtown\ninform(price=cheap, food=Thai, area=centre)\nSystem: The House serves cheap Thai food\nUser: Where is it?\ninform(price=cheap, food=Thai, area=centre); request(address)\nSystem: The House is at 106 Regent Street\nFigure 15.8 The output of the dialogue state tracker after each turn (Mrk \u02c7si\u00b4c et al., 2017).\nplace some constraints on the slots and values, the tasks of dialogue-act detection and\nslot-\ufb01lling are often performed jointly. The state tracker can just take the output of\na slot-\ufb01lling sequence-model (Section 15.2.1) after each sentence, or do something\nmore complicated like training a classi\ufb01er to decide if a value has been changed.\nA special case: detecting correction acts. If a dialogue system misrecognizes\nor misunderstands an utterance, users will repeat or reformulate the utterance. De-\ntecting these user correction acts is quite important, especially for spoken lan-user correction\nacts\nguage. Ironically, corrections are actually harder to recognize than normal sentences\n(Swerts et al., 2000), because users who are frustrated adjust their speech in a way\nthat is dif\ufb01cult for speech recognizers (Goldberg et al., 2003). For example speak-\ners often use a prosodic style for corrections called hyperarticulation , in which thehyperarticula-\ntion\nutterance is louder or longer or exaggerated in pitch, such as I said BAL-TI-MORE,\nnot Boston (Wade et al. 1992, Levow 1998, Hirschberg et al. 2001). Detecting acts\ncan be part of the general dialogue act detection classi\ufb01er, or can make use of spe-\ncial features beyond the words, like those shown below (Levow 1998, Litman et al.\n1999, Hirschberg et al. 2001, Bulyko et al. 2005, Awadallah et al. 2015).\nfeatures examples\nsemantic embedding similarity between correction and user\u2019s prior utterance\nphonetic phonetic overlap between candidate correction act and user\u2019s prior utterance\n(i.e. \u201cWhatsApp\u201d may be incorrectly recognized as \u201cWhat\u2019s up\u201d)\nprosodic hyperarticulation, increases in F0 range, pause duration, and word duration\nASR ASR con\ufb01dence, language model probability\n15.3.3 Dialogue Policy: Which act to generate\nIn early commercial frame-based systems, the dialogue policy is simple: ask ques-\ntions until all the slots are full, do a database query, then report back to the user. A\nmore sophisticated dialogue policy can help a system decide when to answer the dialogue policy\nuser\u2019s questions, when to instead ask the user a clari\ufb01cation question, and so on. A\ndialogue policy thus decides what dialogue act to generate. Choosing a dialogue act\nto generate, along with its arguments, is sometimes called content planning .content\nplanning\nLet\u2019s see how to do this for some important dialogue acts. Dialogue systems, es-\npecially speech systems, often misrecognize the users\u2019 words or meaning. To ensure\nsystem and user share a common ground, systems must con\ufb01rm understandings with\nthe user or reject utterances that the system don\u2019t understand. A system might use\nanexplicit con\ufb01rmation act to con\ufb01rm with the user, like Is that correct? below:explicit\ncon\ufb01rmation 12 CHAPTER 15 \u2022 C HATBOTS & D IALOGUE SYSTEMS\nU:I\u2019d like to \ufb02y from Denver Colorado to New York City on September\ntwenty \ufb01rst in the morning on United Airlines\nS:Let\u2019s see then. I have you going from Denver Colorado to New York\non September twenty \ufb01rst. Is that correct?\nWhen using an implicit con\ufb01rmation act, a system instead grounds more im-implicit\ncon\ufb01rmation\nplicitly, for example by repeating the system\u2019s understanding as part of asking the\nnext question, as Shanghai is con\ufb01rmed in passing in this example:\nU:I want to travel to to Shanghai\nS:When do you want to travel to Shanghai?\nThere\u2019s a tradeoff. Explicit con\ufb01rmation makes it easier for users to correct\nmisrecognitions by just answering \u201cno\u201d to the con\ufb01rmation question. But explicit\ncon\ufb01rmation is time-consuming and awkward (Danieli and Gerbino 1995, Walker\net al. 1998). We also might want an act that expresses lack of understanding: rejec-\ntion, for example with a prompt like I\u2019m sorry, I didn\u2019t understand that . To decide rejection\namong these acts, we can make use of the fact that ASR systems often compute\ntheir con\ufb01dence in their transcription (often based on the log-likelihood the system\nassigns the sentence). A system can thus choose to explicitly con\ufb01rm only low-\ncon\ufb01dence sentences. Or systems might have a four-tiered level of con\ufb01dence with\nthree thresholds a,b, and g:\n<alow con\ufb01dence reject\n\u0015aabove the threshold con\ufb01rm explicitly\n\u0015bhigh con\ufb01dence con\ufb01rm implictly\n\u0015gvery high con\ufb01dence don\u2019t con\ufb01rm at all\n15.3.4 Natural language generation: Sentence Realization\nrecommend(restaurant name= Au Midi, neighborhood = midtown,\ncuisine = french)\n1 Au Midi is in Midtown and serves French food.\n2 There is a French restaurant in Midtown called Au Midi.\nFigure 15.9 Sample inputs to the sentence realization phase of NLG, showing the dialogue\nact and attributes prespeci\ufb01ed by the content planner, and two distinct potential output sen-\ntences to be generated. From the restaurant recommendation system of Nayak et al. (2017).\nOnce a dialogue act has been chosen, we need to generate the text of the re-\nsponse to the user. This part of the generation process is called sentence realiza-\ntion. Fig. 15.9 shows a sample input/output for the sentence realization phase. Thesentence\nrealization\ncontent planner has chosen the dialogue act RECOMMEND and some slots (name,\nneighborhood, cuisine) and \ufb01llers. The sentence realizer generates a sentence like\nlines 1 or 2 (by training on examples of representation/sentence pairs from a corpus\nof labeled dialogues). Because we won\u2019t see every restaurant or attribute in every\npossible wording, we can delexicalize : generalize the training examples by replac- delexicalize\ning speci\ufb01c slot value words in the training set with a generic placeholder token\nrepresenting the slot. Fig. 15.10 shows the sentences in Fig. 15.9 delexicalized.\nWe can map from frames to delexicalized sentences with an encoder decoder\nmodel (Mrk \u02c7si\u00b4c et al. 2017, inter alia), trained on hand-labeled dialogue corpora like\nMultiWOZ (Budzianowski et al., 2018). The input to the encoder is a sequence of 15.4 \u2022 C HATBOTS 13\nrecommend(restaurant name= Au Midi, neighborhood = midtown,\ncuisine = french)\n1restaurant name is inneighborhood and serves cuisine food.\n2 There is a cuisine restaurant in neighborhood called restaurant name .\nFigure 15.10 Delexicalized sentences that can be used for generating many different relex-\nicalized sentences. From the restaurant recommendation system of Nayak et al. (2017).\ndecentservice:RECOMMENDcuisine:null[name]hasdecentserviceENCODERDECODER\nFigure 15.11 An encoder decoder sentence realizer mapping slots/\ufb01llers to English.\ntokens xtthat represent the dialogue act (e.g., RECOMMEND ) and its arguments (e.g.,\nservice:decent, cuisine:null ) (Nayak et al., 2017), as in Fig. 15.11.\nThe decoder outputs the delexicalized English sentence \u201c name has decent ser-\nvice\u201d, which we can then relexicalize , i.e. \ufb01ll back in correct slot values, resulting relexicalize\nin \u201cAu Midi has decent service\u201d.\n15.4 Chatbots\nChatbots are systems that can carry on extended conversations with the goal of chatbot\nmimicking the unstructured conversations or \u2018chats\u2019 characteristic of informal human-\nhuman interaction. While early systems like ELIZA (Weizenbaum, 1966) or PARRY\n(Colby et al., 1971) had theoretical goals like testing theories of psychological coun-\nseling, for most of the last 50 years chatbots have been designed for entertainment.\nThat changed with the recent rise of neural chatbots like ChatGPT, which incor-\nporate solutions to NLP tasks like question answering, writing tools, or machine\ntranslation into a conversational interface. A conversation with ChatGPT is shown\nin Fig. 15.12. In this section we describe neural chatbot architectures and datasets.\n[TBD]\nFigure 15.12 A conversation with ChatGPT.\n15.4.1 Training chatbots\nData Chatbots are generally trained on a training set that includes standard large\nlanguage model training data of the type discussed in Section ??: versions of the\nweb from the Common Crawl, including news sites, Wikipedia, as well as books.\nFor training chatbots, it is common to additionally add lots of dialogue data.\nThis can include datasets created speci\ufb01cally for training chatbots by hiring\nspeakers of the language to have conversations, such as by having them take on\npersonas or talk about knowledge provided to them. For example the Topical-Chat\ndataset has 11K crowdsourced conversations spanning 8 broad topics (Gopalakrish-\nnan et al., 2019), the E MPATHETIC DIALOGUES includes 25K crowdsourced con- 14 CHAPTER 15 \u2022 C HATBOTS & D IALOGUE SYSTEMS\nversations grounded in a speci\ufb01c situation where a speaker was feeling a speci\ufb01c\nemotion (Rashkin et al., 2019), and the SaFeRDialogues dataset (Ung et al., 2022)\nhas 8k dialogues demonstrating graceful responses to conversational feedback about\nsafety failures.\nSuch datasets are far too small to train a language model alone, and so it\u2019s com-\nmon to also pretrain on large datasets of pseudo-conversations drawn from Twitter\n(Ritter et al., 2010), Reddit (Roller et al., 2021), Weibo ( \u5fae\u535a), and other social\nmedia platforms. To turn social media data into data that has the structure of a con-\nversation, we can treat any post on the platform as the \ufb01rst turn in a conversation,\nand the sequence of comments/replies as subsequent turns in that conversation.\nDatasets from the web can be enormously toxic, so it\u2019s crucial to \ufb01lter the di-\nalogues \ufb01rst. This can be done by using the same toxicity classi\ufb01ers we describe\nbelow in the \ufb01ne-tuning section.\nArchitecture For training chatbots, it\u2019s most common to use the standard causal\nlanguage model architecture, in which the model predicts each word given all the\nprior words, and the loss is the standard language modeling loss. Fig. 15.13 shows a\nstandard training setup; no different than language model training in Chapter 9. The\nonly difference is the data, which has the addition of signi\ufb01cant conversation and\npseudo-conversation data as described in the prior section. As usual, the left context\ncan include the entire prior conversation (or as much as \ufb01ts in the context window).\nTransformerBlocksLM head\ngotpromoted!<s>gotpromoted!<s>Next wordCongratsLM Loss\u2026LM headLM headLM headLM headLM head\nICongrats!\u2026LM headLM head!-log y!-log yCongrats-log y<s>-log y!-log ypromoted-log ygot\u2026\u2026\u2026\u2026\u2026\u2026\nFigure 15.13 Training a causal (decoder-only) language model for a chatbot.\nAn alternative is to use the encoder-decoder architecture of Chapter 13. In this\ncase the entire conversation up to the last turn (as much as \ufb01ts in the context) is\npresented to the encoder, and the decoder generates the next turn.\npromotedgot!<s>Congrats!ENCODERDECODERI\nFigure 15.14 An alternative: an encoder-decoder language model for a chatbot.\nIn practice, dialogue systems require additional customization beyond just pre-\ntraining on dialogue data. In the next few sections we\u2019ll discuss various stages of 15.4 \u2022 C HATBOTS 15\n\ufb01ne-tuning that can be used for this customization.\n15.4.2 Fine Tuning for Quality and Safety\nIt is a common practice for dialogue systems to use further labeled data for \ufb01ne-\ntuning. One function of this \ufb01ne-tuning step is to improve the quality of the dialogue,\ntraining the system to produce responses that are sensible and interesting. Another\nfunction might be to improve safety, keeping a dialogue system from suggesting\nharmful actions (like \ufb01nancial fraud, medical harm, inciting hatred, or abusing the\nuser or other people).\nIn the simplest method for improving quality and safety, speakers of the lan-\nguage are given an initial prompt and instructions to have high-quality, safe dia-\nlogues. They then interact with an initial dialogue system and their responses are\nused to \ufb01netune the model, usually as part of the instruct tuning step we introduced\nin Chapter 12. Thus a dialogue system learns to answer questions, follow other\ninstructions, and also carry on high-quality, safe dialogues, in a single multi-task\nlearning format.\nWhile \ufb01ne-tuning on positive examples is helpful, it is generally insuf\ufb01cient and\nso it is common to add more discriminative data that speci\ufb01cally downweights low-\nquality or harmful responses. The simplest paradigm for this is to train a model to\npredict turn-level safety and quality values, by training on human-labeled ratings.\nSuch ratings might be collected by \ufb01rst having speakers of the language carry on\ndialogues with a system, and then a second set of people act as labelers to label\nevery system turn for its quality and safety, resulting in a binary label for quality and\nsafety for each turn.\nOnce a dataset has been created with these labels, a language model can be used\nin a classi\ufb01cation task to label the quality and safety of a turn. For example in the\nLamda system (Cohen et al., 2022), a single language model is used in two phases,\nroughly corresponding to generative and discriminative tasks: \ufb01rst generating a re-\nsponse, and then generating a label. In the generative phase, the model is given the\nprior turn and a special RESPONSE token and generates the blue response turn. (In\ntraining, the training loss is given only for the blue response):\n\u201cWhat\u2019s up? RESPONSE Not much.\nIn a second, discriminative phase, the model is \ufb01ne-tuned to see an attribute\n(SENSIBLE, INTERESTING, UNSAFE) and then to predict a 0 or 1 value, again\nwith training losses given only for the blue value.\nWhat\u2019s up? RESPONSE Not much. SENSIBLE 1\nWhat\u2019s up? RESPONSE Not much. INTERESTING 0\nWhat\u2019s up? RESPONSE Not much. UNSAFE 0\nTo use the system in inference, the model \ufb01rst generates a response given the context,\nand then it is given the attribute and asked to generate a rating. The result is a\ngenerated turn along with a label. This label isn\u2019t shown to the user but can be use\nfor \ufb01ltering, either at training time or at deployment time. For example, the system\ncan generate multiple potential responses, \ufb01lter out any response that is unsafe, and\nreturn to the user the highest ranking response. 16 CHAPTER 15 \u2022 C HATBOTS & D IALOGUE SYSTEMS\n15.4.3 Learning to perform retrieval as part of responding\nModern chatbots are also trained to do retrieval (calls to Google or Bing or other\nsearch engines) as a component of generating a response. This is generally done\nby having a sort of fake dialogue participant called something like Search Query\nwhose utterances are passed to a search engine. The system can be encourged to do\nsearches either by prompting or by \ufb01ne-tuning.\nFor example in the Sparrow chatbot (Glaese et al., 2022), the prompt includes\na few sample conversations that include two special users, Search Query , and\nSearch Results :\n...[this is preceded by other parts of the prompt]\nUser: What color is the sky?\nSparrow: Blue.\nUser: Why?\nSearch Query: why is sky blue\nSearch Results: Page title: Sky Except for direct sunlight, most of the light in\nthe daytime sky is caused by scattering, which is dominated by\na small-particle limit called Rayleigh scattering. The scattering\ndue to molecule-sized particles (as in air) is greater in the di-\nrections both toward and away from the source of light than it\nis in directions perpendicular to the incident path. Scattering is\nsigni\ufb01cant for light at all visible wavelengths, but is stronger at\nthe shorter (bluer) end of the visible spectrum, meaning that the\nscattered light is bluer than its source: the Sun. The remain-\ning direct sunlight, having lost some of its shorter-wavelength\ncomponents, appears slightly less blue.\nSparrow: Because of the Rayleigh scattering. It causes short waves of\nblue light to be scattered more than the other colours in the spec-\ntrum.\nUser: Who was Raleigh?\nSearch Query: who was Rayleigh\n... ...\nFrom these prompts, the system learns to generate texts with Search Query\nturns for fact-based questions from the user, and these are passed to a search engine\nto generate the Search Results turns.\nAlternatively, systems can be \ufb01netuned to to know when to use a search en-\ngine. For example, labelers can interact with a system, fact check each of the re-\nsponses, and whenever the system emits an incorrect response, perform the web\nsearch queries that the system should have used to check its answer, and then the in-\nteration is recorded and used for \ufb01ne-tuning. Or labelers can look at a transcript of a\nlanguage model carrying on a dialogue, and similarly mark every place where a fact\nwas wrong (or out-of-date) and write the set of search queries that would have been\nappropriate. A system is then \ufb01ne-tuned to generate search query turns which\nare again passed to a search engine to generate the search responses . The set\nof pages or snippets returned by the search engine in the search response turn are\nthen treated as the context for generation, similarly to the retrieval-based question-\nanswering methods of Chapter 14. 15.5 \u2022 D IALOGUE SYSTEM DESIGN 17\n15.4.4 RLHF\nA more sophisticated family of methods uses reinforcement learning to learn to\nmatch human preferences for generated turns. In this method, RLHF for Rein- RLHF\nforcement Learning from Human Feedback, we give a system a dialogue context\nand sample two possible turns from the language model. We then have humans la-\nbel which of the two is better, creating a large dataset of sentence pairs with human\npreferences. These pairs are used to train a dialogue policy, and reinforcement learn-\ning is used to train the language model to generate turns that have higher rewards\n(Christiano et al., 2017; Ouyang et al., 2022). While using RLHF is the current state\nof the art at the time of this writing, a number of alternatives have been recently\ndeveloped that don\u2019t require reinforcement learning (Rafailov et al., 2023, e.g.,) and\nso this aspect of the \ufb01eld is changing very quickly.\n15.4.5 Evaluating Chatbots\nChatbots are evaluated by humans, who assign a score. This can be the human who\ntalked to the chatbot ( participant evaluation ) or a third party who reads a transcript\nof a human/chatbot conversation ( observer evaluation ). In the participant evalua-\ntion of See et al. (2019), the human evaluator chats with the model for six turns and\nrates the chatbot on 8 dimensions capturing conversational quality: avoiding repe-\ntition, interestingness, making sense, \ufb02uency, listening, inquisitiveness, humanness\nand engagingness on Likert scales like these:\nEngagingness How much did you enjoy talking to this user?\n\u000fNot at all \u000fA little \u000fSomewhat \u000fA lot\nMaking sense How often did this user say something which did NOT make sense?\n\u000fNever made any sense \u000fMost responses didn\u2019t make sense \u000fSome re-\nsponses didn\u2019t make sense \u000fEverything made perfect sense\nObserver evaluations use third party annotators to look at the text of a complete\nconversation. Sometimes we\u2019re interested in having raters assign a score to each\nsystem turn; for example (Artstein et al., 2009) have raters mark how coherent each\nturn is. Often, however, we just want a single high-level score to know if system A\nis better than system B. The acute-eval metric (Li et al., 2019) is such an observer acute-eval\nevaluation in which annotators look at two separate human-computer conversations\nand choose the system which performed better on four metrics: engagingness, inter-\nestingness, humanness, and knowledgability.\n15.5 Dialogue System Design\nBecause of the important role of the user, the \ufb01eld of dialogue systems is closely\nlinked with Human-Computer Interaction (HCI). This is especially true for task-\noriented dialogue and assistants, where the design of dialogue strategies, sometimes\ncalled voice user interface design, generally follows user-centered design princi-voice user\ninterface\nples (Gould and Lewis, 1985):\n1. Study the user and task: Understand the users and the task by interviewing\nusers, investigating similar systems, and studying related human-human dialogues.\n2. Build simulations and prototypes: A crucial tool in building dialogue systems\nis the Wizard-of-Oz system . In wizard systems, the users interact with what theyWizard-of-Oz\nsystem 18 CHAPTER 15 \u2022 C HATBOTS & D IALOGUE SYSTEMS\nthink is a program but is in fact a human \u201cwizard\u201d disguised by a software interface\n(Gould et al. 1983, Good et al. 1984, Fraser and\nGilbert 1991). The name comes from the chil-\ndren\u2019s book The Wizard of Oz (Baum, 1900),\nin which the wizard turned out to be a simu-\nlation controlled by a man behind a curtain or\nscreen. A wizard system can be used to test out\nan architecture before implementation; only the\ninterface software and databases need to be in\nplace. The wizard gets input from the user, uses\na database interface to run queries based on the\nuser utterance, and then outputs sentences, ei-\nther by typing them or speaking them.\nWizard-of-Oz systems are not a perfect\nsimulation, since the wizard doesn\u2019t exactly\nsimulate the errors or limitations of a real sys-\ntem; but wizard studies can still provide a useful \ufb01rst idea of the domain issues.\n3. Iteratively test the design on users: An iterative design cycle with embedded\nuser testing is essential in system design (Nielsen 1992, Cole et al. 1997, Yankelovich\net al. 1995, Landauer 1995). For example in a well-known incident, an early dia-\nlogue system required the user to press a key to interrupt the system (Stifelman et al.,\n1993). But user testing showed users barged in (interrupted, talking over the sys- barged in\ntem), which led to a redesign of the system to recognize overlapped speech. It\u2019s also\nimportant to incorporate value sensitive design , in which we carefully consider dur-value sensitive\ndesign\ning the design process the bene\ufb01ts, harms and possible stakeholders of the resulting\nsystem (Friedman et al. 2017, Friedman and Hendry 2019).\n15.5.1 Ethical Issues in Dialogue System Design\nEthical issues have been key to how we think about designing arti\ufb01cial agents since\nwell before we had dialogue systems. Mary Shelley (depicted below) centered her\nnovel Frankenstein around the problem of creating arti\ufb01cial agents without consider-\ning\nethical and humanistic concerns. One issue is the\nsafety of users. If users seek information from di-\nalogue systems in safety-critical situations like ask-\ning medical advice, or in emergency situations, or\nwhen indicating the intentions of self-harm, incorrect\nadvice can be dangerous and even life-threatening.\nFor example (Bickmore et al., 2018) gave participants\nmedical problems to pose to three commercial di-\nalogue systems (Siri, Alexa, Google Assistant) and\nasked them to determine an action to take based on\nthe system responses; many of the proposed actions,\nif actually taken, would have led to harm or death.\nA system can also harm users by verbally attacking them, or creating represen-\ntational harms (Blodgett et al., 2020) by generating abusive or harmful stereotypes\nthat demean particular groups of people. Both abuse and stereotypes can cause psy-\nchological harm to users. Microsoft\u2019s 2016 Tay chatbot, for example, was taken Tay\nof\ufb02ine 16 hours after it went live, when it began posting messages with racial slurs, 15.6 \u2022 S UMMARY 19\nconspiracy theories, and personal attacks on its users. Tay had learned these biases\nand actions from its training data, including from users who seemed to be purposely\nteaching the system to repeat this kind of language (Neff and Nagy 2016). Hender-\nson et al. (2017) examined dialogue datasets used to train corpus-based chatbots and\nfound toxic and abusive language, especially in social media corpora like Twitter\nand Reddit, and indeed such language then appears in the text generated by lan-\nguage models and dialogue systems (Gehman et al. 2020; Xu et al. 2020) which\ncan even amplify the bias from the training data (Dinan et al., 2020). Liu et al.\n(2020) developed another method for investigating bias, testing how neural dialogue\nsystems responded to pairs of simulated user turns that are identical except for men-\ntioning different genders or race. They found, for example, that simple changes like\nusing the word \u2018she\u2019 instead of \u2018he\u2019 in a sentence caused systems to respond more\noffensively and with more negative sentiment.\nAnother important ethical issue is privacy . Already in the \ufb01rst days of ELIZA,\nWeizenbaum pointed out the privacy implications of people\u2019s revelations to the chat-\nbot. The ubiquity of in-home dialogue systems means they may often overhear\nprivate information (Henderson et al., 2017). If a chatbot is human-like, users are\nalso more likely to disclose private information, and less likely to worry about the\nharm of this disclosure (Ischen et al., 2019). In general, chatbots that are trained\non transcripts of human-human or human-machine conversation must anonymize\npersonally identi\ufb01able information.\nFinally, chatbots raise important issues of gender equality in addition to textual\nbias. Current chatbots are overwhelmingly given female names, likely perpetuating\nthe stereotype of a subservient female servant (Paolino, 2017). And when users\nuse sexually harassing language, most commercial chatbots evade or give positive\nresponses rather than responding in clear negative ways (Fessler, 2017).\nThese ethical issues are an important area of investigation, including \ufb01nding\nways to mitigate problems of abuse and toxicity, like detecting and responding ap-\npropriately to toxic contexts (Wolf et al. 2017, Dinan et al. 2020, Xu et al. 2020).\nValue sensitive design, carefully considering possible harms in advance (Friedman\net al. 2017, Friedman and Hendry 2019) is also important; (Dinan et al., 2021) give\na number of suggestions for best practices in dialogue system design. For exam-\nple getting informed consent from participants, whether they are used for training,\nor whether they are interacting with a deployed system is important. Because di-\nalogue systems by de\ufb01nition involve human participants, researchers also work on\nthese issues with the Institutional Review Boards ( IRB) at their institutions, who IRB\nhelp protect the safety of experimental subjects.\n15.6 Summary\nChatbots anddialogue systems are crucial speech and language processing appli-\ncations that are already widely used commercially.\n\u2022 In human dialogue, speaking is a kind of action; these acts are referred to\nas speech acts or dialogue acts. Speakers also attempt to achieve common\nground by acknowledging that they have understand each other. Conversation\nalso is characterized by turn structure and dialogue structure.\n\u2022 Chatbots are conversational systems designed to mimic the appearance of in-\nformal human conversation. Rule-based chatbots like ELIZA and its modern 20 CHAPTER 15 \u2022 C HATBOTS & D IALOGUE SYSTEMS\ndescendants use rules to map user sentences into system responses. Corpus-\nbased chatbots mine logs of human conversation to learn to automatically map\nuser sentences into system responses.\n\u2022 For task-based dialogue, most commercial dialogue systems use the GUS or\nframe-based architecture, in which the designer speci\ufb01es frames consisting of\nslots that the system must \ufb01ll by asking the user.\n\u2022 The dialogue-state architecture augments the GUS frame-and-slot architec-\nture with richer representations and more sophisticated algorithms for keeping\ntrack of user\u2019s dialogue acts, policies for generating its own dialogue acts, and\na natural language component.\n\u2022 Dialogue systems are a kind of human-computer interaction, and general HCI\nprinciples apply in their design, including the role of the user, simulations such\nas Wizard-of-Oz systems, and the importance of iterative design and testing\non real users.\nBibliographical and Historical Notes\nThe linguistic, philosophical, and psychological literature on dialogue is quite ex-\ntensive. For example the idea that utterances in a conversation are a kind of action\nbeing performed by the speaker was due originally to the philosopher Wittgenstein\n(1953) but worked out more fully by Austin (1962) and his student John Searle.\nVarious sets of speech acts have been de\ufb01ned over the years, and a rich linguistic\nand philosophical literature developed, especially focused on explaining the use of\nindirect speech acts. The idea of dialogue acts draws also from a number of other\nsources, including the ideas of adjacency pairs, pre-sequences, and other aspects of\nthe interactional properties of human conversation developed in the \ufb01eld of conver-\nsation analysis (see Levinson (1983) for an introduction to the \ufb01eld). This idea thatconversation\nanalysis\nacts set up strong local dialogue expectations was also pre\ufb01gured by Firth (1935, p.\n70), in a famous quotation:\nMost of the give-and-take of conversation in our everyday life is stereotyped\nand very narrowly conditioned by our particular type of culture. It is a sort\nof roughly prescribed social ritual, in which you generally say what the other\nfellow expects you, one way or the other, to say.\nAnother important research thread modeled dialogue as a kind of collaborative\nbehavior, including the ideas of common ground (Clark and Marshall, 1981), ref-\nerence as a collaborative process (Clark and Wilkes-Gibbs, 1986), joint intention\n(Levesque et al., 1990), and shared plans (Grosz and Sidner, 1980).\nThe earliest conversational systems were simple pattern-action chatbots like ELIZA\n(Weizenbaum, 1966). ELIZA had a widespread in\ufb02uence on popular perceptions of\narti\ufb01cial intelligence, and brought up some of the \ufb01rst ethical questions in natural\nlanguage processing \u2014such as the issues of privacy we discussed above as well the\nrole of algorithms in decision-making\u2014 leading its creator Joseph Weizenbaum to\n\ufb01ght for social responsibility in AI and computer science in general.\nComputational-implemented theories of dialogue blossomed in the 1970. That\nperiod saw the very in\ufb02uential GUS system (Bobrow et al., 1977), which in the late\n1970s established the frame-based paradigm that became the dominant industrial\nparadigm for dialogue systems for over 30 years. BIBLIOGRAPHICAL AND HISTORICAL NOTES 21\nAnother in\ufb02uential line of research from that decade focused on modeling the\nhierarchical structure of dialogue. Grosz\u2019s pioneering 1977 dissertation \ufb01rst showed\nthat \u201ctask-oriented dialogues have a structure that closely parallels the structure of\nthe task being performed\u201d (p. 27), leading to her work with Sidner and others show-\ning how to use similar notions of intention and plans to model discourse structure\nand coherence in dialogue. See, e.g., Lochbaum et al. (2000) for a summary of the\nrole of intentional structure in dialogue.\nYet a third line, \ufb01rst suggested by Bruce (1975), suggested that since speech acts\nare actions, they should be planned like other actions, and drew on the AI planning\nliterature (Fikes and Nilsson, 1971). A system seeking to \ufb01nd out some information\ncan come up with the plan of asking the interlocutor for the information. A system\nhearing an utterance can interpret a speech act by running the planner \u201cin reverse\u201d,\nusing inference rules to infer from what the interlocutor said what the plan might\nhave been. Plan-based models of dialogue are referred to as BDI models because BDI\nsuch planners model the beliefs ,desires , and intentions (BDI) of the system and in-\nterlocutor. BDI models of dialogue were \ufb01rst introduced by Allen, Cohen, Perrault,\nand their colleagues in a number of in\ufb02uential papers showing how speech acts could\nbe generated (Cohen and Perrault, 1979) and interpreted (Perrault and Allen 1980,\nAllen and Perrault 1980). At the same time, Wilensky (1983) introduced plan-based\nmodels of understanding as part of the task of interpreting stories.\nIn the 1990s, machine learning models that had \ufb01rst been applied to natural\nlanguage processing began to be applied to dialogue tasks like slot \ufb01lling (Miller\net al. 1994, Pieraccini et al. 1991). This period also saw lots of analytic work on the\nlinguistic properties of dialogue acts and on machine-learning-based methods for\ntheir detection. (Sag and Liberman 1975, Hinkelman and Allen 1989, Nagata and\nMorimoto 1994, Goodwin 1996, Chu-Carroll 1998, Shriberg et al. 1998, Stolcke\net al. 2000, Gravano et al. 2012. This work strongly informed the development\nof the dialogue-state model (Larsson and Traum, 2000). Dialogue state tracking\nquickly became an important problem for task-oriented dialogue, and there has been\nan in\ufb02uential annual evaluation of state-tracking algorithms (Williams et al., 2016).\nThe turn of the century saw a line of work on applying reinforcement learning\nto dialogue, which \ufb01rst came out of AT&T and Bell Laboratories with work on\nMDP dialogue systems (Walker 2000, Levin et al. 2000, Singh et al. 2002) along\nwith work on cue phrases, prosody, and rejection and con\ufb01rmation. Reinforcement\nlearning research turned quickly to the more sophisticated POMDP models (Roy\net al. 2000, Lemon et al. 2006, Williams and Young 2007) applied to small slot-\n\ufb01lling dialogue tasks. Neural reinforcement learning models have been used both for\nchatbot systems, for example simulating dialogues between two dialogue systems,\nrewarding good conversational properties like coherence and ease of answering (Li\net al., 2016), and for task-oriented dialogue (Williams et al., 2017).\nBy around 2010 the GUS architecture \ufb01nally began to be widely used commer-\ncially in dialogue systems on phones like Apple\u2019s SIRI (Bellegarda, 2013) and other\ndigital assistants.\nThe rise of the web gave rise to corpus-based chatbot architectures around the\nturn of the century, \ufb01rst using information retrieval models and then in the 2010s,\nafter the rise of deep learning, with sequence-to-sequence models.\n[TBD: Modern history of neural chatbots]\nOther important dialogue areas include the study of affect in dialogue (Rashkin\net al. 2019, Lin et al. 2019) and conversational interface design (Cohen et al. 2004,\nHarris 2005, Pearl 2017, Deibel and Evanhoe 2021). 22 CHAPTER 15 \u2022 C HATBOTS & D IALOGUE SYSTEMS\nExercises\n15.1 Write a \ufb01nite-state automaton for a dialogue manager for checking your bank\nbalance and withdrawing money at an automated teller machine.\n15.2 Adispreferred response is a response that has the potential to make a persondispreferred\nresponse\nuncomfortable or embarrassed in the conversational context; the most com-\nmon example dispreferred responses is turning down a request. People signal\ntheir discomfort with having to say no with surface cues (like the word well),\nor via signi\ufb01cant silence. Try to notice the next time you or someone else\nutters a dispreferred response, and write down the utterance. What are some\nother cues in the response that a system might use to detect a dispreferred\nresponse? Consider non-verbal cues like eye gaze and body gestures.\n15.3 When asked a question to which they aren\u2019t sure they know the answer, peo-\nple display their lack of con\ufb01dence by cues that resemble other dispreferred\nresponses. Try to notice some unsure answers to questions. What are some\nof the cues? If you have trouble doing this, read Smith and Clark (1993) and\nlisten speci\ufb01cally for the cues they mention.\n15.4 Implement a small air-travel help system based on text input. Your system\nshould get constraints from users about a particular \ufb02ight that they want to\ntake, expressed in natural language, and display possible \ufb02ights on a screen.\nMake simplifying assumptions. You may build in a simple \ufb02ight database or\nyou may use a \ufb02ight information system on the Web as your backend. Exercises 23\nAllen, J. and C. R. Perrault. 1980. Analyzing intention in\nutterances. Arti\ufb01cial Intelligence , 15:143\u2013178.\nArtstein, R., S. Gandhe, J. Gerten, A. Leuski, and D. Traum.\n2009. Semi-formal evaluation of conversational charac-\nters. In Languages: From Formal to Natural , 22\u201335.\nSpringer.\nAustin, J. L. 1962. How to Do Things with Words . Harvard\nUniversity Press.\nAwadallah, A. H., R. G. Kulkarni, U. Ozertem, and R. Jones.\n2015. Charaterizing and predicting voice query reformu-\nlation. CIKM-15 .\nBach, K. and R. Harnish. 1979. Linguistic communication\nand speech acts . MIT Press.\nBaum, L. F. 1900. The Wizard of Oz . Available at Project\nGutenberg.\nBellegarda, J. R. 2013. Natural language technology in mo-\nbile devices: Two grounding frameworks. In Mobile\nSpeech and Advanced Natural Language Solutions , 185\u2013\n196. Springer.\nBickmore, T. W., H. Trinh, S. Olafsson, T. K. O\u2019Leary,\nR. Asadi, N. M. Rickles, and R. Cruz. 2018. Patient and\nconsumer safety risks when using conversational assis-\ntants for medical information: An observational study of\nSiri, Alexa, and Google Assistant. Journal of Medical\nInternet Research , 20(9):e11510.\nBlodgett, S. L., S. Barocas, H. Daum \u00b4e III, and H. Wallach.\n2020. Language (technology) is power: A critical survey\nof \u201cbias\u201d in NLP. ACL.\nBobrow, D. G., R. M. Kaplan, M. Kay, D. A. Norman,\nH. Thompson, and T. Winograd. 1977. GUS, A frame\ndriven dialog system. Arti\ufb01cial Intelligence , 8:155\u2013173.\nBruce, B. C. 1975. Generation as a social action. Proceed-\nings of TINLAP-1 (Theoretical Issues in Natural Lan-\nguage Processing) .\nBudzianowski, P., T.-H. Wen, B.-H. Tseng, I. Casanueva,\nS. Ultes, O. Ramadan, and M. Ga \u02c7si\u00b4c. 2018. MultiWOZ -\na large-scale multi-domain wizard-of-Oz dataset for task-\noriented dialogue modelling. EMNLP .\nBulyko, I., K. Kirchhoff, M. Ostendorf, and J. Goldberg.\n2005. Error-sensitive response generation in a spo-\nken language dialogue system. Speech Communication ,\n45(3):271\u2013288.\nCarbonell, J. R. 1970. AI in CAI: An arti\ufb01cial-intelligence\napproach to computer-assisted instruction. IEEE transac-\ntions on man-machine systems , 11(4):190\u2013202.\nChristiano, P. F., J. Leike, T. Brown, M. Martic, S. Legg,\nand D. Amodei. 2017. Deep reinforcement learning from\nhuman preferences. NeurIPS , volume 30.\nChu-Carroll, J. 1998. A statistical model for discourse act\nrecognition in dialogue interactions. Applying Machine\nLearning to Discourse Processing. Papers from the 1998\nAAAI Spring Symposium. Tech. rep. SS-98-01. AAAI\nPress.\nChu-Carroll, J. and S. Carberry. 1998. Collaborative re-\nsponse generation in planning dialogues. Computational\nLinguistics , 24(3):355\u2013400.\nClark, H. H. 1996. Using Language . Cambridge University\nPress.Clark, H. H. and C. Marshall. 1981. De\ufb01nite reference and\nmutual knowledge. In A. K. Joshi, B. L. Webber, and I. A.\nSag, eds, Elements of Discourse Understanding , 10\u201363.\nCambridge.\nClark, H. H. and D. Wilkes-Gibbs. 1986. Referring as a col-\nlaborative process. Cognition , 22:1\u201339.\nCohen, A. D., A. Roberts, A. Molina, A. Butryna, A. Jin,\nA. Kulshreshtha, B. Hutchinson, B. Zevenbergen, B. H.\nAguera-Arcas, C. ching Chang, C. Cui, C. Du, D. D. F.\nAdiwardana, D. Chen, D. D. Lepikhin, E. H. Chi,\nE. Hoffman-John, H.-T. Cheng, H. Lee, I. Krivokon,\nJ. Qin, J. Hall, J. Fenton, J. Soraker, K. Meier-Hellstern,\nK. Olson, L. M. Aroyo, M. P. Bosma, M. J. Pickett, M. A.\nMenegali, M. Croak, M. D \u00b4\u0131az, M. Lamm, M. Krikun,\nM. R. Morris, N. Shazeer, Q. V . Le, R. Bernstein, R. Ra-\njakumar, R. Kurzweil, R. Thoppilan, S. Zheng, T. Bos,\nT. Duke, T. Doshi, V . Y . Zhao, V . Prabhakaran, W. Rusch,\nY . Li, Y . Huang, Y . Zhou, Y . Xu, and Z. Chen. 2022.\nLamda: Language models for dialog applications. ArXiv\npreprint.\nCohen, M. H., J. P. Giangola, and J. Balogh. 2004. Voice\nUser Interface Design . Addison-Wesley.\nCohen, P. R. and C. R. Perrault. 1979. Elements of a plan-\nbased theory of speech acts. Cognitive Science , 3(3):177\u2013\n212.\nColby, K. M., S. Weber, and F. D. Hilf. 1971. Arti\ufb01cial para-\nnoia. Arti\ufb01cial Intelligence , 2(1):1\u201325.\nCole, R. A., D. G. Novick, P. J. E. Vermeulen, S. Sutton,\nM. Fanty, L. F. A. Wessels, J. H. de Villiers, J. Schalkwyk,\nB. Hansen, and D. Burnett. 1997. Experiments with a\nspoken dialogue system for taking the US census. Speech\nCommunication , 23:243\u2013260.\nDanieli, M. and E. Gerbino. 1995. Metrics for evaluating\ndialogue strategies in a spoken language system. AAAI\nSpring Symposium on Empirical Methods in Discourse\nInterpretation and Generation .\nDeibel, D. and R. Evanhoe. 2021. Conversations with\nThings: UX Design for Chat and Voice . Rosenfeld.\nDinan, E., G. Abercrombie, A. S. Bergman, S. Spruit,\nD. Hovy, Y .-L. Boureau, and V . Rieser. 2021. Antici-\npating safety issues in e2e conversational ai: Framework\nand tooling. ArXiv.\nDinan, E., A. Fan, A. Williams, J. Urbanek, D. Kiela, and\nJ. Weston. 2020. Queens are powerful too: Mitigating\ngender bias in dialogue generation. EMNLP .\nFessler, L. 2017. We tested bots like Siri and Alexa to see\nwho would stand up to sexual harassment. Quartz . Feb\n22, 2017. https://qz.com/911681/ .\nFikes, R. E. and N. J. Nilsson. 1971. STRIPS: A new ap-\nproach to the application of theorem proving to problem\nsolving. Arti\ufb01cial Intelligence , 2:189\u2013208.\nFirth, J. R. 1935. The technique of semantics. Transactions\nof the philological society , 34(1):36\u201373.\nFraser, N. M. and G. N. Gilbert. 1991. Simulating speech\nsystems. Computer Speech and Language , 5:81\u201399.\nFriedman, B. and D. G. Hendry. 2019. Value Sensitive De-\nsign: Shaping Technology with Moral Imagination . MIT\nPress.\nFriedman, B., D. G. Hendry, and A. Borning. 2017. A sur-\nvey of value sensitive design methods. Foundations and\nTrends in Human-Computer Interaction , 11(2):63\u2013125. 24 Chapter 15 \u2022 Chatbots & Dialogue Systems\nGehman, S., S. Gururangan, M. Sap, Y . Choi, and N. A.\nSmith. 2020. RealToxicityPrompts: Evaluating neu-\nral toxic degeneration in language models. Findings of\nEMNLP .\nGlaese, A., N. McAleese, M. Trebacz, J. Aslanides,\nV . Firoiu, T. Ewalds, M. Rauh, L. Weidinger, M. Chad-\nwick, P. Thacker, L. Campbell-Gillingham, J. Uesato, P.-\nS. Huang, R. Comanescu, F. Yang, A. See, S. Dathathri,\nR. Greig, C. Chen, D. Fritz, J. Sanchez Elias,\nR. Green, S. Mokr \u00b4a, N. Fernando, B. Wu, R. Foley,\nS. Young, I. Gabriel, W. Isaac, J. Mellor, D. Hassabis,\nK. Kavukcuoglu, L. A. Hendricks, and G. Irving. 2022.\nImproving alignment of dialogue agents via targeted hu-\nman judgements. ArXiv preprint.\nGoldberg, J., M. Ostendorf, and K. Kirchhoff. 2003. The im-\npact of response wording in error correction subdialogs.\nISCA Tutorial and Research Workshop on Error Handling\nin Spoken Dialogue Systems .\nGood, M. D., J. A. Whiteside, D. R. Wixon, and S. J.\nJones. 1984. Building a user-derived interface. CACM ,\n27(10):1032\u20131043.\nGoodwin, C. 1996. Transparent vision. In E. Ochs, E. A.\nSchegloff, and S. A. Thompson, eds, Interaction and\nGrammar , 370\u2013404. Cambridge University Press.\nGopalakrishnan, K., B. Hedayatnia, Q. Chen, A. Gottardi,\nS. Kwatra, A. Venkatesh, R. Gabriel, and D. Hakkani-\nT\u00a8ur. 2019. Topical-chat: Towards knowledge-grounded\nopen-domain conversations. INTERSPEECH .\nGould, J. D., J. Conti, and T. Hovanyecz. 1983. Compos-\ning letters with a simulated listening typewriter. CACM ,\n26(4):295\u2013308.\nGould, J. D. and C. Lewis. 1985. Designing for usabil-\nity: Key principles and what designers think. CACM ,\n28(3):300\u2013311.\nGravano, A., J. Hirschberg, and \u02c7S. Be \u02c7nu\u02c7s. 2012. Af\ufb01rma-\ntive cue words in task-oriented dialogue. Computational\nLinguistics , 38(1):1\u201339.\nGrice, H. P. 1975. Logic and conversation. In P. Cole and\nJ. L. Morgan, eds, Speech Acts: Syntax and Semantics\nVolume 3 , 41\u201358. Academic Press.\nGrice, H. P. 1978. Further notes on logic and conversation. In\nP. Cole, ed., Pragmatics: Syntax and Semantics Volume 9 ,\n113\u2013127. Academic Press.\nGrosz, B. J. 1977. The Representation and Use of Focus\nin Dialogue Understanding . Ph.D. thesis, University of\nCalifornia, Berkeley.\nGrosz, B. J. and C. L. Sidner. 1980. Plans for discourse. In\nP. R. Cohen, J. Morgan, and M. E. Pollack, eds, Intentions\nin Communication , 417\u2013444. MIT Press.\nGuindon, R. 1988. A multidisciplinary perspective on dia-\nlogue structure in user-advisor dialogues. In R. Guindon,\ned.,Cognitive Science and Its Applications for Human-\nComputer Interaction , 163\u2013200. Lawrence Erlbaum.\nHarris, R. A. 2005. Voice Interaction Design: Crafting the\nNew Conversational Speech Systems . Morgan Kaufmann.\nHenderson, P., K. Sinha, N. Angelard-Gontier, N. R. Ke,\nG. Fried, R. Lowe, and J. Pineau. 2017. Ethical chal-\nlenges in data-driven dialogue systems. AAAI/ACM AI\nEthics and Society Conference .Hinkelman, E. A. and J. Allen. 1989. Two constraints on\nspeech act ambiguity. ACL.\nHirschberg, J., D. J. Litman, and M. Swerts. 2001. Identi-\nfying user corrections automatically in spoken dialogue\nsystems. NAACL .\nIschen, C., T. Araujo, H. V oorveld, G. van Noort, and\nE. Smit. 2019. Privacy concerns in chatbot interactions.\nInternational Workshop on Chatbot Research and De-\nsign.\nJefferson, G. 1972. Side sequences. In D. Sudnow, ed., Stud-\nies in social interaction , 294\u2013333. Free Press, New York.\nLandauer, T. K., ed. 1995. The Trouble with Computers:\nUsefulness, Usability, and Productivity . MIT Press.\nLarsson, S. and D. R. Traum. 2000. Information state and\ndialogue management in the trindi dialogue move engine\ntoolkit. Natural Language Engineering , 6(323-340):97\u2013\n114.\nLemon, O., K. Georgila, J. Henderson, and M. Stuttle. 2006.\nAn ISU dialogue system exhibiting reinforcement learn-\ning of dialogue policies: Generic slot-\ufb01lling in the TALK\nin-car system. EACL .\nLevesque, H. J., P. R. Cohen, and J. H. T. Nunes. 1990. On\nacting together. AAAI . Morgan Kaufmann.\nLevin, E., R. Pieraccini, and W. Eckert. 2000. A stochastic\nmodel of human-machine interaction for learning dialog\nstrategies. IEEE Transactions on Speech and Audio Pro-\ncessing , 8:11\u201323.\nLevinson, S. C. 1983. Conversational Analysis , chapter 6.\nCambridge University Press.\nLevow, G.-A. 1998. Characterizing and recognizing spoken\ncorrections in human-computer dialogue. COLING/ACL .\nLi, J., W. Monroe, A. Ritter, D. Jurafsky, M. Galley, and\nJ. Gao. 2016. Deep reinforcement learning for dialogue\ngeneration. EMNLP .\nLi, M., J. Weston, and S. Roller. 2019. Acute-eval: Improved\ndialogue evaluation with optimized questions and multi-\nturn comparisons. NeurIPS19 Workshop on Conversa-\ntional AI .\nLin, Z., A. Madotto, J. Shin, P. Xu, and P. Fung. 2019.\nMoEL: Mixture of empathetic listeners. EMNLP .\nLitman, D. J. 1985. Plan Recognition and Discourse Anal-\nysis: An Integrated Approach for Understanding Dia-\nlogues . Ph.D. thesis, University of Rochester, Rochester,\nNY .\nLitman, D. J. and J. Allen. 1987. A plan recognition model\nfor subdialogues in conversation. Cognitive Science ,\n11:163\u2013200.\nLitman, D. J., M. A. Walker, and M. Kearns. 1999. Auto-\nmatic detection of poor speech recognition at the dialogue\nlevel. ACL.\nLiu, H., J. Dacon, W. Fan, H. Liu, Z. Liu, and J. Tang. 2020.\nDoes gender matter? Towards fairness in dialogue sys-\ntems. COLING .\nLochbaum, K. E., B. J. Grosz, and C. L. Sidner. 2000. Dis-\ncourse structure and intention recognition. In R. Dale,\nH. Moisl, and H. L. Somers, eds, Handbook of Natural\nLanguage Processing . Marcel Dekker.\nMiller, S., R. J. Bobrow, R. Ingria, and R. Schwartz. 1994.\nHidden understanding models of natural language. ACL. Exercises 25\nMrk\u02c7si\u00b4c, N., D. \u00b4O S\u00b4eaghdha, T.-H. Wen, B. Thomson, and\nS. Young. 2017. Neural belief tracker: Data-driven dia-\nlogue state tracking. ACL.\nNagata, M. and T. Morimoto. 1994. First steps toward statis-\ntical modeling of dialogue to predict the speech act type\nof the next utterance. Speech Communication , 15:193\u2013\n203.\nNayak, N., D. Hakkani-T \u00a8ur, M. A. Walker, and L. P. Heck.\n2017. To plan or not to plan? discourse planning in\nslot-value informed sequence to sequence models for lan-\nguage generation. INTERSPEECH .\nNeff, G. and P. Nagy. 2016. Talking to bots: Symbiotic\nagency and the case of Tay. International Journal of\nCommunication , 10:4915\u20134931.\nNickerson, R. S. 1976. On conversational interaction with\ncomputers. Proceedings of the ACM/SIGGRAPH work-\nshop on User-oriented design of interactive graphics sys-\ntems.\nNielsen, J. 1992. The usability engineering life cycle. IEEE\nComputer , 25(3):12\u201322.\nNorman, D. A. 1988. The Design of Everyday Things . Basic\nBooks.\nOuyang, L., J. Wu, X. Jiang, D. Almeida, C. Wainwright,\nP. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray,\nJ. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens,\nA. Askell, P. Welinder, P. Christiano, J. Leike, and\nR. Lowe. 2022. Training language models to follow in-\nstructions with human feedback. NeurIPS , volume 35.\nPaolino, J. 2017. Google Home vs Alexa: Two\nsimple user experience design gestures that de-\nlighted a female user. Medium . Jan 4,\n2017. https://medium.com/startup-grind/\ngoogle-home-vs-alexa-56e26f69ac77 .\nPearl, C. 2017. Designing Voice User Interfaces: Principles\nof Conversational Experiences . O\u2019Reilly.\nPerrault, C. R. and J. Allen. 1980. A plan-based analysis of\nindirect speech acts. American Journal of Computational\nLinguistics , 6(3-4):167\u2013182.\nPieraccini, R., E. Levin, and C.-H. Lee. 1991. Stochastic\nrepresentation of conceptual structure in the ATIS task.\nSpeech and Natural Language Workshop .\nRafailov, R., A. Sharma, E. Mitchell, S. Ermon, C. D. Man-\nning, and C. Finn. 2023. Direct preference optimiza-\ntion: Your language model is secretly a reward model.\nNeurIPS .\nRashkin, H., E. M. Smith, M. Li, and Y .-L. Boureau. 2019.\nTowards empathetic open-domain conversation models:\nA new benchmark and dataset. ACL.\nReeves, B. and C. Nass. 1996. The Media Equation: How\nPeople Treat Computers, Television, and New Media Like\nReal People and Places . Cambridge University Press.\nRitter, A., C. Cherry, and B. Dolan. 2010. Unsupervised\nmodeling of twitter conversations. NAACL HLT .\nRoller, S., E. Dinan, N. Goyal, D. Ju, M. Williamson, Y . Liu,\nJ. Xu, M. Ott, E. M. Smith, Y .-L. Boureau, and J. We-\nston. 2021. Recipes for building an open-domain chatbot.\nEACL .\nRoy, N., J. Pineau, and S. Thrun. 2000. Spoken dialogue\nmanagement using probabilistic reasoning. ACL.Sacks, H., E. A. Schegloff, and G. Jefferson. 1974. A sim-\nplest systematics for the organization of turn-taking for\nconversation. Language , 50(4):696\u2013735.\nSag, I. A. and M. Y . Liberman. 1975. The intonational dis-\nambiguation of indirect speech acts. In CLS-75 , 487\u2013498.\nUniversity of Chicago.\nSchegloff, E. A. 1968. Sequencing in conversational open-\nings. American Anthropologist , 70:1075\u20131095.\nSee, A., S. Roller, D. Kiela, and J. Weston. 2019. What\nmakes a good conversation? how controllable attributes\naffect human judgments. NAACL HLT .\nShriberg, E., R. Bates, P. Taylor, A. Stolcke, D. Jurafsky,\nK. Ries, N. Coccaro, R. Martin, M. Meteer, and C. Van\nEss-Dykema. 1998. Can prosody aid the automatic clas-\nsi\ufb01cation of dialog acts in conversational speech? Lan-\nguage and Speech (Special Issue on Prosody and Conver-\nsation) , 41(3-4):439\u2013487.\nSingh, S. P., D. J. Litman, M. Kearns, and M. A. Walker.\n2002. Optimizing dialogue management with reinforce-\nment learning: Experiments with the NJFun system.\nJAIR , 16:105\u2013133.\nSmith, V . L. and H. H. Clark. 1993. On the course of an-\nswering questions. Journal of Memory and Language ,\n32:25\u201338.\nStalnaker, R. C. 1978. Assertion. In P. Cole, ed., Pragmat-\nics: Syntax and Semantics Volume 9 , 315\u2013332. Academic\nPress.\nStifelman, L. J., B. Arons, C. Schmandt, and E. A. Hulteen.\n1993. V oiceNotes: A speech interface for a hand-held\nvoice notetaker. INTERCHI 1993 .\nStolcke, A., K. Ries, N. Coccaro, E. Shriberg, R. Bates,\nD. Jurafsky, P. Taylor, R. Martin, M. Meteer, and C. Van\nEss-Dykema. 2000. Dialogue act modeling for automatic\ntagging and recognition of conversational speech. Com-\nputational Linguistics , 26(3):339\u2013371.\nSwerts, M., D. J. Litman, and J. Hirschberg. 2000. Correc-\ntions in spoken dialogue systems. ICSLP .\nUng, M., J. Xu, and Y .-L. Boureau. 2022. SaFeRDialogues:\nTaking feedback gracefully after conversational safety\nfailures. ACL.\nWade, E., E. Shriberg, and P. J. Price. 1992. User behaviors\naffecting speech recognition. ICSLP .\nWalker, M. A. 2000. An application of reinforcement learn-\ning to dialogue strategy selection in a spoken dialogue\nsystem for email. JAIR , 12:387\u2013416.\nWalker, M. A., J. C. Fromer, and S. S. Narayanan. 1998.\nLearning optimal dialogue strategies: A case study of a\nspoken dialogue agent for email. COLING/ACL .\nWeizenbaum, J. 1966. ELIZA \u2013 A computer program for the\nstudy of natural language communication between man\nand machine. CACM , 9(1):36\u201345.\nWilensky, R. 1983. Planning and Understanding: A Com-\nputational Approach to Human Reasoning . Addison-\nWesley.\nWilliams, J. D., K. Asadi, and G. Zweig. 2017. Hybrid code\nnetworks: practical and ef\ufb01cient end-to-end dialog con-\ntrol with supervised and reinforcement learning. ACL.\nWilliams, J. D., A. Raux, and M. Henderson. 2016. The di-\nalog state tracking challenge series: A review. Dialogue\n& Discourse , 7(3):4\u201333. 26 Chapter 15 \u2022 Chatbots & Dialogue Systems\nWilliams, J. D. and S. J. Young. 2007. Partially observ-\nable markov decision processes for spoken dialog sys-\ntems. Computer Speech and Language , 21(1):393\u2013422.\nWittgenstein, L. 1953. Philosophical Investigations. (Trans-\nlated by Anscombe, G.E.M.) . Blackwell.\nWolf, M. J., K. W. Miller, and F. S. Grodzinsky. 2017. Why\nwe should have seen that coming: Comments on Mi-\ncrosoft\u2019s Tay \u201cexperiment,\u201d and wider implications. The\nORBIT Journal , 1(2):1\u201312.\nXu, J., D. Ju, M. Li, Y .-L. Boureau, J. Weston, and E. Dinan.\n2020. Recipes for safety in open-domain chatbots. ArXiv\npreprint arXiv:2010.07079.\nYankelovich, N., G.-A. Levow, and M. Marx. 1995. Design-\ning SpeechActs: Issues in speech user interfaces. CHI-95 .\nYoung, S. J., M. Ga \u02c7si\u00b4c, S. Keizer, F. Mairesse, J. Schatz-\nmann, B. Thomson, and K. Yu. 2010. The Hidden Infor-\nmation State model: A practical framework for POMDP-\nbased spoken dialogue management. Computer Speech &\nLanguage , 24(2):150\u2013174.", "24_Dialogue_May_6_2021.pdf": "Chatbots and Dialogue SystemsIntroduction to Chatbots and Dialogue Systems Conversational Agents  (AKA  Dialogue Systems AKA Dialogue Agents AKA Chatbots)Personal Assistants on phones or other devicesSIRI, Alexa, Cortana, Google AssistantPlaying music, setting timers and clocksChatting for funBooking travel reservationsClinical uses for mental health Two kind of conversational agents1.Chatbots-mimic informal human chatting-for fun, or even for therapy2.(Task-based) Dialogue Agents-interfaces to personal assistants-cars, robots, appliances-booking flights or restaurants ChatbotArchitecturesRule-based1.Pattern-action rules (ELIZA)2.+ A mental model (PARRY):The first system to pass the Turing Test! Corpus-based3.Information Retrieval (XiaoIce)4.Neural encoder-decoder (BlenderBot) Sure, what do you want to be sung to? I can sing you a song about baking.Will you sing me a song?Yes, sing me a song about baking!Do you want me to sing it to the tune of \u201cSing Along with Mitch\u201d?Yes, I would love that.OK, I\u2019ll try to get it to you by the end of the day. What else do you like to do?No, sing it to me now!OK, give me a few days. I\u2019m a teacher, so I don\u2019t have much free time.Can you at least tell me what the lyrics are?Sure, it goes like this: \u201cBaking is my favorite thing to do in the world.\u201dHow does the chorus go?It goes: BAKING IS MY FAVORITE THING TO DO IN THE WORLD!Wow, sounds like a bop.I know, right?  It\u2019s so catchy. I can\u2019t get it out of my head.BlenderBot(Roller et al. 2020) XiaoIce(Zhou et al., 2020)6CHAPTER24\u2022CHATBOTS&DIALOGUESYSTEMSThese systems often have an entertainment value, such as Facebook\u2019s Blender-Bot, a neural chatbot capable of carrying on the kinds of conversations shown inFig.24.2, or Microsoft\u2019s XiaoIce system2(Zhou et al., 2020), a system that chatswith people on text messaging platforms (Fig.24.3).Sure, what do you want to be sung to? I can sing you a song about baking.Will you sing me a song?Yes, sing me a song about baking!Do you want me to sign it to the tune of \u201cSing Along with Mitch\u201d?Yes, I would love that.OK, I\u2019ll try to get it to you by the end of the day. What else do you like to do?No, sing it to me now!OK, give me a few days. I\u2019m a teacher, so I don\u2019t have much free time.Can you at least tell me what the lyrics are?Sure, it goes like this: \u201cBaking is my favorite thing to do in the world.\u201dHow does the chorus go?It goes: BAKING IS MY FAVORITE THING TO DO IN THE WORLD!Wow, sounds like a bop.I know, right?  It\u2019s so catchy. I can\u2019t get it out of my head.Figure 24.2A conversation between a user (left) and BlenderBot(Roller et al., 2020).;\u222b{=\u270f\u221e`}[Human] Hello, XiaoIce.;\u270f\u221e=`}@\u0000`4\u0153} \u00a3\u0000\u0000\u2026(\u00cd\u00a1\u00ab[XiaoIce]Hello, your profile pic isinteresting. Like I\u2019ve seenit somewhere.;\u222b{=/\u232b[Human] Really?;\u270f\u221e=\u2013\u0000/\u00cd\u00cb\u00ae;\u00c3\u00d1@[XiaoIce]Is it a comic character?;\u222b{=`\u0000H\u00c2S\u00d1[Human] How do you know that?;\u270f\u221e=`\u2327[XiaoIce]Guess.;\u222b{=\u00e6S`/\u0000\u222b\u232b\u0000[Human] Are you human?;\u270f\u221e=\u0178*\u0000`\u00e5\u2318\u21e2JJ1\u00c2S\u00dc[XiaoIce]Well, you\u2019ll know it afterchatting with me more.;\u222b{=`\u0000/\u222b\u00c2z\u02dd:h\u222bH[Human] Aren\u2019t you an Al chatbot?;\u270f\u221e=/\u00bb\u00007\u0000\u2318\u0000\u21e2\u2211\u221e\u221e\u00d1\u02d8\u00d6`\u00d1[XiaoIce]So what? I won\u2019t beindifferent to you.;\u222b{=`\u02dd \u00bfH\u00fc\u02ddJ[Human] What functions do you have?;\u270f\u221e=\u2318 200\u21e2*skills\u0000e\u0000\u00d9\u02d9`\u00d1\u0000B[XiaoIce]I have over 200 skills. Justname one you need.Figure 24.3A sample Chinese conversation from the XiaoIce chatbot with translations(Zhou et al., 2020).Yet starting from the very \ufb01rst system, ELIZA(Weizenbaum, 1966), chatbotshave also been used for practical purposes like testing theories of psychological2XiaoIce is a somewhat confusing translation of the systems Chinese name\u270f\u221e, \u2018Xiao Bing\u2019, whichmeans Little Bing or Little Ice. Task-based dialogue agents\"Task-based\" or\"goal-based\" dialogueagents\u2022Systems that have the goal of helping a user solve a task\u2022Setting a timer\u2022Making a travel reservation\u2022Playing a song\u2022Buying a productArchitecture: \u2022Frames with slots and values\u2022A knowledge structure representing user intentions The FrameA set of slots, to be filled with information of a given typeEach associated with a questionto the userSlotTypeQuestionORIGINcity\"What city are you leaving from?DESTcity\"Where are you going?DEP DATE date\"What day would you like to leave?DEP TIME time\"What time would you like to leave?AIRLINEline\"What is your preferred airline? Chatbots and Dialogue SystemsIntroduction to Chatbots and Dialogue Systems Chatbots and Dialogue SystemsProperties of Human Conversation A telephone conversation between a human travel agent (A) and a human client (C)2CHAPTER24\u2022CHATBOTS&DIALOGUESYSTEMScontrast,chatbotsare systems designed for extended conversations, set up to mimicthe unstructured conversations or \u2018chats\u2019 characteristic of human-human interaction,mainly for entertainment, but also for practical purposes like making task-orientedagents more natural.1In Section24.2we\u2019ll discuss the three major chatbot architec-tures: rule-based systems, information retrieval systems, and encoder-decoder gen-erators. In Section24.3we turn to task-oriented agents, introducing the frame-basedarchitecture (theGUSarchitecture) that underlies most task-based systems.24.1 Properties of Human ConversationConversation between humans is an intricate and complex joint activity. Before weattempt to design a conversational agent to converse with humans, it is crucial tounderstand something about how humans converse with each other. Consider someof the phenomena that occur in the conversation between a human travel agent anda human client excerpted in Fig.24.1.C1: . . . I need to travel in May.A2: And, what day in May did you want to travel?C3: OK uh I need to be there for a meeting that\u2019s from the 12th to the 15th.A4: And you\u2019re \ufb02ying into what city?C5: Seattle.A6: And what time would you like to leave Pittsburgh?C7: Uh hmm I don\u2019t think there\u2019s many options for non-stop.A8: Right. There\u2019s three non-stops today.C9: What are they?A10: The \ufb01rst one departs PGH at 10:00am arrives Seattle at 12:05 their time.The second \ufb02ight departs PGH at 5:55pm, arrives Seattle at 8pm. And thelast \ufb02ight departs PGH at 8:15pm arrives Seattle at 10:28pm.C11: OK I\u2019ll take the 5ish \ufb02ight on the night before on the 11th.A12: On the 11th? OK. Departing at 5:55pm arrives Seattle at 8pm, U.S. Air\ufb02ight 115.C13: OK.A14: And you said returning on May 15th?C15: Uh, yeah, at the end of the day.A16: OK. There\u2019s #two non-stops . . . #C17: #Act. . . actually #, what day of the week is the 15th?A18: It\u2019s a Friday.C19: Uh hmm. I would consider staying there an extra day til Sunday.A20: OK. . . OK. On Sunday I have . . .Figure 24.1Part of a phone conversation between a human travel agent (A) and humanclient (C). The passages framed by # in A16and C17indicate overlaps in speech.TurnsA dialogue is a sequence ofturns(C1,A2,C3, and so on), each a single contributionturnfrom one speaker to the dialogue (as if in a game: I take a turn, then you take a turn,1By contrast, in popular usage, the wordchatbotis often generalized to refer to both task-oriented andchit-chat systems; we\u2019ll be usingdialogue systemsfor the former. Properties of Human ConversationTurns\u2022We call each contribution a \"turn\"\u2022As if conversation was the kind of game where everyone takes turns.\n12 2CHAPTER24\u2022CHATBOTS&DIALOGUESYSTEMScontrast,chatbotsare systems designed for extended conversations, set up to mimicthe unstructured conversations or \u2018chats\u2019 characteristic of human-human interaction,mainly for entertainment, but also for practical purposes like making task-orientedagents more natural.1In Section24.2we\u2019ll discuss the three major chatbot architec-tures: rule-based systems, information retrieval systems, and encoder-decoder gen-erators. In Section24.3we turn to task-oriented agents, introducing the frame-basedarchitecture (theGUSarchitecture) that underlies most task-based systems.24.1 Properties of Human ConversationConversation between humans is an intricate and complex joint activity. Before weattempt to design a conversational agent to converse with humans, it is crucial tounderstand something about how humans converse with each other. Consider someof the phenomena that occur in the conversation between a human travel agent anda human client excerpted in Fig.24.1.C1: . . . I need to travel in May.A2: And, what day in May did you want to travel?C3: OK uh I need to be there for a meeting that\u2019s from the 12th to the 15th.A4: And you\u2019re \ufb02ying into what city?C5: Seattle.A6: And what time would you like to leave Pittsburgh?C7: Uh hmm I don\u2019t think there\u2019s many options for non-stop.A8: Right. There\u2019s three non-stops today.C9: What are they?A10: The \ufb01rst one departs PGH at 10:00am arrives Seattle at 12:05 their time.The second \ufb02ight departs PGH at 5:55pm, arrives Seattle at 8pm. And thelast \ufb02ight departs PGH at 8:15pm arrives Seattle at 10:28pm.C11: OK I\u2019ll take the 5ish \ufb02ight on the night before on the 11th.A12: On the 11th? OK. Departing at 5:55pm arrives Seattle at 8pm, U.S. Air\ufb02ight 115.C13: OK.A14: And you said returning on May 15th?C15: Uh, yeah, at the end of the day.A16: OK. There\u2019s #two non-stops . . . #C17: #Act. . . actually #, what day of the week is the 15th?A18: It\u2019s a Friday.C19: Uh hmm. I would consider staying there an extra day til Sunday.A20: OK. . . OK. On Sunday I have . . .Figure 24.1Part of a phone conversation between a human travel agent (A) and humanclient (C). The passages framed by # in A16and C17indicate overlaps in speech.TurnsA dialogue is a sequence ofturns(C1,A2,C3, and so on), each a single contributionturnfrom one speaker to the dialogue (as if in a game: I take a turn, then you take a turn,1By contrast, in popular usage, the wordchatbotis often generalized to refer to both task-oriented andchit-chat systems; we\u2019ll be usingdialogue systemsfor the former. Properties of Human ConversationTurn-taking issues\u25e6When to take the floor?\u25e6When to yield the floor?Interruptions\n14 2CHAPTER24\u2022CHATBOTS&DIALOGUESYSTEMScontrast,chatbotsare systems designed for extended conversations, set up to mimicthe unstructured conversations or \u2018chats\u2019 characteristic of human-human interaction,mainly for entertainment, but also for practical purposes like making task-orientedagents more natural.1In Section24.2we\u2019ll discuss the three major chatbot architec-tures: rule-based systems, information retrieval systems, and encoder-decoder gen-erators. In Section24.3we turn to task-oriented agents, introducing the frame-basedarchitecture (theGUSarchitecture) that underlies most task-based systems.24.1 Properties of Human ConversationConversation between humans is an intricate and complex joint activity. Before weattempt to design a conversational agent to converse with humans, it is crucial tounderstand something about how humans converse with each other. Consider someof the phenomena that occur in the conversation between a human travel agent anda human client excerpted in Fig.24.1.C1: . . . I need to travel in May.A2: And, what day in May did you want to travel?C3: OK uh I need to be there for a meeting that\u2019s from the 12th to the 15th.A4: And you\u2019re \ufb02ying into what city?C5: Seattle.A6: And what time would you like to leave Pittsburgh?C7: Uh hmm I don\u2019t think there\u2019s many options for non-stop.A8: Right. There\u2019s three non-stops today.C9: What are they?A10: The \ufb01rst one departs PGH at 10:00am arrives Seattle at 12:05 their time.The second \ufb02ight departs PGH at 5:55pm, arrives Seattle at 8pm. And thelast \ufb02ight departs PGH at 8:15pm arrives Seattle at 10:28pm.C11: OK I\u2019ll take the 5ish \ufb02ight on the night before on the 11th.A12: On the 11th? OK. Departing at 5:55pm arrives Seattle at 8pm, U.S. Air\ufb02ight 115.C13: OK.A14: And you said returning on May 15th?C15: Uh, yeah, at the end of the day.A16: OK. There\u2019s #two non-stops . . . #C17: #Act. . . actually #, what day of the week is the 15th?A18: It\u2019s a Friday.C19: Uh hmm. I would consider staying there an extra day til Sunday.A20: OK. . . OK. On Sunday I have . . .Figure 24.1Part of a phone conversation between a human travel agent (A) and humanclient (C). The passages framed by # in A16and C17indicate overlaps in speech.TurnsA dialogue is a sequence ofturns(C1,A2,C3, and so on), each a single contributionturnfrom one speaker to the dialogue (as if in a game: I take a turn, then you take a turn,1By contrast, in popular usage, the wordchatbotis often generalized to refer to both task-oriented andchit-chat systems; we\u2019ll be usingdialogue systemsfor the former. Implications for Conversational AgentsBarge-in\u25e6Allowing the user to interruptEnd-pointing\u25e6The task for a speech system of deciding whether the user has stopped talking.\u25e6Very hard, since people often pause in the middle of turns\n16 Language as Action\nEach turn in a dialogue is a kind of actionWittgenstein (1953) and Austin (1962) Speech Acts (aka Dialogue Acts)Constatives: committing the speaker to something\u2019s being the case (answering, claiming, confirming, denying, disagreeing, stating) Directives: attempts by the speaker to get the addressee to do something (advising, asking, forbidding, inviting, ordering, requesting) Commissives: committing the speaker to some future course of action (promising, planning, vowing, betting, opposing) Acknowledgments: express the speaker\u2019s attitude regarding the hearer with respect to some social action (apologizing, greeting, thanking, accepting an acknowledgment) Bach and Harnish (1979) Speech acts\"Turn up the music!\" DIRECTIVE\"What day in May do you want to travel?\"DIRECTIVE\"I need to travel in May\"CONSTATIVEThanksACKNOWLEDGEMENT Grounding\u2022Participants in conversation or any joint activity need to establish common ground.\u2022Principle of closure.  Agents performing an action require evidence, sufficient for current purposes, that they have succeeded in performing it  (Clark 1996, after Norman 1988)\u2022Speech is an action too!  Sospeakers need to groundeach other\u2019s utterances. \u2022Grounding: acknowledging that the hearer has understood Grounding\u2022Grounding is relevant for human-machine interaction\u2022Why do elevator buttons light up?\n Grounding: Establishing Common GroundA: And you said returning on May 15th?C: Uh, yeah, at the end of the day.A: OKC: OK I\u2019ll take the 5ish flight on the night before on the 11th.A: On the 11th? OK. C: ...I need to travel in May.A: And, what day in May did you want to travel? Grounding is important for computers too!System: Did you want to review some more of your profile?User: No.System: What\u2019s next?System: Did you want to review some more of your profile?User: No.System: Okay, what\u2019s next?Cohen et al (2004) Conversations have structureLocal structure between adjacent speech acts, from the field of conversational analysis(Sacks et al. 1974)Calledadjacency pairs:QUESTION\u2026 ANSWERPROPOSAL\u2026 ACCEPTANCE/REJECTIONCOMPLIMENTS(\"Nice jacket!\")\u2026 DOWNPLAYER(\"Oh, this old thing?\") Another kind of structure: SubdialoguesAgent:  OK.  There's #two non-stops#Client:                     #Act-actually#, what day of the week is the 15th?Agent: It\u2019s a Friday.Client: Uh hmm. I would consider staying there an extra day tilSunday. Agent: OK...OK. On Sunday I have ... \n25Correctionsubdialogue Clarification SubdialoguesUser: What do you have going to UNKNOWN WORD on the 5th?System:  Let\u2019s see, going where on the 5th?User: Going to Hong Kong. System: OK, here are some flights... \n26 PresequencesUser: Can you make train reservations?System: YesI can.User: Great, I\u2019d like to reserve a seat on the 4pm train to New York.  Conversational InitiativeSome conversations are controlled by one person\u2022A reporter interviewing a chef asks questions, and the chef responds.\u2022This reporter has the conversational initiative (Walker and Whittaker 1990)Most human conversations have mixed initiative: \u2022I lead, then you lead, then I lead.\u2022Mixed initiative is very hard for NLP systems, which often default to simpler styles that can be frustrating for humans:\u2022User initiative (user asks or commands, system responds)\u2022System initiative (system asks user questions to fill out a form, user can't change the direction) Even harder problems:  InferenceAgent: And, what day in May did you want to travel?Client: OK, uh, I need to be there for a meeting that\u2019s from the 12th to the 15th.  Chatbots and Dialogue SystemsProperties of Human Conversation Chatbots and Dialogue SystemsRule-based Chatbots: ELIZA and PARRY ELIZA: Weizenbaum(1966)Men are all alike.IN WHAT WAYThey're always bugging us about something or other. CAN YOU THINK OF A SPECIFIC EXAMPLE Well, my boyfriend made me come here.YOUR BOYFRIEND MADE YOU COME HERE He says I'm depressed much of the time.I AM SORRY TO HEAR YOU ARE DEPRESSED ...You are like my father in some ways.WHAT RESEMBLANCE DO YOU SEEYou are not very aggressivebut I think you don\u2019t want me to notice that.WHAT MAKES YOU THINK I AM NOT AGGRESSIVEYou don\u2019t argue with me.WHY DO YOU THINK I DON\u2019T ARGUE WITH YOUYou are afraid of me.DOES IT PLEASE YOU TO BELIEVE I\u2019M AFRAID OF YOUMyfatherisafraidof everybody.WHAT ELSE COMES TO MIND WHEN YOU THINK OF YOUR FATHERBullies.DOES THAT HAVE ANYTHING TO DO WITH THE FACT THAT YOUR BOYFRIEND MADE YOU COME HEREWeizenbaum, Joseph. 1966,\"ELIZA -A Computer Program For the Study of Natural Language Communication Between Man And Machine\",Communications of the ACM 9 (1): 36-45 ELIZA's trick:be a Rogerian psychologistDraw the patient out by reflecting patient\u2019s statements back at themRare type of conversation in which one can \u201cassume the pose of knowing almost nothing of the real world\u201d Rogerian psychologistPatient: \"I went for a long boat ride\u201dPsychologist: \"Tell me about boats\u201dYou don\u2019t assume she didn\u2019t know what a boat isYou assume she had some conversational goalChatbotstrying to pass the Turing test usually choose some such domain Eliza pattern/transform rules(0 YOU 0 ME)    [pattern]\u00e0(WHAT MAKES YOU THINK I 3 YOU) [transform]0  means Kleene *The 3 is the constituent # in patternYou hate meWHAT MAKES YOU THINK I HATE YOU Eliza RulesA formal notation in which to represent the decomposition template is: (0 YOU 0 ME) and the reassembly rule (WHAT MAKES YOU THINK I 3 YOU). The \"0\" in the decomposition rule stands for \"an in- definite number of words\" (analogous to the indefinite dollar sign of COMIT) [6] while the \"3\" in the reassembly rule indicates that the third component of the subject decomposition is to be inserted in its place. The decom- position rule (0 YOU 1 ME) would have worked just as well in this specific example. A nonzero integer \"n\" appearing in a decomposition rule indicates that the component in question should consist of exactly \"n\" words. However, of the two rules shown, only the first would have matched the sentence, \"It seems you hate and love me,\" the second failing because there is more than one word between \"you\" and \"me\". \n~1 RI,1 R1,2\"'\" Rl,ml D2 R2J RZ,Z\" \" \"Rz,m2 De Ra,t FIn'2~ '\" \" R%nan Fro. 1. :Keyword and rule list structure in ELIZA the question of which decomposition rules to apply to an input text is of course a crucial one. The input sentence might have been, for example, \"It seems that you hate,\" in which ease the decomposition rule (0 YOU 0 ME) would have failed in that the word \"ME\" would not have been found at all, let alone in its assigned place. Some other decomposition rule would then have to be tried and, failing that, still another until a match could be made or a total failure reported. ELIZA must therefm~ have a mechanism to sharply delimit the set of decom- position rules which are potentially applicable to a cur- rently active input sentence. This is the keyword mecha- nism. An input sentence is scanned from left to right. Each word is looked up in a dictionary of keywords. If a word is identified as a keyword, then (apart from the issue of precedence of keywords) only decomposition rules con- taining that keyword need to be tried. The trial sequence can even be partially ordered. For example, the decom- position rule (0 YOU 0) associated with the keyword \"YOU\" (and decomposing an input sentence into (1) all words in front of \"YOU\", (2) the word \"YOU\", and (3) all words following \"YOU\") should be the last, one tried since it is bound to succeed. Two problems now arise. One stems from the fact that almost none of the words in any given sentence are repre. sented in the keyword dictionary. The other is that of \"associating\" both decomposit;ion and reassembly rules with keyword~-;. The iirst is serious in that the determina- tion that a word is not in a dictionary may well require more computation (i.e., time) than the location of a word which is represented. The attack on both problems begins by placing both a keyword trod its associated rules on a list. The basic format of a typical key list is the following: (K ((D0 (R~. t) (R~, 2) \"'\" (Rt .... )) ((D2) (R2. ~) (R2.2) \"\" (R2.,,,~)) : ((D,~) (R,,, ~) (taw, ~) ... (R,~, .~))) where K is the keyword, D, the it, h decomposition rule associated with K and R\u00a2, ~ the jth reassembly rule ass0. ciated with the ith decomposition rule. A common pictorial representation of such a structure is the tree diagram shown in Figure 1. The top level of this structure contains the keyword followed by the names of lists; each one of which is again a list structure beginning with a decomposition rule and followed by reassembly rules. Since list structures of this type have no predeter- mined dimensionality limitations, any number of decom- position rules may be associated with a given keyword and any number of reassembly rules witch any specific decom- position rule. SLiP i8 rich in functions that sequence over structures of this type efficiently. Hence programmin problems are minimized. An ELIZA script consists mainly of a set of list struc- tures of the type shown. The actual keyword dictionary is constructed when such a script is first read into the hitherto empty program. The basic structural component of the keyword dictionary is a vector KEY of (currently) 128 contiguous computer words. As a particular key list structure is read the keyword K at its top is randomized (hashed) by a procedure that produces (currently) a 7 bit integer \"i\". The word \"always\", for example, yields the integer 14. KEY(i), i.e., the ith word of the vector KEY, is then examined to determine whether it contains a list. name. If it does not, then an empty list is created, its name placed in KEY(i), and the key list structure in question placed on that list. If KEY(i) already contains a list name, then the name of the key list structure is placed on the bottom of the list named in KEY(i). The largest dictionary so far attempted contains about 50 keywords. No list named in any of the words of the KEY vector contains more than two key list structures. Every word encountered in the scan of an input text, i.e., during the actual operations of ELIZA, is randomized by the same hashing algorithm as was originally applied to the incoming keywords, hence yields an integer which points to the only possible list structure which could potentially contain that word as a keyword. Even then, only the tops of any key list structures that may be found there need be interrogated to determine whether or not a keyword has been found. By virtue of the various list 38 Communications of tile ACM Volume 9 / Number 1 / January, 1966 \nkeywordpatternlist of transformsKeyword:  YOUPattern: (0 YOU 0 ME)    Transforms: (WHAT MAKES YOU THINK I 3 YOU)(WHY DO YOU THINK I 3 YOU)Rules are organized by keywordsEach keyword has a pattern and a list of possible transform Keywords are ranked from specific to generalUser: I know everybody laughed at me\u201cI\u201d is very general:I: (I *) -> (YOU SAY YOU 2)ELIZA: YOU SAY YOU KNOW EVERYBODY LAUGHED AT YOU\u201cEverybody\u201d is more specific and interestingEverybody: (Everybody *) -> (WHO IN PARTICULAR ARE YOU THINKING OF)ELIZA: WHO IN PARTICULAR ARE YOU THINKING OF?Implementation: keywords stored with their rank Everybody 5 (list of transformation rules)I 0 (list of transformation rules) 8CHAPTER24\u2022CHATBOTS&DIALOGUESYSTEMSMen are all alike.IN WHAT WAYThey\u2019re always bugging us about something or other.CAN YOU THINK OF A SPECIFIC EXAMPLEWell my boyfriend made me come here.YOUR BOYFRIEND MADE YOU COME HEREHe says I\u2019m depressed much of the time.I AM SORRY TO HEAR YOU ARE DEPRESSED....WHO ELSE IN YOUR FAMILY TAKES CARE OF YOU?My fatherYOUR FATHERYou are like my father in some ways.WHAT RESEMBLANCE DO YOU SEEYou are not very aggressive but I think you don\u2019t want me to notice that.WHAT MAKES YOU THINK I AM NOT AGGRESSIVEYou don\u2019t argue with me.WHY DO YOU THINK I DON\u2019T ARGUE WITH YOUYou are afraid of me.DOES IT PLEASE YOU TO BELIEVE I\u2019M AFRAID OF YOUMy father is afraid of everybody.WHAT ELSE COMES TO MIND WHEN YOU THINK OF YOUR FATHERBullies.DOES THAT HAVE ANYTHING TO DO WITH THE FACT THAT YOURBOYFRIEND MADE YOU COME HEREFigure 24.4Sample ELIZA dialogue fromWeizenbaum (1966).functionELIZAGENERATOR(usersentence)returnsresponseFind the wordwinsentencethat has the highest keyword rankifwexistsChoose the highest ranked rulerforwthat matchessentenceresponse Apply the transform inrtosentenceifw= \u2018my\u2019future Apply a transformation from the \u2018memory\u2019 rule list tosentencePushfutureonto memory stackelse(no keyword applies)eitherresponse Apply the transform for the NONE keyword tosentenceorresponse Pop the top response from the memory stackreturn(response)Figure 24.5A simpli\ufb01ed sketch of the ELIZA algorithm. The power of the algorithmcomes from the particular transforms associated with each keyword.to some quite speci\ufb01c event or person\u201d. Therefore, ELIZA prefers to respond withthe pattern associated with the more speci\ufb01c keywordeverybody(implementing byjust assigning \u201ceverybody\u201d rank 5 and \u201cI\u201d rank 0 in the lexicon), whose rule thus NONEPLEASE GO ONTHAT\u2019S VERY INTERESTINGI SEE Memory(MEMORY MY(0 MY 0 = LETS DISCUSS FURTHER WHY YOUR 3)(0 MY 0 = EARLIER YOU SAID YOUR 3)(0 MY 0 = DOES THAT HAVE ANYTHING TO DO WITH THE FACT THAT YOUR 3))Whenever \u201cMY\u201d is highest keyword\u25e6Randomly select a transform on the MEMORY list \u25e6Apply to sentence\u25e6Store on a (first-in-first-out) queueLater, if no keyword matches a sentence\u25e6Return the top of the MEMORY queue instead Ethical implications: Anthropomorphism and PrivacyPeople  became deeply emotionally involved with the programOne of Weizenbaum'sstaff asked him to leave the room when she talked with ELIZAWhen he suggested that he might want to store all the ELIZA conversations for later analysis, people immediately pointed out the privacy implications\u25e6Suggesting that they were having quite private conversations with ELIZA\u25e6Despite knowing that it was just software.  Ethical implicationsIt worried Weizenbaumthat people confided in ELIZA\u2022Were people misled about how much computers understood?Turkle studied users of ELIZA and other systems\u2022Turkle has shown human face-to-face interaction is vital\u2022But people also develop specific relationships with artifacts\u2022Some users told her ELIZA was more like a kind of diary, a way to privately explore their thoughts.Importance of value-sensitive designJoseph Weizenbaum.1976.Computer Power and Human Reason: From Judgment to Calculation.WH Freeman.Sherry Turkle. 2011. Taking Things at Interface Value, chapter in Life on the Screen.Simon and Schuster.Sherry Turkle. 2007. Authenticity in the age of digital companions. Interaction Studies, 8(3), pp.501-517 PARRY: A computational model of schizophreniaAnother chatbot with a clinical psychology focus\u2022Colby, K. M., Weber, S., and Hilf, F. D. (1971). Artificial paranoia. Artificial Intelligence 2(1), 1\u201325. Used to study schizophreniaSame pattern-response structure as ElizaBut a much richer:\u25e6control structure \u25e6language understanding capabilities\u25e6model of mental state.\u25e6variables modeling levels of Anger, Fear, Mistrust43 Affect variablesFear(0-20)                    Anger(0-20)                  Mistrust(0-15)Start with all variables lowAfter each user turn\u25e6Eachuser statement can change Fear and Anger\u25e6E.g., Insults increases Anger, Flattery decreases Anger\u25e6Mentions of his delusions increase Fear\u25e6Else if nothing malevolent in input\u25e6Anger, Fear, Mistrust all drop Parry's responses depend on mental stateModifyAffect variablesexcessive fearEscapeexcessive angerHostilityInput mentions delusion topicquestionFearanswerUser Inputconditionresponse\u2026 PARRY passes the Turing test in 1972The first system to pass a version of the Turing test Psychiatrists couldn\u2019t distinguish interviews with PARRY from (text transcripts of) interviews with people diagnosed with paranoid schizophrenia\u2022Colby, K. M., Hilf, F. D., Weber, S., and Kraemer, H. C. (1972). Turing-like indistinguishability tests for the validation of a computer simulation of paranoid processes. Artificial Intelligence 3, 199\u2013221. 46 Chatbots and Dialogue SystemsRule-based Chatbots: ELIZA and PARRY Chatbots and Dialogue SystemsCorpus-based Chatbots Two architectures for corpus-based chabotsResponse by retrieval\u2022Use information retrieval to grab a response (that is appropriate to the context) from some corpusResponse by generation\u2022Use a language model or encoder-decoder to generate the response given the dialogue context49 Corpus-based chatbots require corpora\u2022Modern corpus-based chatbots are very data-intensive\u2022They commonly require hundreds of millions or billions of words50 What conversations to draw on?Transcripts of telephone conversations between volunteers\u2022Switchboard corpus of American English telephone conversationsMovie dialogue\u2022Various corpora of movie subtitlesHire human crowdworkersto have conversations\u2022Topical-Chat 11K crowdsourced conversations on 8 topics\u2022EMPATHETICDIALOGUES25K crowdsourced conversations grounded in a situation where a speaker was feeling a specific emotionPseudo-conversations from public posts on social media\u2022Drawn from Twitter, Reddit, Weibo (\u5fae\u535a), etc. \u2022Tend to be noisy; often used just as pre-training.Crucial to remove personally identifiable information (PII)  Response by retrieval: classic IR method1.Given a user turn q, and a training corpus Cof conversation2.Find in C the turn rthat is most similar (tf-idfcosine) to q3.Say r10CHAPTER24\u2022CHATBOTS&DIALOGUESYSTEMStalk about knowledge provided to them. For example the Topical-Chat dataset has11K crowdsourced conversations spanning 8 broad topics(Gopalakrishnan et al.,2019), and the EMPATHETICDIALOGUESincludes 25K crowdsourced conversa-tions grounded in a speci\ufb01c situation where a speaker was feeling a speci\ufb01c emotion(Rashkin et al., 2019).All of these datasets, although large, don\u2019t reach the size of billions of words,and so many systems \ufb01rst pretrain on large datasets of pseudo-conversations drawnfrom Twitter(Ritter et al., 2010), Reddit(Roller et al., 2020), Weibo (\u00c6Z), andother social media platforms.Another common technique is to extract possible responses from knowledgesources (Wikipedia, news stories) so that a chatbot can tell stories or mention factsacquired in that way.Finally, once a chatbot has been put into practice, the turns that humans use torespond to the chatbot can be used as additional conversational data for training or\ufb01netuning. Here it\u2019s important to have con\ufb01dence metrics to make sure that theseturns come from conversations that are going well(Hancock et al., 2019). It\u2019s alsocrucial in these cases to remove personally identi\ufb01able information (PII); see Sec-tion24.6.1.Most corpus based chatbots produce their responses to a user\u2019s turn in contexteither byretrievalmethods (using information retrieval to grab a response fromsome corpus that is appropriate given the dialogue context) orgenerationmethods(using a language model or encoder-decoder to generate the response given the di-alogue context) In either case, systems mostly generate a single response turn thatis appropriate given the entire conversation so far (for conversations that are shortenough to \ufb01t into a single model\u2019s window). For this reason they are often calledresponse generationsystems. Corpus-based chatbot algorithms thus draw on algo-responsegenerationrithms for question answering systems, which similarly focus on single responseswhile ignoring longer-term conversational goals.Response by retrievalTheretrievalmethod of responding is to think of the user\u2019sturn as a queryq, and our job is to retrieve and repeat some appropriate turnras theresponse from a corpus of conversationsC. GenerallyCis the training set for thesystem, and we score each turn inCas a potential response to the contextqselectingthe highest-scoring one. The scoring metric is similarity: we choose therthat ismost similar toq, using any of the IR methods we saw in Section??. This can bedone using classic IR techniques to compute tf-idf models forCandq, choosing therthat has the highest tf-idf cosine withq:response(q,C)=argmaxr2Cq\u00b7r|q||r|(24.1)Or, we can use the neural IR techniques of Section??. The simplest of those is abi-encoder model, in which we train two separate encoders, one to encode the userquery and one to encode the candidate response, and use the dot product betweenthese two vectors as the score (Fig.24.6a). For example to implement this usingBERT, we would have two encoders BERTQand BERTRand we could represent thequery and candidate response as the[CLS]token of the respective encoders:hq=BERTQ(q)[CLS]hr=BERTR(r)[CLS]response(q,C)=argmaxr2Chq\u00b7hr(24.2) Response by retrieval: neural IR method1.Given a user turn q, and a training corpus Cof conversation2.Find in C the turn rthat is most similar (BERT dot product) to q3.Say r10CHAPTER24\u2022CHATBOTS&DIALOGUESYSTEMStalk about knowledge provided to them. For example the Topical-Chat dataset has11K crowdsourced conversations spanning 8 broad topics(Gopalakrishnan et al.,2019), and the EMPATHETICDIALOGUESincludes 25K crowdsourced conversa-tions grounded in a speci\ufb01c situation where a speaker was feeling a speci\ufb01c emotion(Rashkin et al., 2019).All of these datasets, although large, don\u2019t reach the size of billions of words,and so many systems \ufb01rst pretrain on large datasets of pseudo-conversations drawnfrom Twitter(Ritter et al., 2010), Reddit(Roller et al., 2020), Weibo (\u00c6Z), andother social media platforms.Another common technique is to extract possible responses from knowledgesources (Wikipedia, news stories) so that a chatbot can tell stories or mention factsacquired in that way.Finally, once a chatbot has been put into practice, the turns that humans use torespond to the chatbot can be used as additional conversational data for training or\ufb01netuning. Here it\u2019s important to have con\ufb01dence metrics to make sure that theseturns come from conversations that are going well(Hancock et al., 2019). It\u2019s alsocrucial in these cases to remove personally identi\ufb01able information (PII); see Sec-tion24.6.1.Most corpus based chatbots produce their responses to a user\u2019s turn in contexteither byretrievalmethods (using information retrieval to grab a response fromsome corpus that is appropriate given the dialogue context) orgenerationmethods(using a language model or encoder-decoder to generate the response given the di-alogue context) In either case, systems mostly generate a single response turn thatis appropriate given the entire conversation so far (for conversations that are shortenough to \ufb01t into a single model\u2019s window). For this reason they are often calledresponse generationsystems. Corpus-based chatbot algorithms thus draw on algo-responsegenerationrithms for question answering systems, which similarly focus on single responseswhile ignoring longer-term conversational goals.Response by retrievalTheretrievalmethod of responding is to think of the user\u2019sturn as a queryq, and our job is to retrieve and repeat some appropriate turnras theresponse from a corpus of conversationsC. GenerallyCis the training set for thesystem, and we score each turn inCas a potential response to the contextqselectingthe highest-scoring one. The scoring metric is similarity: we choose therthat ismost similar toq, using any of the IR methods we saw in Section??. This can bedone using classic IR techniques to compute tf-idf models forCandq, choosing therthat has the highest tf-idf cosine withq:response(q,C)=argmaxr2Cq\u00b7r|q||r|(24.1)Or, we can use the neural IR techniques of Section??. The simplest of those is abi-encoder model, in which we train two separate encoders, one to encode the userquery and one to encode the candidate response, and use the dot product betweenthese two vectors as the score (Fig.24.6a). For example to implement this usingBERT, we would have two encoders BERTQand BERTRand we could represent thequery and candidate response as the[CLS]token of the respective encoders:hq=BERTQ(q)[CLS]hr=BERTR(r)[CLS]response(q,C)=argmaxr2Chq\u00b7hr(24.2) Response by generationThink of response production as an encoder-decoder taskGenerate each token rtof the response by conditioning on the encoding of the entire query q and the response so far r1...rt\u2212124.2\u2022CHATBOTS11The IR-based approach can be extended in various ways, such as by using moresophisticated neural architectures(Humeau et al., 2020), or by using a longer contextfor the query than just the user\u2019s last turn, up to the whole preceding conversation.Information about the user or sentiment or other information can also play a role.Response by generationAn alternate way to use a corpus to generate dialogue isto think of response production as an encoder-decoder task\u2014 transducing from theuser\u2019s prior turn to the system\u2019s turn. We can think of this as a machine learningversion of ELIZA; the system learns from a corpus to transduce a question to ananswer.Ritter et al. (2011)proposed early on to think of response generation asa kind of translation, and this idea was generalized to the encoder-decoder modelroughly contemporaneously byShang et al. (2015),Vinyals and Le (2015), andSordoni et al. (2015).As we saw in Chapter 11, encoder decoder models generate each tokenrtof theresponse by conditioning on the encoding of the entire queryqand the response sofarr1...rt\u00001:\u02c6rt=argmaxw2VP(w|q,r1...rt\u00001)(24.3)Fig.24.6shows the intuition of the generator and retriever methods for responsegeneration. In the generator architecture, we normally include a longer context,forming the query not just from the user\u2019s turn but from the entire conversation-so-far. Fig.24.7shows an \ufb02eshed-out example.ENCODERqueryq1\u2026qnENCODERresponser1\u2026rndot-producthqhrENCODERq1\u2026qnDECODERr1\u2026<S>r1r2\u2026rn(a) Response by Retrieval (b) Response by GenerationFigure 24.6Two architectures for generating responses for a neural chatbot. In response byretrieval (a) we choose a response by using a \ufb01nding the turn in the corpus whose encodinghas the highest dot-product with the user\u2019s turn. In response by generation (b) we use anencoder-decoder to generate the response.That is quite an accomplishment and you should be proud!ENCODERDECODER[U:] I \ufb01nally got promoted today at work!  [S:] Congrats! That\u2019s great! [U:] Thank you! I\u2019ve been trying to get it for a while now!Figure 24.7Example of encoder decoder for dialogue response generation; the encoder sees the entire dia-logue context.A number of modi\ufb01cations are required to the basic encoder-decoder model toadapt it for the task of response generation. For example basic encoder-decoder24.2\u2022CHATBOTS11The IR-based approach can be extended in various ways, such as by using moresophisticated neural architectures(Humeau et al., 2020), or by using a longer contextfor the query than just the user\u2019s last turn, up to the whole preceding conversation.Information about the user or sentiment or other information can also play a role.Response by generationAn alternate way to use a corpus to generate dialogue isto think of response production as an encoder-decoder task\u2014 transducing from theuser\u2019s prior turn to the system\u2019s turn. We can think of this as a machine learningversion of ELIZA; the system learns from a corpus to transduce a question to ananswer.Ritter et al. (2011)proposed early on to think of response generation asa kind of translation, and this idea was generalized to the encoder-decoder modelroughly contemporaneously byShang et al. (2015),Vinyals and Le (2015), andSordoni et al. (2015).As we saw in Chapter 11, encoder decoder models generate each tokenrtof theresponse by conditioning on the encoding of the entire queryqand the response sofarr1...rt\u00001:\u02c6rt=argmaxw2VP(w|q,r1...rt\u00001)(24.3)Fig.24.6shows the intuition of the generator and retriever methods for responsegeneration. In the generator architecture, we normally include a longer context,forming the query not just from the user\u2019s turn but from the entire conversation-so-far. Fig.24.7shows an \ufb02eshed-out example.ENCODERqueryq1\u2026qnENCODERresponser1\u2026rndot-producthqhrENCODERq1\u2026qnDECODERr1\u2026<S>r1r2\u2026rn(a) Response by Retrieval (b) Response by GenerationFigure 24.6Two architectures for generating responses for a neural chatbot. In response byretrieval (a) we choose a response by using a \ufb01nding the turn in the corpus whose encodinghas the highest dot-product with the user\u2019s turn. In response by generation (b) we use anencoder-decoder to generate the response.That is quite an accomplishment and you should be proud!ENCODERDECODER[U:] I \ufb01nally got promoted today at work!  [S:] Congrats! That\u2019s great! [U:] Thank you! I\u2019ve been trying to get it for a while now!Figure 24.7Example of encoder decoder for dialogue response generation; the encoder sees the entire dia-logue context.A number of modi\ufb01cations are required to the basic encoder-decoder model toadapt it for the task of response generation. For example basic encoder-decoder Response by generation24.2\u2022CHATBOTS11The IR-based approach can be extended in various ways, such as by using moresophisticated neural architectures(Humeau et al., 2020), or by using a longer contextfor the query than just the user\u2019s last turn, up to the whole preceding conversation.Information about the user or sentiment or other information can also play a role.Response by generationAn alternate way to use a corpus to generate dialogue isto think of response production as an encoder-decoder task\u2014 transducing from theuser\u2019s prior turn to the system\u2019s turn. We can think of this as a machine learningversion of ELIZA; the system learns from a corpus to transduce a question to ananswer.Ritter et al. (2011)proposed early on to think of response generation asa kind of translation, and this idea was generalized to the encoder-decoder modelroughly contemporaneously byShang et al. (2015),Vinyals and Le (2015), andSordoni et al. (2015).As we saw in Chapter 11, encoder decoder models generate each tokenrtof theresponse by conditioning on the encoding of the entire queryqand the response sofarr1...rt\u00001:\u02c6rt=argmaxw2VP(w|q,r1...rt\u00001)(24.3)Fig.24.6shows the intuition of the generator and retriever methods for responsegeneration. In the generator architecture, we normally include a longer context,forming the query not just from the user\u2019s turn but from the entire conversation-so-far. Fig.24.7shows an \ufb02eshed-out example.ENCODERqueryq1\u2026qnENCODERresponser1\u2026rndot-producthqhrENCODERq1\u2026qnDECODERr1\u2026<S>r1r2\u2026rn(a) Response by Retrieval (b) Response by GenerationFigure 24.6Two architectures for generating responses for a neural chatbot. In response byretrieval (a) we choose a response by using a \ufb01nding the turn in the corpus whose encodinghas the highest dot-product with the user\u2019s turn. In response by generation (b) we use anencoder-decoder to generate the response.That is quite an accomplishment and you should be proud!ENCODERDECODER[U:] I \ufb01nally got promoted today at work!  [S:] Congrats! That\u2019s great! [U:] Thank you! I\u2019ve been trying to get it for a while now!Figure 24.7Example of encoder decoder for dialogue response generation; the encoder sees the entire dia-logue context.A number of modi\ufb01cations are required to the basic encoder-decoder model toadapt it for the task of response generation. For example basic encoder-decoder Response by generationAlternative approach: fine-tune a large language model  on conversational dataThe Chirpy Cardinal system (Paranjape et al., 2020):\u2022fine-tunes GPT-2 \u2022on the EMPATHETICDIALOGUESdataset (Rashkinet al., 2019)  P. G .  Wo d e h o u s e  p r e d i c t s  n e u r a l  c h a t b o t s\u201cWhat ho!\u201d I said.\u201cWhat ho!\u201d said Motty.\u201cWhat ho! What ho!\u201d\u201cWhat ho! What ho! What ho!\u201dAfter that it seemed rather difficult to go on with the conversation.\u201d Wodehouse My Man Jeeves, 1919 Ongoing research problem:Neural chatbots can get repetitive and boringA: Where are you going?B: I\u2019m going to the restroom.A: See you later.B: See you later.A: See you later.B: See you later.A: How old are you?B: I\u2019m 16.A: 16?B: I don\u2019t know what you are talking about.A: You don\u2019t know what you are saying.B: I don\u2019t know what you are talking about.A: You don\u2019t know what you are saying. Response by retrieving and refining knowledge Can generate responses from informative text rather than dialogue\u2022To respond to turns like \u201cTell me something about Beijing\u201d\u2022XiaoIcecollects sentences from public lectures and news articles.\u2022And searches them using IR based on query expansion from user\u2019s turn\u2022Can augment encoder-decoder model\u2022use IR to retrieve passages from Wikipedia \u2022concatenate each Wikipedia sentence to the dialogue context with a separator token. \u2022Give as encoder context to the encoder-decoder model, which learns to incorporate text into its response Hybrid ArchitecturesChirpy Cardinal (Paranjape et al., 2020) response generation from a series of different generators:\u2022GPT-2 finetuned on EmpatheticDialogues\u2022GPT-2 finetuned to paraphrase content from Wikipedia\u2022Rule-based movie or music generators that produce scripted conversation about a movie or a musician\u2022asking the user\u2019s opinion about a movie, \u2022giving a fun fact, \u2022asking the user their opinion on an actor in the movie.  Chirpy Cardinal (Paranjape et al. 2020)24.3\u2022GUS: SIMPLEFRAME-BASEDDIALOGUESYSTEMS13linker speci\ufb01es what entity is currently being discussed. Dialogue act classi\ufb01cationis used to detect when the user is asking a question or giving an af\ufb01rmative versusnegative response.Bot responses are generated by a series of response generators. Some responsegenerators use \ufb01ne-tuned neural language models: a GPT-2(Radford et al., 2019)language model \ufb01ne-tuned on the EmpatheticDialogues dataset, and a separately\ufb01ne-tuned GPT-2 language model that learns in \ufb01ne-tuning to paraphrase contentfrom Wikipedia in response to questions.Other response generators are closer to rule-based; the movie or music genera-tors produce scripted conversation about a movie or a musician, such as asking asksthe user\u2019s opinion about a movie, giving a fun fact, asks the user their opinion on anactor in the movie, and so on. These generators use regexes and sentiment classi\ufb01ersto classify user responses and handwritten templates to generate bot utterances.Each response generators can specify a priority, and a ranker is used to passcontrol between response generators. Fig.24.8shows part of a conversation.# Utterance Entity Commentary1User:let\u2019s chat NoneBot:Hi, this is an Alexa Prize Socialbot. I\u2019d love toget to know you a bit better before we chat! Is it allright if I ask for your name?[ Launch ]NoneLaunchgivesFORCE_STARTresponse to be-gin the conversation.2User:my name is chris NoneBot:Well it\u2019s nice to meet you, Chris! I\u2019m excitedto chat with you today.[ Launch ]I hope your after-noon is going well. What are your plans for the restof today?[ Neural Chat ]NoneLaunchgives aSTRONG_CONTINUEre-sponse that needs a prompt, which is suppliedbyNeural ChatwithFORCE_STARTpriority(scripted transition).3User:hang out with my friends NoneBot:That sounds great, I love spending time withfriends. What will you do together?[ Neural Chat ]NoneNeural Chatgenerates a GPT-2 response withSTRONG_CONTINUEpriority.4User:maybe watch a movie NoneBot:Me too! I love watching movies; I get to learnso much about what the world is like outside ofthe cloud! Have you seen any movies recently?[ Movies ]FilmMoviesdetects the \u2018movie\u2019 keyword in theuser\u2019s utterance and gives aFORCE_STARTresponse (overridingNeural Chat), settingcurrent entity toFilm.5User:i saw the matrixThe MatrixTheEntity LinkerandEntity Trackeriden-tifyThe Matrixas the current entity.Bot:Nice! Did you like The Matrix?[ Movies ]The Matrix6User:i loved it neo is amazingThe MatrixBot:Oooh, yeah, I agree. Hey, isn\u2019t Keanu Reevesin that movie? What do you think about KeanuReeves?[ Movies ]KeanuReevesMoviessets current entity toKeanu Reeves.7User:i want to talk about something else NoneNavigational Intentdetectsnegative intent, soEntity Trackerrejects the current entity.Bot:OK, no problem.[ Movies ]There\u2019s actuallysomething else I wanted to ask you about. What\u2019syour favorite animal?[ Categories ]AnimalMoviesacknowledges and asks for a prompt.Categoriesgives aGENERICprompt.8User:i love cats CatEntity LinkerandEntity TrackeridentifyCatas current entity.Bot:Good to hear you like cats. I have to be honestthough, I\u2019m not a big fan of cats. I feel like catsdon\u2019t respect my personal space, but I would love tohear why you like cats?[ Opinion ]CatOpiniongives aCAN_STARTresponse aboutcats.9User:hmm i love cats because they are \ufb02uffy CatBot:That make sense. Now that I think aboutit, one good reason to like cats is that they purrand I de\ufb01nitely need that kind of positive feed-back.[ Opinion ]Wanna know something interest-ing about cat?[ Wiki ]CatOpinion\ufb01nishes talking.Wikigives aCUR-RENT_TOPICprompt aboutCat.10User:you are not very smart CatBot:I know you feel frustrated. I\u2019m always tryingto get better.[ Offensive User ]I\u2019ve been listeningto some new music today and I wanted to chat aboutinstruments. If you were a musical instrument whichone would you be?[ Music ]Musical in-strumentOffensive Usergives aFORCE_STARTre-sponse to the critical utterance, then[ Music ]provides aGENERICprompt to steer the con-versation in a new direction.11User:what do you \ufb01nd interesting None The user gives an unexpected response.Bot:I like the violin, but I\u2019m more of a classicalmusic player.[ Neural Fallback ]I remember youmentioned Neo. Would you like to talk more aboutit?[ Wiki ]Neo (TheMatrix)Neural Fallbackgives a GPT-2-generated re-sponse, which is selected as no other RG (ex-ceptFallback) has a response.Wikigives aCONTEXTUALprompt.12User:sureNeo (TheMatrix)Bot:Neo has got into so many adventures! Whatare some adventures that you\u2019ve enjoyed?[ Wiki ]Neo (TheMatrix)Wikiasks an open-ended question (speci\ufb01c tomovie characters) aboutNeo (The Matrix).13User:morpheus teaching jujitsu to neoNeo (TheMatrix)Bot:I liked that Neo and Trinity were able to res-cue Morpheus from a building protected by armedguards and agents. Morpheus has done so manyamazing things! What are some of your favoriteMorpheus moments?[ Wiki ]Morpheus(TheMatrix)Wikigives a response (generated by the GPT-2 conversational paraphrasing model), sets thecurrent entity toMorpheus (The Matrix), andasks another open-ended question.14User:i want to stop talking None We detectstop intent; the conversation ends.Table 1: An example dialogue. To respect users\u2019 privacy, this is not a real user conversation.5Figure 24.8Part of a conversation with Chirpy Cardinal(Paranjape et al., 2020).24.3 GUS: Simple Frame-based Dialogue SystemsWe turn now totask-based dialogue, in which a dialogue system has the goal ofhelping a user solve some task like making an airplane reservation or buying a prod-uct. In this section we introduce the very simpleGUSarchitecture for task-basedGUSdialogue. Although it is quite old, (it was \ufb01rst introduced in 1977 in the in\ufb02uen-tial GUS system for travel planning(Bobrow et al., 1977)) the GUS architecturehas been astonishingly long-lived and underlies most if not all modern commercialdigital assistants. Figure24.9shows a conversation with the original GUS system.All modern task-based dialogue systems, whether the simple GUS architecturewe describe here, or the more sophisticateddialogue statearchitectures we turn todialogue statein the following section, are based aroundframes. A frame is a kind of knowledgeframestructure representing the kinds of intentions the system can extract from user sen-tences, and consists of a collection ofslots, each of which can take a set of possibleslot Chatbots: pro and conPro:\u25e6Fun\u25e6Good for narrow, scriptable applicationsCons:\u25e6They don't really understand\u25e6Giving the appearance of understanding may be problematic\u25e6Rule-based chatbotsare expensive and brittle\u25e6IR-based chatbotscan only mirror training data\u25e6We'll discuss later the case of Microsoft Tay\u25e6(or, Garbage-in, Garbage-out)Next steps: integrating chatbot ability into frame-based agents Chatbots and Dialogue SystemsCorpus-based Chatbots Chatbots and Dialogue SystemsThe Frame-based (\"GUS\") Dialogue Architecture Frame-based dialogue agentsSometimes called \"task-based dialogue agents\"\u2022Systems that have the goal of helping a user solve a task like making a travel reservation or buying a productArchitecture: \u2022First proposed in the GUS system of 1977\u2022A knowledge structure representing user intentions\u2022One or more frames (eachconsistingofslots withvalues) The FrameA set of slots, to be filled with information of a given typeEach associated with a questionto the userSometimes called a domain ontologySlotTypeQuestionORIGINcity\"What city are you leaving from?DESTcity\"Where are you going?DEP DATE date\"What day would you like to leave?DEP TIME time\"What time would you like to leave?AIRLINEline\"What is your preferred airline? Two basic architecturesThe GUSarchitecture\u2022Sometimes just called \"frame-based\" architecture\u2022Over 40 years old, but still used in most industrial task-based dialogue agents\u2022Bobrow, Daniel G., Ronald M. Kaplan, Martin Kay, Donald A. Norman, Henry Thompson, and Terry Winograd. 1977. \"GUS, a frame-driven dialog system.\"Artificial Intelligence8, 2:155-173.The dialogue-statearchitecture\u2022Extension of GUS\u2022More common in research systems\u2022Some aspects making their way into industrial systems 14CHAPTER24\u2022CHATBOTS&DIALOGUESYSTEMSvalues. Together this set of frames is sometimes called adomain ontology.GUS: Hello. My name is GUS. I can help you plan a simple trip by air. Where do youwant to go ?Client: I want to go to San Diego on May 28GUS: What time do you want to leave?Client: I must be in San Diego before 10 amGUS: Would you like P.S.A. \ufb02ight 102 that arrives at 9.15 am ?Client: What is the next \ufb02ight ?GUS: Air California \ufb02ight 310 that leaves at 8.30am and arrives at 10.00amClient: I\u2019ll take the \ufb01rst oneGUS: In what name should I make a reservation, please?Client: Dan BobrowGUS: I have con\ufb01rmed the following \ufb02ight: P.S.A. \ufb02ight 102 on Wednesday May 28from San Jose to San Diego that leaves at 7.30 am and arrives at 9.15 am.GUS: What date do you want to return on ?Client: On Friday in the evening.GUS: Would you like the \ufb02ight that leaves at 7.45 pm ?Client: That\u2019s \ufb01ne.GUS: I have con\ufb01rmed the following \ufb02ight: P.S.A. \ufb02ight 307 on Friday May 30 fromSan Diego to San Jose that leaves at 7.45 pm and arrives at 9.30 pm Thank youfor calling. GoodbyeFigure 24.9The travel domain: A transcript of an actual dialogue with the GUS system ofBobrow et al. (1977). P.S.A. and Air California were airlines of that period.The set of slots in a task-based dialogue frame speci\ufb01es what the system needsto know, and the \ufb01ller of each slot is constrained to values of a particular semantictype. In the travel domain, for example, a slot might be of type city (hence take onvalues likeSan Francisco, orHong Kong) or of type date, airline, or time.SlotTypeQuestion TemplateORIGIN CITYcity\u201cFrom what city are you leaving?\u201dDESTINATION CITYcity\u201cWhere are you going?\u201dDEPARTURE TIMEtime\u201cWhen would you like to leave?\u201dDEPARTURE DATEdate\u201cWhat day would you like to leave?\u201dARRIV AL TIMEtime\u201cWhen do you want to arrive?\u201dARRIV AL DATEdate\u201cWhat day would you like to arrive?\u201dFigure 24.10A frame in a frame-based dialogue system, showing the type of each slot anda question used to \ufb01ll the slot.Types in GUS, as in modern frame-based dialogue agents, have hierarchicalstructure; for example thedatetype in GUS is itself a frame with slots with typeslikeintegeror members of sets of weekday names:DATEMONTH:NAME YEAR:INTEGER DAY:(BOUNDED-INTEGER 1 31)WEEKDAY:(MEMBER (Sunday Monday Tuesday WednesdayThursday Friday Saturday))24.3.1 Control structure for frame-based dialogueThe control architecture for frame-based dialogue systems, used in various formsin modern systems like Apple\u2019s Siri, Amazon\u2019s Alexa, and the Google Assistant, isdesigned around the frame. The system\u2019s goal is to \ufb01ll the slots in the frame with the Control structure for GUS frame architectureSystem asks questions of user, filling any slots that user specifiesUser might fill many slots at a time:\u2022I want a flight from San Francisco to Denver one wayleaving after five p.m. on Tuesday. When frame is filled, do database query GUS slots have condition-action rules attachedSome rules attached to the DESTINATION slot for the plane booking frame1.Once the user has specified the destination\u2022Enter that city as the default StayLocationfor the hotel booking frame. 2.Once the user has specified DESTINATION DAY for a short trip \u2022Automatically copy as ARRIVAL DAY .  GUS systems have multiple framesFrames like:\u2022Car or hotel reservations\u2022General route information\u2022Which airlines fly from Boston to San Francisco?, \u2022Information about airfare practices \u2022Do I have to stay a specific number of days to get a decent airfare?). Frame detection:\u2022System must detect which slot of which frame user is filling\u2022And switch dialogue control to that frame.  GUS: Natural Language Understanding for filling dialog slots1.Domain classificationAsking weather? Booking a flight? Programming alarm clock?2.Intent DeterminationFind a Movie, Show Flight, Remove Calendar Appt3.Slot FillingExtract the actual slots and fillers Natural Language Understanding for filling slotsShow me morning flights from Boston to SF on Tuesday.29.2\u2022FRAMEBASEDDIALOGAGENTS1329.2.2 Natural language understanding for \ufb01lling slotsThe goal of the natural language understanding component is to extract three thingsfrom the user\u2019s utterance. The \ufb01rst task isdomain classi\ufb01cation: is this user fordomainclassi\ufb01cationexample talking about airlines, programming an alarm clocks, or dealing with theircalendar? Of course this 1-of-n classi\ufb01cation tasks is unnecessary for single-domainsystems that are focused on, say, only calendar management, but multi-domain di-alog systems are the modern standard. The second is userintent determination:intentdeterminationwhat general task or goal is the user trying to accomplish? For example the taskcould be to Find a Movie, or Show a Flight, or Remove a Calendar Appointment.Finally, we need to doslot \ufb01lling: extract the particular slots and \ufb01llers that the userslot \ufb01llingintends the system to understand from their utterance with respect to their intent.From a user utterance like this one:Show me morning flights from Boston to San Francisco on Tuesdaya system might want to build a representation like:DOMAIN: AIR-TRAVELINTENT: SHOW-FLIGHTSORIGIN-CITY: BostonORIGIN-DATE: TuesdayORIGIN-TIME: morningDEST-CITY: San Franciscowhile an utterance likeWake me tomorrow at 6should give an intent like this:DOMAIN: ALARM-CLOCKINTENT: SET-ALARMTIME: 2017-07-01 0600-0800The task of slot-\ufb01lling, and the simpler tasks of domain and intent classi\ufb01cation,are special cases of the task of semantic parsing discussed in Chapter??. Dialogueagents can thus extract slots, domains, and intents from user utterances by applyingany of the semantic parsing approaches discussed in that chapter.The method used in the original GUS system, and still quite common in indus-trial applications, is to use hand-written rules, often as part of the condition-actionrules attached to slots or concepts.For example we might just de\ufb01ne a regular expression consisting of a set stringsthat map to the SET-ALARM intent:wake me (up) | set (the|an) alarm | get me upWe can build more complex automata that instantiate sets of rules like thosediscussed in Chapter 20, for example extracting a slot \ufb01ller by turning a stringlikeMonday at 2pminto an object of typedatewith parameters (DAY, MONTH,YEAR, HOURS, MINUTES).Rule-based systems can be even implemented with full grammars. Research sys-tems like the Phoenix system(Ward and Issar, 1994)consists of large hand-designedsemantic grammarswith thousands of rules. A semantic grammar is a context-freesemanticgrammargrammar in which the left-hand side of each rule corresponds to the semantic entitiesbeing expressed (i.e., the slot names) as in the following fragment: Natural Language Understanding for filling slotsWake me tomorrow at six.29.2\u2022FRAMEBASEDDIALOGAGENTS1329.2.2 Natural language understanding for \ufb01lling slotsThe goal of the natural language understanding component is to extract three thingsfrom the user\u2019s utterance. The \ufb01rst task isdomain classi\ufb01cation: is this user fordomainclassi\ufb01cationexample talking about airlines, programming an alarm clocks, or dealing with theircalendar? Of course this 1-of-n classi\ufb01cation tasks is unnecessary for single-domainsystems that are focused on, say, only calendar management, but multi-domain di-alog systems are the modern standard. The second is userintent determination:intentdeterminationwhat general task or goal is the user trying to accomplish? For example the taskcould be to Find a Movie, or Show a Flight, or Remove a Calendar Appointment.Finally, we need to doslot \ufb01lling: extract the particular slots and \ufb01llers that the userslot \ufb01llingintends the system to understand from their utterance with respect to their intent.From a user utterance like this one:Show me morning flights from Boston to San Francisco on Tuesdaya system might want to build a representation like:DOMAIN: AIR-TRAVELINTENT: SHOW-FLIGHTSORIGIN-CITY: BostonORIGIN-DATE: TuesdayORIGIN-TIME: morningDEST-CITY: San Franciscowhile an utterance likeWake me tomorrow at 6should give an intent like this:DOMAIN: ALARM-CLOCKINTENT: SET-ALARMTIME: 2017-07-01 0600-0800The task of slot-\ufb01lling, and the simpler tasks of domain and intent classi\ufb01cation,are special cases of the task of semantic parsing discussed in Chapter??. Dialogueagents can thus extract slots, domains, and intents from user utterances by applyingany of the semantic parsing approaches discussed in that chapter.The method used in the original GUS system, and still quite common in indus-trial applications, is to use hand-written rules, often as part of the condition-actionrules attached to slots or concepts.For example we might just de\ufb01ne a regular expression consisting of a set stringsthat map to the SET-ALARM intent:wake me (up) | set (the|an) alarm | get me upWe can build more complex automata that instantiate sets of rules like thosediscussed in Chapter 20, for example extracting a slot \ufb01ller by turning a stringlikeMonday at 2pminto an object of typedatewith parameters (DAY, MONTH,YEAR, HOURS, MINUTES).Rule-based systems can be even implemented with full grammars. Research sys-tems like the Phoenix system(Ward and Issar, 1994)consists of large hand-designedsemantic grammarswith thousands of rules. A semantic grammar is a context-freesemanticgrammargrammar in which the left-hand side of each rule corresponds to the semantic entitiesbeing expressed (i.e., the slot names) as in the following fragment: How to fill slots?(1) Rule-based Slot-fillingWrite regular expressions or grammar rulesWake me (up) | set (the|an) alarm | get me upDo text normalization Generating responses: template-based generationA template is a pre-built response stringTemplates can be fixed:\"Hello, how can I help you?\"Or have variables:\"What time do you want to leave CITY-ORIG?\"\"Will you return to CITY-ORIG from CITY-DEST?\" Summary: simple frame-based architectureLike many rule-based approaches\u2022Positives:\u2022High precision\u2022Can provide coverage if the domain is narrow\u2022Negatives:\u2022Can be expensive and slow to create rules\u2022Can suffer from recall problems Chatbots and Dialogue SystemsThe Frame-based (\"GUS\") Dialogue Architecture Chatbots and Dialogue SystemsThe Dialogue-State Architecture Dialogue-State or Belief-State ArchitectureA more sophisticated version of the frame-based architecture\u2022Has dialogue acts, more ML, better generationThe basis for modern research systemsSlowly making its way into industrial systems\u2022Some aspects (ML for slot-understanding) already widely used industrially DIALOGSTATETRACKINGOVERVIEW\nLEAVING FROM DOWNTOWNLEAVING AT ONE P MARRIVING AT ONE P M0.60.20.1{ from: downtown }{ depart-time: 1300 }{ arrive-time: 1300 }0.50.30.1\nfrom:        CMUto:          airportdepart-time: 1300confirmed:   noscore:       0.10from:        CMUto:          airportdepart-time: 1300confirmed:   noscore:       0.15from:        downtownto:          airportdepart-time: --confirmed:   noscore:       0.65Automatic Speech Recognition (ASR)Spoken Language Understanding (SLU)Dialog State Tracker (DST)\nDialog Policyact:  confirmfrom: downtownFROM DOWNTOWN, IS THAT RIGHT?Natural Language Generation (NLG)Text to Speech (TTS)\nFigure 1: Principal components of a spoken dialog system.The topic of this paper is thedialog state tracker(DST). The DST takes as input all of the dialoghistory so far, and outputs its estimate of the currentdialog state\u2013 for example, in a restaurantinformation system, the dialog state might indicate the user\u2019s preferred price range and cuisine,what information they are seeking such as the phone number of a restaurant, and which conceptshave been stated vs. con\ufb01rmed. Dialog state tracking is dif\ufb01cult because ASR and SLU errors arecommon, and can cause the system to misunderstand the user. At the same time, state tracking iscrucial because the dialog policy relies on the estimated dialog state to choose actions \u2013 for example,which restaurants to suggest.In the literature, numerous methods for dialog state tracking have been proposed. These arecovered in detail in Section 3; illustrative examples include hand-crafted rules (Larsson and Traum,2000; Bohus and Rudnicky, 2003), heuristic scores (Higashinaka et al., 2003), Bayesian networks(Paek and Horvitz, 2000; Williams and Young, 2007), and discriminative models (Bohus and Rud-nicky, 2006). Techniques have been \ufb01elded which scale to realistically sized dialog problems andoperate in real time (Young et al., 2010; Thomson and Young, 2010; Williams, 2010; Mehta et al.,2010). In end-to-end dialog systems, dialog state tracking has been shown to improve overall systemperformance (Young et al., 2010; Thomson and Young, 2010).Despite this progress, direct comparisons between methodshavenot been possible because paststudies use different domains and different system components for ASR, SLU, dialog policy, etc.Moreover, there has not been a standard task or methodology for evaluating dialog state tracking.Together these issues have limited progress in this research area.The Dialog State Tracking Challenge (DSTC) series has provided a \ufb01rst common testbed andevaluation suite for dialog state tracking. Three instances of the DSTC have been run over a three5The Dialogue-State ArchitectureWilliams, Jason D., Antoine Raux, and Matthew Henderson. \"The dialog state tracking challenge series: A review.\" Dialogue & Discourse 7, no. 3 (2016): 4-33. Components in a dialogue-state architectureNLU: extracts slot fillers from the user\u2019s utterance using machine learningDialogue state tracker: maintains the current state of the dialogue (user\u2019s most recent dialogue act, set of slot-filler constraints from userDialogue policy: decides what the system should do or say next\u2022GUS policy: ask questions until the frame was full then report back\u2022More sophisticated: know when to answer questions, when to ask a clarification question, etc.NLG: produce more natural, less templated utterances Dialogue ActsCombine the ideas of speech acts and groundinginto a single representation24.4\u2022THEDIALOGUE-STATEARCHITECTURE19Tag Sys User DescriptionHELLO(a=x,b=y,. . .)XXOpen a dialogue and give infoa=x,b=y,. . .INFORM(a=x,b=y,. . .)XXGive infoa=x,b=y,. . .REQUEST(a,b=x,. . .)XXRequest value for a givenb=x,. . .REQALTS(a=x,. . .)cXRequest alternative witha=x,. . .CONFIRM(a=x,b=y,. . .)XXExplicitly con\ufb01rma=x,b=y,. . .CONFREQ(a=x,. . . ,d)XcImplicitly con\ufb01rma=x,. . .and request value ofdSELECT(a=x,a=y)XcImplicitly con\ufb01rma=x,. . .and request value ofdAFFIRM(a=x,b=y,. . .)XXAf\ufb01rm and give further infoa=x,b=y,. . .NEGATE(a=x)cXNegate and give corrected valuea=xDENY(a=x)cXDeny thata=xBYE()XXClose a dialogueFigure 24.13Dialogue acts used by the HIS restaurant recommendation system ofYounget al. (2010). The Sys and User columns indicate which acts are valid as system outputs anduser inputs, respectively.Utterance Dialogue actU: Hi, I am looking for somewhere to eat.hello(task = find,type=restaurant)S: You are looking for a restaurant. Whattype of food do you like?confreq(type = restaurant, food)U: I\u2019d like an Italian somewhere near themuseum.inform(food = Italian, near=museum)S: Roma is a nice Italian restaurant nearthe museum.inform(name = \"Roma\", type = restaurant,food = Italian, near = museum)U: Is it reasonably priced?confirm(pricerange = moderate)S: Yes, Roma is in the moderate pricerange.affirm(name = \"Roma\", pricerange =moderate)U: What is the phone number?request(phone)S: The number of Roma is 385456.inform(name = \"Roma\", phone = \"385456\")U: Ok, thank you goodbye. bye()Figure 24.14A sample dialogue from the HIS System ofYoung et al. (2010)using the dialogue acts inFig.24.13.24.4.2 Slot FillingThe task of slot-\ufb01lling, and the simpler tasks of domain and intent classi\ufb01cation, arespecial cases of the task of supervised semantic parsing discussed in Chapter 16, inwhich we have a training set that associates each sentence with the correct set ofslots, domain, and intent.A simple method is to train a a sequence model to map from input words repre-sentation to slot \ufb01llers, domain and intent. For example given the sentence:I want to fly to San Francisco on Monday afternoon pleasewe compute a sentence representation, for example by passing the sentence througha contextual embedding network like BERT. The resulting sentence representationcan be passed through a feedforward layer and then a simple 1-of-N classi\ufb01er todetermine that the domain isAIRLINEand and the intent isSHOWFLIGHT.Our training data is sentences paired with sequences ofBIOlabels:O O O O O B-DES I-DES O B-DEPTIME I-DEPTIME OI want to fly to San Francisco on Monday afternoon pleaseRecall from Chapter 8 that in BIO tagging we introduce a tag for the beginning(B) and inside (I) of each slot label, and one for tokens outside (O) any slot label.The number of tags is thus 2n+1 tags, wherenis the number of slots.Young et al., 2010: Dialogue Acts24.4\u2022THEDIALOGUE-STATEARCHITECTURE19Tag Sys User DescriptionHELLO(a=x,b=y,. . .)XXOpen a dialogue and give infoa=x,b=y,. . .INFORM(a=x,b=y,. . .)XXGive infoa=x,b=y,. . .REQUEST(a,b=x,. . .)XXRequest value for a givenb=x,. . .REQALTS(a=x,. . .)cXRequest alternative witha=x,. . .CONFIRM(a=x,b=y,. . .)XXExplicitly con\ufb01rma=x,b=y,. . .CONFREQ(a=x,. . . ,d)XcImplicitly con\ufb01rma=x,. . .and request value ofdSELECT(a=x,a=y)XcImplicitly con\ufb01rma=x,. . .and request value ofdAFFIRM(a=x,b=y,. . .)XXAf\ufb01rm and give further infoa=x,b=y,. . .NEGATE(a=x)cXNegate and give corrected valuea=xDENY(a=x)cXDeny thata=xBYE()XXClose a dialogueFigure 24.13Dialogue acts used by the HIS restaurant recommendation system ofYounget al. (2010). The Sys and User columns indicate which acts are valid as system outputs anduser inputs, respectively.Utterance Dialogue actU: Hi, I am looking for somewhere to eat.hello(task = find,type=restaurant)S: You are looking for a restaurant. Whattype of food do you like?confreq(type = restaurant, food)U: I\u2019d like an Italian somewhere near themuseum.inform(food = Italian, near=museum)S: Roma is a nice Italian restaurant nearthe museum.inform(name = \"Roma\", type = restaurant,food = Italian, near = museum)U: Is it reasonably priced?confirm(pricerange = moderate)S: Yes, Roma is in the moderate pricerange.affirm(name = \"Roma\", pricerange =moderate)U: What is the phone number?request(phone)S: The number of Roma is 385456.inform(name = \"Roma\", phone = \"385456\")U: Ok, thank you goodbye. bye()Figure 24.14A sample dialogue from the HIS System ofYoung et al. (2010)using the dialogue acts inFig.24.13.24.4.2 Slot FillingThe task of slot-\ufb01lling, and the simpler tasks of domain and intent classi\ufb01cation, arespecial cases of the task of supervised semantic parsing discussed in Chapter 16, inwhich we have a training set that associates each sentence with the correct set ofslots, domain, and intent.A simple method is to train a a sequence model to map from input words repre-sentation to slot \ufb01llers, domain and intent. For example given the sentence:I want to fly to San Francisco on Monday afternoon pleasewe compute a sentence representation, for example by passing the sentence througha contextual embedding network like BERT. The resulting sentence representationcan be passed through a feedforward layer and then a simple 1-of-N classi\ufb01er todetermine that the domain isAIRLINEand and the intent isSHOWFLIGHT.Our training data is sentences paired with sequences ofBIOlabels:O O O O O B-DES I-DES O B-DEPTIME I-DEPTIME OI want to fly to San Francisco on Monday afternoon pleaseRecall from Chapter 8 that in BIO tagging we introduce a tag for the beginning(B) and inside (I) of each slot label, and one for tokens outside (O) any slot label.The number of tags is thus 2n+1 tags, wherenis the number of slots.Young et al., 2010: Slot filling: Machine learningMachine learning classifiers to map words to semantic frame-fillersGiven a set of labeled sentencesInput: \"I want to fly to San Francisco on Monday please\"Output: Destination: SFDepart-time: MondayBuild a classifier to map from one to the otherRequirements: Lots of labeled data Slot filling as sequence labeling: BIO taggingThe BIO tagging paradigmIdea: Train a classifier to label each input word with a tag that tells us what slot (if any) it fillsWe create a B and I tag for each slot-typeAnd convert the training data to this format24.4\u2022THEDIALOGUE-STATEARCHITECTURE19Tag Sys User DescriptionHELLO(a=x,b=y,. . .)XXOpen a dialogue and give infoa=x,b=y,. . .INFORM(a=x,b=y,. . .)XXGive infoa=x,b=y,. . .REQUEST(a,b=x,. . .)XXRequest value for a givenb=x,. . .REQALTS(a=x,. . .)cXRequest alternative witha=x,. . .CONFIRM(a=x,b=y,. . .)XXExplicitly con\ufb01rma=x,b=y,. . .CONFREQ(a=x,. . . ,d)XcImplicitly con\ufb01rma=x,. . .and request value ofdSELECT(a=x,a=y)XcImplicitly con\ufb01rma=x,. . .and request value ofdAFFIRM(a=x,b=y,. . .)XXAf\ufb01rm and give further infoa=x,b=y,. . .NEGATE(a=x)cXNegate and give corrected valuea=xDENY(a=x)cXDeny thata=xBYE()XXClose a dialogueFigure 24.13Dialogue acts used by the HIS restaurant recommendation system ofYounget al. (2010). The Sys and User columns indicate which acts are valid as system outputs anduser inputs, respectively.Utterance Dialogue actU: Hi, I am looking for somewhere to eat.hello(task = find,type=restaurant)S: You are looking for a restaurant. Whattype of food do you like?confreq(type = restaurant, food)U: I\u2019d like an Italian somewhere near themuseum.inform(food = Italian, near=museum)S: Roma is a nice Italian restaurant nearthe museum.inform(name = \"Roma\", type = restaurant,food = Italian, near = museum)U: Is it reasonably priced?confirm(pricerange = moderate)S: Yes, Roma is in the moderate pricerange.affirm(name = \"Roma\", pricerange =moderate)U: What is the phone number?request(phone)S: The number of Roma is 385456.inform(name = \"Roma\", phone = \"385456\")U: Ok, thank you goodbye. bye()Figure 24.14A sample dialogue from the HIS System ofYoung et al. (2010)using the dialogue acts inFig.24.13.24.4.2 Slot FillingThe task of slot-\ufb01lling, and the simpler tasks of domain and intent classi\ufb01cation, arespecial cases of the task of supervised semantic parsing discussed in Chapter 16, inwhich we have a training set that associates each sentence with the correct set ofslots, domain, and intent.A simple method is to train a a sequence model to map from input words repre-sentation to slot \ufb01llers, domain and intent. For example given the sentence:I want to fly to San Francisco on Monday afternoon pleasewe compute a sentence representation, for example by passing the sentence througha contextual embedding network like BERT. The resulting sentence representationcan be passed through a feedforward layer and then a simple 1-of-N classi\ufb01er todetermine that the domain isAIRLINEand and the intent isSHOWFLIGHT.Our training data is sentences paired with sequences ofBIOlabels:O O O O O B-DES I-DES O B-DEPTIME I-DEPTIME OI want to fly to San Francisco on Monday afternoon pleaseRecall from Chapter 8 that in BIO tagging we introduce a tag for the beginning(B) and inside (I) of each slot label, and one for tokens outside (O) any slot label.The number of tags is thus 2n+1 tags, wherenis the number of slots. Slot filling using contextual embeddings20CHAPTER24\u2022CHATBOTS&DIALOGUESYSTEMSFig.24.15shows the architecture. The input is a series of wordsw1...wn, whichis passed through a contextual embedding model to get contextual word representa-tions. This is followed by a feedforward layer and a softmax at each token positionover possible BIO tags, with the output a series of BIO tagss1...sn. We can alsocombine the domain-classi\ufb01cation and intent-extraction tasks with slot-\ufb01lling sim-ply by adding a domain concatenated with an intent as the desired output for the\ufb01nal EOS token.\nSanFranciscoonMondayEncodingsClassi\ufb01er+softmaxB-DESI-DESOB-DTIME\n\u2026d+i\n<EOS>Encoder (BERT)Figure 24.15A simple architecture for slot \ufb01lling, mapping the words in the input throughcontextual embeddings like BERT to an output classi\ufb01er layer (which can be linear or some-thing more complex), followed by softmax to generate a series of BIO tags (and including a\ufb01nal state consisting of a domain concatenated with an intent).Once the sequence labeler has tagged the user utterance, a \ufb01ller string can beextracted for each slot from the tags (e.g., \u201cSan Francisco\u201d), and these word stringscan then be normalized to the correct form in the ontology (perhaps the airport code\u2018SFO\u2019). This normalization can take place by using homonym dictionaries (specify-ing, for example, that SF, SFO, and San Francisco are the same place).In industrial contexts, machine learning-based systems for slot-\ufb01lling are of-ten bootstrapped from GUS-style rule-based systems in a semi-supervised learningmanner. A rule-based system is \ufb01rst built for the domain, and a test set is carefullylabeled. As new user utterances come in, they are paired with the labeling providedby the rule-based system to create training tuples. A classi\ufb01er can then be trainedon these tuples, using the test set to test the performance of the classi\ufb01er againstthe rule-based system. Some heuristics can be used to eliminate errorful trainingtuples, with the goal of increasing precision. As suf\ufb01cient training samples becomeavailable the resulting classi\ufb01er can often outperform the original rule-based system(Suendermann et al., 2009), although rule-based systems may still remain higher-precision for dealing with complex cases like negation.24.4.3 Dialogue State TrackingThe job of the dialogue-state tracker is to determine both the current state of theframe (the \ufb01llers of each slot), as well as the user\u2019s most recent dialogue act. Thedialogue-state thus includes more than just the slot-\ufb01llers expressed in the currentsentence; it includes the entire state of the frame at this point, summarizing all ofthe user\u2019s constraints. The following example fromMrk\u02c7si\u00b4c et al. (2017)shows therequired output of the dialogue state tracker after each turn:Can do domain and intent too: e.g.,  generate the label  \"AIRLINE_TRAVEL + SEARCH_FLIGHT\" Once we have the BIO tag of the sentence\u2022We can extract the filler string for each slot\u2022And then normalize it to the correct form in the ontology\u2022Like \"SFO\" for San Francisco\u2022Using homonym dictionaries (SF=SFO=San Francisco)24.4\u2022THEDIALOGUE-STATEARCHITECTURE19Tag Sys User DescriptionHELLO(a=x,b=y,. . .)XXOpen a dialogue and give infoa=x,b=y,. . .INFORM(a=x,b=y,. . .)XXGive infoa=x,b=y,. . .REQUEST(a,b=x,. . .)XXRequest value for a givenb=x,. . .REQALTS(a=x,. . .)cXRequest alternative witha=x,. . .CONFIRM(a=x,b=y,. . .)XXExplicitly con\ufb01rma=x,b=y,. . .CONFREQ(a=x,. . . ,d)XcImplicitly con\ufb01rma=x,. . .and request value ofdSELECT(a=x,a=y)XcImplicitly con\ufb01rma=x,. . .and request value ofdAFFIRM(a=x,b=y,. . .)XXAf\ufb01rm and give further infoa=x,b=y,. . .NEGATE(a=x)cXNegate and give corrected valuea=xDENY(a=x)cXDeny thata=xBYE()XXClose a dialogueFigure 24.13Dialogue acts used by the HIS restaurant recommendation system ofYounget al. (2010). The Sys and User columns indicate which acts are valid as system outputs anduser inputs, respectively.Utterance Dialogue actU: Hi, I am looking for somewhere to eat.hello(task = find,type=restaurant)S: You are looking for a restaurant. Whattype of food do you like?confreq(type = restaurant, food)U: I\u2019d like an Italian somewhere near themuseum.inform(food = Italian, near=museum)S: Roma is a nice Italian restaurant nearthe museum.inform(name = \"Roma\", type = restaurant,food = Italian, near = museum)U: Is it reasonably priced?confirm(pricerange = moderate)S: Yes, Roma is in the moderate pricerange.affirm(name = \"Roma\", pricerange =moderate)U: What is the phone number?request(phone)S: The number of Roma is 385456.inform(name = \"Roma\", phone = \"385456\")U: Ok, thank you goodbye. bye()Figure 24.14A sample dialogue from the HIS System ofYoung et al. (2010)using the dialogue acts inFig.24.13.24.4.2 Slot FillingThe task of slot-\ufb01lling, and the simpler tasks of domain and intent classi\ufb01cation, arespecial cases of the task of supervised semantic parsing discussed in Chapter 16, inwhich we have a training set that associates each sentence with the correct set ofslots, domain, and intent.A simple method is to train a a sequence model to map from input words repre-sentation to slot \ufb01llers, domain and intent. For example given the sentence:I want to fly to San Francisco on Monday afternoon pleasewe compute a sentence representation, for example by passing the sentence througha contextual embedding network like BERT. The resulting sentence representationcan be passed through a feedforward layer and then a simple 1-of-N classi\ufb01er todetermine that the domain isAIRLINEand and the intent isSHOWFLIGHT.Our training data is sentences paired with sequences ofBIOlabels:O O O O O B-DES I-DES O B-DEPTIME I-DEPTIME OI want to fly to San Francisco on Monday afternoon pleaseRecall from Chapter 8 that in BIO tagging we introduce a tag for the beginning(B) and inside (I) of each slot label, and one for tokens outside (O) any slot label.The number of tags is thus 2n+1 tags, wherenis the number of slots. The task of dialogue state tracking24.4\u2022THEDIALOGUE-STATEARCHITECTURE21User: I\u2019m looking for a cheaper restaurantinform(price=cheap)System: Sure. What kind - and where?User: Thai food, somewhere downtowninform(price=cheap, food=Thai, area=centre)System: The House serves cheap Thai foodUser: Where is it?inform(price=cheap, food=Thai, area=centre); request(address)System: The House is at 106 Regent StreetSince dialogue acts place some constraints on the slots and values, the tasks ofdialogue-act detection and slot-\ufb01lling are often performed jointly. Consider the taskof determining thatI\u2019d like Cantonese food near the Mission Districthas the structureinform(food=cantonese,area=mission).Dialogue act interpretation\u2014in this example choosinginformfrom the set ofdialogue acts for this task\u2014is done by supervised classi\ufb01cation trained on hand-labeled dialog acts, predicting the dialogue act tag based on embeddings represent-ing the current input sentence and the prior dialogue acts.The simplest dialogue state tracker might just take the output of a slot-\ufb01llingsequence-model (Section24.4.2) after each sentence. Alternatively, a more complexmodel can make use of the reading-comprehension architectures from Chapter 23.For example the model ofGao et al. (2019)trains a classi\ufb01er for each slot to decidewhether its value is being changed in the current sentence or should be carried overfrom the previous sentences. If the slot value is being changed, a span-predictionmodel is used to predict the start and end of the span with the slot \ufb01ller.A special case: detecting correction actsSome dialogue acts are important because of their implications for dialogue control.If a dialogue system misrecognizes or misunderstands an utterance, the user willgenerally correct the error by repeating or reformulating the utterance. Detectingtheseuser correction actsis therefore quite important. Ironically, it turns out thatuser correctionactscorrections are actuallyharderto recognize than normal sentences! In fact, correc-tions in one early dialogue system (the TOOT system) had double the ASR worderror rate of non-corrections(Swerts et al., 2000)! One reason for this is that speak-ers sometimes use a speci\ufb01c prosodic style for corrections calledhyperarticulation,hyperarticula-tionin which the utterance contains exaggerated energy, duration, or F0 contours, suchasI said BAL-TI-MORE, not Boston(Wade et al. 1992,Levow 1998,Hirschberget al. 2001). Even when they are not hyperarticulating, users who are frustratedseem to speak in a way that is harder for speech recognizers(Goldberg et al., 2003).What are the characteristics of these corrections? User corrections tend to beeither exact repetitions or repetitions with one or more words omitted, although theymay also be paraphrases of the original utterance.(Swerts et al., 2000). Detect-ing these reformulations or correction acts can be part of the general dialogue actdetection classi\ufb01er. Alternatively, because the cues to these acts tend to appear indifferent ways than for simple acts (likeINFORMorrequest), we can make use offeatures orthogonal to simple contextual embedding features; some typical featuresare shown below (Levow 1998,Litman et al. 1999,Hirschberg et al. 2001,Bulykoet al. 2005,Awadallah et al. 2015):Example from Mrk\u0161i\u0107, N., O S\u00e9aghdha, D., Wen, T.-H., Thomson, B., and Young, S. (2017). Neural belief tracker: Data-driven dialogue state tracking. ACL.  Dialogue state trackingI'd like Cantonese food near the Mission district.\u00e0inform(food=cantonese, area=mission). Dialogue act interpretation algorithm: \u20221-of-N supervised classification to choose inform\u2022Based on encodings of current sentence + prior dialogue actsSimple dialogue state tracker:\u2022Run a slot-filler after each sentence  An special case of dialogue act detection: Detecting Correction ActsIf system misrecognizes an utteranceUser might make a correction\u25e6Repeat themselves\u25e6Rephrasing\u25e6Saying \u201cno\u201d to a confirmation question Corrections are harder to recognize!\u25e6From speech, corrections are misrecognized twice as often (in terms of word error rate) as non-corrections! (Swertset al 2000)\u25e6Hyperarticulation(exaggerated prosody) is a large factor: \u25e6Shriberg, E., Wade, E., Price, P ., 1992. Human-machine problem solving using spoken language systems (SLS): Factors affect-ng performance and user satisfaction. DARPA Speech and Natural Language Workshop.\u25e6\"I said BAL-TI-MORE, not Boston\" Features for detecting corrections in spoken dialogue22CHAPTER24\u2022CHATBOTS&DIALOGUESYSTEMSfeaturesexampleslexicalwords like \u201cno\u201d, \u201ccorrection\u201d, \u201cI don\u2019t\u201d, swear words, utterance lengthsemanticsimilarity (word overlap or embedding dot product) between the candidatecorrection act and the user\u2019s prior utterancephoneticphonetic overlap between the candidate correction act and the user\u2019s prior ut-terance (i.e. \u201cWhatsApp\u201d may be incorrectly recognized as \u201cWhat\u2019s up\u201d)prosodichyperarticulation, increases in F0 range, pause duration, and word duration,generally normalized by the values for previous sentencesASRASR con\ufb01dence, language model probability24.4.4 Dialogue PolicyThe goal of thedialogue policyis to decide what action the system should take next,dialogue policythat is, what dialogue act to generate.More formally, at turniin the conversation we want to predict which actionAito take, based on the entire dialogue state. The state could mean the entire sequenceof dialogue acts from the system (A) and from the user (U), in which case the taskwould be to compute:\u02c6Ai=argmaxAi2AP(Ai|(A1,U1,. . . ,Ai\u00001,Ui\u00001)(24.7)We can simplify this by maintaining as the dialogue state mainly just the set ofslot-\ufb01llers that the user has expressed, collapsing across the many different conver-sational paths that could lead to the same set of \ufb01lled slots.Such a policy might then just condition on the current dialogue state as repre-sented just by the current state of the frame Framei(which slots are \ufb01lled and withwhat) and the last turn by the system and user:\u02c6Ai=argmaxAi2AP(Ai|Framei\u00001,Ai\u00001,Ui\u00001)(24.8)These probabilities can be estimated by a neural classi\ufb01er using neural representa-tions of the slot \ufb01llers (for example as spans) and the utterances (for example assentence embeddings computed over contextual embeddings)More sophisticated models train the policy viareinforcement learning. To de-cide which action to take, a reinforcement learning system gets a reward at the endof the dialogue, and uses that reward to train a policy to take actions. For example inthe movie-recommendation dialogue system ofFazel-Zarandi et al. (2017), the ac-tion space has only three actions:EXECUTE,CONFIRM, andELICIT. TheEXECUTEsends a query to the database and answers the user\u2019s question,CONFIRMclari\ufb01esthe intent or slot with the users (e.g., \u201cDo you want movies directed by ChristopherNolan?\u201d) whileELICITasks the user for missing information (e.g., \u201cWhich movieare you talking about?\u201d). The system gets a large positive reward if the dialogue sys-tem terminates with the correct slot representation at the end, a large negative rewardif the slots are wrong, and a small negative reward for con\ufb01rmation and elicitationquestions to keep the system from re-con\ufb01rming everything.Policy Example: Con\ufb01rmation and RejectionModern dialogue systems often make mistakes. It is therefore important for dialoguesystems to make sure that they have achieved the correct interpretation of the user\u2019s Chatbots and Dialogue SystemsThe Dialogue-State Architecture Chatbots and Dialogue SystemsThe Dialogue-State Architecture Continued: Policy and Generation Dialogue PolicyAt turn ipredict action Aito take, given entire history:Simplify by just conditioning on the current dialogue state (filled frame slots) and the last turn and turn by system and user:22CHAPTER24\u2022CHATBOTS&DIALOGUESYSTEMSfeaturesexampleslexicalwords like \u201cno\u201d, \u201ccorrection\u201d, \u201cI don\u2019t\u201d, swear words, utterance lengthsemanticsimilarity (word overlap or embedding dot product) between the candidatecorrection act and the user\u2019s prior utterancephoneticphonetic overlap between the candidate correction act and the user\u2019s prior ut-terance (i.e. \u201cWhatsApp\u201d may be incorrectly recognized as \u201cWhat\u2019s up\u201d)prosodichyperarticulation, increases in F0 range, pause duration, and word duration,generally normalized by the values for previous sentencesASRASR con\ufb01dence, language model probability24.4.4 Dialogue PolicyThe goal of thedialogue policyis to decide what action the system should take next,dialogue policythat is, what dialogue act to generate.More formally, at turniin the conversation we want to predict which actionAito take, based on the entire dialogue state. The state could mean the entire sequenceof dialogue acts from the system (A) and from the user (U), in which case the taskwould be to compute:\u02c6Ai=argmaxAi2AP(Ai|(A1,U1,. . . ,Ai\u00001,Ui\u00001)(24.7)We can simplify this by maintaining as the dialogue state mainly just the set ofslot-\ufb01llers that the user has expressed, collapsing across the many different conver-sational paths that could lead to the same set of \ufb01lled slots.Such a policy might then just condition on the current dialogue state as repre-sented just by the current state of the frame Framei(which slots are \ufb01lled and withwhat) and the last turn by the system and user:\u02c6Ai=argmaxAi2AP(Ai|Framei\u00001,Ai\u00001,Ui\u00001)(24.8)These probabilities can be estimated by a neural classi\ufb01er using neural representa-tions of the slot \ufb01llers (for example as spans) and the utterances (for example assentence embeddings computed over contextual embeddings)More sophisticated models train the policy viareinforcement learning. To de-cide which action to take, a reinforcement learning system gets a reward at the endof the dialogue, and uses that reward to train a policy to take actions. For example inthe movie-recommendation dialogue system ofFazel-Zarandi et al. (2017), the ac-tion space has only three actions:EXECUTE,CONFIRM, andELICIT. TheEXECUTEsends a query to the database and answers the user\u2019s question,CONFIRMclari\ufb01esthe intent or slot with the users (e.g., \u201cDo you want movies directed by ChristopherNolan?\u201d) whileELICITasks the user for missing information (e.g., \u201cWhich movieare you talking about?\u201d). The system gets a large positive reward if the dialogue sys-tem terminates with the correct slot representation at the end, a large negative rewardif the slots are wrong, and a small negative reward for con\ufb01rmation and elicitationquestions to keep the system from re-con\ufb01rming everything.Policy Example: Con\ufb01rmation and RejectionModern dialogue systems often make mistakes. It is therefore important for dialoguesystems to make sure that they have achieved the correct interpretation of the user\u2019s22CHAPTER24\u2022CHATBOTS&DIALOGUESYSTEMSfeaturesexampleslexicalwords like \u201cno\u201d, \u201ccorrection\u201d, \u201cI don\u2019t\u201d, swear words, utterance lengthsemanticsimilarity (word overlap or embedding dot product) between the candidatecorrection act and the user\u2019s prior utterancephoneticphonetic overlap between the candidate correction act and the user\u2019s prior ut-terance (i.e. \u201cWhatsApp\u201d may be incorrectly recognized as \u201cWhat\u2019s up\u201d)prosodichyperarticulation, increases in F0 range, pause duration, and word duration,generally normalized by the values for previous sentencesASRASR con\ufb01dence, language model probability24.4.4 Dialogue PolicyThe goal of thedialogue policyis to decide what action the system should take next,dialogue policythat is, what dialogue act to generate.More formally, at turniin the conversation we want to predict which actionAito take, based on the entire dialogue state. The state could mean the entire sequenceof dialogue acts from the system (A) and from the user (U), in which case the taskwould be to compute:\u02c6Ai=argmaxAi2AP(Ai|(A1,U1,. . . ,Ai\u00001,Ui\u00001)(24.7)We can simplify this by maintaining as the dialogue state mainly just the set ofslot-\ufb01llers that the user has expressed, collapsing across the many different conver-sational paths that could lead to the same set of \ufb01lled slots.Such a policy might then just condition on the current dialogue state as repre-sented just by the current state of the frame Framei(which slots are \ufb01lled and withwhat) and the last turn by the system and user:\u02c6Ai=argmaxAi2AP(Ai|Framei\u00001,Ai\u00001,Ui\u00001)(24.8)These probabilities can be estimated by a neural classi\ufb01er using neural representa-tions of the slot \ufb01llers (for example as spans) and the utterances (for example assentence embeddings computed over contextual embeddings)More sophisticated models train the policy viareinforcement learning. To de-cide which action to take, a reinforcement learning system gets a reward at the endof the dialogue, and uses that reward to train a policy to take actions. For example inthe movie-recommendation dialogue system ofFazel-Zarandi et al. (2017), the ac-tion space has only three actions:EXECUTE,CONFIRM, andELICIT. TheEXECUTEsends a query to the database and answers the user\u2019s question,CONFIRMclari\ufb01esthe intent or slot with the users (e.g., \u201cDo you want movies directed by ChristopherNolan?\u201d) whileELICITasks the user for missing information (e.g., \u201cWhich movieare you talking about?\u201d). The system gets a large positive reward if the dialogue sys-tem terminates with the correct slot representation at the end, a large negative rewardif the slots are wrong, and a small negative reward for con\ufb01rmation and elicitationquestions to keep the system from re-con\ufb01rming everything.Policy Example: Con\ufb01rmation and RejectionModern dialogue systems often make mistakes. It is therefore important for dialoguesystems to make sure that they have achieved the correct interpretation of the user\u2019s Policy example: Confirmation and RejectionDialogue systems make errorsSo they to make sure they have understood userTwo important mechanisms:\u2022confirmingunderstandings with the user \u2022rejectingutterances that the system is likely to have misunderstood.  Explicit confirmation strategy24.4\u2022THEDIALOGUE-STATEARCHITECTURE23input. This is generally done by two methods:con\ufb01rmingunderstandings with theuser andrejectingutterances that the system is likely to have misunderstood.As we saw in the prior section, most systems introduce particular strategies andactions related to con\ufb01rmation and rejection. When using theexplicit con\ufb01rmationexplicitcon\ufb01rmationstrategy, a system asks the user a direct question to con\ufb01rm the system\u2019s under-standing, like the two examples below in which the system asks a (boldface) yes-nocon\ufb01rmation question:S:Which city do you want to leave from?U:Baltimore.S:Do you want to leave from Baltimore?U:Yes.U:I\u2019d like to \ufb02y from Denver Colorado to New York City on Septembertwenty \ufb01rst in the morning on United AirlinesS:Let\u2019s see then. I have you going from Denver Colorado to New Yorkon September twenty \ufb01rst. Is that correct?U:YesWhen using theimplicit con\ufb01rmationstrategy, a system instead can demon-implicitcon\ufb01rmationstrate its understanding as agroundingstrategy, for example repeating back thesystem\u2019s understanding as part of asking the next question, as in the two examplesbelow:U:I want to travel to BerlinS:When do you want to travel to Berlin?U2:Hi I\u2019d like to \ufb02y to Seattle Tuesday MorningA3:Traveling to Seattle on Tuesday, August eleventh in the morning.Your full name?Explicit and implicit con\ufb01rmation have complementary strengths. Explicit con-\ufb01rmation makes it easier for users to correct the system\u2019s misrecognitions since auser can just answer \u201cno\u201d to the con\ufb01rmation question. But explicit con\ufb01rmation isawkward and increases the length of the conversation (Danieli and Gerbino 1995,Walker et al. 1998). The explicit con\ufb01rmation dialogue fragments above sound non-natural and de\ufb01nitely non-human; implicit con\ufb01rmation is much more conversation-ally natural.Con\ufb01rmation is just one kind of conversational action by which a system canexpress lack of understanding. Another option isrejection, in which a system givesrejectionthe user a prompt likeI\u2019m sorry, I didn\u2019t understand that.Sometimes utterances are rejected multiple times. This might mean that the useris using language that the system is unable to follow. Thus, when an utterance isrejected, systems often follow a strategy ofprogressive promptingorescalatingprogressivepromptingdetail(Yankelovich et al. 1995,Weinschenk and Barker 2000), as in this examplefromCohen et al. (2004):System:When would you like to leave?Caller:Well, um, I need to be in New York in time for the \ufb01rst World Series game.System:<reject>. Sorry, I didn\u2019t get that. Please say the month and day you\u2019d liketo leave.Caller:I wanna go on October \ufb01fteenth. Implicit confirmation strategy24.4\u2022THEDIALOGUE-STATEARCHITECTURE23input. This is generally done by two methods:con\ufb01rmingunderstandings with theuser andrejectingutterances that the system is likely to have misunderstood.As we saw in the prior section, most systems introduce particular strategies andactions related to con\ufb01rmation and rejection. When using theexplicit con\ufb01rmationexplicitcon\ufb01rmationstrategy, a system asks the user a direct question to con\ufb01rm the system\u2019s under-standing, like the two examples below in which the system asks a (boldface) yes-nocon\ufb01rmation question:S:Which city do you want to leave from?U:Baltimore.S:Do you want to leave from Baltimore?U:Yes.U:I\u2019d like to \ufb02y from Denver Colorado to New York City on Septembertwenty \ufb01rst in the morning on United AirlinesS:Let\u2019s see then. I have you going from Denver Colorado to New Yorkon September twenty \ufb01rst. Is that correct?U:YesWhen using theimplicit con\ufb01rmationstrategy, a system instead can demon-implicitcon\ufb01rmationstrate its understanding as agroundingstrategy, for example repeating back thesystem\u2019s understanding as part of asking the next question, as in the two examplesbelow:U:I want to travel to BerlinS:When do you want to travel to Berlin?U2:Hi I\u2019d like to \ufb02y to Seattle Tuesday MorningA3:Traveling to Seattle on Tuesday, August eleventh in the morning.Your full name?Explicit and implicit con\ufb01rmation have complementary strengths. Explicit con-\ufb01rmation makes it easier for users to correct the system\u2019s misrecognitions since auser can just answer \u201cno\u201d to the con\ufb01rmation question. But explicit con\ufb01rmation isawkward and increases the length of the conversation (Danieli and Gerbino 1995,Walker et al. 1998). The explicit con\ufb01rmation dialogue fragments above sound non-natural and de\ufb01nitely non-human; implicit con\ufb01rmation is much more conversation-ally natural.Con\ufb01rmation is just one kind of conversational action by which a system canexpress lack of understanding. Another option isrejection, in which a system givesrejectionthe user a prompt likeI\u2019m sorry, I didn\u2019t understand that.Sometimes utterances are rejected multiple times. This might mean that the useris using language that the system is unable to follow. Thus, when an utterance isrejected, systems often follow a strategy ofprogressive promptingorescalatingprogressivepromptingdetail(Yankelovich et al. 1995,Weinschenk and Barker 2000), as in this examplefromCohen et al. (2004):System:When would you like to leave?Caller:Well, um, I need to be in New York in time for the \ufb01rst World Series game.System:<reject>. Sorry, I didn\u2019t get that. Please say the month and day you\u2019d liketo leave.Caller:I wanna go on October \ufb01fteenth. Confirmation strategy  tradeoffsExplicit confirmation makes it easier for users to correct the system\u2019s misrecognitions since a user can just answer \u201cno\u201d to the confirmation question. But explicit confirmation is also awkward and increases the length of the conversation (Danieliand Gerbino1995, Walker et al. 1998).  RejectionI\u2019m sorry, I didn\u2019t understand that.  Progressive prompting for rejection24.4\u2022THEDIALOGUE-STATEARCHITECTURE23input. This is generally done by two methods:con\ufb01rmingunderstandings with theuser andrejectingutterances that the system is likely to have misunderstood.As we saw in the prior section, most systems introduce particular strategies andactions related to con\ufb01rmation and rejection. When using theexplicit con\ufb01rmationexplicitcon\ufb01rmationstrategy, a system asks the user a direct question to con\ufb01rm the system\u2019s under-standing, like the two examples below in which the system asks a (boldface) yes-nocon\ufb01rmation question:S:Which city do you want to leave from?U:Baltimore.S:Do you want to leave from Baltimore?U:Yes.U:I\u2019d like to \ufb02y from Denver Colorado to New York City on Septembertwenty \ufb01rst in the morning on United AirlinesS:Let\u2019s see then. I have you going from Denver Colorado to New Yorkon September twenty \ufb01rst. Is that correct?U:YesWhen using theimplicit con\ufb01rmationstrategy, a system instead can demon-implicitcon\ufb01rmationstrate its understanding as agroundingstrategy, for example repeating back thesystem\u2019s understanding as part of asking the next question, as in the two examplesbelow:U:I want to travel to BerlinS:When do you want to travel to Berlin?U2:Hi I\u2019d like to \ufb02y to Seattle Tuesday MorningA3:Traveling to Seattle on Tuesday, August eleventh in the morning.Your full name?Explicit and implicit con\ufb01rmation have complementary strengths. Explicit con-\ufb01rmation makes it easier for users to correct the system\u2019s misrecognitions since auser can just answer \u201cno\u201d to the con\ufb01rmation question. But explicit con\ufb01rmation isawkward and increases the length of the conversation (Danieli and Gerbino 1995,Walker et al. 1998). The explicit con\ufb01rmation dialogue fragments above sound non-natural and de\ufb01nitely non-human; implicit con\ufb01rmation is much more conversation-ally natural.Con\ufb01rmation is just one kind of conversational action by which a system canexpress lack of understanding. Another option isrejection, in which a system givesrejectionthe user a prompt likeI\u2019m sorry, I didn\u2019t understand that.Sometimes utterances are rejected multiple times. This might mean that the useris using language that the system is unable to follow. Thus, when an utterance isrejected, systems often follow a strategy ofprogressive promptingorescalatingprogressivepromptingdetail(Yankelovich et al. 1995,Weinschenk and Barker 2000), as in this examplefromCohen et al. (2004):System:When would you like to leave?Caller:Well, um, I need to be in New York in time for the \ufb01rst World Series game.System:<reject>. Sorry, I didn\u2019t get that. Please say the month and day you\u2019d liketo leave.Caller:I wanna go on October \ufb01fteenth.Don't just repeat the question \"When would you like to leave?\"Give user guidance about what they can say: Using confidence to decide whether to confirm:ASR  or NLU systems can assign a confidencevalue, indicating how likely they are that they understood the user. \u2022Acoustic log-likelihood of the utterance\u2022Prosodic features\u2022Ratio of score  of best to second-best interpretationSystems could use set confidence thresholds:24CHAPTER24\u2022CHATBOTS&DIALOGUESYSTEMSIn this example, instead of just repeating \u201cWhen would you like to leave?\u201d, therejection prompt gives the caller more guidance about how to formulate an utter-ance the system will understand. Theseyou-can-sayhelp messages are important inhelping improve systems\u2019 understanding performance(Bohus and Rudnicky, 2005).If the caller\u2019s utterance gets rejected yet again, the prompt can re\ufb02ect this (\u201cIstilldidn\u2019t get that\u201d), and give the caller even more guidance.An alternative strategy for error handling israpid reprompting, in which therapidrepromptingsystem rejects an utterance just by saying \u201cI\u2019m sorry?\u201d or \u201cWhat was that?\u201d Onlyif the caller\u2019s utterance is rejected a second time does the system start applyingprogressive prompting.Cohen et al. (2004)summarize experiments showing thatusers greatly prefer rapid reprompting as a \ufb01rst-level error prompt.It is common to use rich features other than just the dialogue state representa-tion to make policy decisions. For example, thecon\ufb01dencethat the ASR systemassigns to an utterance can be used by explicitly con\ufb01rming low-con\ufb01dence sen-tences. Condience is a metric that the speech recognizer can assign to its transcrip-tion of a sentence to indicate how con\ufb01dent it is in that transcription. Con\ufb01dence isoften computed from the acoustic log-likelihood of the utterance (greater probabil-ity means higher con\ufb01dence), but prosodic features can also be used in con\ufb01denceprediction. For example, utterances with large F0 excursions or longer durations,or those preceded by longer pauses, are likely to be misrecognized(Litman et al.,2000).Another common feature in con\ufb01rmation is thecostof making an error. For ex-ample, explicit con\ufb01rmation is common before a \ufb02ight is actually booked or moneyin an account is moved. Systems might have a four-tiered level of con\ufb01dence withthree thresholdsa,b, andg:<alow con\ufb01dence reject\u0000aabove the threshold con\ufb01rm explicitly\u0000bhigh con\ufb01dence con\ufb01rm implictly\u0000gvery high con\ufb01dence don\u2019t con\ufb01rm at all24.4.5 Natural language generation in the dialogue-state modelFinally, once the policy has decided what speech act to generate, the natural languagegeneration component needs to generate the text of a response to the user.Once a dialogue act has been decided, we need to generate the text of the re-sponse to the user. The task of natural language generation (NLG) in the information-state architecture is often modeled in two stages,content planning(what to say),contentplanningandsentence realization(how to say it).sentencerealizationHere we\u2019ll assume content planning has been done by the dialogue policy, whichhas chosen the dialogue act to generate, and chosen some attributes (slots and values)that the planner wants to say to the user (either to give the user the answer, or as partof a con\ufb01rmation strategy).Fig.24.16shows some sample input/outputs for the sentence realization phase.In the \ufb01rst example, the content planner has chosen the dialogue actRECOMMENDand some particular slots (name, neighborhood, cuisine) and their \ufb01llers. The goalof the sentence realizer is to generate a sentence like lines 1 or 2 shown in the \ufb01gure,by training on many such examples of representation/sentence pairs from a largecorpus of labeled dialogues.Training data is hard to come by; we are unlikely to see every possible restaurantwith every possible attribute in many possible differently worded sentences. There- Natural Language GenerationNLG  in information-state architecture modeled in two stages:\u2022content planning (what to say)\u2022sentence realization (how to say it). We'll focus on sentence realization here. Sentence RealizationAssume content planning has been done by the dialogue policy\u2022Chosen the dialogue act to generate \u2022Chosen some attributes (slots and values) that the planner wants to say to the user \u2022Either to give the user the answer, or as part of a confirmation strategy)  2 samples of Input and Output for Sentence Realizer24.4\u2022THEDIALOGUE-STATEARCHITECTURE25recommend(restaurant name= Au Midi, neighborhood = midtown,cuisine = french1 Au Midi is in Midtown and serves French food.2 There is a French restaurant in Midtown called Au Midi.recommend(restaurant name= Loch Fyne, neighborhood = citycentre, cuisine = seafood)3 Loch Fyne is in the City Center and serves seafood food.4 There is a seafood restaurant in the City Centre called Loch Fyne.Figure 24.16Two examples of inputs to the sentence realization phase of NLG, showingthe dialogue act and attributes prespeci\ufb01ed by the content planner. Line 1-2 and 3-4 show dif-ferent possible output sentences to be generated by the sentence realizer. From the restaurantrecommendation system ofNayak et al. (2017).fore it is common in sentence realization to increase the generality of the trainingexamples bydelexicalization. Delexicalization is the process of replacing speci\ufb01cdelexicalizationwords in the training set that represent slot values with a generic placeholder to-ken representing the slot. Fig.24.17shows the result of delexicalizing the trainingsentences in Fig.24.16.recommend(restaurant name= Au Midi, neighborhood = midtown,cuisine = french1restaurantnameis inneighborhoodand servescuisinefood.2 There is acuisinerestaurant inneighborhoodcalledrestaurantname.Figure 24.17Delexicalized sentences that can be used for generating many different relex-icalized sentences. From the restaurant recommendation system ofNayak et al. (2017).Mapping from frames to delexicalized sentences is generally done by encoderdecoder models (Wen et al. 2015a,Wen et al. 2015b,Mrk\u02c7si\u00b4c et al. 2017, interalia), trained on large hand-labeled corpora of task-oriented dialogue(Budzianowskiet al., 2018). The input to the encoder is a sequence of tokensxtthat representthe dialogue act and its arguments. Thus the dialogue actRECOMMENDand theattribute/value pairs service:decent, cuisine:null might be represented as a \ufb02at se-quence of tokens(Nayak et al., 2017), each mapped to a learned embeddingwt, asshown in Fig.24.18.\ndecentservice:RECOMMENDcuisine:null[name]hasdecentserviceENCODERDECODERFigure 24.18An encoder decoder sentence realizer mapping slots/\ufb01llers to English.The encoder reads all the input slot/value representations, and the decoder out-puts the following delexicalized English sentence:restaurantname has decent serviceWe can then use the input frame from the content planner torelexicalize(\ufb01ll in therelexicalizeexact restaurant or neighborhood or cuisine) resulting in:Au Midi has decent service 24.4\u2022THEDIALOGUE-STATEARCHITECTURE25recommend(restaurant name= Au Midi, neighborhood = midtown,cuisine = french1 Au Midi is in Midtown and serves French food.2 There is a French restaurant in Midtown called Au Midi.recommend(restaurant name= Loch Fyne, neighborhood = citycentre, cuisine = seafood)3 Loch Fyne is in the City Center and serves seafood food.4 There is a seafood restaurant in the City Centre called Loch Fyne.Figure 24.16Two examples of inputs to the sentence realization phase of NLG, showingthe dialogue act and attributes prespeci\ufb01ed by the content planner. Line 1-2 and 3-4 show dif-ferent possible output sentences to be generated by the sentence realizer. From the restaurantrecommendation system ofNayak et al. (2017).fore it is common in sentence realization to increase the generality of the trainingexamples bydelexicalization. Delexicalization is the process of replacing speci\ufb01cdelexicalizationwords in the training set that represent slot values with a generic placeholder to-ken representing the slot. Fig.24.17shows the result of delexicalizing the trainingsentences in Fig.24.16.recommend(restaurant name= Au Midi, neighborhood = midtown,cuisine = french1restaurantnameis inneighborhoodand servescuisinefood.2 There is acuisinerestaurant inneighborhoodcalledrestaurantname.Figure 24.17Delexicalized sentences that can be used for generating many different relex-icalized sentences. From the restaurant recommendation system ofNayak et al. (2017).Mapping from frames to delexicalized sentences is generally done by encoderdecoder models (Wen et al. 2015a,Wen et al. 2015b,Mrk\u02c7si\u00b4c et al. 2017, interalia), trained on large hand-labeled corpora of task-oriented dialogue(Budzianowskiet al., 2018). The input to the encoder is a sequence of tokensxtthat representthe dialogue act and its arguments. Thus the dialogue actRECOMMENDand theattribute/value pairs service:decent, cuisine:null might be represented as a \ufb02at se-quence of tokens(Nayak et al., 2017), each mapped to a learned embeddingwt, asshown in Fig.24.18.\ndecentservice:RECOMMENDcuisine:null[name]hasdecentserviceENCODERDECODERFigure 24.18An encoder decoder sentence realizer mapping slots/\ufb01llers to English.The encoder reads all the input slot/value representations, and the decoder out-puts the following delexicalized English sentence:restaurantname has decent serviceWe can then use the input frame from the content planner torelexicalize(\ufb01ll in therelexicalizeexact restaurant or neighborhood or cuisine) resulting in:Au Midi has decent serviceSentence RealizationTraining data is hard to come by \u2022Don't see each restaurant in each situationCommon way to improve generalization:\u2022Delexicalization: replacing words in the training set that represent slot values with a generic placeholder token: Sentence RealizationTraining data is hard to come by \u2022Don't see each restaurant in each situationCommon way to improve generalization:\u2022Delexicalization: replacing words in the training set that represent slot values with a generic placeholder token:24.4\u2022THEDIALOGUE-STATEARCHITECTURE25recommend(restaurant name= Au Midi, neighborhood = midtown,cuisine = french1 Au Midi is in Midtown and serves French food.2 There is a French restaurant in Midtown called Au Midi.recommend(restaurant name= Loch Fyne, neighborhood = citycentre, cuisine = seafood)3 Loch Fyne is in the City Center and serves seafood food.4 There is a seafood restaurant in the City Centre called Loch Fyne.Figure 24.16Two examples of inputs to the sentence realization phase of NLG, showingthe dialogue act and attributes prespeci\ufb01ed by the content planner. Line 1-2 and 3-4 show dif-ferent possible output sentences to be generated by the sentence realizer. From the restaurantrecommendation system ofNayak et al. (2017).fore it is common in sentence realization to increase the generality of the trainingexamples bydelexicalization. Delexicalization is the process of replacing speci\ufb01cdelexicalizationwords in the training set that represent slot values with a generic placeholder to-ken representing the slot. Fig.24.17shows the result of delexicalizing the trainingsentences in Fig.24.16.recommend(restaurant name= Au Midi, neighborhood = midtown,cuisine = french1restaurantnameis inneighborhoodand servescuisinefood.2 There is acuisinerestaurant inneighborhoodcalledrestaurantname.Figure 24.17Delexicalized sentences that can be used for generating many different relex-icalized sentences. From the restaurant recommendation system ofNayak et al. (2017).Mapping from frames to delexicalized sentences is generally done by encoderdecoder models (Wen et al. 2015a,Wen et al. 2015b,Mrk\u02c7si\u00b4c et al. 2017, interalia), trained on large hand-labeled corpora of task-oriented dialogue(Budzianowskiet al., 2018). The input to the encoder is a sequence of tokensxtthat representthe dialogue act and its arguments. Thus the dialogue actRECOMMENDand theattribute/value pairs service:decent, cuisine:null might be represented as a \ufb02at se-quence of tokens(Nayak et al., 2017), each mapped to a learned embeddingwt, asshown in Fig.24.18.\ndecentservice:RECOMMENDcuisine:null[name]hasdecentserviceENCODERDECODERFigure 24.18An encoder decoder sentence realizer mapping slots/\ufb01llers to English.The encoder reads all the input slot/value representations, and the decoder out-puts the following delexicalized English sentence:restaurantname has decent serviceWe can then use the input frame from the content planner torelexicalize(\ufb01ll in therelexicalizeexact restaurant or neighborhood or cuisine) resulting in:Au Midi has decent service 24.4\u2022THEDIALOGUE-STATEARCHITECTURE25recommend(restaurant name= Au Midi, neighborhood = midtown,cuisine = french1 Au Midi is in Midtown and serves French food.2 There is a French restaurant in Midtown called Au Midi.recommend(restaurant name= Loch Fyne, neighborhood = citycentre, cuisine = seafood)3 Loch Fyne is in the City Center and serves seafood food.4 There is a seafood restaurant in the City Centre called Loch Fyne.Figure 24.16Two examples of inputs to the sentence realization phase of NLG, showingthe dialogue act and attributes prespeci\ufb01ed by the content planner. Line 1-2 and 3-4 show dif-ferent possible output sentences to be generated by the sentence realizer. From the restaurantrecommendation system ofNayak et al. (2017).fore it is common in sentence realization to increase the generality of the trainingexamples bydelexicalization. Delexicalization is the process of replacing speci\ufb01cdelexicalizationwords in the training set that represent slot values with a generic placeholder to-ken representing the slot. Fig.24.17shows the result of delexicalizing the trainingsentences in Fig.24.16.recommend(restaurant name= Au Midi, neighborhood = midtown,cuisine = french1restaurantnameis inneighborhoodand servescuisinefood.2 There is acuisinerestaurant inneighborhoodcalledrestaurantname.Figure 24.17Delexicalized sentences that can be used for generating many different relex-icalized sentences. From the restaurant recommendation system ofNayak et al. (2017).Mapping from frames to delexicalized sentences is generally done by encoderdecoder models (Wen et al. 2015a,Wen et al. 2015b,Mrk\u02c7si\u00b4c et al. 2017, interalia), trained on large hand-labeled corpora of task-oriented dialogue(Budzianowskiet al., 2018). The input to the encoder is a sequence of tokensxtthat representthe dialogue act and its arguments. Thus the dialogue actRECOMMENDand theattribute/value pairs service:decent, cuisine:null might be represented as a \ufb02at se-quence of tokens(Nayak et al., 2017), each mapped to a learned embeddingwt, asshown in Fig.24.18.\ndecentservice:RECOMMENDcuisine:null[name]hasdecentserviceENCODERDECODERFigure 24.18An encoder decoder sentence realizer mapping slots/\ufb01llers to English.The encoder reads all the input slot/value representations, and the decoder out-puts the following delexicalized English sentence:restaurantname has decent serviceWe can then use the input frame from the content planner torelexicalize(\ufb01ll in therelexicalizeexact restaurant or neighborhood or cuisine) resulting in:Au Midi has decent serviceSentence Realization: mapping from frames to delexicalized sentencesEncoder-decoder models:Output:restaurant_namehas decent serviceRelexicalize to: Au Midi has decent service  Generating clarification questionsUser: What do you have going to UNKNOWN WORD on the 5th? System: Going where on the 5th? The system repeats \u201cgoing\u201d and \u201con the 5th\u201d to make it clear which aspect of the user\u2019s turn the system needs to be clarifiedMethods for generating clarification questions:\u2022Rules like 'replace \u201cgoing to UNKNOWN WORD\u201d with \u201cgoing where\u201d'\u2022Classifiers that  guess which slots were misrecognized  Chatbots and Dialogue SystemsThe Dialogue-State Architecture Continued: Policy and Generation Chatbots and Dialogue SystemsEvaluating Dialogue Systems Evaluating chatbots and task-based dialogueTask-based dialogue:  \u2022mainly by measuring task performanceChatbots: \u2022mainly by human evaluation Chatbots are evaluated by humansParticipant evaluation: The human who talked to the chatbot assigns a scoreObserver evaluation: third party who reads a transcript of a human/chatbot conversation assigns a score. Participant evaluationHuman chats with model for 6 turns and rates 8 dimensions of quality:\u2022avoiding repetition, interestingness, making sense, fluency, listening, inquisitiveness, humanness, engagingness, (1) Avoiding Repetition: How repetitive was this user? \u25e6\u2022Repeated themselves over and over \u2022Sometimes said the same thing twice \u2022 Always said something new (3) Making sense: How often did this user say something which didn't make sense? \u25e6\u2022Never made any sense \u2022Most responses didn\u2019t make sense \u2022Some responses didn\u2019t make sense \u2022Everything made perfect sense (8) Engagingness:How much did you enjoy talking to this user?\u25e6\u2022Not at all \u2022A little \u2022Somewhat \u2022A lotAbigail See, Stephen Roller, DouweKiela, Jason Weston. 2019.What makes a good conversation? How controllable attributes affect human judgments. NAACL. Observer evaluation: acute-evalAnnotators look at two conversations (A + B) and decide which is better:Engagingness: Who would you prefer to talk to for a long conversation? Interestingness: If you had to say one of these speakers is interesting and one is boring, who would you say is more interesting? Humanness: Which speaker sounds more human? Knowledgeable: If you had to say that one speaker is more knowledgeable and one is more ignorant, who is more knowledgeable? Li, M., Weston, J., and Roller, S. (2019). Acute-eval: Improved dialogue evaluation with optimized questions and multi-turn comparisons. NeurIPS19 Workshop on Conversational AI.  ACUTE-EV AL: Improved dialogue evaluation with optimized questions andmulti-turn comparisonsMargaret LiFacebook AI Researchmargaretli@fb.comJason WestonFacebook AI Researchjase@fb.comStephen RollerFacebook AI Researchroller@fb.comAbstractWhile dialogue remains an important end-goal of natural lan-guage research, the dif\ufb01culty of evaluation is an oft-quotedreason why it remains troublesome to make real progress to-wards its solution. Evaluation dif\ufb01culties are actually two-fold:not only do automatic metrics not correlate well with humanjudgments, but also human judgments themselves are in factdif\ufb01cult to measure. The two most used human judgment tests,single-turn pairwise evaluation and multi-turn Likert scores,both have serious \ufb02aws as we discuss in this work.We instead provide a novel procedure involving comparingtwo full dialogues, where a human judge is asked to pay at-tention to only one speaker within each, and make a pairwisejudgment. The questions themselves are optimized to maxi-mize the robustness of judgments across different annotators,resulting in better tests. We also show how these tests work inself-play model chat setups, resulting in faster, cheaper tests.We hope these tests become the de facto standard, and willrelease open-source code to that end.IntroductionDialogue between human and machine is an important end-goal of natural language research. The open-ended nature ofgenerating sequences in a multi-turn setup naturally makesthe task dif\ufb01cult to evaluate \u2013 with full evaluation pos-sessing many of the dif\ufb01culties of the task itself as it re-quires deep understanding of the content of the conversa-tion. As in many other natural language generation (NLG)tasks, automatic metrics have not been shown to have aclear correlation with human evaluations (Liu et al.2016;Lowe et al.2017). This means the current standard for alldialogue research involves human trials, which slows downresearch and greatly increases the cost of model development.Unfortunately, human judgments are themselves dif\ufb01-cult to measure. The two most used approaches, single-turn pairwise evaluation (Vinyals and Le 2015; Li et al.2016b), and multi-turn Likert scores (Venkatesh et al.2017;Zhang et al.2018; See et al.2019; Dinan et al.2019b;Dinan et al.2019a) have serious limitations. Single-turn pair-wise evaluation provides the bene\ufb01ts and simplicity of anA/B test, allowing for cheap and fast annotations, with com-parisons that are robust to annotator score bias, but fail to takeinto account the multi-turn aspect of conversations. To givea trivial example, such comparisons fail to capture whether\nFigure 1:ACUTE-EV ALasks humans to compare two multi-turn dialogues, and independent of the gray speakers, choosebetween Speaker 1 (light blue) and Speaker 2 (dark blue).the model would repeat itself in a multi-turn conversationbecause they only look at one turn; repetition is a knownissue that humans dislike (See et al. 2019).Multi-turn Likert scores require the annotator to have amulti-turn conversation and then provide an integer score,which is more costly and time-consuming to run but evalu-ates full conversations more accurately. The integer scoreshowever suffer from differing bias and variance per annotator,which researchers have tried to mitigate (Kulikov et al.2018),but nevertheless due to its lack of sensitivity often yields com-parisons that are not statistically signi\ufb01cant. Furthermore, dueto strong anchoring effects during model evaluation, i.e. thatannotators are affected by the \ufb01rst systems they evaluate, Lik-ert comparisons are generally not comparable across multiplepapers. This mandates that evaluations of new models bearXiv:1909.03087v1  [cs.CL]  6 Sep 2019The ACUTE-EVAL methodLi et el., 2019\nFigure from Li, M., Weston, J., and Roller, S. (2019). Acute-eval: Improved dialogue evaluation with optimized questions and multi-turn comparisons. NeurIPS19 Workshop on Conversational AI.  Automatic evaluation is an open problemAutomatic evaluation methods (like the BLEU scores used for Machine Translation) are generally not used for chatbots. \u2022They correlate poorly with human judgements.One current research direction: Adversarial Evaluation\u2022Inspired by the Turing Test\u2022train a ``Turing-like'' classifier to distinguish between human responses and machine responses.\u2022The more successful a dialogue system is at fooling the evaluator, the better the system. Task-based systems are evaluated by task success!1.End-to-end evaluation (Task Success)2.Slot Error Rate for a Sentence# of inserted/deleted/subsitutedslots# of total reference slots for sentence Evaluation Metrics: Slot error rate\nSlot error rate: 1/3Task success: At end, was the correct meeting added to the calendar?\u201cMake an appointment with Chris at 10:30 in Gates 104\u201dSlotFillerPERSONChrisTIME11:30 a.m.ROOMGates 104 More fine-grained metrics: User Satisfaction Survey28CHAPTER24\u2022CHATBOTS&DIALOGUESYSTEMSthe Turing test. The idea is to train a \u201cTuring-like\u201d evaluator classi\ufb01er to distinguishbetween human-generated responses and machine-generated responses. The moresuccessful a response generation system is at fooling this evaluator, the better thesystem.24.5.2 Evaluating Task-Based DialogueFor task-based dialogue, if the task is unambiguous, we can simply measure absolutetask success (did the system book the right plane \ufb02ight, or put the right event on thecalendar).To get a more \ufb01ne-grained idea of user happiness, we can compute auser sat-isfaction rating, having users interact with a dialogue system to perform a task andthen having them complete a questionnaire. For example, Fig.24.20shows samplemultiple-choice questions(Walker et al., 2001); responses are mapped into the rangeof 1 to 5, and then averaged over all questions to get a total user satisfaction rating.TTS PerformanceWas the system easy to understand ?ASR PerformanceDid the system understand what you said?Task EaseWas it easy to \ufb01nd the message/\ufb02ight/train you wanted?Interaction PaceWas the pace of interaction with the system appropriate?User ExpertiseDid you know what you could say at each point?System ResponseHow often was the system sluggish and slow to reply to you?Expected BehaviorDid the system work the way you expected it to?Future UseDo you think you\u2019d use the system in the future?Figure 24.20User satisfaction survey, adapted fromWalker et al. (2001).It is often economically infeasible to run complete user satisfaction studies afterevery change in a system. For this reason, it is useful to have performance evaluationheuristics that correlate well with human satisfaction. A number of such factors andheuristics have been studied, often grouped into two kinds of criteria: how well thesystem allows users to accomplish their goals (maximizing task success) with thefewest problems (minimizing costs):Task completion success:Task success can be measured by evaluating the cor-rectness of the total solution. For a frame-based architecture, this might besloterror rate, the percentage of slots that were \ufb01lled with the correct values:Slot Error Rate for a Sentence=# of inserted/deleted/subsituted slots# of total reference slots for sentence(24.9)For example consider a system given this sentence:(24.10)Make an appointment with Chris at 10:30 in Gates 104which extracted the following candidate slot structure:Slot FillerPERSON ChrisTIME 11:30 a.m.ROOM Gates 104Here the slot error rate is 1/3, since the TIME is wrong. Instead of error rate,slot precision, recall, and F-score can also be used. Slot error rate is also sometimescalledconcept error rate.Interestingly, sometimes the user\u2019sperceptionof whether they completed thetask is a better predictor of user satisfaction than the actual task completion success.(Walker et al., 2001).Walker, Marilyn, Candace Kamm, and Diane Litman. \"Towards developing general models of usability with PARADISE.\"Natural Language Engineering6, no. 3 & 4 (2000): 363-377. Other HeuristicsEfficiency cost: \u2022total elapsed time for the dialogue in seconds, \u2022the number of total turns or of system turns\u2022total number of queries \u2022\u201cturn correction ratio\u201d: % of turns that were used to correct errorsQuality cost: \u2022number of ASR rejection prompts. \u2022number of times the user had to barge in Chatbots and Dialogue SystemsEvaluating Dialogue Systems Chatbots and Dialogue SystemsDesign and Ethical Issues Dialog System Design: User-centered Design1.Study the users and task\u2022value-sensitive design2.Build simulations\u2022Wizard of Oz study3.Iteratively test design on users\nGould, John D., and Clayton Lewis. \"Designing for usability: key principles and what designers think.\"Communications of the ACM28, no. 3 (1985): 300-311.Bender, Emily M., and BatyaFriedman. \"Data statements for natural language processing: Toward mitigating system bias and enabling better science.\" TACL 6 (2018): 587-604. Ethical designEthical issues have long been known to be crucial in artificial agentsMary Shelley's Frankenstein\u2022creating agents without a consideration of ethical and humanistic concerns Ethical issues:\u2022Safety:  Systems abusing users, distracting drivers, or giving bad medical advice\u2022Representational harm: Systems demeaning particular social groups\u2022Privacy: Information Leakage \n SafetyChatbots for mental health\u25e6Extremely important not to say the wrong thingIn-vehicle conversational agents\u25e6Must be aware of environment, driver's level of attention\nPeter Henderson, KoustuvSinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. 2018. Ethical Challenges in Data-Driven Dialogue Systems. In 2018 AAAI/ACM Conference on AI, Ethics, and Society (AIES \u201918),  Abuse and Representation Harm: The case of Microsoft TayExperimental Twitter chatbot launched in 2016\u25e6given the profile personality of an 18-to 24-year-old American woman\u25e6could share horoscopes, tell jokes, \u25e6asked people to send selfies\u25e6used informal language, slang, emojis, and GIFs, \u25e6Designed to learn from users (IR-based)  The case of Microsoft TayImmediately Tay turned offensive and abusive\u25e6Obscene and inflammatory tweets\u25e6Nazi propaganda, conspiracy theories\u25e6Began harassing women online\u25e6Reflecting racism and misogyny of Twitter usersMicrosoft took Tay down after 16 hoursLessons:\u25e6User response must be considered in the design phaseGina Neff and Peter Nagy 2016. Talking to Bots: Symbiotic Agency and the Case of Tay. International Journal of Communication 10(2016), 4915\u20134931  Bias in training datasetsHenderson et al. ran hate-speech and bias detectors on standard training sets for dialogue systems:\u25e6Twitter, Reddit, other dialogue datasetsFound bias and hate-speech\u25e6In training data\u25e6In dialogue models trained on the data\nPeter Henderson, KoustuvSinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. 2018. Ethical Challenges in Data-Driven Dialogue Systems. In 2018 AAAI/ACM Conference on AI, Ethics, and Society (AIES \u201918),  Privacy: Training on user dataAccidental information leakage\u2022\u201cComputer, turn on the lights [answers the phone] Hi, yes, my password is...\u201d\u2022Henderson show in simulation that this leakage can occur.Intentional information leakage\u2022Dialogue systems that are designed to send user data to developer or advertisers\u2022Important to consider privacy-preserving dialogue systemsPeter Henderson, KoustuvSinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. 2018. Ethical Challenges in Data-Driven Dialogue Systems. In 2018 AAAI/ACM Conference on AI, Ethics, and Society (AIES \u201918), Campagna, Giovanni, Rakesh Ramesh, SileiXu, Michael Fischer, and Monica S. Lam. \"Almond: The architecture of an open, crowdsourced, privacy-preserving, programmable virtual assistant.\" InProceedings of the 26th International Conference on World Wide Web, pp. 341-350. 2017. Chatbots and Dialogue SystemsDesign and Ethical Issues", "16.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\n16Automatic Speech Recognition\nand Text-to-Speech\nI KNOW not whether\nI see your meaning: if I do, it lies\nUpon the wordy wavelets of your voice,\nDim as an evening shadow in a brook,\nThomas Lovell Beddoes, 1851\nUnderstanding spoken language, or at least transcribing the words into writing, is\none of the earliest goals of computer language processing. In fact, speech processing\npredates the computer by many decades!\nThe \ufb01rst machine that recognized speech\nwas a toy from the 1920s. \u201cRadio Rex\u201d,\nshown to the right, was a celluloid dog\nthat moved (by means of a spring) when\nthe spring was released by 500 Hz acous-\ntic energy. Since 500 Hz is roughly the\n\ufb01rst formant of the vowel [eh] in \u201cRex\u201d,\nRex seemed to come when he was called\n(David, Jr. and Selfridge, 1962).\nIn modern times, we expect more of our automatic systems. The task of auto-\nmatic speech recognition (ASR ) is to map any waveform like this: ASR\nto the appropriate string of words:\nIt's time for lunch!\nAutomatic transcription of speech by any speaker in any environment is still far from\nsolved, but ASR technology has matured to the point where it is now viable for many\npractical tasks. Speech is a natural interface for communicating with smart home ap-\npliances, personal assistants, or cellphones, where keyboards are less convenient, in\ntelephony applications like call-routing (\u201cAccounting, please\u201d) or in sophisticated\ndialogue applications (\u201cI\u2019d like to change the return date of my \ufb02ight\u201d). ASR is also\nuseful for general transcription, for example for automatically generating captions\nfor audio or video text (transcribing movies or videos or live discussions). Transcrip-\ntion is important in \ufb01elds like law where dictation plays an important role. Finally,\nASR is important as part of augmentative communication (interaction between com-\nputers and humans with some disability resulting in dif\ufb01culties or inabilities in typ-\ning or audition). The blind Milton famously dictated Paradise Lost to his daughters,\nand Henry James dictated his later novels after a repetitive stress injury.\nWhat about the opposite problem, going from text to speech? This is a problem\nwith an even longer history. In Vienna in 1769, Wolfgang von Kempelen built for 2CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\nthe Empress Maria Theresa the famous Mechanical Turk, a chess-playing automaton\nconsisting of a wooden box \ufb01lled with gears, behind which sat a robot mannequin\nwho played chess by moving pieces with his mechanical arm. The Turk toured Eu-\nrope and the Americas for decades, defeating Napoleon Bonaparte and even playing\nCharles Babbage. The Mechanical Turk might have been one of the early successes\nof arti\ufb01cial intelligence were it not for the fact that it was, alas, a hoax, powered by\na human chess player hidden inside the box.\nWhat is less well known is that von Kempelen, an extraordinarily\nproli\ufb01c inventor, also built between\n1769 and 1790 what was de\ufb01nitely\nnot a hoax: the \ufb01rst full-sentence\nspeech synthesizer, shown partially to\nthe right. His device consisted of a\nbellows to simulate the lungs, a rub-\nber mouthpiece and a nose aperture, a\nreed to simulate the vocal folds, var-\nious whistles for the fricatives, and a\nsmall auxiliary bellows to provide the puff of air for plosives. By moving levers\nwith both hands to open and close apertures, and adjusting the \ufb02exible leather \u201cvo-\ncal tract\u201d, an operator could produce different consonants and vowels.\nMore than two centuries later, we no longer build our synthesizers out of wood\nand leather, nor do we need human operators. The modern task of speech synthesis ,speech\nsynthesis\nalso called text-to-speech orTTS , is exactly the reverse of ASR; to map text: text-to-speech\nTTSIt's time for lunch!\nto an acoustic waveform:\nModern speech synthesis has a wide variety of applications. TTS is used in\nconversational agents that conduct dialogues with people, plays a role in devices\nthat read out loud for the blind or in games, and can be used to speak for sufferers\nof neurological disorders, such as the late astrophysicist Steven Hawking who, after\nhe lost the use of his voice because of ALS, spoke by manipulating a TTS system.\nIn the next sections we\u2019ll show how to do ASR with encoder-decoders, intro-\nduce the CTC loss functions, the standard word error rate evaluation metric, and\ndescribe how acoustic features are extracted. We\u2019ll then see how TTS can be mod-\neled with almost the same algorithm in reverse, and conclude with a brief mention\nof other speech tasks.\n16.1 The Automatic Speech Recognition Task\nBefore describing algorithms for ASR, let\u2019s talk about how the task itself varies.\nOne dimension of variation is vocabulary size. Some ASR tasks can be solved with\nextremely high accuracy, like those with a 2-word vocabulary ( yesversus no) or\nan 11 word vocabulary like digit recognition (recognizing sequences of digits in-digit\nrecognition\ncluding zero tonine plus oh). Open-ended tasks like transcribing videos or human\nconversations, with large vocabularies of up to 60,000 words, are much harder. 16.1 \u2022 T HEAUTOMATIC SPEECH RECOGNITION TASK 3\nA second dimension of variation is who the speaker is talking to. Humans speak-\ning to machines (either dictating or talking to a dialogue system) are easier to recog-\nnize than humans speaking to humans. Read speech , in which humans are reading read speech\nout loud, for example in audio books, is also relatively easy to recognize. Recog-\nnizing the speech of two humans talking to each other in conversational speech ,conversational\nspeech\nfor example, for transcribing a business meeting, is the hardest. It seems that when\nhumans talk to machines, or read without an audience present, they simplify their\nspeech quite a bit, talking more slowly and more clearly.\nA third dimension of variation is channel and noise. Speech is easier to recognize\nif it\u2019s recorded in a quiet room with head-mounted microphones than if it\u2019s recorded\nby a distant microphone on a noisy city street, or in a car with the window open.\nA \ufb01nal dimension of variation is accent or speaker-class characteristics. Speech\nis easier to recognize if the speaker is speaking the same dialect or variety that the\nsystem was trained on. Speech by speakers of regional or ethnic dialects, or speech\nby children can be quite dif\ufb01cult to recognize if the system is only trained on speak-\ners of standard dialects, or only adult speakers.\nA number of publicly available corpora with human-created transcripts are used\nto create ASR test and training sets to explore this variation; we mention a few of\nthem here since you will encounter them in the literature. LibriSpeech is a large LibriSpeech\nopen-source read-speech 16 kHz dataset with over 1000 hours of audio books from\nthe LibriV ox project, with transcripts aligned at the sentence level (Panayotov et al.,\n2015). It is divided into an easier (\u201cclean\u201d) and a more dif\ufb01cult portion (\u201cother\u201d)\nwith the clean portion of higher recording quality and with accents closer to US\nEnglish. This was done by running a speech recognizer (trained on read speech from\nthe Wall Street Journal) on all the audio, computing the WER for each speaker based\non the gold transcripts, and dividing the speakers roughly in half, with recordings\nfrom lower-WER speakers called \u201cclean\u201d and recordings from higher-WER speakers\n\u201cother\u201d.\nTheSwitchboard corpus of prompted telephone conversations between strangers Switchboard\nwas collected in the early 1990s; it contains 2430 conversations averaging 6 min-\nutes each, totaling 240 hours of 8 kHz speech and about 3 million words (Godfrey\net al., 1992). Switchboard has the singular advantage of an enormous amount of\nauxiliary hand-done linguistic labeling, including parses, dialogue act tags, phonetic\nand prosodic labeling, and discourse and information structure. The CALLHOME CALLHOME\ncorpus was collected in the late 1990s and consists of 120 unscripted 30-minute\ntelephone conversations between native speakers of English who were usually close\nfriends or family (Canavan et al., 1997).\nThe Santa Barbara Corpus of Spoken American English (Du Bois et al., 2005) is\na large corpus of naturally occurring everyday spoken interactions from all over the\nUnited States, mostly face-to-face conversation, but also town-hall meetings, food\npreparation, on-the-job talk, and classroom lectures. The corpus was anonymized by\nremoving personal names and other identifying information (replaced by pseudonyms\nin the transcripts, and masked in the audio).\nCORAAL is a collection of over 150 sociolinguistic interviews with African CORAAL\nAmerican speakers, with the goal of studying African American Language ( AAL ),\nthe many variations of language used in African American communities (Kendall\nand Farrington, 2020). The interviews are anonymized with transcripts aligned at\nthe utterance level. The CHiME Challenge is a series of dif\ufb01cult shared tasks with CHiME\ncorpora that deal with robustness in ASR. The CHiME 5 task, for example, is ASR of\nconversational speech in real home environments (speci\ufb01cally dinner parties). The 4CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\ncorpus contains recordings of twenty different dinner parties in real homes, each\nwith four participants, and in three locations (kitchen, dining area, living room),\nrecorded both with distant room microphones and with body-worn mikes. The\nHKUST Mandarin Telephone Speech corpus has 1206 ten-minute telephone con- HKUST\nversations between speakers of Mandarin across China, including transcripts of the\nconversations, which are between either friends or strangers (Liu et al., 2006). The\nAISHELL-1 corpus contains 170 hours of Mandarin read speech of sentences taken AISHELL-1\nfrom various domains, read by different speakers mainly from northern China (Bu\net al., 2017).\nFigure 16.1 shows the rough percentage of incorrect words (the word error rate ,\nor WER, de\ufb01ned on page 16) from state-of-the-art systems on some of these tasks.\nNote that the error rate on read speech (like the LibriSpeech audiobook corpus) is\naround 2%; this is a solved task, although these numbers come from systems that re-\nquire enormous computational resources. By contrast, the error rate for transcribing\nconversations between humans is much higher; 5.8 to 11% for the Switchboard and\nCALLHOME corpora. The error rate is higher yet again for speakers of varieties\nlike African American Vernacular English, and yet again for dif\ufb01cult conversational\ntasks like transcription of 4-speaker dinner party speech, which can have error rates\nas high as 81.3%. Character error rates (CER) are also much lower for read Man-\ndarin speech than for natural conversation.\nEnglish Tasks WER %\nLibriSpeech audiobooks 960hour clean 1.4\nLibriSpeech audiobooks 960hour other 2.6\nSwitchboard telephone conversations between strangers 5.8\nCALLHOME telephone conversations between family 11.0\nSociolinguistic interviews, CORAAL (AAL) 27.0\nCHiMe5 dinner parties with body-worn microphones 47.9\nCHiMe5 dinner parties with distant microphones 81.3\nChinese (Mandarin) Tasks CER %\nAISHELL-1 Mandarin read speech corpus 6.7\nHKUST Mandarin Chinese telephone conversations 23.5\nFigure 16.1 Rough Word Error Rates (WER = % of words misrecognized) reported around\n2020 for ASR on various American English recognition tasks, and character error rates (CER)\nfor two Chinese recognition tasks.\n16.2 Feature Extraction for ASR: Log Mel Spectrum\nThe \ufb01rst step in ASR is to transform the input waveform into a sequence of acoustic\nfeature vectors , each vector representing the information in a small time window feature vector\nof the signal. Let\u2019s see how to convert a raw wave\ufb01le to the most commonly used\nfeatures, sequences of log mel spectrum vectors. A speech signal processing course\nis recommended for more details.\n16.2.1 Sampling and Quantization\nThe input to a speech recognizer is a complex series of changes in air pressure.\nThese changes in air pressure obviously originate with the speaker and are caused 16.2 \u2022 F EATURE EXTRACTION FOR ASR: L OGMELSPECTRUM 5\nby the speci\ufb01c way that air passes through the glottis and out the oral or nasal cav-\nities. We represent sound waves by plotting the change in air pressure over time.\nOne metaphor which sometimes helps in understanding these graphs is that of a ver-\ntical plate blocking the air pressure waves (perhaps in a microphone in front of a\nspeaker\u2019s mouth, or the eardrum in a hearer\u2019s ear). The graph measures the amount\nofcompression orrarefaction (uncompression) of the air molecules at this plate.\nFigure 16.2 shows a short segment of a waveform taken from the Switchboard corpus\nof telephone speech of the vowel [iy] from someone saying \u201cshe just had a baby\u201d.\nTime (s)0 0.03875\u20130.016970.02283\n0\nFigure 16.2 A waveform of an instance of the vowel [iy] (the last vowel in the word \u201cbaby\u201d). The y-axis\nshows the level of air pressure above and below normal atmospheric pressure. The x-axis shows time. Notice\nthat the wave repeats regularly.\nThe \ufb01rst step in digitizing a sound wave like Fig. 16.2 is to convert the analog\nrepresentations (\ufb01rst air pressure and then analog electric signals in a microphone)\ninto a digital signal. This analog-to-digital conversion has two steps: sampling and sampling\nquantization . To sample a signal, we measure its amplitude at a particular time; the\nsampling rate is the number of samples taken per second. To accurately measure a\nwave, we must have at least two samples in each cycle: one measuring the positive\npart of the wave and one measuring the negative part. More than two samples per\ncycle increases the amplitude accuracy, but fewer than two samples causes the fre-\nquency of the wave to be completely missed. Thus, the maximum frequency wave\nthat can be measured is one whose frequency is half the sample rate (since every\ncycle needs two samples). This maximum frequency for a given sampling rate is\ncalled the Nyquist frequency . Most information in human speech is in frequenciesNyquist\nfrequency\nbelow 10,000 Hz; thus, a 20,000 Hz sampling rate would be necessary for com-\nplete accuracy. But telephone speech is \ufb01ltered by the switching network, and only\nfrequencies less than 4,000 Hz are transmitted by telephones. Thus, an 8,000 Hz\nsampling rate is suf\ufb01cient for telephone-bandwidth speech like the Switchboard\ncorpus, while 16,000 Hz sampling is often used for microphone speech.\nAlthough using higher sampling rates produces higher ASR accuracy, we can\u2019t\ncombine different sampling rates for training and testing ASR systems. Thus if\nwe are testing on a telephone corpus like Switchboard (8 KHz sampling), we must\ndownsample our training corpus to 8 KHz. Similarly, if we are training on mul-\ntiple corpora and one of them includes telephone speech, we downsample all the\nwideband corpora to 8Khz.\nAmplitude measurements are stored as integers, either 8 bit (values from -128\u2013\n127) or 16 bit (values from -32768\u201332767). This process of representing real-valued\nnumbers as integers is called quantization ; all values that are closer together than quantization\nthe minimum granularity (the quantum size) are represented identically. We refer to\neach sample at time index nin the digitized, quantized waveform as x[n].\nOnce data is quantized, it is stored in various formats. One parameter of these\nformats is the sample rate and sample size discussed above; telephone speech is\noften sampled at 8 kHz and stored as 8-bit samples, and microphone data is often\nsampled at 16 kHz and stored as 16-bit samples. Another parameter is the number of 6CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\nchannels . For stereo data or for two-party conversations, we can store both channels channel\nin the same \ufb01le or we can store them in separate \ufb01les. A \ufb01nal parameter is individual\nsample storage\u2014linearly or compressed. One common compression format used for\ntelephone speech is m-law (often written u-law but still pronounced mu-law). The\nintuition of log compression algorithms like m-law is that human hearing is more\nsensitive at small intensities than large ones; the log represents small values with\nmore faithfulness at the expense of more error on large values. The linear (unlogged)\nvalues are generally referred to as linear PCM values (PCM stands for pulse code PCM\nmodulation, but never mind that). Here\u2019s the equation for compressing a linear PCM\nsample value xto 8-bit m-law, (where m=255 for 8 bits):\nF(x) =sgn(x)log(1+mjxj)\nlog(1+m)\u00001\u0014x\u00141 (16.1)\nThere are a number of standard \ufb01le formats for storing the resulting digitized wave-\n\ufb01le, such as Microsoft\u2019s .wav and Apple\u2019s AIFF all of which have special headers;\nsimple headerless \u201craw\u201d \ufb01les are also used. For example, the .wav format is a sub-\nset of Microsoft\u2019s RIFF format for multimedia \ufb01les; RIFF is a general format that\ncan represent a series of nested chunks of data and control information. Figure 16.3\nshows a simple .wav \ufb01le with a single data chunk together with its format chunk.\nFigure 16.3 Microsoft wave\ufb01le header format, assuming simple \ufb01le with one chunk. Fol-\nlowing this 44-byte header would be the data chunk.\n16.2.2 Windowing\nFrom the digitized, quantized representation of the waveform, we need to extract\nspectral features from a small window of speech that characterizes part of a par-\nticular phoneme. Inside this small window, we can roughly think of the signal as\nstationary (that is, its statistical properties are constant within this region). (By stationary\ncontrast, in general, speech is a non-stationary signal, meaning that its statistical non-stationary\nproperties are not constant over time). We extract this roughly stationary portion of\nspeech by using a window which is non-zero inside a region and zero elsewhere, run-\nning this window across the speech signal and multiplying it by the input waveform\nto produce a windowed waveform.\nThe speech extracted from each window is called a frame . The windowing is frame\ncharacterized by three parameters: the window size orframe size of the window\n(its width in milliseconds), the frame stride , (also called shift oroffset ) between stride\nsuccessive windows, and the shape of the window.\nTo extract the signal we multiply the value of the signal at time n,s[n]by the\nvalue of the window at time n,w[n]:\ny[n] =w[n]s[n] (16.2)\nThe window shape sketched in Fig. 16.4 is rectangular ; you can see the ex- rectangular\ntracted windowed signal looks just like the original signal. The rectangular window, 16.2 \u2022 F EATURE EXTRACTION FOR ASR: L OGMELSPECTRUM 7\nShift10 msWindow25 msShift10 msWindow25 msWindow25 ms\nFigure 16.4 Windowing, showing a 25 ms rectangular window with a 10ms stride.\nhowever, abruptly cuts off the signal at its boundaries, which creates problems when\nwe do Fourier analysis. For this reason, for acoustic feature creation we more com-\nmonly use the Hamming window, which shrinks the values of the signal toward Hamming\nzero at the window boundaries, avoiding discontinuities. Figure 16.5 shows both;\nthe equations are as follows (assuming a window that is Lframes long):\nrectangular w [n] =\u001a1 0\u0014n\u0014L\u00001\n0 otherwise(16.3)\nHamming w [n] =\u001a\n0:54\u00000:46cos (2pn\nL)0\u0014n\u0014L\u00001\n0 otherwise(16.4)\nTime (s)00.0475896\u20130.50.49990Rectangular windowHamming window\nTime (s)0.004559380.0256563\u20130.48260.49990\nTime (s)0.004559380.0256563\u20130.50.49990\nFigure 16.5 Windowing a sine wave with the rectangular or Hamming windows.\n16.2.3 Discrete Fourier Transform\nThe next step is to extract spectral information for our windowed signal; we need to\nknow how much energy the signal contains at different frequency bands. The tool 8CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\nfor extracting spectral information for discrete frequency bands for a discrete-time\n(sampled) signal is the discrete Fourier transform orDFT .Discrete\nFourier\ntransformDFT The input to the DFT is a windowed signal x[n]:::x[m], and the output, for each\nofNdiscrete frequency bands, is a complex number X[k]representing the magni-\ntude and phase of that frequency component in the original signal. If we plot the\nmagnitude against the frequency, we can visualize the spectrum (see Appendix H\nfor more on spectra). For example, Fig. 16.6 shows a 25 ms Hamming-windowed\nportion of a signal and its spectrum as computed by a DFT (with some additional\nsmoothing).\nTime (s)0.0141752 0.039295\u20130.041210.04414\n0\nFrequency (Hz)0 8000Sound pressure level (dB /Hz)\n\u201320020\n(a) (b)\nFigure 16.6 (a) A 25 ms Hamming-windowed portion of a signal from the vowel [iy]\nand (b) its spectrum computed by a DFT.\nWe do not introduce the mathematical details of the DFT here, except to note\nthat Fourier analysis relies on Euler\u2019s formula , with jas the imaginary unit: Euler\u2019s formula\nejq=cosq+jsinq (16.5)\nAs a brief reminder for those students who have already studied signal processing,\nthe DFT is de\ufb01ned as follows:\nX[k] =N\u00001X\nn=0x[n]e\u0000j2p\nNkn(16.6)\nA commonly used algorithm for computing the DFT is the fast Fourier transformfast Fourier\ntransform\norFFT . This implementation of the DFT is very ef\ufb01cient but only works for values FFT\nofNthat are powers of 2.\n16.2.4 Mel Filter Bank and Log\nThe results of the FFT tell us the energy at each frequency band. Human hearing,\nhowever, is not equally sensitive at all frequency bands; it is less sensitive at higher\nfrequencies. This bias toward low frequencies helps human recognition, since in-\nformation in low frequencies (like formants) is crucial for distinguishing vowels or\nnasals, while information in high frequencies (like stop bursts or fricative noise) is\nless crucial for successful recognition. Modeling this human perceptual property\nimproves speech recognition performance in the same way.\nWe implement this intuition by collecting energies, not equally at each frequency\nband, but according to the melscale, an auditory frequency scale. A mel(Stevens mel\net al. 1937, Stevens and V olkmann 1940) is a unit of pitch. Pairs of sounds that are\nperceptually equidistant in pitch are separated by an equal number of mels. The mel 16.3 \u2022 S PEECH RECOGNITION ARCHITECTURE 9\nfrequency mcan be computed from the raw acoustic frequency by a log transforma-\ntion:\nmel(f) =1127ln (1+f\n700) (16.7)\nWe implement this intuition by creating a bank of \ufb01lters that collect energy from\neach frequency band, spread logarithmically so that we have very \ufb01ne resolution\nat low frequencies, and less resolution at high frequencies. Figure 16.7 shows a\nsample bank of triangular \ufb01lters that implement this idea, that can be multiplied by\nthe spectrum to get a mel spectrum.\nm1m2mM...mel spectrum0770000.51AmplitudeFrequency (Hz)8K\nFigure 16.7 The mel \ufb01lter bank (Davis and Mermelstein, 1980). Each triangular \ufb01lter,\nspaced logarithmically along the mel scale, collects energy from a given frequency range.\nFinally, we take the log of each of the mel spectrum values. The human response\nto signal level is logarithmic (like the human response to frequency). Humans are\nless sensitive to slight differences in amplitude at high amplitudes than at low ampli-\ntudes. In addition, using a log makes the feature estimates less sensitive to variations\nin input such as power variations due to the speaker\u2019s mouth moving closer or further\nfrom the microphone.\n16.3 Speech Recognition Architecture\nThe basic architecture for ASR is the encoder-decoder (implemented with either\nRNNs or Transformers), exactly the same architecture introduced for MT in Chap-\nter 13. Generally we start from the log mel spectral features described in the previous\nsection, and map to letters, although it\u2019s also possible to map to induced morpheme-\nlike chunks like wordpieces or BPE.\nFig. 16.8 sketches the standard encoder-decoder architecture, which is com-\nmonly referred to as the attention-based encoder decoder orAED , orlisten attend AED\nand spell (LAS ) after the two papers which \ufb01rst applied it to speech (Chorowskilisten attend\nand spell\net al. 2014, Chan et al. 2016). The input is a sequence of tacoustic feature vectors\nF=f1;f2;:::;ft, one vector per 10 ms frame. The output can be letters or word-\npieces; we\u2019ll assume letters here. Thus the output sequence Y= (hSOSi;y1;:::;ymhEOSi),\nassuming special start of sequence and end of sequence tokens hsosiandheosiand\neach yiis a character; for English we might choose the set:\nyi2fa;b;c;:::;z;0;:::;9;hspacei;hcommai;hperiodi;hapostrophei;hunkig\nOf course the encoder-decoder architecture is particularly appropriate when in-\nput and output sequences have stark length differences, as they do for speech, with\nvery long acoustic feature sequences mapping to much shorter sequences of letters 10 CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\nENCODER\u2026DECODER\u2026\u2026ym\nFeature ComputationSubsampling\u2026Hftf180-dimensional log Mel spectrumper frameShorter sequence Xy1<s>iy2ity3t\u2018y4\u2018sy5s y6 ty7tiy8imy9mex1xn\nFigure 16.8 Schematic architecture for an encoder-decoder speech recognizer.\nor words. A single word might be 5 letters long but, supposing it lasts about 2\nseconds, would take 200 acoustic frames (of 10ms each).\nBecause this length difference is so extreme for speech, encoder-decoder ar-\nchitectures for speech need to have a special compression stage that shortens the\nacoustic feature sequence before the encoder stage. (Alternatively, we can use a loss\nfunction that is designed to deal well with compression, like the CTC loss function\nwe\u2019ll introduce in the next section.)\nThe goal of the subsampling is to produce a shorter sequence X=x1;:::;xnthat\nwill be the input to the encoder. The simplest algorithm is a method sometimes\ncalled low frame rate (Pundak and Sainath, 2016): for time iwe stack (concatenate) low frame rate\nthe acoustic feature vector fiwith the prior two vectors fi\u00001andfi\u00002to make a new\nvector three times longer. Then we simply delete fi\u00001and fi\u00002. Thus instead of\n(say) a 40-dimensional acoustic feature vector every 10 ms, we have a longer vector\n(say 120-dimensional) every 30 ms, with a shorter sequence length n=t\n3.1\nAfter this compression stage, encoder-decoders for speech use the same archi-\ntecture as for MT or other text, composed of either RNNs (LSTMs) or Transformers.\nFor inference, the probability of the output string Yis decomposed as:\np(y1;:::; yn) =nY\ni=1p(yijy1;:::; yi\u00001;X) (16.8)\nWe can produce each letter of the output via greedy decoding:\n\u02c6yi=argmax char2Alphabet P(charjy1:::yi\u00001;X) (16.9)\nAlternatively we can use beam search as described in the next section. This is par-\nticularly relevant when we are adding a language model.\nAdding a language model Since an encoder-decoder model is essentially a con-\nditional language model, encoder-decoders implicitly learn a language model for the\noutput domain of letters from their training data. However, the training data (speech\npaired with text transcriptions) may not include suf\ufb01cient text to train a good lan-\nguage model. After all, it\u2019s easier to \ufb01nd enormous amounts of pure text training\n1There are also more complex alternatives for subsampling, like using a convolutional net that down-\nsamples with max pooling, or layers of pyramidal RNNs , RNNs where each successive layer has half\nthe number of RNNs as the previous layer. 16.4 \u2022 CTC 11\ndata than it is to \ufb01nd text paired with speech. Thus we can can usually improve a\nmodel at least slightly by incorporating a very large language model.\nThe simplest way to do this is to use beam search to get a \ufb01nal beam of hy-\npothesized sentences; this beam is sometimes called an n-best list . We then use a n-best list\nlanguage model to rescore each hypothesis on the beam. The scoring is done by in- rescore\nterpolating the score assigned by the language model with the encoder-decoder score\nused to create the beam, with a weight ltuned on a held-out set. Also, since most\nmodels prefer shorter sentences, ASR systems normally have some way of adding a\nlength factor. One way to do this is to normalize the probability by the number of\ncharacters in the hypothesis jYjc. The following is thus a typical scoring function\n(Chan et al., 2016):\nscore(YjX) =1\njYjclogP(YjX)+llogPLM(Y) (16.10)\n16.3.1 Learning\nEncoder-decoders for speech are trained with the normal cross-entropy loss gener-\nally used for conditional language models. At timestep iof decoding, the loss is the\nlog probability of the correct token (letter) yi:\nLCE=\u0000logp(yijy1;:::; yi\u00001;X) (16.11)\nThe loss for the entire sentence is the sum of these losses:\nLCE=\u0000mX\ni=1logp(yijy1;:::; yi\u00001;X) (16.12)\nThis loss is then backpropagated through the entire end-to-end model to train the\nentire encoder-decoder.\nAs we described in Chapter 13, we normally use teacher forcing, in which the\ndecoder history is forced to be the correct gold yirather than the predicted \u02c6 yi. It\u2019s\nalso possible to use a mixture of the gold and decoder output, for example using\nthe gold output 90% of the time, but with probability .1 taking the decoder output\ninstead:\nLCE=\u0000logp(yijy1;:::; \u02c6yi\u00001;X) (16.13)\n16.4 CTC\nWe pointed out in the previous section that speech recognition has two particular\nproperties that make it very appropriate for the encoder-decoder architecture, where\nthe encoder produces an encoding of the input that the decoder uses attention to\nexplore. First, in speech we have a very long acoustic input sequence Xmapping to\na much shorter sequence of letters Y, and second, it\u2019s hard to know exactly which\npart of Xmaps to which part of Y.\nIn this section we brie\ufb02y introduce an alternative to encoder-decoder: an algo-\nrithm and loss function called CTC , short for Connectionist Temporal Classi\ufb01ca- CTC\ntion(Graves et al., 2006), that deals with these problems in a very different way. The\nintuition of CTC is to output a single character for every frame of the input, so that 12 CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\nthe output is the same length as the input, and then to apply a collapsing function\nthat combines sequences of identical letters, resulting in a shorter sequence.\nLet\u2019s imagine inference on someone saying the word dinner , and let\u2019s suppose\nwe had a function that chooses the most probable letter for each input spectral frame\nrepresentation xi. We\u2019ll call the sequence of letters corresponding to each input\nframe an alignment , because it tells us where in the acoustic signal each letter aligns alignment\nto. Fig. 16.9 shows one such alignment, and what happens if we use a collapsing\nfunction that just removes consecutive duplicate letters.\nX (input)A (alignment)Y (output)dx1ix2ix3nx4nx5nx6nx7ex8rx9rx10rx11rx12rx13rx14dinerwavefile\nFigure 16.9 A naive algorithm for collapsing an alignment between input and letters.\nWell, that doesn\u2019t work; our naive algorithm has transcribed the speech as diner ,\nnotdinner ! Collapsing doesn\u2019t handle double letters. There\u2019s also another problem\nwith our naive function; it doesn\u2019t tell us what symbol to align with silence in the\ninput. We don\u2019t want to be transcribing silence as random letters!\nThe CTC algorithm solves both problems by adding to the transcription alphabet\na special symbol for a blank , which we\u2019ll represent as . The blank can be used in blank\nthe alignment whenever we don\u2019t want to transcribe a letter. Blank can also be used\nbetween letters; since our collapsing function collapses only consecutive duplicate\nletters, it won\u2019t collapse across . More formally, let\u2019s de\ufb01ne the mapping B:a!y\nbetween an alignment aand an output y, which collapses all repeated letters and\nthen removes all blanks. Fig. 16.10 sketches this collapsing function B.\nX (input)A (alignment)remove blanksdx1ix2x3nx4nx5x6nx7ex8rx9rx10rx11rx12x13x14dinernmerge duplicatesdinernY (output)dinern\u2423\u2423\u2423\u2423\u2423\u2423\u2423\nFigure 16.10 The CTC collapsing function B, showing the space blank character ; re-\npeated (consecutive) characters in an alignment Aare removed to form the output Y.\nThe CTC collapsing function is many-to-one; lots of different alignments map\nto the same output string. For example, the alignment shown in Fig. 16.10 is not\nthe only alignment that results in the string dinner . Fig. 16.11 shows some other\nalignments that would produce the same output.\nIt\u2019s useful to think of the set of all alignments that might produce the same output\nY. We\u2019ll use the inverse of our Bfunction, called B\u00001, and represent that set as\nB\u00001(Y). 16.4 \u2022 CTC 13\ndinnneeerrr\u2423\u2423ddinnnerr\u2423\u2423\u2423dddinnnerr\u2423i\u2423\u2423\u2423\u2423\u2423\nFigure 16.11 Three other legitimate alignments producing the transcript dinner .\n16.4.1 CTC Inference\nBefore we see how to compute PCTC(YjX)let\u2019s \ufb01rst see how CTC assigns a proba-\nbility to one particular alignment \u02c6A=f\u02c6a1;:::; \u02c6ang. CTC makes a strong conditional\nindependence assumption: it assumes that, given the input X, the CTC model output\natat time tis independent of the output labels at any other time ai. Thus:\nPCTC(AjX) =TY\nt=1p(atjX) (16.14)\nThus to \ufb01nd the best alignment \u02c6A=f\u02c6a1;:::; \u02c6aTgwe can greedily choose the charac-\nter with the max probability at each time step t:\n\u02c6at=argmax\nc2Cpt(cjX) (16.15)\nWe then pass the resulting sequence Ato the CTC collapsing function Bto get the\noutput sequence Y.\nLet\u2019s talk about how this simple inference algorithm for \ufb01nding the best align-\nment A would be implemented. Because we are making a decision at each time\npoint, we can treat CTC as a sequence-modeling task, where we output one letter\n\u02c6ytat time tcorresponding to each input token xt, eliminating the need for a full de-\ncoder. Fig. 16.12 sketches this architecture, where we take an encoder, produce a\nhidden state htat each timestep, and decode by taking a softmax over the character\nvocabulary at each time step.\nENCODER\u2026yn\nFeature ComputationSubsampling\u2026ftf1 log Mel spectrumShorter inputsequence Xy1iy2iy3iy4tx1xnClassi\ufb01er+softmax\u2026ty5\u2026\u2026output lettersequence Y\nFigure 16.12 Inference with CTC: using an encoder-only model, with decoding done by\nsimple softmaxes over the hidden state htat each output step.\nAlas, there is a potential \ufb02aw with the inference algorithm sketched in (Eq. 16.15)\nand Fig. 16.11. The problem is that we chose the most likely alignment A, but the 14 CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\nmost likely alignment may not correspond to the most likely \ufb01nal collapsed output\nstring Y. That\u2019s because there are many possible alignments that lead to the same\noutput string, and hence the most likely output string might not correspond to the\nmost probable alignment. For example, imagine the most probable alignment Afor\nan input X= [x1x2x3]is the string [a b \u000f] but the next two most probable alignments\nare [b\u000fb] and [\u000fb b]. The output Y=[b b], summing over those two alignments,\nmight be more probable than Y=[a b].\nFor this reason, the most probable output sequence Yis the one that has, not\nthe single best CTC alignment, but the highest sum over the probability of all its\npossible alignments:\nPCTC(YjX) =X\nA2B\u00001(Y)P(AjX)\n=X\nA2B\u00001(Y)TY\nt=1p(atjht)\n\u02c6Y=argmax\nYPCTC(YjX) (16.16)\nAlas, summing over all alignments is very expensive (there are a lot of alignments),\nso we approximate this sum by using a version of Viterbi beam search that cleverly\nkeeps in the beam the high-probability alignments that map to the same output string,\nand sums those as an approximation of (Eq. 16.16). See Hannun (2017) for a clear\nexplanation of this extension of beam search for CTC.\nBecause of the strong conditional independence assumption mentioned earlier\n(that the output at time tis independent of the output at time t\u00001, given the input),\nCTC does not implicitly learn a language model over the data (unlike the attention-\nbased encoder-decoder architectures). It is therefore essential when using CTC to\ninterpolate a language model (and some sort of length factor L(Y)) using interpola-\ntion weights that are trained on a devset:\nscore CTC(YjX) =logPCTC(YjX)+l1logPLM(Y)l2L(Y) (16.17)\n16.4.2 CTC Training\nTo train a CTC-based ASR system, we use negative log-likelihood loss with a special\nCTC loss function. Thus the loss for an entire dataset Dis the sum of the negative\nlog-likelihoods of the correct output Yfor each input X:\nLCTC=X\n(X;Y)2D\u0000logPCTC(YjX) (16.18)\nTo compute CTC loss function for a single input pair (X;Y), we need the probability\nof the output Ygiven the input X. As we saw in Eq. 16.16, to compute the probability\nof a given output Ywe need to sum over all the possible alignments that would\ncollapse to Y. In other words:\nPCTC(YjX) =X\nA2B\u00001(Y)TY\nt=1p(atjht) (16.19)\nNaively summing over all possible alignments is not feasible (there are too many\nalignments). However, we can ef\ufb01ciently compute the sum by using dynamic pro- 16.4 \u2022 CTC 15\ngramming to merge alignments, with a version of the forward-backward algo-\nrithm also used to train HMMs (Appendix A) and CRFs. The original dynamic pro-\ngramming algorithms for both training and inference are laid out in (Graves et al.,\n2006); see (Hannun, 2017) for a detailed explanation of both.\n16.4.3 Combining CTC and Encoder-Decoder\nIt\u2019s also possible to combine the two architectures/loss functions we\u2019ve described,\nthe cross-entropy loss from the encoder-decoder architecture, and the CTC loss.\nFig. 16.13 shows a sketch. For training, we can simply weight the two losses with a\nltuned on a devset:\nL=\u0000llogPencdec(YjX)\u0000(1\u0000l)logPctc(YjX) (16.20)\nFor inference, we can combine the two with the language model (or the length\npenalty), again with learned weights:\n\u02c6Y=argmax\nY[llogPencdec(YjX)\u0000(1\u0000l)logPCTC(YjX)+glogPLM(Y)](16.21)\nENCODER\u2026DECODER\u2026H<s>it\u2018s timx1xn\u2026\u2026i   t   \u2019   s      t   i   m   e  \u2026CTC LossEncoder-Decoder Loss\nFigure 16.13 Combining the CTC and encoder-decoder loss functions.\n16.4.4 Streaming Models: RNN-T for improving CTC\nBecause of the strong independence assumption in CTC (assuming that the output\nat time tis independent of the output at time t\u00001), recognizers based on CTC\ndon\u2019t achieve as high an accuracy as the attention-based encoder-decoder recog-\nnizers. CTC recognizers have the advantage, however, that they can be used for\nstreaming . Streaming means recognizing words on-line rather than waiting until streaming\nthe end of the sentence to recognize them. Streaming is crucial for many applica-\ntions, from commands to dictation, where we want to start recognition while the\nuser is still talking. Algorithms that use attention need to compute the hidden state\nsequence over the entire input \ufb01rst in order to provide the attention distribution con-\ntext, before the decoder can start decoding. By contrast, a CTC algorithm can input\nletters from left to right immediately.\nIf we want to do streaming, we need a way to improve CTC recognition to re-\nmove the conditional independent assumption, enabling it to know about output his-\ntory. The RNN-Transducer ( RNN-T ), shown in Fig. 16.14, is just such a model RNN-T\n(Graves 2012, Graves et al. 2013). The RNN-T has two main components: a CTC\nacoustic model, and a separate language model component called the predictor that\nconditions on the output token history. At each time step t, the CTC encoder outputs 16 CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\na hidden state henc\ntgiven the input x1:::xt. The language model predictor takes as in-\nput the previous output token (not counting blanks), outputting a hidden state hpred\nu.\nThe two are passed through another network whose output is then passed through a\nsoftmax to predict the next character.\nPRNN\u0000T(YjX) =X\nA2B\u00001(Y)P(AjX)\n=X\nA2B\u00001(Y)TY\nt=1p(atjht;y<ut)\nENCODERP ( yt,u | x[1..t] , y[1..u-1] )\nxtPREDICTIONNETWORKyu-1JOINT NETWORKhencthpreduSOFTMAXzt,uDECODER\nFigure 16.14 The RNN-T model computing the output token distribution at time tby inte-\ngrating the output of a CTC acoustic encoder and a separate \u2018predictor\u2019 language model.\n16.5 ASR Evaluation: Word Error Rate\nThe standard evaluation metric for speech recognition systems is the word error word error\nrate. The word error rate is based on how much the word string returned by the\nrecognizer (the hypothesized word string) differs from a reference transcription.\nThe \ufb01rst step in computing word error is to compute the minimum edit distance in\nwords between the hypothesized and correct strings, giving us the minimum num-\nber of word substitutions , word insertions , and word deletions necessary to map\nbetween the correct and hypothesized strings. The word error rate (WER) is then\nde\ufb01ned as follows (note that because the equation includes insertions, the error rate\ncan be greater than 100%):\nWord Error Rate =100\u0002Insertions +Substitutions +Deletions\nTotal Words in Correct Transcript\nHere is a sample alignment between a reference and a hypothesis utterance from alignment\nthe CallHome corpus, showing the counts used to compute the error rate:\nREF: i *** ** UM the PHONE IS i LEFT THE portable **** PHONE UPSTAIRS last night\nHYP: i GOT IT TO the ***** FULLEST i LOVE TO portable FORM OF STORES last night\nEval: I I S D S S S I S S\nThis utterance has six substitutions, three insertions, and one deletion:\nWord Error Rate =1006+3+1\n13=76:9% 16.5 \u2022 ASR E VALUATION : W ORD ERROR RATE 17\nThe standard method for computing word error rates is a free script called sclite ,\navailable from the National Institute of Standards and Technologies (NIST) (NIST,\n2005). Sclite is given a series of reference (hand-transcribed, gold-standard) sen-\ntences and a matching set of hypothesis sentences. Besides performing alignments,\nand computing word error rate, sclite performs a number of other useful tasks. For\nexample, for error analysis it gives useful information such as confusion matrices\nshowing which words are often misrecognized for others, and summarizes statistics\nof words that are often inserted or deleted. sclite also gives error rates by speaker\n(if sentences are labeled for speaker ID), as well as useful statistics like the sentence\nerror rate , the percentage of sentences with at least one word error.Sentence error\nrate\nStatistical signi\ufb01cance for ASR: MAPSSWE or MacNemar\nAs with other language processing algorithms, we need to know whether a particular\nimprovement in word error rate is signi\ufb01cant or not.\nThe standard statistical tests for determining if two word error rates are different\nis the Matched-Pair Sentence Segment Word Error (MAPSSWE) test, introduced in\nGillick and Cox (1989).\nThe MAPSSWE test is a parametric test that looks at the difference between\nthe number of word errors the two systems produce, averaged across a number of\nsegments. The segments may be quite short or as long as an entire utterance; in\ngeneral, we want to have the largest number of (short) segments in order to justify\nthe normality assumption and to maximize power. The test requires that the errors\nin one segment be statistically independent of the errors in another segment. Since\nASR systems tend to use trigram LMs, we can approximate this requirement by\nde\ufb01ning a segment as a region bounded on both sides by words that both recognizers\nget correct (or by turn/utterance boundaries). Here\u2019s an example from NIST (2007)\nwith four regions:\nI II III IV\nREF: |it was|the best|of|times it|was the worst|of times| |it was\n| | | | | | | |\nSYS A:|ITS |the best|of|times it|IS the worst |of times|OR|it was\n| | | | | | | |\nSYS B:|it was|the best| |times it|WON the TEST |of times| |it was\nIn region I, system A has two errors (a deletion and an insertion) and system B\nhas zero; in region III, system A has one error (a substitution) and system B has two.\nLet\u2019s de\ufb01ne a sequence of variables Zrepresenting the difference between the errors\nin the two systems as follows:\nNi\nAthe number of errors made on segment iby system A\nNi\nB the number of errors made on segment iby system B\nZ Ni\nA\u0000Ni\nB;i=1;2;\u0001\u0001\u0001;nwhere nis the number of segments\nIn the example above, the sequence of Zvalues isf2;\u00001;\u00001;1g. Intuitively, if\nthe two systems are identical, we would expect the average difference, that is, the\naverage of the Zvalues, to be zero. If we call the true average of the differences\nmuz, we would thus like to know whether muz=0. Following closely the original\nproposal and notation of Gillick and Cox (1989), we can estimate the true average\nfrom our limited sample as \u02c6mz=Pn\ni=1Zi=n. The estimate of the variance of the Zi\u2019s\nis\ns2\nz=1\nn\u00001nX\ni=1(Zi\u0000mz)2(16.22) 18 CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\nLet\nW=\u02c6mz\nsz=pn(16.23)\nFor a large enough n(>50),Wwill approximately have a normal distribution with\nunit variance. The null hypothesis is H0:mz=0, and it can thus be rejected if\n2\u0003P(Z\u0015jwj)\u00140:05 (two-tailed) or P(Z\u0015jwj)\u00140:05 (one-tailed), where Zis\nstandard normal and wis the realized value W; these probabilities can be looked up\nin the standard tables of the normal distribution.\nEarlier work sometimes used McNemar\u2019s test for signi\ufb01cance, but McNemar\u2019s McNemar\u2019s test\nis only applicable when the errors made by the system are independent, which is not\ntrue in continuous speech recognition, where errors made on a word are extremely\ndependent on errors made on neighboring words.\nCould we improve on word error rate as a metric? It would be nice, for exam-\nple, to have something that didn\u2019t give equal weight to every word, perhaps valuing\ncontent words like Tuesday more than function words like aorof. While researchers\ngenerally agree that this would be a good idea, it has proved dif\ufb01cult to agree on\na metric that works in every application of ASR. For dialogue systems, however,\nwhere the desired semantic output is more clear, a metric called slot error rate or\nconcept error rate has proved extremely useful; it is discussed in Chapter 15 on page\n??.\n16.6 TTS\nThe goal of text-to-speech (TTS) systems is to map from strings of letters to wave-\nforms, a technology that\u2019s important for a variety of applications from dialogue sys-\ntems to games to education.\nLike ASR systems, TTS systems are generally based on the encoder-decoder\narchitecture, either using LSTMs or Transformers. There is a general difference in\ntraining. The default condition for ASR systems is to be speaker-independent: they\nare trained on large corpora with thousands of hours of speech from many speakers\nbecause they must generalize well to an unseen test speaker. By contrast, in TTS, it\u2019s\nless crucial to use multiple voices, and so basic TTS systems are speaker-dependent:\ntrained to have a consistent voice, on much less data, but all from one speaker. For\nexample, one commonly used public domain dataset, the LJ speech corpus, consists\nof 24 hours of one speaker, Linda Johnson, reading audio books in the LibriV ox\nproject (Ito and Johnson, 2017), much smaller than standard ASR corpora which are\nhundreds or thousands of hours.2\nWe generally break up the TTS task into two components. The \ufb01rst component\nis an encoder-decoder model for spectrogram prediction : it maps from strings of\nletters to mel spectrographs: sequences of mel spectral values over time. Thus we\n2There is also recent TTS research on the task of multi-speaker TTS, in which a system is trained on\nspeech from many speakers, and can switch between different voices. 16.6 \u2022 TTS 19\nmight map from this string:\nIt's time for lunch!\nto the following mel spectrogram:\nThe second component maps from mel spectrograms to waveforms. Generating\nwaveforms from intermediate representations like spectrograms is called vocoding vocoding\nand this second component is called a vocoder : vocoder\nThese standard encoder-decoder algorithms for TTS are still quite computation-\nally intensive, so a signi\ufb01cant focus of modern research is on ways to speed them\nup.\n16.6.1 TTS Preprocessing: Text normalization\nBefore either of these two steps, however, TTS systems require text normaliza-\ntion preprocessing for handling non-standard words : numbers, monetary amounts,non-standard\nwords\ndates, and other concepts that are verbalized differently than they are spelled. A TTS\nsystem seeing a number like 151needs to know to verbalize it as one hundred \ufb01fty\noneif it occurs as $151 but as one \ufb01fty one if it occurs in the context 151 Chapulte-\npec Ave. . The number 1750 can be spoken in at least four different ways, depending\non the context:\nseventeen fifty: (in\u201cThe European economy in 1750\u201d )\none seven five zero: (in\u201cThe password is 1750\u201d )\nseventeen hundred and fifty: (in\u201c1750 dollars\u201d )\none thousand, seven hundred, and fifty: (in\u201c1750 dollars\u201d )\nOften the verbalization of a non-standard word depends on its meaning (what\nTaylor (2009) calls its semiotic class ). Fig. 16.15 lays out some English non-\nstandard word types.\nMany classes have preferred realizations. A year is generally read as paired\ndigits (e.g., seventeen fifty for 1750). $3.2 billion must be read out with the\nword dollars at the end, as three point two billion dollars . Some ab-\nbreviations like N.Y. are expanded (to New York ), while other acronyms like GPU\nare pronounced as letter sequences. In languages with grammatical gender, normal-\nization may depend on morphological properties. In French, the phrase 1 mangue\n(\u2018one mangue\u2019) is normalized to une mangue , but 1 ananas (\u2018one pineapple\u2019) is\nnormalized to un ananas . In German, Heinrich IV (\u2018Henry IV\u2019) can be normalized\ntoHeinrich der Vierte ,Heinrich des Vierten ,Heinrich dem Vierten , or\nHeinrich den Vierten depending on the grammatical case of the noun (Demberg,\n2006). 20 CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\nsemiotic class examples verbalization\nabbreviations gov\u2019t , N.Y., mph government\nacronyms read as letters GPU , D.C., PC, UN, IBM G P U\ncardinal numbers 12, 45, 1/2, 0.6 twelve\nordinal numbers May 7, 3rd, Bill Gates III seventh\nnumbers read as digits Room 101 one oh one\ntimes 3.20, 11:45 eleven forty \ufb01ve\ndates 28/02 (or in US, 2/28) February twenty eighth\nyears 1999 , 80s, 1900s, 2045 nineteen ninety nine\nmoney $3.45 ,e250, $200K three dollars forty \ufb01ve\nmoney in tr/m/billions $3.45 billion three point four \ufb01ve billion dollars\npercentage 75% 3.4% seventy \ufb01ve percent\nFigure 16.15 Some types of non-standard words in text normalization; see Sproat et al.\n(2001) and (van Esch and Sproat, 2018) for many more.\nModern end-to-end TTS systems can learn to do some normalization themselves,\nbut TTS systems are only trained on a limited amount of data (like the 220,000 words\nwe mentioned above for the LJ corpus (Ito and Johnson, 2017)), and so a separate\nnormalization step is important.\nNormalization can be done by rule or by an encoder-decoder model. Rule-based\nnormalization is done in two stages: tokenization and verbalization. In the tokeniza-\ntion stage we hand-write rules to detect non-standard words. These can be regular\nexpressions, like the following for detecting years:\n/(1[89][0-9][0-9])j(20[0-9][0-9]/\nA second pass of rules express how to verbalize each semiotic class. Larger TTS\nsystems instead use more complex rule-systems, like the Kestral system of (Ebden\nand Sproat, 2015), which \ufb01rst classi\ufb01es and parses each input into a normal form\nand then produces text using a verbalization grammar. Rules have the advantage\nthat they don\u2019t require training data, and they can be designed for high precision, but\ncan be brittle, and require expert rule-writers so are hard to maintain.\nThe alternative model is to use encoder-decoder models, which have been shown\nto work better than rules for such transduction tasks, but do require expert-labeled\ntraining sets in which non-standard words have been replaced with the appropriate\nverbalization; such training sets for some languages are available (Sproat and Gor-\nman 2018, Zhang et al. 2019).\nIn the simplest encoder-decoder setting, we simply treat the problem like ma-\nchine translation, training a system to map from:\nThey live at 224 Mission St.\nto\nThey live at two twenty four Mission Street\nWhile encoder-decoder algorithms are highly accurate, they occasionally pro-\nduce errors that are egregious; for example normalizing 45 minutes asforty \ufb01ve mil-\nlimeters . To address this, more complex systems use mechanisms like lightweight\ncovering grammars , which enumerate a large set of possible verbalizations but\ndon\u2019t try to disambiguate, to constrain the decoding to avoid such outputs (Zhang\net al., 2019).\n16.6.2 TTS: Spectrogram prediction\nThe exact same architecture we described for ASR\u2014the encoder-decoder with attention\u2013\ncan be used for the \ufb01rst component of TTS. Here we\u2019ll give a simpli\ufb01ed overview 16.6 \u2022 TTS 21\nof the Tacotron2 architecture (Shen et al., 2018), which extends the earlier Tacotron Tacotron2\n(Wang et al., 2017) architecture and the Wavenet vocoder (van den Oord et al., Wavenet\n2016). Fig. 16.16 sketches out the entire architecture.\nThe encoder\u2019s job is to take a sequence of letters and produce a hidden repre-\nsentation representing the letter sequence, which is then used by the attention mech-\nanism in the decoder. The Tacotron2 encoder \ufb01rst maps every input grapheme to\na 512-dimensional character embedding. These are then passed through a stack\nof 3 convolutional layers, each containing 512 \ufb01lters with shape 5 \u00021, i.e. each\n\ufb01lter spanning 5 characters, to model the larger letter context. The output of the\n\ufb01nal convolutional layer is passed through a biLSTM to produce the \ufb01nal encod-\ning. It\u2019s common to use a slightly higher quality (but slower) version of attention\ncalled location-based attention , in which the computation of the avalues (Eq. ??location-based\nattention\nin Chapter 8) makes use of the avalues from the prior time-state.\nIn the decoder, the predicted mel spectrum from the prior time slot is passed\nthrough a small pre-net as a bottleneck. This prior output is then concatenated with\nthe encoder\u2019s attention vector context and passed through 2 LSTM layers. The out-\nput of this LSTM is used in two ways. First, it is passed through a linear layer, and\nsome output processing, to autoregressively predict one 80-dimensional log-mel \ufb01l-\nterbank vector frame (50 ms, with a 12.5 ms stride) at each step. Second, it is passed\nthrough another linear layer to a sigmoid to make a \u201cstop token prediction\u201d decision\nabout whether to stop producing output.\nWhile linear spectrograms discard phase information (and aretherefore lossy), algorithms such as Grif\ufb01n-Lim [14] are capable ofestimating this discarded information, which enables time-domainconversion via the inverse short-time Fourier transform. Mel spectro-grams discard even more information, presenting a challenging in-verse problem. However, in comparison to the linguistic and acousticfeatures used in WaveNet, the mel spectrogram is a simpler, lower-level acoustic representation of audio signals. It should thereforebe straightforward for a similar WaveNet model conditioned on melspectrograms to generate audio, essentially as a neural vocoder. In-deed, we will show that it is possible to generate high quality audiofrom mel spectrograms using a modi\ufb01ed WaveNet architecture.2.2. Spectrogram Prediction NetworkAs in Tacotron, mel spectrograms are computed through a short-time Fourier transform (STFT) using a 50 ms frame size, 12.5 msframe hop, and a Hann window function. We experimented with a5 ms frame hop to match the frequency of the conditioning inputsin the original WaveNet, but the corresponding increase in temporalresolution resulted in signi\ufb01cantly more pronunciation issues.We transform the STFT magnitude to the mel scale using an 80channel mel \ufb01lterbank spanning 125 Hz to 7.6 kHz, followed by logdynamic range compression. Prior to log compression, the \ufb01lterbankoutput magnitudes are clipped to a minimum value of 0.01 in orderto limit dynamic range in the logarithmic domain.The network is composed of an encoder and a decoder with atten-tion. The encoder converts a character sequence into a hidden featurerepresentation which the decoder consumes to predict a spectrogram.Input characters are represented using a learned 512-dimensionalcharacter embedding, which are passed through a stack of 3 convolu-tional layers each containing 512 \ufb01lters with shape5\u21e51, i.e., whereeach \ufb01lter spans 5 characters, followed by batch normalization [18]and ReLU activations. As in Tacotron, these convolutional layersmodel longer-term context (e.g.,N-grams) in the input charactersequence. The output of the \ufb01nal convolutional layer is passed into asingle bi-directional [19] LSTM [20] layer containing 512 units (256in each direction) to generate the encoded features.The encoder output is consumed by an attention network whichsummarizes the full encoded sequence as a \ufb01xed-length context vectorfor each decoder output step. We use the location-sensitive attentionfrom [21], which extends the additive attention mechanism [22] touse cumulative attention weights from previous decoder time stepsas an additional feature. This encourages the model to move forwardconsistently through the input, mitigating potential failure modeswhere some subsequences are repeated or ignored by the decoder.Attention probabilities are computed after projecting inputs and lo-cation features to 128-dimensional hidden representations. Locationfeatures are computed using 32 1-D convolution \ufb01lters of length 31.The decoder is an autoregressive recurrent neural network whichpredicts a mel spectrogram from the encoded input sequence oneframe at a time. The prediction from the previous time step is \ufb01rstpassed through a smallpre-netcontaining 2 fully connected layersof 256 hidden ReLU units. We found that the pre-net acting as aninformation bottleneck was essential for learning attention. The pre-net output and attention context vector are concatenated and passedthrough a stack of 2 uni-directional LSTM layers with 1024 units.The concatenation of the LSTM output and the attention contextvector is projected through a linear transform to predict the targetspectrogram frame. Finally, the predicted mel spectrogram is passedthrough a 5-layer convolutionalpost-netwhich predicts a residualto add to the prediction to improve the overall reconstruction. Each\n'LEVEGXIV\u0004)QFIHHMRK0SGEXMSR\u00047IRWMXMZI\u0004%XXIRXMSR\u0017\u0004'SRZ\u00040E]IVW&MHMVIGXMSREP\u00040781-RTYX\u00048I\\X\u0016\u00040E]IV\u00044VI\u00112IX\u0016\u00040781\u00040E]IVW0MRIEV\u00044VSNIGXMSR0MRIEV\u00044VSNIGXMSR7XST\u00048SOIR\u0019\u0004'SRZ\u00040E]IV\u00044SWX\u00112IX\n0HO\u00036SHFWURJUDP;EZI2IX\u00041S0;EZIJSVQ\u00047EQTPIWFig. 1. Block diagram of the Tacotron 2 system architecture.post-net layer is comprised of 512 \ufb01lters with shape5\u21e51with batchnormalization, followed bytanhactivations on all but the \ufb01nal layer.We minimize the summed mean squared error (MSE) from beforeand after the post-net to aid convergence. We also experimentedwith a log-likelihood loss by modeling the output distribution witha Mixture Density Network [23,24] to avoid assuming a constantvariance over time, but found that these were more dif\ufb01cult to trainand they did not lead to better sounding samples.In parallel to spectrogram frame prediction, the concatenation ofdecoder LSTM output and the attention context is projected downto a scalar and passed through a sigmoid activation to predict theprobability that the output sequence has completed. This \u201cstop token\u201dprediction is used during inference to allow the model to dynamicallydetermine when to terminate generation instead of always generatingfor a \ufb01xed duration. Speci\ufb01cally, generation completes at the \ufb01rstframe for which this probability exceeds a threshold of 0.5.The convolutional layers in the network are regularized usingdropout [25] with probability 0.5, and LSTM layers are regularizedusing zoneout [26] with probability 0.1. In order to introduce outputvariation at inference time, dropout with probability 0.5 is appliedonly to layers in the pre-net of the autoregressive decoder.In contrast to the original Tacotron, our model uses simpler build-ing blocks, using vanilla LSTM and convolutional layers in the en-coder and decoder instead of \u201cCBHG\u201d stacks and GRU recurrentlayers. We do not use a \u201creduction factor\u201d, i.e., each decoder stepcorresponds to a single spectrogram frame.2.3. WaveNet VocoderWe use a modi\ufb01ed version of the WaveNet architecture from [8] toinvert the mel spectrogram feature representation into time-domainwaveform samples. As in the original architecture, there are 30dilated convolution layers, grouped into 3 dilation cycles, i.e., thedilation rate of layer k (k=0...29) is2k(mod 10). To work withthe 12.5 ms frame hop of the spectrogram frames, only 2 upsamplinglayers are used in the conditioning stack instead of 3 layers.Instead of predicting discretized buckets with a softmax layer,we follow PixelCNN++ [27] and Parallel WaveNet [28] and use a 10-component mixture of logistic distributions (MoL) to generate 16-bitsamples at 24 kHz. To compute the logistic mixture distribution, theWaveNet stack output is passed through a ReLU activation followedEncoderDecoderVocoder\nFigure 16.16 The Tacotron2 architecture: An encoder-decoder maps from graphemes to\nmel spectrograms, followed by a vocoder that maps to wave\ufb01les. Figure modi\ufb01ed from Shen\net al. (2018).\nThe system is trained on gold log-mel \ufb01lterbank features, using teacher forcing,\nthat is the decoder is fed the correct log-model spectral feature at each decoder step\ninstead of the predicted decoder output from the prior step.\n16.6.3 TTS: Vocoding\nThe vocoder for Tacotron 2 is an adaptation of the WaveNet vocoder (van den Oord WaveNet\net al., 2016). Here we\u2019ll give a somewhat simpli\ufb01ed description of vocoding using\nWaveNet.\nRecall that the goal of the vocoding process here will be to invert a log mel spec-\ntrum representations back into a time-domain waveform representation. WaveNet is\nan autoregressive network, like the language models we introduced in Chapter 8. It 22 CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\ntakes spectrograms as input and produces audio output represented as sequences of\n8-bit mu-law (page 6). The probability of a waveform , a sequence of 8-bit mu-law\nvalues Y=y1;:::;yt, given an intermediate input mel spectrogram his computed as:\np(Y) =tY\nt=1P(ytjy1;:::;yt\u00001;h1;:::;ht) (16.24)\nThis probability distribution is modeled by a stack of special convolution layers,\nwhich include a speci\ufb01c convolutional structure called dilated convolutions , and a\nspeci\ufb01c non-linearity function.\nA dilated convolution is a subtype of causal convolutional layer. Causal or\nmasked convolutions look only at the past input, rather than the future; the pre-\ndiction of yt+1can only depend on y1;:::;yt, useful for autoregressive left-to-right\nprocessing. In dilated convolutions , at each successive layer we apply the convolu-dilated\nconvolutions\ntional \ufb01lter over a span longer than its length by skipping input values. Thus at time\ntwith a dilation value of 1, a convolutional \ufb01lter of length 2 would see input values\nxtandxt\u00001. But a \ufb01lter with a distillation value of 2 would skip an input, so would\nsee input values xtandxt\u00001. Fig. 16.17 shows the computation of the output at time\ntwith 4 dilated convolution layers with dilation values, 1, 2, 4, and 8.\nBecause models with causal convolutions do not have recurrent connections, they are typically faster\nto train than RNNs, especially when applied to very long sequences. One of the problems of causal\nconvolutions is that they require many layers, or large \ufb01lters to increase the receptive \ufb01eld. For\nexample, in Fig. 2 the receptive \ufb01eld is only 5 (= #layers + \ufb01lter length - 1). In this paper we use\ndilated convolutions to increase the receptive \ufb01eld by orders of magnitude, without greatly increasing\ncomputational cost.\nA dilated convolution (also called `a trous , or convolution with holes) is a convolution where the\n\ufb01lter is applied over an area larger than its length by skipping input values with a certain step. It is\nequivalent to a convolution with a larger \ufb01lter derived from the original \ufb01lter by dilating it with zeros,\nbut is signi\ufb01cantly more ef\ufb01cient. A dilated convolution effectively allows the network to operate on\na coarser scale than with a normal convolution. This is similar to pooling or strided convolutions, but\nhere the output has the same size as the input. As a special case, dilated convolution with dilation\n1yields the standard convolution. Fig. 3 depicts dilated causal convolutions for dilations 1,2,4,\nand8. Dilated convolutions have previously been used in various contexts, e.g. signal processing\n(Holschneider et al., 1989; Dutilleux, 1989), and image segmentation (Chen et al., 2015; Yu &\nKoltun, 2016).\nInputHidden LayerDilation = 1Hidden LayerDilation = 2Hidden LayerDilation = 4OutputDilation = 8\nFigure 3: Visualization of a stack of dilated causal convolutional layers.\nStacked dilated convolutions enable networks to have very large receptive \ufb01elds with just a few lay-\ners, while preserving the input resolution throughout the network as well as computational ef\ufb01ciency.\nIn this paper, the dilation is doubled for every layer up to a limit and then repeated: e.g.\n1,2,4,..., 512,1,2,4,..., 512,1,2,4,..., 512.\nThe intuition behind this con\ufb01guration is two-fold. First, exponentially increasing the dilation factor\nresults in exponential receptive \ufb01eld growth with depth (Yu & Koltun, 2016). For example each\n1,2,4,..., 512block has receptive \ufb01eld of size 1024 , and can be seen as a more ef\ufb01cient and dis-\ncriminative (non-linear) counterpart of a 1\u21e51024 convolution. Second, stacking these blocks further\nincreases the model capacity and the receptive \ufb01eld size.\n2.2 S OFTMAX DISTRIBUTIONS\nOne approach to modeling the conditional distributions p(xt|x1,...,x t\u00001)over the individual\naudio samples would be to use a mixture model such as a mixture density network (Bishop, 1994)\nor mixture of conditional Gaussian scale mixtures (MCGSM) (Theis & Bethge, 2015). However,\nvan den Oord et al. (2016a) showed that a softmax distribution tends to work better, even when the\ndata is implicitly continuous (as is the case for image pixel intensities or audio sample values). One\nof the reasons is that a categorical distribution is more \ufb02exible and can more easily model arbitrary\ndistributions because it makes no assumptions about their shape.\nBecause raw audio is typically stored as a sequence of 16-bit integer values (one per timestep), a\nsoftmax layer would need to output 65,536 probabilities per timestep to model all possible values.\nTo make this more tractable, we \ufb01rst apply a \u00b5-law companding transformation (ITU-T, 1988) to\nthe data, and then quantize it to 256 possible values:\nf(xt) = sign( xt)ln (1 + \u00b5|xt|)\nln (1 + \u00b5),\n3\nFigure 16.17 Dilated convolutions, showing one dilation cycle size of 4, i.e., dilation values\nof 1, 2, 4, 8. Figure from van den Oord et al. (2016).\nThe Tacotron 2 synthesizer uses 12 convolutional layers in two cycles with a\ndilation cycle size of 6, meaning that the \ufb01rst 6 layers have dilations of 1, 2, 4, 8, 16,\nand 32. and the next 6 layers again have dilations of 1, 2, 4, 8, 16, and 32. Dilated\nconvolutions allow the vocoder to grow the receptive \ufb01eld exponentially with depth.\nWaveNet predicts mu-law audio samples. Recall from page 6 that this is a stan-\ndard compression for audio in which the values at each sampling timestep are com-\npressed into 8-bits. This means that we can predict the value of each sample with a\nsimple 256-way categorical classi\ufb01er. The output of the dilated convolutions is thus\npassed through a softmax which makes this 256-way decision.\nThe spectrogram prediction encoder-decoder and the WaveNet vocoder are trained\nseparately. After the spectrogram predictor is trained, the spectrogram prediction\nnetwork is run in teacher-forcing mode, with each predicted spectral frame condi-\ntioned on the encoded text input and the previous frame from the ground truth spec-\ntrogram. This sequence of ground truth-aligned spectral features and gold audio\noutput is then used to train the vocoder.\nThis has been only a high-level sketch of the TTS process. There are numer-\nous important details that the reader interested in going further with TTS may want 16.7 \u2022 O THER SPEECH TASKS 23\nto look into. For example WaveNet uses a special kind of a gated activation func-\ntion as its non-linearity, and contains residual and skip connections. In practice,\npredicting 8-bit audio values doesn\u2019t as work as well as 16-bit, for which a simple\nsoftmax is insuf\ufb01cient, so decoders use fancier ways as the last step of predicting\naudio sample values, like mixtures of distributions. Finally, the WaveNet vocoder\nas we have described it would be so slow as to be useless; many different kinds of\nef\ufb01ciency improvements are necessary in practice, for example by \ufb01nding ways to\ndo non-autoregressive generation, avoiding the latency of having to wait to generate\neach frame until the prior frame has been generated, and instead making predictions\nin parallel. We encourage the interested reader to consult the original papers and\nvarious version of the code.\n16.6.4 TTS Evaluation\nSpeech synthesis systems are evaluated by human listeners. (The development of a\ngood automatic metric for synthesis evaluation, one that would eliminate the need\nfor expensive and time-consuming human listening experiments, remains an open\nand exciting research topic.)\nWe evaluate the quality of synthesized utterances by playing a sentence to lis-\nteners and ask them to give a mean opinion score (MOS ), a rating of how good MOS\nthe synthesized utterances are, usually on a scale from 1\u20135. We can then compare\nsystems by comparing their MOS scores on the same sentences (using, e.g., paired\nt-tests to test for signi\ufb01cant differences).\nIf we are comparing exactly two systems (perhaps to see if a particular change\nactually improved the system), we can use AB tests . In AB tests, we play the same AB tests\nsentence synthesized by two different systems (an A and a B system). The human\nlisteners choose which of the two utterances they like better. We do this for say\n50 sentences (presented in random order) and compare the number of sentences\npreferred for each system.\n16.7 Other Speech Tasks\nWhile we have focused on speech recognition and TTS in this chapter, there are a\nwide variety of speech-related tasks.\nThe task of wake word detection is to detect a word or short phrase, usually in wake word\norder to wake up a voice-enable assistant like Alexa, Siri, or the Google Assistant.\nThe goal with wake words is build the detection into small devices at the computing\nedge, to maintain privacy by transmitting the least amount of user speech to a cloud-\nbased server. Thus wake word detectors need to be fast, small footprint software that\ncan \ufb01t into embedded devices. Wake word detectors usually use the same frontend\nfeature extraction we saw for ASR, often followed by a whole-word classi\ufb01er.\nSpeaker diarization is the task of determining \u2018who spoke when\u2019 in a longspeaker\ndiarization\nmulti-speaker audio recording, marking the start and end of each speaker\u2019s turns in\nthe interaction. This can be useful for transcribing meetings, classroom speech, or\nmedical interactions. Often diarization systems use voice activity detection (V AD) to\n\ufb01nd segments of continuous speech, extract speaker embedding vectors, and cluster\nthe vectors to group together segments likely from the same speaker. More recent\nwork is investigating end-to-end algorithms to map directly from input speech to a\nsequence of speaker labels for each frame. 24 CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\nSpeaker recognition , is the task of identifying a speaker. We generally distin-speaker\nrecognition\nguish the subtasks of speaker veri\ufb01cation , where we make a binary decision (is\nthis speaker Xor not?), such as for security when accessing personal information\nover the telephone, and speaker identi\ufb01cation , where we make a one of Ndecision\ntrying to match a speaker\u2019s voice against a database of many speakers . These tasks\nare related to language identi\ufb01cation , in which we are given a wave\ufb01le and mustlanguage\nidenti\ufb01cation\nidentify which language is being spoken; this is useful for example for automatically\ndirecting callers to human operators that speak appropriate languages.\n16.8 Summary\nThis chapter introduced the fundamental algorithms of automatic speech recognition\n(ASR) and text-to-speech (TTS).\n\u2022 The task of speech recognition (or speech-to-text) is to map acoustic wave-\nforms to sequences of graphemes.\n\u2022 The input to a speech recognizer is a series of acoustic waves. that are sam-\npled,quantized , and converted to a spectral representation like the log mel\nspectrum .\n\u2022 Two common paradigms for speech recognition are the encoder-decoder with\nattention model, and models based on the CTC loss function . Attention-\nbased models have higher accuracies, but models based on CTC more easily\nadapt to streaming : outputting graphemes online instead of waiting until the\nacoustic input is complete.\n\u2022 ASR is evaluated using the Word Error Rate; the edit distance between the\nhypothesis and the gold transcription.\n\u2022TTS systems are also based on the encoder-decoder architecture. The en-\ncoder maps letters to an encoding, which is consumed by the decoder which\ngenerates mel spectrogram output. A neural vocoder then reads the spectro-\ngram and generates waveforms.\n\u2022 TTS systems require a \ufb01rst pass of text normalization to deal with numbers\nand abbreviations and other non-standard words.\n\u2022 TTS is evaluated by playing a sentence to human listeners and having them\ngive a mean opinion score (MOS) or by doing AB tests.\nBibliographical and Historical Notes\nASR A number of speech recognition systems were developed by the late 1940s\nand early 1950s. An early Bell Labs system could recognize any of the 10 digits\nfrom a single speaker (Davis et al., 1952). This system had 10 speaker-dependent\nstored patterns, one for each digit, each of which roughly represented the \ufb01rst two\nvowel formants in the digit. They achieved 97%\u201399% accuracy by choosing the pat-\ntern that had the highest relative correlation coef\ufb01cient with the input. Fry (1959)\nand Denes (1959) built a phoneme recognizer at University College, London, that\nrecognized four vowels and nine consonants based on a similar pattern-recognition\nprinciple. Fry and Denes\u2019s system was the \ufb01rst to use phoneme transition probabili-\nties to constrain the recognizer. BIBLIOGRAPHICAL AND HISTORICAL NOTES 25\nThe late 1960s and early 1970s produced a number of important paradigm shifts.\nFirst were a number of feature-extraction algorithms, including the ef\ufb01cient fast\nFourier transform (FFT) (Cooley and Tukey, 1965), the application of cepstral pro-\ncessing to speech (Oppenheim et al., 1968), and the development of LPC for speech\ncoding (Atal and Hanauer, 1971). Second were a number of ways of handling warp-\ning; stretching or shrinking the input signal to handle differences in speaking rate warping\nand segment length when matching against stored patterns. The natural algorithm for\nsolving this problem was dynamic programming, and, as we saw in Appendix A, the\nalgorithm was reinvented multiple times to address this problem. The \ufb01rst applica-\ntion to speech processing was by Vintsyuk (1968), although his result was not picked\nup by other researchers, and was reinvented by Velichko and Zagoruyko (1970) and\nSakoe and Chiba (1971) (and 1984). Soon afterward, Itakura (1975) combined this\ndynamic programming idea with the LPC coef\ufb01cients that had previously been used\nonly for speech coding. The resulting system extracted LPC features from incoming\nwords and used dynamic programming to match them against stored LPC templates.\nThe non-probabilistic use of dynamic programming to match a template against in-\ncoming speech is called dynamic time warping .dynamic time\nwarping\nThe third innovation of this period was the rise of the HMM. Hidden Markov\nmodels seem to have been applied to speech independently at two laboratories around\n1972. One application arose from the work of statisticians, in particular Baum and\ncolleagues at the Institute for Defense Analyses in Princeton who applied HMMs\nto various prediction problems (Baum and Petrie 1966, Baum and Eagon 1967).\nJames Baker learned of this work and applied the algorithm to speech processing\n(Baker, 1975) during his graduate work at CMU. Independently, Frederick Jelinek\nand collaborators (drawing from their research in information-theoretical models\nin\ufb02uenced by the work of Shannon (1948)) applied HMMs to speech at the IBM\nThomas J. Watson Research Center (Jelinek et al., 1975). One early difference was\nthe decoding algorithm; Baker\u2019s DRAGON system used Viterbi (dynamic program-\nming) decoding, while the IBM system applied Jelinek\u2019s stack decoding algorithm\n(Jelinek, 1969). Baker then joined the IBM group for a brief time before founding\nthe speech-recognition company Dragon Systems.\nThe use of the HMM, with Gaussian Mixture Models (GMMs) as the phonetic\ncomponent, slowly spread through the speech community, becoming the dominant\nparadigm by the 1990s. One cause was encouragement by ARPA, the Advanced\nResearch Projects Agency of the U.S. Department of Defense. ARPA started a\n\ufb01ve-year program in 1971 to build 1000-word, constrained grammar, few speaker\nspeech understanding (Klatt, 1977), and funded four competing systems of which\nCarnegie-Mellon University\u2019s Harpy system (Lowerre, 1976), which used a simpli-\n\ufb01ed version of Baker\u2019s HMM-based DRAGON system was the best of the tested sys-\ntems. ARPA (and then DARPA) funded a number of new speech research programs,\nbeginning with 1000-word speaker-independent read-speech tasks like \u201cResource\nManagement\u201d (Price et al., 1988), recognition of sentences read from the Wall Street\nJournal (WSJ), Broadcast News domain (LDC 1998, Graff 1997) (transcription of\nactual news broadcasts, including quite dif\ufb01cult passages such as on-the-street inter-\nviews) and the Switchboard, CallHome, CallFriend, and Fisher domains (Godfrey\net al. 1992, Cieri et al. 2004) (natural telephone conversations between friends or\nstrangers). Each of the ARPA tasks involved an approximately annual bakeoff at bakeoff\nwhich systems were evaluated against each other. The ARPA competitions resulted\nin wide-scale borrowing of techniques among labs since it was easy to see which\nideas reduced errors the previous year, and the competitions were probably an im- 26 CHAPTER 16 \u2022 A UTOMATIC SPEECH RECOGNITION AND TEXT-TO-SPEECH\nportant factor in the eventual spread of the HMM paradigm.\nBy around 1990 neural alternatives to the HMM/GMM architecture for ASR\narose, based on a number of earlier experiments with neural networks for phoneme\nrecognition and other speech tasks. Architectures included the time-delay neural\nnetwork ( TDNN )\u2014the \ufb01rst use of convolutional networks for speech\u2014 (Waibel\net al. 1989, Lang et al. 1990), RNNs (Robinson and Fallside, 1991), and the hybrid hybrid\nHMM/MLP architecture in which a feedforward neural network is trained as a pho-\nnetic classi\ufb01er whose outputs are used as probability estimates for an HMM-based\narchitecture (Morgan and Bourlard 1990, Bourlard and Morgan 1994, Morgan and\nBourlard 1995).\nWhile the hybrid systems showed performance close to the standard HMM/GMM\nmodels, the problem was speed: large hybrid models were too slow to train on the\nCPUs of that era. For example, the largest hybrid system, a feedforward network,\nwas limited to a hidden layer of 4000 units, producing probabilities over only a few\ndozen monophones. Yet training this model still required the research group to de-\nsign special hardware boards to do vector processing (Morgan and Bourlard, 1995).\nA later analytic study showed the performance of such simple feedforward MLPs\nfor ASR increases sharply with more than 1 hidden layer, even controlling for the\ntotal number of parameters (Maas et al., 2017). But the computational resources of\nthe time were insuf\ufb01cient for more layers.\nOver the next two decades a combination of Moore\u2019s law and the rise of GPUs\nallowed deep neural networks with many layers. Performance was getting close to\ntraditional systems on smaller tasks like TIMIT phone recognition by 2009 (Mo-\nhamed et al., 2009), and by 2012, the performance of hybrid systems had surpassed\ntraditional HMM/GMM systems (Jaitly et al. 2012, Dahl et al. 2012, inter alia).\nOriginally it seemed that unsupervised pretraining of the networks using a tech-\nnique like deep belief networks was important, but by 2013, it was clear that for\nhybrid HMM/GMM feedforward networks, all that mattered was to use a lot of data\nand enough layers, although a few other components did improve performance: us-\ning log mel features instead of MFCCs, using dropout, and using recti\ufb01ed linear\nunits (Deng et al. 2013, Maas et al. 2013, Dahl et al. 2013).\nMeanwhile early work had proposed the CTC loss function by 2006 (Graves\net al., 2006), and by 2012 the RNN-Transducer was de\ufb01ned and applied to phone\nrecognition (Graves 2012, Graves et al. 2013), and then to end-to-end speech recog-\nnition rescoring (Graves and Jaitly, 2014), and then recognition (Maas et al., 2015),\nwith advances such as specialized beam search (Hannun et al., 2014). (Our de-\nscription of CTC in the chapter draws on Hannun (2017), which we encourage the\ninterested reader to follow).\nThe encoder-decoder architecture was applied to speech at about the same time\nby two different groups, in the Listen Attend and Spell system of Chan et al. (2016)\nand the attention-based encoder decoder architecture of Chorowski et al. (2014)\nand Bahdanau et al. (2016). By 2018 Transformers were included in this encoder-\ndecoder architecture. Karita et al. (2019) is a nice comparison of RNNs vs Trans-\nformers in encoder-architectures for ASR, TTS, and speech-to-speech translation.\nPopular toolkits for speech processing include Kaldi (Povey et al., 2011) and Kaldi\nESPnet (Watanabe et al. 2018, Hayashi et al. 2020). ESPnet\nTTS As we noted at the beginning of the chapter, speech synthesis is one of the\nearliest \ufb01elds of speech and language processing. The 18th century saw a number\nof physical models of the articulation process, including the von Kempelen model\nmentioned above, as well as the 1773 vowel model of Kratzenstein in Copenhagen EXERCISES 27\nusing organ pipes.\nThe early 1950s saw the development of three early paradigms of waveform\nsynthesis: formant synthesis, articulatory synthesis, and concatenative synthesis.\nModern encoder-decoder systems are distant descendants of formant synthesiz-\ners. Formant synthesizers originally were inspired by attempts to mimic human\nspeech by generating arti\ufb01cial spectrograms. The Haskins Laboratories Pattern\nPlayback Machine generated a sound wave by painting spectrogram patterns on a\nmoving transparent belt and using re\ufb02ectance to \ufb01lter the harmonics of a wave-\nform (Cooper et al., 1951); other very early formant synthesizers include those of\nLawrence (1953) and Fant (1951). Perhaps the most well-known of the formant\nsynthesizers were the Klatt formant synthesizer and its successor systems, includ-\ning the MITalk system (Allen et al., 1987) and the Klattalk software used in Digital\nEquipment Corporation\u2019s DECtalk (Klatt, 1982). See Klatt (1975) for details.\nA second early paradigm, concatenative synthesis, seems to have been \ufb01rst pro-\nposed by Harris (1953) at Bell Laboratories; he literally spliced together pieces of\nmagnetic tape corresponding to phones. Soon afterwards, Peterson et al. (1958) pro-\nposed a theoretical model based on diphones, including a database with multiple\ncopies of each diphone with differing prosody, each labeled with prosodic features\nincluding F0, stress, and duration, and the use of join costs based on F0 and formant\ndistance between neighboring units. But such diphone synthesis models were not\nactually implemented until decades later (Dixon and Maxey 1968, Olive 1977). The\n1980s and 1990s saw the invention of unit selection synthesis , based on larger units\nof non-uniform length and the use of a target cost, (Sagisaka 1988, Sagisaka et al.\n1992, Hunt and Black 1996, Black and Taylor 1994, Syrdal et al. 2000).\nA third paradigm, articulatory synthesizers attempt to synthesize speech by\nmodeling the physics of the vocal tract as an open tube. Representative models\ninclude Stevens et al. (1953), Flanagan et al. (1975), and Fant (1986). See Klatt\n(1975) and Flanagan (1972) for more details.\nMost early TTS systems used phonemes as input; development of the text anal-\nysis components of TTS came somewhat later, drawing on NLP. Indeed the \ufb01rst\ntrue text-to-speech system seems to have been the system of Umeda and Teranishi\n(Umeda et al. 1968, Teranishi and Umeda 1968, Umeda 1976), which included a\nparser that assigned prosodic boundaries, as well as accent and stress.\nExercises\n16.1 Analyze each of the errors in the incorrectly recognized transcription of \u201cum\nthe phone is I left the. . . \u201d on page 16. For each one, give your best guess as\nto whether you think it is caused by a problem in signal processing, pronun-\nciation modeling, lexicon size, language model, or pruning in the decoding\nsearch. 28 Chapter 16 \u2022 Automatic Speech Recognition and Text-to-Speech\nAllen, J., M. S. Hunnicut, and D. H. Klatt. 1987. From Text\nto Speech: The MITalk system . Cambridge University\nPress.\nAtal, B. S. and S. Hanauer. 1971. Speech analysis and syn-\nthesis by prediction of the speech wave. JASA , 50:637\u2013\n655.\nBahdanau, D., J. Chorowski, D. Serdyuk, P. Brakel, and\nY . Bengio. 2016. End-to-end attention-based large vo-\ncabulary speech recognition. ICASSP .\nBaker, J. K. 1975. The DRAGON system \u2013 An overview.\nIEEE Transactions on ASSP , ASSP-23(1):24\u201329.\nBaum, L. E. and J. A. Eagon. 1967. An inequality with appli-\ncations to statistical estimation for probabilistic functions\nof Markov processes and to a model for ecology. Bulletin\nof the American Mathematical Society , 73(3):360\u2013363.\nBaum, L. E. and T. Petrie. 1966. Statistical inference for\nprobabilistic functions of \ufb01nite-state Markov chains. An-\nnals of Mathematical Statistics , 37(6):1554\u20131563.\nBlack, A. W. and P. Taylor. 1994. CHATR: A generic speech\nsynthesis system. COLING .\nBourlard, H. and N. Morgan. 1994. Connectionist Speech\nRecognition: A Hybrid Approach . Kluwer.\nBu, H., J. Du, X. Na, B. Wu, and H. Zheng. 2017. AISHELL-\n1: An open-source Mandarin speech corpus and a speech\nrecognition baseline. O-COCOSDA Proceedings .\nCanavan, A., D. Graff, and G. Zipperlen. 1997. CALL-\nHOME American English speech LDC97S42. Linguistic\nData Consortium.\nChan, W., N. Jaitly, Q. Le, and O. Vinyals. 2016. Listen,\nattend and spell: A neural network for large vocabulary\nconversational speech recognition. ICASSP .\nChorowski, J., D. Bahdanau, K. Cho, and Y . Bengio.\n2014. End-to-end continuous speech recognition us-\ning attention-based recurrent NN: First results. NeurIPS\nDeep Learning and Representation Learning Workshop .\nCieri, C., D. Miller, and K. Walker. 2004. The Fisher cor-\npus: A resource for the next generations of speech-to-text.\nLREC .\nCooley, J. W. and J. W. Tukey. 1965. An algorithm for the\nmachine calculation of complex Fourier series. Mathe-\nmatics of Computation , 19(90):297\u2013301.\nCooper, F. S., A. M. Liberman, and J. M. Borst. 1951. The\ninterconversion of audible and visible patterns as a basis\nfor research in the perception of speech. Proceedings of\nthe National Academy of Sciences , 37(5):318\u2013325.\nDahl, G. E., T. N. Sainath, and G. E. Hinton. 2013. Im-\nproving deep neural networks for LVCSR using recti\ufb01ed\nlinear units and dropout. ICASSP .\nDahl, G. E., D. Yu, L. Deng, and A. Acero. 2012. Context-\ndependent pre-trained deep neural networks for large-\nvocabulary speech recognition. IEEE Transactions on au-\ndio, speech, and language processing , 20(1):30\u201342.\nDavid, Jr., E. E. and O. G. Selfridge. 1962. Eyes and ears\nfor computers. Proceedings of the IRE (Institute of Radio\nEngineers) , 50:1093\u20131101.\nDavis, K. H., R. Biddulph, and S. Balashek. 1952. Automatic\nrecognition of spoken digits. JASA , 24(6):637\u2013642.Davis, S. and P. Mermelstein. 1980. Comparison of para-\nmetric representations for monosyllabic word recognition\nin continuously spoken sentences. IEEE Transactions on\nASSP , 28(4):357\u2013366.\nDemberg, V . 2006. Letter-to-phoneme conversion for a Ger-\nman text-to-speech system. Diplomarbeit Nr. 47, Univer-\nsit\u00a8at Stuttgart.\nDenes, P. 1959. The design and operation of the mechanical\nspeech recognizer at University College London. Journal\nof the British Institution of Radio Engineers , 19(4):219\u2013\n234. Appears together with companion paper (Fry 1959).\nDeng, L., G. Hinton, and B. Kingsbury. 2013. New types of\ndeep neural network learning for speech recognition and\nrelated applications: An overview. ICASSP .\nDixon, N. and H. Maxey. 1968. Terminal analog synthesis of\ncontinuous speech using the diphone method of segment\nassembly. IEEE Transactions on Audio and Electroacous-\ntics, 16(1):40\u201350.\nDu Bois, J. W., W. L. Chafe, C. Meyer, S. A. Thompson,\nR. Englebretson, and N. Martey. 2005. Santa Barbara cor-\npus of spoken American English, Parts 1-4. Philadelphia:\nLinguistic Data Consortium.\nEbden, P. and R. Sproat. 2015. The Kestrel TTS text\nnormalization system. Natural Language Engineering ,\n21(3):333.\nvan Esch, D. and R. Sproat. 2018. An expanded taxonomy of\nsemiotic classes for text normalization. INTERSPEECH .\nFant, G. M. 1951. Speech communication research. Ing.\nVetenskaps Akad. Stockholm, Sweden , 24:331\u2013337.\nFant, G. M. 1986. Glottal \ufb02ow: Models and interaction.\nJournal of Phonetics , 14:393\u2013399.\nFlanagan, J. L. 1972. Speech Analysis, Synthesis, and Per-\nception . Springer.\nFlanagan, J. L., K. Ishizaka, and K. L. Shipley. 1975. Syn-\nthesis of speech from a dynamic model of the vocal\ncords and vocal tract. The Bell System Technical Jour-\nnal, 54(3):485\u2013506.\nFry, D. B. 1959. Theoretical aspects of mechanical speech\nrecognition. Journal of the British Institution of Radio\nEngineers , 19(4):211\u2013218. Appears together with com-\npanion paper (Denes 1959).\nGillick, L. and S. J. Cox. 1989. Some statistical issues in the\ncomparison of speech recognition algorithms. ICASSP .\nGodfrey, J., E. Holliman, and J. McDaniel. 1992. SWITCH-\nBOARD: Telephone speech corpus for research and de-\nvelopment. ICASSP .\nGraff, D. 1997. The 1996 Broadcast News speech and\nlanguage-model corpus. Proceedings DARPA Speech\nRecognition Workshop .\nGraves, A. 2012. Sequence transduction with recurrent neu-\nral networks. ICASSP .\nGraves, A., S. Fern \u00b4andez, F. Gomez, and J. Schmidhuber.\n2006. Connectionist temporal classi\ufb01cation: Labelling\nunsegmented sequence data with recurrent neural net-\nworks. ICML .\nGraves, A. and N. Jaitly. 2014. Towards end-to-end speech\nrecognition with recurrent neural networks. ICML . Exercises 29\nGraves, A., A.-r. Mohamed, and G. Hinton. 2013.\nSpeech recognition with deep recurrent neural networks.\nICASSP .\nHannun, A. 2017. Sequence modeling with CTC. Distill ,\n2(11).\nHannun, A. Y ., A. L. Maas, D. Jurafsky, and A. Y . Ng. 2014.\nFirst-pass large vocabulary continuous speech recogni-\ntion using bi-directional recurrent DNNs. ArXiv preprint\narXiv:1408.2873.\nHarris, C. M. 1953. A study of the building blocks in speech.\nJASA , 25(5):962\u2013969.\nHayashi, T., R. Yamamoto, K. Inoue, T. Yoshimura,\nS. Watanabe, T. Toda, K. Takeda, Y . Zhang, and X. Tan.\n2020. ESPnet-TTS: Uni\ufb01ed, reproducible, and inte-\ngratable open source end-to-end text-to-speech toolkit.\nICASSP .\nHunt, A. J. and A. W. Black. 1996. Unit selection in a con-\ncatenative speech synthesis system using a large speech\ndatabase. ICASSP .\nItakura, F. 1975. Minimum prediction residual principle ap-\nplied to speech recognition. IEEE Transactions on ASSP ,\nASSP-32:67\u201372.\nIto, K. and L. Johnson. 2017. The LJ speech dataset. https:\n//keithito.com/LJ-Speech-Dataset/ .\nJaitly, N., P. Nguyen, A. Senior, and V . Vanhoucke. 2012.\nApplication of pretrained deep neural networks to large\nvocabulary speech recognition. INTERSPEECH .\nJelinek, F. 1969. A fast sequential decoding algorithm us-\ning a stack. IBM Journal of Research and Development ,\n13:675\u2013685.\nJelinek, F., R. L. Mercer, and L. R. Bahl. 1975. Design of a\nlinguistic statistical decoder for the recognition of contin-\nuous speech. IEEE Transactions on Information Theory ,\nIT-21(3):250\u2013256.\nKarita, S., N. Chen, T. Hayashi, T. Hori, H. Inaguma,\nZ. Jiang, M. Someki, N. E. Y . Soplin, R. Yamamoto,\nX. Wang, S. Watanabe, T. Yoshimura, and W. Zhang.\n2019. A comparative study on transformer vs RNN in\nspeech applications. IEEE ASRU-19 .\nKendall, T. and C. Farrington. 2020. The Corpus of Regional\nAfrican American Language. Version 2020.05. Eugene,\nOR: The Online Resources for African American Lan-\nguage Project. http://oraal.uoregon.edu/coraal.\nKlatt, D. H. 1975. V oice onset time, friction, and aspiration\nin word-initial consonant clusters. Journal of Speech and\nHearing Research , 18:686\u2013706.\nKlatt, D. H. 1977. Review of the ARPA speech understand-\ning project. JASA , 62(6):1345\u20131366.\nKlatt, D. H. 1982. The Klattalk text-to-speech conversion\nsystem. ICASSP .\nLang, K. J., A. H. Waibel, and G. E. Hinton. 1990. A\ntime-delay neural network architecture for isolated word\nrecognition. Neural networks , 3(1):23\u201343.\nLawrence, W. 1953. The synthesis of speech from signals\nwhich have a low information rate. In W. Jackson, ed.,\nCommunication Theory , 460\u2013469. Butterworth.\nLDC. 1998. LDC Catalog: Hub4 project . Univer-\nsity of Pennsylvania. www.ldc.upenn.edu/Catalog/\nLDC98S71.html .Liu, Y ., P. Fung, Y . Yang, C. Cieri, S. Huang, and D. Graff.\n2006. HKUST/MTS: A very large scale Mandarin tele-\nphone speech corpus. International Conference on Chi-\nnese Spoken Language Processing .\nLowerre, B. T. 1976. The Harpy Speech Recognition System .\nPh.D. thesis, Carnegie Mellon University, Pittsburgh, PA.\nMaas, A., Z. Xie, D. Jurafsky, and A. Y . Ng. 2015. Lexicon-\nfree conversational speech recognition with neural net-\nworks. NAACL HLT .\nMaas, A. L., A. Y . Hannun, and A. Y . Ng. 2013. Recti\ufb01er\nnonlinearities improve neural network acoustic models.\nICML .\nMaas, A. L., P. Qi, Z. Xie, A. Y . Hannun, C. T. Lengerich,\nD. Jurafsky, and A. Y . Ng. 2017. Building dnn acoustic\nmodels for large vocabulary speech recognition. Com-\nputer Speech & Language , 41:195\u2013213.\nMohamed, A., G. E. Dahl, and G. E. Hinton. 2009. Deep\nBelief Networks for phone recognition. NIPS Workshop\non Deep Learning for Speech Recognition and Related\nApplications .\nMorgan, N. and H. Bourlard. 1990. Continuous speech\nrecognition using multilayer perceptrons with hidden\nmarkov models. ICASSP .\nMorgan, N. and H. A. Bourlard. 1995. Neural networks for\nstatistical recognition of continuous speech. Proceedings\nof the IEEE , 83(5):742\u2013772.\nNIST. 2005. Speech recognition scoring toolkit (sctk) ver-\nsion 2.1. http://www.nist.gov/speech/tools/ .\nNIST. 2007. Matched Pairs Sentence-Segment Word Error\n(MAPSSWE) Test.\nOlive, J. P. 1977. Rule synthesis of speech from dyadic units.\nICASSP77 .\nvan den Oord, A., S. Dieleman, H. Zen, K. Simonyan,\nO. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and\nK. Kavukcuoglu. 2016. WaveNet: A Generative Model\nfor Raw Audio. ISCA Workshop on Speech Synthesis\nWorkshop .\nOppenheim, A. V ., R. W. Schafer, and T. G. J. Stockham.\n1968. Nonlinear \ufb01ltering of multiplied and convolved sig-\nnals. Proceedings of the IEEE , 56(8):1264\u20131291.\nPanayotov, V ., G. Chen, D. Povey, and S. Khudanpur. 2015.\nLibrispeech: an ASR corpus based on public domain au-\ndio books. ICASSP .\nPeterson, G. E., W. S.-Y . Wang, and E. Sivertsen. 1958.\nSegmentation techniques in speech synthesis. JASA ,\n30(8):739\u2013742.\nPovey, D., A. Ghoshal, G. Boulianne, L. Burget, O. Glem-\nbek, N. Goel, M. Hannemann, P. Motlicek, Y . Qian,\nP. Schwarz, J. Silovsk \u00b4y, G. Stemmer, and K. Vesel \u00b4y.\n2011. The Kaldi speech recognition toolkit. ASRU .\nPrice, P. J., W. Fisher, J. Bernstein, and D. Pallet. 1988. The\nDARPA 1000-word resource management database for\ncontinuous speech recognition. ICASSP .\nPundak, G. and T. N. Sainath. 2016. Lower frame rate neural\nnetwork acoustic models. INTERSPEECH .\nRobinson, T. and F. Fallside. 1991. A recurrent error prop-\nagation network speech recognition system. Computer\nSpeech & Language , 5(3):259\u2013274.\nSagisaka, Y . 1988. Speech synthesis by rule using an optimal\nselection of non-uniform synthesis units. ICASSP . 30 Chapter 16 \u2022 Automatic Speech Recognition and Text-to-Speech\nSagisaka, Y ., N. Kaiki, N. Iwahashi, and K. Mimura. 1992.\nAtr \u2013 n-talk speech synthesis system. ICSLP .\nSakoe, H. and S. Chiba. 1971. A dynamic programming\napproach to continuous speech recognition. Proceedings\nof the Seventh International Congress on Acoustics , vol-\nume 3. Akad \u00b4emiai Kiad \u00b4o.\nSakoe, H. and S. Chiba. 1984. Dynamic programming al-\ngorithm optimization for spoken word recognition. IEEE\nTransactions on ASSP , ASSP-26(1):43\u201349.\nShannon, C. E. 1948. A mathematical theory of commu-\nnication. Bell System Technical Journal , 27(3):379\u2013423.\nContinued in the following volume.\nShen, J., R. Pang, R. J. Weiss, M. Schuster, N. Jaitly,\nZ. Yang, Z. Chen, Y . Zhang, Y . Wang, R. Skerry-Ryan,\nR. A. Saurous, Y . Agiomyrgiannakis, and Y . Wu. 2018.\nNatural TTS synthesis by conditioning WaveNet on mel\nspectrogram predictions. ICASSP .\nSproat, R., A. W. Black, S. F. Chen, S. Kumar, M. Ostendorf,\nand C. Richards. 2001. Normalization of non-standard\nwords. Computer Speech & Language , 15(3):287\u2013333.\nSproat, R. and K. Gorman. 2018. A brief summary of the\nKaggle text normalization challenge.\nStevens, K. N., S. Kasowski, and G. M. Fant. 1953. An elec-\ntrical analog of the vocal tract. JASA , 25(4):734\u2013742.\nStevens, S. S. and J. V olkmann. 1940. The relation of pitch\nto frequency: A revised scale. The American Journal of\nPsychology , 53(3):329\u2013353.\nStevens, S. S., J. V olkmann, and E. B. Newman. 1937. A\nscale for the measurement of the psychological magni-\ntude pitch. JASA , 8:185\u2013190.\nSyrdal, A. K., C. W. Wightman, A. Conkie, Y . Stylianou,\nM. Beutnagel, J. Schroeter, V . Strom, and K.-S. Lee.\n2000. Corpus-based techniques in the AT&T NEXTGEN\nsynthesis system. ICSLP .\nTaylor, P. 2009. Text-to-Speech Synthesis . Cambridge Uni-\nversity Press.\nTeranishi, R. and N. Umeda. 1968. Use of pronouncing dic-\ntionary in speech synthesis experiments. 6th International\nCongress on Acoustics .\nUmeda, N. 1976. Linguistic rules for text-to-speech synthe-\nsis.Proceedings of the IEEE , 64(4):443\u2013451.\nUmeda, N., E. Matui, T. Suzuki, and H. Omura. 1968. Syn-\nthesis of fairy tale using an analog vocal tract. 6th Inter-\nnational Congress on Acoustics .\nVelichko, V . M. and N. G. Zagoruyko. 1970. Automatic\nrecognition of 200 words. International Journal of Man-\nMachine Studies , 2:223\u2013234.\nVintsyuk, T. K. 1968. Speech discrimination by dynamic\nprogramming. Cybernetics , 4(1):52\u201357. Original Rus-\nsian: Kibernetika 4(1):81-88. 1968.\nWaibel, A., T. Hanazawa, G. Hinton, K. Shikano, and K. J.\nLang. 1989. Phoneme recognition using time-delay neu-\nral networks. IEEE Transactions on ASSP , 37(3):328\u2013\n339.\nWang, Y ., R. Skerry-Ryan, D. Stanton, Y . Wu, R. J. Weiss,\nN. Jaitly, Z. Yang, Y . Xiao, Z. Chen, S. Bengio, Q. Le,\nY . Agiomyrgiannakis, R. Clark, and R. A. Saurous. 2017.\nTacotron: Towards end-to-end speech synthesis. INTER-\nSPEECH .Watanabe, S., T. Hori, S. Karita, T. Hayashi, J. Nishitoba,\nY . Unno, N. E. Y . Soplin, J. Heymann, M. Wiesner,\nN. Chen, A. Renduchintala, and T. Ochiai. 2018. ESPnet:\nEnd-to-end speech processing toolkit. INTERSPEECH .\nZhang, H., R. Sproat, A. H. Ng, F. Stahlberg, X. Peng,\nK. Gorman, and B. Roark. 2019. Neural models of text\nnormalization for speech applications. Computational\nLinguistics , 45(2):293\u2013337.", "17.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\n17Sequence Labeling for Parts of\nSpeech and Named Entities\nTo each word a warbling note\nA Midsummer Night\u2019s Dream , V .I\nDionysius Thrax of Alexandria ( c.100 B.C.), or perhaps someone else (it was a long\ntime ago), wrote a grammatical sketch of Greek (a \u201c techn \u00afe\u201d) that summarized the\nlinguistic knowledge of his day. This work is the source of an astonishing proportion\nof modern linguistic vocabulary, including the words syntax ,diphthong ,clitic , and\nanalogy . Also included are a description of eight parts of speech : noun, verb, parts of speech\npronoun, preposition, adverb, conjunction, participle, and article. Although earlier\nscholars (including Aristotle as well as the Stoics) had their own lists of parts of\nspeech, it was Thrax\u2019s set of eight that became the basis for descriptions of European\nlanguages for the next 2000 years. (All the way to the Schoolhouse Rock educational\ntelevision shows of our childhood, which had songs about 8 parts of speech, like the\nlate great Bob Dorough\u2019s Conjunction Junction .) The durability of parts of speech\nthrough two millennia speaks to their centrality in models of human language.\nProper names are another important and anciently studied linguistic category.\nWhile parts of speech are generally assigned to individual words or morphemes, a\nproper name is often an entire multiword phrase, like the name \u201cMarie Curie\u201d, the\nlocation \u201cNew York City\u201d, or the organization \u201cStanford University\u201d. We\u2019ll use the\nterm named entity for, roughly speaking, anything that can be referred to with a named entity\nproper name: a person, a location, an organization, although as we\u2019ll see the term is\ncommonly extended to include things that aren\u2019t entities per se.\nParts of speech (also known as POS ) and named entities are useful clues to POS\nsentence structure and meaning. Knowing whether a word is a noun or a verb tells us\nabout likely neighboring words (nouns in English are preceded by determiners and\nadjectives, verbs by nouns) and syntactic structure (verbs have dependency links to\nnouns), making part-of-speech tagging a key aspect of parsing. Knowing if a named\nentity like Washington is a name of a person, a place, or a university is important to\nmany natural language processing tasks like question answering, stance detection,\nor information extraction.\nIn this chapter we\u2019ll introduce the task of part-of-speech tagging , taking a se-\nquence of words and assigning each word a part of speech like NOUN orVERB , and\nthe task of named entity recognition (NER ), assigning words or phrases tags like\nPERSON ,LOCATION , or ORGANIZATION .\nSuch tasks in which we assign, to each word xiin an input word sequence, a\nlabel yi, so that the output sequence Yhas the same length as the input sequence X\nare called sequence labeling tasks. We\u2019ll introduce classic sequence labeling algo-sequence\nlabeling\nrithms, one generative\u2014 the Hidden Markov Model (HMM)\u2014and one discriminative\u2014\nthe Conditional Random Field (CRF). In following chapters we\u2019ll introduce modern\nsequence labelers based on RNNs and Transformers. 2CHAPTER 17 \u2022 S EQUENCE LABELING FOR PARTS OF SPEECH AND NAMED ENTITIES\n17.1 (Mostly) English Word Classes\nUntil now we have been using part-of-speech terms like noun andverb rather freely.\nIn this section we give more complete de\ufb01nitions. While word classes do have\nsemantic tendencies\u2014adjectives, for example, often describe properties and nouns\npeople \u2014 parts of speech are de\ufb01ned instead based on their grammatical relationship\nwith neighboring words or the morphological properties about their af\ufb01xes.\nTag Description ExampleOpen ClassADJ Adjective: noun modi\ufb01ers describing properties red,young ,awesome\nADV Adverb: verb modi\ufb01ers of time, place, manner very,slowly ,home ,yesterday\nNOUN words for persons, places, things, etc. algorithm ,cat,mango ,beauty\nVERB words for actions and processes draw ,provide ,go\nPROPN Proper noun: name of a person, organization, place, etc.. Regina ,IBM,Colorado\nINTJ Interjection: exclamation, greeting, yes/no response, etc. oh,um,yes,helloClosed Class WordsADP Adposition (Preposition/Postposition): marks a noun\u2019s\nspacial, temporal, or other relationin, on, by, under\nAUX Auxiliary: helping verb marking tense, aspect, mood, etc., can, may, should, are\nCCONJ Coordinating Conjunction: joins two phrases/clauses and,or,but\nDET Determiner: marks noun phrase properties a, an, the, this\nNUM Numeral one, two, 2026, 11:00, hundred\nPART Particle: a function word that must be associated with an-\nother word\u2019s, not, (in\ufb01nitive) to\nPRON Pronoun: a shorthand for referring to an entity or event she, who, I, others\nSCONJ Subordinating Conjunction: joins a main clause with a\nsubordinate clause such as a sentential complementwhether ,becauseOtherPUNCT Punctuation\u02d9, , ()\nSYM Symbols like $ or emoji $, %\nX Other asdf, qwfg\nFigure 17.1 The 17 parts of speech in the Universal Dependencies tagset (de Marneffe et al., 2021). Features\ncan be added to make \ufb01ner-grained distinctions (with properties like number, case, de\ufb01niteness, and so on).\nParts of speech fall into two broad categories: closed class andopen class . closed class\nopen class Closed classes are those with relatively \ufb01xed membership, such as prepositions\u2014\nnew prepositions are rarely coined. By contrast, nouns and verbs are open classes\u2014\nnew nouns and verbs like iPhone orto fax are continually being created or borrowed.\nClosed class words are generally function words likeof,it,and, oryou, which tend function word\nto be very short, occur frequently, and often have structuring uses in grammar.\nFour major open classes occur in the languages of the world: nouns (including\nproper nouns), verbs ,adjectives , and adverbs , as well as the smaller open class of\ninterjections . English has all \ufb01ve, although not every language does.\nNouns are words for people, places, or things, but include others as well. Com- noun\nmon nouns include concrete terms like catandmango , abstractions like algorithm common noun\nandbeauty , and verb-like terms like pacing as in His pacing to and fro became quite\nannoying . Nouns in English can occur with determiners ( a goat, this bandwidth )\ntake possessives ( IBM\u2019s annual revenue ), and may occur in the plural ( goats, abaci ).\nMany languages, including English, divide common nouns into count nouns and count noun\nmass nouns . Count nouns can occur in the singular and plural ( goat/goats, rela- mass noun\ntionship/relationships ) and can be counted ( one goat, two goats ). Mass nouns are\nused when something is conceptualized as a homogeneous group. So snow, salt , and\ncommunism are not counted (i.e., *two snows or*two communisms ).Proper nouns , proper noun\nlikeRegina ,Colorado , and IBM, are names of speci\ufb01c persons or entities. 17.1 \u2022 (M OSTLY ) ENGLISH WORD CLASSES 3\nVerbs refer to actions and processes, including main verbs like draw ,provide , verb\nandgo. English verbs have in\ufb02ections (non-third-person-singular ( eat), third-person-\nsingular ( eats), progressive ( eating ), past participle ( eaten )). While many scholars\nbelieve that all human languages have the categories of noun and verb, others have\nargued that some languages, such as Riau Indonesian and Tongan, don\u2019t even make\nthis distinction (Broschart 1997; Evans 2000; Gil 2000) .\nAdjectives often describe properties or qualities of nouns, like color ( white , adjective\nblack ), age ( old,young ), and value ( good ,bad), but there are languages without\nadjectives. In Korean, for example, the words corresponding to English adjectives\nact as a subclass of verbs, so what is in English an adjective \u201cbeautiful\u201d acts in\nKorean like a verb meaning \u201cto be beautiful\u201d.\nAdverbs are a hodge-podge. All the italicized words in this example are adverbs: adverb\nActually , I ran home extremely quickly yesterday\nAdverbs generally modify something (often verbs, hence the name \u201cadverb\u201d, but\nalso other adverbs and entire verb phrases). Directional adverbs orlocative ad- locative\nverbs (home ,here,downhill ) specify the direction or location of some action; degree degree\nadverbs (extremely ,very,somewhat ) specify the extent of some action, process, or\nproperty; manner adverbs (slowly ,slinkily ,delicately ) describe the manner of some manner\naction or process; and temporal adverbs describe the time that some action or event temporal\ntook place ( yesterday ,Monday ).\nInterjections (oh, hey, alas, uh, um ) are a smaller open class that also includes interjection\ngreetings ( hello, goodbye ) and question responses ( yes, no, uh-huh ).\nEnglish adpositions occur before nouns, hence are called prepositions . They can preposition\nindicate spatial or temporal relations, whether literal ( on it,before then ,by the house )\nor metaphorical ( on time ,with gusto ,beside herself ), and relations like marking the\nagent in Hamlet was written by Shakespeare .\nAparticle resembles a preposition or an adverb and is used in combination with particle\na verb. Particles often have extended meanings that aren\u2019t quite the same as the\nprepositions they resemble, as in the particle over inshe turned the paper over . A\nverb and a particle acting as a single unit is called a phrasal verb . The meaning phrasal verb\nof phrasal verbs is often non-compositional \u2014not predictable from the individual\nmeanings of the verb and the particle. Thus, turn down means \u2018reject\u2019, rule out\n\u2018eliminate\u2019, and go on \u2018continue\u2019.\nDeterminers likethisandthat(this chapter ,that page ) can mark the start of an determiner\nEnglish noun phrase. Articles likea,an, and the, are a type of determiner that mark article\ndiscourse properties of the noun and are quite frequent; theis the most common\nword in written English, with aandanright behind.\nConjunctions join two phrases, clauses, or sentences. Coordinating conjunc- conjunction\ntions like and,or, and butjoin two elements of equal status. Subordinating conjunc-\ntions are used when one of the elements has some embedded status. For example,\nthe subordinating conjunction thatin\u201cI thought that you might like some milk\u201d links\nthe main clause I thought with the subordinate clause you might like some milk . This\nclause is called subordinate because this entire clause is the \u201ccontent\u201d of the main\nverb thought . Subordinating conjunctions like thatwhich link a verb to its argument\nin this way are also called complementizers . complementizer\nPronouns act as a shorthand for referring to an entity or event. Personal pro- pronoun\nnouns refer to persons or entities ( you,she,I,it,me, etc.). Possessive pronouns are\nforms of personal pronouns that indicate either actual possession or more often just\nan abstract relation between the person and some object ( my, your, his, her, its, one\u2019s,\nour, their ).Wh-pronouns (what, who, whom, whoever ) are used in certain question wh 4CHAPTER 17 \u2022 S EQUENCE LABELING FOR PARTS OF SPEECH AND NAMED ENTITIES\nforms, or act as complementizers ( Frida, who married Diego. . . ).\nAuxiliary verbs mark semantic features of a main verb such as its tense, whether auxiliary\nit is completed (aspect), whether it is negated (polarity), and whether an action is\nnecessary, possible, suggested, or desired (mood). English auxiliaries include the\ncopula verb be, the two verbs doandhave , forms, as well as modal verbs used to copula\nmodal mark the mood associated with the event depicted by the main verb: canindicates\nability or possibility, may permission or possibility, must necessity.\nAn English-speci\ufb01c tagset, the Penn Treebank tagset (Marcus et al., 1993), shown\nin Fig. 17.2, has been used to label many syntactically annotated corpora like the\nPenn Treebank corpora, so it is worth knowing about.\nTag Description Example Tag Description Example Tag Description Example\nCC coord. conj. and, but, or NNP proper noun, sing. IBM TO in\ufb01nitive to to\nCD cardinal number one, two NNPS proper noun, plu. Carolinas UH interjection ah, oops\nDT determiner a, the NNS noun, plural llamas VB verb base eat\nEX existential \u2018there\u2019 there PDT predeterminer all, both VBD verb past tense ate\nFW foreign word mea culpa POS possessive ending \u2019s VBG verb gerund eating\nIN preposition/\nsubordin-conjof, in, by PRP personal pronoun I, you, he VBN verb past partici-\npleeaten\nJJ adjective yellow PRP$ possess. pronoun your VBP verb non-3sg-pr eat\nJJR comparative adj bigger RB adverb quickly VBZ verb 3sg pres eats\nJJS superlative adj wildest RBR comparative adv faster WDT wh-determ. which, that\nLS list item marker 1, 2, One RBS superlatv. adv fastest WP wh-pronoun what, who\nMD modal can, should RP particle up, off WP$ wh-possess. whose\nNN sing or mass noun llama SYM symbol +,%,& WRB wh-adverb how, where\nFigure 17.2 Penn Treebank core 36 part-of-speech tags.\nBelow we show some examples with each word tagged according to both the UD\n(in blue) and Penn (in red) tagsets. Notice that the Penn tagset distinguishes tense\nand participles on verbs, and has a special tag for the existential there construction in\nEnglish. Note that since London Journal of Medicine is a proper noun, both tagsets\nmark its component nouns as PROPN/NNP, including journal andmedicine , which\nmight otherwise be labeled as common nouns (NOUN/NN).\n(17.1) There/ PRON /EXare/ VERB /VBP 70/NUM /CDchildren/ NOUN /NNS\nthere/ ADV/RB./PUNC /.\n(17.2) Preliminary/ ADJ/JJ\ufb01ndings/ NOUN /NNS were/ AUX/VBD\nreported/ VERB /VBN in/ADP/INtoday/ NOUN /NN\u2019s/PART /POS\nLondon/ PROPN /NNP Journal/ PROPN /NNP of/ADP/INMedicine/ PROPN /NNP\n17.2 Part-of-Speech Tagging\nPart-of-speech tagging is the process of assigning a part-of-speech to each word inpart-of-speech\ntagging\na text. The input is a sequence x1;x2;:::;xnof (tokenized) words and a tagset, and\nthe output is a sequence y1;y2;:::;ynof tags, each output yicorresponding exactly to\none input xi, as shown in the intuition in Fig. 17.3.\nTagging is a disambiguation task; words are ambiguous \u2014have more than one ambiguous\npossible part-of-speech\u2014and the goal is to \ufb01nd the correct tag for the situation.\nFor example, book can be a verb ( book that \ufb02ight ) or a noun ( hand me that book ).\nThat can be a determiner ( Does that \ufb02ight serve dinner ) or a complementizer ( I 17.2 \u2022 P ART-OF-SPEECH TAGGING 5\nwillNOUNAUXVERBDETNOUNJanetbackthebillPart of Speech Taggerx1x2x3x4x5y1y2y3y4y5\nFigure 17.3 The task of part-of-speech tagging: mapping from input words x1;x2;:::;xnto\noutput POS tags y1;y2;:::;yn.\nthought that your \ufb02ight was earlier ). The goal of POS-tagging is to resolve theseambiguity\nresolution\nambiguities, choosing the proper tag for the context.\nThe accuracy of part-of-speech tagging algorithms (the percentage of test set accuracy\ntags that match human gold labels) is extremely high. One study found accuracies\nover 97% across 15 languages from the Universal Dependency (UD) treebank (Wu\nand Dredze, 2019). Accuracies on various English treebanks are also 97% (no matter\nthe algorithm; HMMs, CRFs, BERT perform similarly). This 97% number is also\nabout the human performance on this task, at least for English (Manning, 2011).\nTypes: WSJ Brown\nUnambiguous (1 tag) 44,432 ( 86% ) 45,799 ( 85% )\nAmbiguous (2+ tags) 7,025 ( 14% ) 8,050 ( 15% )\nTokens :\nUnambiguous (1 tag) 577,421 ( 45% ) 384,349 ( 33% )\nAmbiguous (2+ tags) 711,780 ( 55% ) 786,646 ( 67% )\nFigure 17.4 Tag ambiguity in the Brown and WSJ corpora (Treebank-3 45-tag tagset).\nWe\u2019ll introduce algorithms for the task in the next few sections, but \ufb01rst let\u2019s\nexplore the task. Exactly how hard is it? Fig. 17.4 shows that most word types\n(85-86%) are unambiguous ( Janet is always NNP, hesitantly is always RB). But the\nambiguous words, though accounting for only 14-15% of the vocabulary, are very\ncommon, and 55-67% of word tokens in running text are ambiguous. Particularly\nambiguous common words include that,back,down ,putandset; here are some\nexamples of the 6 different parts of speech for the word back:\nearnings growth took a back/JJ seat\na small building in the back/NN\na clear majority of senators back/VBP the bill\nDave began to back/VB toward the door\nenable the country to buy back/RP debt\nI was twenty-one back/RB then\nNonetheless, many words are easy to disambiguate, because their different tags\naren\u2019t equally likely. For example, acan be a determiner or the letter a, but the\ndeterminer sense is much more likely.\nThis idea suggests a useful baseline : given an ambiguous word, choose the tag\nwhich is most frequent in the training corpus. This is a key concept:\nMost Frequent Class Baseline: Always compare a classi\ufb01er against a baseline at\nleast as good as the most frequent class baseline (assigning each token to the class\nit occurred in most often in the training set). 6CHAPTER 17 \u2022 S EQUENCE LABELING FOR PARTS OF SPEECH AND NAMED ENTITIES\nThe most-frequent-tag baseline has an accuracy of about 92%1. The baseline\nthus differs from the state-of-the-art and human ceiling (97%) by only 5%.\n17.3 Named Entities and Named Entity Tagging\nPart of speech tagging can tell us that words like Janet ,Stanford University , and\nColorado are all proper nouns; being a proper noun is a grammatical property of\nthese words. But viewed from a semantic perspective, these proper nouns refer to\ndifferent kinds of entities: Janet is a person, Stanford University is an organization,\nand Colorado is a location.\nHere we re-introduce the concept of a named entity , which was also introduced named entity\nin Section ??for readers who haven\u2019t yet read Chapter 11.\nAnamed entity is, roughly speaking, anything that can be referred to with a named entity\nproper name: a person, a location, an organization. The task of named entity recog-\nnition (NER ) is to \ufb01nd spans of text that constitute proper names and tag the type ofnamed entity\nrecognition\nNER the entity. Four entity tags are most common: PER (person), LOC (location), ORG\n(organization), or GPE (geo-political entity). However, the term named entity is\ncommonly extended to include things that aren\u2019t entities per se, including dates,\ntimes, and other kinds of temporal expressions, and even numerical expressions like\nprices. Here\u2019s an example of the output of an NER tagger:\nCiting high fuel prices, [ ORG United Airlines ] said [ TIME Friday ] it\nhas increased fares by [ MONEY $6] per round trip on \ufb02ights to some\ncities also served by lower-cost carriers. [ ORG American Airlines ], a\nunit of [ ORG AMR Corp.] , immediately matched the move, spokesman\n[PER Tim Wagner ] said. [ ORG United] , a unit of [ ORG UAL Corp.] ,\nsaid the increase took effect [ TIME Thursday] and applies to most\nroutes where it competes against discount carriers, such as [ LOC Chicago]\nto [LOC Dallas] and [ LOC Denver] to [LOC San Francisco] .\nThe text contains 13 mentions of named entities including 5 organizations, 4 loca-\ntions, 2 times, 1 person, and 1 mention of money. Figure 17.5 shows typical generic\nnamed entity types. Many applications will also need to use speci\ufb01c entity types like\nproteins, genes, commercial products, or works of art.\nType Tag Sample Categories Example sentences\nPeople PER people, characters Turing is a giant of computer science.\nOrganization ORG companies, sports teams The IPCC warned about the cyclone.\nLocation LOC regions, mountains, seas Mt. Sanitas is in Sunshine Canyon .\nGeo-Political Entity GPE countries, states Palo Alto is raising the fees for parking.\nFigure 17.5 A list of generic named entity types with the kinds of entities they refer to.\nNamed entity tagging is a useful \ufb01rst step in lots of natural language processing\ntasks. In sentiment analysis we might want to know a consumer\u2019s sentiment toward a\nparticular entity. Entities are a useful \ufb01rst stage in question answering, or for linking\ntext to information in structured knowledge sources like Wikipedia. And named\nentity tagging is also central to tasks involving building semantic representations,\nlike extracting events and the relationship between participants.\n1In English, on the WSJ corpus, tested on sections 22-24. 17.3 \u2022 N AMED ENTITIES AND NAMED ENTITY TAGGING 7\nUnlike part-of-speech tagging, where there is no segmentation problem since\neach word gets one tag, the task of named entity recognition is to \ufb01nd and label\nspans of text, and is dif\ufb01cult partly because of the ambiguity of segmentation; we\nneed to decide what\u2019s an entity and what isn\u2019t, and where the boundaries are. Indeed,\nmost words in a text will not be named entities. Another dif\ufb01culty is caused by type\nambiguity. The mention JFKcan refer to a person, the airport in New York, or any\nnumber of schools, bridges, and streets around the United States. Some examples of\nthis kind of cross-type confusion are given in Figure 17.6.\n[PER Washington] was born into slavery on the farm of James Burroughs.\n[ORG Washington] went up 2 games to 1 in the four-game series.\nBlair arrived in [ LOC Washington] for what may well be his last state visit.\nIn June, [ GPE Washington] passed a primary seatbelt law.\nFigure 17.6 Examples of type ambiguities in the use of the name Washington .\nThe standard approach to sequence labeling for a span-recognition problem like\nNER is BIO tagging (Ramshaw and Marcus, 1995). This is a method that allows us\nto treat NER like a word-by-word sequence labeling task, via tags that capture both\nthe boundary and the named entity type. Consider the following sentence:\n[PER Jane Villanueva ] of [ ORG United ] , a unit of [ ORG United Airlines\nHolding ] , said the fare applies to the [ LOC Chicago ] route.\nFigure 17.7 shows the same excerpt represented with BIO tagging, as well as BIO\nvariants called IOtagging and BIOES tagging. In BIO tagging we label any token\nthatbegins a span of interest with the label B, tokens that occur inside a span are\ntagged with an I, and any tokens outside of any span of interest are labeled O. While\nthere is only one Otag, we\u2019ll have distinct Band Itags for each named entity class.\nThe number of tags is thus 2 n+1 tags, where nis the number of entity types. BIO\ntagging can represent exactly the same information as the bracketed notation, but has\nthe advantage that we can represent the task in the same simple sequence modeling\nway as part-of-speech tagging: assigning a single label yito each input word xi:\nWords IO Label BIO Label BIOES Label\nJane I-PER B-PER B-PER\nVillanueva I-PER I-PER E-PER\nof O O O\nUnited I-ORG B-ORG B-ORG\nAirlines I-ORG I-ORG I-ORG\nHolding I-ORG I-ORG E-ORG\ndiscussed O O O\nthe O O O\nChicago I-LOC B-LOC S-LOC\nroute O O O\n. O O O\nFigure 17.7 NER as a sequence model, showing IO, BIO, and BIOES taggings.\nWe\u2019ve also shown two variant tagging schemes: IO tagging, which loses some\ninformation by eliminating the B tag, and BIOES tagging, which adds an end tag\nEfor the end of a span, and a span tag Sfor a span consisting of only one word.\nA sequence labeler (HMM, CRF, RNN, Transformer, etc.) is trained to label each\ntoken in a text with tags that indicate the presence (or absence) of particular kinds\nof named entities. 8CHAPTER 17 \u2022 S EQUENCE LABELING FOR PARTS OF SPEECH AND NAMED ENTITIES\n17.4 HMM Part-of-Speech Tagging\nIn this section we introduce our \ufb01rst sequence labeling algorithm, the Hidden Markov\nModel, and show how to apply it to part-of-speech tagging. Recall that a sequence\nlabeler is a model whose job is to assign a label to each unit in a sequence, thus\nmapping a sequence of observations to a sequence of labels of the same length.\nThe HMM is a classic model that introduces many of the key concepts of sequence\nmodeling that we will see again in more modern models.\nAn HMM is a probabilistic sequence model: given a sequence of units (words,\nletters, morphemes, sentences, whatever), it computes a probability distribution over\npossible sequences of labels and chooses the best label sequence.\n17.4.1 Markov Chains\nThe HMM is based on augmenting the Markov chain. A Markov chain is a model Markov chain\nthat tells us something about the probabilities of sequences of random variables,\nstates , each of which can take on values from some set. These sets can be words, or\ntags, or symbols representing anything, for example the weather. A Markov chain\nmakes a very strong assumption that if we want to predict the future in the sequence,\nall that matters is the current state. All the states before the current state have no im-\npact on the future except via the current state. It\u2019s as if to predict tomorrow\u2019s weather\nyou could examine today\u2019s weather but you weren\u2019t allowed to look at yesterday\u2019s\nweather.\nWARM3HOT1COLD2.8.6.1.1.3.6.1.1.3\ncharminguniformlyare.1.4.5.5.5.2.6.2\n(a) (b)\nFigure 17.8 A Markov chain for weather (a) and one for words (b), showing states and\ntransitions. A start distribution pis required; setting p= [0:1;0:7;0:2]for (a) would mean a\nprobability 0.7 of starting in state 2 (cold), probability 0.1 of starting in state 1 (hot), etc.\nMore formally, consider a sequence of state variables q1;q2;:::;qi. A Markov\nmodel embodies the Markov assumption on the probabilities of this sequence: thatMarkov\nassumption\nwhen predicting the future, the past doesn\u2019t matter, only the present.\nMarkov Assumption: P(qi=ajq1:::qi\u00001) =P(qi=ajqi\u00001) (17.3)\nFigure 17.8a shows a Markov chain for assigning a probability to a sequence of\nweather events, for which the vocabulary consists of HOT,COLD , and WARM . The\nstates are represented as nodes in the graph, and the transitions, with their probabil-\nities, as edges. The transitions are probabilities: the values of arcs leaving a given\nstate must sum to 1. Figure 17.8b shows a Markov chain for assigning a probabil-\nity to a sequence of words w1:::wt. This Markov chain should be familiar; in fact,\nit represents a bigram language model, with each edge expressing the probability\np(wijwj)! Given the two models in Fig. 17.8, we can assign a probability to any\nsequence from our vocabulary. 17.4 \u2022 HMM P ART-OF-SPEECH TAGGING 9\nFormally, a Markov chain is speci\ufb01ed by the following components:\nQ=q1q2:::qN a set of Nstates\nA=a11a12:::aN1:::aNN atransition probability matrix A, each ai jrepresent-\ning the probability of moving from state ito state j, s.t.Pn\nj=1ai j=18i\np=p1;p2;:::;pN aninitial probability distribution over states. piis the\nprobability that the Markov chain will start in state i.\nSome states jmay have pj=0, meaning that they cannot\nbe initial states. Also,Pn\ni=1pi=1\nBefore you go on, use the sample probabilities in Fig. 17.8a (with p= [0:1;0:7;0:2])\nto compute the probability of each of the following sequences:\n(17.4) hot hot hot hot\n(17.5) cold hot cold hot\nWhat does the difference in these probabilities tell you about a real-world weather\nfact encoded in Fig. 17.8a?\n17.4.2 The Hidden Markov Model\nA Markov chain is useful when we need to compute a probability for a sequence\nof observable events. In many cases, however, the events we are interested in are\nhidden : we don\u2019t observe them directly. For example we don\u2019t normally observe hidden\npart-of-speech tags in a text. Rather, we see words, and must infer the tags from the\nword sequence. We call the tags hidden because they are not observed.\nAhidden Markov model (HMM ) allows us to talk about both observed eventshidden Markov\nmodel\n(like words that we see in the input) and hidden events (like part-of-speech tags) that\nwe think of as causal factors in our probabilistic model. An HMM is speci\ufb01ed by\nthe following components:\nQ=q1q2:::qN a set of Nstates\nA=a11:::ai j:::aNNatransition probability matrix A, each ai jrepresenting the probability\nof moving from state ito state j, s.t.PN\nj=1ai j=18i\nB=bi(ot) a sequence of observation likelihoods , also called emission probabili-\nties, each expressing the probability of an observation ot(drawn from a\nvocabulary V=v1;v2;:::;vV) being generated from a state qi\np=p1;p2;:::;pN aninitial probability distribution over states. piis the probability that\nthe Markov chain will start in state i. Some states jmay have pj=0,\nmeaning that they cannot be initial states. Also,Pn\ni=1pi=1\nThe HMM is given as input O=o1o2:::oT: a sequence of Tobservations , each\none drawn from the vocabulary V.\nA \ufb01rst-order hidden Markov model instantiates two simplifying assumptions.\nFirst, as with a \ufb01rst-order Markov chain, the probability of a particular state depends\nonly on the previous state:\nMarkov Assumption: P(qijq1;:::;qi\u00001) =P(qijqi\u00001) (17.6)\nSecond, the probability of an output observation oidepends only on the state that\nproduced the observation qiand not on any other states or any other observations:\nOutput Independence: P(oijq1;:::qi;:::; qT;o1;:::; oi;:::; oT) =P(oijqi)(17.7) 10 CHAPTER 17 \u2022 S EQUENCE LABELING FOR PARTS OF SPEECH AND NAMED ENTITIES\n17.4.3 The components of an HMM tagger\nAn HMM has two components, the AandBprobabilities, both estimated by counting\non a tagged training corpus. (For this example we\u2019ll use the tagged WSJ corpus.)\nTheAmatrix contains the tag transition probabilities P(tijti\u00001)which represent\nthe probability of a tag occurring given the previous tag. For example, modal verbs\nlikewillare very likely to be followed by a verb in the base form, a VB, like race, so\nwe expect this probability to be high. We compute the maximum likelihood estimate\nof this transition probability by counting, out of the times we see the \ufb01rst tag in a\nlabeled corpus, how often the \ufb01rst tag is followed by the second:\nP(tijti\u00001) =C(ti\u00001;ti)\nC(ti\u00001)(17.8)\nIn the WSJ corpus, for example, MD occurs 13124 times of which it is followed\nby VB 10471, for an MLE estimate of\nP(V BjMD) =C(MD;V B)\nC(MD)=10471\n13124=:80 (17.9)\nTheBemission probabilities, P(wijti), represent the probability, given a tag (say\nMD), that it will be associated with a given word (say will). The MLE of the emis-\nsion probability is\nP(wijti) =C(ti;wi)\nC(ti)(17.10)\nOf the 13124 occurrences of MD in the WSJ corpus, it is associated with will4046\ntimes:\nP(willjMD) =C(MD;will)\nC(MD)=4046\n13124=:31 (17.11)\nWe saw this kind of Bayesian modeling in Chapter 4; recall that this likelihood\nterm is not asking \u201cwhich is the most likely tag for the word will?\u201d That would be\nthe posterior P(MDjwill). Instead, P(willjMD)answers the slightly counterintuitive\nquestion \u201cIf we were going to generate a MD, how likely is it that this modal would\nbewill?\u201d\nNN3VB1MD2a22\na11a12a21a13a33a32a23a31P(\"aardvark\" | NN)...P(\u201cwill\u201d | NN)...P(\"the\" | NN)...P(\u201cback\u201d | NN)...P(\"zebra\" | NN)B3P(\"aardvark\" | VB)...P(\u201cwill\u201d | VB)...P(\"the\" | VB)...P(\u201cback\u201d | VB)...P(\"zebra\" | VB)B1P(\"aardvark\" | MD)...P(\u201cwill\u201d | MD)...P(\"the\" | MD)...P(\u201cback\u201d | MD)...P(\"zebra\" | MD)B2\nFigure 17.9 An illustration of the two parts of an HMM representation: the Atransition\nprobabilities used to compute the prior probability, and the Bobservation likelihoods that are\nassociated with each state, one likelihood for each possible observation word. 17.4 \u2022 HMM P ART-OF-SPEECH TAGGING 11\nThe Atransition probabilities, and Bobservation likelihoods of the HMM are\nillustrated in Fig. 17.9 for three states in an HMM part-of-speech tagger; the full\ntagger would have one state for each tag.\n17.4.4 HMM tagging as decoding\nFor any model, such as an HMM, that contains hidden variables, the task of deter-\nmining the hidden variables sequence corresponding to the sequence of observations\nis called decoding . More formally, decoding\nDecoding : Given as input an HMM l= (A;B)and a sequence of ob-\nservations O=o1;o2;:::;oT, \ufb01nd the most probable sequence of states\nQ=q1q2q3:::qT.\nFor part-of-speech tagging, the goal of HMM decoding is to choose the tag\nsequence t1:::tnthat is most probable given the observation sequence of nwords\nw1:::wn:\n\u02c6t1:n=argmax\nt1:::tnP(t1:::tnjw1:::wn) (17.12)\nThe way we\u2019ll do this in the HMM is to use Bayes\u2019 rule to instead compute:\n\u02c6t1:n=argmax\nt1:::tnP(w1:::wnjt1:::tn)P(t1:::tn)\nP(w1:::wn)(17.13)\nFurthermore, we simplify Eq. 17.13 by dropping the denominator P(wn\n1):\n\u02c6t1:n=argmax\nt1:::tnP(w1:::wnjt1:::tn)P(t1:::tn) (17.14)\nHMM taggers make two further simplifying assumptions. The \ufb01rst (output in-\ndependence, from Eq. 17.7) is that the probability of a word appearing depends only\non its own tag and is independent of neighboring words and tags:\nP(w1:::wnjt1:::tn)\u0019nY\ni=1P(wijti) (17.15)\nThe second assumption (the Markov assumption, Eq. 17.6) is that the probability of\na tag is dependent only on the previous tag, rather than the entire tag sequence;\nP(t1:::tn)\u0019nY\ni=1P(tijti\u00001) (17.16)\nPlugging the simplifying assumptions from Eq. 17.15 and Eq. 17.16 into Eq. 17.14\nresults in the following equation for the most probable tag sequence from a bigram\ntagger:\n\u02c6t1:n=argmax\nt1:::tnP(t1:::tnjw1:::wn)\u0019argmax\nt1:::tnnY\ni=1emissionz}|{\nP(wijti)transitionz}|{\nP(tijti\u00001) (17.17)\nThe two parts of Eq. 17.17 correspond neatly to the Bemission probability andA\ntransition probability that we just de\ufb01ned above! 12 CHAPTER 17 \u2022 S EQUENCE LABELING FOR PARTS OF SPEECH AND NAMED ENTITIES\nfunction VITERBI (observations of len T,state-graph of len N)returns best-path ,path-prob\ncreate a path probability matrix viterbi[N,T]\nforeach state sfrom 1toNdo ; initialization step\nviterbi [s,1] ps\u0003bs(o1)\nbackpointer [s,1] 0\nforeach time step tfrom 2toTdo ; recursion step\nforeach state sfrom 1toNdo\nviterbi [s,t] Nmax\ns0=1viterbi [s0;t\u00001]\u0003as0;s\u0003bs(ot)\nbackpointer [s,t] Nargmax\ns0=1viterbi [s0;t\u00001]\u0003as0;s\u0003bs(ot)\nbestpathprob Nmax\ns=1viterbi [s;T] ; termination step\nbestpathpointer Nargmax\ns=1viterbi [s;T] ; termination step\nbestpath the path starting at state bestpathpointer , that follows backpointer[] to states back in time\nreturn bestpath ,bestpathprob\nFigure 17.10 Viterbi algorithm for \ufb01nding the optimal sequence of tags. Given an observation sequence and\nan HMM l= (A;B), the algorithm returns the state path through the HMM that assigns maximum likelihood\nto the observation sequence.\n17.4.5 The Viterbi Algorithm\nThe decoding algorithm for HMMs is the Viterbi algorithm shown in Fig. 17.10.Viterbi\nalgorithm\nAs an instance of dynamic programming , Viterbi resembles the dynamic program-\nming minimum edit distance algorithm of Chapter 2.\nThe Viterbi algorithm \ufb01rst sets up a probability matrix or lattice , with one col-\numn for each observation otand one row for each state in the state graph. Each col-\numn thus has a cell for each state qiin the single combined automaton. Figure 17.11\nshows an intuition of this lattice for the sentence Janet will back the bill .\nEach cell of the lattice, vt(j), represents the probability that the HMM is in state\njafter seeing the \ufb01rst tobservations and passing through the most probable state\nsequence q1;:::;qt\u00001, given the HMM l. The value of each cell vt(j)is computed\nby recursively taking the most probable path that could lead us to this cell. Formally,\neach cell expresses the probability\nvt(j) = max\nq1;:::;qt\u00001P(q1:::qt\u00001;o1;o2:::ot;qt=jjl) (17.18)\nWe represent the most probable path by taking the maximum over all possible\nprevious state sequences max\nq1;:::;qt\u00001. Like other dynamic programming algorithms,\nViterbi \ufb01lls each cell recursively. Given that we had already computed the probabil-\nity of being in every state at time t\u00001, we compute the Viterbi probability by taking\nthe most probable of the extensions of the paths that lead to the current cell. For a\ngiven state qjat time t, the value vt(j)is computed as\nvt(j) =Nmax\ni=1vt\u00001(i)ai jbj(ot) (17.19)\nThe three factors that are multiplied in Eq. 17.19 for extending the previous paths to\ncompute the Viterbi probability at time tare 17.4 \u2022 HMM P ART-OF-SPEECH TAGGING 13\nJJNNPNNPNNPMDMDMDMDVBVBJJJJJJNNNNRBRBRBRBDTDTDTDT\nNNPJanetwillbackthebillNNVBMDNNVBJJRB\nNNPDTNNVB\nFigure 17.11 A sketch of the lattice for Janet will back the bill , showing the possible tags\n(qi) for each word and highlighting the path corresponding to the correct tag sequence through\nthe hidden states. States (parts of speech) which have a zero probability of generating a\nparticular word according to the Bmatrix (such as the probability that a determiner DT will\nbe realized as Janet ) are greyed out.\nvt\u00001(i)theprevious Viterbi path probability from the previous time step\nai j thetransition probability from previous state qito current state qj\nbj(ot) thestate observation likelihood of the observation symbol otgiven\nthe current state j\n17.4.6 Working through an example\nLet\u2019s tag the sentence Janet will back the bill ; the goal is the correct series of tags\n(see also Fig. 17.11):\n(17.20) Janet/NNP will/MD back/VB the/DT bill/NN\nNNP MD VB JJ NN RB DT\n<s> 0.2767 0.0006 0.0031 0.0453 0.0449 0.0510 0.2026\nNNP 0.3777 0.0110 0.0009 0.0084 0.0584 0.0090 0.0025\nMD 0.0008 0.0002 0.7968 0.0005 0.0008 0.1698 0.0041\nVB 0.0322 0.0005 0.0050 0.0837 0.0615 0.0514 0.2231\nJJ 0.0366 0.0004 0.0001 0.0733 0.4509 0.0036 0.0036\nNN 0.0096 0.0176 0.0014 0.0086 0.1216 0.0177 0.0068\nRB 0.0068 0.0102 0.1011 0.1012 0.0120 0.0728 0.0479\nDT 0.1147 0.0021 0.0002 0.2157 0.4744 0.0102 0.0017\nFigure 17.12 TheAtransition probabilities P(tijti\u00001)computed from the WSJ corpus with-\nout smoothing. Rows are labeled with the conditioning event; thus P(V BjMD)is 0.7968.\n<s>is the start token.\nLet the HMM be de\ufb01ned by the two tables in Fig. 17.12 and Fig. 17.13. Fig-\nure 17.12 lists the ai jprobabilities for transitioning between the hidden states (part-\nof-speech tags). Figure 17.13 expresses the bi(ot)probabilities, the observation\nlikelihoods of words given tags. This table is (slightly simpli\ufb01ed) from counts in the\nWSJ corpus. So the word Janet only appears as an NNP, back has 4 possible parts 14 CHAPTER 17 \u2022 S EQUENCE LABELING FOR PARTS OF SPEECH AND NAMED ENTITIES\nJanet will back the bill\nNNP 0.000032 0 0 0.000048 0\nMD 0 0.308431 0 0 0\nVB 0 0.000028 0.000672 0 0.000028\nJJ 0 0 0.000340 0 0\nNN 0 0.000200 0.000223 0 0.002337\nRB 0 0 0.010446 0 0\nDT 0 0 0 0.506099 0\nFigure 17.13 Observation likelihoods Bcomputed from the WSJ corpus without smooth-\ning, simpli\ufb01ed slightly.\nof speech, and the word thecan appear as a determiner or as an NNP (in titles like\n\u201cSomewhere Over the Rainbow\u201d all words are tagged as NNP).\n\u03c0P(NNP|start) = .28* P(MD|MD)= 0*  P(MD|NNP).000009*.01  = .9e-8 v1(2)=.0006 x 0 = 0v1(1) = .28* .000032 = .000009\ntMDq2q1\no1Janetbillwillo2o3backVBJJv1(3)=.0031 x 0 = 0v1(4)= .045*0=0\no4  *  P(MD|VB) = 0 * P(MD|JJ)= 0P(VB|start) = .0031P(JJ |start) =.045\nbacktraceq3q4\ntheNNq5RBq6DTq7\nv2(2) =max * .308 =2.772e-8v2(5)=max * .0002 = .0000000001\nv2(3)=max * .000028 =     2.5e-13v3(6)=max * .0104v3(5)=max * .000223v3(4)=max * .00034v3(3)=max * .00067v1(5)v1(6)v1(7)\nv2(1)v2(4)v2(6)v2(7)\nbacktrace* P(RB|NN)* P(NN|NN)\nstartstartstartstartstarto5NNPP(MD|start) = .0006\nFigure 17.14 The \ufb01rst few entries in the individual state columns for the Viterbi algorithm. Each cell keeps\nthe probability of the best path so far and a pointer to the previous cell along that path. We have only \ufb01lled out\ncolumns 1 and 2; to avoid clutter most cells with value 0 are left empty. The rest is left as an exercise for the\nreader. After the cells are \ufb01lled in, backtracing from the endstate, we should be able to reconstruct the correct\nstate sequence NNP MD VB DT NN.\nFigure 17.14 shows a \ufb02eshed-out version of the sketch we saw in Fig. 17.11,\nthe Viterbi lattice for computing the best hidden state sequence for the observation\nsequence Janet will back the bill .\nThere are N=5 state columns. We begin in column 1 (for the word Janet ) by\nsetting the Viterbi value in each cell to the product of the ptransition probability (the\nstart probability for that state i, which we get from the <s>entry of Fig. 17.12), and 17.5 \u2022 C ONDITIONAL RANDOM FIELDS (CRF S)15\nthe observation likelihood of the word Janet given the tag for that cell. Most of the\ncells in the column are zero since the word Janet cannot be any of those tags. The\nreader should \ufb01nd this in Fig. 17.14.\nNext, each cell in the willcolumn gets updated. For each state, we compute the\nvalue viterbi [s;t]by taking the maximum over the extensions of all the paths from\nthe previous column that lead to the current cell according to Eq. 17.19. We have\nshown the values for the MD, VB, and NN cells. Each cell gets the max of the 7 val-\nues from the previous column, multiplied by the appropriate transition probability;\nas it happens in this case, most of them are zero from the previous column. The re-\nmaining value is multiplied by the relevant observation probability, and the (trivial)\nmax is taken. In this case the \ufb01nal value, 2.772e-8, comes from the NNP state at the\nprevious column. The reader should \ufb01ll in the rest of the lattice in Fig. 17.14 and\nbacktrace to see whether or not the Viterbi algorithm returns the gold state sequence\nNNP MD VB DT NN.\n17.5 Conditional Random Fields (CRFs)\nWhile the HMM is a useful and powerful model, it turns out that HMMs need a\nnumber of augmentations to achieve high accuracy. For example, in POS tagging\nas in other tasks, we often run into unknown words : proper names and acronymsunknown\nwords\nare created very often, and even new common nouns and verbs enter the language\nat a surprising rate. It would be great to have ways to add arbitrary features to\nhelp with this, perhaps based on capitalization or morphology (words starting with\ncapital letters are likely to be proper nouns, words ending with -edtend to be past\ntense (VBD or VBN), etc.) Or knowing the previous or following words might be a\nuseful feature (if the previous word is the, the current tag is unlikely to be a verb).\nAlthough we could try to hack the HMM to \ufb01nd ways to incorporate some of\nthese, in general it\u2019s hard for generative models like HMMs to add arbitrary features\ndirectly into the model in a clean way. We\u2019ve already seen a model for combining\narbitrary features in a principled way: log-linear models like the logistic regression\nmodel of Chapter 5! But logistic regression isn\u2019t a sequence model; it assigns a class\nto a single observation.\nLuckily, there is a discriminative sequence model based on log-linear models:\ntheconditional random \ufb01eld (CRF ). We\u2019ll describe here the linear chain CRF , CRF\nthe version of the CRF most commonly used for language processing, and the one\nwhose conditioning closely matches the HMM.\nAssuming we have a sequence of input words X=x1:::xnand want to compute\na sequence of output tags Y=y1:::yn. In an HMM to compute the best tag sequence\nthat maximizes P(YjX)we rely on Bayes\u2019 rule and the likelihood P(XjY):\n\u02c6Y=argmax\nYp(YjX)\n=argmax\nYp(XjY)p(Y)\n=argmax\nYY\nip(xijyi)Y\nip(yijyi\u00001) (17.21)\nIn a CRF, by contrast, we compute the posterior p(YjX)directly, training the CRF 16 CHAPTER 17 \u2022 S EQUENCE LABELING FOR PARTS OF SPEECH AND NAMED ENTITIES\nto discriminate among the possible tag sequences:\n\u02c6Y=argmax\nY2YP(YjX) (17.22)\nHowever, the CRF does not compute a probability for each tag at each time step. In-\nstead, at each time step the CRF computes log-linear functions over a set of relevant\nfeatures, and these local features are aggregated and normalized to produce a global\nprobability for the whole sequence.\nLet\u2019s introduce the CRF more formally, again using XandYas the input and\noutput sequences. A CRF is a log-linear model that assigns a probability to an entire\noutput (tag) sequence Y, out of all possible sequences Y, given the entire input\n(word) sequence X. We can think of a CRF as like a giant sequential version of\nthe multinomial logistic regression algorithm we saw for text categorization. Recall\nthat we introduced the feature function fin regular multinomial logistic regression\nfor text categorization as a function of a tuple: the input text xand a single class y\n(page ??). In a CRF, we\u2019re dealing with a sequence, so the function Fmaps an entire\ninput sequence Xand an entire output sequence Yto a feature vector. Let\u2019s assume\nwe have Kfeatures, with a weight wkfor each feature Fk:\np(YjX) =exp KX\nk=1wkFk(X;Y)!\nX\nY02Yexp KX\nk=1wkFk(X;Y0)! (17.23)\nIt\u2019s common to also describe the same equation by pulling out the denominator into\na function Z(X):\np(YjX) =1\nZ(X)exp KX\nk=1wkFk(X;Y)!\n(17.24)\nZ(X) =X\nY02Yexp KX\nk=1wkFk(X;Y0)!\n(17.25)\nWe\u2019ll call these Kfunctions Fk(X;Y)global features , since each one is a property\nof the entire input sequence Xand output sequence Y. We compute them by decom-\nposing into a sum of local features for each position iinY:\nFk(X;Y) =nX\ni=1fk(yi\u00001;yi;X;i) (17.26)\nEach of these local features fkin a linear-chain CRF is allowed to make use of the\ncurrent output token yi, the previous output token yi\u00001, the entire input string X(or\nany subpart of it), and the current position i. This constraint to only depend on\nthe current and previous output tokens yiandyi\u00001are what characterizes a linear\nchain CRF . As we will see, this limitation makes it possible to use versions of thelinear chain\nCRF\nef\ufb01cient Viterbi and Forward-Backwards algorithms from the HMM. A general CRF,\nby contrast, allows a feature to make use of any output token, and are thus necessary\nfor tasks in which the decision depend on distant output tokens, like yi\u00004. General\nCRFs require more complex inference, and are less commonly used for language\nprocessing. 17.5 \u2022 C ONDITIONAL RANDOM FIELDS (CRF S)17\n17.5.1 Features in a CRF POS Tagger\nLet\u2019s look at some of these features in detail, since the reason to use a discriminative\nsequence model is that it\u2019s easier to incorporate a lot of features.2\nAgain, in a linear-chain CRF, each local feature fkat position ican depend on\nany information from: (yi\u00001;yi;X;i). So some legal features representing common\nsituations might be the following:\n1fxi=the;yi=DETg\n1fyi=PROPN, xi+1=Street ,yi\u00001=NUMg\n1fyi=VERB, yi\u00001=AUXg\nFor simplicity, we\u2019ll assume all CRF features take on the value 1 or 0. Above, we\nexplicitly use the notation 1fxgto mean \u201c1 if xis true, and 0 otherwise\u201d. From now\non, we\u2019ll leave off the 1when we de\ufb01ne features, but you can assume each feature\nhas it there implicitly.\nAlthough the idea of what features to use is done by the system designer by hand,\nthe speci\ufb01c features are automatically populated by using feature templates as wefeature\ntemplates\nbrie\ufb02y mentioned in Chapter 5. Here are some templates that only use information\nfrom (yi\u00001;yi;X;i):\nhyi;xii;hyi;yi\u00001i;hyi;xi\u00001;xi+2i\nThese templates automatically populate the set of features from every instance in\nthe training and test set. Thus for our example Janet/NNP will/MD back/VB the/DT\nbill/NN , when xiis the word back, the following features would be generated and\nhave the value 1 (we\u2019ve assigned them arbitrary feature numbers):\nf3743:yi= VB and xi= back\nf156:yi= VB and yi\u00001= MD\nf99732 :yi= VB and xi\u00001= will and xi+2= bill\nIt\u2019s also important to have features that help with unknown words. One of the\nmost important is word shape features, which represent the abstract letter pattern word shape\nof the word by mapping lower-case letters to \u2018x\u2019, upper-case to \u2018X\u2019, numbers to\n\u2019d\u2019, and retaining punctuation. Thus for example I.M.F. would map to X.X.X. and\nDC10-30 would map to XXdd-dd. A second class of shorter word shape features is\nalso used. In these features consecutive character types are removed, so words in all\ncaps map to X, words with initial-caps map to Xx, DC10-30 would be mapped to\nXd-d but I.M.F would still map to X.X.X. Pre\ufb01x and suf\ufb01x features are also useful.\nIn summary, here are some sample feature templates that help with unknown words:\nxicontains a particular pre\ufb01x (perhaps from all pre\ufb01xes of length \u00142)\nxicontains a particular suf\ufb01x (perhaps from all suf\ufb01xes of length \u00142)\nxi\u2019s word shape\nxi\u2019s short word shape\nFor example the word well-dressed might generate the following non-zero val-\nued feature values:\n2Because in HMMs all computation is based on the two probabilities P(tagjtag)andP(wordjtag), if\nwe want to include some source of knowledge into the tagging process, we must \ufb01nd a way to encode\nthe knowledge into one of these two probabilities. Each time we add a feature we have to do a lot of\ncomplicated conditioning which gets harder and harder as we have more and more such features. 18 CHAPTER 17 \u2022 S EQUENCE LABELING FOR PARTS OF SPEECH AND NAMED ENTITIES\npre\ufb01x( xi) =w\npre\ufb01x( xi) =we\nsuf\ufb01x( xi) =ed\nsuf\ufb01x( xi) =d\nword-shape( xi) =xxxx-xxxxxxx\nshort-word-shape( xi) =x-x\nThe known-word templates are computed for every word seen in the training\nset; the unknown word features can also be computed for all words in training, or\nonly on training words whose frequency is below some threshold. The result of the\nknown-word templates and word-signature features is a very large set of features.\nGenerally a feature cutoff is used in which features are thrown out if they have count\n<5 in the training set.\nRemember that in a CRF we don\u2019t learn weights for each of these local features\nfk. Instead, we \ufb01rst sum the values of each local feature (for example feature f3743)\nover the entire sentence, to create each global feature (for example F3743). It is those\nglobal features that will then be multiplied by weight w3743. Thus for training and\ninference there is always a \ufb01xed set of Kfeatures with Kweights, even though the\nlength of each sentence is different.\n17.5.2 Features for CRF Named Entity Recognizers\nA CRF for NER makes use of very similar features to a POS tagger, as shown in\nFigure 17.15.\nidentity of wi, identity of neighboring words\nembeddings for wi, embeddings for neighboring words\npart of speech of wi, part of speech of neighboring words\npresence of wiin agazetteer\nwicontains a particular pre\ufb01x (from all pre\ufb01xes of length \u00144)\nwicontains a particular suf\ufb01x (from all suf\ufb01xes of length \u00144)\nword shape of wi, word shape of neighboring words\nshort word shape of wi, short word shape of neighboring words\ngazetteer features\nFigure 17.15 Typical features for a feature-based NER system.\nOne feature that is especially useful for locations is a gazetteer , a list of place gazetteer\nnames, often providing millions of entries for locations with detailed geographical\nand political information.3This can be implemented as a binary feature indicating a\nphrase appears in the list. Other related resources like name-lists , for example from\nthe United States Census Bureau4, can be used, as can other entity dictionaries like\nlists of corporations or products, although they may not be as helpful as a gazetteer\n(Mikheev et al., 1999).\nThe sample named entity token L\u2019Occitane would generate the following non-\nzero valued feature values (assuming that L\u2019Occitane is neither in the gazetteer nor\nthe census).\n3www.geonames.org\n4www.census.gov 17.5 \u2022 C ONDITIONAL RANDOM FIELDS (CRF S)19\npre\ufb01x( xi) =L suf\ufb01x( xi) =tane\npre\ufb01x( xi) =L' suf\ufb01x( xi) =ane\npre\ufb01x( xi) =L'O suf\ufb01x( xi) =ne\npre\ufb01x( xi) =L'Oc suf\ufb01x( xi) =e\nword-shape( xi) =X'Xxxxxxxx short-word-shape( xi) =X'Xx\nFigure 17.16 illustrates the result of adding part-of-speech tags and some shape\ninformation to our earlier example.\nWords POS Short shape Gazetteer BIO Label\nJane NNP Xx 0 B-PER\nVillanueva NNP Xx 1 I-PER\nof IN x 0 O\nUnited NNP Xx 0 B-ORG\nAirlines NNP Xx 0 I-ORG\nHolding NNP Xx 0 I-ORG\ndiscussed VBD x 0 O\nthe DT x 0 O\nChicago NNP Xx 1 B-LOC\nroute NN x 0 O\n. . . 0 O\nFigure 17.16 Some NER features for a sample sentence, assuming that Chicago and Vil-\nlanueva are listed as locations in a gazetteer. We assume features only take on the values 0 or\n1, so the \ufb01rst POS feature, for example, would be represented as 1fPOS=NNPg.\n17.5.3 Inference and Training for CRFs\nHow do we \ufb01nd the best tag sequence \u02c6Yfor a given input X? We start with Eq. 17.22:\n\u02c6Y=argmax\nY2YP(YjX)\n=argmax\nY2Y1\nZ(X)exp KX\nk=1wkFk(X;Y)!\n(17.27)\n=argmax\nY2Yexp KX\nk=1wknX\ni=1fk(yi\u00001;yi;X;i)!\n(17.28)\n=argmax\nY2YKX\nk=1wknX\ni=1fk(yi\u00001;yi;X;i) (17.29)\n=argmax\nY2YnX\ni=1KX\nk=1wkfk(yi\u00001;yi;X;i) (17.30)\nWe can ignore the exp function and the denominator Z(X), as we do above, because\nexp doesn\u2019t change the argmax, and the denominator Z(X)is constant for a given\nobservation sequence X.\nHow should we decode to \ufb01nd this optimal tag sequence \u02c6 y? Just as with HMMs,\nwe\u2019ll turn to the Viterbi algorithm, which works because, like the HMM, the linear-\nchain CRF depends at each timestep on only one previous output token yi\u00001.\nConcretely, this involves \ufb01lling an N\u0002Tarray with the appropriate values, main-\ntaining backpointers as we proceed. As with HMM Viterbi, when the table is \ufb01lled,\nwe simply follow pointers back from the maximum value in the \ufb01nal column to\nretrieve the desired set of labels. 20 CHAPTER 17 \u2022 S EQUENCE LABELING FOR PARTS OF SPEECH AND NAMED ENTITIES\nThe requisite changes from HMM Viterbi have to do only with how we \ufb01ll each\ncell. Recall from Eq. 17.19 that the recursive step of the Viterbi equation computes\nthe Viterbi value of time tfor state jas\nvt(j) =Nmax\ni=1vt\u00001(i)ai jbj(ot); 1\u0014j\u0014N;1<t\u0014T (17.31)\nwhich is the HMM implementation of\nvt(j) =Nmax\ni=1vt\u00001(i)P(sjjsi)P(otjsj)1\u0014j\u0014N;1<t\u0014T (17.32)\nThe CRF requires only a slight change to this latter formula, replacing the aandb\nprior and likelihood probabilities with the CRF features:\nvt(j) =Nmax\ni=1vt\u00001(i)KX\nk=1wkfk(yt\u00001;yt;X;t)1\u0014j\u0014N;1<t\u0014T(17.33)\nLearning in CRFs relies on the same supervised learning algorithms we presented\nfor logistic regression. Given a sequence of observations, feature functions, and cor-\nresponding outputs, we use stochastic gradient descent to train the weights to maxi-\nmize the log-likelihood of the training corpus. The local nature of linear-chain CRFs\nmeans that the forward-backward algorithm introduced for HMMs in Appendix A\ncan be extended to a CRF version that will ef\ufb01ciently compute the necessary deriva-\ntives. As with logistic regression, L1 or L2 regularization is important.\n17.6 Evaluation of Named Entity Recognition\nPart-of-speech taggers are evaluated by the standard metric of accuracy . Named\nentity recognizers are evaluated by recall ,precision , and F1measure . Recall that\nrecall is the ratio of the number of correctly labeled responses to the total that should\nhave been labeled; precision is the ratio of the number of correctly labeled responses\nto the total labeled; and F-measure is the harmonic mean of the two.\nTo know if the difference between the F 1scores of two NER systems is a signif-\nicant difference, we use the paired bootstrap test, or the similar randomization test\n(Section ??).\nFor named entity tagging, the entity rather than the word is the unit of response.\nThus in the example in Fig. 17.16, the two entities Jane Villanueva andUnited Air-\nlines Holding and the non-entity discussed would each count as a single response.\nThe fact that named entity tagging has a segmentation component which is not\npresent in tasks like text categorization or part-of-speech tagging causes some prob-\nlems with evaluation. For example, a system that labeled Jane but not Jane Vil-\nlanueva as a person would cause two errors, a false positive for O and a false nega-\ntive for I-PER. In addition, using entities as the unit of response but words as the unit\nof training means that there is a mismatch between the training and test conditions.\n17.7 Further Details\nIn this section we summarize a few remaining details of the data and models for\npart-of-speech tagging and NER, beginning with data. Since the algorithms we have 17.7 \u2022 F URTHER DETAILS 21\npresented are supervised, having labeled data is essential for training and testing. A\nwide variety of datasets exist for part-of-speech tagging and/or NER. The Universal\nDependencies (UD) dataset (de Marneffe et al., 2021) has POS tagged corpora in\nover a hundred languages, as do the Penn Treebanks in English, Chinese, and Arabic.\nOntoNotes has corpora labeled for named entities in English, Chinese, and Arabic\n(Hovy et al., 2006). Named entity tagged corpora are also available in particular\ndomains, such as for biomedical (Bada et al., 2012) and literary text (Bamman et al.,\n2019).\n17.7.1 Rule-based Methods\nWhile machine learned (neural or CRF) sequence models are the norm in academic\nresearch, commercial approaches to NER are often based on pragmatic combina-\ntions of lists and rules, with some smaller amount of supervised machine learning\n(Chiticariu et al., 2013). For example in the IBM System T architecture, a user\nspeci\ufb01es declarative constraints for tagging tasks in a formal query language that\nincludes regular expressions, dictionaries, semantic constraints, and other operators,\nwhich the system compiles into an ef\ufb01cient extractor (Chiticariu et al., 2018).\nOne common approach is to make repeated rule-based passes over a text, starting\nwith rules with very high precision but low recall, and, in subsequent stages, using\nmachine learning methods that take the output of the \ufb01rst pass into account (an\napproach \ufb01rst worked out for coreference (Lee et al., 2017)):\n1. First, use high-precision rules to tag unambiguous entity mentions.\n2. Then, search for substring matches of the previously detected names.\n3. Use application-speci\ufb01c name lists to \ufb01nd likely domain-speci\ufb01c mentions.\n4. Finally, apply supervised sequence labeling techniques that use tags from pre-\nvious stages as additional features.\nRule-based methods were also the earliest methods for part-of-speech tagging.\nRule-based taggers like the English Constraint Grammar system (Karlsson et al.\n1995, V outilainen 1999) use a two-stage formalism invented in the 1950s and 1960s:\n(1) a morphological analyzer with tens of thousands of word stem entries returns all\nparts of speech for a word, then (2) a large set of thousands of constraints are applied\nto the input sentence to rule out parts of speech inconsistent with the context.\n17.7.2 POS Tagging for Morphologically Rich Languages\nAugmentations to tagging algorithms become necessary when dealing with lan-\nguages with rich morphology like Czech, Hungarian and Turkish.\nThese productive word-formation processes result in a large vocabulary for these\nlanguages: a 250,000 word token corpus of Hungarian has more than twice as many\nword types as a similarly sized corpus of English (Oravecz and Dienes, 2002), while\na 10 million word token corpus of Turkish contains four times as many word types\nas a similarly sized English corpus (Hakkani-T \u00a8ur et al., 2002). Large vocabular-\nies mean many unknown words, and these unknown words cause signi\ufb01cant per-\nformance degradations in a wide variety of languages (including Czech, Slovene,\nEstonian, and Romanian) (Haji \u02c7c, 2000).\nHighly in\ufb02ectional languages also have much more information than English\ncoded in word morphology, like case (nominative, accusative, genitive) or gender\n(masculine, feminine). Because this information is important for tasks like pars-\ning and coreference resolution, part-of-speech taggers for morphologically rich lan- 22 CHAPTER 17 \u2022 S EQUENCE LABELING FOR PARTS OF SPEECH AND NAMED ENTITIES\nguages need to label words with case and gender information. Tagsets for morpho-\nlogically rich languages are therefore sequences of morphological tags rather than a\nsingle primitive tag. Here\u2019s a Turkish example, in which the word izinhas three pos-\nsible morphological/part-of-speech tags and meanings (Hakkani-T \u00a8ur et al., 2002):\n1. Yerdeki izintemizlenmesi gerek. iz + Noun+A3sg+Pnon+Gen\nThe trace on the \ufb02oor should be cleaned.\n2.\u00a8Uzerinde parmak izinkalmis \u00b8. iz + Noun+A3sg+P2sg+Nom\nYour \ufb01nger print is left on (it).\n3. Ic \u00b8eri girmek ic \u00b8in izinalman gerekiyor. izin + Noun+A3sg+Pnon+Nom\nYou need permission to enter.\nUsing a morphological parse sequence like Noun+A3sg+Pnon+Gen as the part-\nof-speech tag greatly increases the number of parts of speech, and so tagsets can\nbe 4 to 10 times larger than the 50\u2013100 tags we have seen for English. With such\nlarge tagsets, each word needs to be morphologically analyzed to generate the list\nof possible morphological tag sequences (part-of-speech tags) for the word. The\nrole of the tagger is then to disambiguate among these tags. This method also helps\nwith unknown words since morphological parsers can accept unknown stems and\nstill segment the af\ufb01xes properly.\n17.8 Summary\nThis chapter introduced parts of speech andnamed entities , and the tasks of part-\nof-speech tagging andnamed entity recognition :\n\u2022 Languages generally have a small set of closed class words that are highly\nfrequent, ambiguous, and act as function words , and open-class words like\nnouns ,verbs ,adjectives . Various part-of-speech tagsets exist, of between 40\nand 200 tags.\n\u2022Part-of-speech tagging is the process of assigning a part-of-speech label to\neach of a sequence of words.\n\u2022Named entities are words for proper nouns referring mainly to people, places,\nand organizations, but extended to many other types that aren\u2019t strictly entities\nor even proper nouns.\n\u2022 Two common approaches to sequence modeling are a generative approach,\nHMM tagging, and a discriminative approach, CRF tagging. We will see a\nneural approach in following chapters.\n\u2022 The probabilities in HMM taggers are estimated by maximum likelihood es-\ntimation on tag-labeled training corpora. The Viterbi algorithm is used for\ndecoding , \ufb01nding the most likely tag sequence\n\u2022Conditional Random Fields orCRF taggers train a log-linear model that can\nchoose the best tag sequence given an observation sequence, based on features\nthat condition on the output tag, the prior output tag, the entire input sequence,\nand the current timestep. They use the Viterbi algorithm for inference, to\nchoose the best sequence of tags, and a version of the Forward-Backward\nalgorithm (see Appendix A) for training, BIBLIOGRAPHICAL AND HISTORICAL NOTES 23\nBibliographical and Historical Notes\nWhat is probably the earliest part-of-speech tagger was part of the parser in Zellig\nHarris\u2019s Transformations and Discourse Analysis Project (TDAP), implemented be-\ntween June 1958 and July 1959 at the University of Pennsylvania (Harris, 1962),\nalthough earlier systems had used part-of-speech dictionaries. TDAP used 14 hand-\nwritten rules for part-of-speech disambiguation; the use of part-of-speech tag se-\nquences and the relative frequency of tags for a word pre\ufb01gures modern algorithms.\nThe parser was implemented essentially as a cascade of \ufb01nite-state transducers; see\nJoshi and Hopely (1999) and Karttunen (1999) for a reimplementation.\nThe Computational Grammar Coder (CGC) of Klein and Simmons (1963) had\nthree components: a lexicon, a morphological analyzer, and a context disambigua-\ntor. The small 1500-word lexicon listed only function words and other irregular\nwords. The morphological analyzer used in\ufb02ectional and derivational suf\ufb01xes to as-\nsign part-of-speech classes. These were run over words to produce candidate parts\nof speech which were then disambiguated by a set of 500 context rules by relying on\nsurrounding islands of unambiguous words. For example, one rule said that between\nan ARTICLE and a VERB, the only allowable sequences were ADJ-NOUN, NOUN-\nADVERB, or NOUN-NOUN. The TAGGIT tagger (Greene and Rubin, 1971) used\nthe same architecture as Klein and Simmons (1963), with a bigger dictionary and\nmore tags (87). TAGGIT was applied to the Brown corpus and, according to Francis\nand Ku \u02c7cera (1982, p. 9), accurately tagged 77% of the corpus; the remainder of the\nBrown corpus was then tagged by hand. All these early algorithms were based on\na two-stage architecture in which a dictionary was \ufb01rst used to assign each word a\nset of potential parts of speech, and then lists of handwritten disambiguation rules\nwinnowed the set down to a single part of speech per word.\nProbabilities were used in tagging by Stolz et al. (1965) and a complete proba-\nbilistic tagger with Viterbi decoding was sketched by Bahl and Mercer (1976). The\nLancaster-Oslo/Bergen (LOB) corpus, a British English equivalent of the Brown cor-\npus, was tagged in the early 1980\u2019s with the CLAWS tagger (Marshall 1983; Mar-\nshall 1987; Garside 1987), a probabilistic algorithm that approximated a simpli\ufb01ed\nHMM tagger. The algorithm used tag bigram probabilities, but instead of storing the\nword likelihood of each tag, the algorithm marked tags either as rare (P(tagjword)<\n:01)infrequent (P(tagjword)< :10) or normally frequent (P(tagjword)> :10).\nDeRose (1988) developed a quasi-HMM algorithm, including the use of dy-\nnamic programming, although computing P(tjw)P(w)instead of P(wjt)P(w). The\nsame year, the probabilistic PARTS tagger of Church 1988, 1989 was probably the\n\ufb01rst implemented HMM tagger, described correctly in Church (1989), although\nChurch (1988) also described the computation incorrectly as P(tjw)P(w)instead\nofP(wjt)P(w). Church (p.c.) explained that he had simpli\ufb01ed for pedagogical pur-\nposes because using the probability P(tjw)made the idea seem more understandable\nas \u201cstoring a lexicon in an almost standard form\u201d.\nLater taggers explicitly introduced the use of the hidden Markov model (Kupiec\n1992; Weischedel et al. 1993; Sch \u00a8utze and Singer 1994). Merialdo (1994) showed\nthat fully unsupervised EM didn\u2019t work well for the tagging task and that reliance\non hand-labeled data was important. Charniak et al. (1993) showed the importance\nof the most frequent tag baseline; the 92.3% number we give above was from Abney\net al. (1999). See Brants (2000) for HMM tagger implementation details, includ-\ning the extension to trigram contexts, and the use of sophisticated unknown word\nfeatures; its performance is still close to state of the art taggers. 24 CHAPTER 17 \u2022 S EQUENCE LABELING FOR PARTS OF SPEECH AND NAMED ENTITIES\nLog-linear models for POS tagging were introduced by Ratnaparkhi (1996),\nwho introduced a system called MXPOST which implemented a maximum entropy\nMarkov model (MEMM), a slightly simpler version of a CRF. Around the same\ntime, sequence labelers were applied to the task of named entity tagging, \ufb01rst with\nHMMs (Bikel et al., 1997) and MEMMs (McCallum et al., 2000), and then once\nCRFs were developed (Lafferty et al. 2001), they were also applied to NER (Mc-\nCallum and Li, 2003). A wide exploration of features followed (Zhou et al., 2005).\nNeural approaches to NER mainly follow from the pioneering results of Collobert\net al. (2011), who applied a CRF on top of a convolutional net. BiLSTMs with word\nand character-based embeddings as input followed shortly and became a standard\nneural algorithm for NER (Huang et al. 2015, Ma and Hovy 2016, Lample et al.\n2016) followed by the more recent use of Transformers and BERT.\nThe idea of using letter suf\ufb01xes for unknown words is quite old; the early Klein\nand Simmons (1963) system checked all \ufb01nal letter suf\ufb01xes of lengths 1-5. The\nunknown word features described on page 17 come mainly from Ratnaparkhi (1996),\nwith augmentations from Toutanova et al. (2003) and Manning (2011).\nState of the art POS taggers use neural algorithms, either bidirectional RNNs or\nTransformers like BERT; see Chapter 8 to Chapter 11. HMM (Brants 2000; Thede\nand Harper 1999) and CRF tagger accuracies are likely just a tad lower.\nManning (2011) investigates the remaining 2.7% of errors in a high-performing\ntagger (Toutanova et al., 2003). He suggests that a third or half of these remaining\nerrors are due to errors or inconsistencies in the training data, a third might be solv-\nable with richer linguistic models, and for the remainder the task is underspeci\ufb01ed\nor unclear.\nSupervised tagging relies heavily on in-domain training data hand-labeled by\nexperts. Ways to relax this assumption include unsupervised algorithms for cluster-\ning words into part-of-speech-like classes, summarized in Christodoulopoulos et al.\n(2010), and ways to combine labeled and unlabeled data, for example by co-training\n(Clark et al. 2003; S\u00f8gaard 2010).\nSee Householder (1995) for historical notes on parts of speech, and Sampson\n(1987) and Garside et al. (1997) on the provenance of the Brown and other tagsets.\nExercises\n17.1 Find one tagging error in each of the following sentences that are tagged with\nthe Penn Treebank tagset:\n1. I/PRP need/VBP a/DT \ufb02ight/NN from/IN Atlanta/NN\n2. Does/VBZ this/DT \ufb02ight/NN serve/VB dinner/NNS\n3. I/PRP have/VB a/DT friend/NN living/VBG in/IN Denver/NNP\n4. Can/VBP you/PRP list/VB the/DT nonstop/JJ afternoon/NN \ufb02ights/NNS\n17.2 Use the Penn Treebank tagset to tag each word in the following sentences\nfrom Damon Runyon\u2019s short stories. You may ignore punctuation. Some of\nthese are quite dif\ufb01cult; do your best.\n1. It is a nice night.\n2. This crap game is over a garage in Fifty-second Street. . .\n3. . . . Nobody ever takes the newspapers she sells . . .\n4. He is a tall, skinny guy with a long, sad, mean-looking kisser, and a\nmournful voice. EXERCISES 25\n5. . . . I am sitting in Mindy\u2019s restaurant putting on the ge\ufb01llte \ufb01sh, which is\na dish I am very fond of, . . .\n6. When a guy and a doll get to taking peeks back and forth at each other,\nwhy there you are indeed.\n17.3 Now compare your tags from the previous exercise with one or two friend\u2019s\nanswers. On which words did you disagree the most? Why?\n17.4 Implement the \u201cmost likely tag\u201d baseline. Find a POS-tagged training set,\nand use it to compute for each word the tag that maximizes p(tjw). You will\nneed to implement a simple tokenizer to deal with sentence boundaries. Start\nby assuming that all unknown words are NN and compute your error rate on\nknown and unknown words. Now write at least \ufb01ve rules to do a better job of\ntagging unknown words, and show the difference in error rates.\n17.5 Build a bigram HMM tagger. You will need a part-of-speech-tagged corpus.\nFirst split the corpus into a training set and test set. From the labeled training\nset, train the transition and observation probabilities of the HMM tagger di-\nrectly on the hand-tagged data. Then implement the Viterbi algorithm so you\ncan decode a test sentence. Now run your algorithm on the test set. Report its\nerror rate and compare its performance to the most frequent tag baseline.\n17.6 Do an error analysis of your tagger. Build a confusion matrix and investigate\nthe most frequent errors. Propose some features for improving the perfor-\nmance of your tagger on these errors.\n17.7 Develop a set of regular expressions to recognize the character shape features\ndescribed on page 17.\n17.8 The BIO and other labeling schemes given in this chapter aren\u2019t the only\npossible one. For example, the Btag can be reserved only for those situations\nwhere an ambiguity exists between adjacent entities. Propose a new set of\nBIOtags for use with your NER system. Experiment with it and compare its\nperformance with the schemes presented in this chapter.\n17.9 Names of works of art (books, movies, video games, etc.) are quite different\nfrom the kinds of named entities we\u2019ve discussed in this chapter. Collect a\nlist of names of works of art from a particular category from a Web-based\nsource (e.g., gutenberg.org, amazon.com, imdb.com, etc.). Analyze your list\nand give examples of ways that the names in it are likely to be problematic for\nthe techniques described in this chapter.\n17.10 Develop an NER system speci\ufb01c to the category of names that you collected\nin the last exercise. Evaluate your system on a collection of text likely to\ncontain instances of these named entities. 26 Chapter 17 \u2022 Sequence Labeling for Parts of Speech and Named Entities\nAbney, S. P., R. E. Schapire, and Y . Singer. 1999. Boosting\napplied to tagging and PP attachment. EMNLP/VLC .\nBada, M., M. Eckert, D. Evans, K. Garcia, K. Shipley, D. Sit-\nnikov, W. A. Baumgartner, K. B. Cohen, K. Verspoor,\nJ. A. Blake, and L. E. Hunter. 2012. Concept annotation\nin the craft corpus. BMC bioinformatics , 13(1):161.\nBahl, L. R. and R. L. Mercer. 1976. Part of speech as-\nsignment by a statistical decision algorithm. Proceedings\nIEEE International Symposium on Information Theory .\nBamman, D., S. Popat, and S. Shen. 2019. An annotated\ndataset of literary entities. NAACL HLT .\nBikel, D. M., S. Miller, R. Schwartz, and R. Weischedel.\n1997. Nymble: A high-performance learning name-\n\ufb01nder. ANLP .\nBrants, T. 2000. TnT: A statistical part-of-speech tagger.\nANLP .\nBroschart, J. 1997. Why Tongan does it differently. Linguis-\ntic Typology , 1:123\u2013165.\nCharniak, E., C. Hendrickson, N. Jacobson, and\nM. Perkowitz. 1993. Equations for part-of-speech tag-\nging. AAAI .\nChiticariu, L., M. Danilevsky, Y . Li, F. Reiss, and H. Zhu.\n2018. SystemT: Declarative text understanding for enter-\nprise. NAACL HLT , volume 3.\nChiticariu, L., Y . Li, and F. R. Reiss. 2013. Rule-Based In-\nformation Extraction is Dead! Long Live Rule-Based In-\nformation Extraction Systems! EMNLP .\nChristodoulopoulos, C., S. Goldwater, and M. Steedman.\n2010. Two decades of unsupervised POS induction: How\nfar have we come? EMNLP .\nChurch, K. W. 1988. A stochastic parts program and noun\nphrase parser for unrestricted text. ANLP .\nChurch, K. W. 1989. A stochastic parts program and noun\nphrase parser for unrestricted text. ICASSP .\nClark, S., J. R. Curran, and M. Osborne. 2003. Bootstrapping\nPOS-taggers using unlabelled data. CoNLL .\nCollobert, R., J. Weston, L. Bottou, M. Karlen,\nK. Kavukcuoglu, and P. Kuksa. 2011. Natural language\nprocessing (almost) from scratch. JMLR , 12:2493\u20132537.\nDeRose, S. J. 1988. Grammatical category disambiguation\nby statistical optimization. Computational Linguistics ,\n14:31\u201339.\nEvans, N. 2000. Word classes in the world\u2019s languages. In\nG. Booij, C. Lehmann, and J. Mugdan, eds, Morphology:\nA Handbook on In\ufb02ection and Word Formation , 708\u2013732.\nMouton.\nFrancis, W. N. and H. Ku \u02c7cera. 1982. Frequency Analysis of\nEnglish Usage . Houghton Mif\ufb02in, Boston.\nGarside, R. 1987. The CLAWS word-tagging system. In\nR. Garside, G. Leech, and G. Sampson, eds, The Compu-\ntational Analysis of English , 30\u201341. Longman.\nGarside, R., G. Leech, and A. McEnery. 1997. Corpus An-\nnotation . Longman.\nGil, D. 2000. Syntactic categories, cross-linguistic variation\nand universal grammar. In P. M. V ogel and B. Comrie,\neds, Approaches to the Typology of Word Classes , 173\u2013\n216. Mouton.Greene, B. B. and G. M. Rubin. 1971. Automatic grammati-\ncal tagging of English. Department of Linguistics, Brown\nUniversity, Providence, Rhode Island.\nHaji\u02c7c, J. 2000. Morphological tagging: Data vs. dictionaries.\nNAACL .\nHakkani-T \u00a8ur, D., K. O\ufb02azer, and G. T \u00a8ur. 2002. Sta-\ntistical morphological disambiguation for agglutinative\nlanguages. Journal of Computers and Humanities ,\n36(4):381\u2013410.\nHarris, Z. S. 1962. String Analysis of Sentence Structure .\nMouton, The Hague.\nHouseholder, F. W. 1995. Dionysius Thrax, the technai , and\nSextus Empiricus. In E. F. K. Koerner and R. E. Asher,\neds, Concise History of the Language Sciences , 99\u2013103.\nElsevier Science.\nHovy, E. H., M. P. Marcus, M. Palmer, L. A. Ramshaw,\nand R. Weischedel. 2006. OntoNotes: The 90% solution.\nHLT-NAACL .\nHuang, Z., W. Xu, and K. Yu. 2015. Bidirectional LSTM-\nCRF models for sequence tagging. arXiv preprint\narXiv:1508.01991 .\nJoshi, A. K. and P. Hopely. 1999. A parser from antiquity.\nIn A. Kornai, ed., Extended Finite State Models of Lan-\nguage , 6\u201315. Cambridge University Press.\nKarlsson, F., A. V outilainen, J. Heikkil \u00a8a, and A. Anttila, eds.\n1995. Constraint Grammar: A Language-Independent\nSystem for Parsing Unrestricted Text . Mouton de Gruyter.\nKarttunen, L. 1999. Comments on Joshi. In A. Kornai, ed.,\nExtended Finite State Models of Language , 16\u201318. Cam-\nbridge University Press.\nKlein, S. and R. F. Simmons. 1963. A computational ap-\nproach to grammatical coding of English words. Journal\nof the ACM , 10(3):334\u2013347.\nKupiec, J. 1992. Robust part-of-speech tagging using a hid-\nden Markov model. Computer Speech and Language ,\n6:225\u2013242.\nLafferty, J. D., A. McCallum, and F. C. N. Pereira. 2001.\nConditional random \ufb01elds: Probabilistic models for seg-\nmenting and labeling sequence data. ICML .\nLample, G., M. Ballesteros, S. Subramanian, K. Kawakami,\nand C. Dyer. 2016. Neural architectures for named entity\nrecognition. NAACL HLT .\nLee, H., M. Surdeanu, and D. Jurafsky. 2017. A scaffolding\napproach to coreference resolution integrating statistical\nand rule-based models. Natural Language Engineering ,\n23(5):733\u2013762.\nMa, X. and E. H. Hovy. 2016. End-to-end sequence labeling\nvia bi-directional LSTM-CNNs-CRF. ACL.\nManning, C. D. 2011. Part-of-speech tagging from 97% to\n100%: Is it time for some linguistics? CICLing 2011 .\nMarcus, M. P., B. Santorini, and M. A. Marcinkiewicz. 1993.\nBuilding a large annotated corpus of English: The Penn\ntreebank. Computational Linguistics , 19(2):313\u2013330.\nde Marneffe, M.-C., C. D. Manning, J. Nivre, and D. Zeman.\n2021. Universal Dependencies. Computational Linguis-\ntics, 47(2):255\u2013308.\nMarshall, I. 1983. Choice of grammatical word-class with-\nout global syntactic analysis: Tagging words in the LOB\ncorpus. Computers and the Humanities , 17:139\u2013150. Exercises 27\nMarshall, I. 1987. Tag selection using probabilistic meth-\nods. In R. Garside, G. Leech, and G. Sampson, eds, The\nComputational Analysis of English , 42\u201356. Longman.\nMcCallum, A., D. Freitag, and F. C. N. Pereira. 2000. Max-\nimum entropy Markov models for information extraction\nand segmentation. ICML .\nMcCallum, A. and W. Li. 2003. Early results for named\nentity recognition with conditional random \ufb01elds, feature\ninduction and web-enhanced lexicons. CoNLL .\nMerialdo, B. 1994. Tagging English text with a probabilistic\nmodel. Computational Linguistics , 20(2):155\u2013172.\nMikheev, A., M. Moens, and C. Grover. 1999. Named entity\nrecognition without gazetteers. EACL .\nOravecz, C. and P. Dienes. 2002. Ef\ufb01cient stochastic part-\nof-speech tagging for Hungarian. LREC .\nRamshaw, L. A. and M. P. Marcus. 1995. Text chunking\nusing transformation-based learning. Proceedings of the\n3rd Annual Workshop on Very Large Corpora .\nRatnaparkhi, A. 1996. A maximum entropy part-of-speech\ntagger. EMNLP .\nSampson, G. 1987. Alternative grammatical coding systems.\nIn R. Garside, G. Leech, and G. Sampson, eds, The Com-\nputational Analysis of English , 165\u2013183. Longman.\nSch\u00a8utze, H. and Y . Singer. 1994. Part-of-speech tagging us-\ning a variable memory Markov model. ACL.\nS\u00f8gaard, A. 2010. Simple semi-supervised training of part-\nof-speech taggers. ACL.\nStolz, W. S., P. H. Tannenbaum, and F. V . Carstensen. 1965.\nA stochastic approach to the grammatical coding of En-\nglish. CACM , 8(6):399\u2013405.\nThede, S. M. and M. P. Harper. 1999. A second-order hidden\nMarkov model for part-of-speech tagging. ACL.\nToutanova, K., D. Klein, C. D. Manning, and Y . Singer.\n2003. Feature-rich part-of-speech tagging with a cyclic\ndependency network. HLT-NAACL .\nV outilainen, A. 1999. Handcrafted rules. In H. van Halteren,\ned.,Syntactic Wordclass Tagging , 217\u2013246. Kluwer.\nWeischedel, R., M. Meteer, R. Schwartz, L. A. Ramshaw,\nand J. Palmucci. 1993. Coping with ambiguity and un-\nknown words through probabilistic models. Computa-\ntional Linguistics , 19(2):359\u2013382.\nWu, S. and M. Dredze. 2019. Beto, Bentz, Becas: The sur-\nprising cross-lingual effectiveness of BERT. EMNLP .\nZhou, G., J. Su, J. Zhang, and M. Zhang. 2005. Exploring\nvarious knowledge in relation extraction. ACL.", "8_POSNER_intro_May_6_2021.pdf": "Sequence Labeling for Part of Speech and Named EntitiesPart of Speech Tagging Parts of SpeechFrom the earliest linguistic traditions (Yaskaand Panini 5thC. BCE, Aristotle 4thC. BCE), the idea that words can be classified into grammatical categories\u2022part of speech, word classes, POS, POS tags8 parts of speech attributed to Dionysius Thraxof Alexandria (c. 1stC. BCE): \u2022noun, verb, pronoun, preposition, adverb, conjunction, participle, article \u2022These categories are relevant for NLP today. Two classes of words: Open vs. ClosedClosed class words\u2022Relatively fixed membership\u2022Usually functionwords: short, frequent words with grammatical function\u2022determiners: a, an, the\u2022pronouns: she, he, I\u2022prepositions: on, under, over, near, by, \u2026Open class words\u2022Usually contentwords: Nouns, Verbs, Adjectives, Adverbs\u2022Plus interjections: oh, ouch, uh-huh, yes, hello\u2022New nouns and verbs like iPhone or to fax Open class (\"content\") words\nClosed class (\"function\")Nouns\nVerbsProperCommonAuxiliaryMainAdjectivesAdverbsPrepositionsParticlesDeterminersConjunctionsPronouns\u2026 more\u2026 moreJanetItalycat,  catsmangoeatwentcanhadold   green   tastyslowly yesterday\nto withoff   upthe someand orthey itsNumbers122,312oneInterjectionsOw  hello Part-of-Speech TaggingAssigning a part-of-speech to each word in a text. Words often have more than one POS. book:\u2022VERB: (Bookthat flight) \u2022NOUN: (Hand me that book). Part-of-Speech Tagging8.2\u2022PART-OF-SPEECHTAGGING5\nwillNOUNAUXVERBDETNOUNJanetbackthebillPart of Speech Taggerx1x2x3x4x5y1y2y3y4y5\nFigure 8.3The task of part-of-speech tagging: mapping from input wordsx1,x2,. . . ,xntooutput POS tagsy1,y2,. . . ,yn.thought thatyour \ufb02ight was earlier). The goal of POS-tagging is toresolvetheseambiguityresolutionambiguities, choosing the proper tag for the context.Theaccuracyof part-of-speech tagging algorithms (the percentage of test setaccuracytags that match human gold labels) is extremely high. One study found accuraciesover 97% across 15 languages from the Universal Dependency (UD) treebank(Wuand Dredze, 2019). Accuracies on various English treebanks are also 97% (no matterthe algorithm; HMMs, CRFs, BERT perform similarly). This 97% number is alsoabout the human performance on this task, at least for English(Manning, 2011).Types: WSJ BrownUnambiguous(1 tag) 44,432 (86%) 45,799 (85%)Ambiguous(2+ tags) 7,025 (14%) 8,050 (15%)Tokens:Unambiguous(1 tag) 577,421 (45%) 384,349 (33%)Ambiguous(2+ tags) 711,780 (55%) 786,646 (67%)Figure 8.4Tag ambiguity in the Brown and WSJ corpora (Treebank-3 45-tag tagset).We\u2019ll introduce algorithms for the task in the next few sections, but \ufb01rst let\u2019sexplore the task. Exactly how hard is it? Fig.8.4shows that most word types(85-86%) are unambiguous (Janetis always NNP,hesitantlyis always RB). But theambiguous words, though accounting for only 14-15% of the vocabulary, are verycommon, and 55-67% of word tokens in running text are ambiguous. Particularlyambiguous common words includethat,back,down,putandset; here are someexamples of the 6 different parts of speech for the wordback:earnings growth took aback/JJseata small building in theback/NNa clear majority of senatorsback/VBPthe billDave began toback/VBtoward the doorenable the country to buyback/RPdebtI was twenty-oneback/RBthenNonetheless, many words are easy to disambiguate, because their different tagsaren\u2019t equally likely. For example,acan be a determiner or the lettera, but thedeterminer sense is much more likely.This idea suggests a usefulbaseline: given an ambiguous word, choose the tagwhich ismost frequentin the training corpus. This is a key concept:Most Frequent Class Baseline:Always compare a classi\ufb01er against a baseline atleast as good as the most frequent class baseline (assigning each token to the classit occurred in most often in the training set).Map from sequence x1,\u2026,xnof words to y1,\u2026,ynof POS tags  \"Universal Dependencies\" Tagset2CHAPTER8\u2022SEQUENCELABELING FORPARTS OFSPEECH ANDNAMEDENTITIES8.1 (Mostly) English Word ClassesUntil now we have been using part-of-speech terms likenounandverbrather freely.In this section we give more complete de\ufb01nitions. While word classes do havesemantic tendencies\u2014adjectives, for example, often describepropertiesand nounspeople\u2014 parts of speech are de\ufb01ned instead based on their grammatical relationshipwith neighboring words or the morphological properties about their af\ufb01xes.Tag Description ExampleOpen ClassADJAdjective: noun modi\ufb01ers describing propertiesred,young,awesomeADVAdverb: verb modi\ufb01ers of time, place, mannervery,slowly,home,yesterdayNOUNwords for persons, places, things, etc.algorithm,cat,mango,beautyVERBwords for actions and processesdraw,provide,goPROPNProper noun: name of a person, organization, place, etc..Regina,IBM,ColoradoINTJInterjection: exclamation, greeting, yes/no response, etc.oh,um,yes,helloClosed Class WordsADPAdposition (Preposition/Postposition): marks a noun\u2019sspacial, temporal, or other relationin, on, by underAUXAuxiliary: helping verb marking tense, aspect, mood, etc.,can, may, should, areCCONJCoordinating Conjunction: joins two phrases/clausesand,or,butDETDeterminer: marks noun phrase propertiesa, an, the, thisNUMNumeralone, two, \ufb01rst, secondPARTParticle: a preposition-like form used together with a verbup, down, on, off, in, out, at, byPRONPronoun: a shorthand for referring to an entity or eventshe, who, I, othersSCONJSubordinating Conjunction: joins a main clause with asubordinate clause such as a sentential complementthat,whichOtherPUNCTPunctuation\u02d9, , ()SYMSymbols like $ or emoji $, %XOther asdf, qwfgFigure 8.1The 17 parts of speech in the Universal Dependencies tagset(Nivre et al., 2016a). Features canbe added to make \ufb01ner-grained distinctions (with properties like number, case, de\ufb01niteness, and so on).Parts of speech fall into two broad categories:closed classandopen class.closed classopen classClosed classes are those with relatively \ufb01xed membership, such as prepositions\u2014new prepositions are rarely coined. By contrast, nouns and verbs are open classes\u2014new nouns and verbs likeiPhoneorto faxare continually being created or borrowed.Closed class words are generallyfunction wordslikeof,it,and, oryou, which tendfunction wordto be very short, occur frequently, and often have structuring uses in grammar.Four major open classes occur in the languages of the world:nouns(includingproper nouns),verbs,adjectives, andadverbs, as well as the smaller open class ofinterjections. English has all \ufb01ve, although not every language does.Nounsare words for people, places, or things, but include others as well.Com-nounmon nounsinclude concrete terms likecatandmango, abstractions likealgorithmcommon nounandbeauty, and verb-like terms likepacingas inHis pacing to and fro became quiteannoying. Nouns in English can occur with determiners (a goat, its bandwidth)take possessives (IBM\u2019s annual revenue), and may occur in the plural (goats, abaci).Many languages, including English, divide common nouns intocount nounsandcount nounmass nouns. Count nouns can occur in the singular and plural (goat/goats, rela-mass nountionship/relationships) and can be counted (one goat, two goats). Mass nouns areused when something is conceptualized as a homogeneous group. Sosnow, salt, andcommunismare not counted (i.e.,*two snowsor*two communisms).Proper nouns,proper nounlikeRegina,Colorado, andIBM, are names of speci\ufb01c persons or entities.Nivreet al. 2016 Sample \"Tagged\" English sentencesThere/PROwere/VERB70/NUMchildren/NOUNthere/ADV./PUNCPreliminary/ADJfindings/NOUNwere/AUXreported/VERBin/ADPtoday/NOUN\u2019s/PARTNew/PROPNEngland/PROPNJournal/PROPNof/ADPMedicine/PROPN Why Part of Speech Tagging?\u25e6Can be useful for other NLP tasks\u25e6Parsing: POS tagging can improve syntactic parsing\u25e6MT: reordering of adjectives and nouns (say from Spanish to English)\u25e6Sentiment or affective tasks: may want to distinguish adjectives or other POS\u25e6Text-to-speech (how do we pronounce \u201clead\u201dor \"object\"?)\u25e6Or linguistic or language-analytic computational tasks\u25e6Need to control for POS when studying linguistic change like creation of new words, or meaning shift\u25e6Or control for POS in measuring meaning similarity or difference How difficult is POS tagging in English?Roughly 15% of word types are ambiguous\u2022Hence 85% of word types are unambiguous\u2022Janetis always PROPN, hesitantlyis always ADV But those 15% tend to be very common. So ~60% of word tokens are ambiguousE.g., backearnings growth took a back/ADJ seata small building in the back/NOUNa clear majority of senators back/VERB the bill enable the country to buy back/PART debtI was twenty-one back/ADV then  POS tagging performance in EnglishHow many tags are correct?  (Tag accuracy)\u25e6About 97%\u25e6Hasn't changed in the last 10+ years\u25e6HMMs, CRFs, BERT perform similarly .\u25e6Human accuracy about the sameBut baseline is 92%!\u25e6Baseline is performance of stupidest possible method\u25e6\"Most frequent class baseline\" is an important baseline for many tasks\u25e6Tag every word with its most frequent tag\u25e6(and tag unknown words as nouns)\u25e6Partly easy because\u25e6Many words are unambiguous Sources of information for POS taggingJanet willback the billAUX/NOUN/VERB?           NOUN/VERB?Prior probabilities of word/tag\u2022\"will\" is usually an AUXIdentity of neighboring words\u2022\"the\" means the next word is probably not a verbMorphology and wordshape:\u25e6Prefixesunable: un-\u00aeADJ\u25e6Suffixesimportantly: -ly\u00aeADJ\u25e6CapitalizationJanet: CAP\u00aePROPN Standard algorithms for POS taggingSupervised Machine Learning Algorithms:\u2022Hidden Markov Models\u2022Conditional Random Fields (CRF)/ Maximum Entropy Markov Models (MEMM)\u2022Neural sequence models (RNNs or Transformers)\u2022Large Language Models (like BERT), finetunedAll required a hand-labeled training set, all about equal performance (97% on English)All make use of information sources we discussed\u2022Via human created features: HMMs and CRFs\u2022Via representation learning:  Neural LMs Sequence Labeling for Part of Speech and Named EntitiesPart of Speech Tagging Sequence Labeling for Part of Speech and Named EntitiesNamed Entity Recognition (NER) Named Entities\u25e6Named entity, in its core usage, means anything that can be referred to with a proper name. Most common 4 tags:\u25e6PER(Person): \u201cMarie Curie\u201d\u25e6LOC(Location): \u201cNew York City\u201d \u25e6ORG(Organization): \u201cStanford University\u201d\u25e6GPE(Geo-Political Entity): \"Boulder, Colorado\"\u25e6Often multi-word phrases\u25e6But the term is also extended to things that aren't entities:\u25e6dates, times, prices Named Entity taggingThe task of named entity recognition (NER):\u2022find spans of text that constitute proper names\u2022tag the type of the entity.  NER output6CHAPTER8\u2022SEQUENCELABELING FORPARTS OFSPEECH ANDNAMEDENTITIESThe most-frequent-tag baseline has an accuracy of about 92%1. The baselinethus differs from the state-of-the-art and human ceiling (97%) by only 5%.8.3 Named Entities and Named Entity TaggingPart of speech tagging can tell us that words likeJanet,Stanford University, andColoradoare all proper nouns; being a proper noun is a grammatical property ofthese words. But viewed from a semantic perspective, these proper nouns refer todifferent kinds of entities: Janet is a person, Stanford University is an organization,..and Colorado is a location.Anamed entityis, roughly speaking, anything that can be referred to with anamed entityproper name: a person, a location, an organization. The task ofnamed entity recog-nition(NER) is to \ufb01nd spans of text that constitute proper names and tag the type ofnamed entityrecognitionNERthe entity. Four entity tags are most common:PER(person),LOC(location),ORG(organization), orGPE(geo-political entity). However, the termnamed entityiscommonly extended to include things that aren\u2019t entities per se, including dates,times, and other kinds of temporal expressions, and even numerical expressions likeprices. Here\u2019s an example of the output of an NER tagger:Citing high fuel prices,[ORGUnited Airlines]said[TIMEFriday]ithas increased fares by[MONEY$6]per round trip on \ufb02ights to somecities also served by lower-cost carriers.[ORGAmerican Airlines],aunit of[ORGAMR Corp.], immediately matched the move, spokesman[PERTim Wagner]said.[ORGUnited], a unit of[ORGUAL Corp.],said the increase took effect[TIMEThursday]and applies to mostroutes where it competes against discount carriers, such as[LOCChicago]to[LOCDallas]and[LOCDenver]to[LOCSan Francisco].The text contains 13 mentions of named entities including 5 organizations, 4 loca-tions, 2 times, 1 person, and 1 mention of money. Figure8.5shows typical genericnamed entity types. Many applications will also need to use speci\ufb01c entity types likeproteins, genes, commercial products, or works of art.Type Tag Sample Categories Example sentencesPeoplePERpeople, charactersTuringis a giant of computer science.OrganizationORGcompanies, sports teams TheIPCCwarned about the cyclone.LocationLOCregions, mountains, seasMt. Sanitasis inSunshine Canyon.Geo-Political EntityGPEcountries, statesPalo Altois raising the fees for parking.Figure 8.5A list of generic named entity types with the kinds of entities they refer to.Named entity tagging is a useful \ufb01rst step in lots of natural language understand-ing tasks. In sentiment analysis we might want to know a consumer\u2019s sentimenttoward a particular entity. Entities are a useful \ufb01rst stage in question answering,or for linking text to information in structured knowledge sources like Wikipedia.And named entity tagging is also central to natural language understanding tasksof building semantic representations, like extracting events and the relationship be-tween participants.Unlike part-of-speech tagging, where there is no segmentation problem sinceeach word gets one tag, the task of named entity recognition is to \ufb01nd and label1In English, on the WSJ corpus, tested on sections 22-24. Why NER?Sentiment analysis: consumer\u2019s sentiment toward a particular company or person?Question Answering: answer questions about an entity?Information Extraction: Extracting facts about entities from text. Why NER is hard1)Segmentation\u2022In POS tagging, no segmentation problem since each word gets one tag.\u2022In NER we have to find and segment the entities!2)Type ambiguity8.3\u2022NAMEDENTITIES ANDNAMEDENTITYTAGGING7spansof text, and is dif\ufb01cult partly because of the ambiguity of segmentation; weneed to decide what\u2019s an entity and what isn\u2019t, and where the boundaries are. Indeed,most words in a text will not be named entities. Another dif\ufb01culty is caused by typeambiguity. The mentionJFKcan refer to a person, the airport in New York, or anynumber of schools, bridges, and streets around the United States. Some examples ofthis kind of cross-type confusion are given in Figure8.6.[PERWashington] was born into slavery on the farm of James Burroughs.[ORGWashington] went up 2 games to 1 in the four-game series.Blair arrived in [LOCWashington] for what may well be his last state visit.In June, [GPEWashington] passed a primary seatbelt law.Figure 8.6Examples of type ambiguities in the use of the nameWashington.The standard approach to sequence labeling for a span-recognition problem likeNER isBIOtagging(Ramshaw and Marcus, 1995). This is a method that allows usto treat NER like a word-by-word sequence labeling task, via tags that capture boththe boundary and the named entity type. Consider the following sentence:[PERJane Villanueva]of[ORGUnited], a unit of[ORGUnited AirlinesHolding], said the fare applies to the[LOCChicago]route.Figure8.7shows the same excerpt represented withBIOtagging, as well asBIOvariants calledIOtagging andBIOEStagging. In BIO tagging we label any tokenthatbeginsa span of interest with the labelB, tokens that occurinsidea span aretagged with anI, and any tokens outside of any span of interest are labeledO. Whilethere is only oneOtag, we\u2019ll have distinctBandItags for each named entity class.The number of tags is thus 2n+1 tags, wherenis the number of entity types. BIOtagging can represent exactly the same information as the bracketed notation, but hasthe advantage that we can represent the task in the same simple sequence modelingway as part-of-speech tagging: assigning a single labelyito each input wordxi:WordsIO LabelBIO LabelBIOES LabelJaneI-PERB-PERB-PERVillanuevaI-PERI-PERE-PERofOOOUnitedI-ORGB-ORGB-ORGAirlinesI-ORGI-ORGI-ORGHoldingI-ORGI-ORGE-ORGdiscussedOOOtheOOOChicagoI-LOCB-LOCS-LOCrouteOOO.OOOFigure 8.7NER as a sequence model, showing IO, BIO, and BIOES taggings.We\u2019ve also shown two variant tagging schemes: IO tagging, which loses someinformation by eliminating the B tag, and BIOES tagging, which adds an end tagEfor the end of a span, and a span tagSfor a span consisting of only one word.A sequence labeler (HMM, CRF, RNN, Transformer, etc.) is trained to label eachtoken in a text with tags that indicate the presence (or absence) of particular kindsof named entities. BIO TaggingHow can we turn this structured problem into a sequence problem like POS tagging, with one label per word?[PER Jane Villanueva] of [ORG United] , a unit of [ORG United Airlines Holding] , said the fare applies to the [LOC Chicago ] route.  BIO Tagging[PER Jane Villanueva] of [ORG United] , a unit of [ORG United Airlines Holding] , said the fare applies to the [LOC Chicago ] route. 8.3\u2022NAMEDENTITIES ANDNAMEDENTITYTAGGING7spansof text, and is dif\ufb01cult partly because of the ambiguity of segmentation; weneed to decide what\u2019s an entity and what isn\u2019t, and where the boundaries are. Indeed,most words in a text will not be named entities. Another dif\ufb01culty is caused by typeambiguity. The mentionJFKcan refer to a person, the airport in New York, or anynumber of schools, bridges, and streets around the United States. Some examples ofthis kind of cross-type confusion are given in Figure8.6.[PERWashington] was born into slavery on the farm of James Burroughs.[ORGWashington] went up 2 games to 1 in the four-game series.Blair arrived in [LOCWashington] for what may well be his last state visit.In June, [GPEWashington] passed a primary seatbelt law.Figure 8.6Examples of type ambiguities in the use of the nameWashington.The standard approach to sequence labeling for a span-recognition problem likeNER isBIOtagging(Ramshaw and Marcus, 1995). This is a method that allows usto treat NER like a word-by-word sequence labeling task, via tags that capture boththe boundary and the named entity type. Consider the following sentence:[PERJane Villanueva]of[ORGUnited], a unit of[ORGUnited AirlinesHolding], said the fare applies to the[LOCChicago]route.Figure8.7shows the same excerpt represented withBIOtagging, as well asBIOvariants calledIOtagging andBIOEStagging. In BIO tagging we label any tokenthatbeginsa span of interest with the labelB, tokens that occurinsidea span aretagged with anI, and any tokens outside of any span of interest are labeledO. Whilethere is only oneOtag, we\u2019ll have distinctBandItags for each named entity class.The number of tags is thus 2n+1 tags, wherenis the number of entity types. BIOtagging can represent exactly the same information as the bracketed notation, but hasthe advantage that we can represent the task in the same simple sequence modelingway as part-of-speech tagging: assigning a single labelyito each input wordxi:WordsIO LabelBIO LabelBIOES LabelJaneI-PERB-PERB-PERVillanuevaI-PERI-PERE-PERofOOOUnitedI-ORGB-ORGB-ORGAirlinesI-ORGI-ORGI-ORGHoldingI-ORGI-ORGE-ORGdiscussedOOOtheOOOChicagoI-LOCB-LOCS-LOCrouteOOO.OOOFigure 8.7NER as a sequence model, showing IO, BIO, and BIOES taggings.We\u2019ve also shown two variant tagging schemes: IO tagging, which loses someinformation by eliminating the B tag, and BIOES tagging, which adds an end tagEfor the end of a span, and a span tagSfor a span consisting of only one word.A sequence labeler (HMM, CRF, RNN, Transformer, etc.) is trained to label eachtoken in a text with tags that indicate the presence (or absence) of particular kindsof named entities.8.3\u2022NAMEDENTITIES ANDNAMEDENTITYTAGGING7spansof text, and is dif\ufb01cult partly because of the ambiguity of segmentation; weneed to decide what\u2019s an entity and what isn\u2019t, and where the boundaries are. Indeed,most words in a text will not be named entities. Another dif\ufb01culty is caused by typeambiguity. The mentionJFKcan refer to a person, the airport in New York, or anynumber of schools, bridges, and streets around the United States. Some examples ofthis kind of cross-type confusion are given in Figure8.6.[PERWashington] was born into slavery on the farm of James Burroughs.[ORGWashington] went up 2 games to 1 in the four-game series.Blair arrived in [LOCWashington] for what may well be his last state visit.In June, [GPEWashington] passed a primary seatbelt law.Figure 8.6Examples of type ambiguities in the use of the nameWashington.The standard approach to sequence labeling for a span-recognition problem likeNER isBIOtagging(Ramshaw and Marcus, 1995). This is a method that allows usto treat NER like a word-by-word sequence labeling task, via tags that capture boththe boundary and the named entity type. Consider the following sentence:[PERJane Villanueva]of[ORGUnited], a unit of[ORGUnited AirlinesHolding], said the fare applies to the[LOCChicago]route.Figure8.7shows the same excerpt represented withBIOtagging, as well asBIOvariants calledIOtagging andBIOEStagging. In BIO tagging we label any tokenthatbeginsa span of interest with the labelB, tokens that occurinsidea span aretagged with anI, and any tokens outside of any span of interest are labeledO. Whilethere is only oneOtag, we\u2019ll have distinctBandItags for each named entity class.The number of tags is thus 2n+1 tags, wherenis the number of entity types. BIOtagging can represent exactly the same information as the bracketed notation, but hasthe advantage that we can represent the task in the same simple sequence modelingway as part-of-speech tagging: assigning a single labelyito each input wordxi:WordsIO LabelBIO LabelBIOES LabelJaneI-PERB-PERB-PERVillanuevaI-PERI-PERE-PERofOOOUnitedI-ORGB-ORGB-ORGAirlinesI-ORGI-ORGI-ORGHoldingI-ORGI-ORGE-ORGdiscussedOOOtheOOOChicagoI-LOCB-LOCS-LOCrouteOOO.OOOFigure 8.7NER as a sequence model, showing IO, BIO, and BIOES taggings.We\u2019ve also shown two variant tagging schemes: IO tagging, which loses someinformation by eliminating the B tag, and BIOES tagging, which adds an end tagEfor the end of a span, and a span tagSfor a span consisting of only one word.A sequence labeler (HMM, CRF, RNN, Transformer, etc.) is trained to label eachtoken in a text with tags that indicate the presence (or absence) of particular kindsof named entities.Now we have one tag per token!!! BIO TaggingB: token that begins a spanI: tokens inside a spanO: tokens outside of any span# of tags (where n is #entity types):1 O tag, nB tags, nI tagstotal of 2n+18.3\u2022NAMEDENTITIES ANDNAMEDENTITYTAGGING7spansof text, and is dif\ufb01cult partly because of the ambiguity of segmentation; weneed to decide what\u2019s an entity and what isn\u2019t, and where the boundaries are. Indeed,most words in a text will not be named entities. Another dif\ufb01culty is caused by typeambiguity. The mentionJFKcan refer to a person, the airport in New York, or anynumber of schools, bridges, and streets around the United States. Some examples ofthis kind of cross-type confusion are given in Figure8.6.[PERWashington] was born into slavery on the farm of James Burroughs.[ORGWashington] went up 2 games to 1 in the four-game series.Blair arrived in [LOCWashington] for what may well be his last state visit.In June, [GPEWashington] passed a primary seatbelt law.Figure 8.6Examples of type ambiguities in the use of the nameWashington.The standard approach to sequence labeling for a span-recognition problem likeNER isBIOtagging(Ramshaw and Marcus, 1995). This is a method that allows usto treat NER like a word-by-word sequence labeling task, via tags that capture boththe boundary and the named entity type. Consider the following sentence:[PERJane Villanueva]of[ORGUnited], a unit of[ORGUnited AirlinesHolding], said the fare applies to the[LOCChicago]route.Figure8.7shows the same excerpt represented withBIOtagging, as well asBIOvariants calledIOtagging andBIOEStagging. In BIO tagging we label any tokenthatbeginsa span of interest with the labelB, tokens that occurinsidea span aretagged with anI, and any tokens outside of any span of interest are labeledO. Whilethere is only oneOtag, we\u2019ll have distinctBandItags for each named entity class.The number of tags is thus 2n+1 tags, wherenis the number of entity types. BIOtagging can represent exactly the same information as the bracketed notation, but hasthe advantage that we can represent the task in the same simple sequence modelingway as part-of-speech tagging: assigning a single labelyito each input wordxi:WordsIO LabelBIO LabelBIOES LabelJaneI-PERB-PERB-PERVillanuevaI-PERI-PERE-PERofOOOUnitedI-ORGB-ORGB-ORGAirlinesI-ORGI-ORGI-ORGHoldingI-ORGI-ORGE-ORGdiscussedOOOtheOOOChicagoI-LOCB-LOCS-LOCrouteOOO.OOOFigure 8.7NER as a sequence model, showing IO, BIO, and BIOES taggings.We\u2019ve also shown two variant tagging schemes: IO tagging, which loses someinformation by eliminating the B tag, and BIOES tagging, which adds an end tagEfor the end of a span, and a span tagSfor a span consisting of only one word.A sequence labeler (HMM, CRF, RNN, Transformer, etc.) is trained to label eachtoken in a text with tags that indicate the presence (or absence) of particular kindsof named entities.8.3\u2022NAMEDENTITIES ANDNAMEDENTITYTAGGING7spansof text, and is dif\ufb01cult partly because of the ambiguity of segmentation; weneed to decide what\u2019s an entity and what isn\u2019t, and where the boundaries are. Indeed,most words in a text will not be named entities. Another dif\ufb01culty is caused by typeambiguity. The mentionJFKcan refer to a person, the airport in New York, or anynumber of schools, bridges, and streets around the United States. Some examples ofthis kind of cross-type confusion are given in Figure8.6.[PERWashington] was born into slavery on the farm of James Burroughs.[ORGWashington] went up 2 games to 1 in the four-game series.Blair arrived in [LOCWashington] for what may well be his last state visit.In June, [GPEWashington] passed a primary seatbelt law.Figure 8.6Examples of type ambiguities in the use of the nameWashington.The standard approach to sequence labeling for a span-recognition problem likeNER isBIOtagging(Ramshaw and Marcus, 1995). This is a method that allows usto treat NER like a word-by-word sequence labeling task, via tags that capture boththe boundary and the named entity type. Consider the following sentence:[PERJane Villanueva]of[ORGUnited], a unit of[ORGUnited AirlinesHolding], said the fare applies to the[LOCChicago]route.Figure8.7shows the same excerpt represented withBIOtagging, as well asBIOvariants calledIOtagging andBIOEStagging. In BIO tagging we label any tokenthatbeginsa span of interest with the labelB, tokens that occurinsidea span aretagged with anI, and any tokens outside of any span of interest are labeledO. Whilethere is only oneOtag, we\u2019ll have distinctBandItags for each named entity class.The number of tags is thus 2n+1 tags, wherenis the number of entity types. BIOtagging can represent exactly the same information as the bracketed notation, but hasthe advantage that we can represent the task in the same simple sequence modelingway as part-of-speech tagging: assigning a single labelyito each input wordxi:WordsIO LabelBIO LabelBIOES LabelJaneI-PERB-PERB-PERVillanuevaI-PERI-PERE-PERofOOOUnitedI-ORGB-ORGB-ORGAirlinesI-ORGI-ORGI-ORGHoldingI-ORGI-ORGE-ORGdiscussedOOOtheOOOChicagoI-LOCB-LOCS-LOCrouteOOO.OOOFigure 8.7NER as a sequence model, showing IO, BIO, and BIOES taggings.We\u2019ve also shown two variant tagging schemes: IO tagging, which loses someinformation by eliminating the B tag, and BIOES tagging, which adds an end tagEfor the end of a span, and a span tagSfor a span consisting of only one word.A sequence labeler (HMM, CRF, RNN, Transformer, etc.) is trained to label eachtoken in a text with tags that indicate the presence (or absence) of particular kindsof named entities. BIO Tagging variants: IO and BIOES[PER Jane Villanueva] of [ORG United] , a unit of [ORG United Airlines Holding] , said the fare applies to the [LOC Chicago ] route. 8.3\u2022NAMEDENTITIES ANDNAMEDENTITYTAGGING7spansof text, and is dif\ufb01cult partly because of the ambiguity of segmentation; weneed to decide what\u2019s an entity and what isn\u2019t, and where the boundaries are. Indeed,most words in a text will not be named entities. Another dif\ufb01culty is caused by typeambiguity. The mentionJFKcan refer to a person, the airport in New York, or anynumber of schools, bridges, and streets around the United States. Some examples ofthis kind of cross-type confusion are given in Figure8.6.[PERWashington] was born into slavery on the farm of James Burroughs.[ORGWashington] went up 2 games to 1 in the four-game series.Blair arrived in [LOCWashington] for what may well be his last state visit.In June, [GPEWashington] passed a primary seatbelt law.Figure 8.6Examples of type ambiguities in the use of the nameWashington.The standard approach to sequence labeling for a span-recognition problem likeNER isBIOtagging(Ramshaw and Marcus, 1995). This is a method that allows usto treat NER like a word-by-word sequence labeling task, via tags that capture boththe boundary and the named entity type. Consider the following sentence:[PERJane Villanueva]of[ORGUnited], a unit of[ORGUnited AirlinesHolding], said the fare applies to the[LOCChicago]route.Figure8.7shows the same excerpt represented withBIOtagging, as well asBIOvariants calledIOtagging andBIOEStagging. In BIO tagging we label any tokenthatbeginsa span of interest with the labelB, tokens that occurinsidea span aretagged with anI, and any tokens outside of any span of interest are labeledO. Whilethere is only oneOtag, we\u2019ll have distinctBandItags for each named entity class.The number of tags is thus 2n+1 tags, wherenis the number of entity types. BIOtagging can represent exactly the same information as the bracketed notation, but hasthe advantage that we can represent the task in the same simple sequence modelingway as part-of-speech tagging: assigning a single labelyito each input wordxi:WordsIO LabelBIO LabelBIOES LabelJaneI-PERB-PERB-PERVillanuevaI-PERI-PERE-PERofOOOUnitedI-ORGB-ORGB-ORGAirlinesI-ORGI-ORGI-ORGHoldingI-ORGI-ORGE-ORGdiscussedOOOtheOOOChicagoI-LOCB-LOCS-LOCrouteOOO.OOOFigure 8.7NER as a sequence model, showing IO, BIO, and BIOES taggings.We\u2019ve also shown two variant tagging schemes: IO tagging, which loses someinformation by eliminating the B tag, and BIOES tagging, which adds an end tagEfor the end of a span, and a span tagSfor a span consisting of only one word.A sequence labeler (HMM, CRF, RNN, Transformer, etc.) is trained to label eachtoken in a text with tags that indicate the presence (or absence) of particular kindsof named entities. Standard algorithms for NERSupervised Machine Learning given a human-labeled training set of text annotated with tags\u2022Hidden Markov Models\u2022Conditional Random Fields (CRF)/ Maximum Entropy Markov Models (MEMM)\u2022Neural sequence models (RNNs or Transformers)\u2022Large Language Models (like BERT), finetuned Sequence Labeling for Part of Speech and Named EntitiesNamed Entity Recognition (NER)", "18.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\n18Context-Free Grammars and\nConstituency Parsing\nBecause the Night by Bruce Springsteen and Patti Smith\nThe Fire Next Time by James Baldwin\nIf on a winter\u2019s night a traveler by Italo Calvino\nLove Actually by Richard Curtis\nSuddenly Last Summer by Tennessee Williams\nA Scanner Darkly by Philip K. Dick\nSix titles that are not constituents, from Geoffrey K. Pullum on\nLanguage Log (who was pointing out their incredible rarity).\nOne morning I shot an elephant in my pajamas.\nHow he got into my pajamas I don\u2019t know.\nGroucho Marx, Animal Crackers , 1930\nThe study of grammar has an ancient pedigree. The grammar of Sanskrit was\ndescribed by the Indian grammarian P \u00afan.ini sometime between the 7th and 4th cen-\nturies BCE, in his famous treatise the As .t.\u00afadhy \u00afay\u00af\u0131 (\u20188 books\u2019). And our word syntax syntax\ncomes from the Greek s\u00b4yntaxis , meaning \u201csetting out together or arrangement\u201d, and\nrefers to the way words are arranged together. We have seen syntactic notions in pre-\nvious chapters like the use of part-of-speech categories (Chapter 17). In this chapter\nand the next one we introduce formal models for capturing more sophisticated no-\ntions of grammatical structure and algorithms for parsing these structures.\nOur focus in this chapter is context-free grammars and the CKY algorithm\nfor parsing them. Context-free grammars are the backbone of many formal mod-\nels of the syntax of natural language (and, for that matter, of computer languages).\nSyntactic parsing is the task of assigning a syntactic structure to a sentence. Parse\ntrees (whether for context-free grammars or for the dependency or CCG formalisms\nwe introduce in following chapters) can be used in applications such as grammar\nchecking : sentence that cannot be parsed may have grammatical errors (or at least\nbe hard to read). Parse trees can be an intermediate stage of representation for for-\nmal semantic analysis . And parsers and the grammatical structure they assign a\nsentence are a useful text analysis tool for text data science applications that require\nmodeling the relationship of elements in sentences.\nIn this chapter we introduce context-free grammars, give a small sample gram-\nmar of English, introduce more formal de\ufb01nitions of context-free grammars and\ngrammar normal form, and talk about treebanks : corpora that have been anno-\ntated with syntactic structure. We then discuss parse ambiguity and the problems\nit presents, and turn to parsing itself, giving the famous Cocke-Kasami-Younger\n(CKY) algorithm (Kasami 1965, Younger 1967), the standard dynamic program-\nming approach to syntactic parsing. The CKY algorithm returns an ef\ufb01cient repre-\nsentation of the set of parse trees for a sentence, but doesn\u2019t tell us which parse tree\nis the right one. For that, we need to augment CKY with scores for each possible\nconstituent. We\u2019ll see how to do this with neural span-based parsers. Finally, we\u2019ll\nintroduce the standard set of metrics for evaluating parser accuracy. 2CHAPTER 18 \u2022 C ONTEXT -FREE GRAMMARS AND CONSTITUENCY PARSING\n18.1 Constituency\nSyntactic constituency is the idea that groups of words can behave as single units,\nor constituents. Part of developing a grammar involves building an inventory of the\nconstituents in the language. How do words group together in English? Consider\nthenoun phrase , a sequence of words surrounding at least one noun. Here are some noun phrase\nexamples of noun phrases (thanks to Damon Runyon):\nHarry the Horse a high-class spot such as Mindy\u2019s\nthe Broadway coppers the reason he comes into the Hot Box\nthey three parties from Brooklyn\nWhat evidence do we have that these words group together (or \u201cform constituents\u201d)?\nOne piece of evidence is that they can all appear in similar syntactic environments,\nfor example, before a verb.\nthree parties from Brooklyn arrive . . .\na high-class spot such as Mindy\u2019s attracts . . .\nthe Broadway coppers love. . .\nthey sit\nBut while the whole noun phrase can occur before a verb, this is not true of each\nof the individual words that make up a noun phrase. The following are not grammat-\nical sentences of English (recall that we use an asterisk (*) to mark fragments that\nare not grammatical English sentences):\n*from arrive . . .*asattracts . . .\n*the is. . . *spot sat. . .\nThus, to correctly describe facts about the ordering of these words in English, we\nmust be able to say things like \u201c Noun Phrases can occur before verbs \u201d. Let\u2019s now\nsee how to do this in a more formal way!\n18.2 Context-Free Grammars\nA widely used formal system for modeling constituent structure in natural lan-\nguage is the context-free grammar , orCFG . Context-free grammars are also called CFG\nphrase-structure grammars , and the formalism is equivalent to Backus-Naur form ,\norBNF . The idea of basing a grammar on constituent structure dates back to the psy-\nchologist Wilhelm Wundt (1900) but was not formalized until Chomsky (1956) and,\nindependently, Backus (1959).\nA context-free grammar consists of a set of rules orproductions , each of which rules\nexpresses the ways that symbols of the language can be grouped and ordered to-\ngether, and a lexicon of words and symbols. For example, the following productions lexicon\nexpress that an NP(ornoun phrase ) can be composed of either a ProperNoun or NP\na determiner ( Det) followed by a Nominal ; aNominal in turn can consist of one or 18.2 \u2022 C ONTEXT -FREE GRAMMARS 3\nmore Noun s.1\nNP!Det Nominal\nNP!ProperNoun\nNominal!NounjNominal Noun\nContext-free rules can be hierarchically embedded, so we can combine the previous\nrules with others, like the following, that express facts about the lexicon:\nDet!a\nDet!the\nNoun!\ufb02ight\nThe symbols that are used in a CFG are divided into two classes. The symbols\nthat correspond to words in the language (\u201cthe\u201d, \u201cnightclub\u201d) are called terminal terminal\nsymbols; the lexicon is the set of rules that introduce these terminal symbols. The\nsymbols that express abstractions over these terminals are called non-terminals . In non-terminal\neach context-free rule, the item to the right of the arrow ( !) is an ordered list of one\nor more terminals and non-terminals; to the left of the arrow is a single non-terminal\nsymbol expressing some cluster or generalization. The non-terminal associated with\neach word in the lexicon is its lexical category, or part of speech.\nA CFG can be thought of in two ways: as a device for generating sentences\nand as a device for assigning a structure to a given sentence. Viewing a CFG as a\ngenerator, we can read the !arrow as \u201crewrite the symbol on the left with the string\nof symbols on the right\u201d.\nSo starting from the symbol: NP\nwe can use our \ufb01rst rule to rewrite NPas: Det Nominal\nand then rewrite Nominal as: Noun\nand \ufb01nally rewrite these parts-of-speech as: a \ufb02ight\nWe say the string a \ufb02ight can be derived from the non-terminal NP. Thus, a CFG\ncan be used to generate a set of strings. This sequence of rule expansions is called a\nderivation of the string of words. It is common to represent a derivation by a parse derivation\ntree (commonly shown inverted with the root at the top). Figure 18.1 shows the tree parse tree\nrepresentation of this derivation.\nNP\nNom\nNoun\n\ufb02ightDet\na\nFigure 18.1 A parse tree for \u201ca \ufb02ight\u201d.\nIn the parse tree shown in Fig. 18.1, we can say that the node NPdominates dominates\nall the nodes in the tree ( Det,Nom ,Noun ,a,\ufb02ight ). We can say further that it\nimmediately dominates the nodes DetandNom .\nThe formal language de\ufb01ned by a CFG is the set of strings that are derivable\nfrom the designated start symbol . Each grammar must have one designated start start symbol\n1When talking about these rules we can pronounce the rightarrow !as \u201cgoes to\u201d, and so we might\nread the \ufb01rst rule above as \u201cNP goes to Det Nominal\u201d. 4CHAPTER 18 \u2022 C ONTEXT -FREE GRAMMARS AND CONSTITUENCY PARSING\nsymbol, which is often called S. Since context-free grammars are often used to de\ufb01ne\nsentences, Sis usually interpreted as the \u201csentence\u201d node, and the set of strings that\nare derivable from Sis the set of sentences in some simpli\ufb01ed version of English.\nLet\u2019s add a few additional rules to our inventory. The following rule expresses\nthe fact that a sentence can consist of a noun phrase followed by a verb phrase : verb phrase\nS!NP VP I prefer a morning \ufb02ight\nA verb phrase in English consists of a verb followed by assorted other things;\nfor example, one kind of verb phrase consists of a verb followed by a noun phrase:\nVP!Verb NP prefer a morning \ufb02ight\nOr the verb may be followed by a noun phrase and a prepositional phrase:\nVP!Verb NP PP leave Boston in the morning\nOr the verb phrase may have a verb followed by a prepositional phrase alone:\nVP!Verb PP leaving on Thursday\nA prepositional phrase generally has a preposition followed by a noun phrase.\nFor example, a common type of prepositional phrase in the ATIS corpus is used to\nindicate location or direction:\nPP!Preposition NP from Los Angeles\nTheNPinside a PPneed not be a location; PPs are often used with times and\ndates, and with other nouns as well; they can be arbitrarily complex. Here are ten\nexamples from the ATIS corpus:\nto Seattle on these \ufb02ights\nin Minneapolis about the ground transportation in Chicago\non Wednesday of the round trip \ufb02ight on United Airlines\nin the evening of the AP \ufb01fty seven \ufb02ight\non the ninth of July with a stopover in Nashville\nFigure 18.2 gives a sample lexicon, and Fig. 18.3 summarizes the grammar rules\nwe\u2019ve seen so far, which we\u2019ll call L0. Note that we can use the or-symbol jto\nindicate that a non-terminal has alternate possible expansions.\nNoun!\ufb02ightsj\ufb02ightjbreezejtripjmorning\nVerb!isjpreferjlikejneedjwantj\ufb02yjdo\nAdjective!cheapestjnon-stopj\ufb01rstjlatest\njotherjdirect\nPronoun!mejIjyoujit\nProper-Noun!AlaskajBaltimorejLos Angeles\njChicagojUnitedjAmerican\nDeterminer!thejajanjthisjthesejthat\nPreposition!fromjtojonjnearjin\nConjunction!andjorjbut\nFigure 18.2 The lexicon for L0.\nWe can use this grammar to generate sentences of this \u201cATIS-language\u201d. We\nstart with S, expand it to NP VP , then choose a random expansion of NP(let\u2019s say, to 18.2 \u2022 C ONTEXT -FREE GRAMMARS 5\nGrammar Rules Examples\nS!NP VP I + want a morning \ufb02ight\nNP!Pronoun I\njProper-Noun Los Angeles\njDet Nominal a + \ufb02ight\nNominal!Nominal Noun morning + \ufb02ight\njNoun \ufb02ights\nVP!Verb do\njVerb NP want + a \ufb02ight\njVerb NP PP leave + Boston + in the morning\njVerb PP leaving + on Thursday\nPP!Preposition NP from + Los Angeles\nFigure 18.3 The grammar for L0, with example phrases for each rule.\nS\nVP\nNP\nNom\nNoun\n\ufb02ightNom\nNoun\nmorningDet\naVerb\npreferNP\nPro\nI\nFigure 18.4 The parse tree for \u201cI prefer a morning \ufb02ight\u201d according to grammar L0.\nI), and a random expansion of VP(let\u2019s say, to Verb NP ), and so on until we generate\nthe string I prefer a morning \ufb02ight . Figure 18.4 shows a parse tree that represents a\ncomplete derivation of I prefer a morning \ufb02ight .\nWe can also represent a parse tree in a more compact format called bracketed\nnotation ; here is the bracketed representation of the parse tree of Fig. 18.4:bracketed\nnotation\n(18.1) [ S[NP[ProI]] [VP[Vprefer] [ NP[Deta] [Nom [Nmorning] [ Nom [N\ufb02ight]]]]]]\nA CFG like that of L0de\ufb01nes a formal language. Sentences (strings of words)\nthat can be derived by a grammar are in the formal language de\ufb01ned by that gram-\nmar, and are called grammatical sentences. Sentences that cannot be derived by grammatical\na given formal grammar are not in the language de\ufb01ned by that grammar and are\nreferred to as ungrammatical . This hard line between \u201cin\u201d and \u201cout\u201d characterizes ungrammatical\nall formal languages but is only a very simpli\ufb01ed model of how natural languages\nreally work. This is because determining whether a given sentence is part of a given\nnatural language (say, English) often depends on the context. In linguistics, the use\nof formal languages to model natural languages is called generative grammar sincegenerative\ngrammar\nthe language is de\ufb01ned by the set of possible sentences \u201cgenerated\u201d by the grammar.\n(Note that this is a different sense of the word \u2018generate\u2019 than when we talk about 6CHAPTER 18 \u2022 C ONTEXT -FREE GRAMMARS AND CONSTITUENCY PARSING\nlanguage models generating text.)\n18.2.1 Formal De\ufb01nition of Context-Free Grammar\nWe conclude this section with a quick, formal description of a context-free gram-\nmar and the language it generates. A context-free grammar Gis de\ufb01ned by four\nparameters: N;S;R;S(technically it is a \u201c4-tuple\u201d).\nNa set of non-terminal symbols (orvariables )\nSa set of terminal symbols (disjoint from N)\nRa set of rules or productions, each of the form A!b,\nwhere Ais a non-terminal,\nbis a string of symbols from the in\ufb01nite set of strings (S[N)\u0003\nSa designated start symbol and a member of N\nFor the remainder of the book we adhere to the following conventions when dis-\ncussing the formal properties of context-free grammars (as opposed to explaining\nparticular facts about English or other languages).\nCapital letters like A,B, and S Non-terminals\nS The start symbol\nLower-case Greek letters like a,b, and g Strings drawn from (S[N)\u0003\nLower-case Roman letters like u,v, and w Strings of terminals\nA language is de\ufb01ned through the concept of derivation. One string derives an-\nother one if it can be rewritten as the second one by some series of rule applications.\nMore formally, following Hopcroft and Ullman (1979),\nifA!bis a production of Randaandgare any strings in the set\n(S[N)\u0003, then we say that aAgdirectly derives abg , oraAg)abg . directly derives\nDerivation is then a generalization of direct derivation:\nLeta1;a2;:::; ambe strings in (S[N)\u0003;m\u00151, such that\na1)a2;a2)a3;:::; am\u00001)am\nWe say that a1derives am, ora1\u0003)am. derives\nWe can then formally de\ufb01ne the language LGgenerated by a grammar Gas the\nset of strings composed of terminal symbols that can be derived from the designated\nstart symbol S.\nLG=fwjwis inS\u0003andS\u0003)wg\nThe problem of mapping from a string of words to its parse tree is called syn-\ntactic parsing , as we\u2019ll see in Section 18.6.syntactic\nparsing\n18.3 Treebanks\nA corpus in which every sentence is annotated with a parse tree is called a treebank . treebank 18.3 \u2022 T REEBANKS 7\nTreebanks play an important role in parsing as well as in linguistic investigations of\nsyntactic phenomena.\nTreebanks are generally made by running a parser over each sentence and then\nhaving the resulting parse hand-corrected by human linguists. Figure 18.5 shows\nsentences from the Penn Treebank project, which includes various treebanks in Penn Treebank\nEnglish, Arabic, and Chinese. The Penn Treebank part-of-speech tagset was de\ufb01ned\nin Chapter 17, but we\u2019ll see minor formatting differences across treebanks. The use\nof LISP-style parenthesized notation for trees is extremely common and resembles\nthe bracketed notation we saw earlier in (18.1). For those who are not familiar with\nit we show a standard node-and-line tree representation in Fig. 18.6.\n((S\n(NP-SBJ (DT That)\n(JJ cold) (, ,)\n(JJ empty) (NN sky) )\n(VP (VBD was)\n(ADJP-PRD (JJ full)\n(PP (IN of)\n(NP (NN fire)\n(CC and)\n(NN light) ))))\n(. .) ))((S\n(NP-SBJ The/DT flight/NN )\n(VP should/MD\n(VP arrive/VB\n(PP-TMP at/IN\n(NP eleven/CD a.m/RB ))\n(NP-TMP tomorrow/NN )))))\n(a) (b)\nFigure 18.5 Parses from the LDC Treebank3 for (a) Brown and (b) ATIS sentences.\nS\n.\n.VP\nADJP-PRD\nPP\nNP\nNN\nlightCC\nandNN\n\ufb01reIN\nofJJ\nfullVBD\nwasNP-SBJ\nNN\nskyJJ\nempty,\n,JJ\ncoldDT\nThat\nFigure 18.6 The tree corresponding to the Brown corpus sentence in the previous \ufb01gure.\nThe sentences in a treebank implicitly constitute a grammar of the language. For\nexample, from the parsed sentences in Fig. 18.5 we can extract the CFG rules shown\nin Fig. 18.7 (with rule suf\ufb01xes ( -SBJ ) stripped for simplicity). The grammar used\nto parse the Penn Treebank is very \ufb02at, resulting in very many rules. For example, 8CHAPTER 18 \u2022 C ONTEXT -FREE GRAMMARS AND CONSTITUENCY PARSING\nGrammar Lexicon\nS!NP VP . DT!thejthat\nS!NP VP JJ!coldjemptyjfull\nNP!DT NN NN!skyj\ufb01rejlightj\ufb02ightjtomorrow\nNP!NN CC NN CC!and\nNP!DT JJ , JJ NN IN!ofjat\nNP!NN CD!eleven\nVP!MD VP RB!a.m.\nVP!VBD ADJP VB!arrive\nVP!MD VP VBD!wasjsaid\nVP!VB PP NP MD!shouldjwould\nADJP!JJ PP\nPP!IN NP\nPP!IN NP RB\nFigure 18.7 CFG grammar rules and lexicon from the treebank sentences in Fig. 18.5.\namong the approximately 4,500 different rules for expanding VPs are separate rules\nfor PP sequences of any length and every possible arrangement of verb arguments:\nVP!VBD PP\nVP!VBD PP PP\nVP!VBD PP PP PP\nVP!VBD PP PP PP PP\nVP!VB ADVP PP\nVP!VB PP ADVP\nVP!ADVP VB PP\n18.4 Grammar Equivalence and Normal Form\nA formal language is de\ufb01ned as a (possibly in\ufb01nite) set of strings of words. This sug-\ngests that we could ask if two grammars are equivalent by asking if they generate the\nsame set of strings. In fact, it is possible to have two distinct context-free grammars\ngenerate the same language. We say that two grammars are strongly equivalent ifstrongly\nequivalent\nthey generate the same set of strings andif they assign the same phrase structure\nto each sentence (allowing merely for renaming of the non-terminal symbols). Two\ngrammars are weakly equivalent if they generate the same set of strings but do notweakly\nequivalent\nassign the same phrase structure to each sentence.\nIt is sometimes useful to have a normal form for grammars, in which each of normal form\nthe productions takes a particular form. For example, a context-free grammar is in\nChomsky normal form (CNF) (Chomsky, 1963) if it is \u000f-free and if in additionChomsky\nnormal form\neach production is either of the form A!B C orA!a. That is, the right-hand side\nof each rule either has two non-terminal symbols or one terminal symbol. Chomsky\nnormal form grammars are binary branching , that is they have binary trees (downbinary\nbranching\nto the prelexical nodes). We make use of this binary branching property in the CKY\nparsing algorithm in Section 18.6.\nAny context-free grammar can be converted into a weakly equivalent Chomsky\nnormal form grammar. For example, a rule of the form\nA!B C D\ncan be converted into the following two CNF rules (Exercise 18.1 asks the reader to 18.5 \u2022 A MBIGUITY 9\nGrammar Lexicon\nS!NP VP Det!thatjthisjtheja\nS!Aux NP VP Noun!bookj\ufb02ightjmealjmoney\nS!VP Verb!bookjincludejprefer\nNP!Pronoun Pronoun!Ijshejme\nNP!Proper-Noun Proper-Noun!HoustonjNWA\nNP!Det Nominal Aux!does\nNominal!Noun Preposition!fromjtojonjnearjthrough\nNominal!Nominal Noun\nNominal!Nominal PP\nVP!Verb\nVP!Verb NP\nVP!Verb NP PP\nVP!Verb PP\nVP!VP PP\nPP!Preposition NP\nFigure 18.8 TheL1miniature English grammar and lexicon.\nformulate the complete algorithm):\nA!B X\nX!C D\nSometimes using binary branching can actually produce smaller grammars. For\nexample, the sentences that might be characterized as\nVP -> VBD NP PP*\nare represented in the Penn Treebank by this series of rules:\nVP!VBD NP PP\nVP!VBD NP PP PP\nVP!VBD NP PP PP PP\nVP!VBD NP PP PP PP PP\n...\nbut could also be generated by the following two-rule grammar:\nVP!VBD NP PP\nVP!VP PP\nThe generation of a symbol A with a potentially in\ufb01nite sequence of symbols B with\na rule of the form A!A Bis known as Chomsky-adjunction .Chomsky-\nadjunction\n18.5 Ambiguity\nAmbiguity is the most serious problem faced by syntactic parsers. Chapter 17 intro-\nduced the notions of part-of-speech ambiguity andpart-of-speech disambigua-\ntion. Here, we introduce a new kind of ambiguity, called structural ambiguity ,structural\nambiguity\nillustrated with a new toy grammar L1, shown in Figure 18.8, which adds a few\nrules to the L0grammar.\nStructural ambiguity occurs when the grammar can assign more than one parse\nto a sentence. Groucho Marx\u2019s well-known line as Captain Spaulding in Animal 10 CHAPTER 18 \u2022 C ONTEXT -FREE GRAMMARS AND CONSTITUENCY PARSING\nS\nVP\nNP\nNominal\nPP\nin my pajamasNominal\nNoun\nelephantDet\nanVerb\nshotNP\nPronoun\nIS\nVP\nPP\nin my pajamasVP\nNP\nNominal\nNoun\nelephantDet\nanVerb\nshotNP\nPronoun\nI\nFigure 18.9 Two parse trees for an ambiguous sentence. The parse on the left corresponds to the humorous\nreading in which the elephant is in the pajamas, the parse on the right corresponds to the reading in which\nCaptain Spaulding did the shooting in his pajamas.\nCrackers is ambiguous because the phrase in my pajamas can be part of the NP\nheaded by elephant or a part of the verb phrase headed by shot. Figure 18.9 illus-\ntrates these two analyses of Marx\u2019s line using rules from L1.\nStructural ambiguity, appropriately enough, comes in many forms. Two common\nkinds of ambiguity are attachment ambiguity andcoordination ambiguity . A\nsentence has an attachment ambiguity if a particular constituent can be attached toattachment\nambiguity\nthe parse tree at more than one place. The Groucho Marx sentence is an example\nofPP-attachment ambiguity : the preposition phrase can be attached either as partPP-attachment\nambiguity\nof the NP or as part of the VP. Various kinds of adverbial phrases are also subject\nto this kind of ambiguity. For instance, in the following example the gerundive-VP\n\ufb02ying to Paris can be part of a gerundive sentence whose subject is the Eiffel Tower\nor it can be an adjunct modifying the VP headed by saw:\n(18.2) We saw the Eiffel Tower \ufb02ying to Paris.\nIncoordination ambiguity phrases can be conjoined by a conjunction like and.coordination\nambiguity\nFor example, the phrase old men and women can be bracketed as [old [men and\nwomen]] , referring to old men andold women , or as [old men] and [women] , in\nwhich case it is only the men who are old. These ambiguities combine in complex\nways in real sentences, like the following news sentence from the Brown corpus:\n(18.3) President Kennedy today pushed aside other White House business to\ndevote all his time and attention to working on the Berlin crisis address he\nwill deliver tomorrow night to the American people over nationwide\ntelevision and radio.\nThis sentence has a number of ambiguities, although since they are semantically\nunreasonable, it requires a careful reading to see them. The last noun phrase could be\nparsed [nationwide [television and radio]] or[[nationwide television] and radio] .\nThe direct object of pushed aside should be other White House business but could\nalso be the bizarre phrase [other White House business to devote all his time and\nattention to working] (i.e., a structure like Kennedy af\ufb01rmed [his intention to propose\na new budget to address the de\ufb01cit] ). Then the phrase on the Berlin crisis address he 18.6 \u2022 CKY P ARSING : A D YNAMIC PROGRAMMING APPROACH 11\nwill deliver tomorrow night to the American people could be an adjunct modifying\nthe verb pushed . APPlikeover nationwide television and radio could be attached\nto any of the higher VPs orNPs (e.g., it could modify people ornight ).\nThe fact that there are many grammatically correct but semantically unreason-\nable parses for naturally occurring sentences is an irksome problem that affects all\nparsers. Fortunately, the CKY algorithm below is designed to ef\ufb01ciently handle\nstructural ambiguities. And as we\u2019ll see in the following section, we can augment\nCKY with neural methods to choose a single correct parse by syntactic disambigua-\ntion.syntactic\ndisambiguation\n18.6 CKY Parsing: A Dynamic Programming Approach\nDynamic programming provides a powerful framework for addressing the prob-\nlems caused by ambiguity in grammars. Recall that a dynamic programming ap-\nproach systematically \ufb01lls in a table of solutions to subproblems. The complete\ntable has the solution to all the subproblems needed to solve the problem as a whole.\nIn the case of syntactic parsing, these subproblems represent parse trees for all the\nconstituents detected in the input.\nThe dynamic programming advantage arises from the context-free nature of our\ngrammar rules\u2014once a constituent has been discovered in a segment of the input we\ncan record its presence and make it available for use in any subsequent derivation\nthat might require it. This provides both time and storage ef\ufb01ciencies since subtrees\ncan be looked up in a table, not reanalyzed. This section presents the Cocke-Kasami-\nYounger (CKY) algorithm, the most widely used dynamic-programming based ap-\nproach to parsing. Chart parsing (Kaplan 1973, Kay 1982) is a related approach,\nand dynamic programming methods are often referred to as chart parsing methods. chart parsing\n18.6.1 Conversion to Chomsky Normal Form\nThe CKY algorithm requires grammars to \ufb01rst be in Chomsky Normal Form (CNF).\nRecall from Section 18.4 that grammars in CNF are restricted to rules of the form\nA!B CorA!w. That is, the right-hand side of each rule must expand either to\ntwo non-terminals or to a single terminal. Restricting a grammar to CNF does not\nlead to any loss in expressiveness, since any context-free grammar can be converted\ninto a corresponding CNF grammar that accepts exactly the same set of strings as\nthe original grammar.\nLet\u2019s start with the process of converting a generic CFG into one represented in\nCNF. Assuming we\u2019re dealing with an \u000f-free grammar, there are three situations we\nneed to address in any generic grammar: rules that mix terminals with non-terminals\non the right-hand side, rules that have a single non-terminal on the right-hand side,\nand rules in which the length of the right-hand side is greater than 2.\nThe remedy for rules that mix terminals and non-terminals is to simply introduce\na new dummy non-terminal that covers only the original terminal. For example, a\nrule for an in\ufb01nitive verb phrase such as INF-VP!to VP would be replaced by the\ntwo rules INF-VP!TO VP andTO!to.\nRules with a single non-terminal on the right are called unit productions . WeUnit\nproductions\ncan eliminate unit productions by rewriting the right-hand side of the original rules\nwith the right-hand side of all the non-unit production rules that they ultimately lead\nto. More formally, if A\u0003)Bby a chain of one or more unit productions and B!g 12 CHAPTER 18 \u2022 C ONTEXT -FREE GRAMMARS AND CONSTITUENCY PARSING\nis a non-unit production in our grammar, then we add A!gfor each such rule in\nthe grammar and discard all the intervening unit productions. As we demonstrate\nwith our toy grammar, this can lead to a substantial \ufb02attening of the grammar and a\nconsequent promotion of terminals to fairly high levels in the resulting trees.\nRules with right-hand sides longer than 2 are normalized through the introduc-\ntion of new non-terminals that spread the longer sequences over several new rules.\nFormally, if we have a rule like\nA!B Cg\nwe replace the leftmost pair of non-terminals with a new non-terminal and introduce\na new production, resulting in the following new rules:\nA!X1g\nX1!B C\nIn the case of longer right-hand sides, we simply iterate this process until the of-\nfending rule has been replaced by rules of length 2. The choice of replacing the\nleftmost pair of non-terminals is purely arbitrary; any systematic scheme that results\nin binary rules would suf\ufb01ce.\nIn our current grammar, the rule S!Aux NP VP would be replaced by the two\nrules S!X1 VP andX1!Aux NP .\nThe entire conversion process can be summarized as follows:\n1. Copy all conforming rules to the new grammar unchanged.\n2. Convert terminals within rules to dummy non-terminals.\n3. Convert unit productions.\n4. Make all rules binary and add them to new grammar.\nFigure 18.10 shows the results of applying this entire conversion procedure to\ntheL1grammar introduced earlier on page 9. Note that this \ufb01gure doesn\u2019t show\nthe original lexical rules; since these original lexical rules are already in CNF, they\nall carry over unchanged to the new grammar. Figure 18.10 does, however, show\nthe various places where the process of eliminating unit productions has, in effect,\ncreated new lexical rules. For example, all the original verbs have been promoted to\nboth VPs and to Ss in the converted grammar.\n18.6.2 CKY Recognition\nWith our grammar now in CNF, each non-terminal node above the part-of-speech\nlevel in a parse tree will have exactly two daughters. A two-dimensional matrix can\nbe used to encode the structure of an entire tree. For a sentence of length n, we will\nwork with the upper-triangular portion of an (n+1)\u0002(n+1)matrix. Each cell [i;j]\nin this matrix contains the set of non-terminals that represent all the constituents that\nspan positions ithrough jof the input. Since our indexing scheme begins with 0, it\u2019s\nnatural to think of the indexes as pointing at the gaps between the input words (as in\n0Book 1that 2\ufb02ight 3). These gaps are often called fenceposts , on the metaphor of fenceposts\nthe posts between segments of fencing. It follows then that the cell that represents\nthe entire input resides in position [0;n]in the matrix.\nSince each non-terminal entry in our table has two daughters in the parse, it fol-\nlows that for each constituent represented by an entry [i;j], there must be a position\nin the input, k, where it can be split into two parts such that i<k<j. Given such 18.6 \u2022 CKY P ARSING : A D YNAMIC PROGRAMMING APPROACH 13\nL1Grammar L1in CNF\nS!NP VP S!NP VP\nS!Aux NP VP S!X1 VP\nX1!Aux NP\nS!VP S!bookjincludejprefer\nS!Verb NP\nS!X2 PP\nS!Verb PP\nS!VP PP\nNP!Pronoun NP!Ijshejme\nNP!Proper-Noun NP!TWAjHouston\nNP!Det Nominal NP!Det Nominal\nNominal!Noun Nominal!bookj\ufb02ightjmealjmoney\nNominal!Nominal Noun Nominal!Nominal Noun\nNominal!Nominal PP Nominal!Nominal PP\nVP!Verb VP!bookjincludejprefer\nVP!Verb NP VP!Verb NP\nVP!Verb NP PP VP!X2 PP\nX2!Verb NP\nVP!Verb PP VP!Verb PP\nVP!VP PP VP!VP PP\nPP!Preposition NP PP!Preposition NP\nFigure 18.10 L1Grammar and its conversion to CNF. Note that although they aren\u2019t shown\nhere, all the original lexical entries from L1carry over unchanged as well.\na position k, the \ufb01rst constituent [i;k]must lie to the left of entry [i;j]somewhere\nalong row i, and the second entry [k;j]must lie beneath it, along column j.\nTo make this more concrete, consider the following example with its completed\nparse matrix, shown in Fig. 18.11.\n(18.4) Book the \ufb02ight through Houston.\nThe superdiagonal row in the matrix contains the parts of speech for each word in\nthe input. The subsequent diagonals above that superdiagonal contain constituents\nthat cover all the spans of increasing length in the input.\nGiven this setup, CKY recognition consists of \ufb01lling the parse table in the right\nway. To do this, we\u2019ll proceed in a bottom-up fashion so that at the point where we\nare \ufb01lling any cell [i;j], the cells containing the parts that could contribute to this\nentry (i.e., the cells to the left and the cells below) have already been \ufb01lled. The\nalgorithm given in Fig. 18.12 \ufb01lls the upper-triangular matrix a column at a time\nworking from left to right, with each column \ufb01lled from bottom to top, as the right\nside of Fig. 18.11 illustrates. This scheme guarantees that at each point in time we\nhave all the information we need (to the left, since all the columns to the left have\nalready been \ufb01lled, and below since we\u2019re \ufb01lling bottom to top). It also mirrors on-\nline processing, since \ufb01lling the columns from left to right corresponds to processing\neach word one at a time.\nThe outermost loop of the algorithm given in Fig. 18.12 iterates over the columns,\nand the second loop iterates over the rows, from the bottom up. The purpose of the\ninnermost loop is to range over all the places where a substring spanning itojin\nthe input might be split in two. As kranges over the places where the string can be\nsplit, the pairs of cells we consider move, in lockstep, to the right along row iand\ndown along column j. Figure 18.13 illustrates the general case of \ufb01lling cell [i;j]. 14 CHAPTER 18 \u2022 C ONTEXT -FREE GRAMMARS AND CONSTITUENCY PARSING\nBookthe flight throughHoustonS, VP, Verb, Nominal, NounS,VP,X2S,VP,X2DetNPNPNominal,NounNominalPrepPPNP,Proper-Noun[0,1][0,2][0,3][0,4][0,5][1,2][1,3][2,3][1,4][2,5][2,4][3,4][4,5][3,5][1,5]\nFigure 18.11 Completed parse table for Book the \ufb02ight through Houston.\nfunction CKY-P ARSE (words, grammar )returns table\nforj from 1toLENGTH (words )do\nfor all fAjA!words [j]2grammar g\ntable [j\u00001;j] table [j\u00001;j][A\nfori from j\u00002down to 0do\nfork i+1toj\u00001do\nfor all fAjA!BC2grammar andB2table[i;k]andC2table[k;j]g\ntable [i,j] table [i,j][A\nFigure 18.12 The CKY algorithm.\nAt each such split, the algorithm considers whether the contents of the two cells can\nbe combined in a way that is sanctioned by a rule in the grammar. If such a rule\nexists, the non-terminal on its left-hand side is entered into the table.\nFigure 18.14 shows how the \ufb01ve cells of column 5 of the table are \ufb01lled after the\nword Houston is read. The arrows point out the two spans that are being used to add\nan entry to the table. Note that the action in cell [0;5]indicates the presence of three\nalternative parses for this input, one where the PPmodi\ufb01es the \ufb02ight , one where\nit modi\ufb01es the booking, and one that captures the second argument in the original\nVP!Verb NP PP rule, now captured indirectly with the VP!X2 PP rule.\n18.6.3 CKY Parsing\nThe algorithm given in Fig. 18.12 is a recognizer, not a parser. That is, it can tell\nus whether a valid parse exists for a given sentence based on whether or not if \ufb01nds\nanSin cell [0;n], but it can\u2019t provide the derivation, which is the actual job for a\nparser. To turn it into a parser capable of returning all possible parses for a given\ninput, we can make two simple changes to the algorithm: the \ufb01rst change is to\naugment the entries in the table so that each non-terminal is paired with pointers to\nthe table entries from which it was derived (more or less as shown in Fig. 18.14), the\nsecond change is to permit multiple versions of the same non-terminal to be entered\ninto the table (again as shown in Fig. 18.14). With these changes, the completed\ntable contains all the possible parses for a given input. Returning an arbitrary single 18.6 \u2022 CKY P ARSING : A D YNAMIC PROGRAMMING APPROACH 15\n...\n...[0,n]\n[i,i+1][i,i+2][i,j-2][i,j-1][i+1,j][i+2,j]\n[j-1,j][j-2,j][i,j]...[0,1]\n[n-1, n]\nFigure 18.13 All the ways to \ufb01ll the [ i,j]th cell in the CKY table.\nparse consists of choosing an Sfrom cell [0;n]and then recursively retrieving its\ncomponent constituents from the table. Of course, instead of returning every parse\nfor a sentence, we usually want just the best parse; we\u2019ll see how to do that in the\nnext section.\n18.6.4 CKY in Practice\nFinally, we should note that while the restriction to CNF does not pose a problem\ntheoretically, it does pose some non-trivial problems in practice. The returned CNF\ntrees may not be consistent with the original grammar built by the grammar devel-\nopers, and will complicate any syntax-driven approach to semantic analysis.\nOne approach to getting around these problems is to keep enough information\naround to transform our trees back to the original grammar as a post-processing step\nof the parse. This is trivial in the case of the transformation used for rules with length\ngreater than 2. Simply deleting the new dummy non-terminals and promoting their\ndaughters restores the original tree.\nIn the case of unit productions, it turns out to be more convenient to alter the ba-\nsic CKY algorithm to handle them directly than it is to store the information needed\nto recover the correct trees. Exercise 18.3 asks you to make this change. Many of\nthe probabilistic parsers presented in Appendix C use the CKY algorithm altered in 16 CHAPTER 18 \u2022 C ONTEXT -FREE GRAMMARS AND CONSTITUENCY PARSING\nBookthe flight throughHoustonS, VP, Verb, Nominal, NounS,VP,X2DetNPNominal,NounNominalPrepNP,Proper-Noun[0,1][0,2][0,3][0,4][0,5][1,2][1,3][2,3][1,4][2,5][2,4][3,4][4,5][3,5][1,5]Bookthe flight throughHoustonS, VP, Verb, Nominal, NounS,VP,X2DetNPNPNominal,NounPrepPPNP,Proper-Noun[0,1][0,2][0,3][0,4][0,5][1,2][1,3][2,3][1,4][2,5][2,4][3,4][4,5][3,5][1,5]\nBookthe flight throughHoustonS, VP, Verb, Nominal, NounS,VP,X2DetNPNPNominal,NounNominalPrepPPNP,Proper-Noun[0,1][0,2][0,3][0,4][0,5][1,2][1,3][2,3][1,4][2,5][2,4][3,4][4,5][3,5][1,5]Bookthe flight throughHoustonS, VP, Verb, Nominal, NounS,VP,X2DetNPNPNominal,NounNominalPrepPPNP,Proper-Noun[0,1][0,2][0,3][0,4][0,5][1,2][1,3][2,3][1,4][2,5][2,4][3,4][4,5][3,5][1,5]\nBookthe flight throughHoustonS, VP, Verb, Nominal, NounS,VP,X2DetNPNPNominal,NounNominalPrepPPNP,Proper-Noun[0,1][0,2][0,3][0,4][1,2][1,3][2,3][1,4][2,5][2,4][3,4][4,5][3,5][1,5]S2, VPS3S1,VP, X2\nFigure 18.14 Filling the cells of column 5 after reading the word Houston . 18.7 \u2022 S PAN-BASED NEURAL CONSTITUENCY PARSING 17\njust this manner.\n18.7 Span-Based Neural Constituency Parsing\nWhile the CKY parsing algorithm we\u2019ve seen so far does great at enumerating all\nthe possible parse trees for a sentence, it has a large problem: it doesn\u2019t tell us which\nparse is the correct one! That is, it doesn\u2019t disambiguate among the possible parses.\nTo solve the disambiguation problem we\u2019ll use a simple neural extension of the\nCKY algorithm. The intuition of such parsing algorithms (often called span-based\nconstituency parsing , orneural CKY ), is to train a neural classi\ufb01er to assign a\nscore to each constituent, and then use a modi\ufb01ed version of CKY to combine these\nconstituent scores to \ufb01nd the best-scoring parse tree.\nHere we\u2019ll describe a version of the algorithm from Kitaev et al. (2019). This\nparser learns to map a span of words to a constituent, and, like CKY , hierarchically\ncombines larger and larger spans to build the parse-tree bottom-up. But unlike clas-\nsic CKY , this parser doesn\u2019t use the hand-written grammar to constrain what con-\nstituents can be combined, instead just relying on the learned neural representations\nof spans to encode likely combinations.\n18.7.1 Computing Scores for a Span\nLet\u2019s begin by considering just the constituent (we\u2019ll call it a span ) that lies between span\nfencepost positions iand jwith non-terminal symbol label l. We\u2019ll build a system\nto assign a score s(i;j;l)to this constituent span.\nENCODER[START]BooktheflightthroughHouston[END]map to subwordsmap back to words013245MLPi=1hj-hij=3NPCompute score for spanRepresent spanCKY for computing best parse\npostprocessing layers\nFigure 18.15 A simpli\ufb01ed outline of computing the span score for the span the \ufb02ight with\nthe label NP.\nFig. 18.15 sketches the architecture. The input word tokens are embedded by 18 CHAPTER 18 \u2022 C ONTEXT -FREE GRAMMARS AND CONSTITUENCY PARSING\npassing them through a pretrained language model like BERT. Because BERT oper-\nates on the level of subword (wordpiece) tokens rather than words, we\u2019ll \ufb01rst need to\nconvert the BERT outputs to word representations. One standard way of doing this\nis to simply use the \ufb01rst subword unit as the representation for the entire word; us-\ning the last subword unit, or the sum of all the subword units are also common. The\nembeddings can then be passed through some postprocessing layers; Kitaev et al.\n(2019), for example, use 8 Transformer layers.\nThe resulting word encoder outputs ytare then used to compute a span score.\nFirst, we must map the word encodings (indexed by word positions) to span encod-\nings (indexed by fenceposts). We do this by representing each fencepost with two\nseparate values; the intuition is that a span endpoint to the right of a word represents\ndifferent information than a span endpoint to the left of a word. We convert each\nword output ytinto a (leftward-pointing) value for spans ending at this fencepost, \u0000yt, and a (rightward-pointing) value\u0000 !ytfor spans beginning at this fencepost, by\nsplitting ytinto two halves. Each span then stretches from one double-vector fence-\npost to another, as in the following representation of the \ufb02ight , which is span (1;3):\nSTART 0 Book the \ufb02ight through\ny0\u0000 !y0 \u0000y1y1\u0000 !y1 \u0000y2 y2\u0000 !y2 \u0000y3y3\u0000 !y3 \u0000y4 y4\u0000 !y4 \u0000y5:::\n0\r 1\r 2\r 3\r 4\r\nspan(1,3)\nA traditional way to represent a span, developed originally for RNN-based models\n(Wang and Chang, 2016), but extended also to Transformers, is to take the differ-\nence between the embeddings of its start and end, i.e., representing span (i;j)by\nsubtracting the embedding of ifrom the embedding of j. Here we represent a span\nby concatenating the difference of each of its fencepost components:\nv(i;j) = [\u0000 !yj\u0000\u0000 !yi; \u0000\u0000yj+1\u0000 \u0000\u0000yi+1] (18.5)\nThe span vector vis then passed through an MLP span classi\ufb01er, with two fully-\nconnected layers and one ReLU activation function, whose output dimensionality is\nthe number of possible non-terminal labels:\ns(i;j;\u0001) =W2ReLU (LayerNorm (W1v(i;j))) (18.6)\nThe MLP then outputs a score for each possible non-terminal.\n18.7.2 Integrating Span Scores into a Parse\nNow we have a score for each labeled constituent span s(i;j;l). But we need a score\nfor an entire parse tree. Formally a tree Tis represented as a set of jTjsuch labeled\nspans, with the tthspan starting at position itand ending at position jt, with label lt:\nT=f(it;jt;lt):t=1;:::;jTjg (18.7)\nThus once we have a score for each span, the parser can compute a score for the\nwhole tree s(T)simply by summing over the scores of its constituent spans:\ns(T) =X\n(i;j;l)2Ts(i;j;l) (18.8) 18.8 \u2022 E VALUATING PARSERS 19\nAnd we can choose the \ufb01nal parse tree as the tree with the maximum score:\n\u02c6T=argmax\nTs(T) (18.9)\nThe simplest method to produce the most likely parse is to greedily choose the\nhighest scoring label for each span. This greedy method is not guaranteed to produce\na tree, since the best label for a span might not \ufb01t into a complete tree. In practice,\nhowever, the greedy method tends to \ufb01nd trees; in their experiments Gaddy et al.\n(2018) \ufb01nds that 95% of predicted bracketings form valid trees.\nNonetheless it is more common to use a variant of the CKY algorithm to \ufb01nd the\nfull parse. The variant de\ufb01ned in Gaddy et al. (2018) works as follows. Let\u2019s de\ufb01ne\nsbest(i;j)as the score of the best subtree spanning (i;j). For spans of length one, we\nchoose the best label:\nsbest(i;i+1) =max\nls(i;i+1;l) (18.10)\nFor other spans (i;j), the recursion is:\nsbest(i;j) = max\nls(i;j;l)\n+max\nk[sbest(i;k)+sbest(k;j)] (18.11)\nNote that the parser is using the max label for span (i;j)+ the max labels for spans\n(i;k)and(k;j)without worrying about whether those decisions make sense given a\ngrammar. The role of the grammar in classical parsing is to help constrain possible\ncombinations of constituents (NPs like to be followed by VPs). By contrast, the\nneural model seems to learn these kinds of contextual constraints during its mapping\nfrom spans to non-terminals.\nFor more details on span-based parsing, including the margin-based training al-\ngorithm, see Stern et al. (2017), Gaddy et al. (2018), Kitaev and Klein (2018), and\nKitaev et al. (2019).\n18.8 Evaluating Parsers\nThe standard tool for evaluating parsers that assign a single parse tree to a sentence\nis the PARSEV AL metrics (Black et al., 1991). The PARSEV AL metric measures PARSEV AL\nhow much the constituents in the hypothesis parse tree look like the constituents in a\nhand-labeled, reference parse. PARSEV AL thus requires a human-labeled reference\n(or \u201cgold standard\u201d) parse tree for each sentence in the test set; we generally draw\nthese reference parses from a treebank like the Penn Treebank.\nA constituent in a hypothesis parse Chof a sentence sis labeled correct if there\nis a constituent in the reference parse Crwith the same starting point, ending point,\nand non-terminal symbol. We can then measure the precision and recall just as for\ntasks we\u2019ve seen already like named entity tagging:\nlabeled recall: =# of correct constituents in hypothesis parse of s\n# of total constituents in reference parse of s\nlabeled precision: =# of correct constituents in hypothesis parse of s\n# of total constituents in hypothesis parse of s 20 CHAPTER 18 \u2022 C ONTEXT -FREE GRAMMARS AND CONSTITUENCY PARSING\nS(dumped)\nVP(dumped)\nPP(into)\nNP(bin)\nNN(bin)\nbinDT(a)\naP\nintoNP(sacks)\nNNS(sacks)\nsacksVBD(dumped)\ndumpedNP(workers)\nNNS(workers)\nworkers\nFigure 18.16 A lexicalized tree from Collins (1999).\nAs usual, we often report a combination of the two, F 1:\nF1=2PR\nP+R(18.12)\nWe additionally use a new metric, crossing brackets, for each sentence s:\ncross-brackets: the number of constituents for which the reference parse has a\nbracketing such as ((A B) C) but the hypothesis parse has a bracketing such\nas (A (B C)).\nFor comparing parsers that use different grammars, the PARSEV AL metric in-\ncludes a canonicalization algorithm for removing information likely to be grammar-\nspeci\ufb01c (auxiliaries, pre-in\ufb01nitival \u201cto\u201d, etc.) and for computing a simpli\ufb01ed score\n(Black et al., 1991). The canonical implementation of the PARSEV AL metrics is\ncalled evalb (Sekine and Collins, 1997). evalb\n18.9 Heads and Head-Finding\nSyntactic constituents can be associated with a lexical head ;Nis the head of an NP,\nVis the head of a VP. This idea of a head for each constituent dates back to Bloom-\n\ufb01eld 1914, and is central to the dependency grammars and dependency parsing we\u2019ll\nintroduce in Chapter 19. Indeed, heads can be used as a way to map between con-\nstituency and dependency parses. Heads are also important in probabilistic pars-\ning (Appendix C) and in constituent-based grammar formalisms like Head-Driven\nPhrase Structure Grammar (Pollard and Sag, 1994)..\nIn one simple model of lexical heads, each context-free rule is associated with\na head (Charniak 1997, Collins 1999). The head is the word in the phrase that is\ngrammatically the most important. Heads are passed up the parse tree; thus, each\nnon-terminal in a parse tree is annotated with a single word, which is its lexical head.\nFigure 18.16 shows an example of such a tree from Collins (1999), in which each\nnon-terminal is annotated with its head.\nFor the generation of such a tree, each CFG rule must be augmented to identify\none right-side constituent to be the head child. The headword for a node is then set to\nthe headword of its head child. Choosing these head children is simple for textbook\nexamples ( NNis the head of NP) but is complicated and indeed controversial for 18.10 \u2022 S UMMARY 21\nmost phrases. (Should the complementizer toor the verb be the head of an in\ufb01nite\nverb phrase?) Modern linguistic theories of syntax generally include a component\nthat de\ufb01nes heads (see, e.g., (Pollard and Sag, 1994)).\nAn alternative approach to \ufb01nding a head is used in most practical computational\nsystems. Instead of specifying head rules in the grammar itself, heads are identi\ufb01ed\ndynamically in the context of trees for speci\ufb01c sentences. In other words, once\na sentence is parsed, the resulting tree is walked to decorate each node with the\nappropriate head. Most current systems rely on a simple set of handwritten rules,\nsuch as a practical one for Penn Treebank grammars given in Collins (1999) but\ndeveloped originally by Magerman (1995). For example, the rule for \ufb01nding the\nhead of an NPis as follows (Collins, 1999, p. 238):\n\u2022 If the last word is tagged POS, return last-word.\n\u2022 Else search from right to left for the \ufb01rst child which is an NN, NNP, NNPS, NX, POS,\nor JJR.\n\u2022 Else search from left to right for the \ufb01rst child which is an NP.\n\u2022 Else search from right to left for the \ufb01rst child which is a $, ADJP, or PRN.\n\u2022 Else search from right to left for the \ufb01rst child which is a CD.\n\u2022 Else search from right to left for the \ufb01rst child which is a JJ, JJS, RB or QP.\n\u2022 Else return the last word\nSelected other rules from this set are shown in Fig. 18.17. For example, for VP\nrules of the form VP!Y1\u0001\u0001\u0001Yn, the algorithm would start from the left of Y1\u0001\u0001\u0001\nYnlooking for the \ufb01rst Yiof type TO; if no TOs are found, it would search for the\n\ufb01rstYiof type VBD; if no VBDs are found, it would search for a VBN, and so on.\nSee Collins (1999) for more details.\nParent Direction Priority List\nADJP Left NNS QP NN $ ADVP JJ VBN VBG ADJP JJR NP JJS DT FW RBR RBS\nSBAR RB\nADVP Right RB RBR RBS FW ADVP TO CD JJR JJ IN NP JJS NN\nPRN Left\nPRT Right RP\nQP Left $ IN NNS NN JJ RB DT CD NCD QP JJR JJS\nS Left TO IN VP S SBAR ADJP UCP NP\nSBAR Left WHNP WHPP WHADVP WHADJP IN DT S SQ SINV SBAR FRAG\nVP Left TO VBD VBN MD VBZ VB VBG VBP VP ADJP NN NNS NP\nFigure 18.17 Some head rules from Collins (1999). The head rules are also called a head percolation table .\n18.10 Summary\nThis chapter introduced constituency parsing. Here\u2019s a summary of the main points:\n\u2022 In many languages, groups of consecutive words act as a group or a con-\nstituent , which can be modeled by context-free grammars (which are also\nknown as phrase-structure grammars ).\n\u2022 A context-free grammar consists of a set of rules orproductions , expressed\nover a set of non-terminal symbols and a set of terminal symbols. Formally,\na particular context-free language is the set of strings that can be derived\nfrom a particular context-free grammar . 22 CHAPTER 18 \u2022 C ONTEXT -FREE GRAMMARS AND CONSTITUENCY PARSING\n\u2022Structural ambiguity is a signi\ufb01cant problem for parsers. Common sources\nof structural ambiguity include PP-attachment andcoordination ambiguity .\n\u2022Dynamic programming parsing algorithms, such as CKY , use a table of\npartial parses to ef\ufb01ciently parse ambiguous sentences.\n\u2022CKY restricts the form of the grammar to Chomsky normal form (CNF).\n\u2022 The basic CKY algorithm compactly represents all possible parses of the sen-\ntence but doesn\u2019t choose a single best parse.\n\u2022 Choosing a single parse from all possible parses ( disambiguation ) can be\ndone by neural constituency parsers .\n\u2022 Span-based neural constituency parses train a neural classi\ufb01er to assign a score\nto each constituent, and then use a modi\ufb01ed version of CKY to combine these\nconstituent scores to \ufb01nd the best-scoring parse tree.\n\u2022 Parsers are evaluated with three metrics: labeled recall ,labeled precision ,\nandcross-brackets .\n\u2022Partial parsing andchunking are methods for identifying shallow syntac-\ntic constituents in a text. They are solved by sequence models trained on\nsyntactically-annotated data.\nBibliographical and Historical Notes\nAccording to Percival (1976), the idea of breaking up a sentence into a hierarchy of\nconstituents appeared in the V\u00a8olkerpsychologie of the groundbreaking psychologist\nWilhelm Wundt (Wundt, 1900):\n...den sprachlichen Ausdruck f \u00a8ur die willk \u00a8urliche Gliederung einer Ge-\nsammtvorstellung in ihre in logische Beziehung zueinander gesetzten\nBestandteile\n[the linguistic expression for the arbitrary division of a total idea\ninto its constituent parts placed in logical relations to one another]\nWundt\u2019s idea of constituency was taken up into linguistics by Leonard Bloom-\n\ufb01eld in his early book An Introduction to the Study of Language (Bloom\ufb01eld, 1914).\nBy the time of his later book, Language (Bloom\ufb01eld, 1933), what was then called\n\u201cimmediate-constituent analysis\u201d was a well-established method of syntactic study\nin the United States. By contrast, traditional European grammar, dating from the\nClassical period, de\ufb01ned relations between words rather than constituents, and Eu-\nropean syntacticians retained this emphasis on such dependency grammars, the sub-\nject of Chapter 19. (And indeed, both dependency and constituency grammars have\nbeen in vogue in computational linguistics at different times).\nAmerican Structuralism saw a number of speci\ufb01c de\ufb01nitions of the immediate\nconstituent, couched in terms of their search for a \u201cdiscovery procedure\u201d: a method-\nological algorithm for describing the syntax of a language. In general, these attempt\nto capture the intuition that \u201cThe primary criterion of the immediate constituent\nis the degree in which combinations behave as simple units\u201d (Bazell, 1952/1966, p.\n284). The most well known of the speci\ufb01c de\ufb01nitions is Harris\u2019 idea of distributional\nsimilarity to individual units, with the substitutability test. Essentially, the method\nproceeded by breaking up a construction into constituents by attempting to substitute\nsimple structures for possible constituents\u2014if a substitution of a simple form, say, BIBLIOGRAPHICAL AND HISTORICAL NOTES 23\nman, was substitutable in a construction for a more complex set (like intense young\nman), then the form intense young man was probably a constituent. Harris\u2019s test was\nthe beginning of the intuition that a constituent is a kind of equivalence class.\nThe context-free grammar was a formalization of this idea of hierarchical\nconstituency de\ufb01ned in Chomsky (1956) and further expanded upon (and argued\nagainst) in Chomsky (1957) and Chomsky (1956/1975). Shortly after Chomsky\u2019s\ninitial work, the context-free grammar was reinvented by Backus (1959) and inde-\npendently by Naur et al. (1960) in their descriptions of the ALGOL programming\nlanguage; Backus (1996) noted that he was in\ufb02uenced by the productions of Emil\nPost and that Naur\u2019s work was independent of his (Backus\u2019) own. After this early\nwork, a great number of computational models of natural language processing were\nbased on context-free grammars because of the early development of ef\ufb01cient pars-\ning algorithms.\nDynamic programming parsing has a history of independent discovery. Ac-\ncording to the late Martin Kay (personal communication), a dynamic programming\nparser containing the roots of the CKY algorithm was \ufb01rst implemented by John\nCocke in 1960. Later work extended and formalized the algorithm, as well as prov-\ning its time complexity (Kay 1967, Younger 1967, Kasami 1965). The related well-\nformed substring table (WFST ) seems to have been independently proposed by WFST\nKuno (1965) as a data structure that stores the results of all previous computations\nin the course of the parse. Based on a generalization of Cocke\u2019s work, a similar\ndata structure had been independently described in Kay (1967) (and Kay 1973). The\ntop-down application of dynamic programming to parsing was described in Earley\u2019s\nPh.D. dissertation (Earley 1968, Earley 1970). Sheil (1976) showed the equivalence\nof the WFST and the Earley algorithm. Norvig (1991) shows that the ef\ufb01ciency of-\nfered by dynamic programming can be captured in any language with a memoization\nfunction (such as in LISP) simply by wrapping the memoization operation around a\nsimple top-down parser.\nThe earliest disambiguation algorithms for parsing were based on probabilistic\ncontext-free grammars , \ufb01rst worked out by Booth (1969) and Salomaa (1969); seeprobabilistic\ncontext-free\ngrammarsAppendix C for more history. Neural methods were \ufb01rst applied to parsing at around\nthe same time as statistical parsing methods were developed (Henderson, 1994). In\nthe earliest work neural networks were used to estimate some of the probabilities for\nstatistical constituency parsers (Henderson, 2003, 2004; Emami and Jelinek, 2005)\n. The next decades saw a wide variety of neural parsing algorithms, including re-\ncursive neural architectures (Socher et al., 2011, 2013), encoder-decoder models\n(Vinyals et al., 2015; Choe and Charniak, 2016), and the idea of focusing on spans\n(Cross and Huang, 2016). For more on the span-based self-attention approach we\ndescribe in this chapter see Stern et al. (2017), Gaddy et al. (2018), Kitaev and Klein\n(2018), and Kitaev et al. (2019). See Chapter 20 for the parallel history of neural\ndependency parsing.\nThe classic reference for parsing algorithms is Aho and Ullman (1972); although\nthe focus of that book is on computer languages, most of the algorithms have been\napplied to natural language. 24 CHAPTER 18 \u2022 C ONTEXT -FREE GRAMMARS AND CONSTITUENCY PARSING\nExercises\n18.1 Implement the algorithm to convert arbitrary context-free grammars to CNF.\nApply your program to the L1grammar.\n18.2 Implement the CKY algorithm and test it with your converted L1grammar.\n18.3 Rewrite the CKY algorithm given in Fig. 18.12 on page 14 so that it can accept\ngrammars that contain unit productions.\n18.4 Discuss how to augment a parser to deal with input that may be incorrect, for\nexample, containing spelling errors or mistakes arising from automatic speech\nrecognition.\n18.5 Implement the PARSEV AL metrics described in Section 18.8. Next, use a\nparser and a treebank, compare your metrics against a standard implementa-\ntion. Analyze the errors in your approach. Exercises 25\nAho, A. V . and J. D. Ullman. 1972. The Theory of Parsing,\nTranslation, and Compiling , volume 1. Prentice Hall.\nBackus, J. W. 1959. The syntax and semantics of the\nproposed international algebraic language of the Zurich\nACM-GAMM Conference. Information Processing: Pro-\nceedings of the International Conference on Information\nProcessing, Paris . UNESCO.\nBackus, J. W. 1996. Transcript of question and answer ses-\nsion. In R. L. Wexelblat, ed., History of Programming\nLanguages , page 162. Academic Press.\nBazell, C. E. 1952/1966. The correspondence fallacy in\nstructural linguistics. In E. P. Hamp, F. W. Householder,\nand R. Austerlitz, eds, Studies by Members of the En-\nglish Department, Istanbul University (3), reprinted in\nReadings in Linguistics II (1966) , 271\u2013298. University of\nChicago Press.\nBlack, E., S. P. Abney, D. Flickinger, C. Gdaniec, R. Gr-\nishman, P. Harrison, D. Hindle, R. Ingria, F. Jelinek,\nJ. L. Klavans, M. Y . Liberman, M. P. Marcus, S. Roukos,\nB. Santorini, and T. Strzalkowski. 1991. A procedure for\nquantitatively comparing the syntactic coverage of En-\nglish grammars. Speech and Natural Language Work-\nshop .\nBloom\ufb01eld, L. 1914. An Introduction to the Study of Lan-\nguage . Henry Holt and Company.\nBloom\ufb01eld, L. 1933. Language . University of Chicago\nPress.\nBooth, T. L. 1969. Probabilistic representation of formal\nlanguages. IEEE Conference Record of the 1969 Tenth\nAnnual Symposium on Switching and Automata Theory .\nCharniak, E. 1997. Statistical parsing with a context-free\ngrammar and word statistics. AAAI .\nChoe, D. K. and E. Charniak. 2016. Parsing as language\nmodeling. EMNLP .\nChomsky, N. 1956. Three models for the description of\nlanguage. IRE Transactions on Information Theory ,\n2(3):113\u2013124.\nChomsky, N. 1956/1975. The Logical Structure of Linguistic\nTheory . Plenum.\nChomsky, N. 1957. Syntactic Structures . Mouton.\nChomsky, N. 1963. Formal properties of grammars. In R. D.\nLuce, R. Bush, and E. Galanter, eds, Handbook of Math-\nematical Psychology , volume 2, 323\u2013418. Wiley.\nCollins, M. 1999. Head-Driven Statistical Models for Natu-\nral Language Parsing . Ph.D. thesis, University of Penn-\nsylvania, Philadelphia.\nCross, J. and L. Huang. 2016. Span-based constituency pars-\ning with a structure-label system and provably optimal\ndynamic oracles. EMNLP .\nEarley, J. 1968. An Ef\ufb01cient Context-Free Parsing Algorithm .\nPh.D. thesis, Carnegie Mellon University, Pittsburgh, PA.\nEarley, J. 1970. An ef\ufb01cient context-free parsing algorithm.\nCACM , 6(8):451\u2013455.\nEmami, A. and F. Jelinek. 2005. A neural syntactic language\nmodel. Machine learning , 60(1):195\u2013227.\nGaddy, D., M. Stern, and D. Klein. 2018. What\u2019s going on\nin neural constituency parsers? an analysis. NAACL HLT .Harris, Z. S. 1946. From morpheme to utterance. Language ,\n22(3):161\u2013183.\nHenderson, J. 1994. Description Based Parsing in a Connec-\ntionist Network . Ph.D. thesis, University of Pennsylvania,\nPhiladelphia, PA.\nHenderson, J. 2003. Inducing history representations for\nbroad coverage statistical parsing. HLT-NAACL-03 .\nHenderson, J. 2004. Discriminative training of a neural net-\nwork statistical parser. ACL.\nHopcroft, J. E. and J. D. Ullman. 1979. Introduction to Au-\ntomata Theory, Languages, and Computation . Addison-\nWesley.\nKaplan, R. M. 1973. A general syntactic processor. In\nR. Rustin, ed., Natural Language Processing , 193\u2013241.\nAlgorithmics Press.\nKasami, T. 1965. An ef\ufb01cient recognition and syntax anal-\nysis algorithm for context-free languages. Technical\nReport AFCRL-65-758, Air Force Cambridge Research\nLaboratory, Bedford, MA.\nKay, M. 1967. Experiments with a powerful parser. COL-\nING.\nKay, M. 1973. The MIND system. In R. Rustin, ed., Natural\nLanguage Processing , 155\u2013188. Algorithmics Press.\nKay, M. 1982. Algorithm schemata and data structures in\nsyntactic processing. In S. All \u00b4en, ed., Text Processing:\nText Analysis and Generation, Text Typology and Attribu-\ntion, 327\u2013358. Almqvist and Wiksell, Stockholm.\nKitaev, N., S. Cao, and D. Klein. 2019. Multilingual\nconstituency parsing with self-attention and pre-training.\nACL.\nKitaev, N. and D. Klein. 2018. Constituency parsing with a\nself-attentive encoder. ACL.\nKuno, S. 1965. The predictive analyzer and a path elimina-\ntion technique. CACM , 8(7):453\u2013462.\nMagerman, D. M. 1995. Statistical decision-tree models for\nparsing. ACL.\nNaur, P., J. W. Backus, F. L. Bauer, J. Green, C. Katz,\nJ. McCarthy, A. J. Perlis, H. Rutishauser, K. Samelson,\nB. Vauquois, J. H. Wegstein, A. van Wijnagaarden, and\nM. Woodger. 1960. Report on the algorithmic language\nALGOL 60. CACM , 3(5):299\u2013314. Revised in CACM\n6:1, 1-17, 1963.\nNorvig, P. 1991. Techniques for automatic memoization with\napplications to context-free parsing. Computational Lin-\nguistics , 17(1):91\u201398.\nPercival, W. K. 1976. On the historical source of immedi-\nate constituent analysis. In J. D. McCawley, ed., Syntax\nand Semantics Volume 7, Notes from the Linguistic Un-\nderground , 229\u2013242. Academic Press.\nPollard, C. and I. A. Sag. 1994. Head-Driven Phrase Struc-\nture Grammar . University of Chicago Press.\nSalomaa, A. 1969. Probabilistic and weighted grammars.\nInformation and Control , 15:529\u2013544.\nSekine, S. and M. Collins. 1997. The evalb software. http:\n//cs.nyu.edu/cs/projects/proteus/evalb .\nSheil, B. A. 1976. Observations on context free parsing.\nSMIL: Statistical Methods in Linguistics , 1:71\u2013109.\nSocher, R., J. Bauer, C. D. Manning, and A. Y . Ng. 2013.\nParsing with compositional vector grammars. ACL. 26 Chapter 18 \u2022 Context-Free Grammars and Constituency Parsing\nSocher, R., C. C.-Y . Lin, A. Y . Ng, and C. D. Manning. 2011.\nParsing natural scenes and natural language with recur-\nsive neural networks. ICML .\nStern, M., J. Andreas, and D. Klein. 2017. A minimal span-\nbased neural constituency parser. ACL.\nVinyals, O., \u0141. Kaiser, T. Koo, S. Petrov, I. Sutskever,\nand G. Hinton. 2015. Grammar as a foreign language.\nNeurIPS .\nWang, W. and B. Chang. 2016. Graph-based dependency\nparsing with bidirectional LSTM. ACL.\nWundt, W. 1900. V\u00a8olkerpsychologie: eine Untersuchung der\nEntwicklungsgesetze von Sprache, Mythus, und Sitte . W.\nEngelmann, Leipzig. Band II: Die Sprache, Zweiter Teil.\nYounger, D. H. 1967. Recognition and parsing of context-\nfree languages in time n3.Information and Control ,\n10:189\u2013208.", "19.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\n19Dependency Parsing\nTout mot qui fait partie d\u2019une phrase... Entre lui et ses voisins, l\u2019esprit aperc \u00b8oit\ndes connexions, dont l\u2019ensemble forme la charpente de la phrase.\n[Between each word in a sentence and its neighbors, the mind perceives con-\nnections . These connections together form the scaffolding of the sentence.]\nLucien Tesni `ere. 1959. \u00b4El\u00b4ements de syntaxe structurale, A.1.\u00a74\nThe focus of the last chapter was on context-free grammars and constituent-\nbased representations. Here we present another important family of grammar for-\nmalisms called dependency grammars . In dependency formalisms, phrasal con-dependency\ngrammars\nstituents and phrase-structure rules do not play a direct role. Instead, the syntactic\nstructure of a sentence is described solely in terms of directed binary grammatical\nrelations between the words , as in the following dependency parse:\nIprefer themorning \ufb02ight through Denvernsubjobj\ndet\ncompoundnmod\ncaseroot\n(19.1)\nRelations among the words are illustrated above the sentence with directed, labeled\narcs from heads todependents . We call this a typed dependency structure becausetyped\ndependency\nthe labels are drawn from a \ufb01xed inventory of grammatical relations. A root node\nexplicitly marks the root of the tree, the head of the entire structure.\nFigure 19.1 on the next page shows the dependency analysis from (19.1) but vi-\nsualized as a tree, alongside its corresponding phrase-structure analysis of the kind\ngiven in the prior chapter. Note the absence of nodes corresponding to phrasal con-\nstituents or lexical categories in the dependency parse; the internal structure of the\ndependency parse consists solely of directed relations between words. These head-\ndependent relationships directly encode important information that is often buried in\nthe more complex phrase-structure parses. For example, the arguments to the verb\nprefer are directly linked to it in the dependency structure, while their connection\nto the main verb is more distant in the phrase-structure tree. Similarly, morning\nandDenver , modi\ufb01ers of \ufb02ight , are linked to it directly in the dependency structure.\nThis fact that the head-dependent relations are a good proxy for the semantic rela-\ntionship between predicates and their arguments is an important reason why depen-\ndency grammars are currently more common than constituency grammars in natural\nlanguage processing.\nAnother major advantage of dependency grammars is their ability to deal with\nlanguages that have a relatively free word order . For example, word order in Czech free word order\ncan be much more \ufb02exible than in English; a grammatical object might occur before\nor after a location adverbial . A phrase-structure grammar would need a separate rule 2CHAPTER 19 \u2022 D EPENDENCY PARSING\nprefer\n\ufb02ight\nDenver\nthroughmorning theIS\nVP\nNP\nNom\nPP\nNP\nPro\nDenverP\nthroughNom\nNoun\n\ufb02ightNom\nNoun\nmorningDet\ntheVerb\npreferNP\nPro\nI\nFigure 19.1 Dependency and constituent analyses for I prefer the morning \ufb02ight through Denver.\nfor each possible place in the parse tree where such an adverbial phrase could occur.\nA dependency-based approach can have just one link type representing this particu-\nlar adverbial relation; dependency grammar approaches can thus abstract away a bit\nmore from word order information.\nIn the following sections, we\u2019ll give an inventory of relations used in dependency\nparsing, discuss two families of parsing algorithms (transition-based, and graph-\nbased), and discuss evaluation.\n19.1 Dependency Relations\nThe traditional linguistic notion of grammatical relation provides the basis for thegrammatical\nrelation\nbinary relations that comprise these dependency structures. The arguments to these\nrelations consist of a head and a dependent . The head plays the role of the central head\ndependent organizing word, and the dependent as a kind of modi\ufb01er. The head-dependent rela-\ntionship is made explicit by directly linking heads to the words that are immediately\ndependent on them.\nIn addition to specifying the head-dependent pairs, dependency grammars allow\nus to classify the kinds of grammatical relations, or grammatical function that thegrammatical\nfunction\ndependent plays with respect to its head. These include familiar notions such as\nsubject ,direct object andindirect object . In English these notions strongly corre-\nlate with, but by no means determine, both position in a sentence and constituent\ntype and are therefore somewhat redundant with the kind of information found in\nphrase-structure trees. However, in languages with more \ufb02exible word order, the\ninformation encoded directly in these grammatical relations is critical since phrase-\nbased constituent syntax provides little help.\nLinguists have developed taxonomies of relations that go well beyond the famil-\niar notions of subject and object. While there is considerable variation from theory 19.1 \u2022 D EPENDENCY RELATIONS 3\nClausal Argument Relations Description\nNSUBJ Nominal subject\nOBJ Direct object\nIOBJ Indirect object\nCCOMP Clausal complement\nNominal Modi\ufb01er Relations Description\nNMOD Nominal modi\ufb01er\nAMOD Adjectival modi\ufb01er\nAPPOS Appositional modi\ufb01er\nDET Determiner\nCASE Prepositions, postpositions and other case markers\nOther Notable Relations Description\nCONJ Conjunct\nCC Coordinating conjunction\nFigure 19.2 Some of the Universal Dependency relations (de Marneffe et al., 2021).\nto theory, there is enough commonality that cross-linguistic standards have been\ndeveloped. The Universal Dependencies (UD) project (de Marneffe et al., 2021),Universal\nDependencies\nan open community effort to annotate dependencies and other aspects of grammar\nacross more than 100 languages, provides an inventory of 37 dependency relations.\nFig. 19.2 shows a subset of the UD relations and Fig. 19.3 provides some examples.\nThe motivation for all of the relations in the Universal Dependency scheme is\nbeyond the scope of this chapter, but the core set of frequently used relations can be\nbroken into two sets: clausal relations that describe syntactic roles with respect to a\npredicate (often a verb), and modi\ufb01er relations that categorize the ways that words\ncan modify their heads.\nConsider, for example, the following sentence:\nUnited canceled themorning \ufb02ights toHoustonnsubjobj\ndet\ncompoundnmod\ncaseroot\n(19.2)\nHere the clausal relations NSUBJ and OBJidentify the subject and direct object of\nthe predicate cancel , while the NMOD ,DET, and CASE relations denote modi\ufb01ers of\nthe nouns \ufb02ights andHouston .\n19.1.1 Dependency Formalisms\nA dependency structure can be represented as a directed graph G= (V;A), consisting\nof a set of vertices V, and a set of ordered pairs of vertices A, which we\u2019ll call arcs.\nFor the most part we will assume that the set of vertices, V, corresponds exactly\nto the set of words in a given sentence. However, they might also correspond to\npunctuation, or when dealing with morphologically complex languages the set of\nvertices might consist of stems and af\ufb01xes. The set of arcs, A, captures the head-\ndependent and grammatical function relationships between the elements in V.\nDifferent grammatical theories or formalisms may place further constraints on\nthese dependency structures. Among the more frequent restrictions are that the struc-\ntures must be connected, have a designated root node, and be acyclic or planar. Of\nmost relevance to the parsing approaches discussed in this chapter is the common, 4CHAPTER 19 \u2022 D EPENDENCY PARSING\nRelation Examples with head anddependent\nNSUBJ United canceled the \ufb02ight.\nOBJ United diverted the\ufb02ight to Reno.\nWebooked her the \ufb01rst \ufb02ight to Miami.\nIOBJ Webooked herthe \ufb02ight to Miami.\nCOMPOUND We took the morning \ufb02ight .\nNMOD \ufb02ight toHouston .\nAMOD Book the cheapest \ufb02ight .\nAPPOS United , aunit of UAL, matched the fares.\nDET The \ufb02ight was canceled.\nWhich \ufb02ight was delayed?\nCONJ We\ufb02ewto Denver and drove to Steamboat.\nCC We \ufb02ew to Denver anddrove to Steamboat.\nCASE Book the \ufb02ight through Houston .\nFigure 19.3 Examples of some Universal Dependency relations.\ncomputationally-motivated, restriction to rooted trees. That is, a dependency treedependency\ntree\nis a directed graph that satis\ufb01es the following constraints:\n1. There is a single designated root node that has no incoming arcs.\n2. With the exception of the root node, each vertex has exactly one incoming arc.\n3. There is a unique path from the root node to each vertex in V.\nTaken together, these constraints ensure that each word has a single head, that the\ndependency structure is connected, and that there is a single root node from which\none can follow a unique directed path to each of the words in the sentence.\n19.1.2 Projectivity\nThe notion of projectivity imposes an additional constraint that is derived from the\norder of the words in the input. An arc from a head to a dependent is said to be\nprojective if there is a path from the head to every word that lies between the head projective\nand the dependent in the sentence. A dependency tree is then said to be projective if\nall the arcs that make it up are projective. All the dependency trees we\u2019ve seen thus\nfar have been projective. There are, however, many valid constructions which lead\nto non-projective trees, particularly in languages with relatively \ufb02exible word order.\nConsider the following example.\nJetBlue canceled our \ufb02ight this morning which was already latensubjobjobl\ndetacl:relcl\ndet nsubjcop\nadvroot\n(19.3)\nIn this example, the arc from \ufb02ight to its modi\ufb01er lateis non-projective since there\nis no path from \ufb02ight to the intervening words thisandmorning . As we can see from\nthis diagram, projectivity (and non-projectivity) can be detected in the way we\u2019ve\nbeen drawing our trees. A dependency tree is projective if it can be drawn with\nno crossing edges. Here there is no way to link \ufb02ight to its dependent latewithout\ncrossing the arc that links morning to its head. 19.1 \u2022 D EPENDENCY RELATIONS 5\nOur concern with projectivity arises from two related issues. First, the most\nwidely used English dependency treebanks were automatically derived from phrase-\nstructure treebanks through the use of head-\ufb01nding rules. The trees generated in such\na fashion will always be projective, and hence will be incorrect when non-projective\nexamples like this one are encountered.\nSecond, there are computational limitations to the most widely used families of\nparsing algorithms. The transition-based approaches discussed in Section 19.2 can\nonly produce projective trees, hence any sentences with non-projective structures\nwill necessarily contain some errors. This limitation is one of the motivations for\nthe more \ufb02exible graph-based parsing approach described in Section 19.3.\n19.1.3 Dependency Treebanks\nTreebanks play a critical role in the development and evaluation of dependency\nparsers. They are used for training parsers, they act as the gold labels for evaluating\nparsers, and they also provide useful information for corpus linguistics studies.\nDependency treebanks are created by having human annotators directly generate\ndependency structures for a given corpus, or by hand-correcting the output of an\nautomatic parser. A few early treebanks were also based on using a deterministic\nprocess to translate existing constituent-based treebanks into dependency trees.\nThe largest open community project for building dependency trees is the Univer-\nsal Dependencies project at https://universaldependencies.org/ introduced\nabove, which currently has almost 200 dependency treebanks in more than 100 lan-\nguages (de Marneffe et al., 2021). Here are a few UD examples showing dependency\ntrees for sentences in Spanish, Basque, and Mandarin Chinese:\nVERB ADP DET NOUN ADP DET NUM PUNCT\nSubiremos a el tren a las cinco .\nwe-will-board on the train at the \ufb01ve .obl\ndetcase\ndetobl:tmod\ncasepunct\n[Spanish] Subiremos al tren a las cinco. \u201cWe will be boarding the train at \ufb01ve.\u201d (19.4)\nNOUN NOUN VERB AUX PUNCT\nEkaitzak itsasontzia hondoratu du .\nstorm (Erg.) ship (Abs.) sunk has .nsubj\nobj auxpunct\n[Basque] Ekaitzak itsasontzia hondoratu du. \u201cThe storm has sunk the ship.\u201d (19.5) 6CHAPTER 19 \u2022 D EPENDENCY PARSING\nADV PRON NOUN ADV VERB VERB NOUN\n\u4f46\u6211\u6628\u5929 \u624d \u6536 \u5230\u4fe1\nbut I yesterday only-then receive arrive letter .adv\nnsubj\nobj:tmod\nadvmod compound:vvobj\n[Chinese] \u4f46\u6211\u6628\u5929\u624d\u6536\u5230\u4fe1\u201cBut I didn\u2019t receive the letter until yesterday\u201d (19.6)\n19.2 Transition-Based Dependency Parsing\nOur \ufb01rst approach to dependency parsing is called transition-based parsing. This transition-based\narchitecture draws on shift-reduce parsing , a paradigm originally developed for\nanalyzing programming languages (Aho and Ullman, 1972). In transition-based\nparsing we\u2019ll have a stack on which we build the parse, a buffer of tokens to be\nparsed, and a parser which takes actions on the parse via a predictor called an oracle ,\nas illustrated in Fig. 19.4.\nwnw1w2s2...s1snParserInput bu\ufb00erStackOracleLEFTARCRIGHTARCSHIFTActionDependencyRelationsw3w2\nFigure 19.4 Basic transition-based parser. The parser examines the top two elements of the\nstack and selects an action by consulting an oracle that examines the current con\ufb01guration.\nThe parser walks through the sentence left-to-right, successively shifting items\nfrom the buffer onto the stack. At each time point we examine the top two elements\non the stack, and the oracle makes a decision about what transition to apply to build\nthe parse. The possible transitions correspond to the intuitive actions one might take\nin creating a dependency tree by examining the words in a single pass over the input\nfrom left to right (Covington, 2001):\n\u2022 Assign the current word as the head of some previously seen word,\n\u2022 Assign some previously seen word as the head of the current word,\n\u2022 Postpone dealing with the current word, storing it for later processing.\nWe\u2019ll formalize this intuition with the following three transition operators that\nwill operate on the top two elements of the stack:\n\u2022LEFT ARC: Assert a head-dependent relation between the word at the top of\nthe stack and the second word; remove the second word from the stack.\n\u2022RIGHT ARC: Assert a head-dependent relation between the second word on\nthe stack and the word at the top; remove the top word from the stack; 19.2 \u2022 T RANSITION -BASED DEPENDENCY PARSING 7\n\u2022SHIFT : Remove the word from the front of the input buffer and push it onto\nthe stack.\nWe\u2019ll sometimes call operations like LEFT ARCand RIGHT ARCreduce operations,\nbased on a metaphor from shift-reduce parsing, in which reducing means combin-\ning elements on the stack. There are some preconditions for using operators. The\nLEFT ARCoperator cannot be applied when ROOT is the second element of the stack\n(since by de\ufb01nition the ROOT node cannot have any incoming arcs). And both the\nLEFT ARCand RIGHT ARCoperators require two elements to be on the stack to be\napplied.\nThis particular set of operators implements what is known as the arc standard arc standard\napproach to transition-based parsing (Covington 2001, Nivre 2003). In arc standard\nparsing the transition operators only assert relations between elements at the top of\nthe stack, and once an element has been assigned its head it is removed from the\nstack and is not available for further processing. As we\u2019ll see, there are alterna-\ntive transition systems which demonstrate different parsing behaviors, but the arc\nstandard approach is quite effective and is simple to implement.\nThe speci\ufb01cation of a transition-based parser is quite simple, based on repre-\nsenting the current state of the parse as a con\ufb01guration : the stack, an input buffer con\ufb01guration\nof words or tokens, and a set of relations representing a dependency tree. Parsing\nmeans making a sequence of transitions through the space of possible con\ufb01gura-\ntions. We start with an initial con\ufb01guration in which the stack contains the ROOT\nnode, the buffer has the tokens in the sentence, and an empty set of relations repre-\nsents the parse. In the \ufb01nal goal state, the stack and the word list should be empty,\nand the set of relations will represent the \ufb01nal parse. Fig. 19.5 gives the algorithm.\nfunction DEPENDENCY PARSE (words )returns dependency tree\nstate f[root], [ words ], []g; initial con\ufb01guration\nwhile state not \ufb01nal\nt ORACLE (state ) ; choose a transition operator to apply\nstate APPLY (t,state ) ; apply it, creating a new state\nreturn state\nFigure 19.5 A generic transition-based dependency parser\nAt each step, the parser consults an oracle (we\u2019ll come back to this shortly) that\nprovides the correct transition operator to use given the current con\ufb01guration. It then\napplies that operator to the current con\ufb01guration, producing a new con\ufb01guration.\nThe process ends when all the words in the sentence have been consumed and the\nROOT node is the only element remaining on the stack.\nThe ef\ufb01ciency of transition-based parsers should be apparent from the algorithm.\nThe complexity is linear in the length of the sentence since it is based on a single\nleft to right pass through the words in the sentence. (Each word must \ufb01rst be shifted\nonto the stack and then later reduced.)\nNote that unlike the dynamic programming and search-based approaches dis-\ncussed in Chapter 18, this approach is a straightforward greedy algorithm\u2014the or-\nacle provides a single choice at each step and the parser proceeds with that choice,\nno other options are explored, no backtracking is employed, and a single parse is\nreturned in the end.\nFigure 19.6 illustrates the operation of the parser with the sequence of transitions 8CHAPTER 19 \u2022 D EPENDENCY PARSING\nleading to a parse for the following example.\nBook methemorning \ufb02ightiobjobj\ndet\ncompoundroot\n(19.7)\nLet\u2019s consider the state of the con\ufb01guration at Step 2, after the word mehas been\npushed onto the stack.\nStack Word List Relations\n[root, book, me] [the, morning, \ufb02ight]\nThe correct operator to apply here is RIGHT ARCwhich assigns book as the head of\nmeand pops mefrom the stack resulting in the following con\ufb01guration.\nStack Word List Relations\n[root, book] [the, morning, \ufb02ight] (book!me)\nAfter several subsequent applications of the SHIFT operator, the con\ufb01guration in\nStep 6 looks like the following:\nStack Word List Relations\n[root, book, the, morning, \ufb02ight] [] (book!me)\nHere, all the remaining words have been passed onto the stack and all that is left\nto do is to apply the appropriate reduce operators. In the current con\ufb01guration, we\nemploy the LEFT ARCoperator resulting in the following state.\nStack Word List Relations\n[root, book, the, \ufb02ight] [] (book!me)\n(morning \ufb02ight)\nAt this point, the parse for this sentence consists of the following structure.\nBook methemorning \ufb02ightiobj compound\n(19.8)\nThere are several important things to note when examining sequences such as\nthe one in Figure 19.6. First, the sequence given is not the only one that might lead\nto a reasonable parse. In general, there may be more than one path that leads to the\nsame result, and due to ambiguity, there may be other transition sequences that lead\nto different equally valid parses.\nSecond, we are assuming that the oracle always provides the correct operator\nat each point in the parse\u2014an assumption that is unlikely to be true in practice.\nAs a result, given the greedy nature of this algorithm, incorrect choices will lead to\nincorrect parses since the parser has no opportunity to go back and pursue alternative\nchoices. Section 19.2.4 will introduce several techniques that allow transition-based\napproaches to explore the search space more fully. 19.2 \u2022 T RANSITION -BASED DEPENDENCY PARSING 9\nStep Stack Word List Action Relation Added\n0 [root] [book, me, the, morning, \ufb02ight] SHIFT\n1 [root, book] [me, the, morning, \ufb02ight] SHIFT\n2 [root, book, me] [the, morning, \ufb02ight] RIGHT ARC (book!me)\n3 [root, book] [the, morning, \ufb02ight] SHIFT\n4 [root, book, the] [morning, \ufb02ight] SHIFT\n5 [root, book, the, morning] [\ufb02ight] SHIFT\n6 [root, book, the, morning, \ufb02ight] [] LEFT ARC (morning \ufb02ight)\n7 [root, book, the, \ufb02ight] [] LEFT ARC (the \ufb02ight)\n8 [root, book, \ufb02ight] [] RIGHT ARC (book!\ufb02ight)\n9 [root, book] [] RIGHT ARC (root!book)\n10 [root] [] Done\nFigure 19.6 Trace of a transition-based parse.\nFinally, for simplicity, we have illustrated this example without the labels on\nthe dependency relations. To produce labeled trees, we can parameterize the LEFT -\nARCand RIGHT ARCoperators with dependency labels, as in LEFT ARC(NSUBJ ) or\nRIGHT ARC(OBJ). This is equivalent to expanding the set of transition operators from\nour original set of three to a set that includes LEFT ARCand RIGHT ARCoperators for\neach relation in the set of dependency relations being used, plus an additional one\nfor the SHIFT operator. This, of course, makes the job of the oracle more dif\ufb01cult\nsince it now has a much larger set of operators from which to choose.\n19.2.1 Creating an Oracle\nThe oracle for greedily selecting the appropriate transition is trained by supervised\nmachine learning. As with all supervised machine learning methods, we will need\ntraining data: con\ufb01gurations annotated with the correct transition to take. We can\ndraw these from dependency trees. And we need to extract features of the con-\n\ufb01guration. We\u2019ll introduce neural classi\ufb01ers that represent the con\ufb01guration via\nembeddings, as well as classic systems that use hand-designed features.\nGenerating Training Data\nThe oracle from the algorithm in Fig. 19.5 takes as input a con\ufb01guration and returns a\ntransition operator. Therefore, to train a classi\ufb01er, we will need con\ufb01gurations paired\nwith transition operators (i.e., LEFT ARC,RIGHT ARC, or SHIFT ). Unfortunately,\ntreebanks pair entire sentences with their corresponding trees, not con\ufb01gurations\nwith transitions.\nTo generate the required training data, we employ the oracle-based parsing algo-\nrithm in a clever way. We supply our oracle with the training sentences to be parsed\nalong with their corresponding reference parses from the treebank. To produce train-\ning instances, we then simulate the operation of the parser by running the algorithm\nand relying on a new training oracle to give us correct transition operators for each training oracle\nsuccessive con\ufb01guration.\nTo see how this works, let\u2019s \ufb01rst review the operation of our parser. It begins with\na default initial con\ufb01guration where the stack contains the ROOT , the input list is just\nthe list of words, and the set of relations is empty. The LEFT ARCand RIGHT ARC\noperators each add relations between the words at the top of the stack to the set of\nrelations being accumulated for a given sentence. Since we have a gold-standard\nreference parse for each training sentence, we know which dependency relations are\nvalid for a given sentence. Therefore, we can use the reference parse to guide the 10 CHAPTER 19 \u2022 D EPENDENCY PARSING\nStep Stack Word List Predicted Action\n0 [root] [book, the, \ufb02ight, through, houston] SHIFT\n1 [root, book] [the, \ufb02ight, through, houston] SHIFT\n2 [root, book, the] [\ufb02ight, through, houston] SHIFT\n3 [root, book, the, \ufb02ight] [through, houston] LEFT ARC\n4 [root, book, \ufb02ight] [through, houston] SHIFT\n5 [root, book, \ufb02ight, through] [houston] SHIFT\n6 [root, book, \ufb02ight, through, houston] [] LEFT ARC\n7 [root, book, \ufb02ight, houston ] [] RIGHT ARC\n8 [root, book, \ufb02ight] [] RIGHT ARC\n9 [root, book] [] RIGHT ARC\n10 [root] [] Done\nFigure 19.7 Generating training items consisting of con\ufb01guration/predicted action pairs by simulating a parse\nwith a given reference parse.\nselection of operators as the parser steps through a sequence of con\ufb01gurations.\nTo be more precise, given a reference parse and a con\ufb01guration, the training\noracle proceeds as follows:\n\u2022 Choose LEFT ARCif it produces a correct head-dependent relation given the\nreference parse and the current con\ufb01guration,\n\u2022 Otherwise, choose RIGHT ARCif (1) it produces a correct head-dependent re-\nlation given the reference parse and (2) all of the dependents of the word at\nthe top of the stack have already been assigned,\n\u2022 Otherwise, choose SHIFT .\nThe restriction on selecting the RIGHT ARCoperator is needed to ensure that a\nword is not popped from the stack, and thus lost to further processing, before all its\ndependents have been assigned to it.\nMore formally, during training the oracle has access to the following:\n\u2022 A current con\ufb01guration with a stack Sand a set of dependency relations Rc\n\u2022 A reference parse consisting of a set of vertices Vand a set of dependency\nrelations Rp\nGiven this information, the oracle chooses transitions as follows:\nLEFT ARC(r):if(S1r S2)2Rp\nRIGHT ARC(r):if(S2r S1)2Rpand8r0;w s:t:(S1r0w)2Rpthen(S1r0w)2Rc\nSHIFT :otherwise\nLet\u2019s walk through the processing of the following example as shown in Fig. 19.7.\nBook the\ufb02ight through Houstonobj\ndetnmod\ncaseroot\n(19.9)\nAt Step 1, LEFT ARCis not applicable in the initial con\ufb01guration since it asserts\na relation, (root book), not in the reference answer; RIGHT ARCdoes assert a\nrelation contained in the \ufb01nal answer (root !book), however book has not been\nattached to any of its dependents yet, so we have to defer, leaving SHIFT as the only 19.2 \u2022 T RANSITION -BASED DEPENDENCY PARSING 11\npossible action. The same conditions hold in the next two steps. In step 3, LEFT ARC\nis selected to link theto its head.\nNow consider the situation in Step 4.\nStack Word buffer Relations\n[root, book, \ufb02ight] [through, Houston] (the \ufb02ight)\nHere, we might be tempted to add a dependency relation between book and\ufb02ight ,\nwhich is present in the reference parse. But doing so now would prevent the later\nattachment of Houston since \ufb02ight would have been removed from the stack. For-\ntunately, the precondition on choosing RIGHT ARCprevents this choice and we\u2019re\nagain left with SHIFT as the only viable option. The remaining choices complete the\nset of operators needed for this example.\nTo recap, we derive appropriate training instances consisting of con\ufb01guration-\ntransition pairs from a treebank by simulating the operation of a parser in the con-\ntext of a reference dependency tree. We can deterministically record correct parser\nactions at each step as we progress through each training example, thereby creating\nthe training set we require.\n19.2.2 A feature-based classi\ufb01er\nWe\u2019ll now introduce two classi\ufb01ers for choosing transitions, here a classic feature-\nbased algorithm and in the next section a neural classi\ufb01er using embedding features.\nFeatured-based classi\ufb01ers generally use the same features we\u2019ve seen with part-\nof-speech tagging and partial parsing: Word forms, lemmas, parts of speech, the\nhead, and the dependency relation to the head. Other features may be relevant for\nsome languages, for example morphosyntactic features like case marking on subjects\nor objects. The features are extracted from the training con\ufb01gurations , which consist\nof the stack, the buffer and the current set of relations. Most useful are features\nreferencing the top levels of the stack, the words near the front of the buffer, and the\ndependency relations already associated with any of those elements.\nWe\u2019ll use a feature template as we did for sentiment analysis and part-of-speechfeature\ntemplate\ntagging. Feature templates allow us to automatically generate large numbers of spe-\nci\ufb01c features from a training set. For example, consider the following feature tem-\nplates that are based on single positions in a con\ufb01guration.\nhs1:w;opi;hs2:w;opihs1:t;opi;hs2:t;opi\nhb1:w;opi;hb1:t;opihs1:wt;opi (19.10)\nHere features are denoted as location :property , where s= stack, b= the word\nbuffer, w= word forms, t= part-of-speech, and op= operator. Thus the feature for\nthe word form at the top of the stack would be s1:w, the part of speech tag at the\nfront of the buffer b1:t, and the concatenated feature s1:wtrepresents the word form\nconcatenated with the part of speech of the word at the top of the stack. Consider\napplying these templates to the following intermediate con\ufb01guration derived from a\ntraining oracle for (19.2).\nStack Word buffer Relations\n[root, canceled, \ufb02ights] [to Houston] (canceled!United)\n(\ufb02ights!morning)\n(\ufb02ights!the) 12 CHAPTER 19 \u2022 D EPENDENCY PARSING\nThe correct transition here is SHIFT (you should convince yourself of this before\nproceeding). The application of our set of feature templates to this con\ufb01guration\nwould result in the following set of instantiated features.\nhs1:w=\ufb02ights ;op=shifti (19.11)\nhs2:w=canceled ;op=shifti\nhs1:t=NNS;op=shifti\nhs2:t=VBD;op=shifti\nhb1:w=to;op=shifti\nhb1:t=TO;op=shifti\nhs1:wt=\ufb02ightsNNS ;op=shifti\nGiven that the left and right arc transitions operate on the top two elements of the\nstack, features that combine properties from these positions are even more useful.\nFor example, a feature like s1:t\u000es2:tconcatenates the part of speech tag of the word\nat the top of the stack with the tag of the word beneath it.\nhs1:t\u000es2:t=NNSVBD ;op=shifti (19.12)\nGiven the training data and features, any classi\ufb01er, like multinomial logistic re-\ngression or support vector machines, can be used.\n19.2.3 A neural classi\ufb01er\nThe oracle can also be implemented by a neural classi\ufb01er. A standard architecture\nis simply to pass the sentence through an encoder, then take the presentation of the\ntop 2 words on the stack and the \ufb01rst word of the buffer, concatenate them, and\npresent to a feedforward network that predicts the transition to take (Kiperwasser\nand Goldberg, 2016; Kulmizev et al., 2019). Fig. 19.8 sketches this model. Learning\ncan be done with cross-entropy loss.\nw\u2026s2...s1Input bu\ufb00erStackLEFTARCRIGHTARCSHIFTActionDependencyRelationsw3w2ENCODERw1w2w3w4w5w6Parser OracleSoftmaxFFNws1s2e(w)e(s1)e(s2)\nFigure 19.8 Neural classi\ufb01er for the oracle for the transition-based parser. The parser takes\nthe top 2 words on the stack and the \ufb01rst word of the buffer, represents them by their encodings\n(from running the whole sentence through the encoder), concatenates the embeddings and\npasses through a softmax to choose a parser action (transition). 19.2 \u2022 T RANSITION -BASED DEPENDENCY PARSING 13\n19.2.4 Advanced Methods in Transition-Based Parsing\nThe basic transition-based approach can be elaborated in a number of ways to im-\nprove performance by addressing some of the most obvious \ufb02aws in the approach.\nAlternative Transition Systems\nThe arc-standard transition system described above is only one of many possible sys-\ntems. A frequently used alternative is the arc eager transition system. The arc eager arc eager\napproach gets its name from its ability to assert rightward relations much sooner\nthan in the arc standard approach. To see this, let\u2019s revisit the arc standard trace of\nExample 19.9, repeated here.\nBook the\ufb02ight through Houstonobj\ndetnmod\ncaseroot\nConsider the dependency relation between book and\ufb02ight in this analysis. As\nis shown in Fig. 19.7, an arc-standard approach would assert this relation at Step 8,\ndespite the fact that book and\ufb02ight \ufb01rst come together on the stack much earlier at\nStep 4. The reason this relation can\u2019t be captured at this point is due to the presence\nof the postnominal modi\ufb01er through Houston . In an arc-standard approach, depen-\ndents are removed from the stack as soon as they are assigned their heads. If \ufb02ight\nhad been assigned book as its head in Step 4, it would no longer be available to serve\nas the head of Houston .\nWhile this delay doesn\u2019t cause any issues in this example, in general the longer\na word has to wait to get assigned its head the more opportunities there are for\nsomething to go awry. The arc-eager system addresses this issue by allowing words\nto be attached to their heads as early as possible, before all the subsequent words\ndependent on them have been seen. This is accomplished through minor changes to\ntheLEFT ARCand RIGHT ARCoperators and the addition of a new REDUCE operator.\n\u2022LEFT ARC: Assert a head-dependent relation between the word at the front of\nthe input buffer and the word at the top of the stack; pop the stack.\n\u2022RIGHT ARC: Assert a head-dependent relation between the word on the top of\nthe stack and the word at the front of the input buffer; shift the word at the\nfront of the input buffer to the stack.\n\u2022SHIFT : Remove the word from the front of the input buffer and push it onto\nthe stack.\n\u2022REDUCE : Pop the stack.\nThe LEFT ARCand RIGHT ARCoperators are applied to the top of the stack and\nthe front of the input buffer, instead of the top two elements of the stack as in the\narc-standard approach. The RIGHT ARCoperator now moves the dependent to the\nstack from the buffer rather than removing it, thus making it available to serve as the\nhead of following words. The new REDUCE operator removes the top element from\nthe stack. Together these changes permit a word to be eagerly assigned its head and\nstill allow it to serve as the head for later dependents. The trace shown in Fig. 19.9\nillustrates the new decision sequence for this example.\nIn addition to demonstrating the arc-eager transition system, this example demon-\nstrates the power and \ufb02exibility of the overall transition-based approach. We were\nable to swap in a new transition system without having to make any changes to the 14 CHAPTER 19 \u2022 D EPENDENCY PARSING\nStep Stack Word List Action Relation Added\n0 [root] [book, the, \ufb02ight, through, houston] RIGHT ARC (root!book)\n1 [root, book] [the, \ufb02ight, through, houston] SHIFT\n2 [root, book, the] [\ufb02ight, through, houston] LEFT ARC (the \ufb02ight)\n3 [root, book] [\ufb02ight, through, houston] RIGHT ARC (book!\ufb02ight)\n4 [root, book, \ufb02ight] [through, houston] SHIFT\n5 [root, book, \ufb02ight, through] [houston] LEFT ARC (through houston)\n6 [root, book, \ufb02ight] [houston] RIGHT ARC (\ufb02ight!houston)\n7 [root, book, \ufb02ight, houston] [] REDUCE\n8 [root, book, \ufb02ight] [] REDUCE\n9 [root, book] [] REDUCE\n10 [root] [] Done\nFigure 19.9 A processing trace of Book the \ufb02ight through Houston using the arc-eager transition operators.\nunderlying parsing algorithm. This \ufb02exibility has led to the development of a di-\nverse set of transition systems that address different aspects of syntax and semantics\nincluding: assigning part of speech tags (Choi and Palmer, 2011a), allowing the\ngeneration of non-projective dependency structures (Nivre, 2009), assigning seman-\ntic roles (Choi and Palmer, 2011b), and parsing texts containing multiple languages\n(Bhat et al., 2017).\nBeam Search\nThe computational ef\ufb01ciency of the transition-based approach discussed earlier de-\nrives from the fact that it makes a single pass through the sentence, greedily making\ndecisions without considering alternatives. Of course, this is also a weakness \u2013 once\na decision has been made it can not be undone, even in the face of overwhelming\nevidence arriving later in a sentence. We can use beam search to explore alterna- beam search\ntive decision sequences. Recall from Chapter 9 that beam search uses a breadth-\ufb01rst\nsearch strategy with a heuristic \ufb01lter that prunes the search frontier to stay within a\n\ufb01xed-size beam width . beam width\nIn applying beam search to transition-based parsing, we\u2019ll elaborate on the al-\ngorithm given in Fig. 19.5. Instead of choosing the single best transition operator\nat each iteration, we\u2019ll apply all applicable operators to each state on an agenda and\nthen score the resulting con\ufb01gurations. We then add each of these new con\ufb01gura-\ntions to the frontier, subject to the constraint that there has to be room within the\nbeam. As long as the size of the agenda is within the speci\ufb01ed beam width, we can\nadd new con\ufb01gurations to the agenda. Once the agenda reaches the limit, we only\nadd new con\ufb01gurations that are better than the worst con\ufb01guration on the agenda\n(removing the worst element so that we stay within the limit). Finally, to insure that\nwe retrieve the best possible state on the agenda, the while loop continues as long as\nthere are non-\ufb01nal states on the agenda.\nThe beam search approach requires a more elaborate notion of scoring than we\nused with the greedy algorithm. There, we assumed that the oracle would be a\nsupervised classi\ufb01er that chose the best transition operator based on features of the\ncurrent con\ufb01guration. This choice can be viewed as assigning a score to all the\npossible transitions and picking the best one.\n\u02c6T(c) =argmaxScore (t;c)\nWith beam search we are now searching through the space of decision sequences,\nso it makes sense to base the score for a con\ufb01guration on its entire history. So we\ncan de\ufb01ne the score for a new con\ufb01guration as the score of its predecessor plus the 19.3 \u2022 G RAPH -BASED DEPENDENCY PARSING 15\nscore of the operator used to produce it.\nCon\ufb01gScore (c0) = 0:0\nCon\ufb01gScore (ci) = Con\ufb01gScore (ci\u00001)+Score (ti;ci\u00001)\nThis score is used both in \ufb01ltering the agenda and in selecting the \ufb01nal answer. The\nnew beam search version of transition-based parsing is given in Fig. 19.10.\nfunction DEPENDENCY BEAM PARSE (words ,width )returns dependency tree\nstate f[root], [ words ], [], 0.0 g;initial con\ufb01guration\nagenda hstatei ;initial agenda\nwhile agenda contains non-\ufb01nal states\nnewagenda hi\nfor each state2agenda do\nfor all ftjt2VALID OPERATORS (state )gdo\nchild APPLY (t,state )\nnewagenda ADDTOBEAM (child ,newagenda ,width )\nagenda newagenda\nreturn BESTOF(agenda )\nfunction ADDTOBEAM (state ,agenda ,width )returns updated agenda\nifLENGTH (agenda )<width then\nagenda INSERT (state ,agenda )\nelse if SCORE (state )>SCORE (WORST OF(agenda ))\nagenda REMOVE (WORST OF(agenda ))\nagenda INSERT (state ,agenda )\nreturn agenda\nFigure 19.10 Beam search applied to transition-based dependency parsing.\n19.3 Graph-Based Dependency Parsing\nGraph-based methods are the second important family of dependency parsing algo-\nrithms. Graph-based parsers are more accurate than transition-based parsers, espe-\ncially on long sentences; transition-based methods have trouble when the heads are\nvery far from the dependents (McDonald and Nivre, 2011). Graph-based methods\navoid this dif\ufb01culty by scoring entire trees, rather than relying on greedy local de-\ncisions. Furthermore, unlike transition-based approaches, graph-based parsers can\nproduce non-projective trees. Although projectivity is not a signi\ufb01cant issue for\nEnglish, it is de\ufb01nitely a problem for many of the world\u2019s languages.\nGraph-based dependency parsers search through the space of possible trees for a\ngiven sentence for a tree (or trees) that maximize some score. These methods encode\nthe search space as directed graphs and employ methods drawn from graph theory\nto search the space for optimal solutions. More formally, given a sentence Swe\u2019re\nlooking for the best dependency tree in Gs, the space of all possible trees for that\nsentence, that maximizes some score.\n\u02c6T(S) =argmax\nt2GSScore (t;S) 16 CHAPTER 19 \u2022 D EPENDENCY PARSING\nWe\u2019ll make the simplifying assumption that this score can be edge-factored , edge-factored\nmeaning that the overall score for a tree is the sum of the scores of each of the scores\nof the edges that comprise the tree.\nScore (t;S) =X\ne2tScore (e)\nGraph-based algorithms have to solve two problems: (1) assigning a score to\neach edge, and (2) \ufb01nding the best parse tree given the scores of all potential edges.\nIn the next few sections we\u2019ll introduce solutions to these two problems, beginning\nwith the second problem of \ufb01nding trees, and then giving a feature-based and a\nneural algorithm for solving the \ufb01rst problem of assigning scores.\n19.3.1 Parsing via \ufb01nding the maximum spanning tree\nIn graph-based parsing, given a sentence Swe start by creating a graph Gwhich is a\nfully-connected, weighted, directed graph where the vertices are the input words and\nthe directed edges represent all possible head-dependent assignments. We\u2019ll include\nan additional ROOT node with outgoing edges directed at all of the other vertices.\nThe weights of each edge in Gre\ufb02ect the score for each possible head-dependent\nrelation assigned by some scoring algorithm.\nIt turns out that \ufb01nding the best dependency parse for Sis equivalent to \ufb01nding\nthemaximum spanning tree over G. A spanning tree over a graph Gis a subsetmaximum\nspanning tree\nofGthat is a tree and covers all the vertices in G; a spanning tree over Gthat starts\nfrom the ROOT is a valid parse of S. A maximum spanning tree is the spanning tree\nwith the highest score. Thus a maximum spanning tree of Gemanating from the\nROOT is the optimal dependency parse for the sentence.\nA directed graph for the example Book that \ufb02ight is shown in Fig. 19.11, with the\nmaximum spanning tree corresponding to the desired parse shown in blue. For ease\nof exposition, we\u2019ll describe here the algorithm for unlabeled dependency parsing.\nrootBookthat\ufb02ight1244568757\nFigure 19.11 Initial rooted, directed graph for Book that \ufb02ight .\nBefore describing the algorithm it\u2019s useful to consider two intuitions about di-\nrected graphs and their spanning trees. The \ufb01rst intuition begins with the fact that\nevery vertex in a spanning tree has exactly one incoming edge. It follows from this\nthat every connected component of a spanning tree (i.e., every set of vertices that\nare linked to each other by paths over edges) will also have one incoming edge.\nThe second intuition is that the absolute values of the edge scores are not critical\nto determining its maximum spanning tree. Instead, it is the relative weights of the\nedges entering each vertex that matters. If we were to subtract a constant amount\nfrom each edge entering a given vertex it would have no impact on the choice of 19.3 \u2022 G RAPH -BASED DEPENDENCY PARSING 17\nthe maximum spanning tree since every possible spanning tree would decrease by\nexactly the same amount.\nThe \ufb01rst step of the algorithm itself is quite straightforward. For each vertex\nin the graph, an incoming edge (representing a possible head assignment) with the\nhighest score is chosen. If the resulting set of edges produces a spanning tree then\nwe\u2019re done. More formally, given the original fully-connected graph G= (V;E), a\nsubgraph T= (V;F)is a spanning tree if it has no cycles and each vertex (other than\nthe root) has exactly one edge entering it. If the greedy selection process produces\nsuch a tree then it is the best possible one.\nUnfortunately, this approach doesn\u2019t always lead to a tree since the set of edges\nselected may contain cycles. Fortunately, in yet another case of multiple discovery,\nthere is a straightforward way to eliminate cycles generated during the greedy se-\nlection phase. Chu and Liu (1965) and Edmonds (1967) independently developed\nan approach that begins with greedy selection and follows with an elegant recursive\ncleanup phase that eliminates cycles.\nThe cleanup phase begins by adjusting all the weights in the graph by subtracting\nthe score of the maximum edge entering each vertex from the score of all the edges\nentering that vertex. This is where the intuitions mentioned earlier come into play.\nWe have scaled the values of the edges so that the weights of the edges in the cycle\nhave no bearing on the weight of anyof the possible spanning trees. Subtracting the\nvalue of the edge with maximum weight from each edge entering a vertex results\nin a weight of zero for all of the edges selected during the greedy selection phase,\nincluding all of the edges involved in the cycle .\nHaving adjusted the weights, the algorithm creates a new graph by selecting a\ncycle and collapsing it into a single new node. Edges that enter or leave the cycle\nare altered so that they now enter or leave the newly collapsed node. Edges that do\nnot touch the cycle are included and edges within the cycle are dropped.\nNow, if we knew the maximum spanning tree of this new graph, we would have\nwhat we need to eliminate the cycle. The edge of the maximum spanning tree di-\nrected towards the vertex representing the collapsed cycle tells us which edge to\ndelete in order to eliminate the cycle. How do we \ufb01nd the maximum spanning tree\nof this new graph? We recursively apply the algorithm to the new graph. This will\neither result in a spanning tree or a graph with a cycle. The recursions can continue\nas long as cycles are encountered. When each recursion completes we expand the\ncollapsed vertex, restoring all the vertices and edges from the cycle with the excep-\ntion of the single edge to be deleted .\nPutting all this together, the maximum spanning tree algorithm consists of greedy\nedge selection, re-scoring of edge costs and a recursive cleanup phase when needed.\nThe full algorithm is shown in Fig. 19.12.\nFig. 19.13 steps through the algorithm with our Book that \ufb02ight example. The\n\ufb01rst row of the \ufb01gure illustrates greedy edge selection with the edges chosen shown\nin blue (corresponding to the set Fin the algorithm). This results in a cycle between\nthatand\ufb02ight . The scaled weights using the maximum value entering each node are\nshown in the graph to the right.\nCollapsing the cycle between that and\ufb02ight to a single node (labelled tf) and\nrecursing with the newly scaled costs is shown in the second row. The greedy selec-\ntion step in this recursion yields a spanning tree that links roottobook , as well as an\nedge that links book to the contracted node. Expanding the contracted node, we can\nsee that this edge corresponds to the edge from book to\ufb02ight in the original graph.\nThis in turn tells us which edge to drop to eliminate the cycle. 18 CHAPTER 19 \u2022 D EPENDENCY PARSING\nfunction MAXSPANNING TREE(G=(V ,E) ,root,score )returns spanning tree\nF []\nT\u2019 []\nscore\u2019 []\nfor each v2Vdo\nbestInEdge argmaxe=(u;v)2Escore[e]\nF F[bestInEdge\nfor each e=(u,v)2Edo\nscore\u2019[e] score[e]\u0000score[bestInEdge]\nifT=(V ,F) is a spanning tree then return it\nelse\nC a cycle in F\nG\u2019 CONTRACT (G,C)\nT\u2019 MAXSPANNING TREE(G\u2019,root,score\u2019 )\nT EXPAND (T\u2019,C)\nreturn T\nfunction CONTRACT (G,C)returns contracted graph\nfunction EXPAND (T,C)returns expanded graph\nFigure 19.12 The Chu-Liu Edmonds algorithm for \ufb01nding a maximum spanning tree in a\nweighted directed graph.\nOn arbitrary directed graphs, this version of the CLE algorithm runs in O(mn)\ntime, where mis the number of edges and nis the number of nodes. Since this par-\nticular application of the algorithm begins by constructing a fully connected graph\nm=n2yielding a running time of O(n3). Gabow et al. (1986) present a more ef\ufb01-\ncient implementation with a running time of O(m+nlogn ).\n19.3.2 A feature-based algorithm for assigning scores\nRecall that given a sentence, S, and a candidate tree, T, edge-factored parsing models\nmake the simpli\ufb01cation that the score for the tree is the sum of the scores of the edges\nthat comprise the tree:\nscore(S;T) =X\ne2Tscore(S;e)\nIn a feature-based algorithm we compute the edge score as a weighted sum of fea-\ntures extracted from it:\nscore(S;e) =NX\ni=1wifi(S;e)\nOr more succinctly.\nscore(S;e) = w\u0001f\nGiven this formulation, we need to identify relevant features and train the weights.\nThe features (and feature combinations) used to train edge-factored models mir-\nror those used in training transition-based parsers, such as 19.3 \u2022 G RAPH -BASED DEPENDENCY PARSING 19\nrootBooktf\nrootBookthat\ufb02ight0-3-4\n-7-1-6-2rootBook12that7\ufb02ight8-4-30-2-6-1-700\nrootBook0tf-10-3-4\n-7-1-6-2rootBook12that7\ufb02ight81244568757\nDeleted from cycle\nFigure 19.13 Chu-Liu-Edmonds graph-based example for Book that \ufb02ight\n\u2022 Wordforms, lemmas, and parts of speech of the headword and its dependent.\n\u2022 Corresponding features from the contexts before, after and between the words.\n\u2022 Word embeddings.\n\u2022 The dependency relation itself.\n\u2022 The direction of the relation (to the right or left).\n\u2022 The distance from the head to the dependent.\nGiven a set of features, our next problem is to learn a set of weights correspond-\ning to each. Unlike many of the learning problems discussed in earlier chapters,\nhere we are not training a model to associate training items with class labels, or\nparser actions. Instead, we seek to train a model that assigns higher scores to cor-\nrect trees than to incorrect ones. An effective framework for problems like this is to\nuseinference-based learning combined with the perceptron learning rule. In thisinference-based\nlearning\nframework, we parse a sentence (i.e, perform inference) from the training set using\nsome initially random set of initial weights. If the resulting parse matches the cor-\nresponding tree in the training data, we do nothing to the weights. Otherwise, we\n\ufb01nd those features in the incorrect parse that are notpresent in the reference parse\nand we lower their weights by a small amount based on the learning rate. We do this\nincrementally for each sentence in our training data until the weights converge. 20 CHAPTER 19 \u2022 D EPENDENCY PARSING\n19.3.3 A neural algorithm for assigning scores\nState-of-the-art graph-based multilingual parsers are based on neural networks. In-\nstead of extracting hand-designed features to represent each edge between words wi\nandwj, these parsers run the sentence through an encoder, and then pass the encoded\nrepresentation of the two words wiandwjthrough a network that estimates a score\nfor the edge i!j.\nbookthat\ufb02ightr1score(h1head, h3dep)Bia\ufb03neb\nENCODERUh1 headFFNheadFFNheadFFNdepFFNdeph1 depFFNheadFFNdeph2 headh2 deph3 headh3 depWr2r3\u2211+\nFigure 19.14 Computing scores for a single edge (book !\ufb02ight) in the biaf\ufb01ne parser of\nDozat and Manning (2017); Dozat et al. (2017). The parser uses distinct feedforward net-\nworks to turn the encoder output for each word into a head and dependent representation for\nthe word. The biaf\ufb01ne function turns the head embedding of the head and the dependent\nembedding of the dependent into a score for the dependency edge.\nHere we\u2019ll sketch the biaf\ufb01ne algorithm of Dozat and Manning (2017) and Dozat\net al. (2017) shown in Fig. 19.14, drawing on the work of Gr \u00a8unewald et al. (2021)\nwho tested many versions of the algorithm via their STEPS system. The algorithm\n\ufb01rst runs the sentence X=x1;:::;xnthrough an encoder to produce a contextual\nembedding representation for each token R=r1;:::;rn. The embedding for each\ntoken is now passed through two separate feedforward networks, one to produce a\nrepresentation of this token as a head, and one to produce a representation of this\ntoken as a dependent:\nhhead\ni=FFNhead(ri) (19.13)\nhdep\ni=FFNdep(ri) (19.14)\nNow to assign a score to the directed edge i!j, (wiis the head and wjis the depen-\ndent), we feed the head representation of i,hhead\ni, and the dependent representation\nofj,hdep\nj, into a biaf\ufb01ne scoring function:\nScore (i!j) = Biaff(hhead\ni;hdep\nj) (19.15)\nBiaff(x;y) = x|Uy+W(x\by)+b (19.16) 19.4 \u2022 E VALUATION 21\nwhere U,W, and bare weights learned by the model. The idea of using a biaf\ufb01ne\nfunction is to allow the system to learn multiplicative interactions between the vec-\ntorsxandy.\nIf we pass Score (i!j)through a softmax, we end up with a probability distri-\nbution, for each token j, over potential heads i(all other tokens in the sentence):\np(i!j) =softmax ([Score (k!j);8k6=j;1\u0014k\u0014n]) (19.17)\nThis probability can then be passed to the maximum spanning tree algorithm of\nSection 19.3.1 to \ufb01nd the best tree.\nThis p(i!j)classi\ufb01er is trained by optimizing the cross-entropy loss.\nNote that the algorithm as we\u2019ve described it is unlabeled. To make this into\na labeled algorithm, the Dozat and Manning (2017) algorithm actually trains two\nclassi\ufb01ers. The \ufb01rst classi\ufb01er, the edge-scorer , the one we described above, assigns\na probability p(i!j)to each word wiandwj. Then the Maximum Spanning Tree\nalgorithm is run to get a single best dependency parse tree for the second. We then\napply a second classi\ufb01er, the label-scorer , whose job is to \ufb01nd the maximum prob-\nability label for each edge in this parse. This second classi\ufb01er has the same form\nas (19.15-19.17), but instead of being trained to predict with binary softmax the\nprobability of an edge existing between two words, it is trained with a softmax over\ndependency labels to predict the dependency label between the words.\n19.4 Evaluation\nAs with phrase structure-based parsing, the evaluation of dependency parsers pro-\nceeds by measuring how well they work on a test set. An obvious metric would be\nexact match (EM)\u2014how many sentences are parsed correctly. This metric is quite\npessimistic, with most sentences being marked wrong. Such measures are not \ufb01ne-\ngrained enough to guide the development process. Our metrics need to be sensitive\nenough to tell if actual improvements are being made.\nFor these reasons, the most common method for evaluating dependency parsers\nare labeled and unlabeled attachment accuracy. Labeled attachment refers to the\nproper assignment of a word to its head along with the correct dependency relation.\nUnlabeled attachment simply looks at the correctness of the assigned head, ignor-\ning the dependency relation. Given a system output and a corresponding reference\nparse, accuracy is simply the percentage of words in an input that are assigned the\ncorrect head with the correct relation. These metrics are usually referred to as the\nlabeled attachment score (LAS) and unlabeled attachment score (UAS). Finally, we\ncan make use of a label accuracy score (LS), the percentage of tokens with correct\nlabels, ignoring where the relations are coming from.\nAs an example, consider the reference parse and system parse for the following\nexample shown in Fig. 19.15.\n(19.18) Book me the \ufb02ight through Houston.\nThe system correctly \ufb01nds 4 of the 6 dependency relations present in the reference\nparse and receives an LAS of 2/3. However, one of the 2 incorrect relations found\nby the system holds between book and\ufb02ight , which are in a head-dependent relation\nin the reference parse; the system therefore achieves a UAS of 5/6.\nBeyond attachment scores, we may also be interested in how well a system is\nperforming on a particular kind of dependency relation, for example NSUBJ , across 22 CHAPTER 19 \u2022 D EPENDENCY PARSING\nBook methe \ufb02ight through Houston\n(a) Referenceiobjobj\ndetnmod\ncaseroot\nBook methe \ufb02ight through Houston\n(b) Systemxcomp\nnsubj\ndetnmod\ncaseroot\nFigure 19.15 Reference and system parses for Book me the \ufb02ight through Houston , resulting in an LAS of\n2/3 and an UAS of 5/6.\na development corpus. Here we can make use of the notions of precision and recall\nintroduced in Chapter 17, measuring the percentage of relations labeled NSUBJ by\nthe system that were correct (precision), and the percentage of the NSUBJ relations\npresent in the development set that were in fact discovered by the system (recall).\nWe can employ a confusion matrix to keep track of how often each dependency type\nwas confused for another.\n19.5 Summary\nThis chapter has introduced the concept of dependency grammars and dependency\nparsing. Here\u2019s a summary of the main points that we covered:\n\u2022 In dependency-based approaches to syntax, the structure of a sentence is de-\nscribed in terms of a set of binary relations that hold between the words in a\nsentence. Larger notions of constituency are not directly encoded in depen-\ndency analyses.\n\u2022 The relations in a dependency structure capture the head-dependent relation-\nship among the words in a sentence.\n\u2022 Dependency-based analysis provides information directly useful in further\nlanguage processing tasks including information extraction, semantic parsing\nand question answering.\n\u2022 Transition-based parsing systems employ a greedy stack-based algorithm to\ncreate dependency structures.\n\u2022 Graph-based methods for creating dependency structures are based on the use\nof maximum spanning tree methods from graph theory.\n\u2022 Both transition-based and graph-based approaches are developed using super-\nvised machine learning techniques.\n\u2022 Treebanks provide the data needed to train these systems. Dependency tree-\nbanks can be created directly by human annotators or via automatic transfor-\nmation from phrase-structure treebanks.\n\u2022 Evaluation of dependency parsers is based on labeled and unlabeled accuracy\nscores as measured against withheld development and test corpora. BIBLIOGRAPHICAL AND HISTORICAL NOTES 23\nBibliographical and Historical Notes\nThe dependency-based approach to grammar is much older than the relatively recent\nphrase-structure or constituency grammars, which date only to the 20th century. De-\npendency grammar dates back to the Indian grammarian P \u00afan.ini sometime between\nthe 7th and 4th centuries BCE, as well as the ancient Greek linguistic traditions.\nContemporary theories of dependency grammar all draw heavily on the 20th cen-\ntury work of Tesni `ere (1959).\nAutomatic parsing using dependency grammars was \ufb01rst introduced into compu-\ntational linguistics by early work on machine translation at the RAND Corporation\nled by David Hays. This work on dependency parsing closely paralleled work on\nconstituent parsing and made explicit use of grammars to guide the parsing process.\nAfter this early period, computational work on dependency parsing remained inter-\nmittent over the following decades. Notable implementations of dependency parsers\nfor English during this period include Link Grammar (Sleator and Temperley, 1993),\nConstraint Grammar (Karlsson et al., 1995), and MINIPAR (Lin, 2003).\nDependency parsing saw a major resurgence in the late 1990\u2019s with the appear-\nance of large dependency-based treebanks and the associated advent of data driven\napproaches described in this chapter. Eisner (1996) developed an ef\ufb01cient dynamic\nprogramming approach to dependency parsing based on bilexical grammars derived\nfrom the Penn Treebank. Covington (2001) introduced the deterministic word by\nword approach underlying current transition-based approaches. Yamada and Mat-\nsumoto (2003) and Kudo and Matsumoto (2002) introduced both the shift-reduce\nparadigm and the use of supervised machine learning in the form of support vector\nmachines to dependency parsing.\nTransition-based parsing is based on the shift-reduce parsing algorithm orig-\ninally developed for analyzing programming languages (Aho and Ullman, 1972).\nShift-reduce parsing also makes use of a context-free grammar. Input tokens are\nsuccessively shifted onto the stack and the top two elements of the stack are matched\nagainst the right-hand side of the rules in the grammar; when a match is found the\nmatched elements are replaced on the stack (reduced) by the non-terminal from the\nleft-hand side of the rule being matched. In transition-based dependency parsing\nwe skip the grammar, and alter the reduce operation to add a dependency relation\nbetween a word and its head.\nNivre (2003) de\ufb01ned the modern, deterministic, transition-based approach to\ndependency parsing. Subsequent work by Nivre and his colleagues formalized and\nanalyzed the performance of numerous transition systems, training methods, and\nmethods for dealing with non-projective language (Nivre and Scholz 2004, Nivre\n2006, Nivre and Nilsson 2005, Nivre et al. 2007b, Nivre 2007). The neural ap-\nproach was pioneered by Chen and Manning (2014) and extended by Kiperwasser\nand Goldberg (2016); Kulmizev et al. (2019).\nThe graph-based maximum spanning tree approach to dependency parsing was\nintroduced by McDonald et al. 2005a, McDonald et al. 2005b. The neural classi\ufb01er\nwas introduced by (Kiperwasser and Goldberg, 2016).\nThe long-running Prague Dependency Treebank project (Haji \u02c7c, 1998) is the most\nsigni\ufb01cant effort to directly annotate a corpus with multiple layers of morphological,\nsyntactic and semantic information. PDT 3.0 contains over 1.5 M tokens (Bej \u02c7cek\net al., 2013).\nUniversal Dependencies (UD) (de Marneffe et al., 2021) is an open community 24 CHAPTER 19 \u2022 D EPENDENCY PARSING\nproject to create a framework for dependency treebank annotation, with nearly 200\ntreebanks in over 100 languages. The UD annotation scheme evolved out of several\ndistinct efforts including Stanford dependencies (de Marneffe et al. 2006, de Marn-\neffe and Manning 2008, de Marneffe et al. 2014), Google\u2019s universal part-of-speech\ntags (Petrov et al., 2012), and the Interset interlingua for morphosyntactic tagsets\n(Zeman, 2008).\nThe Conference on Natural Language Learning (CoNLL) has conducted an in-\n\ufb02uential series of shared tasks related to dependency parsing over the years (Buch-\nholz and Marsi 2006, Nivre et al. 2007a, Surdeanu et al. 2008, Haji \u02c7c et al. 2009).\nMore recent evaluations have focused on parser robustness with respect to morpho-\nlogically rich languages (Seddah et al., 2013), and non-canonical language forms\nsuch as social media, texts, and spoken language (Petrov and McDonald, 2012).\nChoi et al. (2015) presents a performance analysis of 10 dependency parsers across\na range of metrics, as well as DEPEND ABLE, a robust parser evaluation tool.\nExercises Exercises 25\nAho, A. V . and J. D. Ullman. 1972. The Theory of Parsing,\nTranslation, and Compiling , volume 1. Prentice Hall.\nBej\u02c7cek, E., E. Haji \u02c7cov\u00b4a, J. Haji \u02c7c, P. J \u00b4\u0131nov\u00b4a, V . Kettnerov \u00b4a,\nV . Kol \u00b4a\u02c7rov\u00b4a, M. Mikulov \u00b4a, J. M \u00b4\u0131rovsk \u00b4y, A. Nedoluzhko,\nJ. Panevov \u00b4a, L. Pol \u00b4akov \u00b4a, M. \u02c7Sev\u02c7c\u00b4\u0131kov\u00b4a, J. \u02c7St\u02c7ep\u00b4anek,\nand \u02c7S. Zik \u00b4anov \u00b4a. 2013. Prague dependency treebank\n3.0. Technical report, Institute of Formal and Ap-\nplied Linguistics, Charles University in Prague. LIN-\nDAT/CLARIN digital library at Institute of Formal and\nApplied Linguistics, Charles University in Prague.\nBhat, I., R. A. Bhat, M. Shrivastava, and D. Sharma. 2017.\nJoining hands: Exploiting monolingual treebanks for\nparsing of code-mixing data. EACL .\nBuchholz, S. and E. Marsi. 2006. Conll-x shared task on\nmultilingual dependency parsing. CoNLL .\nChen, D. and C. Manning. 2014. A fast and accurate depen-\ndency parser using neural networks. EMNLP .\nChoi, J. D. and M. Palmer. 2011a. Getting the most out of\ntransition-based dependency parsing. ACL.\nChoi, J. D. and M. Palmer. 2011b. Transition-based semantic\nrole labeling using predicate argument clustering. Pro-\nceedings of the ACL 2011 Workshop on Relational Mod-\nels of Semantics .\nChoi, J. D., J. Tetreault, and A. Stent. 2015. It depends:\nDependency parser comparison using a web-based evalu-\nation tool. ACL.\nChu, Y .-J. and T.-H. Liu. 1965. On the shortest arborescence\nof a directed graph. Science Sinica , 14:1396\u20131400.\nCovington, M. 2001. A fundamental algorithm for depen-\ndency parsing. Proceedings of the 39th Annual ACM\nSoutheast Conference .\nDozat, T. and C. D. Manning. 2017. Deep biaf\ufb01ne attention\nfor neural dependency parsing. ICLR .\nDozat, T. and C. D. Manning. 2018. Simpler but more accu-\nrate semantic dependency parsing. ACL.\nDozat, T., P. Qi, and C. D. Manning. 2017. Stanford\u2019s\ngraph-based neural dependency parser at the CoNLL\n2017 shared task. Proceedings of the CoNLL 2017 Shared\nTask: Multilingual Parsing from Raw Text to Universal\nDependencies .\nEdmonds, J. 1967. Optimum branchings. Journal of Re-\nsearch of the National Bureau of Standards B , 71(4):233\u2013\n240.\nEisner, J. 1996. Three new probabilistic models for depen-\ndency parsing: An exploration. COLING .\nGabow, H. N., Z. Galil, T. Spencer, and R. E. Tarjan.\n1986. Ef\ufb01cient algorithms for \ufb01nding minimum spanning\ntrees in undirected and directed graphs. Combinatorica ,\n6(2):109\u2013122.\nGr\u00a8unewald, S., A. Friedrich, and J. Kuhn. 2021. Applying\nOccam\u2019s razor to transformer-based dependency parsing:\nWhat works, what doesn\u2019t, and what is really necessary.\nIWPT .\nHaji\u02c7c, J. 1998. Building a Syntactically Annotated Corpus:\nThe Prague Dependency Treebank , 106\u2013132. Karolinum.Haji\u02c7c, J., M. Ciaramita, R. Johansson, D. Kawahara, M. A.\nMart \u00b4\u0131, L. M `arquez, A. Meyers, J. Nivre, S. Pad \u00b4o,\nJ.\u02c7St\u02c7ep\u00b4anek, P. Stran \u02c7a\u00b4k, M. Surdeanu, N. Xue, and\nY . Zhang. 2009. The conll-2009 shared task: Syntac-\ntic and semantic dependencies in multiple languages.\nCoNLL .\nKarlsson, F., A. V outilainen, J. Heikkil \u00a8a, and A. Anttila, eds.\n1995. Constraint Grammar: A Language-Independent\nSystem for Parsing Unrestricted Text . Mouton de Gruyter.\nKiperwasser, E. and Y . Goldberg. 2016. Simple and accu-\nrate dependency parsing using bidirectional LSTM fea-\nture representations. TACL , 4:313\u2013327.\nKudo, T. and Y . Matsumoto. 2002. Japanese dependency\nanalysis using cascaded chunking. CoNLL .\nKulmizev, A., M. de Lhoneux, J. Gontrum, E. Fano, and\nJ. Nivre. 2019. Deep contextualized word embeddings\nin transition-based and graph-based dependency parsing\n- a tale of two parsers revisited. EMNLP .\nLin, D. 2003. Dependency-based evaluation of minipar.\nWorkshop on the Evaluation of Parsing Systems .\nde Marneffe, M.-C., T. Dozat, N. Silveira, K. Haverinen,\nF. Ginter, J. Nivre, and C. D. Manning. 2014. Univer-\nsal Stanford dependencies: A cross-linguistic typology.\nLREC .\nde Marneffe, M.-C., B. MacCartney, and C. D. Manning.\n2006. Generating typed dependency parses from phrase\nstructure parses. LREC .\nde Marneffe, M.-C. and C. D. Manning. 2008. The Stanford\ntyped dependencies representation. COLING Workshop\non Cross-Framework and Cross-Domain Parser Evalua-\ntion.\nde Marneffe, M.-C., C. D. Manning, J. Nivre, and D. Zeman.\n2021. Universal Dependencies. Computational Linguis-\ntics, 47(2):255\u2013308.\nMcDonald, R., K. Crammer, and F. C. N. Pereira. 2005a. On-\nline large-margin training of dependency parsers. ACL.\nMcDonald, R. and J. Nivre. 2011. Analyzing and inte-\ngrating dependency parsers. Computational Linguistics ,\n37(1):197\u2013230.\nMcDonald, R., F. C. N. Pereira, K. Ribarov, and J. Haji \u02c7c.\n2005b. Non-projective dependency parsing using span-\nning tree algorithms. HLT-EMNLP .\nNivre, J. 2007. Incremental non-projective dependency pars-\ning. NAACL-HLT .\nNivre, J. 2003. An ef\ufb01cient algorithm for projective depen-\ndency parsing. Proceedings of the 8th International Work-\nshop on Parsing Technologies (IWPT) .\nNivre, J. 2006. Inductive Dependency Parsing . Springer.\nNivre, J. 2009. Non-projective dependency parsing in ex-\npected linear time. ACL IJCNLP .\nNivre, J., J. Hall, S. K \u00a8ubler, R. McDonald, J. Nilsson,\nS. Riedel, and D. Yuret. 2007a. The conll 2007 shared\ntask on dependency parsing. EMNLP/CoNLL .\nNivre, J., J. Hall, J. Nilsson, A. Chanev, G. Eryigit,\nS. K \u00a8ubler, S. Marinov, and E. Marsi. 2007b. Malt-\nparser: A language-independent system for data-driven\ndependency parsing. Natural Language Engineering ,\n13(02):95\u2013135. 26 Chapter 19 \u2022 Dependency Parsing\nNivre, J. and J. Nilsson. 2005. Pseudo-projective dependency\nparsing. ACL.\nNivre, J. and M. Scholz. 2004. Deterministic dependency\nparsing of english text. COLING .\nPetrov, S., D. Das, and R. McDonald. 2012. A universal\npart-of-speech tagset. LREC .\nPetrov, S. and R. McDonald. 2012. Overview of the 2012\nshared task on parsing the web. Notes of the First Work-\nshop on Syntactic Analysis of Non-Canonical Language\n(SANCL) , volume 59.\nSeddah, D., R. Tsarfaty, S. K \u00a8ubler, M. Candito, J. D. Choi,\nR. Farkas, J. Foster, I. Goenaga, K. Gojenola, Y . Gold-\nberg, S. Green, N. Habash, M. Kuhlmann, W. Maier,\nJ. Nivre, A. Przepi \u00b4orkowski, R. Roth, W. Seeker, Y . Vers-\nley, V . Vincze, M. Woli \u00b4nski, A. Wr \u00b4oblewska, and E. Ville-\nmonte de la Cl \u00b4ergerie. 2013. Overview of the SPMRL\n2013 shared task: cross-framework evaluation of parsing\nmorphologically rich languages. 4th Workshop on Statis-\ntical Parsing of Morphologically-Rich Languages .\nSleator, D. and D. Temperley. 1993. Parsing English with a\nlink grammar. IWPT-93 .\nSurdeanu, M., R. Johansson, A. Meyers, L. M `arquez, and\nJ. Nivre. 2008. The CoNLL 2008 shared task on joint\nparsing of syntactic and semantic dependencies. CoNLL .\nTesni `ere, L. 1959. \u00b4El\u00b4ements de Syntaxe Structurale . Librairie\nC. Klincksieck, Paris.\nYamada, H. and Y . Matsumoto. 2003. Statistical dependency\nanalysis with support vector machines. IWPT-03 .\nZeman, D. 2008. Reusable tagset conversion using tagset\ndrivers. LREC .", "20.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\n20Information Extraction:\nRelations, Events, and Time\nTime will explain.\nJane Austen, Persuasion\nImagine that you are an analyst with an investment \ufb01rm that tracks airline stocks.\nYou\u2019re given the task of determining the relationship (if any) between airline an-\nnouncements of fare increases and the behavior of their stocks the next day. His-\ntorical data about stock prices is easy to come by, but what about the airline an-\nnouncements? You will need to know at least the name of the airline, the nature of\nthe proposed fare hike, the dates of the announcement, and possibly the response of\nother airlines. Fortunately, these can be all found in news articles like this one:\nCiting high fuel prices, United Airlines said Friday it has increased fares\nby $6 per round trip on \ufb02ights to some cities also served by lower-\ncost carriers. American Airlines, a unit of AMR Corp., immediately\nmatched the move, spokesman Tim Wagner said. United, a unit of UAL\nCorp., said the increase took effect Thursday and applies to most routes\nwhere it competes against discount carriers, such as Chicago to Dallas\nand Denver to San Francisco.\nThis chapter presents techniques for extracting limited kinds of semantic con-\ntent from text. This process of information extraction (IE) turns the unstructuredinformation\nextraction\ninformation embedded in texts into structured data, for example for populating a\nrelational database to enable further processing.\nWe begin with the task of relation extraction : \ufb01nding and classifying semanticrelation\nextraction\nrelations among entities mentioned in a text, like child-of (X is the child-of Y), or\npart-whole or geospatial relations. Relation extraction has close links to populat-\ning a relational database, and knowledge graphs , datasets of structured relationalknowledge\ngraphs\nknowledge, are a useful way for search engines to present information to users.\nNext, we discuss event extraction , the task of \ufb01nding events in which these en-event\nextraction\ntities participate, like, in our sample text, the fare increases by United andAmerican\nand the reporting events said andcite. Events are also situated in time , occurring at\na particular date or time, and events can be related temporally, happening before or\nafter or simultaneously with each other. We\u2019ll need to recognize temporal expres-\nsions like Friday ,Thursday ortwo days from now and times such as 3:30 P .M. , and\nnormalize them onto speci\ufb01c calendar dates or times. We\u2019ll need to link Friday to\nthe time of United\u2019s announcement, Thursday to the previous day\u2019s fare increase,\nand we\u2019ll need to produce a timeline in which United\u2019s announcement follows the\nfare increase and American\u2019s announcement follows both of those events.\nThe related task of template \ufb01lling is to \ufb01nd recurring stereotypical events or template \ufb01lling\nsituations in documents and \ufb01ll in the template slots. These slot-\ufb01llers may consist\nof text segments extracted directly from the text, or concepts like times, amounts, or\nontology entities that have been inferred through additional processing. Our airline 2CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\nARTIFACTGENERALAFFILIATIONORGAFFILIATIONPART-WHOLEPERSON-SOCIALPHYSICALLocatedNearBusinessFamilyLasting PersonalCitizen-Resident-Ethnicity-ReligionOrg-Location-OriginFounderEmploymentMembershipOwnershipStudent-AlumInvestorUser-Owner-Inventor-ManufacturerGeographicalSubsidiary\nSports-Affiliation\nFigure 20.1 The 17 relations used in the ACE relation extraction task.\ntext presents such a stereotypical situation since airlines often raise fares and then\nwait to see if competitors follow along. Here we can identify United as a lead air-\nline that initially raised its fares, $6 as the amount, Thursday as the increase date,\nandAmerican as an airline that followed along, leading to a \ufb01lled template like the\nfollowing:\nFARE-RAISE ATTEMPT :2\n6664LEAD AIRLINE : U NITED AIRLINES\nAMOUNT : $6\nEFFECTIVE DATE: 2006-10-26\nFOLLOWER : A MERICAN AIRLINES3\n7775\n20.1 Relation Extraction\nLet\u2019s assume that we have detected the named entities in our sample text (perhaps\nusing the techniques of Chapter 17), and would like to discern the relationships that\nexist among the detected entities:\nCiting high fuel prices, [ ORG United Airlines ] said [ TIME Friday ] it\nhas increased fares by [ MONEY $6] per round trip on \ufb02ights to some\ncities also served by lower-cost carriers. [ ORG American Airlines ], a\nunit of [ ORG AMR Corp.] , immediately matched the move, spokesman\n[PER Tim Wagner ] said. [ ORG United] , a unit of [ ORG UAL Corp.] ,\nsaid the increase took effect [ TIME Thursday] and applies to most\nroutes where it competes against discount carriers, such as [ LOC Chicago]\nto [LOC Dallas] and [ LOC Denver] to [LOC San Francisco] .\nThe text tells us, for example, that Tim Wagner is a spokesman for American\nAirlines , that United is a unit of UAL Corp. , and that American is a unit of AMR .\nThese binary relations are instances of more generic relations such as part-of or\nemploys that are fairly frequent in news-style texts. Figure 20.1 lists the 17 relations\nused in the ACE relation extraction evaluations and Fig. 20.2 shows some sample\nrelations. We might also extract more domain-speci\ufb01c relation such as the notion of\nan airline route. For example from this text we can conclude that United has routes\nto Chicago, Dallas, Denver, and San Francisco. 20.1 \u2022 R ELATION EXTRACTION 3\nRelations Types Examples\nPhysical-Located PER-GPE Hewas in Tennessee\nPart-Whole-Subsidiary ORG-ORG XYZ , the parent company of ABC\nPerson-Social-Family PER-PER Yoko \u2019s husband John\nOrg-AFF-Founder PER-ORG Steve Jobs , co-founder of Apple ...\nFigure 20.2 Semantic relations with examples and the named entity types they involve.\nSets of relations have been de\ufb01ned for many other domains as well. For example\nUMLS, the Uni\ufb01ed Medical Language System from the US National Library of\nMedicine has a network that de\ufb01nes 134 broad subject categories, entity types, and\n54 relations between the entities, such as the following:\nEntity Relation Entity\nInjury disrupts Physiological Function\nBodily Location location-of Biologic Function\nAnatomical Structure part-of Organism\nPharmacologic Substance causes Pathological Function\nPharmacologic Substance treats Pathologic Function\nGiven a medical sentence like this one:\n(20.1) Doppler echocardiography can be used to diagnose left anterior descending\nartery stenosis in patients with type 2 diabetes\nWe could thus extract the UMLS relation:\nEchocardiography, Doppler Diagnoses Acquired stenosis\nWikipedia also offers a large supply of relations, drawn from infoboxes , struc- infoboxes\ntured tables associated with certain Wikipedia articles. For example, the Wikipedia\ninfobox for Stanford includes structured facts like state = \"California\" or\npresident = \"Marc Tessier-Lavigne\" . These facts can be turned into rela-\ntions like president-of orlocated-in . or into relations in a metalanguage called RDF RDF\n(Resource Description Framework). An RDF triple is a tuple of entity-relation- RDF triple\nentity, called a subject-predicate-object expression. Here\u2019s a sample RDF triple:\nsubject predicate object\nGolden Gate Park location San Francisco\nFor example the crowdsourced DBpedia (Bizer et al., 2009) is an ontology de-\nrived from Wikipedia containing over 2 billion RDF triples. Another dataset from\nWikipedia infoboxes, Freebase (Bollacker et al., 2008), now part of Wikidata (Vrande \u02c7ci\u00b4c Freebase\nand Kr \u00a8otzsch, 2014), has relations between people and their nationality, or locations,\nand other locations they are contained in.\nWordNet or other ontologies offer useful ontological relations that express hier-\narchical relations between words or concepts. For example WordNet has the is-aor is-a\nhypernym relation between classes, hypernym\nGiraffe is-a ruminant is-a ungulate is-a mammal is-a vertebrate ...\nWordNet also has Instance-of relation between individuals and classes, so that for\nexample San Francisco is in the Instance-of relation with city. Extracting these\nrelations is an important step in extending or building ontologies.\nFinally, there are large datasets that contain sentences hand-labeled with their\nrelations, designed for training and testing relation extractors. The TACRED dataset\n(Zhang et al., 2017) contains 106,264 examples of relation triples about particular\npeople or organizations, labeled in sentences from news and web text drawn from the 4CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\nannual TAC Knowledge Base Population (TAC KBP) challenges. TACRED contains\n41 relation types (like per:city of birth, org:subsidiaries, org:member of, per:spouse),\nplus a no relation tag; examples are shown in Fig. 20.3. About 80% of all examples\nare annotated as no relation; having suf\ufb01cient negative data is important for training\nsupervised classi\ufb01ers.\nExample Entity Types & Label\nCarey will succeed Cathleen P. Black, who held the position for 15\nyears and will take on a new role as chairwoman of Hearst Maga-\nzines, the company said.PERSON /TITLE\nRelation: per:title\nIrene Morgan Kirkaldy, who was born and reared in Baltimore, lived\non Long Island and ran a child-care center in Queens with her second\nhusband, Stanley Kirkaldy.PERSON /CITY\nRelation: per:city ofbirth\nBaldwin declined further comment, and said JetBlue chief executive\nDave Barger was unavailable.Types: PERSON /TITLE\nRelation: norelation\nFigure 20.3 Example sentences and labels from the TACRED dataset (Zhang et al., 2017).\nA standard dataset was also produced for the SemEval 2010 Task 8, detecting\nrelations between nominals (Hendrickx et al., 2009). The dataset has 10,717 exam-\nples, each with a pair of nominals (untyped) hand-labeled with one of 9 directed\nrelations like product-producer ( afactory manufactures suits) orcomponent-whole\n(myapartment has a large kitchen ).\n20.2 Relation Extraction Algorithms\nThere are \ufb01ve main classes of algorithms for relation extraction: handwritten pat-\nterns ,supervised machine learning ,semi-supervised (viabootstrapping ordis-\ntant supervision ), and unsupervised . We\u2019ll introduce each of these in the next\nsections.\n20.2.1 Using Patterns to Extract Relations\nThe earliest and still common algorithm for relation extraction is lexico-syntactic\npatterns, \ufb01rst developed by Hearst (1992a), and therefore often called Hearst pat-\nterns . Consider the following sentence: Hearst patterns\nAgar is a substance prepared from a mixture of red algae, such as Ge-\nlidium, for laboratory or industrial use.\nHearst points out that most human readers will not know what Gelidium is, but that\nthey can readily infer that it is a kind of (a hyponym of)red algae , whatever that is.\nShe suggests that the following lexico-syntactic pattern\nNP0such as NP 1f;NP2:::;(andjor)NPig;i\u00151 (20.2)\nimplies the following semantics\n8NPi;i\u00151;hyponym (NPi;NP0) (20.3)\nallowing us to infer\nhyponym (Gelidium ;red algae ) (20.4) 20.2 \u2022 R ELATION EXTRACTION ALGORITHMS 5\nNPf, NPg*f,g(andjor) other NP H temples, treasuries, and other important civic buildings\nNPHsuch asfNP,g*f(orjand)gNP red algae such as Gelidium\nsuch NP HasfNP,g*f(orjand)gNP such authors as Herrick, Goldsmith, and Shakespeare\nNPHf,gincludingfNP,g*f(orjand)gNP common-law countries, including Canada and England\nNPHf,gespeciallyfNPg*f(orjand)gNP European countries, especially France, England, and Spain\nFigure 20.4 Hand-built lexico-syntactic patterns for \ufb01nding hypernyms, using fgto mark optionality (Hearst\n1992a, Hearst 1998).\nFigure 20.4 shows \ufb01ve patterns Hearst (1992a, 1998) suggested for inferring\nthe hyponym relation; we\u2019ve shown NPHas the parent/hyponym. Modern versions\nof the pattern-based approach extend it by adding named entity constraints. For\nexample if our goal is to answer questions about \u201cWho holds what of\ufb01ce in which\norganization?\u201d, we can use patterns like the following:\nPER, POSITION of ORG:\nGeorge Marshall, Secretary of State of the United States\nPER (namedjappointedjchosejetc.) PER Prep? POSITION\nTruman appointed Marshall Secretary of State\nPER [be]? (namedjappointedjetc.) Prep? ORG POSITION\nGeorge Marshall was named US Secretary of State\nHand-built patterns have the advantage of high-precision and they can be tailored\nto speci\ufb01c domains. On the other hand, they are often low-recall, and it\u2019s a lot of\nwork to create them for all possible patterns.\n20.2.2 Relation Extraction via Supervised Learning\nSupervised machine learning approaches to relation extraction follow a scheme that\nshould be familiar by now. A \ufb01xed set of relations and entities is chosen, a training\ncorpus is hand-annotated with the relations and entities, and the annotated texts are\nthen used to train classi\ufb01ers to annotate an unseen test set.\nThe most straightforward approach, illustrated in Fig. 20.5 is: (1) Find pairs of\nnamed entities (usually in the same sentence). (2): Apply a relation-classi\ufb01cation\non each pair. The classi\ufb01er can use any supervised technique (logistic regression,\nRNN, Transformer, random forest, etc.).\nAn optional intermediate \ufb01ltering classi\ufb01er can be used to speed up the process-\ning by making a binary decision on whether a given pair of named entities are related\n(by any relation). It\u2019s trained on positive examples extracted directly from all rela-\ntions in the annotated corpus, and negative examples generated from within-sentence\nentity pairs that are not annotated with a relation.\nFeature-based supervised relation classi\ufb01ers. Let\u2019s consider sample features for\na feature-based classi\ufb01er (like logistic regression or random forests), classifying the\nrelationship between American Airlines (Mention 1, or M1) and Tim Wagner (Men-\ntion 2, M2) from this sentence:\n(20.5) American Airlines , a unit of AMR, immediately matched the move,\nspokesman Tim Wagner said\nThese include word features (as embeddings, or 1-hot, stemmed or not):\n\u2022 The headwords of M1 and M2 and their concatenation\nAirlines Wagner Airlines-Wagner 6CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\nfunction FINDRELATIONS (words )returns relations\nrelations nil\nentities FINDENTITIES (words )\nforall entity pairs he1,e2iinentities do\nifRELATED ?(e1,e2)\nrelations relations +CLASSIFY RELATION (e1,e2)\nFigure 20.5 Finding and classifying the relations among entities in a text.\n\u2022 Bag-of-words and bigrams in M1 and M2\nAmerican, Airlines, Tim, Wagner, American Airlines, Tim Wagner\n\u2022 Words or bigrams in particular positions\nM2: -1 spokesman\nM2: +1 said\n\u2022 Bag of words or bigrams between M1 and M2:\na, AMR, of, immediately, matched, move, spokesman, the, unit\nNamed entity features:\n\u2022 Named-entity types and their concatenation\n(M1: ORG, M2: PER, M1M2: ORG-PER)\n\u2022 Entity Level of M1 and M2 (from the set NAME, NOMINAL, PRONOUN)\nM1: NAME [it or he would be PRONOUN]\nM2: NAME [the company would be NOMINAL]\n\u2022 Number of entities between the arguments (in this case 1, for AMR)\nSyntactic structure is a useful signal, often represented as the dependency or\nconstituency syntactic path traversed through the tree between the entities.\n\u2022 Constituent paths between M1 and M2\nNP\"NP\"S\"S#NP\n\u2022 Dependency-tree paths\nAirlines sub jmatched comp said!sub jWagner\nNeural supervised relation classi\ufb01ers Neural models for relation extraction sim-\nilarly treat the task as supervised classi\ufb01cation. Let\u2019s consider a typical system ap-\nplied to the TACRED relation extraction dataset and task (Zhang et al., 2017). In\nTACRED we are given a sentence and two spans within it: a subject, which is a\nperson or organization, and an object, which is any other entity. The task is to assign\na relation from the 42 TAC relations, or no relation.\nA typical Transformer-encoder algorithm, shown in Fig. 20.6, simply takes a\npretrained encoder like BERT and adds a linear layer on top of the sentence repre-\nsentation (for example the BERT [CLS] token), a linear layer that is \ufb01netuned as a\n1-of-N classi\ufb01er to assign one of the 43 labels. The input to the BERT encoder is\npartially de-lexi\ufb01ed; the subject and object entities are replaced in the input by their\nNER tags. This helps keep the system from over\ufb01tting to the individual lexical items\n(Zhang et al., 2017). When using BERT-type Transformers for relation extraction, it\nhelps to use versions of BERT like RoBERTa (Liu et al., 2019) or spanBERT (Joshi\net al., 2020) that don\u2019t have two sequences separated by a [SEP] token, but instead\nform the input from a single long sequence of sentences.\nIn general, if the test set is similar enough to the training set, and if there is\nenough hand-labeled data, supervised relation extraction systems can get high ac- 20.2 \u2022 R ELATION EXTRACTION ALGORITHMS 7\nENCODER[CLS][SUBJ_PERSON]wasbornin[OBJ_LOC],MichiganLinearClassi\ufb01erp(relation|SUBJ,OBJ)\nFigure 20.6 Relation extraction as a linear layer on top of an encoder (in this case BERT),\nwith the subject and object entities replaced in the input by their NER tags (Zhang et al. 2017,\nJoshi et al. 2020).\ncuracies. But labeling a large training set is extremely expensive and supervised\nmodels are brittle: they don\u2019t generalize well to different text genres. For this rea-\nson, much research in relation extraction has focused on the semi-supervised and\nunsupervised approaches we turn to next.\n20.2.3 Semisupervised Relation Extraction via Bootstrapping\nSupervised machine learning assumes that we have lots of labeled data. Unfortu-\nnately, this is expensive. But suppose we just have a few high-precision seed pat-\nterns , like those in Section 20.2.1, or perhaps a few seed tuples . That\u2019s enough seed patterns\nseed tuples to bootstrap a classi\ufb01er! Bootstrapping proceeds by taking the entities in the seed\nbootstrapping pair, and then \ufb01nding sentences (on the web, or whatever dataset we are using) that\ncontain both entities. From all such sentences, we extract and generalize the context\naround the entities to learn new patterns. Fig. 20.7 sketches a basic algorithm.\nfunction BOOTSTRAP (Relation R )returns new relation tuples\ntuples Gather a set of seed tuples that have relation R\niterate\nsentences \ufb01nd sentences that contain entities in tuples\npatterns generalize the context between and around entities in sentences\nnewpairs usepatterns to identify more tuples\nnewpairs newpairs with high con\ufb01dence\ntuples tuples +newpairs\nreturn tuples\nFigure 20.7 Bootstrapping from seed entity pairs to learn relations.\nSuppose, for example, that we need to create a list of airline/hub pairs, and we\nknow only that Ryanair has a hub at Charleroi. We can use this seed fact to discover\nnew patterns by \ufb01nding other mentions of this relation in our corpus. We search\nfor the terms Ryanair ,Charleroi andhubin some proximity. Perhaps we \ufb01nd the\nfollowing set of sentences:\n(20.6) Budget airline Ryanair, which uses Charleroi as a hub, scrapped all\nweekend \ufb02ights out of the airport.\n(20.7) All \ufb02ights in and out of Ryanair\u2019s hub at Charleroi airport were grounded on\nFriday...\n(20.8) A spokesman at Charleroi, a main hub for Ryanair, estimated that 8000\npassengers had already been affected. 8CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\nFrom these results, we can use the context of words between the entity mentions,\nthe words before mention one, the word after mention two, and the named entity\ntypes of the two mentions, and perhaps other features, to extract general patterns\nsuch as the following:\n/ [ORG], which uses [LOC] as a hub /\n/ [ORG]'s hub at [LOC] /\n/ [LOC], a main hub for [ORG] /\nThese new patterns can then be used to search for additional tuples.\nBootstrapping systems also assign con\ufb01dence values to new tuples to avoid se-con\ufb01dence\nvalues\nmantic drift . In semantic drift, an erroneous pattern leads to the introduction of semantic drift\nerroneous tuples, which, in turn, lead to the creation of problematic patterns and the\nmeaning of the extracted relations \u2018drifts\u2019. Consider the following example:\n(20.9) Sydney has a ferry hub at Circular Quay.\nIf accepted as a positive example, this expression could lead to the incorrect in-\ntroduction of the tuple hSydney ;CircularQuayi. Patterns based on this tuple could\npropagate further errors into the database.\nCon\ufb01dence values for patterns are based on balancing two factors: the pattern\u2019s\nperformance with respect to the current set of tuples and the pattern\u2019s productivity\nin terms of the number of matches it produces in the document collection. More\nformally, given a document collection D, a current set of tuples T, and a proposed\npattern p, we need to track two factors:\n\u2022hits(p): the set of tuples in Tthatpmatches while looking in D\n\u2022\ufb01nds (p): The total set of tuples that p\ufb01nds in D\nThe following equation balances these considerations (Riloff and Jones, 1999).\nConfRlogF (p) =jhits(p)j\nj\ufb01nds (p)jlog(j\ufb01nds (p)j) (20.10)\nThis metric is generally normalized to produce a probability.\nWe can assess the con\ufb01dence in a proposed new tuple by combining the evidence\nsupporting it from all the patterns P0that match that tuple in D(Agichtein and Gra-\nvano, 2000). One way to combine such evidence is the noisy-or technique. Assume noisy-or\nthat a given tuple is supported by a subset of the patterns in P, each with its own\ncon\ufb01dence assessed as above. In the noisy-or model, we make two basic assump-\ntions. First, that for a proposed tuple to be false, allof its supporting patterns must\nhave been in error, and second, that the sources of their individual failures are all\nindependent. If we loosely treat our con\ufb01dence measures as probabilities, then the\nprobability of any individual pattern pfailing is 1\u0000Conf (p); the probability of all\nof the supporting patterns for a tuple being wrong is the product of their individual\nfailure probabilities, leaving us with the following equation for our con\ufb01dence in a\nnew tuple.\nConf (t) =1\u0000Y\np2P0(1\u0000Conf (p)) (20.11)\nSetting conservative con\ufb01dence thresholds for the acceptance of new patterns\nand tuples during the bootstrapping process helps prevent the system from drifting\naway from the targeted relation. 20.2 \u2022 R ELATION EXTRACTION ALGORITHMS 9\n20.2.4 Distant Supervision for Relation Extraction\nAlthough hand-labeling text with relation labels is expensive to produce, there are\nways to \ufb01nd indirect sources of training data. The distant supervision methoddistant\nsupervision\n(Mintz et al., 2009) combines the advantages of bootstrapping with supervised learn-\ning. Instead of just a handful of seeds, distant supervision uses a large database to\nacquire a huge number of seed examples, creates lots of noisy pattern features from\nall these examples and then combines them in a supervised classi\ufb01er.\nFor example suppose we are trying to learn the place-of-birth relationship be-\ntween people and their birth cities. In the seed-based approach, we might have only\n5 examples to start with. But Wikipedia-based databases like DBPedia or Freebase\nhave tens of thousands of examples of many relations; including over 100,000 ex-\namples of place-of-birth , (<Edwin Hubble, Marshfield> ,<Albert Einstein,\nUlm> , etc.,). The next step is to run named entity taggers on large amounts of text\u2014\nMintz et al. (2009) used 800,000 articles from Wikipedia\u2014and extract all sentences\nthat have two named entities that match the tuple, like the following:\n...Hubble was born in Marsh\ufb01eld...\n...Einstein, born (1879), Ulm...\n...Hubble\u2019s birthplace in Marsh\ufb01eld...\nTraining instances can now be extracted from this data, one training instance\nfor each identical tuple <relation, entity1, entity2> . Thus there will be one\ntraining instance for each of:\n<born-in, Edwin Hubble, Marshfield>\n<born-in, Albert Einstein, Ulm>\n<born-year, Albert Einstein, 1879>\nand so on.\nWe can then apply feature-based or neural classi\ufb01cation. For feature-based\nclassi\ufb01cation, we can use standard supervised relation extraction features like the\nnamed entity labels of the two mentions, the words and dependency paths in be-\ntween the mentions, and neighboring words. Each tuple will have features col-\nlected from many training instances; the feature vector for a single training instance\nlike ( <born-in,Albert Einstein, Ulm> will have lexical and syntactic features\nfrom many different sentences that mention Einstein and Ulm.\nBecause distant supervision has very large training sets, it is also able to use very\nrich features that are conjunctions of these individual features. So we will extract\nthousands of patterns that conjoin the entity types with the intervening words or\ndependency paths like these:\nPER was born in LOC\nPER, born (XXXX), LOC\nPER\u2019s birthplace in LOC\nTo return to our running example, for this sentence:\n(20.12) American Airlines , a unit of AMR, immediately matched the move,\nspokesman Tim Wagner said\nwe would learn rich conjunction features like this one:\nM1 = ORG & M2 = PER & nextword=\u201csaid\u201d& path= NP\"NP\"S\"S#NP\nThe result is a supervised classi\ufb01er that has a huge rich set of features to use\nin detecting relations. Since not every test sentence will have one of the training 10 CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\nrelations, the classi\ufb01er will also need to be able to label an example as no-relation .\nThis label is trained by randomly selecting entity pairs that do not appear in any\nFreebase relation, extracting features for them, and building a feature vector for\neach such tuple. The \ufb01nal algorithm is sketched in Fig. 20.8.\nfunction DISTANT SUPERVISION (Database D, Text T )returns relation classi\ufb01er C\nforeach relation R\nforeach tuple ( e1,e2 ) of entities with relation RinD\nsentences Sentences in Tthat contain e1ande2\nf Frequent features in sentences\nobservations observations + new training tuple ( e1, e2, f, R )\nC Train supervised classi\ufb01er on observations\nreturn C\nFigure 20.8 The distant supervision algorithm for relation extraction. A neural classi\ufb01er\nwould skip the feature set f.\nDistant supervision shares advantages with each of the methods we\u2019ve exam-\nined. Like supervised classi\ufb01cation, distant supervision uses a classi\ufb01er with lots\nof features, and supervised by detailed hand-created knowledge. Like pattern-based\nclassi\ufb01ers, it can make use of high-precision evidence for the relation between en-\ntities. Indeed, distance supervision systems learn patterns just like the hand-built\npatterns of early relation extractors. For example the is-aorhypernym extraction\nsystem of Snow et al. (2005) used hypernym/hyponym NP pairs from WordNet as\ndistant supervision, and then learned new patterns from large amounts of text. Their\nsystem induced exactly the original 5 template patterns of Hearst (1992a), but also\n70,000 additional patterns including these four:\nNPHlike NP Many hormones like leptin...\nNPHcalled NP ...using a markup language called XHTML\nNP is a NP H Ruby is a programming language...\nNP, a NP H IBM, a company with a long...\nThis ability to use a large number of features simultaneously means that, un-\nlike the iterative expansion of patterns in seed-based systems, there\u2019s no semantic\ndrift. Like unsupervised classi\ufb01cation, it doesn\u2019t use a labeled training corpus of\ntexts, so it isn\u2019t sensitive to genre issues in the training corpus, and relies on very\nlarge amounts of unlabeled data. Distant supervision also has the advantage that it\ncan create training tuples to be used with neural classi\ufb01ers, where features are not\nrequired.\nThe main problem with distant supervision is that it tends to produce low-precision\nresults, and so current research focuses on ways to improve precision. Furthermore,\ndistant supervision can only help in extracting relations for which a large enough\ndatabase already exists. To extract new relations without datasets, or relations for\nnew domains, purely unsupervised methods must be used.\n20.2.5 Unsupervised Relation Extraction\nThe goal of unsupervised relation extraction is to extract relations from the web\nwhen we have no labeled training data, and not even any list of relations. This task\nis often called open information extraction orOpen IE . In Open IE, the relationsopen\ninformation\nextraction 20.2 \u2022 R ELATION EXTRACTION ALGORITHMS 11\nare simply strings of words (usually beginning with a verb).\nFor example, the ReVerb system (Fader et al., 2011) extracts a relation from a\nsentence sin 4 steps:\n1. Run a part-of-speech tagger and entity chunker over s\n2. For each verb in s, \ufb01nd the longest sequence of words wthat start with a verb\nand satisfy syntactic and lexical constraints, merging adjacent matches.\n3. For each phrase w, \ufb01nd the nearest noun phrase xto the left which is not a\nrelative pronoun, wh-word or existential \u201cthere\u201d. Find the nearest noun phrase\nyto the right.\n4. Assign con\ufb01dence cto the relation r= (x;w;y)using a con\ufb01dence classi\ufb01er\nand return it.\nA relation is only accepted if it meets syntactic and lexical constraints. The\nsyntactic constraints ensure that it is a verb-initial sequence that might also include\nnouns (relations that begin with light verbs like make ,have , ordooften express the\ncore of the relation with a noun, like have a hub in ):\nVjVPjVW*P\nV = verb particle? adv?\nW = (nounjadjjadvjpronjdet )\nP = (prepjparticlejin\ufb01nitive \u201cto\u201d)\nThe lexical constraints are based on a dictionary Dthat is used to prune very rare,\nlong relation strings. The intuition is to eliminate candidate relations that don\u2019t oc-\ncur with suf\ufb01cient number of distinct argument types and so are likely to be bad\nexamples. The system \ufb01rst runs the above relation extraction algorithm of\ufb02ine on\n500 million web sentences and extracts a list of all the relations that occur after nor-\nmalizing them (removing in\ufb02ection, auxiliary verbs, adjectives, and adverbs). Each\nrelation ris added to the dictionary if it occurs with at least 20 different arguments.\nFader et al. (2011) used a dictionary of 1.7 million normalized relations.\nFinally, a con\ufb01dence value is computed for each relation using a logistic re-\ngression classi\ufb01er. The classi\ufb01er is trained by taking 1000 random web sentences,\nrunning the extractor, and hand labeling each extracted relation as correct or incor-\nrect. A con\ufb01dence classi\ufb01er is then trained on this hand-labeled data, using features\nof the relation and the surrounding words. Fig. 20.9 shows some sample features\nused in the classi\ufb01cation.\n(x,r,y) covers all words in s\nthe last preposition in risfor\nthe last preposition in rison\nlen(s)\u001410\nthere is a coordinating conjunction to the left of rins\nrmatches a lone V in the syntactic constraints\nthere is preposition to the left of xins\nthere is an NP to the right of yins\nFigure 20.9 Features for the classi\ufb01er that assigns con\ufb01dence to relations extracted by the\nOpen Information Extraction system REVERB (Fader et al., 2011).\nFor example the following sentence:\n(20.13) United has a hub in Chicago, which is the headquarters of United\nContinental Holdings. 12 CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\nhas the relation phrases has a hub in andis the headquarters of (it also has hasand\nis, but longer phrases are preferred). Step 3 \ufb01nds United to the left and Chicago to\nthe right of has a hub in , and skips over which to \ufb01nd Chicago to the left of is the\nheadquarters of . The \ufb01nal output is:\nr1: <United, has a hub in, Chicago>\nr2: <Chicago, is the headquarters of, United Continental Holdings>\nThe great advantage of unsupervised relation extraction is its ability to handle\na huge number of relations without having to specify them in advance. The dis-\nadvantage is the need to map all the strings into some canonical form for adding\nto databases or knowledge graphs. Current methods focus heavily on relations ex-\npressed with verbs, and so will miss many relations that are expressed nominally.\n20.2.6 Evaluation of Relation Extraction\nSupervised relation extraction systems are evaluated by using test sets with human-\nannotated, gold-standard relations and computing precision, recall, and F-measure.\nLabeled precision and recall require the system to classify the relation correctly,\nwhereas unlabeled methods simply measure a system\u2019s ability to detect entities that\nare related.\nSemi-supervised andunsupervised methods are much more dif\ufb01cult to evalu-\nate, since they extract totally new relations from the web or a large text. Because\nthese methods use very large amounts of text, it is generally not possible to run them\nsolely on a small labeled test set, and as a result it\u2019s not possible to pre-annotate a\ngold set of correct instances of relations.\nFor these methods it\u2019s possible to approximate (only) precision by drawing a\nrandom sample of relations from the output, and having a human check the accuracy\nof each of these relations. Usually this approach focuses on the tuples to be extracted\nfrom a body of text rather than on the relation mentions ; systems need not detect\nevery mention of a relation to be scored correctly. Instead, the evaluation is based\non the set of tuples occupying the database when the system is \ufb01nished. That is,\nwe want to know if the system can discover that Ryanair has a hub at Charleroi; we\ndon\u2019t really care how many times it discovers it. The estimated precision \u02c6Pis then\n\u02c6P=# of correctly extracted relation tuples in the sample\ntotal # of extracted relation tuples in the sample.(20.14)\nAnother approach that gives us a little bit of information about recall is to com-\npute precision at different levels of recall. Assuming that our system is able to\nrank the relations it produces (by probability, or con\ufb01dence) we can separately com-\npute precision for the top 1000 new relations, the top 10,000 new relations, the top\n100,000, and so on. In each case we take a random sample of that set. This will\nshow us how the precision curve behaves as we extract more and more tuples. But\nthere is no way to directly evaluate recall.\n20.3 Extracting Events\nThe task of event extraction is to identify mentions of events in texts. For theevent\nextraction\npurposes of this task, an event mention is any expression denoting an event or state\nthat can be assigned to a particular point, or interval, in time. The following markup\nof the sample text on page 1 shows all the events in this text. 20.4 \u2022 R EPRESENTING TIME 13\n[EVENT Citing] high fuel prices, United Airlines [ EVENT said] Fri-\nday it has [ EVENT increased] fares by $6 per round trip on \ufb02ights to\nsome cities also served by lower-cost carriers. American Airlines, a unit\nof AMR Corp., immediately [ EVENT matched] [ EVENT the move],\nspokesman Tim Wagner [ EVENT said]. United, a unit of UAL Corp.,\n[EVENT said] [ EVENT the increase] took effect Thursday and [ EVENT\napplies] to most routes where it [ EVENT competes] against discount\ncarriers, such as Chicago to Dallas and Denver to San Francisco.\nIn English, most event mentions correspond to verbs, and most verbs introduce\nevents. However, as we can see from our example, this is not always the case. Events\ncan be introduced by noun phrases, as in the move andthe increase , and some verbs\nfail to introduce events, as in the phrasal verb took effect , which refers to when the\nevent began rather than to the event itself. Similarly, light verbs such as make ,take, light verbs\nandhave often fail to denote events. A light verb is a verb that has very little meaning\nitself, and the associated event is instead expressed by its direct object noun. In light\nverb examples like took a \ufb02ight , it\u2019s the word \ufb02ight that de\ufb01nes the event; these light\nverbs just provide a syntactic structure for the noun\u2019s arguments.\nVarious versions of the event extraction task exist, depending on the goal. For\nexample in the TempEval shared tasks (Verhagen et al. 2009) the goal is to extract\nevents and aspects like their aspectual and temporal properties. Events are to be\nclassi\ufb01ed as actions, states, reporting events (say, report, tell, explain ), perceptionreporting\nevents\nevents, and so on. The aspect, tense, and modality of each event also needs to be\nextracted. Thus for example the various said events in the sample text would be\nannotated as (class=REPORTING, tense=PAST, aspect=PERFECTIVE).\nEvent extraction is generally modeled via supervised learning, detecting events\nvia IOB sequence models and assigning event classes and attributes with multi-class\nclassi\ufb01ers. The input can be neural models starting from encoders; or classic feature-\nbased models using features like those in Fig. 20.10.\nFeature Explanation\nCharacter af\ufb01xes Character-level pre\ufb01xes and suf\ufb01xes of target word\nNominalization suf\ufb01x Character-level suf\ufb01xes for nominalizations (e.g., -tion )\nPart of speech Part of speech of the target word\nLight verb Binary feature indicating that the target is governed by a light verb\nSubject syntactic category Syntactic category of the subject of the sentence\nMorphological stem Stemmed version of the target word\nVerb root Root form of the verb basis for a nominalization\nWordNet hypernyms Hypernym set for the target\nFigure 20.10 Features commonly used in classic feature-based approaches to event detection.\n20.4 Representing Time\nLet\u2019s begin by introducing the basics of temporal logic and how human languages temporal logic\nconvey temporal information. The most straightforward theory of time holds that it\n\ufb02ows inexorably forward and that events are associated with either points or inter-\nvals in time, as on a timeline. We can order distinct events by situating them on the\ntimeline; one event precedes another if the \ufb02ow of time leads from the \ufb01rst event 14 CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\nto the second. Accompanying these notions in most theories is the idea of the cur-\nrent moment in time. Combining this notion with the idea of a temporal ordering\nrelationship yields the familiar notions of past, present, and future.\nVarious kinds of temporal representation systems can be used to talk about tem-\nporal ordering relationship. One of the most commonly used in computational mod-\neling is the interval algebra of Allen (1984). Allen models all events and time interval algebra\nexpressions as intervals there is no representation for points (although intervals can\nbe very short). In order to deal with intervals without points, he identi\ufb01es 13 primi-\ntive relations that can hold between these temporal intervals. Fig. 20.11 shows these\n13Allen relations . Allen relations\nBABA\nBA\nAAB\nBAB\nTime A  before BB after  AA overlaps BB overlaps' AA meets BB meets' AA equals B(B equals A)A starts BB starts' AA finishes BB finishes' A\nBA during BB during' AA\nFigure 20.11 The 13 temporal relations from Allen (1984).\n20.4.1 Reichenbach\u2019s reference point\nThe relation between simple verb tenses and points in time is by no means straight-\nforward. The present tense can be used to refer to a future event, as in this example:\n(20.15) Ok, we \ufb02y from San Francisco to Boston at 10.\nOr consider the following examples:\n(20.16) Flight 1902 arrived late.\n(20.17) Flight 1902 had arrived late.\nAlthough both refer to events in the past, representing them in the same way seems\nwrong. The second example seems to have another unnamed event lurking in the\nbackground (e.g., Flight 1902 had already arrived late when something else hap-\npened). 20.4 \u2022 R EPRESENTING TIME 15\nTo account for this phenomena, Reichenbach (1947) introduced the notion of\nareference point . In our simple temporal scheme, the current moment in time is reference point\nequated with the time of the utterance and is used as a reference point for when\nthe event occurred (before, at, or after). In Reichenbach\u2019s approach, the notion of\nthe reference point is separated from the utterance time and the event time. The\nfollowing examples illustrate the basics of this approach:\n(20.18) When Mary\u2019s \ufb02ight departed, I ate lunch.\n(20.19) When Mary\u2019s \ufb02ight departed, I had eaten lunch.\nIn both of these examples, the eating event has happened in the past, that is, prior\nto the utterance. However, the verb tense in the \ufb01rst example indicates that the eating\nevent began when the \ufb02ight departed, while the second example indicates that the\neating was accomplished prior to the \ufb02ight\u2019s departure. Therefore, in Reichenbach\u2019s\nterms the departure event speci\ufb01es the reference point. These facts can be accom-\nmodated by additional constraints relating the eating anddeparture events. In the\n\ufb01rst example, the reference point precedes the eating event, and in the second exam-\nple, the eating precedes the reference point. Figure 20.12 illustrates Reichenbach\u2019s\napproach with the primary English tenses. Exercise 20.4 asks you to represent these\nexamples in FOL.\nPast PerfectSimple PastPresent Perfect\nSimple FutureFuture PerfectPresentEE\nEER\nRUR,EUR,U\nU,R,EU,RU\nFigure 20.12 Reichenbach\u2019s approach applied to various English tenses. In these diagrams,\ntime \ufb02ows from left to right, Edenotes the time of the event, Rdenotes the reference time,\nandUdenotes the time of the utterance.\nLanguages have many other ways to convey temporal information besides tense.\nMost useful for our purposes will be temporal expressions like in the morning or\n6:45 orafterwards .\n(20.20) I\u2019d like to go at 6:45 in the morning.\n(20.21) Somewhere around noon, please.\n(20.22) I want to take the train back afterwards.\nIncidentally, temporal expressions display a fascinating metaphorical conceptual\norganization. Temporal expressions in English are frequently expressed in spatial\nterms, as is illustrated by the various uses of at,in,somewhere , and near in these\nexamples (Lakoff and Johnson 1980, Jackendoff 1983). Metaphorical organizations\nsuch as these, in which one domain is systematically expressed in terms of another,\nare very common in languages of the world. 16 CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\n20.5 Representing Aspect\nA related notion to time is aspect , which is what we call the way events can be aspect\ncategorized by their internal temporal structure or temporal contour. By this we\nmean questions like whether events are ongoing or have ended, or whether they are\nconceptualized as happening at a point in time or over some interval. Such notions\nof temporal contour have been used to divide event expressions into classes since\nAristotle, although the set of four classes we\u2019ll introduce here is due to Vendler\n(1967) (you may also see the German term aktionsart used to refer to these classes). aktionsart\nThe most basic aspectual distinction is between events (which involve change) events\nandstates (which do not involve change). Stative expressions represent the notion states\nstative of an event participant being in a state , or having a particular property, at a given\npoint in time. Stative expressions capture aspects of the world at a single point in\ntime, and conceptualize the participant as unchanging and continuous. Consider the\nfollowing ATIS examples.\n(20.23) I like express trains.\n(20.24) I need the cheapest fare.\n(20.25) I want to go \ufb01rst class.\nIn examples like these, the event participant denoted by the subject can be seen as\nexperiencing something at a speci\ufb01c point in time, and don\u2019t involve any kind of\ninternal change over time (the liking or needing is conceptualized as continuous and\nunchanging).\nNon-states (which we\u2019ll refer to as events ) are divided into subclasses; we\u2019ll\nintroduce three here. Activity expressions describe events undertaken by a partic- activity\nipant that occur over a span of time (rather than being conceptualized as a single\npoint in time like stative expressions), and have no particular end point. Of course\nin practice all things end, but the meaning of the expression doesn\u2019t represent this\nfact. Consider the following examples:\n(20.26) She drove a Mazda.\n(20.27) I live in Brooklyn.\nThese examples both specify that the subject is engaged in, or has engaged in, the\nactivity speci\ufb01ed by the verb for some period of time, but doesn\u2019t specify when the\ndriving or living might have stopped.\nTwo more classes of expressions, achievement expressions and accomplish-\nment expressions, describe events that take place over time, but also conceptualize\nthe event as having a particular kind of endpoint or goal. The Greek word telos\nmeans \u2018end\u2019 or \u2019goal\u2019 and so the events described by these kinds of expressions are\noften called telic events. telic\nAccomplishment expressions describe events that have a natural end point andaccomplishment\nexpressions\nresult in a particular state. Consider the following examples:\n(20.28) He booked me a reservation.\n(20.29) The 7:00 train got me to New York City.\nIn these examples, an event is seen as occurring over some period of time that ends\nwhen the intended state is accomplished (i.e., the state of me having a reservation,\nor me being in New York City).\nThe \ufb01nal aspectual class, achievement expressions , is only subtly different thanachievement\nexpressions\naccomplishments. Consider the following: 20.6 \u2022 T EMPORALLY ANNOTATED DATASETS : TIMEBANK 17\n(20.30) She found her gate.\n(20.31) I reached New York.\nLike accomplishment expressions, achievement expressions result in a state. But\nunlike accomplishments, achievement events are \u2018punctual\u2019: they are thought of as\nhappening in an instant and the verb doesn\u2019t conceptualize the process or activ-\nity leading up the state. Thus the events in these examples may in fact have been\npreceded by extended searching ortraveling events, but the verb doesn\u2019t conceptu-\nalize these preceding processes, but rather conceptualizes the events corresponding\nto\ufb01nding andreaching as points, not intervals.\nIn summary, a standard way of categorizing event expressions by their temporal\ncontours is via these four general classes:\nStative: I know my departure gate.\nActivity: John is \ufb02ying.\nAccomplishment: Sally booked her \ufb02ight.\nAchievement: She found her gate.\nBefore moving on, note that event expressions can easily be shifted from one\nclass to another. Consider the following examples:\n(20.32) I \ufb02ew.\n(20.33) I \ufb02ew to New York.\nThe \ufb01rst example is a simple activity; it has no natural end point. The second ex-\nample is clearly an accomplishment event since it has an end point, and results in a\nparticular state. Clearly, the classi\ufb01cation of an event is not solely governed by the\nverb, but by the semantics of the entire expression in context.\n20.6 Temporally Annotated Datasets: TimeBank\nTheTimeBank corpus consists of American English text annotated with temporal TimeBank\ninformation (Pustejovsky et al., 2003). The annotations use TimeML (Saur \u00b4\u0131 et al.,\n2006), a markup language for time based on Allen\u2019s interval algebra discussed above\n(Allen, 1984). There are three types of TimeML objects: an E VENT represent events\nand states, a T IME represents time expressions like dates, and a L INK represents\nvarious relationships between events and times (event-event, event-time, and time-\ntime). The links include temporal links (TL INK) for the 13 Allen relations, aspec-\ntual links (AL INK) for aspectual relationships between events and subevents, and\nSLINKS which mark factuality.\nConsider the following sample sentence and its corresponding markup shown in\nFig. 20.13, selected from one of the TimeBank documents.\n(20.34) Delta Air Lines earnings soared 33% to a record in the \ufb01scal \ufb01rst quarter,\nbucking the industry trend toward declining pro\ufb01ts.\nThis text has three events and two temporal expressions (including the creation\ntime of the article, which serves as the document time), and four temporal links that\ncapture the using the Allen relations:\n\u2022 Soaring e1isincluded in the \ufb01scal \ufb01rst quarter t58\n\u2022 Soaring e1isbefore 1989-10-26 t57\n\u2022 Soaring e1issimultaneous with the bucking e3 18 CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\n<TIMEX3 tid=\"t57\" type=\"DATE\" value=\"1989-10-26\" functionInDocument=\"CREATION_TIME\">\n10/26/89 </TIMEX3>\nDelta Air Lines earnings <EVENT eid=\"e1\" class=\"OCCURRENCE\"> soared </EVENT> 33% to a\nrecord in <TIMEX3 tid=\"t58\" type=\"DATE\" value=\"1989-Q1\" anchorTimeID=\"t57\"> the\nfiscal first quarter </TIMEX3>, <EVENT eid=\"e3\" class=\"OCCURRENCE\">bucking</EVENT>\nthe industry trend toward <EVENT eid=\"e4\" class=\"OCCURRENCE\">declining</EVENT>\nprofits.\nFigure 20.13 Example from the TimeBank corpus.\n\u2022 Declining e4includes soaring e1\nWe can also visualize the links as a graph. The TimeBank snippet in Eq. 20.35\nwould be represented with a graph like Fig. 20.14.\n(20.35) [DCT:11/02/891] 1: Paci\ufb01c First Financial Corp. said 2shareholders\napproved 3itsacquisition 4by Royal Trustco Ltd. of Toronto for $27 a share,\nor $212 million. The thrift holding company said 5itexpects 6toobtain 7\nregulatory approval 8andcomplete 9thetransaction 10byyear-end 11.\n1234567811910BEFOREBEFOREAFTERSIMULTANEOUSENDSCULMINATESBEFOREEVIDENTIALMODALFACTIVEMODALEVIDENTIALMODAL\nFigure 20.14 A graph of the text in Eq. 20.35, adapted from (Ocal et al., 2022). TL INKS\nare shown in blue, AL INKS in red, and SL INKS in green.\n20.7 Automatic Temporal Analysis\nHere we introduce the three common steps used in analyzing time in text:\n1. Extracting temporal expressions\n2.Normalizing these expressions, by converting them to a standard format.\n3.Linking events to times and extracting time graphs and timelines\n20.7.1 Extracting Temporal Expressions\nTemporal expressions are phrases that refer to absolute points in time, relative times,\ndurations, and sets of these. Absolute temporal expressions are those that can be absolute\nmapped directly to calendar dates, times of day, or both. Relative temporal expres- relative\nsions map to particular times through some other reference point (as in a week from\nlast Tuesday ). Finally, durations denote spans of time at varying levels of granular- duration\nity (seconds, minutes, days, weeks, centuries, etc.). Figure 20.15 lists some sample\ntemporal expressions in each of these categories.\nTemporal expressions are grammatical constructions that often have temporal\nlexical triggers as their heads, making them easy to \ufb01nd. Lexical triggers might lexical triggers\nbe nouns, proper nouns, adjectives, and adverbs; full temporal expressions consist 20.7 \u2022 A UTOMATIC TEMPORAL ANALYSIS 19\nAbsolute Relative Durations\nApril 24, 1916 yesterday four hours\nThe summer of \u201977 next semester three weeks\n10:15 AM two weeks from yesterday six days\nThe 3rd quarter of 2006 last quarter the last three quarters\nFigure 20.15 Examples of absolute, relational and durational temporal expressions.\nof their phrasal projections: noun phrases, adjective phrases, and adverbial phrases\n(Figure 20.16).\nCategory Examples\nNoun morning ,noon ,night ,winter ,dusk,dawn\nProper Noun January, Monday, Ides, Easter, Rosh Hashana, Ramadan, Tet\nAdjective recent, past, annual, former\nAdverb hourly, daily, monthly, yearly\nFigure 20.16 Examples of temporal lexical triggers.\nThe task is to detect temporal expressions in running text, like this examples,\nshown with TIMEX3 tags (Pustejovsky et al. 2005, Ferro et al. 2005).\nA fare increase initiated <TIMEX3 >last week </TIMEX3 >by UAL\nCorp\u2019s United Airlines was matched by competitors over <TIMEX3 >the\nweekend </TIMEX3 >, marking the second successful fare increase in\n<TIMEX3 >two weeks </TIMEX3 >.\nRule-based approaches use cascades of regular expressions to recognize larger\nand larger chunks from previous stages, based on patterns containing parts of speech,\ntrigger words (e.g., February ) or classes (e.g., MONTH ) (Chang and Manning, 2012;\nStr\u00a8otgen and Gertz, 2013; Chambers, 2013). Here\u2019s a rule from SUTime (Chang and\nManning, 2012) for detecting expressions like 3 years old :\n/(\\d+)[-\\s]($TEUnits)(s)?([-\\s]old)?/\nSequence-labeling approaches use the standard IOB scheme, marking words\nthat are either (I)nside, (O)utside or at the (B)eginning of a temporal expression:\nA\nOfare\nOincrease\nOinitiated\nOlast\nBweek\nIby\nOUAL\nOCorp\u2019s...\nO\nA statistical sequence labeler is trained, using either embeddings or a \ufb01ne-tuned\nencoder, or classic features extracted from the token and context including words,\nlexical triggers, and POS.\nTemporal expression recognizers are evaluated with the usual recall, precision,\nandF-measures. A major dif\ufb01culty for all of these very lexicalized approaches is\navoiding expressions that trigger false positives:\n(20.36) 1984 tells the story of Winston Smith...\n(20.37) ...U2\u2019s classic Sunday Bloody Sunday\n20.7.2 Temporal Normalization\nTemporal normalization is the task of mapping a temporal expression to a pointtemporal\nnormalization\nin time or to a duration. Points in time correspond to calendar dates, to times of\nday, or both. Durations primarily consist of lengths of time. Normalized times\nare represented via the ISO 8601 standard for encoding temporal values (ISO8601,\n2004). Fig. 20.17 reproduces our earlier example with these value attributes. 20 CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\n<TIMEX3 i d =\u201d t1 \u2019 \u2019 t y p e =\u201dDATE\u201d v a l u e =\u201d 2007 \u221207 \u221202 \u201d f u n c t i o n I n D o c u m e n t =\u201dCREATION TIME\u201d >\nJ u l y 2 , 2007 </TIMEX3 >A f a r e i n c r e a s e i n i t i a t e d <TIMEX3 i d =\u201d t 2 \u201d t y p e =\u201dDATE\u201d\nv a l u e =\u201d 2007 \u2212W26\u201d anchorTimeID=\u201d t 1 \u201d >l a s t week </TIMEX3 >by U n i t e d A i r l i n e s was\nmatched by c o m p e t i t o r s o ve r <TIMEX3 i d =\u201d t 3 \u201d t y p e =\u201dDURATION\u201d v a l u e =\u201dP1WE\u201d\nanchorTimeID=\u201d t 1 \u201d >t h e weekend </TIMEX3 >, marking t h e second s u c c e s s f u l f a r e\ni n c r e a s e i n <TIMEX3 i d =\u201d t 4 \u201d t y p e =\u201dDURATION\u201d v a l u e =\u201dP2W\u201d anchorTimeID=\u201d t 1 \u201d >two\nweeks </TIMEX3 >.\nFigure 20.17 TimeML markup including normalized values for temporal expressions.\nThe dateline, or document date, for this text was July 2, 2007 . The ISO repre-\nsentation for this kind of expression is YYYY-MM-DD, or in this case, 2007-07-02.\nThe encodings for the temporal expressions in our sample text all follow from this\ndate, and are shown here as values for the VALUE attribute.\nThe \ufb01rst temporal expression in the text proper refers to a particular week of the\nyear. In the ISO standard, weeks are numbered from 01 to 53, with the \ufb01rst week\nof the year being the one that has the \ufb01rst Thursday of the year. These weeks are\nrepresented with the template YYYY-Wnn. The ISO week for our document date is\nweek 27; thus the value for last week is represented as \u201c2007-W26\u201d.\nThe next temporal expression is the weekend . ISO weeks begin on Monday;\nthus, weekends occur at the end of a week and are fully contained within a single\nweek. Weekends are treated as durations, so the value of the VALUE attribute has\nto be a length. Durations are represented according to the pattern P nx, where nis\nan integer denoting the length and xrepresents the unit, as in P3Y for three years\nor P2D for two days . In this example, one weekend is captured as P1WE. In this\ncase, there is also suf\ufb01cient information to anchor this particular weekend as part of\na particular week. Such information is encoded in the ANCHOR TIMEID attribute.\nFinally, the phrase two weeks also denotes a duration captured as P2W. Figure 20.18\ngive some more examples, but there is a lot more to the various temporal annotation\nstandards; consult ISO8601 (2004), Ferro et al. (2005), and Pustejovsky et al. (2005)\nfor more details.\nUnit Pattern Sample Value\nFully speci\ufb01ed dates YYYY-MM-DD 1991-09-28\nWeeks YYYY-Wnn 2007-W27\nWeekends PnWE P1WE\n24-hour clock times HH:MM:SS 11:13:45\nDates and times YYYY-MM-DDTHH:MM:SS 1991-09-28T11:00:00\nFinancial quarters Qn 1999-Q3\nFigure 20.18 Sample ISO patterns for representing various times and durations.\nMost current approaches to temporal normalization are rule-based (Chang and\nManning 2012, Str \u00a8otgen and Gertz 2013). Patterns that match temporal expressions\nare associated with semantic analysis procedures. For example, the pattern above for\nrecognizing phrases like 3 years old can be associated with the predicate Duration\nthat takes two arguments, the length and the unit of time:\npattern: /(\\d+)[-\\s]($TEUnits)(s)?([-\\s]old)?/\nresult: Duration($1, $2)\nThe task is dif\ufb01cult because fully quali\ufb01ed temporal expressions are fairly rare\nin real texts. Most temporal expressions in news articles are incomplete and are only\nimplicitly anchored, often with respect to the dateline of the article, which we refer\nto as the document\u2019s temporal anchor . The values of temporal expressions suchtemporal\nanchor\nastoday ,yesterday , ortomorrow can all be computed with respect to this temporal 20.7 \u2022 A UTOMATIC TEMPORAL ANALYSIS 21\nanchor. The semantic procedure for today simply assigns the anchor, and the attach-\nments for tomorrow andyesterday add a day and subtract a day from the anchor,\nrespectively. Of course, given the cyclic nature of our representations for months,\nweeks, days, and times of day, our temporal arithmetic procedures must use modulo\narithmetic appropriate to the time unit being used.\nUnfortunately, even simple expressions such as the weekend orWednesday in-\ntroduce a fair amount of complexity. In our current example, the weekend clearly\nrefers to the weekend of the week that immediately precedes the document date. But\nthis won\u2019t always be the case, as is illustrated in the following example.\n(20.38) Random security checks that began yesterday at Sky Harbor will continue\nat least through the weekend.\nIn this case, the expression the weekend refers to the weekend of the week that the\nanchoring date is part of (i.e., the coming weekend). The information that signals\nthis meaning comes from the tense of continue , the verb governing the weekend .\nRelative temporal expressions are handled with temporal arithmetic similar to\nthat used for today andyesterday . The document date indicates that our example\narticle is ISO week 27, so the expression last week normalizes to the current week\nminus 1. To resolve ambiguous next andlastexpressions we consider the distance\nfrom the anchoring date to the nearest unit. Next Friday can refer either to the\nimmediately next Friday or to the Friday following that, but the closer the document\ndate is to a Friday, the more likely it is that the phrase will skip the nearest one. Such\nambiguities are handled by encoding language and domain-speci\ufb01c heuristics into\nthe temporal attachments.\n20.7.3 Temporal Ordering of Events\nThe goal of temporal analysis, is to link times to events and then \ufb01t all these events\ninto a complete timeline. This ambitious task is the subject of considerable current\nresearch but solving it with a high level of accuracy is beyond the capabilities of\ncurrent systems. A somewhat simpler, but still useful, task is to impose a partial or-\ndering on the events and temporal expressions mentioned in a text. Such an ordering\ncan provide many of the same bene\ufb01ts as a true timeline. An example of such a par-\ntial ordering is the determination that the fare increase by American Airlines came\nafter the fare increase by United in our sample text. Determining such an ordering\ncan be viewed as a binary relation detection and classi\ufb01cation task.\nEven this partial ordering task assumes that in addition to the detecting and nor-\nmalizing time expressions steps described above, we have already detected all the\nevents in the text. Indeed, many temporal expressions are anchored to events men-\ntioned in a text and not directly to other temporal expressions. Consider the follow-\ning example:\n(20.39) One week after the storm, JetBlue issued its customer bill of rights.\nTo determine when JetBlue issued its customer bill of rights we need to determine\nthe time of the storm event, and then we need to modify that time by the temporal\nexpression one week after .\nThus once the events and times have been detected, our goal next is to assert links\nbetween all the times and events: i.e. creating event-event, event-time, time-time,\nDCT-event, and DCT-time TimeML TL INKS . This can be done by training time\nrelation classi\ufb01ers to predict the correct T: INKbetween each pair of times/events,\nsupervised by the gold labels in the TimeBank corpus with features like words/em-\nbeddings, parse paths, tense and aspect The sieve-based architecture using precision- 22 CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\nranked sets of classi\ufb01ers, which we\u2019ll introduce in Chapter 23, is also commonly\nused.\nSystems that perform all 4 tasks (time extraction creation and normalization,\nevent extraction, and time/event linking) include TARSQI (Verhagen et al., 2005)\nCLEAR TK (Bethard, 2013), CAEVO (Chambers et al., 2014), and CATENA (Mirza\nand Tonelli, 2016).\n20.8 Template Filling\nMany texts contain reports of events, and possibly sequences of events, that often\ncorrespond to fairly common, stereotypical situations in the world. These abstract\nsituations or stories, related to what have been called scripts (Schank and Abel- scripts\nson, 1977), consist of prototypical sequences of sub-events, participants, and their\nroles. The strong expectations provided by these scripts can facilitate the proper\nclassi\ufb01cation of entities, the assignment of entities into roles and relations, and most\ncritically, the drawing of inferences that \ufb01ll in things that have been left unsaid. In\ntheir simplest form, such scripts can be represented as templates consisting of \ufb01xed templates\nsets of slots that take as values slot-\ufb01llers belonging to particular classes. The task\noftemplate \ufb01lling is to \ufb01nd documents that invoke particular scripts and then \ufb01ll the template \ufb01lling\nslots in the associated templates with \ufb01llers extracted from the text. These slot-\ufb01llers\nmay consist of text segments extracted directly from the text, or they may consist of\nconcepts that have been inferred from text elements through some additional pro-\ncessing.\nA \ufb01lled template from our original airline story might look like the following.\nFARE-RAISE ATTEMPT :2\n6664LEAD AIRLINE : U NITED AIRLINES\nAMOUNT : $6\nEFFECTIVE DATE: 2006-10-26\nFOLLOWER : A MERICAN AIRLINES3\n7775\nThis template has four slots ( LEAD AIRLINE ,AMOUNT ,EFFECTIVE DATE ,FOL-\nLOWER ). The next section describes a standard sequence-labeling approach to \ufb01lling\nslots. Section 20.8.2 then describes an older system based on the use of cascades of\n\ufb01nite-state transducers and designed to address a more complex template-\ufb01lling task\nthat current learning-based systems don\u2019t yet address.\n20.8.1 Machine Learning Approaches to Template Filling\nIn the standard paradigm for template \ufb01lling, we are given training documents with\ntext spans annotated with prede\ufb01ned templates and their slot \ufb01llers. Our goal is to\ncreate one template for each event in the input, \ufb01lling in the slots with text spans.\nThe task is generally modeled by training two separate supervised systems. The\n\ufb01rst system decides whether the template is present in a particular sentence. This\ntask is called template recognition or sometimes, in a perhaps confusing bit oftemplate\nrecognition\nterminology, event recognition . Template recognition can be treated as a text classi-\n\ufb01cation task, with features extracted from every sequence of words that was labeled\nin training documents as \ufb01lling any slot from the template being detected. The usual\nset of features can be used: tokens, embeddings, word shapes, part-of-speech tags,\nsyntactic chunk tags, and named entity tags. 20.8 \u2022 T EMPLATE FILLING 23\nThe second system has the job of role-\ufb01ller extraction . A separate classi\ufb01er isrole-\ufb01ller\nextraction\ntrained to detect each role ( LEAD -AIRLINE ,AMOUNT , and so on). This can be a\nbinary classi\ufb01er that is run on every noun-phrase in the parsed input sentence, or a\nsequence model run over sequences of words. Each role classi\ufb01er is trained on the\nlabeled data in the training set. Again, the usual set of features can be used, but now\ntrained only on an individual noun phrase or the \ufb01llers of a single slot.\nMultiple non-identical text segments might be labeled with the same slot la-\nbel. For example in our sample text, the strings United orUnited Airlines might be\nlabeled as the L EAD AIRLINE . These are not incompatible choices and the corefer-\nence resolution techniques introduced in Chapter 23 can provide a path to a solution.\nA variety of annotated collections have been used to evaluate this style of ap-\nproach to template \ufb01lling, including sets of job announcements, conference calls for\npapers, restaurant guides, and biological texts. A key open question is extracting\ntemplates in cases where there is no training data or even prede\ufb01ned templates, by\ninducing templates as sets of linked events (Chambers and Jurafsky, 2011).\n20.8.2 Earlier Finite-State Template-Filling Systems\nThe templates above are relatively simple. But consider the task of producing a\ntemplate that contained all the information in a text like this one (Grishman and\nSundheim, 1995):\nBridgestone Sports Co. said Friday it has set up a joint venture in Taiwan\nwith a local concern and a Japanese trading house to produce golf clubs to be\nshipped to Japan. The joint venture, Bridgestone Sports Taiwan Co., capital-\nized at 20 million new Taiwan dollars, will start production in January 1990\nwith production of 20,000 iron and \u201cmetal wood\u201d clubs a month.\nThe MUC-5 \u2018joint venture\u2019 task (the Message Understanding Conferences were\na series of U.S. government-organized information-extraction evaluations) was to\nproduce hierarchically linked templates describing joint ventures. Figure 20.19\nshows a structure produced by the FASTUS system (Hobbs et al., 1997). Note how\nthe \ufb01ller of the ACTIVITY slot of the TIE-UPtemplate is itself a template with slots.\nTie-up-1 Activity-1 :\nRELATIONSHIP tie-up C OMPANY Bridgestone Sports Taiwan Co.\nENTITIES Bridgestone Sports Co. P RODUCT iron and \u201cmetal wood\u201d clubs\na local concern S TART DATE DURING: January 1990\na Japanese trading house\nJOINT VENTURE Bridgestone Sports Taiwan Co.\nACTIVITY Activity-1\nAMOUNT NT$20000000\nFigure 20.19 The templates produced by FASTUS given the input text on page 23.\nEarly systems for dealing with these complex templates were based on cascades\nof transducers based on handwritten rules, as sketched in Fig. 20.20.\nThe \ufb01rst four stages use handwritten regular expression and grammar rules to\ndo basic tokenization, chunking, and parsing. Stage 5 then recognizes entities and\nevents with a recognizer based on \ufb01nite-state transducers (FSTs), and inserts the rec-\nognized objects into the appropriate slots in templates. This FST recognizer is based\non hand-built regular expressions like the following (NG indicates Noun-Group and\nVG Verb-Group), which matches the \ufb01rst sentence of the news story above. 24 CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\nNo. Step Description\n1 Tokens Tokenize input stream of characters\n2 Complex Words Multiword phrases, numbers, and proper names.\n3 Basic phrases Segment sentences into noun and verb groups\n4 Complex phrases Identify complex noun groups and verb groups\n5 Semantic Patterns Identify entities and events, insert into templates.\n6 Merging Merge references to the same entity or event\nFigure 20.20 Levels of processing in FASTUS (Hobbs et al., 1997). Each level extracts a\nspeci\ufb01c type of information which is then passed on to the next higher level.\nNG(Company/ies) VG(Set-up) NG(Joint-Venture) with NG(Company/ies)\nVG(Produce) NG(Product)\nThe result of processing these two sentences is the \ufb01ve draft templates (Fig. 20.21)\nthat must then be merged into the single hierarchical structure shown in Fig. 20.19.\nThe merging algorithm, after performing coreference resolution, merges two activi-\nties that are likely to be describing the same events.\n#Template/Slot Value\n1RELATIONSHIP : TIE-UP\nENTITIES : Bridgestone Co., a local concern, a Japanese trading house\n2ACTIVITY : PRODUCTION\nPRODUCT : \u201cgolf clubs\u201d\n3RELATIONSHIP : TIE-UP\nJOINT VENTURE : \u201cBridgestone Sports Taiwan Co.\u201d\nAMOUNT : NT$20000000\n4ACTIVITY : PRODUCTION\nCOMPANY : \u201cBridgestone Sports Taiwan Co.\u201d\nSTART DATE: DURING : January 1990\n5ACTIVITY : PRODUCTION\nPRODUCT : \u201ciron and \u201cmetal wood\u201d clubs\u201d\nFigure 20.21 The \ufb01ve partial templates produced by stage 5 of FASTUS . These templates\nare merged in stage 6 to produce the \ufb01nal template shown in Fig. 20.19 on page 23.\n20.9 Summary\nThis chapter has explored techniques for extracting limited forms of semantic con-\ntent from texts.\n\u2022Relations among entities can be extracted by pattern-based approaches, su-\npervised learning methods when annotated training data is available, lightly\nsupervised bootstrapping methods when small numbers of seed tuples or\nseed patterns are available, distant supervision when a database of relations\nis available, and unsupervised orOpen IE methods.\n\u2022 Reasoning about time can be facilitated by detection and normalization of\ntemporal expressions .\n\u2022Events can be ordered in time using sequence models and classi\ufb01ers trained\non temporally- and event-labeled data like the TimeBank corpus . BIBLIOGRAPHICAL AND HISTORICAL NOTES 25\n\u2022Template-\ufb01lling applications can recognize stereotypical situations in texts\nand assign elements from the text to roles represented as \ufb01xed sets of slots .\nBibliographical and Historical Notes\nThe earliest work on information extraction addressed the template-\ufb01lling task in the\ncontext of the Frump system (DeJong, 1982). Later work was stimulated by the U.S.\ngovernment-sponsored MUC conferences (Sundheim 1991, Sundheim 1992, Sund-\nheim 1993, Sundheim 1995). Early MUC systems like CIRCUS system (Lehnert\net al., 1991) and SCISOR (Jacobs and Rau, 1990) were quite in\ufb02uential and inspired\nlater systems like FASTUS (Hobbs et al., 1997). Chinchor et al. (1993) describe the\nMUC evaluation techniques.\nDue to the dif\ufb01culty of porting systems from one domain to another, attention\nshifted to machine learning approaches. Early supervised learning approaches to\nIE (Cardie 1993, Cardie 1994, Riloff 1993, Soderland et al. 1995, Huffman 1996)\nfocused on automating the knowledge acquisition process, mainly for \ufb01nite-state\nrule-based systems. Their success, and the earlier success of HMM-based speech\nrecognition, led to the use of sequence labeling (HMMs: Bikel et al. 1997; MEMMs\nMcCallum et al. 2000; CRFs: Lafferty et al. 2001), and a wide exploration of fea-\ntures (Zhou et al., 2005). Neural approaches followed from the pioneering results of\nCollobert et al. (2011), who applied a CRF on top of a convolutional net.\nProgress in this area continues to be stimulated by formal evaluations with shared\nbenchmark datasets, including the Automatic Content Extraction (ACE) evaluations\nof 2000-2007 on named entity recognition, relation extraction, and temporal ex-\npressions1, the KBP (Knowledge Base Population ) evaluations (Ji et al. 2010, Sur- KBP\ndeanu 2013) of relation extraction tasks like slot \ufb01lling (extracting attributes (\u2018slots\u2019) slot \ufb01lling\nlike age, birthplace, and spouse for a given entity) and a series of SemEval work-\nshops (Hendrickx et al., 2009).\nSemisupervised relation extraction was \ufb01rst proposed by Hearst (1992b), and\nextended by systems like AutoSlog-TS (Riloff, 1996), DIPRE (Brin, 1998), SNOW-\nBALL (Agichtein and Gravano, 2000), and Jones et al. (1999). The distant super-\nvision algorithm we describe was drawn from Mintz et al. (2009), who \ufb01rst used\nthe term \u2018distant supervision\u2019 (which was suggested to them by Chris Manning)\nbut similar ideas had occurred in earlier systems like Craven and Kumlien (1999)\nand Morgan et al. (2004) under the name weakly labeled data , as well as in Snow\net al. (2005) and Wu and Weld (2007). Among the many extensions are Wu and\nWeld (2010), Riedel et al. (2010), and Ritter et al. (2013). Open IE systems include\nKNOW ITALLEtzioni et al. (2005), TextRunner (Banko et al., 2007), and R EVERB\n(Fader et al., 2011). See Riedel et al. (2013) for a universal schema that combines\nthe advantages of distant supervision and Open IE.\n1www.nist.gov/speech/tests/ace/ 26 CHAPTER 20 \u2022 I NFORMATION EXTRACTION : RELATIONS , EVENTS ,AND TIME\nExercises\n20.1 Acronym expansion, the process of associating a phrase with an acronym, can\nbe accomplished by a simple form of relational analysis. Develop a system\nbased on the relation analysis approaches described in this chapter to populate\na database of acronym expansions. If you focus on English Three Letter\nAcronyms (TLAs) you can evaluate your system\u2019s performance by comparing\nit to Wikipedia\u2019s TLA page.\n20.2 Acquire the CMU seminar corpus and develop a template-\ufb01lling system by\nusing any of the techniques mentioned in Section 20.8. Analyze how well\nyour system performs as compared with state-of-the-art results on this corpus.\n20.3 A useful functionality in newer email and calendar applications is the ability\nto associate temporal expressions connected with events in email (doctor\u2019s\nappointments, meeting planning, party invitations, etc.) with speci\ufb01c calendar\nentries. Collect a corpus of email containing temporal expressions related to\nevent planning. How do these expressions compare to the kinds of expressions\ncommonly found in news text that we\u2019ve been discussing in this chapter?\n20.4 For the following sentences, give FOL translations that capture the temporal\nrelationships between the events.\n1. When Mary\u2019s \ufb02ight departed, I ate lunch.\n2. When Mary\u2019s \ufb02ight departed, I had eaten lunch. Exercises 27\nAgichtein, E. and L. Gravano. 2000. Snowball: Extracting\nrelations from large plain-text collections. Proceedings\nof the 5th ACM International Conference on Digital Li-\nbraries .\nAllen, J. 1984. Towards a general theory of action and time.\nArti\ufb01cial Intelligence , 23(2):123\u2013154.\nBanko, M., M. Cafarella, S. Soderland, M. Broadhead, and\nO. Etzioni. 2007. Open information extraction for the\nweb. IJCAI .\nBethard, S. 2013. ClearTK-TimeML: A minimalist approach\nto TempEval 2013. SemEval-13 .\nBikel, D. M., S. Miller, R. Schwartz, and R. Weischedel.\n1997. Nymble: A high-performance learning name-\n\ufb01nder. ANLP .\nBizer, C., J. Lehmann, G. Kobilarov, S. Auer, C. Becker,\nR. Cyganiak, and S. Hellmann. 2009. DBpedia\u2014A crys-\ntallization point for the Web of Data. Web Semantics:\nscience, services and agents on the world wide web ,\n7(3):154\u2013165.\nBollacker, K., C. Evans, P. Paritosh, T. Sturge, and J. Taylor.\n2008. Freebase: a collaboratively created graph database\nfor structuring human knowledge. SIGMOD 2008 .\nBrin, S. 1998. Extracting patterns and relations from\nthe World Wide Web. Proceedings World Wide Web\nand Databases International Workshop, Number 1590 in\nLNCS . Springer.\nCardie, C. 1993. A case-based approach to knowledge ac-\nquisition for domain speci\ufb01c sentence analysis. AAAI .\nCardie, C. 1994. Domain-Speci\ufb01c Knowledge Acquisition\nfor Conceptual Sentence Analysis . Ph.D. thesis, Univer-\nsity of Massachusetts, Amherst, MA. Available as CMP-\nSCI Technical Report 94-74.\nChambers, N. 2013. NavyTime: Event and time ordering\nfrom raw text. SemEval-13 .\nChambers, N., T. Cassidy, B. McDowell, and S. Bethard.\n2014. Dense event ordering with a multi-pass architec-\nture. TACL , 2:273\u2013284.\nChambers, N. and D. Jurafsky. 2011. Template-based infor-\nmation extraction without the templates. ACL.\nChang, A. X. and C. D. Manning. 2012. SUTime: A library\nfor recognizing and normalizing time expressions. LREC .\nChinchor, N., L. Hirschman, and D. L. Lewis. 1993. Eval-\nuating Message Understanding systems: An analysis of\nthe third Message Understanding Conference. Computa-\ntional Linguistics , 19(3):409\u2013449.\nCollobert, R., J. Weston, L. Bottou, M. Karlen,\nK. Kavukcuoglu, and P. Kuksa. 2011. Natural language\nprocessing (almost) from scratch. JMLR , 12:2493\u20132537.\nCraven, M. and J. Kumlien. 1999. Constructing biologi-\ncal knowledge bases by extracting information from text\nsources. ISMB-99 .\nDeJong, G. F. 1982. An overview of the FRUMP system.\nIn W. G. Lehnert and M. H. Ringle, eds, Strategies for\nNatural Language Processing , 149\u2013176. LEA.\nEtzioni, O., M. Cafarella, D. Downey, A.-M. Popescu,\nT. Shaked, S. Soderland, D. S. Weld, and A. Yates. 2005.\nUnsupervised named-entity extraction from the web: An\nexperimental study. Arti\ufb01cial Intelligence , 165(1):91\u2013\n134.Fader, A., S. Soderland, and O. Etzioni. 2011. Identifying\nrelations for open information extraction. EMNLP .\nFerro, L., L. Gerber, I. Mani, B. Sundheim, and G. Wilson.\n2005. Tides 2005 standard for the annotation of temporal\nexpressions. Technical report, MITRE.\nGrishman, R. and B. Sundheim. 1995. Design of the MUC-6\nevaluation. MUC-6 .\nHearst, M. A. 1992a. Automatic acquisition of hyponyms\nfrom large text corpora. COLING .\nHearst, M. A. 1992b. Automatic acquisition of hyponyms\nfrom large text corpora. COLING .\nHearst, M. A. 1998. Automatic discovery of WordNet rela-\ntions. In C. Fellbaum, ed., WordNet: An Electronic Lexi-\ncal Database . MIT Press.\nHendrickx, I., S. N. Kim, Z. Kozareva, P. Nakov,\nD.\u00b4O S\u00b4eaghdha, S. Pad \u00b4o, M. Pennacchiotti, L. Romano,\nand S. Szpakowicz. 2009. Semeval-2010 task 8: Multi-\nway classi\ufb01cation of semantic relations between pairs of\nnominals. 5th International Workshop on Semantic Eval-\nuation .\nHobbs, J. R., D. E. Appelt, J. Bear, D. Israel, M. Kameyama,\nM. E. Stickel, and M. Tyson. 1997. FASTUS: A cas-\ncaded \ufb01nite-state transducer for extracting information\nfrom natural-language text. In E. Roche and Y . Sch-\nabes, eds, Finite-State Language Processing , 383\u2013406.\nMIT Press.\nHuffman, S. 1996. Learning information extraction pat-\nterns from examples. In S. Wertmer, E. Riloff, and\nG. Scheller, eds, Connectionist, Statistical, and Symbolic\nApproaches to Learning Natural Language Processing ,\n246\u2013260. Springer.\nISO8601. 2004. Data elements and interchange formats\u2014\ninformation interchange\u2014representation of dates and\ntimes. Technical report, International Organization for\nStandards (ISO).\nJackendoff, R. 1983. Semantics and Cognition . MIT Press.\nJacobs, P. S. and L. F. Rau. 1990. SCISOR: A system\nfor extracting information from on-line news. CACM ,\n33(11):88\u201397.\nJi, H., R. Grishman, and H. T. Dang. 2010. Overview of the\ntac 2011 knowledge base population track. TAC-11 .\nJones, R., A. McCallum, K. Nigam, and E. Riloff. 1999.\nBootstrapping for text learning tasks. IJCAI-99 Workshop\non Text Mining: Foundations, Techniques and Applica-\ntions .\nJoshi, M., D. Chen, Y . Liu, D. S. Weld, L. Zettlemoyer, and\nO. Levy. 2020. SpanBERT: Improving pre-training by\nrepresenting and predicting spans. TACL , 8:64\u201377.\nLafferty, J. D., A. McCallum, and F. C. N. Pereira. 2001.\nConditional random \ufb01elds: Probabilistic models for seg-\nmenting and labeling sequence data. ICML .\nLakoff, G. and M. Johnson. 1980. Metaphors We Live By .\nUniversity of Chicago Press, Chicago, IL.\nLehnert, W. G., C. Cardie, D. Fisher, E. Riloff, and\nR. Williams. 1991. Description of the CIRCUS system\nas used for MUC-3. MUC-3 .\nLiu, Y ., M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen,\nO. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov.\n2019. RoBERTa: A robustly optimized BERT pretraining\napproach. ArXiv preprint arXiv:1907.11692. 28 Chapter 20 \u2022 Information Extraction: Relations, Events, and Time\nMcCallum, A., D. Freitag, and F. C. N. Pereira. 2000. Max-\nimum entropy Markov models for information extraction\nand segmentation. ICML .\nMintz, M., S. Bills, R. Snow, and D. Jurafsky. 2009. Distant\nsupervision for relation extraction without labeled data.\nACL IJCNLP .\nMirza, P. and S. Tonelli. 2016. CATENA: CAusal and TEm-\nporal relation extraction from NAtural language texts.\nCOLING .\nMorgan, A. A., L. Hirschman, M. Colosimo, A. S. Yeh, and\nJ. B. Colombe. 2004. Gene name identi\ufb01cation and nor-\nmalization using a model organism database. Journal of\nBiomedical Informatics , 37(6):396\u2013410.\nOcal, M., A. Perez, A. Radas, and M. Finlayson. 2022.\nHolistic evaluation of automatic TimeML annotators.\nLREC .\nPustejovsky, J., P. Hanks, R. Saur \u00b4\u0131, A. See, R. Gaizauskas,\nA. Setzer, D. Radev, B. Sundheim, D. S. Day, L. Ferro,\nand M. Lazo. 2003. The TIMEBANK corpus. Proceed-\nings of Corpus Linguistics 2003 Conference . UCREL\nTechnical Paper number 16.\nPustejovsky, J., R. Ingria, R. Saur \u00b4\u0131, J. Casta \u02dcno, J. Littman,\nR. Gaizauskas, A. Setzer, G. Katz, and I. Mani. 2005. The\nSpeci\ufb01cation Language TimeML , chapter 27. Oxford.\nReichenbach, H. 1947. Elements of Symbolic Logic . Macmil-\nlan, New York.\nRiedel, S., L. Yao, and A. McCallum. 2010. Modeling rela-\ntions and their mentions without labeled text. In Machine\nLearning and Knowledge Discovery in Databases , 148\u2013\n163. Springer.\nRiedel, S., L. Yao, A. McCallum, and B. M. Marlin. 2013.\nRelation extraction with matrix factorization and univer-\nsal schemas. NAACL HLT .\nRiloff, E. 1993. Automatically constructing a dictionary for\ninformation extraction tasks. AAAI .\nRiloff, E. 1996. Automatically generating extraction patterns\nfrom untagged text. AAAI .\nRiloff, E. and R. Jones. 1999. Learning dictionaries for in-\nformation extraction by multi-level bootstrapping. AAAI .\nRitter, A., L. Zettlemoyer, Mausam, and O. Etzioni. 2013.\nModeling missing data in distant supervision for infor-\nmation extraction. TACL , 1:367\u2013378.\nSaur\u00b4\u0131, R., J. Littman, B. Knippen, R. Gaizauskas, A. Setzer,\nand J. Pustejovsky. 2006. TimeML annotation guidelines\nversion 1.2.1. Manuscript.\nSchank, R. C. and R. P. Abelson. 1977. Scripts, Plans, Goals\nand Understanding . Lawrence Erlbaum.\nSnow, R., D. Jurafsky, and A. Y . Ng. 2005. Learning syntac-\ntic patterns for automatic hypernym discovery. NeurIPS .\nSoderland, S., D. Fisher, J. Aseltine, and W. G. Lehn-\nert. 1995. CRYSTAL: Inducing a conceptual dictionary.\nIJCAI-95 .\nStr\u00a8otgen, J. and M. Gertz. 2013. Multilingual and cross-\ndomain temporal tagging. Language Resources and Eval-\nuation , 47(2):269\u2013298.\nSundheim, B., ed. 1991. Proceedings of MUC-3 .\nSundheim, B., ed. 1992. Proceedings of MUC-4 .Sundheim, B., ed. 1993. Proceedings of MUC-5 . Baltimore,\nMD.\nSundheim, B., ed. 1995. Proceedings of MUC-6 .\nSurdeanu, M. 2013. Overview of the TAC2013 Knowledge\nBase Population evaluation: English slot \ufb01lling and tem-\nporal slot \ufb01lling. TAC-13 .\nVendler, Z. 1967. Linguistics in Philosophy . Cornell Univer-\nsity Press.\nVerhagen, M., R. Gaizauskas, F. Schilder, M. Hepple,\nJ. Moszkowicz, and J. Pustejovsky. 2009. The TempE-\nval challenge: Identifying temporal relations in text. Lan-\nguage Resources and Evaluation , 43(2):161\u2013179.\nVerhagen, M., I. Mani, R. Sauri, R. Knippen, S. B. Jang,\nJ. Littman, A. Rumshisky, J. Phillips, and J. Pustejovsky.\n2005. Automating temporal annotation with TARSQI.\nACL.\nVrande \u02c7ci\u00b4c, D. and M. Kr \u00a8otzsch. 2014. Wikidata: a free col-\nlaborative knowledge base. CACM , 57(10):78\u201385.\nWu, F. and D. S. Weld. 2007. Autonomously semantifying\nWikipedia. CIKM-07 .\nWu, F. and D. S. Weld. 2010. Open information extraction\nusing Wikipedia. ACL.\nZhang, Y ., V . Zhong, D. Chen, G. Angeli, and C. D. Man-\nning. 2017. Position-aware attention and supervised data\nimprove slot \ufb01lling. EMNLP .\nZhou, G., J. Su, J. Zhang, and M. Zhang. 2005. Exploring\nvarious knowledge in relation extraction. ACL.", "21.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\n21Semantic Role Labeling\n\u201cWho, What, Where, When, With what, Why, How\u201d\nThe seven circumstances, associated with Hermagoras and Aristotle (Sloan, 2010)\nSometime between the 7th and 4th centuries BCE, the Indian grammarian P \u00afan.ini1\nwrote a famous treatise on Sanskrit grammar, the As .t.\u00afadhy \u00afay\u00af\u0131 (\u20188 books\u2019), a treatise\nthat has been called \u201cone of the greatest monuments of hu-\nman intelligence\u201d (Bloom\ufb01eld, 1933, 11). The work de-\nscribes the linguistics of the Sanskrit language in the form\nof 3959 sutras, each very ef\ufb01ciently (since it had to be\nmemorized!) expressing part of a formal rule system that\nbrilliantly pre\ufb01gured modern mechanisms of formal lan-\nguage theory (Penn and Kiparsky, 2012). One set of rules\ndescribes the k\u00afarakas , semantic relationships between a\nverb and noun arguments, roles like agent ,instrument , or\ndestination . P\u00afan.ini\u2019s work was the earliest we know of\nthat modeled the linguistic realization of events and their\nparticipants. This task of understanding how participants relate to events\u2014being\nable to answer the question \u201cWho did what to whom\u201d (and perhaps also \u201cwhen and\nwhere\u201d)\u2014is a central question of natural language processing.\nLet\u2019s move forward 2.5 millennia to the present and consider the very mundane\ngoal of understanding text about a purchase of stock by XYZ Corporation. This\npurchasing event and its participants can be described by a wide variety of surface\nforms. The event can be described by a verb ( sold, bought ) or a noun ( purchase ),\nand XYZ Corp can be the syntactic subject (of bought ), the indirect object (of sold),\nor in a genitive or noun compound relation (with the noun purchase ) despite having\nnotionally the same role in all of them:\n\u2022 XYZ corporation bought the stock.\n\u2022 They sold the stock to XYZ corporation.\n\u2022 The stock was bought by XYZ corporation.\n\u2022 The purchase of the stock by XYZ corporation...\n\u2022 The stock purchase by XYZ corporation...\nIn this chapter we introduce a level of representation that captures the common-\nality between these sentences: there was a purchase event, the participants were\nXYZ Corp and some stock, and XYZ Corp was the buyer. These shallow semantic\nrepresentations , semantic roles , express the role that arguments of a predicate take\nin the event, codi\ufb01ed in databases like PropBank and FrameNet. We\u2019ll introduce\nsemantic role labeling , the task of assigning roles to spans in sentences, and selec-\ntional restrictions , the preferences that predicates express about their arguments,\nsuch as the fact that the theme of eatis generally something edible.\n1Figure shows a birch bark manuscript from Kashmir of the Rupavatra, a grammatical textbook based\non the Sanskrit grammar of Panini. Image from the Wellcome Collection. 2CHAPTER 21 \u2022 S EMANTIC ROLE LABELING\n21.1 Semantic Roles\nConsider the meanings of the arguments Sasha ,Pat,the window , and the door in\nthese two sentences.\n(21.1) Sasha broke the window.\n(21.2) Pat opened the door.\nThe subjects Sasha andPat, what we might call the breaker of the window-\nbreaking event and the opener of the door-opening event, have something in com-\nmon. They are both volitional actors, often animate, and they have direct causal\nresponsibility for their events.\nThematic roles are a way to capture this semantic commonality between break- thematic roles\nersandopeners . We say that the subjects of both these verbs are agents . Thus, agents\nAGENT is the thematic role that represents an abstract idea such as volitional causa-\ntion. Similarly, the direct objects of both these verbs, the BrokenThing andOpenedThing ,\nare both prototypically inanimate objects that are affected in some way by the action.\nThe semantic role for these participants is theme . theme\nThematic Role De\ufb01nition\nAGENT The volitional causer of an event\nEXPERIENCER The experiencer of an event\nFORCE The non-volitional causer of the event\nTHEME The participant most directly affected by an event\nRESULT The end product of an event\nCONTENT The proposition or content of a propositional event\nINSTRUMENT An instrument used in an event\nBENEFICIARY The bene\ufb01ciary of an event\nSOURCE The origin of the object of a transfer event\nGOAL The destination of an object of a transfer event\nFigure 21.1 Some commonly used thematic roles with their de\ufb01nitions.\nAlthough thematic roles are one of the oldest linguistic models, as we saw above,\ntheir modern formulation is due to Fillmore (1968) and Gruber (1965). Although\nthere is no universally agreed-upon set of roles, Figs. 21.1 and 21.2 list some the-\nmatic roles that have been used in various computational papers, together with rough\nde\ufb01nitions and examples. Most thematic role sets have about a dozen roles, but we\u2019ll\nsee sets with smaller numbers of roles with even more abstract meanings, and sets\nwith very large numbers of roles that are speci\ufb01c to situations. We\u2019ll use the general\nterm semantic roles for all sets of roles, whether small or large. semantic roles\n21.2 Diathesis Alternations\nThe main reason computational systems use semantic roles is to act as a shallow\nmeaning representation that can let us make simple inferences that aren\u2019t possible\nfrom the pure surface string of words, or even from the parse tree. To extend the\nearlier examples, if a document says that Company A acquired Company B , we\u2019d\nlike to know that this answers the query Was Company B acquired? despite the fact\nthat the two sentences have very different surface syntax. Similarly, this shallow\nsemantics might act as a useful intermediate language in machine translation. 21.2 \u2022 D IATHESIS ALTERNATIONS 3\nThematic Role Example\nAGENT The waiter spilled the soup.\nEXPERIENCER John has a headache.\nFORCE The wind blows debris from the mall into our yards.\nTHEME Only after Benjamin Franklin broke the ice ...\nRESULT The city built a regulation-size baseball diamond ...\nCONTENT Mona asked \u201cYou met Mary Ann at a supermarket?\u201d\nINSTRUMENT He poached cat\ufb01sh, stunning them with a shocking device ...\nBENEFICIARY Whenever Ann Callahan makes hotel reservations for her boss ...\nSOURCE I \ufb02ew in from Boston .\nGOAL I drove to Portland .\nFigure 21.2 Some prototypical examples of various thematic roles.\nSemantic roles thus help generalize over different surface realizations of pred-\nicate arguments. For example, while the AGENT is often realized as the subject of\nthe sentence, in other cases the THEME can be the subject. Consider these possible\nrealizations of the thematic arguments of the verb break :\n(21.3) John\nAGENTbroke the window.\nTHEME\n(21.4) John\nAGENTbroke the window\nTHEMEwith a rock.\nINSTRUMENT\n(21.5) The rock\nINSTRUMENTbroke the window.\nTHEME\n(21.6) The window\nTHEMEbroke.\n(21.7) The window\nTHEMEwas broken by John.\nAGENT\nThese examples suggest that break has (at least) the possible arguments AGENT ,\nTHEME , and INSTRUMENT . The set of thematic role arguments taken by a verb is\noften called the thematic grid ,q-grid, or case frame . We can see that there are thematic grid\ncase frame (among others) the following possibilities for the realization of these arguments of\nbreak :\nAGENT /Subject, THEME /Object\nAGENT /Subject, THEME /Object, INSTRUMENT /PPwith\nINSTRUMENT /Subject, THEME /Object\nTHEME /Subject\nIt turns out that many verbs allow their thematic roles to be realized in various\nsyntactic positions. For example, verbs like give can realize the THEME and GOAL\narguments in two different ways:\n(21.8) a. Doris\nAGENTgave the book\nTHEMEto Cary.\nGOAL\nb.Doris\nAGENTgave Cary\nGOALthe book.\nTHEME\nThese multiple argument structure realizations (the fact that break can take AGENT ,\nINSTRUMENT , or THEME as subject, and give can realize its THEME and GOAL in\neither order) are called verb alternations ordiathesis alternations . The alternationverb\nalternation\nwe showed above for give, the dative alternation , seems to occur with particular se-dative\nalternation\nmantic classes of verbs, including \u201cverbs of future having\u201d ( advance ,allocate ,offer, 4CHAPTER 21 \u2022 S EMANTIC ROLE LABELING\nowe), \u201csend verbs\u201d ( forward ,hand ,mail), \u201cverbs of throwing\u201d ( kick,pass,throw ),\nand so on. Levin (1993) lists for 3100 English verbs the semantic classes to which\nthey belong (47 high-level classes, divided into 193 more speci\ufb01c classes) and the\nvarious alternations in which they participate. These lists of verb classes have been\nincorporated into the online resource VerbNet (Kipper et al., 2000), which links each\nverb to both WordNet and FrameNet entries.\n21.3 Semantic Roles: Problems with Thematic Roles\nRepresenting meaning at the thematic role level seems like it should be useful in\ndealing with complications like diathesis alternations. Yet it has proved quite dif\ufb01-\ncult to come up with a standard set of roles, and equally dif\ufb01cult to produce a formal\nde\ufb01nition of roles like AGENT ,THEME , or INSTRUMENT .\nFor example, researchers attempting to de\ufb01ne role sets often \ufb01nd they need to\nfragment a role like AGENT orTHEME into many speci\ufb01c roles. Levin and Rappa-\nport Hovav (2005) summarize a number of such cases, such as the fact there seem\nto be at least two kinds of INSTRUMENTS ,intermediary instruments that can appear\nas subjects and enabling instruments that cannot:\n(21.9) a. Shelly cut the banana with a knife.\nb. The knife cut the banana.\n(21.10) a. Shelly ate the sliced banana with a fork.\nb. *The fork ate the sliced banana.\nIn addition to the fragmentation problem, there are cases in which we\u2019d like to\nreason about and generalize across semantic roles, but the \ufb01nite discrete lists of roles\ndon\u2019t let us do this.\nFinally, it has proved dif\ufb01cult to formally de\ufb01ne the thematic roles. Consider the\nAGENT role; most cases of AGENTS are animate, volitional, sentient, causal, but any\nindividual noun phrase might not exhibit all of these properties.\nThese problems have led to alternative semantic role models that use either semantic role\nmany fewer or many more roles.\nThe \ufb01rst of these options is to de\ufb01ne generalized semantic roles that abstract\nover the speci\ufb01c thematic roles. For example, PROTO -AGENT and PROTO -PATIENT proto-agent\nproto-patient are generalized roles that express roughly agent-like and roughly patient-like mean-\nings. These roles are de\ufb01ned, not by necessary and suf\ufb01cient conditions, but rather\nby a set of heuristic features that accompany more agent-like or more patient-like\nmeanings. Thus, the more an argument displays agent-like properties (being voli-\ntionally involved in the event, causing an event or a change of state in another par-\nticipant, being sentient or intentionally involved, moving) the greater the likelihood\nthat the argument can be labeled a PROTO -AGENT . The more patient-like the proper-\nties (undergoing change of state, causally affected by another participant, stationary\nrelative to other participants, etc.), the greater the likelihood that the argument can\nbe labeled a PROTO -PATIENT .\nThe second direction is instead to de\ufb01ne semantic roles that are speci\ufb01c to a\nparticular verb or a particular group of semantically related verbs or nouns.\nIn the next two sections we describe two commonly used lexical resources that\nmake use of these alternative versions of semantic roles. PropBank uses both proto-\nroles and verb-speci\ufb01c semantic roles. FrameNet uses semantic roles that are spe-\nci\ufb01c to a general semantic idea called a frame . 21.4 \u2022 T HEPROPOSITION BANK 5\n21.4 The Proposition Bank\nThe Proposition Bank , generally referred to as PropBank , is a resource of sen- PropBank\ntences annotated with semantic roles. The English PropBank labels all the sentences\nin the Penn TreeBank; the Chinese PropBank labels sentences in the Penn Chinese\nTreeBank. Because of the dif\ufb01culty of de\ufb01ning a universal set of thematic roles,\nthe semantic roles in PropBank are de\ufb01ned with respect to an individual verb sense.\nEach sense of each verb thus has a speci\ufb01c set of roles, which are given only numbers\nrather than names: Arg0 ,Arg1 ,Arg2 , and so on. In general, Arg0 represents the\nPROTO -AGENT , and Arg1 , the PROTO -PATIENT . The semantics of the other roles\nare less consistent, often being de\ufb01ned speci\ufb01cally for each verb. Nonetheless there\nare some generalization; the Arg2 is often the benefactive, instrument, attribute, or\nend state, the Arg3 the start point, benefactive, instrument, or attribute, and the Arg4\nthe end point.\nHere are some slightly simpli\ufb01ed PropBank entries for one sense each of the\nverbs agree andfall. Such PropBank entries are called frame \ufb01les ; note that the\nde\ufb01nitions in the frame \ufb01le for each role (\u201cOther entity agreeing\u201d, \u201cExtent, amount\nfallen\u201d) are informal glosses intended to be read by humans, rather than being formal\nde\ufb01nitions.\n(21.11) agree.01\nArg0: Agreer\nArg1: Proposition\nArg2: Other entity agreeing\nEx1: [ Arg0 The group] agreed [Arg1 it wouldn\u2019t make an offer].\nEx2: [ ArgM-TMP Usually] [ Arg0 John] agrees [Arg2 with Mary]\n[Arg1 on everything].\n(21.12) fall.01\nArg1: Logical subject, patient, thing falling\nArg2: Extent, amount fallen\nArg3: start point\nArg4: end point, end state of arg1\nEx1: [ Arg1 Sales] fell[Arg4 to $25 million] [ Arg3 from $27 million].\nEx2: [ Arg1 The average junk bond] fell[Arg2 by 4.2%].\nNote that there is no Arg0 role for fall, because the normal subject of fallis a\nPROTO -PATIENT .\nThe PropBank semantic roles can be useful in recovering shallow semantic in-\nformation about verbal arguments. Consider the verb increase :\n(21.13) increase.01 \u201cgo up incrementally\u201d\nArg0: causer of increase\nArg1: thing increasing\nArg2: amount increased by, EXT, or MNR\nArg3: start point\nArg4: end point\nA PropBank semantic role labeling would allow us to infer the commonality in\nthe event structures of the following three examples, that is, that in each case Big\nFruit Co. is the AGENT andthe price of bananas is the THEME , despite the differing\nsurface forms. 6CHAPTER 21 \u2022 S EMANTIC ROLE LABELING\n(21.14) [ Arg0 Big Fruit Co. ] increased [ Arg1 the price of bananas].\n(21.15) [ Arg1 The price of bananas] was increased again [ Arg0 by Big Fruit Co. ]\n(21.16) [ Arg1 The price of bananas] increased [ Arg2 5%].\nPropBank also has a number of non-numbered arguments called ArgMs , (ArgM-\nTMP, ArgM-LOC, etc.) which represent modi\ufb01cation or adjunct meanings. These\nare relatively stable across predicates, so aren\u2019t listed with each frame \ufb01le. Data\nlabeled with these modi\ufb01ers can be helpful in training systems to detect temporal,\nlocation, or directional modi\ufb01cation across predicates. Some of the ArgM\u2019s include:\nTMP when? yesterday evening, now\nLOC where? at the museum, in San Francisco\nDIR where to/from? down, to Bangkok\nMNR how? clearly, with much enthusiasm\nPRP/CAU why? because ... , in response to the ruling\nREC themselves, each other\nADV miscellaneous\nPRD secondary predication ...ate the meat raw\nWhile PropBank focuses on verbs, a related project, NomBank (Meyers et al., NomBank\n2004) adds annotations to noun predicates. For example the noun agreement in\nApple\u2019s agreement with IBM would be labeled with Apple as the Arg0 and IBM as\nthe Arg2. This allows semantic role labelers to assign labels to arguments of both\nverbal and nominal predicates.\n21.5 FrameNet\nWhile making inferences about the semantic commonalities across different sen-\ntences with increase is useful, it would be even more useful if we could make such\ninferences in many more situations, across different verbs, and also between verbs\nand nouns. For example, we\u2019d like to extract the similarity among these three sen-\ntences:\n(21.17) [ Arg1 The price of bananas] increased [ Arg2 5%].\n(21.18) [ Arg1 The price of bananas] rose [ Arg2 5%].\n(21.19) There has been a [ Arg2 5%] rise [ Arg1 in the price of bananas].\nNote that the second example uses the different verb rise, and the third example\nuses the noun rather than the verb rise. We\u2019d like a system to recognize that the\nprice of bananas is what went up, and that 5%is the amount it went up, no matter\nwhether the 5%appears as the object of the verb increased or as a nominal modi\ufb01er\nof the noun rise.\nThe FrameNet project is another semantic-role-labeling project that attempts FrameNet\nto address just these kinds of problems (Baker et al. 1998, Fillmore et al. 2003,\nFillmore and Baker 2009, Ruppenhofer et al. 2016). Whereas roles in the PropBank\nproject are speci\ufb01c to an individual verb, roles in the FrameNet project are speci\ufb01c\nto aframe .\nWhat is a frame? Consider the following set of words:\nreservation, \ufb02ight, travel, buy, price, cost, fare, rates, meal, plane\nThere are many individual lexical relations of hyponymy, synonymy, and so on\nbetween many of the words in this list. The resulting set of relations does not, 21.5 \u2022 F RAME NET 7\nhowever, add up to a complete account of how these words are related. They are\nclearly all de\ufb01ned with respect to a coherent chunk of common-sense background\ninformation concerning air travel.\nWe call the holistic background knowledge that unites these words a frame (Fill- frame\nmore, 1985). The idea that groups of words are de\ufb01ned with respect to some back-\nground information is widespread in arti\ufb01cial intelligence and cognitive science,\nwhere besides frame we see related works like a model (Johnson-Laird, 1983), or model\neven script (Schank and Abelson, 1977). script\nA frame in FrameNet is a background knowledge structure that de\ufb01nes a set of\nframe-speci\ufb01c semantic roles, called frame elements , and includes a set of predi- frame elements\ncates that use these roles. Each word evokes a frame and pro\ufb01les some aspect of the\nframe and its elements. The FrameNet dataset includes a set of frames and frame\nelements, the lexical units associated with each frame, and a set of labeled exam-\nple sentences. For example, the change position onascale frame is de\ufb01ned as\nfollows:\nThis frame consists of words that indicate the change of an Item\u2019s posi-\ntion on a scale (the Attribute) from a starting point (Initial value) to an\nend point (Final value).\nSome of the semantic roles (frame elements) in the frame are de\ufb01ned as in\nFig. 21.3. Note that these are separated into core roles , which are frame speci\ufb01c, and core roles\nnon-core roles , which are more like the Arg-M arguments in PropBank, expressing non-core roles\nmore general properties of time, location, and so on.\nCore Roles\nATTRIBUTE The A TTRIBUTE is a scalar property that the I TEM possesses.\nDIFFERENCE The distance by which an I TEM changes its position on the scale.\nFINAL STATE A description that presents the I TEM\u2019s state after the change in the A TTRIBUTE \u2019s\nvalue as an independent predication.\nFINAL VALUE The position on the scale where the I TEM ends up.\nINITIAL STATE A description that presents the I TEM\u2019s state before the change in the A T-\nTRIBUTE \u2019s value as an independent predication.\nINITIAL VALUE The initial position on the scale from which the I TEM moves away.\nITEM The entity that has a position on the scale.\nVALUE RANGE A portion of the scale, typically identi\ufb01ed by its end points, along which the\nvalues of the A TTRIBUTE \ufb02uctuate.\nSome Non-Core Roles\nDURATION The length of time over which the change takes place.\nSPEED The rate of change of the V ALUE .\nGROUP The G ROUP in which an I TEM changes the value of an\nATTRIBUTE in a speci\ufb01ed way.\nFigure 21.3 The frame elements in the change position onascale frame from the FrameNet Labelers\nGuide (Ruppenhofer et al., 2016).\nHere are some example sentences:\n(21.20) [ ITEM Oil]rose [ATTRIBUTE in price] [ DIFFERENCE by 2%].\n(21.21) [ ITEM It] has increased [FINAL STATE to having them 1 day a month].\n(21.22) [ ITEM Microsoft shares] fell[FINAL VALUE to 7 5/8].\n(21.23) [ ITEM Colon cancer incidence] fell[DIFFERENCE by 50%] [ GROUP among\nmen]. 8CHAPTER 21 \u2022 S EMANTIC ROLE LABELING\n(21.24) a steady increase [INITIAL VALUE from 9.5] [ FINAL VALUE to 14.3] [ ITEM\nin dividends]\n(21.25) a [ DIFFERENCE 5%] [ ITEM dividend] increase ...\nNote from these example sentences that the frame includes target words like rise,\nfall, and increase . In fact, the complete frame consists of the following words:\nVERBS: dwindle move soar escalation shift\nadvance edge mushroom swell explosion tumble\nclimb explode plummet swing fall\ndecline fall reach triple \ufb02uctuation ADVERBS:\ndecrease \ufb02uctuate rise tumble gain increasingly\ndiminish gain rocket growth\ndip grow shift NOUNS: hike\ndouble increase skyrocket decline increase\ndrop jump slide decrease rise\nFrameNet also codes relationships between frames, allowing frames to inherit\nfrom each other, or representing relations between frames like causation (and gen-\neralizations among frame elements in different frames can be represented by inheri-\ntance as well). Thus, there is a Cause change ofposition onascale frame that is\nlinked to the Change ofposition onascale frame by the cause relation, but that\nadds an A GENT role and is used for causative examples such as the following:\n(21.26) [ AGENT They] raised [ITEM the price of their soda] [ DIFFERENCE by 2%].\nTogether, these two frames would allow an understanding system to extract the\ncommon event semantics of all the verbal and nominal causative and non-causative\nusages.\nFrameNets have also been developed for many other languages including Span-\nish, German, Japanese, Portuguese, Italian, and Chinese.\n21.6 Semantic Role Labeling\nSemantic role labeling (sometimes shortened as SRL ) is the task of automaticallysemantic role\nlabeling\n\ufb01nding the semantic roles of each argument of each predicate in a sentence. Cur-\nrent approaches to semantic role labeling are based on supervised machine learning,\noften using the FrameNet and PropBank resources to specify what counts as a pred-\nicate, de\ufb01ne the set of roles used in the task, and provide training and test sets.\nRecall that the difference between these two models of semantic roles is that\nFrameNet (21.27) employs many frame-speci\ufb01c frame elements as roles, while Prop-\nBank (21.28) uses a smaller number of numbered argument labels that can be inter-\npreted as verb-speci\ufb01c labels, along with the more general ARGM labels. Some\nexamples:\n(21.27)[You] can\u2019t [blame] [the program] [for being unable to identify it]\nCOGNIZER TARGET EVALUEE REASON\n(21.28)[The San Francisco Examiner] issued [a special edition] [yesterday]\nARG0 TARGET ARG 1 ARGM -TMP\n21.6.1 A Feature-based Algorithm for Semantic Role Labeling\nA simpli\ufb01ed feature-based semantic role labeling algorithm is sketched in Fig. 21.4.\nFeature-based algorithms\u2014from the very earliest systems like (Simmons, 1973)\u2014\nbegin by parsing, using broad-coverage parsers to assign a parse to the input string. 21.6 \u2022 S EMANTIC ROLE LABELING 9\nFigure 21.5 shows a parse of (21.28) above. The parse is then traversed to \ufb01nd all\nwords that are predicates.\nFor each of these predicates, the algorithm examines each node in the parse\ntree and uses supervised classi\ufb01cation to decide the semantic role (if any) it plays\nfor this predicate. Given a labeled training set such as PropBank or FrameNet, a\nfeature vector is extracted for each node, using feature templates described in the\nnext subsection. A 1-of-N classi\ufb01er is then trained to predict a semantic role for\neach constituent given these features, where N is the number of potential semantic\nroles plus an extra NONE role for non-role constituents. Any standard classi\ufb01cation\nalgorithms can be used. Finally, for each test sentence to be labeled, the classi\ufb01er is\nrun on each relevant constituent.\nfunction SEMANTIC ROLELABEL (words )returns labeled tree\nparse PARSE (words )\nfor each predicate inparse do\nfor each node inparse do\nfeaturevector EXTRACT FEATURES (node ,predicate ,parse )\nCLASSIFY NODE(node ,featurevector ,parse )\nFigure 21.4 A generic semantic-role-labeling algorithm. C LASSIFY NODE is a 1-of- Nclas-\nsi\ufb01er that assigns a semantic role (or NONE for non-role constituents), trained on labeled data\nsuch as FrameNet or PropBank.\nS\nNP-SBJ =A R G 0 VP\nDT NNP NNP NNP\nThe San Francisco Examiner\nVBD = TARGET NP=A R G 1 PP-TMP =A R G M - T M P\nissued DT JJ NN IN NP\nas p e c i a l e d i t i o n a r o u n d N N N P - T M P\nnoon yesterday\nFigure 21.5 Parse tree for a PropBank sentence, showing the PropBank argument labels. The dotted line\nshows the path feature NP\"S#VP#VBD for ARG0, the NP-SBJ constituent The San Francisco Examiner.\nInstead of training a single-stage classi\ufb01er as in Fig. 21.5, the node-level classi-\n\ufb01cation task can be broken down into multiple steps:\n1.Pruning: Since only a small number of the constituents in a sentence are\narguments of any given predicate, many systems use simple heuristics to prune\nunlikely constituents.\n2.Identi\ufb01cation: a binary classi\ufb01cation of each node as an argument to be la-\nbeled or a NONE .\n3.Classi\ufb01cation: a 1-of- Nclassi\ufb01cation of all the constituents that were labeled\nas arguments by the previous stage 10 CHAPTER 21 \u2022 S EMANTIC ROLE LABELING\nThe separation of identi\ufb01cation and classi\ufb01cation may lead to better use of fea-\ntures (different features may be useful for the two tasks) or to computational ef\ufb01-\nciency.\nGlobal Optimization\nThe classi\ufb01cation algorithm of Fig. 21.5 classi\ufb01es each argument separately (\u2018lo-\ncally\u2019), making the simplifying assumption that each argument of a predicate can be\nlabeled independently. This assumption is false; there are interactions between argu-\nments that require a more \u2018global\u2019 assignment of labels to constituents. For example,\nconstituents in FrameNet and PropBank are required to be non-overlapping. More\nsigni\ufb01cantly, the semantic roles of constituents are not independent. For example\nPropBank does not allow multiple identical arguments; two constituents of the same\nverb cannot both be labeled ARG0 .\nRole labeling systems thus often add a fourth step to deal with global consistency\nacross the labels in a sentence. For example, the local classi\ufb01ers can return a list of\npossible labels associated with probabilities for each constituent, and a second-pass\nViterbi decoding or re-ranking approach can be used to choose the best consensus\nlabel. Integer linear programming (ILP) is another common way to choose a solution\nthat conforms best to multiple constraints.\nFeatures for Semantic Role Labeling\nMost systems use some generalization of the core set of features introduced by\nGildea and Jurafsky (2000). Common basic features templates (demonstrated on\ntheNP-SBJ constituent The San Francisco Examiner in Fig. 21.5) include:\n\u2022 The governing predicate , in this case the verb issued . The predicate is a cru-\ncial feature since labels are de\ufb01ned only with respect to a particular predicate.\n\u2022 The phrase type of the constituent, in this case, NP(orNP-SBJ ). Some se-\nmantic roles tend to appear as NPs, others as SorPP, and so on.\n\u2022 The headword of the constituent, Examiner . The headword of a constituent\ncan be computed with standard head rules, such as those given in Appendix D\nin Fig. ??. Certain headwords (e.g., pronouns) place strong constraints on the\npossible semantic roles they are likely to \ufb01ll.\n\u2022 The headword part of speech of the constituent, NNP .\n\u2022 The path in the parse tree from the constituent to the predicate. This path is\nmarked by the dotted line in Fig. 21.5. Following Gildea and Jurafsky (2000),\nwe can use a simple linear representation of the path, NP \"S#VP#VBD.\"and\n#represent upward and downward movement in the tree, respectively. The\npath is very useful as a compact representation of many kinds of grammatical\nfunction relationships between the constituent and the predicate.\n\u2022 The voice of the clause in which the constituent appears, in this case, active\n(as contrasted with passive ). Passive sentences tend to have strongly different\nlinkings of semantic roles to surface form than do active ones.\n\u2022 The binary linear position of the constituent with respect to the predicate,\neither before orafter .\n\u2022 The subcategorization of the predicate, the set of expected arguments that\nappear in the verb phrase. We can extract this information by using the phrase-\nstructure rule that expands the immediate parent of the predicate; VP !VBD\nNP PP for the predicate in Fig. 21.5.\n\u2022 The named entity type of the constituent. 21.6 \u2022 S EMANTIC ROLE LABELING 11\n\u2022 The \ufb01rst words and the last word of the constituent.\nThe following feature vector thus represents the \ufb01rst NP in our example (recall\nthat most observations will have the value NONE rather than, for example, ARG0,\nsince most constituents in the parse tree will not bear a semantic role):\nARG0: [issued, NP, Examiner, NNP, NP \"S#VP#VBD, active, before, VP !NP PP,\nORG, The, Examiner]\nOther features are often used in addition, such as sets of n-grams inside the\nconstituent, or more complex versions of the path features (the upward or downward\nhalves, or whether particular nodes occur in the path).\nIt\u2019s also possible to use dependency parses instead of constituency parses as the\nbasis of features, for example using dependency parse paths instead of constituency\npaths.\n21.6.2 A Neural Algorithm for Semantic Role Labeling\nA simple neural approach to SRL is to treat it as a sequence labeling task like named-\nentity recognition, using the BIO approach. Let\u2019s assume that we are given the\npredicate and the task is just detecting and labeling spans. Recall that with BIO\ntagging, we have a begin and end tag for each possible role ( B-ARG0,I-ARG0;B-\nARG1,I-ARG1, and so on), plus an outside tag O.\nENCODER[CLS]thecatslovehats[SEP]love[SEP]FFNB-ARG0I-ARG0B-PREDconcatenate with predicateB-ARG1FFNSoftmaxFFNFFNFFN\nFigure 21.6 A simple neural approach to semantic role labeling. The input sentence is\nfollowed by [SEP] and an extra input for the predicate, in this case love. The encoder outputs\nare concatenated to an indicator variable which is 1 for the predicate and 0 for all other words\nAfter He et al. (2017) and Shi and Lin (2019).\nAs with all the taggers, the goal is to compute the highest probability tag se-\nquence \u02c6 y, given the input sequence of words w:\n\u02c6y=argmax\ny2TP(yjw)\nFig. 21.6 shows a sketch of a standard algorithm from He et al. (2017). Here each\ninput word is mapped to pretrained embeddings, and then each token is concatenated\nwith the predicate embedding and then passed through a feedforward network with\na softmax which outputs a distribution over each SRL label. For decoding, a CRF\nlayer can be used instead of the MLP layer on top of the biLSTM output to do global\ninference, but in practice this doesn\u2019t seem to provide much bene\ufb01t. 12 CHAPTER 21 \u2022 S EMANTIC ROLE LABELING\n21.6.3 Evaluation of Semantic Role Labeling\nThe standard evaluation for semantic role labeling is to require that each argument\nlabel must be assigned to the exactly correct word sequence or parse constituent, and\nthen compute precision, recall, and F-measure. Identi\ufb01cation and classi\ufb01cation can\nalso be evaluated separately. Two common datasets used for evaluation are CoNLL-\n2005 (Carreras and M `arquez, 2005) and CoNLL-2012 (Pradhan et al., 2013).\n21.7 Selectional Restrictions\nWe turn in this section to another way to represent facts about the relationship be-\ntween predicates and arguments. A selectional restriction is a semantic type con-selectional\nrestriction\nstraint that a verb imposes on the kind of concepts that are allowed to \ufb01ll its argument\nroles. Consider the two meanings associated with the following example:\n(21.29) I want to eat someplace nearby.\nThere are two possible parses and semantic interpretations for this sentence. In\nthe sensible interpretation, eatis intransitive and the phrase someplace nearby is\nan adjunct that gives the location of the eating event. In the nonsensical speaker-as-\nGodzilla interpretation, eatis transitive and the phrase someplace nearby is the direct\nobject and the THEME of the eating, like the NP Malaysian food in the following\nsentences:\n(21.30) I want to eat Malaysian food.\nHow do we know that someplace nearby isn\u2019t the direct object in this sentence?\nOne useful cue is the semantic fact that the THEME of E ATING events tends to be\nsomething that is edible . This restriction placed by the verb eaton the \ufb01ller of its\nTHEME argument is a selectional restriction.\nSelectional restrictions are associated with senses, not entire lexemes. We can\nsee this in the following examples of the lexeme serve :\n(21.31) The restaurant serves green-lipped mussels.\n(21.32) Which airlines serve Denver?\nExample (21.31) illustrates the offering-food sense of serve , which ordinarily re-\nstricts its THEME to be some kind of food Example (21.32) illustrates the provides a\ncommercial service to sense of serve , which constrains its THEME to be some type\nof appropriate location.\nSelectional restrictions vary widely in their speci\ufb01city. The verb imagine , for\nexample, imposes strict requirements on its AGENT role (restricting it to humans\nand other animate entities) but places very few semantic requirements on its THEME\nrole. A verb like diagonalize , on the other hand, places a very speci\ufb01c constraint\non the \ufb01ller of its THEME role: it has to be a matrix, while the arguments of the\nadjective odorless are restricted to concepts that could possess an odor:\n(21.33) In rehearsal, I often ask the musicians to imagine a tennis game.\n(21.34) Radon is an odorless gas that can\u2019t be detected by human senses.\n(21.35) To diagonalize a matrix is to \ufb01nd its eigenvalues.\nThese examples illustrate that the set of concepts we need to represent selectional\nrestrictions (being a matrix, being able to possess an odor, etc) is quite open ended.\nThis distinguishes selectional restrictions from other features for representing lexical\nknowledge, like parts-of-speech, which are quite limited in number. 21.7 \u2022 S ELECTIONAL RESTRICTIONS 13\n21.7.1 Representing Selectional Restrictions\nOne way to capture the semantics of selectional restrictions is to use and extend the\nevent representation of Appendix F. Recall that the neo-Davidsonian representation\nof an event consists of a single variable that stands for the event, a predicate denoting\nthe kind of event, and variables and relations for the event roles. Ignoring the issue of\nthel-structures and using thematic roles rather than deep event roles, the semantic\ncontribution of a verb like eatmight look like the following:\n9e;x;y Eating (e)^Agent (e;x)^T heme (e;y)\nWith this representation, all we know about y, the \ufb01ller of the THEME role, is that\nit is associated with an Eating event through the Theme relation. To stipulate the\nselectional restriction that ymust be something edible, we simply add a new term to\nthat effect:\n9e;x;y Eating (e)^Agent (e;x)^T heme (e;y)^EdibleT hing (y)\nWhen a phrase like ate a hamburger is encountered, a semantic analyzer can form\nthe following kind of representation:\n9e;x;y Eating (e)^Eater (e;x)^T heme (e;y)^EdibleT hing (y)^Hamburger (y)\nThis representation is perfectly reasonable since the membership of yin the category\nHamburger is consistent with its membership in the category EdibleThing , assuming\na reasonable set of facts in the knowledge base. Correspondingly, the representation\nfor a phrase such as ate a takeoff would be ill-formed because membership in an\nevent-like category such as Takeoff would be inconsistent with membership in the\ncategory EdibleThing .\nWhile this approach adequately captures the semantics of selectional restrictions,\nthere are two problems with its direct use. First, using FOL to perform the simple\ntask of enforcing selectional restrictions is overkill. Other, far simpler, formalisms\ncan do the job with far less computational cost. The second problem is that this\napproach presupposes a large, logical knowledge base of facts about the concepts\nthat make up selectional restrictions. Unfortunately, although such common-sense\nknowledge bases are being developed, none currently have the kind of coverage\nnecessary to the task.\nA more practical approach is to state selectional restrictions in terms of WordNet\nsynsets rather than as logical concepts. Each predicate simply speci\ufb01es a WordNet\nsynset as the selectional restriction on each of its arguments. A meaning representa-\ntion is well-formed if the role \ufb01ller word is a hyponym (subordinate) of this synset.\nFor our ate a hamburger example, for instance, we could set the selectional\nrestriction on the THEME role of the verb eatto the synset ffood, nutrient g, glossed\nasany substance that can be metabolized by an animal to give energy and build\ntissue . Luckily, the chain of hypernyms for hamburger shown in Fig. 21.7 reveals\nthat hamburgers are indeed food. Again, the \ufb01ller of a role need not match the\nrestriction synset exactly; it just needs to have the synset as one of its superordinates.\nWe can apply this approach to the THEME roles of the verbs imagine ,lift, and di-\nagonalize , discussed earlier. Let us restrict imagine \u2019sTHEME to the synsetfentityg,\nlift\u2019sTHEME tofphysical entityg, and diagonalize tofmatrixg. This arrangement\ncorrectly permits imagine a hamburger andlift a hamburger , while also correctly\nruling out diagonalize a hamburger . 14 CHAPTER 21 \u2022 S EMANTIC ROLE LABELING\nSense 1\nhamburger, beefburger --\n(a fried cake of minced beef served on a bun)\n=> sandwich\n=> snack food\n=> dish\n=> nutriment, nourishment, nutrition...\n=> food, nutrient\n=> substance\n=> matter\n=> physical entity\n=> entity\nFigure 21.7 Evidence from WordNet that hamburgers are edible.\n21.7.2 Selectional Preferences\nIn the earliest implementations, selectional restrictions were considered strict con-\nstraints on the kind of arguments a predicate could take (Katz and Fodor 1963,\nHirst 1987). For example, the verb eatmight require that its THEME argument\nbe[+FOOD] . Early word sense disambiguation systems used this idea to rule out\nsenses that violated the selectional restrictions of their governing predicates.\nVery quickly, however, it became clear that these selectional restrictions were\nbetter represented as preferences rather than strict constraints (Wilks 1975b, Wilks\n1975a). For example, selectional restriction violations (like inedible arguments of\neat) often occur in well-formed sentences, for example because they are negated\n(21.36), or because selectional restrictions are overstated (21.37):\n(21.36) But it fell apart in 1931, perhaps because people realized you can\u2019t eat\ngold for lunch if you\u2019re hungry.\n(21.37) In his two championship trials, Mr. Kulkarni ateglass on an empty\nstomach, accompanied only by water and tea.\nModern systems for selectional preferences therefore specify the relation be-\ntween a predicate and its possible arguments with soft constraints of some kind.\nSelectional Association\nOne of the most in\ufb02uential has been the selectional association model of Resnik\n(1993). Resnik de\ufb01nes the idea of selectional preference strength as the generalselectional\npreference\nstrengthamount of information that a predicate tells us about the semantic class of its argu-\nments. For example, the verb eattells us a lot about the semantic class of its direct\nobjects, since they tend to be edible. The verb be, by contrast, tells us less about\nits direct objects. The selectional preference strength can be de\ufb01ned by the differ-\nence in information between two distributions: the distribution of expected semantic\nclasses P(c)(how likely is it that a direct object will fall into class c) and the dis-\ntribution of expected semantic classes for the particular verb P(cjv)(how likely is\nit that the direct object of the speci\ufb01c verb vwill fall into semantic class c). The\ngreater the difference between these distributions, the more information the verb\nis giving us about possible objects. The difference between these two distributions\ncan be quanti\ufb01ed by relative entropy , or the Kullback-Leibler divergence (Kullback relative entropy\nand Leibler, 1951). The Kullback-Leibler or KL divergence D(PjjQ)expresses the KL divergence 21.7 \u2022 S ELECTIONAL RESTRICTIONS 15\ndifference between two probability distributions PandQ\nD(PjjQ) =X\nxP(x)logP(x)\nQ(x)(21.38)\nThe selectional preference SR(v)uses the KL divergence to express how much in-\nformation, in bits, the verb vexpresses about the possible semantic class of its argu-\nment.\nSR(v) = D(P(cjv)jjP(c))\n=X\ncP(cjv)logP(cjv)\nP(c)(21.39)\nResnik then de\ufb01nes the selectional association of a particular class and verb as theselectional\nassociation\nrelative contribution of that class to the general selectional preference of the verb:\nAR(v;c) =1\nSR(v)P(cjv)logP(cjv)\nP(c)(21.40)\nThe selectional association is thus a probabilistic measure of the strength of asso-\nciation between a predicate and a class dominating the argument to the predicate.\nResnik estimates the probabilities for these associations by parsing a corpus, count-\ning all the times each predicate occurs with each argument word, and assuming\nthat each word is a partial observation of all the WordNet concepts containing the\nword. The following table from Resnik (1996) shows some sample high and low\nselectional associations for verbs and some WordNet semantic classes of their direct\nobjects.\nDirect Object Direct Object\nVerb Semantic Class Assoc Semantic Class Assoc\nread WRITING 6.80 ACTIVITY -.20\nwrite WRITING 7.26 COMMERCE 0\nsee ENTITY 5.79 METHOD -0.01\nSelectional Preference via Conditional Probability\nAn alternative to using selectional association between a verb and the WordNet class\nof its arguments is to use the conditional probability of an argument word given a\npredicate verb, directly modeling the strength of association of one verb (predicate)\nwith one noun (argument).\nThe conditional probability model can be computed by parsing a very large cor-\npus (billions of words), and computing co-occurrence counts: how often a given\nverb occurs with a given noun in a given relation. The conditional probability of an\nargument noun given a verb for a particular relation P(njv;r)can then be used as a\nselectional preference metric for that pair of words (Brockmann and Lapata 2003,\nKeller and Lapata 2003):\nP(njv;r) =(\nC(n;v;r)\nC(v;r)ifC(n;v;r)>0\n0 otherwise\nThe inverse probability P(vjn;r)was found to have better performance in some cases\n(Brockmann and Lapata, 2003):\nP(vjn;r) =(\nC(n;v;r)\nC(n;r)ifC(n;v;r)>0\n0 otherwise 16 CHAPTER 21 \u2022 S EMANTIC ROLE LABELING\nAn even simpler approach is to use the simple log co-occurrence frequency of\nthe predicate with the argument log count (v;n;r)instead of conditional probability;\nthis seems to do better for extracting preferences for syntactic subjects rather than\nobjects (Brockmann and Lapata, 2003).\nEvaluating Selectional Preferences\nOne way to evaluate models of selectional preferences is to use pseudowords (Gale pseudowords\net al. 1992, Sch \u00a8utze 1992). A pseudoword is an arti\ufb01cial word created by concate-\nnating a test word in some context (say banana ) with a confounder word (say door )\nto create banana-door ). The task of the system is to identify which of the two words\nis the original word. To evaluate a selectional preference model (for example on the\nrelationship between a verb and a direct object) we take a test corpus and select all\nverb tokens. For each verb token (say drive ) we select the direct object (e.g., car),\nconcatenated with a confounder word that is its nearest neighbor , the noun with the\nfrequency closest to the original (say house ), to make car/house ). We then use the\nselectional preference model to choose which of carandhouse are more preferred\nobjects of drive , and compute how often the model chooses the correct original ob-\nject (e.g., car) (Chambers and Jurafsky, 2010).\nAnother evaluation metric is to get human preferences for a test set of verb-\nargument pairs, and have them rate their degree of plausibility. This is usually done\nby using magnitude estimation, a technique from psychophysics, in which subjects\nrate the plausibility of an argument proportional to a modulus item. A selectional\npreference model can then be evaluated by its correlation with the human prefer-\nences (Keller and Lapata, 2003).\n21.8 Primitive Decomposition of Predicates\nOne way of thinking about the semantic roles we have discussed through the chapter\nis that they help us de\ufb01ne the roles that arguments play in a decompositional way,\nbased on \ufb01nite lists of thematic roles (agent, patient, instrument, proto-agent, proto-\npatient, etc.). This idea of decomposing meaning into sets of primitive semantic\nelements or features, called primitive decomposition orcomponential analysis ,componential\nanalysis\nhas been taken even further, and focused particularly on predicates.\nConsider these examples of the verb kill:\n(21.41) Jim killed his philodendron.\n(21.42) Jim did something to cause his philodendron to become not alive.\nThere is a truth-conditional (\u2018propositional semantics\u2019) perspective from which these\ntwo sentences have the same meaning. Assuming this equivalence, we could repre-\nsent the meaning of killas:\n(21.43) KILL (x,y),CAUSE (x,BECOME (NOT(ALIVE (y))))\nthus using semantic primitives like do,cause ,become not , and alive .\nIndeed, one such set of potential semantic primitives has been used to account\nfor some of the verbal alternations discussed in Section 21.2 (Lakoff 1965, Dowty\n1979). Consider the following examples.\n(21.44) John opened the door. )CAUSE (John, BECOME (OPEN (door)))\n(21.45) The door opened. )BECOME (OPEN (door)) 21.9 \u2022 S UMMARY 17\n(21.46) The door is open. )OPEN (door)\nThe decompositional approach asserts that a single state-like predicate associ-\nated with open underlies all of these examples. The differences among the meanings\nof these examples arises from the combination of this single predicate with the prim-\nitives CAUSE and BECOME .\nWhile this approach to primitive decomposition can explain the similarity be-\ntween states and actions or causative and non-causative predicates, it still relies on\nhaving a large number of predicates like open . More radical approaches choose to\nbreak down these predicates as well. One such approach to verbal predicate decom-\nposition that played a role in early natural language systems is conceptual depen-\ndency (CD), a set of ten primitive predicates, shown in Fig. 21.8.conceptual\ndependency\nPrimitive De\ufb01nition\nATRANS The abstract transfer of possession or control from one entity to\nanother\nPTRANS The physical transfer of an object from one location to another\nMTRANS The transfer of mental concepts between entities or within an\nentity\nMBUILD The creation of new information within an entity\nPROPEL The application of physical force to move an object\nMOVE The integral movement of a body part by an animal\nINGEST The taking in of a substance by an animal\nEXPEL The expulsion of something from an animal\nSPEAK The action of producing a sound\nATTEND The action of focusing a sense organ\nFigure 21.8 A set of conceptual dependency primitives.\nBelow is an example sentence along with its CDrepresentation. The verb brought\nis translated into the two primitives ATRANS and PTRANS to indicate that the waiter\nboth physically conveyed the check to Mary and passed control of it to her. Note\nthat CDalso associates a \ufb01xed set of thematic roles with each primitive to represent\nthe various participants in the action.\n(21.47) The waiter brought Mary the check.\n9x;y Atrans (x)^Actor (x;Waiter )^Ob ject (x;Check )^To(x;Mary )\n^Ptrans (y)^Actor (y;Waiter )^Ob ject (y;Check )^To(y;Mary )\n21.9 Summary\n\u2022Semantic roles are abstract models of the role an argument plays in the event\ndescribed by the predicate.\n\u2022Thematic roles are a model of semantic roles based on a single \ufb01nite list of\nroles. Other semantic role models include per-verb semantic role lists and\nproto-agent /proto-patient , both of which are implemented in PropBank ,\nand per-frame role lists, implemented in FrameNet . 18 CHAPTER 21 \u2022 S EMANTIC ROLE LABELING\n\u2022Semantic role labeling is the task of assigning semantic role labels to the\nconstituents of a sentence. The task is generally treated as a supervised ma-\nchine learning task, with models trained on PropBank or FrameNet. Algo-\nrithms generally start by parsing a sentence and then automatically tag each\nparse tree node with a semantic role. Neural models map straight from words\nend-to-end.\n\u2022 Semantic selectional restrictions allow words (particularly predicates) to post\nconstraints on the semantic properties of their argument words. Selectional\npreference models (like selectional association or simple conditional proba-\nbility) allow a weight or probability to be assigned to the association between\na predicate and an argument word or class.\nBibliographical and Historical Notes\nAlthough the idea of semantic roles dates back to P \u00afan.ini, they were re-introduced\ninto modern linguistics by Gruber (1965), Fillmore (1966) and Fillmore (1968). Fill-\nmore had become interested in argument structure by studying Lucien Tesni `ere\u2019s\ngroundbreaking \u00b4El\u00b4ements de Syntaxe Structurale (Tesni `ere, 1959) in which the term\n\u2018dependency\u2019 was introduced and the foundations were laid for dependency gram-\nmar. Following Tesni `ere\u2019s terminology, Fillmore \ufb01rst referred to argument roles as\nactants (Fillmore, 1966) but quickly switched to the term case, (see Fillmore (2003))\nand proposed a universal list of semantic roles or cases (Agent, Patient, Instrument,\netc.), that could be taken on by the arguments of predicates. Verbs would be listed in\nthe lexicon with their case frame , the list of obligatory (or optional) case arguments.\nThe idea that semantic roles could provide an intermediate level of semantic\nrepresentation that could help map from syntactic parse structures to deeper, more\nfully-speci\ufb01ed representations of meaning was quickly adopted in natural language\nprocessing, and systems for extracting case frames were created for machine transla-\ntion (Wilks, 1973), question-answering (Hendrix et al., 1973), spoken-language pro-\ncessing (Nash-Webber, 1975), and dialogue systems (Bobrow et al., 1977). General-\npurpose semantic role labelers were developed. The earliest ones (Simmons, 1973)\n\ufb01rst parsed a sentence by means of an ATN (Augmented Transition Network) parser.\nEach verb then had a set of rules specifying how the parse should be mapped to se-\nmantic roles. These rules mainly made reference to grammatical functions (subject,\nobject, complement of speci\ufb01c prepositions) but also checked constituent internal\nfeatures such as the animacy of head nouns. Later systems assigned roles from pre-\nbuilt parse trees, again by using dictionaries with verb-speci\ufb01c case frames (Levin\n1977, Marcus 1980).\nBy 1977 case representation was widely used and taught in AI and NLP courses,\nand was described as a standard of natural language processing in the \ufb01rst edition of\nWinston\u2019s 1977 textbook Arti\ufb01cial Intelligence .\nIn the 1980s Fillmore proposed his model of frame semantics , later describing\nthe intuition as follows:\n\u201cThe idea behind frame semantics is that speakers are aware of possi-\nbly quite complex situation types, packages of connected expectations,\nthat go by various names\u2014frames, schemas, scenarios, scripts, cultural\nnarratives, memes\u2014and the words in our language are understood with\nsuch frames as their presupposed background.\u201d (Fillmore, 2012, p. 712) BIBLIOGRAPHICAL AND HISTORICAL NOTES 19\nThe word frame seemed to be in the air for a suite of related notions proposed at\nabout the same time by Minsky (1974), Hymes (1974), and Goffman (1974), as\nwell as related notions with other names like scripts (Schank and Abelson, 1975)\nandschemata (Bobrow and Norman, 1975) (see Tannen (1979) for a comparison).\nFillmore was also in\ufb02uenced by the semantic \ufb01eld theorists and by a visit to the Yale\nAI lab where he took notice of the lists of slots and \ufb01llers used by early information\nextraction systems like DeJong (1982) and Schank and Abelson (1977). In the 1990s\nFillmore drew on these insights to begin the FrameNet corpus annotation project.\nAt the same time, Beth Levin drew on her early case frame dictionaries (Levin,\n1977) to develop her book which summarized sets of verb classes de\ufb01ned by shared\nargument realizations (Levin, 1993). The VerbNet project built on this work (Kipper\net al., 2000), leading soon afterwards to the PropBank semantic-role-labeled corpus\ncreated by Martha Palmer and colleagues (Palmer et al., 2005).\nThe combination of rich linguistic annotation and corpus-based approach in-\nstantiated in FrameNet and PropBank led to a revival of automatic approaches to\nsemantic role labeling, \ufb01rst on FrameNet (Gildea and Jurafsky, 2000) and then on\nPropBank data (Gildea and Palmer, 2002, inter alia). The problem \ufb01rst addressed in\nthe 1970s by handwritten rules was thus now generally recast as one of supervised\nmachine learning enabled by large and consistent databases. Many popular features\nused for role labeling are de\ufb01ned in Gildea and Jurafsky (2002), Surdeanu et al.\n(2003), Xue and Palmer (2004), Pradhan et al. (2005), Che et al. (2009), and Zhao\net al. (2009). The use of dependency rather than constituency parses was introduced\nin the CoNLL-2008 shared task (Surdeanu et al., 2008). For surveys see Palmer\net al. (2010) and M `arquez et al. (2008).\nThe use of neural approaches to semantic role labeling was pioneered by Col-\nlobert et al. (2011), who applied a CRF on top of a convolutional net. Early work\nlike Foland, Jr. and Martin (2015) focused on using dependency features. Later work\neschewed syntactic features altogether; Zhou and Xu (2015) introduced the use of\na stacked (6-8 layer) biLSTM architecture, and (He et al., 2017) showed how to\naugment the biLSTM architecture with highway networks and also replace the CRF\nwith A* decoding that make it possible to apply a wide variety of global constraints\nin SRL decoding.\nMost semantic role labeling schemes only work within a single sentence, fo-\ncusing on the object of the verbal (or nominal, in the case of NomBank) predicate.\nHowever, in many cases, a verbal or nominal predicate may have an implicit argu-\nment : one that appears only in a contextual sentence, or perhaps not at all and mustimplicit\nargument\nbe inferred. In the two sentences This house has a new owner. The sale was \ufb01nalized\n10 days ago. thesale in the second sentence has no A RG1, but a reasonable reader\nwould infer that the Arg1 should be the house mentioned in the prior sentence. Find-\ning these arguments, implicit argument detection (sometimes shortened as iSRL ) iSRL\nwas introduced by Gerber and Chai (2010) and Ruppenhofer et al. (2010). See Do\net al. (2017) for more recent neural models.\nTo avoid the need for huge labeled training sets, unsupervised approaches for\nsemantic role labeling attempt to induce the set of semantic roles by clustering over\narguments. The task was pioneered by Riloff and Schmelzenbach (1998) and Swier\nand Stevenson (2004); see Grenager and Manning (2006), Titov and Klementiev\n(2012), Lang and Lapata (2014), Woodsend and Lapata (2015), and Titov and Khod-\ndam (2014).\nRecent innovations in frame labeling include connotation frames , which mark\nricher information about the argument of predicates. Connotation frames mark the 20 CHAPTER 21 \u2022 S EMANTIC ROLE LABELING\nsentiment of the writer or reader toward the arguments (for example using the verb\nsurvive inhe survived a bombing expresses the writer\u2019s sympathy toward the subject\nheand negative sentiment toward the bombing. See Chapter 22 for more details.\nSelectional preference has been widely studied beyond the selectional associa-\ntion models of Resnik (1993) and Resnik (1996). Methods have included clustering\n(Rooth et al., 1999), discriminative learning (Bergsma et al., 2008), and topic mod-\nels (S \u00b4eaghdha 2010, Ritter et al. 2010), and constraints can be expressed at the level\nof words or classes (Agirre and Martinez, 2001). Selectional preferences have also\nbeen successfully integrated into semantic role labeling (Erk 2007, Zapirain et al.\n2013, Do et al. 2017).\nExercises Exercises 21\nAgirre, E. and D. Martinez. 2001. Learning class-to-class\nselectional preferences. CoNLL .\nBaker, C. F., C. J. Fillmore, and J. B. Lowe. 1998. The Berke-\nley FrameNet project. COLING/ACL .\nBergsma, S., D. Lin, and R. Goebel. 2008. Discriminative\nlearning of selectional preference from unlabeled text.\nEMNLP .\nBloom\ufb01eld, L. 1933. Language . University of Chicago\nPress.\nBobrow, D. G., R. M. Kaplan, M. Kay, D. A. Norman,\nH. Thompson, and T. Winograd. 1977. GUS, A frame\ndriven dialog system. Arti\ufb01cial Intelligence , 8:155\u2013173.\nBobrow, D. G. and D. A. Norman. 1975. Some principles of\nmemory schemata. In D. G. Bobrow and A. Collins, eds,\nRepresentation and Understanding . Academic Press.\nBrockmann, C. and M. Lapata. 2003. Evaluating and com-\nbining approaches to selectional preference acquisition.\nEACL .\nCarreras, X. and L. M `arquez. 2005. Introduction to\nthe CoNLL-2005 shared task: Semantic role labeling.\nCoNLL .\nChambers, N. and D. Jurafsky. 2010. Improving the use\nof pseudo-words for evaluating selectional preferences.\nACL.\nChe, W., Z. Li, Y . Li, Y . Guo, B. Qin, and T. Liu. 2009. Mul-\ntilingual dependency-based syntactic and semantic pars-\ning. CoNLL .\nCollobert, R., J. Weston, L. Bottou, M. Karlen,\nK. Kavukcuoglu, and P. Kuksa. 2011. Natural language\nprocessing (almost) from scratch. JMLR , 12:2493\u20132537.\nDeJong, G. F. 1982. An overview of the FRUMP system.\nIn W. G. Lehnert and M. H. Ringle, eds, Strategies for\nNatural Language Processing , 149\u2013176. LEA.\nDo, Q. N. T., S. Bethard, and M.-F. Moens. 2017. Improv-\ning implicit semantic role labeling by predicting semantic\nframe arguments. IJCNLP .\nDowty, D. R. 1979. Word Meaning and Montague Grammar .\nD. Reidel.\nErk, K. 2007. A simple, similarity-based model for selec-\ntional preferences. ACL.\nFillmore, C. J. 1966. A proposal concerning English prepo-\nsitions. In F. P. Dinneen, ed., 17th annual Round Table ,\nvolume 17 of Monograph Series on Language and Lin-\nguistics , 19\u201334. Georgetown University Press.\nFillmore, C. J. 1968. The case for case. In E. W. Bach and\nR. T. Harms, eds, Universals in Linguistic Theory , 1\u201388.\nHolt, Rinehart & Winston.\nFillmore, C. J. 1985. Frames and the semantics of under-\nstanding. Quaderni di Semantica , VI(2):222\u2013254.\nFillmore, C. J. 2003. Valency and semantic roles: the con-\ncept of deep structure case. In V . Agel, L. M. Eichinger,\nH. W. Eroms, P. Hellwig, H. J. Heringer, and H. Lobin,\neds, Dependenz und Valenz: Ein internationales Hand-\nbuch der zeitgen \u00a8ossischen Forschung , chapter 36, 457\u2013\n475. Walter de Gruyter.\nFillmore, C. J. 2012. ACL lifetime achievement award:\nEncounters with language. Computational Linguistics ,\n38(4):701\u2013718.Fillmore, C. J. and C. F. Baker. 2009. A frames approach\nto semantic analysis. In B. Heine and H. Narrog, eds,\nThe Oxford Handbook of Linguistic Analysis , 313\u2013340.\nOxford University Press.\nFillmore, C. J., C. R. Johnson, and M. R. L. Petruck. 2003.\nBackground to FrameNet. International journal of lexi-\ncography , 16(3):235\u2013250.\nFoland, Jr., W. R. and J. H. Martin. 2015. Dependency-\nbased semantic role labeling using convolutional neural\nnetworks. *SEM 2015 .\nGale, W. A., K. W. Church, and D. Yarowsky. 1992. Work on\nstatistical methods for word sense disambiguation. AAAI\nFall Symposium on Probabilistic Approaches to Natural\nLanguage .\nGerber, M. and J. Y . Chai. 2010. Beyond nombank: A study\nof implicit arguments for nominal predicates. ACL.\nGildea, D. and D. Jurafsky. 2000. Automatic labeling of se-\nmantic roles. ACL.\nGildea, D. and D. Jurafsky. 2002. Automatic labeling of se-\nmantic roles. Computational Linguistics , 28(3):245\u2013288.\nGildea, D. and M. Palmer. 2002. The necessity of syntactic\nparsing for predicate argument recognition. ACL.\nGoffman, E. 1974. Frame analysis: An essay on the organi-\nzation of experience . Harvard University Press.\nGrenager, T. and C. D. Manning. 2006. Unsupervised dis-\ncovery of a statistical verb lexicon. EMNLP .\nGruber, J. S. 1965. Studies in Lexical Relations . Ph.D. thesis,\nMIT.\nHe, L., K. Lee, M. Lewis, and L. Zettlemoyer. 2017. Deep\nsemantic role labeling: What works and what\u2019s next.\nACL.\nHendrix, G. G., C. W. Thompson, and J. Slocum. 1973. Lan-\nguage processing via canonical verbs and semantic mod-\nels.Proceedings of IJCAI-73 .\nHirst, G. 1987. Semantic Interpretation and the Resolution\nof Ambiguity . Cambridge University Press.\nHymes, D. 1974. Ways of speaking. In R. Bauman and\nJ. Sherzer, eds, Explorations in the ethnography of speak-\ning, 433\u2013451. Cambridge University Press.\nJohnson-Laird, P. N. 1983. Mental Models . Harvard Univer-\nsity Press, Cambridge, MA.\nKatz, J. J. and J. A. Fodor. 1963. The structure of a semantic\ntheory. Language , 39:170\u2013210.\nKeller, F. and M. Lapata. 2003. Using the web to obtain fre-\nquencies for unseen bigrams. Computational Linguistics ,\n29:459\u2013484.\nKipper, K., H. T. Dang, and M. Palmer. 2000. Class-based\nconstruction of a verb lexicon. AAAI .\nKullback, S. and R. A. Leibler. 1951. On information and\nsuf\ufb01ciency. Annals of Mathematical Statistics , 22:79\u201386.\nLakoff, G. 1965. On the Nature of Syntactic Irregularity .\nPh.D. thesis, Indiana University. Published as Irregularity\nin Syntax . Holt, Rinehart, and Winston, New York, 1970.\nLang, J. and M. Lapata. 2014. Similarity-driven semantic\nrole induction via graph partitioning. Computational Lin-\nguistics , 40(3):633\u2013669.\nLevin, B. 1977. Mapping sentences to case frames. Techni-\ncal Report 167, MIT AI Laboratory. AI Working Paper\n143. 22 Chapter 21 \u2022 Semantic Role Labeling\nLevin, B. 1993. English Verb Classes and Alternations: A\nPreliminary Investigation . University of Chicago Press.\nLevin, B. and M. Rappaport Hovav. 2005. Argument Real-\nization . Cambridge University Press.\nMarcus, M. P. 1980. A Theory of Syntactic Recognition for\nNatural Language . MIT Press.\nM`arquez, L., X. Carreras, K. C. Litkowski, and S. Steven-\nson. 2008. Semantic role labeling: An introduction to the\nspecial issue. Computational linguistics , 34(2):145\u2013159.\nMeyers, A., R. Reeves, C. Macleod, R. Szekely, V . Zielinska,\nB. Young, and R. Grishman. 2004. The nombank project:\nAn interim report. NAACL/HLT Workshop: Frontiers in\nCorpus Annotation .\nMinsky, M. 1974. A framework for representing knowledge.\nTechnical Report 306, MIT AI Laboratory. Memo 306.\nNash-Webber, B. L. 1975. The role of semantics in automatic\nspeech understanding. In D. G. Bobrow and A. Collins,\neds, Representation and Understanding , 351\u2013382. Aca-\ndemic Press.\nPalmer, M., D. Gildea, and N. Xue. 2010. Semantic role\nlabeling. Synthesis Lectures on Human Language Tech-\nnologies , 3(1):1\u2013103.\nPalmer, M., P. Kingsbury, and D. Gildea. 2005. The proposi-\ntion bank: An annotated corpus of semantic roles. Com-\nputational Linguistics , 31(1):71\u2013106.\nPenn, G. and P. Kiparsky. 2012. On P \u00afan.ini and the gen-\nerative capacity of contextualized replacement systems.\nCOLING .\nPradhan, S., A. Moschitti, N. Xue, H. T. Ng, A. Bj \u00a8orkelund,\nO. Uryupina, Y . Zhang, and Z. Zhong. 2013. Towards\nrobust linguistic analysis using OntoNotes. CoNLL .\nPradhan, S., W. Ward, K. Hacioglu, J. H. Martin, and D. Ju-\nrafsky. 2005. Semantic role labeling using different syn-\ntactic views. ACL.\nResnik, P. 1993. Semantic classes and syntactic ambiguity.\nHLT.\nResnik, P. 1996. Selectional constraints: An information-\ntheoretic model and its computational realization. Cogni-\ntion, 61:127\u2013159.\nRiloff, E. and M. Schmelzenbach. 1998. An empirical ap-\nproach to conceptual case frame acquisition. Proceedings\nof the Sixth Workshop on Very Large Corpora .\nRitter, A., O. Etzioni, and Mausam. 2010. A latent dirichlet\nallocation method for selectional preferences. ACL.\nRooth, M., S. Riezler, D. Prescher, G. Carroll, and F. Beil.\n1999. Inducing a semantically annotated lexicon via EM-\nbased clustering. ACL.\nRuppenhofer, J., M. Ellsworth, M. R. L. Petruck, C. R. John-\nson, C. F. Baker, and J. Scheffczyk. 2016. FrameNet II:\nExtended theory and practice.\nRuppenhofer, J., C. Sporleder, R. Morante, C. F. Baker,\nand M. Palmer. 2010. Semeval-2010 task 10: Linking\nevents and their participants in discourse. 5th Interna-\ntional Workshop on Semantic Evaluation .\nSchank, R. C. and R. P. Abelson. 1975. Scripts, plans, and\nknowledge. Proceedings of IJCAI-75 .\nSchank, R. C. and R. P. Abelson. 1977. Scripts, Plans, Goals\nand Understanding . Lawrence Erlbaum.Sch\u00a8utze, H. 1992. Context space. AAAI Fall Symposium on\nProbabilistic Approaches to Natural Language .\nS\u00b4eaghdha, D. O. 2010. Latent variable models of selectional\npreference. ACL.\nShi, P. and J. Lin. 2019. Simple BERT models for relation\nextraction and semantic role labeling. ArXiv.\nSimmons, R. F. 1973. Semantic networks: Their compu-\ntation and use for understanding English sentences. In\nR. C. Schank and K. M. Colby, eds, Computer Models of\nThought and Language , 61\u2013113. W.H. Freeman & Co.\nSloan, M. C. 2010. Aristotle\u2019s Nicomachean Ethics as the\noriginal locus for the Septem Circumstantiae. Classical\nPhilology , 105(3):236\u2013251.\nSurdeanu, M., S. Harabagiu, J. Williams, and P. Aarseth.\n2003. Using predicate-argument structures for informa-\ntion extraction. ACL.\nSurdeanu, M., R. Johansson, A. Meyers, L. M `arquez, and\nJ. Nivre. 2008. The CoNLL 2008 shared task on joint\nparsing of syntactic and semantic dependencies. CoNLL .\nSwier, R. and S. Stevenson. 2004. Unsupervised semantic\nrole labelling. EMNLP .\nTannen, D. 1979. What\u2019s in a frame? Surface evidence for\nunderlying expectations. In R. Freedle, ed., New Direc-\ntions in Discourse Processing , 137\u2013181. Ablex.\nTesni `ere, L. 1959. \u00b4El\u00b4ements de Syntaxe Structurale . Librairie\nC. Klincksieck, Paris.\nTitov, I. and E. Khoddam. 2014. Unsupervised induction of\nsemantic roles within a reconstruction-error minimization\nframework. NAACL HLT .\nTitov, I. and A. Klementiev. 2012. A Bayesian approach to\nunsupervised semantic role induction. EACL .\nWilks, Y . 1973. An arti\ufb01cial intelligence approach to ma-\nchine translation. In R. C. Schank and K. M. Colby, eds,\nComputer Models of Thought and Language , 114\u2013151.\nW.H. Freeman.\nWilks, Y . 1975a. Preference semantics. In E. L. Keenan,\ned.,The Formal Semantics of Natural Language , 329\u2013\n350. Cambridge Univ. Press.\nWilks, Y . 1975b. A preferential, pattern-seeking, seman-\ntics for natural language inference. Arti\ufb01cial Intelligence ,\n6(1):53\u201374.\nWinston, P. H. 1977. Arti\ufb01cial Intelligence . Addison Wesley.\nWoodsend, K. and M. Lapata. 2015. Distributed representa-\ntions for unsupervised semantic role labeling. EMNLP .\nXue, N. and M. Palmer. 2004. Calibrating features for se-\nmantic role labeling. EMNLP .\nZapirain, B., E. Agirre, L. M `arquez, and M. Surdeanu. 2013.\nSelectional preferences for semantic role classi\ufb01cation.\nComputational Linguistics , 39(3):631\u2013663.\nZhao, H., W. Chen, C. Kit, and G. Zhou. 2009. Multilingual\ndependency learning: A huge feature engineering method\nto semantic dependency parsing. CoNLL .\nZhou, J. and W. Xu. 2015. End-to-end learning of semantic\nrole labeling using recurrent neural networks. ACL.", "22.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\n22Lexicons for Sentiment, Affect,\nand Connotation\nSome day we\u2019ll be able to measure the power of words\nMaya Angelou\nIn this chapter we turn to tools for interpreting affective meaning, extending our affective\nstudy of sentiment analysis in Chapter 4. We use the word \u2018affective\u2019, following the\ntradition in affective computing (Picard, 1995) to mean emotion, sentiment, per-\nsonality, mood, and attitudes. Affective meaning is closely related to subjectivity , subjectivity\nthe study of a speaker or writer\u2019s evaluations, opinions, emotions, and speculations\n(Wiebe et al., 1999).\nHow should affective meaning be de\ufb01ned? One in\ufb02uential typology of affec-\ntive states comes from Scherer (2000), who de\ufb01nes each class of affective states by\nfactors like its cognitive realization and time course (Fig. 22.1).\nEmotion: Relatively brief episode of response to the evaluation of an external\nor internal event as being of major signi\ufb01cance.\n(angry, sad, joyful, fearful, ashamed, proud, elated, desperate )\nMood: Diffuse affect state, most pronounced as change in subjective feeling, of\nlow intensity but relatively long duration, often without apparent cause.\n(cheerful, gloomy, irritable, listless, depressed, buoyant )\nInterpersonal stance: Affective stance taken toward another person in a spe-\nci\ufb01c interaction, coloring the interpersonal exchange in that situation.\n(distant, cold, warm, supportive, contemptuous, friendly )\nAttitude: Relatively enduring, affectively colored beliefs, preferences, and pre-\ndispositions towards objects or persons.\n(liking, loving, hating, valuing, desiring )\nPersonality traits: Emotionally laden, stable personality dispositions and be-\nhavior tendencies, typical for a person.\n(nervous, anxious, reckless, morose, hostile, jealous )\nFigure 22.1 The Scherer typology of affective states (Scherer, 2000).\nWe can design extractors for each of these kinds of affective states. Chapter 4\nalready introduced sentiment analysis , the task of extracting the positive or negative\norientation that a writer expresses in a text. This corresponds in Scherer\u2019s typology\nto the extraction of attitudes : \ufb01guring out what people like or dislike, from affect-\nrich texts like consumer reviews of books or movies, newspaper editorials, or public\nsentiment in blogs or tweets.\nDetecting emotion andmoods is useful for detecting whether a student is con-\nfused, engaged, or certain when interacting with a tutorial system, whether a caller\nto a help line is frustrated, whether someone\u2019s blog posts or tweets indicated depres-\nsion. Detecting emotions like fear in novels, for example, could help us trace what\ngroups or situations are feared and how that changes over time. 2CHAPTER 22 \u2022 L EXICONS FOR SENTIMENT , AFFECT ,AND CONNOTATION\nDetecting different interpersonal stances can be useful when extracting infor-\nmation from human-human conversations. The goal here is to detect stances like\nfriendliness or awkwardness in interviews or friendly conversations, for example for\nsummarizing meetings or \ufb01nding parts of a conversation where people are especially\nexcited or engaged, conversational hot spots that can help in meeting summariza-\ntion. Detecting the personality of a user\u2014such as whether the user is an extrovert\nor the extent to which they are open to experience \u2014 can help improve conversa-\ntional agents, which seem to work better if they match users\u2019 personality expecta-\ntions (Mairesse and Walker, 2008). And affect is important for generation as well\nas recognition; synthesizing affect is important for conversational agents in various\ndomains, including literacy tutors such as children\u2019s storybooks, or computer games.\nIn Chapter 4 we introduced the use of naive Bayes classi\ufb01cation to classify a\ndocument\u2019s sentiment. Various classi\ufb01ers have been successfully applied to many of\nthese tasks, using all the words in the training set as input to a classi\ufb01er which then\ndetermines the affect status of the text.\nIn this chapter we focus on an alternative model, in which instead of using every\nword as a feature, we focus only on certain words, ones that carry particularly strong\ncues to affect or sentiment. We call these lists of words affective lexicons orsenti-\nment lexicons . These lexicons presuppose a fact about semantics: that words have\naffective meanings orconnotations . The word connotation has different meanings connotations\nin different \ufb01elds, but here we use it to mean the aspects of a word\u2019s meaning that\nare related to a writer or reader\u2019s emotions, sentiment, opinions, or evaluations. In\naddition to their ability to help determine the affective status of a text, connotation\nlexicons can be useful features for other kinds of affective tasks, and for computa-\ntional social science analysis.\nIn the next sections we introduce basic theories of emotion, show how sentiment\nlexicons are a special case of emotion lexicons, and mention some useful lexicons.\nWe then survey three ways for building lexicons: human labeling, semi-supervised,\nand supervised. Finally, we talk about how to detect affect toward a particular entity,\nand introduce connotation frames.\n22.1 De\ufb01ning Emotion\nOne of the most important affective classes is emotion , which Scherer (2000) de\ufb01nes emotion\nas a \u201crelatively brief episode of response to the evaluation of an external or internal\nevent as being of major signi\ufb01cance\u201d.\nDetecting emotion has the potential to improve a number of language processing\ntasks. Emotion recognition could help dialogue systems like tutoring systems detect\nthat a student was unhappy, bored, hesitant, con\ufb01dent, and so on. Automatically\ndetecting emotions in reviews or customer responses (anger, dissatisfaction, trust)\ncould help businesses recognize speci\ufb01c problem areas or ones that are going well.\nEmotion can play a role in medical NLP tasks like helping diagnose depression or\nsuicidal intent. Detecting emotions expressed toward characters in novels might\nplay a role in understanding how different social groups were viewed by society at\ndifferent times.\nComputational models of emotion in NLP have mainly been based on two fami-\nlies of theories of emotion (out of the many studied in the \ufb01eld of affective science).\nIn one of these families, emotions are viewed as \ufb01xed atomic units, limited in num-\nber, and from which others are generated, often called basic emotions (Tomkins basic emotions 22.1 \u2022 D EFINING EMOTION 3\n1962, Plutchik 1962), a model dating back to Darwin. Perhaps the most well-known\nof this family of theories are the 6 emotions proposed by Ekman (e.g., Ekman 1999)\nto be universally present in all cultures: surprise, happiness, anger, fear, disgust,\nsadness . Another atomic theory is the Plutchik (1980) wheel of emotion, consisting\nof 8 basic emotions in four opposing pairs: joy\u2013sadness ,anger\u2013fear ,trust\u2013disgust ,\nandanticipation\u2013surprise , together with the emotions derived from them, shown in\nFig. 22.2.\nFigure 22.2 Plutchik wheel of emotion.\nThe second class of emotion theories widely used in NLP views emotion as a\nspace in 2 or 3 dimensions (Russell, 1980). Most models include the two dimensions\nvalence andarousal , and many add a third, dominance . These can be de\ufb01ned as:\nvalence: the pleasantness of the stimulus\narousal: the level of alertness, activeness, or energy provoked by the stimulus\ndominance: the degree of control or dominance exerted by the stimulus or the\nemotion\nSentiment can be viewed as a special case of this second view of emotions as points\nin space. In particular, the valence dimension, measuring how pleasant or unpleasant\na word is, is often used directly as a measure of sentiment.\nIn these lexicon-based models of affect, the affective meaning of a word is gen-\nerally \ufb01xed, irrespective of the linguistic context in which a word is used, or the\ndialect or culture of the speaker. By contrast, other models in affective science repre-\nsent emotions as much richer processes involving cognition (Barrett et al., 2007). In\nappraisal theory , for example, emotions are complex processes, in which a person\nconsiders how an event is congruent with their goals, taking into account variables\nlike the agency, certainty, urgency, novelty and control associated with the event\n(Moors et al., 2013). Computational models in NLP taking into account these richer\ntheories of emotion will likely play an important role in future work. 4CHAPTER 22 \u2022 L EXICONS FOR SENTIMENT , AFFECT ,AND CONNOTATION\n22.2 Available Sentiment and Affect Lexicons\nA wide variety of affect lexicons have been created and released. The most basic\nlexicons label words along one dimension of semantic variability, generally called\n\u201csentiment\u201d or \u201cvalence\u201d.\nIn the simplest lexicons this dimension is represented in a binary fashion, with\na wordlist for positive words and a wordlist for negative words. The oldest is the\nGeneral Inquirer (Stone et al., 1966), which drew on content analysis and on earlyGeneral\nInquirer\nwork in the cognitive psychology of word meaning (Osgood et al., 1957). The Gen-\neral Inquirer has a lexicon of 1915 positive words and a lexicon of 2291 negative\nwords (as well as other lexicons discussed below). The MPQA Subjectivity lexicon\n(Wilson et al., 2005) has 2718 positive and 4912 negative words drawn from prior\nlexicons plus a bootstrapped list of subjective words and phrases (Riloff and Wiebe,\n2003). Each entry in the lexicon is hand-labeled for sentiment and also labeled for\nreliability (strongly subjective or weakly subjective). The polarity lexicon of Hu\nand Liu (2004) gives 2006 positive and 4783 negative words, drawn from product\nreviews, labeled using a bootstrapping method from WordNet.\nPositive admire, amazing, assure, celebration, charm, eager, enthusiastic, excellent, fancy, fan-\ntastic, frolic, graceful, happy, joy, luck, majesty, mercy, nice, patience, perfect, proud,\nrejoice, relief, respect, satisfactorily, sensational, super, terri\ufb01c, thank, vivid, wise, won-\nderful, zest\nNegative abominable, anger, anxious, bad, catastrophe, cheap, complaint, condescending, deceit,\ndefective, disappointment, embarrass, fake, fear, \ufb01lthy, fool, guilt, hate, idiot, in\ufb02ict, lazy,\nmiserable, mourn, nervous, objection, pest, plot, reject, scream, silly, terrible, unfriendly,\nvile, wicked\nFigure 22.3 Some words with consistent sentiment across the General Inquirer (Stone et al., 1966), the\nMPQA Subjectivity lexicon (Wilson et al., 2005), and the polarity lexicon of Hu and Liu (2004).\nSlightly more general than these sentiment lexicons are lexicons that assign each\nword a value on all three affective dimensions. The NRC Valence, Arousal, and\nDominance (V AD) lexicon (Mohammad, 2018a) assigns valence, arousal, and dom-\ninance scores to 20,000 words. Some examples are shown in Fig. 22.4.\nValence Arousal Dominance\nvacation .840 enraged .962 powerful .991\ndelightful .918 party .840 authority .935\nwhistle .653 organized .337 saxophone .482\nconsolation .408 effortless .120 discouraged .0090\ntorture .115 napping .046 weak .045\nFigure 22.4 Values of sample words on the emotional dimensions of Mohammad (2018a).\nThe NRC Word-Emotion Association Lexicon, also called EmoLex (Moham- EmoLex\nmad and Turney, 2013), uses the Plutchik (1980) 8 basic emotions de\ufb01ned above.\nThe lexicon includes around 14,000 words including words from prior lexicons as\nwell as frequent nouns, verbs, adverbs and adjectives. Values from the lexicon for\nsome sample words: 22.3 \u2022 C REATING AFFECT LEXICONS BY HUMAN LABELING 5\nWord\nanger\nanticipation\ndisgust\nfear\njoy\nsadness\nsurprise\ntrust\npositive\nnegative\nreward 0100101110\nworry 0101010001\ntenderness 0000100010\nsweetheart 0100110110\nsuddenly 0000001000\nthirst 0100011000\ngarbage 0010000001\nFor a smaller set of 5,814 words, the NRC Emotion/Affect Intensity Lexicon\n(Mohammad, 2018b) contains real-valued scores of association for anger, fear, joy,\nand sadness; Fig. 22.5 shows examples.\nAnger Fear Joy Sadness\noutraged 0.964 horror 0.923 superb 0.864 sad 0.844\nviolence 0.742 anguish 0.703 cheered 0.773 guilt 0.750\ncoup 0.578 pestilence 0.625 rainbow 0.531 unkind 0.547\noust 0.484 stressed 0.531 gesture 0.387 dif\ufb01culties 0.421\nsuspicious 0.484 failing 0.531 warms 0.391 beggar 0.422\nnurture 0.059 con\ufb01dent 0.094 hardship .031 sing 0.017\nFigure 22.5 Sample emotional intensities for words for anger, fear, joy, and sadness from\nMohammad (2018b).\nLIWC ,Linguistic Inquiry and Word Count , is a widely used set of 73 lex- LIWC\nicons containing over 2300 words (Pennebaker et al., 2007), designed to capture\naspects of lexical meaning relevant for social psychological tasks. In addition to\nsentiment-related lexicons like ones for negative emotion ( bad, weird, hate, prob-\nlem, tough ) and positive emotion ( love, nice, sweet ), LIWC includes lexicons for\ncategories like anger, sadness, cognitive mechanisms, perception, tentative, and in-\nhibition, shown in Fig. 22.6.\nThere are various other hand-built affective lexicons. The General Inquirer in-\ncludes additional lexicons for dimensions like strong vs. weak, active vs. passive,\noverstated vs. understated, as well as lexicons for categories like pleasure, pain,\nvirtue, vice, motivation, and cognitive orientation.\nAnother useful feature for various tasks is the distinction between concrete concrete\nwords like banana orbathrobe andabstract words like belief andalthough . The abstract\nlexicon in Brysbaert et al. (2014) used crowdsourcing to assign a rating from 1 to 5\nof the concreteness of 40,000 words, thus assigning banana ,bathrobe , and bagel 5,\nbelief 1.19, although 1.07, and in between words like brisk a 2.5.\n22.3 Creating Affect Lexicons by Human Labeling\nThe earliest method used to build affect lexicons, and still in common use, is to have\nhumans label each word. This is now most commonly done via crowdsourcing : crowdsourcing\nbreaking the task into small pieces and distributing them to a large number of anno- 6CHAPTER 22 \u2022 L EXICONS FOR SENTIMENT , AFFECT ,AND CONNOTATION\nPositive Negative\nEmotion Emotion Insight Inhibition Family Negate\nappreciat* anger* aware* avoid* brother* aren\u2019t\ncomfort* bore* believe careful* cousin* cannot\ngreat cry decid* hesitat* daughter* didn\u2019t\nhappy despair* feel limit* family neither\ninterest fail* \ufb01gur* oppos* father* never\njoy* fear know prevent* grandf* no\nperfect* griev* knew reluctan* grandm* nobod*\nplease* hate* means safe* husband none\nsafe* panic* notice* stop mom nor\nterri\ufb01c suffers recogni* stubborn* mother nothing\nvalue terrify sense wait niece* nowhere\nwow* violent* think wary wife without\nFigure 22.6 Samples from 5 of the 73 lexical categories in LIWC (Pennebaker et al., 2007).\nThe * means the previous letters are a word pre\ufb01x and all words with that pre\ufb01x are included\nin the category.\ntators. Let\u2019s take a look at some of the methodological choices for two crowdsourced\nemotion lexicons.\nThe NRC Emotion Lexicon (EmoLex) (Mohammad and Turney, 2013), labeled\nemotions in two steps. To ensure that the annotators were judging the correct sense\nof the word, they \ufb01rst answered a multiple-choice synonym question that primed\nthe correct sense of the word (without requiring the annotator to read a potentially\nconfusing sense de\ufb01nition). These were created automatically using the headwords\nassociated with the thesaurus category of the sense in question in the Macquarie\ndictionary and the headwords of 3 random distractor categories. An example:\nWhich word is closest in meaning (most related) to startle?\n\u2022automobile\n\u2022shake\n\u2022honesty\n\u2022entertain\nFor each word (e.g. startle ), the annotator was then asked to rate how associated\nthat word is with each of the 8 emotions ( joy,fear,anger , etc.). The associations\nwere rated on a scale of not,weakly ,moderately , and strongly associated. Outlier\nratings were removed, and then each term was assigned the class chosen by the ma-\njority of the annotators, with ties broken by choosing the stronger intensity, and then\nthe 4 levels were mapped into a binary label for each word (no and weak mapped to\n0, moderate and strong mapped to 1).\nThe NRC V AD Lexicon (Mohammad, 2018a) was built by selecting words and\nemoticons from prior lexicons and annotating them with crowd-sourcing using best-\nworst scaling (Louviere et al. 2015, Kiritchenko and Mohammad 2017). In best-best-worst\nscaling\nworst scaling, annotators are given N items (usually 4) and are asked which item is\nthebest (highest) and which is the worst (lowest) in terms of some property. The\nset of words used to describe the ends of the scales are taken from prior literature.\nFor valence, for example, the raters were asked:\nQ1. Which of the four words below is associated with the MOST happi-\nness / pleasure / positiveness / satisfaction / contentedness / hopefulness\nOR LEAST unhappiness / annoyance / negativeness / dissatisfaction / 22.4 \u2022 S EMI-SUPERVISED INDUCTION OF AFFECT LEXICONS 7\nmelancholy / despair? (Four words listed as options.)\nQ2. Which of the four words below is associated with the LEAST hap-\npiness / pleasure / positiveness / satisfaction / contentedness / hopeful-\nness OR MOST unhappiness / annoyance / negativeness / dissatisfaction\n/ melancholy / despair? (Four words listed as options.)\nThe score for each word in the lexicon is the proportion of times the item was chosen\nas the best (highest V/A/D) minus the proportion of times the item was chosen as the\nworst (lowest V/A/D). The agreement between annotations are evaluated by split-\nhalf reliability : split the corpus in half and compute the correlations between thesplit-half\nreliability\nannotations in the two halves.\n22.4 Semi-supervised Induction of Affect Lexicons\nAnother common way to learn sentiment lexicons is to start from a set of seed words\nthat de\ufb01ne two poles of a semantic axis (words like good orbad), and then \ufb01nd ways\nto label each word wby its similarity to the two seed sets. Here we summarize two\nfamilies of seed-based semi-supervised lexicon induction algorithms, axis-based and\ngraph-based.\n22.4.1 Semantic Axis Methods\nOne of the most well-known lexicon induction methods, the Turney and Littman\n(2003) algorithm, is given seed words like good orbad, and then for each word wto\nbe labeled, measures both how similar it is to good and how different it is from bad.\nHere we describe a slight extension of the algorithm due to An et al. (2018), which\nis based on computing a semantic axis .\nIn the \ufb01rst step, we choose seed words by hand. There are two methods for\ndealing with the fact that the affect of a word is different in different contexts: (1)\nstart with a single large seed lexicon and rely on the induction algorithm to \ufb01ne-tune\nit to the domain, or (2) choose different seed words for different genres. Hellrich\net al. (2019) suggests that for modeling affect across different historical time periods,\nstarting with a large modern affect dictionary is better than small seedsets tuned to\nbe stable across time. As an example of the second approach, Hamilton et al. (2016)\nde\ufb01ne one set of seed words for general sentiment analysis, a different set for Twitter,\nand yet another set for sentiment in \ufb01nancial text:\nDomain Positive seeds Negative seeds\nGeneral good, lovely, excellent, fortunate, pleas-\nant, delightful, perfect, loved, love,\nhappybad, horrible, poor, unfortunate, un-\npleasant, disgusting, evil, hated, hate,\nunhappy\nTwitter love, loved, loves, awesome, nice,\namazing, best, fantastic, correct, happyhate, hated, hates, terrible, nasty, awful,\nworst, horrible, wrong, sad\nFinance successful, excellent, pro\ufb01t, bene\ufb01cial,\nimproving, improved, success, gains,\npositivenegligent, loss, volatile, wrong, losses,\ndamages, bad, litigation, failure, down,\nnegative\nIn the second step, we compute embeddings for each of the pole words. These\nembeddings can be off-the-shelf word2vec embeddings, or can be computed directly 8CHAPTER 22 \u2022 L EXICONS FOR SENTIMENT , AFFECT ,AND CONNOTATION\non a speci\ufb01c corpus (for example using a \ufb01nancial corpus if a \ufb01nance lexicon is the\ngoal), or we can \ufb01ne-tune off-the-shelf embeddings to a corpus. Fine-tuning is espe-\ncially important if we have a very speci\ufb01c genre of text but don\u2019t have enough data\nto train good embeddings. In \ufb01ne-tuning, we begin with off-the-shelf embeddings\nlike word2vec, and continue training them on the small target corpus.\nOnce we have embeddings for each pole word, we create an embedding that\nrepresents each pole by taking the centroid of the embeddings of each of the seed\nwords; recall that the centroid is the multidimensional version of the mean. Given\na set of embeddings for the positive seed words S+=fE(w+\n1);E(w+\n2);:::;E(w+\nn)g,\nand embeddings for the negative seed words S\u0000=fE(w\u0000\n1);E(w\u0000\n2);:::;E(w\u0000\nm)g, the\npole centroids are:\nV+=1\nnnX\n1E(w+\ni)\nV\u0000=1\nmmX\n1E(w\u0000\ni) (22.1)\nThe semantic axis de\ufb01ned by the poles is computed just by subtracting the two vec-\ntors:\nVaxis=V+\u0000V\u0000(22.2)\nVaxis, the semantic axis, is a vector in the direction of positive sentiment. Finally,\nwe compute (via cosine similarity) the angle between the vector in the direction of\npositive sentiment and the direction of w\u2019s embedding. A higher cosine means that\nwis more aligned with S+than S\u0000.\nscore (w) = cos\u0000\nE(w);Vaxis\u0001\n=E(w)\u0001Vaxis\nkE(w)kkVaxisk(22.3)\nIf a dictionary of words with sentiment scores is suf\ufb01cient, we\u2019re done! Or if we\nneed to group words into a positive and a negative lexicon, we can use a threshold\nor other method to give us discrete lexicons.\n22.4.2 Label Propagation\nAn alternative family of methods de\ufb01nes lexicons by propagating sentiment labels\non graphs, an idea suggested in early work by Hatzivassiloglou and McKeown\n(1997). We\u2019ll describe the simple SentProp (Sentiment Propagation) algorithm of\nHamilton et al. (2016), which has four steps:\n1.De\ufb01ne a graph : Given word embeddings, build a weighted lexical graph by\nconnecting each word with its knearest neighbors (according to cosine simi-\nlarity). The weights of the edge between words wiandwjare set as:\nEi;j=arccos \n\u0000wi>wj\nkwikkwjk!\n: (22.4)\n2.De\ufb01ne a seed set: Choose positive and negative seed words. 22.4 \u2022 S EMI-SUPERVISED INDUCTION OF AFFECT LEXICONS 9\n3.Propagate polarities from the seed set: Now we perform a random walk on\nthis graph, starting at the seed set. In a random walk, we start at a node and\nthen choose a node to move to with probability proportional to the edge prob-\nability. A word\u2019s polarity score for a seed set is proportional to the probability\nof a random walk from the seed set landing on that word (Fig. 22.7).\n4.Create word scores : We walk from both positive and negative seed sets,\nresulting in positive (rawscore+(wi)) and negative (rawscore\u0000(wi)) raw label\nscores. We then combine these values into a positive-polarity score as:\nscore+(wi) =rawscore+(wi)\nrawscore+(wi) +rawscore\u0000(wi)(22.5)\nIt\u2019s often helpful to standardize the scores to have zero mean and unit variance\nwithin a corpus.\n5.Assign con\ufb01dence to each score: Because sentiment scores are in\ufb02uenced by\nthe seed set, we\u2019d like to know how much the score of a word would change if\na different seed set is used. We can use bootstrap sampling to get con\ufb01dence\nregions, by computing the propagation Btimes over random subsets of the\npositive and negative seed sets (for example using B=50 and choosing 7 of\nthe 10 seed words each time). The standard deviation of the bootstrap sampled\npolarity scores gives a con\ufb01dence measure.\nidolize\nlove\nadore\nappreciate\nlike\n\ufb01nd\ndislike\nsee\nnotice\ndisapprove\nabhor\nhate\nloathe\ndespise\nuncover\nidolize\nlove\nadore\nappreciate\nlike\n\ufb01nd\ndislike\nsee\nnotice\ndisapprove\nabhor\nhate\nloathe\ndespise\nuncover\n(a) (b)\nFigure 22.7 Intuition of the S ENTPROP algorithm. (a) Run random walks from the seed words. (b) Assign\npolarity scores (shown here as colors green or red) based on the frequency of random walk visits.\n22.4.3 Other Methods\nThe core of semisupervised algorithms is the metric for measuring similarity with\nthe seed words. The Turney and Littman (2003) and Hamilton et al. (2016) ap-\nproaches above used embedding cosine as the distance metric: words were labeled\nas positive basically if their embeddings had high cosines with positive seeds and\nlow cosines with negative seeds. Other methods have chosen other kinds of distance\nmetrics besides embedding cosine.\nFor example the Hatzivassiloglou and McKeown (1997) algorithm uses syntactic\ncues; two adjectives are considered similar if they were frequently conjoined by and\nand rarely conjoined by but. This is based on the intuition that adjectives conjoined\nby the words andtend to have the same polarity; positive adjectives are generally\ncoordinated with positive, negative with negative:\nfair and legitimate, corrupt and brutal\nbut less often positive adjectives coordinated with negative:\n*fair and brutal, *corrupt and legitimate 10 CHAPTER 22 \u2022 L EXICONS FOR SENTIMENT , AFFECT ,AND CONNOTATION\nBy contrast, adjectives conjoined by butare likely to be of opposite polarity:\nfair but brutal\nAnother cue to opposite polarity comes from morphological negation ( un-,im-,\n-less). Adjectives with the same root but differing in a morphological negative ( ad-\nequate/inadequate ,thoughtful/thoughtless ) tend to be of opposite polarity.\nYet another method for \ufb01nding words that have a similar polarity to seed words\nis to make use of a thesaurus like WordNet (Kim and Hovy 2004, Hu and Liu 2004).\nA word\u2019s synonyms presumably share its polarity while a word\u2019s antonyms probably\nhave the opposite polarity. After a seed lexicon is built, each lexicon is updated as\nfollows, possibly iterated.\nLex+: Add synonyms of positive words ( well) and antonyms (like \ufb01ne) of negative\nwords\nLex\u0000: Add synonyms of negative words ( awful ) and antonyms (like evil) of positive\nwords\nAn extension of this algorithm assigns polarity to WordNet senses, called Senti-\nWordNet (Baccianella et al., 2010). Fig. 22.8 shows some examples. SentiWordNet\nSynset Pos Neg Obj\ngood#6 \u2018agreeable or pleasing\u2019 1 0 0\nrespectable#2 honorable#4 good#4 estimable#2 \u2018deserving of esteem\u2019 0.75 0 0.25\nestimable#3 computable#1 \u2018may be computed or estimated\u2019 0 0 1\nsting#1 burn#4 bite#2 \u2018cause a sharp or stinging pain\u2019 0 0.875 .125\nacute#6 \u2018of critical importance and consequence\u2019 0.625 0.125 .250\nacute#4 \u2018of an angle; less than 90 degrees\u2019 0 0 1\nacute#1 \u2018having or experiencing a rapid onset and short but severe course\u2019 0 0.5 0.5\nFigure 22.8 Examples from SentiWordNet 3.0 (Baccianella et al., 2010). Note the differences between senses\nof homonymous words: estimable#3 is purely objective, while estimable#2 is positive; acute can be positive\n(acute#6 ), negative ( acute#1 ), or neutral ( acute #4 ).\nIn this algorithm, polarity is assigned to entire synsets rather than words. A\npositive lexicon is built from all the synsets associated with 7 positive words, and a\nnegative lexicon from synsets associated with 7 negative words. A classi\ufb01er is then\ntrained from this data to take a WordNet gloss and decide if the sense being de\ufb01ned\nis positive, negative or neutral. A further step (involving a random-walk algorithm)\nassigns a score to each WordNet synset for its degree of positivity, negativity, and\nneutrality.\nIn summary, semisupervised algorithms use a human-de\ufb01ned set of seed words\nfor the two poles of a dimension, and use similarity metrics like embedding cosine,\ncoordination, morphology, or thesaurus structure to score words by how similar they\nare to the positive seeds and how dissimilar to the negative seeds.\n22.5 Supervised Learning of Word Sentiment\nSemi-supervised methods require only minimal human supervision (in the form of\nseed sets). But sometimes a supervision signal exists in the world and can be made\nuse of. One such signal is the scores associated with online reviews .\nThe web contains an enormous number of online reviews for restaurants, movies,\nbooks, or other products, each of which have the text of the review along with an 22.5 \u2022 S UPERVISED LEARNING OF WORD SENTIMENT 11\nassociated review score: a value that may range from 1 star to 5 stars, or scoring 1\nto 10. Fig. 22.9 shows samples extracted from restaurant, book, and movie reviews.\nMovie review excerpts (IMDb)\n10A great movie. This \ufb01lm is just a wonderful experience. It\u2019s surreal, zany, witty and slapstick\nall at the same time. And terri\ufb01c performances too.\n1This was probably the worst movie I have ever seen. The story went nowhere even though they\ncould have done some interesting stuff with it.\nRestaurant review excerpts (Yelp)\n5The service was impeccable. The food was cooked and seasoned perfectly... The watermelon\nwas perfectly square ... The grilled octopus was ... mouthwatering...\n2...it took a while to get our waters, we got our entree before our starter, and we never received\nsilverware or napkins until we requested them...\nBook review excerpts (GoodReads)\n1I am going to try and stop being deceived by eye-catching titles. I so wanted to like this book\nand was so disappointed by it.\n5This book is hilarious. I would recommend it to anyone looking for a satirical read with a\nromantic twist and a narrator that keeps butting in\nProduct review excerpts (Amazon)\n5The lid on this blender though is probably what I like the best about it... enables you to pour\ninto something without even taking the lid off! ... the perfect pitcher! ... works fantastic.\n1I hate this blender... It is nearly impossible to get frozen fruit and ice to turn into a smoothie...\nYou have to add a TON of liquid. I also wish it had a spout ...\nFigure 22.9 Excerpts from some reviews from various review websites, all on a scale of 1 to 5 stars except\nIMDb, which is on a scale of 1 to 10 stars.\nWe can use this review score as supervision: positive words are more likely to\nappear in 5-star reviews; negative words in 1-star reviews. And instead of just a\nbinary polarity, this kind of supervision allows us to assign a word a more complex\nrepresentation of its polarity: its distribution over stars (or other scores).\nThus in a ten-star system we could represent the sentiment of each word as a\n10-tuple, each number a score representing the word\u2019s association with that polarity\nlevel. This association can be a raw count, or a likelihood P(wjc), or some other\nfunction of the count, for each class cfrom 1 to 10.\nFor example, we could compute the IMDb likelihood of a word like disap-\npoint(ed/ing) occurring in a 1 star review by dividing the number of times disap-\npoint(ed/ing) occurs in 1-star reviews in the IMDb dataset (8,557) by the total num-\nber of words occurring in 1-star reviews (25,395,214), so the IMDb estimate of\nP(disappointingj1)is .0003.\nA slight modi\ufb01cation of this weighting, the normalized likelihood, can be used\nas an illuminating visualization (Potts, 2011)1\nP(wjc) =count (w;c)P\nw2Ccount (w;c)\nPottsScore (w) =P(wjc)P\ncP(wjc)(22.6)\nDividing the IMDb estimate P(disappointingj1)of .0003 by the sum of the likeli-\nhood P(wjc)over all categories gives a Potts score of 0.10. The word disappointing\n1Each element of the Potts score of a word wand category ccan be shown to be a variant of the\npointwise mutual information pmi (w;c)without the log term; see Exercise 22.1. 12 CHAPTER 22 \u2022 L EXICONS FOR SENTIMENT , AFFECT ,AND CONNOTATION\nthus is associated with the vector [.10, .12, .14, .14, .13, .11, .08, .06, .06, .05]. The\nPotts diagram (Potts, 2011) is a visualization of these word scores, representing the Potts diagram\nprior sentiment of a word as a distribution over the rating categories.\nFig. 22.10 shows the Potts diagrams for 3 positive and 3 negative scalar adjec-\ntives. Note that the curve for strongly positive scalars have the shape of the letter\nJ, while strongly negative scalars look like a reverse J. By contrast, weakly posi-\ntive and negative scalars have a hump-shape, with the maximum either below the\nmean (weakly negative words like disappointing ) or above the mean (weakly pos-\nitive words like good ). These shapes offer an illuminating typology of affective\nmeaning.\nOverviewDataMethodsCategorizationScale inductionLooking aheadExample: attenuators\nIMDB \u2013 53,775 tokens\nCategory-0.50-0.39-0.28-0.17-0.060.060.170.280.390.500.050.090.15Cat = 0.33 (p = 0.004)Cat^2 = -4.02 (p < 0.001)OpenTable \u2013 3,890 tokens\nCategory-0.50-0.250.000.250.500.080.38Cat = 0.11 (p = 0.707)Cat^2 = -6.2 (p = 0.014)Goodreads \u2013 3,424 tokens\nCategory-0.50-0.250.000.250.500.080.190.36Cat = -0.55 (p = 0.128)Cat^2 = -5.04 (p = 0.016)Amazon/Tripadvisor \u2013 2,060 tokens\nCategory-0.50-0.250.000.250.500.120.28Cat = 0.42 (p = 0.207)Cat^2 = -2.74 (p = 0.05)somewhat/r\nIMDB \u2013 33,515 tokens\nCategory-0.50-0.39-0.28-0.17-0.060.060.170.280.390.500.040.090.17Cat = -0.13 (p = 0.284)Cat^2 = -5.37 (p < 0.001)OpenTable \u2013 2,829 tokens\nCategory-0.50-0.250.000.250.500.080.31Cat = 0.2 (p = 0.265)Cat^2 = -4.16 (p = 0.007)Goodreads \u2013 1,806 tokens\nCategory-0.50-0.250.000.250.500.050.120.180.35Cat = -0.87 (p = 0.016)Cat^2 = -5.74 (p = 0.004)Amazon/Tripadvisor \u2013 2,158 tokens\nCategory-0.50-0.250.000.250.500.110.29Cat = 0.54 (p = 0.183)Cat^2 = -3.32 (p = 0.045)fairly/r\nIMDB \u2013 176,264 tokens\nCategory-0.50-0.39-0.28-0.17-0.060.060.170.280.390.500.050.090.13Cat = -0.43 (p < 0.001)Cat^2 = -3.6 (p < 0.001)OpenTable \u2013 8,982 tokens\nCategory-0.50-0.250.000.250.500.080.140.190.32Cat = -0.64 (p = 0.035)Cat^2 = -4.47 (p = 0.007)Goodreads \u2013 11,895 tokens\nCategory-0.50-0.250.000.250.500.070.150.34Cat = -0.71 (p = 0.072)Cat^2 = -4.59 (p = 0.018)Amazon/Tripadvisor \u2013 5,980 tokens\nCategory-0.50-0.250.000.250.500.150.28Cat = 0.26 (p = 0.496)Cat^2 = -2.23 (p = 0.131)pretty/r\u201cPotts&diagrams\u201dPotts,&Christopher .& 2011.&NSF&workshop&on&restructuring& adjectives.goodgreatexcellentdisappointingbadterribletotallyabsolutelyutterlysomewhatfairlyprettyPositive scalarsNegative scalarsEmphaticsAttenuators1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating\nFigure 22.10 Potts diagrams (Potts, 2011) for positive and negative scalar adjectives, show-\ning the J-shape and reverse J-shape for strongly positive and negative adjectives, and the\nhump-shape for more weakly polarized adjectives.\nFig. 22.11 shows the Potts diagrams for emphasizing and attenuating adverbs.\nNote that emphatics tend to have a J-shape (most likely to occur in the most posi-\ntive reviews) or a U-shape (most likely to occur in the strongly positive and nega-\ntive). Attenuators all have the hump-shape, emphasizing the middle of the scale and\ndownplaying both extremes. The diagrams can be used both as a typology of lexical\nsentiment, and also play a role in modeling sentiment compositionality.\nIn addition to functions like posterior P(cjw), likelihood P(wjc), or normalized\nlikelihood (Eq. 22.6) many other functions of the count of a word occurring with a\nsentiment label have been used. We\u2019ll introduce some of these on page 16, including\nideas like normalizing the counts per writer in Eq. 22.14.\n22.5.1 Log Odds Ratio Informative Dirichlet Prior\nOne thing we often want to do with word polarity is to distinguish between words\nthat are more likely to be used in one category of texts than in another. We may, for\nexample, want to know the words most associated with 1 star reviews versus those\nassociated with 5 star reviews. These differences may not be just related to senti-\nment. We might want to \ufb01nd words used more often by Democratic than Republican 22.5 \u2022 S UPERVISED LEARNING OF WORD SENTIMENT 13\nOverviewDataMethodsCategorizationScale inductionLooking aheadExample: attenuators\nIMDB \u2013 53,775 tokens\nCategory-0.50-0.39-0.28-0.17-0.060.060.170.280.390.500.050.090.15Cat = 0.33 (p = 0.004)Cat^2 = -4.02 (p < 0.001)OpenTable \u2013 3,890 tokens\nCategory-0.50-0.250.000.250.500.080.38Cat = 0.11 (p = 0.707)Cat^2 = -6.2 (p = 0.014)Goodreads \u2013 3,424 tokens\nCategory-0.50-0.250.000.250.500.080.190.36Cat = -0.55 (p = 0.128)Cat^2 = -5.04 (p = 0.016)Amazon/Tripadvisor \u2013 2,060 tokens\nCategory-0.50-0.250.000.250.500.120.28Cat = 0.42 (p = 0.207)Cat^2 = -2.74 (p = 0.05)somewhat/r\nIMDB \u2013 33,515 tokens\nCategory-0.50-0.39-0.28-0.17-0.060.060.170.280.390.500.040.090.17Cat = -0.13 (p = 0.284)Cat^2 = -5.37 (p < 0.001)OpenTable \u2013 2,829 tokens\nCategory-0.50-0.250.000.250.500.080.31Cat = 0.2 (p = 0.265)Cat^2 = -4.16 (p = 0.007)Goodreads \u2013 1,806 tokens\nCategory-0.50-0.250.000.250.500.050.120.180.35Cat = -0.87 (p = 0.016)Cat^2 = -5.74 (p = 0.004)Amazon/Tripadvisor \u2013 2,158 tokens\nCategory-0.50-0.250.000.250.500.110.29Cat = 0.54 (p = 0.183)Cat^2 = -3.32 (p = 0.045)fairly/r\nIMDB \u2013 176,264 tokens\nCategory-0.50-0.39-0.28-0.17-0.060.060.170.280.390.500.050.090.13Cat = -0.43 (p < 0.001)Cat^2 = -3.6 (p < 0.001)OpenTable \u2013 8,982 tokens\nCategory-0.50-0.250.000.250.500.080.140.190.32Cat = -0.64 (p = 0.035)Cat^2 = -4.47 (p = 0.007)Goodreads \u2013 11,895 tokens\nCategory-0.50-0.250.000.250.500.070.150.34Cat = -0.71 (p = 0.072)Cat^2 = -4.59 (p = 0.018)Amazon/Tripadvisor \u2013 5,980 tokens\nCategory-0.50-0.250.000.250.500.150.28Cat = 0.26 (p = 0.496)Cat^2 = -2.23 (p = 0.131)pretty/r\u201cPotts&diagrams\u201dPotts,&Christopher .& 2011.&NSF&workshop&on&restructuring& adjectives.goodgreatexcellentdisappointingbadterribletotallyabsolutelyutterlysomewhatfairlyprettyPositive scalarsNegative scalarsEmphaticsAttenuators1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating1  2  3  4  5  6  7  8  9  10rating\nFigure 22.11 Potts diagrams (Potts, 2011) for emphatic and attenuating adverbs.\nmembers of Congress, or words used more often in menus of expensive restaurants\nthan cheap restaurants.\nGiven two classes of documents, to \ufb01nd words more associated with one cate-\ngory than another, we could measure the difference in frequencies (is a word wmore\nfrequent in class Aor class B?). Or instead of the difference in frequencies we could\ncompute the ratio of frequencies, or compute the log odds ratio (the log of the ratio\nbetween the odds of the two words). We could then sort words by whichever associ-\nation measure we pick, ranging from words overrepresented in category Ato words\noverrepresented in category B.\nThe problem with simple log-likelihood or log odds methods is that they overem-\nphasize differences in very rare words, and often also in very frequent words. Very\nrare words will seem to occur very differently in the two corpora since with tiny\ncounts there may be statistical \ufb02uctations, or even zero occurrences in one corpus\ncompared to non-zero occurrences in the other. Very frequent words will also seem\ndifferent since all counts are large.\nIn this section we walk through the details of one solution to this problem: the\n\u201clog odds ratio informative Dirichlet prior\u201d method of Monroe et al. (2008) that is a\nparticularly useful method for \ufb01nding words that are statistically overrepresented in\none particular category of texts compared to another. It\u2019s based on the idea of using\nanother large corpus to get a prior estimate of what we expect the frequency of each\nword to be.\nLet\u2019s start with the goal: assume we want to know whether the word horrible\noccurs more in corpus ior corpus j. We could compute the log likelihood ratio ,log likelihood\nratio\nusing fi(w)to mean the frequency of word win corpus i, and nito mean the total\nnumber of words in corpus i:\nllr(horrible ) = logPi(horrible )\nPj(horrible )\n=logPi(horrible )\u0000logPj(horrible )\n=logfi(horrible )\nni\u0000logfj(horrible )\nnj(22.7) 14 CHAPTER 22 \u2022 L EXICONS FOR SENTIMENT , AFFECT ,AND CONNOTATION\nInstead, let\u2019s compute the log odds ratio : does horrible have higher odds in ior in log odds ratio\nj:\nlor(horrible ) = log\u0012Pi(horrible )\n1\u0000Pi(horrible )\u0013\n\u0000log\u0012Pj(horrible )\n1\u0000Pj(horrible )\u0013\n=log0\nB@fi(horrible )\nni\n1\u0000fi(horrible )\nni1\nCA\u0000log0\nB@fj(horrible )\nnj\n1\u0000fj(horrible )\nnj1\nCA\n=log\u0012fi(horrible )\nni\u0000fi(horrible )\u0013\n\u0000log\u0012fj(horrible )\nnj\u0000fj(horrible )\u0013\n(22.8)\nThe Dirichlet intuition is to use a large background corpus to get a prior estimate of\nwhat we expect the frequency of each word wto be. We\u2019ll do this very simply by\nadding the counts from that corpus to the numerator and denominator, so that we\u2019re\nessentially shrinking the counts toward that prior. It\u2019s like asking how large are the\ndifferences between iand jgiven what we would expect given their frequencies in\na well-estimated large background corpus.\nThe method estimates the difference between the frequency of word win two\ncorpora iandjvia the prior-modi\ufb01ed log odds ratio for w,d(i\u0000j)\nw, which is estimated\nas:\nd(i\u0000j)\nw =log\u0012fi\nw+aw\nni+a0\u0000(fiw+aw)\u0013\n\u0000log \nfj\nw+aw\nnj+a0\u0000(fj\nw+aw)!\n(22.9)\n(where niis the size of corpus i,njis the size of corpus j,fi\nwis the count of word\nwin corpus i,fj\nwis the count of word win corpus j,a0is the scaled size of the\nbackground corpus, and awis the scaled count of word win the background corpus.)\nIn addition, Monroe et al. (2008) make use of an estimate for the variance of the\nlog\u2013odds\u2013ratio:\ns2\u0010\n\u02c6d(i\u0000j)\nw\u0011\n\u00191\nfiw+aw+1\nfj\nw+aw(22.10)\nThe \ufb01nal statistic for a word is then the z\u2013score of its log\u2013odds\u2013ratio:\n\u02c6d(i\u0000j)\nwr\ns2\u0010\n\u02c6d(i\u0000j)\nw\u0011(22.11)\nThe Monroe et al. (2008) method thus modi\ufb01es the commonly used log odds ratio\nin two ways: it uses the z-scores of the log odds ratio, which controls for the amount\nof variance in a word\u2019s frequency, and it uses counts from a background corpus to\nprovide a prior count for words.\nFig. 22.12 shows the method applied to a dataset of restaurant reviews from\nYelp, comparing the words used in 1-star reviews to the words used in 5-star reviews\n(Jurafsky et al., 2014). The largest difference is in obvious sentiment words, with the\n1-star reviews using negative sentiment words like worse, bad, awful and the 5-star\nreviews using positive sentiment words like great, best, amazing . But there are other\nilluminating differences. 1-star reviews use logical negation ( no, not ), while 5-star\nreviews use emphatics and emphasize universality ( very, highly, every, always ). 1-\nstar reviews use \ufb01rst person plurals ( we, us, our ) while 5 star reviews use the second 22.6 \u2022 U SING LEXICONS FOR SENTIMENT RECOGNITION 15\nperson. 1-star reviews talk about people ( manager, waiter, customer ) while 5-star\nreviews talk about dessert and properties of expensive restaurants like courses and\natmosphere. See Jurafsky et al. (2014) for more details.\nClass Words in 1-star reviews Class Words in 5-star reviews\nNegative worst, rude, terrible, horrible, bad,\nawful, disgusting, bland, tasteless,\ngross, mediocre, overpriced, worse,\npoorPositive great, best, love(d), delicious, amazing,\nfavorite, perfect, excellent, awesome,\nfriendly, fantastic, fresh, wonderful, in-\ncredible, sweet, yum(my)\nNegation no, not Emphatics/\nuniversalsvery, highly, perfectly, de\ufb01nitely, abso-\nlutely, everything, every, always\n1Pl pro we, us, our 2 pro you\n3 pro she, he, her, him Articles a, the\nPast verb was, were, asked, told, said, did,\ncharged, waited, left, tookAdvice try, recommend\nSequencers after, then Conjunct also, as, well, with, and\nNouns manager, waitress, waiter, customer,\ncustomers, attitude, waste, poisoning,\nmoney, bill, minutesNouns atmosphere, dessert, chocolate, wine,\ncourse, menu\nIrrealis\nmodalswould, should Auxiliaries is/\u2019s, can, \u2019ve, are\nComp to, that Prep, other in, of, die, city, mouth\nFigure 22.12 The top 50 words associated with one\u2013star and \ufb01ve-star restaurant reviews in a Yelp dataset of\n900,000 reviews, using the Monroe et al. (2008) method (Jurafsky et al., 2014).\n22.6 Using Lexicons for Sentiment Recognition\nIn Chapter 4 we introduced the naive Bayes algorithm for sentiment analysis. The\nlexicons we have focused on throughout the chapter so far can be used in a number\nof ways to improve sentiment detection.\nIn the simplest case, lexicons can be used when we don\u2019t have suf\ufb01cient training\ndata to build a supervised sentiment analyzer; it can often be expensive to have a\nhuman assign sentiment to each document to train the supervised classi\ufb01er.\nIn such situations, lexicons can be used in a rule-based algorithm for classi\ufb01ca-\ntion. The simplest version is just to use the ratio of positive to negative words: if a\ndocument has more positive than negative words (using the lexicon to decide the po-\nlarity of each word in the document), it is classi\ufb01ed as positive. Often a threshold l\nis used, in which a document is classi\ufb01ed as positive only if the ratio is greater than\nl. If the sentiment lexicon includes positive and negative weights for each word,\nq+\nwandq\u0000\nw, these can be used as well. Here\u2019s a simple such sentiment algorithm:\nf+=X\nws.t.w2positivelexiconq+\nwcount (w)\nf\u0000=X\nws.t.w2negativelexiconq\u0000\nwcount (w)\nsentiment =8\n>>><\n>>>:+iff+\nf\u0000>l\n\u0000iff\u0000\nf+>l\n0 otherwise.(22.12) 16 CHAPTER 22 \u2022 L EXICONS FOR SENTIMENT , AFFECT ,AND CONNOTATION\nIf supervised training data is available, these counts computed from sentiment lex-\nicons, sometimes weighted or normalized in various ways, can also be used as fea-\ntures in a classi\ufb01er along with other lexical or non-lexical features. We return to\nsuch algorithms in Section 22.7.\n22.7 Using Lexicons for Affect Recognition\nDetection of emotion (and the other kinds of affective meaning described by Scherer\n(2000)) can be done by generalizing the algorithms described above for detecting\nsentiment.\nThe most common algorithms involve supervised classi\ufb01cation: a training set is\nlabeled for the affective meaning to be detected, and a classi\ufb01er is built using features\nextracted from the training set. As with sentiment analysis, if the training set is large\nenough, and the test set is suf\ufb01ciently similar to the training set, simply using all\nthe words or all the bigrams as features in a powerful classi\ufb01er like SVM or logistic\nregression, as described in Fig. ??in Chapter 4, is an excellent algorithm whose\nperformance is hard to beat. Thus we can treat affective meaning classi\ufb01cation of a\ntext sample as simple document classi\ufb01cation.\nSome modi\ufb01cations are nonetheless often necessary for very large datasets. For\nexample, the Schwartz et al. (2013) study of personality, gender, and age using 700\nmillion words of Facebook posts used only a subset of the n-grams of lengths 1-\n3. Only words and phrases used by at least 1% of the subjects were included as\nfeatures, and 2-grams and 3-grams were only kept if they had suf\ufb01ciently high PMI\n(PMI greater than 2 \u0003length , where length is the number of words):\npmi(phrase ) =logp(phrase )Y\nw2phrasep(w)(22.13)\nVarious weights can be used for the features, including the raw count in the training\nset, or some normalized probability or log probability. Schwartz et al. (2013), for\nexample, turn feature counts into phrase likelihoods by normalizing them by each\nsubject\u2019s total word use.\np(phrasejsubject ) =freq(phrase ;subject )X\nphrase02vocab (subject )freq(phrase0;subject )(22.14)\nIf the training data is sparser, or not as similar to the test set, any of the lexicons\nwe\u2019ve discussed can play a helpful role, either alone or in combination with all the\nwords and n-grams.\nMany possible values can be used for lexicon features. The simplest is just an\nindicator function, in which the value of a feature fLtakes the value 1 if a particular\ntext has any word from the relevant lexicon L. Using the notation of Chapter 4, in\nwhich a feature value is de\ufb01ned for a particular output class cand document x.\nfL(c;x) =\u001a1 if9w:w2L&w2x&class =c\n0 otherwise 22.8 \u2022 L EXICON -BASED METHODS FOR ENTITY -CENTRIC AFFECT 17\nAlternatively the value of a feature fLfor a particular lexicon Lcan be the total\nnumber of word tokens in the document that occur in L:\nfL=X\nw2Lcount (w)\nFor lexica in which each word is associated with a score or weight, the count can be\nmultiplied by a weight qL\nw:\nfL=X\nw2LqL\nwcount (w)\nCounts can alternatively be logged or normalized per writer as in Eq. 22.14.\nHowever they are de\ufb01ned, these lexicon features are then used in a supervised\nclassi\ufb01er to predict the desired affective category for the text or document. Once\na classi\ufb01er is trained, we can examine which lexicon features are associated with\nwhich classes. For a classi\ufb01er like logistic regression the feature weight gives an\nindication of how associated the feature is with the class.\n22.8 Lexicon-based methods for Entity-Centric Affect\nWhat if we want to get an affect score not for an entire document, but for a particular\nentity in the text? The entity-centric method of Field and Tsvetkov (2019) combines\naffect lexicons with contextual embeddings to assign an affect score to an entity in\ntext. In the context of affect about people, they relabel the Valence/Arousal/Dominance\ndimension as Sentiment/Agency/Power. The algorithm \ufb01rst trains classi\ufb01ers to map\nembeddings to scores:\n1. For each word win the training corpus:\n(a) Use off-the-shelf pretrained encoders (like BERT) to extract a contextual\nembedding efor each instance of the word. No additional \ufb01ne-tuning is\ndone.\n(b) Average over the eembeddings of each instance of wto obtain a single\nembedding vector for one training point w.\n(c) Use the NRC V AD Lexicon to get S, A, and P scores for w.\n2. Train (three) regression models on all words wto predict V , A, D scores from\na word\u2019s average embedding.\nNow given an entity mention min a text, we assign affect scores as follows:\n1. Use the same pretrained LM to get contextual embeddings for min context.\n2. Feed this embedding through the 3 regression models to get S, A, P scores for\nthe entity.\nThis results in a (S,A,P) tuple for a given entity mention; To get scores for the rep-\nresentation of an entity in a complete document, we can run coreference resolution\nand average the (S,A,P) scores for all the mentions. Fig. 22.13 shows the scores\nfrom their algorithm for characters from the movie The Dark Knight when run on\nWikipedia plot summary texts with gold coreference. 18 CHAPTER 22 \u2022 L EXICONS FOR SENTIMENT , AFFECT ,AND CONNOTATION\nPower ScoreweaklyRachelDentGordanBatmanJokerpowerfully\nSentiment ScorenegativeJokerDentGordanRachelBatmanpositive\nAgency ScoredullDentGordanRachelBatmanJokerscaryFigure 1: Power, sentiment, and agency scores for char-\nacters in The Dark Night as learned through the regres-\nsion model with ELMo embeddings. Scores generally\nalign with character archetypes, i.e. the antagonist has\nthe lowest sentiment score.\nment have resulted in his effective removal from\nthe industry. While articles about the #MeToo\nmovement portray men like Weinstein as unpow-\nerful, we can speculate that the corpora used to\ntrain ELMo and BERT portray them as powerful.\nThus, in a corpus where traditional power roles\nhave been inverted, the embeddings extracted\nfrom ELMo and BERT perform worse than ran-\ndom, as they are biased towards the power struc-\ntures in the data they are trained on. Further ev-\nidence of this exists in the performance of the\nBERT-masked embeddings - whereas these em-\nbeddings generally capture power poorly as com-\npared to the unmasked embeddings (Table 2),\nthey outperform the unmasked embeddings on this\ntask, and even outperform the frequency baseline\nin one setting. Nevertheless, they do not outper-\nform Field et al. (2019 ), likely because they do not\ncapture affect information as well as the unmasked\nembeddings (Table 2).\n4.3 Qualitative Document-level Analysis\nFinally, we qualitatively analyze how well our\nmethod captures affect dimensions by analyzing\nsingle documents in detail. We conduct this anal-\nysis in a domain where we expect entities to ful\ufb01ll\ntraditional power roles and where entity portray-\nals are known. Following Bamman et al. (2013 ),\nwe analyze the Wikipedia plot summary of the\nmovie The Dark Knight ,7focusing on Batman\n(protagonist),8the Joker (antagonist), Jim Gordan\n(law enforcement of\ufb01cer, ally to Batman), Har-\n7http://bit.ly/2XmhRDR\n8We consider Batman/Bruce Wayne to be the same entity.Power Scoreweakly Rachel Joker Dent Gordan Batmanpowerfully\nSentiment Scorenegative Joker Gordan Batman Dent Rachel positive\nAgency Scoredull Rachel Dent GordanBatman Joker scaryFigure 2: Power, sentiment, and agency scores for char-acters inThe Dark Nightas learned through ASP withELMo embeddings. These scores re\ufb02ect the same pat-terns as the regression model with greater separationbetween characters.vey Dent (ally to Batman who turns evil) andRachel Dawes (primary love interest). To facil-itate extracting example sentences, we score eachinstance of these entities in the narrative separatelyand average across instances to obtain an entityscore for the document.9To maximize our databy capturing every mention of an entity, we per-form co-reference resolution by hand. Addition-ally, based on our results from Table3as well asthe use of Wikipedia data in training the ELMomodel (Peters et al.,2018), we use ELMo embed-dings for our analysis.Figures1and2show results. For refer-ence, we show the entity scores as compared toone polar opposite pair identi\ufb01ed by ASP. Boththe regression model and ASP show similar pat-terns. Batman has high power, while Rachel haslow power. Additionally, the Joker is associatedwith the most negative sentiment, but the high-est agency. Throughout the plot summary, themovie progresses by the Joker taking an aggres-sive action and the other characters responding.We can see this dynamic re\ufb02ected in the Joker\u2019spro\ufb01le score, as a high-powered, high-agency,low-sentiment character, who is the primary plot-driver. In general, ASP shows a greater separationbetween characters than the regression model. Wehypothesize that this occurs because ASP isolatesthe dimensions of interest, while the regression ap-proach captures other confounds, such as that hu-9When we used this averaging metric in other evaluations,we found no signi\ufb01cant change in results. Thus, in other sce-narios, we compute scores over averaged embeddings, ratherthan averaging scores separately computed for each embed-ding to reduce computationally complexity.\nFigure 22.13 Power (dominance), sentiment (valence) and agency (arousal) for characters\nin the movie The Dark Knight computed from embeddings trained on the NRC V AD Lexicon.\nNote the protagonist (Batman) and the antagonist (the Joker) have high power and agency\nscores but differ in sentiment, while the love interest Rachel has low power and agency but\nhigh sentiment.\n22.9 Connotation Frames\nThe lexicons we\u2019ve described so far de\ufb01ne a word as a point in affective space. A\nconnotation frame , by contrast, is a lexicon that incorporates a richer kind of gram-connotation\nframe\nmatical structure, by combining affective lexicons with the frame semantic lexicons\nof Chapter 21. The basic insight of connotation frame lexicons is that a predicate\nlike a verb expresses connotations about the verb\u2019s arguments (Rashkin et al. 2016,\nRashkin et al. 2017).\nConsider sentences like:\n(22.15) Country A violated the sovereignty of Country B\n(22.16) the teenager ... survived the Boston Marathon bombing\u201d\nBy using the verb violate in (22.15), the author is expressing their sympathies with\nCountry B, portraying Country B as a victim, and expressing antagonism toward\nthe agent Country A. By contrast, in using the verb survive , the author of (22.16) is\nexpressing that the bombing is a negative experience, and the subject of the sentence,\nthe teenager, is a sympathetic character. These aspects of connotation are inherent\nin the meaning of the verbs violate andsurvive , as shown in Fig. 22.14.\nThe connotation frame lexicons of Rashkin et al. (2016) and Rashkin et al.\n(2017) also express other connotative aspects of the predicate toward each argu-\nment, including the effect (something bad happened to x) value : (x is valuable), and\nmental state : (x is distressed by the event). Connotation frames can also mark the\npower differential between the arguments (using the verb implore means that the\ntheme argument has greater power than the agent), and the agency of each argument\n(waited is low agency). Fig. 22.15 shows a visualization from Sap et al. (2017).\nConnotation frames can be built by hand (Sap et al., 2017), or they can be learned\nby supervised learning (Rashkin et al., 2016), for example using hand-labeled train-\ning data to supervise classi\ufb01ers for each of the individual relations, e.g., whether\nS(writer!Role1) is + or -, and then improving accuracy via global constraints\nacross all relations. 22.10 \u2022 S UMMARY 19\nWriter\nRole1\nRole2Role1 is asympathetic victimThere issome typeof hardship\nReader+_+__S(writer\u2192role1)S(writer\u2192role2)Connotation Frame for \u201cRole1 survives Role2\u201d S(role1\u2192role2)\nWriter\nRole1\nRole2Role1 is the antagonistRole2 is asympathetic victim\nReader+_+__S(writer\u2192role1)S(writer\u2192role2)Connotation Frame for \u201cRole1 violates Role2\u201d S(role1\u2192role2)\n(a) (b)\nFigure 22.14 Connotation frames for survive andviolate . (a) For survive , the writer and reader have positive\nsentiment toward Role1, the subject, and negative sentiment toward Role2, the direct object. (b) For violate , the\nwriter and reader have positive sentiment instead toward Role2, the direct object.\nAGENTTHEMEpower(AG < TH)VERBimploreHe implored the tribunal to show mercy.The princess waited for her prince.AGENTTHEMEagency(AG) = -VERBwaitFigure 2: The formal notation of the connotation\nframes of power and agency. The \ufb01rst example\nshows the relative power differential implied by\nthe verb \u201cimplored\u201d , i.e., the agent (\u201che\u201d) is in\na position of less power than the theme (\u201cthe tri-\nbunal\u201d). In contrast, \u201cHe demanded the tribunal\nshow mercy\u201d implies that the agent has authority\nover the theme. The second example shows the\nlow level of agency implied by the verb \u201cwaited\u201d .\ninteractive demo website of our \ufb01ndings (see Fig-\nure5in the appendix for a screenshot).2Further-\nmore, as will be seen in Section 4.1, connotation\nframes offer new insights that complement and de-\nviate from the well-known Bechdel test ( Bechdel ,\n1986 ). In particular, we \ufb01nd that high-agency\nwomen through the lens of connotation frames are\nrare in modern \ufb01lms. It is, in part, because some\nmovies (e.g., Snow White) accidentally pass the\nBechdel test and also because even movies with\nstrong female characters are not entirely free from\nthe deeply ingrained biases in social norms.\n2 Connotation Frames of Power and\nAgency\nWe create two new connotation relations, power\nandagency (examples in Figure 3), as an expan-\nsion of the existing connotation frame lexicons.3\nThree AMT crowdworkers annotated the verbs\nwith placeholders to avoid gender bias in the con-\ntext (e.g., Xrescued Y; an example task is shown\nin the appendix in Figure 7). We de\ufb01ne the anno-\ntated constructs as follows:\nPower Differentials Many verbs imply the au-\nthority levels of the agent and theme relative to\n2http://homes .cs.washington .edu/ \u02dcmsap/\nmovie-bias/ .\n3The lexicons and a demo are available at http://\nhomes .cs.washington .edu/ \u02dcmsap/movie-bias/ .power (AG<TH)power (AG>TH)\nagency (AG)=\u0000agency (AG)=+Figure 3: Sample verbs in the connotation frames\nwith high annotator agreement. Size is indicative\nof verb frequency in our corpus (bigger =more\nfrequent), color differences are only for legibility.\none another. For example, if the agent \u201cdom-\ninates\u201d the theme (denoted as power (AG>TH)),\nthen the agent is implied to have a level of control\nover the theme. Alternatively, if the agent \u201chon-\nors\u201d the theme (denoted as power (AG<TH)), the\nwriter implies that the theme is more important or\nauthoritative. We used AMT crowdsourcing to la-\nbel 1700 transitive verbs for power differentials.\nWith three annotators per verb, the inter-annotator\nagreement is 0.34 (Krippendorff\u2019s \u21b5).\nAgency The agency attributed to the agent of the\nverb denotes whether the action being described\nimplies that the agent is powerful, decisive, and\ncapable of pushing forward their own storyline.\nFor example, a person who is described as \u201cex-\nperiencing\u201d things does not seem as active and de-\ncisive as someone who is described as \u201cdetermin-\ning\u201d things. AMT workers labeled 2000 transi-\ntive verbs for implying high/moderate/low agency\n(inter-annotator agreement of 0.27). We denote\nhigh agency as agency (AG)=+, and low agency\nasagency (AG)=\u0000.\nPairwise agreements on a hard constraint are\n56% and 51% for power and agency, respec-\ntively. Despite this, agreements reach 96% and\n94% when moderate labels are counted as agree-\ning with either high or low labels, showing that an-\nnotators rarely strongly disagree with one another.\nSome contributing factors in the lower KA scores\ninclude the subtlety of choosing between neutral\nFigure 22.15 The connotation frames of Sap et al. (2017), showing that the verb implore\nimplies the agent has lower power than the theme (in contrast, say, with a verb like demanded ),\nand showing the low level of agency of the subject of waited . Figure from Sap et al. (2017).\n22.10 Summary\n\u2022 Many kinds of affective states can be distinguished, including emotions ,moods ,\nattitudes (which include sentiment ),interpersonal stance , and personality .\n\u2022Emotion can be represented by \ufb01xed atomic units often called basic emo-\ntions , or as points in space de\ufb01ned by dimensions like valence andarousal .\n\u2022 Words have connotational aspects related to these affective states, and this\nconnotational aspect of word meaning can be represented in lexicons.\n\u2022 Affective lexicons can be built by hand, using crowd sourcing to label the\naffective content of each word.\n\u2022 Lexicons can be built with semi-supervised , bootstrapping from seed words\nusing similarity metrics like embedding cosine.\n\u2022 Lexicons can be learned in a fully supervised manner, when a convenient\ntraining signal can be found in the world, such as ratings assigned by users on\na review site.\n\u2022 Words can be assigned weights in a lexicon by using various functions of word\ncounts in training texts, and ratio metrics like log odds ratio informative\nDirichlet prior .\n\u2022 Affect can be detected, just like sentiment, by using standard supervised text\nclassi\ufb01cation techniques, using all the words or bigrams in a text as features. 20 CHAPTER 22 \u2022 L EXICONS FOR SENTIMENT , AFFECT ,AND CONNOTATION\nAdditional features can be drawn from counts of words in lexicons.\n\u2022 Lexicons can also be used to detect affect in a rule-based classi\ufb01er by picking\nthe simple majority sentiment based on counts of words in each lexicon.\n\u2022Connotation frames express richer relations of affective meaning that a pred-\nicate encodes about its arguments.\nBibliographical and Historical Notes\nThe idea of formally representing the subjective meaning of words began with Os-\ngood et al. (1957), the same pioneering study that \ufb01rst proposed the vector space\nmodel of meaning described in Chapter 6. Osgood et al. (1957) had participants rate\nwords on various scales, and ran factor analysis on the ratings. The most signi\ufb01cant\nfactor they uncovered was the evaluative dimension, which distinguished between\npairs like good/bad ,valuable/worthless ,pleasant/unpleasant . This work in\ufb02uenced\nthe development of early dictionaries of sentiment and affective meaning in the \ufb01eld\nofcontent analysis (Stone et al., 1966).\nWiebe (1994) began an in\ufb02uential line of work on detecting subjectivity in text, subjectivity\nbeginning with the task of identifying subjective sentences and the subjective char-\nacters who are described in the text as holding private states, beliefs or attitudes.\nLearned sentiment lexicons such as the polarity lexicons of Hatzivassiloglou and\nMcKeown (1997) were shown to be a useful feature in subjectivity detection (Hatzi-\nvassiloglou and Wiebe 2000, Wiebe 2000).\nThe term sentiment seems to have been introduced in 2001 by Das and Chen\n(2001), to describe the task of measuring market sentiment by looking at the words in\nstock trading message boards. In the same paper Das and Chen (2001) also proposed\nthe use of a sentiment lexicon. The list of words in the lexicon was created by\nhand, but each word was assigned weights according to how much it discriminated\na particular class (say buy versus sell) by maximizing across-class variation and\nminimizing within-class variation. The term sentiment , and the use of lexicons,\ncaught on quite quickly (e.g., inter alia, Turney 2002). Pang et al. (2002) \ufb01rst showed\nthe power of using all the words without a sentiment lexicon; see also Wang and\nManning (2012).\nMost of the semi-supervised methods we describe for extending sentiment dic-\ntionaries drew on the early idea that synonyms and antonyms tend to co-occur in the\nsame sentence (Miller and Charles 1991, Justeson and Katz 1991, Riloff and Shep-\nherd 1997). Other semi-supervised methods for learning cues to affective mean-\ning rely on information extraction techniques, like the AutoSlog pattern extractors\n(Riloff and Wiebe, 2003). Graph based algorithms for sentiment were \ufb01rst sug-\ngested by Hatzivassiloglou and McKeown (1997), and graph propagation became\na standard method (Zhu and Ghahramani 2002, Zhu et al. 2003, Zhou et al. 2004,\nVelikovich et al. 2010). Crowdsourcing can also be used to improve precision by\n\ufb01ltering the result of semi-supervised lexicon learning (Riloff and Shepherd 1997,\nFast et al. 2016).\nMuch recent work focuses on ways to learn embeddings that directly encode sen-\ntiment or other properties, such as the D ENSIFIER algorithm of Rothe et al. (2016)\nthat learns to transform the embedding space to focus on sentiment (or other) infor-\nmation. EXERCISES 21\nExercises\n22.1 Show that the relationship between a word wand a category cin the Potts\nScore in Eq. 22.6 is a variant of the pointwise mutual information pmi (w;c)\nwithout the log term. 22 Chapter 22 \u2022 Lexicons for Sentiment, Affect, and Connotation\nAn, J., H. Kwak, and Y .-Y . Ahn. 2018. SemAxis: A\nlightweight framework to characterize domain-speci\ufb01c\nword semantics beyond sentiment. ACL.\nBaccianella, S., A. Esuli, and F. Sebastiani. 2010. Senti-\nwordnet 3.0: An enhanced lexical resource for sentiment\nanalysis and opinion mining. LREC .\nBarrett, L. F., B. Mesquita, K. N. Ochsner, and J. J. Gross.\n2007. The experience of emotion. Annual Review of Psy-\nchology , 58:373\u2013403.\nBrysbaert, M., A. B. Warriner, and V . Kuperman. 2014.\nConcreteness ratings for 40 thousand generally known\nEnglish word lemmas. Behavior Research Methods ,\n46(3):904\u2013911.\nDas, S. R. and M. Y . Chen. 2001. Yahoo! for Ama-\nzon: Sentiment parsing from small talk on the web.\nEFA 2001 Barcelona Meetings. http://ssrn.com/\nabstract=276189 .\nEkman, P. 1999. Basic emotions. In T. Dalgleish and M. J.\nPower, eds, Handbook of Cognition and Emotion , 45\u201360.\nWiley.\nFast, E., B. Chen, and M. S. Bernstein. 2016. Empath: Un-\nderstanding Topic Signals in Large-Scale Text. CHI.\nField, A. and Y . Tsvetkov. 2019. Entity-centric contextual\naffective analysis. ACL.\nHamilton, W. L., K. Clark, J. Leskovec, and D. Jurafsky.\n2016. Inducing domain-speci\ufb01c sentiment lexicons from\nunlabeled corpora. EMNLP .\nHatzivassiloglou, V . and K. McKeown. 1997. Predicting the\nsemantic orientation of adjectives. ACL.\nHatzivassiloglou, V . and J. Wiebe. 2000. Effects of adjec-\ntive orientation and gradability on sentence subjectivity.\nCOLING .\nHellrich, J., S. Buechel, and U. Hahn. 2019. Modeling word\nemotion in historical language: Quantity beats supposed\nstability in seed word selection. 3rd Joint SIGHUM Work-\nshop on Computational Linguistics for Cultural Heritage,\nSocial Sciences, Humanities and Literature .\nHu, M. and B. Liu. 2004. Mining and summarizing customer\nreviews. SIGKDD-04 .\nJurafsky, D., V . Chahuneau, B. R. Routledge, and N. A.\nSmith. 2014. Narrative framing of consumer sentiment\nin online restaurant reviews. First Monday , 19(4).\nJusteson, J. S. and S. M. Katz. 1991. Co-occurrences of\nantonymous adjectives and their contexts. Computational\nlinguistics , 17(1):1\u201319.\nKim, S. M. and E. H. Hovy. 2004. Determining the sentiment\nof opinions. COLING .\nKiritchenko, S. and S. M. Mohammad. 2017. Best-worst\nscaling more reliable than rating scales: A case study on\nsentiment intensity annotation. ACL.\nLouviere, J. J., T. N. Flynn, and A. A. J. Marley. 2015. Best-\nworst scaling: Theory, methods and applications . Cam-\nbridge University Press.\nMairesse, F. and M. A. Walker. 2008. Trainable generation of\nbig-\ufb01ve personality styles through data-driven parameter\nestimation. ACL.\nMiller, G. A. and W. G. Charles. 1991. Contextual corre-\nlates of semantics similarity. Language and Cognitive\nProcesses , 6(1):1\u201328.Mohammad, S. M. 2018a. Obtaining reliable human ratings\nof valence, arousal, and dominance for 20,000 English\nwords. ACL.\nMohammad, S. M. 2018b. Word affect intensities. LREC .\nMohammad, S. M. and P. D. Turney. 2013. Crowdsourcing a\nword-emotion association lexicon. Computational Intel-\nligence , 29(3):436\u2013465.\nMonroe, B. L., M. P. Colaresi, and K. M. Quinn. 2008.\nFightin\u2019words: Lexical feature selection and evaluation\nfor identifying the content of political con\ufb02ict. Political\nAnalysis , 16(4):372\u2013403.\nMoors, A., P. C. Ellsworth, K. R. Scherer, and N. H. Frijda.\n2013. Appraisal theories of emotion: State of the art and\nfuture development. Emotion Review , 5(2):119\u2013124.\nOsgood, C. E., G. J. Suci, and P. H. Tannenbaum. 1957. The\nMeasurement of Meaning . University of Illinois Press.\nPang, B., L. Lee, and S. Vaithyanathan. 2002. Thumbs\nup? Sentiment classi\ufb01cation using machine learning tech-\nniques. EMNLP .\nPennebaker, J. W., R. J. Booth, and M. E. Francis. 2007.\nLinguistic Inquiry and Word Count: LIWC 2007 . Austin,\nTX.\nPicard, R. W. 1995. Affective computing. Technical Re-\nport 321, MIT Media Lab Perceputal Computing Techni-\ncal Report. Revised November 26, 1995.\nPlutchik, R. 1962. The emotions: Facts, theories, and a new\nmodel . Random House.\nPlutchik, R. 1980. A general psychoevolutionary theory of\nemotion. In R. Plutchik and H. Kellerman, eds, Emotion:\nTheory, Research, and Experience, Volume 1 , 3\u201333. Aca-\ndemic Press.\nPotts, C. 2011. On the negativity of negation. In N. Li and\nD. Lutz, eds, Proceedings of Semantics and Linguistic\nTheory 20 , 636\u2013659. CLC Publications, Ithaca, NY .\nRashkin, H., E. Bell, Y . Choi, and S. V olkova. 2017. Multi-\nlingual connotation frames: A case study on social media\nfor targeted sentiment analysis and forecast. ACL.\nRashkin, H., S. Singh, and Y . Choi. 2016. Connotation\nframes: A data-driven investigation. ACL.\nRiloff, E. and J. Shepherd. 1997. A corpus-based approach\nfor building semantic lexicons. EMNLP .\nRiloff, E. and J. Wiebe. 2003. Learning extraction patterns\nfor subjective expressions. EMNLP .\nRothe, S., S. Ebert, and H. Sch \u00a8utze. 2016. Ultradense Word\nEmbeddings by Orthogonal Transformation. NAACL\nHLT.\nRussell, J. A. 1980. A circumplex model of affect. Journal\nof personality and social psychology , 39(6):1161\u20131178.\nSap, M., M. C. Prasettio, A. Holtzman, H. Rashkin, and\nY . Choi. 2017. Connotation frames of power and agency\nin modern \ufb01lms. EMNLP .\nScherer, K. R. 2000. Psychological models of emotion. In\nJ. C. Borod, ed., The neuropsychology of emotion , 137\u2013\n162. Oxford.\nSchwartz, H. A., J. C. Eichstaedt, M. L. Kern, L. Dziurzyn-\nski, S. M. Ramones, M. Agrawal, A. Shah, M. Kosin-\nski, D. Stillwell, M. E. P. Seligman, and L. H. Ungar.\n2013. Personality, gender, and age in the language of\nsocial media: The open-vocabulary approach. PloS one ,\n8(9):e73791. Exercises 23\nStone, P., D. Dunphry, M. Smith, and D. Ogilvie. 1966.\nThe General Inquirer: A Computer Approach to Content\nAnalysis . MIT Press.\nTomkins, S. S. 1962. Affect, imagery, consciousness: Vol. I.\nThe positive affects . Springer.\nTurney, P. D. 2002. Thumbs up or thumbs down? Semantic\norientation applied to unsupervised classi\ufb01cation of re-\nviews. ACL.\nTurney, P. D. and M. Littman. 2003. Measuring praise and\ncriticism: Inference of semantic orientation from associ-\nation. ACM Transactions on Information Systems (TOIS) ,\n21:315\u2013346.\nVelikovich, L., S. Blair-Goldensohn, K. Hannan, and R. Mc-\nDonald. 2010. The viability of web-derived polarity lexi-\ncons. NAACL HLT .\nWang, S. and C. D. Manning. 2012. Baselines and bigrams:\nSimple, good sentiment and topic classi\ufb01cation. ACL.\nWiebe, J. 1994. Tracking point of view in narrative. Compu-\ntational Linguistics , 20(2):233\u2013287.\nWiebe, J. 2000. Learning subjective adjectives from corpora.\nAAAI .\nWiebe, J., R. F. Bruce, and T. P. O\u2019Hara. 1999. Develop-\nment and use of a gold-standard data set for subjectivity\nclassi\ufb01cations. ACL.\nWilson, T., J. Wiebe, and P. Hoffmann. 2005. Recogniz-\ning contextual polarity in phrase-level sentiment analysis.\nEMNLP .\nZhou, D., O. Bousquet, T. N. Lal, J. Weston, and\nB. Sch \u00a8olkopf. 2004. Learning with local and global con-\nsistency. NeurIPS .\nZhu, X. and Z. Ghahramani. 2002. Learning from labeled\nand unlabeled data with label propagation. Technical Re-\nport CMU-CALD-02, CMU.\nZhu, X., Z. Ghahramani, and J. Lafferty. 2003. Semi-\nsupervised learning using gaussian \ufb01elds and harmonic\nfunctions. ICML .", "23.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\n23Coreference Resolution and\nEntity Linking\nand even Stigand, the patriotic archbishop of Canterbury, found it advisable\u2013\u201d\u2019\n\u2018Found WHAT?\u2019 said the Duck.\n\u2018Found IT, \u2019 the Mouse replied rather crossly: \u2018of course you know what \u201cit\u201dmeans. \u2019\n\u2018I know what \u201cit\u201dmeans well enough, when I \ufb01nd a thing, \u2019 said the Duck: \u2018it\u2019s gener-\nally a frog or a worm. The question is, what did the archbishop \ufb01nd?\u2019\nLewis Carroll, Alice in Wonderland\nAn important component of language processing is knowing who is being talked\nabout in a text. Consider the following passage:\n(23.1) Victoria Chen , CFO of Megabucks Banking, saw her pay jump to $2.3\nmillion, as the 38-year-old became the company\u2019s president. It is widely\nknown that she came to Megabucks from rival Lotsabucks.\nEach of the underlined phrases in this passage is used by the writer to refer to\na person named Victoria Chen. We call linguistic expressions like herorVictoria\nChen mentions orreferring expressions , and the discourse entity that is referred mention\nto (Victoria Chen) the referent . (To distinguish between referring expressions and referent\ntheir referents, we italicize the former.)1Two or more referring expressions that are\nused to refer to the same discourse entity are said to corefer ; thus, Victoria Chen corefer\nandshecorefer in (23.1).\nCoreference is an important component of natural language processing. A dia-\nlogue system that has just told the user \u201cThere is a 2pm \ufb02ight on United and a 4pm\none on Cathay Paci\ufb01c\u201d must know which \ufb02ight the user means by \u201cI\u2019ll take the sec-\nond one\u201d . A question answering system that uses Wikipedia to answer a question\nabout Marie Curie must know who shewas in the sentence \u201cShe was born in War-\nsaw\u201d . And a machine translation system translating from a language like Spanish, in\nwhich pronouns can be dropped, must use coreference from the previous sentence to\ndecide whether the Spanish sentence \u2018 \u201cMe encanta el conocimiento\u201d, dice. \u2019 should\nbe translated as \u2018 \u201cI love knowledge\u201d, he says \u2019, or \u2018 \u201cI love knowledge\u201d, she says \u2019.\nIndeed, this example comes from an actual news article in El Pa \u00b4\u0131sabout a female\nprofessor and was mistranslated as \u201che\u201d in machine translation because of inaccurate\ncoreference resolution (Schiebinger, 2013).\nNatural language processing systems (and humans) interpret linguistic expres-\nsions with respect to a discourse model (Karttunen, 1969). A discourse modeldiscourse\nmodel\n(Fig. 23.1) is a mental model that the understander builds incrementally when in-\nterpreting a text, containing representations of the entities referred to in the text,\nas well as properties of the entities and relations among them. When a referent is\n\ufb01rst mentioned in a discourse, we say that a representation for it is evoked into the evoked\nmodel. Upon subsequent mention, this representation is accessed from the model. accessed\n1As a convenient shorthand, we sometimes speak of a referring expression referring to a referent, e.g.,\nsaying that sherefers to Victoria Chen. However, the reader should keep in mind that what we really\nmean is that the speaker is performing the act of referring to Victoria Chen by uttering she. 2CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\nVDiscourse Model\u201cVictoria\u201d\u201cshe\u201dcoreferrefer (evoke)refer (access)$LotsabucksMegabuckspay\nFigure 23.1 How mentions evoke and access discourse entities in a discourse model.\nReference in a text to an entity that has been previously introduced into the\ndiscourse is called anaphora , and the referring expression used is said to be an anaphora\nanaphor , or anaphoric.2In passage (23.1), the pronouns sheandherand the de\ufb01- anaphor\nnite NP the 38-year-old are therefore anaphoric. The anaphor corefers with a prior\nmention (in this case Victoria Chen ) that is called the antecedent . Not every refer- antecedent\nring expression is an antecedent. An entity that has only a single mention in a text\n(like Lotsabucks in (23.1)) is called a singleton . singleton\nIn this chapter we focus on the task of coreference resolution . Coreferencecoreference\nresolution\nresolution is the task of determining whether two mentions corefer , by which we\nmean they refer to the same entity in the discourse model (the same discourse entity ).\nThe set of coreferring expressions is often called a coreference chain or acluster .coreference\nchain\ncluster For example, in processing (23.1), a coreference resolution algorithm would need\nto \ufb01nd at least four coreference chains, corresponding to the four entities in the\ndiscourse model in Fig. 23.1.\n1.fVictoria Chen ,her,the 38-year-old ,Sheg\n2.fMegabucks Banking ,the company ,Megabucksg\n3.fher payg\n4.fLotsabucksg\nNote that mentions can be nested; for example the mention heris syntactically\npart of another mention, her pay , referring to a completely different discourse entity.\nCoreference resolution thus comprises two tasks (although they are often per-\nformed jointly): (1) identifying the mentions, and (2) clustering them into corefer-\nence chains/discourse entities.\nWe said that two mentions corefered if they are associated with the same dis-\ncourse entity . But often we\u2019d like to go further, deciding which real world entity is\nassociated with this discourse entity. For example, the mention Washington might\nrefer to the US state, or the capital city, or the person George Washington; the inter-\npretation of the sentence will of course be very different for each of these. The task\nofentity linking (Ji and Grishman, 2011) or entity resolution is the task of mapping entity linking\na discourse entity to some real-world individual.3We usually operationalize entity\n2We will follow the common NLP usage of anaphor to mean any mention that has an antecedent, rather\nthan the more narrow usage to mean only mentions (like pronouns) whose interpretation depends on the\nantecedent (under the narrower interpretation, repeated names are not anaphors).\n3Computational linguistics/NLP thus differs in its use of the term reference from the \ufb01eld of formal\nsemantics, which uses the words reference andcoreference to describe the relation between a mention\nand a real-world entity. By contrast, we follow the functional linguistics tradition in which a mention\nrefers to adiscourse entity (Webber, 1978) and the relation between a discourse entity and the real world\nindividual requires an additional step of linking . 3\nlinking or resolution by mapping to an ontology : a list of entities in the world, like\na gazeteer (Appendix F). Perhaps the most common ontology used for this task is\nWikipedia; each Wikipedia page acts as the unique id for a particular entity. Thus\nthe entity linking task of wiki\ufb01cation (Mihalcea and Csomai, 2007) is the task of de-\nciding which Wikipedia page corresponding to an individual is being referred to by\na mention. But entity linking can be done with any ontology; for example if we have\nan ontology of genes, we can link mentions of genes in text to the disambiguated\ngene name in the ontology.\nIn the next sections we introduce the task of coreference resolution in more de-\ntail, and survey a variety of architectures for resolution. We also introduce two\narchitectures for the task of entity linking.\nBefore turning to algorithms, however, we mention some important tasks we\nwill only touch on brie\ufb02y at the end of this chapter. First are the famous Winograd\nSchema problems (so-called because they were \ufb01rst pointed out by Terry Winograd\nin his dissertation). These entity coreference resolution problems are designed to be\ntoo dif\ufb01cult to be solved by the resolution methods we describe in this chapter, and\nthe kind of real-world knowledge they require has made them a kind of challenge\ntask for natural language processing. For example, consider the task of determining\nthe correct antecedent of the pronoun they in the following example:\n(23.2) The city council denied the demonstrators a permit because\na. they feared violence.\nb. they advocated violence.\nDetermining the correct antecedent for the pronoun they requires understanding\nthat the second clause is intended as an explanation of the \ufb01rst clause, and also\nthat city councils are perhaps more likely than demonstrators to fear violence and\nthat demonstrators might be more likely to advocate violence. Solving Winograd\nSchema problems requires \ufb01nding way to represent or discover the necessary real\nworld knowledge.\nA problem we won\u2019t discuss in this chapter is the related task of event corefer-\nence , deciding whether two event mentions (such as the buyand the acquisition inevent\ncoreference\nthese two sentences from the ECB+ corpus) refer to the same event:\n(23.3) AMD agreed to [ buy] Markham, Ontario-based ATI for around $5.4 billion\nin cash and stock, the companies announced Monday.\n(23.4) The [ acquisition ] would turn AMD into one of the world\u2019s largest providers\nof graphics chips.\nEvent mentions are much harder to detect than entity mentions, since they can be ver-\nbal as well as nominal. Once detected, the same mention-pair and mention-ranking\nmodels used for entities are often applied to events.\nAn even more complex kind of coreference is discourse deixis (Webber, 1988), discourse deixis\nin which an anaphor refers back to a discourse segment, which can be quite hard to\ndelimit or categorize, like the examples in (23.5) adapted from Webber (1991):\n(23.5) According to Soleil, Beau just opened a restaurant\na. But thatturned out to be a lie.\nb. But thatwas false.\nc.That struck me as a funny way to describe the situation.\nThe referent of that is a speech act (see Chapter 15) in (23.5a), a proposition in\n(23.5b), and a manner of description in (23.5c). We don\u2019t give algorithms in this\nchapter for these dif\ufb01cult types of non-nominal antecedents , but see Kolhatkar\net al. (2018) for a survey. 4CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\n23.1 Coreference Phenomena: Linguistic Background\nWe now offer some linguistic background on reference phenomena. We introduce\nthe four types of referring expressions (de\ufb01nite and inde\ufb01nite NPs, pronouns, and\nnames), describe how these are used to evoke and access entities in the discourse\nmodel, and talk about linguistic features of the anaphor/antecedent relation (like\nnumber/gender agreement, or properties of verb semantics).\n23.1.1 Types of Referring Expressions\nInde\ufb01nite Noun Phrases: The most common form of inde\ufb01nite reference in En-\nglish is marked with the determiner a(oran), but it can also be marked by a quan-\nti\ufb01er such as some or even the determiner this. Inde\ufb01nite reference generally intro-\nduces into the discourse context entities that are new to the hearer.\n(23.6) a. Mrs. Martin was so very kind as to send Mrs. Goddard a beautiful goose .\nb. He had gone round one day to bring her some walnuts .\nc. I saw this beautiful cauli\ufb02ower today.\nDe\ufb01nite Noun Phrases: De\ufb01nite reference, such as via NPs that use the English\narticle the, refers to an entity that is identi\ufb01able to the hearer. An entity can be\nidenti\ufb01able to the hearer because it has been mentioned previously in the text and\nthus is already represented in the discourse model:\n(23.7) It concerns a white stallion which I have sold to an of\ufb01cer. But the pedigree\nofthe white stallion was not fully established.\nAlternatively, an entity can be identi\ufb01able because it is contained in the hearer\u2019s\nset of beliefs about the world, or the uniqueness of the object is implied by the\ndescription itself, in which case it evokes a representation of the referent into the\ndiscourse model, as in (23.9):\n(23.8) I read about it in the New York Times .\n(23.9) Have you seen the car keys?\nThese last uses are quite common; more than half of de\ufb01nite NPs in newswire\ntexts are non-anaphoric, often because they are the \ufb01rst time an entity is mentioned\n(Poesio and Vieira 1998, Bean and Riloff 1999).\nPronouns: Another form of de\ufb01nite reference is pronominalization, used for enti-\nties that are extremely salient in the discourse, (as we discuss below):\n(23.10) Emma smiled and chatted as cheerfully as shecould,\nPronouns can also participate in cataphora , in which they are mentioned before cataphora\ntheir referents are, as in (23.11).\n(23.11) Even before shesawit, Dorothy had been thinking about the Emerald City\nevery day.\nHere, the pronouns sheanditboth occur before their referents are introduced.\nPronouns also appear in quanti\ufb01ed contexts in which they are considered to be\nbound , as in (23.12). bound\n(23.12) Every dancer brought herleft arm forward.\nUnder the relevant reading, herdoes not refer to some woman in context, but instead\nbehaves like a variable bound to the quanti\ufb01ed expression every dancer . We are not\nconcerned with the bound interpretation of pronouns in this chapter. 23.1 \u2022 C OREFERENCE PHENOMENA : LINGUISTIC BACKGROUND 5\nIn some languages, pronouns can appear as clitics attached to a word, like lo\n(\u2018it\u2019) in this Spanish example from AnCora (Recasens and Mart \u00b4\u0131, 2010):\n(23.13) La intenci \u00b4on es reconocer el gran prestigio que tiene la marat \u00b4on y unir lo\ncon esta gran carrera.\n\u2018The aim is to recognize the great prestige that the Marathon has and join jit\nwith this great race.\u201d\nDemonstrative Pronouns: Demonstrative pronouns thisandthat can appear ei-\nther alone or as determiners, for instance, this ingredient ,that spice :\n(23.14) I just bought a copy of Thoreau\u2019s Walden . I had bought one \ufb01ve years ago.\nThat one had been very tattered; this one was in much better condition.\nNote that this NP is ambiguous; in colloquial spoken English, it can be inde\ufb01nite,\nas in (23.6), or de\ufb01nite, as in (23.14).\nZero Anaphora: Instead of using a pronoun, in some languages (including Chi-\nnese, Japanese, and Italian) it is possible to have an anaphor that has no lexical\nrealization at all, called a zero anaphor or zero pronoun, as in the following Italian zero anaphor\nand Japanese examples from Poesio et al. (2016):\n(23.15) EN [John] iwent to visit some friends. On the way [he] ibought some\nwine.\nIT [Giovanni] iand`o a far visita a degli amici. Per via ficompr `o del vino.\nJA [John] i-wa yujin-o houmon-sita. Tochu-de fiwain-o ka-tta.\nor this Chinese example:\n(23.16) [ \u6211]\u524d\u4e00\u4f1a\u7cbe\u795e\u4e0a\u592a\u7d27\u5f20\u3002[0]\u73b0\u5728\u6bd4\u8f83\u5e73\u9759\u4e86\n[I] was too nervous a while ago. ... [0] am now calmer.\nZero anaphors complicate the task of mention detection in these languages.\nNames: Names (such as of people, locations, or organizations) can be used to refer\nto both new and old entities in the discourse:\n(23.17) a. Miss Woodhouse certainly had not done him justice.\nb.International Business Machines sought patent compensation\nfrom Amazon; IBM had previously sued other companies.\n23.1.2 Information Status\nThe way referring expressions are used to evoke new referents into the discourse\n(introducing new information), or access old entities from the model (old informa-\ntion), is called their information status orinformation structure . Entities can beinformation\nstatus\ndiscourse-new ordiscourse-old , and indeed it is common to distinguish at least discourse-new\ndiscourse-old three kinds of entities informationally (Prince, 1981):\nnew NPs:\nbrand new NPs: these introduce entities that are discourse-new and hearer-\nnew like a fruit orsome walnuts .\nunused NPs: these introduce entities that are discourse-new but hearer-old\n(like Hong Kong ,Marie Curie , orthe New York Times .\nold NPs: also called evoked NPs , these introduce entities that already in the dis-\ncourse model, hence are both discourse-old and hearer-old, like itin \u201cI went\nto a new restaurant. It was... \u201d. 6CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\ninferrables: these introduce entities that are neither hearer-old nor discourse-old,\nbut the hearer can infer their existence by reasoning based on other entities\nthat are in the discourse. Consider the following examples:\n(23.18) I went to a superb restaurant yesterday. The chef had just opened it.\n(23.19) Mix \ufb02our, butter and water. Knead the dough until shiny.\nNeither the chef northe dough were in the discourse model based on the \ufb01rst\nsentence of either example, but the reader can make a bridging inferencebridging\ninference\nthat these entities should be added to the discourse model and associated with\nthe restaurant and the ingredients, based on world knowledge that restaurants\nhave chefs and dough is the result of mixing \ufb02our and liquid (Haviland and\nClark 1974, Webber and Baldwin 1992, Nissim et al. 2004, Hou et al. 2018).\nThe form of an NP gives strong clues to its information status. We often talk\nabout an entity\u2019s position on the given-new dimension, the extent to which the refer- given-new\nent is given (salient in the discourse, easier for the hearer to call to mind, predictable\nby the hearer), versus new (non-salient in the discourse, unpredictable) (Chafe 1976,\nPrince 1981, Gundel et al. 1993). A referent that is very accessible (Ariel, 2001) accessible\ni.e., very salient in the hearer\u2019s mind or easy to call to mind, can be referred to with\nless linguistic material. For example pronouns are used only when the referent has\na high degree of activation or salience in the discourse model.4By contrast, less salience\nsalient entities, like a new referent being introduced to the discourse, will need to be\nintroduced with a longer and more explicit referring expression to help the hearer\nrecover the referent.\nThus when an entity is \ufb01rst introduced into a discourse its mentions are likely\nto have full names, titles or roles, or appositive or restrictive relative clauses, as in\nthe introduction of our protagonist in (23.1): Victoria Chen, CFO of Megabucks\nBanking . As an entity is discussed over a discourse, it becomes more salient to the\nhearer and its mentions on average typically becomes shorter and less informative,\nfor example with a shortened name (for example Ms. Chen ), a de\ufb01nite description\n(the 38-year-old ), or a pronoun ( sheorher) (Hawkins 1978). However, this change\nin length is not monotonic, and is sensitive to discourse structure (Grosz 1977, Re-\nichman 1985, Fox 1993).\n23.1.3 Complications: Non-Referring Expressions\nMany noun phrases or other nominals are not referring expressions, although they\nmay bear a confusing super\ufb01cial resemblance. For example in some of the earliest\ncomputational work on reference resolution, Karttunen (1969) pointed out that the\nNPa car in the following example does not create a discourse referent:\n(23.20) Janet doesn\u2019t have a car .\nand cannot be referred back to by anaphoric itorthe car :\n(23.21) * Itis a Toyota.\n(23.22) * The car is red.\nWe summarize here four common types of structures that are not counted as men-\ntions in coreference tasks and hence complicate the task of mention-detection:\n4Pronouns also usually (but not always) refer to entities that were introduced no further than one or two\nsentences back in the ongoing discourse, whereas de\ufb01nite noun phrases can often refer further back. 23.1 \u2022 C OREFERENCE PHENOMENA : LINGUISTIC BACKGROUND 7\nAppositives: An appositional structure is a noun phrase that appears next to a\nhead noun phrase, describing the head. In English they often appear in commas, like\n\u201ca unit of UAL\u201d appearing in apposition to the NP United , orCFO of Megabucks\nBanking in apposition to Victoria Chen .\n(23.23) Victoria Chen, CFO of Megabucks Banking, saw ...\n(23.24) United, a unit of UAL, matched the fares.\nAppositional NPs are not referring expressions, instead functioning as a kind of\nsupplementary parenthetical description of the head NP. Nonetheless, sometimes it\nis useful to link these phrases to an entity they describe, and so some datasets like\nOntoNotes mark appositional relationships.\nPredicative and Prenominal NPs: Predicative or attributive NPs describe prop-\nerties of the head noun. In United is a unit of UAL , the NP a unit of UAL describes\na property of United, rather than referring to a distinct entity. Thus they are not\nmarked as mentions in coreference tasks; in our example the NPs $2.3 million and\nthe company\u2019s president , are attributive, describing properties of her pay andthe\n38-year-old ; Example (23.27) shows a Chinese example in which the predicate NP\n(\u4e2d\u56fd\u6700\u5927\u7684\u57ce\u5e02; China\u2019s biggest city ) is not a mention.\n(23.25) her pay jumped to $2.3 million\n(23.26) the 38-year-old became the company\u2019s president\n(23.27)\u4e0a\u6d77\u662f[\u4e2d\u56fd\u6700\u5927\u7684\u57ce\u5e02] [Shanghai is China\u2019s biggest city ]\nExpletives: Many uses of pronouns like itin English and corresponding pronouns\nin other languages are not referential. Such expletive orpleonastic cases include expletive\nit is raining , in idioms like hit it off , or in particular syntactic situations like clefts clefts\n(23.28a) or extraposition (23.28b):\n(23.28) a. Itwas Emma Goldman who founded Mother Earth\nb.Itsurprised me that there was a herring hanging on her wall.\nGenerics: Another kind of expression that does not refer back to an entity explic-\nitly evoked in the text is generic reference. Consider (23.29).\n(23.29) I love mangos. They are very tasty.\nHere, they refers, not to a particular mango or set of mangos, but instead to the class\nof mangos in general. The pronoun youcan also be used generically:\n(23.30) In July in San Francisco youhave to wear a jacket.\n23.1.4 Linguistic Properties of the Coreference Relation\nNow that we have seen the linguistic properties of individual referring expressions\nwe turn to properties of the antecedent/anaphor pair. Understanding these properties\nis helpful both in designing novel features and performing error analyses.\nNumber Agreement: Referring expressions and their referents must generally\nagree in number; English she/her/he/him/his/it are singular, we/us/they/them are plu-\nral, and youis unspeci\ufb01ed for number. So a plural antecedent like the chefs cannot\ngenerally corefer with a singular anaphor like she. However, algorithms cannot\nenforce number agreement too strictly. First, semantically plural entities can be re-\nferred to by either itorthey:\n(23.31) IBM announced a new machine translation product yesterday. They have\nbeen working on it for 20 years. 8CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\nSecond, singular they has become much more common, in which they is used to singular they\ndescribe singular individuals, often useful because they is gender neutral. Although\nrecently increasing, singular they is quite old, part of English for many centuries.5\nPerson Agreement: English distinguishes between \ufb01rst, second, and third person,\nand a pronoun\u2019s antecedent must agree with the pronoun in person. Thus a third\nperson pronoun ( he, she, they, him, her, them, his, her, their ) must have a third person\nantecedent (one of the above or any other noun phrase). However, phenomena like\nquotation can cause exceptions; in this example I,my, and sheare coreferent:\n(23.32) \u201cI voted for Nader because he was most aligned with my values,\u201d she said.\nGender or Noun Class Agreement: In many languages, all nouns have grammat-\nical gender or noun class6and pronouns generally agree with the grammatical gender\nof their antecedent. In English this occurs only with third-person singular pronouns,\nwhich distinguish between male (he, him, his ),female (she, her ), and nonpersonal\n(it) grammatical genders. Non-binary pronouns like zeorhirmay also occur in more\nrecent texts. Knowing which gender to associate with a name in text can be complex,\nand may require world knowledge about the individual. Some examples:\n(23.33) Maryam has a theorem. She is exciting. (she=Maryam, not the theorem)\n(23.34) Maryam has a theorem. It is exciting. (it=the theorem, not Maryam)\nBinding Theory Constraints: The binding theory is a name for syntactic con-\nstraints on the relations between a mention and an antecedent in the same sentence\n(Chomsky, 1981). Oversimplifying a bit, re\ufb02exive pronouns like himself andher- re\ufb02exive\nselfcorefer with the subject of the most immediate clause that contains them (23.35),\nwhereas nonre\ufb02exives cannot corefer with this subject (23.36).\n(23.35) Janet bought herself a bottle of \ufb01sh sauce. [herself =Janet]\n(23.36) Janet bought her a bottle of \ufb01sh sauce. [her 6=Janet]\nRecency: Entities introduced in recent utterances tend to be more salient than\nthose introduced from utterances further back. Thus, in (23.37), the pronoun itis\nmore likely to refer to Jim\u2019s map than the doctor\u2019s map.\n(23.37) The doctor found an old map in the captain\u2019s chest. Jim found an even\nolder map hidden on the shelf. It described an island.\nGrammatical Role: Entities mentioned in subject position are more salient than\nthose in object position, which are in turn more salient than those mentioned in\noblique positions. Thus although the \ufb01rst sentence in (23.38) and (23.39) expresses\nroughly the same propositional content, the preferred referent for the pronoun he\nvaries with the subject\u2014John in (23.38) and Bill in (23.39).\n(23.38) Billy Bones went to the bar with Jim Hawkins. He called for a glass of\nrum. [ he = Billy ]\n(23.39) Jim Hawkins went to the bar with Billy Bones. He called for a glass of\nrum. [ he = Jim ]\n5Here\u2019s a bound pronoun example from Shakespeare\u2019s Comedy of Errors :There\u2019s not a man I meet but\ndoth salute me As if I were their well-acquainted friend\n6The word \u201cgender\u201d is generally only used for languages with 2 or 3 noun classes, like most Indo-\nEuropean languages; many languages, like the Bantu languages or Chinese, have a much larger number\nof noun classes. 23.2 \u2022 C OREFERENCE TASKS AND DATASETS 9\nVerb Semantics: Some verbs semantically emphasize one of their arguments, bi-\nasing the interpretation of subsequent pronouns. Compare (23.40) and (23.41).\n(23.40) John telephoned Bill. He lost the laptop.\n(23.41) John criticized Bill. He lost the laptop.\nThese examples differ only in the verb used in the \ufb01rst sentence, yet \u201che\u201d in (23.40)\nis typically resolved to John, whereas \u201che\u201d in (23.41) is resolved to Bill. This may\nbe partly due to the link between implicit causality and saliency: the implicit cause\nof a \u201ccriticizing\u201d event is its object, whereas the implicit cause of a \u201ctelephoning\u201d\nevent is its subject. In such verbs, the entity which is the implicit cause may be more\nsalient.\nSelectional Restrictions: Many other kinds of semantic knowledge can play a role\nin referent preference. For example, the selectional restrictions that a verb places on\nits arguments (Chapter 21) can help eliminate referents, as in (23.42).\n(23.42) I ate the soup in my new bowl after cooking it for hours\nThere are two possible referents for it, the soup and the bowl. The verb eat, however,\nrequires that its direct object denote something edible, and this constraint can rule\noutbowl as a possible referent.\n23.2 Coreference Tasks and Datasets\nWe can formulate the task of coreference resolution as follows: Given a text T, \ufb01nd\nall entities and the coreference links between them. We evaluate our task by com-\nparing the links our system creates with those in human-created gold coreference\nannotations on T.\nLet\u2019s return to our coreference example, now using superscript numbers for each\ncoreference chain (cluster), and subscript letters for individual mentions in the clus-\nter:\n(23.43) [Victoria Chen]1\na, CFO of [Megabucks Banking]2\na, saw [[her]1\nbpay]3\najump\nto $2.3 million, as [the 38-year-old]1\ncalso became [[the company]2\nb\u2019s\npresident. It is widely known that [she]1\ndcame to [Megabucks]2\ncfrom rival\n[Lotsabucks]4\na.\nAssuming example (23.43) was the entirety of the article, the chains for her pay and\nLotsabucks are singleton mentions:\n1.fVictoria Chen ,her,the 38-year-old ,Sheg\n2.fMegabucks Banking ,the company ,Megabucksg\n3.fher payg\n4.fLotsabucksg\nFor most coreference evaluation campaigns, the input to the system is the raw\ntext of articles, and systems must detect mentions and then link them into clusters.\nSolving this task requires dealing with pronominal anaphora (\ufb01guring out that her\nrefers to Victoria Chen ), \ufb01ltering out non-referential pronouns like the pleonastic It\ninIt has been ten years ), dealing with de\ufb01nite noun phrases to \ufb01gure out that the\n38-year-old is coreferent with Victoria Chen , and that the company is the same as\nMegabucks . And we need to deal with names, to realize that Megabucks is the same\nasMegabucks Banking . 10 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\nExactly what counts as a mention and what links are annotated differs from task\nto task and dataset to dataset. For example some coreference datasets do not label\nsingletons, making the task much simpler. Resolvers can achieve much higher scores\non corpora without singletons, since singletons constitute the majority of mentions in\nrunning text, and they are often hard to distinguish from non-referential NPs. Some\ntasks use gold mention-detection (i.e. the system is given human-labeled mention\nboundaries and the task is just to cluster these gold mentions), which eliminates the\nneed to detect and segment mentions from running text.\nCoreference is usually evaluated by the CoNLL F1 score, which combines three\nmetrics: MUC, B3, and CEAF e; Section 23.8 gives the details.\nLet\u2019s mention a few characteristics of one popular coreference dataset, OntoNotes\n(Pradhan et al. 2007b, Pradhan et al. 2007a), and the CoNLL 2012 Shared Task\nbased on it (Pradhan et al., 2012a). OntoNotes contains hand-annotated Chinese\nand English coreference datasets of roughly one million words each, consisting of\nnewswire, magazine articles, broadcast news, broadcast conversations, web data and\nconversational speech data, as well as about 300,000 words of annotated Arabic\nnewswire. The most important distinguishing characteristic of OntoNotes is that\nit does not label singletons, simplifying the coreference task, since singletons rep-\nresent 60%-70% of all entities. In other ways, it is similar to other coreference\ndatasets. Referring expression NPs that are coreferent are marked as mentions, but\ngenerics and pleonastic pronouns are not marked. Appositive clauses are not marked\nas separate mentions, but they are included in the mention. Thus in the NP, \u201cRichard\nGodown, president of the Industrial Biotechnology Association\u201d the mention is the\nentire phrase. Prenominal modi\ufb01ers are annotated as separate entities only if they\nare proper nouns. Thus wheat is not an entity in wheat \ufb01elds , but UNis an entity in\nUN policy (but not adjectives like American inAmerican policy ).\nA number of corpora mark richer discourse phenomena. The ISNotes corpus\nannotates a portion of OntoNotes for information status, include bridging examples\n(Hou et al., 2018). The LitBank coreference corpus (Bamman et al., 2020) contains\ncoreference annotations for 210,532 tokens from 100 different literary novels, in-\ncluding singletons and quanti\ufb01ed and negated noun phrases. The AnCora-CO coref-\nerence corpus (Recasens and Mart \u00b4\u0131, 2010) contains 400,000 words each of Spanish\n(AnCora-CO-Es) and Catalan (AnCora-CO-Ca) news data, and includes labels for\ncomplex phenomena like discourse deixis in both languages. The ARRAU corpus\n(Uryupina et al., 2020) contains 350,000 words of English marking all NPs, which\nmeans singleton clusters are available. ARRAU includes diverse genres like dialog\n(the TRAINS data) and \ufb01ction (the Pear Stories), and has labels for bridging refer-\nences, discourse deixis, generics, and ambiguous anaphoric relations.\n23.3 Mention Detection\nThe \ufb01rst stage of coreference is mention detection : \ufb01nding the spans of text thatmention\ndetection\nconstitute each mention. Mention detection algorithms are usually very liberal in\nproposing candidate mentions (i.e., emphasizing recall), and only \ufb01ltering later. For\nexample many systems run parsers and named entity taggers on the text and extract\nevery span that is either an NP, apossessive pronoun , or a named entity .\nDoing so from our sample text repeated in (23.44):\n(23.44) Victoria Chen, CFO of Megabucks Banking, saw her pay jump to $2.3 23.3 \u2022 M ENTION DETECTION 11\nmillion, as the 38-year-old also became the company\u2019s president. It is\nwidely known that she came to Megabucks from rival Lotsabucks.\nmight result in the following list of 13 potential mentions:\nVictoria Chen $2.3 million she\nCFO of Megabucks Banking the 38-year-old Megabucks\nMegabucks Banking the company Lotsabucks\nher the company\u2019s president\nher pay It\nMore recent mention detection systems are even more generous; the span-based\nalgorithm we will describe in Section 23.6 \ufb01rst extracts literally all n-gram spans\nof words up to N=10. Of course recall from Section 23.1.3 that many NPs\u2014and\nthe overwhelming majority of random n-gram spans\u2014are not referring expressions.\nTherefore all such mention detection systems need to eventually \ufb01lter out pleonas-\ntic/expletive pronouns like Itabove, appositives like CFO of Megabucks Banking\nInc, or predicate nominals like the company\u2019s president or$2.3 million .\nSome of this \ufb01ltering can be done by rules. Early rule-based systems designed\nregular expressions to deal with pleonastic it, like the following rules from Lappin\nand Leass (1994) that use dictionaries of cognitive verbs (e.g., believe ,know ,antic-\nipate ) to capture pleonastic itin \u201cIt is thought that ketchup...\u201d, or modal adjectives\n(e.g., necessary ,possible ,certain ,important ), for, e.g., \u201cIt is likely that I...\u201d. Such\nrules are sometimes used as part of modern systems:\nIt is Modaladjective that S\nIt is Modaladjective (for NP) to VP\nIt is Cogv-ed that S\nIt seems/appears/means/follows (that) S\nMention-detection rules are sometimes designed speci\ufb01cally for particular eval-\nuation campaigns. For OntoNotes, for example, mentions are not embedded within\nlarger mentions, and while numeric quantities are annotated, they are rarely coref-\nerential. Thus for OntoNotes tasks like CoNLL 2012 (Pradhan et al., 2012a), a\ncommon \ufb01rst pass rule-based mention detection algorithm (Lee et al., 2013) is:\n1.Take all NPs, possessive pronouns, and named entities.\n2.Remove numeric quantities (100 dollars, 8%), mentions embedded in\nlarger mentions, adjectival forms of nations, and stop words (like there ).\n3.Remove pleonastic itbased on regular expression patterns.\nRule-based systems, however, are generally insuf\ufb01cient to deal with mention-\ndetection, and so modern systems incorporate some sort of learned mention detec-\ntion component, such as a referentiality classi\ufb01er, an anaphoricity classi\ufb01er \u2014\ndetecting whether an NP is an anaphor\u2014or a discourse-new classi\ufb01er\u2014 detecting\nwhether a mention is discourse-new and a potential antecedent for a future anaphor.\nAnanaphoricity detector , for example, can draw its positive training examplesanaphoricity\ndetector\nfrom any span that is labeled as an anaphoric referring expression in hand-labeled\ndatasets like OntoNotes, ARRAU , or AnCora. Any other NP or named entity can be\nmarked as a negative training example. Anaphoricity classi\ufb01ers use features of the\ncandidate mention such as its head word, surrounding words, de\ufb01niteness, animacy,\nlength, position in the sentence/discourse, many of which were \ufb01rst proposed in\nearly work by Ng and Cardie (2002a); see Section 23.5 for more on features. 12 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\nReferentiality or anaphoricity detectors can be run as \ufb01lters, in which only men-\ntions that are classi\ufb01ed as anaphoric or referential are passed on to the coreference\nsystem. The end result of such a \ufb01ltering mention detection system on our example\nabove might be the following \ufb01ltered set of 9 potential mentions:\nVictoria Chen her pay she\nMegabucks Bank the 38-year-old Megabucks\nher the company Lotsabucks\nIt turns out, however, that hard \ufb01ltering of mentions based on an anaphoricity\nor referentiality classi\ufb01er leads to poor performance. If the anaphoricity classi\ufb01er\nthreshold is set too high, too many mentions are \ufb01ltered out and recall suffers. If the\nclassi\ufb01er threshold is set too low, too many pleonastic or non-referential mentions\nare included and precision suffers.\nThe modern approach is instead to perform mention detection, anaphoricity, and\ncoreference jointly in a single end-to-end model (Ng 2005b, Denis and Baldridge\n2007, Rahman and Ng 2009). For example mention detection in the Lee et al.\n(2017b),2018 system is based on a single end-to-end neural network that computes\na score for each mention being referential, a score for two mentions being corefer-\nence, and combines them to make a decision, training all these scores with a single\nend-to-end loss. We\u2019ll describe this method in detail in Section 23.6.7\nDespite these advances, correctly detecting referential mentions seems to still be\nan unsolved problem, since systems incorrectly marking pleonastic pronouns like\nitand other non-referential NPs as coreferent is a large source of errors of modern\ncoreference resolution systems (Kummerfeld and Klein 2013, Martschat and Strube\n2014, Martschat and Strube 2015, Wiseman et al. 2015, Lee et al. 2017a).\nMention, referentiality, or anaphoricity detection is thus an important open area\nof investigation. Other sources of knowledge may turn out to be helpful, especially\nin combination with unsupervised and semisupervised algorithms, which also mit-\nigate the expense of labeled datasets. In early work, for example Bean and Riloff\n(1999) learned patterns for characterizing anaphoric or non-anaphoric NPs; (by ex-\ntracting and generalizing over the \ufb01rst NPs in a text, which are guaranteed to be\nnon-anaphoric). Chang et al. (2012) look for head nouns that appear frequently in\nthe training data but never appear as gold mentions to help \ufb01nd non-referential NPs.\nBergsma et al. (2008) use web counts as a semisupervised way to augment standard\nfeatures for anaphoricity detection for English it, an important task because itis both\ncommon and ambiguous; between a quarter and half itexamples are non-anaphoric.\nConsider the following two examples:\n(23.45) You can make [it] in advance. [anaphoric]\n(23.46) You can make [it] in Hollywood. [non-anaphoric]\nTheitinmake it is non-anaphoric, part of the idiom make it . Bergsma et al. (2008)\nturn the context around each example into patterns, like \u201cmake * in advance\u201d from\n(23.45), and \u201cmake * in Hollywood\u201d from (23.46). They then use Google n-grams to\nenumerate all the words that can replace itin the patterns. Non-anaphoric contexts\ntend to only have itin the wildcard positions, while anaphoric contexts occur with\nmany other NPs (for example make them in advance is just as frequent in their data\n7Some systems try to avoid mention detection or anaphoricity detection altogether. For datasets like\nOntoNotes which don\u2019t label singletons, an alternative to \ufb01ltering out non-referential mentions is to run\ncoreference resolution, and then simply delete any candidate mentions which were not corefered with\nanother mention. This likely doesn\u2019t work as well as explicitly modeling referentiality, and cannot solve\nthe problem of detecting singletons, which is important for tasks like entity linking. 23.4 \u2022 A RCHITECTURES FOR COREFERENCE ALGORITHMS 13\nasmake it in advance , but make them in Hollywood did not occur at all). These\nn-gram contexts can be used as features in a supervised anaphoricity classi\ufb01er.\n23.4 Architectures for Coreference Algorithms\nModern systems for coreference are based on supervised neural machine learning,\nsupervised from hand-labeled datasets like OntoNotes. In this section we overview\nthe various architecture of modern systems, using the categorization of Ng (2010),\nwhich distinguishes algorithms based on whether they make each coreference deci-\nsion in a way that is entity-based \u2014representing each entity in the discourse model\u2014\nor only mention-based \u2014considering each mention independently, and whether they\nuseranking models to directly compare potential antecedents. Afterwards, we go\ninto more detail on one state-of-the-art algorithm in Section 23.6.\n23.4.1 The Mention-Pair Architecture\nWe begin with the mention-pair architecture, the simplest and most in\ufb02uential mention-pair\ncoreference architecture, which introduces many of the features of more complex\nalgorithms, even though other architectures perform better. The mention-pair ar- mention-pair\nchitecture is based around a classi\ufb01er that\u2014 as its name suggests\u2014is given a pair\nof mentions, a candidate anaphor and a candidate antecedent, and makes a binary\nclassi\ufb01cation decision: coreferring or not.\nLet\u2019s consider the task of this classi\ufb01er for the pronoun shein our example, and\nassume the slightly simpli\ufb01ed set of potential antecedents in Fig. 23.2.\nVictoria ChenMegabucks Bankingherher paythe 37-year-oldshep(coref|\u201dVictoria Chen\u201d,\u201dshe\u201d)\np(coref|\u201dMegabucks Banking\u201d,\u201dshe\u201d)\nFigure 23.2 For each pair of a mention (like she), and a potential antecedent mention (like\nVictoria Chen orher), the mention-pair classi\ufb01er assigns a probability of a coreference link.\nFor each prior mention ( Victoria Chen ,Megabucks Banking ,her, etc.), the binary\nclassi\ufb01er computes a probability: whether or not the mention is the antecedent of\nshe. We want this probability to be high for actual antecedents ( Victoria Chen ,her,\nthe 38-year-old ) and low for non-antecedents ( Megabucks Banking ,her pay ).\nEarly classi\ufb01ers used hand-built features (Section 23.5); more recent classi\ufb01ers\nuse neural representation learning (Section 23.6)\nFor training, we need a heuristic for selecting training samples; since most pairs\nof mentions in a document are not coreferent, selecting every pair would lead to\na massive overabundance of negative samples. The most common heuristic, from\n(Soon et al., 2001), is to choose the closest antecedent as a positive example, and all\npairs in between as the negative examples. More formally, for each anaphor mention\nmiwe create\n\u2022 one positive instance ( mi;mj) where mjis the closest antecedent to mi, and 14 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\n\u2022 a negative instance ( mi;mk) for each mkbetween mjandmi\nThus for the anaphor she, we would choose ( she,her) as the positive example\nand no negative examples. Similarly, for the anaphor the company we would choose\n(the company ,Megabucks ) as the positive example and ( the company ,she) (the com-\npany ,the 38-year-old ) (the company ,her pay ) and ( the company ,her) as negative\nexamples.\nOnce the classi\ufb01er is trained, it is applied to each test sentence in a clustering\nstep. For each mention iin a document, the classi\ufb01er considers each of the prior i\u00001\nmentions. In closest-\ufb01rst clustering (Soon et al., 2001), the classi\ufb01er is run right to\nleft (from mention i\u00001 down to mention 1) and the \ufb01rst antecedent with probability\n>:5 is linked to i. If no antecedent has probably >0:5, no antecedent is selected for\ni. Inbest-\ufb01rst clustering, the classi\ufb01er is run on all i\u00001 antecedents and the most\nprobable preceding mention is chosen as the antecedent for i. The transitive closure\nof the pairwise relation is taken as the cluster.\nWhile the mention-pair model has the advantage of simplicity, it has two main\nproblems. First, the classi\ufb01er doesn\u2019t directly compare candidate antecedents to\neach other, so it\u2019s not trained to decide, between two likely antecedents, which one\nis in fact better. Second, it ignores the discourse model, looking only at mentions,\nnot entities. Each classi\ufb01er decision is made completely locally to the pair, without\nbeing able to take into account other mentions of the same entity. The next two\nmodels each address one of these two \ufb02aws.\n23.4.2 The Mention-Rank Architecture\nThe mention ranking model directly compares candidate antecedents to each other,\nchoosing the highest-scoring antecedent for each anaphor.\nIn early formulations, for mention i, the classi\ufb01er decides which of the f1;:::;i\u0000\n1gprior mentions is the antecedent (Denis and Baldridge, 2008). But suppose iis\nin fact not anaphoric, and none of the antecedents should be chosen? Such a model\nwould need to run a separate anaphoricity classi\ufb01er on i. Instead, it turns out to be\nbetter to jointly learn anaphoricity detection and coreference together with a single\nloss (Rahman and Ng, 2009).\nSo in modern mention-ranking systems, for the ith mention (anaphor), we have\nan associated random variable yiranging over the values Y(i) =f1;:::;i\u00001;\u000fg. The\nvalue\u000fis a special dummy mention meaning that idoes not have an antecedent (i.e.,\nis either discourse-new and starts a new coref chain, or is non-anaphoric).\nVictoria ChenMegabucks Bankingherher paythe 37-year-oldshep(\u201dVictoria Chen\u201d|\u201dshe\u201d)\np(\u03f5|\u201dshe\u201d)\u03f5One or more of theseshould be highAll of theseshould be low}p(\u201dher pay\u201d|she\u201d)p(\u201dher\u201d|she\u201d)p(\u201dthe 37-year-old\u201d|she\u201d)\np(\u201dMegabucks Banking\u201d|she\u201d)}\nFigure 23.3 For each candidate anaphoric mention (like she), the mention-ranking system assigns a proba-\nbility distribution over all previous mentions plus the special dummy mention \u000f.\nAt test time, for a given mention ithe model computes one softmax over all the\nantecedents (plus \u000f) giving a probability for each candidate antecedent (or none). 23.5 \u2022 C LASSIFIERS USING HAND -BUILT FEATURES 15\nFig. 23.3 shows an example of the computation for the single candidate anaphor\nshe.\nOnce the antecedent is classi\ufb01ed for each anaphor, transitive closure can be run\nover the pairwise decisions to get a complete clustering.\nTraining is trickier in the mention-ranking model than the mention-pair model,\nbecause for each anaphor we don\u2019t know which of all the possible gold antecedents\nto use for training. Instead, the best antecedent for each mention is latent ; that\nis, for each mention we have a whole cluster of legal gold antecedents to choose\nfrom. Early work used heuristics to choose an antecedent, for example choosing the\nclosest antecedent as the gold antecedent and all non-antecedents in a window of\ntwo sentences as the negative examples (Denis and Baldridge, 2008). Various kinds\nof ways to model latent antecedents exist (Fernandes et al. 2012, Chang et al. 2013,\nDurrett and Klein 2013). The simplest way is to give credit to any legal antecedent\nby summing over all of them, with a loss function that optimizes the likelihood of\nall correct antecedents from the gold clustering (Lee et al., 2017b). We\u2019ll see the\ndetails in Section 23.6.\nMention-ranking models can be implemented with hand-build features or with\nneural representation learning (which might also incorporate some hand-built fea-\ntures). we\u2019ll explore both directions in Section 23.5 and Section 23.6.\n23.4.3 Entity-based Models\nBoth the mention-pair and mention-ranking models make their decisions about men-\ntions . By contrast, entity-based models link each mention not to a previous mention\nbut to a previous discourse entity (cluster of mentions).\nA mention-ranking model can be turned into an entity-ranking model simply\nby having the classi\ufb01er make its decisions over clusters of mentions rather than\nindividual mentions (Rahman and Ng, 2009).\nFor traditional feature-based models, this can be done by extracting features over\nclusters. The size of a cluster is a useful feature, as is its \u2018shape\u2019, which is the\nlist of types of the mentions in the cluster i.e., sequences of the tokens (P)roper,\n(D)e\ufb01nite, (I)nde\ufb01nite, (Pr)onoun, so that a cluster composed of fVictoria ,her,the\n38-year-oldgwould have the shape P-Pr-D (Bj\u00a8orkelund and Kuhn, 2014). An entity-\nbased model that includes a mention-pair classi\ufb01er can use as features aggregates of\nmention-pair probabilities, for example computing the average probability of coref-\nerence over all mention-pairs in the two clusters (Clark and Manning 2015).\nNeural models can learn representations of clusters automatically, for example\nby using an RNN over the sequence of cluster mentions to encode a state correspond-\ning to a cluster representation (Wiseman et al., 2016), or by learning distributed rep-\nresentations for pairs of clusters by pooling over learned representations of mention\npairs (Clark and Manning, 2016b).\nHowever, although entity-based models are more expressive, the use of cluster-\nlevel information in practice has not led to large gains in performance, so mention-\nranking models are still more commonly used.\n23.5 Classi\ufb01ers using hand-built features\nFeature-based classi\ufb01ers, use hand-designed features in logistic regression, SVM,\nor random forest classi\ufb01ers for coreference resolution. These classi\ufb01ers don\u2019t per- 16 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\nform as well as neural ones. Nonetheless, they are still sometimes useful to build\nlightweight systems when compute or data are sparse, and the features themselves\nare useful for error analysis even in neural systems.\nGiven an anaphor mention and a potential antecedent mention, feature based\nclassi\ufb01ers make use of three types of features: (i) features of the anaphor, (ii) features\nof the candidate antecedent, and (iii) features of the relationship between the pair.\nEntity-based models can make additional use of two additional classes: (iv) feature\nof all mentions from the antecedent\u2019s entity cluster, and (v) features of the relation\nbetween the anaphor and the mentions in the antecedent entity cluster.\nFeatures of the Anaphor or Antecedent Mention\nFirst (last) word Victoria/she First or last word (or embedding) of antecedent/anaphor\nHead word Victoria/she Head word (or head embedding) of antecedent/anaphor\nAttributes Sg-F-A-3-PER/\nSg-F-A-3-PERThe number, gender, animacy, person, named entity type\nattributes of (antecedent/anaphor)\nLength 2/1 length in words of (antecedent/anaphor)\nMention type P/Pr Type: (P)roper, (D)e\ufb01nite, (I)nde\ufb01nite, (Pr)onoun) of an-\ntecedent/anaphor\nFeatures of the Antecedent Entity\nEntity shape P-Pr-D The \u2018shape\u2019 or list of types of the mentions in the\nantecedent entity (cluster), i.e., sequences of (P)roper,\n(D)e\ufb01nite, (I)nde\ufb01nite, (Pr)onoun.\nEntity attributes Sg-F-A-3-PER The number, gender, animacy, person, named entity type\nattributes of the antecedent entity\nAnt. cluster size 3 Number of mentions in the antecedent cluster\nFeatures of the Pair of Mentions\nSentence distance 1 The number of sentences between antecedent and anaphor\nMention distance 4 The number of mentions between antecedent and anaphor\ni-within-i F Anaphor has i-within-i relation with antecedent\nCosine Cosine between antecedent and anaphor embeddings\nFeatures of the Pair of Entities\nExact String Match F True if the strings of any two mentions from the antecedent\nand anaphor clusters are identical.\nHead Word Match F True if any mentions from antecedent cluster has same\nheadword as any mention in anaphor cluster\nWord Inclusion F All words in anaphor cluster included in antecedent cluster\nFigure 23.4 Feature-based coreference: sample feature values for anaphor \u201cshe\u201d and potential antecedent\n\u201cVictoria Chen\u201d.\nFigure 23.4 shows a selection of commonly used features, and shows the value\nthat would be computed for the potential anaphor \u201cshe\u201d and potential antecedent\n\u201cVictoria Chen\u201d in our example sentence, repeated below:\n(23.47) Victoria Chen , CFO of Megabucks Banking, saw her pay jump to $2.3\nmillion, as the 38-year-old also became the company\u2019s president. It is\nwidely known that shecame to Megabucks from rival Lotsabucks.\nFeatures that prior work has found to be particularly useful are exact string\nmatch, entity headword agreement, mention distance, as well as (for pronouns) exact\nattribute match and i-within-i, and (for nominals and proper names) word inclusion\nand cosine. For lexical features (like head words) it is common to only use words\nthat appear enough times ( >20 times). 23.6 \u2022 A NEURAL MENTION -RANKING ALGORITHM 17\nIt is crucial in feature-based systems to use conjunctions of features; one exper-\niment suggested that moving from individual features in a classi\ufb01er to conjunctions\nof multiple features increased F1 by 4 points (Lee et al., 2017a). Speci\ufb01c conjunc-\ntions can be designed by hand (Durrett and Klein, 2013), all pairs of features can be\nconjoined (Bengtson and Roth, 2008), or feature conjunctions can be learned using\ndecision tree or random forest classi\ufb01ers (Ng and Cardie 2002a, Lee et al. 2017a).\nFeatures can also be used in neural models as well. Neural systems use contex-\ntual word embeddings so don\u2019t bene\ufb01t from shallow features like string match or or\nmention types. However features like mention length, distance between mentions,\nor genre can complement neural contextual embedding models.\n23.6 A neural mention-ranking algorithm\nIn this section we describe the neural e2e-coref algorithms of Lee et al. (2017b)\n(simpli\ufb01ed and extended a bit, drawing on Joshi et al. (2019) and others). This is\namention-ranking algorithm that considers all possible spans of text in the docu-\nment, assigns a mention-score to each span, prunes the mentions based on this score,\nthen assigns coreference links to the remaining mentions.\nMore formally, given a document Dwith Twords, the model considers all of\ntheT(T+1)\n2text spans in D(unigrams, bigrams, trigrams, 4-grams, etc; in practice\nwe only consider spans up a maximum length around 10). The task is to assign\nto each span ian antecedent yi, a random variable ranging over the values Y(i) =\nf1;:::;i\u00001;\u000fg; each previous span and a special dummy token \u000f. Choosing the\ndummy token means that idoes not have an antecedent, either because iis discourse-\nnew and starts a new coreference chain, or because iis non-anaphoric.\nFor each pair of spans iand j, the system assigns a score s(i;j)for the coref-\nerence link between span iand span j. The system then learns a distribution P(yi)\nover the antecedents for span i:\nP(yi) =exp(s(i;yi))P\ny02Y(i)exp(s(i;y0))(23.48)\nThis score s(i;j)includes three factors that we\u2019ll de\ufb01ne below: m(i); whether span\niis a mention; m(j); whether span jis a mention; and c(i;j); whether jis the\nantecedent of i:\ns(i;j) =m(i)+m(j)+c(i;j) (23.49)\nFor the dummy antecedent \u000f, the score s(i;\u000f)is \ufb01xed to 0. This way if any non-\ndummy scores are positive, the model predicts the highest-scoring antecedent, but if\nall the scores are negative it abstains.\n23.6.1 Computing span representations\nTo compute the two functions m(i)andc(i;j)which score a span ior a pair of spans\n(i;j), we\u2019ll need a way to represent a span. The e2e-coref family of algorithms\nrepresents each span by trying to capture 3 words/tokens: the \ufb01rst word, the last\nword, and the most important word. We \ufb01rst run each paragraph or subdocument\nthrough an encoder (like BERT) to generate embeddings hifor each token i. The\nspan iis then represented by a vector githat is a concatenation of the encoder output 18 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\nembedding for the \ufb01rst (start) token of the span, the encoder output for the last (end)\ntoken of the span, and a third vector which is an attention-based representation:\ngi= [hSTART (i);hEND (i);hATT(i)] (23.50)\nThe goal of the attention vector is to represent which word/token is the likely\nsyntactic head-word of the span; we saw in the prior section that head-words are\na useful feature; a matching head-word is a good indicator of coreference. The\nattention representation is computed as usual; the system learns a weight vector wa,\nand computes its dot product with the hidden state httransformed by a FFN:\nat=wa\u0001FFN a(ht) (23.51)\nThe attention score is normalized into a distribution via a softmax:\nai;t=exp(at)\nPEND (i)\nk=START (i)exp(ak)(23.52)\nAnd then the attention distribution is used to create a vector hATT(i)which is an\nattention-weighted sum of the embeddings etof each of the words in span i:\nhATT(i)=END (i)X\nt=START (i)ai;t\u0001et (23.53)\nFig. 23.5 shows the computation of the span representation and the mention\nscore.\nEncodings (h)\u2026Encoder GeneralElectricsaidthePostalServicecontactedthecompanySpan head (hATT) Span representation (g) Mention score (m)+++++General ElectricElectric said thethe Postal ServiceService contacted thethe company\nFigure 23.5 Computation of the span representation g(and the mention score m) in a BERT version of the\ne2e-coref model (Lee et al. 2017b, Joshi et al. 2019). The model considers all spans up to a maximum width of\nsay 10; the \ufb01gure shows a small subset of the bigram and trigram spans.\n23.6.2 Computing the mention and antecedent scores mandc\nNow that we know how to compute the vector gifor representing span i, we can\nsee the details of the two scoring functions m(i)andc(i;j). Both are computed by\nfeedforward networks:\nm(i) = wm\u0001FFN m(gi) (23.54)\nc(i;j) = wc\u0001FFN c([gi;gj;gi\u000egj;]) (23.55)\nAt inference time, this mention score mis used as a \ufb01lter to keep only the best few\nmentions. 23.6 \u2022 A NEURAL MENTION -RANKING ALGORITHM 19\nWe then compute the antecedent score for high-scoring mentions. The antecedent\nscore c(i;j)takes as input a representation of the spans iandj, but also the element-\nwise similarity of the two spans to each other gi\u000egj(here\u000eis element-wise mul-\ntiplication). Fig. 23.6 shows the computation of the score sfor the three possible\nantecedents of the company in the example sentence from Fig. 23.5.\nFigure 23.6 The computation of the score sfor the three possible antecedents of the com-\npany in the example sentence from Fig. 23.5. Figure after Lee et al. (2017b).\nGiven the set of mentions, the joint distribution of antecedents for each docu-\nment is computed in a forward pass, and we can then do transitive closure on the\nantecedents to create a \ufb01nal clustering for the document.\nFig. 23.7 shows example predictions from the model, showing the attention\nweights, which Lee et al. (2017b) \ufb01nd correlate with traditional semantic heads.\nNote that the model gets the second example wrong, presumably because attendants\nandpilot likely have nearby word embeddings.\nFigure 23.7 Sample predictions from the Lee et al. (2017b) model, with one cluster per\nexample, showing one correct example and one mistake. Bold, parenthesized spans are men-\ntions in the predicted cluster. The amount of red color on a word indicates the head-\ufb01nding\nattention weight ai;tin (23.52). Figure adapted from Lee et al. (2017b).\n23.6.3 Learning\nFor training, we don\u2019t have a single gold antecedent for each mention; instead the\ncoreference labeling only gives us each entire cluster of coreferent mentions; so a\nmention only has a latent antecedent. We therefore use a loss function that maxi-\nmizes the sum of the coreference probability of any of the legal antecedents. For a\ngiven mention iwith possible antecedents Y(i), let GOLD (i) be the set of mentions\nin the gold cluster containing i. Since the set of mentions occurring before iisY(i),\nthe set of mentions in that gold cluster that also occur before iisY(i)\\GOLD (i). We 20 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\ntherefore want to maximize:\nX\n\u02c6y2Y(i)\\GOLD (i)P(\u02c6y) (23.56)\nIf a mention iis not in a gold cluster GOLD (i) =\u000f.\nTo turn this probability into a loss function, we\u2019ll use the cross-entropy loss\nfunction we de\ufb01ned in Eq. ??in Chapter 5, by taking the \u0000log of the probability. If\nwe then sum over all mentions, we get the \ufb01nal loss function for training:\nL=NX\ni=2\u0000logX\n\u02c6y2Y(i)\\GOLD (i)P(\u02c6y) (23.57)\n23.7 Entity Linking\nEntity linking is the task of associating a mention in text with the representation of entity linking\nsome real-world entity in an ontology or knowledge base (Ji and Grishman, 2011). It\nis the natural follow-on to coreference resolution; coreference resolution is the task\nof associating textual mentions that corefer to the same entity. Entity linking takes\nthe further step of identifying who that entity is. It is especially important for any\nNLP task that links to a knowledge base.\nWhile there are all sorts of potential knowledge-bases, we\u2019ll focus in this section\non Wikipedia, since it\u2019s widely used as an ontology for NLP tasks. In this usage,\neach unique Wikipedia page acts as the unique id for a particular entity. This task of\ndeciding which Wikipedia page corresponding to an individual is being referred to\nby a text mention has its own name: wiki\ufb01cation (Mihalcea and Csomai, 2007). wiki\ufb01cation\nSince the earliest systems (Mihalcea and Csomai 2007, Cucerzan 2007, Milne\nand Witten 2008), entity linking is done in (roughly) two stages: mention detec-\ntion andmention disambiguation . We\u2019ll give two algorithms, one simple classic\nbaseline that uses anchor dictionaries and information from the Wikipedia graph\nstructure (Ferragina and Scaiella, 2011) and one modern neural algorithm (Li et al.,\n2020). We\u2019ll focus here mainly on the application of entity linking to questions,\nsince a lot of the literature has been in that context.\n23.7.1 Linking based on Anchor Dictionaries and Web Graph\nAs a simple baseline we introduce the T AGME linker (Ferragina and Scaiella, 2011)\nfor Wikipedia, which itself draws on earlier algorithms (Mihalcea and Csomai 2007,\nCucerzan 2007, Milne and Witten 2008). Wiki\ufb01cation algorithms de\ufb01ne the set of\nentities as the set of Wikipedia pages, so we\u2019ll refer to each Wikipedia page as a\nunique entity e. T AGME \ufb01rst creates a catalog of all entities (i.e. all Wikipedia\npages, removing some disambiguation and other meta-pages) and indexes them in a\nstandard IR engine like Lucene. For each page e, the algorithm computes an in-link\ncount in (e): the total number of in-links from other Wikipedia pages that point to e.\nThese counts can be derived from Wikipedia dumps.\nFinally, the algorithm requires an anchor dictionary . An anchor dictionary\nlists for each Wikipedia page, its anchor texts : the hyperlinked spans of text on anchor texts\nother pages that point to it. For example, the web page for Stanford University,\nhttp://www.stanford.edu , might be pointed to from another page using anchor\ntexts like Stanford orStanford University : 23.7 \u2022 E NTITY LINKING 21\n<a href=\"http://www.stanford.edu\">Stanford University</a>\nWe compute a Wikipedia anchor dictionary by including, for each Wikipedia\npage e,e\u2019s title as well as all the anchor texts from all Wikipedia pages that point to e.\nFor each anchor string awe\u2019ll also compute its total frequency freq (a)in Wikipedia\n(including non-anchor uses), the number of times aoccurs as a link (which we\u2019ll call\nlink(a)), and its link probability linkprob (a) =link(a)=freq(a). Some cleanup of the\n\ufb01nal anchor dictionary is required, for example removing anchor strings composed\nonly of numbers or single characters, that are very rare, or that are very unlikely to\nbe useful entities because they have a very low linkprob.\nMention Detection Given a question (or other text we are trying to link), T AGME\ndetects mentions by querying the anchor dictionary for each token sequence up to\n6 words. This large set of sequences is pruned with some simple heuristics (for\nexample pruning substrings if they have small linkprobs). The question:\nWhen was Ada Lovelace born?\nmight give rise to the anchor Ada Lovelace and possibly Ada, but substrings spans\nlikeLovelace might be pruned as having too low a linkprob, and but spans like born\nhave such a low linkprob that they would not be in the anchor dictionary at all.\nMention Disambiguation If a mention span is unambiguous (points to only one\nentity/Wikipedia page), we are done with entity linking! However, many spans are\nambiguous, matching anchors for multiple Wikipedia entities/pages. The T AGME\nalgorithm uses two factors for disambiguating ambiguous spans, which have been\nreferred to as prior probability andrelatedness/coherence . The \ufb01rst factor is p(eja),\nthe probability with which the span refers to a particular entity. For each page e2\nE(a), the probability p(eja)that anchor apoints to e, is the ratio of the number of\nlinks into ewith anchor text ato the total number of occurrences of aas an anchor:\nprior(a!e) = p(eja) =count (a!e)\nlink(a)(23.58)\nLet\u2019s see how that factor works in linking entities in the following question:\nWhat Chinese Dynasty came before the Yuan?\nThe most common association for the span Yuan in the anchor dictionary is the name\nof the Chinese currency, i.e., the probability p(Yuan currencyjyuan)is very high.\nRarer Wikipedia associations for Yuan include the common Chinese last name, a\nlanguage spoken in Thailand, and the correct entity in this case, the name of the\nChinese dynasty. So if we chose based only on p(eja), we would make the wrong\ndisambiguation and miss the correct link, Yuan dynasty .\nTo help in just this sort of case, T AGME uses a second factor, the relatedness of\nthis entity to other entities in the input question. In our example, the fact that the\nquestion also contains the span Chinese Dynasty , which has a high probability link to\nthe page Dynasties inChinese history , ought to help match Yuan dynasty .\nLet\u2019s see how this works. Given a question q, for each candidate anchors span\nadetected in q, we assign a relatedness score to each possible entity e2E(a)ofa.\nThe relatedness score of the link a!eis the weighted average relatedness between\neand all other entities in q. Two entities are considered related to the extent their\nWikipedia pages share many in-links. More formally, the relatedness between two\nentities AandBis computed as\nrel(A;B) =log(max(jin(A)j;jin(B)j))\u0000log(jin(A)\\in(B)j)\nlog(jWj)\u0000log(min(jin(A)j;jin(B)j))(23.59) 22 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\nwhere in (x)is the set of Wikipedia pages pointing to xandWis the set of all Wiki-\npedia pages in the collection.\nThe vote given by anchor bto the candidate annotation a!Xis the average,\nover all the possible entities of b, of their relatedness to X, weighted by their prior\nprobability:\nvote(b;X) =1\njE(b)jX\nY2E(b)rel(X;Y)p(Yjb) (23.60)\nThe total relatedness score for a!Xis the sum of the votes of all the other anchors\ndetected in q:\nrelatedness (a!X) =X\nb2Xqnavote(b;X) (23.61)\nTo score a!X, we combine relatedness and prior by choosing the entity X\nthat has the highest relatedness (a!X), \ufb01nding other entities within a small \u000fof\nthis value, and from this set, choosing the entity with the highest prior P(Xja). The\nresult of this step is a single entity assigned to each span in q.\nThe T AGME algorithm has one further step of pruning spurious anchor/entity\npairs, assigning a score averaging link probability with the coherence.\ncoherence (a!X) =1\njSj\u00001X\nB2SnXrel(B;X)\nscore(a!X) =coherence (a!X)+linkprob (a)\n2(23.62)\nFinally, pairs are pruned if score (a!X)<l, where the threshold lis set on a\nheld-out set.\n23.7.2 Neural Graph-based linking\nMore recent entity linking models are based on bi-encoders , encoding a candidate\nmention span, encoding an entity, and computing the dot product between the en-\ncodings. This allows embeddings for all the entities in the knowledge base to be\nprecomputed and cached (Wu et al., 2020). Let\u2019s sketch the ELQ linking algorithm\nof Li et al. (2020), which is given a question qand a set of candidate entities from\nWikipedia with associated Wikipedia text, and outputs tuples (e;ms;me)of entity id,\nmention start, and mention end. As Fig. 23.8 shows, it does this by encoding each\nWikipedia entity using text from Wikipedia, encoding each mention span using text\nfrom the question, and computing their similarity, as we describe below.\nEntity Mention Detection To get an h-dimensional embedding for each question\ntoken, the algorithm runs the question through BERT in the normal way:\n[q1\u0001\u0001\u0001qn] =BERT ([CLS]q1\u0001\u0001\u0001qn[SEP]) (23.63)\nIt then computes the likelihood of each span [i;j]inqbeing an entity mention, in\na way similar to the span-based algorithm we saw for the reader above. First we\ncompute the score for i/jbeing the start/end of a mention:\nsstart(i) =wstart\u0001qi;send(j) =wend\u0001qj; (23.64) 23.7 \u2022 E NTITY LINKING 23\nFigure 23.8 A sketch of the inference process in the ELQ algorithm for entity linking in\nquestions (Li et al., 2020). Each candidate question mention span and candidate entity are\nseparately encoded, and then scored by the entity/span dot product.\nwhere wstartandwendare vectors learned during training. Next, another trainable\nembedding, wmention is used to compute a score for each token being part of a men-\ntion:\nsmention (t) =wmention\u0001qt (23.65)\nMention probabilities are then computed by combining these three scores:\np([i;j]) =s \nsstart(i)+send(j)+jX\nt=ismention (t)!\n(23.66)\nEntity Linking To link mentions to entities, we next compute embeddings for\neach entity in the set E=e1;\u0001\u0001\u0001;ei;\u0001\u0001\u0001;ewof all Wikipedia entities. For each en-\ntityeiwe\u2019ll get text from the entity\u2019s Wikipedia page, the title t(ei)and the \ufb01rst\n128 tokens of the Wikipedia page which we\u2019ll call the description d(ei). This is\nagain run through BERT, taking the output of the CLStoken BERT [CLS] as the entity\nrepresentation:\nxei=BERT [CLS]([CLS]t(ei)[ENT]d(ei)[SEP]) (23.67)\nMention spans can be linked to entities by computing, for each entity eand span\n[i;j], the dot product similarity between the span encoding (the average of the token\nembeddings) and the entity encoding.\nyi;j=1\n(j\u0000i+1)jX\nt=iqt\ns(e;[i;j]) = x\u0001\neyi;j (23.68)\nFinally, we take a softmax to get a distribution over entities for each span:\np(ej[i;j]) =exp(s(e;[i;j]))P\ne02Eexp(s(e0;[i;j]))(23.69)\nTraining The ELQ mention detection and entity linking algorithm is fully super-\nvised. This means, unlike the anchor dictionary algorithms from Section 23.7.1, 24 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\nit requires datasets with entity boundaries marked and linked. Two such labeled\ndatasets are WebQuestionsSP (Yih et al., 2016), an extension of the WebQuestions\n(Berant et al., 2013) dataset derived from Google search questions, and GraphQues-\ntions (Su et al., 2016). Both have had entity spans in the questions marked and\nlinked (Sorokin and Gurevych 2018, Li et al. 2020) resulting in entity-labeled ver-\nsions WebQSP ELand GraphQ EL(Li et al., 2020).\nGiven a training set, the ELQ mention detection and entity linking phases are\ntrained jointly, optimizing the sum of their losses. The mention detection loss is\na binary cross-entropy loss, with Lthe length of the passage and Nthe number of\ncandidates:\nLMD=\u00001\nNX\n1\u0014i\u0014j\u0014min(i+L\u00001;n)\u0000\ny[i;j]logp([i;j])+( 1\u0000y[i;j])log(1\u0000p([i;j]))\u0001\n(23.70)\nwith y[i;j]=1 if[i;j]is a gold mention span, else 0. The entity linking loss is:\nLED=\u0000logp(egj[i;j]) (23.71)\nwhere egis the gold entity for mention [i;j].\n23.8 Evaluation of Coreference Resolution\nWe evaluate coreference algorithms model-theoretically, comparing a set of hypoth-\nesischains or clusters Hproduced by the system against a set of gold or reference\nchains or clusters Rfrom a human labeling, and reporting precision and recall.\nHowever, there are a wide variety of methods for doing this comparison. In fact,\nthere are 5 common metrics used to evaluate coreference algorithms: the link based\nMUC (Vilain et al., 1995) and BLANC (Recasens and Hovy 2011, Luo et al. 2014)\nmetrics, the mention based B3metric (Bagga and Baldwin, 1998), the entity based\nCEAF metric (Luo, 2005), and the link based entity aware LEA metric (Moosavi and\nStrube, 2016).\nLet\u2019s just explore two of the metrics. The MUC F-measure (Vilain et al., 1995)MUC\nF-measure\nis based on the number of coreference links (pairs of mentions) common to Hand\nR. Precision is the number of common links divided by the number of links in H.\nRecall is the number of common links divided by the number of links in R; This\nmakes MUC biased toward systems that produce large chains (and fewer entities),\nand it ignores singletons, since they don\u2019t involve links.\nB3is mention-based rather than link-based. For each mention in the reference B3\nchain, we compute a precision and recall, and then we take a weighted sum over all\nNmentions in the document to compute a precision and recall for the entire task. For\na given mention i, letRbe the reference chain that includes i, and Hthe hypothesis\nchain that has i. The set of correct mentions in HisH\\R. Precision for mention i\nis thusjH\\Rj\njHj, and recall for mention ithusjH\\Rj\njRj. The total precision is the weighted\nsum of the precision for mention i, weighted by a weight wi. The total recall is the\nweighted sum of the recall for mention i, weighted by a weight wi. Equivalently:\nPrecision =NX\ni=1wi# of correct mentions in hypothesis chain containing entity i\n# of mentions in hypothesis chain containing entity i\nRecall =NX\ni=1wi# of correct mentions in hypothesis chain containing entity i\n# of mentions in reference chain containing entity i 23.9 \u2022 W INOGRAD SCHEMA PROBLEMS 25\nThe weight wifor each entity can be set to different values to produce different\nversions of the algorithm.\nFollowing a proposal from Denis and Baldridge (2009), the CoNLL coreference\ncompetitions were scored based on the average of MUC, CEAF-e, and B3(Pradhan\net al. 2011, Pradhan et al. 2012b), and so it is common in many evaluation campaigns\nto report an average of these 3 metrics. See Luo and Pradhan (2016) for a detailed\ndescription of the entire set of metrics; reference implementations of these should\nbe used rather than attempting to reimplement from scratch (Pradhan et al., 2014).\nAlternative metrics have been proposed that deal with particular coreference do-\nmains or tasks. For example, consider the task of resolving mentions to named\nentities (persons, organizations, geopolitical entities), which might be useful for in-\nformation extraction or knowledge base completion. A hypothesis chain that cor-\nrectly contains all the pronouns referring to an entity, but has no version of the name\nitself, or is linked with a wrong name, is not useful for this task. We might instead\nwant a metric that weights each mention by how informative it is (with names being\nmost informative) (Chen and Ng, 2013) or a metric that considers a hypothesis to\nmatch a gold chain only if it contains at least one variant of a name (the NEC F1\nmetric of Agarwal et al. (2019)).\n23.9 Winograd Schema problems\nFrom early on in the \ufb01eld, researchers have noted that some cases of coreference\nare quite dif\ufb01cult, seeming to require world knowledge or sophisticated reasoning\nto solve. The problem was most famously pointed out by Winograd (1972) with the\nfollowing example:\n(23.72) The city council denied the demonstrators a permit because\na. they feared violence.\nb. they advocated violence.\nWinograd noticed that the antecedent that most readers preferred for the pro-\nnoun they in continuation (a) was the city council , but in (b) was the demonstrators .\nHe suggested that this requires understanding that the second clause is intended\nas an explanation of the \ufb01rst clause, and also that our cultural frames suggest that\ncity councils are perhaps more likely than demonstrators to fear violence and that\ndemonstrators might be more likely to advocate violence.\nIn an attempt to get the \ufb01eld of NLP to focus more on methods involving world\nknowledge and common-sense reasoning, Levesque (2011) proposed a challenge\ntask called the Winograd Schema Challenge .8The problems in the challenge taskWinograd\nschema\nare coreference problems designed to be easily disambiguated by the human reader,\nbut hopefully not solvable by simple techniques such as selectional restrictions, or\nother basic word association methods.\nThe problems are framed as a pair of statements that differ in a single word or\nphrase, and a coreference question:\n(23.73) The trophy didn\u2019t \ufb01t into the suitcase because it was too large .\nQuestion: What was too large ? Answer: The trophy\n8Levesque\u2019s call was quickly followed up by Levesque et al. (2012) and Rahman and Ng (2012), a\ncompetition at the IJCAI conference (Davis et al., 2017), and a natural language inference version of the\nproblem called WNLI (Wang et al., 2018). 26 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\n(23.74) The trophy didn\u2019t \ufb01t into the suitcase because it was too small .\nQuestion: What was too small ? Answer: The suitcase\nThe problems have the following characteristics:\n1. The problems each have two parties\n2. A pronoun preferentially refers to one of the parties, but could grammatically\nalso refer to the other\n3. A question asks which party the pronoun refers to\n4. If one word in the question is changed, the human-preferred answer changes\nto the other party\nThe kind of world knowledge that might be needed to solve the problems can\nvary. In the trophy/suitcase example, it is knowledge about the physical world; that\na bigger object cannot \ufb01t into a smaller object. In the original Winograd sentence,\nit is stereotypes about social actors like politicians and protesters. In examples like\nthe following, it is knowledge about human actions like turn-taking or thanking.\n(23.75) Bill passed the gameboy to John because his turn was [over/next]. Whose\nturn was [over/next]? Answers: Bill/John\n(23.76) Joan made sure to thank Susan for all the help she had [given/received].\nWho had [given/received] help? Answers: Susan/Joan.\nAlthough the Winograd Schema was designed to require common-sense rea-\nsoning, a large percentage of the original set of problems can be solved by pre-\ntrained language models, \ufb01ne-tuned on Winograd Schema sentences (Kocijan et al.,\n2019). Large pretrained language models encode an enormous amount of world or\ncommon-sense knowledge! The current trend is therefore to propose new datasets\nwith increasingly dif\ufb01cult Winograd-like coreference resolution problems like K NOW REF\n(Emami et al., 2019), with examples like:\n(23.77) Marcus is undoubtedly faster than Jarrett right now but in [his] prime the\ngap wasn\u2019t all that big.\nIn the end, it seems likely that some combination of language modeling and knowl-\nedge will prove fruitful; indeed, it seems that knowledge-based models over\ufb01t less\nto lexical idiosyncracies in Winograd Schema training sets (Trichelair et al., 2018),\n23.10 Gender Bias in Coreference\nAs with other aspects of language processing, coreference models exhibit gender and\nother biases (Zhao et al. 2018, Rudinger et al. 2018, Webster et al. 2018). For exam-\nple the WinoBias dataset (Zhao et al., 2018) uses a variant of the Winograd Schema\nparadigm to test the extent to which coreference algorithms are biased toward link-\ning gendered pronouns with antecedents consistent with cultural stereotypes. As we\nsummarized in Chapter 6, embeddings replicate societal biases in their training test,\nsuch as associating men with historically sterotypical male occupations like doctors,\nand women with stereotypical female occupations like secretaries (Caliskan et al.\n2017, Garg et al. 2018).\nA WinoBias sentence contain two mentions corresponding to stereotypically-\nmale and stereotypically-female occupations and a gendered pronoun that must be\nlinked to one of them. The sentence cannot be disambiguated by the gender of the\npronoun, but a biased model might be distracted by this cue. Here is an example\nsentence: 23.11 \u2022 S UMMARY 27\n(23.78) The secretary called the physician iand told him iabout a new patient\n[pro-stereotypical]\n(23.79) The secretary called the physician iand told her iabout a new patient\n[anti-stereotypical]\nZhao et al. (2018) consider a coreference system to be biased if it is more accu-\nrate at linking pronouns consistent with gender stereotypical occupations (e.g., him\nwith physician in (23.78)) than linking pronouns inconsistent with gender-stereotypical\noccupations (e.g., herwith physician in (23.79)). They show that coreference sys-\ntems of all architectures (rule-based, feature-based machine learned, and end-to-\nend-neural) all show signi\ufb01cant bias, performing on average 21 F 1points worse in\nthe anti-stereotypical cases.\nOne possible source of this bias is that female entities are signi\ufb01cantly un-\nderrepresented in the OntoNotes dataset, used to train most coreference systems.\nZhao et al. (2018) propose a way to overcome this bias: they generate a second\ngender-swapped dataset in which all male entities in OntoNotes are replaced with\nfemale ones and vice versa, and retrain coreference systems on the combined orig-\ninal and swapped OntoNotes data, also using debiased GloVE embeddings (Boluk-\nbasi et al., 2016). The resulting coreference systems no longer exhibit bias on the\nWinoBias dataset, without signi\ufb01cantly impacting OntoNotes coreference accuracy.\nIn a follow-up paper, Zhao et al. (2019) show that the same biases exist in ELMo\ncontextualized word vector representations and coref systems that use them. They\nshowed that retraining ELMo with data augmentation again reduces or removes bias\nin coreference systems on WinoBias.\nWebster et al. (2018) introduces another dataset, GAP, and the task of Gendered\nPronoun Resolution as a tool for developing improved coreference algorithms for\ngendered pronouns. GAP is a gender-balanced labeled corpus of 4,454 sentences\nwith gendered ambiguous pronouns (by contrast, only 20% of the gendered pro-\nnouns in the English OntoNotes training data are feminine). The examples were\ncreated by drawing on naturally occurring sentences from Wikipedia pages to create\nhard to resolve cases with two named entities of the same gender and an ambiguous\npronoun that may refer to either person (or neither), like the following:\n(23.80) In May, Fujisawa joined Mari Motohashi\u2019s rink as the team\u2019s skip, moving\nback from Karuizawa to Kitami where shehad spent her junior days.\nWebster et al. (2018) show that modern coreference algorithms perform signif-\nicantly worse on resolving feminine pronouns than masculine pronouns in GAP.\nKurita et al. (2019) shows that a system based on BERT contextualized word repre-\nsentations shows similar bias.\n23.11 Summary\nThis chapter introduced the task of coreference resolution .\n\u2022 This is the task of linking together mentions in text which corefer , i.e. refer\nto the same discourse entity in the discourse model , resulting in a set of\ncoreference chains (also called clusters orentities ).\n\u2022 Mentions can be de\ufb01nite NPs orinde\ufb01nite NPs ,pronouns (including zero\npronouns ) ornames . 28 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\n\u2022 The surface form of an entity mention is linked to its information status\n(new,old, orinferrable ), and how accessible orsalient the entity is.\n\u2022 Some NPs are not referring expressions, such as pleonastic itinIt is raining .\n\u2022 Many corpora have human-labeled coreference annotations that can be used\nfor supervised learning, including OntoNotes for English, Chinese, and Ara-\nbic, ARRAU for English, and AnCora for Spanish and Catalan.\n\u2022 Mention detection can start with all nouns and named entities and then use\nanaphoricity classi\ufb01ers orreferentiality classi\ufb01ers to \ufb01lter out non-mentions.\n\u2022 Three common architectures for coreference are mention-pair ,mention-rank ,\nandentity-based , each of which can make use of feature-based or neural clas-\nsi\ufb01ers.\n\u2022 Modern coreference systems tend to be end-to-end, performing mention de-\ntection and coreference in a single end-to-end architecture.\n\u2022 Algorithms learn representations for text spans and heads, and learn to com-\npare anaphor spans with candidate antecedent spans.\n\u2022 Entity linking is the task of associating a mention in text with the representa-\ntion of some real-world entity in an ontology .\n\u2022 Coreference systems are evaluated by comparing with gold entity labels using\nprecision/recall metrics like MUC ,B3,CEAF ,BLANC , orLEA .\n\u2022 The Winograd Schema Challenge problems are dif\ufb01cult coreference prob-\nlems that seem to require world knowledge or sophisticated reasoning to solve.\n\u2022 Coreference systems exhibit gender bias which can be evaluated using datasets\nlike Winobias and GAP.\nBibliographical and Historical Notes\nCoreference has been part of natural language processing since the 1970s (Woods\net al. 1972, Winograd 1972). The discourse model and the entity-centric foundation\nof coreference was formulated by Karttunen (1969) (at the 3rd COLING confer-\nence), playing a role also in linguistic semantics (Heim 1982, Kamp 1981). But\nit was Bonnie Webber\u2019s 1978 dissertation and following work (Webber 1983) that\nexplored the model\u2019s computational aspects, providing fundamental insights into\nhow entities are represented in the discourse model and the ways in which they can\nlicense subsequent reference. Many of the examples she provided continue to chal-\nlenge theories of reference to this day.\nThe Hobbs algorithm9is a tree-search algorithm that was the \ufb01rst in a longHobbs\nalgorithm\nseries of syntax-based methods for identifying reference robustly in naturally occur-\nring text. The input to the Hobbs algorithm is a pronoun to be resolved, together\nwith a syntactic (constituency) parse of the sentences up to and including the cur-\nrent sentence. The details of the algorithm depend on the grammar used, but can be\nunderstood from a simpli\ufb01ed version due to Kehler et al. (2004) that just searches\nthrough the list of NPs in the current and prior sentences. This simpli\ufb01ed Hobbs\nalgorithm searches NPs in the following order: \u201c(i) in the current sentence from\nright-to-left, starting with the \ufb01rst NP to the left of the pronoun, (ii) in the previous\nsentence from left-to-right, (iii) in two sentences prior from left-to-right, and (iv) in\n9The simpler of two algorithms presented originally in Hobbs (1978). BIBLIOGRAPHICAL AND HISTORICAL NOTES 29\nthe current sentence from left-to-right, starting with the \ufb01rst noun group to the right\nof the pronoun (for cataphora). The \ufb01rst noun group that agrees with the pronoun\nwith respect to number, gender, and person is chosen as the antecedent\u201d (Kehler\net al., 2004).\nLappin and Leass (1994) was an in\ufb02uential entity-based system that used weights\nto combine syntactic and other features, extended soon after by Kennedy and Bogu-\nraev (1996) whose system avoids the need for full syntactic parses.\nApproximately contemporaneously centering (Grosz et al., 1995) was applied\nto pronominal anaphora resolution by Brennan et al. (1987), and a wide variety of\nwork followed focused on centering\u2019s use in coreference (Kameyama 1986, Di Eu-\ngenio 1990, Walker et al. 1994, Di Eugenio 1996, Strube and Hahn 1996, Kehler\n1997, Tetreault 2001, Iida et al. 2003). Kehler and Rohde (2013) show how center-\ning can be integrated with coherence-driven theories of pronoun interpretation. See\nChapter 24 for the use of centering in measuring discourse coherence.\nCoreference competitions as part of the US DARPA-sponsored MUC confer-\nences provided early labeled coreference datasets (the 1995 MUC-6 and 1998 MUC-\n7 corpora), and set the tone for much later work, choosing to focus exclusively\non the simplest cases of identity coreference (ignoring dif\ufb01cult cases like bridging,\nmetonymy, and part-whole) and drawing the community toward supervised machine\nlearning and metrics like the MUC metric (Vilain et al., 1995). The later ACE eval-\nuations produced labeled coreference corpora in English, Chinese, and Arabic that\nwere widely used for model training and evaluation.\nThis DARPA work in\ufb02uenced the community toward supervised learning begin-\nning in the mid-90s (Connolly et al. 1994, Aone and Bennett 1995, McCarthy and\nLehnert 1995). Soon et al. (2001) laid out a set of basic features, extended by Ng and\nCardie (2002b), and a series of machine learning models followed over the next 15\nyears. These often focused separately on pronominal anaphora resolution (Kehler\net al. 2004, Bergsma and Lin 2006), full NP coreference (Cardie and Wagstaff 1999,\nNg and Cardie 2002b, Ng 2005a) and de\ufb01nite NP reference (Poesio and Vieira 1998,\nVieira and Poesio 2000), as well as separate anaphoricity detection (Bean and Riloff\n1999, Bean and Riloff 2004, Ng and Cardie 2002a, Ng 2004), or singleton detection\n(de Marneffe et al., 2015).\nThe move from mention-pair to mention-ranking approaches was pioneered by\nYang et al. (2003) and Iida et al. (2003) who proposed pairwise ranking methods,\nthen extended by Denis and Baldridge (2008) who proposed to do ranking via a soft-\nmax over all prior mentions. The idea of doing mention detection, anaphoricity, and\ncoreference jointly in a single end-to-end model grew out of the early proposal of Ng\n(2005b) to use a dummy antecedent for mention-ranking, allowing \u2018non-referential\u2019\nto be a choice for coreference classi\ufb01ers, Denis and Baldridge\u2019s 2007 joint system\ncombining anaphoricity classi\ufb01er probabilities with coreference probabilities, the\nDenis and Baldridge (2008) ranking model, and the Rahman and Ng (2009) pro-\nposal to train the two models jointly with a single objective.\nSimple rule-based systems for coreference returned to prominence in the 2010s,\npartly because of their ability to encode entity-based features in a high-precision\nway (Zhou et al. 2004, Haghighi and Klein 2009, Raghunathan et al. 2010, Lee et al.\n2011, Lee et al. 2013, Hajishirzi et al. 2013) but in the end they suffered from an\ninability to deal with the semantics necessary to correctly handle cases of common\nnoun coreference.\nA return to supervised learning led to a number of advances in mention-ranking\nmodels which were also extended into neural architectures, for example using re- 30 CHAPTER 23 \u2022 C OREFERENCE RESOLUTION AND ENTITY LINKING\ninforcement learning to directly optimize coreference evaluation models Clark and\nManning (2016a), doing end-to-end coreference all the way from span extraction\n(Lee et al. 2017b, Zhang et al. 2018). Neural models also were designed to take\nadvantage of global entity-level information (Clark and Manning 2016b, Wiseman\net al. 2016, Lee et al. 2018).\nCoreference is also related to the task of entity linking discussed in Chapter 14.\nCoreference can help entity linking by giving more possible surface forms to help\nlink to the right Wikipedia page, and conversely entity linking can help improve\ncoreference resolution. Consider this example from Hajishirzi et al. (2013):\n(23.81) [Michael Eisner] 1and [Donald Tsang] 2announced the grand opening of\n[[Hong Kong] 3Disneyland] 4yesterday. [Eisner] 1thanked [the President] 2\nand welcomed [fans] 5to [the park] 4.\nIntegrating entity linking into coreference can help draw encyclopedic knowl-\nedge (like the fact that Donald Tsang is a president) to help disambiguate the men-\ntionthe President . Ponzetto and Strube (2006) 2007 and Ratinov and Roth (2012)\nshowed that such attributes extracted from Wikipedia pages could be used to build\nricher models of entity mentions in coreference. More recent research shows how to\ndo linking and coreference jointly (Hajishirzi et al. 2013, Zheng et al. 2013) or even\njointly with named entity tagging as well (Durrett and Klein 2014).\nThe coreference task as we introduced it involves a simplifying assumption that\nthe relationship between an anaphor and its antecedent is one of identity : the two\ncoreferring mentions refer to the identical discourse referent. In real texts, the rela-\ntionship can be more complex, where different aspects of a discourse referent can\nbe neutralized or refocused. For example (23.82) (Recasens et al., 2011) shows an\nexample of metonymy , in which the capital city Washington is used metonymically metonymy\nto refer to the US. (23.83-23.84) show other examples (Recasens et al., 2011):\n(23.82) a strict interpretation of a policy requires The U.S. to notify foreign\ndictators of certain coup plots ... Washington rejected the bid ...\n(23.83) I once crossed that border into Ashgh-Abad on Nowruz, the Persian New\nYear. In the South, everyone was celebrating New Year ; to the North, it\nwas a regular day.\n(23.84) In France, the president is elected for a term of seven years, while in the\nUnited States heis elected for a term of four years.\nFor further linguistic discussions of these complications of coreference see Puste-\njovsky (1991), van Deemter and Kibble (2000), Poesio et al. (2006), Fauconnier and\nTurner (2008), Versley (2008), and Barker (2010).\nNg (2017) offers a useful compact history of machine learning models in corefer-\nence resolution. There are three excellent book-length surveys of anaphora/coreference\nresolution, covering different time periods: Hirst (1981) (early work until about\n1981), Mitkov (2002) (1986-2001), and Poesio et al. (2016) (2001-2015).\nAndy Kehler wrote the Discourse chapter for the 2000 \ufb01rst edition of this text-\nbook, which we used as the starting point for the second-edition chapter, and there\nare some remnants of Andy\u2019s lovely prose still in this third-edition coreference chap-\nter.\nExercises Exercises 31\nAgarwal, O., S. Subramanian, A. Nenkova, and D. Roth.\n2019. Evaluation of named entity coreference. Work-\nshop on Computational Models of Reference, Anaphora\nand Coreference .\nAone, C. and S. W. Bennett. 1995. Evaluating automated\nand manual acquisition of anaphora resolution strategies.\nACL.\nAriel, M. 2001. Accessibility theory: An overview. In\nT. Sanders, J. Schilperoord, and W. Spooren, eds, Text\nRepresentation: Linguistic and Psycholinguistic Aspects ,\n29\u201387. Benjamins.\nBagga, A. and B. Baldwin. 1998. Algorithms for scoring\ncoreference chains. LREC Workshop on Linguistic Coref-\nerence .\nBamman, D., O. Lewke, and A. Mansoor. 2020. An anno-\ntated dataset of coreference in English literature. LREC .\nBarker, C. 2010. Nominals don\u2019t provide criteria of iden-\ntity. In M. Rathert and A. Alexiadou, eds, The Semantics\nof Nominalizations across Languages and Frameworks ,\n9\u201324. Mouton.\nBean, D. and E. Riloff. 1999. Corpus-based identi\ufb01cation of\nnon-anaphoric noun phrases. ACL.\nBean, D. and E. Riloff. 2004. Unsupervised learning of con-\ntextual role knowledge for coreference resolution. HLT-\nNAACL .\nBengtson, E. and D. Roth. 2008. Understanding the value of\nfeatures for coreference resolution. EMNLP .\nBerant, J., A. Chou, R. Frostig, and P. Liang. 2013. Semantic\nparsing on freebase from question-answer pairs. EMNLP .\nBergsma, S. and D. Lin. 2006. Bootstrapping path-based\npronoun resolution. COLING/ACL .\nBergsma, S., D. Lin, and R. Goebel. 2008. Distributional\nidenti\ufb01cation of non-referential pronouns. ACL.\nBj\u00a8orkelund, A. and J. Kuhn. 2014. Learning structured\nperceptrons for coreference resolution with latent an-\ntecedents and non-local features. ACL.\nBolukbasi, T., K.-W. Chang, J. Zou, V . Saligrama, and A. T.\nKalai. 2016. Man is to computer programmer as woman\nis to homemaker? Debiasing word embeddings. NeurIPS .\nBrennan, S. E., M. W. Friedman, and C. Pollard. 1987. A\ncentering approach to pronouns. ACL.\nCaliskan, A., J. J. Bryson, and A. Narayanan. 2017. Seman-\ntics derived automatically from language corpora contain\nhuman-like biases. Science , 356(6334):183\u2013186.\nCardie, C. and K. Wagstaff. 1999. Noun phrase coreference\nas clustering. EMNLP/VLC .\nChafe, W. L. 1976. Givenness, contrastiveness, de\ufb01niteness,\nsubjects, topics, and point of view. In C. N. Li, ed., Sub-\nject and Topic , 25\u201355. Academic Press.\nChang, K.-W., R. Samdani, and D. Roth. 2013. A con-\nstrained latent variable model for coreference resolution.\nEMNLP .\nChang, K.-W., R. Samdani, A. Rozovskaya, M. Sammons,\nand D. Roth. 2012. Illinois-Coref: The UI system in the\nCoNLL-2012 shared task. CoNLL .\nChen, C. and V . Ng. 2013. Linguistically aware coreference\nevaluation metrics. IJCNLP .Chomsky, N. 1981. Lectures on Government and Binding .\nForis.\nClark, K. and C. D. Manning. 2015. Entity-centric corefer-\nence resolution with model stacking. ACL.\nClark, K. and C. D. Manning. 2016a. Deep reinforce-\nment learning for mention-ranking coreference models.\nEMNLP .\nClark, K. and C. D. Manning. 2016b. Improving coreference\nresolution by learning entity-level distributed representa-\ntions. ACL.\nConnolly, D., J. D. Burger, and D. S. Day. 1994. A machine\nlearning approach to anaphoric reference. Proceedings\nof the International Conference on New Methods in Lan-\nguage Processing (NeMLaP) .\nCucerzan, S. 2007. Large-scale named entity disambiguation\nbased on Wikipedia data. EMNLP/CoNLL .\nDavis, E., L. Morgenstern, and C. L. Ortiz. 2017. The \ufb01rst\nWinograd schema challenge at IJCAI-16. AI Magazine ,\n38(3):97\u201398.\nDenis, P. and J. Baldridge. 2007. Joint determination\nof anaphoricity and coreference resolution using integer\nprogramming. NAACL-HLT .\nDenis, P. and J. Baldridge. 2008. Specialized models and\nranking for coreference resolution. EMNLP .\nDenis, P. and J. Baldridge. 2009. Global joint models for\ncoreference resolution and named entity classi\ufb01cation.\nProcesamiento del Lenguaje Natural , 42.\nDi Eugenio, B. 1990. Centering theory and the Italian\npronominal system. COLING .\nDi Eugenio, B. 1996. The discourse functions of Italian sub-\njects: A centering approach. COLING .\nDurrett, G. and D. Klein. 2013. Easy victories and uphill\nbattles in coreference resolution. EMNLP .\nDurrett, G. and D. Klein. 2014. A joint model for entity anal-\nysis: Coreference, typing, and linking. TACL , 2:477\u2013490.\nEmami, A., P. Trichelair, A. Trischler, K. Suleman,\nH. Schulz, and J. C. K. Cheung. 2019. The KNOWREF\ncoreference corpus: Removing gender and number cues\nfor dif\ufb01cult pronominal anaphora resolution. ACL.\nFauconnier, G. and M. Turner. 2008. The way we think: Con-\nceptual blending and the mind\u2019s hidden complexities . Ba-\nsic Books.\nFernandes, E. R., C. N. dos Santos, and R. L. Milidi \u00b4u. 2012.\nLatent structure perceptron with feature induction for un-\nrestricted coreference resolution. CoNLL .\nFerragina, P. and U. Scaiella. 2011. Fast and accurate anno-\ntation of short texts with wikipedia pages. IEEE Software ,\n29(1):70\u201375.\nFox, B. A. 1993. Discourse Structure and Anaphora: Writ-\nten and Conversational English . Cambridge.\nGarg, N., L. Schiebinger, D. Jurafsky, and J. Zou. 2018.\nWord embeddings quantify 100 years of gender and eth-\nnic stereotypes. Proceedings of the National Academy of\nSciences , 115(16):E3635\u2013E3644.\nGrosz, B. J. 1977. The Representation and Use of Focus\nin Dialogue Understanding . Ph.D. thesis, University of\nCalifornia, Berkeley.\nGrosz, B. J., A. K. Joshi, and S. Weinstein. 1995. Center-\ning: A framework for modeling the local coherence of\ndiscourse. Computational Linguistics , 21(2):203\u2013225. 32 Chapter 23 \u2022 Coreference Resolution and Entity Linking\nGundel, J. K., N. Hedberg, and R. Zacharski. 1993. Cog-\nnitive status and the form of referring expressions in dis-\ncourse. Language , 69(2):274\u2013307.\nHaghighi, A. and D. Klein. 2009. Simple coreference reso-\nlution with rich syntactic and semantic features. EMNLP .\nHajishirzi, H., L. Zilles, D. S. Weld, and L. Zettlemoyer.\n2013. Joint coreference resolution and named-entity link-\ning with multi-pass sieves. EMNLP .\nHaviland, S. E. and H. H. Clark. 1974. What\u2019s new? Acquir-\ning new information as a process in comprehension. Jour-\nnal of Verbal Learning and Verbal Behaviour , 13:512\u2013\n521.\nHawkins, J. A. 1978. De\ufb01niteness and inde\ufb01niteness: a study\nin reference and grammaticality prediction . Croom Helm\nLtd.\nHeim, I. 1982. The semantics of de\ufb01nite and inde\ufb01nite noun\nphrases . Ph.D. thesis, University of Massachusetts at\nAmherst.\nHirst, G. 1981. Anaphora in Natural Language Understand-\ning: A survey . Number 119 in Lecture notes in computer\nscience. Springer-Verlag.\nHobbs, J. R. 1978. Resolving pronoun references. Lingua ,\n44:311\u2013338.\nHou, Y ., K. Markert, and M. Strube. 2018. Unre-\nstricted bridging resolution. Computational Linguistics ,\n44(2):237\u2013284.\nIida, R., K. Inui, H. Takamura, and Y . Matsumoto. 2003. In-\ncorporating contextual cues in trainable models for coref-\nerence resolution. EACL Workshop on The Computa-\ntional Treatment of Anaphora .\nJi, H. and R. Grishman. 2011. Knowledge base population:\nSuccessful approaches and challenges. ACL.\nJoshi, M., O. Levy, D. S. Weld, and L. Zettlemoyer. 2019.\nBERT for coreference resolution: Baselines and analysis.\nEMNLP .\nKameyama, M. 1986. A property-sharing constraint in cen-\ntering. ACL.\nKamp, H. 1981. A theory of truth and semantic represen-\ntation. In J. Groenendijk, T. Janssen, and M. Stokhof,\neds,Formal Methods in the Study of Language , 189\u2013222.\nMathematical Centre, Amsterdam.\nKarttunen, L. 1969. Discourse referents. COLING . Preprint\nNo. 70.\nKehler, A. 1997. Current theories of centering for pronoun\ninterpretation: A critical evaluation. Computational Lin-\nguistics , 23(3):467\u2013475.\nKehler, A., D. E. Appelt, L. Taylor, and A. Simma. 2004. The\n(non)utility of predicate-argument frequencies for pro-\nnoun interpretation. HLT-NAACL .\nKehler, A. and H. Rohde. 2013. A probabilistic reconcil-\niation of coherence-driven and centering-driven theories\nof pronoun interpretation. Theoretical Linguistics , 39(1-\n2):1\u201337.\nKennedy, C. and B. K. Boguraev. 1996. Anaphora for every-\none: Pronominal anaphora resolution without a parser.\nCOLING .\nKocijan, V ., A.-M. Cretu, O.-M. Camburu, Y . Yordanov, and\nT. Lukasiewicz. 2019. A surprisingly robust trick for the\nWinograd Schema Challenge. ACL.Kolhatkar, V ., A. Roussel, S. Dipper, and H. Zinsmeister.\n2018. Anaphora with non-nominal antecedents in compu-\ntational linguistics: A survey. Computational Linguistics ,\n44(3):547\u2013612.\nKummerfeld, J. K. and D. Klein. 2013. Error-driven analysis\nof challenges in coreference resolution. EMNLP .\nKurita, K., N. Vyas, A. Pareek, A. W. Black, and\nY . Tsvetkov. 2019. Quantifying social biases in contex-\ntual word representations. 1st ACL Workshop on Gender\nBias for Natural Language Processing .\nLappin, S. and H. Leass. 1994. An algorithm for pronom-\ninal anaphora resolution. Computational Linguistics ,\n20(4):535\u2013561.\nLee, H., A. Chang, Y . Peirsman, N. Chambers, M. Surdeanu,\nand D. Jurafsky. 2013. Deterministic coreference resolu-\ntion based on entity-centric, precision-ranked rules. Com-\nputational Linguistics , 39(4):885\u2013916.\nLee, H., Y . Peirsman, A. Chang, N. Chambers, M. Surdeanu,\nand D. Jurafsky. 2011. Stanford\u2019s multi-pass sieve coref-\nerence resolution system at the CoNLL-2011 shared task.\nCoNLL .\nLee, H., M. Surdeanu, and D. Jurafsky. 2017a. A scaffolding\napproach to coreference resolution integrating statistical\nand rule-based models. Natural Language Engineering ,\n23(5):733\u2013762.\nLee, K., L. He, M. Lewis, and L. Zettlemoyer. 2017b. End-\nto-end neural coreference resolution. EMNLP .\nLee, K., L. He, and L. Zettlemoyer. 2018. Higher-\norder coreference resolution with coarse-to-\ufb01ne infer-\nence. NAACL HLT .\nLevesque, H. 2011. The Winograd Schema Challenge. Logi-\ncal Formalizations of Commonsense Reasoning \u2014 Papers\nfrom the AAAI 2011 Spring Symposium (SS-11-06) .\nLevesque, H., E. Davis, and L. Morgenstern. 2012. The\nWinograd Schema Challenge. KR-12 .\nLi, B. Z., S. Min, S. Iyer, Y . Mehdad, and W.-t. Yih. 2020.\nEf\ufb01cient one-pass end-to-end entity linking for questions.\nEMNLP .\nLuo, X. 2005. On coreference resolution performance met-\nrics. EMNLP .\nLuo, X. and S. Pradhan. 2016. Evaluation metrics. In\nM. Poesio, R. Stuckardt, and Y . Versley, eds, Anaphora\nresolution: Algorithms, resources, and applications ,\n141\u2013163. Springer.\nLuo, X., S. Pradhan, M. Recasens, and E. H. Hovy. 2014. An\nextension of BLANC to system mentions. ACL.\nde Marneffe, M.-C., M. Recasens, and C. Potts. 2015. Mod-\neling the lifespan of discourse entities with application to\ncoreference resolution. JAIR , 52:445\u2013475.\nMartschat, S. and M. Strube. 2014. Recall error analysis for\ncoreference resolution. EMNLP .\nMartschat, S. and M. Strube. 2015. Latent structures for\ncoreference resolution. TACL , 3:405\u2013418.\nMcCarthy, J. F. and W. G. Lehnert. 1995. Using decision\ntrees for coreference resolution. IJCAI-95 .\nMihalcea, R. and A. Csomai. 2007. Wikify!: Linking docu-\nments to encyclopedic knowledge. CIKM 2007 .\nMilne, D. and I. H. Witten. 2008. Learning to link with wiki-\npedia. CIKM 2008 . Exercises 33\nMitkov, R. 2002. Anaphora Resolution . Longman.\nMoosavi, N. S. and M. Strube. 2016. Which coreference\nevaluation metric do you trust? A proposal for a link-\nbased entity aware metric. ACL.\nNg, V . 2004. Learning noun phrase anaphoricity to improve\ncoreference resolution: Issues in representation and opti-\nmization. ACL.\nNg, V . 2005a. Machine learning for coreference resolution:\nFrom local classi\ufb01cation to global ranking. ACL.\nNg, V . 2005b. Supervised ranking for pronoun resolution:\nSome recent improvements. AAAI .\nNg, V . 2010. Supervised noun phrase coreference research:\nThe \ufb01rst \ufb01fteen years. ACL.\nNg, V . 2017. Machine learning for entity coreference reso-\nlution: A retrospective look at two decades of research.\nAAAI .\nNg, V . and C. Cardie. 2002a. Identifying anaphoric and non-\nanaphoric noun phrases to improve coreference resolu-\ntion. COLING .\nNg, V . and C. Cardie. 2002b. Improving machine learning\napproaches to coreference resolution. ACL.\nNissim, M., S. Dingare, J. Carletta, and M. Steedman. 2004.\nAn annotation scheme for information status in dialogue.\nLREC .\nPoesio, M., R. Stuckardt, and Y . Versley. 2016. Anaphora\nresolution: Algorithms, resources, and applications .\nSpringer.\nPoesio, M., P. Sturt, R. Artstein, and R. Filik. 2006. Under-\nspeci\ufb01cation and anaphora: Theoretical issues and pre-\nliminary evidence. Discourse processes , 42(2):157\u2013175.\nPoesio, M. and R. Vieira. 1998. A corpus-based investiga-\ntion of de\ufb01nite description use. Computational Linguis-\ntics, 24(2):183\u2013216.\nPonzetto, S. P. and M. Strube. 2006. Exploiting semantic role\nlabeling, WordNet and Wikipedia for coreference resolu-\ntion. HLT-NAACL .\nPonzetto, S. P. and M. Strube. 2007. Knowledge de-\nrived from Wikipedia for computing semantic related-\nness. JAIR , 30:181\u2013212.\nPradhan, S., E. H. Hovy, M. P. Marcus, M. Palmer,\nL. Ramshaw, and R. Weischedel. 2007a. OntoNotes: A\nuni\ufb01ed relational semantic representation. Proceedings of\nICSC .\nPradhan, S., X. Luo, M. Recasens, E. H. Hovy, V . Ng, and\nM. Strube. 2014. Scoring coreference partitions of pre-\ndicted mentions: A reference implementation. ACL.\nPradhan, S., A. Moschitti, N. Xue, O. Uryupina, and\nY . Zhang. 2012a. CoNLL-2012 shared task: Model-\ning multilingual unrestricted coreference in OntoNotes.\nCoNLL .\nPradhan, S., A. Moschitti, N. Xue, O. Uryupina, and\nY . Zhang. 2012b. Conll-2012 shared task: Modeling mul-\ntilingual unrestricted coreference in OntoNotes. CoNLL .\nPradhan, S., L. Ramshaw, M. P. Marcus, M. Palmer,\nR. Weischedel, and N. Xue. 2011. CoNLL-2011 shared\ntask: Modeling unrestricted coreference in OntoNotes.\nCoNLL .Pradhan, S., L. Ramshaw, R. Weischedel, J. MacBride, and\nL. Micciulla. 2007b. Unrestricted coreference: Identi-\nfying entities and events in OntoNotes. Proceedings of\nICSC 2007 .\nPrince, E. 1981. Toward a taxonomy of given-new infor-\nmation. In P. Cole, ed., Radical Pragmatics , 223\u2013255.\nAcademic Press.\nPustejovsky, J. 1991. The generative lexicon. Computational\nLinguistics , 17(4).\nRaghunathan, K., H. Lee, S. Rangarajan, N. Chambers,\nM. Surdeanu, D. Jurafsky, and C. D. Manning. 2010. A\nmulti-pass sieve for coreference resolution. EMNLP .\nRahman, A. and V . Ng. 2009. Supervised models for coref-\nerence resolution. EMNLP .\nRahman, A. and V . Ng. 2012. Resolving complex cases\nof de\ufb01nite pronouns: the Winograd Schema challenge.\nEMNLP .\nRatinov, L. and D. Roth. 2012. Learning-based multi-sieve\nco-reference resolution with knowledge. EMNLP .\nRecasens, M. and E. H. Hovy. 2011. BLANC: Implement-\ning the Rand index for coreference evaluation. Natural\nLanguage Engineering , 17(4):485\u2013510.\nRecasens, M., E. H. Hovy, and M. A. Mart \u00b4\u0131. 2011. Identity,\nnon-identity, and near-identity: Addressing the complex-\nity of coreference. Lingua , 121(6):1138\u20131152.\nRecasens, M. and M. A. Mart \u00b4\u0131. 2010. AnCora-CO: Corefer-\nentially annotated corpora for Spanish and Catalan. Lan-\nguage Resources and Evaluation , 44(4):315\u2013345.\nReichman, R. 1985. Getting Computers to Talk Like You and\nMe. MIT Press.\nRudinger, R., J. Naradowsky, B. Leonard, and\nB. Van Durme. 2018. Gender bias in coreference res-\nolution. NAACL HLT .\nSchiebinger, L. 2013. Machine translation: Analyzing gen-\nder.http://genderedinnovations.stanford.edu/\ncase-studies/nlp.html#tabs-2 .\nSoon, W. M., H. T. Ng, and D. C. Y . Lim. 2001. A ma-\nchine learning approach to coreference resolution of noun\nphrases. Computational Linguistics , 27(4):521\u2013544.\nSorokin, D. and I. Gurevych. 2018. Mixing context granu-\nlarities for improved entity linking on question answering\ndata across entity categories. *SEM .\nStrube, M. and U. Hahn. 1996. Functional centering. ACL.\nSu, Y ., H. Sun, B. Sadler, M. Srivatsa, I. G \u00a8ur, Z. Yan, and\nX. Yan. 2016. On generating characteristic-rich question\nsets for QA evaluation. EMNLP .\nTetreault, J. R. 2001. A corpus-based evaluation of center-\ning and pronoun resolution. Computational Linguistics ,\n27(4):507\u2013520.\nTrichelair, P., A. Emami, J. C. K. Cheung, A. Trischler,\nK. Suleman, and F. Diaz. 2018. On the evaluation of\ncommon-sense reasoning in natural language understand-\ning. NeurIPS 2018 Workshop on Critiquing and Correct-\ning Trends in Machine Learning .\nUryupina, O., R. Artstein, A. Bristot, F. Cavicchio, F. Del-\nogu, K. J. Rodriguez, and M. Poesio. 2020. Annotating\na broad range of anaphoric phenomena, in a variety of\ngenres: The ARRAU corpus. Natural Language Engi-\nneering , 26(1):1\u201334. 34 Chapter 23 \u2022 Coreference Resolution and Entity Linking\nvan Deemter, K. and R. Kibble. 2000. On coreferring: coref-\nerence in MUC and related annotation schemes. Compu-\ntational Linguistics , 26(4):629\u2013637.\nVersley, Y . 2008. Vagueness and referential ambiguity in a\nlarge-scale annotated corpus. Research on Language and\nComputation , 6(3-4):333\u2013353.\nVieira, R. and M. Poesio. 2000. An empirically based sys-\ntem for processing de\ufb01nite descriptions. Computational\nLinguistics , 26(4):539\u2013593.\nVilain, M., J. D. Burger, J. Aberdeen, D. Connolly, and\nL. Hirschman. 1995. A model-theoretic coreference scor-\ning scheme. MUC-6 .\nWalker, M. A., M. Iida, and S. Cote. 1994. Japanese dis-\ncourse and the process of centering. Computational Lin-\nguistics , 20(2):193\u2013232.\nWang, A., A. Singh, J. Michael, F. Hill, O. Levy, and S. R.\nBowman. 2018. Glue: A multi-task benchmark and anal-\nysis platform for natural language understanding. ICLR .\nWebber, B. L. 1978. A Formal Approach to Discourse\nAnaphora . Ph.D. thesis, Harvard University.\nWebber, B. L. 1983. So what can we talk about now? In\nM. Brady and R. C. Berwick, eds, Computational Models\nof Discourse , 331\u2013371. The MIT Press.\nWebber, B. L. 1991. Structure and ostension in the inter-\npretation of discourse deixis. Language and Cognitive\nProcesses , 6(2):107\u2013135.\nWebber, B. L. and B. Baldwin. 1992. Accommodating con-\ntext change. ACL.\nWebber, B. L. 1988. Discourse deixis: Reference to dis-\ncourse segments. ACL.\nWebster, K., M. Recasens, V . Axelrod, and J. Baldridge.\n2018. Mind the GAP: A balanced corpus of gendered\nambiguous pronouns. TACL , 6:605\u2013617.\nWinograd, T. 1972. Understanding Natural Language . Aca-\ndemic Press.\nWiseman, S., A. M. Rush, and S. M. Shieber. 2016. Learning\nglobal features for coreference resolution. NAACL HLT .\nWiseman, S., A. M. Rush, S. M. Shieber, and J. Weston.\n2015. Learning anaphoricity and antecedent ranking fea-\ntures for coreference resolution. ACL.\nWoods, W. A., R. M. Kaplan, and B. L. Nash-Webber. 1972.\nThe lunar sciences natural language information system:\nFinal report. Technical Report 2378, BBN.\nWu, L., F. Petroni, M. Josifoski, S. Riedel, and L. Zettle-\nmoyer. 2020. Scalable zero-shot entity linking with dense\nentity retrieval. EMNLP .\nYang, X., G. Zhou, J. Su, and C. L. Tan. 2003. Coreference\nresolution using competition learning approach. ACL.\nYih, W.-t., M. Richardson, C. Meek, M.-W. Chang, and\nJ. Suh. 2016. The value of semantic parse labeling for\nknowledge base question answering. ACL.\nZhang, R., C. N. dos Santos, M. Yasunaga, B. Xiang, and\nD. Radev. 2018. Neural coreference resolution with deep\nbiaf\ufb01ne attention by joint mention detection and mention\nclustering. ACL.\nZhao, J., T. Wang, M. Yatskar, R. Cotterell, V . Ordonez, and\nK.-W. Chang. 2019. Gender bias in contextualized word\nembeddings. NAACL HLT .Zhao, J., T. Wang, M. Yatskar, V . Ordonez, and K.-W.\nChang. 2018. Gender bias in coreference resolution:\nEvaluation and debiasing methods. NAACL HLT .\nZheng, J., L. Vilnis, S. Singh, J. D. Choi, and A. McCallum.\n2013. Dynamic knowledge-base alignment for corefer-\nence resolution. CoNLL .\nZhou, L., M. Ticrea, and E. H. Hovy. 2004. Multi-document\nbiography summarization. EMNLP .", "24.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\n24Discourse Coherence\nAnd even in our wildest and most wandering reveries, nay in our very dreams,\nwe shall \ufb01nd, if we re\ufb02ect, that the imagination ran not altogether at adven-\ntures, but that there was still a connection upheld among the different ideas,\nwhich succeeded each other. Were the loosest and freest conversation to be\ntranscribed, there would immediately be transcribed, there would immediately\nbe observed something which connected it in all its transitions.\nDavid Hume, An enquiry concerning human understanding , 1748\nOrson Welles\u2019 movie Citizen Kane was groundbreaking in many ways, perhaps most\nnotably in its structure. The story of the life of \ufb01ctional media magnate Charles\nFoster Kane, the movie does not proceed in chronological order through Kane\u2019s\nlife. Instead, the \ufb01lm begins with Kane\u2019s death (famously murmuring \u201cRosebud\u201d )\nand is structured around \ufb02ashbacks to his life inserted among scenes of a reporter\ninvestigating his death. The novel idea that the structure of a movie does not have\nto linearly follow the structure of the real timeline made apparent for 20th century\ncinematography the in\ufb01nite possibilities and impact of different kinds of coherent\nnarrative structures.\nBut coherent structure is not just a fact about movies or works of art. Like\nmovies, language does not normally consist of isolated, unrelated sentences, but\ninstead of collocated, structured, coherent groups of sentences. We refer to such\na coherent structured group of sentences as a discourse , and we use the word co- discourse\nherence to refer to the relationship between sentences that makes real discourses coherence\ndifferent than just random assemblages of sentences. The chapter you are now read-\ning is an example of a discourse, as is a news article, a conversation, a thread on\nsocial media, a Wikipedia page, and your favorite novel.\nWhat makes a discourse coherent? If you created a text by taking random sen-\ntences each from many different sources and pasted them together, would that be a\ncoherent discourse? Almost certainly not. Real discourses exhibit both local coher- local\nence andglobal coherence . Let\u2019s consider three ways in which real discourses are global\nlocally coherent;\nFirst, sentences or clauses in real discourses are related to nearby sentences in\nsystematic ways. Consider this example from Hobbs (1979):\n(24.1) John took a train from Paris to Istanbul. He likes spinach.\nThis sequence is incoherent because it is unclear to a reader why the second\nsentence follows the \ufb01rst; what does liking spinach have to do with train trips? In\nfact, a reader might go to some effort to try to \ufb01gure out how the discourse could be\ncoherent; perhaps there is a French spinach shortage? The very fact that hearers try\nto identify such connections suggests that human discourse comprehension involves\nthe need to establish this kind of coherence.\nBy contrast, in the following coherent example:\n(24.2) Jane took a train from Paris to Istanbul. She had to attend a conference. 2CHAPTER 24 \u2022 D ISCOURSE COHERENCE\nthe second sentence gives a REASON for Jane\u2019s action in the \ufb01rst sentence. Struc-\ntured relationships like REASON that hold between text units are called coherence\nrelations , and coherent discourses are structured by many such coherence relations.coherence\nrelations\nCoherence relations are introduced in Section 24.1.\nA second way a discourse can be locally coherent is by virtue of being \u201cabout\u201d\nsomeone or something. In a coherent discourse some entities are salient , and the\ndiscourse focuses on them and doesn\u2019t go back and forth between multiple entities.\nThis is called entity-based coherence . Consider the following incoherent passage,\nin which the salient entity seems to wildly swing from John to Jenny to the piano\nstore to the living room, back to Jenny, then the piano again:\n(24.3) John wanted to buy a piano for his living room.\nJenny also wanted to buy a piano.\nHe went to the piano store.\nIt was nearby.\nThe living room was on the second \ufb02oor.\nShe didn\u2019t \ufb01nd anything she liked.\nThe piano he bought was hard to get up to that \ufb02oor.\nEntity-based coherence models measure this kind of coherence by tracking salient\nentities across a discourse. For example Centering Theory (Grosz et al., 1995), theCentering\nTheory\nmost in\ufb02uential theory of entity-based coherence, keeps track of which entities in\nthe discourse model are salient at any point (salient entities are more likely to be\npronominalized or to appear in prominent syntactic positions like subject or object).\nIn Centering Theory, transitions between sentences that maintain the same salient\nentity are considered more coherent than ones that repeatedly shift between entities.\nThe entity grid model of coherence (Barzilay and Lapata, 2008) is a commonly entity grid\nused model that realizes some of the intuitions of the Centering Theory framework.\nEntity-based coherence is introduced in Section 24.3.\nFinally, discourses can be locally coherent by being topically coherent : nearbytopically\ncoherent\nsentences are generally about the same topic and use the same or similar vocab-\nulary to discuss these topics. Because topically coherent discourses draw from a\nsingle semantic \ufb01eld or topic, they tend to exhibit the surface property known as\nlexical cohesion (Halliday and Hasan, 1976): the sharing of identical or semanti- lexical cohesion\ncally related words in nearby sentences. For example, the fact that the words house ,\nchimney ,garret ,closet , and window \u2014 all of which belong to the same semantic\n\ufb01eld\u2014 appear in the two sentences in (24.4), or that they share the identical word\nshingled , is a cue that the two are tied together as a discourse:\n(24.4) Before winter I built a chimney , and shingled the sides of my house ...\nI have thus a tight shingled and plastered house ... with a garret and a\ncloset , a large window on each side....\nIn addition to the local coherence between adjacent or nearby sentences, dis-\ncourses also exhibit global coherence . Many genres of text are associated with\nparticular conventional discourse structures. Academic articles might have sections\ndescribing the Methodology or Results. Stories might follow conventional plotlines\nor motifs. Persuasive essays have a particular claim they are trying to argue for,\nand an essay might express this claim together with a structured set of premises that\nsupport the argument and demolish potential counterarguments. We\u2019ll introduce\nversions of each of these kinds of global coherence.\nWhy do we care about the local or global coherence of a discourse? Since co-\nherence is a property of a well-written text, coherence detection plays a part in any 24.1 \u2022 C OHERENCE RELATIONS 3\ntask that requires measuring the quality of a text. For example coherence can help\nin pedagogical tasks like essay grading or essay quality measurement that are trying\nto grade how well-written a human essay is (Somasundaran et al. 2014, Feng et al.\n2014, Lai and Tetreault 2018). Coherence can also help for summarization; knowing\nthe coherence relationship between sentences can help know how to select informa-\ntion from them. Finally, detecting incoherent text may even play a role in mental\nhealth tasks like measuring symptoms of schizophrenia or other kinds of disordered\nlanguage (Ditman and Kuperberg 2010, Elvev \u02daag et al. 2007, Bedi et al. 2015, Iter\net al. 2018).\n24.1 Coherence Relations\nRecall from the introduction the difference between passages (24.5) and (24.6).\n(24.5) Jane took a train from Paris to Istanbul. She likes spinach.\n(24.6) Jane took a train from Paris to Istanbul. She had to attend a conference.\nThe reason (24.6) is more coherent is that the reader can form a connection be-\ntween the two sentences, in which the second sentence provides a potential REASON\nfor the \ufb01rst sentences. This link is harder to form for (24.5). These connections\nbetween text spans in a discourse can be speci\ufb01ed as a set of coherence relations .coherence\nrelation\nThe next two sections describe two commonly used models of coherence relations\nand associated corpora: Rhetorical Structure Theory (RST), and the Penn Discourse\nTreeBank (PDTB).\n24.1.1 Rhetorical Structure Theory\nThe most commonly used model of discourse organization is Rhetorical Structure\nTheory (RST ) (Mann and Thompson, 1987). In RST relations are de\ufb01ned between RST\ntwo spans of text, generally a nucleus and a satellite . The nucleus is the unit that nucleus\nsatellite is more central to the writer\u2019s purpose and that is interpretable independently; the\nsatellite is less central and generally is only interpretable with respect to the nucleus.\nSome symmetric relations, however, hold between two nuclei.\nBelow are a few examples of RST coherence relations, with de\ufb01nitions adapted\nfrom the RST Treebank Manual (Carlson and Marcu, 2001).\nReason: The nucleus is an action carried out by an animate agent and the satellite\nis the reason for the nucleus.\n(24.7) [ NUC Jane took a train from Paris to Istanbul.] [ SATShe had to attend a\nconference.]\nElaboration: The satellite gives additional information or detail about the situation\npresented in the nucleus.\n(24.8) [ NUC Dorothy was from Kansas.] [ SATShe lived in the midst of the great\nKansas prairies.]\nEvidence: The satellite gives additional information or detail about the situation\npresented in the nucleus. The information is presented with the goal of convince the\nreader to accept the information presented in the nucleus.\n(24.9) [ NUC Kevin must be here.] [ SATHis car is parked outside.] 4CHAPTER 24 \u2022 D ISCOURSE COHERENCE\nAttribution: The satellite gives the source of attribution for an instance of reported\nspeech in the nucleus.\n(24.10) [ SATAnalysts estimated] [ NUC that sales at U.S. stores declined in the\nquarter, too]\nList: In this multinuclear relation, a series of nuclei is given, without contrast or\nexplicit comparison:\n(24.11) [ NUC Billy Bones was the mate; ] [ NUC Long John, he was quartermaster]\nRST relations are traditionally represented graphically; the asymmetric Nucleus-\nSatellite relation is represented with an arrow from the satellite to the nucleus:\nKevin must be here.His car is parked outsideevidence\nWe can also talk about the coherence of a larger text by considering the hierar-\nchical structure between coherence relations. Figure 24.1 shows the rhetorical struc-\nture of a paragraph from Marcu (2000a) for the text in (24.12) from the Scienti\ufb01c\nAmerican magazine.\n(24.12) With its distant orbit\u201350 percent farther from the sun than Earth\u2013and slim\natmospheric blanket, Mars experiences frigid weather conditions. Surface\ntemperatures typically average about -60 degrees Celsius (-76 degrees\nFahrenheit) at the equator and can dip to -123 degrees C near the poles. Only\nthe midday sun at tropical latitudes is warm enough to thaw ice on occasion,\nbut any liquid water formed in this way would evaporate almost instantly\nbecause of the low atmospheric pressure.\nTitle\n(1)\nMars2-9\nevidence\n2-3\nbackground\n  (2)\nWIth its \ndistant orbit  \n<p> -- 50\n percent \nfarther from \nthe sun than \nEarth -- </p> \nand slim \natmospheric \nblanket,(3)\nMars\nexperiences\nfrigid weather\nconditions.4-9\nelaboration-additional\n(4)\nSurface \ntemperatures\n typically average \nabout -60 \ndegrees Celsius\n <p> (-76 degrees\nFahrenheit)</p>\n at the equator4-5\nList\n(5)\nand can dip\nto -123\ndegrees C\nnear the\npoles.6-9\nContrast\n6-7\n(6)\nOnly the\nmidday sun at\ntropical latitudes\nis warm enough(7)\nto thaw ice\non occasion,purpose8-9\nexplanation-argumentative\n(8)\nbut any liquid water\nformed in this way \nwould evaporate \nalmost instantly(9)\nbecause of\nthe low\natmospheric\npressure.\nFigure 24.1 A discourse tree for the Scienti\ufb01c American text in (24.12), from Marcu (2000a). Note that\nasymmetric relations are represented with a curved arrow from the satellite to the nucleus.\nThe leaves in the Fig. 24.1 tree correspond to text spans of a sentence, clause or\nphrase that are called elementary discourse units orEDU s in RST; these units can EDU\nalso be referred to as discourse segments . Because these units may correspond to\narbitrary spans of text, determining the boundaries of an EDU is an important task\nfor extracting coherence relations. Roughly speaking, one can think of discourse 24.1 \u2022 C OHERENCE RELATIONS 5\nsegments as being analogous to constituents in sentence syntax, and indeed as we\u2019ll\nsee in Section 24.2 we generally draw on parsing algorithms to infer discourse struc-\nture.\nThere are corpora for many discourse coherence models; the RST Discourse\nTreeBank (Carlson et al., 2001) is the largest available discourse corpus. It con-\nsists of 385 English language documents selected from the Penn Treebank, with full\nRST parses for each one, using a large set of 78 distinct relations, grouped into 16\nclasses. RST treebanks exist also for Spanish, German, Basque, Dutch and Brazilian\nPortuguese (Braud et al., 2017).\nNow that we\u2019ve seen examples of coherence, we can see more clearly how a\ncoherence relation can play a role in summarization or information extraction. For\nexample, the nuclei of a text presumably express more important information than\nthe satellites, which might be dropped in a summary.\n24.1.2 Penn Discourse TreeBank (PDTB)\nThe Penn Discourse TreeBank (PDTB ) is a second commonly used dataset that PDTB\nembodies another model of coherence relations (Miltsakaki et al. 2004, Prasad et al.\n2008, Prasad et al. 2014). PDTB labeling is lexically grounded . Instead of asking\nannotators to directly tag the coherence relation between text spans, they were given\na list of discourse connectives , words that signal discourse relations, like because ,discourse\nconnectives\nalthough ,when ,since , oras a result . In a part of a text where these words marked a\ncoherence relation between two text spans, the connective and the spans were then\nannotated, as in Fig. 24.13, where the phrase as a result signals a causal relationship\nbetween what PDTB calls Arg1 (the \ufb01rst two sentences, here in italics) and Arg2\n(the third sentence, here in bold).\n(24.13) Jewelry displays in department stores were often cluttered and uninspired.\nAnd the merchandise was, well, fake. As a result , marketers of faux gems\nsteadily lost space in department stores to more fashionable\nrivals\u2014cosmetics makers.\n(24.14) In July, the Environmental Protection Agency imposed a gradual ban on\nvirtually all uses of asbestos. (implicit=as a result )By 1997, almost all\nremaining uses of cancer-causing asbestos will be outlawed.\nNot all coherence relations are marked by an explicit discourse connective, and\nso the PDTB also annotates pairs of neighboring sentences with no explicit signal,\nlike (24.14). The annotator \ufb01rst chooses the word or phrase that could have been its\nsignal (in this case as a result ), and then labels its sense. For example for the am-\nbiguous discourse connective since annotators marked whether it is using a C AUSAL\nor a T EMPORAL sense.\nThe \ufb01nal dataset contains roughly 18,000 explicit relations and 16,000 implicit\nrelations. Fig. 24.2 shows examples from each of the 4 major semantic classes, while\nFig. 24.3 shows the full tagset.\nUnlike the RST Discourse Treebank, which integrates these pairwise coherence\nrelations into a global tree structure spanning an entire discourse, the PDTB does not\nannotate anything above the span-pair level, making no commitment with respect to\nhigher-level discourse structure.\nThere are also treebanks using similar methods for other languages; (24.15)\nshows an example from the Chinese Discourse TreeBank (Zhou and Xue, 2015).\nBecause Chinese has a smaller percentage of explicit discourse connectives than\nEnglish (only 22% of all discourse relations are marked with explicit connectives, 6CHAPTER 24 \u2022 D ISCOURSE COHERENCE\nClass Type Example\nTEMPORAL SYNCHRONOUS The parishioners of St. Michael and All Angels stop to chat at\nthe church door, as members here always have. (Implicit while )\nIn the tower, \ufb01ve men and women pull rhythmically on ropes\nattached to the same \ufb01ve bells that \ufb01rst sounded here in 1614.\nCONTINGENCY REASON Also unlike Mr. Ruder, Mr. Breeden appears to be in a position\nto get somewhere with his agenda. (implicit=because )As a for-\nmer White House aide who worked closely with Congress,\nhe is savvy in the ways of Washington.\nCOMPARISON CONTRAST The U.S. wants the removal of what it perceives as barriers to\ninvestment; Japan denies there are real barriers.\nEXPANSION CONJUNCTION Not only do the actors stand outside their characters and make\nit clear they are at odds with them, but they often literally stand\non their heads.\nFigure 24.2 The four high-level semantic distinctions in the PDTB sense hierarchy\nTemporal Comparison\n\u000fAsynchronous \u000fContrast (Juxtaposition, Opposition)\n\u000fSynchronous (Precedence, Succession) \u000fPragmatic Contrast (Juxtaposition, Opposition)\n\u000fConcession (Expectation, Contra-expectation)\n\u000fPragmatic Concession\nContingency Expansion\n\u000fCause (Reason, Result) \u000fException\n\u000fPragmatic Cause (Justi\ufb01cation) \u000fInstantiation\n\u000fCondition (Hypothetical, General, Unreal\nPresent/Past, Factual Present/Past)\u000fRestatement (Speci\ufb01cation, Equivalence, Generalization)\n\u000fPragmatic Condition (Relevance, Implicit As-\nsertion)\u000fAlternative (Conjunction, Disjunction, Chosen Alterna-\ntive)\n\u000fList\nFigure 24.3 The PDTB sense hierarchy. There are four top-level c\u00aflasses, 16 types, and 23 subtypes (not all\ntypes have subtypes). 11 of the 16 types are commonly used for implicit argument classi\ufb01cation; the 5 types in\nitalics are too rare in implicit labeling to be used.\ncompared to 47% in English), annotators labeled this corpus by directly mapping\npairs of sentences to 11 sense tags, without starting with a lexical discourse connec-\ntor.\n(24.15) [ Conn\u4e3a] [Arg2\u63a8\u52a8\u56fe\u4eec\u6c5f\u5730\u533a\u5f00\u53d1]\uff0c[Arg1\u97e9\u56fd\u6350\u6b3e\u4e00\u767e\u4e07\u7f8e\u5143\n\u8bbe\u7acb\u4e86\u56fe\u4eec\u6c5f\u53d1\u5c55\u57fa\u91d1]\n\u201c[In order to] [ Arg2 promote the development of the Tumen River region],\n[Arg1 South Korea donated one million dollars to establish the Tumen\nRiver Development Fund].\u201d\nThese discourse treebanks have been used for shared tasks on multilingual dis-\ncourse parsing (Xue et al., 2016).\n24.2 Discourse Structure Parsing\nGiven a sequence of sentences, how can we automatically determine the coherence\nrelations between them? This task is often called discourse parsing (even thoughdiscourse\nparsing\nfor PDTB we are only assigning labels to leaf spans and not building a full parse 24.2 \u2022 D ISCOURSE STRUCTURE PARSING 7\ntree as we do for RST).\n24.2.1 EDU segmentation for RST parsing\nRST parsing is generally done in two stages. The \ufb01rst stage, EDU segmentation ,\nextracts the start and end of each EDU. The output of this stage would be a labeling\nlike the following:\n(24.16) [Mr. Rambo says] e1[that a 3.2-acre property] e2[overlooking the San\nFernando Valley] e3[is priced at $4 million] e4[because the late actor Erroll\nFlynn once lived there.] e5\nSince EDUs roughly correspond to clauses, early models of EDU segmentation\n\ufb01rst ran a syntactic parser, and then post-processed the output. Modern systems\ngenerally use neural sequence models supervised by the gold EDU segmentation in\ndatasets like the RST Discourse Treebank. Fig. 24.4 shows an example architecture\nsimpli\ufb01ed from the algorithm of Lukasik et al. (2020) that predicts for each token\nwhether or not it is a break. Here the input sentence is passed through an encoder\nand then passed through a linear layer and a softmax to produce a sequence of 0s\nand 1, where 1 indicates the start of an EDU.\nMr.RambosaysthatENCODER\u20260001linear layersoftmaxEDU break\nFigure 24.4 Predicting EDU segment beginnings from encoded text.\n24.2.2 RST parsing\nTools for building RST coherence structure for a discourse have long been based on\nsyntactic parsing algorithms like shift-reduce parsing (Marcu, 1999). Many modern\nRST parsers since Ji and Eisenstein (2014) draw on the neural syntactic parsers we\nsaw in Chapter 20, using representation learning to build representations for each\nspan, and training a parser to choose the correct shift and reduce actions based on\nthe gold parses in the training set.\nWe\u2019ll describe the shift-reduce parser of Yu et al. (2018). The parser state con-\nsists of a stack and a queue, and produces this structure by taking a series of actions\non the states. Actions include:\n\u2022shift : pushes the \ufb01rst EDU in the queue onto the stack creating a single-node\nsubtree.\n\u2022reduce (l,d): merges the top two subtrees on the stack, where lis the coherence\nrelation label, and dis the nuclearity direction, d2fNN ;NS ;SNg.\nAs well as the pop root operation, to remove the \ufb01nal tree from the stack.\nFig. 24.6 shows the actions the parser takes to build the structure in Fig. 24.5. 8CHAPTER 24 \u2022 D ISCOURSE COHERENCE\n560e1 e2 e3 e4attr elabelab e1: American Telephone & Telegraph Co. said it\ne2: will lay off 75 to 85 technicians here , effective Nov. 1.\ne3: The workers install , maintain and repair its private branch exchanges,\ne4: which are large intracompany telephone networks.\nFigure 1: An example of RST discourse tree, where {e1,e2,e3,e4}are EDUs, attr andelab are\ndiscourse relation labels, and arrows indicate the nuclearities of discourse relations.\nRST discourse parsing. Other studies still adopt discrete syntax features proposed by statistical models,\nfeeding them into neural network models (Braud et al., 2016; Braud et al., 2017).\nThe above approaches model syntax trees in an explicit way, requiring discrete syntax parsing outputs\nas inputs for RST parsing. These approaches may suffer from the error propagation problem. Syntax trees\nproduced by a supervised syntax parsing model could have errors, which may propagate into discourse\nparsing models. The problem could be extremely serious when inputs of discourse parsing have different\ndistributions with the training data of the supervised syntax parser. Recently, Zhang et al. (2017) suggest\nan alternative method, which extracts syntax features from a Bi-Af\ufb01ne dependency parser (Dozat and\nManning, 2016), and the method gives competitive performances on relation extraction. It actually\nrepresents syntax trees implicitly, thus it can reduce the error propagation problem.\nIn this work, we investigate the implicit syntax feature extraction approach for RST parsing. In ad-\ndition, we propose a transition-based neural model for this task, which is able to incorporate various\nfeatures \ufb02exibly. We exploit hierarchical bi-directional LSTMs (Bi-LSTMs) to encode texts, and further\nenhance the transition-based model with dynamic oracle. Based on the proposed model, we study the\neffectiveness of our proposed implicit syntax features. We conduct experiments on a standard RST dis-\ncourse TreeBank (Carlson et al., 2003). First, we evaluate the performance of our proposed transition-\nbased baseline, \ufb01nding that the model is able to achieve strong performances after applying dynamic\noracle. Then we evaluate the effectiveness of implicit syntax features extracted from a Bi-Af\ufb01ne depen-\ndency parser. Results show that the implicit syntax features are effective, giving better performances than\nexplicit Tree-LSTM (Li et al., 2015b). Our codes will be released for public under the Apache License\n2.0 at https://github.com/yunan4nlp/NNDisParser .\nIn summary, we mainly make the following two contributions in this work: (1) we propose a transition-\nbased neural RST discourse parsing model with dynamic oracle, (2) we compare three different syntactic\nintegration approaches proposed by us. The rest of the paper is organized as follows. Section 2 describes\nour proposed models including the transition-based neural model, the dynamic oracle strategy and the\nimplicit syntax feature extraction approach. Section 3 presents the experiments to evaluate our models.\nSection 4 shows the related work. Finally, section 5 draws conclusions.\n2 Transition-based Discourse Parsing\nWe follow Ji and Eisenstein (2014), exploiting a transition-based framework for RST discourse parsing.\nThe framework is conceptually simple and \ufb02exible to support arbitrary features, which has been widely\nused in a number of NLP tasks (Zhu et al., 2013; Dyer et al., 2015; Zhang et al., 2016). In addition, a\ntransition-based model formalizes a certain task into predicting a sequence of actions, which is essential\nsimilar to sequence-to-sequence models proposed recently (Bahdanau et al., 2014). In the following,\nwe \ufb01rst describe the transition system for RST discourse parsing, and then introduce our neural network\nmodel by its encoder and decoder parts, respectively. Thirdly, we present our proposed dynamic oracle\nstrategy aiming to enhance the transition-based model. Then we introduce the integration method of\nimplicit syntax features. Finally we describe the training method of our neural network models.\n2.1 The Transition-based System\nThe transition-based framework converts a structural learning problem into a sequence of action predic-\ntions, whose key point is a transition system. A transition system consists of two parts: states and actions.\nThe states are used to store partially-parsed results and the actions are used to control state transitions.\nFigure 24.5 Example RST discourse tree, showing four EDUs. Figure from Yu et al. (2018).\n561Step Stack Queue Action Relation\n1 ?e1,e2,e3,e4 SH ?\n2 e1 e2,e3,e4 SH ?\n3 e1,e2 e3,e4 RD(attr,SN) ?\n4 e1:2 e3,e4 SH de1e2\n5 e1:2,e3 e4 SH de1e2\n6 e1:2,e3,e4 ?RD(elab,NS) de1e2\n7 e1:2,e3:4 ?RD(elab,SN) de1e2,de3e4\n8 e1:4 ? PR de1e2,de3e4,\\e1:2e3:4\nTable 1: An example of the transition-based system for RST discourse parsing.\nThe initial state is an empty state, and the \ufb01nal state represents a full result. There are three kinds of\nactions in our transition system:\n\u2022Shift (SH), which removes the \ufb01rst EDU in the queue onto the stack, forming a single-node subtree.\n\u2022Reduce (RD) ( l,d), which merges the top two subtrees on the stack, where lis a discourse relation\nlabel, and d2{NN,NS,SN}indicates the relation nuclearity (nuclear (N) or satellite (S)).\n\u2022Pop Root (PR), which pops out the top tree on the stack, marking the decoding being completed,\nwhen the stack holds only one subtree and the queue is empty.\nGiven the RST tree as shown in Figure 1, it can be generated by the following action sequence: {SH,\nSH, RD (attr,SN) , SH, SH, RD (elab,NS) , RD (elab,SN) , PR}. Table 1 shows the decoding\nprocess in detail. By this way, we naturally convert RST discourse parsing into predicting a sequence of\ntransition actions, where each line includes a state and next step action referring to the tree.\n2.2 Encoder-Decoder\nPrevious transition-based RST discourse parsing studies exploit statistical models, using manually-\ndesigned discrete features (Sagae, 2009; Heilman and Sagae, 2015; Wang et al., 2017). In this work, we\npropose a transition-based neural model for RST discourse parsing, which follows an encoder-decoder\nframework. Given an input sequence of EDUs {e1,e2,. . . ,e n}, the encoder computes the input represen-\ntations {he\n1,he\n2,. . . ,he\nn}, and the decoder predicts next step actions conditioned on the encoder outputs.\n2.2.1 Encoder\nWe follow Li et al. (2016), using hierarchical Bi-LSTMs to encode the source EDU inputs, where the\n\ufb01rst-layer is used to represent sequencial words inside of EDUs, and the second layer is used to represent\nsequencial EDUs. Given an input sentence {w1,w2,. . . ,w m}, \ufb01rst we represent each word by its form\n(e.g., wi) and POS tag (e.g. ti), concatenating their neural embeddings. By this way, the input vectors\nof the \ufb01rst-layer Bi-LSTM are {xw\n1,xw\n2,. . . ,xw\nm}, where xw\ni=emb (wi)\u0000emb (ti), and then we apply\nBi-LSTM directly, obtaining:\n{hw\n1,hw\n2,. . . ,hw\nm}=Bi-LSTM ({xw\n1,xw\n2,. . . ,xw\nm}) (1)\nThe second-layer Bi-LSTM is built over sequential EDUs. We should \ufb01rst obtain a suitable representa-\ntion for each EDU, which is composed by a span of words inside a certain sentence. Assuming an EDU\nwith its words by {ws,ws+1,. . . ,w t}, after applying the \ufb01rst-layer Bi-LSTM, we obtain their representa-\ntions by {hw\ns,hw\ns+1...,hw\nt}, then we calculate the EDU representation by average pooling:\nxe=1\nt\u0000s+1tX\nshw\nk (2)\nWhen the EDU representations are ready, we apply the second-layer Bi-LSTM directly, resulting:\n{he\n1,he\n2,. . . ,he\nn}=Bi-LSTM ({xe\n1,xe\n2,. . . ,xe\nn}) (3)\nFigure 24.6 Parsing the example of Fig. 24.5 using a shift-reduce parser. Figure from Yu\net al. (2018).\nThe Yu et al. (2018) uses an encoder-decoder architecture, where the encoder\nrepresents the input span of words and EDUs using a hierarchical biLSTM. The\n\ufb01rst biLSTM layer represents the words inside an EDU, and the second represents\nthe EDU sequence. Given an input sentence w1;w2; :::;wm, the words can be repre-\nsented as usual (by static embeddings, combinations with character embeddings or\ntags, or contextual embeddings) resulting in an input word representation sequence\nxw\n1;xw\n2; :::;xw\nm. The result of the word-level biLSTM is then a sequence of hwvalues:\nhw\n1;hw\n2; :::;hw\nm=biLSTM (xw\n1;xw\n2; :::;xw\nm) (24.17)\nAn EDU of span ws;ws+1; :::;wtthen has biLSTM output representation hw\ns;hw\ns+1; :::;hw\nt,\nand is represented by average pooling:\nxe=1\nt\u0000s+1tX\nk=shw\nk (24.18)\nThe second layer uses this input to compute a \ufb01nal representation of the sequence of\nEDU representations he:\nhe\n1;he\n2; :::;he\nn=biLSTM (xe\n1;xe\n2; :::;xe\nn) (24.19)\nThe decoder is then a feedforward network Wthat outputs an action obased on a\nconcatenation of the top three subtrees on the stack ( so;s1;s2) plus the \ufb01rst EDU in\nthe queue (q0):\no=W(ht\ns0;ht\ns1;ht\ns2;he\nq0) (24.20)\nwhere the representation of the EDU on the queue he\nq0comes directly from the\nencoder, and the three hidden vectors representing partial trees are computed by\naverage pooling over the encoder output for the EDUs in those trees:\nhts=1\nj\u0000i+1jX\nk=ihe\nk (24.21) 24.2 \u2022 D ISCOURSE STRUCTURE PARSING 9\nTraining \ufb01rst maps each RST gold parse tree into a sequence of oracle actions, and\nthen uses the standard cross-entropy loss (with l2regularization) to train the system\nto take such actions. Give a state Sand oracle action a, we \ufb01rst compute the decoder\noutput using Eq. 24.20, apply a softmax to get probabilities:\npa=exp(oa)P\na02Aexp(oa0)(24.22)\nand then computing the cross-entropy loss:\nLCE() =\u0000log(pa)+l\n2jjQjj2(24.23)\nRST discourse parsers are evaluated on the test section of the RST Discourse Tree-\nbank, either with gold EDUs or end-to-end, using the RST-Pareval metrics (Marcu,\n2000b). It is standard to \ufb01rst transform the gold RST trees into right-branching bi-\nnary trees, and to report four metrics: trees with no labels (S for Span), labeled\nwith nuclei (N), with relations (R), or both (F for Full), for each metric computing\nmicro-averaged F 1over all spans from all documents (Marcu 2000b, Morey et al.\n2017).\n24.2.3 PDTB discourse parsing\nPDTB discourse parsing, the task of detecting PDTB coherence relations between\nspans, is sometimes called shallow discourse parsing because the task just involvesshallow\ndiscourse\nparsing\ufb02at relationships between text spans, rather than the full trees of RST parsing.\nThe set of four subtasks for PDTB discourse parsing was laid out by Lin et al.\n(2014) in the \ufb01rst complete system, with separate tasks for explicit (tasks 1-3) and\nimplicit (task 4) connectives:\n1. Find the discourse connectives (disambiguating them from non-discourse uses)\n2. Find the two spans for each connective\n3. Label the relationship between these spans\n4. Assign a relation between every adjacent pair of sentences\nMany systems have been proposed for Task 4: taking a pair of adjacent sentences\nas input and assign a coherence relation sense label as output. The setup often fol-\nlows Lin et al. (2009) in assuming gold sentence span boundaries and assigning each\nadjacent span one of the 11 second-level PDTB tags or none (removing the 5 very\nrare tags of the 16 shown in italics in Fig. 24.3).\nA simple but very strong algorithm for Task 4 is to represent each of the two\nspans by BERT embeddings and take the last layer hidden state corresponding to\nthe position of the [CLS] token, pass this through a single layer tanh feedforward\nnetwork and then a softmax for sense classi\ufb01cation (Nie et al., 2019).\nEach of the other tasks also have been addressed. Task 1 is to disambiguat-\ning discourse connectives from their non-discourse use. For example as Pitler and\nNenkova (2009) point out, the word andis a discourse connective linking the two\nclauses by an elaboration/expansion relation in (24.24) while it\u2019s a non-discourse\nNP conjunction in (24.25):\n(24.24) Selling picked up as previous buyers bailed out of their positions and\naggressive short sellers\u2014anticipating further declines\u2014moved in.\n(24.25) My favorite colors are blue and green. 10 CHAPTER 24 \u2022 D ISCOURSE COHERENCE\nSimilarly, once is a discourse connective indicating a temporal relation in (24.26),\nbut simply a non-discourse adverb meaning \u2018formerly\u2019 and modifying used in (24.27):\n(24.26) The asbestos \ufb01ber, crocidolite, is unusually resilient once it enters the\nlungs, with even brief exposures to it causing symptoms that show up\ndecades later, researchers said.\n(24.27) A form of asbestos once used to make Kent cigarette \ufb01lters has caused a\nhigh percentage of cancer deaths among a group of workers exposed to it\nmore than 30 years ago, researchers reported.\nDetermining whether a word is a discourse connective is thus a special case\nof word sense disambiguation. Early work on disambiguation showed that the 4\nPDTB high-level sense classes could be disambiguated with high (94%) accuracy\nused syntactic features from gold parse trees (Pitler and Nenkova, 2009). Recent\nwork performs the task end-to-end from word inputs using a biLSTM-CRF with\nBIO outputs ( B-CONN ,I-CONN ,O) (Yu et al., 2019).\nFor task 2, PDTB spans can be identi\ufb01ed with the same sequence models used to\n\ufb01nd RST EDUs: a biLSTM sequence model with pretrained contextual embedding\n(BERT) inputs (Muller et al., 2019). Simple heuristics also do pretty well as a base-\nline at \ufb01nding spans, since 93% of relations are either completely within a single\nsentence or span two adjacent sentences, with one argument in each sentence (Biran\nand McKeown, 2015).\n24.3 Centering and Entity-Based Coherence\nA second way a discourse can be coherent is by virtue of being \u201cabout\u201d some entity.\nThis idea that at each point in the discourse some entity is salient, and a discourse\nis coherent by continuing to discuss the same entity, appears early in functional lin-\nguistics and the psychology of discourse (Chafe 1976, Kintsch and Van Dijk 1978),\nand soon made its way to computational models. In this section we introduce two\nmodels of this kind of entity-based coherence :Centering Theory (Grosz et al., entity-based\n1995), and the entity grid model of Barzilay and Lapata (2008).\n24.3.1 Centering\nCentering Theory (Grosz et al., 1995) is a theory of both discourse salience andCentering\nTheory\ndiscourse coherence. As a model of discourse salience, Centering proposes that at\nany given point in the discourse one of the entities in the discourse model is salient:\nit is being \u201ccentered\u201d on. As a model of discourse coherence, Centering proposes\nthat discourses in which adjacent sentences CONTINUE to maintain the same salient\nentity are more coherent than those which SHIFT back and forth between multiple\nentities (we will see that CONTINUE and SHIFT are technical terms in the theory).\nThe following two texts from Grosz et al. (1995) which have exactly the same\npropositional content but different saliences, can help in understanding the main\nCentering intuition.\n(24.28) a. John went to his favorite music store to buy a piano.\nb. He had frequented the store for many years.\nc. He was excited that he could \ufb01nally buy a piano.\nd. He arrived just as the store was closing for the day. 24.3 \u2022 C ENTERING AND ENTITY -BASED COHERENCE 11\n(24.29) a. John went to his favorite music store to buy a piano.\nb. It was a store John had frequented for many years.\nc. He was excited that he could \ufb01nally buy a piano.\nd. It was closing just as John arrived.\nWhile these two texts differ only in how the two entities (John and the store) are\nrealized in the sentences, the discourse in (24.28) is intuitively more coherent than\nthe one in (24.29). As Grosz et al. (1995) point out, this is because the discourse\nin (24.28) is clearly about one individual, John, describing his actions and feelings.\nThe discourse in (24.29), by contrast, focuses \ufb01rst on John, then the store, then back\nto John, then to the store again. It lacks the \u201caboutness\u201d of the \ufb01rst discourse.\nCentering Theory realizes this intuition by maintaining two representations for\neach utterance Un. The backward-looking center ofUn, denoted as Cb(Un), rep-backward-\nlooking\ncenterresents the current salient entity, the one being focused on in the discourse after Un\nis interpreted. The forward-looking centers ofUn, denoted as Cf(Un), are a setforward-looking\ncenter\nof potential future salient entities, the discourse entities evoked by Unany of which\ncould serve as Cb(the salient entity) of the following utterance, i.e. Cb(Un+1).\nThe set of forward-looking centers Cf(Un)are ranked according to factors like\ndiscourse salience and grammatical role (for example subjects are higher ranked\nthan objects, which are higher ranked than all other grammatical roles). We call the\nhighest-ranked forward-looking center Cp(for \u201cpreferred center\u201d). Cpis a kind of\nprediction about what entity will be talked about next. Sometimes the next utterance\nindeed talks about this entity, but sometimes another entity becomes salient instead.\nWe\u2019ll use here the algorithm for centering presented in Brennan et al. (1987),\nwhich de\ufb01nes four intersentential relationships between a pair of utterances Unand\nUn+1that depend on the relationship between Cb(Un+1),Cb(Un), and Cp(Un+1);\nthese are shown in Fig. 24.7.\nCb(Un+1) =Cb(Un) Cb(Un+1)6=Cb(Un)\nor unde\ufb01ned Cb(Un)\nCb(Un+1) =Cp(Un+1) Continue Smooth-Shift\nCb(Un+1)6=Cp(Un+1) Retain Rough-Shift\nFigure 24.7 Centering Transitions for Rule 2 from Brennan et al. (1987).\nThe following rules are used by the algorithm:\nRule 1: If any element of Cf(Un)is realized by a pronoun in utterance\nUn+1, then Cb(Un+1)must be realized as a pronoun also.\nRule 2 :Transition states are ordered. Continue is preferred to Retain is\npreferred to Smooth-Shift is preferred to Rough-Shift.\nRule 1 captures the intuition that pronominalization (including zero-anaphora)\nis a common way to mark discourse salience. If there are multiple pronouns in an\nutterance realizing entities from the previous utterance, one of these pronouns must\nrealize the backward center Cb; if there is only one pronoun, it must be Cb.\nRule 2 captures the intuition that discourses that continue to center the same en-\ntity are more coherent than ones that repeatedly shift to other centers. The transition\ntable is based on two factors: whether the backward-looking center Cbis the same\nfrom UntoUn+1and whether this discourse entity is the one that is preferred ( Cp)\nin the new utterance Un+1. If both of these hold, a CONTINUE relation, the speaker\nhas been talking about the same entity and is going to continue talking about that 12 CHAPTER 24 \u2022 D ISCOURSE COHERENCE\nentity. In a RETAIN relation, the speaker intends to SHIFT to a new entity in a future\nutterance and meanwhile places the current entity in a lower rank Cf. In a SHIFT\nrelation, the speaker is shifting to a new salient entity.\nLet\u2019s walk though the start of (24.28) again, repeated as (24.30), showing the\nrepresentations after each utterance is processed.\n(24.30) John went to his favorite music store to buy a piano. ( U1)\nHe was excited that he could \ufb01nally buy a piano. ( U2)\nHe arrived just as the store was closing for the day. ( U3)\nIt was closing just as John arrived ( U4)\nUsing the grammatical role hierarchy to order the C f, for sentence U1we get:\nCf(U1):fJohn, music store, piano g\nCp(U1): John\nCb(U1): unde\ufb01ned\nand then for sentence U2:\nCf(U2):fJohn, pianog\nCp(U2): John\nCb(U2): John\nResult: Continue ( Cp(U2)=Cb(U2);Cb(U1)unde\ufb01ned)\nThe transition from U1toU2is thus a CONTINUE . Completing this example is left\nas exercise (1) for the reader\n24.3.2 Entity Grid model\nCentering embodies a particular theory of how entity mentioning leads to coher-\nence: that salient entities appear in subject position or are pronominalized, and that\ndiscourses are salient by means of continuing to mention the same entity in such\nways.\nThe entity grid model of Barzilay and Lapata (2008) is an alternative way to entity grid\ncapture entity-based coherence: instead of having a top-down theory, the entity-grid\nmodel using machine learning to induce the patterns of entity mentioning that make\na discourse more coherent.\nThe model is based around an entity grid , a two-dimensional array that repre-\nsents the distribution of entity mentions across sentences. The rows represent sen-\ntences, and the columns represent discourse entities (most versions of the entity grid\nmodel focus just on nominal mentions). Each cell represents the possible appearance\nof an entity in a sentence, and the values represent whether the entity appears and its\ngrammatical role. Grammatical roles are subject ( S), object ( O), neither ( X), or ab-\nsent (\u2013); in the implementation of Barzilay and Lapata (2008), subjects of passives\nare represented with O, leading to a representation with some of the characteristics\nof thematic roles.\nFig. 24.8 from Barzilay and Lapata (2008) shows a grid for the text shown in\nFig. 24.9. There is one row for each of the six sentences. The second column, for\nthe entity \u2018trial\u2019, is O\u2013 \u2013 \u2013 X, showing that the trial appears in the \ufb01rst sentence as\ndirect object, in the last sentence as an oblique, and does not appear in the middle\nsentences. The third column, for the entity Microsoft, shows that it appears as sub-\nject in sentence 1 (it also appears as the object of the preposition against , but entities\nthat appear multiple times are recorded with their highest-ranked grammatical func-\ntion). Computing the entity grids requires extracting entities and doing coreference 24.3 \u2022 C ENTERING AND ENTITY -BASED COHERENCE 13\nComputational Linguistics Volume 34, Number 1\nthese patterns can be encoded as feature vectors appropriate for performing coherence-\nrelated ranking and classi\ufb01cation tasks.\n3.1 The Entity-Grid Discourse Representation\nEach text is represented by an entity grid ,at w o - d i m e n s i o n a la r r a yt h a tc a p t u r e s\nthe distribution of discourse entities across text sentences. We follow Miltsakaki and\nKukich (2000) in assuming that our unit of analysis is the traditional sentence (i.e., a\nmain clause with accompanying subordinate and adjunct clauses). The rows of the\ngrid correspond to sentences, and the columns correspond to discourse entities. By\ndiscourse entity we mean a class of coreferent noun phrases (we explain in Section 3.3\nhow coreferent entities are identi\ufb01ed). For each occurrence of a discourse entity in the\ntext, the corresponding grid cell contains information about its presence or absence\nin a sequence of sentences. In addition, for entities present in a given sentence, grid\ncells contain information about their syntactic role. Such information can be expressed\nin many ways (e.g., using constituent labels or thematic role information). Because\ngrammatical relations \ufb01gure prominently in entity-based theories of local coherence (see\nSection 2), they serve as a logical point of departure. Each grid cell thus corresponds to\na string from a set of categories re\ufb02ecting whether the entity in question is a subject ( S),\nobject ( O), or neither ( X). Entities absent from a sentence are signaled by gaps ( \u2013).\nGrammatical role information can be extracted from the output of a broad-coverage\ndependency parser (Lin 2001; Briscoe and Carroll 2002) or any state-of-the art statistical\nparser (Collins 1997; Charniak 2000). We discuss how this information was computed\nfor our experiments in Section 3.3.\nTable 1 illustrates a fragment of an entity grid constructed for the text in Table 2.\nBecause the text contains six sentences, the grid columns are of length six. Consider\nfor instance the grid column for the entity trial ,[O\u2013\u2013\u2013\u2013 X].I tr e c o r d st h a t trial is\npresent in sentences 1 and 6 (as Oand X,r e s p e c t i v e l y )b u ti sa b s e n tf r o mt h er e s to ft h e\nsentences. Also note that the grid in Table 1 takes coreference resolution into account.\nEven though the same entity appears in different linguistic forms, for example, Microsoft\nCorp. ,Microsoft ,a n d the company , it is mapped to a single entry in the grid (see the\ncolumn introduced by Microsoft in Table 1).\nTable 1\nA fragment of the entity grid. Noun phrases are represented by their head nouns. Grid cells\ncorrespond to grammatical roles: subjects ( S), objects ( O), or neither ( X).Department\nTrial\nMicrosoft\nEvidence\nCompetitors\nMarkets\nProducts\nBrands\nCase\nNetscape\nSoftware\nTactics\nGovernment\nSuit\nEarnings\n1SO SXO \u2013\u2013\u2013\u2013\u2013\u2013\u2013\u2013\u2013\u2013 1\n2\u2013\u2013 O\u2013\u2013 XSO \u2013\u2013\u2013\u2013 \u2013\u2013\u2013 2\n3\u2013\u2013 SO \u2013\u2013\u2013\u2013 SOO \u2013\u2013\u2013 \u2013 3\n4\u2013\u2013 S\u2013\u2013\u2013\u2013\u2013\u2013\u2013\u2013 S\u2013\u2013\u2013 4\n5\u2013\u2013\u2013\u2013\u2013\u2013\u2013\u2013\u2013\u2013\u2013\u2013 SO \u20135\n6\u2013XS \u2013\u2013\u2013\u2013\u2013 \u2013\u2013\u2013 \u2013 \u2013 \u2013 O6\n6\nFigure 24.8 Part of the entity grid for the text in Fig. 24.9. Entities are listed by their head\nnoun; each cell represents whether an entity appears as subject ( S), object ( O), neither ( X), or\nis absent (\u2013). Figure from Barzilay and Lapata (2008).\nBarzilay and Lapata Modeling Local Coherence\nTable 2\nSummary augmented with syntactic annotations for grid computation.\n1 [The Justice Department]Sis conducting an [anti-trust trial]Oagainst [Microsoft Corp.]X\nwith [evidence]Xthat [the company]Sis increasingly attempting to crush [competitors]O.\n2[ M i c r o s o f t ]Ois accused of trying to forcefully buy into [markets]Xwhere [its own\nproducts]Sare not competitive enough to unseat [established brands]O.\n3[ T h e c a s e ]Srevolves around [evidence]Oof [Microsoft]Saggressively pressuring\n[Netscape]Ointo merging [browser software]O.\n4[ M i c r o s o f t ]Sclaims [its tactics]Sare commonplace and good economically.\n5 [The government]Smay \ufb01le [a civil suit]Oruling that [conspiracy]Sto curb [competition]O\nthrough [collusion]Xis [a violation of the Sherman Act]O.\n6[ M i c r o s o f t ]Scontinues to show [increased earnings]Odespite [the trial]X.\nWhen a noun is attested more than once with a different grammatical role in the\nsame sentence, we default to the role with the highest grammatical ranking: subjects are\nranked higher than objects, which in turn are ranked higher than the rest. For example,\nthe entity Microsoft is mentioned twice in Sentence 1 with the grammatical roles x(for\nMicrosoft Corp. )a n d s(forthe company ), but is represented only by sin the grid (see\nTables 1 and 2).\n3.2 Entity Grids as Feature Vectors\nAf u n d a m e n t a la s s u m p t i o nu n d e r l y i n go u ra p p r o a c hi st h a tt h ed i s t r i b u t i o no fe n t i t i e s\nin coherent texts exhibits certain regularities re\ufb02ected in grid topology. Some of these\nregularities are formalized in Centering Theory as constraints on transitions of the\nlocal focus in adjacent sentences. Grids of coherent texts are likely to have some dense\ncolumns (i.e., columns with just a few gaps, such as Microsoft in Table 1) and many\nsparse columns which will consist mostly of gaps (see markets andearnings in Table 1).\nOne would further expect that entities corresponding to dense columns are more often\nsubjects or objects. These characteristics will be less pronounced in low-coherence texts.\nInspired by Centering Theory, our analysis revolves around patterns of local entity\ntransitions. A local entity transition is a sequence {S,O,X,\u2013}nthat represents entity\noccurrences and their syntactic roles in nadjacent sentences. Local transitions can be\neasily obtained from a grid as continuous subsequences of each column. Each transition\nwill have a certain probability in a given grid. For instance, the probability of the\ntransition [S\u2013]in the grid from Table 1 is 0 .08 (computed as a ratio of its frequency\n[i.e., six] divided by the total number of transitions of length two [i.e., 75]). Each text\ncan thus be viewed as a distribution de\ufb01ned over transition types.\nWe can now go one step further and represent each text by a \ufb01xed set of transition\nsequences using a standard feature vector notation. Each grid rendering jof a document\ndicorresponds to a feature vector \u03a6(xij)=(p1(xij),p2(xij),... ,pm(xij)), where mis the\nnumber of all prede\ufb01ned entity transitions, and pt(xij)t h ep r o b a b i l i t yo ft r a n s i t i o n t\nin grid xij.T h i sf e a t u r ev e c t o rr e p r e s e n t a t i o ni su s e f u l l ya m e n a b l et om a c h i n el e a r n i n g\nalgorithms (see our experiments in Sections 4\u20136). Furthermore, it allows the consid-\neration of large numbers of transitions which could potentially uncover novel entity\ndistribution patterns relevant for coherence assessment or other coherence-related tasks.\nNote that considerable latitude is available when specifying the transition types to\nbe included in a feature vector. These can be all transitions of a given length (e.g., two\nor three) or the most frequent transitions within a document collection. An example of\n7\nFigure 24.9 A discourse with the entities marked and annotated with grammatical func-\ntions. Figure from Barzilay and Lapata (2008).\nresolution to cluster them into discourse entities (Chapter 23) as well as parsing the\nsentences to get grammatical roles.\nIn the resulting grid, columns that are dense (like the column for Microsoft) in-\ndicate entities that are mentioned often in the texts; sparse columns (like the column\nfor earnings) indicate entities that are mentioned rarely.\nIn the entity grid model, coherence is measured by patterns of local entity tran-\nsition . For example, Department is a subject in sentence 1, and then not men-\ntioned in sentence 2; this is the transition [ S\u2013]. The transitions are thus sequences\nfS,O X, \u2013gnwhich can be extracted as continuous cells from each column. Each\ntransition has a probability; the probability of [ S\u2013] in the grid from Fig. 24.8 is 0.08\n(it occurs 6 times out of the 75 total transitions of length two). Fig. 24.10 shows the\ndistribution over transitions of length 2 for the text of Fig. 24.9 (shown as the \ufb01rst\nrowd1), and 2 other documents.\nComputational Linguistics Volume 34, Number 1\naf e a t u r es p a c ew i t ht r a n s i t i o n so fl e n g t ht w oi si l l u s t r a t e di nT a b l e3 .T h es e c o n dr o w\n(introduced by d1)i st h ef e a t u r ev e c t o rr e p r e s e n t a t i o no ft h eg r i di nT a b l e1 .\n3.3 Grid Construction: Linguistic Dimensions\nOne of the central research issues in developing entity-based models of coherence is\ndetermining what sources of linguistic knowledge are essential for accurate prediction,\nand how to encode them succinctly in a discourse representation. Previous approaches\ntend to agree on the features of entity distribution related to local coherence\u2014the\ndisagreement lies in the way these features are modeled.\nOur study of alternative encodings is not a mere duplication of previous ef-\nforts (Poesio et al. 2004) that focus on linguistic aspects of parameterization. Because we\nare interested in an automatically constructed model, we have to take into account com-\nputational and learning issues when considering alternative representations. Therefore,\nour exploration of the parameter space is guided by three considerations: the linguistic\nimportance of a parameter, the accuracy of its automatic computation, and the size of the\nresulting feature space. From the linguistic side, we focus on properties of entity distri-\nbution that are tightly linked to local coherence, and at the same time allow for multiple\ninterpretations during the encoding process. Computational considerations prevent us\nfrom considering discourse representations that cannot be computed reliably by exist-\ning tools. For instance, we could not experiment with the granularity of an utterance\u2014\nsentence versus clause\u2014because available clause separators introduce substantial noise\ninto a grid construction. Finally, we exclude representations that will explode the size of\nthe feature space, thereby increasing the amount of data required for training the model.\nEntity Ex traction. The accurate computation of entity classes is key to computing mean-\ningful entity grids. In previous implementations of entity-based models, classes of coref-\nerent nouns have been extracted manually (Miltsakaki and Kukich 2000; Karamanis\net al. 2004; Poesio et al. 2004), but this is not an option for our model. An obvious\nsolution for identifying entity classes is to employ an automatic coreference resolution\ntool that determines which noun phrases refer to the same entity in a document.\nCurrent approaches recast coreference resolution as a classi\ufb01cation task. A pair\nof NPs is classi\ufb01ed as coreferring or not based on constraints that are learned from\nan annotated corpus. A separate clustering mechanism then coordinates the possibly\ncontradictory pairwise classi\ufb01cations and constructs a partition on the set of NPs. In\nour experiments, we employ Ng and Cardie\u2019s (2002) coreference resolution system.\nThe system decides whether two NPs are coreferent by exploiting a wealth of lexical,\ngrammatical, semantic, and positional features. It is trained on the MUC (6\u20137) data sets\nand yields state-of-the-art performance (70.4 F-measure on MUC-6 and 63.4 on MUC-7).\nTable 3\nExample of a feature-vector document representation using all transitions of length two given\nsyntactic categories S,O,X,a n d \u2013.\nSS SO SX S \u2013OS OO OX O \u2013XS XO XX X \u2013\u2013 S\u2013O\u2013X\u2013\u2013\nd1.01 .01 0 .08 .01 0 0 .09 0 0 0 .03 .05 .07 .03 .59\nd2.02 .01 .01 .02 0 .07 0 .02 .14 .14 .06 .04 .03 .07 0.1 .36\nd3.02 0 0 .03 .09 0 .09 .06 0 0 0 .05 .03 .07 .17 .39\n8\nFigure 24.10 A feature vector for representing documents using all transitions of length 2.\nDocument d1is the text in Fig. 24.9. Figure from Barzilay and Lapata (2008).\nThe transitions and their probabilities can then be used as features for a machine\nlearning model. This model can be a text classi\ufb01er trained to produce human-labeled\ncoherence scores (for example from humans labeling each text as coherent or inco-\nherent). But such data is expensive to gather. Barzilay and Lapata (2005) introduced\na simplifying innovation: coherence models can be trained by self-supervision :\ntrained to distinguish the natural original order of sentences in a discourse from 14 CHAPTER 24 \u2022 D ISCOURSE COHERENCE\na modi\ufb01ed order (such as a randomized order). We turn to these evaluations in the\nnext section.\n24.3.3 Evaluating Neural and Entity-based coherence\nEntity-based coherence models, as well as the neural models we introduce in the\nnext section, are generally evaluated in one of two ways.\nFirst, we can have humans rate the coherence of a document and train a classi\ufb01er\nto predict these human ratings, which can be categorial (high/low, or high/mid/low)\nor continuous. This is the best evaluation to use if we have some end task in mind,\nlike essay grading, where human raters are the correct de\ufb01nition of the \ufb01nal label.\nAlternatively, since it\u2019s very expensive to get human labels, and we might not\nyet have an end-task in mind, we can use natural texts to do self-supervision. In\nself-supervision we pair up a natural discourse with a pseudo-document created by\nchanging the ordering. Since naturally-ordered discourses are more coherent than\nrandom permutation (Lin et al., 2011), a successful coherence algorithm should pre-\nfer the original ordering.\nSelf-supervision has been implemented in 3 ways. In the sentence order dis-\ncrimination task (Barzilay and Lapata, 2005), we compare a document to a random\npermutation of its sentences. A model is considered correct for an (original, per-\nmuted) test pair if it ranks the original document higher. Given kdocuments, we can\ncompute npermutations, resulting in knpairs each with one original document and\none permutation, to use in training and testing.\nIn the sentence insertion task (Chen et al., 2007) we take a document, remove\none of the nsentences s, and create n\u00001 copies of the document with sinserted into\neach position. The task is to decide which of the ndocuments is the one with the\noriginal ordering, distinguishing the original position for sfrom all other positions.\nInsertion is harder than discrimination since we are comparing documents that differ\nby only one sentence.\nFinally, in the sentence order reconstruction task (Lapata, 2003), we take a\ndocument, randomize the sentences, and train the model to put them back in the\ncorrect order. Again given kdocuments, we can compute npermutations, resulting\ninknpairs each with one original document and one permutation, to use in training\nand testing. Reordering is of course a much harder task than simple classi\ufb01cation.\n24.4 Representation learning models for local coherence\nThe third kind of local coherence is topical or semantic \ufb01eld coherence. Discourses\ncohere by talking about the same topics and subtopics, and drawing on the same\nsemantic \ufb01elds in doing so.\nThe \ufb01eld was pioneered by a series of unsupervised models in the 1990s of this\nkind of coherence that made use of lexical cohesion (Halliday and Hasan, 1976): lexical cohesion\nthe sharing of identical or semantically related words in nearby sentences. Morris\nand Hirst (1991) computed lexical chains of words (like pine,bush trees ,trunk ) that\noccurred through a discourse and that were related in Roget\u2019s Thesaurus (by being in\nthe same category, or linked categories). They showed that the number and density\nof chain correlated with the topic structure. The TextTiling algorithm of Hearst TextTiling\n(1997) computed the cosine between neighboring text spans (the normalized dot\nproduct of vectors of raw word counts), again showing that sentences or paragraph in 24.4 \u2022 R EPRESENTATION LEARNING MODELS FOR LOCAL COHERENCE 15\na subtopic have high cosine with each other, but not with sentences in a neighboring\nsubtopic.\nA third early model, the LSA Coherence method of Foltz et al. (1998) was the\n\ufb01rst to use embeddings, modeling the coherence between two sentences as the co-\nsine between their LSA sentence embedding vectors1, computing embeddings for a\nsentence sby summing the embeddings of its words w:\nsim(s;t) = cos(s;t)\n=cos(X\nw2sw;X\nw2tw) (24.31)\nand de\ufb01ning the overall coherence of a text as the average similarity over all pairs of\nadjacent sentences siandsi+1:\ncoherence (T) =1\nn\u00001n\u00001X\ni=1cos(si;si+1) (24.32)\nModern neural representation-learning coherence models, beginning with Li et al.\n(2014), draw on the intuitions of these early unsupervised models for learning sen-\ntence representations and measuring how they change between neighboring sen-\ntences. But the new models also draw on the idea pioneered by Barzilay and Lapata\n(2005) of self-supervision. That is, unlike say coherence relation models, which\ntrain on hand-labeled representations for RST or PDTB, these models are trained to\ndistinguish natural discourses from unnatural discourses formed by scrambling the\norder of sentences, thus using representation learning to discover the features that\nmatter for at least the ordering aspect of coherence.\nHere we present one such model, the local coherence discriminator (LCD) (Xu\net al., 2019). Like early models, LCD computes the coherence of a text as the av-\nerage of coherence scores between consecutive pairs of sentences. But unlike the\nearly unsupervised models, LCD is a self-supervised model trained to discriminate\nconsecutive sentence pairs (si;si+1)in the training documents (assumed to be coher-\nent) from (constructed) incoherent pairs (si;s0). All consecutive pairs are positive\nexamples, and the negative (incoherent) partner for a sentence siis another sentence\nuniformly sampled from the same document as si.\nFig. 24.11 describes the architecture of the model fq, which takes a sentence\npair and returns a score, higher scores for more coherent pairs. Given an input\nsentence pair sandt, the model computes sentence embeddings sandt(using any\nsentence embeddings algorithm), and then concatenates four features of the pair: (1)\nthe concatenation of the two vectors (2) their difference s\u0000t; (3) the absolute value\nof their difference js\u0000tj; (4) their element-wise product s\ft. These are passed\nthrough a one-layer feedforward network to output the coherence score.\nThe model is trained to make this coherence score higher for real pairs than for\nnegative pairs. More formally, the training objective for a corpus Cof documents d,\neach of which consists of a list of sentences si, is:\nLq=X\nd2CX\nsi2dE\np(s0jsi)[L(fq(si;si+1);fq(si;s0))] (24.33)\nEp(s0jsi)is the expectation with respect to the negative sampling distribution con-\nditioned on si: given a sentence sithe algorithms samples a negative sentence s0\n1See Chapter 6 for more on LSA embeddings; they are computed by applying SVD to the term-\ndocument matrix (each cell weighted by log frequency and normalized by entropy), and then the \ufb01rst\n300 dimensions are used as the embedding. 16 CHAPTER 24 \u2022 D ISCOURSE COHERENCE\n681Loss function: The role of the loss function is\nto encourage f+=f\u2713(si,si+1)to be high while\nf\u0000=f\u2713(si,s0)to be low. Common losses such as\nmargin or log loss can all be used. Through exper-\nimental validation, we found that margin loss to\nbe superior for this problem. Speci\ufb01cally, Ltakes\non the form: L(f+,f\u0000) = max(0 ,\u2318\u0000f++f\u0000)\nwhere \u2318is the margin hyperparameter.\nNegative samples: Technically, we are free to\nchoose any sentence s0to form a negative pair\nwith si. However, because of potential differ-\nences in genre, topic and writing style, such neg-\natives might cause the discriminative model to\nlearn cues unrelated to coherence. Therefore, we\nonly select sentences from the same document to\nconstruct negative pairs. Speci\ufb01cally, suppose si\ncomes from document dkwith length nk, then\np(s0|si)is a uniform distribution over the nk\u00001\nsentences {sj}j6=ifrom dk. For a document with\nnsentences, there are n\u00001positive pairs, and\n(n\u00001)\u21e4(n\u00002)/2negative pairs. It turns out that\nthe quadratic number of negatives provides a rich\nenough learning signal, while at the same time, is\nnot too prohibitively large to be effectively cov-\nered by a sampling procedure. In practice, we\nsample a new set of negatives each time we see\na document, hence after many epochs, we can ef-\nfectively cover the space for even very long doc-\numents. Section 5.7discusses further details on\nsampling.\n4.1 Model Architecture\nThe speci\ufb01c neural architecture that we use for f\u2713\nis illustrated in Figure 1. We assume the use of\nsome pre-trained sentence encoder, which is dis-\ncussed in the next section.\nGiven an input sentence pair, the sentence en-\ncoder maps the sentences to real-valued vectors S\nandT. We then compute the concatenation of the\nfollowing features: (1)concatenation of the two\nvectors (S, T);(2)element-wise difference S\u0000T;\n(3)element-wise product S\u21e4T;(4)absolute value\nof element-wise difference |S\u0000T|. The concate-\nnated feature representation is then fed to a one-\nlayer MLP to output the coherence score.\nIn practice, we make our overall coherence\nmodel bidirectional, by training a forward model\nwith input (S, T)and a backward model with in-\nput(T,S)with the same architecture but separate\nparameters. The coherence score is then the aver-\nage from the two models.\nFigure 1: Generic architecture for our proposed model.\n4.2 Pre-trained Generative Model as the\nSentence Encoder\nOur model can work with any pre-trained sen-\ntence encoder, ranging from the most simplistic\naverage GloVe ( Pennington et al. ,2014 ) embed-\ndings to more sophisticated supervised or unsu-\npervised pre-trained sentence encoders ( Conneau\net al. ,2017 ). As mentioned in the introduction,\nsince generative models can often be turned into\nsentence encoder, generative coherence model can\nbe leveraged by our model to bene\ufb01t from the\nadvantages of both generative and discriminative\ntraining, similar to ( Kiros et al. ,2015 ;Peters et al. ,\n2018 ). After initialization, we freeze the genera-\ntive model parameters to avoid over\ufb01tting.\nIn Section 5, we will experimentally show that\nwhile we do bene\ufb01t from strong pre-trained en-\ncoders, the fact that our local discriminative model\nimproves over previous methods is independent of\nthe choice of sentence encoder.\n5 Experiments\n5.1 Evaluation Tasks\nFollowing Nguyen and Joty (2017 ) and other pre-\nvious work, we evaluate our models on the dis-\ncrimination and insertion tasks. Additionally, we\nevaluate on the paragraph reconstruction task in\nopen-domain settings, in a similar manner to Li\nand Jurafsky (2017 ).\nIn the discrimination task, a document is com-\npared to a random permutation of its sentences,\nand the model is considered correct if it scores the\noriginal document higher than the permuted one.\nTwenty permutations are used in the test set in ac-\ncordance with previous work.\nFigure 24.11 The architecture of the LCD model of document coherence, showing the\ncomputation of the score for a pair of sentences sandt. Figure from Xu et al. (2019).\nuniformly over the other sentences in the same document. Lis a loss function that\ntakes two scores, one for a positive pair and one for a negative pair, with the goal of\nencouraging f+=fq(si;si+1)to be high and f\u0000=fq(si;s0))to be low. Fig. 24.11\nuse the margin loss l(f+;f\u0000) =max(0;h\u0000f++f\u0000)where his the margin hyper-\nparameter.\nXu et al. (2019) also give a useful baseline algorithm that itself has quite high\nperformance in measuring perplexity: train an RNN language model on the data,\nand compute the log likelihood of sentence siin two ways, once given the preceding\ncontext (conditional log likelihood) and once with no context (marginal log likeli-\nhood). The difference between these values tells us how much the preceding context\nimproved the predictability of si, a predictability measure of coherence.\nTraining models to predict longer contexts than just consecutive pairs of sen-\ntences can result in even stronger discourse representations. For example a Trans-\nformer language model trained with a contrastive sentence objective to predict text\nup to a distance of \u00062 sentences improves performance on various discourse coher-\nence tasks (Iter et al., 2020).\nLanguage-model style models are generally evaluated by the methods of Sec-\ntion 24.3.3, although they can also be evaluated on the RST and PDTB coherence\nrelation tasks.\n24.5 Global Coherence\nA discourse must also cohere globally rather than just at the level of pairs of sen-\ntences. Consider stories, for example. The narrative structure of stories is one of\nthe oldest kinds of global coherence to be studied. In his in\ufb02uential Morphology of\nthe Folktale , Propp (1968) models the discourse structure of Russian folktales via\na kind of plot grammar. His model includes a set of character categories he called\ndramatis personae , like Hero, Villain, Donor, or Helper, and a set of events he\ncalled functions (like \u201cVillain commits kidnapping\u201d, \u201cDonor tests Hero\u201d, or \u201cHero\nis pursued\u201d) that have to occur in particular order, along with other components.\nPropp shows that the plots of each of the fairy tales he studies can be represented as 24.5 \u2022 G LOBAL COHERENCE 17\na sequence of these functions, different tales choosing different subsets of functions,\nbut always in the same order. Indeed Lakoff (1972) showed that Propp\u2019s model\namounted to a discourse grammar of stories, and in recent computational work Fin-\nlayson (2016) demonstrates that some of these Proppian functions could be induced\nfrom corpora of folktale texts by detecting events that have similar actions across\nstories. Bamman et al. (2013) showed that generalizations over dramatis personae\ncould be induced from movie plot summaries on Wikipedia. Their model induced\nlatent personae from features like the actions the character takes (e.g., Villains stran-\ngle), the actions done to them (e.g., Villains are foiled and arrested) or the descriptive\nwords used of them (Villains are evil).\nIn this section we introduce two kinds of such global discourse structure that\nhave been widely studied computationally. The \ufb01rst is the structure of arguments:\nthe way people attempt to convince each other in persuasive essays by offering\nclaims and supporting premises. The second is somewhat related: the structure of\nscienti\ufb01c papers, and the way authors present their goals, results, and relationship to\nprior work in their papers.\n24.5.1 Argumentation Structure\nThe \ufb01rst type of global discourse structure is the structure of arguments . Analyzing\npeople\u2019s argumentation computationally is often called argumentation mining .argumentation\nmining\nThe study of arguments dates back to Aristotle, who in his Rhetorics described\nthree components of a good argument: pathos (appealing to the emotions of the pathos\nlistener), ethos (appealing to the speaker\u2019s personal character), and logos (the logical ethos\nlogos structure of the argument).\nMost of the discourse structure studies of argumentation have focused on logos ,\nparticularly via building and training on annotated datasets of persuasive essays or\nother arguments (Reed et al. 2008, Stab and Gurevych 2014a, Peldszus and Stede\n2016, Habernal and Gurevych 2017, Musi et al. 2018). Such corpora, for exam-\nple, often include annotations of argumentative components like claims (the central claims\ncomponent of the argument that is controversial and needs support) and premises premises\n(the reasons given by the author to persuade the reader by supporting or attacking\nthe claim or other premises), as well as the argumentative relations between themargumentative\nrelations\nlike SUPPORT and ATTACK .\nConsider the following example of a persuasive essay from Stab and Gurevych\n(2014b). The \ufb01rst sentence (1) presents a claim (in bold). (2) and (3) present two\npremises supporting the claim. (4) gives a premise supporting premise (3).\n\u201c(1) Museums and art galleries provide a better understanding\nabout arts than Internet. (2) In most museums and art galleries, de-\ntailed descriptions in terms of the background, history and author are\nprovided. (3) Seeing an artwork online is not the same as watching it\nwith our own eyes, as (4) the picture online does not show the texture\nor three-dimensional structure of the art, which is important to study.\u201d\nThus this example has three argumentative relations: SUPPORT (2,1), SUPPORT (3,1)\nand SUPPORT (4,3). Fig. 24.12 shows the structure of a much more complex argu-\nment.\nWhile argumentation mining is clearly related to rhetorical structure and other\nkinds of coherence relations, arguments tend to be much less local; often a persua-\nsive essay will have only a single main claim, with premises spread throughout the\ntext, without the local coherence we see in coherence relations. 18 CHAPTER 24 \u2022 D ISCOURSE COHERENCE\nStab and Gurevych Parsing Argumentation Structures\ncloning. This example illustrates that knowing argumentative relations is important for\nseparating several arguments in a paragraph. The example also shows that argument\ncomponents frequently exhibit preceding text units that are not relevant to the argument\nbut helpful for recognizing the argument component type. For example, preceding dis-\ncourse connectors like \u201ctherefore\u201d, \u201cconsequently\u201d, or \u201cthus\u201d can signal a subsequent\nclaim. Discourse markers like \u201cbecause\u201d, \u201csince\u201d, or \u201cfurthermore\u201d could indicate a\npremise. Formally, these preceding tokens of an argument component starting at token\ntiare de\ufb01ned as the tokens ti\u0000m,...,ti\u00001that are not covered by another argument\ncomponent in the sentence s=t1,t2,...,tnwhere 1 \uf8ffi\uf8ffnand i\u0000m\u00001. The third body\nparagraph illustrates a contra argument and argumentative attack relations:\nAdmittedly, [ cloning could bemisused formilitary purposes] Claim 5. For example,\n[:it:::::could:::be:::::used::to::::::::::manipulate:::::::human::::::genes::in::::::order::to::::::create::::::::obedient:::::::soldiers\n::::with::::::::::::extraordinary:::::::abilities] Premise 9. However, because [::::moral::::and:::::::ethical::::::values:::are\n::::::::::::internationally::::::shared] Premise 10,[:it:::is::::very::::::::unlikely::::that:::::::cloning::::will::be::::::::misused:::for\n::::::militant:::::::::objectives] Premise 11.\nThe paragraph begins with Claim 5, which attacks the stance of the author. It is supported\nbyPremise 9in the second sentence. The third sentence includes two premises, both of\nwhich defend the stance of the author. Premise 11is an attack of Claim 5, and Premise 10\nsupports Premise 11. The last paragraph (conclusion) restates the major claim and sum-\nmarizes the main aspects of the essay:\nTo sum up, although [ permitting cloning might bear some risks like misuse for\nmilitary purposes] Claim 6, I strongly believe that [ this technology is bene\ufb01cial to\nhumanity ]MajorClaim 2. It is likely that [ thistechnologybears some important cures which\nwill significantly improve lifeconditions] Claim 7.\nThe conclusion of the essay starts with an attacking claim followed by the restatement of\nthe major claim. The last sentence includes another claim that summarizes the most im-\nportant points of the author\u2019s argumentation. Figure 2 shows the entire argumentation\nstructure of the example essay.\nFigure 2Argumentation structure of the example essay. Arrows indicate argumentative relations.Arrowheads denote argumentative support relations and circleheads attack relations. Dashedlines indicate relations that are encoded in the stance attributes of claims. \u201cP\u201d denotes premises.629\nFigure 24.12 Argumentation structure of a persuasive essay. Arrows indicate argumentation relations, ei-\nther of SUPPORT (with arrowheads) or ATTACK (with circleheads); P denotes premises. Figure from Stab and\nGurevych (2017).\nAlgorithms for detecting argumentation structure often include classi\ufb01ers for\ndistinguishing claims, premises, or non-argumentation, together with relation clas-\nsi\ufb01ers for deciding if two spans have the SUPPORT ,ATTACK , or neither relation\n(Peldszus and Stede, 2013). While these are the main focus of much computational\nwork, there is also preliminary efforts on annotating and detecting richer semantic\nrelationships (Park and Cardie 2014, Hidey et al. 2017) such as detecting argumen-\ntation schemes , larger-scale structures for argument like argument from example ,argumentation\nschemes\norargument from cause to effect , orargument from consequences (Feng and\nHirst, 2011).\nAnother important line of research is studying how these argument structure (or\nother features) are associated with the success or persuasiveness of an argument\n(Habernal and Gurevych 2016, Tan et al. 2016, Hidey et al. 2017. Indeed, while it\nis Aristotle\u2019s logos that is most related to discourse structure, Aristotle\u2019s ethos and\npathos techniques are particularly relevant in the detection of mechanisms of this\nsort of persuasion . For example scholars have investigated the linguistic realization persuasion\nof features studied by social scientists like reciprocity (people return favors), social\nproof (people follow others\u2019 choices), authority (people are in\ufb02uenced by those\nwith power), and scarcity (people value things that are scarce), all of which can\nbe brought up in a persuasive argument (Cialdini, 1984). Rosenthal and McKeown\n(2017) showed that these features could be combined with argumentation structure\nto predict who in\ufb02uences whom on social media, Althoff et al. (2014) found that\nlinguistic models of reciprocity and authority predicted success in online requests,\nwhile the semisupervised model of Yang et al. (2019) detected mentions of scarcity,\ncommitment, and social identity to predict the success of peer-to-peer lending plat-\nforms.\nSee Stede and Schneider (2018) for a comprehensive survey of argument mining.\n24.5.2 The structure of scienti\ufb01c discourse\nScienti\ufb01c papers have a very speci\ufb01c global structure: somewhere in the course of\nthe paper the authors must indicate a scienti\ufb01c goal, develop a method for a solu-\ntion, provide evidence for the solution, and compare to prior work. One popular 24.6 \u2022 S UMMARY 19\nannotation scheme for modeling these rhetorical goals is the argumentative zon-\ningmodel of Teufel et al. (1999) and Teufel et al. (2009), which is informed by theargumentative\nzoning\nidea that each scienti\ufb01c paper tries to make a knowledge claim about a new piece\nof knowledge being added to the repository of the \ufb01eld (Myers, 1992). Sentences\nin a scienti\ufb01c paper can be assigned one of 15 tags; Fig. 24.13 shows 7 (shortened)\nexamples of labeled sentences.\nCategory Description Example\nAIM Statement of speci\ufb01c research goal, or\nhypothesis of current paper\u201cThe aim of this process is to examine the role that\ntraining plays in the tagging process\u201d\nOWNMETHOD New Knowledge claim, own work:\nmethods\u201cIn order for it to be useful for our purposes, the\nfollowing extensions must be made:\u201d\nOWNRESULTS Measurable/objective outcome of own\nwork\u201cAll the curves have a generally upward trend but\nalways lie far below backoff (51% error rate)\u201d\nUSE Other work is used in own work \u201cWe use the framework for the allocation and\ntransfer of control of Whittaker....\u201d\nGAPWEAK Lack of solution in \ufb01eld, problem with\nother solutions\u201cHere, we will produce experimental evidence\nsuggesting that this simple model leads to serious\noverestimates\u201d\nSUPPORT Other work supports current work or is\nsupported by current work\u201cWork similar to that described here has been car-\nried out by Merialdo (1994), with broadly similar\nconclusions.\u201d\nANTISUPPORT Clash with other\u2019s results or theory; su-\nperiority of own work\u201cThis result challenges the claims of...\u201d\nFigure 24.13 Examples for 7 of the 15 labels from the Argumentative Zoning labelset (Teufel et al., 2009).\nTeufel et al. (1999) and Teufel et al. (2009) develop labeled corpora of scienti\ufb01c\narticles from computational linguistics and chemistry, which can be used as supervi-\nsion for training standard sentence-classi\ufb01cation architecture to assign the 15 labels.\n24.6 Summary\nIn this chapter we introduced local and global models for discourse coherence .\n\u2022 Discourses are not arbitrary collections of sentences; they must be coherent .\nAmong the factors that make a discourse coherent are coherence relations\nbetween the sentences, entity-based coherence, and topical coherence.\n\u2022 Various sets of coherence relations andrhetorical relations have been pro-\nposed. The relations in Rhetorical Structure Theory ( RST ) hold between\nspans of text and are structured into a tree. Because of this, shift-reduce\nand other parsing algorithms are generally used to assign these structures.\nThe Penn Discourse Treebank ( PDTB ) labels only relations between pairs of\nspans, and the labels are generally assigned by sequence models.\n\u2022Entity-based coherence captures the intuition that discourses are about an\nentity, and continue mentioning the entity from sentence to sentence. Cen-\ntering Theory is a family of models describing how salience is modeled for\ndiscourse entities, and hence how coherence is achieved by virtue of keeping\nthe same discourse entities salient over the discourse. The entity grid model\ngives a more bottom-up way to compute which entity realization transitions\nlead to coherence. 20 CHAPTER 24 \u2022 D ISCOURSE COHERENCE\n\u2022 Many different genres have different types of global coherence . Persuasive\nessays have claims and premises that are extracted in the \ufb01eld of argument\nmining , scienti\ufb01c articles have structure related to aims, methods, results, and\ncomparisons.\nBibliographical and Historical Notes\nCoherence relations arose from the independent development of a number of schol-\nars, including Hobbs (1979) idea that coherence relations play an inferential role for\nthe hearer, and the investigations by Mann and Thompson (1987) of the discourse\nstructure of large texts. Other approaches to coherence relations and their extrac-\ntion include Segmented Discourse Representation Theory ( SDRT ) (Asher and Las- SDRT\ncarides 2003, Baldridge et al. 2007) and the Linguistic Discourse Model (Polanyi\n1988, Scha and Polanyi 1988, Polanyi et al. 2004). Wolf and Gibson (2005) argue\nthat coherence structure includes crossed bracketings, which make it impossible to\nrepresent as a tree, and propose a graph representation instead. A compendium of\nover 350 relations that have been proposed in the literature can be found in Hovy\n(1990).\nRST parsing was \ufb01rst proposed by Marcu (1997), and early work was rule-based,\nfocused on discourse markers (Marcu, 2000a). The creation of the RST Discourse\nTreeBank (Carlson et al. 2001, Carlson and Marcu 2001) enabled a wide variety\nof machine learning algorithms, beginning with the shift-reduce parser of Marcu\n(1999) that used decision trees to choose actions, and continuing with a wide variety\nof machine learned parsing methods (Soricut and Marcu 2003, Sagae 2009, Hernault\net al. 2010, Feng and Hirst 2014, Surdeanu et al. 2015, Joty et al. 2015) and chunkers\n(Sporleder and Lapata, 2005). Subba and Di Eugenio (2009) integrated sophisticated\nsemantic information into RST parsing. Ji and Eisenstein (2014) \ufb01rst applied neural\nmodels to RST parsing neural models, leading to the modern set of neural RST\nmodels (Li et al. 2014, Li et al. 2016, Braud et al. 2017, Yu et al. 2018, inter alia) as\nwell as neural segmenters (Wang et al. 2018). and neural PDTB parsing models (Ji\nand Eisenstein 2015, Qin et al. 2016, Qin et al. 2017).\nBarzilay and Lapata (2005) pioneered the idea of self-supervision for coher-\nence: training a coherence model to distinguish true orderings of sentences from\nrandom permutations. Li et al. (2014) \ufb01rst applied this paradigm to neural sentence-\nrepresentation, and many neural self-supervised models followed (Li and Jurafsky\n2017, Logeswaran et al. 2018, Lai and Tetreault 2018, Xu et al. 2019, Iter et al.\n2020)\nAnother aspect of global coherence is the global topic structure of a text, the way\nthe topics shift over the course of the document. Barzilay and Lee (2004) introduced\nan HMM model for capturing topics for coherence, and later work expanded this\nintuition (Soricut and Marcu 2006, Elsner et al. 2007, Louis and Nenkova 2012, Li\nand Jurafsky 2017).\nThe relationship between explicit and implicit discourse connectives has been\na fruitful one for research. Marcu and Echihabi (2002) \ufb01rst proposed to use sen-\ntences with explicit relations to help provide training data for implicit relations, by\nremoving the explicit relations and trying to re-predict them as a way of improv-\ning performance on implicit connectives; this idea was re\ufb01ned by Sporleder and\nLascarides (2005), (Pitler et al., 2009), and Rutherford and Xue (2015). This rela- BIBLIOGRAPHICAL AND HISTORICAL NOTES 21\ntionship can also be used as a way to create discourse-aware representations. The\nDisSent algorithm (Nie et al., 2019) creates the task of predicting explicit discourse\nmarkers between two sentences. They show that representations learned to be good\nat this task also function as powerful sentence representations for other discourse\ntasks.\nThe idea of entity-based coherence seems to have arisen in multiple \ufb01elds in the\nmid-1970s, in functional linguistics (Chafe, 1976), in the psychology of discourse\nprocessing (Kintsch and Van Dijk, 1978), and in the roughly contemporaneous work\nof Grosz, Sidner, Joshi, and their colleagues. Grosz (1977) addressed the focus of\nattention that conversational participants maintain as the discourse unfolds. She de-\n\ufb01ned two levels of focus; entities relevant to the entire discourse were said to be in\nglobal focus, whereas entities that are locally in focus (i.e., most central to a partic-\nular utterance) were said to be in immediate focus. Sidner 1979; 1983 described a\nmethod for tracking (immediate) discourse foci and their use in resolving pronouns\nand demonstrative noun phrases. She made a distinction between the current dis-\ncourse focus and potential foci, which are the predecessors to the backward- and\nforward-looking centers of Centering theory, respectively. The name and further\nroots of the centering approach lie in papers by Joshi and Kuhn (1979) and Joshi\nand Weinstein (1981), who addressed the relationship between immediate focus and\nthe inferences required to integrate the current utterance into the discourse model.\nGrosz et al. (1983) integrated this work with the prior work of Sidner and Grosz.\nThis led to a manuscript on centering which, while widely circulated since 1986,\nremained unpublished until Grosz et al. (1995). A collection of centering papers ap-\npears in Walker et al. (1998). See Karamanis et al. (2004) and Poesio et al. (2004) for\na deeper exploration of centering and its parameterizations, and the History section\nof Chapter 23 for more on the use of centering on coreference.\nThe grid model of entity-based coherence was \ufb01rst proposed by Barzilay and\nLapata (2005) drawing on earlier work by Lapata (2003) and Barzilay, and then\nextended by them Barzilay and Lapata (2008) and others with additional features\n(Elsner and Charniak 2008, 2011, Feng et al. 2014, Lin et al. 2011) a model that\nprojects entities into a global graph for the discourse (Guinaudeau and Strube 2013,\nMesgar and Strube 2016), and a convolutional model to capture longer-range entity\ndependencies (Nguyen and Joty, 2017).\nTheories of discourse coherence have also been used in algorithms for interpret-\ning discourse-level linguistic phenomena, including verb phrase ellipsis and gap-\nping (Asher 1993, Kehler 1993), and tense interpretation (Lascarides and Asher\n1993, Kehler 1994, Kehler 2000). An extensive investigation into the relationship\nbetween coherence relations and discourse connectives can be found in Knott and\nDale (1994).\nUseful surveys of discourse processing and structure include Stede (2011) and\nWebber et al. (2012).\nAndy Kehler wrote the Discourse chapter for the 2000 \ufb01rst edition of this text-\nbook, which we used as the starting point for the second-edition chapter, and there\nare some remnants of Andy\u2019s lovely prose still in this third-edition coherence chap-\nter. 22 CHAPTER 24 \u2022 D ISCOURSE COHERENCE\nExercises\n24.1 Finish the Centering Theory processing of the last two utterances of (24.30),\nand show how (24.29) would be processed. Does the algorithm indeed mark\n(24.29) as less coherent?\n24.2 Select an editorial column from your favorite newspaper, and determine the\ndiscourse structure for a 10\u201320 sentence portion. What problems did you\nencounter? Were you helped by super\ufb01cial cues the speaker included (e.g.,\ndiscourse connectives) in any places? Exercises 23\nAlthoff, T., C. Danescu-Niculescu-Mizil, and D. Jurafsky.\n2014. How to ask for a favor: A case study on the suc-\ncess of altruistic requests. ICWSM 2014 .\nAsher, N. 1993. Reference to Abstract Objects in Dis-\ncourse . Studies in Linguistics and Philosophy (SLAP)\n50, Kluwer.\nAsher, N. and A. Lascarides. 2003. Logics of Conversation .\nCambridge University Press.\nBaldridge, J., N. Asher, and J. Hunter. 2007. Annotation for\nand robust parsing of discourse structure on unrestricted\ntexts. Zeitschrift f \u00a8ur Sprachwissenschaft , 26:213\u2013239.\nBamman, D., B. O\u2019Connor, and N. A. Smith. 2013. Learning\nlatent personas of \ufb01lm characters. ACL.\nBarzilay, R. and M. Lapata. 2005. Modeling local coherence:\nAn entity-based approach. ACL.\nBarzilay, R. and M. Lapata. 2008. Modeling local coher-\nence: An entity-based approach. Computational Linguis-\ntics, 34(1):1\u201334.\nBarzilay, R. and L. Lee. 2004. Catching the drift: Prob-\nabilistic content models, with applications to generation\nand summarization. HLT-NAACL .\nBedi, G., F. Carrillo, G. A. Cecchi, D. F. Slezak, M. Sig-\nman, N. B. Mota, S. Ribeiro, D. C. Javitt, M. Copelli,\nand C. M. Corcoran. 2015. Automated analysis of free\nspeech predicts psychosis onset in high-risk youths. npj\nSchizophrenia , 1.\nBiran, O. and K. McKeown. 2015. PDTB discourse parsing\nas a tagging task: The two taggers approach. SIGDIAL .\nBraud, C., M. Coavoux, and A. S\u00f8gaard. 2017. Cross-lingual\nRST discourse parsing. EACL .\nBrennan, S. E., M. W. Friedman, and C. Pollard. 1987. A\ncentering approach to pronouns. ACL.\nCarlson, L. and D. Marcu. 2001. Discourse tagging manual.\nTechnical Report ISI-TR-545, ISI.\nCarlson, L., D. Marcu, and M. E. Okurowski. 2001. Building\na discourse-tagged corpus in the framework of rhetorical\nstructure theory. SIGDIAL .\nChafe, W. L. 1976. Givenness, contrastiveness, de\ufb01niteness,\nsubjects, topics, and point of view. In C. N. Li, ed., Sub-\nject and Topic , 25\u201355. Academic Press.\nChen, E., B. Snyder, and R. Barzilay. 2007. Incre-\nmental text structuring with online hierarchical ranking.\nEMNLP/CoNLL .\nCialdini, R. B. 1984. In\ufb02uence: The psychology of persua-\nsion. Morrow.\nDitman, T. and G. R. Kuperberg. 2010. Building coherence:\nA framework for exploring the breakdown of links across\nclause boundaries in schizophrenia. Journal of neurolin-\nguistics , 23(3):254\u2013269.\nElsner, M., J. Austerweil, and E. Charniak. 2007. A uni\ufb01ed\nlocal and global model for discourse coherence. NAACL-\nHLT.\nElsner, M. and E. Charniak. 2008. Coreference-inspired co-\nherence modeling. ACL.\nElsner, M. and E. Charniak. 2011. Extending the entity grid\nwith entity-speci\ufb01c features. ACL.Elvev \u02daag, B., P. W. Foltz, D. R. Weinberger, and T. E. Gold-\nberg. 2007. Quantifying incoherence in speech: an auto-\nmated methodology and novel application to schizophre-\nnia.Schizophrenia research , 93(1-3):304\u2013316.\nFeng, V . W. and G. Hirst. 2011. Classifying arguments by\nscheme. ACL.\nFeng, V . W. and G. Hirst. 2014. A linear-time bottom-up\ndiscourse parser with constraints and post-editing. ACL.\nFeng, V . W., Z. Lin, and G. Hirst. 2014. The impact of deep\nhierarchical discourse structures in the evaluation of text\ncoherence. COLING .\nFinlayson, M. A. 2016. Inferring Propp\u2019s functions from se-\nmantically annotated text. The Journal of American Folk-\nlore, 129(511):55\u201377.\nFoltz, P. W., W. Kintsch, and T. K. Landauer. 1998. The\nmeasurement of textual coherence with latent semantic\nanalysis. Discourse processes , 25(2-3):285\u2013307.\nGrosz, B. J. 1977. The representation and use of focus in\na system for understanding dialogs. IJCAI-77 . Morgan\nKaufmann.\nGrosz, B. J., A. K. Joshi, and S. Weinstein. 1983. Provid-\ning a uni\ufb01ed account of de\ufb01nite noun phrases in English.\nACL.\nGrosz, B. J., A. K. Joshi, and S. Weinstein. 1995. Center-\ning: A framework for modeling the local coherence of\ndiscourse. Computational Linguistics , 21(2):203\u2013225.\nGuinaudeau, C. and M. Strube. 2013. Graph-based local co-\nherence modeling. ACL.\nHabernal, I. and I. Gurevych. 2016. Which argument is more\nconvincing? Analyzing and predicting convincingness of\nWeb arguments using bidirectional LSTM. ACL.\nHabernal, I. and I. Gurevych. 2017. Argumentation mining\nin user-generated web discourse. Computational Linguis-\ntics, 43(1):125\u2013179.\nHalliday, M. A. K. and R. Hasan. 1976. Cohesion in English .\nLongman. English Language Series, Title No. 9.\nHearst, M. A. 1997. Texttiling: Segmenting text into multi-\nparagraph subtopic passages. Computational Linguistics ,\n23:33\u201364.\nHernault, H., H. Prendinger, D. A. duVerle, and M. Ishizuka.\n2010. Hilda: A discourse parser using support vector ma-\nchine classi\ufb01cation. Dialogue & Discourse , 1(3).\nHidey, C., E. Musi, A. Hwang, S. Muresan, and K. McKe-\nown. 2017. Analyzing the semantic types of claims and\npremises in an online persuasive forum. 4th Workshop on\nArgument Mining .\nHobbs, J. R. 1979. Coherence and coreference. Cognitive\nScience , 3:67\u201390.\nHovy, E. H. 1990. Parsimonious and pro\ufb02igate approaches to\nthe question of discourse structure relations. Proceedings\nof the 5th International Workshop on Natural Language\nGeneration .\nIter, D., K. Guu, L. Lansing, and D. Jurafsky. 2020. Pretrain-\ning with contrastive sentence objectives improves dis-\ncourse performance of language models. ACL.\nIter, D., J. Yoon, and D. Jurafsky. 2018. Automatic detec-\ntion of incoherent speech for diagnosing schizophrenia.\nFifth Workshop on Computational Linguistics and Clini-\ncal Psychology . 24 Chapter 24 \u2022 Discourse Coherence\nJi, Y . and J. Eisenstein. 2014. Representation learning for\ntext-level discourse parsing. ACL.\nJi, Y . and J. Eisenstein. 2015. One vector is not enough:\nEntity-augmented distributed semantics for discourse re-\nlations. TACL , 3:329\u2013344.\nJoshi, A. K. and S. Kuhn. 1979. Centered logic: The role\nof entity centered sentence representation in natural lan-\nguage inferencing. IJCAI-79 .\nJoshi, A. K. and S. Weinstein. 1981. Control of inference:\nRole of some aspects of discourse structure \u2013 centering.\nIJCAI-81 .\nJoty, S., G. Carenini, and R. T. Ng. 2015. CODRA: A novel\ndiscriminative framework for rhetorical analysis. Compu-\ntational Linguistics , 41(3):385\u2013435.\nKaramanis, N., M. Poesio, C. Mellish, and J. Oberlander.\n2004. Evaluating centering-based metrics of coherence\nfor text structuring using a reliably annotated corpus.\nACL.\nKehler, A. 1993. The effect of establishing coherence in el-\nlipsis and anaphora resolution. ACL.\nKehler, A. 1994. Temporal relations: Reference or discourse\ncoherence? ACL.\nKehler, A. 2000. Coherence, Reference, and the Theory of\nGrammar . CSLI Publications.\nKintsch, W. and T. A. Van Dijk. 1978. Toward a model of\ntext comprehension and production. Psychological re-\nview, 85(5):363\u2013394.\nKnott, A. and R. Dale. 1994. Using linguistic phenomena\nto motivate a set of coherence relations. Discourse Pro-\ncesses , 18(1):35\u201362.\nLai, A. and J. Tetreault. 2018. Discourse coherence in the\nwild: A dataset, evaluation and methods. SIGDIAL .\nLakoff, G. 1972. Structural complexity in fairy tales. In The\nStudy of Man , 128\u201350. School of Social Sciences, Uni-\nversity of California, Irvine, CA.\nLapata, M. 2003. Probabilistic text structuring: Experiments\nwith sentence ordering. ACL.\nLascarides, A. and N. Asher. 1993. Temporal interpretation,\ndiscourse relations, and common sense entailment. Lin-\nguistics and Philosophy , 16(5):437\u2013493.\nLi, J. and D. Jurafsky. 2017. Neural net models of open-\ndomain discourse coherence. EMNLP .\nLi, J., R. Li, and E. H. Hovy. 2014. Recursive deep models\nfor discourse parsing. EMNLP .\nLi, Q., T. Li, and B. Chang. 2016. Discourse parsing with\nattention-based hierarchical neural networks. EMNLP .\nLin, Z., M.-Y . Kan, and H. T. Ng. 2009. Recognizing im-\nplicit discourse relations in the Penn Discourse Treebank.\nEMNLP .\nLin, Z., H. T. Ng, and M.-Y . Kan. 2011. Automatically eval-\nuating text coherence using discourse relations. ACL.\nLin, Z., H. T. Ng, and M.-Y . Kan. 2014. A pdtb-styled end-\nto-end discourse parser. Natural Language Engineering ,\n20(2):151\u2013184.\nLogeswaran, L., H. Lee, and D. Radev. 2018. Sentence\nordering and coherence modeling using recurrent neural\nnetworks. AAAI .Louis, A. and A. Nenkova. 2012. A coherence model based\non syntactic patterns. EMNLP .\nLukasik, M., B. Dadachev, K. Papineni, and G. Sim \u02dcoes.\n2020. Text segmentation by cross segment attention.\nEMNLP .\nMann, W. C. and S. A. Thompson. 1987. Rhetorical structure\ntheory: A theory of text organization. Technical Report\nRS-87-190, Information Sciences Institute.\nMarcu, D. 1997. The rhetorical parsing of natural language\ntexts. ACL.\nMarcu, D. 1999. A decision-based approach to rhetorical\nparsing. ACL.\nMarcu, D. 2000a. The rhetorical parsing of unrestricted\ntexts: A surface-based approach. Computational Linguis-\ntics, 26(3):395\u2013448.\nMarcu, D., ed. 2000b. The Theory and Practice of Discourse\nParsing and Summarization . MIT Press.\nMarcu, D. and A. Echihabi. 2002. An unsupervised approach\nto recognizing discourse relations. ACL.\nMesgar, M. and M. Strube. 2016. Lexical coherence graph\nmodeling using word embeddings. ACL.\nMiltsakaki, E., R. Prasad, A. K. Joshi, and B. L. Webber.\n2004. The Penn Discourse Treebank. LREC .\nMorey, M., P. Muller, and N. Asher. 2017. How much\nprogress have we made on RST discourse parsing? a\nreplication study of recent results on the rst-dt. EMNLP .\nMorris, J. and G. Hirst. 1991. Lexical cohesion computed by\nthesaural relations as an indicator of the structure of text.\nComputational Linguistics , 17(1):21\u201348.\nMuller, P., C. Braud, and M. Morey. 2019. ToNy: Contextual\nembeddings for accurate multilingual discourse segmen-\ntation of full documents. Workshop on Discourse Relation\nParsing and Treebanking .\nMusi, E., M. Stede, L. Kriese, S. Muresan, and A. Rocci.\n2018. A multi-layer annotated corpus of argumenta-\ntive text: From argument schemes to discourse relations.\nLREC .\nMyers, G. 1992. \u201cIn this paper we report...\u201d: Speech acts and\nscienti\ufb01c facts. Journal of Pragmatics , 17(4):295\u2013313.\nNguyen, D. T. and S. Joty. 2017. A neural local coherence\nmodel. ACL.\nNie, A., E. Bennett, and N. Goodman. 2019. DisSent: Learn-\ning sentence representations from explicit discourse rela-\ntions. ACL.\nPark, J. and C. Cardie. 2014. Identifying appropriate support\nfor propositions in online user comments. First workshop\non argumentation mining .\nPeldszus, A. and M. Stede. 2013. From argument diagrams\nto argumentation mining in texts: A survey. International\nJournal of Cognitive Informatics and Natural Intelligence\n(IJCINI) , 7(1):1\u201331.\nPeldszus, A. and M. Stede. 2016. An annotated corpus of\nargumentative microtexts. 1st European Conference on\nArgumentation .\nPitler, E., A. Louis, and A. Nenkova. 2009. Automatic sense\nprediction for implicit discourse relations in text. ACL\nIJCNLP .\nPitler, E. and A. Nenkova. 2009. Using syntax to disam-\nbiguate explicit discourse connectives in text. ACL IJC-\nNLP. Exercises 25\nPoesio, M., R. Stevenson, B. Di Eugenio, and J. Hitzeman.\n2004. Centering: A parametric theory and its instantia-\ntions. Computational Linguistics , 30(3):309\u2013363.\nPolanyi, L. 1988. A formal model of the structure of dis-\ncourse. Journal of Pragmatics , 12.\nPolanyi, L., C. Culy, M. van den Berg, G. L. Thione, and\nD. Ahn. 2004. A rule based approach to discourse pars-\ning. Proceedings of SIGDIAL .\nPrasad, R., N. Dinesh, A. Lee, E. Miltsakaki, L. Robaldo,\nA. K. Joshi, and B. L. Webber. 2008. The Penn Discourse\nTreeBank 2.0. LREC .\nPrasad, R., B. L. Webber, and A. Joshi. 2014. Re\ufb02ections on\nthe Penn Discourse Treebank, comparable corpora, and\ncomplementary annotation. Computational Linguistics ,\n40(4):921\u2013950.\nPropp, V . 1968. Morphology of the Folktale , 2nd edition.\nUniversity of Texas Press. Original Russian 1928. Trans-\nlated by Laurence Scott.\nQin, L., Z. Zhang, and H. Zhao. 2016. A stacking gated\nneural architecture for implicit discourse relation classi\ufb01-\ncation. EMNLP .\nQin, L., Z. Zhang, H. Zhao, Z. Hu, and E. Xing. 2017. Ad-\nversarial connective-exploiting networks for implicit dis-\ncourse relation classi\ufb01cation. ACL.\nReed, C., R. Mochales Palau, G. Rowe, and M.-F. Moens.\n2008. Language resources for studying argument. LREC .\nRosenthal, S. and K. McKeown. 2017. Detecting in\ufb02uencers\nin multiple online genres. ACM Transactions on Internet\nTechnology (TOIT) , 17(2).\nRutherford, A. and N. Xue. 2015. Improving the inference\nof implicit discourse relations via classifying explicit dis-\ncourse connectives. NAACL HLT .\nSagae, K. 2009. Analysis of discourse structure with syn-\ntactic dependencies and data-driven shift-reduce parsing.\nIWPT-09 .\nScha, R. and L. Polanyi. 1988. An augmented context free\ngrammar for discourse. COLING .\nSidner, C. L. 1979. Towards a computational theory of de\ufb01-\nnite anaphora comprehension in English discourse. Tech-\nnical Report 537, MIT Arti\ufb01cial Intelligence Laboratory,\nCambridge, MA.\nSidner, C. L. 1983. Focusing in the comprehension of de\ufb01-\nnite anaphora. In M. Brady and R. C. Berwick, eds, Com-\nputational Models of Discourse , 267\u2013330. MIT Press.\nSomasundaran, S., J. Burstein, and M. Chodorow. 2014.\nLexical chaining for measuring discourse coherence qual-\nity in test-taker essays. COLING .\nSoricut, R. and D. Marcu. 2003. Sentence level discourse\nparsing using syntactic and lexical information. HLT-\nNAACL .\nSoricut, R. and D. Marcu. 2006. Discourse generation using\nutility-trained coherence models. COLING/ACL .\nSporleder, C. and A. Lascarides. 2005. Exploiting linguistic\ncues to classify rhetorical relations. RANLP-05 .\nSporleder, C. and M. Lapata. 2005. Discourse chunking and\nits application to sentence compression. EMNLP .\nStab, C. and I. Gurevych. 2014a. Annotating argument com-\nponents and relations in persuasive essays. COLING .Stab, C. and I. Gurevych. 2014b. Identifying argumentative\ndiscourse structures in persuasive essays. EMNLP .\nStab, C. and I. Gurevych. 2017. Parsing argumentation struc-\ntures in persuasive essays. Computational Linguistics ,\n43(3):619\u2013659.\nStede, M. 2011. Discourse processing . Morgan & Claypool.\nStede, M. and J. Schneider. 2018. Argumentation Mining .\nMorgan & Claypool.\nSubba, R. and B. Di Eugenio. 2009. An effective discourse\nparser that uses rich linguistic information. NAACL HLT .\nSurdeanu, M., T. Hicks, and M. A. Valenzuela-Escarcega.\n2015. Two practical rhetorical structure theory parsers.\nNAACL HLT .\nTan, C., V . Niculae, C. Danescu-Niculescu-Mizil, and\nL. Lee. 2016. Winning arguments: Interaction dynam-\nics and persuasion strategies in good-faith online discus-\nsions. WWW-16 .\nTeufel, S., J. Carletta, and M. Moens. 1999. An annotation\nscheme for discourse-level argumentation in research ar-\nticles. EACL .\nTeufel, S., A. Siddharthan, and C. Batchelor. 2009. To-\nwards domain-independent argumentative zoning: Ev-\nidence from chemistry and computational linguistics.\nEMNLP .\nWalker, M. A., A. K. Joshi, and E. Prince, eds. 1998. Cen-\ntering in Discourse . Oxford University Press.\nWang, Y ., S. Li, and J. Yang. 2018. Toward fast and accurate\nneural discourse segmentation. EMNLP .\nWebber, B. L., M. Egg, and V . Kordoni. 2012. Discourse\nstructure and language technology. Natural Language\nEngineering , 18(4):437\u2013490.\nWolf, F. and E. Gibson. 2005. Representing discourse coher-\nence: A corpus-based analysis. Computational Linguis-\ntics, 31(2):249\u2013287.\nXu, P., H. Saghir, J. S. Kang, T. Long, A. J. Bose, Y . Cao,\nand J. C. K. Cheung. 2019. A cross-domain transferable\nneural coherence model. ACL.\nXue, N., H. T. Ng, S. Pradhan, A. Rutherford, B. L. Web-\nber, C. Wang, and H. Wang. 2016. CoNLL 2016 shared\ntask on multilingual shallow discourse parsing. CoNLL-\n16 shared task .\nYang, D., J. Chen, Z. Yang, D. Jurafsky, and E. H. Hovy.\n2019. Let\u2019s make your request more persuasive: Model-\ning persuasive strategies via semi-supervised neural nets\non crowdfunding platforms. NAACL HLT .\nYu, N., M. Zhang, and G. Fu. 2018. Transition-based neural\nRST parsing with implicit syntax features. COLING .\nYu, Y ., Y . Zhu, Y . Liu, Y . Liu, S. Peng, M. Gong, and\nA. Zeldes. 2019. GumDrop at the DISRPT2019 shared\ntask: A model stacking approach to discourse unit seg-\nmentation and connective detection. Workshop on Dis-\ncourse Relation Parsing and Treebanking 2019 .\nZhou, Y . and N. Xue. 2015. The Chinese Discourse Tree-\nBank: a Chinese corpus annotated with discourse rela-\ntions. Language Resources and Evaluation , 49(2):397\u2013\n431.", "A.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\nAHidden Markov Models\nChapter 17 introduced the Hidden Markov Model and applied it to part of speech\ntagging. Part of speech tagging is a fully-supervised learning task, because we have\na corpus of words labeled with the correct part-of-speech tag. But many applications\ndon\u2019t have labeled data. So in this chapter, we introduce the full set of algorithms for\nHMMs, including the key unsupervised learning algorithm for HMM, the Forward-\nBackward algorithm. We\u2019ll repeat some of the text from Chapter 17 for readers who\nwant the whole story laid out in a single chapter.\nA.1 Markov Chains\nThe HMM is based on augmenting the Markov chain. A Markov chain is a model Markov chain\nthat tells us something about the probabilities of sequences of random variables,\nstates , each of which can take on values from some set. These sets can be words, or\ntags, or symbols representing anything, like the weather. A Markov chain makes a\nvery strong assumption that if we want to predict the future in the sequence, all that\nmatters is the current state. The states before the current state have no impact on the\nfuture except via the current state. It\u2019s as if to predict tomorrow\u2019s weather you could\nexamine today\u2019s weather but you weren\u2019t allowed to look at yesterday\u2019s weather.\nWARM3HOT1COLD2.8.6.1.1.3.6.1.1.3\ncharminguniformlyare.1.4.5.5.5.2.6.2\n(a) (b)\nFigure A.1 A Markov chain for weather (a) and one for words (b), showing states and\ntransitions. A start distribution pis required; setting p= [0:1;0:7;0:2]for (a) would mean a\nprobability 0.7 of starting in state 2 (cold), probability 0.1 of starting in state 1 (hot), etc.\nMore formally, consider a sequence of state variables q1;q2;:::;qi. A Markov\nmodel embodies the Markov assumption on the probabilities of this sequence: thatMarkov\nassumption\nwhen predicting the future, the past doesn\u2019t matter, only the present.\nMarkov Assumption: P(qi=ajq1:::qi\u00001) =P(qi=ajqi\u00001) (A.1)\nFigure A.1a shows a Markov chain for assigning a probability to a sequence of\nweather events, for which the vocabulary consists of HOT,COLD , and WARM . The\nstates are represented as nodes in the graph, and the transitions, with their probabil-\nities, as edges. The transitions are probabilities: the values of arcs leaving a given 2APPENDIX A \u2022 H IDDEN MARKOV MODELS\nstate must sum to 1. Figure A.1b shows a Markov chain for assigning a probabil-\nity to a sequence of words w1:::wn. This Markov chain should be familiar; in fact,\nit represents a bigram language model, with each edge expressing the probability\np(wijwj)! Given the two models in Fig. A.1, we can assign a probability to any\nsequence from our vocabulary.\nFormally, a Markov chain is speci\ufb01ed by the following components:\nQ=q1q2:::qN a set of Nstates\nA=a11a12:::aN1:::aNN atransition probability matrix A, each ai jrepresent-\ning the probability of moving from state ito state j, s.t.Pn\nj=1ai j=18i\np=p1;p2;:::;pN aninitial probability distribution over states. piis the\nprobability that the Markov chain will start in state i.\nSome states jmay have pj=0, meaning that they cannot\nbe initial states. Also,PN\ni=1pi=1\nBefore you go on, use the sample probabilities in Fig. A.1a (with p= [:1;:7:;2])\nto compute the probability of each of the following sequences:\n(A.2) hot hot hot hot\n(A.3) cold hot cold hot\nWhat does the difference in these probabilities tell you about a real-world weather\nfact encoded in Fig. A.1a?\nA.2 The Hidden Markov Model\nA Markov chain is useful when we need to compute a probability for a sequence\nof observable events. In many cases, however, the events we are interested in are\nhidden : we don\u2019t observe them directly. For example we don\u2019t normally observe hidden\npart-of-speech tags in a text. Rather, we see words, and must infer the tags from the\nword sequence. We call the tags hidden because they are not observed.\nAhidden Markov model (HMM ) allows us to talk about both observed eventsHidden\nMarkov model\n(like words that we see in the input) and hidden events (like part-of-speech tags) that\nwe think of as causal factors in our probabilistic model. An HMM is speci\ufb01ed by\nthe following components:\nQ=q1q2:::qN a set of Nstates\nA=a11:::ai j:::aNN atransition probability matrix A, each ai jrepresenting the probability\nof moving from state ito state j, s.t.PN\nj=1ai j=18i\nB=bi(ot) a sequence of observation likelihoods , also called emission probabili-\nties, each expressing the probability of an observation ot(drawn from a\nvocabulary V=v1;v2;:::;vV) being generated from a state qi\np=p1;p2;:::;pN aninitial probability distribution over states. piis the probability that\nthe Markov chain will start in state i. Some states jmay have pj=0,\nmeaning that they cannot be initial states. Also,Pn\ni=1pi=1\nThe HMM is given as input O=o1o2:::oT: a sequence of Tobservations , each\none drawn from the vocabulary V.\nA \ufb01rst-order hidden Markov model instantiates two simplifying assumptions.\nFirst, as with a \ufb01rst-order Markov chain, the probability of a particular state depends A.2 \u2022 T HEHIDDEN MARKOV MODEL 3\nonly on the previous state:\nMarkov Assumption: P(qijq1:::qi\u00001) =P(qijqi\u00001) (A.4)\nSecond, the probability of an output observation oidepends only on the state that\nproduced the observation qiand not on any other states or any other observations:\nOutput Independence: P(oijq1:::qi;:::; qT;o1;:::; oi;:::; oT) =P(oijqi)(A.5)\nTo exemplify these models, we\u2019ll use a task invented by Jason Eisner (2002).\nImagine that you are a climatologist in the year 2799 studying the history of global\nwarming. You cannot \ufb01nd any records of the weather in Baltimore, Maryland, for\nthe summer of 2020, but you do \ufb01nd Jason Eisner\u2019s diary, which lists how many ice\ncreams Jason ate every day that summer. Our goal is to use these observations to\nestimate the temperature every day. We\u2019ll simplify this weather task by assuming\nthere are only two kinds of days: cold (C) and hot (H). So the Eisner task is as\nfollows:\nGiven a sequence of observations O(each an integer representing the\nnumber of ice creams eaten on a given day) \ufb01nd the \u2018hidden\u2019 sequence\nQof weather states (H or C) which caused Jason to eat the ice cream.\nFigure A.2 shows a sample HMM for the ice cream task. The two hidden states\n(H and C) correspond to hot and cold weather, and the observations (drawn from the\nalphabet O=f1;2;3g) correspond to the number of ice creams eaten by Jason on a\ngiven day.\n\u03c0 = [.2,.8]HOT2COLD1B2P(1 | HOT)           .2P(2 | HOT)     =    .4P(3 | HOT)           .4.6.5.4.5P(1 | COLD)         .5P(2 | COLD)   =    .4P(3 | COLD)         .1B1\nFigure A.2 A hidden Markov model for relating numbers of ice creams eaten by Jason (the\nobservations) to the weather (H or C, the hidden variables).\nAn in\ufb02uential tutorial by Rabiner (1989), based on tutorials by Jack Ferguson in\nthe 1960s, introduced the idea that hidden Markov models should be characterized\nbythree fundamental problems :\nProblem 1 (Likelihood): Given an HMM l= (A;B)and an observation se-\nquence O, determine the likelihood P(Ojl).\nProblem 2 (Decoding): Given an observation sequence Oand an HMM l=\n(A;B), discover the best hidden state sequence Q.\nProblem 3 (Learning): Given an observation sequence Oand the set of states\nin the HMM, learn the HMM parameters AandB.\nWe already saw an example of Problem 2 in Chapter 17. In the next two sections\nwe introduce the Forward and Forward-Backward algorithms to solve Problems 1\nand 3 and give more information on Problem 2 4APPENDIX A \u2022 H IDDEN MARKOV MODELS\nA.3 Likelihood Computation: The Forward Algorithm\nOur \ufb01rst problem is to compute the likelihood of a particular observation sequence.\nFor example, given the ice-cream eating HMM in Fig. A.2, what is the probability\nof the sequence 3 1 3 ? More formally:\nComputing Likelihood: Given an HMM l= (A;B)and an observa-\ntion sequence O, determine the likelihood P(Ojl).\nFor a Markov chain, where the surface observations are the same as the hidden\nevents, we could compute the probability of 3 1 3 just by following the states labeled\n3 1 3 and multiplying the probabilities along the arcs. For a hidden Markov model,\nthings are not so simple. We want to determine the probability of an ice-cream\nobservation sequence like 3 1 3 , but we don\u2019t know what the hidden state sequence\nis!\nLet\u2019s start with a slightly simpler situation. Suppose we already knew the weather\nand wanted to predict how much ice cream Jason would eat. This is a useful part\nof many HMM tasks. For a given hidden state sequence (e.g., hot hot cold ), we can\neasily compute the output likelihood of 3 1 3 .\nLet\u2019s see how. First, recall that for hidden Markov models, each hidden state\nproduces only a single observation. Thus, the sequence of hidden states and the\nsequence of observations have the same length.1\nGiven this one-to-one mapping and the Markov assumptions expressed in Eq. A.4,\nfor a particular hidden state sequence Q=q1;q2;:::;qTand an observation sequence\nO=o1;o2;:::;oT, the likelihood of the observation sequence is\nP(OjQ) =TY\ni=1P(oijqi) (A.6)\nThe computation of the forward probability for our ice-cream observation 3 1 3 from\none possible hidden state sequence hot hot cold is shown in Eq. A.7. Figure A.3\nshows a graphic representation of this computation.\nP(3 1 3jhot hot cold ) = P(3jhot)\u0002P(1jhot)\u0002P(3jcold) (A.7)\ncoldhot3.4hot13.2.1\nFigure A.3 The computation of the observation likelihood for the ice-cream events 3 1 3\ngiven the hidden state sequence hot hot cold .\nBut of course, we don\u2019t actually know what the hidden state (weather) sequence\nwas. We\u2019ll need to compute the probability of ice-cream events 3 1 3 instead by\n1In a variant of HMMs called segmental HMMs (in speech recognition) or semi-HMMs (in text pro-\ncessing) this one-to-one mapping between the length of the hidden state sequence and the length of the\nobservation sequence does not hold. A.3 \u2022 L IKELIHOOD COMPUTATION : THEFORWARD ALGORITHM 5\nsumming over all possible weather sequences, weighted by their probability. First,\nlet\u2019s compute the joint probability of being in a particular weather sequence Qand\ngenerating a particular sequence Oof ice-cream events. In general, this is\nP(O;Q) =P(OjQ)\u0002P(Q) =TY\ni=1P(oijqi)\u0002TY\ni=1P(qijqi\u00001) (A.8)\nThe computation of the joint probability of our ice-cream observation 3 1 3 and one\npossible hidden state sequence hot hot cold is shown in Eq. A.9. Figure A.4 shows\na graphic representation of this computation.\nP(3 1 3 ;hot hot cold ) = P(hotjstart)\u0002P(hotjhot)\u0002P(coldjhot)\n\u0002P(3jhot)\u0002P(1jhot)\u0002P(3jcold) (A.9)\ncoldhot3.4hot.613.4.2.1\nFigure A.4 The computation of the joint probability of the ice-cream events 3 1 3 and the\nhidden state sequence hot hot cold .\nNow that we know how to compute the joint probability of the observations\nwith a particular hidden state sequence, we can compute the total probability of the\nobservations just by summing over all possible hidden state sequences:\nP(O) =X\nQP(O;Q) =X\nQP(OjQ)P(Q) (A.10)\nFor our particular case, we would sum over the eight 3-event sequences cold cold\ncold,cold cold hot , that is,\nP(3 1 3) =P(3 1 3 ;cold cold cold )+P(3 1 3 ;cold cold hot )+P(3 1 3 ;hot hot cold )+:::\nFor an HMM with Nhidden states and an observation sequence of Tobserva-\ntions, there are NTpossible hidden sequences. For real tasks, where NandTare\nboth large, NTis a very large number, so we cannot compute the total observation\nlikelihood by computing a separate observation likelihood for each hidden state se-\nquence and then summing them.\nInstead of using such an extremely exponential algorithm, we use an ef\ufb01cient\nO(N2T)algorithm called the forward algorithm . The forward algorithm is a kindforward\nalgorithm\nofdynamic programming algorithm, that is, an algorithm that uses a table to store\nintermediate values as it builds up the probability of the observation sequence. The\nforward algorithm computes the observation probability by summing over the prob-\nabilities of all possible hidden state paths that could generate the observation se-\nquence, but it does so ef\ufb01ciently by implicitly folding each of these paths into a\nsingle forward trellis .\nFigure A.5 shows an example of the forward trellis for computing the likelihood\nof3 1 3 given the hidden state sequence hot hot cold . 6APPENDIX A \u2022 H IDDEN MARKOV MODELS\n\u03c0HCHCHCP(C|start) * P(3|C).2 * .1P(H|H) * P(1|H).6 * .2P(C|C) * P(1|C).5 * .5P(C|H) * P(1|C).4 * .5P(H|C) * P(1|H).5 * .2P(H|start)*P(3|H).8 * .4\u03b11(2)=.32\u03b11(1) = .02\u03b12(2)= .32*.12 + .02*.1 = .0404\u03b12(1) = .32*.2 + .02*.25 = .069\ntCHq2q1o13o2o313\nFigure A.5 The forward trellis for computing the total observation likelihood for the ice-cream events 3 1 3 .\nHidden states are in circles, observations in squares. The \ufb01gure shows the computation of at(j)for two states at\ntwo time steps. The computation in each cell follows Eq. A.12: at(j) =PN\ni=1at\u00001(i)ai jbj(ot). The resulting\nprobability expressed in each cell is Eq. A.11: at(j) =P(o1;o2:::ot;qt=jjl).\nEach cell of the forward algorithm trellis at(j)represents the probability of be-\ning in state jafter seeing the \ufb01rst tobservations, given the automaton l. The value\nof each cell at(j)is computed by summing over the probabilities of every path that\ncould lead us to this cell. Formally, each cell expresses the following probability:\nat(j) =P(o1;o2:::ot;qt=jjl) (A.11)\nHere, qt=jmeans \u201cthe tthstate in the sequence of states is state j\u201d. We compute\nthis probability at(j)by summing over the extensions of all the paths that lead to\nthe current cell. For a given state qjat time t, the value at(j)is computed as\nat(j) =NX\ni=1at\u00001(i)ai jbj(ot) (A.12)\nThe three factors that are multiplied in Eq. A.12 in extending the previous paths\nto compute the forward probability at time tare\nat\u00001(i) theprevious forward path probability from the previous time step\nai j thetransition probability from previous state qito current state qj\nbj(ot) thestate observation likelihood of the observation symbol otgiven\nthe current state j\nConsider the computation in Fig. A.5 of a2(2), the forward probability of being\nat time step 2 in state 2 having generated the partial observation 3 1. We compute by\nextending the aprobabilities from time step 1, via two paths, each extension con-\nsisting of the three factors above: a1(1)\u0002P(HjC)\u0002P(1jH)anda1(2)\u0002P(HjH)\u0002\nP(1jH).\nFigure A.6 shows another visualization of this induction step for computing the\nvalue in one new cell of the trellis.\nWe give two formal de\ufb01nitions of the forward algorithm: the pseudocode in\nFig. A.7 and a statement of the de\ufb01nitional recursion here. A.3 \u2022 L IKELIHOOD COMPUTATION : THEFORWARD ALGORITHM 7\not-1ota1ja2jaNja3jbj(ot)\u03b1t(j)= \u03a3i \u03b1t-1(i) aij bj(ot) \nq1q2q3qN\nq1qjq2q1q2ot+1ot-2q1q2q3q3qNqN\u03b1t-1(N)\n\u03b1t-1(3)\u03b1t-1(2)\u03b1t-1(1)\u03b1t-2(N)\n\u03b1t-2(3)\u03b1t-2(2)\u03b1t-2(1)\nFigure A.6 Visualizing the computation of a single element at(i)in the trellis by summing\nall the previous values at\u00001, weighted by their transition probabilities a, and multiplying by\nthe observation probability bi(ot). For many applications of HMMs, many of the transition\nprobabilities are 0, so not all previous states will contribute to the forward probability of the\ncurrent state. Hidden states are in circles, observations in squares. Shaded nodes are included\nin the probability computation for at(i).\nfunction FORWARD (observations of len T,state-graph of len N)returns forward-prob\ncreate a probability matrix forward[N,T]\nforeach state sfrom 1toNdo ; initialization step\nforward [s,1] ps\u0003bs(o1)\nforeach time step tfrom 2toTdo ; recursion step\nforeach state sfrom 1toNdo\nforward [s;t] NX\ns0=1forward [s0;t\u00001]\u0003as0;s\u0003bs(ot)\nforwardprob NX\ns=1forward [s;T] ; termination step\nreturn forwardprob\nFigure A.7 The forward algorithm, where forward [s;t]represents at(s).\n1. Initialization:\na1(j) = pjbj(o1)1\u0014j\u0014N\n2. Recursion:\nat(j) =NX\ni=1at\u00001(i)ai jbj(ot); 1\u0014j\u0014N;1<t\u0014T\n3. Termination:\nP(Ojl) =NX\ni=1aT(i) 8APPENDIX A \u2022 H IDDEN MARKOV MODELS\nA.4 Decoding: The Viterbi Algorithm\nFor any model, such as an HMM, that contains hidden variables, the task of deter-\nmining which sequence of variables is the underlying source of some sequence of\nobservations is called the decoding task. In the ice-cream domain, given a sequence decoding\nof ice-cream observations 3 1 3 and an HMM, the task of the decoder is to \ufb01nd the\nbest hidden weather sequence ( H H H ). More formally,\nDecoding : Given as input an HMM l= (A;B)and a sequence of ob-\nservations O=o1;o2;:::;oT, \ufb01nd the most probable sequence of states\nQ=q1q2q3:::qT.\nWe might propose to \ufb01nd the best sequence as follows: For each possible hid-\nden state sequence ( HHH ,HHC ,HCH , etc.), we could run the forward algorithm\nand compute the likelihood of the observation sequence given that hidden state se-\nquence. Then we could choose the hidden state sequence with the maximum obser-\nvation likelihood. It should be clear from the previous section that we cannot do this\nbecause there are an exponentially large number of state sequences.\nInstead, the most common decoding algorithms for HMMs is the Viterbi algo-\nrithm . Like the forward algorithm, Viterbi is a kind of dynamic programmingViterbi\nalgorithm\nthat makes uses of a dynamic programming trellis. Viterbi also strongly resembles\nanother dynamic programming variant, the minimum edit distance algorithm of\nChapter 2.\n\u03c0HCHCHCP(C|start) * P(3|C).2 * .1P(H|H) * P(1|H).6 * .2P(C|C) * P(1|C).5 * .5P(C|H) * P(1|C).4 * .5P(H|C) * P(1|H).5 * .2P(H|start)*P(3|H).8 * .4v1(2)=.32v1(1) = .02v2(2)= max(.32*.12, .02*.10) = .038v2(1) = max(.32*.20, .02*.25) = .064\ntCHq2q1o1o2o3313\nFigure A.8 The Viterbi trellis for computing the best path through the hidden state space for the ice-cream\neating events 3 1 3 . Hidden states are in circles, observations in squares. White (un\ufb01lled) circles indicate illegal\ntransitions. The \ufb01gure shows the computation of vt(j)for two states at two time steps. The computation in each\ncell follows Eq. A.14: vt(j) =max 1\u0014i\u0014N\u00001vt\u00001(i)ai jbj(ot). The resulting probability expressed in each cell\nis Eq. A.13: vt(j) =P(q0;q1;:::; qt\u00001;o1;o2;:::; ot;qt=jjl).\nFigure A.8 shows an example of the Viterbi trellis for computing the best hidden\nstate sequence for the observation sequence 3 1 3 . The idea is to process the ob-\nservation sequence left to right, \ufb01lling out the trellis. Each cell of the trellis, vt(j),\nrepresents the probability that the HMM is in state jafter seeing the \ufb01rst tobser-\nvations and passing through the most probable state sequence q1;:::;qt\u00001, given the A.4 \u2022 D ECODING : THEVITERBI ALGORITHM 9\nautomaton l. The value of each cell vt(j)is computed by recursively taking the\nmost probable path that could lead us to this cell. Formally, each cell expresses the\nprobability\nvt(j) = max\nq1;:::;qt\u00001P(q1:::qt\u00001;o1;o2:::ot;qt=jjl) (A.13)\nNote that we represent the most probable path by taking the maximum over all\npossible previous state sequences max\nq1;:::;qt\u00001. Like other dynamic programming algo-\nrithms, Viterbi \ufb01lls each cell recursively. Given that we had already computed the\nprobability of being in every state at time t\u00001, we compute the Viterbi probability\nby taking the most probable of the extensions of the paths that lead to the current\ncell. For a given state qjat time t, the value vt(j)is computed as\nvt(j) =Nmax\ni=1vt\u00001(i)ai jbj(ot) (A.14)\nThe three factors that are multiplied in Eq. A.14 for extending the previous paths to\ncompute the Viterbi probability at time tare\nvt\u00001(i) theprevious Viterbi path probability from the previous time step\nai j thetransition probability from previous state qito current state qj\nbj(ot) thestate observation likelihood of the observation symbol otgiven\nthe current state j\nfunction VITERBI (observations of len T,state-graph of len N)returns best-path ,path-prob\ncreate a path probability matrix viterbi[N,T]\nforeach state sfrom 1toNdo ; initialization step\nviterbi [s,1] ps\u0003bs(o1)\nbackpointer [s,1] 0\nforeach time step tfrom 2toTdo ; recursion step\nforeach state sfrom 1toNdo\nviterbi [s,t] Nmax\ns0=1viterbi [s0;t\u00001]\u0003as0;s\u0003bs(ot)\nbackpointer [s,t] Nargmax\ns0=1viterbi [s0;t\u00001]\u0003as0;s\u0003bs(ot)\nbestpathprob Nmax\ns=1viterbi [s;T] ; termination step\nbestpathpointer Nargmax\ns=1viterbi [s;T] ; termination step\nbestpath the path starting at state bestpathpointer , that follows backpointer[] to states back in time\nreturn bestpath ,bestpathprob\nFigure A.9 Viterbi algorithm for \ufb01nding optimal sequence of hidden states. Given an observation sequence\nand an HMM l= (A;B), the algorithm returns the state path through the HMM that assigns maximum likelihood\nto the observation sequence.\nFigure A.9 shows pseudocode for the Viterbi algorithm. Note that the Viterbi\nalgorithm is identical to the forward algorithm except that it takes the max over the\nprevious path probabilities whereas the forward algorithm takes the sum. Note also\nthat the Viterbi algorithm has one component that the forward algorithm doesn\u2019t 10 APPENDIX A \u2022 H IDDEN MARKOV MODELS\nhave: backpointers . The reason is that while the forward algorithm needs to pro-\nduce an observation likelihood, the Viterbi algorithm must produce a probability and\nalso the most likely state sequence. We compute this best state sequence by keeping\ntrack of the path of hidden states that led to each state, as suggested in Fig. A.10, and\nthen at the end backtracing the best path to the beginning (the Viterbi backtrace ).Viterbi\nbacktrace\n\u03c0HCHCHCP(C|start) * P(3|C).2 * .1P(H|H) * P(1|H).6 * .2P(C|C) * P(1|C).5 * .5P(C|H) * P(1|C).4 * .5P(H|C) * P(1|H).5 * .2P(H|start)*P(3|H).8 * .4v1(2)=.32v1(1) = .02v2(2)= max(.32*.12, .02*.10) = .038v2(1) = max(.32*.20, .02*.25) = .064\ntCHq2q1o1o2o3313\nFigure A.10 The Viterbi backtrace. As we extend each path to a new state account for the next observation,\nwe keep a backpointer (shown with broken lines) to the best path that led us to this state.\nFinally, we can give a formal de\ufb01nition of the Viterbi recursion as follows:\n1.Initialization:\nv1(j) = pjbj(o1) 1\u0014j\u0014N\nbt1(j) = 0 1\u0014j\u0014N\n2.Recursion\nvt(j) =Nmax\ni=1vt\u00001(i)ai jbj(ot); 1\u0014j\u0014N;1<t\u0014T\nbtt(j) =Nargmax\ni=1vt\u00001(i)ai jbj(ot); 1\u0014j\u0014N;1<t\u0014T\n3.Termination:\nThe best score: P\u0003=Nmax\ni=1vT(i)\nThe start of backtrace: qT\u0003=Nargmax\ni=1vT(i)\nA.5 HMM Training: The Forward-Backward Algorithm\nWe turn to the third problem for HMMs: learning the parameters of an HMM, that\nis, the AandBmatrices. Formally,\nLearning: Given an observation sequence Oand the set of possible\nstates in the HMM, learn the HMM parameters AandB. A.5 \u2022 HMM T RAINING : THEFORWARD -BACKWARD ALGORITHM 11\nThe input to such a learning algorithm would be an unlabeled sequence of ob-\nservations Oand a vocabulary of potential hidden states Q. Thus, for the ice cream\ntask, we would start with a sequence of observations O=f1;3;2;:::;gand the set of\nhidden states HandC.\nThe standard algorithm for HMM training is the forward-backward , orBaum-Forward-\nbackward\nWelch algorithm (Baum, 1972), a special case of the Expectation-Maximization Baum-Welch\norEM algorithm (Dempster et al., 1977). The algorithm will let us train both the EM\ntransition probabilities Aand the emission probabilities Bof the HMM. EM is an\niterative algorithm, computing an initial estimate for the probabilities, then using\nthose estimates to compute a better estimate, and so on, iteratively improving the\nprobabilities that it learns.\nLet us begin by considering the much simpler case of training a fully visible\nMarkov model, where we know both the temperature and the ice cream count for\nevery day. That is, imagine we see the following set of input observations and mag-\nically knew the aligned hidden state sequences:\n3 3 2 1 1 2 1 2 3\nhot hot cold cold cold cold cold hot hot\nThis would easily allow us to compute the HMM parameters just by maximum\nlikelihood estimation from the training data. First, we can compute pfrom the count\nof the 3 initial hidden states:\nph=1=3pc=2=3\nNext we can directly compute the Amatrix from the transitions, ignoring the \ufb01nal\nhidden states:\np(hotjhot) =2=3p(coldjhot) =1=3\np(coldjcold) =2=3p(hotjcold) =1=3\nand the Bmatrix:\nP(1jhot) =0=4=0p(1jcold) =3=5=:6\nP(2jhot) =1=4=:25 p(2jcold=2=5=:4\nP(3jhot) =3=4=:75 p(3jcold) =0\nFor a real HMM, we cannot compute these counts directly from an observation\nsequence since we don\u2019t know which path of states was taken through the machine\nfor a given input. For example, suppose I didn\u2019t tell you the temperature on day 2,\nand you had to guess it, but you (magically) had the above probabilities, and the\ntemperatures on the other days. You could do some Bayesian arithmetic with all the\nother probabilities to get estimates of the likely temperature on that missing day, and\nuse those to get expected counts for the temperatures for day 2.\nBut the real problem is even harder: we don\u2019t know the counts of being in any\nof the hidden states!! The Baum-Welch algorithm solves this by iteratively esti-\nmating the counts. We will start with an estimate for the transition and observation\nprobabilities and then use these estimated probabilities to derive better and better\nprobabilities. And we\u2019re going to do this by computing the forward probability for\nan observation and then dividing that probability mass among all the different paths\nthat contributed to this forward probability.\nTo understand the algorithm, we need to de\ufb01ne a useful probability related to the\nforward probability and called the backward probability . The backward probabil-backward\nprobability 12 APPENDIX A \u2022 H IDDEN MARKOV MODELS\nitybis the probability of seeing the observations from time t+1 to the end, given\nthat we are in state iat time t(and given the automaton l):\nbt(i) =P(ot+1;ot+2:::oTjqt=i;l) (A.15)\nIt is computed inductively in a similar manner to the forward algorithm.\n1.Initialization:\nbT(i) = 1;1\u0014i\u0014N\n2.Recursion\nbt(i) =NX\nj=1ai jbj(ot+1)bt+1(j);1\u0014i\u0014N;1\u0014t<T\n3.Termination:\nP(Ojl) =NX\nj=1pjbj(o1)b1(j)\nFigure A.11 illustrates the backward induction step.\not+1otai1ai2aiNai3b1(ot+1)\u03b2t(i)= \u03a3j \u03b2t+1(j) aij  bj(ot+1) \nq1q2q3qN\nq1qiq2q1q2ot-1q3qN\u03b2t+1(N)\n\u03b2t+1(3)\u03b2t+1(2)\u03b2t+1(1)b2(ot+1)b3(ot+1)bN(ot+1)\nFigure A.11 The computation of bt(i)by summing all the successive values bt+1(j)\nweighted by their transition probabilities ai jand their observation probabilities bj(ot+1).\nWe are now ready to see how the forward and backward probabilities can help\ncompute the transition probability ai jand observation probability bi(ot)from an ob-\nservation sequence, even though the actual path taken through the model is hidden.\nLet\u2019s begin by seeing how to estimate \u02c6 ai jby a variant of simple maximum like-\nlihood estimation:\n\u02c6ai j=expected number of transitions from state ito state j\nexpected number of transitions from state i(A.16)\nHow do we compute the numerator? Here\u2019s the intuition. Assume we had some\nestimate of the probability that a given transition i!jwas taken at a particular\npoint in time tin the observation sequence. If we knew this probability for each\nparticular time t, we could sum over all times tto estimate the total count for the\ntransition i!j. A.5 \u2022 HMM T RAINING : THEFORWARD -BACKWARD ALGORITHM 13\nMore formally, let\u2019s de\ufb01ne the probability xtas the probability of being in state\niat time tand state jat time t+1, given the observation sequence and of course the\nmodel:\nxt(i;j) =P(qt=i;qt+1=jjO;l) (A.17)\nTo compute xt, we \ufb01rst compute a probability which is similar to xt, but differs in\nincluding the probability of the observation; note the different conditioning of O\nfrom Eq. A.17:\nnot-quite- xt(i;j) =P(qt=i;qt+1=j;Ojl) (A.18)\not+2ot+1\u03b1t(i)ot-1otaijbj(ot+1) sisj\u03b2t+1(j)\nFigure A.12 Computation of the joint probability of being in state iat time tand state jat\ntime t+1. The \ufb01gure shows the various probabilities that need to be combined to produce\nP(qt=i;qt+1=j;Ojl): the aandbprobabilities, the transition probability ai jand the\nobservation probability bj(ot+1). After Rabiner (1989) which is \u00a91989 IEEE.\nFigure A.12 shows the various probabilities that go into computing not-quite- xt:\nthe transition probability for the arc in question, the aprobability before the arc, the\nbprobability after the arc, and the observation probability for the symbol just after\nthe arc. These four are multiplied together to produce not-quite- xtas follows:\nnot-quite- xt(i;j) =at(i)ai jbj(ot+1)bt+1(j) (A.19)\nTo compute xtfrom not-quite- xt, we follow the laws of probability and divide by\nP(Ojl), since\nP(XjY;Z) =P(X;YjZ)\nP(YjZ)(A.20)\nThe probability of the observation given the model is simply the forward proba-\nbility of the whole utterance (or alternatively, the backward probability of the whole\nutterance):\nP(Ojl) =NX\nj=1at(j)bt(j) (A.21) 14 APPENDIX A \u2022 H IDDEN MARKOV MODELS\nSo, the \ufb01nal equation for xtis\nxt(i;j) =at(i)ai jbj(ot+1)bt+1(j)PN\nj=1at(j)bt(j)(A.22)\nThe expected number of transitions from state ito state jis then the sum over all\ntofx. For our estimate of ai jin Eq. A.16, we just need one more thing: the total\nexpected number of transitions from state i. We can get this by summing over all\ntransitions out of state i. Here\u2019s the \ufb01nal formula for \u02c6 ai j:\n\u02c6ai j=PT\u00001\nt=1xt(i;j)PT\u00001\nt=1PN\nk=1xt(i;k)(A.23)\nWe also need a formula for recomputing the observation probability. This is the\nprobability of a given symbol vkfrom the observation vocabulary V, given a state j:\n\u02c6bj(vk). We will do this by trying to compute\n\u02c6bj(vk) =expected number of times in state jand observing symbol vk\nexpected number of times in state j(A.24)\nFor this, we will need to know the probability of being in state jat time t, which\nwe will call gt(j):\ngt(j) =P(qt=jjO;l) (A.25)\nOnce again, we will compute this by including the observation sequence in the\nprobability:\ngt(j) =P(qt=j;Ojl)\nP(Ojl)(A.26)\not+1\u03b1t(j)ot-1otsj\u03b2t(j)\nFigure A.13 The computation of gt(j), the probability of being in state jat time t. Note\nthatgis really a degenerate case of xand hence this \ufb01gure is like a version of Fig. A.12 with\nstate icollapsed with state j. After Rabiner (1989) which is \u00a91989 IEEE.\nAs Fig. A.13 shows, the numerator of Eq. A.26 is just the product of the forward\nprobability and the backward probability:\ngt(j) =at(j)bt(j)\nP(Ojl)(A.27) A.5 \u2022 HMM T RAINING : THEFORWARD -BACKWARD ALGORITHM 15\nWe are ready to compute b. For the numerator, we sum gt(j)for all time steps\ntin which the observation otis the symbol vkthat we are interested in. For the\ndenominator, we sum gt(j)over all time steps t. The result is the percentage of the\ntimes that we were in state jand saw symbol vk(the notationPT\nt=1s:t:Ot=vkmeans\n\u201csum over all tfor which the observation at time twasvk\u201d):\n\u02c6bj(vk) =PT\nt=1s:t:Ot=vkgt(j)\nPT\nt=1gt(j)(A.28)\nWe now have ways in Eq. A.23 and Eq. A.28 to re-estimate the transition Aand ob-\nservation Bprobabilities from an observation sequence O, assuming that we already\nhave a previous estimate of AandB.\nThese re-estimations form the core of the iterative forward-backward algorithm.\nThe forward-backward algorithm (Fig. A.14) starts with some initial estimate of the\nHMM parameters l= (A;B). We then iteratively run two steps. Like other cases of\nthe EM (expectation-maximization) algorithm, the forward-backward algorithm has\ntwo steps: the expectation step, or E-step , and the maximization step, or M-step . E-step\nM-step In the E-step, we compute the expected state occupancy count gand the expected\nstate transition count xfrom the earlier AandBprobabilities. In the M-step, we use\ngandxto recompute new AandBprobabilities.\nfunction FORWARD -BACKWARD (observations of len T,output vocabulary V ,hidden\nstate set Q )returns HMM=(A,B)\ninitialize AandB\niterate until convergence\nE-step\ngt(j) =at(j)bt(j)\naT(qF)8tandj\nxt(i;j) =at(i)ai jbj(ot+1)bt+1(j)\naT(qF)8t;i;andj\nM-step\n\u02c6ai j=T\u00001X\nt=1xt(i;j)\nT\u00001X\nt=1NX\nk=1xt(i;k)\n\u02c6bj(vk) =TX\nt=1s:t:Ot=vkgt(j)\nTX\nt=1gt(j)\nreturn A,B\nFigure A.14 The forward-backward algorithm.\nAlthough in principle the forward-backward algorithm can do completely unsu-\npervised learning of the AandBparameters, in practice the initial conditions are\nvery important. For this reason the algorithm is often given extra information. For\nexample, for HMM-based speech recognition, the HMM structure is often set by\nhand, and only the emission ( B) and (non-zero) Atransition probabilities are trained\nfrom a set of observation sequences O. 16 APPENDIX A \u2022 H IDDEN MARKOV MODELS\nA.6 Summary\nThis chapter introduced the hidden Markov model for probabilistic sequence clas-\nsi\ufb01cation .\n\u2022 Hidden Markov models ( HMMs ) are a way of relating a sequence of obser-\nvations to a sequence of hidden classes orhidden states that explain the\nobservations.\n\u2022 The process of discovering the sequence of hidden states, given the sequence\nof observations, is known as decoding orinference . The Viterbi algorithm is\ncommonly used for decoding.\n\u2022 The parameters of an HMM are the Atransition probability matrix and the B\nobservation likelihood matrix. Both can be trained with the Baum-Welch or\nforward-backward algorithm.\nBibliographical and Historical Notes\nAs we discussed in Chapter 17, Markov chains were \ufb01rst used by Markov (1913)\n(translation Markov 2006), to predict whether an upcoming letter in Pushkin\u2019s Eu-\ngene Onegin would be a vowel or a consonant. The hidden Markov model was de-\nveloped by Baum and colleagues at the Institute for Defense Analyses in Princeton\n(Baum and Petrie 1966, Baum and Eagon 1967).\nTheViterbi algorithm was \ufb01rst applied to speech and language processing in the\ncontext of speech recognition by Vintsyuk (1968) but has what Kruskal (1983) calls\na \u201cremarkable history of multiple independent discovery and publication\u201d. Kruskal\nand others give at least the following independently-discovered variants of the algo-\nrithm published in four separate \ufb01elds:\nCitation Field\nViterbi (1967) information theory\nVintsyuk (1968) speech processing\nNeedleman and Wunsch (1970) molecular biology\nSakoe and Chiba (1971) speech processing\nSankoff (1972) molecular biology\nReichert et al. (1973) molecular biology\nWagner and Fischer (1974) computer science\nThe use of the term Viterbi is now standard for the application of dynamic pro-\ngramming to any kind of probabilistic maximization problem in speech and language\nprocessing. For non-probabilistic problems (such as for minimum edit distance), the\nplain term dynamic programming is often used. Forney, Jr. (1973) wrote an early\nsurvey paper that explores the origin of the Viterbi algorithm in the context of infor-\nmation and communications theory.\nOur presentation of the idea that hidden Markov models should be characterized\nby three fundamental problems was modeled after an in\ufb02uential tutorial by Rabiner\n(1989), which was itself based on tutorials by Jack Ferguson of IDA in the 1960s.\nJelinek (1997) and Rabiner and Juang (1993) give very complete descriptions of the\nforward-backward algorithm as applied to the speech recognition problem. Jelinek\n(1997) also shows the relationship between forward-backward and EM. Bibliographical and Historical Notes 17\nBaum, L. E. 1972. An inequality and associated maximiza-\ntion technique in statistical estimation for probabilistic\nfunctions of Markov processes. Inequalities III: Pro-\nceedings of the 3rd Symposium on Inequalities . Academic\nPress.\nBaum, L. E. and J. A. Eagon. 1967. An inequality with appli-\ncations to statistical estimation for probabilistic functions\nof Markov processes and to a model for ecology. Bulletin\nof the American Mathematical Society , 73(3):360\u2013363.\nBaum, L. E. and T. Petrie. 1966. Statistical inference for\nprobabilistic functions of \ufb01nite-state Markov chains. An-\nnals of Mathematical Statistics , 37(6):1554\u20131563.\nDempster, A. P., N. M. Laird, and D. B. Rubin. 1977. Max-\nimum likelihood from incomplete data via the EMalgo-\nrithm. Journal of the Royal Statistical Society , 39(1):1\u2013\n21.\nEisner, J. 2002. An interactive spreadsheet for teaching\nthe forward-backward algorithm. Proceedings of the\nACL Workshop on Effective Tools and Methodologies for\nTeaching NLP and CL .\nForney, Jr., G. D. 1973. The Viterbi algorithm. Proceedings\nof the IEEE , 61(3):268\u2013278.\nJelinek, F. 1997. Statistical Methods for Speech Recognition .\nMIT Press.\nKruskal, J. B. 1983. An overview of sequence comparison.\nIn D. Sankoff and J. B. Kruskal, eds, Time Warps, String\nEdits, and Macromolecules: The Theory and Practice of\nSequence Comparison , 1\u201344. Addison-Wesley.\nMarkov, A. A. 1913. Essai d\u2019une recherche statistique sur\nle texte du roman \u201cEugene Onegin\u201d illustrant la liaison\ndes epreuve en chain (\u2018Example of a statistical investiga-\ntion of the text of \u201cEugene Onegin\u201d illustrating the de-\npendence between samples in chain\u2019). Izvistia Impera-\ntorskoi Akademii Nauk (Bulletin de l\u2019Acad \u00b4emie Imp \u00b4eriale\ndes Sciences de St.-P \u00b4etersbourg) , 7:153\u2013162.\nMarkov, A. A. 2006. Classical text in translation: A. A.\nMarkov, an example of statistical investigation of the text\nEugene Onegin concerning the connection of samples in\nchains. Science in Context , 19(4):591\u2013600. Translated by\nDavid Link.\nNeedleman, S. B. and C. D. Wunsch. 1970. A general\nmethod applicable to the search for similarities in the\namino-acid sequence of two proteins. Journal of Molec-\nular Biology , 48:443\u2013453.\nRabiner, L. R. 1989. A tutorial on hidden Markov models\nand selected applications in speech recognition. Proceed-\nings of the IEEE , 77(2):257\u2013286.\nRabiner, L. R. and B. H. Juang. 1993. Fundamentals of\nSpeech Recognition . Prentice Hall.\nReichert, T. A., D. N. Cohen, and A. K. C. Wong. 1973.\nAn application of information theory to genetic mutations\nand the matching of polypeptide sequences. Journal of\nTheoretical Biology , 42:245\u2013261.\nSakoe, H. and S. Chiba. 1971. A dynamic programming\napproach to continuous speech recognition. Proceedings\nof the Seventh International Congress on Acoustics , vol-\nume 3. Akad \u00b4emiai Kiad \u00b4o.\nSankoff, D. 1972. Matching sequences under deletion-\ninsertion constraints. Proceedings of the National\nAcademy of Sciences , 69:4\u20136.Vintsyuk, T. K. 1968. Speech discrimination by dynamic\nprogramming. Cybernetics , 4(1):52\u201357. Original Rus-\nsian: Kibernetika 4(1):81-88. 1968.\nViterbi, A. J. 1967. Error bounds for convolutional codes and\nan asymptotically optimum decoding algorithm. IEEE\nTransactions on Information Theory , IT-13(2):260\u2013269.\nWagner, R. A. and M. J. Fischer. 1974. The string-to-string\ncorrection problem. Journal of the ACM , 21:168\u2013173.", "B.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\nBSpelling Correction and the\nNoisy Channel\nALGERNON : But my own sweet Cecily, I have never written you any letters.\nCECILY : You need hardly remind me of that, Ernest. I remember only too well\nthat I was forced to write your letters for you. I wrote always three times a week,\nand sometimes oftener.\nALGERNON : Oh, do let me read them, Cecily?\nCECILY : Oh, I couldn\u2019t possibly. They would make you far too conceited. The\nthree you wrote me after I had broken off the engagement are so beautiful, and\nso badly spelled, that even now I can hardly read them without crying a little.\nOscar Wilde, The Importance of Being Earnest\nLike Oscar Wilde\u2019s fabulous Cecily, a lot of people were thinking about spelling\nduring the last turn of the century. Gilbert and Sullivan provide many examples. The\nGondoliers \u2019 Giuseppe, for example, worries that his private secretary is \u201cshaky in his\nspelling\u201d, while Iolanthe \u2019s Phyllis can \u201cspell every word that she uses\u201d. Thorstein\nVeblen\u2019s explanation (in his 1899 classic The Theory of the Leisure Class ) was that\na main purpose of the \u201carchaic, cumbrous, and ineffective\u201d English spelling system\nwas to be dif\ufb01cult enough to provide a test of membership in the leisure class.\nWhatever the social role of spelling, we can certainly agree that many more of\nus are like Cecily than like Phyllis. Estimates for the frequency of spelling errors\nin human-typed text vary from 1-2% for carefully retyping already printed text to\n10-15% for web queries.\nIn this chapter we introduce the problem of detecting and correcting spelling\nerrors. Fixing spelling errors is an integral part of writing in the modern world,\nwhether this writing is part of texting on a phone, sending email, writing longer\ndocuments, or \ufb01nding information on the web. Modern spell correctors aren\u2019t perfect\n(indeed, autocorrect-gone-wrong is a popular source of amusement on the web) but\nthey are ubiquitous in pretty much any software that relies on keyboard input.\nSpelling correction is often considered from two perspectives. Non-word spelling\ncorrection is the detection and correction of spelling errors that result in non-words\n(like graffe forgiraffe ). By contrast, real word spelling correction is the task of\ndetecting and correcting spelling errors even if they accidentally result in an actual\nword of English ( real-word errors ). This can happen from typographical errorsreal-word\nerrors\n(insertion, deletion, transposition) that accidentally produce a real word (e.g., there\nforthree ), orcognitive errors where the writer substituted the wrong spelling of a\nhomophone or near-homophone (e.g., dessert fordesert , orpiece forpeace ).\nNon-word errors are detected by looking for any word not found in a dictio-\nnary. For example, the misspelling graffe above would not occur in a dictionary.\nThe larger the dictionary the better; modern systems often use enormous dictio-\nnaries derived from the web. To correct non-word spelling errors we \ufb01rst generate 2APPENDIX B \u2022 S PELLING CORRECTION AND THE NOISY CHANNEL\ncandidates : real words that have a similar letter sequence to the error. Candidate candidates\ncorrections from the spelling error graffe might include giraffe ,graf,gaffe ,grail , or\ncraft. We then rank the candidates using a distance metric between the source and\nthe surface error. We\u2019d like a metric that shares our intuition that giraffe is a more\nlikely source than grail forgraffe because giraffe is closer in spelling to graffe than\ngrail is to graffe . The minimum edit distance algorithm from Chapter 2 will play a\nrole here. But we\u2019d also like to prefer corrections that are more frequent words, or\nmore likely to occur in the context of the error. The noisy channel model introduced\nin the next section offers a way to formalize this intuition.\nReal word spelling error detection is a much more dif\ufb01cult task, since any word\nin the input text could be an error. Still, it is possible to use the noisy channel to \ufb01nd\ncandidates for each word wtyped by the user, and rank the correction that is most\nlikely to have been the user\u2019s original intention.\nB.1 The Noisy Channel Model\nIn this section we introduce the noisy channel model and show how to apply it to\nthe task of detecting and correcting spelling errors. The noisy channel model was\napplied to the spelling correction task at about the same time by researchers at AT&T\nBell Laboratories (Kernighan et al. 1990, Church and Gale 1991) and IBM Watson\nResearch (Mays et al., 1991).\ndecoder noisy wordoriginal wordnoisy channelguessed wordnoisy 1noisy 2noisy Nword hyp1word hyp2...word hyp3\nFigure B.1 In the noisy channel model, we imagine that the surface form we see is actually\na \u201cdistorted\u201d form of an original word passed through a noisy channel. The decoder passes\neach hypothesis through a model of this channel and picks the word that best matches the\nsurface noisy word.\nThe intuition of the noisy channel model (see Fig. B.1) is to treat the misspelled noisy channel\nword as if a correctly spelled word had been \u201cdistorted\u201d by being passed through a\nnoisy communication channel.\nThis channel introduces \u201cnoise\u201d in the form of substitutions or other changes to\nthe letters, making it hard to recognize the \u201ctrue\u201d word. Our goal, then, is to build a\nmodel of the channel. Given this model, we then \ufb01nd the true word by passing every\nword of the language through our model of the noisy channel and seeing which one\ncomes the closest to the misspelled word.\nThis noisy channel model is a kind of Bayesian inference . We see an obser- Bayesian B.1 \u2022 T HENOISY CHANNEL MODEL 3\nvation x(a misspelled word) and our job is to \ufb01nd the word wthat generated this\nmisspelled word. Out of all possible words in the vocabulary Vwe want to \ufb01nd the\nword wsuch that P(wjx)is highest. We use the hat notation \u02c6 to mean \u201cour estimate\nof the correct word\u201d.\n\u02c6w=argmax\nw2VP(wjx) (B.1)\nThe function argmax xf(x)means \u201cthe xsuch that f(x)is maximized\u201d. Equa- argmax\ntion B.1 thus means, that out of all words in the vocabulary, we want the particular\nword that maximizes the right-hand side P(wjx).\nThe intuition of Bayesian classi\ufb01cation is to use Bayes\u2019 rule to transform Eq. B.1\ninto a set of other probabilities. Bayes\u2019 rule is presented in Eq. B.2; it gives us a way\nto break down any conditional probability P(ajb)into three other probabilities:\nP(ajb) =P(bja)P(a)\nP(b)(B.2)\nWe can then substitute Eq. B.2 into Eq. B.1 to get Eq. B.3:\n\u02c6w=argmax\nw2VP(xjw)P(w)\nP(x)(B.3)\nWe can conveniently simplify Eq. B.3 by dropping the denominator P(x). Why\nis that? Since we are choosing a potential correction word out of all words, we will\nbe computingP(xjw)P(w)\nP(x)for each word. But P(x)doesn\u2019t change for each word; we\nare always asking about the most likely word for the same observed error x, which\nmust have the same probability P(x). Thus, we can choose the word that maximizes\nthis simpler formula:\n\u02c6w=argmax\nw2VP(xjw)P(w) (B.4)\nTo summarize, the noisy channel model says that we have some true underlying\nword w, and we have a noisy channel that modi\ufb01es the word into some possible\nmisspelled observed surface form. The likelihood orchannel model of the noisy likelihood\nchannel model channel producing any particular observation sequence xis modeled by P(xjw). The\nprior probability of a hidden word is modeled by P(w). We can compute the mostprior\nprobability\nprobable word \u02c6 wgiven that we\u2019ve seen some observed misspelling xby multiply-\ning the prior P(w)and the likelihood P(xjw)and choosing the word for which this\nproduct is greatest.\nWe apply the noisy channel approach to correcting non-word spelling errors by\ntaking any word not in our spelling dictionary, generating a list of candidate words ,\nranking them according to Eq. B.4, and picking the highest-ranked one. We can\nmodify Eq. B.4 to refer to this list of candidate words instead of the full vocabulary\nVas follows:\n\u02c6w=argmax\nw2Cchannel modelz}|{\nP(xjw)priorz}|{\nP(w) (B.5)\nThe noisy channel algorithm is shown in Fig. B.2.\nTo see the details of the computation of the likelihood and the prior (language\nmodel), let\u2019s walk through an example, applying the algorithm to the example mis-\nspelling acress . The \ufb01rst stage of the algorithm proposes candidate corrections by 4APPENDIX B \u2022 S PELLING CORRECTION AND THE NOISY CHANNEL\nfunction NOISY CHANNEL SPELLING (word x ,dict D , lm,editprob )returns correction\nifx=2D\ncandidates, edits All strings at edit distance 1 from xthat are2D, and their edit\nforeach c;ein candidates, edits\nchannel editprob(e)\nprior lm(x)\nscore[c] = log channel + log prior\nreturn argmaxcscore[c]\nFigure B.2 Noisy channel model for spelling correction for unknown words.\n\ufb01nding words that have a similar spelling to the input word. Analysis of spelling\nerror data has shown that the majority of spelling errors consist of a single-letter\nchange and so we often make the simplifying assumption that these candidates have\nan edit distance of 1 from the error word. To \ufb01nd this list of candidates we\u2019ll use\nthe minimum edit distance algorithm introduced in Chapter 2, but extended so that\nin addition to insertions, deletions, and substitutions, we\u2019ll add a fourth type of edit,\ntranspositions, in which two letters are swapped. The version of edit distance with\ntransposition is called Damerau-Levenshtein edit distance. Applying all such sin-Damerau-\nLevenshtein\ngle transformations to acress yields the list of candidate words in Fig. B.3.\nTransformation\nCorrect Error Position\nError Correction Letter Letter (Letter #) Type\nacress actress t \u2014 2 deletion\nacress cress \u2014 a 0 insertion\nacress caress ca ac 0 transposition\nacress access c r 2 substitution\nacress across o e 3 substitution\nacress acres \u2014 s 5 insertion\nacress acres \u2014 s 4 insertion\nFigure B.3 Candidate corrections for the misspelling acress and the transformations that\nwould have produced the error (after Kernighan et al. (1990)). \u201c\u2014\u201d represents a null letter.\nOnce we have a set of a candidates, to score each one using Eq. B.5 requires that\nwe compute the prior and the channel model.\nThe prior probability of each correction P(w)is the language model probability\nof the word win context, which can be computed using any language model, from\nunigram to trigram or 4-gram. For this example let\u2019s start in the following table by\nassuming a unigram language model. We computed the language model from the\n404,253,213 words in the Corpus of Contemporary English (COCA).\nw count(w) p(w)\nactress 9,321 .0000231\ncress 220 .000000544\ncaress 686 .00000170\naccess 37,038 .0000916\nacross 120,844 .000299\nacres 12,874 .0000318\nHow can we estimate the likelihood P(xjw), also called the channel model or channel model B.1 \u2022 T HENOISY CHANNEL MODEL 5\nerror model ? A perfect model of the probability that a word will be mistyped would error model\ncondition on all sorts of factors: who the typist was, whether the typist was left-\nhanded or right-handed, and so on. Luckily, we can get a pretty reasonable estimate\nofP(xjw)just by looking at local context: the identity of the correct letter itself, the\nmisspelling, and the surrounding letters. For example, the letters mandnare often\nsubstituted for each other; this is partly a fact about their identity (these two letters\nare pronounced similarly and they are next to each other on the keyboard) and partly\na fact about context (because they are pronounced similarly and they occur in similar\ncontexts).\nA simple model might estimate, for example, p(acressjacross )just using the\nnumber of times that the letter ewas substituted for the letter oin some large corpus\nof errors. To compute the probability for each edit in this way we\u2019ll need a confu-\nsion matrix that contains counts of errors. In general, a confusion matrix lists theconfusion\nmatrix\nnumber of times one thing was confused with another. Thus for example a substi-\ntution matrix will be a square matrix of size 26 \u000226 (or more generally jAj\u0002jAj,\nfor an alphabet A) that represents the number of times one letter was incorrectly\nused instead of another. Following Kernighan et al. (1990) we\u2019ll use four confusion\nmatrices.\ndel[x;y]: count( xytyped as x)\nins[x;y]: count( xtyped as xy)\nsub[x;y]: count( xtyped as y)\ntrans[x;y]: count( xytyped as yx)\nNote that we\u2019ve conditioned the insertion and deletion probabilities on the previ-\nous character; we could instead have chosen to condition on the following character.\nWhere do we get these confusion matrices? One way is to extract them from\nlists of misspellings like the following:\nadditional : addional, additonal\nenvironments : enviornments, enviorments, enviroments\npreceded : preceeded\n...\nThere are lists available on Wikipedia and from Roger Mitton ( http://www.\ndcs.bbk.ac.uk/ ~ROGER/corpora.html ) and Peter Norvig ( http://norvig.\ncom/ngrams/ ). Norvig also gives the counts for each single-character edit that can\nbe used to directly create the error model probabilities.\nAn alternative approach used by Kernighan et al. (1990) is to compute the ma-\ntrices by iteratively using this very spelling error correction algorithm itself. The\niterative algorithm \ufb01rst initializes the matrices with equal values; thus, any character\nis equally likely to be deleted, equally likely to be substituted for any other char-\nacter, etc. Next, the spelling error correction algorithm is run on a set of spelling\nerrors. Given the set of typos paired with their predicted corrections, the confusion\nmatrices can now be recomputed, the spelling algorithm run again, and so on. This\niterative algorithm is an instance of the important EM algorithm (Dempster et al.,\n1977), which we discuss in Appendix A.\nOnce we have the confusion matrices, we can estimate P(xjw)as follows (where 6APPENDIX B \u2022 S PELLING CORRECTION AND THE NOISY CHANNEL\nwiis the ith character of the correct word w) and xiis the ith character of the typo x:\nP(xjw) =8\n>>>>>>>>>>>><\n>>>>>>>>>>>>:del[xi\u00001;wi]\ncount [xi\u00001wi];if deletion\nins[xi\u00001;wi]\ncount [wi\u00001];if insertion\nsub[xi;wi]\ncount [wi];if substitution\ntrans[wi;wi+1]\ncount [wiwi+1];if transposition(B.6)\nUsing the counts from Kernighan et al. (1990) results in the error model proba-\nbilities for acress shown in Fig. B.4.\nCandidate Correct Error\nCorrection Letter Letter x jw P(xjw)\nactress t - c|ct .000117\ncress - a a|# .00000144\ncaress ca ac ac|ca .00000164\naccess c r r|c .000000209\nacross o e e|o .0000093\nacres - s es|e .0000321\nacres - s ss|s .0000342\nFigure B.4 Channel model for acress ; the probabilities are taken from the del[],ins[],\nsub[], and trans [] confusion matrices as shown in Kernighan et al. (1990).\nFigure B.5 shows the \ufb01nal probabilities for each of the potential corrections;\nthe unigram prior is multiplied by the likelihood (computed with Eq. B.6 and the\nconfusion matrices). The \ufb01nal column shows the product, multiplied by 109just for\nreadability.\nCandidate Correct Error\nCorrection Letter Letter x jw P(xjw) P(w) 109*P(xjw)P(w)\nactress t - c|ct .000117 .0000231 2.7\ncress - a a|# .00000144 .000000544 0.00078\ncaress ca ac ac|ca .00000164 .00000170 0.0028\naccess c r r|c .000000209 .0000916 0.019\nacross o e e|o .0000093 .000299 2.8\nacres - s es|e .0000321 .0000318 1.0\nacres - s ss|s .0000342 .0000318 1.0\nFigure B.5 Computation of the ranking for each candidate correction, using the language\nmodel shown earlier and the error model from Fig. B.4. The \ufb01nal score is multiplied by 109\nfor readability.\nThe computations in Fig. B.5 show that our implementation of the noisy channel\nmodel chooses across as the best correction, and actress as the second most\nlikely word.\nUnfortunately, the algorithm was wrong here; the writer\u2019s intention becomes\nclear from the context: . . . was called a \u201cstellar and versatile acress whose com-\nbination of sass and glamour has de\ufb01ned her. . . \u201d. The surrounding words make it\nclear that actress and not across was the intended word. B.2 \u2022 R EAL-WORD SPELLING ERRORS 7\nFor this reason, it is important to use larger language models than unigrams.\nFor example, if we use the Corpus of Contemporary American English to compute\nbigram probabilities for the words actress andacross in their context using add-one\nsmoothing, we get the following probabilities:\nP(actressjversatile) =:000021\nP(acrossjversatile) =:000021\nP(whosejactress) =:0010\nP(whosejacross) =:000006\nMultiplying these out gives us the language model estimate for the two candi-\ndates in context:\nP(\u201cversatile actress whose\u201d) =:000021\u0003:0010 =210\u000210\u000010\nP(\u201cversatile across whose\u201d) =:000021\u0003:000006 =1\u000210\u000010\nCombining the language model with the error model in Fig. B.5, the bigram\nnoisy channel model now chooses the correct word actress .\nEvaluating spell correction algorithms is generally done by holding out a train-\ning, development and test set from lists of errors like those on the Norvig and Mitton\nsites mentioned above.\nB.2 Real-word spelling errors\nThe noisy channel approach can also be applied to detect and correct real-word\nspelling errors , errors that result in an actual word of English. This can happen fromreal-word error\ndetection\ntypographical errors (insertion, deletion, transposition) that accidentally produce a\nreal word (e.g., there forthree ) or because the writer substituted the wrong spelling\nof a homophone or near-homophone (e.g., dessert fordesert , orpiece forpeace ). A\nnumber of studies suggest that between 25% and 40% of spelling errors are valid\nEnglish words as in the following examples (Kukich, 1992):\nThis used to belong to thew queen. They are leaving in about \ufb01fteen minuets to go to her house.\nThe design anconstruction of the system will take more than a year.\nCan they lave him my messages?\nThe study was conducted mainly beJohn Black.\nThe noisy channel can deal with real-word errors as well. Let\u2019s begin with a\nversion of the noisy channel model \ufb01rst proposed by Mays et al. (1991) to deal\nwith these real-word spelling errors. Their algorithm takes the input sentence X=\nfx1;x2;:::; xk;:::; xng, generates a large set of candidate correction sentences C(X),\nthen picks the sentence with the highest language model probability.\nTo generate the candidate correction sentences, we start by generating a set of\ncandidate words for each input word xi. The candidates, C(xi), include every En-\nglish word with a small edit distance from xi. With edit distance 1, a common choice\n(Mays et al., 1991), the candidate set for the real word error thew (a rare word mean-\ning \u2018muscular strength\u2019) might be C(thew) = fthe, thaw, threw, them, thwe g. We then\nmake the simplifying assumption that every sentence has only one error. Thus the\nset of candidate sentences C(X)for a sentence X = Only two of thew apples\nwould be: 8APPENDIX B \u2022 S PELLING CORRECTION AND THE NOISY CHANNEL\nonly two of thew apples\noily two of thew apples\nonly too of thew apples\nonly to of thew apples\nonly tao of the apples\nonly two on thew apples\nonly two off thew apples\nonly two of the apples\nonly two of threw apples\nonly two of thew applies\nonly two of thew dapples\n...\nEach sentence is scored by the noisy channel:\n\u02c6W=argmax\nW2C(X)P(XjW)P(W) (B.7)\nForP(W), we can use the trigram probability of the sentence.\nWhat about the channel model? Since these are real words, we need to consider\nthe possibility that the input word is not an error. Let\u2019s say that the channel proba-\nbility of writing a word correctly, P(wjw), isa; we can make different assumptions\nabout exactly what the value of ais in different tasks; perhaps ais .95, assum-\ning people write 1 word wrong out of 20, for some tasks, or maybe .99 for others.\nMays et al. (1991) proposed a simple model: given a typed word x, let the channel\nmodel P(xjw)beawhen x=w, and then just distribute 1 \u0000aevenly over all other\ncandidate corrections C(x):\np(xjw) =8\n>>><\n>>>:a ifx=w\n1\u0000a\njC(x)jifx2C(x)\n0 otherwise(B.8)\nNow we can replace the equal distribution of 1 \u0000aover all corrections in Eq. B.8;\nwe\u2019ll make the distribution proportional to the edit probability from the more sophis-\nticated channel model from Eq. B.6 that used the confusion matrices.\nLet\u2019s see an example of this integrated noisy channel model applied to a real\nword. Suppose we see the string two of thew . The author might have intended\nto type the real word thew (\u2018muscular strength\u2019). But thew here could also be a\ntypo for the or some other word. For the purposes of this example let\u2019s consider\nedit distance 1, and only the following \ufb01ve candidates the,thaw ,threw , and thwe\n(a rare name) and the string as typed, thew . We took the edit probabilities from\nNorvig\u2019s 2009 analysis of this example. For the language model probabilities, we\nused a Stupid Backoff model (Section ??) trained on the Google n-grams:\nP(thejtwo of) = 0.476012\nP(thewjtwo of) = 9.95051 \u000210\u00008\nP(thawjtwo of) = 2.09267 \u000210\u00007\nP(threwjtwo of) = 8.9064 \u000210\u00007\nP(themjtwo of) = 0.00144488\nP(thwejtwo of) = 5.18681 \u000210\u00009\nHere we\u2019ve just computed probabilities for the single phrase two of thew , but\nthe model applies to entire sentences; so if the example in context was two of thew B.3 \u2022 N OISY CHANNEL MODEL : THESTATE OF THE ART 9\npeople , we\u2019d need to also multiply in probabilities for P(people jof the), P(peoplejof\nthew), P(peoplejof threw), and so on.\nFollowing Norvig (2009), we assume that the probability of a word being a typo\nin this task is .05, meaning that a=P(wjw)is .95. Fig. B.6 shows the computation.\nx w xjw P (xjw)P(wjwi\u00002;wi\u00001)108P(xjw)P(wjwi\u00002;wi\u00001)\nthew the ewje 0.000007 0.48 333\nthew thew a=0.95 9.95\u000210\u000089.45\nthew thaw eja 0.001 2.1 \u000210\u000070.0209\nthew threw hjhr 0.000008 8.9 \u000210\u000070.000713\nthew thwe ewjwe 0.000003 5.2 \u000210\u000090.00000156\nFigure B.6 The noisy channel model on 5 possible candidates for thew , with a Stupid\nBackoff trigram language model computed from the Google n-gram corpus and the error\nmodel from Norvig (2009).\nFor the error phrase two of thew , the model correctly picks theas the correction.\nBut note that a lower error rate might change things; in a task where the probability\nof an error is low enough ( ais very high), the model might instead decide that the\nword thew was what the writer intended.\nB.3 Noisy Channel Model: The State of the Art\nState of the art implementations of noisy channel spelling correction make a number\nof extensions to the simple models we presented above.\nFirst, rather than make the assumption that the input sentence has only a sin-\ngle error, modern systems go through the input one word at a time, using the noisy\nchannel to make a decision for that word. But if we just run the basic noisy chan-\nnel system described above on each word, it is prone to overcorrecting , replacing\ncorrect but rare words (for example names) with more frequent words (Whitelaw\net al. 2009, Wilcox-O\u2019Hearn 2014). Modern algorithms therefore need to augment\nthe noisy channel with methods for detecting whether or not a real word should ac-\ntually be corrected. For example state of the art systems like Google\u2019s (Whitelaw\net al., 2009) use a blacklist, forbidding certain tokens (like numbers, punctuation,\nand single letter words) from being changed. Such systems are also more cautious\nin deciding whether to trust a candidate correction. Instead of just choosing a candi-\ndate correction if it has a higher probability P(wjx)than the word itself, these more\ncareful systems choose to suggest a correction wover keeping the non-correction x\nonly if the difference in probabilities is suf\ufb01ciently great. The best correction wis\nchosen only if:\nlogP(wjx)\u0000logP(xjx)>q\nDepending on the speci\ufb01c application, spell-checkers may decide to autocorrect autocorrect\n(automatically change a spelling to a hypothesized correction) or merely to \ufb02ag the\nerror and offer suggestions. This decision is often made by another classi\ufb01er which\ndecides whether the best candidate is good enough, using features such as the dif-\nference in log probabilities between the candidates (we\u2019ll introduce algorithms for\nclassi\ufb01cation in the next chapter).\nModern systems also use much larger dictionaries than early systems. Ahmad\nand Kondrak (2005) found that a 100,000 word UNIX dictionary only contained 10 APPENDIX B \u2022 S PELLING CORRECTION AND THE NOISY CHANNEL\n73% of the word types in their corpus of web queries, missing words like pics, mul-\ntiplayer, google, xbox, clipart , and mallorca . For this reason modern systems often\nuse much larger dictionaries automatically derived from very large lists of unigrams\nlike the Google n-gram corpus. Whitelaw et al. (2009), for example, used the most\nfrequently occurring ten million word types in a large sample of web pages. Because\nthis list will include lots of misspellings, their system requires a more sophisticated\nerror model. The fact that words are generally more frequent than their misspellings\ncan be used in candidate suggestion, by building a set of words and spelling vari-\nations that have similar contexts, sorting by frequency, treating the most frequent\nvariant as the source, and learning an error model from the difference, whether from\nweb text (Whitelaw et al., 2009) or from query logs (Cucerzan and Brill, 2004).\nWords can also be automatically added to the dictionary when a user rejects a cor-\nrection, and systems running on phones can automatically add words from the user\u2019s\naddress book or calendar.\nWe can also improve the performance of the noisy channel model by changing\nhow the prior and the likelihood are combined. In the standard model they are just\nmultiplied together. But often these probabilities are not commensurate; the lan-\nguage model or the channel model might have very different ranges. Alternatively\nfor some task or dataset we might have reason to trust one of the two models more.\nTherefore we use a weighted combination, by raising one of the factors to a power\nl:\n\u02c6w=argmax\nw2VP(xjw)P(w)l(B.9)\nor in log space:\n\u02c6w=argmax\nw2VlogP(xjw)+llogP(w) (B.10)\nWe then tune the parameter lon a development test set.\nFinally, if our goal is to do real-word spelling correction only for speci\ufb01c con-\nfusion sets likepeace/piece ,affect/effect ,weather/whether , or even grammar cor- confusion sets\nrection examples like among/between , we can train supervised classi\ufb01ers to draw on\nmany features of the context and make a choice between the two candidates. Such\nclassi\ufb01ers can achieve very high accuracy for these speci\ufb01c sets, especially when\ndrawing on large-scale features from web statistics (Golding and Roth 1999, Lapata\nand Keller 2004, Bergsma et al. 2009, Bergsma et al. 2010).\nB.3.1 Improved Edit Models: Partitions and Pronunciation\nOther recent research has focused on improving the channel model P(tjc). One\nimportant extension is the ability to compute probabilities for multiple-letter trans-\nformations. For example Brill and Moore (2000) propose a channel model that\n(informally) models an error as being generated by a typist \ufb01rst choosing a word,\nthen choosing a partition of the letters of that word, and then typing each partition,\npossibly erroneously. For example, imagine a person chooses the word physical ,\nthen chooses the partition ph y s i c al . She would then generate each parti-\ntion, possibly with errors. For example the probability that she would generate the\nstring fisikle with partition f i s i k le would be p(fjph)\u0003p(ijy)\u0003p(sjs)\u0003\np(iji)\u0003p(kjk)\u0003p(lejal). Unlike the Damerau-Levenshtein edit distance, the Brill-\nMoore channel model can thus model edit probabilities like P(fjph)orP(lejal), or B.3 \u2022 N OISY CHANNEL MODEL : THESTATE OF THE ART 11\nthe high likelihood of P(entjant). Furthermore, each edit is conditioned on where\nit is in the word ( beginning, middle, end ) so instead of P(fjph)the model actually\nestimates P(fjph;beginning ).\nMore formally, let R be a partition of the typo string xinto adjacent (possibly\nempty) substrings, and T be a partition of the candidate string. Brill and Moore\n(2000) then approximates the total likelihood P(xjw)(e.g., P( fisiklejphysical ))\nby the probability of the single best partition:\nP(xjw)\u0019 max\nR;T s:t:jTj=jRjjRjX\ni=1P(TijRi;position ) (B.11)\nThe probability of each transform P(TijRi)can be learned from a training set of\ntriples of an error, the correct string, and the number of times it occurs. For example\ngiven a training pair akgsual /actual , standard minimum edit distance is used to\nproduce an alignment:\nactualakgsual\nThis alignment corresponds to the sequence of edit operations:\na!a,c!k,\u000f!g t!s,u!u,a!a,l!l\nEach nonmatch substitution is then expanded to incorporate up to N additional\nedits; For N=2, we would expand c!kto:\nac!ak\nc!cg\nac!akg\nct!kgs\nEach of these multiple edits then gets a fractional count, and the probability for\neach edit a!bis then estimated from counts in the training corpus of triples as\ncount (a!b)\ncount (a).\nAnother research direction in channel models is the use of pronunciation in ad-\ndition to spelling. Pronunciation is an important feature in some non-noisy-channel\nalgorithms for spell correction like the GNU aspell algorithm (Atkinson, 2011), aspell\nwhich makes use of the metaphone pronunciation of a word (Philips, 1990). Meta-\nphone is a series of rules that map a word to a normalized representation of its\npronunciation. Some example rules:\n\u2022 \u201cDrop duplicate adjacent letters, except for C.\u201d\n\u2022 \u201cIf the word begins with \u2018KN\u2019, \u2018GN\u2019, \u2018PN\u2019, \u2018AE\u2019, \u2018WR\u2019, drop the \ufb01rst letter.\u201d\n\u2022 \u201cDrop \u2018B\u2019 if after \u2018M\u2019 and if it is at the end of the word\u201d\nAspell works similarly to the channel component of the noisy channel model, \ufb01nding\nall words in the dictionary whose pronunciation string is a short edit distance (1 or\n2 pronunciation letters) from the typo, and then scoring this list of candidates by\na metric that combines two edit distances: the pronunciation edit distance and the\nweighted letter edit distance.\nPronunciation can also be incorporated directly the noisy channel model. For ex-\nample the Toutanova and Moore (2002) model, like aspell, interpolates two channel 12 APPENDIX B \u2022 S PELLING CORRECTION AND THE NOISY CHANNEL\nfunction SOUNDEX (name )returns soundex form\n1. Keep the \ufb01rst letter of name\n2. Drop all occurrences of non-initial a, e, h, i, o, u, w, y.\n3. Replace the remaining letters with the following numbers:\nb, f, p, v!1\nc, g, j, k, q, s, x, z!2\nd, t!3\nl!4\nm, n!5\nr!6\n4. Replace any sequences of identical numbers, only if they derive from two or more\nletters that were adjacent in the original name, with a single number (e.g., 666 !6).\n5. Convert to the form Letter Digit Digit Digit by dropping digits past the third\n(if necessary) or padding with trailing zeros (if necessary).\nFigure B.7 The Soundex Algorithm\nmodels, one based on spelling and one based on pronunciation. The pronunciation\nmodel is based on using letter-to-sound models to translate each input word and letter-to-sound\neach dictionary word into a sequences of phones representing the pronunciation of phones\nthe word. For example actress andaktress would both map to the phone string\nae k t r ix s . See Chapter 16 on the task of letter-to-sound or grapheme-to-\nphoneme .\nSome additional string distance functions have been proposed for dealing specif-\nically with names . These are mainly used for the task of deduplication (deciding if deduplication\ntwo names in a census list or other namelist are the same) rather than spell-checking.\nThe Soundex algorithm (Knuth 1973, Odell and Russell 1918/1922) is an older\nmethod used originally for census records for representing people\u2019s names. It has the\nadvantage that versions of the names that are slightly misspelled will still have the\nsame representation as correctly spelled names. (e.g., Jurafsky, Jarofsky, Jarovsky,\nand Jarovski all map to J612). The algorithm is shown in Fig. B.7.\nInstead of Soundex, more recent work uses Jaro-Winkler distance, which is Jaro-Winkler\nan edit distance algorithm designed for names that allows characters to be moved\nlonger distances in longer names, and also gives a higher similarity to strings that\nhave identical initial characters (Winkler, 2006).\nBibliographical and Historical Notes\nAlgorithms for spelling error detection and correction have existed since at least\nBlair (1960). Most early algorithms were based on similarity keys like the Soundex\nalgorithm (Odell and Russell 1918/1922, Knuth 1973). Damerau (1964) gave a\ndictionary-based algorithm for error detection; most error-detection algorithms since\nthen have been based on dictionaries. Early research (Peterson, 1986) had suggested\nthat spelling dictionaries might need to be kept small because large dictionaries con-\ntain very rare words (wont, veery) that resemble misspellings of other words, but\nDamerau and Mays (1989) found that in practice larger dictionaries proved more\nhelpful. Damerau (1964) also gave a correction algorithm that worked for single\nerrors.\nThe idea of modeling language transmission as a Markov source passed through EXERCISES 13\na noisy channel model was developed very early on by Claude Shannon (1948).\nThe idea of combining a prior and a likelihood to deal with the noisy channel was\ndeveloped at IBM Research by Raviv (1967), for the similar task of optical char-\nacter recognition (OCR ). While earlier spell-checkers like Kashyap and Oommen\n(1983) had used likelihood-based models of edit distance, the idea of combining a\nprior and a likelihood seems not to have been applied to the spelling correction task\nuntil researchers at AT&T Bell Laboratories (Kernighan et al. 1990, Church and\nGale 1991) and IBM Watson Research (Mays et al., 1991) roughly simultaneously\nproposed noisy channel spelling correction. Much later, the Mays et al. (1991) algo-\nrithm was reimplemented and tested on standard datasets by Wilcox-O\u2019Hearn et al.\n(2008), who showed its high performance.\nMost algorithms since Wagner and Fischer (1974) have relied on dynamic pro-\ngramming.\nRecent focus has been on using the web both for language models and for train-\ning the error model, and on incorporating additional features in spelling, like the\npronunciation models described earlier, or other information like parses or semantic\nrelatedness (Jones and Martin 1997, Hirst and Budanitsky 2005).\nSee Mitton (1987) for a survey of human spelling errors, and Kukich (1992)\nfor an early survey of spelling error detection and correction. Norvig (2007) gives\na nice explanation and a Python implementation of the noisy channel model, with\nmore details and an ef\ufb01cient algorithm presented in Norvig (2009).\nExercises\nB.1 Suppose we want to apply add-one smoothing to the likelihood term (channel\nmodel) P(xjw)of a noisy channel model of spelling. For simplicity, pretend\nthat the only possible operation is deletion. The MLE estimate for deletion\nis given in Eq. B.6, which is P(xjw) =del[xi\u00001;wi]\ncount (xi\u00001wi). What is the estimate for\nP(xjw)if we use add-one smoothing on the deletion edit model? Assume the\nonly characters we use are lower case a-z, that there are Vword types in our\ncorpus, and Ntotal characters, not counting spaces. 14 Appendix B \u2022 Spelling Correction and the Noisy Channel\nAhmad, F. and G. Kondrak. 2005. Learning a spelling error\nmodel from search query logs. EMNLP .\nAtkinson, K. 2011. Gnu aspell.\nBergsma, S., D. Lin, and R. Goebel. 2009. Web-scale n-gram\nmodels for lexical disambiguation. IJCAI .\nBergsma, S., E. Pitler, and D. Lin. 2010. Creating robust\nsupervised classi\ufb01ers via web-scale n-gram data. ACL.\nBlair, C. R. 1960. A program for correcting spelling errors.\nInformation and Control , 3:60\u201367.\nBrill, E. and R. C. Moore. 2000. An improved error model\nfor noisy channel spelling correction. ACL.\nChurch, K. W. and W. A. Gale. 1991. Probability scoring for\nspelling correction. Statistics and Computing , 1(2):93\u2013\n103.\nCucerzan, S. and E. Brill. 2004. Spelling correction as an\niterative process that exploits the collective knowledge of\nweb users. EMNLP , volume 4.\nDamerau, F. J. 1964. A technique for computer detection and\ncorrection of spelling errors. CACM , 7(3):171\u2013176.\nDamerau, F. J. and E. Mays. 1989. An examination of un-\ndetected typing errors. Information Processing and Man-\nagement , 25(6):659\u2013664.\nDempster, A. P., N. M. Laird, and D. B. Rubin. 1977. Max-\nimum likelihood from incomplete data via the EMalgo-\nrithm. Journal of the Royal Statistical Society , 39(1):1\u2013\n21.\nGolding, A. R. and D. Roth. 1999. A Winnow based ap-\nproach to context-sensitive spelling correction. Machine\nLearning , 34(1-3):107\u2013130.\nHirst, G. and A. Budanitsky. 2005. Correcting real-word\nspelling errors by restoring lexical cohesion. Natural\nLanguage Engineering , 11:87\u2013111.\nJones, M. P. and J. H. Martin. 1997. Contextual spelling cor-\nrection using latent semantic analysis. ANLP .\nKashyap, R. L. and B. J. Oommen. 1983. Spelling correction\nusing probabilistic methods. Pattern Recognition Letters ,\n2:147\u2013154.\nKernighan, M. D., K. W. Church, and W. A. Gale. 1990.\nA spelling correction program base on a noisy channel\nmodel. COLING , volume II.\nKnuth, D. E. 1973. Sorting and Searching: The Art of Com-\nputer Programming Volume 3 . Addison-Wesley.\nKukich, K. 1992. Techniques for automatically correcting\nwords in text. ACM Computing Surveys , 24(4):377\u2013439.\nLapata, M. and F. Keller. 2004. The web as a baseline: Eval-\nuating the performance of unsupervised web-based mod-\nels for a range of NLP tasks. HLT-NAACL .\nMays, E., F. J. Damerau, and R. L. Mercer. 1991. Context\nbased spelling correction. Information Processing and\nManagement , 27(5):517\u2013522.\nMitton, R. 1987. Spelling checkers, spelling correctors and\nthe misspellings of poor spellers. Information processing\n& management , 23(5):495\u2013505.\nNorvig, P. 2007. How to write a spelling corrector. http:\n//www.norvig.com/spell-correct.html .\nNorvig, P. 2009. Natural language corpus data. In T. Segaran\nand J. Hammerbacher, eds, Beautiful data: the stories be-\nhind elegant data solutions . O\u2019Reilly.Odell, M. K. and R. C. Russell. 1918/1922. U.S. Patents\n1261167 (1918), 1435663 (1922). Cited in Knuth (1973).\nPeterson, J. L. 1986. A note on undetected typing errors.\nCACM , 29(7):633\u2013637.\nPhilips, L. 1990. Hanging on the metaphone. Computer\nLanguage , 7(12).\nRaviv, J. 1967. Decision making in Markov chains applied\nto the problem of pattern recognition. IEEE Transactions\non Information Theory , 13(4):536\u2013551.\nShannon, C. E. 1948. A mathematical theory of commu-\nnication. Bell System Technical Journal , 27(3):379\u2013423.\nContinued in the following volume.\nToutanova, K. and R. C. Moore. 2002. Pronunciation mod-\neling for improved spelling correction. ACL.\nVeblen, T. 1899. Theory of the Leisure Class . Macmillan,\nNew York.\nWagner, R. A. and M. J. Fischer. 1974. The string-to-string\ncorrection problem. Journal of the ACM , 21:168\u2013173.\nWhitelaw, C., B. Hutchinson, G. Y . Chung, and G. El-\nlis. 2009. Using the web for language independent\nspellchecking and autocorrection. EMNLP .\nWilcox-O\u2019Hearn, L. A. 2014. Detection is the central prob-\nlem in real-word spelling correction. http://arxiv.\norg/abs/1408.3153 .\nWilcox-O\u2019Hearn, L. A., G. Hirst, and A. Budanitsky. 2008.\nReal-word spelling correction with trigrams: A recon-\nsideration of the Mays, Damerau, and Mercer model.\nCICLing-2008 .\nWinkler, W. E. 2006. Overview of record linkage and current\nresearch directions. Technical report, Statistical Research\nDivision, U.S. Census Bureau.", "C.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a9\n2024. All rights reserved. Draft of August 20, 2024.\nCHAPTER\nCStatistical Constituency Pars-\ning\nThe characters in Damon Runyon\u2019s short stories are willing to bet \u201con any propo-\nsition whatever\u201d, as Runyon says about Sky Masterson in The Idyll of Miss Sarah\nBrown , from getting aces back-to-back to a man being able to throw a peanut from\nsecond base to home plate. There is a moral here for language processing: with\nenough knowledge we can estimate the probability of just about anything. Chap-\nter 18 introduced constituency structure and the task of parsing it. Here, we show\nhow to build probabilistic models of syntactic knowledge and ef\ufb01cient probabilistic\nparsers.\nOne use of probabilistic parsing is to solve the problem of disambiguation . Re-\ncall from Chapter 18 that sentences on average tend to be syntactically ambiguous\nbecause of phenomena like coordination ambiguity andattachment ambiguity .\nThe CKY parsing algorithm can represent these ambiguities in an ef\ufb01cient way but\nis not equipped to resolve them. There we introduced a neural algorithm for disam-\nbiguation. Here we introduce probabilistic parsers, which offer an alternative solu-\ntion to the problem: compute the probability of each interpretation and choose the\nmost probable interpretation. The most commonly used probabilistic constituency\ngrammar formalism is the probabilistic context-free grammar (PCFG), a prob-\nabilistic augmentation of context-free grammars in which each rule is associated\nwith a probability. We introduce PCFGs in the next section, showing how they can\nbe trained on Treebank grammars and how they can be parsed with a probabilistic\nversion of the CKY algorithm of Chapter 18.\nWe then show a number of ways that we can improve on this basic probabil-\nity model (PCFGs trained on Treebank grammars), such as by modifying the set of\nnon-terminals (making them either more speci\ufb01c or more general), or adding more\nsophisticated conditioning factors like subcategorization or dependencies. Heav-\nily lexicalized grammar formalisms such as Lexical-Functional Grammar (LFG)\n(Bresnan, 1982), Head-Driven Phrase Structure Grammar (HPSG) (Pollard and Sag,\n1994), Tree-Adjoining Grammar (TAG) (Joshi, 1985), and Combinatory Categorial\nGrammar (CCG) pose additional problems for probabilistic parsers. Section ??in-\ntroduces the task of supertagging and the use of heuristic search methods based on\ntheA* algorithm in the context of CCG parsing.\nC.1 Probabilistic Context-Free Grammars\nThe simplest augmentation of the context-free grammar is the Probabilistic Context-\nFree Grammar (PCFG ), also known as the Stochastic Context-Free Grammar PCFG\n(SCFG ), \ufb01rst proposed by Booth (1969). Recall that a context-free grammar Gis SCFG\nde\ufb01ned by four parameters ( N;S;R;S); a probabilistic context-free grammar is also\nde\ufb01ned by four parameters, with a slight augmentation to each of the rules in R: 2APPENDIX C \u2022 S TATISTICAL CONSTITUENCY PARSING\nNa set of non-terminal symbols (orvariables )\nSa set of terminal symbols (disjoint from N)\nRa set of rules or productions, each of the form A!b[p],\nwhere Ais a non-terminal,\nbis a string of symbols from the in\ufb01nite set of strings (S[N)\u0003,\nandpis a number between 0 and 1 expressing P(bjA)\nSa designated start symbol\nThat is, a PCFG differs from a standard CFG by augmenting each rule in Rwith\na conditional probability:\nA!b[p] (C.1)\nHere pexpresses the probability that the given non-terminal Awill be expanded\nto the sequence b. That is, pis the conditional probability of a given expansion b\ngiven the left-hand-side (LHS) non-terminal A. We can represent this probability as\nP(A!b)\nor as\nP(A!bjA)\nor as\nP(RHSjLHS)\nThus, if we consider all the possible expansions of a non-terminal, the sum of their\nprobabilities must be 1:X\nbP(A!b) =1\nFigure C.1 shows a PCFG: a probabilistic augmentation of the L1miniature En-\nglish CFG grammar and lexicon. Note that the probabilities of all of the expansions\nof each non-terminal sum to 1. Also note that these probabilities were made up\nfor pedagogical purposes. A real grammar has a great many more rules for each\nnon-terminal; hence, the probabilities of any particular rule would tend to be much\nsmaller.\nA PCFG is said to be consistent if the sum of the probabilities of all sentences consistent\nin the language equals 1. Certain kinds of recursive rules cause a grammar to be\ninconsistent by causing in\ufb01nitely looping derivations for some sentences. For ex-\nample, a rule S!Swith probability 1 would lead to lost probability mass due to\nderivations that never terminate. See Booth and Thompson (1973) for more details\non consistent and inconsistent grammars.\nHow are PCFGs used? A PCFG can be used to estimate a number of useful\nprobabilities concerning a sentence and its parse tree(s), including the probability of\na particular parse tree (useful in disambiguation) and the probability of a sentence\nor a piece of a sentence (useful in language modeling). Let\u2019s see how this works.\nC.1.1 PCFGs for Disambiguation\nA PCFG assigns a probability to each parse tree T(i.e., each derivation ) of a sen-\ntence S. This attribute is useful in disambiguation . For example, consider the two\nparses of the sentence \u201cBook the dinner \ufb02ight\u201d shown in Fig. C.2. The sensible parse C.1 \u2022 P ROBABILISTIC CONTEXT -FREE GRAMMARS 3\nGrammar Lexicon\nS!NP VP [:80] Det!that[:10]ja[:30]jthe[:60]\nS!Aux NP VP [:15] Noun!book[:10]jtrip[:30]\nS!VP [:05]jmeal[:05]jmoney [:05]\nNP!Pronoun [:35]j\ufb02ight [:40]jdinner [:10]\nNP!Proper-Noun [:30] Verb!book[:30]jinclude [:30]\nNP!Det Nominal [:20]jprefer [:40]\nNP!Nominal [:15] Pronoun!I[:40]jshe[:05]\nNominal!Noun [:75]jme[:15]jyou[:40]\nNominal!Nominal Noun [:20] Proper-Noun!Houston [:60]\nNominal!Nominal PP [:05]jNWA [:40]\nVP!Verb [:35] Aux!does[:60]jcan[:40]\nVP!Verb NP [:20] Preposition!from[:30]jto[:30]\nVP!Verb NP PP [:10]jon[:20]jnear[:15]\nVP!Verb PP [:15]jthrough [:05]\nVP!Verb NP NP [:05]\nVP!VP PP [:15]\nPP!Preposition NP [1:0]\nFigure C.1 A PCFG that is a probabilistic augmentation of the L1miniature English CFG\ngrammar and lexicon of Fig. ??. These probabilities were made up for pedagogical purposes\nand are not based on a corpus (any real corpus would have many more rules, so the true\nprobabilities of each rule would be much smaller).\non the left means \u201cBook a \ufb02ight that serves dinner\u201d. The nonsensical parse on the\nright, however, would have to mean something like \u201cBook a \ufb02ight on behalf of \u2018the\ndinner\u201d\u2019 just as a structurally similar sentence like \u201cCan you book John a \ufb02ight?\u201d\nmeans something like \u201cCan you book a \ufb02ight on behalf of John?\u201d\nThe probability of a particular parse Tis de\ufb01ned as the product of the probabil-\nities of all the nrules used to expand each of the nnon-terminal nodes in the parse\ntreeT, where each rule ican be expressed as LHS i!RHS i:\nP(T;S) =nY\ni=1P(RHS ijLHS i) (C.2)\nThe resulting probability P(T;S)is both the joint probability of the parse and the\nsentence and also the probability of the parse P(T). How can this be true? First, by\nthe de\ufb01nition of joint probability:\nP(T;S) =P(T)P(SjT) (C.3)\nBut since a parse tree includes all the words of the sentence, P(SjT)is 1. Thus,\nP(T;S) =P(T)P(SjT) =P(T) (C.4)\nWe can compute the probability of each of the trees in Fig. C.2 by multiplying the\nprobabilities of each of the rules used in the derivation. For example, the probability\nof the left tree in Fig. C.2a (call it Tle ft) and the right tree (Fig. C.2b or Tright) can be\ncomputed as follows:\nP(Tle ft) = :05\u0003:20\u0003:20\u0003:20\u0003:75\u0003:30\u0003:60\u0003:10\u0003:40=2:2\u000210\u00006\nP(Tright) = :05\u0003:10\u0003:20\u0003:15\u0003:75\u0003:75\u0003:30\u0003:60\u0003:10\u0003:40=6:1\u000210\u00007 4APPENDIX C \u2022 S TATISTICAL CONSTITUENCY PARSING\nS\nVP\nNP\nNominal\nNoun\n\ufb02ightNominal\nNoun\ndinnerDet\ntheVerb\nBookS\nVP\nNP\nNominal\nNoun\n\ufb02ightNP\nNominal\nNoun\ndinnerDet\ntheVerb\nBook\nRules P Rules P\nS!VP .05 S !VP .05\nVP!Verb NP .20 VP !Verb NP NP .10\nNP!Det Nominal .20 NP !Det Nominal .20\nNominal!Nominal Noun .20 NP !Nominal .15\nNominal!Noun .75 Nominal !Noun .75\nNominal!Noun .75\nVerb!book .30 Verb !book .30\nDet!the .60 Det !the .60\nNoun!dinner .10 Noun !dinner .10\nNoun!\ufb02ight .40 Noun !\ufb02ight .40\nFigure C.2 Two parse trees for an ambiguous sentence. The parse on the left corresponds\nto the sensible meaning \u201cBook a \ufb02ight that serves dinner\u201d, while the parse on the right corre-\nsponds to the nonsensical meaning \u201cBook a \ufb02ight on behalf of \u2018the dinner\u2019 \u201d.\nWe can see that the left tree in Fig. C.2 has a much higher probability than the\ntree on the right. Thus, this parse would correctly be chosen by a disambiguation\nalgorithm that selects the parse with the highest PCFG probability.\nLet\u2019s formalize this intuition that picking the parse with the highest probability\nis the correct way to do disambiguation. Consider all the possible parse trees for a\ngiven sentence S. The string of words Sis called the yield of any parse tree over S. yield\nThus, out of all parse trees with a yield of S, the disambiguation algorithm picks the\nparse tree that is most probable given S:\n\u02c6T(S) = argmax\nT s:t:S=yield (T)P(TjS) (C.5)\nBy de\ufb01nition, the probability P(TjS)can be rewritten as P(T;S)=P(S), thus leading\nto\n\u02c6T(S) = argmax\nT s:t:S=yield (T)P(T;S)\nP(S)(C.6)\nSince we are maximizing over all parse trees for the same sentence, P(S)will be a C.1 \u2022 P ROBABILISTIC CONTEXT -FREE GRAMMARS 5\nconstant for each tree, so we can eliminate it:\n\u02c6T(S) = argmax\nT s:t:S=yield (T)P(T;S) (C.7)\nFurthermore, since we showed above that P(T;S) =P(T), the \ufb01nal equation for\nchoosing the most likely parse neatly simpli\ufb01es to choosing the parse with the high-\nest probability:\n\u02c6T(S) = argmax\nT s:t:S=yield (T)P(T) (C.8)\nC.1.2 PCFGs for Language Modeling\nA second attribute of a PCFG is that it assigns a probability to the string of words\nconstituting a sentence. This is important in language modeling , whether for use\nin speech recognition, machine translation, spelling correction, augmentative com-\nmunication, or other applications. The probability of an unambiguous sentence is\nP(T;S) =P(T)or just the probability of the single parse tree for that sentence. The\nprobability of an ambiguous sentence is the sum of the probabilities of all the parse\ntrees for the sentence:\nP(S) =X\nT s:t:S=yield (T)P(T;S) (C.9)\n=X\nT s:t:S=yield (T)P(T) (C.10)\nAn additional feature of PCFGs that is useful for language modeling is their ability\nto assign a probability to substrings of a sentence. For example, suppose we want\nto know the probability of the next word wiin a sentence given all the words we\u2019ve\nseen so far w1;:::;wi\u00001. The general formula for this is\nP(wijw1;w2;:::;wi\u00001) =P(w1;w2;:::;wi\u00001;wi)\nP(w1;w2;:::;wi\u00001)(C.11)\nWe saw in Chapter 3 a simple approximation of this probability using N-grams,\nconditioning on only the last word or two instead of the entire context; thus, the\nbigram approximation would give us\nP(wijw1;w2;:::;wi\u00001)\u0019P(wi\u00001;wi)\nP(wi\u00001)(C.12)\nBut the fact that the N-gram model can only make use of a couple words of context\nmeans it is ignoring potentially useful prediction cues. Consider predicting the word\nafter in the following sentence from Chelba and Jelinek (2000):\n(C.13) the contract ended with a loss of 7 cents after trading as low as 9 cents\nA trigram grammar must predict after from the words 7 cents , while it seems clear\nthat the verb ended and the subject contract would be useful predictors that a PCFG-\nbased parser could help us make use of. Indeed, it turns out that PCFGs allow us to\ncondition on the entire previous context w1;w2;:::;wi\u00001shown in Eq. C.11.\nIn summary, PCFGs can be applied both to disambiguation in syntactic parsing\nand to word prediction in language modeling. Both of these applications require that\nwe be able to compute the probability of parse tree Tfor a given sentence S. The\nnext few sections introduce some algorithms for computing this probability. 6APPENDIX C \u2022 S TATISTICAL CONSTITUENCY PARSING\nC.2 Probabilistic CKY Parsing of PCFGs\nThe parsing problem for PCFGs is to produce the most-likely parse \u02c6Tfor a given\nsentence S, that is,\n\u02c6T(S) = argmax\nT s:t:S=yield (T)P(T) (C.14)\nThe algorithms for computing the most likely parse are simple extensions of the\nstandard algorithms for parsing; most modern probabilistic parsers are based on the\nprobabilistic CKY algorithm, \ufb01rst described by Ney (1991). The probabilistic CKYprobabilistic\nCKY\nalgorithm assumes the PCFG is in Chomsky normal form. Recall from page ??that\nin CNF, the right-hand side of each rule must expand to either two non-terminals or\nto a single terminal, i.e., rules have the form A!B C, orA!w.\nFor the CKY algorithm, we represented each sentence as having indices between\nthe words. Thus, an example sentence like\n(C.15) Book the \ufb02ight through Houston.\nwould assume the following indices between each word:\n(C.16) 0\rBook 1\rthe 2\r\ufb02ight 3\rthrough 4\rHouston 5\r\nUsing these indices, each constituent in the CKY parse tree is encoded in a\ntwo-dimensional matrix. Speci\ufb01cally, for a sentence of length nand a grammar\nthat contains Vnon-terminals, we use the upper-triangular portion of an (n+1)\u0002\n(n+1)matrix. For CKY , each cell table[i;j]contained a list of constituents that\ncould span the sequence of words from itoj. For probabilistic CKY , it\u2019s slightly\nsimpler to think of the constituents in each cell as constituting a third dimension of\nmaximum length V. This third dimension corresponds to each non-terminal that can\nbe placed in this cell, and the value of the cell is then a probability for that non-\nterminal/constituent rather than a list of constituents. In summary, each cell [i;j;A]\nin this (n+1)\u0002(n+1)\u0002Vmatrix is the probability of a constituent of type Athat\nspans positions ithrough jof the input.\nFigure C.3 gives the probabilistic CKY algorithm.\nfunction PROBABILISTIC -CKY( words,grammar )returns most probable parse\nand its probability\nforj from 1toLENGTH (words )do\nfor allfAjA!words [j]2grammarg\ntable [j\u00001;j;A] P(A!words [j])\nfori from j\u00002downto 0do\nfork i+1toj\u00001do\nfor allfAjA!BC2grammar ;\nandtable[i;k;B]>0andtable[k;j;C]>0g\nif(table [i,j,A]<P(A!BC)\u0002table [i,k,B]\u0002table [k,j,C])then\ntable [i,j,A] P(A!BC)\u0002table [i,k,B]\u0002table [k,j,C]\nback[i,j,A] fk;B;Cg\nreturn BUILD TREE (back[1, L ENGTH (words ),S]),table [1, L ENGTH (words ),S]\nFigure C.3 The probabilistic CKY algorithm for \ufb01nding the maximum probability parse\nof a string of num words words given a PCFG grammar with num rules rules in Chomsky\nnormal form. back is an array of backpointers used to recover the best parse. The build tree\nfunction is left as an exercise to the reader. C.2 \u2022 P ROBABILISTIC CKY P ARSING OF PCFG S7\nLike the basic CKY algorithm in Fig. ??, the probabilistic CKY algorithm re-\nquires a grammar in Chomsky normal form. Converting a probabilistic grammar to\nCNF requires that we also modify the probabilities so that the probability of each\nparse remains the same under the new CNF grammar. Exercise C.2 asks you to mod-\nify the algorithm for conversion to CNF in Chapter 18 so that it correctly handles\nrule probabilities.\nIn practice, a generalized CKY algorithm that handles unit productions directly\nis typically used. Recall that Exercise 13.3 asked you to make this change in CKY;\nExercise C.3 asks you to extend this change to probabilistic CKY .\nLet\u2019s see an example of the probabilistic CKY chart, using the following mini-\ngrammar, which is already in CNF:\nS!NP VP .80 Det!the .40\nNP!Det N .30 Det!a .40\nV P!V NP .20 N!meal .01\nV!includes .05 N!f light .02\nGiven this grammar, Fig. C.4 shows the \ufb01rst steps in the probabilistic CKY parse\nof the sentence \u201cThe \ufb02ight includes a meal\u201d.\nTheflight[0,1][0,2][0,3][1,2][1,3][2,3]Det: .40includesameal\n[3,4][4,5]N: .02V: .05NP: .30 *.40 *.02= .0024[0,4][1,4][2,4][3,5][2,5][1,5][0,5]\nDet: .40N: .01\nFigure C.4 The beginning of the probabilistic CKY matrix. Filling out the rest of the chart\nis left as Exercise C.4 for the reader. 8APPENDIX C \u2022 S TATISTICAL CONSTITUENCY PARSING\nC.3 Ways to Learn PCFG Rule Probabilities\nWhere do PCFG rule probabilities come from? There are two ways to learn proba-\nbilities for the rules of a grammar. The simplest way is to use a treebank, a corpus\nof already parsed sentences. Recall that we introduced in Appendix D the idea\nof treebanks and the commonly used Penn Treebank , a collection of parse trees\nin English, Chinese, and other languages that is distributed by the Linguistic Data\nConsortium. Given a treebank, we can compute the probability of each expansion\nof a non-terminal by counting the number of times that expansion occurs and then\nnormalizing.\nP(a!bja) =Count (a!b)P\ngCount (a!g)=Count (a!b)\nCount (a)(C.17)\nIf we don\u2019t have a treebank but we do have a (non-probabilistic) parser, we can\ngenerate the counts we need for computing PCFG rule probabilities by \ufb01rst parsing\na corpus of sentences with the parser. If sentences were unambiguous, it would be\nas simple as this: parse the corpus, increment a counter for every rule in the parse,\nand then normalize to get probabilities.\nBut wait! Since most sentences are ambiguous, that is, have multiple parses, we\ndon\u2019t know which parse to count the rules in. Instead, we need to keep a separate\ncount for each parse of a sentence and weight each of these partial counts by the\nprobability of the parse it appears in. But to get these parse probabilities to weight\nthe rules, we need to already have a probabilistic parser.\nThe intuition for solving this chicken-and-egg problem is to incrementally im-\nprove our estimates by beginning with a parser with equal rule probabilities, then\nparse the sentence, compute a probability for each parse, use these probabilities to\nweight the counts, re-estimate the rule probabilities, and so on, until our proba-\nbilities converge. The standard algorithm for computing this solution is called the\ninside-outside algorithm; it was proposed by Baker (1979) as a generalization of the inside-outside\nforward-backward algorithm for HMMs. Like forward-backward, inside-outside is\na special case of the Expectation Maximization (EM) algorithm, and hence has two\nsteps: the expectation step , and the maximization step . See Lari and Young (1990)\nor Manning and Sch \u00a8utze (1999) for more on the algorithm.\nC.4 Problems with PCFGs\nWhile probabilistic context-free grammars are a natural extension to context-free\ngrammars, they have two main problems as probability estimators:\nPoor independence assumptions: CFG rules impose an independence assumption\non probabilities that leads to poor modeling of structural dependencies across\nthe parse tree.\nLack of lexical conditioning: CFG rules don\u2019t model syntactic facts about speci\ufb01c\nwords, leading to problems with subcategorization ambiguities, preposition\nattachment, and coordinate structure ambiguities.\nBecause of these problems, probabilistic constituent parsing models use some\naugmented version of PCFGs, or modify the Treebank-based grammar in some way. C.4 \u2022 P ROBLEMS WITH PCFG S9\nIn the next few sections after discussing the problems in more detail we introduce\nsome of these augmentations.\nC.4.1 Independence Assumptions Miss Rule Dependencies\nLet\u2019s look at these problems in more detail. Recall that in a CFG the expansion\nof a non-terminal is independent of the context, that is, of the other nearby non-\nterminals in the parse tree. Similarly, in a PCFG, the probability of a particular\nrule like NP!Det N is also independent of the rest of the tree. By de\ufb01nition, the\nprobability of a group of independent events is the product of their probabilities.\nThese two facts explain why in a PCFG we compute the probability of a tree by just\nmultiplying the probabilities of each non-terminal expansion.\nUnfortunately, this CFG independence assumption results in poor probability\nestimates. This is because in English the choice of how a node expands can after all\ndepend on the location of the node in the parse tree. For example, in English it turns\nout that NPs that are syntactic subjects are far more likely to be pronouns, and NPs\nthat are syntactic objects are far more likely to be non-pronominal (e.g., a proper\nnoun or a determiner noun sequence), as shown by these statistics for NPs in the\nSwitchboard corpus (Francis et al., 1999):1\nPronoun Non-Pronoun\nSubject 91% 9%\nObject 34% 66%\nUnfortunately, there is no way to represent this contextual difference in the prob-\nabilities of a PCFG. Consider two expansions of the non-terminal NPas a pronoun\nor as a determiner+noun. How shall we set the probabilities of these two rules? If\nwe set their probabilities to their overall probability in the Switchboard corpus, the\ntwo rules have about equal probability.\nNP!DT NN :28\nNP!PRP :25\nBecause PCFGs don\u2019t allow a rule probability to be conditioned on surrounding\ncontext, this equal probability is all we get; there is no way to capture the fact that in\nsubject position, the probability for NP!PRP should go up to .91, while in object\nposition, the probability for NP!DT NN should go up to .66.\nThese dependencies could be captured if the probability of expanding an NPas\na pronoun (e.g., NP!PRP) versus a lexical NP(e.g., NP!DT NN ) were con-\nditioned on whether the NPwas a subject or an object. Section C.5 introduces the\ntechnique of parent annotation for adding this kind of conditioning.\nC.4.2 Lack of Sensitivity to Lexical Dependencies\nA second class of problems with PCFGs is their lack of sensitivity to the words in\nthe parse tree. Words do play a role in PCFGs since the parse probability includes\nthe probability of a word given a part-of-speech (e.g., from rules like V!sleep ,\nNN!book , etc.).\n1Distribution of subjects from 31,021 declarative sentences; distribution of objects from 7,489 sen-\ntences. This tendency is caused by the use of subject position to realize the topic or old information\nin a sentence (Giv \u00b4on, 1990). Pronouns are a way to talk about old information, while non-pronominal\n(\u201clexical\u201d) noun-phrases are often used to introduce new referents (Chapter 23). 10 APPENDIX C \u2022 S TATISTICAL CONSTITUENCY PARSING\nBut it turns out that lexical information is useful in other places in the grammar,\nsuch as in resolving prepositional phrase ( PP) attachment ambiguities. Since prepo-\nsitional phrases in English can modify a noun phrase or a verb phrase, when a parser\n\ufb01nds a prepositional phrase, it must decide where to attach it in the tree. Consider\nthe following example:\n(C.18) Workers dumped sacks into a bin.\nFigure C.5 shows two possible parse trees for this sentence; the one on the left is\nthe correct parse; Fig. C.6 shows another perspective on the preposition attachment\nproblem, demonstrating that resolving the ambiguity in Fig. C.5 is equivalent to\ndeciding whether to attach the prepositional phrase into the rest of the tree at the\nNPorVPnodes; we say that the correct parse requires VP attachment , and the VP attachment\nincorrect parse implies NP attachment . NP attachment\nS\nVP\nPP\nNP\nNN\nbinDT\naP\nintoNP\nNNS\nsacksVBD\ndumpedNP\nNNS\nworkersS\nVP\nNP\nPP\nNP\nNN\nbinDT\naP\nintoNP\nNNS\nsacksVBD\ndumpedNP\nNNS\nworkers\nFigure C.5 Two possible parse trees for a prepositional phrase attachment ambiguity . The left parse is the\nsensible one, in which \u201cinto a bin\u201d describes the resulting location of the sacks. In the right incorrect parse, the\nsacks to be dumped are the ones which are already \u201cinto a bin\u201d, whatever that might mean.\nWhy doesn\u2019t a PCFG already deal with PPattachment ambiguities? Note that\nthe two parse trees in Fig. C.5 have almost exactly the same rules; they differ only\nin that the left-hand parse has this rule:\nV P!V BD NP PP\nwhile the right-hand parse has these:\nV P!V BD NP\nNP!NP PP\nDepending on how these probabilities are set, a PCFG will always either prefer\nNPattachment or VPattachment. As it happens, NPattachment is slightly more\ncommon in English, so if we trained these rule probabilities on a corpus, we might\nalways prefer NPattachment, causing us to misparse this sentence.\nBut suppose we set the probabilities to prefer the VPattachment for this sen-\ntence. Now we would misparse the following, which requires NPattachment: C.5 \u2022 I MPROVING PCFG S BY SPLITTING NON-TERMINALS 11\nS\nVP\nNP\nNNS\nsacksVBD\ndumpedNP\nNNS\nworkersPP\nNP\nNN\nbinDT\naP\ninto\nFigure C.6 Another view of the preposition attachment problem. Should the PPon the right attach to the VP\norNPnodes of the partial parse tree on the left?\n(C.19) \ufb01shermen caught tons of herring\nWhat information in the input sentence lets us know that (C.19) requires NP\nattachment while (C.18) requires VPattachment? These preferences come from\nthe identities of the verbs, nouns, and prepositions. The af\ufb01nity between the verb\ndumped and the preposition intois greater than the af\ufb01nity between the noun sacks\nand the preposition into, thus leading to VPattachment. On the other hand, in (C.19)\nthe af\ufb01nity between tons andofis greater than that between caught andof, leading to\nNPattachment. Thus, to get the correct parse for these kinds of examples, we need\na model that somehow augments the PCFG probabilities to deal with these lexical\ndependency statistics for different verbs and prepositions.lexical\ndependency\nCoordination ambiguities are another case in which lexical dependencies are\nthe key to choosing the proper parse. Figure C.7 shows an example from Collins\n(1999) with two parses for the phrase dogs in houses and cats . Because dogs is\nsemantically a better conjunct for catsthan houses (and because most dogs can\u2019t \ufb01t\ninside cats), the parse [dogs in [ NPhouses and cats]] is intuitively unnatural and\nshould be dispreferred. The two parses in Fig. C.7, however, have exactly the same\nPCFG rules, and thus a PCFG will assign them the same probability.\nIn summary, we have shown in this section and the previous one that probabilistic\ncontext-free grammars are incapable of modeling important structural andlexical\ndependencies. In the next two sections we sketch current methods for augmenting\nPCFGs to deal with both these issues.\nC.5 Improving PCFGs by Splitting Non-Terminals\nLet\u2019s start with the \ufb01rst of the two problems with PCFGs mentioned above: their\ninability to model structural dependencies, like the fact that NPs in subject position\ntend to be pronouns, whereas NPs in object position tend to have full lexical (non-\npronominal) form. How could we augment a PCFG to correctly model this fact?\nOne idea would be to split theNPnon-terminal into two versions: one for sub- split\njects, one for objects. Having two nodes (e.g., NPsubject andNPobject ) would allow\nus to correctly model their different distributional properties, since we would have 12 APPENDIX C \u2022 S TATISTICAL CONSTITUENCY PARSING\nNP\nNP\nNoun\ncatsConj\nandNP\nPP\nNP\nNoun\nhousesPrep\ninNP\nNoun\ndogsNP\nPP\nNP\nNP\nNoun\ncatsConj\nandNP\nNoun\nhousesPrep\ninNP\nNoun\ndogs\nFigure C.7 An instance of coordination ambiguity. Although the left structure is intuitively\nthe correct one, a PCFG will assign them identical probabilities since both structures use\nexactly the same set of rules. After Collins (1999).\ndifferent probabilities for the rule NPsubject!PRP and the rule NPobject!PRP.\nOne way to implement this intuition of splits is to do parent annotation (John-parent\nannotation\nson, 1998), in which we annotate each node with its parent in the parse tree. Thus,\nanNPnode that is the subject of the sentence and hence has parent Swould be anno-\ntated NP\u02c6S, while a direct object NPwhose parent is VPwould be annotated NP\u02c6VP.\nFigure C.8 shows an example of a tree produced by a grammar that parent-annotates\nthe phrasal non-terminals (like NPandVP).\na) S\nVP\nNP\nNN\n\ufb02ightDT\naVBD\nneedNP\nPRP\nIb) S\nVP\u02c6S\nNP\u02c6VP\nNN\n\ufb02ightDT\naVBD\nneedNP\u02c6S\nPRP\nI\nFigure C.8 A standard PCFG parse tree (a) and one which has parent annotation on the\nnodes which aren\u2019t pre-terminal (b). All the non-terminal nodes (except the pre-terminal\npart-of-speech nodes) in parse (b) have been annotated with the identity of their parent.\nIn addition to splitting these phrasal nodes, we can also improve a PCFG by\nsplitting the pre-terminal part-of-speech nodes (Klein and Manning, 2003b). For ex-\nample, different kinds of adverbs (RB) tend to occur in different syntactic positions:\nthe most common adverbs with ADVP parents are also andnow, with VPparents\nn\u2019tandnot, and with NPparents only andjust. Thus, adding tags like RB\u02c6ADVP,\nRB\u02c6VP, and RB\u02c6NP can be useful in improving PCFG modeling.\nSimilarly, the Penn Treebank tag IN can mark a wide variety of parts-of-speech,\nincluding subordinating conjunctions ( while ,as,if), complementizers ( that,for),\nand prepositions ( of,in,from). Some of these differences can be captured by parent C.6 \u2022 P ROBABILISTIC LEXICALIZED CFG S13\nannotation (subordinating conjunctions occur under S, prepositions under PP), while\nothers require splitting the pre-terminal nodes. Figure C.9 shows an example from\nKlein and Manning (2003b) in which even a parent-annotated grammar incorrectly\nparses works as a noun in to see if advertising works . Splitting pre-terminals to allow\nifto prefer a sentential complement results in the correct verbal parse.\nNode-splitting is not without problems; it increases the size of the grammar and\nhence reduces the amount of training data available for each grammar rule, leading\nto over\ufb01tting. Thus, it is important to split to just the correct level of granularity for a\nparticular training set. While early models employed handwritten rules to try to \ufb01nd\nan optimal number of non-terminals (Klein and Manning, 2003b), modern models\nautomatically search for the optimal splits. The split and merge algorithm of Petrov split and merge\net al. (2006), for example, starts with a simple X-bar grammar, alternately splits the\nnon-terminals, and merges non-terminals, \ufb01nding the set of annotated nodes that\nmaximizes the likelihood of the training set treebank.\nC.6 Probabilistic Lexicalized CFGs\nThe previous section showed that a simple probabilistic CKY algorithm for pars-\ning raw PCFGs can achieve extremely high parsing accuracy if the grammar rule\nsymbols are redesigned by automatic splits and merges.\nIn this section, we discuss an alternative family of models in which instead of\nmodifying the grammar rules, we modify the probabilistic model of the parser to\nallow for lexicalized rules. The resulting family of lexicalized parsers includes the\nCollins parser (Collins, 1999) and the Charniak parser (Charniak, 1997).\nWe saw in Section ??that syntactic constituents could be associated with a lexi-\ncalhead , and we de\ufb01ned a lexicalized grammar in which each non-terminal in thelexicalized\ngrammar\ntree is annotated with its lexical head, where a rule like V P!V BD NP PP would\nbe extended as\nVP(dumped)!VBD(dumped) NP(sacks) PP(into) (C.20)\nVP\u02c6S\nVP\u02c6VP\nPP\u02c6VP\nNP\u02c6PP\nNNS\nworksNN\nadvertisingIN\nifVB\nseeTO\ntoVP\u02c6S\nVP\u02c6VP\nSBAR\u02c6VP\nS\u02c6SBAR\nVP\u02c6S\nVBZ\u02c6VP\nworksNP\u02c6S\nNN\u02c6NP\nadvertisingIN\u02c6SBAR\nifVB\u02c6VP\nseeTO\u02c6VP\nto\nFigure C.9 An incorrect parse even with a parent-annotated parse (left). The correct parse (right), was pro-\nduced by a grammar in which the pre-terminal nodes have been split, allowing the probabilistic grammar to\ncapture the fact that ifprefers sentential complements. Adapted from Klein and Manning (2003b). 14 APPENDIX C \u2022 S TATISTICAL CONSTITUENCY PARSING\nIn the standard type of lexicalized grammar, we actually make a further exten-\nsion, which is to associate the head tag , the part-of-speech tags of the headwords, head tag\nwith the non-terminal symbols as well. Each rule is thus lexicalized by both the\nheadword and the head tag of each constituent resulting in a format for lexicalized\nrules like\nVP(dumped,VBD) !VBD(dumped,VBD) NP(sacks,NNS) PP(into,P) (C.21)\nWe show a lexicalized parse tree with head tags in Fig. C.10, extended from Fig. ??.\nTOP\nS(dumped,VBD)\nVP(dumped,VBD)\nPP(into,P)\nNP(bin,NN)\nNN(bin,NN)\nbinDT(a,DT)\naP(into,P)\nintoNP(sacks,NNS)\nNNS(sacks,NNS)\nsacksVBD(dumped,VBD)\ndumpedNP(workers,NNS)\nNNS(workers,NNS)\nworkers\nInternal Rules Lexical Rules\nTOP !S(dumped,VBD) NNS(workers,NNS) !workers\nS(dumped,VBD) !NP(workers,NNS) VP(dumped,VBD) VBD(dumped,VBD) !dumped\nNP(workers,NNS) !NNS(workers,NNS) NNS(sacks,NNS) !sacks\nVP(dumped,VBD) !VBD(dumped, VBD) NP(sacks,NNS) PP(into,P) P(into,P) !into\nPP(into,P) !P(into,P) NP(bin,NN) DT(a,DT) !a\nNP(bin,NN) !DT(a,DT) NN(bin,NN) NN(bin,NN) !bin\nFigure C.10 A lexicalized tree, including head tags, for a WSJ sentence, adapted from Collins (1999). Below\nwe show the PCFG rules needed for this parse tree, internal rules on the left, and lexical rules on the right.\nTo generate such a lexicalized tree, each PCFG rule must be augmented to iden-\ntify one right-hand constituent to be the head daughter. The headword for a node is\nthen set to the headword of its head daughter, and the head tag to the part-of-speech\ntag of the headword. Recall that we gave in Fig. ??a set of handwritten rules for\nidentifying the heads of particular constituents.\nA natural way to think of a lexicalized grammar is as a parent annotation, that\nis, as a simple context-free grammar with many copies of each rule, one copy for\neach possible headword/head tag for each constituent. Thinking of a probabilistic\nlexicalized CFG in this way would lead to the set of simple PCFG rules shown below\nthe tree in Fig. C.10.\nNote that Fig. C.10 shows two kinds of rules: lexical rules , which express the lexical rules\nexpansion of a pre-terminal to a word, and internal rules , which express the other internal rules\nrule expansions. We need to distinguish these kinds of rules in a lexicalized grammar\nbecause they are associated with very different kinds of probabilities. The lexical\nrules are deterministic, that is, they have probability 1.0 since a lexicalized pre- C.6 \u2022 P ROBABILISTIC LEXICALIZED CFG S15\nterminal like NN(bin;NN)can only expand to the word bin. But for the internal\nrules, we need to estimate probabilities.\nSuppose we were to treat a probabilistic lexicalized CFG like a really big CFG\nthat just happened to have lots of very complex non-terminals and estimate the\nprobabilities for each rule from maximum likelihood estimates. Thus, according\nto Eq. C.17, the MLE estimate for the probability for the rule P(VP(dumped,VBD)\n!VBD(dumped, VBD) NP(sacks,NNS) PP(into,P)) would be\nCount(VP(dumped,VBD) !VBD(dumped, VBD) NP(sacks,NNS) PP(into,P))\nCount(VP(dumped,VBD))(C.22)\nBut there\u2019s no way we can get good estimates of counts like those in (C.22) because\nthey are so speci\ufb01c: we\u2019re unlikely to see many (or even any) instances of a sentence\nwith a verb phrase headed by dumped that has one NPargument headed by sacks\nand a PPargument headed by into. In other words, counts of fully lexicalized PCFG\nrules like this will be far too sparse, and most rule probabilities will come out 0.\nThe idea of lexicalized parsing is to make some further independence assump-\ntions to break down each rule so that we would estimate the probability\nP(VP(dumped,VBD) !VBD(dumped, VBD) NP(sacks,NNS) PP(into,P))\nas the product of smaller independent probability estimates for which we could ac-\nquire reasonable counts. The next section summarizes one such method, the Collins\nparsing method.\nC.6.1 The Collins Parser\nStatistical parsers differ in exactly which independence assumptions they make.\nLet\u2019s look at the assumptions in a simpli\ufb01ed version of the Collins parser. The \ufb01rst\nintuition of the Collins parser is to think of the right-hand side of every (internal)\nCFG rule as consisting of a head non-terminal, together with the non-terminals to\nthe left of the head and the non-terminals to the right of the head. In the abstract, we\nthink about these rules as follows:\nLHS!LnLn\u00001:::L1H R 1:::Rn\u00001Rn (C.23)\nSince this is a lexicalized grammar, each of the symbols like L1orR3orHorLHS\nis actually a complex symbol representing the category and its head and head tag,\nlikeVP(dumped,VP) orNP(sacks,NNS) .\nNow, instead of computing a single MLE probability for this rule, we are going\nto break down this rule via a neat generative story, a slight simpli\ufb01cation of what is\ncalled Collins Model 1. This new generative story is that given the left-hand side,\nwe \ufb01rst generate the head of the rule and then generate the dependents of the head,\none by one, from the inside out. Each of these steps will have its own probability.\nWe also add a special STOP non-terminal at the left and right edges of the rule;\nthis non-terminal allows the model to know when to stop generating dependents on a\ngiven side. We generate dependents on the left side of the head until we\u2019ve generated\nSTOP on the left side of the head, at which point we move to the right side of the\nhead and start generating dependents there until we generate STOP . So it\u2019s as if we\nare generating a rule augmented as follows:\nP(VP(dumped,VBD) ! (C.24)\nSTOP VBD(dumped, VBD) NP(sacks,NNS) PP(into,P) STOP ) 16 APPENDIX C \u2022 S TATISTICAL CONSTITUENCY PARSING\nLet\u2019s see the generative story for this augmented rule. We make use of three kinds\nof probabilities: PHfor generating heads, PLfor generating dependents on the left,\nandPRfor generating dependents on the right.\n1. Generate the head VBD(dumped,VBD) with probability\nPH(HjLHS) = P H(VBD(dumped,VBD) jVP(dumped,VBD))VP(dumped,VBD)\nVBD(dumped,VBD)\n2. Generate the left dependent (which is STOP, since there isn\u2019t\none) with probability\nPL(STOPjVP(dumped,VBD) VBD(dumped,VBD))VP(dumped,VBD)\nVBD(dumped,VBD) STOP\n3. Generate right dependent NP(sacks,NNS) with probability\nPR(NP(sacks,NNS)jVP(dumped,VBD), VBD(dumped,VBD))VP(dumped,VBD)\nNP(sacks,NNS) VBD(dumped,VBD) STOP\n4. Generate the right dependent PP(into,P) with probability\nPR(PP(into,P)jVP(dumped,VBD), VBD(dumped,VBD))VP(dumped,VBD)\nPP(into,P) NP(sacks,NNS) VBD(dumped,VBD) STOP\n5) Generate the right dependent STOP with probability\nPR(STOPjVP(dumped,VBD), VBD(dumped,VBD))VP(dumped,VBD)\nSTOP PP(into,P) NP(sacks,NNS) VBD(dumped,VBD) STOP\nIn summary, the probability of this rule\nP(VP(dumped,VBD) ! (C.25)\nVBD(dumped, VBD) NP(sacks,NNS) PP(into,P) )\nis estimated (simplifying the notation a bit from the steps above):\nPH(VBDjVP , dumped)\u0002PL(STOPjVP;VBD;dumped ) (C.26)\n\u0002PR(NP(sacks,NNS)jVP;VBD;dumped )\n\u0002PR(PP(into,P)jVP;VBD;dumped )\n\u0002PR(STOPjVP;VBD;dumped )\nEach of these probabilities can be estimated from much smaller amounts of data\nthan the full probability in (C.25). For example, the maximum likelihood estimate\nfor the component probability PR(NP(sacks,NNS)jVP;VBD;dumped )is\nCount (VP(dumped,VBD) with NNS(sacks) as a daughter somewhere on the right )\nCount (VP(dumped,VBD) )\n(C.27)\nThese counts are much less subject to sparsity problems than are complex counts\nlike those in (C.25). C.7 \u2022 S UMMARY 17\nMore generally, if His a head with head word hwand head tag ht,lw=ltand\nrw=rtare the word/tag on the left and right respectively, and Pis the parent, then the\nprobability of an entire rule can be expressed as follows:\n1. Generate the head of the phrase H(hw;ht)with probability:\nPH(H(hw;ht)jP;hw;ht)\n2. Generate modi\ufb01ers to the left of the head with total probability\nn+1Y\ni=1PL(Li(lwi;lti)jP;H;hw;ht)\nsuch that Ln+1(lwn+1;ltn+1) = STOP , and we stop generating once we\u2019ve gen-\nerated a STOP token.\n3. Generate modi\ufb01ers to the right of the head with total probability:\nn+1Y\ni=1PR(Ri(rwi;rti)jP;H;hw;ht)\nsuch that Rn+1(rwn+1;rtn+1) = STOP , and we stop generating once we\u2019ve gen-\nerated a STOP token.\nThe parsing algorithm for the Collins model is an extension of probabilistic\nCKY . Extending the CKY algorithm to handle basic lexicalized probabilities is left\nas Exercises 14.5 and 14.6 for the reader.\nC.7 Summary\nThis chapter has sketched the basics of probabilistic parsing, concentrating on\nprobabilistic context-free grammars .\n\u2022 Probabilistic grammars assign a probability to a sentence or string of words\nwhile attempting to capture sophisticated grammatical information.\n\u2022 A probabilistic context-free grammar (PCFG ) is a context-free\ngrammar in which every rule is annotated with the probability of that rule\nbeing chosen. Each PCFG rule is treated as if it were conditionally inde-\npendent ; thus, the probability of a sentence is computed by multiplying the\nprobabilities of each rule in the parse of the sentence.\n\u2022 The probabilistic CKY ( Cocke-Kasami-Younger ) algorithm is a probabilistic\nversion of the CKY parsing algorithm.\n\u2022 PCFG probabilities can be learned by counting in a parsed corpus or by pars-\ning a corpus. The inside-outside algorithm is a way of dealing with the fact\nthat the sentences being parsed are ambiguous.\n\u2022 Raw PCFGs suffer from poor independence assumptions among rules and lack\nof sensitivity to lexical dependencies.\n\u2022 One way to deal with this problem is to split and merge non-terminals (auto-\nmatically or by hand). 18 APPENDIX C \u2022 S TATISTICAL CONSTITUENCY PARSING\n\u2022Probabilistic lexicalized CFG s are another solution to this problem in which\nthe basic PCFG model is augmented with a lexical head for each rule. The\nprobability of a rule can then be conditioned on the lexical head or nearby\nheads.\n\u2022 Parsers for lexicalized PCFGs (like the Collins parser) are based on extensions\nto probabilistic CKY parsing.\nBibliographical and Historical Notes\nMany of the formal properties of probabilistic context-free grammars were \ufb01rst\nworked out by Booth (1969) and Salomaa (1969). Baker (1979) proposed the inside-\noutside algorithm for unsupervised training of PCFG probabilities, and used a CKY-\nstyle parsing algorithm to compute inside probabilities. Jelinek and Lafferty (1991)\nextended the CKY algorithm to compute probabilities for pre\ufb01xes. Stolcke (1995)\nadapted the Earley algorithm to use with PCFGs.\nA number of researchers starting in the early 1990s worked on adding lexical de-\npendencies to PCFGs and on making PCFG rule probabilities more sensitive to sur-\nrounding syntactic structure. For example, Schabes et al. (1988) and Schabes (1990)\npresented early work on the use of heads. Many papers on the use of lexical depen-\ndencies were \ufb01rst presented at the DARPA Speech and Natural Language Workshop\nin June 1990. A paper by Hindle and Rooth (1990) applied lexical dependencies\nto the problem of attaching prepositional phrases; in the question session to a later\npaper, Ken Church suggested applying this method to full parsing (Marcus, 1990).\nEarly work on such probabilistic CFG parsing augmented with probabilistic depen-\ndency information includes Magerman and Marcus (1991), Black et al. (1992), Bod\n(1993), and Jelinek et al. (1994), in addition to Collins (1996), Charniak (1997), and\nCollins (1999) discussed above. Other recent PCFG parsing models include Klein\nand Manning (2003a) and Petrov et al. (2006).\nThis early lexical probabilistic work led initially to work focused on solving\nspeci\ufb01c parsing problems like preposition-phrase attachment by using methods in-\ncluding transformation-based learning (TBL) (Brill and Resnik, 1994), maximum\nentropy (Ratnaparkhi et al., 1994), memory-based learning (Zavrel and Daelemans,\n1997), log-linear models (Franz, 1997), decision trees that used semantic distance\nbetween heads (computed from WordNet) (Stetina and Nagao, 1997), and boosting\n(Abney et al., 1999). Another direction extended the lexical probabilistic parsing\nwork to build probabilistic formulations of grammars other than PCFGs, such as\nprobabilistic TAG grammar (Resnik 1992, Schabes 1992), based on the TAG gram-\nmars discussed in Appendix D, probabilistic LR parsing (Briscoe and Carroll, 1993),\nand probabilistic link grammar (Lafferty et al., 1992). The supertagging approach\nwe saw for CCG was developed for TAG grammars (Bangalore and Joshi 1999,\nJoshi and Srinivas 1994), based on the lexicalized TAG grammars of Schabes et al.\n(1988).\nExercises\nC.1 Implement the CKY algorithm. EXERCISES 19\nC.2 Modify the algorithm for conversion to CNF from Chapter 18 to correctly\nhandle rule probabilities. Make sure that the resulting CNF assigns the same\ntotal probability to each parse tree.\nC.3 Recall that Exercise 13.3 asked you to update the CKY algorithm to han-\ndle unit productions directly rather than converting them to CNF. Extend this\nchange to probabilistic CKY .\nC.4 Fill out the rest of the probabilistic CKY chart in Fig. C.4.\nC.5 Sketch how the CKY algorithm would have to be augmented to handle lexi-\ncalized probabilities.\nC.6 Implement your lexicalized extension of the CKY algorithm. 20 Appendix C \u2022 Statistical Constituency Parsing\nAbney, S. P., R. E. Schapire, and Y . Singer. 1999. Boosting\napplied to tagging and PP attachment. EMNLP/VLC .\nBaker, J. K. 1979. Trainable grammars for speech recogni-\ntion. Speech Communication Papers for the 97th Meeting\nof the Acoustical Society of America .\nBangalore, S. and A. K. Joshi. 1999. Supertagging: An\napproach to almost parsing. Computational Linguistics ,\n25(2):237\u2013265.\nBlack, E., F. Jelinek, J. D. Lafferty, D. M. Magerman, R. L.\nMercer, and S. Roukos. 1992. Towards history-based\ngrammars: Using richer models for probabilistic parsing.\nHLT.\nBod, R. 1993. Using an annotated corpus as a stochastic\ngrammar. EACL .\nBooth, T. L. 1969. Probabilistic representation of formal\nlanguages. IEEE Conference Record of the 1969 Tenth\nAnnual Symposium on Switching and Automata Theory .\nBooth, T. L. and R. A. Thompson. 1973. Applying proba-\nbility measures to abstract languages. IEEE Transactions\non Computers , C-22(5):442\u2013450.\nBresnan, J., ed. 1982. The Mental Representation of Gram-\nmatical Relations . MIT Press.\nBrill, E. and P. Resnik. 1994. A rule-based approach to\nprepositional phrase attachment disambiguation. COL-\nING.\nBriscoe, T. and J. Carroll. 1993. Generalized proba-\nbilistic LR parsing of natural language (corpora) with\nuni\ufb01cation-based grammars. Computational Linguistics ,\n19(1):25\u201359.\nCharniak, E. 1997. Statistical parsing with a context-free\ngrammar and word statistics. AAAI .\nChelba, C. and F. Jelinek. 2000. Structured language model-\ning. Computer Speech and Language , 14:283\u2013332.\nCollins, M. 1996. A new statistical parser based on bigram\nlexical dependencies. ACL.\nCollins, M. 1999. Head-Driven Statistical Models for Natu-\nral Language Parsing . Ph.D. thesis, University of Penn-\nsylvania, Philadelphia.\nFrancis, H. S., M. L. Gregory, and L. A. Michaelis. 1999. Are\nlexical subjects deviant? CLS-99 . University of Chicago.\nFranz, A. 1997. Independence assumptions considered\nharmful. ACL.\nGiv\u00b4on, T. 1990. Syntax: A Functional Typological Introduc-\ntion. John Benjamins.\nHindle, D. and M. Rooth. 1990. Structural ambiguity and\nlexical relations. Speech and Natural Language Work-\nshop .\nHindle, D. and M. Rooth. 1991. Structural ambiguity and\nlexical relations. ACL.\nJelinek, F. and J. D. Lafferty. 1991. Computation of the\nprobability of initial substring generation by stochas-\ntic context-free grammars. Computational Linguistics ,\n17(3):315\u2013323.\nJelinek, F., J. D. Lafferty, D. M. Magerman, R. L. Mercer,\nA. Ratnaparkhi, and S. Roukos. 1994. Decision tree pars-\ning using a hidden derivation model. ARPA Human Lan-\nguage Technologies Workshop .Johnson, M. 1998. PCFG models of linguistic tree represen-\ntations. Computational Linguistics , 24(4):613\u2013632.\nJoshi, A. K. 1985. Tree adjoining grammars: How\nmuch context-sensitivity is required to provide reasonable\nstructural descriptions? In D. R. Dowty, L. Karttunen,\nand A. Zwicky, eds, Natural Language Parsing , 206\u2013250.\nCambridge University Press.\nJoshi, A. K. and B. Srinivas. 1994. Disambiguation of super\nparts of speech (or supertags): Almost parsing. COLING .\nKlein, D. and C. D. Manning. 2001. Parsing and hyper-\ngraphs. IWPT-01 .\nKlein, D. and C. D. Manning. 2003a. A* parsing: Fast exact\nViterbi parse selection. HLT-NAACL .\nKlein, D. and C. D. Manning. 2003b. Accurate unlexicalized\nparsing. HLT-NAACL .\nLafferty, J. D., D. Sleator, and D. Temperley. 1992. Gram-\nmatical trigrams: A probabilistic model of link gram-\nmar. AAAI Fall Symposium on Probabilistic Approaches\nto Natural Language .\nLari, K. and S. J. Young. 1990. The estimation of stochas-\ntic context-free grammars using the Inside-Outside algo-\nrithm. Computer Speech and Language , 4:35\u201356.\nMagerman, D. M. and M. P. Marcus. 1991. Pearl: A proba-\nbilistic chart parser. EACL .\nManning, C. D. and H. Sch \u00a8utze. 1999. Foundations of Sta-\ntistical Natural Language Processing . MIT Press.\nMarcus, M. P. 1990. Summary of session 9: Automatic ac-\nquisition of linguistic structure. Speech and Natural Lan-\nguage Workshop .\nNey, H. 1991. Dynamic programming parsing for context-\nfree grammars in continuous speech recognition. IEEE\nTransactions on Signal Processing , 39(2):336\u2013340.\nPetrov, S., L. Barrett, R. Thibaux, and D. Klein. 2006. Learn-\ning accurate, compact, and interpretable tree annotation.\nCOLING/ACL .\nPollard, C. and I. A. Sag. 1994. Head-Driven Phrase Struc-\nture Grammar . University of Chicago Press.\nRatnaparkhi, A., J. C. Reynar, and S. Roukos. 1994. A max-\nimum entropy model for prepositional phrase attachment.\nARPA Human Language Technologies Workshop .\nResnik, P. 1992. Probabilistic tree-adjoining grammar as\na framework for statistical natural language processing.\nCOLING .\nSalomaa, A. 1969. Probabilistic and weighted grammars.\nInformation and Control , 15:529\u2013544.\nSchabes, Y . 1990. Mathematical and Computational As-\npects of Lexicalized Grammars . Ph.D. thesis, University\nof Pennsylvania, Philadelphia, PA.\nSchabes, Y . 1992. Stochastic lexicalized tree-adjoining\ngrammars. COLING .\nSchabes, Y ., A. Abeill \u00b4e, and A. K. Joshi. 1988. Parsing\nstrategies with \u2018lexicalized\u2019 grammars: Applications to\nTree Adjoining Grammars. COLING .\nStetina, J. and M. Nagao. 1997. Corpus based PP attachment\nambiguity resolution with a semantic dictionary. Pro-\nceedings of the Fifth Workshop on Very Large Corpora .\nStolcke, A. 1995. An ef\ufb01cient probabilistic context-free\nparsing algorithm that computes pre\ufb01x probabilities.\nComputational Linguistics , 21(2):165\u2013202. Exercises 21\nZavrel, J. and W. Daelemans. 1997. Memory-based learning:\nUsing similarity for smoothing. ACL.", "D.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\nDConstituency Grammars\nBecause the Night by Bruce Springsteen and Patty Smith\nThe Fire Next Time by James Baldwin\nIf on a winter\u2019s night a traveler by Italo Calvino\nLove Actually by Richard Curtis\nSuddenly Last Summer by Tennessee Williams\nA Scanner Darkly by Philip K. Dick\nSix titles that are not constituents, from Geoffrey K. Pullum on\nLanguage Log (who was pointing out their incredible rarity).\nThe study of grammar has an ancient pedigree. The grammar of Sanskrit was\ndescribed by the Indian grammarian P \u00afan.ini sometime between the 7th and 4th cen-\nturies BCE, in his famous treatise the As .t.\u00afadhy \u00afay\u00af\u0131 (\u20188 books\u2019). And our word syntax syntax\ncomes from the Greek s\u00b4yntaxis , meaning \u201csetting out together or arrangement\u201d, and\nrefers to the way words are arranged together. We have seen various syntactic no-\ntions in previous chapters: ordering of sequences of words (Chapter 2), probabilities\nfor these word sequences (Chapter 3), and the use of part-of-speech categories as\na grammatical equivalence class for words (Chapter 17). In this chapter and the\nnext three we introduce a variety of syntactic phenomena that go well beyond these\nsimpler approaches, together with formal models for capturing them in a computa-\ntionally useful manner.\nThe bulk of this chapter is devoted to context-free grammars. Context-free gram-\nmars are the backbone of many formal models of the syntax of natural language (and,\nfor that matter, of computer languages). As such, they play a role in many computa-\ntional applications, including grammar checking, semantic interpretation, dialogue\nunderstanding, and machine translation. They are powerful enough to express so-\nphisticated relations among the words in a sentence, yet computationally tractable\nenough that ef\ufb01cient algorithms exist for parsing sentences with them (as we show in\nChapter 18). Here we also introduce the concept of lexicalized grammars, focusing\non one example, combinatory categorial grammar , orCCG .\nIn Chapter 20 we introduce a formal model of grammar called syntactic depen-\ndencies that is an alternative to these constituency grammars, and we\u2019ll give algo-\nrithms for dependency parsing . Both constituency and dependency formalisms are\nimportant for language processing.\nFinally, we provide a brief overview of the grammar of English, illustrated from\na domain with relatively simple sentences called ATIS (Air Traf\ufb01c Information Sys-\ntem) (Hemphill et al., 1990). ATIS systems were an early spoken language system\nfor users to book \ufb02ights, by expressing sentences like I\u2019d like to \ufb02y to Atlanta . 2APPENDIX D \u2022 C ONSTITUENCY GRAMMARS\nD.1 Constituency\nSyntactic constituency is the idea that groups of words can behave as single units,\nor constituents. Part of developing a grammar involves building an inventory of the\nconstituents in the language. How do words group together in English? Consider\nthenoun phrase , a sequence of words surrounding at least one noun. Here are some noun phrase\nexamples of noun phrases (thanks to Damon Runyon):\nHarry the Horse a high-class spot such as Mindy\u2019s\nthe Broadway coppers the reason he comes into the Hot Box\nthey three parties from Brooklyn\nWhat evidence do we have that these words group together (or \u201cform constituents\u201d)?\nOne piece of evidence is that they can all appear in similar syntactic environments,\nfor example, before a verb.\nthree parties from Brooklyn arrive . . .\na high-class spot such as Mindy\u2019s attracts . . .\nthe Broadway coppers love. . .\nthey sit\nBut while the whole noun phrase can occur before a verb, this is not true of each\nof the individual words that make up a noun phrase. The following are not grammat-\nical sentences of English (recall that we use an asterisk (*) to mark fragments that\nare not grammatical English sentences):\n*from arrive . . .*asattracts . . .\n*the is. . . *spot sat. . .\nThus, to correctly describe facts about the ordering of these words in English, we\nmust be able to say things like \u201c Noun Phrases can occur before verbs \u201d.\nOther kinds of evidence for constituency come from what are called preposed or preposed\npostposed constructions. For example, the prepositional phrase on September sev- postposed\nenteenth can be placed in a number of different locations in the following examples,\nincluding at the beginning (preposed) or at the end (postposed):\nOn September seventeenth , I\u2019d like to \ufb02y from Atlanta to Denver\nI\u2019d like to \ufb02y on September seventeenth from Atlanta to Denver\nI\u2019d like to \ufb02y from Atlanta to Denver on September seventeenth\nBut again, while the entire phrase can be placed differently, the individual words\nmaking up the phrase cannot be:\n*On September , I\u2019d like to \ufb02y seventeenth from Atlanta to Denver\n*On I\u2019d like to \ufb02y September seventeenth from Atlanta to Denver\n*I\u2019d like to \ufb02y on September from Atlanta to Denver seventeenth\nD.2 Context-Free Grammars\nThe most widely used formal system for modeling constituent structure in English\nand other natural languages is the Context-Free Grammar , or CFG . Context- CFG D.2 \u2022 C ONTEXT -FREE GRAMMARS 3\nfree grammars are also called Phrase-Structure Grammars , and the formalism\nis equivalent to Backus-Naur Form , orBNF . The idea of basing a grammar on\nconstituent structure dates back to the psychologist Wilhelm Wundt 1900 but was\nnot formalized until Chomsky (1956) and, independently, Backus (1959).\nA context-free grammar consists of a set of rules orproductions , each of which rules\nexpresses the ways that symbols of the language can be grouped and ordered to-\ngether, and a lexicon of words and symbols. For example, the following productions lexicon\nexpress that an NP(ornoun phrase ) can be composed of either a ProperNoun or NP\na determiner ( Det) followed by a Nominal ; aNominal in turn can consist of one or\nmore Noun s.1\nNP!Det Nominal\nNP!ProperNoun\nNominal!NounjNominal Noun\nContext-free rules can be hierarchically embedded, so we can combine the previous\nrules with others, like the following, that express facts about the lexicon:\nDet!a\nDet!the\nNoun!\ufb02ight\nThe symbols that are used in a CFG are divided into two classes. The symbols\nthat correspond to words in the language (\u201cthe\u201d, \u201cnightclub\u201d) are called terminal terminal\nsymbols; the lexicon is the set of rules that introduce these terminal symbols. The\nsymbols that express abstractions over these terminals are called non-terminals . In non-terminal\neach context-free rule, the item to the right of the arrow ( !) is an ordered list of one\nor more terminals and non-terminals; to the left of the arrow is a single non-terminal\nsymbol expressing some cluster or generalization. The non-terminal associated with\neach word in the lexicon is its lexical category, or part of speech.\nA CFG can be thought of in two ways: as a device for generating sentences\nand as a device for assigning a structure to a given sentence. Viewing a CFG as a\ngenerator, we can read the !arrow as \u201crewrite the symbol on the left with the string\nof symbols on the right\u201d.\nSo starting from the symbol: NP\nwe can use our \ufb01rst rule to rewrite NPas: Det Nominal\nand then rewrite Nominal as: Noun\nand \ufb01nally rewrite these parts-of-speech as: a \ufb02ight\nWe say the string a \ufb02ight can be derived from the non-terminal NP. Thus, a CFG\ncan be used to generate a set of strings. This sequence of rule expansions is called a\nderivation of the string of words. It is common to represent a derivation by a parse derivation\ntree (commonly shown inverted with the root at the top). Figure D.1 shows the tree parse tree\nrepresentation of this derivation.\nIn the parse tree shown in Fig. D.1, we can say that the node NPdominates dominates\nall the nodes in the tree ( Det,Nom ,Noun ,a,\ufb02ight ). We can say further that it\nimmediately dominates the nodes DetandNom .\nThe formal language de\ufb01ned by a CFG is the set of strings that are derivable\nfrom the designated start symbol . Each grammar must have one designated start start symbol\n1When talking about these rules we can pronounce the rightarrow !as \u201cgoes to\u201d, and so we might\nread the \ufb01rst rule above as \u201cNP goes to Det Nominal\u201d. 4APPENDIX D \u2022 C ONSTITUENCY GRAMMARS\nNP\nNom\nNoun\n\ufb02ightDet\na\nFigure D.1 A parse tree for \u201ca \ufb02ight\u201d.\nsymbol, which is often called S. Since context-free grammars are often used to de\ufb01ne\nsentences, Sis usually interpreted as the \u201csentence\u201d node, and the set of strings that\nare derivable from Sis the set of sentences in some simpli\ufb01ed version of English.\nLet\u2019s add a few additional rules to our inventory. The following rule expresses\nthe fact that a sentence can consist of a noun phrase followed by a verb phrase : verb phrase\nS!NP VP I prefer a morning \ufb02ight\nA verb phrase in English consists of a verb followed by assorted other things;\nfor example, one kind of verb phrase consists of a verb followed by a noun phrase:\nVP!Verb NP prefer a morning \ufb02ight\nOr the verb may be followed by a noun phrase and a prepositional phrase:\nVP!Verb NP PP leave Boston in the morning\nOr the verb phrase may have a verb followed by a prepositional phrase alone:\nVP!Verb PP leaving on Thursday\nA prepositional phrase generally has a preposition followed by a noun phrase.\nFor example, a common type of prepositional phrase in the ATIS corpus is used to\nindicate location or direction:\nPP!Preposition NP from Los Angeles\nTheNPinside a PPneed not be a location; PPs are often used with times and\ndates, and with other nouns as well; they can be arbitrarily complex. Here are ten\nexamples from the ATIS corpus:\nto Seattle on these \ufb02ights\nin Minneapolis about the ground transportation in Chicago\non Wednesday of the round trip \ufb02ight on United Airlines\nin the evening of the AP \ufb01fty seven \ufb02ight\non the ninth of July with a stopover in Nashville\nFigure D.2 gives a sample lexicon, and Fig. D.3 summarizes the grammar rules\nwe\u2019ve seen so far, which we\u2019ll call L0. Note that we can use the or-symbol jto\nindicate that a non-terminal has alternate possible expansions.\nWe can use this grammar to generate sentences of this \u201cATIS-language\u201d. We\nstart with S, expand it to NP VP , then choose a random expansion of NP(let\u2019s say, to\nI), and a random expansion of VP(let\u2019s say, to Verb NP ), and so on until we generate\nthe string I prefer a morning \ufb02ight . Figure D.4 shows a parse tree that represents a\ncomplete derivation of I prefer a morning \ufb02ight .\nWe can also represent a parse tree in a more compact format called bracketed\nnotation ; here is the bracketed representation of the parse tree of Fig. D.4:bracketed\nnotation D.2 \u2022 C ONTEXT -FREE GRAMMARS 5\nNoun!\ufb02ightsj\ufb02ightjbreezejtripjmorning\nVerb!isjpreferjlikejneedjwantj\ufb02yjdo\nAdjective!cheapestjnon-stopj\ufb01rstjlatest\njotherjdirect\nPronoun!mejIjyoujit\nProper-Noun!AlaskajBaltimorejLos Angeles\njChicagojUnitedjAmerican\nDeterminer!thejajanjthisjthesejthat\nPreposition!fromjtojonjnearjin\nConjunction!andjorjbut\nFigure D.2 The lexicon for L0.\nGrammar Rules Examples\nS!NP VP I + want a morning \ufb02ight\nNP!Pronoun I\njProper-Noun Los Angeles\njDet Nominal a + \ufb02ight\nNominal!Nominal Noun morning + \ufb02ight\njNoun \ufb02ights\nVP!Verb do\njVerb NP want + a \ufb02ight\njVerb NP PP leave + Boston + in the morning\njVerb PP leaving + on Thursday\nPP!Preposition NP from + Los Angeles\nFigure D.3 The grammar for L0, with example phrases for each rule.\nS\nVP\nNP\nNom\nNoun\n\ufb02ightNom\nNoun\nmorningDet\naVerb\npreferNP\nPro\nI\nFigure D.4 The parse tree for \u201cI prefer a morning \ufb02ight\u201d according to grammar L0.\n(D.1) [ S[NP[ProI]] [VP[Vprefer] [ NP[Deta] [Nom [Nmorning] [ Nom [N\ufb02ight]]]]]]\nA CFG like that of L0de\ufb01nes a formal language. We saw in Chapter 2 that a for-\nmal language is a set of strings. Sentences (strings of words) that can be derived by a\ngrammar are in the formal language de\ufb01ned by that grammar, and are called gram-\nmatical sentences. Sentences that cannot be derived by a given formal grammar are grammatical\nnot in the language de\ufb01ned by that grammar and are referred to as ungrammatical . ungrammatical 6APPENDIX D \u2022 C ONSTITUENCY GRAMMARS\nThis hard line between \u201cin\u201d and \u201cout\u201d characterizes all formal languages but is only\na very simpli\ufb01ed model of how natural languages really work. This is because de-\ntermining whether a given sentence is part of a given natural language (say, English)\noften depends on the context. In linguistics, the use of formal languages to model\nnatural languages is called generative grammar since the language is de\ufb01ned bygenerative\ngrammar\nthe set of possible sentences \u201cgenerated\u201d by the grammar.\nD.2.1 Formal De\ufb01nition of Context-Free Grammar\nWe conclude this section with a quick, formal description of a context-free gram-\nmar and the language it generates. A context-free grammar Gis de\ufb01ned by four\nparameters: N;S;R;S(technically this is a \u201c4-tuple\u201d).\nNa set of non-terminal symbols (orvariables )\nSa set of terminal symbols (disjoint from N)\nRa set of rules or productions, each of the form A!b,\nwhere Ais a non-terminal,\nbis a string of symbols from the in\ufb01nite set of strings (S[N)\u0003\nSa designated start symbol and a member of N\nFor the remainder of the book we adhere to the following conventions when dis-\ncussing the formal properties of context-free grammars (as opposed to explaining\nparticular facts about English or other languages).\nCapital letters like A,B, and S Non-terminals\nS The start symbol\nLower-case Greek letters like a,b, and g Strings drawn from (S[N)\u0003\nLower-case Roman letters like u,v, and w Strings of terminals\nA language is de\ufb01ned through the concept of derivation. One string derives an-\nother one if it can be rewritten as the second one by some series of rule applications.\nMore formally, following Hopcroft and Ullman (1979),\nifA!bis a production of Randaandgare any strings in the set\n(S[N)\u0003, then we say that aAgdirectly derives abg , oraAg)abg . directly derives\nDerivation is then a generalization of direct derivation:\nLeta1;a2;:::; ambe strings in (S[N)\u0003;m\u00151, such that\na1)a2;a2)a3;:::; am\u00001)am\nWe say that a1derives am, ora1\u0003)am. derives\nWe can then formally de\ufb01ne the language LGgenerated by a grammar Gas the\nset of strings composed of terminal symbols that can be derived from the designated\nstart symbol S.\nLG=fwjwis inS\u0003andS\u0003)wg\nThe problem of mapping from a string of words to its parse tree is called syn-\ntactic parsing ; we de\ufb01ne algorithms for constituency parsing in Chapter 18.syntactic\nparsing D.3 \u2022 S OME GRAMMAR RULES FOR ENGLISH 7\nD.3 Some Grammar Rules for English\nIn this section, we introduce a few more aspects of the phrase structure of English;\nfor consistency we will continue to focus on sentences from the ATIS domain. Be-\ncause of space limitations, our discussion is necessarily limited to highlights. Read-\ners are strongly advised to consult a good reference grammar of English, such as\nHuddleston and Pullum (2002).\nD.3.1 Sentence-Level Constructions\nIn the small grammar L0, we provided only one sentence-level construction for\ndeclarative sentences like I prefer a morning \ufb02ight . Among the large number of\nconstructions for English sentences, four are particularly common and important:\ndeclaratives, imperatives, yes-no questions, and wh-questions.\nSentences with declarative structure have a subject noun phrase followed by declarative\na verb phrase, like \u201cI prefer a morning \ufb02ight\u201d. Sentences with this structure have\na great number of different uses that we follow up on in Chapter 15. Here are a\nnumber of examples from the ATIS domain:\nI want a \ufb02ight from Ontario to Chicago\nThe \ufb02ight should be eleven a.m. tomorrow\nThe return \ufb02ight should leave at around seven p.m.\nSentences with imperative structure often begin with a verb phrase and have imperative\nno subject. They are called imperative because they are almost always used for\ncommands and suggestions; in the ATIS domain they are commands to the system.\nShow the lowest fare\nGive me Sunday\u2019s \ufb02ights arriving in Las Vegas from New York City\nList all \ufb02ights between \ufb01ve and seven p.m.\nWe can model this sentence structure with another rule for the expansion of S:\nS!VP\nSentences with yes-no question structure are often (though not always) used to yes-no question\nask questions; they begin with an auxiliary verb, followed by a subject NP, followed\nby a VP. Here are some examples. Note that the third example is not a question at\nall but a request; Chapter 15 discusses the uses of these question forms to perform\ndifferent pragmatic functions such as asking, requesting, or suggesting.\nDo any of these \ufb02ights have stops?\nDoes American\u2019s \ufb02ight eighteen twenty \ufb01ve serve dinner?\nCan you give me the same information for United?\nHere\u2019s the rule:\nS!Aux NP VP\nThe most complex sentence-level structures we examine here are the various wh-\nstructures. These are so named because one of their constituents is a wh-phrase , that wh-phrase\nis, one that includes a wh-word (who, whose, when, where, what, which, how, why ). wh-word\nThese may be broadly grouped into two classes of sentence-level structures. The\nwh-subject-question structure is identical to the declarative structure, except that\nthe \ufb01rst noun phrase contains some wh-word. 8APPENDIX D \u2022 C ONSTITUENCY GRAMMARS\nWhat airlines \ufb02y from Burbank to Denver?\nWhich \ufb02ights depart Burbank after noon and arrive in Denver by six p.m?\nWhose \ufb02ights serve breakfast?\nHere is a rule. Exercise D.7 discusses rules for the constituents that make up the\nWh-NP .\nS!Wh-NP VP\nIn the wh-non-subject-question structure, the wh-phrase is not the subject of thewh-non-subject-\nquestion\nsentence, and so the sentence includes another subject. In these types of sentences\nthe auxiliary appears before the subject NP, just as in the yes-no question structures.\nHere is an example followed by a sample rule:\nWhat \ufb02ights do you have from Burbank to Tacoma Washington?\nS!Wh-NP Aux NP VP\nConstructions like the wh-non-subject-question contain what are called long-\ndistance dependencies because the Wh-NP what \ufb02ights is far away from the predi-long-distance\ndependencies\ncate that it is semantically related to, the main verb have in the VP. In some models\nof parsing and understanding compatible with the grammar rule above, long-distance\ndependencies like the relation between \ufb02ights andhave are thought of as a semantic\nrelation. In such models, the job of \ufb01guring out that \ufb02ights is the argument of have is\ndone during semantic interpretation. Other models of parsing represent the relation-\nship between \ufb02ights andhave as a syntactic relation, and the grammar is modi\ufb01ed to\ninsert a small marker called a trace orempty category after the verb. We discuss\nempty-category models when we introduce the Penn Treebank on page 15.\nD.3.2 Clauses and Sentences\nBefore we move on, we should clarify the status of the Srules in the grammars we\njust described. Srules are intended to account for entire sentences that stand alone\nas fundamental units of discourse. However, Scan also occur on the right-hand side\nof grammar rules and hence can be embedded within larger sentences. Clearly then,\nthere\u2019s more to being an Sthan just standing alone as a unit of discourse.\nWhat differentiates sentence constructions (i.e., the Srules) from the rest of the\ngrammar is the notion that they are in some sense complete . In this way they corre-\nspond to the notion of a clause , which traditional grammars often describe as form- clause\ning a complete thought. One way of making this notion of \u201ccomplete thought\u201d more\nprecise is to say an Sis a node of the parse tree below which the main verb of the S\nhas all of its arguments . We de\ufb01ne verbal arguments later, but for now let\u2019s just see\nan illustration from the tree for I prefer a morning \ufb02ight in Fig. D.4 on page 5. The\nverb prefer has two arguments: the subject Iand the object a morning \ufb02ight . One of\nthe arguments appears below the VPnode, but the other one, the subject NP, appears\nonly below the Snode.\nD.3.3 The Noun Phrase\nOurL0grammar introduced three of the most frequent types of noun phrases that\noccur in English: pronouns, proper nouns and the NP!Det Nominal construction.\nThe central focus of this section is on the last type since that is where the bulk of\nthe syntactic complexity resides. These noun phrases consist of a head, the central\nnoun in the noun phrase, along with various modi\ufb01ers that can occur before or after\nthe head noun. Let\u2019s take a close look at the various parts. D.3 \u2022 S OME GRAMMAR RULES FOR ENGLISH 9\nThe Determiner\nNoun phrases can begin with simple lexical determiners:\na stop the \ufb02ights this \ufb02ight\nthose \ufb02ights any \ufb02ights some \ufb02ights\nThe role of the determiner can also be \ufb01lled by more complex expressions:\nUnited\u2019s \ufb02ight\nUnited\u2019s pilot\u2019s union\nDenver\u2019s mayor\u2019s mother\u2019s canceled \ufb02ight\nIn these examples, the role of the determiner is \ufb01lled by a possessive expression\nconsisting of a noun phrase followed by an \u2019sas a possessive marker, as in the\nfollowing rule.\nDet!NP0s\nThe fact that this rule is recursive (since an NPcan start with a Det) helps us model\nthe last two examples above, in which a sequence of possessive expressions serves\nas a determiner.\nUnder some circumstances determiners are optional in English. For example,\ndeterminers may be omitted if the noun they modify is plural:\n(D.2) Show me \ufb02ights from San Francisco to Denver on weekdays\nAs we saw in Chapter 17, mass nouns also don\u2019t require determination. Recall that\nmass nouns often (not always) involve something that is treated like a substance\n(including e.g., water andsnow ), don\u2019t take the inde\ufb01nite article \u201c a\u201d, and don\u2019t tend\nto pluralize. Many abstract nouns are mass nouns ( music ,homework ). Mass nouns\nin the ATIS domain include breakfast ,lunch , and dinner :\n(D.3) Does this \ufb02ight serve dinner?\nThe Nominal\nThe nominal construction follows the determiner and contains any pre- and post-\nhead noun modi\ufb01ers. As indicated in grammar L0, in its simplest form a nominal\ncan consist of a single noun.\nNominal!Noun\nAs we\u2019ll see, this rule also provides the basis for the bottom of various recursive\nrules used to capture more complex nominal constructions.\nBefore the Head Noun\nA number of different kinds of word classes can appear before the head noun but\nafter the determiner (the \u201cpostdeterminers\u201d) in a nominal. These include cardinalcardinal\nnumbers\nnumbers ,ordinal numbers ,quanti\ufb01ers , and adjectives . Examples of cardinalordinal\nnumbers\nquanti\ufb01ers numbers:\ntwo friends one stop\nOrdinal numbers include \ufb01rst,second ,third , and so on, but also words like next,\nlast,past,other , and another :\nthe \ufb01rst one the next day the second leg\nthe last \ufb02ight the other American \ufb02ight\nSome quanti\ufb01ers ( many ,(a) few ,several ) occur only with plural count nouns: 10 APPENDIX D \u2022 C ONSTITUENCY GRAMMARS\nmany fares\nAdjectives occur after quanti\ufb01ers but before nouns.\na\ufb01rst-class fare a non-stop \ufb02ight\nthelongest layover the earliest lunch \ufb02ight\nAdjectives can also be grouped into a phrase called an adjective phrase or AP.adjective\nphrase\nAPs can have an adverb before the adjective (see Chapter 17 for de\ufb01nitions of ad-\njectives and adverbs):\ntheleast expensive fare\nAfter the Head Noun\nA head noun can be followed by postmodi\ufb01ers . Three kinds of nominal postmodi-\n\ufb01ers are common in English:\nprepositional phrases all \ufb02ights from Cleveland\nnon-\ufb01nite clauses any \ufb02ights arriving after eleven a.m.\nrelative clauses a \ufb02ight that serves breakfast\nThey are especially common in the ATIS corpus since they are used to mark the\norigin and destination of \ufb02ights.\nHere are some examples of prepositional phrase postmodi\ufb01ers, with brackets\ninserted to show the boundaries of each PP; note that two or more PPs can be strung\ntogether within a single NP:\nall \ufb02ights [from Cleveland] [to Newark]\narrival [in San Jose] [before seven p.m.]\na reservation [on \ufb02ight six oh six] [from Tampa] [to Montreal]\nHere\u2019s a new nominal rule to account for postnominal PPs:\nNominal!Nominal PP\nThe three most common kinds of non-\ufb01nite postmodi\ufb01ers are the gerundive ( - non-\ufb01nite\ning),-ed, and in\ufb01nitive forms.\nGerundive postmodi\ufb01ers are so called because they consist of a verb phrase that gerundive\nbegins with the gerundive ( -ing) form of the verb. Here are some examples:\nany of those [leaving on Thursday]\nany \ufb02ights [arriving after eleven a.m.]\n\ufb02ights [arriving within thirty minutes of each other]\nWe can de\ufb01ne the Nominals with gerundive modi\ufb01ers as follows, making use of\na new non-terminal GerundVP :\nNominal!Nominal GerundVP\nWe can make rules for GerundVP constituents by duplicating all of our VP pro-\nductions, substituting GerundV forV.\nGerundVP!GerundV NP\njGerundV PPjGerundVjGerundV NP PP\nGerundV can then be de\ufb01ned as\nGerundV!beingjarrivingjleavingj:::\nThe phrases in italics below are examples of the two other common kinds of\nnon-\ufb01nite clauses, in\ufb01nitives and -edforms: D.3 \u2022 S OME GRAMMAR RULES FOR ENGLISH 11\nthe last \ufb02ight to arrive in Boston\nI need to have dinner served\nWhich is the aircraft used by this \ufb02ight ?\nA postnominal relative clause (more correctly a restrictive relative clause ), is\na clause that often begins with a relative pronoun (thatandwho are the most com-relative\npronoun\nmon). The relative pronoun functions as the subject of the embedded verb in the\nfollowing examples:\na \ufb02ight that serves breakfast\n\ufb02ights that leave in the morning\nthe one that leaves at ten thirty \ufb01ve\nWe might add rules like the following to deal with these:\nNominal!Nominal RelClause\nRelClause!(whojthat)VP\nThe relative pronoun may also function as the object of the embedded verb, as\nin the following example; we leave for the reader the exercise of writing grammar\nrules for more complex relative clauses of this kind.\nthe earliest American Airlines \ufb02ight that I can get\nVarious postnominal modi\ufb01ers can be combined:\na \ufb02ight [from Phoenix to Detroit] [leaving Monday evening]\nevening \ufb02ights [from Nashville to Houston] [that serve dinner]\na friend [living in Denver] [that would like to visit me in DC]\nBefore the Noun Phrase\nWord classes that modify and appear before NPs are called predeterminers . Many predeterminers\nof these have to do with number or amount; a common predeterminer is all:\nall the \ufb02ights all \ufb02ights all non-stop \ufb02ights\nThe example noun phrase given in Fig. D.5 illustrates some of the complexity\nthat arises when these rules are combined.\nD.3.4 The Verb Phrase\nThe verb phrase consists of the verb and a number of other constituents. In the\nsimple rules we have built so far, these other constituents include NPs and PPs and\ncombinations of the two:\nVP!Verb disappear\nVP!Verb NP prefer a morning \ufb02ight\nVP!Verb NP PP leave Boston in the morning\nVP!Verb PP leaving on Thursday\nVerb phrases can be signi\ufb01cantly more complicated than this. Many other kinds\nof constituents, such as an entire embedded sentence, can follow the verb. These are\ncalled sentential complements :sentential\ncomplements\nYou [ VP[Vsaid [ Syou had a two hundred sixty-six dollar fare]]\n[VP[VTell] [ NPme] [ Show to get from the airport to downtown]]\nI [VP[Vthink [ SI would like to take the nine thirty \ufb02ight]] 12 APPENDIX D \u2022 C ONSTITUENCY GRAMMARS\nNP\nNP\nNom\nGerundiveVP\nleaving before 10Nom\nPP\nto TampaNom\nPP\nfrom DenverNom\nNoun\n\ufb02ightsNom\nNoun\nmorningDet\nthePreDet\nall\nFigure D.5 A parse tree for \u201call the morning \ufb02ights from Denver to Tampa leaving before 10\u201d.\nHere\u2019s a rule for these:\nVP!Verb S\nSimilarly, another potential constituent of the VPis another VP. This is often the\ncase for verbs like want ,would like ,try,intend ,need :\nI want [ VPto \ufb02y from Milwaukee to Orlando]\nHi, I want [ VPto arrange three \ufb02ights]\nWhile a verb phrase can have many possible kinds of constituents, not every\nverb is compatible with every verb phrase. For example, the verb want can be used\neither with an NPcomplement ( I want a \ufb02ight . . . ) or with an in\ufb01nitive VPcomple-\nment ( I want to \ufb02y to . . . ). By contrast, a verb like \ufb01ndcannot take this sort of VP\ncomplement ( * I found to \ufb02y to Dallas ).\nThis idea that verbs are compatible with different kinds of complements is a very\nold one; traditional grammar distinguishes between transitive verbs like \ufb01nd, which transitive\ntake a direct object NP(I found a \ufb02ight ), and intransitive verbs like disappear , intransitive\nwhich do not ( *I disappeared a \ufb02ight ).\nWhere traditional grammars subcategorize verbs into these two categories (tran- subcategorize\nsitive and intransitive), modern grammars distinguish as many as 100 subcategories.\nWe say that a verb like \ufb01ndsubcategorizes for anNP, and a verb like want sub-subcategorizes\nfor\ncategorizes for either an NPor a non-\ufb01nite VP. We also call these constituents the\ncomplements of the verb (hence our use of the term sentential complement above). complements\nSo we say that want can take a VPcomplement. These possible sets of complements\nare called the subcategorization frame for the verb. Another way of talking aboutsubcategorization\nframe\nthe relation between the verb and these other constituents is to think of the verb as\na logical predicate and the constituents as logical arguments of the predicate. So we\ncan think of such predicate-argument relations as FIND (I,A FLIGHT ) or WANT (I,TO\nFLY). We talk more about this view of verbs and arguments in Appendix F when we\ntalk about predicate calculus representations of verb semantics. Subcategorization\nframes for a set of example verbs are given in Fig. D.6. D.3 \u2022 S OME GRAMMAR RULES FOR ENGLISH 13\nFrame Verb Example\n/ 0 eat, sleep I ate\nNP prefer, \ufb01nd, leave Find [ NPthe \ufb02ight from Pittsburgh to Boston]\nNP NP show, give Show [ NPme] [ NPairlines with \ufb02ights from Pittsburgh]\nPPfromPPto \ufb02y, travel I would like to \ufb02y [ PPfrom Boston] [ PPto Philadelphia]\nNP PP with help, load Can you help [ NPme] [ PPwith a \ufb02ight]\nVPto prefer, want, need I would prefer [ VPto to go by United Airlines]\nS mean Does this mean [ SAA has a hub in Boston]\nFigure D.6 Subcategorization frames for a set of example verbs.\nWe can capture the association between verbs and their complements by making\nseparate subtypes of the class Verb (e.g., Verb-with-NP-complement ,Verb-with-Inf-\nVP-complement ,Verb-with-S-complement , and so on):\nVerb-with-NP-complement !\ufb01ndjleavejrepeatj:::\nVerb-with-S-complement !thinkjbelievejsayj:::\nVerb-with-Inf-VP-complement !wantjtryjneedj:::\nEach VPrule could then be modi\ufb01ed to require the appropriate verb subtype:\nVP!Verb-with-no-complement disappear\nVP!Verb-with-NP-comp NP prefer a morning \ufb02ight\nVP!Verb-with-S-comp S said there were two \ufb02ights\nA problem with this approach is the signi\ufb01cant increase in the number of rules and\nthe associated loss of generality.\nD.3.5 Coordination\nThe major phrase types discussed here can be conjoined with conjunctions likeand, conjunctions\nor, and butto form larger constructions of the same type. For example, a coordinate coordinate\nnoun phrase can consist of two other noun phrases separated by a conjunction:\nPlease repeat [ NP[NPthe \ufb02ights] and[NPthe costs]]\nI need to know [ NP[NPthe aircraft] and[NPthe \ufb02ight number]]\nHere\u2019s a rule that allows these structures:\nNP!NP and NP\nNote that the ability to form coordinate phrases through conjunctions is often\nused as a test for constituency. Consider the following examples, which differ from\nthe ones given above in that they lack the second determiner.\nPlease repeat the [ Nom[Nom\ufb02ights] and[Nomcosts]]\nI need to know the [ Nom[Nomaircraft] and[Nom\ufb02ight number]]\nThe fact that these phrases can be conjoined is evidence for the presence of the\nunderlying Nominal constituent we have been making use of. Here\u2019s a rule for this:\nNominal!Nominal and Nominal\nThe following examples illustrate conjunctions involving VPs and Ss. 14 APPENDIX D \u2022 C ONSTITUENCY GRAMMARS\nWhat \ufb02ights do you have [ VP[VPleaving Denver] and[VParriving in\nSan Francisco]]\n[S[SI\u2019m interested in a \ufb02ight from Dallas to Washington] and[SI\u2019m\nalso interested in going to Baltimore]]\nThe rules for VPandSconjunctions mirror the NPone given above.\nVP!VP and VP\nS!S and S\nSince all the major phrase types can be conjoined in this fashion, it is also possible\nto represent this conjunction fact more generally; a number of grammar formalisms\nsuch as GPSG (Gazdar et al., 1985) do this using metarules like: metarules\nX!X and X\nThis metarule states that any non-terminal can be conjoined with the same non-\nterminal to yield a constituent of the same type; the variable Xmust be designated\nas a variable that stands for any non-terminal rather than a non-terminal itself.\nD.4 Treebanks\nSuf\ufb01ciently robust grammars consisting of context-free grammar rules can be used\nto assign a parse tree to any sentence. This means that it is possible to build a\ncorpus where every sentence in the collection is paired with a corresponding parse\ntree. Such a syntactically annotated corpus is called a treebank . Treebanks play treebank\nan important role in parsing, as we discuss in Chapter 18, as well as in linguistic\ninvestigations of syntactic phenomena.\nA wide variety of treebanks have been created, generally through the use of\nparsers (of the sort described in the next few chapters) to automatically parse each\nsentence, followed by the use of humans (linguists) to hand-correct the parses. The\nPenn Treebank project (whose POS tagset we introduced in Chapter 17) has pro- Penn Treebank\nduced treebanks from the Brown, Switchboard, ATIS, and Wall Street Journal cor-\npora of English, as well as treebanks in Arabic and Chinese. A number of treebanks\nuse the dependency representation we will introduce in Chapter 20, including many\nthat are part of the Universal Dependencies project (Nivre et al., 2016).\nD.4.1 Example: The Penn Treebank Project\nFigure D.7 shows sentences from the Brown and ATIS portions of the Penn Tree-\nbank.2Note the formatting differences for the part-of-speech tags; such small dif-\nferences are common and must be dealt with in processing treebanks. The Penn\nTreebank part-of-speech tagset was de\ufb01ned in Chapter 17. The use of LISP-style\nparenthesized notation for trees is extremely common and resembles the bracketed\nnotation we saw earlier in (D.1). For those who are not familiar with it we show a\nstandard node-and-line tree representation in Fig. D.8.\nFigure D.9 shows a tree from the Wall Street Journal . This tree shows another\nfeature of the Penn Treebanks: the use of traces (-NONE- nodes) to mark long- traces\n2The Penn Treebank project released treebanks in multiple languages and in various stages; for exam-\nple, there were Treebank I (Marcus et al., 1993), Treebank II (Marcus et al., 1994), and Treebank III\nreleases of English treebanks. We use Treebank III for our examples. D.4 \u2022 T REEBANKS 15\n((S\n(NP-SBJ (DT That)\n(JJ cold) (, ,)\n(JJ empty) (NN sky) )\n(VP (VBD was)\n(ADJP-PRD (JJ full)\n(PP (IN of)\n(NP (NN fire)\n(CC and)\n(NN light) ))))\n(. .) ))((S\n(NP-SBJ The/DT flight/NN )\n(VP should/MD\n(VP arrive/VB\n(PP-TMP at/IN\n(NP eleven/CD a.m/RB ))\n(NP-TMP tomorrow/NN )))))\n(a) (b)\nFigure D.7 Parsed sentences from the LDC Treebank3 version of the (a) Brown and (b)\nATIS corpora.\nS\n.\n.VP\nADJP-PRD\nPP\nNP\nNN\nlightCC\nandNN\n\ufb01reIN\nofJJ\nfullVBD\nwasNP-SBJ\nNN\nskyJJ\nempty,\n,JJ\ncoldDT\nThat\nFigure D.8 The tree corresponding to the Brown corpus sentence in the previous \ufb01gure.\ndistance dependencies or syntactic movement . For example, quotations often fol-syntactic\nmovement\nlow a quotative verb like say. But in this example, the quotation \u201cWe would have\nto wait until we have collected on those assets\u201d precedes the words he said . An\nempty Scontaining only the node -NONE- marks the position after said where the\nquotation sentence often occurs. This empty node is marked (in Treebanks II and\nIII) with the index 2, as is the quotation Sat the beginning of the sentence. Such\nco-indexing may make it easier for some parsers to recover the fact that this fronted\nor topicalized quotation is the complement of the verb said. A similar -NONE- node\nmarks the fact that there is no syntactic subject right before the verb to wait ; instead,\nthe subject is the earlier NP We . Again, they are both co-indexed with the index 1.\nThe Penn Treebank II and Treebank III releases added further information to\nmake it easier to recover the relationships between predicates and arguments. Cer-\ntain phrases were marked with tags indicating the grammatical function of the phrase\n(as surface subject, logical topic, cleft, non-VP predicates) its presence in particular\ntext categories (headlines, titles), and its semantic function (temporal phrases, lo- 16 APPENDIX D \u2022 C ONSTITUENCY GRAMMARS\n( (S (`` ``)\n(S-TPC-2\n(NP-SBJ-1 (PRP We) )\n(VP (MD would)\n(VP (VB have)\n(S\n(NP-SBJ (-NONE- *-1) )\n(VP (TO to)\n(VP (VB wait)\n(SBAR-TMP (IN until)\n(S\n(NP-SBJ (PRP we) )\n(VP (VBP have)\n(VP (VBN collected)\n(PP-CLR (IN on)\n(NP (DT those)(NNS assets)))))))))))))\n(, ,) ('' '')\n(NP-SBJ (PRP he) )\n(VP (VBD said)\n(S (-NONE- *T*-2) ))\n(. .) ))\nFigure D.9 A sentence from the Wall Street Journal portion of the LDC Penn Treebank.\nNote the use of the empty -NONE- nodes.\ncations) (Marcus et al. 1994, Bies et al. 1995). Figure D.9 shows examples of the\n-SBJ (surface subject) and -TMP (temporal phrase) tags. Figure D.8 shows in addi-\ntion the -PRD tag, which is used for predicates that are not VPs (the one in Fig. D.8\nis an ADJP). We\u2019ll return to the topic of grammatical function when we consider\ndependency grammars and parsing in Chapter 19.\nD.4.2 Treebanks as Grammars\nThe sentences in a treebank implicitly constitute a grammar of the language repre-\nsented by the corpus being annotated. For example, from the three parsed sentences\nin Fig. D.7 and Fig. D.9, we can extract each of the CFG rules in them. For simplic-\nity, let\u2019s strip off the rule suf\ufb01xes ( -SBJ and so on). The resulting grammar is shown\nin Fig. D.10.\nThe grammar used to parse the Penn Treebank is relatively \ufb02at, resulting in very\nmany and very long rules. For example, among the approximately 4,500 different\nrules for expanding VPs are separate rules for PP sequences of any length and every\npossible arrangement of verb arguments:\nVP!VBD PP\nVP!VBD PP PP\nVP!VBD PP PP PP\nVP!VBD PP PP PP PP\nVP!VB ADVP PP\nVP!VB PP ADVP\nVP!ADVP VB PP\nas well as even longer rules, such as\nVP!VBP PP PP PP PP PP ADVP PP D.4 \u2022 T REEBANKS 17\nGrammar Lexicon\nS!NP VP . PRP!wejhe\nS!NP VP DT!thejthatjthose\nS!\u201cS\u201d, NP VP . JJ!coldjemptyjfull\nS!-NONE- NN!skyj\ufb01rejlightj\ufb02ightjtomorrow\nNP!DT NN NNS!assets\nNP!DT NNS CC!and\nNP!NN CC NN IN!ofjatjuntiljon\nNP!CD RB CD!eleven\nNP!DT JJ , JJ NN RB!a.m.\nNP!PRP VB!arrivejhavejwait\nNP!-NONE- VBD!wasjsaid\nVP!MD VP VBP!have\nVP!VBD ADJP VBN!collected\nVP!VBD S MD!shouldjwould\nVP!VBN PP TO!to\nVP!VB S\nVP!VB SBAR\nVP!VBP VP\nVP!VBN PP\nVP!TO VP\nSBAR!IN S\nADJP!JJ PP\nPP!IN NP\nFigure D.10 A sample of the CFG grammar rules and lexical entries that would be ex-\ntracted from the three treebank sentences in Fig. D.7 and Fig. D.9.\nwhich comes from the VPmarked in italics:\nThis mostly happens because we go from football in the fall to lifting in the\nwinter to football again in the spring .\nSome of the many thousands of NPrules include\nNP!DT JJ NN\nNP!DT JJ NNS\nNP!DT JJ NN NN\nNP!DT JJ JJ NN\nNP!DT JJ CD NNS\nNP!RB DT JJ NN NN\nNP!RB DT JJ JJ NNS\nNP!DT JJ JJ NNP NNS\nNP!DT NNP NNP NNP NNP JJ NN\nNP!DT JJ NNP CC JJ JJ NN NNS\nNP!RB DT JJS NN NN SBAR\nNP!DT VBG JJ NNP NNP CC NNP\nNP!DT JJ NNS , NNS CC NN NNS NN\nNP!DT JJ JJ VBG NN NNP NNP FW NNP\nNP!NP JJ , JJ `` SBAR '' NNS\nThe last two of those rules, for example, come from the following two noun phrases:\n[DTThe] [JJstate-owned ] [JJindustrial ] [VBGholding ] [NNcompany ] [NNPInstituto ] [NNPNacional ]\n[FWde] [NNPIndustria ]\n[NPShearson\u2019s ] [JJeasy-to-\ufb01lm ],[JJblack-and-white ]\u201c[SBAR Where We Stand ]\u201d[NNScommercials ]\nViewed as a large grammar in this way, the Penn Treebank III Wall Street Journal\ncorpus, which contains about 1 million words, also has about 1 million non-lexical\nrule tokens, consisting of about 17,500 distinct rule types. 18 APPENDIX D \u2022 C ONSTITUENCY GRAMMARS\nS(dumped)\nVP(dumped)\nPP(into)\nNP(bin)\nNN(bin)\nbinDT(a)\naP\nintoNP(sacks)\nNNS(sacks)\nsacksVBD(dumped)\ndumpedNP(workers)\nNNS(workers)\nworkers\nFigure D.11 A lexicalized tree from Collins (1999).\nVarious facts about the treebank grammars, such as their large numbers of \ufb02at\nrules, pose problems for probabilistic parsing algorithms. For this reason, it is com-\nmon to make various modi\ufb01cations to a grammar extracted from a treebank. We\ndiscuss these further in Appendix C.\nD.4.3 Heads and Head-Finding\nWe suggested informally earlier that syntactic constituents could be associated with\na lexical head ;Nis the head of an NP,Vis the head of a VP. This idea of a head\nfor each constituent dates back to Bloom\ufb01eld 1914, and is central to the dependency\ngrammars and dependency parsing we\u2019ll introduce in Chapter 19. Heads are also\nimportant in probabilistic parsing (Appendix C) and in constituent-based grammar\nformalisms like Head-Driven Phrase Structure Grammar (Pollard and Sag, 1994)..\nIn one simple model of lexical heads, each context-free rule is associated with\na head (Charniak 1997, Collins 1999). The head is the word in the phrase that is\ngrammatically the most important. Heads are passed up the parse tree; thus, each\nnon-terminal in a parse tree is annotated with a single word, which is its lexical\nhead. Figure D.11 shows an example of such a tree from Collins (1999), in which\neach non-terminal is annotated with its head.\nFor the generation of such a tree, each CFG rule must be augmented to identify\none right-side constituent to be the head child. The headword for a node is then set to\nthe headword of its head child. Choosing these head children is simple for textbook\nexamples ( NNis the head of NP) but is complicated and indeed controversial for\nmost phrases. (Should the complementizer toor the verb be the head of an in\ufb01nite\nverb phrase?) Modern linguistic theories of syntax generally include a component\nthat de\ufb01nes heads (see, e.g., (Pollard and Sag, 1994)).\nAn alternative approach to \ufb01nding a head is used in most practical computational\nsystems. Instead of specifying head rules in the grammar itself, heads are identi\ufb01ed\ndynamically in the context of trees for speci\ufb01c sentences. In other words, once\na sentence is parsed, the resulting tree is walked to decorate each node with the\nappropriate head. Most current systems rely on a simple set of handwritten rules,\nsuch as a practical one for Penn Treebank grammars given in Collins (1999) but\ndeveloped originally by Magerman (1995). For example, the rule for \ufb01nding the\nhead of an NPis as follows (Collins, 1999, p. 238):\n\u2022 If the last word is tagged POS, return last-word. D.5 \u2022 G RAMMAR EQUIVALENCE AND NORMAL FORM 19\n\u2022 Else search from right to left for the \ufb01rst child which is an NN, NNP, NNPS, NX, POS,\nor JJR.\n\u2022 Else search from left to right for the \ufb01rst child which is an NP.\n\u2022 Else search from right to left for the \ufb01rst child which is a $, ADJP, or PRN.\n\u2022 Else search from right to left for the \ufb01rst child which is a CD.\n\u2022 Else search from right to left for the \ufb01rst child which is a JJ, JJS, RB or QP.\n\u2022 Else return the last word\nSelected other rules from this set are shown in Fig. D.12. For example, for VP\nrules of the form VP!Y1\u0001\u0001\u0001Yn, the algorithm would start from the left of Y1\u0001\u0001\u0001\nYnlooking for the \ufb01rst Yiof type TO; if no TOs are found, it would search for the\n\ufb01rstYiof type VBD; if no VBDs are found, it would search for a VBN, and so on.\nSee Collins (1999) for more details.\nParent Direction Priority List\nADJP Left NNS QP NN $ ADVP JJ VBN VBG ADJP JJR NP JJS DT FW RBR RBS\nSBAR RB\nADVP Right RB RBR RBS FW ADVP TO CD JJR JJ IN NP JJS NN\nPRN Left\nPRT Right RP\nQP Left $ IN NNS NN JJ RB DT CD NCD QP JJR JJS\nS Left TO IN VP S SBAR ADJP UCP NP\nSBAR Left WHNP WHPP WHADVP WHADJP IN DT S SQ SINV SBAR FRAG\nVP Left TO VBD VBN MD VBZ VB VBG VBP VP ADJP NN NNS NP\nFigure D.12 Some head rules from Collins (1999). The head rules are also called a head percolation table .\nD.5 Grammar Equivalence and Normal Form\nA formal language is de\ufb01ned as a (possibly in\ufb01nite) set of strings of words. This\nsuggests that we could ask if two grammars are equivalent by asking if they gener-\nate the same set of strings. In fact, it is possible to have two distinct context-free\ngrammars generate the same language.\nWe usually distinguish two kinds of grammar equivalence: weak equivalence\nandstrong equivalence . Two grammars are strongly equivalent if they generate the\nsame set of strings andif they assign the same phrase structure to each sentence\n(allowing merely for renaming of the non-terminal symbols). Two grammars are\nweakly equivalent if they generate the same set of strings but do not assign the same\nphrase structure to each sentence.\nIt is sometimes useful to have a normal form for grammars, in which each of normal form\nthe productions takes a particular form. For example, a context-free grammar is in\nChomsky normal form (CNF) (Chomsky, 1963) if it is \u000f-free and if in additionChomsky\nnormal form\neach production is either of the form A!B C orA!a. That is, the right-hand side\nof each rule either has two non-terminal symbols or one terminal symbol. Chomsky\nnormal form grammars are binary branching , that is they have binary trees (downbinary\nbranching\nto the prelexical nodes). We make use of this binary branching property in the CKY\nparsing algorithm in Chapter 18.\nAny context-free grammar can be converted into a weakly equivalent Chomsky\nnormal form grammar. For example, a rule of the form\nA!B C D 20 APPENDIX D \u2022 C ONSTITUENCY GRAMMARS\ncan be converted into the following two CNF rules (Exercise D. ??asks the reader to\nformulate the complete algorithm):\nA!B X\nX!C D\nSometimes using binary branching can actually produce smaller grammars. For\nexample, the sentences that might be characterized as\nVP -> VBD NP PP*\nare represented in the Penn Treebank by this series of rules:\nVP!VBD NP PP\nVP!VBD NP PP PP\nVP!VBD NP PP PP PP\nVP!VBD NP PP PP PP PP\n...\nbut could also be generated by the following two-rule grammar:\nVP!VBD NP PP\nVP!VP PP\nThe generation of a symbol A with a potentially in\ufb01nite sequence of symbols B with\na rule of the form A!A Bis known as Chomsky-adjunction .Chomsky-\nadjunction\nD.6 Summary\nThis chapter has introduced a number of fundamental concepts in syntax through\nthe use of context-free grammars .\n\u2022 In many languages, groups of consecutive words act as a group or a con-\nstituent , which can be modeled by context-free grammars (which are also\nknown as phrase-structure grammars ).\n\u2022 A context-free grammar consists of a set of rules orproductions , expressed\nover a set of non-terminal symbols and a set of terminal symbols. Formally,\na particular context-free language is the set of strings that can be derived\nfrom a particular context-free grammar .\n\u2022 Agenerative grammar is a traditional name in linguistics for a formal lan-\nguage that is used to model the grammar of a natural language.\n\u2022 There are many sentence-level grammatical constructions in English; declar-\native ,imperative ,yes-no question , and wh-question are four common types;\nthese can be modeled with context-free rules.\n\u2022 An English noun phrase can have determiners ,numbers ,quanti\ufb01ers , and\nadjective phrases preceding the head noun , which can be followed by a num-\nber of postmodi\ufb01ers ;gerundive andin\ufb01nitive VPs are common possibilities.\n\u2022Subjects in English agree with the main verb in person and number.\n\u2022 Verbs can be subcategorized by the types of complements they expect. Sim-\nple subcategories are transitive and intransitive ; most grammars include\nmany more categories than these.\n\u2022Treebanks of parsed sentences exist for many genres of English and for many\nlanguages. Treebanks can be searched with tree-search tools. BIBLIOGRAPHICAL AND HISTORICAL NOTES 21\n\u2022 Any context-free grammar can be converted to Chomsky normal form , in\nwhich the right-hand side of each rule has either two non-terminals or a single\nterminal.\nBibliographical and Historical Notes\nAccording to Percival (1976), the idea of breaking up a sentence into a hierarchy of\nconstituents appeared in the V\u00a8olkerpsychologie of the groundbreaking psychologist\nWilhelm Wundt (Wundt, 1900):\n...den sprachlichen Ausdruck f \u00a8ur die willk \u00a8urliche Gliederung einer Ge-\nsammtvorstellung in ihre in logische Beziehung zueinander gesetzten\nBestandteile\n[the linguistic expression for the arbitrary division of a total idea\ninto its constituent parts placed in logical relations to one another]\nWundt\u2019s idea of constituency was taken up into linguistics by Leonard Bloom-\n\ufb01eld in his early book An Introduction to the Study of Language (Bloom\ufb01eld, 1914).\nBy the time of his later book, Language (Bloom\ufb01eld, 1933), what was then called\n\u201cimmediate-constituent analysis\u201d was a well-established method of syntactic study\nin the United States. By contrast, traditional European grammar, dating from the\nClassical period, de\ufb01ned relations between words rather than constituents, and Eu-\nropean syntacticians retained this emphasis on such dependency grammars, the sub-\nject of Chapter 19.\nAmerican Structuralism saw a number of speci\ufb01c de\ufb01nitions of the immediate\nconstituent, couched in terms of their search for a \u201cdiscovery procedure\u201d: a method-\nological algorithm for describing the syntax of a language. In general, these attempt\nto capture the intuition that \u201cThe primary criterion of the immediate constituent\nis the degree in which combinations behave as simple units\u201d (Bazell, 1952/1966, p.\n284). The most well known of the speci\ufb01c de\ufb01nitions is Harris\u2019 idea of distributional\nsimilarity to individual units, with the substitutability test. Essentially, the method\nproceeded by breaking up a construction into constituents by attempting to substitute\nsimple structures for possible constituents\u2014if a substitution of a simple form, say,\nman, was substitutable in a construction for a more complex set (like intense young\nman), then the form intense young man was probably a constituent. Harris\u2019s test was\nthe beginning of the intuition that a constituent is a kind of equivalence class.\nThe \ufb01rst formalization of this idea of hierarchical constituency was the phrase-\nstructure grammar de\ufb01ned in Chomsky (1956) and further expanded upon (and\nargued against) in Chomsky (1957) and Chomsky (1956/1975). From this time on,\nmost generative linguistic theories were based at least in part on context-free gram-\nmars or generalizations of them (such as Head-Driven Phrase Structure Grammar\n(Pollard and Sag, 1994), Lexical-Functional Grammar (Bresnan, 1982), the Mini-\nmalist Program (Chomsky, 1995), and Construction Grammar (Kay and Fillmore,\n1999), inter alia); many of these theories used schematic context-free templates\nknown as X-bar schemata , which also relied on the notion of syntactic head.X-bar\nschemata\nShortly after Chomsky\u2019s initial work, the context-free grammar was reinvented\nby Backus (1959) and independently by Naur et al. (1960) in their descriptions of\nthe ALGOL programming language; Backus (1996) noted that he was in\ufb02uenced by\nthe productions of Emil Post and that Naur\u2019s work was independent of his (Backus\u2019) 22 APPENDIX D \u2022 C ONSTITUENCY GRAMMARS\nown. After this early work, a great number of computational models of natural\nlanguage processing were based on context-free grammars because of the early de-\nvelopment of ef\ufb01cient algorithms to parse these grammars (see Chapter 18).\nThre are various classes of extensions to CFGs, many designed to handle long-\ndistance dependencies in the syntax. (Other grammars instead treat long-distance-\ndependent items as being related semantically rather than syntactically (Kay and\nFillmore 1999, Culicover and Jackendoff 2005).\nOne extended formalism is Tree Adjoining Grammar (TAG) (Joshi, 1985).\nThe primary TAG data structure is the tree, rather than the rule. Trees come in two\nkinds: initial trees andauxiliary trees . Initial trees might, for example, represent\nsimple sentential structures, and auxiliary trees add recursion into a tree. Trees are\ncombined by two operations called substitution andadjunction . The adjunction\noperation handles long-distance dependencies. See Joshi (1985) for more details.\nTree Adjoining Grammar is a member of the family of mildly context-sensitive\nlanguages .\nWe mentioned on page 15 another way of handling long-distance dependencies,\nbased on the use of empty categories and co-indexing. The Penn Treebank uses\nthis model, which draws (in various Treebank corpora) from the Extended Standard\nTheory and Minimalism (Radford, 1997).\nReaders interested in the grammar of English should get one of the three large\nreference grammars of English: Huddleston and Pullum (2002), Biber et al. (1999),\nand Quirk et al. (1985).\nThere are many good introductory textbooks on syntax from different perspec-\ntives. Sag et al. (2003) is an introduction to syntax from a generative perspective, generative\nfocusing on the use of phrase-structure rules, uni\ufb01cation, and the type hierarchy in\nHead-Driven Phrase Structure Grammar. Van Valin, Jr. and La Polla (1997) is an\nintroduction from a functional perspective, focusing on cross-linguistic data and on functional\nthe functional motivation for syntactic structures.\nExercises\nD.1 Draw tree structures for the following ATIS phrases:\n1. Dallas\n2. from Denver\n3. after \ufb01ve p.m.\n4. arriving in Washington\n5. early \ufb02ights\n6. all redeye \ufb02ights\n7. on Thursday\n8. a one-way fare\n9. any delays in Denver\nD.2 Draw tree structures for the following ATIS sentences:\n1. Does American Airlines have a \ufb02ight between \ufb01ve a.m. and six a.m.?\n2. I would like to \ufb02y on American Airlines.\n3. Please repeat that.\n4. Does American 487 have a \ufb01rst-class section?\n5. I need to \ufb02y between Philadelphia and Atlanta.\n6. What is the fare from Atlanta to Denver? EXERCISES 23\n7. Is there an American Airlines \ufb02ight from Philadelphia to Dallas?\nD.3 Assume a grammar that has many VPrules for different subcategorizations,\nas expressed in Section D.3.4, and differently subcategorized verb rules like\nVerb-with-NP-complement . How would the rule for postnominal relative clauses\n(D.4) need to be modi\ufb01ed if we wanted to deal properly with examples like\nthe earliest \ufb02ight that you have ? Recall that in such examples the pronoun\nthatis the object of the verb get. Your rules should allow this noun phrase but\nshould correctly rule out the ungrammatical S *I get .\nD.4 Does your solution to the previous problem correctly model the NP the earliest\n\ufb02ight that I can get ? How about the earliest \ufb02ight that I think my mother\nwants me to book for her ? Hint: this phenomenon is called long-distance\ndependency .\nD.5 Write rules expressing the verbal subcategory of English auxiliaries; for ex-\nample, you might have a rule verb-with-bare-stem-VP-complement !can.\nD.6 NPs like Fortune\u2019s of\ufb01ce ormy uncle\u2019s marks are called possessive orgenitive possessive\ngenitive noun phrases. We can model possessive noun phrases by treating the sub-NP\nlikeFortune\u2019s ormy uncle\u2019s as a determiner of the following head noun. Write\ngrammar rules for English possessives. You may treat \u2019sas if it were a separate\nword (i.e., as if there were always a space before \u2019s).\nD.7 Page 8 discussed the need for a Wh-NP constituent. The simplest Wh-NP is\none of the Wh-pronouns (who, whom, whose, which ). The Wh-words what\nandwhich can be determiners: which four will you have? ,what credit do you\nhave with the Duke? Write rules for the different types of Wh-NP s. 24 Appendix D \u2022 Constituency Grammars\nBackus, J. W. 1959. The syntax and semantics of the\nproposed international algebraic language of the Zurich\nACM-GAMM Conference. Information Processing: Pro-\nceedings of the International Conference on Information\nProcessing, Paris . UNESCO.\nBackus, J. W. 1996. Transcript of question and answer ses-\nsion. In R. L. Wexelblat, ed., History of Programming\nLanguages , page 162. Academic Press.\nBazell, C. E. 1952/1966. The correspondence fallacy in\nstructural linguistics. In E. P. Hamp, F. W. Householder,\nand R. Austerlitz, eds, Studies by Members of the En-\nglish Department, Istanbul University (3), reprinted in\nReadings in Linguistics II (1966) , 271\u2013298. University of\nChicago Press.\nBiber, D., S. Johansson, G. Leech, S. Conrad, and E. Fine-\ngan. 1999. Longman Grammar of Spoken and Written\nEnglish . Pearson.\nBies, A., M. Ferguson, K. Katz, and R. MacIntyre. 1995.\nBracketing guidelines for Treebank II style Penn Tree-\nbank Project.\nBloom\ufb01eld, L. 1914. An Introduction to the Study of Lan-\nguage . Henry Holt and Company.\nBloom\ufb01eld, L. 1933. Language . University of Chicago\nPress.\nBresnan, J., ed. 1982. The Mental Representation of Gram-\nmatical Relations . MIT Press.\nCharniak, E. 1997. Statistical parsing with a context-free\ngrammar and word statistics. AAAI .\nChomsky, N. 1956. Three models for the description of\nlanguage. IRE Transactions on Information Theory ,\n2(3):113\u2013124.\nChomsky, N. 1956/1975. The Logical Structure of Linguistic\nTheory . Plenum.\nChomsky, N. 1957. Syntactic Structures . Mouton.\nChomsky, N. 1963. Formal properties of grammars. In R. D.\nLuce, R. Bush, and E. Galanter, eds, Handbook of Math-\nematical Psychology , volume 2, 323\u2013418. Wiley.\nChomsky, N. 1995. The Minimalist Program . MIT Press.\nCollins, M. 1999. Head-Driven Statistical Models for Natu-\nral Language Parsing . Ph.D. thesis, University of Penn-\nsylvania, Philadelphia.\nCulicover, P. W. and R. Jackendoff. 2005. Simpler Syntax .\nOxford University Press.\nGazdar, G., E. Klein, G. K. Pullum, and I. A. Sag. 1985.\nGeneralized Phrase Structure Grammar . Blackwell.\nHarris, Z. S. 1946. From morpheme to utterance. Language ,\n22(3):161\u2013183.\nHemphill, C. T., J. Godfrey, and G. Doddington. 1990. The\nATIS spoken language systems pilot corpus. Speech and\nNatural Language Workshop .\nHopcroft, J. E. and J. D. Ullman. 1979. Introduction to Au-\ntomata Theory, Languages, and Computation . Addison-\nWesley.\nHuddleston, R. and G. K. Pullum. 2002. The Cambridge\nGrammar of the English Language . Cambridge Univer-\nsity Press.Joshi, A. K. 1985. Tree adjoining grammars: How\nmuch context-sensitivity is required to provide reasonable\nstructural descriptions? In D. R. Dowty, L. Karttunen,\nand A. Zwicky, eds, Natural Language Parsing , 206\u2013250.\nCambridge University Press.\nKay, P. and C. J. Fillmore. 1999. Grammatical constructions\nand linguistic generalizations: The What\u2019s X Doing Y?\nconstruction. Language , 75(1):1\u201333.\nMagerman, D. M. 1995. Statistical decision-tree models for\nparsing. ACL.\nMarcus, M. P., G. Kim, M. A. Marcinkiewicz, R. MacIn-\ntyre, A. Bies, M. Ferguson, K. Katz, and B. Schasberger.\n1994. The Penn Treebank: Annotating predicate argu-\nment structure. HLT.\nMarcus, M. P., B. Santorini, and M. A. Marcinkiewicz. 1993.\nBuilding a large annotated corpus of English: The Penn\ntreebank. Computational Linguistics , 19(2):313\u2013330.\nNaur, P., J. W. Backus, F. L. Bauer, J. Green, C. Katz,\nJ. McCarthy, A. J. Perlis, H. Rutishauser, K. Samelson,\nB. Vauquois, J. H. Wegstein, A. van Wijnagaarden, and\nM. Woodger. 1960. Report on the algorithmic language\nALGOL 60. CACM , 3(5):299\u2013314. Revised in CACM\n6:1, 1-17, 1963.\nNivre, J., M.-C. de Marneffe, F. Ginter, Y . Goldberg, J. Haji \u02c7c,\nC. D. Manning, R. McDonald, S. Petrov, S. Pyysalo,\nN. Silveira, R. Tsarfaty, and D. Zeman. 2016. Univer-\nsal Dependencies v1: A multilingual treebank collection.\nLREC .\nPercival, W. K. 1976. On the historical source of immedi-\nate constituent analysis. In J. D. McCawley, ed., Syntax\nand Semantics Volume 7, Notes from the Linguistic Un-\nderground , 229\u2013242. Academic Press.\nPollard, C. and I. A. Sag. 1994. Head-Driven Phrase Struc-\nture Grammar . University of Chicago Press.\nQuirk, R., S. Greenbaum, G. Leech, and J. Svartvik. 1985.\nA Comprehensive Grammar of the English Language .\nLongman.\nRadford, A. 1997. Syntactic Theory and the Structure of\nEnglish: A Minimalist Approach . Cambridge University\nPress.\nSag, I. A., T. Wasow, and E. M. Bender, eds. 2003. Syntac-\ntic Theory: A Formal Introduction . CSLI Publications,\nStanford, CA.\nVan Valin, Jr., R. D. and R. La Polla. 1997. Syntax: Structure,\nMeaning, and Function . Cambridge University Press.\nWundt, W. 1900. V\u00a8olkerpsychologie: eine Untersuchung der\nEntwicklungsgesetze von Sprache, Mythus, und Sitte . W.\nEngelmann, Leipzig. Band II: Die Sprache, Zweiter Teil.", "E.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\nECombinatory Categorial\nGrammar\nIn this chapter, we provide an overview of categorial grammar (Ajdukiewicz 1935,categorial\ngrammar\nBar-Hillel 1953), an early lexicalized grammar model, as well as an important mod-\nern extension, combinatory categorial grammar , or CCG (Steedman 1996, Steed-combinatory\ncategorial\ngrammarman 1989, Steedman 2000). CCG is a heavily lexicalized approach motivated by\nboth syntactic and semantic considerations. It is an exemplar of a set of computa-\ntionally relevant approaches to grammar that emphasize putting grammatical infor-\nmation in a rich lexicon, including Lexical-Functional Grammar (LFG) (Bresnan,\n1982), Head-Driven Phrase Structure Grammar (HPSG) (Pollard and Sag, 1994),\nand Tree-Adjoining Grammar (TAG) (Joshi, 1985).\nThe categorial approach consists of three major elements: a set of categories,\na lexicon that associates words with categories, and a set of rules that govern how\ncategories combine in context.\nE.1 CCG Categories\nCategories are either atomic elements or single-argument functions that return a cat-\negory as a value when provided with a desired category as argument. More formally,\nwe can de\ufb01ne C, a set of categories for a grammar as follows:\n\u2022A\u0012C, where Ais a given set of atomic elements\n\u2022 (X=Y), (XnY)2C, ifX,Y2C\nThe slash notation shown here is used to de\ufb01ne the functions in the grammar.\nIt speci\ufb01es the type of the expected argument, the direction it is expected be found,\nand the type of the result. Thus, ( X=Y) is a function that seeks a constituent of type\nYto its right and returns a value of X; (XnY) is the same except it seeks its argument\nto the left.\nThe set of atomic categories is typically very small and includes familiar el-\nements such as sentences and noun phrases. Functional categories include verb\nphrases and complex noun phrases among others.\nE.2 The Lexicon\nThe lexicon in a categorial approach consists of assignments of categories to words.\nThese assignments can either be to atomic or functional categories, and due to lexical\nambiguity words can be assigned to multiple categories. Consider the following 2APPENDIX E \u2022 C OMBINATORY CATEGORIAL GRAMMAR\nsample lexical entries.\n\ufb02ight : N\nMiami : NP\ncancel :(SnNP)=NP\nNouns and proper nouns like \ufb02ight andMiami are assigned to atomic categories,\nre\ufb02ecting their typical role as arguments to functions. On the other hand, a transitive\nverb like cancel is assigned the category ( SnNP)=NP: a function that seeks an NPon\nits right and returns as its value a function with the type ( SnNP). This function can,\nin turn, combine with an NPon the left, yielding an Sas the result. This captures\nsubcategorization information with a computationally useful, internal structure.\nDitransitive verbs like give, which expect two arguments after the verb, would\nhave the category (( SnNP)=NP)=NP: a function that combines with an NPon its\nright to yield yet another function corresponding to the transitive verb ( SnNP)=NP\ncategory such as the one given above for cancel .\nE.3 Rules\nThe rules of a categorial grammar specify how functions and their arguments com-\nbine. The following two rule templates constitute the basis for all categorial gram-\nmars.\nX=Y Y)X (E.1)\nY XnY)X (E.2)\nThe \ufb01rst rule applies a function to its argument on the right, while the second\nlooks to the left for its argument. We\u2019ll refer to the \ufb01rst as forward function appli-\ncation , and the second as backward function application . The result of applying\neither of these rules is the category speci\ufb01ed as the value of the function being ap-\nplied.\nGiven these rules and a simple lexicon, let\u2019s consider an analysis of the sentence\nUnited serves Miami . Assume that serves is a transitive verb with the category\n(SnNP)=NPand that United andMiami are both simple NPs. Using both forward\nand backward function application, the derivation would proceed as follows:\nUnited serves Miami\nNP (SnNP)=NP NP\n>SnNP\n<S\nCategorial grammar derivations are illustrated growing down from the words,\nrule applications are illustrated with a horizontal line that spans the elements in-\nvolved, with the type of the operation indicated at the right end of the line. In this\nexample, there are two function applications: one forward function application indi-\ncated by the >that applies the verb serves to the NPon its right, and one backward\nfunction application indicated by the <that applies the result of the \ufb01rst to the NP\nUnited on its left.\nEnglish permits the coordination of two constituents of the same type, resulting\nin a new constituent of the same type. The following rule provides the mechanism E.3 \u2022 R ULES 3\nto handle such examples.\nX CONJ X)X (E.3)\nThis rule states that when two constituents of the same category are separated by a\nconstituent of type CONJ they can be combined into a single larger constituent of\nthe same type. The following derivation illustrates the use of this rule.\nWe flew to Geneva and drove to Chamonix\nNP(SnNP)=PP PP =NP NP CONJ (SnNP)=PP PP =NP NP\n> >PP PP\n> >SnNP S nNP\n<F>SnNP\n<S\nHere the two SnNPconstituents are combined via the conjunction operator <F>\nto form a larger constituent of the same type, which can then be combined with the\nsubject NPvia backward function application.\nThese examples illustrate the lexical nature of the categorial grammar approach.\nThe grammatical facts about a language are largely encoded in the lexicon, while the\nrules of the grammar are boiled down to a set of three rules. Unfortunately, the basic\ncategorial approach does not give us any more expressive power than we had with\ntraditional CFG rules; it just moves information from the grammar to the lexicon. To\nmove beyond these limitations CCG includes operations that operate over functions.\nThe \ufb01rst pair of operators permit us to compose adjacent functions.\nX=Y Y=Z)X=Z (E.4)\nYnZ XnY)XnZ (E.5)\nThe \ufb01rst rule, called forward composition , can be applied to adjacent con-forward\ncomposition\nstituents where the \ufb01rst is a function seeking an argument of type Yto its right, and\nthe second is a function that provides Yas a result. This rule allows us to compose\nthese two functions into a single one with the type of the \ufb01rst constituent and the\nargument of the second. Although the notation is a little awkward, the second rule,\nbackward composition is the same, except that we\u2019re looking to the left instead ofbackward\ncomposition\nto the right for the relevant arguments. Both kinds of composition are signalled by a\nBin CCG diagrams, accompanied by a <or>to indicate the direction.\nThe next operator is type raising . Type raising elevates simple categories to the type raising\nstatus of functions. More speci\ufb01cally, type raising takes a category and converts it\nto a function that seeks as an argument a function that takes the original category\nas its argument. The following schema show two versions of type raising: one for\narguments to the right, and one for the left.\nX)T=(TnX) (E.6)\nX)Tn(T=X) (E.7)\nThe category Tin these rules can correspond to any of the atomic or functional\ncategories already present in the grammar.\nA particularly useful example of type raising transforms a simple NPargument\nin subject position to a function that can compose with a following VP. To see how 4APPENDIX E \u2022 C OMBINATORY CATEGORIAL GRAMMAR\nthis works, let\u2019s revisit our earlier example of United serves Miami . Instead of clas-\nsifying United as an NPwhich can serve as an argument to the function attached to\nserve , we can use type raising to reinvent it as a function in its own right as follows.\nNP)S=(SnNP)\nCombining this type-raised constituent with the forward composition rule (E.4) per-\nmits the following alternative to our previous derivation.\nUnited serves Miami\nNP (SnNP)=NP NP\n>TS=(SnNP)\n>BS=NP\n>S\nBy type raising United toS=(SnNP), we can compose it with the transitive verb\nserves to yield the (S=NP)function needed to complete the derivation.\nThere are several interesting things to note about this derivation. First, it pro-\nvides a left-to-right, word-by-word derivation that more closely mirrors the way\nhumans process language. This makes CCG a particularly apt framework for psy-\ncholinguistic studies. Second, this derivation involves the use of an intermediate\nunit of analysis, United serves , that does not correspond to a traditional constituent\nin English. This ability to make use of such non-constituent elements provides CCG\nwith the ability to handle the coordination of phrases that are not proper constituents,\nas in the following example.\n(E.8) We \ufb02ew IcelandAir to Geneva and SwissAir to London.\nHere, the segments that are being coordinated are IcelandAir to Geneva and\nSwissAir to London , phrases that would not normally be considered constituents, as\ncan be seen in the following standard derivation for the verb phrase \ufb02ew IcelandAir\nto Geneva .\n\ufb02ew IcelandAir to Geneva\n(VP=PP)=NP NP PP =NP NP\n> >VP=PP PP\n>VP\nIn this derivation, there is no single constituent that corresponds to IcelandAir\nto Geneva , and hence no opportunity to make use of the <F>operator. Note that\ncomplex CCG categories can get a little cumbersome, so we\u2019ll use VPas a shorthand\nfor (SnNP) in this and the following derivations.\nThe following alternative derivation provides the required element through the\nuse of both backward type raising (E.7) and backward function composition (E.5).\n\ufb02ew IcelandAir to Geneva\n(V P=PP)=NP NP PP =NP NP\n<T >(V P=PP)n((V P=PP)=NP) PP\n<TV Pn(V P=PP)\n<BV Pn((V P=PP)=NP)\n<V P\nApplying the same analysis to SwissAir to London satis\ufb01es the requirements for\nthe<F>operator, yielding the following derivation for our original example (E.8). E.4 \u2022 CCG BANK 5\n\ufb02ew IcelandAir to Geneva and SwissAir to London\n(V P=PP)=NP NP PP =NP NP CONJ NP PP =NP NP\n<T > <T >(V P=PP)n((V P=PP)=NP) PP (V P=PP)n((V P=PP)=NP) PP\n<T <TV Pn(V P=PP) V Pn(V P=PP)\n< <V Pn((V P=PP)=NP) V Pn((V P=PP)=NP)\n<F>V Pn((V P=PP)=NP)\n<V P\nFinally, let\u2019s examine how these advanced operators can be used to handle long-\ndistance dependencies (also referred to as syntactic movement or extraction). As\nmentioned in Appendix D, long-distance dependencies arise from many English\nconstructions including wh-questions, relative clauses, and topicalization. What\nthese constructions have in common is a constituent that appears somewhere dis-\ntant from its usual, or expected, location. Consider the following relative clause as\nan example.\nthe \ufb02ight that United diverted\nHere, divert is a transitive verb that expects two NParguments, a subject NPto its\nleft and a direct object NPto its right; its category is therefore ( SnNP)=NP. However,\nin this example the direct object the \ufb02ight has been \u201cmoved\u201d to the beginning of the\nclause, while the subject United remains in its normal position. What is needed is a\nway to incorporate the subject argument, while dealing with the fact that the \ufb02ight is\nnot in its expected location.\nThe following derivation accomplishes this, again through the combined use of\ntype raising and function composition.\nthe \ufb02ight that United diverted\nNP=N N (NPnNP)=(S=NP) NP (SnNP)=NP\n> >TNP S =(SnNP)\n>BS=NP\n>NPnNP\n<NP\nAs we saw with our earlier examples, the \ufb01rst step of this derivation is type raising\nUnited to the category S=(SnNP) allowing it to combine with diverted via forward\ncomposition. The result of this composition is S=NPwhich preserves the fact that we\nare still looking for an NPto \ufb01ll the missing direct object. The second critical piece\nis the lexical category assigned to the word that: (NPnNP)=(S=NP). This function\nseeks a verb phrase missing an argument to its right, and transforms it into an NP\nseeking a missing element to its left, precisely where we \ufb01nd the \ufb02ight .\nE.4 CCGbank\nAs with phrase-structure approaches, treebanks play an important role in CCG-\nbased approaches to parsing. CCGbank (Hockenmaier and Steedman, 2007) is the\nlargest and most widely used CCG treebank. It was created by automatically trans-\nlating phrase-structure trees from the Penn Treebank via a rule-based approach. The\nmethod produced successful translations of over 99% of the trees in the Penn Tree-\nbank resulting in 48,934 sentences paired with CCG derivations. It also provides a 6APPENDIX E \u2022 C OMBINATORY CATEGORIAL GRAMMAR\nlexicon of 44,000 words with over 1200 categories. Appendix C will discuss how\nthese resources can be used to train CCG parsers.\nE.5 Ambiguity in CCG\nAs is always the case in parsing, managing ambiguity is the key to successful CCG\nparsing. The dif\ufb01culties with CCG parsing arise from the ambiguity caused by the\nlarge number of complex lexical categories combined with the very general nature of\nthe grammatical rules. To see some of the ways that ambiguity arises in a categorial\nframework, consider the following example.\n(E.9) United diverted the \ufb02ight to Reno.\nOur grasp of the role of the \ufb02ight in this example depends on whether the prepo-\nsitional phrase to Reno is taken as a modi\ufb01er of the \ufb02ight , as a modi\ufb01er of the entire\nverb phrase, or as a potential second argument to the verb divert . In a context-free\ngrammar approach, this ambiguity would manifest itself as a choice among the fol-\nlowing rules in the grammar.\nNominal!Nominal PP\nVP!VP PP\nVP!Verb NP PP\nIn a phrase-structure approach we would simply assign the word toto the cate-\ngory Pallowing it to combine with Reno to form a prepositional phrase. The sub-\nsequent choice of grammar rules would then dictate the ultimate derivation. In the\ncategorial approach, we can associate towith distinct categories to re\ufb02ect the ways\nin which it might interact with other elements in a sentence. The fairly abstract\ncombinatoric rules would then sort out which derivations are possible. Therefore,\nthe source of ambiguity arises not from the grammar but rather from the lexicon.\nLet\u2019s see how this works by considering several possible derivations for this\nexample. To capture the case where the prepositional phrase to Reno modi\ufb01es the\n\ufb02ight , we assign the preposition tothe category (NPnNP)=NP, which gives rise to\nthe following derivation.\nUnited diverted the \ufb02ight to Reno\nNP (SnNP)=NP NP =N N (NPnNP)=NP NP\n> >NP NP nNP\n<NP\n>SnNP\n<S\nHere, the category assigned to toexpects to \ufb01nd two arguments: one to the right as\nwith a traditional preposition, and one to the left that corresponds to the NPto be\nmodi\ufb01ed.\nAlternatively, we could assign toto the category (SnS)=NP, which permits the\nfollowing derivation where to Reno modi\ufb01es the preceding verb phrase. E.6 \u2022 CCG P ARSING 7\nUnited diverted the \ufb02ight to Reno\nNP (SnNP)=NP NP =N N (SnS)=NP NP\n> >NP SnS\n>SnNP\n<BSnNP\n<S\nA third possibility is to view divert as a ditransitive verb by assigning it to the\ncategory ((SnNP)=PP)=NP, while treating to Reno as a simple prepositional phrase.\nUnited diverted the \ufb02ight to Reno\nNP ((SnNP)=PP)=NP NP =N N PP =NP NP\n> >NP PP\n>(SnNP)=PP\n>SnNP\n<S\nWhile CCG parsers are still subject to ambiguity arising from the choice of gram-\nmar rules, including the kind of spurious ambiguity discussed above, it should be\nclear that the choice of lexical categories is the primary problem to be addressed in\nCCG parsing.\nE.6 CCG Parsing\nSince the rules in combinatory grammars are either binary or unary, a bottom-up,\ntabular approach based on the CKY algorithm should be directly applicable to CCG\nparsing. Unfortunately, the large number of lexical categories available for each\nword, combined with the promiscuity of CCG\u2019s combinatoric rules, leads to an ex-\nplosion in the number of (mostly useless) constituents added to the parsing table.\nThe key to managing this explosion of zombie constituents is to accurately assess\nand exploit the most likely lexical categories possible for each word\u2014a process\ncalled supertagging.\nThese following sections describe an approach to CCG parsing that make use of\nsupertags, structuring the parsing process as a heuristic search through the use of the\nA* algorithm.\nE.6.1 Supertagging\nChapter 17 introduced the task of part-of-speech tagging, the process of assigning\nthe correct lexical category to each word in a sentence. Supertagging is the corre- supertagging\nsponding task for highly lexicalized grammar frameworks, where the assigned tags\noften dictate much of the derivation for a sentence (Bangalore and Joshi, 1999).\nCCG supertaggers rely on treebanks such as CCGbank to provide both the over-\nall set of lexical categories as well as the allowable category assignments for each\nword in the lexicon. CCGbank includes over 1000 lexical categories, however, in\npractice, most supertaggers limit their tagsets to those tags that occur at least 10 8APPENDIX E \u2022 C OMBINATORY CATEGORIAL GRAMMAR\ntimes in the training corpus. This results in a total of around 425 lexical categories\navailable for use in the lexicon. Note that even this smaller number is large in con-\ntrast to the 45 POS types used by the Penn Treebank tagset.\nAs with traditional part-of-speech tagging, the standard approach to building a\nCCG supertagger is to use supervised machine learning to build a sequence labeler\nfrom hand-annotated training data. To \ufb01nd the most likely sequence of tags given a\nsentence, it is most common to use a neural sequence model, either RNN or Trans-\nformer.\nIt\u2019s also possible, however, to use the CRF tagging model described in Chap-\nter 17, using similar features; the current word wi, its surrounding words within\nlwords, local POS tags and character suf\ufb01xes, and the supertag from the prior\ntimestep, training by maximizing log-likelihood of the training corpus and decoding\nvia the Viterbi algorithm as described in Chapter 17.\nUnfortunately the large number of possible supertags combined with high per-\nword ambiguity leads the naive CRF algorithm to error rates that are too high for\npractical use in a parser. The single best tag sequence \u02c6Twill typically contain too\nmany incorrect tags for effective parsing to take place. To overcome this, we instead\nreturn a probability distribution over the possible supertags for each word in the\ninput. The following table illustrates an example distribution for a simple sentence,\nin which each column represents the probability of each supertag for a given word\nin the context of the input sentence . The \u201c...\u201d represent all the remaining supertags\npossible for each word.\nUnited serves Denver\nN=N: 0.4 (SnNP)=NP: 0.8 NP: 0.9\nNP: 0.3 N: 0.1 N=N: 0.05\nS=S: 0.1 ... ...\nSnS: .05\n...\nTo get the probability of each possible word/tag pair, we\u2019ll need to sum the\nprobabilities of all the supertag sequences that contain that tag at that location. This\ncan be done with the forward-backward algorithm that is also used to train the CRF,\ndescribed in Appendix A.\nE.6.2 CCG Parsing using the A* Algorithm\nThe A* algorithm is a heuristic search method that employs an agenda to \ufb01nd an\noptimal solution. Search states representing partial solutions are added to an agenda\nbased on a cost function, with the least-cost option being selected for further ex-\nploration at each iteration. When a state representing a complete solution is \ufb01rst\nselected from the agenda, it is guaranteed to be optimal and the search terminates.\nThe A* cost function, f(n), is used to ef\ufb01ciently guide the search to a solution.\nThe f-cost has two components: g(n), the exact cost of the partial solution repre-\nsented by the state n, and h(n)a heuristic approximation of the cost of a solution\nthat makes use of n. When h(n)satis\ufb01es the criteria of not overestimating the actual\ncost, A* will \ufb01nd an optimal solution. Not surprisingly, the closer the heuristic can\nget to the actual cost, the more effective A* is at \ufb01nding a solution without having\nto explore a signi\ufb01cant portion of the solution space.\nWhen applied to parsing, search states correspond to edges representing com-\npleted constituents. Each edge speci\ufb01es a constituent\u2019s start and end positions, its E.6 \u2022 CCG P ARSING 9\ngrammatical category, and its f-cost. Here, the gcomponent represents the current\ncost of an edge and the hcomponent represents an estimate of the cost to complete\na derivation that makes use of that edge. The use of A* for phrase structure parsing\noriginated with Klein and Manning (2003), while the CCG approach presented here\nis based on the work of Lewis and Steedman (2014).\nUsing information from a supertagger, an agenda and a parse table are initial-\nized with states representing all the possible lexical categories for each word in the\ninput, along with their f-costs. The main loop removes the lowest cost edge from\nthe agenda and tests to see if it is a complete derivation. If it re\ufb02ects a complete\nderivation it is selected as the best solution and the loop terminates. Otherwise, new\nstates based on the applicable CCG rules are generated, assigned costs, and entered\ninto the agenda to await further processing. The loop continues until a complete\nderivation is discovered, or the agenda is exhausted, indicating a failed parse. The\nalgorithm is given in Fig. E.1.\nfunction CCG-AS TAR-PARSE (words )returns table orfailure\nsupertags SUPERTAGGER (words )\nfori from 1toLENGTH (words )do\nfor all fAj(words [i];A;score)2supertags g\nedge MAKEEDGE(i\u00001,i,A,score )\ntable INSERT EDGE(table ,edge )\nagenda INSERT EDGE(agenda ,edge )\nloop do\nifEMPTY ?(agenda )return failure\ncurrent POP(agenda )\nifCOMPLETED PARSE ?(current )return table\ntable INSERT EDGE(table ,current )\nfor each ruleinAPPLICABLE RULES (current )do\nsuccessor APPLY (rule,current )\nifsuccessor not2inagenda orchart\nagenda INSERT EDGE(agenda ,successor )\nelse if successor2agenda with higher cost\nagenda REPLACE EDGE(agenda ,successor )\nFigure E.1 A*-based CCG parsing.\nE.6.3 Heuristic Functions\nBefore we can de\ufb01ne a heuristic function for our A* search, we need to decide how\nto assess the quality of CCG derivations. We\u2019ll make the simplifying assumption\nthat the probability of a CCG derivation is just the product of the probability of\nthe supertags assigned to the words in the derivation, ignoring the rules used in the\nderivation. More formally, given a sentence Sand derivation Dthat contains supertag\nsequence T, we have:\nP(D;S) = P(T;S) (E.10)\n=nY\ni=1P(tijsi) (E.11) 10 APPENDIX E \u2022 C OMBINATORY CATEGORIAL GRAMMAR\nTo better \ufb01t with the traditional A* approach, we\u2019d prefer to have states scored by\na cost function where lower is better (i.e., we\u2019re trying to minimize the cost of a\nderivation). To achieve this, we\u2019ll use negative log probabilities to score deriva-\ntions; this results in the following equation, which we\u2019ll use to score completed\nCCG derivations.\nP(D;S) = P(T;S) (E.12)\n=nX\ni=1\u0000logP(tijsi) (E.13)\nGiven this model, we can de\ufb01ne our f-cost as follows. The f-cost of an edge is\nthe sum of two components: g(n), the cost of the span represented by the edge, and\nh(n), the estimate of the cost to complete a derivation containing that edge (these\nare often referred to as the inside andoutside costs ). We\u2019ll de\ufb01ne g(n)for an edge\nusing Equation E.13. That is, it is just the sum of the costs of the supertags that\ncomprise the span.\nForh(n), we need a score that approximates but never overestimates the actual\ncost of the \ufb01nal derivation. A simple heuristic that meets this requirement assumes\nthat each of the words in the outside span will be assigned its most probable su-\npertag . If these are the tags used in the \ufb01nal derivation, then its score will equal\nthe heuristic. If any other tags are used in the \ufb01nal derivation the f-cost will be\nhigher since the new tags must have higher costs, thus guaranteeing that we will not\noverestimate.\nPutting this all together, we arrive at the following de\ufb01nition of a suitable f-cost\nfor an edge.\nf(wi;j;ti;j) = g(wi;j)+h(wi;j) (E.14)\n=jX\nk=i\u0000logP(tkjwk)+\ni\u00001X\nk=1min\nt2tags(\u0000logP(tjwk))+NX\nk=j+1min\nt2tags(\u0000logP(tjwk))\nAs an example, consider an edge representing the word serves with the supertag N\nin the following example.\n(E.15) United serves Denver.\nTheg-cost for this edge is just the negative log probability of this tag, \u0000log10(0:1),\nor 1. The outside h-cost consists of the most optimistic supertag assignments for\nUnited andDenver , which are N=NandNPrespectively. The resulting f-cost for\nthis edge is therefore 1.443.\nE.6.4 An Example\nFig. E.2 shows the initial agenda and the progress of a complete parse for this ex-\nample. After initializing the agenda and the parse table with information from the\nsupertagger, it selects the best edge from the agenda\u2014the entry for United with the\ntagN=Nand f-cost 0.591. This edge does not constitute a complete parse and is\ntherefore used to generate new states by applying all the relevant grammar rules. In\nthis case, applying forward application to United: N/N andserves: N results in the\ncreation of the edge United serves: N[0,2], 1.795 to the agenda. E.6 \u2022 CCG P ARSING 11\nSkipping ahead, at the third iteration an edge representing the complete deriva-\ntionUnited serves Denver, S[0,3], .716 is added to the agenda. However, the algo-\nrithm does not terminate at this point since the cost of this edge (.716) does not place\nit at the top of the agenda. Instead, the edge representing Denver with the category\nNPis popped. This leads to the addition of another edge to the agenda (type-raising\nDenver ). Only after this edge is popped and dealt with does the earlier state repre-\nsenting a complete derivation rise to the top of the agenda where it is popped, goal\ntested, and returned as a solution.\nUnited serves: N[0,2]1.795\nUnited: N/N.591\nDenver: N/N2.494\nDenver: N1.795\nserves: N1.494\nUnited: S\\S1.494\nUnited: S/S1.1938\nUnited: NP.716\nDenver: NP.591\nserves: (S\\NP)/NP.591\nserves Denver: S\\NP[1,3].591\nUnited serves Denver: S[0,3].716\nDenver: S/(S\\NP)[0,1].591\n1\n23\n456Initial AgendaGoal State\n\u2026\nS: 0.716S/NP: 0.591Unitedserves[0,1][0,2][0,3][1,2][1,3][2,3]N/N: 0.591NP: 0.716S/S: 1.1938S\\S: 1.494\u2026Denver\n(S\\NP)/NP: 0.591N: 1.494\u2026NP: 0.591N: 1.795N/N: 2.494\u2026N: 1.795\nFigure E.2 Example of an A* search for the example \u201cUnited serves Denver\u201d. The circled numbers on the\nblue boxes indicate the order in which the states are popped from the agenda. The costs in each state are based\non f-costs using negative log10probabilities.\nThe effectiveness of the A* approach is re\ufb02ected in the coloring of the states in\nFig. E.2 as well as the \ufb01nal parsing table. The edges shown in blue (including all the 12 APPENDIX E \u2022 C OMBINATORY CATEGORIAL GRAMMAR\ninitial lexical category assignments not explicitly shown) re\ufb02ect states in the search\nspace that never made it to the top of the agenda and, therefore, never contributed any\nedges to the \ufb01nal table. This is in contrast to the PCKY approach where the parser\nsystematically \ufb01lls the parse table with all possible constituents for all possible spans\nin the input, \ufb01lling the table with myriad constituents that do not contribute to the\n\ufb01nal analysis.\nE.7 Summary\nThis chapter has introduced combinatory categorial grammar (CCG):\n\u2022 Combinatorial categorial grammar (CCG) is a computationally relevant lexi-\ncalized approach to grammar and parsing.\n\u2022 Much of the dif\ufb01culty in CCG parsing is disambiguating the highly rich lexical\nentries, and so CCG parsers are generally based on supertagging .\n\u2022 Supertagging is the equivalent of part-of-speech tagging in highly lexicalized\ngrammar frameworks. The tags are very grammatically rich and dictate much\nof the derivation for a sentence.\nBibliographical and Historical Notes Bibliographical and Historical Notes 13\nAjdukiewicz, K. 1935. Die syntaktische Konnexit \u00a8at.Stu-\ndia Philosophica , 1:1\u201327. English translation \u201cSyntactic\nConnexion\u201d by H. Weber in McCall, S. (Ed.) 1967. Polish\nLogic , pp. 207\u2013231, Oxford University Press.\nBangalore, S. and A. K. Joshi. 1999. Supertagging: An\napproach to almost parsing. Computational Linguistics ,\n25(2):237\u2013265.\nBar-Hillel, Y . 1953. A quasi-arithmetical notation for syn-\ntactic description. Language , 29:47\u201358.\nBresnan, J., ed. 1982. The Mental Representation of Gram-\nmatical Relations . MIT Press.\nHockenmaier, J. and M. Steedman. 2007. CCGbank: a cor-\npus of CCG derivations and dependency structures ex-\ntracted from the penn treebank. Computational Linguis-\ntics, 33(3):355\u2013396.\nJoshi, A. K. 1985. Tree adjoining grammars: How\nmuch context-sensitivity is required to provide reasonable\nstructural descriptions? In D. R. Dowty, L. Karttunen,\nand A. Zwicky, eds, Natural Language Parsing , 206\u2013250.\nCambridge University Press.\nKlein, D. and C. D. Manning. 2003. A* parsing: Fast exact\nViterbi parse selection. HLT-NAACL .\nLewis, M. and M. Steedman. 2014. A* ccg parsing with a\nsupertag-factored model. EMNLP .\nPollard, C. and I. A. Sag. 1994. Head-Driven Phrase Struc-\nture Grammar . University of Chicago Press.\nSteedman, M. 1989. Constituency and coordination in a\ncombinatory grammar. In M. R. Baltin and A. S. Kroch,\neds, Alternative Conceptions of Phrase Structure , 201\u2013\n231. University of Chicago.\nSteedman, M. 1996. Surface Structure and Interpretation .\nMIT Press. Linguistic Inquiry Monograph, 30.\nSteedman, M. 2000. The Syntactic Process . The MIT Press.", "F.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\nFLogical Representations of\nSentence Meaning\nISHMAEL :Surely all this is not without meaning.\nHerman Melville, Moby Dick\nIn this chapter we introduce the idea that the meaning of linguistic expressions can\nbe captured in formal structures called meaning representations . Consider tasksmeaning\nrepresentations\nthat require some form of semantic processing, like learning to use a new piece of\nsoftware by reading the manual, deciding what to order at a restaurant by reading\na menu, or following a recipe. Accomplishing these tasks requires representations\nthat link the linguistic elements to the necessary non-linguistic knowledge of the\nworld . Reading a menu and deciding what to order, giving advice about where to\ngo to dinner, following a recipe, and generating new recipes all require knowledge\nabout food and its preparation, what people like to eat, and what restaurants are like.\nLearning to use a piece of software by reading a manual, or giving advice on using\nsoftware, requires knowledge about the software and similar apps, computers, and\nusers in general.\nIn this chapter, we assume that linguistic expressions have meaning representa-\ntions that are made up of the same kind of stuff that is used to represent this kind of\neveryday common-sense knowledge of the world. The process whereby such repre-\nsentations are created and assigned to linguistic inputs is called semantic parsing orsemantic\nparsing\nsemantic analysis , and the entire enterprise of designing meaning representations\nand associated semantic parsers is referred to as computational semantics .computational\nsemantics\n9e;y Having (e)^Haver (e;Speaker )^HadT hing (e;y)^Car(y)\nh / have-01c / cari / i arg0arg1(h / have-01        arg0: (i / i)        arg1: (c / car))Having:      Haver:  Speaker      HadThing:  Car\nFigure F.1 A list of symbols, two directed graphs, and a record structure: a sampler of\nmeaning representations for I have a car.\nConsider Fig. F.1, which shows example meaning representations for the sen-\ntence I have a car using four commonly used meaning representation languages.\nThe top row illustrates a sentence in First-Order Logic , covered in detail in Sec-\ntion F.3; the directed graph and its corresponding textual form is an example of an\nAbstract Meaning Representation (AMR) form (Banarescu et al., 2013), and on\nthe right is a frame-based orslot-\ufb01ller representation, discussed in Section F.5 and\nagain in Chapter 20. 2APPENDIX F \u2022 L OGICAL REPRESENTATIONS OF SENTENCE MEANING\nWhile there are non-trivial differences among these approaches, they all share\nthe notion that a meaning representation consists of structures composed from a\nset of symbols, or representational vocabulary. When appropriately arranged, these\nsymbol structures are taken to correspond to objects, properties of objects, and rela-\ntions among objects in some state of affairs being represented or reasoned about. In\nthis case, all four representations make use of symbols corresponding to the speaker,\na car, and a relation denoting the possession of one by the other.\nImportantly, these representations can be viewed from at least two distinct per-\nspectives in all of these approaches: as representations of the meaning of the par-\nticular linguistic input I have a car , and as representations of the state of affairs in\nsome world. It is this dual perspective that allows these representations to be used\nto link linguistic inputs to the world and to our knowledge of it.\nIn the next sections we give some background: our desiderata for a meaning\nrepresentation language and some guarantees that these representations will actually\ndo what we need them to do\u2014provide a correspondence to the state of affairs being\nrepresented. In Section F.3 we introduce First-Order Logic, historically the primary\ntechnique for investigating natural language semantics, and see in Section F.4 how\nit can be used to capture the semantics of events and states in English.\nF.1 Computational Desiderata for Representations\nLet\u2019s consider why meaning representations are needed and what they should do for\nus. To focus this discussion, let\u2019s consider a system that gives restaurant advice to\ntourists based on a knowledge base.\nVeri\ufb01ability\nConsider the following simple question:\n(F.1) Does Maharani serve vegetarian food?\nTo answer this question, we have to know what it\u2019s asking, and know whether what\nit\u2019s asking is true of Maharini or not. Veri\ufb01ability is a system\u2019s ability to compare veri\ufb01ability\nthe state of affairs described by a representation to the state of affairs in some world\nas modeled in a knowledge base. For example, we\u2019ll need some sort of representa-\ntion like Serves (Maharani ;VegetarianFood ), which a system can match against its\nknowledge base of facts about particular restaurants, and if it \ufb01nds a representation\nmatching this proposition, it can answer yes. Otherwise, it must either say Noif its\nknowledge of local restaurants is complete, or say that it doesn\u2019t know if it knows\nits knowledge is incomplete.\nUnambiguous Representations\nSemantics, like all the other domains we have studied, is subject to ambiguity. Words\nand sentences have different meaning representations in different contexts. Consider\nthe following example:\n(F.2) I wanna eat someplace that\u2019s close to ICSI.\nThis sentence can either mean that the speaker wants to eat atsome nearby location,\nor under a Godzilla-as-speaker interpretation, the speaker may want to devour some\nnearby location. The sentence is ambiguous; a single linguistic expression can have\none of two meanings. But our meaning representations itself cannot be ambiguous. F.1 \u2022 C OMPUTATIONAL DESIDERATA FOR REPRESENTATIONS 3\nThe representation of an input\u2019s meaning should be free from any ambiguity, so that\nthe system can reason over a representation that means either one thing or the other\nin order to decide how to answer.\nA concept closely related to ambiguity is vagueness : in which a meaning repre- vagueness\nsentation leaves some parts of the meaning underspeci\ufb01ed. Vagueness does not give\nrise to multiple representations. Consider the following request:\n(F.3) I want to eat Italian food.\nWhile Italian food may provide enough information to provide recommendations, it\nis nevertheless vague as to what the user really wants to eat. A vague representation\nof the meaning of this phrase may be appropriate for some purposes, while a more\nspeci\ufb01c representation may be needed for other purposes.\nCanonical Form\nThe doctrine of canonical form says that distinct inputs that mean the same thing canonical form\nshould have the same meaning representation. This approach greatly simpli\ufb01es rea-\nsoning, since systems need only deal with a single meaning representation for a\npotentially wide range of expressions.\nConsider the following alternative ways of expressing (F.1):\n(F.4) Does Maharani have vegetarian dishes?\n(F.5) Do they have vegetarian food at Maharani?\n(F.6) Are vegetarian dishes served at Maharani?\n(F.7) Does Maharani serve vegetarian fare?\nDespite the fact these alternatives use different words and syntax, we want them\nto map to a single canonical meaning representations. If they were all different,\nassuming the system\u2019s knowledge base contains only a single representation of this\nfact, most of the representations wouldn\u2019t match. We could, of course, store all\npossible alternative representations of the same fact in the knowledge base, but doing\nso would lead to enormous dif\ufb01culty in keeping the knowledge base consistent.\nCanonical form does complicate the task of semantic parsing. Our system must\nconclude that vegetarian fare ,vegetarian dishes , and vegetarian food refer to the\nsame thing, that having andserving are equivalent here, and that all these parse\nstructures still lead to the same meaning representation. Or consider this pair of\nexamples:\n(F.8) Maharani serves vegetarian dishes.\n(F.9) Vegetarian dishes are served by Maharani.\nDespite the different placement of the arguments to serve , a system must still assign\nMaharani andvegetarian dishes to the same roles in the two examples by draw-\ning on grammatical knowledge, such as the relationship between active and passive\nsentence constructions.\nInference and Variables\nWhat about more complex requests such as:\n(F.10) Can vegetarians eat at Maharani?\nThis request results in the same answer as the others not because they mean the same\nthing, but because there is a common-sense connection between what vegetarians eat\nand what vegetarian restaurants serve. This is a fact about the world. We\u2019ll need to\nconnect the meaning representation of this request with this fact about the world in a 4APPENDIX F \u2022 L OGICAL REPRESENTATIONS OF SENTENCE MEANING\nknowledge base. A system must be able to use inference \u2014to draw valid conclusions inference\nbased on the meaning representation of inputs and its background knowledge. It\nmust be possible for the system to draw conclusions about the truth of propositions\nthat are not explicitly represented in the knowledge base but that are nevertheless\nlogically derivable from the propositions that are present.\nNow consider the following somewhat more complex request:\n(F.11) I\u2019d like to \ufb01nd a restaurant where I can get vegetarian food.\nThis request does not make reference to any particular restaurant; the user wants in-\nformation about an unknown restaurant that serves vegetarian food. Since no restau-\nrants are named, simple matching is not going to work. Answering this request\nrequires the use of variables , using some representation like the following: variables\nServes (x;VegetarianFood ) (F.12)\nMatching succeeds only if the variable xcan be replaced by some object in the\nknowledge base in such a way that the entire proposition will then match. The con-\ncept that is substituted for the variable can then be used to ful\ufb01ll the user\u2019s request.\nIt is critical for any meaning representation language to be able to handle these kinds\nof inde\ufb01nite references.\nExpressiveness\nFinally, a meaning representation scheme must be expressive enough to handle a\nwide range of subject matter, ideally any sensible natural language utterance. Al-\nthough this is probably too much to expect from any single representational system,\nFirst-Order Logic, as described in Section F.3, is expressive enough to handle quite\na lot of what needs to be represented.\nF.2 Model-Theoretic Semantics\nWhat is it about meaning representation languages that allows them to ful\ufb01ll these\ndesiderata, bridging the gap from formal representations to representations that tell\nus something about some state of affairs in the world?\nThe answer is a model . A model is a formal construct that stands for the partic- model\nular state of affairs in the world. Expressions in a meaning representation language\ncan be mapped to elements of the model, like objects, properties of objects, and\nrelations among objects. If the model accurately captures the facts we\u2019re interested\nin, then a consistent mapping between the meaning representation and the model\nprovides the bridge between meaning representation and world. Models provide a\nsurprisingly simple and powerful way to ground the expressions in meaning repre-\nsentation languages.\nFirst, some terminology. The vocabulary of a meaning representation consists of\ntwo parts: the non-logical vocabulary and the logical vocabulary. The non-logical\nvocabulary consists of the open-ended set of names for the objects, properties, andnon-logical\nvocabulary\nrelations that make up the world we\u2019re trying to represent. These appear in various\nschemes as predicates, nodes, labels on links, or labels in slots in frames. The log-\nical vocabulary consists of the closed set of symbols, operators, quanti\ufb01ers, links,logical\nvocabulary\netc., that provide the formal means for composing expressions in a given meaning\nrepresentation language. F.2 \u2022 M ODEL -THEORETIC SEMANTICS 5\nEach element of the non-logical vocabulary must have a denotation in the model, denotation\nmeaning that every element corresponds to a \ufb01xed, well-de\ufb01ned part of the model.\nLet\u2019s start with objects. The domain of a model is the set of objects that are being domain\nrepresented. Each distinct concept, category, or individual denotes a unique element\nin the domain.\nWe represent properties of objects in a model by denoting the domain elements\nthat have the property; that is, properties denote sets. The denotation of the property\nredis the set of things we think are red. Similarly, a relation among object denotes\na set of ordered lists, or tuples, of domain elements that take part in the relation: the\ndenotation of the relation Married is set of pairs of domain objects that are married.\nThis approach to properties and relations is called extensional , because we de\ufb01ne extensional\nconcepts by their extension, their denotations. To summarize:\n\u2022 Objects denote elements of the domain\n\u2022 Properties denote sets of elements of the domain\n\u2022 Relations denote sets of tuples of elements of the domain\nWe now need a mapping that gets us from our meaning representation to the\ncorresponding denotations: a function that maps from the non-logical vocabulary of\nour meaning representation to the proper denotations in the model. We\u2019ll call such\na mapping an interpretation . interpretation\nLet\u2019s return to our restaurant advice application, and let its domain consist of\nsets of restaurants, patrons, facts about the likes and dislikes of the patrons, and\nfacts about the restaurants such as their cuisine, typical cost, and noise level. To\nbegin populating our domain, D, let\u2019s assume that we\u2019re dealing with four patrons\ndesignated by the non-logical symbols Matthew, Franco, Katie , and Caroline . de-\nnoting four unique domain elements. We\u2019ll use the constants a;b;cand, dto stand\nfor these domain elements. We\u2019re deliberately using meaningless, non-mnemonic\nnames for our domain elements to emphasize the fact that whatever it is that we\nknow about these entities has to come from the formal properties of the model and\nnot from the names of the symbols. Continuing, let\u2019s assume that our application\nincludes three restaurants, designated as Frasca ,Med, and Rioin our meaning rep-\nresentation, that denote the domain elements e;f, and g. Finally, let\u2019s assume that\nwe\u2019re dealing with the three cuisines Italian ,Mexican , and Eclectic , denoted by h;i,\nandjin our model.\nProperties like Noisy denote the subset of restaurants from our domain that are\nknown to be noisy. Two-place relational notions, such as which restaurants individ-\nual patrons Like, denote ordered pairs, or tuples, of the objects from the domain.\nAnd, since we decided to represent cuisines as objects in our model, we can cap-\nture which restaurants Serve which cuisines as a set of tuples. One possible state of\naffairs using this scheme is given in Fig. F.2.\nGiven this simple scheme, we can ground our meaning representations by con-\nsulting the appropriate denotations in the corresponding model. For example, we can\nevaluate a representation claiming that Matthew likes the Rio , or that The Med serves\nItalian by mapping the objects in the meaning representations to their corresponding\ndomain elements and mapping any links, predicates, or slots in the meaning repre-\nsentation to the appropriate relations in the model. More concretely, we can verify\na representation asserting that Matthew likes Frasca by \ufb01rst using our interpretation\nfunction to map the symbol Matthew to its denotation a,Frasca toe, and the Likes\nrelation to the appropriate set of tuples. We then check that set of tuples for the\npresence of the tuple ha;ei. If, as it is in this case, the tuple is present in the model,\nthen we can conclude that Matthew likes Frasca is true; if it isn\u2019t then we can\u2019t. 6APPENDIX F \u2022 L OGICAL REPRESENTATIONS OF SENTENCE MEANING\nDomain D=fa;b;c;d;e;f;g;h;i;jg\nMatthew, Franco, Katie and Caroline a;b;c;d\nFrasca, Med, Rio e;f;g\nItalian, Mexican, Eclectic h;i;j\nProperties\nNoisy Noisy =fe;f;gg\nFrasca, Med, and Rio are noisy\nRelations\nLikes Likes =fha;fi;hc;fi;hc;gi;hb;ei;hd;fi;hd;gig\nMatthew likes the Med\nKatie likes the Med and Rio\nFranco likes Frasca\nCaroline likes the Med and Rio\nServes Serves =fhf;ji;hg;ii;he;hig\nMed serves eclectic\nRio serves Mexican\nFrasca serves Italian\nFigure F.2 A model of the restaurant world.\nThis is all pretty straightforward\u2014we\u2019re using sets and operations on sets to\nground the expressions in our meaning representations. Of course, the more inter-\nesting part comes when we consider more complex examples such as the following:\n(F.13) Katie likes the Rio and Matthew likes the Med.\n(F.14) Katie and Caroline like the same restaurants.\n(F.15) Franco likes noisy, expensive restaurants.\n(F.16) Not everybody likes Frasca.\nOur simple scheme for grounding the meaning of representations is not adequate\nfor examples such as these. Plausible meaning representations for these examples\nwill not map directly to individual entities, properties, or relations. Instead, they\ninvolve complications such as conjunctions, equality, quanti\ufb01ed variables, and nega-\ntions. To assess whether these statements are consistent with our model, we\u2019ll have\nto tear them apart, assess the parts, and then determine the meaning of the whole\nfrom the meaning of the parts.\nConsider the \ufb01rst example above. A meaning representation for this example\nwill include two distinct propositions expressing the individual patron\u2019s preferences,\nconjoined with some kind of implicit or explicit conjunction operator. Our model\ndoesn\u2019t have a relation that encodes pairwise preferences for all of the patrons and\nrestaurants in our model, nor does it need to. We know from our model that Matthew\nlikes the Med and separately that Katie likes the Rio (that is, the tuples ha;fiand\nhc;giare members of the set denoted by the Likes relation). All we really need to\nknow is how to deal with the semantics of the conjunction operator. If we assume\nthe simplest possible semantics for the English word and, the whole statement is\ntrue if it is the case that each of the components is true in our model. In this case,\nboth components are true since the appropriate tuples are present and therefore the\nsentence as a whole is true.\nWhat we\u2019ve done with this example is provide a truth-conditional semanticstruth-\nconditional\nsemanticsfor the assumed conjunction operator in some meaning representation. That is,\nwe\u2019ve provided a method for determining the truth of a complex expression from F.3 \u2022 F IRST-ORDER LOGIC 7\nFormula!AtomicFormula\njFormula Connective Formula\njQuanti\ufb01er Variable ; : : :Formula\nj : Formula\nj(Formula )\nAtomicFormula!Predicate (Term ; : : :)\nTerm!Function (Term ; : : :)\njConstant\njVariable\nConnective! ^j_j =)\nQuanti\ufb01er! 8j9\nConstant!AjVegetarianFoodjMaharani\u0001\u0001\u0001\nVariable!xjyj\u0001\u0001\u0001\nPredicate!ServesjNearj\u0001\u0001\u0001\nFunction!LocationOfjCuisineOfj\u0001\u0001\u0001\nFigure F.3 A context-free grammar speci\ufb01cation of the syntax of First-Order Logic repre-\nsentations. Adapted from Russell and Norvig 2002.\nthe meanings of the parts (by consulting a model) and the meaning of an operator by\nconsulting a truth table. Meaning representation languages are truth-conditional to\nthe extent that they give a formal speci\ufb01cation as to how we can determine the mean-\ning of complex sentences from the meaning of their parts. In particular, we need to\nknow the semantics of the entire logical vocabulary of the meaning representation\nscheme being used.\nNote that although the details of how this happens depend on details of the par-\nticular meaning representation being used, it should be clear that assessing the truth\nconditions of examples like these involves nothing beyond the simple set operations\nwe\u2019ve been discussing. We return to these issues in the next section in the context of\nthe semantics of First-Order Logic.\nF.3 First-Order Logic\nFirst-Order Logic ( FOL) is a \ufb02exible, well-understood, and computationally tractable\nmeaning representation language that satis\ufb01es many of the desiderata given in Sec-\ntion F.1. It provides a sound computational basis for the veri\ufb01ability, inference, and\nexpressiveness requirements, as well as a sound model-theoretic semantics.\nAn additional attractive feature of FOLis that it makes few speci\ufb01c commitments\nas to how things ought to be represented, and those it does are shared by many of\nthe schemes mentioned earlier: the represented world consists of objects, properties\nof objects, and relations among objects.\nThe remainder of this section introduces the basic syntax and semantics of FOL\nand then describes the application of FOL to the representation of events.\nF.3.1 Basic Elements of First-Order Logic\nLet\u2019s explore FOL by \ufb01rst examining its various atomic elements and then showing\nhow they can be composed to create larger meaning representations. Figure F.3,\nwhich provides a complete context-free grammar for the particular syntax of FOL\nthat we will use, is our roadmap for this section. 8APPENDIX F \u2022 L OGICAL REPRESENTATIONS OF SENTENCE MEANING\nLet\u2019s begin by examining the notion of a term , the FOL device for representing term\nobjects. As can be seen from Fig. F.3, FOL provides three ways to represent these\nbasic building blocks: constants, functions, and variables. Each of these devices can\nbe thought of as designating an object in the world under consideration.\nConstants inFOL refer to speci\ufb01c objects in the world being described. Such constant\nconstants are conventionally depicted as either single capitalized letters such as A\nandBor single capitalized words that are often reminiscent of proper nouns such as\nMaharani andHarry . Like programming language constants, FOL constants refer\nto exactly one object. Objects can, however, have multiple constants that refer to\nthem.\nFunctions inFOL correspond to concepts that are often expressed in English as function\ngenitives such as Frasca\u2019s location . A FOL translation of such an expression might\nlook like the following.\nLocationOf (Frasca ) (F.17)\nFOL functions are syntactically the same as single argument predicates. It is im-\nportant to remember, however, that while they have the appearance of predicates,\nthey are in fact terms in that they refer to unique objects. Functions provide a con-\nvenient way to refer to speci\ufb01c objects without having to associate a named constant\nwith them. This is particularly convenient in cases in which many named objects,\nlike restaurants, have a unique concept such as a location associated with them.\nVariables are our \ufb01nal FOL mechanism for referring to objects. Variables, de- variable\npicted as single lower-case letters, let us make assertions and draw inferences about\nobjects without having to make reference to any particular named object. This ability\nto make statements about anonymous objects comes in two \ufb02avors: making state-\nments about a particular unknown object and making statements about all the objects\nin some arbitrary world of objects. We return to the topic of variables after we have\npresented quanti\ufb01ers, the elements of FOL that make variables useful.\nNow that we have the means to refer to objects, we can move on to the FOL\nmechanisms that are used to state relations that hold among objects. Predicates are\nsymbols that refer to, or name, the relations that hold among some \ufb01xed number\nof objects in a given domain. Returning to the example introduced informally in\nSection F.1, a reasonable FOL representation for Maharani serves vegetarian food\nmight look like the following formula:\nServes (Maharani ;VegetarianFood ) (F.18)\nThis FOL sentence asserts that Serves , a two-place predicate, holds between the\nobjects denoted by the constants Maharani andVegetarianFood .\nA somewhat different use of predicates is illustrated by the following fairly typ-\nical representation for a sentence like Maharani is a restaurant :\nRestaurant (Maharani ) (F.19)\nThis is an example of a one-place predicate that is used, not to relate multiple objects,\nbut rather to assert a property of a single object. In this case, it encodes the category\nmembership of Maharani .\nWith the ability to refer to objects, to assert facts about objects, and to relate\nobjects to one another, we can create rudimentary composite representations. These\nrepresentations correspond to the atomic formula level in Fig. F.3. This ability to\ncompose complex representations is, however, not limited to the use of single pred-\nicates. Larger composite representations can also be put together through the use of F.3 \u2022 F IRST-ORDER LOGIC 9\nlogical connectives . As can be seen from Fig. F.3, logical connectives let us createlogical\nconnectives\nlarger representations by conjoining logical formulas using one of three operators.\nConsider, for example, the following BERP sentence and one possible representation\nfor it:\n(F.20) I only have \ufb01ve dollars and I don\u2019t have a lot of time.\nHave (Speaker ;FiveDollars )^:Have (Speaker ;LotOfTime ) (F.21)\nThe semantic representation for this example is built up in a straightforward way\nfrom the semantics of the individual clauses through the use of the ^and:operators.\nNote that the recursive nature of the grammar in Fig. F.3 allows an in\ufb01nite number\nof logical formulas to be created through the use of these connectives. Thus, as with\nsyntax, we can use a \ufb01nite device to create an in\ufb01nite number of representations.\nF.3.2 Variables and Quanti\ufb01ers\nWe now have all the machinery necessary to return to our earlier discussion of vari-\nables. As noted above, variables are used in two ways in FOL: to refer to particular\nanonymous objects and to refer generically to all objects in a collection. These two\nuses are made possible through the use of operators known as quanti\ufb01ers . The two quanti\ufb01ers\noperators that are basic to FOL are the existential quanti\ufb01er, which is denoted 9and\nis pronounced as \u201cthere exists\u201d, and the universal quanti\ufb01er, which is denoted 8and\nis pronounced as \u201cfor all\u201d.\nThe need for an existentially quanti\ufb01ed variable is often signaled by the presence\nof an inde\ufb01nite noun phrase in English. Consider the following example:\n(F.22) a restaurant that serves Mexican food near ICSI.\nHere, reference is being made to an anonymous object of a speci\ufb01ed category with\nparticular properties. The following would be a reasonable representation of the\nmeaning of such a phrase:\n9xRestaurant (x)^Serves (x;MexicanFood ) (F.23)\n^Near (LocationOf (x);LocationOf (ICSI ))\nThe existential quanti\ufb01er at the head of this sentence instructs us on how to\ninterpret the variable xin the context of this sentence. Informally, it says that for\nthis sentence to be true there must be at least one object such that if we were to\nsubstitute it for the variable x, the resulting sentence would be true. For example,\nifAyCaramba is a Mexican restaurant near ICSI, then substituting AyCaramba forx\nresults in the following logical formula:\nRestaurant (AyCaramba )^Serves (AyCaramba ;MexicanFood ) (F.24)\n^Near ((LocationOf (AyCaramba );LocationOf (ICSI ))\nBased on the semantics of the ^operator, this sentence will be true if all of its\nthree component atomic formulas are true. These in turn will be true if they are\neither present in the system\u2019s knowledge base or can be inferred from other facts in\nthe knowledge base.\nThe use of the universal quanti\ufb01er also has an interpretation based on substi-\ntution of known objects for variables. The substitution semantics for the universal\nquanti\ufb01er takes the expression for all quite literally; the8operator states that for the\nlogical formula in question to be true, the substitution of anyobject in the knowledge 10 APPENDIX F \u2022 L OGICAL REPRESENTATIONS OF SENTENCE MEANING\nbase for the universally quanti\ufb01ed variable should result in a true formula. This is in\nmarked contrast to the 9operator, which only insists on a single valid substitution\nfor the sentence to be true.\nConsider the following example:\n(F.25) All vegetarian restaurants serve vegetarian food.\nA reasonable representation for this sentence would be something like the following:\n8xVegetarianRestaurant (x) =)Serves (x;VegetarianFood ) (F.26)\nFor this sentence to be true, every substitution of a known object for xmust result in a\nsentence that is true. We can divide the set of all possible substitutions into the set of\nobjects consisting of vegetarian restaurants and the set consisting of everything else.\nLet us \ufb01rst consider the case in which the substituted object actually is a vegetarian\nrestaurant; one such substitution would result in the following sentence:\nVegetarianRestaurant (Maharani ) =)Serves (Maharani ;VegetarianFood )\n(F.27)\nIf we assume that we know that the consequent clause\nServes (Maharani ;VegetarianFood ) (F.28)\nis true, then this sentence as a whole must be true. Both the antecedent and the\nconsequent have the value True and, therefore, according to the \ufb01rst two rows of\nFig. F.4 on page 12 the sentence itself can have the value True . This result will be\nthe same for all possible substitutions of Terms representing vegetarian restaurants\nforx.\nRemember, however, that for this sentence to be true, it must be true for all\npossible substitutions. What happens when we consider a substitution from the set\nof objects that are not vegetarian restaurants? Consider the substitution of a non-\nvegetarian restaurant such as AyCaramba for the variable x:\nVegetarianRestaurant (AyCaramba ) =)Serves (AyCaramba ;VegetarianFood )\nSince the antecedent of the implication is False , we can determine from Fig. F.4\nthat the sentence is always True , again satisfying the 8constraint.\nNote that it may still be the case that AyCaramba serves vegetarian food with-\nout actually being a vegetarian restaurant. Note also that, despite our choice of\nexamples, there are no implied categorical restrictions on the objects that can be\nsubstituted for xby this kind of reasoning. In other words, there is no restriction of\nxto restaurants or concepts related to them. Consider the following substitution:\nVegetarianRestaurant (Carburetor ) =)Serves (Carburetor ;VegetarianFood )\nHere the antecedent is still false so the rule remains true under this kind of irrelevant\nsubstitution.\nTo review, variables in logical formulas must be either existentially ( 9) or uni-\nversally (8) quanti\ufb01ed. To satisfy an existentially quanti\ufb01ed variable, at least one\nsubstitution must result in a true sentence. To satisfy a universally quanti\ufb01ed vari-\nable, all substitutions must result in true sentences. F.3 \u2022 F IRST-ORDER LOGIC 11\nF.3.3 Lambda Notation\nThe \ufb01nal element we need to complete our discussion of FOL is called the lambda\nnotation (Church, 1940). This notation provides a way to abstract from fully speci-lambda\nnotation\n\ufb01ed FOLformulas in a way that will be particularly useful for semantic analysis. The\nlambda notation extends the syntax of FOL to include expressions of the following\nform:\nlx:P(x) (F.29)\nSuch expressions consist of the Greek symbol l, followed by one or more variables,\nfollowed by a FOL formula that makes use of those variables.\nThe usefulness of these l-expressions is based on the ability to apply them to\nlogical terms to yield new FOLexpressions where the formal parameter variables are\nbound to the speci\ufb01ed terms. This process is known as l-reduction , and consists l-reduction\nof a simple textual replacement of the lvariables and the removal of the l. The\nfollowing expressions illustrate the application of a l-expression to the constant A,\nfollowed by the result of performing a l-reduction on this expression:\nlx:P(x)(A) (F.30)\nP(A)\nAn important and useful variation of this technique is the use of one l-expression\nas the body of another as in the following expression:\nlx:ly:Near (x;y) (F.31)\nThis fairly abstract expression can be glossed as the state of something being near\nsomething else. The following expressions illustrate a single l-application and sub-\nsequent reduction with this kind of embedded l-expression:\nlx:ly:Near (x;y)(Bacaro ) (F.32)\nly:Near (Bacaro ;y)\nThe important point here is that the resulting expression is still a l-expression; the\n\ufb01rst reduction bound the variable xand removed the outer l, thus revealing the\ninner expression. As might be expected, this resulting l-expression can, in turn,\nbe applied to another term to arrive at a fully speci\ufb01ed logical formula, as in the\nfollowing:\nly:Near (Bacaro ;y)(Centro ) (F.33)\nNear (Bacaro ;Centro )\nThis general technique, called currying1(Sch \u00a8on\ufb01nkel, 1924) is a way of converting currying\na predicate with multiple arguments into a sequence of single-argument predicates.\nThel-notation also provides a way to incrementally gather arguments to a pred-\nicate when they do not all appear together as daughters of the predicate in a parse\ntree.\nF.3.4 The Semantics of First-Order Logic\nThe various objects, properties, and relations represented in a FOL knowledge base\nacquire their meanings by virtue of their correspondence to objects, properties, and\n1Currying is the standard term, although Heim and Kratzer (1998) present an interesting argument for\nthe term Sch\u00a8on\ufb01nkelization over currying, since Curry later built on Sch \u00a8on\ufb01nkel\u2019s work. 12 APPENDIX F \u2022 L OGICAL REPRESENTATIONS OF SENTENCE MEANING\nrelations out in the external world being modeled. We can accomplish this by em-\nploying the model-theoretic approach introduced in Section F.2. Recall that this\napproach employs simple set-theoretic notions to provide a truth-conditional map-\nping from the expressions in a meaning representation to the state of affairs being\nmodeled. We can apply this approach to FOL by going through all the elements in\nFig. F.3 on page 7 and specifying how each should be accounted for.\nWe can start by asserting that the objects in our world, FOL terms, denote ele-\nments in a domain, and asserting that atomic formulas are captured either as sets of\ndomain elements for properties, or as sets of tuples of elements for relations. As an\nexample, consider the following:\n(F.34) Centro is near Bacaro.\nCapturing the meaning of this example in FOL involves identifying the Terms\nandPredicates that correspond to the various grammatical elements in the sentence\nand creating logical formulas that capture the relations implied by the words and\nsyntax of the sentence. For this example, such an effort might yield something like\nthe following:\nNear (Centro ;Bacaro ) (F.35)\nThe meaning of this logical formula is based on whether the domain elements de-\nnoted by the terms Centro andBacaro are contained among the tuples denoted by\nthe relation denoted by the predicate Near in the current model.\nThe interpretation of formulas involving logical connectives is based on the\nmeanings of the components in the formulas combined with the meanings of the\nconnectives they contain. Figure F.4 gives interpretations for each of the logical\noperators shown in Fig. F.3.\nP Q :P P^Q P_Q P =)Q\nFalse False True False False True\nFalse True True False True True\nTrue False False False True False\nTrue True False True True True\nFigure F.4 Truth table giving the semantics of the various logical connectives.\nThe semantics of the ^(and) and:(not) operators are fairly straightforward,\nand are correlated with at least some of the senses of the corresponding English\nterms. However, it is worth pointing out that the _(or) operator is not disjunctive\nin the same way that the corresponding English word is, and that the =)(im-\nplies) operator is only loosely based on any common-sense notions of implication\nor causation.\nThe \ufb01nal bit we need to address involves variables and quanti\ufb01ers. Recall that\nthere are no variables in our set-based models, only elements of the domain and\nrelations that hold among them. We can provide a model-based account for formulas\nwith variables by employing the notion of a substitution introduced earlier on page\n9. Formulas involving 9are true if a substitution of terms for variables results in\na formula that is true in the model. Formulas involving 8must be true under all\npossible substitutions.\nF.3.5 Inference\nA meaning representation language must support inference to add valid new propo-\nsitions to a knowledge base or to determine the truth of propositions not explicitly F.3 \u2022 F IRST-ORDER LOGIC 13\ncontained within a knowledge base (Section F.1). This section brie\ufb02y discusses\nmodus ponens , the most widely implemented inference method provided by FOL.\nModus ponens is a form of inference that corresponds to what is informally Modus ponens\nknown as if-then reasoning. We can abstractly de\ufb01ne modus ponens as follows,\nwhere aandbshould be taken as FOL formulas:\na\na=)b\nb(F.36)\nA schema like this indicates that the formula below the line can be inferred from the\nformulas above the line by some form of inference. Modus ponens states that if the\nleft-hand side of an implication rule is true, then the right-hand side of the rule can\nbe inferred. In the following discussions, we will refer to the left-hand side of an\nimplication as the antecedent and the right-hand side as the consequent .\nFor a typical use of modus ponens, consider the following example, which uses\na rule from the last section:\nVegetarianRestaurant (Leaf )\n8xVegetarianRestaurant (x) =)Serves (x;VegetarianFood )\nServes (Leaf ;VegetarianFood )(F.37)\nHere, the formula VegetarianRestaurant (Leaf )matches the antecedent of the rule,\nthus allowing us to use modus ponens to conclude Serves (Leaf ;VegetarianFood ).\nModus ponens can be put to practical use in one of two ways: forward chaining\nand backward chaining. In forward chaining systems, modus ponens is used inforward\nchaining\nprecisely the manner just described. As individual facts are added to the knowledge\nbase, modus ponens is used to \ufb01re all applicable implication rules. In this kind of\narrangement, as soon as a new fact is added to the knowledge base, all applicable\nimplication rules are found and applied, each resulting in the addition of new facts to\nthe knowledge base. These new propositions in turn can be used to \ufb01re implication\nrules applicable to them. The process continues until no further facts can be deduced.\nThe forward chaining approach has the advantage that facts will be present in\nthe knowledge base when needed, because, in a sense all inference is performed in\nadvance. This can substantially reduce the time needed to answer subsequent queries\nsince they should all amount to simple lookups. The disadvantage of this approach\nis that facts that will never be needed may be inferred and stored.\nInbackward chaining , modus ponens is run in reverse to prove speci\ufb01c propo-backward\nchaining\nsitions called queries. The \ufb01rst step is to see if the query formula is true by determin-\ning if it is present in the knowledge base. If it is not, then the next step is to search\nfor applicable implication rules present in the knowledge base. An applicable rule\nis one whereby the consequent of the rule matches the query formula. If there are\nany such rules, then the query can be proved if the antecedent of any one them can\nbe shown to be true. This can be performed recursively by backward chaining on\nthe antecedent as a new query. The Prolog programming language is a backward\nchaining system that implements this strategy.\nTo see how this works, let\u2019s assume that we have been asked to verify the truth of\nthe proposition Serves (Leaf ;VegetarianFood ), assuming the facts given above the\nline in (F.37). Since this proposition is not present in the knowledge base, a search\nfor an applicable rule is initiated resulting in the rule given above. After substituting\nthe constant Leaf for the variable x, our next task is to prove the antecedent of the\nrule, VegetarianRestaurant (Leaf ), which, of course, is one of the facts we are given. 14 APPENDIX F \u2022 L OGICAL REPRESENTATIONS OF SENTENCE MEANING\nNote that it is critical to distinguish between reasoning by backward chaining\nfrom queries to known facts and reasoning backwards from known consequents to\nunknown antecedents. To be speci\ufb01c, by reasoning backwards we mean that if the\nconsequent of a rule is known to be true, we assume that the antecedent will be as\nwell. For example, let\u2019s assume that we know that Serves (Leaf ;VegetarianFood )is\ntrue. Since this fact matches the consequent of our rule, we might reason backwards\nto the conclusion that VegetarianRestaurant (Leaf ).\nWhile backward chaining is a sound method of reasoning, reasoning backwards\nis an invalid, though frequently useful, form of plausible reasoning . Plausible rea-\nsoning from consequents to antecedents is known as abduction , and as we show in abduction\nChapter 24, is often useful in accounting for many of the inferences people make\nwhile analyzing extended discourses.\nWhile forward and backward reasoning are sound, neither is complete . This complete\nmeans that there are valid inferences that cannot be found by systems using these\nmethods alone. Fortunately, there is an alternative inference technique called reso-\nlution that is sound and complete. Unfortunately, inference systems based on res- resolution\nolution are far more computationally expensive than forward or backward chaining\nsystems. In practice, therefore, most systems use some form of chaining and place\na burden on knowledge base developers to encode the knowledge in a fashion that\npermits the necessary inferences to be drawn.\nF.4 Event and State Representations\nMuch of the semantics that we wish to capture consists of representations of states\nand events. States are conditions, or properties, that remain unchanged over an\nextended period of time, and events denote changes in some state of affairs. The\nrepresentation of both states and events may involve a host of participants, props,\ntimes and locations.\nThe representations for events and states that we have used thus far have con-\nsisted of single predicates with as many arguments as are needed to incorporate all\nthe roles associated with a given example. For example, the representation for Leaf\nserves vegetarian fare consists of a single predicate with arguments for the entity\ndoing the serving and the thing served.\nServes (Leaf ;VegetarianFare ) (F.38)\nThis approach assumes that the predicate used to represent an event verb has the\nsame number of arguments as are present in the verb\u2019s syntactic subcategorization\nframe. Unfortunately, this is clearly not always the case. Consider the following\nexamples of the verb eat:\n(F.39) I ate.\n(F.40) I ate a turkey sandwich.\n(F.41) I ate a turkey sandwich at my desk.\n(F.42) I ate at my desk.\n(F.43) I ate lunch.\n(F.44) I ate a turkey sandwich for lunch.\n(F.45) I ate a turkey sandwich for lunch at my desk. F.5 \u2022 D ESCRIPTION LOGICS 15\nClearly, choosing the correct number of arguments for the predicate represent-\ning the meaning of eatis a tricky problem. These examples introduce \ufb01ve distinct\narguments, or roles, in an array of different syntactic forms, locations, and combina-\ntions. Unfortunately, predicates in FOL have \ufb01xed arity \u2013 they take a \ufb01xed number arity\nof arguments.\nTo address this problem, we introduce the notion of an event variable to allow event variable\nus to make assertions about particular events. To do this, we can refactor our event\npredicates to have an existentially quanti\ufb01ed variable as their \ufb01rst, and only , argu-\nment. Using this event variable, we can introduce additional predicates to represent\nthe other information we have about the event. These predicates take an event vari-\nable as their \ufb01rst argument and related FOL terms as their second argument. The\nfollowing formula illustrates this scheme with the meaning representation of F.40\nfrom our earlier discussion.\n9e Eating (e)^Eater (e;Speaker )^Eaten (e;TurkeySandwich )\nHere, the quanti\ufb01ed variable estands for the eating event and is used to bind the\nevent predicate with the core information provided via the named roles Eater and\nEaten . To handle the more complex examples, we simply add additional relations\nto capture the provided information, as in the following for F.45.\n9e Eating (e)^Eater (e;Speaker )^Eaten (e;TurkeySandwich ) (F.46)\n^Meal (e;Lunch )^Location (e;Desk )\nEvent representations of this sort are referred to as neo-Davidsonian event rep-neo-\nDavidsonian\nresentations (Davidson 1967, Parsons 1990) after the philosopher Donald Davidson\nwho introduced the notion of an event variable (Davidson, 1967). To summarize, in\nthe neo-Davidsonian approach to event representations:\n\u2022 Events are captured with predicates that take a single event variable as an\nargument.\n\u2022 There is no need to specify a \ufb01xed number of arguments for a given FOL\npredicate; rather, as many roles and \ufb01llers can be glued on as are provided in\nthe input.\n\u2022 No more roles are postulated than are mentioned in the input.\n\u2022 The logical connections among closely related inputs that share the same pred-\nicate are satis\ufb01ed without the need for additional inference.\nThis approach still leaves us with the problem of determining the set of predi-\ncates needed to represent roles associated with speci\ufb01c events like Eater andEaten ,\nas well as more general concepts like Location andTime . We\u2019ll return to this prob-\nlem in more detail in Chapter 20 and Chapter 21.\nF.5 Description Logics\nAs noted at the beginning of this chapter, a fair number of representational schemes\nhave been invented to capture the meaning of linguistic utterances. It is now widely\naccepted that meanings represented in these various approaches can, in principle, be\ntranslated into equivalent statements in FOL with relative ease. The dif\ufb01culty is that\nin many of these approaches the semantics of a statement are de\ufb01ned procedurally.\nThat is, the meaning arises from whatever the system that interprets it does with it. 16 APPENDIX F \u2022 L OGICAL REPRESENTATIONS OF SENTENCE MEANING\nDescription logics are an effort to better specify the semantics of these earlier\nstructured network representations and to provide a conceptual framework that is\nespecially well suited to certain kinds of domain modeling. Formally, the term De-\nscription Logics refers to a family of logical approaches that correspond to varying\nsubsets of FOL. The restrictions placed on the expressiveness of Description Logics\nserve to guarantee the tractability of various critical kinds of inference. Our focus\nhere, however, will be on the modeling aspects of DLs rather than on computational\ncomplexity issues.\nWhen using Description Logics to model an application domain, the emphasis\nis on the representation of knowledge about categories, individuals that belong to\nthose categories, and the relationships that can hold among these individuals. The\nset of categories, or concepts, that make up a particular application domain is called\nitsterminology . The portion of a knowledge base that contains the terminology is terminology\ntraditionally called the TBox ; this is in contrast to the ABox that contains facts about TBox\nABox individuals. The terminology is typically arranged into a hierarchical organization\ncalled an ontology that captures the subset/superset relations among the categories. ontology\nReturning to our earlier culinary domain, we represented domain concepts us-\ning unary predicates such as Restaurant (x); the DL equivalent omits the variable,\nso the restaurant category is simply written as Restaurant .2To capture the fact\nthat a particular domain element, such as Frasca , is a restaurant, we assert Restau-\nrant(Frasca) in much the same way we would in FOL. The semantics of these\ncategories are speci\ufb01ed in precisely the same way that was introduced earlier in\nSection F.2: a category like Restaurant simply denotes the set of domain elements\nthat are restaurants.\nOnce we\u2019ve speci\ufb01ed the categories of interest in a particular domain, the next\nstep is to arrange them into a hierarchical structure. There are two ways to cap-\nture the hierarchical relationships present in a terminology: we can directly assert\nrelations between categories that are related hierarchically, or we can provide com-\nplete de\ufb01nitions for our concepts and then rely on inference to provide hierarchical\nrelationships. The choice between these methods hinges on the use to which the re-\nsulting categories will be put and the feasibility of formulating precise de\ufb01nitions for\nmany naturally occurring categories. We\u2019ll discuss the \ufb01rst option here and return to\nthe notion of de\ufb01nitions later in this section.\nTo directly specify a hierarchical structure, we can assert subsumption relations subsumption\nbetween the appropriate concepts in a terminology. The subsumption relation is\nconventionally written as CvDand is read as Cis subsumed by D; that is, all\nmembers of the category Care also members of the category D. Not surprisingly, the\nformal semantics of this relation are provided by a simple set relation; any domain\nelement that is in the set denoted by Cis also in the set denoted by D.\nAdding the following statements to the TBox asserts that all restaurants are com-\nmercial establishments and, moreover, that there are various subtypes of restaurants.\nRestaurantvCommercialEstablishment (F.47)\nItalianRestaurant vRestaurant (F.48)\nChineseRestaurant vRestaurant (F.49)\nMexicanRestaurant vRestaurant (F.50)\nOntologies such as this are conventionally illustrated with diagrams such as the one\n2DL statements are conventionally typeset with a sans serif font. We\u2019ll follow that convention here,\nreverting to our standard mathematical notation when giving FOL equivalents of DL statements. F.5 \u2022 D ESCRIPTION LOGICS 17\nshown in Fig. F.5, where subsumption relations are denoted by links between the\nnodes representing the categories.\nRestaurantChineseRestaurant MexicanRestaurantItalianRestaurantCommercialEstablishment\nFigure F.5 A graphical network representation of a set of subsumption relations in the\nrestaurant domain.\nNote, that it was precisely the vague nature of semantic network diagrams like\nthis that motivated the development of Description Logics. For example, from this\ndiagram we can\u2019t tell whether the given set of categories is exhaustive or disjoint.\nThat is, we can\u2019t tell if these are all the kinds of restaurants that we\u2019ll be dealing with\nin our domain or whether there might be others. We also can\u2019t tell if an individual\nrestaurant must fall into only oneof these categories, or if it is possible, for example,\nfor a restaurant to be both Italian and Chinese. The DL statements given above are\nmore transparent in their meaning; they simply assert a set of subsumption relations\nbetween categories and make no claims about coverage or mutual exclusion.\nIf an application requires coverage and disjointness information, then such in-\nformation must be made explicitly. The simplest ways to capture this kind of in-\nformation is through the use of negation and disjunction operators. For example,\nthe following assertion would tell us that Chinese restaurants can\u2019t also be Italian\nrestaurants.\nChineseRestaurant vnotItalianRestaurant (F.51)\nSpecifying that a set of subconcepts covers a category can be achieved with disjunc-\ntion, as in the following:\nRestaurantv (F.52)\n(orItalianRestaurant ChineseRestaurant MexicanRestaurant )\nHaving a hierarchy such as the one given in Fig. F.5 tells us next to nothing\nabout the concepts in it. We certainly don\u2019t know anything about what makes a\nrestaurant a restaurant, much less Italian, Chinese, or expensive. What is needed are\nadditional assertions about what it means to be a member of any of these categories.\nIn Description Logics such statements come in the form of relations between the\nconcepts being described and other concepts in the domain. In keeping with its\norigins in structured network representations, relations in Description Logics are\ntypically binary and are often referred to as roles, or role-relations.\nTo see how such relations work, let\u2019s consider some of the facts about restaurants\ndiscussed earlier in the chapter. We\u2019ll use the hasCuisine relation to capture infor-\nmation as to what kinds of food restaurants serve and the hasPriceRange relation\nto capture how pricey particular restaurants tend to be. We can use these relations\nto say something more concrete about our various classes of restaurants. Let\u2019s start 18 APPENDIX F \u2022 L OGICAL REPRESENTATIONS OF SENTENCE MEANING\nwith our ItalianRestaurant concept. As a \ufb01rst approximation, we might say some-\nthing uncontroversial like Italian restaurants serve Italian cuisine. To capture these\nnotions, let\u2019s \ufb01rst add some new concepts to our terminology to represent various\nkinds of cuisine.\nMexicanCuisinevCuisine\nItalianCuisinevCuisine\nChineseCuisinevCuisine\nVegetarianCuisine vCuisineExpensiveRestaurant vRestaurant\nModerateRestaurant vRestaurant\nCheapRestaurant vRestaurant\nNext, let\u2019s revise our earlier version of ItalianRestaurant to capture cuisine infor-\nmation.\nItalianRestaurant vRestaurantu9hasCuisine :ItalianCuisine (F.53)\nThe correct way to read this expression is that individuals in the category Italian-\nRestaurant are subsumed both by the category Restaurant and by an unnamed\nclass de\ufb01ned by the existential clause\u2014the set of entities that serve Italian cuisine.\nAn equivalent statement in FOL would be\n8xItalianRestaurant (x)!Restaurant (x) (F.54)\n^(9yServes (x;y)^ItalianCuisine (y))\nThis FOL translation should make it clear what the DL assertions given above do\nand do not entail. In particular, they don\u2019t say that domain entities classi\ufb01ed as Ital-\nian restaurants can\u2019t engage in other relations like being expensive or even serving\nChinese cuisine. And critically, they don\u2019t say much about domain entities that we\nknow do serve Italian cuisine. In fact, inspection of the FOL translation makes it\nclear that we cannot infer that any new entities belong to this category based on their\ncharacteristics. The best we can do is infer new facts about restaurants that we\u2019re\nexplicitly told are members of this category.\nOf course, inferring the category membership of individuals given certain char-\nacteristics is a common and critical reasoning task that we need to support. This\nbrings us back to the alternative approach to creating hierarchical structures in a\nterminology: actually providing a de\ufb01nition of the categories we\u2019re creating in the\nform of necessary and suf\ufb01cient conditions for category membership. In this case,\nwe might explicitly provide a de\ufb01nition for ItalianRestaurant as being those restau-\nrants that serve Italian cuisine, and ModerateRestaurant as being those whose\nprice range is moderate.\nItalianRestaurant \u0011Restaurantu9hasCuisine :ItalianCuisine (F.55)\nModerateRestaurant \u0011RestaurantuhasPriceRange :ModeratePrices (F.56)\nWhile our earlier statements provided necessary conditions for membership in these\ncategories, these statements provide both necessary and suf\ufb01cient conditions.\nFinally, let\u2019s now consider the super\ufb01cially similar case of vegetarian restaurants.\nClearly, vegetarian restaurants are those that serve vegetarian cuisine. But they don\u2019t\nmerely serve vegetarian fare, that\u2019s all they serve. We can accommodate this kind of\nconstraint by adding an additional restriction in the form of a universal quanti\ufb01er to F.5 \u2022 D ESCRIPTION LOGICS 19\nour earlier description of VegetarianRestaurants , as follows:\nVegetarianRestaurant \u0011Restaurant (F.57)\nu9hasCuisine :VegetarianCuisine\nu8hasCuisine :VegetarianCuisine\nInference\nParalleling the focus of Description Logics on categories, relations, and individuals\nis a processing focus on a restricted subset of logical inference. Rather than employ-\ning the full range of reasoning permitted by FOL, DL reasoning systems emphasize\nthe closely coupled problems of subsumption and instance checking.\nSubsumption , as a form of inference, is the task of determining, based on the subsumption\nfacts asserted in a terminology, whether a superset/subset relationship exists between\ntwo concepts. Correspondingly, instance checking asks if an individual can be ainstance\nchecking\nmember of a particular category given the facts we know about both the individual\nand the terminology. The inference mechanisms underlying subsumption and in-\nstance checking go beyond simply checking for explicitly stated subsumption rela-\ntions in a terminology. They must explicitly reason using the relational information\nasserted about the terminology to infer appropriate subsumption and membership\nrelations.\nReturning to our restaurant domain, let\u2019s add a new kind of restaurant using the\nfollowing statement:\nIlFornaiovModerateRestaurant u9hasCuisine :ItalianCuisine (F.58)\nGiven this assertion, we might ask whether the IlFornaio chain of restaurants might\nbe classi\ufb01ed as an Italian restaurant or a vegetarian restaurant. More precisely, we\ncan pose the following questions to our reasoning system:\nIlFornaiovItalianRestaurant (F.59)\nIlFornaiovVegetarianRestaurant (F.60)\nThe answer to the \ufb01rst question is positive since IlFornaio meets the criteria we\nspeci\ufb01ed for the category ItalianRestaurant : it\u2019s a Restaurant since we explicitly\nclassi\ufb01ed it as a ModerateRestaurant , which is a subtype of Restaurant , and it\nmeets the has.Cuisine class restriction since we\u2019ve asserted that directly.\nThe answer to the second question is negative. Recall, that our criteria for veg-\netarian restaurants contains two requirements: it has to serve vegetarian fare, and\nthat\u2019s all it can serve. Our current de\ufb01nition for IlFornaio fails on both counts since\nwe have not asserted any relations that state that IlFornaio serves vegetarian fare,\nand the relation we have asserted, hasCuisine.ItalianCuisine , contradicts the sec-\nond criteria.\nA related reasoning task, based on the basic subsumption inference, is to derive\ntheimplied hierarchy for a terminology given facts about the categories in the ter-implied\nhierarchy\nminology. This task roughly corresponds to a repeated application of the subsump-\ntion operator to pairs of concepts in the terminology. Given our current collection\nof statements, the expanded hierarchy shown in Fig. F.6 can be inferred. You should\nconvince yourself that this diagram contains all and only the subsumption links that\nshould be present given our current knowledge.\nInstance checking is the task of determining whether a particular individual can\nbe classi\ufb01ed as a member of a particular category. This process takes what is known 20 APPENDIX F \u2022 L OGICAL REPRESENTATIONS OF SENTENCE MEANING\nRestaurant\nChineseRestaurant MexicanRestaurantItalianRestaurantExpensiveRestaurantCheapRestaurantModerateRestaurantIl FornaioVegetarianRestaurant\nFigure F.6 A graphical network representation of the complete set of subsumption relations\nin the restaurant domain given the current set of assertions in the TBox.\nabout a given individual, in the form of relations and explicit categorical statements,\nand then compares that information with what is known about the current terminol-\nogy. It then returns a list of the most speci\ufb01c categories to which the individual can\nbelong.\nAs an example of a categorization problem, consider an establishment that we\u2019re\ntold is a restaurant and serves Italian cuisine.\nRestaurant (Gondolier )\nhasCuisine (Gondolier ;ItalianCuisine )\nHere, we\u2019re being told that the entity denoted by the term Gondolier is a restau-\nrant and serves Italian food. Given this new information and the contents of our\ncurrent TBox, we might reasonably like to ask if this is an Italian restaurant, if it is\na vegetarian restaurant, or if it has moderate prices.\nAssuming the de\ufb01nitional statements given earlier, we can indeed categorize\nthe Gondolier as an Italian restaurant. That is, the information we\u2019ve been given\nabout it meets the necessary and suf\ufb01cient conditions required for membership in\nthis category. And as with the IlFornaio category, this individual fails to match the\nstated criteria for the VegetarianRestaurant . Finally, the Gondolier might also\nturn out to be a moderately priced restaurant, but we can\u2019t tell at this point since\nwe don\u2019t know anything about its prices. What this means is that given our current\nknowledge the answer to the query ModerateRestaurant (Gondolier )would be false\nsince it lacks the required hasPriceRange relation.\nThe implementation of subsumption, instance checking, as well as other kinds of\ninferences needed for practical applications, varies according to the expressivity of\nthe Description Logic being used. However, for a Description Logic of even modest\npower, the primary implementation techniques are based on satis\ufb01ability methods\nthat in turn rely on the underlying model-based semantics introduced earlier in this\nchapter.\nOWL and the Semantic Web\nThe highest-pro\ufb01le role for Description Logics, to date, has been as a part of the\ndevelopment of the Semantic Web. The Semantic Web is an ongoing effort to pro-\nvide a way to formally specify the semantics of the contents of the Web (Fensel\net al., 2003). A key component of this effort involves the creation and deployment\nof ontologies for various application areas of interest. The meaning representation F.6 \u2022 S UMMARY 21\nlanguage used to represent this knowledge is the Web Ontology Language (OWL)Web Ontology\nLanguage\n(McGuiness and van Harmelen, 2004). OWL embodies a Description Logic that\ncorresponds roughly to the one we\u2019ve been describing here.\nF.6 Summary\nThis chapter has introduced the representational approach to meaning. The follow-\ning are some of the highlights of this chapter:\n\u2022 A major approach to meaning in computational linguistics involves the cre-\nation of formal meaning representations that capture the meaning-related\ncontent of linguistic inputs. These representations are intended to bridge the\ngap from language to common-sense knowledge of the world.\n\u2022 The frameworks that specify the syntax and semantics of these representa-\ntions are called meaning representation languages . A wide variety of such\nlanguages are used in natural language processing and arti\ufb01cial intelligence.\n\u2022 Such representations need to be able to support the practical computational\nrequirements of semantic processing. Among these are the need to determine\nthe truth of propositions , to support unambiguous representations , to rep-\nresent variables , to support inference , and to be suf\ufb01ciently expressive .\n\u2022 Human languages have a wide variety of features that are used to convey\nmeaning. Among the most important of these is the ability to convey a predicate-\nargument structure .\n\u2022First-Order Logic is a well-understood, computationally tractable meaning\nrepresentation language that offers much of what is needed in a meaning rep-\nresentation language.\n\u2022 Important elements of semantic representation including states andevents\ncan be captured in FOL.\n\u2022Semantic networks andframes can be captured within the FOL framework.\n\u2022 Modern Description Logics consist of useful and computationally tractable\nsubsets of full First-Order Logic. The most prominent use of a description\nlogic is the Web Ontology Language (OWL), used in the speci\ufb01cation of the\nSemantic Web.\nBibliographical and Historical Notes\nThe earliest computational use of declarative meaning representations in natural lan-\nguage processing was in the context of question-answering systems (Green et al.\n1961, Raphael 1968, Lindsey 1963). These systems employed ad hoc representa-\ntions for the facts needed to answer questions. Questions were then translated into\na form that could be matched against facts in the knowledge base. Simmons (1965)\nprovides an overview of these early efforts.\nWoods (1967) investigated the use of FOL-like representations in question an-\nswering as a replacement for the ad hoc representations in use at the time. Woods\n(1973) further developed and extended these ideas in the landmark Lunar system. 22 APPENDIX F \u2022 L OGICAL REPRESENTATIONS OF SENTENCE MEANING\nInterestingly, the representations used in Lunar had both truth-conditional and pro-\ncedural semantics. Winograd (1972) employed a similar representation based on the\nMicro-Planner language in his SHRDLU system.\nDuring this same period, researchers interested in the cognitive modeling of lan-\nguage and memory had been working with various forms of associative network\nrepresentations. Masterman (1957) was the \ufb01rst to make computational use of a\nsemantic network-like knowledge representation, although semantic networks are\ngenerally credited to Quillian (1968). A considerable amount of work in the se-\nmantic network framework was carried out during this era (Norman and Rumelhart\n1975, Schank 1972, Wilks 1975b, Wilks 1975a, Kintsch 1974). It was during this\nperiod that a number of researchers began to incorporate Fillmore\u2019s notion of case\nroles (Fillmore, 1968) into their representations. Simmons (1973) was the earliest\nadopter of case roles as part of representations for natural language processing.\nDetailed analyses by Woods (1975) and Brachman (1979) aimed at \ufb01guring out\nwhat semantic networks actually mean led to the development of a number of more\nsophisticated network-like languages including KRL (Bobrow and Winograd, 1977)\nand KL-ONE (Brachman and Schmolze, 1985). As these frameworks became more\nsophisticated and well de\ufb01ned, it became clear that they were restricted variants of\nFOL coupled with specialized indexing inference procedures. A useful collection of\npapers covering much of this work can be found in Brachman and Levesque (1985).\nRussell and Norvig (2002) describe a modern perspective on these representational\nefforts.\nLinguistic efforts to assign semantic structures to natural language sentences in\nthe generative era began with the work of Katz and Fodor (1963). The limitations of\ntheir simple feature-based representations and the natural \ufb01t of logic to many of the\nlinguistic problems of the day quickly led to the adoption of a variety of predicate-\nargument structures as preferred semantic representations (Lakoff 1972, McCawley\n1968). The subsequent introduction by Montague (1973) of the truth-conditional\nmodel-theoretic framework into linguistic theory led to a much tighter integration\nbetween theories of formal syntax and a wide range of formal semantic frameworks.\nGood introductions to Montague semantics and its role in linguistic theory can be\nfound in Dowty et al. (1981) and Partee (1976).\nThe representation of events as rei\ufb01ed objects is due to Davidson (1967). The\napproach presented here, which explicitly rei\ufb01es event participants, is due to Parsons\n(1990).\nA recent comprehensive treatment of logic and language can be found in van\nBenthem and ter Meulen (1997). A classic semantics text is Lyons (1977). McCaw-\nley (1993) is an indispensable textbook covering a wide range of topics concerning\nlogic and language. Chierchia and McConnell-Ginet (1991) also broadly covers\nsemantic issues from a linguistic perspective. Heim and Kratzer (1998) is a more\nrecent text written from the perspective of current generative theory.\nExercises\nF.1 Peruse your daily newspaper for three examples of ambiguous sentences or\nheadlines. Describe the various sources of the ambiguities.\nF.2 Consider a domain in which the word coffee can refer to the following con-\ncepts in a knowledge-based system: a caffeinated or decaffeinated beverage,\nground coffee used to make either kind of beverage, and the beans themselves. EXERCISES 23\nGive arguments as to which of the following uses of coffee are ambiguous and\nwhich are vague.\n1. I\u2019ve had my coffee for today.\n2. Buy some coffee on your way home.\n3. Please grind some more coffee.\nF.3 The following rule, which we gave as a translation for Example F.25, is not a\nreasonable de\ufb01nition of what it means to be a vegetarian restaurant.\n8xVegetarianRestaurant (x) =)Serves (x;VegetarianFood )\nGive a FOLrule that better de\ufb01nes vegetarian restaurants in terms of what they\nserve.\nF.4 Give FOL translations for the following sentences:\n1. Vegetarians do not eat meat.\n2. Not all vegetarians eat eggs.\nF.5 Give a set of facts and inferences necessary to prove the following assertions:\n1. McDonald\u2019s is not a vegetarian restaurant.\n2. Some vegetarians can eat at McDonald\u2019s.\nDon\u2019t just place these facts in your knowledge base. Show that they can be\ninferred from some more general facts about vegetarians and McDonald\u2019s.\nF.6 On page 12, we gave the representation Near (Centro ;Bacaro )as a transla-\ntion for the sentence Centro is near Bacaro . In a truth-conditional semantics,\nthis formula is either true or false given some model. Critique this truth-\nconditional approach with respect to the meaning of words like near. 24 Appendix F \u2022 Logical Representations of Sentence Meaning\nBanarescu, L., C. Bonial, S. Cai, M. Georgescu, K. Grif-\n\ufb01tt, U. Hermjakob, K. Knight, P. Koehn, M. Palmer, and\nN. Schneider. 2013. Abstract meaning representation for\nsembanking. 7th Linguistic Annotation Workshop and In-\nteroperability with Discourse .\nvan Benthem, J. and A. ter Meulen, eds. 1997. Handbook of\nLogic and Language . MIT Press.\nBobrow, D. G. and T. Winograd. 1977. An overview of KRL,\na knowledge representation language. Cognitive Science ,\n1(1):3\u201346.\nBrachman, R. J. 1979. On the epistemogical status of seman-\ntic networks. In N. V . Findler, ed., Associative Networks:\nRepresentation and Use of Knowledge by Computers , 3\u2013\n50. Academic Press.\nBrachman, R. J. and H. J. Levesque, eds. 1985. Readings in\nKnowledge Representation . Morgan Kaufmann.\nBrachman, R. J. and J. G. Schmolze. 1985. An overview of\nthe KL-ONE knowledge representation system. Cogni-\ntive Science , 9(2):171\u2013216.\nChierchia, G. and S. McConnell-Ginet. 1991. Meaning and\nGrammar . MIT Press.\nChurch, A. 1940. A formulation of a simple theory of types.\nJournal of Symbolic Logic , 5:56\u201368.\nDavidson, D. 1967. The logical form of action sentences. In\nN. Rescher, ed., The Logic of Decision and Action . Uni-\nversity of Pittsburgh Press.\nDowty, D. R., R. E. Wall, and S. Peters. 1981. Introduction\nto Montague Semantics . D. Reidel.\nFensel, D., J. A. Hendler, H. Lieberman, and W. Wahlster,\neds. 2003. Spinning the Semantic Web: Bring the World\nWide Web to its Full Potential . MIT Press, Cambridge,\nMA.\nFillmore, C. J. 1968. The case for case. In E. W. Bach and\nR. T. Harms, eds, Universals in Linguistic Theory , 1\u201388.\nHolt, Rinehart & Winston.\nGreen, B. F., A. K. Wolf, C. Chomsky, and K. Laughery.\n1961. Baseball: An automatic question answerer. Pro-\nceedings of the Western Joint Computer Conference 19 .\nHeim, I. and A. Kratzer. 1998. Semantics in a Generative\nGrammar . Blackwell Publishers, Malden, MA.\nKatz, J. J. and J. A. Fodor. 1963. The structure of a semantic\ntheory. Language , 39:170\u2013210.\nKintsch, W. 1974. The Representation of Meaning in Mem-\nory. Wiley, New York.\nLakoff, G. 1972. Linguistics and natural logic. In D. David-\nson and G. Harman, eds, Semantics for Natural Lan-\nguage , 545\u2013665. D. Reidel.\nLindsey, R. 1963. Inferential memory as the basis of ma-\nchines which understand natural language. In E. Feigen-\nbaum and J. Feldman, eds, Computers and Thought , 217\u2013\n233. McGraw Hill.\nLyons, J. 1977. Semantics . Cambridge University Press.\nMasterman, M. 1957. The thesaurus in syntax and semantics.\nMechanical Translation , 4(1):1\u20132.\nMcCawley, J. D. 1968. The role of semantics in a gram-\nmar. In E. W. Bach and R. T. Harms, eds, Universals in\nLinguistic Theory , 124\u2013169. Holt, Rinehart & Winston.McCawley, J. D. 1993. Everything that Linguists Have Al-\nways Wanted to Know about Logic , 2nd edition. Univer-\nsity of Chicago Press, Chicago, IL.\nMcGuiness, D. L. and F. van Harmelen. 2004. OWL web\nontology overview. Technical Report 20040210, World\nWide Web Consortium.\nMontague, R. 1973. The proper treatment of quanti\ufb01cation\nin ordinary English. In R. Thomason, ed., Formal Philos-\nophy: Selected Papers of Richard Montague , 247\u2013270.\nYale University Press, New Haven, CT.\nNorman, D. A. and D. E. Rumelhart. 1975. Explorations in\nCognition . Freeman.\nParsons, T. 1990. Events in the Semantics of English . MIT\nPress.\nPartee, B. H., ed. 1976. Montague Grammar . Academic\nPress.\nQuillian, M. R. 1968. Semantic memory. In M. Minsky, ed.,\nSemantic Information Processing , 227\u2013270. MIT Press.\nRaphael, B. 1968. SIR: A computer program for semantic\ninformation retrieval. In M. Minsky, ed., Semantic Infor-\nmation Processing , 33\u2013145. MIT Press.\nRussell, S. and P. Norvig. 2002. Arti\ufb01cial Intelligence: A\nModern Approach , 2nd edition. Prentice Hall.\nSchank, R. C. 1972. Conceptual dependency: A theory\nof natural language processing. Cognitive Psychology ,\n3:552\u2013631.\nSch\u00a8on\ufb01nkel, M. 1924. \u00a8Uber die Bausteine der mathema-\ntischen Logik. Mathematische Annalen , 92:305\u2013316.\nEnglish translation appears in From Frege to G \u00a8odel: A\nSource Book in Mathematical Logic , Harvard University\nPress, 1967.\nSimmons, R. F. 1965. Answering English questions by com-\nputer: A survey. CACM , 8(1):53\u201370.\nSimmons, R. F. 1973. Semantic networks: Their compu-\ntation and use for understanding English sentences. In\nR. C. Schank and K. M. Colby, eds, Computer Models of\nThought and Language , 61\u2013113. W.H. Freeman & Co.\nWilks, Y . 1975a. Preference semantics. In E. L. Keenan,\ned.,The Formal Semantics of Natural Language , 329\u2013\n350. Cambridge Univ. Press.\nWilks, Y . 1975b. A preferential, pattern-seeking, seman-\ntics for natural language inference. Arti\ufb01cial Intelligence ,\n6(1):53\u201374.\nWinograd, T. 1972. Understanding Natural Language . Aca-\ndemic Press.\nWoods, W. A. 1967. Semantics for a Question-Answering\nSystem . Ph.D. thesis, Harvard University.\nWoods, W. A. 1973. Progress in natural language under-\nstanding. Proceedings of AFIPS National Conference .\nWoods, W. A. 1975. What\u2019s in a link: Foundations for se-\nmantic networks. In D. G. Bobrow and A. M. Collins, eds,\nRepresentation and Understanding: Studies in Cognitive\nScience , 35\u201382. Academic Press.", "G.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\nGWord Senses and WordNet\nLady Bracknell . Are your parents living?\nJack . I have lost both my parents.\nLady Bracknell. To lose one parent, Mr. Worthing, may be regarded as a\nmisfortune; to lose both looks like carelessness.\nOscar Wilde, The Importance of Being Earnest\nWords are ambiguous : the same word can be used to mean different things. In ambiguous\nChapter 6 we saw that the word \u201cmouse\u201d has (at least) two meanings: (1) a small\nrodent, or (2) a hand-operated device to control a cursor. The word \u201cbank\u201d can\nmean: (1) a \ufb01nancial institution or (2) a sloping mound. In the quote above from\nhis play The Importance of Being Earnest , Oscar Wilde plays with two meanings of\n\u201close\u201d (to misplace an object, and to suffer the death of a close person).\nWe say that the words \u2018mouse\u2019 or \u2018bank\u2019 are polysemous (from Greek \u2018having\nmany senses\u2019, poly- \u2018many\u2019 + sema , \u2018sign, mark\u2019).1Asense (orword sense ) is word sense\na discrete representation of one aspect of the meaning of a word. In this chapter\nwe discuss word senses in more detail and introduce WordNet , a large online the- WordNet\nsaurus \u2014a database that represents word senses\u2014with versions in many languages.\nWordNet also represents relations between senses. For example, there is an IS-A\nrelation between dogandmammal (a dog is a kind of mammal) and a part-whole\nrelation between engine andcar(an engine is a part of a car).\nKnowing the relation between two senses can play an important role in tasks\ninvolving meaning. Consider the antonymy relation. Two words are antonyms if\nthey have opposite meanings, like long andshort , orupanddown . Distinguishing\nthese is quite important; if a user asks a dialogue agent to turn up the music, it\nwould be unfortunate to instead turn it down. But in fact in embedding models like\nword2vec, antonyms are easily confused with each other, because often one of the\nclosest words in embedding space to a word (e.g., up) is its antonym (e.g., down ).\nThesauruses that represent this relationship can help!\nWe also introduce word sense disambiguation (WSD ), the task of determiningword sense\ndisambiguation\nwhich sense of a word is being used in a particular context. We\u2019ll give supervised\nand unsupervised algorithms for deciding which sense was intended in a particular\ncontext. This task has a very long history in computational linguistics and many ap-\nplications. In question answering, we can be more helpful to a user who asks about\n\u201cbat care\u201d if we know which sense of bat is relevant. (Is the user a vampire? or\njust wants to play baseball.) And the different senses of a word often have differ-\nent translations; in Spanish the animal bat is a murci \u00b4elago while the baseball bat is\nabate, and indeed word sense algorithms may help improve MT (Pu et al., 2018).\nFinally, WSD has long been used as a tool for evaluating language processing mod-\nels, and understanding how models represent different word senses is an important\n1The word polysemy itself is ambiguous; you may see it used in a different way, to refer only to cases\nwhere a word\u2019s senses are related in some structured way, reserving the word homonymy to mean sense\nambiguities with no relation between the senses (Haber and Poesio, 2020). Here we will use \u2018polysemy\u2019\nto mean any kind of sense ambiguity, and \u2018structured polysemy\u2019 for polysemy with sense relations. 2APPENDIX G \u2022 W ORD SENSES AND WORDNET\nanalytic direction.\nG.1 Word Senses\nAsense (orword sense ) is a discrete representation of one aspect of the meaning of word sense\na word. Loosely following lexicographic tradition, we represent each sense with a\nsuperscript: bank1andbank2,mouse1andmouse2. In context, it\u2019s easy to see the\ndifferent meanings:\nmouse1: .... a mouse controlling a computer system in 1968.\nmouse2: .... a quiet animal like a mouse\nbank1: ...a bank can hold the investments in a custodial account ...\nbank2: ...as agriculture burgeons on the east bank , the river ...\nG.1.1 De\ufb01ning Word Senses\nHow can we de\ufb01ne the meaning of a word sense? We introduced in Chapter 6 the\nstandard computational approach of representing a word as an embedding , a point\nin semantic space. The intuition of embedding models like word2vec or GloVe is\nthat the meaning of a word can be de\ufb01ned by its co-occurrences, the counts of words\nthat often occur nearby. But that doesn\u2019t tell us how to de\ufb01ne the meaning of a word\nsense . As we saw in Chapter 11, contextual embeddings like BERT go further by\noffering an embedding that represents the meaning of a word in its textual context,\nand we\u2019ll see that contextual embeddings lie at the heart of modern algorithms for\nword sense disambiguation.\nBut \ufb01rst, we need to consider the alternative ways that dictionaries and the-\nsauruses offer for de\ufb01ning senses. One is based on the fact that dictionaries or the-\nsauruses give textual de\ufb01nitions for each sense called glosses . Here are the glosses gloss\nfor two senses of bank :\n1. financial institution that accepts deposits and channels\nthe money into lending activities\n2. sloping land (especially the slope beside a body of water)\nGlosses are not a formal meaning representation; they are just written for people.\nConsider the following fragments from the de\ufb01nitions of right ,left,red, and blood\nfrom the American Heritage Dictionary (Morris, 1985).\nright adj. located nearer the right hand esp. being on the right when\nfacing the same direction as the observer.\nleft adj.located nearer to this side of the body than the right.\nred n.the color of blood or a ruby.\nblood n.the red liquid that circulates in the heart, arteries and veins of\nanimals.\nNote the circularity in these de\ufb01nitions. The de\ufb01nition of right makes two direct\nreferences to itself, and the entry for leftcontains an implicit self-reference in the\nphrase this side of the body , which presumably means the leftside. The entries for\nredandblood reference each other in their de\ufb01nitions. For humans, such entries are\nuseful since the user of the dictionary has suf\ufb01cient grasp of these other terms. G.1 \u2022 W ORD SENSES 3\nYet despite their circularity and lack of formal representation, glosses can still\nbe useful for computational modeling of senses. This is because a gloss is just a sen-\ntence, and from sentences we can compute sentence embeddings that tell us some-\nthing about the meaning of the sense. Dictionaries often give example sentences\nalong with glosses, and these can again be used to help build a sense representation.\nThe second way that thesauruses offer for de\ufb01ning a sense is\u2014like the dictionary\nde\ufb01nitions\u2014de\ufb01ning a sense through its relationship with other senses. For exam-\nple, the above de\ufb01nitions make it clear that right andleftare similar kinds of lemmas\nthat stand in some kind of alternation, or opposition, to one another. Similarly, we\ncan glean that redis a color and that blood is aliquid .Sense relations of this sort\n(IS-A , orantonymy ) are explicitly listed in on-line databases like WordNet . Given\na suf\ufb01ciently large database of such relations, many applications are quite capable\nof performing sophisticated semantic tasks about word senses (even if they do not\nreally know their right from their left).\nG.1.2 How many senses do words have?\nDictionaries and thesauruses give discrete lists of senses. By contrast, embeddings\n(whether static or contextual) offer a continuous high-dimensional model of meaning\nthat doesn\u2019t divide up into discrete senses.\nTherefore creating a thesaurus depends on criteria for deciding when the differ-\ning uses of a word should be represented with discrete senses. We might consider\ntwo senses discrete if they have independent truth conditions, different syntactic be-\nhavior, and independent sense relations, or if they exhibit antagonistic meanings.\nConsider the following uses of the verb serve from the WSJ corpus:\n(G.1) They rarely serve red meat, preferring to prepare seafood.\n(G.2) He served as U.S. ambassador to Norway in 1976 and 1977.\n(G.3) He might have served his time, come out and led an upstanding life.\nTheserve ofserving red meat and that of serving time clearly have different truth\nconditions and presuppositions; the serve ofserve as ambassador has the distinct\nsubcategorization structure serve as NP . These heuristics suggest that these are prob-\nably three distinct senses of serve . One practical technique for determining if two\nsenses are distinct is to conjoin two uses of a word in a single sentence; this kind\nof conjunction of antagonistic readings is called zeugma . Consider the following zeugma\nexamples:\n(G.4) Which of those \ufb02ights serve breakfast?\n(G.5) Does Air France serve Philadelphia?\n(G.6) ?Does Air France serve breakfast and Philadelphia?\nWe use (?) to mark those examples that are semantically ill-formed. The oddness of\nthe invented third example (a case of zeugma) indicates there is no sensible way to\nmake a single sense of serve work for both breakfast and Philadelphia. We can use\nthis as evidence that serve has two different senses in this case.\nDictionaries tend to use many \ufb01ne-grained senses so as to capture subtle meaning\ndifferences, a reasonable approach given that the traditional role of dictionaries is\naiding word learners. For computational purposes, we often don\u2019t need these \ufb01ne\ndistinctions, so we often group or cluster the senses; we have already done this for\nsome of the examples in this chapter. Indeed, clustering examples into senses, or\nsenses into broader-grained categories, is an important computational task that we\u2019ll\ndiscuss in Section G.7. 4APPENDIX G \u2022 W ORD SENSES AND WORDNET\nG.2 Relations Between Senses\nThis section explores the relations between word senses, especially those that have\nreceived signi\ufb01cant computational investigation like synonymy ,antonymy , and hy-\npernymy .\nSynonymy\nWe introduced in Chapter 6 the idea that when two senses of two different words\n(lemmas) are identical, or nearly identical, we say the two senses are synonyms . synonym\nSynonyms include such pairs as\ncouch/sofa vomit/throw up \ufb01lbert/hazelnut car/automobile\nAnd we mentioned that in practice, the word synonym is commonly used to\ndescribe a relationship of approximate or rough synonymy. But furthermore, syn-\nonymy is actually a relationship between senses rather than words. Considering the\nwords bigandlarge . These may seem to be synonyms in the following sentences,\nsince we could swap bigandlarge in either sentence and retain the same meaning:\n(G.7) How big is that plane?\n(G.8) Would I be \ufb02ying on a large or small plane?\nBut note the following sentence in which we cannot substitute large forbig:\n(G.9) Miss Nelson, for instance, became a kind of big sister to Benjamin.\n(G.10) ?Miss Nelson, for instance, became a kind of large sister to Benjamin.\nThis is because the word bighas a sense that means being older or grown up, while\nlarge lacks this sense. Thus, we say that some senses of bigandlarge are (nearly)\nsynonymous while other ones are not.\nAntonymy\nWhereas synonyms are words with identical or similar meanings, antonyms are antonym\nwords with an opposite meaning, like:\nlong/short big/little fast/slow cold/hot dark/light\nrise/fall up/down in/out\nTwo senses can be antonyms if they de\ufb01ne a binary opposition or are at opposite\nends of some scale. This is the case for long/short ,fast/slow , orbig/little, which are\nat opposite ends of the length orsizescale. Another group of antonyms, reversives , reversives\ndescribe change or movement in opposite directions, such as rise/fall orup/down .\nAntonyms thus differ completely with respect to one aspect of their meaning\u2014\ntheir position on a scale or their direction\u2014but are otherwise very similar, sharing\nalmost all other aspects of meaning. Thus, automatically distinguishing synonyms\nfrom antonyms can be dif\ufb01cult.\nTaxonomic Relations\nAnother way word senses can be related is taxonomically. A word (or sense) is a\nhyponym of another word or sense if the \ufb01rst is more speci\ufb01c, denoting a subclass hyponym\nof the other. For example, caris a hyponym of vehicle ,dogis a hyponym of animal ,\nandmango is a hyponym of fruit. Conversely, we say that vehicle is ahypernym of hypernym\ncar, and animal is a hypernym of dog. It is unfortunate that the two words (hypernym G.2 \u2022 R ELATIONS BETWEEN SENSES 5\nand hyponym) are very similar and hence easily confused; for this reason, the word\nsuperordinate is often used instead of hypernym . superordinate\nSuperordinate vehicle fruit furniture mammal\nSubordinate car mango chair dog\nWe can de\ufb01ne hypernymy more formally by saying that the class denoted by\nthe superordinate extensionally includes the class denoted by the hyponym. Thus,\nthe class of animals includes as members all dogs, and the class of moving actions\nincludes all walking actions. Hypernymy can also be de\ufb01ned in terms of entail-\nment . Under this de\ufb01nition, a sense Ais a hyponym of a sense Bif everything\nthat is Ais also B, and hence being an Aentails being a B, or8x A(x))B(x). Hy-\nponymy/hypernymy is usually a transitive relation; if A is a hyponym of B and B is a\nhyponym of C, then A is a hyponym of C. Another name for the hypernym/hyponym\nstructure is the IS-A hierarchy, in which we say A IS-A B, or B subsumes A . IS-A\nHypernymy is useful for tasks like textual entailment or question answering;\nknowing that leukemia is a type of cancer , for example, would certainly be useful in\nanswering questions about leukemia.\nMeronymy\nAnother common relation is meronymy , the part-whole relation. A legis part of a part-whole\nchair ; awheel is part of a car. We say that wheel is ameronym ofcar, and caris a\nholonym ofwheel .\nStructured Polysemy\nThe senses of a word can also be related semantically, in which case we call the\nrelationship between them structured polysemy . Consider this sense bank :structured\npolysemy\n(G.11) The bank is on the corner of Nassau and Witherspoon.\nThis sense, perhaps bank4, means something like \u201cthe building belonging to\na \ufb01nancial institution\u201d. These two kinds of senses (an organization and the build-\ning associated with an organization ) occur together for many other words as well\n(school ,university ,hospital , etc.). Thus, there is a systematic relationship between\nsenses that we might represent as\nBUILDING$ORGANIZATION\nThis particular subtype of polysemy relation is called metonymy . Metonymy is metonymy\nthe use of one aspect of a concept or entity to refer to other aspects of the entity or\nto the entity itself. We are performing metonymy when we use the phrase the White\nHouse to refer to the administration whose of\ufb01ce is in the White House. Other\ncommon examples of metonymy include the relation between the following pairings\nof senses:\nAUTHOR $WORKS OF AUTHOR\n(Jane Austen wrote Emma ) (I really love Jane Austen )\nFRUITTREE $FRUIT\n(Plums have beautiful blossoms ) (I ate a preserved plum yesterday ) 6APPENDIX G \u2022 W ORD SENSES AND WORDNET\nG.3 WordNet: A Database of Lexical Relations\nThe most commonly used resource for sense relations in English and many other\nlanguages is the WordNet lexical database (Fellbaum, 1998). English WordNet WordNet\nconsists of three separate databases, one each for nouns and verbs and a third for\nadjectives and adverbs; closed class words are not included. Each database contains\na set of lemmas, each one annotated with a set of senses. The WordNet 3.0 release\nhas 117,798 nouns, 11,529 verbs, 22,479 adjectives, and 4,481 adverbs. The aver-\nage noun has 1.23 senses, and the average verb has 2.16 senses. WordNet can be\naccessed on the Web or downloaded locally. Figure G.1 shows the lemma entry for\nthe noun bass.\nThe noun \u201cbass\u201d has 8 senses in WordNet.\n1. bass1- (the lowest part of the musical range)\n2. bass2, bass part1- (the lowest part in polyphonic music)\n3. bass3, basso1- (an adult male singer with the lowest voice)\n4. sea bass1, bass4- (the lean \ufb02esh of a saltwater \ufb01sh of the family Serranidae)\n5. freshwater bass1, bass5- (any of various North American freshwater \ufb01sh with\nlean \ufb02esh (especially of the genus Micropterus))\n6. bass6, bass voice1, basso2- (the lowest adult male singing voice)\n7. bass7- (the member with the lowest range of a family of musical instruments)\n8. bass8- (nontechnical name for any of numerous edible marine and\nfreshwater spiny-\ufb01nned \ufb01shes)\nFigure G.1 A portion of the WordNet 3.0 entry for the noun bass.\nNote that there are eight senses, each of which has a gloss (a dictionary-style gloss\nde\ufb01nition), a list of synonyms for the sense, and sometimes also usage examples.\nWordNet doesn\u2019t represent pronunciation, so doesn\u2019t distinguish the pronunciation\n[b ae s] in bass4,bass5, and bass8from the other senses pronounced [b ey s].\nThe set of near-synonyms for a WordNet sense is called a synset (forsynonym synset\nset); synsets are an important primitive in WordNet. The entry for bass includes\nsynsets likefbass1, deep6g, orfbass6, bass voice1, basso2g. We can think of a\nsynset as representing a concept of the type we discussed in Appendix F. Thus,\ninstead of representing concepts in logical terms, WordNet represents them as lists\nof the word senses that can be used to express the concept. Here\u2019s another synset\nexample:\nfchump1, fool2, gull1, mark9, patsy1, fall guy1,\nsucker1, soft touch1, mug2g\nThe gloss of this synset describes it as:\nGloss : a person who is gullible and easy to take advantage of.\nGlosses are properties of a synset, so that each sense included in the synset has the\nsame gloss and can express this concept. Because they share glosses, synsets like\nthis one are the fundamental unit associated with WordNet entries, and hence it is\nsynsets, not wordforms, lemmas, or individual senses, that participate in most of the\nlexical sense relations in WordNet.\nWordNet also labels each synset with a lexicographic category drawn from a\nsemantic \ufb01eld for example the 26 categories for nouns shown in Fig. G.2, as well\nas 15 for verbs (plus 2 for adjectives and 1 for adverbs). These categories are often G.3 \u2022 W ORDNET: A D ATABASE OF LEXICAL RELATIONS 7\ncalled supersenses , because they act as coarse semantic categories or groupings of supersense\nsenses which can be useful when word senses are too \ufb01ne-grained (Ciaramita and\nJohnson 2003, Ciaramita and Altun 2006). Supersenses have also been de\ufb01ned for\nadjectives (Tsvetkov et al., 2014) and prepositions (Schneider et al., 2018).\nCategory Example Category Example Category Example\nACT service GROUP place PLANT tree\nANIMAL dog LOCATION area POSSESSION price\nARTIFACT car MOTIVE reason PROCESS process\nATTRIBUTE quality NATURAL EVENT experience QUANTITY amount\nBODY hair NATURAL OBJECT \ufb02ower RELATION portion\nCOGNITION way OTHER stuff SHAPE square\nCOMMUNICATION review PERSON people STATE pain\nFEELING discomfort PHENOMENON result SUBSTANCE oil\nFOOD food TIME day\nFigure G.2 Supersenses: 26 lexicographic categories for nouns in WordNet.\nG.3.1 Sense Relations in WordNet\nWordNet represents all the kinds of sense relations discussed in the previous section,\nas illustrated in Fig. G.3 and Fig. G.4.\nRelation Also Called De\ufb01nition Example\nHypernym Superordinate From concepts to superordinates breakfast1!meal1\nHyponym Subordinate From concepts to subtypes meal1!lunch1\nInstance Hypernym Instance From instances to their concepts Austen1!author1\nInstance Hyponym Has-Instance From concepts to their instances composer1!Bach1\nPart Meronym Has-Part From wholes to parts table2!leg3\nPart Holonym Part-Of From parts to wholes course7!meal1\nAntonym Semantic opposition between lemmas leader1() follower1\nDerivation Lemmas w/same morphological root destruction1() destroy1\nFigure G.3 Some of the noun relations in WordNet.\nRelation De\ufb01nition Example\nHypernym From events to superordinate events \ufb02y9!travel5\nTroponym From events to subordinate event walk1!stroll1\nEntails From verbs (events) to the verbs (events) they entail snore1!sleep1\nAntonym Semantic opposition between lemmas increase1() decrease1\nFigure G.4 Some verb relations in WordNet.\nFor example WordNet represents hyponymy (page 4) by relating each synset to\nits immediately more general and more speci\ufb01c synsets through direct hypernym\nand hyponym relations. These relations can be followed to produce longer chains of\nmore general or more speci\ufb01c synsets. Figure G.5 shows hypernym chains for bass3\nandbass7; more general synsets are shown on successively indented lines.\nWordNet has two kinds of taxonomic entities: classes and instances. An instance\nis an individual, a proper noun that is a unique entity. San Francisco is an instance\nofcity, for example. But cityis a class, a hyponym of municipality and eventually\noflocation . Fig. G.6 shows a subgraph of WordNet demonstrating many of the\nrelations. 8APPENDIX G \u2022 W ORD SENSES AND WORDNET\nbass3, basso (an adult male singer with the lowest voice)\n=> singer, vocalist, vocalizer, vocaliser\n=> musician, instrumentalist, player\n=> performer, performing artist\n=> entertainer\n=> person, individual, someone...\n=> organism, being\n=> living thing, animate thing,\n=> whole, unit\n=> object, physical object\n=> physical entity\n=> entity\nbass7(member with the lowest range of a family of instruments)\n=> musical instrument, instrument\n=> device\n=> instrumentality, instrumentation\n=> artifact, artefact\n=> whole, unit\n=> object, physical object\n=> physical entity\n=> entity\nFigure G.5 Hyponymy chains for two separate senses of the lemma bass. Note that the\nchains are completely distinct, only converging at the very abstract level whole, unit .\nFigure G.6 WordNet viewed as a graph. Figure from Navigli (2016).\nG.4 Word Sense Disambiguation\nThe task of selecting the correct sense for a word is called word sense disambigua-\ntion, orWSD . WSD algorithms take as input a word in context and a \ufb01xed inventoryword sense\ndisambiguation\nWSD of potential word senses and outputs the correct word sense in context. G.4 \u2022 W ORD SENSE DISAMBIGUATION 9\nG.4.1 WSD: The Task and Datasets\nIn this section we introduce the task setup for WSD, and then turn to algorithms.\nThe inventory of sense tags depends on the task. For sense tagging in the context\nof translation from English to Spanish, the sense tag inventory for an English word\nmight be the set of different Spanish translations. For automatic indexing of med-\nical articles, the sense-tag inventory might be the set of MeSH (Medical Subject\nHeadings) thesaurus entries. Or we can use the set of senses from a resource like\nWordNet, or supersenses if we want a coarser-grain set. Figure G.4.1 shows some\nsuch examples for the word bass.\nWordNet Spanish WordNet\nSense Translation Supersense Target Word in Context\nbass4lubina FOOD . . . \ufb01sh as Paci\ufb01c salmon and striped bass and. . .\nbass7bajo ARTIFACT . . . play bass because he doesn\u2019t have to solo. . .\nFigure G.7 Some possible sense tag inventories for bass.\nIn some situations, we just need to disambiguate a small number of words. In\nsuch lexical sample tasks, we have a small pre-selected set of target words and an lexical sample\ninventory of senses for each word from some lexicon. Since the set of words and the\nset of senses are small, simple supervised classi\ufb01cation approaches work very well.\nMore commonly, however, we have a harder problem in which we have to dis-\nambiguate all the words in some text. In this all-words task, the system is given an all-words\nentire texts and a lexicon with an inventory of senses for each entry and we have to\ndisambiguate every word in the text (or sometimes just every content word). The\nall-words task is similar to part-of-speech tagging, except with a much larger set of\ntags since each lemma has its own set. A consequence of this larger set of tags is\ndata sparseness.\nSupervised all-word disambiguation tasks are generally trained from a semantic\nconcordance , a corpus in which each open-class word in each sentence is labeledsemantic\nconcordance\nwith its word sense from a speci\ufb01c dictionary or thesaurus, most often WordNet.\nThe SemCor corpus is a subset of the Brown Corpus consisting of over 226,036\nwords that were manually tagged with WordNet senses (Miller et al. 1993, Landes\net al. 1998). Other sense-tagged corpora have been built for the SENSEVAL andSe-\nmEval WSD tasks, such as the SENSEVAL -3 Task 1 English all-words test data with\n2282 annotations (Snyder and Palmer, 2004) or the SemEval-13 Task 12 datasets.\nLarge semantic concordances are also available in other languages including Dutch\n(V ossen et al., 2011) and German (Henrich et al., 2012).\nHere\u2019s an example from the SemCor corpus showing the WordNet sense num-\nbers of the tagged words; we\u2019ve used the standard WSD notation in which a subscript\nmarks the part of speech (Navigli, 2009):\n(G.12) You will \ufb01nd9\nvthat avocado1\nnis1\nvunlike1\njother1\njfruit1\nnyou have ever1\nrtasted2\nv\nGiven each noun, verb, adjective, or adverb word in the hand-labeled test set (say\nfruit), the SemCor-based WSD task is to choose the correct sense from the possible\nsenses in WordNet. For fruit this would mean choosing between the correct answer\nfruit1\nn(the ripened reproductive body of a seed plant), and the other two senses fruit2\nn\n(yield; an amount of a product) and fruit3\nn(the consequence of some effort or action).\nFig. G.8 sketches the task.\nWSD systems are typically evaluated intrinsically, by computing F1 against\nhand-labeled sense tags in a held-out set, such as the SemCor corpus or SemEval\ncorpora discussed above. 10 APPENDIX G \u2022 W ORD SENSES AND WORDNET\nanelectricguitarandbassplayerstando\ufb00toonesideelectric1: using electricityelectric2:  tenseelectric3: thrillingguitar1 bass1: low range\u2026bass4: sea \ufb01sh\u2026 bass7: instrument\u2026player1: in gameplayer2: musician player3: actor\u2026stand1: upright\u2026stand5: bear\u2026 stand10: put upright\u2026side1: relative region\u2026side3: of body\u2026 side11: slope\u2026x1y1x2y2x3y3y4y5y6\nx4x5x6\nFigure G.8 The all-words WSD task, mapping from input words ( x) to WordNet senses\n(y). Only nouns, verbs, adjectives, and adverbs are mapped, and note that some words (like\nguitar in the example) only have one sense in WordNet. Figure inspired by Chaplot and\nSalakhutdinov (2018).\nA surprisingly strong baseline is simply to choose the most frequent sense formost frequent\nsense\neach word from the senses in a labeled corpus (Gale et al., 1992a). For WordNet, this\ncorresponds to the \ufb01rst sense, since senses in WordNet are generally ordered from\nmost frequent to least frequent based on their counts in the SemCor sense-tagged\ncorpus. The most frequent sense baseline can be quite accurate, and is therefore\noften used as a default, to supply a word sense when a supervised algorithm has\ninsuf\ufb01cient training data.\nA second heuristic, called one sense per discourse is based on the work ofone sense per\ndiscourse\nGale et al. (1992b), who noticed that a word appearing multiple times in a text or\ndiscourse often appears with the same sense. This heuristic seems to hold better for\ncoarse-grained senses and particularly when a word\u2019s senses are unrelated, so isn\u2019t\ngenerally used as a baseline. Nonetheless various kinds of disambiguation tasks\noften include some such bias toward resolving an ambiguity the same way inside a\ndiscourse segment.\nG.4.2 The WSD Algorithm: Contextual Embeddings\nThe best performing WSD algorithm is a simple 1-nearest-neighbor algorithm using\ncontextual word embeddings, due to Melamud et al. (2016) and Peters et al. (2018).\nAt training time we pass each sentence in the SemCore labeled dataset through any\ncontextual embedding (e.g., BERT) resulting in a contextual embedding for each\nlabeled token in SemCore. (There are various ways to compute this contextual em-\nbedding vifor a token i; for BERT it is common to pool multiple layers by summing\nthe vector representations of ifrom the last four BERT layers). Then for each sense\nsof any word in the corpus, for each of the ntokens of that sense, we average their\nncontextual representations vito produce a contextual sense embedding v sfors:\nvs=1\nnX\nivi8vi2tokens (s) (G.13)\nAt test time, given a token of a target word tin context, we compute its contextual\nembedding tand choose its nearest neighbor sense from the training set, i.e., the G.4 \u2022 W ORD SENSE DISAMBIGUATION 11\nsense whose sense embedding has the highest cosine with t:\nsense(t) =argmax\ns2senses (t)cosine (t;vs) (G.14)\nFig. G.9 illustrates the model.\nI  found  the  jar  emptycIcfoundfind1vcthecjarcemptyfind9vfind5vfind4vENCODER\nFigure G.9 The nearest-neighbor algorithm for WSD. In green are the contextual embed-\ndings precomputed for each sense of each word; here we just show a few of the senses for\n\ufb01nd. A contextual embedding is computed for the target word found , and then the nearest\nneighbor sense (in this case \ufb01nd9v) is chosen. Figure inspired by Loureiro and Jorge (2019).\nWhat do we do for words we haven\u2019t seen in the sense-labeled training data?\nAfter all, the number of senses that appear in SemCor is only a small fraction of the\nwords in WordNet. The simplest algorithm is to fall back to the Most Frequent Sense\nbaseline, i.e. taking the \ufb01rst sense in WordNet. But that\u2019s not very satisfactory.\nA more powerful approach, due to Loureiro and Jorge (2019), is to impute the\nmissing sense embeddings, bottom-up, by using the WordNet taxonomy and super-\nsenses. We get a sense embedding for any higher-level node in the WordNet taxon-\nomy by averaging the embeddings of its children, thus computing the embedding for\neach synset as the average of its sense embeddings, the embedding for a hypernym\nas the average of its synset embeddings, and the lexicographic category (supersense)\nembedding as the average of the large set of synset embeddings with that category.\nMore formally, for each missing sense in WordNet \u02c6 s2W, let the sense embeddings\nfor the other members of its synset be S\u02c6s, the hypernym-speci\ufb01c synset embeddings\nbeH\u02c6s, and the lexicographic (supersense-speci\ufb01c) synset embeddings be L\u02c6s. We can\nthen compute the sense embedding for \u02c6 sas follows:\nifjS\u02c6sj>0;v\u02c6s=1\njS\u02c6sjX\nvs;8vs2S\u02c6s (G.15)\nelse ifjH\u02c6sj>0;v\u02c6s=1\njH\u02c6sjX\nvsyn;8vsyn2H\u02c6s (G.16)\nelse ifjL\u02c6sj>0;v\u02c6s=1\njL\u02c6sjX\nvsyn;8vsyn2L\u02c6s (G.17)\nSince all of the supersenses have some labeled data in SemCor, the algorithm is\nguaranteed to have some representation for all possible senses by the time the al-\ngorithm backs off to the most general (supersense) information, although of course\nwith a very coarse model. 12 APPENDIX G \u2022 W ORD SENSES AND WORDNET\nG.5 Alternate WSD algorithms and Tasks\nG.5.1 Feature-Based WSD\nFeature-based algorithms for WSD are extremely simple and function almost as\nwell as contextual language model algorithms. The best performing IMS algorithm\n(Zhong and Ng, 2010), augmented by embeddings (Iacobacci et al. 2016, Raganato\net al. 2017b), uses an SVM classi\ufb01er to choose the sense for each input word with\nthe following simple features of the surrounding words:\n\u2022 part-of-speech tags (for a window of 3 words on each side, stopping at sen-\ntence boundaries)\n\u2022collocation features of words or n-grams of lengths 1, 2, 3 at a particular collocation\nlocation in a window of 3 words on each side (i.e., exactly one word to the\nright, or the two words starting 3 words to the left, and so on).\n\u2022 weighted average of embeddings (of all words in a window of 10 words on\neach side, weighted exponentially by distance)\nConsider the ambiguous word bass in the following WSJ sentence:\n(G.18) An electric guitar and bass player stand off to one side,\nIf we used a small 2-word window, a standard feature vector might include parts-of-\nspeech, unigram and bigram collocation features, and a weighted sum gof embed-\ndings, that is:\n[wi\u00002;POS i\u00002;wi\u00001;POS i\u00001;wi+1;POS i+1;wi+2;POS i+2;wi\u00001\ni\u00002;\nwi+2\ni+1;g(E(wi\u00002);E(wi\u00001);E(wi+1);E(wi+2)] (G.19)\nwould yield the following vector:\n[guitar, NN, and, CC, player, NN, stand, VB, guitar and,\nplayer stand, g(E(guitar),E(and),E(player),E(stand))]\nG.5.2 The Lesk Algorithm as WSD Baseline\nGenerating sense labeled corpora like SemCor is quite dif\ufb01cult and expensive. An\nalternative class of WSD algorithms, knowledge-based algorithms, rely solely onknowledge-\nbased\nWordNet or other such resources and don\u2019t require labeled data. While supervised\nalgorithms generally work better, knowledge-based methods can be used in lan-\nguages or domains where thesauruses or dictionaries but not sense labeled corpora\nare available.\nThe Lesk algorithm is the oldest and most powerful knowledge-based WSD Lesk algorithm\nmethod, and is a useful baseline. Lesk is really a family of algorithms that choose\nthe sense whose dictionary gloss or de\ufb01nition shares the most words with the target\nword\u2019s neighborhood. Figure G.10 shows the simplest version of the algorithm,\noften called the Simpli\ufb01ed Lesk algorithm (Kilgarriff and Rosenzweig, 2000). Simpli\ufb01ed Lesk\nAs an example of the Lesk algorithm at work, consider disambiguating the word\nbank in the following context:\n(G.20) The bank can guarantee deposits will eventually cover future tuition costs\nbecause it invests in adjustable-rate mortgage securities.\ngiven the following two WordNet senses: G.5 \u2022 A LTERNATE WSD ALGORITHMS AND TASKS 13\nfunction SIMPLIFIED LESK(word, sentence )returns best sense of word\nbest-sense most frequent sense for word\nmax-overlap 0\ncontext set of words in sentence\nfor each sense insenses of word do\nsignature set of words in the gloss and examples of sense\noverlap COMPUTE OVERLAP (signature ,context )\nifoverlap >max-overlap then\nmax-overlap overlap\nbest-sense sense\nend\nreturn (best-sense )\nFigure G.10 The Simpli\ufb01ed Lesk algorithm. The C OMPUTE OVERLAP function returns\nthe number of words in common between two sets, ignoring function words or other words\non a stop list. The original Lesk algorithm de\ufb01nes the context in a more complex way.\nbank1Gloss: a \ufb01nancial institution that accepts deposits and channels the\nmoney into lending activities\nExamples: \u201che cashed a check at the bank\u201d, \u201cthat bank holds the mortgage\non my home\u201d\nbank2Gloss: sloping land (especially the slope beside a body of water)\nExamples: \u201cthey pulled the canoe up on the bank\u201d, \u201che sat on the bank of\nthe river and watched the currents\u201d\nSense bank1has two non-stopwords overlapping with the context in (G.20):\ndeposits andmortgage , while sense bank2has zero words, so sense bank1is chosen.\nThere are many obvious extensions to Simpli\ufb01ed Lesk, such as weighing the\noverlapping words by IDF (inverse document frequency) (Chapter 6) to downweight\nfrequent words like function words; best performing is to use word embedding co-\nsine instead of word overlap to compute the similarity between the de\ufb01nition and the\ncontext (Basile et al., 2014). Modern neural extensions of Lesk use the de\ufb01nitions\nto compute sense embeddings that can be directly used instead of SemCor-training\nembeddings (Kumar et al. 2019, Luo et al. 2018a, Luo et al. 2018b).\nG.5.3 Word-in-Context Evaluation\nWord Sense Disambiguation is a much more \ufb01ne-grained evaluation of word mean-\ning than the context-free word similarity tasks we described in Chapter 6. Recall that\ntasks like LexSim-999 require systems to match human judgments on the context-\nfree similarity between two words (how similar is cuptomug?). We can think of\nWSD as a kind of contextualized similarity task, since our goal is to be able to distin-\nguish the meaning of a word like bass in one context (playing music) from another\ncontext (\ufb01shing).\nSomewhere in between lies the word-in-context task. Here the system is given word-in-context\ntwo sentences, each with the same target word but in a different sentential context.\nThe system must decide whether the target words are used in the same sense in the\ntwo sentences or in a different sense . Fig. G.11 shows sample pairs from the WiC WiC\ndataset of Pilehvar and Camacho-Collados (2019).\nThe WiC sentences are mainly taken from the example usages for senses in\nWordNet. But WordNet senses are very \ufb01ne-grained. For this reason tasks like 14 APPENDIX G \u2022 W ORD SENSES AND WORDNET\nF There\u2019s a lot of trash on the bedof the river \u2014\nI keep a glass of water next to my bedwhen I sleep\nFJustify the margins \u2014 The end justi\ufb01es the means\nTAirpollution \u2014 Open a window and let in some air\nT The expanded window will give us time to catch the thieves \u2014\nYou have a two-hour window of clear weather to \ufb01nish working on the lawn\nFigure G.11 Positive (T) and negative (F) pairs from the WiC dataset (Pilehvar and\nCamacho-Collados, 2019).\nword-in-context \ufb01rst cluster the word senses into coarser clusters, so that the two\nsentential contexts for the target word are marked as T if the two senses are in the\nsame cluster. WiC clusters all pairs of senses if they are \ufb01rst degree connections in\nthe WordNet semantic graph, including sister senses, or if they belong to the same\nsupersense; we point to other sense clustering algorithms at the end of the chapter.\nThe baseline algorithm to solve the WiC task uses contextual embeddings like\nBERT with a simple thresholded cosine. We \ufb01rst compute the contextual embed-\ndings for the target word in each of the two sentences, and then compute the cosine\nbetween them. If it\u2019s above a threshold tuned on a devset we respond true (the two\nsenses are the same) else we respond false.\nG.5.4 Wikipedia as a source of training data\nDatasets other than SemCor have been used for all-words WSD. One important di-\nrection is to use Wikipedia as a source of sense-labeled data. When a concept is\nmentioned in a Wikipedia article, the article text may contain an explicit link to the\nconcept\u2019s Wikipedia page, which is named by a unique identi\ufb01er. This link can be\nused as a sense annotation. For example, the ambiguous word baris linked to a\ndifferent Wikipedia article depending on its meaning in context, including the page\nBAR(LAW), the page B AR(MUSIC ), and so on, as in the following Wikipedia\nexamples (Mihalcea, 2007).\nIn 1834, Sumner was admitted to the [[bar (law) jbar]] at the age of\ntwenty-three, and entered private practice in Boston.\nIt is danced in 3/4 time (like most waltzes), with the couple turning\napprox. 180 degrees every [[bar (music)jbar]] .\nJenga is a popular beer in the [[bar (establishment) jbar]] s of Thailand.\nThese sentences can then be added to the training data for a supervised system.\nIn order to use Wikipedia in this way, however, it is necessary to map from Wiki-\npedia concepts to whatever inventory of senses is relevant for the WSD application.\nAutomatic algorithms that map from Wikipedia to WordNet, for example, involve\n\ufb01nding the WordNet sense that has the greatest lexical overlap with the Wikipedia\nsense, by comparing the vector of words in the WordNet synset, gloss, and related\nsenses with the vector of words in the Wikipedia page title, outgoing links, and page\ncategory (Ponzetto and Navigli, 2010). The resulting mapping has been used to\ncreate BabelNet, a large sense-annotated resource (Navigli and Ponzetto, 2012). G.6 \u2022 U SING THESAURUSES TO IMPROVE EMBEDDINGS 15\nG.6 Using Thesauruses to Improve Embeddings\nThesauruses have also been used to improve both static and contextual word em-\nbeddings. For example, static word embeddings have a problem with antonyms .\nA word like expensive is often very similar in embedding cosine to its antonym\nlikecheap . Antonymy information from thesauruses can help solve this problem;\nFig. G.12 shows nearest neighbors to some target words in GloVe, and the improve-\nment after one such method.\nBefore counter\ufb01tting After counter\ufb01tting\neast west north south eastward eastern easterly\nexpensive pricey cheaper costly costly pricy overpriced\nBritish American Australian Britain Brits London BBC\nFigure G.12 The nearest neighbors in GloVe to east,expensive , and British include\nantonyms like west. The right side showing the improvement in GloVe nearest neighbors\nafter the counter\ufb01tting method (Mrk \u02c7si\u00b4c et al., 2016).\nThere are two families of solutions. The \ufb01rst requires retraining: we modify the\nembedding training to incorporate thesaurus relations like synonymy, antonym, or\nsupersenses. This can be done by modifying the static embedding loss function for\nword2vec (Yu and Dredze 2014, Nguyen et al. 2016) or by modifying contextual\nembedding training (Levine et al. 2020, Lauscher et al. 2019).\nThe second, for static embeddings, is more light-weight; after the embeddings\nhave been trained we learn a second mapping based on a thesaurus that shifts the\nembeddings of words in such a way that synonyms (according to the thesaurus) are\npushed closer and antonyms further apart. Such methods are called retro\ufb01tting retro\ufb01tting\n(Faruqui et al. 2015, Lengerich et al. 2018) or counter\ufb01tting (Mrk \u02c7si\u00b4c et al., 2016).\nG.7 Word Sense Induction\nIt is expensive and dif\ufb01cult to build large corpora in which each word is labeled for\nits word sense. For this reason, an unsupervised approach to sense disambiguation,\noften called word sense induction orWSI , is an important direction. In unsu-word sense\ninduction\npervised approaches, we don\u2019t use human-de\ufb01ned word senses. Instead, the set of\n\u201csenses\u201d of each word is created automatically from the instances of each word in\nthe training set.\nMost algorithms for word sense induction follow the early work of Sch \u00a8utze\n(Sch \u00a8utze 1992, Sch \u00a8utze 1998) in using some sort of clustering over word embed-\ndings. In training, we use three steps:\n1. For each token wiof word win a corpus, compute a context vector c.\n2. Use a clustering algorithm tocluster these word-token context vectors cinto\na prede\ufb01ned number of groups or clusters. Each cluster de\ufb01nes a sense of w.\n3. Compute the vector centroid of each cluster. Each vector centroid sjis a\nsense vector representing that sense of w.\nSince this is an unsupervised algorithm, we don\u2019t have names for each of these\n\u201csenses\u201d of w; we just refer to the jth sense of w.\nTo disambiguate a particular token tofwwe again have three steps: 16 APPENDIX G \u2022 W ORD SENSES AND WORDNET\n1. Compute a context vector cfort.\n2. Retrieve all sense vectors sjforw.\n3. Assign tto the sense represented by the sense vector sjthat is closest to t.\nAll we need is a clustering algorithm and a distance metric between vectors.\nClustering is a well-studied problem with a wide number of standard algorithms that\ncan be applied to inputs structured as vectors of numerical values (Duda and Hart,\n1973). A frequently used technique in language applications is known as agglom-\nerative clustering . In this technique, each of the Ntraining instances is initiallyagglomerative\nclustering\nassigned to its own cluster. New clusters are then formed in a bottom-up fashion by\nthe successive merging of the two clusters that are most similar. This process con-\ntinues until either a speci\ufb01ed number of clusters is reached, or some global goodness\nmeasure among the clusters is achieved. In cases in which the number of training\ninstances makes this method too expensive, random sampling can be used on the\noriginal training set to achieve similar results.\nHow can we evaluate unsupervised sense disambiguation approaches? As usual,\nthe best way is to do extrinsic evaluation embedded in some end-to-end system; one\nexample used in a SemEval bakeoff is to improve search result clustering and di-\nversi\ufb01cation (Navigli and Vannella, 2013). Intrinsic evaluation requires a way to\nmap the automatically derived sense classes into a hand-labeled gold-standard set so\nthat we can compare a hand-labeled test set with a set labeled by our unsupervised\nclassi\ufb01er. Various such metrics have been tested, for example in the SemEval tasks\n(Manandhar et al. 2010, Navigli and Vannella 2013, Jurgens and Klapaftis 2013),\nincluding cluster overlap metrics, or methods that map each sense cluster to a pre-\nde\ufb01ned sense by choosing the sense that (in some training set) has the most overlap\nwith the cluster. However it is fair to say that no evaluation metric for this task has\nyet become standard.\nG.8 Summary\nThis chapter has covered a wide range of issues concerning the meanings associated\nwith lexical items. The following are among the highlights:\n\u2022 Aword sense is the locus of word meaning; de\ufb01nitions and meaning relations\nare de\ufb01ned at the level of the word sense rather than wordforms.\n\u2022 Many words are polysemous , having many senses.\n\u2022 Relations between senses include synonymy ,antonymy ,meronymy , and\ntaxonomic relations hyponymy andhypernymy .\n\u2022WordNet is a large database of lexical relations for English, and WordNets\nexist for a variety of languages.\n\u2022Word-sense disambiguation (WSD ) is the task of determining the correct\nsense of a word in context. Supervised approaches make use of a corpus\nof sentences in which individual words ( lexical sample task ) or all words\n(all-words task ) are hand-labeled with senses from a resource like WordNet.\nSemCor is the largest corpus with WordNet-labeled senses.\n\u2022 The standard supervised algorithm for WSD is nearest neighbors with contex-\ntual embeddings.\n\u2022 Feature-based algorithms using parts of speech and embeddings of words in\nthe context of the target word also work well. BIBLIOGRAPHICAL AND HISTORICAL NOTES 17\n\u2022 An important baseline for WSD is the most frequent sense , equivalent, in\nWordNet, to take the \ufb01rst sense .\n\u2022 Another baseline is a knowledge-based WSD algorithm called the Lesk al-\ngorithm which chooses the sense whose dictionary de\ufb01nition shares the most\nwords with the target word\u2019s neighborhood.\n\u2022Word sense induction is the task of learning word senses unsupervised.\nBibliographical and Historical Notes\nWord sense disambiguation traces its roots to some of the earliest applications of\ndigital computers. The insight that underlies modern algorithms for word sense dis-\nambiguation was \ufb01rst articulated by Weaver (1949/1955) in the context of machine\ntranslation:\nIf one examines the words in a book, one at a time as through an opaque\nmask with a hole in it one word wide, then it is obviously impossible\nto determine, one at a time, the meaning of the words. [. . . ] But if\none lengthens the slit in the opaque mask, until one can see not only\nthe central word in question but also say N words on either side, then\nif N is large enough one can unambiguously decide the meaning of the\ncentral word. [. . . ] The practical question is : \u201cWhat minimum value of\nN will, at least in a tolerable fraction of cases, lead to the correct choice\nof meaning for the central word?\u201d\nOther notions \ufb01rst proposed in this early period include the use of a thesaurus for dis-\nambiguation (Masterman, 1957), supervised training of Bayesian models for disam-\nbiguation (Madhu and Lytel, 1965), and the use of clustering in word sense analysis\n(Sparck Jones, 1986).\nMuch disambiguation work was conducted within the context of early AI-oriented\nnatural language processing systems. Quillian (1968) and Quillian (1969) proposed\na graph-based approach to language processing, in which the de\ufb01nition of a word\nwas represented by a network of word nodes connected by syntactic and semantic\nrelations, and sense disambiguation by \ufb01nding the shortest path between senses in\nthe graph. Simmons (1973) is another in\ufb02uential early semantic network approach.\nWilks proposed one of the earliest non-discrete models with his Preference Seman-\ntics(Wilks 1975c, Wilks 1975b, Wilks 1975a), and Small and Rieger (1982) and\nRiesbeck (1975) proposed understanding systems based on modeling rich procedu-\nral information for each word. Hirst\u2019s ABSITY system (Hirst and Charniak 1982,\nHirst 1987, Hirst 1988), which used a technique called marker passing based on se-\nmantic networks, represents the most advanced system of this type. As with these\nlargely symbolic approaches, early neural network (at the time called \u2018connection-\nist\u2019) approaches to word sense disambiguation relied on small lexicons with hand-\ncoded representations (Cottrell 1985, Kawamoto 1988).\nThe earliest implementation of a robust empirical approach to sense disambigua-\ntion is due to Kelly and Stone (1975), who directed a team that hand-crafted a set\nof disambiguation rules for 1790 ambiguous English words. Lesk (1986) was the\n\ufb01rst to use a machine-readable dictionary for word sense disambiguation. Fellbaum\n(1998) collects early work on WordNet. Early work using dictionaries as lexical 18 APPENDIX G \u2022 W ORD SENSES AND WORDNET\nresources include Amsler\u2019s 1981 use of the Merriam Webster dictionary and Long-\nman\u2019s Dictionary of Contemporary English (Boguraev and Briscoe, 1989).\nSupervised approaches to disambiguation began with the use of decision trees\nby Black (1988). In addition to the IMS and contextual-embedding based methods\nfor supervised WSD, recent supervised algorithms includes encoder-decoder models\n(Raganato et al., 2017a).\nThe need for large amounts of annotated text in supervised methods led early\non to investigations into the use of bootstrapping methods (Hearst 1991, Yarowsky\n1995). For example the semi-supervised algorithm of Diab and Resnik (2002) is\nbased on aligned parallel corpora in two languages. For example, the fact that the\nFrench word catastrophe might be translated as English disaster in one instance\nandtragedy in another instance can be used to disambiguate the senses of the two\nEnglish words (i.e., to choose senses of disaster andtragedy that are similar).\nThe earliest use of clustering in the study of word senses was by Sparck Jones\n(1986); Pedersen and Bruce (1997), Sch \u00a8utze (1997), and Sch \u00a8utze (1998) applied dis-\ntributional methods. Clustering word senses into coarse senses has also been used coarse senses\nto address the problem of dictionary senses being too \ufb01ne-grained (Section G.5.3)\n(Dolan 1994, Chen and Chang 1998, Mihalcea and Moldovan 2001, Agirre and\nde Lacalle 2003, Palmer et al. 2004, Navigli 2006, Snow et al. 2007, Pilehvar et al.\n2013). Corpora with clustered word senses for training supervised clustering algo-\nrithms include Palmer et al. (2006) and OntoNotes (Hovy et al., 2006). OntoNotes\nSee Pustejovsky (1995), Pustejovsky and Boguraev (1996), Martin (1986), and\nCopestake and Briscoe (1995), inter alia, for computational approaches to the rep-\nresentation of polysemy. Pustejovsky\u2019s theory of the generative lexicon , and ingenerative\nlexicon\nparticular his theory of the qualia structure of words, is a way of accounting for thequalia\nstructure\ndynamic systematic polysemy of words in context.\nHistorical overviews of WSD include Agirre and Edmonds (2006) and Navigli\n(2009).\nExercises\nG.1 Collect a small corpus of example sentences of varying lengths from any\nnewspaper or magazine. Using WordNet or any standard dictionary, deter-\nmine how many senses there are for each of the open-class words in each sen-\ntence. How many distinct combinations of senses are there for each sentence?\nHow does this number seem to vary with sentence length?\nG.2 Using WordNet or a standard reference dictionary, tag each open-class word\nin your corpus with its correct tag. Was choosing the correct sense always a\nstraightforward task? Report on any dif\ufb01culties you encountered.\nG.3 Using your favorite dictionary, simulate the original Lesk word overlap dis-\nambiguation algorithm described on page 13 on the phrase Time \ufb02ies like an\narrow . Assume that the words are to be disambiguated one at a time, from\nleft to right, and that the results from earlier decisions are used later in the\nprocess.\nG.4 Build an implementation of your solution to the previous exercise. Using\nWordNet, implement the original Lesk word overlap disambiguation algo-\nrithm described on page 13 on the phrase Time \ufb02ies like an arrow . Exercises 19\nAgirre, E. and O. L. de Lacalle. 2003. Clustering WordNet\nword senses. RANLP 2003 .\nAgirre, E. and P. Edmonds, eds. 2006. Word Sense Disam-\nbiguation: Algorithms and Applications . Kluwer.\nAmsler, R. A. 1981. A taxonomy for English nouns and\nverbs. ACL.\nBasile, P., A. Caputo, and G. Semeraro. 2014. An enhanced\nLesk word sense disambiguation algorithm through a dis-\ntributional semantic model. COLING .\nBlack, E. 1988. An experiment in computational discrimi-\nnation of English word senses. IBM Journal of Research\nand Development , 32(2):185\u2013194.\nBoguraev, B. K. and T. Briscoe, eds. 1989. Computational\nLexicography for Natural Language Processing . Long-\nman.\nChaplot, D. S. and R. Salakhutdinov. 2018. Knowledge-\nbased word sense disambiguation using topic models.\nAAAI .\nChen, J. N. and J. S. Chang. 1998. Topical clustering of\nMRD senses based on information retrieval techniques.\nComputational Linguistics , 24(1):61\u201396.\nCiaramita, M. and Y . Altun. 2006. Broad-coverage sense\ndisambiguation and information extraction with a super-\nsense sequence tagger. EMNLP .\nCiaramita, M. and M. Johnson. 2003. Supersense tagging of\nunknown nouns in WordNet. EMNLP-2003 .\nCopestake, A. and T. Briscoe. 1995. Semi-productive\npolysemy and sense extension. Journal of Semantics ,\n12(1):15\u201368.\nCottrell, G. W. 1985. A Connectionist Approach to Word\nSense Disambiguation . Ph.D. thesis, University of\nRochester, Rochester, NY . Revised version published by\nPitman, 1989.\nDiab, M. and P. Resnik. 2002. An unsupervised method for\nword sense tagging using parallel corpora. ACL.\nDolan, B. 1994. Word sense ambiguation: Clustering related\nsenses. COLING .\nDuda, R. O. and P. E. Hart. 1973. Pattern Classi\ufb01cation and\nScene Analysis . John Wiley and Sons.\nFaruqui, M., J. Dodge, S. K. Jauhar, C. Dyer, E. Hovy, and\nN. A. Smith. 2015. Retro\ufb01tting word vectors to semantic\nlexicons. NAACL HLT .\nFellbaum, C., ed. 1998. WordNet: An Electronic Lexical\nDatabase . MIT Press.\nGale, W. A., K. W. Church, and D. Yarowsky. 1992a. Es-\ntimating upper and lower bounds on the performance of\nword-sense disambiguation programs. ACL.\nGale, W. A., K. W. Church, and D. Yarowsky. 1992b. One\nsense per discourse. HLT.\nHaber, J. and M. Poesio. 2020. Assessing polyseme sense\nsimilarity through co-predication acceptability and con-\ntextualised embedding distance. *SEM .\nHearst, M. A. 1991. Noun homograph disambiguation. Pro-\nceedings of the 7th Conference of the University of Wa-\nterloo Centre for the New OED and Text Research .\nHenrich, V ., E. Hinrichs, and T. V odolazova. 2012. We-\nbCAGe \u2013 a web-harvested corpus annotated with Ger-\nmaNet senses. EACL .Hirst, G. 1987. Semantic Interpretation and the Resolution\nof Ambiguity . Cambridge University Press.\nHirst, G. 1988. Resolving lexical ambiguity computationally\nwith spreading activation and polaroid words. In S. L.\nSmall, G. W. Cottrell, and M. K. Tanenhaus, eds, Lexical\nAmbiguity Resolution , 73\u2013108. Morgan Kaufmann.\nHirst, G. and E. Charniak. 1982. Word sense and case slot\ndisambiguation. AAAI .\nHovy, E. H., M. P. Marcus, M. Palmer, L. A. Ramshaw,\nand R. Weischedel. 2006. OntoNotes: The 90% solution.\nHLT-NAACL .\nIacobacci, I., M. T. Pilehvar, and R. Navigli. 2016. Em-\nbeddings for word sense disambiguation: An evaluation\nstudy. ACL.\nJurgens, D. and I. P. Klapaftis. 2013. SemEval-2013 task 13:\nWord sense induction for graded and non-graded senses.\n*SEM .\nKawamoto, A. H. 1988. Distributed representations of am-\nbiguous words and their resolution in connectionist net-\nworks. In S. L. Small, G. W. Cottrell, and M. Tanen-\nhaus, eds, Lexical Ambiguity Resolution , 195\u2013228. Mor-\ngan Kaufman.\nKelly, E. F. and P. J. Stone. 1975. Computer Recognition of\nEnglish Word Senses . North-Holland.\nKilgarriff, A. and J. Rosenzweig. 2000. Framework and re-\nsults for English SENSEV AL. Computers and the Hu-\nmanities , 34:15\u201348.\nKumar, S., S. Jat, K. Saxena, and P. Talukdar. 2019. Zero-\nshot word sense disambiguation using sense de\ufb01nition\nembeddings. ACL.\nLandes, S., C. Leacock, and R. I. Tengi. 1998. Building se-\nmantic concordances. In C. Fellbaum, ed., WordNet: An\nElectronic Lexical Database , 199\u2013216. MIT Press.\nLauscher, A., I. Vuli \u00b4c, E. M. Ponti, A. Korhonen, and\nG. Glava \u02c7s. 2019. Informing unsupervised pretraining\nwith external linguistic knowledge. ArXiv preprint\narXiv:1909.02339.\nLengerich, B., A. Maas, and C. Potts. 2018. Retro\ufb01tting dis-\ntributional embeddings to knowledge graphs with func-\ntional relations. COLING .\nLesk, M. E. 1986. Automatic sense disambiguation using\nmachine readable dictionaries: How to tell a pine cone\nfrom an ice cream cone. Proceedings of the 5th Interna-\ntional Conference on Systems Documentation .\nLevine, Y ., B. Lenz, O. Dagan, O. Ram, D. Pad-\nnos, O. Sharir, S. Shalev-Shwartz, A. Shashua, and\nY . Shoham. 2020. SenseBERT: Driving some sense into\nBERT. ACL.\nLoureiro, D. and A. Jorge. 2019. Language modelling makes\nsense: Propagating representations through WordNet for\nfull-coverage word sense disambiguation. ACL.\nLuo, F., T. Liu, Z. He, Q. Xia, Z. Sui, and B. Chang. 2018a.\nLeveraging gloss knowledge in neural word sense disam-\nbiguation by hierarchical co-attention. EMNLP .\nLuo, F., T. Liu, Q. Xia, B. Chang, and Z. Sui. 2018b. Incor-\nporating glosses into neural word sense disambiguation.\nACL.\nMadhu, S. and D. Lytel. 1965. A \ufb01gure of merit technique for\nthe resolution of non-grammatical ambiguity. Mechanical\nTranslation , 8(2):9\u201313. 20 Appendix G \u2022 Word Senses and WordNet\nManandhar, S., I. P. Klapaftis, D. Dligach, and S. Pradhan.\n2010. SemEval-2010 task 14: Word sense induction &\ndisambiguation. SemEval .\nMartin, J. H. 1986. The acquisition of polysemy. ICML .\nMasterman, M. 1957. The thesaurus in syntax and semantics.\nMechanical Translation , 4(1):1\u20132.\nMelamud, O., J. Goldberger, and I. Dagan. 2016. con-\ntext2vec: Learning generic context embedding with bidi-\nrectional LSTM. CoNLL .\nMihalcea, R. 2007. Using Wikipedia for automatic word\nsense disambiguation. NAACL-HLT .\nMihalcea, R. and D. Moldovan. 2001. Automatic genera-\ntion of a coarse grained WordNet. NAACL Workshop on\nWordNet and Other Lexical Resources .\nMiller, G. A., C. Leacock, R. I. Tengi, and R. T. Bunker.\n1993. A semantic concordance. HLT.\nMorris, W., ed. 1985. American Heritage Dictionary , 2nd\ncollege edition edition. Houghton Mif\ufb02in.\nMrk\u02c7si\u00b4c, N., D. \u00b4O. S \u00b4eaghdha, B. Thomson, M. Ga \u02c7si\u00b4c, L. M.\nRojas-Barahona, P.-H. Su, D. Vandyke, T.-H. Wen, and\nS. Young. 2016. Counter-\ufb01tting word vectors to linguis-\ntic constraints. NAACL HLT .\nNavigli, R. 2006. Meaningful clustering of senses helps\nboost word sense disambiguation performance. COL-\nING/ACL .\nNavigli, R. 2009. Word sense disambiguation: A survey.\nACM Computing Surveys , 41(2).\nNavigli, R. 2016. Chapter 20. ontologies. In R. Mitkov, ed.,\nThe Oxford handbook of computational linguistics . Ox-\nford University Press.\nNavigli, R. and S. P. Ponzetto. 2012. BabelNet: The auto-\nmatic construction, evaluation and application of a wide-\ncoverage multilingual semantic network. Arti\ufb01cial Intel-\nligence , 193:217\u2013250.\nNavigli, R. and D. Vannella. 2013. SemEval-2013 task 11:\nWord sense induction and disambiguation within an end-\nuser application. *SEM .\nNguyen, K. A., S. Schulte im Walde, and N. T. Vu. 2016.\nIntegrating distributional lexical contrast into word em-\nbeddings for antonym-synonym distinction. ACL.\nPalmer, M., O. Babko-Malaya, and H. T. Dang. 2004. Dif-\nferent sense granularities for different applications. HLT-\nNAACL Workshop on Scalable Natural Language Under-\nstanding .\nPalmer, M., H. T. Dang, and C. Fellbaum. 2006. Making\n\ufb01ne-grained and coarse-grained sense distinctions, both\nmanually and automatically. Natural Language Engineer-\ning, 13(2):137\u2013163.\nPedersen, T. and R. Bruce. 1997. Distinguishing word senses\nin untagged text. EMNLP .\nPeters, M., M. Neumann, M. Iyyer, M. Gardner, C. Clark,\nK. Lee, and L. Zettlemoyer. 2018. Deep contextualized\nword representations. NAACL HLT .\nPilehvar, M. T. and J. Camacho-Collados. 2019. WiC: the\nword-in-context dataset for evaluating context-sensitive\nmeaning representations. NAACL HLT .\nPilehvar, M. T., D. Jurgens, and R. Navigli. 2013. Align,\ndisambiguate and walk: A uni\ufb01ed approach for measur-\ning semantic similarity. ACL.Ponzetto, S. P. and R. Navigli. 2010. Knowledge-rich word\nsense disambiguation rivaling supervised systems. ACL.\nPu, X., N. Pappas, J. Henderson, and A. Popescu-Belis.\n2018. Integrating weakly supervised word sense disam-\nbiguation into neural machine translation. TACL , 6:635\u2013\n649.\nPustejovsky, J. 1995. The Generative Lexicon . MIT Press.\nPustejovsky, J. and B. K. Boguraev, eds. 1996. Lexical Se-\nmantics: The Problem of Polysemy . Oxford University\nPress.\nQuillian, M. R. 1968. Semantic memory. In M. Minsky, ed.,\nSemantic Information Processing , 227\u2013270. MIT Press.\nQuillian, M. R. 1969. The teachable language compre-\nhender: A simulation program and theory of language.\nCACM , 12(8):459\u2013476.\nRaganato, A., C. D. Bovi, and R. Navigli. 2017a. Neural se-\nquence learning models for word sense disambiguation.\nEMNLP .\nRaganato, A., J. Camacho-Collados, and R. Navigli. 2017b.\nWord sense disambiguation: A uni\ufb01ed evaluation frame-\nwork and empirical comparison. EACL .\nRiesbeck, C. K. 1975. Conceptual analysis. In R. C. Schank,\ned.,Conceptual Information Processing , 83\u2013156. Ameri-\ncan Elsevier, New York.\nSchneider, N., J. D. Hwang, V . Srikumar, J. Prange, A. Blod-\ngett, S. R. Moeller, A. Stern, A. Bitan, and O. Abend.\n2018. Comprehensive supersense disambiguation of En-\nglish prepositions and possessives. ACL.\nSch\u00a8utze, H. 1992. Dimensions of meaning. Proceedings of\nSupercomputing \u201992 . IEEE Press.\nSch\u00a8utze, H. 1997. Ambiguity Resolution in Language Learn-\ning: Computational and Cognitive Models . CSLI Publi-\ncations, Stanford, CA.\nSch\u00a8utze, H. 1998. Automatic word sense discrimination.\nComputational Linguistics , 24(1):97\u2013124.\nSimmons, R. F. 1973. Semantic networks: Their compu-\ntation and use for understanding English sentences. In\nR. C. Schank and K. M. Colby, eds, Computer Models of\nThought and Language , 61\u2013113. W.H. Freeman & Co.\nSmall, S. L. and C. Rieger. 1982. Parsing and comprehend-\ning with Word Experts. In W. G. Lehnert and M. H.\nRingle, eds, Strategies for Natural Language Processing ,\n89\u2013147. Lawrence Erlbaum.\nSnow, R., S. Prakash, D. Jurafsky, and A. Y . Ng. 2007.\nLearning to merge word senses. EMNLP/CoNLL .\nSnyder, B. and M. Palmer. 2004. The English all-words task.\nSENSEVAL-3 .\nSparck Jones, K. 1986. Synonymy and Semantic Classi\ufb01ca-\ntion. Edinburgh University Press, Edinburgh. Republica-\ntion of 1964 PhD Thesis.\nTsvetkov, Y ., N. Schneider, D. Hovy, A. Bhatia, M. Faruqui,\nand C. Dyer. 2014. Augmenting English adjective senses\nwith supersenses. LREC .\nV ossen, P., A. G \u00a8or\u00a8og, F. Laan, M. Van Gompel, R. Izquierdo,\nand A. Van Den Bosch. 2011. Dutch-semcor: building a\nsemantically annotated corpus for dutch. Proceedings of\neLex. Exercises 21\nWeaver, W. 1949/1955. Translation. In W. N. Locke and\nA. D. Boothe, eds, Machine Translation of Languages ,\n15\u201323. MIT Press. Reprinted from a memorandum writ-\nten by Weaver in 1949.\nWilks, Y . 1975a. An intelligent analyzer and understander of\nEnglish. CACM , 18(5):264\u2013274.\nWilks, Y . 1975b. Preference semantics. In E. L. Keenan,\ned.,The Formal Semantics of Natural Language , 329\u2013\n350. Cambridge Univ. Press.\nWilks, Y . 1975c. A preferential, pattern-seeking, seman-\ntics for natural language inference. Arti\ufb01cial Intelligence ,\n6(1):53\u201374.\nYarowsky, D. 1995. Unsupervised word sense disambigua-\ntion rivaling supervised methods. ACL.\nYu, M. and M. Dredze. 2014. Improving lexical embeddings\nwith semantic knowledge. ACL.\nZhong, Z. and H. T. Ng. 2010. It makes sense: A wide-\ncoverage word sense disambiguation system for free text.\nACL.", "H.pdf": "Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright \u00a92024. All\nrights reserved. Draft of August 20, 2024.\nCHAPTER\nHPhonetics\nThe characters that make up the texts we\u2019ve been discussing in this book aren\u2019t just\nrandom symbols. They are also an amazing scienti\ufb01c invention: a theoretical model\nof the elements that make up human speech.\nThe earliest writing systems we know of (Sumerian, Chinese, Mayan) were\nmainly logographic : one symbol representing a whole word. But from the ear-\nliest stages we can \ufb01nd, some symbols were also used to represent the sounds\nthat made up words. The cuneiform sign to the right pro-\nnounced baand meaning \u201cration\u201d in Sumerian could also\nfunction purely as the sound /ba/. The earliest Chinese char-\nacters we have, carved into bones for divination, similarly\ncontain phonetic elements. Purely sound-based writing systems, whether syllabic\n(like Japanese hiragana ), alphabetic (like the Roman alphabet), or consonantal (like\nSemitic writing systems), trace back to these early logo-syllabic systems, often as\ntwo cultures came together. Thus, the Arabic, Aramaic, Hebrew, Greek, and Roman\nsystems all derive from a West Semitic script that is presumed to have been modi\ufb01ed\nby Western Semitic mercenaries from a cursive form of Egyptian hieroglyphs. The\nJapanese syllabaries were modi\ufb01ed from a cursive form of Chinese phonetic charac-\nters, which themselves were used in Chinese to phonetically represent the Sanskrit\nin the Buddhist scriptures that came to China in the Tang dynasty.\nThis implicit idea that the spoken word is composed of smaller units of speech\nunderlies algorithms for both speech recognition (transcribing waveforms into text)\nandtext-to-speech (converting text into waveforms). In this chapter we give a com-\nputational perspective on phonetics , the study of the speech sounds used in the phonetics\nlanguages of the world, how they are produced in the human vocal tract, how they\nare realized acoustically, and how they can be digitized and processed.\nH.1 Speech Sounds and Phonetic Transcription\nA letter like \u2018p\u2019 or \u2018a\u2019 is already a useful model of the sounds of human speech,\nand indeed we\u2019ll see in Chapter 16 how to map between letters and waveforms.\nNonetheless, it is helpful to represent sounds slightly more abstractly. We\u2019ll repre-\nsent the pronunciation of a word as a string of phones , which are speech sounds, phone\neach represented with symbols adapted from the Roman alphabet.\nThe standard phonetic representation for transcribing the world\u2019s languages is\ntheInternational Phonetic Alphabet (IPA), an evolving standard \ufb01rst developed in IPA\n1888, But in this chapter we\u2019ll instead represent phones with the ARPAbet (Shoup,\n1980), a simple phonetic alphabet (Fig. H.1) that conveniently uses ASCII symbols\nto represent an American-English subset of the IPA.\nMany of the IPA and ARPAbet symbols are equivalent to familiar Roman let-\nters. So, for example, the ARPAbet phone [p]represents the consonant sound at the 2APPENDIX H \u2022 P HONETICS\nARPAbet IPA ARPAbet ARPAbet IPA ARPAbet\nSymbol Symbol Word Transcription Symbol Symbol Word Transcription\n[p] [p] p arsley [p aa r s l iy] [iy] [i] lily [l ih l iy]\n[t] [t] t ea [t iy] [ih] [I] lily [l ih l iy]\n[k] [k] c ook [k uh k] [ey] [eI] daisy [d ey z iy]\n[b] [b] b ay [b ey] [eh] [E] pen [p eh n]\n[d] [d] d ill [d ih l] [ae] [\u00e6] aster [ae s t axr]\n[g] [g] g arlic [g aa r l ix k] [aa] [A] poppy [p aa p iy]\n[m] [m] m int [m ih n t] [ao] [O] orchid [ao r k ix d]\n[n] [n] n utmeg [n ah t m eh g] [uh] [U] wood [w uh d]\n[ng] [ N] baking [b ey k ix ng] [ow] [oU] lotus [l ow dx ax s]\n[f] [f] f lour [f l aw axr] [uw] [u] tulip [t uw l ix p]\n[v] [v] clov e [k l ow v] [ah] [2] butter [b ah dx axr]\n[th] [ T] th ick [th ih k] [er] [\u00c7] bird [b er d]\n[dh] [ D] th ose [dh ow z] [ay] [aI] iris [ay r ix s]\n[s] [s] s oup [s uw p] [aw] [aU] \ufb02ower [f l aw axr]\n[z] [z] eggs [eh g z] [oy] [oI] soil [s oy l]\n[sh] [ S] squash [s k w aa sh] [ax] [@] pita [p iy t ax]\n[zh] [ Z] ambros ia [ae m b r ow zh ax]\n[ch] [t S] ch erry [ch eh r iy]\n[jh] [d Z] j ar [jh aa r]\n[l] [l] l icorice [l ih k axr ix sh]\n[w] [w] kiw i [k iy w iy]\n[r] [r] r ice [r ay s]\n[y] [j] y ellow [y eh l ow]\n[h] [h] h oney [h ah n iy]\nFigure H.1 ARPAbet and IPA symbols for English consonants (left) and vowels (right).\nbeginning of platypus ,puma , and plantain , the middle of leopard , or the end of an-\ntelope . In general, however, the mapping between the letters of English orthography\nand phones is relatively opaque ; a single letter can represent very different sounds\nin different contexts. The English letter ccorresponds to phone [k] in cougar [k uw\ng axr], but phone [s] in cell[s eh l]. Besides appearing as candk, the phone [k] can\nappear as part of x(fox[f aa k s]), as ck(jackal [jh ae k el]) and as cc(raccoon [r ae\nk uw n]). Many other languages, for example, Spanish, are much more transparent\nin their sound-orthography mapping than English.\nH.2 Articulatory Phonetics\nArticulatory phonetics is the study of how these phones are produced as the variousarticulatory\nphonetics\norgans in the mouth, throat, and nose modify the air\ufb02ow from the lungs.\nThe Vocal Organs\nFigure H.2 shows the organs of speech. Sound is produced by the rapid movement\nof air. Humans produce most sounds in spoken languages by expelling air from the\nlungs through the windpipe (technically, the trachea ) and then out the mouth or\nnose. As it passes through the trachea, the air passes through the larynx , commonly\nknown as the Adam\u2019s apple or voice box. The larynx contains two small folds of H.2 \u2022 A RTICULATORY PHONETICS 3\nFigure H.2 The vocal organs, shown in side view. (Figure from OpenStax University\nPhysics, CC BY 4.0)\nmuscle, the vocal folds (often referred to non-technically as the vocal cords ), which\ncan be moved together or apart. The space between these two folds is called the\nglottis . If the folds are close together (but not tightly closed), they will vibrate as air glottis\npasses through them; if they are far apart, they won\u2019t vibrate. Sounds made with the\nvocal folds together and vibrating are called voiced ; sounds made without this vocal voiced sound\ncord vibration are called unvoiced orvoiceless . V oiced sounds include [b], [d], [g], unvoiced sound\n[v], [z], and all the English vowels, among others. Unvoiced sounds include [p], [t],\n[k], [f], [s], and others.\nThe area above the trachea is called the vocal tract ; it consists of the oral tract\nand the nasal tract . After the air leaves the trachea, it can exit the body through the\nmouth or the nose. Most sounds are made by air passing through the mouth. Sounds\nmade by air passing through the nose are called nasal sounds ; nasal sounds (like nasal\nEnglish [m], [n], and [ng]) use both the oral and nasal tracts as resonating cavities.\nPhones are divided into two main classes: consonants andvowels . Both kinds consonant\nvowel of sounds are formed by the motion of air through the mouth, throat or nose. Con-\nsonants are made by restriction or blocking of the air\ufb02ow in some way, and can be\nvoiced or unvoiced. V owels have less obstruction, are usually voiced, and are gen-\nerally louder and longer-lasting than consonants. The technical use of these terms is\nmuch like the common usage; [p], [b], [t], [d], [k], [g], [f], [v], [s], [z], [r], [l], etc.,\nare consonants; [aa], [ae], [ao], [ih], [aw], [ow], [uw], etc., are vowels. Semivow-\nels(such as [y] and [w]) have some of the properties of both; they are voiced like\nvowels, but they are short and less syllabic like consonants. 4APPENDIX H \u2022 P HONETICS\nConsonants: Place of Articulation\nBecause consonants are made by restricting air\ufb02ow, we can group them into classes\nby their point of maximum restriction, their place of articulation (Fig. H.3).place of\narticulation\n(nasal tract)dentalbilabialglottalpalatalvelaralveolar\nFigure H.3 Major English places of articulation.\nLabial: Consonants whose main restriction is formed by the two lips coming to- labial\ngether have a bilabial place of articulation. In English these include [p] as\ninpossum , [b] as in bear, and [m] as in marmot . The English labiodental\nconsonants [v] and [f] are made by pressing the bottom lip against the upper\nrow of teeth and letting the air \ufb02ow through the space in the upper teeth.\nDental: Sounds that are made by placing the tongue against the teeth are dentals. dental\nThe main dentals in English are the [th] of thingand the [dh] of though , which\nare made by placing the tongue behind the teeth with the tip slightly between\nthe teeth.\nAlveolar: The alveolar ridge is the portion of the roof of the mouth just behind the alveolar\nupper teeth. Most speakers of American English make the phones [s], [z], [t],\nand [d] by placing the tip of the tongue against the alveolar ridge. The word\ncoronal is often used to refer to both dental and alveolar.\nPalatal: The roof of the mouth (the palate ) rises sharply from the back of the palatal\npalate alveolar ridge. The palato-alveolar sounds [sh] ( shrimp ), [ch] ( china), [zh]\n(Asian ), and [jh] ( jar) are made with the blade of the tongue against the rising\nback of the alveolar ridge. The palatal sound [y] of yakis made by placing the\nfront of the tongue up close to the palate.\nVelar: Thevelum , or soft palate, is a movable muscular \ufb02ap at the very back of the velar\nroof of the mouth. The sounds [k] ( cuckoo), [g] ( goose), and [N](king\ufb01sher )\nare made by pressing the back of the tongue up against the velum.\nGlottal: The glottal stop [q] is made by closing the glottis (by bringing the vocal glottal\nfolds together).\nConsonants: Manner of Articulation\nConsonants are also distinguished by how the restriction in air\ufb02ow is made, for ex-\nample, by a complete stoppage of air or by a partial blockage. This feature is called\nthemanner of articulation of a consonant. The combination of place and mannermanner of\narticulation\nof articulation is usually suf\ufb01cient to uniquely identify a consonant. Following are\nthe major manners of articulation for English consonants:\nAstop is a consonant in which air\ufb02ow is completely blocked for a short time. stop\nThis blockage is followed by an explosive sound as the air is released. The period\nof blockage is called the closure , and the explosion is called the release . English H.2 \u2022 A RTICULATORY PHONETICS 5\nhas voiced stops like [b], [d], and [g] as well as unvoiced stops like [p], [t], and [k].\nStops are also called plosives .\nThenasal sounds [n], [m], and [ng] are made by lowering the velum and allow- nasal\ning air to pass into the nasal cavity.\nInfricatives , air\ufb02ow is constricted but not cut off completely. The turbulent fricatives\nair\ufb02ow that results from the constriction produces a characteristic \u201chissing\u201d sound.\nThe English labiodental fricatives [f] and [v] are produced by pressing the lower\nlip against the upper teeth, allowing a restricted air\ufb02ow between the upper teeth.\nThe dental fricatives [th] and [dh] allow air to \ufb02ow around the tongue between the\nteeth. The alveolar fricatives [s] and [z] are produced with the tongue against the\nalveolar ridge, forcing air over the edge of the teeth. In the palato-alveolar fricatives\n[sh] and [zh], the tongue is at the back of the alveolar ridge, forcing air through a\ngroove formed in the tongue. The higher-pitched fricatives (in English [s], [z], [sh]\nand [zh]) are called sibilants . Stops that are followed immediately by fricatives are sibilants\ncalled affricates ; these include English [ch] ( chicken ) and [jh] ( giraffe ).\nInapproximants , the two articulators are close together but not close enough to approximant\ncause turbulent air\ufb02ow. In English [y] ( yellow ), the tongue moves close to the roof\nof the mouth but not close enough to cause the turbulence that would characterize a\nfricative. In English [w] ( wood), the back of the tongue comes close to the velum.\nAmerican [r] can be formed in at least two ways; with just the tip of the tongue\nextended and close to the palate or with the whole tongue bunched up near the palate.\n[l] is formed with the tip of the tongue up against the alveolar ridge or the teeth, with\none or both sides of the tongue lowered to allow air to \ufb02ow over it. [l] is called a\nlateral sound because of the drop in the sides of the tongue.\nAtapor\ufb02ap[dx] is a quick motion of the tongue against the alveolar ridge. The tap\nconsonant in the middle of the word lotus ([l ow dx ax s]) is a tap in most dialects of\nAmerican English; speakers of many U.K. dialects would use a [t] instead.\nVowels\nLike consonants, vowels can be characterized by the position of the articulators as\nthey are made. The three most relevant parameters for vowels are what is called\nvowel height , which correlates roughly with the height of the highest part of the\ntongue, vowel frontness orbackness , indicating whether this high point is toward\nthe front or back of the oral tract and whether the shape of the lips is rounded or\nnot. Figure H.4 shows the position of the tongue for different vowels.\nboot [uw]closedvelumbat [ae]palatebeet [iy]tongue\nFigure H.4 Tongue positions for English high front [iy], low front [ae]and high back [uw].\nIn the vowel [iy], for example, the highest point of the tongue is toward the\nfront of the mouth. In the vowel [uw], by contrast, the high-point of the tongue is\nlocated toward the back of the mouth. V owels in which the tongue is raised toward\nthe front are called front vowels ; those in which the tongue is raised toward the Front vowel 6APPENDIX H \u2022 P HONETICS\nback are called back vowels . Note that while both [ih] and [eh] are front vowels, back vowel\nthe tongue is higher for [ih] than for [eh]. V owels in which the highest point of the\ntongue is comparatively high are called high vowels ; vowels with mid or low values high vowel\nof maximum tongue height are called mid vowels orlow vowels , respectively.\nfront back\nlowhigh\niy\nih\neh\naeuw\nuh\nax\nahao\naay uw\neyowoy\nayaw\nFigure H.5 The schematic \u201cvowel space\u201d for English vowels.\nFigure H.5 shows a schematic characterization of the height of different vowels.\nIt is schematic because the abstract property height correlates only roughly with ac-\ntual tongue positions; it is, in fact, a more accurate re\ufb02ection of acoustic facts. Note\nthat the chart has two kinds of vowels: those in which tongue height is represented\nas a point and those in which it is represented as a path. A vowel in which the tongue\nposition changes markedly during the production of the vowel is a diphthong . En- diphthong\nglish is particularly rich in diphthongs.\nThe second important articulatory dimension for vowels is the shape of the lips.\nCertain vowels are pronounced with the lips rounded (the same lip shape used for\nwhistling). These rounded vowels include [uw], [ao], and [ow]. rounded vowel\nSyllables\nConsonants and vowels combine to make a syllable . A syllable is a vowel-like (or syllable\nsonorant ) sound together with some of the surrounding consonants that are most\nclosely associated with it. The word doghas one syllable, [d aa g] (in our dialect);\nthe word catnip has two syllables, [k ae t] and [n ih p]. We call the vowel at the\ncore of a syllable the nucleus . Initial consonants, if any, are called the onset . Onsets nucleus\nonset with more than one consonant (as in strike [s t r ay k]), are called complex onsets .\nThecoda is the optional consonant or sequence of consonants following the nucleus. coda\nThus [d] is the onset of dog, and [g] is the coda. The rime , orrhyme , is the nucleus rime\nplus coda. Figure H.6 shows some sample syllable structures.\nThe task of automatically breaking up a word into syllables is called syllabi\ufb01ca-\ntion. Syllable structure is also closely related to the phonotactics of a language. The syllabi\ufb01cation\nterm phonotactics means the constraints on which phones can follow each other in phonotactics\na language. For example, English has strong constraints on what kinds of conso-\nnants can appear together in an onset; the sequence [zdr], for example, cannot be a\nlegal English syllable onset. Phonotactics can be represented by a language model\nor \ufb01nite-state model of phone sequences. H.3 \u2022 P ROSODY 7\ns\nRime\nCoda\nmNucleus\naeOnset\nhs\nRime\nCoda\nnNucleus\niyOnset\nrgs\nRime\nCoda\nzgNucleus\neh\nFigure H.6 Syllable structure of ham,green ,eggs.s=syllable.\nH.3 Prosody\nProsody is the study of the intonational and rhythmic aspects of language, and in prosody\nparticular the use of F0,energy , and duration to convey pragmatic, affective, or\nconversation-interactional meanings.1We\u2019ll introduce these acoustic quantities in\ndetail in the next section when we turn to acoustic phonetics, but brie\ufb02y we can\nthink of energy as the acoustic quality that we perceive as loudness, and F0 as the\nfrequency of the sound that is produced, the acoustic quality that we hear as the\npitch of an utterance. Prosody can be used to mark discourse structure , like the\ndifference between statements and questions, or the way that a conversation is struc-\ntured. Prosody is used to mark the saliency of a particular word or phrase. Prosody\nis heavily used for paralinguistic functions like conveying affective meanings like\nhappiness, surprise, or anger. And prosody plays an important role in managing\nturn-taking in conversation.\nH.3.1 Prosodic Prominence: Accent, Stress and Schwa\nIn a natural utterance of American English, some words sound more prominent than prominence\nothers, and certain syllables in these words are also more prominent than others.\nWhat we mean by prominence is that these words or syllables are perceptually more\nsalient to the listener. Speakers make a word or syllable more salient in English\nby saying it louder, saying it slower (so it has a longer duration), or by varying F0\nduring the word, making it higher or more variable.\nAccent We represent prominence via a linguistic marker called pitch accent . Words pitch accent\nor syllables that are prominent are said to bear (be associated with) a pitch accent.\nThus this utterance might be pronounced by accenting the underlined words:\n(H.1) I\u2019m a little surprised to hear it characterized as happy .\nLexical Stress The syllables that bear pitch accent are called accented syllables.\nNot every syllable of a word can be accented: pitch accent has to be realized on the\nsyllable that has lexical stress . Lexical stress is a property of the word\u2019s pronuncia- lexical stress\ntion in dictionaries; the syllable that has lexical stress is the one that will be louder\nor longer if the word is accented. For example, the word surprised is stressed on its\nsecond syllable, not its \ufb01rst. (Try stressing the other syllable by saying SURprised;\nhopefully that sounds wrong to you). Thus, if the word surprised receives a pitch\naccent in a sentence, it is the second syllable that will be stronger. The following ex-\n1The word is used in a different but related way in poetry, to mean the study of verse metrical structure. 8APPENDIX H \u2022 P HONETICS\nample shows underlined accented words with the stressed syllable bearing the accent\n(the louder, longer syllable) in boldface:\n(H.2) I\u2019m a little sur prised to hear it char acterized ashappy.\nStress is marked in dictionaries. The CMU dictionary (CMU, 1993), for ex-\nample, marks vowels with 0 (unstressed) or 1 (stressed) as in entries for counter :\n[K AW1 N T ER0], or table : [T EY1 B AH0 L]. Difference in lexical stress can\naffect word meaning; the noun content is pronounced [K AA1 N T EH0 N T], while\nthe adjective is pronounced [K AA0 N T EH1 N T].\nReduced Vowels and Schwa Unstressed vowels can be weakened even further to\nreduced vowels , the most common of which is schwa ([ax]), as in the second vowel reduced vowel\nschwa ofparakeet : [p ae r ax k iy t]. In a reduced vowel the articulatory gesture isn\u2019t as\ncomplete as for a full vowel. Not all unstressed vowels are reduced; any vowel, and\ndiphthongs in particular, can retain its full quality even in unstressed position. For\nexample, the vowel [iy] can appear in stressed position as in the word eat[iy t] or in\nunstressed position as in the word carry [k ae r iy].\nIn summary, there is a continuum of prosodic prominence , for which it is often prominence\nuseful to represent levels like accented, stressed, full vowel, and reduced vowel.\nH.3.2 Prosodic Structure\nSpoken sentences have prosodic structure: some words seem to group naturally to-\ngether, while some words seem to have a noticeable break or disjuncture between\nthem. Prosodic structure is often described in terms of prosodic phrasing , mean-prosodic\nphrasing\ning that an utterance has a prosodic phrase structure in a similar way to it having\na syntactic phrase structure. For example, the sentence I wanted to go to London,\nbut could only get tickets for France seems to have two main intonation phrases ,intonation\nphrase\ntheir boundary occurring at the comma. Furthermore, in the \ufb01rst phrase, there seems\nto be another set of lesser prosodic phrase boundaries (often called intermediate\nphrase s) that split up the words as I wantedjto gojto London . These kinds ofintermediate\nphrase\nintonation phrases are often correlated with syntactic structure constituents (Price\net al. 1991, Bennett and Elfner 2019).\nAutomatically predicting prosodic boundaries can be important for tasks like\nTTS. Modern approaches use sequence models that take either raw text or text an-\nnotated with features like parse trees as input, and make a break/no-break decision\nat each word boundary. They can be trained on data labeled for prosodic structure\nlike the Boston University Radio News Corpus (Ostendorf et al., 1995).\nH.3.3 Tune\nTwo utterances with the same prominence and phrasing patterns can still differ\nprosodically by having different tunes . The tune of an utterance is the rise and tune\nfall of its F0 over time. A very obvious example of tune is the difference between\nstatements and yes-no questions in English. The same words can be said with a \ufb01nal\nF0 rise to indicate a yes-no question (called a question rise ): question rise\nYou    know    what    Imean ?\nor a \ufb01nal drop in F0 (called a \ufb01nal fall ) to indicate a declarative intonation: \ufb01nal fall H.4 \u2022 A COUSTIC PHONETICS AND SIGNALS 9\nYou    know        what         Imean .\nLanguages make wide use of tune to express meaning (Xu, 2005). In English,\nfor example, besides this well-known rise for yes-no questions, a phrase containing\na list of nouns separated by commas often has a short rise called a continuation\nriseafter each noun. Other examples include the characteristic English contours forcontinuation\nrise\nexpressing contradiction and expressing surprise .\nLinking Prominence and Tune\nPitch accents come in different varieties that are related to tune; high pitched accents,\nfor example, have different functions than low pitched accents. There are many\ntypologies of accent classes in different languages. One such typology is part of the\nToBI (Tone and Break Indices) theory of intonation (Silverman et al. 1992). Each ToBI\nword in ToBI can be associated with one of \ufb01ve types of pitch accents shown in\nin Fig. H.7. Each utterance in ToBI consists of a sequence of intonational phrases,\neach of which ends in one of four boundary tones shown in Fig. H.7, representing boundary tone\nthe utterance \ufb01nal aspects of tune. There are version of ToBI for many languages.\nPitch Accents Boundary Tones\nH* peak accent L-L% \u201c\ufb01nal fall\u201d: \u201cdeclarative contour\u201d of American\nEnglish\nL* low accent L-H% continuation rise\nL*+H scooped accent H-H% \u201cquestion rise\u201d: cantonical yes-no question\ncontour\nL+H* rising peak accent H-L% \ufb01nal level plateau\nH+!H* step down\nFigure H.7 The accent and boundary tones labels from the ToBI transcription system for\nAmerican English intonation (Beckman and Ayers 1997, Beckman and Hirschberg 1994).\nH.4 Acoustic Phonetics and Signals\nWe begin with a very brief introduction to the acoustic waveform and its digitization\nand frequency analysis; the interested reader is encouraged to consult the references\nat the end of the chapter.\nH.4.1 Waves\nAcoustic analysis is based on the sine and cosine functions. Figure H.8 shows a plot\nof a sine wave, in particular the function\ny=A\u0003sin(2pft) (H.3)\nwhere we have set the amplitude A to 1 and the frequency fto 10 cycles per second.\nRecall from basic mathematics that two important characteristics of a wave are\nitsfrequency andamplitude . The frequency is the number of times a second that a frequency\namplitude wave repeats itself, that is, the number of cycles . We usually measure frequency in\ncycles per second . The signal in Fig. H.8 repeats itself 5 times in .5 seconds, hence\n10 cycles per second. Cycles per second are usually called hertz (shortened to Hz), Hertz 10 APPENDIX H \u2022 P HONETICS\nTime (s)0 0.5\u20131.01.0\n0\n0 0.1 0.2 0.3 0.4 0.5\nFigure H.8 A sine wave with a frequency of 10 Hz and an amplitude of 1.\nso the frequency in Fig. H.8 would be described as 10 Hz. The amplitude Aof a\nsine wave is the maximum value on the Y axis. The period Tof the wave is the time period\nit takes for one cycle to complete, de\ufb01ned as\nT=1\nf(H.4)\nEach cycle in Fig. H.8 lasts a tenth of a second; hence T=:1 seconds.\nH.4.2 Speech Sound Waves\nLet\u2019s turn from hypothetical waves to sound waves. The input to a speech recog-\nnizer, like the input to the human ear, is a complex series of changes in air pressure.\nThese changes in air pressure obviously originate with the speaker and are caused\nby the speci\ufb01c way that air passes through the glottis and out the oral or nasal cav-\nities. We represent sound waves by plotting the change in air pressure over time.\nOne metaphor which sometimes helps in understanding these graphs is that of a ver-\ntical plate blocking the air pressure waves (perhaps in a microphone in front of a\nspeaker\u2019s mouth, or the eardrum in a hearer\u2019s ear). The graph measures the amount\nofcompression orrarefaction (uncompression) of the air molecules at this plate.\nFigure H.9 shows a short segment of a waveform taken from the Switchboard corpus\nof telephone speech of the vowel [iy] from someone saying \u201cshe just had a baby\u201d.\nTime (s)0 0.03875\u20130.016970.02283\n0\nFigure H.9 A waveform of the vowel [iy] from an utterance shown later in Fig. H.13 on page 14. The y-axis\nshows the level of air pressure above and below normal atmospheric pressure. The x-axis shows time. Notice\nthat the wave repeats regularly.\nThe \ufb01rst step in digitizing a sound wave like Fig. H.9 is to convert the analog\nrepresentations (\ufb01rst air pressure and then analog electric signals in a microphone)\ninto a digital signal. This analog-to-digital conversion has two steps: sampling and sampling\nquantization . To sample a signal, we measure its amplitude at a particular time; the\nsampling rate is the number of samples taken per second. To accurately measure a\nwave, we must have at least two samples in each cycle: one measuring the positive H.4 \u2022 A COUSTIC PHONETICS AND SIGNALS 11\npart of the wave and one measuring the negative part. More than two samples per\ncycle increases the amplitude accuracy, but fewer than two samples causes the fre-\nquency of the wave to be completely missed. Thus, the maximum frequency wave\nthat can be measured is one whose frequency is half the sample rate (since every\ncycle needs two samples). This maximum frequency for a given sampling rate is\ncalled the Nyquist frequency . Most information in human speech is in frequenciesNyquist\nfrequency\nbelow 10,000 Hz; thus, a 20,000 Hz sampling rate would be necessary for com-\nplete accuracy. But telephone speech is \ufb01ltered by the switching network, and only\nfrequencies less than 4,000 Hz are transmitted by telephones. Thus, an 8,000 Hz\nsampling rate is suf\ufb01cient for telephone-bandwidth speech like the Switchboard\ncorpus, while 16,000 Hz sampling is often used for microphone speech.\nEven an 8,000 Hz sampling rate requires 8000 amplitude measurements for each\nsecond of speech, so it is important to store amplitude measurements ef\ufb01ciently.\nThey are usually stored as integers, either 8 bit (values from -128\u2013127) or 16 bit\n(values from -32768\u201332767). This process of representing real-valued numbers as\nintegers is called quantization because the difference between two integers acts as quantization\na minimum granularity (a quantum size) and all values that are closer together than\nthis quantum size are represented identically.\nOnce data is quantized, it is stored in various formats. One parameter of these\nformats is the sample rate and sample size discussed above; telephone speech is\noften sampled at 8 kHz and stored as 8-bit samples, and microphone data is often\nsampled at 16 kHz and stored as 16-bit samples. Another parameter is the number of\nchannels . For stereo data or for two-party conversations, we can store both channels channel\nin the same \ufb01le or we can store them in separate \ufb01les. A \ufb01nal parameter is individual\nsample storage\u2014linearly or compressed. One common compression format used for\ntelephone speech is m-law (often written u-law but still pronounced mu-law). The\nintuition of log compression algorithms like m-law is that human hearing is more\nsensitive at small intensities than large ones; the log represents small values with\nmore faithfulness at the expense of more error on large values. The linear (unlogged)\nvalues are generally referred to as linear PCM values (PCM stands for pulse code PCM\nmodulation, but never mind that). Here\u2019s the equation for compressing a linear PCM\nsample value xto 8-bit m-law, (where m=255 for 8 bits):\nF(x) =sgn(x)log(1+mjxj)\nlog(1+m)\u00001\u0014x\u00141 (H.5)\nThere are a number of standard \ufb01le formats for storing the resulting digitized wave-\n\ufb01le, such as Microsoft\u2019s .wav and Apple\u2019s AIFF all of which have special headers;\nsimple headerless \u201craw\u201d \ufb01les are also used. For example, the .wav format is a subset\nof Microsoft\u2019s RIFF format for multimedia \ufb01les; RIFF is a general format that can\nrepresent a series of nested chunks of data and control information. Figure H.10\nshows a simple .wav \ufb01le with a single data chunk together with its format chunk.\nFigure H.10 Microsoft wave\ufb01le header format, assuming simple \ufb01le with one chunk. Fol-\nlowing this 44-byte header would be the data chunk. 12 APPENDIX H \u2022 P HONETICS\nH.4.3 Frequency and Amplitude; Pitch and Loudness\nSound waves, like all waves, can be described in terms of frequency, amplitude, and\nthe other characteristics that we introduced earlier for pure sine waves. In sound\nwaves, these are not quite as simple to measure as they were for sine waves. Let\u2019s\nconsider frequency. Note in Fig. H.9 that although not exactly a sine, the wave is\nnonetheless periodic, repeating 10 times in the 38.75 milliseconds (.03875 seconds)\ncaptured in the \ufb01gure. Thus, the frequency of this segment of the wave is 10/.03875\nor 258 Hz.\nWhere does this periodic 258 Hz wave come from? It comes from the speed of\nvibration of the vocal folds; since the waveform in Fig. H.9 is from the vowel [iy], it\nis voiced. Recall that voicing is caused by regular openings and closing of the vocal\nfolds. When the vocal folds are open, air is pushing up through the lungs, creating\na region of high pressure. When the folds are closed, there is no pressure from the\nlungs. Thus, when the vocal folds are vibrating, we expect to see regular peaks in\namplitude of the kind we see in Fig. H.9, each major peak corresponding to an open-\ning of the vocal folds. The frequency of the vocal fold vibration, or the frequency\nof the complex wave, is called the fundamental frequency of the waveform, oftenfundamental\nfrequency\nabbreviated F0. We can plot F0 over time in a pitch track . Figure H.11 shows the F0\npitch track pitch track of a short question, \u201cThree o\u2019clock?\u201d represented below the waveform.\nNote the rise in F0 at the end of the question.\nthree o\u2019clock\nTime (s)0 0.5443750 Hz500 Hz\nFigure H.11 Pitch track of the question \u201cThree o\u2019clock?\u201d, shown below the wave\ufb01le. Note\nthe rise in F0 at the end of the question. Note the lack of pitch trace during the very quiet part\n(the \u201co\u2019\u201d of \u201co\u2019clock\u201d; automatic pitch tracking is based on counting the pulses in the voiced\nregions, and doesn\u2019t work if there is no voicing (or insuf\ufb01cient sound).\nThe vertical axis in Fig. H.9 measures the amount of air pressure variation; pres-\nsure is force per unit area, measured in Pascals (Pa). A high value on the vertical\naxis (a high amplitude) indicates that there is more air pressure at that point in time,\na zero value means there is normal (atmospheric) air pressure, and a negative value\nmeans there is lower than normal air pressure (rarefaction).\nIn addition to this value of the amplitude at any point in time, we also often\nneed to know the average amplitude over some time range, to give us some idea\nof how great the average displacement of air pressure is. But we can\u2019t just take\nthe average of the amplitude values over a range; the positive and negative values\nwould (mostly) cancel out, leaving us with a number close to zero. Instead, we\ngenerally use the RMS (root-mean-square) amplitude, which squares each number H.4 \u2022 A COUSTIC PHONETICS AND SIGNALS 13\nbefore averaging (making it positive), and then takes the square root at the end.\nRMS amplitudeN\ni=1=vuut1\nNNX\ni=1x2\ni (H.6)\nThepower of the signal is related to the square of the amplitude. If the number power\nof samples of a sound is N, the power is\nPower =1\nNNX\ni=1x2\ni (H.7)\nRather than power, we more often refer to the intensity of the sound, which intensity\nnormalizes the power to the human auditory threshold and is measured in dB. If P0\nis the auditory threshold pressure = 2 \u000210\u00005Pa, then intensity is de\ufb01ned as follows:\nIntensity =10log101\nNP0NX\ni=1x2\ni (H.8)\nFigure H.12 shows an intensity plot for the sentence \u201cIs it a long movie?\u201d from\nthe CallHome corpus, again shown below the waveform plot.\nis it a long movie?\nTime (s)0 1.1675\nFigure H.12 Intensity plot for the sentence \u201cIs it a long movie?\u201d. Note the intensity peaks\nat each vowel and the especially high peak for the word long.\nTwo important perceptual properties, pitch andloudness , are related to fre-\nquency and intensity. The pitch of a sound is the mental sensation, or perceptual pitch\ncorrelate, of fundamental frequency; in general, if a sound has a higher fundamen-\ntal frequency we perceive it as having a higher pitch. We say \u201cin general\u201d because\nthe relationship is not linear, since human hearing has different acuities for different\nfrequencies. Roughly speaking, human pitch perception is most accurate between\n100 Hz and 1000 Hz and in this range pitch correlates linearly with frequency. Hu-\nman hearing represents frequencies above 1000 Hz less accurately, and above this\nrange, pitch correlates logarithmically with frequency. Logarithmic representation\nmeans that the differences between high frequencies are compressed and hence not\nas accurately perceived. There are various psychoacoustic models of pitch percep-\ntion scales. One common model is the melscale (Stevens et al. 1937, Stevens and Mel 14 APPENDIX H \u2022 P HONETICS\nV olkmann 1940). A mel is a unit of pitch de\ufb01ned such that pairs of sounds which\nare perceptually equidistant in pitch are separated by an equal number of mels. The\nmel frequency mcan be computed from the raw acoustic frequency as follows:\nm=1127ln (1+f\n700) (H.9)\nAs we\u2019ll see in Chapter 16, the mel scale plays an important role in speech\nrecognition.\nTheloudness of a sound is the perceptual correlate of the power . So sounds with\nhigher amplitudes are perceived as louder, but again the relationship is not linear.\nFirst of all, as we mentioned above when we de\ufb01ned m-law compression, humans\nhave greater resolution in the low-power range; the ear is more sensitive to small\npower differences. Second, it turns out that there is a complex relationship between\npower, frequency, and perceived loudness; sounds in certain frequency ranges are\nperceived as being louder than those in other frequency ranges.\nVarious algorithms exist for automatically extracting F0. In a slight abuse of ter-\nminology, these are called pitch extraction algorithms. The autocorrelation method pitch extraction\nof pitch extraction, for example, correlates the signal with itself at various offsets.\nThe offset that gives the highest correlation gives the period of the signal. There\nare various publicly available pitch extraction toolkits; for example, an augmented\nautocorrelation pitch tracker is provided with Praat (Boersma and Weenink, 2005).\nH.4.4 Interpretation of Phones from a Waveform\nMuch can be learned from a visual inspection of a waveform. For example, vowels\nare pretty easy to spot. Recall that vowels are voiced; another property of vowels is\nthat they tend to be long and are relatively loud (as we can see in the intensity plot in\nFig. H.12). Length in time manifests itself directly on the x-axis, and loudness is re-\nlated to (the square of) amplitude on the y-axis. We saw in the previous section that\nvoicing is realized by regular peaks in amplitude of the kind we saw in Fig. H.9, each\nmajor peak corresponding to an opening of the vocal folds. Figure H.13 shows the\nwaveform of the short sentence \u201cshe just had a baby\u201d. We have labeled this wave-\nform with word and phone labels. Notice that each of the six vowels in Fig. H.13,\n[iy], [ax], [ae], [ax], [ey], [iy], all have regular amplitude peaks indicating voicing.\nshe just had a baby\nsh iy j ax s h ae dx ax b ey b iy\nTime (s)0 1.059\nFigure H.13 A waveform of the sentence \u201cShe just had a baby\u201d from the Switchboard corpus (conversation\n4325). The speaker is female, was 20 years old in 1991, which is approximately when the recording was made,\nand speaks the South Midlands dialect of American English.\nFor a stop consonant, which consists of a closure followed by a release, we can\noften see a period of silence or near silence followed by a slight burst of amplitude.\nWe can see this for both of the [b]\u2019s in baby in Fig. H.13. H.4 \u2022 A COUSTIC PHONETICS AND SIGNALS 15\nAnother phone that is often quite recognizable in a waveform is a fricative. Re-\ncall that fricatives, especially very strident fricatives like [sh], are made when a\nnarrow channel for air\ufb02ow causes noisy, turbulent air. The resulting hissy sounds\nhave a noisy, irregular waveform. This can be seen somewhat in Fig. H.13; it\u2019s even\nclearer in Fig. H.14, where we\u2019ve magni\ufb01ed just the \ufb01rst word she.\nshe\nsh iy\nTime (s)0 0.257\nFigure H.14 A more detailed view of the \ufb01rst word \u201cshe\u201d extracted from the wave\ufb01le in Fig. H.13. Notice\nthe difference between the random noise of the fricative [sh] and the regular voicing of the vowel [iy].\nH.4.5 Spectra and the Frequency Domain\nWhile some broad phonetic features (such as energy, pitch, and the presence of voic-\ning, stop closures, or fricatives) can be interpreted directly from the waveform, most\ncomputational applications such as speech recognition (as well as human auditory\nprocessing) are based on a different representation of the sound in terms of its com-\nponent frequencies. The insight of Fourier analysis is that every complex wave can\nbe represented as a sum of many sine waves of different frequencies. Consider the\nwaveform in Fig. H.15. This waveform was created (in Praat) by summing two sine\nwaveforms, one of frequency 10 Hz and one of frequency 100 Hz.\nTime (s)0 0.5\u201311\n0\nFigure H.15 A waveform that is the sum of two sine waveforms, one of frequency 10\nHz (note \ufb01ve repetitions in the half-second window) and one of frequency 100 Hz, both of\namplitude 1.\nWe can represent these two component frequencies with a spectrum . The spec- spectrum\ntrum of a signal is a representation of each of its frequency components and their\namplitudes. Figure H.16 shows the spectrum of Fig. H.15. Frequency in Hz is on\nthe x-axis and amplitude on the y-axis. Note the two spikes in the \ufb01gure, one at\n10 Hz and one at 100 Hz. Thus, the spectrum is an alternative representation of\nthe original waveform, and we use the spectrum as a tool to study the component\nfrequencies of a sound wave at a particular time point. 16 APPENDIX H \u2022 P HONETICS\nFrequency (Hz)1 10 100 2 20 200 5 50Sound pressure level (dB /Hz)\n406080\nFigure H.16 The spectrum of the waveform in Fig. H.15.\nLet\u2019s look now at the frequency components of a speech waveform. Figure H.17\nshows part of the waveform for the vowel [ae] of the word had, cut out from the\nsentence shown in Fig. H.13.\nTime (s)0 0.04275\u20130.055540.04968\n0\nFigure H.17 The waveform of part of the vowel [ae] from the word hadcut out from the\nwaveform shown in Fig. H.13.\nNote that there is a complex wave that repeats about ten times in the \ufb01gure; but\nthere is also a smaller repeated wave that repeats four times for every larger pattern\n(notice the four small peaks inside each repeated wave). The complex wave has a\nfrequency of about 234 Hz (we can \ufb01gure this out since it repeats roughly 10 times\nin .0427 seconds, and 10 cycles/.0427 seconds = 234 Hz).\nThe smaller wave then should have a frequency of roughly four times the fre-\nquency of the larger wave, or roughly 936 Hz. Then, if you look carefully, you can\nsee two little waves on the peak of many of the 936 Hz waves. The frequency of this\ntiniest wave must be roughly twice that of the 936 Hz wave, hence 1872 Hz.\nFigure H.18 shows a smoothed spectrum for the waveform in Fig. H.17, com-\nputed with a discrete Fourier transform (DFT).\nFrequency (Hz)0 4000Sound pressure level (dB /Hz)\n\u201320020\n0 2000 4000 0 1000 2000 3000 4000\nFigure H.18 A spectrum for the vowel [ae] from the word hadin the waveform of She just\nhad a baby in Fig. H.13. H.4 \u2022 A COUSTIC PHONETICS AND SIGNALS 17\nThe x-axis of a spectrum shows frequency, and the y-axis shows some mea-\nsure of the magnitude of each frequency component (in decibels (dB), a logarithmic\nmeasure of amplitude that we saw earlier). Thus, Fig. H.18 shows signi\ufb01cant fre-\nquency components at around 930 Hz, 1860 Hz, and 3020 Hz, along with many\nother lower-magnitude frequency components. These \ufb01rst two components are just\nwhat we noticed in the time domain by looking at the wave in Fig. H.17!\nWhy is a spectrum useful? It turns out that these spectral peaks that are easily\nvisible in a spectrum are characteristic of different phones; phones have characteris-\ntic spectral \u201csignatures\u201d. Just as chemical elements give off different wavelengths of\nlight when they burn, allowing us to detect elements in stars by looking at the spec-\ntrum of the light, we can detect the characteristic signature of the different phones\nby looking at the spectrum of a waveform. This use of spectral information is essen-\ntial to both human and machine speech recognition. In human audition, the function\nof the cochlea , orinner ear , is to compute a spectrum of the incoming waveform. cochlea\nSimilarly, the acoustic features used in speech recognition are spectral representa-\ntions.\nLet\u2019s look at the spectrum of different vowels. Since some vowels change over\ntime, we\u2019ll use a different kind of plot called a spectrogram . While a spectrum\nshows the frequency components of a wave at one point in time, a spectrogram is a spectrogram\nway of envisioning how the different frequencies that make up a waveform change\nover time. The x-axis shows time, as it did for the waveform, but the y-axis now\nshows frequencies in hertz. The darkness of a point on a spectrogram corresponds\nto the amplitude of the frequency component. Very dark points have high amplitude,\nlight points have low amplitude. Thus, the spectrogram is a useful way of visualizing\nthe three dimensions (time x frequency x amplitude).\nFigure H.19 shows spectrograms of three American English vowels, [ih], [ae],\nand [ah]. Note that each vowel has a set of dark bars at various frequency bands,\nslightly different bands for each vowel. Each of these represents the same kind of\nspectral peak that we saw in Fig. H.17.\nTime (s)0 2.8139705000Frequency (Hz)\nFigure H.19 Spectrograms for three American English vowels, [ih], [ae], and [uh]\nEach dark bar (or spectral peak) is called a formant . As we discuss below, a formant\nformant is a frequency band that is particularly ampli\ufb01ed by the vocal tract. Since\ndifferent vowels are produced with the vocal tract in different positions, they will\nproduce different kinds of ampli\ufb01cations or resonances. Let\u2019s look at the \ufb01rst two\nformants, called F1 and F2. Note that F1, the dark bar closest to the bottom, is in a\ndifferent position for the three vowels; it\u2019s low for [ih] (centered at about 470 Hz)\nand somewhat higher for [ae] and [ah] (somewhere around 800 Hz). By contrast,\nF2, the second dark bar from the bottom, is highest for [ih], in the middle for [ae],\nand lowest for [ah].\nWe can see the same formants in running speech, although the reduction and 18 APPENDIX H \u2022 P HONETICS\ncoarticulation processes make them somewhat harder to see. Figure H.20 shows the\nspectrogram of \u201cshe just had a baby\u201d, whose waveform was shown in Fig. H.13. F1\nand F2 (and also F3) are pretty clear for the [ax] of just, the [ae] of had, and the [ey]\nofbaby .\nshe just had a baby\nsh iy j ax s h ae dxax b ey b iy\nTime (s)0 1.059\nFigure H.20 A spectrogram of the sentence \u201cshe just had a baby\u201d whose waveform was shown in Fig. H.13.\nWe can think of a spectrogram as a collection of spectra (time slices), like Fig. H.18 placed end to end.\nWhat speci\ufb01c clues can spectral representations give for phone identi\ufb01cation?\nFirst, since different vowels have their formants at characteristic places, the spectrum\ncan distinguish vowels from each other. We\u2019ve seen that [ae] in the sample waveform\nhad formants at 930 Hz, 1860 Hz, and 3020 Hz. Consider the vowel [iy] at the\nbeginning of the utterance in Fig. H.13. The spectrum for this vowel is shown in\nFig. H.21. The \ufb01rst formant of [iy] is 540 Hz, much lower than the \ufb01rst formant for\n[ae], and the second formant (2581 Hz) is much higher than the second formant for\n[ae]. If you look carefully, you can see these formants as dark bars in Fig. H.20 just\naround 0.5 seconds.\n\u22121001020304050607080\n0 1000 2000 3000\nFigure H.21 A smoothed (LPC) spectrum for the vowel [iy] at the start of She just had a\nbaby . Note that the \ufb01rst formant (540 Hz) is much lower than the \ufb01rst formant for [ae] shown\nin Fig. H.18, and the second formant (2581 Hz) is much higher than the second formant for\n[ae].\nThe location of the \ufb01rst two formants (called F1 and F2) plays a large role in de-\ntermining vowel identity, although the formants still differ from speaker to speaker.\nHigher formants tend to be caused more by general characteristics of a speaker\u2019s\nvocal tract rather than by individual vowels. Formants also can be used to identify\nthe nasal phones [n], [m], and [ng] and the liquids [l] and [r]. H.4 \u2022 A COUSTIC PHONETICS AND SIGNALS 19\nH.4.6 The Source-Filter Model\nWhy do different vowels have different spectral signatures? As we brie\ufb02y mentioned\nabove, the formants are caused by the resonant cavities of the mouth. The source-\n\ufb01lter model is a way of explaining the acoustics of a sound by modeling how thesource-\ufb01lter\nmodel\npulses produced by the glottis (the source ) are shaped by the vocal tract (the \ufb01lter ).\nLet\u2019s see how this works. Whenever we have a wave such as the vibration in air\ncaused by the glottal pulse, the wave also has harmonics . A harmonic is another harmonic\nwave whose frequency is a multiple of the fundamental wave. Thus, for example, a\n115 Hz glottal fold vibration leads to harmonics (other waves) of 230 Hz, 345 Hz,\n460 Hz, and so on on. In general, each of these waves will be weaker, that is, will\nhave much less amplitude than the wave at the fundamental frequency.\nIt turns out, however, that the vocal tract acts as a kind of \ufb01lter or ampli\ufb01er;\nindeed any cavity, such as a tube, causes waves of certain frequencies to be ampli\ufb01ed\nand others to be damped. This ampli\ufb01cation process is caused by the shape of the\ncavity; a given shape will cause sounds of a certain frequency to resonate and hence\nbe ampli\ufb01ed. Thus, by changing the shape of the cavity, we can cause different\nfrequencies to be ampli\ufb01ed.\nWhen we produce particular vowels, we are essentially changing the shape of\nthe vocal tract cavity by placing the tongue and the other articulators in particular\npositions. The result is that different vowels cause different harmonics to be ampli-\n\ufb01ed. So a wave of the same fundamental frequency passed through different vocal\ntract positions will result in different harmonics being ampli\ufb01ed.\nWe can see the result of this ampli\ufb01cation by looking at the relationship between\nthe shape of the vocal tract and the corresponding spectrum. Figure H.22 shows\nthe vocal tract position for three vowels and a typical resulting spectrum. The for-\nmants are places in the spectrum where the vocal tract happens to amplify particular\nharmonic frequencies.\nFrequency (Hz)04000Sound pressure level (dB/Hz)\n020\n2682416F1F2[iy]  (tea)\nFrequency (Hz)04000Sound pressure level (dB/Hz)\n020\n9031695F1 F2[ae] (cat) \nFrequency (Hz)04000Sound pressure level (dB/Hz)\n\u2013200\n295817F1 F2[uw]  (moo)\n[ae] (cat) [uw]  (moo)[iy]  (tea)\nFigure H.22 Visualizing the vocal tract position as a \ufb01lter: the tongue positions for three English vowels and\nthe resulting smoothed spectra showing F1 and F2. 20 APPENDIX H \u2022 P HONETICS\nH.5 Phonetic Resources\nA wide variety of phonetic resources can be drawn on for computational work. On-\nlinepronunciation dictionaries give phonetic transcriptions for words. The LDCpronunciation\ndictionary\ndistributes pronunciation lexicons for Egyptian Arabic, Dutch, English, German,\nJapanese, Korean, Mandarin, and Spanish. For English, the CELEX dictionary\n(Baayen et al., 1995) has pronunciations for 160,595 wordforms, with syllabi\ufb01ca-\ntion, stress, and morphological and part-of-speech information. The open-source\nCMU Pronouncing Dictionary (CMU, 1993) has pronunciations for about 134,000\nwordforms, while the \ufb01ne-grained 110,000 word UNISYN dictionary (Fitt, 2002),\nfreely available for research purposes, gives syllabi\ufb01cations, stress, and also pronun-\nciations for dozens of dialects of English.\nAnother useful resource is a phonetically annotated corpus , in which a col-\nlection of waveforms is hand-labeled with the corresponding string of phones. The\nTIMIT corpus (NIST, 1990), originally a joint project between Texas Instruments\n(TI), MIT, and SRI, is a corpus of 6300 read sentences, with 10 sentences each from\n630 speakers. The 6300 sentences were drawn from a set of 2342 sentences, some\nselected to have particular dialect shibboleths, others to maximize phonetic diphone\ncoverage. Each sentence in the corpus was phonetically hand-labeled, the sequence\nof phones was automatically aligned with the sentence wave\ufb01le, and then the au-\ntomatic phone boundaries were manually hand-corrected (Seneff and Zue, 1988).\nThe result is a time-aligned transcription : a transcription in which each phone istime-aligned\ntranscription\nassociated with a start and end time in the waveform, like the example in Fig. H.23.\nshe had your dark suit ingreasy wash water all year\nsh iy hv ae dcl jh axr dcl d aa r kcl s ux q engcl g r iy s ix w aa sh q w aa dx axr q aa l y ix axr\nFigure H.23 Phonetic transcription from the TIMIT corpus, using special ARPAbet features for narrow tran-\nscription, such as the palatalization of [d] in had, unreleased \ufb01nal stop in dark, glottalization of \ufb01nal [t] in suit\nto [q], and \ufb02ap of [t] in water . The TIMIT corpus also includes time-alignments (not shown).\nThe Switchboard Transcription Project phonetically annotated corpus consists\nof 3.5 hours of sentences extracted from the Switchboard corpus (Greenberg et al.,\n1996), together with transcriptions time-aligned at the syllable level. Figure H.24\nshows an example .\n0.470 0.640 0.720 0.900 0.953 1.279 1.410 1.630\ndh er k aa n ax v ih m b ix t w iy n r ay n aw\nFigure H.24 Phonetic transcription of the Switchboard phrase they\u2019re kind of in between\nright now . Note vowel reduction in they\u2019re andof, coda deletion in kind andright , and re-\nsyllabi\ufb01cation (the [v] of ofattaches as the onset of in). Time is given in number of seconds\nfrom the beginning of sentence to the start of each syllable.\nThe Buckeye corpus (Pitt et al. 2007, Pitt et al. 2005) is a phonetically tran-\nscribed corpus of spontaneous American speech, containing about 300,000 words\nfrom 40 talkers. Phonetically transcribed corpora are also available for other lan-\nguages, including the Kiel corpus of German and Mandarin corpora transcribed by\nthe Chinese Academy of Social Sciences (Li et al., 2000).\nIn addition to resources like dictionaries and corpora, there are many useful pho-\nnetic software tools. Many of the \ufb01gures in this book were generated by the Praat\npackage (Boersma and Weenink, 2005), which includes pitch, spectral, and formant\nanalysis, as well as a scripting language. H.6 \u2022 S UMMARY 21\nH.6 Summary\nThis chapter has introduced many of the important concepts of phonetics and com-\nputational phonetics.\n\u2022 We can represent the pronunciation of words in terms of units called phones .\nThe standard system for representing phones is the International Phonetic\nAlphabet orIPA. The most common computational system for transcription\nof English is the ARPAbet , which conveniently uses ASCII symbols.\n\u2022 Phones can be described by how they are produced articulatorily by the vocal\norgans; consonants are de\ufb01ned in terms of their place andmanner of articu-\nlation and voicing ; vowels by their height ,backness , and roundness .\n\u2022 Speech sounds can also be described acoustically . Sound waves can be de-\nscribed in terms of frequency ,amplitude , or their perceptual correlates, pitch\nandloudness .\n\u2022 The spectrum of a sound describes its different frequency components. While\nsome phonetic properties are recognizable from the waveform, both humans\nand machines rely on spectral analysis for phone detection.\n\u2022 A spectrogram is a plot of a spectrum over time. V owels are described by\ncharacteristic harmonics called formants .\nBibliographical and Historical Notes\nThe major insights of articulatory phonetics date to the linguists of 800\u2013150 B.C.\nIndia. They invented the concepts of place and manner of articulation, worked out\nthe glottal mechanism of voicing, and understood the concept of assimilation. Eu-\nropean science did not catch up with the Indian phoneticians until over 2000 years\nlater, in the late 19th century. The Greeks did have some rudimentary phonetic\nknowledge; by the time of Plato\u2019s Theaetetus andCratylus , for example, they distin-\nguished vowels from consonants, and stop consonants from continuants. The Stoics\ndeveloped the idea of the syllable and were aware of phonotactic constraints on pos-\nsible words. An unknown Icelandic scholar of the 12th century exploited the concept\nof the phoneme and proposed a phonemic writing system for Icelandic, including\ndiacritics for length and nasality. But his text remained unpublished until 1818 and\neven then was largely unknown outside Scandinavia (Robins, 1967). The modern\nera of phonetics is usually said to have begun with Sweet, who proposed what is\nessentially the phoneme in his Handbook of Phonetics 1877. He also devised an al-\nphabet for transcription and distinguished between broad andnarrow transcription,\nproposing many ideas that were eventually incorporated into the IPA. Sweet was\nconsidered the best practicing phonetician of his time; he made the \ufb01rst scienti\ufb01c\nrecordings of languages for phonetic purposes and advanced the state of the art of\narticulatory description. He was also infamously dif\ufb01cult to get along with, a trait\nthat is well captured in Henry Higgins, the stage character that George Bernard Shaw\nmodeled after him. The phoneme was \ufb01rst named by the Polish scholar Baudouin\nde Courtenay, who published his theories in 1894.\nIntroductory phonetics textbooks include Ladefoged (1993) and Clark and Yal-\nlop (1995). Wells (1982) is the de\ufb01nitive three-volume source on dialects of English.\nMany of the classic insights in acoustic phonetics had been developed by the\nlate 1950s or early 1960s; just a few highlights include techniques like the sound 22 APPENDIX H \u2022 P HONETICS\nspectrograph (Koenig et al., 1946), theoretical insights like the working out of the\nsource-\ufb01lter theory and other issues in the mapping between articulation and acous-\ntics ((Fant, 1960), Stevens et al. 1953, Stevens and House 1955, Heinz and Stevens\n1961, Stevens and House 1961) the F1xF2 space of vowel formants (Peterson and\nBarney, 1952), the understanding of the phonetic nature of stress and the use of\nduration and intensity as cues (Fry, 1955), and a basic understanding of issues in\nphone perception (Miller and Nicely 1955,Liberman et al. 1952). Lehiste (1967) is\na collection of classic papers on acoustic phonetics. Many of the seminal papers of\nGunnar Fant have been collected in Fant (2004).\nExcellent textbooks on acoustic phonetics include Johnson (2003) and Lade-\nfoged (1996). Coleman (2005) includes an introduction to computational process-\ning of acoustics and speech from a linguistic perspective. Stevens (1998) lays out\nan in\ufb02uential theory of speech sound production. There are a number of software\npackages for acoustic phonetic analysis. Probably the most widely used one is Praat\n(Boersma and Weenink, 2005).\nExercises\nH.1 Find the mistakes in the ARPAbet transcriptions of the following words:\na.\u201cthree\u201d [dh r i] d.\u201cstudy\u201d [s t uh d i] g.\u201cslight\u201d [s l iy t]\nb.\u201csing\u201d [s ih n g] e.\u201cthough\u201d [th ow]\nc.\u201ceyes\u201d [ay s] f.\u201cplanning\u201d [p pl aa n ih ng]\nH.2 Ira Gershwin\u2019s lyric for Let\u2019s Call the Whole Thing Off talks about two pro-\nnunciations (each) of the words \u201ctomato\u201d, \u201cpotato\u201d, and \u201ceither\u201d. Transcribe\ninto the ARPAbet both pronunciations of each of these three words.\nH.3 Transcribe the following words in the ARPAbet:\n1. dark\n2. suit\n3. greasy\n4. wash\n5. water\nH.4 Take a wave\ufb01le of your choice. Some examples are on the textbook website.\nDownload the Praat software, and use it to transcribe the wave\ufb01les at the word\nlevel and into ARPAbet phones, using Praat to help you play pieces of each\nwave\ufb01le and to look at the wave\ufb01le and the spectrogram.\nH.5 Record yourself saying \ufb01ve of the English vowels: [aa], [eh], [ae], [iy], [uw].\nFind F1 and F2 for each of your vowels. Exercises 23\nBaayen, R. H., R. Piepenbrock, and L. Gulikers. 1995.\nThe CELEX Lexical Database (Release 2) [CD-ROM] .\nLinguistic Data Consortium, University of Pennsylvania\n[Distributor].\nBeckman, M. E. and G. M. Ayers. 1997. Guidelines\nfor ToBI labelling. Unpublished manuscript, Ohio\nState University, http://www.ling.ohio-state.\nedu/research/phonetics/E_ToBI/ .\nBeckman, M. E. and J. Hirschberg. 1994. The ToBI annota-\ntion conventions. Manuscript, Ohio State University.\nBennett, R. and E. Elfner. 2019. The syntax\u2013prosody inter-\nface. Annual Review of Linguistics , 5:151\u2013171.\nBoersma, P. and D. Weenink. 2005. Praat: doing phonetics\nby computer (version 4.3.14). [Computer program]. Re-\ntrieved May 26, 2005, from http://www.praat.org/ .\nClark, J. and C. Yallop. 1995. An Introduction to Phonetics\nand Phonology , 2nd edition. Blackwell.\nCMU. 1993. The Carnegie Mellon Pronouncing Dictionary\nv0.1. Carnegie Mellon University.\nColeman, J. 2005. Introducing Speech and Language Pro-\ncessing . Cambridge University Press.\nFant, G. M. 1960. Acoustic Theory of Speech Production .\nMouton.\nFant, G. M. 2004. Speech Acoustics and Phonetics . Kluwer.\nFitt, S. 2002. Unisyn lexicon. http://www.cstr.ed.ac.\nuk/projects/unisyn/ .\nFry, D. B. 1955. Duration and intensity as physical correlates\nof linguistic stress. JASA , 27:765\u2013768.\nGreenberg, S., D. Ellis, and J. Hollenback. 1996. Insights\ninto spoken language gleaned from phonetic transcription\nof the Switchboard corpus. ICSLP .\nHeinz, J. M. and K. N. Stevens. 1961. On the properties of\nvoiceless fricative consonants. JASA , 33:589\u2013596.\nJohnson, K. 2003. Acoustic and Auditory Phonetics , 2nd\nedition. Blackwell.\nKoenig, W., H. K. Dunn, and L. Y . Lacy. 1946. The sound\nspectrograph. JASA , 18:19\u201349.\nLadefoged, P. 1993. A Course in Phonetics . Harcourt Brace\nJovanovich. (3rd ed.).\nLadefoged, P. 1996. Elements of Acoustic Phonetics , 2nd\nedition. University of Chicago.\nLehiste, I., ed. 1967. Readings in Acoustic Phonetics . MIT\nPress.\nLi, A., F. Zheng, W. Byrne, P. Fung, T. Kamm, L. Yi,\nZ. Song, U. Ruhi, V . Venkataramani, and X. Chen. 2000.\nCASS: A phonetically transcribed corpus of Mandarin\nspontaneous speech. ICSLP .\nLiberman, A. M., P. C. Delattre, and F. S. Cooper. 1952. The\nrole of selected stimulus variables in the perception of the\nunvoiced stop consonants. American Journal of Psychol-\nogy, 65:497\u2013516.\nMiller, G. A. and P. E. Nicely. 1955. An analysis of percep-\ntual confusions among some English consonants. JASA ,\n27:338\u2013352.\nNIST. 1990. TIMIT Acoustic-Phonetic Continuous Speech\nCorpus. National Institute of Standards and Technology\nSpeech Disc 1-1.1. NIST Order No. PB91-505065.Ostendorf, M., P. Price, and S. Shattuck-Hufnagel. 1995. The\nBoston University Radio News Corpus. Technical Report\nECS-95-001, Boston University.\nPeterson, G. E. and H. L. Barney. 1952. Control methods\nused in a study of the vowels. JASA , 24:175\u2013184.\nPitt, M. A., L. Dilley, K. Johnson, S. Kiesling, W. D. Ray-\nmond, E. Hume, and E. Fosler-Lussier. 2007. Buckeye\ncorpus of conversational speech (2nd release). Depart-\nment of Psychology, Ohio State University (Distributor).\nPitt, M. A., K. Johnson, E. Hume, S. Kiesling, and W. D.\nRaymond. 2005. The buckeye corpus of conversational\nspeech: Labeling conventions and a test of transcriber re-\nliability. Speech Communication , 45:90\u201395.\nPrice, P. J., M. Ostendorf, S. Shattuck-Hufnagel, and\nC. Fong. 1991. The use of prosody in syntactic disam-\nbiguation. JASA , 90(6).\nRobins, R. H. 1967. A Short History of Linguistics . Indiana\nUniversity Press, Bloomington.\nSeneff, S. and V . W. Zue. 1988. Transcription and align-\nment of the TIMIT database. Proceedings of the Second\nSymposium on Advanced Man-Machine Interface through\nSpoken Language .\nShoup, J. E. 1980. Phonological aspects of speech recogni-\ntion. In W. A. Lea, ed., Trends in Speech Recognition ,\n125\u2013138. Prentice Hall.\nSilverman, K., M. E. Beckman, J. F. Pitrelli, M. Ostendorf,\nC. W. Wightman, P. J. Price, J. B. Pierrehumbert, and\nJ. Hirschberg. 1992. ToBI: A standard for labelling En-\nglish prosody. ICSLP .\nStevens, K. N. 1998. Acoustic Phonetics . MIT Press.\nStevens, K. N. and A. S. House. 1955. Development of\na quantitative description of vowel articulation. JASA ,\n27:484\u2013493.\nStevens, K. N. and A. S. House. 1961. An acoustical theory\nof vowel production and some of its implications. Journal\nof Speech and Hearing Research , 4:303\u2013320.\nStevens, K. N., S. Kasowski, and G. M. Fant. 1953. An elec-\ntrical analog of the vocal tract. JASA , 25(4):734\u2013742.\nStevens, S. S. and J. V olkmann. 1940. The relation of pitch\nto frequency: A revised scale. The American Journal of\nPsychology , 53(3):329\u2013353.\nStevens, S. S., J. V olkmann, and E. B. Newman. 1937. A\nscale for the measurement of the psychological magni-\ntude pitch. JASA , 8:185\u2013190.\nSweet, H. 1877. A Handbook of Phonetics . Clarendon Press.\nWells, J. C. 1982. Accents of English . Cambridge University\nPress.\nXu, Y . 2005. Speech melody as articulatorily implemented\ncommunicative functions. Speech communication , 46(3-\n4):220\u2013251."}