course/pr_1213/en/chapter6/10.md

# End-of-chapter quiz[[end-of-chapter-quiz]]

Let's test what you learned in this chapter!

### 1. When should you train a new tokenizer?

### 2. What is the advantage of using a generator of lists of texts compared to a list of lists of texts when using `train_new_from_iterator()`?

train_new_from_iterator() accepts.",
			explain: "A list of lists of texts is a particular kind of generator of lists of texts, so the method will accept this too. Try again!"
		},
		{
			text: "You will avoid loading the whole dataset into memory at once.",
			explain: "Right! Each batch of texts will be released from memory when you iterate, and the gain will be especially visible if you use 🤗 Datasets to store your texts.",
			correct: true
		},
		{
			text: "This will allow the 🤗 Tokenizers library to use multiprocessing.",
			explain: "No, it will use multiprocessing either way."
		},
        {
			text: "The tokenizer you train will generate better texts.",
			explain: "The tokenizer does not generate text -- are you confusing it with a language model?"
		}
	]}
/>

### 3. What are the advantages of using a "fast" tokenizer?

### 4. How does the `token-classification` pipeline handle entities that span over several tokens?

### 5. How does the `question-answering` pipeline handle long contexts?

### 6. What is normalization?

### 7. What is pre-tokenization for a subword tokenizer?

### 8. Select the sentences that apply to the BPE model of tokenization.

### 9. Select the sentences that apply to the WordPiece model of tokenization.

### 10. Select the sentences that apply to the Unigram model of tokenization.