| --- |
| language: |
| - en |
| thumbnail: https://github.com/karanchahal/distiller/blob/master/distiller.jpg |
| tags: |
| - question-answering |
| license: apache-2.0 |
| datasets: |
| - squad |
| metrics: |
| - squad |
| --- |
| |
| # DistilBERT with a second step of distillation |
|
|
| ## Model description |
|
|
| This model replicates the "DistilBERT (D)" model from Table 2 of the [DistilBERT paper](https://arxiv.org/pdf/1910.01108.pdf). In this approach, a DistilBERT student is fine-tuned on SQuAD v1.1, but with a BERT model (also fine-tuned on SQuAD v1.1) acting as a teacher for a second step of task-specific distillation. |
|
|
| In this version, the following pre-trained models were used: |
|
|
| * Student: `distilbert-base-uncased` |
| * Teacher: `lewtun/bert-base-uncased-finetuned-squad-v1` |
|
|
| ## Training data |
|
|
| This model was trained on the SQuAD v1.1 dataset which can be obtained from the `datasets` library as follows: |
|
|
| ```python |
| from datasets import load_dataset |
| squad = load_dataset('squad') |
| ``` |
|
|
| ## Training procedure |
|
|
| ## Eval results |
|
|
| | | Exact Match | F1 | |
| |------------------|-------------|------| |
| | DistilBERT paper | 79.1 | 86.9 | |
| | Ours | 78.4 | 86.5 | |
|
|
| The scores were calculated using the `squad` metric from `datasets`. |
|
|
| ### BibTeX entry and citation info |
|
|
| ```bibtex |
| @misc{sanh2020distilbert, |
| title={DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter}, |
| author={Victor Sanh and Lysandre Debut and Julien Chaumond and Thomas Wolf}, |
| year={2020}, |
| eprint={1910.01108}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CL} |
| } |
| ``` |
