| --- |
| language: |
| - zh |
| license: "apache-2.0" |
| pipeline_tag: "fill-mask" |
| --- |
| |
| **Please use `ElectraForPreTraining` for `discriminator` and `ElectraForMaskedLM` for `generator` if you are re-training these models.** |
|
|
| ## Chinese ELECTRA |
| Google and Stanford University released a new pre-trained model called ELECTRA, which has a much compact model size and relatively competitive performance compared to BERT and its variants. |
| For further accelerating the research of the Chinese pre-trained model, the Joint Laboratory of HIT and iFLYTEK Research (HFL) has released the Chinese ELECTRA models based on the official code of ELECTRA. |
| ELECTRA-small could reach similar or even higher scores on several NLP tasks with only 1/10 parameters compared to BERT and its variants. |
|
|
| This project is based on the official code of ELECTRA: [https://github.com/google-research/electra](https://github.com/google-research/electra) |
|
|
| You may also interested in, |
| - Chinese BERT series: https://github.com/ymcui/Chinese-BERT-wwm |
| - Chinese ELECTRA: https://github.com/ymcui/Chinese-ELECTRA |
| - Chinese XLNet: https://github.com/ymcui/Chinese-XLNet |
| - Knowledge Distillation Toolkit - TextBrewer: https://github.com/airaria/TextBrewer |
|
|
| More resources by HFL: https://github.com/ymcui/HFL-Anthology |
|
|
|
|
| ## Citation |
| If you find our resource or paper is useful, please consider including the following citation in your paper. |
| - https://arxiv.org/abs/2004.13922 |
| ``` |
| @inproceedings{cui-etal-2020-revisiting, |
| title = "Revisiting Pre-Trained Models for {C}hinese Natural Language Processing", |
| author = "Cui, Yiming and |
| Che, Wanxiang and |
| Liu, Ting and |
| Qin, Bing and |
| Wang, Shijin and |
| Hu, Guoping", |
| booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings", |
| month = nov, |
| year = "2020", |
| address = "Online", |
| publisher = "Association for Computational Linguistics", |
| url = "https://www.aclweb.org/anthology/2020.findings-emnlp.58", |
| pages = "657--668", |
| } |
| ``` |
|
|
