| license: mit | |
| library_name: fasttext | |
| pipeline_tag: text-classification | |
| This is the fastText pretraining data filter targeting the LAMBADA DE task, discussed in the main text of the Perplexity Correlations paper: https://arxiv.org/abs/2409.05816 | |
| Code: https://github.com/TristanThrush/perplexity-correlations |