|
|
--- |
|
|
language: |
|
|
- nl |
|
|
tags: |
|
|
- kenlm |
|
|
license: apache-2.0 |
|
|
--- |
|
|
# KenLM (arpa) models for Dutch based on SONAR |
|
|
|
|
|
This repository contains KenLM models (n=5) for Dutch, based on the SONAR corpus - sentence-segmented (one sentence per line). Models are provided on tokens, part-of-speech, dependency labels, and lemmas, as processed with spaCy `nl_core_news_sm`: |
|
|
|
|
|
- kenlm_sonar_token.arpa[.bin]: token |
|
|
- kenlm_sonar_pos.arpa[.bin]: part-of-speech tag |
|
|
- kenlm_sonar_dep.arpa[.bin]: dependency label |
|
|
- kenlm_sonar_lemma.arpa[.bin]: lemma |
|
|
|
|
|
More noisy SONAR components (WRPEA, WRPED, WRUEA, WRUED, WRUEB) were excluded. |
|
|
|
|
|
Both regular `.arpa` files as well as more efficient KenLM binary files (`.arpa.bin`) are provided. You probably want to use the binary versions. |
|
|
|
|
|
## Usage from within Python |
|
|
|
|
|
Make sure to install dependencies: |
|
|
|
|
|
```shell |
|
|
pip install huggingface_hub |
|
|
pip install https://github.com/kpu/kenlm/archive/master.zip |
|
|
|
|
|
# If you want to use spaCy preprocessing |
|
|
pip install spacy |
|
|
python -m spacy download nl_core_news_sm |
|
|
``` |
|
|
|
|
|
We can then use the Hugging Face hub software to download and cache the model file that we want, and directly use it with KenLM. |
|
|
|
|
|
```python |
|
|
import kenlm |
|
|
from huggingface_hub import hf_hub_download |
|
|
|
|
|
model_file = hf_hub_download(repo_id="BramVanroy/kenlm_sonar", filename="kenlm_sonar_token.arpa.bin") |
|
|
model = kenlm.Model(model_file) |
|
|
|
|
|
text = "Ik eet graag koekjes !" # pre-tokenized |
|
|
model.perplexity(text) |
|
|
# 148.21996373689134 |
|
|
``` |
|
|
|
|
|
It is recommended to use spaCy as a preprocessor to automatically use the same tagsets and tokenization as were used when creating the LMs. |
|
|
|
|
|
|
|
|
```python |
|
|
import kenlm |
|
|
import spacy |
|
|
from huggingface_hub import hf_hub_download |
|
|
|
|
|
model_file = hf_hub_download(repo_id="BramVanroy/kenlm_sonar", filename="kenlm_sonar_pos.arpa.bin") # pos file |
|
|
model = kenlm.Model(model_file) |
|
|
|
|
|
nlp = spacy.load("nl_core_news_sm") |
|
|
|
|
|
text = "Ik eet graag koekjes!" |
|
|
pos_sequence = " ".join([token.pos_ for token in nlp(text)]) |
|
|
# 'PRON VERB ADV NOUN PUNCT' |
|
|
model.perplexity(pos_sequence) |
|
|
# 6.916279238079976 |
|
|
``` |
|
|
|
|
|
|
|
|
## Reproduction |
|
|
|
|
|
```sh |
|
|
bin/lmplz -o 5 -S 75% -T ../data/tmp/ < ../data/processed_sonar_token_dedup.txt > ../data/kenlm_sonar_token.arpa |
|
|
``` |
|
|
|
|
|
For class-based LMs (POS and DEP), the `--discount_fallback` was used and the parsed data was not deduplicated (but it was deduplicated on the sentence-level for token and lemma models). |
