fasttext-language-detection-ms-id

FastText model to classify standard Malay, local Malay, local Indonesian and others.

how to use

from huggingface_hub import hf_hub_download
import fasttext

filename = hf_hub_download(
    repo_id="mesolitica/fasttext-language-detection-ms-id", 
    filename="fasttext.ftz"
)
lang_model = fasttext.load_model(filename)
lang_model.predict('hello name saye', k = 10)

Output,

(('__label__standard-malay',
  '__label__other',
  '__label__socialmedia-indonesian',
  '__label__local-malay'),
 array([8.16933990e-01, 1.58563450e-01, 2.42817383e-02, 2.62394664e-04]))