Language Detector (TF-IDF char n-grams + linear SVM Classifier)
Module 1 of the RAG-Based Mental Health Support Chatbot. Predicts the language
code of a short text (e.g. en, fr, ar).
Files
vectorizer.joblib— fittedTfidfVectorizer(character n-grams)classifier.joblib— fitted linear classifier
Usage
import joblib
from huggingface_hub import hf_hub_download
vec = joblib.load(hf_hub_download("GAD01/svm-language-detector", "vectorizer.joblib"))
clf = joblib.load(hf_hub_download("GAD01/svm-language-detector", "classifier.joblib"))
print(clf.predict(vec.transform(["Bonjour, comment allez-vous ?"]))[0]) # -> "fr"
Trained on papluca/language-identification.
- Downloads last month
- -
from huggingface_hub import hf_hub_download import joblib model = joblib.load( hf_hub_download("GAD01/svm-language-detector", "sklearn_model.joblib") ) # only load pickle files from sources you trust # read more about it here https://skops.readthedocs.io/en/stable/persistence.html