Jyotiprakash4357/language-detection-lstm

Language Detection LSTM

Character-level Bidirectional LSTM trained on papluca/language-identification.

Supports 20 Languages

Arabic, Bulgarian, German, Greek, English, Spanish, French, Hindi, Italian, Japanese, Dutch, Polish, Portuguese, Russian, Swahili, Thai, Turkish, Urdu, Vietnamese, Chinese

Performance

• Val Accuracy: ~98.9%
• Test Accuracy: ~98%+

Usage

import torch, pickle
from huggingface_hub import hf_hub_download

char2idx = pickle.load(open(hf_hub_download("Jyotiprakash4357/language-detection-lstm", "char2idx.pkl"), "rb"))
le       = pickle.load(open(hf_hub_download("Jyotiprakash4357/language-detection-lstm", "label_encoder.pkl"), "rb"))

model = LangLSTM(len(char2idx), 64, 256, 2, len(le.classes_))
model.load_state_dict(torch.load(hf_hub_download("Jyotiprakash4357/language-detection-lstm", "lang_lstm.pt"), map_location="cpu"))

predict("Bonjour le monde")  # → [('fr', 0.99), ...]