bashyaldhiraj2067/nepali-chatbot-embeddings

🇳🇵 Nepali Chatbot Embeddings Model

A Transformer trained from scratch on 40,000 Nepali QA pairs.
Produces 128-dimensional sentence embeddings for Nepali (Devanagari) text,
used for semantic retrieval in a chatbot system.

Model Details

Property	Value
Architecture	Transformer Encoder-Decoder
Embedding dim	128
Attention heads	4
FF dim	256
Layers	2
Tokenizer	Character-level (Devanagari)
Vocab size	127
Max seq len	64
Language	Nepali (ne)
Dataset	bashyaldhiraj2067/50k_nepali_chatbot_dataset

Evaluation Results (Autoregressive — No Teacher Forcing)

Metric	Score
Token Accuracy	79.32%
Precision (macro)	82.11%
Recall (macro)	75.21%
F1 (macro)	77.69%
BLEU-1 (corpus)	0.9315
BLEU-2 (corpus)	0.9040
BLEU-3 (corpus)	0.8845
BLEU-4 (corpus)	0.8703
ROUGE-1	0.1976
ROUGE-2	0.1108
ROUGE-L	0.1976

Training Curves

Files

File	Description
pytorch_model.bin	Full model weights (PyTorch)
config.json	Model architecture & eval config
tokenizer.json	Character-level vocab (token2id)
embeddings_index.pt	Pre-computed embeddings + QA pairs for retrieval
training_curves.png	Loss / Accuracy / BLEU / ROUGE plots

Usage

import torch, json
import torch.nn.functional as F

# 1. Load config
with open("config.json") as f:
    cfg = json.load(f)

# 2. Load tokenizer
with open("tokenizer.json") as f:
    tok_data = json.load(f)
token2id = tok_data["token2id"]

def encode(text, max_len=64):
    ids = [1] + [token2id.get(c, 3) for c in list(text.strip())] + [2]
    ids = ids[:max_len]
    ids += [0] * (max_len - len(ids))
    return ids

# 3. Load pre-built index
index = torch.load("embeddings_index.pt", map_location="cpu")
all_embs = index["embeddings"]   # (N, 128) normalized
questions = index["questions"]
answers   = index["answers"]

# 4. Query
def get_answer(query):
    ids  = torch.tensor([encode(query)], dtype=torch.long)
    # (For full model inference, load pytorch_model.bin + TransformerChatbot class)
    # Quick cosine retrieval using index:
    # ... embed query with model.get_embedding(ids) ...
    # sims = (all_embs @ q_emb.T).squeeze()
    # return answers[sims.argmax()]
    pass

License

MIT