Aksharakuppy — English ↔ Malayalam Translator

A from-scratch Transformer neural machine translation model for English ↔ Malayalam, built and released by endurasolution. A single model translates both directions using direction tags.

Powered by Aksharakuppy.

Models

Two checkpoints are included:

Model	Description
`best.pt`	Base model trained from scratch on ~7.4M En–Ml sentence pairs
`finetunedcorrected.pt`	Base model fine-tuned on BPCC human-annotated data + curated corrections (recommended)

Architecture: 6-layer encoder-decoder Transformer (~52M params), d_model=512
Tokenizer: SentencePiece BPE, 16k vocab, shared En+Ml
Direction control: <2ml> (to Malayalam), <2en> (to English)

Usage

Install dependencies:

bash pip install torch sentencepiece

Translate:

```python import torch, sentencepiece as spm from model import MTModel from config import CFG

DEVICE = "cuda" if torch.cuda.is_available() else "cpu" sp = spm.SentencePieceProcessor(model_file="checkpoints/spm.model") PAD, BOS, EOS = 0, 1, 2 ML_TAG = sp.piece_to_id("<2ml>"); EN_TAG = sp.piece_to_id("<2en>")

model = MTModel(sp.get_piece_size(), CFG.d_model, CFG.nhead, CFG.layers, CFG.dim_ff, dropout=0.0, max_len=CFG.max_len).to(DEVICE) sd = torch.load("checkpoints/finetunedcorrected.pt", map_location=DEVICE)["model"] model.load_state_dict({k.replace("_orig_mod.", ""): v for k, v in sd.items()}) model.eval()

@torch.no_grad() def translate(text, to="ml"): tag = ML_TAG if to == "ml" else EN_TAG ids = [BOS, tag] + sp.encode(text, out_type=int)[:CFG.max_len-2] + [EOS] src = torch.tensor([ids], device=DEVICE) ys = torch.tensor([[BOS]], device=DEVICE) for _ in range(128): nxt = model(src, ys)[0, -1].argmax().item() ys = torch.cat([ys, torch.tensor([[nxt]], device=DEVICE)], 1) if nxt == EOS: break out = [i for i in ys[0].tolist() if i not in (PAD, BOS, EOS, ML_TAG, EN_TAG)] return sp.decode(out)

print(translate("The weather is nice today.", "ml")) print(translate("എനിക്ക് വിശക്കുന്നു.", "en")) ```

Web demo

bash pip install fastapi uvicorn uvicorn translate_server:app --host 0.0.0.0 --port 8081 Open http://localhost:8081 — includes a model selector and side-by-side comparison.

Limitations

Trained from scratch on limited data; conveys meaning well on everyday sentences but may occasionally add or drop a word (e.g. inferring "morning" from context) or pick a synonym. Not intended to match large pretrained systems.
Proper nouns / names may vary in spelling.

License & credits

Model weights: CC-BY-NC-4.0 (non-commercial), following the Samanantar data license. Training data: AI4Bharat Samanantar (CC-BY-NC-4.0) and BPCC (CC0 for mined/seed subsets), plus a custom corpus. Please cite AI4Bharat for the underlying data.