Ransaka/sinhala-n-gram-collection

Sinhala N-gram Language Model

A token-level 5-gram language model for Sinhala, trained on 180K sentences (8.7M tokens). Uses Stupid Backoff scoring. Pure Python implementation.

Files

File	Description
sinhala_5gram.json.gz	Compressed 5-gram model (gzipped JSON, 13 MB)
tokenizer/vocab.json	Sinlib tokenizer vocabulary (850 subword tokens)
tokenizer/config.json	Tokenizer configuration

Model Stats

Metric	Value
Order	5
Vocabulary	850 phonological subword tokens
Training tokens	8,681,739
Unique 1-grams	847
Unique 2-grams	63,325
Unique 3-grams	409,674
Unique 4-grams	927,050
Unique 5-grams	1,341,620

Usage

Loading the model

import gzip
import json
import math


class NgramLM:
    \"\"\"Lightweight n-gram LM with Stupid Backoff scoring.\"\"\"

    def __init__(self, path: str):
        with gzip.open(path, "rt", encoding="utf-8") as f:
            data = json.load(f)
        self.order = data["order"]
        self.total_unigrams = data["total_unigrams"]
        self.counts = [{}] * (self.order + 1)
        for n_str, ngram_counts in data["counts"].items():
            n = int(n_str)
            self.counts[n] = {
                tuple(k.split(" ")): v for k, v in ngram_counts.items()
            }

    def score(self, context: list[str], token: str) -> float:
        \"\"\"Return log10 P(token | context) using Stupid Backoff.\"\"\"
        for n in range(min(len(context) + 1, self.order), 0, -1):
            if n == 1:
                count = self.counts[1].get((token,), 0)
                if count > 0:
                    return math.log10(count / self.total_unigrams)
                return math.log10(1.0 / (self.total_unigrams + 1))
            ngram = tuple(context[-(n - 1):]) + (token,)
            ctx = tuple(context[-(n - 1):])
            nc = self.counts[n].get(ngram, 0)
            cc = self.counts[n - 1].get(ctx, 0)
            if nc > 0 and cc > 0:
                return math.log10(nc / cc)
        return math.log10(1.0 / (self.total_unigrams + 1))

    def score_sequence(self, tokens: list[str]) -> float:
        \"\"\"Score a full token sequence. Returns sum of log10 probs.\"\"\"
        context = ["<s>"]
        total = 0.0
        for t in tokens:
            total += self.score(context, t)
            context.append(t)
        return total

Scoring a token sequence

lm = NgramLM("sinhala_5gram.json.gz")

# Tokens are string representations of sinlib token IDs
tokens = ["166", "96", "433", "28"]
log_prob = lm.score_sequence(tokens)
print(f"Sequence log10-prob: {log_prob:.4f}")

# Score next token given context
context = ["<s>", "166", "96"]
next_token_score = lm.score(context, "433")
print(f"P(433 | context): {10**next_token_score:.6f}")

Tokenizing text with sinlib

from sinlib import Tokenizer

tokenizer = Tokenizer.from_pretrained("Ransaka/sinlib")
encoding = tokenizer("සිංහල වාක්\u200dය", add_bos_token=False, truncate_and_pad=False)
token_ids = [str(tid) for tid in encoding.input_ids]

score = lm.score_sequence(token_ids)

Training Details

• Tokenizer: sinlib phonological subword tokenizer
• Scoring: Stupid Backoff (Brants et al., 2007)
• Implementation: Python