Moonshine-Base-BN: Bengali ASR via Tokenizer Transplantation

moonshine-base-bn is a 61.5M-parameter Bengali Automatic Speech Recognition (ASR) model built by adapting UsefulSensors/Moonshine-Base through a novel tokenizer transplantation pipeline. The model's original English-centric byte-level decoder vocabulary was surgically replaced with the native-script BanglaBERT WordPiece vocabulary, resolving the autoregressive collapse that high-fertility byte tokenization causes on morphologically rich languages like Bengali.

This is the official model release accompanying the paper:

Tokenizer Transplantation: Mitigating Autoregressive Collapse in Edge-Efficient Bengali ASR Sanjid Hasan, Md. Abdur Rahman — MuslimML Workshop @ ICML 2026

Why Tokenizer Transplantation?

Lightweight ASR models like Moonshine are optimized for fast, offline, edge deployment — but their English-centric tokenizers fragment Bengali words into long byte chains. This high tokenizer fertility (tokens-per-word) causes the autoregressive decoder to drift and collapse during inference, even when teacher-forced training loss looks fine.

Tokenizer	Fertility (Φ)	Sequence Length
Original Moonshine (byte-fallback)	9.16	—
Transplanted (BanglaBERT WordPiece)	1.30	85.8% shorter

By replacing the vocabulary with a native Bengali WordPiece tokenizer and re-aligning the model through a two-stage recovery schedule, decoding instability is fully resolved.

Performance

Evaluated on the held-out test split of the Lipi-Ghor-bn-882-SSTT dataset (882 hours, multi-speaker, multi-domain Bengali speech):

Model	Params	WER (%)	CER (%)	RTF
Seamless M4T-v2 (zero-shot)	~2.3B	66.71	45.54	—
Whisper large-v3 (zero-shot)	~1.55B	84.53	72.92	—
Meta MMS 1B (zero-shot)	~1B	43.06	21.16	—
Hishab TITU Conformer Large (zero-shot)	~120M	30.51	18.23	—
Conformer Baseline (fine-tuned)	~120M	24.67	15.56	0.0120
Faster Whisper Medium (fine-tuned)	~769M	21.28	11.18	0.0190
Moonshine-Base-BN (this model)	~61.5M	21.54	10.79	0.0053

This model achieves the lowest CER among all tested architectures and matches the WER of a model 12x its size, while running natively ~3.5x faster than engineered Whisper CTranslate2 pipelines.

Model Architecture

This model is built by surgically replacing the decoder vocabulary of Moonshine-Base with the native-script BanglaBERT WordPiece vocabulary, then re-aligning the decoder through a recovery fine-tuning schedule. Full methodology is detailed in the paper (Section 4).

Usage

Record an audio in .wav format and replace the path with sample.wav!

# Ensure you have librosa installed
# !pip install -q librosa

import torch
import librosa
import numpy as np
from transformers import AutoTokenizer, AutoModelForSpeechSeq2Seq

# ═══════════════════════════════════════════════════════════════════════════════
# CONFIG & PATHS
# ═══════════════════════════════════════════════════════════════════════════════
REPO_ID = "Sanjidh090/moonshine-base-bn"
AUDIO_PATH = "sample.wav"

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if torch.cuda.is_available() else torch.float32

print(f"📡 Calling Model from Hugging Face: {REPO_ID}")

# 1. Load Tokenizer & Model
tokenizer = AutoTokenizer.from_pretrained(REPO_ID)
model = AutoModelForSpeechSeq2Seq.from_pretrained(
    REPO_ID, 
    torch_dtype=dtype, 
    low_cpu_mem_usage=True
).to(device)

# 2. Extract Token IDs (Fallback to your verified defaults if missing)
START_ID = tokenizer.cls_token_id or 2
EOS_ID   = tokenizer.sep_token_id or 3
PAD_ID   = tokenizer.pad_token_id or 0

model.eval()

# ═══════════════════════════════════════════════════════════════════════════════
# INFERENCE FUNCTION
# ═══════════════════════════════════════════════════════════════════════════════
def transcribe(audio_file):
    print("⏳ Processing audio...")
    # Load and resample to 16kHz
    audio, _ = librosa.load(audio_file, sr=16000)
    
    # Pad to multiple of 160
    remainder = len(audio) % 320
    if remainder:
        audio = np.concatenate([audio, np.zeros(320 - remainder, dtype=np.float32)])
    
    # Cast the tensor to matching global precision
    input_values = torch.tensor(audio).unsqueeze(0).to(device, dtype=dtype)

    print("🔮 Running inference pipeline...")
    with torch.no_grad():
        generated_ids = model.generate(
            input_values,
            max_new_tokens=2000,
            num_beams=5,
            no_repeat_ngram_size=3,
            repetition_penalty=1.2,
            decoder_start_token_id=START_ID,
            pad_token_id=PAD_ID,
            eos_token_id=EOS_ID
        )

    # 3. Calculate Token Stats
    output_tokens_len = generated_ids.shape[1]

    # Decode
    transcription = tokenizer.decode(generated_ids[0].tolist(), skip_special_tokens=True)
    
    return transcription, output_tokens_len

# Run it!
if __name__ == "__main__":
    try:
        result, token_count = transcribe(AUDIO_PATH)
        print(f"\n📝 Transcription:\n{result}")
        print(f"\n--- Token Statistics ---")
        print(f"Generated Output Text Tokens: {token_count} tokens")
    except Exception as e:
        print(f"❌ Error during inference execution: {e}")

Citation

@inproceedings{hasan2026tokenizer,
  title={Tokenizer Transplantation: Mitigating Autoregressive Collapse in Edge-Efficient Bengali ASR},
  author={Hasan, Sanjid and Rahman, Md. Abdur},
  booktitle={MuslimML Workshop at the 43rd International Conference on Machine Learning (ICML)},
  year={2026}
}

Paper on Arxiv...

@misc{hasan2026tokenizertransplantationmitigatingautoregressive,
      title={Tokenizer Transplantation: Mitigating Autoregressive Collapse in Edge-Efficient Bengali ASR}, 
      author={Sanjid Hasan and Md. Abdur Rahman},
      year={2026},
      eprint={2607.09598},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2607.09598}, 
}

Acknowledgments

Built on UsefulSensors/Moonshine and BanglaBERT. Trained on the Lipi-Ghor-bn-882-SSTT dataset, with GPU support from the Department of CSE at Khulna University of Engineering & Technology (KUET).