| --- |
| language: |
| - bn |
| license: apache-2.0 |
| tags: |
| - bengali |
| - bangla |
| - causal-lm |
| - llama |
| - custom-tokenizer |
| - parameter-efficient |
| datasets: |
| - HuggingFaceFW/fineweb-2 |
| - hishab/titulm-bangla-corpus |
| metrics: |
| - accuracy |
| --- |
| |
| # Alo-50M-Base-Retied |
|
|
| ## Model Summary |
| **Alo-50M-Base-Retied** is a compact, 50-million parameter foundational Large Language Model (LLM) built exclusively for the Bengali (Bangla) language. Trained entirely from scratch, it aims to lower the compute barrier for Bengali NLP research and provide a viable path for deploying localized AI assistants on standard CPUs and resource-constrained edge devices. |
|
|
| > ⚠️ **Note on Weight Tying (Update):** This model is the updated, mathematically tied version of the original `Alo-70M-Base`. By correctly tying the input and output embeddings (`tie_word_embeddings = True`), we eliminated a duplicate copy of the $16.8\text{M}$ parameter embedding matrix from the checkpoint. This successfully reduced the active parameter footprint from **69M to 52M** and shrank the disk/download size by **24% (from 276MB to 209MB)** while maintaining mathematically identical outputs and evaluation performance. |
|
|
| * **Developer:** Fahad Hossain |
| * **Language:** Bengali (Bangla) |
| * **Model Type:** Causal Language Model (Autoregressive Transformer) |
| * **Parameter Count:** 52 Million (Active/Tied) / Formerly 69 Million (Untied) |
| * **License:** Apache 2.0 |
|
|
| ## Related Resources |
| This model is the **base** foundational model. We also release the instruction-tuned version, the synthetic dataset used for alignment, and the standalone tokenizer: |
| * **Instruction-Tuned Model:** [spitfire4794/Alo-70M](https://huggingface.co/spitfire4794/Alo-70M) |
| * **Alignment Dataset:** [spitfire4794/Bangla-SFT-50k](https://huggingface.co/datasets/spitfire4794/Bangla-SFT-50k) |
| * **Tokenizer:** [spitfire4794/beng_bpe](https://huggingface.co/spitfire4794/beng_bpe) |
|
|
| ## Usage |
|
|
| You can load and generate text with Alo-50M-Base-Retied using the `transformers` library. |
|
|
| ```python |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| |
| model_id = "spitfire4794/Alo-70m-Base-retied" |
| tokenizer_id = "spitfire4794/beng_bpe" |
| |
| # Load the custom Bengali BPE tokenizer and model |
| tokenizer = AutoTokenizer.from_pretrained(tokenizer_id) |
| model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto") |
| |
| prompt = "বাংলাদেশের রাজধানী ঢাকা" |
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
| |
| # Generate text |
| outputs = model.generate( |
| **inputs, |
| max_new_tokens=50, |
| repetition_penalty=1.1, |
| do_sample=True, |
| temperature=0.7 |
| ) |
| |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| ``` |
|
|
| ## Model Details |
| Alo-50M-Base-Retied is based on a scaled-down LLaMA transformer architecture optimized for hardware alignment and parameter efficiency. |
| * **Layers:** 12 |
| * **Hidden Dimension ($d_{model}$):** 512 |
| * **Intermediate FFN Dimension:** 1408 |
| * **Attention Mechanism:** Grouped-Query Attention (GQA) with 8 query heads and 4 key-value heads. |
| * **Positional Embeddings:** Rotary Position Embeddings (RoPE) with a base frequency theta of 10,000. |
| * **Activation Function:** SwiGLU (`silu`) |
| * **Word Embeddings:** Tied (`tie_word_embeddings = True`). The input embeddings (`embed_tokens`) and output language model head (`lm_head`) share the exact same weight matrix. This resolves a prior configuration issue where identical matrices were loaded twice into GPU memory, reducing the static lookup table footprint from 33.5M parameters down to 16.8M parameters. |
| * **Context Window:** 1024 tokens. |
| * **Tokenizer:** Custom Bengali Byte-Pair Encoding (BPE) tokenizer ([spitfire4794/beng_bpe](https://huggingface.co/spitfire4794/beng_bpe)) with a highly optimized vocabulary size of 32,768 (2^15) designed to maximize TPU efficiency. |
| |
| ## Pre-Training Data |
| The model was pre-trained on a curated, strictly Bengali corpus containing approximately **19.25 billion tokens**. The dataset is a blend of: |
| 1. **FineWeb-2 (Bengali Split):** Providing diverse, web-scale filtered documents. |
| 2. **TituLM Bengali Corpus (CommonCrawl Split):** Introducing balanced, general-domain web data. |
| |
| ## Training Procedure |
| Alo-50M-Base-Retied (initially trained as `Alo-70M-Base`) was initialized with completely randomized weights (no transfer learning) and trained from scratch. |
| * **Hardware:** Kaggle Tensor Processing Unit (TPU) v5e-8 |
| * **Training Time:** 12.9 hours (wall time) across two sequential runs. |
| * **Optimizer:** AdamW (`adam_torch_xla`) with beta_1 = 0.9, beta_2 = 0.95, epsilon = 10^-8 |
| * **Weight Decay:** 0.1 |
| * **Learning Rate Schedule:** Cosine Decay with a gradual warmup phase, peaking at 6 times 10^-4. |
| * **Precision & Optimization:** Mixed `bfloat16` (`bf16`) precision with gradient checkpointing enabled to maximize throughput. |
| |
| ## Evaluation Results |
| The model was evaluated in a zero-shot setting across various Bengali reasoning and knowledge benchmarks using a continuation-based evaluation methodology (calculating conditional log-probabilities directly over raw Bengali text to avoid "pointer-reasoning" bottlenecks). |
| |
| The evaluation metrics below represent the actual raw benchmark results computed directly from the tied checkpoint. |
| |
| | Benchmark | Alo-50M-Base-Retied | Gemma-3-270M-IT | TigerLLM-1B-IT | |
| | :--- | :---: | :---: | :---: | |
| | **bangla_mmlu_bn** | 26.30% | 26.81% | 27.66% | |
| | **bangla_commonsenseqa_bn** | **28.58%** | 22.77% | 25.14% | |
| | **indicbench_arc_bn_challenge** | 23.04% | 25.34% | 27.13% | |
| | **boolqa_bn** | 48.97% | 51.30% | 52.40% | |
| | **openbookqa_bn** | 30.78% | 31.99% | 34.21% | |
| | **piqa_bn** | **50.49%** | 49.51% | 49.51% | |
| | **hellaswag_bn** | 27.17% | 27.85% | 31.01% | |
|
|
| *Note: Alo-50M-Base-Retied matches or outperforms both the 270M and 1B baselines on CommonsenseQA and PIQA despite having 5x to 19x fewer active parameters.* |
|
|
| ## Limitations and Biases |
| * **Encyclopedic Knowledge:** Due to the severe parameter constraints (~50M parameters), the model physically lacks the parametric capacity to act as a deep encyclopedic knowledge base, which reflects in its performance on knowledge-intensive retrieval tasks like MMLU and ARC-Challenge. |
| * **Tokenizer Evaluation Bias:** During continuation-based log-likelihood evaluations, Alo-50M's highly efficient native tokenizer may possess a mathematical advantage over multilingual baselines (which heavily fragment Bengali text), contributing partially to its high performance margins on specific tasks. |
| * **Instruction Following:** This is a base model. It will attempt to complete text rather than follow conversational instructions. For chat and instruction-following, please use **Alo-70M**. |
|
|
| ## Citation |
| *Technical paper out soon.* |
