Squash history to single initial commit

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,122 @@
+---
+language:
+- bn
+license: apache-2.0
+tags:
+- bengali
+- bangla
+- causal-lm
+- llama
+- custom-tokenizer
+- parameter-efficient
+- instruction-tuning
+- sft
+datasets:
+- spitfire4794/Bangla-SFT-50k
+metrics:
+- accuracy
+---
+
+# Alo-70M (Instruct)
+
+## Model Summary
+**Alo-70M** is the instruction-tuned version of the ultra-lightweight 69-million parameter Bengali language model, [Alo-70M-Base](https://huggingface.co/spitfire4794/Alo-70M-Base). Built on a scaled-down LLaMA architecture, it is designed to act as a highly efficient, edge-deployable localized AI assistant. 
+
+Fine-tuned on a curated dataset of instruction-response pairs using the **ChatML** format, Alo-70M is aligned for tasks such as summarization, entity extraction, text editing, and question answering in native Bengali. Despite its compact footprint, it offers a viable path for edge AI deployment on standard CPUs and mobile hardware.
+
+* **Developer:** Fahad Hossain
+* **Language:** Bengali (Bangla)
+* **Model Type:** Causal Language Model (Instruction-Tuned Autoregressive Transformer)
+* **Parameter Count:** 69 Million
+* **License:** Apache 2.0
+
+## Related Resources
+* **Base Model:** [spitfire4794/Alo-70M-Base](https://huggingface.co/spitfire4794/Alo-70M-Base)
+* **Alignment Dataset:** [spitfire4794/Bangla-SFT-50k](https://huggingface.co/datasets/spitfire4794/Bangla-SFT-50k)
+* **Tokenizer:** [spitfire4794/beng_bpe](https://huggingface.co/spitfire4794/beng_bpe)
+
+## Usage 
+
+Alo-70M was trained using the ChatML template. The chat template is built directly into the Jinja template of the tokenizer (`spitfire4794/beng_bpe`). You can leverage it using Hugging Face's `apply_chat_template` interface:
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+model_id = "spitfire4794/Alo-70M"
+tokenizer_id = "spitfire4794/beng_bpe"
+
+# Load the custom Bengali BPE tokenizer and model
+tokenizer = AutoTokenizer.from_pretrained(tokenizer_id)
+model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
+
+# Define the instruction in ChatML format
+messages = [
+    {"role": "user", "content": "নিচের অনুচ্ছেদটি সংক্ষেপে সারসংক্ষেপ করুন: [এখানে আপনার টেক্সট লিখুন]"}
+]
+
+# Apply the pre-configured ChatML template
+inputs = tokenizer.apply_chat_template(
+    messages, 
+    add_generation_prompt=True, 
+    return_tensors="pt"
+).to(model.device)
+
+# Generate text
+outputs = model.generate(
+    **inputs, 
+    max_new_tokens=150, 
+    repetition_penalty=1.1,
+    do_sample=True,
+    temperature=0.6,
+    top_p=0.9
+)
+
+# Decode response (omitting user prompt)
+response = outputs[0][inputs.shape[-1]:]
+print(tokenizer.decode(response, skip_special_tokens=True))
+```
+
+## Supervised Fine-Tuning (SFT) Details
+Alo-70M was aligned using a curated subset of the **Bangla-SFT-50k** dataset formatted using **ChatML**.
+* **Dataset Pruning:** Initial SFT experiments revealed that forcing a sub-100M parameter model to learn complex markdown syntax/tables caused severe representation crowding. Thus, the 12,517-sample *Structured Formatting* category was excluded. The final active training mixture consisted of **37,536** aligned pairs.
+* **Engineering Properties:** The training data strictly forbade conversational prefaces (e.g., "নিশ্চয়ই, আমি এটি করে দিচ্ছি") so that responses begin immediately with the target output, optimizing inference speeds.
+* **Hardware:** NVIDIA T4 and L4 GPUs.
+* **Hyperparameters:** 
+  * Optimizer: Fused AdamW (`adamw_torch_fused`) with $\beta_1 = 0.9, \beta_2 = 0.999, \epsilon = 10^{-8}$
+  * Weight Decay: 0.0
+  * Learning Rate Schedule: Cosine decay, peaking at $3 \times 10^{-4}$ with a 10% linear warmup.
+  * Epochs: 3
+  * Effective Batch Size: 32 (per-device 8 with gradient accumulation of 4).
+  * Precision: Native Automatic Mixed Precision (AMP).
+
+## Model Architecture details
+Like its base model, Alo-70M utilizes a parameter-efficient architecture:
+* **Layers:** 12
+* **Hidden Dimension ($d_{model}$):** 512 | **Intermediate FFN:** 1408
+* **Attention:** Grouped-Query Attention (GQA) with 8 query heads / 4 KV heads.
+* **Positional Embeddings:** RoPE (Base freq: 10,000)
+* **Word Embeddings:** Untied (`tie_word_embeddings = False`).
+* **Context Window:** 1024 tokens.
+
+## Evaluation Results
+The model was evaluated zero-shot across Bengali reasoning and knowledge benchmarks (continuation-based log-probability evaluation):
+
+| Benchmark | Alo-70M (SFT) | Alo-70M-Base | Gemma-3-270M-IT | TigerLLM-1B-IT |
+| :--- | :---: | :---: | :---: | :---: |
+| **bangla_mmlu_bn** | 26.29% | 26.31% | 26.81% | 27.66% |
+| **bangla_commonsenseqa_bn** | **25.88%** | 28.42% | 22.77% | 25.14% |
+| **indicbench_arc_bn_challenge** | 24.15% | 22.70% | 25.34% | 27.13% |
+| **boolqa_bn** | 48.70% | 48.42% | 51.30% | 52.40% |
+| **openbookqa_bn** | 30.58% | 31.39% | 31.99% | 34.21% |
+| **piqa_bn** | **50.05%** | 50.49% | 49.51% | 49.51% |
+| **hellaswag_bn** | 26.89% | 27.27% | 27.85% | 31.01% |
+
+*Note: The 69M instruction-tuned model outperforms the larger Gemma-3-270M-IT baseline on tasks like CommonsenseQA and PIQA.*
+
+## Limitations and Biases
+* **Alignment Tax (Catastrophic Forgetting):** While SFT successfully aligned the model for text generation stability and instruction following, it introduced a measurable degradation in pure zero-shot reasoning compared to the Base model (e.g., dropping from 28.42% to 25.88% on CommonsenseQA). This happens because applying instructions to a highly capacity-constrained 69M model over-indexes weights toward output formatting at the expense of some pre-trained logical representations.
+* **Knowledge Retrieval:** With under 100M parameters, the model physically lacks the capacity to serve as a comprehensive encyclopedic knowledge base. It is better suited for text processing tasks (editing, summarizing) than fact-retrieval.
+* **Context Length:** The model is optimized for a 1024-token context window. Prompts exceeding this length will be truncated or result in degraded quality.
+
+## Citation
+*Technical paper out soon.*

chat_template.jinja

@@ -0,0 +1,7 @@
+{{- bos_token }}
+{%- for message in messages %}
+    {{- '<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>\n' }}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

config.json

@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "dtype": "float32",
+  "eos_token_id": 2,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 512,
+  "initializer_range": 0.02,
+  "intermediate_size": 1408,
+  "max_position_embeddings": 1024,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 8,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 4,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.0.0",
+  "use_cache": false,
+  "vocab_size": 32768
+}

generation_config.json

@@ -0,0 +1,12 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": [
+    2
+  ],
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 0,
+  "transformers_version": "5.0.0",
+  "use_cache": false
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:25fd7e5fabea1d37b72b63686dd7954b961f5bdcd9eb13b77d4e8fbd2ff635e4
+size 275839008

tokenizer_config.json

@@ -0,0 +1,23 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|tool_call|>",
+    "<|/tool_call|>",
+    "<|tool_response|>",
+    "<|begin_of_thought|>",
+    "<|end_of_thought|>",
+    "<|reserved_0|>",
+    "<|reserved_1|>"
+  ],
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<|endoftext|>"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c7d45d39aa5dfcde0fc617983ffca808cc137529610a96125bf0c9efb52b251a
+size 5201