eulogik/Bharat-Tiny-LLM-adapter

Bharat-Tiny-LLM LoRA Adapter

This repository contains the LoRA adapter weights for Bharat-Tiny-LLM, a Hinglish (Hindi+English) conversational model fine-tuned on a Mac Mini M4.

Base model: Qwen/Qwen2.5-1.5B

Training Details

Metric	Value
Method	LoRA (16 layers, rank 8, alpha 16)
Training iters	76,420
Best val loss	0.781
Trainable params	5.276M (0.34% of 1.5B)
Data	376K conversations from 5 datasets
Framework	MLX (Apple Silicon)
Hardware	Mac Mini M4 (16GB)

Usage

Important: This adapter was trained with MLX. Due to a fundamental MLX ↔ PEFT LoRA implementation mismatch, the adapter cannot be loaded directly with PEFT/Transformers. Manual fusion is required.

To use this adapter:

import torch
import safetensors.torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from huggingface_hub import hf_hub_download

# Load base model
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-1.5B",
    torch_dtype=torch.float16
).to("mps")  # or "cuda" or "cpu"

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-1.5B")

# Load adapter weights
peft_path = hf_hub_download("eulogik/Bharat-Tiny-LLM-adapter", "adapter_model.safetensors")
peft = safetensors.torch.load_file(peft_path)

# Fuse LoRA weights into base model
scale = 16.0  # alpha / r
for name, param in model.named_parameters():
    if param.ndim != 2:
        continue
    layer_path = name.replace(".weight", "")
    lora_A_key = f"base_model.model.{layer_path}.lora_A.weight"
    lora_B_key = f"base_model.model.{layer_path}.lora_B.weight"
    if lora_A_key in peft and lora_B_key in peft:
        lora_A = peft[lora_A_key].to(device=model.device, dtype=param.dtype)
        lora_B = peft[lora_B_key].to(device=model.device, dtype=param.dtype)
        param.data += torch.mm(lora_B, lora_A) * scale

# Generate
prompt = "<|im_start|>user\nChai peete hain?<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Alternatively, use the pre-fused model: eulogik/Bharat-Tiny-LLM-fused

Limitations

• Experimental quality: The model was trained for 76K iterations only. Output quality varies significantly across prompts.
• Repetition & coherence: The model sometimes produces repetitive or off-topic responses.
• Language switching: May switch between Hinglish, English, or Hindi mid-response.
• Training gap: Further training (100K+ iters with lower LR) would significantly improve quality.

License

Apache 2.0