README.md

metadata

language:
  - si
  - ta
  - en
license: other
tags:
  - qwen
  - qlora
  - cpt
  - sri-lanka
  - sinhala
  - tamil
model_type: causal-lm
pipeline_tag: text-generation
base_model: qwen3.5-4b

Chat2Find-CPT: Qwen 3.5 4B (Sri Lankan Continued Pre-Training)

Chat2Find-CPT is a specialized version of the Qwen 3.5 4B model, enhanced via Continued Pre-Training (CPT) using QLoRA (4-bit) to excel in Sri Lankan linguistic and cultural contexts. It features robust proficiency in Sinhala, Tamil, and English.

Model Details

• Developed by: Sentient (Chat2Find)
• Base Model: Qwen 3.5-4B
• Training Method: Continued Pre-Training (CPT) with QLoRA
• Languages: Sinhala (si), Tamil (ta), English (en)
• Quantization: 4-bit (bitsandbytes)

Technical Specifications

Training Hardware

• GPU: NVIDIA GeForce RTX 3060 (12GB VRAM)
• Frameworks: Unsloth, Hugging Face Transformers, PEFT

Training Hyperparameters

• Method: QLoRA (Rank 32, Alpha 32)
• Learning Rate: 5e-5 (Cosine Scheduler)
• Optimizer: AdamW (8-bit)
• Epochs: 1.0
• Sequence Length: 2048 tokens
• Batch Size: 2 (local) / 8 (global with Gradient Accumulation)

Dataset

The model was trained on a curated corpus of ~270,000 sequences focusing on:

• Sri Lankan News & Media: Current events and reporting styles.
• Business & Logistics: Domain-specific data from Lankabizz and local commerce.
• Cultural Context: General web-scraped data reflecting local nuances.

Capabilities

Chat2Find-CPT excels at:

1. Sinhala & Tamil Generation: Fluent and contextually relevant text generation.
2. Code-Switching: Handling natural language mixes common in Sri Lankan communication.
3. Local Knowledge: Understanding entities, locations, and cultural references specific to Sri Lanka.

Usage

Using Unsloth (Recommended for Speed)

from unsloth import FastLanguageModel
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "SENTIENT_ID/Chat2Find-CPT",
    max_seq_length = 2048,
    load_in_4bit = True,
)
FastLanguageModel.for_inference(model)

inputs = tokenizer(
    ["ශ්‍රී ලංකාව ගැන කෙටි විස්තරයක්:"],
    return_tensors = "pt"
).to("cuda")

outputs = model.generate(**inputs, max_new_tokens = 256)
print(tokenizer.batch_decode(outputs))

Using Standard Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "SENTIENT_ID/Chat2Find-CPT"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")

# Note: The model is a merged 16-bit weight set. 
# You can load it in 4-bit/8-bit using BitsAndBytes.

Limitations & Bias

While Chat2Find-CPT is significantly better at local languages than the base Qwen model, it may still exhibit biases present in the training data or the base model's internal knowledge. Users are encouraged to perform their own safety checks for specific deployment scenarios.

License

This model is subject to the original Qwen License Agreement.