BharatVLM/AssameseGPT2

Assamese GPT-2 Model

This is a GPT-2 language model trained from scratch on Assamese monolingual text, using data from IndicCorpV2 . The model is developed for educational and research purposes to support natural language understanding and generation tasks in Assamese — a low-resource language.

📖 Model Description

The Assamese GPT-2 model is based on the standard GPT-2 decoder-only transformer architecture with 12 layers, 12 attention heads, 768 hidden size. It is capable of generating grammatically coherent and contextually relevant Assamese text and serves as a foundation for downstream NLP tasks such as:

• Language modeling
• Text completion/generation
• Fine-tuning for classification or summarization

✅ Intended Uses

• Academic research on Assamese NLP
• Training and benchmarking in educational settings
• Exploration of low-resource language modeling

🚫 Limitations

• Trained on general-domain monolingual data, may not perform well on domain-specific texts (e.g., legal, medical).
• Might generate biased, incomplete, or hallucinated outputs.
• Not suitable for production use or deployment in sensitive applications.

📚 Training and Evaluation Data

The model was trained using Assamese monolingual data collected from:

• IndicCorpV2: A curated collection of web-crawled and processed data for Indic languages.

Data preprocessing included:

• Unicode normalization
• Removal of noisy characters and malformed tokens
• Sentence segmentation using Assamese-specific heuristics

🧪 Training Procedure

Hyperparameters

• Architecture: GPT2 (12 layers, 12 heads, 768 hidden size)
• Tokenizer vocab size: 50,000
• Context window size: 1024 tokens
• Learning rate: 5e-5
• Epochs: 20
• Batch size: 64
• Optimizer: AdamW (β₁=0.9, β₂=0.999, ε=1e-8)
• Scheduler: Linear
• Mixed Precision: Native AMP
• Seed: 42

Results

• Final Evaluation Loss: -29.1890
• Accuracy: 0.3452

🚀 Example Usage

from transformers import GPT2LMHeadModel, GPT2Tokenizer

model = GPT2LMHeadModel.from_pretrained("BharatVLM/AssameseGPT2")
tokenizer = GPT2Tokenizer.from_pretrained("BharatVLM/AssameseGPT2")

prompt = "অসমৰ ইতিহাস"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=50, do_sample=True)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

📄 License

This model is released under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license. Commercial use is not permitted. Use is allowed for academic and research purposes only.

📬 Citation

Please cite this model as:

@misc{assamesegpt2, author = {BharatVLM}, title = {Assamese GPT-2 Model}, year = 2025, howpublished = {\url{https://huggingface.co/BharatVLM/AssameseGPT2}}, note = {Trained using IndicCorpV2 and OSCAR corpora} }

🧰 Framework Versions

• Transformers: 4.52.0.dev0

• PyTorch: 2.5.1+cu121

• Datasets: 3.6.0

• Tokenizers: 0.21.1

Contact Us

For questions or academic collaboration, please contact: ai.bharatvlm@gmail.com.