README.md

---
language: kha
license: cc-by-nc-4.0
tags:
- khasi
- bert
- roberta
- masked-lm
- foundational-model
- low-resource
- indian-language
- meghalaya
- austroasiatic
library_name: transformers
pipeline_tag: fill-mask
widget:
- text: 'Ka Meghalaya ka <mask> ha ka jingpyrkhat jong ki Khasi.'
- text: 'U bah u la <mask> da ka bazar.'
- text: 'Ki lum Khasi ki <mask> ha ka ri Meghalaya.'
---

[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.17063992.svg)](https://doi.org/10.5281/zenodo.17063992)

# KhasiBERT: Foundational Language Model for Khasi

**KhasiBERT** is a foundational language model for the Khasi language, trained on 3.6 million sentences using the RoBERTa architecture. This model serves as the foundation for all downstream Khasi NLP tasks including text classification, sentiment analysis, question answering, and language generation.

## Model Overview

| Attribute | Value |
| --- | --- |
| Model Name | KhasiBERT |
| Version | 1.0.0 |
| Architecture | RoBERTa-base |
| Parameters | 110,652,416 |
| Model Size | 421 MB |
| Language | Khasi (kha) |
| Language Family | Austroasiatic |
| Training Data | 3,621,116 sentences |
| Vocabulary Size | 32,000 tokens |
| Max Sequence Length | 512 tokens |
| Training Time | ~4 hours |
| GPU Used | NVIDIA RTX A6000 (48GB) |

## Model Architecture

KhasiBERT follows the RoBERTa-base architecture with the following specifications:

| Component | Configuration |
| --- | --- |
| Transformer Layers | 12 |
| Hidden Size | 768 |
| Attention Heads | 12 |
| Intermediate Size | 3,072 |
| Activation Function | GELU |
| Dropout | 0.1 |
| Layer Norm Epsilon | 1e-12 |
| Max Position Embeddings | 514 |

## Training Methodology

### Training Objective
KhasiBERT was trained using Masked Language Modeling (MLM), where 15% of input tokens are randomly masked and the model learns to predict these masked tokens based on bidirectional context.

### Training Configuration

| Hyperparameter | Value |
| --- | --- |
| Training Objective | Masked Language Modeling |
| Masking Probability | 15% |
| Optimizer | AdamW |
| Learning Rate | 5e-5 |
| Learning Rate Schedule | Linear with warmup |
| Warmup Steps | 5,000 |
| Weight Decay | 0.01 |
| Batch Size | 24 |
| Gradient Accumulation | 1 |
| Training Epochs | 1 |
| Total Training Steps | 150,880 |
| Mixed Precision | FP16 |
| Hardware | NVIDIA RTX A6000 (48GB) |

### Tokenization
A custom Byte-Level BPE tokenizer was trained specifically on the Khasi corpus with:
- Vocabulary size: 32,000 tokens
- Special tokens: `<s>`, `</s>`, `<pad>`, `<unk>`, `<mask>`
- Trained on the complete Khasi dataset for optimal language coverage

## Training Data

### Dataset Statistics

| Statistic | Value |
| --- | --- |
| Total Sentences | 3,621,116 |
| Average Sentence Length | 83 characters |
| Estimated Total Tokens | ~50-70 million |
| Data Quality | High-quality, deduplicated |
| Language Coverage | Comprehensive Khasi text |

### Data Preprocessing
- Exact duplicate removal
- Near-duplicate removal (80% similarity threshold)
- Length filtering (10-500 characters)
- Text normalization and cleaning
- Quality validation

## Usage

### Quick Start

```python
from transformers import RobertaForMaskedLM, RobertaTokenizerFast, pipeline

# Load model and tokenizer
model = RobertaForMaskedLM.from_pretrained('MWirelabs/khasibert')
tokenizer = RobertaTokenizerFast.from_pretrained('MWirelabs/khasibert')

# Create fill-mask pipeline
fill_mask = pipeline('fill-mask', model=model, tokenizer=tokenizer)

# Example usage
text = 'Ka Meghalaya ka <mask> ha ka jingpyrkhat jong ki Khasi.'
results = fill_mask(text)
print(results)
```

### Fine-tuning for Downstream Tasks

```python
from transformers import RobertaForSequenceClassification

# For text classification
model = RobertaForSequenceClassification.from_pretrained(
    'MWirelabs/khasibert',
    num_labels=2  # Adjust for your task
)

# Fine-tune for sentiment analysis, document classification, etc.
```

## Model Performance

### Masked Language Modeling Results

KhasiBERT demonstrates strong contextual understanding in Khasi:

| Test Case | Input | Top Prediction | Confidence |
| --- | --- | --- | --- |
| Question Context | Phi lah bam <mask>? | bha | 6.8% |
| Location Context | Ka shnong jongngi ka don ha pdeng ki <mask> | khlaw | 7.1% |
| Place Reference | Ngan sa leit kai sha <mask> lashai. | Delhi | 10.6% |
| Action Context | Ngi donkam ban leit sha iew ban thied <mask>. | jingthied | 25.3% |
| Gratitude Expression | Khublei shibun na ka bynta ka jingiarap jong <mask>. | phi | 44.7% |

### Key Strengths
- **Authentic Khasi understanding** with contextually appropriate predictions
- **Pronoun recognition** - correctly predicts "phi" (you) in gratitude expressions (44.7%)
- **Semantic relationships** - predicts "jingthied" related to "thied" (25.3%)
- **Place name recognition** - identifies proper nouns like "Delhi" (10.6%)
- **Grammatical structure awareness** across question, location, and action contexts

## Applications

KhasiBERT serves as the foundation for various Khasi NLP applications:

### Supported Tasks
- **Text Classification**: Document categorization, topic modeling
- **Sentiment Analysis**: Opinion mining in Khasi text
- **Named Entity Recognition**: Person, place, organization extraction
- **Question Answering**: Khasi reading comprehension systems
- **Text Generation**: Coherent Khasi text creation
- **Language Understanding**: Chatbots and virtual assistants
- **Machine Translation**: English-Khasi translation systems

### Use Cases
- Educational technology for Khasi language learning
- Government services in Meghalaya
- Cultural preservation and digital humanities
- Social media monitoring and analysis
- Content recommendation systems

## Regional Significance

### For Meghalaya State
- **Digital Government Services**: Enables Khasi language interfaces for e-governance
- **Educational Technology**: Powers AI-driven Khasi language learning platforms
- **Cultural Preservation**: Digitally preserves and promotes Khasi linguistic heritage
- **Economic Development**: Creates foundation for local language tech industry

### For Northeast India
- **Linguistic Diversity**: Supports AI development for Northeast India's rich language landscape  
- **Digital Inclusion**: Ensures indigenous communities aren't left behind in AI revolution
- **Research Hub**: Positions Meghalaya as center for indigenous language AI research

## Technical Requirements

### Hardware Requirements

| Task | RAM | GPU Memory | GPU |
| --- | --- | --- | --- |
| Inference (CPU) | 4GB | - | - |
| Inference (GPU) | 8GB | 2GB | Any CUDA GPU |
| Fine-tuning | 16GB | 8GB | RTX 3080+ |
| Full Training | 32GB | 24GB+ | RTX 4090/A6000+ |

### Software Requirements
- Python 3.8+
- PyTorch 1.9+
- Transformers 4.20+
- CUDA 11.0+ (for GPU)

## Research Context

### Significance
KhasiBERT represents a significant advancement in low-resource NLP:

- **A foundational model** for the Khasi language
- **Enables NLP research** for 1.4+ million Khasi speakers
- **Preserves linguistic heritage** through AI technology
- **Demonstrates efficient training** methodology for resource-constrained scenarios

### Limitations
- Trained on 1 epoch (future versions may benefit from additional training)
- Performance may vary on highly domain-specific text
- Requires task-specific fine-tuning for optimal performance
- May not capture all dialectal variations of Khasi

## Citation

If you use KhasiBERT in your research or applications, please cite:

```bibtex
@misc{khasibert2025,
  title        = {KhasiBERT v1.0: A Foundational Language Model for Khasi},
  author       = {MWire Labs},
  year         = {2025},
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.17063992},
  url          = {https://doi.org/10.5281/zenodo.17063992}
}

```

## Contact & Support

- **Organization**: MWirelabs
- **Model Repository**: https://huggingface.co/MWirelabs/khasibert
- **Issues**: Please report issues through the Hugging Face model page

## Acknowledgments

- Training conducted on NVIDIA RTX A6000 GPU
- Built using the Transformers library by Hugging Face
- Inspired by the success of foundational models for major languages
- Dedicated to the preservation and advancement of the Khasi language

## License

This model is released under the Creative Commons BY-NC 4.0 License. You are free to:
- Use for non-commercial research and education
- Modify and distribute for non-commercial purposes  
- Create derivative works for research

**Commercial Use**: Contact MWirelabs for commercial licensing agreements.
**Attribution Required**: Please provide appropriate credit to MWirelabs when using this model.

---

*KhasiBERT: Bridging traditional Khasi language with modern AI technology.*