# Bengali-Code LLM Training Pipeline

A comprehensive pipeline for training a Bengali language model specialized in code understanding and generation. The model is fine-tuned on Bengali programming tutorials, documentation, and code examples.

## 🌟 Features

- Automated data collection from Bengali Wikipedia and Prothom Alo
- Custom tokenizer training with SentencePiece for Bengali text and code
- Model fine-tuning using TinyLlama base model
- Comprehensive evaluation suite for Bengali code generation
- GitHub Actions workflow for automated training
- Weights & Biases integration for experiment tracking

## 📋 Requirements

- Python 3.10 or higher
- CUDA-capable GPU (recommended)
- 16GB+ RAM
- Internet connection for data collection

## 🚀 Quick Start

1. Clone the repository:
```bash
git clone https://github.com/yourusername/bengali-code-llm.git
cd bengali-code-llm
```

2. Install dependencies:
```bash
pip install -r requirements.txt
```

3. Set up environment variables:
```bash
export HUGGINGFACE_TOKEN="your_token_here"
export WANDB_API_KEY="your_wandb_key_here"
```

4. Run the complete pipeline:
```bash
# Collect data
python scripts/data_collector.py

# Train tokenizer
python scripts/tokenizer_trainer.py

# Train model
python scripts/model_trainer.py

# Evaluate model
python scripts/model_evaluator.py
```

## 🏗️ Pipeline Components

### Data Collection (`scripts/data_collector.py`)
- Scrapes Bengali text from Wikipedia and Prothom Alo
- Implements rate limiting and error handling
- Outputs processed data in JSON format

### Tokenizer Training (`scripts/tokenizer_trainer.py`)
- Uses SentencePiece for tokenizer training
- Custom vocabulary with Bengali and code tokens
- Generates HuggingFace-compatible tokenizer files

### Model Training (`scripts/model_trainer.py`)
- Fine-tunes TinyLlama model
- Implements efficient training with gradient accumulation
- Supports mixed precision training
- Integrates with Weights & Biases for tracking

### Model Evaluation (`scripts/model_evaluator.py`)
- Comprehensive evaluation suite
- Tests code generation capabilities
- Measures BLEU and ROUGE scores
- Generates detailed evaluation reports

## 📊 Training Metrics

The training progress can be monitored through Weights & Biases:
- Loss curves
- Evaluation metrics
- Generated samples
- Resource utilization

## 🔄 GitHub Actions Workflow

The repository includes an automated training pipeline that:
- Runs daily to incorporate new data
- Executes the complete training pipeline
- Uploads model artifacts
- Can be triggered manually

## 📁 Directory Structure

```
bengali-code-llm/
├── .github/
│   └── workflows/
│       └── train_model.yml
├── scripts/
│   ├── data_collector.py
│   ├── tokenizer_trainer.py
│   ├── model_trainer.py
│   └── model_evaluator.py
├── data/
│   └── raw/
├── outputs/
│   ├── tokenizer/
│   ├── model/
│   └── evaluation/
├── requirements.txt
└── README.md
```

## 🎯 Model Performance

The model is evaluated on various tasks:
- Code generation in Bengali
- Code explanation and documentation
- Error detection and correction
- Algorithm explanation

## 📜 License

This project is licensed under the MIT License - see the LICENSE file for details.

## 🤝 Contributing

Contributions are welcome! Please feel free to submit issues and pull requests.

## 📧 Contact

For questions and feedback, please open an issue in the repository.

## 🙏 Acknowledgments

- TinyLlama team for the base model
- HuggingFace for the Transformers library
- Weights & Biases for experiment tracking