Amna-Shahzad/code-switching-codesaviours-si26-Amna

README.md

---
language:
- en
- ur
tags:
- token-classification
- code-switching
- roman-urdu
- urdu
- english
- multilingual
- xlm-roberta
- text-classification
- code-mixing
license: apache-2.0
base_model: FacebookAI/xlm-roberta-base
pipeline_tag: token-classification
---

# Urdu-English Code-Switching Detection

A fine-tuned **XLM-RoBERTa** model for identifying whether individual words in code-switched text are **Urdu, English, or Mixed**.

This model was developed as part of the **CodeSaviours SI-26** project by **Amna Shahzad** at **Code Saviours (SMC-PRIVATE) Limited**.

---

## Overview

Code-switching is the practice of using words or expressions from multiple languages within the same sentence or conversation.

In social media and informal communication, Urdu and English are frequently mixed together. This is especially common in:

- YouTube comments
- WhatsApp messages
- Twitter/X posts
- Instagram content

For example:

> `Aaj weather bohat amazing hai`

A human reader can easily understand the sentence, but automatically determining the language of each word can be challenging.

This model addresses this problem by classifying individual words into three categories:

| Label | Meaning |
|------|---------|
| `URD` | Urdu |
| `ENG` | English |
| `MIX` | Mixed Urdu-English / code-switched word |

---

# Project Objective

The primary objective of this project is to develop a language identification model capable of detecting the language category of individual words in **Urdu-English code-switched text**.

The model can be used as a component in larger NLP systems for:

- Code-switching analysis
- Urdu-English language identification
- Social media text analysis
- Multilingual NLP
- Roman Urdu processing
- Language-aware text preprocessing
- Code-mixed sentiment analysis
- Urdu-English text normalization

---

# Model

The model is based on:

**XLM-RoBERTa Base**

XLM-RoBERTa is a multilingual Transformer model designed to process text across multiple languages.

For this project, the pretrained model was fine-tuned using a **token-classification architecture**.

### Architecture

```text
Input Sentence
      │
      ▼
XLM-RoBERTa Tokenizer
      │
      ▼
XLM-RoBERTa Encoder
      │
      ▼
Token Classification Head
      │
      ▼
URD / ENG / MIX

The model performs classification at the token/word level rather than assigning a single label to the complete sentence.

---

Label Definitions

The model uses three labels:

URD

Represents words identified as Urdu.

Example:

Aaj mujhe ghar jana hai
    ↓
URD

ENG

Represents English words.

Example:

Aaj weather amazing hai
    ↓
weather → ENG
amazing → ENG

MIX

Represents words that contain or combine Urdu and English characteristics.

Example:

downloadkarna

may be classified as:

MIX

Note: The exact behavior of MIX depends on the annotation guidelines used during dataset creation.

---

Dataset

The model was trained on a manually collected dataset of 175 sentences.

Split	Number of Sentences
Training	140
Testing	35
Total	175

The dataset consists of Urdu-English code-switched sentences collected from multiple real-world communication platforms.

Data Sources

The collected examples were sourced from:

• YouTube comments
• WhatsApp messages
• Twitter/X posts
• Instagram content

The purpose of using multiple sources was to capture different styles of informal Urdu-English communication.

---

Example

An example of code-switched text:

Aaj meeting bohat important hai

Possible token-level classification:

Aaj        → URD
meeting    → ENG
bohat      → URD
important  → ENG
hai        → URD

Another example:

Mujhe ye file downloadkarni hai

Possible classification:

Mujhe          → URD
ye             → URD
file           → ENG
downloadkarni  → MIX
hai            → URD

---

Training

The pretrained xlm-roberta-base model was fine-tuned for token classification.

Training Configuration

Parameter	Value
Base Model	xlm-roberta-base
Task	Token Classification
Number of Labels	3
Training Samples	140
Testing Samples	35
Epochs	5
Training Batch Size	16
Evaluation Batch Size	8
Learning Rate	5e-5
Optimizer	AdamW
LR Scheduler	Linear
Weight Decay	0.0
Warmup Steps	0
Gradient Accumulation	1
Random Seed	42
Evaluation Strategy	Epoch
Save Strategy	Epoch
Best Model Selection	Validation Loss

---

📈 Performance

The model achieved an overall token-level accuracy of:

99.70%

This result was obtained on the held-out test set containing 35 sentences.

Metric	Result
Overall Accuracy	99.70%
Training Samples	140
Testing Samples	35

Interpretation

An overall accuracy of 99.70% means that approximately 99.7% of the evaluated tokens were assigned the correct language label according to the test annotations.

Because the dataset is relatively small, this result should be interpreted cautiously. Evaluation on a larger and more diverse dataset would provide a stronger estimate of real-world generalization.

---

Intended Use

This model is intended for research and educational purposes involving:

• Urdu-English code-switching
• Roman Urdu
• Multilingual NLP
• Token-level language identification
• Social media text analysis
• Code-mixed NLP pipelines

It can also be used as a preprocessing component for downstream applications such as:

Code-Switched Text
        │
        ▼
Language Identification
        │
   ┌────┼────┐
   ▼    ▼    ▼
  URD  ENG  MIX
        │
        ▼
Downstream NLP Task

---

How to Use

Install the required libraries:

pip install transformers torch

Then load the model:

from transformers import (
    AutoTokenizer,
    AutoModelForTokenClassification,
    pipeline
)

model_name = "Amna-Shahzad/code-switching-codesaviours-si26-Amna"

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForTokenClassification.from_pretrained(
    model_name
)

classifier = pipeline(
    "token-classification",
    model=model,
    tokenizer=tokenizer,
    aggregation_strategy="simple"
)

Run the model on a sentence:

text = "Aaj meeting bohat important hai"

results = classifier(text)

for result in results:
    print(
        result["word"],
        "->",
        result["entity_group"]
    )

Example output:

Aaj        -> URD
meeting    -> ENG
bohat      -> URD
important  -> ENG
hai        -> URD

The exact output may vary depending on tokenization and model predictions.

---

Methodology

The overall development process consisted of the following stages:

1. Data Collection

Code-switched Urdu-English sentences were collected from real-world sources including social media and messaging platforms.

2. Data Preparation

The collected sentences were organized and annotated according to the three target categories:

URD
ENG
MIX

3. Dataset Split

The dataset was divided into:

140 Training Sentences
        +
35 Testing Sentences

4. Tokenization

The text was tokenized using the XLM-RoBERTa tokenizer.

5. Label Alignment

Labels were aligned with the corresponding subword tokens generated by the tokenizer.

6. Fine-Tuning

xlm-roberta-base was fine-tuned using the prepared training dataset.

7. Evaluation

The trained model was evaluated on the held-out test set.

8. Final Performance

The model achieved:

99.70% overall token-level accuracy.

---

Limitations

Although the model achieved high accuracy on the current test set, there are several limitations to consider.

Small Dataset

The dataset contains only 175 sentences, which is relatively small for training and evaluating a robust multilingual NLP system.

Limited Test Set

Only 35 sentences were used for testing. Therefore, the reported 99.70% accuracy may not fully represent performance on unseen real-world data.

Domain Dependence

The data was collected primarily from informal communication platforms. Performance may differ on:

• Formal Urdu
• News articles
• Academic text
• Long documents
• Historical Urdu
• Highly specialized terminology

Roman Urdu Variation

Roman Urdu does not have a single standardized spelling system. The same Urdu word may be written in several different ways.

For example:

bohat
boht
bahut
bohut

Such variations can make language identification more difficult.

Mixed Words

Words combining Urdu and English can be ambiguous and may require contextual understanding.

---

Future Work

Several improvements can be explored in future versions:

Larger Dataset

Increase the dataset from hundreds of sentences to thousands or more.

More Data Sources

Include additional sources such as:

• Reddit
• Public forums
• News comments
• Online discussions

Improved Annotation

Develop more detailed annotation guidelines for ambiguous and mixed words.

Model Comparison

Compare XLM-RoBERTa with other multilingual and language-specific Transformer models.

Additional Metrics

Evaluate the model using:

• Precision
• Recall
• F1-score
• Per-class performance
• Confusion matrix

Data Augmentation

Generate additional realistic variations of Roman Urdu and code-switched sentences.

Better Generalization

Evaluate the model on completely unseen domains and larger external datasets.

---

Repository Information

Model: Amna-Shahzad/code-switching-codesaviours-si26-Amna

Base Model: xlm-roberta-base

Task: Token Classification

Labels:

URD
ENG
MIX

Training Dataset: 140 sentences

Testing Dataset: 35 sentences

Overall Accuracy: 99.70%

---

Author

Amna Shahzad

Project

CodeSaviours SI-26

Organization

Code Saviours (SMC-PRIVATE) Limited

---

License

This model is intended for research and educational use.

Please review the licensing terms of the underlying xlm-roberta-base model and any datasets used before using this model in a commercial application.

---

Acknowledgements

This project builds upon the work of the XLM-RoBERTa authors and the open-source Hugging Face Transformers ecosystem.

Special thanks to the developers and researchers who make multilingual NLP models and tools publicly available.

---

References

• XLM-RoBERTa: Unsupervised Cross-lingual Representation Learning at Scale
• Hugging Face Transformers documentation
• Hugging Face Datasets ecosystem

---