README.md

---
license: apache-2.0
language:
- en
- es
- ml
tags:
- sentiment-analysis
- text-classification
- code-switching
- code-mixed
- data-augmentation
- llm-augmentation
- spanglish
- spanish-english
- malayalam-english
- mbert
- xlm-roberta
- xlm-t
- low-resource
library_name: transformers
pipeline_tag: text-classification
---

# Code-Mixed Sentiment Analysis Models (LLM-augmented)

A working collection of **~104 multilingual BERT and XLM-T models for code-mixed (code-switched) sentiment analysis**,
covering Spanish–English (Spanglish) and Malayalam–English. Models are fine-tuned on a
mix of naturally occurring and LLM-generated (augmented) code-mixed data.

**Status: development repository.** This is the repository owner's research workspace, kept public for convenience. The checkpoints here are organized for the owner's iteration, not packaged as polished, benchmarked releases. Expect naming to be terse.

## Repository structure

Models and tokenizers are organized as Language / Type / Architecture / folder:

```
<Language-Pair>/<Data-Type>/<Architecture>/<model-or-tokenizer folder>
```

- **Language pair** — `Spanish-English`, `Malayalam-English`
- **Data type** — how the training data was produced:
  - `Natural` — naturally occurring code-mixed data (folder tag `NCM`)
  - `Synthetic` — fully LLM-generated code-mixed data (folder tag `SCM`)
  - `Mix` — a blend of natural and synthetic data (folder tag `MIX`)
  - `Zero-shot` — evaluated/trained in a zero-shot setup
- **Architecture** — `mBERT` or `XLM-T`
- **Kind** — a folder is a tokenizer if its name contains `tokenizer`; otherwise it's a
  model. Tokenizers are stored as their own folders next to the models (not merged in).

Example paths:

```
spanish-english/natural/mBERT/spanglish_NCM_..._mBERT_mBERT/       # model
spanish-english/natural/mBERT/spanglish_NCM_tokenizer_..._mBERT/   # its tokenizer
malayalam-english/mix/XLM-T/malay_MIX_..._mBERT_XLMT/              # an XLM-T model
malayalam-english/zero-shot/XLM-T/malay_zero_shot_XLMT/           # zero-shot XLM-T
```

Naming note: in the raw folder names the *architecture is the last arch token* — e.g.
`..._mBERT_XLMT` is an XLM-T model, `..._mBERT_mBERT` is an mBERT model.


### `important-models/`

A separately discovered directory of what are likely the most important models in this repo.
Like the folder below, these do **not** follow the clean `Language / Type / Architecture` layout,
but the folder naming is consistent enough to identify each model. These checkpoints were created
Feb 28 – Apr 11, 2024 (the majority on Apr 11, 2024).

### `preliminary-experiments/`

A separate top-level folder holding my earlier, informally organized experiments. It contains initial mBERT
Spanglish model runs, along with some zero-shot, few-shot, and natural baselines and initial
synthetic and mix models. These were created in February 2024. These folders do **not** follow the `Language / Type / Architecture / folder` layout used
elsewhere in the repo, but the folder naming is still fairly consistent, so you can generally tell
the language, data type, and architecture from a folder's name. 

### Rough timeline

The creation dates give a quick way to tell the informal folders apart:

- `preliminary-experiments/` — **February 2024**
- `important-models/` — **Feb 28 – Apr 11, 2024** (mostly Apr 11)
- everything in the structured `Language / Type / Architecture` tree — generally **March–May 2024**

## How to load a specific model

Because there are many models in one repo, pass the `subfolder` argument. Tokenizers live in
separate folders, so point the tokenizer loader at the matching tokenizer folder:

```python
from transformers import AutoModelForSequenceClassification, AutoTokenizer

REPO = "lindazeng979/codemixed-sentiment-models"

model = AutoModelForSequenceClassification.from_pretrained(
    REPO, subfolder="spanish-english/natural/mBERT/<model_folder_name>"
)
tokenizer = AutoTokenizer.from_pretrained(
    REPO, subfolder="spanish-english/natural/mBERT/<tokenizer_folder_name>"
)
```

Browse the file tree to find exact folder names:
https://huggingface.co/lindazeng979/codemixed-sentiment-models/tree/main

## Intended use & limitations

These models classify sentiment in code-mixed (code-switched) Spanglish and Malayalam–English
social media text. They are used to investigate **LLM-based data augmentation for low-resource,
code-switched sentiment analysis**, how training on naturally occurring code-mixed
data (`natural`), fully LLM-generated code-mixed data (`synthetic`), and blends of the two (`mix`)
affects downstream sentiment performance across mBERT and XLM-T backbones. For the data, experimental setup, and evaluation, see the
GitHub repo and the paper.

Limitations: checkpoints here are organized to accompany the paper rather than as individual releases, so results and quality vary across data types and architectures. Models
trained on synthetic-only data may carry artifacts of the generating LLM, and sentiment models
can reflect biases in their training data. 

## Citation

If you use these models, please cite the paper:

```bibtex
@inproceedings{zeng-2024-leveraging,
    title = "Leveraging Large Language Models for Code-Mixed Data Augmentation in Sentiment Analysis",
    author = "Zeng, Linda",
    editor = "Hale, James and Chawla, Kushal and Garg, Muskan",
    booktitle = "Proceedings of the Second Workshop on Social Influence in Conversations (SICon 2024)",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.sicon-1.6/",
    doi = "10.18653/v1/2024.sicon-1.6",
    pages = "85--101"
}
```
**Paper:** https://aclanthology.org/2024.sicon-1.6/

**Data & code:** https://github.com/lindazeng979/LLM-CMSA