🚧 Work in progress. This repository is the first evaluation stage of an ongoing project. A full paper is in preparation, and fine-tuning experiments are coming next. The dataset, results, and analysis will be expanded and updated.

📌 At a glance

1. The blind spot

Yoruba-Nigerian Pidgin code-switching is a common feature of everyday communication in Lagos and across South-West Nigeria, occurring in families, markets, workplaces, social interactions, and digital communication. Speakers may switch between Yoruba and Nigerian Pidgin within the same sentence or conversation to express meaning, emphasis, identity, or emotion.

Although Yoruba–Pidgin code-switching has been documented in sociolinguistic research and appears in broader Nigerian language resources, it remains relatively underexplored from an NLP perspective, particularly in the evaluation of small language models. Existing benchmarks often focus on individual languages or relatively clean, monolingual inputs, making it difficult to assess how models handle the mixed and informal language patterns found in real-world Nigerian communication.

This creates an important blind spot: a model may perform well on standard Yorùbá or Nigerian Pidgin benchmarks while still struggling to understand code-switched text. Evaluating small language models on Yoruba–Nigerian Pidgin code-switching can therefore provide a more realistic assessment of their ability to handle everyday multilingual communication.

Research question: How well can a small language model translate Yoruba–Nigerian Pidgin code-switched text into English?

2. Evaluation

2.1 Why this model

Qwen3.5-4B is a recent (2026), widely used open-weight model from the Qwen family, and it fits on a single Colab GPU. Its model card claims support for 201 languages and dialects.

2.2 Dataset

• Source: Code-switched Yoruba–Nigerian Pidgin sentences written and validated by the author, a native Yoruba speaker and Nigerian Pidgin speaker.
• Size: 30 validated sentences: 3 used as few-shot examples, 27 used as the test set (the same 27 for both conditions).
• Domain: Education (all 30 items in this release)
• Preprocessing: Unicode NFC normalisation, so Yoruba tone marks are encoded consistently; whitespace cleanup.

Loading the data:

import pandas as pd
df = pd.read_csv("validated_30.csv")

2.3 Setup

You are an expert translator fluent in Yoruba, Nigerian Pidgin, and English. The sentence below is code-mixed: it combines Yoruba and Nigerian Pidgin in the same sentence, as people commonly speak in Lagos and South-West Nigeria. Translate the whole sentence into clear, natural English, preserving the full meaning, including tone, idioms, and cultural references. Do not add explanations, notes, or alternatives. Output only the English translation.

2.4 Metrics

Human meaning score rubric:

2.5 📊 Results

2.6 What goes wrong: error analysis

In the zero-shot condition, the model did not translate the Yoruba parts of a sentence. In the example below, it left the Yoruba phrase Olùkọ́ eré yan untranslated and copied it into the output (as "Olukore eré yan"), while translating the surrounding words into English.

2.7 Hypothesis and conclusion

Hypothesis

It was hypothesised that the small language model would struggle to translate Yoruba–Nigerian Pidgin code-switched text because of limited exposure to low-resource and code-switched data. It was also expected that few-shot prompting would improve translation quality by providing the model with relevant examples.

Conclusion

The results support the hypothesis. Few-shot prompting improved translation performance across all four evaluation metrics. BLEU increased from 27.8 to 30.8, chrF from 55.6 to 58.2, BERTScore from 93.6 to 94.7, and the human meaning score from 3.56 to 3.78. However, the improvement in human-rated meaning was relatively modest (+0.22), suggesting that providing three examples improved surface-level translation quality more than it resolved deeper meaning-related errors.

Overall, the results suggest that the small language model can translate Yoruba–Nigerian Pidgin code-switched text to some extent, and that few-shot prompting provides a measurable improvement. However, few-shot examples alone do not fully address the challenges of translating this type of low-resource, code-switched language.

3. Path forward

To close this gap, I plan to take three steps:

1. Collect more Yoruba–Nigerian Pidgin code-switched data that reflects how people actually speak, so models can learn from real mixed-language examples.
2. Fine-tune the model with LoRA on this code-switched data, to test whether targeted training improves translation beyond what few-shot prompting achieved.
3. Compare different model sizes, to understand whether the gap narrows as models get larger or whether code-switching remains a challenge regardless of size.

📁 Files

📖 Citation

If you use this dataset or evaluation, please cite:

@misc{olusanya2026whenlagostalks,
  title        = {When Lagos Talks: Evaluating Yoruba--Nigerian Pidgin Code-Switching in a Small Frontier LLM},
  author       = {Joy Olusanya},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/buckets/NaomiNLP/when-lagos-talks}},
}

📬 Contact

Joy (Naomi) Olusanya · LinkedIn · Hugging Face