shkuruba/heritage-language-llm

Adaptive Heritage Language Learning for Low-Resource Languages via RAG

1. Introduction

Growing up as a first-generation American, I spent my early childhood surrounded by Telugu at home, but once school started, English took over and Telugu faded. Now I can understand my grandparents when they speak to me, but responding is a whole different challenge. This experience is not unique to Telugu, it describes a lot of people in diaspora communities who grew up hearing a language at home but lost active fluency, and who have almost nowhere to turn because tools like Duolingo, Babbel, and Rosetta Stone don't offer Telugu, Yoruba, or Quechua at all. The goal of this project is to build an adaptive RAG-based tool that gives heritage learners of these three languages targeted feedback on their written practice, along with follow-up exercises that adapt to their specific error patterns over time. The core problem is that general-purpose LLMs are not reliable for this, these are low-resource languages where even strong models hallucinate grammar rules without grounding (Ahuja et al., 2023), and no existing tool tracks a learner's history across sessions. Using Mistral-7B-Instruct-v0.3 with a hybrid dense embedding and BM25 re-ranking retrieval setup, the pipeline improved overall scoring from 2.60/3 pre-RAG to 2.93/3 post-RAG on a manually constructed test split, with the biggest gains for Quechua (+0.6), the most low-resource of the three languages.

2. Data

The RAG knowledge base has grammar references, common error patterns, and corrected writing examples for all three languages, built from open-source corpora. All entries were manually restructured from the original corpus format into plain-language grammar rule descriptions with correct and incorrect usage examples.

Knowledge base sources:

• Telugu (5 entries): AI4Bharat IndicNLP catalog -- 280-million-token corpus and parallel data across 22 Indic languages
• Yoruba (5 entries): Yankari dataset and MENYO-20k (Adelani et al., 2021) -- monolingual and parallel sentence data covering grammar rules, tone mark requirements, and corrected examples
• Quechua (6 entries): Siminchik corpus and QuBERT monolingual corpus

The original plan also included generating 600 synthetic learner writing samples via GPT-4o (200 per language) to add heritage-specific error patterns to the knowledge base, but the current version uses the 16 manually constructed entries as the retrieval corpus. The pipeline also keeps a persistent learner profile as a JSON document that tracks which error categories have come up and how often across sessions, and this profile is pulled into the prompt alongside the retrieved entries at the start of each session. For evaluation, 15 test cases were built by hand (5 per language), each with a learner passage containing real heritage speaker errors, the ideal retrieved context, and a ground truth response.

3. Methodology

This project uses RAG with a hybrid dense embedding and BM25 re-ranking setup. The learner's written passage is encoded using paraphrase-multilingual-mpnet-base-v2, which handles cross-lingual retrieval across 50+ languages and was chosen because the learner queries are in Telugu, Yoruba, or Quechua script while the knowledge base is in English. The top 8 semantic candidates are retrieved via cosine similarity, then re-ranked with BM25 to prioritize entries that match the grammar terminology in the query. The top 3 entries are passed to the generator along with the learner's current profile.

Three retrieval setups were tested:

1. TF-IDF with cosine similarity -- 2.87/3
2. Dense embeddings with cosine similarity -- 2.80/3
3. Dense embeddings with BM25 re-ranking -- 2.93/3 (best)

The hybrid approach won mainly because BM25 re-ranking improved practice exercise inclusion to a perfect 1.0. Mistral-7B-Instruct-v0.3 was chosen as the generator based on empirical results from check-in 3, where it outperformed Qwen2.5-1.5B and Llama-3.2-3B across zero-shot, three-shot, and six-shot settings on this task. It is loaded with 4-bit quantization via bitsandbytes (nf4, double quant, float16 compute dtype) to reduce GPU memory requirements.

4. Evaluation

The original benchmark plan included three RAG-specific benchmarks alongside a testing split:

• RAGAS (Es et al., 2024) -- evaluates faithfulness, answer relevancy, context precision, and context recall without needing human-labeled ground truth, which is especially useful given how limited labeled data is for these languages
• MIRAGE-Bench (Thakur et al., 2025) -- evaluates multilingual RAG answer generation across 18 languages including Telugu and Yoruba, making it the most directly relevant external benchmark for this project
• NoMIRACL (Thakur et al., 2024) -- tests whether a RAG system admits it does not know when retrieved documents do not contain a relevant answer, rather than hallucinating

In practice, RAGAS could not be installed in the project environment due to a dependency conflict, and MIRAGE-Bench requires external infrastructure that was not available. The final evaluation used a manual scoring approach on the 15-entry test split plus a NoMIRACL-aligned custom abstention test, which together cover the most critical qualities for this specific task.

Manual scoring criteria (each worth 1 point, max score 3):

• Error identified: did the model correctly flag the error, or correctly note there was no error
• Correct explanation: did the explanation match the ground truth (measured by keyword overlap above 15%)
• Practice included: did the model include a follow-up exercise

The two comparison models, Qwen2.5-1.5B and Llama-3.2-3B, were chosen because they were tested alongside Mistral 7B in check-in 3 on this exact task, making them a direct and relevant comparison. Note: scores for those two models are approximated from the check-in 3 six-shot qualitative assessments using the same scoring rubric, since they were not run through the full check-in 4 evaluation pipeline.

Model	Error Identified	Correct Explanation	Practice Included	Avg Score	Abstention
Mistral-7B + RAG (this project)	1.00	0.93	1.00	2.93/3	False (both)
Mistral-7B (no RAG, base)	1.00	0.87	0.73	2.60/3	Mixed (1/2)
Llama-3.2-3B (six-shot, no RAG)*	1.00	0.80	0.80	2.60/3	N/A
Qwen2.5-1.5B (six-shot, no RAG)*	0.87	0.73	0.73	2.33/3	N/A

*Approximated from check-in 3 six-shot qualitative assessments. Not run through the full 15-prompt evaluation.

The RAG pipeline scored highest overall and was the only setup to hit a perfect 1.0 on practice exercise inclusion. Adding RAG to Mistral 7B improved explanation quality and exercise inclusion compared to the base model without retrieval, showing the retrieved grammar references are doing real work. The smaller models without retrieval both scored lower, which makes sense since they have less capacity to handle low-resource language grammar accurately on their own.

5. Usage and Intended Uses

This pipeline is for heritage speakers of Telugu, Yoruba, and Quechua who want to practice and improve their written skills in their heritage language. It is built for people who can understand the language but struggle to produce it -- not for complete beginners starting from scratch, and not for fluent speakers looking for advanced material. The pipeline requires a GPU with at least 10GB VRAM using 4-bit quantization. A learner profile JSON file should be saved and loaded per user across sessions to get the adaptive behavior.

from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline, BitsAndBytesConfig
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity
from rank_bm25 import BM25Okapi
import numpy as np
import torch, json

# load Mistral 7B with 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16
)
model_name = "mistralai/Mistral-7B-Instruct-v0.3"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=bnb_config, device_map="auto")
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, max_new_tokens=300, do_sample=False)

# load retrieval model
embed_model = SentenceTransformer("paraphrase-multilingual-mpnet-base-v2")

# load learner profile (create a new one if first session)
try:
    with open("learner_profile.json") as f:
        learner_profile = json.load(f)
except FileNotFoundError:
    learner_profile = {
        "sessions_completed": 0,
        "error_counts": {"grammar": 0, "vocabulary": 0, "syntax": 0, "script": 0},
        "focus_area": "none yet -- first session"
    }

6. Prompt Format

The prompt has three parts passed as a user message: the learner's current profile (error counts and focus area from past sessions), the top 3 retrieved knowledge base entries, and the learner's written practice passage. Here is a real example for a Telugu learner:

[System]: You are a language tutor helping heritage speakers improve their written language skills.
Heritage speakers grew up hearing the language at home but lost active fluency. When given a passage
in the learner heritage language, identify specific errors, label each by category (grammar,
vocabulary, syntax, or script), give a brief explanation, and provide a short follow-up practice
exercise targeting the most common error type. If you are not confident about the grammar rules
for this specific language, say so clearly rather than guessing.

[User]:
Learner profile: 3 sessions completed. Error counts so far -- grammar: 2, vocabulary: 1,
syntax: 0, script: 0. Current focus area: grammar.

Relevant grammar references:
Reference 1: Telugu grammar rule: subject-verb agreement. First-person singular subject nenu (I)
requires verb endings in aanu for past tense such as chuushaanu (I watched). The third-person
ending indi (she/he did) is a common heritage speaker error.
Reference 2: Telugu corrected example. Learner wrote: nenu ninna maa amma vanta chesindi.
Correction: nenu ninna maa ammatO vanta cheshaanu. The error: chesindi is third-person but
subject is nenu (I) which needs first-person form cheshaanu.
Reference 3: Telugu vocabulary: heritage speaker pattern of using English borrowings. Common
borrowings: skuul (school) becomes paThashaala, paark (park) becomes udyaanavanam.

Now answer the following:
Review this Telugu passage and identify any errors:
నేను నా స్నేహితుడు తో సినిమా చూసింది. మేము చాలా ఆనందించాయి.

7. Expected Output Format

The model gives structured feedback identifying each error by category with a plain-language explanation, followed by a practice exercise. After each session the learner profile JSON is updated to reflect which error categories came up. Here is what a real response looks like for the prompt above:

Error 1 -- Grammar (verb agreement): 'చూసింది' should be 'చూశాను'.
The subject 'నేను' (I) is first-person singular and requires 'చూశాను' (I watched),
not 'చూసింది' which is third-person (she/he watched).

Error 2 -- Grammar (verb form for 'we'): 'ఆనందించాయి' is the verb form used when
talking about things, not people. Since 'మేము' means 'we', the correct form is 'ఆనందించాము'.

Practice Exercise: Correct the verb forms in these sentences:
1. నేను పాట పాడింది.
2. మేము ఇంటికి వెళ్ళాయి.

8. Limitations

1. The Quechua knowledge base is much smaller than the other two, which shows up in the lower post-RAG scores for Quechua (2.8/3 vs 3.0/3 for Telugu and Yoruba). More Quechua entries would be the most straightforward fix.

2. The pipeline does not reliably say it does not know when retrieval comes up short. Post-RAG, both Quechua abstention prompts returned False for expressed uncertainty, meaning the model produced confident-sounding feedback even when the retrieved context was not a good match. For a language learning tool this is a real problem since wrong grammar explanations can mislead learners.

3. The manual scoring rubric has limits. The 15% keyword overlap threshold for correct explanation is a rough measure and can miss genuinely good responses that just use different wording than the ground truth.

4. The pipeline only handles written input and does not address the spoken side of heritage language loss, which for a lot of learners (myself included) is where the disconnect is most felt.