| --- |
| license: apache-2.0 |
| language: |
| - en |
| - zh |
| - es |
| - ur |
| tags: |
| - lora |
| - aya |
| - tiny-aya |
| - multilingual |
| - code |
| - legesher |
| - tiny-aya-expedition |
| - language-decoded |
| - unsloth |
| library_name: transformers |
| base_model: |
| - CohereLabs/tiny-aya-base |
| pipeline_tag: text-generation |
| --- |
| |
| # Language Decoded LoRA |
|
|
| QLoRA adapters fine-tuned on multilingual code conditions for the **Language Decoded** project (part of [Cohere's Tiny Aya Expedition](https://aya.for.ai)). |
|
|
| ## ⚠️ Phase 3 eval numbers — read the experiments repo before citing |
|
|
| Original Phase 3 `_summary_*.json` files on [`legesher/language-decoded-experiments`](https://huggingface.co/datasets/legesher/language-decoded-experiments) **under-report cond-5 SIB-200 accuracy by 20–35pp** because the strict inference-time extractor refused native-script answers. Cite the `_summary_reparsed_*.json` siblings (refined extractor) instead. **Five** Phase 3 SIB-200 conclusions also flip win→loss against baseline once the extractor is corrected (`cond-2-es-5k`, `cond-2-es-20k`, `cond-2-ur-20k`, `cond-2-zh-20k`, `cond-3-zh-5k`), and `cond-2-ur-5k`'s gain deflates 4.4×. See the [banner on the experiments repo](https://huggingface.co/datasets/legesher/language-decoded-experiments) (top of the README) for the full picture. |
|
|
| ## Research Question |
|
|
| > Does fine-tuning on non-English code improve multilingual reasoning — and is the benefit language-dependent or structure-dependent? |
|
|
| ## Base Model |
|
|
| All adapters are trained on [CohereLabs/tiny-aya-base](https://huggingface.co/CohereLabs/tiny-aya-base) (3.35B parameters). |
|
|
| ## Model Structure |
|
|
| This repo is the canonical hub for the trained-from-scratch LoRA adapters, organized by experimental condition: |
|
|
| | Subdirectory | Condition | Training Data | |
| | --------------------- | ----------- | ----------------------------------------------------- | |
| | `condition-1-en-32k/` | Condition 1 | English Python from The Stack Dedup (full 32k corpus) | |
| | `condition-1-en-5k/` | Condition 1 | English Python from The Stack Dedup (5k subset) | |
| | `condition-2-zh-5k/` | Condition 2 | Chinese keyword-swapped Python (Legesher-transpiled) | |
| | `condition-2-es-5k/` | Condition 2 | Spanish keyword-swapped Python (Legesher-transpiled) | |
| | `condition-2-ur-5k/` | Condition 2 | Urdu keyword-swapped Python (Legesher-transpiled) | |
| | `condition-3-zh-5k/` | Condition 3 | Transpiled + native Chinese code (blended) | |
|
|
| **Cond-5 (cross-lingual transfer)** is an evaluation pattern that re-uses condition-2 adapters with cross-language prompting — see [`phase3/conditions/condition-5-{zh,es,ur}-5k/`](https://huggingface.co/datasets/legesher/language-decoded-experiments/tree/main/phase3/conditions) on the experiments repo for the cross-lingual eval results. |
|
|
| ### The Experimental Ladder |
|
|
| - **Baseline --> 1**: Does code help at all? |
| - **1 --> 2**: Does the language of keywords matter? |
| - **2 --> 3**: Does diversity of native-language sources add value beyond keyword swap? |
| - **3 --> 4**: Does code written in the cultural context of a language carry unique signal? |
| - **--> 5**: Does shared script or language family create transfer effects when an adapter trained on one language is prompted in another? |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| from peft import PeftModel |
| |
| # Load base model |
| base_model = AutoModelForCausalLM.from_pretrained("CohereLabs/tiny-aya-base") |
| tokenizer = AutoTokenizer.from_pretrained("CohereLabs/tiny-aya-base") |
| |
| # Load a LoRA adapter (e.g., Condition 1 — English code) |
| model = PeftModel.from_pretrained(base_model, "legesher/language-decoded-lora", subfolder="condition-1-en-5k") |
| |
| # Load a language-specific adapter (e.g., Condition 2 — Chinese keyword-swapped) |
| model = PeftModel.from_pretrained(base_model, "legesher/language-decoded-lora", subfolder="condition-2-zh-5k") |
| ``` |
|
|
| ## Training Details |
|
|
| | Parameter | Value | |
| | ------------------ | ------------------------------------------------------------------------------------------------ | |
| | Base model | [CohereLabs/tiny-aya-base](https://huggingface.co/CohereLabs/tiny-aya-base) (3.35B params) | |
| | Method | QLoRA 4-bit (NF4), ~5.4GB VRAM | |
| | Hardware | Kaggle T4 (16GB) | |
| | Tokenizer | CohereLabs/tiny-aya-base | |
| | Transpilation tool | [Legesher](https://github.com/legesher/legesher) v0.7.3 | |
| | Training data | [legesher/language-decoded-data](https://huggingface.co/datasets/legesher/language-decoded-data) | |
|
|
| ### QLoRA Hyperparameters |
|
|
| | Parameter | Value | |
| | --------------- | ------------------------------------------------------------- | |
| | LoRA rank (`r`) | 16 | |
| | LoRA alpha | 32 | |
| | LoRA dropout | 0.0 | |
| | Target modules | q_proj, k_proj, v_proj, o_proj, up_proj, down_proj, gate_proj | |
| | Bias | none | |
| | Task type | CAUSAL_LM | |
| | PEFT version | 0.18.1 | |
| | Quantization | NF4 (4-bit) via Unsloth | |
|
|
| ## Evaluation |
|
|
| Models are evaluated on multilingual reasoning benchmarks with dual prompts (English + language-specific). Phase 3 adds SIB-200 and Belebele to the Phase 2 benchmark set. |
|
|
| | Benchmark | What it measures | Phase | Examples per language | |
| | --------- | -------------------------- | ----- | --------------------- | |
| | MGSM | Math reasoning | 2, 3 | 250 | |
| | X-CSQA | Commonsense reasoning | 2, 3 | ~1,000 | |
| | XNLI | Natural language inference | 2, 3 | ~5,000 | |
| | SIB-200 | Topic classification | 3 | ~204 | |
| | Belebele | Reading comprehension | 3 | ~900 | |
|
|
| Eval results live at [`legesher/language-decoded-experiments`](https://huggingface.co/datasets/legesher/language-decoded-experiments). **Cite `_summary_reparsed_*.json` files for Phase 3 numbers** — see the banner above. |
| |
| ## Limitations |
| |
| - **Single base model**: All adapters are trained on CohereLabs/tiny-aya-base (3.35B params). Results may not generalize to larger or architecturally different models. |
| - **Limited training data**: Each condition uses a 5k-file subset for QLoRA fine-tuning, constrained by Kaggle T4 hardware limits. |
| - **Evaluation scope**: Currently evaluated on 5 benchmarks (MGSM, X-CSQA, XNLI, SIB-200, Belebele). Other reasoning tasks may show different patterns. |
| - **Consumer hardware**: Training on Kaggle T4 (16GB) with 4-bit quantization introduces approximation that may affect adapter quality compared to full-precision training. |
| - **Extractor coverage**: Phase 3 inference-time extractor under-counts native-script SIB-200 answers; refined post-hoc extractor recovers them. See the banner above and [`expedition-tiny-aya/analysis/phase-3/phase3-refined-evaluation.md`](https://github.com/legesher/research/blob/main/expedition-tiny-aya/analysis/phase-3/phase3-refined-evaluation.md) on the research repo. |
| |
| ## Related Resources |
| |
| - **Training data**: [legesher/language-decoded-data](https://huggingface.co/datasets/legesher/language-decoded-data) |
| - **Community code**: [legesher/language-decoded-community](https://huggingface.co/datasets/legesher/language-decoded-community) |
| - **Experiment tracking**: [legesher/language-decoded-experiments](https://huggingface.co/datasets/legesher/language-decoded-experiments) |
| - **Transpilation tool**: [Legesher on GitHub](https://github.com/legesher/legesher) |
| |
| ## Citation |
| |
| ```bibtex |
| @misc{language-decoded-2026, |
| title={Language Decoded: Investigating Language-Dependent vs. Structure-Dependent Reasoning Benefits of Code}, |
| author={Madison Edgar and Saad Ahmed Bazaz and Tom Sherborne and Rashik Shahjahan and Khojasteh Mirza and Sarah Jawaid and Rafay Mustafa and Sohaib Ahmed Bazaz}, |
| year={2026}, |
| publisher={Hugging Face}, |
| url={https://huggingface.co/legesher/language-decoded-lora} |
| } |
| ``` |
| |
| ## License |
| |
| Apache 2.0 |
| |
