--- license: apache-2.0 language: - en - zh - es - ur tags: - lora - aya - tiny-aya - multilingual - code - legesher - tiny-aya-expedition - language-decoded - unsloth library_name: transformers base_model: - CohereLabs/tiny-aya-base pipeline_tag: text-generation --- # Language Decoded LoRA QLoRA adapters fine-tuned on multilingual code conditions for the **Language Decoded** project (part of [Cohere's Tiny Aya Expedition](https://aya.for.ai)). ## ⚠️ Phase 3 eval numbers — read the experiments repo before citing Original Phase 3 `_summary_*.json` files on [`legesher/language-decoded-experiments`](https://huggingface.co/datasets/legesher/language-decoded-experiments) **under-report cond-5 SIB-200 accuracy by 20–35pp** because the strict inference-time extractor refused native-script answers. Cite the `_summary_reparsed_*.json` siblings (refined extractor) instead. **Five** Phase 3 SIB-200 conclusions also flip win→loss against baseline once the extractor is corrected (`cond-2-es-5k`, `cond-2-es-20k`, `cond-2-ur-20k`, `cond-2-zh-20k`, `cond-3-zh-5k`), and `cond-2-ur-5k`'s gain deflates 4.4×. See the [banner on the experiments repo](https://huggingface.co/datasets/legesher/language-decoded-experiments) (top of the README) for the full picture. ## Research Question > Does fine-tuning on non-English code improve multilingual reasoning — and is the benefit language-dependent or structure-dependent? ## Base Model All adapters are trained on [CohereLabs/tiny-aya-base](https://huggingface.co/CohereLabs/tiny-aya-base) (3.35B parameters). ## Model Structure This repo is the canonical hub for the trained-from-scratch LoRA adapters, organized by experimental condition: | Subdirectory | Condition | Training Data | | --------------------- | ----------- | ----------------------------------------------------- | | `condition-1-en-32k/` | Condition 1 | English Python from The Stack Dedup (full 32k corpus) | | `condition-1-en-5k/` | Condition 1 | English Python from The Stack Dedup (5k subset) | | `condition-2-zh-5k/` | Condition 2 | Chinese keyword-swapped Python (Legesher-transpiled) | | `condition-2-es-5k/` | Condition 2 | Spanish keyword-swapped Python (Legesher-transpiled) | | `condition-2-ur-5k/` | Condition 2 | Urdu keyword-swapped Python (Legesher-transpiled) | | `condition-3-zh-5k/` | Condition 3 | Transpiled + native Chinese code (blended) | **Cond-5 (cross-lingual transfer)** is an evaluation pattern that re-uses condition-2 adapters with cross-language prompting — see [`phase3/conditions/condition-5-{zh,es,ur}-5k/`](https://huggingface.co/datasets/legesher/language-decoded-experiments/tree/main/phase3/conditions) on the experiments repo for the cross-lingual eval results. ### The Experimental Ladder - **Baseline --> 1**: Does code help at all? - **1 --> 2**: Does the language of keywords matter? - **2 --> 3**: Does diversity of native-language sources add value beyond keyword swap? - **3 --> 4**: Does code written in the cultural context of a language carry unique signal? - **--> 5**: Does shared script or language family create transfer effects when an adapter trained on one language is prompted in another? ## Usage ```python from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel # Load base model base_model = AutoModelForCausalLM.from_pretrained("CohereLabs/tiny-aya-base") tokenizer = AutoTokenizer.from_pretrained("CohereLabs/tiny-aya-base") # Load a LoRA adapter (e.g., Condition 1 — English code) model = PeftModel.from_pretrained(base_model, "legesher/language-decoded-lora", subfolder="condition-1-en-5k") # Load a language-specific adapter (e.g., Condition 2 — Chinese keyword-swapped) model = PeftModel.from_pretrained(base_model, "legesher/language-decoded-lora", subfolder="condition-2-zh-5k") ``` ## Training Details | Parameter | Value | | ------------------ | ------------------------------------------------------------------------------------------------ | | Base model | [CohereLabs/tiny-aya-base](https://huggingface.co/CohereLabs/tiny-aya-base) (3.35B params) | | Method | QLoRA 4-bit (NF4), ~5.4GB VRAM | | Hardware | Kaggle T4 (16GB) | | Tokenizer | CohereLabs/tiny-aya-base | | Transpilation tool | [Legesher](https://github.com/legesher/legesher) v0.7.3 | | Training data | [legesher/language-decoded-data](https://huggingface.co/datasets/legesher/language-decoded-data) | ### QLoRA Hyperparameters | Parameter | Value | | --------------- | ------------------------------------------------------------- | | LoRA rank (`r`) | 16 | | LoRA alpha | 32 | | LoRA dropout | 0.0 | | Target modules | q_proj, k_proj, v_proj, o_proj, up_proj, down_proj, gate_proj | | Bias | none | | Task type | CAUSAL_LM | | PEFT version | 0.18.1 | | Quantization | NF4 (4-bit) via Unsloth | ## Evaluation Models are evaluated on multilingual reasoning benchmarks with dual prompts (English + language-specific). Phase 3 adds SIB-200 and Belebele to the Phase 2 benchmark set. | Benchmark | What it measures | Phase | Examples per language | | --------- | -------------------------- | ----- | --------------------- | | MGSM | Math reasoning | 2, 3 | 250 | | X-CSQA | Commonsense reasoning | 2, 3 | ~1,000 | | XNLI | Natural language inference | 2, 3 | ~5,000 | | SIB-200 | Topic classification | 3 | ~204 | | Belebele | Reading comprehension | 3 | ~900 | Eval results live at [`legesher/language-decoded-experiments`](https://huggingface.co/datasets/legesher/language-decoded-experiments). **Cite `_summary_reparsed_*.json` files for Phase 3 numbers** — see the banner above. ## Limitations - **Single base model**: All adapters are trained on CohereLabs/tiny-aya-base (3.35B params). Results may not generalize to larger or architecturally different models. - **Limited training data**: Each condition uses a 5k-file subset for QLoRA fine-tuning, constrained by Kaggle T4 hardware limits. - **Evaluation scope**: Currently evaluated on 5 benchmarks (MGSM, X-CSQA, XNLI, SIB-200, Belebele). Other reasoning tasks may show different patterns. - **Consumer hardware**: Training on Kaggle T4 (16GB) with 4-bit quantization introduces approximation that may affect adapter quality compared to full-precision training. - **Extractor coverage**: Phase 3 inference-time extractor under-counts native-script SIB-200 answers; refined post-hoc extractor recovers them. See the banner above and [`expedition-tiny-aya/analysis/phase-3/phase3-refined-evaluation.md`](https://github.com/legesher/research/blob/main/expedition-tiny-aya/analysis/phase-3/phase3-refined-evaluation.md) on the research repo. ## Related Resources - **Training data**: [legesher/language-decoded-data](https://huggingface.co/datasets/legesher/language-decoded-data) - **Community code**: [legesher/language-decoded-community](https://huggingface.co/datasets/legesher/language-decoded-community) - **Experiment tracking**: [legesher/language-decoded-experiments](https://huggingface.co/datasets/legesher/language-decoded-experiments) - **Transpilation tool**: [Legesher on GitHub](https://github.com/legesher/legesher) ## Citation ```bibtex @misc{language-decoded-2026, title={Language Decoded: Investigating Language-Dependent vs. Structure-Dependent Reasoning Benefits of Code}, author={Madison Edgar and Saad Ahmed Bazaz and Tom Sherborne and Rashik Shahjahan and Khojasteh Mirza and Sarah Jawaid and Rafay Mustafa and Sohaib Ahmed Bazaz}, year={2026}, publisher={Hugging Face}, url={https://huggingface.co/legesher/language-decoded-lora} } ``` ## License Apache 2.0