--- license: apache-2.0 language: - en - zh - es - ur tags: - lora - aya - tiny-aya - multilingual - code - legesher - tiny-aya-expedition - language-decoded - unsloth library_name: transformers base_model: - CohereLabs/tiny-aya-base pipeline_tag: text-generation --- # Language Decoded LoRA QLoRA adapters fine-tuned on multilingual code conditions for the **Language Decoded** project (part of [Cohere's Tiny Aya Expedition](https://aya.for.ai)). > **Submitted paper title (2026-05-26):** _Language, Decoded: Exploring the Impact of Fine-Tuning a Multilingual Model on Native-Language Code_ ## ⚠️ Phase 3 eval numbers — read the experiments repo before citing Original Phase 3 `_summary_*.json` files on [`legesher/language-decoded-experiments`](https://huggingface.co/datasets/legesher/language-decoded-experiments) **under-report cond-5 SIB-200 accuracy by 20–35pp** because the strict inference-time extractor refused native-script answers. Cite the `_summary_reparsed_*.json` siblings (refined extractor) instead. **Five** Phase 3 SIB-200 conclusions also flip win→loss against baseline once the extractor is corrected (`cond-2-es-5k`, `cond-2-es-20k`, `cond-2-ur-20k`, `cond-2-zh-20k`, `cond-3-zh-5k`), and `cond-2-ur-5k`'s gain deflates 4.4×. See the [banner on the experiments repo](https://huggingface.co/datasets/legesher/language-decoded-experiments) (top of the README) for the full picture. ## Research Question > **How does fine-tuning Tiny Aya on non-English code — whether transpiled, mixed-native, or fully translated — affect its multilingual reasoning and instruction-following, and how does that impact _differ_ from fine-tuning on English code?** The hypothesis is **not** that non-English code matches or exceeds English code as a generic reasoning aid — rather, that the _kind_ of effect non-English code produces depends on the target language, the data structure, and how the corpus was constructed. See [legesher/language-decoded-experiments](https://huggingface.co/datasets/legesher/language-decoded-experiments) for the full project context. ## Base Model All adapters are trained on [CohereLabs/tiny-aya-base](https://huggingface.co/CohereLabs/tiny-aya-base) (3.35B parameters). Tiny Aya was chosen because it is small (deployable on a single 16 GB T4 GPU via QLoRA), accessible (Apache 2.0-licensed), and supports 70+ languages with explicit emphasis on lower-resourced ones — which makes the experimental ladder viable for `ur` at all. ## Adapter Inventory Each subdirectory is one trained condition × file-volume × seed combination. All adapters share the QLoRA hyperparameters listed under [Training Details](#training-details). | Subdirectory | Condition | Training data | Seeds | | ----------------------------------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `condition-1-en-5k-seed{42,123,456}/` | 1 | Raw English Python from `bigcode/the-stack-v2-dedup` (5k file subset) | 42, 123, 456 | | `condition-1-en-20k-seed42/` | 1 | Raw English Python (20k file subset) | 42 | | `condition-2-{zh,es,ur}-5k-seed{42,123,456}/` | 2 | The **same 5k subset as cond-1**, processed through Legesher v0.7.3 — Python's reserved words (keywords, exceptions, built-in functions, numerical system for some target languages) translated to the target language; user logic preserved | 42, 123, 456 | | `condition-2-{zh,es,ur}-20k-seed42/` | 2 | The **same 20k subset as cond-1**, processed through Legesher v0.7.3 | 42 | | `condition-3-zh-5k-native-code-seed42/` | 3 | Community-collected raw Chinese code from varied online public-source repositories (different source-file population from cond-1/2/5 by design) | 42 | | `condition-5-{zh,es,ur}-5k-c4ai-aya-expanse-32b-seed42/` | 5 | The **same 5k subset as cond-1**, first transpiled by Legesher v0.7.3 to translate Python's reserved words, then run through `c4ai-aya-expanse-32b` via the Cohere API to translate the remaining content (identifiers, comments, docstrings, string literals) | 42 | **Condition 4 ("Community-Contributed Native Code")** is pending sufficient direct community contributions to the [`legesher/legesher-native-code`](https://huggingface.co/spaces/legesher/legesher-native-code) HF Space; no cond-4 adapter exists yet. ### Source-file control Cond-1, cond-2, and cond-5 all train on the **same 5,000-file subset** drawn from `bigcode/the-stack-v2-dedup` (with a parallel 20k subset for the 20k tier). Differences across these conditions reflect the processing pipeline (raw / transpiled / fully translated), not file-quality or content drift. Cond-3 is the deliberate exception — its source files are a different population by design. ### The experimental ladder - **Baseline → cond-1**: Does code help at all? (Replicates [Aryabumi et al., 2024](https://arxiv.org/abs/2408.10914).) - **Cond-1 → cond-2**: Does translating Python's reserved words (keywords, exceptions, built-in functions, numerical system for some target languages) into the target language change the model's behavior? User logic and library calls remain English-derived. - **Cond-2 → cond-3**: Does code pulled from real-world public-source repositories — code humans actually wrote in or with the target language — add value beyond Legesher's mechanical translation? - **Cond-2 → cond-5**: Cond-2 translates only Python's reserved words; cond-5 goes further by translating the rest of the file's content (identifiers, comments, docstrings, string literals) via `c4ai-aya-expanse-32b`. Logic and structure are preserved. - **Cond-3 → cond-5** (implicit): Human-authored vs. machine-synthesized native code. For the full ladder including future directions (natural-language text control, combined-language training, similar-script evaluation), see [legesher/language-decoded-experiments](https://huggingface.co/datasets/legesher/language-decoded-experiments). ## Usage ```python from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel # Load base model base_model = AutoModelForCausalLM.from_pretrained("CohereLabs/tiny-aya-base") tokenizer = AutoTokenizer.from_pretrained("CohereLabs/tiny-aya-base") # Load a LoRA adapter — e.g., cond-1 (English code, seed 42, 5k tier) model = PeftModel.from_pretrained( base_model, "legesher/language-decoded-lora", subfolder="condition-1-en-5k-seed42", ) # Or a language-specific cond-2 adapter (Chinese reserved-word translation, seed 42) model = PeftModel.from_pretrained( base_model, "legesher/language-decoded-lora", subfolder="condition-2-zh-5k-seed42", ) # Or a cond-5 adapter (Synthesized Native Code, Urdu, seed 42) model = PeftModel.from_pretrained( base_model, "legesher/language-decoded-lora", subfolder="condition-5-ur-5k-c4ai-aya-expanse-32b-seed42", ) ``` ## Training Details | Parameter | Value | | ------------------ | ------------------------------------------------------------------------------------------------------ | | Base model | [CohereLabs/tiny-aya-base](https://huggingface.co/CohereLabs/tiny-aya-base) (3.35B params, 70+ languages, low-resource emphasis) | | Method | QLoRA 4-bit (NF4), ~5.4 GB VRAM, Unsloth-accelerated | | Hardware | Kaggle T4 (16 GB) | | Tokenizer | `CohereLabs/tiny-aya-base` | | Transpilation tool | [Legesher](https://github.com/legesher/legesher) v0.7.3 (Phase 3); v0.5.1 / v0.6.0 used in Phase 2 | | Cond-5 translation | [`c4ai-aya-expanse-32b`](https://huggingface.co/CohereLabs/aya-expanse-32b) accessed via the Cohere API (made possible by Cohere credits awarded to Legesher) | | Training data | [legesher/language-decoded-data](https://huggingface.co/datasets/legesher/language-decoded-data) | ### QLoRA hyperparameters | Parameter | Value | | --------------- | ------------------------------------------------------------- | | LoRA rank (`r`) | 16 | | LoRA alpha | 32 | | LoRA dropout | 0.0 | | Target modules | q_proj, k_proj, v_proj, o_proj, up_proj, down_proj, gate_proj | | Bias | none | | Task type | CAUSAL_LM | | PEFT version | 0.18.1 | | Quantization | NF4 (4-bit) via Unsloth | ## Evaluation Phase 3 models are evaluated on four multilingual benchmarks under `template1` (English-prompt) and `template2` (native-prompt) across the full `data_lang × instr_lang` matrix: | Benchmark | What it measures | Examples per language | | --------- | -------------------------- | --------------------- | | XNLI | Natural-language inference | ~5,000 | | X-CSQA | Commonsense reasoning | ~1,000 | | SIB-200 | Topic classification | ~204 | | Belebele | Reading comprehension | ~900 | MGSM was used in Phase 2 and **dropped from Phase 3** — at 3.35B parameters and 250 examples per language, scores ranged 2.8% – 10.8% across all conditions with most condition-to-condition differences within noise. A useful null result; budget was reallocated to SIB-200 and Belebele. Paper-grade evaluation results live on [`legesher/language-decoded-experiments`](https://huggingface.co/datasets/legesher/language-decoded-experiments) — see the refined-tables and the writeup at [`expedition-tiny-aya/analysis/phase-3/phase3-refined-evaluation.md`](https://github.com/legesher/research/blob/main/expedition-tiny-aya/analysis/phase-3/phase3-refined-evaluation.md). ## Limitations - **Single base model**: All adapters are trained on `CohereLabs/tiny-aya-base` (3.35B params). Results may not generalize to larger or architecturally different models. Future iterations will expand to additional base models. - **Per-language fine-tuning only**: Every condition is per-language — each `cond-2-{zh,es,ur}-5k` (and `cond-5-{zh,es,ur}-5k`) is a separate training run. Combined-language training is a planned future condition. - **Limited training data**: 5k and 20k file tiers are constrained by Kaggle T4 hardware limits. 103k variants exist on the [training data repo](https://huggingface.co/datasets/legesher/language-decoded-data) but no 103k adapters have been trained yet. - **Consumer hardware**: Training on Kaggle T4 (16 GB) with 4-bit quantization introduces approximation that may affect adapter quality compared to full-precision training. - **Extractor coverage** — when citing Phase 3 results, use the refined-extractor scores. See the banner at the top of this card and the experiments repo for full details. ## Related Resources - **Experiment tracking and results**: [legesher/language-decoded-experiments](https://huggingface.co/datasets/legesher/language-decoded-experiments) (canonical project source-of-truth) - **Training data**: [legesher/language-decoded-data](https://huggingface.co/datasets/legesher/language-decoded-data) - **Community native code**: [legesher/language-decoded-community](https://huggingface.co/datasets/legesher/language-decoded-community) - **Cond-4 contribution interface**: [`legesher/legesher-native-code`](https://huggingface.co/spaces/legesher/legesher-native-code) HF Space - **Transpilation tool**: [Legesher on GitHub](https://github.com/legesher/legesher) ## Citation ```bibtex @misc{language-decoded-2026, title={Language Decoded: Exploring the Impact of Native Code on Multilingual Models}, author={Madison Edgar and Saad Ahmed Bazaz and Tom Sherborne and Rashik Shahjahan and Khojasteh Mirza and Sarah Jawaid and Rafay Mustafa and Sohaib Ahmed Bazaz}, year={2026}, publisher={Hugging Face}, url={https://huggingface.co/legesher/language-decoded-lora} } ``` ## License Apache 2.0