--- license: apache-2.0 language: - en - zh - es - ur tags: - lora - aya - tiny-aya - multilingual - code - legesher - tiny-aya-expedition - language-decoded - unsloth - arxiv:2408.10914 - arxiv:2603.11510 - arxiv:2211.15533 - arxiv:2510.09591 - arxiv:1809.05053 - arxiv:2308.16884 - arxiv:2106.06937 - arxiv:2210.03057 library_name: transformers base_model: - CohereLabs/tiny-aya-base pipeline_tag: text-generation --- # Language Decoded LoRA QLoRA adapters fine-tuned on multilingual code conditions for the **Language Decoded** project (part of [Cohere's Tiny Aya Expedition](https://aya.for.ai)). > **Submitted paper title (2026-05-26):** _Language, Decoded: Exploring the Impact of Fine-Tuning a Multilingual Model on Native-Language Code_ ## ⚠️ Phase 3 eval numbers — read the experiments repo before citing Original Phase 3 `_summary_*.json` files on [`legesher/language-decoded-experiments`](https://huggingface.co/datasets/legesher/language-decoded-experiments) **under-report cond-5 SIB-200 accuracy by 20–35pp** because the strict inference-time extractor refused native-script answers. Cite the `_summary_reparsed_*.json` siblings (refined extractor) instead. **Five** Phase 3 SIB-200 conclusions also flip win→loss against baseline once the extractor is corrected (`cond-2-es-5k`, `cond-2-es-20k`, `cond-2-ur-20k`, `cond-2-zh-20k`, `cond-3-zh-5k`), and `cond-2-ur-5k`'s gain deflates 4.4×. See the [banner on the experiments repo](https://huggingface.co/datasets/legesher/language-decoded-experiments) (top of the README) for the full picture. ## Research Question > **How does fine-tuning Tiny Aya on non-English code — whether transpiled, mixed-native, or fully translated — affect its multilingual reasoning and instruction-following, and how does that impact _differ_ from fine-tuning on English code?** The hypothesis is **not** that non-English code matches or exceeds English code as a generic reasoning aid — rather, that the _kind_ of effect non-English code produces depends on the target language, the data structure, and how the corpus was constructed. See [legesher/language-decoded-experiments](https://huggingface.co/datasets/legesher/language-decoded-experiments) for the full project context. ## Base Model All adapters are trained on [CohereLabs/tiny-aya-base](https://huggingface.co/CohereLabs/tiny-aya-base) (3.35B parameters). Tiny Aya was chosen because it is small (deployable on a single 16 GB T4 GPU via QLoRA), accessible (Apache 2.0-licensed), and supports 70+ languages with explicit emphasis on lower-resourced ones — which makes the experimental ladder viable for `ur` at all. ## Adapter Inventory This repo holds adapters from **two generations of the project**, kept side by side and clearly separated by folder. See the [Provenance & Manifest](#provenance--manifest) section for a complete path → phase → source-corpus map, and [`MANIFEST.md`](MANIFEST.md) for the machine-readable version. - **Paper adapters (Phase 3 · The Stack v2-dedup)** — live under the **`tiny-aya-base/`** prefix. These are the adapters cited in the submitted paper; cond-1, cond-2, and cond-5 were re-trained from scratch on the cleaner [`bigcode/the-stack-v2-dedup`](https://huggingface.co/datasets/bigcode/the-stack-v2-dedup) corpus. - **Preliminary adapters (Phase 2 · The Stack v1)** — live as **flat top-level folders** (`condition-1-en-32k/`, `condition-2-zh-5k/`, …). These are the original March-2026 hackathon adapters trained on [`bigcode/the-stack`](https://huggingface.co/datasets/bigcode/the-stack) (v1, non-dedup), retained for reproducibility. **Do not cite these for the paper.** ### Paper adapters — Phase 3 · The Stack v2-dedup Each subdirectory under `tiny-aya-base/` is one trained condition × file-volume × seed combination. All adapters share the QLoRA hyperparameters listed under [Training Details](#training-details). | Subdirectory (under `tiny-aya-base/`) | Condition | Training data | Seeds | | ------------------------------------------------------------------------ | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `tiny-aya-base/condition-1-en-5k-seed{42,123,456}/` | 1 | Raw English Python from `bigcode/the-stack-v2-dedup` (5k file subset) | 42, 123, 456 | | `tiny-aya-base/condition-1-en-20k-seed42/` | 1 | Raw English Python (20k file subset) | 42 | | `tiny-aya-base/condition-2-{zh,es,ur}-5k-seed{42,123,456}/` | 2 | The **same 5k subset as cond-1**, processed through Legesher v0.7.3 — Python's reserved words (keywords, exceptions, built-in functions, numerical system for some target languages) translated to the target language; user logic preserved | 42, 123, 456 | | `tiny-aya-base/condition-2-{zh,es,ur}-20k-seed42/` | 2 | The **same 20k subset as cond-1**, processed through Legesher v0.7.3 | 42 | | `tiny-aya-base/condition-3-zh-5k-native-code-seed42/` | 3 | Community-collected raw Chinese code from varied online public-source repositories (different source-file population from cond-1/2/5 by design) | 42 | | `tiny-aya-base/condition-5-{zh,es,ur}-5k-c4ai-aya-expanse-32b-seed42/` | 5 | The **same 5k subset as cond-1**, first transpiled by Legesher v0.7.3 to translate Python's reserved words, then run through `c4ai-aya-expanse-32b` via the Cohere API to translate the remaining content (identifiers, comments, docstrings, string literals) | 42 | **Condition 4 ("Community-Contributed Native Code")** is pending sufficient direct community contributions to the [`legesher/legesher-native-code`](https://huggingface.co/spaces/legesher/legesher-native-code) HF Space; no cond-4 adapter exists yet. ### Preliminary adapters — Phase 2 · The Stack v1 These flat top-level folders are the original hackathon adapters, trained on [`bigcode/the-stack`](https://huggingface.co/datasets/bigcode/the-stack) (v1, non-dedup) with Legesher v0.5.1 / v0.6.0. They are **superseded by the `tiny-aya-base/` Phase 3 adapters above** and are kept only for reproducibility of the preliminary results. The `32k` size and the single-seed setup are Phase 2 signatures. | Subdirectory (top level) | Condition | Source corpus | Notes | | ------------------------ | --------- | ---------------------------------------------- | ---------------------------------- | | `condition-1-en-32k/` | 1 | `bigcode/the-stack` (v1) | Phase 2 32k tier; no Phase 3 equivalent | | `condition-1-en-5k/` | 1 | `bigcode/the-stack` (v1) | Preliminary; use `tiny-aya-base/condition-1-en-5k-seed42/` for the paper | | `condition-2-es-5k/` | 2 | `bigcode/the-stack` (v1), Legesher transpiled | Preliminary | | `condition-2-ur-5k/` | 2 | `bigcode/the-stack` (v1), Legesher transpiled | Preliminary | | `condition-2-zh-5k/` | 2 | `bigcode/the-stack` (v1), Legesher transpiled | Preliminary | | `condition-3-zh-5k/` | 3 | Community-collected raw Chinese code | Preliminary; corpus unchanged across phases | > The standalone per-adapter repos that previously published these Phase 2 / v1 adapters (`legesher/language-decoded-lora-condition-*`) have been renamed to `legesher/language-decoded-lora-phase-2-the-stack-v1-condition-*` and deprecated in favor of this umbrella repo. Their old URLs continue to resolve via Hugging Face redirects. ### Source-file control Cond-1, cond-2, and cond-5 all train on the **same 5,000-file subset** drawn from `bigcode/the-stack-v2-dedup` (with a parallel 20k subset for the 20k tier). Differences across these conditions reflect the processing pipeline (raw / transpiled / fully translated), not file-quality or content drift. Cond-3 is the deliberate exception — its source files are a different population by design. ### The experimental ladder - **Baseline → cond-1**: Does code help at all? (Replicates [Aryabumi et al., 2024](https://arxiv.org/abs/2408.10914).) - **Cond-1 → cond-2**: Does translating Python's reserved words (keywords, exceptions, built-in functions, numerical system for some target languages) into the target language change the model's behavior? User logic and library calls remain English-derived. - **Cond-2 → cond-3**: Does code pulled from real-world public-source repositories — code humans actually wrote in or with the target language — add value beyond Legesher's mechanical translation? - **Cond-2 → cond-5**: Cond-2 translates only Python's reserved words; cond-5 goes further by translating the rest of the file's content (identifiers, comments, docstrings, string literals) via `c4ai-aya-expanse-32b`. Logic and structure are preserved. - **Cond-3 → cond-5** (implicit): Human-authored vs. machine-synthesized native code. For the full ladder including future directions (natural-language text control, combined-language training, similar-script evaluation), see [legesher/language-decoded-experiments](https://huggingface.co/datasets/legesher/language-decoded-experiments). ## Provenance & Manifest The two adapter generations are distinguished by **folder location and source corpus**, matching the convention used across the project's repos (`phase-2-the-stack-v1-*` on [`language-decoded-data`](https://huggingface.co/datasets/legesher/language-decoded-data), `phase2/`÷`phase3/` on [`language-decoded-experiments`](https://huggingface.co/datasets/legesher/language-decoded-experiments)): | Generation | Location in this repo | Source corpus | Legesher | Tier / seeds | Cite for paper? | | --- | --- | --- | --- | --- | --- | | **Phase 3 (paper)** | `tiny-aya-base/…-seed*/` | [`bigcode/the-stack-v2-dedup`](https://huggingface.co/datasets/bigcode/the-stack-v2-dedup) | v0.7.3 | 5k (3 seeds) + 20k (1 seed) | ✅ Yes | | **Phase 2 (preliminary)** | flat top-level `condition-*/` | [`bigcode/the-stack`](https://huggingface.co/datasets/bigcode/the-stack) (v1) | v0.5.1 / v0.6.0 | 5k / 32k (1 seed) | ❌ No | A complete, machine-readable path → phase → corpus → condition map is in [`MANIFEST.md`](MANIFEST.md). Training-data provenance for each condition is detailed on [`language-decoded-data`](https://huggingface.co/datasets/legesher/language-decoded-data); the phase comparison is in the ["Phase 2 → Phase 3 at a glance"](https://huggingface.co/datasets/legesher/language-decoded-experiments#phase-2--phase-3-at-a-glance) table on the experiments repo. ## Usage ```python from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel # Load base model base_model = AutoModelForCausalLM.from_pretrained("CohereLabs/tiny-aya-base") tokenizer = AutoTokenizer.from_pretrained("CohereLabs/tiny-aya-base") # Load a paper (Phase 3 · Stack v2-dedup) adapter — e.g., cond-1 (English code, seed 42, 5k tier). # Paper adapters live under the `tiny-aya-base/` prefix. model = PeftModel.from_pretrained( base_model, "legesher/language-decoded-lora", subfolder="tiny-aya-base/condition-1-en-5k-seed42", ) # Or a language-specific cond-2 adapter (Chinese reserved-word translation, seed 42) model = PeftModel.from_pretrained( base_model, "legesher/language-decoded-lora", subfolder="tiny-aya-base/condition-2-zh-5k-seed42", ) # Or a cond-5 adapter (Synthesized Native Code, Urdu, seed 42) model = PeftModel.from_pretrained( base_model, "legesher/language-decoded-lora", subfolder="tiny-aya-base/condition-5-ur-5k-c4ai-aya-expanse-32b-seed42", ) # To load a *preliminary* Phase 2 / Stack v1 adapter instead, use the flat top-level # folder (no `tiny-aya-base/` prefix) — e.g. the original cond-2 Chinese hackathon adapter: model = PeftModel.from_pretrained( base_model, "legesher/language-decoded-lora", subfolder="condition-2-zh-5k", ) ``` ## Training Details | Parameter | Value | | ------------------ | ------------------------------------------------------------------------------------------------------ | | Base model | [CohereLabs/tiny-aya-base](https://huggingface.co/CohereLabs/tiny-aya-base) (3.35B params, 70+ languages, low-resource emphasis) | | Method | QLoRA 4-bit (NF4), ~5.4 GB VRAM, Unsloth-accelerated | | Hardware | Kaggle T4 (16 GB) | | Tokenizer | `CohereLabs/tiny-aya-base` | | Transpilation tool | [Legesher](https://github.com/legesher/legesher) v0.7.3 (Phase 3); v0.5.1 / v0.6.0 used in Phase 2 | | Cond-5 translation | [`c4ai-aya-expanse-32b`](https://huggingface.co/CohereLabs/aya-expanse-32b) accessed via the Cohere API (made possible by Cohere credits awarded to Legesher) | | Training data | [legesher/language-decoded-data](https://huggingface.co/datasets/legesher/language-decoded-data) | ### QLoRA hyperparameters | Parameter | Value | | --------------- | ------------------------------------------------------------- | | LoRA rank (`r`) | 16 | | LoRA alpha | 32 | | LoRA dropout | 0.0 | | Target modules | q_proj, k_proj, v_proj, o_proj, up_proj, down_proj, gate_proj | | Bias | none | | Task type | CAUSAL_LM | | PEFT version | 0.18.1 | | Quantization | NF4 (4-bit) via Unsloth | ## Evaluation Phase 3 models are evaluated on four multilingual benchmarks under `template1` (English-prompt) and `template2` (native-prompt) across the full `data_lang × instr_lang` matrix: | Benchmark | What it measures | Examples per language | | --------- | -------------------------- | --------------------- | | XNLI | Natural-language inference | ~5,000 | | X-CSQA | Commonsense reasoning | ~1,000 | | SIB-200 | Topic classification | ~204 | | Belebele | Reading comprehension | ~900 | MGSM was used in Phase 2 and **dropped from Phase 3** — at 3.35B parameters and 250 examples per language, scores ranged 2.8% – 10.8% across all conditions with most condition-to-condition differences within noise. A useful null result; budget was reallocated to SIB-200 and Belebele. Paper-grade evaluation results live on [`legesher/language-decoded-experiments`](https://huggingface.co/datasets/legesher/language-decoded-experiments) — see the refined-tables and the writeup at [`expedition-tiny-aya/analysis/phase-3/phase3-refined-evaluation.md`](https://github.com/legesher/research/blob/main/expedition-tiny-aya/analysis/phase-3/phase3-refined-evaluation.md). ## Limitations - **Single base model**: All adapters are trained on `CohereLabs/tiny-aya-base` (3.35B params). Results may not generalize to larger or architecturally different models. Future iterations will expand to additional base models. - **Per-language fine-tuning only**: Every condition is per-language — each `cond-2-{zh,es,ur}-5k` (and `cond-5-{zh,es,ur}-5k`) is a separate training run. Combined-language training is a planned future condition. - **Limited training data**: 5k and 20k file tiers are constrained by Kaggle T4 hardware limits. 103k variants exist on the [training data repo](https://huggingface.co/datasets/legesher/language-decoded-data) but no 103k adapters have been trained yet. - **Consumer hardware**: Training on Kaggle T4 (16 GB) with 4-bit quantization introduces approximation that may affect adapter quality compared to full-precision training. - **Extractor coverage** — when citing Phase 3 results, use the refined-extractor scores. See the banner at the top of this card and the experiments repo for full details. ## Related Resources - **Experiment tracking and results**: [legesher/language-decoded-experiments](https://huggingface.co/datasets/legesher/language-decoded-experiments) (canonical project source-of-truth) - **Training data**: [legesher/language-decoded-data](https://huggingface.co/datasets/legesher/language-decoded-data) - **Community native code**: [legesher/language-decoded-community](https://huggingface.co/datasets/legesher/language-decoded-community) - **Cond-4 contribution interface**: [`legesher/legesher-native-code`](https://huggingface.co/spaces/legesher/legesher-native-code) HF Space - **Transpilation tool**: [Legesher on GitHub](https://github.com/legesher/legesher) ## Citation ```bibtex @misc{language-decoded-2026, title={Language Decoded: Exploring the Impact of Native Code on Multilingual Models}, author={Madison Edgar and Saad Ahmed Bazaz and Tom Sherborne and Rashik Shahjahan and Khojasteh Mirza and Sarah Jawaid and Rafay Mustafa and Sohaib Ahmed Bazaz}, year={2026}, publisher={Hugging Face}, url={https://huggingface.co/legesher/language-decoded-lora} } ``` ## License Apache 2.0