| --- |
| license: apache-2.0 |
| language: |
| - en |
| - zh |
| - es |
| - ur |
| tags: |
| - lora |
| - aya |
| - tiny-aya |
| - multilingual |
| - code |
| - legesher |
| - tiny-aya-expedition |
| - language-decoded |
| - unsloth |
| - arxiv:2408.10914 |
| - arxiv:2603.11510 |
| - arxiv:2211.15533 |
| - arxiv:2510.09591 |
| - arxiv:1809.05053 |
| - arxiv:2308.16884 |
| - arxiv:2106.06937 |
| - arxiv:2210.03057 |
| library_name: transformers |
| base_model: |
| - CohereLabs/tiny-aya-base |
| pipeline_tag: text-generation |
| --- |
| |
| # Language Decoded LoRA |
|
|
| QLoRA adapters fine-tuned on multilingual code conditions for the **Language Decoded** project (part of [Cohere's Tiny Aya Expedition](https://aya.for.ai)). |
|
|
| > **Submitted paper title (2026-05-26):** _Language, Decoded: Exploring the Impact of Fine-Tuning a Multilingual Model on Native-Language Code_ |
|
|
| ## ⚠️ Phase 3 eval numbers — read the experiments repo before citing |
|
|
| Original Phase 3 `_summary_*.json` files on [`legesher/language-decoded-experiments`](https://huggingface.co/datasets/legesher/language-decoded-experiments) **under-report cond-5 SIB-200 accuracy by 20–35pp** because the strict inference-time extractor refused native-script answers. Cite the `_summary_reparsed_*.json` siblings (refined extractor) instead. **Five** Phase 3 SIB-200 conclusions also flip win→loss against baseline once the extractor is corrected (`cond-2-es-5k`, `cond-2-es-20k`, `cond-2-ur-20k`, `cond-2-zh-20k`, `cond-3-zh-5k`), and `cond-2-ur-5k`'s gain deflates 4.4×. See the [banner on the experiments repo](https://huggingface.co/datasets/legesher/language-decoded-experiments) (top of the README) for the full picture. |
|
|
| ## Research Question |
|
|
| > **How does fine-tuning Tiny Aya on non-English code — whether transpiled, mixed-native, or fully translated — affect its multilingual reasoning and instruction-following, and how does that impact _differ_ from fine-tuning on English code?** |
|
|
| The hypothesis is **not** that non-English code matches or exceeds English code as a generic reasoning aid — rather, that the _kind_ of effect non-English code produces depends on the target language, the data structure, and how the corpus was constructed. See [legesher/language-decoded-experiments](https://huggingface.co/datasets/legesher/language-decoded-experiments) for the full project context. |
|
|
| ## Base Model |
|
|
| All adapters are trained on [CohereLabs/tiny-aya-base](https://huggingface.co/CohereLabs/tiny-aya-base) (3.35B parameters). Tiny Aya was chosen because it is small (deployable on a single 16 GB T4 GPU via QLoRA), accessible (Apache 2.0-licensed), and supports 70+ languages with explicit emphasis on lower-resourced ones — which makes the experimental ladder viable for `ur` at all. |
|
|
| ## Adapter Inventory |
|
|
| This repo holds adapters from **two generations of the project**, kept side by side and clearly separated by folder. See the [Provenance & Manifest](#provenance--manifest) section for a complete path → phase → source-corpus map, and [`MANIFEST.md`](MANIFEST.md) for the machine-readable version. |
|
|
| - **Paper adapters (Phase 3 · The Stack v2-dedup)** — live under the **`tiny-aya-base/`** prefix. These are the adapters cited in the submitted paper; cond-1, cond-2, and cond-5 were re-trained from scratch on the cleaner [`bigcode/the-stack-v2-dedup`](https://huggingface.co/datasets/bigcode/the-stack-v2-dedup) corpus. |
| - **Preliminary adapters (Phase 2 · The Stack v1)** — live as **flat top-level folders** (`condition-1-en-32k/`, `condition-2-zh-5k/`, …). These are the original March-2026 hackathon adapters trained on [`bigcode/the-stack`](https://huggingface.co/datasets/bigcode/the-stack) (v1, non-dedup), retained for reproducibility. **Do not cite these for the paper.** |
|
|
| ### Paper adapters — Phase 3 · The Stack v2-dedup |
|
|
| Each subdirectory under `tiny-aya-base/` is one trained condition × file-volume × seed combination. All adapters share the QLoRA hyperparameters listed under [Training Details](#training-details). |
|
|
| | Subdirectory (under `tiny-aya-base/`) | Condition | Training data | Seeds | |
| | ------------------------------------------------------------------------ | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | |
| | `tiny-aya-base/condition-1-en-5k-seed{42,123,456}/` | 1 | Raw English Python from `bigcode/the-stack-v2-dedup` (5k file subset) | 42, 123, 456 | |
| | `tiny-aya-base/condition-1-en-20k-seed42/` | 1 | Raw English Python (20k file subset) | 42 | |
| | `tiny-aya-base/condition-2-{zh,es,ur}-5k-seed{42,123,456}/` | 2 | The **same 5k subset as cond-1**, processed through Legesher v0.7.3 — Python's reserved words (keywords, exceptions, built-in functions, numerical system for some target languages) translated to the target language; user logic preserved | 42, 123, 456 | |
| | `tiny-aya-base/condition-2-{zh,es,ur}-20k-seed42/` | 2 | The **same 20k subset as cond-1**, processed through Legesher v0.7.3 | 42 | |
| | `tiny-aya-base/condition-3-zh-5k-native-code-seed42/` | 3 | Community-collected raw Chinese code from varied online public-source repositories (different source-file population from cond-1/2/5 by design) | 42 | |
| | `tiny-aya-base/condition-5-{zh,es,ur}-5k-c4ai-aya-expanse-32b-seed42/` | 5 | The **same 5k subset as cond-1**, first transpiled by Legesher v0.7.3 to translate Python's reserved words, then run through `c4ai-aya-expanse-32b` via the Cohere API to translate the remaining content (identifiers, comments, docstrings, string literals) | 42 | |
|
|
| **Condition 4 ("Community-Contributed Native Code")** is pending sufficient direct community contributions to the [`legesher/legesher-native-code`](https://huggingface.co/spaces/legesher/legesher-native-code) HF Space; no cond-4 adapter exists yet. |
|
|
| ### Preliminary adapters — Phase 2 · The Stack v1 |
|
|
| These flat top-level folders are the original hackathon adapters, trained on [`bigcode/the-stack`](https://huggingface.co/datasets/bigcode/the-stack) (v1, non-dedup) with Legesher v0.5.1 / v0.6.0. They are **superseded by the `tiny-aya-base/` Phase 3 adapters above** and are kept only for reproducibility of the preliminary results. The `32k` size and the single-seed setup are Phase 2 signatures. |
|
|
| | Subdirectory (top level) | Condition | Source corpus | Notes | |
| | ------------------------ | --------- | ---------------------------------------------- | ---------------------------------- | |
| | `condition-1-en-32k/` | 1 | `bigcode/the-stack` (v1) | Phase 2 32k tier; no Phase 3 equivalent | |
| | `condition-1-en-5k/` | 1 | `bigcode/the-stack` (v1) | Preliminary; use `tiny-aya-base/condition-1-en-5k-seed42/` for the paper | |
| | `condition-2-es-5k/` | 2 | `bigcode/the-stack` (v1), Legesher transpiled | Preliminary | |
| | `condition-2-ur-5k/` | 2 | `bigcode/the-stack` (v1), Legesher transpiled | Preliminary | |
| | `condition-2-zh-5k/` | 2 | `bigcode/the-stack` (v1), Legesher transpiled | Preliminary | |
| | `condition-3-zh-5k/` | 3 | Community-collected raw Chinese code | Preliminary; corpus unchanged across phases | |
|
|
| > The standalone per-adapter repos that previously published these Phase 2 / v1 adapters (`legesher/language-decoded-lora-condition-*`) have been renamed to `legesher/language-decoded-lora-phase-2-the-stack-v1-condition-*` and deprecated in favor of this umbrella repo. Their old URLs continue to resolve via Hugging Face redirects. |
|
|
| ### Source-file control |
|
|
| Cond-1, cond-2, and cond-5 all train on the **same 5,000-file subset** drawn from `bigcode/the-stack-v2-dedup` (with a parallel 20k subset for the 20k tier). Differences across these conditions reflect the processing pipeline (raw / transpiled / fully translated), not file-quality or content drift. Cond-3 is the deliberate exception — its source files are a different population by design. |
|
|
| ### The experimental ladder |
|
|
| - **Baseline → cond-1**: Does code help at all? (Replicates [Aryabumi et al., 2024](https://arxiv.org/abs/2408.10914).) |
| - **Cond-1 → cond-2**: Does translating Python's reserved words (keywords, exceptions, built-in functions, numerical system for some target languages) into the target language change the model's behavior? User logic and library calls remain English-derived. |
| - **Cond-2 → cond-3**: Does code pulled from real-world public-source repositories — code humans actually wrote in or with the target language — add value beyond Legesher's mechanical translation? |
| - **Cond-2 → cond-5**: Cond-2 translates only Python's reserved words; cond-5 goes further by translating the rest of the file's content (identifiers, comments, docstrings, string literals) via `c4ai-aya-expanse-32b`. Logic and structure are preserved. |
| - **Cond-3 → cond-5** (implicit): Human-authored vs. machine-synthesized native code. |
|
|
| For the full ladder including future directions (natural-language text control, combined-language training, similar-script evaluation), see [legesher/language-decoded-experiments](https://huggingface.co/datasets/legesher/language-decoded-experiments). |
|
|
| ## Provenance & Manifest |
|
|
| The two adapter generations are distinguished by **folder location and source corpus**, matching the convention used across the project's repos (`phase-2-the-stack-v1-*` on [`language-decoded-data`](https://huggingface.co/datasets/legesher/language-decoded-data), `phase2/`÷`phase3/` on [`language-decoded-experiments`](https://huggingface.co/datasets/legesher/language-decoded-experiments)): |
|
|
| | Generation | Location in this repo | Source corpus | Legesher | Tier / seeds | Cite for paper? | |
| | --- | --- | --- | --- | --- | --- | |
| | **Phase 3 (paper)** | `tiny-aya-base/…-seed*/` | [`bigcode/the-stack-v2-dedup`](https://huggingface.co/datasets/bigcode/the-stack-v2-dedup) | v0.7.3 | 5k (3 seeds) + 20k (1 seed) | ✅ Yes | |
| | **Phase 2 (preliminary)** | flat top-level `condition-*/` | [`bigcode/the-stack`](https://huggingface.co/datasets/bigcode/the-stack) (v1) | v0.5.1 / v0.6.0 | 5k / 32k (1 seed) | ❌ No | |
|
|
| A complete, machine-readable path → phase → corpus → condition map is in [`MANIFEST.md`](MANIFEST.md). Training-data provenance for each condition is detailed on [`language-decoded-data`](https://huggingface.co/datasets/legesher/language-decoded-data); the phase comparison is in the ["Phase 2 → Phase 3 at a glance"](https://huggingface.co/datasets/legesher/language-decoded-experiments#phase-2--phase-3-at-a-glance) table on the experiments repo. |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| from peft import PeftModel |
| |
| # Load base model |
| base_model = AutoModelForCausalLM.from_pretrained("CohereLabs/tiny-aya-base") |
| tokenizer = AutoTokenizer.from_pretrained("CohereLabs/tiny-aya-base") |
| |
| # Load a paper (Phase 3 · Stack v2-dedup) adapter — e.g., cond-1 (English code, seed 42, 5k tier). |
| # Paper adapters live under the `tiny-aya-base/` prefix. |
| model = PeftModel.from_pretrained( |
| base_model, |
| "legesher/language-decoded-lora", |
| subfolder="tiny-aya-base/condition-1-en-5k-seed42", |
| ) |
| |
| # Or a language-specific cond-2 adapter (Chinese reserved-word translation, seed 42) |
| model = PeftModel.from_pretrained( |
| base_model, |
| "legesher/language-decoded-lora", |
| subfolder="tiny-aya-base/condition-2-zh-5k-seed42", |
| ) |
| |
| # Or a cond-5 adapter (Synthesized Native Code, Urdu, seed 42) |
| model = PeftModel.from_pretrained( |
| base_model, |
| "legesher/language-decoded-lora", |
| subfolder="tiny-aya-base/condition-5-ur-5k-c4ai-aya-expanse-32b-seed42", |
| ) |
| |
| # To load a *preliminary* Phase 2 / Stack v1 adapter instead, use the flat top-level |
| # folder (no `tiny-aya-base/` prefix) — e.g. the original cond-2 Chinese hackathon adapter: |
| model = PeftModel.from_pretrained( |
| base_model, |
| "legesher/language-decoded-lora", |
| subfolder="condition-2-zh-5k", |
| ) |
| ``` |
|
|
| ## Training Details |
|
|
| | Parameter | Value | |
| | ------------------ | ------------------------------------------------------------------------------------------------------ | |
| | Base model | [CohereLabs/tiny-aya-base](https://huggingface.co/CohereLabs/tiny-aya-base) (3.35B params, 70+ languages, low-resource emphasis) | |
| | Method | QLoRA 4-bit (NF4), ~5.4 GB VRAM, Unsloth-accelerated | |
| | Hardware | Kaggle T4 (16 GB) | |
| | Tokenizer | `CohereLabs/tiny-aya-base` | |
| | Transpilation tool | [Legesher](https://github.com/legesher/legesher) v0.7.3 (Phase 3); v0.5.1 / v0.6.0 used in Phase 2 | |
| | Cond-5 translation | [`c4ai-aya-expanse-32b`](https://huggingface.co/CohereLabs/aya-expanse-32b) accessed via the Cohere API (made possible by Cohere credits awarded to Legesher) | |
| | Training data | [legesher/language-decoded-data](https://huggingface.co/datasets/legesher/language-decoded-data) | |
|
|
| ### QLoRA hyperparameters |
|
|
| | Parameter | Value | |
| | --------------- | ------------------------------------------------------------- | |
| | LoRA rank (`r`) | 16 | |
| | LoRA alpha | 32 | |
| | LoRA dropout | 0.0 | |
| | Target modules | q_proj, k_proj, v_proj, o_proj, up_proj, down_proj, gate_proj | |
| | Bias | none | |
| | Task type | CAUSAL_LM | |
| | PEFT version | 0.18.1 | |
| | Quantization | NF4 (4-bit) via Unsloth | |
|
|
| ## Evaluation |
|
|
| Phase 3 models are evaluated on four multilingual benchmarks under `template1` (English-prompt) and `template2` (native-prompt) across the full `data_lang × instr_lang` matrix: |
|
|
| | Benchmark | What it measures | Examples per language | |
| | --------- | -------------------------- | --------------------- | |
| | XNLI | Natural-language inference | ~5,000 | |
| | X-CSQA | Commonsense reasoning | ~1,000 | |
| | SIB-200 | Topic classification | ~204 | |
| | Belebele | Reading comprehension | ~900 | |
|
|
| MGSM was used in Phase 2 and **dropped from Phase 3** — at 3.35B parameters and 250 examples per language, scores ranged 2.8% – 10.8% across all conditions with most condition-to-condition differences within noise. A useful null result; budget was reallocated to SIB-200 and Belebele. |
|
|
| Paper-grade evaluation results live on [`legesher/language-decoded-experiments`](https://huggingface.co/datasets/legesher/language-decoded-experiments) — see the refined-tables and the writeup at [`expedition-tiny-aya/analysis/phase-3/phase3-refined-evaluation.md`](https://github.com/legesher/research/blob/main/expedition-tiny-aya/analysis/phase-3/phase3-refined-evaluation.md). |
|
|
| ## Limitations |
|
|
| - **Single base model**: All adapters are trained on `CohereLabs/tiny-aya-base` (3.35B params). Results may not generalize to larger or architecturally different models. Future iterations will expand to additional base models. |
| - **Per-language fine-tuning only**: Every condition is per-language — each `cond-2-{zh,es,ur}-5k` (and `cond-5-{zh,es,ur}-5k`) is a separate training run. Combined-language training is a planned future condition. |
| - **Limited training data**: 5k and 20k file tiers are constrained by Kaggle T4 hardware limits. 103k variants exist on the [training data repo](https://huggingface.co/datasets/legesher/language-decoded-data) but no 103k adapters have been trained yet. |
| - **Consumer hardware**: Training on Kaggle T4 (16 GB) with 4-bit quantization introduces approximation that may affect adapter quality compared to full-precision training. |
| - **Extractor coverage** — when citing Phase 3 results, use the refined-extractor scores. See the banner at the top of this card and the experiments repo for full details. |
|
|
| ## Related Resources |
|
|
| - **Experiment tracking and results**: [legesher/language-decoded-experiments](https://huggingface.co/datasets/legesher/language-decoded-experiments) (canonical project source-of-truth) |
| - **Training data**: [legesher/language-decoded-data](https://huggingface.co/datasets/legesher/language-decoded-data) |
| - **Community native code**: [legesher/language-decoded-community](https://huggingface.co/datasets/legesher/language-decoded-community) |
| - **Cond-4 contribution interface**: [`legesher/legesher-native-code`](https://huggingface.co/spaces/legesher/legesher-native-code) HF Space |
| - **Transpilation tool**: [Legesher on GitHub](https://github.com/legesher/legesher) |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{language-decoded-2026, |
| title={Language Decoded: Exploring the Impact of Native Code on Multilingual Models}, |
| author={Madison Edgar and Saad Ahmed Bazaz and Tom Sherborne and Rashik Shahjahan and Khojasteh Mirza and Sarah Jawaid and Rafay Mustafa and Sohaib Ahmed Bazaz}, |
| year={2026}, |
| publisher={Hugging Face}, |
| url={https://huggingface.co/legesher/language-decoded-lora} |
| } |
| ``` |
|
|
| ## License |
|
|
| Apache 2.0 |
|
|
