README.md

---
license: cc-by-nc-4.0
language:
- en
- zh
- es
- ur
tags:
- lora
- aya
- tiny-aya
- multilingual
- code
- legesher
- tiny-aya-expedition
- language-decoded
- unsloth
- arxiv:2408.10914
- arxiv:2603.11510
- arxiv:2211.15533
- arxiv:2510.09591
- arxiv:1809.05053
- arxiv:2308.16884
- arxiv:2106.06937
- arxiv:2210.03057
library_name: transformers
base_model:
- CohereLabs/tiny-aya-base
pipeline_tag: text-generation
---

# Language Decoded LoRA

QLoRA adapters fine-tuned on multilingual code conditions for the **Language Decoded** project (part of [Cohere's Tiny Aya Expedition](https://aya.for.ai)).

> **Submitted paper title (2026-05-26):** _Language, Decoded: Exploring the Impact of Fine-Tuning a Multilingual Model on Native-Language Code_

## ⚠️ Phase 3 eval numbers — read the experiments repo before citing

Original Phase 3 `_summary_*.json` files on [`legesher/language-decoded-experiments`](https://huggingface.co/datasets/legesher/language-decoded-experiments) **under-report cond-5 SIB-200 accuracy by 20–35pp** because the strict inference-time extractor refused native-script answers. Cite the `_summary_reparsed_*.json` siblings (refined extractor) instead. **Five** Phase 3 SIB-200 conclusions also flip win→loss against baseline once the extractor is corrected (`cond-2-es-5k`, `cond-2-es-20k`, `cond-2-ur-20k`, `cond-2-zh-20k`, `cond-3-zh-5k`), and `cond-2-ur-5k`'s gain deflates 4.4×. See the [banner on the experiments repo](https://huggingface.co/datasets/legesher/language-decoded-experiments) (top of the README) for the full picture.

## Research Question

> **How does fine-tuning Tiny Aya on non-English code — whether transpiled, mixed-native, or fully translated — affect its multilingual reasoning and instruction-following, and how does that impact _differ_ from fine-tuning on English code?**

The hypothesis is **not** that non-English code matches or exceeds English code as a generic reasoning aid — rather, that the _kind_ of effect non-English code produces depends on the target language, the data structure, and how the corpus was constructed. See [legesher/language-decoded-experiments](https://huggingface.co/datasets/legesher/language-decoded-experiments) for the full project context.

## Base Model

All adapters are trained on [CohereLabs/tiny-aya-base](https://huggingface.co/CohereLabs/tiny-aya-base) (3.35B parameters). Tiny Aya was chosen because it is small (deployable on a single 16 GB T4 GPU via QLoRA), openly available (released under CC-BY-NC-4.0), and supports 70+ languages with explicit emphasis on lower-resourced ones — which makes the experimental ladder viable for `ur` at all.

## Adapter Inventory

This repo holds adapters from **two generations of the project**, kept side by side and clearly separated by folder. See the [Provenance & Manifest](#provenance--manifest) section for a complete path → phase → source-corpus map, and [`MANIFEST.md`](MANIFEST.md) for the machine-readable version.

- **Paper adapters (Phase 3 · The Stack v2-dedup)** — live under the **`tiny-aya-base/`** prefix. These are the adapters cited in the submitted paper; cond-1, cond-2, and cond-5 were re-trained from scratch on the cleaner [`bigcode/the-stack-v2-dedup`](https://huggingface.co/datasets/bigcode/the-stack-v2-dedup) corpus.
- **Preliminary adapters (Phase 2 · The Stack v1)** — live as **flat top-level folders** (`condition-1-en-32k/`, `condition-2-zh-5k/`, …). These are the original March-2026 hackathon adapters trained on [`bigcode/the-stack`](https://huggingface.co/datasets/bigcode/the-stack) (v1, non-dedup), retained for reproducibility. **Do not cite these for the paper.**

### Paper adapters — Phase 3 · The Stack v2-dedup

Each subdirectory under `tiny-aya-base/` is one trained condition × file-volume × seed combination. All adapters share the QLoRA hyperparameters listed under [Training Details](#training-details).

| Subdirectory (under `tiny-aya-base/`)                                    | Condition | Training data                                                                                                                                  | Seeds      |
| ------------------------------------------------------------------------ | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `tiny-aya-base/condition-1-en-5k-seed{42,123,456}/`                      | 1         | Raw English Python from `bigcode/the-stack-v2-dedup` (5k file subset)                                                                          | 42, 123, 456 |
| `tiny-aya-base/condition-1-en-20k-seed42/`                               | 1         | Raw English Python (20k file subset)                                                                                                           | 42         |
| `tiny-aya-base/condition-2-{zh,es,ur}-5k-seed{42,123,456}/`             | 2         | The **same 5k subset as cond-1**, processed through Legesher v0.7.3 — Python's reserved words (keywords, exceptions, built-in functions, numerical system for some target languages) translated to the target language; user logic preserved | 42, 123, 456 |
| `tiny-aya-base/condition-2-{zh,es,ur}-20k-seed42/`                      | 2         | The **same 20k subset as cond-1**, processed through Legesher v0.7.3                                                                           | 42         |
| `tiny-aya-base/condition-3-zh-5k-native-code-seed42/`                   | 3         | Community-collected raw Chinese code from varied online public-source repositories (different source-file population from cond-1/2/5 by design) | 42         |
| `tiny-aya-base/condition-5-{zh,es,ur}-5k-c4ai-aya-expanse-32b-seed42/`  | 5         | The **same 5k subset as cond-1**, first transpiled by Legesher v0.7.3 to translate Python's reserved words, then run through `c4ai-aya-expanse-32b` via the Cohere API to translate the remaining content (identifiers, comments, docstrings, string literals) | 42         |

**Condition 4 ("Community-Contributed Native Code")** is pending sufficient direct community contributions to the [`legesher/legesher-native-code`](https://huggingface.co/spaces/legesher/legesher-native-code) HF Space; no cond-4 adapter exists yet.

### Preliminary adapters — Phase 2 · The Stack v1

These flat top-level folders are the original hackathon adapters, trained on [`bigcode/the-stack`](https://huggingface.co/datasets/bigcode/the-stack) (v1, non-dedup) with Legesher v0.5.1 / v0.6.0. They are **superseded by the `tiny-aya-base/` Phase 3 adapters above** and are kept only for reproducibility of the preliminary results. The `32k` size and the single-seed setup are Phase 2 signatures.

| Subdirectory (top level) | Condition | Source corpus                                  | Notes                              |
| ------------------------ | --------- | ---------------------------------------------- | ---------------------------------- |
| `condition-1-en-32k/`    | 1         | `bigcode/the-stack` (v1)                        | Phase 2 32k tier; no Phase 3 equivalent |
| `condition-1-en-5k/`     | 1         | `bigcode/the-stack` (v1)                        | Preliminary; use `tiny-aya-base/condition-1-en-5k-seed42/` for the paper |
| `condition-2-es-5k/`     | 2         | `bigcode/the-stack` (v1), Legesher transpiled   | Preliminary                        |
| `condition-2-ur-5k/`     | 2         | `bigcode/the-stack` (v1), Legesher transpiled   | Preliminary                        |
| `condition-2-zh-5k/`     | 2         | `bigcode/the-stack` (v1), Legesher transpiled   | Preliminary                        |
| `condition-3-zh-5k/`     | 3         | Community-collected raw Chinese code            | Preliminary; corpus unchanged across phases |

> The standalone per-adapter repos that previously published these Phase 2 / v1 adapters (`legesher/language-decoded-lora-condition-*`) have been renamed to `legesher/language-decoded-lora-phase-2-the-stack-v1-condition-*` and deprecated in favor of this umbrella repo. Their old URLs continue to resolve via Hugging Face redirects.

### Source-file control

Cond-1, cond-2, and cond-5 all train on the **same 5,000-file subset** drawn from `bigcode/the-stack-v2-dedup` (with a parallel 20k subset for the 20k tier). Differences across these conditions reflect the processing pipeline (raw / transpiled / fully translated), not file-quality or content drift. Cond-3 is the deliberate exception — its source files are a different population by design.

### The experimental ladder

- **Baseline → cond-1**: Does code help at all? (Replicates [Aryabumi et al., 2024](https://arxiv.org/abs/2408.10914).)
- **Cond-1 → cond-2**: Does translating Python's reserved words (keywords, exceptions, built-in functions, numerical system for some target languages) into the target language change the model's behavior? User logic and library calls remain English-derived.
- **Cond-2 → cond-3**: Does code pulled from real-world public-source repositories — code humans actually wrote in or with the target language — add value beyond Legesher's mechanical translation?
- **Cond-2 → cond-5**: Cond-2 translates only Python's reserved words; cond-5 goes further by translating the rest of the file's content (identifiers, comments, docstrings, string literals) via `c4ai-aya-expanse-32b`. Logic and structure are preserved.
- **Cond-3 → cond-5** (implicit): Human-authored vs. machine-synthesized native code.

For the full ladder including future directions (natural-language text control, combined-language training, similar-script evaluation), see [legesher/language-decoded-experiments](https://huggingface.co/datasets/legesher/language-decoded-experiments).

## Provenance & Manifest

The two adapter generations are distinguished by **folder location and source corpus**, matching the convention used across the project's repos (`phase-2-the-stack-v1-*` on [`language-decoded-data`](https://huggingface.co/datasets/legesher/language-decoded-data), `phase2/`÷`phase3/` on [`language-decoded-experiments`](https://huggingface.co/datasets/legesher/language-decoded-experiments)):

| Generation | Location in this repo | Source corpus | Legesher | Tier / seeds | Cite for paper? |
| --- | --- | --- | --- | --- | --- |
| **Phase 3 (paper)** | `tiny-aya-base/…-seed*/` | [`bigcode/the-stack-v2-dedup`](https://huggingface.co/datasets/bigcode/the-stack-v2-dedup) | v0.7.3 | 5k (3 seeds) + 20k (1 seed) | ✅ Yes |
| **Phase 2 (preliminary)** | flat top-level `condition-*/` | [`bigcode/the-stack`](https://huggingface.co/datasets/bigcode/the-stack) (v1) | v0.5.1 / v0.6.0 | 5k / 32k (1 seed) | ❌ No |

A complete, machine-readable path → phase → corpus → condition map is in [`MANIFEST.md`](MANIFEST.md). Training-data provenance for each condition is detailed on [`language-decoded-data`](https://huggingface.co/datasets/legesher/language-decoded-data); the phase comparison is in the ["Phase 2 → Phase 3 at a glance"](https://huggingface.co/datasets/legesher/language-decoded-experiments#phase-2--phase-3-at-a-glance) table on the experiments repo.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained("CohereLabs/tiny-aya-base")
tokenizer = AutoTokenizer.from_pretrained("CohereLabs/tiny-aya-base")

# Load a paper (Phase 3 · Stack v2-dedup) adapter — e.g., cond-1 (English code, seed 42, 5k tier).
# Paper adapters live under the `tiny-aya-base/` prefix.
model = PeftModel.from_pretrained(
    base_model,
    "legesher/language-decoded-lora",
    subfolder="tiny-aya-base/condition-1-en-5k-seed42",
)

# Or a language-specific cond-2 adapter (Chinese reserved-word translation, seed 42)
model = PeftModel.from_pretrained(
    base_model,
    "legesher/language-decoded-lora",
    subfolder="tiny-aya-base/condition-2-zh-5k-seed42",
)

# Or a cond-5 adapter (Synthesized Native Code, Urdu, seed 42)
model = PeftModel.from_pretrained(
    base_model,
    "legesher/language-decoded-lora",
    subfolder="tiny-aya-base/condition-5-ur-5k-c4ai-aya-expanse-32b-seed42",
)

# To load a *preliminary* Phase 2 / Stack v1 adapter instead, use the flat top-level
# folder (no `tiny-aya-base/` prefix) — e.g. the original cond-2 Chinese hackathon adapter:
model = PeftModel.from_pretrained(
    base_model,
    "legesher/language-decoded-lora",
    subfolder="condition-2-zh-5k",
)
```

## Training Details

| Parameter          | Value                                                                                                  |
| ------------------ | ------------------------------------------------------------------------------------------------------ |
| Base model         | [CohereLabs/tiny-aya-base](https://huggingface.co/CohereLabs/tiny-aya-base) (3.35B params, 70+ languages, low-resource emphasis) |
| Method             | QLoRA 4-bit (NF4), ~5.4 GB VRAM, Unsloth-accelerated                                                   |
| Hardware           | Kaggle T4 (16 GB)                                                                                      |
| Tokenizer          | `CohereLabs/tiny-aya-base`                                                                             |
| Transpilation tool | [Legesher](https://github.com/legesher/legesher) v0.7.3 (Phase 3); v0.5.1 / v0.6.0 used in Phase 2     |
| Cond-5 translation | [`c4ai-aya-expanse-32b`](https://huggingface.co/CohereLabs/aya-expanse-32b) accessed via the Cohere API (made possible by Cohere credits awarded to Legesher) |
| Training data      | [legesher/language-decoded-data](https://huggingface.co/datasets/legesher/language-decoded-data)       |

### QLoRA hyperparameters

| Parameter       | Value                                                         |
| --------------- | ------------------------------------------------------------- |
| LoRA rank (`r`) | 16                                                            |
| LoRA alpha      | 32                                                            |
| LoRA dropout    | 0.0                                                           |
| Target modules  | q_proj, k_proj, v_proj, o_proj, up_proj, down_proj, gate_proj |
| Bias            | none                                                          |
| Task type       | CAUSAL_LM                                                     |
| PEFT version    | 0.18.1                                                        |
| Quantization    | NF4 (4-bit) via Unsloth                                       |

## Evaluation

Phase 3 models are evaluated on four multilingual benchmarks under `template1` (English-prompt) and `template2` (native-prompt) across the full `data_lang × instr_lang` matrix:

| Benchmark | What it measures           | Examples per language |
| --------- | -------------------------- | --------------------- |
| XNLI      | Natural-language inference | ~5,000                |
| X-CSQA    | Commonsense reasoning      | ~1,000                |
| SIB-200   | Topic classification       | ~204                  |
| Belebele  | Reading comprehension      | ~900                  |

MGSM was used in Phase 2 and **dropped from Phase 3** — at 3.35B parameters and 250 examples per language, scores ranged 2.8% – 10.8% across all conditions with most condition-to-condition differences within noise. A useful null result; budget was reallocated to SIB-200 and Belebele.

Paper-grade evaluation results live on [`legesher/language-decoded-experiments`](https://huggingface.co/datasets/legesher/language-decoded-experiments) — see the refined-tables and the writeup at [`expedition-tiny-aya/analysis/phase-3/phase3-refined-evaluation.md`](https://github.com/legesher/research/blob/main/expedition-tiny-aya/analysis/phase-3/phase3-refined-evaluation.md).

## Limitations

- **Single base model**: All adapters are trained on `CohereLabs/tiny-aya-base` (3.35B params). Results may not generalize to larger or architecturally different models. Future iterations will expand to additional base models.
- **Per-language fine-tuning only**: Every condition is per-language — each `cond-2-{zh,es,ur}-5k` (and `cond-5-{zh,es,ur}-5k`) is a separate training run. Combined-language training is a planned future condition.
- **Limited training data**: 5k and 20k file tiers are constrained by Kaggle T4 hardware limits. 103k variants exist on the [training data repo](https://huggingface.co/datasets/legesher/language-decoded-data) but no 103k adapters have been trained yet.
- **Consumer hardware**: Training on Kaggle T4 (16 GB) with 4-bit quantization introduces approximation that may affect adapter quality compared to full-precision training.
- **Extractor coverage** — when citing Phase 3 results, use the refined-extractor scores. See the banner at the top of this card and the experiments repo for full details.

## Related Resources

- **Experiment tracking and results**: [legesher/language-decoded-experiments](https://huggingface.co/datasets/legesher/language-decoded-experiments) (canonical project source-of-truth)
- **Training data**: [legesher/language-decoded-data](https://huggingface.co/datasets/legesher/language-decoded-data)
- **Community native code**: [legesher/language-decoded-community](https://huggingface.co/datasets/legesher/language-decoded-community)
- **Cond-4 contribution interface**: [`legesher/legesher-native-code`](https://huggingface.co/spaces/legesher/legesher-native-code) HF Space
- **Transpilation tool**: [Legesher on GitHub](https://github.com/legesher/legesher)

## Citation

```bibtex
@misc{language-decoded-2026,
  title={Language Decoded: Exploring the Impact of Native Code on Multilingual Models},
  author={Madison Edgar and Saad Ahmed Bazaz and Tom Sherborne and Rashik Shahjahan and Khojasteh Mirza and Sarah Jawaid and Rafay Mustafa and Sohaib Ahmed Bazaz},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/legesher/language-decoded-lora}
}
```

## License

CC-BY-NC-4.0. The adapters inherit the license of the base model,
[CohereLabs/tiny-aya-base](https://huggingface.co/CohereLabs/tiny-aya-base)
(CC-BY-NC-4.0). The training datasets
([legesher/language-decoded-data](https://huggingface.co/datasets/legesher/language-decoded-data))
are separately licensed under Apache-2.0.

## Provenance, attribution & takedown

These adapters were fine-tuned from
[`CohereLabs/tiny-aya-base`](https://huggingface.co/CohereLabs/tiny-aya-base)
on a specific revision of the
[`legesher/language-decoded-data`](https://huggingface.co/datasets/legesher/language-decoded-data)
training conditions (see each adapter's configuration for the
condition and revision).

If you are the author of source code included in the training data
and would like attribution added or your code removed, open a
discussion on the dataset repository's **Community** tab or email
**support@legesher.com**. Removals are propagated in a new dataset
revision. Adapters already trained are frozen historical artifacts:
a dataset removal does not alter existing adapter weights, but we
will note affected conditions here and take reported concerns about
specific adapters into account.

**Usage caution.** These are research artifacts, not
production-ready models. Documented side effects include code
fragments leaking into natural-language output (strongest for Urdu
adapters) and matched-language regressions on specific evaluation
cells; the Condition 5 adapters were trained on corpora containing
raw translator output. Evaluate per language and per task before any
downstream use.