README.md

metadata

license: apache-2.0
language:
  - en
  - zh
  - es
  - ur
tags:
  - lora
  - aya
  - tiny-aya
  - multilingual
  - code
  - legesher
  - tiny-aya-expedition
  - language-decoded
  - unsloth
library_name: transformers
base_model:
  - CohereLabs/tiny-aya-base
pipeline_tag: text-generation

Language Decoded LoRA

QLoRA adapters fine-tuned on multilingual code conditions for the Language Decoded project (part of Cohere's Tiny Aya Expedition).

Submitted paper title (2026-05-26): Language, Decoded: Exploring the Impact of Fine-Tuning a Multilingual Model on Native-Language Code

⚠️ Phase 3 eval numbers — read the experiments repo before citing

Original Phase 3 _summary_*.json files on legesher/language-decoded-experiments under-report cond-5 SIB-200 accuracy by 20–35pp because the strict inference-time extractor refused native-script answers. Cite the _summary_reparsed_*.json siblings (refined extractor) instead. Five Phase 3 SIB-200 conclusions also flip win→loss against baseline once the extractor is corrected (cond-2-es-5k, cond-2-es-20k, cond-2-ur-20k, cond-2-zh-20k, cond-3-zh-5k), and cond-2-ur-5k's gain deflates 4.4×. See the banner on the experiments repo (top of the README) for the full picture.

Research Question

How does fine-tuning Tiny Aya on non-English code — whether transpiled, mixed-native, or fully translated — affect its multilingual reasoning and instruction-following, and how does that impact differ from fine-tuning on English code?

The hypothesis is not that non-English code matches or exceeds English code as a generic reasoning aid — rather, that the kind of effect non-English code produces depends on the target language, the data structure, and how the corpus was constructed. See legesher/language-decoded-experiments for the full project context.

Base Model

All adapters are trained on CohereLabs/tiny-aya-base (3.35B parameters). Tiny Aya was chosen because it is small (deployable on a single 16 GB T4 GPU via QLoRA), accessible (Apache 2.0-licensed), and supports 70+ languages with explicit emphasis on lower-resourced ones — which makes the experimental ladder viable for ur at all.

Adapter Inventory

Each subdirectory is one trained condition × file-volume × seed combination. All adapters share the QLoRA hyperparameters listed under Training Details.

Subdirectory	Condition	Training data	Seeds
condition-1-en-5k-seed{42,123,456}/	1	Raw English Python from bigcode/the-stack-v2-dedup (5k file subset)	42, 123, 456
condition-1-en-20k-seed42/	1	Raw English Python (20k file subset)	42
condition-2-{zh,es,ur}-5k-seed{42,123,456}/	2	The same 5k subset as cond-1, processed through Legesher v0.7.3 — Python's reserved words (keywords, exceptions, built-in functions, numerical system for some target languages) translated to the target language; user logic preserved	42, 123, 456
condition-2-{zh,es,ur}-20k-seed42/	2	The same 20k subset as cond-1, processed through Legesher v0.7.3	42
condition-3-zh-5k-native-code-seed42/	3	Community-collected raw Chinese code from varied online public-source repositories (different source-file population from cond-1/2/5 by design)	42
condition-5-{zh,es,ur}-5k-c4ai-aya-expanse-32b-seed42/	5	The same 5k subset as cond-1, first transpiled by Legesher v0.7.3 to translate Python's reserved words, then run through c4ai-aya-expanse-32b via the Cohere API to translate the remaining content (identifiers, comments, docstrings, string literals)	42

Condition 4 ("Community-Contributed Native Code") is pending sufficient direct community contributions to the legesher/legesher-native-code HF Space; no cond-4 adapter exists yet.

Source-file control

Cond-1, cond-2, and cond-5 all train on the same 5,000-file subset drawn from bigcode/the-stack-v2-dedup (with a parallel 20k subset for the 20k tier). Differences across these conditions reflect the processing pipeline (raw / transpiled / fully translated), not file-quality or content drift. Cond-3 is the deliberate exception — its source files are a different population by design.

The experimental ladder

• Baseline → cond-1: Does code help at all? (Replicates Aryabumi et al., 2024.)
• Cond-1 → cond-2: Does translating Python's reserved words (keywords, exceptions, built-in functions, numerical system for some target languages) into the target language change the model's behavior? User logic and library calls remain English-derived.
• Cond-2 → cond-3: Does code pulled from real-world public-source repositories — code humans actually wrote in or with the target language — add value beyond Legesher's mechanical translation?
• Cond-2 → cond-5: Cond-2 translates only Python's reserved words; cond-5 goes further by translating the rest of the file's content (identifiers, comments, docstrings, string literals) via c4ai-aya-expanse-32b. Logic and structure are preserved.
• Cond-3 → cond-5 (implicit): Human-authored vs. machine-synthesized native code.

For the full ladder including future directions (natural-language text control, combined-language training, similar-script evaluation), see legesher/language-decoded-experiments.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained("CohereLabs/tiny-aya-base")
tokenizer = AutoTokenizer.from_pretrained("CohereLabs/tiny-aya-base")

# Load a LoRA adapter — e.g., cond-1 (English code, seed 42, 5k tier)
model = PeftModel.from_pretrained(
    base_model,
    "legesher/language-decoded-lora",
    subfolder="condition-1-en-5k-seed42",
)

# Or a language-specific cond-2 adapter (Chinese reserved-word translation, seed 42)
model = PeftModel.from_pretrained(
    base_model,
    "legesher/language-decoded-lora",
    subfolder="condition-2-zh-5k-seed42",
)

# Or a cond-5 adapter (Synthesized Native Code, Urdu, seed 42)
model = PeftModel.from_pretrained(
    base_model,
    "legesher/language-decoded-lora",
    subfolder="condition-5-ur-5k-c4ai-aya-expanse-32b-seed42",
)

Training Details

Parameter	Value
Base model	CohereLabs/tiny-aya-base (3.35B params, 70+ languages, low-resource emphasis)
Method	QLoRA 4-bit (NF4), ~5.4 GB VRAM, Unsloth-accelerated
Hardware	Kaggle T4 (16 GB)
Tokenizer	CohereLabs/tiny-aya-base
Transpilation tool	Legesher v0.7.3 (Phase 3); v0.5.1 / v0.6.0 used in Phase 2
Cond-5 translation	c4ai-aya-expanse-32b accessed via the Cohere API (made possible by Cohere credits awarded to Legesher)
Training data	legesher/language-decoded-data

QLoRA hyperparameters

Parameter	Value
LoRA rank (r)	16
LoRA alpha	32
LoRA dropout	0.0
Target modules	q_proj, k_proj, v_proj, o_proj, up_proj, down_proj, gate_proj
Bias	none
Task type	CAUSAL_LM
PEFT version	0.18.1
Quantization	NF4 (4-bit) via Unsloth

Evaluation

Phase 3 models are evaluated on four multilingual benchmarks under template1 (English-prompt) and template2 (native-prompt) across the full data_lang × instr_lang matrix:

Benchmark	What it measures	Examples per language
XNLI	Natural-language inference	~5,000
X-CSQA	Commonsense reasoning	~1,000
SIB-200	Topic classification	~204
Belebele	Reading comprehension	~900

MGSM was used in Phase 2 and dropped from Phase 3 — at 3.35B parameters and 250 examples per language, scores ranged 2.8% – 10.8% across all conditions with most condition-to-condition differences within noise. A useful null result; budget was reallocated to SIB-200 and Belebele.

Paper-grade evaluation results live on legesher/language-decoded-experiments — see the refined-tables and the writeup at expedition-tiny-aya/analysis/phase-3/phase3-refined-evaluation.md.

Limitations

• Single base model: All adapters are trained on CohereLabs/tiny-aya-base (3.35B params). Results may not generalize to larger or architecturally different models. Future iterations will expand to additional base models.
• Per-language fine-tuning only: Every condition is per-language — each cond-2-{zh,es,ur}-5k (and cond-5-{zh,es,ur}-5k) is a separate training run. Combined-language training is a planned future condition.
• Limited training data: 5k and 20k file tiers are constrained by Kaggle T4 hardware limits. 103k variants exist on the training data repo but no 103k adapters have been trained yet.
• Consumer hardware: Training on Kaggle T4 (16 GB) with 4-bit quantization introduces approximation that may affect adapter quality compared to full-precision training.
• Extractor coverage — when citing Phase 3 results, use the refined-extractor scores. See the banner at the top of this card and the experiments repo for full details.

Related Resources

• Experiment tracking and results: legesher/language-decoded-experiments (canonical project source-of-truth)
• Training data: legesher/language-decoded-data
• Community native code: legesher/language-decoded-community
• Cond-4 contribution interface: legesher/legesher-native-code HF Space
• Transpilation tool: Legesher on GitHub

Citation

@misc{language-decoded-2026,
  title={Language Decoded: Exploring the Impact of Native Code on Multilingual Models},
  author={Madison Edgar and Saad Ahmed Bazaz and Tom Sherborne and Rashik Shahjahan and Khojasteh Mirza and Sarah Jawaid and Rafay Mustafa and Sohaib Ahmed Bazaz},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/legesher/language-decoded-lora}
}

License

Apache 2.0