| | --- |
| | license: cc-by-nc-4.0 |
| | language: |
| | - multilingual |
| | tags: |
| | - lora |
| | - aya |
| | - tiny-aya |
| | - multilingual |
| | - code |
| | - legesher |
| | - tiny-aya-expedition |
| | - language-decoded |
| | library_name: transformers |
| | base_model: |
| | - CohereLabs/tiny-aya-global |
| | - CohereLabs/tiny-aya-fire |
| | - CohereLabs/tiny-aya-earth |
| | - CohereLabs/tiny-aya-water |
| | pipeline_tag: text-generation |
| | --- |
| | |
| | # Language Decoded LoRA |
| |
|
| | LoRA adapters fine-tuned on multilingual code conditions for the **Language Decoded** project (part of Cohere's Tiny Aya Expedition). |
| |
|
| | ## Research Question |
| |
|
| | > Does fine-tuning on non-English code improve multilingual reasoning — and is the benefit language-dependent or structure-dependent? |
| |
|
| | ## Base Models |
| |
|
| | All adapters are trained on [Tiny Aya](https://huggingface.co/collections/CohereLabs/tiny-aya) (3.35B parameters), a multilingual model optimized for 70+ languages. |
| |
|
| | | Model | HF ID | Regional Strength | |
| | |---|---|---| |
| | | **Global** | `CohereLabs/tiny-aya-global` | Balanced across all languages | |
| | | **Fire** | `CohereLabs/tiny-aya-fire` | South Asian (Urdu) | |
| | | **Earth** | `CohereLabs/tiny-aya-earth` | West Asian & African (Amharic) | |
| | | **Water** | `CohereLabs/tiny-aya-water` | European & Asia Pacific (Chinese) | |
| |
|
| | ## Model Structure |
| |
|
| | This repo contains LoRA adapters organized by experimental condition and base model variant: |
| |
|
| | | Subdirectory | Condition | Training Data | |
| | |---|---|---| |
| | | `global/baseline/` | Condition 1 | No code augmentation | |
| | | `global/english-code/` | Condition 2 | English-keyword Python code | |
| | | `global/multilingual-code/` | Condition 3 | Python transpiled to Urdu, Amharic, Chinese keywords | |
| | | `global/multilingual-text/` | Condition 4 | Non-code multilingual text | |
| | | `fire/multilingual-code/` | Regional | Urdu-keyword Python on Fire variant | |
| | | `earth/multilingual-code/` | Regional | Amharic-keyword Python on Earth variant | |
| | | `water/multilingual-code/` | Regional | Chinese-keyword Python on Water variant | |
| |
|
| | ## Usage |
| |
|
| | ```python |
| | from transformers import AutoModelForCausalLM, AutoTokenizer |
| | from peft import PeftModel |
| | |
| | # Load base model (Global variant) |
| | base_model = AutoModelForCausalLM.from_pretrained("CohereLabs/tiny-aya-global") |
| | tokenizer = AutoTokenizer.from_pretrained("CohereLabs/tiny-aya-global") |
| | |
| | # Load a LoRA adapter (e.g., multilingual code on Global) |
| | model = PeftModel.from_pretrained(base_model, "Legesher/language-decoded-lora", subfolder="global/multilingual-code") |
| | |
| | # Or load a regional variant (e.g., Urdu code on Fire) |
| | base_fire = AutoModelForCausalLM.from_pretrained("CohereLabs/tiny-aya-fire") |
| | model_fire = PeftModel.from_pretrained(base_fire, "Legesher/language-decoded-lora", subfolder="fire/multilingual-code") |
| | ``` |
| |
|
| | ## Training Details |
| |
|
| | - **Base models**: Tiny Aya 3.35B — Global, Fire, Earth, Water ([CohereLabs](https://huggingface.co/CohereLabs)) |
| | - **Method**: QLoRA (Quantized Low-Rank Adaptation) |
| | - **Training data**: [Legesher/language-decoded-data](https://huggingface.co/datasets/Legesher/language-decoded-data) |
| | - **Parameters**: 3.35B base, ~0.1% trainable via LoRA |
| |
|
| | *Detailed hyperparameters and training configs will be added as training completes.* |
| |
|
| | ## Evaluation |
| |
|
| | Models are evaluated on multilingual reasoning benchmarks: |
| |
|
| | | Benchmark | Task | Languages | |
| | |---|---|---| |
| | | XNLI | Natural language inference | 15 | |
| | | XStoryCloze | Story completion | 11 | |
| | | TyDi QA | Question answering | 11 | |
| | | MMLU | Knowledge | Multilingual | |
| |
|
| | *Results will be added as evaluation completes.* |
| |
|
| | ## Related Resources |
| |
|
| | - **Base models**: [Tiny Aya Collection](https://huggingface.co/collections/CohereLabs/tiny-aya) |
| | - **Training data**: [Legesher/language-decoded-data](https://huggingface.co/datasets/Legesher/language-decoded-data) |
| | - **Community code**: [Legesher/language-decoded-community](https://huggingface.co/datasets/Legesher/language-decoded-community) |
| | - **Experiments**: [Legesher/language-decoded-experiments](https://huggingface.co/datasets/Legesher/language-decoded-experiments) |
| | - **Transpilation tool**: [Legesher](https://github.com/Legesher/legesher) |
| |
|
| | ## Citation |
| |
|
| | ```bibtex |
| | @misc{language-decoded-2026, |
| | title={Language Decoded: Investigating Language-Dependent vs. Structure-Dependent Reasoning Benefits of Code}, |
| | author={Madison Edgar and Saad Bazaz and Rafay Mustafa and Sarah Jawaid and Rashik Shahjahan and Khojasteh Mirza and Sohaib Bazaz}, |
| | year={2026}, |
| | publisher={Hugging Face}, |
| | url={https://huggingface.co/Legesher/language-decoded-lora} |
| | } |
| | ``` |
| |
|
| | ## License |
| |
|
| | CC-BY-NC 4.0 (inherits from Tiny Aya base models) |
