docs(readme): align with canonical source-of-truth (cond-5 adapters, source-file control, refined extractor)

README.md

@@ -25,40 +25,50 @@ pipeline_tag: text-generation
 
 QLoRA adapters fine-tuned on multilingual code conditions for the **Language Decoded** project (part of [Cohere's Tiny Aya Expedition](https://aya.for.ai)).
 
+> **Submitted paper title (2026-05-26):** _Language, Decoded: Exploring the Impact of Fine-Tuning a Multilingual Model on Native-Language Code_
+
 ## ⚠️ Phase 3 eval numbers — read the experiments repo before citing
 
 Original Phase 3 `_summary_*.json` files on [`legesher/language-decoded-experiments`](https://huggingface.co/datasets/legesher/language-decoded-experiments) **under-report cond-5 SIB-200 accuracy by 20–35pp** because the strict inference-time extractor refused native-script answers. Cite the `_summary_reparsed_*.json` siblings (refined extractor) instead. **Five** Phase 3 SIB-200 conclusions also flip win→loss against baseline once the extractor is corrected (`cond-2-es-5k`, `cond-2-es-20k`, `cond-2-ur-20k`, `cond-2-zh-20k`, `cond-3-zh-5k`), and `cond-2-ur-5k`'s gain deflates 4.4×. See the [banner on the experiments repo](https://huggingface.co/datasets/legesher/language-decoded-experiments) (top of the README) for the full picture.
 
 ## Research Question
 
-> Does fine-tuning on non-English code improve multilingual reasoning — and is the benefit language-dependent or structure-dependent?
+> **How does fine-tuning Tiny Aya on non-English code — whether transpiled, mixed-native, or fully translated — affect its multilingual reasoning and instruction-following, and how does that impact _differ_ from fine-tuning on English code?**
+
+The hypothesis is **not** that non-English code matches or exceeds English code as a generic reasoning aid — rather, that the _kind_ of effect non-English code produces depends on the target language, the data structure, and how the corpus was constructed. See [legesher/language-decoded-experiments](https://huggingface.co/datasets/legesher/language-decoded-experiments) for the full project context.
 
 ## Base Model
 
-All adapters are trained on [CohereLabs/tiny-aya-base](https://huggingface.co/CohereLabs/tiny-aya-base) (3.35B parameters).
+All adapters are trained on [CohereLabs/tiny-aya-base](https://huggingface.co/CohereLabs/tiny-aya-base) (3.35B parameters). Tiny Aya was chosen because it is small (deployable on a single 16 GB T4 GPU via QLoRA), accessible (Apache 2.0-licensed), and supports 70+ languages with explicit emphasis on lower-resourced ones — which makes the experimental ladder viable for `ur` at all.
+
+## Adapter Inventory
+
+Each subdirectory is one trained condition × file-volume × seed combination. All adapters share the QLoRA hyperparameters listed under [Training Details](#training-details).
 
-## Model Structure
+| Subdirectory                                                | Condition | Training data                                                                                                                                  | Seeds      |
+| ----------------------------------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
+| `condition-1-en-5k-seed{42,123,456}/`                       | 1         | Raw English Python from `bigcode/the-stack-v2-dedup` (5k file subset)                                                                          | 42, 123, 456 |
+| `condition-1-en-20k-seed42/`                                | 1         | Raw English Python (20k file subset)                                                                                                           | 42         |
+| `condition-2-{zh,es,ur}-5k-seed{42,123,456}/`               | 2         | The **same 5k subset as cond-1**, processed through Legesher v0.7.3 — Python's reserved words (keywords, exceptions, built-in functions, numerical system for some target languages) translated to the target language; user logic preserved | 42, 123, 456 |
+| `condition-2-{zh,es,ur}-20k-seed42/`                        | 2         | The **same 20k subset as cond-1**, processed through Legesher v0.7.3                                                                           | 42         |
+| `condition-3-zh-5k-native-code-seed42/`                     | 3         | Community-collected raw Chinese code from varied online public-source repositories (different source-file population from cond-1/2/5 by design) | 42         |
+| `condition-5-{zh,es,ur}-5k-c4ai-aya-expanse-32b-seed42/`    | 5         | The **same 5k subset as cond-1**, first transpiled by Legesher v0.7.3 to translate Python's reserved words, then run through `c4ai-aya-expanse-32b` via the Cohere API to translate the remaining content (identifiers, comments, docstrings, string literals) | 42         |
 
-This repo is the canonical hub for the trained-from-scratch LoRA adapters, organized by experimental condition:
+**Condition 4 ("Community-Contributed Native Code")** is pending sufficient direct community contributions to the [`legesher/legesher-native-code`](https://huggingface.co/spaces/legesher/legesher-native-code) HF Space; no cond-4 adapter exists yet.
 
-| Subdirectory          | Condition   | Training Data                                         |
-| --------------------- | ----------- | ----------------------------------------------------- |
-| `condition-1-en-32k/` | Condition 1 | English Python from The Stack Dedup (full 32k corpus) |
-| `condition-1-en-5k/`  | Condition 1 | English Python from The Stack Dedup (5k subset)       |
-| `condition-2-zh-5k/`  | Condition 2 | Chinese keyword-swapped Python (Legesher-transpiled)  |
-| `condition-2-es-5k/`  | Condition 2 | Spanish keyword-swapped Python (Legesher-transpiled)  |
-| `condition-2-ur-5k/`  | Condition 2 | Urdu keyword-swapped Python (Legesher-transpiled)     |
-| `condition-3-zh-5k/`  | Condition 3 | Transpiled + native Chinese code (blended)            |
+### Source-file control
 
-**Cond-5 (cross-lingual transfer)** is an evaluation pattern that re-uses condition-2 adapters with cross-language prompting — see [`phase3/conditions/condition-5-{zh,es,ur}-5k/`](https://huggingface.co/datasets/legesher/language-decoded-experiments/tree/main/phase3/conditions) on the experiments repo for the cross-lingual eval results.
+Cond-1, cond-2, and cond-5 all train on the **same 5,000-file subset** drawn from `bigcode/the-stack-v2-dedup` (with a parallel 20k subset for the 20k tier). Differences across these conditions reflect the processing pipeline (raw / transpiled / fully translated), not file-quality or content drift. Cond-3 is the deliberate exception — its source files are a different population by design.
 
-### The Experimental Ladder
+### The experimental ladder
 
-- **Baseline --> 1**: Does code help at all?
-- **1 --> 2**: Does the language of keywords matter?
-- **2 --> 3**: Does diversity of native-language sources add value beyond keyword swap?
-- **3 --> 4**: Does code written in the cultural context of a language carry unique signal?
-- **--> 5**: Does shared script or language family create transfer effects when an adapter trained on one language is prompted in another?
+- **Baseline → cond-1**: Does code help at all? (Replicates [Aryabumi et al., 2024](https://arxiv.org/abs/2408.10914).)
+- **Cond-1 → cond-2**: Does translating Python's reserved words (keywords, exceptions, built-in functions, numerical system for some target languages) into the target language change the model's behavior? User logic and library calls remain English-derived.
+- **Cond-2 → cond-3**: Does code pulled from real-world public-source repositories — code humans actually wrote in or with the target language — add value beyond Legesher's mechanical translation?
+- **Cond-2 → cond-5**: Cond-2 translates only Python's reserved words; cond-5 goes further by translating the rest of the file's content (identifiers, comments, docstrings, string literals) via `c4ai-aya-expanse-32b`. Logic and structure are preserved.
+- **Cond-3 → cond-5** (implicit): Human-authored vs. machine-synthesized native code.
+
+For the full ladder including future directions (natural-language text control, combined-language training, similar-script evaluation), see [legesher/language-decoded-experiments](https://huggingface.co/datasets/legesher/language-decoded-experiments).
 
 ## Usage
 
@@ -70,25 +80,41 @@ from peft import PeftModel
 base_model = AutoModelForCausalLM.from_pretrained("CohereLabs/tiny-aya-base")
 tokenizer = AutoTokenizer.from_pretrained("CohereLabs/tiny-aya-base")
 
-# Load a LoRA adapter (e.g., Condition 1 — English code)
-model = PeftModel.from_pretrained(base_model, "legesher/language-decoded-lora", subfolder="condition-1-en-5k")
-
-# Load a language-specific adapter (e.g., Condition 2 — Chinese keyword-swapped)
-model = PeftModel.from_pretrained(base_model, "legesher/language-decoded-lora", subfolder="condition-2-zh-5k")
+# Load a LoRA adapter — e.g., cond-1 (English code, seed 42, 5k tier)
+model = PeftModel.from_pretrained(
+    base_model,
+    "legesher/language-decoded-lora",
+    subfolder="condition-1-en-5k-seed42",
+)
+
+# Or a language-specific cond-2 adapter (Chinese reserved-word translation, seed 42)
+model = PeftModel.from_pretrained(
+    base_model,
+    "legesher/language-decoded-lora",
+    subfolder="condition-2-zh-5k-seed42",
+)
+
+# Or a cond-5 adapter (Synthesized Native Code, Urdu, seed 42)
+model = PeftModel.from_pretrained(
+    base_model,
+    "legesher/language-decoded-lora",
+    subfolder="condition-5-ur-5k-c4ai-aya-expanse-32b-seed42",
+)
 ```
 
 ## Training Details
 
-| Parameter          | Value                                                                                            |
-| ------------------ | ------------------------------------------------------------------------------------------------ |
-| Base model         | [CohereLabs/tiny-aya-base](https://huggingface.co/CohereLabs/tiny-aya-base) (3.35B params)       |
-| Method             | QLoRA 4-bit (NF4), ~5.4GB VRAM                                                                   |
-| Hardware           | Kaggle T4 (16GB)                                                                                 |
-| Tokenizer          | CohereLabs/tiny-aya-base                                                                         |
-| Transpilation tool | [Legesher](https://github.com/legesher/legesher) v0.7.3                                          |
-| Training data      | [legesher/language-decoded-data](https://huggingface.co/datasets/legesher/language-decoded-data) |
+| Parameter          | Value                                                                                                  |
+| ------------------ | ------------------------------------------------------------------------------------------------------ |
+| Base model         | [CohereLabs/tiny-aya-base](https://huggingface.co/CohereLabs/tiny-aya-base) (3.35B params, 70+ languages, low-resource emphasis) |
+| Method             | QLoRA 4-bit (NF4), ~5.4 GB VRAM, Unsloth-accelerated                                                   |
+| Hardware           | Kaggle T4 (16 GB)                                                                                      |
+| Tokenizer          | `CohereLabs/tiny-aya-base`                                                                             |
+| Transpilation tool | [Legesher](https://github.com/legesher/legesher) v0.7.3 (Phase 3); v0.5.1 / v0.6.0 used in Phase 2     |
+| Cond-5 translation | [`c4ai-aya-expanse-32b`](https://huggingface.co/CohereLabs/aya-expanse-32b) accessed via the Cohere API (made possible by Cohere credits awarded to Legesher) |
+| Training data      | [legesher/language-decoded-data](https://huggingface.co/datasets/legesher/language-decoded-data)       |
 
-### QLoRA Hyperparameters
+### QLoRA hyperparameters
 
 | Parameter       | Value                                                         |
 | --------------- | ------------------------------------------------------------- |
@@ -103,38 +129,40 @@ model = PeftModel.from_pretrained(base_model, "legesher/language-decoded-lora",
 
 ## Evaluation
 
-Models are evaluated on multilingual reasoning benchmarks with dual prompts (English + language-specific). Phase 3 adds SIB-200 and Belebele to the Phase 2 benchmark set.
+Phase 3 models are evaluated on four multilingual benchmarks under `template1` (English-prompt) and `template2` (native-prompt) across the full `data_lang × instr_lang` matrix:
+
+| Benchmark | What it measures           | Examples per language |
+| --------- | -------------------------- | --------------------- |
+| XNLI      | Natural-language inference | ~5,000                |
+| X-CSQA    | Commonsense reasoning      | ~1,000                |
+| SIB-200   | Topic classification       | ~204                  |
+| Belebele  | Reading comprehension      | ~900                  |
 
-| Benchmark | What it measures           | Phase | Examples per language |
-| --------- | -------------------------- | ----- | --------------------- |
-| MGSM      | Math reasoning             | 2, 3  | 250                   |
-| X-CSQA    | Commonsense reasoning      | 2, 3  | ~1,000                |
-| XNLI      | Natural language inference | 2, 3  | ~5,000                |
-| SIB-200   | Topic classification       | 3     | ~204                  |
-| Belebele  | Reading comprehension      | 3     | ~900                  |
+MGSM was used in Phase 2 and **dropped from Phase 3** — at 3.35B parameters and 250 examples per language, scores ranged 2.8% – 10.8% across all conditions with most condition-to-condition differences within noise. A useful null result; budget was reallocated to SIB-200 and Belebele.
 
-Eval results live at [`legesher/language-decoded-experiments`](https://huggingface.co/datasets/legesher/language-decoded-experiments). **Cite `_summary_reparsed_*.json` files for Phase 3 numbers** — see the banner above.
+Paper-grade evaluation results live on [`legesher/language-decoded-experiments`](https://huggingface.co/datasets/legesher/language-decoded-experiments) — see the refined-tables and the writeup at [`expedition-tiny-aya/analysis/phase-3/phase3-refined-evaluation.md`](https://github.com/legesher/research/blob/main/expedition-tiny-aya/analysis/phase-3/phase3-refined-evaluation.md).
 
 ## Limitations
 
-- **Single base model**: All adapters are trained on CohereLabs/tiny-aya-base (3.35B params). Results may not generalize to larger or architecturally different models.
-- **Limited training data**: Each condition uses a 5k-file subset for QLoRA fine-tuning, constrained by Kaggle T4 hardware limits.
-- **Evaluation scope**: Currently evaluated on 5 benchmarks (MGSM, X-CSQA, XNLI, SIB-200, Belebele). Other reasoning tasks may show different patterns.
-- **Consumer hardware**: Training on Kaggle T4 (16GB) with 4-bit quantization introduces approximation that may affect adapter quality compared to full-precision training.
-- **Extractor coverage**: Phase 3 inference-time extractor under-counts native-script SIB-200 answers; refined post-hoc extractor recovers them. See the banner above and [`expedition-tiny-aya/analysis/phase-3/phase3-refined-evaluation.md`](https://github.com/legesher/research/blob/main/expedition-tiny-aya/analysis/phase-3/phase3-refined-evaluation.md) on the research repo.
+- **Single base model**: All adapters are trained on `CohereLabs/tiny-aya-base` (3.35B params). Results may not generalize to larger or architecturally different models. Future iterations will expand to additional base models.
+- **Per-language fine-tuning only**: Every condition is per-language — each `cond-2-{zh,es,ur}-5k` (and `cond-5-{zh,es,ur}-5k`) is a separate training run. Combined-language training is a planned future condition.
+- **Limited training data**: 5k and 20k file tiers are constrained by Kaggle T4 hardware limits. 103k variants exist on the [training data repo](https://huggingface.co/datasets/legesher/language-decoded-data) but no 103k adapters have been trained yet.
+- **Consumer hardware**: Training on Kaggle T4 (16 GB) with 4-bit quantization introduces approximation that may affect adapter quality compared to full-precision training.
+- **Extractor coverage** — when citing Phase 3 results, use the refined-extractor scores. See the banner at the top of this card and the experiments repo for full details.
 
 ## Related Resources
 
+- **Experiment tracking and results**: [legesher/language-decoded-experiments](https://huggingface.co/datasets/legesher/language-decoded-experiments) (canonical project source-of-truth)
 - **Training data**: [legesher/language-decoded-data](https://huggingface.co/datasets/legesher/language-decoded-data)
-- **Community code**: [legesher/language-decoded-community](https://huggingface.co/datasets/legesher/language-decoded-community)
-- **Experiment tracking**: [legesher/language-decoded-experiments](https://huggingface.co/datasets/legesher/language-decoded-experiments)
+- **Community native code**: [legesher/language-decoded-community](https://huggingface.co/datasets/legesher/language-decoded-community)
+- **Cond-4 contribution interface**: [`legesher/legesher-native-code`](https://huggingface.co/spaces/legesher/legesher-native-code) HF Space
 - **Transpilation tool**: [Legesher on GitHub](https://github.com/legesher/legesher)
 
 ## Citation
 
 ```bibtex
 @misc{language-decoded-2026,
-  title={Language Decoded: Investigating Language-Dependent vs. Structure-Dependent Reasoning Benefits of Code},
+  title={Language Decoded: Exploring the Impact of Native Code on Multilingual Models},
   author={Madison Edgar and Saad Ahmed Bazaz and Tom Sherborne and Rashik Shahjahan and Khojasteh Mirza and Sarah Jawaid and Rafay Mustafa and Sohaib Ahmed Bazaz},
   year={2026},
   publisher={Hugging Face},