Original Phase 3 _summary_*.json files on legesher/language-decoded-experiments under-report cond-5 SIB-200 accuracy by 20–35pp because the strict inference-time extractor refused native-script answers. Cite the _summary_reparsed_*.json siblings (refined extractor) instead. Five Phase 3 SIB-200 conclusions also flip win→loss against baseline once the extractor is corrected (cond-2-es-5k, cond-2-es-20k, cond-2-ur-20k, cond-2-zh-20k, cond-3-zh-5k), and cond-2-ur-5k's gain deflates 4.4×. See the banner on the experiments repo (top of the README) for the full picture.

Research Question

How does fine-tuning Tiny Aya on non-English code — whether transpiled, mixed-native, or fully translated — affect its multilingual reasoning and instruction-following, and how does that impact differ from fine-tuning on English code?

The hypothesis is not that non-English code matches or exceeds English code as a generic reasoning aid — rather, that the kind of effect non-English code produces depends on the target language, the data structure, and how the corpus was constructed. See legesher/language-decoded-experiments for the full project context.

Base Model

All adapters are trained on CohereLabs/tiny-aya-base (3.35B parameters). Tiny Aya was chosen because it is small (deployable on a single 16 GB T4 GPU via QLoRA), openly available (released under CC-BY-NC-4.0), and supports 70+ languages with explicit emphasis on lower-resourced ones — which makes the experimental ladder viable for ur at all.

Adapter Inventory

This repo holds adapters from two generations of the project, kept side by side and clearly separated by folder. See the Provenance & Manifest section for a complete path → phase → source-corpus map, and MANIFEST.md for the machine-readable version.

Paper adapters (Phase 3 · The Stack v2-dedup) — live under the tiny-aya-base/ prefix. These are the adapters cited in the submitted paper; cond-1, cond-2, and cond-5 were re-trained from scratch on the cleaner bigcode/the-stack-v2-dedup corpus.
Preliminary adapters (Phase 2 · The Stack v1) — live as flat top-level folders (condition-1-en-32k/, condition-2-zh-5k/, …). These are the original March-2026 hackathon adapters trained on bigcode/the-stack (v1, non-dedup), retained for reproducibility. Do not cite these for the paper.

Paper adapters — Phase 3 · The Stack v2-dedup

Each subdirectory under tiny-aya-base/ is one trained condition × file-volume × seed combination. All adapters share the QLoRA hyperparameters listed under Training Details.

Subdirectory (under `tiny-aya-base/`)	Condition	Training data	Seeds
`tiny-aya-base/condition-1-en-5k-seed{42,123,456}/`	1	Raw English Python from `bigcode/the-stack-v2-dedup` (5k file subset)	42, 123, 456
`tiny-aya-base/condition-1-en-20k-seed42/`	1	Raw English Python (20k file subset)	42
`tiny-aya-base/condition-2-{zh,es,ur}-5k-seed{42,123,456}/`	2	The same 5k subset as cond-1, processed through Legesher v0.7.3 — Python's reserved words (keywords, exceptions, built-in functions, numerical system for some target languages) translated to the target language; user logic preserved	42, 123, 456
`tiny-aya-base/condition-2-{zh,es,ur}-20k-seed42/`	2	The same 20k subset as cond-1, processed through Legesher v0.7.3	42
`tiny-aya-base/condition-3-zh-5k-native-code-seed42/`	3	Community-collected raw Chinese code from varied online public-source repositories (different source-file population from cond-1/2/5 by design)	42
`tiny-aya-base/condition-5-{zh,es,ur}-5k-c4ai-aya-expanse-32b-seed42/`	5	The same 5k subset as cond-1, first transpiled by Legesher v0.7.3 to translate Python's reserved words, then run through `c4ai-aya-expanse-32b` via the Cohere API to translate the remaining content (identifiers, comments, docstrings, string literals)	42

Condition 4 ("Community-Contributed Native Code") is pending sufficient direct community contributions to the legesher/legesher-native-code HF Space; no cond-4 adapter exists yet.

Preliminary adapters — Phase 2 · The Stack v1

These flat top-level folders are the original hackathon adapters, trained on bigcode/the-stack (v1, non-dedup) with Legesher v0.5.1 / v0.6.0. They are superseded by the tiny-aya-base/ Phase 3 adapters above and are kept only for reproducibility of the preliminary results. The 32k size and the single-seed setup are Phase 2 signatures.

Subdirectory (top level)	Condition	Source corpus	Notes
`condition-1-en-32k/`	1	`bigcode/the-stack` (v1)	Phase 2 32k tier; no Phase 3 equivalent
`condition-1-en-5k/`	1	`bigcode/the-stack` (v1)	Preliminary; use `tiny-aya-base/condition-1-en-5k-seed42/` for the paper
`condition-2-es-5k/`	2	`bigcode/the-stack` (v1), Legesher transpiled	Preliminary
`condition-2-ur-5k/`	2	`bigcode/the-stack` (v1), Legesher transpiled	Preliminary
`condition-2-zh-5k/`	2	`bigcode/the-stack` (v1), Legesher transpiled	Preliminary
`condition-3-zh-5k/`	3	Community-collected raw Chinese code	Preliminary; corpus unchanged across phases

The standalone per-adapter repos that previously published these Phase 2 / v1 adapters (legesher/language-decoded-lora-condition-*) have been renamed to legesher/language-decoded-lora-phase-2-the-stack-v1-condition-* and deprecated in favor of this umbrella repo. Their old URLs continue to resolve via Hugging Face redirects.

Source-file control

Cond-1, cond-2, and cond-5 all train on the same 5,000-file subset drawn from bigcode/the-stack-v2-dedup (with a parallel 20k subset for the 20k tier). Differences across these conditions reflect the processing pipeline (raw / transpiled / fully translated), not file-quality or content drift. Cond-3 is the deliberate exception — its source files are a different population by design.

The experimental ladder

Baseline → cond-1: Does code help at all? (Replicates Aryabumi et al., 2024.)
Cond-1 → cond-2: Does translating Python's reserved words (keywords, exceptions, built-in functions, numerical system for some target languages) into the target language change the model's behavior? User logic and library calls remain English-derived.
Cond-2 → cond-3: Does code pulled from real-world public-source repositories — code humans actually wrote in or with the target language — add value beyond Legesher's mechanical translation?
Cond-2 → cond-5: Cond-2 translates only Python's reserved words; cond-5 goes further by translating the rest of the file's content (identifiers, comments, docstrings, string literals) via c4ai-aya-expanse-32b. Logic and structure are preserved.
Cond-3 → cond-5 (implicit): Human-authored vs. machine-synthesized native code.

For the full ladder including future directions (natural-language text control, combined-language training, similar-script evaluation), see legesher/language-decoded-experiments.

Provenance & Manifest

The two adapter generations are distinguished by folder location and source corpus, matching the convention used across the project's repos (phase-2-the-stack-v1-* on language-decoded-data, phase2/÷phase3/ on language-decoded-experiments):

Generation	Location in this repo	Source corpus	Legesher	Tier / seeds	Cite for paper?
Phase 3 (paper)	`tiny-aya-base/…-seed*/`	`bigcode/the-stack-v2-dedup`	v0.7.3	5k (3 seeds) + 20k (1 seed)	✅ Yes
Phase 2 (preliminary)	flat top-level `condition-*/`	`bigcode/the-stack` (v1)	v0.5.1 / v0.6.0	5k / 32k (1 seed)	❌ No

A complete, machine-readable path → phase → corpus → condition map is in MANIFEST.md. Training-data provenance for each condition is detailed on language-decoded-data; the phase comparison is in the "Phase 2 → Phase 3 at a glance" table on the experiments repo.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained("CohereLabs/tiny-aya-base")
tokenizer = AutoTokenizer.from_pretrained("CohereLabs/tiny-aya-base")

# Load a paper (Phase 3 · Stack v2-dedup) adapter — e.g., cond-1 (English code, seed 42, 5k tier).
# Paper adapters live under the `tiny-aya-base/` prefix.
model = PeftModel.from_pretrained(
    base_model,
    "legesher/language-decoded-lora",
    subfolder="tiny-aya-base/condition-1-en-5k-seed42",
)

# Or a language-specific cond-2 adapter (Chinese reserved-word translation, seed 42)
model = PeftModel.from_pretrained(
    base_model,
    "legesher/language-decoded-lora",
    subfolder="tiny-aya-base/condition-2-zh-5k-seed42",
)

# Or a cond-5 adapter (Synthesized Native Code, Urdu, seed 42)
model = PeftModel.from_pretrained(
    base_model,
    "legesher/language-decoded-lora",
    subfolder="tiny-aya-base/condition-5-ur-5k-c4ai-aya-expanse-32b-seed42",
)

# To load a *preliminary* Phase 2 / Stack v1 adapter instead, use the flat top-level
# folder (no `tiny-aya-base/` prefix) — e.g. the original cond-2 Chinese hackathon adapter:
model = PeftModel.from_pretrained(
    base_model,
    "legesher/language-decoded-lora",
    subfolder="condition-2-zh-5k",
)

Training Details

Parameter	Value
Base model	CohereLabs/tiny-aya-base (3.35B params, 70+ languages, low-resource emphasis)
Method	QLoRA 4-bit (NF4), ~5.4 GB VRAM, Unsloth-accelerated
Hardware	Kaggle T4 (16 GB)
Tokenizer	`CohereLabs/tiny-aya-base`
Transpilation tool	Legesher v0.7.3 (Phase 3); v0.5.1 / v0.6.0 used in Phase 2
Cond-5 translation	`c4ai-aya-expanse-32b` accessed via the Cohere API (made possible by Cohere credits awarded to Legesher)
Training data	legesher/language-decoded-data

QLoRA hyperparameters

Parameter	Value
LoRA rank (`r`)	16
LoRA alpha	32
LoRA dropout	0.0
Target modules	q_proj, k_proj, v_proj, o_proj, up_proj, down_proj, gate_proj
Bias	none
Task type	CAUSAL_LM
PEFT version	0.18.1
Quantization	NF4 (4-bit) via Unsloth

Evaluation

Phase 3 models are evaluated on four multilingual benchmarks under template1 (English-prompt) and template2 (native-prompt) across the full data_lang × instr_lang matrix:

Benchmark	What it measures	Examples per language
XNLI	Natural-language inference	~5,000
X-CSQA	Commonsense reasoning	~1,000
SIB-200	Topic classification	~204
Belebele	Reading comprehension	~900

MGSM was used in Phase 2 and dropped from Phase 3 — at 3.35B parameters and 250 examples per language, scores ranged 2.8% – 10.8% across all conditions with most condition-to-condition differences within noise. A useful null result; budget was reallocated to SIB-200 and Belebele.

Paper-grade evaluation results live on legesher/language-decoded-experiments — see the refined-tables and the writeup at expedition-tiny-aya/analysis/phase-3/phase3-refined-evaluation.md.

Limitations

Single base model: All adapters are trained on CohereLabs/tiny-aya-base (3.35B params). Results may not generalize to larger or architecturally different models. Future iterations will expand to additional base models.
Per-language fine-tuning only: Every condition is per-language — each cond-2-{zh,es,ur}-5k (and cond-5-{zh,es,ur}-5k) is a separate training run. Combined-language training is a planned future condition.
Limited training data: 5k and 20k file tiers are constrained by Kaggle T4 hardware limits. 103k variants exist on the training data repo but no 103k adapters have been trained yet.
Consumer hardware: Training on Kaggle T4 (16 GB) with 4-bit quantization introduces approximation that may affect adapter quality compared to full-precision training.
Extractor coverage — when citing Phase 3 results, use the refined-extractor scores. See the banner at the top of this card and the experiments repo for full details.

Related Resources

Experiment tracking and results: legesher/language-decoded-experiments (canonical project source-of-truth)
Training data: legesher/language-decoded-data
Community native code: legesher/language-decoded-community
Cond-4 contribution interface: legesher/legesher-native-code HF Space
Transpilation tool: Legesher on GitHub

Citation

bibtex

@misc{language-decoded-2026,
  title={Language Decoded: Exploring the Impact of Native Code on Multilingual Models},
  author={Madison Edgar and Saad Ahmed Bazaz and Tom Sherborne and Rashik Shahjahan and Khojasteh Mirza and Sarah Jawaid and Rafay Mustafa and Sohaib Ahmed Bazaz},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/legesher/language-decoded-lora}
}

License

CC-BY-NC-4.0. The adapters inherit the license of the base model, CohereLabs/tiny-aya-base (CC-BY-NC-4.0). The training datasets (legesher/language-decoded-data) are separately licensed under Apache-2.0.

Provenance, attribution & takedown

These adapters were fine-tuned from CohereLabs/tiny-aya-base on a specific revision of the legesher/language-decoded-data training conditions (see each adapter's configuration for the condition and revision).

If you are the author of source code included in the training data and would like attribution added or your code removed, open a discussion on the dataset repository's Community tab or email support@legesher.com. Removals are propagated in a new dataset revision. Adapters already trained are frozen historical artifacts: a dataset removal does not alter existing adapter weights, but we will note affected conditions here and take reported concerns about specific adapters into account.

Usage caution. These are research artifacts, not production-ready models. Documented side effects include code fragments leaking into natural-language output (strongest for Urdu adapters) and matched-language regressions on specific evaluation cells; the Condition 5 adapters were trained on corpora containing raw translator output. Evaluate per language and per task before any downstream use.