This repository contains the Multi-Task Learning (MTL) LoRA adapter weights (Checkpoint 25000) fine-tuned on an mT5-base foundational model. This adapter handles translation alongside two auxiliary structural token-classification sequence tasks: Part-of-Speech (POS) tagging and Semantic tagging.

Status: Accepted to TSD 2026 as part of an ablation study exploring parameter capacity limits and negative transfer in synthetic low-resource settings.

Repository Cross-Links

Training Code & Script (GitHub): achulzhanov/mayan-mt5
Multi-Task Training Dataset (Hugging Face): achulz/mayan-mt5-qeqchi-dataset
Baseline Translation (STL) Variant: achulz/mayan-mt5-qeqchi-adapter

Usage and Inference

This model uses a text-to-text formatting schema to share latent representations across multiple distinct linguistic objectives. You must use the explicit prefixes below to toggle model outputs.

Task 1: Bidirectional Translation

Prefix format: translate [Source Language] to [Target Language]:

python

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
from peft import PeftModel

base_model_id = "google/mt5-base"
peft_model_id = "achulz/mayan-mt5-qeqchi-mtl-adapter"

tokenizer = AutoTokenizer.from_pretrained(peft_model_id)
base_model = AutoModelForSeq2SeqLM.from_pretrained(base_model_id)
model = PeftModel.from_pretrained(base_model, peft_model_id)

# Example Translation
prompt = "translate English to Q'eqchi': The dog is sleeping in the house."
inputs = tokenizer(prompt, return_tensors="pt", max_length=128, truncation=True)
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Task 2: Part-of-Speech Tagging

Prefix format: tag POS Q'eqchi':

python

# Example POS Tagging
prompt = "tag POS Q'eqchi': Li Ch'ina'us naxk'e rib'..."
inputs = tokenizer(prompt, return_tensors="pt", max_length=128, truncation=True)
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
# Output format: Token (UD_TAG) Token (UD_TAG) ...

Task 3: Semantic/Named-Entity Tagging

Prefix format: tag semantic Q'eqchi':

python

# Example Semantic Tagging
prompt = "tag semantic Q'eqchi': Laj Carlos ..."
inputs = tokenizer(prompt, return_tensors="pt", max_length=128, truncation=True)
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
# Output format: Token (BIO-Tag) ...

Architectural Insights & Limitations

This adapter represents an experimental negative result from our TSD 2026 paper. While the joint optimization strategy (using weighted focal loss) functions correctly in-domain, forcing a small-rank LoRA adapter (r=32) to simultaneously memorize structural metadata and cross-lingual translation tokens creates a severe parameter bottleneck. This capacity constraint caused over-optimization to synthetic patterns, inducing a high noise floor and worse out-of-domain translation performance compared to the baseline STL model.