Russian scientific T5 model initialized from EN-T5-Sci using WECHSEL and a language-specific SentencePiece 32k tokenizer.
Model Details
This is one of the non-English scientific T5 transfer models from the paper. The model keeps the EN-T5-Sci Transformer weights and reinitializes the language-specific embeddings with WECHSEL using a target SentencePiece tokenizer.
WECHSEL resources: English fastText embeddings + Russian fastText embeddings (ru) with the russian bilingual dictionary.
Evaluation
Zero-shot Global-MMLU accuracy reported by the paper aggregation:
Metric
Accuracy
Average
26.36
STEM
27.12
Humanities
24.89
Social Sciences
28.86
Other
25.33
Limitations
The model is evaluated primarily with zero-shot Global-MMLU. Downstream task-specific
evaluation is recommended before deployment in specialized scientific workflows.
Citation
Title: Transferring Scientific English Pre-Trained Language Models to Multiple Languages Using Cross-Lingual Transfer
Authors: Nikolas Rauscher, Fabio Barth, Georg Rehm
Venue: LREC-COLING 2026, citation details TBA after publication
