This repository contains a fine-tuned mT5-small (Multilingual Text-to-Text Transfer Transformer) model on the ParaShoot dataset (github). To enhance its performance, a "domain-specific" fine-tuning approach was employed. Initially, the model was pretrained on a Hebrew dataset to capture Hebrew linguistic nuances. Subsequently, I further fine-tuned the model on the ParaShoot dataset, aiming to improve its proficiency in the Question-Answering task. This model builds upon the original work by imvladikon who initially fine-tuned the mT5-small model for the summarization task.

mT5 is pretrained on the mC4 corpus, covering 101 languages. Note: mT5 was only pre-trained on mC4 excluding any supervised training. Therefore, this model has to be fine-tuned before it is useable on a downstream task.

Paper: mT5: A massively multilingual pre-trained text-to-text transformer Authors: Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, Colin Raffel

Paper: Multilingual Sequence-to-Sequence Models for Hebrew NLP Authors: Matan Eyal, Hila Noga, Roee Aharoni, Idan Szpektor, Reut Tsarfaty

Paper: PARASHOOT: A Hebrew Question Answering Dataset Authors: Omri Keren, Omer Levy

Note: In the paper Multilingual Sequence-to-Sequence Models for Hebrew NLP the results were F1 - 48.71, EM - 24.52.

Use the code below to get started with the model.