This model specialises on answering Closed Question Answering in Japanese. Input a piece of reference text, ask a question, and see the model answer based on the reference text.
We trained on equal samples of the following three datasets:
which resulted in a dataset of 13,167 samples total.
These three datasets were chosen as they represent three distinct fine-tuning tasks (Text simplification, question answering, and text summarization, respectively) which we hypothesize can help to improve the language models suitability for dealing with Japanese data.
These three datasets make up the model name: STX.
With these datasets, we achieve the following scores on the JGLUE benchmark:
Model Name
Open-Orca/OpenOrcaxOpenChat-Preview2-13B
lightblue/openorca_stx
jsquad-1.1-0.3
0.692
0.836
jcommonsenseqa-1.1-0.3
0.831
0.782
jnli-1.1-0.3
0.504
0.48
marc_ja-1.1-0.3
0.936
0.959
We achieved these scores by using the lm-evaluation-harness from Stability AI using the below commands:
Our model achieves much better results on the question answering benchmark (JSQuAD) than the base checkpoint without monstrous degradation of performance on multi-choice question benchmarks (JCommonSense, JNLI, MARC-Ja) purely through QLoRA training.
This shows the potential for applying strong language models such as Open-Orca/OpenOrcaxOpenChat-Preview2-13B to minimal QLoRA fine-tuning using Japanese fine-tuning datasets to achieve better results at narrow NLP tasks.
We have found that this model is able to work well using a variety of prompts, including the Alpaca style templated prompts:
python
12f"""
3Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
4### Instruction:
5{instruction}6### Input:
7{input}8### Response:
9"""10
We have found that having a newline at the end of the prompt can be important for signalling that the model must respond and not continue the inputs.
Training details
We trained using the following three minimalistic prompt templates for the three tasks in STX:
SNOW
python
1f"""元の日本語：
2{original_ja}34シンプルな日本語："""
TyDiQA
python
1f"""{passage_text}23{question_text}"""
XLSum
python
1f"""記事：
2{article_text}34要約："""
This model was trained for 1000 steps (1.2 epochs) with the model being evaluated every 50 steps. We then chose the best model from these evaluations based on validation loss.
We used the qlora package from artidoro.
We trained with the following hyperparameters:
