This model is a fine-tuned version of rinna/japanese-roberta-base (pre-trained RoBERTa model provided by rinna Co., Ltd.) trained for extractive question answering.

The model is fine-tuned on JaQuAD dataset provided by Skelter Labs, in which data is collected from Japanese Wikipedia articles and annotated by a human.

Intended uses

When running with a dedicated pipeline :

python

from transformers import pipeline

model_name = "tsmatz/roberta_qa_japanese"
qa_pipeline = pipeline(
    "question-answering",
    model=model_name,
    tokenizer=model_name)
result = qa_pipeline(
    question = "決勝トーナメントで日本に勝ったのはどこでしたか。",
    context = "日本は予選リーグで強豪のドイツとスペインに勝って決勝トーナメントに進んだが、クロアチアと対戦して敗れた。",
    align_to_words = False,
)
print(result)

When manually running through forward pass :

python

import torch
import numpy as np
from transformers import AutoModelForQuestionAnswering, AutoTokenizer

model_name = "tsmatz/roberta_qa_japanese"
model = (AutoModelForQuestionAnswering
         .from_pretrained(model_name))
tokenizer = AutoTokenizer.from_pretrained(model_name)

def inference_answer(question, context):
    question = question
    context = context
    test_feature = tokenizer(
        question,
        context,
        max_length=318,
    )
    with torch.no_grad():
        outputs = model(torch.tensor([test_feature["input_ids"]]))
    start_logits = outputs.start_logits.cpu().numpy()
    end_logits = outputs.end_logits.cpu().numpy()
    answer_ids = test_feature["input_ids"][np.argmax(start_logits):np.argmax(end_logits)+1]
    return "".join(tokenizer.batch_decode(answer_ids))

question = "決勝トーナメントで日本に勝ったのはどこでしたか。"
context = "日本は予選リーグで強豪のドイツとスペインに勝って決勝トーナメントに進んだが、クロアチアと対戦して敗れた。"
answer_pred = inference_answer(question, context)
print(answer_pred)

Training procedure

You can download the source code for fine-tuning from here.

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 7e-05
train_batch_size: 2
eval_batch_size: 1
seed: 42
gradient_accumulation_steps: 16
total_train_batch_size: 32
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
lr_scheduler_warmup_steps: 100
num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
2.1293	0.13	150	1.0311
1.1965	0.26	300	0.6723
1.022	0.39	450	0.4838
0.9594	0.53	600	0.5174
0.9187	0.66	750	0.4671
0.8229	0.79	900	0.4650
0.71	0.92	1050	0.2648
0.5436	1.05	1200	0.2665
0.5045	1.19	1350	0.2686
0.5025	1.32	1500	0.2082
0.5213	1.45	1650	0.1715
0.4648	1.58	1800	0.1563
0.4698	1.71	1950	0.1488
0.4823	1.84	2100	0.1050
0.4482	1.97	2250	0.0821
0.2755	2.11	2400	0.0898
0.2834	2.24	2550	0.0964
0.2525	2.37	2700	0.0533
0.2606	2.5	2850	0.0561
0.2467	2.63	3000	0.0601
0.2799	2.77	3150	0.0562
0.2497	2.9	3300	0.0516

Framework versions

Transformers 4.23.1
Pytorch 1.12.1+cu102
Datasets 2.6.1
Tokenizers 0.13.1