This is a
LayoutLM model pretrained on texts in the Japanese language.
The model is primarily aimed at being fine-tuned on a token classification task. You can use the raw model for masked language modeling, although it is not the primary use case. Refer to
https://github.com/nishiwakikazutaka/shinra2022-task2_jrird for instructions on how to fine-tune the model. Note that the linked repository is written in Japanese.
Use the code below to get started with the model.
1>>> from transformers import AutoTokenizer, AutoModel
2>>> import torch
3
4>>> tokenizer = AutoTokenizer.from_pretrained("jri-advtechlab/layoutlm-wikipedia-ja")
5>>> model = AutoModel.from_pretrained("jri-advtechlab/layoutlm-wikipedia-ja")
6
7>>> tokens = tokenizer.tokenize("こんにちは") # ['こん', '##にち', '##は']
8>>> normalized_token_boxes = [[637, 773, 693, 782], [693, 773, 749, 782], [749, 773, 775, 782]]
9>>> # add bounding boxes of cls + sep tokens
10>>> bbox = [[0, 0, 0, 0]] + normalized_token_boxes + [[1000, 1000, 1000, 1000]]
11
12>>> input_ids = [tokenizer.cls_token_id] \
13 + tokenizer.convert_tokens_to_ids(tokens) \
14 + [tokenizer.sep_token_id]
15>>> attention_mask = [1] * len(input_ids)
16>>> token_type_ids = [0] * len(input_ids)
17>>> encoding = {
18 "input_ids": torch.tensor([input_ids]),
19 "attention_mask": torch.tensor([attention_mask]),
20 "token_type_ids": torch.tensor([token_type_ids]),
21 "bbox": torch.tensor([bbox]),
22 }
23
24>>> outputs = model(**encoding)
The model is trained on the Japanese version of Wikipedia. The training corpus is distributed as
training data of the SHINRA 2022 shared task.
We used the tokenizer of
cl-tohoku/bert-base-japanese-v2 to split texts into tokens (subwords). Each token is wrapped in a
<span> tag with the no-wrap value set for the white-space property and localized by obtaining
BoundingClientRect. The localization process was conducted with Google Chrome (106.0.5249.119) headless mode on Ubuntu 20.04.5 LTS with a 1,280*854 window size.
The vocabulary is the same as
cl-tohoku/bert-base-japanese-v2.
The model was trained using Masked Visual-Language Model (MVLM), but it was not trained using Multi-label Document Classification (MDC). We made this decision because we did not identify significant visual differences, such as those between a contract and an invoice, between the different Wikipedia articles.
All parameters except the 2-D Position Embedding were initialized with weights from
cl-tohoku/bert-base-japanese-v2. We initialized the 2-D Position Embedding with random values.
The model was trained on 8 NVIDIA A100 SXM4 GPUs for 100,000 steps, with a batch size of 256 with a maximum sequence length of 512. The optimizer used is Adam with a learning rate of 5e-5, β1=0.9, β2=0.999, learning rate warmup for 1,000 steps, and linear decay of the learning rate after. Additionally, we utilized fp16 mixed precision during training. The training took about 5.3 hours to finish.
Our fine-tuned model achieved a macro-f1 score of 55.1451 on the leaderboard for the SHINRA 2022 shared task. You can check the leaderboard at
https://2022.shinra-project.info/#leaderboard for detailed information.
1@inproceedings{nishiwaki2023layoutlm-wiki-ja,
2 title = {日本語情報抽出タスクのための{L}ayout{LM}モデルの評価},
3 author = {西脇一尊 and 大沼俊輔 and 門脇一真},
4 booktitle = {言語処理学会第29回年次大会(NLP2023)予稿集},
5 year = {2023},
6 pages = {522--527}
7}
