LayoutLMv3-Japanese-preview is a multimodal pre-trained model for Japanese Document AI,
built on the LayoutLMv3 architecture.
The text tokenizer is replaced with a Japanese tokenizer from
nlp-waseda/roberta-base-japanese,
and the visual tokenizer adopts BEiT v2 VQ-KD.
The model is pre-trained on ~20M Japanese web pages from NDL WARP ,PubLayNet and DocLayNet for document layout analysis.
The tokenizer is an AlbertTokenizer (SentencePiece) inherited from
nlp-waseda/roberta-base-japanese,
so the standard LayoutLMv3Processor cannot be used as-is. Instead,
combine AutoTokenizer with LayoutLMv3ImageProcessor and align each
subword's bbox with the source word.
Requirements: transformers>=4.44, torch, Pillow. PyMuPDF
(imported as fitz) is optional — only needed if you want to
render a PDF on the fly. If you just run the provided sample.png,
you do not need PyMuPDF. Words and bounding boxes can be produced by
any OCR engine that supports Japanese (e.g. PaddleOCR, Tesseract with
jpn traineddata, or manga-ocr). Boxes must be in pixel coordinates
(x0, y0, x1, y1) and normalized to LayoutLMv3's 0–1000 range.
A sample Japanese document image is provided as sample.png in this
repository for quick experimentation. The snippet below supports two
input paths: load sample.png directly, or render the first page of
any PDF via fitz (PyMuPDF) — uncomment the branch you want.
sample document
python
1import io
23import torch
4from PIL import Image
5from transformers import AutoTokenizer, LayoutLMv3ImageProcessor, LayoutLMv3Model
67REPO ="llm-jp/layoutlmv3-japanese-preview"89tokenizer = AutoTokenizer.from_pretrained(REPO)10image_processor = LayoutLMv3ImageProcessor(apply_ocr=False)11model = LayoutLMv3Model.from_pretrained(REPO).eval()1213# --- Option A: use the bundled sample.png (no PyMuPDF required) ---14image = Image.open("sample.png").convert("RGB")1516# --- Option B: render the first page of any PDF via fitz (PyMuPDF) ---17# import fitz18# doc = fitz.open("your_document.pdf")19# pix = doc[0].get_pixmap(dpi=200)20# image = Image.open(io.BytesIO(pix.tobytes("png"))).convert("RGB")2122width, height = image.size
2324# Replace with your OCR output. Each box is (x0, y0, x1, y1) in pixel coords.25words =["石巻市駅前北通り","災害公営住宅","完成資料"]26boxes =[27(160,28,420,70),28(430,28,610,70),29(625,28,770,70),30]3132defnormalize(box, w, h):33 x0, y0, x1, y1 = box
34return[35int(1000* x0 / w),int(1000* y0 / h),36int(1000* x1 / w),int(1000* y1 / h),37]3839# Tokenize each word and assign the word's bbox to all of its subwords.40input_ids =[tokenizer.cls_token_id]41bbox =[[0,0,0,0]]42for word, box inzip(words,[normalize(b, width, height)for b in boxes]):43 ids = tokenizer(word, add_special_tokens=False)["input_ids"]44 input_ids += ids
45 bbox +=[box]*len(ids)46input_ids.append(tokenizer.sep_token_id)47bbox.append([1000,1000,1000,1000])4849input_ids = torch.tensor([input_ids])50bbox = torch.tensor([bbox])51attention_mask = torch.ones_like(input_ids)52pixel_values = image_processor(images=image, return_tensors="pt")["pixel_values"]5354with torch.no_grad():55 outputs = model(56 input_ids=input_ids,57 bbox=bbox,58 attention_mask=attention_mask,59 pixel_values=pixel_values,60)6162# (batch, num_text_tokens + num_image_patches, hidden_size) — e.g. (1, 213, 768)63print(outputs.last_hidden_state.shape)
For downstream tasks (token classification, QA, etc.), swap LayoutLMv3Model
for the corresponding task head class such as
LayoutLMv3ForTokenClassification or LayoutLMv3ForQuestionAnswering
and fine-tune on your labeled data.
License
This model is licensed under Apache 2.0.
Pre-training Pipeline (Text & Layout Extraction)
For the NDL WARP PDF corpus, word-level text and bounding boxes were
extracted with PyMuPDF (fitz) —
not a learned OCR model — to build (word, bbox) pairs used as the 1D
text + 2D layout inputs for pre-training. PyMuPDF was used strictly as
an internal data-processing tool: the library was not modified,
was not redistributed, and is neither embedded in nor linked to the
published model weights. Under AGPL v3, the copyleft obligations
(§5 "Conveying Modified Source Versions" and §13 "Remote Network
Interaction") attach only when the covered software itself is
conveyed or served to users over a network. Running PyMuPDF
locally to produce derived data (text strings and coordinate tuples),
and then training a separate model on that data, is a permitted use
and does not cause AGPL terms to propagate to the resulting model
weights. We therefore consider this usage to be license-compliant.
The weights distributed here remain Apache 2.0.
The document images in PubLayNet originate from the PMC Open Access
Commercial Use Collection, which includes articles under CC0, CC BY,
CC BY-SA, and CC BY-ND licenses. We used these documents for model training under the application of Article 30-4 of the Japanese Copyright Law (2026).
Users outside Japan should assess the applicability of their
local copyright exceptions when using this model.
