OvisOCR is designed for information-dense documents containing natural language text, tables, mathematical formulas, figures, and complex layouts. It preserves fine-grained textual fidelity while maintaining global document structure and human reading order. With only 1.3B parameters, OvisOCR achieves outstanding overall performance on OmniDocBench v1.5.
-
Strictly End-to-End Document Parsing
OvisOCR directly maps full-page visual signals to structured Markdown without localized slicing, layout-dependent recognition, or post-hoc merging. This streamlined paradigm reduces error propagation and improves global serialization consistency.
-
Synergistic Data Construction
Our data construction pipeline builds high-quality supervision by combining the strengths of a specialized OCR engine and a general-purpose MLLM. The specialized perceiver supplies dense local evidence, while the general reasoner checks for hallucinations, content completeness, table validity, formula syntax, and logical reading order.
-
Multi-Granularity Alignment
OvisOCR uses element-aware optimization for heterogeneous document constituents. Text, tables, and formulas are optimized with tailored reward signals, including edit-distance-based text fidelity, TEDS-based table similarity, and CDM-based formula visual correctness.
-
Strong Document Parsing Capability with Compact Scale
With only 1.3B parameters, OvisOCR achieves outstanding performance on OmniDocBench v1.5, surpassing strong specialized parsers, large general MLLMs, and traditional pipeline tools.
1from PIL import Image
2from vllm import LLM, SamplingParams
3
4
5class OvisOCRParser:
6 def __init__(self, model_name_or_path: str):
7 self.model = LLM(
8 model=model_name_or_path,
9 tensor_parallel_size=1,
10 trust_remote_code=True,
11 gpu_memory_utilization=0.8,
12 )
13
14 prompt = 'Extract all readable content from the image in natural human reading order and output the result as a single Markdown document. For charts or images, represent them using an HTML image tag: <' + 'img src="images/bbox_{left}_{top}_{right}_{bottom}.jpg" />, where left, top, right, bottom are bounding box coordinates scaled to [0, 1000). Format formulas as LaTeX. Format tables as HTML: <table>...</table>. Transcribe all other text as standard Markdown. Preserve the original text without translation or paraphrasing.'
15 self.prompt = self.model.get_tokenizer().apply_chat_template(
16 [{"role": "user", "content": f"<image>\n{prompt}"}],
17 tokenize=False,
18 add_generation_prompt=True,
19 enable_thinking=False
20 )
21
22 self.sampling_params = SamplingParams(
23 max_tokens=16384,
24 temperature=0.0,
25 )
26
27 def _clean_truncated_repeats(
28 self,
29 text: str,
30 min_text_len: int = 8000,
31 max_period: int = 200,
32 min_period: int = 1,
33 min_repeat_chars: int = 100,
34 min_repeat_times: int = 5
35 ) -> str:
36 n = len(text)
37 if n < min_text_len:
38 return text
39
40 max_period = min(max_period, n - 1)
41 for unit_len in range(min_period, max_period + 1):
42 if text[n - 1] != text[n - 1 - unit_len]:
43 continue
44
45 match_len = 1
46 idx = n - 2
47 while idx >= unit_len and text[idx] == text[idx - unit_len]:
48 match_len += 1
49 idx -= 1
50
51 total_len = match_len + unit_len
52 repeat_times = total_len // unit_len
53 tail_len = total_len % unit_len
54
55 if repeat_times >= min_repeat_times and total_len >= min_repeat_chars:
56 return text[: n - total_len + unit_len] + text[n - tail_len:]
57
58 return text
59
60 def parse(self, images: list[Image.Image], filter_imgtags: bool = True) -> list[str]:
61 vllm_inputs = [
62 {
63 "prompt": self.prompt,
64 "multi_modal_data": {"image": image},
65 "mm_processor_kwargs": {
66 "images_kwargs": {
67 "min_pixels": 448 * 448,
68 "max_pixels": 2880 * 2880,
69 }
70 }
71 }
72 for image in images
73 ]
74
75 outputs = self.model.generate(vllm_inputs, self.sampling_params)
76
77 markdowns = []
78 for output in outputs:
79 text = output.outputs[0].text.strip()
80 if filter_imgtags:
81 text = "\n\n".join(
82 block
83 for block in text.split("\n\n")
84 if not block.strip().startswith('<img src="images/bbox_')
85 )
86 markdowns.append(self._clean_truncated_repeats(text))
87
88 return markdowns
89
90
91if __name__ == "__main__":
92 parser = OvisOCRParser("AIDC-AI/OvisOCR")
93 images = [Image.open("test1.jpg"), Image.open("test2.jpg")]
94 markdowns = parser.parse(images)
95 print(markdowns[0])
1@inproceedings{jiang2026ovisocr,
2 title = {{OvisOCR}: End-to-End Document Parsing via Aligning Specialized Perception with General Reasoning},
3 author = {Jiang, Jun-Peng and Lu, Shiyin and Ji, An-Yang and Li, Yinglun and Chen, Qing-Guo and Xu, Zhao and Luo, Weihua and Zhang, Kaifu and Zhan, De-Chuan and Ye, Han-Jia},
4 booktitle = {Proceedings of the 43rd International Conference on Machine Learning},
5 series = {Proceedings of Machine Learning Research},
6 volume = {306},
7 address = {Seoul, South Korea},
8 publisher = {PMLR},
9 year = {2026}
10}
This project is licensed under the
Apache License, Version 2.0 (SPDX-License-Identifier: Apache-2.0).
We used automated filtering and quality-assurance procedures during data construction to reduce parsing errors such as repeated hallucinations, incomplete content, invalid table/formula structures, and reading-order inconsistencies. Due to the diversity and complexity of real-world documents, OvisOCR may still produce incorrect or incomplete outputs. Please manually verify results in critical applications.
