Neural Metrics · General OCR that does not flinch at formulas, tables or sheet music.

Attribution

This is an unmodified fork of stepfun-ai/GOT-OCR2_0, created by the Qwen team. All weights, files and behaviour are identical to upstream — we rehost it so our experiments stay reproducible and version-pinned. The original license and all credit remain with the Qwen team. If you want the canonical model, please use the original.

Original model card from stepfun-ai/GOT-OCR2_0 (click to expand)

General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model

🔋Online Demo | 🌟GitHub | 📜Paper

Haoran Wei*, Chenglong Liu*, Jinyue Chen, Jia Wang, Lingyu Kong, Yanming Xu, Zheng Ge, Liang Zhao, Jianjian Sun, Yuang Peng, Chunrui Han, Xiangyu Zhang

Usage

Inference using Huggingface transformers on NVIDIA GPUs. Requirements tested on python 3.10：

torch==2.0.1
torchvision==0.15.2
transformers==4.37.2
tiktoken==0.6.0
verovio==4.3.1
accelerate==0.28.0

python

1from transformers import AutoModel, AutoTokenizer 2 3tokenizer = AutoTokenizer.from_pretrained('ucaslcl/GOT-OCR2_0', trust_remote_code=True) 4model = AutoModel.from_pretrained('ucaslcl/GOT-OCR2_0', trust_remote_code=True, low_cpu_mem_usage=True, device_map='cuda', use_safetensors=True, pad_token_id=tokenizer.eos_token_id) 5model = model.eval().cuda() 6 7 8# input your test image 9image_file = 'xxx.jpg' 10 11# plain texts OCR 12res = model.chat(tokenizer, image_file, ocr_type='ocr') 13 14# format texts OCR: 15# res = model.chat(tokenizer, image_file, ocr_type='format') 16 17# fine-grained OCR: 18# res = model.chat(tokenizer, image_file, ocr_type='ocr', ocr_box='') 19# res = model.chat(tokenizer, image_file, ocr_type='format', ocr_box='') 20# res = model.chat(tokenizer, image_file, ocr_type='ocr', ocr_color='') 21# res = model.chat(tokenizer, image_file, ocr_type='format', ocr_color='') 22 23# multi-crop OCR: 24# res = model.chat_crop(tokenizer, image_file, ocr_type='ocr') 25# res = model.chat_crop(tokenizer, image_file, ocr_type='format') 26 27# render the formatted OCR results: 28# res = model.chat(tokenizer, image_file, ocr_type='format', render=True, save_render_file = './demo.html') 29 30print(res) 31 32

More details about 'ocr_type', 'ocr_box', 'ocr_color', and 'render' can be found at our GitHub. Our training codes are available at our GitHub.

More Multimodal Projects

👏 Welcome to explore more multimodal projects of our team:

Vary | Fox | OneChart

Citation

If you find our work helpful, please consider citing our papers 📝 and liking this project ❤️！

bib

1@article{wei2024general, 2 title={General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model}, 3 author={Wei, Haoran and Liu, Chenglong and Chen, Jinyue and Wang, Jia and Kong, Lingyu and Xu, Yanming and Ge, Zheng and Zhao, Liang and Sun, Jianjian and Peng, Yuang and others}, 4 journal={arXiv preprint arXiv:2409.01704}, 5 year={2024} 6} 7@article{liu2024focus, 8 title={Focus Anywhere for Fine-grained Multi-page Document Understanding}, 9 author={Liu, Chenglong and Wei, Haoran and Chen, Jinyue and Kong, Lingyu and Ge, Zheng and Zhu, Zining and Zhao, Liang and Sun, Jianjian and Han, Chunrui and Zhang, Xiangyu}, 10 journal={arXiv preprint arXiv:2405.14295}, 11 year={2024} 12} 13@article{wei2023vary, 14 title={Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models}, 15 author={Wei, Haoran and Kong, Lingyu and Chen, Jinyue and Zhao, Liang and Ge, Zheng and Yang, Jinrong and Sun, Jianjian and Han, Chunrui and Zhang, Xiangyu}, 16 journal={arXiv preprint arXiv:2312.06109}, 17 year={2023} 18}