You can deploy this model and start earning money today!

sapkotapraful/
FullyOCR-2

Name: FullyOCR-2
Availability: InStock

safetensors

apache-2.0

Views

No views yet

FullyOCR-2 Model Card

FullyOCR-2 is a specialized Vision-Language Model (VLM) optimized for high-fidelity Optical Character Recognition. Built using the Unsloth framework, it excels at transforming document images into structured Markdown, preserving headers, lists, and layout formatting with high efficiency.

Model Details

Developed by: sapkotapraful
Model Type: Vision-Language Model (VLM)
Base Architecture: FastVisionModel (Unsloth-optimized)
License: Apache 2.0
Finetuning Trigger: <|MD|>

Intended Use

Primary Use Case

FullyOCR-2 is designed for Document-to-Markdown conversion. It is ideal for:

Extracting text from complex layouts (multi-column documents).
Converting handwritten or typed notes into digital formats.
Preserving structural elements like bold text, headers, and bullet points.

Technical Specifications

Optimization

The model utilizes Unsloth optimizations to achieve:

4-bit Quantization: Native support for bitsandbytes to run on GPUs with limited VRAM (e.g., 8GB - 12GB).
Inference Speed: Significantly reduced latency during token generation compared to standard Transformers.
Memory Efficiency: Efficient KV caching for long document processing.

Training Details

Prompt Template: Chat-based vision template.
Instruction Token: The model specifically looks for the <|MD|> token to trigger the OCR extraction flow.

Installation

Ensure you have the latest versions of the Unsloth ecosystem installed:

bash

pip install unsloth torch pillow

Quick Start (Inference)

python

from unsloth import FastVisionModel
import torch
from PIL import Image

# 1. Load model + tokenizer
model, tokenizer = FastVisionModel.from_pretrained(
    "sapkotapraful/FullyOCR-2",
    load_in_4bit=True,
)

# 2. Setup Device
image = Image.open('document.jpg')
model.eval()
device = "cuda" if torch.cuda.is_available() else "cpu"
if device == "cuda":
    model = model.to(device)

# 3. Prepare Prompt
instruction = "<|MD|>"
messages = [
    {"role": "user", "content": [
        {"type": "image"},
        {"type": "text", "text": instruction}
    ]}
]

input_text = tokenizer.apply_chat_template(messages, add_generation_prompt=True)

# 4. Generate
inputs = tokenizer(
    image,
    input_text,
    add_special_tokens=False,
    return_tensors="pt",
).to(device)

with torch.no_grad(), torch.amp.autocast(device_type="cuda", enabled=(device=="cuda")):
    output_ids = model.generate(
        **inputs,
        max_new_tokens=1024,
        use_cache=True,
        num_beams=1,
        do_sample=False,
        pad_token_id=tokenizer.pad_token_id,
    )

# 5. Extract Markdown
decoded = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0]
extracted = decoded.split(instruction)[-1].strip()
print(extracted)

Performance & Limitations

Strengths

Format Integrity: Specifically tuned to output valid Markdown.
Resource Friendly: Can be deployed on consumer hardware using 4-bit weights.

Limitations

Resolution Sensitivity: Very small text on low-resolution scans may result in "hallucinated" characters.
Context Window: Designed for single-page extraction; very long documents should be processed page-by-page.

Ethical Considerations

When using FullyOCR-2, ensure compliance with data privacy laws (like GDPR) when processing documents containing sensitive Personal Identifiable Information (PII).