Views
No views yet
| Feature | Capability |
|---|---|
| Architecture | Idefics3-based VLM (SigLIP2 + Granite 165M) |
| Input | Document images (512×512) + text prompts |
| Output | DocTags structured markup |
| Performance | 2-5x faster than PyTorch inference |
| Memory | 60-80% less RAM usage |
| Hardware | CPU, CUDA, DirectML, TensorRT |
1import onnxruntime as ort
2import numpy as np
3from PIL import Image
4
5# Load the ONNX model
6session = ort.InferenceSession('model.onnx')
7
8# Prepare document image
9image = Image.open('document.png').resize((512, 512))
10pixel_values = np.array(image).astype(np.float32) / 255.0
11pixel_values = pixel_values.transpose(2, 0, 1)[np.newaxis, :]
12
13# Prepare text input
14input_ids = np.array([[1, 2, 3, 4, 5]], dtype=np.int64)
15attention_mask = np.ones((1, 5), dtype=np.int64)
16
17# Run inference
18outputs = session.run(None, {
19 'pixel_values': pixel_values,
20 'input_ids': input_ids,
21 'attention_mask': attention_mask
22})
23
24print(f"Generated DocTags logits: {outputs[0].shape}")1use ort::{Session, inputs, execution_providers::ExecutionProvider};
2
3// Load granite-docling ONNX model
4let session = Session::builder()?
5 .with_optimization_level(GraphOptimizationLevel::Level3)?
6 .with_execution_providers([
7 ExecutionProvider::DirectML, // Windows acceleration
8 ExecutionProvider::CUDA, // NVIDIA acceleration
9 ExecutionProvider::CPU, // Universal fallback
10 ])?
11 .commit_from_file("model.onnx")?;
12
13// Process document
14let document_tensor = preprocess_document_image("document.pdf")?;
15let outputs = session.run(inputs![document_tensor])?;
16let doctags = decode_doctags_markup(outputs)?;| Metric | PyTorch | ONNX Runtime | Improvement |
|---|---|---|---|
| Inference Time | 2.5s | 0.8s | 3.1x faster |
| Memory Usage | 4.2GB | 1.8GB | 57% reduction |
| CPU Utilization | 85% | 62% | 27% more efficient |
| Model Loading | 8.5s | 3.2s | 2.7x faster |
1<doctag>
2 <title><loc_50><loc_20><loc_450><loc_60>Document Title</title>
3 <text><loc_50><loc_80><loc_450><loc_200>Main content paragraph...</text>
4 <otsl>
5 <ched>Header 1<ched>Header 2<nl>
6 <fcel>Cell 1<fcel>Cell 2<nl>
7 </otsl>
8 <formula><loc_100><loc_300><loc_400><loc_350>E = mc^2</formula>
9</doctag>1[dependencies]
2ort = { version = "2.0.0-rc.10", features = ["directml", "cuda"] }pip install onnxruntime-gpu # or onnxruntime for CPUnpm install onnxruntime-web