Views
No views yet
document-classification-v2. It ships as
two self-contained ONNX graphs — an image tower and a text tower — that you run with onnxruntime.
embed_dim: 1024; classification p = sigmoid(scale·cos + bias) (calibration in
modules/omni-image/config.json).| Benchmark | v1 (open) | v2 (commercial) | best cloud VLM |
|---|---|---|---|
| DocLayNet | 0.75 | 0.97 | 0.83 |
| Forms | 0.80 | 1.00 | 1.00 |
| Tobacco | 0.61 | 0.74 | 0.85 |
| OOD (unseen types) | 0.86 | 0.95 | — |
| OOV (synonym wording) | 0.73 | 0.83 | — |
v2
and the large cloud VLMs on accuracy, but it's Apache-2.0 and downloadable. Like all embedding models it
trails VLMs most on Tobacco (a read-the-header task). ~5.7 pages/s on an A40 (fused image+text).1import numpy as np, onnxruntime as ort, json
2from transformers import AutoImageProcessor, AutoTokenizer
3from huggingface_hub import hf_hub_download
4from PIL import Image
5
6R = "nutrientdocs/document-classification-v1"
7img_sess = ort.InferenceSession(hf_hub_download(R, "modules/omni-image/image_model.onnx")) # SigLIP image tower
8txt_sess = ort.InferenceSession(hf_hub_download(R, "modules/omni-image/text_model.onnx")) # Qwen text tower
9cal = json.load(open(hf_hub_download(R, "modules/omni-image/config.json")))["calibration"]
10proc = AutoImageProcessor.from_pretrained(R, subfolder="modules/omni-image") # SigLIP image processor
11tok = AutoTokenizer.from_pretrained(R, subfolder="modules/omni-image") # Qwen tokenizer
12
13labels = ["invoice", "letter", "memo", "form", "scientific article", "resume"]
14calib = lambda cos: 1 / (1 + np.exp(-(cal["scale"] * cos + cal["bias"])))
15
16def embed_text(texts, maxlen):
17 e = tok(texts, padding=True, truncation=True, max_length=maxlen, return_tensors="np")
18 return txt_sess.run(["text_emb"], {"input_ids": e["input_ids"].astype(np.int64),
19 "attention_mask": e["attention_mask"].astype(np.int64)})[0] # [.,1024] L2
20
21lab = embed_text(labels, 64) # label embeds, once
22
23# --- image branch: page image vs labels (image ONNX has batch=1; loop+pool for multi-page) ---
24pix = proc(images=[Image.open("doc.png").convert("RGB")], return_tensors="np")["pixel_values"].astype(np.float16)
25ie = img_sess.run(["image_emb"], {"pixel_values": pix})[0] # [1,1024] L2
26image_probs = calib((ie @ lab.T)[0]) # [N]
27
28# --- text branch: the page's OCR text vs labels (up to ~2048 tokens) ---
29doc_text = open("doc.txt").read()
30text_probs = calib((embed_text([doc_text], 2048) @ lab.T)[0]) # [N]
31
32# --- reliability fusion: weight each branch by how DECISIVE it is (top1-top2 margin) ---
33margin = lambda p: float(np.partition(p, -2)[-1] - np.partition(p, -2)[-2])
34wi, wt = margin(image_probs), margin(text_probs); s = wi + wt + 1e-9
35fused = (wi / s) * image_probs + (wt / s) * text_probs
36print(dict(zip(labels, fused.round(3).tolist())))modules/omni-image/{image_model.onnx, text_model.onnx} — the image + text towers (fp16, onnxruntime).modules/omni-image/{config.json, preprocessor_config.json, tokenizer.json} — calibration + the
preprocessor and tokenizer needed to run them. That's it — nothing else required.document-classification-v2.