shrew-ocr-preview converts one document page image per request into a single JSON object containing document metadata, a summary, self-contained semantic chunks (sized for RAG ingestion, not raw OCR lines), and figures/tables with bounding boxes and HTML. A text modality accepts HTML/markdown/plain-text input and produces the same output schema.

GPTQ-8bit quantization of shrew-ocr-preview (fine-tuned from ibm-granite/granite-vision-4.1-4b). Only the language model is quantized (INT8, sym, group_size 32, desc_act=False); the vision tower and projectors remain bf16 — which matches the fine-tune exactly, since all adaptation lives in the language blocks. Measured cost of quantization: +0.25% domain perplexity against the bf16 model through an identical evaluation path, with unchanged output-length distribution. Measured benefit: ~1.8× serving throughput and a 4.9 GB footprint (2.3 GiB per GPU at TP=4) that fits comfortably on a single 16 GB card with KV headroom.

One request = one page. The model returns exactly one JSON object, five keys always present:

{
  "metadata":        {"title", "authors" (list), "organization", "year", "doc_type"} — null where unknown,
  "summary":         str | null,
  "semantic_chunks": [{"chunk_id", "title", "content",
                       "section_type" ∈ {abstract, introduction, methodology, results,
                                          discussion, conclusion, technical_content, appendix}}],
  "figures":         [{"bbox": [x0,y0,x1,y1] | null, "caption", "description"}],
  "tables":          [{"html": "<table>…", "bbox": [...] | null, "caption", "description"}]
}

Bounding boxes are xyxy on a 0–1000 normalized grid over the page image. In text modality, bboxes are null.

Recommended path: shrew-server (MIT, pin tag v0.2.1), the reference server for this model. It implements the model's entire input contract server-side — glyph-routed bucket preprocessing, the structured_extraction request shape, tuned decoding with a schema-enforced retry tier, the streaming repetition guard, schema/coercion gates, and multi-page assembly. POST a PDF, receive structured JSON. It does not serve the model itself; point it at an OpenAI-compatible endpoint (vLLM, below — note the GPTQ-specific flags in the serving section):

Full instructions, including a Docker Compose quickstart, are in the repo README under "Using with shrew-ocr-preview (recommended)".

For direct integration without shrew-server, the requirements below define the input contract. Deviating from any of these degrades output quality:

1. System prompt. Set the system prompt to the literal string structured_extraction. Do not send instruction text; the model was trained on this fixed prompt only.

2. Decoding. Set temperature to 0 and max_tokens to 20000. presence_penalty 0.3–0.6 is measured fidelity-neutral; 0.3 is the reference server's first-pass default. Do not set top_p or any other penalty parameter (measured basis under the repetition guard below). Serve with context length ≥ 32768; dense pages need room for both the image tokens and a long completion.

3. Input resolution ("buckets"). Resize each page image to one of three portrait tile grids, selected by the page's measured glyph height (target ~10 px after resize). Training used exactly this routing. Reference implementation:

The checkpoint's config.json and preprocessor_config.json carry the matching image_grid_pinpoints. Do not remove or modify them; the tile packing must match training.

Use --dtype half, not bfloat16: the fast GPTQ kernel path is fp16-only, and fp16 was measured output-neutral for this model. vLLM detects the mixed-precision layout from the checkpoint automatically — the bf16 vision tower loads unquantized with no extra configuration. Requires a vLLM version whose GPTQ support covers multimodal models with partially-quantized checkpoints (v0.27+).

Scaling note: the model is small (2–5 GB weights). For batch serving on multi-GPU hosts, data-parallel replicas (--data-parallel-size N) outperform tensor parallelism substantially (+54% measured on a 4-GPU node) — prefer DP unless a single GPU cannot hold the weights. On memory-constrained GPUs keep --max-num-batched-tokens at 2048 or below: the vision encoder batches image tiles, and large prefill budgets can OOM the tower on high-tile pages.

Request shape (OpenAI-compatible):

Recommended: streaming repetition guard. On hard pages the failure mode is degenerate repetition, not plausible-but-wrong output. Stream the completion, compute len(window)/len(zlib.compress(window)) over a trailing ~2,000-character window every ~800 characters, and abort after 2 consecutive windows above ~15. Clean pages measure ~2; looping output exceeds 25.

Penalty parameters (measured: 900 first-pass runs, 150 stratified pages × 6 decode arms on the production stack): presence_penalty 0.3 at first pass raised the first-pass success rate (valid JSON passing all schema and degeneration gates, no retry) 0.880→0.887 with extraction precision flat (0.949→0.949 median 10-gram precision vs ground truth), and is the reference server's first-pass default; 0.3–0.6 both measured fidelity-neutral on healthy pages. In a separate rescue experiment, a penalized retry (presence_penalty 0.6) recovered 12/12 sampled loop-failed Latin-script broadsheet pages with fidelity flat; dense non-Latin broadsheets did not recover at any penalty (see Limitations — that class is a training gap, not a decoding one). Do not use frequency or repetition penalties or no_repeat_ngram_size: ngram blocking suppresses JSON tokens that must repeat ("bbox": [); frequency penalties accumulate with each repeated occurrence, and repetition penalties apply to every previously-seen token — both degrade required schema tokens in long structured outputs. Do not use grammar-constrained (schema-enforced) decoding at first pass: it degrades table transcription severely (table one-shot 0.975→0.225), primarily by overrunning the token budget mid-table (grammar-constrained decoding transcribes exhaustively), with degraded table fidelity (0.483→0.280) on the pages that do finish; the reference server applies enforcement only on retry. Two caveats: the Results tables were measured with penalty-free greedy decoding (the presence-penalty recommendation comes from the separate decode matrix), and retry rescue was validated on Latin-script pages only — for pages whose text the vision tower cannot resolve, a penalized retry can convert a detectable loop into plausible hallucination (observed: a penalized retry on an unsupported-script page produced fluent output with zero n-gram overlap with the page), so validate retry output with the same window check and schema gates and mark it lower-confidence downstream.

Which borderline pages loop is sensitive to serving numerics (tensor- vs data-parallel layout, torch.compile, kernel changes); in our measurements loop-page identities changed across serve configs while the overall loop rate stayed constant. Compare deployments by loop rate over a fixed page set, not by which pages failed.

The same model accepts born-digital text — HTML (emails, filings), markdown, source code, plain text — and returns the same 5-key output schema with figures[].bbox and tables[].bbox null. Do not route OCR output of scanned pages here; scanned pages go through the image modality.

Request shape: same envelope, with the raw content as a single text part in place of the image. Send the content as-is — no wrapper, no instructions, no cleaning:

Input sizing: send 2,000–9,000 characters (~500–2,500 tokens) per request; treat 13,000 characters as the ceiling (training inputs never exceeded it). Split longer documents at structural boundaries (headings, sections) and send one request per section. Each request in the recommended range yields roughly 2–6 semantic chunks (median chunk ~820 characters).

Measured on the OHR-Bench corpus (ICCV 2025): 1,261 PDFs / 8,561 pages across 7 domains (textbook, law, finance, newspaper, manual, academic, administration). Every page runs through our full production path (rasterize → bucket routing → model → schema gates → assembly); each system's structured output is chunked under the same budget, embedded with nvidia/llama-nemotron-embed-vl-1b-v2 ("nemotron-vl"), and scored as retrieval hit@5 / MRR@10 over OHR-Bench's ~8.5k human-verified Q&A pairs. These are our own retrieval-harness measurements, not official OHR-Bench generation (LCS/F1) numbers. gt is retrieval over OHR-Bench's human ground-truth structured data; MinerU and PaddleOCR outputs were run through the identical chunking and indexing.

Text retrieval, hit@5 / MRR@10 by evidence type (higher is better, best per row in bold):

† Known failure. OHR-Bench draws reading-order queries almost entirely from dense broadsheet newspaper scans, which fall in this model's repetition-loop failure class (see Limitations); with those pages unparsed, the attainable ceiling is ~0.14. The bf16/INT8 gap on this row is a harness-gating artifact, not quantization: the bf16 number is inflated by hallucinated filler that the final output gate rejects. Treat broadsheet reading order as unsupported in this release.

Figure/table localization vs our own frozen human-annotated gold subset — 551 corpus pages / 1,100 boxes, not an OHR-Bench artifact (greedy match at IoU ≥ 0.5):

Reliability: 84.35% of pages produce valid schema-complete JSON on the first pass (bf16: 84.7%); 97.2% after mechanical schema coercion. Hard failures are ~3% of pages, concentrated in the dense-broadsheet loop class, and terminate as repetition-guard aborts rather than silent bad output.

Quantization cost and benefit: this INT8 variant matches or exceeds bf16 on 5 of 7 retrieval types, is within noise on localization, and measures +0.25% domain perplexity. End-to-end wall clock through the production path measured ~2.3× vs serving base + LoRA adapter in bf16; controlled merged-vs-merged raw serving isolates quantization at ~1.8×, with the remainder from removing the adapter path.

This is a preview: weights update in place under these names as the model improves. Each weight push's commit message records the training and calibration generation — pin a commit (revision=) for reproducibility.

Base model: ibm-granite/granite-vision-4.1-4b (Apache 2.0). The vision tower is unchanged from the base; all fine-tuning lives in the language model.