Rust implementation of the DeepSeek-OCR inference stack with a fast CLI and an OpenAI-compatible HTTP server. The workspace packages multiple OCR backends, prompt tooling, and a serving layer so you can build document understanding pipelines that run locally on CPU, Apple Metal, or (alpha) NVIDIA CUDA GPUs.
Want ready-made binaries? Latest macOS (Metal-enabled) and Windows bundles live in the build-binaries workflow artifacts. Grab them from the newest green run.
Apple Silicon + Metal (FP16), high-VRAM NVIDIA GPUs, 32GB+ RAM desktops
Highest accuracy, SAM+CLIP global/local context, MoE DeepSeek‑V2 decoder (3B params, ~570M active per token). Use when latency is secondary to quality.
PaddleOCR‑VL
≈4.7GB FP16 weights, ≈9GB RAM/VRAM with cache & activations
16GB laptops, CPU-only boxes, mid-range GPUs
Dense 0.9B Ernie decoder with SigLIP vision tower. Faster startup, lower memory, great for batch jobs or lightweight deployments.
DotsOCR
≈9GB FP16 weights, but expect 30–50GB RAM/VRAM for high-res docs due to huge vision tokens
Apple Silicon + Metal BF16, ≥24GB CUDA cards, or 64GB RAM CPU workstations
Unified VLM (DotsVision + Qwen2) that nails layout, reading order, grounding, and multilingual math if you can tolerate the latency and memory bill.
*Measured from the default FP16 safetensors. Runtime footprint varies with sequence length.
Guidance:
Need maximum fidelity, multi-region reasoning, or already have 16–24GB VRAM? Use DeepSeek‑OCR. The hybrid SAM+CLIP tower plus DeepSeek‑V2 MoE decoder handles complex layouts best, but expect higher memory/latency.
Deploying to CPU-only nodes, 16GB laptops, or latency-sensitive services? Choose PaddleOCR‑VL. Its dense Ernie decoder (18 layers, hidden 1024) activates fewer parameters per token and keeps memory under 10GB while staying close in quality on most docs.
Chasing reading-order accuracy, layout grounding, or multi-page multilingual PDFs on roomy hardware? Pick DotsOCR with BF16 on Metal/CUDA. Prefill runs around 40–50 tok/s on M-series GPUs but can fall to ~12 tok/s on CPU because of the heavy vision tower.
Why Rust? 💡
The original DeepSeek-OCR ships as a Python + Transformers stack—powerful, but hefty to deploy and awkward to embed. Rewriting the pipeline in Rust gives us:
Smaller deployable artifacts with zero Python runtime or conda baggage.
Memory-safe, thread-friendly infrastructure that blends into native Rust backends.
Unified tooling (CLI + server) running on Candle + Rocket without the Python GIL overhead.
Drop-in compatibility with OpenAI-style clients while tuned for single-turn OCR prompts.
Technical Stack ⚙️
Candle for tensor compute, with Metal and CUDA backends and FlashAttention support.
Rocket + async streaming for OpenAI-compatible /v1/responses and /v1/chat/completions.
tokenizers (upstream DeepSeek release) wrapped by crates/assets for deterministic caching via Hugging Face and ModelScope mirrors.
Pure Rust vision/prompt pipeline shared by CLI and server to avoid duplicated logic.
Advantages over the Python Release 🥷
Faster cold-start on Apple Silicon, lower RSS, and native binary distribution.
Deterministic dual-source (Hugging Face + ModelScope) asset download + verification built into the workspace.
Automatic single-turn chat compaction so OCR outputs stay stable even when clients send history.
Ready-to-use OpenAI compatibility for tools like Open WebUI without adapters.
Highlights ✨
One repo, two entrypoints – a batteries-included CLI for batch jobs and a Rocket-based server that speaks /v1/responses and /v1/chat/completions.
Works out of the box – pulls model weights, configs, and tokenizer from whichever of Hugging Face or ModelScope responds fastest on first run.
Optimised for Apple Silicon – optional Metal backend with FP16 execution for real-time OCR on laptops.
CUDA (alpha) – experimental support via --features cuda + --device cuda --dtype f16; expect rough edges while we finish kernel coverage.
Intel MKL (preview) – faster BLAS on x86 via --features mkl (install Intel oneMKL beforehand).
OpenAI client compatibility – drop-in replacement for popular SDKs; the server automatically collapses chat history to the latest user turn for OCR-friendly prompts.
Model Matrix 📦
The workspace exposes three base model IDs plus DSQ-quantized variants for DeepSeek‑OCR, PaddleOCR‑VL, and DotsOCR:
Model ID
Base Model
Precision
Suggested Use Case
deepseek-ocr
deepseek-ocr
FP16 (select via --dtype)
Full-fidelity DeepSeek‑OCR stack with SAM+CLIP + MoE decoder; use when you prioritise quality on capable Metal/CUDA/CPU hosts.
deepseek-ocr-q4k
deepseek-ocr
Q4_K
Tight VRAM, local deployments, and batch jobs that still want DeepSeek’s SAM+CLIP pipeline.
deepseek-ocr-q6k
deepseek-ocr
Q6_K
Day‑to‑day balance of quality and size on mid‑range GPUs.
deepseek-ocr-q8k
deepseek-ocr
Q8_0
Stay close to full‑precision quality with manageable memory savings.
paddleocr-vl
paddleocr-vl
FP16 (select via --dtype)
Default choice for lighter hardware; 0.9B Ernie + SigLIP tower with strong doc/table OCR and low latency.
paddleocr-vl-q4k
paddleocr-vl
Q4_K
Heavily compressed doc/table deployments with aggressive memory budgets.
paddleocr-vl-q6k
paddleocr-vl
Q6_K
Common engineering setups; blends accuracy and footprint.
paddleocr-vl-q8k
paddleocr-vl
Q8_0
Accuracy‑leaning deployments that still want a smaller footprint than FP16.
dots-ocr
dots-ocr
FP16 / BF16 (via --dtype)
DotsVision + Qwen2 VLM for high‑precision layout, reading order, grounding, and multilingual docs; expect high memory (30–50GB on large pages).
dots-ocr-q4k
dots-ocr
Q4_K
Sidecar DSQ snapshot over the DotsOCR baseline; reduces weight memory/compute while keeping the heavy vision token profile unchanged.
dots-ocr-q6k
dots-ocr
Q6_K
Recommended balance of size and quality when you already accept DotsOCR’s memory footprint but want cheaper weights.
dots-ocr-q8k
dots-ocr
Q8_0
Accuracy‑leaning DotsOCR deployment that stays close to FP16/BF16 quality with modest memory savings.
Quick Start 🏁
Prerequisites
Rust 1.78+ (edition 2024 support)
Git
Optional: Apple Silicon running macOS 13+ for Metal acceleration
Optional: CUDA 12.2+ toolkit + driver for experimental NVIDIA GPU acceleration on Linux/Windows
Optional: Intel oneAPI MKL for preview x86 acceleration (see below)
(Recommended) Hugging Face account with HF_TOKEN when pulling from the deepseek-ai/DeepSeek-OCR repo (ModelScope is used automatically when it’s faster/reachable).
The first invocation of the CLI or server downloads the config, tokenizer, and model-00001-of-000001.safetensors (~6.3GB) into DeepSeek-OCR/. To prefetch manually:
cargo run -p deepseek-ocr-cli --release -- --help # dev profile is extremely slow; always prefer --release
Always include --release when running from source; debug builds on this model are extremely slow.
Set HF_HOME/HF_TOKEN if you store Hugging Face caches elsewhere (ModelScope downloads land alongside the same asset tree). The full model package is ~6.3GB on disk and typically requires ~13GB of RAM headroom during inference (model + activations).
Configuration & Overrides 🗂️
The CLI and server share the same configuration. On first launch we create a config.toml populated with defaults; later runs reuse it so both entrypoints stay in sync.
Override the location with --config /path/to/config.toml (available on both CLI and server). Missing files are created automatically.
Each [models.entries."<id>"] record can point to custom config, tokenizer, or weights files. When omitted we fall back to the cache directory above and download/update assets as required.
Runtime values resolve in this order: command-line flags → values stored in config.toml → built-in defaults. The HTTP API adds a final layer where request payload fields (for example max_tokens) override everything else for that call.
The generated file starts with the defaults below; adjust them to persistently change behaviour:
[models] picks the active model and lets you add more entries (each entry can point to its own config/tokenizer/weights).
[inference] controls notebook-friendly defaults shared by the CLI and server (device, template, vision sizing, decoding budget, cache usage).
[server] sets the network binding and the model identifier reported by /v1/models.
See crates/cli/README.md and crates/server/README.md for concise override tables.
Benchmark Snapshot 📊
Single-request Rust CLI (Accelerate backend on macOS) compared with the reference Python pipeline on the same prompt and image:
Stage
ref total (ms)
ref avg (ms)
python total
python/ref
Decode – Overall (decode.generate)
30077.840
30077.840
56554.873
1.88x
Decode – Token Loop (decode.iterative)
26930.216
26930.216
39227.974
1.46x
Decode – Prompt Prefill (decode.prefill)
3147.337
3147.337
5759.684
1.83x
Prompt – Build Tokens (prompt.build_tokens)
0.466
0.466
45.434
97.42x
Prompt – Render Template (prompt.render)
0.005
0.005
0.019
3.52x
Vision – Embed Images (vision.compute_embeddings)
6391.435
6391.435
3953.459
0.62x
Vision – Prepare Inputs (vision.prepare_inputs)
62.524
62.524
45.438
0.73x
Command-Line Interface 🖥️
Build and run directly from the workspace:
bash
1cargo run -p deepseek-ocr-cli --release -- \2 --prompt "<image>\n<|grounding|>Convert this receipt to markdown."\3 --image baselines/sample/images/test.png \4 --device cpu --max-new-tokens 512
Tip: --release is required for reasonable throughput; debug builds can be 10x slower.
macOS tip: append --features metal to the cargo run/cargo build commands to compile with Accelerate + Metal backends.
CUDA tip (Linux/Windows): append --features cuda and run with --device cuda --dtype f16 to target NVIDIA GPUs—feature is still alpha, so be ready for quirks.
Intel MKL preview: install Intel oneMKL, then build with --features mkl for faster CPU matmuls on x86.
The CLI (and server) will download the matching config/tokenizer/weights from the appropriate repository (deepseek-ai/DeepSeek-OCR, PaddlePaddle/PaddleOCR-VL, or dots-ocr) into your cache on first use. You can still override paths with --model-config, --tokenizer, or --weights if you maintain local fine-tunes.
HTTP Server ☁️
Launch an OpenAI-compatible endpoint:
bash
1cargo run -p deepseek-ocr-server --release -- \2 --host 0.0.0.0 --port 8000\3 --device cpu --max-new-tokens 512
Keep --release on the server as well; the debug profile is far too slow for inference workloads.
macOS tip: add --features metal to the cargo run -p deepseek-ocr-server command when you want the server binary to link against Accelerate + Metal (and pair it with --device metal at runtime).
CUDA tip: add --features cuda and start the server with --device cuda --dtype f16 to offload inference to NVIDIA GPUs (alpha-quality support).
Intel MKL preview: install Intel oneMKL before building with --features mkl to accelerate CPU workloads on x86.
Notes:
Use data: URLs or remote http(s) links; local paths are rejected.
The server collapses multi-turn chat inputs to the latest user message to keep prompts OCR-friendly.
Works out of the box with tools such as Open WebUI or any OpenAI-compatible client—just point the base URL to your server (http://localhost:8000/v1) and select either the deepseek-ocr or paddleocr-vl model ID exposed in /v1/models.
Adjust the request body limit with Rocket config if you routinely send large images.
Open WebUI connected to deepseek-ocr.rs
GPU Acceleration ⚡
Metal (macOS 13+ Apple Silicon) – pass --device metal --dtype f16 and build binaries with --features metal so Candle links against Accelerate + Metal.
CUDA (alpha, NVIDIA GPUs) – install CUDA 12.2+ toolkits, build with --features cuda, and launch the CLI/server with --device cuda --dtype f16; still experimental.
Intel MKL (preview) – install Intel oneMKL and build with --features mkl to speed up CPU workloads on x86.
For either backend, prefer release builds (e.g. cargo build --release -p deepseek-ocr-cli --features metal|cuda) to maximise throughput.
Combine GPU runs with --max-new-tokens and crop tuning flags to balance latency vs. quality.
Repository Layout 🗂️
crates/core – shared inference pipeline, model loaders, conversation templates.
Where do assets come from? – downloads automatically pick between Hugging Face and ModelScope based on latency; the CLI prints the chosen source for each file.
Slow first response – model load and GPU warm-up (Metal/CUDA alpha) happen on the initial request; later runs are faster.
Large image rejection – increase Rocket JSON limits in crates/server/src/main.rs or downscale the input.
Roadmap 🗺️
✅ Apple Metal backend with FP16 support and CLI/server parity on macOS.
✅ NVIDIA CUDA backend (alpha) – build with --features cuda, run with --device cuda --dtype f16 for Linux/Windows GPUs; polishing in progress.
🔄 Parity polish – finish projector normalisation + crop tiling alignment; extend intermediate-tensor diff suite beyond the current sample baseline.
🔄 Grounding & streaming – port the Python post-processing helpers (box extraction, markdown polish) and refine SSE streaming ergonomics.
🔄 Cross-platform acceleration – continue tuning CUDA kernels, add automatic device detection across CPU/Metal/CUDA, and publish opt-in GPU benchmarks.
🔄 Packaging & Ops – ship binary releases with deterministic asset checksums, richer logging/metrics, and Helm/docker references for server deploys.
🔜 Structured outputs – optional JSON schema tools for downstream automation once parity gaps close.
License 📄
This repository inherits the licenses of its dependencies and the upstream DeepSeek-OCR model. Refer to DeepSeek-OCR/LICENSE for model terms and apply the same restrictions to downstream use.
