SAM 3.1 — ONNX export for ComicApp
ONNX conversion of Meta's
SAM 3.1 for use in
ComicApp.
No retraining, no fine-tuning — this is a
format conversion of the authors' released weights so they can run under
onnxruntime.
Attribution and licence
- Original work: SAM 3.1, Meta Platforms — https://github.com/facebookresearch/sam3
- Licence: the SAM License (19 November 2025). A verbatim copy ships in
this repo as
LICENSE, as §1(b)(i) requires.
- Your use of these files is governed by that Agreement, exactly as the
original weights are. Notable terms: no use for military/warfare, nuclear,
espionage or weapons purposes; no reverse engineering; compliance with export
and sanctions controls.
- Source checkpoint:
sam3.1_multiplex.pt.
What is here
| file | precision | size | what it does |
|---|
sam31_image_encoder.onnx | fp16 | 910 MB | Page encoder. Runs ONCE per page; everything else is cheap against its output. |
sam31_prompt_side.onnx + .data | fp32 | 121 MB | Concept ("find every X") prompting. |
sam31_prompt_box.onnx + .data | fp32 | 134 MB | Box prompting — outline what a person drew a box around. |
The split is deliberate: the encoder is 94% of the bytes and nearly all of the
compute, so a page is encoded once and then prompted repeatedly for almost
nothing. That is what makes an interactive mask editor viable.
Input contract — read this before wiring it up
Normalisation is mean = std = 0.5 (i.e. 2x/255 − 1), from the model's own
processor_config.json. It is NOT ImageNet normalisation. Getting this wrong
degrades results silently rather than failing.
- Encoder input:
image, float16, [1, 3, 1008, 1008], RGB, NCHW.
- Encoder output: four feature levels. The decoder consumes the first
THREE — the model applies
scalp=1 and discards the last.
- Prompt-side outputs:
pred_logits [1,200,1], pred_boxes [1,200,4]
(cxcywh, normalised), pred_masks [1,200,288,288], presence_logit [1,1].
- Thresholding is not in the graph. Score is
sigmoid(logit) * sigmoid(presence); apply your own floor and upsample the
masks to page size. Keeping it outside means the confidence floor is a runtime
setting rather than a property of the file.
Why fp16 for the encoder
Measured, not assumed. On a real comic page, fp16 against fp32:
| fp32 | fp16 |
|---|
| detections kept | 24 | 24 — same set |
| mask IoU on survivors (min / mean) | 0.95060 / 0.99278 | 0.95060 / 0.99321 |
| mask-pixel disagreement | 0.1825% | 0.1803% |
| size | 1819 MB | 910 MB |
Identical decisions, half the download.
Note that at the feature level fp16 looks ~10× worse (relative error 0.11–0.17
vs 0.012–0.016) — but mean|diff| is 0.000089, so that is a handful of outliers
a max-based ratio exaggerates, and nothing downstream notices it. If you are
evaluating a quantisation, compare decisions rather than tensors.
The prompt-side graphs stay fp32: they are 13% of the bytes, so the saving is not
worth an unverified change.
Conversion notes
The stock model does not export as-is. Six changes were needed, all
behaviour-preserving:
perflib/fused.py:addmm_act hardcodes bf16 and has no ONNX symbolic —
replaced with activation(F.linear(...)), which is what it computes.- RoPE uses complex tensors (
torch.polar, view_as_complex); ONNX has no
complex type — rewritten in real arithmetic,
(a+bi)(c+di) = (ac−bd) + (ad+bc)i.
- The 32 complex buffers are converted to real
(…,2) form before tracing;
doing it inside the function puts an aten::view_as_real in the graph.
decoder.py compares stored ints against export-time SymInts —
torch.compiler.is_dynamo_compiling() forced true, which is the honest answer
while tracing.aten::_pin_memory is not implemented by torch.export; pinning is a
host-transfer hint with no effect on values, so it is made identity.concat_padded_sequences short-circuits on zero-length operands. With an
empty geometric prompt the geometry tensors are zero-rowed, and ONNX
shape inference cannot broadcast over a zero-length dim.
Each was verified to reproduce the original path exactly before export.
⚠️ Do not "fix" (6) by seeding a dummy box. A whole-page box with a positive
label is not inert — it instructs the model to segment everything, and the file
will load, run, and quietly answer a different question.
Verification
Both prompt graphs make identical decisions to the PyTorch reference on a
real page — same surviving query set, mask IoU ≥ 0.9991, <0.008% of mask pixels
differing — checked against a control (a different concept) that correctly
disagrees.
