Sona Forge — SD 1.5 ControlNet Canny (ONNX FP16)
ONNX FP16 export of the SD 1.5 ControlNet Canny encoder. Used by the Sona Forge Android app for pose / composition stability on portrait avatars (Phase 7). Pair with
sona-forge/sd15-ipadapter-fp16 (residual-accepting variant, revision ≥ 1.1.0) and
sona-forge/clip-vit-h-14-image-fp16.
ONNX shape
| Input | Shape | dtype | Notes |
|---|
sample | [batch, 4, 64, 64] | FP16 | latent state at step t |
timestep | [batch] | FP16 | scheduler timestep |
encoder_hidden_states | [batch, 77, 768] | FP16 | text embeds |
canny_image | [batch, 3, 512, 512] | FP16 | Canny edge map in [0, 1] (white-on-black, replicated 3×). For CFG, pass zeros for the uncond branch. ControlNet's residual contribution is linear in canny_image, so on-device callers can pre-multiply this by a controlNetScale ∈ [0, 1] factor instead of carrying a scalar input. |
| Output | Shape | dtype | Notes |
|---|
down_residual_0..11 | 12 tensors | FP16 | down-block residuals fed into the SD 1.5 UNet's skip connections |
mid_residual | [batch, 1280, 8, 8] | FP16 | mid-block residual |
Down-block residual canonical shapes (per SD 1.5 UNet):
[batch, 320, 64, 64] ×3, [batch, 320, 32, 32], [batch, 640, 32, 32] ×2, [batch, 640, 16, 16], [batch, 1280, 16, 16] ×2, [batch, 1280, 8, 8] ×3.
How it was made
Pinned conversion environment:
| Package | Version |
|---|
| diffusers | 0.27.2 |
| transformers | 4.40.0 |
| torch | 2.3.0 |
| onnx | 1.16.0 |
| onnxruntime | 1.18.0 |
| numpy | <2 (ABI compat) |
Conversion sequence:
- Load
lllyasviel/control_v11p_sd15_canny ControlNet model at FP16.
- Wrap to expose 13 named outputs (
down_residual_0..11, mid_residual).
torch.onnx.export at opset 17 with FP16 dummy inputs at canonical SD 1.5 shapes.
Re-running the conversion from the same pinned environment produces byte-identical output (same sha256). Conversion artefacts include a spike report with full validation metrics and arithmetic round-trip checks against the PyTorch reference.
Files
| File | Size | sha256 |
|---|
model.onnx | 723,055,316 B (689.6 MB) | 399358929322eb5bb2f0e141e23486397a29ec871e4efa625c0f2ba4d418c698 |
No external-data sidecar — graph + weights fit under the 2 GB protobuf single-file limit.
Licence
Memory footprint
ORT CPU EP promotes FP16 to FP32 at session load (~1.4 GB resident). On Android, NNAPI / XNNPack execute FP16 natively and the on-device working set is closer to the FP16 disk size + activation buffers. Sona Forge gates this pack to Tier C devices (≥ 11 GB total RAM) per RamGate.requiresControlNetTier.
Usage
1import onnxruntime as ort
2import numpy as np
3
4session = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])
5
6# CFG batch=2.
7sample = np.random.randn(2, 4, 64, 64).astype(np.float16)
8timestep = np.array([999.0, 999.0], dtype=np.float16)
9encoder_hidden_states = np.random.randn(2, 77, 768).astype(np.float16)
10
11# canny_image: zeros for the uncond branch, scaled edges for the cond branch.
12canny_one = np.random.rand(1, 3, 512, 512).astype(np.float16) # white-on-black, 3-channel replicated, [0..1]
13controlnet_scale = 0.7
14canny_image = np.concatenate([
15 np.zeros_like(canny_one),
16 canny_one * controlnet_scale,
17], axis=0)
18
19residuals = session.run(None, {
20 "sample": sample,
21 "timestep": timestep,
22 "encoder_hidden_states": encoder_hidden_states,
23 "canny_image": canny_image,
24})
25# 12 down-block residuals + 1 mid-block residual, fed into the residual-accepting IP-Adapter UNet.
Provenance
- Original ControlNet weights:
lllyasviel/control_v11p_sd15_canny.
- Companion SD 1.5 UNet (residual-accepting variant):
sona-forge/sd15-ipadapter-fp16 (revision ≥ 1.1.0 supports the 13-residual signature).
