This document describes the NVDS (Neural Video Depth Stabilizer) ONNX export used
by PTMediaServer as an optional temporal depth stabilizer in the offline
2D-to-3D/VR pipeline.
The exporter is examples/export_nvds_onnx.py. It converts
the NVDS PyTorch checkpoint into fixed-resolution ONNX Runtime graphs so
inference does not need PyTorch at runtime. NVDS smooths the per-frame depth that
DA3 produces, removing the temporal "depth swimming" that causes discomfort when
the stereo output is viewed.
Sharper depth boundaries; reachable via --nvds-res 672x384.
Export Modes
The exporter has two modes.
Monolithic (--width/--height, no --split)
A single graph that takes the whole 4-frame window and runs the backbone on all
4 frames internally. Output name NVDS_Stabilizer_{w}x{h}.onnx.
Split (--split, recommended)
Two graphs that let the runtime run the heavy backbone once per frame and cache
the last 4 results, instead of recomputing 3/4 of it for every sliding window:
File
Role
NVDS_Backbone_{w}x{h}.onnx
Single RGBD frame -> 4 multi-scale feature maps. Run once per frame and cached.
NVDS_Head_{w}x{h}.onnx
The 4 window frames' features (stacked) + last-frame RGB -> stabilized depth.
Both modes are numerically identical (the backbone has no cross-frame ops, so its
features are batch-independent). Measured max_abs_diff between split and
monolith is about 3e-05.
Note on performance: the focal cross-attention head, not the backbone, dominates
NVDS cost, so the split's main practical benefit is enabling the lower-resolution
tier (the head cost scales with input pixel count). See
summary/summary_20260620_NVDS_INTEGRATION_EXTERNAL_REVIEW_CN.md for the
measurements.
Fixed Resolution
Each ONNX file is exported at a fixed input resolution.
Rules:
Height and width are fixed in each ONNX file.
--width and --height must each be a multiple of 32.
The intended tiers are 512 x 288 and 672 x 384 (16:9). Re-export to add
another size.
1input names: feat0..feat3 (each [4, C, h, w] = the 4 window frames stacked)
2 last_rgb ([1, 3, height, width], the current frame's RGB)
34output name : stabilized_depth
5output shape: [1, 1, height, width]
The output is a stabilized normalized near/disparity map (larger = nearer),
already temporally smoothed. Unlike raw DA3 depth, it must NOT be reciprocated or
percentile-normalized again at render time; PTMediaServer feeds it through the
dedicated render_near(...) path.
Preprocessing
For every frame, the runtime wrapper builds a 4-channel RGBD frame at the export
resolution:
Load the frame as RGB and resize to [width, height].
Convert to float32 in 0.0 .. 1.0 and apply ImageNet normalization with mean
[0.485, 0.456, 0.406] and std [0.229, 0.224, 0.225]; transpose to CHW.
Convert the DA3 depth to a normalized near/disparity map (reciprocal +
percentile normalization) and resize it to [width, height].
Concatenate RGB (3 ch) + near (1 ch) into a [4, height, width] frame.
For the monolithic graph the wrapper keeps the last 4 such frames, padding the
first frames by repetition, and stacks them into [1, 4, 4, H, W]. For the split
graphs it runs the backbone on each frame, caches the last 4 feature tuples,
stacks them on the batch axis, and passes the current frame's RGB as last_rgb.
PTMediaServer's offline.nvds_stabilizer.NvdsDepthStabilizer performs all of
this and auto-selects the split graphs when they are present.
Runtime Dependencies
CPU:
pip install onnxruntime numpy opencv-python
GPU:
pip install onnxruntime-gpu numpy opencv-python
TensorRT is not usable for NVDS (see Notes); the runtime uses the CUDA execution
provider with a bounded GPU memory arena.
Quick Inference Example
This example runs the split graphs over a 4-frame window. frame_rgbd_t is a
preprocessed [1, 4, H, W] float32 frame as described in Preprocessing.
python
1import numpy as np
2import onnxruntime as ort
34W, H =512,2885providers =["CUDAExecutionProvider","CPUExecutionProvider"]6backbone = ort.InferenceSession("models/NVDS/NVDS_Backbone_512x288.onnx", providers=providers)7head = ort.InferenceSession("models/NVDS/NVDS_Head_512x288.onnx", providers=providers)89bb_out =[o.name for o in backbone.get_outputs()]# feat0..feat310window =[]# last 4 feature tuples11for frame_rgbd_t in stream:# each [1, 4, H, W] float3212 feats = backbone.run(bb_out,{"frame_rgbd": frame_rgbd_t})13 window.append(feats)14 window = window[-4:]15 pad =[window[0]]*(4-len(window))+ window # causal padding16 feeds ={f"feat{s}": np.concatenate([pad[t][s]for t inrange(4)], axis=0)17for s inrange(4)}18 feeds["last_rgb"]= np.ascontiguousarray(frame_rgbd_t[:,0:3])19 stable_near = head.run(["stabilized_depth"], feeds)[0][0,0]# [H, W]
Convert From PyTorch Weights
Run the exporter from this repository root. The VR_Video_Toolbox_NE virtual
environment already contains the PyTorch dependencies (the runtime venv is
ONNX-only and has no torch):
bash
1G:/GIT/debug/VR_Video_Toolbox_NE/.venv/Scripts/python.exe \2 examples/export_nvds_onnx.py --split --width 512 --height 288 --device cuda
Export the high-quality tier:
python examples/export_nvds_onnx.py --split --width 672 --height 384 --device cuda
Export the monolithic graph instead of the split pair:
python examples/export_nvds_onnx.py --width 672 --height 384 --device cuda
Useful options:
text
1--split Export backbone + head graphs instead of the monolith.
2--width 672 Input width. Must be a multiple of 32.
3--height 384 Input height. Must be a multiple of 32.
4--source-root PATH Vendored NVDS source root (contains full_model.py).
5--checkpoint PATH NVDS_Stabilizer.pth.
6--output PATH Output path for the monolithic export.
7--opset 17 ONNX opset version.
8--device cpu|cuda Device used for tracing.
9--dynamic-batch Mark only the batch axis dynamic (monolithic only).
10--skip-ort-check Skip the ONNX Runtime comparison.
Expected checkpoint layout:
text
1NVDS/
2 NVDS_Stabilizer.pth
Validation
The exporter compares outputs with ONNX Runtime and reports a max/mean absolute
difference. In --split mode it runs the per-frame backbone plus head over a
4-frame window and compares against the monolithic PyTorch forward, so a single
run verifies that the split is equivalent to the original model.
Notes
TensorRT cannot run this graph: the optimized model exceeds the 2 GB protobuf
limit and the graph contains many ScatterND / dynamic ops. The runtime treats
a trt request for NVDS as CUDA and bounds the CUDA arena with gpu_mem_limit
(DA3 still uses TensorRT normally).
NVDS is VRAM-heavy. Sharing the GPU with DA3 and the renderer pushes the
combined working set close to 16 GB; the bounded arena keeps it from spilling
into Windows shared memory.
NVDS is limited to 16:9 input in this project.
The output is a stabilized near/disparity map and must be rendered through the
render_near(...) path, not the raw-depth path.
The split graphs are preferred and auto-detected at runtime; the monolith is a
fallback when the split files are absent.
Citation
If you use NVDS or ONNX exports derived from it, cite the upstream work (verify
against the upstream repository for the authoritative entry):
bibtex
1@InProceedings{Wang_2023_ICCV,
2 author = {Wang, Yiran and Pan, Zhiyu and Li, Xingyi and Cao, Zhiguo and Xian, Ke and Zhang, Jianming},
3 title = {Neural Video Depth Stabilizer},
4 booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
5 year = {2023}
6}
