[2026-05-11] Code and LoRA weights for LTX-2 are available.
🏗️ Method Overview
Modality-wise Advantage Routing — Instead of collapsing all rewards into a single global advantage, OmniNFT computes independent per-reward advantages for video, audio, and cross-modal synchronization, then routes each to its responsible generation branch — uni-modal advantages supervise only their own branch while the synchronization advantage is broadcast to both — resolving the advantage inconsistency where roughly half of samples receive opposing rewards across modalities.
Layer-wise Gradient Surgery — To address gradient imbalance where video-branch gradients leak into shallow audio layers dedicated to intra-modal generation, OmniNFT applies a partial stop-gradient on the audio key-value projections in A2V cross-attention at shallow Transformer blocks, suppressing erroneous gradient injection while preserving full gradient flow through the deeper cross-modal alignment layers (AV-Sync Zone).
Region-wise Loss Reweighting — Leveraging V2A cross-attention maps from late denoising steps as an intrinsic proxy for sound-emitting critical regions, OmniNFT aggregates them into per-token importance weights that modulate the video-branch RL loss, providing fine-grained credit assignment that concentrates optimization capacity on regions most critical for audio-video synchronization without requiring external detection modules.
1python scripts/inference.py \2 --model_path ./merged_model.safetensors \3 --gemma_path $GEMMA_MODEL_PATH\4 --prompt "A man plays acoustic guitar on a wooden stage, warm applause from the audience"\5 --seed 42\6 --output_dir ./results
Arguments
Argument
Default
Description
--model_path
(required)
Path to merged .safetensors model
--gemma_path
env GEMMA_MODEL_PATH
Path to Gemma 3 text encoder
--prompt
(required)
Text prompt for generation
--num_frames
121
Number of video frames
--height / --width
model default
Video resolution
--num_inference_steps
model default
Number of denoising steps
--video_guidance_scale
model default
Video CFG scale
--audio_guidance_scale
model default
Audio CFG scale
--seed
42
Random seed
--no_audio
false
Disable audio generation
--dtype
bf16
Inference precision
Outputs are saved to --output_dir: .mp4 (video with audio) and .wav (audio only).
🖊️ Citation
bibtex
1@article{zhang2026omninft,
2 title={OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generation},
3 author={Zhang, Guohui and Ma, XiaoXiao and Huang, Jie and Xu, Hang and Yu, Hu and Fu, Siming and Li, Yuming and Xue, Zeyue and Song, Lin and Huang, Haoyang and Duan, Nan and Zhao, Feng},
4 journal={arXiv preprint arXiv:2605.12480},
5 year={2026}
6}
