You can deploy this model and start earning money today!

rootlocalghost/
NAVA

Name: NAVA
Rating: 2.5 (12 reviews)

audio-video-generation

Wan-AI/Wan2.2-TI2V-5B

finetune

apache-2.0

Views

No views yet

NAVA

NAVA — Native Audio-Visual Alignment for Generation

State-of-the-art audio-visual synchronization with only 6.3 B parameters.

arXiv

Code

Project Page

License

Params

Base model

ERNIE Team · Baidu Inc. · arXiv 2026

⭐ If you find this model useful, please consider giving our GitHub repo a star! ⭐

📖 中文版 README

TL;DR

NAVA is a 6.3 B-parameter joint audio-video generator that synthesizes synchronized video and audio from a single prompt — including multi-speaker speech with reference-timbre control and image-conditioned continuations.

Instead of post-hoc-aligned dual towers or fully unified tri-modal stacks, NAVA uses an Align-then-Fuse MMDiT: a dedicated alignment space first establishes audio-video correspondence, then context (text, speaker embeddings) is fused via cross-attention. On Verse-Bench it sets new SOTA on Sync-C / Sync-D / video quality / audio WER while using 2× to 5× fewer parameters than open-source baselines.

Highlights

720p 1-min Fast Generation — 720p synchronized audio-video in ~1 minute via 8-GPU Ulysses sequence parallel.

Dual-Channel Audio — stereo audio (scene + speech) jointly denoised with video, no post-hoc vocoder alignment.

Precise Multi-Timbre Control — reference WAVs bound to <S>...<E> speech spans for per-speaker voice identity.

Language-Described Camera Control — shot composition, motion, and pacing directly from the prompt.

Multi-Resolution — landscape / portrait / square aspect ratios from the same checkpoint.

Model Details

Quick Facts


Architecture	Align-then-Fuse MMDiT (Wan2.2 backbone)
Parameters	6.3 B (backbone, joint AV)
Modality	Joint audio + video, text-conditioned
Resolution	1280×704 (recommended) · 960×960 also supported
Frames / FPS	37 frames @ 24 fps ≈ 6 s · 55–61 frames ≈ 9–10 s
Audio	25 latent tokens / sec, ≤ 10 s
Sampling	Flow matching · UniPC scheduler · 50 default steps
Precision	bf16
Parallelism	Single-GPU or Ulysses sequence parallel (up to 8 GPUs)
Base model	Wan-AI/Wan2.2-TI2V-5B

Architecture

NAVA instantiates Native Audio-Visual Alignment as an Align-then-Fuse MMDiT stack:

Hierarchical Alignment Layers — 10 double-stream blocks. Video and audio keep separate QKV projections and FFNs but share a joint self-attention over concatenated [video_tokens; audio_tokens], plus dedicated cross-attention to text. This builds an alignment space where AV correspondence is learned without semantic context interference.
Unified Fusion Layers — 20 single-stream blocks. Video and audio share QKV/FFN; a unified joint attention treats all tokens as one stream, with a single text cross-attention path. This is where context-conditioned denoising happens.
Backbone hyperparameters. dim=3072, ffn_dim=14336, 24 attention heads, 30 layers (10 double + 20 single), text_len=512, patch size (1, 2, 2). RMSNorm on QK; cross-attention norm; ε = 1e-6.
Positional encoding. 3D RoPE for video (temporal + height + width), 1D RoPE for audio, applied jointly inside the joint-attention path.
Timbre-in-Context Conditioning. Reference-WAV speaker embeddings (ReDimNet, 192-d) are injected through the context pathway and bound to <S>...<E> speech spans, enabling per-speaker timbre control in multi-speaker scenes.
3D cross-modal CFG. Independent classifier-free guidance scales for video, audio, and the cross-modal alignment direction (video_align_guidance_scale, audio_align_guidance_scale) keep AV synchronization tight at inference.

What's Different from Existing Open-Source AV Models

Design axis	Typical baselines	NAVA
Stream layout	Dual-tower (post-hoc align) or fully unified tri-modal	Align-then-Fuse — alignment space first, context fused after
Speech control	Caption-only, no per-speaker timbre	Timbre-in-Context via reference WAVs
Param budget	10 B – 32 B	6.3 B

Components Shipped Alongside the Backbone

Component	Description	Size
WanAVModel (backbone)	MMDiT, joint AV attention	6.3 B
Wan2.2 Video VAE	Causal 3D ConvNet · 16×16×4 spatial-temporal compression · 48 latent channels	2.7 GB
LTX Audio VAE + Vocoder	128 latent channels · 25 tokens/sec · built-in waveform decoder	348 MB
umt5-xxl Text Encoder	T5 · 4096-d embeddings	11 GB
ReDimNet	Speaker embedding · 192-d	~50 MB

Evaluation

Table 1 — VerseBench (general AV capability)

NAVA achieves the best AV synchronization (Sync-C / Sync-D), video quality, and audio WER, with the smallest parameter budget.

Model	Params	Resolution	Sync-C ↑	Sync-D ↓	IB ↑	Video Quality ↑	WER ↓	PQ ↑	FD ↓
Ovi 1.1	10 B	720p	7.4839	7.9791	0.199	0.636	0.102	5.8432	0.9418
MOVA	A18B (32 B)	720p	7.2888	7.808	0.269	0.603	0.126	7.2331	0.9222
Davinci	15 B	540p	7.1487	7.8158	0.269	0.600	0.151	5.9559	0.9307
LTX 2.3	19 B	512p	7.2476	7.6902	0.337	0.576	0.106	6.9459	0.8287
NAVA (ours)	6.3 B	720p	7.7914	7.5655	0.313	0.659	0.099	6.8609	0.8328

_{↑ higher is better · ↓ lower is better · bold = best · underline = 2nd best.}

Table 2 — Seed-TTS-eval (speech quality)

Among joint AV models, NAVA delivers speech quality close to dedicated audio-only systems. Audio-only rows are listed for reference; they are not directly comparable.

Category	Model	WER ↓	Speaker Similarity ↑
Audio-Only (reference)	CosyVoice	4.29	60.9
Audio-Only (reference)	Qwen2.5-Omni	2.72	63.2
Audio-Video Joint	DreamID-Omni	33.44	34.1
Audio-Video Joint	NAVA (ours)	5.81	62.4

How to Use

TL;DR command. After §1 setup is complete:

bash
1bash scripts/inference.sh # General T2AV 2bash scripts/inference_timbre.sh # I2AV + timbre control

Outputs land under eval_results/.

1 · Setup (once)

bash

git clone https://github.com/ernie-research/NAVA && cd NAVA

# Python deps
pip install torch torchvision torchaudio
pip install diffusers transformers accelerate safetensors einops scipy PyYAML tqdm sentencepiece
pip install flash-attn --no-build-isolation

# All weights in one shot — main checkpoint + Wan2.2 VAE + T5 + LTX audio VAE
huggingface-cli download <NAVA-repo-id> --local-dir .

Expected on-disk layout

NAVA/
├── NAVA.ckpt                                                    # main checkpoint (24 GB)
├── Wan2.2-TI2V-5B/
│   ├── Wan2.2_VAE.pth                                           # 2.7 GB
│   ├── models_t5_umt5-xxl-enc-bf16.pth                          # 11 GB
│   └── google/umt5-xxl/{spiece.model, tokenizer.json}
├── params/
│   └── LTX2/
│       ├── ltx-2.3-22b-dev_audio_vae.safetensors                # 348 MB
│       └── LICENSE                                              # LTX-2 Community License
└── configs/                                                     # inference YAMLs

The LTX audio-VAE Python code is vendored under nava_src/vendor/ltx_core/ (see its NOTICE.md), so no separate clone of the LTX-Video repo is needed. ReDimNet is fetched via torch.hub on first run.

2 · One-command inference (recommended, 8 GPU SP)

The repo ships two end-to-end scripts that build a JSONL inline and launch SP=8 inference:

bash

# General T2AV (text-only)
bash scripts/inference.sh

# I2AV + Timbre Control (first-frame image + reference voice)
bash scripts/inference_timbre.sh

Override defaults via env vars:

bash

CKPT=/path/to/NAVA.ckpt OUT_DIR=eval_results/run1 bash scripts/inference.sh
TIMBRE_SCALE=3.0 SPK_WAV=/path/to/spk.wav    bash scripts/inference_timbre.sh

3 · Custom batches — write your own JSONL

Each line is one prompt:

jsonl

{"prompt": "一位男子在海边奔跑，镜头跟随。背景是海浪声和风声。"}
{"prompt": "两人对话<S>Hello<E><S>Hi there<E>", "spk_wavs": ["spk1.wav", "spk2.wav"]}
{"prompt": "镜头跟随主体...", "image_path": "/abs/path/first_frame.png"}

Field	Required	Description
`prompt`	yes	Text caption (also accepts legacy `text` field name)
`image_path`	no	Absolute path to first-frame image — auto-enables I2V for this sample
`spk_wavs`	no	List of absolute paths to speaker reference WAVs (max 2)

Then launch:

bash

SETUPTOOLS_USE_DISTUTILS=stdlib torchrun \
    --nnodes=1 --nproc_per_node=8 \
    --master_addr=127.0.0.1 --master_port=29507 \
    inference_nava.py \
    --config configs/baseline_t2av_demo_mmdit_no_split_ltx_control_unipc.yaml \
    --ckpt NAVA.ckpt \
    --out_dir ./outputs \
    --data_format json --data_file my_prompts.jsonl \
    --width 1280 --height 704 --frames 37 --fps 24 \
    --steps 50 --save_sample --gen_turn 1 --use_sp

Outputs land at outputs/{save_path}-{gen_turn}_av.mp4. For timbre-controlled samples, also pass --timbre_cfg --timbre_align_guidance_scale 3.0.

Mode cheatsheet

Goal	JSONL fields	Extra flags
Text → AV	`prompt`	—
Image → AV	`prompt` + `image_path`	(auto-detected)
Timbre-controlled speech	`prompt` + `spk_wavs`	`--timbre_cfg --timbre_align_guidance_scale 3.0`
9-second video	any	`--frames 55`
Single-GPU (slower)	any	omit `--use_sp`

4 · Prompt rewriting (recommended for short / English inputs)

NAVA is trained on Chinese dense captions; short or English prompts benefit substantially from rewriting before inference. Three pathways are provided, all sharing the same system prompt and sampling profile (so output style stays consistent), with <S>...<E> speech spans preserved verbatim.

Pathway	Backend	Speed	Best for
vLLM batch server (`pe_src/`)	Qwen3-4B-Thinking-2507 served via vLLM, async HTTP	< 2 s / prompt	Offline batches
Local transformers, single (`gradio_demo/rewrite_single.py`)	Same model, in-process	40–80 s / prompt	One-off CLI
Gradio "Rewrite" button	Same as above, hosted in Gradio	40–80 s / prompt	Interactive UI

bash

# Batch path: start vLLM server, then rewrite a txt of prompts
bash pe_src/start_server.sh --gpu 0 --low-footprint
python pe_src/rewrite.py -i prompts.txt -o prompts_rewritten.txt

5 · Gradio Web UI

Interactive demo with click-to-rewrite (Qwen3-4B), image upload, and reference-WAV upload:

bash

bash gradio_demo/start_gradio.sh \
    --config configs/baseline_t2av_demo_mmdit_no_split_ltx_control_unipc.yaml \
    --ckpt NAVA.ckpt \
    --rewrite_model pe_src/Qwen3-4B-Thinking-2507 \
    --port 8000 --nproc 8

Debug mode (no models, UI only)

python gradio_demo/gradio_server.py --debug --port 8000

Bias, Safety, and Misuse

NAVA can synthesize video and speech conditioned on a reference image (image_path) and reference voice (spk_wavs). Using it to depict real persons without consent — including face-likeness or voice-likeness reproduction — is prohibited by the license and may also be illegal in your jurisdiction. We recommend:

Only use consent-approved reference media.
Label generated content as synthetic.
Apply provenance / watermarking before redistribution.

Citation

bibtex

@article{nava2026,
  title   = {NAVA: Native Audio-Visual Alignment for Joint Audio-Video Generation},
  author  = {ERNIE Team},
  journal = {arXiv preprint},
  year    = {2026},
}

Acknowledgements

NAVA builds on excellent upstream work: Wan2.2-TI2V-5B (video backbone & VAE), LTX 2.3 (audio VAE + built-in vocoder), umt5-xxl (text encoder), and ReDimNet (speaker embedding). We also thank the open-source AV-generation community — Ovi, MOVA, Davinci, LTX — for releasing strong baselines that made fair benchmarking possible.

License & Contact

Released under Apache-2.0. For research / commercial inquiries, contact the ERNIE team at Baidu Inc.

rootlocalghost/NAVA

NAVA — Native Audio-Visual Alignment for Generation

TL;DR

Model Details

Quick Facts

Architecture

What's Different from Existing Open-Source AV Models

Components Shipped Alongside the Backbone

Evaluation

Table 1 — VerseBench (general AV capability)

Table 2 — Seed-TTS-eval (speech quality)

How to Use

1 · Setup (once)

2 · One-command inference (recommended, 8 GPU SP)

3 · Custom batches — write your own JSONL

Mode cheatsheet

4 · Prompt rewriting (recommended for short / English inputs)

5 · Gradio Web UI

Bias, Safety, and Misuse

Citation

Acknowledgements

License & Contact

rootlocalghost/
NAVA