Views
No views yet
Audio: ...<AUDCAP> ... <ENDAUDCAP>example_prompts/gpt_examples_t2v.csvexample_prompts/gpt_examples_i2v.csvexample_prompts/gpt_examples_t2v.csvexample_prompts/gpt_examples_i2v.csv<S>Your speech content here<E> - Text enclosed in these tags will be converted to speechAudio: YOUR AUDIO DESCRIPTION - Describes the audio or sound effects present in the video at the end of prompt!1# Clone the repository
2git clone https://github.com/character-ai/Ovi.git
3
4cd Ovi
5
6# Create and activate virtual environment
7virtualenv ovi-env
8source ovi-env/bin/activate
9
10# Install PyTorch first
11pip install torch==2.6.0 torchvision torchaudio
12
13# Install other dependencies
14pip install -r requirements.txt
15
16# Install Flash Attention
17pip install flash_attn --no-build-isolation1git clone https://github.com/Dao-AILab/flash-attention.git
2cd flash-attention/hopper
3python setup.py install
4cd ../.. # Return to Ovi directory# Default is downloaded to ./ckpts, and the inference yaml is set to ./ckpts so no change required
# Default installs all versions of Ovi models, 720x720_5s, 960x960_5s, 960x960_10s
python3 download_weights.py
# For qint8 also ues python3 download_weights.py
OR
# Optional can specific --output-dir to download to a specific directory
# but if a custom directory is used, the inference yaml has to be updated with the custom directory
python3 download_weights.py --output-dir <custom_dir>
# Optional can specific --models to download selective versions of Ovi instead of all of them
# but if a custom directory is used, the inference yaml has to be updated with the custom directory
python3 download_weights.py --models 960x960_10s # ["720x720_5s", "960x960_5s", "960x960_10s"]
# Additionally, if you only have ~ 24Gb of GPU vram, please download the fp8 quantized version of the model, and follow the following instructions in sections below to run with fp8
wget -O "./ckpts/Ovi/model_fp8_e4m3fn.safetensors" "https://huggingface.co/rkfg/Ovi-fp8_quantized/resolve/main/model_fp8_e4m3fn.safetensors"1# Output and Model Configuration
2model_name: "960x960_10s" # ["720x720_5s", "960x960_5s", "960x960_10s"]
3output_dir: "/path/to/save/your/videos" # Directory to save generated videos
4ckpt_dir: "/path/to/your/ckpts/dir" # Path to model checkpoints
5
6# Generation Quality Settings
7sample_steps: 50 # Number of denoising steps. Lower (30-40) = faster generation
8solver_name: "unipc" # Sampling algorithm for denoising process
9shift: 5.0 # Timestep shift factor for sampling scheduler
10seed: 100 # Random seed for reproducible results
11
12# Guidance Strength Control
13audio_guidance_scale: 3.0 # Strength of audio conditioning. Higher = better audio-text sync
14video_guidance_scale: 4.0 # Strength of video conditioning. Higher = better video-text adherence
15slg_layer: 11 # Layer for applying SLG (Skip Layer Guidance) technique - feel free to try different layers!
16
17# Multi-GPU and Performance
18sp_size: 1 # Sequence parallelism size. Set equal to number of GPUs used
19cpu_offload: False # CPU offload, will largely reduce peak GPU VRAM but increase end to end runtime by ~20 seconds
20fp8: False # load fp8 version of model, will have quality degradation and will not have speed up in inference time as it still uses bf16 matmuls, but can be paired with cpu_offload=True, to run model with 24Gb of GPU vram
21
22# Input Configuration
23text_prompt: "/path/to/csv" or "your prompt here" # Text prompt OR path to CSV/TSV file with prompts
24mode: ['i2v', 't2v', 't2i2v'] # Generate t2v, i2v or t2i2v; if t2i2v, it will use flux krea to generate starting image and then will follow with i2v
25video_frame_height_width: [704, 1280] # Video dimensions [height, width] for T2V mode only
26each_example_n_times: 1 # Number of times to generate each prompt
27
28# Quality Control (Negative Prompts)
29video_negative_prompt: "jitter, bad hands, blur, distortion" # Artifacts to avoid in video
30audio_negative_prompt: "robotic, muffled, echo, distorted" # Artifacts to avoid in audiopython3 inference.py --config-file ovi/configs/inference/inference_fusion.yamltext_prompt can be a single string or path to a CSV file.torchrun --nnodes 1 --nproc_per_node 8 inference.py --config-file ovi/configs/inference/inference_fusion.yaml| Sequence Parallel Size | FlashAttention-3 Enabled | CPU Offload | With Image Gen Model | Peak VRAM Required | End-to-End Time |
|---|---|---|---|---|---|
| 1 | Yes | No | No | ~80 GB | ~83s |
| 1 | No | No | No | ~80 GB | ~96s |
| 1 | Yes | Yes | No | ~80 GB | ~105s |
| 1 | No | Yes | No | ~32 GB | ~118s |
| 1 | Yes | Yes | Yes | ~32 GB | ~140s |
| 4 | Yes | No | No | ~80 GB | ~55s |
| 8 | Yes | No | No | ~80 GB | ~40s |
ckpt_dir in ovi/configs/inference/inference_fusion.yaml to initialize the model1python3 gradio_app.py
2
3OR
4
5# To enable cpu offload to save GPU VRAM, will slow down end to end inference by ~20 seconds
6python3 gradio_app.py --cpu_offload
7
8OR
9
10# To enable an additional image generation model to generate first frames for I2V, cpu_offload is automatically enabled if image generation model is enabled
11python3 gradio_app.py --use_image_gen
12
13OR
14
15# To run model with 24Gb GPU vram. No need to download additional models.
16python3 gradio_app.py --cpu_offload --qint8
17
18# To run model with 24Gb GPU vram
19python3 gradio_app.py --cpu_offload --fp8
201@misc{low2025ovitwinbackbonecrossmodal,
2 title={Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation},
3 author={Chetwin Low and Weimin Wang and Calder Katyal},
4 year={2025},
5 eprint={2510.01284},
6 archivePrefix={arXiv},
7 primaryClass={cs.MM},
8 url={https://arxiv.org/abs/2510.01284},
9}