Views
No views yet
Ex-Omni is an public release for omni-modal response generation. Given text or speech input, the system can produce response text, speech units / decoded audio, and 52-dimensional facial blendshape coefficients, with optional rendering into a talking-face video.1.
2├── asset/ # Download the mesh templates here, e.g. EmoTalk.npz
3├── cosyvoice/ # Runtime audio decoder modules
4├── deploy.py # Main Gradio entrypoint
5├── deploy_base.py # Shared inference pipeline and UI logic
6├── ex_omni/
7│ ├── constants.py # Runtime constants
8│ ├── flow_inference.py # Audio decoder wrapper
9│ ├── render_utils.py # Blendshape rendering utilities
10│ └── model/
11│ ├── language_model/ # Omni model wrapper
12│ ├── speech_encoder/ # Whisper speech encoder
13│ ├── speech_projector/ # Speech projector
14│ ├── speech_generator/ # Speech generator
15│ └── blendshape_generator/ # Blendshape generator
16├── requirements.txt # Python dependencies
17└── LICENSE.txt # License file1# 1. Create and activate environment
2conda create -n Ex-Omni python=3.10 -y
3conda activate Ex-Omni
4
5# 2. Install PyTorch (example: CUDA 12.6)
6pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu1216
7
8# 3. Install project dependencies
9pip install -r requirements.txt
10
11# 4. Install pytorch3d separately according to your CUDA / PyTorch version
12# See official pytorch3d installation instructions| Component | Expected Path | Source |
|---|---|---|
| Ex-Omni checkpoint | ckpt/Ex-Omni/ |  |
| Flow decoder checkpoint | ckpt/glm-4-voice-decoder/flow.pt |  |
| HiFT decoder checkpoint | ckpt/glm-4-voice-decoder/hift.pt | |
| EmoTalk mesh template | asset/EmoTalk.npz |  |
| Claire mesh template | asset/claire.npz |  |
The Claire template is not redistributed in this repository. Please obtain the original Claire asset yourself and convert it into the.npzformat.
1python deploy.py \
2 --model-path ckpt/Ex-Omni \
3 --flow_ckpt_path ckpt/glm-4-voice-decoder/flow.pt \
4 --hift_ckpt_path ckpt/glm-4-voice-decoder/hift.pt \
5 --template_type emotalk \
6 --port 8080http://localhost:80801@misc{zhang2026exomnienabling3dfacial,
2 title={Ex-Omni: Enabling 3D Facial Animation Generation for Omni-modal Large Language Models},
3 author={Haoyu Zhang and Zhipeng Li and Yiwen Guo and Tianshu Yu},
4 year={2026},
5 eprint={2602.07106},
6 archivePrefix={arXiv},
7 primaryClass={cs.CV},
8 url={https://arxiv.org/abs/2602.07106},
9}