Unified masked-diffusion modeling across textual reasoning, image generation, image editing, multi-modal understanding, text to speech, and speech to text.
Dynin-Omni: Omnimodal Unified Large Diffusion Language Model is an 8B-scale masked-diffusion foundation model that unifies text, image, video, and speech understanding and generation within a single architecture.
Unlike autoregressive (AR) unified models that serialize heterogeneous modalities into a left-to-right sequence, Dynin-Omni models all modalities as discrete tokens in a shared vocabulary and performs generation via iterative masked denoising. This enables bidirectional context modeling, parallel multi-token prediction, and globally conditioned any-to-any inference without modality-specific expert decoders.
Training proceeds in three stages: (1) modality adaptation, (2) omni-modal supervised fine-tuning with model merging, and (3) continual capability scaling.
vLLM-Omni
Dynin-Omni support has been merged into vLLM-Omni through PR #1759 and is scheduled to be included in version 0.19.0. Once 0.19.0 is released, this section will be updated with the official setup and usage instructions.
Prerequisites
Direct local-machine inference and training with Dynin-Omni are supported. Follow the instructions below.
Initialize the environment (installs and builds Python packages):
bash scripts/init_env.sh --overwrite
--overwrite forces the Hugging Face cache root to datasets/huggingface under the project root. Without --overwrite, the cache root is resolved as HF_CACHE_DIR > HF_HOME > project default.
Inference
Dynin-Omni performs multimodal inference through iterative masked denoising. Target tokens are initialized as masks and refined over diffusion steps.
json format: each item includes id (source image filename) and prompt.
Source image directory (default): validation/data/image (override with --origin-img-root).
5. Speech (ASR and TTS)
Speech recognition and synthesis are performed within the same token-level diffusion backbone without a modality-specific decoder.
Validation script: validation/speech.py.
bash scripts/inference.sh --speech
Default source: LibriSpeech ASR test split from Hugging Face (openslr/librispeech_asr).
Optional local audio root: --librispeech-root (directory containing LibriSpeech .flac files).
Training
Training configurations (datasets, hyperparameters, etc.) are defined in configs/*.yaml.
scripts/train.sh path variables (CONFIG_FILE, TRAIN_SCRIPT, EXPERIMENT_CFG, LOG_DIR) must be specified as project-root-relative paths.
The examples below assume a single-node setup; host/runtime variables should be adapted to the target environment.
Accelerate configuration can be prepared by running:
python -m accelerate config
Predefined configurations are also available in accelerate_configs/:
Stage 1 starts from the MMaDA-8B-MixCoT backbone checkpoint and extends it to support video and speech modalities through vocabulary expansion and text-centric alignment.
Stage 2 Omni-Modal Supervised Fine-Tuning
Stage 2 continues from the Stage 1 checkpoint and performs full omni-modal supervised fine-tuning.
Activated modality directions:
Text → Text (Chat & Reasoning)
Image → Text, Video → Text (Multi-Modal Understanding)
Text → Image (Image Generation)
Image → Image (Image Editing)
Speech → Text (ASR)
Text → Speech (TTS)
Before training, model merging is applied between the original backbone and the Stage 1 checkpoint to mitigate catastrophic forgetting. Explicit <EOS> supervision enables stable variable-length generation across modalities.
Stage 3 starts from the Stage 2 checkpoint specified in configs/dynin_omni_stage3_llada_instruct.yaml and performs continual capability scaling under the same unified diffusion objective.
Evaluation
main_result
Dynin-Omni achieves the following results across several multimodal benchmarks as reported in the paper:
Language reasoning: 87.6 on GSM8K
Multimodal understanding: 1733.6 on MME-P
Video understanding: 61.4 on VideoMME
Image generation: 0.87 on GenEval
Speech recognition: 2.1 WER on LibriSpeech test-clean
1@article{aidaslab2026dyninomni,
2 title={Dynin-Omni: Omnimodal Unified Large Diffusion Language Model},
3 author={Kim, Jaeik and Kim, Woojin and Hong, Jihwan and Lee, Yejoon and Hyeon, Sieun and Lim, Mintaek and Han, Yunseok and Kim, Dogeun and Lee, Hoeun and Kim, Hyunggeun and Do, Jaeyoung},
4 journal={arXiv preprint arXiv:2604.00007},
5 year={2026}
6}
