MMFace-DiT: A Dual-Stream Diffusion Transformer for High-Fidelity Multimodal Face Generation
Authors: Bharath Krishnamurthy and Ajita Rattani
Affiliation: University of North Texas, Denton, Texas, USA
Accepted to IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2026)
Abstract
Recent multimodal face generation models address the spatial control limitations of text-to-image diffusion models by augmenting text-based conditioning with spatial priors such as segmentation masks, sketches, or edge maps. However, existing approaches typically append auxiliary control modules or stitch together separate uni-modal networks.
We introduce MMFace-DiT, a unified dual-stream diffusion transformer engineered for synergistic multimodal face synthesis. Its core novelty lies in a dual-stream transformer block that processes spatial (mask/sketch) and semantic (text) tokens in parallel, deeply fusing them through a shared Rotary Position-Embedded (RoPE) Attention mechanism. Furthermore, a novel Modality Embedder enables a single cohesive model to dynamically adapt to varying spatial conditions without retraining. MMFace-DiT achieves a 40% improvement in visual fidelity and prompt alignment over five state-of-the-art multimodal face generation models.
Repository Contents
This repository contains the trained model checkpoints for MMFace-DiT. The models are provided for both Diffusion and Rectified Flow Matching (Flow) paradigms across different resolutions.
dit-unified-flux-vae-256: Diffusion paradigm model for 256x256 resolution using the unified FLUX VAE (checkpoint-440700).dit-unified-flux-vae-256-rfm: Rectified Flow Matching (RFM) paradigm model for 256x256 resolution (checkpoint-283517).dit-unified-flux-vae-512-rfm: Rectified Flow Matching (RFM) paradigm model for 512x512 resolution (checkpoint-44070).VAE: Standalone VAE weights utilizing the compressed 16-channel FLUX latent space.stable-diffusion-2-1-base: Base SD 2.1 component structures required for the pipeline (Tokenizers, Text Encoders, Schedulers).
Usage & Inference
Please refer to our
Official GitHub Project Page for complete inference scripts, training code, and setup instructions.
Example Inference (Flow - Mask Conditioning)
1python sample_flow.py \
2 --config_path "configs/flow/config_256_unified_rfm.yml" \
3 --weights_path "path/to/downloaded/dit-unified-flux-vae-256-rfm/checkpoint-283517/dit_model_weights_ema.safetensors" \
4 --modality "mask" \
5 --conditioning_path "path/to/mask.png" \
6 --prompt "A stunning young woman with long, wavy blonde hair..." \
7 --output_dir "Generated_Samples" \
8 --num_samples 4 \
9 --guidance_scale 7.5
Citation:
If you find this work helpful for your research, please cite our CVPR paper:
1@article{krishnamurthy2026mmface,
2 title={MMFace-DiT: A Dual-Stream Diffusion Transformer for High-Fidelity Multimodal Face Generation},
3 author={Krishnamurthy, Bharath and Rattani, Ajita},
4 journal={arXiv preprint arXiv:2603.29029},
5 year={2026}
6}
