You can deploy this model and start earning money today!

bartduis/
modality_forcing

Name: modality_forcing
Rating: 2.5 (826 reviews)

Views

No views yet

Modality Forcing for Scalable Spatial Generation

Joint text → RGB + depth generation with a single diffusion transformer, built on FLUX.2. Modality Forcing assigns separate noise levels per modality during post-training, so one model supports joint generation (text → RGB-D), image-to-depth, and depth-to-image at inference.

📄 Paper: arXiv:2606.13676
💻 Code: github.com/Duisterhof/modality-forcing
🚀 Demo: Hugging Face Space
🌐 Project page: modality-forcing.github.io

Files

File	Description
`model.safetensors`	FluxRGBD DiT (12B total — 9B-class FLUX.2 backbone + depth streams, bf16)
`config.json`	Model variant config (`flux_rgbd_9b_v2`)
`ae_encoder.safetensors` / `ae_decoder.safetensors`	FLUX.2 autoencoder

The Qwen3-8B text encoder is pulled separately from Qwen/Qwen3-8B.

Usage

bash

git clone https://github.com/Duisterhof/modality-forcing.git
cd modality-forcing
bash install.sh
python scripts/joint.py --prompt "a cozy sunlit kitchen with wooden cabinets"

The scripts download these weights automatically (bartduis/modality_forcing is the default --model).

License

The model weights are released under CC BY-NC 4.0 (non-commercial). The inference code is Apache-2.0; see the GitHub repository.

Citation

bibtex

@article{duisterhof2026mofo,
  title   = {Modality Forcing for Scalable Spatial Generation},
  author  = {Duisterhof, Bardienus Pieter and Ramanan, Deva and Ichnowski, Jeffrey and Johnson, Justin and Park, Keunhong},
  journal = {arXiv preprint arXiv:2606.13676},
  year    = {2026}
}