This framework provides ability to use any combination of text, graph, image and video as conditioning for video synthesisation. We have provided sample configs to run training and inference for all these combinations. Feel free to use our work for comparisons and to cite it!
🔑 Key Features
SWoMo is a neuro-symbolic world model for surgical simulation that decouples interaction dynamics from visual appearance.
Using an inverse pairing strategy, real surgical videos are reconstructed in a simulator to create paired data for training a video diffusion model for sim-to-real translation, with intermediate scene graphs serving as a constraint regularizer.
We demonstrate improved phase recognition, unsupervised style transfer, and strong generalisation to unseen interaction geometries.
We released our interactive SAM2-based annotation tool in a separate repository: IntrekSAM. In our research, we found that there was no existing tool for video segmentation annotation that is free, open-source, locally deployable, easily modifiable, supports multi-class segmentation, and is simple to set up. Therefore, we rewrote the GUI in Python while still keeping the original SAM2 backend.
We also make our processed Cataract-1k data available on Hugging Face, including real videos, simulated videos, simulated segmentations, and scene graphs. If you would like to use our manually annotated segmentations of the real videos (at 16 fps) for the 869 videos from Cataract-1K, please contact the authors via the email address in the paper. We would also be happy to share additional annotations described in the paper, such as phase labels and tracking point annotation, upon request.
Step 2: Train Another VAE (For Video Diffusion Model)
bash
1python swomo/ldm/main.py --base configs/vae/config_autoencoderkl_cataract.yaml -t --gpus 0, --logdir ./checkpoints/Cataract-1K
23# Converting a CompVis VAE to Diffusers VAE Format4# IMPORTANT: First update Diffusers to version 0.31.0, run the script, and then downgrade back to 0.21.25python scripts/ae_compvis_to_diffuser.py \6 --vae_pt_path /path/to/checkpoints/last.ckpt \7 --dump_path /path/to/save/vae_vid_diffusion
Update the finetuned_unet_path in the config with the model trained in Step 4. The training of ControlNet can also be run in a multi-GPU setup, similar to Step 4.
If you are using SWoMo for your paper, please cite the following paper:
bibtex
1@article{sivakumar2026swomo,
2 title={SWoMo: Neuro-Symbolic World Model for Cataract Surgery Simulation},
3 author={Sivakumar, Ssharvien Kumar and Johnson, Akwele and Dhingra, Anirudh and Frisch, Yannik and Ghazaei, Ghazal and Mukhopadhyay, Anirban},
4 journal={arXiv preprint arXiv:2605.16530},
5 year={2026}
6}
⭐ Acknowledgement
Thanks for the following projects and theoretical works that we have either used or inspired from:
