LAMP: Language-Assisted Motion Planning
M. Burak Kizil
·
Enes Sanli
·
Niloy J. Mitra
·
Erkut Erdem
·
Aykut Erdem
·
Duygu Ceylan
arXiv
Webpage
GitHub
Introduction
LAMP defines a motion domain-specific language (DSL), inspired by cinematography conventions. By harnessing program synthesis capabilities of LLMs, LAMP generates structured motion programs from natural language, which are deterministically mapped to 3D trajectories.
🎉 News
⚙️ Installation
The codebase was tested with Python 3.11.13, CUDA version 12.8, and PyTorch >= 2.8.0
Setup for Model Inference
You can setup for LAMP model inference by running:
1git clone https://github.com/mbkizil/LAMP.git && cd LAMP
2pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128 # If PyTorch is not installed.
3pip install -r requirements.txt
4pip install wan@git+https://github.com/Wan-Video/Wan2.1
Download Models
Download the
VACE and finetuned
Qwen2.5-VL-7B-Instruct model weight using
download.sh1chmod +x download.sh
2./download.sh
🚀 Usage
In LAMP, users act as a director, providing natural language descriptions for both object and camera behaviors. The system translates these prompts into precise 3D Motion Programs and conditions the video generation process to produce cinematic shots.
Interactive Demo (Gradio)
To explore the full pipeline—from text-to-motion planning to final video synthesis—we provide an interactive Gradio interface. This single entry point handles the loading of the Motion Planner (Qwen2.5-VL) and the Video Generator (VACE) seamlessly.
1
2python -m src.serve.app --model-path ./qwen_checkpoints/LAMP-Qwen-2.5-VL
3
This script will:
Load the LLM Motion Planner (Qwen2.5-based) into memory.
Initialize the embedded VACE pipeline for trajectory-conditioned generation.
Launch a local web server (default at
http://127.0.0.1:8890).
💡
Notes from VACE:
(1) Please refer to
vace/vace_wan_inference.py for the inference args.
(2) For English language Wan2.1 users, you need prompt extension to unlock the full model performance.
Please follow the
instruction of Wan2.1 and set
--use_prompt_extend while running inference.
Acknowledgement
We are grateful for the following awesome projects that served as the foundation for LAMP, including
VACE for the powerful all-in-one video generation backbone and
Qwen for the robust language reasoning capabilities. We also extend our thanks to
Qwen-VL-Series-Finetune, which provided an efficient framework for training our motion planner.
Additionally, we acknowledge the pioneering works in camera control and trajectory generation, specifically
GenDoP and
Exceptional Trajectories. Their contributions to motion datasets and evaluation methodologies have brought immense inspiration to this project and established essential baselines for the field of controllable video generation.
BibTeX
1@misc{kizil2025lamplanguageassistedmotionplanning,
2 title={LAMP: Language-Assisted Motion Planning for Controllable Video Generation},
3 author={Muhammed Burak Kizil and Enes Sanli and Niloy J. Mitra and Erkut Erdem and Aykut Erdem and Duygu Ceylan},
4 year={2025},
5 eprint={2512.03619},
6 archivePrefix={arXiv},
7 primaryClass={cs.CV},
8 url={https://arxiv.org/abs/2512.03619},
9}
