This project provides the online evaluation and distributed data parallel training code for FantasyVLN. The online evaluation is implemented based on the LH-VLN benchmark, and the training code is built upon ms-swift and qwen-vl.

FantasyVLN is a unified multimodal Chain-of-Thought (CoT) reasoning framework that enables efficient and precise navigation based on natural language instructions and visual observations. FantasyVLN combines the benefits of textual, visual, and multimodal CoT reasoning by constructing a unified representation space across these reasoning modes. To enable efficient reasoning, we align these CoT reasoning modes with non-CoT reasoning during training, while using only non-CoT reasoning at test time. Notably, we perform visual CoT in the latent space of a VAR model, where only low-scale latent representations are predicted. Compared to traditional pixel-level visual CoT methods, our approach significantly improves both training and inference efficiency.

We modify the LH-VLN codebase to support VLMs and multi-GPU inference.

You can use the following commands to install the required environment, or refer to the LH-VLN environment setup tutorial for more details.

LH-VLN uses HM3D as the scene dataset. The required data splits can be downloaded by following the command below. Note that an application must be submitted to Matterport before using the dataset. For more details, please refer to this link.

LH-VLN dataset is available in Hugging Face and ModelScope. The zipped files included in the downloaded dataset are not required for online evaluation.

Your final directory structure should be like this:

fantasy-vln/
├── lhvln/
│   ├── data/
│   │   ├── hm3d/
│   │   │   ├── train/
│   │   │   ├── val/
│   │   │   └── hm3d_annotated_basis.scene_dataset_config.json
│   │   ├── task/
│   │   │   ├── batch_1/
│   │   │   ├── ...
│   │   │   └── batch_8/
│   │   ├── step_task/
│   │   │   ├── batch_1/
│   │   │   ├── ...
│   │   │   └── batch_8/
│   │   └── episode_task/
│   │       ├── batch_1.json.gz
│   │       ├── ...
│   │       └── batch_8.json.gz

You must specify the following parameters before runing the script:

You can generate training data by runing the following commands:

PS: We used Qwen-VL-Max to generate textual CoT annotations for the data in swift_his_20_train_aug.jsonl. However, due to data licensing and privacy compliance considerations, we cannot release these annotations publicly. You may reproduce them by following the same procedure (describled in our paper).

The final directory structure should be like this:

If you find this work helpful, please consider giving us a ⭐️ and citing: