To facilitate community communication, we provide our model weights. We provide our foundation model and its metric version on target datasets (MVSEC and DENSE) respectively. As described in our paper, when applied to the target dataset, we only train the deep decoding head, the frozen image-event joint encoder (consisting of the image encoder, event encoder, and FreDFuse). So the three versions of the model only have different weights in the deep decoding head. For each version, we provide three types: small, base and large.
Foundation Model
Below is the weight of the image event joint estimation foundation model obtained by performing knowledge transfer through the FUSE framework we proposed, using Depth Anything V2 as the foundation model for image depth estimation. The depth decoding head weights below are from Depth Anything V2, which outputs inverse depth instead of depth.
Freeze the image event joint encoder weights of our foundation model and the metric depth estimator weights obtained by training the deep decoding head on the MVSEC dataset. The metrics are the evaluation results under MVSEC outdoor_night1.
Freeze the image event joint encoder weights of our foundation model and the metric depth estimator weights obtained by training the deep decoding head on the DENSE dataset.
Since the image, event, and depth labels in MVSEC are asynchronous, it is necessary to manually construct image-event-depth pairs, which can be achieved by running the script scripts/process_mvsec_hdf5.py.
In the event voxel grid representation, we choose 3 as the number of time bins. DENSE and EventScape initially provide a voxel grid bin of 5, which you can modify using the scripts under scripts directory.
Training
The training process is divided into three stages: Feature Alignment, Feature Fusion, and Adaptation to the Target Dataset.
We use image-event pairs (without using depth ground truth) from EventScape and Depth Anything V2 to train the image-event joint encoder. We use DENSE and MVSEC as target datasets to verify the model effect. On DENSE and MVSEC, we only train the decoder.
Feature Alignment
Run the script align_feature.sh.
You need to modify the variables load_from and save_path in the script, which represent the path of the pre-trained Depth Anything weights used and the path to save the training weights, respectively.
Feature Fusion
Run the script fuse_feature.sh.
The variable prompt_encoder_pretrained in this script should be the event encoder weight path obtained in the feature alignment stage
Adaptation to the Target Dataset
Run the script train.sh.
Inference
Run the script run.sh.
load_from indicates the path of the pre-trained weights to be loaded.
Evaluation
Run the script eval.sh.
predictions_dataset and target_dataset represent the directory paths where prediction results and depth data are stored respectively. clip_distanceindicates the maximum depth to be evaluated, which is 80 for MVSEC and 1000 for DENSE
Acknowledgements
This project includes code from the following repositories:
Depth-Anything-V2 - We use Depth Anything V2 as the image depth foundation model
