Views
No views yet
conda create -n robustclip python==3.11conda activate robustclip pip install -r requirements.txtcd ./open_clip_torchpython setup.py developdataset_path
└─imagenet
└─train
└─n01440764
xxxxxx.JPEG
.....
└─......
└─val
└─n04254680
xxxxxx.JPEG
.....
└─......CUDA_VISIBLE_DEVICES=0,1,2,3 python -m train.training_clip_slots --clip_model_name ViT-L-14 --pretrained openai --dataset imagenet --imagenet_root /.../.../dataset_path/imagenet --template std --output_normalize False --steps 1000000 --warmup 10000 --batch_size 128 --loss l2 --opt adamw --lr 5e-5 --wd 1e-4 --attack pgd --inner_loss l2 --norm linf --eps 4 --iterations_adv 10 --stepsize_adv 1 --wandb False --output_dir ./output_slots --experiment_name SLOTS --log_freq 1000 --eval_freq 1000```CUDA_VISIBLE_DEVICES=0,1,2,3 python -m train.adversarial_training_clip_with_object_token --clip_model_name ViT-L-14 --slots_ckp ./ckps/model_slots_step_300000.pt --pretrained openai --dataset imagenet --imagenet_root /path/to/imagenet --template std --output_normalize False --steps 20000 --warmup 1400 --batch_size 128 --loss l2 --opt adamw --lr 1e-5 --wd 1e-4 --attack pgd --inner_loss l2 --norm linf --eps 4 --iterations_adv 10 --stepsize_adv 1 --wandb False --output_dir ./output --experiment_name with_OT --log_freq 10 --eval_freq 10--eps 2 to obtain SlotVLM2 models.CUDA_VISIBLE_DEVICES=0,1,2,3 python -m train.adversarial_training_clip_with_object_token --clip_model_name ViT-L-14 --slots_ckp ./ckps/model_slots_step_300000.pt --dataset imagenet --imagenet_root /path/to/imagenet --template std --output_normalize False --steps 20000 --warmup 1400 --batch_size 128 --loss l2 --opt adamw --lr 1e-5 --wd 1e-4 --attack pgd --inner_loss l2 --norm linf --eps 4 --iterations_adv 10 --stepsize_adv 1 --wandb False --output_dir ./output --experiment_name with_OT --log_freq 10 --eval_freq 10 --optimizer_state /home/xxx/RobustVLM/output/ViT-L-14_openai_imagenet_l2_imagenet_with_Object_Token_xxxxx/checkpoints/fallback_80000_opt.pt --start_step 80000 --pretrained nonebash directory are executable: chmod +x bash/*python -m CLIP_eval.clip_robustbench --clip_model_name ViT-L-14 --pretrained /path/to/ckpt.pt --dataset imagenet --imagenet_root /path/to/imagenet --wandb False --norm linf --eps 2--pretrained and the --eps 2/4 for SlotVLM2/4 models.CLIP_benchmark/benchmark/models.txt and datasets in CLIP_benchmark/benchmark/datasets.txt
(the datasets are downloaded from HuggingFace). Then run1cd CLIP_benchmark
2./bash/run_benchmark_adv.sh/bash/llava_eval.sh supply paths for the datasets. The required annotation files for the datasets can be obtained from this HuggingFace repository.
Set --vision_encoder_pretrained to openai or supply path to fine-tuned CLIP model checkpoint.
Then run./bash/llava_eval.sh/bash/of_eval_9B.sh and run./bash/of_eval_9B.sh./bash/llava_eval_targeted.shvlm_eval/run_evaluation_qualitative.py and runpython -m vlm_eval.run_evaluation_qualitative --precision float32 --attack apgd --eps 2 --steps 10000 --vlm_model_name llava --vision_encoder_pretrained openai --verbose1./bash/eval_pope.sh openai # for clean model evaluation
2./bash/eval_pope.sh # for robust model evaluation - add path_to_ckpt in bash file1./bash/eval_scienceqa.sh openai # for clean model evaluation
2./bash/eval_scienceqa.sh # for robust model evaluation - add path_to_ckpt in bash file