DMNSP: Dynamic Multi-Layer Null Space Projection for Vision-Language Continual Learning
Official implementation of the paper "Dynamic Multi-Layer Null Space Projection for Vision-Language Continual Learning" (ICCV 2025) in PyTorch.
🎯 Abstract
Vision-Language Models (VLM) have emerged as a highly promising approach for Continual Learning (CL) due to their powerful generalized features. While adapter-based VLM can exploit both task-specific and task-agnostic features, current CL methods have largely overlooked the distinct and evolving parameter distributions in visual and language modalities, which are found crucial for effectively mitigating catastrophic forgetting.In this study, we find that the visual modality experiences a broader parameter distribution and greater variance during class increments than the textual modality, leading to higher vulnerability to forgetting. Consequently, we handle the branches of the two modalities asymmetrically.
Key Contributions
- 🔍 Asymmetric Modality Handling: We propose handling visual and language modalities differently based on their distinct parameter distribution characteristics
- 🚀 Multi-layer Null Space Projection: A novel strategy applied only to the visual modality branch to restrict parameter updates within specific subspaces
- ⚖️ Dynamic Projection Coefficient: Precise control of gradient projection magnitude for optimal stability-plasticity balance
🛠️ Installation
Setup Environment
1# Install dependencies
2pip install -r requirements.txt
📊 Datasets
The framework supports the following datasets for class incremental learning:
- CIFAR100: 100 classes, various incremental settings (2-2, 5-5, 10-10)
- TinyImageNet: 200 classes, incremental settings (200-100-5, 200-100-10, 200-100-20)
Data Preparation
- The datasets will be automatically downloaded when running experiments
- Update the
dataset_root path in your configuration files or command line
- Ensure sufficient disk space for dataset storage
🚀 Quick Start
Basic Usage
1# Run CIFAR100 with 10 initial classes and 10 incremental classes
2sh run_cifar100-10-10.sh
3
4# Or run with custom parameters
5python main.py \
6 --config-path ./configs/class \
7 --config-name cifar100_10-10.yaml \
8 dataset_root="/path/to/your/data" \
9 class_order="./class_orders/cifar100.yaml"
Configuration Options
The project uses Hydra for configuration management. Key parameters include:
1# Model settings
2model_name: "ViT-B/16" # CLIP model variant
3prompt_template: "a bad photo of a {}." # Text prompt template
4
5# Training settings
6batch_size: 128 # Training batch size
7lr: 1e-3 # Learning rate
8weight_decay: 0.0 # Weight decay
9ls: 0.0 # Label smoothing
10
11# Incremental learning settings
12initial_increment: 10 # Initial number of classes
13increment: 10 # Classes per incremental step
14method: "DMNSP" # Method name
🔧 Advanced Usage
Custom Datasets
To add support for new datasets:
- Add dataset configuration in
continual_clip/datasets.py
- Create corresponding class order file in
class_orders/
- Add configuration YAML in
configs/class/
📄 License
This project is licensed under the MIT License - see the
LICENSE file for details.
📚 Citation
If you find this work useful in your research, please consider citing:
1@inproceedings{Kang2025DMNSP,
2 title={Dynamic Multi-Layer Null Space Projection for Vision-Language Continual Learning},
3 author={Borui Kang, Lei Wang, Zhiping Wu, Tao Feng, Yawen Li, Yang Gao, Wenbin Li},
4 journal={ICCV},
5 year={2025}
6}
📞 Contact
For questions or issues, please:
- Open an issue on GitHub
- Contact the authors at [kangborui.cn@gmail.com]
Note: This implementation is for research purposes. Please ensure you comply with the respective licenses of the datasets and models used.
