ViStream: Law-of-Charge-Conservation Inspired Spiking Neural Network for Visual Streaming Perception
ViStream is a novel energy-efficient framework for Visual Streaming Perception (VSP) that leverages Spiking Neural Networks (SNNs) with Law of Charge Conservation (LoCC) properties.
Model Details
Model Description
- Developed by: Kang You, Ziling Wei, Jing Yan, Boning Zhang, Qinghai Guo, Yaoyu Zhang, Zhezhi He
- Model type: Spiking Neural Network for Visual Streaming Perception
- Language(s): PyTorch implementation
- License: CC-BY-4.0
- Paper: CVPR 2025
- Repository: GitHub
Model Architecture
ViStream introduces two key innovations:
- Law of Charge Conservation (LoCC) property in ST-BIF neurons
- Differential Encoding (DiffEncode) scheme for temporal optimization
The framework achieves significant computational reduction while maintaining accuracy equivalent to ANN counterparts.
Uses
Direct Use
ViStream can be directly used for:
- Multiple Object Tracking (MOT)
- Single Object Tracking (SOT)
- Video Object Segmentation (VOS)
- Multiple Object Tracking and Segmentation (MOTS)
- Pose Tracking
Downstream Use
The model can be fine-tuned for various visual streaming perception tasks in:
- Autonomous driving
- UAV navigation
- AR/VR applications
- Real-time surveillance
Bias, Risks, and Limitations
Limitations
- Requires specific hardware optimization for maximum energy benefits
- Performance may vary with different frame rates
- Limited to visual perception tasks
Recommendations
- Test thoroughly on target hardware before deployment
- Consider computational constraints of edge devices
- Validate performance on domain-specific datasets
How to Get Started with the Model
1from huggingface_hub import hf_hub_download
2import torch
3
4# Download the checkpoint
5checkpoint_path = hf_hub_download(
6 repo_id="AndyBlocker/ViStream",
7 filename="checkpoint-90.pth"
8)
9
10# Load the model (requires ViStream implementation)
11checkpoint = torch.load(checkpoint_path, map_location='cpu')
For complete usage examples, see the
GitHub repository.
Training Details
Training Data
The model was trained on multiple datasets for various visual streaming perception tasks including object tracking, video object segmentation, and pose tracking.
Training Procedure
Training Details:
- Framework: PyTorch
- Optimization: Energy-efficient SNN training with Law of Charge Conservation
- Architecture: ResNet-based backbone with spike quantization layers
Evaluation
The model demonstrates competitive performance across multiple visual streaming perception tasks while achieving significant energy efficiency improvements compared to traditional ANN-based approaches. Detailed evaluation results are available in the
CVPR 2025 paper.
Model Card Authors
Kang You, Ziling Wei, Jing Yan, Boning Zhang, Qinghai Guo, Yaoyu Zhang, Zhezhi He
Model Card Contact
For questions about this model, please open an issue in the
GitHub repository.
Citation
1@inproceedings{you2025vistream,
2 title={VISTREAM: Improving Computation Efficiency of Visual Streaming Perception via Law-of-Charge-Conservation Inspired Spiking Neural Network},
3 author={You, Kang and Wei, Ziling and Yan, Jing and Zhang, Boning and Guo, Qinghai and Zhang, Yaoyu and He, Zhezhi},
4 booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference},
5 pages={8796--8805},
6 year={2025}
7}
