You can deploy this model and start earning money today!

s23deepak/
streaming-vlm-qwen3-rocm

Name: streaming-vlm-qwen3-rocm
Availability: InStock

Views

No views yet

StreamingVLM — Qwen3-VL (2B & 4B) for CUDA + ROCm

PitchAI: Your AI Broadcast Companion — Real-time AI-powered football commentary with live tactical vision.

This is an updated port of StreamingVLM, adapted for Qwen3-VL-2B/4B-Instruct. StreamingVLM is designed for real-time, stable understanding of infinite video streams by maintaining a compact KV cache.

Paper: StreamingVLM: Real-Time Understanding for Infinite Video Streams
Project Page: https://streamingvlm.hanlab.ai
Original Code: mit-han-lab/streaming-vlm

This port is specifically optimized for:

Qwen3-VL-2B-Instruct ⚡ (lightweight, runs on consumer GPUs)
Qwen3-VL-4B-Instruct 🏋️ (higher quality commentary)
NVIDIA CUDA with flash_attention_2 (maximum throughput)
AMD ROCm with sdpa (no flash-attn build needed)
Latest Transformers (4.57.0+ from source)

Supported Configurations

Model	GPU	Attention	VRAM	Speed	Use Case
Qwen3-VL-2B	RTX 3090/4090	`flash_attention_2`	~6 GB	~12 FPS	Real-time on consumer HW
Qwen3-VL-2B	MI250X/MI300X	`sdpa`	~6 GB	~8 FPS	ROCm deployment
Qwen3-VL-4B	A100/H100	`flash_attention_2`	~12 GB	~8 FPS	Production quality
Qwen3-VL-4B	MI300X	`sdpa`	~12 GB	~6 FPS	ROCm high quality

Key Changes from Original StreamingVLM

Component	Original	This Port
Model	Qwen2.5-VL-7B-Instruct	Qwen3-VL-2B/4B-Instruct
Attention	flash_attention_2 (CUDA only)	SDPA + flash_attention_2 (both!)
Transformers	4.50.0–4.52.4	4.57.0+ (from source)
Vision Attn	flash_attn_varlen_func	chunked SDPA
ViT Patch Size	14×14	16×16
RoPE	mrope (standard)	mrope_interleaved
DeepStack	❌	✅ (layers 5, 11, 17)
Max Context	128K	256K

Installation

For CUDA (NVIDIA GPUs)

bash

conda create -n streaming_qwen3 python=3.11 -y
conda activate streaming_qwen3

# PyTorch with CUDA
pip install torch torchvision

# Transformers from source (Qwen3-VL requires 4.57.0+)
pip install git+https://github.com/huggingface/transformers

# Flash Attention 2 (recommended for CUDA)
pip install flash-attn --no-build-isolation

# Other deps
pip install -r requirements.txt

For ROCm (AMD GPUs)

bash

conda create -n streaming_qwen3 python=3.11 -y
conda activate streaming_qwen3

# PyTorch with ROCm
pip install torch torchvision --index-url https://download.pytorch.org/whl/rocm6.2

# Transformers from source
pip install git+https://github.com/huggingface/transformers

# No flash-attn needed! SDPA works out of the box.
pip install -r requirements.txt

Quick Start

🚀 2B Model on CUDA

bash

python -m streaming_vlm.inference.inference \
    --model_path Qwen/Qwen3-VL-2B-Instruct \
    --video_path /path/to/football_match.mp4 \
    --output_path ./commentary.vtt \
    --fps 4 \
    --attn_implementation flash_attention_2 \
    --window_size 24 \
    --text_sliding_window 768

💬 Interactive Q&A Mode

bash

python -m streaming_vlm.inference.inference \
    --model_path Qwen/Qwen3-VL-2B-Instruct \
    --video_path /path/to/match.mp4 \
    --interactive \
    --attn_implementation flash_attention_2

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "s23deepak/streaming-vlm-qwen3-rocm"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

Architecture Overview

StreamingVLM maintains a compact KV cache for infinite video streams:

Attention Sinks (512 tokens): Stabilize attention over long sequences.
Visual Window (16–24s): Recent video frames kept in KV cache.
Text Window (512–768 tokens): Recent text context maintained.
Contiguous RoPE: Keeps position indices bounded for infinite streams.

Citation

bibtex

@misc{xu2025streamingvlmrealtimeunderstandinginfinite,
      title={StreamingVLM: Real-Time Understanding for Infinite Video Streams}, 
      author={Ruyi Xu and Guangxuan Xiao and Yukang Chen and Liuning He and Kelly Peng and Yao Lu and Song Han},
      year={2025},
      eprint={2510.09608},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2510.09608}, 
}

Generated by ML Intern

This model repository was generated by ML Intern, an agent for machine learning research and development on the Hugging Face Hub.

Try ML Intern: https://smolagents-ml-intern.hf.space
Source code: https://github.com/huggingface/ml-intern