You can deploy this model and start earning money today!

wwwyyy/
TimeZero-Charades-7B

Name: TimeZero-Charades-7B
Availability: InStock

text-generation-inference

endpoints_compatible

Views

No views yet

TimeZero: Temporal Video Grounding with Reasoning-Guided LVLM

Updates

2025-03-17: TimeZero initial release! Code and evaluation scripts are now available.
2025-03-17: TimeZero achieves SOTA performance on Charades-STA!

Overview

TimeZero is a reasoning-guided Large Vision-Language Model (LVLM) for Temporal Video Grounding (TVG). It excels at identifying temporal segments within videos that correspond to a given natural language query. TimeZero achieves this entirely through a reinforcement learning approach that allows the model to reason about video-language relationships during inference.

Key Features:

Reinforcement Learning Training: TimeZero is trained entirely using reinforcement learning, enhancing its ability to generate accurate temporal boundaries.
Test-Time Reasoning: The model exhibits emergent reasoning capabilities during inference, generating a chain of thought to justify its segment predictions.
SOTA Performance: TimeZero sets a new SOTA on the Charades-STA benchmark.

This README provides an overview of TimeZero, including setup instructions, the training process, and evaluation guidelines.

Example:

Training Visualization:

Setup

bash

conda create -n timezero python=3.11
conda env create -f environment.yml
conda activate timezero

Training

TimeZero training involves the following steps:

Data Preprocessing:

Download the dataset Charades-STA, ActivityNet

Before training, you need to preprocess the video data.

bash preprocess_video.sh

Specify the path to the Charades-STA dataset (video files, annotations, etc.).
GRPO Training:

bash
1cd scripts 2bash run_grpo_video.sh

run_grpo_video.sh

bash
1#!/bin/bash 2 3export DEBUG_MODE="false" # Set to "true" for verbose logging during training. 4export LOG_PATH="./debug_log.txt" 5 6torchrun --nproc_per_node="4" \ 7--nnodes="1" \ 8--node_rank="0" \ 9--master_addr="127.0.0.1" \ 10--master_port="12361" \ 11src/open_r1/grpo_video.py \ 12--deepspeed scripts/zero3_offload.json \ 13--output_dir $OUTDIR \ 14--model_name_or_path mllm/Qwen2.5-VL-7B-Instruct \ 15--preprocessed_data_path ./Charades_preprocessed_data_maxpix_3584 \ 16--train_data_path ./Charades/charades_annotation/train.json \ 17--eval_data_path ./Charades/charades_annotation/val.json \ 18--video_folder ./Charades/Charades_v1 \ 19--dataset_name xxx \ 20--max_prompt_length 8192 \ 21--max_completion_length 1024 \ 22--num_generations 8 \ 23--per_device_train_batch_size 1 \ 24--gradient_accumulation_steps 2 \ 25--logging_steps 1 \ 26--bf16 \ 27--torch_dtype bfloat16 \ 28--data_seed 42 \ 29--gradient_checkpointing true \ 30--attn_implementation flash_attention_2 \ 31--num_train_epochs 2 \ 32--run_name $WANDB_NAME \ 33--report_to wandb \ 34--save_steps 50 \ 35--save_only_model true

Evaluation

After training, evaluate your model's performance:

bash scripts/evaluate.sh # Use evaluate.sh for evaluation.

evaluate.sh

python evaluate.py --model_base <path_to_your_trained_model> --dataset <charades or activitynet>

The evaluation script (evaluate.py) needs to be implemented to load your model, process the test data, and calculate the relevant metrics (R1@0.3, R1@0.5, R1@0.7, etc.).

Results

Charades-STA (Finetuned)

TimeZero outperforms previous state-of-the-art methods by a large margin.

Method	Type	R1@0.3	R1@0.5	R1@0.7
EaTR (VLP sota)	VLP	-	68.4	44.9
TimeSuite (LVLM sota)	SFT	79.4	67.1	43.0
TimeZero (ours)	RL	83.3	72.5	47.9

ActivityNet (Finetuned)

TimeZero surpasses previous state-of-the-art LVLMs.

Method	Type	R1@0.3	R1@0.5	R1@0.7
EaTR (VLP sota)	VLP	-	58.18	37.64
TRACE (LVLM sota)	SFT	54.0	37.7	24.0
TimeZero (ours)	RL	68.6	47.3	26.9

Acknowledgements

We thank the authors of the following projects for their contributions:

Citation

bibtex

@article{wang2025timezero,
  title={TimeZero: Temporal Video Grounding with Reasoning-Guided LVLM},
  author={Wang, Ye and Xu, Boshen and Yue, Zihao and Xiao, Zihan and Wang, Ziheng and Zhang, Liang and Yang, Dingyi and Wang, Wenxuan and Jin, Qin},
  booktitle={arxiv},
  year={2025}
}

wwwyyy/TimeZero-Charades-7B