You can deploy this model and start earning money today!

fztkm/
lc_video_description_videomae_gpt2

Name: lc_video_description_videomae_gpt2
Availability: InStock

transformers

safetensors

length_control_videomae_gpt2

feature-extraction

custom_code

Views

No views yet

Length Controllable Video Description Model

This was introduced in the paper Fine-grained length controllable video captioning with ordinal embeddings, IEEE Access, Vol.12, pp. 189667-189688, 2024 by Nitta et al. It enables fine-grained control over the length of generated video captions, a novel feature achieved using ordinal embeddings. An online demo of the model is available at Hugging Face Spaces.

Model Details

Model Description

This model uses VideoMAE as the encoder and GPT2 as the decoder, with ordinal embeddings designed to manage caption lengths effectively. The training data is sourced from the ActivityNet Captions dataset. The model facilitates both token-length control and temporal duration control for captions, making it a powerful tool for applications such as video narration and storytelling.

Model type: Encoder-Decoder with autoregressive decoding
Languages: English
Fine-tuned from: VideoMAE (ViT-B) and GPT2

Training Details

Datasets: ActivityNet Captions
Encoder: VideoMAE (pretrained on Kinetics-400)
Decoder: GPT2 (small version, pretrained on WebText)
Length embeddings: Ordinal embeddings defined in the paper
Maximum length: 256 tokens
Optimizations: AdamW optimizer with learning rate 1e-4 and weight decay 5e-4
Frame size: 16 frames
Clip duration: 1.5 seconds

How to Use

You can load and use the model as follows:

python

from transformers import AutoModel, AutoTokenizer

def transform(video_pixel):
    transform_list = [
        transforms.Lambda(lambda x: x / 255.),
        Normalize(
            [0.485, 0.456, 0.406],
            [0.229, 0.224, 0.225]),
        ShortSideScale(256),
        CenterCrop(224),
    ]
    transform = Compose(transform_list)
    return transform(video_pixel).transpose(0, 1)

def captioning(video_tensor):

  video = transform(video)

  model = AutoModel.from_pretrained("fztkm/lc_video_description_videomae_gpt2", trust_remote_code=True).requires_grad_(False)
  tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2")
  
  output_tokens = model.generate(pixel_values=video_tensor, desired_length=desired_length, max_new_tokens=100)
  caption = tokenizer.batch_decode(output_tokens, skip_special_tokens=True)

  print(caption)

Citation

BibTeX:

bash

@ARTICLE{Nitta2024_length_control_captioning,
  author={Nitta, Tomoya and Fukuzawa, Takumi and Tamaki, Toru},
  journal={IEEE Access}, 
  title={Fine-grained length controllable video captioning with ordinal embeddings}, 
  year={2024},
  doi={10.1109/ACCESS.2024.3506751}
}