Skip to content

OpenGVLab/video-mamba-suite

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

50 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Introduction

teaser Understanding videos is one of the fundamental directions in computer vision research, with extensive efforts dedicated to exploring various architectures such as RNN, 3D CNN, and Transformers. The newly proposed architecture of state space model, e.g, Mamba, shows promising traits to extend its success in long sequence modeling to video modeling. To assess whether Mamba can be a viable alternative to Transformers in the video understanding domain, in this work, we conduct a comprehensive set of studies, probing different roles Mamba can play in modeling videos, while investigating diverse tasks where Mamba could exhibit superiority. We categorize Mamba into four roles for modeling videos, deriving a Video Mamba Suite composed of 14 models/modules, and evaluating them on 12 video understanding tasks. Our extensive experiments reveal the strong potential of Mamba on both video-only and video-language tasks while showing promising efficiency-performance trade-offs. We hope this work could provide valuable data points and insights for future research on video understanding.

📢 News

(2024/04/01) The code of related tasks for Video Temporal Adapter and Spatial-temporal modeling has been released. They contain multi-instance retrieval, action recognition, long video quesion-answer on EK100, Kinetics400, and EgoSchema. If there exist bug or missing packages, please submit a new issue to tell me.

(2024/03/21) The code of related tasks for Video Temporal Modeling and Cross-modal Interaction has been released.

(2024/03/15) 🔄The repository is public.

(2024/03/12) 🔄The repository is created.

Preliminary Installation

  1. Install the preliminary requirements.
# clone video-mamba-suite
git clone --recursive https://github.com/OpenGVLab/video-mamba-suite.git

# create environment
conda create -n video-mamba-suite python=3.9
conda activate video-mamba-suite

# install pytorch
pip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu118

# install requirements
pip install -r requirement.txt

# install mamba
cd causal-conv1d
python setup.py develop
cd ..
cd mamba
python setup.py develop
cd ..
  1. For each tasks, enter their folders to follow installation instructions.

  2. If requirement.txt is missing some libraries, please propose an issue as soon as possible.

Supported tasks

Mamba for Video Temporal Modeling

task supported datasets
Temporal Action Localization ActivityNet, HACS Segment, FineAction, THUMOS-14
Temporal Action Segmentation GTEA, Breakfast, 50salads
Video dense captioning ActivityNet, YouCook2
Video paragraph captioning ActivityNet, YouCook2
Action Anticipation Epic-kitchen-100

Mamba for Cross-modal Interaction

task supported datasets
Video Temporal Grounding QvHighlight, Charade-STA
Highlight Detection QvHighlight

Mamba as Video Temporal Adapter

task supported datasets
Zero-shot/Finetuned Multi-instance Retrieval Epic-kitchen-100
Finetuned Action Recognition Epic-kitchen-100
Long-form Video Question-Answer EgoSchema

Mamba for Spatial-temporal Modeling

task supported datasets
Zero-shot/Finetuned Multi-instance Retrieval Epic-kitchen-100
Finetuned Action Recognition Kinetics-400

Model Zoo

The models of Mamba as Video Temporal Adapter and Mamba for Spatial-temporal Modeling.

Model Pretrain-I Pretrain-V Finetuned #Frame Weight
TimeSformer-B (Vanilla) CLIP-400M Ego4D-4M - 4 🤗 HF link
TimeSformer-B (Frozen) CLIP-400M Ego4D-4M - 4 🤗 HF link
TimeMamba-B (Vanilla) CLIP-400M Ego4D-4M - 4 🤗 HF link
TimeMamba-B (Frozen) CLIP-400M Ego4D-4M - 4 🤗 HF link
TimeSformer-B (Vanilla) CLIP-400M Ego4D-4M EK100-CLS 4->16 🤗 HF link
TimeSformer-B (Vanilla) CLIP-400M Ego4D-4M EK100-MIR 4->16 🤗 HF link
TimeMamba-B (Vanilla) CLIP-400M Ego4D-4M EK100-CLS 4->16 🤗 HF link
TimeMamba-B (Vanilla) CLIP-400M Ego4D-4M EK100-MIR 4->16 🤗 HF link
ViT-T deit,IN1K Ego4D-4M - 16 🤗 HF link
ViT-S deit,IN1K Ego4D-4M - 16 🤗 HF link
ViViM-T deit,IN1K Ego4D-4M - 16 🤗 HF link
ViViM-S deit,IN1K Ego4D-4M - 16 🤗 HF link
ViViM-T deit,IN1K - K400 16x3x4 🤗 HF link
ViViM-S deit,IN1K - K400 16x3x4 🤗 HF link

Cite

If you find this repository useful, please use the following BibTeX entry for citation.

@misc{2024videomambasuite,
      title={Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding}, 
      author={Guo Chen, Yifei Huang, Jilan Xu, Baoqi Pei, Zhe Chen, Zhiqi Li, Jiahao Wang, Kunchang Li, Tong Lu, Limin Wang},
      year={2024},
      eprint={2403.09626},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

License

This project is released under the MIT License

Acknowledgement

This repository is built based on ActionFormer, UniVTG, ASFormer, PDVC, Testra, MAT, AVION, InternVideo, EgoSchema, ViM and Mamba repository.

About

The suite of modeling video with Mamba

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published