Vjepa2 Vitl Fpc16 256 Ssv2 Uvf101

--- library_name: transformers datasets: - sayakpaul/ucf101-subset metrics: - accuracy base_model: - qubvel-hf/vjepa2-vitl-fpc16-256-ssv2 pipeline_tag: video-classification --- # Fine Tuned V-JEPA 2 on UCF101 Subset A frontier video understanding model developed by FAIR, Meta, which extends the pretraining objectives of [VJEPA](https://ai.meta.com/blog/v-jepa-yann-lecun-ai-model-video-joint-embedding-predictive-architecture/), resulting in state-of-the-art video understanding capabilities, leveraging data and model sizes at scale. The code is released [in this repository](https://github.com/facebookresearch/vjepa2). The base model we used is [vjepa2-vitl-fpc16-256-ssv2](https://hf.co/qubvel-hf/vjepa2-vitl-fpc16-256-ssv2), a V-JEPA 2 model pretrained on the <a href="https://paperswithcode.com/dataset/something-something-v2" style="color: black;">Something-Something-V2</a> dataset. We further fine-tuned this model on a subset of [UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild](https://huggingface.co/papers/1212.0402). This dataset contains just 400 short videos (in total) across 10 different categories. <img src="https://dl.fbaipublicfiles.com/vjepa2/vjepa2-pretrain.gif">&nbsp; ## Installation To run this V-JEPA 2 model, ensure you have installed the latest transformers: ```bash pip install -U git+https://github.com/huggingface/transformers ``` ## Video classification code snippet ```python import torch import numpy as np from torchcodec.decoders import VideoDecoder from transformers import AutoVideoProcessor, AutoModelForVideoClassification device = "cuda" if torch.cuda.is_available() else "cpu" # Load model and video preprocessor hf_repo = "facebook/vjepa2-vitg-fpc64-384-ssv2" model = AutoModelForVideoClassification.from_pretrained(hf_repo).to(device) processor = AutoVideoProcessor.from_pretrained(hf_repo) # To load a video, sample the number of frames according to the model. # For this model, we use 64. video_url = "https://huggingface.co/datasets/nateraw/kinetics-mini/resolve/main/val/bowling/-WH-lxmGJVY_000005_000015.mp4" vr = VideoDecoder(video_url) frame_idx = np.arange(0, model.config.frames_per_clip, 2) # you can define more complex sampling strategy video = vr.get_frames_at(indices=frame_idx).data # frames x channels x height x width # Preprocess and run inference inputs = processor(video, return_tensors="pt"...

Source context: 7 downloads · 0 likes · Pipeline video-classification · Library transformers · Repo ariG23498/vjepa2-vitl-fpc16-256-ssv2-uvf101