Vjepa2 Vitg Fpc32 384 Diving48

--- license: mit pipeline_tag: video-classification tags: - video library_name: transformers datasets: - bkprocovid19/diving48 base_model: - facebook/vjepa2-vitg-fpc64-384 --- # V-JEPA 2 A frontier video understanding model developed by FAIR, Meta, which extends the pretraining objectives of [VJEPA](https://ai.meta.com/blog/v-jepa-yann-lecun-ai-model-video-joint-embedding-predictive-architecture/), resulting in state-of-the-art video understanding capabilities, leveraging data and model sizes at scale. The code is released [in this repository](https://github.com/facebookresearch/vjepa2). <div style="background-color: rgba(251, 255, 120, 0.4); padding: 10px; color: black; border-radius: 5px; box-shadow: 0 4px 8px rgba(0,0,0,0.1);"> 💡 This is V-JEPA 2 model with video classification head pretrained on <a href="http://www.svcl.ucsd.edu/projects/resound/dataset.html" style="color: black;">Diving 48</a> dataset. </div> <br></br> <img src="https://dl.fbaipublicfiles.com/vjepa2/vjepa2-pretrain.gif">&nbsp; ## Installation To run V-JEPA 2 model, ensure you have installed the latest transformers: ```bash pip install -U git+https://github.com/huggingface/transformers ``` ## Video classification code snippet ```python import torch import numpy as np from torchcodec.decoders import VideoDecoder from transformers import AutoVideoProcessor, AutoModelForVideoClassification device = "cuda" if torch.cuda.is_available() else "cpu" # Load model and video preprocessor hf_repo = "facebook/vjepa2-vitg-fpc32-384-diving48" model = AutoModelForVideoClassification.from_pretrained(hf_repo).to(device) processor = AutoVideoProcessor.from_pretrained(hf_repo) # To load a video, sample the number of frames according to the model. video_url = "https://huggingface.co/facebook/vjepa2-vitg-fpc32-384-diving48/resolve/main/sample/diving.mp4" vr = VideoDecoder(video_url) frame_idx = np.arange(0, model.config.frames_per_clip, 8) # you can define more complex sampling strategy video = vr.get_frames_at(indices=frame_idx).data # frames x channels x height x width # Preprocess and run inference inputs = processor(video, return_tensors="pt").to(model.device) with torch.no_grad(): outputs = model(**inputs) logits = outputs.logits print("Top 5 predicted class names:") top5_indices = logits.topk(5).indices[0] top5_probs = torch.softmax(logits, dim=-1).topk(5).values[0] for idx, prob i...

Source context: 4 downloads · 0 likes · Pipeline video-classification · Library transformers · Repo qubvel-hf/vjepa2-vitg-fpc32-384-diving48