--- library_name: keras-hub --- ### Model Overview # Model Summary VideoPrism is a family of foundational video-encoder models from Google Research, designed to be a universal "prism" for understanding the diverse facets of video content. Built on a massive scale of 36 million high-quality video-caption pairs and 582 million video clips, VideoPrism is engineered to excel across a wide range of video understanding tasks, including classification, localization, retrieval, and captioning. VideoPrism models utilize a Vision Transformer (ViT) architecture and are pre-trained using a combination of video-text contrastive learning and masked video modeling. This dual approach allows the model to capture both global semantic meaning and fine-grained spatio-temporal details, making it a powerful backbone for state-of-the-art video AI applications. ## Links * [VideoPrism Technical Paper](https://arxiv.org/abs/2402.13250) * [VideoPrism API Documentation](https://keras.io/api/keras_hub/models/videoprism/) * [VideoPrism on Hugging Face](https://huggingface.co/google/videoprism-base-f16r288) * [KerasHub Beginner Guide](https://keras.io/guides/keras_hub/getting_started/) * [KerasHub Model Publishing Guide](https://keras.io/guides/keras_hub/upload/) ## Installation Keras and KerasHub can be installed with: ``` pip install -U -q keras-hub pip install -U -q keras>=3 ``` JAX, TensorFlow, and Torch come pre-installed in Kaggle Notebooks. For instructions on installing them in another environment, see the [Keras Getting Started](https://keras.io/getting_started/) page. ## Presets The following model checkpoints are provided by the Keras team. For the Video-Text (LvT) variants, both the video encoder and the text encoder are provided to enable multimodal tasks like zero-shot retrieval. | Preset name | Parameters | Description | |---------------------------------|------------|--------------------------------------------------------------------------------------------------------------------------------------------------| | `videoprism_public_v1_base` | 114.00M | 114 million parameter, 12-layer ViT-B, 16-frame, 288x288 resolution, video-only encoder for spatio-temporal representation. | | `videoprism_public_v1_large` | 354.00M | 354 million parameter, 24-layer ViT-L, 16-frame, 288x288 resolution, video-only encoder for spatio-temporal representation. | | `videoprism_lvt_public_v1_base` | 248.00M | 248 million parameter, 12-layer ViT-B video encoder + text encoder, 16-frame, 288x288 resolution, for multimodal video-language tasks. | | `videoprism_lvt_public_v1_large`| 580.00M | 580 million parameter, 24-layer ViT-L video encoder + text encoder, 16-frame, 288x288 resolution, for multimodal video-language tasks. |
