MotionVideoSearch

MotionVideoSearch

MotionVideoSearch is a Python project for extracting motion-based embeddings from videos and storing them in a vector database (FAISS). You can then quickly search those embeddings with an input image, retrieving matching segments of the indexed videos. It offers:

• A command-line interface (CLI) built with Typer.
• Integration with PyTorch for embedding extraction.
• Video sampling, embedding, and database storage with SQLite + FAISS.
• Optional watermark detection and removal (via an external repo).
• Utility nodes for ComfyUI to create and search motion images within ComfyUI’s node graph environment.

Features

• Video Ingestion
  Slice videos into small segments, generate motion-based embeddings for each segment, and store them in FAISS for fast similarity search.

• Search
  Query the index with a single motion image to find relevant video segments, returning top matches ranked by distance in vector space.

• Concurrency Support
  Organize indexing jobs in different subdirectories, then merge (combine) them later into one big index.

• Watermark Removal (Optional)
  If you clone l-comm/WatermarkRemoval, you can automatically detect and remove watermarks from video frames during ingestion.

• ComfyUI Nodes
  Includes two custom nodes:

  1. IG_MotionVideoSearch: Takes an image input and returns the top 5 ranked URLs from the FAISS index.
  2. IG_MotionVideoFrame: Converts 24 consecutive frames into a single “dot frame,” which encodes motion features in a color-coded image.

ComfyUI

We have some ComfyUI nodes ready to be used with a pre-existing video database consisting of roughly 100,000 videos. You can get started using the following workflow: MotionSearch

MotionSearch

So for example, this video

https://github.com/user-attachments/assets/3195c170-e080-498f-be09-a06db53d1a73

Will give the following search results

https://github.com/user-attachments/assets/b793af91-882c-44ea-8dc7-c051a73f74b2

https://github.com/user-attachments/assets/ca9008aa-281e-47c5-aff9-05ebce8fe43a

https://github.com/user-attachments/assets/dc7fd9da-1e62-49e1-98cf-9acd64f025bf

Installation

1. Set Up Environment

Clone the repository:

git clone https://github.com/IDGallagher/MotionVideoSearch
cd MotionVideoSearch

(Optional) Create a conda environment:

conda create -n mvs python=3.9
conda activate mvs

2. Install Dependencies

Install PyTorch and TorchVision compatible with your CUDA setup (example below uses CUDA 11.8):

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

Install ffmpeg 6.x:

conda install -c conda-forge ffmpeg=6.*

Install other dependencies:

pip install tqdm matplotlib einops einshape scipy timm lmdb av mediapy typer imageio imageio-ffmpeg requests opencv-python

Install Faiss (version example below):

conda install -c pytorch faiss-cpu=1.9.0

(Optional) Download the watermark removal repo if you want that functionality:

git clone https://github.com/l-comm/WatermarkRemoval.git

Usage

After installing, you can run the CLI commands via `python main.py [COMMAND] [OPTIONS]`.

1. Storing Embeddings

Use the `store` command to process videos and store their embeddings in the FAISS index:

python main.py store \
  --dir /path/to/videos \
  --max-time 1.0 \
  --debug

• `--dir`: Directory with video files (e.g., `.mp4`, `.avi`, etc.).
• `--max-time`: How many seconds to process from each video (default 1 second).
• `--debug`: Enable debug mode to save intermediate frames and videos in a `deb