ComfyUI-AI-VideoAnalysis

ComfyUI Video Analysis (Gemini Flash)

What it is

A minimal ComfyUI workflow that turns a local video into a clear, human-readable analysis by sampling video frames and sending them to Gemini Flash (multimodal Video Language Model).

You get a practical “what is happening in this clip?” description without building a separate app, backend, or UI.

How it works

Pipeline (in ComfyUI):

VHS_LoadVideo → frame sampling → Gemini Flash (video) → ShowText

1. Load video from ComfyUI/input/ using VideoHelperSuite (VHS_LoadVideo).
2. Sample frames (more than 6; configurable with frame_load_cap and select_every_nth).
3. Send frames to Gemini with a task prompt (e.g., “Analyze the situation in details.”).
4. Display the analysis inside ComfyUI using a text viewer node.

Why it matters

This pattern is a reusable building block for real systems:

• Anomaly detection + temporal segmentation (next step): extend from whole-clip summaries to time-windowed analysis (e.g., 15-second chunks) and export per-segment JSON with timestamps.
• Training & compliance: convert video into structured summaries for healthcare, industrial, and other professional training workflows (SOP adherence, safety audits, incident reviews).
• Video analytics systems engineering: a concrete, end-to-end example of video ingestion → sampling → multimodal reasoning → outputs, ready to evolve into scalable pipelines (batch jobs, caching, evaluation, monitoring).

Scope:

• ✅ Load video
• ✅ Sample more than 6 frames (configurable)
• ✅ Send frames to Gemini and returns understanding of the video frames in text output
• ✅ Show understanding of the video per segment in ComfyUI
• ✅ Show results in ComfyUI

---

What you get

A ComfyUI workflow like:

VHS_LoadVideo → Gemini Flash (video) → Shows understanding of the video in text

Example output:

The image frames extracted from the video show someone preparing raw chicken wings… liquid poured… diced garlic added…

---

Requirements

• ComfyUI Desktop (tested with ComfyUI 0.9.1 on macOS)
• VideoHelperSuite (VHS) for VHS_LoadVideo
• Gemini Flash custom node (the Gemini repo node)
• A Gemini API key (Google AI Studio / Gemini API)

---

Install

1) Install the custom nodes

Put these folders into:

ComfyUI/custom_nodes/

• comfyui-videohelpersuite
• ComfyUI-Gemini_Flash_2.0_Exp (or your fork)

Restart ComfyUI.

If you are using ComfyUI-Manager, install both from Manager and restart.

2) Install Python dependencies (must be in ComfyUI’s venv)

Use the same Python executable ComfyUI runs, e.g.:

/Users/<you>/Documents/ComfyUI/.venv/bin/python -m pip install -U google-genai google-generativeai pillow

---

API Key Configuration

Two valid approaches:

Option A — config.json (recommended)

Create config.json inside the Gemini node folder:

{
  "GEMINI_API_KEY": "YOUR_KEY_HERE"
}

Option B — enter key in the node UI

Some setups are more reliable when the key is entered directly in the node.

---

Workflow Setup

1) Put your video in ComfyUI input directory

Copy video into:

ComfyUI/input/

2) Load the workflow JSON

Load from:

workflows/gemini_video_analysis.json

3) Configure VHS_LoadVideo (important)

Key settings to avoid typical VHS errors and improve coherence:

• force_rate: 8 (do not leave empty / null)
• custom_width/custom_height: 512 (or 640)
• frame_load_cap: 24–48 (increase frames vs default “6” behavior)
• select_every_nth: 1–4 (controls sampling density)
• skip_first_frames: optional (e.g., 0–10)

Recommended presets:

• Short clips (10–30s): frame_load_cap=24, select_every_nth=1
• Longer clips (60–120s): frame_load_cap=32–48, select_every_nth=2–4

4) Configure Gemini Node

• input_type: video
• prompt example:
  • Analyze the situation in details.
• set `ma