ComfyUI-SD-CN-Animation

ComfyUI-SD-CN-Animation

Temporally coherent video generation and stylization for ComfyUI using optical flow. Port of SD-CN-Animation to ComfyUI custom nodes.

Generate videos from text prompts (Txt2Vid) or stylize existing videos (Vid2Vid) with any Stable Diffusion checkpoint, ControlNet, and IPAdapter — while maintaining smooth frame-to-frame consistency via optical flow prediction.

Example output

How It Works

Workflow screenshot

Both pipelines use a two-pass approach per frame to balance creativity with coherence:

Txt2Vid uses the FloweR model to predict optical flow, occlusion, and the next frame from the previous 4 frames. Each frame is refined through two SD sampling passes — an inpaint pass guided by the occlusion mask, and a refinement pass for cleanup — with histogram matching to prevent color drift.

Vid2Vid uses RAFT optical flow estimation between consecutive input video frames to warp the previous stylized frame forward. Occlusion detection identifies areas where the warp breaks down, and those regions get inpainted by SD. A second refinement pass and histogram matching complete the frame.

Nodes

Node	Description
Load FloweR Model	Loads FloweR optical flow prediction model
SD-CN Animation Txt2Vid	Generate video from text prompt using FloweR
Load RAFT Model	Loads RAFT optical flow estimation model
SD-CN Animation Vid2Vid	Stylize video frames using RAFT optical flow
SD-CN Prompt Schedule	Keyframed prompt scheduling with interpolation
SD-CN Motion Control	Control flow, zoom, pan, and rotation in Txt2Vid
FloweR Predict	Standalone FloweR prediction (debug/visualization)
Histogram Match	Standalone color histogram matching utility

Installation

Install via ComfyUI Manager or clone manually into ComfyUI/custom_nodes/.

Model weights (~8MB FloweR + ~20MB RAFT) are downloaded automatically from Hugging Face on first launch.

Workflows

Txt2Vid (Text to Video)

CheckpointLoader ──→ model ──────────────────→ SD-CN Animation Txt2Vid ──→ VHS_VideoCombine
                 ──→ clip ──→ CLIPTextEncode ──→ positive                   (set fps: 12)
                 ──→ vae  ──────────────────→
LoadFloweRModel ──→ flower_model ───────────→

Key parameters:

• num_frames — total frames to generate (30-120+)
• processing_strength — how much SD changes each frame (0.5-0.85). Lower = smoother morphing, higher = more creative variation
• fix_frame_strength — refinement pass strength (0.1-0.2). Keep low for stability, set to 0 to skip entirely
• loop_frames — set to 10-20 to blend the last N frames back toward the first frame for a seamless loop. 0 = disabled
• color_match (optional) — histogram reference for color stability: first_frame (default, most stable), previous_frame (lets colors evolve), or none

For smooth morphing animation: Use processing_strength: 0.55, fix_frame_strength: 0.12, num_frames: 90, and set your video combine node to 12-14 fps.

Vid2Vid (Video Stylization)

VHS_LoadVideo ──→ frames ───────────────────→ SD-CN Animation Vid2Vid ──→ VHS_VideoCombine
CheckpointLoader ──→ model ─────────────────→
                 ──→ clip ──→ CLIPTextEncode → positive
                 ──→ vae  ──────────────────→
LoadRAFTModel ──→ raft_model ───────────────→

Key parameters:

• processing_strength — denoising for main stylization (0.75-0.85)
• fix_frame_strength — refinement pass (0.1-0.2, or 0 to skip)
• blend_alpha — blend between warped styled frame (0.0) and current input (1.0). Higher values keep more of the original video structure
• occlusion_mask_blur — smooths the mask that decides what gets regenerated (2-4)