Wan2.2 Animate + SAM3 - 1 minute long videos — v1.1

Author notes from Wan22AnimateInfinite.json · selected release

Use this group as a pose estimator backup if the main group fails

Author notes from Wan22AnimateInfinite.json · selected release

Use the **positive prompt** to briefly describe the video, focus on mentioning things that are not clearly visible in your reference image, for example, include a description of the back of your character if it turns around at some point in the video, or mention the eye color if your character image is so small that the eye color is not clearly visible. Prompt example: A beautiful woman with green eyes is dancing for a tiktok video. She is wearing an straples croptop with a pink ribbon on her lower back; her hair styled in a messy low bun. You can leave the **negative** prompt as is, it includes keywords to exclude "AI look" and bad quality.

Author notes from Wan22AnimateInfinite.json · selected release

Here is where you select your guiding video, the video with the motion you want to copy or the character you want to swap. The ImageScale node resizes the reference video to a 720p resolution, this ensures a more accurate pose generation.

Author notes from Wan22AnimateInfinite.json · selected release

Set the "fill_holes" field to true for face swap, or if you want to "patch holes" in the maks.

Author notes from Wan22AnimateInfinite.json · selected release

A color correction node is included to give a more realistic touch (low contrast) to the final video.

Author notes from Wan22AnimateInfinite.json · selected release

Default: Expand: 20 Block size: 30 You can play with the expand value if the character mask is not big enough to fit your characer, but don't over do it.

Author notes from Wan22AnimateInfinite.json · selected release

Use the "Fast group bypasser" node from the right to navigate and enable/disable groups. Default settings and groups: - Set your video settings in the "Video settings group".<br> - Load your guiding/reference video in the "Reference video" group.<br> - Load your character image in the "CHARACTER REFERENCE IMAGE" group.<br> - Write a prompt following the guidelines that you can find in the "PROMPT" group.<br> - If your guiding video has more than one characters, enable the "Character isolation mask" group and isolate the character you want to extract the movement from or the one you want to swap; otherwise disable the "Character isolation mask" group.<br> - Enable the "Pose estimation" group to extract the character pose data. In extremely rare occasions, this group throws an error, you can then disable it and use the "Pose estimation alt" group instead (use one or the other); the alt group is less accurate but still works well.<br> - Enable the "Final video P1" group. Enable the "Final video chunks" group too if your video is longer than your batch size; this group is a loop that generates chunks of video until it generates the full video. If you are not sure, you can activate both groups, it won't affect the final result.<br> - Enable the color correction node if you want to adjust brightness, saturation and contrast. By default it sets the contrast to 0.9 to give the final result a more realistic feeling. **Wan animate works in two modes:** **Character animation** (you want to animate your character image using the guiding video movement) - This is the default mode, so you only need to activate the default groups previously explained. --- **Character swap / face swap** (you want to swap one of the characters of your reference video for the character of your reference image, preserving the original reference video background or scene). Aside from the default groups: 1. Make sure the "CHARACTER ISOLATION MASK" group is enabled, even if you are using a guider video with only one character, this is because we need the mask to tell the model where to edit the reference video. 2. Enable the "KEEP REFERENCE VIDEO BACKGROUND" group; this group will generate the background video the model needs. ## Tips: If you need to isolate a character from your guiding/reference video, disable groups 6, 7, 9 and 10 until you are happy with the isolation mask, then keep going activating group by group to control each step of the generation. Set the seed to "fixed" in the "Final video P1" group to fix the generation cache.

Author notes from Wan22AnimateInfinite.json · selected release

Infinite video generation using chunks. Sample config: WITH lighting LoRA: 6 steps, CFG of 1. WITHOUT lighting LoRA: 50 steps, CFG of 1.

Author notes from Wan22AnimateInfinite.json · selected release

The character you want to animate using the guiding video. Try to use an image as close to the first position of your guiding video as possible. Wan animate is really good at re-adjusting so this is not strictly mandatory but, overall, the closest, the better. Optionally, you can change the size of the image in the resize image node. The size of this image doesn't affect the final video resolution, so you can leave it at 1920 regardless of your video resolution.

Author notes from Wan22AnimateInfinite.json · selected release

Use this group to isolate a single character from your guiding video, or to select the character you want to swap. This character will be later used to generate the pose and face expression video. The isolation process is divided in two, so your GPU doesn't explode processing long videos. We use Meta's SAM3 for high quality object isolation. It requires Pytorch, CUDA and Triton to work, and some extra Python libraries for the ComfyUI portable version, you can find those libraries and a complete guide in: [Triton for windows](https://github.com/woct0rdho/triton-windows?tab=readme-ov-file#8-special-notes-for-comfyui-with-embeded-python) Use the "CHARACTER ISOLATION DESCRIPTION" text box to shortly describe the character you want to isolate or use "Face" for face swap. Examples: - Woman - Dancer - Man - Face If your guiding video happens to have for example, 3 women, or 3 dancers, their index will appear in the "MASKS PREVIEW" video; you can then select a single one by selecting its index in the "SAM3 Segmentation" nodes, in the object_ids field. If for example, you want to select character 0, you write 0 on that field. If you leave it blank, it will select all the objects/characters matching your "CHARACTER ISOLATION DESCRIPTION".

Author notes from Wan22AnimateInfinite.json · selected release

First chunk of video generation. Sample config: WITH lighting LoRA: 6 steps, CFG of 1. WITHOUT lighting LoRA: 50 steps, CFG of 1.

Author notes from Wan22AnimateInfinite.json · selected release

Change the seed here, or fix it if you need to.

Author notes from Wan22AnimateInfinite.json · selected release

- **Main model:** [Wan2_2-Animate-14B_fp8_scaled_e4m3fn_KJ_v2.safetensors](https://huggingface.co/Kijai/WanVideo_comfy_fp8_scaled/tree/main/Wan22Animate) - **Text encoder:** [umt5_xxl_fp8_e4m3fn_scaled.safetensors](https://huggingface.co/Comfy-Org/Wan_2.2_ComfyUI_Repackaged/tree/main/split_files/text_encoders) - **VAE:** [wan_2.1_vae.safetensors](https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/tree/main/split_files/vae) This workflow uses the KJ model and CLIP vision, so we use Wan2.1 VAE instead of 2.2 - **CLIP VISION:** [clip_vision_h.safetensors](https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/tree/main/split_files/clip_vision) - **LoRAs:**<br> Speed LoRA: [lightx2v_I2V_14B_480p_cfg_step_distill_rank64_bf16.safetensors](https://huggingface.co/Kijai/WanVideo_comfy/tree/709844db75d2e15582cf204e9a0b5e12b23a35dd/Lightx2v)<br> Religh LoRA: [WanAnimate_relight_lora_fp16.safetensors](https://huggingface.co/Kijai/WanVideo_comfy/tree/main/LoRAs/Wan22_relight) - **SAM3:** [sam3.pt](https://huggingface.co/facebook/sam3/tree/main). See the instructions in the "CHARACTER ISOLATION MASK" group for further information. At first you need to apply for approval to download the model, but don't worry, it's just a formality, you will get approved in minutes<br> "Wan cache optimizer" (CacheDIT nodes) is optional, it speeds up subsequent generations. **File structure:** ``` 📂 ComfyUI/ ├── 📂 models/ │ ├── 📂 diffusion_models/ │ │ └── Wan2_2-Animate-14B_fp8_scaled_e4m3fn_KJ_v2.safetensors │ ├── 📂 text_encoders/ │ │ └── umt5_xxl_fp8_e4m3fn_scaled.safetensors │ ├── 📂 vae/ │ │ └── wan_2.1_vae.safetensors │ ├── 📂 clip_vision/ │ │ └── clip_vision_h.safetensors │ ├── 📂 loras/ │ │ └── <YOUR LORAS GO HERE> │ └── 📂 sam3/ │ └── sam3.pt ```

Author notes from Wan22AnimateInfinite.json · selected release

- **Image size selector:** Here you select your final video resolution and orientation (landscape or portrait mode). 480p is good for an RTX 4090. Higher values will require more VRAM and time. A 4090 can handle 720p at a batch size of 160, but it takes longer to render, it also can handle 20 second videos at 1080p with a batch size of 80, though it takes even longer to render.<br> - **Framerate:** Wan animate physics work at 16 FPS, so don't change it unless you want to try crazy stuff like slow motion or super speed animation.<br> - **Max video duration:** Limits the final video duration, you can leave it high, reduce it if for example you want to only use a portion of your guiding video. If your guiding video is shorter than this limit, it will not affect anything at all, so that's why you can leave it high.<br> - **Batch size (in frames):** change it for what your GPU can handle, 160 frames (10 seconds) is good for an RTX 4090. The complete video is divided in chunks (the batch size), so if your graphics card can handle only 5 second videos for example, change the batch size to 80, this will divide the video generation in 5 second chunks.<br> - **Skip first n seconds:** use it to skip some part of the video (from the start)<br> - **Activate tile decoding:** This node toggles tile decoding mode. When it is set to true, a tile decoding node is activated instead of the default VAE decode node in the video decoding stage. Use this if you are experimenting long decoding times or hangs, otherwise keep it set to false.

Author notes from Wan22AnimateInfinite.json · selected release

You can delete this SystemNotification node, it is not relevant at all. It sends a system notification via your browser when the workflow finish execution; it requires an SSL connection for non localhost setups.

Author notes from Wan22AnimateInfinite.json · selected release

Use relight LoRA for character swap, it helps to blend your character naturally with the source video. For full animation you don't need it.