Capybara: Video Edit
Upload an image or video and a text instruction to perform generation or editing tasks. Generate a new or edited image or video based on the provided instruction.
About
The Capybara: Video Edit workflow is a powerful tool designed for users looking to perform video editing and generation tasks using AI. At its core, this workflow leverages the Capybara model, which is adept at interpreting text instructions to modify or create video content. The workflow begins with the LoadVideo node, which allows users to upload their desired video or image file.
Once the media is loaded, users can input specific text instructions that dictate the editing or generation task they wish to perform. The Capybara model processes these instructions to produce the desired output, which can range from simple edits to entirely new video sequences.
Technically, the workflow is structured to facilitate seamless integration between user inputs and AI-driven video processing. The MarkdownNote node serves as a guide, providing users with essential information and tips for optimal usage. After processing, the SaveVideo node ensures that the output is stored efficiently, ready for review or further use. This workflow is particularly useful for content creators, marketers, and educators who need to quickly generate or edit video content based on specific requirements, without the need for extensive manual editing.
What you provide
- Reading the workflow
Runs in ComfyUI. This workflow can't run on this page. Open it in Comfy Cloud with everything loaded, or download it for your own ComfyUI.
Try in CloudInside the workflow
A read-only preview of the node graph. Open it in Comfy Cloud or download it to edit the nodes.
Loading the graph
Open full-size workflow preview- Author
- ComfyUI
Requires custom nodes
- Reading the workflow
Use this workflow from code
Generated from the workflow graph and inputs when possible; stored examples are shown otherwise. Review credentials and endpoints before running code, which may incur charges.
What you need
- Runs on
- Comfy Cloud
- Key
- COMFY_API_KEY
Needs a paid Cloud plan and credits.
# Python 3.10+: pip install comfy-sdk==0.4.0
import os
from comfy_sdk import Comfy
api_key = os.environ.get("COMFY_API_KEY", "").strip()
if not api_key: raise RuntimeError("Set COMFY_API_KEY")
os.environ.setdefault("COMFY_BASE_URL", "https://cloud.comfy.org")
client = Comfy(api_key=api_key)
workflow = client.workflows.from_json({
"114": {
"class_type": "SaveVideo",
"inputs": {
"filename_prefix": "video/ComfyUI",
"format": "auto",
"format.codec": "auto",
"video": [
"103:113",
0
]
},
"_meta": {
"title": "Save Video"
}
},
"127": {
"class_type": "LoadVideo",
"inputs": {
"file": "horse_running.mp4"
},
"_meta": {
"title": "Load Video"
}
},
"103:10": {
"class_type": "VAELoader",
"inputs": {
"vae_name": "hunyuanvideo15_vae_fp16.safetensors"
},
"_meta": {
"title": "Load VAE"
}
},
"103:11": {
"class_type": "DualCLIPLoader",
"inputs": {
"clip_name1": "qwen_2.5_vl_7b.safetensors",
"clip_name2": "byt5_small_glyphxl_fp16.safetensors",
"device": "default",
"type": "hunyuan_video_15"
},
"_meta": {
"title": "Load CLIP (Dual)"
}
},
"103:113": {
"class_type": "CreateVideo",
"inputs": {
"audio": [
"103:128",
1
],
"bit_depth": "auto",
"codec": "none",
"color_space": "sRGB",
"fps": 24,
"images": [
"103:8",
0
]
},
"_meta": {
"title": "Create Video"
}
},
"103:115": {
"class_type": "HunyuanVideo15ImageToVideo",
"inputs": {
"batch_size": 1,
"clip_vision_output": [
"103:119",
0
],
"height": [
"103:122",
0
],
"length": 121,
"negative": [
"103:93",
0
],
"positive": [
"103:44",
0
],
"start_image": [
"103:131",
0
],
"vae": [
"103:10",
0
],
"width": [
"103:121",
0
]
},
"_meta": {
"title": "HunyuanVideo15ImageToVideo"
}
},
"103:117": {
"class_type": "CLIPVisionLoader",
"inputs": {
"clip_name": "sigclip_vision_patch14_384.safetensors"
},
"_meta": {
"title": "Load CLIP Vision"
}
},
"103:119": {
"class_type": "CLIPVisionEncode",
"inputs": {
"clip_vision": [
"103:117",
0
],
"crop": "center",
"image": [
"103:131",
0
]
},
"_meta": {
"title": "CLIP Vision Encode"
}
},
"103:12": {
"class_type": "UNETLoader",
"inputs": {
"unet_name": "capybara_v0.1.safetensors",
"weight_dtype": "default"
},
"_meta": {
"title": "Load Diffusion Model"
}
},
"103:121": {
"class_type": "PrimitiveInt",
"inputs": {
"value": 960
},
"_meta": {
"title": "Width"
}
},
"103:122": {
"class_type": "PrimitiveInt",
"inputs": {
"value": 960
},
"_meta": {
"title": "Height"
}
},
"103:128": {
"class_type": "GetVideoComponents",
"inputs": {
"video": [
"127",
0
]
},
"_meta": {
"title": "Get Video Components"
}
},
"103:129": {
"class_type": "ImageFromBatch",
"inputs": {
"batch_index": 0,
"image": [
"103:128",
0
],
"length": 121
},
"_meta": {
"title": "Get Image from Batch"
}
},
"103:13": {
"class_type": "SamplerCustomAdvanced",
"inputs": {
"guider": [
"103:91",
0
],
"latent_image": [
"103:115",
2
],
"noise": [
"103:25",
0
],
"sampler": [
"103:16",
0
],
"sigmas": [
"103:17",
0
]
},
"_meta": {
"title": "SamplerCustomAdvanced"
}
},
"103:131": {
"class_type": "ResizeImageMaskNode",
"inputs": {
"input": [
"103:129",
0
],
"resize_type": "scale dimensions",
"resize_type.crop": "center",
"resize_type.height": [
"103:122",
0
],
"resize_type.width": [
"103:121",
0
],
"scale_method": "lanczos"
},
"_meta": {
"title": "Resize Image/Mask"
}
},
"103:16": {
"class_type": "KSamplerSelect",
"inputs": {
"sampler_name": "euler"
},
"_meta": {
"title": "KSamplerSelect"
}
},
"103:17": {
"class_type": "BasicScheduler",
"inputs": {
"denoise": 1,
"model": [
"103:12",
0
],
"scheduler": "simple",
"steps": 20
},
"_meta": {
"title": "BasicScheduler"
}
},
"103:25": {
"class_type": "RandomNoise",
"inputs": {
"noise_seed": 908966882718651
},
"_meta": {
"title": "RandomNoise"
}
},
"103:44": {
"class_type": "CLIPTextEncode",
"inputs": {
"clip": [
"103:11",
0
],
"text": "Change the ocean background in the video to a grassland and make the horse run on the grassland."
},
"_meta": {
"title": "CLIP Text Encode (Positive Prompt)"
}
},
"103:67": {
"class_type": "ModelSamplingSD3",
"inputs": {
"model": [
"103:12",
0
],
"shift": 7
},
"_meta": {
"title": "ModelSamplingSD3"
}
},
"103:8": {
"class_type": "VAEDecode",
"inputs": {
"samples": [
"103:13",
0
],
"vae": [
"103:10",
0
]
},
"_meta": {
"title": "VAE Decode"
}
},
"103:91": {
"class_type": "CFGGuider",
"inputs": {
"cfg": 6,
"model": [
"103:67",
0
],
"negative": [
"103:115",
1
],
"positive": [
"103:115",
0
]
},
"_meta": {
"title": "CFG Guider"
}
},
"103:93": {
"class_type": "CLIPTextEncode",
"inputs": {
"clip": [
"103:11",
0
],
"text": "blurry, low quality, distorted, ugly, watermark, text"
},
"_meta": {
"title": "CLIP Text Encode (Negative Prompt)"
}
}
})
# run() also passes api_key to the partner nodes in this workflow.
job = client.run(workflow, api_key=api_key)Questions & answers
You can upload both video and image files using the LoadVideo node, as the workflow supports various common formats.
Your text instructions should be clear and concise, specifying the desired outcome to help the Capybara model generate accurate results.
Yes, you can configure the resolution settings based on the Capybara Resolution Configuration table provided in the workflow notes.
Consider increasing the number of steps in the inference process, as suggested in the author notes, to improve the quality of the generated video.