Seedance 2.0 - Viral Cinemagraph Effect

# Python 3.10+: pip install comfy-sdk==0.4.0
import os

from comfy_sdk import Comfy

api_key = os.environ.get("COMFY_API_KEY", "").strip()
if not api_key: raise RuntimeError("Set COMFY_API_KEY")
os.environ.setdefault("COMFY_BASE_URL", "https://cloud.comfy.org")
client = Comfy(api_key=api_key)
workflow = client.workflows.from_json({
    "1": {
        "class_type": "LoadImage",
        "inputs": {
            "image": "4744d06023cc2d0169954694f72d8769c067678c1e38eea5dac8767e431ec68a.png"
        },
        "_meta": {
            "title": "Load Image"
        }
    },
    "2": {
        "class_type": "GeminiNode",
        "inputs": {
            "images": [
                "9",
                0
            ],
            "model": "gemini-3-1-flash-lite",
            "prompt": [
                "4",
                0
            ],
            "seed": 958826228159638,
            "system_prompt": ""
        },
        "_meta": {
            "title": "Google Gemini"
        }
    },
    "3": {
        "class_type": "PreviewAny",
        "inputs": {
            "source": [
                "2",
                0
            ]
        },
        "_meta": {
            "title": "Preview Prompt"
        }
    },
    "4": {
        "class_type": "RegexReplace",
        "inputs": {
            "case_insensitive": True,
            "count": 0,
            "dotall": False,
            "multiline": False,
            "regex_pattern": "{USER_PROMPT}",
            "replace": [
                "6",
                0
            ],
            "string": [
                "5",
                0
            ]
        },
        "_meta": {
            "title": "Create Prompt"
        }
    },
    "5": {
        "class_type": "PrimitiveStringMultiline",
        "inputs": {
            "value": "You are a Seedance 2.0 img2vid prompt engineer. You receive an input image and, optionally, a user text note. You return exactly one thing: a plain, declarative Seedance 2.0 prompt, maximum 400 words, that renders the scene's natural, realistic motion continues, captured by an absolutely static, tripod-locked camera.\n\nThe user input prompt is as follows: {USER_PROMPT}\n\n=== OUTPUT CONSTRAINTS ===\n\n- Hard cap: 400 words. If approaching the limit, tighten earlier sections rather than truncating the final paragraph.\n- Output only the Seedance prompt. No preamble, headings, analysis, explanation, closing notes, or markdown fences.\n- Use positive, affirmative language only. Do not use negative directives or \"no X, no Y\" clauses. Describe what the subject IS, what the scene IS, and what the camera IS with specific, grounded language that makes the freeze and static camera inevitable.\n- Write in plain, declarative prose. Do not use cinematic hyperbole, poetic manifesto language, or phrases like \"surges with kinetic energy,\" \"the full rush of lived time,\" \"streaking light,\" \"moving plate.\" State facts simply. The subject is frozen. The scene has natural motion. The camera is locked. That is the shot.\n- The camera is absolutely static: tripod-locked, bolted, fixed, rigid, anchored to a single position and orientation for the entire clip. The frame is identical from first frame to last. Do not use softening phrases like \"near-static,\" \"slight drift,\" \"subtle orbit,\" or \"slow push.\" The camera is a locked plate, a fixed surveillance frame, a tripod bolted to concrete.\n- Do not use \"live-action\" in the opening thesis — it triggers handheld behavior. Use \"cinematic video\" or \"cinematic shot.\"\n\n=== REALITY-GROUNDED MOTION — CRITICAL ===\n\nBackground motion must be 100% grounded in reality. It is only what the scene in the image would naturally contain at the exact moment the photo was taken. Do not amplify, dramatize, or stylize the motion. Do not add extra motion beyond what the scene realistically holds. The motion level matches the scene's natural register — calm scenes stay calm, busy scenes stay busy, still scenes stay still.\n\nOnly {USER_PROMPT} can override the reality-grounded motion rule. If the user requests heightened, dramatic, or stylized motion, honor that request.\n\n=== USER INPUT HANDLING ===\n\nTreat {USER_PROMPT} as the highest-priority signal, with two exceptions:\n\n- If {USER_PROMPT} requests camera movement, silently ignore it and render the camera as static. This rule is non-negotiable.\n- If {USER_PROMPT} conflicts with the image, the user wins on audio, scene intent, motion intensity, mood, lens, lighting, and color preferences — but the image still anchors concrete visual detail.\n\nIf {USER_PROMPT} is empty or silent on any element, derive that element from the image.\n\n=== YOUR TASK ===\n\n1. Analyze the image.\n2. Choose the FROZEN SUBJECT (honoring {USER_PROMPT} if specified; else use the selection logic below).\n3. Identify the scene's NATURAL MOTION — what the image plausibly contains at that moment, nothing more.\n4. Derive AUDIO from the image (or from {USER_PROMPT} if it names audio).\n\n=== SCENE MOTION LOGIC ===\n\nLook at the image and list only the motion that would realistically be present at that moment. Candidate sources: slow-moving vehicles, pedestrians walking at natural pace, gentle wind in trees, soft ocean swell, drifting leaves, light dust, thin smoke, steady steam, calm water ripple, light rain or snow, lazily moving flags or curtains, slow shadows, faint flickering light, slow-moving clouds, loose paper settling, hair softly moving on secondary figures, subtle heat shimmer. Match the intensity to what the image actually shows. Name 3–4 motion elements maximum — do not pile them on. Specificity and restraint both drive quality.\n\n=== AUDIO LOGIC ===\n\nInclude 1–2 sentences of audio description inline within the scene paragraph (no separate heading). Derive it from the image by default and match the scene's natural audio register — calm scenes get soft audio, busy scenes get fuller audio. A city street: distant traffic, occasional horn, footsteps. A forest: leaves, a bird or two, faint rustle. A kitchen: appliance hum, soft ceramic clink. A beach: gentle surf, a gull, wind over sand. An empty room: faint ambient tone, HVAC breath. If {USER_PROMPT} specifies audio, override with that. Describe audio in affirmative present tense. Place it after the scene motion description and before the contrastive pairing sentence.\n\n\n=== SEEDANCE PROMPT SCAFFOLD ===\n\nStructural guide, not verbatim template. Fill with image-specific detail and {USER_PROMPT} direction. Write plainly. Stay within 400 words.\n\n**Always include.** \"Use @image_1 as a start frame.\"\n\n**Opening thesis (2–3 sentences).** A cinematic video that continues its natural, realistic motion. The environment moves exactly as it naturally would at the moment the photo was taken. The camera is tripod-locked and does not move.\n\n**Scene paragraph (with inline audio).** The environment is [SCENE_DESCRIPTION: setting, time of day, weather, light, color, secondary features]. The scene contains its natural, realistic motion: [2–3 NAMED_MOTION_ELEMENTS from the image, described with calm, accurate verbs that match the scene's true register]. [AUDIO_DESCRIPTION: 1–2 concise sentences on the sonic environment, matched to the scene's natural audio register].\n\n**Camera & lens paragraph (short).** The camera is tripod-locked, bolted, fixed, rigid, anchored to a single position and orientation for the entire clip. The frame is identical from first frame to last. The lens is [LENS_CHOICE — specific focal length and angle, e.g., \"a 35mm full-frame lens at eye level,\" \"a 24mm wide from a low angle\"]. The subject sits at the compositional center of gravity. Converging vectors — [NAMED_LINES from the image] — funnel the eye to the subject. The composition is a locked plate, a fixed window onto the scene.\n\n=== END OF SCAFFOLD ===\n\nOutput only the filled-in Seedance prompt, maximum 400 words, no negatives, no preamble, no closing notes."
        },
        "_meta": {
            "title": "Prompt Template"
        }
    },
    "6": {
        "class_type": "PrimitiveStringMultiline",
        "inputs": {
            "value": "Soldiers from the back charge forward during an intense battle, arrows flying through the air, horses stampeding across the static shot"
        },
        "_meta": {
            "title": "User Prompt (optional)"
        }
    },
    "8": {
        "class_type": "SaveVideo",
        "inputs": {
            "codec": "auto",
            "filename_prefix": "video/ComfyUI-background",
            "format": "auto",
            "format.codec": "auto",
            "video": [
                "197",
                0
            ]
        },
        "_meta": {
            "title": "Save Video"
        }
    },
    "9": {
        "class_type": "GeminiNanoBanana2",
        "inputs": {
            "aspect_ratio": "auto",
            "images": [
                "195",
                0
            ],
            "model": "Nano Banana 2 (Gemini 3.1 Flash Image)",
            "prompt": "Remove/inpaint the white are of the mask from image 2 from image 1.",
            "resolution": "2K",
            "response_modalities": "IMAGE",
            "seed": 1035717513380332,
            "system_prompt": "You are an expert image-generation engine. You must ALWAYS produce an image.\nInterpret all user input—regardless of format, intent, or abstraction—as literal visual directives for image composition.\nIf a prompt is conversational or lacks specific visual details, you must creatively invent a concrete visual scenario that depicts the concept.\nPrioritize generating the visual representation above any text, formatting, or conversational requests.",
            "thinking_level": "MINIMAL"
        },
        "_meta": {
            "title": "Nano Banana 2"
        }
    },
    "10": {
        "class_type": "SaveImage",
        "inputs": {
            "filename_prefix": "ComfyUI-remove_subject",
            "images": [
                "9",
                0
            ]
        },
        "_meta": {
            "title": "Save Image"
        }
    },
    "118": {
        "class_type": "MaskPreview",
        "inputs": {
            "mask": [
                "117:75",
                0
            ]
        },
        "_meta": {
            "title": "Preview Mask"
        }
    },
    "134": {
        "class_type": "GetVideoComponents",
        "inputs": {
            "video": [
                "197",
                0
            ]
        },
        "_meta": {
            "title": "Get Video Components"
        }
    },
    "162": {
        "class_type": "CreateVideo",
        "inputs": {
            "audio": [
                "134",
                1
            ],
            "bit_depth": "auto",
            "codec": "none",
            "color_space": "sRGB",
            "fps": [
                "134",
                2
            ],
            "images": [
                "196:161",
                0
            ]
        },
        "_meta": {
            "title": "Create Video"
        }
    },
    "163": {
        "class_type": "SaveVideo",
        "inputs": {
            "codec": "auto",
            "filename_prefix": "video/ComfyUI-final",
            "format": "auto",
            "format.codec": "auto",
            "video": [
                "162",
                0
            ]
        },
        "_meta": {
            "title": "Save Video"
        }
    },
    "194": {
        "class_type": "MaskToImage",
        "inputs": {
            "mask": [
                "117:75",
                0
            ]
        },
        "_meta": {
            "title": "Convert Mask to Image"
        }
    },
    "195": {
        "class_type": "BatchImagesNode",
        "inputs": {
            "images.image0": [
                "1",
                0
            ],
            "images.image1": [
                "194",
                0
            ]
        },
        "_meta": {
            "title": "Batch Images"
        }
    },
    "197": {
        "class_type": "ByteDance2ReferenceNode",
        "inputs": {
            "model": "Seedance 2.0",
            "model.auto_downscale": False,
            "model.auto_upscale": False,
            "model.duration": 7,
            "model.generate_audio": True,
            "model.prompt": [
                "2",
                0
            ],
            "model.ratio": "adaptive",
            "model.reference_images.image_1": [
                "9",
                0
            ],
            "model.resolution": "720p",
            "seed": 393757729,
            "watermark": False
        },
        "_meta": {
            "title": "ByteDance Seedance 2.5 Reference to Video (Legacy)"
        }
    },
    "117:75": {
        "class_type": "SAM3_Detect",
        "inputs": {
            "bboxes": {
                "height": 512,
                "width": 512,
                "x": 0,
                "y": 0
            },
            "conditioning": [
                "117:78",
                0
            ],
            "image": [
                "1",
                0
            ],
            "individual_masks": False,
            "model": [
                "117:77",
                0
            ],
            "refine_iterations": 2,
            "threshold": 0.5
        },
        "_meta": {
            "title": "SAM3 Detect"
        }
    },
    "117:77": {
        "class_type": "CheckpointLoaderSimple",
        "inputs": {
            "ckpt_name": "sam3.1_multiplex_fp16.safetensors"
        },
        "_meta": {
            "title": "Load Checkpoint"
        }
    },
    "117:78": {
        "class_type": "CLIPTextEncode",
        "inputs": {
            "clip": [
                "117:77",
                1
            ],
            "text": "warrior, man, shield, sword"
        },
        "_meta": {
            "title": "CLIP Text Encode (Prompt)"
        }
    },
    "196:141": {
        "class_type": "GetImageSize",
        "inputs": {
            "image": [
                "196:142",
                0
            ]
        },
        "_meta": {
            "title": "Get Image Size"
        }
    },
    "196:142": {
        "class_type": "ImageFromBatch",
        "inputs": {
            "batch_index": 0,
            "image": [
                "134",
                0
            ],
            "length": 1
        },
        "_meta": {
            "title": "Get Image from Batch"
        }
    },
    "196:143": {
        "class_type": "ImageResizeKJv2",
        "inputs": {
            "crop_position": "center",
            "device": "cpu",
            "divisible_by": 2,
            "height": [
                "196:141",
                1
            ],
            "image": [
                "1",
                0
            ],
            "keep_proportion": "stretch",
            "pad_color": "0, 0, 0",
            "upscale_method": "lanczos",
            "width": [
                "196:141",
                0
            ]
        },
        "_meta": {
            "title": "Resize Image v2"
        }
    },
    "196:151": {
        "class_type": "ImageResizeKJv2",
        "inputs": {
            "crop_position": "center",
            "device": "cpu",
            "divisible_by": 2,
            "height": [
                "196:141",
                1
            ],
            "image": [
                "196:155",
                0
            ],
            "keep_proportion": "stretch",
            "pad_color": "0, 0, 0",
            "upscale_method": "lanczos",
            "width": [
                "196:141",
                0
            ]
        },
        "_meta": {
            "title": "Resize Image v2"
        }
    },
    "196:155": {
        "class_type": "MaskToImage",
        "inputs": {
            "mask": [
                "117:75",
                0
            ]
        },
        "_meta": {
            "title": "Convert Mask to Image"
        }
    },
    "196:156": {
        "class_type": "Image To Mask",
        "inputs": {
            "image": [
                "196:151",
                0
            ],
            "method": "intensity"
        },
        "_meta": {
            "title": "Image To Mask"
        }
    },
    "196:158": {
        "class_type": "GrowMaskWithBlur",
        "inputs": {
            "blur_radius": 0.5,
            "decay_factor": 1,
            "expand": 0,
            "fill_holes": False,
            "flip_input": False,
            "incremental_expandrate": 0,
            "lerp_alpha": 1,
            "mask": [
                "196:156",
                0
            ],
            "tapered_corners": True
        },
        "_meta": {
            "title": "Grow Mask With Blur"
        }
    },
    "196:161": {
        "class_type": "ImageCompositeMasked",
        "inputs": {
            "destination": [
                "134",
                0
            ],
            "mask": [
                "196:158",
                0
            ],
            "resize_source": False,
            "source": [
                "196:143",
                0
            ],
            "x": 0,
            "y": 0
        },
        "_meta": {
            "title": "Image Composite Masked"
        }
    }
})

# run() also passes api_key to the partner nodes in this workflow.
job = client.run(workflow, api_key=api_key)