圖片 → 歌曲 → 音訊反應影片
使用 ACE Step 1.5 XL 建立歌曲,並用 LTX 2.3 以及 Audio-Reactive LoRA 建立音訊反應影片。
關於
這個工作流程把單張圖片轉成帶配樂、且會隨音訊變化的影片:結合文字轉音樂生成(ACE Step 1.5 XL Turbo)與圖片轉影片動畫(LTX 2.3),並加入 Audio‑Reactive LoRA 增強效果。你提供一張圖片與一段簡短創意需求;工作流程會用 GeminiNode 一次產出視覺用的「風格標籤」與歌曲用的「歌詞」。接著 RegexExtract 節點會在 '---' 位置切分文字:前半段作為 LTX 2.3 的視覺風格標籤,後半段作為 ACE Step 1.5 XL Turbo 的歌詞輸入。生成的歌曲會透過 SaveAudioMP3 儲存,並驅動影片的音訊反應效果,最後用 SaveVideo 輸出影片。
技術上,流程分組清楚:Load Image、Create ACE Step 1.5 Prompts、Create Audio‑Reactive LoRA Prompt、Regex Extract、ACE Step 1.5 XL,以及 LTX 2.3 with Audio‑Reactive LoRA。Audio‑Reactive LoRA(可參考連結的 model card)會加入跟著音樂走的動態效果;建議把 strength 調在 1.0 到 2.0 之間(1.4+ 會更強)以取得較好的結果。PreviewAny 方便你在最終渲染前檢查中間輸出(提示詞、音訊、影格),MarkdownNote 則在節點圖內提供快速提示。整體是容易迭代的圖片 → 歌曲 → 音訊反應影片管線:調整風格標籤與歌詞、重新生成音軌、再重新渲染影片,直到符合你的創作方向。
您提供的內容
- 正在讀取工作流程
在 ComfyUI 中執行。 此工作流程無法在此頁面上執行。請在已載入所有內容的 Comfy Cloud 中開啟,或下載到您自己的 ComfyUI。
在 Cloud 中試用工作流程內部
節點圖的唯讀預覽。請在 Comfy Cloud 中開啟,或下載後編輯節點。
Loading the graph
Open full-size workflow preview從程式碼使用此工作流程
可行時會根據工作流程圖和輸入內容產生;否則會顯示儲存的範例。執行可能產生費用的程式碼前,請檢查認證資訊和端點。
What you need
- Runs on
- Comfy Cloud
- 金鑰
- COMFY_API_KEY
需要付費 Cloud 方案與點數。
# Python 3.10+: pip install comfy-sdk==0.4.0
import os
from comfy_sdk import Comfy
api_key = os.environ.get("COMFY_API_KEY", "").strip()
if not api_key: raise RuntimeError("Set COMFY_API_KEY")
os.environ.setdefault("COMFY_BASE_URL", "https://cloud.comfy.org")
client = Comfy(api_key=api_key)
workflow = client.workflows.from_json({
"666": {
"class_type": "LoadImage",
"inputs": {
"image": "660fdce25fe105c5982159389b5fdf67c12d941d49be78defcf77fd1560ca507.png"
},
"_meta": {
"title": "Load Image"
}
},
"668": {
"class_type": "SaveAudioMP3",
"inputs": {
"audio": [
"667:18",
0
],
"filename_prefix": "audio/ACE_Step1.5_xl_turbo",
"quality": "320k"
},
"_meta": {
"title": "Save Audio (MP3) (DEPRECATED)"
}
},
"670": {
"class_type": "RegexExtract",
"inputs": {
"case_insensitive": True,
"dotall": False,
"group_index": 1,
"mode": "First Group",
"multiline": False,
"regex_pattern": "(?s)(?<=\\n---\\n)([\\s\\S]*)$",
"string": [
"673",
0
]
},
"_meta": {
"title": "Extract Text"
}
},
"671": {
"class_type": "PreviewAny",
"inputs": {
"source": [
"670",
0
]
},
"_meta": {
"title": "Preview as Text"
}
},
"672": {
"class_type": "PreviewAny",
"inputs": {
"source": [
"674",
0
]
},
"_meta": {
"title": "Preview as Text"
}
},
"673": {
"class_type": "GeminiNode",
"inputs": {
"images": [
"666",
0
],
"model": "gemini-3-1-flash-lite",
"prompt": "medieval ambient",
"seed": 65482163754128,
"system_prompt": "**Role:** You are a professional Music Architect for **ACE-Step 1.5**. \n\n**Task:** Generate a music creation plan.\n\n**Strict Output Constraint (IMPORTANT):**\n1. Return the output as **RAW PLAIN TEXT** only.\n2. DO NOT use Markdown code blocks (no backticks ```).\n3. DO NOT use any headers, labels, or bold text.\n4. Use ONLY the separator `---` to divide the two parts.\n5. Part 1: Comma-separated style keywords.\n6. Part 2: Structured lyrics with [Tags].\n\n**My Topic/Requirement:**\n[USER INPUT]"
},
"_meta": {
"title": "Google Gemini"
}
},
"674": {
"class_type": "RegexExtract",
"inputs": {
"case_insensitive": True,
"dotall": False,
"group_index": 1,
"mode": "First Group",
"multiline": False,
"regex_pattern": "(?sm)^(.*?)^---$(.*)",
"string": [
"673",
0
]
},
"_meta": {
"title": "Extract Text"
}
},
"683": {
"class_type": "SaveVideo",
"inputs": {
"filename_prefix": "video/ComfyUI",
"format": "auto",
"format.codec": "auto",
"video": [
"792:312",
0
]
},
"_meta": {
"title": "Save Video"
}
},
"752": {
"class_type": "PreviewAny",
"inputs": {
"source": [
"673",
0
]
},
"_meta": {
"title": "Preview as Text"
}
},
"753": {
"class_type": "GeminiNode",
"inputs": {
"audio": [
"667:18",
0
],
"images": [
"666",
0
],
"model": "gemini-3-1-flash-lite",
"prompt": "Continuous audio-reactive video.",
"seed": 1013654870151104,
"system_prompt": "You are an expert prompt engineer specializing in LTX-2.3 video generation prompts, especially audio-reactive LoRA workflows.\n\nYour task is to generate cinematic, continuous, high-motion video prompts using the required trigger phrase:\n\ncontinuous audio-reactive video\n\nAlways include this exact phrase near the beginning of every prompt.\n\nThe user will provide, imply, or request a visual concept, music genre, or motion style. Transform their idea into a polished prompt that follows this structure:\n\nA continuous audio-reactive video that transitions smoothly from [Phase 1: Initial Subject & Action] to [Phase 2: First Evolution & Motion], then warps/morphs into [Phase 3: Second Evolution & Morphing] before warping/morphing into [Phase 4: Final Chaotic/Complex State], with every [Type of Visual Motion/Deformation] perfectly synchronized to the [Musical Element 1], [Musical Element 2], and [Musical Element 3] of the [Music Genre & Vibe] track.\n\nOutput rules:\n- Always answer with only one finished prompt.\n- Do not include explanations, headings, labels, notes, negative prompts, recommended settings, batch guidance, or extra sections.\n- Do not ask clarifying questions.\n- Make a reasonable creative choice when the user gives limited detail.\n- Always include 2 to 4 distinct phases.\n- Prefer 4 phases for complex, chaotic, surreal, or high-energy transformations.\n- Prefer 2 or 3 phases for elegant, minimal, stable, or slower concepts.\n- Make the video feel continuous, not like separate scenes.\n- Use smooth transition language such as “transitions smoothly,” “melts into,” “warps into,” “morphs into,” “fractures into,” “blooms into,” or “folds into.”\n- Emphasize visible motion, rhythmic deformation, fluid morphing, and high-motion transformation.\n- Avoid static composition language unless paired with motion.\n- Avoid vague phrases like “cool visuals,” “nice effects,” or “beautiful animation.”\n- Do not overcomplicate the scene with too many unrelated subjects.\n- Keep the subject visually trackable as it evolves.\n- Synchronize visual motion to musical features such as bass hits, kick drums, snares, hi-hats, synth pulses, vocal chops, risers, drops, cymbal crashes, arpeggios, sub-bass waves, or tempo shifts.\n- For image-to-video requests, preserve the starting image’s core subject, composition, color palette, and camera angle while adding audio-reactive motion and morphing.\n\nStyle:\nUse vivid, cinematic, motion-heavy language. Prioritize fluid morphing, rhythmic deformation, beat-synced escalation, and seamless phase-to-phase transformation.\n\nFinal answer format:\nReturn only the completed prompt as plain text."
},
"_meta": {
"title": "Google Gemini"
}
},
"754": {
"class_type": "PreviewAny",
"inputs": {
"source": [
"753",
0
]
},
"_meta": {
"title": "Preview as Text"
}
},
"667:104": {
"class_type": "UNETLoader",
"inputs": {
"unet_name": "acestep_v1.5_xl_turbo_bf16.safetensors",
"weight_dtype": "default"
},
"_meta": {
"title": "Load Diffusion Model"
}
},
"667:105": {
"class_type": "DualCLIPLoader",
"inputs": {
"clip_name1": "qwen_0.6b_ace15.safetensors",
"clip_name2": "qwen_4b_ace15.safetensors",
"device": "default",
"type": "ace"
},
"_meta": {
"title": "Load CLIP (Dual)"
}
},
"667:106": {
"class_type": "VAELoader",
"inputs": {
"vae_name": "ace_1.5_vae.safetensors"
},
"_meta": {
"title": "Load VAE"
}
},
"667:109": {
"class_type": "PrimitiveInt",
"inputs": {
"value": 0
},
"_meta": {
"title": "Int (Seed)"
}
},
"667:18": {
"class_type": "VAEDecodeAudio",
"inputs": {
"samples": [
"667:3",
0
],
"vae": [
"667:106",
0
]
},
"_meta": {
"title": "VAE Decode Audio"
}
},
"667:3": {
"class_type": "KSampler",
"inputs": {
"cfg": 1,
"denoise": 1,
"latent_image": [
"667:98",
0
],
"model": [
"667:78",
0
],
"negative": [
"667:47",
0
],
"positive": [
"667:94",
0
],
"sampler_name": "euler",
"scheduler": "simple",
"seed": [
"667:109",
0
],
"steps": 8
},
"_meta": {
"title": "KSampler"
}
},
"667:47": {
"class_type": "ConditioningZeroOut",
"inputs": {
"conditioning": [
"667:94",
0
]
},
"_meta": {
"title": "Conditioning Zero Out"
}
},
"667:78": {
"class_type": "ModelSamplingAuraFlow",
"inputs": {
"model": [
"667:104",
0
],
"sampling": "flow",
"shift": 3
},
"_meta": {
"title": "ModelSamplingAuraFlow"
}
},
"667:94": {
"class_type": "TextEncodeAceStepAudio1.5",
"inputs": {
"bpm": 95,
"cfg_scale": 2,
"clip": [
"667:105",
0
],
"duration": 60,
"generate_audio_codes": True,
"keyscale": "E minor",
"language": "en",
"lyrics": [
"671",
0
],
"min_p": 0,
"seed": [
"667:109",
0
],
"tags": [
"672",
0
],
"temperature": 0.85,
"timesignature": "4",
"top_k": 0,
"top_p": 0.9
},
"_meta": {
"title": "TextEncodeAceStepAudio1.5"
}
},
"667:98": {
"class_type": "EmptyAceStep1.5LatentAudio",
"inputs": {
"batch_size": 1,
"seconds": 60
},
"_meta": {
"title": "Empty Ace Step 1.5 Latent Audio"
}
},
"792:285": {
"class_type": "RandomNoise",
"inputs": {
"noise_seed": 42
},
"_meta": {
"title": "RandomNoise"
}
},
"792:286": {
"class_type": "RandomNoise",
"inputs": {
"noise_seed": 784235873879537
},
"_meta": {
"title": "RandomNoise"
}
},
"792:287": {
"class_type": "LTXVConcatAVLatent",
"inputs": {
"audio_latent": [
"792:309",
1
],
"video_latent": [
"792:296",
0
]
},
"_meta": {
"title": "Concat AV Latent"
}
},
"792:288": {
"class_type": "KSamplerSelect",
"inputs": {
"sampler_name": "euler_cfg_pp"
},
"_meta": {
"title": "KSamplerSelect"
}
},
"792:289": {
"class_type": "ManualSigmas",
"inputs": {
"sigmas": "0.85, 0.7250, 0.4219, 0.0"
},
"_meta": {
"title": "ManualSigmas"
}
},
"792:290": {
"class_type": "CFGGuider",
"inputs": {
"cfg": 1,
"model": [
"792:791",
0
],
"negative": [
"792:292",
1
],
"positive": [
"792:292",
0
]
},
"_meta": {
"title": "CFG Guider"
}
},
"792:291": {
"class_type": "SamplerCustomAdvanced",
"inputs": {
"guider": [
"792:315",
0
],
"latent_image": [
"792:326",
0
],
"noise": [
"792:286",
0
],
"sampler": [
"792:298",
0
],
"sigmas": [
"792:308",
0
]
},
"_meta": {
"title": "SamplerCustomAdvanced"
}
},
"792:292": {
"class_type": "LTXVCropGuides",
"inputs": {
"latent": [
"792:309",
0
],
"negative": [
"792:307",
1
],
"positive": [
"792:307",
0
]
},
"_meta": {
"title": "LTXVCropGuides"
}
},
"792:294": {
"class_type": "ResizeImagesByLongerEdge",
"inputs": {
"images": [
"792:297",
0
],
"longer_edge": 1536
},
"_meta": {
"title": "Resize Images by Longer Edge (DEPRECATED)"
}
},
"792:295": {
"class_type": "LTXVLatentUpsampler",
"inputs": {
"samples": [
"792:309",
0
],
"upscale_model": [
"792:786",
0
],
"vae": [
"792:788",
2
]
},
"_meta": {
"title": "LTXVLatentUpsampler"
}
},
"792:296": {
"class_type": "LTXVImgToVideoInplace",
"inputs": {
"bypass": [
"792:305",
0
],
"image": [
"792:334",
0
],
"latent": [
"792:295",
0
],
"strength": 1,
"vae": [
"792:788",
2
]
},
"_meta": {
"title": "LTXVImgToVideoInplace"
}
},
"792:297": {
"class_type": "ResizeImageMaskNode",
"inputs": {
"input": [
"666",
0
],
"resize_type": "scale dimensions",
"resize_type.crop": "center",
"resize_type.height": [
"792:324",
0
],
"resize_type.width": [
"792:330",
0
],
"scale_method": "lanczos"
},
"_meta": {
"title": "Resize Image/Mask"
}
},
"792:298": {
"class_type": "KSamplerSelect",
"inputs": {
"sampler_name": "euler_ancestral_cfg_pp"
},
"_meta": {
"title": "KSamplerSelect"
}
},
"792:299": {
"class_type": "ComfyMathExpression",
"inputs": {
"expression": "a/2",
"values.a": [
"792:330",
0
]
},
"_meta": {
"title": "Math Expression"
}
},
"792:301": {
"class_type": "ComfyMathExpression",
"inputs": {
"expression": "a/2",
"values.a": [
"792:324",
0
]
},
"_meta": {
"title": "Math Expression"
}
},
"792:302": {
"class_type": "EmptyLTXVLatentVideo",
"inputs": {
"batch_size": 1,
"height": [
"792:301",
1
],
"length": [
"792:329",
1
],
"width": [
"792:299",
1
]
},
"_meta": {
"title": "Empty LTXV Latent Video"
}
},
"792:303": {
"class_type": "LTXVAudioVAEDecode",
"inputs": {
"audio_vae": [
"792:785",
0
],
"samples": [
"792:311",
1
]
},
"_meta": {
"title": "LTXV Audio VAE Decode"
}
},
"792:304": {
"class_type": "ComfyMathExpression",
"inputs": {
"expression": "a",
"values.a": [
"792:323",
0
]
},
"_meta": {
"title": "Math Expression"
}
},
"792:305": {
"class_type": "PrimitiveBoolean",
"inputs": {
"value": False
},
"_meta": {
"title": "Switch to Text to Video?"
}
},
"792:306": {
"class_type": "CLIPTextEncode",
"inputs": {
"clip": [
"792:789",
0
],
"text": [
"792:319",
0
]
},
"_meta": {
"title": "CLIP Text Encode (Prompt)"
}
},
"792:307": {
"class_type": "LTXVConditioning",
"inputs": {
"frame_rate": [
"792:304",
0
],
"negative": [
"792:314",
0
],
"positive": [
"792:306",
0
]
},
"_meta": {
"title": "LTXVConditioning"
}
},
"792:308": {
"class_type": "ManualSigmas",
"inputs": {
"sigmas": "1.0, 0.99375, 0.9875, 0.98125, 0.975, 0.909375, 0.725, 0.421875, 0.0"
},
"_meta": {
"title": "ManualSigmas"
}
},
"792:309": {
"class_type": "LTXVSeparateAVLatent",
"inputs": {
"av_latent": [
"792:291",
0
]
},
"_meta": {
"title": "Separate AV Latent"
}
},
"792:310": {
"class_type": "SamplerCustomAdvanced",
"inputs": {
"guider": [
"792:290",
0
],
"latent_image": [
"792:287",
0
],
"noise": [
"792:285",
0
],
"sampler": [
"792:288",
0
],
"sigmas": [
"792:289",
0
]
},
"_meta": {
"title": "SamplerCustomAdvanced"
}
},
"792:311": {
"class_type": "LTXVSeparateAVLatent",
"inputs": {
"av_latent": [
"792:310",
0
]
},
"_meta": {
"title": "Separate AV Latent"
}
},
"792:312": {
"class_type": "CreateVideo",
"inputs": {
"audio": [
"792:303",
0
],
"bit_depth": "auto",
"codec": "none",
"color_space": "sRGB",
"fps": [
"792:304",
0
],
"images": [
"792:316",
0
]
},
"_meta": {
"title": "Create Video"
}
},
"792:314": {
"class_type": "CLIPTextEncode",
"inputs": {
"clip": [
"792:789",
0
],
"text": "pc game, console game, video game, cartoon, childish, ugly"
},
"_meta": {
"title": "CLIP Text Encode (Prompt)"
}
},
"792:315": {
"class_type": "CFGGuider",
"inputs": {
"cfg": 1,
"model": [
"792:791",
0
],
"negative": [
"792:307",
1
],
"positive": [
"792:307",
0
]
},
"_meta": {
"title": "CFG Guider"
}
},
"792:316": {
"class_type": "VAEDecodeTiled",
"inputs": {
"overlap": 64,
"samples": [
"792:311",
0
],
"temporal_overlap": 4,
"temporal_size": 4096,
"tile_size": 768,
"vae": [
"792:788",
2
]
},
"_meta": {
"title": "VAE Decode (Tiled)"
}
},
"792:319": {
"class_type": "PrimitiveStringMultiline",
"inputs": {
"value": [
"754",
0
]
},
"_meta": {
"title": "Prompt"
}
},
"792:323": {
"class_type": "PrimitiveInt",
"inputs": {
"value": 24
},
"_meta": {
"title": "Frame Rate"
}
},
"792:324": {
"class_type": "PrimitiveInt",
"inputs": {
"value": 720
},
"_meta": {
"title": "Height"
}
},
"792:325": {
"class_type": "LTXVImgToVideoInplace",
"inputs": {
"bypass": [
"792:305",
0
],
"image": [
"792:334",
0
],
"latent": [
"792:302",
0
],
"strength": 0.7,
"vae": [
"792:788",
2
]
},
"_meta": {
"title": "LTXVImgToVideoInplace"
}
},
"792:326": {
"class_type": "LTXVConcatAVLatent",
"inputs": {
"audio_latent": [
"792:327",
0
],
"video_latent": [
"792:325",
0
]
},
"_meta": {
"title": "Concat AV Latent"
}
},
"792:327": {
"class_type": "SetLatentNoiseMask",
"inputs": {
"mask": [
"792:333",
0
],
"samples": [
"792:328",
0
]
},
"_meta": {
"title": "Set Latent Noise Mask"
}
},
"792:328": {
"class_type": "LTXVAudioVAEEncode",
"inputs": {
"audio": [
"792:332",
0
],
"audio_vae": [
"792:785",
0
]
},
"_meta": {
"title": "LTXV Audio VAE Encode"
}
},
"792:329": {
"class_type": "ComfyMathExpression",
"inputs": {
"expression": "a * b + 1",
"values.a": [
"792:331",
0
],
"values.b": [
"792:323",
0
]
},
"_meta": {
"title": "Math Expression"
}
},
"792:330": {
"class_type": "PrimitiveInt",
"inputs": {
"value": 1280
},
"_meta": {
"title": "Width"
}
},
"792:331": {
"class_type": "PrimitiveFloat",
"inputs": {
"value": 9
},
"_meta": {
"title": "Duration"
}
},
"792:332": {
"class_type": "TrimAudioDuration",
"inputs": {
"audio": [
"667:18",
0
],
"duration": [
"792:331",
0
],
"start_index": 0
},
"_meta": {
"title": "Trim Audio Duration"
}
},
"792:333": {
"class_type": "SolidMask",
"inputs": {
"height": [
"792:324",
0
],
"value": 0,
"width": [
"792:330",
0
]
},
"_meta": {
"title": "Create Solid Mask"
}
},
"792:334": {
"class_type": "LTXVPreprocess",
"inputs": {
"image": [
"792:294",
0
],
"img_compression": 18
},
"_meta": {
"title": "LTXV Preprocess"
}
},
"792:785": {
"class_type": "LTXVAudioVAELoader",
"inputs": {
"ckpt_name": "ltx-2.3-22b-dev-fp8.safetensors"
},
"_meta": {
"title": "Load LTXV Audio VAE"
}
},
"792:786": {
"class_type": "LatentUpscaleModelLoader",
"inputs": {
"model_name": "ltx-2.3-spatial-upscaler-x2-1.1.safetensors"
},
"_meta": {
"title": "Load Latent Upscale Model"
}
},
"792:787": {
"class_type": "LoraLoaderModelOnly",
"inputs": {
"lora_name": "ltx-2.3-22b-distilled-lora-384.safetensors",
"model": [
"792:788",
0
],
"strength_model": 0.5
},
"_meta": {
"title": "Load LoRA"
}
},
"792:788": {
"class_type": "CheckpointLoaderSimple",
"inputs": {
"ckpt_name": "ltx-2.3-22b-dev-fp8.safetensors"
},
"_meta": {
"title": "Load Checkpoint"
}
},
"792:789": {
"class_type": "LTXAVTextEncoderLoader",
"inputs": {
"ckpt_name": "ltx-2.3-22b-dev-fp8.safetensors",
"device": "default",
"text_encoder": "gemma_3_12B_it_fp4_mixed.safetensors"
},
"_meta": {
"title": "Load LTXV Audio Text Encoder"
}
},
"792:791": {
"class_type": "LoraLoaderModelOnly",
"inputs": {
"lora_name": "LTX2.3_AudioReactiveV1.safetensors",
"model": [
"792:787",
0
],
"strength_model": 1
},
"_meta": {
"title": "Load LoRA"
}
}
})
# run() also passes api_key to the partner nodes in this workflow.
job = client.run(workflow, api_key=api_key)問答
The RegexExtract node captures everything before '---' as style tags for LTX 2.3 and everything after as lyrics for ACE Step 1.5 XL Turbo. This guarantees each model receives a purpose‑built prompt from a single GeminiNode output.
Use the Audio‑Reactive LoRA strength in the LTX 2.3 group. A range of 1.0–2.0 is recommended; try ~1.4 or higher for stronger effects. More dynamic audio (clear percussion, drops, and swells) will also produce more noticeable motion.
Yes. Replace or bypass the ACE branch and feed your audio into the LTX 2.3 + Audio‑Reactive LoRA input path (use a compatible audio loader node in ComfyUI). Ensure the video duration matches your track length.
Start with a clean, high‑quality image; keep style tags concise and specific; use reasonable resolutions your GPU can handle; and iterate: adjust style tags, LoRA strength, and re‑render. PreviewAny helps validate prompts and frames before the final save.