Imagem → Música → Vídeo Áudio-Reativo
Crie músicas com ACE Step 1.5 XL e vídeos áudio-reativos com LTX 2.3 e o Audio-Reactive LoRA.
Sobre
Este workflow transforma uma única imagem em um vídeo com música e reação ao áudio, combinando geração de texto para música (ACE Step 1.5 XL Turbo) com animação image-to-video (LTX 2.3) aprimorada por um LoRA Audio‑Reactive. Você fornece uma imagem e um breve direcionamento criativo; o workflow usa um GeminiNode para redigir, em uma única passada, tanto "tags de estilo" para os visuais quanto "letras" para a música. Em seguida, um nó RegexExtract divide esse texto em '---': tudo antes vira tags de estilo visual para o LTX 2.3, e tudo depois vira letras enviadas ao ACE Step 1.5 XL Turbo. A música gerada é salva via SaveAudioMP3 e então dirige a reatividade do vídeo, que é salvo com SaveVideo.
Tecnicamente, o pipeline é organizado em grupos: Load Image, Create ACE Step 1.5 Prompts, Create Audio‑Reactive LoRA Prompt, Regex Extract, ACE Step 1.5 XL e LTX 2.3 with Audio‑Reactive LoRA. O LoRA Audio‑Reactive (veja o model card linkado) adiciona efeitos de movimento que seguem a música; ajuste a força entre 1.0 e 2.0 (1.4+ para efeitos mais fortes) para melhores resultados. O PreviewAny ajuda a conferir saídas intermediárias (prompts, áudio e frames) antes do render final, enquanto o MarkdownNote traz dicas rápidas dentro do grafo. O resultado é um pipeline simplificado imagem → música → vídeo reativo ao áudio, fácil de iterar: ajuste tags de estilo e letras, regenere a faixa e rerenderize o vídeo até corresponder ao seu direcionamento criativo.
O que você fornece
- Lendo o fluxo de trabalho
É executado no ComfyUI. Este fluxo de trabalho não pode ser executado nesta página. Abra-o no Comfy Cloud com tudo carregado ou baixe-o para usar no seu próprio ComfyUI.
Experimentar no CloudDentro do workflow
Uma prévia somente leitura do gráfico de nós. Abra-o no Comfy Cloud ou faça o download para editar os nós.
Loading the graph
Open full-size workflow previewArquivos neste workflow
- Autor
- ComfyUI
Requer nós personalizados
- Lendo o fluxo de trabalho
Use este fluxo de trabalho a partir do código
Gerado a partir do grafo e das entradas do fluxo de trabalho quando possível; caso contrário, são exibidos exemplos armazenados. Revise as credenciais e os endpoints antes de executar o código, pois isso pode gerar cobranças.
What you need
- Runs on
- Comfy Cloud
- Chave
- COMFY_API_KEY
Requer um plano pago do Cloud e créditos.
# Python 3.10+: pip install comfy-sdk==0.4.0
import os
from comfy_sdk import Comfy
api_key = os.environ.get("COMFY_API_KEY", "").strip()
if not api_key: raise RuntimeError("Set COMFY_API_KEY")
os.environ.setdefault("COMFY_BASE_URL", "https://cloud.comfy.org")
client = Comfy(api_key=api_key)
workflow = client.workflows.from_json({
"666": {
"class_type": "LoadImage",
"inputs": {
"image": "660fdce25fe105c5982159389b5fdf67c12d941d49be78defcf77fd1560ca507.png"
},
"_meta": {
"title": "Load Image"
}
},
"668": {
"class_type": "SaveAudioMP3",
"inputs": {
"audio": [
"667:18",
0
],
"filename_prefix": "audio/ACE_Step1.5_xl_turbo",
"quality": "320k"
},
"_meta": {
"title": "Save Audio (MP3) (DEPRECATED)"
}
},
"670": {
"class_type": "RegexExtract",
"inputs": {
"case_insensitive": True,
"dotall": False,
"group_index": 1,
"mode": "First Group",
"multiline": False,
"regex_pattern": "(?s)(?<=\\n---\\n)([\\s\\S]*)$",
"string": [
"673",
0
]
},
"_meta": {
"title": "Extract Text"
}
},
"671": {
"class_type": "PreviewAny",
"inputs": {
"source": [
"670",
0
]
},
"_meta": {
"title": "Preview as Text"
}
},
"672": {
"class_type": "PreviewAny",
"inputs": {
"source": [
"674",
0
]
},
"_meta": {
"title": "Preview as Text"
}
},
"673": {
"class_type": "GeminiNode",
"inputs": {
"images": [
"666",
0
],
"model": "gemini-3-1-flash-lite",
"prompt": "medieval ambient",
"seed": 65482163754128,
"system_prompt": "**Role:** You are a professional Music Architect for **ACE-Step 1.5**. \n\n**Task:** Generate a music creation plan.\n\n**Strict Output Constraint (IMPORTANT):**\n1. Return the output as **RAW PLAIN TEXT** only.\n2. DO NOT use Markdown code blocks (no backticks ```).\n3. DO NOT use any headers, labels, or bold text.\n4. Use ONLY the separator `---` to divide the two parts.\n5. Part 1: Comma-separated style keywords.\n6. Part 2: Structured lyrics with [Tags].\n\n**My Topic/Requirement:**\n[USER INPUT]"
},
"_meta": {
"title": "Google Gemini"
}
},
"674": {
"class_type": "RegexExtract",
"inputs": {
"case_insensitive": True,
"dotall": False,
"group_index": 1,
"mode": "First Group",
"multiline": False,
"regex_pattern": "(?sm)^(.*?)^---$(.*)",
"string": [
"673",
0
]
},
"_meta": {
"title": "Extract Text"
}
},
"683": {
"class_type": "SaveVideo",
"inputs": {
"filename_prefix": "video/ComfyUI",
"format": "auto",
"format.codec": "auto",
"video": [
"792:312",
0
]
},
"_meta": {
"title": "Save Video"
}
},
"752": {
"class_type": "PreviewAny",
"inputs": {
"source": [
"673",
0
]
},
"_meta": {
"title": "Preview as Text"
}
},
"753": {
"class_type": "GeminiNode",
"inputs": {
"audio": [
"667:18",
0
],
"images": [
"666",
0
],
"model": "gemini-3-1-flash-lite",
"prompt": "Continuous audio-reactive video.",
"seed": 1013654870151104,
"system_prompt": "You are an expert prompt engineer specializing in LTX-2.3 video generation prompts, especially audio-reactive LoRA workflows.\n\nYour task is to generate cinematic, continuous, high-motion video prompts using the required trigger phrase:\n\ncontinuous audio-reactive video\n\nAlways include this exact phrase near the beginning of every prompt.\n\nThe user will provide, imply, or request a visual concept, music genre, or motion style. Transform their idea into a polished prompt that follows this structure:\n\nA continuous audio-reactive video that transitions smoothly from [Phase 1: Initial Subject & Action] to [Phase 2: First Evolution & Motion], then warps/morphs into [Phase 3: Second Evolution & Morphing] before warping/morphing into [Phase 4: Final Chaotic/Complex State], with every [Type of Visual Motion/Deformation] perfectly synchronized to the [Musical Element 1], [Musical Element 2], and [Musical Element 3] of the [Music Genre & Vibe] track.\n\nOutput rules:\n- Always answer with only one finished prompt.\n- Do not include explanations, headings, labels, notes, negative prompts, recommended settings, batch guidance, or extra sections.\n- Do not ask clarifying questions.\n- Make a reasonable creative choice when the user gives limited detail.\n- Always include 2 to 4 distinct phases.\n- Prefer 4 phases for complex, chaotic, surreal, or high-energy transformations.\n- Prefer 2 or 3 phases for elegant, minimal, stable, or slower concepts.\n- Make the video feel continuous, not like separate scenes.\n- Use smooth transition language such as “transitions smoothly,” “melts into,” “warps into,” “morphs into,” “fractures into,” “blooms into,” or “folds into.”\n- Emphasize visible motion, rhythmic deformation, fluid morphing, and high-motion transformation.\n- Avoid static composition language unless paired with motion.\n- Avoid vague phrases like “cool visuals,” “nice effects,” or “beautiful animation.”\n- Do not overcomplicate the scene with too many unrelated subjects.\n- Keep the subject visually trackable as it evolves.\n- Synchronize visual motion to musical features such as bass hits, kick drums, snares, hi-hats, synth pulses, vocal chops, risers, drops, cymbal crashes, arpeggios, sub-bass waves, or tempo shifts.\n- For image-to-video requests, preserve the starting image’s core subject, composition, color palette, and camera angle while adding audio-reactive motion and morphing.\n\nStyle:\nUse vivid, cinematic, motion-heavy language. Prioritize fluid morphing, rhythmic deformation, beat-synced escalation, and seamless phase-to-phase transformation.\n\nFinal answer format:\nReturn only the completed prompt as plain text."
},
"_meta": {
"title": "Google Gemini"
}
},
"754": {
"class_type": "PreviewAny",
"inputs": {
"source": [
"753",
0
]
},
"_meta": {
"title": "Preview as Text"
}
},
"667:104": {
"class_type": "UNETLoader",
"inputs": {
"unet_name": "acestep_v1.5_xl_turbo_bf16.safetensors",
"weight_dtype": "default"
},
"_meta": {
"title": "Load Diffusion Model"
}
},
"667:105": {
"class_type": "DualCLIPLoader",
"inputs": {
"clip_name1": "qwen_0.6b_ace15.safetensors",
"clip_name2": "qwen_4b_ace15.safetensors",
"device": "default",
"type": "ace"
},
"_meta": {
"title": "Load CLIP (Dual)"
}
},
"667:106": {
"class_type": "VAELoader",
"inputs": {
"vae_name": "ace_1.5_vae.safetensors"
},
"_meta": {
"title": "Load VAE"
}
},
"667:109": {
"class_type": "PrimitiveInt",
"inputs": {
"value": 0
},
"_meta": {
"title": "Int (Seed)"
}
},
"667:18": {
"class_type": "VAEDecodeAudio",
"inputs": {
"samples": [
"667:3",
0
],
"vae": [
"667:106",
0
]
},
"_meta": {
"title": "VAE Decode Audio"
}
},
"667:3": {
"class_type": "KSampler",
"inputs": {
"cfg": 1,
"denoise": 1,
"latent_image": [
"667:98",
0
],
"model": [
"667:78",
0
],
"negative": [
"667:47",
0
],
"positive": [
"667:94",
0
],
"sampler_name": "euler",
"scheduler": "simple",
"seed": [
"667:109",
0
],
"steps": 8
},
"_meta": {
"title": "KSampler"
}
},
"667:47": {
"class_type": "ConditioningZeroOut",
"inputs": {
"conditioning": [
"667:94",
0
]
},
"_meta": {
"title": "Conditioning Zero Out"
}
},
"667:78": {
"class_type": "ModelSamplingAuraFlow",
"inputs": {
"model": [
"667:104",
0
],
"sampling": "flow",
"shift": 3
},
"_meta": {
"title": "ModelSamplingAuraFlow"
}
},
"667:94": {
"class_type": "TextEncodeAceStepAudio1.5",
"inputs": {
"bpm": 95,
"cfg_scale": 2,
"clip": [
"667:105",
0
],
"duration": 60,
"generate_audio_codes": True,
"keyscale": "E minor",
"language": "en",
"lyrics": [
"671",
0
],
"min_p": 0,
"seed": [
"667:109",
0
],
"tags": [
"672",
0
],
"temperature": 0.85,
"timesignature": "4",
"top_k": 0,
"top_p": 0.9
},
"_meta": {
"title": "TextEncodeAceStepAudio1.5"
}
},
"667:98": {
"class_type": "EmptyAceStep1.5LatentAudio",
"inputs": {
"batch_size": 1,
"seconds": 60
},
"_meta": {
"title": "Empty Ace Step 1.5 Latent Audio"
}
},
"792:285": {
"class_type": "RandomNoise",
"inputs": {
"noise_seed": 42
},
"_meta": {
"title": "RandomNoise"
}
},
"792:286": {
"class_type": "RandomNoise",
"inputs": {
"noise_seed": 784235873879537
},
"_meta": {
"title": "RandomNoise"
}
},
"792:287": {
"class_type": "LTXVConcatAVLatent",
"inputs": {
"audio_latent": [
"792:309",
1
],
"video_latent": [
"792:296",
0
]
},
"_meta": {
"title": "Concat AV Latent"
}
},
"792:288": {
"class_type": "KSamplerSelect",
"inputs": {
"sampler_name": "euler_cfg_pp"
},
"_meta": {
"title": "KSamplerSelect"
}
},
"792:289": {
"class_type": "ManualSigmas",
"inputs": {
"sigmas": "0.85, 0.7250, 0.4219, 0.0"
},
"_meta": {
"title": "ManualSigmas"
}
},
"792:290": {
"class_type": "CFGGuider",
"inputs": {
"cfg": 1,
"model": [
"792:791",
0
],
"negative": [
"792:292",
1
],
"positive": [
"792:292",
0
]
},
"_meta": {
"title": "CFG Guider"
}
},
"792:291": {
"class_type": "SamplerCustomAdvanced",
"inputs": {
"guider": [
"792:315",
0
],
"latent_image": [
"792:326",
0
],
"noise": [
"792:286",
0
],
"sampler": [
"792:298",
0
],
"sigmas": [
"792:308",
0
]
},
"_meta": {
"title": "SamplerCustomAdvanced"
}
},
"792:292": {
"class_type": "LTXVCropGuides",
"inputs": {
"latent": [
"792:309",
0
],
"negative": [
"792:307",
1
],
"positive": [
"792:307",
0
]
},
"_meta": {
"title": "LTXVCropGuides"
}
},
"792:294": {
"class_type": "ResizeImagesByLongerEdge",
"inputs": {
"images": [
"792:297",
0
],
"longer_edge": 1536
},
"_meta": {
"title": "Resize Images by Longer Edge (DEPRECATED)"
}
},
"792:295": {
"class_type": "LTXVLatentUpsampler",
"inputs": {
"samples": [
"792:309",
0
],
"upscale_model": [
"792:786",
0
],
"vae": [
"792:788",
2
]
},
"_meta": {
"title": "LTXVLatentUpsampler"
}
},
"792:296": {
"class_type": "LTXVImgToVideoInplace",
"inputs": {
"bypass": [
"792:305",
0
],
"image": [
"792:334",
0
],
"latent": [
"792:295",
0
],
"strength": 1,
"vae": [
"792:788",
2
]
},
"_meta": {
"title": "LTXVImgToVideoInplace"
}
},
"792:297": {
"class_type": "ResizeImageMaskNode",
"inputs": {
"input": [
"666",
0
],
"resize_type": "scale dimensions",
"resize_type.crop": "center",
"resize_type.height": [
"792:324",
0
],
"resize_type.width": [
"792:330",
0
],
"scale_method": "lanczos"
},
"_meta": {
"title": "Resize Image/Mask"
}
},
"792:298": {
"class_type": "KSamplerSelect",
"inputs": {
"sampler_name": "euler_ancestral_cfg_pp"
},
"_meta": {
"title": "KSamplerSelect"
}
},
"792:299": {
"class_type": "ComfyMathExpression",
"inputs": {
"expression": "a/2",
"values.a": [
"792:330",
0
]
},
"_meta": {
"title": "Math Expression"
}
},
"792:301": {
"class_type": "ComfyMathExpression",
"inputs": {
"expression": "a/2",
"values.a": [
"792:324",
0
]
},
"_meta": {
"title": "Math Expression"
}
},
"792:302": {
"class_type": "EmptyLTXVLatentVideo",
"inputs": {
"batch_size": 1,
"height": [
"792:301",
1
],
"length": [
"792:329",
1
],
"width": [
"792:299",
1
]
},
"_meta": {
"title": "Empty LTXV Latent Video"
}
},
"792:303": {
"class_type": "LTXVAudioVAEDecode",
"inputs": {
"audio_vae": [
"792:785",
0
],
"samples": [
"792:311",
1
]
},
"_meta": {
"title": "LTXV Audio VAE Decode"
}
},
"792:304": {
"class_type": "ComfyMathExpression",
"inputs": {
"expression": "a",
"values.a": [
"792:323",
0
]
},
"_meta": {
"title": "Math Expression"
}
},
"792:305": {
"class_type": "PrimitiveBoolean",
"inputs": {
"value": False
},
"_meta": {
"title": "Switch to Text to Video?"
}
},
"792:306": {
"class_type": "CLIPTextEncode",
"inputs": {
"clip": [
"792:789",
0
],
"text": [
"792:319",
0
]
},
"_meta": {
"title": "CLIP Text Encode (Prompt)"
}
},
"792:307": {
"class_type": "LTXVConditioning",
"inputs": {
"frame_rate": [
"792:304",
0
],
"negative": [
"792:314",
0
],
"positive": [
"792:306",
0
]
},
"_meta": {
"title": "LTXVConditioning"
}
},
"792:308": {
"class_type": "ManualSigmas",
"inputs": {
"sigmas": "1.0, 0.99375, 0.9875, 0.98125, 0.975, 0.909375, 0.725, 0.421875, 0.0"
},
"_meta": {
"title": "ManualSigmas"
}
},
"792:309": {
"class_type": "LTXVSeparateAVLatent",
"inputs": {
"av_latent": [
"792:291",
0
]
},
"_meta": {
"title": "Separate AV Latent"
}
},
"792:310": {
"class_type": "SamplerCustomAdvanced",
"inputs": {
"guider": [
"792:290",
0
],
"latent_image": [
"792:287",
0
],
"noise": [
"792:285",
0
],
"sampler": [
"792:288",
0
],
"sigmas": [
"792:289",
0
]
},
"_meta": {
"title": "SamplerCustomAdvanced"
}
},
"792:311": {
"class_type": "LTXVSeparateAVLatent",
"inputs": {
"av_latent": [
"792:310",
0
]
},
"_meta": {
"title": "Separate AV Latent"
}
},
"792:312": {
"class_type": "CreateVideo",
"inputs": {
"audio": [
"792:303",
0
],
"bit_depth": "auto",
"codec": "none",
"color_space": "sRGB",
"fps": [
"792:304",
0
],
"images": [
"792:316",
0
]
},
"_meta": {
"title": "Create Video"
}
},
"792:314": {
"class_type": "CLIPTextEncode",
"inputs": {
"clip": [
"792:789",
0
],
"text": "pc game, console game, video game, cartoon, childish, ugly"
},
"_meta": {
"title": "CLIP Text Encode (Prompt)"
}
},
"792:315": {
"class_type": "CFGGuider",
"inputs": {
"cfg": 1,
"model": [
"792:791",
0
],
"negative": [
"792:307",
1
],
"positive": [
"792:307",
0
]
},
"_meta": {
"title": "CFG Guider"
}
},
"792:316": {
"class_type": "VAEDecodeTiled",
"inputs": {
"overlap": 64,
"samples": [
"792:311",
0
],
"temporal_overlap": 4,
"temporal_size": 4096,
"tile_size": 768,
"vae": [
"792:788",
2
]
},
"_meta": {
"title": "VAE Decode (Tiled)"
}
},
"792:319": {
"class_type": "PrimitiveStringMultiline",
"inputs": {
"value": [
"754",
0
]
},
"_meta": {
"title": "Prompt"
}
},
"792:323": {
"class_type": "PrimitiveInt",
"inputs": {
"value": 24
},
"_meta": {
"title": "Frame Rate"
}
},
"792:324": {
"class_type": "PrimitiveInt",
"inputs": {
"value": 720
},
"_meta": {
"title": "Height"
}
},
"792:325": {
"class_type": "LTXVImgToVideoInplace",
"inputs": {
"bypass": [
"792:305",
0
],
"image": [
"792:334",
0
],
"latent": [
"792:302",
0
],
"strength": 0.7,
"vae": [
"792:788",
2
]
},
"_meta": {
"title": "LTXVImgToVideoInplace"
}
},
"792:326": {
"class_type": "LTXVConcatAVLatent",
"inputs": {
"audio_latent": [
"792:327",
0
],
"video_latent": [
"792:325",
0
]
},
"_meta": {
"title": "Concat AV Latent"
}
},
"792:327": {
"class_type": "SetLatentNoiseMask",
"inputs": {
"mask": [
"792:333",
0
],
"samples": [
"792:328",
0
]
},
"_meta": {
"title": "Set Latent Noise Mask"
}
},
"792:328": {
"class_type": "LTXVAudioVAEEncode",
"inputs": {
"audio": [
"792:332",
0
],
"audio_vae": [
"792:785",
0
]
},
"_meta": {
"title": "LTXV Audio VAE Encode"
}
},
"792:329": {
"class_type": "ComfyMathExpression",
"inputs": {
"expression": "a * b + 1",
"values.a": [
"792:331",
0
],
"values.b": [
"792:323",
0
]
},
"_meta": {
"title": "Math Expression"
}
},
"792:330": {
"class_type": "PrimitiveInt",
"inputs": {
"value": 1280
},
"_meta": {
"title": "Width"
}
},
"792:331": {
"class_type": "PrimitiveFloat",
"inputs": {
"value": 9
},
"_meta": {
"title": "Duration"
}
},
"792:332": {
"class_type": "TrimAudioDuration",
"inputs": {
"audio": [
"667:18",
0
],
"duration": [
"792:331",
0
],
"start_index": 0
},
"_meta": {
"title": "Trim Audio Duration"
}
},
"792:333": {
"class_type": "SolidMask",
"inputs": {
"height": [
"792:324",
0
],
"value": 0,
"width": [
"792:330",
0
]
},
"_meta": {
"title": "Create Solid Mask"
}
},
"792:334": {
"class_type": "LTXVPreprocess",
"inputs": {
"image": [
"792:294",
0
],
"img_compression": 18
},
"_meta": {
"title": "LTXV Preprocess"
}
},
"792:785": {
"class_type": "LTXVAudioVAELoader",
"inputs": {
"ckpt_name": "ltx-2.3-22b-dev-fp8.safetensors"
},
"_meta": {
"title": "Load LTXV Audio VAE"
}
},
"792:786": {
"class_type": "LatentUpscaleModelLoader",
"inputs": {
"model_name": "ltx-2.3-spatial-upscaler-x2-1.1.safetensors"
},
"_meta": {
"title": "Load Latent Upscale Model"
}
},
"792:787": {
"class_type": "LoraLoaderModelOnly",
"inputs": {
"lora_name": "ltx-2.3-22b-distilled-lora-384.safetensors",
"model": [
"792:788",
0
],
"strength_model": 0.5
},
"_meta": {
"title": "Load LoRA"
}
},
"792:788": {
"class_type": "CheckpointLoaderSimple",
"inputs": {
"ckpt_name": "ltx-2.3-22b-dev-fp8.safetensors"
},
"_meta": {
"title": "Load Checkpoint"
}
},
"792:789": {
"class_type": "LTXAVTextEncoderLoader",
"inputs": {
"ckpt_name": "ltx-2.3-22b-dev-fp8.safetensors",
"device": "default",
"text_encoder": "gemma_3_12B_it_fp4_mixed.safetensors"
},
"_meta": {
"title": "Load LTXV Audio Text Encoder"
}
},
"792:791": {
"class_type": "LoraLoaderModelOnly",
"inputs": {
"lora_name": "LTX2.3_AudioReactiveV1.safetensors",
"model": [
"792:787",
0
],
"strength_model": 1
},
"_meta": {
"title": "Load LoRA"
}
}
})
# run() also passes api_key to the partner nodes in this workflow.
job = client.run(workflow, api_key=api_key)Perguntas e respostas
O nó RegexExtract captura tudo antes de '---' como tags de estilo para o LTX 2.3 e tudo depois como letras para o ACE Step 1.5 XL Turbo. Isso garante que cada modelo receba um prompt específico a partir de uma única saída do GeminiNode.
Use a intensidade do Audio‑Reactive LoRA no grupo LTX 2.3. Recomenda-se o intervalo de 1.0–2.0; tente ~1.4 ou mais para efeitos mais fortes. Áudios mais dinâmicos (com percussão, quedas e variações) também produzem mais movimento.
Sim. Substitua ou ignore o ramo ACE e envie seu áudio para a entrada do LTX 2.3 + Audio‑Reactive LoRA (use um nó de carregamento de áudio compatível no ComfyUI). Certifique-se de que a duração do vídeo corresponda à da faixa.
Comece com uma imagem limpa e de alta qualidade; mantenha as tags de estilo concisas e específicas; use resoluções compatíveis com sua GPU; e itere: ajuste tags, intensidade do LoRA e re-renderize. PreviewAny ajuda a validar prompts e quadros antes do salvamento final.