Imagen → Canción → Vídeo Reactivo al Audio
Crea canciones con ACE Step 1.5 XL y videos audio-reactivos con LTX 2.3 y el Audio-Reactive LoRA.
Acerca de
Este flujo de trabajo convierte una sola imagen en un video audio‑reactivo con música de fondo combinando generación de música desde texto (ACE Step 1.5 XL Turbo) con animación de imagen a video (LTX 2.3) mejorada con un LoRA audio‑reactivo. Proporcionas una imagen y un brief creativo corto; el flujo usa un GeminiNode para redactar tanto “style tags” para los visuales como “lyrics” para la canción en una sola pasada. Luego, un nodo RegexExtract divide ese texto en '---': todo lo anterior se convierte en etiquetas de estilo visual para LTX 2.3, y todo lo posterior se convierte en letras que se envían a ACE Step 1.5 XL Turbo. La canción generada se guarda con SaveAudioMP3 y luego impulsa la reactividad del video, que se guarda con SaveVideo.
Técnicamente, la canalización está organizada en grupos: Load Image, Create ACE Step 1.5 Prompts, Create Audio‑Reactive LoRA Prompt, Regex Extract, ACE Step 1.5 XL y LTX 2.3 con Audio‑Reactive LoRA. El LoRA audio‑reactivo (ver la ficha del modelo enlazada) añade efectos de movimiento que siguen la música; ajusta su fuerza entre 1.0 y 2.0 (1.4+ para efectos más marcados) para mejores resultados. PreviewAny te ayuda a revisar salidas intermedias (prompts, audio y frames) antes del render final, mientras que MarkdownNote incluye consejos rápidos dentro del grafo. El resultado es una canalización optimizada de imagen → canción → video audio‑reactivo fácil de iterar: ajusta style tags y lyrics, vuelve a generar la pista y vuelve a renderizar el video hasta que encaje con tu intención creativa.
Lo que proporcionas
- Leyendo el flujo de trabajo
Se ejecuta en ComfyUI. Este flujo de trabajo no puede ejecutarse en esta página. Ábrelo en Comfy Cloud con todo cargado o descárgalo para tu propio ComfyUI.
Probar en CloudDentro del flujo de trabajo
Una vista previa de solo lectura del gráfico de nodos. Ábrelo en Comfy Cloud o descárgalo para editar los nodos.
Loading the graph
Open full-size workflow previewArchivos de este flujo de trabajo
- Autor
- ComfyUI
Requiere nodos personalizados
- Leyendo el flujo de trabajo
Usa este flujo de trabajo desde código
Se genera a partir del gráfico y las entradas del flujo de trabajo cuando es posible; de lo contrario, se muestran ejemplos almacenados. Revisa las credenciales y los endpoints antes de ejecutar el código, ya que puede generar cargos.
What you need
- Runs on
- Comfy Cloud
- Clave
- COMFY_API_KEY
Requiere un plan de pago de Cloud y créditos.
# Python 3.10+: pip install comfy-sdk==0.4.0
import os
from comfy_sdk import Comfy
api_key = os.environ.get("COMFY_API_KEY", "").strip()
if not api_key: raise RuntimeError("Set COMFY_API_KEY")
os.environ.setdefault("COMFY_BASE_URL", "https://cloud.comfy.org")
client = Comfy(api_key=api_key)
workflow = client.workflows.from_json({
"666": {
"class_type": "LoadImage",
"inputs": {
"image": "660fdce25fe105c5982159389b5fdf67c12d941d49be78defcf77fd1560ca507.png"
},
"_meta": {
"title": "Load Image"
}
},
"668": {
"class_type": "SaveAudioMP3",
"inputs": {
"audio": [
"667:18",
0
],
"filename_prefix": "audio/ACE_Step1.5_xl_turbo",
"quality": "320k"
},
"_meta": {
"title": "Save Audio (MP3) (DEPRECATED)"
}
},
"670": {
"class_type": "RegexExtract",
"inputs": {
"case_insensitive": True,
"dotall": False,
"group_index": 1,
"mode": "First Group",
"multiline": False,
"regex_pattern": "(?s)(?<=\\n---\\n)([\\s\\S]*)$",
"string": [
"673",
0
]
},
"_meta": {
"title": "Extract Text"
}
},
"671": {
"class_type": "PreviewAny",
"inputs": {
"source": [
"670",
0
]
},
"_meta": {
"title": "Preview as Text"
}
},
"672": {
"class_type": "PreviewAny",
"inputs": {
"source": [
"674",
0
]
},
"_meta": {
"title": "Preview as Text"
}
},
"673": {
"class_type": "GeminiNode",
"inputs": {
"images": [
"666",
0
],
"model": "gemini-3-1-flash-lite",
"prompt": "medieval ambient",
"seed": 65482163754128,
"system_prompt": "**Role:** You are a professional Music Architect for **ACE-Step 1.5**. \n\n**Task:** Generate a music creation plan.\n\n**Strict Output Constraint (IMPORTANT):**\n1. Return the output as **RAW PLAIN TEXT** only.\n2. DO NOT use Markdown code blocks (no backticks ```).\n3. DO NOT use any headers, labels, or bold text.\n4. Use ONLY the separator `---` to divide the two parts.\n5. Part 1: Comma-separated style keywords.\n6. Part 2: Structured lyrics with [Tags].\n\n**My Topic/Requirement:**\n[USER INPUT]"
},
"_meta": {
"title": "Google Gemini"
}
},
"674": {
"class_type": "RegexExtract",
"inputs": {
"case_insensitive": True,
"dotall": False,
"group_index": 1,
"mode": "First Group",
"multiline": False,
"regex_pattern": "(?sm)^(.*?)^---$(.*)",
"string": [
"673",
0
]
},
"_meta": {
"title": "Extract Text"
}
},
"683": {
"class_type": "SaveVideo",
"inputs": {
"filename_prefix": "video/ComfyUI",
"format": "auto",
"format.codec": "auto",
"video": [
"792:312",
0
]
},
"_meta": {
"title": "Save Video"
}
},
"752": {
"class_type": "PreviewAny",
"inputs": {
"source": [
"673",
0
]
},
"_meta": {
"title": "Preview as Text"
}
},
"753": {
"class_type": "GeminiNode",
"inputs": {
"audio": [
"667:18",
0
],
"images": [
"666",
0
],
"model": "gemini-3-1-flash-lite",
"prompt": "Continuous audio-reactive video.",
"seed": 1013654870151104,
"system_prompt": "You are an expert prompt engineer specializing in LTX-2.3 video generation prompts, especially audio-reactive LoRA workflows.\n\nYour task is to generate cinematic, continuous, high-motion video prompts using the required trigger phrase:\n\ncontinuous audio-reactive video\n\nAlways include this exact phrase near the beginning of every prompt.\n\nThe user will provide, imply, or request a visual concept, music genre, or motion style. Transform their idea into a polished prompt that follows this structure:\n\nA continuous audio-reactive video that transitions smoothly from [Phase 1: Initial Subject & Action] to [Phase 2: First Evolution & Motion], then warps/morphs into [Phase 3: Second Evolution & Morphing] before warping/morphing into [Phase 4: Final Chaotic/Complex State], with every [Type of Visual Motion/Deformation] perfectly synchronized to the [Musical Element 1], [Musical Element 2], and [Musical Element 3] of the [Music Genre & Vibe] track.\n\nOutput rules:\n- Always answer with only one finished prompt.\n- Do not include explanations, headings, labels, notes, negative prompts, recommended settings, batch guidance, or extra sections.\n- Do not ask clarifying questions.\n- Make a reasonable creative choice when the user gives limited detail.\n- Always include 2 to 4 distinct phases.\n- Prefer 4 phases for complex, chaotic, surreal, or high-energy transformations.\n- Prefer 2 or 3 phases for elegant, minimal, stable, or slower concepts.\n- Make the video feel continuous, not like separate scenes.\n- Use smooth transition language such as “transitions smoothly,” “melts into,” “warps into,” “morphs into,” “fractures into,” “blooms into,” or “folds into.”\n- Emphasize visible motion, rhythmic deformation, fluid morphing, and high-motion transformation.\n- Avoid static composition language unless paired with motion.\n- Avoid vague phrases like “cool visuals,” “nice effects,” or “beautiful animation.”\n- Do not overcomplicate the scene with too many unrelated subjects.\n- Keep the subject visually trackable as it evolves.\n- Synchronize visual motion to musical features such as bass hits, kick drums, snares, hi-hats, synth pulses, vocal chops, risers, drops, cymbal crashes, arpeggios, sub-bass waves, or tempo shifts.\n- For image-to-video requests, preserve the starting image’s core subject, composition, color palette, and camera angle while adding audio-reactive motion and morphing.\n\nStyle:\nUse vivid, cinematic, motion-heavy language. Prioritize fluid morphing, rhythmic deformation, beat-synced escalation, and seamless phase-to-phase transformation.\n\nFinal answer format:\nReturn only the completed prompt as plain text."
},
"_meta": {
"title": "Google Gemini"
}
},
"754": {
"class_type": "PreviewAny",
"inputs": {
"source": [
"753",
0
]
},
"_meta": {
"title": "Preview as Text"
}
},
"667:104": {
"class_type": "UNETLoader",
"inputs": {
"unet_name": "acestep_v1.5_xl_turbo_bf16.safetensors",
"weight_dtype": "default"
},
"_meta": {
"title": "Load Diffusion Model"
}
},
"667:105": {
"class_type": "DualCLIPLoader",
"inputs": {
"clip_name1": "qwen_0.6b_ace15.safetensors",
"clip_name2": "qwen_4b_ace15.safetensors",
"device": "default",
"type": "ace"
},
"_meta": {
"title": "Load CLIP (Dual)"
}
},
"667:106": {
"class_type": "VAELoader",
"inputs": {
"vae_name": "ace_1.5_vae.safetensors"
},
"_meta": {
"title": "Load VAE"
}
},
"667:109": {
"class_type": "PrimitiveInt",
"inputs": {
"value": 0
},
"_meta": {
"title": "Int (Seed)"
}
},
"667:18": {
"class_type": "VAEDecodeAudio",
"inputs": {
"samples": [
"667:3",
0
],
"vae": [
"667:106",
0
]
},
"_meta": {
"title": "VAE Decode Audio"
}
},
"667:3": {
"class_type": "KSampler",
"inputs": {
"cfg": 1,
"denoise": 1,
"latent_image": [
"667:98",
0
],
"model": [
"667:78",
0
],
"negative": [
"667:47",
0
],
"positive": [
"667:94",
0
],
"sampler_name": "euler",
"scheduler": "simple",
"seed": [
"667:109",
0
],
"steps": 8
},
"_meta": {
"title": "KSampler"
}
},
"667:47": {
"class_type": "ConditioningZeroOut",
"inputs": {
"conditioning": [
"667:94",
0
]
},
"_meta": {
"title": "Conditioning Zero Out"
}
},
"667:78": {
"class_type": "ModelSamplingAuraFlow",
"inputs": {
"model": [
"667:104",
0
],
"sampling": "flow",
"shift": 3
},
"_meta": {
"title": "ModelSamplingAuraFlow"
}
},
"667:94": {
"class_type": "TextEncodeAceStepAudio1.5",
"inputs": {
"bpm": 95,
"cfg_scale": 2,
"clip": [
"667:105",
0
],
"duration": 60,
"generate_audio_codes": True,
"keyscale": "E minor",
"language": "en",
"lyrics": [
"671",
0
],
"min_p": 0,
"seed": [
"667:109",
0
],
"tags": [
"672",
0
],
"temperature": 0.85,
"timesignature": "4",
"top_k": 0,
"top_p": 0.9
},
"_meta": {
"title": "TextEncodeAceStepAudio1.5"
}
},
"667:98": {
"class_type": "EmptyAceStep1.5LatentAudio",
"inputs": {
"batch_size": 1,
"seconds": 60
},
"_meta": {
"title": "Empty Ace Step 1.5 Latent Audio"
}
},
"792:285": {
"class_type": "RandomNoise",
"inputs": {
"noise_seed": 42
},
"_meta": {
"title": "RandomNoise"
}
},
"792:286": {
"class_type": "RandomNoise",
"inputs": {
"noise_seed": 784235873879537
},
"_meta": {
"title": "RandomNoise"
}
},
"792:287": {
"class_type": "LTXVConcatAVLatent",
"inputs": {
"audio_latent": [
"792:309",
1
],
"video_latent": [
"792:296",
0
]
},
"_meta": {
"title": "Concat AV Latent"
}
},
"792:288": {
"class_type": "KSamplerSelect",
"inputs": {
"sampler_name": "euler_cfg_pp"
},
"_meta": {
"title": "KSamplerSelect"
}
},
"792:289": {
"class_type": "ManualSigmas",
"inputs": {
"sigmas": "0.85, 0.7250, 0.4219, 0.0"
},
"_meta": {
"title": "ManualSigmas"
}
},
"792:290": {
"class_type": "CFGGuider",
"inputs": {
"cfg": 1,
"model": [
"792:791",
0
],
"negative": [
"792:292",
1
],
"positive": [
"792:292",
0
]
},
"_meta": {
"title": "CFG Guider"
}
},
"792:291": {
"class_type": "SamplerCustomAdvanced",
"inputs": {
"guider": [
"792:315",
0
],
"latent_image": [
"792:326",
0
],
"noise": [
"792:286",
0
],
"sampler": [
"792:298",
0
],
"sigmas": [
"792:308",
0
]
},
"_meta": {
"title": "SamplerCustomAdvanced"
}
},
"792:292": {
"class_type": "LTXVCropGuides",
"inputs": {
"latent": [
"792:309",
0
],
"negative": [
"792:307",
1
],
"positive": [
"792:307",
0
]
},
"_meta": {
"title": "LTXVCropGuides"
}
},
"792:294": {
"class_type": "ResizeImagesByLongerEdge",
"inputs": {
"images": [
"792:297",
0
],
"longer_edge": 1536
},
"_meta": {
"title": "Resize Images by Longer Edge (DEPRECATED)"
}
},
"792:295": {
"class_type": "LTXVLatentUpsampler",
"inputs": {
"samples": [
"792:309",
0
],
"upscale_model": [
"792:786",
0
],
"vae": [
"792:788",
2
]
},
"_meta": {
"title": "LTXVLatentUpsampler"
}
},
"792:296": {
"class_type": "LTXVImgToVideoInplace",
"inputs": {
"bypass": [
"792:305",
0
],
"image": [
"792:334",
0
],
"latent": [
"792:295",
0
],
"strength": 1,
"vae": [
"792:788",
2
]
},
"_meta": {
"title": "LTXVImgToVideoInplace"
}
},
"792:297": {
"class_type": "ResizeImageMaskNode",
"inputs": {
"input": [
"666",
0
],
"resize_type": "scale dimensions",
"resize_type.crop": "center",
"resize_type.height": [
"792:324",
0
],
"resize_type.width": [
"792:330",
0
],
"scale_method": "lanczos"
},
"_meta": {
"title": "Resize Image/Mask"
}
},
"792:298": {
"class_type": "KSamplerSelect",
"inputs": {
"sampler_name": "euler_ancestral_cfg_pp"
},
"_meta": {
"title": "KSamplerSelect"
}
},
"792:299": {
"class_type": "ComfyMathExpression",
"inputs": {
"expression": "a/2",
"values.a": [
"792:330",
0
]
},
"_meta": {
"title": "Math Expression"
}
},
"792:301": {
"class_type": "ComfyMathExpression",
"inputs": {
"expression": "a/2",
"values.a": [
"792:324",
0
]
},
"_meta": {
"title": "Math Expression"
}
},
"792:302": {
"class_type": "EmptyLTXVLatentVideo",
"inputs": {
"batch_size": 1,
"height": [
"792:301",
1
],
"length": [
"792:329",
1
],
"width": [
"792:299",
1
]
},
"_meta": {
"title": "Empty LTXV Latent Video"
}
},
"792:303": {
"class_type": "LTXVAudioVAEDecode",
"inputs": {
"audio_vae": [
"792:785",
0
],
"samples": [
"792:311",
1
]
},
"_meta": {
"title": "LTXV Audio VAE Decode"
}
},
"792:304": {
"class_type": "ComfyMathExpression",
"inputs": {
"expression": "a",
"values.a": [
"792:323",
0
]
},
"_meta": {
"title": "Math Expression"
}
},
"792:305": {
"class_type": "PrimitiveBoolean",
"inputs": {
"value": False
},
"_meta": {
"title": "Switch to Text to Video?"
}
},
"792:306": {
"class_type": "CLIPTextEncode",
"inputs": {
"clip": [
"792:789",
0
],
"text": [
"792:319",
0
]
},
"_meta": {
"title": "CLIP Text Encode (Prompt)"
}
},
"792:307": {
"class_type": "LTXVConditioning",
"inputs": {
"frame_rate": [
"792:304",
0
],
"negative": [
"792:314",
0
],
"positive": [
"792:306",
0
]
},
"_meta": {
"title": "LTXVConditioning"
}
},
"792:308": {
"class_type": "ManualSigmas",
"inputs": {
"sigmas": "1.0, 0.99375, 0.9875, 0.98125, 0.975, 0.909375, 0.725, 0.421875, 0.0"
},
"_meta": {
"title": "ManualSigmas"
}
},
"792:309": {
"class_type": "LTXVSeparateAVLatent",
"inputs": {
"av_latent": [
"792:291",
0
]
},
"_meta": {
"title": "Separate AV Latent"
}
},
"792:310": {
"class_type": "SamplerCustomAdvanced",
"inputs": {
"guider": [
"792:290",
0
],
"latent_image": [
"792:287",
0
],
"noise": [
"792:285",
0
],
"sampler": [
"792:288",
0
],
"sigmas": [
"792:289",
0
]
},
"_meta": {
"title": "SamplerCustomAdvanced"
}
},
"792:311": {
"class_type": "LTXVSeparateAVLatent",
"inputs": {
"av_latent": [
"792:310",
0
]
},
"_meta": {
"title": "Separate AV Latent"
}
},
"792:312": {
"class_type": "CreateVideo",
"inputs": {
"audio": [
"792:303",
0
],
"bit_depth": "auto",
"codec": "none",
"color_space": "sRGB",
"fps": [
"792:304",
0
],
"images": [
"792:316",
0
]
},
"_meta": {
"title": "Create Video"
}
},
"792:314": {
"class_type": "CLIPTextEncode",
"inputs": {
"clip": [
"792:789",
0
],
"text": "pc game, console game, video game, cartoon, childish, ugly"
},
"_meta": {
"title": "CLIP Text Encode (Prompt)"
}
},
"792:315": {
"class_type": "CFGGuider",
"inputs": {
"cfg": 1,
"model": [
"792:791",
0
],
"negative": [
"792:307",
1
],
"positive": [
"792:307",
0
]
},
"_meta": {
"title": "CFG Guider"
}
},
"792:316": {
"class_type": "VAEDecodeTiled",
"inputs": {
"overlap": 64,
"samples": [
"792:311",
0
],
"temporal_overlap": 4,
"temporal_size": 4096,
"tile_size": 768,
"vae": [
"792:788",
2
]
},
"_meta": {
"title": "VAE Decode (Tiled)"
}
},
"792:319": {
"class_type": "PrimitiveStringMultiline",
"inputs": {
"value": [
"754",
0
]
},
"_meta": {
"title": "Prompt"
}
},
"792:323": {
"class_type": "PrimitiveInt",
"inputs": {
"value": 24
},
"_meta": {
"title": "Frame Rate"
}
},
"792:324": {
"class_type": "PrimitiveInt",
"inputs": {
"value": 720
},
"_meta": {
"title": "Height"
}
},
"792:325": {
"class_type": "LTXVImgToVideoInplace",
"inputs": {
"bypass": [
"792:305",
0
],
"image": [
"792:334",
0
],
"latent": [
"792:302",
0
],
"strength": 0.7,
"vae": [
"792:788",
2
]
},
"_meta": {
"title": "LTXVImgToVideoInplace"
}
},
"792:326": {
"class_type": "LTXVConcatAVLatent",
"inputs": {
"audio_latent": [
"792:327",
0
],
"video_latent": [
"792:325",
0
]
},
"_meta": {
"title": "Concat AV Latent"
}
},
"792:327": {
"class_type": "SetLatentNoiseMask",
"inputs": {
"mask": [
"792:333",
0
],
"samples": [
"792:328",
0
]
},
"_meta": {
"title": "Set Latent Noise Mask"
}
},
"792:328": {
"class_type": "LTXVAudioVAEEncode",
"inputs": {
"audio": [
"792:332",
0
],
"audio_vae": [
"792:785",
0
]
},
"_meta": {
"title": "LTXV Audio VAE Encode"
}
},
"792:329": {
"class_type": "ComfyMathExpression",
"inputs": {
"expression": "a * b + 1",
"values.a": [
"792:331",
0
],
"values.b": [
"792:323",
0
]
},
"_meta": {
"title": "Math Expression"
}
},
"792:330": {
"class_type": "PrimitiveInt",
"inputs": {
"value": 1280
},
"_meta": {
"title": "Width"
}
},
"792:331": {
"class_type": "PrimitiveFloat",
"inputs": {
"value": 9
},
"_meta": {
"title": "Duration"
}
},
"792:332": {
"class_type": "TrimAudioDuration",
"inputs": {
"audio": [
"667:18",
0
],
"duration": [
"792:331",
0
],
"start_index": 0
},
"_meta": {
"title": "Trim Audio Duration"
}
},
"792:333": {
"class_type": "SolidMask",
"inputs": {
"height": [
"792:324",
0
],
"value": 0,
"width": [
"792:330",
0
]
},
"_meta": {
"title": "Create Solid Mask"
}
},
"792:334": {
"class_type": "LTXVPreprocess",
"inputs": {
"image": [
"792:294",
0
],
"img_compression": 18
},
"_meta": {
"title": "LTXV Preprocess"
}
},
"792:785": {
"class_type": "LTXVAudioVAELoader",
"inputs": {
"ckpt_name": "ltx-2.3-22b-dev-fp8.safetensors"
},
"_meta": {
"title": "Load LTXV Audio VAE"
}
},
"792:786": {
"class_type": "LatentUpscaleModelLoader",
"inputs": {
"model_name": "ltx-2.3-spatial-upscaler-x2-1.1.safetensors"
},
"_meta": {
"title": "Load Latent Upscale Model"
}
},
"792:787": {
"class_type": "LoraLoaderModelOnly",
"inputs": {
"lora_name": "ltx-2.3-22b-distilled-lora-384.safetensors",
"model": [
"792:788",
0
],
"strength_model": 0.5
},
"_meta": {
"title": "Load LoRA"
}
},
"792:788": {
"class_type": "CheckpointLoaderSimple",
"inputs": {
"ckpt_name": "ltx-2.3-22b-dev-fp8.safetensors"
},
"_meta": {
"title": "Load Checkpoint"
}
},
"792:789": {
"class_type": "LTXAVTextEncoderLoader",
"inputs": {
"ckpt_name": "ltx-2.3-22b-dev-fp8.safetensors",
"device": "default",
"text_encoder": "gemma_3_12B_it_fp4_mixed.safetensors"
},
"_meta": {
"title": "Load LTXV Audio Text Encoder"
}
},
"792:791": {
"class_type": "LoraLoaderModelOnly",
"inputs": {
"lora_name": "LTX2.3_AudioReactiveV1.safetensors",
"model": [
"792:787",
0
],
"strength_model": 1
},
"_meta": {
"title": "Load LoRA"
}
}
})
# run() also passes api_key to the partner nodes in this workflow.
job = client.run(workflow, api_key=api_key)Preguntas y respuestas
El nodo RegexExtract captura todo lo anterior a '---' como etiquetas de estilo para LTX 2.3 y todo lo posterior como letras para ACE Step 1.5 XL Turbo. Esto garantiza que cada modelo reciba un prompt específico desde una sola salida de GeminiNode.
Usa la fuerza del Audio‑Reactive LoRA en el grupo LTX 2.3. Se recomienda un rango de 1.0–2.0; prueba ~1.4 o más para efectos más fuertes. Un audio más dinámico (percusión clara, cambios y subidas) también producirá más movimiento.
Sí. Reemplaza o omite la rama ACE y alimenta tu audio en la ruta de entrada de LTX 2.3 + Audio‑Reactive LoRA (usa un nodo de carga de audio compatible en ComfyUI). Asegúrate de que la duración del video coincida con la de tu pista.
Comienza con una imagen limpia y de alta calidad; mantén las etiquetas de estilo concisas y específicas; usa resoluciones razonables para tu GPU; e itera: ajusta etiquetas de estilo, fuerza de LoRA y vuelve a renderizar. PreviewAny ayuda a validar prompts y frames antes del guardado final.