txt2vid_ltx2

Generate video (and audio) with LTX-2.3

Description

Distilled text-to-video generation: encodes the prompt with Gemma3 + connectors, denoises joint audio/video latents over the official 8-step distilled schedule (no classifier-free guidance), decodes the video with the causal VAE and the audio with the audio VAE + BWE vocoder, and optionally muxes both into an MP4.

Usage

txt2vid_ltx2(prompt, pipeline, text_encoder = NULL, tokenizer = NULL,
             prompt_embeds = NULL, width = 768L, height = 512L,
             num_frames = 121L, frame_rate = 24,
             sigmas = ltx23_distilled_sigmas(), guidance_scale = 1,
             seed = NULL, device = "cuda", dtype = "bfloat16", filename = NULL,
             max_sequence_length = 1024L, decode_video = TRUE,
             decode_audio = TRUE, two_stage = FALSE, upsampler = NULL,
             adain_factor = 1, tone_map_compression = 0, phase_offload = TRUE,
             image = NULL, condition_video = NULL, conditioning_frames = 9L,
             cond_noise_scale = 0, audio = NULL, verbose = TRUE)

Arguments

  • prompt: Character. The prompt.
  • pipeline: An ltx23_pipeline from ltx23_load_pipeline.
  • prompt_embeds: Optional precomputed list with prompt_embeds (raw stacked Gemma3 states) and prompt_attention_mask; bypasses the text encoder.
  • num_frames: Integer. 8k + 1 frames (e.g. 121).
  • frame_rate: Numeric. Frames per second.
  • sigmas: Numeric vector. Denoising schedule (default: official distilled schedule; must end in 0).
  • guidance_scale: Numeric. Only 1 (no CFG) is supported; the distilled checkpoints are trained for CFG-free sampling.
  • seed: Integer or NULL.
  • device: Character. Compute device for the denoising loop.
  • dtype: Character. Model compute dtype (“bfloat16” or “float32”).
  • filename: Character or NULL. Output video path (.mp4). Audio is muxed in when the av package is available.
  • max_sequence_length: Integer. Text token length (multiple of 128).
  • two_stage: Logical. Generate at half resolution, upsample the latents 2x spatially, and refine over the stage-2 schedule (resolution must then be a multiple of 64; requires upsampler).
  • upsampler: An ltx23_latent_upsampler (see ltx23_load_upsampler).
  • adain_factor: Numeric. AdaIN blend of the upsampled latents toward the stage-1 statistics (0 disables).
  • tone_map_compression: Numeric in [0, 1]. Optional latent tone mapping before stage 2.
  • phase_offload: Logical. Move each small component to the compute device only for its phase (text encoding, upsampling, decoding) and back to the CPU afterwards, keeping the denoise phase as the sole GPU tenant.
  • image: Optional start image for image-to-video: a PNG/JPEG path or an [H, W, 3] array in [0, 1]. The image conditions the first frame; the rest of the video is generated (reference i2v).
  • condition_video: Optional continuation source: a video path (its trailing conditioning_frames frames are used) or an [F, H, W, 3] array. The clip’s tail becomes the frozen prefix of the new video, so the output’s first conditioning_frames frames overlap the source (trim or crossfade when concatenating).
  • conditioning_frames: Integer. Trailing pixel frames taken from condition_video (8k + 1, default 9 = 2 latent frames).
  • cond_noise_scale: Numeric in [0, 1]. Optional partial noising of the conditioned tokens (0 = keep them exactly).
  • audio: Optional conditioning audio for audio-driven generation (lip sync): a file path (decoded via av) or a matrix [2, samples] in [-1, 1] at 16 kHz. The audio is encoded into clean, frozen audio latents that the video attends to while denoising, and the original samples are muxed into the output (audio decoding is skipped).
  • verbose: Logical.
  • text_encoder,tokenizer: Gemma3 model and tokenizer (or paths; see load_gemma3_text_encoder and gemma3_tokenizer). Ignored when prompt_embeds is supplied.
  • width,height: Integers. Output resolution (multiples of 32).
  • decode_video,decode_audio: Logicals. Decode the respective latents (disable for latent-space work).

Value

Invisibly, a list with video (array [frames, height, width, 3] in [0, 1]), audio (matrix [2, samples] in [-1, 1]), sample_rate, and the raw latents.