https://cornball.ai/docs/diffuser/Recent content in diffuseR on cornball.aiHugo -- gohugo.ioenhttps://cornball.ai/docs/diffuser/auto_devices/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/auto_devices/Auto-Configure Device AssignmentDescriptionAutomatically determines optimal device configuration for diffusion model components based on available VRAM and GPU architecture. Uses gpuctl for detection if available, otherwise falls back to sensible defaults. Usageauto_devices(model = "sdxl", strategy = "auto") Arguments model: Character. Model type: “sd21” or “sdxl”. strategy: Character. Memory strategy: “auto” (default), “full_gpu”, “unet_gpu”, or “cpu_only”. See Details. DetailsStrategies: “auto”: Detect VRAM and choose best strategy (requires gpuctl) “full_gpu”: All components on CUDA (16GB+ VRAM for SDXL) “unet_gpu”: Only unet on CUDA, rest on CPU (8GB+ VRAM) “cpu_only”: All components on CPU If gpuctl is not installed, “auto” falls back to “unet_gpu” which works on most modern GPUs (8GB+ VRAM).https://cornball.ai/docs/diffuser/bpe_tokenizer/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/bpe_tokenizer/BPE TokenizerDescriptionNative R implementation of Byte-Pair Encoding tokenizer. Loads from HuggingFace tokenizer.json format. Usagebpe_tokenizer(tokenizer_path) Arguments tokenizer_path: Path to tokenizer.json or directory containing it. ValueA bpe_tokenizer object.https://cornball.ai/docs/diffuser/clear_vram/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/clear_vram/Clear VRAM CacheDescriptionForces garbage collection and clears CUDA memory cache. Usageclear_vram(verbose = FALSE) Arguments verbose: Logical. Print memory status before/after. ValueInvisibly returns NULL. Examplesclear_vram()https://cornball.ai/docs/diffuser/clip_pooled_output/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/clip_pooled_output/Pooled CLIP output at the EOS positionDescriptionThe HF CLIPTextModel pooler_output: the final-layer-norm hidden state at the EOS token position, located by argmax over the token ids (EOS is the highest id in the CLIP vocab, and causal attention makes any padding after it irrelevant). No text projection is applied - this is what FLUX uses as pooled_projections. Usageclip_pooled_output(hidden_states, input_ids) Arguments hidden_states: Final-LN hidden states [B, S, D] from text_encoder_native (with apply_final_ln = TRUE).https://cornball.ai/docs/diffuser/cliptokenizer/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/cliptokenizer/Tokenize a promptDescriptionTokenize a prompt UsageCLIPTokenizer(prompt, merges = system.file("tokenizer/merges.txt", package = "diffuseR"), vocab_file = system.file("tokenizer/vocab.json", package = "diffuseR"), pad_token = 0L) Arguments prompt: A character string prompt describing the image to generate. merges: Path to the merges file (BPE merges). vocab_file: Path to the vocabulary file (token->id mapping). pad_token: The token ID used for padding (default is 0). ValueA 2D torch tensor of shape c(1, 77) containing the token IDs.https://cornball.ai/docs/diffuser/ddim_scheduler_create/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ddim_scheduler_create/Create a DDIM SchedulerDescriptionCreates a Denoising Diffusion Implicit Models (DDIM) scheduler for use with diffusion models. DDIM schedulers provide a deterministic sampling process that offers faster inference compared to DDPM while maintaining high quality outputs. Usageddim_scheduler_create(num_train_timesteps = 1000, num_inference_steps = 50, eta = 0, beta_schedule = c("linear", "scaled_linear", "cosine"), beta_start = 0.00085, beta_end = 0.012, rescale_betas_zero_snr = FALSE, dtype = torch::torch_float32(), device = c(torch::torch_device("cpu"), torch::torch_device("cuda"))) Arguments num_train_timesteps: Integer. The number of diffusion steps used to train the model.https://cornball.ai/docs/diffuser/ddim_scheduler_step/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ddim_scheduler_step/Perform a DDIM scheduler stepDescriptionPerforms a single denoising step using the DDIM (Denoising Diffusion Implicit Models) algorithm. This function takes the output from a diffusion model at a specific timestep and computes the previous (less noisy) sample in the diffusion process. Usageddim_scheduler_step(model_output, timestep, sample, schedule, eta = 0, use_clipped_model_output = FALSE, thresholding = FALSE, generator = NULL, variance_noise = NULL, clip_sample = FALSE, set_alpha_to_one = FALSE, prediction_type = c("epsilon", "sample", "v_prediction"), dtype = torch::torch_float32(), device = "cpu") Arguments model_output: Numeric array.https://cornball.ai/docs/diffuser/decode_bpe/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/decode_bpe/Decode token IDs to textDescriptionDecode token IDs to text Usagedecode_bpe(tokenizer, ids, skip_special_tokens = TRUE) Arguments tokenizer: A bpe_tokenizer object. ids: Integer vector or matrix of token IDs. skip_special_tokens: Logical. Skip special tokens in output. ValueCharacter string or vector.https://cornball.ai/docs/diffuser/download_component/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/download_component/Download a single TorchScript model componentDescriptionDownloads a specific model component file (e.g., UNet, decoder, text encoder) using hfhub::hub_download() from the cornball-ai dataset repos. Usagedownload_component(model_name = "sd21", component, device = "cpu", overwrite = FALSE, show_progress = TRUE) Arguments model_name: Character string, the name of the model (e.g., "sd21"). component: Character string, the component to download (e.g., "unet", "decoder"). device: Character string, the device type (e.g., "cpu" or "cuda"). overwrite: Logical; if TRUE, force re-download even if cached.https://cornball.ai/docs/diffuser/download_flux1/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/download_flux1/Download FLUX.1-schnell and build the quantized artifactDescriptionSkips work already done: a valid quantized manifest short-circuits the transformer download; cached files are not re-fetched. Needs HF_TOKEN set for the gated repo (see the error message it raises without one). The bf16 transformer source (~24 GB in the HuggingFace cache) may be deleted after quantization. Usagedownload_flux1(quantize = TRUE, precision = c("nf4", "fp8"), output_dir = NULL, text_encoders = TRUE, verbose = TRUE) Arguments quantize: Logical.https://cornball.ai/docs/diffuser/download_flux2_klein/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/download_flux2_klein/Download FLUX.2-klein-4B and build the quantized artifactDescriptionSkips work already done: a valid quantized manifest short-circuits the transformer download; cached files are not re-fetched. No token is needed (the repo is ungated). The bf16 transformer source (~7.8 GB in the HuggingFace cache) may be deleted after quantization. Usagedownload_flux2_klein(quantize = TRUE, precision = c("fp8", "nf4"), output_dir = NULL, text_encoders = TRUE, verbose = TRUE) Arguments quantize: Logical. Build the quantized artifact. precision: “fp8” (~4 GB, GPU-resident; near-bf16 quality) or “nf4” (~2.https://cornball.ai/docs/diffuser/download_ltx2/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/download_ltx2/Download the LTX-2.3 checkpoint and build the fp8 artifactDescriptionSkips work that is already done: a valid fp8 manifest short-circuits everything; a cached 46 GB source skips the download. The source file may be deleted after quantization (it is never removed automatically). Usagedownload_ltx2(quantize = TRUE, output_dir = file.path(tools::R_user_dir("diffuseR", "data"), "ltx2.3-fp8"), text_encoder = TRUE, verbose = TRUE) Arguments quantize: Logical. Build the fp8 artifact after downloading. output_dir: Directory for the fp8 artifact. text_encoder: Logical.https://cornball.ai/docs/diffuser/download_model/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/download_model/Download TorchScript model files for Stable DiffusionDescriptionDownloads the required model files (e.g., UNet, decoder, text encoder) for a given Stable Diffusion model using hfhub::hub_download(). Usagedownload_model(model_name = "sd21", devices = list(unet = "cpu", decoder = "cpu", text_encoder = "cpu"), unet_dtype_str = NULL, overwrite = FALSE, show_progress = TRUE, download_models = FALSE) Arguments model_name: Name of the model (e.g., “sd21” for stable-diffusion-2-1) devices: Either a single device string or a named list with elements ‘unet’, ‘decoder’, ’text_encoder’; optionally ’encoder’ unet_dtype_str: Optional: “float16” or “float32” (only applies if unet uses CUDA) overwrite: If TRUE, force re-download of model files show_progress: Show download progress messages download_models: If TRUE, download the model files from HuggingFace DetailsFiles are cached by hfhub (typically ~/.https://cornball.ai/docs/diffuser/encode_bpe/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/encode_bpe/Encode text to token IDsDescriptionEncode text to token IDs Usageencode_bpe(tokenizer, text, add_special_tokens = TRUE, max_length = NULL, padding = "none", truncation = FALSE, return_tensors = "list") Arguments tokenizer: A bpe_tokenizer object. text: Character string or vector to encode. add_special_tokens: Logical. Add BOS/EOS tokens. max_length: Integer. Maximum sequence length (NULL for no limit). padding: Character. Padding strategy: “none”, “max_length”, or “longest”. truncation: Logical. Truncate to max_length. return_tensors: Character. Return type: “list” or “pt” (torch tensors).https://cornball.ai/docs/diffuser/encode_qwen/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/encode_qwen/Encode prompts with the Qwen tokenizerDescriptionWith chat_template = TRUE (the FLUX.2 klein pipeline behavior) each prompt is wrapped as a single user turn with the generation prompt and a disabled thinking block, matching apply_chat_template(..., add_generation_prompt = TRUE, enable_thinking = FALSE). Right-pads with <|endoftext|>. Usageencode_qwen(tokenizer, texts, max_length = 512L, chat_template = TRUE) Arguments tokenizer: A qwen_bpe_tokenizer. texts: Character vector of prompts. max_length: Integer. Fixed sequence length (klein: 512). NULL for no truncation/padding.https://cornball.ai/docs/diffuser/encode_unigram/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/encode_unigram/Encode text with a Unigram tokenizerDescriptionNormalizes (strip-right, multi-space collapse, control whitespace to space), applies the Metaspace pre-tokenizer, segments each pre-token by Viterbi over the Unigram scores, fuses consecutive unknowns, and appends EOS. T5 semantics: right padding with <pad> (id 0), truncation to max_length - 1 before the EOS. Usageencode_unigram(tokenizer, texts, max_length = 256L, add_eos = TRUE, pad = TRUE) Arguments tokenizer: A unigram_tokenizer. texts: Character vector of prompts. max_length: Integer. Fixed sequence length (NULL for no truncation/padding).https://cornball.ai/docs/diffuser/encode_with_gemma3/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/encode_with_gemma3/Encode text with Gemma3 for LTX-2DescriptionFull pipeline for encoding text prompts using Gemma3 text encoder. Returns the raw stacked per-layer hidden states (embedding layer plus all transformer layers) for downstream connector modules, which handle normalization and projection themselves. Usageencode_with_gemma3(prompts, model = NULL, tokenizer = NULL, max_sequence_length = 1024L, device = "cuda", dtype = "float16", verbose = TRUE) Arguments prompts: Character vector of prompts. model: Gemma3 text model (or path to load from).https://cornball.ai/docs/diffuser/encode_with_qwen3/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/encode_with_qwen3/Encode prompts with the Qwen3 encoder for FLUX.2DescriptionTokenizes with the chat template, runs the encoder with the padding mask, and concatenates the requested mid-stack hidden states per token, matching Flux2KleinPipeline._get_qwen3_prompt_embeds. Usageencode_with_qwen3(prompts, model, tokenizer, max_sequence_length = 512L, out_layers = c(9L, 18L, 27L), device = NULL) Arguments prompts: Character vector. model: A qwen3_encoder. tokenizer: A qwen_bpe_tokenizer. max_sequence_length: Integer. Fixed token length (klein: 512). out_layers: Integer vector. Hidden-state layers (klein-4B: 9, 18, 27). device: Device for the input ids (defaults to the model’s).https://cornball.ai/docs/diffuser/encode_with_t5/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/encode_with_t5/Encode prompts with the T5 encoderDescriptionTokenizes with encode_unigram (right padding to max_sequence_length) and runs the encoder. Matching the FLUX reference pipeline, no attention mask is used. Usageencode_with_t5(prompts, model, tokenizer, max_sequence_length = 256L, device = NULL) Arguments prompts: Character vector. model: A t5_encoder. tokenizer: A unigram_tokenizer. max_sequence_length: Integer. Fixed token length (schnell: 256). device: Device for the input ids (defaults to the model’s). ValueTensor [length(prompts), max_sequence_length, d_model].https://cornball.ai/docs/diffuser/filename_from_prompt/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/filename_from_prompt/Generate a filename from a promptDescriptionThis function generates a filename from a prompt by removing all non-alphanumeric characters and replacing them with underscores. The filename is limited to 50 characters. If datetime is set to TRUE, the current date and time are prepended to the filename. Usagefilename_from_prompt(prompt, datetime = TRUE) Arguments prompt: A character string representing the prompt. datetime: Logical indicating whether to prepend the current date and time to the filename.https://cornball.ai/docs/diffuser/flowmatch_calculate_shift/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flowmatch_calculate_shift/Calculate shift for dynamic shiftingDescriptionComputes the shift parameter (mu) based on sequence length for resolution-dependent timestep shifting. Usageflowmatch_calculate_shift(seq_len, base_seq_len = 256L, max_seq_len = 4096L, base_shift = 0.5, max_shift = 1.15) Arguments seq_len: Integer. The sequence length (num_patches). base_seq_len: Integer. Base sequence length. Default: 256 max_seq_len: Integer. Maximum sequence length. Default: 4096 base_shift: Numeric. Base shift value. Default: 0.5 max_shift: Numeric. Maximum shift value. Default: 1.15 ValueNumeric. The computed shift value (mu).https://cornball.ai/docs/diffuser/flowmatch_scale_noise/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flowmatch_scale_noise/Scale noise for flow matching forward processDescriptionApplies the forward process in flow-matching: interpolates between the clean sample and noise. Usageflowmatch_scale_noise(sample, timestep, noise, schedule) Arguments sample: torch tensor. The clean sample. timestep: torch tensor. The current timestep. noise: torch tensor. The noise tensor. schedule: List. The FlowMatch scheduler object. Valuetorch tensor. The noisy sample at timestep t.https://cornball.ai/docs/diffuser/flowmatch_scheduler_create/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flowmatch_scheduler_create/Create a FlowMatch Euler Discrete SchedulerDescriptionCreates a FlowMatch scheduler for use with flow-matching diffusion models like LTX-2. FlowMatch schedulers use Euler integration for sampling, which is simpler and often faster than DDIM-style schedulers. Usageflowmatch_scheduler_create(num_train_timesteps = 1000L, shift = 1, use_dynamic_shifting = FALSE, base_shift = 0.5, max_shift = 1.15, base_seq_len = 256L, max_seq_len = 4096L, invert_sigmas = FALSE, shift_terminal = NULL, time_shift_type = c("exponential", "linear")) Arguments num_train_timesteps: Integer. The number of diffusion steps used to train the model.https://cornball.ai/docs/diffuser/flowmatch_scheduler_step/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flowmatch_scheduler_step/Perform a FlowMatch scheduler stepDescriptionPerforms a single denoising step using Euler integration. This is the core sampling function for FlowMatch models. Usageflowmatch_scheduler_step(model_output, timestep, sample, schedule, generator = NULL) Arguments model_output: torch tensor. The output from the diffusion model (velocity prediction). timestep: Numeric. The current timestep. sample: torch tensor. The current noisy sample. schedule: List. The FlowMatch scheduler object. generator: torch generator or NULL. Random generator for reproducibility. DetailsThe FlowMatch Euler step is remarkably simple: prev_sample = sample + dt * model_output where dt = sigma_next - sigma_current.https://cornball.ai/docs/diffuser/flowmatch_set_timesteps/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flowmatch_set_timesteps/Set timesteps for inferenceDescriptionConfigures the scheduler timesteps for a specific number of inference steps. This must be called before using the scheduler for denoising. Usageflowmatch_set_timesteps(schedule, num_inference_steps = 50L, device = "cpu", mu = NULL, sigmas = NULL, timesteps = NULL) Arguments schedule: List. The FlowMatch scheduler object. num_inference_steps: Integer. Number of denoising steps. Default: 50 device: Character or torch device. Device for tensors. Default: “cpu” mu: Numeric or NULL. Shift parameter for dynamic shifting.https://cornball.ai/docs/diffuser/flux_ada_layer_norm_continuous/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux_ada_layer_norm_continuous/FLUX continuous adaLN (final norm)DescriptionScale/shift conditioning of the final norm. Note the chunk order: scale first, then shift (the reverse of adaLN-Zero). Reference: diffusers AdaLayerNormContinuous as used by FLUX norm_out (elementwise_affine = FALSE, eps = 1e-6). Usageflux_ada_layer_norm_continuous(dim, cond_dim = dim, bias = TRUE) Arguments dim: Integer. Model dimension. cond_dim: Integer. Conditioning embedding dimension. bias: Logical. Bias on the projection (TRUE for FLUX.1, FALSE for FLUX.2).https://cornball.ai/docs/diffuser/flux_ada_layer_norm_zero/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux_ada_layer_norm_zero/FLUX adaLN-Zero modulation (double-stream)DescriptionProjects the conditioning embedding to six modulation vectors and returns the msa-modulated input plus the remaining parameters. Reference: diffusers AdaLayerNormZero. Usageflux_ada_layer_norm_zero(dim) Arguments dim: Integer. Model dimension. ValueModule whose forward(x, emb) returns list(x_norm, gate_msa, shift_mlp, scale_mlp, gate_mlp).https://cornball.ai/docs/diffuser/flux_ada_layer_norm_zero_single/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux_ada_layer_norm_zero_single/FLUX adaLN-Zero modulation (single-stream)DescriptionThree modulation vectors: shift, scale, gate. Reference: diffusers AdaLayerNormZeroSingle. Usageflux_ada_layer_norm_zero_single(dim) Arguments dim: Integer. Model dimension. ValueModule whose forward(x, emb) returns list(x_norm, gate).https://cornball.ai/docs/diffuser/flux_apply_rotary_emb/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux_apply_rotary_emb/Apply FLUX rotary embeddings to a per-head tensorDescriptionRotates adjacent element pairs of the last dimension: out = x * cos + rotate_half(x) * sin with pairs interleaved (elements 1,2 form the first complex pair). Math in float32, result cast back to the input dtype. Reference: apply_rotary_emb with use_real_unbind_dim = -1. Usageflux_apply_rotary_emb(x, freqs) Arguments x: Tensor of shape [B, H, S, D] (per-head layout). freqs: List of two tensors (cos, sin), each [S, D], from flux_pos_embed.https://cornball.ai/docs/diffuser/flux_attention/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux_attention/FLUX joint attentionDescriptionMulti-head attention with per-head RMS q/k norms and rotary position embeddings. With added_kv = TRUE (double-stream blocks) the text stream gets its own q/k/v projections and both streams attend jointly (text tokens first); the outputs are split back and projected per stream. With pre_only = TRUE (single-stream blocks) there is no output projection. Reference: diffusers FluxAttention + FluxAttnProcessor. Usageflux_attention(query_dim, heads, dim_head, added_kv = FALSE, pre_only = FALSE, eps = 1e-06, bias = TRUE) Arguments query_dim: Integer.https://cornball.ai/docs/diffuser/flux_double_block/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux_double_block/FLUX double-stream (MMDiT) transformer blockDescriptionImage and text streams each get adaLN-Zero modulation and a feed-forward; attention is joint across both streams. Reference: diffusers FluxTransformerBlock. Usageflux_double_block(dim, num_attention_heads, attention_head_dim) Arguments dim: Integer. Model dimension. num_attention_heads: Integer. Attention heads. attention_head_dim: Integer. Per-head dimension. ValueModule whose forward(hidden_states, encoder_hidden_states, temb, image_rotary_emb) returns list(encoder_hidden_states, hidden_states).https://cornball.ai/docs/diffuser/flux_is_quant_key/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux_is_quant_key/Test whether a FLUX key is in the quantization cast setDescriptionTest whether a FLUX key is in the quantization cast set Usageflux_is_quant_key(key) Arguments key: Character vector of parameter names (diffusers-style). ValueLogical vector.https://cornball.ai/docs/diffuser/flux_load_pipeline/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux_load_pipeline/Load the FLUX.1-schnell pipelineDescriptionLoads the quantized transformer artifact plus the VAE decoder, CLIP and T5 text encoders, tokenizer, and scheduler config (from the HuggingFace cache populated by download_flux1). Components load to the CPU when phase_offload is on and move to the GPU only for their phase of the generation. Usageflux_load_pipeline(model_dir = NULL, device = "cuda", precision = NULL, text_device = "cpu", attn_chunk = NULL, phase_offload = TRUE, verbose = TRUE) Arguments model_dir: Quantized artifact directory (default: the download_flux1 location for precision), or a raw diffusers transformer directory for full-precision loading.https://cornball.ai/docs/diffuser/flux_load_transformer/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux_load_transformer/Load a FLUX transformer from any checkpoint formatDescriptionBuilds flux_transformer from the checkpoint’s embedded config and loads the weights. Dispatches on the checkpoint format: Usageflux_load_transformer(ckpt, device = "cuda", dtype = "bfloat16", pin = TRUE, fp8_resident = FALSE, verbose = TRUE, ...) Arguments ckpt: A checkpoint from flux_open_checkpoint or flux_open_quantized. device: Character. Compute device. dtype: Character. Model dtype (“bfloat16” or “float32”). For quantized formats this sets the resident (non-quantized) tensors and must match the compute dtype: bfloat16 for GPU compute, float32 for CPU compute.https://cornball.ai/docs/diffuser/flux_memory_profile/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux_memory_profile/Resolve a FLUX memory profileDescriptionResolve a FLUX memory profile Usageflux_memory_profile(vram_gb = NULL) Arguments vram_gb: Numeric or NULL. Available VRAM; auto-detected when NULL (via gpu.ctl or nvidia-smi). ValueList with name, precision (“nf4”/“fp8”), attn_chunk, text_device, phase_offload, and max_pixels (largest validated image area).https://cornball.ai/docs/diffuser/flux_open_checkpoint/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux_open_checkpoint/Open a FLUX transformer checkpoint directoryDescriptionOpens a diffusers-layout transformer directory lazily (headers only). Sharded checkpoints are resolved through the index.json weight map; single-file checkpoints are opened directly. The transformer config.json is attached as $config. Usageflux_open_checkpoint(transformer_dir) Arguments transformer_dir: Directory containing config.json and the diffusion_pytorch_model*.safetensors file(s). ValueAn object of class ltx23_checkpoint (shared checkpoint interface): list with handle$get_tensor, keys, config, and path.https://cornball.ai/docs/diffuser/flux_open_quantized/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux_open_quantized/Open a quantized FLUX artifact directoryDescriptionOpens the sharded NF4/fp8 artifact written by flux_quantize through the shared checkpoint interface. The manifest’s embedded transformer config and format ride along, so flux_load_transformer needs nothing else. Usageflux_open_quantized(dir) Arguments dir: The quantized artifact directory (with manifest.json). ValueAn ltx23_checkpoint with $format set.https://cornball.ai/docs/diffuser/flux_pack_latents/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux_pack_latents/Pack FLUX latents into a patch sequenceDescriptionPacks a [B, C, H, W] latent into 2x2 patches, giving a sequence [B, (H/2) * (W/2), C * 4]. Reference: FluxPipeline._pack_latents. Usageflux_pack_latents(latents) Arguments latents: Tensor of shape [B, C, H, W]; H and W must be even. ValueTensor of shape [B, (H/2) * (W/2), C * 4].https://cornball.ai/docs/diffuser/flux_pos_embed/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux_pos_embed/Compute FLUX rotary frequencies from position idsDescriptionPer-axis 1D rotary frequencies (interleaved-real convention), computed in float64 on CPU and concatenated over the axes. Reference: FluxPosEmbed with get_1d_rotary_pos_embed(repeat_interleave_real=TRUE, use_real=TRUE, freqs_dtype=float64). Usageflux_pos_embed(ids, axes_dim = c(16L, 56L, 56L), theta = 10000) Arguments ids: Tensor of shape [S, 3]: concatenated text ids (all zero) and image ids from flux_prepare_latent_image_ids. axes_dim: Integer vector of per-axis rotary dims; must sum to the attention head dim. FLUX uses c(16, 56, 56).https://cornball.ai/docs/diffuser/flux_prepare_latent_image_ids/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux_prepare_latent_image_ids/Build FLUX latent image position idsDescriptionPosition ids over the packed latent grid (latent height/2 x width/2). Channel 1 is always zero, channel 2 holds the row index, channel 3 the column index. Reference: FluxPipeline._prepare_latent_image_ids. Usageflux_prepare_latent_image_ids(height, width, device = "cpu") Arguments height: Integer. Packed grid height (latent height / 2). width: Integer. Packed grid width (latent width / 2). device: Device for the resulting tensor. ValueFloat tensor of shape [height * width, 3].https://cornball.ai/docs/diffuser/flux_quantize/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux_quantize/Quantize a FLUX transformer to NF4 or fp8 shardsDescriptionStreams the bf16 diffusers checkpoint tensor by tensor. Cast-set weights (see flux_is_quant_key) are stored as NF4 (packed uint8 + <key>_absmax float32 blocks) or as float8_e4m3fn with an absmax/448 per-tensor <key>_scale; everything else is copied through unchanged. The manifest embeds the transformer config, so the source checkpoint is not needed again after quantization. Usageflux_quantize(transformer_dir, output_dir = NULL, format = c("nf4", "fp8"), shard_bytes = 4e+09, force = FALSE, verbose = TRUE) Arguments transformer_dir: Source diffusers transformer directory.https://cornball.ai/docs/diffuser/flux_single_block/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux_single_block/FLUX single-stream transformer blockDescriptionParallel attention + MLP over the joint [text; image] sequence with a shared gate: x + gate * proj_out(cat(attn, gelu(mlp))). The reference concatenates the streams inside every block and splits after; here the caller concatenates once before the single-block stack, which is numerically identical. Reference: diffusers FluxSingleTransformerBlock. Usageflux_single_block(dim, num_attention_heads, attention_head_dim, mlp_ratio = 4) Arguments dim: Integer. Model dimension. num_attention_heads: Integer. Attention heads. attention_head_dim: Integer. Per-head dimension. mlp_ratio: Numeric.https://cornball.ai/docs/diffuser/flux_transformer/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux_transformer/FLUX transformer modelDescription19 double-stream (MMDiT) blocks followed by 38 single-stream blocks over the joint [text; image] sequence, with adaLN-Zero conditioning on timestep + pooled CLIP text. Rotary embeddings are precomputed by the caller with flux_pos_embed (they are static across denoise steps). Defaults are the FLUX.1-schnell configuration. Usageflux_transformer(in_channels = 64L, num_layers = 19L, num_single_layers = 38L, attention_head_dim = 128L, num_attention_heads = 24L, joint_attention_dim = 4096L, pooled_projection_dim = 768L, axes_dims_rope = c(16L, 56L, 56L), out_channels = NULL) Arguments in_channels: Integer.https://cornball.ai/docs/diffuser/flux_unpack_latents/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux_unpack_latents/Unpack a FLUX patch sequence back into latentsDescriptionInverse of flux_pack_latents. Height and width are the target image dimensions in pixels; the latent grid is derived via the VAE scale factor and the 2x2 patch size. Reference: FluxPipeline._unpack_latents. Usageflux_unpack_latents(latents, height, width, vae_scale_factor = 8L) Arguments latents: Tensor of shape [B, S, C_packed]. vae_scale_factor: Integer. Spatial downsampling of the VAE (8). height,width: Integers. Image height/width in pixels. ValueTensor of shape [B, C_packed / 4, height / 8, width / 8].https://cornball.ai/docs/diffuser/flux2_bn_normalize/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux2_bn_normalize/Normalize patchified latents with the VAE BatchNorm statisticsDescriptionFLUX.2 has no scalar scaling/shift factor; latents are standardized per packed channel with the VAE’s bn.running_mean / bn.running_var (eps 1e-4). Reference: encode/decode paths of Flux2KleinPipeline. Usageflux2_bn_normalize(latents, bn_mean, bn_var, eps = 1e-04, inverse = FALSE) Arguments latents: Tensor [B, 128, H, W] (patchified). eps: Numeric. BatchNorm epsilon. inverse: Logical. De-normalize (decode path) instead. bn_mean,bn_var: Float tensors [128]. ValueTensor like latents.https://cornball.ai/docs/diffuser/flux2_double_block/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux2_double_block/FLUX.2 double-stream (MMDiT) blockDescriptionImage and text streams with externally supplied (shift, scale, gate) modulation triples, joint attention (txt first), and SwiGLU feed-forwards. Reference: Flux2TransformerBlock. Usageflux2_double_block(dim, num_attention_heads, attention_head_dim, mlp_ratio = 3, eps = 1e-06, bias = FALSE) Arguments dim: Integer. Model dimension. num_attention_heads: Integer. Attention heads. attention_head_dim: Integer. Per-head dimension. mlp_ratio: Numeric. FF multiplier (FLUX.2: 3.0). eps: Numeric. Norm epsilon. bias: Logical. ValueModule whose forward(hidden_states, encoder_hidden_states, temb_mod_img, temb_mod_txt, image_rotary_emb) returns list(encoder_hidden_states, hidden_states).https://cornball.ai/docs/diffuser/flux2_empirical_mu/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux2_empirical_mu/Empirical timestep shift for FLUX.2DescriptionBFL’s piecewise-linear fit of the dynamic-shifting mu as a function of image sequence length and step count; replaces FLUX.1’s calculate_shift. Reference: compute_empirical_mu (adapted from BFL sampling.py). Usageflux2_empirical_mu(image_seq_len, num_steps) Arguments image_seq_len: Integer. Packed image token count. num_steps: Integer. Inference steps. ValueNumeric mu for flowmatch_set_timesteps.https://cornball.ai/docs/diffuser/flux2_feed_forward/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux2_feed_forward/FLUX.2 feed-forward (fused SwiGLU)Descriptionlinear_in projects to twice the inner dim; SwiGLU gates the first half with SiLU and multiplies by the second half; linear_out projects back. Reference: Flux2FeedForward + Flux2SwiGLU. Usageflux2_feed_forward(dim, dim_out = NULL, mult = 3, bias = FALSE) Arguments dim: Integer. Input dimension. dim_out: Integer. Output dimension (defaults to dim). mult: Numeric. Inner dim multiplier (FLUX.2: 3.0). bias: Logical.https://cornball.ai/docs/diffuser/flux2_is_quant_key/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux2_is_quant_key/Test whether a FLUX.2 key is in the quantization cast setDescriptionTest whether a FLUX.2 key is in the quantization cast set Usageflux2_is_quant_key(key) Arguments key: Character vector of parameter names (diffusers-style). ValueLogical vector.https://cornball.ai/docs/diffuser/flux2_load_pipeline/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux2_load_pipeline/Load the FLUX.2 klein pipelineDescriptionLoads the quantized transformer artifact plus the FLUX.2 VAE decoder, Qwen3 text encoder, and tokenizer from the HuggingFace cache populated by download_flux2_klein. With fp8 precision the ~4 GB transformer stays GPU-resident. Usageflux2_load_pipeline(model_dir = NULL, device = "cuda", precision = c("fp8", "nf4"), text_device = NULL, attn_chunk = NULL, phase_offload = TRUE, verbose = TRUE) Arguments model_dir: Quantized artifact directory (default: the download_flux2_klein location for precision), or a raw diffusers transformer directory.https://cornball.ai/docs/diffuser/flux2_modulation/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux2_modulation/FLUX.2 shared modulation projectionDescriptionlinear(silu(temb)) producing mod_param_sets triples of (shift, scale, gate). Computed once per forward at model level and broadcast to every block. Reference: Flux2Modulation. Usageflux2_modulation(dim, mod_param_sets = 2L, bias = FALSE) Arguments dim: Integer. Model dimension. mod_param_sets: Integer. Number of (shift, scale, gate) triples. bias: Logical.https://cornball.ai/docs/diffuser/flux2_pack_latents/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux2_pack_latents/Pack patchified FLUX.2 latents into tokensDescription[B, C, H, W] -> [B, H * W, C] (row-major spatial flatten, channels-last). Reference: Flux2KleinPipeline._pack_latents. Usageflux2_pack_latents(latents) Arguments latents: Tensor [B, C, H, W]. ValueTensor [B, H * W, C].https://cornball.ai/docs/diffuser/flux2_parallel_self_attention/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux2_parallel_self_attention/FLUX.2 parallel self-attention (single-stream)DescriptionViT-22B-style parallel block internals: one fused projection produces QKV and the SwiGLU MLP input; one fused projection consumes cat(attention output, MLP output). Reference: Flux2ParallelSelfAttention + Flux2ParallelSelfAttnProcessor. Usageflux2_parallel_self_attention(query_dim, heads, dim_head, mlp_ratio = 3, eps = 1e-06, bias = FALSE) Arguments query_dim: Integer. Model dimension. heads: Integer. Attention heads. dim_head: Integer. Per-head dimension. mlp_ratio: Numeric. MLP hidden multiplier (FLUX.2: 3.0). eps: Numeric. RMS norm epsilon. bias: Logical.https://cornball.ai/docs/diffuser/flux2_patchify_latents/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux2_patchify_latents/Patchify FLUX.2 latents (2x2 -> channels)Description[B, C, H, W] -> [B, 4C, H/2, W/2], channel order (C, ph, pw). Reference: Flux2KleinPipeline._patchify_latents. Usageflux2_patchify_latents(latents) Arguments latents: Tensor [B, C, H, W]; H and W must be even. ValueTensor [B, C * 4, H / 2, W / 2].https://cornball.ai/docs/diffuser/flux2_prepare_latent_ids/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux2_prepare_latent_ids/Build FLUX.2 latent position idsDescriptionColumns (T, H, W, L) with H and W carrying the packed-grid position (row-major: H varies slowest), T = L = 0. Reference: Flux2KleinPipeline._prepare_latent_ids. Usageflux2_prepare_latent_ids(height, width, device = "cpu") Arguments height: Integer. Packed grid height (pixel height / 16). width: Integer. Packed grid width (pixel width / 16). device: Device for the resulting tensor. ValueFloat tensor [height * width, 4].https://cornball.ai/docs/diffuser/flux2_prepare_text_ids/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux2_prepare_text_ids/Build FLUX.2 text position idsDescriptionColumns (T, H, W, L) with only L varying: 0..len-1. Reference: Flux2KleinPipeline._prepare_text_ids. Usageflux2_prepare_text_ids(len, device = "cpu") Arguments len: Integer. Text sequence length. device: Device for the resulting tensor. ValueFloat tensor [len, 4].https://cornball.ai/docs/diffuser/flux2_single_block/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux2_single_block/FLUX.2 single-stream block (parallel)DescriptionParameterless LayerNorm with external modulation, then the fused parallel attention+MLP. Operates on the pre-concatenated [text; image] sequence (the reference model concatenates once before the stack). Reference: Flux2SingleTransformerBlock. Usageflux2_single_block(dim, num_attention_heads, attention_head_dim, mlp_ratio = 3, eps = 1e-06, bias = FALSE) Arguments dim: Integer. Model dimension. num_attention_heads: Integer. Attention heads. attention_head_dim: Integer. Per-head dimension. mlp_ratio: Numeric. MLP multiplier (FLUX.2: 3.0). eps: Numeric. Norm epsilon. bias: Logical. ValueModule whose forward(hidden_states, temb_mod, image_rotary_emb) returns the joint hidden states.https://cornball.ai/docs/diffuser/flux2_transformer/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux2_transformer/FLUX.2 transformer modelDescriptionShared modulation computed once per forward; double blocks over separate text/image streams, then single (parallel) blocks over the concatenated [text; image] sequence. Rotary embeddings are precomputed by the caller with flux_pos_embed (axes_dim = c(32, 32, 32, 32), theta = 2000) over the concatenated [text; image] 4-axis position ids. Usageflux2_transformer(in_channels = 128L, num_layers = 5L, num_single_layers = 20L, attention_head_dim = 128L, num_attention_heads = 24L, joint_attention_dim = 7680L, mlp_ratio = 3, timestep_guidance_channels = 256L, axes_dims_rope = c(32L, 32L, 32L, 32L), rope_theta = 2000, eps = 1e-06, out_channels = NULL) Arguments in_channels: Integer.https://cornball.ai/docs/diffuser/flux2_unpack_latents_with_ids/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux2_unpack_latents_with_ids/Unpack FLUX.2 tokens back to a latent grid via position idsDescriptionScatters tokens to (H, W) positions taken from the id columns (H = column 2, W = column 3, 0-based values). Reference: Flux2KleinPipeline._unpack_latents_with_ids. Usageflux2_unpack_latents_with_ids(x, ids, height, width) Arguments x: Tensor [B, S, C] of tokens. ids: Tensor [S, 4] (or [B, S, 4]) of position ids. height,width: Integers. Packed grid dimensions. ValueTensor [B, C, height, width].https://cornball.ai/docs/diffuser/flux2_unpatchify_latents/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux2_unpatchify_latents/Unpatchify FLUX.2 latents (channels -> 2x2)DescriptionInverse of flux2_patchify_latents. Reference: Flux2KleinPipeline._unpatchify_latents. Usageflux2_unpatchify_latents(latents) Arguments latents: Tensor [B, 4C, H, W]. ValueTensor [B, C, H * 2, W * 2].https://cornball.ai/docs/diffuser/flux2_vae_decoder/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/flux2_vae_decoder/FLUX.2 VAE decoderDescriptionThe AutoencoderKLFlux2 decode path: post_quant_conv (1x1, 32 channels) followed by the standard AutoencoderKL decoder body (reused from vae_decoder_native), plus the BatchNorm running statistics used for latent (de)normalization. Reference: src/diffusers/models/autoencoders/autoencoder_kl_flux2.py. Usageflux2_vae_decoder(latent_channels = 32L, block_channels = c(512L, 512L, 256L, 128L), norm_groups = 32L) Arguments latent_channels: Integer (32 for FLUX.2). block_channels: Decoder block channels (reversed encoder block_out_channels). norm_groups: Integer. Group norm groups. ValueModule whose forward(z) decodes [B, 32, H, W] latents to [B, 3, 8H, 8W] images; $bn$running_mean / $bn$running_var carry the normalization statistics.https://cornball.ai/docs/diffuser/gemma3_config_ltx2/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/gemma3_config_ltx2/Create Gemma3 configuration for LTX-2DescriptionReturns the default configuration used by LTX-2’s text encoder. Usagegemma3_config_ltx2() ValueList with model configuration parameters.https://cornball.ai/docs/diffuser/gemma3_text_model/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/gemma3_text_model/Gemma3 Text ModelDescriptionFull Gemma3 text encoder model. Usagegemma3_text_model(config) Arguments config: Model configuration list.https://cornball.ai/docs/diffuser/gemma3_tokenizer/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/gemma3_tokenizer/Gemma3 TokenizerDescriptionNative R tokenizer for Gemma3 using BPE. Loads from HuggingFace tokenizer.json format. Usagegemma3_tokenizer(tokenizer_path) Arguments tokenizer_path: Character. Path to tokenizer directory or tokenizer.json file. ValueA gemma3_tokenizer object (extends bpe_tokenizer).https://cornball.ai/docs/diffuser/img2img/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/img2img/Image-to-Image Generation with Stable DiffusionDescriptionThis function generates an image based on an input image and a text prompt using the Stable Diffusion model. It allows for various configurations such as model name, device, scheduler, and more. Usageimg2img(input_image, prompt, negative_prompt = NULL, img_dim = 512, model_name = c("sd21", "sdxl"), pipeline = NULL, devices = "auto", unet_dtype_str = "float16", download_models = FALSE, scheduler = "ddim", num_inference_steps = 50, strength = 0.8, guidance_scale = 7.https://cornball.ai/docs/diffuser/is_blackwell_gpu/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/is_blackwell_gpu/Check if GPU is Blackwell ArchitectureDescriptionBlackwell GPUs (RTX 50xx) may need special handling. Usageis_blackwell_gpu() ValueLogical. TRUE if Blackwell GPU detected. Examplesif (is_blackwell_gpu()) { message("Using Blackwell-compatible settings") }https://cornball.ai/docs/diffuser/latents_to_video/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/latents_to_video/Create Video from Latents (Helper)DescriptionConvenience function to decode latents and save video in one step. Usagelatents_to_video(latents, vae, file, fps = 24, ...) Arguments latents: Tensor of latents from generation. vae: VAE decoder module. file: Output file path. fps: Frames per second. ...: Additional arguments to save_video. ValueInvisibly returns the output file path.https://cornball.ai/docs/diffuser/load_decoder_safetensors/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/load_decoder_safetensors/Load HF safetensors VAE weights into the native decoderDescriptionLoads the decoder half of a diffusers AutoencoderKL safetensors file (e.g. FLUX.1-schnell’s vae/diffusion_pytorch_model.safetensors). Keys under decoder. map to the native module 1:1; encoder and quant-conv keys are skipped (the FLUX VAE has no quant convs, and txt2img needs no encoder). Usageload_decoder_safetensors(native_decoder, path, verbose = TRUE) Arguments native_decoder: Native VAE decoder module path: Path to the VAE .safetensors file (or a directory containing diffusion_pytorch_model.https://cornball.ai/docs/diffuser/load_decoder_weights/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/load_decoder_weights/Load weights from TorchScript decoder into native decoderDescriptionLoad weights from TorchScript decoder into native decoder Usageload_decoder_weights(native_decoder, torchscript_path, verbose = TRUE) Arguments native_decoder: Native VAE decoder module torchscript_path: Path to TorchScript decoder .pt file verbose: Print loading progress ValueThe native decoder with loaded weights (invisibly)https://cornball.ai/docs/diffuser/load_flux2_vae_decoder/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/load_flux2_vae_decoder/Load the FLUX.2 VAE decoder from safetensorsDescriptionLoads the decoder half plus post_quant_conv and the BatchNorm running statistics; encoder and quant_conv keys are skipped (txt2img needs no encoder). Usageload_flux2_vae_decoder(path, latent_channels = 32L, block_channels = c(512L, 512L, 256L, 128L), norm_groups = 32L, verbose = TRUE) Arguments path: Path to the VAE .safetensors file (or a directory containing diffusion_pytorch_model.safetensors). verbose: Logical. latent_channels,block_channels,norm_groups: Constructor arguments for flux2_vae_decoder. ValueThe loaded flux2_vae_decoder in eval mode.https://cornball.ai/docs/diffuser/load_gemma3_text_encoder/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/load_gemma3_text_encoder/Load Gemma3 Text Model from safetensorsDescriptionLoads pre-trained Gemma3 weights from HuggingFace safetensors files. Usageload_gemma3_text_encoder(model_path, device = "cpu", dtype = "float16", verbose = TRUE) Arguments model_path: Character. Path to directory containing model files. device: Character. Device to load model to. dtype: Character. Data type (“float32”, “float16”, “bfloat16”). verbose: Logical. Print loading progress. ValueInitialized gemma3_text_model with loaded weights.https://cornball.ai/docs/diffuser/load_model_component/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/load_model_component/Load a specific component of a diffusion modelDescriptionLoads a TorchScript model component (UNet, decoder, or text encoder) from the hfhub cache or legacy model directory, downloading it first if necessary. Usageload_model_component(component, model_name = "sd21", device = "cpu", unet_dtype_str = NULL, download = TRUE, use_native = FALSE) Arguments component: Character string, the component to load: “unet”, “decoder”, or “text_encoder”. model_name: Character string, the name of the model to use. device: Character string, the torch device to load the model onto (“cpu” or “cuda”).https://cornball.ai/docs/diffuser/load_pipeline/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/load_pipeline/Load a diffusion model pipelineDescriptionThis function loads a diffusion model pipeline consisting of a UNet, VAE decoder, and text encoder. It initializes the models and sets up the environment for inference. Usageload_pipeline(model_name, m2d, i2i = FALSE, unet_dtype_str, use_native_decoder = FALSE, use_native_text_encoder = FALSE, use_native_unet = FALSE, ...) Arguments model_name: The name of the model to load. m2d: A list containing model-to-device mappings and configurations. i2i: Logical indicating whether to load the encoder for img2img().https://cornball.ai/docs/diffuser/load_qwen3_text_encoder/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/load_qwen3_text_encoder/Load a Qwen3 encoder from a transformers directoryDescriptionStreams the (possibly sharded) safetensors weights into qwen3_encoder. The LM head is tied to the embeddings and skipped. Usageload_qwen3_text_encoder(model_path, device = "cpu", dtype = "float32", verbose = TRUE, ...) Arguments model_path: Directory with config.json and model*.safetensors (FLUX.2-klein’s text_encoder). device: Character. Target device. dtype: Character. “bfloat16” (GPU) or “float32” (CPU). verbose: Logical. ...: Overrides for qwen3_encoder arguments. ValueThe loaded qwen3_encoder in eval mode.https://cornball.ai/docs/diffuser/load_t5_text_encoder/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/load_t5_text_encoder/Load a T5 encoder from a transformers directoryDescriptionStreams the (possibly sharded) safetensors weights into t5_encoder, stripping the encoder. key prefix and aliasing embed_tokens to the shared embedding. Usageload_t5_text_encoder(model_path, device = "cpu", dtype = "float32", verbose = TRUE, ...) Arguments model_path: Directory with config.json and model*.safetensors (FLUX.1-schnell’s text_encoder_2). device: Character. Target device. dtype: Character. “float32” (CPU default; T5 overflows in float16) or “bfloat16”. verbose: Logical. ...: Overrides for t5_encoder arguments. ValueThe loaded t5_encoder in eval mode.https://cornball.ai/docs/diffuser/load_text_encoder_safetensors/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/load_text_encoder_safetensors/Load HF safetensors weights into the native CLIP text encoderDescriptionLoads a HuggingFace CLIPTextModel model.safetensors (e.g. FLUX.1-schnell’s text_encoder) into text_encoder_native, reusing the TorchScript key remaps minus the export prefixes. Usageload_text_encoder_safetensors(native_encoder, path, verbose = TRUE) Arguments native_encoder: Native text encoder module path: Path to model.safetensors (or a directory containing it) verbose: Print loading progress ValueThe native encoder with loaded weights (invisibly)https://cornball.ai/docs/diffuser/load_text_encoder_weights/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/load_text_encoder_weights/Load weights from TorchScript text encoder into native encoderDescriptionLoad weights from TorchScript text encoder into native encoder Usageload_text_encoder_weights(native_encoder, torchscript_path, verbose = TRUE) Arguments native_encoder: Native text encoder module torchscript_path: Path to TorchScript encoder .pt file verbose: Print loading progress ValueThe native encoder with loaded weights (invisibly)https://cornball.ai/docs/diffuser/load_text_encoder2_weights/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/load_text_encoder2_weights/Load weights from TorchScript text encoder 2 into native encoderDescriptionLoad weights from TorchScript text encoder 2 into native encoder Usageload_text_encoder2_weights(native_encoder, torchscript_path, verbose = TRUE) Arguments native_encoder: Native text encoder 2 module torchscript_path: Path to TorchScript encoder .pt file verbose: Print loading progress ValueThe native encoder with loaded weights (invisibly)https://cornball.ai/docs/diffuser/load_to_gpu/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/load_to_gpu/Load Module to GPUDescriptionMoves a torch module and all its parameters to CUDA. Usageload_to_gpu(module, device = "cuda") Arguments module: A torch nn_module. device: Character. Target device (default “cuda”). ValueThe module (modified in place). Examplesload_to_gpu(model) output <- model(x) offload_to_cpu(model)https://cornball.ai/docs/diffuser/load_unet_sdxl_weights/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/load_unet_sdxl_weights/Load weights from TorchScript SDXL UNet into native SDXL UNetDescriptionLoad weights from TorchScript SDXL UNet into native SDXL UNet Usageload_unet_sdxl_weights(native_unet, torchscript_path, verbose = TRUE) Arguments native_unet: Native SDXL UNet module torchscript_path: Path to TorchScript SDXL UNet .pt file verbose: Print loading progress ValueThe native UNet with loaded weights (invisibly)https://cornball.ai/docs/diffuser/load_unet_weights/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/load_unet_weights/Load weights from TorchScript UNet into native UNetDescriptionLoad weights from TorchScript UNet into native UNet Usageload_unet_weights(native_unet, torchscript_path, verbose = TRUE) Arguments native_unet: Native UNet module torchscript_path: Path to TorchScript UNet .pt file verbose: Print loading progress ValueThe native UNet with loaded weights (invisibly)https://cornball.ai/docs/diffuser/ltx23_ada_layer_norm_single/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_ada_layer_norm_single/Adaptive layer norm single (adaLN-single)DescriptionEmbeds a timestep/sigma and projects it to a configurable number of modulation parameter vectors. Usageltx23_ada_layer_norm_single(embedding_dim, num_mod_params = 6L) Arguments embedding_dim: Integer. Model dimension. num_mod_params: Integer. Number of modulation parameter vectors. ValueModule whose forward returns list(mod_params [N, num_mod_params * dim], embedded_timestep [N, dim]).https://cornball.ai/docs/diffuser/ltx23_adain_filter_latent/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_adain_filter_latent/Adaptive instance normalization between latent tensorsDescriptionMatches each (batch, channel) slice’s mean/std to the reference latents, blended by factor (cf. diffusers LTX2LatentUpsamplePipeline.adain_filter_latent). Usageltx23_adain_filter_latent(latents, reference_latents, factor = 1) Arguments latents: Tensor [B, C, F, H, W]. reference_latents: Tensor with the target statistics. factor: Numeric blend in [-10, 10]; 0 is identity. ValueFiltered latents.https://cornball.ai/docs/diffuser/ltx23_antialias_act1d/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_antialias_act1d/Anti-aliased activationDescriptionUpsample 2x, apply the activation, downsample 2x. Usageltx23_antialias_act1d(channels, ratio = 2L, kernel_size = 12L) Arguments channels: Integer. Channels for the SnakeBeta activation. ratio,kernel_size: Integers. Resampling config.https://cornball.ai/docs/diffuser/ltx23_apply_interleaved_rotary_emb/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_apply_interleaved_rotary_emb/Apply interleaved rotary embeddingsDescriptionRotates adjacent element pairs of the last dimension: out = x * cos + rotate_half(x) * sin with pairs interleaved (elements 1,2 form the first complex pair). Usageltx23_apply_interleaved_rotary_emb(x, freqs) Arguments x: Tensor of shape [B, S, C]. freqs: List of two tensors (cos, sin), each [B, S, C]. ValueTensor with the same shape and dtype as x.https://cornball.ai/docs/diffuser/ltx23_apply_split_rotary_emb/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_apply_split_rotary_emb/Apply split rotary embeddingsDescriptionRotates element pairs formed by splitting the last dimension in half: element i pairs with element i + d/2. The cos/sin tensors carry half the head dimension. Usageltx23_apply_split_rotary_emb(x, freqs) Arguments x: Tensor of shape [B, H, T, D] (per-head layout), or [B, T, H*D] which is reshaped per-head when freqs is 4D. freqs: List of two tensors (cos, sin), each [B, H, T, D/2]. ValueTensor with the same shape and dtype as x.https://cornball.ai/docs/diffuser/ltx23_attention/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_attention/LTX-2 attention layerDescriptionAttention with RMS q/k norms across heads, optional per-head output gating (LTX-2.3), separate query/key RoPE (for a2v/v2a cross attention), and optional STG perturbation (skip attention, use the value projection). Usageltx23_attention(query_dim, heads = 8L, kv_heads = NULL, dim_head = 64L, bias = TRUE, cross_attention_dim = NULL, out_bias = TRUE, norm_eps = 1e-06, norm_elementwise_affine = TRUE, rope_type = "split", apply_gated_attention = FALSE) Arguments query_dim: Integer. Query feature dimension. dim_head: Integer. Per-head dimension.https://cornball.ai/docs/diffuser/ltx23_audio_causal_conv2d/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_audio_causal_conv2d/Causal 2D convolution for audio spectrogramsDescriptionPads asymmetrically along the causal axis (“height” = time frames for LTX audio) before an unpadded Conv2d. Usageltx23_audio_causal_conv2d(in_channels, out_channels, kernel_size = 3L, stride = 1L, causality_axis = "height") Arguments kernel_size: Integer or length-2 vector. stride: Integer. causality_axis: “height”, “width”, “width-compatibility”, or “none”. in_channels,out_channels: Integers.https://cornball.ai/docs/diffuser/ltx23_audio_decoder/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_audio_decoder/LTX-2.3 audio VAE decoderDescriptionLatents [B, 8, L, 16] -> mel spectrogram [B, 2, 4L - 3, 64]. Usageltx23_audio_decoder(base_channels = 128L, output_channels = 2L, num_res_blocks = 2L, latent_channels = 8L, ch_mult = c(1L, 2L, 4L), causality_axis = "height", mel_bins = 64L) Arguments base_channels: Integer. output_channels: Integer. Audio channels (2 = stereo). num_res_blocks: Integer. Per-level ResNet count (a stage runs num_res_blocks + 1 blocks). latent_channels: Integer. ch_mult: Integer vector. Channel multipliers per level.https://cornball.ai/docs/diffuser/ltx23_audio_downsample/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_audio_downsample/LTX audio downsamplerDescriptionCausal zero-pad followed by a plain stride-2 conv (reference LTX2AudioDownsample; note the conv is unwrapped, so its checkpoint key is downsample.conv.*). Usageltx23_audio_downsample(in_channels, causality_axis = "height") Arguments in_channels: Integer. causality_axis: Character.https://cornball.ai/docs/diffuser/ltx23_audio_encoder/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_audio_encoder/LTX-2.3 audio VAE encoderDescriptionMel spectrogram [B, 2, T, 64] -> latent distribution moments [B, 2 * latent_channels, ceil(T/4), 16]. Structure mirrors the decoder: causal convs, parameterless pixel norms, ResNet stages with stride-2 downsampling between levels (reference LTX2AudioEncoder). Usageltx23_audio_encoder(base_channels = 128L, in_channels = 2L, num_res_blocks = 2L, latent_channels = 8L, ch_mult = c(1L, 2L, 4L), causality_axis = "height") Arguments in_channels: Integer. Mel channels (2 = stereo). base_channels,num_res_blocks,latent_channels,ch_mult,causality_axis: See ltx23_audio_decoder.https://cornball.ai/docs/diffuser/ltx23_audio_mel_frontend/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_audio_mel_frontend/Build the 16 kHz log-mel frontend for audio conditioningDescriptionAn ltx23_mel_stft whose STFT and mel bases are constructed (not checkpoint-loaded) with the audio VAE’s preprocessing spec. Usageltx23_audio_mel_frontend(filter_length = 1024L, hop_length = 160L, n_mels = 64L, sample_rate = 16000L, fmin = 0, fmax = 8000) Arguments filter_length,hop_length,n_mels,sample_rate,fmin,fmax: The checkpoint preprocessing parameters (defaults are LTX-2.3’s). ValueAn ltx23_mel_stft module.https://cornball.ai/docs/diffuser/ltx23_audio_resnet_block/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_audio_resnet_block/LTX audio ResNet blockDescriptionPixelNorm -> SiLU -> causal conv, twice, with a 1x1 causal conv shortcut (nin_shortcut) on channel change. Usageltx23_audio_resnet_block(in_channels, out_channels = NULL, causality_axis = "height") Arguments causality_axis: Character. in_channels,out_channels: Integers.https://cornball.ai/docs/diffuser/ltx23_audio_upsample/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_audio_upsample/LTX audio upsamplerDescriptionNearest 2x interpolation, causal conv, then a crop of the first element along the causal axis. Usageltx23_audio_upsample(in_channels, causality_axis = "height") Arguments in_channels: Integer. causality_axis: Character.https://cornball.ai/docs/diffuser/ltx23_audio_vae/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_audio_vae/LTX-2.3 audio VAEDescriptionEncoder + decoder plus the per-channel latent statistics loaded from the checkpoint. Encoding is used for audio-conditioned generation (lip sync); decoding for generated audio. Usageltx23_audio_vae(base_channels = 128L, output_channels = 2L, num_res_blocks = 2L, latent_channels = 8L, ch_mult = c(1L, 2L, 4L), causality_axis = "height", mel_bins = 64L, in_channels = 2L) Arguments in_channels: Integer. Mel input channels (2 = stereo). base_channels,output_channels,num_res_blocks,latent_channels,ch_mult,causality_axis,mel_bins: See ltx23_audio_decoder.https://cornball.ai/docs/diffuser/ltx23_causal_conv3d/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_causal_conv3d/Causal 3D convolutionDescriptionSpatial padding is handled by the convolution; temporal padding replicates the first frame (causal) or both edge frames (non-causal), chosen at call time. Usageltx23_causal_conv3d(in_channels, out_channels, kernel_size = 3L, stride = 1L, spatial_padding_mode = "zeros") Arguments kernel_size: Integer or length-3 vector (t, h, w). stride: Integer or length-3 vector. spatial_padding_mode: Character. Conv padding mode. in_channels,out_channels: Integers.https://cornball.ai/docs/diffuser/ltx23_census/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_census/Summarize checkpoint key coverageDescriptionSummarize checkpoint key coverage Usageltx23_census(ckpt) Arguments ckpt: An ltx23_checkpoint. ValueA data.frame with one row per component group and its key count.https://cornball.ai/docs/diffuser/ltx23_connector_transformer_1d/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_connector_transformer_1d/1D connector transformerDescriptionReplaces padded positions with learnable registers (valid tokens are front-aligned in their original order; the tail is filled with registers indexed by absolute position, after which the attention mask is cleared), then runs 1D transformer blocks with rotary embeddings. Usageltx23_connector_transformer_1d(num_attention_heads = 32L, attention_head_dim = 128L, num_layers = 8L, num_learnable_registers = 128L, rope_base_seq_len = 4096L, rope_theta = 10000, rope_double_precision = TRUE, eps = 1e-06, rope_type = "split", gated_attention = TRUE) Arguments num_learnable_registers: Integer or NULL.https://cornball.ai/docs/diffuser/ltx23_denormalize_latents/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_denormalize_latents/Denormalize latents with the VAE’s per-channel statisticsDescriptionDenormalize latents with the VAE’s per-channel statistics Usageltx23_denormalize_latents(latents, latents_mean, latents_std) Arguments latents: Tensor [B, C, F, H, W]. latents_mean,latents_std: Tensors [C]. ValueDenormalized latents ready for the decoder.https://cornball.ai/docs/diffuser/ltx23_distilled_sigmas/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_distilled_sigmas/Official distilled sigma scheduleDescriptionThe distilled LTX sigma values (with terminal zero appended), as published in the Apache-2.0 diffusers reference (pipelines/ltx2/utils.py). Usageltx23_distilled_sigmas() ValueNumeric vector of length 9.https://cornball.ai/docs/diffuser/ltx23_downsample1d/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_downsample1d/Anti-aliasing 1D downsampler (low-pass then stride)DescriptionAnti-aliasing 1D downsampler (low-pass then stride) Usageltx23_downsample1d(ratio = 2L, kernel_size = NULL) Arguments ratio: Integer. Downsampling ratio. kernel_size: Integer or NULL (default 6*ratio rounded even).https://cornball.ai/docs/diffuser/ltx23_encode_audio/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_encode_audio/Encode audio into normalized, packed conditioning latentsDescriptionPads or trims the waveform so the latent length equals audio_num_frames (mel frames 4L - 3, mirroring the decoder’s target_frames), computes the log-mel, encodes in argmax mode, packs, and normalizes with the checkpoint statistics. Usageltx23_encode_audio(audio_vae, wav, audio_num_frames, frontend = NULL) Arguments audio_vae: An ltx23_audio_vae (with encoder weights). wav: Matrix [2, samples] in [-1, 1] at 16 kHz (see ltx23_read_audio). audio_num_frames: Integer. Target latent length. frontend: Optional prebuilt ltx23_audio_mel_frontend.https://cornball.ai/docs/diffuser/ltx23_encode_video_frames/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_encode_video_frames/Encode pixel frames to normalized video latentsDescriptionVAE encode in “argmax” mode (the distribution mean), then normalize with the checkpoint’s per-channel statistics — the exact inverse of the decode path. Usageltx23_encode_video_frames(vae, frames) Arguments vae: An ltx23_video_vae. frames: Tensor [1, 3, F, H, W] in [-1, 1] (see ltx23_preprocess_frames). ValueNormalized latents [1, 128, F’, H/32, W/32] (float32).https://cornball.ai/docs/diffuser/ltx23_feed_forward/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_feed_forward/LTX feed-forward layerDescriptionLinear -> GELU (tanh approximation) -> Linear with 4x hidden dim, matching diffusers FeedForward(activation_fn="gelu-approximate") state-dict names (net.0.proj, net.2). Usageltx23_feed_forward(dim, mult = 4L) Arguments dim: Integer. Input/output dimension. mult: Integer. Hidden dimension multiplier.https://cornball.ai/docs/diffuser/ltx23_fp8_linear/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_fp8_linear/FP8 linear layerDescriptionWeight lives as float8_e4m3fn plus a float32 scale in plain module fields (so $to(device) moves only the bias); the forward pass ships 1 byte/param to the input’s device, upcasts, rescales, and runs nnf_linear. Usageltx23_fp8_linear(out_features, in_features, bias = TRUE) Arguments bias: Logical. out_features,in_features: Integers.https://cornball.ai/docs/diffuser/ltx23_get_timestep_embedding/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_get_timestep_embedding/Sinusoidal timestep embeddingDescriptionDDPM-style sinusoidal embedding. LTX uses flip_sin_to_cos=TRUE (cos first) and downscale_freq_shift=0. Usageltx23_get_timestep_embedding(timesteps, embedding_dim, flip_sin_to_cos = TRUE, downscale_freq_shift = 0, max_period = 10000) Arguments timesteps: 1D tensor of timestep values. embedding_dim: Integer. Output embedding size. flip_sin_to_cos: Logical. Put cos before sin. downscale_freq_shift: Numeric. Frequency delta control. max_period: Numeric. Maximum embedding frequency period. ValueTensor [N, embedding_dim].https://cornball.ai/docs/diffuser/ltx23_is_fp8_cast_key/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_is_fp8_cast_key/Test whether a mapped DiT key is in the official fp8 cast setDescriptionTest whether a mapped DiT key is in the official fp8 cast set Usageltx23_is_fp8_cast_key(mapped_key) Arguments mapped_key: Character vector of mapped (diffusers-style) parameter names. ValueLogical vector.https://cornball.ai/docs/diffuser/ltx23_kaiser_sinc_filter1d/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_kaiser_sinc_filter1d/Kaiser sinc low-pass filter kernelDescriptionKaiser sinc low-pass filter kernel Usageltx23_kaiser_sinc_filter1d(cutoff, half_width, kernel_size) Arguments cutoff: Numeric. Normalized cutoff in (0, 0.5]. half_width: Numeric. Transition band half width. kernel_size: Integer. ValueTensor [kernel_size].https://cornball.ai/docs/diffuser/ltx23_latent_upsampler/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_latent_upsampler/LTX-2.3 latent upsampler modelDescriptionLatents [B, 128, F, H, W] -> [B, 128, F, 2H, 2W]. Usageltx23_latent_upsampler(in_channels = 128L, mid_channels = 1024L, num_blocks_per_stage = 4L) Arguments in_channels: Integer. Latent channels. mid_channels: Integer. num_blocks_per_stage: Integer.https://cornball.ai/docs/diffuser/ltx23_load_group/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_load_group/Stream a checkpoint key group into a moduleDescriptionReads tensors one at a time from an open checkpoint and copies them into the matching parameters/buffers of module. Destination names are derived by map_key; $copy_() handles any dtype/device conversion, so the module may already live on its target device in its target dtype. Usageltx23_load_group(ckpt, keys, module, map_key = identity, verbose = TRUE, gc_every = 50L) Arguments ckpt: An ltx23_checkpoint. keys: Character vector of checkpoint keys to load (one group from ltx23_split_keys).https://cornball.ai/docs/diffuser/ltx23_load_pipeline/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_load_pipeline/Load the LTX-2.3 generation components from a single-file checkpointDescriptionBuilds the transformer, connectors, video VAE, audio VAE, and vocoder with the LTX 2.3 configuration and streams the checkpoint weights into them. The Gemma3 text encoder ships separately (see load_gemma3_text_encoder). Usageltx23_load_pipeline(checkpoint_path, device = "cuda", dtype = "bfloat16", transformer_device = "cpu", components = c("dit", "connectors", "vae", "audio_vae", "vocoder"), pin = TRUE, attn_chunk = NULL, phase_offload = TRUE, verbose = TRUE) Arguments checkpoint_path: Path to the single-file checkpoint (e.https://cornball.ai/docs/diffuser/ltx23_load_transformer_fp8/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_load_transformer_fp8/Load the LTX-2.3 transformer with FP8 weightsDescriptionBuilds the transformer, swaps the official cast-set linears for ltx23_fp8_linear, loads fp8 weights CPU-side (optionally pinned) and everything else as bfloat16 on device. Sets options(diffuseR.block_gc = TRUE) so the transformer runs per-block garbage collection over the dequantized temporaries. Usageltx23_load_transformer_fp8(ckpt, device = "cuda", pin = TRUE, verbose = TRUE, ...) Arguments ckpt: An fp8 ltx23_checkpoint (ltx23_open_fp8_checkpoint). device: Character. Device for the resident (non-fp8) weights. pin: Logical.https://cornball.ai/docs/diffuser/ltx23_load_transformer_nf4/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_load_transformer_nf4/Load the LTX-2.3 transformer with resident NF4 weightsDescriptionBuilds the transformer, swaps the cast-set linears for ltx23_nf4_linear, and loads everything onto device: at ~4.5 bits/parameter the whole 22B transformer stays GPU-resident, avoiding per-step weight transfers. Usageltx23_load_transformer_nf4(ckpt, device = "cuda", verbose = TRUE, ...) Arguments ckpt: An NF4 ltx23_checkpoint (ltx23_open_fp8_checkpoint reads any shard artifact). device: Character. verbose: Logical. ...: Passed to ltx23_transformer (tiny test configs). ValueThe loaded ltx23_transformer.https://cornball.ai/docs/diffuser/ltx23_load_upsampler/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_load_upsampler/Load the LTX-2.3 spatial upscaler weightsDescriptionThe checkpoint keys match this module tree directly. Usageltx23_load_upsampler(path, device = "cuda", dtype = "bfloat16", verbose = TRUE) Arguments path: Path to e.g. ltx-2.3-spatial-upscaler-x2-1.1.safetensors. verbose: Logical. device,dtype: Placement for the loaded model. ValueThe loaded ltx23_latent_upsampler.https://cornball.ai/docs/diffuser/ltx23_map_audio_vae_key/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_map_audio_vae_key/Map an official audio VAE checkpoint key to the R module nameDescriptionMap an official audio VAE checkpoint key to the R module name Usageltx23_map_audio_vae_key(key) Arguments key: Character. Checkpoint key. ValueCharacter.https://cornball.ai/docs/diffuser/ltx23_map_connector_key/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_map_connector_key/Map an official connectors checkpoint key to the R module nameDescriptionMap an official connectors checkpoint key to the R module name Usageltx23_map_connector_key(key) Arguments key: Character. Checkpoint key. ValueCharacter. Module parameter name.https://cornball.ai/docs/diffuser/ltx23_map_dit_key/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_map_dit_key/Map an official DiT checkpoint key to the R module nameDescriptionApplies the official-to-diffusers renames for the LTX-2.3 transformer (cf. diffusers scripts/convert_ltx2_to_diffusers.py). Our module tree matches the diffusers names, so this is the full mapping. Usageltx23_map_dit_key(key) Arguments key: Character. Checkpoint key (with or without the model.diffusion_model. prefix). ValueCharacter. Module parameter/buffer name.https://cornball.ai/docs/diffuser/ltx23_map_vae_key/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_map_vae_key/Map an official VAE checkpoint key to the R module nameDescriptionThe official checkpoint stores the encoder/decoder as flat block lists (down_blocks.0-8 / up_blocks.0-8) where downsamplers/upsamplers and the mid block are separate entries; diffusers (and this port) nest them. Index mapping per diffusers convert_ltx2_to_diffusers.py. Usageltx23_map_vae_key(key) Arguments key: Character. Checkpoint key (with or without “vae.” prefix). ValueCharacter. Module parameter/buffer name.https://cornball.ai/docs/diffuser/ltx23_map_vocoder_key/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_map_vocoder_key/Map an official vocoder checkpoint key to the R module nameDescriptionMap an official vocoder checkpoint key to the R module name Usageltx23_map_vocoder_key(key) Arguments key: Character. Checkpoint key. ValueCharacter. Module parameter/buffer name.https://cornball.ai/docs/diffuser/ltx23_mel_stft/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_mel_stft/Causal log-mel spectrogram with checkpoint-loaded basesDescriptionCausal log-mel spectrogram with checkpoint-loaded bases Usageltx23_mel_stft(filter_length = 512L, hop_length = 80L, window_length = 512L, num_mel_channels = 64L) Arguments filter_length,hop_length,window_length,num_mel_channels: Integers.https://cornball.ai/docs/diffuser/ltx23_memory_profile/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_memory_profile/Get an LTX-2.3 memory profileDescriptionSelects transformer precision, component placement, and attention chunking for the available VRAM. Measured on an RTX 5060 Ti (16 GB): fp8 streaming peaks ~11.6 GB (without phase offloading) at 512x320x49; NF4 keeps the whole 22B transformer resident (~12.5 GB) and removes the ~21 GB/step PCIe weight streaming. The NF4 profile renders 1280x704x121 with audio in ~23 min at a 15.7 GB peak (tiled VAE decode, in-place feed-forward GELU, and the default diffuseR.https://cornball.ai/docs/diffuser/ltx23_nf4_dequantize/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_nf4_dequantize/Dequantize NF4 data to a float tensorDescriptionDequantize NF4 data to a float tensor Usageltx23_nf4_dequantize(packed, absmax, shape, dtype = torch::torch_bfloat16(), chunk_elements = 8388608L, out = NULL) Arguments packed: uint8 tensor of packed index pairs. absmax: float32 tensor of per-block scales. shape: Integer vector. Original tensor shape. dtype: Target torch dtype. chunk_elements: Integer. Elements dequantized per slice (bounds the int64 index temporary). out: Optional preallocated tensor of shape to write into (avoids allocating a fresh weight tensor per call).https://cornball.ai/docs/diffuser/ltx23_nf4_linear/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_nf4_linear/NF4 linear layerDescriptionPacked weights and per-block scales are registered as buffers, so they move with the module (uint8 packs are untouched by dtype conversions). The forward pass dequantizes on the weight’s device. Usageltx23_nf4_linear(out_features, in_features, bias = TRUE) Arguments bias: Logical. out_features,in_features: Integers.https://cornball.ai/docs/diffuser/ltx23_nf4_quantize/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_nf4_quantize/Quantize a tensor to NF4DescriptionQuantize a tensor to NF4 Usageltx23_nf4_quantize(x) Arguments x: Float tensor (any shape; total elements must be a multiple of 128, i.e. two 64-element blocks - always true for the LTX linears). ValueList with packed (uint8, two indices per byte) and absmax (float32, one per 64-element block).https://cornball.ai/docs/diffuser/ltx23_normalize_latents/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_normalize_latents/Normalize latents with the VAE’s per-channel statisticsDescriptionNormalize latents with the VAE’s per-channel statistics Usageltx23_normalize_latents(latents, latents_mean, latents_std) Arguments latents: Tensor [B, C, F, H, W]. latents_mean,latents_std: Tensors [C]. ValueNormalized latents.https://cornball.ai/docs/diffuser/ltx23_open_checkpoint/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_open_checkpoint/Open an LTX-2.3 checkpointDescriptionOpens a single-file LTX checkpoint lazily (header only), validates the model_version metadata, and parses the embedded component configuration. Usageltx23_open_checkpoint(path, require_version = "2.3") Arguments path: Path to the checkpoint .safetensors file. require_version: Character. Required model_version prefix (default “2.3”). Set to NULL to skip the check. ValueAn object of class ltx23_checkpoint: a list with handle (safetensors reader), keys, version, config (parsed component configs, or NULL), and path. Examplesckpt <- ltx23_open_checkpoint("ltx-2.https://cornball.ai/docs/diffuser/ltx23_open_fp8_checkpoint/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_open_fp8_checkpoint/Open an FP8 shard directory as a checkpointDescriptionPresents the sharded fp8 artifact through the same interface as ltx23_open_checkpoint so the group loaders work unchanged. Usageltx23_open_fp8_checkpoint(dir) Arguments dir: The fp8 artifact directory (with manifest.json). ValueAn ltx23_checkpoint.https://cornball.ai/docs/diffuser/ltx23_per_channel_rms_norm/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_per_channel_rms_norm/Per-channel RMS normalizationDescriptionNormalizes by the root-mean-square across the channel dimension (dim 2 of [B, C, F, H, W]); no learned parameters. Usageltx23_per_channel_rms_norm(eps = 1e-08) Arguments eps: Numeric. Stability epsilon.https://cornball.ai/docs/diffuser/ltx23_per_token_rms_norm/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_per_token_rms_norm/Per-token RMS normalization over the channel axisDescriptionPer-token RMS normalization over the channel axis Usageltx23_per_token_rms_norm(x, eps = 1e-06) Arguments x: Tensor [B, S, C, L] of stacked per-layer hidden states. eps: Numeric. Stability epsilon. ValueTensor of the same shape.https://cornball.ai/docs/diffuser/ltx23_prepare_conditioned_latents/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_prepare_conditioned_latents/Build conditioned initial latents and the conditioning maskDescriptioni2v (cond_latents has one latent frame): the encoded frame is repeated across all latent frames and only latent frame 0 is marked conditioned. Continuation (k latent frames): the prefix tokens are replaced and marked. Unconditioned positions start as pure noise. Usageltx23_prepare_conditioned_latents(cond_latents, latent_frames, latent_height, latent_width, noise, cond_noise_scale = 0) Arguments cond_latents: Normalized condition latents [1, 128, k, H’, W’] from ltx23_encode_video_frames. noise: Tensor [1, 128, F’, H’, W’] of standard noise (caller provides so seeding stays in one place).https://cornball.ai/docs/diffuser/ltx23_preprocess_frames/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_preprocess_frames/Preprocess an image (or frame stack) for VAE encodingDescriptionMirrors the diffusers VideoProcessor: bilinear resize so the shorter relative side matches, center-crop to the exact target, and scale to [-1, 1]. Usageltx23_preprocess_frames(x, height, width) Arguments x: Path to a PNG/JPEG, or an array [H, W, 3] (values in [0, 1]), or a [F, H, W, 3] array of frames. height,width: Integers. Target size (multiples of 32). ValueFloat32 tensor [1, 3, F, height, width] in [-1, 1].https://cornball.ai/docs/diffuser/ltx23_quantize_fp8/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_quantize_fp8/Quantize an LTX-2.3 checkpoint to FP8 shardsDescriptionStreams the single-file bf16 checkpoint tensor by tensor. DiT attention/FFN linear weights are stored as float8_e4m3fn with a float32 absmax/448 per-tensor scale (<key>_scale sibling); everything else is copied through unchanged. Output shards carry the original key names plus a manifest for skip-if-exists. Usageltx23_quantize_fp8(checkpoint_path, output_dir = file.path(tools::R_user_dir("diffuseR", "data"), "ltx2.3-fp8"), shard_bytes = 4e+09, force = FALSE, verbose = TRUE) Arguments checkpoint_path: Source .safetensors (46 GB bf16 single file).https://cornball.ai/docs/diffuser/ltx23_quantize_nf4/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_quantize_nf4/Quantize an LTX-2.3 checkpoint to NF4 shardsDescriptionSame streaming layout and cast policy as ltx23_quantize_fp8, but cast-set weights are stored as NF4 (<key> packed uint8 + <key>_absmax float32 blocks + the original shape recovered from the model config at load time). Non-cast tensors are copied through unchanged. The manifest carries format = "nf4". Usageltx23_quantize_nf4(checkpoint_path, output_dir = file.path(tools::R_user_dir("diffuseR", "data"), "ltx2.3-nf4"), shard_bytes = 4e+09, force = FALSE, verbose = TRUE) Arguments checkpoint_path: Source .https://cornball.ai/docs/diffuser/ltx23_read_audio/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_read_audio/Read an audio file as 16 kHz stereo PCMDescriptionDecodes MP3/WAV/etc. via av to 16-bit PCM at the target rate and parses the RIFF container in base R. Usageltx23_read_audio(path, sample_rate = 16000L) Arguments path: Audio file. sample_rate: Integer. ValueMatrix [2, samples] in [-1, 1].https://cornball.ai/docs/diffuser/ltx23_read_tail_frames/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_read_tail_frames/Read the trailing frames of a video fileDescriptionExtracts the last n frames of an MP4 (via av) for use as continuation conditioning. Usageltx23_read_tail_frames(path, n = 9L) Arguments path: Video file. n: Integer. Trailing frame count. ValueArray [n, H, W, 3] in [0, 1].https://cornball.ai/docs/diffuser/ltx23_release_dequant_buffers/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_release_dequant_buffers/Release the NF4 dequantization buffersDescriptionFrees the cached per-shape weight buffers (e.g. before decoding at high resolution). Usageltx23_release_dequant_buffers() ValueInvisibly, NULL.https://cornball.ai/docs/diffuser/ltx23_rms_norm/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_rms_norm/RMS normalizationDescriptionVariance is computed in float32; the result is cast back to the input dtype (or the weight dtype when elementwise affine). Usageltx23_rms_norm(dim, eps = 1e-06, elementwise_affine = TRUE) Arguments dim: Integer. Normalized dimension size. eps: Numeric. Stability epsilon. elementwise_affine: Logical. Learn a scale weight.https://cornball.ai/docs/diffuser/ltx23_rotary_pos_embed/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_rotary_pos_embed/LTX-2.3 audio/video rotary position embedderDescriptionComputes RoPE cos/sin frequency tensors from spatiotemporal patch coordinates. Video coordinates are 3D (frames scaled to seconds via fps, height, width in pixel space); audio coordinates are 1D (seconds). Coordinates are patch boundaries [start, end); the midpoint is used as the position. Usageltx23_rotary_pos_embed(dim, patch_size = 1L, patch_size_t = 1L, base_num_frames = 20L, base_height = 2048L, base_width = 2048L, sampling_rate = 16000L, hop_length = 160L, scale_factors = c(8L, 32L, 32L), theta = 10000, causal_offset = 1L, modality = "video", double_precision = TRUE, rope_type = "split", num_attention_heads = 32L) Arguments dim: Integer.https://cornball.ai/docs/diffuser/ltx23_rotary_pos_embed_1d/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_rotary_pos_embed_1d/1D rotary embeddings for the text connectorsDescription1D rotary embeddings for the text connectors Usageltx23_rotary_pos_embed_1d(dim, base_seq_len = 4096L, theta = 10000, double_precision = TRUE, rope_type = "split", num_attention_heads = 32L) Arguments dim: Integer. Rotary dimension (connector inner dim). base_seq_len: Integer. Base sequence length for normalization. theta: Numeric. RoPE theta. double_precision: Logical. Compute base frequencies in float64. rope_type: “split” (LTX-2.3) or “interleaved”. num_attention_heads: Integer. For the split per-head layout.https://cornball.ai/docs/diffuser/ltx23_set_attn_chunk/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_set_attn_chunk/Set the attention query-chunk size across a transformerDescriptionR torch has no fused attention, so the [B, H, S, S] matrix materializes; chunking queries bounds the peak. NULL disables chunking. Usageltx23_set_attn_chunk(transformer, chunk) Arguments transformer: An ltx23_transformer. chunk: Integer or NULL. ValueInvisibly, the transformer.https://cornball.ai/docs/diffuser/ltx23_snake_beta/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_snake_beta/SnakeBeta activationDescriptionx + (1 / (beta + eps)) * sin(x * alpha)^2 with per-channel log-scale alpha/beta parameters. Usageltx23_snake_beta(channels, eps = 1e-09) Arguments channels: Integer. eps: Numeric.https://cornball.ai/docs/diffuser/ltx23_split_keys/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_split_keys/Split checkpoint keys by componentDescriptionSplits the flat key space of an LTX single-file checkpoint into its component groups. Connector tensors live under the model.diffusion_model. prefix alongside the transformer, plus a top-level text_embedding_projection. group; both are routed to the connectors component. Usageltx23_split_keys(keys) Arguments keys: Character vector of checkpoint tensor names. ValueNamed list of character vectors: dit, connectors, vae, audio_vae, vocoder, and other (anything unrecognized; should be empty).https://cornball.ai/docs/diffuser/ltx23_stage2_distilled_sigmas/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_stage2_distilled_sigmas/Stage-2 distilled sigma schedule (two-stage refinement)DescriptionStage-2 distilled sigma schedule (two-stage refinement) Usageltx23_stage2_distilled_sigmas() ValueNumeric vector of length 4.https://cornball.ai/docs/diffuser/ltx23_text_connectors/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_text_connectors/LTX-2.3 text connectorsDescriptionTakes raw stacked per-layer text encoder hidden states and produces the video and audio text embeddings for the DiT: per-token RMS norm, per-modality sqrt(dim ratio) rescaling and projection, then a per-modality 1D connector transformer. Usageltx23_text_connectors(caption_channels = 3840L, text_proj_in_factor = 49L, video_connector_num_attention_heads = 32L, video_connector_attention_head_dim = 128L, video_connector_num_layers = 8L, video_connector_num_learnable_registers = 128L, video_gated_attn = TRUE, audio_connector_num_attention_heads = 32L, audio_connector_attention_head_dim = 64L, audio_connector_num_layers = 8L, audio_connector_num_learnable_registers = 128L, audio_gated_attn = TRUE, connector_rope_base_seq_len = 4096L, rope_theta = 10000, rope_double_precision = TRUE, rope_type = "split", video_hidden_dim = 4096L, audio_hidden_dim = 2048L, proj_bias = TRUE) Arguments caption_channels: Integer.https://cornball.ai/docs/diffuser/ltx23_tone_map_latents/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_tone_map_latents/Sigmoid tone mapping for latentsDescriptionCompresses the latent dynamic range (cf. diffusers tone_map_latents). compression 0 is identity, 1 is the full effect. Usageltx23_tone_map_latents(latents, compression) Arguments latents: Tensor. compression: Numeric in [0, 1]. ValueTone-mapped latents.https://cornball.ai/docs/diffuser/ltx23_transformer/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_transformer/LTX-2.3 video transformer modelDescriptionDual-stream audio/video DiT. Text embeddings arrive already projected to the video (inner_dim) and audio (audio_inner_dim) dimensions by the connector modules. Usageltx23_transformer(in_channels = 128L, out_channels = 128L, patch_size = 1L, patch_size_t = 1L, num_attention_heads = 32L, attention_head_dim = 128L, cross_attention_dim = 4096L, vae_scale_factors = c(8L, 32L, 32L), pos_embed_max_pos = 20L, base_height = 2048L, base_width = 2048L, gated_attn = TRUE, cross_attn_mod = TRUE, audio_in_channels = 128L, audio_out_channels = 128L, audio_patch_size = 1L, audio_patch_size_t = 1L, audio_num_attention_heads = 32L, audio_attention_head_dim = 64L, audio_cross_attention_dim = 2048L, audio_scale_factor = 4L, audio_pos_embed_max_pos = 20L, audio_sampling_rate = 16000L, audio_hop_length = 160L, audio_gated_attn = TRUE, audio_cross_attn_mod = TRUE, num_layers = 48L, norm_eps = 1e-06, rope_theta = 10000, rope_double_precision = TRUE, causal_offset = 1L, timestep_scale_multiplier = 1000, cross_attn_timestep_scale_multiplier = 1000, rope_type = "split", perturbed_attn = TRUE) Arguments cross_attention_dim: Integer.https://cornball.ai/docs/diffuser/ltx23_transformer_block/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_transformer_block/LTX-2 transformer blockDescriptionDual-stream (video + audio) block: modulated self-attention per modality, text cross-attention per modality (with LTX-2.3 query and key/value modulation), bidirectional audio-video cross-attention with global+per-block modulation, and modulated feed-forward. Usageltx23_transformer_block(dim, num_attention_heads, attention_head_dim, cross_attention_dim, audio_dim, audio_num_attention_heads, audio_attention_head_dim, audio_cross_attention_dim, video_gated_attn = TRUE, video_cross_attn_adaln = TRUE, audio_gated_attn = TRUE, audio_cross_attn_adaln = TRUE, eps = 1e-06, elementwise_affine = FALSE, rope_type = "split", perturbed_attn = TRUE) Arguments cross_attention_dim: Integer. Text embedding dim for video. audio_cross_attention_dim: Integer.https://cornball.ai/docs/diffuser/ltx23_tune_gc/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_tune_gc/Tune the torch CUDA allocator for large-resident inferenceDescriptionStops the allocator GC storm (cf. ~/skills/torch torch-jit-gc-performance.md): lantern proactively calls R’s gc() whenever reserved memory exceeds torch.cuda_allocator_reserved_rate (default 0.20) of the card. With ~75 weights that fires on nearly every allocation. Raising the rate to the actual footprint is safe here because the LTX hot loops compute into persistent scratch buffers (near-zero per-step garbage). Also raises the host-allocation GC threshold and defaults PYTORCH_CUDA_ALLOC_CONF to expandable segments.https://cornball.ai/docs/diffuser/ltx23_upsample1d/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_upsample1d/Anti-aliasing 1D upsampler (transposed low-pass)DescriptionAnti-aliasing 1D upsampler (transposed low-pass) Usageltx23_upsample1d(ratio = 2L, kernel_size = NULL, window_type = "kaiser", persistent = TRUE) Arguments ratio: Integer. Upsampling ratio. kernel_size: Integer or NULL. window_type: “kaiser” (BigVGAN default) or “hann” (final resampler). persistent: Logical. Register the filter as a buffer (present in checkpoints); FALSE stores the computed filter as a plain field.https://cornball.ai/docs/diffuser/ltx23_video_decoder3d/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_video_decoder3d/LTX-2.3 video decoderDescriptionLatents [B, 128, F, H, W] -> pixel video [B, 3, 8F - 7, 32H, 32W]. Block channel lists are given encoder-side (as in the config) and reversed internally; upsample_type is indexed directly. Usageltx23_video_decoder3d(in_channels = 128L, out_channels = 3L, block_out_channels = c(256L, 512L, 512L, 1024L), spatio_temporal_scaling = c(TRUE, TRUE, TRUE, TRUE), layers_per_block = c(4L, 6L, 4L, 2L, 2L), upsample_type = c("spatiotemporal", "spatiotemporal", "temporal", "spatial"), patch_size = 4L, patch_size_t = 1L, resnet_norm_eps = 1e-06, is_causal = FALSE, upsample_residual = c(FALSE, FALSE, FALSE, FALSE), upsample_factor = c(2L, 2L, 1L, 2L), spatial_padding_mode = "zeros") Arguments block_out_channels: Integer vector (config order).https://cornball.ai/docs/diffuser/ltx23_video_down_block3d/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_video_down_block3d/LTX video down blockDescriptionResNet stack (at the input channel count) followed by a pixel-unshuffle downsampler that also changes the channel count. Usageltx23_video_down_block3d(in_channels, out_channels = NULL, num_layers = 1L, resnet_eps = 1e-06, spatio_temporal_scale = TRUE, downsample_type = "spatiotemporal", spatial_padding_mode = "zeros") Arguments num_layers: Integer. ResNet count. resnet_eps: Numeric. spatio_temporal_scale: Logical. Whether to downsample at all. downsample_type: “spatial”, “temporal”, or “spatiotemporal”. spatial_padding_mode: Character. in_channels,out_channels: Integers.https://cornball.ai/docs/diffuser/ltx23_video_downsampler3d/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_video_downsampler3d/Pixel-unshuffle 3D downsamplerDescriptionConv followed by space/time-to-channel rearrangement, plus a grouped channel-mean residual of the same rearrangement. Usageltx23_video_downsampler3d(in_channels, out_channels, stride = c(1L, 1L, 1L), spatial_padding_mode = "zeros") Arguments stride: Length-3 integer vector (t, h, w). spatial_padding_mode: Character. in_channels,out_channels: Integers.https://cornball.ai/docs/diffuser/ltx23_video_encoder3d/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_video_encoder3d/LTX-2.3 video encoderDescriptionPixel video [B, 3, F, H, W] -> latent statistics [B, 2 * latent_channels, F/8, H/32, W/32] (mean and a uniform log-var channel broadcast across the latent channels). Usageltx23_video_encoder3d(in_channels = 3L, out_channels = 128L, block_out_channels = c(256L, 512L, 1024L, 1024L), spatio_temporal_scaling = c(TRUE, TRUE, TRUE, TRUE), layers_per_block = c(4L, 6L, 4L, 2L, 2L), downsample_type = c("spatial", "temporal", "spatiotemporal", "spatiotemporal"), patch_size = 4L, patch_size_t = 1L, resnet_norm_eps = 1e-06, is_causal = TRUE, spatial_padding_mode = "zeros") Arguments block_out_channels: Integer vector.https://cornball.ai/docs/diffuser/ltx23_video_mid_block3d/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_video_mid_block3d/LTX video mid blockDescriptionA plain ResNet stack at a fixed channel count. Usageltx23_video_mid_block3d(in_channels, num_layers = 1L, resnet_eps = 1e-06, spatial_padding_mode = "zeros") Arguments in_channels: Integer. num_layers: Integer. resnet_eps: Numeric. spatial_padding_mode: Character.https://cornball.ai/docs/diffuser/ltx23_video_resnet_block3d/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_video_resnet_block3d/LTX 3D ResNet blockDescriptionPerChannelRMSNorm -> SiLU -> causal conv, twice, with a LayerNorm + 1x1 Conv3d shortcut when the channel count changes. Usageltx23_video_resnet_block3d(in_channels, out_channels = NULL, eps = 1e-06, spatial_padding_mode = "zeros") Arguments eps: Numeric. Shortcut LayerNorm epsilon. spatial_padding_mode: Character. in_channels,out_channels: Integers.https://cornball.ai/docs/diffuser/ltx23_video_up_block3d/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_video_up_block3d/LTX video up blockDescriptionOptional channel-changing conv-in ResNet, pixel-shuffle upsampler, then a ResNet stack at the output channel count. Usageltx23_video_up_block3d(in_channels, out_channels = NULL, num_layers = 1L, resnet_eps = 1e-06, spatio_temporal_scale = TRUE, upsample_type = "spatiotemporal", upsample_residual = FALSE, upscale_factor = 1L, spatial_padding_mode = "zeros") Arguments num_layers: Integer. resnet_eps: Numeric. spatio_temporal_scale: Logical. upsample_type: “spatial”, “temporal”, or “spatiotemporal”. upsample_residual: Logical. upscale_factor: Integer. spatial_padding_mode: Character. in_channels,out_channels: Integers.https://cornball.ai/docs/diffuser/ltx23_video_upsampler3d/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_video_upsampler3d/Pixel-shuffle 3D upsamplerDescriptionConv followed by channel-to-space/time rearrangement, with an optional channel-repeat residual and an upscale factor that divides the conv output channels. Usageltx23_video_upsampler3d(in_channels, stride = c(1L, 1L, 1L), residual = FALSE, upscale_factor = 1L, spatial_padding_mode = "zeros") Arguments in_channels: Integer. stride: Length-3 integer vector (t, h, w). residual: Logical. Add the rearranged input as a residual. upscale_factor: Integer. spatial_padding_mode: Character.https://cornball.ai/docs/diffuser/ltx23_video_vae/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_video_vae/LTX-2.3 video VAEDescriptionEncoder + decoder + per-channel latent statistics (loaded from the checkpoint’s per_channel_statistics). The checkpoint’s scaling_factor is 1.0, so latent (de)normalization is purely the per-channel affine map. Usageltx23_video_vae(in_channels = 3L, out_channels = 3L, latent_channels = 128L, block_out_channels = c(256L, 512L, 1024L, 1024L), decoder_block_out_channels = c(256L, 512L, 512L, 1024L), layers_per_block = c(4L, 6L, 4L, 2L, 2L), decoder_layers_per_block = c(4L, 6L, 4L, 2L, 2L), spatio_temporal_scaling = c(TRUE, TRUE, TRUE, TRUE), decoder_spatio_temporal_scaling = c(TRUE, TRUE, TRUE, TRUE), downsample_type = c("spatial", "temporal", "spatiotemporal", "spatiotemporal"), upsample_type = c("spatiotemporal", "spatiotemporal", "temporal", "spatial"), upsample_residual = c(FALSE, FALSE, FALSE, FALSE), upsample_factor = c(2L, 2L, 1L, 2L), patch_size = 4L, patch_size_t = 1L, resnet_norm_eps = 1e-06, encoder_causal = TRUE, decoder_causal = FALSE, encoder_spatial_padding_mode = "zeros", decoder_spatial_padding_mode = "zeros") Arguments latent_channels: Integer.https://cornball.ai/docs/diffuser/ltx23_vocoder/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_vocoder/LTX-2.3 vocoder stageDescriptionMel spectrogram [B, C, T, M] -> waveform [B, out_channels, samples]. Channel and mel dims are flattened into conv channels; each upsample stage halves the channel count and averages three parallel ResNet branches. Usageltx23_vocoder(in_channels = 128L, hidden_channels = 1536L, out_channels = 2L, upsample_kernel_sizes = c(11L, 4L, 4L, 4L, 4L, 4L), upsample_factors = c(5L, 2L, 2L, 2L, 2L, 2L), resnet_kernel_sizes = c(3L, 7L, 11L), resnet_dilations = list(c(1L, 3L, 5L), c(1L, 3L, 5L), c(1L, 3L, 5L)), antialias_ratio = 2L, antialias_kernel_size = 12L, final_bias = FALSE) Arguments in_channels: Integer.https://cornball.ai/docs/diffuser/ltx23_vocoder_resblock/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_vocoder_resblock/Vocoder ResNet block (AMP)DescriptionDilated conv pairs, each preceded by an anti-aliased SnakeBeta activation, with residual connections. Usageltx23_vocoder_resblock(channels, kernel_size = 3L, dilations = c(1L, 3L, 5L), antialias_ratio = 2L, antialias_kernel_size = 12L) Arguments channels: Integer. kernel_size: Integer. dilations: Integer vector. antialias_ratio,antialias_kernel_size: Integers.https://cornball.ai/docs/diffuser/ltx23_vocoder_with_bwe/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/ltx23_vocoder_with_bwe/LTX-2.3 vocoder with bandwidth extensionDescriptionFull mel [B, 2, T, 64] -> 48 kHz stereo waveform pipeline: 16 kHz vocoder, causal mel re-analysis, BWE vocoder residual added to a Hann-resampled skip path, clamped to [-1, 1]. Usageltx23_vocoder_with_bwe(in_channels = 128L, hidden_channels = 1536L, out_channels = 2L, upsample_kernel_sizes = c(11L, 4L, 4L, 4L, 4L, 4L), upsample_factors = c(5L, 2L, 2L, 2L, 2L, 2L), resnet_kernel_sizes = c(3L, 7L, 11L), resnet_dilations = list(c(1L, 3L, 5L), c(1L, 3L, 5L), c(1L, 3L, 5L)), bwe_in_channels = 128L, bwe_hidden_channels = 512L, bwe_upsample_kernel_sizes = c(12L, 11L, 4L, 4L, 4L), bwe_upsample_factors = c(6L, 5L, 2L, 2L, 2L), bwe_resnet_kernel_sizes = c(3L, 7L, 11L), bwe_resnet_dilations = list(c(1L, 3L, 5L), c(1L, 3L, 5L), c(1L, 3L, 5L)), filter_length = 512L, hop_length = 80L, window_length = 512L, num_mel_channels = 64L, input_sampling_rate = 16000L, output_sampling_rate = 48000L) Arguments out_channels: Integer.https://cornball.ai/docs/diffuser/models2devices/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/models2devices/models2devicesDescriptionThis function sets up the model directory, device configuration, and data types for diffusion models. It checks the validity of the model name and devices, detects model type, and downloads the model if necessary. Usagemodels2devices(model_name, devices = "cpu", unet_dtype_str = NULL, download_models = FALSE) Arguments model_name: A character string representing the name of the model to be used. devices: A character string or a named list specifying the devices for different components of the model.https://cornball.ai/docs/diffuser/offload_to_cpu/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/offload_to_cpu/Offload Module to CPUDescriptionMoves a torch module and all its parameters to CPU. Usageoffload_to_cpu(module, gc = TRUE) Arguments module: A torch nn_module. gc: Logical. Run garbage collection after offload. ValueThe module (modified in place). Examplesmodel$to(device = "cuda") output <- model(x) offload_to_cpu(model)https://cornball.ai/docs/diffuser/post_quant_conv/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/post_quant_conv/Post Quant ConvDescriptionThis function applies a quantized convolution operation to an input tensor. It is typically used in the context of image post processing, particularly in generative models like Stable Diffusion XL. Usagepost_quant_conv(x, dtype, device) Arguments x: Input tensor to be processed. dtype: Data type for the tensor (e.g., “torch_float16” or “torch_float32”). device: Device on which the tensor is located (e.g., “cpu” or “cuda”). ValueProcessed tensor after applying the quantized convolution.https://cornball.ai/docs/diffuser/preprocess_image/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/preprocess_image/Preprocess image for Stable DiffusionDescriptionPreprocess image for Stable Diffusion Usagepreprocess_image(input, device = "cpu", width = 512, height = 512) Arguments input: File path to .jpg or .png, or a 3D array device: Target device for torch (“cpu” or “cuda”) width: Desired width of the output image height: Desired height of the output image ValueTorch tensor of shape c(1, 3, 512, 512), scaled to c(-1, 1)https://cornball.ai/docs/diffuser/quant_conv/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/quant_conv/Quant ConvDescriptionThis function applies a quantized convolution operation to an input tensor. It is typically used in the context of image processing, particularly in generative models like Stable Diffusion. Usagequant_conv(x, dtype, device) Arguments x: Input tensor to be processed. dtype: Data type for the tensor (e.g., “torch_float16” or “torch_float32”). device: Device on which the tensor is located (e.g., “cpu” or “cuda”). ValueProcessed tensor after applying the quantized convolution.https://cornball.ai/docs/diffuser/qwen_bpe_tokenizer/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/qwen_bpe_tokenizer/Load a Qwen2 byte-level BPE tokenizerDescriptionLoad a Qwen2 byte-level BPE tokenizer Usageqwen_bpe_tokenizer(tokenizer_path) Arguments tokenizer_path: Path to a tokenizer.json (or a directory containing one). ValueA qwen_tokenizer object.https://cornball.ai/docs/diffuser/qwen3_encoder/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/qwen3_encoder/Qwen3 encoder stackDescriptionDefaults are the Qwen3-4B configuration used by FLUX.2 klein. The module tree mirrors the checkpoint keys (model.embed_tokens, model.layers.*, model.norm); the tied LM head carries no weights of its own and is not implemented. Usageqwen3_encoder(vocab_size = 151936L, hidden_size = 2560L, intermediate_size = 9728L, num_hidden_layers = 36L, num_attention_heads = 32L, num_key_value_heads = 8L, head_dim = 128L, rope_theta = 1e+06, rms_norm_eps = 1e-06) Arguments rope_theta: Numeric. rms_norm_eps: Numeric. vocab_size,hidden_size,intermediate_size,num_hidden_layers: Integers. num_attention_heads,num_key_value_heads,head_dim: Integers.https://cornball.ai/docs/diffuser/save_frames/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/save_frames/Save Video Frames as Individual ImagesDescriptionSave Video Frames as Individual Images Usagesave_frames(video, dir, prefix = "frame_", format = "png", verbose = TRUE) Arguments video: Array of video frames [T, H, W, C]. dir: Directory to save frames in. prefix: Character. Filename prefix (default “frame_”). format: Character. Image format: “png” or “jpg”. verbose: Logical. ValueInvisibly returns vector of saved file paths.https://cornball.ai/docs/diffuser/save_image/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/save_image/Save and Display an Image from a Torch TensorDescriptionConverts a Torch tensor to a normalized RGB image array, saves it as a PNG file, and optionally displays it in the RStudio Viewer pane using grid::grid.raster(). Usagesave_image(img, save_to = "output.png", normalize = TRUE) Arguments img: A numeric with shape [3, H, W]. save_to: File path for the PNG image (default is "output.png"). normalize: Logical; whether to normalize pixel values to [0, 1].https://cornball.ai/docs/diffuser/save_video/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/save_video/Save Video to FileDescriptionSaves a video array to a file in various formats. Usagesave_video(video, file, fps = 24, format = NULL, backend = "auto", quality = 85, verbose = TRUE) Arguments video: Array of video frames with shape [T, H, W, C] where C is 3 (RGB). Values should be in [0, 1] range. file: Character. Output file path. Extension determines format. fps: Numeric. Frames per second (default 24). format: Character.https://cornball.ai/docs/diffuser/save_video_ltx23/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/save_video_ltx23/Save an LTX video (optionally with audio) to MP4DescriptionUses the av package (Suggests) to encode frames and mux the audio track. Usagesave_video_ltx23(video, filename, fps = 24, audio = NULL, sample_rate = 48000L, verbose = TRUE) Arguments video: Array [frames, height, width, 3] in [0, 1]. filename: Output path (.mp4). fps: Numeric. audio: Optional numeric matrix [channels, samples] in [-1, 1]. sample_rate: Integer. verbose: Logical. ValueInvisibly, the filename.https://cornball.ai/docs/diffuser/scheduler_add_noise/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/scheduler_add_noise/Add noise to latents using DDIM schedulerDescriptionThis function adds noise to the original latents according to the DDIM scheduler’s diffusion process. It computes the noisy latents based on the original latents, noise, and the current timestep. Usagescheduler_add_noise(original_latents, noise, timestep, scheduler_obj) Arguments original_latents: A torch tensor representing the original latents. noise: A torch tensor representing the noise to be added. timestep: An integer representing the current timestep in the diffusion process. scheduler_obj: A list containing the DDIM scheduler parameters, including alphas_cumprod and timesteps.https://cornball.ai/docs/diffuser/sdxl_memory_profile/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/sdxl_memory_profile/Get SDXL Memory ProfileDescriptionDetermines optimal memory configuration for SDXL image generation based on available VRAM. Usagesdxl_memory_profile(vram_gb = NULL) Arguments vram_gb: Numeric. Available VRAM in GB, or NULL for auto-detection. DetailsMemory profiles for SDXL: full_gpu: 16GB+ - All components on CUDA balanced: 10-12GB - UNet + decoder on CUDA, text encoders on CPU unet_gpu: 6-10GB - Only UNet on CUDA, everything else CPU cpu_only: <6GB - All on CPU Each profile also specifies:https://cornball.ai/docs/diffuser/t5_encoder/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/t5_encoder/T5 encoder stackDescriptionDefaults are the T5-v1.1-XXL configuration used by FLUX. Usaget5_encoder(vocab_size = 32128L, d_model = 4096L, d_kv = 64L, num_heads = 64L, d_ff = 10240L, num_layers = 24L, relative_attention_num_buckets = 32L, relative_attention_max_distance = 128L, layer_norm_epsilon = 1e-06) Arguments layer_norm_epsilon: Numeric. vocab_size,d_model,d_kv,num_heads,d_ff,num_layers: Integers. relative_attention_num_buckets,relative_attention_max_distance: Integers. Relative position bias shape. ValueModule whose forward(input_ids) (1-based ids [B, S]) returns the last hidden state [B, S, d_model].https://cornball.ai/docs/diffuser/text_encoder_native/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/text_encoder_native/Native CLIP Text EncoderDescriptionNative R torch implementation of CLIP text encoder. Replaces TorchScript for better GPU compatibility. Usagetext_encoder_native(vocab_size = 49408, context_length = 77, embed_dim = 768, num_layers = 12, num_heads = 12, mlp_dim = 3072, apply_final_ln = TRUE, gelu_type = "tanh") Arguments vocab_size: Vocabulary size (default 49408) context_length: Maximum sequence length (default 77) embed_dim: Embedding dimension num_layers: Number of transformer layers num_heads: Number of attention heads mlp_dim: MLP hidden dimension apply_final_ln: Whether to apply final layer norm (default TRUE).https://cornball.ai/docs/diffuser/text_encoder2_native/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/text_encoder2_native/Native CLIP Text Encoder 2 (OpenCLIP ViT-bigG for SDXL)DescriptionNative R torch implementation of OpenCLIP text encoder used in SDXL. Returns both hidden states and pooled output. Usagetext_encoder2_native(vocab_size = 49408, context_length = 77, embed_dim = 1280, num_layers = 32, num_heads = 20, mlp_dim = 5120) Arguments vocab_size: Vocabulary size (default 49408) context_length: Maximum sequence length (default 77) embed_dim: Embedding dimension (default 1280) num_layers: Number of transformer layers (default 32) num_heads: Number of attention heads (default 20) mlp_dim: MLP hidden dimension (default 5120) ValueAn nn_module representing the text encoderhttps://cornball.ai/docs/diffuser/tokenize_gemma3/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/tokenize_gemma3/Tokenize text for Gemma3DescriptionTokenize text for Gemma3 Usagetokenize_gemma3(tokenizer, text, max_length = 1024L, padding = "max_length", return_tensors = "pt") Arguments tokenizer: Gemma3 tokenizer object. text: Character vector of prompts. max_length: Integer. Maximum sequence length. padding: Character. Padding strategy (“left”, “right”, “max_length”, “none”). return_tensors: Character. Return type (“list” or “pt” for torch tensors). ValueList with input_ids and attention_mask.https://cornball.ai/docs/diffuser/txt2img/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/txt2img/Generate an image from a text prompt using a diffusion pipelineDescriptionGenerate an image from a text prompt using a diffusion pipeline Usagetxt2img(prompt, model_name = c("sd21", "sdxl", "flux1", "flux2"), ...) Arguments prompt: A character string prompt describing the image to generate. model_name: Name of the model to use (e.g., "sd21"). ...: Additional parameters passed to the diffusion process. ValueA tensor or image object, depending on implementation. Examplesimg <- txt2img("a cat wearing sunglasses in space", device = "cuda")https://cornball.ai/docs/diffuser/txt2img_flux/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/txt2img_flux/Generate an image with FLUX.1-schnellDescription4-step distilled text-to-image generation (no classifier-free guidance): T5 + CLIP prompt encoding, flow-matching Euler denoising over the packed latent sequence, and 16-channel VAE decode. With phase offloading each component is the sole GPU tenant for its phase. Usagetxt2img_flux(prompt, pipeline = NULL, width = 1024L, height = 1024L, num_inference_steps = 4L, max_sequence_length = 256L, seed = NULL, prompt_embeds = NULL, pooled_prompt_embeds = NULL, save_file = TRUE, filename = NULL, verbose = TRUE, .https://cornball.ai/docs/diffuser/txt2img_flux2/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/txt2img_flux2/Generate an image with FLUX.2 kleinDescriptionStep-distilled text-to-image (klein-4B: 4 steps, no guidance): Qwen3 prompt encoding (chat template, mid-stack hidden states), FlowMatch denoising with the empirical dynamic shift, and 32-channel VAE decode through the BatchNorm latent statistics. Usagetxt2img_flux2(prompt, pipeline = NULL, width = 1024L, height = 1024L, num_inference_steps = 4L, max_sequence_length = 512L, seed = NULL, prompt_embeds = NULL, save_file = TRUE, filename = NULL, verbose = TRUE, ...) Arguments prompt: Character.https://cornball.ai/docs/diffuser/txt2img_sd21/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/txt2img_sd21/Generate an image from a text prompt using a diffusion pipelineDescriptionThis function generates an image based on a text prompt using the Stable Diffusion model. It allows for various configurations such as model name, device, scheduler, and more. Usagetxt2img_sd21(prompt, negative_prompt = NULL, img_dim = 768, pipeline = NULL, devices = "auto", unet_dtype_str = NULL, download_models = FALSE, scheduler = "ddim", timesteps = NULL, initial_latents = NULL, num_inference_steps = 50, guidance_scale = 7.https://cornball.ai/docs/diffuser/txt2img_sdxl/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/txt2img_sdxl/Generate an image from a text prompt using SDXLDescriptionGenerate an image from a text prompt using SDXL Usagetxt2img_sdxl(prompt, negative_prompt = NULL, img_dim = 1024, pipeline = NULL, devices = "auto", memory_profile = NULL, unet_dtype_str = NULL, download_models = FALSE, scheduler = "ddim", timesteps = NULL, initial_latents = NULL, num_inference_steps = 30, guidance_scale = 7.5, seed = NULL, save_file = TRUE, filename = NULL, metadata_path = NULL, use_native_decoder = FALSE, use_native_text_encoder = FALSE, use_native_unet = FALSE, verbose = TRUE, .https://cornball.ai/docs/diffuser/txt2vid_ltx2/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/txt2vid_ltx2/Generate video (and audio) with LTX-2.3DescriptionDistilled text-to-video generation: encodes the prompt with Gemma3 + connectors, denoises joint audio/video latents over the official 8-step distilled schedule (no classifier-free guidance), decodes the video with the causal VAE and the audio with the audio VAE + BWE vocoder, and optionally muxes both into an MP4. Usagetxt2vid_ltx2(prompt, pipeline, text_encoder = NULL, tokenizer = NULL, prompt_embeds = NULL, width = 768L, height = 512L, num_frames = 121L, frame_rate = 24, sigmas = ltx23_distilled_sigmas(), guidance_scale = 1, seed = NULL, device = "cuda", dtype = "bfloat16", filename = NULL, max_sequence_length = 1024L, decode_video = TRUE, decode_audio = TRUE, two_stage = FALSE, upsampler = NULL, adain_factor = 1, tone_map_compression = 0, phase_offload = TRUE, image = NULL, condition_video = NULL, conditioning_frames = 9L, cond_noise_scale = 0, audio = NULL, verbose = TRUE) Arguments prompt: Character.https://cornball.ai/docs/diffuser/unet_native/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/unet_native/Native UNet for Stable DiffusionDescriptionNative R torch implementation of UNet2DConditionModel. Replaces TorchScript for better GPU compatibility. Usageunet_native(in_channels = 4L, out_channels = 4L, block_out_channels = c(320L, 640L, 1280L, 1280L), layers_per_block = 2L, cross_attention_dim = 1024L, attention_head_dim = 64L) Arguments in_channels: Input channels (default 4 for latent space) out_channels: Output channels (default 4) block_out_channels: Channel multipliers per block layers_per_block: Number of ResBlocks per down/up block cross_attention_dim: Context dimension from text encoder attention_head_dim: Dimension per attention head ValueAn nn_module representing the UNethttps://cornball.ai/docs/diffuser/unet_native_from_torchscript/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/unet_native_from_torchscript/Create native UNet from TorchScriptDescriptionDetects architecture and loads weights from a TorchScript UNet file. Usageunet_native_from_torchscript(torchscript_path, verbose = TRUE) Arguments torchscript_path: Path to TorchScript UNet .pt file verbose: Print loading progress ValueA native UNet module with loaded weightshttps://cornball.ai/docs/diffuser/unet_sdxl_native/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/unet_sdxl_native/Native SDXL UNetDescriptionNative R torch implementation of SDXL UNet2DConditionModel. SDXL has a different architecture from SD21: 3 down/up blocks (not 4) Variable transformer depth per block Additional conditioning via add_embedding Usageunet_sdxl_native(in_channels = 4L, out_channels = 4L, block_out_channels = c(320L, 640L, 1280L), layers_per_block = 2L, transformer_layers_per_block = c(0L, 2L, 10L), cross_attention_dim = 2048L, attention_head_dim = 64L, addition_embed_dim = 1280L, addition_time_embed_dim = 256L) Arguments in_channels: Input channels (default 4 for latent space) out_channels: Output channels (default 4) block_out_channels: Channel multipliers per block layers_per_block: Number of ResBlocks per down/up block transformer_layers_per_block: Transformer depth per block cross_attention_dim: Context dimension from text encoder attention_head_dim: Dimension per attention head addition_embed_dim: Dimension for additional embeddings addition_time_embed_dim: Dimension for time embedding projection ValueAn nn_module representing the SDXL UNethttps://cornball.ai/docs/diffuser/unet_sdxl_native_from_torchscript/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/unet_sdxl_native_from_torchscript/Create native SDXL UNet from TorchScriptDescriptionCreate native SDXL UNet from TorchScript Usageunet_sdxl_native_from_torchscript(torchscript_path, verbose = TRUE) Arguments torchscript_path: Path to TorchScript SDXL UNet .pt file verbose: Print loading progress ValueA native SDXL UNet module with loaded weightshttps://cornball.ai/docs/diffuser/unigram_tokenizer/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/unigram_tokenizer/Load a Unigram tokenizer from tokenizer.jsonDescriptionLoad a Unigram tokenizer from tokenizer.json Usageunigram_tokenizer(tokenizer_path) Arguments tokenizer_path: Path to a HuggingFace tokenizer.json with a Unigram model, or a directory containing one. ValueA unigram_tokenizer object.https://cornball.ai/docs/diffuser/vae_decoder_native/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/vae_decoder_native/Native VAE DecoderDescriptionNative R torch implementation of the SDXL VAE decoder. Replaces TorchScript decoder for better GPU compatibility. Usagevae_decoder_native(latent_channels = 4, out_channels = 3, block_channels = c(512, 512, 256, 128), norm_groups = 32) Arguments latent_channels: Number of latent channels (4 for SD/SDXL, 16 for FLUX/SD3) out_channels: Number of output channels (default 3 for RGB) block_channels: Decoder block channels (reversed encoder block_out_channels; default matches SD/SDXL and FLUX) norm_groups: Group norm groups (default 32; must divide every entry of block_channels) ValueAn nn_module representing the VAE decoderhttps://cornball.ai/docs/diffuser/vocab_size/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/vocab_size/Get vocabulary sizeDescriptionGet vocabulary size Usagevocab_size(tokenizer) Arguments tokenizer: A bpe_tokenizer object. ValueInteger vocabulary size.https://cornball.ai/docs/diffuser/vram_report/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/vram_report/Report VRAM UsageDescriptionPrints current VRAM usage using gpuctl. Usagevram_report(label = "") Arguments label: Character. Label for the report. ValueInvisibly returns a list with used and free VRAM in GB. Examplesvram_report("After model load")https://cornball.ai/docs/diffuser/write_wav/Mon, 01 Jan 0001 00:00:00 +0000https://cornball.ai/docs/diffuser/write_wav/Write a 16-bit PCM WAV fileDescriptionMinimal RIFF writer in base R. Usagewrite_wav(audio, path, sample_rate = 48000L) Arguments audio: Numeric matrix [channels, samples] in [-1, 1]. path: Output path. sample_rate: Integer. ValueInvisibly, the path.