flux_attention

Packages

chatterbox
- chatterbox Reference
cornfab
- cornfab Reference
diffuseR
earshot
- run_app
fyi
- fyi Reference
llamaR
- llamaR Reference
rapt
- rapt Reference
rformat
- rformat Reference
saber
- saber Reference
stt.api
tinypkgr
- tinypkgr Reference
tinyrox
- tinyrox Reference
tts.api
- tts.api Reference
whisper

FLUX joint attention

Description

Multi-head attention with per-head RMS q/k norms and rotary position embeddings. With added_kv = TRUE (double-stream blocks) the text stream gets its own q/k/v projections and both streams attend jointly (text tokens first); the outputs are split back and projected per stream. With pre_only = TRUE (single-stream blocks) there is no output projection. Reference: diffusers FluxAttention + FluxAttnProcessor.

Usage

flux_attention(query_dim, heads, dim_head, added_kv = FALSE, pre_only = FALSE,
               eps = 1e-06, bias = TRUE)

Arguments

query_dim: Integer. Model dimension.
heads: Integer. Number of attention heads.
dim_head: Integer. Per-head dimension.
added_kv: Logical. Add text-stream projections (double blocks).
pre_only: Logical. Skip the output projection (single blocks).
eps: Numeric. RMS norm epsilon.
bias: Logical. Bias on the linear projections (TRUE for FLUX.1, FALSE for FLUX.2).