Qwen3 encoder stack
Description
Defaults are the Qwen3-4B configuration used by FLUX.2 klein. The
module tree mirrors the checkpoint keys (model.embed_tokens,
model.layers.*, model.norm); the tied LM head carries
no weights of its own and is not implemented.
Usage
qwen3_encoder(vocab_size = 151936L, hidden_size = 2560L,
intermediate_size = 9728L, num_hidden_layers = 36L,
num_attention_heads = 32L, num_key_value_heads = 8L,
head_dim = 128L, rope_theta = 1e+06, rms_norm_eps = 1e-06)
Arguments
rope_theta: Numeric.rms_norm_eps: Numeric.vocab_size,hidden_size,intermediate_size,num_hidden_layers: Integers.num_attention_heads,num_key_value_heads,head_dim: Integers.
Value
Module whose forward(input_ids, attention_mask = NULL,
out_layers) returns a list of hidden-state tensors [B, S, hidden],
one per requested layer (a value of k means the state after k
layers, matching HF output.hidden_states[k]). Runs only to
max(out_layers). input_ids are 1-based.