encode_qwen

Packages

chatterbox
- chatterbox Reference
cornfab
- cornfab Reference
diffuseR
earshot
- run_app
fyi
- fyi Reference
llamaR
- llamaR Reference
rapt
- rapt Reference
rformat
- rformat Reference
saber
- saber Reference
stt.api
tinypkgr
- tinypkgr Reference
tinyrox
- tinyrox Reference
tts.api
- tts.api Reference
whisper

Encode prompts with the Qwen tokenizer

Description

With chat_template = TRUE (the FLUX.2 klein pipeline behavior) each prompt is wrapped as a single user turn with the generation prompt and a disabled thinking block, matching apply_chat_template(..., add_generation_prompt = TRUE, enable_thinking = FALSE). Right-pads with <|endoftext|>.

Usage

encode_qwen(tokenizer, texts, max_length = 512L, chat_template = TRUE)

Arguments

tokenizer: A qwen_bpe_tokenizer.
texts: Character vector of prompts.
max_length: Integer. Fixed sequence length (klein: 512). NULL for no truncation/padding.
chat_template: Logical. Wrap in the Qwen3 chat template.

Value

List with input_ids and attention_mask integer matrices [length(texts), max_length] (ragged lists when max_length is NULL). Ids are 0-based.