encode_unigram

Packages

chatterbox
- chatterbox Reference
cornfab
- cornfab Reference
diffuseR
earshot
- run_app
fyi
- fyi Reference
llamaR
- llamaR Reference
rapt
- rapt Reference
rformat
- rformat Reference
saber
- saber Reference
stt.api
tinypkgr
- tinypkgr Reference
tinyrox
- tinyrox Reference
tts.api
- tts.api Reference
whisper

Encode text with a Unigram tokenizer

Description

Normalizes (strip-right, multi-space collapse, control whitespace to space), applies the Metaspace pre-tokenizer, segments each pre-token by Viterbi over the Unigram scores, fuses consecutive unknowns, and appends EOS. T5 semantics: right padding with <pad> (id 0), truncation to max_length - 1 before the EOS.

Usage

encode_unigram(tokenizer, texts, max_length = 256L, add_eos = TRUE, pad = TRUE)

Arguments

tokenizer: A unigram_tokenizer.
texts: Character vector of prompts.
max_length: Integer. Fixed sequence length (NULL for no truncation/padding).
add_eos: Logical. Append the EOS token.
pad: Logical. Right-pad to max_length.

Value

List with input_ids and attention_mask, each an integer matrix [length(texts), max_length] (or ragged lists when max_length is NULL). Ids are 0-based (HuggingFace convention); add 1 for R torch embedding lookups.