decode_bpe

Decode token IDs to text

Description

Decode token IDs to text

Usage

decode_bpe(tokenizer, ids, skip_special_tokens = TRUE)

Arguments

  • tokenizer: A bpe_tokenizer object.
  • ids: Integer vector or matrix of token IDs.
  • skip_special_tokens: Logical. Skip special tokens in output.

Value

Character string or vector.