text_to_tokens

Convert text to token tensor

Description

Convert text to token tensor

Usage

1text_to_tokens(tokenizer, text, normalize = TRUE, device = "cpu")

Arguments

  • tokenizer: Tokenizer object
  • text: Input text
  • normalize: Whether to apply punctuation normalization
  • device: Target device

Value

Token tensor (1, seq_len)