bpe_tokenizer

BPE Tokenizer

Description

Native R implementation of Byte-Pair Encoding tokenizer. Loads from HuggingFace tokenizer.json format.

Usage

bpe_tokenizer(tokenizer_path)

Arguments

  • tokenizer_path: Path to tokenizer.json or directory containing it.

Value

A bpe_tokenizer object.