gemma3_tokenizer

Gemma3 Tokenizer

Description

Native R tokenizer for Gemma3 using BPE. Loads from HuggingFace tokenizer.json format.

Usage

gemma3_tokenizer(tokenizer_path)

Arguments

  • tokenizer_path: Character. Path to tokenizer directory or tokenizer.json file.

Value

A gemma3_tokenizer object (extends bpe_tokenizer).