qwen_bpe_tokenizer

Load a Qwen2 byte-level BPE tokenizer

Description

Load a Qwen2 byte-level BPE tokenizer

Usage

qwen_bpe_tokenizer(tokenizer_path)

Arguments

  • tokenizer_path: Path to a tokenizer.json (or a directory containing one).

Value

A qwen_tokenizer object.