load_bpe_tokenizer

Load tokenizer from JSON file

Description

Load tokenizer from JSON file

Usage

1load_bpe_tokenizer(vocab_path)

Arguments

  • vocab_path: Path to tokenizer.json

Value

Tokenizer object (list)