unigram_tokenizer

Load a Unigram tokenizer from tokenizer.json

Description

Load a Unigram tokenizer from tokenizer.json

Usage

unigram_tokenizer(tokenizer_path)

Arguments

  • tokenizer_path: Path to a HuggingFace tokenizer.json with a Unigram model, or a directory containing one.

Value

A unigram_tokenizer object.