vocab_size

Get vocabulary size

Description

Get vocabulary size

Usage

vocab_size(tokenizer)

Arguments

  • tokenizer: A bpe_tokenizer object.

Value

Integer vocabulary size.