Struct Tokenizer 

pub struct Tokenizer { /* private fields */ }

BERT-compatible tokenizer.

Combines vocabulary, pre-tokenization, and WordPiece into a single pipeline.

Implementations

impl Tokenizer

pub fn new<P: AsRef<Path>>(vocab_path: P, do_lower_case: bool) -> Result<Self>

Create a new tokenizer from a vocabulary file.

Arguments

• vocab_path: Path to vocab.txt
• do_lower_case: Whether to lowercase input text

pub fn from_vocab(vocab: Vocab, do_lower_case: bool) -> Self

Create a tokenizer from a pre-loaded vocabulary.

pub fn encode(&self, text: &str, max_length: usize) -> Result<Encoding>

Tokenize and encode a single text.

Pipeline: text -> pre-tokenize -> WordPiece -> add [CLS] and [SEP] -> truncate -> pad

Arguments

• text: Input text
• max_length: Maximum sequence length (including special tokens)

pub fn encode_batch( &self, texts: &[&str], max_length: usize, ) -> Result<Vec<Encoding>>

Encode a batch of texts in parallel.

Uses rayon for parallel tokenization. All encodings are padded to max_length.

pub fn vocab_size(&self) -> usize

Get the vocabulary size.

pub fn vocab(&self) -> &Vocab

Access the vocabulary.

Trait Implementations

impl Clone for Tokenizer

fn clone(&self) -> Tokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Tokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.