Interface Tokenizer

A tokenizer that reads tokens from text and returns them by invoking a callback.

See

createTokenizer

interface Tokenizer {
tokenize(text: string, callback: TokenCallback): void;
}

Index

Methods

Methods

tokenize

tokenize(text: string, callback: TokenCallback): void
Reads tokens from text and returns them by invoking a callback.
Parameters
- text: string
  The text string to read tokens from.
- callback: TokenCallback
  The callback that is invoked when a token is read.
Returns void