Interface Tokenizer

A tokenizer that reads tokens from text and returns them by invoking a callback.

See

createTokenizer

interface Tokenizer {
tokenizeDocument(text: string, callback: TokenCallback): void;
tokenizeFragment(text: string, callback: TokenCallback): void;
}

Index

Methods

tokenizeDocument tokenizeFragment

Methods

tokenizeDocument

tokenizeDocument(text: string, callback: TokenCallback): void
Reads tokens from text and returns them by invoking a callback.
Parameters
- text: string
  The text string to read tokens from.
- callback: TokenCallback
  The callback that is invoked when a token is read.
Returns void

tokenizeFragment

tokenizeFragment(text: string, callback: TokenCallback): void
Reads tokens from text and returns them by invoking a callback.
Parameters
- text: string
  The text string to read tokens from.
- callback: TokenCallback
  The callback that is invoked when a token is read.
Returns void