The tokenizer stage type.
The context passed to the tokenizer.
Reads tokens from the input in a non-streaming fashion.
The input string to tokenize.
The callbacks that are invoked when tokens are read from the string.
The context that should be passed to readers and stage providers.
The mutable state used by the tokenizer.
The result state of the tokenizer.
Reads remaining tokens from the TokenizerState.chunk.
The callbacks that are invoked when tokens are read from the string.
The mutable state returned by the previous Tokenizer.write call.
The context that should be passed to readers and stage providers.
The result state of the tokenizer.
Reads tokens from the chunk in a streaming fashion. During streaming, TokenHandler is triggered only with confirmed tokens. Token is confirmed if the consequent token was successfully read.
let state = tokenizer.write('foo', handler);
tokenizer.write('bar', handler, state);
tokenizer.end(handler, state);
The input chunk to tokenize.
The callbacks that are invoked when tokens are read from the string.
The mutable state returned by the previous Tokenizer.write call.
The context that should be passed to readers and stage providers.
The result state of the tokenizer.
Generated using TypeDoc
The type of the token emitted by the tokenizer.