The state that is used by the Tokenizer to track tokenization progress.
The tokenizer stage type.
The chunk that is being processed.
The offset of the chunk in the input stream.
The offset in the chunk from which the tokenization should proceed.
The current tokenizer stage.
Generated using TypeDoc
The state that is used by the Tokenizer to track tokenization progress.