Function createTokenizer

createTokenizer(options?: TokenizerOptions): Tokenizer
Reads tokens from text and returns them by invoking a callback.

Tokens are guaranteed to be returned in correct order. Missing tokens are inserted to restore the correct order if needed.
Parameters
- options: TokenizerOptions = {}
  Tokenizer options.
Returns Tokenizer
Example
```
import { createTokenizer, htmlTokenizerOptions } from 'tag-soup';

const tokenizer = createTokenizer(htmlTokenizerOptions);

tokenizer.tokenize(
  'Hello, <b>Bob</b>!',
  (token, startIndex, endIndex) => {
    // Handle token here
  },
);
```