Tokenizer

The Tokenizer generates a stream of double-linked Token objects.

class pyTokenizer.Tokenizer

Bases: object

class TokenKind(value)

Bases: enum.Enum

Enumeration of token kinds.