Package EasyExtend :: Module eetokenizer :: Class StdTokenizer
[hide private]
[frames] | no frames]

Class StdTokenizer



object --+
         |
        StdTokenizer

Instance Methods [hide private]
 
__init__(self, token, typ=None, use_comment=True)
x.__init__(...) initializes x; see x.__class__.__doc__ for signature
 
tokenize_file(self, filename)
 
tokenize_string(self, inString)
 
tokenize_input(self, readline)
 
next_token(self)
 
tokenized(self)
 
generate_tokens(self, readline)
The generate_tokens() generator requires one argment, readline, which must be a callable object which provides the same interface as the readline() method of built-in file objects.

Inherited from object: __delattr__, __getattribute__, __hash__, __new__, __reduce__, __reduce_ex__, __repr__, __setattr__, __str__

Properties [hide private]

Inherited from object: __class__

Method Details [hide private]

__init__(self, token, typ=None, use_comment=True)
(Constructor)

 
x.__init__(...) initializes x; see x.__class__.__doc__ for signature
Overrides: object.__init__
(inherited documentation)

generate_tokens(self, readline)

 

The generate_tokens() generator requires one argment, readline, which must be a callable object which provides the same interface as the readline() method of built-in file objects. Each call to the function should return one line of input as a string.

The generator produces 5-tuples with these members: the token type; the token string; a 2-tuple (srow, scol) of ints specifying the row and column where the token begins in the source; a 2-tuple (erow, ecol) of ints specifying the row and column where the token ends in the source; and the line on which the token was found. The line passed is the logical line; continuation lines are included.