Package EasyExtend :: Module eetokenizer :: Class Tokenizer

Class Tokenizer

      object --+        
               |        
eeoptions.EEShow --+    
                   |    
       BaseTokenizer --+
                       |
                      Tokenizer

Instance Methods

[hide private]

__init__(self, langlet)
x.__init__(...) initializes x; see x.__class__.__doc__ for signature

post_lex(self, scan)

default(self, pos, tok, stream)

terminate_stream(self, stream)

dot_start(self, pos, tok, stream)
dot_start: '.' | '.' A_DIGIT+ [Exponent] ['j'|'J']

INTRON(self, pos, tok, stream)

multiple_introns(self, introns, tok, stream)

single_intron(self, intron, tok, stream)

dedent_to(self, k, stream)

LEFT(self, pos, tok, stream)

RIGHT(self, pos, tok, stream)

Inherited from BaseTokenizer: add_token, set_handler, tokenize_file, tokenize_fileobj, tokenize_string

Inherited from eeoptions.EEShow: maybe_grammar_check, maybe_show_cst_after, maybe_show_cst_before, maybe_show_python, maybe_show_scan, maybe_show_token

Inherited from object: __delattr__, __getattribute__, __hash__, __new__, __reduce__, __reduce_ex__, __repr__, __setattr__, __str__

Class Methods

[hide private]

Inherited from eeoptions.EEShow: show_tokenstream

Properties

[hide private]

Inherited from object: __class__

Method Details

[hide private]

init(self, langlet)
(Constructor)

x.__init__(...) initializes x; see x.__class__.__doc__ for signature

Overrides: object.__init__: (inherited documentation)

post_lex(self, scan)

Overrides: BaseTokenizer.post_lex

dot_start(self, pos, tok, stream)

dot_start: '.' | '.' A_DIGIT+ [Exponent] ['j'|'J']

Decorators:

@post

INTRON(self, pos, tok, stream)

Decorators:

@post

LEFT(self, pos, tok, stream)

Decorators:

@post

RIGHT(self, pos, tok, stream)

Decorators:

@post

Class Tokenizer

__init__(self, langlet) (Constructor)

post_lex(self, scan)

dot_start(self, pos, tok, stream)

INTRON(self, pos, tok, stream)

LEFT(self, pos, tok, stream)

RIGHT(self, pos, tok, stream)

init(self, langlet)
(Constructor)