Package EasyExtend :: Module eetokenizer :: Class BaseTokenizer

Class BaseTokenizer

      object --+    
               |    
eeoptions.EEShow --+
                   |
                  BaseTokenizer

Known Subclasses:

Tokenizer

Abstract base class of Tokenizer classes.

Instance Methods

[hide private]

__init__(self, langlet)
x.__init__(...) initializes x; see x.__class__.__doc__ for signature

post_lex(self, scan)

add_token(self, tok, stream)

set_handler(self)

tokenize_file(*args, **kwargs)

tokenize_fileobj(*args, **kwargs)

tokenize_string(*args, **kwargs)

Inherited from eeoptions.EEShow: maybe_grammar_check, maybe_show_cst_after, maybe_show_cst_before, maybe_show_python, maybe_show_scan, maybe_show_token

Inherited from object: __delattr__, __getattribute__, __hash__, __new__, __reduce__, __reduce_ex__, __repr__, __setattr__, __str__

Class Methods

[hide private]

Inherited from eeoptions.EEShow: show_tokenstream

Properties

[hide private]

Inherited from object: __class__

Method Details

[hide private]

init(self, langlet)
(Constructor)

x.__init__(...) initializes x; see x.__class__.__doc__ for signature

Overrides: object.__init__: (inherited documentation)

tokenize_file(*args, **kwargs)

Decorators:

@EasyExtend.util.psyco_optimized

tokenize_fileobj(*args, **kwargs)

Decorators:

@EasyExtend.util.psyco_optimized

tokenize_string(*args, **kwargs)

Decorators:

@EasyExtend.util.psyco_optimized

Class BaseTokenizer

__init__(self, langlet) (Constructor)

tokenize_file(*args, **kwargs)

tokenize_fileobj(*args, **kwargs)

tokenize_string(*args, **kwargs)

init(self, langlet)
(Constructor)