Easy Extend



                             

       Coverage







            Author: Kay Schluehr

            Date: 04.05.2006
            Version: 1/2



1  Description

The coverage.py module is dedicated to code coverage tasks. It is an example of the simplicity of EasyExtend: the derived Transformer has to implement only one method. The basic idea of coverage.py is to compile a function call into each suite
( i.e. code block ) of the analyzed module. If the the suite is entered the function will be called and the call will be notified.
For each suite a Sensor object is created. You can imagine a bunch of sensors spreaded all over the code-blocks and signaling their presence to an external Monitor object. The Monitor holds a reference to each sensor and draws an image of their activation after execution of the analyzed module. The ratio between all sensors distributed over the code against those that were activated is a measure of code coverage. It can be used as one factor of test-quality measurement.

Since we change behaviour of an existing Python module it is reasonable not to create a *.pyc file and overwrite existing compilations but using an own file extension. The coverage tool uses the pycov extension. The resulting *.pycov-files are copied into the directory Fibers/coverage/temp.

2. Usage


usage: python fiber.py [option] ... file
Options:
-o   : redirect output in report output file instead of stdout
-p   : pattern used to describe names of modules imported using the
       coverage importer. pattern is a quoted regular expression.   
-d   : deactivate default pattern ( see explanation below ) 
file : script file to start with

The -p option defines a selection among modules using a regular expression. As default pattern the expression '(?P<test>test\_(?P<mod_name>\w+))' is used. This pattern extracts the group of strings ("test_module", "module") from the string "test_module" where "module" is an arbitary alphanumeric string. It can be deactivated using the -d option.

Examples:
 (1) python coverage.py test_all.py
 (2) python coverage.py -d "eetransformer" language.py

In (1) the module test_all.py will be covered but also each module test_xxx.py that is imported during execution of test_all.py as well as each module xxx.py.

In (2) we deactivate the default import rules of (1) and cover language.py. The language module imports the eetransformer module that will be covered as well.


3 Output and Interpretation

The following example shows the result of executing the command  python fiber.py test_demo.py where fiber.py belongs to the coverage fiber. Small s symbols are drawn right from the line numbers. They mark places where sensors are inserted. Each new codeblock is marked by a sensor. One exception is made with blocks

            if __name__ == '__main__':
         BLOCK

which are defined in modules imported by the main module ( here test_demo.py ) They will never get called and the ratio between activated/deactivated sensors does not provide valuable information. An arrow "==>" will be drawn right from a sensor mark s if the sensor keeps silent and does not respond. Each measurement on a module is terminated by a status message. If more than one module is involved a summary is provided.


     _____
    /  __ \
    | /  \/ _____   _____ _ __ __ _  __ _  ___
    | |    / _ \ \ / / _ \ '__/ _` |/ _` |/ _ \
    | \__/\ (_) \ V /  __/ | | (_| | (_| |  __/
     \____/\___/ \_/ \___|_|  \__,_|\__, |\___|
                                     __/ |
                                    |___/


----------------------------------------------------------.
 Coverage : c:\Python24\Fibers\pytools\test\test_demo.py  |
---------------------------------------------------------------------------------------.
0
1         from demo import Demo
2        
3        
4 s       if __name__ == '__main__':
5             Demo(0,-1)
6             d = Demo(1,0)
7 s  ==>      if 0:                     # this sensor does nothing for obvious reasons
8                 d.f()

---------------------------------------------------------.
Status:                                                  |
  2 sensors created                                      |
  1 sensors did not respond                              |
---------------------------------------------------------------------------------------'

-----------------------------------------------------.
 Coverage : c:\Python24\Fibers\pytools\test\demo.py  |
---------------------------------------------------------------------------------------.
00
01 s       class Demo:
02 s           def __init__(self,a,b):

03 s               if a:
04                     print "Demo a +:",a
05 s               else:
06                     print "Demo a -:",a
07 s               if b<0:
08 s                   while 1:
09 s  ==>                  if b<-1:
10                             return
11                         b+=1
12 s                       if b==0:
13                             break
14                             a-5        # ...does not detect all unreachables
15 s  ==>                  b-2            # may not be reached
16 s               else:
17                     print "Demo b:",b
18        
19 s  ==>      def f(self):
20                 return 0
21        
22        
23         if __name__ == '__main__':
24             Demo(0,-1)
25             d = Demo(1,0)
26             if 0:
27                 d.f()

----------------------------------------------------.
Status:                                             |
  10 sensors created                                |
  3 sensors did not respond                         |
---------------------------------------------------------------------------------------'

========================================================================================
Summary:
  12 sensors created
  4 sensors did not respond

========================================================================================



4  Modules that can't

Some modules do not respond to the coverage specific Importer because they were already imported by at startup. Those modules are listed here. You can do a workaround simply by copying those modules to other location and call coverage from there.

5  Prospects

With statement coverage we do not cover some more subtle conditional expressions. Let's modify the __init__ method of the Demo class in the following way:

07               if b<0 or a+b>7:
08                   x = 0
09                   y = 9
10               else:
11                   print b

With statement coverage we do not know which branch of the condition is executed. If b<0 is always true for any test the alternative branch defined by a+b>7 will never be checked. What we have to do is to replace

if b<0 or a+b>7:
    SUITE
else:
   
print b


==>
if b<0:
    SUITE
elif a+b>7:          
    SUITE
else:
    print b

A corresonding conjunction  b<0 and a+b>7 could be rephrased using a nested if-statement:

if b<0 and a+b>7:
    SUITE
else:
   
print b


==>
if b<0:
    if a+b>7:
        SUITE
    else:       
        print b
else:
    print b

Those transformations would not necessarily require an inplace expression -> statement transformation. Instead we may encapsulate those statements in a predicate i.e. a function returning a bool typed value and substitute the conditional expression by a call of the predicate accordingly. But we will likely not draw the sensor image of those helper functions but remap the sensor activities onto our original expressions.