Easy Extend


                   



                                            Author: Kay Schluehr
                                            Date: 2008-09-04
                                            Version: 1.2.4
Download
API documentation
Bytelets


 
                                  

1. Introduction 

A short version of an introduction to P4D can be summarized by a small formula:

P4D = E4X without E and X but P and P4D trees

A slightly more extended version goes like this:

P4D ( = Python For Data ) is a full formed Python dialect that enables writing tree shaped data directly in source code. The main sources of inspiration for P4D were E4X which handles access to XML quite gracefully on language level and provides XML literals and SliP which preserves the structural properties of XML while being more apt to the concise Python syntax. As a result P4D keeps much of the original E4X design idea ( and also the syntax ) but drops XML literals and replaces them by more pythonic data structures. The transitions between P4D and XML are smooth. So you can use P4D to convert an XML document into a P4D datastructure edit and transform this structure and output an XML document again.

P4D is implemented as an EasyExtend langlet which means among other things that P4D files compile to Python bytecode. So accessing P4D modules from Python is feasible ( it is actually feasible on source level as well but this will be explained later in this doc ) while accessing Python from P4D is obviously possible.

2. Using P4D

2.1 Install and run

P4D requires a Python 2.5 installation. Other versions of Python won't work with P4D.

You can download P4D here. It is a Python package that is installed on the command line using the canonical

               python setup.py install


               <MyPythonDir>/lib/site-packages/EasyExtend/langlets/p4d

P4D can be started as a Python script placed in the <MyPythonDir>/Scripts folder of you cd to the above directory and type

               python run_p4d.py

A P4D specific console shall be opened immediately:


C:\lang\Python25\Lib\site-packages\EasyExtend\langlets\p4d>python run_p4d.py
__________________________________________________________________________________

 p4d

 On Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)]
__________________________________________________________________________________

p4d>

2.2 Philosophers Football

A P4D statement is a tree of elements containing other P4D statements or elements. Following example encodes some data of Monty Pythons philosophers football game.

game = monty-python-football:
    location: "München"
    team(country="Deutschland"):
        player(no=1):"Leibniz"
        player(no=2):"I.Kant"
        player(no=3):"Hegel"
        player(no=4):"Schopenhauer"
        player(no=5):"Schelling"
        player(no=6):"Beckenbauer"     # Beckenbauer obviously a bit of a surprise there
        player(no=7):"Jaspers"
        player(no=8):"Schlegel"
        player(no=9):"Wittgenstein"   
        player(no=10):"Nietzsche"
        player(no=11):"Heidegger"

    team(country="Griechenland"):
        player(no=1):"Plato"
        player(no=2):"Epiktet"
        player(no=3):"Aristoteles"
        player(no=4):"Sophokles"
        player(no=5):"Empedokles"
        player(no=6):"Plotin"
        player(no=7):"Epikur"
        player(no=8):"Heraklit"
        player(no=9):"Demokrit"
        player(no=10):"Sokrates"
        player(no=11):"Archimedes"

    referees:
        main:"K'ung fu-tsze"
        assistants:
            linesman(no=1): "Saint Thomas Aquinas"
            linesman(no=2): "Saint Augustine"
    play:
        half-time(no = 1):
            team(country="Germany" goals = 0)
            team(country="Greece" goals = 0)
        half-time(no = 2):
            team(country="Germany" goals = 0):
                substitutes:
                   subst(p4d:in="Marx" p4d:out="Wittgenstein")                 
                matchwinner: "False"
            team(country="Greece", goals = 2):
                substitutes
                goal-scorer(goal=1): "Sokrates"
                goal-scorer(goal=2): "Sokrates"
                matchwinner: "True"

2.3 Basic P4D element access

Now we show how to access elements using P4Ds syntax. The game object which is an ordinary Python object serves as an entry point.

p4d> game
P4D<monty-python-football>

Notice here one particular detail. In P4D one can write hyphenated names: NAME ('-' NAME)*. However one can write them within P4D elements only. In other contexts the same syntactical form will be interpreted as an arithmetic difference.

p4d> game.location
[P4D<location>]

We can access elements using the dot-operator. The return value is a P4DList object. A P4DList is derived from Pythons list type. It provides some additional methods which it shares with basic P4D objects. So one can access text directly for instance:

p4d> location = game.location
p4d> location.text()

This however works only if the P4DList contains exactly one element. Otherwise an AttributeError is raised as in the following example.

p4d> game.team.player
Traceback
...
AttributeError...

Notice that the attributes in the attribute list as in

team(country="Greece" goals = 2)

don't have to be separated by comma. Commas are optional though and you can write the element in Python style:

team(country="Greece", goals = 2)

2.3.1 Attribute access using the @ operator

We can access P4D object attributes either using the attributes() method or we use the @-operator which is a syntactical shortcut.

p4d> game.team[0].@country
"Deutschland"
p4d> winning_team_goals = int(game.play.child("final-score").team[1].@goals)

In the latter case we have to use the child() method of the P4D element to access the final-score element. This is necessary because we do not use final-score within the context of an P4D object definition.

Another use of the @-operator is to select all attributes of an element. This is done by the combination of @ with the wildcard symbol *.

p4d> game.play.child("half-time").team[1].@*
{"country":"Greece", "goals": "2"}

2.4 Declarative filters

Subelement filters are an E4X feature also available in P4D. A subelement filter is basically a statement about element properties used to filter subelements of the element to which the filter is applied.

X.(filter-declaration)

    -->

for item in X.children():
    filter-function(item)

The main purpose of filters is navigation. Intuitively speaking one tries to reduce a list of child nodes to a list of exactly 1 element. On this 1 element list one can apply another filter and so on.

Example 1:

a = a:
    b(x = 0 y = 1): "abc"
    b(x = 0 y = 2): "a"
    c(x = 1 y = 2): "abc"
    c(x = 0 y = 2):
        d: "def"
        d: "efg"

With on element a we can apply several different filters:
p4d> a.(@x == 0)              # -- elements X in a with X.@x == '0'
[P4D<b>, P4D<b>, P4D<c>]  

p4d> a.b.(.text() == "abc")   # -- elements b in a.b with b.text() == "abc"
[P4D<b>]                  

p4d> a.c.(d == "abc")         # -- elements c in a.c with subelement d and d.text() == "efg"
[P4D<c>]                  

p4d> a.(@x == 1 or @y == 1)   # -- elements X in a with X.@x == 1 or X.@y == 1
[P4D<b>, P4D<c>]          

p4d> a.b.(.children() == [])  # -- elements b in a.b with b.children() == []
[P4D<b>, P4D<b>]                  


Example 2:

Applying filters to the philosophers football game:

p4d> game.(@country == "Deutschland").player.(@no == 3).text()
'Hegel'

p4d> game.play.children("half-time")[1].team.(matchwinner == "True").@country
'Greece'

As you see the "half-time" element must be selected using the children() object method because "half-time" is not a Python name.

2.4.1  Using wildcards in filters

In the second example above we mentioned the subelement tag explicitely in the filter declaration:

p4d> a.(b == "abc")         # lambda this: this.text( ) == "abc" and this.tag == 'b'
[P4D<b>]                  

Using a single underscore _ instead of b leads to a different translation where the clause this.tag == '_' will be just omitted:

p4d> a.(_ == "abc")               # lambda this: this.text( ) == "abc"
[P4D<b>, P4D<c>]                  

p4d> a.(_.text()[0] == "a")       # lambda this: this.text( )[ 0 ] == "a"
[P4D<b>, P4D<b>, P4D<c>]                  


2.4.2  Pitfalls

Filters operate just skindeep i.e. on the level of child elements of a current node. You see this when trying to use a filter declaration to filter among nodes on a deeper level of hierarchy e.g.

a.(c.d.text() == "def")         # lambda this: this.d.text() == "def" and this.tag == 'c'

Looks quite o.k. but isn't. Actually this query fails with following exception:

                    P4DAccessError: Cannot access P4D text on P4DList with length!=1.

The message means that c.d is a list of nodes of length != 1 ( two nodes in our example ) and P4D only treats P4DList objects of length = 1 like P4DNode objects!

2.5  The elm keyword

So far we have assigned P4D elements to regular Python names using the assignment form

        py_name '=' p4d-element

However each P4D element starts itself with a name so we have

        py_name '=' p4d_name ...

Whenever a p4d_name is also a valid py_name valid we can reduce boilerplate and just write

        elm p4d_name ...



# explicit assignment:

philosopher = philosopher:
    name: "Gottfried Wilhelm Leibniz"
    birth:
        date: "1646-07-1"
        city: "Leipzig"
    death:
        date: "1716-11-14"
        city: "Hannover"

# implicit assignment:

elm philosopher:
    name: "Georg Wilhelm Friedrich Hegel"
    birth:
        date: "1770-08-27"
        city: "Stuttgart"
    death:
        date: "1831-11-14"
        city: "Berlin"

Definitions using elm will always be expanded into the explicit assignment form.

Implicit assignment cannot be used with hyphened names such as monty-python-football or prefixed names like p4d:if. In both cases a SyntaxError is raised and an explicit assignment using a valid Python name is recommended.

Why elm and not the more intuitively appealing elem? An elem keyword would be like var or val. Any second Python program has defined one and would be invalid.

2.6 namespaces

XML and P4D elements provide context and scope. Elements shall therefore be sufficient for most practical disambiguation purposes. However XML moves beyond this and provides  XML namespaces which lets you mix elements of different XML based domain specific languages freely. Although P4D supported XML namespaces initially just for compliency reasons

A namespace consists of two parts: a prefix and an URI ( or URN ). The URI shall serve as a strong name providing real means of contextual disambiguation. The prefix is kind of a variable that binds to an URI. When in use namespace prefixes are separated by colon from a so called local name. Prefxixes are the visible clue you have most likely noticed in XML elements like this

             <xsl:stylesheet version "1.0" ...>

Library authors who take namespaces really serious substitute prefixes by their corresponding URIs ( URNs ) entirely when XML documents are getting dumped. The elementtree package of the Python standard library is an example of this practice.

P4D considers namespaces prefixes basically as an additional syntactical detail. One can retrieve namespace information from elements though. This doesn't mean much more than a variable lookup.

a = a:
    b:c(xmlns:b = "http://bla.bla.bla")

p4d> a.b::c
P4D<b:c>
p4d> a.b::c.namespace()
<EasyExtend.langlets.p4d.p4dbase.P4DNamespace object at 0x01487C10>
p4d> b_ns = a.b::c.namespace()
p4d> b_ns.prefix
'b'
p4d> b_ns.uri
'http://bla.bla.bla'

The doublecolon operator :: is used to express full element access of a prefixed name. Alternatively one can use the child method with the full tag name:

p4d> a.child("b:c")
P4D<b:c>

If an element is not prefixed the invocation of namespace() on it returns a P4DNamespace object with empty content.

p4d> a.namespace().uri          # None
p4d> a.namespace().prefix       # None

If an element is not prefixed the invocation of namespace() on it returns a P4DNamespace object with empty content.

2.6.1  The p4d namespace

There is one default namespace URL in P4D which is http://fiber-space.de/p4dns. The corresponding prefix is p4d. The p4d namespace is introduced for a very particular reason: it wraps language keywords. Just look at the following example:

if:
    while(x=1):
        pass

P4D would fail to parse this expression. Keywords are treated special by the parser. However you might prefix these element tags using p4d:

p4d:if:
    p4d:while(x=1):
        p4d:pass

In this case the parser reads p4d:NAME as a single token.

How to navigate through this structure?

For namespace information retrieval you can simply access elements using a double colon:

a = p4d:if:
    p4d:while(x=1):
        p4d:pass

p4d> a.p4d::while.@x
'1'

Alternatively one can use the child(..) accessor method

p4d = p4d:if:
    p4d:while(x=1):
        p4d:pass

p4d> a.child("p4d:while").@x
'1'

2.7 Python expressions in P4D elements

So far we considered just static content. Content of P4D attributes or elements had to be strings or numbers. If the content was a number it was automatically converted into a string.

Now we want to place arbitrary Python expressions within P4D elements or attributes. The value of the expression shall be either a string or a P4DNode object. Embedding however might cause trouble. Just look at the following example:

p4d> x = 1
p4d> y:x
P4D<y:x>

Here y is understood as a namespace prefix and the assignment of x is ignored. In order to let the value of x be the content of y we write:

p4d> x = 1
p4d> y:&x
P4D<y>

The ampersand operator can prefix an arbitrary Python expression. So the following more compex expression shall work

p4d> x = 0
p4d> y:&(x if x>0 else -x)
P4D<y>

The expression is evaluated when the root P4D element is created. I considered lazy evaluation ( call by need ) as well. This enables dataflow and partial constructions. However it is not available in this release.

Attribute values actually wouldn't require the ampersand operator because there is nothing to disambiguate. However I abandoned modality in favor for simplicity: evaluation in P4D elements always only takes place when they are prefixed by an ampersand operator.

Notice that omitting the ampersand operator leads to failure in case of general expressions:

p4d> x = 0
p4d> y:(x if x>0 else -x)              # wrong - expr needs to be prefixed with '&'
-> ParseError
p4d> y(a = x if x>0 else -x): 0        # wrong - expr needs to be prefixed with '&'
-> SyntaxError
p4d> y(a = x): 0                       # wrong - expr needs to be prefixed with '&'
-> SyntaxError
p4d> elm y(a = "x if x>0 else -x"): 0  # o.k. strings are permitted
p4d> elm y(a = 42): 0                  # o.k. numbers are permitted
p4d> elm y(a = &x if x>0 else -x): 0   # o.k. ampersand prefix used
p4d> y.@a
'0'

2.7.1  Object lists as element content

This feature enables a more compact notion of lists of elements. It is not directly representable in XML and might be the single biggest departure.

Suppose you have a list L = [1,2,3,4] and for each entry in L you want to create an Item element with the entry as content.
The usual way to do this is to write:
# variant 1

elm X:
    Item: 1
    Item: 2
    Item: 3
    Item: 4

# variant 2 using the ampersand operator

elm X:
    Item: &L[0]
    Item: &L[1]
    Item: &L[2]
    Item: &L[3]

In P4D this boilerplate can be eliminated entirely and you can simply write

# variant 1

elm X:
    Item: [1,2,3,4]

# variant 2 using the ampersand operator

elm X:
    Item: &L

Those lists will be expanded and for each entry an Item object is created internally:

p4d> X.Item
[P4D<Item>, P4D<Item>, P4D<Item>, P4D<Item>]

Attributes will be replicated and P4D nodes become embedded:

elm a:
   b(x = 0): [1,2,3]

elm Y:
    Item: &a.b

p4d> Y.Item
[P4D<Item>, P4D<Item>, P4D<Item>]
p4d> Y.Item[1].b.text()
'2'
p4d> Y.Item[1].@x
'0'

2.7.2  Subelement rules

Suppose you have a list L = ["blu", x, 2, y] where x and y are P4D elements and you make following insertion:

elm C:
    D:
        &L

For "blu" and 2 are new elements of tag D created. They are used as content. This is not true for x and y because they serve as subelements. So there will always only be one D for which P4D elements in a list are used as subelements.

elm x:
    a
elm y:
    b
elm C:
    D:
        &["blu", x, 2, y]

Now display C:

p4d> print C.p4dstr()
C:
    D: "blu"
    D:
        y:
            b
        x:
            a
    D: "2"

2.7.3  Non P4D element content

Suppose you have the following statement:
x = "Some Text"
y = False

elm A:
    B:
        &x
        &y

What can be used as content here? Shall an exception be raised because x and y can be both content? Shall only y be content because it is a string? Shall y be used because it is the last definition in the row or x because it is the first one?

I decided to use both and defined a new type which is called P4DContentList. P4DContentList is derived from list and you usually don't have to access it explicitely except for the special case of having more than one objects that can be content. The implementation of P4DContentList is rather simple:

class P4DContentList(list):
    def __str__(self):
        return ''.join(str(item) for item in self)

The P4DContentList object will be printed as an ordinary string not as a list. Notice that the sequence of non P4D objects is not preserved.

2.7.4  None is not content

The None object will not be inserted as content. Writing a P4D element in this style

y = None

elm A:
    &None
    &y

yields an element A without children and without content. This can be used to optionally insert P4D elements in A:

y = (compute_element(args) if cond else None)

elm A:
    &y


2.8 P4D comments

Using single or triple quoted strings are used to denotate P4D content within P4D elements. They can't be used to create XML like comments which are significant. Signifcant means that the comments are usually not dropped by the parser but content is sourced out into comments that doesn't fit well into the document structure otherwise.

The comment style of P4D is {*  ... *}.

elm foo:
    {* This is a
       comment *}

    """... and this is
       content.
    """

You can access a comment by either using the comment() object method or the child(tag) object method with tag = '*'. Notice that a P4D comment is a regular P4D element! A P4D comment is much like a docstring in Python which is just a usual string.

2.9 Element assignments and deletions ( new in P4D 1.2 )

2.9.1 Assignments

You can modify content or replacing elements by assigning values to them:
elm a:
    b: 1

elm b:
    c: 2

p4d> a.b.content()
'1'
p4d> a.b = 2
p4d> a.b.content()
'2'
p4d> a.b = b
p4d> a.b.c.content()
'2'

Notice that you can't assign a P4D element A to a P4D element B. When you really want to change a tag you have to rebind the tag attribute:

p4d> a.b[0].tag = 'x'
p4d> a.x
[P4D<x>]

Further limitations: you can't assign to b if b is a list of nodes with length > 1.

2.9.2 Deletions

Subelement deletions are as easy as assignments

p4d> elm a:
....    b
....    c
....
p4d> a.children()
[P4D<b>, P4D<c>]
p4d> del a['c']                # remove 'key'
p4d> a.children()
[P4D<b>]

A KeyError is raised when no child element with tag = 'c' is available.

3. Support for XML languages

Support for XML languages is an obvious goal of P4D. The transition between P4D and XML shall be smooth since internally the same datastructure is used to hold elements. In section 5.1 we start with some conversion groundwork. Section 5.2. examines Adobe Flex and MXML as one particular use case.

3.1. xmlstr and p4dstr

p4d> elm philosopher:
....    name: "Hegel"
....    main-work: "Phaenomenologie des Geistes"
....    first-edition: 1807
....
p4d> hegel_data = philosopher.xmlstr()
p4d> print hegel_data
<philosopher>
    <name>Hegel</name>
    <main-work>Phaenomenologie des Geistes</main-work>
    <first-edition>1807</first-edition>
</philosopher>
p4d>

You can reconstruct the original P4D element with the same ease:

p4d> philosopher = P4D.from_xml(hegel_data)
p4d> print philosopher.p4dstr()
philosopher:
    name: "Hegel"
    main-work: "Phaenomenologie des Geistes"
    first-edition: "1807"

You can print additional data that are optional e.g. a standard XML declaration:

p4d> print philosopher.xmlstr(xml_declaration = True)
<?xml version="1.0" encoding="UTF-8" ?>
<philosopher>
    <name>Hegel</name>
    <main-work>Phaenomenologie des Geistes</main-work>
    <first-edition>1807</first-edition>
</philosopher>

Another option is to create an assignment when a P4D representation is created:

p4d> print philosopher.p4dstr(name = "p4ddoc")
p4ddoc = philosopher:
    name: "Hegel"
    main-work: "Phaenomenologie des Geistes"
    first-edition: "1807"

3.2  xml2p4d conversion from the command-line

Let's say you have an XML file foo.xml and you want to convert it into P4D. Then you can use the --xml2p4d command line option and run

               python run_p4d.py --xml2p4d foo.xml

This will create a file foo.p4d in the directory of foo.xml. If you omit the name foo but just write

               python run_p4d.py --xml2p4d *.xml

all files with extension xml in the directory you are pointing to will be converted to p4d.

3.3  The p4d namespace and its transformations

The namespace that is characterized by the URL http://fiber-space.de/p4dns and the prefix p4d is somewhat special as we have noted in section two already.

Now take a look at the following XML element kept from a catalog.xml file which lists resources of Adobe Flex components.

<library path="library.swf">
   <script name="mx/core/mx_internal" mod="1146526998000" >
      <def id="mx.core:mx_internal" />
      <dep id="AS3" type="n" />
   </script>
</library>

It would translate into

library(path="library.swf"):
   script(name="mx/core/mx_internal", mod="1146526998000"):
      def(id="mx.core:mx_internal")
      dep(id="AS3" type="n")

and this fragment would immediately fail to compile because def which is used as an element name is a P4D ( and Python ) keyword. That's why the p4d namespaces prefix is added automatically by default:

p4d> p4d_lib = P4D.from_xml(xml_lib).p4dstr()
p4d> print p4d_lib
library( path="library.swf" ):
    script( name="mx/core/mx_internal", mod="1146526998000" ):
        p4d:def( id="mx.core:mx_internal" )
        dep( type="n", id="AS3" )

The reverse operation works also in the other direction i.e. when creating the XML fragment from P4D. Applying  xmlstr()

p4d> print P4D.eval(p4d_lib).xmlstr()
<library path="library.swf">
   <script name="mx/core/mx_internal" mod="1146526998000" >
      <def id="mx.core:mx_internal" />
      <dep id="AS3" type="n" />
   </script>
</library>

We can also suppress adding the p4d prefix on loading from XML or we can suppress stripping the prefix away:

p4d> print library.xmlstr(strip_p4d = False)
<library path="library.swf">
    <script name="mx/core/mx_internal" mod="1146526998000">
        <p4d:def xmlns:p4d="http://fiber-space.de/p4dns" id="mx.core:mx_internal"/>
        <dep type="n" id="AS3"/>
    </script>
</library>

Not only has the namespace prefix been preserved but also the correct namespace declaration has been inserted. The latter will be done only once.

3.4  Translating XML comments and CDATA

XML comments will be translated into P4D comments. That's why P4D comments were introduced in the first place. A slightly different translation will be performed for CDATA sections. A P4D comment is fully specified by {* content *}. If we replace the single star by a double star the P4D comment will be translated into <![CDATA[ content ]]>.

p4d> elm stuff:
....    '''
....    Some text
....    '''
....    {* and a simple comment ... *}
....    {** and a cdata
....    section
....    **}
....
p4d> print stuff.xmlstr()
<stuff>
    Some text

    <!-- and a simple comment ...     -->
    <![CDATA[ and a cdata
    section
    ]]>
</stuff>
p4d>
p4d> print P4D.from_xml(stuff.xmlstr()).p4dstr()     # and the other way round
stuff:
    "Some text"
    {*
        and a simple comment ...
    *}
    {**
        and a cdata
    section
    **}

p4d>

Whitespace handling is not perfect. I hope I'll drive it perfection in the future :)

Notice that CDATA comments are treated much like regular P4D comments i.e. they are normal P4D elements, there is a CDATA() method used to access a CDATA section and there is an element tag which is '**'.

There is obviously some redundancy about comments in P4D but usually you don't have to care much about them unless you are converting from / to XML for particular reasons.

3.4. Case Study - Adobe Flex

Adobe Flex is a framework used to support so called Rich Internet Applications ( RIA ). This is such a hot topic right now that it is somewhat too hot for me to discuss here. So my interest is focussed mainly on MXML which is an XML language for declarative GUIs descriptions and part of Flex. The idea of using XML to describe programming language constructs has been critizised a lot in the past and I mostly agree with it myself. However one has to change the perspective to do justice about it. When you consider MXML from the point of view of a tool vendor who offers a GUI designer that enables roundtrip engineering in a safe way, it makes good sense. MXML becomes essentially a human readable serialization format for the FlexBuilder tool. This is much in alignment with Flash which is also basically a tool and a player. However Adobe has released an SDK a while ago and now you can build your applications from the ground up in a way that is more appealing to programmers like me ( I don't alway rate flexibility over comfort even when programming but I nevertheless tend to be fundamentalist as a programmer ).

I added the flexutils.p4d module to P4D. It might not be for everyone and I'll refuse to explain Flex related issues ( like configuration ) here. If interested read the code and adapt it to your advantage. It is not much anyway. However flexutils.p4d touches some framework related topics which might be of interest for a general readership.

3.4.1  Import flexutils.p4d from Python

Any langlet ( *.p4d ) module compiles to Python bytecodes and can therefore be imported as a bytecode ( *.p4d ) file. But now suppose you have a Python module that imports a p4d file. That's what run_p4d.py does:

  # run_p4d.py
  # ----------
 
  import flexutils   # imports a p4d-module

and you change flexutils.p4d. Only as long as flexutils.p4d gets compiled before run_p4d.py is executed everything works well and the most recent changes will be notified. Otherwise the old flexutils.pyc will be imported. In order to enforce recompilation on change we replace the import directive:

  # run_p4d.py
  # ----------
  import EasyExtend.eeimporter as eeimporter
  import langlet as p4d_langlet
  flexutils = eeimporter.import_from_langlet(p4d_langlet, "flexutils")

The import_from_langlet function is generic over langlets. You need to pass the right langlet of course to apply the correct transformations on flexutils.p4d. This import pattern doesn't have to be applied when you import Python modules from P4D.


You can't run
            if __name__ == '__main__':
                BODY

defined inside a .py module from P4D. You can run this inside .p4d modules though. The reason is that EasyExtend transforms this statement into an ordinary method call and calls this method but only for langlet modules not for Python modules that won't get transformed at all.

4. Bytelets ( new in P4D 1.2 )

A detailed treatment of Bytelets can be found here.

At this place it shall only be mentioned that Bytelets affect P4D as a whole in following respects:
  • Two more namespaces are introduced. Their prefixes are bl and bl-schema.
  • Hexadecimal literals like 0x89 or 0xA57 are not mapped on integers in P4D but on Hex objects. Moreover it is possible to write sequences of such numbers separated by a space e.g. 0x01 0x78 0x67. This is an important departure from Python and might break existing semantics.

5. Token.ext & Grammar.ext

The syntactical extensions involve both the lexer and the parser.

5.1 Token.ext

DOUBLECOLON
OPERATOR
NAME
P4_NS
Name
P4D_Comment
 ::=
::=
::=
::=
::=
::=
 '::'
OPERATOR_DEF | DOUBLECOLON
Name ('-' NAME)* | P4_NS
'p4d' ':' NAME
A_CHAR ( A_CHAR | A_DIGIT )*
'{' '*' ('*'|ANY)* '*' '}'
Most noteworthy is the redefinition of NAME and the introduction of Name that carries on the standard definition of the Token module.
Enabling hyphenated names of the kind monty-python-football is problematic in other contexts where it has to be understood as an arithmetic difference or isn't permitted entirely. The transformation actually takes care:

p4d> a = b = 0
p4d> a-b
0
p4d> a-b:0
P4D<a-b>
p4d> import a-b
Traceback...
...
SyntaxError: invalid syntax. No hyphenation permitted in Python names.

With P4_NS we can escape keywords. Just look at the difference in behaviour:

p4d> a:pass:""
Traceback...
...
ParserError: Failed to parse input 'pass' at (line 1 , column 6).

    a:pass:""

      ^^^^
                                                                                         
p4d> p4d:pass:""
P4D<p4d:pass>

This wouldn't be feasible with the current parser if we won't deal with "p4d:pass" as a single chunk and doing some postprocessing.

5.2 Grammar.ext

compound_stmt
p4d_compound_stmt
p4d_element
p4d_attribute_list
p4d_attribute
p4d_suite
p4d_simple_stmt
p4d_stmt
p4d_expr

p4d_accessor
trailer
p4d_name
p4d_attr_access
atom




HEXNUM
subscript


 ::=
::=
::=
::=
::=
::=
::=
::=
::=

::=
::=
::=
::=
::=


::=
::=


 if_stmt | while_stmt | for_stmt | try_stmt | ... | p4d_compound_stmt
['elm' | NAME '='] p4d_element ':' p4d_suite
p4d_name ['(' [p4d_attribute_list] ')']
p4d_attribute ([','] p4d_attribute)*
p4d_name '=' ['&'] test
p4d_simple_stmt | NEWLINE INDENT p4d_stmt+ DEDENT
(p4d_element | p4d_expr) NEWLINE
p4d_simple_stmt | p4d_compound_stmt
'&' '.'* test | '(' [ p4d_expr (',' p4d_expr)*] ')' | '[' [ p4d_expr (',' p4d_expr)*] ']' | STRING | NUMBER | HEXNUM | P4D_Comment
'.' p4d_attr_access |'::' NAME | '.' '(' test ')'
'(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME | p4d_accessor
NAME [':' NAME]
'@' ( NAME | '*')
atom: ('(' [yield_expr|testlist_gexp] ')' |
       '[' [listmaker] ']' |
       '{' [dictmaker] '}' |
       '`' testlist1 '`' |
       NAME | NUMBER | STRING+ | HEXNUM | p4d_attr_accesss
Hexnumber+
'.' '.' '.' | test | [test] ':' test [sliceop] | [test] '::' test |
'::' test | [test] ':'

Rules that substitute those of Pythons grammar are atom, trailer and compound_stmt. P4D specific statements are realized as compound statements and they follow the rules of Pythons indentation.

5.2.1  Translation of p4d_compound_stmt

A P4D compound statement is translated into a P4D object constructor call. Instead of giving a formal specification here we simply show some translations of common P4D compound statement structures.

p4d> a:
....    b:0
....
[python-source>
P4D( mapeval(['a', {}, [['b', {}, ['0']]]] , globals( )))

<python-source]
P4D<a>

The list that is passed to mapeval has the canonical form

             [TAG, ATTRS, CHILDREN, TEXT]

The textual content isn't stored separately but among the CHILDREN. P4D automatically stringifies numbers such as the 0. Actually the
content can be omitted completely

p4d> a:
....    b
....
[python-source>
P4D( mapeval(['a', {}, [['b', {}, []]]] , globals( )))

<python-source]
P4D<a>

Here is another example using attributes ( and also namespaces ).

p4d> p4d:if:
....    p4d:while(x=1 y=2):
....            p4d:pass
....
[python-source>
P4D( mapeval(['p4d:if', {}, [['p4d:while', {'y': 'evaltostr_1', 'x': 'evaltostr_2'}, [['p4d:pass', {}, []]]]]] ,
 globals( )))

<python-source]
P4D<p4d:if>

What's more remarkable here are the attribute value strings 'evaltostr_1' and 'evaltostr_2'. These strings are the reason for the presence of mapeval. The mapeval function seeks for strings beginning with "evaltostr_" and "evaltoobj_" and evaluates the appended part of the string which is "1" and "2" here. Finally it turns them into strings again. What is redundant in our particular example makes sense in the more general case.

p4d> a = 89
p4d> b(x = 1+a):""
[python-source>
P4D( mapeval(['b', {'x': 'evaltostr_1 + a'}, []] , globals( )))

<python-source]
P4D<b>

Now we can access the evaluated attribute value:

p4d> c = b(x = 1+a):""
p4d> c.@x
'90'


6. History


Date
 Version
 Change
2008-09-04
 1.2.4
  • Fix subelement rules for lists.
  • Change root element in p4d scripts created with options --xml2p4d --flex
2008-08-16 1.2.3
  • Bugfix in evalutils: elements of the kind

          A:
             B: ["x1", "x2"]
             C: ["y1", "y2"]

          caused problems when C was expanded.
  • Add P4DContentList for storing content that stems from multiple inputs as in the following case:
          A:
             "A string"
             "Another String"
  • add get() method to P4D and P4DList objects. It's close to child() but unlike child it doesn't raise a ValueError exception if child element is unavailable.
2008-08-13
 1.2.2
  • Bugfix in xmlutils: attribute accepts empty string values.
  • Bugfix in p4dutils: xml2p4d conversion failed due to import error on P4D when not started from P4D.
2008-08-10 1.2.1
  • Worst case error i.e. grammar bug fixed. Token.ext specified (* ... *) as comments within P4D elements. I completely overlooked that this can't be used since foo(*bar) is a valid. This bug remained unrecognized because Grammar / Token are separated from each other. We need to replace (* ... *) by {* ... *}. This is both Python 2.X as well as Python 3.0 compliant. This change is not downwards compatible!
2008-07-12 1.2
  • One can now set attribute values and content easily without referring to the internal datastructure.
  • Introduction of Bytelets and the bl namespace which is supported on language level.
  • Adding a Hex object in this context.
  • elm foo:bar: ... works now like an assignment  bar = foo:bar:... which is valid  for all for all non keywords and non hyphened names.
2008-07-11 1.1.1
  • Fix: remove py4as3 import from xmlutils and p4dbase
2008-07-11 1.1
  • Fix XML formatAttributes() in xmlutils.
  • Make locals() available for mapeval.
  • Add flexutils module
  • Add elm keyword
  • Add P4D comments
  • Fix double-colon access for namespaces with p4d prefix.
  • Add element lists or tuples.
  • Add P4D.eval()
  • Add CDATA handling
  • Change filter semantics
2008-06-26
 1.0
 Initial release