Codegeneration

An upcoming release of XBMC will change the python interface from the existing hand coded one, with one whose code is generated as part of building XBMC. There are several advantages to this approach including:

1. More flexible and straightforward means of managing the API. The API will be defined in terms of a C++ library independent of the fact that it can be called from a scripting language.
2. Provide a means of merging different APIs into a single API accessed through different means. It would be nice to define the API independent of the means of access.
3. Tackle some longstanding stability issues resulting from the mixing of Python mechanisms with XBMC mechanisms. These are typically deadlocks.
4. Smaller source codebase to manage
5. Provide a basis to support other scripting language.

There are some distinct disadvantages.

1. Object code size will be larger.
2. As with any large and new module, there will be issues that need to be worked out.
3. More work will be required to parse out the documentation and include it in the scripting language interface. This could be done with either a future SWIG release (a current GCOS project for SWIG is to include documentation) or a separate (optional) doxygen pass.

How it works

In general, the code-generator is build in two phases. In the first phase SWIG is used to parse C++ header files that define the API. SWIG outputs an XML file that contains a complete description of the structure of the API. In the second phase, the XML file is ingested by a Groovy program that then creates C++ code that forms the bridge to the scripting language (Python).

Referring to the figure above, the following steps contain a little more detail.

1. A SWIG interface file is created per API module. The SWIG interface file is documented in the SWIG documentation. It is mainly a SWIG configuration file that includes each of the header files that is meant to be included in the given module. It is also used to insert code snippets into the generated code so that, for example, the appropriate dependencies can be '#include'-ed.
2. SWIG is run (currently from the Makefile) and XML is output for the given module. The XML file contains a full description of all of the code included via the SWIG interface file.
3. The XML file is ingested by a Groovy program in order to generate the final .cpp. Groovy has native XML parsing support as well as a built in template processor that makes it very powerful and flexible. When invoking the Groovy program, a template is supplied. The template drives the code-generation for a particular scripting language. It's basically the rule set for how to construct the language interface from the structured, parsed, API representation in the XML. The result is a C++ file that contains the scripting language interface code.
4. Finally the resulting C++ file is compiled into XBMC.

An Example

This is a walkthrough of a simple example to show how this works and how it can be extended in the future. Let's say we have the following simple API function in XBMC that we want available to python. Since this is actually a piece of the current API's module 'XBMC,' let's say this is the complete contents of the file "ModuleXbmc.h":

    void log(const char* msg, int level = lLOGNOTICE);

SWIG takes SWIG interface files as input. Interface files allow the SWIG user to direct the code generation a many different levels. The interface file has a C++-like syntax and must at least specify the module name and identify the include files that are to be parsed as part of the API. For the purposes of our simplified example, an AddonModuleXbmc.i interface file that only includes our "ModuleXbmc.h" would look like:

%module xbmc

%include "native/ModuleXbmc.h"

The SWIG interface file syntax is very rich and can give very fine grained control over the code-generation and the parse-tree that SWIG creates. Please look at the interface file tutorial and the complete SWIG documentation.

The first pass, the call to SWIG, generates a really ugly, but very detailed XML file. This file contains a lot of extraneous information that SWIG has attached to the parse tree, but the section we care about looks like this (this is only here for completeness, even if you write your own templates you don't need to know this):

    <include id="208" addr="82de5b50" >
        <attributelist id="209" addr="82de5b50" >
            <attribute name="name" value="AddonModuleXbmc.i" id="210" addr="82de68f0" />
            <attribute name="module" value="" id="211" addr="82de5bf0" />
            <attribute name="options" value="82de5ad0" id="212" addr="82de5ad0" />
        </attributelist >
        <module id="213" addr="82de5b90" >
            <attributelist id="214" addr="82de5b90" >
                <attribute name="name" value="xbmc" id="215" addr="82de68f0" />
            </attributelist >
        </module >
        <include id="216" addr="82de5c50" >
            <attributelist id="217" addr="82de5c50" >
                <attribute name="name" value="[absolute path to header]/ModuleXbmc.h" id="218" addr="82de68f0" />
            </attributelist >
            <cdecl id="219" addr="82de5fb0" >
                <attributelist id="220" addr="82de5fb0" >
                    <attribute name="name" value="log" id="221" addr="82de68f0" />
                    <attribute name="sym_symtab" value="82dd27f0" id="222" addr="82dd27f0" />
                    <attribute name="sym_nextSibling" value="82de61f0" id="223" addr="82de61f0" />
                    <attribute name="csym_nextSibling" value="82de61f0" id="224" addr="82de61f0" />
                    <attribute name="kind" value="function" id="225" addr="82de68f0" />
                    <attribute name="sym_name" value="log" id="226" addr="82de68f0" />
                    <attribute name="decl" value="f(p.q(const).char,int)." id="227" addr="82de68f0" />
                    <attribute name="sym_overloaded" value="82de5fb0" id="228" addr="82de5fb0" />
                    <parmlist id="229" addr="82de5db0" >
                        <parm id="230">
                            <attributelist id="231" addr="82de5db0" >
                                <attribute name="name" value="msg" id="232" addr="82de68f0" />
                                <attribute name="type" value="p.q(const).char" id="233" addr="82de68f0" />
                            </attributelist >
                        </parm >
                        <parm id="234">
                            <attributelist id="235" addr="82de5f10" >
                                <attribute name="name" value="level" id="236" addr="82de68f0" />
                                <attribute name="value" value="LOGNOTICE" id="237" addr="82de68f0" />
                                <attribute name="type" value="int" id="238" addr="82de68f0" />
                            </attributelist >
                        </parm >
                    </parmlist >
                    <attribute name="type" value="void" id="239" addr="82de68f0" />
                    <attribute name="sym_overname" value="__SWIG_0" id="240" addr="82de68f0" />
                </attributelist >
            </cdecl >
            <cdecl id="241" addr="82de61f0" >
                <attributelist id="242" addr="82de61f0" >
                    <attribute name="sym_name" value="log" id="243" addr="82de68f0" />
                    <attribute name="name" value="log" id="244" addr="82de68f0" />
                    <attribute name="decl" value="f(p.q(const).char)." id="245" addr="82de68f0" />
                    <parmlist id="246" addr="82de6150" >
                        <parm id="247">
                            <attributelist id="248" addr="82de6150" >
                                <attribute name="name" value="msg" id="249" addr="82de68f0" />
                                <attribute name="type" value="p.q(const).char" id="250" addr="82de68f0" />
                            </attributelist >
                        </parm >
                    </parmlist >
                    <attribute name="kind" value="function" id="251" addr="82de68f0" />
                    <attribute name="sym_overloaded" value="82de5fb0" id="252" addr="82de5fb0" />
                    <attribute name="defaultargs" value="82de5fb0" id="253" addr="82de5fb0" />
                    <attribute name="type" value="void" id="254" addr="82de68f0" />
                    <attribute name="sym_previousSibling" value="82de5fb0" id="255" addr="82de5fb0" />
                    <attribute name="sym_symtab" value="82dd27f0" id="256" addr="82dd27f0" />
                    <attribute name="sym_overname" value="__SWIG_1" id="257" addr="82de68f0" />
                </attributelist >
             
            </cdecl >
        </include >
    </include >

Ugly, eh? Well, it's too complicated and verbose to provide directly to a Groovy template so the generator transforms the XML prior to providing it to the template. The transformed XML can be obtained by running the Groovy generator script and passing the '-verbose' flag. For this example the transformed XML would look like:

<?xml version="1.0" encoding="UTF-8"?>
<module name="xbmc">
  <function name="log" sym_name="log" decl="f(p.q(const).char,int)." sym_overloaded="82de5fb0" type="void" id="82de5fb0">
    <parm name="msg" type="p.q(const).char"/>
    <parm name="level" value="LOGNOTICE" type="int"/>
  </function>
  <typetab/>
</module>

This is much simpler. Notice, how this xml reflects the original C++ structure. This forms the input 'bindings' (in Groovy template parlance) provided to the template. I cannot provide a full tutorial on Groovy templates because they are basically, in and of themselves, Groovy scripts. They are very analogous to JSPs in Java, but not specific to generating web pages.

The Template

Trivial Example

As an example, let's suppose, rather than generate C++, our template simply provides information about the API. If our template looks like this:

Module Name: ${module.@name}

Then the resulting output file would look like:

Module Name: xbmc

Notice, we directly reference the xml element by name in the template. Use the '@' sign to reference an attribute of that element. The template is interpreted as a Groovy GString and therefore anything within the ${ ... } has its value, as a string, substituted.

We can access children of the element as a list (since child elements can be repeated). This combined with Groovy's support for closures makes the templating very powerful. Let's modify the template to print out all of the function names in the module.

Module Name: ${module.@name}
<%
module.function.each { 
%>
  function: ${it.@name}
<%
}
%>

Note that anything within <% ... %> can be straight Groovy code with direct access to the parse tree. In this case we apply a closure to 'each' function in the module. Within that closure we exit the <% ... %> brackets and here, whatever we type ends up in the output file. The above template results in:

Module Name: xbmc

  function: log

Let's extend the template to provide parameter information.

Module Name: ${module.@name}
<%
module.function.each { functionNode ->
%>
  function: ${functionNode.@name}
<%
  functionNode.parm.eachWithIndex { param, index ->
%>
    parameter ${index}= name:${param.@name}, type:${param.@type}<% if (param.@value) { %>, default value: ${param.@value} <% } %>
<%
   }
}
%>

Now, within the closure for each function, we're providing a closure that's applied over each parameter. The resulting output file now contains:

Module Name: xbmc

  function: log

    parameter 0= name:msg, type:p.q(const).char

    parameter 1= name:level, type:int, default value: LOGNOTICE