Codegeneration

As of XBMC v12 Frodo, the python interface has changed from the existing hand coded one, with one whose code is generated as part of building XBMC. There are several advantages to this approach including:


 * 1) More flexible and straightforward means of managing the API. The API will be defined in terms of a C++ library independent of the fact that it can be called from a scripting language.
 * 2) Provide a means of merging different APIs into a single API accessed through different means. It would be nice to define the API independent of the means of access.
 * 3) Tackle some longstanding stability issues resulting from the mixing of Python mechanisms with XBMC mechanisms. These are typically deadlocks.
 * 4) Smaller source codebase to manage
 * 5) Provide a basis to support other scripting language.

There are some distinct disadvantages.


 * 1) Object code size will be larger.
 * 2) As with any large and new module, there will be issues that need to be worked out.
 * 3) More work will be required to parse out the documentation and include it in the scripting language interface. This could be done with either a future SWIG release (a current GCOS project for SWIG is to include documentation) or a separate (optional) doxygen pass.

How to use the codegenerator
Now the ability to extend the API should be much simpler. You simply write C++ code and make sure that it gets incorporated in the appropriate module file. The directories within XBMC are organized as follows:


 * xbmc/interfaces/legacy contains the C++ code that represents the current (now, legacy) API.
 * xbmc/interfaces/swig contains a SWIG interface file for each module in the API. SWIG interface files end with '.i'
 * xbmc/interfaces/python contains the template for generating the python specific bindings.

In order to run the codegenerator on any changes you made to the native api in interfaces/legacy you need to run 'make' from xbmc/interfaces/python.

An example of extending the API
The simplest thing to do is to add a trivial method to an existing module. Suppose we want to add a method called 'test' that can be called from python. Let's add this to the file ModuleXbmc.h inside of the namespaces. Simply add this code:

That's it. Now you can call it from a python script. For example:

Current requirements/restrictions
The codegenerator cannot handle every possible case so here is a list of things to keep in mind:


 * It can handle all built in types as return values or as parameters (let me know if you find any missing and I can add them).
 * It can handle several container types:
 * 1) std::map - as long as the template parameters can also be handled
 * 2) std::vector - as long as the template parameters can also be handled
 * 3) XBMCAddon::Tuple - a 'varadic' template Tuple of (currently) up to four entries. Again the template parameter types must also be able to be handled
 * 4) XBMCAddon::Dictionary - this is probably redundant with the std::map and may be removed.
 * 5) std::string, XBMCAddon::String (typedef of std::string).
 * It can handle API classes but they MUST:
 * 1) descend from XBMCAddon::AddonClass
 * 2) have a single explicit constructor
 * 3) Not have multiple base classes that are visible to the parser/Python. You need to hide all but one using #ifndef SWIG/#endif pairs.
 * The codegenerator cannot (yet) handle overloaded methods or functions.
 * The codegenerator doesn't handle operator overloading and all defined operators will simply be ignored - EXCEPT for the following situation:
 * For python you can define a "comparator" (NOT a "rich comparator") if you define these three operators on your API class (you cannot use global operator definitions):
 * 1) bool operator ==(const T&)
 * 2) bool operator >(const T&)
 * 3) bool operator >(const T&)
 * Unlike SWIG, this codegenerator doesn't handle typedefs so you'll need to define your API in terms of explicit types.

If this is too restrictive feel free to suggest enhancements. As it is, this is all that's needed to support the entire existing API.

Adding a new Class
As mentioned, your class must inherit from AddonClass and define a single constructor. You can hide any implementation details from the scripting language using #ifndef SWIG/#endif so if you want to define multiple constructors, hide all but one from the parser.

Please use the namespace convention already set forth. Then %include your header file in the appropriate SWIG interface file in the xbmc/interfaces/swig directory. Don't forget to make sure your header file is also #included in the generated code using the header section of the interface file. See the SWIG documentation for more details.

Cross-language Polymorphism
Make sure you read "Cross-language Polymorphism in XBMC" before implementing a callback.

If you want to be able to extend your new class in Python and override methods that get called when they are invoked on the parent class (a.k.a. callbacks), then you need to declare the class as a "director". Turn on "directors" at the top of the interface file using the module attribute "directors" as follows:

All of the existing modules do this already.

Next you need to enable it for your particular class. Prior to the %include of your header file use the %feature directive to tell the parser it's a "director" as follows:

Now, every virtual function visible to SWIG will be a method that could be overloaded in a Python subclass which will be called when invoked from either side, Python or C++.

If you have virtual methods that you don't want to be used for Cross-language polymorphism, use the macro 'SWIGHIDDENVIRTUAL' in place of the keyword 'virtual' and it will be 'virtual' in C++, but simply callable in Python. An example from the 'Player' in the existing API:

Cross-language Polymorphism in XBMC
Unfortunately, it's not all that simple in XBMC. Since XBMC is a complicated multi-threaded application, and since Python requires absolute control over the threading context, you cannot simply invoke a method on a reference to the base class and expect the Python child class to execute without any problems. You must make sure you're already in a Python thread that's running your script in order for this to work.

Fortunately, there are facilities in the Addon library to make this somewhat painless, but you do need to explicitly take advantage of them.

First, your "director" class should inherit from 'XBMCAddon::AddonCallback' rather than XBMCAddon::AddonClass. Note, AddonCallback inherits from AddonClass so this satisfies the requirement.

Next, you need to "proxy" the callback method. Currently the convention is that this is done with a method invoked from the C++ side that has the same name and signature as the method that will be called on the Python side, but starting with a capital letter (this is not a requirement, just a convention).

Looking at the "Player" class you will see the following:

'OnPlayBackStarted' is a "proxy" for calling "onPlayBackStarted" that ensures that "onPlayBackStarted" will be invoked in the proper context. Note that "onPlayBackStarted" will be invoked asynchronously, and possibly some time after 'OnPlayBackStarted' returns. This means the methods should be 'void'.

The implementation of the 'OnPlayBackStarted' proxy is simple using the functionality provided. You use the "Callback" templates from "CallbackFunction.h" to send the callback message. Again, an example from the existing API:

The 'type' parameter for 'CallbackFunction' is the type of the class the callback is to be invoked on. The constructor parameters are then the object reference to invoke the callback on and a pointer the member function to be invoked.

'CallbackFunction' is a 'varadic' template to allow you to bind parameters to the callback to be invoked. Currently it supports up to 1 parameter but this can be easily extended upon request. We currently have callbacks on Window objects that take a parameter. An example is the Window::OnAction method:

Adding a new Module
To add a new module you need to define another interface file. Use one of the existing ones (AddonModuleXbmc.i is a good one as it has most permutations but isn't as big as AddonModuleXbmcgui.i), as well as the SWIG documentation, as a guide.

Tweaking
Currently you can tweak the codegeneration results using the following feature directives:


 * By default, in the existing python API, all returned strings are ascii or UTF8 stored in a std::string. In order to force the return value to be Unicode and use the PyUnicode_DecodeUTF8 call, in the interface file you can attach the "python:coerceToUnicode" directive. An example from the current API follows:


 * The codegenerator will give warnings if base classes for API classes are unknown. These can be suppressed:


 * When one function take as a parameter a type from another module, a warning will be given about an "unknown type." To suppress these warnings you can tell the parser about that unknown type in the given module as follows:


 * By default, all parameter names are used as "keywords" in the python methods. This isn't always desired, especially in constructors of classes that are meant to be extended in python. To use positional parameter information only use:


 * If someone get's the bright idea to name an Python accessible api function so that it matches a C++ keyword, you can fix this using the %rename directive. The following allows the method XBMCAddon::xbmcvfs::deleteFile to be called from Python as 'xbmcvfs.delete':


 * You can add a method by writing the native CPython implementation of it directly. Please DON'T DO THIS. It's ONLY here to support the legacy Python API. Anything you do this way will ONLY be available to Python. Once this is used for other scripting languages or for JSON-RPC, methods implemented this way will be invisible to those other clients.

Notice that the arguments passed to your custom methods will be 'self', 'args', and 'kwds'.

How it works


In general, the code-generator is build in two phases. In the first phase SWIG is used to parse C++ header files that define the API. SWIG outputs an XML file that contains a complete description of the structure of the API. In the second phase, the XML file is ingested by a Groovy program that then creates C++ code that forms the bridge to the scripting language (Python).

Referring to the figure above, the following steps contain a little more detail.


 * 1) A SWIG interface file is created per API module. The SWIG interface file is documented in the SWIG documentation. It is mainly a SWIG configuration file that includes each of the header files that is meant to be included in the given module. It is also used to insert code snippets into the generated code so that, for example, the appropriate dependencies can be '#include'-ed.
 * 2) SWIG is run (currently from the Makefile) and XML is output for the given module. The XML file contains a full description of all of the code included via the SWIG interface file.
 * 3) The XML file is ingested by a Groovy program in order to generate the final .cpp. Groovy has native XML parsing support as well as a built in template processor that makes it very powerful and flexible. When invoking the Groovy program, a template is supplied. The template drives the code-generation for a particular scripting language. It's basically the rule set for how to construct the language interface from the structured, parsed, API representation in the XML. The result is a C++ file that contains the scripting language interface code.
 * 4) Finally the resulting C++ file is compiled into XBMC.

An Example
This is a walkthrough of a simple example to show how this works and how it can be extended in the future. Let's say we have the following simple API function in XBMC that we want available to python. Since this is actually a piece of the current API's module 'XBMC,' let's say this is the complete contents of the file "ModuleXbmc.h":

SWIG takes SWIG interface files as input. Interface files allow the SWIG user to direct the code generation a many different levels. The interface file has a C++-like syntax and must at least specify the module name and identify the include files that are to be parsed as part of the API. For the purposes of our simplified example, an AddonModuleXbmc.i interface file that only includes our "ModuleXbmc.h" would look like:

%module xbmc

%include "native/ModuleXbmc.h"

The SWIG interface file syntax is very rich and can give very fine grained control over the code-generation and the parse-tree that SWIG creates. Please look at the interface file tutorial and the complete SWIG documentation.

The first pass, the call to SWIG, generates a really ugly, but very detailed XML file. This file contains a lot of extraneous information that SWIG has attached to the parse tree, but the section we care about looks like this (this is only here for completeness, even if you write your own templates you don't need to know this):

Ugly, eh? Well, it's too complicated and verbose to provide directly to a Groovy template so the generator transforms the XML prior to providing it to the template. The transformed XML can be obtained by running the Groovy generator script and passing the '-verbose' flag. For this example the transformed XML would look like:

This is much simpler. Notice, how this xml reflects the original C++ structure. This forms the input 'bindings' (in Groovy template parlance) provided to the template. I cannot provide a full tutorial on Groovy templates because they are basically, in and of themselves, Groovy scripts. They are very analogous to JSPs in Java, but not specific to generating web pages.

Trivial Example
As an example, let's suppose, rather than generate C++, our template simply provides information about the API. If our template looks like this:

Then the resulting output file would look like:

Module Name: xbmc

Notice, we directly reference the xml element by name in the template. Use the '@' sign to reference an attribute of that element. The template is interpreted as a Groovy GString and therefore anything within the ${ ... } has its value, as a string, substituted.

We can access children of the element as a list (since child elements can be repeated). This combined with Groovy's support for closures makes the templating very powerful. Let's modify the template to print out all of the function names in the module.

Note that anything within <% ... %> can be straight Groovy code with direct access to the parse tree. In this case we apply a closure to 'each' function in the module. Within that closure we exit the <% ... %> brackets and here, whatever we type ends up in the output file. The above template results in:

Module Name: xbmc

function: log

Let's extend the template to provide parameter information.

Now, within the closure for each function, we're providing a closure that's applied over each parameter. The resulting output file now contains:

Module Name: xbmc

function: log

parameter 0= name:msg, type:p.q(const).char

parameter 1= name:level, type:int, default value: LOGNOTICE

Type information
You may have noticed the type information for the parameters above. The type information is generated from the SWIG type system. This type system is very powerful and most of it has been 'transliterated' into Groovy and is accessible to the template. Reading the type information is fairly straightforward and documented in the SWIG documentation on the Type System.

The type system includes the means of representing types that can be used on the left hand side of the assignment operator. These types are called 'lvalues'. We can modify the template to show the 'type' information both as an "lvalue" and straight as follows:

Access to the Type System is through the class SwigTypeParser. Therefore the "SwigTypeParser" needs to be imported. SwigType_str, and SwigType_lstr convert a type string to a representative C++ type in either a straightforward manner, or as its corresponding 'lvalue' type, respectively.

Module Name: xbmc

function: log

parameter 0= name:msg, type:p.q(const).char type:char const * lvalue-type:char *

parameter 1= name:level, type:int, default value: LOGNOTICE type:int lvalue-type:int

Notice, an instance of the 'lvalue-type' can be assigned an instance of the 'type.' This is the correct way to handle 'const'-ness in parameters.

Typemaps - Handling Parameters
The Groovy template has access to "typemaps" for the conversion of input parameters and return values. These are completely analogous to SWIG Typemaps but are actually mini-Groovy templates. They contains snippets that allow the generalization of how to convert various type to and from the scripting language. An "in typemap" is used to map parameter values from the scripting language, to the API types. An "out typemap" will map types returned from the API to the scripting language.

In this example an "in typemap" is used to map the 'log' input parameters from a hypothetical scripting language to the API values.

Typemap handling needs to be initialized by calling the 'setup' method on the "Helper" Groovy class (which, of course, needs to be imported).

A Map of type to "in typemap" is the fourth parameter to the "Helper.setup" method. The key is the 'type' string (ltype' also works). In the example above the typemap has only two entries (given that our example only has two parameter types).

The "in typemap" has a set of predefined 'bindings' (or variable values) that it has access to. Each of the typemaps above uses:


 * 1) api - is a string containing the name of the variable that will be passed to the API class.
 * 2) slarg - is the variable name holding the value that was passed in from the scripting language that needs to be converted and stored in 'api'.

If the scripting language is Python then the 'slarg' variable's type will be a "PyObject*". In our example this value is just passed to the function 'convertSl*'.

The Template needs to then use the typemap functionality through the "Helper" class. The method "Helper.getInConversion" is used from the template to execute the typemap for a given parameter. Augmenting the script so far gives:

Now, for each parameter (in each function - though there is only one in our example), a section is printed out that shows how that parameter would be converted from the scripting language to the api type given the provided "in typemaps." The results are:

Module Name: xbmc

function: log

parameter 0= name:msg, type:p.q(const).char type:char const * lvalue-type:char * code to handle parameter 0 { // declare and set the value that came in from the scripting languge ScriptingLanguageType sl_msg = /* set the value from the scripting language */; // declare and set the variable that will contain the api parameter char * p_msg; p_msg = convertSlString(sl_msg); }

parameter 1= name:level, type:int, default value: LOGNOTICE type:int lvalue-type:int code to handle parameter 1 { // declare and set the value that came in from the scripting languge ScriptingLanguageType sl_level = /* set the value from the scripting language */; // declare and set the variable that will contain the api parameter int p_level; p_level = convertSlInt(sl_level); }

Typemaps - Return types
As you might have suspected by now, the "out typemap" is for converting return values. The Map of "out typemaps" is the second parameter of the "Helper.setup" method. In this case, the "return" type (as such) is 'void.' Languages like Python always have a return so an API call that's 'void' still needs to return a Py_None instance to the Python caller. In a case like this the following typemap makes sense:

Here, we can see one of the bindings used in the "out typemap." The value of 'result' holds the name of the variable that will be returned from the scripting language. Using the "out typemap" via the "Helper.getOutConversion":

Notice, the "Helper" class contains a convenience method (Helper.getReturnSwigType) for extracting the return type from the functionNode of the parsed XML. Adding these two snippets to the correct places in the template gives a generated output file that contains:

code to handle return value { // This is an example of how Python handles return values Py_Object* result; Py_INCREF(Py_None); result = Py_None; return result; }

Calling the API method
Of course, the API method invocation needs to be output from the template logic. There is convenience method on the "Helper" class for this. Adding the following to the template will output the method invocation:

which would put the following into the output file:

code to invoke the api method { log( p_msg,  p_level  ); }

The final Template and Output
For reference then, this Template:

Gives this output with the simple ModuleXbmc.h file and AddonModuleXbmc.i file we provided earlier:

Module Name: xbmc

function: log

parameter 0= name:msg, type:p.q(const).char lvalue:char * rvalue:char const * code to handle parameter 0 { // declare and set the value that came in from the scripting languge ScriptingLanguageType sl_msg = /* set the value from the scripting language */; // declare and set the variable that will contain the api parameter char * p_msg; p_msg = convertSlString(sl_msg); }

parameter 1= name:level, type:int, default value: LOGNOTICE lvalue:int rvalue:int code to handle parameter 1 { // declare and set the value that came in from the scripting languge ScriptingLanguageType sl_level = /* set the value from the scripting language */; // declare and set the variable that will contain the api parameter int p_level; p_level = convertSlInt(sl_level); }

code to invoke the api method { log( p_msg,  p_level  ); }

code to handle return value { // This is an example of how Python handles return values Py_Object* result; Py_INCREF(Py_None); result = Py_None; return result; }

Why not simply use SWIG?
We spent a long time trying to get direct SWIG output of Python to work correctly. There are several issues that became intractable. For example, the SWIG code when run inside of XBMC could be very unstable. See this bug report.

On top of that the SWIG output code is incredibly large and complex. This is because it needs to satisfy the general case and so must be complete.

With the addition of the Groovy code-generation we have complete control over the actual code that's generated. As a result the generated code is specific to XBMC and can be tuned the way we need it to be.

Of course, there is a cost to this in terms of both maintenance and build complexity.