Reflection in libgdx’s GWT backend

About a year ago, i finished our GWT backend, which allows you to compile your Java code to straight Javascript. This means no applets, just pure WebGL goodness (except for IE). There were multiple issues that i had to solve:

  • There is no filesystem in the browser. However, libgdx’s Files module exposes just that to you
  • Many of the Java runtime classes are not available or incomplete in GWT
  • WebGL doesn’t map 100% to your plain old OpenGL ES 2.0 APIs. You work with objects instead of integer ids, requiring a kind of mapping
  • No native code support, so our box2D wrapper is useless.
  • GWT does not support reflection

I may cover some of the above in future blog posts. In this post i want to discuss how i solved the problem of reflection in GWT.

Libgdx started out without any reflection code. Then came Nate. He wrote quite a few very handy things, like our JSON parser or scene2d’s UI, among many other things. Nate is in love with reflection, so naturally he uses it whenever the oportunity arises.

This turned out to be a huge problem for the GWT backend. Without reflection, all of these nice things wouldn’t be available in GWT, so i had to come up with a solution. So, how does it work?

The first thing that needed to be done was emulating all those libgdx core classes that use reflection. Emulation in GWT means duplicating the original code, and replacing problematic portitions with GWT specific solutions. Those emulation classes then overwrite the normal Java implementation, via the super source mechanism of GWT.

Once the emulation classes where in place, i started coming up with a reflection API that would be used in those classes instead of the original Java reflection API. The end result is largely based on GWT’s own Type and related classes.

Here’s one original method from the Array class of libgdx:

This class creates a new typed array at runtime, which has a specific element type. The corresponding method in the GWT emulation class uses the custom reflection API instead and looks like this:

I basically went through all the reflection API calls in the original sources, and added them to the reflection API for GWT. As such, the API for the GWT backend is pretty close the original Java API. Here’s a rundown of the classes:

  • Type, equivalent to Java’s Class. Holds constructors, methods and fields for a type along with information on the type itself, e.g. it’s superclass, implemented interfaces, access qualifiers and so on.
  • Constructor, represents a default constructor of a type, without arguments. Our GWT reflection API can only instantiate classes with such a constructor!
  • Field, represents a field of a type. Stores access qualifiers as well as a reference to the type of the field. Provides methods to get and set the field value on an instance of that type.
  • Method, you guessed it, represents the method of a type. Stores a list of parameters, the return type, access qualifiers and provides methods to invoke that method on an instance of the type.
  • ReflectionCache, equivalent to the static methods in the normal Java Class. Provides methods to find the type for a Class or fully qualified class name, instantiate arrays with specific element type and so on.

If you look at the implementations of say the get and set methods of the Field class, you see that they call into ReflectionCache.instance (sometimes they also call into the methods of ReflectionCache, which just delegate to that instance). This is a static field of the ReflectionCache class, and implements the IReflectionCache interface. Here’s how that field is initialized:

And this is where the magic begins. GWT provides a way to hook into its compiler. When the compiler encounters a call to GWT.create(), it checks if it can find a Generator that can provide the Java source for the class in question. This is also known as deferred binding and is used to solve a variety of problems in GWT. I abuse it to generate an implementation of the IReflectionCache interface that:

  • Contains a map from the fully qualified name of a Java class to a corresponding Type instance representing the class and its fields, methods, constructors and so on.
  • Implementations for the IReflectionCache methods that allow us to query Type instances, invoke methods, get and set fields and construct new objects and arrays.

The generation of the source of this implementation is performed in the ReflectionCacheSourceCreator. This class is called by the GWT compiler when the above call is encountered, and asked to basically print the Java source implementation for that interface to a file.

The entire workflow of this class is present in the ReflectionCacheSourceCreator#create() method. After some initial setup, I start by generating the Type lookup map by calling ReflectionCacheSourceCreator#generateLookup().

This method spits out the map as a field for the IReflectionCache implementation, then it tries to gather all the classes that are used in the application. The method ReflectionCacheSourceCreator#gatherTypes() plays a crucial role here, as it decides whether information is generated for a class or not. It will also recursively gather the types found in fields and method return types and parameters!

Once all the classes are gathered, the actual lookup generation is performed. For each class, a method is generated, which creates and instance of Type, and fills it with information on the class (constructor, fields, methods, super classes, etc.). Each of this methods is called c#number, e.g. c0() for the first type, c1() for the next type and so on. At the end of each of these methods, the instantiated Type is put into the map for later lookup. As you can see, the method decides wheter to include a class or not by checking either its package name or its class name. If we want to open up reflection to the public, we’d need to add a way for the user to specify additional classes for which reflection information needs to be generated.

Once all these methods are generated, i generate the constructor of the IReflectionCache implementation. The constructor is responsible for calling all these methods we generated previously, thereby populating the lookup map. You may ask yourself why all those methods are generated in the first place. Couldn’t i just put all that generated code in the constructor? The answer is no, Java methods and constructors can not be larger than 64KB. That limit is easily reached when lots of class information is generated.

Next i generate setter and getter methods for every field as well as a warpper method for every method of every class. Each setter/getter/method is assigned a unique name, e.g. s0 for the first field’s setter, or m10 for the 11th method. These methods basically wrap access to fields and methods, we will call them later in the implementations of the IReflectionCache interface methods we are about to generate. On interesting aspect of this is that i use JSNI for the implementatino of these wrappers. This allows me to set/get protected and private fields, and call protected and private methods. However, there are limitations. If the type of a field is not public, it can not be accessed. If the type of a field is long, it can not be accessed either, as GWT doesn’t support returning longs from JSNI back to Java.

Let’s recap what we did so far:

  • Generated a map field in the IReflectionCache implementation that stores class name to Type instance key/value pairs
  • One method per class that creates a Type instance, fills it with the class’ information, and puts the Type instance into the map for later lookup
  • The constructor of the IReflectionCache implementation tht calls all these methods
  • A setter and getter method for each field of every class, that uses JSNI to access protected and private fields as well.
  • A wrapper method for every method of every class that uses JSNI to access protected and private methods as well.

Note that every Field and Method instance has a field that records the id of its wrapper method(s)!

All this information is now used to implement the actual IReflectionCache methods. For each of the methods of the interface, there is one method in ReflectionCacheSourceGenerator responsible to output its implementation, e.g. the call to getKnownTypesC() generates the IReflectionCache#getKnownTypes() implementation:

The p() method just prints a line to the Java source file the generator is writting. The implementation of the getKnownTypes() method is pretty straight forward. It simply returns the values of the map we fill when the implementation is constructed.

The IReflectionCache#forName() method is equally simple, it just looks up the Type instance by name in the map.

The IReflectionCache#newArray and IReflectionCache#newInstance methods are a little bit more involved to implement. The generated implementations are basically huge if-statements that compare the fully qualified class name of the passed in type to the names of all the types we generated information for previously. That’s nasty, and could be solved differently, but it was easy to get going and also easier to debug than say an index based O(1) lookup… As you can see, we can only create new instances for classes that have a public default constructor. Constructors that are not public or have parameters are not supported!

IReflectionCache also has methods to get and set array elements as well as get an array’s length. I again generted a ton of if-statements and compare qualified class names. Once an if-statement matches, i just have to perform a little bit of casting.

Finally i generate the methods for invoking a method on an object, and getting and setting fields. It’s just basically more of the same, big if-statements, some casting.

Here’s an excerpt of the generted Java source:

Once we are done generating the Java source file for the IReflectionCache implementation, the GWT compiler will take the file and integrate it into the build on-the-fly. The GWT.create() call will then be replaced with new IReflectionCacheGenerated() statement, and our static field is initialized.

Now all the emulated classes call into our generated IReflectionCache class if they need reflection and all is well in GWT land.

As you can see from the above code dump, the way things are done is far from being optimal. I also didn’t go into a lot of the nitty gritty details, otherwise this post would be even more massive. I hope this serves as a good intro the reflection mechanism in the GWT backend.

The reason i’m telling you all this is that we are playing with the thought of implementing a reflection API in libdgx. This would allow folks to use reflection freely in their libgdx games while being still compatible with the GWT backend. It would also allow us to get rid of many of the emulated core classes, reducing code duplication and the potential to miss updates to core classes.

More libgdx games on iOS App Store!

New releases for iOS are always fun to report on 🙂

Tescott aka Gushiku Studios released Toy Slots on iOS about a week ago. He’s contributed a new audio backend to the iOS libgdx backend based on ObjectAL. Super cool!

Also, Ironhide Games released Clash of the Olympians on iOS today! The game was released a while ago on Android. Team Gemserk have been working on both the Android and iOS version for Ironhide, and contributed a new OpenGL backend to the libgdx ios backend.

All around excellent stuff, super excited we finally see some libgdx games appearing on the app store.

I already created the new iOS template for the setup UI, it’s pending integration. I’ll try to get this to you guys this week. Pinky promise 😀