Reflection API with GWT support!

Nex just finished his work on our new reflection API. You may ask why we need that? We not only target proper JVMs, but also GWT. GWT has no support for reflection. To give you reflection support, we created a simple API close the the standard Java APIs, that is now supported across platforms.

This also allows us to remove quite a few of the emulation classes in the GWT backend. An emulation class is a special version for GWT, that uses GWT specifics to implement its functionality. This mostly concerned things involving reflection. With the addition of this new API, we were able to eliminate quite a few of these emulated classes, and hence reduce code duplication.

Thanks to Nex for his awesome work. You can find more information on the API on this wiki page.

GWT backend updates

A couple of nice updates have arrived in the GWT backend thanks for Noblemaster, Nex and Vincent Bousquet on Github.

New Progress Bar

Previously, when your HTML5 version of your game loaded, you’d see a really ugly black/white/greyish “progress” bar. Programmer art, clearly.

A conspirative group of contributors banded together to fix my faux pas (see this thread). The end result looks way better.

null

Yay, branding :p You can change the appearance of the loading screen, as described here.

Reflection Support V1

GWT does not support standard Java reflection for various reasons. This posed a problem, as our UI library and other bits and pieces of libgdx like the JSON loader rely on reflection. Here’s the full story. TL;DR: i added reflection to the GWT backend to fix those issues. However, i did not expose this capabilities to users, as it’s one of the biggest hacks in my coding history. Only a small, fixed subset of classes had reflection information generated. If you used the Json class to read in Pojos, you found that that doesn’t work in the GWT backend, at all.

Vincent send us a pull request that now allows you to specify packages and class names for which reflection information should be generated. If you have Java Pojos you want to read from a Json file, you simply have to add their class name or package name to your GWT project’s gwt.xml file, like this:

You can specify multiple classes and packages by simply adding another extend-configuration-property element to your GWT file.

Note that this highly experimental, so ymmv.

We began talking about implementing a Reflection utility class that works across all platforms, including HTML5. We’ll add that in the upcoming weeks. It will be a bit more limited than standard Java reflection, but should suffice for almost all needs you have.

Preloader Improvements

Assets are preloaded when your GWT application is loaded. This is necessary to expose a virtual (read-only) file system to your application, so Gdx.files works as expected. The progress bar indicated how many of those assets have been preloaded, but lacked concrete byte-wise information. Vincent added this to the preloading mechanism, in addition to MIME-types.

Along the way he refactored the GWT AssetLoader responsible for preloading the assets. Binary files are now directly read into a typed Int8Array, which should improve performance considerably when working with binary files.

He also added switches that control things like caching and so on. Browser caching is now used by default. Another option is to use base64 encoding, but that’s a bit esoteric and unlikely what you want. See the pull request for more information.

Reflection in libgdx’s GWT backend

About a year ago, i finished our GWT backend, which allows you to compile your Java code to straight Javascript. This means no applets, just pure WebGL goodness (except for IE). There were multiple issues that i had to solve:

  • There is no filesystem in the browser. However, libgdx’s Files module exposes just that to you
  • Many of the Java runtime classes are not available or incomplete in GWT
  • WebGL doesn’t map 100% to your plain old OpenGL ES 2.0 APIs. You work with objects instead of integer ids, requiring a kind of mapping
  • No native code support, so our box2D wrapper is useless.
  • GWT does not support reflection

I may cover some of the above in future blog posts. In this post i want to discuss how i solved the problem of reflection in GWT.

Libgdx started out without any reflection code. Then came Nate. He wrote quite a few very handy things, like our JSON parser or scene2d’s UI, among many other things. Nate is in love with reflection, so naturally he uses it whenever the oportunity arises.

This turned out to be a huge problem for the GWT backend. Without reflection, all of these nice things wouldn’t be available in GWT, so i had to come up with a solution. So, how does it work?

The first thing that needed to be done was emulating all those libgdx core classes that use reflection. Emulation in GWT means duplicating the original code, and replacing problematic portitions with GWT specific solutions. Those emulation classes then overwrite the normal Java implementation, via the super source mechanism of GWT.

Once the emulation classes where in place, i started coming up with a reflection API that would be used in those classes instead of the original Java reflection API. The end result is largely based on GWT’s own Type and related classes.

Here’s one original method from the Array class of libgdx:

This class creates a new typed array at runtime, which has a specific element type. The corresponding method in the GWT emulation class uses the custom reflection API instead and looks like this:

I basically went through all the reflection API calls in the original sources, and added them to the reflection API for GWT. As such, the API for the GWT backend is pretty close the original Java API. Here’s a rundown of the classes:

  • Type, equivalent to Java’s Class. Holds constructors, methods and fields for a type along with information on the type itself, e.g. it’s superclass, implemented interfaces, access qualifiers and so on.
  • Constructor, represents a default constructor of a type, without arguments. Our GWT reflection API can only instantiate classes with such a constructor!
  • Field, represents a field of a type. Stores access qualifiers as well as a reference to the type of the field. Provides methods to get and set the field value on an instance of that type.
  • Method, you guessed it, represents the method of a type. Stores a list of parameters, the return type, access qualifiers and provides methods to invoke that method on an instance of the type.
  • ReflectionCache, equivalent to the static methods in the normal Java Class. Provides methods to find the type for a Class or fully qualified class name, instantiate arrays with specific element type and so on.

If you look at the implementations of say the get and set methods of the Field class, you see that they call into ReflectionCache.instance (sometimes they also call into the methods of ReflectionCache, which just delegate to that instance). This is a static field of the ReflectionCache class, and implements the IReflectionCache interface. Here’s how that field is initialized:

And this is where the magic begins. GWT provides a way to hook into its compiler. When the compiler encounters a call to GWT.create(), it checks if it can find a Generator that can provide the Java source for the class in question. This is also known as deferred binding and is used to solve a variety of problems in GWT. I abuse it to generate an implementation of the IReflectionCache interface that:

  • Contains a map from the fully qualified name of a Java class to a corresponding Type instance representing the class and its fields, methods, constructors and so on.
  • Implementations for the IReflectionCache methods that allow us to query Type instances, invoke methods, get and set fields and construct new objects and arrays.

The generation of the source of this implementation is performed in the ReflectionCacheSourceCreator. This class is called by the GWT compiler when the above call is encountered, and asked to basically print the Java source implementation for that interface to a file.

The entire workflow of this class is present in the ReflectionCacheSourceCreator#create() method. After some initial setup, I start by generating the Type lookup map by calling ReflectionCacheSourceCreator#generateLookup().

This method spits out the map as a field for the IReflectionCache implementation, then it tries to gather all the classes that are used in the application. The method ReflectionCacheSourceCreator#gatherTypes() plays a crucial role here, as it decides whether information is generated for a class or not. It will also recursively gather the types found in fields and method return types and parameters!

Once all the classes are gathered, the actual lookup generation is performed. For each class, a method is generated, which creates and instance of Type, and fills it with information on the class (constructor, fields, methods, super classes, etc.). Each of this methods is called c#number, e.g. c0() for the first type, c1() for the next type and so on. At the end of each of these methods, the instantiated Type is put into the map for later lookup. As you can see, the method decides wheter to include a class or not by checking either its package name or its class name. If we want to open up reflection to the public, we’d need to add a way for the user to specify additional classes for which reflection information needs to be generated.

Once all these methods are generated, i generate the constructor of the IReflectionCache implementation. The constructor is responsible for calling all these methods we generated previously, thereby populating the lookup map. You may ask yourself why all those methods are generated in the first place. Couldn’t i just put all that generated code in the constructor? The answer is no, Java methods and constructors can not be larger than 64KB. That limit is easily reached when lots of class information is generated.

Next i generate setter and getter methods for every field as well as a warpper method for every method of every class. Each setter/getter/method is assigned a unique name, e.g. s0 for the first field’s setter, or m10 for the 11th method. These methods basically wrap access to fields and methods, we will call them later in the implementations of the IReflectionCache interface methods we are about to generate. On interesting aspect of this is that i use JSNI for the implementatino of these wrappers. This allows me to set/get protected and private fields, and call protected and private methods. However, there are limitations. If the type of a field is not public, it can not be accessed. If the type of a field is long, it can not be accessed either, as GWT doesn’t support returning longs from JSNI back to Java.

Let’s recap what we did so far:

  • Generated a map field in the IReflectionCache implementation that stores class name to Type instance key/value pairs
  • One method per class that creates a Type instance, fills it with the class’ information, and puts the Type instance into the map for later lookup
  • The constructor of the IReflectionCache implementation tht calls all these methods
  • A setter and getter method for each field of every class, that uses JSNI to access protected and private fields as well.
  • A wrapper method for every method of every class that uses JSNI to access protected and private methods as well.

Note that every Field and Method instance has a field that records the id of its wrapper method(s)!

All this information is now used to implement the actual IReflectionCache methods. For each of the methods of the interface, there is one method in ReflectionCacheSourceGenerator responsible to output its implementation, e.g. the call to getKnownTypesC() generates the IReflectionCache#getKnownTypes() implementation:

The p() method just prints a line to the Java source file the generator is writting. The implementation of the getKnownTypes() method is pretty straight forward. It simply returns the values of the map we fill when the implementation is constructed.

The IReflectionCache#forName() method is equally simple, it just looks up the Type instance by name in the map.

The IReflectionCache#newArray and IReflectionCache#newInstance methods are a little bit more involved to implement. The generated implementations are basically huge if-statements that compare the fully qualified class name of the passed in type to the names of all the types we generated information for previously. That’s nasty, and could be solved differently, but it was easy to get going and also easier to debug than say an index based O(1) lookup… As you can see, we can only create new instances for classes that have a public default constructor. Constructors that are not public or have parameters are not supported!

IReflectionCache also has methods to get and set array elements as well as get an array’s length. I again generted a ton of if-statements and compare qualified class names. Once an if-statement matches, i just have to perform a little bit of casting.

Finally i generate the methods for invoking a method on an object, and getting and setting fields. It’s just basically more of the same, big if-statements, some casting.

Here’s an excerpt of the generted Java source:

Once we are done generating the Java source file for the IReflectionCache implementation, the GWT compiler will take the file and integrate it into the build on-the-fly. The GWT.create() call will then be replaced with new IReflectionCacheGenerated() statement, and our static field is initialized.

Now all the emulated classes call into our generated IReflectionCache class if they need reflection and all is well in GWT land.

As you can see from the above code dump, the way things are done is far from being optimal. I also didn’t go into a lot of the nitty gritty details, otherwise this post would be even more massive. I hope this serves as a good intro the reflection mechanism in the GWT backend.

The reason i’m telling you all this is that we are playing with the thought of implementing a reflection API in libdgx. This would allow folks to use reflection freely in their libgdx games while being still compatible with the GWT backend. It would also allow us to get rid of many of the emulated core classes, reducing code duplication and the potential to miss updates to core classes.