Mixing Java & C++ in one Eclipse project and debugging the crap out of it.

While porting the audio framework i wrote for the onset detection tutorial to C++ i wondered wheter i can mix Java and C++ projects in Eclipse. The goal was it to have both sources in a single Eclipse project, have the C++ code automatically compile and produce a shared library to be used for JNI stuff. I came up with a pretty simple solution using Eclipse, the CDT plugin as well as MinGW on Windows. For Linux it works the same, when you want to do cross plattform JNI stuff you will do a little extra work.

So here’s what we learn in this article:

  • Install and setup MinGW on Windows
  • Install CDT for Eclipse
  • Create a Java project with a simple class that has a single native method
  • Add a CDT C++ nature to the java project so we can create a shared library that contains the implementation of the native method
  • Debug both the Java code as well as the C++ Code in Eclipse
  • Profit!

First of let’s start with installing MinGW. MinGW is a port of the GNU compiler suite for Windows 32-bit (a 64-bit version is available too, albeit a it experimental). You can find a handy installer at http://sourceforge.net/projects/mingw/files/Automated%20MinGW%20Installer/. Once downloaded simply execute it and install MinGW to some directory. When you are asked what components to install make sure you check “g++ compiler” otherwise we’ll only have the C compiler available. Now that MinGW is installed we have to install another little thing called gdb. For some reason the MinGW installer does not offer an option to install the MinGW version of gdb so we have to do that manually. Download the tar.gz file from http://sourceforge.net/projects/mingw/files/GNU%20Source-Level%20Debugger/GDB-7.0.50.20100202/gdb-7.0.50.20100202-mingw32-bin.tar.gz/download. Once downloaded extract the content of the file. You’ll end up with a bin/ and a share/ directory. Copy those into the directory of your MinGW install (for me that was c:/MinGW for example). The last thing we have to do is point the PATH environment variable to the MinGW/bin directory so gcc (the compiler) and gdb (the debugger) are available system wide. If you are not sure how to do this here’s a guide how to do that on Windows 7, it’s nearly the same on Vista and XP. Just add “;C:\Mingw\bin” to the system variable Path. To verify that everything works correctly open up a console window and type in “gcc” and “gdb”. The first call will tell you that there’s no input files, the second will start the prompt of gdb which you can exit via typing q+enter. Now we’re done with MinGW, on to CDT.

CDT is an Eclipse plugin which allows you to create C/C++ projects in Eclipse as well as running produced programs and debugging them. Apart from that it features all those handy features you are used to in Java under Eclipse like code completion, refactoring and so on. To install CDT simply click Help->Install New Software in Eclipse. In the dialog appearing click the Add button and enter cdt in the first field and the address “http://download.eclipse.org/tools/cdt/releases/galileo”. Back in the original dialog select “CDT Main Features” as well as “CDT Optional Features” and press next. Press next and finish a couple more time than grab yourself a cup of coffee and wait. The CDT server is kinda slow today :). After having installed the plugin Eclipse will ask you to restart it, so we do that (we really do! It’s vital!).

Now that CDT is installed we create a new Java Project, i called mine jnidemo. Next we create a new package to place our Java class into, i called mine com.badlogic.jnidemo. Create a new Java class in that package and call it JNIDemo. The implementation of JNIDemo looks like this:

When we execute this we get an error something like “Exception in thread “main” java.lang.UnsatisfiedLinkError: com.badlogic.jnidemo.JNIDemo.add(II)I”. The JVM is kindly telling us that we are stupid and didn’t provide it with the implementation of the native JNIDemo.add() method. That’s what we are going to implement now in C++.

Here comes the magic. In order to mix Java and C++ in a single Eclipse project we have to give our Java Project a C++ nature. This is achied by right-clicking the project in the package explorer, selecting “New” and then “Other”. A new dialog pops up from where we select “C/C++” -> “Convert to a C/C++ Project”.

In the next dialog we select “Shared Library” as the project type and “MinGW GCC” as the tool chain.

After clicking “Finish” you’ll be asked wheter you want to open the C/C++ perspective. Click “Yes” to open it and be amazed by the new C/C++ perspective that CDT offers you! Now we want to add the C++ code that implements our native method. First we have to create a source folder to place our C++ files in. We do this by right clicking the project, select “New” and then “Source Folder” i named mine “jni”.

Now we are ready to generate the C++ header file for our Java class. This is done via the tool “javah” on the command line. Javah is part of the Java JDK that you must have installed already to do something useful in Eclipse. You also need to put the bin/ directory of your JDK in PATH environment variable, as you have done with the MinGW bin/ directory. That makes all the Java JDK tools available on the command line, like javac and so on. Now open a console and change to the jnidemo folder. To generate the header file invoke javah as follows:

Back in eclipse we refresh the project and get a new file in the jni/ folder called JNIDemo.h. From this we copy the function signature to a new file in the same folder we call JNIDemo.cpp. Here’s the code in that file:

Ahhh, C++ how i missed you. Now let’s try to build our shared library that contains this awesome native method. For this we have to click on the hammer icon which is available in the C/C++ perspective (which we still haven’t left, right?). Here’s what we get when we do that:


**** Build of configuration Debug for project jnidemo ****

**** Internal Builder is used for build ****
g++ -O0 -g3 -Wall -c -fmessage-length=0 -ojni\JNIDemo.o ..\jni\JNIDemo.cpp
In file included from ..\jni\JNIDemo.cpp:7:
..\jni\JNIDemo.h:2:17: jni.h: No such file or directory
In file included from ..\jni\JNIDemo.cpp:7:
..\jni\JNIDemo.h:15: error: JNIEXPORT' does not name a type
..\jni\JNIDemo.cpp:9: error:
JNIEXPORT’ does not name a type
Build error occurred, build is stopped
Time consumed: 82 ms.

Well, it seems we are missing an include file, namely “jni.h”. This file is provided by the JDK installation and is located in $JDK_HOME/include. In my case $JDK_HOME is “c:\Program Files (x86)\Java\jdk1.6.0_18\”, go figure out what it is on your machine, i’ll wait here… Welcome back. Now what we have to do is tell the C++ compiler to use this directory when searching for header files. For this we right-click the project and then click on properties. In the dialog click on “C/C++ Builder” in the left tree view and select “Settings” in the same tree view. In the right part of the window click on “GCC C++ Compiler/Directories”. Next add the $JDK_HOME/include directory as well as the $JDK_HOME/include/win32 directory. I hardcoded the $JDK_HOME path for simplicity, here’s how it looks:

Close the dialog and rebuild the project. Here’s the output:

**** Build of configuration Debug for project jnidemo ****

**** Internal Builder is used for build ****
g++ -Ic:\Program Files (x86)\Java\jdk1.6.0_18\include\win32 -Ic:\Program Files (x86)\Java\jdk1.6.0_18\include -O0 -g3 -Wall -c -fmessage-length=0 -ojni\JNIDemo.o ..\jni\JNIDemo.cpp
g++ -shared -olibjnidemo.dll jni\JNIDemo.o
Build complete for project jnidemo
Time consumed: 364 ms.

Awesome! We just build a shared library that contains the implementation of the native method JNIDemo.add(). The shared library is called libjnidemo.dll and located in the “Debug” folder of our project. Now all we have to do is load this library in the JNIDemo class so that Java knows where to find the native library. Here’s the modified code:

In the static block we call System.loadLibrary with the shared libraries name minus the suffix. Let’s start the JNIDemo class and see what it outputs:


java.lang.UnsatisfiedLinkError: no libjnidemo in java.library.path
at java.lang.ClassLoader.loadLibrary(Unknown Source)
at java.lang.Runtime.loadLibrary0(Unknown Source)
at java.lang.System.loadLibrary(Unknown Source)
at com.badlogic.jnidemo.JNIDemo.(JNIDemo.java:7)
Exception in thread “main”

Well, say hello to the dreaded UnsatisfiedLinkError! This will happen if the System.loadLibrary() call failed because it couldn’t find the shared library file. As our shared library is located in the “Debug” folder we have to tell the JVM where to find it. Open the Run configuration of the JNIDemo class and add “-Djava.library.path=Debug/” as a VM argument. When we rerun the program we get this output:


Exception in thread “main” java.lang.UnsatisfiedLinkError: com.badlogic.jnidemo.JNIDemo.add(II)I
at com.badlogic.jnidemo.JNIDemo.add(Native Method)
at com.badlogic.jnidemo.JNIDemo.main(JNIDemo.java:20)

Now that is a bit strange. We again get an UnsatisfiedLinkError but this time it seems that the native method itself can’t be found in the shared library. The problem is that exporting symbols to shared libraries on Windows is kinda fucked up. To fix that we have to pass a flag to the C++ linker that generates the shared library file. For this open the properties dialog of the project and go to “C/C++Builder/Settings” again. In the right hand side of the window select “MinGW C++ Linker/Miscellanious”. Now you see an input box called “Linker Flags”. Put the string “-Wl,–kill-at” in there:

Close the dialog and click the hammer again. Everything should still compile fine. Now let’s run the JNIDemo Java program once more. Here’s the output:


5

Yes, it finally works! Now we have everything working. When you modify your C++ source files all you need to do is rebuilding the project in the C++ view. You can also let CDT automatically rebuild the C++ portion of your code in case you modified a file. I found that to be to time consuming so i don’t turn it on normally.

Now the final piece of the puzzle is debugging. There’s not a lot of information out there how to debug JNI code in Eclipse so i hope this little guide will help you. Let’s start of by placing breakpoints. The first one goes into line 20 of the JNIDemo.java file, the second one into line 11 in the JNIDemo.cpp file. Next we create a new Debug Configuration. For this we have to go to the “Debug configurations” dialog which we can find here:

In the dialog that pops up double click on “C/C++ Attach to Application”. This will create a new C++ debugger configuration that works with our shared library file named libjnidemo.dll.

Next we have to select what debugger to use. Remember how we installed MinGW gdb in the beginning? That’s what we’ll tell the configuration to use now. Click on the Debugger tab and select “MinGW gdb Debugger” from the Debugger drop down list:

Now close the dialog and start the Java class JNIDemo in debug mode. Wait for the debugger to arrive at the break point in line 20. When the debugger got there start the C++ debugger configuration you just created. A dialog will pop up that asks you to what executable to attach the debugger to. Choose “javaw.exe”. Wait for the debugger to load, then resume “Thread 1” of the jnidemo C++ debugger:

Next resume the Java debugger. And tada! The c++ debugger will break at line 11 in JNIDemo.cpp. You can even inspect the parameters. Awesome.

That’s all there is to this. Well, one thing i left out. Up until now, all the changes we did to the C++ properties of our project have been made to it’s Debug configuration. By default CDT will also create a Release configuration. If you want a release ready shared library you have to set the same things in the Release configuration too. To select which configuration is active just click the arrow beside the hammer icon. Also, if you want to make your project cross plattform you’ll have to have seperate configurations for each platform.

That’s it for now, hope that helps some people out there. For Android this approach is not working 100%. You can’t build a shared library this way (well, you can but setting it up is a bit of a pain in the ass) and you also can’t debug. But what you can do is using the features CDT offers you like code completion and so on. Just make sure you set the proper include paths for the C++ compiler in the projects properties and indexer will pick up the header files for code completion.

Newton on Archos

Today i received a nicely written mail by Melanie Hossler from Archos. She asked me to put out a version of Newton that will work on their devices which are basically internet tablets running Android 1.5 and as of lately 1.6. As i don’t have a device to test Newton on i was a bit worried that i might run into trouble. Melanie pointed out that i could download a skin to create a proper AVD. However, all the skin does is removing the standard Android buttons home, back, menu and search and instead of the standard red android look a like you get a frame that looks like an Archos 5. Another difference is that in portrait mode the width is larger than the height of the screen (at least that’s what i deciphered from using the provided skin). So i had to tweak Newton a bit in that respect which was a simple one liner. Another problem are the missing system buttons as mentioned above. Up until now Newton had no exit button as you’d normally use the back or home button to exit it. So i also had to add a button to exit Newton in the main menu, again not really a big deal. After messing around with the skin a bit and getting things to work i uploaded the latest Newton Lite version to appslib without a hazzle. Melanie was kind enough to test it on a real device and confirmed that it worked properly. Newton Lite is now officially Archos approved by Melanie :). If you write apps for Android consider porting them to Archos it’s not all that hard. The site itself needs some more work, some things are still a bit rough.

The full version of Newton is still not ready. Stefanie created another 7 levels so we have a whooping amount of 14 new levels now (not counting the 4 i put out earlier this week to celebrate the 10.000th download). We both just don’t find time to do them levels… If any of you out there want to help out drop us a line. There’s a desktop version of Newton with which you could create levels even more easily than on the device.

On a side note: Tower Raiders, a nice tower defense game albeit a bit weak on the overall performance, just broke into the top ten paid apps on the market. The developer seems to have that achieved by listening to his dedicated player community and has a straight 5 star rating on the market along with more than 1000 downloads already. Congrats to you!

Onset Detection Part 7: Thresholding & Peak picking

In the last article we saw how to reduce a complex spectrum that evolves over time to a simple one dimensional function called the spectral flux. On the way we did some improvements to the spectral flux function like rectifying it and enabling Hamming smoothing of the audio samples. In this article we will talk about a so called threshold function which we derive from the spectral flux function.

That’s a rectified, Hamming smoothed spectral flux function. It’s for an excerpt of the song “Judith” by “A Perfect Circle” and already shows quiet some peaks which we could directly interpret as onsets. However, there’s still some noise in that function, especially when the crashs are hit for example. We can’t do any better filtering, like for example smoothing out the function a bit as this would get rid of a lot of the small peaks. Instead we do something totally simple: we calculate a threshold function which is derived from the spectral flux function. This threshold function is then later used to discard spectral flux values that are noise.

The threshold function is actually pretty simple. Given our spectral flux function we calculate the mean or average value of a window around each spectral flux value. Say we have 3000 spectral flux values, each specifying the spectral flux of a 1024 sample window. 1024 samples at a sampling rate of 44100Hz equal a time span of around 43ms. The window size we use for the threshold function should be derived from some time value, say we want the average spectral flux of a time span of 0.5 seconds. That’s 0.5 / 0.043 = 11 sample windows or 11 spectral flux values. For each spectral flux value we take the 5 samples before it, 5 samples after it and the current value and calculate their average. We thus get for each spectral flux value a single value we call the threshold (for reasons that will become apearant shortly). Here’s some code that does this:

Not a lot of new things in there. First we calculate the spectral flux function as we did in the last article. Based on this spectral flux function we then calculate the threshold function. For each spectral flux value in the ArrayList spectralFlux we take the THRESHOLD_WINDOW_SIZE spectral flux values before and after it and calculate the average. The resulting mean value is then stored in an ArrayList called threshold. Note that we also multiply each threshold value by MULTIPLIER which is set to 1.5 in this example. After we finished calculating all our functions we plot them. The result looks like this:

What we just did is calculating a running average of the spectral flux function. With this we can detect so called outliers. Anything above the threshold function is an outlier and marks an onset of some kind! It should also be clear why the threshold function values are multiplied by a constant > 1. Outliers must be bigger than the average, in this case 1.5 times bigger than the average. This multiplier is an important parameter of our onset detector as it governs the sensitivity. However, when i apply the detector to songs i try to come up with a single value that works for all of them. I actually did that and arrived at the magic 1.5 multiplier used above. It works well for all the samples that you can find in the svn as well as a lot of other songs i tested.

Now let us combine the spectral flux function and the threshold function. Basically we want a prunned spectral flux function that only contains values that are bigger or equal to the threshold function. We extend the above program by the following lines:

The variable prunnedSpectralFlux is just another ArrayList. The loop is pretty simple, we add zero to the prunnedSpectralFlux list of the current spectral flux is less than the corresponding threshold function value or we add the spectrul flux value minus the threshold value at position i. The resulting prunned spectral flux function looks like this:

Awesome, we are nearly finished! All that is left is to search for peaks in this prunned spectral flux. A Peak is a value that is bigger then the next value. That’s all there is to peak detection. We are nearly done. Let’s write the small code for the peak detection producing a peak ArrayList:

And that’s it. Any value > 0 in the ArrayList peaks is an onset or beat now. To calculate the point in time for each peak in peaks we simply take its index and multiply it by the time span that the original sample window takes up. Say we used a sample window of 1024 samples at a sampling rate of 44100Hz then we have the simple forumula time = index * (1024 / 44100). That’s it. Here’s the output:

We are done. Kind off. That’s the most basic onset detector we can write. Let’s review some of it’s properties. As with many systems we have a few parametes we can tweak. The first one is the sample window size. I almost always use 1024. Lowering this will result in a more fine grained spectrum but might not gain you much. The next parameter is wheter to use Hamming smoothing on the samples before doing the FFT on them. I did see a bit of improvement turning that on but it will also cost you some cycles calculating it. This might be of importance if you plan on doing this on a mobile device with not FPU. Every cycle saved is a good cycle there. Next is the threshold window size. This can have a huge impact on the outcome of the detection stage. I stick to a window of 20 spectal flux values in total as in the example above. This is motivated by calculating how big of a time span those 20 values represent. In the case of a sample window 1024 samples at a sampling rate of 44100Hz that’s roughly have a second worth of data. The value worked well on all the genres i tested the simple detector on but your mileage may vary. The last and probably most important parameter is the multiplier for the threshold function. To low and the detector is to sensitive, to high and nothing get’s cought. In my day job i sometimes have to do outlier analysis which is pretty similar to what we do here. Sometimes i use simple threshold based methods and a value between 1.3-1.6 seems to be some sort of magic number when doing statistics based outlier detection relative to some means. I’d say that the threshold multiplier and the threshold window size should be the only parameters you tweak. Forget about the rest and use the defaults of 1024 for sample window size and Hamming smoothing on. Doing a parameter seach for to many parameters is no fun so boiling it down to “only” two makes sense in this case. The defaults i gave you above work reasonably well with the detector we developed in this series. If you do multi-band analysis you might have to tweak them individually for each band.

What’s left is some improvements like doing this complete process not for the whole spectrum but sub-bands so that we can do onset detection for various instruments (sort of, as we know they overlap in frequency range). Another improvement is to use overlapping sample windows, say by 50%. This smooths out the Fourier transform a bit more and gives better results. One could also try to do dynamic range compression on the whole audio signal before passing it to the FFT and so on and so forth. The code in the framework allows you to experiment easily with various additional steps. You will also want to maybe clean up the resulting peak list and remove any peaks that are to close together, say < 10ms. Learning by doing is the big mantra. Now go out and write awesome music games! Here's a couple of papers and links i found very useful in the process of getting this to work: http://www.dspguide.com/ An awesome online book (also available as hard cover) called “The Scientist and Engineer’s Guide to Digital Signal Processing” featuring anything you’ll ever want to know about digital signal processing (as an engineer :)).
https://ccrma.stanford.edu/~jos/dft/ Another online book called “Mathematics of the discrete Fourier Transform with audio applications” coping with anything related to the discrete Fourier Transform. Pretty heavy on math but a very good read.
http://old.lam.jussieu.fr/src/Membres/Daudet/Publications_files/Onset_Tutorial.pdf A very good scientific paper called “A tutorial on onset detection for musical signals”. Features various onset detection functions (we used the un-banded spectral flux here) as well as some data on the performance of various functions on a standardized data set (Mirex 2006 Onset Detection Challenge). Look out for any other papers by Bello, he’s a big player in the field it seems.
http://www.dafx.ca/proceedings/papers/p_133.pdf “Onset detection revisited” by Dixon, another big name in the field (at least it seems so from the citation counts, but we all know how much they really tell you…). Very good paper, must read.

If you check out the citation graph of the two papers mentioned on google schoolar you will many more papers talking about onset detection that might interest you. Google *is* your friend in this case. Always read your fine papers!