Playing videos in libgdx, the dirty way

Any community contributions to libgdx go here! Some may get included in the core API when permission is granted.

Playing videos in libgdx, the dirty way

Postby DieJay » Sun Feb 01, 2015 1:23 am

I desesperately wanted to be able to play videos with libgdx, so instead of waiting for an implementation of video codecs which may never come, I did the next best thing; created my own method.

I didn't write a universal video decoder. Screw that. What I did was basically used VLC to create a sequence of still images, rip the audio into an OGG file, made a script to pack the stills into a single file, feed it to libgdx and synchronize the audio with the corresponding stills.

I know, I know, it's dirty, it's stupid and it's a lot of work for little benefit. But hey! It works!

Here's the Java classes I made. First the generator for the stills;

Code: Select all
package com.melloland.videoforgdx;

import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.nio.ByteBuffer;
import java.nio.file.Files;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;

public class VideoGenerator
{
    public static void create(String input, String output, String videoFileName, float fps)
    {
        int i;
       
        // Lists files in directory and assign audio and video files
        // Searches for files with regex patterns
        File directory = new File(input);
        File[] files = directory.listFiles();
        ArrayList<File> videoFiles = new ArrayList<File>();
       
        for (i = 0; i < files.length; i++)
        {
            String fileName = files[i].getName();
            if (fileName.matches(videoFileName))
            {
                videoFiles.add(files[i]);
            }
        }
       
        // sorts the video files by name
        Collections.sort(videoFiles, new Comparator<File>()
        {
            public int compare(File fileA, File fileB)
            {
                return fileA.getName().compareTo(fileB.getName());
            }
        });
       
        // read byte arrays of audio and video files
        ArrayList<byte[]> videoFilesBytes = new ArrayList<byte[]>();
        try
        {
            for (i = 0; i < videoFiles.size(); i++)
            {
                videoFilesBytes.add(Files.readAllBytes(videoFiles.get(i).toPath()));
            }
        }
        catch (IOException e)
        {
            e.printStackTrace();
        }
       
        // creates header
        // header consists of frame rate, number of video files and each video file's position and size
        int headerSize = Float.BYTES + Integer.BYTES + (videoFilesBytes.size() * Integer.BYTES * 2);
        ByteBuffer headerBuffer = ByteBuffer.allocate(headerSize);
        int headerPosition = 0;
        headerBuffer.putFloat(fps);
        headerBuffer.putInt(videoFilesBytes.size());
        for (i = 0; i < videoFilesBytes.size(); i++)
        {
            headerBuffer.putInt(headerSize + headerPosition);
            headerBuffer.putInt(videoFilesBytes.get(i).length);
            headerPosition += videoFilesBytes.get(i).length;
        }
       
        // outputs the data to new file
        OutputStream outputStream = null;
        try
        {
            outputStream = new BufferedOutputStream(new FileOutputStream(output));
            outputStream.write(headerBuffer.array());
            for (i = 0; i < videoFilesBytes.size(); i++)
            {
                outputStream.write(videoFilesBytes.get(i));
            }
        }
        catch (FileNotFoundException e)
        {
            e.printStackTrace();
        }
        catch (IOException e)
        {
            e.printStackTrace();
        }
        finally
        {
            try
            {
                outputStream.close();
            }
            catch (IOException e)
            {
                e.printStackTrace();
            }
        }
    }
}


Then the class that puts it all back together.

Code: Select all
package com.melloland.videoforgdx;

import java.nio.ByteBuffer;
import java.util.ArrayList;

import com.badlogic.gdx.audio.Music;
import com.badlogic.gdx.files.FileHandle;
import com.badlogic.gdx.graphics.Pixmap;
import com.badlogic.gdx.graphics.Pixmap.Format;
import com.badlogic.gdx.graphics.Texture;
import com.badlogic.gdx.graphics.Texture.TextureFilter;

public class VideoLoader
{
    private float fps;
    private long milliDelay;
    private Music audio;
    private ArrayList<byte[]> videoDataArray;
    private Pixmap pixmap = null;
    private Texture texture = null;
    public long lastTime = 0;
   
    public VideoLoader(Music audio, FileHandle video)
    {
        this.audio = audio;
       
        int i;
       
        ByteBuffer buffer = ByteBuffer.wrap(video.readBytes());
       
        fps = buffer.getFloat();
        milliDelay = (long)(1000 / fps);
        int videoCount = buffer.getInt();
        int[] videoPositions = new int[videoCount];
        int[] videoSizes = new int[videoCount];
        for (i = 0; i < videoCount; i++)
        {
            videoPositions[i] = buffer.getInt();
            videoSizes[i] = buffer.getInt();
        }
       
        videoDataArray = new ArrayList<byte[]>();
        for (i = 0; i < videoCount; i++)
        {
            byte[] videoData = new byte[videoSizes[i]];
            buffer.get(videoData);
            videoDataArray.add(videoData);
        }
       
        texture = new Texture(2048, 2048, Format.RGB888);
        texture.setFilter(TextureFilter.Nearest, TextureFilter.Nearest);
    }
   
    public long getMilliDelay()
    {
        return milliDelay;
    }
   
    public Texture getFrame()
    {
        return getFrameAtTime(audio.getPosition());
    }
   
    public Texture getFrame(int frame)
    {
        long currentTime = System.currentTimeMillis();
        if (currentTime - lastTime >= getMilliDelay())
        {
            if (videoDataArray.get(frame).length > 0)
            {
                pixmap = new Pixmap(videoDataArray.get(frame), 0, videoDataArray.get(frame).length);
                texture.draw(pixmap, 0, 0);
                pixmap.dispose();
                pixmap = null;
            }
            lastTime = currentTime;
        }
       
        return texture;
    }
   
    public Texture getFrameAtTime(float seconds)
    {
        return getFrame((int)(fps * seconds));
    }
   
    public void play()
    {
        audio.play();
    }
   
    public void stop()
    {
        audio.stop();
    }
}


You first call VideoGenerator.create() in a separate script (just create a new class with a main() function and run it) with the parameters input (path to the directory where your stills are), output (the path to the file that will be created), videoFileName (the name of your stills as a regex pattern, ex. "scene[0-9]+\\.jpeg") and fps (the frame rate, ex. 30). Once you have your file, put it in your assets folder along with the audio file. Then in your libgdx project, load with the assets manager the audio as a Music object and the video as a FileHandle object (I assume you know how to do that, there are tutorials out there about it). When the files are loaded, create a VideoLoader object with your Music and FileHandle as parameters and call the play() function. On each frame, you can call the getFrame() function which will return a Texture object corresponding with the current position of the Music object.

That's pretty much the best I could come up with. What do you think?

Edit: Also, as per the requirements for posting in the contributions section, I hereby release this code under an Apache 2.0 license (http://www.apache.org/licenses/LICENSE-2.0).
Last edited by DieJay on Sun Feb 01, 2015 7:08 pm, edited 3 times in total.
DieJay
 
Posts: 9
Joined: Wed Oct 01, 2014 2:43 pm

Re: Playing videos in libgdx, the dirty way

Postby DieJay » Sun Feb 01, 2015 1:48 am

By the way, be careful if you run this. Currently the memory goes through the roof even though I dispose the texture on every frame. Still looking into it.

Edit: Nevermind, I fixed it in the code above! I just had to set the pixmap and texture to null before reassigning them.
DieJay
 
Posts: 9
Joined: Wed Oct 01, 2014 2:43 pm

Re: Playing videos in libgdx, the dirty way

Postby chutia » Sun Feb 01, 2015 2:10 am

cool hack nice work! 8-) Im sure it will come in handy for people who want to add videos into their apps.
chutia
 
Posts: 171
Joined: Fri Jun 06, 2014 7:01 am

Re: Playing videos in libgdx, the dirty way

Postby DieJay » Sun Feb 01, 2015 2:19 am

Thanks! :)

I found that it's painfully slow on android though, probably due to the fact a new pixmap object must be generated on each frame. Runs flawlessly on PC though. There must be a way to improve it. I guess generating all the textures before starting the video would do, but it would be a memory hog, not to mention the time it would take before the video is ready. Maybe if I tinker with opengl?
DieJay
 
Posts: 9
Joined: Wed Oct 01, 2014 2:43 pm

Re: Playing videos in libgdx, the dirty way

Postby Magnesus » Sun Feb 01, 2015 8:05 am

On Android you could probably use WebKit to view the video in a html5 container. If it is supported...
Magnesus
 
Posts: 1620
Joined: Sun Sep 25, 2011 3:50 pm

Re: Playing videos in libgdx, the dirty way

Postby DieJay » Sun Feb 01, 2015 7:05 pm

Yeah, if it's supported. That's the problem, each platform has its own way of doing things.

What I propose is for libgdx to have its own video file format, which could basically just be a packaging of an mp3 or ogg file with a sequence of still jpegs. That kind of format would be easily readable on every platform that can play standard music and image files. I would do it myself (tried it too) but you can't to my knowledge create Music object with a byte array, and as I proved it, feeding the jpeg data into Pixmap objects and drawing them on a Texture is lamentably slow on Android devices.
DieJay
 
Posts: 9
Joined: Wed Oct 01, 2014 2:43 pm

Re: Playing videos in libgdx, the dirty way

Postby slebed » Sun Feb 01, 2015 11:11 pm

I've been able to play h.264 clips that I generated that on Android. The only issue was that there is a particular flag that needed to be set in the mp4 file in order for it to play correctly. I can't find the links at the moment, but I originally just googled for them. I haven't tried vp8 yet, but it should work very well on Android considering its targeted to 4.3+
slebed
 
Posts: 204
Joined: Fri Dec 28, 2012 3:29 am

Re: Playing videos in libgdx, the dirty way

Postby xoppa » Mon Feb 02, 2015 1:16 pm

xoppa
 
Posts: 689
Joined: Thu Aug 23, 2012 11:27 pm

Re: Playing videos in libgdx, the dirty way

Postby DieJay » Mon Feb 02, 2015 5:27 pm

So you mean there's an extension in development?! Well I'll be damned.

Can it be used already? I played around with the source a little but couldn't figure how to make it run properly.
DieJay
 
Posts: 9
Joined: Wed Oct 01, 2014 2:43 pm

Re: Playing videos in libgdx, the dirty way

Postby aznphatb0i » Thu Jul 28, 2016 2:32 am

I am trying to run a video in my libgdx project on intel cherry trail stick PC's which have the Z8300 processor and Windows with 2GB RAM

The same project runs fine on Android or on desktop PC with I3, but on the Z8300 sticks it has a low frame rate and stutters.

I would love to try with VLCJ instead, do you have any other tips like how you got VLCJ to do hardware decode?
aznphatb0i
 
Posts: 8
Joined: Wed Jul 08, 2015 7:57 pm

Next

Return to Libgdx Contributions

Who is online

Users browsing this forum: No registered users and 1 guest