Michael Sheldon's gstreamer Stuff

Michael Sheldon (mike at mikeasoft dot com)

June 23, 2012

Writing native Tizen applications using the 1.0 (Larkspur) SDK Scratchbox and EFL
Mike @ 3:29 pm

Thanks to Daniel Juyung Seo this post is also available in Korean.

Overview

Tizen has the capacity for native applications written using the Enlightenment Foundation Libraries (EFL), however in the 1.0 SDK release this functionality isn’t particularly well exposed. Instead the main focus is currently on HTML5/JavaScript based applications, which is fine, but it’s not appropriate for all applications. As a way to get to grips with Tizen in more detail I decided to try porting one of my existing applications, Eyrie, from MeeGo Harmattan (as used by the Nokia N9) to Tizen.

Eyrie is one such application which clearly isn’t especially suited to being written as a web app. It requires the ability to get raw audio samples from the microphone and perform some signal processing on them to extract acoustic fingerprints. As far as I can see the current Tizen Web API doesn’t really have the capacity to handle audio input at such a low level, and even if it did making use of it for Eyrie would involve completely rewriting the EchoPrint codegen library in JavaScript to handle the fingerprinting.

However by making use of EFL I was able to get the bulk of Eyrie ported to Tizen in less than 24 hours. Thanks to both MeeGo and Tizen using GStreamer as their multimedia framework and the EchoPrint codegen library being easy enough to compile for both platforms all that was really needed was a new UI using EFL instead of QML. Below you can see a video of the initial port running on the Tizen emulator (I don’t have a real device to play with at the moment), the UI is still a little rough around the edges but all the main functionality is working.


Video of the Eyrie music identifier’s initial port to EFL on Tizen

In this tutorial we’ll work through the creation, building, packaging and deployment of a very simple EFL application, which I hope will give people a handle on a potential workflow for native application development using the 1.0 SDK release.

Scratchbox

Although it’s not made obvious, the SDK actually contains a full scratchbox environment which the IDE drops into when build packages. While it might potentially be possible to write EFL apps purely using the IDE I found it easier to ignore this and jump into scratchbox directly, this then frees you up to make use of whatever editor/IDE you’re most comfortable with. If you are looking to work just within the IDE you might find the Tizen Platform Developer’s Guide a good starting point.

I’ll be assuming that you installed the Tizen SDK to ~/tizen_sdk/

Note: After installing the SDK it’s necessary to open the IDE and create (and possibly compile) a platform project. Simply select a new platform project and then hit compile. This sets up the scratchbox targets the first time around. You only need to do this once after installing the SDK.

Before we enter the scratchbox environment we should find out what targets are available to us. We can do this by running:

~/tizen_sdk/SDK/build-system/toolchains/scratchbox2/bin/sb2-config -l

With the 1.0 SDK release this produces the following output:

tizen-device-1.0.sb2_gcc45sbox2.armel.cdeb
tizen-device-1.0.sb2_gcc45sbox2.armel.platform
tizen-emulator-1.0.sb2_gcc45sbox2.i386.platform

Since I’m working with the emulator in x86 mode I’ll be using the tizen-emulator-1.0.sb2_gcc45sbox2.i386.platform target, to compile packages for real devices you’ll need to replace this with one of the armel targets in the following commands. I’m unclear on the intended separation between the two armel targets, but I expect either should be usable for our purposes.

To access the scratchbox as your normal user you can simply run:

~/tizen_sdk/SDK/build-system/toolchains/scratchbox2/bin/sb2 -t tizen-emulator-1.0.sb2_gcc45sbox2.i386.platform

This is perfectly usable for building and editing packages, and you should find that your home directory is available within the scratchbox. However the rest of the environment will be mounted read-only, preventing you from installing new packages.

To start a session in which it’s possible to install packages we can run:

~/tizen_sdk/SDK/build-system/toolchains/scratchbox2/bin/sb2 -e -R -t tizen-emulator-1.0.sb2_gcc45sbox2.i386.platform

This mode is only really suitable for installing packages and isn’t usable for general development as most compilation tools aren’t available.

We want to try writing a simple EFL application using the Elementary toolkit, so to start with we’re going to need to install the Elementary development package:

apt-get install libelm-dev

As you can probably tell from the above scratchbox is a Debian based environment, so most of the normal Debian tools are available to us. So for example if we didn’t know the exact name of the Elementary package we need we could simply run:

apt-cache search elementary

And we’d get a list of likely looking candidates to choose from. As a side note the Eclipse based IDE does have a simple package management interface which can be found on the default toolbar, however this appears to lacks any way of searching for packages, but if you are trying to do everything through the IDE without dropping into scratchbox you might find it helpful.

Elementary

So now that we have a compilation environment capable of targeting both the emulator and real devices we can start writing our applications. For this tutorial we’ll just create a very simple Elementary application that displays a label:

#include <Elementary.h>
        
static void on_quit(void *data, Evas_Object *obj, void *event_info) {
        elm_exit();
}

EAPI int elm_main(int argc, char *argv[]) {
        Evas_Object *win, *bg, *box, *lbl;

        /* Create our window */
        win = elm_win_add(NULL, "eflexample", ELM_WIN_BASIC);
        elm_win_title_set(win, "Tizen EFL Example");
        evas_object_smart_callback_add(win, "delete,request", on_quit, NULL);

        /* Give it a background */
        bg = elm_bg_add(win);
        evas_object_size_hint_weight_set(bg, 0.0, 0.0);
        elm_win_resize_object_add(win, bg);
        evas_object_show(bg);

        /* Create a box to place our label inside */
        box = elm_box_add(win);
        elm_box_horizontal_set(box, EINA_FALSE);
        evas_object_size_hint_weight_set(box, EVAS_HINT_EXPAND, 0.0);
        evas_object_size_hint_align_set(box, EVAS_HINT_FILL, 0.0);
        elm_win_resize_object_add(win, box);
        evas_object_show(box);

        /* Make a label */
        lbl = elm_label_add(win);
        evas_object_size_hint_weight_set(lbl, EVAS_HINT_EXPAND, EVAS_HINT_EXPAND);
        elm_object_part_text_set(lbl, NULL, "Hello Tizen!");
        elm_box_pack_end(box, lbl);
        evas_object_show(lbl);

        /* Display the window */
        evas_object_show(win);

        /* Start the Elementary main loop */
        elm_run();

        return 0;
}

ELM_MAIN();

I’ll assume we’re working in the directory ~/src/tizen/eflexample/, so we can save this as ~/src/tizen/eflexample/eflexample.c

More comprehensive documentation on Elementary can be found on the Enlightenment Wiki.

We’ll keep the build process very simple for this example and just have a small Makefile handle compilation. For more complex projects both autotools and cmake are available.

CFLAGS=`pkg-config --cflags elementary`
LDFLAGS=`pkg-config --libs elementary`

eflexample:
	$(CC) $(CFLAGS) eflexample.c -o eflexample $(LDFLAGS)

all: eflexample

clean:
	rm -f eflexample

install: all
	mkdir -p $(DESTDIR)/opt/share/applications
	cp com.mikeasoft.eflexample.desktop $(DESTDIR)/opt/share/applications/
	mkdir -p $(DESTDIR)/opt/com.mikeasoft.eflexample
	cp eflexample $(DESTDIR)/opt/com.mikeasoft.eflexample/
	cp icon.png $(DESTDIR)/opt/com.mikeasoft.eflexample/

This gets saved as ~/src/tizen/eflexample/Makefile

By including the $(DESTDIR) variable in our install paths we allow the Debian packaging we’re about to write to override the install destination and place it inside a temporary directory prior to being packed into a .deb.

If we run make in the ~/src/tizen/eflexample directory whilst inside our scratchbox session it should then compile successfully.

Launcher

You might notice in the earlier Makefile a couple of files that we haven’t yet discussed, com.mikeasoft.eflexample.desktop and icon.png. These are going to be used for adding our application to Tizen’s launcher. The .desktop file gets installed to /opt/share/applications/ alongside all other Tizen .desktop launchers, and should look something like:

Name=EFL Example
Type=Application
Exec=/opt/com.mikeasoft.eflexample/eflexample
Icon=/opt/com.mikeasoft.eflexample/icon.png
Comment=An example of how to write native EFL applications for Tizen
Version=0.1
X-TIZEN-TaskManage=True
X-TIZEN-Multiple=False
X-TIZEN-Removable=True

This is then saved to ~/src/tizen/eflexample/com.mikeasoft.eflexample.desktop

You’ll probably notice that there’s a selection of Tizen specific entries. The first, X-TIZEN-TaskManage, controls whether or not our application will appear in the Tizen task manager. The second, X-TIZEN-Multiple, tells the task manager if it is allowed to launch more than one copy of our application. If this is set to false the task manager will instead try to restore a currently running version of our application to the foreground. Because our simple example doesn’t perform the extra necessary work to allow Tizen to restore it to the foreground the task manager will instead simply kill our application when clicked a second time. Finally the last setting, X-TIZEN-Removable, specifies whether or not our application can be uninstalled through the application manager.

For the icon file we’ll simply use the Enlightenment logo:

Enlightenment Logo

Which needs to be saved to ~/src/tizen/eflexample/icon.png

Packaging

At this stage we could potentially just copy our eflexample binary to the device and start it via the command line, but a much more complete and redistributable solution is to construct a Debian package for it.

First we create our control file, which describes our package and its various dependencies:

Source: com.mikeasoft.eflexample
Section: user/other
Priority: extra
Maintainer: Michael Sheldon <elleo@gnu.org>
Build-Depends: debhelper (>= 7), libelm-dev
Standards-Version: 3.8.4
Homepage: http://blog.mikeasoft.com

Package: com.mikeasoft.eflexample
Depends: ${shlibs:Depends}, ${misc:Depends}
Description: An example of creating EFL applications for Tizen
Architecture: any

This gets placed in ~/src/tizen/eflexample/debian/control

Because our build system is fairly standard and handles the DESTDIR parameter being passed to it we can use debhelper to automate all of our rules file for us:

#!/usr/bin/make -f
%:
	dh $@

This is then saved to ~/src/tizen/eflexample/debian/rules

Then we have our changelog:

com.mikeasoft.eflexample (0.1.0-1) unstable; urgency=low

  * Initial release

 -- Michael Sheldon <elleo@gnu.org>  Sat, 23 Jun 2012 04:32:27 +0100

Which is saved to ~/src/tizen/eflexample/debian/changelog

Finally we can specify which version of debhelper we’re compatible with by running:

echo 7 > ~/src/tizen/eflexample/debian/compat

Now we’re ready to actually build our package. To do this we need to be inside our scratchbox session as a normal user, which we achieve by running:

~/tizen_sdk/SDK/build-system/toolchains/scratchbox2/bin/sb2 -t tizen-emulator-1.0.sb2_gcc45sbox2.i386.platform

Then we can begin building our package:

cd ~/src/tizen/eflexample
dpkg-buildpackage -rfakeroot

After our application has been successfully compiled and packaged it’ll be saved in the parent of our current directory (i.e. ~/src/tizen/).

Deploying

Now that we’ve built a package for our application we can deploy it using the sdb tool provided with the SDK. This should work with either the emulator or a connected device. First we transfer our Debian package to the device and place it in the /tmp/ directory, then we run dpkg on the device to install our new package. If we had used any additional dependencies that aren’t already installed by default we’d need to use apt-get to install them prior to installing our own package. However since libelm is part of the default Tizen system all we need to do is:

~/tizen_sdk/SDK/sdb/sdb push ~/src/tizen/com.mikeasoft.eflexample_0.1.0-1_i386.deb /tmp/ && ~/tizen_sdk/SDK/sdb/sdb shell dpkg -i /tmp/com.mikeasoft.eflexample_0.1.0-1_i386.deb

We should now see a new icon in the Tizen application launcher for our example app, which we can simply click to start.



Source

The complete source for this tutorial can be download here: eflexample.tar.gz.


June 18, 2010

Jokosher on the Nokia n900
Mike @ 1:05 am

I’ve had another stab at getting Jokosher running on the Nokia n900 and I’m getting much closer to something actually usable now, as this screenshot attests:

Jokosher on the Nokia n900

There’s still a number of issues that need resolving before it’s really ready for use (most notably some playback/recording problems and some dialog boxes that are too large for the screen), but it’s getting there. When it’s working fully it could make the n900 a very useful device for portable podcasting, allowing users to record, edit, mix, encode and upload their roaming shows with nothing more than their phone.


June 17, 2010

GStreamer OpenCV plugins on the Nokia n900
Mike @ 3:49 pm

A while back I wrote a few GStreamer plugins that expose OpenCV functionality as GStreamer elements (source code), I haven’t had much time to work on these recently myself, but thankfully a number of other folks have started contributing now. Yesterday Daniil Ivanov kindly packaged gst-opencv for the maemo extras-devel repository, and the n900 performs surprisingly well considering how CPU intensive many of the vision operations performed are.

This first video shows edge detection being performed from the n900’s main camera (whilst simultaneously being encoded):

Example gst-launch line: gst-launch v4l2camsrc device=/dev/video0 ! video/x-raw-yuv,width=480,height=272 ! videorate ! video/x-raw-yuv,framerate=12/1 ! ffmpegcolorspace ! edgedetect ! ffmpegcolorspace ! xvimagesink

This second video shows the faceblur element in action, it detects any faces in the current scene and blurs them out, the frame rate and resolution on this one had to be reduced somewhat due to the complexity of the operation, it looks clearer when performed directly to an xvimagesink rather than attempting to encode at the same time.

Example gst-launch line: gst-launch v4l2camsrc device=/dev/video0 ! video/x-raw-yuv,width=240,height=136 ! videorate ! video/x-raw-yuv,width=240,height=136,framerate=6/1 ! videoscale ! video/x-raw-yuv,width=120,height=68 ! ffmpegcolorspace ! faceblur profile=/home/user/haarcascade_frontalface_default.xml ! ffmpegcolorspace ! xvimagesink

For some more examples of the gst-opencv plugins in action on a normal desktop machine take a look at thiagoss’ blog post and a couple of videos by Alexandre Poltorak (edge detection and face blurring).


June 2, 2010

Jokosher Summer of Code Projects
Mike @ 4:07 pm

This year we’re lucky enough to have three students working on Jokosher as part of the Google Summer of Code, two under GNOME and one under GStreamer.

Andi Miller

Andi is looking into making it possible to do collaborative editing tasks between multiple Jokosher instances, between Jokosher and Pitivi and potentially between Jokosher and a small remote control (so you can use your phone/MID to start Jokosher recording/playing without needing to be sat at your PC). The project is progressing well with some Jokosher information and events already being exposed via a dbus interface.

Pēteris Krišjānis

PÄ“teris is working on finishing up some old work to provide telepathy support within Jokosher and then further extending this to add support for telepathy tubes. This will make it very easy for users to record VoIP sessions within Jokosher and the tubes support will also tie-in with the collaborative editing project, allowing Andi to send dbus messages via telepathy connections to remote users. The most interesting use case from my perspective is for podcasters working over VoIP, in the scenario where both participants are using Jokosher it should be possible for them to carry out a standard VoIP call and then afterwards have the two Jokosher instances automatically synchronise a high quality recording of each participant’s side of the conversation; so while the VoIP call quality might not be perfect the final audio will sound as if they’re both in a studio together.

David Williams

David is attempting to add musical score editing support to Jokosher, allowing people to sketch out musical ideas that can be played back as MIDI instruments alongside normal recorded audio. He’s already made some good progress in creating a python GStreamer element that can output some simple MusicXML based on an internal model (which can then be rendered to MIDI via the musicxml2midi element), this will then later be connected to a score editing UI, with the potential for multiple interface types (traditional scores, guitar tablature, drum events, etc.).

So hopefully by the end of the summer we’ll not only have a number of very exciting new features but also three more core Jokosher developers :).


March 19, 2010

GStreamer MusicXML2MIDI Release 0.1
Mike @ 3:55 pm

A few months back I started writing a GStreamer element for converting MusicXML into MIDI, the eventual goal of this from my perspective is to allow for score editing inside Jokosher (without Jokosher having to deal with all the pain of the MIDI format itself). It’s far from being perfect and still has trouble with more complicated files, but hopefully it’ll be of use to some people in its current state so I’m releasing version 0.1.

Example Uses

  • Synthesising MusicXML directly:
    gst-launch filesrc location=song.xml ! musicxml2midi ! wildmidi ! audioconvert ! autoaudiosink
  • Generating a MIDI file:
    gst-launch filesrc location=song.xml ! musicxml2midi ! filesink location=song.mid
  • Generating an OGG Vorbis file:
    gst-launch filesrc location=song.xml ! musicxml2midi ! wildmidi ! audioconvert ! vorbisenc ! oggmux ! filesink location=song.ogg

Running the OGG Vorbis pipeline on the twovoices.xml test file produces the following output:

Download

MusicXML2MIDI 0.1 – Source Package
MusicXML2MIDI 0.1 – Debian/Ubuntu Package for 32-bit systems
MusicXML2MIDI 0.1 – Debian/Ubuntu Package for 64-bit systems

Alternatively there’s a PPA available: https://launchpad.net/~gst-musicxml2midi/+archive/ppa/

Contribute

All the source code is stored in a git repository: http://github.com/Elleo/gst-musicxml2midi. If you’d like to help out simply clone the repository and start hacking away, once you’re happy with your changes you can propose your branch for merging with my own.


Powered by WordPress