Pekka Paalanen

Broken connection to DiNovo bluetooth device: a solution

2013-02-03T22:59:00.000+02:00

I have a Logitech DiNovo Mini (combined keyboard & touchpad), which at first worked just fine on my Gentoo laptop, Asus G50V, using the laptop's built-in bluetooth adapter, Bluez major version 4 (4.101-r5 today), and manual connection. Then I tried to connect the DiNovo to other devices, both without and with the USB-bluetooth-dongle that came with the DiNovo. Then I wanted it back to my laptop. There was a time when it worked only if I temporarily removed the battery from DiNovo. In the end, after several weeks if not months, it did not work anymore, at all. Blindly poking around, I now found how to fix it.

The DiNovo showed a green light, saying it got a connection, but on the laptop, all I could see was the device appearing and very soon disappearing in udev (confirmed with udevadm monitor). I tried to pair it again, many times, and while the pairing seemed to succeed, the device just did not work. I also installed blueman, which indicated the same: when I touched the DiNovo, it connected, and was immediately disconnected. In the system log I got:

bluetoothd[280]: Refusing input device connect: Operation already in progress (114)
bluetoothd[280]: Refusing connection from ##:##:##:##:##:##: setup in progress
(bluetoothd:280): GLib-WARNING **: Invalid file descriptor.

Ok, so is it trying to connect twice for some odd reason? Where is the state kept? Could I manually fix it?

Apparently, I could. In the /var/lib/bluetooth/*/ directory I saw several files that seemed to be about the bluetooth settings on my Gentoo. Not knowing anything about how Bluez works, I looked at the files there, to see if I could spot something suspicious. Luckily they were all plain-enough text files, so I did spot something.

The file /var/lib/bluetooth/*/spd had two lines with my DiNovo's device address on it. The first line was a long one, the second line short. Not knowing what I'm doing, I stopped the bluetooth service, removed the short line, and restarted the bluetooth service. Like magic, the DiNovo started working again, connecting automatically. No errors in the system log anymore, either.

I have not used the DiNovo much after the repair yet, remains to be seen if I broke anything, but so far so good. Apparently when I was playing around with the DiNovo, somehow that file got a second entry for the same device, and caused malfunction. Is it my fault or a bug, I do not know. Googling did not give any helpful hints on solving this, so I am recording this note here, hoping it helps someone.

-- A note, barely readable, scratched with a broken SD-card on the wall of a passageway in the huge, monster crawling dungeon they call the Intternets.

On supporting Wayland GL clients and proprietary embedded platforms

2012-11-21T15:58:00.000+02:00

How would one start implementing support for graphics hardware accelerated Wayland clients on an embedded platform that only has proprietary drivers?

This is a question I have answered more than once recently. Presumably you already have some ways to implement a Wayland compositor, some APIs you can use to composite and get images on screen. You may have wl_shm based clients already working, and now you want hardware rendered clients to work. Where to start?

First I will explain the architecture a bit. There are basically three things related to Wayland graphics:

the client side support for graphics hardware acceleration
the compositor's support for hardware accelerated clients
the compositor's own rendering or compositing, and output to screen

Usually the graphics hardware accelerated applications (clients) use EGL for the window system glue, and GL ES (2) for rendering. The client side is the EGL Wayland platform, plus wayland-egl API. The EGL Wayland platform means, that you can pass Wayland types as EGL native types, for instance a struct wl_display * as the EGLNativeDisplayType parameter of the eglGetDisplay() function.

The compositor's support for hardware accelerated clients is the server side of Wayland-enabled libEGL. Normally it consists of the EGL_WL_bind_wayland_display extension. For a compositor programmer, this extension allows you to create an EGLImageKHR object from a struct wl_buffer *, and then bind that as a GL texture, so you can composite it.

The compositor's own rendering mechanisms are largely irrelevant to the client support. The only requirement is, that the compositor can effectively use the kinds of buffers the clients send to it. If you turn a wl_buffer object via EGLImageKHR into a GL texture, you would better be compositing with a GL API, naturally. Apart from that, it does not matter what APIs the compositor uses for compositing and displaying.

Now, what do we actually need for supporting hardware accelerated clients?

First, forget about using wl_shm buffers, they are not suitable for hardware accelerated clients. Buffers that GPUs render into are often badly suited for CPU usage, or not directly accessible by the CPU at all. Due to GPU requirements, you likely cannot make a GPU to render into an shm buffer, either. Therefore to get the pixel data into an shm buffer you would need to do a copy, like glReadPixels(). Then you send the shm buffer to the server, and the server needs to copy the pixels again to make them accessible to the GPU for compositing, e.g. by calling glTexImage2D(). That is two copies between CPU and GPU domains, and that is slow. I would say unusably slow. It is far better to not move the pixels into CPU domain at all, and avoid all copying.

Therefore, the most important thing is graphics buffer sharing or passing. Buffer sharing works by creating a handle for a buffer, and passing that handle to another process which then uses the handle to make the GPU access again the same buffer. On your graphics platform, find out:

Do such handles exist at all?
How do you create a buffer and a handle?
How do you direct GL ES rendering into that buffer?
What is the handle? Does it contain integers, open file descriptors, or opaque pointers? Integers and file descriptors are not a problem, but you cannot pass (host virtual) pointers from one process to another.
How do you create something usable, like an EGLImageKHR or a GL texture, from the handle?

It would be good to test that the buffer passing actually works, too.

Once you know what the handle is, and whether clients can allocate their own buffers (preferred), or must the compositor hand out buffers to clients for some obscure security reasons, you can think about how to use the Wayland protocol to pass buffers around. You must invent a new Wayland protocol extension. The extension should allow a client to create a wl_buffer object from the handle. All the standard Wayland interfaces deal with wl_buffer objects, and the server will detect the type of each wl_buffer object when used by a client. Examples of the protocol extension are wl_drm of Mesa, and my experimental android_wlegl.

I recommend you do the first implementation of the protocol extension completely ad hoc. Hack the server to work with your buffer types, and write a custom client that directly uses the protocol without any fancy API like wayland-egl. Once you confirm it works, you can design the real implementation, whether it should be in a wrapper library around the proprietary libEGL or something else.

EGL is the standard interface to implement accelerated Wayland client support and it conveniently hides the details from both servers and clients, but it is not the only way. If you control both server and client code, you can use any other API, or create your own. That is a big if, though.

The key point is buffer sharing, copying will kill your system performance. After you know how to share graphics buffers, your work has only begun. Good luck!

Wayland on Android: upgrade to 4.0.4 and new build integration

2012-09-24T13:46:00.000+03:00

We at Collabora have been working on a new Android build system integration with autotools projects, still based on Androgenizer (git). Now we have our own repo manifest repository, and a tool called anagrman for managing optional feature packages (aggregates). Wayland on Android is one feature package, and the first to become available. We also upgraded to Ice Cream Sandwhich 4.0.4_r2.1. Instead of a snapshot release, this particular announcement is about live branches.

Weston (git) was upgraded to upstream as of Sept. 18th, 2012, though there are no user visible changes on Android. This brings the 0.95 protocol, and the new evdev input rework from upstream, which went through some changes since the 4.0.1_r1.2-b Wayland on Android release. Weston now has its GLESv2 renderer separated in code, and shaders are faster and simpler (I have not done any shader benchmarks myself).

Libxkbcommon lost its final dependencies to kbproto and xproto, and our Android build files are upstream. Thanks to Daniel Stone, we can use libxkbcommon straight from upstream.

The new Android build system integration requires you to manually download only anagrman in addition to Android repo. There is no make wayland-aggregate-configure step anymore, all generated Android makefiles are created during the full build. Also androgenizer and wayland-scanner are built automatically as needed. All this is possible using the makefile update feature of GNU Make. If there is a rule that can update a makefile, GNU Make will update the makefile as needed. If any makefiles were updated, Make will then start from scratch, reading in all makefiles again, before continuing to the actual build phase. This causes Make to reload all Android makefiles 2-3 times during the first build. It should also solve any dependency issues of the explicit configure step, like when one project's configure depends on another project's fully built library. A big thank you to Helio Chissini de Castro for doing most of the build system work.

This work is available in two ways:

The source code you can build yourself, see README. Use the module wayland where anagrman is invoked.
A snapshot image you can flash into a Samsung Galaxy Nexus: wayland-on-android-20120924.zip (105 MB) (sha1: b75cb4e325bc7a2960464128ae7f569c21741471). You can flash it with the command: $ fastboot update wayland-on-android-20120924.zip

The ready-made image is configured to launch Weston at boot instead of SurfaceFlinger (the setting is in device/samsung/maguro/system.prop in the source tree). The source code, however, does not start Weston nor SurfaceFlinger automatically. You have to use the commands # adb root and # adb shell to log into the phone, and run one of:

# setprop service.compositor surfaceflinger
# setprop service.compositor weston

You can also run Weston by just # weston & and start other demo clients manually. Weston will automatically start simple-touch finger drawing demo, and the power button will cause Weston to power off the phone. The available demo apps are: simple-touch, simple-shm, flower, and clickdot. Any GL based demos are not included, since the EGL Wayland platform for clients is still unimplemented.

Even though this is a live release, i.e. not tagged to a specific revision, I do not expect much changes in the near future. We are now researching other ways to enable Wayland on Android and other embedded-like devices.

Wayland on Android snapshot release: input

2012-07-13T15:40:00.000+03:00

It is time to announce the android-4.0.1_r1.2-b snapshot release of the Wayland on Android project at Collabora! We give you: input support in Weston and a finger-painting demo!

Collabora will have people at GUADEC demoing this on real devices, though not me personally.

Click to see the video!

This release provides ports of the following projects (git repositories, really) to Android 4.0.1 on Samsung Galaxy Nexus:

wayland
weston
pixman
mtdev
cairo
libxkbcommon, and its dependencies xproto, kbproto, and xkeyboard-config
libffi
mouse cursors

It also includes some changes to Android internals, and the aggregate for building it all.

This is just a snapshot release of a work in progress, and you cannot do much with it. Everything an end user would have known about Android is still gone.

In Weston, the three device buttons are working, and the touchscreen is working. Unfortunately, the only application really supporting touch devices is simple-touch, but I turned that into a demo that is automatically launched. If you install this release into a Galaxy Nexus device, it will boot into Weston and you can play with simple-touch. The power button is hooked up in Weston to power off the device immediately, so a computer is not necessary to show and exit the demo.

The main advancement compared to my previous posts is that the touchscreen is fully working now. Also, this time I am providing a proper release:

an image you can flash into your device, if you have the right tools: android-4.0.1_r1.2-b.tar.gz (106 MB) (sha1: 1bd52cef8b53574452b4e2feac76c5191e815884)
instructions for building a full Android OS with Weston: README

You can get the fastboot tool needed for flashing the images from the Android SDK, I think. I have never used the SDK myself, I have always gone with the full AOSP tree.

Please, if you try this on your device, let me know how it went. If you find problems that I can fix, I might push the fixes to the android-4.0.1_r1.2-b release branches, and update the ChangeLog for this release, but I will not provide new images. Before August I probably won't react, though.

If you look at the histories of the git branches mentioned towards the top, you will find many ugly hacky commits. All commits marked as HACK will be replaced by the proper changes during the course of this project. We are planning to send almost all changes to respective upstream projects, too. The input enablement patch series in Weston needs a rewrite, before it gets upstream.

Thanks to the whole Android team at Collabora for making this happen!

Forwarding evdev devices to Android

2012-07-06T17:16:00.002+03:00

I'm working on adding input support for Weston's Android backend, and to test a normal keyboard and a mouse, I needed a way to get those as evdev devices on Android. I don't have any Bluetooth devices here, so I started hacking up evdev forwarding from a laptop PC. I could not find much information on anyone doing this before.

First I tried the various forwarding types of adb other than TCP. None of them seemed to work at all. Therefore I needed to use TCP, since that worked. Thanks to a co-worker at Collabora, I found out that busybox has a netcat tool I could use on Android. Laptop of course has the original netcat. Nethid is a quick & dirty program for creating a fake evdev device node using uinput, so I fixed that to work on Android.

First on the host, set up TCP forwarding (port number just made up):

$ adb forward tcp:7776 tcp:7776

On the Android device, start uinput:

$ busybox nc -l -p 7776 | uinput-pipe

Then on the host start feeding input events:

$ cat /dev/input/event10 | nc localhost 7776

And Weston now finds my forwarded evdev input device and receives input from it!

But there are some minor problems with this. Something in this pipe spaghetti is caching data, so input events arrive in bursts, not as soon as possible. Also, something is eating over 99% of the REL_X events, so it is quite impossible to move the cursor horizontally. REL_Y events seem go through fine. Strange...

Anyway, this fills its purpose: I was able to test that pointer devices work on my Weston android backend, and I also got mouse cursors to show up after some fixes. The video below is the proof.

Wayland on Android: intermediate report

2012-06-19T15:06:00.001+03:00

We at Collabora had an internal meeting, where I presented the Wayland on Android project and its current state. You can find the slides by clicking the picture below.

Note: epdfview may get the colors wrong, at least Evince shows it right.

Some links related to the slides:

Weston on Android: desktop-shell

2012-05-24T10:40:00.000+03:00

Last week I got weston-desktop-shell working on Android, along with some toytoolkit clients. This means I have Cairo androgenized, and it can even render text. I did have trouble with Cairo's configure script, so the build lacks all thread-safety. For the curious, all sources can be found via my wayland-aggregate. Right now I'm working on Android's libEGL, trying to add Wayland support.

Galaxy Nexus sideways on a table, running Weston and desktop-shell.

Wayland anti-FUD

2012-05-11T10:16:00.000+03:00

I was replying to an email, and got side-tracked into writing some Wayland anti-FUD. There are lots of myths about Wayland out there, so I thought to better make it into a blog post.

This post is about the very small overhead of a Wayland (system) compositor, and why Wayland over network will be much better than X-over-ssh.

I predict that on desktops and other systems that may have accounts for more than one person, there will actually be two Wayland compositors stacked. There is a system compositor at the bottom, handling fast user switching, replacing VT switching, etc., and then a session compositor that actually provides the desktop environment. This is not my idea, it has been written in the Wayland FAQ under "Is Wayland replacing the X server?" for a long time.

My point is: Wayland compositors will not make 3D games suck because of compositing. While explaining why, I also continue to explaining why network transparency will not suck either. Now, do not mix up these things, I am not claiming that remoting 3D games over network will magically become feasible.

The overhead of adding a system compositor in the Wayland stack will be very small. A system compositor normally does not do any real work for compositing, it only takes the buffer handle from a client compositor, and flips it onto the screen. No rendering and no image copying involved in the system compositor.

It is the same with a full-screen game vs. any Wayland compositor: the compositor will not do any real work. A game renders its image into a buffer, passes the buffer handle to the compositor, and the compositor tells the hardware to scan out the buffer. No extra copying, no extra rendering.

The overhead that will appear with adding a system compositor, is relaying input events and buffer flips. The amount of data is small, and at least buffer flips will happen at most once per vertical refresh per monitor. There is also the idea of relaying input events only once per frame. This means that CPU process context switches will increase only by few per frame, when adding another compositor in the stack. Ideally the increase is 2 per frame: a switch to system compositor, system compositor handles input and output, and a switch back.

The overhead can be this small, because the protocol has been designed to avoid round-trips. A round-trip means that one process is waiting for another to reply before it can continue. The protocol also favors batching: accumulate a bunch of data, and then send as a batch. Both of these principles minimize the number CPU process context switches.

Because of these design principles, no Wayland developer is worried about the performance of a possible network transparency layer. Minimizing CPU context switches translates directly to minimizing the effect of network latency. Some believe, that even a simple Wayland network transport which practically just relays the Wayland protocol messages as is, and adds transferring of buffer data, will clearly outperform the traditional X-over-ssh.

Now, if you still claim that X-over-ssh would be better, you a) underestimate the effect of latency, and b) forget that modern applications do not send small rendering commands through the X protocol like 10-20 years ago. Modern applications render their content client-side, and send images to the X server. Wayland simply makes images the only way to send content to the server, allowing to drop the whole rendering machinery from the server and avoiding a huge amount of protocol.

First light from Weston on Android

2012-04-27T15:33:00.000+03:00

A couple of months ago, Collabora assigned me first to research and then make a proof of concept port of Wayland on Android. I had never even seen an Android before. Yesterday, Weston on Android achieved first light!

Galaxy Nexus running Weston and simple-shm.

That is a Samsung Galaxy Nexus smart phone, running a self-built image of Android 4.0.1. Weston is driving the screen, where you see the simple-shm Wayland client. There is no desktop nor wallpaper, because right now, simple-shm is the only ported client.

How is that possible? Android has no DRI, no DRM, no KMS (the DRM API), no GBM, no Mesa, and for this device the graphics drivers are proprietary and I do not have access to the closed driver source.

Fortunately, Android's self-invented graphics stack has pretty similar requirements to Weston. All it took was to write a new Android specific backend for Weston, that interfaces to the Android APIs. Writing it took roughly three days.

And the rest of the two months? I spent some time in studying Android's graphics stack, but the majority of the time sunk into porting the minimum required library dependencies, libwayland, Weston, and simple-shm to the Android platform and build environment. Simply getting the Android build system to build stuff properly took a huge effort, and then I got to write workarounds to features missing in Android's C library (Bionic). Features, that we have taken for granted on standard Linux operating systems for years. I also had to completely remove signal handling and timers from libwayland, because signalfd and timerfd interfaces do not exist in Bionic. Those need to be reinvented still.

Android has gralloc and fb hardware abstraction layer (HAL) APIs. Hardware vendors are required to implement these APIs, and provide EGL and GL support libraries. These implementations are usually closed and proprietary. On top of these is the Android wrapper-libEGL, written in C++, open source. My first thought was to use the gralloc and fb HAL APIs directly, but turned out that the wrapper-libEGL does not support using them in the application side. Instead, I was forced to use some Android C++ API (there is no C API for this, as far as I can tell) to get access to the framebuffer in an EGL-compatible way. In the end, I had to write a lot less code than using the HALs directly.

The Android backend for Weston so far only provides basic graphics output to the screen, and offers (presumably) accelerated GLES2 via EGL for the server. No input devices are hooked up yet, so you cannot interact with Weston. I do not know how to get pageflip completion events (if possible?), so that is hacked over.

Simple-shm is the only client that runs for now. There is no support for EGL/GL in Wayland clients. Toytoolkit clients are waiting for Cairo and dependencies to be ported.

The framebuffer can be used by one program at a time. Normally that program is SurfaceFlinger, the Android system compositor. To be able to run Weston, I have to kill SurfaceFlinger and make sure it stays down. Killing SurfaceFlinger also kills the whole Android UI infrastructure. You cannot even power off the phone by pressing or holding down the physical power button!

A video about simple-shm running on Galaxy Nexus:

The sources with Android build integration and other hacks can be found here:

The wayland_aggregate is how I actually connect to the Android build system. Building is not trivial, and you cannot simply do a checkout and start compiling. You have to get the right Android tree for your device, add my local manifest (which still points to repositories on my hard drive, i.e., won't work for you), download and extract binary blobs, and whatnot. Be warned, I will rebase the above branches.

This is the beginning of pushing a Wayland stack into Android. Next I need to clean up, send stuff upstream, add input support, find out about that pageflip, reinvent signal handling and timerfd, and then move on to the second major task: supporting Wayland GL clients. I hope it is possible to implement the Wayland platform in the wrapper-libEGL.

This work is sponsored by Collabora, Ltd, and I also thank my fellow collaborans for guidance through the vast jungles of Android.

Improved appearance for the simplest Wayland client

2012-04-27T10:17:00.000+03:00

Of the Wayland demo clients in the Weston repository, simple-shm is the simplest. All the related code is in that one file, and it interfaces directly with libwayland. It does not use GL or EGL, so it can be ran on systems where the EGL stack does not support the Wayland platform nor extensions. However, what it renders, is surprising:

The original simple-shm client on a Weston desktop.

The square with apparently garbage texture is the original simple-shm. To any graphics developer, who does not know any better, that immediately looks like something is wrong with the image stride somewhere in the graphics stack. That really is what it was supposed to look like, not a bug.

I decided to propose a different rendering, that would not look so much like a bug, and had some real diagnostic value.

The proposed appearance of simple-shm, the way it is supposed to look like.

The new appearance has some vertical bars moving from left to right, some horizontal bars moving upwards, and some circles that shrink into the center. With these, you can actually see if there is a stride bug somewhere, or non-uniform scaling. There is one more diagnostic feature.

This is how the proposed simple-shm looks like when the X-channel is mistaken as alpha.

Simple-shm uses XRGB buffers. If the compositor does not properly ignore the X-channel, and uses it as alpha, you will see a cross over the image. Depending on whether the compositor repaints what is below simple-shm or not, the cross will either saturate to white or show the background through. It is best to have a bright background picture to clearly see it.

I do hope no-one gets hypnotized by the animation. ;-)

What does EGL do in the Wayland stack

2012-03-10T11:51:00.001+02:00

Recently I drew some diagrams of how an EGL library relates to the Wayland stack. Here I am presenting the Mesa EGL version of them with the details explained.

Mesa EGL with Wayland, and simplified X as comparison.

X11 part

The X11 part of the diagram is very much simplified. It completely ignores indirect rendering, DRI1, details of DRI2, and others. It only shows, that a direct rendering X11 EGL application uses the X11 protocol to create an X11 window, and the Mesa EGL X11 platform uses the DRI2 protocol in some way to communicate with the X server. Naturally the application also uses one of the OpenGL interfaces. The X server has hardware or platform specific drivers that are generally referred to as DDX. On the Linux DRI stack, these call into libdrm and the various driver specific sub-libraries. In the end they use the kernel DRM services, like kernel mode setting (KMS). All this in the diagram is just for comparison with a Wayland stack.

Wayland server

The Wayland server in the diagram is Weston with the DRM backend. The server does its rendering using GL ES 2, which it initialises by calling EGL. Since the server runs on "bare KMS", it uses the EGL DRM platform, which could really be called as the GBM platform, since it relies on the Mesa GBM interface. Mesa GBM is an abstraction of the graphics driver specific buffer management APIs (for instance the various libdrm_* libraries), implemented internally by calling into the Mesa GPU drivers.

Mesa GBM provides graphics memory buffers to Weston. Weston then uses EGL calls to bind them into GL objects, and renders into them with GL ES 2. A rendered buffer is shown on an output (monitor) by queuing a page flip via the libdrm KMS API.

If the EGL implementation offers the extension EGL_WL_bind_wayland_display, Weston will use it to register its wl_display object (facing the clients) to EGL. In practice, the Mesa EGL then adds a new global Wayland object to the wl_display. That object (or interface) is called wl_drm, and the server will automatically advertise that to all clients. Clients will use wl_drm for DRM authentication, getting the right DRM device node, and sharing graphics buffers with the server without copying pixels.

Wayland client

A Wayland client, naturally, connects to a Wayland server, and gets the main Wayland protocol object wl_display. The client creates a window, which is a Wayland object of type wl_surface. All what follows is enabled by the Wayland platform support in Mesa EGL.

The client passes the wl_display object to eglGetDisplay() and receives an EGLDisplay to be used with EGL calls. Then comes the trick that is denoted by the double-arrowed blue line from Wayland client to Mesa EGL in the diagram. The client calls the wayland-egl API (implemented in Mesa) function wl_egl_window_create() to get the native window handle. Normally you would just use the "real" native window object wl_surface (or an X11 Window if you were using X). The native window handle is used to create the EGLSurface EGL handle. Wayland has this extra step and the wayland-egl API because a wl_surface carries no information of its size. When the EGL library allocates buffers, it needs to know the size, and wayland-egl API is the only way to tell that.

Once EGL Wayland platform knows the size, it can allocate a graphics buffer by calling the Mesa GPU driver. Then this graphics buffer needs to be mapped into a Wayland protocol object wl_buffer. A wl_buffer object is created by sending a request through the wl_drm interface carrying the name of the (DRM) graphics buffer. In the server side, wl_drm requests are handled in the Mesa EGL library, where the corresponding server side part of the wl_buffer object is created. In the diagram this is shown as the blue dotted arrow from EGL Wayland platform to itself. Now, whenever the wl_buffer object is referenced in the Wayland protocol, the server knows exactly what it is.

The client now has an EGLSurface ready, and renders into it by using one of the GL APIs or OpenVG offered by Mesa. Finally, the client calls eglSwapBuffers() to show the result in its Wayland window.

The buffer swap in Mesa EGL Wayland platform uses the Wayland core protocol and sends an attach request to the wl_surface, with the wl_buffer as an argument. This is the blue dotted arrow from EGL Wayland platform to Wayland server.

Weston itself processes the attach request. It knows the buffer is not a shm buffer, so it passes the wl_buffer object to the Mesa EGL library in an eglCreateImageKHR() function call. In return Weston gets an EGLImage handle, which is then turned into a 2D texture, and used in drawing the surface (window). This operation is enabled by EGL_WL_bind_wayland_display extension.

Summary

The important facts, that should be apparent in the diagram, are:

There are two different EGL platforms in play: one for the server, and one for the clients.
A Wayland server does not contain any graphics hardware or driver specific code, it is all in the generic libraries of DRM, EGL and GL (libdrm and Mesa).
Everything about wl_drm is an implementation detail internal to the EGL library in use.

The system dependent part of Weston is the backend, which somehow must be able to drive the outputs. The new abstractions in Mesa (GBM API) make it completely hardware agnostic on standard Linux systems. Therefore every Wayland server implementation does not need its own set of graphics drivers, like X does.

It is also worth to note, that 3D graphics on X uses very much the same drivers as Wayland. However, due to the Wayland requirements from the EGL framework (extensions, EGL Wayland platform), proprietary driver stacks need to specifically implement Wayland support, or they need to be wrapped into a meta-EGL-library, that glues Wayland support on top. Proprietary drivers also need to provide a way to use accelerated graphics without X, for a Wayland server to run without X beneath. Therefore the desktop proprietary drivers like Nvidia's have a long way to go, as currently nvidia does not implement EGL at all, no support for Wayland, and no support for running without X, or even setting a video mode without X.

Due to the way wl_drm is totally encapsulated into Mesa EGL and how the interfaces are defined for the EGL Wayland platform and the EGL extension, another EGL implementor can choose their very own way of sharing graphics buffers instead of using wl_drm.

There are already plans to change to some of the architecture described in this article, so it is possible that details in the diagram become outdated fairly soon. This article also does not consider a purely software rendered Wayland stack, which certainly would lift all these requirements, but quite likely be too slow in practice for the desktop.

See also: the authoritative description of the Wayland architecture

Wayland R&D at Collabora

2012-02-02T14:57:00.000+02:00

While being contracted by Collabora, I started a Wayland R&D project in October 2011 with the primary goal of getting to know Wayland, and strengthening Wayland expertise in Collabora. During the four months I started the wl_shell_surface protocol for desktops, added screen locking, ported an X screensaver to Wayland with new protocol, and most recently implemented surface transformations in Weston (the reference compositor, originally the wayland-demos compositor). All this sponsored by Collabora.

The project started by getting wayland-demos running under X, and then looking into the bugs I hit. To rule out problems in hardware GL renderer, I also got the demos running with softpipe and llvmpipe. Trying to fix segmentation faults and other obvious problems was my stepping stone into the Wayland code base.

My first real piece of work was screen locking. That included adding special protocol for it, having a way to have privileged Wayland clients, implementing locking in the shell plugin in the compositor, and writing an unlock dialog for the desktop-shell client. Those are the obvious parts. I also had to extend the shell plugin interface, find a way to hide surfaces so they do not render while the screen is locked, and of course bug hunting and patch set rebasing and rewriting, before screen locking landed upstream.

Next was porting an X screensaver as a regular Wayland client. Once that worked, I extended the protocol by adding a screensaver interface, and made the shell plugin automatically start the screensaver application. Handling screensavers would have been a walk in the park, except I needed shell-specific data to be attached to all surfaces. I wrote a hacky solution, but in the end, Kristian Høgsberg wanted me to add a whole new interface into the shell protocol for this. It became the wl_shell_surface interface, and all demo clients needed to adopt it. Yet that was not all. Since we are used to have per-monitor screensavers, I needed my screensaver to set different instances for each monitor. Hence I had to add output event callbacks in the toytoolkit.

A cleanup phase came next, I took Valgrind and ran. I fixed a pile of memory leaks and wrote missing destructor functions all over, in compositor, clients and the toytoolkit, at the same time collecting a Valgrind suppressions list to ease Valgrinding in the future. This work included adding some ad hoc way of cleanly exiting demo clients.

In January there were some discussions on maximised and full-screen surfaces, what they are and how they should be implemented. Surface scaling was raised as one point. Weston already had the zoom effect, and full-screen scaling would be another surface transformation, so I decided to write a transformation matrix stack for supporting any number of simultaneous transformations. It turned out to be a three week task.

Implementing surface transformations required changes all over Weston. First, I needed a way to invert the transformation which is a 4-by-4 matrix. After searching in vain for a MIT-licenced C implementation I wrote one myself, based on LU-decomposition. I believe LU-decomposition is more efficient on a 4x4 matrix than the cofactor method. Along the inversion routines, I wrote a unit test application for testing the speed and precision of the inversion. Detecting and dealing with non-invertible transformations is also important.

Going through the transformation stack every time you need to transform a point might be costly, so I added a cached total transform and its inverse. Implementing input redirection was a simple matter of applying the inverse total transform to pointer coordinates. Needing a way to test transformations, I added a Weston key binding for rotating surfaces, and modified an existing demo application to mark the clicked point. Adding functions for explicitly converting between display global coordinates and surface local coordinates (surface local are the only ones a client knows of) clarified some of the coordinate computations.

Surface painting and damage region tracking needed fixes, too. Previously, a zoomed surface was repainted as a whole, and it forced a full display redraw, i.e. damaging the whole display. Transformed surface repaint needed to start honoring the repaint regions, so we could avoid excessive repainting. Damage and repaint regions are tracked as global coordinate axis aligned rectangles. Whenever a transformed surface is damaged (requires repainting), we need to compute the bounding box for the damage instead of simply using the global x, y of the top-left corner and the surface width, height. Then during surface painting, we take the list of damage rectangles, and render only those. Surface local coordinates (texture coordinates) are computed via the inverse transformation. This method may result in sampling outside of a surface's buffer (texture), so those samples need to be discarded in the fragment shader.

Other things that needed fixing after the surface transformations were window move and resize. Before fixing, moving a surface would not follow the pointer but move in the surface local orientation. Resize needed the same orientation fix, and another fix in relative surface motion that a client can set in the surface's attach request.

What you mostly see as the result of the surface transformations work is, that you can rotate any normal window, no application support needed. The pointer position on screen, over a window, accurately corresponds to what the application receives as the local pointer location. I did not realise it at the time, but this input redirection working flawlessly became an appreciated feature. Apparently it is hard or impossible to do in X, I would not know. In Wayland, and for me, it was just another relatively easy bug to be fixed. The window rotation feature was meant purely for debugging surface transformations.

Two rotated windows and some flowers.

There are still further issues to be fixed with surface transformations. Relative surfaces, like pop-up windows and menus, are not transformed and appear at a wrong location. Pointer cursors are not transformed; you would want the text bar cursor to be aligned with the text orientation. Continuously resizing a transformed window from its (locally) top-left corner makes the window drift away. We are probably still damaging larger regions than absolutely necessary for repaints. Repaint optimisation of opaque surfaces does not work with transformations.

During all this work of four months there were also the usual bug hunts, enhancements and fixes all over. For example, decorationless EGL apps, which turned out to have been a bug in Cairo, and moving the configuration file parser into a helper library that is shared between clients and the compositor.

Now, I am done with the Wayland R&D project and moving into another project at Collabora. In the new project I will continue working on Wayland, Weston, and the demos.

Nokia N9 Music Player and Album Cover Art

2012-01-15T16:54:00.003+02:00

I recently got a Nokia N9 phone. One of the first things I did was copy my music collection into it. Since the player shows also album cover images, if such are stored, I started adding them -- not by embedding them into ID3v2 tags but as separate files, to avoid useless copies of images.

Usually it is as simple as putting a cover.jpg file into a directory, that contains a single album. Sometimes and in some cases, though, that does not work. I found out, that the N9's default music player is supposed to follow Media Art Storage specification. That gave me hints.

If a directory contains more than one album, you can name the cover image files according to the album, for example 'Back in Black.jpg' and 'Flick of the Switch.jpg', as long as the names correspond the ID3 tag album name (somehow?).

My real problem case was a directory full of songs downloaded from Nectarine. I edited them all (EasyTAG is a wonderful tool) to make the ID3 album tag "Nectarine" because I wanted to have them all under the same "album", and there are over 50 songs in that single directory. Simply adding a cover.jpg or Nectarine.jpg did not work.

There are two possible reasons that I found. First, the directory contains too many files, according to the Media Art Storage spec. Second, apparently the cover art is not taken into use, unless at least one song file, which would use that cover art, is touched (modification date updated).

I created a new directory, moved one Nectarine song into it, and put Nectarine.jpg there, too. And it started to work, for all my Nectarine songs.

There is software called Tracker in the N9, which maintains some sort of database of all media. Also album cover art gets used via Tracker. If you ssh into your phone, and move around your media files, Tracker update is not automatically triggered. ~~You could use the command tracker-control -r to force a full rebuild when you launch e.g. the music player the next time, but the rebuild will take a long time.~~ An easy way to force a faster rebuild is to plug the N9 into a computer via USB, and then unplug it.

Wayland screensaver integration

2011-12-09T12:12:00.000+02:00

Continuing on the Wayland screensaver track, I sent a branch for review. The screensaver interface is now fully implemented in both the demo compositor and the demo screensaver. Screenshots below...

The compositor shell plugin of desktop-shell now implements the screensaver interface. This allows a client to register a surface as an idle animation for a given output (monitor). These surfaces remain hidden until the compositor's idle timer triggers, and the compositor fades to black. If there are any screensaver surfaces, the compositor will fade them in, showing the idle animation. The compositor can also be configured to automatically start a screensaver client.

While an idle animation is running, if the shell implements screen locking, the unlock dialog will appear on the first input event, for instance when moving the mouse. The idle animation continues as the background for the unlock screen.

There is another idle timeout running with the idle animation. When that timeout triggers, the compositor fades to black and will seize updating the screen. This also causes properly written animating clients to stop rendering, and we can hit zero CPU usage, even when there is a screensaver active. The compositor will wake from this sleep as usual, and fade in either the desktop directly, or the unlock dialog with the animation in the background.

On returning to the normal desktop, the compositor (the shell plugin, really) will kill the screensaver client if it started it in the first place.

The demo implementation also supports multiple outputs, which is convenient to demonstrate on X. The three Wayland compositor windows are the outputs of a single demo compositor running.

Normal desktop, spread over three outputs, with a few flower clients and a terminal.

The idle animations running on each output with separate state. There is only one screensaver client running.

Idle animation as the background for the unlock screen.

A Wayland screensaver

2011-11-22T15:38:00.000+02:00

Now that screen locking is done in Wayland demos, it is time to go for the eye-candy: full-screen idle animations, also known as screensavers. The first step was to port an existing screensaver to Wayland. I chose glmatrix from XScreenSaver, because it is cool, and it renders with OpenGL. This way I did not have to port Xlib based rendering to Cairo (yay!).

Here is GLMatrix running as a regular, windowed application on Wayland, using the toytoolkit:

GLMatrix on the Wayland demo compositor.

On Wayland, screensavers can be reduced to pure animation applications, while the compositor handles everything about locking. Next, we need a Wayland protocol extension to actually use this idle-animation in a screensaver'y way.

GLMatrix is already in the Wayland demo repository as a client called wscreensaver, and it requires cairo-gl, just like gears does.

Screen locking in Wayland

2011-11-21T17:09:00.000+02:00

This is continuation to my Wayland desktop-shell post.

My goal was to implement a simple screen locking feature, a similar idea to what xlockmore does for X. In Wayland it is much simpler and more reliable to implement than in X, because the implementation will be in the display server (compositor). While the "lock" itself is in the compositor, also an unlock dialog is required. The unlock dialog usually asks the user to input his password, but I settled for "click the green ball". Screenshots below...

First a protocol (commit) is needed to drive the compositor locking and unlocking, since the unlock dialog is exported to the desktop-shell client. When the compositor hits the idle timeout, it fades out to black, and then locks itself in shell plugin. The compositor is woken up by input events, and sends prepare_lock_surface event to desktop-shell. The client replies with set_lock_surface request, with the unlock dialog's surface as an argument. Only on getting the surface, the compositor fades in, to have a nice transition to the dialog. The dialog then runs like any other application on screen, and when the user has dismissed it, desktop-shell sends unlock request to the compositor. On unlock, compositor brings all windows (surfaces) back to the desktop.

The shell plugin implements screen locking by stealing all the surfaces from the compositor's rendering list. Only the background surface and pointer cursor surfaces are left. This has the side-effect that none of the stolen (hidden) surfaces can be activated nor receive input. The compositor-side surface objects still continue living as usual. New surfaces can be created and they are automatically hidden. Output assignment of the hidden surfaces is set to NULL, which prevents sending any frame events for them, effectively also stopping any animations that might have been running. On unlock, the surfaces are simply put back into the compositor's list, and assigned to outputs.

After the last commit in the screen locking series, you can enjoy automatic screen locking in the Wayland demo compositor:

Normal desktop.

Locked, with the unlock dialog.

Note, that locking does not imply a fancy animated screensaver. The black screen is the screensaver ;-)

Thanks to Kristian Høgsberg for his reviews, comments and bug fixes.

This feature is sponsored by Collabora, Ltd.

Wayland misconceptions: Window

2011-11-21T11:26:00.000+02:00

There are many misconceptions about Wayland, and I want to try to correct one. Let's start with the statement:

There is no object in the Wayland protocol that corresponds to Window in X.

Surprised? We need to take a step back to explain what that really means, and I will do it with the help of an example of a complex application: Firefox.

Take a Firefox instance, one window with several tabs open. The active tab contains a flash video running.

Notice, what I called as a "window". It is the window from the end user's perspective, it is "the Firefox" on screen. If there was a text terminal window open, too, that would be another "window". Let's call "the Firefox" an application window.

In X (I imagine, but I do not know if this is really the case) the application window would be an instance of the Xlib data type Window. This Window contains lots of child Windows, for example: every button could be a separate Window. Every scrollable text box could be a separate Window, that is only partially visible. Every tab is probably a Window, too. The hierarchy, the relationships, and event masks are all sent to the X server, so the X server can dispatch input events to the right Windows, and render all those Windows on screen.

Wayland is nothing like that.

Notice how I said the X server renders stuff? A Wayland compositor does not render that stuff. Wayland is not a rendering API.

In Wayland, the whole application window is a wl_surface. Period. There are no child surfaces (except perhaps menus, but those are an exception, because they can extend beyond the application window). All rendering must happen client-side. The application will acquire a new wl_buffer, render everything into it, including even the video image and window decorations, and then do a single attach request to get the updated application window contents on screen, in one go, without flicker. Yes, that really must be done for every video frame playing (you can cache the non-changing graphics client side to avoid full redraws).

The simple statement, that wl_surface corresponds to a window, is roughly correct, when one means an application window. There is nothing like an Xlib Window, except if you invent one in your toolkit, and then it will be specific to your toolkit, and never transmitted over the Wayland protocol.

Wayland desktop-shell

2011-11-18T17:05:00.001+02:00

When I started on Wayland, it already had a fade to black type of screensaver, which triggered after an idle timeout. Actually, one should talk about wayland-demos and the demo compositor, as Wayland itself is just a protocol and a C-library implementing the protocol.

My plan was to add a locking feature. People usually like to lock their desktop when they leave, to prevent unauthorized access. Locking a desktop means that all windows should be hidden and interaction with any windows should be disabled. In X, implementing a lock requires lots and lots of tricks, and even then, you are not absolutely guaranteed, that it really works. In Wayland on the other hand, where locking can be implemented in the display server which also manages all windows and input, you can be sure the lock holds.

The Wayland demo compositor consists of the core, backend plugins, and a shell plugin. The backend plugins handle interfacing with the system: getting images out and input events in. There are three backends: x11 for X, drm for running on DRM/KMS without X, and wayland for running the compositor inside another wayland compositor. The shell plugin is where all the excitement happens with respect to locking.

The shell plugin has a client counterpart, called desktop-shell. Between these two is a special interface, also called desktop-shell. The desktop-shell interface is an extension to the Wayland core protocol. Desktop-shell client is responsible for drawing the background and the panel.

Not every client should be able to lock the screen at will, or unlock it. Therefore I made the desktop-shell interface "privileged", which means that only a known client can bind to it. The easiest way to have a known or authenticated client is to have the compositor execute the client directly, and set up a secure communication channel to it, allowing only this client to bind to the interface. Having the whole interface protected, it is easy to define an unlock request that cannot be abused.

As a result, desktop-shell client is nowadays started automatically by the demo compositor. Also, desktop-shell cannot be started manually anymore because it would not be privileged for the compositor. (commit)

Later, Kristian Høgsberg added a configuration file for the desktop-shell client. You can set the panel buttons and the background image (which must be JPEG) in it. The sample file is wayland-desktop-shell.ini and you must copy it to $HOME/.config/ directory and edit for the background image to have a nice demo experience.

The actual screen locking in Wayland demos will be explained in a future post.

The past summer

2011-11-18T16:02:00.001+02:00

In the past summer I joined Collabora. After settling in and getting familiar with the self-employed status, I worked on Compiz for a moment, fixing bugs and helping the GLES 2 port. Then, I turned my interest to Wayland, and decided to implement some sort of screen locking.