[yol]

Ah OK - I didn't look specifically into VDPAU. But if that's clear there is no reason to include it in the proposal, so I've removed it.

Could substitute VDPAU with VAAPI (VA API)? http://en.wikipedia.org/wiki/Video_Acceleration_API

Well, VA-API is part of the proposal (for the cards/drivers that support it).

[daniels]

Congratulations on being accepted this year! I'm one of the Wayland developers, and would be more than happy to help and answer any questions you have, either here or as 'daniels' on Freenode.

Using wp_presentation would be really great. Unfortunately it's still not implemented in Mutter for boring internal infrastructural reasons, but I'd expect that to change soon.

Native YUV support would also be really nice to have, though again, Mutter doesn't yet support it. :\ The news article linked to is very old, which mainly describes work with the wl_drm protocol, something which is an internal implementation detail of Mesa and thus not supported on other platforms. We do also reserve the right to break that interface, so please don't rely on it! Instead, I'd be looking more towards the zwp_linux_dmabuf_v1 interface, which is again supported by Weston, and will be supported by Mutter within the next couple of months I hope.

Using dmabuf directly allows you to be in full control of the presentation layer, not having to route around EGL/VA-API's own presentation support. If this is a viable option then I'd personally be very happy to see it, though I wouldn't be surprised if this required a fair bit of rework on Kodi, so please don't necessarily take this as good advice. This is the route VA-API is going down as well: allowing clients to access dmabuf handles directly and drive presentation themselves.

Best of luck with your summer, and please get in touch if I can help at all!

[FernetMenta]

Kodi's presentation component is independent from decoding APIs like VAAPi or VDPAU. Kodi renders GL textures. The requirement on a windowing system is that Kodi needs to know when a texture rendered through GL becomes visible on screen.
Kodi also needs to sync its clock with vertical retrace aka vblank. The windowing system should provide some mechanism to do so.

[daniels]

Kodi's presentation component is independent from decoding APIs like VAAPi or VDPAU. Kodi renders GL textures. The requirement on a windowing system is that Kodi needs to know when a texture rendered through GL becomes visible on screen.

That's reasonable. Bearing in mind that this isn't the bad old days of destructive X11 compositors though, I would be interested to explore direct buffer attachment with you guys. In other words, not using the GPU (via GL) to copy the buffers and do conversion, but offload that to the compositor, where we can potentially use the display controller overlays to do so, with no copying. Our colour management isn't quite up to scratch for perfectly lossless pipelines yet, but we're hoping to make it so in the fairly near future. Having at least a pathway to an implementation in Kodi would be a great testbed for this. This usually gives quite good filtering/scaling/conversion through the overlays, as well as helping bring down power usage, and maybe even bring up performance thanks to reduced memory bandwidth. Anyway, one for the future.

Kodi also needs to sync its clock with vertical retrace aka vblank. The windowing system should provide some mechanism to do so.

Hm, vblank itself is not quite what you want to be synchronising to. The kernel needs to submit its final configuration to the hardware just before vblank. In turn, the compositor needs to submit its final display configuration to the kernel, before the kernel reaches that critical timing point. In turn, the client needs to submit its final frame to the compositor, a bit before the compositor submits to the kernel. The exact mechanics of the frame timings are a little fluid, but the point is that 'vblank' is a very specific term, where what you want is more like the critical presentation commit point, wherever that may lie. (For Weston's default configuration, this is 7ms before vblank.)

[FernetMenta]

1) presentation
For this thread we stay with GL rendering and our own color management. We have things like 3dlut and I think doing this at the app level leaves us with more options.

Rendering video in paralled to GL may be an option for an additional path. As you said, something for the futur.

2) vblank
We don't use this for precise timing but to sync the clock. We allow an variance of 30% and also have some error correction if we miss a vblank. The point here is that display fps never matches video frame rate to 100% and if you sync to system clock sooner or later the error gets bigger than frametime. Dropping a frame or duplicating one causes noticeable stutter.

[daniels]

1) presentation
For this thread we stay with GL rendering and our own color management. We have things like 3dlut and I think doing this at the app level leaves us with more options.

Rendering video in paralled to GL may be an option for an additional path. As you said, something for the futur.

Totally understand.

2) vblank
We don't use this for precise timing but to sync the clock. We allow an variance of 30% and also have some error correction if we miss a vblank. The point here is that display fps never matches video frame rate to 100% and if you sync to system clock sooner or later the error gets bigger than frametime. Dropping a frame or duplicating one causes noticeable stutter.

I see what you mean, but I'm not sure I agree with the sentiment. Like I say, the deadline by which you need to present for a given frame, will always fall at a certain point within CPU time. The compositor decides when it will repaint for the next frame, and it is this deadline the app must meet. Using vblank is one way to get this information, but not the only way and not necessarily entirely correct. I feel it is useful in that sense to avoid talking about vblank (in the very specific CRT-derived timing sense) unless you truly mean the actual display device, but this is primarily useful for obtaining time-to-light information for A/V sync, rather than trying to derive the deadline you must meet to submit your next frame.

[FernetMenta]

Not sure we are talking about the same thing. We use vblank information not for a/v sync but for adjusting playback speed to the display. Kodi does not need to know whether refresh rate is exaclty 23.976 or 24.0 or someting slightly off. If you play some 23.976 material on a 24Hz display it will run sllightly faster but smooth. I am not talking about presentation time. Kodi needs to get the beat of vertical retrace.

[yol]

Congratulations on being accepted this year! I'm one of the Wayland developers, and would be more than happy to help and answer any questions you have, either here or as 'daniels' on Freenode.

Using wp_presentation would be really great. Unfortunately it's still not implemented in Mutter for boring internal infrastructural reasons, but I'd expect that to change soon.

Native YUV support would also be really nice to have, though again, Mutter doesn't yet support it. :\ The news article linked to is very old, which mainly describes work with the wl_drm protocol, something which is an internal implementation detail of Mesa and thus not supported on other platforms. We do also reserve the right to break that interface, so please don't rely on it! Instead, I'd be looking more towards the zwp_linux_dmabuf_v1 interface, which is again supported by Weston, and will be supported by Mutter within the next couple of months I hope.

Using dmabuf directly allows you to be in full control of the presentation layer, not having to route around EGL/VA-API's own presentation support. If this is a viable option then I'd personally be very happy to see it, though I wouldn't be surprised if this required a fair bit of rework on Kodi, so please don't necessarily take this as good advice. This is the route VA-API is going down as well: allowing clients to access dmabuf handles directly and drive presentation themselves.

Best of luck with your summer, and please get in touch if I can help at all!

Thanks! Having a Wayland developer around to ask directly will certainly come in handy :-) As FernetMenta already said though, support for non-RGB surfaces will probably be very low priority as far as this GSoC project goes.

Allow me to ask one quick question: Are you aware of any "good" C++ bindings for Wayland?

[yol]

Hi community,

I've set up a GitHub repository for my GSoC project: https://github.com/pkerling/xbmc

I've added the tasks from the proposal as issues there so you can track the progress. Do feel free to comment there, test the code, report bugs. etc. as soon as there is something to test :-)

[yol]

Hi again,

basic Wayland support is now integrated into the master branch at https://github.com/pkerling/xbmc

To summarize what I did over the first two weeks of coding period:

• Decide on what library to use for communication with the compositor: I tried out libwayland-client, the old Kodi-specific Wayland protocol wrappers (from the prior implementation from 2013 or so) and waylandpp. In the end I chose waylandpp because the object-based protocol of Wayland really shines when used with C++ (which rules out libwayland-client, which is C) and the home-grown C++ wrappers were too much of a burden to write by hand, considering that the list of interfaces and methods that needs to be supported grows with each feature. waylandpp seemed abandoned at first but I put out a few pull requests on GitHub which the author promptly merged. This is a very good sign and means that it probably won't be necessary to maintain a Kodi-specific fork of waylandpp. I do have a fork on GitHub though that should be used for the time being for compiling Kodi since I often add new features to waylandpp, use them in Kodi, and then submit a PR to the waylandpp owner afterwards, which means that necessary features might not be upstreamed yet. I expect this to change once things get more stable.
  
• Finish porting all features that the prior implementation had - and more :-) among the additional features are things such as full Unicode input (think e.g. Cyrillic), having a window icon in the task switcher, not displaying a mouse cursor when no mouse input is available, and selecting a monitor for fullscreen display
  
• Work on getting VAAPI support back (in-progress)
  
• Start xdg_shell integration
  
• Do some clean-up in Kodi related to the event system which was based on SDL, which is thoroughly obsolete by now
  

I regularly update the project board which should give you an idea about what I'm working on at the moment. Next step is cleaning up VAAPI and xdg_shell support.

In the end I did rewrite most of the prior implementation, but I used it a lot for reference. Note that there is nothing inherently wrong with the old code, the basic stuff did work after all. It was just using a deprecated implementation model (CWinSystemEGL) that was slated for removal anyway because it does not make sense from an architectural POV - EGL is not a windowing system, just a means to get a GL surface. Actually, most other code depending on CWinSystemEGL has also already been refactored now. Additionally, the old implementation included a whole lot of code for integration testing that I decided not to add. It complicates the implementation a lot, is very time-intensive to port and maintain, and provides not enough benefit to make up for it in my opinion.

The repository also has basic build instructions, and it would be cool if some interested parties would get the code, try to run it on their compositor of choice and report the results.

[yol]

Wayland Support
Google Summer of Code 2017 Work Product Submission
Student: Philipp Kerling

Motivation
The Wayland protocol is slowly but steadily gaining traction in the Linux world as successor to the X11 display server and is generally seen to replace it in the long term, which is why it is essential for Kodi to support it if it wants to continue to deliver a good user experience on Linux. Kodi did already have a Wayland windowing implementation originally written by Sam Spilsbury during GSoC 2013, but it got removed from master for unsolved infrastructure reasons some time ago and was never on-par with the X11 implementation feature-wise.

Goals
My main goal was quite simply to get Kodi to natively support Wayland again, if possible as well as it currently supports X11. This was refined into a number of sub-goals/features in my initial proposal categorized as "basic" (i.e. necessary so you can reasonably use it at all), "additional," and "possible extension." These goals can be found in full in the original proposal. They are not revisited goal-for-goal here, but in summary all basic and additional features were completed. One possible extension feature (namely full touch support) was implemented. The remaining ones are listed further down in this document as future work.

Implemented features
The following major features were implemented (links go to GitHub issues, in approximate implementation order):

• Basic Wayland windowing support: Got Kodi running on Wayland again with basic features (only full screen, only full GL, mouse/keyboard input)
  
• xdg_shell: Wayland has the core wl_shell protocol for communicating details about toplevel windows, but it is deprecated for all practical purposes. To get better integration, support for the still unstable designated successor xdg_shell (version 6 only) was added.
  
• VAAPI support
  
• GLES support
  
• Screensaver inhibition: Support for both the generic org.freedesktop.ScreenSaver DBus interface as well as the Wayland-specific idle inhibit protocol was added.
  
• Touch input: Kodi/X11 does not have the ability to navigate using multi-touch gestures since touch is only supported via the default X pointer emulation. Kodi/Wayland has full native touch input support.
  
• Paste from clipboard
  
• wp_presentation support: The stable wp_presentation protocol provides detailed timing information about displayed frames and is now used for determining display latency and for syncing to video clock on Kodi/Wayland.
  
• Windowed mode: Most Wayland compositors require applications to draw and handle window decorations themselves (called client-side decorations) if they want to have any. I discussed with my mentor how to go about this and we decided to once again keep things local to the Wayland implementation and draw the decorations there as none of the other windowing systems currently requires this. The other option would have been to somehow integrate the decorations into the Kodi skinning system. This is of course nicer, but far more complicated and probably not worth the trouble. As media center, most users will use Kodi in full screen mode which is also the default, so window decorations that don't look that nice will not be a big problem.
  
• tools/depends integration for CI: Dependencies for the Wayland build (mostly waylandpp) needed to be added to tools/depends in order to have it build on Jenkins, which the Kodi project uses for continuous integration.
  

Code
During development I commited all work to my Kodi fork at GitHub. Further improvements might be made there before they get integrated into mainline.

In preparation of the GSoC work product submission, all Wayland-related changes were submitted as one pull request against the master branch of the Kodi mainline repository. It was fully merged in the last week of the coding period. Kodi can be built as usual. To select the Wayland platform, the additional CMake arguments

Code:

-DCORE_PLATFORM_NAME=wayland -DWAYLAND_RENDER_SYSTEM=gl

must be used. Also, some extra libraries such as waylandpp are required. For details, see the updated Linux build instructions.
The main pull request against master had 124 changed files with 9,832 added and 292 removed lines.

Development process
I had planned to salvage as much of the previous code as possible, but different design choices I made ultimately meant that I ended up writing most of it from scratch. Still, I was frequently checking the prior implementation to see which parts of Kodi I have to touch and how I can go about solving some specific aspects.
To get a bit more concrete here, where the approach differs most is how the Wayland protocol objects are used in object-oriented C++. Previously a home-grown approach was used, probably because there were no C++ bindings for libwayland-client at the time. The resulting wrapper code is quite tedious to write and maintain, which is why I decided to use the C++ waylandpp library as base. Even though it is not quite mature yet, I figured that my time would be better spent improving waylandpp than trying to come up with something new on my own. It is of course possible to use the C API of libwayland-client directly without any wrappers, but this is not a very nice solution since the Wayland protocol is really made out to be used in an object-oriented fashion if the language allows for it. Much to my delight, the author of waylandpp was very responsive and supporting, which means that all improvements and feature additions I made to waylandpp at this point are already merged upstream.

Unfortunately, it soon became clear that the novel approaches that Wayland took compared to X11, Windows, and most other windowing systems were not a good match for the Kodi windowing infrastructure: Wayland is designed such that clients have minimal control over the desktop for security reasons. This for example means that applications cannot just set arbitrary video modes. They can only tell what their preference would be to the compositor, which then decides and in turn tells the applications what size they get. On the one hand, I do think that this is a good design decision since it solves a lot of problems that XRandR has like leaving the desktop in a bad resolution when a program crashes. On the other hand, it does mean that a lot of things have to be done differently and that Kodi cannot provide a perfect experience on Wayland for the time being. The reason is that there currently is no practically useful way to tell the compositor e.g. what refresh rate the output should use, so switching to the correct modeline for playback at original speed without occasional micro-stutters is not possible.

As far as the actual implementation is concernced, the Kodi code assumes the traditional procedure of fetching a list of valid resolutions and then being able to switch to any resolution immediately and basically without failure. Changing this in a satisfying way would require quite a lot of refactoring that would have to touch all of the current windowing implementations, too. This is a lot of work, requires coordination with all platform maintainers, and is not realistic to be completed within the GSoC timeframe, so I instead tried to make minimal adjustments to the Kodi resolution switching code that allow the Wayland implementation to work without impacting any other windowing system. The downside to this is that the Wayland code is not as clean as it could or should be and that there is an increased risk of weird stuff happening.

During the past three months I also had to learn that Wayland is still quite new to the scene and that a lot of bugs still lurk in the various components. I had to report and sometimes fix bugs in compositors and other third-party components such as mutter, weston, and libva. Finding these bugs and determining whose fault it is that something did not work as expected took more time than I had initially anticipated.

Future work

• Refactor Kodi resolution setting
  
• Kodi skin cursor image could be placed into the Wayland cursor surface to reduce input latency (at the moment the Wayland cursor surface is empty/unused and Kodi paints the cursor directly into its surface with GL)
  
• Allow X11 and Wayland in one Kodi build (at the moment Kodi can only be built with one windowing implementation set at compile time): While this would be desirable to facilitate quick adoption of a Wayland, it was ultimately decided that the Kodi core changes if done right would be too extensive to finish in time. Possible work-arounds include shipping two builds plus a wrapper script that starts the version corresponding to the active display server. See here for more details.
  
• Add facilities to switch resolution and/or frame rate: Wayland currently has no satisfying protocol support for clients that need specific output parameters (except for fullscreen-shell, see below). Therefore, solving this and enabling the resolution adaption in Kodi would require designing and upstreaming a Wayland protocol extension that enables applications to communicate their requirements to the compositor as well as implementing it at least in weston.
  
• Support for unstable fullscreen-shell protocol to provide modesetting: fullscreen-shell is an alternative shell protocol for a single application to run on a compositor in full screen mode. It has modesetting capabilities. Unclear if needed since Kodi now has a native gbm backend that will result in approximately the same thing (minus the Wayland compositor in between). The gbm backend currently only supports GLES, but it could easily be expanded to full GL.
  
• Support for zero-copy video rendering of video to YUV surfaces: Still questionable if benefits justify the necessary work
  

Acknowledgements
I am very grateful to the Kodi team and community for accepting me as student this year and my mentor FernetMenta for always being quick to respond, giving valuable advice, and investing a lot of his private time to review my work. This also goes for a lot of other team members (namely fritsch, lrusak, martijn, paxxi, rechi, and wsnipex) that followed my journey and supported me at specific tasks.

Furthermore, I would like to thank the #wayland IRC community and specifically pq and SardemFF7 for their time and advice that helped me move forward several times.

Contributions to Kodi apart from Wayland PR

• Stop application on SIGINT and SIGTERM
  
• Fix fallout
  
• Fix XBMC_event fallout in WinEventsWin32
  
• Posix/Win32 main refactor
  
• SDL Event cleanup
  
• More SDL event cleanup
  
• Rework OS screen saver handling
  
• DBus refactoring
  
• DeviceKit cleanup
  
• Restore = default constructor for DBusConnectionDeleter
  
• Modernize CEventGroup
  
• Support more yuv420p bitwidths for direct rendering on Linux
  
• Change CVideoSync stop flag from atomic bool to CEvent
  
• Log filter graph only when video component logging is enabled
  
• Remove autotools files from gitignore
  
• Fix missing include when building for X11 without VAAPI and VDPAU
  
• Remove unused parameters on ResetRenderSystem()
  
• Build libusb without -Werror
  
• Explicitly disable guile in GnuTLS build
  
• Fix crash when VAAPI is not available at runtime
  
• Remove dependency on zip always and on unzip for everything but Android
  
• Fix path autoconf-pre-depends looks in for m4
  
• Assorted touch fixes
  
• Integrate Wayland build into buildsteps
  
• [wayland] Fix compilation with older gcc
  

Contributions to third-party projects
waylandpp

• Allow copying of empty proxy_t
  
• Allow public access to proxy_t c_ptr() and state
  
• Make proxy_t bool operator and proxy_has_object() const
  
• Fix enum protocol type
  
• Introduce read_intent for announcing FD read intent with RAII
  
• Add wayland:et_log_handler
  
• Add proxy_t::get_version
  
• Add interface_name constant to generated interfaces
  
• Reorganize EGL stuff
  
• Fix empty instance creation from proxy
  
• Install scanner
  
• Unify operators and functions available on wrappers
  
• Improve thread-safety when copying proxy_t
  
• Throw std:ystem_exception when display_t operations fail
  
• Add some const markers and fix dump example
  
• Add helpers for checking availability of functions
  
• Introduce wrapper for wl_fixed_t
  
• Various fixes
  
• Accept empty event_queue_t for proxy_t:et_queue
  
• Fix handling of objects that are owned by other libraries
  
• Accept const reference in egl_window_t constructor
  
• Add support for proxy wrappers
  
• pkgconfig
  

libva

• Wayland thread-safety
  
• wayland: Also support wl_drm version 1
  
• wayland: Do not rely on external library for wl_drm_interface
  
• Wayland fixes
  

intel-vaapi-driver

• Wayland thread-safety
  
• wayland: Also support wl_drm version 1
  
• Remove generated file from tree and on clean
  
• Fix Wayland crash when globals are removed
  

cpython

• bpo-30654: Do not reset SIGINT handler to SIG_DFL in finisignal
  

weston

• desktop-shell: Track focused shell surface by main surface
  
• libweston-desktop/xdg-shell: Check window geometry instead of surface size against configured size
  
• desktop-shell: Set surface resizing state during interactive resize
  
• libweston: Do not include subsurfaces with NULL buffer in bounding box
  

Bugs reported to third-party projects
mutter/gnome-shell

• screensaver-proxy: Check gnome-session Inhibit call return value
  
• Edge-swipe gestures should be inhibited for full-screen applications
  
• Lockup when using wl_shell_surface:et_transient on same surface
  
• xdg_shell set_fullscreen() output setting is misinterpreted sometimes
  
• surface sizing issues when using zxdg_toplevel_v6:et_fullscreen() with monitors of differing scale
  
• wayland: Allow multitouch on client surfaces with > 2 contact points
  
• wayland: First configure is wrong for subcomposited surfaces
  
• wayland: Crash when destroying subsurface before surface: "assertion failed: (surface->role)"
  

weston

• subcomposited surfaces with content outside shell surface area lack proper damage on some occasions
  

[wsnipex]

Congratulations and thank you for your awesome work!