[Info] RetroPlayer shaders and overlays

  Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
FREEZX Offline
Junior Member
Posts: 10
Joined: Mar 2015
Reputation: 0
Post: #1
Hello,
I am interested in working on the shaders and overlays parts of RetroPlayer for GSoC.
I have worked with shaders with Nvidia's CG (in Unity3d) in the past and i'd like to work on implementing shaders and overlays in here as well.
Since Nvidia's CG language is now deprecated, i would create HLSL shaders and also convert them to GLSL using something like https://github.com/aras-p/hlsl2glslfork in order to support both GLSL and HLSL.
I am not yet accustomed to the XBMC code base, so i don't quite know how everything works internally, so i need some guidance for writing the "implementation" part of my proposal.
What would be the timeline of my project? Is there a system in place that would allow me to create these features?

Thanks.
find quote
garbear Offline
Team-Kodi Developer
Posts: 1,752
Joined: Dec 2010
Reputation: 112
Location: city of angels
Post: #2
hi FREEZX,

shader support in RetroPlayer would be way cool. for this to make a viable GSoC project, we'll need to figure out the best way to implement this in Kodi. I actually have next to no experience with shaders, so I'm probably not the person to ask. how do other open source projects (like RetroArch) implement shaders and overlays? can we re-use any code from these projects? if you can describe one of these systems, I might be able to help adapt it to RetroPlayer.

there's been a big push to move as much code as possible (literally hundreds of thousands of lines over the last few months) into binary add-ons. small hint: the more code we can place into add-ons, the better
find quote
FREEZX Offline
Junior Member
Posts: 10
Joined: Mar 2015
Reputation: 0
Post: #3
Hi garbear and thank you for the response,

I took a glance at RetroArch's shader drivers, they work with GLSL, HLSL as well as Cg:
https://github.com/libretro/RetroArch/tr...ers_shader

I also noticed that apparently they are also supporting software filters that run on the CPU:
https://github.com/libretro/RetroArch/bl...o_filter.c

The way shaders work is the renderer loads up and compiles shader files optimizing them for the specific graphics card it's working on.
See how it's done in (OpenGL)
After compiling, when you want to render something with the shader, you call glUseProgram (OpenGL-specific), and the next renders are executed with the given shader prog.

I am not sure how the rendering pipeline of Kodi works, but there is probably vendor-specific implementations (d3d/gl) of this process somewhere in the code, and i'm not sure how/if it could be pushed as a binary add-on.
Is there any add-on that modifies the rendering pipeline that i could use for reference?

Thanks
find quote
garbear Offline
Team-Kodi Developer
Posts: 1,752
Joined: Dec 2010
Reputation: 112
Location: city of angels
Post: #4
check out RetroPlayerVideo.cpp. Those 200 lines are the entirety of RetroPlayer's video handling. Basically just a thin wrapper around the renderer, with some colorspace conversion sprinkled in. eventually the colorspace conversion will be done within the renderer, but I'm aiming to be minimally invasive until then.

we will need to identify where in the rendering pipeline to inject our transformations (probably multiple places and for multiple [directx/gl] APIs), and then build an API around this for shader/overlay add-ons. In concept this is very similar to audio DSP add-ons. I haven't read through the code in that branch yet, but it might offer some insights.

in RetroPlayer, we only have access to the raw RGB data (and if/when GameStreaming is merged, only h264 packets). full-blown shader support might have to dig a little deeper into the rendering system, but if the API for shader add-ons has multiple entry points, CPU-based shaders can be added directly in retroplayer for a start.
find quote
FREEZX Offline
Junior Member
Posts: 10
Joined: Mar 2015
Reputation: 0
Post: #5
I will check it out and create a sample soft shader tonight as a proof of concept, i'll report here when i'm done
find quote
da-anda Offline
Team-Kodi Member
Posts: 5,781
Joined: Jun 2009
Reputation: 77
Location: germany
Post: #6
hint hint: a blur shader would be nice to have for our skinning engine Wink
find quote
colek42 Offline
Junior Member
Posts: 31
Joined: Sep 2013
Reputation: 1
Post: #7
@Freezx, it might be worth taking a look at openCV. It is a great cross platform computer vision library.
find quote
topfs2 Offline
Team-Kodi Developer
Posts: 4,548
Joined: Dec 2007
Reputation: 17
Post: #8
(2015-03-26 07:53)colek42 Wrote:  @Freezx, it might be worth taking a look at openCV. It is a great cross platform computer vision library.

I'm no OpenCV expert but initially my scare is that the image will have to jump between ram and vram like crazy, atleast if OpenCV is using an graphic card accelerated route.

RetroPlayer image RAM -> OpenCV upload to vram -> OpenCV processes in vram -> OpenCV downloads from vram -> RetroPlayer uploads to vram for display

If we do it in shaders we'd upload once and all processing is done in vram. But it might be that OpenCV has some options to either keep processing in ram (slow probably) or it can give us back an object in vram. If it can do the latter then yeah it might be a valid option!

If you have problems please read this before posting

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.

[Image: badge.gif]

"Well Im gonna download the code and look at it a bit but I'm certainly not a really good C/C++ programer but I'd help as much as I can, I mostly write in C#."
find quote
RockerC Offline
Posting Freak
Posts: 1,485
Joined: May 2011
Reputation: 28
Post: #9
(2015-03-26 07:53)colek42 Wrote:  @Freezx, it might be worth taking a look at openCV. It is a great cross platform computer vision library.
Another problem with that is that Kodi is multi-platform and most embedded systems / devices out there today does not even support OpenCV in hardware, with exception of the very newest high-end devices released during the last 12-months or so. Also, you don't need a cannon to shoot sparrows, as while OpenCV is much more powerful it is better suited for more advanced processing of example camera input for motion control Kinect like features and such, rather than adding more simple overlays or distorting/blurring effects where using a Shading Language will probably be best.

On the other hand, almost all embedded system / device hardware supports OpenGL ES 2.0 and GLSL-ES shaders. Be sure to work with GLSL-ES shaders and not just at desktop GLSL shaders as those require full OpenGL support. GLSL has progressed far in the many years since GLSL-ES split off, so newer desktop GLSL shaders will not work on GLSL-ES, meaning you need to reuse to the algorithm and rewrite it for OpenGL ES 2.0

https://www.khronos.org/files/opengles_s...nguage.pdf
https://www.khronos.org/opengles/sdk/docs/man/

Anyway, if you are going to be using shader languages then it might be best to target the shader language for the lowest denominator as a reference first, which is GLSL-ES for OpenGL ES 2.0 (or possibly GLSL-ES for version OpenGL ES 2.1), then see if that also runs on full GLSL hardware too, modify for full GLSL if needed, and only then convert to HLSL.
(This post was last modified: 2015-03-26 10:44 by RockerC.)
find quote
colek42 Offline
Junior Member
Posts: 31
Joined: Sep 2013
Reputation: 1
Post: #10
(2015-03-26 09:40)topfs2 Wrote:  
(2015-03-26 07:53)colek42 Wrote:  @Freezx, it might be worth taking a look at openCV. It is a great cross platform computer vision library.

I'm no OpenCV expert but initially my scare is that the image will have to jump between ram and vram like crazy, atleast if OpenCV is using an graphic card accelerated route.

RetroPlayer image RAM -> OpenCV upload to vram -> OpenCV processes in vram -> OpenCV downloads from vram -> RetroPlayer uploads to vram for display

If we do it in shaders we'd upload once and all processing is done in vram. But it might be that OpenCV has some options to either keep processing in ram (slow probably) or it can give us back an object in vram. If it can do the latter then yeah it might be a valid option!

That sounds horrible. I don't have much experience with rendering, I use OpenCV for computer vision. Sounds like it is definitely not the right tool.
find quote
FREEZX Offline
Junior Member
Posts: 10
Joined: Mar 2015
Reputation: 0
Post: #11
(2015-03-26 07:53)colek42 Wrote:  @Freezx, it might be worth taking a look at openCV. It is a great cross platform computer vision library.

OpenCV is used for computer vision rather than processing pixels, rendering libraries have shaders for the purpose of creating effects.

@da-anda @garbear
I don't currently have enough free time to create any soft shader and proposal before the GSoC deadline, and i am sort of overworked from creating proposals and working on demos the whole past week.
I am, however, willing to contribute to this feature in my free time after the following week (uni exams), to get some more experience with C++ and explore the rendering pipeline of Kodi.
find quote
da-anda Offline
Team-Kodi Member
Posts: 5,781
Joined: Jun 2009
Reputation: 77
Location: germany
Post: #12
hey, no preasure, it's open source and supposed to be fun
find quote
natethomas Offline
XBMC Chief and Kodi Project Manager
Posts: 6,233
Joined: Apr 2008
Reputation: 128
Location: Kansas
Post: #13
It's worth noting that you only need the proposal by tomorrow; a working softshader is definitely not required for the proposal. We can continue talking for quite some time after that before any decisions are made. But if you decide just to play around with this in your free time instead, that'd also work. That's at least theoretically what all the rest of us do. Smile
find quote
FREEZX Offline
Junior Member
Posts: 10
Joined: Mar 2015
Reputation: 0
Post: #14
(2015-03-26 19:44)natethomas Wrote:  It's worth noting that you only need the proposal by tomorrow; a working softshader is definitely not required for the proposal. We can continue talking for quite some time after that before any decisions are made. But if you decide just to play around with this in your free time instead, that'd also work. That's at least theoretically what all the rest of us do. Smile

In that case i will submit a proposal, and we'll keep talking to see how it can/should be done later.
find quote
natethomas Offline
XBMC Chief and Kodi Project Manager
Posts: 6,233
Joined: Apr 2008
Reputation: 128
Location: Kansas
Post: #15
I've looked at the calendar, just to remind myself of timing. It seems like we've got until April 13th to request slots and April 21st to finish making the hard decisions.
(This post was last modified: 2015-03-27 01:07 by natethomas.)
find quote
Post Reply