[Info] RetroPlayer shaders and overlays
#1
Hello,
I am interested in working on the shaders and overlays parts of RetroPlayer for GSoC.
I have worked with shaders with Nvidia's CG (in Unity3d) in the past and i'd like to work on implementing shaders and overlays in here as well.
Since Nvidia's CG language is now deprecated, i would create HLSL shaders and also convert them to GLSL using something like https://github.com/aras-p/hlsl2glslfork in order to support both GLSL and HLSL.
I am not yet accustomed to the XBMC code base, so i don't quite know how everything works internally, so i need some guidance for writing the "implementation" part of my proposal.
What would be the timeline of my project? Is there a system in place that would allow me to create these features?

Thanks.
Reply
#2
hi FREEZX,

shader support in RetroPlayer would be way cool. for this to make a viable GSoC project, we'll need to figure out the best way to implement this in Kodi. I actually have next to no experience with shaders, so I'm probably not the person to ask. how do other open source projects (like RetroArch) implement shaders and overlays? can we re-use any code from these projects? if you can describe one of these systems, I might be able to help adapt it to RetroPlayer.

there's been a big push to move as much code as possible (literally hundreds of thousands of lines over the last few months) into binary add-ons. small hint: the more code we can place into add-ons, the better
Reply
#3
Hi garbear and thank you for the response,

I took a glance at RetroArch's shader drivers, they work with GLSL, HLSL as well as Cg:
https://github.com/libretro/RetroArch/tr...ers_shader

I also noticed that apparently they are also supporting software filters that run on the CPU:
https://github.com/libretro/RetroArch/bl...o_filter.c

The way shaders work is the renderer loads up and compiles shader files optimizing them for the specific graphics card it's working on.
See how it's done in (OpenGL)
After compiling, when you want to render something with the shader, you call glUseProgram (OpenGL-specific), and the next renders are executed with the given shader prog.

I am not sure how the rendering pipeline of Kodi works, but there is probably vendor-specific implementations (d3d/gl) of this process somewhere in the code, and i'm not sure how/if it could be pushed as a binary add-on.
Is there any add-on that modifies the rendering pipeline that i could use for reference?

Thanks
Reply
#4
check out RetroPlayerVideo.cpp. Those 200 lines are the entirety of RetroPlayer's video handling. Basically just a thin wrapper around the renderer, with some colorspace conversion sprinkled in. eventually the colorspace conversion will be done within the renderer, but I'm aiming to be minimally invasive until then.

we will need to identify where in the rendering pipeline to inject our transformations (probably multiple places and for multiple [directx/gl] APIs), and then build an API around this for shader/overlay add-ons. In concept this is very similar to audio DSP add-ons. I haven't read through the code in that branch yet, but it might offer some insights.

in RetroPlayer, we only have access to the raw RGB data (and if/when GameStreaming is merged, only h264 packets). full-blown shader support might have to dig a little deeper into the rendering system, but if the API for shader add-ons has multiple entry points, CPU-based shaders can be added directly in retroplayer for a start.
Reply
#5
I will check it out and create a sample soft shader tonight as a proof of concept, i'll report here when i'm done
Reply
#6
hint hint: a blur shader would be nice to have for our skinning engine Wink
Reply
#7
@FREEZX, it might be worth taking a look at openCV. It is a great cross platform computer vision library.
Reply
#8
(2015-03-26, 07:53)colek42 Wrote: @FREEZX, it might be worth taking a look at openCV. It is a great cross platform computer vision library.

I'm no OpenCV expert but initially my scare is that the image will have to jump between ram and vram like crazy, atleast if OpenCV is using an graphic card accelerated route.

RetroPlayer image RAM -> OpenCV upload to vram -> OpenCV processes in vram -> OpenCV downloads from vram -> RetroPlayer uploads to vram for display

If we do it in shaders we'd upload once and all processing is done in vram. But it might be that OpenCV has some options to either keep processing in ram (slow probably) or it can give us back an object in vram. If it can do the latter then yeah it might be a valid option!
If you have problems please read this before posting

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.

Image

"Well Im gonna download the code and look at it a bit but I'm certainly not a really good C/C++ programer but I'd help as much as I can, I mostly write in C#."
Reply
#9
(2015-03-26, 07:53)colek42 Wrote: @FREEZX, it might be worth taking a look at openCV. It is a great cross platform computer vision library.
Another problem with that is that Kodi is multi-platform and most embedded systems / devices out there today does not even support OpenCV in hardware, with exception of the very newest high-end devices released during the last 12-months or so. Also, you don't need a cannon to shoot sparrows, as while OpenCV is much more powerful it is better suited for more advanced processing of example camera input for motion control Kinect like features and such, rather than adding more simple overlays or distorting/blurring effects where using a Shading Language will probably be best.

On the other hand, almost all embedded system / device hardware supports OpenGL ES 2.0 and GLSL-ES shaders. Be sure to work with GLSL-ES shaders and not just at desktop GLSL shaders as those require full OpenGL support. GLSL has progressed far in the many years since GLSL-ES split off, so newer desktop GLSL shaders will not work on GLSL-ES, meaning you need to reuse to the algorithm and rewrite it for OpenGL ES 2.0

https://www.khronos.org/files/opengles_s...nguage.pdf
https://www.khronos.org/opengles/sdk/docs/man/

Anyway, if you are going to be using shader languages then it might be best to target the shader language for the lowest denominator as a reference first, which is GLSL-ES for OpenGL ES 2.0 (or possibly GLSL-ES for version OpenGL ES 2.1), then see if that also runs on full GLSL hardware too, modify for full GLSL if needed, and only then convert to HLSL.
Reply
#10
(2015-03-26, 09:40)topfs2 Wrote:
(2015-03-26, 07:53)colek42 Wrote: @FREEZX, it might be worth taking a look at openCV. It is a great cross platform computer vision library.

I'm no OpenCV expert but initially my scare is that the image will have to jump between ram and vram like crazy, atleast if OpenCV is using an graphic card accelerated route.

RetroPlayer image RAM -> OpenCV upload to vram -> OpenCV processes in vram -> OpenCV downloads from vram -> RetroPlayer uploads to vram for display

If we do it in shaders we'd upload once and all processing is done in vram. But it might be that OpenCV has some options to either keep processing in ram (slow probably) or it can give us back an object in vram. If it can do the latter then yeah it might be a valid option!

That sounds horrible. I don't have much experience with rendering, I use OpenCV for computer vision. Sounds like it is definitely not the right tool.
Reply
#11
(2015-03-26, 07:53)colek42 Wrote: @FREEZX, it might be worth taking a look at openCV. It is a great cross platform computer vision library.

OpenCV is used for computer vision rather than processing pixels, rendering libraries have shaders for the purpose of creating effects.

@da-anda @garbear
I don't currently have enough free time to create any soft shader and proposal before the GSoC deadline, and i am sort of overworked from creating proposals and working on demos the whole past week.
I am, however, willing to contribute to this feature in my free time after the following week (uni exams), to get some more experience with C++ and explore the rendering pipeline of Kodi.
Reply
#12
hey, no preasure, it's open source and supposed to be fun
Reply
#13
It's worth noting that you only need the proposal by tomorrow; a working softshader is definitely not required for the proposal. We can continue talking for quite some time after that before any decisions are made. But if you decide just to play around with this in your free time instead, that'd also work. That's at least theoretically what all the rest of us do. Smile
Reply
#14
(2015-03-26, 19:44)natethomas Wrote: It's worth noting that you only need the proposal by tomorrow; a working softshader is definitely not required for the proposal. We can continue talking for quite some time after that before any decisions are made. But if you decide just to play around with this in your free time instead, that'd also work. That's at least theoretically what all the rest of us do. Smile

In that case i will submit a proposal, and we'll keep talking to see how it can/should be done later.
Reply
#15
I've looked at the calendar, just to remind myself of timing. It seems like we've got until April 13th to request slots and April 21st to finish making the hard decisions.
Reply

Logout Mark Read Team Forum Stats Members Help
[Info] RetroPlayer shaders and overlays0