[Montellese]

Also good practice will be to ask for records in chunks to maintain snappy performance. There is no way to display thousands of ROMs on the screen at once, and should not ask the database to give us all of them. The gui should ask for a certain number of records above and below what is currently displayed. If we need all of the titles at once and reads become a problem (I don't see this happening) I can always use SQLAlchemy Core and bypass the ORM to get some more speed.

This will require a rather large refactoring in our core code because currently we always retrieve all the items and then start listing them. We don't necessarily retrieve all the metadata for all the items at once and load some of it in the background after having already built the list.
I also assume that some of the JSON-RPC clients use this and retrieve all items at once when they e.g. perform an offline sync with their own database.

[RockerC]

A few more general points that come to mind from internal discussions we had in the team about how we should do the persistence logic in Kodi and how potential server/client setups could look like:

• Most users won't have a dedicated server machine where the backend would run on separated from clients.
  
• It should be possible to use Kodi standalone or in a server/client setup.
  
• Users shouldn't have to manually setup a database unless they have a more advanced setup in which case they usually know what they are doing.

This! In my mind Kodi should primarily be made for home users, and not be made to be primarily for commercial setups where a enterprise client-server design would actually makes sense.

So rather than having a client-server model design that is made to have a dedicated server and dedicated clients, I think that a more peer-to-peer design would be much better suited for Kodi, where in a peer-to-peer design every client is also a full server, with all data being fully replicated two-ways between all the clients (which again also all acts as full servers).

I believe Kodi having a peer-to-peer design where all stand-alone "peers" share resources amongst each other without the use of a centralized server would be more user-friendly.

Dedicated server setup would be a nice OPTION for the few advanced users who want it, but I think that is in essence a different target audience than what the existing Kodi application was design for. I bet the majority of users will only ever use Kodi in a stand-alone setup and will likely be put off if it became more complicated to install and configure with a server.

Am I right to assume that the rPi is about the slowest platform Kodi will run on?

Yeah you need to keep the existing Kodi audience in mind here too, as with "slower embedded systems" you need to think of original Raspberry Pi and older single-core Android devices.

Intel i3/i5/i7 or even Celeron based system are still all in the high-end HTPC class in the world of Kodi, so those are not even close to being in the "slower embedded systems" class.

[jjd-uk]

I think that a more peer-to-peer design would be much better suited for Kodi, where in a peer-to-peer design every client is also a full server, with all data being fully replicated two-ways between all the clients (which again also all acts as full servers).

That is exactly the route we want to go.

The headless work will likely still be a full Kodi install, but with the added ability to turn off the GUI and all it's dependences, so headless Kodi would simply be an always available Master (that can happen to be installed on a server/NAS) that can be used to sync all other clients that join the network.

[colek42]

This will require a rather large refactoring in our core code because currently we always retrieve all the items and then start listing them. We don't necessarily retrieve all the metadata for all the items at once and load some of it in the background after having already built the list.

Will have to test it the see if it will be needed, but I can get down to the metal in the database with SQLAlchemy Core if it is too slow on embedded devices. My testing and experience shows that we should be fine on most systems

Also do you know where the code that handles the listing of items is? I would like to take a look at it.

[colek42]

I think that a more peer-to-peer design would be much better suited for Kodi, where in a peer-to-peer design every client is also a full server, with all data being fully replicated two-ways between all the clients (which again also all acts as full servers).

I agree. I hope to be working (at my paid job) in solving this problem in a swarm of MAVs this summer. The biggest issue comes with the split mind problem. How is this problem solved in Kodi's domain? How to we account for being on a guest network? How can it be written so it is easily maintainable? What should happen when we are on a WAN? How do you handle media that is sitting on a NAS when you go to a friends house, or on a road trip? How to deal with version fragmentation?

How do you add all of these features without everything turning into a giant pile of spaghetti?

My experience with large distributed systems is that you need a schema. You need a single piece of truth for your moving bits of code. This is why in ROS you have a messaging system. It is why google uses protobuffs. Protobuffs is a perfect schema system for Kodi, IMHO. It is well supported, well tested, versionable, and has libraries for languages I have never even heard of.

The core of my proposal is using the Retroplayer project for proof of concept in just one of it's parts.

If using Python for the ORM bits doesn't work out, we just replace it with something in C++, or maybe golang (I have been looking for a reason to learn Go) -- it would not affect any other pieces of the system!. We still have the schema defined in our protobuff definition.

[Montellese]

Also do you know where the code that handles the listing of items is? I would like to take a look at it.

Basically for GUI listings we wrap the database into our virtual filesystem using different paths (like videodb://movies/titles/) which is then translated to the appropriate database call with a matching SQL query (see e.g. https://github.com/xbmc/xbmc/tree/master...eDirectory for the classes that wrap the video database). The actual retrieval of the items for the GUI in case of the video and music library happens at https://github.com/xbmc/xbmc/blob/master...w.cpp#L667 where we call into the VFS and get a list of items.

[RockerC]

I hope to be working (at my paid job) in solving this problem in a swarm of MAVs this summer.

Are you applying to GSOC and also plan to have a other paid job this summer? It is just that I'm pretty sure that the while idea with the GSOC stipend is that it will be your full time paid summer job, so wondering if you have enough time to focus on two paid jobs this summer? Would be sad if that would become a conflict of interest.

http://www.google-melange.com/gsoc/docum...equired_to

9. How much time is required to participate as a student in Google Summer of Code?

The amount of time you will need depends on both the scope of your project and the requirements of your mentoring organization. While your organization may offer some flexibility around milestone completion dates, you should expect your project to be your primary focus this summer. If you have a great internship starting soon or you're planning a month long backpacking trip, you likely won't be a good candidate for the program.

[colek42]

Also do you know where the code that handles the listing of items is? I would like to take a look at it.

Basically for GUI listings we wrap the database into our virtual filesystem using different paths (like videodb://movies/titles/) which is then translated to the appropriate database call with a matching SQL query (see e.g. https://github.com/xbmc/xbmc/tree/master...eDirectory for the classes that wrap the video database). The actual retrieval of the items for the GUI in case of the video and music library happens at https://github.com/xbmc/xbmc/blob/master...w.cpp#L667 where we call into the VFS and get a list of items.

Thank you for that. It may make more sense to implement the Retroplayer library similar to how Movies and TV are implemented now. After that I can start factoring it out to use the message system as I described. This would allow be to become familiar with the code base which will be necessary to implement some of the suggestions I suggested -as well as have a baseline to test against.

I work up a design and implementation proposal with the things I learned here and get it officially submitted.

[colek42]

I hope to be working (at my paid job) in solving this problem in a swarm of MAVs this summer.

Are you applying to GSOC and also plan to have a other paid job this summer? It is just that I'm pretty sure that the while idea with the GSOC stipend is that it will be your full time paid summer job, so wondering if you have enough time to focus on two paid jobs this summer? Would be sad if that would become a conflict of interest.

http://www.google-melange.com/gsoc/docum...equired_to

If the team members selecting the students have an issue with this please send me a pm. Details of why this will not be an issue are not something I wish to publish on a public forum.

[Maxoo]

So... Has this project been accepted ?
If not, will you still try to go forward with it ?

[garbear]

The accepted GSoC projects were announced on the blog a few days ago: http://kodi.tv/kodi-community-may-2015/ . unfortunately, as the only mentor for the game-related proposals, I had to choose between this and the gamestream proposal. I will say this, both were strong proposals and the choice wasn't easy.

colek42, the game library you've envisioned is something that Kodi will need as RetroPlayer becomes more popular and capable. It's highly likely that I'll take on this project myself (yet again ) and if you're still keen on contributing to Kodi I'd love your input when the time rolls around.