DB's central servers improvments
#1
All,

I am thinking about starting development to improve the way we handle synchronized systems.

My thought was each kodi instance will have their own local dB's that is independent of all other instances and the central server. Starting off there would be one main setup. Then from there the central server would "mirror", not in the literal sense because I would be working with different database types, the main. The first time that this happens will be slow, it has to build everything, also when a new instance connects for the first time.

Now to keep everything synchronized, there will be a special table that monitors if anything has changed. This table will also record the id of each instance with a changed version That I will use for comparison to see if it is up to date. There will be multiple change types so that I know which items need to be updated. This will allow for quick connections since I will only be sending what needs to be changed. Or at least, what values should be updated.

I am thinking it might be better sending an XML to the central server which would have UDF's along with stored procedures to full execute the needed operations, be that creating a new table, inserting rows, or deleting items
.
The updates would only happen in the following ways, if and only if there is a central server connected.
1. The user pauses the media. This would send a quick update of the current time.
2. A users stops playing media. Again same as number 1.
3. The devices is turned on. This would check to see if any updates are needed, usually quick unless added a bunch of media
4. The instance has been idle for x amount of time.

I would also make an easy settings page that would allow the user to input the central servers information along with other settings that could be directly related based on how the users wants it to behave.

As far as setting the central server up, for those that have little know how, I could create an auto executable that would automatically install all the needed software. Including the UDF's and possibly a utility that I can build to assist in parsing.
This would be a huge improvement because we would not be mediadown to the central server therefore certain instances could be completely mobile and only updating when there is a connection to the central server.

As far as my credentials, I have been working with enterprise software that connects to many different systems and uses multiple databases that have to talk to each other. One of my applications is kind of set up this way already just not to the scale of what we are going to be doing here.

Please let me know thoughts, suggestions, concerns etc. Also feel free to pm me.
Reply
#2
Montellese is working on a shared library feature based on upnp. I'm not entirely sure what branch the latest work resides in but its one of the media_import branches iirc. https://github.com/Montellese/xbmc/tree/media_import

I'm sure PRs against his branch would be appreciated!
If you have problems please read this before posting

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.

Image

"Well Im gonna download the code and look at it a bit but I'm certainly not a really good C/C++ programer but I'd help as much as I can, I mostly write in C#."
Reply
#3
I don't have time to go into details right now but as topfs2 has mentioned I'm working on something similar but with UPnP and there's no restrictions on who can synchronise with who. I've given a presentation about the current state of my work at the last DevCon and you can take a look at it here: https://dl.dropboxusercontent.com/u/4929...orting.pdf. There's only support for video library items. The system is designed to not only work with UPnP and it should be possible to write other synchronisers. In the end you need to have a component that retrieves the remote items either in "raw" format or as a changeset against the local database. The rest of the system will take care of adding/updating/removing the synchronised items to/from the local database.

The system isn't finished yet as it's completely separate from the usual video library scanning and running both in parallel will kill the local database. Starting playback will also not pause media importing and there's no updating interval or anything. It simply updates whenever it auto-detects the media provider and you can also manually trigger synchronisation.

I just rebased and updated my branch to current master and it's located at https://github.com/Montellese/xbmc/tree/...t_redesign.

There's also some other guy (don't remember his name) that was trying to do something with keeping track of changed records etc and using another UPnP specification/protocol to only transfer the items that have actually changed. In my implementation I always transfer all the items and then determine what has changed. That's obviously not ideal and a bit slow but it was the easiest to go on for a start.
Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
Reply
#4
Thanks. What is the main thread for this development? While I like UPNP, I am not too happy that it is completely separate from the local library, which makes it a pain to access (e.g., I cannot go into movie menu from the main screen unless I map things separately). Are there plans to fully integrate UPNP with the libraries?

Have the developers decided to push UPNP forward and eventually abandon mysql for this purpose?
Server: Asus Sabertooth Z77 | Intel Core i5 3.4 GHz | 16 GB DDR3 | 128 GB SSD, 82 TB (9 x 6 TB, 7 x 4 TB)
HTPC 1: Raspberry Pi 2 | HTPC 2: Raspberry Pi 2 | HTPC 3: Raspberry Pi
Reply
#5
(2014-12-29, 08:52)steve1977 Wrote: Thanks. What is the main thread for this development?
There's no main thread or anything. The problem is that I only have very very limited time to work on it so I never created a thread for it because the main thing that will happen is people asking when it will be ready and to get test builds etc.

(2014-12-29, 08:52)steve1977 Wrote: While I like UPNP, I am not too happy that it is completely separate from the local library, which makes it a pain to access (e.g., I cannot go into movie menu from the main screen unless I map things separately). Are there plans to fully integrate UPNP with the libraries?
The way it works is that it uses UPnP to retrieve all items of a certain media type (movies, tvshows, ...) and then integrates the retrieved items into the local library. So you don't even need to know that the items are imported/accessed using UPnP and you also don't need to browse into a UPnP server as the items will be listed as part of your local library.

(2014-12-29, 08:52)steve1977 Wrote: Have the developers decided to push UPNP forward and eventually abandon mysql for this purpose?
Ideally yes but it's not really limited to UPnP.
Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
Reply
#6
http://forum.kodi.tv/showthread.php?tid=200911&page=12

This thread is pretty long and all about redoing the database to allow far better usage for those of us that utilize SQL centralized database. Also has great side effects like parental controls and Plex-like functions.
Reply

Logout Mark Read Team Forum Stats Members Help
DB's central servers improvments0