[DecK]

Based on the discussion here and my own disappointment with the performance of symlinks for thumbnails, I decided to write an addon to keep data synchronized between multiple XBMC machines.

The code is available here:
https://github.com/declankenny/XBMC_datasync

Brief
Create an addon that allows the user to synchronize data between one master (server) XBMC instance and multiple client instances.

• The addon shall allow the user to specify the frequency of updates.
  
• The addon shall allow the user to specify which data to synchronize.
  

Usage

• The addon needs to be installed on both the server and client instances.
  
• Don't use this with anything that is a symlink or a path replacement in advanced settings.
  
• You must have one, and only one, instance where the "Mode" setting is "Server". On that instance, you must Allow control of XBMC via HTTP in the network settings.
  
• Only data for the master profile and common data will be synchronized.
  

Settings

• General
  • Mode - Server or Client
    
  • IP Address - Only available for client instances, refers to the IP address of the Server instance.
    
  • Port - Only available for client instances, refers to the HTTP port of the Server instance.
    
• Items to Sync - options only available when mode is client
  • Thumbnails
    
  • Media Sources - both sources.xml and mediasources.xml
    
  • Addons - addons folder, addons database and userdata/addon_data
    
  • RSS - RssFeeds.xml
    
• Schedule - options only available when mode is client
  • Period - Sets the base frequency for pulls from the server instance. Can be Hour, Day or Week
    
  • Hourly Frequency - How often to pull the data. Only available if the Period = Hour
    
  • Day of the Week - Which day to pull the data. Only available if the Period = Week
    
  • Staring Hour - When the period is Day or Week, this is the hour at which the data is pulled. When the period is hour, this the the seed hour, and will be incrrmented by the Hourly Frequency setting.
    
  • Staring Minute - The minutes after the hour when the data is pulled.
    

Cheers
Declan

[robweber]

This is a really great idea, I can see where lots of people with multiple setups would like this.

Regarding the timing feature. I see you allow in the settings to sync thumbnails, sources, addons, and rss. Have you given any thought to checking the modified time on the "master" for a change and updating based on that instead of a timer? Granted some people may prefer the timer over that method; however this would ensure the most timely updates. I belive os.path.getmtime would return the right info, you'd need to store the last modified timestamp persistently though for comparison.

Also, looking at rootPath function you have in a few places that is based on the OS. Will you be able to handle userdata folders in non-traditional locations, ie portable mode? The Windows path could also be Documents and Settings\username\Application Data\XBMC if using WinXP. You may need two different OS entries for Windows XP vs Vista/7. Since you could use the special:// protocol and the xbmc.translatePath python function to get the exact userdata location you might be able to get away with not hardcoding locations altogether. It may be a little harder to get the paths to the target folders but since you'll have to communicate with them somehow to send the files you could have them pass that information back to you, or handle the path translation themselves.

Overall a nice start, looking forward to a working prototype.

[DecK]

Rob
I hadn't thought about an on-demand option, but it is a good idea. Scanning the thumbnails folder for changes worries my from a performance perspective, mine has 12K files, but I will run some tests. Give the user more options can't hurt. I've added it to the issue list in github.
Yes, remote folder locations is a problem, and is part of a much bigger one - the ability to communicate with the targets. Once I get that address, which is the current show stopper, I hope to be able to be more effective in the target folder calc.

Thanks for the input.

[ppic]

you can have an addon on the other computer that can be launched with parameter via jsonRPC to make a dialog.

[DecK]

Not sure I like the idea of the user having to install a seperate addon on the targets. Just had a look at some of the jsonRPC pages over on the wiki and can already tell that I'm in over my head...
But I'm wondering if I could use a remote call (is that even the right expression) from the addon to the target using json to get the OS and root path info? that would at least address Rob's concern.

[robweber]

Ok, here is a thought that would totally reverse how you are thinking about the data sync, but may provide a solution.

What if instead of the master pushing files to the targets, the targets can pull files from the master? This would provide a mechanism for an unlimited number of targets (instead of having separate addon settings for each one) and allow you to use the jsonrpc as a transport mechanism.

I guess what I'm thinking is that you could use the Files.GetDirectory function to get a list of files in a directory and then use Files.Download to actually get the file from the master and store it on the target. The target could use the special:// protocol to translate the path and make sure it is put into the correct location. I'm fairly certain that the jsonrpc commands will recognize the special:// syntax so from the target you could use something like special://masterprofile/sources.xml and it would give you the correct file on the master xbmc instance. You could use http as the transfer mechanism to download the file within python (Files.Download returns an http location)

Granted there are probably some details I'm missing here but in theory this sounds like it might work. Would also solve the problem of trying to figure out a way to get files from a->b reliably cross platform.

[Bstrdsmkr]

DecK, you mention that getting the remote folder is the current show stopper. I'd be glad to help out, but if you could be more specific in asking targeted questions, I could give you better advice.

From the sound of it, I'm of the same opinion as ppic. The most robust (and easiest on your sanity) is going to be a service addon installed on both source and target. In your service settings, have an option to switch between client mode and server mode. The easiest way to communicate between instances is going to be the JSONRPC, although I'm still learning how the documentation is lain out myself. I would say in your service loop, if it's set to server mode, check the mtime of all monitored folders. If the mtime of a folder has changed since the last loop, loop through the list of clients and signal each one which folder has changed. If you're in client mode, wait for a folder changed signal, pull the new contents from the instance that signaled you, then signal back once the update completes.

A setup like that gives you good control over how many instances are pulling from the server at once, when they're pulling from the server, and if something happens to the server, just switch one of the others into server mode.

Even if you have to walk every file in the directory, it's not terribly resource intensive. My addon has a function that walks down a directory and verifies that the files match what's in a database and about 8gb worth TV Show/Movie metadata takes less than 5 sec.

[DecK]

I had considered a pull rather than a push approach, but went with the latter becasue of a logic issue.
1/ The sync, regardless of push or pull is left to right, [if new or change on left then copy, if only on right then delete].
2/ I really want to include addons in the sync.
3/ This is an addon

With a pull model, won't the client/target/right side versions of this addon get overwritten every time the sync is executed? that would override any local settings, make every instance look like the master, etc...

I'm not saying that pull won't work, just that I can't get my head around part of it. Again, this is supposed to be a learning exercise.

I'd be glad to help out, but if you could be more specific in asking targeted questions, I could give you better advice.

OK

1. How, in python, do a inititiate a call to a remote instance of XBMC (identifed by a IP4 address) to return:
   
   1. the OS of the remote instance.
      
   2. the full path to the userdata folder on the remote machine
      

Again, not dismissing the pull approach, just trying to see if I can get the push to work first.

[robweber]

I had considered a pull rather than a push approach, but went with the latter becasue of a logic issue.
1/ The sync, regardless of push or pull is left to right, [if new or change on left then copy, if only on right then delete].
2/ I really want to include addons in the sync.
3/ This is an addon

With a pull model, won't the client/target/right side versions of this addon get overwritten every time the sync is executed? that would override any local settings, make every instance look like the master, etc...

I'm not saying that pull won't work, just that I can't get my head around part of it. Again, this is supposed to be a learning exercise.

Addressing the add-on getting overwritten, yes the addon core python files will get copied. The settings however are stored in a separate addon_data folder under the userdata directory. If the master/target settings and any other add-on data you need is here, and not overwritten, then you shouldn't have any issues copying the python files and causing the addon to stop working. It would almost be the same process that XBMC would use if doing an addon update from within. It would unpack the new addon and replace the current files with the new ones - the settings don't get touched as they are in a separate space. You'll have to be selective on which files you are copying, but if you ignore the addon_data directory each xbmc instance could have the same addons with different settings.

Regarding getting OS and the user data directory from the JSONRPC, I don't think there are calls for that type of information. Bstrdsmkr may have more insight but as I understand it the jsonrpc and the virtual file system are supposed to abstract that kind of info so you don't need to know it (ie can't).

One last thing that I just thought of is depending on the files being synced you may need to restart the target XBMC instance in order for some things to take effect. For instance the sources.xml file only gets read upon startup (or when you change it within the xbmc gui) so simply copying the file won't add the new sources unless xbmc is restarted so that it will read them back in. After some experimenting you could probably figure out which operations require a restart and code the add-on either do an xbmc restart, or prompt for one.

[DecK]

OK, I've come around, lets go with a pull approach. I am still going to focus on a scheduled approach for the first release. Triggered updates can wait for now.

Rob - my intention was to copy all the addon data, including user settings and the db. I will add an exclusion rule for this addon if necessary.

In the settings, I already have the ip for the 'sever'/master instance. Will I need the username and password for a json connection or will the ip suffice? Do I need a port number as a user setting?

I think, once I can establish a json connection and retrive a file list (I assume into a list of some sort of dictionary object), it should just need a rewrite of the Synchronize Function to compare lists rather than directories.

So, any good samples of opening a json connection in python and getting a file list?

[Bstrdsmkr]

1. How, in python, do a inititiate a call to a remote instance of XBMC (identifed by a IP4 address) to return:
   
   1. the OS of the remote instance.
      
   2. the full path to the userdata folder on the remote machine
      

Again, not dismissing the pull approach, just trying to see if I can get the push to work first.

1. Adapted from http://www.voidspace.org.uk/python/artic...lib2.shtml:

PHP Code:

import urllib
import urllib2
username = 'user'
password = 'pass'
host = '192.168.1.1'
port = '9090'
url = 'http://%s:%s@%s:%s/jsonrpc' %(username, pass, host, port)
user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
values = {"jsonrpc": "2.0", "method": "AudioPlayer.PlayPause", "id": "1"}
headers = { 'User-Agent' : user_agent }

data = urllib.urlencode(values)
req = urllib2.Request(url, data, headers)
response = urllib2.urlopen(req)
the_page = response.read() 


I'm doing this on the fly and about to leave, but that should get you started, you can set any necessary headers in the headers dict change the JSON string in values as needed. Reference http://wiki.xbmc.org/index.php?title=JSON-RPC_API/v4 to see what you need to change "values" to to get the desired result

[robweber]

Here is a post with a python class for sending a json command via http to another xbmc instance:

http://forum.xbmc.org/showthread.php?tid...#pid678603

[Bstrdsmkr]

Also, the userdata folder on the remote machine will always be special://userdata

Not sure you can get the OS, so you may have to configure it in settings. I think the easy approach is to feed your service the general information and let it figure things out locally.

[DecK]

Thank you both. Guess I won't be getting any sleep tonight....

[robweber]

Just checking back to see if you've made any progress on this? I'm watching your github repo and have seen a few commits.