HOW-TO syncronize libraries between multiple XBMC via Dropbox - An Experimental Guide
#1
Thumbs Up 
Overview:
Over the past week or two, I've been experimenting with using Dropbox to sync my xbmc library over multiple computers and I wanted to document my experiences. This "guide" is for windows (although could easily be adapted to work on linux & osx) and is HIGHLY EXPERIMENT AND DOESN'T WORK 100% OF THE TIME! If you're not into experimenting/tinkering/trying out stuff that might not work... well then, what the heck are ya doing on the xbmc forums in the first place!? Anyway this setup has been work pretty well for me for the past week, but because of the way dropbox handles conflicting files, I still felt that warning was necessary. Now first off, so that you can understand why this works for me, let me explain how I use XBMC...

My XBMC Setups:
I've got two windows 7 machines running xbmc in my house. Machine 1 (which i will call my server) is always on and doing something (fairly powerful dual core athalon). Machine 2 (which I will call my node) is in the bedroom and is a tiny Acer Aspire Revo R1600 (single-core) ION-based pc. The node, as opposed to the server, is put to sleep whenever it's not needed. Also, the sources on both machines are identical (but not synced). All my media files are on a samba share, so the paths are the same on both machines, which is vital for this to work. The important thing to know about my setup is that 98% of the time, I only have one instance of xbmc running at one time. So file conflicts (i.e. both machines writing to the library at the same time) are a rarity for me (but does happen from time to time). The good news is that dropbox will never write to a file that is in use on the machine, so there is never a fatal error that will lock up xbmc and corrupt your library. The bad news is that dropbox will at no point try and merge the files (some might see this is a good thing too, since dropbox doesn't speak SQL and would probably screw up if it tried), so if two machines write to the library at the same time, you will wind up with 2 copies of the offending library .db file in your dropbox/database folder. One will be named correctly and one will have some extra text appended to it (https://www.dropbox.com/help/36). I'm hoping to do some more testing of both machines running at the same time, this week.

Dropbox Setup:
First off, start by picking one machine to work from for now. Update your library, make sure you thumbnails show up correctly on one machine. This whole process will go a lot smoother if you start with a complete and functioning library as opposed to trying to combine 2 or 3 of them or trying to set up syncing before building the library. You'll be able to update the library from any machine after this setup is complete, but it's definitely easier if it's as complete as possible to start with.

The trick to getting dropbox working with XBMC is symlinks. My first instinct was to just create a symlink in the dropbox folder that linked to XBMC's database folder, but this will NOT work. Dropbox will update the files there, but only when the machine (or the dropbox app) is restarted. So instead, you have to copy the Database & Thumbnail folders to your dropbox first, and then create hard symlinks (/J) in xbmc's userdata folder (if you don't maintain a library, and are only concerned with the watched/unwatched flag & resume times, then you probably don't need to sync the thumbnails folder, but if you scrape your library and have cover art, then it's a must... more on that later). I also renamed the Database & Thumbnail folders to XBMCDatabase & XBMCThumbnails once they were in my dropbox, just to make them a little more distinctive.

Helpful Tip: if your other machines all have similar setups to the one you're working from now, make a little batch file to rename or delete the old folders and create the symlinks, and put it in your dropbox. This will make setting up the nodes a lot easier later on, since all you'll have to do is install dropbox, let the files sync, and then double click the batch file. My batch file looks like this (keep in mind, I use XBMC in portable mode, so your userdata folder will probably be in a different location).
Code:
rename "C:\Program Files\XBMC\userdata\Database" "C:\Program Files\XBMC\userdata\Database.bak"
rename "C:\Program Files\XBMC\userdata\Thumbnails" "C:\Program Files\XBMC\userdata\Thumbnails.bak"
mklink /j "C:\Program Files\XBMC\userdata\Thumbnails" "C:\Users\[myusername]\Documents\My Dropbox\XBMCThumbnails"
mklink /j "C:\Program Files\XBMC\userdata\Database" "C:\Users\[myusername]\Documents\My Dropbox\XBMCDatabase"

(on the first 2 lines, you could also do a "del /s /q" on those folders instead of a rename, which is what I did, but this is a bit safer)

Now if you have a large library, dropbox is gonna take a while to upload all your thumbnails. Also you should be weary of how big that thumbnail folder is (mine was just over a gig, luckily i had enough free space in my dropbox to accommodate it). But you should be able to use XBMC on this machine to verify that the symlinks are working (even while dropbox is still syncing).

IMPORTANT: You should let dropbox finish its initial sync before you attempt to set it up on another machine. It's not that dropbox would cause an error, but if you try and run XBMC before you're thumbnails have all been uploaded & synced, then you're probably gonna wind up with a whole bunch of useless .tbn files that are random frames from the videos (cause xbmc will create the ones that are missing instead of reloading them from their source.

After the initial upload, setting up your other xbmc nodes should be pretty easy. Just install dropbox, let it is sync (the downloads should be a lot quicker than the uploads), and double click the batch file (or create your own symlinks manually).
IMPORTANT: When setting up your nodes, make SURE that dropbox has finished it's initial sync BEFORE you try to run XBMC, to avoid the same problem described above (losing thumbnails).

After that, you're done. You now have 2 (or more) machines synced to the same library and with all thumbnails. You should be able to start playing something on one machine, stop it, and pick up where you left off on the other machine, within a matter of a few seconds. You should also be able to add or delete from the library on one machine and see the changes on the other (as long as both machines aren't making changes at the exact same time).

Real World Usage, Caveats and Concessions:
-Syncing the Thumbnails folder: When I first set this up, I was only syncing the Database folder. But what i noticed was when I updated the library on one machine (adding new content), the data would appear on the other machine, but none of the images would (even if the .tbn's were on the shared drive with the movies). To get the images to show up I would have to refresh the info for each movie/tv show that wasn't showing artwork (doing a normal library update would not solve the problem). This is when I started syncing the Thumbnails directory as well.

-Updating the Library: When updating the library, make sure all other instances of xbmc are not running! I have no problem with playing video on multiple boxes at the same time, since if there is a conflict the result will be nothing more than a missing resume point or watched flag, but if I'm updating my library on one machine, I'm making damn sure that the others are OFF, just to be safe.

-Putting to Sleep: So I usually put my xbmc node to sleep when I'm done watching, but here's the problem... I've configured eventghost to handle the power button on my remote, so that it always attempts to stop any playing video before going to sleep. This is great because I can just press power and xbmc will save the resume point, however I noticed that the machine was going to sleep before dropbox was finished uploading the diff created by that tiny database update. To rectify this, I altered my eventghost config on that machine to pause for 15 seconds in between stopping the video and actually going to sleep. This seems to do the trick for me, but your mileage may vary based on the speed of your net connection & the speed of your pc (remember, dropbox only uploads binary diffs when changes are made to files, not the complete file, so the upload time is pretty small when the only thing that changed was a watched flag and a resume point).

-Duplicate files: I haven't yet played too much with file conflicts as I've only encountered a small handful. But to keep my dropbox tidy I scheduled a batch script on my server to run at 4am every day and delete any duplicate files that dropbox created. ll my batch file does is "del *(*).db" (it's stored in the XBMCDatabase folder in my dropbox, and scheduled with Windows Task Scheduler). But I definitely need to do more testing with this feature. What might be a good idea is to have a script or small app determine which version is bigger and delete the smaller one (and then make sure the leftover one is named properly), so I might try and throw a little .net app together to do this.
Reply
#2
(had to split my post)

Conclusion:
This is clearly not a perfect solution, but for someone with a similar configuration and/or usage patterns, the simplicity of the set up could make it the perfect "good-enough" solution until we finally have a centralized way to manage our xbmc libraries. After all, I've been using it for a while now, and it works pretty darn well for my needs.

Pros:
-Will NOT cause errors when two machines are writing to the same database file at the same time (https://www.dropbox.com/help/36)
-Will keep full library in sync across all machines including watched flag, resume times and links from movies to TV shows
-You can run a library update from any machine (not just a main one) and see the changes on all other machines (I know that's pretty much the definition of syncing, but I wanted to differentiate between that and a centralized management machine)
-Syncing is near-real time
-Syncing is pretty fast even on internet connections with a slow upload speed, since dropbox uploads binary diffs instead of complete files (https://www.dropbox.com/help/8), also downloads are performed over your LAN if the file exists there, this is a Dropbox feature I was unaware of until today (https://www.dropbox.com/help/137)
-Fairly simple to set up (depending on your xbmc configuration)
-Especially good if you only care about the watched/resume flag (i.e. you don't use the library, but just file mode)
-Even with multiple boxes playing at the same time, sync will usually work, as long as videos aren't stopped at the exact same time (since that seems to be when xbmc writes to the database)

Cons:
-You kind of need to make sure only 1 xbmc instance is running when you do a library update (i.e. adding new content)
-Not ideal if multiple boxes run their own library updates at the same time
-Internet based (but downloads are done over the local network)
-Initial set up takes a long time if you've got a big library with lots of thumbnails
-Depending on the size of your library, can take up a lot of space in your dropbox
-Depends on uptime of a 3rd party service


I strongly encourage anyone who has tried this type of thing on any platform to share your tips and experiences. Whether using dropbox or another method to keep your libraries in sync across multiple PCs (Anyone tried this kind of thing with SugarSync? Has anyone tried running SugarSync & Dropbox on the same machine?). I don't think I'm the only one who'd be interested in experimenting with alternatives. Also feel free to ask me questions or ask me to try out different scenario's, and I'll do my best to accommodate and report back. I will also continue to update this post as I continue to sync via dropbox.
Reply
#3
Thanks for the handy guide. I do this now with sugarsync on my windows machines, but I'm getting ready to add an Aspire Revo to the bedroom for XBMC. I'm most likely going to run XBMC Live as that seems to be the best way to go for the revo, but I want to use dropbox to sync it with the other computers. Anybody have any suggestions on using drop box with XBMC Live or am I going to have to use a minimal Ubuntu install if I want to use dropbox?
Reply
#4
I've got a similar setup to yours (NAS, 1 always on PC, revo on the TV, both on Win 7), and in a similar attempt to keep the libraries synced I've just been using the NAS and a bunch of symlinks, instead of drop box

I made a folder on the NAS called XBMC_LIB to keep all the XBMC related stuff, then created symlinks on both machines to point to them. I also recently switched to using a MYSQL database since I had MySQL running on the main PC anyway, but this was working fine with the regular DB as well.

Figured I'd post my method as well if anyone wants to give it a try:
Code:
mklink /d "C:\Program Files\XBMC\portable_data\userdata\Database" "\\naspath\XBMC_LIB\Database"
mklink /d "C:\Program Files\XBMC\portable_data\userdata\Thumbnails" "\\naspath\XBMC_LIB\Thumbnails"
mklink /d "C:\Program Files\XBMC\addons\plugin.program.launcher" "\\naspath\XBMC_LIB\plugin.program.launcher"
mklink    "C:\Program Files\XBMC\portable_data\userdata\sources.xml" "\\naspath\XBMC_LIB\sources.xml"
mklink    "C:\Program Files\XBMC\portable_data\userdata\plugin_data\programs\Launcher\launchers.xml" "\\naspath\XBMC_LIB\launchers.xml"

Keeps the databases for the library and thumbnails synced up, as well as the sources since they are the same on all PCs, and the launcher files are for the launcher addon for my emulators, so those lines can be removed if you're not using that.

I make no guarantees, but its been working fine for me for awhile now.
Reply
#5
ln -s is the right way for MacOS? or should i use another command?

it is! Wink tried it just yet and everything works fine. now i have thumbnails everywhere. and: i prefer the mysql database way Wink i only use dropbox fior syncing the thumbnails.
-= XBMC Lover © 2006 =-
---------------------------------------
XBMC @ Lenovo Q180, harmony one, Sony Bravia KDL-55W905, Marantz SR5006, MySQL XBMC Database @ Synology DS-409+ (thx Firnsy)
XBMC @ Lenovo Q150 and a Panasonic Plasma for bedrooming ;)
Reply
#6
why is not thumbnails working in the "mysql" way?
Image
Reply
#7
Because thumbnails are always stored on the local machine, independent from the SQL database (which only equals to the original MyVideos34.db file). That is also the reason why you need for each .db a respective SQL database if you not only want to have a central video database, but also one for music and so on.
Reply
#8
I use a combination of both. MySQL for databases and Dropbox for the Thumbail folder.
I run Live, Win7 and a MacOS X client. This way there are no database conflicts and navigation is instant. I really wish there was an easier and more elegant way.
Reply
#9
maxinc, you've got XBMC Live to work with Dropbox?

If this is true can you share how you got it to work? I'm struggling to get the Thumbnails folder to sync from Win 7 smb share to Live.

thanks
Reply
#10
amsquires83 Wrote:maxinc, you've got XBMC Live to work with Dropbox?

If this is true can you share how you got it to work? I'm struggling to get the Thumbnails folder to sync from Win 7 smb share to Live.

thanks

http://wiki.dropbox.com/TipsAndTricks/Te...nuxInstall

This is what I've followed and it worked first time. Mind you I had to google "tar" and "wget" as I'm totally worthless in linux but it worked like a charm.

As for the library sync, I moved the Thumbanail folder inside the Dropbox foler and created symbolic links instead inside ~/.xbmc/userdata.
XBMC Live: i3 530 / GT210 / 2GB / SSD + 2 x Zotac HD01 / 2GB / SSD
unRAID Pro: 6 x 1TB + 2 x 1.5TB + 2 x 2TB + 2 x 500G over GbE
HP Micro Server: SABnzbd+, Sickbeard, Couchpotato, uTorrent, Media Companion, MySQL, MKV Toolnix
Reply
#11
thanks a bunch! I didn't realise that a non gui version of dropbox was available. I've got it working with a symlink to a smb Thumbnails share.

Last outstanding task... I have tried both methods under "Running on system startup" to have dropbox automatically start with XBMC Live, but neither were successful. Is this something that you have attempted?

For future readers of the thread I'm running the xbmcfreak version on a Revo 3600.
Reply
#12
I've added this line to the /etc/rc.local as sugested on that page.

/root/bin/dropbox.py start

Obviously, you have to replace the path with the location where you downloaded the dropbox.py script. It was /root/bin in my case.
XBMC Live: i3 530 / GT210 / 2GB / SSD + 2 x Zotac HD01 / 2GB / SSD
unRAID Pro: 6 x 1TB + 2 x 1.5TB + 2 x 2TB + 2 x 500G over GbE
HP Micro Server: SABnzbd+, Sickbeard, Couchpotato, uTorrent, Media Companion, MySQL, MKV Toolnix
Reply
#13
amsquires83 Wrote:maxinc, you've got XBMC Live to work with Dropbox?

If this is true can you share how you got it to work? I'm struggling to get the Thumbnails folder to sync from Win 7 smb share to Live.

thanks

I got dropbox working on openelec, however I imagine the procedure is near identical to XBMCLive

I followed this guide on installing dropbox through the CLI

http://wiki.dropbox.com/TipsAndTricks/Te...nuxInstall
Reply
#14
Here is a link that I used as the basis of my setup which includes using a central SQL database for all the instances, and a single dropbox directory used to sync each of the thumbnail directories on each xbmc instance.

http://xbmcstuff.tumblr.com/post/7400497...pbox-style

I am using a mixture of xbmc-live (sam's crystalbuntu image), xbmc on OSX, and XBMC on linux/ubuntu.

I made a couple of changes from the guide;

1) I had to change the permissions on the crystalbuntu image as they use the user atv and has root permissions on the /root/.xbmc/userdata/Thumbnails directory.

If you're signing up for Dropbox use this link and you'll get an extra 250mb of space http://db.tt/ksxR7pF

Regards.
Reply
#15
How did you get a network path to work? I also have an NAS on my network and would like to server my XBMC stuff from there. When I run mklink command I get an error saying Local files are required. Thanks!
Reply



Logout Mark Read Team Forum Stats Members Help
HOW-TO syncronize libraries between multiple XBMC via Dropbox - An Experimental Guide0
This forum uses Lukasz Tkacz MyBB addons.