Frodo - Shared Library Thumbnail Updates & Mulitple PCs

  Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
greenman Offline
Junior Member
Posts: 11
Joined: Aug 2007
Reputation: 0
Question  Frodo - Shared Library Thumbnail Updates & Mulitple PCs
Post: #1
Hi All,

I have a number of PCs in my house running XBMC 11 and am looking to upgrade to 12 but I am concerned with the new library update changes, the removal of the path substitution and how it will different PCs updating the library and thumbnails.

Currently the PCs are turned off when not in use but they all update the library when XBMC first loads. TV/Movies etc. are on the NAS, as are all the thumbnails (using path substitution on each XBMC instance). This means everything is in sync because of the central database and central thumbnail storage.

The upgrade instructions for Frodo (http://wiki.xbmc.org/index.php?title=HOW...BMC_to_v12) explain how to remove the path substitution and copy everything locally and then it will keep it in sync itself but it doesn't explain how this actually works and I don't see how it can.

What if all the computers aren't turned on or the computer that did the update is then turned off when one of the others is turned on? The library will be up to date but the thumbnails missing I am guessing until that PC is turned back on at the same time as one of the others.

A good example is our home theatre which is used rarely compared to the living room. If I start XBMC and it updates the library and the computer is turned off for a few weeks then none of the other instances will "sync" the thumbnails stored locally on that PC only.

It appears to me that I'd need to somehow run a headless XBMC instance that is updating the library all the time on some type of server to have a centrally fully up to date database and then not do any updates on the actual clients.

Am I misunderstanding something with this new implementation?
Can path substitutions still work in Frodo (exactly the same as Eden) to get around this problem?

Regards,
Andrew
(This post was last modified: 2013-01-31 03:27 by greenman.)
find quote
Nitroburner77 Offline
Junior Member
Posts: 40
Joined: Sep 2011
Reputation: 0
Post: #2
If I'm understanding your question correctly, it sounds like you are using MySQL for the centralized database. In Frodo, the database logs the location of the originating artwork, so if one device updates the library, when you turn on another and go to a view that shows the new media without art, XBMC goes right to the path set in the database and collects it. This is quite different from previous releases where if you updated the library on one machine you had to manually sync the others or select the movie/show and click refresh/update or whatever it was. On a side note, if you change the art on one machine with either online or local art, the new art will be collected from the original source (NAS for you, since that is where you store it) and updated automagically on the other devices.

I'm a low post newbie, so somebody please correct me if I am incorrect or if it isn't working for everybody.
find quote
greenman Offline
Junior Member
Posts: 11
Joined: Aug 2007
Reputation: 0
Post: #3
(2013-01-31 04:24)Nitroburner77 Wrote:  If I'm understanding your question correctly, it sounds like you are using MySQL for the centralized database.

Yes I am using mySQL, apologies I should have mentioned this in the initial post.

(2013-01-31 04:24)Nitroburner77 Wrote:  In Frodo, the database logs the location of the originating artwork, so if one device updates the library, when you turn on another and go to a view that shows the new media without art, XBMC goes right to the path set in the database and collects it.

This is my big issue and question. The machine that performed the update has the thumbnail/banner/fanart locally for the newly updated library entries may not be turned on so there is no way any of the other xbmc instances can access the files so they won't be able to copy them locally. This doesn't happen currently because the artwork is stored and accessed on the NAS which is always turned on.

(2013-01-31 04:24)Nitroburner77 Wrote:  This is quite different from previous releases where if you updated the library on one machine you had to manually sync the others or select the movie/show and click refresh/update or whatever it was. On a side note, if you change the art on one machine with either online or local art, the new art will be collected from the original source (NAS for you, since that is where you store it) and updated automagically on the other devices.

No you didn't, there was no manual syncing at all. With path substitution and mySQL every instance looked at the same database and artwork location so it was always up to date. The library update on startup which I run on all the XBMC instances would scan the NAS for new shows or movies and update the library.

I can install XBMC on a new PC (or Mac), update the advancedsettings.xml with the mySQL and path substitutions and it is instantly up to date with the central library and artwork, watched flags etc.

Updating the art (say a new movie cover) works the same, update it on one PC then its updated on the others since they are all looking at the same thing.

I understand that the new system would be faster as everything is access locally (not that I have any issues with the current system) but I can't see how any of my eight XBMC instances would ever have a complete copy of the artwork.
find quote
jmarshall Offline
Team-XBMC Developer
Posts: 26,221
Joined: Oct 2003
Reputation: 178
Post: #4
Quote:The machine that performed the update has the thumbnail/banner/fanart locally
.

No, it has a local cache of the image which is not shared at all. This is completely separate from the shared database. The original URL of the image is kept in the shared database. See the art table.

Your other clients have the original URL from the shared database, and create their own local cache of the image automatically.

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


[Image: badge.gif]
find quote
greenman Offline
Junior Member
Posts: 11
Joined: Aug 2007
Reputation: 0
Post: #5
(2013-01-31 05:15)jmarshall Wrote:  
Quote:The machine that performed the update has the thumbnail/banner/fanart locally
.

No, it has a local cache of the image which is not shared at all. This is completely separate from the shared database. The original URL of the image is kept in the shared database. See the art table.

Your other clients have the original URL from the shared database, and create their own local cache of the image automatically.

Ahh, now that makes perfect sense. I didn't think of looking at the new database table schema. Thanks for the clarification.

Not that it really matters but it is worth pointing out that with it working like this each XBMC client will need to download all the missing artwork each time from the internet (using the URL stored in the database) so if you have 10 clients then you'll be downloading all the artwork 10 times. Am I understanding that correctly? This will also mean that if you add a new instance in say 5 years some of the URLs may no longer work, what happens in this case?

If my understanding is correct an additional feature could be added to have a local central library that all machines use to update their local cache to avoid the internet downloading. The URL stored in the database could be the image location on the NAS.

Appreciate your time.
(This post was last modified: 2013-01-31 05:30 by greenman.)
find quote
greenman Offline
Junior Member
Posts: 11
Joined: Aug 2007
Reputation: 0
Post: #6
I thought about this a little more. If you wanted to create a new XBMC instance then you'd just copy the cache off an existing machine.

It's quite possible that the URL stored though may no longer exist if a machine wasn't turned on for a period of time so what would you do in this instance? We have a couple of machines that may not be turned on for 6 months.
find quote
robweber Offline
Posting Freak
Posts: 935
Joined: Sep 2009
Reputation: 27
Post: #7
You are really overthinking this in my opinion. Artwork URLs should exist for over 6 months, at least from the sources xbmc is scraping from. Ive been using the new cache system almost as soon as it was available in a nightly build. My raspberry pi (secondary xbmc in my house) is constantly being wiped as a try new things and i have yet to see miss any artwork when rebuilding the library.

As a side note. If you export your library, it will also export the artwork. If you ever need to reimport or find specific artwork you can get it locally then. If you are super paranoid just do an export every few months.

The new cache system really works great in my opinion.
find quote
greenman Offline
Junior Member
Posts: 11
Joined: Aug 2007
Reputation: 0
Post: #8
Thanks for your thoughts. I am possibly over thinking it, the XBMC team have always had well thought out solutions so I doubt this time is any different.

Having 9 machines to update with a few different operating systems I wanted to understand how the new system works before fully I started the upgrade process.

Thanks to everyone who contributed to this thread.

Thanks XBMC team for a wonderful piece of software.
find quote
grumbler Offline
Member
Posts: 58
Joined: Jan 2010
Reputation: 0
Post: #9
Excuse me if I jump into this topic, but I am really worried about what I am reading here:

(2013-01-31 06:20)robweber Wrote:  Artwork URLs should exist for over 6 months, at least from the sources xbmc is scraping from.

But what if the artwork was never scraped by XBMC, therefore there are no URLs?
I usually retrieve the artwork by external tools (ember or MediaElch), some of it by scraping the well known medie sites as XBMC would do, but some others by using pictures I downloaded myseld and simply putting
them in the movie directories as poster.jpg or fanart.jpg or whatever.

These files are used by the Transparency-Skin of my XBMC without any problems, there is no way any of this images is connected to a valid URL.
The Thumbnail-Cache is currently synced between 3 different XBMC-Installations by a robocopy script.

So my question: If I add an new fanart.jpg into a movie directory on machine X (just by storing the file in the movie directory, not scraping it from a URL), will it ever be visible on machine Y? As machine Y can not scrape it itself from a URL in the internet, were will it get the image from?
find quote
Kib Offline
Team-Kodi Server Dude
Posts: 3,674
Joined: Jan 2010
Reputation: 55
Location: NL
Post: #10
(2013-01-31 08:54)grumbler Wrote:  So my question: If I add an new fanart.jpg into a movie directory on machine X (just by storing the file in the movie directory, not scraping it from a URL), will it ever be visible on machine Y? As machine Y can not scrape it itself from a URL in the internet, were will it get the image from?
If you have the fanart on a movie directory on machine X and machine X is turned off, you should be worrying about where machine Y will get the movie itself from.

I could make a living out of tuning MirrorBrain. 86 million requests per day.
find quote
robweber Offline
Posting Freak
Posts: 935
Joined: Sep 2009
Reputation: 27
Post: #11
(2013-01-31 08:54)grumbler Wrote:  So my question: If I add an new fanart.jpg into a movie directory on machine X (just by storing the file in the movie directory, not scraping it from a URL), will it ever be visible on machine Y? As machine Y can not scrape it itself from a URL in the internet, were will it get the image from?

The image path does not have to be an http URL. As an example, when I decided to move from SQLite to the MySQL database structure I did an export of my library, which exported all artwork into the movie/tv show folders along with NFO files. Once I had MySQL working I did a rescrape of the library to get the contents into MySQL. The URLs to the image paths were now nfs://ip/path/folder/movie/whatever instead of http://website/imagepath. Looking in the artwork table of the database all the image paths just point to local network images. As I added new media some of these paths became web URLs, others local paths as I added custom artwork.

The media location doesn't matter so much as long as XBMC can a) read the path from the DB and b) the path exists. As Kibje mentioned, if machine X is turned off and that is where you movies/artwork are; you'll have quite a few problems.
find quote
Pr.Sinister Offline
Fan
Posts: 710
Joined: Sep 2007
Reputation: 15
Location: Montreal, Canada
Post: #12
I am using MySQL with 1 TV Server, 1 Movie Server and 1 Music Server. I also have 4 XBMC Clients. I have all my posters, fanart and thumbnails in the source folders. For anyone using a central database, i believe this is the best way to go about it.

I upgraded to Frodo yesterday but decided NOT to go the extra safe way like explained in the upgrade FAQ. At first, i didn't want to have to copy my 12 gig thumbnail folder to the XBMC clients that have small SSDs and especially not to my Raspberry PI! I figured that over the years, some of those thumbnails were obsolete.

When XBMC started the Video Art Upgrade, i saw this would be a lengthy process so it definitely stuck to my original idea of just letting all the clients generate their own art.

This worked just fine. While the main DB was being upgraded, i fired up the clients and started navigating and the artwork slowly got generated.

For those of you with large libraries like myself (4250+ Movies, 785+ TV Shows, 30000+ MP3s), here is a list of how long different steps of the conversion took (all on the main server) :

Updating the database Textures13 from version 6 to 13 : 1 Minute
Updating the database xbmc_music32 from version 18 to 32 : 45 Minutes
Updating the database xbmc_video75 from version 60 to 75 : 49 Minutes

Updating the Movies & TV Shows Art : 4 Hours 56 Minutes
Updating the Actors Art : 5 Hours 41 Minutes
Updating the Music Art : Started at 10:19am and it is now 11:35am and it is 40% done.

This is on an i5-750 with 8 GB of RAM.

-Pr.

------------------------------------------
[4 XBMC Clients + 3 Norco RPC-4224 Media Servers w/87 TB HDD Space]
Do you know what the +/- button does? :)
(This post was last modified: 2013-01-31 18:38 by Pr.Sinister.)
find quote
BigO Offline
Senior Member
Posts: 168
Joined: May 2010
Reputation: 0
Post: #13
PR i feel your pain, i really dont want to do local images since my XBMC client are using USB drives or small disks. I really wanted to keep all my images on a larger SSD that i have on my machine that holds my DB and does the auto update since its always on. Ive been tinkering with using the path substitution but ran into some issues since the machine in question is running LION.

I need to upgrade to full FRODO this weekend so will do some other testing with path substitution on my nas but directly to another SSD drive to see how that pans out.

XBMC-ATV2 - Frodo
XBMC-LR - Zotac Zbox Plus - 36gb SSD - 3Gb Mem - Frodo
XBMC-BR - Foxcon 330i - 16gb SSD - 2Gb Mem - FrodoRc3
XBMC-Mini - MacMini - 128Gb OCZ Agility - 5Gb Mem - Frodo - Running SQL - Auto Library Update
Synology DiskStation Ds1512+(Dsm 4.3-3776) - 4gb Mem - 5*3tb Wd Reds [SHR] - 2GB Bonded Link - (SSD Cache WIP)

find quote
jmarshall Offline
Team-XBMC Developer
Posts: 26,221
Joined: Oct 2003
Reputation: 178
Post: #14
Pr. Sinister: No way in hell it should take that long. The textures update for instance is essentially: "drop everything and change a couple tables" so anything more than a few seconds is way too long. Same for the others.

The music scan could well be about right if you have a really, really large collection - essentially it's a complete (local) rescan of your music. I've only got a small collection (some 5k songs) and it takes about 5-10 minutes (over wifi).

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


[Image: badge.gif]
find quote
greenman Offline
Junior Member
Posts: 11
Joined: Aug 2007
Reputation: 0
Post: #15
From the previous discussion my assumption would be that the URL stored would be the URL of the computer not and internet URL so if X is always on then XBMC on Y could get the file from the original location.

If it's not on then as Kibje says it won't be able to get the fanart.jpg but it also couldn't play the movie.

I still think a local central cache on a NAS (where the DB stores the NAS location rather than the internet location) would be a useful addition that would save downloading the same thing for every XBMC instance. I'd be interested to know if this was considered with the new design.


I think the idea of storing everything locally is a great idea. My systems are not slow with the current Eden implementation but locally stored images/fanart are definitely faster though I do see the problem if the clients have small disks.
(This post was last modified: 2013-02-01 05:35 by greenman.)
find quote
Post Reply