• 1
  • 59
  • 60
  • 61(current)
  • 62
  • 63
  • 197
[RELEASE] Texture Cache Maintenance utility
Cool. I got it working mostly! This should help tremendously!

I think I'm on the same page with your method now. I've ran through the scripts and have mostly got what I was expecting to get which is awesome (Ie movies are now synced up using mysql data base and all movies are correctly cached). However I used basically the same approach and can't get my tvshows to respond the same way.

1: Update / Scrape TVshows folder to mysql
2: Scraping xbmc loads all images fine
3: run: python texturecache.py c tvshows

This is what I get:
http://pastebin.com/SgLmbsxe

I'm fine with the two SNL jpgs not downloading. But I'm not sure what all the duplicates are all about and none of the images are showing up for the tv shows on the non-scraped xbmc? Thoughts?

This is really slick. Not too sure why this isn't implemented directly into xbmc core!
Reply
(2014-04-25, 15:11)rbusenet Wrote: I'm fine with the two SNL jpgs not downloading.

Yes, unfortunately they're no longer available - you'll need to find alternatives for those two. This is biggest problem when using remote content.

(2014-04-25, 15:11)rbusenet Wrote: But I'm not sure what all the duplicates are all about

The different conditions are described here - have a read and hopefully it should become clear, if not ask away...

(2014-04-25, 15:11)rbusenet Wrote: and none of the images are showing up for the tv shows on the non-scraped xbmc? Thoughts?
That's unusual, is there anything in xbmc.log on the client that has this problem (you might need to enable debug)?

If the artwork has been cached successfully then it should be visible in the GUI, unless it's a skin problem - test with stock Confluence - or you're using some sort of path substitution which is preventing XBMC from accessing/reading the cached artwork correctly.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
I deleted the texture.db and tv32.db and now it works. I noticed that all of the most recent tv scrapes/additions worked fine so it was just old files that were not playing nice. I could have C'ed it I guess but either way it was a quick check to the theory and so far it is all looking good.

I think I'm all squared away - Great product man!
Reply
My guess is that the database (Textures13.db) had rows claiming the artwork was present in the cache, but the files didn't actually exist in the Thumbnails folder, so XBMC had nothing to display as it wouldn't have found any of the cached files. XBMC should have complained about this in the log, though as you have worked out, deleting Textures13.db or running C fixes the problem and gets your XBMC back in sync. Smile

A more optimal solution is to use the "X"/"Xd" options: "X" will list any rows from Textures13.db that reference non-existent Thumbnails files, while "Xd" will delete these rows.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
I'm having a problem with mklocal.py. One of my sources in XBMC is smb:\\WOPR\HD Rips\ I think mklocal.py is choking on the space in the path name because it returns:

Code:
mklocal.py: error: unrecognized arguments: Rips

How can I work around this?
Reply
(2014-04-27, 01:56)RockDawg Wrote: I'm having a problem with mklocal.py. One of my sources in XBMC is smb:\\WOPR\HD Rips\ I think mklocal.py is choking on the space in the path name because it returns:

Code:
mklocal.py: error: unrecognized arguments: Rips

How can I work around this?

If you're passing "smb:\\WOPR\HD Rips" as the --prefix make sure it's quoted, or escape the space ie. smb:\\WOPR\HD\ Rips - could be the argument parsing that is (quite rightly) choking on the space, interpreting Rips as an argument.

Failing that, can you PM me the input you are feeding to mklocal.py (or post it here, just one movie or tvshow with the path should be sufficient), and also tell me the exact command line you are using to run mklocal.py.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2014-04-27, 01:58)MilhouseVH Wrote:
(2014-04-27, 01:56)RockDawg Wrote: I'm having a problem with mklocal.py. One of my sources in XBMC is smb:\\WOPR\HD Rips\ I think mklocal.py is choking on the space in the path name because it returns:

Code:
mklocal.py: error: unrecognized arguments: Rips

How can I work around this?

If you're passing "smb:\\WOPR\HD Rips" as the --prefix make sure it's quoted, or escape the space ie. smb:\\WOPR\HD\ Rips - could be the argument parsing that is (quite rightly) choking on the space, interpreting Rips as an argument.

Failing that, can you PM me the input you are feeding to mklocal.py (or post it here, just one movie or tvshow with the path should be sufficient), and also tell me the exact command line you are using to run mklocal.py.

Escaping the space didn't work, but the quotes did. Thanks a lot!!
Reply
(2014-04-26, 04:09)MilhouseVH Wrote: My guess is that the database (Textures13.db) had rows claiming the artwork was present in the cache, but the files didn't actually exist in the Thumbnails folder, so XBMC had nothing to display as it wouldn't have found any of the cached files.

Yeah I'm sure that's exactly what happened. I'm still trying to figure out all of the commands but I'll give he Xd a go. Thanks!
Reply
Thanks for this, Milhouse - it's incredibly useful and time-saving!
Reply
Milhouse - I saw where you described how to attach this to a bash cron job. I got that working but was wondering how you have your's setup. I can see that a cron job isn't exactly the most efficient setup for this script. Ideally the scraper xbmc should trigger the client xbmc to start the script when something new has been scrapped. That, or have the NAS running the msql trigger it. Either way I'm not too sure how to trigger the script externally. I'd probably be ok with it running on boot. My client xbmc is off most of the time so I'd probably prefer it to just run on startup. What are your thoughts?
Reply
(2014-04-29, 17:41)rbusenet Wrote: I got that working but was wondering how you have your's setup.

I have a NAS (FreeNAS, exporting NFS, running MySQL), a headless Raspbian-based Raspberry Pi(*) on which I run Sickbeard/Transmission and other scheduled tasks including all the scripts to handle XBMC scraping, an XBMC "scraper" client (OpenELEC Raspberry Pi, this is the "scraper" as it's always "on") and an XBMC non-scraper client (OpenELEC x86).

*This could have been the NAS instead of a Raspbian Raspberry Pi, but I prefer to keep the NAS "as a NAS" as much as possible and I'd have put MySQL on the Pi too if it had the grunt!

This is pretty much my approach:

TV Shows:
These all come via Sickbeard running on a Raspbian Raspberry Pi. I have a Sickbeard post-processing script that is called by Sickbeard whenever it has copied a new episode to my NAS, and this post processing script is passed the file name of the new episode media file (mkv, avi etc.). The post-processing script uses mediainfo to add streamdetails to the corresponding NFO for the new media file, and then the following actions are performed:
  1. a library scan is performed on the "scraper" client (Raspberry Pi running OpenELEC). Only the season folder is scanned (vscan "nfs://path/to/tvshow/Season XX"), not the whole library
  2. mklocal.py is used to convert any remote artwork to local (unlikely there would be any with my tv show setup, but just in case...) and also load any new non-standard local artwork (eg. clearlogo, clearart, etc.)
  3. pre-load texture cache (c tvshows) on the scraper client if any artwork is converted or new artwork loaded by mklocal.py
  4. runs "qax tvshows @qaperiod=90" to reload any episodes that fail qa (eg. when episode thumb wasn't available yesterday but now it's been downloaded)
  5. wakes any sleeping remote non-scraper clients (if sleeping), runs "c tvshows" (or "lc tvshows" if mklocal.py made no changes) against each non-scraper client then puts the client back to sleep if originally sleeping

Movies:
These I rip myself, copy to the NAS along with artwork, then I create the NFO file using Ember. A daemon (started at boot, running on the Raspbian Pi, wakes at 5 minute intervals) uses "find" to locate any new movies, and only when all of the new movies have a matching NFO then the following actions are performed:
  1. a library scan is run on the scraper client, performing a whole library scan (vscan - yes, this could inadvertently pick up a new tv show episode but given the infrequency with which movies are added this isn't usually a problem)
  2. after the scan, Artwork Downloader is executed (exec script.artwork.downloader silent=true mediatype=movie). As I store multiple movies per folder (ie. I use the movie-name prefix rather than one movie per folder) "Use local files" is not enabled and new remote artwork is associated with movies. The script waits for AD to finish by tailing xbmc.log. AD is configured to find only clearart and clearlogo artwork on http://fanart.tv.
  3. mklocal.py is used to convert any remote artwork to local (ie. any new artwork found by AD), and also load any other new non-standard local artwork (unlikely there would be any, but just in case...)
  4. pre-load texture cache ("c movies") on the scraper client if any artwork is converted or new artwork loaded by mklocal.py
  5. wakes any sleeping remote non-scraper clients (if sleeping), runs "c movies" (or "lc movies" if mklocal.py made no changes) against each non-scraper client then puts the client back to sleep if originally sleeping

So my workflow for TV shows is... precisely nothing, everything works automatically - shows are loaded automatically as they become available, artwork is updated and backfilled, all caches are pre-loaded.

For movies, I copy the new movie files(s) and associated fanart/poster artwork to my NAS after ripping, scrape each new movie into Ember which creates the NFO file (with streamdetails etc.) on the NAS, and once all the movies have their NFO files, within 5 minutes the scraping process will have started automatically on the "scraper" client. Once the database optimisations have landed I will consider adding a call to update imdb movie rating/vote information on a more regular basis - the current JSON/database update performance doesn't really make this feasible right now.

(2014-04-29, 17:41)rbusenet Wrote: I can see that a cron job isn't exactly the most efficient setup for this script. Ideally the scraper xbmc should trigger the client xbmc to start the script when something new has been scrapped.

There is an addon, I think - I can't recall the name[1][2] - that allows you to execute external shell scripts. One solution might be to call a shell script from within XBMC (you'll probably want to fork a new process within the shell script, as it can take a while to finish). The shell script can then perform all or some of the above steps - that is, the shell script initiates the library scan and performs all the other steps, not XBMC starting the scan and then somehow calling the script to complete the remaining steps.

Maybe the following addons will be of use - not sure if they're Gotham compatible:
1: http://forum.xbmc.org/showthread.php?tid=85724
2: http://forum.xbmc.org/showthread.php?tid=151011
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Hey Millhouse i also use Ember to scrape movies automatically. I call it within my post processing script. Bur i don't rip my movies manually though!
Image Image
Reply
@ Milhouse, i know it is off-topic, but I hope you could point me in the right direction.
I see that you wake some of your xbmc non scraping clients, I assume you use some kind of Wake On Lan.
I am not sure if all your non scraping clients are x86 machine, just in case one is a raspberry, since I tried waking over lan my raspberry through wol and etherwake but I couldn't do it, is there any recommendation you could give me?

Thanks a lot

M
Reply
(2014-05-01, 22:16)mcarni Wrote: I see that you wake some of your xbmc non scraping clients, I assume you use some kind of Wake On Lan.

Yes, it's built-in to texturecache.py.

Code:
./texturecache,py wake @network.mac=d0:27:88:0e:7a:42

or configure an appropriate "section" in your texturecache.cfg and specify the section name (@section=name) eg.
Code:
#globals
qaperiod=9999
download.threads = 10

#use the following section as the default
section = rpi

[rpi]
xbmc.host = 192.168.0.8
network.mac=d0:27:88:0e:7a:42

[lounge]
xbmc.host = 192.168.0.12
network.mac=ab:cd:ef:12:34:45
then
Code:
./texturecache,py @section=lounge wake

to wake the "lounge" PC, and
Code:
./texturecache,py @section=lounge power suspend
to suspend it.

(2014-05-01, 22:16)mcarni Wrote: I am not sure if all your non scraping clients are x86 machine, just in case one is a raspberry, since I tried waking over lan my raspberry through wol and etherwake but I couldn't do it, is there any recommendation you could give me?

The Pi doesn't support a suspend mode - once it's powered down (only red light visible) you need to power cycle it to restart and there is no WOL support (just leave it on 24x7).
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
@Milhouse, thanks a lot, as always very clear and helpful
M
Reply
  • 1
  • 59
  • 60
  • 61(current)
  • 62
  • 63
  • 197

Logout Mark Read Team Forum Stats Members Help
[RELEASE] Texture Cache Maintenance utility17