• 1
  • 51
  • 52
  • 53(current)
  • 54
  • 55
  • 197
[RELEASE] Texture Cache Maintenance utility
texturecache.py (Version 1.4.7)
  • Add: Send GUI notification with notify option, eg. notify "Title" "Message". Optionally, the display time (in milliseconds, default 5000) can be specified, as well as the location of a suitable image file (which must be accessible to the XBMC client)
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Hi Milhouse - I saw your response to this thread, and I am trying to get this to work correctly.

I thought I had everything set up correctly, but am getting an error when trying to refresh my nfo items

Code:
C:\Users\Kevin\Stuff\XBMC\texturecache.py-master>python texturecache.py qax movies @qa.nfo.refresh=today
Traceback (most recent call last):
  File "texturecache.py", line 6945, in <module>
    main(sys.argv[1:])
  File "texturecache.py", line 6683, in main
    loadConfig(argv)
  File "texturecache.py", line 6276, in loadConfig
    gConfig = MyConfiguration(argv)
  File "texturecache.py", line 294, in __init__
    (self.QA_NFO_REFRESH, self.qa_nfo_refresh_date, self.qa_nfo_refresh_date_fmt
) = self.getRelativeDateAndFormat(config, "qa.nfo.refresh", "")
  File "texturecache.py", line 522, in getRelativeDateAndFormat
    date_seconds = int(temp_date.strftime("%s"))
ValueError: Invalid format string

From your examples in the thread, it looks like I have the syntax correct, but I can't be sure. Any help is greatly appreciated.
Reply
OK, looks like a platform issue (Windows), will work on a fix.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Thanks!
Reply
@feerlessleadr - try updating to v1.4.9, I've just pushed an update that should fix the problem on Windows.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Thanks, worked perfectly!
Reply
I would like to Prune all the entries that youtube and other addons make, I ran

nc then C then I ran P that went OK.
Code:
|    banner   |    fanart   |    poster   |    thumb    |  thumbnail  |    TOTAL
--------------+-------------+-------------+-------------+-------------+-------------+-------------
Cached        |     272     |     1376    |     1322    |     6857    |     152     |     9979
Deleted       |     280     |     1384    |     1325    |     6857    |     154     |    10000
Duplicate     |     7027    |     7177    |     7031    |      70     |      7      |    21312
Error         |      8      |      15     |      3      |      9      |     129     |     164
Ignored       |      -      |      -      |      -      |      45     |      27     |      72
Season-all    |      -      |      -      |      -      |      -      |      -      |      0
Skipped       |      -      |      -      |      -      |      -      |      -      |      0
Undefined     |      -      |      53     |      -      |      -      |      84     |     137
==================================================================================================
TOTAL         |     7587    |    10005    |     9681    |    13838    |     553     |    41664
Download Time | 00:03:48.34 | 00:00:01.52 | 00:02:09.73 | 00:06:15.13 | 00:00:14.01 | 02:25:55.78

then R i get a massive amount to get out of the way and texturecache tools tells me something could be wrong?

Code:
sudo ./texturecache.py R
Found 5,995 orphaned files for 13,978 database files.

This is far too many orphaned files for this number of
database files - more than 5% - and something may be wrong.

Check your configuration, database, and Thumbnails folder.

Add "orphan.limit.check = no" to the properties file
if you wish to disable this check.

Obviously not done it yet but I feel tempted.
Reply
(2014-03-05, 00:33)uNiversal Wrote: Obviously not done it yet but I feel tempted.

Yeah, that's just a safety check after a bug wiped out one users cache - oops. Blush

As per the message, add "orphan.limit.check = no" to your configuration file (or add "@Orphan.limit.check=no" on your command line), and it will then remove those 5,995 files that are no longer being referenced by Textures13.db. Presumably these have been left over from a pre-Frodo installation, I'm guessing. If this explanation makes sense, then go ahead and do it (but don't blame me afterwards!) Smile

If you want to remove all your youtube cached artwork, the best option is probably "purge all <pattern>" where <pattern> is something unique about the urls you want to remove (eg. "purge all www.youtube.com/"). You can perform a dryrun with "purgetest all <pattern>", just to be on the safe side!
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Hi,

I've noticed my thumbnails folder is currently at 10.7gb so I'm trying to run the following command, "python texturecache.py p" on my Win7 machine with python 2.7 installed and I am receive the following error:

Code:
Loading Songs: Rob Ford | VictoriaÔÇÖs Secret Fashion Show Pt. 1 | Wha Happened
Loading Songs: Saturday...
Loading Genres: club  trance house VA  ACID TECHNO HARDCORE HARDSTYLE DEEP UNDER
Loading Genres: Compilation...
Traceback (most recent call last):
  File "texturecache.py", line 6982, in <module>
    main(sys.argv[1:])
  File "texturecache.py", line 6843, in main
    pruneCache(remove_nonlibrary_artwork=False)
  File "texturecache.py", line 5222, in pruneCache
    libraryFiles = getAllFiles(keyFunction=getKeyFromFilename)
  File "texturecache.py", line 5537, in getAllFiles
    files[keyFunction(picture["thumbnail"])] = "thumbnail"
  File "texturecache.py", line 5367, in getKeyFromFilename
    return MyUtility.normalise(filename, strip=True)
  File "texturecache.py", line 3233, in normalise
    v = bytes(v.encode("iso-8859-1")).decode("utf-8")
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 39-41: ordinal not in range(256)

I think its giving an error while scanning my music which I must admit is a mess when it comes to tagging.

Any advise? Thanks in advance.
Reply
Sigh... Sad

Can you add @logfile=c:\tc.log when running the script, then send me a link to the log? Thanks.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2014-03-05, 01:06)MilhouseVH Wrote:
(2014-03-05, 00:33)uNiversal Wrote: Obviously not done it yet but I feel tempted.

Yeah, that's just a safety check after a bug wiped out one users cache - oops. Blush

As per the message, add "orphan.limit.check = no" to your configuration file (or add "@Orphan.limit.check=no" on your command line), and it will then remove those 5,995 files that are no longer being referenced by Textures13.db. Presumably these have been left over from a pre-Frodo installation, I'm guessing. If this explanation makes sense, then go ahead and do it (but don't blame me afterwards!) Smile

If you want to remove all your youtube cached artwork, the best option is probably "purge all <pattern>" where <pattern> is something unique about the urls you want to remove (eg. "purge all www.youtube.com/"). You can perform a dryrun with "purgetest all <pattern>", just to be on the safe side!

Would it be possible to add an option that purges all addon like youtube/pbs cached art and watched status from youtube and pbs like addons? Really any addon stuff thats not actual media to me could go away.

I often thought there should be a separate database for these so its easier to identify and purge and de-clutter things, I dont want to nuke the whole lot just the crude.
Reply
well purgetest all http://www.youtube.com/* has 0 matches so idk what the 5,995 orphaned files for 13,978 database files are for.

Also when I try to run qax i get a silly error

Error: media class [addons] is not currently supported by qax

In fact it says

Valid media classes: addons, pvr.tv, pvr.radio, artists, albums, songs, movies, sets, tags, tvshows
Valid meta classes: audio (artists + albums + songs) and video (movies + sets + tvshows) and all (music + video + addons + pvr.tv + pvr.radio)
Meta classes can be used in place of media classes for: c/C/nc/lc/lnc/j/J/jd/Jd/qa/qax options.

but most of these meta classes are not supported at all by qax even though qa outputs something for them.

This is 1.4.9 btw.

Code:
./texturecache.py lnc audio

Cache pre-load activity summary for "albums/artists/songs":

              |    fanart   |  thumbnail  |    TOTAL
--------------+-------------+-------------+-------------
Cached        |      -      |      -      |      0
Deleted       |      -      |      -      |      0
Duplicate     |      -      |      -      |      0
Error         |      -      |      -      |      0
Ignored       |     2363    |     2442    |     4805
Not in Cache  |      -      |      -      |      0
Skipped       |      -      |      -      |      0
Undefined     |      94     |      15     |     109
========================================================
TOTAL         |     2457    |     2457    |     4914

I get a lot of ignored stuff and here in I think lies the orphaned files... I get that in videos and tvshows too though numbers are diffrent mainly its the ignored numbers andtype of art which is being ignored.


EDIT

I think some of these orpahaned files could be cast/actor/addons/fanart/otherthumbs which cannot be cached even though Im prestty sure they exist, as neither cast actors is a class and idk how to relink oprpahned files to actual content or why they were orphaned anyway.. So to really purge the cruft idk really how to do it without nuking things that shouldnt go.
This is in addition to my posts above.
Reply
(2014-03-05, 08:58)uNiversal Wrote: Would it be possible to add an option that purges all addon like youtube/pbs cached art and watched status from youtube and pbs like addons? Really any addon stuff thats not actual media to me could go away.

I often thought there should be a separate database for these so its easier to identify and purge and de-clutter things, I dont want to nuke the whole lot just the crude.

Prune (P) should get rid of most addons already, as they're not associated with your media library items.

"purge" is a faster option if you know the artwork urls you wish to get rid off, it will accept multiple patterns, and you can run it on a more frequent basis than "P" which can take quite a while to complete.

Which you prefer is up to you, prune is automated, slow and thorough, while purge is not automated (you have to tell it what to remove) but a lot faster than prune.

(2014-03-05, 10:43)uNiversal Wrote: well purgetest all http://www.youtube.com/* has 0 matches so idk what the 5,995 orphaned files for 13,978 database files are for.

If you'd already run prune (P), it would have got rid of most of your youtube artwork.

However the pattern I gave is just an example, I don't know what youtube urls look like in your Textures13.db - try a search (s "youtube") and see what urls appear, or run a "p" (prune, dry run) to see what unassociated urls are in your cache.

(2014-03-05, 10:43)uNiversal Wrote: Also when I try to run qax i get a silly error

Error: media class [addons] is not currently supported by qax

In fact it says

Valid media classes: addons, pvr.tv, pvr.radio, artists, albums, songs, movies, sets, tags, tvshows
Valid meta classes: audio (artists + albums + songs) and video (movies + sets + tvshows) and all (music + video + addons + pvr.tv + pvr.radio)
Meta classes can be used in place of media classes for: c/C/nc/lc/lnc/j/J/jd/Jd/qa/qax options.

but most of these meta classes are not supported at all by qax even though qa outputs something for them.

OK I'll remove that check so that qax works for addons, genres etc. However, since there is no Addons.RemoveAddons JSON method (the same goes for genres, songs, albums, artists etc.) any failures reported by qax will not be actioned, so qax will be ineffective. The only classes that truly support qax are movies and tvshows as these media items can be removed and then re-scraped, hopefully fixing whatever problem has been identified by qax.

(2014-03-05, 10:43)uNiversal Wrote: I get a lot of ignored stuff and here in I think lies the orphaned files...

"Ignored" items are those artwork files that match a pattern in the "cache.ignore.types" property - this defaults to "^video, ^music".

So your ignored items are embedded video or music artwork with urls starting with either music or video, and which unfortunately can't be cached using this script - you'll need to browse the GUI for those thumbnails to be extracted.

(2014-03-05, 10:43)uNiversal Wrote: I think some of these orpahaned files could be cast/actor/addons/fanart/otherthumbs

Orphaned files are files that are present in your Thumbnails folder but which are not referenced by a row in Textures13.db - this means they're taking up file space and cannot/will not be displayed by XBMC as without a row in Textures13.db it will not know that these files exist. Most often this can happen if at some point you delete Textures13.db without deleting Thumbnails, so now you have a bunch of files with no corresponding rows in the database. This might also happen if you have thumbnails left over from a pre-Frodo installation.

In order to re-create the rows in Textures13.db you need to re-cache the artwork, ie. "c audio", "c video", as these will recreate any missing texture cache rows for artwork that can be cached (which is pretty much everything apart from embedded artwork, and pictures).

Other than nuking your Textures13.db and Thumbnails folder then running "c all" to re-cache everything, my advice to keep your cache in tip-top condition would be to run the following on a semi-regular basis:
Code:
./texturecache.py P        <--- Prune: remove non-media library artwork, including addon cruft
./texturecache.py R        <--- Orphans: remove files that XBMC has forgotten about
./texturecache.py c all    <--- Cache: cache any missing artwork, for all media classes (basically, everything)

texturecache.py (Version 1.5.1 - this thread is a year old!)
  • Add: Include suspend capability (whether suspend is supported), and idle timer state in status
  • Add: Include songs as tc.members in jd albums (also j/jr/J/Jr/Jd) when @songmembers=yes
  • Fix: Remove unsupported class check on qax - now runs qax on all classes, even if there is no way to fix them
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Hi -

Really like this script, and have been experimenting with it. Trying to integrate it in to a scheduled (via cron) workflow and hoping for some help with the command-line options I need. Here's what I want to accomplish:

Once a day, run the script via cron to update all movies and TV shows that (a) have had their local NFO file change since the last run (could use a relative date to determine this since it will run at the same time every day), AND/OR (b) have had their artwork (poster, banner, logo, etc) change.

Some details:

My movies and TV shows are stored on a file server, and accessed by my XBMC client(s) via SMB.
Every media file has an associated NFO file. There are also show and season-specific NFO files for TV series
NFO files and all artwork are stored in the item's folder (same location as the media files, accessed via SMB)
I use a 3rd party tool to "update" my library, which basically writes a new NFO file for each media file if I make a change to any information
That same 3rd party tool replaces artwork (poster, fanart, banner, logo, etc...) if I download a different/new file

I've been experimenting with the -qax option, qa.nfo.refresh option. etc, but still haven't found a way to do exactly what I'm looking for. If I need to run the script multiple times with different options, I can script/batch that if need be.

If it matters, I'm running Gotham Beta1 on Windows 7, using a shared mySQL database, and again, accessing all media via SMB. Only one of my clients "manages" the Library/Database, and it would presumably be the one I use to execute this script.

Thanks in advance for any help...
Reply
qa.nfo.refresh should solve the problem of (a) - you just need to specify a relative (or absolute) date. Absolute dates takes the form "YYYY-MM-DD HH:MM:SS". Relative dates are a positive number eg. 0 (same as "today"), 1 would be yesterday etc.

If you want to re-cache artwork that has been recently updated, then specify "@cache.refresh=today|<relative date>|<absolute date>" when running "C <class>" and only artwork that is more recently modified than the specified date will be re-cached (normally, C without specifying a @cache.refresh property will re-cache everything so don't mis-type this parameter!)

So, work out your relative or absolute date, then run qax followed by C, specifying the relevant @qa.nfo.refresh/@cache.refresh parameter. Some artwork that is (re-)loaded by qax may also be re-cached by C, particularly if it's a new movie or episode, but this is an unavoidable inefficiency.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
  • 1
  • 51
  • 52
  • 53(current)
  • 54
  • 55
  • 197

Logout Mark Read Team Forum Stats Members Help
[RELEASE] Texture Cache Maintenance utility17