[RELEASE] Texture Cache Maintenance utility

[RELEASE] Texture Cache Maintenance utility - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Supplementary Tools for Kodi (https://forum.kodi.tv/forumdisplay.php?fid=116)
+--- Thread: [RELEASE] Texture Cache Maintenance utility (/showthread.php?tid=158373)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197

RE: [RELEASE] Texture Cache Maintenance utility - cgrey - 2014-03-06

(2014-03-06, 06:47)MilhouseVH Wrote: Hmmm..... all qax does is remove items, then initiate a library rescan, usually just of the directory containing the removed item... if you can reproduce, a log would be handy. Strange that one of the items was unavailable (what was the path?), stranger still that there should be two - the first issue would suggest a problem with the scan, the second issue would suggest the item wasn't successfully removed in the first place, but then if the episode remained in the library, xbmc shouldn't have had any reason to scan in the duplicate...

I'm doing too much at once...

I don't know if this is related to Gotham B1 or not. So I'm going back to Frodo (partly due to a usability issue that just cropped up on my Gotham system). Once I've got things back to stable, I'll revisit this.

That said, I still truly appreciate this script as a tremendously useful library management tool - much more than image/artwork management Smile

RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2014-03-06

texturecache.py (Version 1.5.4)

Add: IPv6 support. When connecting to JSON, IPv6 will be attempted first, followed by IPv4. If you wish to use only a specific version, set rpc.ipversion=4 or rpc.ipversion=6 and only that version will be attempted. Default is no value for this property. Closes issue #11.
Add: Extra logging added to rbphdmi to record calls to @bin.tvservice and response received

RE: [RELEASE] Texture Cache Maintenance utility - un1versal - 2014-03-06

can you add a switch to write the default settings to file like ./texturecache wcfg so that all defaults are put into a file and we can edit those after to suit any particular changes we want to do like -R etc.?

RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2014-03-06

(2014-03-06, 23:49)uNiversal Wrote: can you add a switch to write the default settings to file like ./texturecache wcfg so that all defaults are put into a file and we can edit those after to suit any particular changes we want to do like -R etc.?

"./texturecache.py config" will show you your current settings, and without a config file these will all be the defaults.

You could redirect the output of "config" to a file and with a bit of editing use it as a configuration file but I really don't recommend adding properties to the confguration file unless you want to override the default value - if you add properties to the config file using what are the current default values, if/when the default value of a property changes in future you won't get the benefit as you'll still be using the "old" default value.

My advice is always to add only the properties you need to your config file. Smile

RE: [RELEASE] Texture Cache Maintenance utility - ADA2D - 2014-03-07

Just wanted to say thanks again MilhouseVH. Your changes are working like a charm.

According to the last line I will be Purging a total of 31,504 files; Total size: 3,122,384 KB

Big Grin

RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2014-03-07

3GB is quite a reduction - presumably you've eyeballed the files and are happy with what will be removed, although it's a lot to check thoroughly. If you want to keep pictures you will need to set "@prune.retain.pictures=yes", not sure if you've done that or not. Picture previews (prune.retain.previews) is enabled by default, but the cached versions of the pictures are not retained by default (debatable whether they should even be in the cache, or if you'd want to keep them when pruning is all about trimming away the crud, and the pictures could huge).

RE: [RELEASE] Texture Cache Maintenance utility - ADA2D - 2014-03-07

Well the purge completed. TV shows, music, picture all look good at a quick glance. I did lose about 10% of my posters and fanart in my movies. Not the end of the world; I can manually refresh the ones that are missing in an hour or so. Overall, worth the 3GB of overhead I recovered.

I'm wasn't surprised about the 3GB. I haven't done a clean refresh of XBMC since v9 back in 2009.

RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2014-03-07

You shouldn't really have lost any "active" posters or fanart, that's a bit odd. Should be easy enough to re-cache though (as long as the originals remain available).

RE: [RELEASE] Texture Cache Maintenance utility - un1versal - 2014-03-09

Hey Milhouse

Am coming across some oddities.

embedded art in albums etc not coming up properly, and it refuses to read the Folder.jpg in each album, so I really dont want to manually pull info > choose thumb for each to load local music thumbs for albums or songs etc as its already embedded in tags.

Code:
C          Re-cache artwork even when it exists. Class can be movies, tags, sets, tvshows, artists, albums or songs. Filter mandatory.

However running

Code:
./texturecache.py C songs

 ./texturecache.py C albums

 ./texturecache.py C artists

I gets something like

Code:
File "./texturecache.py", line 7063, in <module>

    main(sys.argv[1:])

  File "./texturecache.py", line 6801, in main

    loadConfig(argv)

  File "./texturecache.py", line 6392, in loadConfig

    gConfig = MyConfiguration(argv)

  File "./texturecache.py", line 291, in __init__

    self.QAPERIOD = int(self.getValue(config, "qaperiod", "30"))

ValueError: invalid literal for int() with base 10: '30 (added after 2014-02-05)'

So is this me or something worng in script, I though filter mandatory measnt you could pass movies, tags, sets, tvshows, artists, albums or song to C

Yes Im adding allow.recacheall=yes to the cfg.

RE: [RELEASE] Texture Cache Maintenance utility - un1versal - 2014-03-09

right I think for some reason the cfg had

qaperiod = 30 (added after 2014-02-07)

removing (added after 2014-02-07)

so its

qaperiod = 30

fixed that though idk if it will recache the embedded art I have in alums etc.

RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2014-03-09

(2014-03-09, 20:21)uNiversal Wrote: embedded art in albums etc not coming up properly, and it refuses to read the Folder.jpg in each album, so I really dont want to manually pull info > choose thumb for each to load local music thumbs for albums or songs etc as its already embedded in tags.

Embedded artwork isn't supported unfortunately, as it can only be cached by browsing the GUI (if you have "Extract thumbnails" enabled in Settings) or - presumably - by the original scraper, and not by downloading the embedded artwork from the webserver, which is the method the script is using.

Any artwork that is embedded (where the url begins with video or music, eg. "music@nfs://blah/blah/song.mp3") will not be processed by the script when performing a caching operation, so if you run "C songs" it will not remove any of your embedded artwork (as it can't be re-cached) but it will remove and re-cache any non-embedded artwork.

As for Folder.jpg, the first thing is to confirm if any of your albums are actually using the Folder.jpg file as their "thumbnail" artwork - check with "jd albums", then if you find an album search the cache (s "you-folder.jpg-thumbnail-path") to see if the corresponding url is present in the cache.

All of my albums have "cover.jpg" (not folder.jpg) in the album folder and these are scraped and cached OK, is "folder.jpg" the correct filename for album artwork when scraping the music db?

(2014-03-09, 20:21)uNiversal Wrote:
Code:
C Re-cache artwork even when it exists. Class can be movies, tags, sets, tvshows, artists, albums or songs. Filter mandatory.
...
Yes Im adding allow.recacheall=yes to the cfg.

Thanks, I should probably re-word that to be "Filter mandatory, unless allow.recacheall=yes" or similar.

RE: [RELEASE] Texture Cache Maintenance utility - un1versal - 2014-03-09

Even after

./texturecache.py C artists

you have to set the allow.recacheall=yes script kicked me straight away with that message because I was using C even if I was using C atrists or C albums.

idk what the correct artwork is, everything in wiki suggests folder.jpg and not cover.jpg.

As for embedded art in music albums only working when you browse it or play it, not true as far as I can tell. Yes I have extract thumbnails set in xbmc. but this is not video files and its not streaminfo but meh music art and all this handling even when tagged to to ad nauseaum doesnt work as expected.

Its not about if script supports it or not, Im using script to help xbmc along and apparently xbmc is having none of it when it comes to music stuffs.

for some laugh heres what it said.

Cache pre-load activity summary for "albums":

| fanart | thumbnail | TOTAL
--------------+-------------+-------------+-------------
Cached | 35 | 58 | 93
Deleted | 35 | 58 | 93
Duplicate | 126 | 1 | 127
Error | 3 | 79 | 82
Ignored | - | 30 | 30
Skipped | - | - | 0
Undefined | 5 | 1 | 6
========================================================
TOTAL | 204 | 227 | 431
Download Time | 00:00:55.42 | 00:01:46.66 | 00:01:52.43

Threads Used: 3
Min/Avg/Max: 0.23 / 0.97 / 2.08

Loading: 00:00:00.22
Parsing: 00:00:00.05
Comparing: 00:00:04.30
Downloading: 00:01:53.11
TOTAL RUNTIME: 00:01:57.73

RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2014-03-10

(2014-03-09, 23:20)uNiversal Wrote: Even after

./texturecache.py C artists

you have to set the allow.recacheall=yes script kicked me straight away with that message because I was using C even if I was using C atrists or C albums.

Yes you're right, media class AND filter are both mandatory when allow.recacheall=no (the default). It's quite a "destructive" option to use on an entire library, which could happen by mistake, hence the "allow" property that absolves me of all responsibility. Smile

(2014-03-09, 23:20)uNiversal Wrote: As for embedded art in music albums only working when you browse it or play it, not true as far as I can tell. Yes I have extract thumbnails set in xbmc. but this is not video files and its not streaminfo but meh music art and all this handling even when tagged to to ad nauseaum doesnt work as expected.

It used to be the case that embedded music and video couldn't be pre-cached, but I just did a quick test and managed to pre-cache an embedded album cover, so perhaps embedded music artwork can now be cached (is this new in Gotham?)

If you set "cache.ignore.types = ^video" (the default value is "^video, ^music") you should be able to process embedded music artwork when caching - if you have success caching embedded music artwork let me know and I'll consider changing the default so that embedded music artwork is no longer excluded when caching.

However embedded video artwork still doesn't download from the XBMC webserver (just tested it using a tip of master Gotham build), so embedded video artwork remains unsupported by this script as far as caching is concerned.

(2014-03-09, 23:20)uNiversal Wrote: Its not about if script supports it or not, Im using script to help xbmc along and apparently xbmc is having none of it when it comes to music stuffs.

With the best will in the world I can only support what XBMC supports. If/when XBMC support changes I'm happy to update the script to follow suit. When I originally wrote the caching functionality, I had little success caching embedded video and music artwork, hence the current logic in the script.

Now it appears that it may be possible to cache embedded music artwork, but I have no idea if this is a recent Gotham change - which would cause problems for Frodo users if I allow embedded music artwork to be cached - or what. If this only applies to Gotham and a Frodo user runs "C" on their music library they'll find their embedded artwork is removed from the cache but not replaced which is hardly desirable.

RE: [RELEASE] Texture Cache Maintenance utility - tesona1977 - 2014-03-10

@MilhouseVH

first of all...thanks for this great tool. it makes a lot of things easier Wink

two questions:

* is it possible to print the entire movie library as one big json dump? i want to convert this whole json dump to html afterwards (to get a movie listing for friends and family)

* second question regarding the "imdb" option. where can i find a listing of all the valid fields i can use for this option?

thanks!

RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2014-03-10

(2014-03-10, 07:06)tesona1977 Wrote: * is it possible to print the entire movie library as one big json dump? i want to convert this whole json dump to html afterwards (to get a movie listing for friends and family)

For a "basic" set of movie fields (title, file, art, id):

Code:
./texturecache.py jd movies

If you want to specify additional movie fields, use the extrajson.movies property with "Jd movies", for example:

Code:
./texturecache.py Jd movies @extrajson.movies=trailer,streamdetails,file,mpaa,rating,plot,dateadded,playcount,cast,tag,studio,runtime,country,lastplayed,imdbnumber

(2014-03-10, 07:06)tesona1977 Wrote: * second question regarding the "imdb" option. where can i find a listing of all the valid fields i can use for this option?

Have a look at the results from omdbapi.com as this is the source of the information.

This is the result for Avatar, tt0499549:

Code:
{"Title":"Avatar","Year":"2009","Rated":"PG-13","Released":"18 Dec 2009","Runtime":"162 min","Genre":"Action, Adventure, Fantasy","Director":"James Cameron","Writer":"James Cameron","Actors":"Sam Worthington, Zoe Saldana, Sigourney Weaver, Stephen Lang","Plot":"A paraplegic Marine dispatched to the moon Pandora on a unique mission becomes torn between following his orders and protecting the world he feels is his home.","Language":"English, Spanish","Country":"USA, UK","Awards":"Won 3 Oscars. Another 58 wins & 74 nominations.","Poster":"http://ia.media-imdb.com/images/M/MV5BMTYwOTEwNjAzMl5BMl5BanBnXkFtZTcwODc5MTUwMw@@._V1_SX300.jpg","Metascore":"83","imdbRating":"8.0","imdbVotes":"651,019","imdbID":"tt0499549","Type":"movie","Response":"True"}

Many (although not all) of the omdabpi.com fields map directly to XBMC JSON fields. "Metascore", "Released", "Actors", "Poster", "Awards" and "Language" are not supported, either because there is no associated field in XBMC, or the corresponding field cannot be set via JSON.

Specify the XBMC field names when assigning a value to the imdb.fields property. "plotoutline" is supported by querying omdapi.com and requesting only the short plot description. If "plot" and "plotoutline" are specified on imdb.fields, omdapi.com will be queried twice per movie (once for the short plot/plot outline, and again for the "full" plot).