[RELEASE] Texture Cache Maintenance utility - Printable Version +- Kodi Community Forum (https://forum.kodi.tv) +-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33) +--- Forum: Supplementary Tools for Kodi (https://forum.kodi.tv/forumdisplay.php?fid=116) +--- Thread: [RELEASE] Texture Cache Maintenance utility (/showthread.php?tid=158373) Pages:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
|
[RELEASE] Texture Cache Maintenance utility - Milhouse - 2013-03-05 It's annoying having to delete the entire Textures13.db+Thumbnails folder to try and correct scraping errors and/or corrupted images that just never go away, so I've written a very trivial Python script that can be used to interrogate the texture cache database, and can also be used to remove - with absolute precision - those database rows and cached files that relate to problematic artwork. The script has been tested with local installations of XBMC Frodo on OpenELEC R-Pi and Ubuntu 12.10, and also Windows with Python 2.7.3 running against a remote Frodo installation of XBMC. A separate properties file can be used to specify non-default configurations (see property file details at the end of this post). This is not an addon, and it requires that the webserver is enabled in XBMC on port 8080 (unless another port is specified in the properties file). It also assumes you are comfortable working at the command line (ssh in Linux, a large CMD window may also work OK for Windows users). Summary of features
Installation instructions For instructions on how to run a Python script, see here. Windows users will need to install Python 2.7.x or Python 3.3.x, if not already installed (see how-to). Download the single Python script file - use "Save as" in your browser - from github. A default properties file is available on github, rename this to texturecache.cfg in order to use it, although in many cases it's not required and should just be considered a template - pick and choose what options you wish to override. Alternatively, non-Windows users can download the script directly at the command line using curl: Code: curl https://raw.githubusercontent.com/MilhouseVH/texturecache.py/master/texturecache.py -o texturecache.py ATV2 (iOS) users Python 2.6+ is required to run this script, and although Python can be installed on iOS using "apt-get install python", the version installed (typically v2.5.1 - check with "python --version") is very old and lacks language features required by the script. It is possible to install a more recent Python 2.7.3 package as follows: Code: ssh [email protected] Example usage Let's say the poster image for my "Dr. No" movie is corrupted, and I want to delete it so that XBMC will automatically re-cache it (hopefully correctly) next time it is displayed: 1) Execute: ./texturecache.py s "Dr. No" to search for my Dr. No related artwork 2) Several rows should be returned, relating to different cached artwork - one row will be for the poster, the other fanart, and there may also be rows for other image types too (logo, clearart etc.). This is what I get: Code: 000226|5/596edd13.jpg|0720|1280|0011|2013-03-05 02:07:40|2013-03-04 21:27:37|nfs://192.168.0.3/mnt/share/media/Video/Movies/James Bond/Dr. No (1962)[DVDRip]-fanart.jpg 3) Since I want to remove only the poster (.tbn) I can execute ./texturecache.py d 227 and both the database row and cached poster image will be deleted. If I wanted to remove both images, I would simply execute ./texturecache.py d 226 227 and the two rows and their corresponding cached images will be removed. Alternatively, ./texturecache.py C movies "Dr. No" would achieve the same result including re-caching the deleted items. Format of "dumped" records When displaying rows from the database, the following fields (columns) are shown: Code: rowid, cachedurl, height, width, usecount, lastusetime, lasthashcheck, url Additional usage examples To view your most recently accessed artwork: Code: ./texturecache.py x | sort -t"|" -k6 or your top 10 accessed artwork... Code: ./texturecache.py x | sort -t"|" -k5r | head -10 Use texturecache.py to identify artwork for deletion, then cutting and pasting the matched ids into the "d" option or via a script: For example, to delete those small remote thumbnails you might have viewed when downloading artwork (and which still clutter up your cache): Code: ./texturecache.py s "size=thumb" and the same, but automatically: Code: IDS=$(./texturecache.py s "size=thumb" 2>&1 1>/dev/null | sed "s/.*: //") Delete any artwork that has not been accessed since a particular date: Code: ./texturecache.py x "where lastusetime <= '2013-03-05'" or hasn't been accessed more than once: Code: ./texturecache.py x "where usecount <= 1" Query the media library returning JSON results: First, just the default fields for a particular class (movies), filtering for a specific item (Avatar): Code: ./texturecache.py j movies "Avatar" With "extrajson.movies = trailer, streamdetails, file" in the properties file, the same query but returning the extra fields too: Code: ./texturecache.py J movies "Avatar" Optional Properties File By default the script will run fine on distributions where the .xbmc/userdata folder is within the users Home folder (ie. userdata=~/.xbmc/userdata). To override this default, specify a properties file with a different value for the userdata property. The properties file should be called texturecache.cfg, and be in the same directory as the texturecache.py script. What follows is an example properties file with the default values shown - DO NOT paste this file as your own properties, set only the properties you require in your own file: Code: sep = | The dbfile and thumbbnails properties represent folders that are both relative to the userdata property. Specify webserver.username and webserver.password if you require webserver authentication. extrajson.* properties allow the specification of additional JSON audio/video fields to be returned by the J query option. Cast thumbnails will not be updated by default, so specify cache.castthumb = yes if you require cast artwork to be re-cached. Ignore specific URLs when pre-loading the cache (c/C options), by specifying comma delimited regex patterns for the cache.ignore.types property. Default value is image://video. Set to None to process all URLs. Any matching URL will not be cached (c/C), nor will it be deleted when forcing a reload (option C). Specify a filename for the logfile property, to log detailed processing information. See github for full changelog and further details in README. Run the script without arguments for usage and current configuration information. Hopefully someone else will find this useful! For mklocal.py, see here For explanation of dropped/skipped/ignored, see here RE: Texture Cache interrogation utility - Milhouse - 2013-03-08 New version (0.0.5) - see installation instructions in first post for download details. Added "c" option to re-cache all artwork. It uses JSON to query the media library for all artwork, and then retrieves each item of artwork so that it is cached locally. Items that are already in the cache will be ignored. Use the "d" option to remove items from the cache, and then refresh the cache. This is much easier than laboriously browsing through your media library while the Pi slowly re-caches all the missing items! To run: Code: ./texturecache.py c If no additional arguments are supplied when using the "c" command, all but songs will be refreshed (ie. movies+sets+tvshows+albums+artists). A third argument can be used to filter queries, partially matching on movie/set/tvshow/album/song title, or artist name. As items of artwork are processed, their type (poster, fanart, etc.) will be displayed. A "+" alongside the artwork type will indicate that the item was not found in the cache, and has now been cached. The lack of a "+" indicates that the item was found in the cache, and it has been skipped (not retrieved). This has been tested with a media library hosted on a remote MySQL server and NFS content, and is working well. Should work with local databases and other media solutions. RE: Texture Cache interrogation utility - leechguy - 2013-03-10 I haven't tested the script yet, but this sure is going to be usefull! Thanks! RE: Texture Cache interrogation utility - Milhouse - 2013-03-10 (2013-03-10, 02:19)leechguy Wrote: I haven't tested the script yet, but this sure is going to be usefull! Thanks! Well it's been useful for me, so I figured it might be useful for others too! Hopefully it works for you, if not let me know and I'll try to fix. I've lost count of the number of times I've had to drop the texture cache database and remove Thumbnails, only to then have to re-cache items one by one... but no more now that I've got the re-caching option working! I've also added another caching option ("C") which will remove any existing artwork before re-caching new items, eg. Code: ./texturecache.py C tvshows "big bang theory" will automatically remove any existing artwork from the cache (both database and filesystem) before retrieving new artwork items, thus ensuring the local texture cache is updated come what may (this is just a more convenient way of deleting items - option "d" - before refreshing them with option "c"). RE: Texture Cache interrogation utility - nickr - 2013-03-12 @Milhouse - could you please clarify the licensing of your code? I would like to play with the code a bit, and would hope that in the spirit of xbmc you might release it under GPL2? RE: Texture Cache interrogation utility - Milhouse - 2013-03-12 (2013-03-12, 10:55)nickr Wrote: @Milhouse - could you please clarify the licensing of your code? I would like to play with the code a bit, and would hope that in the spirit of xbmc you might release it under GPL2? Good question, I've just uploaded an updated version 0.0.8 using the GPLv2 license so you can do what you like with it now, though a small credit or attribution would be nice on any derived work! RE: Texture Cache interrogation utility - nickr - 2013-03-12 Excellent. Possibly my interest will come to nothing, but a license is good RE: Texture Cache interrogation utility - Milhouse - 2013-03-13 Version 0.1.0. Small update, now outputs JSON results (j), optionally with additional fields (J) as specified in the properties file. Potentially JSON output could be parsed by another script for further processing. RE: Texture Cache interrogation utility - popcornmix - 2013-03-13 @MilhouseVH One annoyance with xbcm is that some tv episodes have no screenshot/plot/rating presumably because the episode was scanned before the scraping site was updated. This never gets fixed without a manual episode info/refresh. Is it possible for your script to trigger a refesh on episodes with missing content? (or does this feature exist anywhere else?) RE: Texture Cache interrogation utility - Milhouse - 2013-03-13 (2013-03-13, 14:50)popcornmix Wrote: @MilhouseVH As long as you aren't extracting thumbnails (which will create a thumbnail), then yes, I guess so... you'd need to execute the extended JSON query and determine when there is a missing thumbnail or plot from the returned results. ie, if you add "plot" to "extrajson.tvshows.episode" in the properties file, you could run something like "./texturecache.py J tvshows" and then parse the JSON results for any missing items. The trouble with triggering a refresh of items that have missing content is that you normally need to delete the item first - and not just from the texture cache but deletion from the media library. Definitely possible though, either by extending texturecache.py or by a script that parses the JSON results. I'll give it some thought, though not entirely sure how to fix the problem once having identified there are missing items. (2013-03-13, 14:50)popcornmix Wrote: (or does this feature exist anywhere else?) I load all my meta-data from NFO files and locally stored artwork, so I wrote a script that I run on my NAS which parses the NFO files to check for missing plots, and to identify missing artwork.. not aware of any other solution, other than querying the media library database direct. RE: Texture Cache interrogation utility - Milhouse - 2013-03-13 @popcornmix: Try the version I just uploaded (version 0.1.1) which now includes a "qa" option for movies and tvshows, and should identify missing artwork (poster and fanart for movies, thumb for tvshows) and plots. The formatting of the output could do with some work, but as an initial step is this of any use to you? To run it, execute: Code: ./freenas/data/texturecache.py qa movies Code: ./freenas/data/texturecache.py qa tvshows RE: Texture Cache interrogation utility - bonelifer - 2013-03-13 Now if someone could do this in an addon that was crossplatform. RE: Texture Cache interrogation utility - Milhouse - 2013-03-13 (2013-03-13, 16:58)bonelifer Wrote: Now if someone could do this in an addon that was crossplatform. Off you go then. Sorry I've no interest in writing an addon, I'm much happier at the command line and the UI for this as an addon would be quite horrific. Although this command line script should work cross platform, with a bit of jiggery pockery. RE: Texture Cache interrogation utility - Milhouse - 2013-03-13 Version 0.1.2 now up... when performing a "qa" check, only movies or tvshow episodes added during the previous 30 days will be considered. Increase/reduce this period by specifying a value (in days) for qaperiod in the properties file, eg. "qaperiod=9999" would effectively restore v0.1.1 behaviour which didn't restrict by period. Also added additional qa tests: rating and mpaa certification (movies), plot/fanart/banner/poster (tv show) and rating (tv show episode). RE: Texture Cache interrogation utility - popcornmix - 2013-03-13 (2013-03-13, 17:54)MilhouseVH Wrote: Version 0.1.2 now up... when performing a "qa" check, only movies or tvshow episodes added during the previous 30 days will be considered. Increase/reduce this period by specifying a value (in days) for qaperiod in the properties file, eg. "qaperiod=9999" would effectively restore v0.1.1 behaviour which didn't restrict by period. Sounds very interesting. I'll have a play with this when I get the chance. |