[RELEASE] Texture Cache Maintenance utility - Printable Version +- Kodi Community Forum (https://forum.kodi.tv) +-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33) +--- Forum: Supplementary Tools for Kodi (https://forum.kodi.tv/forumdisplay.php?fid=116) +--- Thread: [RELEASE] Texture Cache Maintenance utility (/showthread.php?tid=158373) Pages:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
|
RE: [RELEASE] Texture Cache maintenance utility - Milhouse - 2013-03-24 Version 0.2.3: * Added logfile property, to log processing information. To a file. Version 0.2.4: * Added cache.ignore.types property to ignore troublesome image://video urls which may or may not work, depending. Disable by setting to None. Multiple image types can be specified, eg. "image://video, image://http" to ignore local video images, and also all remote images. Each pattern is matched against the image url, so you could exclude specific remote web sites too (eg. "cache.ignore.types = image://video, .*imdb.com.*"). * Added additional QA rule to highlight missing artwork (as in, missing from cache). For qax option, this will not initiate a rescan of the directory unless other existing QA rules also fail - if the missing artwork is image://video then try and cache it via the GUI, for other artwork try pre-loading by using the c option. RE: [RELEASE] Texture Cache maintenance utility - ctawn - 2013-03-24 (2013-03-24, 06:16)MilhouseVH Wrote: @ctawn: new version (0.2.2) uploaded, working with Python 2.6.6 which had the same error you had with 2.5.1, so hopefully it's now fixed. Not sure if I can support 2.5.1 going forward but we'll see (doesn't the ATV2 have a more recent version of Python?) Thanks Milhouse, for the fast work. This time it fails on a json dependency: Code: ATV:/bin root# ./texturecache.py c I don't know if there's a newer Python available, but 2.5.1 is what "apt-get install" yields. I'm actually surprised that XBMC doesn't already depend on python, and that it wasn't installed already. Could not get json via apt-get install either. This is all a little involved for me anyway, so I'm going to get around my problems (hopefully) by reverting to my old userdata folder before I used the Thumbnail Cleaner script, and then deleting all the old pre-Frodo thumbnails (as described by jmarshall). RE: [RELEASE] Texture Cache maintenance utility - charrua - 2013-03-24 (2013-03-24, 10:44)MilhouseVH Wrote: Version 0.2.3:Perfect! A very good call to add the property cache.ignore.types The log file is also an amazing addition to know in more detail what's going on. Thanks again for all your excellent work. Now the bad news I was testing the other options, and I found that there's something wrong with r/R, as it is showing all the files in the cache as orphaned. While testing the R option, the script deleted the entire XBMC cache, so I'm now rebuilding it using the c option. Please let me know if you need any more data on this issue. Option qax is working partially: it requests XBMC the rescan of the affected sources, but XBMC doesn't refresh the info on the files on that source, probably because the already calculated hash of the source (stored in the media DB in the field path.strHash) has not changed. Deleting the hash stored in that field and then asking XBMC for a new scan of that source should do the trick. RE: [RELEASE] Texture Cache maintenance utility - Milhouse - 2013-03-24 (2013-03-24, 14:29)charrua Wrote: Perfect! A very good call to add the property cache.ignore.typesThanks. (2013-03-24, 14:29)charrua Wrote: Now the bad news Ouch! Sorry about that, but unfortunately it's working fine for me here... can you run the script without parameters and check your config is correct, and pointing at the correct database etc.? Not sure how it could get it quite so spectacularly wrong unless the texture db really was empty... avoiding the R option would be sensible until we can get to the bottom of this. (2013-03-24, 14:29)charrua Wrote: Option qax is working partially: it requests XBMC the rescan of the affected sources, but XBMC doesn't refresh the info on the files on that source, probably because the already calculated hash of the source (stored in the media DB in the field path.strHash) has not changed. Yeah, that's what I was afraid of. Not sure if I can change the hash as I'm using JSON rather than accessing the media library directly but will have a hunt around, thanks for the tip. (2013-03-24, 13:20)ctawn Wrote: Thanks Milhouse, for the fast work. This time it fails on a json dependency: To be honest, I have no experience of the ATV2 and no idea what version of Python is running or capable of running and I'm really surprised it's as old as you say, however the lack of json makes me think something isn't quite right with your setup, maybe it's a paths issue. Also, skeeney in this thread had the script working fine on an ATV2, so if you're still interested you might want to follow up or hope someone can suggest something because it should work. RE: [RELEASE] Texture Cache maintenance utility - charrua - 2013-03-24 (2013-03-24, 14:41)MilhouseVH Wrote:No problem, I made a backup of the Thumbnails folder before trying the R option, but I'm using the c option anyway to build it again from scratch.(2013-03-24, 14:29)charrua Wrote: Now the bad newsOuch! Sorry about that, but unfortunately it's working fine for me here... can you run the script without parameters and check your config is correct, and pointing at the correct database etc.? Not sure how it could get it quite so spectacularly wrong unless the texture db really was empty... avoiding the R option would be sensible until we can get to the bottom of this. The config seems fine: Code: Current properties (if exists, read from D:\programas\multimedia\XBMC\scripts\texturecache.cfg): And the options s, qa, p and c are also working fine. The problem is only with the r/R options. Maybe you can add some info to the log file while using those options to try and fix the problem. Please let me know if you want me to test anything else. RE: [RELEASE] Texture Cache maintenance utility - Milhouse - 2013-03-24 Ah... Windows user... penny just dropped. I think I know what happened. A week or so ago I updated the code to use os.sep on paths to improve Windows compatability (os.sep being "/" on Linux, "\" on Windows), and I also used os.sep when forming the hash (directory + filename) that is calculated as the orphan scan option walks the Thumbnails directory. So on Linux the hash for a file would be formed as "f/ffd2826e.jpg", and on Windows it would be "f\ffd2826e.jpg". This value is then matched against the database hash, to see if it exists or not (is an orphan), and this is where the problem lies... The database hash, I've now realised, always uses the Linux "/" format, so _nothing_ in the database matched against the calculated hashes under Windows, and all files were considered orphans! Oops, very sorry. My fault. v0.2.5 is updated as follows: * Fix hash so that always uses Linux path seperator * Apply 5% limit when identifying files for orphan removal. If the number of orphaned files exceeds 5% of the total database files (rows), the R operation will be aborted before any files are removed. RE: [RELEASE] Texture Cache maintenance utility - charrua - 2013-03-24 (2013-03-24, 15:42)MilhouseVH Wrote: Ah... Windows user... penny just dropped. I think I know what happened.Hehe, I was just about to post that probably the problem was in this line: Code: hash = "%s%s%s" % (dir, os.sep, f) The safety measure for the R option is very thoughtful. RE: [RELEASE] Texture Cache maintenance utility - Milhouse - 2013-03-24 (2013-03-24, 14:41)MilhouseVH Wrote:(2013-03-24, 14:29)charrua Wrote: Option qax is working partially: it requests XBMC the rescan of the affected sources, but XBMC doesn't refresh the info on the files on that source, probably because the already calculated hash of the source (stored in the media DB in the field path.strHash) has not changed. I've had a quick look at the JSON API again, and there's nothing there that seems likely to do the job, so the basic scan is all there is. Unless @jmarshall can suggest anything? I really haven't looked at how the scanner works, but it would be nice if there was a JSON method to force a re-scrape of media, rather than have the scraper do nothing because it doesn't believe there is anything wrong, or that requires updating. RE: [RELEASE] Texture Cache maintenance utility - charrua - 2013-03-24 (2013-03-24, 16:08)MilhouseVH Wrote:Maybe you could use VideoLibrary.RemoveMovie first (or VideoLibrary.RemoveTVShow, VideoLibrary.RemoveTVShow, VideoLibrary.RemoveEpisode, etc.) and then call for a source scan.(2013-03-24, 14:41)MilhouseVH Wrote:(2013-03-24, 14:29)charrua Wrote: Option qax is working partially: it requests XBMC the rescan of the affected sources, but XBMC doesn't refresh the info on the files on that source, probably because the already calculated hash of the source (stored in the media DB in the field path.strHash) has not changed. RE: [RELEASE] Texture Cache maintenance utility - Milhouse - 2013-03-24 (2013-03-24, 16:18)charrua Wrote:(2013-03-24, 16:08)MilhouseVH Wrote:Maybe you could use VideoLibrary.RemoveMovie first (or VideoLibrary.RemoveTVShow, VideoLibrary.RemoveTVShow, VideoLibrary.RemoveEpisode, etc.) and then call for a source scan.(2013-03-24, 14:41)MilhouseVH Wrote: Yeah, that's what I was afraid of. Not sure if I can change the hash as I'm using JSON rather than accessing the media library directly but will have a hunt around, thanks for the tip. Hmm, yeah, I'll have a go with RemoveEpisode/RemoveMovie - removing the entire TVShow is possibly a little excessive so hopefully it doesn't come to that! Still, would be nice if it were possible to make the scraper run unconditionally on a specific folder, or even media entry. RE: [RELEASE] Texture Cache maintenance utility - charrua - 2013-03-24 (2013-03-24, 16:20)MilhouseVH Wrote: Still, would be nice if it were possible to make the scraper run unconditionally on a specific folder, or even media entry.1+ (2013-03-24, 16:20)MilhouseVH Wrote:As a safety measure, it'd be wise reading the media details before asking XBMC to remove it.(2013-03-24, 16:18)charrua Wrote:Hmm, yeah, I'll have a go with RemoveEpisode/RemoveMovie - removing the entire TVShow is possibly a little excessive so hopefully it doesn't come to that!(2013-03-24, 16:08)MilhouseVH Wrote: I've had a quick look at the JSON API again, and there's nothing there that seems likely to do the job, so the basic scan is all there is. Unless @jmarshall can suggest anything? I really haven't looked at how the scanner works, but it would be nice if there was a JSON method to force a re-scrape of media, rather than have the scraper do nothing because it doesn't believe there is anything wrong, or that requires updating.Maybe you could use VideoLibrary.RemoveMovie first (or VideoLibrary.RemoveTVShow, VideoLibrary.RemoveTVShow, VideoLibrary.RemoveEpisode, etc.) and then call for a source scan. After source refresh is done you could even check the new info against the old one and restore fields as needed. RE: [RELEASE] Texture Cache maintenance utility - Milhouse - 2013-03-24 (2013-03-24, 16:28)charrua Wrote:(2013-03-24, 16:20)MilhouseVH Wrote:As a safety measure, it'd be wise reading the media details before asking XBMC to remove it.(2013-03-24, 16:18)charrua Wrote: Maybe you could use VideoLibrary.RemoveMovie first (or VideoLibrary.RemoveTVShow, VideoLibrary.RemoveTVShow, VideoLibrary.RemoveEpisode, etc.) and then call for a source scan.Hmm, yeah, I'll have a go with RemoveEpisode/RemoveMovie - removing the entire TVShow is possibly a little excessive so hopefully it doesn't come to that! Good news... removing media items seems to do the trick as far as scanning local NFOs is concerned. Thanks charrua. As a test, I modified the plot for a tvshow episode NFO, ran the qax option on the tvshow and the episode - as expected - failed QA, it was removed (just the episode) from the media libary, then the directory re-scanned and the new modified plot became visible in the GUI. I've tested the same with a movie, and no problem, it worked fine. I've uploaded version v0.2.6 with this change, so that qax now removes media items from the media library database before re-scanning - use at your own discretion! Feedback would be most welcome as I've got so few test cases (all my media is local, and pretty much complete). Regarding the safety measure... probably a lot of work! RE: [RELEASE] Texture Cache maintenance utility - ctawn - 2013-03-24 (2013-03-24, 14:41)MilhouseVH Wrote: To be honest, I have no experience of the ATV2 and no idea what version of Python is running or capable of running and I'm really surprised it's as old as you say, however the lack of json makes me think something isn't quite right with your setup, maybe it's a paths issue. Also, skeeney in this thread had the script working fine on an ATV2, so if you're still interested you might want to follow up or hope someone can suggest something because it should work. OK, thanks. as I said over here, I wound up just manually reverting to my /userdata/ folder minus the pre-Frodo thumbnails, and got the results I wanted that way (saving 1 GB). If the XBMC team doesn't implement some cache cleanup internally by the next time I need to reclaim space, I will return to this script! Thanks. Edit: Note that I see no indication that skeeney got it to work on an ATV2. Looks like he was using Windows. RE: [RELEASE] Texture Cache maintenance utility - Milhouse - 2013-03-24 (2013-03-24, 20:04)ctawn Wrote: Edit: Note that I see no indication that skeeney got it to work on an ATV2. Looks like he was using Windows. Post #12? That particular post is discussing an issue with the web server (an issue which is now closed) and appears to be running on an Apple TV2, suggesting there is no problem with the json module. RE: [RELEASE] Texture Cache maintenance utility - charrua - 2013-03-25 (2013-03-24, 17:26)MilhouseVH Wrote:I'm glad you could make it work, thank you.(2013-03-24, 16:28)charrua Wrote:Good news... removing media items seems to do the trick as far as scanning local NFOs is concerned. Thanks charrua.(2013-03-24, 16:20)MilhouseVH Wrote: Hmm, yeah, I'll have a go with RemoveEpisode/RemoveMovie - removing the entire TVShow is possibly a little excessive so hopefully it doesn't come to that!As a safety measure, it'd be wise reading the media details before asking XBMC to remove it. (2013-03-24, 17:26)MilhouseVH Wrote: Regarding the safety measure... probably a lot of work!Maybe it's simpler that it seems... JSON: VideoLibrary.GetMovieDetails (or VideoLibrary.GetEpisodeDetails, etc.) ---> Save to temp nfo file Just to have a backup of the movie details (or other media) before XBMC deletes it from the DB And if you want to complicate it a little with a data restoring option: read info from the temp nfo JSON: VideoLibrary.SetMovieDetails (or VideoLibrary.SetEpisodeDetails, etc.) |