[RELEASE] Texture Cache Maintenance utility - Printable Version +- Kodi Community Forum (https://forum.kodi.tv) +-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33) +--- Forum: Supplementary Tools for Kodi (https://forum.kodi.tv/forumdisplay.php?fid=116) +--- Thread: [RELEASE] Texture Cache Maintenance utility (/showthread.php?tid=158373) Pages:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
|
RE: [RELEASE] Texture Cache maintenance utility - Milhouse - 2013-03-26 Version 0.2.7: * Added new QA test, qa.file (yes/no, default:no) to check if the underlying video file still exists. Won't initiate remove/rescan in qax, obviously , and is thus purely informational. * Added "Duplicate" statistic summary class shown at the end of c/C. This is for when the same artwork is used multiple times. Rather than attempting to re-cache the same item as it is referenced each time by different media (which is pointless, and worse, with the C option could result in the same image being removed then re-cached multiple times), the script now remembers each image item as it is referenced during the current run, and will only cache an image once, quickly ignoring any duplicate references, resulting in improved run-times. * Tidied up logging, c/C statistics (now with added key), and a little bit of code refactoring (hope it didn't break anything...) RE: [RELEASE] Texture Cache maintenance utility - Milhouse - 2013-03-26 Version 0.2.8: * Added jd and Jd options. Functionally equivalent to the existing j and J options, these new options will also decode encoded URLs. RE: [RELEASE] Texture Cache maintenance utility - Milhouse - 2013-03-26 Version 0.2.9: * Added support for Video Library Tags. Use the "tags" keyword as you would movies/sets/tvshows etc., and optionally filter by a tag name (including partial tag names). Options with tag support: j, J, jd, Jd, c, C, qa and qax. For example: Code: ./texturecache.py j tags live-comedy The last example will process all movies that have at least one tag assigned. Not currently possible to filter by more than one tag. RE: [RELEASE] Texture Cache maintenance utility - charrua - 2013-03-26 (2013-03-26, 12:52)MilhouseVH Wrote: Version 0.2.9:Thanks for all the new features! One more suggestion for the future: multiple threads for speeding up long tasks RE: [RELEASE] Texture Cache maintenance utility - Milhouse - 2013-03-26 (2013-03-26, 13:09)charrua Wrote: One more suggestion for the future: Run multiple separate processes? At the end of the day, the speed is determined by the server... some systems (eg. Pi) are pretty much maxed out by a single task. RE: [RELEASE] Texture Cache maintenance utility - un1versal - 2013-03-26 I have a question regarding cache. Currently total duration mostly noticeable on TVshows needs to be calculated/cached in Im not sure where this info is cached (I do believe its textures#.db since this file grows as this info is cached) but its a slow process that delays displaying of SD/HD (blue/green) flags in confluence and other flagging is also delayed until total duration is calculated for all materials. To calculate this for TV shows you need to be in library view and under all seasons/episodes (lists all episodes oof all seasons view) so total duration starts being calculated and thus SD/HD and codec info flags start showing up. Can this script somehow trigger this caching in future? Particularly where a user doesnt have to navigate around and stay in that root dir waiting for this caching/calculation to take place. its a long monotonous job.# as soon as you leave any directory this caching stops so basically means if you want everything cached its a pita. uNi RE: [RELEASE] Texture Cache maintenance utility - charrua - 2013-03-26 (2013-03-26, 13:14)MilhouseVH Wrote:Specially while using c/C options, I've noticed that the script is slower at making XBMC preload the images than XBMC loading the images (not yet cached) when using the GUI. I think that's because XBMC loads multiple images at once (even when they aren't yet cached) while the script is loading them one by one.(2013-03-26, 13:09)charrua Wrote: One more suggestion for the future:Run multiple separate processes? At the end of the day, the speed is determined by the server... some systems (eg. Pi) are pretty much maxed out by a single task. The process is not CPU intensive, probably the speed is determined by the http requests to get the images, which I think could be greatly improved by implementing multiple threads. RE: [RELEASE] Texture Cache maintenance utility - Milhouse - 2013-03-26 (2013-03-26, 13:26)charrua Wrote: Specially while using c/C options, I've noticed that the script is slower at making XBMC preload the images than XBMC loading the images (not yet cached) when using the GUI. I think that's because XBMC loads multiple images at once (even when they aren't yet cached) while the script is loading them one by one. I think the default is for 2 threads on XBMC (certainly on the Pi). You could run the script on a more powerful machine, such as a desktop, or even multiple scripts on a desktop, and point the desktop at an XBMC client. However it's the image conversion that has to be done by the XBMC which is where the load/delays can occur, particularly on a Pi. Not even sure if multiple http downloads are possible (this is what triggers the caching process). Thing is, though, I really don't want to to get involved in writing threaded code if I can help it! If I run out of ideas I'll kick it around to see what's involved... but no promises! (2013-03-26, 13:17)uNiversal Wrote: I have a question regarding cache. Certainly this script will cache the tvshow images, but not entirely sure about the stream details (SD/HD etc.), it may be possible to "pre-load" that information by requesting the stream details via json (can already do this with the J/Jd option and adding streamdetails property to extra.json.tvshows.episode)... I'll investigate shortly, just adding some extra tag support, or give it a try and see if you get valid stream details for newly added tvshows (and then see if they're there in the GUI when you access the Season/Episode list). RE: [RELEASE] Texture Cache maintenance utility - charrua - 2013-03-26 (2013-03-26, 13:43)MilhouseVH Wrote:Thanks! (only a suggestion anyway... )(2013-03-26, 13:26)charrua Wrote: Specially while using c/C options, I've noticed that the script is slower at making XBMC preload the images than XBMC loading the images (not yet cached) when using the GUI. I think that's because XBMC loads multiple images at once (even when they aren't yet cached) while the script is loading them one by one.I think the default is for 2 threads on XBMC (certainly on the Pi). I tried running multiple instances of the script simultaneously, and it worked, so it will probably work using multiple threads too, but I understand completely that using threads in a script is not always the easiest way to go. (2013-03-26, 13:17)uNiversal Wrote: I have a question regarding cache.The total duration of media is not cached, it's calculated in real time each time you access a list. If you want the loading of lists to be faster you can use a skin that doesn't show that info (there are a few). RE: [RELEASE] Texture Cache maintenance utility - un1versal - 2013-03-26 (2013-03-26, 13:43)MilhouseVH Wrote: Certainly this script will cache the tvshow images, but not entirely sure about the stream details (SD/HD etc.), it may be possible to "pre-load" that information by requesting the stream details via json (can already do this with the J/Jd option and adding streamdetails property to extra.json.tvshows.episode)... I'll investigate shortly, just adding some extra tag support, or give it a try and see if you get valid stream details for newly added tvshows (and then see if they're there in the GUI when you access the Season/Episode list). Excellent, if you wish to duplicate the behaiviour I described, try this with a TV show that has at least 50 episodes. Remove it from library and re-add it then navigate to TVShows>Library>Tittle>All Seasons you will notice all stream details are being re-cached and not listed immediately, as total duration gets calculated and to make flags show up and they get calculated jump from listing to scroll bar (this triggers refresh of flags). Would be extra nice if script could trigger the calculation of the total duration as well but am unsure if this is at all possible. Thanks for response will be keeping an eye. uNi RE: [RELEASE] Texture Cache maintenance utility - Milhouse - 2013-03-26 v0.3.0 is up: * Added combinatorial logic for tags, eg.: Code: ./texturecache.py c tags "comedy and standup" RE: [RELEASE] Texture Cache maintenance utility - Milhouse - 2013-03-26 (2013-03-26, 13:57)uNiversal Wrote: Would be extra nice if script could trigger the calculation of the total duration as well but am unsure if this is at all possible. Sadly not having much luck. I've tried all the methods that return stream details (Files.GetFileDetails, VideoLibrary.GetEpisodeDetails, individually and in combination), even downloading the video file (Files.PrepareDownload then downloading over HTTP, tried Files.Download), and I can't find any JSON method that causes XBMC to extract and return the stream details. Basically, unless you've played back the video in XBMC (or listed it in the GUI, waiting for the SD/HD tags to appear), then XBMC will refuse to return stream details over JSON, which sucks, as I know the problem you are referring to but it looks like it's being handled specially by the GUI and beyond JSON at this time. Probably should be raised as an issue, as this renders the streamdetails side of JSON nigh on useless - virtually my entire library has no stream details according to JSON. I don't have any iOS or Android apps here, but I'm guessing they don't show streamdetails for videos that haven't yet been pulled up in the GUI, or never played back,but if they are showing these stream details for new media then there must be a way... Suggestions welcomed! (2013-03-26, 13:55)charrua Wrote: The total duration of media is not cached, it's calculated in real time each time you access a list. It seems this has to be extracted from the media on the fly, if it's not already specified in the NFOs, and this is only done the first time the item is listed (or played back)... once that information is extracted, and stored in the media library, it should be much quicker to calculate the total runtime of a given list of items. The trick is finding the trigger that causes XBMC to extract this information from specific media items - it certainly doesn't look like this has been implemented for JSON, unfortunately. Grrrr... RE: [RELEASE] Texture Cache maintenance utility - Milhouse - 2013-03-26 (2013-03-26, 15:07)MilhouseVH Wrote: The trick is finding the trigger that causes XBMC to extract this information from specific media items - it certainly doesn't look like this has been implemented for JSON, unfortunately. Grrrr... Here's the official answer - can't be done right now, and no easy solution. RE: [RELEASE] Texture Cache maintenance utility - un1versal - 2013-03-26 I dont have iOS or android apps either Im in Linux Ubuntu. But yea sure is a shortcomming of JSON, but who knows if it will get implemented anytime soon. Thanks anyway for looking into this and for prompt replies. Fantastic While Im here would make some suggestion, it would be great if this was a addon in XBMC which one could use with a remote/keyboard via UI or even better if it was part of a web interface like XWMM. Perhaps you can even consider using github for development/delivery of anything like this as well... Thanks for tool anyway though haven't used it I can see this being a part of my maintenance tools for XBMC in future. uNi RE: [RELEASE] Texture Cache maintenance utility - Milhouse - 2013-03-26 Thanks for your kind words, however if you ran the script you might realise why it won't be an addon any time soon! I'm much happier with it at the command line for now, and I'm really not sure if I'd be able to create a suitable gui for this, plus half it's current functionality would most likely be lost. Hope you find a use for it soon, I'd be interested in your feedback. I do intend to put it on github eventually. |