[RELEASE] Texture Cache Maintenance utility - Printable Version +- Kodi Community Forum (https://forum.kodi.tv) +-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33) +--- Forum: Supplementary Tools for Kodi (https://forum.kodi.tv/forumdisplay.php?fid=116) +--- Thread: [RELEASE] Texture Cache Maintenance utility (/showthread.php?tid=158373) Pages:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
|
RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2014-01-07 (2014-01-07, 04:35)rubpa Wrote: I'm running the first command (./texturecache.py P) right now and I see lines like below. I assume it means this path does not exist in my library but I've checked that this file exists. Yes, that url (image://) doesn't exist in your media library so the texture cache database row and the corresponding thumbnail file are being deleted. (2014-01-07, 04:35)rubpa Wrote: I see thousands of such lines (it's running for more than an hour now, I've more than 15000 pictures). Is the utility doing the right thing? Should be working correctly, assuming you really wanted those cached items to be removed. What it's doing is removing any cached artwork that is not part of your media library. Pictures/photos are not represented in the media library (they're not scanned or scraped into the library as songs, tvshows and movies are) so any associated cached artwork (such as the above "thumb" - or preview - version of your camera photo) will be removed from the cache by "pruning" (P). I hasten to add the script will never modify or remove your original images, it will only be removing the cached versions and these are typically created in order to preview a photo when browsing through the file system. I'm not sure if their removal is what you intended, but I got the impression you wanted to get rid of photos (from the cache). I've never really looked into how pictures are handled by xbmc, but as far as I can see only the smaller thumbnail previews are being cached and the original artwork is always displayed directly, not via the cache. If caching of the original picture artwork is possible (and beneficial) then this could be supported in a similar way to extrafanart/extrathumbs. At least then the previews wouldn't be cleared out when pruning as they would be matched against the original artwork. Edit: The contents of directories not within a media source cannot be viewed using JSON so scanning an arbitrary directory of photos so that they might be pre-cached is sadly not possible. If you want to keep your photo previews, I would suggest adding the following entry to texturecache.cfg: Code: prune.retain.types = <some path>/.* where <some path> is your photo path (doesn't have to be the whole path, just enough to be unlike any other non-picture path, eg. "/mnt/mypictures/.*"). Edit2: OK arbitrary photo directories are out, but if the folder is within your Pictures source then it can be scanned... and cached. Note: I recently uploaded v1.2.8 of the script as these photo urls should be being decoded - now they are (you should automatically be updated to the latest version). RE: [RELEASE] Texture Cache Maintenance utility - rubpa - 2014-01-07 (2014-01-07, 05:38)MilhouseVH Wrote: What it's doing is removing any cached artwork that is not part of your media library. Pictures/photos are not represented in the media library (they're not scanned or scraped into the library as songs, tvshows and movies are) so any associated cached artwork (such as the above "thumb" - or preview - version of your camera photo) will be removed from the cache by "pruning" (P). I have only a 10GB partition for my xbmc installation and it was full (no warning by xbmc whatsoever except slowness ). Around 7GB was consumed by Thumbnails (over 1+ yrs I think). I intended to definitely remove unnecessary thumbnails but not the thumbnails of my "Pictures". Anyway, it's not a big problem since it gets regenerated locally when I visit the folder again even without any network. Quote:I've never really looked into how pictures are handled by xbmc, but as far as I can see only the smaller thumbnail previews are being cached and the original artwork is always displayed directly, not via the cache. If caching of the original picture artwork is possible (and beneficial) then this could be supported in a similar way to extrafanart/extrathumbs. At least then the previews wouldn't be cleared out when pruning as they would be matched against the original artwork. A little confused by all this. I downloaded your script yesterday. What do you recommend for future runs? Do I need to set some options to not delete the thumbnails of my Pictures folders. After the P command, I ran texturecache.py R and got this: Code: Found 13,199 orphaned files for 1,326 database files. Also, the actual scanning & caching is done by xbmc when I view the pictures. I'm confused by "...then it can be scanned... and cached" and also your signature states "Preload your texture cache for optimal UI performance". Are these related? Does this utility improve xbmc performance by doing something more at every boot? Can you explain? Thanks a lot for your detailed response! RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2014-01-07 (2014-01-07, 09:53)rubpa Wrote: I intended to definitely remove unnecessary thumbnails but not the thumbnails of my "Pictures". Anyway, it's not a big problem since it gets regenerated locally when I visit the folder again even without any network. I'm working on a version that will not prune your picture and folder previews. Your pictures (ie. those you view, eg. full screen in a slideshow etc.) are never cached. And unfortunately it's not possible to pre-cache the previews - it looks like it's another one of those situations where the GUI does something special which can't be replicated with JSON. I should have an update shortly. (2014-01-07, 09:53)rubpa Wrote: After the P command, I ran texturecache.py R and got this:P would fail to delete the files if there is a permission problem preventing file deletion. Assuming that isn't the case, another possibility is that the orphaned files already existed before you ran the P command and may have accumulated over the previous year or so. Orphaned files often result whenever XBMC is upgraded to a new major version (eg. Dharma to Frodo or Gotham), as the thumbnail files from the old system are left lying around. (2014-01-07, 09:53)rubpa Wrote: I guess P command just cleaned the database rows and now the R command would remove the actual thumbnail file referenced in these deleted rows. Can you confirm? Only if there is a permission problem when deleting from the Thumbnails folder. But yes, the R command will clean up the left over files and get things back in to shape. However if P couldn't delete the files, then it's unlikely that R will either. (2014-01-07, 09:53)rubpa Wrote: Also, the actual scanning & caching is done by xbmc when I view the pictures. I'm confused by "...then it can be scanned... and cached" I had hoped it would be possible to cache the artwork previews, but it turns out that isn't supported by JSON, and although it is possible to cache the full resolution pictures there's absolutely no point in doing so as they're always displayed directly from source and not via the cache (which is just as well, otherwise the storage requirements for the cache would balloon). So the best I can do with this script is simply to not prune your previews, assuming most people want to keep the previews. It's not much, but it might help. And if you do want to delete the previews, there'll be an option to set. Or there is the "purge" option which takes one or more string patterns and removes items (database rows and files) from the cache. (2014-01-07, 09:53)rubpa Wrote: and also your signature states "Preload your texture cache for optimal UI performance". Are these related? Does this utility improve xbmc performance by doing something more at every boot? Can you explain? As you browse around the XBMC GUI, images will be loaded into the cache which involves resizing and format conversion etc. The images you see on the screen are loaded from the cache (except for full size pictures, which load directly from the source). On low power platforms (such as the Raspberry Pi) the extra work populating the cache can be quite noticeable, so pre-loading the cache ensures that when you browse the GUI there are no stutters or delays as all your artwork is already cached. RE: [RELEASE] Texture Cache Maintenance utility - rubpa - 2014-01-07 Thanks again for your response. I'll investigate my system a little more. (2014-01-07, 10:37)MilhouseVH Wrote: ...so pre-loading the cache ensures that when you browse the GUI there are no stutters or delays as all your artwork is already cached. Can you explain "pre-loading the cache" again? I guess it's the cC option of your utility. Should I run it at startup on my system? If it matters, my xbmc box has inbuilt storage and no network drives are added as libraries. RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2014-01-07 (2014-01-07, 12:03)rubpa Wrote: Can you explain "pre-loading the cache" again? I guess it's the cC option of your utility. Should I run it at startup on my system? If it matters, my xbmc box has inbuilt storage and no network drives are added as libraries. No, you don't run it at boot, you only need to update the cache when you have loaded new media (movies, tvshows). And if you only have one machine, which is your scraping machine (ie. you're not using MySQL and sharing metadata amongst several clients) then the texture cache will be up to date anyway as the scraper client always has a fully populated texture cache. So in your case it sounds like you're running a single machine, that is also your scraper, in which case you're not going to see any benefit from running the "c" option (there's no harm in running it, there just won't be any artwork that needs to be cached). Of course if for any reason you deleted your Textures13.db or Thumbnails files, then you could run the "c" option and repopulate your cache with the minimum of effort (the alternative is to browse slowly through your Library, displaying every possible item of artwork...) When running a distributed MySQL setup it's very likely that the non-scraper clients will be missing artwork, so whenever new artwork is added by the scraper client its usually beneficial to initiate a cache pre-load on all the remaining clients using the "c" option (which will only cache artwork that isn't yet cached). The "C" option, on the other hand, will force a cache update by removing any currently cached artwork before caching the artwork again - this is useful if you have changed artwork but kept the file name the same, as the texture cache has a tendency to ignore updated artwork for a few days until a background process realises the file content has changed. So if you're impatient, or have loaded corrupt artwork, the C option can be useful. texturecache.py (Version 1.2.9)
RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2014-01-07 texturecache.py (Version 1.3.1)
When choosing the correct number of items (eg. 444) use the item count shown at the foot of the view (in Confluence), eg. "(444) Items - page (1/45)". Make sure the GUI is on the correct view/window before commencing a stress-test run, otherwise the remote input sent to the client will cause unpredictable results... Note that with an overclocked and over-volted Raspberry Pi it's quite possible to hit the thermal limit at which point the the CPU frequency will be throttled back to 700Mhz. RE: [RELEASE] Texture Cache Maintenance utility - rubpa - 2014-01-07 I understand cC option now, thanks! I ran v1.2.9 texturecache.py P command. It was in Loading: Pictures... state for 1.5 hrs with my 15000+ photo library. I could not wait longer and killed it. edit: BTW, it's 2014 and your changelog says 2013 for the last 4 entries! RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2014-01-07 It certainly shouldn't take 1.5 hours unless the xbmc client has stopped responding. If you enable logging with @log=tc.log and upload the log somewhere (it may need to be zipped) I could take a look and see what the problem is, but most likely it's a performance issue with the client trying to iterate over so many file items (maybe). To obtain a listing of your files without pruning, try the rdirectory option as follows. First view your sources and note each path: Code: $ ./texturecache.py sources To traverse the entire path for the "Photos" picture source: Code: $ ./texturecache.py rdirectory nfs://192.168.0.3/mnt/share/media/Photos If rdirectory also hangs or takes very, very long then please add @log=tc.log to the command line and send me the log - thanks. RE: [RELEASE] Texture Cache Maintenance utility - rubpa - 2014-01-08 (2014-01-07, 20:01)MilhouseVH Wrote: It certainly shouldn't take 1.5 hours unless the xbmc client has stopped responding. It progressed in less than a minute this time and I noticed that it started deleting some thumbnails related to my photos. After some analysis, xbmc seems to be storing multiple thumbnails per photo. As an example: Code: ./texturecache.py s DSC_0669 texturecache.py P is not deleting the 1st row but deleting the 2nd row. I checked a few random photos and not always both lines exist. I guess we need to figure out more how xbmc is caching Pictures. I've seen lots of variations in the stored path in the Texture13.db in the last few days. RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2014-01-08 (2014-01-08, 05:00)rubpa Wrote: That's weird, I couldn't get XBMC to cache the original full size picture (the second row in your example above), although I only tested the "slideshow" functionality - maybe an addon has caused the original artwork to be cached, which IMHO is probably not a good thing due to the extra storage requirements, whereas slideshow appears to avoid the cache by accessing the artwork direct. It shouldn't be a problem to include the original full size artwork paths in the prune processing so that both the preview and full size images are retained in the cache, but is it sensible to keep all those full size images in the cache? Personally I'd want the full size images to be pruned. I guess it can be added as an option, prune.retain.pictures, disabled by default. RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2014-01-08 @rubpa: Try version 1.3.3, just pushed with support for prune.retain.pictures. This is not enabled by default, so you'll need to add prune.retain.pictures=yes to your properties file if you want to keep the pictures in the cache. RE: [RELEASE] Texture Cache Maintenance utility - DaveInBracknell - 2014-01-09 Hi, Sorry for the dumb new user post I have searched high and low, put in a default cfg then taken it away again. I have been trying to use this great tool with no luck. Here is the tc.log 2014-01-09 00:07:31.568749:MainThread: Command line args: ['./texturecache.py', 'C', '@logfile=tc.log', '@logfile.verbose '] 2014-01-09 00:07:31.568856:MainThread: Current version #: v1.3.3 2014-01-09 00:07:31.568890:MainThread: Current platform : linux2 2014-01-09 00:07:31.568929:MainThread: Python version #: v2.7.3.0 (final) 2014-01-09 00:07:31.569052:MainThread: libPing.JSON WEB REQUEST: [{"jsonrpc": "2.0", "method": "JSONRPC.Ping", "id": "lib Ping"}] 2014-01-09 00:07:31.571446:MainThread: libPing.RECEIVED DATA: {"id":"libPing","jsonrpc":"2.0","result":"pong"} 2014-01-09 00:07:31.571998:MainThread: libVersion.JSON SOCKET REQUEST: [{"jsonrpc": "2.0", "method": "JSONRPC.Version", " id": "libVersion"}] It just sits there for ever. I have no cfg file, it is running latest frodo version (12.3) webserver is enabled with no password standard port 8080. Help I have a load of thumbnails on the tv show view in ACE showing black boxes and it is driving me mad Thanks Dave RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2014-01-09 Looks like your XBMC client isn't responding for some reason. Can sometimes happen, just interrupt the script and try again. Although if your plan is to run "C" without limiting by media class (ie. songs, artists, movies, tvshows etc.) then you may as well just delete your Textures13.db and Thumbnails folder, restart XBMC (so that it re-creates an empty database and folders) then use the "c" option to pre-load the cache. RE: [RELEASE] Texture Cache Maintenance utility - goslo - 2014-01-10 Code: Found 2,777 orphaned files for 12,002 database files. George RE: [RELEASE] Texture Cache Maintenance utility - rubpa - 2014-01-10 (2014-01-08, 05:15)MilhouseVH Wrote: That's weird, I couldn't get XBMC to cache the original full size picture (the second row in your example above), although I only tested the "slideshow" functionality - maybe an addon has caused the original artwork to be cached, which IMHO is probably not a good thing due to the extra storage requirements, whereas slideshow appears to avoid the cache by accessing the artwork direct. I did some trials on my windows laptop with XBMC 12.3 + Aeon Nox skin - similar to my actual media center which is xbmcbuntu. I added a folder with 8 trial pics and when I open the folder, the thumbnails are created and here is the output of texturecache.py f Code: 000006|2/2ad34bd6.jpg|0240|0360|0001|2014-01-10 14:47:45|2014-01-10 20:17:45|C:\Temp\Pics\P1050811.JPG/transform?size=thumb The skin uses InfoWall view by default which shows a larger image to the side when I just use the arrow keys to move through the pictures. When I selected one picture, 1 additional row was added to the database. When I browsed through all the 8 pictures with arrow keys, this is what it looked like: Code: 000010|1/1809f893.jpg|0720|1080|0001|2014-01-10 14:58:23|2014-01-10 20:28:23|C:\Temp\Pics\DSC_0956.JPG So, if the skin needs a larger preview, I guess these additional db lines & thumbnails are created. Thanks for the last update. I now understand what xbmc/skin is doing and I've your script with all options to clean my cache! |