• 1
  • 66
  • 67
  • 68(current)
  • 69
  • 70
  • 197
[RELEASE] Texture Cache Maintenance utility
Cool No problem.. I updated from 1.6.4 to 1.6.6 successfully and everything is fine now Smile cheers !
~Web and graphic designer... Linux enthusiast... Python Fan... A Gooner~
[AMD A10-7850K 3.7 Ghz, Radeon R7]
[Fedora 27] | [Kodi - 17.6 / Skin - Grid]

Image
Reply
Hi there! I screwed up my cache and I thought I'd try fixing it by force with `./texturecache.py C` (with the config set appropriately). Weirdly enough, it goes through everything and errors out on every one (the following items could not be downloaded...). However, each of the items it tells me it couldn't download includes a URL that leads to a perfectly downloadable file.

Help?
Reply
Check xbmc.log for errors. Turn on debugging, particularly curl logging. Some websites may reject multiple concurrent requests, try reducing the number of download threads to 1.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
I've been using your very useful utility as needed for a while now and it's worked very well.

I now have an issue with some addon icons and fanart. The cached images appear to be missing as the icons are displayed as the jigsaw puzzle and the backgrounds are the skin default. I checked the logs and noticed an error that a particular fanart was missing so I manually copied it from the addon folder to the appropriate cache location and that resolved the fanart problem for that addon. The icon is still the jigsaw puzzle and no error log was produced.

I've tried a number of things to figure out what the issue is.

./texturecache.py C addons "TMZ" gave an error on the thumbnail. With debug on there was nothing in the log to indicate the issue.

python ./texturecache.py j addons gave a list of addons as expected with the correct icons and fanart in the addons directory.


So I have a couple of questions.

1) How can I fix this with the script?

2) How can I find the location of the offending cached icon? I have no problem fixing the few broken ones I have by hand if necessary, as I did with the missing fanart.

Thanks
Reply
If you run "./texturecache.py jd addons tmz" you should see the artwork urls associated with the addon - check that these artwork items exist, and are accessible to XBMC (as long as they're in the .xbmc/addons folder, they should be accessible, but if you're doing something like path substitution then all bets are off!)

You can search for the cached TMZ artwork with "./texturecache.py s tmz" - this will list the database rows with matching urls (urls containing tmz - use the same urls from the above "jd" call if you want to be more precise). You can then delete individual rows and associated cached artwork files with "./texturecache.py d #" where # is the rowid (left-most column in the list).

After removing any cached artwork items you can then perform a normal cache pre-load with "./texturecache.py c addons tmz", that should pre-load any missing (ie. recently deleted) artwork items.

If you still have errors then enable debug log (wiki) in XBMC, ideally with curl debugging enabled, and upload it to pastebin after trying to preload the cache for this addon. A logfile from texturecache.py would also be useful (add @logfile=/tmp/tc.log when pre-loading the cache and upload tc.log to pastebin.com).
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Thanks for the quick response.

I don't use path substitution.

In all my cases the addons were installed correctly and had tha jpg's and png's in the proper location. The files in cache indicated in the "s tmz", etc queries did not exist. I simply copied the files (and renamed them) to the proper place in cache and full joy.

I would have thought that "addons c tmz" should have fixed this, but for sure "addons C tmz" would write the missing image file, but they did not.

Anyway, thanks again for the quick accurate response.
Reply
(2014-07-13, 05:36)gargamon Wrote: In all my cases the addons were installed correctly and had tha jpg's and png's in the proper location. The files in cache indicated in the "s tmz", etc queries did not exist. I simply copied the files (and renamed them) to the proper place in cache and full joy.

OK, so what you're saying is that you have rows in the database but the corresponding files (cachedurl) are not in the cache?

This would make sense if there are problems writing the artwork files into the cache (Thumbnails folder), and would suggest the image conversion (resizing etc.) is failing for some reason, after the database row is inserted. Maybe there's something peculiar about the artwork being used by this addon?

By the way, "./texturecache.py X" will identify any Textures13 database rows that do not have a corresponding cache file. And "./texturecache.py Xd" will automatically remove these rows from the database.

To identify cached artwork that is no longer referenced by the database , use "./texturecache.py r", and to automatically remove this artwork use "./texturecache.py R".

(2014-07-13, 05:36)gargamon Wrote: I would have thought that "addons c tmz" should have fixed this, but for sure "addons C tmz" would write the missing image file, but they did not.

It should have, but if XBMC can't write the file to the cache then that's a problem and not one the script can solve.

As a test, remove the artwork from the cache (remove from both Textures13.db and the Thumbnails folder), restart XBMC, enable debug logging (and CURL component debugging, if available in your XBMC build) and then browse to the TMZ addon icon/fanart in the GUI. Can you confirm if you still have the same problem, ie. no artwork displayed for the addon? If so, there should be errors present in the XBMC log.

It would be useful to know why XBMC is failing to cache these artwork files, it could be a jpeg/png format issue, and/or could be dependent on the device you are using for XBMC (R-Pi, Android, x86 etc.).

If you could upload a full debug log (wiki) that would be great, although it's almost certainly an XBMC-related issue (which would be confirmed if XBMC fails to update the cache via the GUI) and should probably be addressed in the general help section.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Ok, here we go. I used raspbmc.browser instead of tmz to do my tests. Raspbmc.browser is in the same state tmz was before I "fixed" it. I saw nothing in the logs about that addon, but here they are:
http://pastebin.com/tTJVnEDQ

Because I saw nothing there I thought it might be useful to grab the texturecache logs. I ran 2 scenarios, with "c addons browser"
http://pastebin.com/dbqCbyGQ

and with "C addons browser"

http://pastebin.com/wubP9Sea

I hope that's enough to give a clue as to the problem.

Some historical info that may be helpful. I used to run R-Pi openelec and recently converted to Raspbmc. Originally the TMZ icon worked in Openelec. Some time before I migrated to Raspbmc I needed to rebuild my cache database in openelec. I deleted all the thumbnails and the texturesXX.db and ran the texturecache utility to repopulate the cache. I think sometime around then I lost the TMZ and other icons.

On the change from openelec to raspbmc I migrated from nfs to fstab mounting of my media. This also necessitated a rebuild of the texture cache as above.

Please let me know if you find anything.
Reply
(2014-07-13, 10:10)gargamon Wrote: Ok, here we go. I used raspbmc.browser instead of tmz to do my tests. Raspbmc.browser is in the same state tmz was before I "fixed" it. I saw nothing in the logs about that addon, but here they are:
http://pastebin.com/tTJVnEDQ

Hmmm... as you say, there's no evidence of XBMC loading the icon.png or fanart.jpg artwork for raspbmc.browser at all in your debug log...

What I can see from the texturecache logs is that the fanart.jpg has been cached successfully, but the icon.png fails - why, I don't know, it's XBMC that is failing here.

When I re-cache artwork for an addon (eg. "C addons artwork.downloader") I'm seeing confirmation that the artwork is successfully cached (written) to the file system - note the "Fast Caching image" and "cached image" entries here.

Can you upload the XBMC debug log when running "C addons raspbmc.browser @Download.threads=1" - using a single download thread will serialise the webserver requests and make it easier to follow. If there are still no errors in your debug log, then I would suggest deleting Textures13.db in case it is corrupt, and if that still doesn't help then try re-installing...
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Here's the XBMC debug logs you requested. http://pastebin.com/rHXuhLc8

I'm happy with the way my XBMC is running now. The icon's I care about have all been fixed and they were the last thing I needed to get right. The later debugging stuff was just to assist in tracking down a potential bug.

Thanks again for all your help.
Reply
This is the only debug written for the icon.png requests:
Code:
18:14:48 T:2903000128   DEBUG: webserver: request received for /jsonrpc
18:14:49 T:2821715008   DEBUG: Previous line repeats 3 times.
18:14:49 T:2821715008    INFO: JSONRPC Server: Disconnection detected

texturecache.py will retry 3 times when communicating with the web server (4 failed requests in total). So what we see here is the web server failing to complete the Files.PrepareDownload method for this file - it doesn't appear to get as far as actually loading the file, but without more detail it's hard to say why it is failing, though I'd double check file permissions.

Enabling CURL debug might reveal more detail regarding the failure, but if you're happy I'm happy.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Interestingly enough, after my last test I decided to clean up the textures db and ran the "Xd" and "R" commands and then the browser icon magically appeared.

Are you sure the code checks if there is an actual file in the cache or only if the db says there is one?
Reply
(2014-07-13, 12:48)gargamon Wrote: Interestingly enough, after my last test I decided to clean up the textures db and ran the "Xd" and "R" commands and then the browser icon magically appeared.

Are you sure the code checks if there is an actual file in the cache or only if the db says there is one?

The db is king. The caching code looks only at the db for a few reasons: 1) performance - hitting the filesystem would be very slow (plus, for remote users, there is no JSON API for that), 2) it's basically the same approach used by XBMC, and 3) the db is supposed to be correct!

However we do know from time to time that the db lies, which is when problems start - and that's why the Xd/R options exist to help recover from these problems and synchronise the db (Textures13.db) and filesystem (Thumbnails) once more.

Having a file in the Thumbnails folder that isn't referenced by the db - fixed by R - isn't generally a problem. When XBMC tries to display the artwork and fails to find a row in the db it will create a new db row and then overwrite the existing Thumbnails file. At worst it wastes a little file space but shouldn't ever result in any visual/display issues.

However a row in the db with no underlying Thumbnails file is a different matter entirely. In this case XBMC will find the row in the db, then look in the filesystem but fail to find the file, the net result being that no artwork is displayed (you'll see only the standard placeholders).

In theory XBMC should log an error when it can't find the file but as you've seen that doesn't always happen so it can be a very perplexing problem. The only way for XBMC to create the missing file is for the offending db row to be removed. Fortunately "Xd" will do this automatically.

Anyway, glad you got it sorted.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Hi Milhouse,

just a quick question. I was browsing my library and looking for "obscure/rare" movies, therefore i would love to sort/list my movies according to the imdb vote numbers. Is this somehow possibile with your script, for example with a json query?
Reply
(2014-07-13, 13:10)Milhouse Wrote:
(2014-07-13, 12:48)gargamon Wrote: Interestingly enough, after my last test I decided to clean up the textures db and ran the "Xd" and "R" commands and then the browser icon magically appeared.

Are you sure the code checks if there is an actual file in the cache or only if the db says there is one?

The db is king. The caching code looks only at the db for a few reasons: 1) performance - hitting the filesystem would be very slow (plus, for remote users, there is no JSON API for that), 2) it's basically the same approach used by XBMC, and 3) the db is supposed to be correct!

However we do know from time to time that the db lies, which is when problems start - and that's why the Xd/R options exist to help recover from these problems and synchronise the db (Textures13.db) and filesystem (Thumbnails) once more.

Having a file in the Thumbnails folder that isn't referenced by the db - fixed by R - isn't generally a problem. When XBMC tries to display the artwork and fails to find a row in the db it will create a new db row and then overwrite the existing Thumbnails file. At worst it wastes a little file space but shouldn't ever result in any visual/display issues.

However a row in the db with no underlying Thumbnails file is a different matter entirely. In this case XBMC will find the row in the db, then look in the filesystem but fail to find the file, the net result being that no artwork is displayed (you'll see only the standard placeholders).

In theory XBMC should log an error when it can't find the file but as you've seen that doesn't always happen so it can be a very perplexing problem. The only way for XBMC to create the missing file is for the offending db row to be removed. Fortunately "Xd" will do this automatically.

Anyway, glad you got it sorted.


I was thinking about this again. It seems to me that when you're doing the "C" option, you really want to overwrite whatever is there, whether it's nothing, a corrupted file, or who knows what. Wouldn't that imply that you no longer trust the database and want to rewrite that row anyway? I know it would be slower than now, but the added functionality of being able to do "C tvshows some_show_name" is much more versatile than doing "Xd"on a particular row when you may want to do 20 or more rows. Maybe a new command could handle this.
Reply
  • 1
  • 66
  • 67
  • 68(current)
  • 69
  • 70
  • 197

Logout Mark Read Team Forum Stats Members Help
[RELEASE] Texture Cache Maintenance utility17