[RELEASE] Texture Cache Maintenance utility

  Thread Rating:
  • 16 Vote(s) - 4.75 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
Milhouse Offline
Team-Kodi Member
Posts: 12,679
Joined: Jan 2011
Reputation: 624
Post: #16
Updated to version 0.1.5.

Added "p" option (prune) to identify and display cached files that are not present in the media library, eg. image/thumbnail previews that accumulate in the cache over time, or deleted movies/tv shows/music that remain in the texture cache even after cleaning the database. Option "P" will automatically remove these unnecessary files.

This version also significantly improves the performance of option r/R, which identifies "orphaned" thumbnail files (ie. image files not referenced in the texture cache db)

Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
find quote
Milhouse Offline
Team-Kodi Member
Posts: 12,679
Joined: Jan 2011
Reputation: 624
Post: #17
Update to version 0.1.6.

Added additional support for season-all-fanart.jpg and season-all-banner.jpg images.

Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
find quote
charrua Offline
Senior Member
Posts: 273
Joined: Nov 2008
Reputation: 14
Post: #18
Thanks for the great script, it's very useful.

Running on Win8 with Python 2.7.3, XBMC 13.0a1 using a remote MySQL DB, I'm having an error with the c/C option, it downloads the first image fine, but then before getting the second image to download it returns the following error:

Code:
texturecache.py C movies "30 Days of Night"
Querying CHYCH-HTPC001:9090 for all Movies...
Movie: 30 Days of Night                          [ + fanart     ]Traceback (most recent call last):
  File "D:\programas\multimedia\XBMC\scripts\texturecache.py", line 1341, in <module>
    main(sys.argv[1:])
  File "D:\programas\multimedia\XBMC\scripts\texturecache.py", line 1306, in main
    libraryQuery("cache", argv[1], argv[2], force=True)
  File "D:\programas\multimedia\XBMC\scripts\texturecache.py", line 355, in libraryQuery
    libraryAllMovies(action, filter, force, extraFields, rescan)
  File "D:\programas\multimedia\XBMC\scripts\texturecache.py", line 482, in libraryAllMovies
    for artwork in art: loadImage(force, artwork, art[artwork], imgErrors)
  File "D:\programas\multimedia\XBMC\scripts\texturecache.py", line 786, in loadImage
    URL = getImageURL(filename)
  File "D:\programas\multimedia\XBMC\scripts\texturecache.py", line 763, in getImageURL
    data = sendJSON(REQUEST, "1")
  File "D:\programas\multimedia\XBMC\scripts\texturecache.py", line 271, in sendJSON
    return json.loads(web.getresponse().read())
  File "C:\Python\lib\httplib.py", line 1030, in getresponse
    response.begin()
  File "C:\Python\lib\httplib.py", line 407, in begin
    version, status, reason = self._read_status()
  File "C:\Python\lib\httplib.py", line 365, in _read_status
    line = self.fp.readline()
  File "C:\Python\lib\socket.py", line 430, in readline
    data = recv(1)
socket.error: [Errno 10053]

Any ideas to solve this problem?
(This post was last modified: 2013-03-23 16:05 by charrua.)
find quote
popcornmix Offline
Team-Kodi Member
Posts: 7,824
Joined: Feb 2011
Reputation: 324
Post: #19
(2013-03-13 17:54)MilhouseVH Wrote:  Version 0.1.2 now up... when performing a "qa" check, only movies or tvshow episodes added during the previous 30 days will be considered. Increase/reduce this period by specifying a value (in days) for qaperiod in the properties file, eg. "qaperiod=9999" would effectively restore v0.1.1 behaviour which didn't restrict by period.

Also added additional qa tests: rating and mpaa certification (movies), plot/fanart/banner/poster (tv show) and rating (tv show episode).

Seemed to work great detecting missing information.
The question is, is it possible to force a refresh (i.e. download new plot/rating/thumbnail) on the episodes with missing items?
find quote
Milhouse Offline
Team-Kodi Member
Posts: 12,679
Joined: Jan 2011
Reputation: 624
Post: #20
(2013-03-23 15:53)charrua Wrote:  Any tips to solve this problem?

Hi yes, I've just been debugging this with another forum member. Not sure why it happens, as I don't get it on either Ubuntu 12.10 or Raspberry Pi, but it seems necessary to close the web server connection between image requests.

Can you try downloading the latest version (0.1.9):
Code:
rm ./texturecache.py
wget http://www.nmacleod.com/public/texturecache.py && chmod +x ./texturecache.py

and then add "webserver.singleshot = yes" to the properties file texturecache.cfg

Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
(This post was last modified: 2013-03-23 16:01 by Milhouse.)
find quote
charrua Offline
Senior Member
Posts: 273
Joined: Nov 2008
Reputation: 14
Post: #21
(2013-03-23 16:01)MilhouseVH Wrote:  
(2013-03-23 15:53)charrua Wrote:  Any tips to solve this problem?
Hi yes, I've just been debugging this with another forum member. Not sure why it happens, as I don't get it on either Ubuntu 12.10 or Raspberry Pi, but it seems necessary to close the web server connection between image requests.
Can you try downloading the latest version (0.1.9):
Code:
rm ./texturecache.py
wget http://www.nmacleod.com/public/texturecache.py && chmod +x ./texturecache.py
and then add "webserver.singleshot = yes" to the properties file texturecache.cfg
It's fixed, thanks a bunch for the quick answer and solution!
find quote
Milhouse Offline
Team-Kodi Member
Posts: 12,679
Joined: Jan 2011
Reputation: 624
Post: #22
(2013-03-23 15:56)popcornmix Wrote:  Seemed to work great detecting missing information.
The question is, is it possible to force a refresh (i.e. download new plot/rating/thumbnail) on the episodes with missing items?

Yes, it's possible, but not sure how effective it will be as it depends on the XBMC scraper replacing the missing information - from my experience defective meta-data can be incredibly "sticky"... Smile

I've added a "qax" option (it's in latest version 0.1.9), and it's experimental, but if you run it (eg. ./texturecache.py qax tvshows or ./texturecache.py qax tvshows "big bang", it also works for movies) then it will initiate a library rescan for those episode directories (eg. Season 5, Season 6 etc.) containing episodes that fail to pass QA - let me know how you get on...

Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
(This post was last modified: 2013-03-23 16:16 by Milhouse.)
find quote
Milhouse Offline
Team-Kodi Member
Posts: 12,679
Joined: Jan 2011
Reputation: 624
Post: #23
(2013-03-23 16:12)charrua Wrote:  It's fixed, thanks a bunch for the quick answer and solution!

I've just updated the latest version so that it shouldn't require the extra property - if it experiences a web connection problem it will switch to "single shot" mode, can you test this and let me know if it works (you'll need to remove the webserver.singleshot property from the configuration file first).

Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
find quote
charrua Offline
Senior Member
Posts: 273
Joined: Nov 2008
Reputation: 14
Post: #24
(2013-03-23 17:01)MilhouseVH Wrote:  I've just updated the latest version so that it shouldn't require the extra property - if it experiences a web connection problem it will switch to "single shot" mode, can you test this and let me know if it works (you'll need to remove the webserver.singleshot property from the configuration file first).
The new version is running perfectly (at least in my environment) without the 'webserver.singleshot' property in the config file.
I'm even running two instances at once without any problem.
Great work!
(This post was last modified: 2013-03-23 23:07 by charrua.)
find quote
Milhouse Offline
Team-Kodi Member
Posts: 12,679
Joined: Jan 2011
Reputation: 624
Post: #25
Thanks for confirming!

Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
find quote
jmarshall Offline
Team-XBMC Developer
Posts: 26,216
Joined: Oct 2003
Reputation: 185
Post: #26
MilhouseVH: mind detailing the queries you use on Textures13.db and what you'd thus need exposed the exact same functionality through JSON-RPC?

Cheers,
Jonathan

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


[Image: badge.gif]
find quote
Milhouse Offline
Team-Kodi Member
Posts: 12,679
Joined: Jan 2011
Reputation: 624
Post: #27
(2013-03-23 22:39)jmarshall Wrote:  MilhouseVH: mind detailing the queries you use on Textures13.db and what you'd thus need exposed the exact same functionality through JSON-RPC?

Cheers,
Jonathan

The main query to extract from Textures13.db is:
Code:
SELECT t.id, t.cachedurl, t.lasthashcheck, t.url, s.height, s.width, s.usecount, s.lastusetime FROM texture t JOIN sizes s ON (t.id = s.idtexture)

with an optional filter (WHERE) when searching.

I'd also want the ability to delete rows (by id), and also remove the underlying file system object.

In addition, I walk the files in Thumbnails folder, so being able to get a list of all the files in Thumbnails would be nice.

So in summary:

1) Method to search the texture table (joined with sizes table) and return all rows and columns, or a filtered subset
2) Method to delete rows from database tables (using texture.id), with optional deletion of associated filesystem (cachedurl) object
3) Method to get list of all files present in the Thumbnails folder
4) Method to delete files that are in the Thumbnails folder

Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
(This post was last modified: 2013-03-26 04:45 by Milhouse.)
find quote
ctawn Offline
Fan
Posts: 311
Joined: May 2009
Reputation: 5
Post: #28
I tried to run this on my ATV2 and got this error:

Code:
ATV2:/bin root# ./texturecache.py c
  File "./texturecache.py", line 1209
    for action in {"Skipped", "Deleted", "Error", "Updated" }:
                            ^
SyntaxError: invalid syntax

I installed it in /bin/ and I first installed Python 2.5.1 with "apt-get install python".

What am I doing wrong here?
find quote
Milhouse Offline
Team-Kodi Member
Posts: 12,679
Joined: Jan 2011
Reputation: 624
Post: #29
(2013-03-24 02:32)ctawn Wrote:  What am I doing wrong here?

Just to make sure there wasn't a problem with versions, can you remove texturecache.py and download it again. I've only ever tested this on Python 2.7.3, and the current version runs fine, and unfortunately I don't have Python 2.5.1 to test with...

EDIT: OK, thought I'd try this on my old OpenELEC 1.0 x86 box which is running Python 2.6.6 and have reproduced - working on a fix...

Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
(This post was last modified: 2013-03-24 04:58 by Milhouse.)
find quote
Milhouse Offline
Team-Kodi Member
Posts: 12,679
Joined: Jan 2011
Reputation: 624
Post: #30
@ctawn: new version (0.2.2) uploaded, working with Python 2.6.6 which had the same error you had with 2.5.1, so hopefully it's now fixed. Not sure if I can support 2.5.1 going forward but we'll see (doesn't the ATV2 have a more recent version of Python?)

Also in this version, webserver authentication is now supported - add webserver.username and webserver.password to the properties file.

Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
find quote
Post Reply