• 1
  • 45
  • 46
  • 47(current)
  • 48
  • 49
  • 197
[RELEASE] Texture Cache Maintenance utility
Curl is a utility, like wget, to download files from web servers. If you want to use this script, copy & paste those installation commands into your ssh window (ie. PuTTY if you're on Windows, connecting to your OpenELEC client).
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
I have another question regarding the QA NFO check. It's working well, and the files are rescanned accordingly, but i witnessed a strange behaviour. For example:

1. Movie is in library with poster and fanart.
2. I change the local(!) poster and the nfo file.
3. QA Check -> Movie is rescanned and updated with new metadata and a new poster.
4. I change the local poster and the nfo file again (10 minutes later).
5. QA Check -> Movie is rescanned but the poster isn't updated this time.

Why isn't the poster updated? Am i missing something?
Reply
Your old poster is still in the cache after the movie is removed, so when the movie is added back you still have your old poster.

It would be necessary to remove all artwork from the cache of the movie being removed from the library, or forcibly re-cache the movie after it has been added back, both of which are possible but would massively slow down qax (and may not be desirable behaviour for all users). In addition, although the cache of the client on which you run the qax option would be correct (ie. the local cache now contains the new artwork), you'd still have the old artwork present on all your other clients as their cache would still be stale, and you'd need to refresh their cache individually ("C movie" etc.), so maybe a better cache pre-load option is required.

I'm thinking a cache pre-load option which compares the timestamp from the cache (lasthashcheck, or failing that lastusetime) against the modification date of the artwork file, and reloads the artwork if the modification time is more recent.

With such a change, you would run qax (reload movie, nice and quick) followed by re-cache of any artwork that is now stale in the cache (and you'd need to repeat just the re-cache step against all of your other clients). Alternatively much like qa.nfo.refresh, specify a relative date/time period and re-cache any artwork modified within that period, but right now I'm thinking the timestamp comparison method will be more useful.

Edit: Acquiring the modification timestamp of remote (http://) artwork isn't going to work, whereas specifying a relative date/time period will work for both local and remote artwork... so something like cache.refresh=today|#|date-time should work (and will be quicker too - obtaining the modification date for all local files would have been pretty slow...).

The cache.refresh modifier would only apply to the "C" and "nc" options, as it will restrict the artwork that is to be removed and reloaded, and would have no effect on the "c" option.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Ok, thanks for the info. I tried to recache the particular movie, but the script (or my Pi) crashes everytime. I tried before the "nc" or the "c" option and ran into problems too (the pi is always unresponsible after 2-3 minutes).

Code:
./texturecache.py C movies moviename

According to the log this happens:

Code:
... (snipped the ok-looking part)
2014-01-25 01:45:29.362798:MainThread: libDirectory.FINISHED, elapsed time: 0.029251 seconds
2014-01-25 01:45:29.386471:MainThread: libTextures.JSON SOCKET REQUEST: [{"jsonrpc": "2.0", "params": {"properties": ["cachedurl", "url"]}, "method": "Textures.GetTextures", "id": "libTextures"}]
2014-01-25 01:46:11.110427:MainThread: libTextures.BUFFER RECEIVED (len 21856)
2014-01-25 01:46:11.112242:MainThread: libTextures.BUFFER RECEIVED (len 21856)
2014-01-25 01:46:11.116008:MainThread: libTextures.BUFFER RECEIVED (len 32768)
... (last line is repeated 100-150 times)
2014-01-25 01:56:13.374352:MainThread: libTextures.Incomplete JSON data - continue reading socket
2014-01-25 01:56:13.383738:MainThread: Ignored Value Error: Expecting object: line 1 column 6957759 (char 6957759)
2014-01-25 01:56:13.385397:MainThread: libTextures.BUFFER RECEIVED (len 32768)
... (last line is repeated again ca. 50 times)
Reply
Is your Pi stable? Check if there are any associated errors in the xbmc.log. Or send me your Textures13.db (upload it somewhere and PM me a link) as it seems to be having a problem retrieving the rows from the Texture cache - hopefully not another foreign encoding problem...
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Nothing unusual in the xbmc.log (except for not finding some fonts).

Here is the output of texturecache.py c

Code:
./texturecache.py c
Traceback (most recent call last):                          
  File "./texturecache.py", line 6350, in <module>
    main(sys.argv[1:])
  File "./texturecache.py", line 6206, in main
    extraFields=_extraFields, query=_query)
  File "./texturecache.py", line 3373, in jsonQuery
    cacheImages(mediatype, jcomms, database, data, title_name, id_name, force, nodownload)
  File "./texturecache.py", line 3428, in cacheImages
    for r in database.getRows(allfields=False):
  File "./texturecache.py", line 1270, in getRows
    data = self.mydb.getTextures(filter, order, allfields)
  File "./texturecache.py", line 2622, in getTextures
    return self.sendJSON(REQUEST, "libTextures", checkResult=False)
  File "./texturecache.py", line 1567, in sendJSON
    udata = MyUtility.toUnicode(data)
  File "./texturecache.py", line 3070, in toUnicode
    data = unicode(data, encoding="utf-8", errors="ignore")

Texture.db link is on your way.
Reply
You'll need to send me your Textures13.db - it's as I feared... Sad
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Thanks for the database. I think you're hitting an out of memory (OOM) situation, rather than any encoding issues as I first thought.

On my Pi with your Textures13.db I ran:
Code:
./texturecache.py x
which dumps the contents of Textures13.db to the console, and with bcmstat.sh running in a separate window it becomes pretty clear that memory is the issue.

The problem is that JSON is now being used to access the texture cache database by default in recent builds, and in your case this means almost 60,000 rows are being loaded into memory objects by XBMC (~110MB), these objects are then transferred using JSON to texturecache.py where they're again stored in memory. Combined, this is a lot of data and it exhausts the available physical memory.

With 128MB swap on my 512MB Pi (256MB/256MB split) it didn't crash, but it's still running after 15 minutes and the Pi is unresponsive (though still responding to pings). Allocating more memory to the ARM might help make it run more quickly, even complete, until eventually more database rows mean that memory allocation isn't sufficient, so it's not really a long term solution.

You do however have two options:
1) Since you're running this directly on the Pi, you can disable JSON for Textures database access with @dbjson=no and then SQLite direct access will be used instead of JSON
2) Run texturecache.py on a remote PC and not the Pi, so that the memory usage on the Pi is significantly reduced (shouldn't be necessary to disable dbjson for this, although you can but it then means you have to mount the userdata folder on the remote PC).

#1 is likely to be the easiest for you. I may give some thought to making dbjson=no the default when being run locally, but dbjson=yes the default for remote access.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
texturecache.py (Version 1.4.0)
  • Add: For use with C and nc options, @cache.refresh=YYYY-MM-DD HH:MM: SS|today|#, will re-cache (C) or list (nc) only stale cache items. Stale cache items are those original local artwork files that have been modified on-disk since the specified date. Remote and inaccessible artwork will not classed as stale and instead ignored (skipped).
  • Chg: dbjson will now default to no when the script is running on localhost, and yes when xbmc.host is a remote client.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Thanks for your fast fix, but i'm not sure if it is working here:

I tried first without the "nc" option without any change (i updated manually to 1.4.0) and the Pi was again unresponsive (but still responding to pings). I have "xbmc.host = localhost" in my texturecache.cfg. Nevertheless i tried a second run with "dbjson = no" in my texturecache.cfg. Same effect. The process starts and the first output is visible (for addons), but it hangs here:

Code:
2014-01-25 12:52:47.154932:MainThread: libMovies.JSON SOCKET REQUEST: [{"jsonrpc": "2.0", "params": {"sort": {"order": "ascending", "method": "title"}, "properties": ["title", "art", "cast", "file"]}, "method": "VideoLibrary.GetMovies", "id": "libMovies"}]

Which looks like JSON to me?
Reply
Yes, that's accessing movies from the media library - dbjson applies only to the Textures database. Access to the media library is performed only with JSON, there's no alternative for that as there is for the Textures db.

What's the memory situation on your Pi like (bcmstat.sh cxd10) while running texturecache.py? Is this a 256MB or 512MB Pi, what's your memory split?
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
It's a 512MB Pi - i don't know what memory split means, therefore i'm guessing it is the usual?!

bcmstat.sh cxd10 output while running texturecache.py nc:

Code:
OpenELEC:~ # ./bcmstat.sh cxd10
  Config: v0.0.9, args "cxd10", priority lowest (+19)
Governor: ondemand
  Memory: 512MB (split 384MB ARM, 128MB GPU)
HW Block: |   ARM   |  Core  |  H264  |   SDRAM   |
Min Freq: | 1000Mhz | 500Mhz |   0Mhz |   500Mhz  |
Max Freq: | 1000Mhz | 500Mhz | 250Mhz |   500Mhz  |
Voltages: |         +4, 1.30V         |  0, 1.20V |
   Other: temp_limit=85, force_turbo=1
Firmware: Jan 10 2014 16:54:51, version efa116b5c8859c352322cb27e13baccbea583ef7 (clean) (release)
  Codecs: H264 VP8 VORBIS MJPG
  Booted: Sat Jan 25 13:11:41 2014

Time          ARM     Core     H264  Core Temp (Max)   IRQ/s      RX B/s      TX B/s   %user   %nice %system   %idle %iowait    %irq  %s/irq  %total  Memory Free/Used
========  =======  =======  =======  ===============  ======  ==========  ==========  ======  ======  ======  ======  ======  ======  ======  ======  ================
13:14:50  1000Mhz   500Mhz     0Mhz  55.00C (55.00C)     343       1,870       8,513   24.65   36.97   24.65    0.00    0.00    0.00   12.32  100.00  308,728 kB/19.1%
13:15:00  1000Mhz   500Mhz     0Mhz  54.00C (55.00C)     271         706         398   12.94    0.69    1.68   81.88    0.00    0.00    0.00   18.12  308,728 kB/19.1%
13:15:13  1000Mhz   500Mhz     0Mhz  55.00C (55.00C)     414         615       2,116   89.28    1.08    5.03    3.48    0.39    0.00    0.39   96.52  285,740 kB/25.2%
13:15:27  1000Mhz   500Mhz     0Mhz  55.00C (55.00C)     449          54          54   94.87    0.74    3.68    0.00    0.37    0.00    0.37  100.00  259,236 kB/32.1%
13:15:42  1000Mhz   499Mhz     0Mhz  56.00C (56.00C)     305          13          32   94.57    0.67    4.38    0.00    0.13    0.00    0.13  100.00  209,920 kB/45.0%
13:15:58  1000Mhz   500Mhz     0Mhz  56.00C (56.00C)     287         325       1,010   95.59    0.64    3.57    0.00    0.06    0.00    0.19  100.00  188,136 kB/50.7%
13:16:12  1000Mhz   500Mhz     0Mhz  56.00C (56.00C)     278          59          33   97.27    0.69    1.93    0.00    0.07    0.00    0.07  100.00  187,904 kB/50.8%
13:16:27   999Mhz   500Mhz     0Mhz  56.00C (56.00C)     272          34          33   97.45    0.62    2.28    0.00    0.07    0.00    0.07  100.00  188,192 kB/50.7%
13:16:41  1000Mhz   499Mhz     0Mhz  56.00C (56.00C)     277          64         648   95.88    0.55    2.83    0.00    0.14    0.00    0.07  100.00  187,156 kB/51.0%
13:16:56  1000Mhz   500Mhz     0Mhz  56.00C (56.00C)     260          31          33   97.50    0.62    1.86    0.00    0.00    0.00    0.00  100.00  187,172 kB/51.0%
13:17:10  1000Mhz   500Mhz     0Mhz  56.00C (56.00C)     258           7          33   97.15    0.69    2.22    0.00    0.00    0.00    0.00  100.00  187,176 kB/51.0%
13:17:25   999Mhz   500Mhz     0Mhz  56.00C (56.00C)     262          82         276   96.92    0.54    3.18    0.00    0.00    0.00    0.00  100.00  177,240 kB/53.6%
13:17:40  1000Mhz   500Mhz     0Mhz  56.00C (56.00C)     644           6          31   90.36    0.58    5.78    0.00    1.75    0.00    0.58  100.00  170,212 kB/55.4%
13:17:54  1000Mhz   500Mhz     0Mhz  57.00C (57.00C)     328           7          35   94.28    0.71    4.73    0.00    0.14    0.00    0.21  100.00  161,332 kB/57.7%
13:18:09  1000Mhz   500Mhz     0Mhz  56.00C (57.00C)     324          30          32   96.80    0.54    4.75    0.00    0.07    0.00    0.14  100.00  145,108 kB/62.0%

The JSON Socket Request started at 13:17:25 and bcmstat.sh did not give any other output after 13:18:09
Reply
How big is your library?
Code:
./texturecache.py stats

It looks like you're using SQLite for your media library (judging by the absence of any network activity), would you mind sending me a zip of your Database folder so that I may try and reproduce?
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Code:
OpenELEC:~ # ./texturecache.py stats
Albums     : 0
Artists    : 0
Songs      : 0
Movies     : 1279
TVShows    : 144
Episodes   : 3924
MusicVideos: 0
Addons     : 60

Database.zip -> PM
Reply
Thanks for the zip.

I still think it's a memory issue.

Running texturecache.py remotely, for just "nc movies", I can see over 80MB of RAM being consumed on the Pi, which is the memory being used by XBMC just to query and then serialise the JSON data (media libary and texture db). Obviously, running texturecache,py on the Pi itself would at least double this RAM requirement.

I'm not sure of your precise configuration, but you obviously have logging enabled, and I'll take a guess that you also have cache.castthumb enabled.

Disabling the logfile (@logfile=) will prevent the creation of one or two large buffers and may leave enough RAM for the script to load everything it needs. "nc movies" worked here without logging, but failed to complete with logging enabled.

Also, caching cast artwork massively increases the number of items to be loaded and processed. Although it's only 1279 movies, you have a total of 26,695 cast thumbnails out of a total of 63,697 cast members, all of which need to be loaded into memory then processed (although all of the cast members without thumbnails will be discarded, they still need to be loaded first). Disabling cast caching (@cache.castthumb=no) will, I'm sure, eliminate memory issues although it does of course mean you will not be pre-loading the cache with cast thumbnails.

I think there will be instances when it's just not going to be practical to run texturecache.py directly on a memory limited device, in which case running texturecache.py remotely is probably the best option, particularly if you want to cache cast members.

I'll give some thought to how the memory overhead could be reduced, but other than executing multiple queries and discarding as much as possible between queries, nothing is leaping out at me. Unfortunately there's no way to query XBMC for only those cast members with thumbnails... now that would be nice. Smile
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
  • 1
  • 45
  • 46
  • 47(current)
  • 48
  • 49
  • 197

Logout Mark Read Team Forum Stats Members Help
[RELEASE] Texture Cache Maintenance utility17