•   
  • 1
  • 74
  • 75
  • 76(current)
  • 77
  • 78
  • 179
  •   
[RELEASE] Texture Cache Maintenance utility
Here is a problem I have. I tried using texturecache.py on it but maybe I'm not using the right options.

This is for music library. When new files are scanned, if a new album is found the first album artist will be assigned and the scanner will look for (local art) folder.jpg and fanart.jpg in the album's parent folder. If it exists it will be added to the album artist and in the database "art" table. The problem is if there is more than one album artist or compilation only (song) artists. There is no mechanism to assign local art for these during a scan. IF the artist is scraped, then it is possible to use the "artist info" screen to browse for local thumb and fanart. If the artist didn't scrape, "artist info" isn't accessible, without setting "lastScraped" in the artist table, either by "import library" or directly updating the artist table. Even when the "artist info"screen is available, it is very tedious to use the UI to browse and set local art.

What I am looking for is something that can go through the "artist" and "art" table and find all artists where thumb and or fanart are not set and provide some means of updating the art table. It is sort of possible to do this with the music library import, but the only way I get it to work is to use the
Code:
<thumb>...</thumb>
<fanart><thumb>...</thumb></fanart>
elements, but it seems like if you do this yes you can get local art into the art table but it locks out any url/online art that might have scraped.

Any ideas?
Reply
(2014-10-13, 06:29)scott967 Wrote: Here is a problem I have. I tried using texturecache.py on it but maybe I'm not using the right options.

This is for music library. When new files are scanned, if a new album is found the first album artist will be assigned and the scanner will look for (local art) folder.jpg and fanart.jpg in the album's parent folder. If it exists it will be added to the album artist and in the database "art" table. The problem is if there is more than one album artist or compilation only (song) artists. There is no mechanism to assign local art for these during a scan. IF the artist is scraped, then it is possible to use the "artist info" screen to browse for local thumb and fanart. If the artist didn't scrape, "artist info" isn't accessible, without setting "lastScraped" in the artist table, either by "import library" or directly updating the artist table. Even when the "artist info"screen is available, it is very tedious to use the UI to browse and set local art.

What I am looking for is something that can go through the "artist" and "art" table and find all artists where thumb and or fanart are not set and provide some means of updating the art table. It is sort of possible to do this with the music library import, but the only way I get it to work is to use the
Code:
<thumb>...</thumb>
<fanart><thumb>...</thumb></fanart>
elements, but it seems like if you do this yes you can get local art into the art table but it locks out any url/online art that might have scraped.

Any ideas?

I don't really know much about updating the media library database directly, particularly the music database - you might have more luck asking in the Universal Music Scraper thread. I've only concerned myself with JSON queries when extracting data from the media library (or making changes), but unfortunately there are no JSON methods that would allow you to make artwork changes in the music database.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Milhouse, could you help with this interpretation after running the script with the c option.

Is it correct on my large library I have gained only 28 items that needed cache-ing ?

Image

Cheers

Reply
Yes, it would look that way. "Skipped" would be items already in your cache, and "Duplicates" would be artwork used more than once (which only needs to be skipped/cached once). You've got some "Undefined" artwork that you might want fix - this would be library entries such as fanart="" (a qa check might reveal them, remember to set @qaperiod=9999).
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Thanks, as expected a bit of missing mpaa's and obscure docos with no fanart, and lots of vgenres [tvshow, ***** and "vgenres [movie, *****]
This looks skin related and is confirmed by no genres subfolders in the extras folder contained inside the individual skin add-on directory.

What Milhouse's very useful script does confirm, is that using a good external media manager. One that parses filenames to store media in XBMC preferred individual folders and scrapes and stores full metadata within. This makes managing a large media library very easy. My RPi library has over 5000 media files and only 28 textures were not cached.

Happy as.

I would go so far as to suggest this should be mainlined into XBMC, with a some of simple GUI options. Wish I was a skilled coder.

Wink

Reply
For any of that to work, first the artwork and meta has to exist in the databases this gets scraped from, obscure stuff is hardly excluded from this.
Reply
Version 1.7.7
  • Add: Support for @clean.showdialogs and @Scan.showdialogs, default no. With JSON API 6.19.0, JSON initiated library scans and cleans will not show any GUI progress or dialogs. Enable GUI progress/dialogs by setting these new parameters to yes.
  • Add: play, stop and pause options to start, stop and pause media playback. pause will toggle playback. Examples:
    Code:
    ./texturecache.py play http://clips.vorwaerts-gmbh.de/big_buck_bunny.mp4
    ./texturecache.py pause
    ./texturecache.py stop
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Thanks Milhouse!

This Clean feature was a must have for me as cleaning my kodi library was a PITA with that dialogs Smile
Image Image
Reply
Version 1.7.8
  • Add: playw option. Same as play, but wait until end of playback.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
After restoring an OpenElec backup I lost most of my fanart and posters. When trying this script I get errors exactly for those movies that lost it's images.

Code:
OpenELEC:~/downloads # ./texturecache.py C movies "2 Guns" @allow.recacheall=yes @download.threads=1
                                                                                  
Caching artwork: 0 items remaining of 2 (qs: 0, qm: 0), 2 errors, 0 threads active (00.57 downloads per second, ETA: 00:00:00)

The following items could not be downloaded:

[fanart    ] [2 Guns                                  ] http://image.tmdb.org/t/p/original/wZsX7160l5feO646shRx3liMutx.jpg
[poster    ] [2 Guns                                  ] http://image.tmdb.org/t/p/original/30lM3Uvzs6HOG5l4hzhwxYTWgd3.jpg

Cache pre-load activity summary for "movies", filtered by "2 Guns":

              |    fanart   |    poster   |    TOTAL    
--------------+-------------+-------------+-------------
Cached        |      -      |      -      |      0      
Deleted       |      -      |      -      |      0      
Duplicate     |      -      |      -      |      0      
Error         |      1      |      1      |      2      
Ignored       |      -      |      -      |      0      
Skipped       |      -      |      -      |      0      
Undefined     |      -      |      -      |      0      
========================================================
TOTAL         |      1      |      1      |      2      
Download Time | 00:00:01.74 | 00:00:01.74 | 00:00:03.48

  Threads Used: 1
   Min/Avg/Max: 0.57 / 0.57 / 0.57

       Loading: 00:00:00.24
       Parsing: 00:00:00.01
     Comparing: 00:00:03.60
   Downloading: 00:00:04.00
TOTAL RUNTIME: 00:00:07.90
OpenELEC:~/downloads #

There have been more post with similar issues, but those displayed a network or relative url instead of a plain accessible internet url.
Manually downloading one of the images with the following on the same system works fine so it's not the internet connection that's causing issues:
Code:
OpenELEC:~/downloads # curl http://image.tmdb.org/t/p/original/wZsX7160l5feO646shRx3liMutx.jpg -o wZsX7160l5feO646shRx3liMutx.jpg
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1043k  100 1043k    0     0  3402k      0 --:--:-- --:--:-- --:--:-- 3442k
OpenELEC:~/downloads #

Enabling debugging, including component specific for curl, gives me nothing related in the xbmc.log. The following is all that's logged during the time the script is running:
Code:
18:54:20 317.615112 T:2755408976   DEBUG: Thread JobWorker 2755408976 terminating (autodelete)
18:54:35 332.493042 T:2773480528   DEBUG: webserver: request received for /jsonrpc
18:54:35 332.532013 T:2730243152   DEBUG: JSONRPC Server: New connection detected
18:54:35 332.532440 T:2730243152    INFO: JSONRPC Server: New connection added
18:54:35 332.556763 T:2730243152    INFO: JSONRPC Server: Disconnection detected
18:54:36 333.596283 T:2730243152   DEBUG: JSONRPC Server: New connection detected
18:54:36 333.596741 T:2730243152    INFO: JSONRPC Server: New connection added
18:54:36 333.710236 T:2730243152   DEBUG: RunQuery took 8 ms for 1 items query: select * from movieview  WHERE ((movieview.c00 LIKE '%2 Gun$
18:54:36 333.816772 T:2730243152   DEBUG: JSONRPC Server: New connection detected
18:54:36 333.817169 T:2730243152    INFO: JSONRPC Server: New connection added
18:54:39 337.169250 T:2730243152    INFO: JSONRPC Server: Disconnection detected
18:54:39 337.227020 T:2830906448   DEBUG: webserver: request received for /jsonrpc
18:54:43 341.207306 T:2730243152   DEBUG: Previous line repeats 7 times.
18:54:43 341.207581 T:2730243152    INFO: JSONRPC Server: Disconnection detected

Manually refreshing the movie also doesn't restore the images.
What I do get in the log when manually refreshing is:
Code:
19:15:21 1578.453003 T:2830906448  NOTICE: Thread JobWorker start, auto delete: true
19:15:21 1578.464722 T:2830906448   ERROR: COMXImageFile::ReadFile special://masterprofile/Thumbnails/6/6c1ec998.jpg not found
19:15:21 1578.465088 T:2830906448  NOTICE: LoadJpeg: unable to load special://masterprofile/Thumbnails/6/6c1ec998.jpg
19:15:29 1586.925171 T:2679911504  NOTICE: Thread VideoInfoDownloader start, auto delete: false
19:15:44 1602.097046 T:2679911504  NOTICE: Previous line repeats 1 times.
19:15:44 1602.097412 T:2679911504  NOTICE: Thread JobWorker start, auto delete: true
19:15:47 1604.883545 T:2679911504   ERROR: COMXImageFile::ReadFile special://masterprofile/Thumbnails/6/685e3b42.jpg not found
19:15:47 1604.883911 T:2679911504  NOTICE: LoadJpeg: unable to load special://masterprofile/Thumbnails/6/685e3b42.jpg
19:15:47 1604.886963 T:2830906448   ERROR: COMXImageFile::ReadFile special://masterprofile/Thumbnails/6/6c1ec998.jpg not found
19:15:47 1604.887329 T:2830906448  NOTICE: LoadJpeg: unable to load special://masterprofile/Thumbnails/6/6c1ec998.jpg
19:15:47 1605.216797 T:2830906448   ERROR: COMXImageFile::ReadFile special://masterprofile/Thumbnails/6/6c1ec998.jpg not found
19:15:47 1605.217163 T:2830906448  NOTICE: LoadJpeg: unable to load special://masterprofile/Thumbnails/6/6c1ec998.jpg

What could be the cause of the issue and how can I fix it?
Reply
Did you try enabling CURL component debugging? The problem is with the Files.PrepareDownload call or the download of the artwork by XBMC itself. If you enable texturecache.py logging with @logfile= you should see which step is failing, though why it would be failing is harder to answer - sometimes it's just one of those things that XBMC (annoyingly) does.

(2014-10-17, 19:18)Forage Wrote: What I do get in the log when manually refreshing is:
Code:
19:15:21 1578.453003 T:2830906448  NOTICE: Thread JobWorker start, auto delete: true
19:15:21 1578.464722 T:2830906448   ERROR: COMXImageFile::ReadFile special://masterprofile/Thumbnails/6/6c1ec998.jpg not found
19:15:21 1578.465088 T:2830906448  NOTICE: LoadJpeg: unable to load special://masterprofile/Thumbnails/6/6c1ec998.jpg
19:15:29 1586.925171 T:2679911504  NOTICE: Thread VideoInfoDownloader start, auto delete: false
19:15:44 1602.097046 T:2679911504  NOTICE: Previous line repeats 1 times.
19:15:44 1602.097412 T:2679911504  NOTICE: Thread JobWorker start, auto delete: true
19:15:47 1604.883545 T:2679911504   ERROR: COMXImageFile::ReadFile special://masterprofile/Thumbnails/6/685e3b42.jpg not found
19:15:47 1604.883911 T:2679911504  NOTICE: LoadJpeg: unable to load special://masterprofile/Thumbnails/6/685e3b42.jpg
19:15:47 1604.886963 T:2830906448   ERROR: COMXImageFile::ReadFile special://masterprofile/Thumbnails/6/6c1ec998.jpg not found
19:15:47 1604.887329 T:2830906448  NOTICE: LoadJpeg: unable to load special://masterprofile/Thumbnails/6/6c1ec998.jpg
19:15:47 1605.216797 T:2830906448   ERROR: COMXImageFile::ReadFile special://masterprofile/Thumbnails/6/6c1ec998.jpg not found
19:15:47 1605.217163 T:2830906448  NOTICE: LoadJpeg: unable to load special://masterprofile/Thumbnails/6/6c1ec998.jpg

I would advise running "./texturecache.py Xd" as it looks like your Textures13.db is out of sync with the contents of the Thumbnails folder, with the db rows referencing thumbnails that don't exist.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2014-10-17, 19:28)Milhouse Wrote: I would advise running "./texturecache.py Xd" as it looks like your Textures13.db is out of sync with the contents of the Thumbnails folder, with the db rows referencing thumbnails that don't exist
Worked like a charm! Muchas gracias!
I would have expected this to be unneeded when you are already forcing a re-cache/download with the C option.
Reply
(2014-10-17, 19:42)Forage Wrote: Worked like a charm! Muchas gracias!
I would have expected this to be unneeded when you are already forcing a re-cache/download with the C option.

Indeed, however this is what happens during a force cache "C" option:

Code:
1. Acquire the url of the artwork to be cached by calling Files.PrepareDownload
2. If the url is obtained successfully in step #1, remove the existing artwork from cache (deleting the row and corresponding thumbnail file)
3. Use the url to download the new artwork, creating a new entry in the cache (new row and new thumbnail)

In normal (ie. working circumstances) this sequence acts as a safety net - if the artwork you are attempting to re-cache is no longer available to be cached then texturecache.py won't delete the existing thumbnail you already have in your cache, so at least you'll be left with "something" at the end of the process rather than nothing at all. Users might become rather hacked off if their cache is wiped out because of a temporary networking issue.

However in your case, the db is confused as it thinks it has a thumbnail when it doesn't and this is what causes step #1 to fail so steps #2 and #3 are never attempted. As the db is the main driver when determining if the artwork is "in the cache" or not, once your db is out of sync with respect to the thumbnails folder, "bad things" will start to happen.

It would be possible to add some sort of sanity check to determine if the reason for the failure in step #1 is because the thumbnail file doesn't exist (and if not to then remove the db row and repeat step #1), but this will slow things down and should not in normal circumstances be required. Also, there are several legitimate reasons why step #1 should fail, and querying the Thumbnails folder could become quite an expensive process.

I'll think about it though... I don't want to make it any messier than it already is. Smile

In the meantime, running "Xd" from time to time is probably the best idea, or just when "bad things" start to happen.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Update: I can add a fix, but it's only going to work if texturecache.py has direct file system access to the Thumbnails folder as the JSON Textures API "lies" about thumbnail file existence - it responds with details from Textures13.db even though the underlying cached file is missing, so the fix is not going to be ideal. Those that are maintaining remote systems won't see any change, but those running the script on a local system (or that have mounted the remote Thumbnails folder locally) will have the benefit of automatic "fixing" of any missing thumbnails.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Version 1.7.9
  • Chg: When direct file system access to the Thumbnails folder is available, sanity check Files.PrepareDownload failures to determine if the thumbnail file is missing and if so delete the database row before calling Files.PrepareDownload again. See http://forum.xbmc.org/showthread.php?tid...pid1815613
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
  •   
  • 1
  • 74
  • 75
  • 76(current)
  • 77
  • 78
  • 179
  •   
 
Thread Rating:
  • 17 Vote(s) - 4.76 Average



Logout Mark Read Team Forum Stats Members Help
[RELEASE] Texture Cache Maintenance utility4.7617