20 Nov 2019 - TVDB Scraper v3.2.0 is now available which reinstates scraping. TVDB are still in the process of fixing a number of bugs so there may still be further breakages. See this post. 2901570 (post)

  •   
  • 1
  • 162
  • 163
  • 164(current)
  • 165
  • 166
  • 180
  •   
[RELEASE] Texture Cache Maintenance utility
(2018-04-23, 21:31)Milhouse Wrote: but not storing non-IMDB IDs in the IMDB field might be a better solution.
I agree but I don't know why there is this ID. My NFO file next to the movie contains <id>tt4044364</id>.
Any idea why few movies have IMDb ID and other TMDb ID?
(2018-04-23, 21:31)Milhouse Wrote:   2. Yes. ./texturecache.py imdb tvshows @query.seasons=no
Great thank you!
(2018-04-23, 21:31)Milhouse Wrote: After dropping mpaa, if the movies continue to be refreshed then run ./texturecache.py qax movies @qa.nfo.refresh=0 @logfile=/tmp/tc.log and upload /tmp/tc.log to a pastebin site (or if too large, zip it and use Googledrive/Dropbox etc.) then post the link.
I have dropped it until I can have this info in my nfo. Now only movies with poster issue are displayed.
It seems that the python script is struggling with the step about Rescanning directories, following the failures. It is stuck with the first one for 15 minutes now.
Please find here the log until it get stuck: https://pastebin.com/iupz9Ver

After a reboot, I have restarted and went through: https://pastebin.com/YqRQyKDb
Reply
(2018-04-23, 22:13)raspdealer Wrote: Any idea why few movies have IMDb ID and other TMDb ID?
No sorry, maybe it's the scraper you are using? If you only want information from your local NFO then you should use the "Local Information Only" scraper. If you use "The Movie Database" scraper then it may also include TMDB information.

(2018-04-23, 22:13)raspdealer Wrote: After a reboot, I have restarted and went through: https://pastebin.com/YqRQyKDb

Now that you've dropped mpaa (and rebooted), and you've run qax through to completion, if you run it again does it refresh the same movies? Based on the second log, it should have refreshed as follows:
Movie    [I Am Love                                         ]: FAIL (missing poster, local is available)
Movie [Molly's Game ]: FAIL (missing poster, local is available)
Movie [Star Wars: The Last Jedi ]: FAIL (missing poster, local is available)
Movie [Strike a Pose ]: FAIL (missing poster, local is available)
Rescanning directory: nfs://192.168.1.121/volume1/Family/Movies/I Am Love (2009)...
Updating library: New movieid 963 [I Am Love]
Rescanning directory: nfs://192.168.1.121/volume1/Family/Movies/Molly's Game (2017)...
Updating library: New movieid 964 [Molly's Game]
Rescanning directory: nfs://192.168.1.121/volume1/Family/Movies/Star Wars Episode VIII - The Last Jedi (2017)...
Updating library: New movieid 965 [Star Wars: The Last Jedi]
Rescanning directory: nfs://192.168.1.121/volume1/Family/Movies/Strike a Pose (2016)...
Updating library: New movieid 966 [Strike a Pose]

So if you run it a third time, the movies above should not now refresh (as they should be associated with their local poster). If any movies do refresh please send a new log.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
hoping for my sake I'm not alone here, but is anyone getting a lot of errors when they run this utility lately?
of 1400 or so files the utility says it is going to cache, at around 630 or so it errors on every file thereafter.
I am using the latest version of the utility and on alpha 2 of leia. every attempt returns this to the log file.


2018-04-23 21:43:16.310119:Thread-1  : preparedl.JSON WEB REQUEST: [POST] [{"method": "Files.PrepareDownload", "params": {"path": "image://https%3a%2f%2fassets.fanart.tv%2ffanart%2fmovies%2f23398%2fmoviebanner%2falvin-and-the-chipmunks-the-squeakquel-53f60665859d9.jpg/"}, "jsonrpc": "2.0", "id": "preparedl"}]
2018-04-23 21:43:21.312405:Thread-1  : ** iotimeout occurred during web request **
2018-04-23 21:43:21.312405:Thread-1  : preparedl.RECEIVED WEB DATA: 408, Request Timeout, 
2018-04-23 21:43:21.312405:Thread-1  : primeImage.DIRECT WEB REQUEST: [GET], HTTPS=True, [assets.fanart.tv] [/fanart/movies/23398/moviebanner/alvin-and-the-chipmunks-the-squeakquel-53f60665859d9.jpg]
2018-04-23 21:43:29.350865:Thread-1  : primeImage.RECEIVED WEB DATA: 200, OK, <raw data>
2018-04-23 21:43:29.351865:Thread-1  : Primed request of: HTTPS=True, Domain [assets.fanart.tv] with URL [/fanart/movies/23398/moviebanner/alvin-and-the-chipmunks-the-squeakquel-53f60665859d9.jpg], result [200, OK]
2018-04-23 21:43:29.351865:Thread-1  : Retrying getDownloadURL(), 3 attempts remaining
2018-04-23 21:43:29.851894:Thread-1  : preparedl.JSON WEB REQUEST: [POST] [{"method": "Files.PrepareDownload", "params": {"path": "image://https%3a%2f%2fassets.fanart.tv%2ffanart%2fmovies%2f23398%2fmoviebanner%2falvin-and-the-chipmunks-the-squeakquel-53f60665859d9.jpg/"}, "jsonrpc": "2.0", "id": "preparedl"}]
2018-04-23 21:43:34.853180:Thread-1  : ** iotimeout occurred during web request **
2018-04-23 21:43:34.853180:Thread-1  : preparedl.RECEIVED WEB DATA: 408, Request Timeout, 
2018-04-23 21:43:34.853180:Thread-1  : Retrying getDownloadURL(), 2 attempts remaining
2018-04-23 21:43:35.353208:Thread-1  : preparedl.JSON WEB REQUEST: [POST] [{"method": "Files.PrepareDownload", "params": {"path": "image://https%3a%2f%2fassets.fanart.tv%2ffanart%2fmovies%2f23398%2fmoviebanner%2falvin-and-the-chipmunks-the-squeakquel-53f60665859d9.jpg/"}, "jsonrpc": "2.0", "id": "preparedl"}]
2018-04-23 21:43:40.353494:Thread-1  : ** iotimeout occurred during web request **
2018-04-23 21:43:40.353494:Thread-1  : preparedl.RECEIVED WEB DATA: 408, Request Timeout, 
2018-04-23 21:43:40.353494:Thread-1  : Retrying getDownloadURL(), 1 attempts remaining
2018-04-23 21:43:40.853523:Thread-1  : preparedl.JSON WEB REQUEST: [POST] [{"method": "Files.PrepareDownload", "params": {"path": "image://https%3a%2f%2fassets.fanart.tv%2ffanart%2fmovies%2f23398%2fmoviebanner%2falvin-and-the-chipmunks-the-squeakquel-53f60665859d9.jpg/"}, "jsonrpc": "2.0", "id": "preparedl"}]
2018-04-23 21:43:45.853809:Thread-1  : ** iotimeout occurred during web request **
2018-04-23 21:43:45.853809:Thread-1  : preparedl.RECEIVED WEB DATA: 408, Request Timeout, 
2018-04-23 21:43:45.853809:Thread-1  : Image not available for download - uncacheable (embedded?), or doesn't exist. Filename [image://https%3a%2f%2fassets.fanart.tv%2ffanart%2fmovies%2f23398%2fmoviebanner%2falvin-and-the-chipmunks-the-squeakquel-53f60665859d9.jpg/]
Reply
I reduced the amount of movies to 9 movies and tried again. of 107 to cache it starts throwing errors again at number 45. so both were just a little over half.
Reply
(2018-04-23, 23:23)Milhouse Wrote: you should use the "Local Information Only" scraper.
FIXED thanks
(2018-04-23, 23:23)Milhouse Wrote: So if you run it a third time, the movies above should not now refresh (as they should be associated with their local poster). If any movies do refresh please send a new log. 
[/quote]
It seems ok now, I will change it back to the default 30 days period.

Few TV Shows are not recognized by the OMDB Api, is it possible to create some aliases in the cfg file?

Can you please explain to me what does "Duplicate/Skipped" means in the report?
Code:

              |    banner   |  clearlogo  |    fanart   |    poster   | season.banner | season.poster |  seasonposter|    thumb    |    TOTAL    
--------------+-------------+-------------+-------------+-------------+---------------+---------------+--------------+-------------+-------------
Cached        |      -      |      -      |      -      |      -      |       -       |       -       |       -      |      -      |      0      
Deleted       |      -      |      -      |      -      |      -      |       -       |       -       |       -      |      -      |      0      
Duplicate     |     1234    |     772     |     1201    |     1276    |      513      |      1149     |      1184    |      20     |     7349    
Error         |      -      |      -      |      -      |      -      |       -       |       -       |       -      |      -      |      0      
Ignored       |      -      |      -      |      -      |      -      |       -       |       -       |       -      |      -      |      0      
Season-all    |      -      |      -      |      -      |      -      |       -       |       -       |       -      |      -      |      0      
Skipped       |      49     |      34     |     238     |     254     |       33      |       33      |       47     |     1061    |     1749    
Undefined     |      -      |      -      |      -      |      -      |       -       |       -       |       -      |      -      |      0      
=================================================================================================================================================
TOTAL         |     1283    |     806     |     1439    |     1530    |      546      |      1182     |      1231    |     1081    |     9098   

Thanks for everything!
Reply
(2018-04-24, 08:23)raspdealer Wrote: Few TV Shows are not recognized by the OMDB Api, is it possible to create some aliases in the cfg file?

There are some settings you can add - these are mine:
imdb.ignore.tvtitles = ^Brass Eye|^Connections|^Dynamo|^Fresh Meat|^Hamish Macbeth|^The Mighty Boosh|^The Killing|^Sanctuary|^Outnumbered|^V \(2009\)$|Treasure Island
imdb.map.tvtitles = ^Battlestar Galactica=tt0076984|^Wallander$=tt0907702|^Wallander \(UK\)$=tt1178618|DC's Legends of Tomorrow=tt4532368|^Lucifer$=tt4052886
imdb.translate.tvtitles = ^Frank Herbert's=|^Marvel's=|Hitchhiker's=Hitch Hikers
imdb.ignore.tvtitles uses a regex and matching tvshows will be ignored (ie. not queried - omdb coverage is sometimes very poor, missing entire tv shows)
imdb.map.tvtitles uses a regex and matching tvshows will use the specified IMDb ID instead of whatever ID is in the library
imdb.translate.tvtitles uses a regex and the matching tvshow title text will be replaced by the substitution text (or removed if the substitution text is blank)

For example, on omdb Frank Herbert's Children of Dune is known as Children of Dune etc.

(2018-04-24, 08:23)raspdealer Wrote: Can you please explain to me what does "Duplicate/Skipped" means in the report?

Skipped = artwork is already cached

Duplicate = for tvshows, the same artwork appears at the episode level and season level, so the repeated artwork is classed as duplicate. If two movies have the same fanart then one of the instances will be seen as a duplicate (there would be no reason to cache both). Songs in an album will often share the same (ie. duplicate) fanart/cover artwork. Etc.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2018-04-24, 06:30)cabrio Wrote: I reduced the amount of movies to 9 movies and tried again. of 107 to cache it starts throwing errors again at number 45. so both were just a little over half.

Remote artwork sucks. How many times do I have to repeat this... Smile

Kodi is performing the download and not the script, so it's a problem between Kodi and the remote website.

It may be the remote website is imposing some sort of rate limiting. Or the artwork no longer exists. Or the remote website is having a bad day.

If you look at the log you'll see the encoded urls that are being processed - an example from your log is:
2018-04-23 21:43:40.853523:Thread-1  : preparedl.JSON WEB REQUEST: [POST] [{"method": "Files.PrepareDownload", "params": {"path": "image://https%3a%2f%2fassets.fanart.tv%2ffanart%2fmovies%2f23398%2fmoviebanner%2falvin-and-the-chipmunks-the-squeakquel-53f60665859d9.jpg/"}, "jsonrpc": "2.0", "id": "preparedl"}]

The decoded url is:
https://assets.fanart.tv/fanart/movies/23398/moviebanner/alvin-and-the-chipmunks-the-squeakquel-53f60665859d9.jpg

Loading this url in a browser results in the error:
{ "status": "error", "error message": "Not found" }

(I also had a "500 Internal Server Error")

So either the artwork no longer exists (gone forever) or fanart.tv is having a bad day. If the latter, which seems likely, maybe the artwork will be accessible later, or tomorrow. If the former you'll need to fix your media library and associate new artwork that does exist.

If you want to avoid this sort of problem in future you should use only local artwork. I've written the mklocal.py script which you can use to convert your remote artwork into local artwork, and it will remove from your library any remote artwork that no longer exists (allowing you to use something like the Artwork Beef add-on to re-associate new artwork) or you can manually download the artwork you prefer and re-run mklocal.py and it will associate your movies/tvshows with that artwork.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
@cabrio: There's another possible explanation:

2018-04-23 21:43:16.310119:Thread-1  : preparedl.JSON WEB REQUEST: [POST] [{"method": "Files.PrepareDownload", "params": {"path": "image://https%3a%2f%2fassets.fanart.tv%2ffanart%2fmovies%2f23398%2fmoviebanner%2falvin-and-the-chipmunks-the-squeakquel-53f60665859d9.jpg/"}, "jsonrpc": "2.0", "id": "preparedl"}]
2018-04-23 21:43:21.312405:Thread-1 : ** iotimeout occurred during web request **
2018-04-23 21:43:21.312405:Thread-1 : preparedl.RECEIVED WEB DATA: 408, Request Timeout,

The iotimeout error (which I haven't seen for a long time) might suggest a problem/bug with your Kodi client, so check your Kodi log for errors.

Edit: It's likely the misbehaving fanart.tv website is causing Kodi to take longer than the 5 second communication timeout when the script requests information from Kodi, resulting in the iotimeout error (unfortunately this particular timeout is not configurable). You can try adding an increased timeout with the following patch:
diff --git a/texturecache.py b/texturecache.py
index 820526e..983e3de 100755
--- a/texturecache.py
+++ b/texturecache.py
@@ -2759,7 +2759,7 @@ class MyJSONComms(object):
REQUEST = {"method":"Files.PrepareDownload",
"params":{"path": filename}}

- data = self.sendJSON(REQUEST, "preparedl")
+ data = self.sendJSON(REQUEST, "preparedl", timeout=15.0)

if "result" in data:
return "/%s" % data["result"]["details"]["path"]
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
strange... putting that url to my browser fetches the image 
Image
it takes a few seconds but it does come, as does every other error that is generated.
Reply
Yes it took 4-5 attempts before it appeared once, then I refreshed again and got the errors I mentioned once more. So it's clear the failures were genuine, because the remote website is/was having a bad day. Avoiding remote artwork is the best solution.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
is there a special box i have to tick to make sure it is all downloaded to the local source?
Reply
(2018-04-24, 21:26)cabrio Wrote: is there a special box i have to tick to make sure it is all downloaded to the local source?

In Artwork Beef? Not sure, don't use it - you will need to ask in the AB support thread.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
@Milhouse 

Is this result below normal....
Quote:              |  animatedfanart|  animatedposter|    banner   |   clearart  |  clearlogo  |     db4     |   discart   |    fanart   |   fanart1   |   fanart2   |   fanart3   |   fanart4   |  landscape  |    poster   |     set     |  setfanart  |    thumb    |  thumbnail  |    TOTAL
--------------+----------------+----------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------
Cached        |        -       |        -       |     157     |     148     |     425     |      -      |     376     |      2      |     548     |     700     |     565     |     541     |     1005    |      -      |      -      |      -      |      -      |      1      |     4468
Deleted       |        -       |        -       |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      0
Duplicate     |        -       |        -       |      44     |      44     |      52     |      63     |      47     |      52     |      53     |      54     |      52     |      49     |      48     |      53     |     229     |     225     |      -      |      -      |     1065
Error         |        -       |        -       |      20     |      25     |      67     |      -      |      48     |      -      |      93     |     141     |     237     |     149     |     530     |      -      |      -      |      -      |      -      |      -      |     1310
Ignored       |        -       |        -       |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      4      |      -      |      4
Season-all    |        -       |        -       |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      0
Skipped       |        34      |        65      |     1400    |     1212    |     1240    |      1      |     997     |     1992    |     1316    |     1017    |     982     |     1021    |      2      |     1988    |     500     |     476     |      -      |      99     |    14342
Undefined     |        -       |        -       |      -      |      -      |      -      |      -      |      -      |     117     |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      -      |      51     |     168
==============================================================================================================================================================================================================================================================================================
TOTAL         |        34      |        65      |     1621    |     1429    |     1784    |      64     |     1468    |     2163    |     2010    |     1912    |     1836    |     1760    |     1585    |     2041    |     729     |     701     |      4      |     151     |    21357
Download Time |        -       |        -       | 00:43:18.87 | 00:29:53.86 | 00:53:37.35 |      -      | 00:02:20.92 | 00:00:11.99 | 02:43:41.11 | 03:07:28.50 | 03:24:10.99 | 03:34:37.36 | 00:41:54.94 |      -      |      -      |      -      |      -      | 00:00:00.01 | 1d 07:15:38.96
MY CURRENT MEDIA PLAYER | MY HOME THEATER
MINIX NEO U9-H LIBREELEC v18 Leia | EGREAT A5 | NVIDIA SHIELD
SAMSUNG QLED QN65Q7 | Sony 43 Android TV HDR
XBOX ONE X UHD BLU-RAY | PS4 PRO 4K w/PSVR
ONKYO SBT-A500 DTS X | ATMOS 3.2.1
Reply
It could be normal - what do you think is not normal?

It seems to have taken a very long time (1d 7h), which is far longer than the sum of the individual artwork activities so something might not be right there, but other than that... nothing is obviously wrong.

If you're concerned about the "Undefined" counts, that just means you have artwork entries in your media library such as:
      "fanart": "",
which shouldn't really exist (if the value is blank there shouldn't be an artwork entry in the database) but should also be harmless.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
@Milhouse 

If you don't any error then that is fine, actually it took more 3 days to complete , i don't know it took this long.

Thanks for all your help!
MY CURRENT MEDIA PLAYER | MY HOME THEATER
MINIX NEO U9-H LIBREELEC v18 Leia | EGREAT A5 | NVIDIA SHIELD
SAMSUNG QLED QN65Q7 | Sony 43 Android TV HDR
XBOX ONE X UHD BLU-RAY | PS4 PRO 4K w/PSVR
ONKYO SBT-A500 DTS X | ATMOS 3.2.1
Reply
  •   
  • 1
  • 162
  • 163
  • 164(current)
  • 165
  • 166
  • 180
  •   
 
Thread Rating:
  • 17 Vote(s) - 4.76 Average



Logout Mark Read Team Forum Stats Members Help
[RELEASE] Texture Cache Maintenance utility4.7617