[ilf_]

@Milhouse: Thanks for this tool, very useful!

I am trying to remove all titles from my library that don't have files (any more). Someone told me to check out texturecache.py, and it's indeed very cool and very powerful.

However, I still haven't figured out how to achieve my task.

I can find a lot of duplicates, like this example:

python:


texturecache.py duplicates

<snip>

IMDb Number: tt0028420

        Title: Circus

     Movie ID: 2477

    Playcount: 0

  Last Played:

   Date Added: 2019-08-10 20:08:26

         File: /mnt/unwatched/Tsirk AKA Circus (1936) RUS/Tsirk (1936) Grigori Aleksandrov, Isidor Simkov.mkv



        Title: Circus

     Movie ID: 3262

    Playcount: 0

  Last Played:

   Date Added: 2021-02-15 14:43:42

         File: /mnt/unwatched/Tsirk AKA Circus (1936) RUS/Tsirk.AKA.Circus.1936.720p.BluRay.AC3.x264-trand.mkv

</snip>

The example directory only contains one file:

python:


ls -1 /mnt/unwatched/Tsirk\ AKA\ Circus\ \(1936\)\ RUS

Tsirk.AKA.Circus.1936.720p.BluRay.AC3.x264-trand.mkv

The options/parameter proposed in the other thread are not limited to "no media file exists (any more)", because it finds both:

python:


texturecache.py qa movies @qafile=yes @qaperiod=9999 @qa.art.movies=

<snip>

Movie    [Circus                                            ]: FAIL (missing mpaa)

Movie    [Circus                                            ]: FAIL (missing mpaa)

</snip>

This is the library query output for this movie title:

json:


texturecache.py jd movies "Circus"

[

  {

    "art": {

      "fanart": "image:///home/osmc/kodi_videodb_2020-10-24/movies/Circus_1936-fanart.jpg/",

      "poster": "image:///home/osmc/kodi_videodb_2020-10-24/movies/Circus_1936-poster.jpg/"

    },

    "file": "/mnt/unwatched/Tsirk AKA Circus (1936) RUS/Tsirk (1936) Grigori Aleksandrov, Isidor Simkov.mkv",

    "label": "Circus",

    "movieid": 2477,

    "title": "Circus"

  },

  {

    "art": {

      "fanart": "image://https://image.tmdb.org/t/p/original/zHP6spV7YnF2Kp2Q2II6n6fMbEa.jpg/",

      "poster": "image://https://image.tmdb.org/t/p/original/s9XtFmpePkL6zWJ7wvI3fq82Phf.jpg/"

    },

    "file": "/mnt/unwatched/Tsirk AKA Circus (1936) RUS/Tsirk.AKA.Circus.1936.720p.BluRay.AC3.x264-trand.mkv",

    "label": "Circus",

    "movieid": 3262,

    "title": "Circus"

  }

]

My question is: How can I

1. find/list all movies without files (like #2477)
   
2. remove these movies from the library
   

I'm sure texturecache.py can do this. I just can't figure out how. :)

Thanks, and keep up the good work!

[Rouzax]

Have recently switched to TMDB for all my episode scrapes and since then I have a show (Oogappels) that will lose the first season and than the first season will be recognised as Angry Birds Toons.
This it happens after this until has run.

Should this tool also still work with TMDB as data provider?

[Klojum]

Should this tool also still work with TMDB as data provider?

Usually no scraper will switch or lose an assigned TV shows per season.
I'd try to refresh/rescrape this TV show manually via Kodi. You should get a listing of matching TV shows, and normally 'Oogappels' should be there.

[ilf_]

I am trying to remove all titles from my library that don't have files (any more). Someone told me to check out texturecache.py, and it's indeed very cool and very powerful.

However, I still haven't figured out how to achieve my task.

I ran "texturecache.py vclean" and it seems to have done what I want. This is cool! But weird.

According to Milhouse himself, "'vclean' just executes the same video library cleanup process that you would run from the Kodi GUI".
But according to Kodi wiki, "when you run this option your media may not be deleted as expected".

And indeed, when I run "clean library" from the GUI, titles that have no file are kept in the library.
But when I run "texturecache.py vclean", titles that have no file are removed from the library.

Is this correct?

What exactly does "texturecache.py vclean" do? The documentation only sais "Clean video library" without explaining any further details.
I have looked into the 8672 lines of code in texturecache.py, but unfortunately my Python is not good enough to figure this out myself from source.

Thanks for your work and help!

[Rouzax]

Should this tool also still work with TMDB as data provider?

Usually no scraper will switch or lose an assigned TV shows per season.
I'd try to refresh/rescrape this TV show manually via Kodi. You should get a listing of matching TV shows, and normally 'Oogappels' should be there.

I know and I've removed it from the library over and over again and did a library update which fixed everything, but after each run of this script it will find Angry Bird Toons from the first season of Oogappels

[monkeyman]

As I said somewhere below that, I found a way to add a second sleep time between requests to your script... but fanart.tv still banned me every few requests. I had to cancel the script and restart it, so I could download artwork again.

You're probably right. It has no much sense to implement site by site rules inside the script. May be you could implement some config options so users can "adapt" script behaviour to each site limitation. "cadence" and "reconnect" options maybe? I don't know. You're the master here.

Thanks a lot for your support.

I haven't had time to look at this yet, and assuming the fanart.tv restriction is only temporary I may never get around to it (what they have done is going to screw up Kodi as well, so it's not really a practical solution for the majority of their users such as Kodi).

It's interesting that adding time.sleep(1.0) to force 1 request per second wasn't successful, although I have an idea why that could be. Can you re-try your modified script with the option @Download.prime=no as that will stop the script checking if the artwork exists before it asks kodi to perform the download ("priming" the download seemed to help improve reliability of imdb.com artwork downloads, although it could be a placebo...). Also, the script will ask kodi to download artwork up to 3 times by default, so this could also be blowing the fanart.tv rate limit (it depends where you added the delay, I suppose - between artwork items, or between web requests etc.). You could try @Download.retry=0 to avoid retrying failed downloads, and (in combination with the previous option) see if that helps avoid restarting the script (I've no idea why restarting the script helps, other than to introduce a sufficiently long enough delay to avoid immediately hitting the fanart.tv rate limits again).

The problem with implementing per-site rate limiting is really to do with the multi-threading as that makes it all a bit more complicated - per-site limits could be implemented, but it may mean enforcing a single thread just to simplify things. And when sites do implement rate limits they don't usually make public what those limits actually are - fanart.tv is quite unusual in providing the limits they have implemented. I'm fairly sure imdb.com implement some sort of rate limiting, but the exact details are unknown (at least to me). So configuring per-site limits would just be a guessing game, and crossing fingers, and I'm really not sure that there is a good solution for this issue other than the sledgehammer "1 request per second using a single thread".

Edit: Those options are @ download.prime / @ download.retry (lower case d - use the @ on the command line, leave it out when adding to texturecache.cfg). Unfortunately there's a @Download user (capital D) and the forum software in it's infinite wisdom decides to rewrite the message. Bah.

I dumped and re-scraped my full library this weekend and ran into issues with fanart.tv blocking requests from texturecache.py (503 errors after the first 20-30 artwork download requests).  I was able to work around this by setting download.prime=no and download.retry=0 in texturecache.cfg, and then added time.sleep(0.5) to line 1110 of texturecache.py v2.5.4.  This slowed down the requests from texturecache.py enough to avoid getting blocked. 

Would it be possible to make any modifications to the official version of texturecache.py to avoid this workaround?  It looks like the script already limits requests to fanart.tv to a single thread, so could requests to that domain be rate limited as well?  It's not ideal to have to slow down these requests so much, but the alternative is to run the script, kill it once requests start getting blocked, then restart the script. This can be a tedious process if thousands of items need to be cached.

[Klojum]

Would it be possible to make any modifications to the official version of texturecache.py to avoid this workaround?

I see you quoted a post from Milhouse. In case you didn't know yet, sadly, Milhouse chose to lay down his Kodi/LibreELEC tasks some time ago, and move on to other things in life.

Also, in the meantime a Python-3 version of the TextureCache tool has been created and added to the Video Database Cleaner add-on , you should find it in the github page mentioned at the top of the opening post. It seems like a logical step to go from there and do your additions and/or changes.

[DGenerateKane]

Would it be possible to make any modifications to the official version of texturecache.py to avoid this workaround?

I see you quoted a post from Milhouse. In case you didn't know yet, sadly, Milhouse chose to lay down his Kodi/LibreELEC tasks some time ago, and move on to other things in life.

Also, in the meantime a Python-3 version of the TextureCache tool has been created and added to the Video Database Cleaner add-on , you should find it in the github page mentioned at the top of the opening post. It seems like a logical step to go from there and do your additions and/or changes.

How would anyone know? He's still listed as Team Kodi member.

[Klojum]

How would anyone know? He's still listed as Team Kodi member.

Plenty of people already got the memo on his departure, but his forum details were indeed still unchanged. I've amended them now.

[danmedhurst]

So i've been using texturecache for years and it's never failed me, but on a new matrix install i've getting a few thousand files it's not able to grab for movies mainly..

is this tool no longer the go to? Should i be using something else to cache the artwork?

I've seen artwork beef but i'm really no able to ascertain if it does the same thing..

[queeup]

I am using this command to remove movies under imdb rating 5. Is there any other way to do that differently? I mean more elegant way?

bash:
texturecache.py jd movies @filter=rating @filter.operator=lessthan "5" | jq ".[].movieid" | for i in `cat /dev/stdin`; do texturecache.py remove movie $i; done

Also is there a way to remove from library and also removing video files from harddisk?

[xbmandrea]

Hi Milhouse, thanks for your tool!
I have a simple question, but I can't figure it out on my own.
I need help.
With this command
 

Code:

texturecache.py jd movies "2012"

this is the result:
 

Code:

[
  {
    "art": {
      "fanart": "image://http://image.tmdb.org/t/p/original/ywxrdkfbr8Dg3SBW2gi4kC59qOb.jpg/",
      "poster": "image://http://image.tmdb.org/t/p/original/iYQd1u03APB67e8NYaQiUetxaIu.jpg/"
    },
    "file": "smb://192.168.178.21/Film/2FilmHD/2012 (2009) [1080p] HEVC.mkv",
    "label": "2012",
    "movieid": 797,
    "title": "2012"
  }
]

The problem is that url http://image.tmdb.org/t/p/original/iYQd1...txaIu.jpg/ of the poster image does not exist.
So I execute the command:
 

Code:

texturecache.py c movies "2012"

But this is the error
 

Code:

Need to cache: [  fanart  ] for movie: 2012
Need to cache: [  poster  ] for movie: 2012

Caching artwork: 2 items remaining of 2 (qs: 0, qm: 0), 0 errors, 2 threads active (00.00 downloads per second, ETA: **:Caching artwork: 0 items remaining of 2 (qs: 0, qm: 0), 2 errors, 0 threads active (02.00 downloads per second, ETA: 00:00:00)

The following items could not be downloaded:

[poster    ] [2012                                    ] http://image.tmdb.org/t/p/original/iYQd1u03APB67e8NYaQiUetxaIu.jpg
[fanart    ] [2012                                    ] http://image.tmdb.org/t/p/original/ywxrdkfbr8Dg3SBW2gi4kC59qOb.jpg

Cache pre-load activity summary for "movies", filtered by "2012":

              |    fanart   |    poster   |    TOTAL
--------------+-------------+-------------+-------------
Cached        |      -      |      -      |      0
Deleted       |      -      |      -      |      0
Duplicate     |      -      |      -      |      0
Error         |      1      |      1      |      2
Ignored       |      -      |      -      |      0
Skipped       |      -      |      -      |      0
Undefined     |      -      |      -      |      0
========================================================
TOTAL         |      1      |      1      |      2
Download Time | 00:00:00.18 | 00:00:00.12 | 00:00:00.18

Any help is appreciated.
Thanks

[Klojum]

Hi Milhouse, thanks for your tool!

In case you missed the memo, Milhouse has left the Kodi building some time ago.
But perhaps others can help you out with your question(s).

[littlejeem]

I am using this command to remove movies under imdb rating 5. Is there any other way to do that differently? I mean more elegant way?

bash:
texturecache.py jd movies @filter=rating @filter.operator=lessthan "5" | jq ".[].movieid" | for i in `cat /dev/stdin`; do texturecache.py remove movie $i; done

Also is there a way to remove from library and also removing video files from harddisk?

Hello @queeup

I can't help improve the elegance of the one-liner you've written but perhaps this might help with the functionality you wanted.

bash:
texturecache.py jd movies @filter=rating @filter.operator=lessthan "5" | jq -r ".[].file" | while read i; do find $i -type f -print; done < /dev/stdin

All I've done here is focus on the removal of the file , so we stick with you texturecache use but jq -r asks for raw output (no "" marks). Moving from a 'for' loop to a 'while/where' gets us off the hook for problems with newlines, finally the find command helps us match an existing file to the value of $i so we don't match random stuff and prints the output.

If you are happy with the results you are getting back then change the -print to -delete.

A normal library clean from within Kodi, via texturecache, or via json will remove the, now non existent, movies from the library.

...will leave some empty parent folders though.

[lpt2007]

I try to add missing poster with command:
texturecache.py c tvshows "'Allo 'Allo!"

and I get error:
Traceback (most recent call last):
  File "/usr/bin/texturecache.py", line 8663, in <module>
    main(sys.argv[1:])
  File "/usr/bin/texturecache.py", line 8353, in main
    if not checkConfig(argv[0]): sys.exit(2)
  File "/usr/bin/texturecache.py", line 7893, in checkConfig
    data = wcomms.sendJSON(REQUEST, "libPing", checkResult=False, useWebServer=True)
  File "/usr/bin/texturecache.py", line 2230, in sendJSON
    return self.sendWeb("POST", "/jsonrpc", id, request, {"Content-Type": "application/json"}, timeout=timeout)
  File "/usr/bin/texturecache.py", line 2177, in sendWeb
    raise httplib.HTTPException("Remote web host requires webserver.username/webserver.password properties")
httplib.HTTPException: Remote web host requires webserver.username/webserver.password properties