[rainman74]

@millhouse When cleaning, the castthumbs are always deleted, although I have activated them in the setting. What am I doing wrong?

Code:

python texturecache.py P @profile.name=Parents

texturecache.cfg

Code:

#userdata = ~/.xbmc/userdata/
#dbfile = Database/Textures13.db
#thumbnails = Thumbnails/
xbmc.host = localhost
webserver.port = 8080
rpc.port = 9999
download.threads = 3
#extrajson.albums  =
#extrajson.artists =
#extrajson.songs   =
#extrajson.movies  =
#extrajson.sets    =
#extrajson.tvshows.tvshow =
#extrajson.tvshows.season =
#extrajson.tvshows.episode=
#qaperiod = 30
#qa.file = no
query.seasons = yes
query.episodes = yes
cache.castthumb = yes
cache.extra = yes
prune.retain.previews = yes
prune.retain.pictures = yes
prune.retain.chapters = yes
#logfile =
#logfile.verbose = no
#network.mac = aa:bb:cc:dd:ee:ff
checkupdate = yes
orphan.limit.check = no


Code:

Command line args: ['texturecache.py', 'c', '@profile.name=Parents']
Current version #: v2.5.1
Current platform : win32
Python  version #: v3.7.4.0 (final)


[Milhouse]

I don't know how you've populated the "Parents" media library, or corresponding texture cache. Is the "Parent" profile separate or shared? If separate, were the "Parent" libraries copied from "master" (default) or created from scratch?

If you copied from master, maybe the cached cast artwork isn't present in the "Parents" media library. Check you have enabled "Download actor thumbnails" in Settings > Media > Videos for the "Parents" profile.

Failing that, a logfile (@logfile=tc.log) for the p prune command should help explain what is occurring (lowercase p won't remove anything, but will instead list what it would have deleted). I'll need the entire log file, so probably best to zip it and upload it somewhere.

Note that the cache.castthumb option only dictates whether the script will cache cast artwork (if there are references to the cast artwork in your media library). In the texture cache there's no way to tell a cast artwork from a poster - all cached artwork is just a "thumbnail" without any specific type (ie. poster, fanart, cast etc.) - so there's no way to prevent removal of a thumbnail that happens to be a cast artwork, apart from adding a regex to prune.retain.types - if all your cast artwork is hosted on imdb.com then adding prune.retain.types=http://ia.media-imdb.com might stop unreferenced cast artwork from being removed.

But adding the missing references (ie. fixing the underlying issue) might be easier.

[rainman74]

I don't know how you've populated the "Parents" media library, or corresponding texture cache. Is the "Parent" profile separate or shared? If separate, were the "Parent" libraries copied from "master" (default) or created from scratch?

The parent profile is my master profile ("Eltern" in German), I just renamed it. Strangely enough, the cast thumbs are not deleted from the second "Children" ("Kinder" in German) profile, which I created as an additional standalone separated profile (python texturecache.py p @profile.name=Children).

If you copied from master, maybe the cached cast artwork isn't present in the "Parents" media library. Check you have enabled "Download actor thumbnails" in Settings > Media > Videos for the "Parents" profile.

I had already activated this option.

Failing that, a logfile (@logfile=tc.log) for the p prune command should help explain what is occurring (lowercase p won't remove anything, but will instead list what it would have deleted). I'll need the entire log file, so probably best to zip it and upload it somewhere.

Ok, there is my log file:
log.zip

Note that the cache.castthumb option only dictates whether the script will cache cast artwork (if there are references to the cast artwork in your media library). In the texture cache there's no way to tell a cast artwork from a poster - all cached artwork is just a "thumbnail" without any specific type (ie. poster, fanart, cast etc.) - so there's no way to prevent removal of a thumbnail that happens to be a cast artwork, apart from adding a regex to prune.retain.types - if all your cast artwork is hosted on imdb.com then adding prune.retain.types=http://ia.media-imdb.com might stop unreferenced cast artwork from being removed.

If the root cause can't be found, I should try that, shouldn't I?
prune.retain.types=/.actors/.*


Update:
I have just seen that with selected "Parent" profile also thumbnails of children medias (e.g. episode thumbnails) should also be deleted, but they are not present in the parent profile at all, these media files were never added by me to the library in the parent profile!

Is it possible that Kodi gets confused internally when using profiles, or was it because I cached the textures before? But also here I strictly separated the profiles:

First -> python texturecache.py c @profile.name=Parents
Second -> python texturecache.py c @profile.name=Children

[Milhouse]

Sorry for taking so long to reply. I've just downloaded your log.zip, but it looks like you've reformatted it (pretty printed the JSON data) which has kind of made it a bit useless to me as I need it in the original raw format - the texturecache.py script has a @replayfile option which takes the log file as input and allows me to reproduce and debug your data as though it were my data (assuming the log hasn't been tampered with...).

Any chance you can run the command again, and upload the unmodified log file?

[rainman74]

Sorry for taking so long to reply. I've just downloaded your log.zip, but it looks like you've reformatted it (pretty printed the JSON data) which has kind of made it a bit useless to me as I need it in the original raw format - the texturecache.py script has a @replayfile option which takes the log file as input and allows me to reproduce and debug your data as though it were my data (assuming the log hasn't been tampered with...).

Any chance you can run the command again, and upload the unmodified log file?

ok, here it is ... link

[Milhouse]

Sorry for taking so long to reply. I've just downloaded your log.zip, but it looks like you've reformatted it (pretty printed the JSON data) which has kind of made it a bit useless to me as I need it in the original raw format - the texturecache.py script has a @replayfile option which takes the log file as input and allows me to reproduce and debug your data as though it were my data (assuming the log hasn't been tampered with...).

Any chance you can run the command again, and upload the unmodified log file?

ok, here it is ... link

Sorry for the delay (again) - been really busy.

OK, I've figured out why your log looks the way it does - you've configured:

text:


<jsonrpc>

    <compactoutput>false</compactoutput>

</jsonrpc>

in your Kodi advancedsettings.xml, which means the JSON responses from Kodi are being written to the log in a "pretty" format that I can't currently parse (and replay).

Can you disable this compactoutput setting in your as.xml (or set it to true), restart kodi, reproduce the issue and then re-upload your log (for a third time - sorry!)

Maybe one day I'll store the responses from Kodi with all tabs and newlines stripped so that this as.xml setting doesn't cause a problem, but I'm not likely to get around to it for a while.

[rainman74]

Sorry for taking so long to reply. I've just downloaded your log.zip, but it looks like you've reformatted it (pretty printed the JSON data) which has kind of made it a bit useless to me as I need it in the original raw format - the texturecache.py script has a @replayfile option which takes the log file as input and allows me to reproduce and debug your data as though it were my data (assuming the log hasn't been tampered with...).

Any chance you can run the command again, and upload the unmodified log file?

ok, here it is ... link 

Sorry for the delay (again) - been really busy.

OK, I've figured out why your log looks the way it does - you've configured:

text:


<jsonrpc>

<compactoutput>false</compactoutput>

</jsonrpc>

in your Kodi advancedsettings.xml, which means the JSON responses from Kodi are being written to the log in a "pretty" format that I can't currently parse (and replay).

Can you disable this compactoutput setting in your as.xml (or set it to true), restart kodi, reproduce the issue and then re-upload your log (for a third time - sorry!)

Maybe one day I'll store the responses from Kodi with all tabs and newlines stripped so that this as.xml setting doesn't cause a problem, but I'm not likely to get around to it for a while.  

That's no problem at all! Okay, I'll do that. But it will take a little longer, because in desperation I deleted my whole library & textures db and have it scrapened and cached again. This will take a little longer, because the scraping over SMB is hellish slow...

But maybe in the meantime you can tell me why I get this error message when caching music videos:

Code:

Traceback (most recent call last):0, multi thread 2288
  File "texturecache.py", line 8643, in <modules>
    main(sys.argv[1:])
  File "texturecache.py", line 8448, in main
    extraFields=_extraFields, query=_query, drop_items=_drop_items)
  File "texturecache.py", line 4668, in jsonQuery
    cacheImages(mediatype, jcomms, database, data, title_name, id_name, force, nodownload, drop_items)
  File "texturecache.py", line 4834, in cacheImages
    tCount = gConfig.DOWNLOAD_THREADS["download.threads.%s" % mediatype]
KeyError: 'download.threads.musicvideos'


In texturecache.cfg I use:

Code:

download.threads = 4

[Milhouse]

But maybe in the meantime you can tell me why I get this error message when caching music videos:

Code:

Traceback (most recent call last):0, multi thread 2288
  File "texturecache.py", line 8643, in <modules>
    main(sys.argv[1:])
  File "texturecache.py", line 8448, in main
    extraFields=_extraFields, query=_query, drop_items=_drop_items)
  File "texturecache.py", line 4668, in jsonQuery
    cacheImages(mediatype, jcomms, database, data, title_name, id_name, force, nodownload, drop_items)
  File "texturecache.py", line 4834, in cacheImages
    tCount = gConfig.DOWNLOAD_THREADS["download.threads.%s" % mediatype]
KeyError: 'download.threads.musicvideos'


In texturecache.cfg I use:

Code:

download.threads = 4

Ah, that's a bug - I don't have any musicvideos so have done minimal testing with that media type. I'll push a fix shortly.

[Milhouse]

I've pushed v2.5.2 of the script, it adds the missing musicvideos key which hopefully fixes the issue you found.

I've also add rudimentary support for pretty-formatted JSON responses so you should be able to upload a log I can use without disabling <compactoutput>false</compactoutput>.

[rainman74]

I've pushed v2.5.2 of the script, it adds the missing musicvideos key which hopefully fixes the issue you found.

I've also add rudimentary support for pretty-formatted JSON responses so you should be able to upload a log I can use without disabling <compactoutput>false</compactoutput>.

Thank you!

I was about to create a new log, but noticed that this time no cast thumbs were cached because I used the default config, damn! So I have to run this again with "cache.castthumb = yes" ...  ;-)

[rainman74]

I've pushed v2.5.2 of the script, it adds the missing musicvideos key which hopefully fixes the issue you found.

I've also add rudimentary support for pretty-formatted JSON responses so you should be able to upload a log I can use without disabling <compactoutput>false</compactoutput>.

So, after 2 days now all castthumbs were cached over SMB again, I started a test prune of the texture cache.

Code:

python texturecache.py c @profile.name=Eltern

The program wants to delete 22,196 files after the test run (python texturecache.py p @profile.name=Eltern), although I read everything completely new and then cached. That can therefore definitely not be that these are all files that are no longer used.

You can find the log here. What am I doing wrong?


P.S. Caching of musicvideos works now! ;-()


Update:
Could it be because I renamed the "Master User" profile to "Eltern" and it doesn't find the correct path?
"PROFILE_DIRECTORY": "profiles\\Eltern" <- does not exist and this is generated automatically.

But if I start the program without the "@profile.name=Eltern", it doesn't work:
--> ERROR: Profile "Master user" is not valid!


Update II:
I tested it once, but even this command doesn't work and there are still 22,196 files to delete:

Code:

python texturecache.py p @profile.name=Eltern @profile.directory=C:\Users\Karlheinz\AppData\Roaming\Kodi\userdata

[Milhouse]

Thanks for your patience. I've spent a few hours looking at this today, and I've found two bugs in my code, which I think I've fixed in this version (not yet merged): http://ix.io/20DH - you can try testing it, but I don't think it will make much difference for you (the first issue failed to process Extras correctly for your folder Der 90. Geburtstag oder Dinner for One (1963), while the second issue related to incorrect handling of Windows paths on Linux which is only an issue when replaying a log file generated on a different platform - you're using Windows, I'm replaying your file on Linux etc.).

With this updated version I've been able to replay your log, and it mostly wants to remove textures with paths starting P:\Kinder\ (Filme2, Serien, etc.) - the first 50 can be see here: http://ix.io/20DL

I can also confirm that it wants to remove 22,196 items:

text:


000026|b/b9ac609e.jpg|0404|0720|0002|2019-10-28 16:27:45|                   |video@P:\Kinder\Serien2\The Voice of Germany\Staffel 9\The Voice of Germany - S09E14.mkv

000034|1/177e348f.jpg|0404|0720|0002|2019-10-28 16:27:45|                   |video@P:\Kinder\Serien\Soy Luna\Staffel 1\Soy Luna - S01E01 - Folge 01.mkv



Summary: 22,196 files

This all looks to be pretty much entirely correct, as these are cached textures for one of your other profiles (Kinder)), as you are pruning while using the profile Eltern.

I can't see how you can share a single texture cache (database + Thumbnails folder) for two profiles and avoid pruning the textures for the inactive profiles - each profile should be using it's own dedicated texture cache (ie. Textures13.db database and Thumbnails folder) which is synced to the media library (MyMusic72/MyMovies116 etc) that is used by the profile.

It also wants to remove 1430 other textures, mostly starting with the path P:\Musik\, these look like album cover art (folder.jpg, for albums you no longer have in your MyMusic media library). So pruning these textures would appear to be correct as well.

[rainman74]

...

Thank you very much for your research.

First of all, the 1430 music files are all still there, but the program still wants to delete the textures. This still seems to be a bug to me.

But now to the main problem. Now my guess seems to be correct with the error in the profile handling of Kodi! I had already observed this several times, but first I thought it couldn't be. But there is no error in your script!

Here my observation:

• If your script switched an active sub-profile (in my case "children") back to the master profile (parents), or if you do this in Kodi itself via the profile switch, then the media of the sub-profile will still be displayed for about 5-7 seconds, although the master profile is already active!
  
• The correct media will only be displayed in the widgets (Last added, etc.) after this time.
  
• The whole "mess" even goes so far that you can quickly switch to movies or series after switching, you can even see the media of parents and children mixed first.
  
• This will then be cleaned up automatically after 5-7 seconds (but sometimes only after a restart of Kodi) and you will only see the media of parents (Master Profile).
  
• Thus I strongly suspect that the caching already starts and that elements of the previously active sub-profile are cached in the Master Profile, although they are not present in the profile at all, but are still displayed for some time.
  

What do you think, could that phenomenon be the cause? But that would then be a very unpleasant error in Kodi's caching!

[Milhouse]

First of all, the 1430 music files are all still there, but the program still wants to delete the textures. This still seems to be a bug to me.

There don't appear to be any media library items that reference those P:\Musik textures, unless it's some sort of assumed relationship.

For example - these textures don't have matching media library entries:

text:


000925|c/c3f679d7.jpg|0512|0512|0001|2019-10-28 10:45:16|2019-10-28 11:45:16|P:\Musik\A Life Divided\Passenger\folder.jpg

001027|8/805e5ed2.jpg|0512|0512|0001|2019-10-28 10:45:38|2019-10-28 11:45:38|P:\Musik\A Life Divided\The Great Escape (Deluxe Editi\folder.jpg

001099|a/ae355e19.jpg|0700|0700|0001|2019-10-28 10:48:10|2019-10-28 11:48:10|P:\Musik\A Life Divided\folder.jpg

000653|f/ff8f56e4.jpg|0512|0512|0001|2019-10-28 10:44:19|2019-10-28 11:44:19|P:\Musik\ALMA\Chasing Highs\folder.jpg

000825|5/54ac3d8e.jpg|0512|0512|0001|2019-10-28 10:44:57|2019-10-28 11:44:56|P:\Musik\ALMA\Karma\folder.jpg

001111|c/c41b8bdd.jpg|0600|0600|0001|2019-10-28 10:48:12|2019-10-28 11:48:12|P:\Musik\ALMA\folder.jpg

so try texturecache.py jd albums Passenger and see if anything is found, as there's nothing in the media library for the Eltern profile. It might be an album in the media library of a different profile, which again would explain why the script wants to remove these textures when you are pruning using the Eltern profile.

What do you think, could that phenomenon be the cause? But that would then be a very unpleasant error in Kodi's caching!

What you suggest is entirely possible - unfortunately the profile handling in Kodi has many problems and the plan is to rewrite/reimplement it from scratch, as it's considered beyond fixing in any sane way.

Given that profile handling in Kodi is flaky at best, it might be necessary to avoid using the script to change profiles.

Or try this (last throw of the dice): change the profile using the script then quit Kodi (something like texturecache.py @profile.name=Eltern power exit might work on Windows - ie. switch to the Eltern profile, and then exit/shutdown Kodi). Start kodi again (making sure it is configured to use the "Last used profile" on startup), and *then* cache or prune artwork for the active profile without changing the profile again, ie. texturecache.py @profile.enabled=no c movies would cache movie artwork for the currently active profile (ie. Eltern in this example).

You can automate all of this using a Windows command script.

You can also view the currently active profile with texturecache.py @profile.enabled=no status.

[rainman74]

...

Thanks for the tips.

With the musik artworks you were right, since I had read all media folders again, I had forgotten the music folder with my parent master profile and had not read it yet (only with the children), therefore the deletion is correct, however, it is also so, that no music files exist in the parent profile, but nevertheless after the profile switch briefly erroneously in the master profile are cached!

Since the erroneous behavior does not only happen with your script, but also quite normally when you switch the profiles in Kodi, it doesn't help me to pay attention to it when caching via texturecache.py, because at the latest with the next manual switch the artworks are again wrongly cached in the wrong profile.

So I can only hope that this bug will soon be fixed in one of the upcoming Kodi 19 versions.