• 1
  • 121
  • 122
  • 123(current)
  • 124
  • 125
  • 197
[RELEASE] Texture Cache Maintenance utility
Hi Milhouse

Thanks for the new version of the script. A timely question: where should the scripts & config file be kept and run from in OE ?

Mine are currently sitting in "\\192.168.1.170\Root File System\storage\", but is this the ideal location for this code ?

Thanks.

Supplementary (n00b) Q: is there a complete list of CFG options I can download for reference ?
EDIT: found it (!) on github.
::  LibreELEC 9.2.6 RELEASE - Generic x86_64  ::  Intel 847 NUC  ::  KVR1333D3S9/4G  ::  Kingston SMS200S3/30G mSATA  ::  MS 1044 MCE keyboard  ::  GP-IR02BK remote  ::  Xonfluence  ::  10.9TiB on FreeNAS v11.3-U5 (RAID-Z2)  ::
Reply
(2016-01-23, 08:35)gjwAudio Wrote: where should the scripts & config file be kept and run from in OE ?

Not sure I really understand, you can store the code wherever you like/wherever makes sense. The config should ideally be in the same directory as the script, or in your current directory when running the script. The config search order is (first is #1):

  1. Script directory
  2. Your current directory
  3. ~/.config

The last one means a folder called .config in your users home directory.

Alternatively you can specify the configuration filename with @config=<filename>

(2016-01-23, 08:35)gjwAudio Wrote: Supplementary (n00b) Q: is there a complete list of CFG options I can download for reference ?
EDIT: found it (!) on github.

Or:

Code:
texturecache.py config

which will list your current configuration (most of them will be defaults). It will also tell you the location of the configuration file that is being used.

The sample on github is out of date - I kinda regret creating it as people think it's a good idea to use it as their cfg file. Just add what properties you need, don't copy the sample file.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Hi Milhouse
(2016-01-20, 07:03)Milhouse Wrote: So when you re-scrape/refresh a movie you're going to lose all your non-standard artwork types, ie. your banner, whether it's in your NFO or not.
Yeah, I HATE when this happens (...quite regularly Sad ).
(2016-01-20, 07:03)Milhouse Wrote: This is where you need to... re-associate the banner artwork manually... - knock up a small script that calls "texturecache.py vscan" then mklocal.py followed by "texturecache.py c movies" and you shouldn't have any more problems with non-standard artwork.

Indeed. I've been looking at the post announcing mklocal.py, and can't quite discern the minimum number of parameters required to make it do what I need Eek. I desperately need that knocked-up script, to fix/restore the non-standard art every time a scan wipes it out !

The chicken-and-egg thing is making me crazy... how can I create a process that will recognize these non-standard guys and "automatically" re-associate them with their respective movies ? Also, can it be run repeatedly without messing up any fixes done previously ?

OK, I presume it's possible Angel. Curiosity (on my part) wants to know how the three steps you outline work together. What does the vscan do/create, that mklocal then acts upon ?

(BTW, you might check the 5th paragraph in that mklocal post - I think a typo has crept into the example of mapping a local file to a URL)

Oh, and... I read most of the AD thread as you suggested... man, I need a shower after going through all that ! You deserve uber-points for patience Rofl.

ps: I'm having no luck understanding what fields can be queried with the "extrajson.* = fieldname" parameters in the config file. I thought the info here would show me a list, but no.
::  LibreELEC 9.2.6 RELEASE - Generic x86_64  ::  Intel 847 NUC  ::  KVR1333D3S9/4G  ::  Kingston SMS200S3/30G mSATA  ::  MS 1044 MCE keyboard  ::  GP-IR02BK remote  ::  Xonfluence  ::  10.9TiB on FreeNAS v11.3-U5 (RAID-Z2)  ::
Reply
(2016-01-26, 08:17)gjwAudio Wrote: Indeed. I've been looking at the post announcing mklocal.py, and can't quite discern the minimum number of parameters required to make it do what I need Eek. I desperately need that knocked-up script, to fix/restore the non-standard art every time a scan wipes it out !

The example in the linked post is pretty minimal, you'd just change the artwork types to "--artwork fanart poster banner" and it will automatically download any remote fanart/poster/banner artwork to your local filesystem alongside your movies, and then associate any new local fanart/poster/banner artwork with your movies. If you don't want remote artwork downloaded, add "--readonly".

Change the --local and --prefix arguments to suit your configuration.

(2016-01-26, 08:17)gjwAudio Wrote: The chicken-and-egg thing is making me crazy... how can I create a process that will recognize these non-standard guys and "automatically" re-associate them with their respective movies ?

That's what we're discussing.

(2016-01-26, 08:17)gjwAudio Wrote: Also, can it be run repeatedly without messing up any fixes done previously ?

Yep.

(2016-01-26, 08:17)gjwAudio Wrote: OK, I presume it's possible Angel. Curiosity (on my part) wants to know how the three steps you outline work together. What does the vscan do/create, that mklocal then acts upon ?

"texturecache.py vscan" initiates a video library scan (use ascan for audio/music scans) - this will add any new movies (or tvshows/episodes) to your library. It's basically the same as clicking "Update library" in the GUI. After this any new movies will have poster and fanart - possibly remote if nothing found locally - but no banners.

You then run mklocal.py to download any remote artwork that is being used (although this is optional), and mklocal.py will also automatically associate movies that don't already have associations with any matching local artwork, including banners.

If a movie already has an association with a local artwork file it won't be changed, unless you add the option --nokeep as this will enable removal of any association where the artwork file (remote or local) no longer exists - handy when cleaning up garbage/stale/non-existent associations.

Once you've created new associations in your media library you might want to pre-load the texture cache of every client by executing "texturecache.py c movies" (or sets, or tvshows) against each client in turn.

I run a process similar to this whenever new movies/episodes are detected on my NAS. Or you could just run it on a regular schedule, say every morning at 3am, or just run it whenever you feel like.

(2016-01-26, 08:17)gjwAudio Wrote: (BTW, you might check the 5th paragraph in that mklocal post - I think a typo has crept into the example of mapping a local file to a URL)

Thanks, fixed.

(2016-01-26, 08:17)gjwAudio Wrote: ps: I'm having no luck understanding what fields can be queried with the "extrajson.* = fieldname" parameters in the config file. I thought the info here would show me a list, but no.

For extrajson.movies: Video.Fields.Movie - click on "[show]":
Code:
"items": {
    "enums": [
      "title",
      "genre",
      "year",
      "rating",
      "director",
      "trailer",
      "tagline",
      "plot",
      "plotoutline",
      "originaltitle",
      "lastplayed",
      "playcount",
      "writer",
      "studio",
      "mpaa",
      "cast",
      "country",
      "imdbnumber",
      "runtime",
      "set",
      "showlink",
      "streamdetails",
      "top250",
      "votes",
      "fanart",
      "thumbnail",
      "file",
      "sorttitle",
      "resume",
      "setid",
      "dateadded",
      "tag",
      "art"

eg.
Code:
$ texturecache.py Jd movies avatar @extrajson.movies=rating,votes,dateadded
[
  {
    "art": {
      "clearart": "image://nfs://192.168.0.3/mnt/share/media/Video/MoviesSD/Avatar (2009)[DVDRip]-clearart.png/",
      "clearlogo": "image://nfs://192.168.0.3/mnt/share/media/Video/MoviesSD/Avatar (2009)[DVDRip]-logo.png/",
      "fanart": "image://nfs://192.168.0.3/mnt/share/media/Video/MoviesSD/Avatar (2009)[DVDRip]-fanart.jpg/",
      "poster": "image://nfs://192.168.0.3/mnt/share/media/Video/MoviesSD/Avatar (2009)[DVDRip]-poster.jpg/"
    },
    "dateadded": "2010-07-09 23:32:43",
    "file": "nfs://192.168.0.3/mnt/share/media/Video/MoviesSD/Avatar (2009)[DVDRip].m4v",
    "label": "Avatar",
    "movieid": 2126,
    "rating": 7.900000095367432,
    "title": "Avatar",
    "votes": "840438"
  }
]
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Found my way into the tool

Rescanning directory: smb://SHARK/Usenet/TV/Ballers/Season 1..
it's possible that it keeps stalling here for minutes (+15 min) but no error message
./texturecache.py qax tvshows "Ballers" gives no output
and when starting: qax again .. it will not use above example anymore and goes to the next
Rescanning directory: smb://SHARK/Usenet/TV/Bitten/Season 2...

my config
userdata = /media/kodi/
dbfile = Database/Textures13.db
thumbnails = Thumbnails/
xbmc.host = 192.168.1.154
webserver.port = 80
#rpc.port = 9090
download.threads = 2
#extrajson.albums =
#extrajson.artists =
#extrajson.songs =
#extrajson.movies =
#extrajson.sets =
extrajson.tvshows.tvshow = plot, rating, cast, file, imdbnumber
extrajson.tvshows.episode = streamdetails, file, dateadded, rating, plot, cast
# extrajson.tvshows.tvshow =
# extrajson.tvshows.season =
# extrajson.tvshows.episode=
#qaperiod = 30
#qa.file = no
qa.fail.missinglocalart=yes
qaperiod=9999
cache.extra = yes
cache.castthumb = yes
allow.recacheall=yes
logfile = /home/dennis/thumbs/texturecache.log
logfile.verbose = yes
#network.mac = aa:bb:cc:dd:ee:ff


log tells me
2016-01-29 21:34:59.905087:MainThread: libRescan.PERFORMING CALLBACK: Name [jsonWaitForScanFinished], with Id [None], Method [VideoLibrary.OnCleanFinished], Params [{u'data': None, u'sender': u'xbmc'}]
2016-01-29 21:34:59.905142:MainThread: libRescan.CALLBACK RESULT: [False] Name [jsonWaitForScanFinished], Id [None], Method [VideoLibrary.OnCleanFinished], Params [{u'data': None, u'sender': u'xbmc'}]
2016-01-29 21:34:59.905186:MainThread: libRescan.READING SOCKET UNTIL CALLBACK SUCCEEDS...

and than it stalls ..
Reply
Send me the whole logfile.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Thanks for such quick reply

http://we.tl/G60T1HbY3H

started with empty logfile
started with ./texturecache.py qax tvshows

logfile 5MB ..
(sorry I have too much tvshows)

ps.
When I use ./texturecache.py qax tvshows "Baby Daddy" than the tool is finishing its job
in above log baby daddy as example is stalling on the callback part and in the 2nd log (much smaller btw) you see it is finishing ok

http://we.tl/9aYzHwj4WY
Reply
(2016-01-30, 09:21)zeroday Wrote: ps.
When I use ./texturecache.py qax tvshows "Baby Daddy" than the tool is finishing its job
in above log baby daddy as example is stalling on the callback part and in the 2nd log (much smaller btw) you see it is finishing ok

http://we.tl/9aYzHwj4WY

This isn't doing a scan, which is why it succeeds - from what I can tell it found no QA issues that needed fixing.

(2016-01-30, 09:21)zeroday Wrote: Thanks for such quick reply

http://we.tl/G60T1HbY3H

started with empty logfile
started with ./texturecache.py qax tvshows

logfile 5MB ..
(sorry I have too much tvshows)

It looks like you've got some other process that is starting a video library clean while the script is running, which is conflicting with the subsequent video library scan started by texturecache.py.

In the following log detail, while texturecache.py is removing a Baby Daddy Season 4 episode, a library clean is started. The texturecache.py script sees the "OnCleanStarted" notification, which it simply ignores as it's not relevant to it's current processing. The library clean is not being started by texturecache.py - the library clean must be being started from the GUI, or an addon or some other external script/cron:
Code:
2016-01-30 09:09:10.367642:MainThread: Removing episodeid 6858 from media library.
2016-01-30 09:09:10.367965:MainThread: libRemove.JSON SOCKET REQUEST: [{"jsonrpc": "2.0", "params": {"episodeid": 6858}, "method": "VideoLibrary.RemoveEpisode", "id": "libRemove"}]
2016-01-30 09:09:10.374653:MainThread: libRemove.BUFFER RECEIVED (len 113)
2016-01-30 09:09:10.375537:MainThread: libRemove.PARSING JSON DATA: {"jsonrpc":"2.0","method":"VideoLibrary.OnRemove","params":{"data":{"id":6858,"type":"episode"},"sender":"xbmc"}}
2016-01-30 09:09:10.376097:MainThread: libRemove.PARSING COMPLETE, elapsed time: 0.000556 seconds
2016-01-30 09:09:10.377243:MainThread: libRemove.IGNORING NOTIFICATION [{"params": {"data": {"type": "episode", "id": 6858}, "sender": "xbmc"}, "jsonrpc": "2.0", "method": "VideoLibrary.OnRemove"}]
2016-01-30 09:09:10.377620:MainThread: libRemove.READING SOCKET FOR A RESPONSE...
2016-01-30 09:09:10.411303:MainThread: libRemove.BUFFER RECEIVED (len 95)
2016-01-30 09:09:10.411673:MainThread: libRemove.PARSING JSON DATA: {"jsonrpc":"2.0","method":"VideoLibrary.OnCleanStarted","params":{"data":null,"sender":"xbmc"}}
2016-01-30 09:09:10.411862:MainThread: libRemove.PARSING COMPLETE, elapsed time: 0.000249 seconds
2016-01-30 09:09:10.412136:MainThread: libRemove.IGNORING NOTIFICATION [{"params": {"data": null, "sender": "xbmc"}, "jsonrpc": "2.0", "method": "VideoLibrary.OnCleanStarted"}]
2016-01-30 09:09:10.412302:MainThread: libRemove.READING SOCKET FOR A RESPONSE...
2016-01-30 09:09:10.482488:MainThread: libRemove.BUFFER RECEIVED (len 48)
2016-01-30 09:09:10.482981:MainThread: libRemove.PARSING JSON DATA: {"id":"libRemove","jsonrpc":"2.0","result":"OK"}

Subsequently, when the texturecache.py script tries to start a library scan - when the library clean is still running - the scan request is silently ignored by Kodi because there is a library clean in progress:

Code:
2016-01-30 09:09:10.790270:MainThread: Rescanning directory: smb://SHARK/Usenet/TV/Baby Daddy/Season 4...
2016-01-30 09:09:10.790723:MainThread: libRescan.JSON SOCKET REQUEST: [{"jsonrpc": "2.0", "params": {"directory": "smb://SHARK/Usenet/TV/Baby Daddy/Season 4", "showdialogs": false}, "method": "VideoLibrary.Scan", "id": "libRescan"}]
2016-01-30 09:09:10.792881:MainThread: libRescan.BUFFER RECEIVED (len 48)
2016-01-30 09:09:10.793348:MainThread: libRescan.PARSING JSON DATA: {"id":"libRescan","jsonrpc":"2.0","result":"OK"}
2016-01-30 09:09:10.793600:MainThread: libRescan.PARSING COMPLETE, elapsed time: 0.000292 seconds
2016-01-30 09:09:10.793822:MainThread: libRescan.PERFORMING CALLBACK: Name [jsonWaitForScanFinished], with Id [libRescan], Method [OK], Params [None]
2016-01-30 09:09:10.794070:MainThread: libRescan.CALLBACK RESULT: [False] Name [jsonWaitForScanFinished], Id [libRescan], Method [OK], Params [None]
2016-01-30 09:09:10.794276:MainThread: libRescan.READING SOCKET UNTIL CALLBACK SUCCEEDS...
2016-01-30 09:09:28.459701:MainThread: libRescan.BUFFER RECEIVED (len 96)
2016-01-30 09:09:28.459934:MainThread: libRescan.PARSING JSON DATA: {"jsonrpc":"2.0","method":"VideoLibrary.OnCleanFinished","params":{"data":null,"sender":"xbmc"}}
2016-01-30 09:09:28.460025:MainThread: libRescan.PARSING COMPLETE, elapsed time: 0.000131 seconds
2016-01-30 09:09:28.460109:MainThread: libRescan.PERFORMING CALLBACK: Name [jsonWaitForScanFinished], with Id [None], Method [VideoLibrary.OnCleanFinished], Params [{u'data': None, u'sender': u'xbmc'}]
2016-01-30 09:09:28.460176:MainThread: libRescan.CALLBACK RESULT: [False] Name [jsonWaitForScanFinished], Id [None], Method [VideoLibrary.OnCleanFinished], Params [{u'data': None, u'sender': u'xbmc'}]
2016-01-30 09:09:28.460231:MainThread: libRescan.READING SOCKET UNTIL CALLBACK SUCCEEDS...

Consequently, the texturecache.py script never sees a "VideoLibrary.OnScanFinished" notification and is left hanging, waiting forever for a notification that is never going to come (unless you manually start another library scan).

18 seconds after the library scan request (which didn't start, despite the "OK" result), the library clean finishes but again this notification is not relevant as the texturecache.py script is waiting for a scan to finish, not a clean... so it continues to wait.

Perhaps I could add some sort of "state" so that the script knows if a library clean has been started (or queries the clean/scan status before trying to kick off a scan, and polls/waits for it to complete)... nah, just stop running whatever process is automatically starting the library clean.

If you want to clean your library you can use the texturecache.py script (aclean/vclean), but to avoid hanging the script while it is trying to run a library scan simply stop cleaning your library at the same time!
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
ok, this helped me a lot

The issue is the xbmc auto updater plugin

When we do the scan ./texturecache qax tvshows I see that kodi.log notices that there is an incoming scan
but .. in my config fo the xbmc auto updater plugin I have stated: clean after scan

This is done with a purpose:

make sure that there are no unleft references of removed tvshows after the database have been scanned
(ie. HDTV replaced by WEB-DL)

After changing this setting to daily .. the issue was resolved.

So if I am correct I coudl do following:
disable xbmc auto updater as plugin in Kodi
create simple shell script (anyone?) with a few ./texturecache commands like scan than ./texturecache c | ./texturecache qax than clean .. Or something like this?
Reply
(2016-01-30, 19:34)zeroday Wrote: So if I am correct I coudl do following:
disable xbmc auto updater as plugin in Kodi
create simple shell script (anyone?) with a few ./texturecache commands like scan than ./texturecache c | ./texturecache qax than clean .. Or something like this?

Yep, should work - you can automate pretty much everything with texturecache.py. Run the script on demand (click from a desktop), on a schedule (ie. daily) or run the script constantly (daemon) and then vscan/qax/c/vclean whenever new content becomes available. How creative you want to be is up to you - I do a combination of the last two options, vscan/qax/c as new movies/episodes appear in the NAS filesystem, with a regular daily qax/c to process any new thumbnails/artwork or updated NFOs.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Can I use this: RunScript(script[,args]*) to create a button for this in my kodi home menu?

Like say, Action = RunScript(special://profile/texturecache.py[,c]) ?

Or am I way off base with this?
Reply
Sir, i have an issue here, and i hope texture cache can help me with this.

Recently I deleted a few duplicate movies manually. The movies would show up in Kodi two times, anywas, will this utility help me remove those duplicate entires from Kodi?
I have already run the command using the switch "c", "p", "P" ran all one after another (without the double quote)
It ran did a lot of things, but still the entries remain.
Please guide, or do i need to delete the movies database files and re-update kodi?
Reply
(2016-02-01, 17:24)lstar337 Wrote: Can I use this: RunScript(script[,args]*) to create a button for this in my kodi home menu?

Like say, Action = RunScript(special://profile/texturecache.py[,c]) ?

Or am I way off base with this?

Should be possible. Note the square brackets denote optionality, so you don't include the [] in your actual command.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2016-02-01, 23:30)MANswers Wrote: Recently I deleted a few duplicate movies manually. The movies would show up in Kodi two times, anywas, will this utility help me remove those duplicate entires from Kodi?

The script won't automatically remove duplicate movies, but it will identify duplicate movies based on imdbnumber ("texturecache.py duplicates"). What you do with them is up to you - you can remove them based on the movie id ("texturecache.py remove movie 12345" where 12345 is the movie id).

If you're having problems with duplicate movies you probably need to work out what is causing that and delete or rename the media files that are causing the duplicates to be created by Kodi.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2016-02-02, 01:46)Milhouse Wrote:
(2016-02-01, 17:24)lstar337 Wrote: Can I use this: RunScript(script[,args]*) to create a button for this in my kodi home menu?

Like say, Action = RunScript(special://profile/texturecache.py[,c]) ?

Or am I way off base with this?

Should be possible. Note the square brackets denote optionality, so you don't include the [] in your actual command.
Thanks.

I gave it a go, but I get an addon error. Looking in the log it says this:

Code:
10:27:01 T:2640049056 WARNING: CPythonInvoker(49): Script invoked without an addon. Adding all addon modules installed to python path as fallback. This behaviour will be removed in future version.
10:27:01 T:2640049056   ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
                                             - NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
                                            Error Type: <type 'exceptions.AttributeError'>
                                            Error Contents: xbmcout instance has no attribute 'isatty'
                                            Traceback (most recent call last):
                                              File "/storage/.kodi/userdata/texturecache.py", line 8492, in <module>
                                                main(sys.argv[1:])
                                              File "/storage/.kodi/userdata/texturecache.py", line 8193, in main
                                                loadConfig(argv)
                                              File "/storage/.kodi/userdata/texturecache.py", line 7703, in loadConfig
                                                gLogger = MyLogger()
                                              File "/storage/.kodi/userdata/texturecache.py", line 897, in __init__
                                                self.ISATTY = sys.stdout.isatty()
                                            AttributeError: xbmcout instance has no attribute 'isatty'
                                            -->End of Python script error report<--

This is really reaching the limit of my understanding. Rofl

Here is what I am putting in:
Image
Reply
  • 1
  • 121
  • 122
  • 123(current)
  • 124
  • 125
  • 197

Logout Mark Read Team Forum Stats Members Help
[RELEASE] Texture Cache Maintenance utility17