[RELEASE] Texture Cache Maintenance utility

  Thread Rating:
  • 16 Vote(s) - 4.75 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
Milhouse Offline
Team-Kodi Member
Posts: 12,686
Joined: Jan 2011
Reputation: 626
Post: #46
Version 0.2.7:

* Added new QA test, qa.file (yes/no, default:no) to check if the underlying video file still exists. Won't initiate remove/rescan in qax, obviously Smile, and is thus purely informational.

* Added "Duplicate" statistic summary class shown at the end of c/C. This is for when the same artwork is used multiple times. Rather than attempting to re-cache the same item as it is referenced each time by different media (which is pointless, and worse, with the C option could result in the same image being removed then re-cached multiple times), the script now remembers each image item as it is referenced during the current run, and will only cache an image once, quickly ignoring any duplicate references, resulting in improved run-times.

* Tidied up logging, c/C statistics (now with added key), and a little bit of code refactoring (hope it didn't break anything...)

Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
(This post was last modified: 2013-03-26 06:45 by Milhouse.)
find quote
Milhouse Offline
Team-Kodi Member
Posts: 12,686
Joined: Jan 2011
Reputation: 626
Post: #47
Version 0.2.8:

* Added jd and Jd options. Functionally equivalent to the existing j and J options, these new options will also decode encoded URLs.

Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
find quote
Milhouse Offline
Team-Kodi Member
Posts: 12,686
Joined: Jan 2011
Reputation: 626
Post: #48
Version 0.2.9:

* Added support for Video Library Tags. Use the "tags" keyword as you would movies/sets/tvshows etc., and optionally filter by a tag name (including partial tag names). Options with tag support: j, J, jd, Jd, c, C, qa and qax.

For example:
Code:
./texturecache.py j tags live-comedy
./texturecache.py c tags anime
./texturecache.py qa tags foreign-language
./texturecache.py qax tags

The last example will process all movies that have at least one tag assigned. Not currently possible to filter by more than one tag.

Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
find quote
charrua Offline
Senior Member
Posts: 273
Joined: Nov 2008
Reputation: 14
Post: #49
(2013-03-26 12:52)MilhouseVH Wrote:  Version 0.2.9:

* Added support for Video Library Tags. Use the "tags" keyword as you would movies/sets/tvshows etc., and optionally filter by a tag name (including partial tag names). Options with tag support: j, J, jd, Jd, c, C, qa and qax.

For example:
Code:
./texturecache.py j tags live-comedy
./texturecache.py c tags anime
./texturecache.py qa tags foreign-language
./texturecache.py qax tags

The last example will process all movies that have at least one tag assigned. Not currently possible to filter by more than one tag.
Thanks for all the new features!

One more suggestion for the future:
multiple threads for speeding up long tasks
find quote
Milhouse Offline
Team-Kodi Member
Posts: 12,686
Joined: Jan 2011
Reputation: 626
Post: #50
(2013-03-26 13:09)charrua Wrote:  One more suggestion for the future:
multiple threads for speeding up long tasks

Run multiple separate processes? At the end of the day, the speed is determined by the server... some systems (eg. Pi) are pretty much maxed out by a single task.

Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
find quote
un1versal Offline
Out of Memory (1939–2016)
Posts: 7,134
Joined: Oct 2012
Location: Binary Pulsar
Post: #51
I have a question regarding cache.

Currently total duration mostly noticeable on TVshows needs to be calculated/cached in Im not sure where this info is cached (I do believe its textures#.db since this file grows as this info is cached) but its a slow process that delays displaying of SD/HD (blue/green) flags in confluence and other flagging is also delayed until total duration is calculated for all materials. To calculate this for TV shows you need to be in library view and under all seasons/episodes (lists all episodes oof all seasons view) so total duration starts being calculated and thus SD/HD and codec info flags start showing up.

Can this script somehow trigger this caching in future?
Particularly where a user doesnt have to navigate around and stay in that root dir waiting for this caching/calculation to take place. its a long monotonous job.# as soon as you leave any directory this caching stops so basically means if you want everything cached its a pita.

uNi
(This post was last modified: 2013-03-26 13:24 by un1versal.)
find quote
charrua Offline
Senior Member
Posts: 273
Joined: Nov 2008
Reputation: 14
Post: #52
(2013-03-26 13:14)MilhouseVH Wrote:  
(2013-03-26 13:09)charrua Wrote:  One more suggestion for the future:
multiple threads for speeding up long tasks
Run multiple separate processes? At the end of the day, the speed is determined by the server... some systems (eg. Pi) are pretty much maxed out by a single task.
Specially while using c/C options, I've noticed that the script is slower at making XBMC preload the images than XBMC loading the images (not yet cached) when using the GUI. I think that's because XBMC loads multiple images at once (even when they aren't yet cached) while the script is loading them one by one.
The process is not CPU intensive, probably the speed is determined by the http requests to get the images, which I think could be greatly improved by implementing multiple threads.
find quote
Milhouse Offline
Team-Kodi Member
Posts: 12,686
Joined: Jan 2011
Reputation: 626
Post: #53
(2013-03-26 13:26)charrua Wrote:  Specially while using c/C options, I've noticed that the script is slower at making XBMC preload the images than XBMC loading the images (not yet cached) when using the GUI. I think that's because XBMC loads multiple images at once (even when they aren't yet cached) while the script is loading them one by one.
The process is not CPU intensive, probably the speed is determined by the http requests to get the images, which I think could be greatly improved by implementing multiple threads.

I think the default is for 2 threads on XBMC (certainly on the Pi).

You could run the script on a more powerful machine, such as a desktop, or even multiple scripts on a desktop, and point the desktop at an XBMC client. However it's the image conversion that has to be done by the XBMC which is where the load/delays can occur, particularly on a Pi. Not even sure if multiple http downloads are possible (this is what triggers the caching process).

Thing is, though, I really don't want to to get involved in writing threaded code if I can help it! If I run out of ideas I'll kick it around to see what's involved... but no promises!

(2013-03-26 13:17)uNiversal Wrote:  I have a question regarding cache.

Currently total duration mostly noticeable on TVshows needs to be calculated/cached in Im not sure where this info is cached (I do believe its textures#.db since this file grows as this info is cached) but its a slow process that delays displaying of SD/HD (blue/green) flags in confluence and other flagging is also delayed until total duration is calculated for all materials. To calculate this for TV shows you need to be in library view and under all seasons/episodes (lists all episodes oof all seasons view) so total duration starts being calculated and thus SD/HD and codec info flags start showing up.

Can this script somehow trigger this caching in future?
Particularly where a user doesnt have to navigate around and stay in that root dir waiting for this caching/calculation to take place. its a long monotonous job.# as soon as you leave any directory this caching stops so basically means if you want everything cached its a pita.

uNi

Certainly this script will cache the tvshow images, but not entirely sure about the stream details (SD/HD etc.), it may be possible to "pre-load" that information by requesting the stream details via json (can already do this with the J/Jd option and adding streamdetails property to extra.json.tvshows.episode)... I'll investigate shortly, just adding some extra tag support, or give it a try and see if you get valid stream details for newly added tvshows (and then see if they're there in the GUI when you access the Season/Episode list).

Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
(This post was last modified: 2013-03-26 13:49 by Milhouse.)
find quote
charrua Offline
Senior Member
Posts: 273
Joined: Nov 2008
Reputation: 14
Post: #54
(2013-03-26 13:43)MilhouseVH Wrote:  
(2013-03-26 13:26)charrua Wrote:  Specially while using c/C options, I've noticed that the script is slower at making XBMC preload the images than XBMC loading the images (not yet cached) when using the GUI. I think that's because XBMC loads multiple images at once (even when they aren't yet cached) while the script is loading them one by one.
The process is not CPU intensive, probably the speed is determined by the http requests to get the images, which I think could be greatly improved by implementing multiple threads.
I think the default is for 2 threads on XBMC (certainly on the Pi).
You could run the script on a more powerful machine, such as a desktop, or even multiple scripts on a desktop, and point the desktop at an XBMC client. However it's the image conversion that has to be done by the XBMC which is where the load/delays can occur, particularly on a Pi. Not even sure if multiple http downloads are possible (this is what triggers the caching process).
Thing is, though, I really don't want to to get involved in writing threaded code if I can help it! If I run out of ideas I'll kick it around to see what's involved... but no promises!
Thanks! (only a suggestion anyway... Smile)
I tried running multiple instances of the script simultaneously, and it worked, so it will probably work using multiple threads too, but I understand completely that using threads in a script is not always the easiest way to go.

(2013-03-26 13:17)uNiversal Wrote:  I have a question regarding cache.
Currently total duration mostly noticeable on TVshows needs to be calculated/cached in Im not sure where this info is cached (I do believe its textures#.db since this file grows as this info is cached) but its a slow process that delays displaying of SD/HD (blue/green) flags in confluence and other flagging is also delayed until total duration is calculated for all materials. To calculate this for TV shows you need to be in library view and under all seasons/episodes (lists all episodes oof all seasons view) so total duration starts being calculated and thus SD/HD and codec info flags start showing up.
Can this script somehow trigger this caching in future?
Particularly where a user doesnt have to navigate around and stay in that root dir waiting for this caching/calculation to take place. its a long monotonous job.# as soon as you leave any directory this caching stops so basically means if you want everything cached its a pita.
uNi
The total duration of media is not cached, it's calculated in real time each time you access a list.
If you want the loading of lists to be faster you can use a skin that doesn't show that info (there are a few).
find quote
un1versal Offline
Out of Memory (1939–2016)
Posts: 7,134
Joined: Oct 2012
Location: Binary Pulsar
Post: #55
(2013-03-26 13:43)MilhouseVH Wrote:  Certainly this script will cache the tvshow images, but not entirely sure about the stream details (SD/HD etc.), it may be possible to "pre-load" that information by requesting the stream details via json (can already do this with the J/Jd option and adding streamdetails property to extra.json.tvshows.episode)... I'll investigate shortly, just adding some extra tag support, or give it a try and see if you get valid stream details for newly added tvshows (and then see if they're there in the GUI when you access the Season/Episode list).

Excellent, if you wish to duplicate the behaiviour I described, try this with a TV show that has at least 50 episodes. Remove it from library and re-add it then navigate to TVShows>Library>Tittle>All Seasons you will notice all stream details are being re-cached and not listed immediately, as total duration gets calculated and to make flags show up and they get calculated jump from listing to scroll bar (this triggers refresh of flags).

Would be extra nice if script could trigger the calculation of the total duration as well but am unsure if this is at all possible.

Thanks for response will be keeping an eye.

uNi
find quote
Milhouse Offline
Team-Kodi Member
Posts: 12,686
Joined: Jan 2011
Reputation: 626
Post: #56
v0.3.0 is up:

* Added combinatorial logic for tags, eg.:
Code:
./texturecache.py c tags "comedy and standup"
./texturecache.py c tags "action or adventure"

Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
find quote
Milhouse Offline
Team-Kodi Member
Posts: 12,686
Joined: Jan 2011
Reputation: 626
Post: #57
(2013-03-26 13:57)uNiversal Wrote:  Would be extra nice if script could trigger the calculation of the total duration as well but am unsure if this is at all possible.

Thanks for response will be keeping an eye.

uNi

Sadly not having much luck.

I've tried all the methods that return stream details (Files.GetFileDetails, VideoLibrary.GetEpisodeDetails, individually and in combination), even downloading the video file (Files.PrepareDownload then downloading over HTTP, tried Files.Download), and I can't find any JSON method that causes XBMC to extract and return the stream details.

Basically, unless you've played back the video in XBMC (or listed it in the GUI, waiting for the SD/HD tags to appear), then XBMC will refuse to return stream details over JSON, which sucks, as I know the problem you are referring to but it looks like it's being handled specially by the GUI and beyond JSON at this time. Probably should be raised as an issue, as this renders the streamdetails side of JSON nigh on useless - virtually my entire library has no stream details according to JSON.

I don't have any iOS or Android apps here, but I'm guessing they don't show streamdetails for videos that haven't yet been pulled up in the GUI, or never played back,but if they are showing these stream details for new media then there must be a way...

Suggestions welcomed!

(2013-03-26 13:55)charrua Wrote:  The total duration of media is not cached, it's calculated in real time each time you access a list.
If you want the loading of lists to be faster you can use a skin that doesn't show that info (there are a few).

It seems this has to be extracted from the media on the fly, if it's not already specified in the NFOs, and this is only done the first time the item is listed (or played back)... once that information is extracted, and stored in the media library, it should be much quicker to calculate the total runtime of a given list of items.

The trick is finding the trigger that causes XBMC to extract this information from specific media items - it certainly doesn't look like this has been implemented for JSON, unfortunately. Grrrr...

Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
(This post was last modified: 2013-03-26 15:26 by Milhouse.)
find quote
Milhouse Offline
Team-Kodi Member
Posts: 12,686
Joined: Jan 2011
Reputation: 626
Post: #58
(2013-03-26 15:07)MilhouseVH Wrote:  The trick is finding the trigger that causes XBMC to extract this information from specific media items - it certainly doesn't look like this has been implemented for JSON, unfortunately. Grrrr...

Here's the official answer - can't be done right now, and no easy solution. Sad

Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
find quote
un1versal Offline
Out of Memory (1939–2016)
Posts: 7,134
Joined: Oct 2012
Location: Binary Pulsar
Post: #59
I dont have iOS or android apps either Im in Linux Ubuntu. But yea sure is a shortcomming of JSON, but who knows if it will get implemented anytime soon.

Thanks anyway for looking into this and for prompt replies. Fantastic Wink

While Im here would make some suggestion, it would be great if this was a addon in XBMC which one could use with a remote/keyboard via UI or even better if it was part of a web interface like XWMM.
Perhaps you can even consider using github for development/delivery of anything like this as well...

Thanks for tool anyway though haven't used it I can see this being a part of my maintenance tools for XBMC in future.

uNi
(This post was last modified: 2013-03-26 16:17 by un1versal.)
find quote
Milhouse Offline
Team-Kodi Member
Posts: 12,686
Joined: Jan 2011
Reputation: 626
Post: #60
Thanks for your kind words, however if you ran the script you might realise why it won't be an addon any time soon! Smile

I'm much happier with it at the command line for now, and I'm really not sure if I'd be able to create a suitable gui for this, plus half it's current functionality would most likely be lost. Hope you find a use for it soon, I'd be interested in your feedback.

I do intend to put it on github eventually.

Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
find quote
Post Reply