Guest - Testers are needed for the reworked CDateTime core component. See... https://forum.kodi.tv/showthread.php?tid=378981 (September 29) x
  • 1
  • 110
  • 111
  • 112(current)
  • 113
  • 114
  • 197
[RELEASE] Texture Cache Maintenance utility
(2015-09-25, 23:26)Milhouse Wrote: To re-cache already cached artwork you need to use the "C" option.

OK:
Code:
./texturecache.py C

Quote:I use Universal Movie Scraper and The TVDB, and never had a problem.
me too

Quote:My as.xml settings are:
Code:
<fanartres>1080</fanartres>
<imageres>720</imageres>
which results in good quality cached artwork (my clients are a mixture of RPi and x86 - all source artwork is high quality).

The quality of cached artwork also has an impact on the quality (and storage requirements) of artwork displayed on tablet devices, eg. Yatse, Kore etc.
OK, I set 720 for imageres Big Grin


SOLVED: Thank you very much. :*
Reply
@Milhouse, Question regarding multiple Pi's.

Usually, when I finish a vscan on a new movie, i'll run remotely:
Code:
tc1 exec script.artwork.downloader silent=true mediatype=movie dbid=xxx
followed by
Code:
tc1 c movies <moviename> && tc2 c movies <moviename>
and finally
Code:
tc1 exec script.artworkorganizer silent=true && tc2 exec script.artworkorganizer silent=true

I obviously only need to run AD on one Pi because it puts new lines in MySQL, but caching and Artwork Organizer are specific to each Pi. So aside from the fact that I use an alias for ./texturecache.py @section=xxx, I was wondering if there is a more efficient way to run the commands here all at once, or do I need the && to separate each command. I only ask because of your impressive knowledge of the command line. Hope you can help.
Reply
Yes you need to run them as two separate commands.

If you wanted to reduce the duration of the whole process you could consider processing each client in parallel, eg.:

Code:
(tc1 c movies <moviename> && tc1 exec ...) &
(tc2 c movies <moviename> && tc2 exec ...) &

Although caching specific movies is usually pretty quick, I've no idea how long Artwork Organizer takes to finish.

Knowing when a script like AD or AO has finished can also be a challenge as the JSON exec method isn't synchronous, unfortunately... If you need to wait until an add-on has finished before continuing with another stage of the process then tailing kodi.log is often the only solution and is best run directly on the remote client (using scp/ssh etc.)
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2015-09-26, 02:06)Milhouse Wrote: Yes you need to run them as two separate commands.

If you wanted to reduce the duration of the whole process you could consider processing each client in parallel, eg.:

Code:
(tc1 c movies <moviename> && tc1 exec ...) &
(tc2 c movies <moviename> && tc2 exec ...) &

Although caching specific movies is usually pretty quick, I've no idea how long Artwork Organizer takes to finish.

Knowing when a script like AD or AO has finished can also be a challenge as the JSON exec method isn't synchronous, unfortunately... If you need to wait until an add-on has finished before continuing with another stage of the process then tailing kodi.log is often the only solution and is best run directly on the remote client (using scp/ssh etc.)

Yeah, AD is the only one that is a real concern as far as stacking the commands. That one Kicks back to the prompt after starting, so may be an issue running another right behind it. AO is the last command, so not a big issue there. I'll play with Parallel though, not sure exactly how you did that. I've run two terminal windows at the same time, each caching the entire movie library on a different pi, but never tried parallel in the same window. Thanks for getting me on the right track, i'll check it out.

edit: and unfortunately I have to run AD first, because texture cache has no logo/clearart/disc otherwise.
edit2: ok, I see what you did there. Running each line in the background.
Reply
Just wanna say thank you Milhouse this script helps speeding up my huge library Big Grin

Code:
The following items could not be downloaded:

[banner    ] [Be Arrogant                             ] http://thetvdb.com/banners/graphical/294951-g.jpg
[fanart    ] [Could We Love?                          ] http://thetvdb.com/banners/fanart/original/276373-1.jpg
[fanart    ] [Freezing                                ] http://thetvdb.com/banners/fanart/original/81376-2.jpg
[fanart    ] [Room Alone 401-410                      ] http://thetvdb.com/banners/fanart/original/293955-1.jpg
[fanart    ] [Scrap Teacher                           ] http://thetvdb.com/banners/fanart/original/264145-1.jpg
[fanart    ] [Water Boys                              ] http://thetvdb.com/banners/fanart/original/82330-1.jpg
[fanart    ] [Why Does 16-Year-Old Toudouin Seiya Have] http://thetvdb.com/banners/fanart/original/291367-1.jpg
[poster    ] [Goosebumps                              ] http://thetvdb.com/banners/posters/78790-4.jpg
[poster    ] [Naniwa Shonen Tanteidan                 ] http://thetvdb.com/banners/posters/280115-1.jpg

Cache pre-load activity summary for "addons/agenres/vgenres/albums/artists/movie
s/sets/tvshows":

              |    banner   |    fanart   |  landscape  |    poster   |  seasonposter|    thumb    |  thumbnail  |    TOTAL
              |-------------+-------------+-------------+-------------+--------------+-------------+-------------+-------------+
Cached        |     256     |      15     |      -      |     5755    |       -          |     4104    |      1      |    10131
Deleted       |      -      |      -      |      -      |      -      |       -      |      -      |      -      |      0
Duplicate     |    39279    |    40316    |      14     |    40480    |       14     |     1737    |      7      |    121847
Error         |      1      |      9      |      -      |      8      |       -      |      -      |      -      |      18
Ignored       |      -      |      -      |      -      |      -      |       -      |     5032    |      -      |     5032
Season-all    |      -      |      -      |      -      |      -      |       -      |      -      |      -      |      0
Skipped       |     1500    |     9411    |      1      |     5625    |       1      |    24809    |      67     |    41414
Undefined     |      -      |      73     |      -      |      -      |       -      |      -      |      63     |     136
===============================================================================================================================
TOTAL         |    41036    |    49824    |      15     |    51868    |       15     |    35682    |     138     |    178578
Download Time | 00:06:28.43 | 00:00:21.25 |      -      | 00:10:36.93 |       -     | 01:46:05.41 | 00:00:02.04 | 05:34:24.21

  Threads Used: 8
   Min/Avg/Max: 01.00 / 00.51 / 02.00 downloads per second
   Min/Avg/Max: 01.54 / 03.95 / 50.03 seconds per download

       Loading: 00:39:03.45
       Parsing: 00:00:12.39
     Comparing: 00:00:41.54
   Downloading: 05:34:24.21
TOTAL RUNTIME: 06:14:22.31

So i have to run this script everytime the library gets updated right?

btw i haven't tested this for diskless though, this is for local installation.. I'll report back once i've tried it in diskless
Reply
Hmm something is strange milhouse, when i first run Kodi after running your script, the thumbs load alot faster....
Then i close it, reopen it, and browse my collection to test the posters load speed,
And now the thumbs in my home screen won't load at all... any idea why?
I put Container.IsUpdating and it keeps spinning...

Don't see anything strange on the log... Just long load times...

http://pastebin.com/ch2tHD0t

well it's finally loading the thumbs, but it takes much longer time than before i use the script.. almost 3 mins
Edit : Sorry was in another PC, fixed it
Reply
Don't upload logs to the forum! Use a paste site (pastebin.com, xbmclogs.com) and paste the link - please remove your log from the forum (which is unreadable, and now makes browsing this thread on mobile devices almost impossible). Also, upload the full log to the paste site - you've left out important information from your forum pasted log.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2015-09-26, 12:18)denywinarto Wrote: Hmm something is strange milhouse, when i first run Kodi after running your script, the thumbs load alot faster....
Then i close it, reopen it, and browse my collection to test the posters load speed,
And now the thumbs in my home screen won't load at all... any idea why?
I put Container.IsUpdating and it keeps spinning...

Don't see anything strange on the log... Just long load times...

http://pastebin.com/ch2tHD0t

well it's finally loading the thumbs, but it takes much longer time than before i use the script.. almost 3 mins

It seems to be taking an extraordinarily long time to load images from the cache. Is there a performance problem with the PC (not enough RAM, too slow a disk etc.)?

A few movie posters and fanarts are still being cached, which is odd - they should have been cached by texturecache.py.

It does just seem to be that you have a problem retrieving artwork from the cache, which is basically a SQLite database lookup followed by a read from your filesystem, so it shouldn't be slow. I'd check the rest of your system.

Maybe also try a current build of Kodi 15.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2015-09-26, 12:59)Milhouse Wrote:
(2015-09-26, 12:18)denywinarto Wrote: Hmm something is strange milhouse, when i first run Kodi after running your script, the thumbs load alot faster....
Then i close it, reopen it, and browse my collection to test the posters load speed,
And now the thumbs in my home screen won't load at all... any idea why?
I put Container.IsUpdating and it keeps spinning...

Don't see anything strange on the log... Just long load times...

http://pastebin.com/ch2tHD0t

well it's finally loading the thumbs, but it takes much longer time than before i use the script.. almost 3 mins

It seems to be taking an extraordinarily long time to load images from the cache. Is there a performance problem with the PC (not enough RAM, too slow a disk etc.)?

A few movie posters and fanarts are still being cached, which is odd - they should have been cached by texturecache.py.

It does just seem to be that you have a problem retrieving artwork from the cache, which is basically a SQLite database lookup followed by a read from your filesystem, so it shouldn't be slow. I'd check the rest of your system.

Maybe also try a current build of Kodi 15.

I have a few movies and series not yet scanned.. Does that have any effect?

Hmm my PC has 2Gb of RAM, no VGA,
Processor is Core 2 Duo CPU @2.66 Ghz (2 CPUs), ~2.7 Ghz
The HDD is fairly new, and i just did chkdsk few days ago so i doubt that's the problem...

Can't really upgrade my specs though Big Grin

So it's performance problem ? I'm using custom skin as well (modded Ace)
Can wrongly coded skin cause this?

Was pretty excited because i thought got faster..
About Kodi 15, gonna be a little difficult to try it because it might break something in my custom skin...

I think my PC can still handle it.. In task manager Kodi only tops 40-50% CPU
Reply
(2015-09-26, 13:06)denywinarto Wrote: I have a few movies and series not yet scanned.. Does that have any effect?

It shouldn't do, the problem seems to be the amount of time it takes to retrieve artwork from the cache.

(2015-09-26, 13:06)denywinarto Wrote: Hmm my PC has 2Gb of RAM, no VGA,

What does "no VGA" mean - you have a graphics card, right? Maybe Kodi is doing all the artwork decode in software.

(2015-09-26, 13:06)denywinarto Wrote: So it's performance problem ? I'm using custom skin as well (modded Ace)

On the face of it, yes. It normally takes less than 100ms-150ms to load an item of artwork from the cache, but in your case it's often taking 500ms, and even as long as 11723 milliseconds (that's 11.7 seconds!)

Here are the attempts to load artwork from your cache, sorted into ascending order of time taken: http://sprunge.us/DMRN

What size artwork have you cached, perhaps if you've cached 4K artwork that might explain it... particularly if it's all software decoded.

(2015-09-26, 13:06)denywinarto Wrote: Can wrongly coded skin cause this?

Was pretty excited because i thought got faster..
About Kodi 15, gonna be a little difficult to try it because it might break something in my custom skin...

It seems unlikely, but it would be worth testing with stock Confluence just to rule out the third-party skin.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2015-09-26, 13:24)Milhouse Wrote:
(2015-09-26, 13:06)denywinarto Wrote: I have a few movies and series not yet scanned.. Does that have any effect?

It shouldn't do, the problem seems to be the amount of time it takes to retrieve artwork from the cache.

(2015-09-26, 13:06)denywinarto Wrote: Hmm my PC has 2Gb of RAM, no VGA,

What does "no VGA" mean - you have a graphics card, right? Maybe Kodi is doing all the artwork decode in software.

(2015-09-26, 13:06)denywinarto Wrote: So it's performance problem ? I'm using custom skin as well (modded Ace)

On the face of it, yes. It normally takes less than 100ms-150ms to load an item of artwork from the cache, but in your case it's often taking 500ms, and even as long as 11723 milliseconds (that's 11.7 seconds!)

Here are the attempts to load artwork from your cache, sorted into ascending order of time taken: http://sprunge.us/DMRN

What size artwork have you cached, perhaps if you've cached 4K artwork that might explain it... particularly if it's all software decoded.

(2015-09-26, 13:06)denywinarto Wrote: Can wrongly coded skin cause this?

Was pretty excited because i thought got faster..
About Kodi 15, gonna be a little difficult to try it because it might break something in my custom skin...

It seems unlikely, but it would be worth testing with stock Confluence just to rule out the third-party skin.

Yeah, can't use any VGA unfortunately... There will be too many to replace Smile
I don't think there's any 4k artwork, i used filebot to automatically generate artworks to each movie folder in the library...
(Does that effect the caching somehow?)
And IMDB &thetvdb don't have 4k images IIRC

I forgot to mention, my NAS is structured like this :

NAS - Gigabit Switch - 20 clients

I suspected network traffic might have something to do with this,
so i tested it when the traffic is not crowded...
And yep.. the loading times seems to speed up a bit
http://pastebin.com/stXjSGRj
But it's still too slow i think..

One thing i notice i keep seeing this in the log, what does it mean?

XFILE::CDirectory::GetDirectory - Error getting

Can't really duplicate the test with confluence because the skin structure is very different for home menu..

Speaking of xbmc versions, i read some people complaining about performance of 15.1.. Has this been fixed?
Reply
(2014-01-17, 01:46)theowiesengrund Wrote: Another thing: Is there a specific reason, that you don't support tvshows for the imdb function?

Version 2.1.6:

* Add: imdb support for tvshows, ie. imdb tvshows [showname]. The usable fields for tvshows and episodes are votes, ratings, plot (typically an overview at the tvshow level, detail at the episode level) and genre (which is set only at the tvshow level). Configure the fields to be set using @imdb.fields.tvshows, eg. @imdb.fields.tvshows=plot,votes,ratings. The default fields are votes and ratings.

This facility uses the title and year of each tvshow to query http://omdbapi.com so it can fail if the tvshow title or year differs from that held by imdb.com. In order to overcome this potential mismatch, use @imdb.translate.tvtitles to translate your tvshow titles to what imdb.com/omdbapi.com expects

You may also use @imdb.translate.tvyears to temporarily alter the year when a tvshow first aired, for when your library uses a different date to that in use by IMDb/OMDb.

Finally, use imdb.ignore.tvtitles to disable the lookup of those shows that will never be fixed or added to omdabpi.com.

All of the above settings accept a list of regex patterns (case insensitive, delimited by pipe character). In the case of the translate options, a replacement value will follow the equal sign (either a year, or a new string - a blank string essentially means the matched string is removed). Note also that the patterns for imdb.ignore.tvtitles and imdb.translate.tvyears will be matched against the original tvshow title and not the temporarily translated title.

Examples:
Code:
imdb.ignore.tvtitles = ^Brass Eye|^Connections|^Dynamo: Magician Impossible|^Fresh Meat|^Hamish Macbeth|^The Mighty Boosh|^The Killing|^Sanctuary|^Outnumbered
imdb.translate.tvtitles = ^Frank Herbert's=|^Marvel's=|Hitchhiker's=Hitch Hikers
imdb.translate.tvyears = ^Battlestar Galactica=1978|^Wallander$=2005

With the above imdb.translate.tvtitles example: "Frank Herbert's Children of Dune" will be temporarily changed to "Children of Dune", "Marvel's Agent Carter" and "Marvel's Daredevil" to "Agent Carter" and "Daredevil" respectively, and "The Hitchhiker's Guide to the Galaxy" as "The Hitch Hikers Guide to the Galaxy".

In the case of imdb.translate.tvyears, the tvshow "Battlestar Galactica" (aka "Galactica 1980") will have its year temporarily changed from 1980 to 1978 when querying omdbapi.com as the episode data for "Galactica 1980" (by which it is known on tvdb.com) is listed under "Battlestar Galactica (1978)" on imdb.com/omdbapi.com. As for Wallander, I have two Wallander tvshows: "Wallander" (the original Swedish, with the incorrect year - 2006 instead of 2005) and "Wallander (UK)" (the UK 2008 remake). The ^Wallander$ pattern ensures that the year is changed only for the Swedish tvshow.

One new setting that applies to movies and episodes: If you only wish to process movies or episodes added to the media library within a specific period then set @imdb.period=#, eg. imdb.period=30 would consider only movies or episodes added during the previous 30 days. There is no default value in which case all movies/episodes will be considered.

See the logfile for additional information relating to omdbapi.com query failures. When a tvshow can't be found on omdbapi.com it's almost always going to be due to a mismatched title and/or year. When an episode can't be found - but the tvshow does exist - then the episode is simply not available on omdbapi.com - many series appear to be incomplete, sometimes missing just one or two episodes, sometimes almost all episodes may be missing. It's possible a missing tvshow or the missing episodes may be added to omdbapi.com in the future, however if there is a lot of missing episodes for an old tvshow the chances are slim of (IMHO) of this happening in which case you have the option to ignore the tvshow entirely by adding a suitable pattern to imdb.ignore.tvtitles.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
I was hoping to run this using pythonw.exe in Windows instead of python.exe, but it seems to not run properly. When running this with python.exe it steals focus of XBMC which is very annoying. Has this issue been looked at in the past?
Reply
Nope. Not sure what you mean by "steals focus", what command are you running? Most of the script options have no interaction with the Kodi GUI (at least, not directly).
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
I run your program to update thumbnails for movies/tv when my computer wakes up from sleep, so it can cache newly added items. Kodi is always open, so it has initial focus. When I run the python script via the task scheduler, it minimizes Kodi and shows the python console. I've tried running it as "minimized" in the scheduler, tried running a vbs script with different window settings to keep it minimized, etc, but every time it steals focus and minimizes Kodi.

pythonw.exe would launch your script asynchronously with no output, but I believe your script needs a no output option for that to work properly. If you try to write to stdout/stderr/etc I believe it causes pythonw to silently fail.
Reply
  • 1
  • 110
  • 111
  • 112(current)
  • 113
  • 114
  • 197

Logout Mark Read Team Forum Stats Members Help
[RELEASE] Texture Cache Maintenance utility17
This forum uses Lukasz Tkacz MyBB addons.