• 1
  • 105
  • 106
  • 107(current)
  • 108
  • 109
  • 197
[RELEASE] Texture Cache Maintenance utility
(2015-08-17, 03:50)Milhouse Wrote:
(2015-08-17, 03:29)raspberry_pd Wrote: Some pages in your GitHub repo? Maybe?!!

Kodi Wiki would be more appropriate.

It's unlikely I'll be adding significantly to the documentation for this utility, beyond what already exists. Most people seem able to work out what to do from the built-in help and existing documentation, and you can always look at the logfile to see the JSON requests that are being generated, and if you have a JSON problem then refer to the JSON-RPC documentation for help on what is valid and what isn't. Failing that, ask a question in the forum.

In the case of your YouTube url, all I did was enable debug logging in Kodi and play a random video using the YouTube addon, then extracted the most likely looking url and tried it with texturecache.py play - it worked. I then modified the url so that it played your video. Repeating this for any other addon is probably the easiest solution when trying to work out what the video url needs to be.

Fair enough. Hope my response didn't come across as implying documentation for this utility is insufficient. I'm sure it could be expanded upon but I think it's pretty good at the moment.
Reply
Hi Milhouse, I need your help Big Grin

My system configuration: Raspberry Pi2 with Openelec and Aeon Nox theme;
all media file stored into a Synology Nas (MySQL database created with MariaDB and phpMyAdmin).

From when I transfered all into the NAS, Kodi is became slow to load images.
I discovered your script: I installed it, I activated the webserver on port 8080 (instead of the default 80), reboot kodi and gave it a try.

With jd option I seen that all the images are downloaded every time from Internet:
I used your avatar example, but the situation is the same for all my media files (audio and movies).

Code:
OpenELEC:~ # ./texturecache.py jd movies avatar
[
  {
    "art": {
      "fanart": "image://http://assets.fanart.tv/fanart/movies/19995/moviebackground/avatar-4fec1d99872da.jpg/",
      "poster": "image://http://image.tmdb.org/t/p/original/44aaqZf0BfT5bdtIMFHgT5VYOCQ.jpg/"
    },
    "file": "nfs://192.168.1.68/volume1/FILM/Avatar - Extended Cut.mkv",
    "label": "Avatar",
    "movieid": 2051,
    "title": "Avatar"
  }
]

Into my media NAS folder there aren't the .nfo and .jpg files.
Maybe for this reason it grab all from the Web.

So I tryed

Code:
OpenELEC:~ # ./texturecache.py nc

but nothing changed (Kodi is still slow and no info and image are saved into nas directory):

Code:
|    banner   |    fanart   |    poster   |    thumb    |  thumbnail  |    TOTAL
--------------+-------------+-------------+-------------+-------------+-------------+-------------
Cached        |      -      |      -      |      -      |      -      |      -      |      0
Deleted       |      -      |      -      |      -      |      -      |      -      |      0
Duplicate     |     868     |     1929    |     861     |      -      |      58     |     3716
Error         |      -      |      -      |      -      |      -      |      -      |      0
Ignored       |      -      |      -      |      -      |      -      |      36     |      36
Not in Cache  |      32     |      3      |      13     |     378     |     790     |     1216
Season-all    |      -      |      -      |      5      |      -      |      -      |      5
Skipped       |      20     |     788     |     526     |     427     |     700     |     2461
Undefined     |      -      |     284     |      -      |      -      |     359     |     643
==================================================================================================
TOTAL         |     920     |     3004    |     1405    |     805     |     1943    |     8077

I used also the "prune" command (it delete a lot of things, but nothing).
I don't know what to do now.
Thx for any help
:*
Reply
jd doesn't cache your images. You need to use C option.
Image Image
Reply
(2015-08-20, 01:05)redglory Wrote: jd doesn't cache your images. You need to use C option.

No no, the "C" option will force everything item to be re-cached!

(2015-08-20, 00:56)lollo78 Wrote: Hi Milhouse, I need your help Big Grin

Try "./texturecache.py c" - this will cache only the remaining 1216 artwork items that are not yet in your cache.

If the RPi is still slow once the texture cache is full populated then your problem is due to something else, maybe a very slow SD card?

(2015-08-20, 00:56)lollo78 Wrote: Into my media NAS folder there aren't the .nfo and .jpg files.
Maybe for this reason it grab all from the Web.

Yes, that would be the reason for Kodi using remote artwork urls.

(2015-08-20, 00:56)lollo78 Wrote: but nothing changed (Kodi is still slow and no info and image are saved into nas directory):

I'm not sure if you're expecting texturecache.py to create NFO and image files in your NAS directory - it won't do that.

If you want to convert your remote artwork into local artwork, then use mklocal.py in conjunction with texturecache.py (see earlier posts for details on mklocal.py).
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Hi guys, with

Code:
./texturecache.py c

now I have this:

Code:
|    banner   |    fanart   |    poster   |    thumb    |  thumbnail  |    TOTAL
--------------+-------------+-------------+-------------+-------------+-------------+-------------
Cached        |      32     |      1      |      12     |     374     |     783     |     1202
Deleted       |      -      |      -      |      -      |      -      |      -      |      0
Duplicate     |     868     |     1929    |     861     |      -      |      58     |     3716
Error         |      -      |      2      |      1      |      -      |      2      |      5
Ignored       |      -      |      -      |      -      |      -      |      36     |      36
Season-all    |      -      |      -      |      5      |      -      |      -      |      5
Skipped       |      20     |     788     |     526     |     431     |     705     |     2470
Undefined     |      -      |     284     |      -      |      -      |     359     |     643
==================================================================================================
TOTAL         |     920     |     3004    |     1405    |     805     |     1943    |     8077
Download Time | 00:00:06.16 | 00:00:01.54 | 00:00:01.56 | 00:01:20.06 | 00:00:00.12 | 00:06:09.54

with
Code:
./texturecache.py p
error are solved, but the others "issues" remains:

Code:
./texturecache.py nc

Code:
|    banner   |    fanart   |    poster   |    thumb    |  thumbnail  |    TOTAL
--------------+-------------+-------------+-------------+-------------+-------------+-------------
Cached        |      -      |      -      |      -      |      -      |      -      |      0
Deleted       |      -      |      -      |      -      |      -      |      -      |      0
Duplicate     |     868     |     1929    |     861     |      -      |      58     |     3716
Error         |      -      |      -      |      -      |      -      |      -      |      0
Ignored       |      -      |      -      |      -      |      -      |      36     |      36
Not in Cache  |      -      |      3      |      1      |      -      |      3      |      7
Season-all    |      -      |      -      |      5      |      -      |      -      |      5
Skipped       |      52     |     788     |     538     |     805     |     1487    |     3670
Undefined     |      -      |     284     |      -      |      -      |     359     |     643
==================================================================================================
TOTAL         |     920     |     3004    |     1405    |     805     |     1943    |     8077


How to manage Ignored, Not in cache Skipped and Undefined?
This voice means that is not all cached, right?


now if I run
Code:
./texturecache.py jd movies avatar
I have always
Code:
[
  {
    "art": {
      "fanart": "image://http://assets.fanart.tv/fanart/movies/19995/moviebackground/avatar-4fec1d99872da.jpg/",
      "poster": "image://http://image.tmdb.org/t/p/original/44aaqZf0BfT5bdtIMFHgT5VYOCQ.jpg/"
    },
    "file": "nfs://192.168.1.68/volume1/FILM/Avatar - Extended Cut [BDRip-1080p-Ita-Eng-Sub-Ita-Eng-Ara-Chapters][RiP By MaX].mkv",
    "label": "Avatar",
    "movieid": 2051,
    "title": "Avatar"
  }
]

so, mklocal.py, seems to be exactly wath I need.
I just installed it and I searched in all the thread to understand how to use it, but I'm a little confused. Confused
I have to simply run
Code:
./mklocal.py --readonly

and after

Code:
/mklocal.py --nokeep
?
Reply
"Error" corresponds to those items that were unable to be downloaded (you should have seen those listed). Not much you can do about those as they're probably no longer available remotely, so you need to use alternative artwork - your best bet would be to download alternative artwork localy and then replace the missing artwork in your library via the GUI (as we're only talking about 5 items, until you're up to speed with mklocal.py which would do this for you automatically).

"Ignored" items are those artwork types that were not in your @cache.artwork list. If you add @logfile=tc.log" it will log detailed information during the run which will include details of the ignored artwork (search for "Ignored") - not sure why it should ignore thumbnails though, upload your log somewhere if you need further help.

Edit: Oh yes, there's a second reason to ignore items - if the artwork matches an entry in "cache.ignore.types" (this defaults to "^video, ^music") the item will be ignored. The default settings avoid trying to cache embedded video or music artwork, as these can't be cached - thumbnail extraction is a GUI process and not supported by the web server which is being used here to cache artwork. Again, this is logged (search for "Ignored").

For mklocal.py, maybe start with this post - some of the arguments might have changed, but hopefully you should get the gist of it. Be aware that it will download new artwork unless you add the --readonly argument. Nothing will be changed in your media library unless you process the output from mklocal.py through "texturecache.py set".
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2015-08-20, 22:00)Milhouse Wrote: If you add @logfile=tc.log" it will log detailed information during the run which will include details of the ignored artwork (search for "Ignored") - not sure why it should ignore thumbnails though, upload your log somewhere if you need further help.

After
Code:
./texturecache.py c @logfile=tc.log

errors are still there, so don't consider my previous post edit.

Code:
|    banner   |    fanart   |    poster   |    thumb    |  thumbnail  |    TOTAL
--------------+-------------+-------------+-------------+-------------+-------------+-------------
Cached        |      -      |      1      |      -      |      -      |      1      |      2
Deleted       |      -      |      -      |      -      |      -      |      -      |      0
Duplicate     |     868     |     1929    |     861     |      -      |      58     |     3716
Error         |      -      |      2      |      1      |      -      |      2      |      5
Ignored       |      -      |      -      |      -      |      -      |      36     |      36
Season-all    |      -      |      -      |      5      |      -      |      -      |      5
Skipped       |      52     |     788     |     538     |     805     |     1487    |     3670
Undefined     |      -      |     284     |      -      |      -      |     359     |     643
==================================================================================================
TOTAL         |     920     |     3004    |     1405    |     805     |     1943    |     8077
Download Time |      -      | 00:00:01.55 | 00:00:01.57 |      -      | 00:00:00.02 | 00:00:03.91

Please, could you download my tc.log from
https://www.dropbox.com/s/ndu22pyml90xzy7/tc.zip?dl=0?
Thx

Quote:For mklocal.py, maybe start with this post
Thx a lot, I read it and I do some test.
Reply
(2015-08-20, 22:29)lollo78 Wrote: Please, could you download my tc.log from

See edit to my previous post - it's due to embedded music artwork, eg:
Code:
2015-08-20 22:20:01.148861:MainThread: Ignored [thumbnail   ] image due to [^music] rule: music@nfs://192.168.1.68/volume1/music/MP3/Robben Ford/2014 - A Day In Nashville/01 Green Grass Rainwater.mp3
2015-08-20 22:20:01.200466:MainThread: Ignored [thumbnail   ] image due to [^music] rule: music@nfs://192.168.1.68/volume1/music/FLAC/Scorpions/1980 - Animal Magnetism/01 - Make It Real.flac
2015-08-20 22:20:01.206941:MainThread: Ignored [thumbnail   ] image due to [^music] rule: music@nfs://192.168.1.68/volume1/music/FLAC/Steve Vai/2000 - The 7th Song/01 For the Love of God.flac
etc.

This type of artwork can only be extracted (and eventually cached) by a GUI process, ie. navigate using the GUI to the item in your library and display it - the artwork should be extracted/cached.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
OK, I'll try to solve this issue via GUI. Thx a lot.
And what is Skipped and Undefined? It's an issue?

For mklocal this is right?
For example, if I want to save all movie artworks directly inside my Synology NAS film directory (with all the MKV files inside the FILM directory):
Code:
./texturecache.py jd movies | mklocal.py --local /volume1/FILM/ --prefix nfs://192.168.1.68/volume1/FILM/ --artwork fanart poster clearlogo clearart --output fixup.dat

and after
Code:
cat fixup.dat | ./texturecache.py set

For tv show (stored in SERIE TV folder like Series_name with Seasons_subfolder)?!? Rolleyes
Code:
./texturecache.py jd tvshow | mklocal.py -s --local /volume1/SERIE\040TV/ --prefix nfs://192.168.1.68/volume1/SERIE\040TV/ --season --artwork fanart poster clearlogo clearart --output fixup.dat

If I undestood well, mklocal works only for movies and tv show and not for music, right? Confused
Reply
(2015-08-20, 23:27)lollo78 Wrote: For mklocal, I'm unable to find the local mounting point (Openelec). Blush

For local mounting point, this needs to be a path to your media files that the OS can access. If you're using OpenELEC, you'll need to add an NFS mount point and use that to access your media files over NFS. Or, as I'm assuming you're not posting this from OpenELEC, you can use your current PC to run mklocal.py assuming that has access to your media files (if it's a Windows PC, your --local path would be something like "E:\FILM", assuming "E:\" is mapped to "\\192.168.1.68\volume1").

(2015-08-20, 23:27)lollo78 Wrote: mklocal works only for movies and tv show, right? Not for music? Frown

Movies, TVShows and Movie Sets only.

Music (albums, artists, songs) is not currently supported, maybe one day.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2015-08-20, 01:39)Milhouse Wrote:
(2015-08-20, 01:05)redglory Wrote: jd doesn't cache your images. You need to use C option.

No no, the "C" option will force everything item to be re-cached!

(2015-08-20, 00:56)lollo78 Wrote: Hi Milhouse, I need your help Big Grin

Try "./texturecache.py c" - this will cache only the remaining 1216 artwork items that are not yet in your cache.

If the RPi is still slow once the texture cache is full populated then your problem is due to something else, maybe a very slow SD card?

(2015-08-20, 00:56)lollo78 Wrote: Into my media NAS folder there aren't the .nfo and .jpg files.
Maybe for this reason it grab all from the Web.

Yes, that would be the reason for Kodi using remote artwork urls.

(2015-08-20, 00:56)lollo78 Wrote: but nothing changed (Kodi is still slow and no info and image are saved into nas directory):

I'm not sure if you're expecting texturecache.py to create NFO and image files in your NAS directory - it won't do that.

If you want to convert your remote artwork into local artwork, then use mklocal.py in conjunction with texturecache.py (see earlier posts for details on mklocal.py).
Sorry i misread his post. I thought he was trying to re cache! Smile
Image Image
Reply
I found the NAS mounting point:
Code:
/volume1/xxx/
but, as you know, I'm searching to get a full speed GUI (no lag when browsing media libray), so if I use mklocal to save all the artwork on the NAS, what I expect about performance? I think that local storage (microSD) is better (mine is a Samsung Evo 32GB class 10) and if all the thumbnail remains around 3 GB, I can archive it on the microSD.

Actually my
Code:
/userdata/Thumbnails
folder is about 1,5 GB, but I doubt that is used well for the cache.
I gave a look inside this folder. It's full of actor images, shitty addon images (damned videodevil Rofl), music artists, movies posters (I don't see fanart)...

but if I use texturecache on a movie, ad example:

Code:
OpenELEC:~ # ./texturecache.py jd movies "come cani e gatti"
[
  {
    "art": {
      "fanart": "image://http://image.tmdb.org/t/p/original/2GoKwjA0K7b3BnsvBXwm68PkaJk.jpg/",
      "poster": "image://http://image.tmdb.org/t/p/original/lkvYSafgMWo1n6kjwPm9qeLiCuw.jpg/"
    },
    "file": "nfs://192.168.1.68/volume1/FILM/Come cani e gatti.mkv",
    "label": "Come cani e gatti",
    "movieid": 1743,
    "title": "Come cani e gatti"
  }
]

I don't know why it search the poster from the web, when the same image is inside my Thumbnail folder.
This happens with all movies.
In the past I tested other media scraper, maybe this compromised the right system working. Shocked

Maybe I have to delete all the Thumbnails folder and recreate it in the right way? But how?
Thanks a lot for all the help :*

EDIT: I looked also into my /userdata/Database folder. There are something missing?
I have this:
CDDB (folder)
Addons16.db
Addons19.db
Addons19.db-journal
Epg8.db
Texture13.db
TV26.db
ViewModes.db
Reply
(2015-08-21, 20:00)lollo78 Wrote: but, as you know, I'm searching to get a full speed GUI (no lag when browsing media libray), so if I use mklocal to save all the artwork on the NAS, what I expect about performance?

Once artwork is cached, whether it's local or remote artwork, the performance of the client will be the same.

The time taken to cache (or scrape, when re-scraping your library) will be quicker with local artwork. Local artwork also makes more sense when you have multiple clients as it avoids repeated downloads, consistent artwork on all clients (a client could display different artwork if between library scrapes some posters or fanart became more or less popular on the remote artwork website, returning different results to those shown by your other clients), and eliminates the risk of the remote artwork disappearing entirely (in a MySQL environment potentially leaving the client unable to display the specified artwork).

(2015-08-21, 20:00)lollo78 Wrote: I think that local storage (microSD) is better (mine is a Samsung Evo 32GB class 10) and if all the thumbnail remains around 3 GB, I can archive it on the microSD.

Not really sure what you mean by "archive it on the microSD". Having the Thumbnails folder on local storage is definitely best for performance. Your original artwork should ideally be stored locally on your NAS and will only be accessed once from the NAS when it is first cached, after which point the artwork will always be displayed using the cached version.

(2015-08-21, 20:00)lollo78 Wrote: Actually my
Code:
/userdata/Thumbnails
folder is about 1,5 GB, but I doubt that is used well for the cache.
I gave a look inside this folder. It's full of actor images, shitty addon images (damned videodevil Rofl), music artists, movies posters (I don't see fanart)...

Yep, that's your cached artwork - your fanart will be in there. Unsure what you mean by "I doubt that is used well for the cache".

The artwork in the cache will be using an alternative filename - use "texturecache.py s <yourfanartname>.jpg" to view the row in the Textures13.db, which will reveal the filename (cachedurl) as it exists in the Thumbnails cache folder.

You can use "texturecache.py P" to prune the texture cache of unused artwork items and keep the size of the texture cache in check.

(2015-08-21, 20:00)lollo78 Wrote: but if I use texturecache on a movie, ad example:

Code:
OpenELEC:~ # ./texturecache.py jd movies "come cani e gatti"
[
  {
    "art": {
      "fanart": "image://http://image.tmdb.org/t/p/original/2GoKwjA0K7b3BnsvBXwm68PkaJk.jpg/",
      "poster": "image://http://image.tmdb.org/t/p/original/lkvYSafgMWo1n6kjwPm9qeLiCuw.jpg/"
    },
    "file": "nfs://192.168.1.68/volume1/FILM/Come cani e gatti.mkv",
    "label": "Come cani e gatti",
    "movieid": 1743,
    "title": "Come cani e gatti"
  }
]

I don't know why it search the poster from the web, when the same image is inside my Thumbnail folder.

This happens with all movies.

That's how it works - your media library references the remote (or, ideally, local) artwork, and when you display this artwork for the first time (or scrape it) on the client the artwork is downloaded, reformatted, resized, optimised etc., and then stored in the texture cache (a row is added to Textures13.db, an image written into the Thumbnails folder).

Next time you display the artwork it will be retrieved directly from the cache, and not the remote web site or locally from your NAS.

(2015-08-21, 20:00)lollo78 Wrote: Maybe I have to delete all the Thumbnails folder and recreate it in the right way? But how?
Thanks a lot for all the help :*

No, your Thumbnails folder is fine, you just need to understand the caching process better.

(2015-08-21, 20:00)lollo78 Wrote: EDIT: I looked also into my /userdata/Database folder. There are something missing?

Can't see a problem - what do you think is missing? You don't have MyVideosXX.db or MyMusicXX.db which is normal when using MySQL/MariaDB.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
hi,
i am not sure if this is the right thread for my question...
the problem is, after a clean kodi installation the folder .../roaming/kodi (windows) is growing very fast up to 1.5 gb in 5 month. there are a lot of thumbnails. is there a tool available to delete all that unneeded files without to delete the needed thumbnails and data for viedo database (fanart, cover....). something like the "ccleaner" for windows? is this possible with "texture cache maintenance utility"?
Reply
(2015-08-22, 11:59)Ed76 Wrote: is this possible with "texture cache maintenance utility"?

Yep - the "prune" option:
Code:
texturecache.py P
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
  • 1
  • 105
  • 106
  • 107(current)
  • 108
  • 109
  • 197

Logout Mark Read Team Forum Stats Members Help
[RELEASE] Texture Cache Maintenance utility17