Kodi Community Forum
[RELEASE] Texture Cache Maintenance utility - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Supplementary Tools for Kodi (https://forum.kodi.tv/forumdisplay.php?fid=116)
+--- Thread: [RELEASE] Texture Cache Maintenance utility (/showthread.php?tid=158373)



RE: [RELEASE] Texture Cache Maintenance utility - bit phreak - 2013-11-19

(2013-11-19, 22:21)MilhouseVH Wrote:
(2013-11-19, 20:16)bit phreak Wrote: then i tried the R command (not quite sure of what it is doing) but anyway....
this is the result i got


./texturecache.py R
Found 31,197 orphaned files for 2,621 database files.

This is far too many orphaned files for this number of
database files - more than 5% - and something may be wrong.

Check your configuration, database, and Thumbnails folder.

Add "orphan.limit.check = no" to the properties file
if you wish to disable this check.



can some please explain in detail what the "R" does. and perhaps how I get around the error above?

The "R" command (it was originally named as it's a kind of reverse lookup) finds those files in your Thumbnails folder that are NOT referenced in your Textures13.db database - ie. they are those files that are serving no purpose whatsoever and just taking up disk space, as they're no longer being used by XBMC. I refer to these files as "orphans" as usually they would have been referenced by XBMC at some point, perhaps in an earlier version of XBMC (ie. Eden?) but are now no longer used or needed.

How you have managed to acquire so many orphans you don't say, but I imagine it's because you have upgraded from a much earlier version of XBMC (pre-Frodo).

As things stand, you currently have 31,197 orphaned files (that can be deleted) and only 2,621 non-orphan files (that you want to keep). Under normal circumstances, you may have only a handful of orphan files, if that. Hence the safety check to avoid a catastrophic coding error on my part (it happened once - oops!).

However if my explanation makes sense, and you can think of a reason why you would have so many orphan files, then add the "orphan.limit.check=no" property to your texturecache.cfg file, and run the "R" option again - this time it will remove all the orphan files.

Your answer makes perfectly sense!
I think that you may be right about all my orphaned files. it is properly because of a very old db and numerus upgrades.
anyway i've got a backup of everything so i can practically do anything i want, and still restore it if nessesary.

I did take a look at the configuration file.
but i thought that is was very confusing. I'm a windows user and opened it up in a webeditor. I couldnt pinpoint the exact location to put the command in the file.
can you tell me wich line number to put the command


RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2013-11-19

(2013-11-19, 22:48)bit phreak Wrote: I did take a look at the configuration file.
but i thought that is was very confusing. I'm a windows user and opened it up in a webeditor. I couldnt pinpoint the exact location to put the command in the file.
can you tell me wich line number to put the command

It doesn't matter where the property is added, although if you add it more than once the last entry will be used.

However I've updated my previous post so you don't need to add anything to the configuration file, just pass the property on the command line. Also, I don't recommend creating a configuration file at all unless you need to override default properties - the sample file on github is just an example, it's not what you're meant to install (I think I will change that by commenting out all the lines...).


RE: [RELEASE] Texture Cache Maintenance utility - bit phreak - 2013-11-19

(2013-11-19, 22:54)MilhouseVH Wrote:
(2013-11-19, 22:48)bit phreak Wrote: I did take a look at the configuration file.
but i thought that is was very confusing. I'm a windows user and opened it up in a webeditor. I couldnt pinpoint the exact location to put the command in the file.
can you tell me wich line number to put the command

It doesn't matter where the property is added, although if you add it more than once the last entry will be used.

However I've updated my previous post so you don't need to add anything to the configuration file, just pass the property on the command line. Also, I don't recommend creating a configuration file at all unless you need to override default properties - the sample file on github is just an example, it's not what you're meant to install (I think I will change that by commenting out all the lines...).


okay Smile

i will give it a spin.
thanks for your help. i really appreciate it

This is crazy shit!!
It worked perfect and i do now have about 5000 files left.and all the thumbs that was in use, is still there.
I will spread the word to my friends. I think some of them have the exact same problem


RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2013-11-19

If you run:
Code:
./texturecache.py f

you'll get a summary of how many files you have in your Thumbnails folder, and how much disk space is being used.


Album art file name other than folder.jpg or cover.jpg - epoke - 2013-11-21

Very helpful for my small-memory atv1. It works fine for movie and tvshow. Thanks for your work.
My music folder stucture is Music - AlbumArtist - Ablum - (CD#) - tracks...
I have album art file in each album folder, but file names are [AlbumArtist] - [Album].jpg, not folder.jpg or cover.jpg. And all tracks have embedded album art in tag.
./texturecache.py c Albums could not cache my album art, maybe album art file name is issue, right?
How can I cache album arts without renaming album art file name?


RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2013-11-21

(2013-11-21, 05:43)epoke Wrote: I have album art file in each album folder, but file names are [AlbumArtist] - [Album].jpg, not folder.jpg or cover.jpg. And all tracks have embedded album art in tag.
./texturecache.py c Albums could not cache my album art, maybe album art file name is issue, right?
How can I cache album arts without renaming album art file name?

Yep, the problem is your naming convention - it's not a supported filename so the local artwork files are ignored, and the embedded artwork is used instead.

Unfortunately, embedded artwork can't be pre-cached - it's a GUI-only process that kicks in as you browse through your library.

Your only option would be to rename your album artwork with a supported filename (eg. folder.jpg), then assign the new artwork to each album - unfortunately that probably means re-scraping your entire music library, unless you're adept at SQL manipulation.


RE: [RELEASE] Texture Cache Maintenance utility - epoke - 2013-11-21

(2013-11-21, 06:21)MilhouseVH Wrote:
(2013-11-21, 05:43)epoke Wrote: I have album art file in each album folder, but file names are [AlbumArtist] - [Album].jpg, not folder.jpg or cover.jpg. And all tracks have embedded album art in tag.
./texturecache.py c Albums could not cache my album art, maybe album art file name is issue, right?
How can I cache album arts without renaming album art file name?

Yep, the problem is your naming convention - it's not a supported filename so the local artwork files are ignored, and the embedded artwork is used instead.

Unfortunately, embedded artwork can't be pre-cached - it's a GUI-only process that kicks in as you browse through your library.

Your only option would be to rename your album artwork with a supported filename (eg. folder.jpg), then assign the new artwork to each album - unfortunately that probably means re-scraping your entire music library, unless you're adept at SQL manipulation.

Thanks for quick answer.
I have renamed all album arts to Folder.jpg and trying to replace url field in art table from mysql database. But it's not easy.
I will drop db and scrape music again.


RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2013-11-22

texturecache.py (Version 1.1.3):
  • Add: cache.artwork property (comma-delimited list) to restrict caching of specific artwork types, eg. @cache.artwork=poster,fanart to cache only posters and fanart. Default is an empty list, which will cache all artwork types.



RE: [RELEASE] Texture Cache Maintenance utility - Enlightened - 2013-11-22

10GB of orphaned files after I ran r. I've had this same thumbnail folder for a very long time. I love this program after I figured out how to use it with a lot of help from MilhouseVH.
Is there a way to use this to cache missing thumbs for tv episodes.

Edit. Wow another 7GB with P.Went from thumbnail folder at almost 20GB down to 3.5 now. which is awesome as my HTPC only has a 60GB ssd.


RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2013-11-22

(2013-11-22, 08:04)Enlightened Wrote: Is there a way to use this to cache missing thumbs for tv episodes.

Code:
texturecache.py c tvshows

Of course it will only cache artwork your media library "knows" about, so if you're missing thumbs artwork for tv show episodes try something like the Artwork Downloader addon, or Ember Media Manager (or similar), to fill in your gaps first.


RE: [RELEASE] Texture Cache Maintenance utility - Enlightened - 2013-11-22

| banner | characterart| clearart | clearlogo | discart | fanart | landscape | poster | thumb | thumbnail |
TOTAL
--------------+-------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+----
---------
Cached | - | - | - | - | - | 1 | - | - | - | 312 |
313
Deleted | - | - | - | - | - | - | - | - | - | - |
0
Duplicate | 42405 | 12028 | 28880 | 34273 | 2 | 44482 | 26960 | 43175 | 484 | 5 |
232694
Error | 1 | 3 | 9 | 25 | 2 | 1 | - | 3 | 1 | 15 |
60
Ignored | - | - | - | - | - | - | - | - | 10 | 250 |
260
Season-all | - | - | - | - | - | - | - | 116 | - | - |
116
Skipped | 647 | 141 | 1160 | 1638 | 543 | 2617 | 599 | 4048 | 36272 | 1370 |
49035
Undefined | - | - | - | - | - | 118 | - | - | - | 324 |
442
================================================================================================================================================================
=========
TOTAL | 43053 | 12172 | 30049 | 35936 | 547 | 47219 | 27559 | 47342 | 36767 | 2276 |
282920
Download Time | 00:00:05.88 | 00:00:19.21 | 00:00:35.57 | 00:00:17.14 | 00:00:11.85 | 00:00:02.00 | - | 00:00:18.75 | 00:00:05.56 | 00:06:01.47 | 00:
09:55.40

I'm having a problem with my music not caching artwork I've manually added. I'm guessing that's what undefined above is. And therefore not being pushed to my other machines. I've used c to recache but its not doing anything
for the artwork. It's being read locally c:/music/artist/extrafanart. When I delete extrafanart my main machine no longer has the artwork either.
This problem and advance launcher not showing fanart in mq5 are my only problems I can't seem to find an answer to.


RE: [RELEASE] Texture Cache Maintenance utility - tinybutstrong - 2013-11-22

Good to see more people confirming my old issue with AD downloading german images:

http://forum.xbmc.org/showthread.php?tid=114633&pid=1555751#pid1555751

If you disable the "no limit", you can pick the specific language, ie. english, but I don't want to do that, because in this mode the extrafanart is limited to 20 items. I want all available images downloaded. Example, for Dexter serie I have 39 extrafanart.

IMO, this is a bug. Anyway, your script is my workaround, a bit of extra work, but there is no alternative.


RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2013-11-22

(2013-11-22, 13:57)Enlightened Wrote: I'm having a problem with my music not caching artwork I've manually added. I'm guessing that's what undefined above is. And therefore not being pushed to my other machines. I've used c to recache but its not doing anything for the artwork. It's being read locally c:/music/artist/extrafanart.

"Undefined" means an artwork item with a value of "", eg.
Code:
./texturecache.py jd movies zombie
[
  {
    "art": {
      "discart": ""
      "clearart": "image://http://assets.fanart.tv/fanart/movies/19908/movieart/zombieland-4fd8c70fd673c.png/",
      "clearlogo": "image://http://assets.fanart.tv/fanart/movies/19908/hdmovielogo/zombieland-5145e97ed73a4.png/",
      "fanart": "image://nfs://192.168.0.3/mnt/share/media/Video/MoviesHD/Zombieland (2009)[BDRip]-fanart.jpg/",
      "poster": "image://nfs://192.168.0.3/mnt/share/media/Video/MoviesHD/Zombieland (2009)[BDRip]-poster.jpg/"
    },
    "file": "nfs://192.168.0.3/mnt/share/media/Video/MoviesHD/Zombieland (2009)[BDRip].mkv",
    "label": "Zombieland",
    "movieid": 667,
    "title": "Zombieland"
  }
]

In the above example, "discart" would be classified as "Undefined".

BTW, "Ignored" would be those items that match one of the patterns specified in cache.ignore.types, the defaults being ^video and ^music, which serve to exclude "embedded" artwork as embedded artwork cannot be pre-loaded into the cache.

You can run a "QA" check on your media with "qa tvshows" (or movies, or music), and if you've now downloaded the artwork replace them with "qax tvshows" which will remove and re-scan the episodes (set @qaperiod=9999 if you do this to widen the default search window, which is media added during the previous 30 days).

As for your music artwork, the best bet would be to look at what you have defined in the media library using the "jd" option, eg. "jd artists <artistname>" and post up the details if you're not sure how to interpret. Or add "@logfile=c:\tc.log" to the command line you used to generate the above output, then send me a link to the log (you may need to zip it).

Though using a path such as "c:/music/artist/extrafanart" doesn't sound like a good idea if you are expecting this artwork to be shared amongst multiple clients (machines). You should be adding you music artwork to the media library with an "smb://" path and then sharing your c:/music folder so that all clients can access it via SMB. This would most likely explain why your other clients can't see the artwork as they can't access it (check for errors in their xbmc.log) - only the client with access to c:/music can cache this artwork.

(2013-11-22, 13:57)Enlightened Wrote: When I delete extrafanart my main machine no longer has the artwork either.
This problem and advance launcher not showing fanart in mq5 are my only problems I can't seem to find an answer to.

I don't know what problem you have or are describing here - more detail?


RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2013-11-22

(2013-11-22, 15:37)tinybutstrong Wrote: Good to see more people confirming my old issue with AD downloading german images:

http://forum.xbmc.org/showthread.php?tid=114633&pid=1555751#pid1555751

If you disable the "no limit", you can pick the specific language, ie. english, but I don't want to do that, because in this mode the extrafanart is limited to 20 items. I want all available images downloaded. Example, for Dexter serie I have 39 extrafanart.

IMO, this is a bug. Anyway, your script is my workaround, a bit of extra work, but there is no alternative.

If AD isn't downloading the artwork - leaving your media library pointing at remote artwork - you might want to look at tools/mklocal.py to download the original artwork into your file system. It works for posters/fanart/clearart/clearlogo/discart/etc but I've not tried it with extrafanart (as I don't have any and not entirely sure how they're stored - I can make changes where necessary if you send me the jd output from one of your movies with extrafanart).

mklocal.py can also be used to load artwork if you have it already local, replacing any remote references or adding references where they didn't previously exist (so basically doing the job of AD, plus it supports movie-name prefixes).


RE: [RELEASE] Texture Cache Maintenance utility - Enlightened - 2013-11-23

(2013-11-22, 13:57)Enlightened Wrote: When I delete extrafanart my main machine no longer has the artwork either.
This problem and advance launcher not showing fanart in mq5 are my only problems I can't seem to find an answer to.
Quote:I don't know what problem you have or are describing here - more detail?

I know xbmc isn't caching the fanart I've added myself because when I go into the extrafanart folder and delete everything on my main machine
the art no longer shows up. When I re add it shows up again. So xbmc is reading the local HDD instead of adding to the thumbnail folder.


(2013-11-22, 20:54)MilhouseVH Wrote: Though using a path such as "c:/music/artist/extrafanart" doesn't sound like a good idea if you are expecting this artwork to be shared amongst multiple clients (machines). You should be adding you music artwork to the media library with an "smb://" path and then sharing your c:/music folder so that all clients can access it via SMB. This would most likely explain why your other clients can't see the artwork as they can't access it (check for errors in their xbmc.log) - only the client with access to c:/music can cache this artwork.

My music is a smb path. But when I added in fanart myself the browser doesn't give me the option of adding it with a smb path so I have to add the folder. Everything I have is on a flexraid D:/music/artist
with full sharing enabled. I ran jd artist and got this

C:\WINDOWS\system32>c:\python27\python.exe c:\texturecache.py jd artists zz ward
[
{
"artist": "Ozzy Osbourne",
"artistid": 1486,
"fanart": "image://http://assets.fanart.tv/fanart/music/8aa5b65a-5b3c-4029-92bf-47a544356934/artistbackground/ozzy-osbourne-4ec36e648cafb.jpg/",
"label": "Ozzy Osbourne",
"thumbnail": "image://http://assets.fanart.tv/fanart/music/8aa5b65a-5b3c-4029-92bf-47a544356934/artistthumb/osbourne-ozzy-4fd0a787e0707.jpg/"
},
{
"artist": "ZZ Ward",
"artistid": 1596,
"fanart": "",
"label": "ZZ Ward",
"thumbnail": "image://D:\\Music\\ZZ Ward\\folder.jpg/"
}
]

C:\WINDOWS\system32>