•   
  • 1
  • 175
  • 176
  • 177
  • 178(current)
  • 179
  •   
[RELEASE] Texture Cache Maintenance utility
Hello @Milhouse,

I read in a very old post where you said that Cygwin with bash was not needed, Python on cmd works fine.
I happen to have Cygwin and its Python package(s) installed.  Do you recommend not using this, and to install "Windows Python" to use with cmd?

Also, do you know whether your utility has been used successfully on Android?
I have been using Termux for a while, and it has Python packages available.  I thought this might be a way to use your utility on Android.

TIA,
Doug
Reply
(2019-08-13, 01:12)grossdm Wrote: I read in a very old post where you said that Cygwin with bash was not needed, Python on cmd works fine.
I happen to have Cygwin and its Python package(s) installed.  Do you recommend not using this, and to install "Windows Python" to use with cmd?

I'd say "use whatever works". A "Windows Python" install will definitely work. A "Cygwin Python" installation may work, so if that's what you have installed - and it's working - then sure, keep using it!

But I wouldn't recommend installing Cygwin just to use this script, as that's not required and may potentially result in unforeseen problems.

(2019-08-13, 01:12)grossdm Wrote: Also, do you know whether your utility has been used successfully on Android?
I have been using Termux for a while, and it has Python packages available. I thought this might be a way to use your utility on Android.

So long as the Android device has a fairly standard version of Python2 (or Python3) available then it should be possible to run the script on the device. If not, then running the script from another device (Linux/Windows PC, Raspberry Pi etc.) and specifying the IP address of the Android client is probably your only option.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
@Milhouse 
I'm struggling with the qax option. 
I have the config with qaperiod = 30 but it looks like it is retrying everything that is missing.

What date is it checking against for this option, is that the date added to library?
If that is the case that would explain my issue since qax removes it from the library and re-scans it again.

Command line
python "C:\GitHub\KodiTextureCache\texturecache.py" qax video @xbmc.host=192.168.2.50

Code:
##
## THIS IS JUST A SAMPLE CONFIGURATION FILE
##
## All of the following lines are comments.
##
## Remove the # at the beginning of a line
## to make that property "active".
##
## Add extra properties to the end of the file.
##
## Don't specify propreties unless you want or
## need to override the default value.
##

#userdata = ~/.xbmc/userdata/
#dbfile = Database/Textures13.db
#thumbnails = Thumbnails/
#xbmc.host = localhost
webserver.port = 80
#rpc.port = 9090
download.threads = 15
#extrajson.albums  =
#extrajson.artists =
#extrajson.songs   =
#extrajson.movies  =
#extrajson.sets    =
#extrajson.tvshows.tvshow =
#extrajson.tvshows.season =
#extrajson.tvshows.episode=
qaperiod = 30
#qa.file = no
#cache.castthumb = no
#logfile =
#logfile.verbose = no
#network.mac = aa:bb:cc:dd:ee:ff
Reply
(2019-08-28, 16:06)Rouzax Wrote: What date is it checking against for this option, is that the date added to library?

Yes, it's dateadded

(2019-08-28, 16:06)Rouzax Wrote: If that is the case that would explain my issue since qax removes it from the library and re-scans it again.

Command line
python "C:\GitHub\KodiTextureCache\texturecache.py" qax video @xbmc.host=192.168.2.50

I can't tell without seeing your screen (which should explain why it is being removed) or a log file.

You'll need to check what date the item being removed had been added - this defaults to the filesystem timestamp, so qax should only be considering media items added within the previous 30 days, and then it will only remove them if certain fields are currently missing (eg. poster/fanart, which now exists locally) - this should all be explained on your screen (whatever check "FAILS").

Use qa if you want to check your media without refreshing it.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Hi Folks,

i am hoping someone can explain some results i am getting that don't make sense to me.

if i run texturecache.py c to force caching files that are in the db but not in the thumbnails folder to the thumbnails folder, i have a few (maybe 20) items that always fail. everything else is cached and only these same files are identified when i run the command. Note all my images are stored locally with the media. i checked the paths and all failed files are 0kb, so i deleted them from the file system.

i ran texturecahce.py X expecting to see only these items, then run texturecahce.py Xd to remove them however, texturecahce.py X returns thousands of results - do i misunderstand the commands or is something amiss? (note i am using kodi 18.3)

Thanks in advance!!
Reply
(2019-09-06, 11:59)thecubasekid Wrote: texturecahce.py X returns thousands of results - do i misunderstand the commands or is something amiss? (note i am using kodi 18.3)

From the help:

X Same as "x" (extract) but will validate cachedurl file exists, displaying only those that fail validation
Xd Same as "x" (extract) but will DELETE those rows for which no cachedurl file exists

The "validation" being performed is a check that the referenced file (from the Textures13.db database) also exists in the Thumbnails folder. If the row from the database is being listed then the file doesn't exist, or is not accessible. If thousands of rows are being unexpectedly listed then it would be best NOT to run Xd until you've got to the bottom of the problem!

If you're using path substitution for the Thumbnails folder then that could be the problem. Otherwise I've no idea without a kodi.log and texturecache log (@logfile=tc.log).
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2019-09-06, 19:48)Milhouse Wrote:
(2019-09-06, 11:59)thecubasekid Wrote: texturecahce.py X returns thousands of results - do i misunderstand the commands or is something amiss? (note i am using kodi 18.3)

From the help:

X Same as "x" (extract) but will validate cachedurl file exists, displaying only those that fail validation
Xd Same as "x" (extract) but will DELETE those rows for which no cachedurl file exists

The "validation" being performed is a check that the referenced file (from the Textures13.db database) also exists in the Thumbnails folder. If the row from the database is being listed then the file doesn't exist, or is not accessible. If thousands of rows are being unexpectedly listed then it would be best NOT to run Xd until you've got to the bottom of the problem!

If you're using path substitution for the Thumbnails folder then that could be the problem. Otherwise I've no idea without a kodi.log and texturecache log (@logfile=tc.log). 
thanks Millhouse

i understood the commands then, I'm not using any sort of path substitution. This is a straight forward Kodi install oto android xtreme custom rom from xda forums on a M8 II box with MySQL DB running on a windows PC. I run texturecache from same Windows PC that hosts MySQL. Guess i need to dig a little deeper, also thanks for the advice about setting log on texturecache, for the Kodi log, would simply turning on debug be enough to capture what is needed?
Reply
(2019-09-06, 22:05)thecubasekid Wrote: I run texturecache from same Windows PC that hosts MySQL. Guess i need to dig a little deeper, also thanks for the advice about setting log on texturecache, for the Kodi log, would simply turning on debug be enough to capture what is needed?

The X and Xd commands require direct access to the filesystem of the device running Kodi, so if you're running the script on Windows for a remote Android client then it's likely your Windows PC simply can't access the Android filesystem, hence all the rows being listed (the textures database for the client can be accessed via JSON, but to test that the cached image files exist in the Thumbnails folder requires - unfortunately - direct filesystem access).

Presumably you've configured the userdata and/or thumbnails property in some way so that it references the Android filesystem from the Windows PC, so I would suggest checking this path and ensuring your Windows PC has read/write access. Run texturecache.py config to see the paths being used by userdata and thumbnails (they will be concatenated together userdata+thumbnails to generate the full path to the client Thumbnails image folder).
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2019-09-07, 02:57)Milhouse Wrote:
(2019-09-06, 22:05)thecubasekid Wrote: I run texturecache from same Windows PC that hosts MySQL. Guess i need to dig a little deeper, also thanks for the advice about setting log on texturecache, for the Kodi log, would simply turning on debug be enough to capture what is needed?

The X and Xd commands require direct access to the filesystem of the device running Kodi, so if you're running the script on Windows for a remote Android client then it's likely your Windows PC simply can't access the Android filesystem, hence all the rows being listed (the textures database for the client can be accessed via JSON, but to test that the cached image files exist in the Thumbnails folder requires - unfortunately - direct filesystem access).

Presumably you've configured the userdata and/or thumbnails property in some way so that it references the Android filesystem from the Windows PC, so I would suggest checking this path and ensuring your Windows PC has read/write access. Run texturecache.py config to see the paths being used by userdata and thumbnails (they will be concatenated together userdata+thumbnails to generate the full path to the client Thumbnails image folder). 
Thanks, this is the problem... my Windows PC has no access to the android file system, guess i wont be using X anytime soon!

thanks also for the fantastic script, i have kodi in most rooms of the house all sharing the SQL DB. Your script helps massively in maintaining all clients in an easy way!!!
Reply
I just learned of "texturecache.py" and I am trying to figure it out.  The reason why I am intrested in it is because my Thumbnails cache is 5GB and tens of thousand of images in size.  I didn't realize that it was growing so large and I need a way to clean it up, hence I found "texturecache.py".

I've been reviewing the docs and follow some of the examples, but I'm not sure how to clean only the Thumbs directory.  I use a centralized Thumbnails storage, on my NAS, and point all of my devices to the same location, probably the reason why it's so large.    I ran this example  ./texturecache.py x "where lastusetime <= '2019-01-01'" and I get a lot of stuff, but nothing from the thumbnails.  

I copied the config example from the docs and added in the Thumbnails share on the NAS, but I'm still not seeing anything from the Thumbnails.  What is the correct syntax to clean out only the Thumbnails and not other cached images?  After I clean up the thumbnails, then I'll review other images and see what needs to be cleaned.

Thanks

Daryl
Reply
(2019-09-24, 14:12)rosede Wrote: What is the correct syntax to clean out only the Thumbnails and not other cached images?
The "prune" command texturecache.py P would automatically remove cached images that are not referenced by a movie, tvshow or album/song/artist, from your media library.

However your biggest problem is that you're sharing a single texture cache between multiple clients, which is generally a bad idea.

Are all your clients using a centralised MySQL database for the media library? If not, then each client is likely to have a different view of the texture cache, and purging the cache based on the media library from one client is likely to cause issues for your other clients that have different media libraries as they suddenly lose access to movie/tv/music images (now deleted) that they have cached.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2019-09-24, 17:24)Milhouse Wrote:
(2019-09-24, 14:12)rosede Wrote: What is the correct syntax to clean out only the Thumbnails and not other cached images?
The "prune" command texturecache.py P would automatically remove cached images that are not referenced by a movie, tvshow or album/song/artist, from your media library.

However your biggest problem is that you're sharing a single texture cache between multiple clients, which is generally a bad idea.

Are all your clients using a centralised MySQL database for the media library? If not, then each client is likely to have a different view of the texture cache, and purging the cache based on the media library from one client is likely to cause issues for your other clients that have different media libraries as they suddenly lose access to movie/tv/music images (now deleted) that they have cached. 

Yes, I am using MySQL to share the media library.  

So will the purge clean out only the Thumbnails?  I don't want it deleting fan art and other images.  

If I change the location of the Thumbnails back to local storage, can I use texturecache.py to regenrate those thumbnails?   

Thanks

Daryl
Reply
(2019-09-24, 17:53)rosede Wrote: Yes, I am using MySQL to share the media library.  

OK, that's one less issue... Smile

(2019-09-24, 17:53)rosede Wrote: So will the purge clean out only the Thumbnails?  I don't want it deleting fan art and other images.  

Yes, texturecache.py will only ever delete images from the Texture Cache (Textures13.db, and Thumbnails folder)- it won't delete your media library artwork (fanart, posters alongside your movies etc.)

(2019-09-24, 17:53)rosede Wrote: If I change the location of the Thumbnails back to local storage, can I use texturecache.py to regenrate those thumbnails?   

texturecache.py c will cache all artwork (fanart, posters etc.) referenced by your media library.

Note that any remote artwork that is no longer available (posters on websites, actor images etc.) will not be possible to cache/re-cache - you'll need to update your media library with alternative source urls for the now missing artwork (ideally, don't use remote artwork...)
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Hi,

I am seeing this each time I do a vscan now:

Updating library: New id           -1

Always -1, and today.... two of them!

/home/osmc/texturecache.py/texturecache.py vscan
Rescanning library...
Updating library: New id           -1
Updating library: New id           -1


Any idea how I could try to resolve whatever these may be?
Reply
(2019-09-27, 21:19)yhs4260 Wrote: Hi,

I am seeing this each time I do a vscan now:

Updating library: New id           -1

Always -1, and today.... two of them!

/home/osmc/texturecache.py/texturecache.py vscan
Rescanning library...
Updating library: New id           -1
Updating library: New id           -1


Any idea how I could try to resolve whatever these may be?

Can you run /home/osmc/texturecache.py/texturecache.py @logfile=/tmp/tc.log vscan, reproduce the issue, and upload /tmp/tc.log somewhere (zip it if necessary) and send me a link to the log?

The -1 id being displayed is coming directly from the Kodi JSON response, so this looks like a Kodi bug/issue... check your Kodi log for errors as well.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
  •   
  • 1
  • 175
  • 176
  • 177
  • 178(current)
  • 179
  •   
 
Thread Rating:
  • 17 Vote(s) - 4.76 Average



Logout Mark Read Team Forum Stats Members Help
[RELEASE] Texture Cache Maintenance utility4.7617