Guest - Testers are needed for the reworked CDateTime core component. See... https://forum.kodi.tv/showthread.php?tid=378981 (September 29) x
  • 1
  • 14
  • 15
  • 16(current)
  • 17
  • 18
  • 197
[RELEASE] Texture Cache Maintenance utility
(2013-07-04, 05:14)HueyHq Wrote: You make particular reference to client DBs, so what happens on the server machine?

The reason I ask is because, against the trend, I use path substitution for the Thumbnails folder. In Frodo.
(I know, I know, I don't get the performance, but man, does it make maintenance sooo much easier! Each new Pi on my network gets the same advancedsettings.xml thrown on an OpenELEC install - no fuss, no stress, and it just works! AFAIK, nothing pertaining to the media is stored on the Pi - it all comes from the server.)

If access is done via JSON, will Textures13.db still be relevant in my case?

I'm curious to see if I delete Textures13.db, whether or not XBMC will recreate it!

If you delete Textures13.db, XBMC will recreate it and slowly re-populate it with cached file details.

I'm not sure if XBMC will try to recreate the optimised artwork files, as it may realise there is already a matching file in the Thumbnails folder since you are sharing Thumbnails (this can lead to a problem where the new image won't replace the existing image). Then again, XBMC may just go right ahead and overwrite the existing Thumbnail file with the new version. Should XBMC overwrite existing files, and if the original artwork file has changed, this may leave your other clients with inconsistent information in their Textures13.db (image height, width etc. is all stored in the Textures13.db rather than extracted from the image each time). This can cause problems when those images are subsequently displayed with the incorrect image proportions.

Think of it like this: you're sharing everything (media library, Thumbnails) but not the "bridge" between your media library and the cached Thumbnails - that bridge is Textures13.db. Ideally, when sharing Thumbnails, you'd also have a single copy of Textures13.db on each client so that all clients would have a consistent "view" of the Thumbnails folder. Currently, this isn't really possible (maybe you could path-sub the Databases folder, but I wouldn't recommend it).

I'm not saying you're wrong to use path substitution to share Thumbnails, but it is a creaky solution that can fall apart quite easily. Deleting artwork from a shared Thumbnails folder is particularly problematic, as any clients that still have the deleted file referenced in their Textures13.db are likely to show a blank/default image.

This script is one alternative to path substitution as you can easily write a scheduled job that keeps all of your Pi clients and their local Textures13.db/Thumbnail caches up to date.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2013-07-04, 05:22)MilhouseVH Wrote:
(2013-07-04, 05:17)wishie Wrote: Should searching for empty folders be an option?
In my case, I was glad I could find them, and therefore remove them.

Empty folders are not particularly relevant as far as XBMC or the media library is concerned, so I'm not sure there needs to be an option to detect them.

It should also be relatively easy to identify empty folders on whatever OS you are using (OK, that's a lie - probably a pain in the @rse on Windows, but it's trivial on Linux: "find . -type d -empty").

Valid point. Just doing './texturecache.py c movies' now, and I am shocked that it basically wants to cache everything.. I thought XBMC would have been doing that as it fetched the data while scraping? Or does XBMC download each part of art for each movie as you browse around the library?
Reply
(2013-07-04, 05:44)wishie Wrote: Valid point. Just doing './texturecache.py c movies' now, and I am shocked that it basically wants to cache everything.. I thought XBMC would have been doing that as it fetched the data while scraping? Or does XBMC download each part of art for each movie as you browse around the library?

XBMC should download (and cache) the artwork on the machine that is performing the scrape, but any other non-scraper clients will not have any artwork cached until you start browsing the GUI (or pre-load using this script).

You can see what is in your cache with "./texturecache.py x", and see what isn't in your cache with "./texturecache.py nc movies".
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2013-07-04, 05:17)wishie Wrote: Ive just spent nearly a day undoing the mess that a shared Thumbnails did caused. I used to use path sub for it too, thinking it was a good idea, even though I was told otherwise.
The performance on the clients suffer (especially if Pi/ATV2/Android devices), network load increases, etc etc.. Its simply not worth it.

My understanding now is, if you remove the Textures.db and path sub from the clients, they will automatically download the artwork (based from the URLs stored in the video library) store them in its own Thumbnails cache and build Textures db locally with optimised images. This may be a pain to start with, but will be a very minimal (and automated thing) afterwards.

You end up with less network traffic, and faster client performance.

I still use MySQL for my video and music libraries, but artwork should be unique for each client. I guess there is no harm having a shared thumbnail cache for 2 Pi's or 2 ATV2's or something, but I still wouldn't do it.
Well that's interesting then; thanks for the info.

I did spend a lot of time setting up the scenario with the specific goal of ease of use. Performance wasn't really my primary concern because with other performance tweaks, the wife hasn't noticed any difference!

I was also concerned about any tweaks to my collection wouldn't get updated at the client. Does this happen automatically? Or do you have to use texturecache.py on each client each time? I also hated how long it took to load the client DB, but I have since found a bottle-neck, so it may be better next time!

I have 2 Pi's currently, with a third imminent. I see that number increasing as the kids get older! I will be interested in how the network responds.
Reply
Im using the machine that did all the scraping, and it seems as though it stored all the metadata in the video library as you would expect, including URLs for artwork but didn't actually download the artwork until that movie was browsed in the GUI.. otherwise, I can't explain why your script is wanting to download 1700+ pieces of artwork..

Anyway, once the run is complete I will try with 'x' and have a look, then re-run with 'c' to see what happens then.
Reply
(2013-07-04, 05:43)MilhouseVH Wrote: Think of it like this: you're sharing everything (media library, Thumbnails) but not the "bridge" between your media library and the cached Thumbnails - that bridge is Textures13.db.
Excellent - that is the piece of advice I was looking for! Thanks!
Reply
I actually boot my OpenELEC based Pi with /storage mounted on NFS, so my Thumbnails and Database folders are both accessed across the (wired) network, and performance really isn't an issue... This type of setup also makes accessing the Pi "remotely" much easier, as any machine that can access NFS can read/write the Textures13.db and Thumbnails folders (although OpenELEC automatically exports the Userdata folder as an SMB share, which also permits full remote administration with this script).

(2013-07-04, 05:53)wishie Wrote: Im using the machine that did all the scraping, and it seems as though it stored all the metadata in the video library as you would expect, including URLs for artwork but didn't actually download the artwork until that movie was browsed in the GUI.. otherwise, I can't explain why your script is wanting to download 1700+ pieces of artwork..

Anyway, once the run is complete I will try with 'x' and have a look, then re-run with 'c' to see what happens then.

Is the artwork located on a remote site - maybe it was down/overloaded, are there any scraper errors in your XBMC log? I have all my artwork stored locally, and I'm pretty sure the scraping client will automatically cache local artwork... not so sure about remote artwork though, I assumed it was the same but maybe not?
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
I basically started fresh.. No artwork in the movie dirs, no Thumbnails dir, no Textures13.db

It looks like the scraper simply fills the video library with metadata including the URLs to artwork.. Only when you browse that title does XBMC download and cache the artwork.

While you're active, Im having issues getting the script to run on ATV2, since sqlite3 doesn't seem to be installable..

sqlite3 depends on sqlite3-lib which depends on sqlite3-dylib

The issue is sqlite3-dylib will only install (firmware < 3.0)

Any suggestions?
Reply
(2013-07-04, 06:28)wishie Wrote: While you're active, Im having issues getting the script to run on ATV2, since sqlite3 doesn't seem to be installable..

sqlite3 depends on sqlite3-lib which depends on sqlite3-dylib

The issue is sqlite3-dylib will only install (firmware < 3.0)

Any suggestions?

Sorry, I know next to nothing about ATV2.

Did you try installing the Python 2.7.3 package mentioned in the note for ATV2 users under the "Installation Instructions" section in the first post? Ned Scott found an ATV2 Python package (but wasn't specifically testing sqlite), and I've added a link to a more recent version.

If you go to the download page for the Python package, it has links to additional Debian Package Dependencies, this list includes sqlite3 and sqlite3-lib. Although it doesn't mention sqlite3-dylib as a dependency, a deb installer for iphone is listed here as sqlite3-dylib_3.5.9-1_iphoneos-arm.deb - maybe it will work?
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
without sqlite3 installed, your script is basically useless Sad Im trying to find any info I can, but it looks as though sqlite3-dylib hasn't been updated in a while.. I mean, it wants firmware < 3.0 .. I currently have firmware 6.1 installed.
Reply
Eugh! - get a Raspberry Pi. Wink

Maybe someone else can chime in, but the ATV2 seems to be a bit of a PITA.

You could mount your ATV2 Userdata folder on a remote machine (SMB? NFS? Not sure what kind of sharing is possible on the ATV2) and then administer your ATV2 remotely using a suitably configured texturecache.py./texturecache.cfg.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
So, in a strange turn of events, I forced Ned Scott's linked version to install, which then prompted me to fix broken depends.. which I did use 'apt-get -f install' and it installed sqlite3 and sqlite3-lib without requiring sqlite3-dylib...

Strange.
Reply
Hmmm... do you think the older Python package is better then?
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
well Python isn't the issue, which is strange.

Python didn't want to install because sqlite3 wasn't installed.
sqlite3 wouldn't install because sqlite3-lib wasn't installed
sqlite3-lib wouldn't install because sqlite3-dylib wasn't installed.

But for some reason, when I forced the older Python to install, everything else fell into place. Perhaps the older Python depends on a slightly older package of sqlite3 that doesn't have the dependency on the -dylib package..

Ill see what info I can dig up.
Reply
Another question I have is, why do I get more errors on ATV2 that I did on my iMac... both are using the same NFS share and MySQL video library. Both had no Thumbnails/ dir when I started..

Here is some of the current output from the ATV2 scan

Caching artwork: 1032 items remaining of 3121 (s: 0, m: 1032), 640 errors, 2 threads active (00.40 downloads per second, ETA: 00:43:31)2)

Thats when doing 'texturecache.py c movies'
Reply
  • 1
  • 14
  • 15
  • 16(current)
  • 17
  • 18
  • 197

Logout Mark Read Team Forum Stats Members Help
[RELEASE] Texture Cache Maintenance utility17
This forum uses Lukasz Tkacz MyBB addons.