19 Nov 2019 - TVDB are still in the process of fixing a number of bugs that have broken the TVDB scraper and any add-on that relies on TVDB data. TVDB are still working to rectify the problems.

  •   
  • 1
  • 29
  • 30
  • 31(current)
  • 32
  • 33
  • 179
  •   
[RELEASE] Texture Cache Maintenance utility
You really need to read the first post, particularly the part about configuration... Wink

Set your texturecache.cfg as follows:
Code:
dbjson=no
userdata=/User/Library/Preferences/XBMC/userdata
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Thanks so much! Got it working now. (Saved almost 500 MB or 25-30% space.) It takes many minutes to delete the images from each site, and I have dozens here.

A Shell Script

As you can tell I'm only marginally competent with unix, but I was able to make a simple shell script ("tckill.sh") that looks like this:

Code:
#!/bin/bash
# tckill.sh
echo "starting delete for youtube"
IDS=$(texturecache.py x 'where lasthashcheck = "" and url like "%youtube%"' 2>&1 1>/dev/null | grep "Matching row ids" | cut -b19-)
[ -n "$IDS" ] && texturecache.py d $IDS
echo "starting delete for nickelodeon"
IDS=$(texturecache.py x 'where lasthashcheck = "" and url like "%nickelodeon%"' 2>&1 1>/dev/null | grep "Matching row ids" | cut -b19-)
[ -n "$IDS" ] && texturecache.py d $IDS
# repeat with as many sites as needed
echo "all done"

Notes for the similarly marginally competent:

• I put the script in /usr/bin/ (in my $PATH) and now, as expected, so I could type "texturecache.py" rather than "./texturecache.py". But I didn't expect that I could no longer use "./texturecache.py". Not sure why. Minor thing. But that explains why the above shell script is written that way rather than the way Millhouse put it in post #440.

• I was not able to find a text editor that worked on the ATV. neither vi, emacs, or pico are installed. Rather than figure that out, I just created and edited the shell script on my Mac and uploaded it to /usr/bin/ next to texturecache.py

• Of course that has to be made executable like texturecache.py itself (with "chmod +x tckill.sh")

• on the ATV, it really is necessary to create the texturecache.cfg file as given kindly by Millhouse in post #451.

• it might not be obvious to others (as it wasn't to me at first) that the texturecache.cfg file should live next to (at the same directory level) as texturecache.py

• I was able to get this all to work without messing with any additional installation of SQlite. The only thing I did other than install this script was install Python 2.7.3 as directed in post #1.

Scheduling

cron is not installed on the ATV. Apple uses launchd for scheduling on OS X and someone here found it works on the ATV too. I'm sure one could find more concise examples. I will look into that further at a later date if needed.

Questions

• any bright way to make the above script give a bit of feedback, such as how many IDs found, and how many IDs deleted? and/or time started and finished for each site?

• and or even more advanced, how to loop it and give a list of sites as a variable?

Suggestions

This all works fine in the end. But you might consider making this simpler by having a format in texturecache.py itself like:

Code:
texturecache.py kill 'site1, site2, site3' -unhashed

(with -unhashed being an option along the lines explained in post #440).

Thanks again for your help.
Reply
(2013-11-07, 18:50)MilhouseVH Wrote: Why did you blank out the filenames in your image of the database rows, that might have been useful in determining what is going on as it looks like one discart has a forward slash, the other a backward slash. If you want to send me your Video and Texture databases I can try taking one last look at resolving any remaining issues with mangled filenames, as I suspect that is your problem.

Edit: I've just noticed you also have two each of clearart and logo images, one with a forward slash the other with a backslash, chances are XBMC is still using the "old" clearart/logo images. I can only imagine you've used an addon that has knackered your urls in a similar way to Movie Set Artwork Automator (now fixed) - maybe contact the addon maintainer about that. Send me your databases and I'll see if I can work around it (though thought I had last time, will be interesting to see what is in your MyVideo db).

Edit2: It's a problem with your sets, right? Try running the following:
Code:
python "c:\folder\texturecache.py" C sets "007" @logfile=c:\tc.log @logfile.verbose=yes
and then upload the tc.log somewhere.

Hey, thanks for the response. All my images are displaying fine, I'll stop to worry about it for now. Anyway, I still have a bad experience with XBMC. What I do every time I add a new movie to my library:

1) Scan for updates;
2) Run Artwork Downloader in auto mode;
3) I have no limit set in AD options, then is very common it downloading some German (always that nazi hijacking my htpc lol) images, specially disc art, logo and clear art;
4) Then I open the top menu to run AD GUI mode (Ace skin), picking each English image to replace the German one. All downloaded fine, I confirm in debug.log and physical location. But in XBMC screen it still using the damn old German images!! grrr, even restarting XBMC several times doesn't fix it.
5) ok calm down, now using your amazing script I'm fixing each item after to hit step 4. (texturecache.py C movies "partial name"), done everything is cool now.

I'm happy? no, not yet. I really need to use your script every time to get new image in cache really? I want to understand, this is a XBMC fault or should I blame the AD author? btw, this last one I already reported the issue, but he was very rude (just a bit impatience guy) and basically ignored.

Not sure if still necessary, but I sent my db's files to you. Check PM.

Thanks
I wish to have Rotten Tomatoes (user+critics) + IMDB both ratings display in skin.
Reply
(2013-11-09, 16:15)tinybutstrong Wrote: I'm happy? no, not yet. I really need to use your script every time to get new image in cache really? I want to understand, this is a XBMC fault or should I blame the AD author? btw, this last one I already reported the issue, but he was very rude (just a bit impatience guy) and basically ignored.

That's the risk with internet based artwork I'm afraid, what you get can sometimes be a mixed bag. If the addon doesn't include a way to filter out German artwork, then your current approach is what you'll have to stick with.

(2013-11-09, 16:15)tinybutstrong Wrote: Not sure if still necessary, but I sent my db's files to you. Check PM.

Thanks. I can confirm the artwork on your movies has mangled paths, for instance on Quantum of Solace:
Code:
"art": {
      "banner": "image://I:\\Filmes\\007 Collection\\22.James.Bond.007.Quantum.of.Solace/banner.jpg/",
      "clearart": "image://I:\\Filmes\\007 Collection\\22.James.Bond.007.Quantum.of.Solace/clearart.png/",
      "clearlogo": "image://I:\\Filmes\\007 Collection\\22.James.Bond.007.Quantum.of.Solace/logo.png/",
      "discart": "image://I:\\Filmes\\007 Collection\\22.James.Bond.007.Quantum.of.Solace/disc.png/",
      "fanart": "image://http://d3gtl9l2a4fn1j.cloudfront.net/t/p/original/hfZVY8lMiE7HH1cDc2qzSFF6Kbt.jpg/",
      "landscape": "image://I:\\Filmes\\007 Collection\\22.James.Bond.007.Quantum.of.Solace/landscape.jpg/",
      "poster": "image://http://d3gtl9l2a4fn1j.cloudfront.net/t/p/original/fuQuV4JF6WflG22mgszXuAaaQVb.jpg/"
    },

Your banner, clearart, logo, discart and landscape artwork all have mangled paths with a mixture of backslashes (Windows, correct) and finally a forward slash (Linux, incorrect). If it is Artwork Downoader that is setting these mangled artwork items for you, you might want to have a (nice) word with Martijn Wink

Although I fixed my script so that it would successfully match these mangled paths against the database (containing unmangled paths), the script is still downloading the mangled path (as that is what is set in the media library) which then appears to be creating mangled versions in the database - and XBMC continues to use the unmangled version... Clearly I need to fix the downloaded path too (sighs).
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
@ctawn: I'll add an option for you, and also make the script auto-detect the ATV2 userdata folder (which should mean one less reason to create the cfg file). Note that the script will look in your current working directory for the cfg file, then if not found it will look in the directory containing the texturecache.py script.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2013-11-09, 18:31)MilhouseVH Wrote: ...you might want to have a (nice) word with Martijn Wink

Out of question for now. I want to avoid contacting him if possible... forever. lawl

(2013-11-09, 18:31)MilhouseVH Wrote: Although I fixed my script so that it would successfully match these mangled paths against the database (containing unmangled paths), the script is still downloading the mangled path (as that is what is set in the media library) which then appears to be creating mangled versions in the database - and XBMC continues to use the unmangled version... Clearly I need to fix the downloaded path too (sighs).

Great! I'll try it when you update the github source.
I wish to have Rotten Tomatoes (user+critics) + IMDB both ratings display in skin.
Reply
(2013-11-09, 20:38)tinybutstrong Wrote: Out of question for now. I want to avoid contacting him if possible... forever. lawl

Not sure if he's following this thread, if not I'll ping him.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
texturecache.py (Version 1.0.5):
  • Add: Augment movie set dump (j sets, J sets etc.) with list of member movies. Movie entries will be ordered by sorttitle if present in media library. Suppress this behaviour with @setmembers=no. When using J, Jd and Jr options, specify additional movie fields with @extrajson.movies, otherwise default movie fields will be returned (movieid, file, title and sorttile).
  • Add: purge hashed site [site] and purge unhashed site [site] options to remove artwork associated with sites, with or without hashes. Use purgetest in place of purge to see which artwork items would be selected for removal.
  • Chg: If SQLite3 module not available, don't complain just force @dbjson=yes (should simplify ATV2/Gotham installations)
  • Chg: Make @logfile.verbose=yes the default setting - has no impact unless logging is actually enabled with @logfile=<filename>
  • Chg: If platform is darwin (ie. ATV2), use a better default userdata path
  • Fix: Correct mangled slashes also in the quoted download url, so that the requested (and now unmangled) url creates an unmangled entry in the Textures db
  • Fix: Correctly validate streamdetails during qa/qax if specified on qa.blank.*
tools\mklocal.py:
  • Add: Movie sets support, finding common parent folder for each set (requires texturecache.py v1.0.5+)

@tinybutstrong: I would recommend you run a "./texturecache.py P" to automatically remove the database entries with mangled paths (or simply search and delete using a search such as s "/disc.png" etc. to remove just the artwork that contains the forward slash) then anything you cache (c or C) from now on should hopefully create entries in the database that XBMC can find...

@ctawn: Give purgetest unhashed youtube nickelodeon a try and see if it gives you expected results (the purgetest to purge to actually delete the artwork).
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
I've done some more testing with mangled filenames, and I'm close to reverting all the attempted fixes, as it's an impossible situation to resolve.

The problem being that when downloading a mangled filename (in order to cache an item) the webserver will insert corrected urls into the texture cache. However when the same artwork is viewed in the GUI, it will be unable to find the now corrected url so the GUI will create another row using the mangled url and cache the artwork a second time.

So although this script can correct the urls in order to match against the texture database, and then delete or re-cache those items, it's doing so only for it's own benefit - the GUI won't use those cached items with corrected urls that no longer match the media library, nor is it possible for this script to create rows with mangled urls as per the media library because the webserver is frustratingly correcting them on-the-fly...

So we have a situation where two parts of XBMC are not in sync: the GUI, which blindly uses what it's given by the media library (which is probably correct), and the webserver which, when asked to download a mangled url by this script (or any other means), apparently corrects the url before inserting it into the texture cache to create what is, to all intents and purposes, a perfectly good but utterly useless cache item (in GUI terms, anyway).

I'll contact the webserver maintainer (is it Montellese?) to see if this inconsistency can be resolved, but I'm not going to waste any more time trying to work around mangled urls that are being set by addons in the media library. The best solution is to use addons that do not create mangled urls in the first place but failing that, consistent behaviour between XBMC front-end and back-end would be helpful! Smile
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Hopefully the last word on mangled urls...

texturecache.py (Version 1.0.6):
  • Revert: Removed all attempts at fixing slashes on-the-fly, utterly pointless given the current front-end/back-end disconnect
  • Add: Option fixurls to identify media library urls with mixed forward and backward slashes. Output can be piped into texturecache.py set to apply corrective changes to the media library, eg:
    Code:
    ./texturecache.py fixurls | ./texturecache.py set

    Once corrective changes have been applied, pruning (to remove old mangled artwork from cache) and caching of new, unmangled artwork may be required.

    Subsequent execution of troublesome addons may re-apply mangled urls to the media library - contact the add-on author/maintainer to notify them of the problem.

    Applying corrective changes to Movie Sets will require a recent nightly build of Gotham (requires JSON API v6.12.0+).

So if you've got stuffed up urls, fix 'em. Wink
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Oh, good explanation. Now I understand the cause of my longgggg time issue. I hope more people read your post, specially the add-on devs.
I wish to have Rotten Tomatoes (user+critics) + IMDB both ratings display in skin.
Reply
(2013-11-09, 21:59)MilhouseVH Wrote: [*]Add: purge hashed site [site] and purge unhashed site [site] options…

@ctawn: Give purgetest unhashed youtube nickelodeon a try and see if it gives you expected results (the purgetest to purge to actually delete the artwork).
Great work, Awesome! I did some tests with v1.06 and seems to work as expected.

One caveat is that it revealed one underlying problem/curiosity which was the same whether I used the new purge command or the "x" command with the parameters I was using above (lasthashcheck = "" and url like "%youtube%"'). That is among the results:

Code:
Dry-run, would remove: image://[email protected]%3a%2f%2fctwan%3apassword%40Computer.local%2fHardDrive%2fVideo%20%5bDriveName%5d%2fMusic%20Videos%20%5bNot%20in%20Library%5d%2fLongName%20-%20YouTube.flv/

That's a .flv video, not an image, and it's on a network volume (afp:// protocol). What business does the script have finding anything on a network volume, especially a video? (I had the same problem in a big way with the Thumbnail Cleaner script – it was returning hundreds of such results – which is how I got over here to the command line in the first place.)

I made a copy of the file in question as a backup and then went ahead and used the purge command. Script reports it is "Removing" it but thankfully the file is not actually deleted on the networked volume. I did the same with "x" command followed by the "d" command as I had been using with the same results (no actual removal), but no feedback either way.

Second issue, might want to include a bit of a safeguard for search terms that are too short with the purge command. I used a long space separated list and inadvertently had "4" as a search term when that was supposed to be just the first character of the search term. I'm afraid that was a bit destructive. I'd say a min of 3 characters is fair unless overridden?

(2013-11-09, 21:59)MilhouseVH Wrote: [*]Chg: If SQLite3 module not available, don't complain just force @dbjson=yes (should simplify ATV2/Gotham installations)
[*]Chg: If platform is darwin (ie. ATV2), use a better default userdata path
nice idea on force dbjson=yes. thanks.
note that Mac OS X also uses a Darwin platform underneath. so…
Reply
(2013-11-10, 19:12)ctawn Wrote: One caveat is that it revealed one underlying problem/curiosity which was the same whether I used the new purge command or the "x" command with the parameters I was using above (lasthashcheck = "" and url like "%youtube%"'). That is among the results:

Code:
Dry-run, would remove: image://[email protected]%3a%2f%2fctwan%3apassword%40Computer.local%2fHardDrive%2fVideo%20%5bDriveName%5d%2fMusic%20Videos%20%5bNot%20in%20Library%5d%2fLongName%20-%20YouTube.flv/

That's a .flv video, not an image, and it's on a network volume (afp:// protocol). What business does the script have finding anything on a network volume, especially a video? (I had the same problem in a big way with the Thumbnail Cleaner script – it was returning hundreds of such results – which is how I got over here to the command line in the first place.)

That's a thumbnail that's been extracted from the referenced flv (that's the meaning of the [email protected] notation, you'll also see [email protected] for artwork extracted from audio files). XBMC will extract thumbnails from a video/music file if a thumb isn't available locally or on the internet. Deleting the cached artwork is harmless and will have no effect on the source media file. There's an option in Settings to disable this extraction behaviour.

Unfortunately these embedded artwork items can't be pre-cached by this script as there's no mechanism to support this, only browsing through the GUI can extract them so the downside to the removal of embedded artwork (if you want to keep them) is that it's a slow process to recreate them. When forcing items to be pre-cached (C movies etc.) this script will by default ignore these embedded artworks and not delete/re-cache them (change this behaviour with the cache.ignore.types property) so by default it's safe to force a re-cache without clobbering all the slow extraction stuff. Smile

You can also identify any music, movies or tv shows that are using embedded artwork by running the "qa" option (eg. "qa movies")

(2013-11-10, 19:12)ctawn Wrote: note that Mac OS X also uses a Darwin platform underneath. so…

Yeah, I realise that which is why darwin devices will only use the ATV2 path if it exists (does that path exist for MacOSX?) otherwise MacOSX will use the default path (~/.xbmc/userdata), so hopfeully MacOSX users should see no change. I've no idea what the default installation path is for MacOSX, if anyone wants to let me know I'll add it in future...
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
FYI:

Code:
./texturecache.py purge hashed transform?size

is a good option for quickly getting rid of image previews, much quicker than a full prune.

(2013-11-10, 19:12)ctawn Wrote: Second issue, might want to include a bit of a safeguard for search terms that are too short with the purge command. I used a long space separated list and inadvertently had "4" as a search term when that was supposed to be just the first character of the search term. I'm afraid that was a bit destructive. I'd say a min of 3 characters is fair unless overridden?

That's why I added the purgetest option... Smile
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
texturecache.py (Version 1.0.7):
  • Add: Auto-update facility, rather than run update manually each time to update. If @checkupdate=yes and @autoupdate=yes (both being the defaults), at the beginning of each script execution the latest github version will be checked and, if a newer version is available, the script automatically updated before execution commences with the latest version. Disable by setting @autoupdate=no. If @checkupdate (which simply warns when the version is out of date) is disabled, auto-updating will also be disabled.
  • Add: In addition to hashed and unhashed on purge/purgetest, added new filter all, being either hashed or unhashed.
  • Fix: Not stripping leading "image://" from artwork in qa/qax.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
  •   
  • 1
  • 29
  • 30
  • 31(current)
  • 32
  • 33
  • 179
  •   
 
Thread Rating:
  • 17 Vote(s) - 4.76 Average



Logout Mark Read Team Forum Stats Members Help
[RELEASE] Texture Cache Maintenance utility4.7617