[Milhouse]

Argghhh...

I'm totally confused why your artwork isn't caching - it can't be because of the backslashes, because even those files with only forward slashes are not caching. You shouldn't, in theory, have had any artwork to cache for the "nc" run, and the run above should have deleted all 82 items, not just 5!

Can you do me a favour... run the following commands in the following sequence (top to bottom) - do not access the XBMC GUI until all commands have been run:

Code:

texturecache.py x > C:\Temp\x0.log
texturecache.py C sets @download.payload=yes @logfile=C:\Temp\tc1.log @logfile.verbose=yes > C:\Temp\out1.log
texturecache.py x > C:\Temp\x1.log
texturecache.py C sets @download.payload=yes @logfile=C:\Temp\tc2.log @logfile.verbose=yes > C:\Temp\out2.log
texturecache.py x > C:\Temp\x2.log

then zip up the seven log files (x0.log, tc1.log, out1.log, x1.log, tc2.log, out2.log and x2.log) and PM me a link - that would be much appreciated. Maybe I can spot what is happening. Also, can you check your XBMC log to make sure there are no errors there (filesystem errors, or database errors?) Maybe include xbmc.log in the zip... Cheers.

[Milhouse]

Thanks to CaptainKen for helping debug his problem, which is being caused by those backslashes. It turns out that XBMC is automatically converting the erroneous backslashes into forward slashes when creating entries in Textures13.db, so reconciling the media library paths (with backslashes) against the texture cache (without backslahes) was problematic...

The script will now inspect the path and replace the slashes whenever it can't find a match, so the overhead should be fairly minimal.

Version 0.8.9

• Fix: Swap back/forward slashes on JSON artwork paths whenever a slash is used incorrectly (see here for details)
  

[teeedubb]

Hi MilhouseVH

I'm getting the following error when tryig to use your script:

Code:

./texturecache.py P
Traceback (most recent call last):
  File "./texturecache.py", line 4031, in <module>
    main(sys.argv[1:])
  File "./texturecache.py", line 3861, in main
    if gConfig.CHECKUPDATE and not argv[0] in ["version","update"]: checkUpdate()
  File "./texturecache.py", line 3779, in checkUpdate
    (remoteVersion, remoteHash) = getLatestVersion()
  File "./texturecache.py", line 3795, in getLatestVersion
    response = urllib2.urlopen("%s/%s" % (gConfig.GITHUB, "VERSION"))
  File "./Lib/urllib2.py", line 126, in urlopen
  File "./Lib/urllib2.py", line 406, in open
  File "./Lib/urllib2.py", line 519, in http_response
  File "./Lib/urllib2.py", line 444, in error
  File "./Lib/urllib2.py", line 378, in _call_chain
  File "./Lib/urllib2.py", line 527, in http_error_default
urllib2.HTTPError: HTTP Error 503: Service Temporarily Unavailable

May it be because GitHub is down?

EDIT: Github is back up and the script is now working.

[teeedubb]

I'm trying to clean up my 12gb userdata/9gb thumbnails directory, but I'm not not having much luck using p/P. The same library scanned into a fresh install of xbmc weighs in at 1.1gb. texturecache a few hundred megabytes on first run and has removed small amounts in certain runs since (like in the log below). If it matters, I have removed all the old *.db files in the ~/.xbmc/userdata/Database directory and currently on have one of each.

./texturecache.py p @logfile=/media/storage/tc.log @logfile.verbose=yes

https://dl.dropboxusercontent.com/u/2241...ogs/tc.log

I have been thinking that I may need to delete the thumbnails folder and use the c/C switch to re-cache everything? This way I wouldn't lose any additional data thats stored in the xbmc database.

[Milhouse]

According to the log, a very large number of files (71,051 to be precise) are being detected as "orphans" - these are files that exist in the Thumbnails folder, but do not have an entry in the Textures13.db database, ie. they were created and referenced by the cache at some point in the past, but are now no longer referenced. These are most likely cached files left over from a previous pre-Frodo installation.

I would suggest running "texturecache.py r" to see details of the orphan files, then "texturecache.py R" to automatically remove them. If there are "too many" files identified as orphans (ie. more than 5% of your entire Thumbnails folder) you will need to enable "orphan.limit.check = no" in the properties file for the script to proceed with removal - only set this property if you think removing this number of thumbnails is reasonable for your system! Since you only have 9,328 rows in your Textures13.db, it would seem entirely reasonable to set/enable the property - the orphaned files really do look like genuine orphaned files and not some error in the script.

Once you have removed the orphaned files, what remains should be files that are indexed within the texture cache. Pruning (p/P) should then identify these files that are known to the texture cache, but have no matching entry in your media library (and should therefore be safe to delete).

I suppose it would be a good idea if the "prune" options notified the user if orphaned files are detected.

Not sure what you mean by

a few hundred megabytes on first run and has removed small amounts in certain runs since

as the files identified by "texturecache.py p" should be consistent from one run to the next (unless you access new artwork in the GUI which would create more "prunable" files).

If you were to run "texturecache.py P", and then follow it up with "texturecache.py p" there should be no files listed as a result of the latter run. If you did see files listed for "p" it would suggest a problem.

Keeping the logs for any runs would be useful just in case!

[teeedubb]

According to the log, a very large number of files (71,051 to be precise) are being detected as "orphans" - these are files that exist in the Thumbnails folder, but do not have an entry in the Textures13.db database, ie. they were created and referenced by the cache at some point in the past, but are now no longer referenced. These are most likely cached files left over from a previous pre-Frodo installation.

I would suggest running "texturecache.py r" to see details of the orphan files, then "texturecache.py R" to automatically remove them. If there are "too many" files identified as orphans (ie. more than 5% of your entire Thumbnails folder) you will need to enable "orphan.limit.check = no" in the properties file for the script to proceed with removal - only set this property if you think removing this number of thumbnails is reasonable for your system!

Once you have removed the orphaned files, what remains should be files that are indexed within the texture cache. Pruning (p/P) should then identify these files that are known to the texture cache, but have no matching entry in your media library (and should therefore be safe to delete).

This worked nicely, now the thumbnails folder is 2.9gb, thanks!

I suppose it would be a good idea if the "prune" options notified the user if orphaned files are detected.

Not sure what you mean by

a few hundred megabytes on first run and has removed small amounts in certain runs since

as the files identified by "texturecache.py p" should be consistent from one run to the next (unless you access new artwork in the GUI which would create more "prunable" files). If you were to run "texturecache.py P", and then follow it up with "texturecache.py p" there should be no files listed as a result of the latter run. If you did see files listed for "p" it would suggest a problem.

Sorry, I should have been clearer - I was using a mix of p and P and the small amounts of deleted files would have been new artwork. No problems with files being listed from p after P.

Thanks for your help Milhouse!

[teeedubb]

I've also disabled dds fanart which was enabled a long time ago and isn't needed with my current setup, after deleting all *.dds fanart the Thumbnails folder is now 810mb

[Milhouse]

Yes... after pruning the artwork items, the corresponding dds files would have been left as orphans - will see if I can clobber the dds files whenever the original artwork is pruned away.

[Martijn]

iirc it is doubtful if the dds files still have any form of performance improvement will all the code improvements done of the time.

[Milhouse]

dds is certainly disabled by default on the Pi, but could still be enabled on an x86 system (or the old dds files could be left lying around after upgrading an x86 system to a more recent version which has dds disabled) so they still need to be processed.

[Maetrik]

When using option "c" i always get this message after movies where checked:

Code:

Caching artwork: 0 items remaining of 320 (qs: 0, qm: 0), 0 errors, 0 threads active (06.10 downloads per second, ETA: 00:00:00)
libSets.ERROR: JSON response has no result!
{u'jsonrpc': u'2.0', u'id': u'libSets', u'error': {u'message': u'Internal error.', u'code': -32603}}

Should i be worried about that? Music and TVShows dont show that error message.

[Milhouse]

What media class are you using to generate this error - "video"?

I guess you have no movie Sets defined... will try and suppress that error, somehow.

[Maetrik]

Yes, i do not have any sets defined. What do you mean with media class?

[Milhouse]

Yes, i do not have any sets defined.

What command are you using to run the script and generate this error?

What do you mean with media class?

A "media class" is one of artists, albums, songs, movies, sets, tags, tvshows, addons, genres.

There are also three "meta media classes": "music" (artists + albums + songs), "video" (movies + sets + tvshows) and all (basically, all media classes).

So, running "texturecache.py c video" would cache movies + sets + tvshows, but since you don't have any Sets defined you are seeing this error.

[Maetrik]

Well, i just ran the c option, ./texturecache.py c

I deleted Textures13.db and ran your script again, so i can post a full "log":

http://pastebin.com/yZc6BqBB (error is in line 283 and 284)

My setup:

Win7 HTPC, MySQL on a Synology NAS, ATV2 and MacBook also connected through MySQL. My .cfg file:

Code:

sep = |
userdata = /Volumes/XBMC/userdata/
dbfile = Database/Textures13.db
thumbnails = Thumbnails/
xbmc.host = 192.168.100.40
webserver.port = 8080
rpc.port = 9090
download.threads = 2
extrajson.albums =
extrajson.artists =
extrajson.songs =
extrajson.movies =
extrajson.sets =
extrajson.tvshows.tvshow =
extrajson.tvshows.season =
extrajson.tvshows.episode=
qaperiod = 30
qa.file = no
cache.castthumb = no
logfile =
logfile.verbose = no
network.mac =

I've mounted the HTPCs XBMC folder on my MacBook so that i can do everthing like scraping and running the script from the OS X machine.