[Milhouse]

Version 0.2.3:

* Added logfile property, to log processing information. To a file.

Version 0.2.4:

* Added cache.ignore.types property to ignore troublesome image://video urls which may or may not work, depending. Disable by setting to None. Multiple image types can be specified, eg. "image://video, image://http" to ignore local video images, and also all remote images. Each pattern is matched against the image url, so you could exclude specific remote web sites too (eg. "cache.ignore.types = image://video, .*imdb.com.*").

* Added additional QA rule to highlight missing artwork (as in, missing from cache). For qax option, this will not initiate a rescan of the directory unless other existing QA rules also fail - if the missing artwork is image://video then try and cache it via the GUI, for other artwork try pre-loading by using the c option.

[ctawn]

@ctawn: new version (0.2.2) uploaded, working with Python 2.6.6 which had the same error you had with 2.5.1, so hopefully it's now fixed. Not sure if I can support 2.5.1 going forward but we'll see (doesn't the ATV2 have a more recent version of Python?)

Also in this version, webserver authentication is now supported - add webserver.username and webserver.password to the properties file.

Thanks Milhouse, for the fast work. This time it fails on a json dependency:

Code:

ATV:/bin root# ./texturecache.py c
Traceback (most recent call last):
  File "./texturecache.py", line 209, in <module>
    import os, sys, ConfigParser, StringIO, httplib, urllib2 as urllib, json, re, datetime
ImportError: No module named json

I don't know if there's a newer Python available, but 2.5.1 is what "apt-get install" yields. I'm actually surprised that XBMC doesn't already depend on python, and that it wasn't installed already. Could not get json via apt-get install either. This is all a little involved for me anyway, so I'm going to get around my problems (hopefully) by reverting to my old userdata folder before I used the Thumbnail Cleaner script, and then deleting all the old pre-Frodo thumbnails (as described by jmarshall).

[charrua]

Version 0.2.3:

* Added logfile property, to log processing information. To a file.

Version 0.2.4:

* Added cache.ignore.types property to ignore troublesome image://video urls which may or may not work, depending. Disable by setting to None. Multiple image types can be specified, eg. "image://video, image://http" to ignore local video images, and also all remote images. Each pattern is matched against the image url, so you could exclude specific remote web sites too (eg. "cache.ignore.types = image://video, .*imdb.com.*").

* Added additional QA rule to highlight missing artwork (as in, missing from cache). For qax option, this will not initiate a rescan of the directory unless other existing QA rules also fail - if the missing artwork is image://video then try and cache it via the GUI, for other artwork try pre-loading by using the c option.

Perfect! A very good call to add the property cache.ignore.types
The log file is also an amazing addition to know in more detail what's going on.
Thanks again for all your excellent work.

Now the bad news
I was testing the other options, and I found that there's something wrong with r/R, as it is showing all the files in the cache as orphaned.
While testing the R option, the script deleted the entire XBMC cache, so I'm now rebuilding it using the c option.
Please let me know if you need any more data on this issue.

Option qax is working partially: it requests XBMC the rescan of the affected sources, but XBMC doesn't refresh the info on the files on that source, probably because the already calculated hash of the source (stored in the media DB in the field path.strHash) has not changed.
Deleting the hash stored in that field and then asking XBMC for a new scan of that source should do the trick.

[Milhouse]

Perfect! A very good call to add the property cache.ignore.types
The log file is also an amazing addition to know in more detail what's going on.
Thanks again for all your excellent work.

Thanks.

Now the bad news
I was testing the other options, and I found that there's something wrong with r/R, as it is showing all the files in the cache as orphaned.
While testing the R option, the script deleted the entire XBMC cache, so I'm now rebuilding it using the c option.
Please let me know if you need any more data on this issue.

Ouch! Sorry about that, but unfortunately it's working fine for me here... can you run the script without parameters and check your config is correct, and pointing at the correct database etc.? Not sure how it could get it quite so spectacularly wrong unless the texture db really was empty... avoiding the R option would be sensible until we can get to the bottom of this.

Option qax is working partially: it requests XBMC the rescan of the affected sources, but XBMC doesn't refresh the info on the files on that source, probably because the already calculated hash of the source (stored in the media DB in the field path.strHash) has not changed.
Deleting the hash stored in that field and then asking XBMC for a new scan of that source should do the trick.

Yeah, that's what I was afraid of. Not sure if I can change the hash as I'm using JSON rather than accessing the media library directly but will have a hunt around, thanks for the tip.

---

Thanks Milhouse, for the fast work. This time it fails on a json dependency:

Code:

ATV:/bin root# ./texturecache.py c
Traceback (most recent call last):
  File "./texturecache.py", line 209, in <module>
    import os, sys, ConfigParser, StringIO, httplib, urllib2 as urllib, json, re, datetime
ImportError: No module named json

I don't know if there's a newer Python available, but 2.5.1 is what "apt-get install" yields. I'm actually surprised that XBMC doesn't already depend on python, and that it wasn't installed already. Could not get json via apt-get install either. This is all a little involved for me anyway, so I'm going to get around my problems (hopefully) by reverting to my old userdata folder before I used the Thumbnail Cleaner script, and then deleting all the old pre-Frodo thumbnails (as described by jmarshall).

To be honest, I have no experience of the ATV2 and no idea what version of Python is running or capable of running and I'm really surprised it's as old as you say, however the lack of json makes me think something isn't quite right with your setup, maybe it's a paths issue. Also, skeeney in this thread had the script working fine on an ATV2, so if you're still interested you might want to follow up or hope someone can suggest something because it should work.

[charrua]

Now the bad news
I was testing the other options, and I found that there's something wrong with r/R, as it is showing all the files in the cache as orphaned.
While testing the R option, the script deleted the entire XBMC cache, so I'm now rebuilding it using the c option.
Please let me know if you need any more data on this issue.

Ouch! Sorry about that, but unfortunately it's working fine for me here... can you run the script without parameters and check your config is correct, and pointing at the correct database etc.? Not sure how it could get it quite so spectacularly wrong unless the texture db really was empty... avoiding the R option would be sensible until we can get to the bottom of this.

No problem, I made a backup of the Thumbnails folder before trying the R option, but I'm using the c option anyway to build it again from scratch.

The config seems fine:

Code:

Current properties (if exists, read from D:\programas\multimedia\XBMC\scripts\texturecache.cfg):
  sep = |
  userdata = D:\programas\multimedia\XBMC\portable_data\userdata\
  dbfile = Database\Textures13.db
  thumbnails = Thumbnails\
  xbmc.host = CHYCH-HTPC001
  webserver.port = 666
  rpc.port = 9090
  extrajson.albums  = None
  extrajson.artists = None
  extrajson.songs   = None
  extrajson.movies  = None
  extrajson.sets    = None
  extrajson.tvshows.tvshow = None
  extrajson.tvshows.season = None
  extrajson.tvshows.episode= None
  qaperiod = 9999 (added after 1985-11-07)
  qarating = False
  cache.castthumb = False
  cache.ignore.types = image://video
  logfile = texturecache.log

See http://wiki.xbmc.org/index.php?title=JSON-RPC_API/v6 for details of available audio/video fields.

And the options s, qa, p and c are also working fine. The problem is only with the r/R options.
Maybe you can add some info to the log file while using those options to try and fix the problem.
Please let me know if you want me to test anything else.

[Milhouse]

Ah... Windows user... penny just dropped. I think I know what happened.

A week or so ago I updated the code to use os.sep on paths to improve Windows compatability (os.sep being "/" on Linux, "\" on Windows), and I also used os.sep when forming the hash (directory + filename) that is calculated as the orphan scan option walks the Thumbnails directory. So on Linux the hash for a file would be formed as "f/ffd2826e.jpg", and on Windows it would be "f\ffd2826e.jpg". This value is then matched against the database hash, to see if it exists or not (is an orphan), and this is where the problem lies... The database hash, I've now realised, always uses the Linux "/" format, so _nothing_ in the database matched against the calculated hashes under Windows, and all files were considered orphans! Oops, very sorry. My fault.

v0.2.5 is updated as follows:

* Fix hash so that always uses Linux path seperator

* Apply 5% limit when identifying files for orphan removal. If the number of orphaned files exceeds 5% of the total database files (rows), the R operation will be aborted before any files are removed.

[charrua]

Ah... Windows user... penny just dropped. I think I know what happened.

A week or so ago I updated the code to use os.sep on paths to improve Windows compatability (os.sep being "/" on Linux, "\" on Windows), and I also used os.sep when forming the hash (directory + filename) that is calculated as the orphan scan option walks the Thumbnails directory. So on Linux the hash for a file would be formed as "f/ffd2826e.jpg", and on Windows it would be "f\ffd2826e.jpg". This value is then matched against the database hash, to see if it exists or not (is an orphan), and this is where the problem lies... The database hash, I've now realised, always uses the Linux "/" format, so _nothing_ in the database matched against the calculated hashes under Windows, and all files were considered orphans! Oops, very sorry. My fault.

v0.2.5 is updated as follows:

* Fix hash so that always uses Linux path seperator

* Apply 5% limit when identifying files for orphan removal. If the number of orphaned files exceeds 5% of the total database files (rows), the R operation will be aborted before any files are removed.

Hehe, I was just about to post that probably the problem was in this line:

Code:

hash = "%s%s%s" % (dir, os.sep, f)

but it seems that you already found out.

The safety measure for the R option is very thoughtful.

[Milhouse]

Option qax is working partially: it requests XBMC the rescan of the affected sources, but XBMC doesn't refresh the info on the files on that source, probably because the already calculated hash of the source (stored in the media DB in the field path.strHash) has not changed.
Deleting the hash stored in that field and then asking XBMC for a new scan of that source should do the trick.

Yeah, that's what I was afraid of. Not sure if I can change the hash as I'm using JSON rather than accessing the media library directly but will have a hunt around, thanks for the tip.

I've had a quick look at the JSON API again, and there's nothing there that seems likely to do the job, so the basic scan is all there is. Unless @jmarshall can suggest anything? I really haven't looked at how the scanner works, but it would be nice if there was a JSON method to force a re-scrape of media, rather than have the scraper do nothing because it doesn't believe there is anything wrong, or that requires updating.

[charrua]

Option qax is working partially: it requests XBMC the rescan of the affected sources, but XBMC doesn't refresh the info on the files on that source, probably because the already calculated hash of the source (stored in the media DB in the field path.strHash) has not changed.
Deleting the hash stored in that field and then asking XBMC for a new scan of that source should do the trick.

Yeah, that's what I was afraid of. Not sure if I can change the hash as I'm using JSON rather than accessing the media library directly but will have a hunt around, thanks for the tip.

I've had a quick look at the JSON API again, and there's nothing there that seems likely to do the job, so the basic scan is all there is. Unless @jmarshall can suggest anything? I really haven't looked at how the scanner works, but it would be nice if there was a JSON method to force a re-scrape of media, rather than have the scraper do nothing because it doesn't believe there is anything wrong, or that requires updating.

Maybe you could use VideoLibrary.RemoveMovie first (or VideoLibrary.RemoveTVShow, VideoLibrary.RemoveTVShow, VideoLibrary.RemoveEpisode, etc.) and then call for a source scan.

[Milhouse]

Yeah, that's what I was afraid of. Not sure if I can change the hash as I'm using JSON rather than accessing the media library directly but will have a hunt around, thanks for the tip.

I've had a quick look at the JSON API again, and there's nothing there that seems likely to do the job, so the basic scan is all there is. Unless @jmarshall can suggest anything? I really haven't looked at how the scanner works, but it would be nice if there was a JSON method to force a re-scrape of media, rather than have the scraper do nothing because it doesn't believe there is anything wrong, or that requires updating.

Maybe you could use VideoLibrary.RemoveMovie first (or VideoLibrary.RemoveTVShow, VideoLibrary.RemoveTVShow, VideoLibrary.RemoveEpisode, etc.) and then call for a source scan.

Hmm, yeah, I'll have a go with RemoveEpisode/RemoveMovie - removing the entire TVShow is possibly a little excessive so hopefully it doesn't come to that! Still, would be nice if it were possible to make the scraper run unconditionally on a specific folder, or even media entry.

[charrua]

Still, would be nice if it were possible to make the scraper run unconditionally on a specific folder, or even media entry.

1+

I've had a quick look at the JSON API again, and there's nothing there that seems likely to do the job, so the basic scan is all there is. Unless @jmarshall can suggest anything? I really haven't looked at how the scanner works, but it would be nice if there was a JSON method to force a re-scrape of media, rather than have the scraper do nothing because it doesn't believe there is anything wrong, or that requires updating.

Maybe you could use VideoLibrary.RemoveMovie first (or VideoLibrary.RemoveTVShow, VideoLibrary.RemoveTVShow, VideoLibrary.RemoveEpisode, etc.) and then call for a source scan.

Hmm, yeah, I'll have a go with RemoveEpisode/RemoveMovie - removing the entire TVShow is possibly a little excessive so hopefully it doesn't come to that!

As a safety measure, it'd be wise reading the media details before asking XBMC to remove it.
After source refresh is done you could even check the new info against the old one and restore fields as needed.

[Milhouse]

Maybe you could use VideoLibrary.RemoveMovie first (or VideoLibrary.RemoveTVShow, VideoLibrary.RemoveTVShow, VideoLibrary.RemoveEpisode, etc.) and then call for a source scan.

Hmm, yeah, I'll have a go with RemoveEpisode/RemoveMovie - removing the entire TVShow is possibly a little excessive so hopefully it doesn't come to that!

As a safety measure, it'd be wise reading the media details before asking XBMC to remove it.
After source refresh is done you could even check the new info against the old one and restore fields as needed.

Good news... removing media items seems to do the trick as far as scanning local NFOs is concerned. Thanks charrua.

As a test, I modified the plot for a tvshow episode NFO, ran the qax option on the tvshow and the episode - as expected - failed QA, it was removed (just the episode) from the media libary, then the directory re-scanned and the new modified plot became visible in the GUI. I've tested the same with a movie, and no problem, it worked fine.

I've uploaded version v0.2.6 with this change, so that qax now removes media items from the media library database before re-scanning - use at your own discretion! Feedback would be most welcome as I've got so few test cases (all my media is local, and pretty much complete).

Regarding the safety measure... probably a lot of work!

[ctawn]

To be honest, I have no experience of the ATV2 and no idea what version of Python is running or capable of running and I'm really surprised it's as old as you say, however the lack of json makes me think something isn't quite right with your setup, maybe it's a paths issue. Also, skeeney in this thread had the script working fine on an ATV2, so if you're still interested you might want to follow up or hope someone can suggest something because it should work.

OK, thanks. as I said over here, I wound up just manually reverting to my /userdata/ folder minus the pre-Frodo thumbnails, and got the results I wanted that way (saving 1 GB). If the XBMC team doesn't implement some cache cleanup internally by the next time I need to reclaim space, I will return to this script! Thanks.

Edit: Note that I see no indication that skeeney got it to work on an ATV2. Looks like he was using Windows.

[Milhouse]

Edit: Note that I see no indication that skeeney got it to work on an ATV2. Looks like he was using Windows.

Post #12? That particular post is discussing an issue with the web server (an issue which is now closed) and appears to be running on an Apple TV2, suggesting there is no problem with the json module.

[charrua]

Hmm, yeah, I'll have a go with RemoveEpisode/RemoveMovie - removing the entire TVShow is possibly a little excessive so hopefully it doesn't come to that!

As a safety measure, it'd be wise reading the media details before asking XBMC to remove it.
After source refresh is done you could even check the new info against the old one and restore fields as needed.

Good news... removing media items seems to do the trick as far as scanning local NFOs is concerned. Thanks charrua.

I'm glad you could make it work, thank you.

Regarding the safety measure... probably a lot of work!

Maybe it's simpler that it seems...
JSON: VideoLibrary.GetMovieDetails (or VideoLibrary.GetEpisodeDetails, etc.) ---> Save to temp nfo file
Just to have a backup of the movie details (or other media) before XBMC deletes it from the DB

And if you want to complicate it a little with a data restoring option:
read info from the temp nfo
JSON: VideoLibrary.SetMovieDetails (or VideoLibrary.SetEpisodeDetails, etc.)