[wishie]

Ok, so my solution was this:

1) Export Library to NFOs
2) Delete MyVideos database
3) Write a script to read the <id>movieid</id> from the NFO and rewrite the NFO containing only the URL for the movie
4) Rescrape and let XBMC fetch the data based on the movie URL in each NFO (to avoid incorrect movie matching)

Once that is all done..

5) Will probably run TextureCache.py to correct any missing/broken images

[Milhouse]

Sounds like a plan. Once you have scraped your library, you could then run:

Code:

./texturecache.py missing movies <movie source>

to identify any movies that were not scraped.

[wishie]

Really? Thats interesting, because I have 1607 movies, and only 1601 made it into the library..

--EDIT--

So I just tried that, and I get the following errors:

./texturecache.py missing movies Movies
Traceback (most recent call last):0/mnt/user/Movies]
File "./texturecache.py", line 3921, in <module>
main(sys.argv[1:])
File "./texturecache.py", line 3900, in main
jsonQuery(action="missing", mediatype=argv[1], labels=argv[2:])
File "./texturecache.py", line 2093, in jsonQuery
fileList = jcomms.getAllFilesForSource(mediatype, labels)
File "./texturecache.py", line 1359, in getAllFilesForSource
for file in self.getFilesForPath(path):
File "./texturecache.py", line 1384, in getFilesForPath
self.getFilesForPath_recurse(fileList, path)
File "./texturecache.py", line 1401, in getFilesForPath_recurse
self.getFilesForPath_recurse(fileList, os.path.dirname(fname))
File "./texturecache.py", line 1391, in getFilesForPath_recurse
files = data["result"]["files"]
KeyError: 'files'


Perhaps this is because the "Movies" source is:

./texturecache.py sources
video: Movies: nfs://192.168.1.10/mnt/user/Movies

[Milhouse]

Really? Thats interesting, because I have 1607 movies, and only 1601 made it into the library..

Thanks again for your help, and awesome script.

The "missing" functionality is a tad experimental (as I'm not able to test with Blu-Ray and DVD disk rips) but let me know if you have any problems.

[Milhouse]

Perhaps this is because the "Movies" source is:

./texturecache.py sources
video: Movies: nfs://192.168.1.10/mnt/user/Movies

No, it should work, although maybe one of your directories is completely empty (just a guess). Can you run:

Code:

./texturecache.py missing movies Movies @logfile=/tmp/tc.log @logfile.verbose=yes

and upload /tmp/tc.log to pastebin.com (if you're on OpenELEC, "pastebinit -u /tmp/tc.log" and paste the URL).

[AnotherPhil]

Ok, my understanding is ...

The [P]rune command deletes all orphaned images from the cache.

It's the [R]everse query option that locates "orphaned" files and removes them.

The role of Textures13.db is to act as an "index" for files in the Thumbnails folder using a "hash" identifier which is also the filename of each thumbnail file (eg. 2/276c1eb0.jpg). The "[R]everse lookup" option will scan the Thumbnails folder retrieving all of the hashes and then try to find the corresponding rows in the Textures13.db - hence the "reverse" nature of this option. Any thumbnail file for which a row can't be found will be considered an "orphan" and removed - without a row in Textures13.db such files are just wasting file space.

Orphaned files are rare, but most likely to appear following an upgrade (eg. Eden to Frodo) when old Eden thumbnails are left unreferenced by the updated Textures13.db used by Frodo.

The [P]rune option on the other hand will match cached files (ie. a row in Textures13.db) with the original artwork in the Media Library (MySQL etc.). Any cached file that cannot be matched to an artwork file in the Media Library will be removed.

Typically the files to be pruned are "preview" thumbnails and other transient artwork that may be downloaded by plugins (eg. BBC iPlayer episode thumbnails), but can also include artwork associated with movies and TV shows that have been removed from the media library.

The [C]ache command issued for a particular movie, re-downloads & caches the movies' poster and fanart images from the existing URLs.

Correct.

What I'm after is a quick way to remove whatever remnants are left after issuing a JSON "VideoLibrary.RemoveMovie" command. i.e. Delete all the cached images & database records relating to the removed movie. Is this already possible with your script or is the logic there, but needs rejigging into another command?

If you know what artwork needs to be deleted, you could try the [s]earch option and then [d]elete cached items based on the row ids found by [s]earch, but it's unlikely you will know all of the artwork references to search for, in which case the current [P]rune option is still the best method - while slow, this will identify ALL of the artwork that is no longer referenced by the Media Library, including previously removed movies.

So it seems I have a database (MySQL) full of incorrect URLs for thumbnails and other artwork.. so when I exported my library to NFOs, they now too contain invalid/incorrect URLs..

Is there a way I can simply tell your script to fetch everything new from the internet, ignoring the URLs in the NFO files?

Also, how to remove all the invalid entries from my database?

No, sorry - that's not how it works.

The URLs you have in your media library determine what artwork is cached and displayed by XBMC, so even if it were possible for this script to cache alternative URLs, XBMC would still be using the URLs specified by your media library and you'd just end up with both artwork URLs being cached and the artwork from the "incorrect" URLs still appearing on the display. In short, it would be a waste of time and disk space caching "alternative" URLs, as XBMC (driven by your media library) would know nothing about them.

Correcting the URLs in the Media Library is the only option, which you can of course do via the Info screen and "Choose Art" button, but assuming we're talking about more than just a handful of movies/tv shows this quickly becomes an impractical solution.

My advice would be to fix the URLs in the NFO files using a media manager such as Ember or alternatively delete all the NFO files if they're not worth keeping and you're happy with internet-sourced artwork; drop your MySQL databases (MyMusic and MyVideo); restart XBMC so that it re-creates empty databases; and then finally re-scrape your library.

Most excellent of you once again my Good Man, thanks to your help I can now delete the orphaned poster & fanart thumbnails after a "VideoLibrary.RemoveMovie"

http://mediacompanion.codeplex.com/Sourc...changesets

[wishie]

Perhaps this is because the "Movies" source is:

./texturecache.py sources
video: Movies: nfs://192.168.1.10/mnt/user/Movies

No, it should work, although maybe one of your directories is completely empty (just a guess). Can you run:

Code:

./texturecache.py missing movies Movies @logfile=/tmp/tc.log @logfile.verbose=yes

and upload /tmp/tc.log to pastebin.com (if you're on OpenELEC, "pastebinit -u /tmp/tc.log" and paste the URL).

That was indeed the problem.. I had 3 empty directories..
Once I fixed those, it ran correctly and I identified a few things that didn't make it into the library.. some where .sfv files and other various stuff that i dont usually keep, so i deleted those.. The only thing left now is movies that are in parts... I assume your script doesn't notice them in the database because of the way XBMC stores the file path for movies in multiple parts (-part1 and -part2 or cd1 and cd2 as an example).

In any event, your script has proved invaluable in fixing my very broken artwork collection. Thank you very much.

[Milhouse]

Did you manage to produce a log? I want to fix the code so that empty folders are not a problem... I'm pretty sure I know where the problem is, but despite my best efforts today I've been unable to reproduce the problem so your log would have been most useful.

The only thing left now is movies that are in parts... I assume your script doesn't notice them in the database because of the way XBMC stores the file path for movies in multiple parts (-part1 and -part2 or cd1 and cd2 as an example).

Can you run ".texturecache.py Jd movies <multipart-movie-name> @extrajson.movies=file" for one of your multi-part movies and I'll see if I can take multi-part movies into consideration...

[HueyHq]

Because you seem like the right man to ask; does Textures13.db still get populated when XBMC is MySQL-based? My assumption was that MySQL negated the use of SQLite, but I thought I read somewhere here that texturecache.py keeps them in sync?

In either case, what does texturecache.py do with MySQL data?

[Milhouse]

Because you seem like the right man to ask; does Textures13.db still get populated when XBMC is MySQL-based? My assumption was that MySQL negated the use of SQLite, but I thought I read somewhere here that texturecache.py keeps them in sync?

Yes, it still gets populated, switching to MySQL has no impact on Textures13.db

MySQL replaces the MyVideos75.db and MyMusic32.db files (the movie/tvshow and music media libraries, respectively) however Textures13.db remains as a local SQLite database on each and every client because it manages the local artwork cache that is unique to each client.

For every artwork image you have in your media library (whether it be MySQL or SQLite based), an optimised version - resized/re-sampled as appropriate, depending on client capabilities - is cached in the Thumbnails folder on the client and an entry for that cached file is added to Textures13.db.

Each time XBMC wants to display an image, if there is a row for the image in Textures13.db it will pull the cached file from the Thumbnails folder - if not, it will download the original file (which could be on the internet), create the cached image file in Textures13.db/Thumbnails, then use the cached file.

In either case, what does texturecache.py do with MySQL data?

The texturecache.py script accesses the XBMC media library data using JSON so it makes no difference to the script if XBMC is configured to use MySQL or SQLite.

[wishie]

Here is a log file (thankfully it was still in my clipboard)

https://dl.dropboxusercontent.com/u/1373...ecache.log

The folder "Doghouse" was empty (so were some others, but this was the first one the script encountered

I will try your other command for multipart movies shortly.

---

Here is the other info you requested:

./texturecache.py Jd movies "The Pretender" @extrajson.movies=file
A new version of this script is available - use the "update" option to automatically apply update.

[
{
"movieid": 1377,
"title": "The Pretender: Island of the Haunted",
"art": {
"fanart": "image://http://cf2.imgobject.com/t/p/original/bTFrf7bz0RiwgEiLHysdkOVIRap.jpg/",
"poster": "image://http://cf2.imgobject.com/t/p/original/cEHkptTqhynfGb09BJ0XNtc8VjX.jpg/"
},
"file": "stack://nfs://192.168.1.10/mnt/user/Movies/The Pretender/The Pretender The Island of the Haunted (Part-1).avi , nfs://192.168.1.10/mnt/user/Movies/The Pretender/The Pretender The Island of the Haunted (Part-2).avi , nfs://192.168.1.10/mnt/user/Movies/The Pretender/The Pretender The Island of the Haunted (Part-3).avi , nfs://192.168.1.10/mnt/user/Movies/The Pretender/The Pretender The Island of the Haunted (Part-4).avi",
"label": "The Pretender: Island of the Haunted"
}
]

[Milhouse]

Here is a log file (thankfully it was still in my clipboard)

https://dl.dropboxusercontent.com/u/1373...ecache.log

The folder "Doghouse" was empty (so were some others, but this was the first one the script encountered

Many thanks for that, most odd that I couldn't reproduce it - for an empty folder I get a "files" item with a "null" value (ie. no files) but you have a response that doesn't include the "files" item at all. Either way it shouldn't be a problem with the next version of the script.

I will try your other command for multipart movies shortly.

Actually no need to worry about that - I mocked up a couple of multi-part files and scraped them in to my library and can see they are being handled as stacked files which is now supported in the new version of the script. Thanks for reporting the problem.

Version 0.8.6 (04/07/2013)

• Fix: When looking for `missing` (unscraped) media items, allow for empty remote folders (no files)
  
• Add: Include `file` as a default field when dumping (`j`/`jd`/`J`/`Jd`) media library items
  
• Add: Stacked/multi-part movie support to `missing` and `qa`/`qax` options
  
• Add: Group similar `prune` items together, by sorting results on url
  

[HueyHq]

Wow - thanks for replying so quickly!

...however Textures13.db remains as a local SQLite database on each and every client because it manages the local artwork cache that is unique to each client.

You make particular reference to client DBs, so what happens on the server machine?

The reason I ask is because, against the trend, I use path substitution for the Thumbnails folder. In Frodo.
(I know, I know, I don't get the performance, but man, does it make maintenance sooo much easier! Each new Pi on my network gets the same advancedsettings.xml thrown on an OpenELEC install - no fuss, no stress, and it just works! AFAIK, nothing pertaining to the media is stored on the Pi - it all comes from the server.)

If access is done via JSON, will Textures13.db still be relevant in my case?

I'm curious to see if I delete Textures13.db, whether or not XBMC will recreate it!

[wishie]

Here is a log file (thankfully it was still in my clipboard)

https://dl.dropboxusercontent.com/u/1373...ecache.log

The folder "Doghouse" was empty (so were some others, but this was the first one the script encountered

Many thanks for that, most odd that I couldn't reproduce it - for an empty folder I get a "files" item with a "null" value (ie. no files) but you have a response that doesn't include the "files" item at all. Either way it shouldn't be a problem with the next version of the script.

I will try your other command for multipart movies shortly.

Actually no need to worry about that - I mocked up a couple of multi-part files and scraped them in to my library and can see they are being handled as stacked files which is now supported in the new version of the script. Thanks for reporting the problem.

Version 0.8.6 (04/07/2013)

• Fix: When looking for `missing` (unscraped) media items, allow for empty remote folders (no files)
  
• Add: Include `file` as a default field when dumping (`j`/`jd`/`J`/`Jd`) media library items
  
• Add: Stacked/multi-part movie support to `missing` and `qa`/`qax` options
  
• Add: Group similar `prune` items together, by sorting results on url
  

Should searching for empty folders be an option?
In my case, I was glad I could find them, and therefore remove them.

---

Wow - thanks for replying so quickly!

...however Textures13.db remains as a local SQLite database on each and every client because it manages the local artwork cache that is unique to each client.

You make particular reference to client DBs, so what happens on the server machine?

The reason I ask is because, against the trend, I use path substitution for the Thumbnails folder. In Frodo.
(I know, I know, I don't get the performance, but man, does it make maintenance sooo much easier! Each new Pi on my network gets the same advancedsettings.xml thrown on an OpenELEC install - no fuss, no stress, and it just works! AFAIK, nothing pertaining to the media is stored on the Pi - it all comes from the server.)

If access is done via JSON, will Textures13.db still be relevant in my case?

I'm curious to see if I delete Textures13.db, whether or not XBMC will recreate it!

Ive just spent nearly a day undoing the mess that a shared Thumbnails did caused. I used to use path sub for it too, thinking it was a good idea, even though I was told otherwise.
The performance on the clients suffer (especially if Pi/ATV2/Android devices), network load increases, etc etc.. Its simply not worth it.

My understanding now is, if you remove the Textures.db and path sub from the clients, they will automatically download the artwork (based from the URLs stored in the video library) store them in its own Thumbnails cache and build Textures db locally with optimised images. This may be a pain to start with, but will be a very minimal (and automated thing) afterwards.

You end up with less network traffic, and faster client performance.

I still use MySQL for my video and music libraries, but artwork should be unique for each client. I guess there is no harm having a shared thumbnail cache for 2 Pi's or 2 ATV2's or something, but I still wouldn't do it.

[Milhouse]

Should searching for empty folders be an option?
In my case, I was glad I could find them, and therefore remove them.

Empty folders are not particularly relevant as far as XBMC or the media library is concerned, so I'm not sure there needs to be an option to detect them.

It should also be relatively easy to identify empty folders on whatever OS you are using (OK, that's a lie - probably a pain in the @rse on Windows, but it's trivial on Linux: "find . -type d -empty").