• 1
  • 17
  • 18
  • 19(current)
  • 20
  • 21
  • 197
[RELEASE] Texture Cache Maintenance utility
(2013-08-03, 09:31)RockDawg Wrote: texturecache.py config results - http://pastebin.com/HeYax8MX

texturecache.py jd movies five-year results - http://pastebin.com/dUqLMg44

Ah, qaperiod=30 - it's only considering for QA movies added in the last 30 days. Set qaperiod=9999 in the properties file and try again.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
I should have caught that. Sorry. That returned about 40 or so movies with faile URL .tbn. How does it handle movies with no added date? XBMC used to not record that. and my db is pretty old and a lot don't have a date.
Reply
(2013-08-03, 09:55)RockDawg Wrote: How does it handle movies with no added date? XBMC used to not record that. and my db is pretty old and a lot don't have a date.

That I don't know, as I don't have any movies like that. See if the following returns anything:
Code:
texturecache.py query movies 'dateadded = ""'
texturecache.py query movies 'dateadded < "2005"'
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
You know, I am seeing a lot of old stuff. The db has references to artwork for movies i deleted long ago. I'm wondering if maybe I should just wipe out the db and re-scan everything. Is there anything that will backup/restore the date added info? That would be the only other thing keeping me from doing it.
Reply
Not sure if dateadded can be restored/modified, although by default I think XBMC now uses the modification timestamp of the media and not when an item is actually added to the database, so if you were to rescan you might find that your dateadded values approximate to when you ripped your media.

From advancedsettings.xml (wiki):

Code:
<dateadded>2</dateadded> <!-- 0 results in using the current datetime when adding a video;
                              1 (default) results in prefering to use the files mtime (if it's valid) and only using the file's ctime if the mtime isn't valid;
                              2 results in using the newer datetime of the file's mtime and ctime -->
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
texturecache.py query movies 'dateadded = ""' returned nothing and texturecache.py query movies 'dateadded < "2005"' returned "The system cannot find the file specified."

The original results definitely do not include any movies that don't have a added date. I see some in the db that still show the poster as movie.tbn and are also missing an added date and they didn't turn up. I guess I could run a query that would populate all the null values with a date in the date added field. That would solve the problem.

My db is backed up so I'll play around and see what happens. Thanks for all your help!
Reply
(2013-08-03, 10:08)RockDawg Wrote: texturecache.py query movies 'dateadded < "2005"' returned "The system cannot find the file specified."

Bah, that's just Windows being a PITA and interpreting the "<" as a request to read from a file called 2005... Smile

Try:
Code:
texturecache.py query movies 'dateadded < "2005"'

(2013-08-03, 10:08)RockDawg Wrote: The original results definitely do not include any movies that don't have a added date. I see some in the db that still show the poster as movie.tbn and are also missing an added date and they didn't turn up. I guess I could run a query that would populate all the null values with a date in the date added field. That would solve the problem.

My db is backed up so I'll play around and see what happens. Thanks for all your help!

You could also search for the following lines (somewhere around line 1584-ish) and comment out the last two so that it doesn't apply a dateadded filter - put a "#" at the beginning of each line to make it a comment, or delete both lines... you'll need an editor that understands Unix file formats (Notepad++ or TextPad etc.).
Code:
qaSinceDate = self.config.QADATE
      if qaSinceDate and mediatype in ["movies", "tags", "episodes"]:
          self.addFilter(REQUEST, {"field": "dateadded", "operator": "after", "value": qaSinceDate })
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Also, if you know (or think you know) a movie that probably doesn't have a dateadded field, you can view what is stored with the following, for example for the movie Avatar:
Code:
texturecache.py Jd movies avatar @extrajson.movies=dateadded

Additionally, if you don't specify the movie name you'll get everything returned so maybe you can spot what dateadded value is being used for "old movies"
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2013-08-03, 09:18)MilhouseVH Wrote: Try restarting XBMC, maybe the Remote Communication Server has died in XBMC as this is required for JSON.

(2013-08-03, 08:12)CaptainKen Wrote: Here's my command prompt runs:
1st, 2nd, 3rd and 4th runs

Here's the XBMC log after the 4th run:
XBMC Log

This is the 5th run:
TC.log

EDIT:
The fanart for "Big Buck Bunny" is one of the problem artwork that isn't refreshing.

Thanks for the log... I can see there might be a logic bug (or platform behaviour difference) when forcing a re-cache of items, which is resulting in deleted items not being re-cached correctly, so now it's only downloading 10 items which presumably were populated by you browsing the GUI, and not the script. If you add "@Download.payload=yes" to the commands used for your 5th run, it should re-cache all 82 items correctly (please confirm with another log - many thanks).

Thanks again for your dedication in assisting me!
tc.log

BTW, I mistakenly said one of the issues was with big buck bunny artwork, but it was actually the "blender open movies" fanart. Sorry.

EDIT:
I should have mentioned that this last attempt still did not resolve the issue.
HTPC: Dell Optiplex 7050 SFF i7-7700 quad-core, 3.6GHz, 16GB
NAS: Synology DS1813+ and DX513, Hybrid RAID (SHR) 48TB usable space
My Media Center | www.CaptainKen.us | www.YouTube.com/KenInGilbert
Reply
(2013-08-03, 21:03)CaptainKen Wrote: Thanks again for your dedication in assisting me!

No problem, sorry for the delay responding - enjoying the glorious weather... Smile

(2013-08-03, 21:03)CaptainKen Wrote: tc.log

OK, so that appears to have downloaded all 82 images, I'll release an update shortly that reverts the payload behaviour which clearly isn't the same on all platforms (or maybe it's only in recent builds that the payload isn't required).

(2013-08-03, 21:03)CaptainKen Wrote: BTW, I mistakenly said one of the issues was with big buck bunny artwork, but it was actually the "blender open movies" fanart. Sorry.

EDIT:
I should have mentioned that this last attempt still did not resolve the issue.

What issue are you seeing, exactly?

According to the log, there were two Blender related items:

Code:
smb://SERVER2/video2/Movies Anime Sets/Blender Open Movies/Big Buck Bunny/fanart.jpg
smb://SERVER2/video2/Movies Anime Sets/Blender Open Movies\poster.jpg

No, that's not a typo for the second one - your media library really does have it using both forward and backward slashes. It's actually affecting a lot of your set artwork (ie. Austin Powers, Batman etc., for both posters and fanart) so I imagine it's been caused by an addon, possibly Movie Set Artwork Automator, though not sure why.

Fortunately it doesn't appear to be a serious issue as in your most recent run, both of these Blender artwork items were downloaded successfully so should in theory have been cached correctly.

To determine the cached files, try running "texturecache.py s blender" - the values in the second column (eg. 0/046f50ab.jpg) are the cached file names within the Thumbnails folder. Open the files associated with the set fanart and poster to view them - they should be the same as the corresponding poster and fanart files on your SMB share.

If the cached files are not what you expect to see, check the artwork files being used by your Blender Open Movies set are correct:
Code:
./texturecache.py jd sets blender

Let me know if this helps. If you continue to have problems with your set artwork, I would look to fix those backward slashes first which could prompt unpredictable behaviour elsewhere in XBMC (ie. the backslash is an escape sequence in Linux).
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Version 0.8.8
  • Fix: Revert payload download behaviour, as it's not consistent on all platforms/builds - make download.payload=yes the default setting.
  • Fix: Better handle integer lists in query
  • Add: Extra information in JSON service message mentioning the options in Settings -> Remote Control which should be enabled
  • Add: Use a workable default userdata property for win32
  • Chg: Rename music meta class to audio, for consistency
  • Chg: No longer fabricate a url for local content when Files.PrepareDownload fails
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Updated to latest texturecache.py and ran with payload option again.

tc.log

Code:
C:\Python27>texturecache.py C sets @download.payload=yes @logfile=C:\Temp\tc.log @logfile.verbose=yes

Caching artwork: 0 items remaining of 82 (qs: 0, qm: 0), 0 errors, 0 threads active (01.28 downloads per second, ETA: 00:00:00)

Cache pre-load activity summary for "sets":

              |    fanart   |    poster   |    thumb    |    TOTAL
--------------+-------------+-------------+-------------+-------------
Cached        |      41     |      41     |      -      |      82
Deleted       |      5      |      6      |      -      |      11
Duplicate     |      1      |      1      |      41     |      43
Error         |      -      |      -      |      -      |      0
Ignored       |      -      |      -      |      -      |      0
Skipped       |      -      |      -      |      -      |      0
Undefined     |      -      |      -      |      -      |      0
======================================================================
TOTAL         |      47     |      48     |      41     |     136
Download Time | 00:00:23.88 | 00:00:18.51 |      -      | 00:00:40.44

  Threads Used: 5
   Min/Avg/Max: 0.27 / 1.20 / 1.65

       Loading: 00:00:00.13
       Parsing: 00:00:00.00
     Comparing: 00:00:00.42
   Downloading: 00:00:41.00
TOTAL RUNTIME: 00:00:41.57

One of my issues is this the "Blender Open Movies" fanart.jpg for the collection isn't updating to latest. In looking through the latest log below I don't even see it listed.

There are actually five movies in the Blender collection, but only big buck bunny is in the log.
(Ignore the 2 and 2 dups)
Image

Code:
C:\Python27>texturecache.py jd sets blender
[
  {
    "setid": 42,
    "title": "Blender Open Movies",
    "art": {
      "fanart": "image://smb://SERVER2/video2/Movies Anime Sets/Blender Open Movies/Big Buck Bunny/fanart.jpg/",
      "poster": "image://smb://SERVER2/video2/Movies Anime Sets/Blender Open Movies\\poster.jpg/",
      "thumb": "image://smb://SERVER2/video2/Movies Anime Sets/Blender Open Movies\\poster.jpg/"
    },
    "label": "Blender Open Movies"
  }
]

poster.jog is cached for the blender collection, but fanart.jpg isn't?
texturecache.py s blender

I noticed those backslashes too, and was clueless how they got there and how to remedy? Perhaps this is the root of my problems?

EDIT:
This was another issue I was having and never resolved.
No collection fanart displayed until I is pressed
Video showing issue
HTPC: Dell Optiplex 7050 SFF i7-7700 quad-core, 3.6GHz, 16GB
NAS: Synology DS1813+ and DX513, Hybrid RAID (SHR) 48TB usable space
My Media Center | www.CaptainKen.us | www.YouTube.com/KenInGilbert
Reply
(2013-08-04, 18:25)CaptainKen Wrote: Updated to latest texturecache.py and ran with payload option again.

tc.log

Code:
C:\Python27>texturecache.py C sets @download.payload=yes @logfile=C:\Temp\tc.log @logfile.verbose=yes

Caching artwork: 0 items remaining of 82 (qs: 0, qm: 0), 0 errors, 0 threads active (01.28 downloads per second, ETA: 00:00:00)

Cache pre-load activity summary for "sets":

              |    fanart   |    poster   |    thumb    |    TOTAL
--------------+-------------+-------------+-------------+-------------
Cached        |      41     |      41     |      -      |      82
Deleted       |      5      |      6      |      -      |      11
Duplicate     |      1      |      1      |      41     |      43
Error         |      -      |      -      |      -      |      0
Ignored       |      -      |      -      |      -      |      0
Skipped       |      -      |      -      |      -      |      0
Undefined     |      -      |      -      |      -      |      0
======================================================================
TOTAL         |      47     |      48     |      41     |     136
Download Time | 00:00:23.88 | 00:00:18.51 |      -      | 00:00:40.44

  Threads Used: 5
   Min/Avg/Max: 0.27 / 1.20 / 1.65

       Loading: 00:00:00.13
       Parsing: 00:00:00.00
     Comparing: 00:00:00.42
   Downloading: 00:00:41.00
TOTAL RUNTIME: 00:00:41.57

In the log I'm seeing 11 items deleted, and 82 items downloaded - but I thought you'd already downloaded all 82 items with your previously post (#279)?

This makes me wonder if the download then (and even now) is working. When you run "texturecache.py nc sets" are any items identified as needing to be cached? You should now have zero items needing to be cached, which is to say, if you run "C sets" again, all 82 items should be deleted (then re-cached).

(2013-08-04, 18:25)CaptainKen Wrote: One of my issues is this the "Blender Open Movies" fanart.jpg for the collection isn't updating to latest. In looking through the latest log below I don't even see it listed.

They're both there in the log, however the urls are "quoted" - these are the relevant parts of the log (pastebin).

(2013-08-04, 18:25)CaptainKen Wrote: There are actually five movies in the Blender collection, but only big buck bunny is in the log.
(Ignore the 2 and 2 dups)

Code:
C:\Python27>texturecache.py jd sets blender
[
  {
    "setid": 42,
    "title": "Blender Open Movies",
    "art": {
      "fanart": "image://smb://SERVER2/video2/Movies Anime Sets/Blender Open Movies/Big Buck Bunny/fanart.jpg/",
      "poster": "image://smb://SERVER2/video2/Movies Anime Sets/Blender Open Movies\\poster.jpg/",
      "thumb": "image://smb://SERVER2/video2/Movies Anime Sets/Blender Open Movies\\poster.jpg/"
    },
    "label": "Blender Open Movies"
  }
]

Right, but you're re-caching the set artwork, not the artwork of the movies comprising the set so that's why the movies aren't in the log.

In the case of the Blender set, this has artwork of one poster and one fanart (and also one - legacy - thumb which is a duplicate of the poster, so doesn't need to be cached). These two items (poster and fanart) are being cached as they are present in your Textures13.db according to the results of "s blender":

Code:
011136|f/f588847a.jpg|1080|1920|0001|2013-08-04 15:22:29|2013-08-04 08:22:29|smb://SERVER2/video2/Movies Anime Sets/Blender Open Movies/Big Buck Bunny/fanart.jpg
011142|3/3f6d34bf.jpg|0278|0185|0001|2013-08-04 16:17:04|2013-08-04 09:17:04|smb://SERVER2/video2/Movies Anime Sets/Blender Open Movies\poster.jpg

The fanart image for the set is in the Big Buck Bunny folder because (according to your screenshot) you have misspelled the name of fanart.jpg in the root folder - you've named it fanaart.jpg (two a's). This would explain why MSAA has drilled down into one of the sub folders (Big Buck Bunny) to find a usable fanart.jpg.

(2013-08-04, 18:25)CaptainKen Wrote: I noticed those backslashes too, and was clueless how they got there and how to remedy? Perhaps this is the root of my problems?

I can't think of an easy remedy, unless you can force MSAA to overwrite existing values which I don't think it will do by default. I use MSAA and its always worked fine, but maybe it has a quirk which causes it to incorrectly use backslashes (when run on Windows, perhaps?)

Whether it's a problem or not I don't know - is the GUI displaying you the poster for the Blender set? If it's not, then it's probably the backslash that is the problem. If it is being displayed then nothing to worry about! Smile

(2013-08-04, 18:25)CaptainKen Wrote: EDIT:
This was another issue I was having and never resolved.
No collection fanart displayed until I is pressed
Video showing issue

Do you get those problems with stock Confluence - if not, it's most likely a skin issue. Then again, could be those backslashes...
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Regarding, the backslashes, it's because MSAA is using the os.path.join call to construct the filenames, which is fine in Linux for network shares as it uses the correct forward slash, but when MSAA is run on a Windows box it will use a backslash which you don't want for a share (but would be fine for a local drive, ie. D:\something\something).

So MSAA should replace os.path.join(set_path, fanart_name) with something that joins the path using the forward or backward slash as appropriate based on the existing usage within the file path, and not blindly respecting the operating system.

I'll mention this in the MSAA thread.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
MilhouseVH Wrote:
Quote:The fanart image for the set is in the Big Buck Bunny folder because (according to your screenshot) you have misspelled the name of fanart.jpg in the root folder - you've named it fanaart.jpg (two a's). This would explain why MSAA has drilled down into one of the sub folders (Big Buck Bunny) to find a usable fanart.jpg.

Drats, that's embarrassing! Blender Open Movie fanart now shows.

There are numerous items that need caching.
texturecache.py nc sets @Download.payload=yes @logfile=C:\Temp\tc.log @logfile.verbose=yes
http://pastebin.com/embed_js.php?i=MDnWiqr4

tc.log

Code:
C:\Python27>texturecache.py C sets

Caching artwork: 0 items remaining of 82 (qs: 0, qm: 0), 0 errors, 0 threads active (01.24 downloads per second, ETA: 00:00:00)

Cache pre-load activity summary for "sets":

              |    fanart   |    poster   |    thumb    |    TOTAL
--------------+-------------+-------------+-------------+-------------
Cached        |      41     |      41     |      -      |      82
Deleted       |      3      |      2      |      -      |      5
Duplicate     |      1      |      1      |      41     |      43
Error         |      -      |      -      |      -      |      0
Ignored       |      -      |      -      |      -      |      0
Skipped       |      -      |      -      |      -      |      0
Undefined     |      -      |      -      |      -      |      0
======================================================================
TOTAL         |      45     |      44     |      41     |     130
Download Time | 00:00:18.46 | 00:00:19.09 |      -      | 00:00:35.56

  Threads Used: 5
   Min/Avg/Max: 0.48 / 1.31 / 2.42

       Loading: 00:00:00.16
       Parsing: 00:00:00.01
     Comparing: 00:00:00.19
   Downloading: 00:00:36.05
TOTAL RUNTIME: 00:00:36.44

Thanks, I see that you also posted the backslash issue to MSAA.

EDIT:
Code:
I can't think of an easy remedy, unless you can force MSAA to overwrite existing values which I don't think it will do by default. I use MSAA and its always worked fine, but maybe it has a quirk which causes it to incorrectly use backslashes (when run on Windows, perhaps?)

Whether it's a problem or not I don't know - is the GUI displaying you the poster for the Blender set? If it's not, then it's probably the backslash that is the problem. If it is being displayed then nothing to worry about!
If I run AD with "Overwrite all existing artwork", will it overwrite my local artwork files?

Yes the Blender set poster.jpg has been displaying all along.
HTPC: Dell Optiplex 7050 SFF i7-7700 quad-core, 3.6GHz, 16GB
NAS: Synology DS1813+ and DX513, Hybrid RAID (SHR) 48TB usable space
My Media Center | www.CaptainKen.us | www.YouTube.com/KenInGilbert
Reply
  • 1
  • 17
  • 18
  • 19(current)
  • 20
  • 21
  • 197

Logout Mark Read Team Forum Stats Members Help
[RELEASE] Texture Cache Maintenance utility17