• 1
  • 55
  • 56
  • 57(current)
  • 58
  • 59
  • 197
[RELEASE] Texture Cache Maintenance utility
I saw you discovered it was to do with & and other punctuation present in the filename - good find, hopefully it will be fixed.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
well one was because & was present in filename, the other was that it needed punctuation in the filename and other was because it had versus and changing it to vs worked, so, very odd it is. In one case it was th official vs bootleg status in tags.

I know it was those because I tried renaming it as well and if renaming was trigger it would have worked before that and didnt.
Reply
I am attempting to understand your python scripts, as the built-in AD has some limitations that don't work in my specific situation. I've read almost the entire thread, but still have lots of questions.

I'll start off easy.

I'm on Windows 7. I have installed the latest Python 3.x. I have all .py files in the same folder.

When I run mklocal.py --help, I get:
Code:
"E:\Downloads\XBMC Backup\TextureCache\mklocal.py", line 6
<!DOCTYPE html>
^
Syntax Error: invalid syntax
Any advice?

Edit: Turns out I didn't download the correct mklocal.py script. I had some weird html file instead of python code. Trying to decipher options now.
Reply
Hi

This tool has significantly increased the performance of my rpi. Artworks of my movies and tvshows finally work smoothly. Its fun! Thanks a lot!

What about pre-caching of my private picture collection? Is this supported? In my case the previews are still created on-the-fly, which puts too much load on my rpi.

Am I doing something wrong? How can I pre-cache my pictures library too?

Kind regards,
Sonox
Reply
(2014-03-16, 14:45)sonox Wrote: What about pre-caching of my private picture collection? Is this supported? In my case the previews are still created on-the-fly, which puts too much load on my rpi.

Am I doing something wrong? How can I pre-cache my pictures library too?

Picture folder thumbnails (the little composite folder images) are created by the GUI as you browse your picture library and can't currently be pre-loaded by JSON, however such thumbnails, once cached, will be retained by default when pruning (controlled by @prune.retain.previews).

The pictures themselves will not be preloaded into the cache, for several reasons: 1) I didn't have much success caching pictures via JSON; 2) I'm not sure it would be a good idea to have your entire picture library reproduced in your cache, and 3) the standard slideshow facility doesn't appear to use the cache as it doesn't store in the cache the pictures it shows, suggesting the cache is being by passed which is understandable and reasonable. However some addons do dump your pictures in the cache, which is probably not (IMHO) a good thing (think: cache bloat). When pruning the cache, any pictures found in the cache will be deleted by default unless you enable @prune.retain.pictures.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
I am unable to make logos and discart show up in XBMC with these scripts. I am not sure where the problem is.

Specifically, if I run the AD and use it to scan my local artwork, all types (poster, fanart, banner, logo, clearart, landscape, discart, etc...) will properly display themselves in the right places in XBMC. This is the output of
Code:
TextureCache>texturecache.py s "Hunger"
when AD scans and integrates local artwork.

However, if I use texturecache.py and mklocal.py on the same local artwork, all appropriate artwork shows up EXCEPT logos and discart.

My workflow is:
  1. Create test.dat to check if changes are correct.
    Code:
    texturecache.py jd movies | tools\mklocal.py --singlefolders --local "E:\Videos to Archive\Movies - Adult" --prefix "E:\Videos to Archive\Movies - Adult" --artwork poster fanart logo clearart disc banner landscape --verbose --output test.dat
    which outputs this.

  2. Make actual changes to database.
    Code:
    texturecache.py jd movies | tools\mklocal.py --singlefolders --local "E:\Videos to Archive\Movies - Adult" --prefix "E:\Videos to Archive\Movies - Adult" --artwork poster fanart logo clearart disc banner landscape --verbose --output | texturecache.py set
    which outputs this.

  3. Cache artwork
    Code:
    texturecache.py c movies
    which outputs this.

  4. Check database.
    Code:
    texturecache.py s "Hunger"
    which outputs this.

For completeness, my config file is here. I am running XBMC Gotham Beta 1 on Windows 7.

Any help or ideas as to why only the logo and discart do not show up?
Reply
(2014-03-16, 16:35)MilhouseVH Wrote:
(2014-03-16, 14:45)sonox Wrote: What about pre-caching of my private picture collection? Is this supported? In my case the previews are still created on-the-fly, which puts too much load on my rpi.

Am I doing something wrong? How can I pre-cache my pictures library too?

Picture folder thumbnails (the little composite folder images) are created by the GUI as you browse your picture library and can't currently be pre-loaded by JSON, however such thumbnails, once cached, will be retained by default when pruning (controlled by @prune.retain.previews).

The pictures themselves will not be preloaded into the cache, for several reasons: 1) I didn't have much success caching pictures via JSON; 2) I'm not sure it would be a good idea to have your entire picture library reproduced in your cache, and 3) the standard slideshow facility doesn't appear to use the cache as it doesn't store in the cache the pictures it shows, suggesting the cache is being by passed which is understandable and reasonable. However some addons do dump your pictures in the cache, which is probably not (IMHO) a good thing (think: cache bloat). When pruning the cache, any pictures found in the cache will be deleted by default unless you enable @prune.retain.pictures.

Thanks for the fast reply!

My focus is simply to increase the feel and fun of my media on the rpi. I believe there must be a way to improve this with picture libraries too. You mention three reasons:

I can't say much about the first one regarding JSON, because I have not studied your script I do not know how xbmc is creating those photo previews. However, I'm sure there is a way to make this work. Let me quote one of the statements you made some time ago:

(2013-03-08, 10:00)MilhouseVH Wrote: This is much easier than laboriously browsing through your media library while the Pi slowly re-caches all the missing items!

This is exactly what I'm still forced to do at the moment with my pictures library to increase performance. As you say, it is laboriously, especially if pictures are organized in many different folders.

You say it wouldn't be good to reproduce the entire library. Well, why not? Is this not what your script is doing with artwork? Is this not working very well and increasing performance significantly? Why shouldn't this work with pictures? You are afraid of cache bloating? Well, maybe you are right. However, some picture libraries are not too big and pre-caching could help a lot.

The slideshows are working fine. I don't have performance issues there. My problem occurs when I have to search through my pictures library to find a particular one. That's when I need fast-displaying previews.

You say, there are addons out there which can pre-cache the picture library aswell? Can you please direct me?

Kind regards,
Roman
Reply
(2014-03-16, 21:04)sonox Wrote: I can't say much about the first one regarding JSON, because I have not studied your script I do not know how xbmc is creating those photo previews. However, I'm sure there is a way to make this work. Let me quote one of the statements you made some time ago:

If you find a way, let me know. That statement of mine was made long before any support was added for pictures, and only applies to artwork that is capable of being pre-loaded via JSON, which doesn't include the artwork you want to cache.

(2014-03-16, 21:04)sonox Wrote: This is exactly what I'm still forced to do at the moment with my pictures library to increase performance. As you say, it is laboriously, especially if pictures are organized in many different folders.

I'm not a miracle worker! Unless JSON can create the previews, which it can't, I'm not able to pre-load the cache. Since the slideshow doesn't use the cache, filling the cache with pictures seems to be largely a waste of time.

(2014-03-16, 21:04)sonox Wrote: You say it wouldn't be good to reproduce the entire library. Well, why not? Is this not what your script is doing with artwork? Is this not working very well and increasing performance significantly? Why shouldn't this work with pictures? You are afraid of cache bloating? Well, maybe you are right. However, some picture libraries are not too big and pre-caching could help a lot.

I can understand that, but since the slideshow doesn't appear to make use of the cache you'd gain nothing by filling your cache with downgraded copies of your pictures.

(2014-03-16, 21:04)sonox Wrote: The slideshows are working fine. I don't have performance issues there. My problem occurs when I have to search through my pictures library to find a particular one. That's when I need fast-displaying previews.

I've not been able to get the previews to pre-load into the cache using JSON. If you find a way that involves JSON, let me know.

(2014-03-16, 21:04)sonox Wrote: You say, there are addons out there which can pre-cache the picture library aswell? Can you please direct me?

I didn't mean to you the impression there are addons that pre-cache pictures - what I'm saying is that, unlike the slideshow feature, there are picture-based addons out there that do foul the cache with picture artwork. I'm not sure which addons though, I've only seen evidence of what they leave behind (in the cache). But at risk of repeating myself, even if your pictures are in your cache, it's not obvious to me that the slideshow will even use the cached artwork.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2014-03-16, 19:41)radigast Wrote: I am unable to make logos and discart show up in XBMC with these scripts. I am not sure where the problem is.

Thanks for the various logs, but can you upload the output from "texturecache.py jd movies" so I can see what your source data is looking like? I think then it should become fairly clear.

What is apparent is that your library has paths with a mixture of forward and backward slashes, these forward slashes are being added by buggy addons (usually). First thing is to fix these before any further manipulation of your library.

Run:
Code:
texturecache.py fixurls | texturecache.py set
to fix the slashes in your library.

As for "disc" and "logo" not appearing - you should be specifying to mklocal.py the artwork types as used by XBMC. "logo" and "disc" are not valid artwork types as far as XBMC is concerned - artwork that ends in "-logo" corresponds to the "clearlogo" type and artwork that ends in "-disc" corresponds to "discart".

Since you have now created "logo" and "disc" artwork in your library you may want to remove them (although if you leave them it will do no harm - they'll just be ignored, as you've discovered).

To remove an artwork item you set it's value to "" (although this is only supported if you are running a recent Gotham build), eg:
Code:
texturecache.py set movie 2 art.logo "" art.disc ""
where 2 is the movieid of your Hunger Games movie. The above will remove both the logo and disc artwork from your movie.

Then run your mklocal.py command as follows:
Code:
texturecache.py jd movies | tools\mklocal.py --singlefolders --local "E:\Videos to Archive\Movies - Adult" --prefix "E:\Videos to Archive\Movies - Adult" --artwork poster fanart clearlogo clearart discart banner landscape --verbose --output | texturecache.py set

and this should find and load your clearlogo and discart artwork.

When you specify artwork types to mklocal.py you can also specify the associated suffix, using typeConfuseduffix notation. mklocal.py has special logic to handle clearlogo and discart, otherwise you would need to specify clearlogo:logo and discart:disc when processing these types. Most artwork maps the type to the suffix, it's just these two that I'm aware of that require special processing (hence the extra support that is already present) but if there are more you can pass the details on the command line using the typeConfuseduffix notation.

Also, if you have your library organised with individual movies in their own folders, you may want to add the --single argument to mklocal.py - it should have no effect unless you are downloading remote artwork in which case it will dictate how the artwork is named (with --single it will not use the movie name as a prefix).

One last thing, I'm not sure what's going on with your extrathumbs - they seem to be present twice in your cache, once with a single backslash and again with two backslashes - can you confirm if the latter is being created by texturecache.py caching the thumbs (delete the thumbs from the cache, then cache your movie again and see what thumbs are created)? That will need fixing as it's likely that the cached version with two backslashes is being ignored by the GUI. A logfile would be handy when you perform the cache (add @logfile=c:\tc.log).
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Thank you for your constructive feedback and support MilhouseVM. Let me address each part of your post point by point.

First, my movie database is only 2 movies large, as I am simply learning your script on this small test database.

MilhouseVH Wrote:What is apparent is that your library has paths with a mixture of forward and backward slashes, these forward slashes are being added by buggy addons (usually). First thing is to fix these before any further manipulation of your library.
The only addon I used is Artwork Downloader, and the information I posted at the beginning of my previous post showed this virgin run of AD on my brand new database. It seems that it is the culprit for the reversed slashes for landscape, clearart, logo, and disc images. I won't be using Artwork Downloader for my real database once I become more competent with your script. I was simply using it as a comparison to see how your script adds pictures to the database so I could troubleshoot my own problems with logos/discart. But, thanks for the tip to fix the slashes should I ever need it. What would happen if these reverse slashes were never fixed in the database?

MilhouseVH Wrote:Then run your mklocal.py command as follows:
Code:
texturecache.py jd movies | tools\mklocal.py --singlefolders --local "E:\Videos to Archive\Movies - Adult" --prefix "E:\Videos to Archive\Movies - Adult" --artwork poster fanart clearlogo clearart discart banner landscape --verbose --output | texturecache.py set

and this should find and load your clearlogo and discart artwork.
Thanks for the tip on the clearlogo and discart variable mistakes. I cleaned the database and reran the command with the updated variables for clearart and discart. Everything showed up! Worked like a boss! Thanks! My mistake with disc and clearart was because I wasn't aware that XBMC called them differently than the actual filename AD give them. In the future, I am also going to be scanning in TV shows. Where do I find a complete list of the actual proper artwork variable names for TV shows for mklocal.py so that I don't make silly naming mistakes again?

MilhouseVH Wrote:When you specify artwork types to mklocal.py you can also specify the associated suffix, using typeConfuseduffix notation. mklocal.py has special logic to handle clearlogo and discart, otherwise you would need to specify clearlogo:logo and discart:disc when processing these types. Most artwork maps the type to the suffix, it's just these two that I'm aware of that require special processing (hence the extra support that is already present) but if there are more you can pass the details on the command line using the typeConfuseduffix notation.
So...if I called my clearlogo file "prettyandcolorful.png", then I would use clearlogo:prettyandcolorful in the script instead of clearlogo?

MilhouseVH Wrote:can you upload the output from "texturecache.py jd movies"
MilhouseVH Wrote:One last thing, I'm not sure what's going on with your extrathumbs - they seem to be present twice in your cache, once with a single backslash and again with two backslashes - can you confirm if the latter is being created by texturecache.py caching the thumbs (delete the thumbs from the cache, then cache your movie again and see what thumbs are created)? That will need fixing as it's likely that the cached version with two backslashes is being ignored by the GUI. A logfile would be handy when you perform the cache (add @logfile=c:\tc.log).
I will do both later today on a fresh database when I get home from work.

Two final unrelated questions:
  1. What command would I use to get the database ID number? For example, my Hunger Games database ID was 2. However, the only way I knew that it was 2 is because it was the second movie I added in the database. If I have 1000 movies, what command could I use to find the ID number of any one of those movies stored in the database?
  2. Does your script also download missing artwork from fanart.tv or other sources? The posts I have read on here seem to suggest it will, but I have tried on movies missing all artwork and none were downloaded from fanart.
Thanks again for your all of your help and support! Your friendly and helpful posts are a breath of fresh air!
Reply
(2014-03-17, 04:50)radigast Wrote: What would happen if these reverse slashes were never fixed in the database?

It just causes excessive caching as XBMC fails to find items that have already been cached with the wrong slashes.

(2014-03-17, 04:50)radigast Wrote: Where do I find a complete list of the actual proper artwork variable names for TV shows for mklocal.py so that I don't make silly naming mistakes again?

Good question. I'm not aware of a definitive list, apart from the small number of standard artwork types supported by XBMC (see here for the most concise listing).

Generally speaking, the "type" is often the same as the filename suffix, the exceptions being discart (<filename>-disc.png/disc.png etc.) and clearlogo (<filename>-logo.png/logo.png). The types I'm familiar with are poster, fanart, banner, landscape, thumb, clearart, clearlogo and discart. There are undoubtedly others too as there is no restriction on what artwork types can be added to the database (I think cdart may be another type, but is likely only used or understood by specific skins and addons).

(2014-03-17, 04:50)radigast Wrote: So...if I called my clearlogo file "prettyandcolorful.png", then I would use clearlogo:prettyandcolorful in the script instead of clearlogo?

Yep.

(2014-03-17, 04:50)radigast Wrote: What command would I use to get the database ID number? For example, my Hunger Games database ID was 2. However, the only way I knew that it was 2 is because it was the second movie I added in the database. If I have 1000 movies, what command could I use to find the ID number of any one of those movies stored in the database?

If you run
Code:
texturecache.py jd movies
then it's the "movieid" property corresponding to each movie.

If you want details for a specific movie, the following:
Code:
texturecache.py jd movies "hunger games"
should return details of any movie with "Hunger Games" in the title.

However if you want to narrow it down further:
Code:
texturecache.py jd movies "the hunger games" @filter.operator=is
will match the movie title exactly.

(2014-03-17, 04:50)radigast Wrote: Does your script also download missing artwork from fanart.tv or other sources? The posts I have read on here seem to suggest it will, but I have tried on movies missing all artwork and none were downloaded from fanart.

Not exactly, as neither texturecache.py or mklocal.py are scrapers.

What mklocal.py will do is "convert to local" any artwork you already have in your library that is remote (ie. the url begins with http and is therefore not local) by downloading the original artwork from the source web server - which could be fanart.tv or any other artwork service - and write it into your local file system using the correct single-folder or multi-folder naming convention. It can also assign any new local artwork to you library.

For instance, I use Artwork Downloader myself to find new clearart/clearlogo artwork for my movies, but because AD doesn't support movie-name prefixes it can't write the artwork into the local file system alongside the movies. So after running AD I run mklocal.py which then downloads any remote artwork now present in my library and writes it into my file system, and then the media library is updated to replace this remote artwork with the local version. In addition, any artwork I happen to find myself that I drop into my folders alongside the movies will be automatically picked up and added to the library next time mklocal.py is run.

All of the above is handled with a call to "texturecache.py jd movies | mklocal.py ... | texturecache.py set", and is entirely automated so I don't have to worry about this stuff! Smile

And if you're maintaining/curating your own clearart/clearlogo/discart/whatever artwork then you could just load it using mklocal.py without using Artwork Downloader at all. mklocal.py will only match new or changed items so can be run regularly to find and set new items.

All of the above should work for tv shows too. Just look at the options on mklocal.py --help as you need to enable --season and --episode if you also want mklocal.py to process season and episode artwork (by default it will only process the tvshow "header"). Loading missing artwork this way will also work around this defect as any missing season posters will be added by mklocal.py --season next time it runs and the poster is found locally (although the ability to set season posters using JSON is only a recent Gotham feature, so this isn't supported in Frodo).
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Please forgive me if this isn't' the place to ask this question but here goes.

I installed the texturecache.py script in my /tmp directory on my pi and configured several aliases to call the script. Everything worked like a charm until I did a reboot of the pi. When I performed the reboot it appears that the texturecache.py script was deleted from my /tmp directory. I have since downloaded it again and put it in my /etc directory and changed the aliases and not sure what's going to happen if I do another reboot.

Can anyone come up with a reason of why this would happen?
Reply
Not the right place to ask, and this may be a stupid observation, but doesn't temp mean temporary? So if by temp you somehow magically expect some permanence then idk. Im not familiar with the pi storage structures but usually I wouldnt dream of putting something in temp that I want to keep.
Reply
The contents of the /tmp folder is not usually retained between reboots on most Unix/Linux systems, what you experienced is therefore perfectly normal and to be expected.

I wouldn't recommend storing the script in /etc, that's not what the /etc folder is for. Putting the script in your pi user home directory should be sufficient (/home/pi), or maybe even /usr/local.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
See http://en.wikipedia.org/wiki/Filesystem_...y_Standard /tmp is not always retained across reboots (although it is in some systems).

/etc is not the right place, it is for config files.

Binaries belong in a bin directory. To see what directories are in your path use
Code:
echo $PATH
Put the executable in one of them. The natural place for a script such as this would be /usr/bin or /usr/local/bin

(The local version is preferred by some people who say /usr/bin should be for stuff installed by your native packaging system (apt, rpm etc) and /usr/local/bin for ad hoc stuff like texture_cache)
If I have helped you or increased your knowledge, click the 'thumbs up' button to give thanks :) (People with less than 20 posts won't see the "thumbs up" button.)
Reply
  • 1
  • 55
  • 56
  • 57(current)
  • 58
  • 59
  • 197

Logout Mark Read Team Forum Stats Members Help
[RELEASE] Texture Cache Maintenance utility17