[RELEASE] Texture Cache Maintenance utility

[RELEASE] Texture Cache Maintenance utility - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Supplementary Tools for Kodi (https://forum.kodi.tv/forumdisplay.php?fid=116)
+--- Thread: [RELEASE] Texture Cache Maintenance utility (/showthread.php?tid=158373)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197

RE: [RELEASE] Texture Cache maintenance utility - charrua - 2013-04-09

(2013-04-09, 03:29)MilhouseVH Wrote:
(2013-04-09, 03:21)charrua Wrote: Any chances of adding CD art images to music albums cache preload?
AFAIK these images (cdart.png) are downloaded by CDArt Manager AddOn straight to the folder where the audio files of the music album are, and then they are cached from there by the skins that support that feature, but the images links are never loaded into the music DB.
Are these artwork types supported by JSON, as the JSON API only describes fanart and thumbnail as valid artwork for Albums.
What happens if you add the cdart type (whatever type that is) to "extrajson.albums" and run "Jd albums <albumname>" where albumname is an album with cdart - do you get a valid response, or an error (invalid parameter)?

I think it's not supported by JSON, at least in Frodo (but I don't know about Gotham).

e.g.: This album has cdart.png in its folder, the image is cached and showed by the skin (Aeon Nox (last git version)), but the JSON query returns only thumbnail and fanart:

Code:
texturecache.py jd albums "back in black"

[

  {

    "title": "Back in black",

    "fanart": "image://http://assets.fanart.tv/fanart/music/66c662b6-6e2f-4930-8610-912e24c63ed1/artistbackground/acdc-4e19b4d51609c.jpg/",

    "artist": [

      "AC/DC"

    ],

    "label": "Back in black",

    "albumid": 3244,

    "thumbnail": "image://http://userserve-ak.last.fm/serve/300x300/81981459.jpg/"

  }

]

RE: [RELEASE] Texture Cache maintenance utility - Milhouse - 2013-04-09

So presumably the P option will delete all those cdART images...

The Music library is a lot less evolved than the Video library, unfortunately.

I'm just browsing through the cdART code and finding references to non-standard artwork types such as "artistthumb" and "cover" - there's probably going to be cdart or maybe discart too. I just tried the "Jd" call with artistthumb and then discart and JSON returned an error both times, so it looks like this is a no go I'm afraid.

RE: [RELEASE] Texture Cache maintenance utility - Milhouse - 2013-04-16

I've just pushed the latest, and probably (hopefully?) final version... v0.5.2.

* Added lastrunfile property. Modification time of this file will be used to restrict cache updates for movies and tvshows (other media classes, while valid, do not support the dateadded filter so its not possible to restricty by date). With this property enabled, only new content added since the file was last modified will be considered for re-caching. The new options lc and lnc will respect the lastrunfile property. If the file is missing or unreadable, no dateadded filter will be applied. Manually touch the lastrunfile to advance the modification date, the script will only ever read the details of this file, and never update it.

* Lookup song/movie/tvshow/episode title when new items are scanned into media library

* vscan/ascan exit status will reflect the number of new items scanned in - 0 when no new items, +n whenever items are added

* Update JSON socket communication to better handle concatenated messages (response + notification(s))

* Added status option to determine if client is idle or active etc.

* Added monitor option, to view notification events as they occur.

RE: [RELEASE] Texture Cache maintenance utility - derchris - 2013-04-18

Hi, nice tool.

Is it possible to add none Class folders to work with it?
Like I have folders with just video files from YT, Self made stuff, which are not in the Library.
But which I would like to cache the thumbnails for.
So far I have to go into each folder first to catch them, which is annoying.

Many thanks,
derchris

RE: [RELEASE] Texture Cache maintenance utility - 3dfx - 2013-04-18

Hi, great script you did there!
But sadly it doesn't run well for me :/

I got this Error:

Code:
Traceback (most recent call last):

  File "./texturecache.py", line 2896, in <module>

    main(sys.argv[1:])

  File "./texturecache.py", line 2838, in main

    dirScan(removeOrphans=False)

  File "./texturecache.py", line 2108, in dirScan

    rows = database.getAllColumns().fetchall()

sqlite3.OperationalError: Could not decode to UTF-8 column 'url' with text '/mnt/media/Serien/Zurück in die Vergangenheit/folder.jpg'

RE: [RELEASE] Texture Cache maintenance utility - Milhouse - 2013-04-18

(2013-04-18, 12:48)derchris Wrote: Is it possible to add none Class folders to work with it?
Like I have folders with just video files from YT, Self made stuff, which are not in the Library.
But which I would like to cache the thumbnails for.
So far I have to go into each folder first to catch them, which is annoying.

So these files aren't in the media library, in which case you're accessing them just by browsing Files? And are the thumbnails embedded? If the answer is yes to either question, it's unlikely I can make any changes that will help, as without media library information knowing where to look for these files is tricky, and pre-caching embedded thumbnails is very hit and miss (mostly miss).

(2013-04-18, 13:48)3dfx Wrote: Hi, great script you did there!
But sadly it doesn't run well for me :/

I got this Error:

Code:
Traceback (most recent call last): File "./texturecache.py", line 2896, in <module> main(sys.argv[1:]) File "./texturecache.py", line 2838, in main dirScan(removeOrphans=False) File "./texturecache.py", line 2108, in dirScan rows = database.getAllColumns().fetchall() sqlite3.OperationalError: Could not decode to UTF-8 column 'url' with text '/mnt/media/Serien/Zurück in die Vergangenheit/folder.jpg'

Argghhh... this utf-8/unicode/encode/decode nonsense is doing my head in... you think you've got it licked, and then this... Sad

I'll take a look at this tomorrow (don't have time today) and try to reproduce. If possible, could you tar up this folder (without the video files - just the structure, artwork and any nfos) and post it somewhere so that I can just untar it - using an archive might increase my chances of reproducing. Also, what scraper did you use?

RE: [RELEASE] Texture Cache maintenance utility - 3dfx - 2013-04-18

(2013-04-18, 14:08)MilhouseVH Wrote:
(2013-04-18, 12:48)derchris Wrote: Is it possible to add none Class folders to work with it?
Like I have folders with just video files from YT, Self made stuff, which are not in the Library.
But which I would like to cache the thumbnails for.
So far I have to go into each folder first to catch them, which is annoying.

So these files aren't in the media library, in which case you're accessing them just by browsing Files? And are the thumbnails embedded? If the answer is yes to either question, it's unlikely I can make any changes that will help, as without media library information knowing where to look for these files is tricky, and pre-caching embedded thumbnails is very hit and miss (mostly miss).

(2013-04-18, 13:48)3dfx Wrote: Hi, great script you did there!
But sadly it doesn't run well for me :/

I got this Error:

Code:
Traceback (most recent call last): File "./texturecache.py", line 2896, in <module> main(sys.argv[1:]) File "./texturecache.py", line 2838, in main dirScan(removeOrphans=False) File "./texturecache.py", line 2108, in dirScan rows = database.getAllColumns().fetchall() sqlite3.OperationalError: Could not decode to UTF-8 column 'url' with text '/mnt/media/Serien/Zurück in die Vergangenheit/folder.jpg'

Argghhh... this utf-8/unicode/encode/decode nonsense is doing my head in... you think you've got it licked, and then this...

I'll take a look at this tomorrow (don't have time today) and try to reproduce. If possible, could you tar up this folder (without the video files - just the structure, artwork and any nfos) and post it somewhere so that I can just untar it - using an archive might increase my chances of reproducing. Also, what scraper did you use?

Same here :-)
I had some headaches with my Webinterface project too.....

Sure, I will pack it later when I'm back home.
Thanks!

RE: [RELEASE] Texture Cache maintenance utility - Martijn - 2013-04-18

(2013-04-18, 14:08)MilhouseVH Wrote: Argghhh... this utf-8/unicode/encode/decode nonsense is doing my head in... you think you've got it licked, and then this...

welcome to our world Wink

RE: [RELEASE] Texture Cache maintenance utility - Milhouse - 2013-04-18

@3dfx: Thinking about this, I may only need your Textures13.db to reproduce this. If you're willing to share your Textures13.db please PM me a download link, many thanks!

RE: [RELEASE] Texture Cache maintenance utility - nsviper - 2013-04-18

Question for you.

I have directories with all fanart, thumbnails etc for all movies / tv shows / series etc all in folder structure on my NAS, central MySQL database on my NAS, 2 ATV 2's and a Pi, all configured to point to the central database, and all with the same sources.

The ATV2's have all the thumbnails cached ok I think, but the Pi doesn't seem to have everything.

I've tried running this script on my Pi as follows:

./texturecache.py nc movies
returns:

Quote:(snip, 280 movies...)
Need to cache: [ fanart ] for movie: X2
Need to cache: [ poster ] for movie: X2
Need to cache: [ fanart ] for movie: Zodiac
Need to cache: [ poster ] for movie: Zodiac
Need to cache: [ fanart ] for movie: ?on Flux
Need to cache: [ poster ] for movie: ?on Flux

Cache pre-load activity summary for "movies":

| banner | clearart | clearlogo | discart | fanart | landscape | poster | TOTAL
--------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------
Cached | - | - | - | - | - | - | - | 0
Deleted | - | - | - | - | - | - | - | 0
Duplicate | - | - | - | - | - | - | - | 0
Error | - | - | - | - | - | - | - | 0
Ignored | - | - | - | - | - | - | - | 0
Not in Cache | 19 | 83 | 98 | 71 | 271 | 39 | 255 | 836
Skipped | 22 | 1 | - | - | 10 | - | 27 | 60
==============================================================================================================================
TOTAL | 41 | 84 | 98 | 71 | 281 | 39 | 282 | 896

Loading: 00:00:08.23
Parsing: 00:00:00.38
Comparing: 00:00:04.23
TOTAL RUNTIME: 00:00:12.96

So there's lots missing Wink

So now running:
./texturecache.py c movies

Quote:Need to cache: [ poster ] for movie: Austin Powers in Goldmember
Need to cache: [ clearart ] for movie: Austin Powers in Goldmember
Need to cache: [ fanart ] for movie: Austin Powers: International Man of Mystery
Need to cache: [ discart ] for movie: Austin Powers: International Man of Mystery
Need to cache: [ poster ] for movie: Austin Powers: International Man of Mystery
Need to cache: [ clearart ] for movie: Austin Powers: International Man of Mystery
...and many more! (First 100 items shown)

Queueing work item: Single thread 0, Multi thread 836

and then a bit later I get :

Quote:Caching artwork: 829 items remaining of 836 (s: 0, m: 829), 6 errors, 2 threads active (00.10 downloads per second, ETA: 02:18:43)

After stopping the download:

Quote:The following items could not be downloaded:

[banner ] [Blade: Trinity ] image://http://assets.fanart.tv/fanart/movies/36648/moviebanner/blade-trinity-504e09df221c2.jpg/
[banner ] [Blade Runner ] image://http://assets.fanart.tv/fanart/movies/78/moviebanner/blade-runner-504e1388648a4.jpg/
[banner ] [Captain America: The First Avenger ] image://http://assets.fanart.tv/fanart/movies/1771/moviebanner/captain-america-the-first-avenger-50a2cb333a48a.jpg/
[banner ] [Casino Royale ] image://http://assets.fanart.tv/fanart/movies/36557/moviebanner/casino-royale-5055cb76b5343.jpg/
[banner ] [Cloverfield ] image://http://assets.fanart.tv/fanart/movies/7191/moviebanner/cloverfield-50eb6a84a6246.jpg/
[banner ] [Daybreakers ] image://http://assets.fanart.tv/fanart/movies/19901/moviebanner/daybreakers-5083331d9bc15.jpg/
[banner ] [Dredd ] image://http://assets.fanart.tv/fanart/movies/49049/moviebanner/dredd-504d1c84ecb15.jpg/
[banner ] [Groundhog Day ] image://http://assets.fanart.tv/fanart/movies/137/moviebanner/groundhog-day-509617542e626.jpg/
[banner ] [In Time ] image://http://assets.fanart.tv/fanart/movies/49530/moviebanner/in-time-513bba8a90ac6.jpg/
[banner ] [I, Robot ] image://http://assets.fanart.tv/fanart/movies/2048/moviebanner/i-robot-510ea65bd7896.jpg/
[banner ] [Inception ] image://http://assets.fanart.tv/fanart/movies/27205/moviebanner/inception-505266bdeea20.jpg/
[banner ] [Iron Man ] image://http://assets.fanart.tv/fanart/movies/1726/moviebanner/iron-man-50447badecf8a.jpg/

Not sure why its trying to download the artwork from the web when I have all art locally? Am i missing a config option to force it to use local folders?

RE: [RELEASE] Texture Cache maintenance utility - Milhouse - 2013-04-19

(2013-04-18, 23:01)nsviper Wrote: Not sure why its trying to download the artwork from the web when I have all art locally? Am i missing a config option to force it to use local folders?

No config issues, that's where XBMC thinks your artwork is - on the web.

Not sure how you've scraped your library, but for whatever reason XBMC has decided to locate your banners using web resources, not local.

Have you used a utility like Artwork Downloader and "overwritten" your locally referenced fanart/banners with remote artwork? The fact you have clearart, discart etc. in your media library makes me suspect this is what you have done.

If you want to see what is in your media library, run - as an example - ./texturecache.py jd movies cloverfield and you'll see that your artwork (some, if not all) is referenced remotely ie. http: rather than local (c:/nfs:/smb:/etc.).

Note also, that fanart.tv appears to prevent multiple concurrent connections, no doubt to avoid being flooded with requests, so you'll get a lot of download failures running multiple download threads against this site (as, indeed, you are). If you do want to download artwork from fanart.tv, then I would suggest you add "singlethread.urls = assets.fanart.tv" to your properties file (texturecache.cfg) so that this artwork is downloaded sequentially over a single thread - slow, but reliable.

I should probably make this the default setting (@Martijn, are you able to confirm if this behaviour is correct? Totally understandable if it is).

RE: [RELEASE] Texture Cache maintenance utility - nsviper - 2013-04-19

Cheers dude will take a look at that and let u know ...

RE: [RELEASE] Texture Cache maintenance utility - 3dfx - 2013-04-19

(2013-04-18, 18:19)MilhouseVH Wrote: @3dfx: Thinking about this, I may only need your Textures13.db to reproduce this. If you're willing to share your Textures13.db please PM me a download link, many thanks!

You've got a PM Smile

RE: [RELEASE] Texture Cache maintenance utility - Milhouse - 2013-04-19

(2013-04-19, 09:47)3dfx Wrote:
(2013-04-18, 18:19)MilhouseVH Wrote: @3dfx: Thinking about this, I may only need your Textures13.db to reproduce this. If you're willing to share your Textures13.db please PM me a download link, many thanks!

You've got a PM

Thanks, have downloaded the Textures13.db and been able to reproduce...now to find a fix... Smile

RE: [RELEASE] Texture Cache maintenance utility - 3dfx - 2013-04-19

(2013-04-19, 15:26)MilhouseVH Wrote:
(2013-04-19, 09:47)3dfx Wrote:
(2013-04-18, 18:19)MilhouseVH Wrote: @3dfx: Thinking about this, I may only need your Textures13.db to reproduce this. If you're willing to share your Textures13.db please PM me a download link, many thanks!

You've got a PM

Thanks, have downloaded the Textures13.db and been able to reproduce...now to find a fix...

That would be nice, I really want to clean up the mess in the Thumbnails folder Smile