•   
  • 1
  • 6
  • 7
  • 8(current)
  • 9
  • 10
  • 180
  •   
[RELEASE] Texture Cache Maintenance utility
(2013-04-09, 03:29)MilhouseVH Wrote:
(2013-04-09, 03:21)charrua Wrote: Any chances of adding CD art images to music albums cache preload?
AFAIK these images (cdart.png) are downloaded by CDArt Manager AddOn straight to the folder where the audio files of the music album are, and then they are cached from there by the skins that support that feature, but the images links are never loaded into the music DB.
Are these artwork types supported by JSON, as the JSON API only describes fanart and thumbnail as valid artwork for Albums.
What happens if you add the cdart type (whatever type that is) to "extrajson.albums" and run "Jd albums <albumname>" where albumname is an album with cdart - do you get a valid response, or an error (invalid parameter)?
I think it's not supported by JSON, at least in Frodo (but I don't know about Gotham).

e.g.: This album has cdart.png in its folder, the image is cached and showed by the skin (Aeon Nox (last git version)), but the JSON query returns only thumbnail and fanart:
Code:
texturecache.py jd albums "back in black"
[
  {
    "title": "Back in black",
    "fanart": "image://http://assets.fanart.tv/fanart/music/66c662b6-6e2f-4930-8610-912e24c63ed1/artistbackground/acdc-4e19b4d51609c.jpg/",
    "artist": [
      "AC/DC"
    ],
    "label": "Back in black",
    "albumid": 3244,
    "thumbnail": "image://http://userserve-ak.last.fm/serve/300x300/81981459.jpg/"
  }
]
Reply
So presumably the P option will delete all those cdART images...

The Music library is a lot less evolved than the Video library, unfortunately.

I'm just browsing through the cdART code and finding references to non-standard artwork types such as "artistthumb" and "cover" - there's probably going to be cdart or maybe discart too. I just tried the "Jd" call with artistthumb and then discart and JSON returned an error both times, so it looks like this is a no go I'm afraid.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
I've just pushed the latest, and probably (hopefully?) final version... v0.5.2.

* Added lastrunfile property. Modification time of this file will be used to restrict cache updates for movies and tvshows (other media classes, while valid, do not support the dateadded filter so its not possible to restricty by date). With this property enabled, only new content added since the file was last modified will be considered for re-caching. The new options lc and lnc will respect the lastrunfile property. If the file is missing or unreadable, no dateadded filter will be applied. Manually touch the lastrunfile to advance the modification date, the script will only ever read the details of this file, and never update it.

* Lookup song/movie/tvshow/episode title when new items are scanned into media library

* vscan/ascan exit status will reflect the number of new items scanned in - 0 when no new items, +n whenever items are added

* Update JSON socket communication to better handle concatenated messages (response + notification(s))

* Added status option to determine if client is idle or active etc.

* Added monitor option, to view notification events as they occur.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Hi, nice tool.

Is it possible to add none Class folders to work with it?
Like I have folders with just video files from YT, Self made stuff, which are not in the Library.
But which I would like to cache the thumbnails for.
So far I have to go into each folder first to catch them, which is annoying.

Many thanks,
derchris
Reply
Hi, great script you did there!
But sadly it doesn't run well for me :/

I got this Error:
Code:
Traceback (most recent call last):
  File "./texturecache.py", line 2896, in <module>
    main(sys.argv[1:])
  File "./texturecache.py", line 2838, in main
    dirScan(removeOrphans=False)
  File "./texturecache.py", line 2108, in dirScan
    rows = database.getAllColumns().fetchall()
sqlite3.OperationalError: Could not decode to UTF-8 column 'url' with text '/mnt/media/Serien/Zurück in die Vergangenheit/folder.jpg'
Kodi 18.4 | Ubuntu 19.10 | Kernel 5.4.2 | intel i3-8100 | Gigabyte Z370N | G.Skill 2x4096MB (DDR4-2133) | Streacom FC5 Evo | Pioneer VSX-934 | LG 65B7D
Reply
(2013-04-18, 12:48)derchris Wrote: Is it possible to add none Class folders to work with it?
Like I have folders with just video files from YT, Self made stuff, which are not in the Library.
But which I would like to cache the thumbnails for.
So far I have to go into each folder first to catch them, which is annoying.

So these files aren't in the media library, in which case you're accessing them just by browsing Files? And are the thumbnails embedded? If the answer is yes to either question, it's unlikely I can make any changes that will help, as without media library information knowing where to look for these files is tricky, and pre-caching embedded thumbnails is very hit and miss (mostly miss).

(2013-04-18, 13:48)3dfx Wrote: Hi, great script you did there!
But sadly it doesn't run well for me :/

I got this Error:
Code:
Traceback (most recent call last):
  File "./texturecache.py", line 2896, in <module>
    main(sys.argv[1:])
  File "./texturecache.py", line 2838, in main
    dirScan(removeOrphans=False)
  File "./texturecache.py", line 2108, in dirScan
    rows = database.getAllColumns().fetchall()
sqlite3.OperationalError: Could not decode to UTF-8 column 'url' with text '/mnt/media/Serien/Zurück in die Vergangenheit/folder.jpg'

Argghhh... this utf-8/unicode/encode/decode nonsense is doing my head in... you think you've got it licked, and then this... Sad

I'll take a look at this tomorrow (don't have time today) and try to reproduce. If possible, could you tar up this folder (without the video files - just the structure, artwork and any nfos) and post it somewhere so that I can just untar it - using an archive might increase my chances of reproducing. Also, what scraper did you use?
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2013-04-18, 14:08)MilhouseVH Wrote:
(2013-04-18, 12:48)derchris Wrote: Is it possible to add none Class folders to work with it?
Like I have folders with just video files from YT, Self made stuff, which are not in the Library.
But which I would like to cache the thumbnails for.
So far I have to go into each folder first to catch them, which is annoying.

So these files aren't in the media library, in which case you're accessing them just by browsing Files? And are the thumbnails embedded? If the answer is yes to either question, it's unlikely I can make any changes that will help, as without media library information knowing where to look for these files is tricky, and pre-caching embedded thumbnails is very hit and miss (mostly miss).

(2013-04-18, 13:48)3dfx Wrote: Hi, great script you did there!
But sadly it doesn't run well for me :/

I got this Error:
Code:
Traceback (most recent call last):
  File "./texturecache.py", line 2896, in <module>
    main(sys.argv[1:])
  File "./texturecache.py", line 2838, in main
    dirScan(removeOrphans=False)
  File "./texturecache.py", line 2108, in dirScan
    rows = database.getAllColumns().fetchall()
sqlite3.OperationalError: Could not decode to UTF-8 column 'url' with text '/mnt/media/Serien/Zurück in die Vergangenheit/folder.jpg'

Argghhh... this utf-8/unicode/encode/decode nonsense is doing my head in... you think you've got it licked, and then this... Sad

I'll take a look at this tomorrow (don't have time today) and try to reproduce. If possible, could you tar up this folder (without the video files - just the structure, artwork and any nfos) and post it somewhere so that I can just untar it - using an archive might increase my chances of reproducing. Also, what scraper did you use?

Same here :-)
I had some headaches with my Webinterface project too.....

Sure, I will pack it later when I'm back home.
Thanks!
Kodi 18.4 | Ubuntu 19.10 | Kernel 5.4.2 | intel i3-8100 | Gigabyte Z370N | G.Skill 2x4096MB (DDR4-2133) | Streacom FC5 Evo | Pioneer VSX-934 | LG 65B7D
Reply
(2013-04-18, 14:08)MilhouseVH Wrote: Argghhh... this utf-8/unicode/encode/decode nonsense is doing my head in... you think you've got it licked, and then this... Sad

welcome to our world Wink
Read/follow the forum rules.
For troubleshooting and bug reporting, read this first
Interested in seeing some YouTube videos about Kodi? Go here and subscribe
Reply
@3dfx: Thinking about this, I may only need your Textures13.db to reproduce this. If you're willing to share your Textures13.db please PM me a download link, many thanks!
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Question for you.

I have directories with all fanart, thumbnails etc for all movies / tv shows / series etc all in folder structure on my NAS, central MySQL database on my NAS, 2 ATV 2's and a Pi, all configured to point to the central database, and all with the same sources.

The ATV2's have all the thumbnails cached ok I think, but the Pi doesn't seem to have everything.

I've tried running this script on my Pi as follows:

./texturecache.py nc movies
returns:
Quote:(snip, 280 movies...)
Need to cache: [ fanart ] for movie: X2
Need to cache: [ poster ] for movie: X2
Need to cache: [ fanart ] for movie: Zodiac
Need to cache: [ poster ] for movie: Zodiac
Need to cache: [ fanart ] for movie: ?on Flux
Need to cache: [ poster ] for movie: ?on Flux

Cache pre-load activity summary for "movies":

| banner | clearart | clearlogo | discart | fanart | landscape | poster | TOTAL
--------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------
Cached | - | - | - | - | - | - | - | 0
Deleted | - | - | - | - | - | - | - | 0
Duplicate | - | - | - | - | - | - | - | 0
Error | - | - | - | - | - | - | - | 0
Ignored | - | - | - | - | - | - | - | 0
Not in Cache | 19 | 83 | 98 | 71 | 271 | 39 | 255 | 836
Skipped | 22 | 1 | - | - | 10 | - | 27 | 60
==============================================================================================================================
TOTAL | 41 | 84 | 98 | 71 | 281 | 39 | 282 | 896

Loading: 00:00:08.23
Parsing: 00:00:00.38
Comparing: 00:00:04.23
TOTAL RUNTIME: 00:00:12.96

So there's lots missing Wink

So now running:
./texturecache.py c movies

Quote:Need to cache: [ poster ] for movie: Austin Powers in Goldmember
Need to cache: [ clearart ] for movie: Austin Powers in Goldmember
Need to cache: [ fanart ] for movie: Austin Powers: International Man of Mystery
Need to cache: [ discart ] for movie: Austin Powers: International Man of Mystery
Need to cache: [ poster ] for movie: Austin Powers: International Man of Mystery
Need to cache: [ clearart ] for movie: Austin Powers: International Man of Mystery
...and many more! (First 100 items shown)

Queueing work item: Single thread 0, Multi thread 836

and then a bit later I get :
Quote:Caching artwork: 829 items remaining of 836 (s: 0, m: 829), 6 errors, 2 threads active (00.10 downloads per second, ETA: 02:18:43)

After stopping the download:

Quote:The following items could not be downloaded:

[banner ] [Blade: Trinity ] image://http://assets.fanart.tv/fanart/movies/36648/moviebanner/blade-trinity-504e09df221c2.jpg/
[banner ] [Blade Runner ] image://http://assets.fanart.tv/fanart/movies/78/moviebanner/blade-runner-504e1388648a4.jpg/
[banner ] [Captain America: The First Avenger ] image://http://assets.fanart.tv/fanart/movies/1771/moviebanner/captain-america-the-first-avenger-50a2cb333a48a.jpg/
[banner ] [Casino Royale ] image://http://assets.fanart.tv/fanart/movies/36557/moviebanner/casino-royale-5055cb76b5343.jpg/
[banner ] [Cloverfield ] image://http://assets.fanart.tv/fanart/movies/7191/moviebanner/cloverfield-50eb6a84a6246.jpg/
[banner ] [Daybreakers ] image://http://assets.fanart.tv/fanart/movies/19901/moviebanner/daybreakers-5083331d9bc15.jpg/
[banner ] [Dredd ] image://http://assets.fanart.tv/fanart/movies/49049/moviebanner/dredd-504d1c84ecb15.jpg/
[banner ] [Groundhog Day ] image://http://assets.fanart.tv/fanart/movies/137/moviebanner/groundhog-day-509617542e626.jpg/
[banner ] [In Time ] image://http://assets.fanart.tv/fanart/movies/49530/moviebanner/in-time-513bba8a90ac6.jpg/
[banner ] [I, Robot ] image://http://assets.fanart.tv/fanart/movies/2048/moviebanner/i-robot-510ea65bd7896.jpg/
[banner ] [Inception ] image://http://assets.fanart.tv/fanart/movies/27205/moviebanner/inception-505266bdeea20.jpg/
[banner ] [Iron Man ] image://http://assets.fanart.tv/fanart/movies/1726/moviebanner/iron-man-50447badecf8a.jpg/

Not sure why its trying to download the artwork from the web when I have all art locally? Am i missing a config option to force it to use local folders?
QNAP TS670 NAS - 4x4TB Raided with NFS Support & Central MySQL DB
Pi 3 [Xbian] - Samsung 46" ES7000 Smart TV & HT-E5550 3D BD Sound System
Pi 3 [Xbian]  + Hyperion Ambilight- Samsung 48" Curved UE48JS9000 4k
PowerEthernet T1502 Powerline adapters
Reply
(2013-04-18, 23:01)nsviper Wrote: Not sure why its trying to download the artwork from the web when I have all art locally? Am i missing a config option to force it to use local folders?

No config issues, that's where XBMC thinks your artwork is - on the web.

Not sure how you've scraped your library, but for whatever reason XBMC has decided to locate your banners using web resources, not local.

Have you used a utility like Artwork Downloader and "overwritten" your locally referenced fanart/banners with remote artwork? The fact you have clearart, discart etc. in your media library makes me suspect this is what you have done.

If you want to see what is in your media library, run - as an example - ./texturecache.py jd movies cloverfield and you'll see that your artwork (some, if not all) is referenced remotely ie. http: rather than local (c:/nfs:/smb:/etc.).

Note also, that fanart.tv appears to prevent multiple concurrent connections, no doubt to avoid being flooded with requests, so you'll get a lot of download failures running multiple download threads against this site (as, indeed, you are). If you do want to download artwork from fanart.tv, then I would suggest you add "singlethread.urls = assets.fanart.tv" to your properties file (texturecache.cfg) so that this artwork is downloaded sequentially over a single thread - slow, but reliable.

I should probably make this the default setting (@Martijn, are you able to confirm if this behaviour is correct? Totally understandable if it is).
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Cheers dude will take a look at that and let u know ...
QNAP TS670 NAS - 4x4TB Raided with NFS Support & Central MySQL DB
Pi 3 [Xbian] - Samsung 46" ES7000 Smart TV & HT-E5550 3D BD Sound System
Pi 3 [Xbian]  + Hyperion Ambilight- Samsung 48" Curved UE48JS9000 4k
PowerEthernet T1502 Powerline adapters
Reply
(2013-04-18, 18:19)MilhouseVH Wrote: @3dfx: Thinking about this, I may only need your Textures13.db to reproduce this. If you're willing to share your Textures13.db please PM me a download link, many thanks!

You've got a PM Smile
Kodi 18.4 | Ubuntu 19.10 | Kernel 5.4.2 | intel i3-8100 | Gigabyte Z370N | G.Skill 2x4096MB (DDR4-2133) | Streacom FC5 Evo | Pioneer VSX-934 | LG 65B7D
Reply
(2013-04-19, 09:47)3dfx Wrote:
(2013-04-18, 18:19)MilhouseVH Wrote: @3dfx: Thinking about this, I may only need your Textures13.db to reproduce this. If you're willing to share your Textures13.db please PM me a download link, many thanks!

You've got a PM Smile

Thanks, have downloaded the Textures13.db and been able to reproduce...now to find a fix... Smile
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2013-04-19, 15:26)MilhouseVH Wrote:
(2013-04-19, 09:47)3dfx Wrote:
(2013-04-18, 18:19)MilhouseVH Wrote: @3dfx: Thinking about this, I may only need your Textures13.db to reproduce this. If you're willing to share your Textures13.db please PM me a download link, many thanks!

You've got a PM Smile

Thanks, have downloaded the Textures13.db and been able to reproduce...now to find a fix... Smile

That would be nice, I really want to clean up the mess in the Thumbnails folder Smile
Kodi 18.4 | Ubuntu 19.10 | Kernel 5.4.2 | intel i3-8100 | Gigabyte Z370N | G.Skill 2x4096MB (DDR4-2133) | Streacom FC5 Evo | Pioneer VSX-934 | LG 65B7D
Reply
  •   
  • 1
  • 6
  • 7
  • 8(current)
  • 9
  • 10
  • 180
  •   
 
Thread Rating:
  • 17 Vote(s) - 4.76 Average



Logout Mark Read Team Forum Stats Members Help
[RELEASE] Texture Cache Maintenance utility4.7617