[RELEASE] Texture Cache Maintenance utility

[RELEASE] Texture Cache Maintenance utility - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Supplementary Tools for Kodi (https://forum.kodi.tv/forumdisplay.php?fid=116)
+--- Thread: [RELEASE] Texture Cache Maintenance utility (/showthread.php?tid=158373)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197

RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2014-09-20

No, --artwork is still a valid argument, not sure why it's not accepting it. Do you have the latest version of mklocal.py - unlike texturecache.py, it doesn't auto-update so you may need to re-download it (save the "raw" source code content from github).

You can check the supported arguments of the version you have installed with "./mklocal.py -h" - if --artwork isn't listed then you have an old version.

Your "Broken pipe" error is because mklocal.py failed to start (due to the arguments not being valid) so texturecache.py had nowhere to send its data.

RE: [RELEASE] Texture Cache Maintenance utility - raspberry_pd - 2014-09-20

Thanks Milhouse! Brilliant! I thought that mklocal auto-updated. Got the new version, looks like things are progressing now Smile

RE: [RELEASE] Texture Cache Maintenance utility - raspberry_pd - 2014-09-20

Here's an interesting error for you to ponder!

Code:
Traceback (most recent call last):

  File "./mklocal.py", line 793, in <module>

    main(init())

  File "./mklocal.py", line 773, in main

    workitem = processItem(args, "episode", episode, episode_items, showTitle=mediatitle)

  File "./mklocal.py", line 371, in processItem

    newname = processArtwork(args, mediatype, media, mediatitle, artitem["type"], mediafile, oldname, artpath_m, artpath_s)

  File "./mklocal.py", line 443, in processArtwork

    fname = pathToXBMC(getImage(args, mediatype, media, title, atype, filename, currentname, target))

  File "./mklocal.py", line 538, in getImage

    info(args, "%9d bytes" % len(idata), atype, title, None, None, target=target)

  File "./mklocal.py", line 107, in info

    line = "%s%-15s - %-10s - %-45s -> %s" % (line, msg, atype.center(10), addEllipsis(45, title), target)

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 68: ordinal not in range(128)

RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2014-09-20

Ugh... if you can zip up the output from texturecache.py jd tvshows (assuming that is your source) and PM me a link while also mentioning the command you are using to run mklocal.py, I'll see if there's a simple fix.

RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2014-09-24

From @pgjensen:

Your config:

Code:
user@MEDIA:~$ cat /usr/local/bin/texturecache.cfg

userdata = /mnt/storage2/xbmc/userdata

dbfile = Database/Textures13.db

thumbnails = Thumbnails

xbmc.host = 192.168.2.2

webserver.port = 8000

webserver.username = xbmc

webserver.password = HIDDEN

rpc.port = 9090

download.threads = 8

logfile = /home/user/texturecache.log

logfile.verbose = yes

Your log:

Code:
xbmc@MEDIA:~$ cat texturecache.log

2014-09-11 11:26:51.286353:MainThread: Command line args: ['/usr/local/bin/texturecache.py', 'c', 'artists']

2014-09-11 11:26:51.286427:MainThread: Current version #: v1.7.6

2014-09-11 11:26:51.286460:MainThread: Current platform : linux2

2014-09-11 11:26:51.286496:MainThread: Python  version #: v2.7.3.0 (final)

...

2014-09-11 11:26:51.312188:MainThread: JSON Texture DB API available and will be used to access the Texture DB

You tell me (in a PM) that /mnt/storage2/xbmc/userdata is mounted on the host that is 192.168.2.2.

I'm still not clear on what exactly you're trying to do, but from the information you've told me it looks like you're trying to populate a mounted userdata folder with cached artwork, using a kodi client listening on 192.168.2.2.

Thing is, the Kodi client is updating it's own userdata folder which may not be the same userdata folder you have configured the script to use. And since the introduction of the JSON Textures API, the script isn't even going to be accessing the userdata folder so even though you have a userdata folder mounted and configured, it's not being used by the script (or, I'm assuming, the Kodi client).

I don't know if the userdata folder you have mounted is the same userdata folder in use by the client listening on 192.168.2.2, but if it's not (and I'm guessing it's not) then that would explain why it's not being updated - because the client on 192.168.2.2 is looking at it's own userdata folder and updating that folder.

There is no way for you to pre-load the cache of a remote userdata folder using a Kodi client that is configured to use a different userdata folder.

The best way to accomplish what you want to do is rsync at boot.

Or, alternatively, connect in turn to each client and pre-load their cache - don't try and use some sort of "proxy" Kodi client to update each userdata folder.

Should you want to configure multiple clients, add "[sections]" to your texturecache.cfg which makes scripting updates to each client much easier, for instance:

Code:
webserver.port = 8000

webserver.username = xbmc

webserver.password = HIDDEN

rpc.port = 9090

download.threads = 8

logfile = /home/user/texturecache.log

logfile.verbose = yes

[master]

xbmc.host = 192.168.2.2

network.mac = aa:bb:cc:dd:ee:00

[lounge]

xbmc.host = 192.168.2.3

network.mac = aa:bb:cc:dd:ee:01

[bedroom]

xbmc.host = 192.168.2.4

network.mac = aa:bb:cc:dd:ee:02

Then, to update the cached movie artwork for all three clients:

Code:
texturecache.py c movies @section=master

texturecache.py c movies @section=lounge

texturecache.py c movies @section=bedroom

RE: [RELEASE] Texture Cache Maintenance utility - three80 - 2014-09-24

Question: What does it cache when caching songs? does kodi wan't album art for every track?

It keeps dying trying to cache songs on raspberrypi , only message I see everytime is "killedogg$ " it cuts off my prompt with killed but no reason i can see, i'm guessing it choked on 78,000 tracks, so I'm wondering why is it caching things for songs?

I know i have all album art embedded but shouldn't albums be enough or is it for singles and such?

Edit: Is there a debug switch I can use to try and figure it out? it took most of a day and half before it error'd

RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2014-09-24

It will cache whatever thumbnail/fanart has been assigned to each song.

For album songs, this is likely to be the artist fanart and album thumbnail, so probably not necessary to cache all your songs - instead just cache artists and albums. Singles could benefit from caching.

Chances are you're running out of memory - check dmesg for out of memory errors when the script is killed. What hardware are you running this on? Try running it on mute powerful hardware, just point the script at your Kodi client with @xbmc.host=<ipaddress>.

Embedded artwork can't be pre-loaded into the cache, so you're probably wasting your time on that anyway.

RE: [RELEASE] Texture Cache Maintenance utility - three80 - 2014-09-24

(2014-09-24, 08:13)Milhouse Wrote: For album songs, this is likely to be the artist fanart and album thumbnail, so probably not necessary to cache all your songs - instead just cache artists and albums. Singles could benefit from caching.

I started with artists, then albums then everything.

Quote:Chances are you're running out of memory - check dmesg for out of memory errors when the script is killed. What hardware are you running this on? Try running it on mute powerful hardware, just point the script at your Kodi client with @xbmc.host=<ipaddress>.

I ran it on raspberrypi

I will try again from my server like you suggest.

Quote:Embedded artwork can't be pre-loaded into the cache, so you're probably wasting your time on that anyway.

I use beets to organize my music and it embeds and puts folder.jpg for albumart. I just wondered if embeded is why it scans songs for artwork.

RE: [RELEASE] Texture Cache Maintenance utility - un1versal - 2014-09-24

@Milhouse

You remember that bug about the the when theres a new season the last season covers arent downloaded. That this patch resolves https://github.com/xbmc/xbmc/pull/3546

Well, since it doesn't look like anyone is going to pull that in for Helix and a workaround is to refresh that last season and when xbmc/kodi asks Do you wish to refersh info for all episodes you say no, and goto through that process and select the series name from list (almost always top entry), I was wondering if that is possible to automate via your script somehow?

RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2014-09-24

Sure, use mklocal.py - it's what I now use and works a treat. It will assign any local artwork that isn't already in the media library, so will find the new season posters once you've scraped in the first episode of the new season.

RE: [RELEASE] Texture Cache Maintenance utility - un1versal - 2014-09-24

Wont that put the artwork where series is though?

RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2014-09-24

Hmm, maybe I spoke too soon.

Whether mklocal.py will work for you depends on whether you have the season artwork downloaded already or not.

My use case has Sickbeard downloading the new season artwork as and when it becomes available on thetvdb.com, and that's never been a problem. The problem for me has always been associating the media library with any new local season artwork once the first episode of a new season is scraped, and for that I'm now using mklocal.py as a workaround.

However I do not know of a fix if you are relying on the Kodi scraper to find and download the artwork for you. PR3546 does/did resolve that, but for reasons I'm still not sure I fully understand is going nowhere, despite seeming to work just fine.

RE: [RELEASE] Texture Cache Maintenance utility - un1versal - 2014-09-24

mklocal wont work for me then.

The workaround does it, or that patch.

Hence I I asked if its possible to automate the workaround for time being, but if not Ill just manually use workaround as it also works for rating/amount of votes in addition of actaully downloadibg the last season poster which gets ignored on the new episodes dlwd.

Yes, I dont understand why the actual portion that fixes bug isnt cherry picked and committed, but then again I dont understand any of the code.

RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2014-09-24

tools/mklocal.py updated to version 0.2.3 with fix for Unicode filename problem reported by raspberry_pd - please update manually if you are using mklocal.py, it does not auto-update.

RE: [RELEASE] Texture Cache Maintenance utility - brazen1 - 2014-09-25

Just a quick THANK YOU Milhouse for providing this excellent tool and detailed instructions.

Options 'c' and 'P' made an already decent windows system more responsive and reduced thumb folder from 25GB to 18GB on an SSD.
I had doubts at 1st thinking this might screw up a heavily modified Aeon Nox 4.1.9 so backed up the entire DB.
Happily and very surprised that EVERYTHING is in tact and functioning perfectly after using your tool.

Thank you for your ambition, dedicated hard work and free contribution(s) for myself and others.
_____________________________________________________________________________