[RELEASE] Texture Cache Maintenance utility - Printable Version +- Kodi Community Forum (https://forum.kodi.tv) +-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33) +--- Forum: Supplementary Tools for Kodi (https://forum.kodi.tv/forumdisplay.php?fid=116) +--- Thread: [RELEASE] Texture Cache Maintenance utility (/showthread.php?tid=158373) Pages:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
|
RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2016-02-19 (2016-02-19, 09:27)gjwAudio Wrote: Checking in fixup.dat, I find the script is planning to rename the Defendor files with prefix alright... BUT it's using VIDEO_TS as the Movie-Name element. Yes, VIDEO_TS isn't supported - I'll need to add support to mklocal.py, but it's unlikely to be added soon. (2016-02-19, 09:27)gjwAudio Wrote: Best approach (as I see it now) is to remove the Movies_VTS folder from the Library, correct the naming and location anomalies, and then "Scan for new content". I'm pretty sure you can eliminate the VIDEO_TS folder entirely so that all files are in the "Defendor (2009)" folder. This would resolve the issue with mklocal.py. You'll need to remove, clean and rescrape your VIDEO_TS movies. Try one and see how it goes before doing the rest. (2016-02-19, 09:27)gjwAudio Wrote: In the context of a VTS rip, does it make sense to name the artwork as if there were a single video file (ie: with Movie-Name-prefix) ? I think it's personal preference, but it does at least identify the artwork unambiguously which is a plus, and it's why it's my preferred naming convention. (2016-02-19, 09:27)gjwAudio Wrote: In your opinion, are there other practices I should adopt for keeping the artwork under control Nah not really. I just try to avoid remote artwork as much as possible so I run mklocal.py on a regular basis to ensure that any remote artwork (I use Artwork Downloader after scraping new movies to find logos and clearart for older movies) is converted to local artwork. RE: [RELEASE] Texture Cache Maintenance utility - gjwAudio - 2016-02-20 Hi Milhouse (2016-02-19, 16:48)Milhouse Wrote: I'm pretty sure you can eliminate the VIDEO_TS folder entirely so that all files are in the "Defendor (2009)" folder. This would resolve the issue with mklocal.py. You'll need to remove, clean and rescrape your VIDEO_TS movies. Try one and see how it goes before doing the rest. CONFIRMED - tried one as you stated, and it works fine. (2016-02-19, 16:48)Milhouse Wrote: Nah not really. I just try to avoid remote artwork as much as possible so I run mklocal.py on a regular basis to ensure that any remote artwork (I use Artwork Downloader after scraping new movies to find logos and clearart for older movies) is converted to local artwork. So, does this mean you have a copy of all artwork in your media folders ? If I get your workflow right, it goes something like this: 1) new movie comes in and is scraped with Universal Media Scraper 2) run Artwork Downloader to gather any non-standard art to be found (this only writes the remote URLs to the database, since you prevent AD from saving to local media folders) 3) run the Three Amigos to: download those remote URL links to local media folders (both std and non-std artwork), naming them with Movie-Name-prefix and then load the texturecache with ALL the just-downloaded artwork PLUS any local artwork saved in the media folders by hand Rinse and Repeat as new movies come in. Any comment on using MediaElch to administer the Library ? Benefts or Pitfalls ? One thing I'd like to do is create some custom genre tags, to better classify movies as I see them Y'know, Ed Wood films, Dondi, The Creeping Terror, Eagah! - they're all Movies: BAD as far as we're concerned around here Thanks again for help and advice. RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2016-02-20 (2016-02-20, 07:54)gjwAudio Wrote: So, does this mean you have a copy of all artwork in your media folders ? Yes. (2016-02-20, 07:54)gjwAudio Wrote: If I get your workflow right, it goes something like this: Yes, that's pretty much it: 1) Scrape new movies 2) Run mklocal.py to pick up any local non-standard artwork (ie. logo.png, clearart.png) that I may have manually created alongside the new movies scraped in #1 3) Run AD to find logo/clearart for movies that don't already have such artwork - don't allow AD to download anything 4) Run mklocal.py again to download any non-standard local artwork that AD may have found in step #3 (2016-02-20, 07:54)gjwAudio Wrote: Any comment on using MediaElch to administer the Library ? Benefts or Pitfalls ? I use Ember for movies - this works for Posters and Fanart. I manually add logo.png and clearart.png, usually from fanart.tv. (2016-02-20, 07:54)gjwAudio Wrote: One thing I'd like to do is create some custom genre tags, to better classify movies as I see them Y'know, Ed Wood films, Dondi, The Creeping Terror, Eagah! - they're all Movies: BAD as far as we're concerned around here What about using tags? Adding <tag>Ed Wood</tag> to an NFO should create an "Ed Wood" tag which you can use to filter movies. http://kodi.wiki/view/NFO_files/movies#Movie_tags Once you've created your tags you can even query them using texturecache.py - the following will dump your Ed Wood and James Bond items: Code: texturecache.py jd tags "ed wood or james bond" RE: [RELEASE] Texture Cache Maintenance utility - morstef4 - 2016-02-22 Hello I'm doing it wrong then. My setup is your OE V7 build #203 with Aeon Nox Silvo from the github-master and script.extendedinfo-master. EDIT : Running your 2.2.9 version Quote:Yes, that's pretty much it: My current script running on a schedule Code: 1 - /usr/bin/python /storage/script/texturecache/texturecache.py vclean In hope to have a better setup, I would appreciate if you can assist me in my understanding A) I should also run my lines 6-7 after 2 ? B) find the switch and implement "No download" at my line 3 ? C) run again my lines 6-7 after 3 ? D) 4 and 5 are used by Aeon Nox Silvo, I run 6-7 after... that might be a question for Phil65 and mikesilvo164 I'm still confused with the extrafanarts so I have included them all in my line #6. (kodi export: created them as extrafanartx) E) I'm not certain that your 4 is my 6-7 , is it ok? Thank you very much. (this is my config file) Code: userdata = /storage/.kodi/userdata RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2016-02-22 (2016-02-22, 18:48)morstef4 Wrote: My current script running on a schedule To run Artwork Downloader, I use: Code: texturecache.py exec script.artwork.downloader silent=true mediatype=movie The problem when executing scripts is that once the script has started, your command continues. In the case of Artwork Downloader, it can take several minutes to finish and you need to wait before continuing with the next step. To do this I use a script on the client that tails kodi.log until the phrase "Artwork Downloader: script stopped" appears. How you solve this issue will depend on your platform, and whether you are running the scripts remotely or locally. And in some cases it may not be possible to solve if the script gives no indication that it has finished. So what you need is this: Code: 1 - /usr/bin/python /storage/script/texturecache/texturecache.py vclean I notice you're trying to process extrafanart and extrathumbs with mklocal.py. Normally this isn't supported, as Kodi knows nothing about extrafanart/extrathumbs - that they're supported at all by Kodi is a skin hack, so what you are doing here I don't think this would work. However there is a proposal to add extrafanart/extrathumbs to the media library - this is enabled with the --extrafanart and --extrathumbs options. If you add these options then you don't need to include extrafanart1 extrafanart2 extrafanart3 etc., instead specify "fanart#" and "thumb#". ie. Code: 6 - texturecache.py jd movies | ./mklocal.py --local /media/MyBook/_FilmsWD --prefix /media/MyBook/_FilmsWD --artwork discart banner fanart landscape poster thumb clearart logo fanart# thumb# --ignorebadprefix --output 3.log --extrafanart --extrathumbs --extrafanartmax 5 and mklocal.py will create the movie associations to whatever extrafanart/extrathumbs it can find in your filesystem (up to the maximum --extrafanartmax / --extrathumbmax, which both default to 4). Obviously this needs further skin support, which I don't think is there yet. See mklocal.py changelog for 25/08/2015. If you choose not to include --extrafanart --extrathumbs then I don't think adding extrafanart1 extrafanart2 as --artwork will achieve anything (though I could be wrong, I don't use extrafanart and wrote most of this blind). By the way, use logo or clearlogo, not both. The artwork extension is -logo.png, but needs to create an artwork association of "clearlogo". mklocal.py will handle this automatically. Also, in order to cache your extra fanart and thumbs in step #7, you'll need to add "cache.extrafanart = yes" and "cache.extrathumbs = yes" to texturecache.cfg. Although these options would no longer be necessary once extrafanart/extrathumbs is correctly associated with their movies in the media library. RE: [RELEASE] Texture Cache Maintenance utility - morstef4 - 2016-02-23 Great feed-back, I thank you for that ! I will be using your command for my scripts. (simple) I have removed my "sleep 1h" earlier today since my single test made between vclean and vscan showed me a wait state.... (my entire schedule elapsed time was to large) Tk's again for sharing this information, obviously my single test wasn't good enough. Smart move using a tail check on the Artwork Downloader... (Hopefully AD script will write the same text in the log file in the futur....) My script will be running locally therefore I will check cron setup for this monitoring log. No luck on the other scripts, they don't have such end logs. With your information I need to do some homework on the extra-arts stuff allowing me to increase my knowledge. How come that a "new art standard" is not yet defined ? Confluence is not using them but IMO in 2016 a default skin must have more bells and whistles. Yes for a PI2 installation I understand but the install program should be intelligent enough to find this out and propose an alternative for other hardware. (you don't have to comment on this, I just hope that Kodi organization will bring it to the next level soon - I enjoy so much this product that sometimes I'm having hard time to sell it to a non technical user) I was reading the other proposal and it looks good but what I'm using is fine at the moment. My intention first is to complete a "good" cron job. Quote:By the way, use logo or clearlogo, not both. The artwork extension is -logo.png, but needs to create an artwork association of "clearlogo". mklocal.py will handle this automatically.Good to know. Quote:Also, in order to cache your extra fanart and thumbs in step #7, you'll need to add "cache.extrafanart = yes" and "cache.extrathumbs = yes" to texturecache.cfg. Although these options would no longer be necessary once extrafanart/extrathumbs is correctly associated with their movies in the media library.Thank you, very interesting. To immediately see the difference when I add/change an art file I have to run: 1) texturecache.py C movies "movie_name" or 2) texturecache.py C tvshows "show_name" What should be run as a command to grab automatically those changes and only those changes? (I'm interested to include this in my schedule therefore specific names should be automatically detected) Sometime I change the nfo file : Title, Original Title, plot, outline, genre, set. Following this wiki it will refresh the information and make them available immediately. Code: Refresh individually I will do some bash scripting to verify date/time of those nfo files and apply this command upon a file modification. Texturecache / mklocal can't do that right ? From a previous post : http://forum.kodi.tv/showthread.php?tid=158373&pid=2164216#pid2164216 You mentioned a list of command for recurring cleanup Code: texturecache.py P Thank you again for your patience and availability. If you have a chance one day to modify you first post and write your best case scenario DB maintenance for a script run locally. (you have to limit your documentation time, since I haven't see anyone else asking, my case is a very good case for such bullet point list ) :-) It would have the recipe and recurring frequency of each item based on your standards. - Your texturecache should be integrated by default, what is the team-kodi process to activate this ? RE: [RELEASE] Texture Cache Maintenance utility - gjwAudio - 2016-02-23 Hi Milhouse Another fine - and confounding - piece of advice to work through... (2016-02-20, 20:30)Milhouse Wrote: ... Soooooo, are you suggesting the settings in AD should always be like this: Code: Settings > Movies > Enable downloading = UNchecked (which blanks out all the artwork type choices) and then it will only insert URLs to the database when it finds a hit for any art AND then we use nr.4 above to download the URLs into our media folders ? ...finally completed by a texturecache.py c movies to pre-load all this new artwork... it that right ? EDIT: Haw-Haw... that stuff above pales in comparison to the steaming pile of sadness now known as my Video Library Through a confluence of bad juju, I've messed up about a third of the 550 or so films in the collection. Pretty awesome for a night's work. I began with a clean slate (changed content type to NONE and then back, of Movies source). Let the Universal Movie Scraper run, and then set to, with the scripts. The unsupported VIDEO_TS thing got me, adding bogus file references to the database. The mklocal script got so confused it even made up a PNG file for a movie with no such image in its folder. For example... a snip from the fixup.dat file being passed to mklocal.py... Code: { ...which is supposed to represent files actually stored as: Code: FREENAS/media/Movies_VTS/A Noble Lie (2011)/A Noble Lie (2011)-banner.jpg Without dumping the whole Movies DB and searching for confirmation, my intuition says many of the images missing from the screen have scrambled or completely missing file references. Very hard to troubleshoot so many variations on the theme. I've got assorted banners missing, posters & thumbs, some movies are even missing the text version of the movie name, in place of a banner (ie: blank slot). Looking at the media folders, I expected to see image files "made local" but it looks like nothing new is there. For example, this week I added the classic Basil Rathbone Sherlock Holmes movies, taking great pains to fabricate custom banners and extrathumbs. None of the posters found by AD made it to the media folders - I thought that was a main function of mklocal.py - to write a file into a media folder. Hmmm... Before I saw the extent of the problem, I set out to manually fix the 12 VTS-type movies using the Choose Art button. Arrrrgh - that's what this week was supposed to avoid . Then I got curious and ran this: Code: OpenELEC:~/.kodi/userdata # ./texturecache.py jd movies "a noble lie" And then, there's the "new and improved" Artwork Beef, which promises to bring a new era to art-scraping within Kodi. Well, it began failing 3 hours ago on an internal call (pykodi.py maybe...) for no apparent reason. So now I can't use any context menu calls for artwork repair, 'cause The Beef is Broken, and has hijacked the context buttons - and I don't know how to restore AD to it's former place of glory Maybe this is when the crash course on MediaElch saves the day ? Time for this soldier to hit the barracks . RE: [RELEASE] Texture Cache Maintenance utility - zimbo - 2016-02-23 Is there no Solution for my Problem ( Post 1875)? RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2016-02-23 (2016-02-23, 12:40)zimbo Wrote: Is there no Solution for my Problem ( Post 1875)? For artist artwork you may need to use folder.jpg instead of poster.jpg. http://forum.kodi.tv/showthread.php?tid=229703&pid=2031314#pid2031314 RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2016-02-23 (2016-02-23, 07:54)morstef4 Wrote: To immediately see the difference when I add/change an art file I have to run: if you add @cache.refresh=0 to the command line it will only forcibly re-cache artwork items updated/modified since midnight. So if you're modifying artwork this is a quick way to refresh only changed artwork items. (2016-02-23, 07:54)morstef4 Wrote: Sometime I change the nfo file : Title, Original Title, plot, outline, genre, set. You need the "qax" texturecache.py command. With @qa.nfo.refresh=0 it will remove/rescan any nfo modified since midnight. (2016-02-23, 07:54)morstef4 Wrote: From a previous post : Only qax. (2016-02-23, 07:54)morstef4 Wrote: - Your texturecache should be integrated by default, what is the team-kodi process to activate this ? Depends what you mean by integrated - as an add-on, it will require a complete rewrite. Since it's an external Python script this makes including with Kodi somewhat difficult. Various distributions can include it, I think it may be included (or available as an installable addon) in Arch. I include it in my OpenELEC test builds. etc. RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2016-02-23 (2016-02-23, 08:25)gjwAudio Wrote: Soooooo, are you suggesting the settings in AD should always be like this: Just the last one. The first two reference "downloading" but really it should say "processing" in place of "downloading", so you want them enabled/checked if you want them to actually do anything. The downloading is controlled by the confusingly named "Use local files" setting - with this enabled AD will write artwork into your file system (but without movie-name prefixes which aren't support) and it may also read any existing local artwork (again, only those without movie-name prefix). (2016-02-23, 08:25)gjwAudio Wrote: AND then we use nr.4 above to download the URLs into our media folders ? Yep. (2016-02-23, 08:25)gjwAudio Wrote: <snip> I thought you'd got rid of your VIDEO_TS folders? For the logo issue (which points to a non-existant VIDEO_TS-logo.png file) you should try the --nokeep option on mklocal.py - this will remove any non-existant artwork from your library. Once removed, running mklocal.py again should find the correct logo.png (it may even find it on the first run, I can't remember now...) RE: [RELEASE] Texture Cache Maintenance utility - zimbo - 2016-02-23 Thanks RE: [RELEASE] Texture Cache Maintenance utility - morstef4 - 2016-02-23 Wow this is all great ! My cron(s) will almost be perfect with all those informations, but I still need more. (2016-02-23, 12:57)Milhouse Wrote:(2016-02-23, 07:54)morstef4 Wrote: - Your texturecache should be integrated by default, what is the team-kodi process to activate this ? Quote:Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems. I originally started with Fritsch builds and he is the reason I switched to OE from Windows. Was a great discovery for me, later on I saw your builds and how much detailed you specified and finally got involved with texturecache. I think it should be running in Kodi for the maintenance portion and just adapted slightly for a non technical user allowing him to see some reports depending of his choice of menu. (basic to expert) I can use the menu to export/overwrite in multiple files all the nfos. It is clear that I haven't reach the texturecache limit yet and that said, is there anyway to export/overwrite those individual nfos ? I am planning to install from scratch again using those nfo files (Kodi is using the existing nfo and art files if exists.... I think) and with <importwatchedstate>true</importwatchedstate> I shouldn't loose any of my special art files or nfo config right ? (I have learned to stop exporting the arts files, thank's to you) I'm having a hard time to make a valid command for "missing" texturecache.py missing movies xxxx - can't find out what xxxx should be. (I have spend lot of time 6-8 months ago to create a script to compare fresh nfos export with my media files trying to find if any of those were not scrap- later I found an addons call "Missing Movies" but now I'm going full TC) "texturecache.py c sets" : I know that qax command will reload/cache all information from nfo file including the set. "c sets" will do the same thing but just for sets ? I'm starting to check other features available in TC, commands that I should run manually and act on the results. What would you suggest me to start with ? I'm running OE from a NUC SSD therefore I have lots of space and one of my goal is to bring all my bells and whistles on it. (obviously except the movies/tvshows/pictures/musics) I'm used to hear my disk start spinning when I browse my library movies or tvshows and the net is probably requested too. I know for sure that script.extendedinfo will download some stuff but how about Kodi core ? EDIT : ADD : I don't want to hear my disk start to spin anymore. EDIT 2: ADD : Probably with that script I'm creating using TC my disk won't spin anymore when browsing. Thank you RE: [RELEASE] Texture Cache Maintenance utility - gjwAudio - 2016-02-23 Hi Milhouse (2016-02-23, 13:07)Milhouse Wrote: I thought you'd got rid of your VIDEO_TS folders? You're correct - I did. Perhaps a file explorer screenshot would have been more obvious. All 12 V_TS-type movies were manually edited to be like this: The NFO and image files are all prefixed with the name of their containing folder, and no VIDEO_TS anywhere to be found. Something odd gave mklocal.py a conniption. I'll try the --nokeep option - maybe I can save the library Is this option something to always use, or only on special occasions ? Thanks again for your help. edit: BTW: rmrector pushed a fix to his repo - I updated my installation just now, and Artwork Beef runs again Woo-HOO !! Executed it against the whole library, but many movies still don't show banners and backgrounds . Re: RE: [RELEASE] Texture Cache Maintenance utility - Milhouse - 2016-02-23 (2016-02-23, 21:59)gjwAudio Wrote: Is this option something to always use, or only on special occasions ? Shouldn't be necessary once you've got everything setup correctly, but useful when you're trying to clean things up. |