Kodi Community Forum

Never mind, a long day in work has obviously done for my reading comprehension.

At moment Album thumb is sync'd between library and player after a artwork change, but artist fanart isn't.

So artist fanart not updating would not be a problem since there's no change in behaviour, but what about Album thumb? would the simple option mean the Album thumb is no longer updated immediately in both Library and Player OSD?

(2018-02-26, 17:34)jjd-uk Wrote: [ -> ]Never mind, a long day in work has obviously done for my reading comprehension.

I know how you feel.
 

Quote:At moment Album thumb is sync'd between library and player after a artwork change, but artist fanart isn't.

There is an attempt in the current code, that will sync album art changes  sometimes to the song player. But it does so in a way that conflicts with Fernet's (reasonable) requirement that the CApp current item is primary, because it gets left behind while GUI items change. And it can be fooled too because it is trying to match on name not ids. No such attempt is made for fanart (or any other kind of art).
 

Quote:So artist fanart not updating would not be a problem since there's no change in behaviour, but what about Album thumb? Would the simple option mean the Album thumb is no longer updated immediately in both Library and Player OSD?

The simple option means that the currently flakey player OSD album art sync would stop completely.
The library window will be in sync, but it will be done efficiently, not by the overkill of a total list rebuild (that didn't always work either) that it was before.

But don't worry, I think I can get all the art updating efficiently. Maybe I should not have shared my initial doubts, and caused confusion! I just like talking ideas out

I also agree that changing artwork for me at least is a maintenance function, the exception being the Artist Slideshow addon that updates music viz window properties based on the value of player.artist.  Getting feedback in the info window on a change is sufficient for me.  My skin mod (and the various ones it's derived from) loads everything in the music viz window, not the OSD (if that makes a difference).

An additional issue or maybe better concern, is that skins often implement "extrafanart" which relies on passing a folder path to a multiimage control.   Typically the path is derived from the artist name or the media item folder.  A similar concern exists in video, but in video it's more typical that extrafanart is always in a folder in the media folder so it's easier to find.

scott s.
.

OK, my work in progress is here 20180226-a1c597d2-JobAlbumInfoDialog

That has the corrected song info dialog (PR waiting to merge if it funstions OK), and what I have so far of album and artist info dialog. That is extended art support, and art changes being immediately applied everywhere including player. Stil some tidy-up to do, but broadly working.

As for extrafanart: well I really hope that all the art addons that currently make artist folders of there own and put art there will start at least offering the artist info folder, and getting the artist folder name from Kodi. Wouldn't that be great - all using the same folder structure/naming that copes with non-unique artist names, collaborations and artists across multiple music sources.

Perhaps someone nudge the authors of such addons to talk to me?

Multiple fanart doesn't really need to be treated any differently than other artwork, it can be placed into the library (just have to pick a name; I've been adding them as "fanart1", "fanart2", and so on with Artwork Beef). Skins pull them up from ListItems and Player and wherever just like all other artwork, which means plugins can also provide them. Skins shouldn't care about the file system at all, and with Dave's work here all of the music library artwork can be added to the library as well - it is a bit more work for skins to display them in a multiimage control than the folder hack, but totally worth it.

Dave, I've just been working on local files for music artwork in Artwork Beef, and I've been working with the artist info directory. I love the idea, and I kind of want it for all media types, even for those that do usually have one single home in the file system.

This art stuff is a voyage of discovery.... (you know how I like to chat while I work)

Todays topic: Texture caching and the ShowAndGetImage dialog
I think I have known but tried to deny that Kodi's texture cache - both the entries in the textures13.db and the files in the folders under userdata/thumbnails - just grows, and grows, and grows (until you either start a fresh install or use a utility to tidy it up).  Well if you start browsing around your drives looking for images at any time that Kodi offers you to select an image, then this can quickly become out of hand. Remember every thumbnail you see, while say browsing your PC pictures folder, gets cached.  That can be a lots of images that will never be used anywhere else in Kodi.

Now maybe you all knew that, but I didn't, and after some random testing of music art (looking at video art or just the pictures facility does the same) the things left in my cache were quite bizzare.

I have had a little look at if I can tidy up after browsing for music art, and I can't see how to do that so the texture cache bloat lives to grow another day!

Yes I am well aware of this cache behavior after using Kodi for 10 years +

To be honest it hasn't really been a problem in all that time, I think the only time I used the texture cache utility to prune things was when I was Add-on testing one time.

Scraping music picks up a list of URLS for remote thumbs for albums, and thumbs and fanart fo artists. In the (near) future the music scrapers could pick up a wider range of kinds of remote art e.g. clearlogo and banner for artists, and cdArt for albums. I hope for Kodi core to be agnostic about what that art is and where it is shown - that is the business of addons and skins. After scraping whatever kind they are, the list of URLs of possible remote art just get stored in the db as an XML string.

The first thumb for albums, or thumb and fanart for artists is taken as the default to use if a local image has not already been found. Some people would like that to be optional, so they can escape poor quality art that they embedded in their music long ago, but that is a spearate issue. 

I don't see Kodi extending that behaviour to other kinds of art, after all just because the scraper fetched it does not mean you have a skin that shows it. But i need to think the consequences through. It will mean the user either has to manually pick that kind of art for evey artist/album, or that an addon will have to do so to automate it. But it is attractive to have the scraper fetch the list of possibilities in one go, than have repeated requests to online sites by different addons. Thoughts on this welcome - scrape more lists of possible art, but then what?

However my immediate question is how best to present (future) mixed kinds of remote art to the user during manual art selection?

Perhaps an example will help?
The newest version of the artist scraper can optionally pick up clearlogo from fanart.tv, and the resulting list of urls, a mix of thumbs and clearlogo, gets stored in the db. This does not mean that artist has art of type "clearlogo", none of them has been chosen as art yet. User then brings up the artist info dialog for an artist that has possible thumb, fanart and clearlogos all scraped from  Fanart.tv and clicks on "Choose Art". What they see, assuming that no other type of art has been added for any artist,  will be art types "Thumbnail" and "Fanart" (no "Clearlogo").

What next?
Click on "Thumbnail" and you see only the list of available remote thumbs + current image
Click on "Fanart" and you see only the list of available remote fanart + current image
Add art type "clearlogo" (magically knowing that may have been scraped) and then you see the list of available remote logos

or

Click on "Thumbnail" and you see the list of all remote art of every kind + current image, maybe labelled with kind
Click on "Fanart" and you see only the list of available remote fanart + current image
Add any kind of art type and you see the list of all remote art of every kind, maybe labelled with kind

[Scraped fanart has historically always be treated separately, there is no longer any need to do that but we need to transition there]

I hope you can imagine what I describe.

I always envisioned the universal scraper just supporting all this defined extra art, it should in my opinion be automatic and require no user intervention (other than to disable it).

Certainly the metadata site I run is ready for this and already has well established 'defined' artwork types.

Can't Kodi just have these artwork types pre-defined depending on whats supported in the scraper? I don't really understand why it needs to be so extensible (although thats awesome of course), as having clear defined artwork will encourage more skins to support it.

In answer to your real question, I like the 2nd option more as it requires less user intervention i think. I think that 99% of users will never do this and will just want it to look pretty after scraping

Yup, keeping the cache less bloated without tossing out stuff that will just be redownloaded again is a tough problem.

Kodi should add the first of every art type provided automatically; scrapers can have configuration options for each art type (the Universal Movie Scraper recently added extra artwork, and has switches for everything but set artwork), so folks can still decide if they want the artwork or not. With the first of all available art types already assigned by default, the "choose type" dialog only needs to show assigned artwork types, because they are unlikely to be different. Remember that GUI selection of artwork will be a last resort, add-ons should be doing most of this work, and scrapers are add-ons.

In the video library I really dislike that all of the other available but not selected art is stored in the database, as that goes out of date before long, and it won't be accessed very often; it seems like data that should be cached temporarily rather than permanently stored like the actually visible artwork. Python scrapers will be handling their own cache, so I'd prefer Kodi to ask the scraper for the available art when the user does, so a list of the currently available art can be returned (I have had several users wonder why the artwork they just added to the web service wasn't available within the 72 hours that Artwork Beef caches artwork results, and I know Artwork Downloader got the same, so there will be plenty of complaints about the results if they are even older). That's only Python scrapers, though, and I don't want to do that for the XML scrapers because they don't seem to cache particularly well, so that's probably not a good idea yet.

And docwra, I will fight you to keep the art map just as extensible as it is . The generally available art types are fairly well defined for skinners in the video library (Artwork Downloader set the convention for art type naming, the open web services define what is generally available), I just don't see it being a problem in the music library. New artwork types will come up in the forum based on the desire of skins; this way users can add them themselves if necessary, and scrapers and other add-ons can add them much quicker than Kodi core likely ever will.

Yes just to be clear, I like that its extensible and in no way want it removed. I was just hoping to improve the "out of the box" experience for new users.

The last HTPC project I was involved in had extensible artwork and it became a huge mess over time with different Add-ons supporting different naming, locations, sizes and formats. This was a long time ago, but I remember being pleasantly surprised when I used XBMC that it kept things constant with scrapers and local artwork, the wiki guides were nice as well to consult saying where artwork should be put etc. 

Just thought it was worth mentioning

OK, several points in discussion.

Controlling extensible art.... I hope we are OK to let the official addon repo management keep both addons and skins using a consistent set of art types, rather than have it hard coded into core and hard to adapt. We will find out soon enough I guess.
 

(2018-02-28, 05:54)rmrector Wrote: [ -> ]Kodi should add the first of every art type provided automatically.

I understand why you want this Zag & Ryan.
My reply is that I am doing this "grand art extension" in steps, and that taking the first of each type of art scraped as the default would be a later stage than my current PR. I think it combines with the internal change that stops handling fanart separately, and treats all kinds of art as art. It also possibly involves making the use of embedded art over remote scraped art optional, and at the same time fetching more of that embedded art not just front covers at scanning time. There is always more work to do.

 

(2018-02-28, 05:54)rmrector Wrote: [ -> ]Remember that GUI selection of artwork will be a last resort, add-ons should be doing most of this work, and scrapers are add-ons.

Yes I know users want lots of pretty art to magically appear, but there will always be a need for manual adjustments too e.g. replacing some images with those the user prefers. Not quite "last resort", even if addons do the bulk of initial image assignment.

Scrapers are addons true, but they get invoked by a special part of core. Maybe we are going the wrong way with that, stuck in an historic groove. An addon does not have to be a "scraper" to add art, so maybe we leave scraping just to identifying music (or video) and all the fetching of art for things is done by other addons? Just thinking aloud, no action pending.

 

(2018-02-28, 05:54)rmrector Wrote: [ -> ]In the video library I really dislike that all of the other available but not selected art is stored in the database, as that goes out of date before long, and it won't be accessed very often....

On the one hand we need to keep data fresh, but on the other we must avoid the millions of Kodi boxes out there killing the free online resource sites by making too many requests. It really is important that Kodi (both core and addons) behaves responsibly and uses these resources economically or we will lose them.

Once Kodi has art for an item I am absolutely against automatically rechecking for new art for every item just in case some more has appeared on an online site. I don't know what addons do now, but if they are hammering those sites then it must stop. Give the user a refresh method sure, but ensure it is necessary and deliberate request for new art, not in effect creating a DoS attack on the art providers than will in the end deprive all users of the use. We want to avoid some users ending up with no art at all, just because other users were inadvertently greedy.

Do the scraped urls for possible thumb or fanart really get out of date so quickly?

Fetching extended art live, say every time the music is played, could work for those that want it (and the sites support it). But I don't see that as practical approach for browsing where the user could be paging past thousands of albums or artists. Pretty browsing means a certain amount of static art that doesn't change from one year to the next, and just helps the user spot the artist or album they want.

Maybe we need to recognise the different requirements for  minimal and static art (yet still user configurable) used in library browsing, and extensive and dynamic art used in playback art shows?

Finally talking about addons caching etc., why not think of the kodi databases (music + video) as the cache? Storing a list of remote art URLs as XML is no problem, the only change needed is to make it easier to replace that list (or add to it?). At the moment only scrapping can populate it, but I don't see why that can't change. That seems a more robust solution than Python cache, and covers the XML scrapers too.

(2018-02-28, 18:42)DaveBlake Wrote: [ -> ]Scrapers are addons true, but they get invoked by a special part of core. Maybe we are going the wrong way with that, stuck in an historic groove. An addon does not have to be a "scraper" to add art, so maybe we leave scraping just to identifying music (or video) and all the fetching of art for things is done by other addons?

Two separate and unrelated processes to fill library data is awkward and complicates things. I still think composing multiple scrapers is the way to go here.
 

(2018-02-28, 18:42)DaveBlake Wrote: [ -> ]On the one hand we need to keep data fresh, but on the other we must avoid the millions of Kodi boxes out there killing the free online resource sites by making too many requests. It really is important that Kodi (both core and addons) behaves responsibly and uses these resources economically or we will lose them.

I don't think that there are millions of Kodi boxes manually managing artwork by hand in the Kodi GUI so frequently that free resources will even notice, nevermind be killed. Web services can use HTTP caching headers on the API to tell consumers/scrapers how long the cache should live, so control can be firmly in their hands (as long as scrapers respect the headers, but the basics are easy enough for a simple Python module).

This is using the API as they are intended (and I expect puts less strain on the web service than using the browser to find the same art), and when Kodi downloads an image it keeps it "forever", as you've noted, so this won't be what kills them; if they die and it has anything remotely to do with Kodi, it will be because of the shit tools that streampiratey "developers" cobble together that are way too aggressive about flushing the texture cache and whatever other nonsense they get up to.

(2018-02-28, 18:42)DaveBlake Wrote: [ -> ]Do the scraped urls for possible thumb or fanart really get out of date so quickly?

Yes. Services may add new artwork types (TADB just added clearart and clearlogo), and quite often new artwork is made for existing media - TV shows can have new imagery for new seasons, which can spawn new artwork, artists can change their style over time, and new fans of old media may have the right mix of skills and desire to create some missing artwork for the old media. Some video library users (Milhouse suggests this, so it has some weight) prefer to download artwork because the URLs can change at any time, and that applies just as much to the available-but-not-assigned artwork. Fanart.tv has the Personal API keys specifically so that new artwork can be accessed sooner from the API.

(2018-02-28, 18:42)DaveBlake Wrote: [ -> ]Fetching extended art live, say every time the music is played, could work for those that want it (and the sites support it). But I don't see that as practical approach for browsing where the user could be paging past thousands of albums or artists. Pretty browsing means a certain amount of static art that doesn't change from one year to the next, and just helps the user spot the artist or album they want.

Maybe we need to recognise the different requirements for  minimal and static art (yet still user configurable) used in library browsing, and extensive and dynamic art used in playback art shows?

I don't know what "dynamic" artwork would accomplish, nor live artwork updates. I also disagree that artwork during browsing is just for identification: album cover and artist portrait did that just fine, this is about enhancing the browsing experience, so all artwork useful for a playback show could be useful for browsing, if the skin finds a use for it. Sound like over-complication, or something that an add-on could implement first to find out how it should actually work.

(2018-02-28, 18:42)DaveBlake Wrote: [ -> ]Finally talking about addons caching etc., why not think of the kodi databases (music + video) as the cache? Storing a list of remote art URLs as XML is no problem, the only change needed is to make it easier to replace that list (or add to it?). At the moment only scrapping can populate it, but I don't see why that can't change. That seems a more robust solution than Python cache, and covers the XML scrapers too. 

Caching is for data that is short lived, and this list of previously available artwork is short lived and doesn't need to be robust. Doesn't belong in the database. If you gave add-ons the ability to modify this list they'd still have to update it when the user wants to pick new artwork if the list is old anyway, so giving it to Kodi to cache makes the process more complicated.

Very few people like working with the XML scrapers, so they should probably be phased out.

Edit: Took a look at your last build and it looks good, except songs in the current playlist aren't being updated when artwork for their (album)artist or album is changed. Changing the song artwork directly is reflected in the playlist.

For anyone that is confused it turns out that Ryan and I are having a discussion about the future of art handling addons/scrapers and Kodi requirements. I started it (I think) but it is beyond the work I am doing now, but useful all the same.

Ryan I'm not sure I'm getting all you are saying, and I fear you have misunderstood me, so a reprise and some questions.....

Scrapers or other addons to fetch art:

(2018-02-28, 18:42)DaveBlake Wrote: [ -> ]Scrapers are addons true, but they get invoked by a special part of core. Maybe we are going the wrong way with that, stuck in an historic groove. An addon does not have to be a "scraper" to add art, so maybe we leave scraping just to identifying music (or video) and all the fetching of art for things is done by other addons?

(2018-03-01, 06:41)rmrector Wrote: [ -> ]Two separate and unrelated processes to fill library data is awkward and complicates things. I still think composing multiple scrapers is the way to go here.

The downside to that is that scraper addons are bound by core scraper processing, while a general addon that used JSON to change the art in DB would not be. Having more scrapers does not prevent other art addons existsing, so any awkward complication continues. But I was just musing, not suggesting we actually stop scraping!

Fair  use of online resources:

(2018-03-01, 06:41)rmrector Wrote: [ -> ]I don't think that there are millions of Kodi boxes manually managing artwork by hand in the Kodi GUI so frequently that free resources will even notice, nevermind be killed.

I agree, manual management is not a problem, but automated refresh could be.

No it is not actual the download of images that is the issue, but unnecessary repeated requests for info including possible artwork that we need to avoid.

You seemed to not like having scrapper results (art URLs) held in the db, seeing it as too static and getting quickly out of date, and to suggest frequent rechecking of on line sources for changes to the available art instead. This is what gives me concerns.

Resource abuse is not just an issue for pirate hacks, and the first consequence is failed requests (404 and 503 errors) rather than the end of that site, but don't underestimate the danger and irresponsibility of Kodi being greedy. Sites like Musicbrainz enforce limits - must be less than 1 request per second per IP address or all requests get rejected) TADB just runs out of server time and stops working towards the end of the month. And one art site has vanished without trace in the last few years, simply ran out of money for running servers.

I know you know this and are responsible with your addons Ryan, I was not suggesting otherwise, but I do think resource protection needs to be highlighted when there is talk about fetching art, and designed in as much as possible.


Info provider driven refresh:
Having the info provider API tell the scraper the "shelf-life" is an interesting idea, but surely those dates are going to be set to "forever" if the site wants to avoid lots of traffic jambing their servers? Sites have no way to know when a volunteer is going to add more art for an artist or album, or even a plan for adding new art types. Could TADB have told requests made last year that new types would be ready now?

Outdated art:

(2018-02-28, 18:42)DaveBlake Wrote: [ -> ]Do the scraped urls for possible thumb or fanart really get out of date so quickly?

(2018-03-01, 06:41)rmrector Wrote: [ -> ]Yes. Services may add new artwork types (TADB just added clearart and clearlogo), and quite often new artwork is made for existing media...  

No doubt with cloud sourced art (and info) new things get added all the time, but how many get removed? By "getting out of date" I meant when the user goes to pick an alternative image from the scraped list of art URLs then find the image does not exist. Sure it could be out of date because there is now more art to pick from.
Perhaps the real questions are:
a) do the info/art providers want Kodi to try to sync with every site change, and do they have the server power for the traffic it could mean?
b) do users want their music art to keep changing?
It is a guess, but I bet the majority of users get art at the beginning, and then happily keep using the default first image of those scraped for years. Others I know like to pick art and then keep it. They like to fill in any blanks, but I don't believe the want to just use different images sometime later.

In the end I am confused by what it is you see as the problem, or propose it is replaced with. My suggestions were a response to what it seemed you wanted, but you declined them, so I have misunderstood. Try again please.

Where to keep scraper results:
Again I offer the music db as an obvious and useful place to keep common data that relates to artists and albums, as an alternative to having some Python cache sitting in files in userdata. It could even be part of the core scraper processing, ensuring that scraper addons do behave responsibily.

(2018-03-01, 06:41)rmrector Wrote: [ -> ]Caching is for data that is short lived, and this list of previously available artwork is short lived and doesn't need to be robust.

The shelf-life of available art seems to be where we have different perceptions or experiences. In all the testing I have done the available music artwork seems to be very long lived and static. When I scrape the artists now I get pretty much the same possible art as I did 3 years ago. A few blanks get filled in, previously unknown artists have been added, but I can compare the data and it is stable. Perhaps it is down to my taste in music!!

Because of this long life I think it is something that should be in the db, and can only be refreshed manually artist or album at a time.

Your short life view is that it should be only in cache (no list of urls in db at all), and updated whenever an addon (based on data from the site) decides to do it.

Where between these views lies the best solution?
 

(2018-03-01, 06:41)rmrector Wrote: [ -> ]If you gave add-ons the ability to modify this list they'd still have to update it when the user wants to pick new artwork if the list is old anyway, so giving it to Kodi to cache makes the process more complicated.

In effect you want to refresh the list of available remote art every time the user brings up the art selection dialog. What is wrong with the refresh being manual as it is now?


XML Scrapers:

(2018-03-01, 06:41)rmrector Wrote: [ -> ]Very few people like working with the XML scrapers, so they should probably be phased out.

Yes the reg exp on XML makes my head hurt too. A start has been made on Python music scrapers, but they need more work from someone with Python experience. Feel free to volunteer.

Meanwhile I am glad to say that Olympia does a great job keeping the XML based ones working. While the music resource sites still have XML APIs we can at least keep functioning.

Ya, I pulled us off the original path here. It is related, though, so thanks for entertaining my ideas.

(2018-03-01, 13:28)DaveBlake Wrote: [ -> ]In the end I am confused by what it is you see as the problem, or propose it is replaced with. My suggestions were a response to what it seemed you wanted, but you declined them, so I have misunderstood. Try again please.

As a user, I want to be able to fill in the blanks (manually) for artwork or change an existing image (manually) on media that has existed in my library for a year or more without refreshing the entire media item. I want all currently assigned artwork and all other info to stay the same unless I specifically choose otherwise. Nothing automatic, only when I want to choose from currently available artwork as potential replacements. If I can browse to that artist in my web browser and see the available artwork without potentially changing other data on the media item, I should also be able to browse the same place in Kodi (if Kodi offers the option at all) and see the same available artwork without potentially changing other data on the media item.

As a developer I see this data as cache and placing it in the database adds an unnecessary bit of complication to Kodi, and doing more work in Kodi so that scrapers and add-ons can share and update it is more unnecessary complication still. I wanted to simplify Kodi a bit by moving this to scrapers, which do need to be responsible for their caching behavior anyway (sources other than HTTP will have their own rules for caching, and Kodi can't know about them all).

(2018-03-01, 13:28)DaveBlake Wrote: [ -> ]Scrapers or other addons to fetch art:
The downside to that is that scraper addons are bound by core scraper processing, while a general addon that used JSON to change the art in DB would not be. Having more scrapers does not prevent other art addons existsing, so any awkward complication continues.

That may not be a bad boundary, it simplifies things. Kodi is responsible for identifying new items and then asking scrapers for the initial information, however the user configures it. Add-ons have to recreate that "identifying new items" step, but scrapers do not.

It doesn't prevent additional add-ons from existing, but if scrapers / "information providers" can populate all data that a user wants up front then fewer other add-ons need an awkward additional process; other add-ons will just be used to change existing data later if users wanted to. Only a few add-ons will need an automated process to periodically update data for existing items (like IDMB/Trakt/Rotten Tomatoes ratings that change over time).
 

(2018-03-01, 13:28)DaveBlake Wrote: [ -> ]Info provider driven refresh:
Having the info provider API tell the scraper the "shelf-life" is an interesting idea, but surely those dates are going to be set to "forever" if the site wants to avoid lots of traffic jambing their servers? Sites have no way to know when a volunteer is going to add more art for an artist or album, or even a plan for adding new art types. Could TADB have told requests made last year that new types would be ready now?

The same caching mechanisms are available to the API as the web pages, so if browsers aren't killing the web services by refreshing old data when the user visits the respective artist/album page then neither will Kodi when the user wants to manually see new artwork. And they absolutely do have the ability to set it for a long time (I think 1 full year is generally the max), and we'd respect it just like any other number. But there is a big stretch between caching "forever" and lots of traffic jambing their servers, so there is surely a happy medium for services that are specifically designed to collect and disseminate this data, often through an API.

HTTP caching headers are already used by some of these APIs, and they are at 8 and 24 hours (if I remember right for the services Artwork Beef uses, but it is low). HTTP caching has been available for APIs as long as APIs have used HTTP with separately addressable resources. In addition to time-based caching, HTTP caching also offers Etags, which are basically a hash of the resource, and if the Etag of a particular resource is the same then the server can respond "Not modified"*. Web services can also throttle requests, as you've noted for MusicBrainz. TheMovieDB also throttles, but not as aggressively. The time-based cache is fairly effective at catching the most likely clusters of changes; changing the automatically selected artwork after scanning in new media, and then later looking for new artwork and applying several changes.

* Etags are a bust for APIs that have smaller payloads right now, though: HTTP/1.1 requests are expensive enough that the Etag check isn't much cheaper on the HTTP layer than small API responses, and with a caching layer between the web request and their application server the API response itself will add minimal load beyond the HTTP layer, so still a wash. HTTP/2 will likely change the situation, but that's in the future (near-ish, but not yet).

(2018-03-01, 13:28)DaveBlake Wrote: [ -> ]Outdated art:
how many get removed? By "getting out of date" I meant when the user goes to pick an alternative image from the scraped list of art URLs then find the image does not exist. Sure it could be out of date because there is now more art to pick from.
Perhaps the real questions are:
a) do the info/art providers want Kodi to try to sync with every site change, and do they have the server power for the traffic it could mean?
b) do users want their music art to keep changing?

Artwork URLs are moved or removed often enough that a respected voice in this community suggests downloading video library artwork instead of using artwork URLs in the library for assigned artwork. Available artwork has the same concern, with the addition of presenting old artwork options to users that are trying to select new artwork (without refreshing the entire item).

a) No
b) No

(2018-03-01, 13:28)DaveBlake Wrote: [ -> ]In all the testing I have done the available music artwork seems to be very long lived and static.

And all my testing of extended artwork in the video library since Frodo says the opposite to me, too much of it will go out of date and the whole item has to be refreshed for the list of available artwork to be updated. As does Milhouse who suggests users download artwork rather than leave URLs to artwork assigned in the library because they go out of date.

(2018-03-01, 13:28)DaveBlake Wrote: [ -> ]XML Scrapers:

(2018-03-01, 06:41)rmrector Wrote: [ -> ]Very few people like working with the XML scrapers, so they should probably be phased out.

Yes the reg exp on XML makes my head hurt too. A start has been made on Python music scrapers, but they need more work from someone with Python experience. Feel free to volunteer.

Meanwhile I am glad to say that Olympia does a great job keeping the XML based ones working. While the music resource sites still have XML APIs we can at least keep functioning.                                                                  

I do what I can, though I'm iffy on core changes. However, I am a poor communicator (you got pretty much the opposite of what I was trying to say), and enough of the gatekeepers of Kodi have made it clear that they don't respect my contributions unless I communicate in a specific way, so contributing to Kodi core is not a very pleasant experience for me.

I too appreciate that Olympia is keeping the original scrapers working.