Issue with Music Years function after scraping
#1
My album scraper has updated the releasedate and year fields for my albums. Now when I view my music by year the album shows up but its songs do not.

It appears that Kodi is issuing a query on songs based on the year as well (which seems odd). I can see this in my log:

SELECT songview.*, songartistview.*
FROM songview JOIN songartistview ON songartistview.idsong = songview.idsong 
WHERE (songview.iYear = 1959) AND (songview.idAlbum = 852) ORDER BY songartistview.idsong, songartistview.idRole, songartistview.iOrder


This doesn't seem right to me, it is filtering for songs on both the album id and the song date. Once the scrapers have been run there is no guarantee that the song dates will match the album ones.

The query seems to be coming from: xbmc\music\MusicDatabase.cpp

    option = options.find("year");
    if (option != options.end())
      filter.AppendWhere(PrepareSQL("songview.iYear = %i", static_cast<int>(option->second.asInteger())));
Reply
#2
I believe that for songs the year shown is always taken from the value in the database which comes from the Year value in the music file metadata e.g. id3 tag TYER
Reply
#3
(2019-06-15, 16:07)jjd-uk Wrote: I believe that for songs the year shown is always taken from the value in the database which comes from the Year value in the music file metadata e.g. id3 tag TYER

That seems right ( I can't see any place in the kodi code where the scraper can update the song data). This leads to weird results when the scraper has changed the album date

The Listings for year 1970

Image

The song listings  in Year->1970->Let It Be->
Image

My log is attached here:
Log

The query that looks wrong:
query = SELECT songview.*, songartistview.* FROM songview JOIN songartistview ON songartistview.idsong = songview.idsong  WHERE (songview.iYear = 1970) AND (songview.idAlbum = 1309) ORDER BY songartistview.idsong, songartistview.idRole, songartistview.iOrder
Reply
#4
Again I believe, scraped info is only supposed to supplement data taken from the tags and not replace it, info from tags is always king @DaveBlake can give the definitive answer.
Reply
#5
If the album year gets set by scraping to something different from the song year (derrived from the tags) then yes you will have problems.  If you have "Prefer online information" enabled then scraped album results will replace the initial album year value taken from the  songs.  I recommend disabling  "Prefer online information" (it is by default) - tagging is only "king" if it is disabled.

If you like I can suggest some SQL that will restore the album year to match that of the song year.

As for the query looking wrong, well it is as it is designed to be. The approach has always been when navigating down selecting items to pass the criteria applied to one level down to the next. This makes sense for genre > artist > album > song: pick a genre and show artists with albums with songs that have genre X; then pick an artist and see just those albums by that artist that  with songs that have genre X; finally pick an album and see just those songs with genre X. Maybe it is not so helpful with  year > album > song (since both album and song have year data in a denormalisation to speed access), but so far there is no special treatment of year parameters.

Of course we could take a single level filter approach to all navigation e.g. find an album that has some songs with certain criteria ( featuring an artist, or with a genre) but then show all the songs on the album whether they match the criteria. However this would be a change to how Kodi has behaved for many years. Since generally all the songs on an album have the same year, genre(s) and artists this way filter criteria are applied is often not noticed, but there is nothing to say all the songs on an album have to be so uniform.

I agree the year handling in this way has potential flaws, but keeping "Prefer online information" diasabled avoids issues.
Reply
#6
(2019-06-16, 15:56)DaveBlake Wrote: If the album year gets set by scraping to something different from the song year (derrived from the tags) then yes you will have problems.  If you have "Prefer online information" enabled then scraped album results will replace the initial album year value taken from the  songs.  I recommend disabling  "Prefer online information" (it is by default) - tagging is only "king" if it is disabled.

If you like I can suggest some SQL that will restore the album year to match that of the song year.

As for the query looking wrong, well it is as it is designed to be. The approach has always been when navigating down selecting items to pass the criteria applied to one level down to the next. This makes sense for genre > artist > album > song: pick a genre and show artists with albums with songs that have genre X; then pick an artist and see just those albums by that artist that  with songs that have genre X; finally pick an album and see just those songs with genre X. Maybe it is not so helpful with  year > album > song (since both album and song have year data in a denormalisation to speed access), but so far there is no special treatment of year parameters.

Of course we could take a single level filter approach to all navigation e.g. find an album that has some songs with certain criteria ( featuring an artist, or with a genre) but then show all the songs on the album whether they match the criteria. However this would be a change to how Kodi has behaved for many years. Since generally all the songs on an album have the same year, genre(s) and artists this way filter criteria are applied is often not noticed, but there is nothing to say all the songs on an album have to be so uniform.

I agree the year handling in this way has potential flaws, but keeping "Prefer online information" diasabled avoids issues.

Thanks for the prompt replies guys Smile, no need for any SQL this issue came about because I had deliberately modified my album scraper to override the releasedate and year to the original release date of the release group. I did this because the musicbrainz release date for many of my albums is for the remastered CD date and not for the original release date. This makes the year->album filtering not that useful, when you're looking for 60's music and you find it in the 80's and 90's.

It would be nice if the album scraper could also modify the scraped song data as well, this would allow us to set the song date and the album date at the same time as well as allowing us  to update fields like the song comment (I'd love to put performance data such as the recording artists, producers, engineers, location and dates into the song).
Reply
#7
(2019-06-16, 16:21)bog_gob Wrote: I had deliberately modified my album scraper to override the releasedate and year to the original release date of the release group. I did this because the musicbrainz release date for many of my albums is for the remastered CD date and not for the original release date. This makes the year->album filtering not that useful, when you're looking for 60's music and you find it in the 80's and 90's.
I'm aware of the original release date issue and looking at improving the way Kodi processes (or doesn't) the various date information tags to accomodate this.

(2019-06-16, 16:21)bog_gob Wrote: It would be nice if the album scraper could also modify the scraped song data as well, this would allow us to set the song date and the album date at the same time as well as allowing us  to update fields like the song comment (I'd love to put performance data such as the recording artists, producers, engineers, location and dates into the song). 
I don't think it is likely that Kodi will start writing/editing music files tags, so many users are glad that Kodi does not mess with their precious files and there are already good tools to get your tags in good shape. Neither is it likely we swill start to scrape data for songs (much of the tag data identifies the song, all other data can be added as a tag). The idea of synching with Musicbrainz changes without retagging was never implemented, and again I think more ikely to cause user distress than joy. You need to tag well enough to identify the track accurately, having done that few want it to move unexpectedly beneath their feet.

But it is possible that I look at how and where album year is used compared to song year (whichever tag we use to get "year" for each song). There is a conceptual issue here, taking album year from the first music file and assuming that is consistent isn't good enough. Having year as denormalised data makes sense, but it either needs to be protected from scraper changes or the filtering needs to separate album year from song year.
Reply
#8
(2019-06-16, 18:44)DaveBlake Wrote: But it is possible that I look at how and where album year is used compared to song year (whichever tag we use to get "year" for each song). There is a conceptual issue here, taking album year from the first music file and assuming that is consistent isn't good enough. Having year as denormalised data makes sense, but it either needs to be protected from scraper changes or the filtering needs to separate album year from song year. 

I've had some thoughts on this myself, what date should a song be associated with - its release date or its recording date? For normal albums there is little difference but for compilation albums or re-releases it makes a big difference. I guess it depends on whether the songs in kodi are only conceptually a (musicbrainz) track or can alllow for (musicbrainz)  recording data as well.

Having recording style DB fields for songs in kodi would allow:
  1. Scrapers to bring in additional information that music taggers like picard leave out
    1. The original recording date (very useful for songs that are part of compilations/ bet of albums) ,
    2. musicbrainz work data (very useful for classical) e.g the date a work was composed, would also allow us to filter by work (now you can listen to all performers performances of a particular work)
    3. etc....
  2. Allow kodi to provide more filtering features e.g. have an option to only show 1 copy of a song in the song list when the Musicbrainz Recording Id is the same ( a pet peeve of my wife)
Reply
#9
I you accept MB theory of db schema, recordings (a unique audio) do not have an associated date.  MB uses a "release date" and i don't see anything associated with a release group.  My guess is that original date is a derived value from the oldest date of a release in a release group.


scott s.
.
maintainer of skin  Aeon MQ5 mods for post-Gotham Kodi releases:
Matrix see: Aeon MQ5 Mod Matrix release thread
Nexus see: Aeon MQ5 Mod Nexus release thread
Aeon MQ 5 skin and addon repo 11.1.0
Reply
#10
MusicBrainz does have a date field for Recordings, but rarely filled in.
Image

It needs someone to do a lot of editing to catch this up. Recorded At is most common to have a date.  It is rare to see dates next to the artists.  This is one of the more complete examples.

Sometimes the dates will only be stored at a Release level. It depends on the details on the cover art and other research.

A problem can be seen in this example - a range of dates that cross a year barrier. Fun.

As to "original release date" for an album, it is just worked out from "oldest Release in a Release Group". It needs someone to have added the original release from somewhere. Not always the case for lesser known bands.


Everything in the MB DB relies on someone having typed it in. Some bands are better detailed than others.
Reply
#11
(2019-06-17, 19:35)scott967 Wrote: I you accept MB theory of db schema, recordings (a unique audio) do not have an associated date.  MB uses a "release date" and i don't see anything associated with a release group.  My guess is that original date is a derived value from the oldest date of a release in a release group.


scott s.
.

You are correct that there is no explicit date field associated with the recording.  Musicbrainz does still provide a mechanism for determining the recording date range via AREs e.g.: https://musicbrainz.org/recording/e2064c...58f3b9a9ce  you can clearly see the date of recording. We could scrape selective ARE data and store it in a new field, it could even be in the ARE form if that is preferable.

The work for the recording https://musicbrainz.org/work/62c39e8e-18...5f8e2b547d also has links to things like wikidata https://www.wikidata.org/wiki/Q864783. which can provide lots of rich date data.
As you can see here is a lot of additional (non date) data available as well for us to scrape apart from the date which we could associate with the song (the . I'm not suggesting that these fields need to mirror or overwrite existing ID3 tags. We could introduce fields specifically for the scraper to fill. The album/artist scraper I'm playing with grabs the wikidata entry for a musicbrainz release/artist and pulls the wikidata->wikipedia page data (in text form - choosing the appropriate language) into the description/ bio field, and uses other  wikidata fields e.g. the left field is the kodi artist data and the right is the wikidata field:
 
                    ('genre',    'P136'),   
                    ('gender',    'P21'),  
                    ('type',     'P31')
                    ('born',    'P569'),
                    ('died',    'P570'),
                    ('formed',    'P571'),
                    ('formed',    'P571'),
                    ('disbanded',    'P576')


There are similar fields for a recording e.g. publication date (P577), composer (P86) etc...
Reply

Logout Mark Read Team Forum Stats Members Help
Issue with Music Years function after scraping0