Kodi 17 - Music Library Improvements - Krypton

  Thread Rating:
  • 1 Vote(s) - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Thread Closed
DaveBlake Offline
Team-Kodi Member
Posts: 2,158
Joined: Jun 2015
Reputation: 56
Location: South West England
Post: #1
What is this
Please use this thread for both reporting and asking questions about the most recent changes to the way Kodi 17.0 Krypton processes the music file tags (ID3, APE2, Vorbis etc.) to create the music library and the associated new features.

If you are experiencing difficulties, or have questions, about released versions of Kodi then please create a separate thread.

As I hope you all know I have been working on improvements to the music library, slowly adding new features for release in Kodi 17.0 Krypton and fixing a few bugs. One big step has been the addition of the concept of artist roles, and I intend to bring together the various posts on that new feature in one place sometime soon. It is a little bit specialist, many users will not ever care about composers, DJ mixers or lyricists etc. (yet), and that is just fine. But there have also been a number of changes made to improve the way Kodi identifies individual artists from the music file tags, attempting to allow for some of the things that can go wrong with tagging, such as mismatching the number of Musicbrainz Ids to artist names. But also better accommodating music that is not tagged with Musicbrainz Ids, but has things like "artist1 feat. artist2".

What I'm looking for in this thread is for some brave volunteers to install the nightly build of Kodi 17 and test scanning their music collections. Then report back on how well (or not) Kodi behaved, any oddities or successes - both are useful feedback. It is important to know that there are no regressions in testing well tagged music file collections, but also to know that some of the less well tagged music files are being handled more robustly. I have obviously tested things with my own music collection, and with test data I have created specially, but none is as useful as real world testing "in the wild"

Other changes may mean that you want to re-tag some of your music files to take advantage of some of the new features in 17, but I would appreciate that you test with what you have first too. It is important to know what experiences a user upgrading Kodi versions will have.

Although intended as platform independant is also of interest to see how this performs on different platforms - I've only tested it on Windows. Some tests with MySQL databases would also be useful, although this is more relevent to other changes than the tag processing in particular.

Finally many thanks to those that have already tested some of these changes and provided essential feedback elsewhere. There is always more to be done, and I hope that having a common place to provide feedback will make the experience feel more easy, worthwhile (and maybe even fun?) for all of us.

What this isn't
A feature request thread. Please use the appropriate forum for that otherwise we all get lost.

Anything to do with Jarvis, Isengard or any older version of Kodi. I repeat - if you are experiencing difficulties, or have questions, about released versions of Kodi then please create a separate thread.

Download link
You can read about and download the development builds from here Development_builds (wiki)
Beware these change just about every day, and different areas may be more or less stable. Always take backups of your system before you start. I also recommend protable mode as a way of having more than one version of Kodi installed at a time.

New Features Edited: March 2017
v17.1 is released, finally time to summarize what happened.

Artist Roles
Added artist roles feature roles to handle Composer, Conductor, DJMixer etc. Tags PR8015

Added processing for the following tags:
COMPOSER (TCOM)
CONDUCTOR (TPE3)
LYRICIST (TEXT)
REMIXER (TPE4)
ENSEMBLE / BAND / ORCHESTRA (TXXX:Ensemble)
ARRANGER *
ENGINEER *
PRODUCER *
DJMIXER *
MIXER *
PERFORMER=artist (instrument) (TIPL and TMCL)

* from the TIPL tag in ID3v2 which is in pairs of values e.g. "Mike Hatch", "Engineer"

But Kodi also processes the pairs of values in the PERFORMER (TMCL) tag, and any other TIPL entries, so role could be absolutely anything. Commonly it is used for instruments e.g. "Joe Fish", "Drummer", or "Yo Yo Ma", "Cello", but there is lots of diversity.

These tags are all about people (or groups of people) that contribute in some way to the recording as musicians or are in some other way involved. They are added to the music library as artists with a role. They can be listed separately from general artists (those scanned from the albumartist and artist tags) using custom nodes which can be configured to reflect the users interests. They will not be displayed on the original artist lists, so those not interested in this extra data will be uneffected.

Added a Roles node, similar to genre, that shows all the artist roles and navigates to the artists with that role

Currently there are default nodes for the standard roles, but users have to make custom nodes for the other roles they are interested in. Using custom nodes role filtering can be combined with genre e.g. a node for classical music composers, or pop music drummers. That is flexible and powerful, but hard work for new users. A simple node showing all roles that have been scanned into the library is a useful addition.
It creates a new content type of "roles", Estuary modified to display this, Confluence seems to just work.
A node file with
Code:
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<node order="1" type="filter" visible="Library.HasContent(Music)">
<label>Roles content</label>
<content>artists</content>
<group>roles</group>
</node>
or alternatively
Code:
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<node order="1" type="filter" visible="Library.HasContent(Music)">
<label>Roles Path</label>
<path>musicdb://roles</path>
</node>

New default submenu and nodes for the standard roles
  • composer, conductor, orchestra, arranger, lyricist, DJmixer, remixer,
  • all roles - standard roles, plus musician credits e.g. "piano", and people involved e.g. engineer
  • all contributors - every artist that Kodi knows about regardless of role they contribute to the song
  • all artists - song and album artists
  • album artists only


Album (Release) Type
http://forum.kodi.tv/showthread.php?tid=...pid2561628
(This post was last modified: 2017-04-03 16:45 by DaveBlake.)
find
zag Offline
Retired Team-Kodi Member
Posts: 4,006
Joined: Oct 2007
Reputation: 75
Location: UK
Post: #2
I've tested on my collection with last nights nightly and everything is working as it should.

One thing I noticed that "artists who only appear on a compilation" does not get any metadata.

I think this is the correct way it should be(due to speed and api lookup issues) but some users may find this confusing if lots of artists do not show thumbs on initial scan.
find
DaveBlake Offline
Team-Kodi Member
Posts: 2,158
Joined: Jun 2015
Reputation: 56
Location: South West England
Post: #3
Thanks for feedback @zag Smile

Quote:One thing I noticed that "artists who only appear on a compilation" does not get any metadata.

Enabling the "show artists who only appear on a compilation"setting might result in seeing artists without art until you initiate some more scraping. But if you have "show artists who only appear on a compilation" enabled before you load new music and the metadata scraper is set going then it will (try) to get the data for song artists as well as album artists.

The response from MB is so bad at the moment it is hard to test practically, and there could be something in the more selective scraping I added that is limiting things on retries, but the gaps you see could be 403 errors, did you check?
find
zag Offline
Retired Team-Kodi Member
Posts: 4,006
Joined: Oct 2007
Reputation: 75
Location: UK
Post: #4
The universal scraper has a new "mirror" option.

Just change that to http://musicbrainz.fin-alice.de:5000/ and it should work 100%

There is also this mirror as a backup

http://musicbrainz-mirror.eu:5000/
find
scott967 Offline
Posting Freak
Posts: 2,703
Joined: Jul 2012
Reputation: 84
Post: #5
I need to do some testing on this, but IIRC changes were made in Krypton for how Kodi computes "total artists" and it was a lot more useful, but that was before the addition of roles. I think total artists should probably be total performers? Also I have some songs with composer tagged as "traditional" or "unknown". Not sure what the "right" answer is, just throwing it out there.

scott s.
.
find
scott967 Offline
Posting Freak
Posts: 2,703
Joined: Jul 2012
Reputation: 84
Post: #6
Bug: Win32 Kodi 17 A2 0523 (Win 7)

I have several albums with FLAC tags for CONDUCTOR. These are scanned into the library, and looking at "artist" table the conductor is added as an artist. The "song_artist" table has the conductor added as idRole 4 and strArtist is correct in the table. Viewing the music library in Estuary, navigating to roles -> conductors and the view is empty. (musicdb://artists/?role=Conductor).

This is the query:

Code:
16:23:40 T:10112   DEBUG: CGUIMediaWindow::GetDirectory (library://music/musicroles/Conductors.xml/)
16:23:40 T:10112   DEBUG:   ParentPath = [library://music/musicroles/]
16:23:40 T:10784   DEBUG: CMusicDatabase::GetArtistsByWhere query: SELECT artistview.* FROM artistview  WHERE (artistview.idArtist IN (SELECT album_artist.idArtist FROM album_artist ) AND artistview.idArtist IN (SELECT song_artist.idArtist FROM song_artist WHERE song_artist.idRole = 4)) and artistview.strArtist != ''
16:23:40 T:10784   DEBUG: Time to retrieve artists from dataset = 2
16:23:40 T:10112   DEBUG: Custom album format = [%B]
16:23:40 T:10112   DEBUG: Previous line repeats 1 times.

I don't understand why the dependency on album_artist.idArtist? None of the conductors are album artists.

scott s.
.
find
DaveBlake Offline
Team-Kodi Member
Posts: 2,158
Joined: Jun 2015
Reputation: 56
Location: South West England
Post: #7
(2016-05-25 03:38)scott967 Wrote:  I need to do some testing on this, but IIRC changes were made in Krypton for how Kodi computes "total artists" and it was a lot more useful, but that was before the addition of roles. I think total artists should probably be total performers? Also I have some songs with composer tagged as "traditional" or "unknown". Not sure what the "right" answer is, just throwing it out there.

Total artists returned each time I think applies the current filtering, addition of roles has not changed that. So, for example, if you have the albumartists only flag set the artists "total" is the number of album artists (not song artists or performers of any other roles). You can get the total number of performers the contribute something to your music with the right custom node.

The perhaps confusing thing is that in order for the addition of roles not to impact default behaviour the extra astists Kodi now can hold are hidden until you request them.
find
DaveBlake Offline
Team-Kodi Member
Posts: 2,158
Joined: Jun 2015
Reputation: 56
Location: South West England
Post: #8
(2016-05-25 04:38)scott967 Wrote:  I have several albums with FLAC tags for CONDUCTOR. These are scanned into the library, and looking at "artist" table the conductor is added as an artist. The "song_artist" table has the conductor added as idRole 4 and strArtist is correct in the table. Viewing the music library in Estuary, navigating to roles -> conductors and the view is empty. (musicdb://artists/?role=Conductor).
...
I don't understand why the dependency on album_artist.idArtist? None of the conductors are album artists.

By default the global albumartistonly flag (inverse of the incorreclty named "Include artists that onlt appear on compilations" setting) is obyed. So if your artists node shows only album artists then the conductor list shows only conductors that are also album artists. This setting can also be localy overridden by custom node with

<path>musicdb://artists/?role=Conductor&albumartistsonly=false</path>

that will show all conductors.

The use of albumartistsonly option also works in any artist nodes, so you could have a custom node to show song artists and another just album artists.
find
scott967 Offline
Posting Freak
Posts: 2,703
Joined: Jul 2012
Reputation: 84
Post: #9
(2016-05-26 08:15)DaveBlake Wrote:  By default the global albumartistonly flag (inverse of the incorreclty named "Include artists that onlt appear on compilations" setting) is obyed. So if your artists node shows only album artists then the conductor list shows only conductors that are also album artists. This setting can also be localy overridden by custom node with

<path>musicdb://artists/?role=Conductor&albumartistsonly=false</path>

that will show all conductors.

The use of albumartistsonly option also works in any artist nodes, so you could have a custom node to show song artists and another just album artists.

OK got it. I guess it's a feature, but I struggle to see a use case where you would want a role view that only selected from album artists.

scott s.
.
find
DaveBlake Offline
Team-Kodi Member
Posts: 2,158
Joined: Jun 2015
Reputation: 56
Location: South West England
Post: #10
(2016-05-27 21:54)scott967 Wrote:  OK got it. I guess it's a feature, but I struggle to see a use case where you would want a role view that only selected from album artists.

A use case is for similar reasons to seeing just album artists rather than all the song artists, reducing clutter. Say you had tagged so that most tracks have a composer, you may not always want to see all of them but just the primary ones e.g. those that are also album artists. This works particularly well when also combined with a genre filter - a way to see the classical album composers not jumbled with conductors and orchestras etc.

But I am open to feedback. Maybe the default behaviour should be albumartistsonly=false regardless of the global setting?

At the moment the only way to adjust what is seen is using custom nodes, perhaps an easier UI is needed? Initially I guess a clear wiki explantaion or tutorial video will have to suffice.
find
scott967 Offline
Posting Freak
Posts: 2,703
Joined: Jul 2012
Reputation: 84
Post: #11
(2016-05-28 08:56)DaveBlake Wrote:  A use case is for similar reasons to seeing just album artists rather than all the song artists, reducing clutter. Say you had tagged so that most tracks have a composer, you may not always want to see all of them but just the primary ones e.g. those that are also album artists. This works particularly well when also combined with a genre filter - a way to see the classical album composers not jumbled with conductors and orchestras etc.

But I am open to feedback. Maybe the default behaviour should be albumartistsonly=false regardless of the global setting?

At the moment the only way to adjust what is seen is using custom nodes, perhaps an easier UI is needed? Initially I guess a clear wiki explantaion or tutorial video will have to suffice.

I used the official library node editor, and that works but maybe not that easy for a typical user. I suppose a lot depends on what prior conventions users have for managing classical. For my part, I used "artist" for composer and "album artist" for orchestra - conductor and with the new tags don't have as much a need for those any more. I suppose I could use "album artists" hints in which case I guess "compilation artists only" would still be useful.

I am finding working with "composer" to be an issue. I think the only way to get a grouping or filter for classical composers is by getting the strGenres field set in the artist db table. That is a tedious process.

scott s.
.
find
DaveBlake Offline
Team-Kodi Member
Posts: 2,158
Joined: Jun 2015
Reputation: 56
Location: South West England
Post: #12
Thanks for more feedback Scott.
(2016-05-31 01:47)scott967 Wrote:  I used the official library node editor, and that works but maybe not that easy for a typical user. I suppose a lot depends on what prior conventions users have for managing classical. For my part, I used "artist" for composer and "album artist" for orchestra - conductor and with the new tags don't have as much a need for those any more. I suppose I could use "album artists" hints in which case I guess "compilation artists only" would still be useful.
Yes prior user approach to tagging classical does make a difference, but it is quite diverse so almost impossible to allow for.

Unless skins start to show composer on the music player or songs node (info labels make that possible, but I guess not every user wants to see composer every time, so a skin design question) then having the composer in the artist tag remains useful as it gets displayed. This is historically why I put compser in both artist and album artist tags, so it was displayed on the album list and while a song played. Another approach to that was to include composer in the song title, but some titles are way too long already and datawise that felt naff to me.

Quote:I am finding working with "composer" to be an issue. I think the only way to get a grouping or filter for classical composers is by getting the strGenres field set in the artist db table. That is a tedious process.

Save your efforts artist genre is not going to help!!! The scraped genre field in the artist table is only ever used for display, never filtering. Kodi implicitly uses song genre even in artists playlists, a rule of genre=classical will find those artists with songs that have the genre "classical".

Assuming that you have tagged songs with a genre of classical then the following folder type node will show just those composers regardless of the system settings for displaying all song artist.
Code:
<?xml version='1.0' encoding='UTF-8'?>
<node order="13" type="folder" visible="Library.HasContent(Role, Composer)">
    <label>Classical Composers All</label>
    <path>musicdb://artists/?role=Composer&genre=Classical&albumartistsonly=false</path>
</node>

Tempting to make that a standard node... then again not every user has classical music. Equally that approach could get hip-hop DJ Mixers, Jazz Lyricists or Funk Guitarists etc. (if the music files are tagged with that data)
find
scott967 Offline
Posting Freak
Posts: 2,703
Joined: Jul 2012
Reputation: 84
Post: #13
(2016-05-31 09:28)DaveBlake Wrote:  Save your efforts artist genre is not going to help!!! The scraped genre field in the artist table is only ever used for display, never filtering. Kodi implicitly uses song genre even in artists playlists, a rule of genre=classical will find those artists with songs that have the genre "classical".

Assuming that you have tagged songs with a genre of classical then the following folder type node will show just those composers regardless of the system settings for displaying all song artist.
Code:
<?xml version='1.0' encoding='UTF-8'?>
<node order="13" type="folder" visible="Library.HasContent(Role, Composer)">
    <label>Classical Composers All</label>
    <path>musicdb://artists/?role=Composer&genre=Classical&albumartistsonly=false</path>
</node>

Tempting to make that a standard node... then again not every user has classical music. Equally that approach could get hip-hop DJ Mixers, Jazz Lyricists or Funk Guitarists etc. (if the music files are tagged with that data).

That will make it quite a bit easier. I was getting ready to hand-edit my node xmls. IMHO the Kodi wiki doesn't really help much in understanding exactly how the node code works (for example, doesn't explain "filter" vs "folder"). And your example is using "folder". Why is that? could you do the same with "filter" and then create rules?

scott s.
.
find
bsoriano Offline
Donor
Posts: 709
Joined: Mar 2011
Reputation: 19
Location: Santo Domingo, Dominican Republic
Post: #14
Dave,

I am a longtime Kodi user who on and off has used Kodi for music. The reason I keep deleting my music database from Kodi is that I have a large collection (over 145,000 tracks: mp3s, FLAC and some m4a) and it has usually been really difficult (and slow!) to get everything scanned properly into Kodi, or to get meaningful metadata for all of the artists and albums in my collection (which covers a very wide range of genres). I tag all of my files in a way that works for me and does not leave empty or with "garbage" the main tags: All of my files have an artist, albumartist, name, track number, disc number, album, genre and year. All of the songs which have more than one artist performing have "Artist 1 / Artist 2 / Artist n" in the Artist tag, and all of my compilation albums are tagged with an album artist "(Multiple Artists)". However, the program I use to tag my files (and manage and listen to my music collection, J River Media Center) does not use Musicbrainz IDs.

Are you interested in tests with a collection like what I describe above? Please let me know, since I am willing to try once more to use Kodi for music with Krypton. Thanks.

Regards,

Bart
(This post was last modified: 2016-06-01 23:19 by bsoriano.)
find
DaveBlake Offline
Team-Kodi Member
Posts: 2,158
Joined: Jun 2015
Reputation: 56
Location: South West England
Post: #15
(2016-06-01 21:20)scott967 Wrote:  IMHO the Kodi wiki doesn't really help much in understanding exactly how the node code works (for example, doesn't explain "filter" vs "folder"). And your example is using "folder". Why is that? could you do the same with "filter" and then create rules?

There is a thread of example nodes here http://forum.kodi.tv/showthread.php?tid=257378, but I agree that the wiki could do with improvement. My understanding is gained initially from trial and error and then digging through the code.

I use folder type cutom nodes because I prefer how they behave compared to filter type or smart playlist rules.
In folder type nodes:

a) there is an optional "* all" entry that navigates to the next level down e.g. all the albums for those artists, or all the songs for those albums.

b) the filtering passes down the levels e.g. showing artists with (songs with) genre=classical, click on artist you see the albums by that artist with (songs with) genre=classical, click on an album and you see those songs with that genre.

With filter type rules they just apply to the top level. So you could select artists with songs you have played once and they get listed, but driling down from such an artist you would se all their albums, and then all their songs not just the ones played once.

The SQL in folder type is also more efficient, but that is just my database brain working overtime, the filter type works well enough.

But yes you can use role in a rule if you want to. In fact if there are other more complecated filtering you want then filter type is the only way.
find
Thread Closed