Scraping music data
#31
(2015-11-25, 15:47)phool Wrote: I use Foobar2000 to tag, manage and listen to my music library on a regular basis. As such I have become accustomed to being able to manipulate and customise the presentation of my library based on various custom tags.
....
However, my understanding is that custom tags are not readable at this stage and thus I would not be able to display my library in this way.

The way Kodi tag processing and UI is implemented it is very hard to incorporate custom tags. For example every piece of data that a skin can show has a unique ID, and every tag is explicity listed and stored. If I could have a Foobar front end running on Android, controlling a Kodi on RPi backend then I would. Sigh.

Having said that I totally get what you would like to be able to do. I use my own genre system (as a kind of custom tag), and there is the comments, mood and style tags too... all can be seconded into personal use but not quite enough. I have it at the back of my mind, I can see the data structure that would work, UI side of things is harder, so a possible maybe Smile
Reply
#32
Updated wiki again with DaveBlakes explanation.

http://kodi.wiki/view/Music_tagging
Reply
#33
(2015-11-25, 20:54)DaveBlake Wrote:
(2015-11-25, 15:47)phool Wrote: I use Foobar2000 to tag, manage and listen to my music library on a regular basis. As such I have become accustomed to being able to manipulate and customise the presentation of my library based on various custom tags.
....
However, my understanding is that custom tags are not readable at this stage and thus I would not be able to display my library in this way.

The way Kodi tag processing and UI is implemented it is very hard to incorporate custom tags. For example every piece of data that a skin can show has a unique ID, and every tag is explicity listed and stored. If I could have a Foobar front end running on Android, controlling a Kodi on RPi backend then I would. Sigh.

Having said that I totally get what you would like to be able to do. I use my own genre system (as a kind of custom tag), and there is the comments, mood and style tags too... all can be seconded into personal use but not quite enough. I have it at the back of my mind, I can see the data structure that would work, UI side of things is harder, so a possible maybe Smile

Could this be simplified by allowing custom tags that are album or artist specific rather than at the item(song) level? Then the NFO could be used. Maybe not at the finite level the user is wanting and won't work for other music players but...
Reply
#34
Could it be like this...

Define what information Kodi will store, then at what level is most appropriate, then define whether it reads or scrapes this data as defined by the user? Obviously it's more complex by adapting to different tag formats. But still seems like manageable and more flexible.

Track (all read from the track)

-Genre
-Style
-Rating
-Mood
-Occasion

Artist (NFO or Online, user selectable, use matching MB ID's in NFO's)

-Life span (Born, Died)
-Formed, Disbanded
-YearsActive
-Genres
-Moods
-Styles
-Instruments
-Biography
-Discography
-Thumb
-Fanart


Album (NFO or online user selectable)

-Year
-Compilation
-Moods
-Styles
-Themes
-Review (description)
-Thumb (cover)
-Label
-Rating
Reply
#35
(2015-11-29, 21:04)beesmyer Wrote:
(2015-11-25, 15:47)phool Wrote: I use Foobar2000 to tag, manage and listen to my music library on a regular basis. As such I have become accustomed to being able to manipulate and customise the presentation of my library based on various custom tags.
....
However, my understanding is that custom tags are not readable at this stage and thus I would not be able to display my library in this way.

Could this be simplified by allowing custom tags that are album or artist specific rather than at the item(song) level? Then the NFO could be used. Maybe not at the finite level the user is wanting and won't work for other music players but...

The use of the word "tag" for both meta-data intrinsic to music files, and for the indication of xml elements in a NFO file does make my head hurt!

I understand your thinking here, but unfortunately the hard coded nature of XML/NFO "tags" is fairly similar to the music file tags processed. And the unique idenification of every piece of data that a skin can see also applies. Customisation (at all levels and from both sources of meta-data) is not impossible, it just isn't easy from where we are.
Reply
#36
Personally I have never used nfo files for music but I think that it would very useful to complete artist / album metadata especially when dealing with local unknown artists that don't get recognized by any scraper. I have tagged all my music using id3 tags and use the default scrapers to add the content to the library.

One thing I tried but wasn't able to do is to add music files from an online source into the library. I tried to create strm files but they don't get added into the music library. According to the wiki there is no documented way to add an NFO for a single music file. Some music addons have a feature to export the whole music library into strm files ( e.g one of the google music addons). By generating an nfo file for every file it would then be possible to add these files into the music library.

I have used the same technique (.strm + nfo file) for a while to add youtube music videos to my library. There are also a lot of video addons which allow library integration that way. Would be great to be able to do the same for music.
Reply
#37
Have not much to add to this... just a +1 for adding to library strm files.

Maybe another thing i consider important if you look to music in a global way... To add the countries... not sure if the country of the label or the country of the artist... or maybe both... because if you look at music and genres and their trends or historical importance you are going to see that the labels, and the countries matter... alongside the years... so you just might want to browse your library based on that... and for some people even the producers... minding the countries, there is people who like world or ethnic music, so adding the countries option might be interesting.

(more or less like you have with the movies section, 100 best songs and stuff like that are not that relevant).
Reply
#38
(2015-12-18, 23:41)Millencolin007 Wrote: Personally I have never used nfo files for music but I think that it would very useful to complete artist / album metadata especially when dealing with local unknown artists that don't get recognized by any scraper. I have tagged all my music using id3 tags and use the default scrapers to add the content to the library.

One thing I tried but wasn't able to do is to add music files from an online source into the library. I tried to create strm files but they don't get added into the music library. According to the wiki there is no documented way to add an NFO for a single music file. Some music addons have a feature to export the whole music library into strm files ( e.g one of the google music addons). By generating an nfo file for every file it would then be possible to add these files into the music library.

I have used the same technique (.strm + nfo file) for a while to add youtube music videos to my library. There are also a lot of video addons which allow library integration that way. Would be great to be able to do the same for music.

I find it is easier for me to import xml file into the library, rather than use nfo files (at least for artists, it's kind of a wash for albums). I don't use a central MySQL library, each Kodi instance has its own SQLite database. Using my master Kodi I can get and edit the library musicdb.xml file and then import it to as many other Kodi databases as desired. the main shortfall is that import doesn't add the current assigned art, so if the music scan didn't pick it up you have to add it yourself. That's more of an issue with local art files.

The only other problem is when you export the library, you only get records from artists or albums that have a "last scraped" date in the database, so I have a SQL command I run on the database that adds the last scraped date when missing prior to exporting the library.

scott s.
.
Reply
#39
Hi Dave,

got a problem with a recent clean and re-build of my music library. I have artists now that are MB IDs. For instance I've got an old DJ Compilation - FANTAZIA - BRITISH ANTHEMS which is 2 disks mixed by Jeremy Healy & Allister Whitehead.

Now the album has gone through MB Picard and has the correct album Artist of 'Jeremy Healy & Allister Whitehead' but Kodi has assigned an artist name of 1d2bcb45-2c15-4482-bde2-bffac1221573/8d2477a6-03bd-4fc8-a4b5-ac22dc566841

Strange eh?

The only place I can find this in the tags for the songs is in MusicBrainz Release Artist Id.

So do the MD tags now override everything else even when album artist etc is correctly populated?

Any ideas how to fix it as it is not the only album, I have several others now after the re-build for albums like

Zeitgeist : New Directions (Also has Allister Whitehead Huh)
Conan the Barbarian (Several Basil Poledouris album art assigned to long alphanumeric artists?)
Lots of Hans Zimmer albums

The list goes on of incorrect artists in my library.

I've tried re-doing in MB Picard, removing the folders from the source, cleaning the library, and then replacing folders and updating library but still the same?

The new re-build seems to have made a right mess of my library.

Grateful for any help that someone can give to resolving this new issue.
SKIN: Aeon Madnox
RIP Schimi2k, we miss you.
When I grow up I want to be a skilled skinner
Thank me by ⬇ adding to my + reputation
Reply
#40
(2016-04-28, 21:15)Mike_Doc Wrote: Now the album has gone through MB Picard and has the correct album Artist of 'Jeremy Healy & Allister Whitehead' but Kodi has assigned an artist name of 1d2bcb45-2c15-4482-bde2-bffac1221573/8d2477a6-03bd-4fc8-a4b5-ac22dc566841

Strange eh?

Not really Smile
This is a scanning issue, not a scraping or NFO one, and I might ask the forum mods to make a separate thread for us.

Unfortunately Jarvis, you don't say so I'm guessing that you are using 16, still can not handle some of the default ways that Picard tags music files. I am hoping to fix this in Krypton, make it totally Picard compatible and flexible, meanwhile you will need to do some tag tweeking if you want to avoid the outcome you currently have.

First let me explain what has happened, just in case it helps someone else. You have the following tags:
Album Artist = Jeremy Healy & Allister Whitehead
Musicbrainz Album Artist Id = 1d2bcb45-2c15-4482-bde2-bffac1221573/8d2477a6-03bd-4fc8-a4b5-ac22dc566841

But no ALBUMARTISTS (note the s) tag. Picard is going to implement this (AFAIK), but may not have done so yet. Kodi reads 2 mbids, easy to parse because they have a standard format, but only one artist name. Some times "&" is part of a single artist name, sometimes it is joining two names, Kodi can not tell which. Mismatching count of album artist and mbids can result in the mbid being used as the artist name, or other confusion because Kodi in some cases tries to use the artist name instead.

The solution (for Jarvis) is to add ALBUMARTISTS tags to the songs of these albums that identify the individual album artists when there is more than one. This can be separate tags if using ID3 v2.4 or FLAC/Vorbis or a single tag using the Kodi default item separator e.g. ALBUMARTISTS = Jeremy Healy / Allister Whitehead if v2.3

An alternative solution is to change the ALBUMARTIST tag to use the Kodi item separator e.g."Jeremy Healy / Allister Whitehead", but this will be what you see listed and the "&" may be important to you.

I am reasonably sure that where you are seeing mbids as artist names it will always be down to a mis-match in the number of names to mbids, and no ARTISTS or ALBUMARTISTS tags. Remember that Jarvis can only split names using the item separator (space slash space by default).

Sorry that you feel Kodi has made a mess of the library. In the longer term correctly identifying individual arrtists and using mbids will make for better music browsing and selection (and accurate scraping of additional arrtist data. artwork etc.), but unfortunately the tagging process still needs some manual intervention. I could do with someone updating the wiki to explain this but don't have the time myself.
Reply
#41
(2016-04-29, 09:07)DaveBlake Wrote: But no ALBUMARTISTS (note the s) tag. Picard is going to implement this (AFAIK), but may not have done so yet. Kodi reads 2 mbids, easy to parse because they have a standard format, but only one artist name. Some times "&" is part of a single artist name, sometimes it is joining two names, Kodi can not tell which. Mismatching count of album artist and mbids can result in the mbid being used as the artist name, or other confusion because Kodi in some cases tries to use the artist name instead.

On a test with the latest development versions of Picard I can't find any ALBUMARTISTS tag as far as I can tell. It does default to ID3v2 (2.4). I do see the ARTISTS tag though.

Picard (latest nightly)
Image

MP3Tag (v2.77)
Image

Free source album https://mega.nz/#!fNFSiJxa!ePxcgzjVYy6fx...Us7U5myurE

It would be awesome to get Kodi totally picard compliant, they seem to have worked out all the tagging cases now so its just a matter of scanning them and displaying in Kodi properly.

I'll get updating my picard guide as well.
Reply
#42
Well Kodi 16 will use an ALBUMARTISTS tag if you add it manually.... Picard will let you create tags that aren't there, so will other tags tools like Mp3tag.

Zag do you have any contacts in Musicbrainz that you can ask about ALBUMARTISTS? It is a counterpart to the ARTISTS tag, doing the same thing as that does but for album artists. I really thought it was going to be added, but that is just from reading older threads here. It makes perfect sense to do so, or maybe there is a Picard script for it?

Updating the wiki would be great. Will need updating again of course for Krypton Smile
Reply
#43
(2016-04-29, 11:31)zag Wrote: On a test with the latest development versions of Picard I can't find any ALBUMARTISTS tag as far as I can tell. It does default to ID3v2 (2.4). I do see the ARTISTS tag though.

You have to tell Picard to add that tag in the tagger script.

Code:
$setmulti(albumartists,%_albumartists%)

Once done it'll use it for any multiple artists.

Image
Learning Linux the hard way !!
Reply
#44
I spoke with the developer "zas" from metabrainz, he pointed me to this ticket and PR

http://tickets.musicbrainz.org/browse/PICARD-700

and

https://github.com/metabrainz/picard/pull/268

He said make a comment there about the default tagging, maybe it will be accepted but apparently its not supported enough yet.

At the moment, its only possible via the above script.
Reply
#45
(2016-04-29, 09:07)DaveBlake Wrote:
(2016-04-28, 21:15)Mike_Doc Wrote: Now the album has gone through MB Picard and has the correct album Artist of 'Jeremy Healy & Allister Whitehead' but Kodi has assigned an artist name of 1d2bcb45-2c15-4482-bde2-bffac1221573/8d2477a6-03bd-4fc8-a4b5-ac22dc566841

Strange eh?

Not really Smile
This is a scanning issue, not a scraping or NFO one, and I might ask the forum mods to make a separate thread for us.

Unfortunately Jarvis, you don't say so I'm guessing that you are using 16, still can not handle some of the default ways that Picard tags music files. I am hoping to fix this in Krypton, make it totally Picard compatible and flexible, meanwhile you will need to do some tag tweeking if you want to avoid the outcome you currently have.

First let me explain what has happened, just in case it helps someone else. You have the following tags:
Album Artist = Jeremy Healy & Allister Whitehead
Musicbrainz Album Artist Id = 1d2bcb45-2c15-4482-bde2-bffac1221573/8d2477a6-03bd-4fc8-a4b5-ac22dc566841

But no ALBUMARTISTS (note the s) tag. Picard is going to implement this (AFAIK), but may not have done so yet. Kodi reads 2 mbids, easy to parse because they have a standard format, but only one artist name. Some times "&" is part of a single artist name, sometimes it is joining two names, Kodi can not tell which. Mismatching count of album artist and mbids can result in the mbid being used as the artist name, or other confusion because Kodi in some cases tries to use the artist name instead.

The solution (for Jarvis) is to add ALBUMARTISTS tags to the songs of these albums that identify the individual album artists when there is more than one. This can be separate tags if using ID3 v2.4 or FLAC/Vorbis or a single tag using the Kodi default item separator e.g. ALBUMARTISTS = Jeremy Healy / Allister Whitehead if v2.3

An alternative solution is to change the ALBUMARTIST tag to use the Kodi item separator e.g."Jeremy Healy / Allister Whitehead", but this will be what you see listed and the "&" may be important to you.

I am reasonably sure that where you are seeing mbids as artist names it will always be down to a mis-match in the number of names to mbids, and no ARTISTS or ALBUMARTISTS tags. Remember that Jarvis can only split names using the item separator (space slash space by default).

Sorry that you feel Kodi has made a mess of the library. In the longer term correctly identifying individual arrtists and using mbids will make for better music browsing and selection (and accurate scraping of additional arrtist data. artwork etc.), but unfortunately the tagging process still needs some manual intervention. I could do with someone updating the wiki to explain this but don't have the time myself.

Hi Dave,

Thank you for the concise and detail reply, seems you are post on mate. I don't have an ALUBMARITSTS, just Album Artist and Album Artist Sort Order.

And apologies if it came across that I was mad at Kodi, as you know I am one of few who heavily uses Kodi for music and as I had to re-scan due to another strange reason it was just a surprise more than anything to see these. I'm thankfully that Kodi does what it does now with Music, improvements are always good though.

I always have library problems be it scanning or whatever, take for instance anything by Satoshi Tomiie all tagged via MBP with artist as Satoshi Tommie but in Kodi the artist it null which I believe is because Madnox cannot handle Chinese / Japanese chars which I believe the songs have been loaded with. But that is going off track slightly.

I guess I'll have to bite the bullet and start planning to update Madnox to Krypton compiliant, scares me the amount of changes to deal with, and then hopefully I can take adavantage of your improvements in Krypton.

And thank you for continuing to improve Music in Kodi.


Just checked again after seeing one of zag's posts. I do have ARTISTS but it is defaulting to the artist of the track so I would guess that is down to how the album is in MBP, not what kodi is doing.
SKIN: Aeon Madnox
RIP Schimi2k, we miss you.
When I grow up I want to be a skilled skinner
Thank me by ⬇ adding to my + reputation
Reply

Logout Mark Read Team Forum Stats Members Help
Scraping music data0