[DaveBlake]

More "inverse support" (a dev getting input from the wider community) from your friendly but over worked music library dev.

While in the process of improving music scraping I'm finally getting around to looking at the export/import facility. I'm going to use this thread to collate comments about export made in other thread, but if anyone else out there would like to chip in then that is fine.

Preamble - where we are in Krypton
The prime purpose of export is to save the additional artist and album information and art work that has previously been scraped from online sources, so that it can be used by another Kodi set-up, or after a clean reinstall, without having to go online and fetch the information again or repeat manual choices. Hence currently only artist and albums that have been scraped get exported.

Exporting produces either a single xml file, or separate nfo files for each artist or album. For separate NFO files it relies on every scraped album or artist having a unique (folder is for that artist and only that artist) folder that Kodi can identify by looking at where the related music files are stored (not at folder names etc.). This is fine for albums (all the music files in a unique folder is an obvious way to store and album), but there are many situations where artists do not have such a unique folder, and then problems start.

Issues

• Artists without a unique folder (identifiable from the contents):
  • Collaboration albums. Those with multiple album artists e.g. classical music with Composer, conductor orchestra, or "Riding with the King" by BB King and Eric Clapton
    
  • Song artists that do not have their own albums e.g. appear only on Various Artist compilations, or a guest artists on songs on albums by other artists.
    
  • Musicians, or other music professionals, that contribute to songs e.g. Mixer, producer, composer etc. but are not the song or album artist
    
  Export to separate files can make a mess - either putting nfo files and art in odd places, or adding multiple (unrelated) artists to one artist.nfo (for an album artist that does have a folder) that then does not scan in correctly.
  
• No way to export (for backup or transfer) user entered song data e.g. song rating, last played date or play count. Needs to include singles in this too, not just album tracks.
  
• Import can not cope with artists with the same name (always in lib with mbids to make unique)
  
• Import does not set artist artwork, have to visit each artist manually in info dialog to select it.
  
• (OT)Automatically scanning for local artist art when music first added to library:
  • Only works for the first album artist of an album.
    
  • Looks in the parent folder of the album for a default image, and repeats for every album
• Single file export can result in folder.jpg and fanart.jpg wrongly written (e.g. in the /mp3/<genre>/ folder)
  

Todo
Basically solve the export/import of additional info for artists that do not have a unique folder - stop the mess and provide a way to input/edit/store the data locally in xml format.
Add facility to externally save the modified song data (no more loss of playcount when you drop or move a source, or do a clean install etc.)

Older threads/posts
https://forum.kodi.tv/showthread.php?tid...pid2535004
Multi-disc sets - https://forum.kodi.tv/showthread.php?tid...id=2509217
https://forum.kodi.tv/showthread.php?tid...id=2351319
Fanart for compilations - https://forum.kodi.tv/showthread.php?tid=266371
Music scraper returning wrong artist - https://forum.kodi.tv/showthread.php?tid...pid2281544
Art related - https://forum.kodi.tv/showthread.php?tid...pid2221020

Feature requests
https://forum.kodi.tv/showthread.php?tid=248328
https://forum.kodi.tv/showthread.php?tid=246408
https://forum.kodi.tv/showthread.php?tid=246303

[DaveBlake]

A useful older post from Scott is worth repeating here

1. Export should allow to export artists, albums, or both.

2. It would be nice if export would allow for artist-name.nfo and album-name.nfo, though I don't know how to handle illegal file name (maybe standard "makelegalffilename" function) . But the database should add based on nfo file contents, not file name.

3. Single file export for artists will include <art></art> element containing currently assigned thumb and fanart (at least for local image files) but is ignored on import.

In light of the fact that music uses file meta-tags as main data source, I would consider getting rid of nfo file reading for music. Instead just use the music library import function. User can turn off online scraping during update.

1) Allow export of only artist or album in both single or multi file exports is a great idea.

2) Not eager to get into filename parsing, but definitely Kodi should add based on nfo file contents, not file name. Using Musicbrainz ID for match testing would also be essential to deal with artists with the same name.

3) Yeap.

I'll add another idea Scott has mentioned elsewhere:

4) Option to export items that have not been scraped. It is a convenient way to create a template of what needs data for manual editing (yes some people want to do that).

5) Either each export creates new dated folder (not always musicdb.xml) like video, or filenaming is optional.

But question for everyone else: do we remove the automatic artist.nfo scraping facility (as part of "fetching online info"), and just use import?

[Karellen]

Hi DaveBlake,

That sounds great. I will be silently following your work. I am sure there will be some changes required to the wiki pages. I imagine it will not be an easy process, and it will be a while before we see any progress.

Some more easy access links for your list:
http://kodi.wiki/view/Import-Export_library
http://kodi.wiki/view/Music_artwork


---Edit---
Count me in if you need any testing

[DaveBlake]

@Karellen don't be silent, opinions welcome.

The design and code is pretty easy actually, it is the UI addition that is stumping me the most - how to enter the export parameters (separate/sinlge?, where? images?, overwrite?, artists, ablums, both? non-scraped too?). If it was not Kodi I would just add an option dialog, but generally new dialogs are not appreciated (more skinning, different from video lib use etc.). The easy version is an ugly series of yes/no dialogs, but that is too horrible. I'm up for suggestions from those with more Kodi UI/skins experience.

[wyrm]

@Karellen don't be silent, opinions welcome.

The design and code is pretty easy actually, it is the UI addition that is stumping me the most - how to enter the export parameters (separate/sinlge?, where? images?, overwrite?, artists, ablums, both? non-scraped too?). If it was not Kodi I would just add an option dialog, but generally new dialogs are not appreciated (more skinning, different from video lib use etc.). The easy version is an ugly series of yes/no dialogs, but that is too horrible. I'm up for suggestions from those with more Kodi UI/skins experience.

DaveBlake,

I guess as a skin writer I can jump in here. From memory its four yes/no dialogs to do the equivalent in the video library, which I have no problem with as I need to do this so seldom. Potentially how many yes/no dialogs are we talking about here? If its five or six, who cares, but if more, yes probably getting out of hand. In the case of more than six dialogs maybe consider co-opting something like the addons settings dialog and presenting each option as a toggle setting or settings spin control as appropriate. The other thing you could consider is to make some of the options defaults that you change in advanced settings file.

Just my 2c worth, adjusted for inflation.
Wyrm (AppTV)

[scott967]

Here is the problem I see: First, I think album nfo scraping and album xml import both are working OK. So it's down to the artists. I think we can agree there should only be at most 1 nfo file for any artist/contributor. The problem is, how does Kodi find that nfo? I don't see any easy way to do it, since an artist could be associated with many songs or albums. Maybe one approach is like the "set artwork": Provide a "get nfo" button on artist info dialog that pops up the file browser with "browse" option (with top browse entry "* Item Folder" as for art). (OT but for some reason, there is no "* Item Folder" entry for artist fanart, but there is everywhere else in music and video). Of course on multi file export you need to handle multiple nfos per item, that's where I think using artist name (cleaned up) would work, then the user has to "parse" the result (when selecting via the file browser assuming above).

Also it seems like video does single file export to a new dated folder, while music always writes to musicdb.xml. Could we maybe have a filenaming option or do it like video?

scott s.
.

[jjd-uk]

Based on your Slack conversation I would go for a dialog, maybe something like:



Obviously it would be helpful to reuse an existing dialog, either what's used for "Set content" or addon settings as suggest by wrym.

To clarify above:

Export... opens a dialog selection consisting of Albums / Artists / Albums and Artists
Export to... open a dialogs selection consisting of Separate files / Single file

[Mike_Doc]

Hi Dave,

glad to hear import / export getting looked into. A stated before I stopped using this option in Kodi back in Helix or Isengard. I found the .nfo being created for albums in my 'Various Artists' folder were just useless , they pulled in info from I don't know where into the .nfo

But I'll give it another go soon with the latest in Leia, after I've dropped and rebuilt my DB again to send it on so you can see how the music scraping changes are going

[DaveBlake]

I found the .nfo being created for albums in my 'Various Artists' folder were just useless , they pulled in info from I don't know where into the .nfo

Mike like others have said, exported album.nfo files seem to be OK, or at least I have not be able to reproduce any issue (and I always worry I haven't fixed what I can't reproduce in the first place). I can see how artist.nfo would be an issue, so I think you are saying that artist.nfo for the artists on compliation albums made a mess. But you say/remember album.nfo being the problem?

Always interested in how new versions of Kodi deal with your extensive music collection.

[DaveBlake]

@jjd_uk and @wrym thanks for UI input, I'll see what I can work out from looking at the code for CGUIDialogSettingsManualBase and how it is used for "set content" and addon settings dialogs. There is always more to learn about with Kodi.

Also it seems like video does single file export to a new dated folder, while music always writes to musicdb.xml. Could we maybe have a filenaming option or do it like video?

Good point Scott, added to my first post.

I've also just discovered that import (and I guess this thread is really about both operations) can't cope with artists that have the same name. Artists - name is not unique, and they don't always have unique folders. That is not "corner cases" as some of my colleagues would say, but a common situation in the real world!

Now I'm trying to do testing with nfo files I am hating having them scattered through my music folders.
So inconvenient to manage.

The cdArt addon adds a folder (in a nominated place?) for every artist it stores art for, and that could be an approach taken for artist.nfo files too. i.e. add folders for those that don't have one. But adding a folder to hold artist.nfo seems worse than adding <artist-name>.nfo, at least with art there can be multiple images (thumb, fanart, poster etc.). And what about those non-unique artist names?

Having many artists in an xml file is a attractive to me, just need to use more than name to identify who is who.

[DaveBlake]

But for some reason, there is no "* Item Folder" entry for artist fanart, but there is everywhere else in music and video.

Ah yes, that is because AddItemPathToFileBrowserSources() is missing from CGUIDialogMusicInfo::OnGetFanart(). An easily fixed bug

Low hanging fruit that I don't want to forget, so PR https://github.com/xbmc/xbmc/pull/12625 raised to fix for Leia.

[scott967]

Low hanging fruit that I don't want to forget, so PR https://github.com/xbmc/xbmc/pull/12625 raised to fix for Leia.

Thanks. It was sort of an annoyance when assigning local art files.

scott s.
.

[meowmoo]

The cdArt addon adds a folder (in a nominated place?) for every artist it stores art for, and that could be an approach taken for artist.nfo files too. i.e. add folders for those that don't have one. But adding a folder to hold artist.nfo seems worse than adding <artist-name>.nfo, at least with art there can be multiple images (thumb, fanart, poster etc.). And what about those non-unique artist names?

I like the idea, I have a folder on my NAS, where I store all the Music Art.
I use CdArt Manager, Skinhelper and Artist Slideshow, and all 3 addons are using that one folder. So if I could Point the Export Funktion to this Folder also, everything would be in one place. And i dont need to clutter my Music Folder with nfos and jpgs

Folders looks like

Code:

ArtistName
----------extrafanart(folder)
----------banner.jpg
----------fanart.jpg
----------folder.jpg (artist thumb)
----------logo.png

I have never used the Export function, but this could be used as a "backup" function, if there is a reason to wipe the DB, it should be much faster to rescan/import the Music than a "clean rescan" ?

I hope i find the time to install your latest build and do some tests

can some one post a example of a non-unique artist name, i jave 800+ artist in my DB but all of them are unique

But adding a folder to hold artist.nfo seems worse than adding <artist-name>.nfo

I would not care if it add a folder with the artist.nfo in my MusicArt Folder, there are 1000+ Folder allready, because my GF listen to radio alot, and Skinhelper/Artistslideshow fetch art for the "radio artist" witch is not in my DB also, but if we create that folder in my Music folder that i would not like to see

[Karellen]

can some one post a example of a non-unique artist name, i jave 800+ artist in my DB but all of them are unique

First name I picked, I came up with a non-unique artist on MusicBrainz. Example is U2. There are two entries with that exact name. One the well known band, the other a Japanese(?) artist. Funnily, MusicBrainz gives the well known band a lower match score than the relatively unknown artist.

https://musicbrainz.org/search?query=u2&...od=indexed

When I recently added an album at MusicBrainz, I recall there was one song on a Compilation Album and there were 5 different artists listed with the same name. Sorry, don't recall which one it was.

[DaveBlake]

can some one post a example of a non-unique artist name, i jave 800+ artist in my DB but all of them are unique

From my own collection I have two "John Williams", both famous
a) the classical guitar player (of "Cavatina" fame)
b) the composer/conductor of music from films such as Star Wars, Superman etc.

But Musicbrainz is full of them, for example there are 23 bands all called "Eclipse".
I guess the broader (and more obscure) your music collection the more likely it is to hit them.

I like the idea, I have a folder on my NAS, where I store all the Music Art.
I use CdArt Manager, Skinhelper and Artist Slideshow, and all 3 addons are using that one folder. So if I could Point the Export Funktion to this Folder also, everything would be in one place. And i dont need to clutter my Music Folder with nfos and jpgs

Folders looks like

Code:

ArtistName
----------extrafanart(folder)
----------banner.jpg
----------fanart.jpg
----------folder.jpg (artist thumb)
----------logo.png

I am warming to the idea of having separate artist folders in a nominated place, as a way to hold both nfo and various art, and thus avoiding cluttering the music folder with nfos and jpgs. Thanks for the input.

But just need to have a way to handle the non-unique artist names, and also names that don't make acceptable folder names. MusicbrainzID is unique, but does not make a human friendly folder name.

I have never used the Export function, but this could be used as a "backup" function, if there is a reason to wipe the DB, it should be much faster to rescan/import the Music than a "clean rescan" ?

To make a backup copy the MyMusicXX.db. Building the lib by scanning embedded tags is not too arduous so I don't see a need to convert the entire db contents to xml. A rescan of tags is often the fix for a broken DB.

But what a useable and reliable export/import facility can do is replace the time consuming process of fetching extra data and art from online sources. As load on these great free resources grows access seems only likely to become slower or more limited, or resources simply collapse. So Kodi needs to be a "good neighbour", scrape once and only repeat for new music, or fetching new/changd info.