• 1
  • 2
  • 3(current)
  • 4
  • 5
  • 9
Export/Import and local artist art improvements
#31
Thnking aloud again....

Fetching artist and album art from local files, is it
a) part of scanning (looking at tags embedded in music files and building the library)
b) part of scraping (fetching additional artist and album data from online sources or NFO files)

Now b) when done online provides lists of available remote art, and if you don't have any local art it can use the first of these. That makes sorting out art part of scraping. But if it is local it can have a connection with the music files (also local).

Go on, someone say it, do both. And if you are scan + scrape e.g. "fetch online info on update" enabled, then the question is irrelevent.

It was the "Put it here and it'll then show up" comment that got me thinking. I know it gets well frustrating as a user to change art and have Kodi stubbornly ignore it. Dropping source and re-scanning is a palava if you have just added a few images, but enough of them that manually setting them is tedious. So for new art to show up either "Update Library" will need to look for new images as well as changed music files, or "Query info for all" (user initiated scraping) needs to look at local art too.

Of course "import" needs to allow the user to have dictated in xml what art to use too. And "export" needs to save the current decisions, and optionally artwork

I think it is fair for local art to be given priority, taken as the default when it is found. You can manually chose something else (from the list of remote art) and Kodi will remember, but automatically Kodi will use local if it fines it.
Reply
#32
Dave,

As someone who has painstakingly obtained artist thumbs and fanart for several hundred artists so far, and who always embeds album cover in the music files themselves, I agree with you about having local art take precedence, if present. And yes, for me it would be both a) and b), since for albums, it would be a) for me, and for artists, it would be b) for me.

Regards,

Bart
Reply
#33
Exporting art work.

Currently art is only optionally exported (thumbs saved to folder.jpg) when the export destination is separate files, i.e. NFO files are being created as well.
  • Should exporting art also be an option when export is to a single XML file?
  • Should other art be exported too e.g. artist fan art to fanart.jpg, or other art added by addons such as cdArt?
  • Should there be an option to put album art someone other than the folder containing the first song file?
Reply
#34
- No I don't think so
- Yes fanart should be exported as its a standard kodi artwork. I don't think the extra artwork like cdArts should be supported until Kodi supports them natively in scrapers(although I did see some work on this recently on GitHub)
- No that I think
Reply
#35
(2017-08-16, 13:00)docwra Wrote: - No I don't think so
- Yes fanart should be exported as its a standard kodi artwork. I don't think the extra artwork like cdArts should be supported until Kodi supports them natively in scrapers(although I did see some work on this recently on GitHub)
- No that I think

Agree on all points.
Reply
#36
So that's 2 of you, interesting.

My view is that xml + art could be a useful option, a way to get art out wihout splattering album.nfo files amongst the music.

Also if we do all art, not just limited types, then some addon functionality gets replaced completely rather than partially. It will come via scraper changes too, so export may as well be ready.

The final idea was a way to grab the thumbnails e.g. art that has been reduced from it original resolution, without risk of overwritting the original.

I'm not unhappy that you seem to see it opposite to me, or actually want something simpler, but thought it worth sharing my thoughts in a little more detail.
Reply
#37
(2017-08-18, 15:44)bsoriano Wrote:
(2017-08-16, 13:00)docwra Wrote: - No I don't think so
- Yes fanart should be exported as its a standard kodi artwork. I don't think the extra artwork like cdArts should be supported until Kodi supports them natively in scrapers(although I did see some work on this recently on GitHub)
- No that I think

Agree on all points.

+1 in all points too!
Reply
#38
OK guys, anyone up for trying out my prototype improved music export? Smile

A pre-PR test build KodiSetup-20170908-32e443c3cd-XMLExport is available on the mirrors. (That is the win64 link but other platforms are similarly named)

Please use a fresh build in portable mode, and give it some music from a test location too in the begining, just to become familiar with what it does. Take it easy please, I don't want tears before bedtime, but the worst case (depending on the options you pick) is it will overwrite your existing nfo files.

Use a simple skin e.g. Estuary or Confluence, but definitely one that doesn't use the (beta?) Skin Helper Service or you may get confused.

Disable cdArt addon, for similar reasons.

There is a new setting - Setting > media > Music > artists folder. Please set this somewhere away from your music files. It is where the artist nfo and art will go, for song artists and other musicians as well as album artists.

Load some music files, scrape at least some artists and albums
Then look at export from Settings > media > Library > Music section.

This is not the completed product, but it is close, so looking forwards to questions and feedback.
Reply
#39
Yes, I will test after I finish the homework you set me. Wink
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#40
Took a quick look at first thing is, on the mediasettings when I select export, I get libexportsettings and the the settings available are:

Choose kind of export
Destination folder
Items to export
Include items that have not been scraped

Default export is "single file"

On dialogselect for kind of export I get
single file (same as current)
separate files per entry (unclear -- this means use the artist folder defined in mediasettings?)
To library folders (this is old multi-file export?)

Test 1. Select single file albums only. Nothing happened (AFAICT). This is because I didn't have any destination folder set. There should be feedback to the user that the operation will fail due to no destination folder.

Test 2. Set a destination folder and then repeated. Export file was generated with a date stamp in filename (thanks). Settings were album only/include items not scraped. Export file included albums only (thanks) and included both scraped and unscraped albums (thanks).

Test 3. Same as test 2 but items to export: album artists. Again export file generated with a date/time stamp in filename (unclear why a different format is used; first filename is kodi_musicdb2017-09-08.xml, second one is kodi_musicdb20170908_130405.xml). Also no indication that export file first one is albums, second album artists (not major, just would be convenient). On this one none of the artists had been scraped; all were exported (thanks).

I will have to set up a new test source to try song and other artists. But so far, just having this single file export options feature is a big help.

scott s.
.
maintainer of skin  Aeon MQ5 mods for post-Gotham Kodi releases:
Matrix see: Aeon MQ5 Mod Matrix release thread
Nexus see: Aeon MQ5 Mod Nexus release thread
Aeon MQ 5 skin and addon repo 11.1.0
Reply
#41
Test setup

  1. Portable install. Download from link provided for Win 10
  2. Added my usual advancedsettings.xml file. Mainly video functions, artwork resolutions and silent debug mode. No other modification to fresh portable install.
  3. Added test library - 21 Artist albums, 12 Compilation albums (in subfolder), 4 Soundtrack albums (in subfolder). All albums tagged with MBPicard. Most album and artists have local artwork
  4. Set source and scanned media into library- 131 Artists, 38 Albums, 301 Songs (FLAC)
  5. Perform Query Info for all artists
  6. Perform Query info for all albums

First Test
  1. Set Artist information folder to outside of Music Library source
  2. Single File Export.
  3. Destination Folder: Should this be carried over from previous step? I see the previous step as a backup destination,
  4. Items to Export: Albums, Album artists.
  5. No to template
  6. Export ok. All Artists and albums exported.

Second Test
  1. Set Artist information folder to outside of Music Library source
  2. Single File Export.
  3. Destination Folder: Should this be carried over from previous step? I see the previous step as a backup destination,
  4. Items to Export: Albums, Album artists, Song artists, Other artists.
  5. No to template
  6. Export ok. All Artists and albums exported.

Third Test
  1. Separate Files per entry Export.
  2. Set Destination folder to outside of Music Library source
  3. Items to Export: Albums, Album artists, Song artists, Other artists.
  4. No to template
  5. No artwork
  6. No overwrite
  7. Export ok. All Artists and albums exported. Various Artists artist.nfo is full of random data, but that is a MBPicard tagging issue

Fourth Test
  1. Separate Files per entry Export.
  2. Set Destination folder to outside of Music Library source. Different directory from third test
  3. Items to Export: Albums, Album artists, Song artists, Other artists.
  4. No to template
  5. Yes artwork
  6. No overwrite
  7. Export ok. All Artists and albums exported. Various Artists artist.nfo is full of random data, but that is a MBPicard tagging issue

Fifth Test
  1. Export To library folders
  2. Items to Export: Albums, Album artists, Song artists, Other artists.
  3. No to template
  4. Yes artwork
  5. No overwrite
  6. Export not ok- No artist.nfo created
  7. Export not ok- Not all album.nfo created 10:49:53.659 T:9016 DEBUG: CMusicDatabase::ExportToXML - Not exporting album The Very Best Of as unique path not found

---Edit---
For the album.nfo issue- The album is 2cd Fleetwood Mac- The Very Best of. In my folder structure I have treated this as one release. All songs are saved in \Fleetwood Mac\Very Best Of
MusicBrainzPicard has treated each cd as a separate release, even though it is the same album release. When looking at Test 4, it has created two folders for this album The Very Best Of_b444 and The Very Best Of_f665. In Test 5, it has failed to save any album.nfo



General comments
  1. I see a conflict with file names for single file export and multiple exports on the same day. User might be testing, or incrementally building the library and export will overwrite previous
  2. New tag... <scrapedmbid>false</scrapedmbid>
  3. New tag... <musicbrainzreleasegroupid></musicbrainzreleasegroupid> Are there anymore?
  4. Noticed... <track></track> has been removed
  5. Wondering if the "OK" button can be renamed to something like "Start" or "Export Now".

I have one debug log capturing all these tests. If you want to view it, I will send.
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#42
Thanks @scott and @Karellen for such speedy testing and feedback, I greatly appreciate it. So good to have more eyes on my work while I complete it Smile
(2017-09-09, 01:11)scott967 Wrote: separate files per entry (unclear -- this means use the artist folder defined in mediasettings?)
To library folders (this is old multi-file export?)
The beginning of the explanation (I would like some help documenting what I created)....

Destination of output:
  • "Single file" - a new file in folder selected in "Destination Folder". Named with date (+ time subsequently)
  • "Separate files per entry" - a folder structure under folder selected in "Destination Folder".

    PHP Code:
    <Destination folder>
    ---
    ABBA (folder)
    ----------
    artist.nfo
    ----------fanart.jpg
    ----------folder.jpg (artist thumb)
    ----------
    Greatest Hits Vol 1 (folder)
    --------------------
    album.nfo
    --------------------folder.jpg (album thumb)
    ---
    John Williams_53b1(folder)
    ----------
    artist.nfo
    ----------fanart.jpg
    ----------folder.jpg (artist thumb)
    ----------
    Spanish Guitar Music (folder)
    --------------------
    album.nfo
    --------------------folder.jpg (album thumb)
    ----------
    Echoes of London  (folder)
    --------------------
    album.nfo
    --------------------folder.jpg (album thumb)

    ---
    John Williams_8b8a (folder)
    ----------
    artist.nfo
    ----------fanart.jpg
    ----------folder.jpg (artist thumb)
    ----------
    Pops In Space (folder) [an album by John Williams and The Boston Pops]
    --------------------
    album.nfo
    --------------------folder.jpg (album thumb
    • Artist folder name is derrived from artist name plus 4 digits of mbid (if name duplicates in library)
    • Album folder name is taken from the music folder containing the music files (if there is a unique one), otherwise from the album name.
    • Collaboration albums (more than 1 album artist) are placed under the first album artist.
    • All artists can have folders (not just album artists), but only those with albums get album subfolders.
    • How much of this structure gets created depends on what is exported.

  • "To library folders" - destinations are split. The album stuff goes with the music files (as before, but if you have mixed albums in a folder then it rather than overwrite as it did before it logs that album as can't find folder). The artists go to folders located in the new "artist information folder" setting found Settings > media > Music > artists information folder.

    Spliting makes sense to me (at least). Albums usually have folders in the music file structure, but with artists many don't, hence albums with music and artists somewhere else. That artist location could be your music source (if you have only one and have source/artist layout ), and new folders will mix in with your existing ones. But if you have all your U2 music under a folder called U2stuff, then Kodi will create U2 as a new folder (i.e. folder names need to match artist names). Persoanlly I hope users embrace the idea of a new location for artist stuff, it echos what some addons do and I hope we can integerate with those too so that other art e.g. extrafanart, logo.jpg and banner.jpg can go in the same place.

scott967 Wrote:There should be feedback to the user that the operation will fail due to no destination folder.
Noted. I'll add validation to the dialog.
Similarly for the "artist info folder" setting and export type "To library folders".

scott967 Wrote:..export file generated with a date/time stamp in filename (unclear why a different format is used; first filename is kodi_musicdb2017-09-08.xml, second one is kodi_musicdb20170908_130405.xml).
Indecision Smile
I prefer a simple filename (just date), but adding time is an easy way to ensure uniqueness. I used standard utilities to give filename with date or date and time, and they differ in format. So what would people prefer - always have date_time?

scott967 Wrote:Also no indication that export file first one is albums, second album artists (not major, just would be convenient).
Not sure how best to indicate the options used a) in the file name, b) in the xml, c) elsewhere? Happy to oblige if a good idea can be presented.

(2017-09-09, 03:00)Karellen Wrote: Set Artist information folder to outside of Music Library source...
Destination Folder: Should this be carried over from previous step? I see the previous step as a backup destination,
No, the "artist info folder" is the (future) primary location for artist art and NFO, rather than info for limited artists being mixed in with music. Of course for backward compatibility Kodi will fall back to looking for album artist stuff with the music if there isn't anything in the "artist info folder".

Backup is about taking a snapsot your data off somewhere, that is offered by export types "Single file", and "Separate files per entry" that go to the destination folder that you give on the dialog.

If I can present that im a more intuitive way for users then tell me.

Karellen Wrote:Fifth Test
  1. Export not ok- No artist.nfo created
  2. Export not ok- Not all album.nfo created 10:49:53.659 T:9016 DEBUG: CMusicDatabase::ExportToXML - Not exporting album The Very Best Of as unique path not found
The album is 2cd Fleetwood Mac- The Very Best of. In my folder structure I have treated this as one release. All songs are saved in \Fleetwood Mac\Very Best Of
MusicBrainzPicard has treated each cd as a separate release, even though it is the same album release. When looking at Test 4, it has created two folders for this album The Very Best Of_b444 and The Very Best Of_f665. In Test 5, it has failed to save any album.nfo
Interesting example.
If tagged as separate releases then Kodi will have made them separate albums, and the export and new "robust" (= does not let nfo for one album or artist apply to another album or artist) artist and album folder processing is in effect finding 2 albums in the same common folder.

It is a case where I think Kodi is right, and your tagging/folder layout conflicting. But other users are going to have done similar things, so I will try to see if I can do more with it.

Karellen Wrote:I see a conflict with file names for single file export and multiple exports on the same day. User might be testing, or incrementally building the library and export will overwrite previous
Not sure what conflict you are seeing? Overwrite of exitsing art and nfo is optional, and single file adds time to output if it fines an earier one. Can you expand on this?

Karellen Wrote:New tag... <scrapedmbid>false</scrapedmbid>, <musicbrainzreleasegroupid></musicbrainzreleasegroupid> Are there anymore?
Noticed... <track></track> has been removed
There will possibly be more before I finish with v18 Smile

Karellen Wrote:Wondering if the "OK" button can be renamed to something like "Start" or "Export Now".
Not sure what the UI philosophy is about that. Following other dialogs here OK = do it, cancel = give up. I await more UI feedback from the team for this.

You will have noticed that under the guise of improving export, I have also addressed the art and nfo only for compliant album artists issue too. Smile

I think this build has solved the hook-up of new art after import issue i.e. set art in nfo/xml and load (import/scrape), but have to visit each artist/album individually and refresh to actually see that new art. Perhaps confirm that if you know what I am talking about.

I also have a question over the consequences of unicode in artist names. At the moment it does not replace the unicode with ascii when making artist sub-folder names, although it does remove reserved chars, and I wonder if that is OK. On the otherhand replacing unicode can get messy, and many systems can cope with unicode folder names. @scott maybe stick some of your more exotic music into it and let me know.

But I'm still working on the pickup of stuff from the right places, and the export of song rating and playcount etc. Hence more test builds will be coming, so don't get too exhaustive just yet.
Reply
#43
Quote:
Karellen Wrote:Wondering if the "OK" button can be renamed to something like "Start" or "Export Now".
Not sure what the UI philosophy is about that. Following other dialogs here OK = do it, cancel = give up. I await more UI feedback from the team for this.

You will have noticed that under the guise of improving export, I have also addressed the art and nfo only for compliant album artists issue too. Smile
This is more for us dummies. I called up the settings dialogue page. I am entering my backup parameters. I click OK and expect that this will save my settings and take me to the next screen that says something like "export now? Yes/No". Just feel it is a bit more user friendly if I am told pressing this button will start the export instead of thinking this is saving my settings.

Quote:
Karellen Wrote:I see a conflict with file names for single file export and multiple exports on the same day. User might be testing, or incrementally building the library and export will overwrite previous
Not sure what conflict you are seeing? Overwrite of exitsing art and nfo is optional, and single file adds time to output if it fines an earier one. Can you expand on this?
See image... https://ibb.co/iTXy6v
On my install here, those options are greyed out and I could not access them. I also failed to mention that with the single file exports, no artwork was exported. Ok, wasn't aware that Kodi will modify the file name if it finds another export file there.

Quote:Export not ok- No artist.nfo created
This comment was meant to apply to all artists, not just the Fleetwood Mac example. No artist.nfo were created for any artist.
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#44
Test 4/5 was "separate files per entry, 4 was album artist and 5 was albums,album artists. They were extracted to the "destination folder". I did notice that when the album artist and album names were identical the album folder created had _nnnn appended (nnnn is from album mbid, but need to test what happens if I didn't have an album mbid). It doesn't seem like it is necessary to have the name collision test between artists and albums?

I notice on the album.nfo files the <track> data is no longer exported?

Update. Looking through the nfos and folders created, I do have one other album that the export folder name has the album mbid appended, and I don't see that there is any name collision at all between it and artists or songs.

scott s.
.
maintainer of skin  Aeon MQ5 mods for post-Gotham Kodi releases:
Matrix see: Aeon MQ5 Mod Matrix release thread
Nexus see: Aeon MQ5 Mod Nexus release thread
Aeon MQ 5 skin and addon repo 11.1.0
Reply
#45
@Karellen, "OK" => "Start", I'll canvas team opinion, especially the UI/skins guys.

Yes, options disabled for "single file" because they only apply to the other types of export i.e. single file uniquely named so not overwrite, and by design does not export art. Users can export the many art files (and no NFO optionally) in a "separate files" type export.

Karellen Wrote:
Quote:Export not ok- No artist.nfo created
This comment was meant to apply to all artists, not just the Fleetwood Mac example. No artist.nfo were created for any artist.
Ah, no artist.nfo created was a bug. I got briefly distracted by Fleetwood Mac Smile
Can you redo test 5 when I get next text build ready Smile
Reply
  • 1
  • 2
  • 3(current)
  • 4
  • 5
  • 9

Logout Mark Read Team Forum Stats Members Help
Export/Import and local artist art improvements0