20 Nov 2019 - TVDB Scraper v3.2.0 is now available which reinstates scraping. TVDB are still in the process of fixing a number of bugs so there may still be further breakages. See this post. 2901570 (post)

  •   
  • 1
  • 3
  • 4
  • 5(current)
  • 6
  • 7
  • 9
  •   
Export/Import and local artist art improvements
#61
Great let's try some liasing here Marcel Smile
(2017-09-24, 21:32)marcelveldt Wrote: Well, maybe first start this way, I'll review the code you're working on to get some clue of what's going on and the todolist in the forum post.
My current work in progress is broadly to improve the music library export facility. That also includes the new concept (to core Kodi) of an artist info folder - a single location separate from music files where artist art and nfo files can be held.

There is a recent test build on the mirrors f909beb-XMLExport (for win32), other platforms found accordingly.
The relevent branch of my repo is XMLExport, it has many commits which I will squish before raising a PR. See what you make of it.

(2017-09-24, 21:32)marcelveldt Wrote: Yes, ready to get into some C++ but I'll start with the python scrapers, see what needs improving so I have a goal of what to contribute to the core code. I know for example the artwork and json api for music is very minimal compared to videos so I'd properly start there.
Right, my work on a new Python album scraper, based on top what ronie did is here https://github.com/DaveTBlake/metadata.i...tendAlbums It is a mess (my first Python), but perhaps you can see where I was going with what @ronie did. I know what it needs to do but didn't get there yet.

As for extra art as it means changes to the code that I am currently working on it could be easier to add it once this is done. But we can discuss.

(2017-09-24, 21:32)marcelveldt Wrote:
(2017-09-22, 08:54)DaveBlake Wrote: I have done nothing to make the newer data (e.g. Musicbrainz release group id) available to addons. The JSON API needs updating, and whatever route addons use. I would like some input on that.
Sounds like a great project for me ;-)
Could well be :p
How does SHS, or any other addon, currently get Musicbrainz release (album) id form the music db to request online art or does it just use name?

The nominated folder for artist nfo and art...
(2017-09-24, 21:32)marcelveldt Wrote: Yes, this is exactly the part in SHS that drove me (almost) mad haha.
I thought that you would have some useful experiences to share.
(2017-09-24, 21:32)marcelveldt Wrote: In the end it turned out that 50% of the users just want their artwork stored in their media folders (including me) while others want a different folder structure to store their artwork/metadata.
...

If it were me, I would make sure that artwork is by default stored in the media folders, this is also very consistent with the other media types.
Besides that, create some settings in the scraper setup for the more advance power users like a custom folder where artwork should be stored and which artwork/metadata to collect.
Yes, albums can easily have a unique folder containing all the songs and only the songs of that album, and putting other material there - nfo and art - is fine.

But artists are not like that. And while users maybe familiar with having artist art somewhere mixed in with their music files it is a flawed approach. The same would be true for actors in video library. Kodi currently tries to find the artist and nfo for an artist from where the music files are for albums by that artist, and I bet most users have an example in their library of that failing.

Logically all music does not all fit nicely into a
my_music / <album artist name> / <album name> / {song files}
structure.
  1. Artists can appear on multiple sources. I want to start encouraging more users to have multiple music sources, it is a great way to organise larger music collectons e.g. folder for my music, another for the kids, or separating mp3 from FLAC, the stuff you don't play much but have for parties, classical from pop etc.
  2. Collaborations (more than one album artist) don't fit either. These are common in classical music that have compser, conductor and orchestra as album artists, but also occur in pop e.g. "Rising Sand" by Alison Krauss & Robert Plant, or "Riding with the King" by Eric Claptoon and BB King.
  3. Artists that don't have albums, just feature as guest artists on albums by others or on various artist compilations.
  4. Other artists that contribute to the music in some way such as producers, mixers, composers, lyricists, musicians etc., people involved in the music but not the song or album artist.
If you have only one music source, and a music/albumartist/album folder layout then yes you could add a folder for every scraped artist to that. With 4) that could be a lot of folders. For 2) where are the collaboration albums located - first album artist folder or some other named folder? And of course what if the artist folder name used does not match exactly the artist name?

For me the best way forwards is a nominated artist folder. If the user sets this to the same as a music source then they get to live within the limitations and manage the consequences. This may be what 50% of users choose to do, but the more generic design is more robust and flexible.

But obviously there is the need for backwards compatibility - users that have carefully put their art where we told them to and used a suitable folder layout will want it to work in v18. Hence the process will be look for art in the artist info folder. If that does not exist then try to identify a unique folder that contains music by that artist and only that artist (a new check that will prevent some of the funny results users get) and look for art there. However Kodi will not export art to that location (unless that is where the nominated artist info folder is set).

Putting the artist folder setting in the scraper is an interesting idea, but then Kodi can have many different scrapers and it could lead back to having repeats of the same data in many places. So I have added it (in the wip) as a main music setting.

What does need some improvement (in my wip) is how to make setting it more user friendly, especially as it will be initlally blank for both clean installs and upgrades alike. [More on that in a later post where I deal with jjd_uk feedback on UI]

(2017-09-24, 21:32)marcelveldt Wrote: Imo all this base logic should be in Kodi core and/or the official scrapers and the addons (like SHS and such) are either obsolete or only cover some edge cases that are not needed to target main audience. In anyway I want to get rid of the 100's of python processes running to just provide a part of the puzzle.
Sounds good, but may take a while to get there.

In the immediate future I hope we can at least get any addon that fetches additional art (or info) to minimise requests to MB by using the stored MB release group ids, and to put the images it fetches in a common location defined by Kodi settings. What would SHS currently need from Kodi core to do that?

A specific question about your experience with making folder names from artist. I notice that you use unicodedata.normalize('NFKD' to convert unicode artist or album names before creating a folder name with it. There isn't a standard C++ equivalent, and so although I could do some translation of unicode I wonder if it is necessary or worth it. Your thoughts, or any examples of problems if you don't "sanitize" or "normalise"
Reply
#62
(2017-09-09, 22:32)Karellen Wrote:
Quote:
Karellen Wrote:Wondering if the "OK" button can be renamed to something like "Start" or "Export Now".
Not sure what the UI philosophy is about that. Following other dialogs here OK = do it, cancel = give up. I await more UI feedback from the team for this.

You will have noticed that under the guise of improving export, I have also addressed the art and nfo only for compliant album artists issue too. Smile
This is more for us dummies. I called up the settings dialogue page. I am entering my backup parameters. I click OK and expect that this will save my settings and take me to the next screen that says something like "export now? Yes/No". Just feel it is a bit more user friendly if I am told pressing this button will start the export instead of thinking this is saving my settings.

I'm in agreement with this, now it's not clear if ok is to save the settings or to start the export, so there's two options as I see it.

1. Change the OK button on right hand side to Export

Image

2. Alternatively after OK is selected a confirmation dialog is popped up

Do you wish to start export?
Yes - No
Reply
#63
When the artist folder is not set you pop up

Image

Add a button to take the user to the setting in a similar manner to when you try to install an addon from zip but unknown sources is not set

Image

Image

Also while on subject of Artist information folder I guess there a good reason for it being outside the new export dialog, is it now or in future plans going to be used for import or something else as well?
Reply
#64
For the kind of export you have

Image

It's not immediately obvious what is meant by Separate files per entry, as in what entries are these?

Since what is exported in the Separate files is dependant on what is set in Item to export

Image

Perhaps language should be the same to establish linkage so change Separate files per entry to:

Separate files for each item

Furthermore to re-enforce this linkage I'd change of settings to:

Code:
Items to export
==> Choose kind of export
Destination folder
Include items that have not been scraped
Include artwork such as thumbnails and fanart
Output only artwork, not NFO
Overwrite existing files

So Choose kind of export is a child setting of Items to export
Reply
#65
In Items to export it's not clear to me what would fall under Other artists

Image
Reply
#66
Re: Separate files per entry

What is being created is a nfo file. We refer to them as nfo files. They are described in the wiki as nfo files. Most users would be more familiar with the term nfo's. I think it would be best to continue the use of that term.

Maybe the options could be renamed something like...

Separate files per entry = NFO Files to Music folder

To Library folders = NFO Files to Artist Information Folder


This standardises nomenclature, especially for NFO. It also effectively links the use of the term "Artist Information Folder" to the same option in the Settings page.
My Signature
Links to : Forum rules (wiki) | Banned_Add-ons (wiki) | Debug Log (wiki)  
Links to : HOW-TO:Create_Music_Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#67
(2017-09-26, 12:07)Karellen Wrote: To Library folders = NFO Files to Artist Information Folder

While it does do that, it also still creates the album.nfo's in the relevant album folders so may mislead people into thinking it only does the Artist.nfo's. [edit - as I recall from memory]
Reply
#68
Yes, it does still send album.nfo to the album folders where the music files are held.

Open to suggestions for a non-misleading description, but I do feel that it should include the term NFO, whatever the description is.

But trying to describe "Artist nfo files goes to the Artist Information Folder as set in the Music Settings page, and the Album nfo file is still saved in the Album folder with the music file" in six words or less is an impossible task.
My Signature
Links to : Forum rules (wiki) | Banned_Add-ons (wiki) | Debug Log (wiki)  
Links to : HOW-TO:Create_Music_Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#69
Best I do can is:

Separate files per entry -> NFO files in music source
To Library folders -> NFO files in music source & Artist Information Folder

Maybe still a bit long winded I think.
Reply
#70
Thanks for the comprehensive UI feedback @jjd_uk, I had better reply while the forum is still up.

1) OK button to "Export", or a confirmation dialog
Personally it is pretty clear that a dialog with a heading "Export Music library" is going to do something when it is clicked. Hitting users with confimation requests all the time is just bad design, the action can be cancelled from the progress dialog anyway, and it is just going to export not reconfigure their life!

But we are in the settings section so maybe people get could confused. Two of you agree so I will change to use a custom button rather than the Ok button (since the CGUIDialogSettingsManualBase dialog does allow that).

I do wonder if library management (clean, export, import) should not burried under settings>media>library at all?

2) Navigation to Artist information folder setting
Good idea, @Karellen made same, I will see if I can implement. The addon example is slightly different, but a good place for me to start.
Quote:Also while on subject of Artist information folder I guess there a good reason for it being outside the new export dialog, is it now or in future plans going to be used for import or something else as well?
It will be the place that scraping artists looks for local nfo files or art, and that other addons can add additional artist art. See my previous posts.

I am going to try and give the user the option to set this when they add their first music source too. That would be the best time to ensure it is set, not just when exporting.

3) Meaning of "Separate files per entry"
A hangover from before, I will replace with "Separate files for each item".

4) Make "Choose kind of export" a child setting of "Items to export"
Sorry no. The export options, including the items to export, depend upon the kind of export. I am about to add the option to export song playback data (playcount, lastplayed and rating) when kind is single file. This dynamic data, currently lost when user drops a music source or does a clean install, does not belong in separate nfo files (they hold static artist and album data) but can usefully be exported on mass for subsequent import.

5) What are "Other artists"
"Other artists" = those artists (individual people, or groups of individuals like an orchestra, choir or DJ team), involved in the music in some way but not credited with being album or song artists. For example musicians, producer, lyricist, composer, DJ mixer, engineer etc. (even the person that designed the album cover, or made the tea if you have tagged it). Because of the introduction of an artist info folder Kodi will be able to scrape artist.nfo for these artists too. In v17 they can already be scraped online, just not exported or scraped locally.

Elsewhere in Kodi (name of a node that no one bothers to look at) and older forum posts I have referred to these as "contributors" but I'm not sure that is any better.

It needs to be succinct as all the chosen items appear in a row on the dialog. Can we not just explain in the wiki help, after all this is a more advanced feature.

6) Using "NFO" in the kind of export
When exporting to separate files it creates both NFO and image files, not just NFO. Also there is an option with this kind of export to only export images and not NFO at all. So I do not think it should say "nfo" specifically.

We can not explain in just a few words so don't, use the nice wiki. Smile

~~~~~~~~~

Thanks for testing and feeback, it is appreciated even when I don't agree.
Reply
#71
(2017-09-25, 13:25)DaveBlake Wrote: A specific question about your experience with making folder names from artist. I notice that you use unicodedata.normalize('NFKD' to convert unicode artist or album names before creating a folder name with it. There isn't a standard C++ equivalent, and so although I could do some translation of unicode I wonder if it is necessary or worth it. Your thoughts, or any examples of problems if you don't "sanitize" or "normalise"

IIUC if you use NFKD on Korean Hangul it decomposes each syllable into the corresponding Jamo? That can be helpful for doing searches (find all artists whose strArtist starts with "ㄱ" would find Kim - "김") but it makes for kind of unreadable folder/file names. Don't know how users of accented characters deal with it. Also not too sure on the effect it has on Japanese half-width and full-width forms. I think it might create illegal characters for some file systems.

scott s.
.
maintainer of skin  Aeon MQ5 mods for post-Gotham Kodi releases:
Krypton
Leia
Reply
#72
(2017-09-26, 23:09)scott967 Wrote: IIUC if you use NFKD on Korean Hangul it decomposes each syllable into the corresponding Jamo? That can be helpful for doing searches (find all artists whose strArtist starts with "ㄱ" would find Kim - "김") but it makes for kind of unreadable folder/file names. Don't know how users of accented characters deal with it. Also not too sure on the effect it has on Japanese half-width and full-width forms. I think it might create illegal characters for some file systems.
Thanks Scott, so sounds like I continue as I am - removing the reserved characters e.g. "/", but leaving unicode intact. Languages and international setting support is not my strong point, but I have tried some testing and that seems to be fine.
Reply
#73
I think maybe there is an advantage with the music library, in that users have control over their tags so they can do their own normalization. There's still potential conflict with scrape sites, but the use of MBID identifiers reduces the potential impact.

scott s.
.
maintainer of skin  Aeon MQ5 mods for post-Gotham Kodi releases:
Krypton
Leia
Reply
#74
Also metadata sites can usually auto substitute "é" for "e" in their searches, for example.
Image Image
Reply
#75
Now the foum is back....
Here is a screen shot of the latest version oif the Music Library Export dialog


Image

Marcel, any time to respond to 2651226 (post)

EDIT: now raised PR of this work 12891 (PR) plodding slowly forwqards
Reply
  •   
  • 1
  • 3
  • 4
  • 5(current)
  • 6
  • 7
  • 9
  •   
 
Thread Rating:
  • 0 Vote(s) - 0 Average



Logout Mark Read Team Forum Stats Members Help
Export/Import and local artist art improvements00