[DaveBlake]

I am going to try to maintain this thread as a summary of my current work in progress on improving music library features for the Kodi v18 Leia release. Here you will find a basic road map of what is currently being developed along with my todo list. Any help others can offer - testing, discussing design, documenting or just being fun to hang out with and share my moans - is very welcome. If anyone has the skill, time and interest to set up an IDE and get involved that way then great.

At any time there may be a music test build available with my work so far that is ahead of the nightly.
Currently - none available
This is from my repo, and differs from Leia master, because of the delays in getting review and approveal to merge so it can appear in the nightlies. As all my work is focussed in one area it is often cumulative, so I can't raise later PRs until the first is merged without a lot of review confusion. Testing of my work before merge will help with the review/approveal process.

I will probably split any testing feedback or feature discussion posted here into a new thread, to stop this thread growing too big and becoming unreadable. Watch out for those when posting.

Work Completed (inc. waiting for review)

1. (P) Forced scanning of tags
   The need for a forced scan is now held in the db, so with client server setups you only get prompted once after an upgrade, not every client.  PR11557 This also added option (add <promptfulltagscan> to the musiclibrary section of advancedsettings.xml) to be prompted for a forced scan of tags when rescanning a source from context menu of fileview. Very useful when testing changes to tag processing but was later PR12891 replaced by a question folowing selection of "scan to library" on the context menu in file view.
   
2. (F) Cleanonupdate
   Fix implementation (to match video) as since Gotham this option was accidentally preventing any subsequent scraping additional album or artist information, and failed to clean the library when no files were being scanned for tags. https://github.com/xbmc/xbmc/pull/11609
   
3. (N) Tags: WRITER, BAND and ENSEMBLE
   Add processing of  WRITER tag, correct BAND and ENSEMBLE tags (Vorbis and APE formats). https://github.com/xbmc/xbmc/pull/11735
   
   
4. Replaygain reduction clipping protection option
   Restore option for replaygain reduction as clipping protection. See forum thread and https://github.com/xbmc/xbmc/pull/11865
   
   
5. (N) Artist sort names - Phase #1
   Add processing of ARTISTSORT (TSOP), ALBUMARTISTSORT (TS02) and COMPOSERSORT (TSOC) tags to provide a sort name for artists. Optionally use field this when sorting artists, albums or songs by artist name. https://github.com/xbmc/xbmc/pull/11738
   
   
6. (F) Duplicate Musicbrainz track ids within in an album
   It is fairly rare but Musicbrainz track ids are not always unique, tracks with the same id can be repeated e.g. silence or on multi-disc anthology albums. Kodi simply didn't load these. https://github.com/xbmc/xbmc/pull/12079
   
   
7. (F)Cuesheet regressions: replay gain and db design
   Refactor db implementation of cuesheets to avoid needlessly slowing the songs node (disabled for v17 anyway), and actually have replay gain applied for library items. Fix other oddities in playback of embedded cuesheet songs from file view.  https://github.com/xbmc/xbmc/pull/12082
   
   
8. (F)Album Type and Record Label merging
   Fix bug where scraped type and label values were always overwriting those derived from tags. When scraped value was empty this effectively cleared the tag value. Also backported to Krypton v17.4
   https://github.com/xbmc/xbmc/pull/12216
   
   
9. Better Album and Artist Additional Info Scraping From Online Sources - Phase #1
   • When fetching additional info on library update:
     • Scan all metadata tags from the music files first before scraping (rather than do it album at a time).
       • This populates the library with the basic music without delay so the user can start using it.
         
       • Tagging can be mixed quality - some with mbids, some without, so artist mbid may be obtained from music files later in the scan.
         
       • Avoids unwanted consequnces arising from the order in which folders are processed. The common example of this is the compilation/NFO issue where early processing of various artist compilations causes the artist folder to be misidentified and artist.nfo is either not found or scraping goes wrong. The work around was to put all compliations in a folder named "zzzzzCompilations", to ensure it was processed last, otherwise the song artists from these albums was added to the library and additional info scraped before the other albums by that artist had been scanned, and so the artist.nfo file was never found and applied. This is no longer required.
         
       • Having passed through all the music during scanning tags we know how many albums we are processing, so during scraping phases the progress bar can show meaningful position though task.
         
       • Repeat of failed scraping due to server timeouts could be added at a later time.
         
     • Scraping album artist(s) as part of library update, even if the album lookup has failed.
       Otherwise known artists can go unscraped just because the albums in the library are not found.
       
     • Avoid repeatedly requesting artists that are not in the online sources.
       
   • Scanning Musicbrainz release group id from tags as well as release id (album) for use by addons to fetch artwork
     
   • Saving scraped Musicbranz ids for artist/album (both release id and release group id) when not in tags, but need to ensure succesfully identified and/or provide a means to modify
     https://github.com/xbmc/xbmc/pull/12120
     
   • Fix throttling of Musicbrainz requests https://github.com/xbmc/xbmc/pull/12402 Significantly reducing the 503 errors and gaps left after scraper runs. Also backported to v17.4
     
   • Changes to Universal Album Scraper (v2.7.2) https://github.com/xbmc/repo-scrapers/pull/51 avoiding inefficient duplicated requests to Musicbrainz. @olympia has also fixed some additional parsing issues.
     
     
   • Fix Change Information Provider for albums and artists PR12597
     a) Setting artist or album info provider selectively that works!
     b) Facility to set info provider for multiple artists or albums
     c) Facility to set default info provider from within library, and to reset all artists or albums
     The old functionality was so broken I doubt it was used.  Note: library is bumped to MyMusic67.db
     
   •  
     
10. Music Export / NFO and Art handling for All Artists (Better Album and Artist Additional Info Scraping From Online Sources - Phase #2)
    See PR12891, also discussed in detail here https://forum.kodi.tv/showthread.php?tid=319284
    • Support for local art and nfo files for all artists via new Artist Information Folder and numerous fixes to better match art and nfo with correct artist
      
    • Export done asynchronously (not lock UI), with new dialog (rather than series of yes/no dialogs), offering export of all artists and allowing greater control of what exported to where.
      
      
    
    
11. Extra Artist Data (type, gender, disambiguation) PR12963. The music db version is bumped to 68.
    • GUI info label, media filter and smartplaylist support is added, and the data is included in import from and export it to xml and NFO files.
      
    • Add GUI info labels for artist sort name (PR11738) and album release groupd id data (PR12120).
      
    • Update JSON API to include all the above.
      
    
    
12. (N)New Information Provider Dialog PR12987.
    • Easy access to default music scraper (Info Provider) settings when adding music source via a "settings" button.
      
      
      
      
    • The same dialog is also used to provide the UI to change information provider for artists and albums (separately) from the artists or albums nodes. A follow on from PR12957
      
    
    
13. (F)Fix Album Genre Inconsistencies PR13042.
    Genre in library navigation is implicitly song genre.
    Makes album genre similar to artist genre (or album mood, style and theme) - a discriptive string that happens to have the name "genre" but can be completely different from the song genre values of related songs. [Note I had originally thought that both artist genre and album genres should be codified, but there simply is no need to associate ids with these strings, much like mood, style and theme, since we are not using them to navigate the library]
    
    
14. (F)Fix various JSON API issues
    • Breaking change bump to v9.0.0 of API. PR13051
      • GetAlbums() and GetAlbumDetails() - sort out genre use
        
      • SetAlbumDetails() and SetSongDetails() - get these to actually work!!!
        
    • (F)Fix wrong playback order for smartplaylists, playlists and upnp PR13059 Longstanding Trac ticket that had playlists started from JSON always in a set order.
      
    • Add "art" parameter to audio library Get and Set methods PR13101
      
    
    
15. Music library extended artwork support Epic discussion on the forum core framework to support any art type set via JSON API, addons or manually via UI.
    • Artwork consistently available to the GUI via Listitem.Art(...) and Container.Art(...)PR13352.
      
    • Refactor Song Information dialog PR13533.
      • Add "Choose Art" button to support selection of any type of art
        
        
      • Get song art changes to show immediately including on the OSD if currently playing song is changed
        
      • Move data read and write off the main thread to avoid possibility of locking UI.
        
    • Refactor Artist/Album Information dialog PR13672.
      • Add "Choose Art" button to support selection of any type of art
        
      • Get album or artist art changes to show immediately.
        
      • Update the artwork for the currently playing song, or songs on current playlist, if the related album or artist art changes.
        
      • Improve artist discography list to always show all the albums in the music library by that artist, and then any additional titles scraped from online sources.
        
      • Clear local images from cache prior to image selection in attempt to immediately show any recent external changes made to the contents of the file. See https://forum.kodi.tv/showthread.php?tid=330078
        
      • Add an "Artist info" button on the album info dialog that links to the primary artist
        
      • Replace the scraped track listing on album info dialog to show the songs in the library from the album. This change happened in DB v68 and deprecates the albumsonginfo table . See http://forum.kodi.tv/showthread.php?tid=298528 and PR12597.
        
      • Fixed issues with resfresh button after switching between artist and album info dialogs (same dialog so gots confused).
        
      • Move data read and write off the main thread to avoid possibility of locking UI.
        
    
    
16. Disc set local artwork support PR13769
    Some baby steps to improve disc set handling by correctly picking up any thumbs from subfolders when the album is spread across multiple subfolders (one per disc), as well as album (the boxed set) and artist art. See more details here. Note this does not handle round disc/CD art (round images), but cover (thumbs) that each disc in a boxed set could have.
    
    
17. (N)Prefer online album art PR13812
    Add the option to automatically use scraped art in preference to that found embedded in music files. See thread for details.
    
    
18. (N) Fetching any type of art automatically PR13848
    It makes fetching additional art, from local files or scraper results, configurable in a white list rather than hard coded. This means showing additional art, such as "clearlogo" or "discart" or anything else people may dream up in the future, will just be a matter of either putting the correctly named image file in an appropriate place, and/or the scraper returning a URL for it , and then editing the skin. See thread for details, discussion and feedback.
    
    
19. (N) Filtering By Media Source PR14012
    New facility to filter artists, albums and songs by media source, from either side blade filter or smartplaylist rules. It is very natural to categorise music by putting it under separate folders, and with the introduction of the Artist Information Folder, users can arrange their music files into whatever folders they like (there is no longer any need to try to have all the music by an artist, and only that artist, under one unique folder in order to have local art and artist.nfo files). By adding these folder as separate sources (via the media source dialog), items can then be filtered using them. See thread for details, discussion and feedback.
    
    
20. (N) JSON API Music Data Faster Access PR14160
    An internal reworking of the batch access to song, album and artist data via JSON API making it significantly faster.
    
    

------
Test Build: none available, use nightly

(N) = New feature, or major improvement
(P) = Partial, more phases to come later.
(F) = Fixing bug

In Progress (stuck on the backburner)

1. Tags: ORIGINALDATE(TDOR), DATE(TDRC) and RELEASEDATE (TDRL)
   Support for various dates for albums and songs rather than just year.  See discussion http://forum.kodi.tv/showthread.php?tid=313952
   How users can optionally see music by the date it was originally produced as well as the post 1980s release dates of all digital media, and support for month and date.
   
2. Smart Playlist Rules, Filtering and Default Nodes
   A unification of SQL Query building so that filtering and navigation from playlist results combines query logic correctly. Essential ground work before filtering and navigation options can be extended. First attempts are here PR10098, but this needs further work.
   Needs to fix navigation of results when have smart playlist within smart playlist, and navigation of artist search results when role <> 1.
   
3. Custom Properties
   Facility for songs, albums and artists to have user defined properties, either edited within Kodi, or scanned from custom tags embeded in music files or from NFO files
   
4. Music Scraping  From Online Sources
   While others are looking at converting to Python based scrapers, a review of how music is scraped, what data fetched etc., how it is merged and any other core changes to support the Python change.
   • Make scraping of genre data for artists and albums optional
     
   
   

Todo
This is the stuff that I feel I have a reasonable chance of dealing with as a single dev, or have a strong  interest in. I know there are a lot more requests for things people would like, and major changes that are beyond me, for my sanity they are not listed here.  But I am happy to collaborate or offer support where I can if anyone else wants to make improvements or fixes to music.

In no particular order....

1. Smartplaylist and Filter Rules
   • Track and Disc Number
     Fix track number smart playlist rule to correctly split just track number form the combined track and disc filed in song table. Add disc number as a smart playlist rule and filter option for songs and albums.
     
   • Filetimestamp
     Add filetimestamp (misnamed Dateadded field) as smartplaylist and filter rule for artists, albums and songs.
     
   • Recently Added (unscraped) Artists
     A way to list those recently added or not yet sucesfully scraped for re-scraping attempts.
     
2. Consistent Application of Node Options
   Apply folder type node options at genre or role level. Currently can create a node with say <path>musicdb:// genres/?ambumartistonly=true&year=1988</path> that will show all genres, but when navigate down for a selected genre the options are applied only see the album artists with songs from 1988, and there could be nothing to show.
   
   
3. Extended artwork support - Phase #2
   • Embedded Art
     • Fix way APIC tag is handled to correctly fetch front cover art when there is more than one image. Explore possibility to handle more than one image per song or album.
       
     • Include embedded in art browse/selection from album/song information dialog.
       
     • Option to use embedded art only as a last resort (or ignore?) for users that have messy art to be able to have first scraped album art apply by default
       
   • Local file cache refresh
     Add a "refresh" button on the file selection dialog when browsing for local art, so recent file content changes are visible immediately. See https://forum.kodi.tv/showthread.php?tid=330078
     
   
   
4. Genre Handling
   • Genre Separator - Allow for multiple item separators,  add ";" (a sensible separator value). Useful for ID3 v2.3 tags (single frame) as semicolon is the default separator used by meia Monkey when tagging.
     
   • Add support for a genre hierarchy, a way to group genre together as subgenres to unscramble the usual large mix of genres that result from scraping/tags
     
   • Clarify JSON API and smart playlist/filter rules where artist, album and song genre are being used.
     
5. Improve multidisc set Album handling
   Process TOTALDISKS, DISCTOTAL (TPOS) tags so can show disc 2 of 3.
   Process DISCSUBTITLE tag.
   [Adding a disc level GUI for boxed set releases (albums) is under "other issues"]
   
   
6. Album Series
   Add support for album series e.g. “Now That’s what I call music”, to help categorise and navigate various artist complitaions. Some breakup is possible via custom nodes based on path rules, but maybe could scrape Musicbrainz series id and use that to identify series.
   
   
7. (N)(F)Refactor Sorting
   • Fix multi-field sorting – current field contatenation approach is flawed and produces odd results see http://forum.kodi.tv/showthread.php?tid=298544
     
   • Look at initially sorting in DB using ORDER BY and COLLATE for efficiency (currently all sorting is done in memory after fetch from db). But still need to be able produce same sort order in locally held FileItemlist for subsequent on screen order changes.
     
   • Support user defined field combination sorting, not limit to predefined hard coded sort orders.  See http://forum.kodi.tv/showthread.php?tid=...pid2505865
     
   
   
8. Artist Art
   Fix issues where user gets stuck with artist image even when image file has been deleted.
   
   
9. Add Audio properties – bitrate, channels – as data and filter
   TagLib can extract these from music files as part of tag scanning. Store this data in the library, add to smartplaylist and filter rules. Add info labels so skins can display as part of songs node, and expose via JSON API
   
   
10. (F)Fix Other JSON API Issues
    • (F)Cuesheet Support
      Expose song.offset as part of song details. Look at returning track duration for non-library cuecheet items as part of filemode info
      
    
    
11. Export Dynamic Song Data
    Add facility to save and restore song play count, last played datetime and user rating so this data is not lost when music source is dropped, and can be transfered to a clean installation
    
    
12. Review Music Settings UI
    Add to Settings UI for new music options e.g. Forced Scan of tags,  sort artist by sort name
    

I'm just one man so hitting this thread with posts for lots of feature requests (or reminders of what has been requested before) genreally is not going to work; making my todo list unmanageably long  is easy. I suggest you create your own thread for those, I will see them and respond if I can. I would love to be able to do more I really would.  There is a window of opportunity for others to helpfully post some friendly reminders if I have said elsewhere I would do something and I have missed it here. But please just don't tell me how easy you think something is unless you have also submitted a PR to make the changes and just want me to review!

I will be deleting posts and editing etc., so do not take offense if that happens to your post.
I declare this thread is mine to do with as I will!!!

[DaveBlake]

Other Issues
This is the stuff that I am am aware of but do not have the resources as a single dev to deal with at this time. I am happy to collaborate, or offer support where I can, if anyone else wants to take these up.

1. Slow songs node on large libraries
   The known longstanding issue with nodes containing lots of items discussed here
   http://forum.kodi.tv/showthread.php?tid=313962
   http://forum.kodi.tv/showthread.php?tid=213737
   and elsewhere.
   
   
2. Music video support
   Beyond me (sorry Gary), but there is some help here http://forum.kodi.tv/showthread.php?tid=294333
   
   
3. Refactor Library Update
   Replace current delete and re-insert approach when music files are rescanned because of possible metadata changes. This will mean that id remain constant for the time that an item is in the library (usefull for remote apps), and that a true "date added" to library for an item can be known (not just the file timestamp)
   
   
4. Artist Alias
   Support for the various names an individual artist can have, both in different languages and the likes of "Prince".
   
   
5. Improve multidisc set Album handling
   Add a disc level GUI for boxed set releases (albums)
   

[DaveBlake]

#Reserved2

[Uatschitchun]

Nice. Thanks for sharing! :-)
Is "Forced scanning of tags" thought of as a full library/source scan or even on a per-folder/partly scan?
I thought cleanonupdate was already in 17.1...

[DaveBlake]

Nice. Thanks for sharing! :-)

You are welcome. While I am getting this thread sorted please comment on stuff I have forgotten. I will keep the thread generally tidy (i.e. delete posts) later when it becomes a distraction, so note any replies to questions, they may vanish later

Is "Forced scanning of tags" thought of as a full library/source scan or even on a per-folder/partly scan?

Yes, can do per folder. When enabled from advanceded settings, in fileview navigate to a folder and choose "scan to library", then you will be asked if you want to scan tag in files that have not changed. It then skips the hash check and reads the tags. A better UI will come before v18 is released, but it is useful now as a dev tool.

I thought cleanonupdate was already in 17.1...

I think that you are thinking of the fix to the cleanup hanging problem on large libraries https://github.com/xbmc/xbmc/pull/11801, that is in 17.1

What I list here is another issue where having "cleanonupdate" enabled prevents subsequent scraping from doing anything. That didn't get backported.

I am not listing a lot of my recent activity, it was backported to Krypton, but it can all be seen here on Github

Now let's see if I can be brave enough to publish my todo list

[docwra]

Awesome stuff, I ran into cue sheet troubles a fair few years ago now when the cue sheets seemed to be overriding my tags but believe that was fixed a while back. Nice to see some of the more advanced stuff fixed.

Regarding metadata and what is scraped, I am of course here to help with that. I truly believe that the less interaction a user has when scanning in their library the better (with power user features still available if needed) so I would love to see any kind of simplification of this. Documenting what data is scraped on the wiki would also be very useful.

[bsoriano]

Thanks Dave! I can certainly test, as I am already running Leia in my main HTPC.

Regards,

Bart

[scott967]

Artist Sort Name

According to Musicbrainz https://wiki.musicbrainz.org/Style/Artist/Sort_Name, the sort name is supposed to be written in Latin script (assume Latin-1?). So for non-western artists, sort name becomes an "alias" for the artist name which is written in the preferred script. It seems like this would be useful in Kodi, at least as an infolabel for listitems and player. It might also be better if usage was available as a boolean/built-in rather than advancedsetting. How do you foresee skins handling this?

scott s.
.

[Mike_Doc]

Here to help / test in whatever capacity as always Dave, just let me know.

[DaveBlake]

Thanks for the offers of testing help.

Artist Sort Name

According to Musicbrainz https://wiki.musicbrainz.org/Style/Artist/Sort_Name, the sort name is supposed to be written in Latin script (assume Latin-1?). So for non-western artists, sort name becomes an "alias" for the artist name which is written in the preferred script. It seems like this would be useful in Kodi, at least as an infolabel for listitems and player. It might also be better if usage was available as a boolean/built-in rather than advancedsetting. How do you foresee skins handling this?

scott s.

Will keep these comments in mind Scott. I have more phases of artist sorting to implement - info lables are on that list along with JSON access. When I have done all areas in one PR before it has been even slower to get approveal, and the many rebases along the way are painful, so this time I am trying a phased approach.

Seems reasonable that skins should have both artist name and sort name available. But I am undecided about making sorting using sort name rather than artist name as another sort option in the list, I am concerned it will just be clutter for many. Using advancedsetting is just an initial way to make it available, I intend to make a settings up date with various changes all in one go at a later point. I think you are suggesting that it could be a skin level choice?

[DaveBlake]

OK, I have updated the first post with my todo list. It is in no special order, and I may have missed things, so polite reminders are welcome preferably with a link to the post where they were discussed. I will hopefully find time later to edit it into some order of either priority or easiness (unless someone would like to help out with that?). I have also added a know-about-but-no-resources list (2nd post)

There is no music test build available at the moment (the builds get cleaned off the mirrors every month), but the artist sort name PR has been merged so will be in the nightlies if anyone would like to try it out

[tkgafs]

Dave, is the discussion here http://forum.kodi.tv/showthread.php?tid=305853 covering a way to get a playlist to display the artwork for a currently playing track via a switch or the like, likely to get included in your deliberations ?

[docwra]

Only thing I can think of is my eternal request for some kind of A-Z node but done in a programmatic way. Razze gave it a go at one point but didnt find an easy way.

See my original try here: https://github.com/xbmc/xbmc/pull/7518

This would allow us to quickly choose music like a real jukebox.

Someone posted about this recently in feature requests, so I brought it up again just in case

[scott967]

Only thing I can think of is my eternal request for some kind of A-Z node but done in a programmatic way. Razze gave it a go at one point but didnt find an easy way.

See my original try here: https://github.com/xbmc/xbmc/pull/7518

This would allow us to quickly choose music like a real jukebox.

Someone posted about this recently in feature requests, so I brought it up again just in case

A problem is this is affected by general collation issue where other scripts than just ascii are involved.

scott s.
.

[DaveBlake]

Dave, is the discussion here http://forum.kodi.tv/showthread.php?tid=305853 covering a way to get a playlist to display the artwork for a currently playing track via a switch or the like, likely to get included in your deliberations?

I'm still hoping someone else will pick this up, so it is lurking between "todo" and "this I don't have resources for". I won't tidy-up these posts until I decide which, but it will get on the list one way or other.

Only thing I can think of is my eternal request for some kind of A-Z node but done in a programmatic way. Razze gave it a go at one point but didnt find an easy way.

There are non-ascii typescript related difficulties as Scott says, but I get the need to split up large lists of items this way. Would it be wrong to have something for the ascii names, even if it failed for other scripts?

But at the moment my head is in scraping....