Thread Rating:
  • 2 Vote(s) - 5 Average
Kodi 18 - Music Library Development Roadmap - One Man's Plan
#1
I am going to try to maintain this thread as a summary of my current work in progress on improving music library features for the Kodi v18 Leia release. Here you will find a basic road map of what is currently being developed along with my todo list. Any help others can offer - testing, discussing design, documenting or just being fun to hang out with and share my moans - is very welcome. If anyone has the skill, time and interest to set up an IDE and get involved that way then great.

At any time there may be a music test build available with my work so far that is ahead of the nightly.
Currently 20171025-d3186d58ff-ProviderDialog
This is from my repo, and several PRs ahead of Leia master, because of the delays in getting review and approveal to merge so it can appear in the nightlies. As all my work is focussed in one area it is often cumulative, so I can't raise later PRs until the first is merged without a lot of review confusion. Testing of my work before merge will help with the review/approveal process.

I will probably split any testing feedback or feature discussion posted here into a new thread, to stop this thread growing too big and becoming unreadable. Watch out for those when posting.

Work Completed (inc. waiting for review)
  1. (P) Forced scanning of tags
    Option (add <promptfulltagscan> to the musiclibrary section of advancedsettings.xml) to be prompted for a forced scan of tags when rescanning a source from context menu of fileview. Very useful when testing changes to tag processing. The need for a forced scan is now held in the db, so with client server setups you only get prompted once after an upgrade, not every client.  https://github.com/xbmc/xbmc/pull/11557
  2. (F) Cleanonupdate
    Fix implementation (to match video) as since Gotham this option was accidentally preventing any subsequent scraping additional album or artist information, and failed to clean the library when no files were being scanned for tags. https://github.com/xbmc/xbmc/pull/11609
  3. (N) Tags: WRITER, BAND and ENSEMBLE
    Add processing of  WRITER tag, correct BAND and ENSEMBLE tags (Vorbis and APE formats). https://github.com/xbmc/xbmc/pull/11735

  4. Replaygain reduction clipping protection option
    Restore option for replaygain reduction as clipping protection. See forum thread and https://github.com/xbmc/xbmc/pull/11865

  5. (N) Artist sort names - Phase #1
    Add processing of ARTISTSORT (TSOP), ALBUMARTISTSORT (TS02) and COMPOSERSORT (TSOC) tags to provide a sort name for artists. Optionally use field this when sorting artists, albums or songs by artist name. https://github.com/xbmc/xbmc/pull/11738

  6. (F) Duplicate Musicbrainz track ids within in an album
    It is fairly rare but Musicbrainz track ids are not always unique, tracks with the same id can be repeated e.g. silence or on multi-disc anthology albums. Kodi simply didn't load these. https://github.com/xbmc/xbmc/pull/12079

  7. (F)Cuesheet regressions: replay gain and db design
    Refactor db implementation of cuesheets to avoid needlessly slowing the songs node (disabled for v17 anyway), and actually have replay gain applied for library items. Fix other oddities in playback of embedded cuesheet songs from file view.  https://github.com/xbmc/xbmc/pull/12082

  8. (F)Album Type and Record Label merging
    Fix bug where scraped type and label values were always overwriting those derived from tags. When scraped value was empty this effectively cleared the tag value. Also backported to Krypton v17.4
    https://github.com/xbmc/xbmc/pull/12216

  9. Better Album and Artist Additional Info Scraping From Online Sources - Phase #1
    • When fetching additional info on library update:
      • Scan all metadata tags from the music files first before scraping (rather than do it album at a time).
        • This populates the library with the basic music without delay so the user can start using it.
        • Tagging can be mixed quality - some with mbids, some without, so artist mbid may be obtained from music files later in the scan.
        • Avoids unwanted consequnces arising from the order in which folders are processed. The common example of this is the compilation/NFO issue where early processing of various artist compilations causes the artist folder to be misidentified and artist.nfo is either not found or scraping goes wrong. The work around was to put all compliations in a folder named "zzzzzCompilations", to ensure it was processed last, otherwise the song artists from these albums was added to the library and additional info scraped before the other albums by that artist had been scanned, and so the artist.nfo file was never found and applied. This is no longer required.
        • Having passed through all the music during scanning tags we know how many albums we are processing, so during scraping phases the progress bar can show meaningful position though task.
        • Repeat of failed scraping due to server timeouts could be added at a later time.
      • Scraping album artist(s) as part of library update, even if the album lookup has failed.
        Otherwise known artists can go unscraped just because the albums in the library are not found.
      • Avoid repeatedly requesting artists that are not in the online sources.
    • Scanning Musicbrainz release group id from tags as well as release id (album) for use by addons to fetch artwork
    • Saving scraped Musicbranz ids for artist/album (both release id and release group id) when not in tags, but need to ensure succesfully identified and/or provide a means to modify
      https://github.com/xbmc/xbmc/pull/12120
    • Fix throttling of Musicbrainz requests https://github.com/xbmc/xbmc/pull/12402 Significantly reducing the 503 errors and gaps left after scraper runs. Also backported to v17.4
    • Changes to Universal Album Scraper (v2.7.2) https://github.com/xbmc/repo-scrapers/pull/51 avoiding inefficient duplicated requests to Musicbrainz. @Olympia has also fixed some additional parsing issues.

    • Fix Change Information Provider for albums and artists https://github.com/xbmc/xbmc/pull/12597
      a) Setting artist or album info provider selectively that works!
      b) Facility to set info provider for multiple artists or albums
      c) Facility to set default info provider from within library, and to reset all artists or albums
      The old functionality was so broken I doubt it was used. 
  10. That is in the nightly build since 17th August 2017 Note: library is bumped to MyMusic67.db
  11. Music Export / NFO and Art handling for All Artists (Better Album and Artist Additional Info Scraping From Online Sources - Phase #2)
    • Support for local art and nfo files for all artists via new Artist Information Folder and numerous fixes to better match art and nfo with correct artist
    • Export done asynchronously (not lock UI), with new dialog (rather than series of yes/no dialogs), offering export of all artists and allowing greater control of what exported to where.
  12. This is discussed in detail here https://forum.kodi.tv/showthread.php?tid=319284 https://github.com/xbmc/xbmc/pull/12891
------
Test Build 20171025-d3186d58ff-ProviderDialog

(N) = New feature, or major improvement
(P) = Partial, more phases to come later.
(F) = Fixing bug

In Progress (active and backburner)
  1. Tags: ORIGINALDATE(TDOR), DATE(TDRC) and RELEASEDATE (TDRL)
    Support for various dates for albums and songs rather than just year.  See discussion http://forum.kodi.tv/showthread.php?tid=313952
    How users can optionally see music by the date it was originally produced as well as the post 1980s release dates of all digital media, and support for month and date.
  2. Smart Playlist Rules, Filtering and Default Nodes
    A unification of SQL Query building so that filtering and navigation from playlist results combines query logic correctly. Essential ground work before filtering and navigation options can be extended. First attempts are here https://github.com/xbmc/xbmc/pull/10098, but this needs further work.
    Needs to fix navigation of results when have smart playlist within smart playlist, and navigation of artist search results when role <> 1.
  3. Custom Properties
    Facility for songs, albums and artists to have user defined properties, either edited within Kodi, or scanned from custom tags embeded in music files or from NFO files
  4. Music Scraping  From Online Sources
    While others are looking at converting to Python based scrapers, a review of how music is scraped, what data fetched etc., how it is merged and any other core changes to support the Python change.
    • Make scraping of genre data for artists and albums optional
  5. Improved UI - New Info Provider Dialog
    • Easy access to default scraper (Info Provider) settings when Adding Music Source (better work flow)
    • Better UI for change info provider for artists or albums
Todo
This is the stuff that I feel I have a reasonable chance of dealing with as a single dev, or have a strong  interest in. I know there are a lot more requests for things people would like, and major changes that are beyond me, for my sanity they are not listed here.  But I am happy to collaborate or offer support where I can if anyone else wants to make improvements or fixes to music.

In no particular order....
  1. Smartplaylist and Filter Rules
    • Track and Disc Number
      Fix track number smart playlist rule to correctly split just track number form the combined track and disc filed in song table. Add disc number as a smart playlist rule and filter option for songs and albums.
    • Filetimestamp
      Add filetimestamp (misnamed Dateadded field) as smartplaylist and filter rule for artists, albums and songs.
    • Recently Added (unscraped) Artists
      A way to list those recently added or not yet sucesfully scraped for re-scraping attempts.
  2. Consistent Application of Node Options
    Apply folder type node options at genre or role level. Currently can create a node with say <path>musicdb:// genres/?ambumartistonly=true&year=1988</path> that will show all genres, but when navigate down for a selected genre the options are applied only see the album artists with songs from 1988, and there could be nothing to show.
  3. Embedded Art
    Fix way APIC tag is handled to correctly fetch front cover art when there is more than one image. Explore possibility to handle more than one image per song or album.
  4. Genre Handling
    • Genre Separator - Allow for multiple item separators,  add ";" (a sensible separator value). Useful for ID3 v2.3 tags (single frame) as semicolon is the default separator used by meia Monkey when tagging.
    • Artist genre (scraped value) link fully to the genre table rather than leave it as display text.
    • Album genre (initially from tags but also merged scraped value) ensure relationship to genre table values is maintained regardless of source (tags, scraper or JSON).  Rework how initial values determined from song genres, see discussion http://forum.kodi.tv/showthread.php?tid=278315
    • Add support for a genre hierarchy, a way to group genre together as subgenres to unscramble the usual large mix of genres that result from scraping/tags
    • Clarify JSON API and smart playlist/filter rules where artist, album and song genre are being used.
  5. Improve multidisc set Album handling
    Process TOTALDISKS, DISCTOTAL (TPOS) tags so can show disc 2 of 3.
    Process DISCSUBTITLE tag.
    Management of separate cover images for each disc in a set.
  6. Album Series
    Add support for album series e.g. “Now That’s what I call music”, to help categorise and navigate various artist complitaions. Some breakup is possible via custom nodes based on path rules, but maybe could scrape Musicbrainz series id and use that to identify series.
  7. (N)(F)Refactor Sorting
    • Fix multi-field sorting – current field contatenation approach is flawed and produces odd results see http://forum.kodi.tv/showthread.php?tid=298544
    • Look at initially sorting in DB using ORDER BY and COLLATE for efficiency (currently all sorting is done in memory after fetch from db). But still need to be able produce same sort order in locally held FileItemlist for subsequent on screen order changes.
    • Support user defined field combination sorting, not limit to predefined hard coded sort orders.  See http://forum.kodi.tv/showthread.php?tid=...pid2505865
  8. (N) Artist sort names - Phase #2
    Add info labels and expose via JSON API
  9. Artist Art
    Fix issues where user gets stuck with artist image even when image file has been deleted.
  10. Add Audio properties – bitrate, channels – as data and filter
    TagLib can extract these from music files as part of tag scanning. Store this data in the library, add to smartplaylist and filter rules. Add info labels so skins can display as part of songs node, and expose via JSON API
  11. (F)Album Info Dialog
    Replace the display of scraped tracks (when scraped) with always showing the songs in the library from the album. This deprecates the albumsonginfo table. See http://forum.kodi.tv/showthread.php?tid=298528
    Add a click link from each song to that song info? Sort out path song info <-> album info when refresh hit (same dialog so gets confused).

    Will have to refactor to fetch data on a separate thread too see http://forum.kodi.tv/showthread.php?tid=...pid2466103 or PR will at blocked
  12. (F)Fix Other JSON API Issues
    • (F)Artist returned as an array
      Fix CAudioLibrary::GetArtist to always return "artist" as a string (as the spec defines). Some individual artists from MB do have " / " (space-slash-space) in the name, and they get split into an array.
    • (F)Setting album and song details
      Fix CAudioLibrary::SetAlbumDetails SetSongDetails to actualy set artists credits and genre fully, not just the display strings leaving the underlying data unchanged.
    • (F)Cuesheet Support
      Expose song.offset as part of song details. Look at returning track duration for non-library cuecheet items as part of filemode info
  13. Review Music Settings UI
    Add to Settings UI for new music options e.g. Forced Scan of tags,  sort artist by sort name

I'm just one man so hitting this thread with posts for lots of feature requests (or reminders of what has been requested before) genreally is not going to work; making my todo list unmanageably long  is easy. I suggest you create your own thread for those, I will see them and respond if I can. I would love to be able to do more I really would.  There is a window of opportunity for others to helpfully post some friendly reminders if I have said elsewhere I would do something and I have missed it here. But please just don't tell me how easy you think something is unless you have also submitted a PR to make the changes and just want me to review!

I will be deleting posts and editing etc., so do not take offense if that happens to your post.
I declare this thread is mine to do with as I will!!! Smile
Save
Reply
#2
Other Issues
This is the stuff that I am am aware of but do not have the resources as a single dev to deal with at this time. I am happy to collaborate, or offer support where I can, if anyone else wants to take these up. Smile
  1. Slow songs node on large libraries
    The known longstanding issue with nodes containing lots of items discussed here
    http://forum.kodi.tv/showthread.php?tid=313962
    http://forum.kodi.tv/showthread.php?tid=213737
    and elsewhere.

  2. Music video support
    Beyond me (sorry Gary), but there is some help here http://forum.kodi.tv/showthread.php?tid=294333

  3. Refactor Library Update
    Replace current delete and re-insert approach when music files are rescanned because of possible metadata changes. This will mean that id remain constant for the time that an item is in the library (usefull for remote apps), and that a true "date added" to library for an item can be known (not just the file timestamp)

  4. Artist Alias
    Support for the various names an individual artist can have, both in different languages and the likes of "Prince"
Reply
#3
#Reserved2
Reply
#4
Nice. Thanks for sharing! :-)
Is "Forced scanning of tags" thought of as a full library/source scan or even on a per-folder/partly scan?
I thought cleanonupdate was already in 17.1...
Reply
#5
(2017-03-31, 17:44)Uatschitchun Wrote: Nice. Thanks for sharing! :-)
You are welcome. While I am getting this thread sorted please comment on stuff I have forgotten. I will keep the thread generally tidy (i.e. delete posts) later when it becomes a distraction, so note any replies to questions, they may vanish later Smile
Quote:Is "Forced scanning of tags" thought of as a full library/source scan or even on a per-folder/partly scan?
Yes, can do per folder. When enabled from advanceded settings, in fileview navigate to a folder and choose "scan to library", then you will be asked if you want to scan tag in files that have not changed. It then skips the hash check and reads the tags. A better UI will come before v18 is released, but it is useful now as a dev tool.
Quote:I thought cleanonupdate was already in 17.1...
I think that you are thinking of the fix to the cleanup hanging problem on large libraries https://github.com/xbmc/xbmc/pull/11801, that is in 17.1

What I list here is another issue where having "cleanonupdate" enabled prevents subsequent scraping from doing anything. That didn't get backported.

I am not listing a lot of my recent activity, it was backported to Krypton, but it can all be seen here on Github

Now let's see if I can be brave enough to publish my todo list Smile
Reply
#6
Awesome stuff, I ran into cue sheet troubles a fair few years ago now when the cue sheets seemed to be overriding my tags but believe that was fixed a while back. Nice to see some of the more advanced stuff fixed.

Regarding metadata and what is scraped, I am of course here to help with that. I truly believe that the less interaction a user has when scanning in their library the better (with power user features still available if needed) so I would love to see any kind of simplification of this. Documenting what data is scraped on the wiki would also be very useful.
Image Formally chef Zag
Reply
#7
Thanks Dave! I can certainly test, as I am already running Leia in my main HTPC.

Regards,

Bart
Reply
#8
Artist Sort Name

According to Musicbrainz https://wiki.musicbrainz.org/Style/Artist/Sort_Name, the sort name is supposed to be written in Latin script (assume Latin-1?). So for non-western artists, sort name becomes an "alias" for the artist name which is written in the preferred script. It seems like this would be useful in Kodi, at least as an infolabel for listitems and player. It might also be better if usage was available as a boolean/built-in rather than advancedsetting. How do you foresee skins handling this?

scott s.
.
Reply
#9
Here to help / test in whatever capacity as always Dave, just let me know.
SKIN: Aeon Madnox
RIP Schimi2k, we miss you.
When I grow up I want to be a skilled skinner
Thank me by ⬇ adding to my + reputation
Reply
#10
Thanks for the offers of testing help.

(2017-04-05, 01:28)scott967 Wrote: Artist Sort Name

According to Musicbrainz https://wiki.musicbrainz.org/Style/Artist/Sort_Name, the sort name is supposed to be written in Latin script (assume Latin-1?). So for non-western artists, sort name becomes an "alias" for the artist name which is written in the preferred script. It seems like this would be useful in Kodi, at least as an infolabel for listitems and player. It might also be better if usage was available as a boolean/built-in rather than advancedsetting. How do you foresee skins handling this?

scott s.
Will keep these comments in mind Scott. I have more phases of artist sorting to implement - info lables are on that list along with JSON access. When I have done all areas in one PR before it has been even slower to get approveal, and the many rebases along the way are painful, so this time I am trying a phased approach.

Seems reasonable that skins should have both artist name and sort name available. But I am undecided about making sorting using sort name rather than artist name as another sort option in the list, I am concerned it will just be clutter for many. Using advancedsetting is just an initial way to make it available, I intend to make a settings up date with various changes all in one go at a later point. I think you are suggesting that it could be a skin level choice?
Reply
#11
OK, I have updated the first post with my todo list. It is in no special order, and I may have missed things, so polite reminders are welcome preferably with a link to the post where they were discussed. I will hopefully find time later to edit it into some order of either priority or easiness (unless someone would like to help out with that?). I have also added a know-about-but-no-resources list (2nd post)

There is no music test build available at the moment (the builds get cleaned off the mirrors every month), but the artist sort name PR has been merged so will be in the nightlies if anyone would like to try it out Smile
Reply
#12
Dave, is the discussion here http://forum.kodi.tv/showthread.php?tid=305853 covering a way to get a playlist to display the artwork for a currently playing track via a switch or the like, likely to get included in your deliberations ?
Reply
#13
Only thing I can think of is my eternal request for some kind of A-Z node but done in a programmatic way. Razze gave it a go at one point but didnt find an easy way.

See my original try here: https://github.com/xbmc/xbmc/pull/7518

This would allow us to quickly choose music like a real jukebox.

Someone posted about this recently in feature requests, so I brought it up again just in case Wink
Image Formally chef Zag
Reply
#14
(2017-05-15, 10:59)docwra Wrote: Only thing I can think of is my eternal request for some kind of A-Z node but done in a programmatic way. Razze gave it a go at one point but didnt find an easy way.

See my original try here: https://github.com/xbmc/xbmc/pull/7518

This would allow us to quickly choose music like a real jukebox.

Someone posted about this recently in feature requests, so I brought it up again just in case Wink

A problem is this is affected by general collation issue where other scripts than just ascii are involved.

scott s.
.
Reply
#15
(2017-05-15, 10:38)tkgafs Wrote: Dave, is the discussion here http://forum.kodi.tv/showthread.php?tid=305853 covering a way to get a playlist to display the artwork for a currently playing track via a switch or the like, likely to get included in your deliberations?
I'm still hoping someone else will pick this up, so it is lurking between "todo" and "this I don't have resources for". I won't tidy-up these posts until I decide which, but it will get on the list one way or other.

(2017-05-15, 10:59)docwra Wrote: Only thing I can think of is my eternal request for some kind of A-Z node but done in a programmatic way. Razze gave it a go at one point but didnt find an easy way.
There are non-ascii typescript related difficulties as Scott says, but I get the need to split up large lists of items this way. Would it be wrong to have something for the ascii names, even if it failed for other scripts?

But at the moment my head is in scraping....
Reply



Music Library Development Roadmap - One Man's Plan52