Kodi 18 - Music Library Development Roadmap - One Man's Plan

  Thread Rating:
  • 2 Vote(s) - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
DaveBlake Offline
Team-Kodi Member
Posts: 2,209
Joined: Jun 2015
Reputation: 61
Location: South West England
Post: #1
I am going to try to maintain this thread as a summary of my current work in progress on improving music library features for the Kodi v18 Leia release. Here you will find a basic road map of what is currently being developed along with my todo list. Any help others can offer - testing, discussing design, documenting or just being fun to hang out with and share my moans - is very welcome. If anyone has the skill, time and interest to set up an IDE and get involved that way then great.

At any time there may be a music test build available with my work so far that is ahead of the nightly.
Currently None available
This is from my repo, and several PRs ahead of Leia master, because of the delays in getting review and approveal to merge so it can appear in the nightlies. As all my work is focussed in one area it is often cumulative, so I can't raise later PRs until the first is merged without a lot of review confusion. Testing of my work before merge will help with the review/approveal process.

I will probably split any testing feedback or feature discussion posted here into a new thread, to stop this thread growing too big and becoming unreadable. Watch out for those when posting.

Work Completed (inc. waiting for review)
  1. (P) Forced scanning of tags
    Option (add <promptfulltagscan> to the musiclibrary section of advancedsettings.xml) to be prompted for a forced scan of tags when rescanning a source from context menu of fileview. Very useful when testing changes to tag processing. The need for a forced scan is now held in the db, so with client server setups you only get prompted once after an upgrade, not every client. https://github.com/xbmc/xbmc/pull/11557
  2. (F) Cleanonupdate
    Fix implementation (to match video) as since Gotham this option was accidentally preventing any subsequent scraping additional album or artist information, and failed to clean the library when no files were being scanned for tags. https://github.com/xbmc/xbmc/pull/11609
  3. (N) Tags: WRITER, BAND and ENSEMBLE
    Add processing of WRITER tag, correct BAND and ENSEMBLE tags (Vorbis and APE formats). https://github.com/xbmc/xbmc/pull/11735
  4. Replaygain reduction clipping protection option
    Restore option for replaygain reduction as clipping protection. See forum thread and https://github.com/xbmc/xbmc/pull/11865
  5. (N) Artist sort names - Phase #1
    Add processing of ARTISTSORT (TSOP), ALBUMARTISTSORT (TS02) and COMPOSERSORT (TSOC) tags to provide a sort name for artists. Optionally use field this when sorting artists, albums or songs by artist name. https://github.com/xbmc/xbmc/pull/11738
  6. (F) Duplicate Musicbrainz track ids within in an album
    It is fairly rare but Musicbrainz track ids are not always unique, tracks with the same id can be repeated e.g. silence or on multi-disc anthology albums. Kodi simply didn't load these. https://github.com/xbmc/xbmc/pull/12079
  7. (F)Cuesheet regressions: replay gain and db design
    Refactor db implementation of cuesheets to avoid needlessly slowing the songs node (disabled for v17 anyway), and actually have replay gain applied for library items. Fix other oddities in playback of embedded cuesheet songs from file view. https://github.com/xbmc/xbmc/pull/12082
  8. (F)Album Type and Record Label merging
    Fix bug where scraped type and label values were always overwriting those derived from tags. When scraped value was empty this effectively cleared the tag value. Also backported to Krypton v17.4
    https://github.com/xbmc/xbmc/pull/12216

    That is in the nightly build since 2nd June 2017
    -
  9. Better Album and Artist Additional Info Scraping From Online Sources - Phase #1
    • Scraping album artist(s) as part of library update, even if the album lookup has failed.
    • Saving scraped Musicbranz ids for artist/album when not in tags, but need to ensure succesfully identified and/or provide a means to modify https://github.com/xbmc/xbmc/pull/12120

    In test build: None available
    -
(N) = New feature, or major improvement
(P) = Partial, more phases to come later.
(F) = Fixing bug

In Progress (active and backburner)
  1. Tags: ORIGINALDATE(TDOR), DATE(TDRC) and RELEASEDATE (TDRL)
    Support for various dates for albums and songs rather than just year. See discussion http://forum.kodi.tv/showthread.php?tid=313952
    How users can optionally see music by the date it was originally produced as well as the post 1980s release dates of all digital media, and support for month and date.
  2. Smart Playlist Rules, Filtering and Default Nodes
    A unification of SQL Query building so that filtering and navigation from playlist results combines query logic correctly. Essential ground work before filtering and navigation options can be extended. First attempts are here https://github.com/xbmc/xbmc/pull/10098, but this needs further work.
    Needs to fix navigation of results when have smart playlist within smart playlist, and navigation of artist search results when role <> 1.
  3. Custom Properties
    Facility for songs, albums and artists to have user defined properties, either edited within Kodi, or scanned from custom tags embeded in music files or from NFO files
  4. Music Scraping From Online Sources
    While others are looking at converting to Python based scrapers, a review of how music is scraped, what data fetched etc., how it is merged and any other core changes to support the Python change.
    • Make scraping of genre data for artists and albums optional

Todo
This is the stuff that I feel I have a reasonable chance of dealing with as a single dev, or have a strong interest in. I know there are a lot more requests for things people would like, and major changes that are beyond me, for my sanity they are not listed here. But I am happy to collaborate or offer support where I can if anyone else wants to make improvements or fixes to music.

In no particular order....
  1. Smartplaylist and Filter Rules
    • Track and Disc Number
      Fix track number smart playlist rule to correctly split just track number form the combined track and disc filed in song table. Add disc number as a smart playlist rule and filter option for songs and albums.
    • Filetimestamp
      Add filetimestamp (misnamed Dateadded field) as smartplaylist and filter rule for artists, albums and songs.
    • Recently Added (unscraped) Artists
      A way to list those recently added or not yet sucesfully scraped for re-scraping attempts.
  2. Compilations and NFO location
    Fix scraping of NFO files so that folder order of folder containing compilation albums does not cause wrong artist location to be determined.
  3. Consistent Application of Node Options
    Apply folder type node options at genre or role level. Currently can create a node with say <path>musicdb:// genres/?ambumartistonly=true&year=1988</path> that will show all genres, but when navigate down for a selected genre the options are applied only see the album artists with songs from 1988, and there could be nothing to show.
  4. Embedded Art
    Fix way APIC tag is handled to correctly fetch front cover art when there is more than one image. Explore possibility to handle more than one image per song or album.
  5. Genre Handling
    • Genre Separator - Allow for multiple item separators, add ";" (a sensible separator value). Useful for ID3 v2.3 tags (single frame) as semicolon is the default separator used by meia Monkey when tagging.
    • Artist genre (scraped value) link fully to the genre table rather than leave it as display text.
    • Album genre (initially from tags but also merged scraped value) ensure relationship to genre table values is maintained regardless of source (tags, scraper or JSON). Rework how initial values determined from song genres, see discussion http://forum.kodi.tv/showthread.php?tid=278315
    • Add support for a genre hierarchy, a way to group genre together as subgenres to unscramble the usual large mix of genres that result from scraping/tags
    • Clarify JSON API and smart playlist/filter rules where artist, album and song genre are being used.
  6. Improve multidisc set Album handling
    Process TOTALDISKS, DISCTOTAL (TPOS) tags so can show disc 2 of 3.
    Process DISCSUBTITLE tag.
    Management of separate cover images for each disc in a set.
  7. Album Series
    Add support for album series e.g. “Now That’s what I call music”, to help categorise and navigate various artist complitaions. Some breakup is possible via custom nodes based on path rules, but maybe could scrape Musicbrainz series id and use that to identify series.
  8. Refactor Sorting
    • Fix multi-field sorting – current field contatenation approach is flawed and produces odd results see http://forum.kodi.tv/showthread.php?tid=298544
    • Look at initially sorting in DB using ORDER BY and COLLATE for efficiency (currently all sorting is done in memory after fetch from db). But still need to be able produce same sort order in locally held FileItemlist for subsequent on screen order changes.
    • Support user defined field combination sorting, not limit to predefined hard coded sort orders. See http://forum.kodi.tv/showthread.php?tid=...pid2505865
  9. (N) Artist sort names - Phase #2
    Add info labels and expose via JSON API
  10. Artist Art
    Fix issues where user gets stuck with artist image even when image file has been deleted.
  11. Add Audio properties – bitrate, channels – as data and filter
    TagLib can extract these from music files as part of tag scanning. Store this data in the library, add to smartplaylist and filter rules. Add info labels so skins can display as part of songs node, and expose via JSON API
  12. (F)Album Info Dialog
    Replace the display of scraped tracks (when scraped) with always showing the songs in the library from the album. This deprecates the albumsonginfo table. See http://forum.kodi.tv/showthread.php?tid=298528
    Add a click link from each song to that song info? Sort out path song info <-> album info when refresh hit (same dialog so gets confused).

    Will have to refactor to fetch data on a separate thread too see http://forum.kodi.tv/showthread.php?tid=...pid2466103 or PR will at blocked
  13. (F)Fix Other JSON API Issues
    • (F)Artist returned as an array
      Fix CAudioLibrary::GetArtist to always return "artist" as a string (as the spec defines). Some individual artists from MB do have " / " (space-slash-space) in the name, and they get split into an array.
    • (F)Setting album and song details
      Fix CAudioLibrary::SetAlbumDetails SetSongDetails to actualy set artists credits and genre fully, not just the display strings leaving the underlying data unchanged.
    • (F)Cuesheet Support
      Expose song.offset as part of song details. Look at returning track duration for non-library cuecheet items as part of filemode info
  14. Review Music Settings UI
    Add to Settings UI for new music options e.g. Forced Scan of tags, sort artist by sort name

I'm just one man so hitting this thread with posts for lots of feature requests (or reminders of what has been requested before) genreally is not going to work; making my todo list unmanageably long is easy. I suggest you create your own thread for those, I will see them and respond if I can. I would love to be able to do more I really would. There is a window of opportunity for others to helpfully post some friendly reminders if I have said elsewhere I would do something and I have missed it here. But please just don't tell me how easy you think something is unless you have also submitted a PR to make the changes and just want me to review!

I will be deleting posts and editing etc., so do not take offense if that happens to your post.
I declare this thread is mine to do with as I will!!! Smile
(This post was last modified: 2017-06-21 09:01 by DaveBlake.)
find quote
DaveBlake Offline
Team-Kodi Member
Posts: 2,209
Joined: Jun 2015
Reputation: 61
Location: South West England
Post: #2
Other Issues
This is the stuff that I am am aware of but do not have the resources as a single dev to deal with at this time. I am happy to collaborate, or offer support where I can, if anyone else wants to take these up. Smile
  1. Slow songs node on large libraries
    The known longstanding issue with nodes containing lots of items discussed here
    http://forum.kodi.tv/showthread.php?tid=313962
    http://forum.kodi.tv/showthread.php?tid=213737
    and elsewhere.
  2. Music video support
    Beyond me (sorry Gary), but there is some help here http://forum.kodi.tv/showthread.php?tid=294333
  3. Refactor Library Update
    Replace current delete and re-insert approach when music files are rescanned because of possible metadata changes. This will mean that id remain constant for the time that an item is in the library (usefull for remote apps), and that a true "date added" to library for an item can be known (not just the file timestamp)
  4. Artist Alias
    Support for the various names an individual artist can have, both in different languages and the likes of "Prince"
(This post was last modified: 2017-05-23 07:26 by DaveBlake.)
find quote
DaveBlake Offline
Team-Kodi Member
Posts: 2,209
Joined: Jun 2015
Reputation: 61
Location: South West England
Post: #3
#Reserved2
find quote
Uatschitchun Offline
Senior Member
Posts: 204
Joined: Nov 2016
Reputation: 0
Post: #4
Nice. Thanks for sharing! :-)
Is "Forced scanning of tags" thought of as a full library/source scan or even on a per-folder/partly scan?
I thought cleanonupdate was already in 17.1...
find quote
DaveBlake Offline
Team-Kodi Member
Posts: 2,209
Joined: Jun 2015
Reputation: 61
Location: South West England
Post: #5
(2017-03-31 17:44)Uatschitchun Wrote:  Nice. Thanks for sharing! :-)
You are welcome. While I am getting this thread sorted please comment on stuff I have forgotten. I will keep the thread generally tidy (i.e. delete posts) later when it becomes a distraction, so note any replies to questions, they may vanish later Smile
Quote:Is "Forced scanning of tags" thought of as a full library/source scan or even on a per-folder/partly scan?
Yes, can do per folder. When enabled from advanceded settings, in fileview navigate to a folder and choose "scan to library", then you will be asked if you want to scan tag in files that have not changed. It then skips the hash check and reads the tags. A better UI will come before v18 is released, but it is useful now as a dev tool.
Quote:I thought cleanonupdate was already in 17.1...
I think that you are thinking of the fix to the cleanup hanging problem on large libraries https://github.com/xbmc/xbmc/pull/11801, that is in 17.1

What I list here is another issue where having "cleanonupdate" enabled prevents subsequent scraping from doing anything. That didn't get backported.

I am not listing a lot of my recent activity, it was backported to Krypton, but it can all be seen here on Github

Now let's see if I can be brave enough to publish my todo list Smile
find quote
docwra Offline
Fan
Posts: 451
Joined: Nov 2014
Reputation: 18
Post: #6
Awesome stuff, I ran into cue sheet troubles a fair few years ago now when the cue sheets seemed to be overriding my tags but believe that was fixed a while back. Nice to see some of the more advanced stuff fixed.

Regarding metadata and what is scraped, I am of course here to help with that. I truly believe that the less interaction a user has when scanning in their library the better (with power user features still available if needed) so I would love to see any kind of simplification of this. Documenting what data is scraped on the wiki would also be very useful.

[Image: cook.png] Formally chef Zag
(This post was last modified: 2017-04-04 09:48 by docwra.)
find quote
bsoriano Offline
Donor
Posts: 712
Joined: Mar 2011
Reputation: 20
Location: Santo Domingo, Dominican Republic
Post: #7
Thanks Dave! I can certainly test, as I am already running Leia in my main HTPC.

Regards,

Bart
find quote
scott967 Offline
Posting Freak
Posts: 2,716
Joined: Jul 2012
Reputation: 85
Post: #8
Artist Sort Name

According to Musicbrainz https://wiki.musicbrainz.org/Style/Artist/Sort_Name, the sort name is supposed to be written in Latin script (assume Latin-1?). So for non-western artists, sort name becomes an "alias" for the artist name which is written in the preferred script. It seems like this would be useful in Kodi, at least as an infolabel for listitems and player. It might also be better if usage was available as a boolean/built-in rather than advancedsetting. How do you foresee skins handling this?

scott s.
.
find quote
Mike_Doc Offline
Posting Freak
Posts: 1,575
Joined: May 2014
Reputation: 123
Location: In a nice white padded cell
Post: #9
Here to help / test in whatever capacity as always Dave, just let me know.

SKIN: Aeon Madnox

RIP Schimi2k, we miss you.

When I grow up I want to be a skilled skinner
Thank me by ⬇ adding to my + reputation
find quote
DaveBlake Offline
Team-Kodi Member
Posts: 2,209
Joined: Jun 2015
Reputation: 61
Location: South West England
Post: #10
Thanks for the offers of testing help.

(2017-04-05 01:28)scott967 Wrote:  Artist Sort Name

According to Musicbrainz https://wiki.musicbrainz.org/Style/Artist/Sort_Name, the sort name is supposed to be written in Latin script (assume Latin-1?). So for non-western artists, sort name becomes an "alias" for the artist name which is written in the preferred script. It seems like this would be useful in Kodi, at least as an infolabel for listitems and player. It might also be better if usage was available as a boolean/built-in rather than advancedsetting. How do you foresee skins handling this?

scott s.
Will keep these comments in mind Scott. I have more phases of artist sorting to implement - info lables are on that list along with JSON access. When I have done all areas in one PR before it has been even slower to get approveal, and the many rebases along the way are painful, so this time I am trying a phased approach.

Seems reasonable that skins should have both artist name and sort name available. But I am undecided about making sorting using sort name rather than artist name as another sort option in the list, I am concerned it will just be clutter for many. Using advancedsetting is just an initial way to make it available, I intend to make a settings up date with various changes all in one go at a later point. I think you are suggesting that it could be a skin level choice?
find quote
DaveBlake Offline
Team-Kodi Member
Posts: 2,209
Joined: Jun 2015
Reputation: 61
Location: South West England
Post: #11
OK, I have updated the first post with my todo list. It is in no special order, and I may have missed things, so polite reminders are welcome preferably with a link to the post where they were discussed. I will hopefully find time later to edit it into some order of either priority or easiness (unless someone would like to help out with that?). I have also added a know-about-but-no-resources list (2nd post)

There is no music test build available at the moment (the builds get cleaned off the mirrors every month), but the artist sort name PR has been merged so will be in the nightlies if anyone would like to try it out Smile
find quote
tkgafs Offline
Senior Member
Posts: 115
Joined: May 2008
Reputation: 2
Post: #12
Dave, is the discussion here http://forum.kodi.tv/showthread.php?tid=305853 covering a way to get a playlist to display the artwork for a currently playing track via a switch or the like, likely to get included in your deliberations ?
find quote
docwra Offline
Fan
Posts: 451
Joined: Nov 2014
Reputation: 18
Post: #13
Only thing I can think of is my eternal request for some kind of A-Z node but done in a programmatic way. Razze gave it a go at one point but didnt find an easy way.

See my original try here: https://github.com/xbmc/xbmc/pull/7518

This would allow us to quickly choose music like a real jukebox.

Someone posted about this recently in feature requests, so I brought it up again just in case Wink

[Image: cook.png] Formally chef Zag
(This post was last modified: 2017-05-15 11:00 by docwra.)
find quote
scott967 Offline
Posting Freak
Posts: 2,716
Joined: Jul 2012
Reputation: 85
Post: #14
(2017-05-15 10:59)docwra Wrote:  Only thing I can think of is my eternal request for some kind of A-Z node but done in a programmatic way. Razze gave it a go at one point but didnt find an easy way.

See my original try here: https://github.com/xbmc/xbmc/pull/7518

This would allow us to quickly choose music like a real jukebox.

Someone posted about this recently in feature requests, so I brought it up again just in case Wink

A problem is this is affected by general collation issue where other scripts than just ascii are involved.

scott s.
.
find quote
DaveBlake Offline
Team-Kodi Member
Posts: 2,209
Joined: Jun 2015
Reputation: 61
Location: South West England
Post: #15
(2017-05-15 10:38)tkgafs Wrote:  Dave, is the discussion here http://forum.kodi.tv/showthread.php?tid=305853 covering a way to get a playlist to display the artwork for a currently playing track via a switch or the like, likely to get included in your deliberations?
I'm still hoping someone else will pick this up, so it is lurking between "todo" and "this I don't have resources for". I won't tidy-up these posts until I decide which, but it will get on the list one way or other.

(2017-05-15 10:59)docwra Wrote:  Only thing I can think of is my eternal request for some kind of A-Z node but done in a programmatic way. Razze gave it a go at one point but didnt find an easy way.
There are non-ascii typescript related difficulties as Scott says, but I get the need to split up large lists of items this way. Would it be wrong to have something for the ascii names, even if it failed for other scripts?

But at the moment my head is in scraping....
find quote
Post Reply