Guest - Testers are needed for the reworked CDateTime core component. See... https://forum.kodi.tv/showthread.php?tid=378981 (September 29) x
  • 1
  • 2
  • 3
  • 4(current)
  • 5
  • 6
  • 24
Managing music albums Box-Sets properly with KODI library feature
#46
(2019-08-12, 21:04)black_eagle Wrote: @DaveBlake  Don't go complicating things further than they already are Big Grin  I will look at improving matching if I ever get once I get a working example that does the basics.

I'm off again now to stare at code and try to make it do what I want !!!!

-=EDIT=-

Got a bit further.  Kodi now builds a 'boxsets' table.  Using 'idAlbum' I can pull out all the sets in my (small) test directory.

Image

 I can also narrow down the query to just one album.

Image

Using idAlbum and strDiscSubtitle it should now be possible to pull out all the tracks for an individual disc in a set as the song table also contains the disc subtitle where appropriate.

Now I need to figure out how to display it !

just be prepared for a lot of this.. theres a ton of instances on MB where only some discs on box sets have subtitles

Image
Reply
#47
@black_eagle it is probably best I leave you learn by doing until you have something you would like me to look at (meaning something in a Github repo) . I'm happy to answer questions of course, and give suggestions when you really get stuck.

Meanwhile can others not get too excited when he says he has something "working", I know from experience that there will be more required before it is fit to add to Kodi. But I'm delighted that you are giving it a go @black_eagle, it really is the only way to begin to climb the Kodi mountain.
Reply
#48
(2019-08-16, 10:03)DaveBlake Wrote: @black_eagle it is probably best I leave you learn by doing until you have something you would like me to look at (meaning something in a Github repo) . I'm happy to answer questions of course, and give suggestions when you really get stuck.

Meanwhile can others not get too excited when he says he has something "working", I know from experience that there will be more required before it is fit to add to Kodi. But I'm delighted that you are giving it a go @black_eagle, it really is the only way to begin to climb the Kodi mountain.

Thanks Dave, I really appreciate that.  Yeah, I'm aware that it'll have to be in a Github repo.  Currently though, although I cloned Kodi locally, I haven't pushed anything to Github yet and in any case, various changes have already been made so what I have done so far will need rebasing at some point.

There is still much to do before it's anywhere near acceptable for Kodi.  There is no documentation for anything yet, there is a bunch of logging to remove that isn't helpful to anyone but me and some of the stuff might not meet the coding standards, although I have tried to follow the guidelines with regard to variable naming, spacing and {} etc. I haven't touched JSON and I need to add the subtitles to import and export of the library.  There is probably also other stuff that I haven't either considered or tripped up over that will need sorting.

There's also ListItem.DBType.  Do I need to add a 'boxset' to that ?  Guess that might remove the hit & miss method I have at the moment for displaying sets, but in that case, how do I do it ??  Lots to think about, and lots to consider.......

I do have a question though Dave !!  I know you develop on Windows, and I was wondering if it's possible to run Kodi inside VS and be able to step through/into it, if you set a breakpoint in a function and if so, is it possible to get variable values in VS using variable names ?  I can do it to a certain extent on Linux using gdb, but the code is assembly and variables are only memory locations which makes things a bit more difficult to pin down at times.  Mostly I've been dumping stuff out to the log file in order to 'see' what's going on inside Kodi.
Learning Linux the hard way !!
Reply
#49
Yes I use a Windows dev env and Visual Studio can do all those things and more. Many of the team devs use Linux,  I don't know what tools but I'm sure they are as good if not better than VS. Sounds like you need to get some better tools one way or another.

ListItem.DBType is a video library info label thing, maybe music needs something similar but I have doubts. I need to know more what you have done to comment intelligently.

No rush, no pressure, it is a hard mountain you are climbing, but expect lots of critical feedback and more work Smile
Reply
#50
@DaveBlake , well......It's been a busy week or so, what with family, work, beer, beer, barbecues and beer !!  Still, I do have a working example now !!  I have a 'boxsets' node which lists just the albums Kodi has determined to be a boxset (there is a disc subtitle present).  Clicking on an album opens the next node which shows the individual discs.  Clicking on a disc shows the tracks from just that disc.

Along the way I have discovered that I don't need the boxsets table at all !! so I shall remove that at some point.  There is still much to be done though.  Individual art is not supported for the discs inside a set, instead Kodi uses the coverart for the main album so I think that probably needs to be added.  I also need to add to the import and export routines for the music library,add to the update routine to include boxsets and add everything to the JSON routines.

There is also some cosmetic stuff to fix.  Currently, when viewing the boxsets node, the string is 'MUSIC/BOX_SETS', but if you open a set, it changes to 'MUSIC/ALBUM_NAME'.  I'd prefer that to be 'MUSIC/BOX_SETS/ALBUM_NAME' and for the discs and tracks to follow that convention which currently, they don't.

Still, I think at this point that the main thing is the bare bones of it work.

Some screenshots to show what I mean.

Image

Image

Image

Image

Image

Image
Learning Linux the hard way !!
Reply
#51
^ like button..
Reply
#52
So far I have been using a small subset of albums to test with, but I had some free time today so I decided to use a bigger set to see what the results were.  From a set of around 900 albums, Kodi identified 46 boxsets.  This was wrong!  In quite a few cases, albums were added where the first disc has no name, but the second/third disc is named.  Quite often as 'Bonus disc' or 'Live'.  I decided to alter the logic a little so that an album is only added as a set when at least two discs are named and one of them is the first disc.  This cleaned out all of the albums with bonus discs but it still left me with some albums which aren't sets but do have all the discs named.

EG - Kate Bush : Aerial, where the discs are named 'A sea of Honey' & 'A sky of Honey'.

At this point, I decided (and oh, how I wish I hadn't right now !!) that it would be a good idea to look at how the context menu works and to see if I could add an option to remove a set.  This doesn't actually remove anything from the database, it just removes the flag.  Adding to the context menu wasn't difficult and getting it to trigger a removal routine wasn't either.

However, having got that far, I am now stuck !!  Every time that routine gets called, Kodi locks up completely.  Well, not quite completely as some threads that autodelete get reported in the log.  The gui however is frozen, including all the debugging info.  The lockup is triggered by a call to update the album table.
cpp:
bool off = false;
strSQL=PrepareSQL("UPDATE album SET bSet = %i WHERE idAlbum = %i", off, albumId);
    CLog::Log(LOGNOTICE, "%s - query [%s]", __FUNCTION__, strSQL.c_str());
    if (!ExecuteQuery(strSQL))
      return false;
    return true;

I can't see anything wrong with it but it freezes kodi every time.  Suggestions most welcome.
Learning Linux the hard way !!
Reply
#53
(2019-08-26, 22:42)black_eagle Wrote: This was wrong!  In quite a few cases, albums were added where the first disc has no name, but the second/third disc is named.  Quite often as 'Bonus disc' or 'Live'.  I decided to alter the logic a little so that an album is only added as a set when at least two discs are named and one of them is the first disc.  

I can't see anything wrong with it but it freezes kodi every time.  Suggestions most welcome.

While that may work for you, it wont work for a lot.

As mentioned previously theres things like a lot of Bob Dylans releases where Disc 1 (or disc 1 & 2) are the original album, and have no titles.

Image


It depends what you are calling a box set.. if its a multi disc release such as Bob's bootleg series then it would in my mind be a box set. 

same situation with McCartney and the Beatles. Multi disc deluxe editions being released (mainlyt the 50 year copyright protection stuff) with disc 1/2 being the original album (and usually on MB without a subtitle..)

although, interestingly, in the last few weeks someone has gone and filled out missing subtitles on all the beatles/macca stuff that i had seen with missing titles

Image

but, you will be rather at the whims of MB editors as to wether that first disc has a subtitle or not

and what of situations like below, where its a 4 disc compilation, but no disc subtitles.. 

Image




It seems the way you have set it will only apply to "box sets" that are box sets in their truest sense (ie. box sets containing x number of albums from the artists discographY)

of course, just to throw a bit more fat on the fire.. Ani Di Franco's two disc album Revelling / Reckoning would become a boxset.. Big Grin

Image





Personally, I would rather have the odd alien album (such as R/R) appearing in Box sets that probably shouldn't be there, than albums missing from Box sets that should be there..

My personal thought would probably be that if its 3 discs and above, its a box set. Thats roughly equivalent to 5 LP's

2 CD's its a double album or a CD with bonus material. I would not consider any 2 disc release a box set. But the vast majority of 3 CD / 5 LP releases I would
Reply
#54
@black_eagle I think you are going to have to be a bit smater with how you derive what things are boxed sets from the tagging.

EDIT:
Having slept on it, maybe handle it in a similar way to the album compilation flag.
Set if
1) album has multiple discs, and all the discs have subtitles (in all the music files for the album), and they are different for each disc.
or
2) the MUSICBRAINZ ALBUM TYPE/RELEASETYPE tag of all the music files includes "Boxedset" (then give subtitles of "CD1" etc. to those that don't have subtitles, or copy the first per song on disc  to other songs.

Don't confuse the RELEASETYPE tag (held in album.strType) and is a string set by Picard to certyain things but could be anything, with what is held in album.strReleaseType (an internal flag of related to how Kodii handles singles as fake albums).

Users may have to do some manual tag adjustments on top what Picard burps out (based on what Musicbrainz entries users have messed up), but if they do that then Kodi meets them part way.

This can then be overwritten by the value in album.nfo
 
(2019-08-26, 22:42)black_eagle Wrote: At this point, I decided (and oh, how I wish I hadn't right now !!) that it would be a good idea to look at how the context menu works and to see if I could add an option to remove a set.  This doesn't actually remove anything from the database, it just removes the flag.  Adding to the context menu wasn't difficult and getting it to trigger a removal routine wasn't either.
Maybe not record deletion but having Kodi change what it holds that is derived from tags may not be a good thing, for example what happens next library update?

EDIT:
Also why have user rating and boxed set as the only album properties that can be set via GUI? So piecemeal. Why not have a way to set all the scraped album values from your armchair? The 10ft TV remote UI may not be the nicest edting tool, but for some it is easier than wrangling nfo files into an editor on some other computer.

But that aside see if I can get you unstuck despite my misgivings on the set removeal facility
(2019-08-26, 22:42)black_eagle Wrote: However, having got that far, I am now stuck !!  Every time that routine gets called, Kodi locks up completely.  Well, not quite completely as some threads that autodelete get reported in the log.  The gui however is frozen, including all the debugging info.  The lockup is triggered by a call to update the album table.
Called from a context menu on what?  At a guess is it a problem with the db connection not being open at that point? Or that you are using the same dataset twice at the same time (notice m_DS and m_DS2).

EDIT
But still not keen on the context menu edit boxed set  flag approach
Reply
#55
Setting album properties via GUI....
At very least we need a button on the album/artist info dialog to export just that the album/artist, that would ease some of the manual messing.

No idea how much opposition/support there would be for editing scraped album/artist properties via GUI. You can do it via Chorus, and any plugin could use JSON to do it. Is Kodi GUI a step too far? Are there alreday nice nfo file management tools?
Reply
#56
Sorted the db issue.  It wasn't my code after all, but simply that I had left the db open in an sqlite editor.  Kodi will have been restarted at some point, but the OS let it read the db, it was only when it came to writing to it that the OS blocked it and made it wait for the editor to release the db.  Guess that was a combination of being tired and thinking I must have written some bad code somewhere.  Slept on it and realized almost as soon as I woke up what the issue was.

I don't think using the Musicbrainz Release Type tag is in any way useful for this. For instance, https://musicbrainz.org/release/b22a78aa...46f4012842 is undoubtedly a boxset but is tagged as 'album / compilation'.  Same for Queen - Greatest Hits I, II & III.  And Floyd's A Tree full of Secrets.  The boxset for The Joshua Tree 30th Anniversary Deluxe box set is tagged as an album.  So unless I am looking at the wrong tag, or there is a tag missing from my files, then sadly I don't see it as being a solution.

There isn't currently a simple solution to determining a 'boxset' as I have some 3 CD sets that I wouldn't class as a boxset, even though the discs are named via MB.  Chilled Euphoria is one (https://musicbrainz.org/release/72e164b3...f40af6f6dd).  That album meets the criteria however you set them.  All discs named - yes.  More than 2 CD's - yes. Boxset - for me, no.  Perhaps the way around this is to consider the albumartist, in the same way Dave does for determining compilations.  Albumartist for that album is 'Various Artists' so it could be excluded in that case.  I'd need some feedback on that to see if that is indeed viable.

Anyway, for someone to turn an album into a set, it's easy enough with tagging software to add the required tags even if the data from mb doesn't currently include all the disc names.  Even when it does, it isn't necessarily what someone might consider a boxed set (see above).  However, it's not quite so easy to remember which albums suddenly popped up as sets when you weren't expecting them to and then go and remove those tags, remove the albums from Kodi and then re-scan them.  That's why I decided to add a remove set option.  It's easy enough to use, the info is all in front of the user, it's non-destructive and its quick.

If you update the library, then choosing to not do a full tag scan on unchanged files I would think would be sufficient to avoid Kodi re-adding them to the sets, although I have yet to test that theory.

Image

Image

Image

Image
Learning Linux the hard way !!
Reply
#57
(2019-08-27, 23:08)black_eagle Wrote: I don't think using the Musicbrainz Release Type tag is in any way useful for this. For instance, https://musicbrainz.org/release/b22a78aa...46f4012842 is undoubtedly a boxset but is tagged as 'album / compilation'.  Same for Queen - Greatest Hits I, II & III.  And Floyd's A Tree full of Secrets.  The boxset for The Joshua Tree 30th Anniversary Deluxe box set is tagged as an album.  So unless I am looking at the wrong tag, or there is a tag missing from my files, then sadly I don't see it as being a solution.

 

potentially the tag feature on MB, but its seen very limited takeup for "boxset" and theres then the problem you would need to get that into the tag to be read. 

Image

At the moment I think picard only supports tag > genre

Image
Reply
#58
@dm21912 Would that it were that simple !!

Unfortunately, as there appears to be no hard and fast rule at MB for people to follow, there are tags of "box", "box set", "boxset" and "box-set".  Occasionally, there is a Genre of boxset but most albums seem to have one of the versions as a custom tag.  There is a box on the tags page of a release where users can add tags to a release.  I guess this is to make the website easier to search as none of the albums I've loaded into Picard show any sign of such a tag (apart from where it is included in genre).  No official tag is mentioned in the documentation -> https://picard.musicbrainz.org/docs/tags/ either.

The best thing I can suggest at the moment is a set of rules that will produce a result that most people will be happy with, however I'm well aware that there is no 'one size fits all' and that in some cases albums won't be added that should be (there is a 32 disc box-set of Rachmaninoff that has no disc titles at all) or that a few may be added that shouldn't be.

The way it stands at the moment, the Rachmaninoff set could be fixed by adding titles to the discs when tagging the files.  Not ideal I know, as if you re-tag it at some point you would lose the titles unless you added them to MB.  I have to try and work within whats available though.  The releasetype is of no help as it can only be Album, Single, EP, Broadcast or Other.  Secondary type can be Compilation, Soundtrack, Spoken word, etc ,etc (there is a list at https://musicbrainz.org/doc/Release_Grou...mary_types) but no box-set or similar so there is no way to derive a box-set from that.
Learning Linux the hard way !!
Reply
#59
@black_eagle quite a few things I want to discuss or comment oh here, could be a big post do try to keep track.

1) Add/remove boxed set facility
The db lockout you experienced brings up another point about how db access is done. The team is trying hard to improve the design to avoid UI (especially player) lock ups. It was easy in old xbox days, that automatically did the UI on a separate thread, so there are a lot of fundamental flaws in this area of  design that are results of that history. Part of this means that db access must be done on a separate thread to the UI, so if for some reason (network issues, db locked by something else etc.) db access is delayed the user can still move the mouse and cancel what they started. There are still places that lock like that, but you will get jumped on if you accidentally add a new one. Context menu processing that includes a SQL update needs to be done on some other thread from the GUI. Such an update is normally almost instant so you don't think about it, but it has lockout potential.

I am still not keen on that add/remove boxed set feature (even done off the UI thread). I want ease of use for users, sure, but providing a quick UI route for one kind edit and not all the others (which may be equally vital to some other user) makes for yet more inconsitency rather than less. For example genre tagging can be a mess, and users so wish they could tweek that mis-spelled value rather than have to visit the tagging again, or that album that didn't get flagged as a compilation etc.

I know why looking at the diversity of the way sets could be tagged and  how hard it is to untangle which are sets, you are attracted to this. It seems so simple... tidy up the mistakes via the context menu. But think big picture too, more items on the context menu is not an expandable approach to fixing album data - the properties can't have  a menu item each. Even rating, the only other property set via GUI, is not a context menu item.

Editing album properties, if Kodi is going to do that, is better via something like the album info dialog - an approach that could be extended for more than just one property and that dialog already has the db reading on a separate thread.

Library updates that rescan an album will lose manual edits made to properties that can be derived from tag data. If the timestamp/size hash changes on any of the music files from an album that has been manually add/remove as a boxed set then that will get reset (unless you take steps to avoid it). The timestamp can be affected on some platforms by clock change, and some other media played edit the files/folder too, so library update can result in unnecessary and unexpected rescans.

2) Determining what is a boxed set
OK, so what comes out of Picard can be very diverse. Does this mean you want to abandon using tags to determine if something is a boxed set all together? Have "boxed set" as an arbitary custom property that users set via NFO or suitable GUI? You could still pick up DISCSUBTITLE from tagging, but nothing special is done with that other than store it like any other song property.

Or do we go with "is a boxed set" being something determined from tagging and leave it to the user to tag sufficiently to achieve the end results they want. Tagging with Picard may give the user a good start with that, but inevitably they may need to do some manual edits to tidy up the more messy Musicbrainz data entries. 

My suggestion of using RELEASETYPE tag to help determine a boxed set was not because Musicbrainz already have a "boxedset" entry (no they don't), but because it was a custom tag (not the ID3 standard) already added by Picard and used by Kodi. It is an easy place for a user to edit this tag and add "boxedset" and then have Kodi use it intelligently. It is an easy way when the DISCSUBTITLE values are mixed, etc. for a user to set one tag and have Kodi take it as a boxed set regardless of other algorthims.

What I would like to avoid is yet another album property that is lost half-way between one approach and the other. We have a few of those already and they can cause issues later. For example year - set for both songs and album initially from the tag values, but album year can be edited via scraping album.nfo or JSON API and then differ from song year. What gets used for filtering? An "edit year" context menu would make matter much worse and end up with more users with inconsistent year data. But I digress. Devil

At the moment my preference for "Boxed set" to be a fact about music files. That is to derive  it from tags using a clear methodology but allowing for what Picard produces to be a nearish miss that is easy to edit,  and get the user to set those tags according to the result they want.

You can make a case for it being a custom album property instead, but then I might suggest that you implement a generic custom property facility while you are there as well as a property editing UI.  Devil


I guess I find editing tags much easier than messing wth nfo files, and data that I may forget to export to save manual edits. Let's not get too bogged down in the imperfections of the Musicbrainz entries. We could even hope that if Kodi had better boxed set handling then some users would go back to MB and fix the more messy entries in their database. Flaws would be more obvious if the data was being used.

More seriously @black_eagle can you see what I am getting at? I think what is needed is some  clear vision if this feature is about organising songs based on music file tagging, or about selecting certain albums.
Reply
#60
I have not read the whole of the thread so apologies if this has been mentioned, but regarding determining what is a Set, why not use the TPOS tag? that's it's role according to id3v2.3.0 see http://id3.org/id3v2.3.0#Declared_ID3v2_frames where it's defined as 4.2.1 TPOS [#TPOS Part of a set] so if TPOS is filled create a Set.
Reply
  • 1
  • 2
  • 3
  • 4(current)
  • 5
  • 6
  • 24

Logout Mark Read Team Forum Stats Members Help
Managing music albums Box-Sets properly with KODI library feature0
This forum uses Lukasz Tkacz MyBB addons.