• 1
  • 3
  • 4
  • 5(current)
  • 6
  • 7
  • 24
Managing music albums Box-Sets properly with KODI library feature
#61
1)  I quite understand about doing db access on a separate thread to the GUI. You are right in that I didn't think about it in that context.  I also know you're not keen on the GUI option.  Yes it did seem at the time to be an easy way to tidy up some unexpected results that I got during testing but on reflection, that was probably more down to the way I had set the rules up initially rather than anything else. A GUI option seemed to be a quick solution to that, but better rules on what constitutes a set, coupled with decent tagging is a better option.
Quote:At the moment my preference for "Boxed set" to be a fact about music files. That is to derive  it from tags using a clear methodology but allowing for what Picard produces to be a nearish miss that is easy to edit,  and get the user to set those tags according to the result they want.

That is my preference too. Adding to RELEASETYPE would certainly allow Kodi to generate disc names if they are missing which would give users less messing around with tags and is a definitive method for a user to say to Kodi "Treat this album as a boxed set", perhaps falling back to the rules that Kodi follows at the moment if that tag doesn't contain 'boxset'. Not everyone will be aware that they need to update their tags initially and the ability for Kodi to generate some sets automatically without that tag might stimulate more interest in sets as a whole.

It is already possible to override Kodi's rules by either exporting the library as a single file and editing the required album(s) and then importing it, or by creating an album.nfo file, turning on "prefer online info" (which I have to say I find a bit misleading as a local .nfo file won't override anything in the tags without that on Huh ) and then rescanning the album.

Necessary info is
xml:
<boxset>true</boxset>
<disctitles>CD 1 / CD 2 / CD 3 / CD 4 / </disctitles>
 
Quote:I think what is needed is some  clear vision if this feature is about organising songs based on music file tagging, or about selecting certain albums.
For me, it's about the songs ultimately.  I mean, that's what I want to listen to, but getting to some of them currently is a pain in the arse.  A tree full of secrets is 18 discs and kodi lists the entire thing normally.  Thats a lot of stuff to wade through if you're looking for a specific track.  Much easier when MB has split it into 10 volumes (vol 10 is a data cd) of two discs each.  It's the same with other stuff I have.  I might want to just listen to disc 3 of something but then I have to look through a list of stuff I don't want, to find bit I do.  Also, aesthetically it looks better (or I think it does) and nearer to what you would once have had in your hand.

The fact that we start with an 'album' is because that is the container for the songs.  Splitting that container into more manageable chunks (whether that's because it was physically like that originally, or just because it makes it easier to manage a large amount of tracks in that container) just makes more sense to me. 

@jjd-uk TPOS contains the disc number and total discs for a multi-disc album.  It's impossible to determine from that alone if the album is indeed a box-set or just a multi-disc compilation or an anthology.
Learning Linux the hard way !!
Reply
#62
(2019-08-28, 15:23)jjd-uk Wrote: I have not read the whole of the thread so apologies if this has been mentioned, but regarding determining what is a Set, why not use the TPOS tag? that's it's role according to id3v2.3.0 see http://id3.org/id3v2.3.0#Declared_ID3v2_frames where it's defined as 4.2.1 TPOS [#TPOS Part of a set] so if TPOS is filled create a Set.

the problem with TPOS is (as I understand it) it will likely apply to double cd releases that arent "box sets" as such but just double cd releases. 

For instance, Revelling/Reckoning would likely be tagged as TPOS 1/2 2/2 

https://musicbrainz.org/release/4f5ccefc...429faa1c83
Reply
#63
@black_eagle you are right that it will help for users to see some boxed sets, even messy ones, to generate awareness and interest. I also wonder where this may lead longer term another reason for wanting to stay away from short term GUI additions - aim higher!

(2019-08-28, 16:21)black_eagle Wrote: It is already possible to override Kodi's rules ... or by creating an album.nfo file, turning on "prefer online info" (which I have to say I find a bit misleading as a local .nfo file won't override anything in the tags without that on Huh ) and then rescanning the album.
No need to rescan (tags/library update) to apply an NFO, if you hit refresh button on the album info dialog and it finds an album.nfo v18 prompts you if you want to use that or check online. Not sure if you knew that and meant "scrape" not scan here or not. A batch art refresh facility would be nice (you can batch rescrape but the newly fetched art does not get applied to items that have art).

I have plans to nuke that "prefer online info" setting. It was created for one purpose that was never fully implemented, got adapted for another use (original purpose forgotten), I came on the scene and attempted to sort out the issues it caused (after scraping had replaced artists on several of my classical albums because it misidentified them!) and.... well it is just misleading/confusing.

What users need is a better way to control how values from nfo (and possibly online scraping too) are applied to existing data (possibly initially derived from tags, or previously scraped) - merge or overwrite which bits? No one understands what "prefer online info" does (nothing very helpful) so replace it with something more useful.

(2019-08-28, 16:21)black_eagle Wrote: Necessary info is
xml:
<boxset>true</boxset>
<disctitles>CD 1 / CD 2 / CD 3 / CD 4 / </disctitles>
Disc titles in album.nfo? Isn't that really song data? Album has boxset flag, the songs have disc number (in with track) and discsubtitles (OK repeated for songs on disc, but denormalisation is fine for access). Either tag those subtitles in the music files, or put up with "CD<disc number>" being applied when they are missing. We can discuss if they need denormalising into the album table too for ease/speed of use, but I would avoid setting them via nfo.

But you are right, it is about getting to songs in digestable chunks e.g. pick and playback of one disc from a set of 18.
I hope my comments are useful, and not driving you nuts yet. Smile
Reply
#64
(2019-08-28, 17:16)dm21912 Wrote:
(2019-08-28, 15:23)jjd-uk Wrote: I have not read the whole of the thread so apologies if this has been mentioned, but regarding determining what is a Set, why not use the TPOS tag? that's it's role according to id3v2.3.0 see http://id3.org/id3v2.3.0#Declared_ID3v2_frames where it's defined as 4.2.1 TPOS [#TPOS Part of a set] so if TPOS is filled create a Set.

the problem with TPOS is (as I understand it) it will likely apply to double cd releases that arent "box sets" as such but just double cd releases. 

For instance, Revelling/Reckoning would likely be tagged as TPOS 1/2 2/2 

https://musicbrainz.org/release/4f5ccefc...429faa1c83

Have read a bit more and realised I perhaps have missed the point of this, I had assume it would be a general sets feature so used for displaying all albums with multiple discs, but it seems this has perhaps a more specific target of where several albums are grouped within one package, being restrictive like that seems like a wasted opportunity to me.

(2019-08-28, 16:21)black_eagle Wrote: @jjd-uk TPOS contains the disc number and total discs for a multi-disc album.  It's impossible to determine from that alone if the album is indeed a box-set or just a multi-disc compilation or an anthology.

Why not make a node for any album containing multiple discs? then for those that want what just a box sets a filter can be set based on DISCSUBTITLE perhaps controlled by a setting.
Reply
#65
(2019-08-28, 18:31)jjd-uk Wrote: Why not make a node for any album containing multiple discs? then for those that want what just a box sets a filter can be set based on DISCSUBTITLE perhaps controlled by a setting. 
Simply multiple discs would scoop up far too many things that are split purely based on the limitations of physical media. I think we need to be a little more discerning, and there is the tag data to make that possible.
Reply
#66
Probably because I have nothing that would fall within the narrow definition being used here.
Reply
#67
(2019-08-28, 18:31)jjd-uk Wrote:
(2019-08-28, 16:21)black_eagle Wrote:  

Why not make a node for any album containing multiple discs? then for those that want what just a box sets a filter can be set based on DISCSUBTITLE perhaps controlled by a setting. 
Would include a huge amount of non-box set releases (certainly for me anyway)

Ive probably got 100+ releases that are two disc but not box sets (some of these releases can have as few as 2 tracks on disc 2 if they are mixed media discs)

whereas ive probably only got 30 releases that are true "box sets"
Reply
#68
(2019-08-28, 21:01)jjd-uk Wrote: Probably because I have nothing that would fall within the narrow definition being used here.

I don't think its that narrow.

Box sets can include

+ Multi disc re-releases of a number of albums in one collection (the traditional box set)

+ Multi disc re-releases of previously released albums with multiple bonus discs and extra material (things like Sgt Pepper or White album re-releases)

+ Multi disc releases of entirely new content (things like the recent Bob Dylan Rolling Thunder releases)

+ Multi discs greatest hits/anthologies (like the repackaged Queen I/II/III or releases like Bob Dylans "Biograph")


But what likely wouldnt be considered a box set is 2 disc "deluxe" editions of an album with the album on disc 1 and some bonus audio on disc 2.

Of course everyone has their own perspective on what they consider a threshold for becoming a box set, and a lot of it can depend on the content of the release.

For instance, i may not consider the 2 disc deluxe edition of GYBR a box set, https://musicbrainz.org/release/a5c6d485...bd290fb651 but by the 4 disc I do.. https://musicbrainz.org/release/816ab66c...f4d957559d but someone else could think totally differently.
Reply
#69
@dm21912   Having taken on board @DaveBlake 's comments regarding the albumreleasetype, I have re-visited this area and made some changes.

So, if you add 'boxset' (no quotes) to that tag with Picard, Kodi will automatically turn it into a boxset.  It will create disc names for any discs that do not currently have them of the form 'Disc X' where X is the disc number.  If any of the discs are already named then Kodi will use that name.  For example if the last two discs are named but the first two aren't, then kodi will generate 'Disc 1' & 'Disc 2' and use the tag names for discs 3 and 4.

If you don't add the tag then Kodi will create boxsets where

- all discs have titles &
- there are 3 or more discs &
- the album isn't marked as a compilation in the tags

If you do add the tag, the rules don't apply and albums tagged as compilations can be added to boxsets if you so wish.  You can also add a twin disc album as a set if you really want to.
Quote:Disc titles in album.nfo? Isn't that really song data?

It's both.... Laugh  There is an extra level in play for a boxset, which isn't there for an ordinary album.  Normally, an album just contains tracks and there is no intermediary bit to navigate but in the case of a boxset, the 'album' contains discs and the disc title becomes the link between the top container (album) and the tracks.  So yes, an album needs to know which discs belong to it and songs need to know which disc they belong to.  Your comments are indeed useful Dave.  I'd probably not have added the boxset tag without them so keep them coming.  In fact, I'd probably never have got this far if I'd just set off on my own and various people hadn't chipped in with comments and advice etc.
Learning Linux the hard way !!
Reply
#70
Let's talk some relational data design Smile

A normalised logical view would be as flollows:
An album comprises of 1 or more discs. A disc belongs to 1 and only one album.
A disc has one or more songs. A song belongs to one and only one disc

Album  -- PrimaryKey = idAlbum
 |
^
Disc   -- PrimaryKey = idAlbum, iDiscNumber
 |
^
Song -- PrimaryKey = idSong, Secondary Keys = {idAlbum}, {iDiscNumber}

We want to indicate that an album is a "Boxed set", we could get which albums have more than one disc and that they have subtitles by querying but since we are also allowing for the music files to just flag they are from a boxed set too, this is best held as a field in the album table.  You could hold it against each song (or disc) too, but since it is common across all songs on an album normalisation has it in album.

The only data a Disc has is DISCSUBTITLE and many albums have only one disc, hence denormalisation for ease of access makes lots of sense.

So denormalising and fitting to the current physical schema:
Add bBoxedSet field to the album table.
Add a strDiscSubtile field to the song table where disc  number already held. (We may also want to separate iDiscNumber from iTracknumber but I'm less clear on that as combined it does make for easy one field sorting).

To get the disc subtitles, or count of discs for an album you query the song table using idAlbum
It is also easy to get the disc subtile of a song from the song table.

It is much easier to use GROUP BY and DISTINCT to get disc subtiles from song table, than to decode some combined string of concatenated subtitles held in a field in the album table. I'm not sure what you have done @black_eagle but the <disctitles> in NFO idea got me wondering.
Reply
#71
Well, the last time I had anything much to do with databases, was the mid 80's when I was writing CoBoL on an HP3000 mini system to produce various reports for the accounts department of a large company.

Anyway, your deduction was correct Dave, and I did have a string of concatenated subtitles in the album table.  This is no longer the case.  The only new field in there now is a bSet (Which I'll probably rename to bBoxedSet to make it clearer and fit the other fields better) which is a bool to indicate if an album contains a set or not.

There is one extra field in the songs table, which is strDiscSubtitle.  So now, to construct the discs of a set, I query the song table for the DISTINCT titles (I honestly didn't realise in the beginning that I could do that - clever stuff this SQL) and vector them. I then get the album using idAlbum and construct the discs by looping through the vector and adding the album N times, using
cpp:
pItem->SetLabel(discTitle.c_str());
to set the name and
cpp:
pItem->SetPath(path);
to set a URL.  The URL is of the form
xml:
'musicdb://boxsets/%i/%s/', idAlbum, strDiscTitle
and is used by GetBoxedSetSongs() to get the songs for an individual disc.

Thinking about it, do I need to add anything to the SQL query to ensure the disc titles are always returned in the right order ?  They all are in my tests so far, but I was wondering if it was possible for them to be returned in an arbitrary order and if so, what I could do to prevent that.  So far the query is
cpp:
    strSQL=PrepareSQL("SELECT DISTINCT strDiscSubtitle FROM song WHERE song.idAlbum = %i ", idAlbum);

Time permitting, I'm going to try and get it up on git over the weekend when I've removed some of the logging I no longer need, and tidied the code up a little.

Oh, just as an aside, I ran it last night against a larger subset of my music and it found a Marillion boxed set of Misplaced Childhood that I had completely forgotten that I have which I was pleased to see confirms the auto-adding logic. Smile
Learning Linux the hard way !!
Reply
#72
(2019-08-30, 11:14)black_eagle Wrote: Thinking about it, do I need to add anything to the SQL query to ensure the disc titles are always returned in the right order ?  They all are in my tests so far, but I was wondering if it was possible for them to be returned in an arbitrary order and if so, what I could do to prevent that.  So far the query is
cpp:
    strSQL=PrepareSQL("SELECT DISTINCT strDiscSubtitle FROM song WHERE song.idAlbum = %i ", idAlbum);
Without an ORDER BY statement strictly the order of the results of a SQL query are undefined (could be anything). In practice what you often get is the natural order like you are experiencing but it is not guarenteed. So yes, add an ORDER BY to ensure disc order is used.

SELECT DISTINCT strDiscSubtitle FROM song WHERE song.idAlbum = 9853 ORDER BY iTrack

The SQL used by Kodi the ORDER BY has often been ommited, the sorting is done (slowly and using a flawed algorithm) in memory by manipulating the list of fileitems, or the order of the results set is irrelevent. I would like to love that into the db where it belongs (and would be so much faster etc.). Anyway it is why you will see so few ORDER BY statements in the code
Reply
#73
(2019-08-28, 22:47)black_eagle Wrote: @dm21912   Having taken on board @DaveBlake 's comments regarding the albumreleasetype, I have re-visited this area and made some changes.
So, if you add 'boxset' (no quotes) to that tag with Picard, Kodi will automatically turn it into a boxset.  It will create disc names for any discs that do

First of all, I can't find a reference from @DaveBlake talking about a tag called "albumreleasetype". Can you point me there, please?

There is no tag content "boxset" in use at musicbrainz, and also there is no tag "albumreleasetype". You can check what's used where if you try editing any boxset at their website. Musicbrainz uses "MUSICBRAINZ_ALBUMTYPE" (in the field secondary album type under releasegroup) and so-called boxsets are tagged as "Compilation" there. Only in conjunction with the existence of "DISCSUBTITLE" do we, the users, deduct that it is a boxset and evaluating both sets of tags would probably the way to go programatically as well.

A general request just in case: Please do not make up or require new tags. Many users don't even use Picard but use plug-ins for their favorite software, like foobar2000. Kodi needs to work with existing tags and not force the user to create new ones. Thanks!
Reply
#74
(2019-08-30, 16:49)HeresJohnny Wrote:
(2019-08-28, 22:47)black_eagle Wrote: @dm21912   Having taken on board @DaveBlake 's comments regarding the albumreleasetype, I have re-visited this area and made some changes.
So, if you add 'boxset' (no quotes) to that tag with Picard, Kodi will automatically turn it into a boxset.  It will create disc names for any discs that do

 You can check what's used where if you try editing any boxset at their website. Musicbrainz uses "MUSICBRAINZ_ALBUMTYPE" (in the field secondary album type under releasegroup) and so-called boxsets are tagged as "Compilation" there.    
you are correct

MB uses Musicbrainz_albumtype

although I would disagree with the assertion that boxsets are tagged as compilation

I only have 5 boxsets out of up to 50 that are "compilations"

Paul Weller - Hit Parade
Lou Reed - Between thought and Expression 
Bob Dylan - Biograph
Lennon - Anthology
The Pogues - Look them in the eye and say Pogue Mahone

The rest, the vast majority of my "box sets" are not compilations. I think you are quite at odds at what would be considered a box set (for the purposes of this thread)



But, what BE is proposing is a set criteria, ie. 3 discs, disc names present, and not a compilation as the criteria for a box set, but with the option for a user to override that by manually adding boxset to the Musicbrainz_album type tag
Reply
#75
(2019-08-30, 16:49)HeresJohnny Wrote: A general request just in case: Please do not make up or require new tags. Many users don't even use Picard but use plug-ins for their favorite software, like foobar2000. Kodi needs to work with existing tags and not force the user to create new ones. Thanks! 
No, nothing in what we have discussed will make or require new tags @HeresJohnny, but if you want to use the boxed set feature you may need to adjust tagging to get the exactly results you want because it is so difficult to ensure all possibilities are covered given the diverse ways that music can be tagged.

But the boxed set feature will not be mandatory, Kodi will keep on working exactly as before. It need not impact your use in any way.
Reply
  • 1
  • 3
  • 4
  • 5(current)
  • 6
  • 7
  • 24

Logout Mark Read Team Forum Stats Members Help
Managing music albums Box-Sets properly with KODI library feature0