[Uatschitchun]

Hi there, I want to re-organize my music sources and need a little help with it.

My current situation:
I've got 2 main sources:

• Music
  
• Audio books
  

For to not mix them up, I've got 2 profiles setup in kodi and use 2 separate DBs (MariaDB). All fine, but needs profile switching.

Together with that, my Music source is added as only one source, although I've got my music pre-sorted in major genres on my hard drive (like /mp3/blues, /mp3/reggae, etc) and want to make use of the new sources capability of v18.
Same goes for Audio Books, as I've got audio books and radio plays in separate folders, so I would be able to use the source feature there, too.

For convenience, I use Yatse. But maintaining 2 profiles needs 2 hosts setup in yatse for each client, making it 4.
As a sidenote: as I also have 2 separate Movie DBs and mix them up on 2 RPi3 with all in all 3 different profiles:

• Profile 1: Music + Movies A
  
• Profile 2: Audio Books + Movies A
  
• Profile 3: Music + Movies B
  

So within Yatse I'm not able to re-use any of the 4 "hosts" (because 2 boxes with 2 profiles each). That leads to sync 4 hosts within Yatse independently...

My goal:

I want to have only one audio DB and will separate Audio Books and Music by creating separate nodes (easy, as they differ in path).
So no need in switching profiles anymore :-)
Music and Audio Books are accessible from only one profile.
Advantage would also be, to be able to setup a headless kodi instance on my server for scraping and syncing (esp. Yatse) Audio & Video with only one profile.
The headless server could then be synced to Yatse (a lot faster than RPi3!) and within Yatse I'm able to re-use this host for the other 2 hosts (RPi3).

The way to go?

My way to go would be:

• Setup an 'Artist information Folder' (AIF) for each source (Audio Books & Music)
  
• Export each source to AIF
  
• Copy the 2 AIFs to only one
  
• Setup a new kodi instance with proper sources (/mp3/blues, /mp3/reggae..., /audio/books, /audio/radioplay)
  
• Assign the new AIF
  
• Import
  

Or should I export both (Music & Audio books) into single files and import both files on the new instance?

I want to loose as few data as possible.
Music & Audio books are tagged (90%) with Musicbrainz, the rest is tagged with proper Id3. Both, Music & Audio Books are scraped already and have fanart (Artist Screensaver) available.

So my main question is:
How to best do the export of 2 separate DBs with 2 sources and import them into only one DB with around 12 sources?

What would be your way to go? Have you already done something like this?

Thx in advance!

[DaveBlake]

Haven't heard from you and your big library for a while @Uatschitchun hope all is well
I'm sure that the filtering by music source facility of v18 will suit you well. Also Yatse offers source constant filtering across all music tabs e.g. you can pick a source and then see artists, albums and songs for just that source and apply smart filtering on top that too. And yes no need for separate music and audio book profiles, which are just so tedious to switch between.

I my upgrade to v18 I didn't quite have to do the conversion to do that you have because I just had not added all my music into the library before because I didn't want it all mixed together. Now I have it all, and use Yatse to switch which source I'm looking at (or see all sources).

You didn't ask but I know with your size of library it would be nice to avoid re-scanning all the music files, unforunately the manual data editing to achieve the sources rearrangement could get quite knotted so it is probably best to just accept that part will take time to run. But you are absolutely right, you can use export/import to avoid needing to scrape online for additional info and art for artists and albums, and also to keep your song playback history and user rating values. The latter can now be exported/imported too.

Of course make a backup copy of your current MyMusic72.db files (or MySQL/MariaDB equivalent) first. I also am vague on how audiobooks may fare - feature added by someone else with no discussion with me so I tend to ignore it exists (I also don't have any books).

Export/import of additional info for artists and albums will work using either single file or NFO files, really up to you. Artist artwork will need to go in folders in an AIF, so perhaps artist.nfo files is easy with those too. Song playback history is to/from a file, on import it matches by artist, title and mbid so can cope with the music files being in a new location.

I think I would probably do the music first, then add the audiobooks, but no real reason for that approach.

So having created exports from the single music source instance, create a new Kodi instance add the separeate music sources and scan music files into library. You can either set the AIF first and enable "fetch on library update", so that artist nfo and art it picked up, or use "query info for all" afterwards, or just import from the single xml export files - all work equally well.

Let us know how it goes.

[Uatschitchun]

Yes, pfuh ... it's a challenge, esp. with a larger library

Sadly it's not possible to give Kodi just an empty config-dir with an edited sources.xml and advancedsettings.xml (for mysql) and tell it to read in all sources available. It needs to be done by hand source by source. Or am I'm missing something here?

All my music is in quite a good shape (90% MB tagged, thx to adding a good bunch of albums to MB). The 10% that is not tagged with MB is in a good shape, too. Consistently tagged with Album, Artist, Title, Albumartist, Performersortorder, Track & CDNo., Year (if available). All covers are cleaned from border artefacts (mostly white if downloaded from google pictures) as his looks really ugly in kodi (!) and are all in 1000x1000 resolution, consistently named as folder.jpg.

Tipp: If using Linux, like me, XnViewMP is a great tool in combination with gwenview! Xnview allows for recursive file display (right click on a folder) with filter option (jpg or folder.jpg) and full screen view with black background. If you set Xnview to "always on top" and have set gwenview as the standard picture viewer, you're able to quickly browse the covers. Hitting F3 opens an image in gwenview (in background because of Xnview is "always on top"). After a while you've got all images that need work open in a separate gwenview instance, so you're able to quickly and easily crop the images then. (strg+s to save and strg+q to quit).
Prior to cropping the images, I set them to 1200x1200, as cropping is more filigran then. Something like: 

Code:

find . -name 'folder.jpg' -exec convert {} -resize 1000x1000! {} ';'

Sets all images to fixed 1200x1200. 
If all covers are nearly square, the workflow is: set them to 1200x1200, control with xnview and open the ones that need work with gwenview in background, crop or replace and set all images to 1000x1000 afterwards.


All music has been read into kodi multiple times from scratch for to check for consistencies, cover artefacts, wrong albumartistsort, etc. 

One problem I'm not satisfied with is the translation of "Various artists" albums (compilations) without MBIDs into the language kodi runs on ("Verschiedene Interpreten" in german). It adds another entry which is not needed and there's no way to set a cover with AIF as there's no entry created for "Verschiedene Interpreten" if I'm correct!?

Another problem is the diverse and inconsistent handling of compilations within Musicbrainz. Some compilations, esp. series, are all "Various artists" whereas other series do have different Albumartists, although it's a compilation (in my sense . 'Cafe del Mar' is a good example here. Having one Artist "Cafe del Mar" summarizing all compilations released would be nice, but mixing up MB-tagging with manual tagging isn't a good idea! So it seems it's something I have to live with... Or are there ideas or solutions I'm not aware of?

Two questions regarding the entries in "Sources" menu.
1.) The ordering is by order in which the sources were read in? Could this be changed to alphabetically?! The sort order is not changeable?!
2.) Would it be possible to add images to sources? I know it's possible to add images to sources in filebrowser, but would it be possible to also have images in "Sources"? Would make navigation a little more intuitive...

I'll keep posting on my journey

[DaveBlake]

1). Sources ordering. The results depend on how you define the node

Folder type node:

Code:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<node order="7" type="folder" visible="Library.HasContent(Music)">
    <label>Sources (folder type)</label>
    <icon>DefaultMusicSources.png</icon>
    <path>musicdb://sources</path>
</node>

is in the order defined in userdata/sources.xml, i.e. the order in which they were added.

While if you use a filter type node

Code:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<node order="6" type="filter" visible="Library.HasContent(Music)">
<label>Sources (filter type)</label>
<content>artists</content>
<group>sources</group>
</node>

is ordered by source name as you want.

But I agree that is a little inconsitent, so I will make a note

2) Images for sources... I suppose it is abit like images for genres, but not sure how that is done. A feature request!

[Uatschitchun]

The story goes on

In my tries to split my audio content into 'logical' parts and separating audioplays from music, I had a heads up from Dave in this thread: https://forum.kodi.tv/showthread.php?tid...pid2906626

As I have mainly 2 mainfolders (music & audioplays) and genre-like subfolders within music, I wanted to have the genre-like subfolders (Hard, Smooth, Ska, Reggae, Pop etc.) as separate sources and 2 'combining' meta-sources for music & audioplays. 
Attention here! What doesn't work is the following:
Adding a source like /media/music/ as the 'combining' meta-source and scanning that source into the DB, scanning sub-sources afterwards, like /media/music/Ska/ or /media/music/Pop/ afterwards, those sources get removed from the meta-source! The result in the end is an empty meta-source!

The way to go is to scan the sub-sources at first and create the 'meta' source afterwards. Not vice-versa!
What even doesn't work is to add sources without choosing to "scan items although the content hasn't changed"! Meaning: If you choose "scan item into library" from the long-click menu, you get asked, if you want to scan the source although the content maybe hasn't changed. If you choose "No" here, the source is not added to your sources! 

Example:

/media/music/
/media/music/pop
/media/music/rock
/media/music/classic

If you want to have 4 sources (pop, rock, classic & music), the only working way (from my experiences) is to scan in pop, rock & classic and add the source 'music' afterwards.

If you scan in music at first (because why not scan all instead of every sub-source and wait for it to finish) and the sources pop, rock, classic afterwards, you'll have pop, rock & classic as populated sources and music being empty!
Note: If you scan in a source, you have to wait until the scan is finished before starting to scan in the next source. If you don't wait, the scanning simply stops and you have to re-scan!

Nodes:
It's possible to create (folder-type!) Nodes (like with 'Library Node Editor' Addon) for music and define a source as path like this:

Code:

<?xml version='1.0' encoding='UTF-8'?>
<node order="20" type="folder">
       <label>Artists Comedy</label>
       <icon>DefaultMusicGenres.png</icon>
       <path>musicdb://artists/?sourceid=1</path>
</node>

in userdata/library/music/ as maybe pop.xml

You're able to get the sourceid's from looking at the kodi.log or by:

Code:

SELECT * from source;

if you're running MySQL.

[randallspicher]

--SNIP--

Another problem is the diverse and inconsistent handling of compilations within Musicbrainz. Some compilations, esp. series, are all "Various artists" whereas other series do have different Albumartists, although it's a compilation (in my sense . 'Cafe del Mar' is a good example here. Having one Artist "Cafe del Mar" summarizing all compilations released would be nice, but mixing up MB-tagging with manual tagging isn't a good idea! So it seems it's something I have to live with... Or are there ideas or solutions I'm not aware of?



Two questions regarding the entries in "Sources" menu.
1.) The ordering is by order in which the sources were read in? Could this be changed to alphabetically?! The sort order is not changeable?!
2.) Would it be possible to add images to sources? I know it's possible to add images to sources in filebrowser, but would it be possible to also have images in "Sources"? Would make navigation a little more intuitive...

I'll keep posting on my journey

My music is structured as ...../Genres/<genre>/<artist>/<year>:<album> - [audio-format]/song.mp3
So, I also set up a separate source for each of my genre subfolders, and view the library from Source view.  It seems reasonable that since the source itself does have an image available which shows in fileview, that the skin should be able to access it (somehow) when viewing the library by source. (I've been meaning to get around to investigating that from the skinning side of things)  For me, I have a widget displaying sources (icons) on the music home screen, and the sources show up in alphabetical order there. I've also noticed that in library under sources, for icons, the order seems to be the order they are placed in sources.xml, but alphabetical for list view.  (I also wonder if that's a skin dependent issue)

I've handled compilations as follows:
.../Genres/<compilation genre>/<Some series>/<year>ome-comp vol. 1/song.mp3
I have a source (Genre subfolder) such as "Gothic Compilations" [don't judge me].  Underneath that, I have a lot of compilation "sets" (for those with multiple volumes, some series have had over 100 volumes come out over the years).  for "one offs" I dropped a "Various Artists" directory under the Gothic Compilations dir, or I grouped them however I saw fit if they seemed similar.  Next, I used music-brainz picard to scan them all to get all the correct artists and all that metadata. Then, because I want to actually see them in my library under the pseudo-artist-set-name I placed then under, rather then all jumbled together as "various artists", I created a shell script that scans everything under the compilation directory and updates the tags to replace AlbumArtist with the Series name (from the directory name), and strip out the MusicBrainzAlbumArtist tag (because otherwise, Kodi would ignore the Albumartist and look up that various artists musicbrainzid).  Huge pain-in-the-butt, but once I got the scripts working, it's easy enough to re-scan the directory again to clean up if I touch anything with picard (which always resets everything back to various artists again)

I personally think it would be great if we could configure kodi to use a regexes on the path to extract metadata.  Would be useful for both "populate tag X from this regex to fill in when tags are not available" (for example, .dts files for multi-channel music), as well as "override tag X from this regex, ignoring whatever may be populated in the tags on the file" (for example force the genres and album artists to be based on where I dropped the album in my directory tree, regardless of MusicBrainz's opinion on the matter)

[Uatschitchun]

Ok, so let me report back

My current setup:

• On my server I have kodi running autostarted after autologin to KDE. My plans to use kodi-headless in a docker container needed to be set apart (reasons see below).
  
• Kodi is running against a MariaDB (10.4) running locally on the server, too.
  
• That kodi-server is synced with Yatse and is used to scan & clean libraries (actual manually invoked only, don't need sheduled scans, but they could be easily implemented with cron jobs), run ArtistBeef, function as a snapcast server and to play audio locally
  
• My sources are divided by my main genre folders on harddrives for audio & video, so I'm able to set up library nodes for kodi's main menu and filter with "Sources" in music and in yatse
  
• I wanted to have my main menu separated into: Music, Movies (16+), Movies (kids), Concerts, Audiobooks & -plays, Series, Live-TV. Realizing this without profiles was the hardest part (see below)
  
• Besides the server I have a Rpi4 4GB, a S905x and 2 Laptops as Clients syncing against the MariaDB and syncing the sources.xml, library/, favorites.xml and playlists/.
  
• All clients are running a snapcast-client against the server. Together with the Snapcast App I'm able to add clients or remove them from the stream the kodi on server is playing. (see below) 
  

Problems & Solutions:

• Main problem was finding a skin which is able to setup the main menu as I liked it to be. Together with that it should be slim enough for to run on Rpi and s905x. Only skin I found to fulfill these needs was Aeon Nox: Silvo. As I needed to divide the DBs into parts (music <-> audioplays, movies <-> concerts) I needed to write own library nodes and set these as main menu & sub menu entries. It basically was a complete reconstruction of the main menu with own nodes needed to achieve that! It would be nice if kodi had an option to simply set up a filter in front of a whole 'menu set', so one would be able to re-use the nodes already existing and could duplicate the movies menu and set a different filter!
  
• The self-created nodes do DB queries. That's not what kodi likes, esp. with huge libraries!! If I understood Dave correct, that's because kodi does things that would better be done on SQL side (Maria or Mysql). Each query eats up RAM, which isn't released again. Esp. addons like library-data-provider eat up RAM over time. I had to swap my Rpi3 with the Rpi4 4GB as I wasn't able to get my setup running with only 1GB RAM. Even the s905x with 2GB still hangs sometimes, although I added 
  Code:

  MALLOC_ARENA_MAX=1
  to
  Code:

  /storage/.config/kodi.conf
  and moved library-data-provider addon out of
  Code:

  .kodi/addons/
  on the s905x. It would be nice if kodi would do DB queries on DB-side as far as possible and would need less memory (or release it more strict)!
  
• Regarding kodi-headless with Docker: I started with milaq/kodi-headless (uses headless-patch, nice project with regular scans, etc.), tried Celedhrim/docker-kodi-server (uses kodi PPA with xpra, quite ressource hungry), tested linuxserver/docker-kodi-headless (uses headless-patch, used it some times and re-wrote the Dockerfile and built my own version, but it uses a linuxserver base-image which gave problems with locale-settings. Had it running for a while only as counterpart for Yatse as base host for the other clients). Lastly I wanted to have the headless server being able to play audio. That's why I tried ehough/docker-kodi (uses kodi PPA together with mviereck/x11docker - with xpra), but I wasn't satisfied with it. Xpra seems nice, as it makes it possible to connect to kodi's "head" if one wants to. Makes configuration a whole lotta easier. But it wasn't working stable and I couldn't get kodi to recognize a sound card with the modified linuxserver/kodi-headless (or the other docker containers). As my server has a display now, running kodi headless isn't that needed anymore. It would be nice if there was a kodi-headless docker image running a snapcast-server and send audio to a locally running snapcast-client. That way, using the Web-UI or Yatse would allow to play audio to snapcast!
  

So far so good

If there's anything special you'd like to know...