Kodi Community Forum

MediaImport is not restricted to servers. Many users import strm files from Kodi Addons. I haven't checked Mezzmo but the other direct to database addons run into problems with such addons. With the plethora of streaming services, no Media server will be able to supply all the content desired by the user. Playing nicely with other addons is going to be important.

And yes the performance of Kodi's content infrastructure needs to be addressed. Right now there are at least seven persons/teams writing code to bypass it. Just wish that effort was used to addressed it. That's a lot of talent working on a problem without trying to solve it.

Cheers,
LongMan

(2021-02-21, 15:08)LongMan Wrote: [ -> ]MediaImport is not restricted to servers. Many users import strm files from Kodi Addons. I haven't checked Mezzmo but the other direct to database addons run into problems with such addons. With the plethora of streaming services, no Media server will be able to supply all the content desired by the user. Playing nicely with other addons is going to be important.

And yes the performance of Kodi's content infrastructure needs to be addressed. Right now there are at least seven persons/teams writing code to bypass it. Just wish that effort was used to addressed it. That's a lot of talent working on a problem without trying to solve it.

Cheers,
LongMan

It would be easy enough for me to modify the addon to create a strm file for the Mezzmo servers it discovers and provide the content URL etc..  for MediaImport.  Currently the addon does not create one.  Does MediaImport need anything specific or I just output a strm file for Kodi ?

To your point about one media server maybe not doing everything, I can see that.  Mezzmo does have the ability to do live stream captures (i.e. OTA, Internet etc..), has plugins for Internet radio stations and more.  I ended up writing a Mezzmo plugin for iHreat radio because I listen to some of their stations.  The plugin pulls a list of stations via their API and makes all of their stations across the globe available and searchable.  That was a real effort since iHeart has 3 APIs and none of them are really documented.  Just bits and pieces.  Lots of reverse engineering went into that effort.


Jeff

Quote:It would be easy enough for me to modify the addon to create a strm file for the Mezzmo servers it discovers and provide the content URL etc..  for MediaImport.  Currently the addon does not create one.  Does MediaImport need anything specific or I just output a strm file for Kodi ?

 
MediaImport does not need a strm file. It provides an API for syncing to the Kodi library without writing directly to the database.

I linked to 2 examples of mediaimporter addons and the description of how to add an extention to an existing addon in my previous post. plugin.video.waipu.tv is an example of an addon which has added the extention point.

Cheers,
LongMan

@jbinkley60,
How far have you gotten with adding MediaImport support?

Cheers,
LongMan

(2021-02-27, 22:15)LongMan Wrote: [ -> ]@jbinkley60,
How far have you gotten with adding MediaImport support?

Cheers,
LongMan

I started looking at it but due to other priorities didn't make much progress.  Aside from the code your provided is there some place where there is more documentation on the integration ?   Maybe because I don't run MediaImport I am missing how integration with the Mezzmo addon would work since it already imports all Mezzmo data into the Kodi database and plays nice with other data so it peacefully coexists with anything else, including any data which MediaImport adds to the database from other sources.  It verifies separation of the data by keying off of the path table entries.  The Mezzmo addon tracks its path table entries and only touches database entries which join to that data. 

I started picking around looking at the JSON RPC documentation but most of it seems to be for controlling Kodi interface functions and I can't find much on using JSON RPC to add, change delete data in the Kodi database.  Lastly, I thought MediaImport will find and discovers uPNP servers so I would think it will find a Mezzmo server and use its normal uPNP import methods ?  For the Mezzmo addon there are a few special calls in Mezzmo which allows the addon to leverage a bulk import from Mezzmo vs. walking the database tree but generic uPNP browsing of the Mezzmo data works with any uPNP client.


Jeff

(2021-02-28, 00:15)jbinkley60 Wrote: [ -> ]I started looking at it but due to other priorities didn't make much progress.  Aside from the code your provided is there some place where there is more documentation on the integration ?   Maybe because I don't run MediaImport I am missing how integration with the Mezzmo addon would work since it already imports all Mezzmo data into the Kodi database and plays nice with other data so it peacefully coexists with anything else, including any data which MediaImport adds to the database from other sources.  It verifies separation of the data by keying off of the path table entries.  The Mezzmo addon tracks its path table entries and only touches database entries which join to that data. 

Media import would take care of this for you. You wouldn't have to write directly to the Kodi database (which is not well received when it comes to support) and you wouldn't have to take care of separating your media data from the rest of Kodi's database / library. The idea is that the add-on provides a way to connect to the media provider (Mezzmo server) through some custom settings and then the user can specify which media types (movies, tvshows, musicvideos) to import. Then Kodi calls the add-on to provide all media items of a specific media type. The add-on retrieves the items and passes them back to Kodi which takes care of synchronizing them into the database. Obviously there are some advanced concepts like partial synchronization etc. but this should give you a general idea.

(2021-02-28, 00:15)jbinkley60 Wrote: [ -> ]I started picking around looking at the JSON RPC documentation but most of it seems to be for controlling Kodi interface functions and I can't find much on using JSON RPC to add, change delete data in the Kodi database.

Media import has nothing to do with JSON-RPC.

(2021-02-28, 00:15)jbinkley60 Wrote: [ -> ]Lastly, I thought MediaImport will find and discovers uPNP servers so I would think it will find a Mezzmo server and use its normal uPNP import methods ?  For the Mezzmo addon there are a few special calls in Mezzmo which allows the addon to leverage a bulk import from Mezzmo vs. walking the database tree but generic uPNP browsing of the Mezzmo data works with any uPNP client.

I haven't used the UPnP import for a while but the main problem with non-Kodi UPnP servers is that they often don't implement the Search functionality which is necessary to retrieve all media items of a specific media type. I had to add this functionality to Kodi when I started with media import because it didn't support it either. Furthermore UPnP is limited in the type of metadata it can provide.

@Montellese,
I was looking at the Emby and Plex mediaimporters and it looks like you isolated the code that is specific to the servers. Am I getting that correct? If so, maybe @jbinkley60 could start with one of those and adjust the authentication and media info to suit Mezzmo. Would that be simpler than starting from scratch?

Cheers,
LongMan

(2021-03-01, 12:13)Montellese Wrote: [ -> ]Media import would take care of this for you. You wouldn't have to write directly to the Kodi database (which is not well received when it comes to support) and you wouldn't have to take care of separating your media data from the rest of Kodi's database / library. The idea is that the add-on provides a way to connect to the media provider (Mezzmo server) through some custom settings and then the user can specify which media types (movies, tvshows, musicvideos) to import. Then Kodi calls the add-on to provide all media items of a specific media type. The add-on retrieves the items and passes them back to Kodi which takes care of synchronizing them into the database. Obviously there are some advanced concepts like partial synchronization etc. but this should give you a general idea.

Media import has nothing to do with JSON-RPC.

I haven't used the UPnP import for a while but the main problem with non-Kodi UPnP servers is that they often don't implement the Search functionality which is necessary to retrieve all media items of a specific media type. I had to add this functionality to Kodi when I started with media import because it didn't support it either. Furthermore UPnP is limited in the type of metadata it can provide.

When I say separate in the Kodi database I simply mean I track records belonging to Mezzmo vs. not.  As far as the user goes, if they are displaying data from the native Kodi front end vs. using the Mezzmo addon GUI, they won't be able to tell the difference.  I believe this is similar to how it looks like MediaImport operates.  The Mezzmo addon just simply plays nice with other data and is a good citizen.  The JSON RPC comment was simply looking at moving from the current direct SQL access to JSON RPC to populate / maintain data which is what the Wiki says is preferred.  I've had minimal luck finding any documentation on this and would have concerns about performance.  This was just part of a conversation LongMan and I were having and not really related to MediaImporter other than I originally thought that is how MediaImprotter was updating the database.    

I've done some additional reading on the Wiki page and looking at the other importer addon code.  I have a better understanding of MediaImport in that it is native to the Kodi code and not another addon.  It uses an extension point for media provider addons.  I looked through some of the Emby code and many of the various functions it has Discovery, Import etc..  are already written in the Mezzmo addon but operate a bit differently due to having a GUI, leveraging some Mezzmo specific functions and similar.  In addition there are a number of features in the Mezmo addon which are unique or operate differently.  Playcounts, bookmarks and such are passed back to the Mezzmo server for all clients to leverage.  Likewise you can mark watched and unwatched via the GUI and it sets it in the Mezzmo database for all clients.  The GUI has view over rides so the user can remap a view to their preference overriding the default view for the media type.  The bookmarks are user adjustable for 0-30 seconds behind where you stop in case you want to rewatch the last few seconds.  As mentioned earlier, full synchronization is built-in including detecting Mezzmo database metadata changes, a daily sync process and more.  Because of all of this and other integrated functionality it would likely be better to write a new addon for MediaImporter and leverage code from the existing addon.  It's going to be a piece of work and I have a number of questions on how MediaImporter works and the interface to an addon.

I don't have time to tackle this amount of coding right now but I can pick away at it as time permits.  I'd also only do a Matrix version and not maintain 2 code versions.  I have a couple of questions which I am curious about. 

First, how are you handling uPNP server record deletions ?  I handle it in Mezzmo via a daily sync process which runs between midnight and 6AM.   The sync processes detects when nothing is playing and runs in the background.  The daily sync removes all Mezzmo records from the Kodi database and then does a new bulk import.  This takes care of deleted records.  I also detect duplicate records in the Mezzmo database.  I am unaware of uPNP servers having the ability to let you know of a deleted record so a full sync seemed to be the best option.  It also makes it nice if a client needs to be rebuilt or the Kodi DB goes haywire.  The addon rebuilds it as soon as it starts.  I am curious how you handle this ?

Second, does MediaImporter take all of the metadata from the addon vs. rescraping the data ?  If so, what format is needed from the addon ?  The Mezzmo addon gets an XML response and parses it prior to adding to the Kodi database.  In looking at the Emby importer code it looks like you might be doing a field mapping but I can't tall what the exact structure is of the uPNP metadata going to MediaImporter.


Thanks,

Jeff

(2021-03-01, 23:20)jbinkley60 Wrote: [ -> ]When I say separate in the Kodi database I simply mean I track records belonging to Mezzmo vs. not.  As far as the user goes, if they are displaying data from the native Kodi front end vs. using the Mezzmo addon GUI, they won't be able to tell the difference.  I believe this is similar to how it looks like MediaImport operates.  The Mezzmo addon just simply plays nice with other data and is a good citizen.  The JSON RPC comment was simply looking at moving from the current direct SQL access to JSON RPC to populate / maintain data which is what the Wiki says is preferred.  I've had minimal luck finding any documentation on this and would have concerns about performance.  This was just part of a conversation LongMan and I were having and not really related to MediaImporter other than I originally thought that is how MediaImprotter was updating the database.

Sorry I misunderstood the part about JSON-RPC. Since I've written most of the JSON-RPC logic a few years ago and there hasn't been much activity there I'm pretty sure that it's not possible to build your database purely through JSON-RPC and the performance would be bad as well. 

(2021-03-01, 23:20)jbinkley60 Wrote: [ -> ]I've done some additional reading on the Wiki page and looking at the other importer addon code.  I have a better understanding of MediaImport in that it is native to the Kodi code and not another addon.  It uses an extension point for media provider addons.  I looked through some of the Emby code and many of the various functions it has Discovery, Import etc..  are already written in the Mezzmo addon but operate a bit differently due to having a GUI, leveraging some Mezzmo specific functions and similar.

For some things you are also free to use GUI dialogs in media importer add-ons. Every detected media provider has a settings dialog which directly interacts with the media importer add-on. As an example the Plex media importer add-on supports authentication through a PIN. When the user clicks the (add-on defined) button to start authentication the add-on's callback handler is called which asks plex.tv for a new PIN and displays it to the user. The user then has to enter that PIN on a plex website which will automatically be detected by the add-on which then retrieves the access tokens and stores them in the settings.

(2021-03-01, 23:20)jbinkley60 Wrote: [ -> ]In addition there are a number of features in the Mezmo addon which are unique or operate differently.  Playcounts, bookmarks and such are passed back to the Mezzmo server for all clients to leverage.  Likewise you can mark watched and unwatched via the GUI and it sets it in the Mezzmo database for all clients.

These are not at all unique to Mezzmo. Media import supports a specific workflow to report changes to the imported media items back to the media provider (in your case the Mezzmo server). A media importer add-on can provide a callback for this and Kodi takes care of calling this callback with any changed media items. This works for playback, mark as watched / unwatched etc. The Emby media importer e.g. reports playback progress back to the server every few seconds and in the Emby server you can see that playback is active in Kodi.

(2021-03-01, 23:20)jbinkley60 Wrote: [ -> ]The GUI has view over rides so the user can remap a view to their preference overriding the default view for the media type.  The bookmarks are user adjustable for 0-30 seconds behind where you stop in case you want to rewatch the last few seconds.

Since a media importer add-on is just an add-on implementing a specific extension point nothing is stopping you from providing additional extension points and features. The Emby media importer e.g. provides additional custom context menu entries. In the Plex media importer add-on I'm working on integrating the skip intro feature which has nothing to do with media import except that it can be applied to the imported media items.

(2021-03-01, 23:20)jbinkley60 Wrote: [ -> ]As mentioned earlier, full synchronization is built-in including detecting Mezzmo database metadata changes, a daily sync process and more.

Media import support full and partial synchronization. At the beginning it always performs a full synchronization but then (depending on the add-on) it can automatically provide items changed on the server without a manual synchronization. Both Emby and Plex media importers support "fast synchronization" after a full synchronization which only retrieves all changed media items since the last synchronization.

(2021-03-01, 23:20)jbinkley60 Wrote: [ -> ]Because of all of this and other integrated functionality it would likely be better to write a new addon for MediaImporter and leverage code from the existing addon.  It's going to be a piece of work and I have a number of questions on how MediaImporter works and the interface to an addon.

I'm not sure whether this is true. You probably already have all of the necessary code but in a different design. I'm pretty sure you could re-use a lot of it and just put it into different places and use different interfaces.

(2021-03-01, 23:20)jbinkley60 Wrote: [ -> ]I'd also only do a Matrix version and not maintain 2 code versions.

Since media import is not part of Kodi mainline there's no need to suppor anything else but Matrix+.

(2021-03-01, 23:20)jbinkley60 Wrote: [ -> ]First, how are you handling uPNP server record deletions ?  I handle it in Mezzmo via a daily sync process which runs between midnight and 6AM.   The sync processes detects when nothing is playing and runs in the background.  The daily sync removes all Mezzmo records from the Kodi database and then does a new bulk import.  This takes care of deleted records.  I also detect duplicate records in the Mezzmo database.  I am unaware of uPNP servers having the ability to let you know of a deleted record so a full sync seemed to be the best option.  It also makes it nice if a client needs to be rebuilt or the Kodi DB goes haywire.  The addon rebuilds it as soon as it starts.  I am curious how you handle this ?

The UPnP media importer does not support partial synchronization. Also doing a full import every day would probably be too slow right now (but I'm currently looking into performance). The media import logic contains a lot of code to compare the already imported media items with the newly provided media items. It detects whether items have changed or have been added and removed and only makes the changes to the database which are necessary. If an item hasn't changed between imports it isn't touched. Right now there is no scheduled synchronization (identical to the "normal" library scan / update) but it would be very easy to add.

(2021-03-01, 23:20)jbinkley60 Wrote: [ -> ]Second, does MediaImporter take all of the metadata from the addon vs. rescraping the data ?  If so, what format is needed from the addon ?  The Mezzmo addon gets an XML response and parses it prior to adding to the Kodi database.  In looking at the Emby importer code it looks like you might be doing a field mapping but I can't tall what the exact structure is of the uPNP metadata going to MediaImporter.

Media importer add-ons must provide their media items as ListItems (see https://codedocs.xyz/AlwinEsch/kodi/grou...titem.html). UPnP isn't involved and the media import logic doesn't care whether the information was originally retrieved as XML or JSON or anything else because it is the add-on's responsibility to transform this into a list of ListItems. In your Mezzmo add-on you currently take the XML and turn it into SQL queries. In a media importer add-on you turn it into ListItems.

As a general roundup I'm just putting out a few buzz words to try to describe the feature set of media importing:

• Media items can be imported from multiple providers (servers) coexisting. For each provider the user can create imports of specific media types.
  
• Media import uses generic interfaces everywhere: for importers, databases, media type handlers, ...
  
• The following media types can currently be imported:
  • movies and sets / collections
    
  • tvshows, seasons and episodes
    
  • music videos
    
• The following importers are currently supported:
  • UPnP (hardcoded in Kodi core)
    
  • add-ons (new extension point)
    
• A media importer can support the following features
  • automatic provider discovery
    
  • manual provider discovery (with user interaction)
    
  • full (and optionally partial) synchronization
    
  • observing imported providers / imports for changes and synchronizing them with the Kodi library
    
  • passing changes to imported media items from Kodi back to the media provider
    
• Add-on media importers can provide any other additional functionality which is not media import specific
  

(2021-03-02, 08:15)Montellese Wrote: [ -> ] ?The UPnP media importer does not support partial synchronization. Also doing a full import every day would probably be too slow right now (but I'm currently looking into performance). The media import logic contains a lot of code to compare the already imported media items with the newly provided media items. It detects whether items have changed or have been added and removed and only makes the changes to the database which are necessary. If an item hasn't changed between imports it isn't touched. Right now there is no scheduled synchronization (identical to the "normal" library scan / update) but it would be very easy to add.

(2021-03-01, 23:20)jbinkley60 Wrote: [ -> ]Second, does MediaImporter take all of the metadata from the addon vs. rescraping the data ?  If so, what format is needed from the addon ?  The Mezzmo addon gets an XML response and parses it prior to adding to the Kodi database.  In looking at the Emby importer code it looks like you might be doing a field mapping but I can't tall what the exact structure is of the uPNP metadata going to MediaImporter.

Media importer add-ons must provide their media items as ListItems (see https://codedocs.xyz/AlwinEsch/kodi/grou...titem.html). UPnP isn't involved and the media import logic doesn't care whether the information was originally retrieved as XML or JSON or anything else because it is the add-on's responsibility to transform this into a list of ListItems. In your Mezzmo add-on you currently take the XML and turn it into SQL queries. In a media importer add-on you turn it into ListItems.

As a general roundup I'm just putting out a few buzz words to try to describe the feature set of media importing:

• Media items can be imported from multiple providers (servers) coexisting. For each provider the user can create imports of specific media types.
  
• Media import uses generic interfaces everywhere: for importers, databases, media type handlers, ...
  
• The following media types can currently be imported:
  • movies and sets / collections
    
  • tvshows, seasons and episodes
    
  • music videos
    
• The following importers are currently supported:
  • UPnP (hardcoded in Kodi core)
    
  • add-ons (new extension point)
    
• A media importer can support the following features
  • automatic provider discovery
    
  • manual provider discovery (with user interaction)
    
  • full (and optionally partial) synchronization
    
  • observing imported providers / imports for changes and synchronizing them with the Kodi library
    
  • passing changes to imported media items from Kodi back to the media provider
    
• Add-on media importers can provide any other additional functionality which is not media import specific
  

Thanks for the response.  I am still not following the how you handle deletions answer.  Are you marking each record in the database with a last seen timestamp or something similar and then via your synchronization process noting that a Kodi entry wasn't seen during the last media provider scan and thus removing it ?   I originally was going to take this approach in the Mezzmo addon but the full sync seemed easier and speed isn't a problem with the SQL interface. 

The ListIem approach is fine.  The Mezzmo addon already does that for the GUI display when the user is browsing their content prior to database insertion.   I'll probably still either create a new addon or a branch of the current one specific to the MediaIMporter functionality.  There are too many folks using the current code right now and I don't want to disrupt them during development and testing.

Thanks,

Jeff

(2021-03-02, 11:42)jbinkley60 Wrote: [ -> ]Thanks for the response.  I am still not following the how you handle deletions answer.  Are you marking each record in the database with a last seen timestamp or something similar and then via your synchronization process noting that a Kodi entry wasn't seen during the last media provider scan and thus removing it ?   I originally was going to take this approach in the Mezzmo addon but the full sync seemed easier and speed isn't a problem with the SQL interface. 

Media import adds additional tables to the database which keeps track of which media item has been imported from where. Now when an item is removed from the server there are two possible scenarios:

1. The media importer (add-on) does a full sync. Kodi then retrieves all previously imported media items and compares them with the list of "newly" provided media items. All items which are part of the list of already imported items but not part of the list of "newly" provided media items will then be removed from the database.
   
2. The media importer (add-on) detects that the media item has been removed from the server (Emby and Plex support this) and tells Kodi through the media importer interface that the item has been removed. It is then automatically removed from the database.
   

For #2 the important part is that with partial synchronization there are two possibilities:

1. The media importer (add-on) simply provides a list of changed (added or updated) media items and Kodi determines which ones are new and which ones have changed. This approach does not support removing imported media items.
   
2. The media importer (add-on) provides a list of changed (added, updated or removed) media items with an additional flag for each item which specifies the type of change. Kodi then takes care of either adding, updating or removing the media items depending on the specified type of change.
   

Ok, so the net of it is that the addon makes the determination on how to detect deletions.  Got it.  I had planned to code #2 for Mezzmo but it was easier just to use method 1 and do a full sync.  I was considering the exact approach you use, to add a table to track.   The current partial sync code does detect metadata changes.  The current Mezzmo addon crawls through the Mezzmo database throughout the day in the background (starting with the newest items)  and detects changes / additions.  The nightly sync does the full rebuild.  It wouldn't be too difficult to integrate the logic.


Jeff

After spending several hours trying to improve the whole code responsible for adding / writing media items retrieved from a media provider into Kodi's database I realized that I only had to adjust the SQL transaction handling to improve the performance by a factor of up to 50. I only tested this on SQLite (so no idea what the performance with MySQL is) and the improvements vary depending on whether the SQLite database is located on a HDD or SDD but either way the performance improvement should be significant. On my dev computer importing 60 movies to an HDD took ~80s and on an SSD it took ~9s. With the improvements this went down to ~1.4s. Now when importing a large library the main time is spent on retrieving the media items from the media provider - adding the retrieved items to the database is very fast.

And last but not least these new test builds are based on Kodi v19 (final).

Downloads: https://github.com/Montellese/xbmc/wiki/...#downloads
Changelog: https://github.com/Montellese/xbmc/wiki/...erformance

Still can't import large TV Show libraries, everytime it gets to retrieving episodes from media provider it errors out. Probably will never work with large libraries I guess?