WIP Media importing and library integration (UPnP, Emby, Plex, ...) - Printable Version +- Kodi Community Forum (https://forum.kodi.tv) +-- Forum: Development (https://forum.kodi.tv/forumdisplay.php?fid=32) +--- Forum: Kodi Application (https://forum.kodi.tv/forumdisplay.php?fid=93) +--- Thread: WIP Media importing and library integration (UPnP, Emby, Plex, ...) (/showthread.php?tid=224794) |
RE: Media importing and library integration of UPnP - Montellese - 2016-10-30 The Kodi to Kodi integration is achieved through the UPnP importing which is supported with these builds. But it requires the same (test) build on both installations i.e. it is not enough to install one of these (test) builds on your Raspberry Pi and then import your library from an existing Kodi installation. RE: Media importing and library integration of UPnP - LongMan - 2016-10-31 First, thanks for your work on this. Really Nice. Just a few observations.
Great Stuff Cheers, LongMan RE: Media importing and library integration of UPnP - Montellese - 2016-10-31 Wow that was fast. Thanks for the feedback. TBH I've never used Emby before this so I simply installed the server, added some stuff to the library and worked with that. So any feedback/input on what I've misunderstood and/or could be done better is very welcome. (2016-10-31, 07:25)LongMan Wrote: I can start playback from the Home Screen Widgets now using Estuary (haven't been able to install any other skin to test). Maybe you can remove that from your todo list.There probably still is an issue because I didn't change anything so I'll keep it on the TODO list but thanks for trying it out. (2016-10-31, 07:25)LongMan Wrote: Emby Server has a feature to hide a user from the login screen. For a hidden user, the username has to be typed in. It is only possible to select from the list now, so hidden users cannot be added.I can add an option "Manual" and allow the user to manually input his username. I also didn't try password protected accounts so we might need that as well? (2016-10-31, 07:25)LongMan Wrote: Subtiltes inside a ripped DVD file works fine, however an external .srt file accompanying a .mkv file does not show, using T on the keyboard.I never use subtitles but AFAIK our player simply tries to find subtitles next to the video file which obviously doesn't work with APIs like Emby where there's no directory scanning. I store the streamdetails about video, audio and subtitle streams but there's no way to provide a path for them so that would probably require changes to our player infrastructure. (2016-10-31, 07:25)LongMan Wrote: It is now expected behaviour (Emby for Kodi) that each mediaitem gets tagged with the library/view to which it belongs. It appears that the addon does it in the module librarysync.py by calling a function called updateTag from another module kodidb_functions.pyI'll take a look. What exactly is the purpose of this tag? RE: Media importing and library integration of UPnP - LongMan - 2016-10-31 I have always followed the development. Media Import is perfect for Rear Entertainment Systems. I use it in my minivan. If everybody wants to watch the same thing, use the builtin system. If not, use your tablet, the same thing is available there. Significantly less bickering results. A manual sign in would be good. The Emby for Kodi addon works in a similar manner. After install it auto discover the server and walks through a wizard. Otherwise, one can sign in from the settings page which allows manual via username and password or EmbyConnect (allows sign without knowing the external ip address when away from home). And yes, it is a good idea to include password, many people in the Emby forums connect to their server outside of the home. The subtitles work in the Emby for Kodi addon, so i guess there is a way to do it via the API. The tags are used as a way to separate libraries of the same type of content. I have Series, Kids Series, Docu Series and Special Interest as libraries on my Emby server. They are all tv shows, but each get tagged with the name of their respective library name (I think in the API it is called views). We then use playlists or video nodes to separate them on the home screen. The plugin actually generates a set of video node for each library using the tag to separate them. Hope that helps. Cheers, LongMan RE: Media importing and library integration of UPnP - Hedda - 2016-11-02 (2016-10-30, 19:35)Montellese Wrote: I have spent quite some time on supporting direct import from an Emby server through their API. @Montellese, not sure if forbidden topic but this concept could also work great with Plex Media Server too right? That is, code for Kodi that imports metadata from Plex's MediaServer API, maybe ported from PlexKodiConnect (or PleXBMC), rather than UPnP? https://forums.plex.tv/discussion/210023/plexkodiconnect-let-kodi-talk-to-your-plex/ ( https://github.com/croneter/PlexKodiConnect ) http://forum.kodi.tv/showthread.php?tid=92310 ( https://github.com/hippojay/plugin.video.plexbmc ) RE: Media importing and library integration of UPnP - Montellese - 2016-11-02 (2016-10-31, 16:01)LongMan Wrote: A manual sign in would be good. The Emby for Kodi addon works in a similar manner. After install it auto discover the server and walks through a wizard. Otherwise, one can sign in from the settings page which allows manual via username and password or EmbyConnect (allows sign without knowing the external ip address when away from home). And yes, it is a good idea to include password, many people in the Emby forums connect to their server outside of the home.Last night I re-worked the media provider settings for Emby which now allow the following possibilities: Either provide an API key (no login required), choose a publicly visible user with the option to specify a password or to manual enter a (hidden) user with the option to specify a password. I also extended all API calls with the functionality to re-try authentication in case the user has been logged out in the mean time. Unfortunately I forgot to push some minor changes which is why the buildserver failed to build and you'll have to be patient until Thursday for new builds with this change. (2016-10-31, 16:01)LongMan Wrote: The subtitles work in the Emby for Kodi addon, so i guess there is a way to do it via the API.Is it possible that it's using the transcoding feature in which case the subtiltes would be intergrated into the transcoded stream? Or it might be using the direct file access in which case Kodi can search for local subtitles again. Right now I've limited functionality to the static stream which should allow streaming in all cases but might have some drawbacks like external subtitles. (2016-11-02, 12:32)Hedda Wrote:I actually looked at Emby and Plex Media Server but the Emby API is much better documented with swagger-ui so trying it out and using it was much easier. Apart from some unofficial documents and the existing Kodi addons I couldn't find any information about the API of PMS. I did some "reverse engineering" and managed to login, retrieve movies, tvshows etc. which should be enough but those were only manual tests and I first want to get the Emby import working properly before looking at PMS again. But it's certainly possible.(2016-10-30, 19:35)Montellese Wrote: I have spent quite some time on supporting direct import from an Emby server through their API. RE: Media importing and library integration of UPnP - LongMan - 2016-11-02 (2016-11-02, 12:43)Montellese Wrote: Last night I re-worked the media provider settings for Emby which now allow the following possibilities: Either provide an API key (no login required), choose a publicly visible user with the option to specify a password or to manual enter a (hidden) user with the option to specify a password. I also extended all API calls with the functionality to re-try authentication in case the user has been logged out in the mean time. Unfortunately I forgot to push some minor changes which is why the buildserver failed to build and you'll have to be patient until Thursday for new builds with this change. I saw the changes. No problem, it is worth waiting for. (2016-11-02, 12:43)Montellese Wrote: Is it possible that it's using the transcoding feature in which case the subtiltes would be intergrated into the transcoded stream? Or it might be using the direct file access in which case Kodi can search for local subtitles again. Right now I've limited functionality to the static stream which should allow streaming in all cases but might have some drawbacks like external subtitles. I think you are right about that. Well partly anyway. The addon offers DirectPaths (file access) or HTTP playback/streaming; I use HTTP. I use Intel NUCs which able to handle the load, so there is no need for transcoding. It actually says "Direct Playing" on the server web interface. The Emby Server server stores the path to the subtitle files, however I don't think Kodi is accessing the files directly, because the subtitle still works while I am away from home. I am can enable, disable, or cycle through the subtitles as if Kodi was accessing them. @angelblue05 is making adjustments to the addon's playback functions now, and would be better able to explain. I just noticed that my movies didn't import. I thought t i have only added tv shows and music videos to the TestUser, but it should have access to all libraries. If the problem persists after the next build, i will post the logs. Cheers, LongMan RE: Media importing and library integration of UPnP - null_pointer - 2016-11-02 This is looking interesting, it looks like you are getting close to having the import working. I think import is a part of the big picture when dealing with external media management systems but there are other considerations also when trying to fully integrate with external systems. One of these is comms and event management between the systems, keeping each other updated with the latest event and changes to media, this includes not just addning new items as they are added to the server but also that some new info about a media item is available (description, cast etc) but also that and item was just watched in a different room of the house on a different device. Or that the client just finished playing something or is currently playing something. The way we handle this in the Emby4Kodi addon is using a combination of the internal KodiMonitor event manager in Kodi for letting the server know the client is doing something or has done something. Also the Emby server has a WebSocket mechanism that Emby4Kodi addon uses via a WebSocket client to listen for media changes and update the local Kodi DB with the changes. There is also the problem of initial sync when you first start a client, this was a big pain point for Pi users, having to process all items on the server to see if any changed was not really an option so we had to implement two mechanisms. The first is a change HASH, it is effectively a hash of the item in question, it is returned with the items details from the server and we can use it to quickly check if an item has changed just by looking at the hash and the hash we saw last time for the item, if it changed then we need to sync this item. The second mechanism to help reduce the sync workload was to add an endpoint to return a list if items/id of items that changed since last update. This is what we refer to as fast sync and has greatly helped in reducing the time to start. This is critical as people want to boot their client and play something new, i.e. a TV epp that was made available last night, if they have to wait 5 min for a full sync the wife is not happy. Another thing to consider is how users filter their content once it is on the client, most users of a media management system like Emby are going to put a lot of effort into building a well formatted media library with sections and categories and if you dont allow then to mirror this in the client it will feel back. The way we do this in Emby4Kodi is using tags, we add a lot of tags to the items so filters can be applied and to allow user to build menus and lists of items in the same way they did on the server. Anyway I just wanted to point out there is more to a fully integrated system than just importing. So a few questions: - Can I use the new import system from an addon building a bunch of business logic around the import like I have outlined above, i.e. let the addon work out what and when things need to be imported / updated and use the import lib to do all the DB interaction? - What are importers written in, C++, Python? RE: Media importing and library integration of UPnP - angelblue05 - 2016-11-03 I'm late to the party. I haven't tested anything yet, however I do have some questions. What is your plan for long term maintenance? Is this going to replace the need for the Emby for Kodi add-on? I guess I don't understand the approach for a project of this scale. Are you planning on maintaining every external database out there? Would it not be best to allow add-ons to utilize your import system and they each build their own process? Regardless, I appreciate the progress on your project, I'm looking for more information regarding what to expect. Tags: We use them to split the user's library into the library sections they have in Emby. Subtitles: We use the Emby HTTP endpoint to provides the subtitles for files over HTTP. We append them using the Kodi api: http://mirrors.kodi.tv/docs/python-docs/14.x-helix/xbmcgui.html#ListItem-setSubtitles Unfortunately, due to the current Kodi api limitation, there's no way to label HTTP subtitles. So right now, we are downloading the subtitles locally on the fly just to have the right language tag appear. Edit: Also, I wish you reached out to us. I feel slightly blindsided by the approach taken assuming what I asked above is the direction the project is taking... RE: Media importing and library integration of UPnP - Ned Scott - 2016-11-03 Quote:Edit: Also, I wish you reached out to us. I feel slightly blindsided by the approach taken assuming what I asked above is the direction the project is taking... This is probably just a proof of concept at this point, to show off media importing for other things. I'm sure you guys will be included/consulted/something for the long-term regarding Emby and media import. This is just an early preview of what awesome things are now possible. Media importing and library integration of UPnP - angelblue05 - 2016-11-03 I really find the project interesting. That's why I mentioned long term application, how maintenance will be handled and such. RE: Media importing and library integration of UPnP - Montellese - 2016-11-03 (2016-11-02, 16:02)LongMan Wrote: I saw the changes. No problem, it is worth waiting for.I actually have one more question concerning users. Should it still be possible to select a specific user even if an API key has been provided (to limit access to certain items only available to a specific user)? In case this is desired should that user also be authenticated with a password or does the knowledge of the API key overrule any user-specific authentication? (2016-11-02, 23:58)null_pointer Wrote: One of these is comms and event management between the systems, keeping each other updated with the latest event and changes to media, this includes not just addning new items as they are added to the server but also that some new info about a media item is available (description, cast etc) but also that and item was just watched in a different room of the house on a different device. Or that the client just finished playing something or is currently playing something.The media import system supports updating/removing items automatically in general. With UPnP we can receive update calls from the server which are then passed to the proper importer etc. And we can send updates from the importing instance to the server to tell it that an item has been watched/unwatched/... The same goes for the Emby importer but there I only added watched/unwatched updates right now. I haven't looked into the websocket stuff yet but I've seen it in the docs. (2016-11-02, 23:58)null_pointer Wrote: There is also the problem of initial sync when you first start a client, this was a big pain point for Pi users, having to process all items on the server to see if any changed was not really an option so we had to implement two mechanisms. The first is a change HASH, it is effectively a hash of the item in question, it is returned with the items details from the server and we can use it to quickly check if an item has changed just by looking at the hash and the hash we saw last time for the item, if it changed then we need to sync this item.Yes initial sync is not very fast because it has to retrieve all items, compare them to the previously imported items and find all differences. Unfortunately the Kodi database schema doesn't provide anything to improve this right now and I didn't want to make to many changes to the existing layout so far to make the integration easier. (2016-11-02, 23:58)null_pointer Wrote: The second mechanism to help reduce the sync workload was to add an endpoint to return a list if items/id of items that changed since last update. This is what we refer to as fast sync and has greatly helped in reducing the time to start. This is critical as people want to boot their client and play something new, i.e. a TV epp that was made available last night, if they have to wait 5 min for a full sync the wife is not happy.In general this is supported by the importer because it has full control over the list of already imported items and the list of retrieved items which are passed on to the synchronisation task. It could e.g. just remove any item from the list of already imported items which haven't changed and only pass on the ones that might have changed and the synchronisation task will only care about those items and ignore all others. That way partial imports/synchronisations are possible as well. (2016-11-02, 23:58)null_pointer Wrote: Another thing to consider is how users filter their content once it is on the client, most users of a media management system like Emby are going to put a lot of effort into building a well formatted media library with sections and categories and if you dont allow then to mirror this in the client it will feel back. The way we do this in Emby4Kodi is using tags, we add a lot of tags to the items so filters can be applied and to allow user to build menus and lists of items in the same way they did on the server.As explained further below this is not really the main goal of media importing. It can import tags set on the Emby server but it does not care about user presentation. It simply adds the items to the local library and then it's up to the user to configure Kodi's library in the way he/she wants. (2016-11-02, 23:58)null_pointer Wrote: - Can I use the new import system from an addon building a bunch of business logic around the import like I have outlined above, i.e. let the addon work out what and when things need to be imported / updated and use the import lib to do all the DB interaction?Not yet as I'm still working out the interface for importing so providing addon bindings is out of scope right now until the interface has been proven to work for different use cases. (2016-11-02, 23:58)null_pointer Wrote: - What are importers written in, C++, Python?Right now they are written in C++ and as part of Kodi but my goals are to be able to write importers as binary addons (in C/C++) and as python addons. All you need is some kind of thread that handles media provider detection and an implementation of the importer interface capable of importing items from the media provider. (2016-11-03, 06:10)angelblue05 Wrote: I'm late to the party. I haven't tested anything yet, however I do have some questions.Right now this is just a proof of concept. I started out with UPnP importing because Kodi already supports UPnP so it was the easiest remote access path available. But it's very hard to test because you have to run two instances of Kodi etc. After having written the whole media importing interface with UPnP as the first importer I wanted to look into other importers to see if the interface works and if it requires additional stuff. I chose Emby because I've seen before that it has a REST and JSON-based API which is easy enough to use from Kodi. But as already mentioned before I've never used Emby before and I've never installed or used any Emby add-on in Kodi so basically what I did is try to figure out how to access movies, tvshows and music videos. This is not meant to replace any existing add-on and I'm not even sure if it will be part of the initial media importing work if it ever gets merged into Kodi mainline. As I've already explained to some team members there's a big difference between my media importing and backend specific add-ons (like for Emby, Plex, Netflix, ...) and that is that media importing tries to integrate the media items from any backend and make it look like regular Kodi without the user having to know where the data is coming from and how it got into the Kodi user interface etc. Backend specific add-ons on the other hand often try to make Kodi look similar to the backend and give the user the exact same experience as if they were using the backend with the backend's native client. They also try to provide the user with an exact representation of the media structure in the backend. Media importing and library integration of UPnP - angelblue05 - 2016-11-03 Awesome, well keep it up and thanks for your quick reply RE: Media importing and library integration of UPnP - LongMan - 2016-11-03 (2016-11-03, 09:35)Montellese Wrote: I actually have one more question concerning users. Should it still be possible to select a specific user even if an API key has been provided (to limit access to certain items only available to a specific user)? In case this is desired should that user also be authenticated with a password or does the knowledge of the API key overrule any user-specific authentication? I am not sure if the API key limits the items obtained to a specific user. @Angelblue or @xnappo or @null_pointer could answer that. However, with a regards to selecting items for a specific user, yes you need to. From community it is clear that many users use Emby Server's Users, Library Access and Parental Controls and along with separate profiles in Kodi to limit the content that their children has access to. So it would very important to them. Just to mention, there is a sizeable community using the Kodi + Emby setup. The discussions take place mainly in the Emby forums because, i suspect that it is not allowed here. Unfortunate, but rules are rules. Further, there are many users who were previously just Emby users who have started using Kodi + Emby for Kodi addon as a client to the Emby server. So it not just Kodi users who are choosing Emby as a backend, there are many Emby/Mediabrowser users who are warming to Kodi. There is an audience for this. Once the LibreELEC/OpenELEC builds are available there should be more testers. Cheers, LongMan Media importing and library integration of UPnP - angelblue05 - 2016-11-03 The api key is generated upon login. You should reuse the same key unless you explicitly have a way to sign out the user or unless the user's password is changed server side (I believe it revokes the token automatically). If you have a user selection option, you'll need to authenticate the selected user in the event the user has no api key saved client side. You should not use the same api key for all users. Hope this helps. |