+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Development (https://forum.kodi.tv/forumdisplay.php?fid=32)
+--- Forum: Kodi Application (https://forum.kodi.tv/forumdisplay.php?fid=93)
+--- Thread: WIP Media importing and library integration (UPnP, Emby, Plex, ...) (/showthread.php?tid=224794)
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - Montellese - 2021-03-20
I found two issues with the previous release so I prepared new test builds:
Downloads: https://github.com/Montellese/xbmc/wiki/Downloads#downloads
Changelog: https://github.com/Montellese/xbmc/wiki/Downloads#kodi-v19-performance-v2
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - LongMan - 2021-03-20
Quote:I found two issues with the previous release so I prepared new test builds:
Whatever changes you made resulted in my youtube channels loading much faster. I always assumed that was just a problem with Matrix. Thanks for that.
Cheers,
LongMan
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - LongMan - 2021-03-20
In case this is also from MediaImport, the warming dialog when installing from zip files only shows the Tilte, there is no warning description.
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - M4tt0 - 2021-03-20
I've started experimenting with the Mediaimporter again after quite some time and experience the following problem:
My Emby Server runs in Docker on my NAS and is discovered correctly, but authentification requests keep failing. Examining the logs, the problem is obvious: The authentification requests are sent to the wrong IP address, i.e. the add-on is trying to address my Emby Server via the IP it has on the bridge network within docker, and not via the IP address of my server itself. In the network settings of my Emby Server there is an option to override the local IP address of the server. I've put my NAS IP in there. But it doesn't help. It seems the mediaimport.emby addon is trying to evaluate the IP address itself, rather than taking the "overridden" one from the Emby server settings.
Is this a known problem? Is there a workaround? Or am I doing something wrong?
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - LongMan - 2021-03-20
@M4tt0, Montellese had ask about something that sounded similar before. The post is here
https://forum.kodi.tv/showthread.php?tid=224794&pid=3014074#pid3014074
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - M4tt0 - 2021-03-20
Thanks a lot, @LongMan. I guess the answer to his question is "yes", then... :-)
Or alternatively and if possible, the IP auto discovery method takes the override value from the Emby Server network settings into account.
I've looked at the source code to see whether there was some obvious, intuitive way for me to patch that in, but failed to do so...
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - M4tt0 - 2021-03-20
I've created a proper github issue for @Montellese so that this doesn't get lost.
It's here if anybody else wants to track it: https://github.com/Montellese/mediaimporter.emby/issues/41
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - LongMan - 2021-03-20
@M4tt0,
https://github.com/Montellese/mediaimporter.plex/commit/f1f6c8fa4aa6a0ecace55b0ff3afaf0d419b292e
These are changes that were made to the Plex mediaimporter. Maybe that can help you.
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - Croneter - 2021-03-20
(2021-03-15, 13:54)Montellese Wrote:(2021-03-10, 22:47)Croneter Wrote:(2021-03-10, 19:13)Montellese Wrote: I just imported ~9K movies from a Plex media server and it took 95 minutes of which 85 minutes were spent on retrieving the items and 10 minutes were spent on writing the items to the database.I'd think that PlexKodiConnect is a couple of magnitudes faster than that, though I can't test on a large library. Took months to improve... Have you looked at its code, specifically the MySQL and Plex metadata retrieval part?
@Croneter I took a closer look at whatpython-plexapidoes and it first retrieves a list of 100 items (specified bymediaimporter.plex) and then retrieves the metadata for every item separately. I did some manual testing and couldn't figure out how to retrieve multiple items from a library section with all necessary metadata (cast, media parts, ...). How do you do that as part of PKC?
Have a look at this function. Sorry, could be prettier and better documented https://github.com/croneter/PlexKodiConnect/blob/4415ad39cf8223ba34c6819afe301b2e87fbb86a/resources/lib/plex_functions.py#L699
It basically returns an iterator for an entire Plex section's ElementTree xml elements of a certain kind, e.g. movies, seasons or episodes. Absolutely vital are the args (=http headers) checkFiles and skipRefresh. The Plex Media Server is told to not scan an element upon requesting its metadata. This will speed up your sync by an order of magnitude.
I switched to threadsafe iterators, queues, etc. because some Plex libraries are so large that PKC needed to somehow "stream" metadata from the PMS to the Kodi DB.
Let me know if you need something more specific. Happy to help!
Edit - in short: you do NOT need to poll every single item's metadata separately
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - Montellese - 2021-03-21
(2021-03-20, 13:46)LongMan Wrote:Sorry for that. Since I never used a plugin in my media import builds I never noticed that regression I introduced. It was discovered after that particular change was merged into mainline for v20. I've cherry-picked the fix into my media import work now.Quote:I found two issues with the previous release so I prepared new test builds:
Whatever changes you made resulted in my youtube channels loading much faster. I always assumed that was just a problem with Matrix. Thanks for that.
Cheers,
LongMan
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - Montellese - 2021-03-21
(2021-03-20, 14:58)LongMan Wrote: In case this is also from MediaImport, the warming dialog when installing from zip files only shows the Tilte, there is no warning description.I'll have to double check but I don't think I touched that part of the code.
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - Montellese - 2021-03-21
(2021-03-20, 22:28)Croneter Wrote:Thanks for the pointers. I'll take a closer look ASAP.(2021-03-15, 13:54)Montellese Wrote:(2021-03-10, 22:47)Croneter Wrote: I'd think that PlexKodiConnect is a couple of magnitudes faster than that, though I can't test on a large library. Took months to improve... Have you looked at its code, specifically the MySQL and Plex metadata retrieval part?
@Croneter I took a closer look at whatpython-plexapidoes and it first retrieves a list of 100 items (specified bymediaimporter.plex) and then retrieves the metadata for every item separately. I did some manual testing and couldn't figure out how to retrieve multiple items from a library section with all necessary metadata (cast, media parts, ...). How do you do that as part of PKC?
Have a look at this function. Sorry, could be prettier and better documented https://github.com/croneter/PlexKodiConnect/blob/4415ad39cf8223ba34c6819afe301b2e87fbb86a/resources/lib/plex_functions.py#L699
It basically returns an iterator for an entire Plex section's ElementTree xml elements of a certain kind, e.g. movies, seasons or episodes. Absolutely vital are the args (=http headers) checkFiles and skipRefresh. The Plex Media Server is told to not scan an element upon requesting its metadata. This will speed up your sync by an order of magnitude.
I switched to threadsafe iterators, queues, etc. because some Plex libraries are so large that PKC needed to somehow "stream" metadata from the PMS to the Kodi DB.
Let me know if you need something more specific. Happy to help!
Edit - in short: you do NOT need to poll every single item's metadata separately
EDIT: From a quick peek through the code it looks to me like you first get the XML for all media items using the
/library/sections/{sectionid}/all endpoint and then you use a thread pool to get the metadata for every returned media item using the /library/metadata/{key} endpoint in PF.GetPlexMetadata(). That still looks to me like you are retrieving the metadata for every single item separately. What am I missing / misunderstanding?I also manually checked the response for the
/library/sections/{sectionid}/all endpoint and it only contains a few actors, not all media parts and is missing a lot of additional metadata / information.
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - kingkiller - 2021-03-21
(2021-03-19, 06:41)LongMan Wrote: Yeah. I gave Library Integration Tool a spin on Leia and Osmosis on Matrix, a MedaImport Build actually. I actually was looking more at the user experience. I am not really fan of the Osmosis approach, one has to enter Osmosis to export an item, you cannot export from each video addon. With Library Integration Tool, the 'Add to Library" menu entry is present in each video addon, so one can export from within each addon and I prefer that.
I am aware of the state of the addons, @kingkiller mentioned looking into updating Library Integration Tool. Hope he gets a chance to do it.
I made several changes to the addon, I believe that compared to the original repository, things don't work and some new menus need some correction, but Netflix, Amazon, Crunchyroll and Disney + should work, at least the last ones I did a few days ago everything worked well.
Regarding Python3, I believe that the menus are the most complex part to adapt, considering the changes coming in Kodi19.
I started to analyze the creation of custom menus with XML, but I didn't have time to go so far in the tests.
If you want to test the current version and look for BUGs, as they must exist, it would be great.
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - LongMan - 2021-03-21
@kingkiller,
Thanks for taking the time to respond. I am happy to help you test it.
I am not sure if you have read the thread, but the MediaImport Project will add a feature to Kodi that will allow addition of items to Kodi's library without using strm files. At current any Matrix compatible video addon can add an extension point which allows it to be a Media Provider (a source of media, videos only at present). However, I was thinking that instead of asking each addon author to add the extension point, we could add said extension point to an addon/script such as Library Integration Tool. In so doing all video addons will be sources of imports via Library Integration Tool (LIT) or something similar.
Unfortunately, MediaImport requires python 3 and LIT is not yet python 3. Could you outline what are your pain points in doing the port to python 3?
Cheers,
LongMan
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - kingkiller - 2021-03-22
(2021-03-21, 19:35)LongMan Wrote: @kingkiller,Hi,
Thanks for taking the time to respond. I am happy to help you test it.
I am not sure if you have read the thread, but the MediaImport Project will add a feature to Kodi that will allow addition of items to Kodi's library without using strm files. At current any Matrix compatible video addon can add an extension point which allows it to be a Media Provider (a source of media, videos only at present). However, I was thinking that instead of asking each addon author to add the extension point, we could add said extension point to an addon/script such as Library Integration Tool. In so doing all video addons will be sources of imports via Library Integration Tool (LIT) or something similar.
Unfortunately, MediaImport requires python 3 and LIT is not yet python 3. Could you outline what are your pain points in doing the port to python 3?
Cheers,
LongMan
I had not considered that MediaImport is Py3, considering this has no way to work for kodi 18, depending on how MediaImport works, LIT would have to be practically all rewritten, especially because it is based on reading the directors via JsonRPC and collecting data to structure according to information and create .strm, i need to understand how MediaImport will work to get an idea of what could be done at LIT, let's assume that JsonRPC continues to be used, one of the big problems is the ordering and categorization of content, each service orders in a different way, and I had to reorder items manually.
___
Well, in my head I remember that there are two things that I suppose need to be updated.
1. The progress dialogs in Kodi 19 lost some lines of information, at least that's what I understood in the documentation, I understand that the information will not appear or will generate an error, with that I thought about creating personalized menus for everything.
2. Some py2 functions need to be adapted, except that there were unicode errors in the current code (from my tests, everything is ok), but I believe that it is necessary to use six, kodi_six (I did some tests, but it was not working and I paused the tests on that).
At least I understood that these are the points that need to be changed to work in py3.
As soon as you can do tests with kodi19 and see more clearly what needs to be changed.