Kodi Community Forum
WIP Media importing and library integration (UPnP, Emby, Plex, ...) - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Development (https://forum.kodi.tv/forumdisplay.php?fid=32)
+--- Forum: Kodi Application (https://forum.kodi.tv/forumdisplay.php?fid=93)
+--- Thread: WIP Media importing and library integration (UPnP, Emby, Plex, ...) (/showthread.php?tid=224794)



RE: Media importing and library integration (UPnP, Emby, Plex, ...) - Montellese - 2021-12-06

(2021-12-06, 05:00)AkVRkSeN Wrote: I have  no idea if I did the "update" the right (or best) way. I went into github, find that the change was in kodi.py, selected show raw and downloaded the python file. Copied it to a sd card from my PC, put it in the FireTV and copied to the location on the FireTV. It felt clunky but it worked Smile.
That's probably not the easiest way but it also gets the job done ;-) The easier way would have been to download a ZIP archive from Github, copy it to your FireTV and then use "Install from ZIP" in Kodi's add-on browser.

BTW if you want to narrow down which data was causing the issue you can take a look at the log files. It should contain a warning starting with ignoring person followed by the label of the media item. If it's a tvshow or an episode it should be possible to narrow it down. If it's a season the information will probably not be that helpful...
(2021-12-06, 05:00)AkVRkSeN Wrote: In my kids short lives, those are the happiest four lines of code Smile for them. Thanks again.
Haha glad to hear it made your kids happy. Thanks to your feedback I'll push the fix onto master and it will be available with the next official release.
(2021-12-06, 05:00)AkVRkSeN Wrote: I have tried some of the other addons for working with an Emby library and this take the cake. The instructions are clear and easy and the solution executes really quickly. Thank you!
Thanks for the high praise.


RE: Media importing and library integration (UPnP, Emby, Plex, ...) - M4tt0 - 2021-12-06

(2021-10-02, 20:06)Montellese Wrote:
(2021-10-02, 16:05)enen92 Wrote: Can someone share what the blocking issue in Kodi/python3 is referred on above comments? I might be living under a rock Smile
TBH I don't know that much about it either. At first I thought it was a media import related problem. But @sualfred mentioned to me that the same happens with his skins which use a lot of add-ons in Kodi Matrix. The problem seems to be that if too many Python sub-interpreters are running at the same type Python is unable to properly handle catching exceptions with a specific type. So if some code tries to catch a TimeoutError it will not be caught. If the same code would try to catch any exception it would work even though the thrown exception is of type TimeoutError. See e.g. this workaround which solves the issue but obviously is very ugly: https://github.com/Montellese/mediaimporter.plex/commit/07edca203118abf6b32469a5f30114d15fa6a9d3. And a Python add-on cannot control every exception being thrown / caught because lots of libraries / modules depend on proper exception handling.

I did some debugging on Windows with media import and noticed that the problem only appears if the number of sub-interpreters running concurrently is greater than 4 and even. Each media importer add-on runs two service scripts and at least one on-demand script. If I only have one media importer add-on active everything is peachy but if I run two media importer add-ons I run into these issues all the time. If I run an additional add-on which uses a service everything is fine again because most of the time the number of concurrently running sub-interpreters is odd (2 + 2 + 1).

I thought it might have something to do with the locking in our PythonInterpreter and tried to improve it but it didn't help.
@enen92 @Montellese I understand the above prevented finally PRing this WONDERFUL, BEAUTIFUL, LIFE-CHANGING work to Master. I'm living under the rock, too. In fact, it seems I'm living under many rocks, this very earth, on a different planet. Did anything come out of your discussion above or was it a dead end, PLEASE?   ;-)


RE: Media importing and library integration (UPnP, Emby, Plex, ...) - psrkallez06 - 2021-12-08

Just noticed.. @LongMan provided me with a generic build. But it won't clean the library.. I normally use watchdog to clean/update my library. But even tho i delete a lot of movies from my FTP, the library won't clean even when i do it manually. the removed movies/shows are still there, when i try to play them i get "this is no longer available would you like to remove it from your library".

Any idea why that is?


RE: Media importing and library integration (UPnP, Emby, Plex, ...) - Montellese - 2021-12-08

(2021-12-06, 21:52)M4tt0 Wrote:
(2021-10-02, 20:06)Montellese Wrote:
(2021-10-02, 16:05)enen92 Wrote: Can someone share what the blocking issue in Kodi/python3 is referred on above comments? I might be living under a rock Smile
TBH I don't know that much about it either. At first I thought it was a media import related problem. But @sualfred mentioned to me that the same happens with his skins which use a lot of add-ons in Kodi Matrix. The problem seems to be that if too many Python sub-interpreters are running at the same type Python is unable to properly handle catching exceptions with a specific type. So if some code tries to catch a TimeoutError it will not be caught. If the same code would try to catch any exception it would work even though the thrown exception is of type TimeoutError. See e.g. this workaround which solves the issue but obviously is very ugly: https://github.com/Montellese/mediaimporter.plex/commit/07edca203118abf6b32469a5f30114d15fa6a9d3. And a Python add-on cannot control every exception being thrown / caught because lots of libraries / modules depend on proper exception handling.

I did some debugging on Windows with media import and noticed that the problem only appears if the number of sub-interpreters running concurrently is greater than 4 and even. Each media importer add-on runs two service scripts and at least one on-demand script. If I only have one media importer add-on active everything is peachy but if I run two media importer add-ons I run into these issues all the time. If I run an additional add-on which uses a service everything is fine again because most of the time the number of concurrently running sub-interpreters is odd (2 + 2 + 1).

I thought it might have something to do with the locking in our PythonInterpreter and tried to improve it but it didn't help.
@enen92 @Montellese I understand the above prevented finally PRing this WONDERFUL, BEAUTIFUL, LIFE-CHANGING work to Master. I'm living under the rock, too. In fact, it seems I'm living under many rocks, this very earth, on a different planet. Did anything come out of your discussion above or was it a dead end, PLEASE?   ;-)
Unfortunately I haven't had any time to look into the issue more and I have no idea if anyone else on Team Kodi is looking into it because it's very hard to reproduce reliably.


RE: Media importing and library integration (UPnP, Emby, Plex, ...) - Montellese - 2021-12-08

(2021-12-08, 21:12)psrkallez06 Wrote: Just noticed.. @LongMan provided me with a generic build. But it won't clean the library.. I normally use watchdog to clean/update my library. But even tho i delete a lot of movies from my FTP, the library won't clean even when i do it manually. the removed movies/shows are still there, when i try to play them i get "this is no longer available would you like to remove it from your library".

Any idea why that is?
Yeah one of the SQL queries in the database cleaning logic is broken in media import. I never use library cleaning so I didn't notice this myself. I'll try to fix it ASAP.


RE: Media importing and library integration (UPnP, Emby, Plex, ...) - psrkallez06 - 2021-12-09

(2021-12-08, 22:34)Montellese Wrote:
(2021-12-08, 21:12)psrkallez06 Wrote: Just noticed.. @LongMan provided me with a generic build. But it won't clean the library.. I normally use watchdog to clean/update my library. But even tho i delete a lot of movies from my FTP, the library won't clean even when i do it manually. the removed movies/shows are still there, when i try to play them i get "this is no longer available would you like to remove it from your library".

Any idea why that is?
Yeah one of the SQL queries in the database cleaning logic is broken in media import. I never use library cleaning so I didn't notice this myself. I'll try to fix it ASAP.

Thank you very much! I really appriciate it. Please post a build for generic when you have fixed it. I use library clean all The time Smile


RE: Media importing and library integration (UPnP, Emby, Plex, ...) - LongMan - 2021-12-09

I will run the build for you ASAP the fix is in.


RE: Media importing and library integration (UPnP, Emby, Plex, ...) - psrkallez06 - 2021-12-09

(2021-12-09, 20:44)LongMan Wrote: I will run the build for you ASAP the fix is in.

Thank you very much! Big Grin


RE: Media importing and library integration (UPnP, Emby, Plex, ...) - psrkallez06 - 2021-12-10

Anything i can do to help btw? Do you need any log files etc? Smile


RE: Media importing and library integration (UPnP, Emby, Plex, ...) - LongMan - 2021-12-10

Doesn't look like it. The problem has already been identified. It's just finding the time to fix it. He will fix it as soon as he can.

Cheers,
LongMan


RE: Media importing and library integration (UPnP, Emby, Plex, ...) - psrkallez06 - 2021-12-11

(2021-12-10, 23:48)LongMan Wrote: Doesn't look like it. The problem has already been identified. It's just finding the time to fix it. He will fix it as soon as he can.

Cheers,
LongMan

I understand. Thanks! Smile


RE: Media importing and library integration (UPnP, Emby, Plex, ...) - LongMan - 2021-12-13

I am getting high memory usage. On the machines with a lot of memory, the effect is not detrimental,  however the Pi and NUCs with 2GB RAM, eventually slow down and then crash and restart. Anybody else seeing this?


RE: Media importing and library integration (UPnP, Emby, Plex, ...) - Montellese - 2021-12-13

(2021-12-13, 19:56)LongMan Wrote: I am getting high memory usage. On the machines with a lot of memory, the effect is not detrimental,  however the Pi and NUCs with 2GB RAM, eventually slow down and then crash and restart. Anybody else seeing this?
Did this also happen with older builds? Or just with the latest ones based on Matrix 19.3? If so it might be worth checking if the issue also happens with vanilla Matrix 19.3.


RE: Media importing and library integration (UPnP, Emby, Plex, ...) - LongMan - 2021-12-13

I don't remember it happening before. I will check when I get home.


RE: Media importing and library integration (UPnP, Emby, Plex, ...) - LongMan - 2021-12-15

I ran some test. Not quite sure how to structure it but this is what I did.

I started with a fresh vanilla install of LibreELEC 10.0 based on Lodi 19.1. I not the memory use values and let it sit for more than 30 mins and record the values again. I then upgrade to LibreELEC 10.01 based on Kodi 19.3 and repeat the process. Next I upgrade to MediaImport based on LibreELEC 10.01 and 19.3. I repeat value taking process again. I then carried out the actions listed and waited and recorded again. Hopefully you understand my process.

                                                      Initial (MB)        After (MB)              
Vanilla LibreELEC 10.0.0                   206                 205
Vanilla LibreELEC 10.0.1                   155                 157
MI      LibreELEC  10.01                   158                 158
  • Enable MI Repo                       182                184
  •  Install Emby mediaimporter    300                294
  •  Add User                                333                334
  •  Add Imports                           259                670       (at the end of the imports about 20 mins)
  •  (No further action)                 670              1406      (about 4.5 hrs later)

EDIT: After another 12 hours the memory usage is still 1406.


This forum uses Lukasz Tkacz MyBB addons.