[angelblue05]

Yes I've been working on that already. Just need more time, juggling multiple projects atm

[Montellese]

I'm testing the latest build currently. Do you have a decent library set up in Emby? My only question is why is it slow to process episodes? Is this the add-on or C side? It takes half a second to a second to import a single episode. I have over 15000, at 67% imported after 23mins. Just to give you an idea, with Emby for kodi, we are talking 2-3 minutes to import all of them on the same hardware, so I'm looking to see if we can speed this up a bit. The only reason I mention this is that I have a relatively small library compares to others.

Edit: Never mind, it was writing to an old database so that's why it was slow. It was overwriting entries that already exists. Looking a bit faster now that I started with a fresh database. Will let you know once it's done importing.
Edit 2: 30mins to complete only my tv shows - so not much faster than before it seems, which took 38 mins. Now the question is it something we can improve within the add-on, or is it C ++ side?

If that's the speed we are looking at, it must be really slow on a low powered device. My episodes on a firetvstick takes 28 mins to import, so it must take more than an hour with media import.

I haven't really looked into performance yet. For Emby import testing I've only setup a small library with a ~200 movies and 10 tvshows because I usually don't use Emby server. I'm pretty sure there are potential improvements in the C++ code since I haven't done any performance optimization yet. Currently it processes one item after another, tries to find a matching already imported item, checks if they are the same or need to be updated etc. Furthermore it's possible to displaying every imported item title in the progress indicator might also slow things down (we had the same problem with database cleaning).

[angelblue05]

Ok thanks for the info. Great, I hope it's something that can be improved down the line.

[Montellese]

I'm testing the latest build currently. Do you have a decent library set up in Emby? My only question is why is it slow to process episodes? Is this the add-on or C side? It takes half a second to a second to import a single episode. I have over 15000, at 67% imported after 23mins. Just to give you an idea, with Emby for kodi, we are talking 2-3 minutes to import all of them on the same hardware, so I'm looking to see if we can speed this up a bit. The only reason I mention this is that I have a relatively small library compares to others.

Edit: Never mind, it was writing to an old database so that's why it was slow. It was overwriting entries that already exists. Looking a bit faster now that I started with a fresh database. Will let you know once it's done importing.
Edit 2: 30mins to complete only my tv shows - so not much faster than before it seems, which took 38 mins. Now the question is it something we can improve within the add-on, or is it C ++ side?

If that's the speed we are looking at, it must be really slow on a low powered device. My episodes on a firetvstick takes 28 mins to import, so it must take more than an hour with media import.

Do you have any details on how you manage to do this task in 2-3 minutes in Emby for Kodi? Are you connecting to the database directly? If so that's no fair comparison because you'd basically bypass any sanity checking that's being done in Kodi's core code which obviously results in a lot of additional SQL queries. I've taken a look at where most of the time is being used up and it's for writing to the database. I've eliminated a few of the calls that are not absolutely necessary which basically left me with a few INSERT/UPDATE/DELETE statements per item and it had no effect on the overall time.

I'm going to dig a little deeper to see if there are things in CVideoDatabase that can be optimized or if it makes sense to support batch insert/update/delete (which didn't have a use case so far).

Either way this probably would mainly affect initial importing. After that an importer implementation can either take care of only updating the items that need updating or it can provide Kodi with a full list and let it figure it out for itself which should still result in a lot less changes to the database.

[angelblue05]

Yes, of course it's not a fair comparison. I was just saying that so you had an idea of where we are coming from, what emby users are used to. I'd like for it to be a bit faster than what it currently is if it's possible, just because I know I don't have a big library compared to other users.

Emby for Kodi uses it's own database to hold all the kodi info for each item imported. This allows to speed up lookups in the Kodi database and avoid unnecessary queries.

Yes it does only effect the initial importing, but speaking from experience, it's the thing people will complain most about (especially if there's no way to backup and restore). Let me know if you need more info.

[da-anda]

sorry for the probably stupid question, but why do we have to do full imports? Would Kodi simply asking various registered backends (Kodi internal + content providers) to return a list of items and then merge it realtime not be better? Combined with lazyloading for metadata of fileitems this might be the best approach. Downside ofc is that there is no "offline view" unless we implement caching.

[Montellese]

sorry for the probably stupid question, but why do we have to do full imports? Would Kodi simply asking various registered backends (Kodi internal + content providers) to return a list of items and then merge it realtime not be better? Combined with lazyloading for metadata of fileitems this might be the best approach. Downside ofc is that there is no "offline view" unless we implement caching.

That was my initial idea a few years ago as well before I started working on this but I've been burnt too many times with the whole "dynamically add non-library items (like movie sets) into a library view". It is a pain for navigation and for cases like filtering etc. because you suddenly have to do all of this manually and can't rely on the database anymore. Furthermore without a proper cache it would result in constantly asking all the different backends for their full list of movies or tvshows etc. which might be even slower as well.

And if we start doing caching for "offline viewing" you basically end up with what I'm doing here.

[LongMan]

It is true that the initial sync is a pain point for Emby for Kodi. However, I think the most severe part of the pain the fact fact that the system cannot be used while the it is occurring. In addition if you mistakenly press the remote or keyboard, it could freeze and then one need to start over again. I suspect that the howls would be less with Media Import, because it is quite usable during the import process. In addition, with the import browser, one has the option to import a single library and start using that while the rest of the import process continues.

The critical thing imo is that the GUI does not freeze.

[LongMan]

With regards to 'realtime caching', I think Media Import is superior for persons who wish to have libraries. One could say that "realtime caching' is basically the method of all the video addons in the repo, which i think most will agree does not offer as good a user experience as does having a local library. Furthermore, having the filtering capabilities that the library offers via playlists, smart playlists, video and audio nodes make for a much better user experience in spite of the pain of the initial sync. I have numerous YouTube .strm files is my library because of this.

Amazon have been emailing that I am not taking advantage of my Prime benefits and I hardly ever stream from Netflix. If I could integrate strm files from them into my library, I would probably use them more.

[Montellese]

Finally had some time to look into this again. I've done a few minor improvements to how the add-on scripts are called and how their callbacks are handled / protected. Now if a script calls a callback with an invalid handle or that doesn't match the current action / mode it will receive an exception.
Furthermore I've figured out why imported items are always removed and re-added instead of synced and updated and added a fix (which will probably need discussing with other team members / developers).

I haven't really changed the API to the mediaimporter add-ons (except for the exceptions) though. The only thing missing in the API right now is the possibility to add / update / remove single items independent of the whole automated synchronisation process (e.g. using asynchronous notifications from Emby).

[bry]

@Montellese this may be jumping the gun a bit but I was wondering if somewhere down the line you could take a look at using live TV via emby.

[Montellese]

@Montellese this may be jumping the gun a bit but I was wondering if somewhere down the line you could take a look at using live TV via emby.

Since I'm not a live TV user and don't have such a setup I wouldn't get your hopes up too much

[da-anda]

why not simply write a PVR add-on for the emby backend? I don't see why this would have to be integrated into the media import interface

[Montellese]

why not simply write a PVR add-on for the emby backend? I don't see why this would have to be integrated into the media import interface

Yup that would probably be a better idea. But that requires someone who knows the Emby API and the Kodi PVR API.

[LongMan]

Getting access to the Emby PVR Backend from Kodi, has been constant goal over in the Emby forum. @angelblue05 added basic access to the channels in the Emby for Kodi plugin

https://emby.media/community/index.php?/...od/?hl=pvr

@marcelveldt was the last person to have a go at integration.

https://emby.media/community/index.php?/...-dor-emby/

However, it sounds like the API Knowledge is there but time and the C/C++ skills do not match the desire. Montellese's decision to go with a python API for media import may turn out to be critical for its success.