+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Development (https://forum.kodi.tv/forumdisplay.php?fid=32)
+--- Forum: Kodi Application (https://forum.kodi.tv/forumdisplay.php?fid=93)
+--- Thread: WIP Media importing and library integration (UPnP, Emby, Plex, ...) (/showthread.php?tid=224794)
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - Montellese - 2021-12-15
Did you also try to scan media items to the library with Vanilla LibreELEC? Because maybe it's just a Kodi library thing and not directly related to media import. Another question is what happens to the memory when you restart Kodi / LibreELEC after you imported everything from Emby and wait for a while? Thanks for taking the time to investigate this in detail.
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - LongMan - 2021-12-16
Restart after import - Memory use went up immediately and stopped at 1351MB.
Disable Emby mediaimporter and restart - memory use went up to 572MB
Still using the MI Build but Reset to Defaults using LibreELEC settings "Reset to Defaults"
No Media - 207 MB
Movies - 292 MB
TV Shows - 337 MB
Music Videos - 343 MB
During each scan the memory usage went up but fell after the scan was completed to the values listed above.
Cheers,
LongMan
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - LongMan - 2021-12-16
After after finishing the scans above I let it sit for about 30 mins. The memory used rose to 398 MB (from 343 listed above). I then left it for 8 hrs and the memory used stayed at 398 MB. This is with the MediaImport build based on LibreELEC 10.01 and Kodi 19.3 Performance.
Let me know if I still need to do it with vanilla LibreELEC.
Cheers,
LongMan
BTW: Is this related? https://github.com/xbmc/xbmc/pull/20688
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - psrkallez06 - 2021-12-17
How is it going with The library cleaning thing?
not in a hurry just curious.Also @Montellese you said that you dont use The library cleaning feature. Is there another way to clean it? Or how do you do it?
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - ontap - 2021-12-29
Had trouble recently using this addon , TV shows would disappear on reboot of kodi , both android and OSX. I would have to go into the importer and rescan them. So thought I would do re install, but neither the repo or the plugin or the 19.3 xbmc build will install on a mac ?
Matrix.zip fails due to an invalid structure.
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - LongMan - 2021-12-30
@ontap,
Sounds like you are describing 2 problems. The first is a problem with a mediaimporter and the second with a installation problem on the mac.
For the importer could you say which one. Also, has there been any change in the provider ie media server? If there hasn't been made any changes on the client side then issue may be on the server side.
With regards to the installation on mac, are you using the latest mac build from the downloads page or a build that you use for the previous installation. Just want to check if any problem was introduced with the last update.
Cheers,
LongMan
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - ontap - 2021-12-30
Please ignore my post above , got it working last. last night , will check on whether "TV Shows" stays linked .
EDIT : Have latest build installed on mac ( which is the device I have my plex server running on ). Exited kodi , took dog for a walk , about an hour , turned kodi back on and TV Shows are all missing . Movies fine but TV Shows have "unsynced".
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - LongMan - 2021-12-30
What does the debug log show?
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - psrkallez06 - 2022-01-06
Hello again. I recently did a clean of on my media box. And now i have a loooot of movies and tv shows that wont play since i cant clear The library. Can i clean it manually somehow? Or remove it all and resan? If so any tips in how? Thanks!
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - LongMan - 2022-01-07
@psrkallez06,
The easiest thing may be just to rename the video database and re-import. The import process is pretty quick so you should up and running again quite quickly.
He asked previously why we are not concerned about the library cleaning feature> The short answer is, most of us are hoarders so there is nothing to clean.
We just get another or a bigger hard drive.BTW why are you having so many broken links? Are you removing that many files from your server?
With regards to the fix, i beg you to just be patient. Usually errors are fixed pretty quickly. However, we just went through a season where one's free time becomes family and friends' time. After that we need to catch up on our normal task before we can find some more free time. So hold on a little bit longer.
Cheers,
LongMan
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - psrkallez06 - 2022-01-08
Thank you very much for your answer! Aah horders.. O dont have The money for new hdds right now haha. I have 1 tb.
I Will be patient. And yes i did remove a lot. But thats fine. I will rename and rescan. Chefers!
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - jbinkley60 - 2022-01-09
(2022-01-07, 02:24)LongMan Wrote: @psrkallez06,
The easiest thing may be just to rename the video database and re-import. The import process is pretty quick so you should up and running again quite quickly.
He asked previously why we are not concerned about the library cleaning feature> The short answer is, most of us are hoarders so there is nothing to clean.
BTW why are you having so many broken links? Are you removing that many files from your server?
With regards to the fix, i beg you to just be patient. Usually errors are fixed pretty quickly. However, we just went through a season where one's free time becomes family and friends' time. After that we need to catch up on our normal task before we can find some more free time. So hold on a little bit longer.
Cheers,
LongMan
A potential suggestion here is to create a full sync option which will sync the Kodi DB to the uPNP server DB. I ran into this issue with my Mezzmo uPNP server Kodi addon. What I did to resolve was create a daily sync process which removes all existing uPNP server data in the Kodi DB (Mezzmo in my case) and then does a full sync with the uPNP server. This eliminates all old stuff in Kodi which may have been deleted on the uPNP server. Unfortunately I haven't seen a uPNP server yet which has a query like, "tell me everything which has been deleted in the last 24 hours or similar". If so, it could just be queried and the associated records removed from the Kodi DB.
The sync service process runs once a day between midnight and 6AM and checks to ensure a video isn't playing before running, to minimize resource usage. I also have an on demand user option where the user can force a full resync if they don't want to wait till the daily process. This is in addition to the hourly sync which catches new things getting added to the uPNP server. I found this strategy works well and avoids deleting the database and similar. I used to have the same challenge prior to implementing this approach.
Here's a sample of the daily sync logs so the user can monitor what is going on. I also write these logs to a separate database where the user can query via the addon GUI vs. going into the Kodi.log file:
2022-01-08 00:56:13.572 T:946 INFO <general>: Mezzmo sync beginning.
2022-01-08 00:56:13.584 T:946 INFO <general>: Mezzmo daily sync process flag set.
2022-01-08 00:56:14.688 T:946 INFO <general>: Kodi database Mezzmo data cleared.
2022-01-08 00:56:14.749 T:946 INFO <general>: Mezzmo duplicate logging is enabled.
2022-01-08 00:56:18.954 T:946 INFO <general>: Mezzmo checkDBpath parent path added: 1 http://192.168.0.34:53168/
2022-01-08 00:57:41.103 T:946 INFO <general>: Mezzmo total Kodi DB record count: 15899
2022-01-08 00:57:41.109 T:946 INFO <general>: Mezzmo total Mezzmo record count: 16178
2022-01-08 00:57:41.114 T:946 INFO <general>: Mezzmo total Live Channels count: 0
2022-01-08 00:57:41.119 T:946 INFO <general>: Mezzmo total nosync videos count: 279
2022-01-08 00:57:42.057 T:946 INFO <general>: Mezzmo database reindex and vacuum complete.
2022-01-08 00:57:42.072 T:946 INFO <general>: Mezzmo sync completed. 16178 videos in 1m 28s checked.
2022-01-08 00:57:42.083 T:946 INFO <general>: Mezzmo sync setting is: Newest
2022-01-08 00:57:42.091 T:946 INFO <general>: Mezzmo performance logging is enabled.
Maybe some of this can help with ideas.
Jeff
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - LongMan - 2022-01-13
@jbinkley60,
Thanks for your suggestions. I was recommended deleting the database in this case, because there is a bug that needs to be addressed which concerns removing items from the database.
With regards to syncing, each import already has a manual sync process and a force full synchronization option. If I understand you correctly, your suggestion would be to add a feature to put it on a schedule.
With regards to being able to request changes since a specified time. My understanding is that neither Emby nor Jellyfin server has the feature builtin. The feature is provided by a server plugin, Kodi Companion for Emby and Kodi Sync Queue for Jellyfin. Maybe a similar type of progarm could be written for a UPnP server. TBH I don't even know if it possible for a UPnP server. What are your thoughts? Is that possible with Mezzmo?
Cheers,
LongMan
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - jbinkley60 - 2022-01-13
(2022-01-13, 17:01)LongMan Wrote: @jbinkley60,
Thanks for your suggestions. I was recommended deleting the database in this case, because there is a bug that needs to be addressed which concerns removing items from the database.
With regards to syncing, each import already has a manual sync process and a force full synchronization option. If I understand you correctly, your suggestion would be to add a feature to put it on a schedule.
With regards to being able to request changes since a specified time. My understanding is that neither Emby nor Jellyfin server has the feature builtin. The feature is provided by a server plugin, Kodi Companion for Emby and Kodi Sync Queue for Jellyfin. Maybe a similar type of progarm could be written for a UPnP server. TBH I don't even know if it possible for a UPnP server. What are your thoughts? Is that possible with Mezzmo?
Cheers,
LongMan
I’ll discuss how I do synchronization within the Mezzmo addon, since I wrote all the code for it and am familiar with how it works. Mezzmo is basically a database driven uPNP server, similar to Plex and others. When we say syncing I think of the Mezzmo and Kodi databases being fully in sync (i.e. 100% match between the records). To sync 2 databases like this there are two typical approaches. First, querying the source database (either partial or all records) and then comparing it against the target, in this case Kodi. The second is deleting all of the records for the source database in the target database (in this case deleting all Mezzmo records in the Kodi database) and then querying all of the source records and adding them to the target. The first requires some lookup table or tracking mechanism in the software doing the sync process (in this case the Mezzmo addon) to look for records no longer in the source (i.e. Mezzmo) but are still on the target (i.e. Kodi). I never implemented this lookup table approach in the Mezzmo addon since I wrote the full sync 2nd option code first and it seems effective. I’ve thought about adding it but it wouldn’t add a lot of value at this point.
The Mezzmo addon does a combination of these sync processes. There is an hourly sync service process which runs in the background. The user can select how this process runs. There’s normal option which first checks for the newest 400 Mezzmo records and will add anything new or if something has changed (i.e. updated metadata, codecs etc.. on the Mezzmo server for a given record) and then queries 1/6th of all other Mezzmo records to do the same comparison (looking for metadata and codec changes). A counter keeps track of the last record it queried from Mezzmo in the previous hour. When it gets to the end it starts over at the newest record in the Mezzmo database. So worst case all records get refreshed every 6 hours. I didn’t implement a look-up table or tracker which would identify any Kodi records for Mezzmo which weren’t seen (i.e. deleted from Mezzmo server). That is handled by the daily sync process. So worst case a stale record will remain in the Kodi database for up to 24 hours but from a user GUI perspective Mezzmo won't present it when browsing the playlists since it was removed from the Mezzmo database. A user might stumble across a stale record in the 24 hour period if they did an actor search, which queries the Kodi database. But this is rare since a full sync is done daily, as described below. The other hourly sync option is called Newest is to just review the newest 400 records each hour which would catch newly added records to the Mezzmo server which can be manually added by the user to the Mezzmo server or automatically discovered and scraped by the Mezzmo server.
For a full sync (i.e. delete all Mezzmo records in Kodi and read them) this is done daily between midnight and 6AM and can be forced manually (this is great if a Kodi database blows up, Kodi client rebuild etc..). The daily full sync process will remove all stale Mezzmo records in Kodi since it starts by removing all Mezzmo records and only adds back what is current. The user can also disable the background sync process altogether and use real time updates (i.e. as they browse the Mezzmo playlists in the GUI new records and changes are detected) or they can set sync to auto which tracks whether Mezzmo and Kodi are in sync and will adjust between normal and the newest 400 records based up the status of sync between the two databases.
For the last part of your question, querying based upon time. I am not aware of a uPNP server which tacks deletes and that can be queried by time. Mezzmo does allow a query of the newest x number of records and there is a uPNP XML field which tracks when a record was added to Mezzmo or you can query any set of records (ii.e. record numbers x to y) That feature is used by the addon Newest sync process. It could be used for date type queries / parsing but it wouldn’t have deleted records, just new ones.
Sorry for the long winded answer but I hope this makes sense.
Jeff
RE: Media importing and library integration (UPnP, Emby, Plex, ...) - LongMan - 2022-01-13
Thanks for taking the time to explain. Your UPnP XML sounds like what the server plugins do. Except they use a database and track deletions as well. I had said it is time based, i may be wrong there it could be since last sync.
I am not sure how long it would take to remove the entries from a given media provider. However, with the improvements that Montellese has made, the import process is so fast that it probably would not be much of a pain to remove and then re-add. So that may be a viable option.
Cheers,
LongMan