Work in Progress - Media importing and library integration (UPnP, Emby, ...)

  Thread Rating:
  • 5 Vote(s) - 4 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
Montellese Online
Team Kodi Developer
Posts: 4,846
Joined: Jan 2009
Reputation: 72
Location: Switzerland
Post: #1
Hey everyone

As some of you know I've been working on media importing and library integration for almost two years now (with a lot of breaks and very little free time) and I've demonstrated my work during the last two devcons and while my work was always publicly available on github I've never provided any test builds or asked the wider community for input because I first wanted to get the core design and implementation into a shape where it can be tested before having to deal with the flood of feature requests. In the last few days I finally had time to resolve some of the outstanding larger issues, bugs and TODOs and now it's finally time to get more people on board to test this, provide feedback and bug reports because there's only so much testing I can do myself and if you designed and implemented a whole system you don't really see any (better) alternatives anymore.

What's this all about?
The general idea is to provide a framework to be able to import media items from all kinds of services and locations which are not supported by our current library scanners which are purely filesystem based. Examples are importing some or all items from a plugin or from a UPnP server. This will allow users to directly integrate these items into their usual library so when they go through their library they'll also see items that are not directly available as a file on their disks/NAS but are available through another service and can play them like any other library item (as long as the service is available).

How does it work?
The new concept introduces the terms media providers and media imports. A media provider is an entity that can provide a set of media items and maybe also a hierarchy (filesystem). A media import is a pointer to a location in the hierarchy of a provider in combination with a list of media items (movies, tvshows, ...) that should be imported from that specific provider and location. Therefore there can be several imports for a single provider (e.g. multiple specific paths of a plugin). While this concept is aimed at specialized providers like UPnP servers etc. it could theoretically also be applied to the current library scanning and scraping. The provider would then be a specific server (or the local filesystem) and an import would just be a combination of a specific path on that server and a media type to import from that path. To handle providers and imports there are provider repositories which are just database specific implementation and right now there's just one for storing and retrieving the providers and imports from the video database.

To do the actual work of importing media items there's the media import manager. It has access to provider repositories to know about the existing providers/imports and to add/remove them. Furthermore it has access to specific implementations of import handlers and media handlers.

Import handlers are specific implementations which can get a list of media items of specific media types from a provider/import. Right now there's just an implementation for UPnP which uses the Search() request specified for UPnP DirectoryProvider implementations to request items of a specific media type.

Media handlers are media type specific implementations which know how to add items of a specific media type into the local library. That task involves getting all previously imported items of a specific provider, comparing the list of imported items with the list of previously imported items and then deciding which items to add, remove and update in the local library.

So how does importing items work?
  1. When Kodi starts up it disables all library items that were imported from a provider because it doesn't know if the provider is available right now. Disabled is just a state of an item and could be used in skins to e.g. grey out these items in the list so that the user sees the item as part of the library but realizes that the source it came from is not available right now.
  2. When Kodi discovers a (new) provider (e.g. a UPnP server) it tells the media import manager that a source is available.
    1. The media import manager checks if it already knows the provider or not.
    2. If it doesn't know the provider, it starts a provider registration task
      1. The provider registration task checks (based on the provider identifier/path) if there's an import handler capable of importing media items from that provider or not (for UPnP we e.g. require the UPnP server to support searching for a specific "upnp:class" property).
      2. After that new media imports can be added for the registered provider in Settings -> Media sources -> Media imports -> By media provider -> Add import...
    3. If there are imports defined for the detected provider, start an import retrieval task which will use the import handler capable of handling the detected provider to grab all items of the media types chosen by the user for this import
    4. If the import retrieval task was successful, start a media synchronisation task for every media type (using the proper media handler) chosen by the user for this import and provide it with the list of imported items
    5. The media handler takes care of synchronising the already imported items from that provider/import with the just previously imported ones
This whole process has been implemented using jobs and tasks (basically sub-jobs) which are run in a seperate thread in the background. They won't interfer with any media playback so if the user is watching a movie and a new UPnP server pops up, he won't see any dialog on the screen until he stops playback. Furthermore there's feedback through the progress bar dialog during the whole import process (similar to how it works for good old library updates).

What is already working/there?
Right now most of the retrieval, synchronisation and cleanup logic is there in a generic form. Importing is only supported from UPnP servers which are also provided by a compatible Kodi version (i.e. this specific test build). Furthermore it only supports importing items into the video library i.e. movies, tvshows, seasons, episodes and music videos. There's no support for the music library yet.

What isn't working?
A lot. I only have limited testing time and possibilities so there are probably a ton of bugs that I don't even know about.

How can the community help?
You can help by installing these test builds, test the new features and provide feedback and bug reports.

Test builds are available in the second post of this thread: http://forum.kodi.tv/showthread.php?tid=...id1986017.
The source code is located in my Kodi fork on github (https://github.com/Montellese/xbmc) in the media_import_redesign branch: https://github.com/Montellese/xbmc/tree/...t_redesign
It is based on a recent version of the master branch from April 18 2015: https://github.com/xbmc/xbmc/commit/6e5a...6a9fb3c11a

Once you've installed one (or more) of these test builds on multiple devices you are ready to go:
  1. Start (at least) two of the test build installations on the same local network
  2. Make sure that the UPnP server is enabled (see Settings -> Services -> UPnP/DLNA -> Share video and music libraries through UPnP
  3. Kodi should automatically detect the other installation as a potential media provider and show a Kai toast with the name and the logo of the detected media provider
  4. Go to Settings -> Media sources -> Media imports -> By media providers and you should see a list of detected media providers by their name
  5. Open one of the listed media providers and you should see a list of defined media imports (obviously this will be empty at the beginning)
  6. Choose Add import.. to create a new import and then choose one of the listed media types from the dialog
  7. You will see the Media import information dialog with some details about the import and some settings which you can adjust to your liking (these are not fully integrated yet)
  8. Close the dialog and Kodi will automatically start synchronising all media items of the specified media type from the media provider
  9. Repeat the last three steps for all media types you'd like to import items from the media provider

Once you've setup the media synchronisation for one or more media providers Kodi will automatically snychronise the imported media items whenever it detects a known media provider (e.g. after startup or when the media provider is started/restarted). You can also manually synchronise media items by going to Settings -> Media sources -> Media imports and use the Synchronise context option on a media provider or media import.

IMPORTANT:
To test this you need to be able to install the same build on (at least) two different devices and they should have access to different media items so that they can be synchronised between the two installations. You need to use the same test builds i.e. you can't synchronise a test build with an existing Kodi Helix or Isengard Alpha/Beta build.

Remember to backup any existing installation and its data (specifically Userdata_folder (wiki)) because this will perform an update of your (video) database!!!

I think I found a bug. What should I do?
  • Provide your Debug log (wiki) and if it crashed also provide the crash dump / stack trace / mini dump (Windows-specific in combination with the PDB file)
  • Provide a detailed description what exactly went wrong and what the expected behaviour is.
  • Provide a detailed description (ideally step by step) how to reproduce the problem.

Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
(This post was last modified: 2016-12-18 16:31 by Montellese.)
find quote
Montellese Online
Team Kodi Developer
Posts: 4,846
Joined: Jan 2009
Reputation: 72
Location: Switzerland
Post: #2
These builds are based on the Kodi Krypton Betas and not on Kodi Jarvis

If you ran a previous version of this test build you need to delete your MyVideosXX.db

Latest test builds: December 10 2016:
Changelog
November 15 2016:
Changelog
  • Rebased onto Krypton Beta: https://github.com/xbmc/xbmc/commit/e9c9...c6165f5deb
  • Reworked media import settings to use the same underlying system as media provider settings.
  • Added a new category to media import settings for Emby which support choosing the libraries/views to import.
  • Extended the emby media importer to add the library/view an item belongs to as a tag.

November 04 2016:
Changelog
  • Support user authentication against Emby server for both public and hidden accounts

October 30 2016:
Changelog
  • Rebased onto current master
  • Improved media import GUI window to manage media imports
  • Support importing video items from an Emby server
  • Added more events

October 10 2015:
Changelog
  • Rebased onto current master: 783db620754bf4b89f96247f20f295603f1c8e1a
  • Improved media import GUI window to manage media imports

September 17 2015:
Changelog
  • Rebased onto current master: b9d1772d274ac766f920aa822457a6a3be373f92
  • Improved the handling of tvshows, seasons and episodes
  • Fixed importing a lot of empty seasons and then deleting them again during cleanup
  • Fixed importing tvshows which also exist as local files in the library

August 16 2015:
Changelog
  • Rebased onto current master: 2c3a5413080dbe1ab5771ce043f4946c015f5da5
  • Added an event for newly detected media providers which will take you directly to the media providers import configuration view.
  • If you ran a previous version of this test build you need to delete your MyVideos92.db and newer

June 23 2015:
Changelog
  • Rebased onto current master: 95a4fdc3200d584a4079d84267ad7a76fde37961
  • If you ran a previous version of this test build you need to delete your MyVideos92.db and MyVideos93.db

May 17 2015:
Changelog
  • Rebased onto current master: fcf7d8882098c3ffee6e56058225c6992d7e4625

May 12 2015:
Changelog
  • Fixed saving file state of watched videos and updating the changes on the media provider

April 22 2015:
Changelog
  • Remove season and tvshow imports after removing an episode import.
  • Marking an imported tvshow and/or season as watched also properly marks all its episodes as watched
  • Updating the watched state on the client applies the same update on the media provider if it is available

April 21 2015:
Changelog
  • Fixed video database update from version 91 and 92
  • Removed the kai toast notification about incompatible media providers (log message is still there)
  • Improved season import/cleanup handling

Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
(This post was last modified: 2016-12-10 09:53 by Montellese.)
find quote
Montellese Online
Team Kodi Developer
Posts: 4,846
Joined: Jan 2009
Reputation: 72
Location: Switzerland
Post: #3
Known bugs:
  • "Unable to import" kai toast is shown every time an incompatible media provider is detected.
  • "All seasons" artwork is broken due to a bad merge conflict resolution in CVideoDatabase::UpdateTables() and CVideoDatabase::GetSchemaVersion().
  • Some entries from the seasons table don't end up in the season_view view due to missing paths.
  • Removing an episode import does not properly remove the season and tvshow imports.
  • The settings window for a newly added "TV shows, seasons & episodes" does not automatically open and synchronisation doesn't automatically start
  • Marking an imported tvshow or season as watched doesn't mark its episodes as watched
  • Can't play imported items from home screen widgets
  • Updating playcount of an imported item doesn't update the playcount of the original item on the media provider
  • Imported tvshows aren't properly matched with identical local tvshows and lead to duplicate entries in the library.
  • Confused

TODOs:
  • Artwork isn't re-synchronised (right now this is disabled on purpose because the IP of the UPnP server is not always the same which results in constantly re-synchronising all items)
  • "Scan for new items" context menu on imported tvshows doesn't make much sense
  • "Update library" doesn't synchronise imported items
  • Add "Synchronise" and "Cancel" buttons to the "Media import information" dialog
  • Allow external VFS access to imported items
  • UPnP
    • Chapter bookmarks and their screenshots are not synchronised
    • Actor/Artist/Person artwork is not supported
  • Emby
    • Unable to synchronise resume points with server
    • Import collections from server
    • Listen for changes on the server using websockets
  • Confused

Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
(This post was last modified: 2016-12-12 09:02 by Montellese.)
find quote
natethomas Offline
XBMC Chief and Kodi Project Manager
Posts: 6,299
Joined: Apr 2008
Reputation: 133
Location: Kansas
Post: #4
First note: watching all my tv shows get imported into my tablet is really satisfying.

Second, it'd be nice if Kodi somehow recognized it was a clean install and offered to immediately send you to the media provider list to begin importing, rather than simply saying an import was available and then doing nothing excepting waiting for the user to know to navigate to the importer.

Third, it looks like Kodi recognizes the HDHomerun as a media source, but gives an error about it. "Unable to import items." This isn't so bad, but it's annoying to get the error every time I boot.

Here's a log of most recent boot on android with the error.

http://xbmclogs.com/pylsxmvum
(This post was last modified: 2015-04-20 12:34 by natethomas.)
find quote
natethomas Offline
XBMC Chief and Kodi Project Manager
Posts: 6,299
Joined: Apr 2008
Reputation: 133
Location: Kansas
Post: #5
Next problem. It appears as though season folder support is tricky. When I beginning scanning in a TV source, I can browse season folders pretty easily, but when the import process is complete those folders seem to disappear.

http://xbmclogs.com/poywfxgya
find quote
Montellese Online
Team Kodi Developer
Posts: 4,846
Joined: Jan 2009
Reputation: 72
Location: Switzerland
Post: #6
(2015-04-20 12:32)natethomas Wrote:  Second, it'd be nice if Kodi somehow recognized it was a clean install and offered to immediately send you to the media provider list to begin importing, rather than simply saying an import was available and then doing nothing excepting waiting for the user to know to navigate to the importer.
I had that behaviour a while ago but it did that all the time which became a huge PITA because it started changing windows while I was doing something completely else. Another problem is when multiple media providers are detected in a short time span. The first one throws you into the proper window and while you add an import a new one is detected and throws you into another path of the same window (and so on).
IMO this will be improved with the event log functionality where you'll be able to click on the event/notification and be taken to the proper window. But I'm open to any suggestions on how to improve this.

(2015-04-20 12:32)natethomas Wrote:  Third, it looks like Kodi recognizes the HDHomerun as a media source, but gives an error about it. "Unable to import items." This isn't so bad, but it's annoying to get the error every time I boot.
Yeah it's known and I'll put it on the TODO list. I'll probably simply remove the kai toast for now. Maybe we should keep a list of incompatible providers and only show the kai toast the very first time such a provider is detected. It can still be written into the debug log though to be able to see that something didn't go right.

Thanks a lot for the feedback.

Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
find quote
Montellese Online
Team Kodi Developer
Posts: 4,846
Joined: Jan 2009
Reputation: 72
Location: Switzerland
Post: #7
(2015-04-20 13:21)natethomas Wrote:  Next problem. It appears as though season folder support is tricky. When I beginning scanning in a TV source, I can browse season folders pretty easily, but when the import process is complete those folders seem to disappear.

http://xbmclogs.com/poywfxgya

Thanks for the log. There is some unideal behaviour due to the fact that we seem to store seasons in the database which don't have any episodes. This results in the client importing all the seasons and then during the cleanup realizes that all these seasons don't have any episodes and removes them again. But that should only affect seasons that are also empty on the media provider (and special seasons like "All seasons" and "Specials" which are always added to the database). Probably best would be to not synchronise these seasons at all in the first place but I ran into some issue with that as well but I don't remember what it was.

Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
find quote
rterblanche Offline
Member
Posts: 95
Joined: Mar 2012
Reputation: 0
Post: #8
Love playing with betas. Just to be sure. Which folders do I need to backup
find quote
Montellese Online
Team Kodi Developer
Posts: 4,846
Joined: Jan 2009
Reputation: 72
Location: Switzerland
Post: #9
(2015-04-20 15:12)rterblanche Wrote:  Love playing with betas. Just to be sure. Which folders do I need to backup

Mainly Userdata_folder (wiki).

Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
find quote
RockerC Offline
Posting Freak
Posts: 1,518
Joined: May 2011
Reputation: 31
Post: #10
(2015-04-19 14:08)Montellese Wrote:  While this concept is aimed at specialized providers like UPnP servers etc. it could theoretically also be applied to the current library scanning and scraping. The provider would then be a specific server (or the local filesystem) and an import would just be a combination of a specific path on that server and a media type to import from that path.
Does this mean that once your initial framework is merged it could then be extended to reimplemented SpotyXBMC2 in an acceptable way? http://forum.kodi.tv/showthread.php?tid=67012

The idea behind SpotyXBMC is also explained as an unified music addon frontend concept which is a feature that is discussed here http://forum.kodi.tv/showthread.php?tid=105147

Sorry if off-topic or too early to ask about this, and maybe it is better to instead continue that part as a separate discussion in the above thread that is in the feature suggestion forum?
(This post was last modified: 2015-04-20 16:40 by RockerC.)
find quote
Montellese Online
Team Kodi Developer
Posts: 4,846
Joined: Jan 2009
Reputation: 72
Location: Switzerland
Post: #11
(2015-04-20 16:38)RockerC Wrote:  Does this mean that once your initial framework is merged it could then be extended to reimplemented SpotyXBMC2 in an acceptable way? http://forum.kodi.tv/showthread.php?tid=67012

The idea behind SpotyXBMC is also explained as an unified music addon frontend concept which is a feature that is discussed here http://forum.kodi.tv/showthread.php?tid=105147

Sorry if off-topic or too early to ask about this, and maybe it is better to instead continue that part as a separate discussion in the above thread that is in the feature suggestion forum?

I haven't taken a close look at SpotyXBMC2 but if the functionality is extended to support importing items from addons/plugins you "just" need a plugin that provides all artists, albums and songs and they can then be imported into your local library.
Another example for something that would become easier and has been requested a lot would be an addon which provides a list of all episodes that are missing from your tvshow library and they could be imported and then you'd see them alongside your existing episodes.

Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
(This post was last modified: 2015-04-20 18:53 by Montellese.)
find quote
rterblanche Offline
Member
Posts: 95
Joined: Mar 2012
Reputation: 0
Post: #12
Ok very impressed.

Running on one central kodi with most of the media and then have a Pi 2 connecting to the main instance.

Setup was easy. Just a funny behaviour when adding multiple sources. Settings window took a while to open and when I added the second source some of the settings windows never opened. Think you mentioned you know about that behaviour and it will be fixed with the events system.

Will let you know if I find any issues
find quote
rterblanche Offline
Member
Posts: 95
Joined: Mar 2012
Reputation: 0
Post: #13
First funny.

Doing a mark as watched or play on a tv series level doesn't mark the episode in that tv series as played. Going into tv series and play or mark as watched worked
find quote
LongMan Online
Senior Member
Posts: 163
Joined: Apr 2013
Reputation: 0
Post: #14
Beautiful work. I have been waiting a long time for this day. Thanks for your efforts.

A small request. Would be possible to allow the user to select sources via the upnp browse menu? I am asking because I use video nodes on my Home menu based on paths in the source.

<?xml version="1.0" encoding="UTF-8" standalone="true"?>
-<node visible="True" type="filter" order="3">
<label>KIDS MOVIES</label>
<content>movies</content>
<match>all</match>
<rule operator="contains" field="path">
<value>Kids Movies</value>
</rule>
<order direction="ascending">title</order>
</node>.


This worked beautifully for the local database, and I have a little system that I use to share the library via UPnP using other corresponding nodes.

<?xml version="1.0" encoding="UTF-8" standalone="true"?>
<node visible="True" type="folder" order="3">
<label>KIDS MOVIES</label>
<path pathversion="1">upnp://UUID/library%3a%2f%2fvideo%2fMenus%2fMain%2fMain3(kids movies).xml%2f/</path>
</node>


In the new media import version, the upnp path listed on the info screen, does not include the "Kids Movies" part, so, my system will no longer work. Is it possible to have

upnp://UUID/library://video/Kids Movies/ etc
instead of
upnp://UUID/videodb://movies etc.

I am assuming if this is possible then the separation would still work. Maybe you can suggest another way for me to do it.

Cheers,
(This post was last modified: 2015-04-20 23:14 by LongMan.)
find quote
LongMan Online
Senior Member
Posts: 163
Joined: Apr 2013
Reputation: 0
Post: #15
Unexpected Behaviour

The items in recently added does not play when they are selected. Neither movies or episodes.

Cheers

Edit: ie The recently added icons on the Home Screen.
(This post was last modified: 2015-04-21 00:00 by LongMan.)
find quote
Post Reply