• 1(current)
  • 2
  • 3
  • 4
  • 5
  • 96
WIP Media importing and library integration (UPnP, Emby, Plex, ...)
#1
Hey everyone

As some of you know I've been working on media importing and library integration for almost two years now (with a lot of breaks and very little free time) and I've demonstrated my work during the last two devcons and while my work was always publicly available on github I've never provided any test builds or asked the wider community for input because I first wanted to get the core design and implementation into a shape where it can be tested before having to deal with the flood of feature requests. In the last few days I finally had time to resolve some of the outstanding larger issues, bugs and TODOs and now it's finally time to get more people on board to test this, provide feedback and bug reports because there's only so much testing I can do myself and if you designed and implemented a whole system you don't really see any (better) alternatives anymore.

What's this all about?
The general idea is to provide a framework to be able to import media items from all kinds of services and locations which are not supported by our current library scanners which are purely filesystem based. Examples are importing some or all items from a plugin or from a UPnP server. This will allow users to directly integrate these items into their usual library so when they go through their library they'll also see items that are not directly available as a file on their disks/NAS but are available through another service and can play them like any other library item (as long as the service is available).

How does it work?
The new concept introduces the terms media providers and media imports. A media provider is an entity that can provide a set of media items and maybe also a hierarchy (filesystem). A media import is a pointer to a location in the hierarchy of a provider in combination with a list of media items (movies, tvshows, ...) that should be imported from that specific provider and location. Therefore there can be several imports for a single provider (e.g. multiple specific paths of a plugin). While this concept is aimed at specialized providers like UPnP servers etc. it could theoretically also be applied to the current library scanning and scraping. The provider would then be a specific server (or the local filesystem) and an import would just be a combination of a specific path on that server and a media type to import from that path. To handle providers and imports there are provider repositories which are just database specific implementation and right now there's just one for storing and retrieving the providers and imports from the video database.

To do the actual work of importing media items there's the media import manager. It has access to provider repositories to know about the existing providers/imports and to add/remove them. Furthermore it has access to specific implementations of import handlers and media handlers.

Import handlers are specific implementations which can get a list of media items of specific media types from a provider/import. Right now there's just an implementation for UPnP which uses the Search() request specified for UPnP DirectoryProvider implementations to request items of a specific media type.

Media handlers are media type specific implementations which know how to add items of a specific media type into the local library. That task involves getting all previously imported items of a specific provider, comparing the list of imported items with the list of previously imported items and then deciding which items to add, remove and update in the local library.

So how does importing items work?
  1. When Kodi starts up it disables all library items that were imported from a provider because it doesn't know if the provider is available right now. Disabled is just a state of an item and could be used in skins to e.g. grey out these items in the list so that the user sees the item as part of the library but realizes that the source it came from is not available right now.
  2. When Kodi discovers a (new) provider (e.g. a UPnP server) it tells the media import manager that a source is available.
    1. The media import manager checks if it already knows the provider or not.
    2. If it doesn't know the provider, it starts a provider registration task
      1. The provider registration task checks (based on the provider identifier/path) if there's an import handler capable of importing media items from that provider or not (for UPnP we e.g. require the UPnP server to support searching for a specific "upnp:class" property).
      2. After that new media imports can be added for the registered provider in Settings -> Media sources -> Media imports -> By media provider -> Add import...
    3. If there are imports defined for the detected provider, start an import retrieval task which will use the import handler capable of handling the detected provider to grab all items of the media types chosen by the user for this import
    4. If the import retrieval task was successful, start a media synchronisation task for every media type (using the proper media handler) chosen by the user for this import and provide it with the list of imported items
    5. The media handler takes care of synchronising the already imported items from that provider/import with the just previously imported ones
This whole process has been implemented using jobs and tasks (basically sub-jobs) which are run in a seperate thread in the background. They won't interfer with any media playback so if the user is watching a movie and a new UPnP server pops up, he won't see any dialog on the screen until he stops playback. Furthermore there's feedback through the progress bar dialog during the whole import process (similar to how it works for good old library updates).

What is already working/there?
Right now most of the retrieval, synchronisation and cleanup logic is there in a generic form. Importing is only supported from UPnP servers which are also provided by a compatible Kodi version (i.e. this specific test build). Furthermore it only supports importing items into the video library i.e. movies, tvshows, seasons, episodes and music videos. There's no support for the music library yet.

What isn't working?
A lot. I only have limited testing time and possibilities so there are probably a ton of bugs that I don't even know about.

How can the community help?
You can help by installing these test builds, test the new features and provide feedback and bug reports.

Test builds are available in the second post of this thread: http://forum.kodi.tv/showthread.php?tid=...pid1986017.
The source code is located in my Kodi fork on github (https://github.com/Montellese/xbmc) in the media_import_redesign branch: https://github.com/Montellese/xbmc/tree/...t_redesign
It is based on a recent version of the master branch from April 18 2015: https://github.com/xbmc/xbmc/commit/6e5a...6a9fb3c11a

Once you've installed one (or more) of these test builds on multiple devices you are ready to go:
  1. Start (at least) two of the test build installations on the same local network
  2. Make sure that the UPnP server is enabled (see Settings -> Services -> UPnP/DLNA -> Share video and music libraries through UPnP
  3. Kodi should automatically detect the other installation as a potential media provider and show a Kai toast with the name and the logo of the detected media provider
  4. Go to Settings -> Media sources -> Media imports -> By media providers and you should see a list of detected media providers by their name
  5. Open one of the listed media providers and you should see a list of defined media imports (obviously this will be empty at the beginning)
  6. Choose Add import.. to create a new import and then choose one of the listed media types from the dialog
  7. You will see the Media import information dialog with some details about the import and some settings which you can adjust to your liking (these are not fully integrated yet)
  8. Close the dialog and Kodi will automatically start synchronising all media items of the specified media type from the media provider
  9. Repeat the last three steps for all media types you'd like to import items from the media provider

Once you've setup the media synchronisation for one or more media providers Kodi will automatically snychronise the imported media items whenever it detects a known media provider (e.g. after startup or when the media provider is started/restarted). You can also manually synchronise media items by going to Settings -> Media sources -> Media imports and use the Synchronise context option on a media provider or media import.

IMPORTANT:
To test this you need to be able to install the same build on (at least) two different devices and they should have access to different media items so that they can be synchronised between the two installations. You need to use the same test builds i.e. you can't synchronise a test build with an existing Kodi Helix or Isengard Alpha/Beta build.

Remember to backup any existing installation and its data (specifically Userdata_folder (wiki)) because this will perform an update of your (video) database!!!

I think I found a bug. What should I do?
  • Provide your Debug log (wiki) and if it crashed also provide the crash dump / stack trace / mini dump (Windows-specific in combination with the PDB file)
  • Provide a detailed description what exactly went wrong and what the expected behaviour is.
  • Provide a detailed description (ideally step by step) how to reproduce the problem.
Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
Reply
#2
Please see https://github.com/Montellese/xbmc/wiki/Downloads for the latest Kodi test builds and the changelog.
Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
Reply
#3
Please see https://github.com/Montellese/xbmc/issue...+Import%22 for a list of known bugs / issues, improvements and feature requests.
Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
Reply
#4
First note: watching all my tv shows get imported into my tablet is really satisfying.

Second, it'd be nice if Kodi somehow recognized it was a clean install and offered to immediately send you to the media provider list to begin importing, rather than simply saying an import was available and then doing nothing excepting waiting for the user to know to navigate to the importer.

Third, it looks like Kodi recognizes the HDHomerun as a media source, but gives an error about it. "Unable to import items." This isn't so bad, but it's annoying to get the error every time I boot.

Here's a log of most recent boot on android with the error.

http://xbmclogs.com/pylsxmvum
Reply
#5
Next problem. It appears as though season folder support is tricky. When I beginning scanning in a TV source, I can browse season folders pretty easily, but when the import process is complete those folders seem to disappear.

http://xbmclogs.com/poywfxgya
Reply
#6
(2015-04-20, 12:32)natethomas Wrote: Second, it'd be nice if Kodi somehow recognized it was a clean install and offered to immediately send you to the media provider list to begin importing, rather than simply saying an import was available and then doing nothing excepting waiting for the user to know to navigate to the importer.
I had that behaviour a while ago but it did that all the time which became a huge PITA because it started changing windows while I was doing something completely else. Another problem is when multiple media providers are detected in a short time span. The first one throws you into the proper window and while you add an import a new one is detected and throws you into another path of the same window (and so on).
IMO this will be improved with the event log functionality where you'll be able to click on the event/notification and be taken to the proper window. But I'm open to any suggestions on how to improve this.

(2015-04-20, 12:32)natethomas Wrote: Third, it looks like Kodi recognizes the HDHomerun as a media source, but gives an error about it. "Unable to import items." This isn't so bad, but it's annoying to get the error every time I boot.
Yeah it's known and I'll put it on the TODO list. I'll probably simply remove the kai toast for now. Maybe we should keep a list of incompatible providers and only show the kai toast the very first time such a provider is detected. It can still be written into the debug log though to be able to see that something didn't go right.

Thanks a lot for the feedback.
Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
Reply
#7
(2015-04-20, 13:21)natethomas Wrote: Next problem. It appears as though season folder support is tricky. When I beginning scanning in a TV source, I can browse season folders pretty easily, but when the import process is complete those folders seem to disappear.

http://xbmclogs.com/poywfxgya

Thanks for the log. There is some unideal behaviour due to the fact that we seem to store seasons in the database which don't have any episodes. This results in the client importing all the seasons and then during the cleanup realizes that all these seasons don't have any episodes and removes them again. But that should only affect seasons that are also empty on the media provider (and special seasons like "All seasons" and "Specials" which are always added to the database). Probably best would be to not synchronise these seasons at all in the first place but I ran into some issue with that as well but I don't remember what it was.
Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
Reply
#8
Love playing with betas. Just to be sure. Which folders do I need to backup
Reply
#9
(2015-04-20, 15:12)rterblanche Wrote: Love playing with betas. Just to be sure. Which folders do I need to backup

Mainly Userdata_folder (wiki).
Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
Reply
#10
(2015-04-19, 14:08)Montellese Wrote: While this concept is aimed at specialized providers like UPnP servers etc. it could theoretically also be applied to the current library scanning and scraping. The provider would then be a specific server (or the local filesystem) and an import would just be a combination of a specific path on that server and a media type to import from that path.
Does this mean that once your initial framework is merged it could then be extended to reimplemented SpotyXBMC2 in an acceptable way? http://forum.kodi.tv/showthread.php?tid=67012

The idea behind SpotyXBMC is also explained as an unified music addon frontend concept which is a feature that is discussed here http://forum.kodi.tv/showthread.php?tid=105147

Sorry if off-topic or too early to ask about this, and maybe it is better to instead continue that part as a separate discussion in the above thread that is in the feature suggestion forum?
Reply
#11
(2015-04-20, 16:38)RockerC Wrote: Does this mean that once your initial framework is merged it could then be extended to reimplemented SpotyXBMC2 in an acceptable way? http://forum.kodi.tv/showthread.php?tid=67012

The idea behind SpotyXBMC is also explained as an unified music addon frontend concept which is a feature that is discussed here http://forum.kodi.tv/showthread.php?tid=105147

Sorry if off-topic or too early to ask about this, and maybe it is better to instead continue that part as a separate discussion in the above thread that is in the feature suggestion forum?

I haven't taken a close look at SpotyXBMC2 but if the functionality is extended to support importing items from addons/plugins you "just" need a plugin that provides all artists, albums and songs and they can then be imported into your local library.
Another example for something that would become easier and has been requested a lot would be an addon which provides a list of all episodes that are missing from your tvshow library and they could be imported and then you'd see them alongside your existing episodes.
Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
Reply
#12
Ok very impressed.

Running on one central kodi with most of the media and then have a Pi 2 connecting to the main instance.

Setup was easy. Just a funny behaviour when adding multiple sources. Settings window took a while to open and when I added the second source some of the settings windows never opened. Think you mentioned you know about that behaviour and it will be fixed with the events system.

Will let you know if I find any issues
Reply
#13
First funny.

Doing a mark as watched or play on a tv series level doesn't mark the episode in that tv series as played. Going into tv series and play or mark as watched worked
Reply
#14
Beautiful work. I have been waiting a long time for this day. Thanks for your efforts.

A small request. Would be possible to allow the user to select sources via the upnp browse menu? I am asking because I use video nodes on my Home menu based on paths in the source.

<?xml version="1.0" encoding="UTF-8" standalone="true"?>
-<node visible="True" type="filter" order="3">
<label>KIDS MOVIES</label>
<content>movies</content>
<match>all</match>
<rule operator="contains" field="path">
<value>Kids Movies</value>
</rule>
<order direction="ascending">title</order>
</node>.


This worked beautifully for the local database, and I have a little system that I use to share the library via UPnP using other corresponding nodes.

<?xml version="1.0" encoding="UTF-8" standalone="true"?>
<node visible="True" type="folder" order="3">
<label>KIDS MOVIES</label>
<path pathversion="1">upnp://UUID/library%3a%2f%2fvideo%2fMenus%2fMain%2fMain3(kids movies).xml%2f/</path>
</node>


In the new media import version, the upnp path listed on the info screen, does not include the "Kids Movies" part, so, my system will no longer work. Is it possible to have

upnp://UUID/library://video/Kids Movies/ etc
instead of
upnp://UUID/videodb://movies etc.

I am assuming if this is possible then the separation would still work. Maybe you can suggest another way for me to do it.

Cheers,
Reply
#15
Unexpected Behaviour

The items in recently added does not play when they are selected. Neither movies or episodes.

Cheers

Edit: ie The recently added icons on the Home Screen.
Reply
  • 1(current)
  • 2
  • 3
  • 4
  • 5
  • 96

Logout Mark Read Team Forum Stats Members Help
Media importing and library integration (UPnP, Emby, Plex, ...)10