2015-04-19, 14:08
Hey everyone
As some of you know I've been working on media importing and library integration for almost two years now (with a lot of breaks and very little free time) and I've demonstrated my work during the last two devcons and while my work was always publicly available on github I've never provided any test builds or asked the wider community for input because I first wanted to get the core design and implementation into a shape where it can be tested before having to deal with the flood of feature requests. In the last few days I finally had time to resolve some of the outstanding larger issues, bugs and TODOs and now it's finally time to get more people on board to test this, provide feedback and bug reports because there's only so much testing I can do myself and if you designed and implemented a whole system you don't really see any (better) alternatives anymore.
What's this all about?
The general idea is to provide a framework to be able to import media items from all kinds of services and locations which are not supported by our current library scanners which are purely filesystem based. Examples are importing some or all items from a plugin or from a UPnP server. This will allow users to directly integrate these items into their usual library so when they go through their library they'll also see items that are not directly available as a file on their disks/NAS but are available through another service and can play them like any other library item (as long as the service is available).
How does it work?
The new concept introduces the terms media providers and media imports. A media provider is an entity that can provide a set of media items and maybe also a hierarchy (filesystem). A media import is a pointer to a location in the hierarchy of a provider in combination with a list of media items (movies, tvshows, ...) that should be imported from that specific provider and location. Therefore there can be several imports for a single provider (e.g. multiple specific paths of a plugin). While this concept is aimed at specialized providers like UPnP servers etc. it could theoretically also be applied to the current library scanning and scraping. The provider would then be a specific server (or the local filesystem) and an import would just be a combination of a specific path on that server and a media type to import from that path. To handle providers and imports there are provider repositories which are just database specific implementation and right now there's just one for storing and retrieving the providers and imports from the video database.
To do the actual work of importing media items there's the media import manager. It has access to provider repositories to know about the existing providers/imports and to add/remove them. Furthermore it has access to specific implementations of import handlers and media handlers.
Import handlers are specific implementations which can get a list of media items of specific media types from a provider/import. Right now there's just an implementation for UPnP which uses the Search() request specified for UPnP DirectoryProvider implementations to request items of a specific media type.
Media handlers are media type specific implementations which know how to add items of a specific media type into the local library. That task involves getting all previously imported items of a specific provider, comparing the list of imported items with the list of previously imported items and then deciding which items to add, remove and update in the local library.
So how does importing items work?
What is already working/there?
Right now most of the retrieval, synchronisation and cleanup logic is there in a generic form. Importing is only supported from UPnP servers which are also provided by a compatible Kodi version (i.e. this specific test build). Furthermore it only supports importing items into the video library i.e. movies, tvshows, seasons, episodes and music videos. There's no support for the music library yet.
What isn't working?
A lot. I only have limited testing time and possibilities so there are probably a ton of bugs that I don't even know about.
How can the community help?
You can help by installing these test builds, test the new features and provide feedback and bug reports.
Test builds are available in the second post of this thread: http://forum.kodi.tv/showthread.php?tid=...pid1986017.
The source code is located in my Kodi fork on github (https://github.com/Montellese/xbmc) in the media_import_redesign branch: https://github.com/Montellese/xbmc/tree/...t_redesign
It is based on a recent version of the master branch from April 18 2015: https://github.com/xbmc/xbmc/commit/6e5a...6a9fb3c11a
Once you've installed one (or more) of these test builds on multiple devices you are ready to go:
Once you've setup the media synchronisation for one or more media providers Kodi will automatically snychronise the imported media items whenever it detects a known media provider (e.g. after startup or when the media provider is started/restarted). You can also manually synchronise media items by going to Settings -> Media sources -> Media imports and use the Synchronise context option on a media provider or media import.
IMPORTANT:
To test this you need to be able to install the same build on (at least) two different devices and they should have access to different media items so that they can be synchronised between the two installations. You need to use the same test builds i.e. you can't synchronise a test build with an existing Kodi Helix or Isengard Alpha/Beta build.
Remember to backup any existing installation and its data (specifically Userdata_folder (wiki)) because this will perform an update of your (video) database!!!
I think I found a bug. What should I do?
As some of you know I've been working on media importing and library integration for almost two years now (with a lot of breaks and very little free time) and I've demonstrated my work during the last two devcons and while my work was always publicly available on github I've never provided any test builds or asked the wider community for input because I first wanted to get the core design and implementation into a shape where it can be tested before having to deal with the flood of feature requests. In the last few days I finally had time to resolve some of the outstanding larger issues, bugs and TODOs and now it's finally time to get more people on board to test this, provide feedback and bug reports because there's only so much testing I can do myself and if you designed and implemented a whole system you don't really see any (better) alternatives anymore.
What's this all about?
The general idea is to provide a framework to be able to import media items from all kinds of services and locations which are not supported by our current library scanners which are purely filesystem based. Examples are importing some or all items from a plugin or from a UPnP server. This will allow users to directly integrate these items into their usual library so when they go through their library they'll also see items that are not directly available as a file on their disks/NAS but are available through another service and can play them like any other library item (as long as the service is available).
How does it work?
The new concept introduces the terms media providers and media imports. A media provider is an entity that can provide a set of media items and maybe also a hierarchy (filesystem). A media import is a pointer to a location in the hierarchy of a provider in combination with a list of media items (movies, tvshows, ...) that should be imported from that specific provider and location. Therefore there can be several imports for a single provider (e.g. multiple specific paths of a plugin). While this concept is aimed at specialized providers like UPnP servers etc. it could theoretically also be applied to the current library scanning and scraping. The provider would then be a specific server (or the local filesystem) and an import would just be a combination of a specific path on that server and a media type to import from that path. To handle providers and imports there are provider repositories which are just database specific implementation and right now there's just one for storing and retrieving the providers and imports from the video database.
To do the actual work of importing media items there's the media import manager. It has access to provider repositories to know about the existing providers/imports and to add/remove them. Furthermore it has access to specific implementations of import handlers and media handlers.
Import handlers are specific implementations which can get a list of media items of specific media types from a provider/import. Right now there's just an implementation for UPnP which uses the Search() request specified for UPnP DirectoryProvider implementations to request items of a specific media type.
Media handlers are media type specific implementations which know how to add items of a specific media type into the local library. That task involves getting all previously imported items of a specific provider, comparing the list of imported items with the list of previously imported items and then deciding which items to add, remove and update in the local library.
So how does importing items work?
- When Kodi starts up it disables all library items that were imported from a provider because it doesn't know if the provider is available right now. Disabled is just a state of an item and could be used in skins to e.g. grey out these items in the list so that the user sees the item as part of the library but realizes that the source it came from is not available right now.
- When Kodi discovers a (new) provider (e.g. a UPnP server) it tells the media import manager that a source is available.
- The media import manager checks if it already knows the provider or not.
- If it doesn't know the provider, it starts a provider registration task
- The provider registration task checks (based on the provider identifier/path) if there's an import handler capable of importing media items from that provider or not (for UPnP we e.g. require the UPnP server to support searching for a specific "upnp:class" property).
- After that new media imports can be added for the registered provider in Settings -> Media sources -> Media imports -> By media provider -> Add import...
- The provider registration task checks (based on the provider identifier/path) if there's an import handler capable of importing media items from that provider or not (for UPnP we e.g. require the UPnP server to support searching for a specific "upnp:class" property).
- If there are imports defined for the detected provider, start an import retrieval task which will use the import handler capable of handling the detected provider to grab all items of the media types chosen by the user for this import
- If the import retrieval task was successful, start a media synchronisation task for every media type (using the proper media handler) chosen by the user for this import and provide it with the list of imported items
- The media handler takes care of synchronising the already imported items from that provider/import with the just previously imported ones
- The media import manager checks if it already knows the provider or not.
What is already working/there?
Right now most of the retrieval, synchronisation and cleanup logic is there in a generic form. Importing is only supported from UPnP servers which are also provided by a compatible Kodi version (i.e. this specific test build). Furthermore it only supports importing items into the video library i.e. movies, tvshows, seasons, episodes and music videos. There's no support for the music library yet.
What isn't working?
A lot. I only have limited testing time and possibilities so there are probably a ton of bugs that I don't even know about.
How can the community help?
You can help by installing these test builds, test the new features and provide feedback and bug reports.
Test builds are available in the second post of this thread: http://forum.kodi.tv/showthread.php?tid=...pid1986017.
The source code is located in my Kodi fork on github (https://github.com/Montellese/xbmc) in the media_import_redesign branch: https://github.com/Montellese/xbmc/tree/...t_redesign
It is based on a recent version of the master branch from April 18 2015: https://github.com/xbmc/xbmc/commit/6e5a...6a9fb3c11a
Once you've installed one (or more) of these test builds on multiple devices you are ready to go:
- Start (at least) two of the test build installations on the same local network
- Make sure that the UPnP server is enabled (see Settings -> Services -> UPnP/DLNA -> Share video and music libraries through UPnP
- Kodi should automatically detect the other installation as a potential media provider and show a Kai toast with the name and the logo of the detected media provider
- Go to Settings -> Media sources -> Media imports -> By media providers and you should see a list of detected media providers by their name
- Open one of the listed media providers and you should see a list of defined media imports (obviously this will be empty at the beginning)
- Choose Add import.. to create a new import and then choose one of the listed media types from the dialog
- You will see the Media import information dialog with some details about the import and some settings which you can adjust to your liking (these are not fully integrated yet)
- Close the dialog and Kodi will automatically start synchronising all media items of the specified media type from the media provider
- Repeat the last three steps for all media types you'd like to import items from the media provider
Once you've setup the media synchronisation for one or more media providers Kodi will automatically snychronise the imported media items whenever it detects a known media provider (e.g. after startup or when the media provider is started/restarted). You can also manually synchronise media items by going to Settings -> Media sources -> Media imports and use the Synchronise context option on a media provider or media import.
IMPORTANT:
To test this you need to be able to install the same build on (at least) two different devices and they should have access to different media items so that they can be synchronised between the two installations. You need to use the same test builds i.e. you can't synchronise a test build with an existing Kodi Helix or Isengard Alpha/Beta build.
Remember to backup any existing installation and its data (specifically Userdata_folder (wiki)) because this will perform an update of your (video) database!!!
I think I found a bug. What should I do?
- Provide your Debug log (wiki) and if it crashed also provide the crash dump / stack trace / mini dump (Windows-specific in combination with the PDB file)
- Provide a detailed description what exactly went wrong and what the expected behaviour is.
- Provide a detailed description (ideally step by step) how to reproduce the problem.
