[pieh]

Proposal: Aggregate media from multiple sources and present them in unified library view

Summary:
Project aims to aggregate remote sources content (for start from UPnP shares or by using xbmc JSON RPC API) and present it as part of unified library. If UPnP share or xbmc instance is discovered in network, metadata could be fetched in background and results cached in local database. Additionally, special plugins could be added to populate library.
How will I achieve this:
Background service that will discover, fetch metadata from UPnP shares and cache it will need to be implemented. Aggregating content from multiple sources will likely results in some duplicates so mechanism to group duplicates.
What will the project focus on:
Most important part of this project is to fetch metadata from remote sources, inspect it and cache it for future quick use. It is critical this feature sould not cause performance hit when browsing library.
Benefits:
It will benefit users with multiple xbmc instances making content sharing much easier.
Goals:
Goal is to aggregate and present metadata from remote shares in simple way without any configuration needed. Subgoals:
- discover and fetch metadata from remote sources
- cache metadata from remote sources
- do housekeeping of cached metadata
What does it touch in XBMC:
This project would require some changes to library databases: mostly adding atributes like source, optional expiration time, etc to media tables. Services to discover remote shares and fetch metodata will need to be implemented.
Requirements:
No special hardware is needed for this project. c++ will be used.


Note: I submitted this proposal before deadline!

[pieh]

And followup post with more details:

How will I do it?

• Use UPnP search feature to get list of all movies ( object.item.videoItem.movie ), etc. This need some work on search capabilities of xbmc upnp server and also need code to issue searches in xbmc upnp media browser (stubs are already there, just need some utilities to make it usable). This is already done in my Proof of Concept.
  
• Import media to database: movies are pretty straight forward, only thing needed is to add "source" column to keep track of non-local movies. TV shows potentially require more work. We assume there is one root directory for tv show episodes and this won't work in long term when aggregating medias from multiple sources. Will need to make sure that stored root directory is used only as information for video info scanner - potentially adding another table to allow 1:N mapping (allow to have tv show episodes placed in multiple paths). Songs and music videos should be straight forward too.

Quirk:
Stating files for dateAdded value is taking considerable ammount of importing time. We can avoid stating if this would be passed with UPnP data. In my PoC I repurposed dc:date member of media object to carry dateAdded rather than premeried or first aired date. This had great impact on import time (around 300 seconds to import 410 movies with stating went down to 100 seconds when getting date directly from UPnP). Ultimately dateAdded shouldn't use dc:date and use something like xbmc:dateAdded (?)

Demo of my Proof of Concept: http://www.youtube.com/watch?v=HfQTGclBfpY
Note: This is on my dev machine with ssd so database operations are really fast (as in importing lasts 10s instead of 100s). I measure time using same library on my old notebook that acts as media center now to get more realistic values.

Ideas to explore:
I expect that importing on low powered boxes will take longer than in my tests. Doing full import each time we discover UPnP server is not really an option then. I'll try to implement method to compare stored (from previous fetches) media with newly fetched ones and compile needed database operation based on differences between 2 media lists.

[pecinko]

Hmm.. Just a few thoughts.

So all movie sources from all upnp servers are getting merged into one MOVIES on a client?
I can see a lot of reasons people would find it useful to have them as separate home menu items - Movies (server1), Kids movies (server2) etc. Theoretically, if I add friends xbmc server with his videos, it should probably stay separate from my media?

[pieh]

Hmm.. Just a few thoughts.

So all movie sources from all upnp servers are getting merged into one MOVIES on a client?
I can see a lot of reasons people would find it useful to have them as separate home menu items - Movies (server1), Kids movies (server2) etc. Theoretically, if I add friends xbmc server with his videos, it should probably stay separate from my media?

That's the whole plan to have all media from whatever sources (local, upnp, possibly plugins) in same place with minimal effort (no mysql / advancedsettings). This doesn't disable option to seperate them later (f.e. via smart playlists with rule "media from source X"). Obviously feature might just be turned off if You don't want to mix content from multiple sources. Having them seperated actually looks similiar to how it is now - add "upnp servers" in video files and you get list of servers and from there you can navigate where you want.

On sidenote, surely home window customization might be painful here: we don't have way to do it from core actually as skinner is master of home menu

[pecinko]

That's the whole plan to have all media from whatever sources (local, upnp, possibly plugins) in same place with minimal effort (no mysql / advancedsettings). This doesn't disable option to seperate them later (f.e. via smart playlists with rule "media from source X"). Obviously feature might just be turned off if You don't want to mix content from multiple sources. Having them seperated actually looks similiar to how it is now - add "upnp servers" in video files and you get list of servers and from there you can navigate where you want.

Thanks for info.

I have tried to gather pieces of info regarding planned changes but did not have much luck, so I might be missing either info or the whole point. So the big plan is - every XBMC instance is a client and upnp server at the same time. Sources are aggregated and presented as unified library on every client? If turned off, aggregation won't take place but those sources would still be available in "sources" node.

What happens with aggregation if user prefers to have all media in one place as opposed to distributed model? I suppose all remains the same and aggregation is done with one source only. Practically this means I would have a linux server with XBMC running and than turn on aggregation on bedroom and living room's htpc. This would make those htpcs pull library data from a remote to local DB. What happens if remote instance is having 2 profiles? What happens if you change art on media item - is it changed locally, on remote instance or both? In order to delete media one would need to do it on a source upnp server, I suppose?

In a case htpc is on low powered device it might be interesting to do aggregation, but keep DB data on a remote upnp server while caching thumbs locally.

On sidenote, surely home window customization might be painful here: we don't have way to do it from core actually as skinner is master of home menu

I suppose it could be done as all system containers are right now - there's a container with reserved ID, list content is populated by XBMC, while container layouts are defined by the skinner (viz DialogSelect and similar).

As for separation via smart playlists I believe I was expecting something more in line with custom nodes. In case of Confluence skin container id=9000 with main list items getting populated by aggregator deamon, while say container id=9001 for submenu items would get populated with filter items for each media type, while filter items are configurable via custom nodes.

[pieh]

I have tried to gather pieces of info regarding planned changes but did not have much luck, so I might be missing either info or the whole point. So the big plan is - every XBMC instance is a client and upnp server at the same time. Sources are aggregated and presented as unified library on every client?

Yes, big plan is to create distributed system.

If turned off, aggregation won't take place but those sources would still be available in "sources" node.

If turned off then nothing changes - to browse upnp servers you have to to video window -> Files -> UPnP Devices.

What happens with aggregation if user prefers to have all media in one place as opposed to distributed model? I suppose all remains the same and aggregation is done with one source only. Practically this means I would have a linux server with XBMC running and than turn on aggregation on bedroom and living room's htpc. This would make those htpcs pull library data from a remote to local DB.

Correct. This is distributed model but still can be used as single server with many clients.

What happens if remote instance is having 2 profiles?

This will be using UPnP and so only one 1 profile is active and serving media at the time. For now any changes to that part are out of the scope of this proposal but might be added as optional goals to research.

What happens if you change art on media item - is it changed locally, on remote instance or both? In order to delete media one would need to do it on a source upnp server, I suppose?

I plan to use UpdateObject method from UPnP Content Directory spec to allow making changes on clients (same mechanism that is used to sync resume point / watched status now). Ofcourse it's still up to upnp server to allow and/or support it (this is optional part of upnp spec) - remember that this will be based on UPnP so we're not talking only about xbmc acting as upnp server.

In a case htpc is on low powered device it might be interesting to do aggregation, but keep DB data on a remote upnp server while caching thumbs locally.

This would work in single server - many clients scenario. With multiple upnp servers in LAN this would mean browsing would be quite painful as it would need to join results on the fly (remember that we will need process them first to merge any duplicates). That's why I believe storing/caching full db on first import (might take some time) and later doing only updates better approach. I understand that You'd like sources seperated and then Your idea would work well. But then again this is what we already have.

I suppose it could be done as all system containers are right now - there's a container with reserved ID, list content is populated by XBMC, while container layouts are defined by the skinner (viz DialogSelect and similar).

As for separation via smart playlists I believe I was expecting something more in line with custom nodes. In case of Confluence skin container id=9000 with main list items getting populated by aggregator deamon, while say container id=9001 for submenu items would get populated with filter items for each media type, while filter items are configurable via custom nodes.

I meant that it would force certain layout of home window (i.e. main menu with sub menu in your example). This might not work for some skins - skin.xeebo for example. Still this could be added to library main node.

[Montellese]

@pieh: Are you planning to write a generic interface to allow populating the database with items from different sources/servers or will it be very UPnP-specific?
I'm asking because I could see this getting extended in the future to e.g. allow plugins to "inject" media items (e.g. youtube videos etc) into the library or to allow the user to manually add a plugin item to the library. It would also need some management functionality because the "external" media items might not always be present or might need to be removed complete. There probably won't be any metadata updating back to the source but basically it would also be an aggregation of media items from different sources, just not UPnP sources.

[pieh]

I'm not planning to make this UPnP specific, but UPnP will be first "external" source that will use it and that's why I use it so much as example. This will be quite simple actually - I'll just add indexed source column (this might be UUID of upnp device or id of new kind of plugin) and possibly more attributes like expireDate for plugin items. I was actually thinking of this kind of plugins when I wrote vague "Additionally, special plugins could be added to populate library." in summary of proposal.

As for managment I plan to do housekeeping of imported media. Using UPnP I will check when upnp server is being disconnected and mark all its items as inactive (not delete). When server will be discovered again I will pull content and compare stored media list and newly fetched list and compile needed database operations based on differences (remove, add, update). I have something similiar in mind with plugins - xbmc will ask library.plugin (might give it some name for discussion sake) for all movies, etc on startup (+ maybe option to do manual refresh and/or set refresh time interval). I think this will make sure that there won't be left any orphaned/forgotten media in database.

I'd say updating metadata back to plugin would be secondary goal - won't be used by youtube plugin for sure, but could be used by plexbmc plugin propably.

[Montellese]

Cool, because I had this as an idea on my TODO list for possible future features I'd like to implement / see implemented

[Bstrdsmkr]

How would you handle plugins that don't have access to all movies that they can provide in a reasonable fashion? For example, some sites only list say 25 items at a time and don't have a full content listing on one page, so it would require a bunch of page requests to list everything so you can do the comparison.

[LongMan]

This is really good. Since discovering the MyLibrary plugin (http://forum.xbmc.org/showthread.php?tid=98210) I have been trying to figure out a way to scan files to my library with the upnp path. I tried .strm files which worked, but I could not find a way to get the path of all the media easily. I really appreciate your work here. Upnp sharing is really nice, but, because it uses just one profile it is not ideal for me. The setup is one profile in each bedroom/or tablet and all profiles in living room. Easy with mysql, and very pretty with the new artwork syncing in Frodo. Unable to achieve the same with upnp.

The problem is restore point. Hopefully, your enhancement will store the restore point locally so that even though, the restore point is not per user/profile, it could be per installation. At least one could use the send to plugin to continue watching on a another machine. As oppose to losing the restore point if someone else starts the movie on a tablet

[pieh]

How would you handle plugins that don't have access to all movies that they can provide in a reasonable fashion? For example, some sites only list say 25 items at a time and don't have a full content listing on one page, so it would require a bunch of page requests to list everything so you can do the comparison.

I must admit I didn't think of it earlier and assumed that plugin can get data in single call via some api.

There's nothing we can do about that - we will have to wait for plugin to gather all available media (this is needed to do proper housekeeping of stored items). What we can do is to allow plugins to insert items in batches - one for each page and it could enable media added in batch. And if there is no more media to add plugin might just state it and that would trigger cleaning up of outdated medias. This will be done in background so fact that it will take longer to gather media shouldn't affect browsing much - it won't block gui to import media!

This is really good. Since discovering the MyLibrary plugin (http://forum.xbmc.org/showthread.php?tid=98210) I have been trying to figure out a way to scan files to my library with the upnp path. I tried .strm files which worked, but I could not find a way to get the path of all the media easily. I really appreciate your work here. Upnp sharing is really nice, but, because it uses just one profile it is not ideal for me. The setup is one profile in each bedroom/or tablet and all profiles in living room. Easy with mysql, and very pretty with the new artwork syncing in Frodo. Unable to achieve the same with upnp.

The problem is restore point. Hopefully, your enhancement will store the restore point locally so that even though, the restore point is not per user/profile, it could be per installation. At least one could use the send to plugin to continue watching on a another machine. As oppose to losing the restore point if someone else starts the movie on a tablet

So in Your case problem is that we sync restore point between upnp clients and upnp server AND that upnp server doesn't care about profiles (or more acurately - it cares only about current profile)? To make use of multiple profiles we'd need to authenticate client - client "logs in" as X profile on server and then server use library from X profile. This could propably be done using subset of optional UPnP Device Protection specs ( http://www.upnp.org/specs/gw/UPnP-gw-Dev...110224.pdf ) - will take a look at it.

[Montellese]

How would you handle plugins that don't have access to all movies that they can provide in a reasonable fashion? For example, some sites only list say 25 items at a time and don't have a full content listing on one page, so it would require a bunch of page requests to list everything so you can do the comparison.

I must admit I didn't think of it earlier and assumed that plugin can get data in single call via some api.

There's nothing we can do about that - we will have to wait for plugin to gather all available media (this is needed to do proper housekeeping of stored items). What we can do is to allow plugins to insert items in batches - one for each page and it could enable media added in batch. And if there is no more media to add plugin might just state it and that would trigger cleaning up of outdated medias. This will be done in background so fact that it will take longer to gather media shouldn't affect browsing much - it won't block gui to import media!

IMO any plugin that can't list all the media items in a single call provides so many items that it wouldn't make sense to blindly add them all to the library. Nobody wants all the Youtube videos in their library but users might want to manually add a few ones when browsing in the youtube addon. For other plugins (like e.g. Netflix) however it makes sense that all the items are added to the library but those shouldn't have a problem to provide them all in one go.

[alcoheca]

This could propably be done using subset of optional UPnP Device Protection specs ( http://www.upnp.org/specs/gw/UPnP-gw-Dev...110224.pdf ) - will take a look at it.

I thought about that - but it's pretty complicated, and wouldn't help non-xbmc devices as clients (none of which seem to support device protection). What we could do though is better monitoring of clients, associating a particular profile to them. By default they would see and 'act' as the master profile, but for instance we could assign a 'kids' profile to the UUID of a DLNA Tv in the playroom that way they'd only see database items available to the kids profile and any watched status / resume point tracking would stick to that profile.

btw I'm very excited by this proposal!

[LongMan]

"This could propably be done using subset of optional UPnP Device Protection specs ( http://www.upnp.org/specs/gw/UPnP-gw-Dev...110224.pdf ) - will take a look at it." Pieh

"I thought about that - but it's pretty complicated..........we could assign a 'kids' profile to the UUID of a DLNA Tv in the playroom that way they'd only see database items available to the kids profile and any watched status / resume point tracking would stick to that profile." Alcoheca

As deducted the restore point is the main sticking point. The libraries are actually subsets of the whole and could be achieve with smart playlists which are already supported in UPNP. Truth be told the profile system is XBMC is a beauty. It allows so much customization, that from profile to profile on the same computer, one could have a radically different user experience. I would not be surprised if it was very complicated to do that in UPNP.

Having said that, I am reminded that XBMC itself has made simple what many others have tried, failed at, or produced a solution that is as complex, if not more complex than the problem being solved. You guys make what others see as impossible, just improbable. And,what they see as improbable, possible. I suspect that whatever solution you implement will be more than enough. In fact what both have you have done so far is more than I expected and has made my media experience much better. Thanks for your work.