Aggregate media from multiple sources and present them in unified library view

  Thread Rating:
  • 1 Vote(s) - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
Bstrdsmkr Offline
Posting Freak
Posts: 802
Joined: Oct 2010
Reputation: 17
Post: #16
(2013-05-18 17:39)Montellese Wrote:  
(2013-05-18 16:47)pieh Wrote:  
(2013-05-17 17:09)Bstrdsmkr Wrote:  How would you handle plugins that don't have access to all movies that they can provide in a reasonable fashion? For example, some sites only list say 25 items at a time and don't have a full content listing on one page, so it would require a bunch of page requests to list everything so you can do the comparison.
I must admit I didn't think of it earlier and assumed that plugin can get data in single call via some api.

There's nothing we can do about that - we will have to wait for plugin to gather all available media (this is needed to do proper housekeeping of stored items). What we can do is to allow plugins to insert items in batches - one for each page and it could enable media added in batch. And if there is no more media to add plugin might just state it and that would trigger cleaning up of outdated medias. This will be done in background so fact that it will take longer to gather media shouldn't affect browsing much - it won't block gui to import media!

IMO any plugin that can't list all the media items in a single call provides so many items that it wouldn't make sense to blindly add them all to the library. Nobody wants all the Youtube videos in their library but users might want to manually add a few ones when browsing in the youtube addon. For other plugins (like e.g. Netflix) however it makes sense that all the items are added to the library but those shouldn't have a problem to provide them all in one go.

That's pretty much what I was thinking, but when you're generalizing it like this you'd need some way to know that X plugin doesn't support full listing

On a similar topic, what about upper limits? For example if a plugin can provide 40,000 TV Shows all at once is the client going to crash and burn?
find quote
Montellese Offline
Team Kodi Developer
Posts: 4,843
Joined: Jan 2009
Reputation: 72
Location: Switzerland
Post: #17
(2013-05-19 04:11)Bstrdsmkr Wrote:  
(2013-05-18 17:39)Montellese Wrote:  
(2013-05-18 16:47)pieh Wrote:  I must admit I didn't think of it earlier and assumed that plugin can get data in single call via some api.

There's nothing we can do about that - we will have to wait for plugin to gather all available media (this is needed to do proper housekeeping of stored items). What we can do is to allow plugins to insert items in batches - one for each page and it could enable media added in batch. And if there is no more media to add plugin might just state it and that would trigger cleaning up of outdated medias. This will be done in background so fact that it will take longer to gather media shouldn't affect browsing much - it won't block gui to import media!

IMO any plugin that can't list all the media items in a single call provides so many items that it wouldn't make sense to blindly add them all to the library. Nobody wants all the Youtube videos in their library but users might want to manually add a few ones when browsing in the youtube addon. For other plugins (like e.g. Netflix) however it makes sense that all the items are added to the library but those shouldn't have a problem to provide them all in one go.

That's pretty much what I was thinking, but when you're generalizing it like this you'd need some way to know that X plugin doesn't support full listing

On a similar topic, what about upper limits? For example if a plugin can provide 40,000 TV Shows all at once is the client going to crash and burn?

We could ask plugin authors to specify that in the addon.xml i.e. they specify whether they are able to provide all of the media items at once or not. Furthermore IMO the user should always specify from which plugins to automatically pull data instead of doing it magically as soon as the plugin has been installed. Then we could just not provide that option for plugins that don't have that ability.

Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
find quote
RockerC Offline
Posting Freak
Posts: 1,516
Joined: May 2011
Reputation: 31
Post: #18
topfs2 and akezeke discussed a similar great about having a unified media library for both addons and other backends here

http://forum.xbmc.org/showthread.php?tid=105147


Please also look at SpotyXBMC2 implementation for some inspiration ideas for this. If not its code then at least its concept.

https://github.com/akezeke/spotyxbmc2

(2013-05-19 09:08)Montellese Wrote:  
(2013-05-19 04:11)Bstrdsmkr Wrote:  
(2013-05-18 17:39)Montellese Wrote:  IMO any plugin that can't list all the media items in a single call provides so many items that it wouldn't make sense to blindly add them all to the library. Nobody wants all the Youtube videos in their library but users might want to manually add a few ones when browsing in the youtube addon. For other plugins (like e.g. Netflix) however it makes sense that all the items are added to the library but those shouldn't have a problem to provide them all in one go.

That's pretty much what I was thinking, but when you're generalizing it like this you'd need some way to know that X plugin doesn't support full listing

On a similar topic, what about upper limits? For example if a plugin can provide 40,000 TV Shows all at once is the client going to crash and burn?

We could ask plugin authors to specify that in the addon.xml i.e. they specify whether they are able to provide all of the media items at once or not. Furthermore IMO the user should always specify from which plugins to automatically pull data instead of doing it magically as soon as the plugin has been installed. Then we could just not provide that option for plugins that don't have that ability.
I think that SpotyXBMC2 have solved this issue in a very good way, as it aggregates only your selected "stared" artists, albums, and playlists from Spotify online music library and mixes those with your existing local stored music files in the same library view.

You could imagine the same thing for Netflix or Hulu, where you would XBMC's library to only display your favorite movies and tv shows when browsing.

But you should still be able to search the entire Netflix or Hulu catalog somehow, or go into to only that addon to browse it and mark your favorites as "stared" or add them to a "watch later" queue.

So XBMC's library should only aggregate media items inside addons that the user have marked as "favorites", instead of aggregating the entire Spotify online catalog into XBMC's library and mix all those with your locally stored music files.


(This post was last modified: 2013-05-22 16:39 by RockerC.)
find quote
pieh Offline
Team-Kodi Member
Posts: 672
Joined: Aug 2010
Reputation: 13
Location: Poland
Post: #19
(2013-05-19 04:11)Bstrdsmkr Wrote:  That's pretty much what I was thinking, but when you're generalizing it like this you'd need some way to know that X plugin doesn't support full listing

On a similar topic, what about upper limits? For example if a plugin can provide 40,000 TV Shows all at once is the client going to crash and burn?
(2013-05-19 09:08)Montellese Wrote:  We could ask plugin authors to specify that in the addon.xml i.e. they specify whether they are able to provide all of the media items at once or not. Furthermore IMO the user should always specify from which plugins to automatically pull data instead of doing it magically as soon as the plugin has been installed. Then we could just not provide that option for plugins that don't have that ability.
I actually don't want existing plugins to populate library - they do not fit here. Making seperate kind of addon could also solve problems with what to import - they could much more specialised, i.e. while we might want Hulu plugin to browse entire Hulu collection, we would only want to import just items from your hulu favourites and/or recently added hulu content (both represented by separate addons). This way You can manage what goes in your library with simply enabling these single purpose addons.

(2013-05-22 16:31)RockerC Wrote:  topfs2 and akezeke discussed a similar great about having a unified media library for both addons and other backends here

http://forum.xbmc.org/showthread.php?tid=105147
Yes, I remember. I wanted to tackle this for a long time now Wink

(2013-05-22 16:31)RockerC Wrote:  Please also look at SpotyXBMC2 implementation for some inspiration ideas for this. If not its code then at least its concept.

https://github.com/akezeke/spotyxbmc2
This approach might work with single service, but it won't scale well if you'll want to add more services. I might use some code to use spotify as my test case though.

(2013-05-22 16:31)RockerC Wrote:  I think that SpotyXBMC2 have solved this issue in a very good way, as it aggregates only your selected "stared" artists, albums, and playlists from Spotify online music library and mixes those with your existing local stored music files in the same library view.
Yes, that's pretty much how I see it - add media that are owned / starred. But really, this would be up to addon creators to use common sense and decide what to add and of course up to user to decide if and what addons he want to use. I don't plan to limit anything here.

Always read the XBMC online-manual, FAQ and search the forums before posting.
Do NOT e-mail Team-XBMC members asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting, make sure you read this first

My previous forum/trac nickname: grajen3
(This post was last modified: 2013-05-26 22:22 by pieh.)
find quote
Bstrdsmkr Offline
Posting Freak
Posts: 802
Joined: Oct 2010
Reputation: 17
Post: #20
Have you given any thought about handling "limited time" content as well? For example, I star an old movie in Hulu and next week it gets removed from the catalog. It's still in my favorites list, but not really available. Would the addon have to verify each favorite each time the list is updated? (might cause unnecessary strain on the host site)
find quote
Montellese Offline
Team Kodi Developer
Posts: 4,843
Joined: Jan 2009
Reputation: 72
Location: Switzerland
Post: #21
(2013-05-27 04:15)Bstrdsmkr Wrote:  Have you given any thought about handling "limited time" content as well? For example, I star an old movie in Hulu and next week it gets removed from the catalog. It's still in my favorites list, but not really available. Would the addon have to verify each favorite each time the list is updated? (might cause unnecessary strain on the host site)

http://forum.xbmc.org/showthread.php?tid...pid1423151

Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
find quote
pieh Offline
Team-Kodi Member
Posts: 672
Joined: Aug 2010
Reputation: 13
Location: Poland
Post: #22
(2013-05-27 04:15)Bstrdsmkr Wrote:  Have you given any thought about handling "limited time" content as well? For example, I star an old movie in Hulu and next week it gets removed from the catalog. It's still in my favorites list, but not really available. Would the addon have to verify each favorite each time the list is updated? (might cause unnecessary strain on the host site)
It is really addon's job to provide up to date media listing. There's really not much we can do on XBMC side to handle that as it would really just bounce request "is this movie valid" back to addon anyway. This would cause unnecessary strain on host site only in cases like this - when media references are not removed together with media itself or not marked as orphaned.

My personal stance on this is if site doesn't provide (public) API, it really doesn't want other way of consuming its medias than what it already provides. Omitting this and doing web scraping might just cause bad impressions. I know this is not what users would like to hear as they would probably love netflix, hulu, etc integrated but that's really not our choice to make Smile

Always read the XBMC online-manual, FAQ and search the forums before posting.
Do NOT e-mail Team-XBMC members asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting, make sure you read this first

My previous forum/trac nickname: grajen3
find quote
Kib Offline
Team Kodi 🐤 Member
Posts: 5,033
Joined: Jan 2010
Reputation: 87
Location: The Hague
Post: #23
Congratulations on having your proposal accepted!
find quote
LongMan Offline
Senior Member
Posts: 162
Joined: Apr 2013
Reputation: 0
Post: #24
Congratulations. I am looking forward to this.
find quote
pettergulbra Offline
Fan
Posts: 722
Joined: Jun 2009
Reputation: 1
Post: #25
This looks really nice..
Main thing for me is to get all my clients in to the library. With upnp, not with MySQL.
And that the watched status it updated over the clients etc. So I could start watching someplace, finish another.

nVidia Shield TV, Samsung UE55F8005STXXE, BenQ W1060 and Yamaha RX-V767

Many thanks for all the effort YOU all do! THANKS! :-)
find quote
pettergulbra Offline
Fan
Posts: 722
Joined: Jun 2009
Reputation: 1
Post: #26
Is this thread died out?

nVidia Shield TV, Samsung UE55F8005STXXE, BenQ W1060 and Yamaha RX-V767

Many thanks for all the effort YOU all do! THANKS! :-)
find quote
Shred17 Offline
Junior Member
Posts: 19
Joined: Oct 2013
Reputation: 0
Post: #27
Did this project ever get off the ground?
find quote
Montellese Offline
Team Kodi Developer
Posts: 4,843
Joined: Jan 2009
Reputation: 72
Location: Switzerland
Post: #28
As part of GSoC 2013 only partly but I've picked up the work that has been done by pieh and I'm still working on it.

Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
find quote
rterblanche Offline
Member
Posts: 95
Joined: Mar 2012
Reputation: 0
Post: #29
Nice. You might be the best to ask. Any news on the GSOC work around UPnP. Did that ever happen
find quote
Montellese Offline
Team Kodi Developer
Posts: 4,843
Joined: Jan 2009
Reputation: 72
Location: Switzerland
Post: #30
Not that I know of, no.

Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
find quote