+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Discussions (https://forum.kodi.tv/forumdisplay.php?fid=222)
+--- Forum: Kodi related discussions (https://forum.kodi.tv/forumdisplay.php?fid=6)
+--- Thread: Native Object-Based Storage Support for XBMC (/showthread.php?tid=183747)
RE: Native Object-Based Storage Support for XBMC - jacintech.fire - 2014-01-23
@nickr,
There is NONE right now.
There are implementations of using an Object Store w/ a mapping layer that presents the objects as files and directories (pretty much ALL the online storage sites fall into that category). OpenStack Swift w/Samba is what I am experimenting on right now.
Doing that is much easier, but it only solves the storage problem (i.e infinitely scalable storage with built in redundancy and fail over). Interacting with it leaves you with thousands upon thousands of file and directories; which is not much improvement on the way things are right now...
The closest to such an unlimited scale would be google's youtube (but I don't think google would appreciate me publishing their code)...
But if you read the openStack API documentation (there is a python API binding as well) is not difficult to see how it could be implemented...if I had the resources and skills to pull this off on my own; I would not have come to this list in the first place. I would have done it on my own and reap all the glory, don't ya think :-)
RE: Native Object-Based Storage Support for XBMC - jacintech.fire - 2014-01-28
Progress: Openstack (in all its wondrous glory) is up and running!!!
Next UP: Pushing and Pulling data into/outta the Object Store.....
RE: Native Object-Based Storage Support for XBMC - nickr - 2014-01-28
This seems very complex for the average home user's 10-20T of movies.
We understand that you can install openstack, or any other software. There seems to be a plethora of documentation for that on the intarwebbies.
But how do you propose to integrate this into xbmc? From what I have seen you describe, the interesting part of the object based system is that the object store contains the metadata as well as the file the metadata describes. Therefore the object store will not only have a movie file, but also the title, subtitle, plot summary, a list of actors, studio, fanart etc etc.
So are you proposing that the objectstore metadata replaces the xbmc database? Or that the xbmc database is populated from the object store and remains a separate database? If so, should that happen dynamically so that changes in the object sotre are immediately reflected in the xbmc database, or should it only happen when you "scan" - as at present.
Also how does the metadata get into the object store in the first place?
What are the performance advantages/disadvantages? There seems an awful lot of abstraction involved and abstraction requires processing.
The current model of a database containing text items (title, plot summary etc) and pointers to binary blobs (the movie file, the fanart etc) seems pretty darn efficient to me, and with a properly tuned database is unlikely to be beaten on performance by some cloud technology. And one constant theme around here is "how can I speed up browsing my movie collection?"
Looking forward to your comments. I still want to know it beats my mhddfs combined filesystem!
(edited for typos)
RE: Native Object-Based Storage Support for XBMC - jacintech.fire - 2014-01-28
@nickr,
I want to setup a working testbed before I start "flapping my gums". There are multitude of wrinkles I have to iron out first. I want to post actual screenshots of the setup at work, to speak...
Sufice to say you hit the nail squarely in the head with some of your comments...specifically with the issue of virtualization/abstraction. One quick note about that is that you are pooling your resources (specifically with openstack) and the processing is done at the cluster level. Even with all the abstraction layers, when you are dealing with 3, 5, 8, or 10 nodes it begins to be aparent...
But again, I want to have something up and running before I go off on a tangent...
RE: Native Object-Based Storage Support for XBMC - nickr - 2014-01-28
By 'nodes' are you talking 'computers'? Because if so, again I think there are very very few xbmc users who want to devote more than one, maybe two computers to act as their media servers.
RE: Native Object-Based Storage Support for XBMC - poplap - 2014-01-29
I can see most not wanting to fallow along with sort of model (though I would love to if i had the money) but more likely it would be a rented cloud similar to Amazon S3 but maybe a bit more consumer.
Now when it comes to performance it would actually be slower then a traditional file system, both a limit in objs but also scale of the clients in his case. See objs systems are best at archiving, things that only ever really get read rather then written to (though there are ways around this) but in his case he will have only a few clients (well i assume less then 50 or so) and large traffic is where objs shines being able to load balance many servers. Where traditional servers performance scaled with number of drives/net IO/etc objs system scale with servers. Many large scale setups may have only 1 or 2 gig nics per server but with a good couple hundred nodes allowing for many clients. This model doesn't make much sense for the home server but rather places like Netflix or YouTube or even as backup system. To get around the speed problem many use ssd banks or more traditional systems (but still enterprise) that can be integrated in to the cloud system using the API but still a little impractical to consumers.
I give it him, 512tb is amazing and with his setup objs makes sense but his is a rare case. I can see people renting storage for their moves sense it can be REALLY cheap thanks to the scale at which they are build and with a fast internet connection you could stream just about everything. It would be awesome if XBMC supported cloud storage backed in but it could also be a little more ideal if something like this could be made into an Addon but with lower system control to act natively in XBMC, replacing the central mySQL and adding in the files.
The power of objs systems is amazing but maybe out of place for most homes but renting is always an option and it should be looked at for XBMC in the near future, just my .02
RE: Native Object-Based Storage Support for XBMC - jacintech.fire - 2014-01-29
@poplap,
WOW you have come closer than most to figure this out. You came right at the door, then turned back. Read your comments again...you had the answer right there....
RE: Native Object-Based Storage Support for XBMC - poplap - 2014-01-29
Thanks but I'm not a 100% what you talking about but if you're talking about the solution to the OP about XBMC supporting cloud storage systems, I feel it should if not in the same way as a traditional filesystem. Sense there are a HUGE number of APIs it would make more sense to use an Addon based system much like the PVR system (In fact i think that more of XBMC should be like that, but thats for another post). Separately maintained Addons that can present XBMC with content and maybe even all the metadata and DB like functions (though that may muddy things up if there would be more than one). The simplest approach I could think of would be to extend the python API to allow lower level control over XBMC (maybe with some safeguards for security), basicly a Source addon category. Not to mention there are some nice libraries for REST APIs (most common cloud API system) for python rather than trying to sort that out in C/C++.
I would be willing to commit code to this but as a full time student with a part time job and volunteering at a lab my schedule is a little tight. Also not to familiar with the XBMC code base so there would be overhead with learning that.
RE: Native Object-Based Storage Support for XBMC - jacintech.fire - 2014-01-29
@poplap,
I hope you can understand my reluctance to get too specific; I don't want to taint the evolution of this concept with my still-unfinished, still-evolving ideas...
I have choosen OpenStack for my own experiment because I believe it offers the most flexibility in terms of what could be acomplished down the line. Given its modular structure, I can start experimenting with only those parts I want to use right now and leave the rest for when my own proficiency improves...
Also, given the backing of heavy hitters like Red Hat and others, I figure this project has legs...
I am hoping, that as the discusion evolves, other better ideas will surface from other members with skills and experience vastly superior to mine...
RE: Native Object-Based Storage Support for XBMC - poplap - 2014-01-30
Ah ok I think I get. I do have to agree that Openstack is the best option that I can see, great feature set and expanding. (Though I do have a bias to FOSS)
I do have one question how did you go about setting up Openstack, do you have a controller node, network node or is it just a sift system. My biggest problem with setting up my own Openstack system is that I would only have room (and money) for two servers, file and Virtual machines, but not a controller nor do I want to complicate my network. Every guide I see is for a larger system with a full Openstack setup.
RE: Native Object-Based Storage Support for XBMC - jacintech.fire - 2014-01-31
@poplap,
That is why my storage pods are GNU/Linux based; you can always re-purpose (re-task) if you need to...
RE: Native Object-Based Storage Support for XBMC - Harro - 2014-01-31
Can I ask what your end goal is?
Is it to be like a Netflix or other streaming services or an all in one type of datastack which would include movie, metadata.etc..?
RE: Native Object-Based Storage Support for XBMC - jacintech.fire - 2014-02-01
(2014-01-31, 21:34)Harro Wrote: Can I ask what your end goal is?
Is it to be like a Netflix or other streaming services or an all in one type of datastack which would include movie, metadata.etc..?
Jeezzz
To be (or beat) Netflix one would need billions of dollars...I don't think the XBMC community is that rich
RE: Native Object-Based Storage Support for XBMC - nickr - 2014-02-01
I don't think you got the point of the question. I don't think Harro was suggesting some sort of publicly available service, imply something like netflix for your own personal use.
Unless you think you can secure the rights for millions of copyrighted movies and tv shows?
RE: Native Object-Based Storage Support for XBMC - jacintech.fire - 2014-02-01
@nickr,
oh, I never would have thought of that...
I wouldn't even know where to start...and for what end...