• 1
  • 2
  • 3
  • 4(current)
  • 5
  • 6
  • 22
Native Object-Based Storage Support for XBMC
#46
With openstack you should be able to mount it at the OS level and then have XBMC connect to it from there, you would just make an object and use a program that can mount swift systems, there are a few open source ones out there.

I'm rather impressed with your storage, I always wanted to set up a OpenStack system but never could justify the hardware cost. Now you just needs some blades for VMs
Reply
#47
(2014-01-19, 17:10)prawnee Wrote: Yeah, yeah, but your scientists were so preoccupied with whether or not they could that they didn't stop to think if they should. Rolleyes


Dude,
When it comes to sience (computer, or otherwise); if you can...you should. It's what drives evolution. I am NOT afraid to be ridiculed; that allows an unprecedented degree of freedom to experiment.we have been told that for large dataset storage, an implementation of RAID was a MUST. Most of us accept this without question; I, on the other hand decided to experiment.

My question was: given the technological advances in manufacturing and materials engineering (and the years of research and development spent to improve these products),wouldn't the hard drives produced today be more resilient and reliable; and if this is so, would a large dataset survive a traditional storage mechanism?

Well, 16x 4TB in 8 nodes says it can...
Reply
#48
@poplap,
If I remember correctly, you can even export OpenStack Swift as a samba share...
What I would like is a mechanism to manipulate the object store directly, with full API access, without the need of a mapping layer...
Effectively combining what MYSQL and Samba do today into a single implementation...
Reply
#49
I have to say a few things, and I mean this all with all due respect:

1). Amazing rack setup. Nobody here is questioning the awesomeness of the 512TB of data storage. I want! BUT, unless you're fully utilizing over 300TB of data your setup is not smart for multiple reasons:
I) you've boxed yourself into a super rigid setup which although may "work", is by no means even close to optimal.
II). Just because you've made it 3 years, doesn't mean you'll last 4... Or 5... Unless you really don't care about the data, why in the world would you not setup a RAID / Storage Pool on Windows so that you can recover from bad disks? You WILL lose data.
III). If you aren't using all the space why would you spend so much up front, when you will be able to buy probably 8TB drives in 2 years for the same price? If you had set it up properly, you could just buy additional disks and have 1024TB for the same price, with no extra work.

2). If you had setup storage pool or raid (which I laugh that you "think" is obsolete), you could have EASILY and automagically) accomplished what you are trying to do via links/junction points. In Linux inotifywait could have been used to automate, and in Windows, there are tools.

3). A question and two points if you don't mind:
Q). What is your profession?
I). You are mad if you are comparing your ignorance / stubbornness of this convoluted setup to Einstein.
II). You are clearly not an IT Professional. I'm not in Infra, but I was an IT Developer, then Team Lead, and now a Dev Manager. I say this because if you told me you had this setup in our pre-interview chat, I would not hire you because I see a lack of foresight, planning, and a strange arrogance and inability to accept / think that when many smart people are tell him that this is not good, it might actually BE WRONG.


My two cents, and probably my last post on this thread... You don't want to listen to advice, and I can promise you that this won't get into XBMC unless you develop it yourself....
Reply
#50
(2014-01-19, 18:33)edrikk Wrote: I have to say a few things, and I mean this all with all due respect:

1). Amazing rack setup. Nobody here is questioning the awesomeness of the 512TB of data storage. I want! BUT, unless you're fully utilizing over 300TB of data your setup is not smart for multiple reasons:
I) you've boxed yourself into a super rigid setup which although may "work", is by no means even close to optimal.
II). Just because you've made it 3 years, doesn't mean you'll last 4... Or 5... Unless you really don't care about the data, why in the world would you not setup a RAID / Storage Pool on Windows so that you can recover from bad disks? You WILL lose data.
III). If you aren't using all the space why would you spend so much up front, when you will be able to buy probably 8TB drives in 2 years for the same price? If you had set it up properly, you could just buy additional disks and have 1024TB for the same price, with no extra work.

2). If you had setup storage pool or raid (which I laugh that you "think" is obsolete), you could have EASILY and automagically) accomplished what you are trying to do via links/junction points. In Linux inotifywait could have been used to automate, and in Windows, there are tools.

3). A question and two points if you don't mind:
Q). What is your profession?
I). You are mad if you are comparing your ignorance / stubbornness of this convoluted setup to Einstein.
II). You are clearly not an IT Professional. I'm not in Infra, but I was an IT Developer, then Team Lead, and now a Dev Manager. I say this because if you told me you had this setup in our pre-interview chat, I would not hire you because I see a lack of foresight, planning, and a strange arrogance and inability to accept / think that when many smart people are tell him that this is not good, it might actually BE WRONG.


My two cents, and probably my last post on this thread... You don't want to listen to advice, and I can promise you that this won't get into XBMC unless you develop it yourself....


Dude,
If you read ALL my post, you would see that I am NOT arguing for the validity of my setup. I specifically alluded to my prime motivation being BRAGGING RIGHTS.

See my actual question: "...My question is: Are there any plans for XBMC to support distributed, cloud storage mechanism that do not rely on the traditional File(s)/Directory(ies) mechanism of FTP, Samba, and the rest?..."

At best, all I am saying that I have not experienced any adverse effect...it works for now until I can find a better solution; which I thought I found with an Object Store; and as you can see from my actual question, I was ONLY asking about a mechanism to access and manipulate the object store from XBMC; which onto itself is a tacit admission of the sustainability (or sheer stupidity) of my original setup.

Seriously, one would have to be crazy to implement this in a real world setup with real data!!!

My entire argument has been that new and exciting possibilities would open for XBMC if it supported (natively) an Object Store Mechanism.
And I would ask you, with you fine IT pedigree, what would be the advantage of XBMC having native support to an Object Store the same way it does with Samba...?
Not to worry; I am not offended, this is a discussion after all :-)
Reply
#51
(2014-01-19, 10:19)jacintech.fire Wrote: @DrDaxxy you need to import the media into XBMC one way or another (files, folders, images, nfo files). When dealing with thousands upon thousands of titles dealing with individuals files is a pain. You can surely see that. Case in point, managing the media metadata (library) with a MYSQL database vs local files...

If your house is brand new why didn't you have them drop in cat5/6 Smile That said, check your phone lines. Most new houses are wired with cat5/6 for the phone lines now. They can easily be converted to network jacks if that is the case. It also isn't impossible to run wires without creating damage, you need a really long drill bit, patience, and pictures of your house prior to dry wall certainly does not hurt.. I just ran wires from my basement, through first floor and second floor to attic and back down to behind a TV. The only hole in the wall was the one I cut behind the tv to put a box there for the rg6 and cat6.

I assume as well based on your details so far that you also have no redundancy? Have you considered the possibility of running in some sort of raid setup? You certainly do not need to break it into 1TB chunks. I'd consider these systems good candidates for ZFS. Maybe on nas4free.

btw, you clearly have too much spare money.
Reply
#52
(2014-01-19, 18:53)juggie Wrote:
(2014-01-19, 10:19)jacintech.fire Wrote: @DrDaxxy you need to import the media into XBMC one way or another (files, folders, images, nfo files). When dealing with thousands upon thousands of titles dealing with individuals files is a pain. You can surely see that. Case in point, managing the media metadata (library) with a MYSQL database vs local files...

If your house is brand new why didn't you have them drop in cat5/6 Smile That said, check your phone lines. Most new houses are wired with cat5/6 for the phone lines now. They can easily be converted to network jacks if that is the case. It also isn't impossible to run wires without creating damage, you need a really long drill bit, patience, and pictures of your house prior to dry wall certainly does not hurt.. I just ran wires from my basement, through first floor and second floor to attic and back down to behind a TV. The only hole in the wall was the one I cut behind the tv to put a box there for the rg6 and cat6.

I assume as well based on your details so far that you also have no redundancy? Have you considered the possibility of running in some sort of raid setup? You certainly do not need to break it into 1TB chunks. I'd consider these systems good candidates for ZFS. Maybe on nas4free.

btw, you clearly have too much spare money.

New house...but already built at the time of purchase...

In the case of rain, refundancy will not help (unless backup are off site...)

The Powerline adapter works fine (24Mbps versus 63-70Mbps on Gigabit Ethernet), so I saw no need to think about it further...
Reply
#53
@edrikk,
"...My two cents, and probably my last post on this thread... You don't want to listen to advice, and I can promise you that this won't get into XBMC unless you develop it yourself..."

I would be willing to bet my ridiculously large collection of BluRay and DVDs that native support for an Object Store model will be supported by XBMC. It is only a matter of time. Either that, or some other project will. The advantages are just too obvious to ignore...
Reply
#54
(2014-01-19, 18:57)jacintech.fire Wrote:
(2014-01-19, 18:53)juggie Wrote:
(2014-01-19, 10:19)jacintech.fire Wrote: @DrDaxxy you need to import the media into XBMC one way or another (files, folders, images, nfo files). When dealing with thousands upon thousands of titles dealing with individuals files is a pain. You can surely see that. Case in point, managing the media metadata (library) with a MYSQL database vs local files...

If your house is brand new why didn't you have them drop in cat5/6 Smile That said, check your phone lines. Most new houses are wired with cat5/6 for the phone lines now. They can easily be converted to network jacks if that is the case. It also isn't impossible to run wires without creating damage, you need a really long drill bit, patience, and pictures of your house prior to dry wall certainly does not hurt.. I just ran wires from my basement, through first floor and second floor to attic and back down to behind a TV. The only hole in the wall was the one I cut behind the tv to put a box there for the rg6 and cat6.

I assume as well based on your details so far that you also have no redundancy? Have you considered the possibility of running in some sort of raid setup? You certainly do not need to break it into 1TB chunks. I'd consider these systems good candidates for ZFS. Maybe on nas4free.

btw, you clearly have too much spare money.

New house...but already built at the time of purchase...

In the case of rain, refundancy will not help (unless backup are off site...)

The Powerline adapter works fine (24Mbps versus 63-70Mbps on Gigabit Ethernet), so I saw no need to think about it further...

This must of been typed on a phone with auto correct Smile

do check your phone lines none the less, you may be pleasantly surprised.

Raid would help if a drive failed etc, and ensure you don't have to re-acquire all your media from which ever source that is. Given you have so many drives, the risk of losing a drive is high. So to me that would be essential.
In terms of clustering you could use something like unionfs to blend all your file systems together however you want and then share that one file system via NFS/Samba. However I am not sure it would of been tested to this scale with this many partitions.

Redhat GFS may also be something worth looking into but I am not sure if you would find many people here with expirience at this scale. You can try here: http://hardforum.com/forumdisplay.php?f=29 this audience may be able to better assist.
Reply
#55
@juggie
I don't know what else to say; as all the response I have gotten seem to zero-in the fact that I have 8x 16x 4TB drives spinning in my basement without RAID support. I think I already established the prime motivation for building that rack was BRAGGING RIGHTS and nothing more (well, actually it was the best solution until I could decide on a permanent approach

My first and only question was (and still is): "...Are there any plans for XBMC to support distributed, cloud storage mechanism that do not rely on the traditional File(s)/Directory(ies) mechanism of FTP, Samba, and the rest?..."

REMEMBER: I can built and configure the Object Store myself (most of them ALREADY have built-in redundancy, duplication..and pixie dust). There are plenty open source projects to choose from. I am ONLY interested in XBMC supporting direct (Native) access and manipulation to ANY object store model. THAT IS ALL.

AGAIN: "..mechanism that do not rely on the traditional File(s)/Directory(ies) mechanism of FTP, Samba, and the rest?..."
Any mechanism that will have to be either exported via Samba/NFS or use a mapping (object ---> file) mechanism to interact with XBMC is not what I am looking for. I am looking for something that no longer relies on traditional files and directories...
Reply
#56
You've already had a response from a team member. What more do you want?
Reply
#57
You aren't reading people's responses.

People have already given you options, and on TOP of that have said your setup is not good.
You keep saying that it's for bragging rights and an experiment. Well. Great. Yes it "runs"... Good luck...


Anyways, this thread is getting to the pointing where locking may be considered the right thing... It's just going in circles. I think your question has been answered a couple of times. Maybe not to your liking, but it has... Actually not "it", but rather "they" have been answered...
Reply
#58
(2014-01-19, 19:33)prawnee Wrote: You've already had a response from a team member. What more do you want?
Yes. That was rather clear and final.
But all other responses since have been about what a bad idea is to have 8x 16x 4TB drives spinning without RAID. What I wanted to do was move the conversation to what the advantages would be of XMBC doing just that (supporting an Object Store model, the same way as it natively supports Samba). That is all...
With ALL due respect :-)
Reply
#59
Currently the best thing you could do is NFS/Samba/CIFS/Upnp on your distributed cluster which XBMC can then access.
Reply
#60
@juggie,
You are right of course. It could be argued that "Currently" that is the best approach (which holds true regardless of the underlying storage solution). But can you imagine what else can be accomplished if we are not longer limited by NFS/Samba/CIFS/Upnp?

I think (and I don't mean this as an insult to anyone), we have been so caught up on what works today that we may have failed to lift our heads up and look towards the horizon; see emerging technologies and think of ways can use them to move the development of XBMC forward; which is, after all the raison d'être of any open source project.
Reply
  • 1
  • 2
  • 3
  • 4(current)
  • 5
  • 6
  • 22

Logout Mark Read Team Forum Stats Members Help
Native Object-Based Storage Support for XBMC5