Kodi Community Forum
Native Object-Based Storage Support for XBMC - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Discussions (https://forum.kodi.tv/forumdisplay.php?fid=222)
+--- Forum: Kodi related discussions (https://forum.kodi.tv/forumdisplay.php?fid=6)
+--- Thread: Native Object-Based Storage Support for XBMC (/showthread.php?tid=183747)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22


RE: URGENT: Object Store 512TB Array - jacintech.fire - 2014-01-22

(2014-01-22, 17:08)Ned Scott Wrote: I don't think you're listening. It's not the idea that I have an issue with. It's your continued lack of credibility in the face of you telling me that I'm the one who's being narrow minded. The idea itself is neat, but you are your own worst enemy in this thread.

(2014-01-22, 14:08)jacintech.fire Wrote: @nickr, @ned scott, @universal,
At risk of beating a dead horse, please consider this: should I join all the volumes into two pools (one for movie, one for tv) and export said volumes as movies and tv (remember, I have THOUSANDS of titles) where to browse these volumes I would see a single folder with THOUSANDS upon THOUSANDS of entries. Try finding ANYTHING with THAT mess. If however, I label each partition such as "Movie Server #8 - Feature Films Drive #4"? Now, when I need to find a title, I just look up the Movie Information from XBMC (or c22 from movie in MYSQL), look at the PATH variable and viola, I know exactly where it is.

This is a perfect example. One, there are several reasons why this really shouldn't matter. Two, if it really matters that much to you, there's still half a dozen ways to handle tracking like this but still only show a limited number (even just two, if you wanted) of sources in XBMC.

There are perfectly good arguments for this object store method you want to use. You've even presented many of them yourself. However, you also keep making statements that just kill your argument because they have nothing to do with why objective store might be better than the alternatives. It makes us think you really don't know what you're talking about.

EDIT: holy crap, is there really 6 pages of posts between this post and the one I'm replying to?
You are looking too close. Please take a step back and consider this: at a certain scale (i.e tens of thousands of media titles) this current mechanism of files and folders becomes a nightmare (RAID or JBOD). An object store is infinitely scalable has built in redundancy, is distributed and the metadata mechanism unifies with content.
Othern than having that, my current setup gives me more than it takes...but like a RAID setup IT DOES NOT SCALE WELL...


RE: URGENT: Object Store 512TB Array - edrikk - 2014-01-22

LOL @ Ned Scott... hahahahaha!


Simple solution to your "I label each partition such as "Movie Server #8 - Feature Films Drive #4"... How about just creating a DIRECTORY with that name? In Windows 8, the search is actually quite good... Press the Windows Key, and type a part of the file name..... Within XBMC, use Global Search, or any of the remote control Apps.

Your "finding things" example is flawed, and actually involves more time wasted (waiting for all the disks to spin-up / show-up in your Explorer, etc etc.)


RE: URGENT: Object Store 512TB Array - jacintech.fire - 2014-01-22

(2014-01-22, 17:17)edrikk Wrote: LOL @ Ned Scott... hahahahaha!


Simple solution to your "I label each partition such as "Movie Server #8 - Feature Films Drive #4"... How about just creating a DIRECTORY with that name? In Windows 8, the search is actually quite good... Press the Windows Key, and type a part of the file name..... Within XBMC, use Global Search, or any of the remote control Apps.

Your "finding things" example is flawed, and actually involves more time wasted (waiting for all the disks to spin-up / show-up in your Explorer, etc etc.)

Exactly. As far as windows go, its a folder ANYWAY. So having 1TB partition, or a folder makes NO DIFFERENCE...


RE: URGENT: Object Store 512TB Array - edrikk - 2014-01-22

That's not what I said... Read it again.
Anyways, this isn't going anywhere. I'm outta this.


RE: URGENT: Object Store 512TB Array - Milhouse - 2014-01-22

(2014-01-22, 17:04)jacintech.fire Wrote: If you look at my post, I have already addressed ZFS as a possible candidate...
...how would that data loss express itself: Across an entire drive? Or most likely across a partition..? Bit rot...or even bit "flipping" occur across small areas,and if those areas are within a single partition, so is the extend of the data loss...

What if the bit rot is in your partition table? You potentially lose the entire disk (all 4 partitions). Your setup has no defence against bit-rot and other data errors. You will have data integrity issues already, but you won't even know it because you have no way to identify such errors (let alone correct them).

(2014-01-22, 17:04)jacintech.fire Wrote: "...When your file system is good enough, the probability of an error occurring without being detected becomes so low that you don't have to care about that any longer and you might decide that having checksums built into the data storage format you're using is unnecessary..."

And what filesystem are you using? NTFS, EXT4, which one automatically detects and corrects bit errors?


RE: URGENT: Object Store 512TB Array - un1versal - 2014-01-22

FAT32


RE: URGENT: Object Store 512TB Array - jacintech.fire - 2014-01-22

(2014-01-22, 17:22)MilhouseVH Wrote:
(2014-01-22, 17:04)jacintech.fire Wrote: If you look at my post, I have already addressed ZFS as a possible candidate...
...how would that data loss express itself: Across an entire drive? Or most likely across a partition..? Bit rot...or even bit "flipping" occur across small areas,and if those areas are within a single partition, so is the extend of the data loss...

What if the bit rot is in your partition table? You potentially lose the entire disk (all 4 partitions). Your setup has no defence against bit-rot and other data errors. You will have data integrity issues already, but you won't even know it because you have no way to identify such errors (let alone correct them).

(2014-01-22, 17:04)jacintech.fire Wrote: "...When your file system is good enough, the probability of an error occurring without being detected becomes so low that you don't have to care about that any longer and you might decide that having checksums built into the data storage format you're using is unnecessary..."

And what filesystem are you using? NTFS, EXT4, which one automatically detects and corrects bit errors?
...but then yoy get into the statistical probabilities of such an error, versus potential for data loss from RAID itself...


RE: URGENT: Object Store 512TB Array - un1versal - 2014-01-22




RE: URGENT: Object Store 512TB Array - Ned Scott - 2014-01-22

I'm out. He's not even responding to what I said.

It's hard, because what he says is just so wrong that it feels like a black hole sucking you in, pulling you into the debate/argument. It won't change anything. I'm tapping out.


RE: URGENT: Object Store 512TB Array - Milhouse - 2014-01-22

(2014-01-22, 17:25)jacintech.fire Wrote: ...but then yoy get into the statistical probabilities of such an error, versus potential for data loss from RAID itself...

At least if you're going to play Russian Roulette with bit rot, can you tell me if you've calculated the probability of bit error across your entire array? All 128 4TB disks.

And please stop suggesting that RAID is likely to increase the likelihood of data loss, do you really think you'd run RAID6 on 128 disks? There's more to RAID than levels 5 and 6, so suggesting it increases your risk only highlights your total lack of knowledge in this area.


RE: URGENT: Object Store 512TB Array - jacintech.fire - 2014-01-22

(2014-01-22, 17:31)MilhouseVH Wrote:
(2014-01-22, 17:25)jacintech.fire Wrote: ...but then yoy get into the statistical probabilities of such an error, versus potential for data loss from RAID itself...

At least if you're going to play Russian Roulette with bit rot, can you tell me if you've calculated the probability of bit error across your entire array? All 128 4TB disks.

And please stop suggesting that RAID is likely to increase the likelihood of data loss, do you really think you'd run RAID6 on 128 disks? There's more to RAID than levels 5 and 6, so suggesting it increases your risk only highlights your total lack of knowledge in this area.

As a matter of fact I did...but there is more to it than simply statistical analysis...

There is a reason RAID is no longer used by the largest data storage providers: it is an obsolete technology. Even more so for large scale.

Other than the distant possibility of bit rot/bit flipping you have not given me ONE scenario where I expetience a nightmare...GIVE ME ONE EXAMPLE (as it applies toXBMC) that trumps 3.5 years to ZERO HDD Loss...
You keep point at some distant future...heck, in the distant future the universe will reach maximum entrophy and all particle interaction will cease...so what?


RE: URGENT: Object Store 512TB Array - un1versal - 2014-01-22




RE: URGENT: Object Store 512TB Array - jacintech.fire - 2014-01-22

(2014-01-22, 17:48)uNiversal Wrote:



http://www.jacintech.com.br/

What is your point...


RE: URGENT: Object Store 512TB Array - Milhouse - 2014-01-22

(2014-01-22, 17:44)jacintech.fire Wrote: There is a reason RAID is no longer used by the largest data storage providers: it is an obsolete technology. Even more so for large scale.

Utter, UTTER codswallop!

I refer you back to my link to the recently installed 55 PetaByte system from IBM (who know a thing about large scale systems) where RAID and other data integrity measures were deemed "essential" (hence the deployment of ZFS).

(2014-01-22, 17:44)jacintech.fire Wrote: that trumps 3.5 years to ZERO HDD Loss...

You've been lucky, it's highly unlikely your luck will continue.


RE: URGENT: Object Store 512TB Array - jacintech.fire - 2014-01-22

(2014-01-22, 17:55)MilhouseVH Wrote:
(2014-01-22, 17:44)jacintech.fire Wrote: There is a reason RAID is no longer used by the largest data storage providers: it is an obsolete technology. Even more so for large scale.

Utter, UTTER codswallop!

I refer you back to my link to the recently installed 55 PetaByte system from IBM (who know a thing about large scale systems) where RAID and other data integrity measures were deemed "essential" (hence the deployment of ZFS).

(2014-01-22, 17:44)jacintech.fire Wrote: that trumps 3.5 years to ZERO HDD Loss...

You've been lucky, it's highly unlikely your luck will continue.

At some point, luck becomes hard data...
"...During a RAID rebuild, data becomes unprotected (RAID5) or reduced in protection (RAID6). Statistically, this increases the chance that another one (or more) of the remaining disks will fail as well due to the increased activity. Studies have also shown that RAID sets show increasing probability of a subsequent disk failure after the first failure. This is due to several factors including the fact that these disks are usually of the same age, from the same manufacturing lot, and have been subject to similar read/write patterns as the other disks in the RAID group..."