Source 'abstraction layer'
#16
It can only be done by looking at the content of the file and yes CRC32 or similar of whole file would be very heavy. Some algorithm much faster but still unique should be found.
If this is to work with anything, then random access cannot be taken for granted (for example not all ftp servers support resume).
Question is how unique are media files at the beginning - how many bytes should be used for hash.
Reply
#17
jmarshall Wrote:I think hashing a portion of the file data (plus use file size) is probably the most reliable method - that way you pick up renames.

Why use the file data itself if you are collecting other meta anyway? You could just hash some other unique data with filesize like media length, resolution, aspect ratio, etc. Hashing a lot of file data just doesn't sound very fast.
Reply
#18
I never said hashing the whole file - that would indeed be silly. Just a tiny bit would suffice. For instance, the hash used by opensubtitles I believe uses just the first 64k of the file and combines this with the size of the file in bytes. That is plenty unique enough, and is much faster than reading metadata information anyway.

@technojunkie - yes, that has been considered. The actual metadata side of the library is planned to basically be a service that can be tapped into by the VFS or plugins. As long as there is a unique file identifier available (URI is the obvious one) we can pair up metadata. This is quite separate to hashing ofcourse.

Cheers,
Jonathan
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


Image
Reply
#19
Keeping in mind that the metadata in the files themselves should probably not be part of this hash, to allow for re-tagging without breaking the hash.

TheUni
Reply
#20
Won't re-tagging void using file size as part of hash like opensubtitles? Or is space allocated for the tags always?
Reply
#21
The hash technique isn't really important, so long as it's generally understood that XBMC has a reliable way to identify the file (although hashes are not technically a guarantee)

I'm not diminishing the hash, please don't think that, but it's too easy to get bogged down in how the hash is created rather than working on the problem domain...shifting media around.

The 'fix' I demonstrated in my batch file really was a nasty horrible workaround that works for me personally in the interim, and was just to help another user get by in the mean time... it's a very simplistic abstraction that would probably improve the lives of 80% of users, and who tend to keep their media pretty organised, but keep moving to new servers, or increased capacity disks. It still works with the same media type split across multiple sources, but not when you start moving media 'across' sources.

Now, if we look at Mr Marshall's suggestion, which is another a higher level of abstraction still...

You could distribute your media wherever you like, providing you can tell XBMC the content types of each 'source', then XBMC need only only perform an automated scan at boot (to find all moved media, or it's a manual process, since the need to rescan is relatively infrequent)....

Why not abandon XBMC having to remember the 'sources' AND their content type and preferred scraper etc and utilise a 'contenttype.nfo' type arrangement.
This way... the user simply tells XBMC which paths to scan for media. No more.

The contenttype.nfo would describe the media to XBMC (TV, Music, Movie etc).
It can work on a 'per folder' basis and/or simply a root folder basis.
The advantage of this, is that media can be stored along with its content type definition, so any new XBMC installation doesn't have to be told the content type for every source. Just give it the paths to search, and voila...it knows what's music and what's movie etc...just an idea.
I expect some people will object to this.... BUT, I put this to you:
XBMC has clearly embarked on a strategy of utilising NFO's if they exist, and the NFO's either help, or literally dictate how XBMC should interpret the media. Why shouldn't this philosophy be extended to help XBMC understand the source content types?
Reply
#22
What do you gain? You still need to define the paths anyway, and defining the content types only adds a few seconds. Also it's only a "one time" operation per source and doen't generally change for the lifetime of the XBMC installation (or very infrequently anyway.)
Reply
#23
AaronD Wrote:What do you gain? You still need to define the paths anyway, and defining the content types only adds a few seconds. Also it's only a "one time" operation per source and doen't generally change for the lifetime of the XBMC installation (or very infrequently anyway.)

You gain the following:

1) No longer necessary to dedicate a folder to Movies, a folder to TV a folder to Music... you can have Mixed Folders, and only need to define ONE root folder/server for all your media.

2) If you deploy multiple XBMC installations, as I do.... 8 of them in my instance, I only need to do this once. So it's scalable

3) If you don't HAVE to reconfigure it multiple times, they why should you? if there's a way to avoid that, I'm all for it (of course it's a cost to develop/benefit judgment)

4) In my view it's a natural continuation of the logic that was used to NFO files for media.

5) It comes into it's own much more once the concept of being able to move your media around is fully supported... rearrange the folders as you please and the NFO will stay with it.

6) There's still a LOT of newbies who forget to set the content type of folders. It's true that they could forget to create and NFO too, but at least 3rd party tools can autocreate it.

You're right about in a single instance... it's not going to offer very much... since it does only take a few seconds to choose a content type for a source, but if you have a lot of sources, a fair number of installations and you move your data around a lot, there's a benefit to be had.


I suppose it's just a matter of opinion as to how much it's worth to any given individual... maybe I'm in the weird minority ;-)
Reply
#24
Creating a text file on the server to hold content information seems like overkill to me, and is certainly nothing "joe user" would ever conceive of doing.

We want to simplify the whole process here. The goal is that the user should simply say "This is where (some of) my media is", and (optionally) "Oh, that dir is (mostly) movies" or (even more optionally) "I'd prefer you used _foo_ to lookup the information" or "oh, these hints may help you to identify things".

Anything more than that is too complicated for the average joe as seen here on the forums time and time again (and those average joes have the stamina to come to the forums and ask - many would just give up).

Most users don't move things around onto new machines regularly, and if they do then really they wouldn't have much trouble in pointing XBMC at the new path(s). If XBMC would then detect that actually all this media is already scanned previously, it's just another copy (or a moved copy) then the user is happy.

Cheers,
Jonathan
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


Image
Reply
#25
jmarshall Wrote:Creating a text file on the server to hold content information seems like overkill to me, and is certainly nothing "joe user" would ever conceive of doing.

We want to simplify the whole process here. The goal is that the user should simply say "This is where (some of) my media is", and (optionally) "Oh, that dir is (mostly) movies" or (even more optionally) "I'd prefer you used _foo_ to lookup the information" or "oh, these hints may help you to identify things".

Anything more than that is too complicated for the average joe as seen here on the forums time and time again (and those average joes have the stamina to come to the forums and ask - many would just give up).

Most users don't move things around onto new machines regularly, and if they do then really they wouldn't have much trouble in pointing XBMC at the new path(s). If XBMC would then detect that actually all this media is already scanned previously, it's just another copy (or a moved copy) then the user is happy.

Cheers,
Jonathan

Well, I can see the logic in that thinking too...
But (always but ;-))

By that reckoning, why bother with 'source' abstraction at all, since people aren't moving media around that much, and if they've exported to multiple NFO files anyway, a rescan should be no trouble.
It seems the primary motivation for source abstraction is to allow media to be moved about and (in theory) not have to rebuild the library or start a new build of the image cache. If people aren't relocating their content much, then is there a point? (I personally think there is, but others might disagree).

It's just my opinion but I think a lot of users are 'kinda' moving their data around... using external USB drives and discovering that (windows in particular) can mount the USB drive under a different drive letter sometimes and it's screwing up XBMC. So they work around it with drive mapping/subst (common) or symlinks (rare).
Could be just my perception though, since I'm sensitive to it I notice others who are too... could still be a minority.

I still believe that a 'Source' should be a logical path/device and XBMC only works with 'Sources' and never a physical path. Which was the initial premise of the thread (the contenttype xml was a mere distraction!)
Reply
#26
AnalogKid Wrote:Why not abandon XBMC having to remember the 'sources' AND their content type and preferred scraper etc and utilise a 'contenttype.nfo' type arrangement.

:-)) I thought of the same thing yesterday. I just wanted to wait until final release as I supposed devs would be busy (more than usual) ATM.

Side effect could be library sharing in for configurations with several PCs with NAS, as they would all use same nfos, stored on NAS.
My skins:

Amber
Quartz

Reply
#27
jmarshall Wrote:Most users don't move things around onto new machines regularly, and if they do then really they wouldn't have much trouble in pointing XBMC at the new path(s). If XBMC would then detect that actually all this media is already scanned previously, it's just another copy (or a moved copy) then the user is happy.

Cheers,
Jonathan

I agree, but please consider adding it through advanced settings as it can make things easier. In configurations with all media on shared NAS, which I suppose is most used scenario, we could have a simplified library sharing through this. We would need:

- automatically generated and refreshed nfos (library changes reflected)
- manually or automatically refreshed library on startup
- xml line storing timestamp for movies you didn't finished watching

There is a ongoing discussion when user asks for sharing NAS content between 2 PC's. Others are explaining how to setup mysql. This is not a solution for John Average. EDIT: http://forum.xbmc.org/showthread.php?tid=70603

Makes sense?
My skins:

Amber
Quartz

Reply

Logout Mark Read Team Forum Stats Members Help
Source 'abstraction layer'0