[da-anda]

@Tolriq - you have the same authentication issues with a pure MySQL setup, but there is always a way around those issues.

@m.savazzi I'm using ORM, QOM and everything in a php framework (almost) day by day, but I'm not a pro on this. I once contributed to that framework as well and fixed some bugs in their relational mapping, so I know a little about the internals and how this stuff is done, but I'm really no professional on this. I can only tell that implementing such a thing is anything but easy - at least if you want to do it in a clean object oriented way. You'd need a solid framework that's already providing all these things and integrate this in XBMC. But be warned, we already had several internal discussions about making use of some other frameworks for certain things (like dependency injection, ...) and everything has been discarded so far because of some drawbacks with any of them (can't recall the exact reasons or the names of those frameworks)

[Tolriq]

I know that why I talked earlier of the need of a secure way to share / sync sources Since all other solutions will need to have an XBMC serving the file with the already evoked drawbacks.

I was just trying to explain him why UNC is nothing a miracle solution to this problem.

[phate89]

@Tolriq - you have the same authentication issues with a pure MySQL setup, but there is always a way around those issues.

@m.savazzi I'm using ORM, QOM and everything in a php framework (almost) day by day, but I'm not a pro on this. I once contributed to that framework as well and fixed some bugs in their relational mapping, so I know a little about the internals and how this stuff is done, but I'm really no professional on this. I can only tell that implementing such a thing is anything but easy - at least if you want to do it in a clean object oriented way. You'd need a solid framework that's already providing all these things and integrate this in XBMC. But be warned, we already had several internal discussions about making use of some other frameworks for certain things (like dependency injection, ...) and everything has been discarded so far because of some drawbacks with any of them (can't recall the exact reasons or the names of those frameworks)

but does xbmc need a full framework? Can't you only define several actions to the database with rules and the "db addon" handles how to perform every single action? Actually any full framework that translate to something else makes everything slower and it's not a great thing for raspberry pi and low power devices...

[Tolriq]

The problem of this approach is that each and every addon needs to be updated on every database changes.

And if one add support for something, it will be hard to maintain if other do not master it.

This would be worse than current Mysql support I guess.

[phate89]

I don't think it's a great problem. db changes are not so common and usually are small changes. When the devs change the db they bump the api and the mantainer of the mysql db addon (or any addon) have to update it to make it work.
Of course someone have to keep the work alive but it's like any other addon that every new release have to be updated.
I don't think it will be hard to find someone at least for mysql (and sqlite is updated from the team) and most requested dbs.
It's still 100 Times easier than maintaining MySQL in the current state.
Of course you won't have everything ready but i think it's better to have faster access on low power devices...

[da-anda]

I just noticed that db-addons might not work, because the addons system is also relying on the DB. So at least the addon system itself can't be handled via non-core db-addons. Imagine you update XBMC which requires a new DB API and thus a new MySQL addon. But while addon manager is checking for available addon updates, some skin script is quering the DB and trying to fetch recently added movies. Boom, here it goes. And delaying startup until addonmanager has done it's job is also no option, because this can take ages.

Your mentioned "several functions with rules" have to be validated by something, have to allow to query for values of related records (which you need a global mapping configuration for) etc - and then you already have your QOM. So it's probably easier to reuse an existing QOM framework thingy.

[m.savazzi]

Library support way more than smb and UNC or whatever will not handle login and password unless plain text and it's not possible for obvious security reasons ....

Can you explain what else is in the Video and Audio libraries?
What you have in Library?

Furthermore Xbmc is gpl v2 and cannot use gpl v3 library without upgrading to v3 and from what I read this is not desired...

Interesting to know, I was not aware of that.

I'm not an expert of everything. What I know is that what is in is not working... then I suggested to contribute and I'm providing indeas and information as I found them to see if they have already been evaluated and vetted.


Sharing a content library among devices requires, obviously, that every device do have access to the library,
Have you thought that IF a use is SOOOOO smart to be able to connect 2 pc to the same NAS probably he has already overcome the huge hurdle of typing twice username/password?
We are talking of a Content and Library sharing functionality. It's pretty straightforward that every device must be able to reach both the content and the db server, and should do it with the correct credentials.

Why in the world you would want to send username and password between the different PC/Devices?

Even in the headless xbmc implementation the "headless" entity should have the credentials to access content and db so where is the point? worst than that you will have a lot of streaming traffic between the playback xbmc and the headless one.

@da-anda
in my experience if you can use a stable framework there is a huge advantage in terms of compatibility and code maintenance. Is easier for people to jump in, easier to fix and to manage.
This especially for some "standardized" functionality like DB access where having a custom implementation will not generate effective advantage but issues.
Current XBMC situation is an example, would have they used ODBC from day 0 there would have not been any issue around DBm sharing etc...
having a custom DB stack lead to issues, non compatible implementations, issues to change and maintain code, etc...

[da-anda]

I'm not against using a framework for this, I only pointed out that certain frameworks might be denied by our devs because of unwanted dependencies or whatever. So be prepared that devs might say no to suggested frameworks.
But regardless of framework or not, first step IMO would be to analyse current DB usage and come up with a concept of how it could work. Then ask core devs for opinions. Once the concept is settled and approved we can move forward and either look for a suitable framework or write something on our own. Concept should include a concept for DB version migrations and handling of mismatching API versions, taking into account that additional DB addons won't be part of the XBMC distribution and be loaded from a repo.

oh, and consider that every additional processing layer will decrease performance. So while a abstract QOM is nice to have, it's always slower than discrete queries. But the point here is not just how to get things faster, but also how to be able to keep MySQL around or use other DB systems

[Tolriq]

I'm not against using a framework for this, I only pointed out that certain frameworks might be denied by our devs because of unwanted dependencies or whatever. So be prepared that devs might say no to suggested frameworks.
But regardless of framework or not, first step IMO would be to analyse current DB usage and come up with a concept of how it could work. Then ask core devs for opinions. Once the concept is settled and approved we can move forward and either look for a suitable framework or write something on our own. Concept should include a concept for DB version migrations and handling of mismatching API versions, taking into account that additional DB addons won't be part of the XBMC distribution and be loaded from a repo.

oh, and consider that every additional processing layer will decrease performance. So while a abstract QOM is nice to have, it's always slower than discrete queries. But the point here is not just how to get things faster, but also how to be able to keep MySQL around or use other DB systems

IMO First step would be to know the medium target about this central library / media distribution and the final one.
Major question being what is the schema :
- Star schema around a central database / library distribution and cached copies with sync.
- Distributed schema with all hosts having all data / library media.
- One of the 2 previous but with split around database info and libray (meaning for example central database but each host have it's own medias)
- Other schemas

Each of the possible ways will generate further needs and limit / open the possibilities some solutions could be made to avoid the need of Mysql.

Same question with APIs, having possibility to offline download files, then on reconnect update the status would be good but needs advanced timing fields.

To me a distributed schema with multiple library, each library having a master host and an Xbmc client host security connection system is the future.
Add some mdns broadcasting and a guest permission easily added for end user and you have something
At first this would still needs one host per library to be up for streaming, but could easily be extended to push the sources info to clients on a permission basis.

[m.savazzi]

Tolriq
Maybe I've not understood what you proposed as:

To me a distributed schema with multiple library, each library having a master host and an Xbmc client host security connection system is the future.

which kind of connection you have in the house and on your pc?
are you already in 10Gb ?

1 headend have to stream to the playback entity at a bitrate that is the one of the video. For blueray is 20mbps+

for SD satellite tv is around 9mbps
for dvd is 9,8mps
for hd satellite tv is around 12 mpbs

if you connect 2 headends to the "central" you need 40+mbps

I have 4 endpoints in my house and one nas as source... (with truked Gb) and sometime I have clashes just with NFS/smb

No way it would ever work on wifi.

Anyway this sounds like a great starting point, thank you for your advice. Will try to undestand how DB is used/misused and see how it could be evolved, if I have ideas

[Tolriq]

You missed the second part, and all that I said before as if only streaming then the headless need to run on nas.

This thread is a discussion I can't repeat all what I said before each time

Anyway as you know 1Gb is common and is 1000mbps, so you can stream 20 full bluray at the same time without problems if your drives follows.

Of course if you decide your main XBMC is a rpi on wifi and connect 5 1Gb hosts on it, it will be problematic unless sources sharing

[Tolriq]

And just though of one little things that was not discussed at all in this central things, it's profiles

How are profiles handled so watched status are still tied to a profile with a central database ?

[da-anda]

profiles have to be handled somehow™

[m.savazzi]

Tolriq, sorry sometimes i do not understand everything you write, thank you for repetitions; they help me


once the libraries are stable on a shared central db to add the profiles/groups is quite trivial and common practice, we can find good info...

an easy way to do it is to have a "users" table (aka the profile) and then have a binding on the core objects so you can create a view that filters by user id.

filtering the core elements (image, movie, episode, track, picture) will make queries far more efficient

specifically for the viewed flag you can have a very simple table
viewed with 2 columns: asset id, used id
IF a row is present the content has been watched.

simple and super quick

[Tolriq]

Thanks for basic sql :p

But the problem was not SQL but the rest How do you sync profiles between hosts ? Who is responsible for the main data for disabling one account, how does all XBMC known which profile is logged to update correctly everywhere the status ?, how do you handle what library what profile have access too, ....
There's lot's of question to answers like for the target schema.

And adding such new table (that also needs lastplayed date, and all needed for resume points, ...) also needs to add an admin profile with an id so queries are consistent and not complex to gather data in multiple spots.

This needs a complete rewrite of how profiles are handled since right now they all have specific sources and databases, and this needs a major update of the current XBMC database schema.

As I said and others too, things have to be done in the correct order, and order is know what the target is, define the target needs, choose if there's a easy path from now to target then implement.

Implement needing to handle things in order too depending on target, but if multiple database support, then define QOM, then implement

So no nothing is simple and quick, real life is not like dreams

You really need to think globally and take the necessary stand back from the details