[m.savazzi]

Source is 99% another kodi device that share and automatically adds and keeps updated (if the user wants) its media and its metadata. Upnp is just the protocol for the exchange but you need to know what media comes from what kodi device and know if the user wants its files directly in the database or not. Read the example in my last post

I understood that... can you maka an example of the media path? I've not found the example you refer to.

If the metadata IS in the DB, the DB is shared among ALL kodi instances so there is no need to "share"
If the metadata IS in a local DB then the metadata should be provided to the other Kodi device.


Let's say there is Kodi1 on a PC and Kodi2 on an HTPC.
There is a NAS Server with a /Video share and the DB is hosted on the NAS.

So Kodi1 and Kodi2 will access the same share and will have the same DB on the NAS.

So why they have to talk in UPnP?

[phate89]

Wrong approach with MySQL in mind. 2 kodi pcs. The first have 5 movies, the 2nd 5 tv shows in their local databases each with local files. you have them both in the same network. Kodi1 detects that there is kodi2 in the network so ask to the user:
"You want to add kodi2 media to the local database and keep them updated?". If the user says yes kodi1 get all the metadata from kodi2 and it adds them to the local database so now it has 5 tv shows and 5 movies in the local database. Of course kodi1 will show the tv shows only if kodi2 is online and when someone change kodi2 metadata (for example watched status) kodi2 will Push the changes to kodi1 so kodi1 will be always up to date. Of course the user can say no so kodi needs to know what source (what kodi instance) the user wants to keep updated and what tv show or movie to hide if kodi2 is offline

[m.savazzi]

Wrong approach with MySQL in mind. 2 kodi pcs. The first have 5 movies, the 2nd 5 tv shows in their local databases each with local files. you have them both in the same network. Kodi1 detects that there is kodi2 in the network so ask to the user:
"You want to add kodi2 media to the local database and keep them updated?". If the user says yes kodi1 get all the metadata from kodi2 and it adds them to the local database so now it has 5 tv shows and 5 movies in the local database. Of course kodi1 will show the tv shows only if kodi2 is online and when someone change kodi2 metadata (for example watched status) kodi2 will Push the changes to kodi1 so kodi1 will be always up to date. Of course the user can say no so kodi needs to know what source (what kodi instance) the user wants to keep updated and what tv show or movie to hide if kodi2 is offline

I do not have MySQL in mind, I have in mind a shared DB scenario
99,9999% of the users has a library on a nas and 99% of the nas natively supports DB or can host SQLite db files (thus being a shared DB).
The UPnP scenario is a niche scenario where content is on 2 different PC and, for very weird reasons I cannot get, instead of doing a file share they want to use UPnP.
Even in your case you need to have Kodi1 and Kodi2 on to watch the content. Why you should use UPnP when Kodi2 can simply access the share of Kodi1?

Different topic is: how to manage offline assets? I have to add that to the DB. Offline assets can be DVD, Blueray, External USB Storage, Network storage, another Kodi PC, etc...

Question:
On Kopdi 2 which is the path of the kodi1 movie 1?

In your scenario the DB I proposed works perfectly even if is NOT efficient as a lot of assets will be duplicated (i.e. poster, fanart, etc...), UPnP is not done to send those files across the network so when you add on Kodi1 an asset that is on Kodi2 on Kodi1 you will need to re-scrape the information from internet and this is not efficient. It will lead to having the same movie on Kodi1 and Kodi2 and have 2 different poster or fanart on Kodi1 and Kodi2... is a very bad user experience.

So in the essence you will have Kodi 1 with DB 1 and Kodi 2 with DB 2 when the Kodi2 is detected Kodi 1 will simply scan (via UPnP) Kodi 2 assets and add them to the library.

Note that there is a high probability that when Kodi 1 detects Kodi 2 the reverse happens so Kodi 2 will detect Kodi 1 library... this because both PC have UPnP active.
Now how you avoid circular references? Kodi 1 detects a movie on Kodi 2 and adds it to the library. Kodi 2 detects Kodi 1 asks for the library and gets even the reference to it's own library via UPnP... this becomes an infinite loop


The second part of your example is more complex, the push of watched status etc... as today Kodi does not keep track of all changes so you'll need to create some sort of "log" of changes.

What happens if on Kodi2 I delete the show and on Kodi1 I update the fanart? who will win?
What if on Kodi1 I delete a show that is on Kodi 2 and on Kodi2 I watch it? what will happen?

On a scenario like the one you described there are a lot of sync issues.

Anyway here is not to discuss if the UPnP scenario makes sense or not... is the new DB structure. The structure I posted fully supports your scenario.

[phate89]

The path is always upnp and it's a mistake to think you need a rescrape. Kodi instances will communicate metadata changes keeping themselves updated. So no rescraping problems, updated info etc also duplicated poster, cover etc etc is a false problem because kodi always cache them. Actually with upnp you can share the changed cover even if it's a local file, impossible to do with a shared library.
Of course there are sync problems but not impossible to solve. I prefer a shared db too (it's what I use) but this is still a very advanced and impressive concept, Way more advanced and dynamic and if they both live it's going to b great but that's not the point. How can you support this scenario without even a "sources" table? How can kodi knows if it has to get the updates from a specific instance and what media to hide without a source table connected to the media?

[Tolriq]

Keep in mind also that the way to access media can be dual as a file might be accessible by one instance and not another one and then should access the media from the first instance (Either automatic fallback or configurable per source).

Or include a way to sync sources with password but with tons of security problems.

About scraping you also need to keep in mind some actual limitation / configuration of XBMC. Images are cached at some defined sizes depending on lots of things.

And since each client could need different size in it's cache the main source either needs to store the best quality one and the local cache for it's need. As sending a low resolution image to be upscaled on the client would just be awful...

There's really a lot of things to think and as Montellese said it needs to be possible to do it step by step. (This image quality problem is already present in current XBMC UPnP sharing from an rpi for example, don't know if something is planned about it).

[m.savazzi]

The path is always upnp and it's a mistake to think you need a rescrape. Kodi instances will communicate metadata changes keeping themselves updated.

This is good, once you push metadata it can be inserted in the DB

So no rescraping problems, updated info etc also duplicated poster, cover etc etc is a false problem because kodi always cache them. Actually with upnp you can share the changed cover even if it's a local file, impossible to do with a shared library.

Cache is local in Kodi so you will have multiple copies of the poster, etc...
Is not correct is impossible to do it with a shared library, you just have to save it with the asset itself

Of course there are sync problems but not impossible to solve. I prefer a shared db too (it's what I use) but this is still a very advanced and impressive concept, Way more advanced and dynamic and if they both live it's going to b great but that's not the point.

I agree with that, only I do not think a lot of people will use it. Having to have multiple PC on to share libraries is a nuisance... everyone I know does have a NAS.

How can you support this scenario without even a "sources" table? How can kodi knows if it has to get the updates from a specific instance and what media to hide without a source table connected to the media?

It's very simple:
in Path table you put the Source
in filename you put the filename.

In the end it's exactly what it it the "path" to the asset is your UPnP Source

The model it's completely abstracted from physical file or location. Obviously at code level you need to be able to act accordingly to a physical file, to a UPnP path, to an HTTP url, etc...

Consider it from this perspective: In Kodi an asset is an asset, independently from where it is physically stored.

---

Keep in mind also that the way to access media can be dual as a file might be accessible by one instance and not another one and then should access the media from the first instance (Either automatic fallback or configurable per source).

Or include a way to sync sources with password but with tons of security problems.

About scraping you also need to keep in mind some actual limitation / configuration of XBMC. Images are cached at some defined sizes depending on lots of things.

And since each client could need different size in it's cache the main source either needs to store the best quality one and the local cache for it's need. As sending a low resolution image to be upscaled on the client would just be awful...

There's really a lot of things to think and as Montellese said it needs to be possible to do it step by step. (This image quality problem is already present in current XBMC UPnP sharing from an rpi for example, don't know if something is planned about it).

To have UPnP work well is a HUGE amount of work and troubles, security, circular references (direct and multiple ways, consider you have 3 PC with Kodi each one will have K1, K2, K3 library. Now K1 will import K2, K2 import K3, K3 imports K1.... BUM!)

Not saying it should not be done but do consider not even Microsoft, Samsung, LG, Google, Netflix got it working right... UPnP has been there for years and it's usage is very limited...
Maybe is better to focus efforts to get DB and Library management working very well as it will impact all users... not just a few.

Sharing the folder and a shared DB achieves the same functionality and is far easier.

IF thumbs are small enough they can be saved in the DB as Objects... thus no duplicates at all!

Anyway as future reference note that once the new DB is in place a Kodi istance can connect to multiple DBs on multiple NAS/PC if you want.
It's just a code problem... not anymore a data issue

[Martijn]

FYI. I don't have a NAS and will never have one.
And I want a shared DB

[m.savazzi]

FYI. I don't have a NAS and will never have one.
And I want a shared DB

Here to serve you ... shared DB is cooking
Question: where you save the content?

---

Question: why don't we save the thumbs in the DB as BLOBs?
It should be MUCH more efficient than the OS especially on low power devices or low speed disks, the speed is the one of the lan for a remote DBs.
For local SQLite there should not be any relevant difference.

I've checked SQLite documentation and BLOB size is huge
(here: http://sqlite.org/limits.html)
The maximum number of bytes in a string or BLOB in SQLite is defined by the preprocessor macro SQLITE_MAX_LENGTH. The default value of this macro is 1 billion (1 thousand million or 1,000,000,000).

[phate89]

It's very simple:
in Path table you put the Source
in filename you put the filename.

In the end it's exactly what it it the "path" to the asset is your UPnP Source

It's not so simple. Kodi needs to know if it has to update a source so you can't just say a path is a source. You need to distinguish them from regular paths and some option to store if the user wants it in the db or not. Also kodi can't every time dig into all the paths only to estract the upnp already in place. And if the user adds as a source a upnp server and scrape the data? How did you distinguish kodi synced libraries from locally scraped upnp servers?
Sources needs to be separated from actual content because they're not the same thing and behave very differently..

[niih]

Thanks, will do!

[quote='m.savazzi' pid='1761748' dateline='1406664729']Added them in latest version. I use inheritance here too.
There is a table Person then there are the Childs that are Actor, Singer... I do not know if we need to add Director or Staff kind of table

Note that I added a Table MusicGroup to group People

Bikeshedding this now but Director for movies and Composer for (classical) music should be on the same level as Actors, Singers, and Music Groups IMHO. Many movie entusiasts also like to follow Writters and Sceenplay writters. Though bike shed color should not really be that important at this early phase or?

I do agree. Director, Composer, Writers, Screenplay are childs of Person and can be bound to any asset.

My question was: do they have any specific field for them?
If they have I will create a separate logic element.
If they have not then we need a "type" field in person that will contain "Actor, Director, Compser, Singer, etc..."

---

I think if you have a "type" field, it should not be in the Person, but should be a part of the relationship metadata. The person per se, does not have a type, it is the relationship which is typed, so a Person is bound to the asset with a relationship of type Actor, Director, Composer, etc

[Fice]

We basically need two things:
a) we have an asset, then we need to be able to get the data of that asset, e.g. to feed it to our players
b) we have an asset, we would like to know where it came from. An addon, a tv backend (e.g. a recorded asset) or a upnp server.
When both are possible (and m.savazzi said it is), we should be fine. I (for my upnp sync stuff) can't think of any other necessary restrictions on the overall DB layout. So, I guess we could move on from 'wheter or not upnp syncing is useful' back to what should the new db layout provide,
and I have one more thing: Have you thought of a 'mobile mode'? Where someone wants to temporarely copy a file to a tablet, watch that file on holiday, and delete the copy once you're back home? During that holiday we would have 2 paths... the one of the original file/source (probably a nas) and the one on the local harddrive. Would it be doable to allow sth. like that in the long run? The goal of the distributed, syncable DB is to make such a mobile mode possible.

[Tolriq]

Keep in mind also that the way to access media can be dual as a file might be accessible by one instance and not another one and then should access the media from the first instance (Either automatic fallback or configurable per source).

Or include a way to sync sources with password but with tons of security problems.

About scraping you also need to keep in mind some actual limitation / configuration of XBMC. Images are cached at some defined sizes depending on lots of things.

And since each client could need different size in it's cache the main source either needs to store the best quality one and the local cache for it's need. As sending a low resolution image to be upscaled on the client would just be awful...

There's really a lot of things to think and as Montellese said it needs to be possible to do it step by step. (This image quality problem is already present in current XBMC UPnP sharing from an rpi for example, don't know if something is planned about it).

To have UPnP work well is a HUGE amount of work and troubles, security, circular references (direct and multiple ways, consider you have 3 PC with Kodi each one will have K1, K2, K3 library. Now K1 will import K2, K2 import K3, K3 imports K1.... BUM!)

Not saying it should not be done but do consider not even Microsoft, Samsung, LG, Google, Netflix got it working right... UPnP has been there for years and it's usage is very limited...
Maybe is better to focus efforts to get DB and Library management working very well as it will impact all users... not just a few.

Sharing the folder and a shared DB achieves the same functionality and is far easier.

IF thumbs are small enough they can be saved in the DB as Objects... thus no duplicates at all!

Anyway as future reference note that once the new DB is in place a Kodi istance can connect to multiple DBs on multiple NAS/PC if you want.
It's just a code problem... not anymore a data issue

You need to read better before answering
I never talk about UPnP in what I said. And I do not talk either about source import (that if done correctly as source with a master, does handle the 3 cases easily ...)

I talk about media access and thumbnail access.

And no storing images in DB for display is not a good idea at all, and you'll always need images at the correct size stored in cache as you'll need local cache as there's no faster way to handle this .... (And sharing the source path does not work for local thumbs near media for local media for example) ....

For storage, if Kodi1 have access to a local drive, Kodi2 need to have access to the media too or source sharing is useless ... So a path like /var/media does not work and you either need upnp / http sharing this is the base of central db.

Same for NAS sources with for example NFS exported only to one host and not all host, or password protected smb shares.

[da-anda]

as for the "source" identification. To be able to indicate that some asset is offline (NAS down, Kodi instance holding the file down), we need to know where it came from, and this must not be some query parsing for a specific flag in a "path" field - so along with the "path" you need some "source" field holding the unique local identifier for the source. And with local I really mean local. From how I understood Montellese, the import sources are speficic to each installation (and probably stored in a XML?) - thus each source will have a instance specific unique ID and this ID should be linked to in a separate field within the files table (along with the path). Having an UNC path in the table and your instance is not able to access it (maybe credentials required) doesn't necessarily mean that the source which could forward it to you isn't able to access the file. So file access checks by just the "path" only won't probably be enough - and at least then you'd need to know from which configured import source this is coming to a) check if that source is even online and b) it can serve that file.

And I don't agree that users MUST be stored in the DB and also don't like the bitfield for user IDs. I'd rather see a permission groups table IN the DB which profiles can be assigned to. So permission check in DB will be done via permission groups and not users/profiles. This also allows profiles to share permissions + simplifies reusing of permission configurations. Also, profiles could have their own DB (completely separate) for which there might be good reasons - so it wouldn't be possible to store them IN the db anyways.

[Tolriq]

Accounts needs to be in database and synced to allow multi instance personal information sync like watched status / resume point per profiles on all devices.
But yes bitfield for permissions is kinda strange and groups are better.

[da-anda]

well, there are two conflicting use cases here - profiles vs users and I think we have to separate them in future. A (system) profile can't be in the DB because profile specific settings can mean that a different DB should be used. So profiles are client specific IMO, but could probably also be synced if we'd use UUIDs as identifiers. Users otoh do not necessarily depend on a specific profile. The same user could have two profiles, or be part of more (private/adult profile, family profile). Maybe we need to differenciate even more (contexts?) and decouple system profiles from user profiles/accounts? Needs some more thinking I guess.