[Tolriq]

For the slow start with rpi and music this is not related to DB at all, but to some bugs and / or no optimization in internals

I have reports of users with 120 000 songs without any problems.

Queue 10 songs it takes 1 minute to start, queue an album of 10 songs (resulting in the exact final playlist) it takes 10 sec.

There was tons of reports, but since no one really fully master the queuing / playlist mechanism in Kodi no one ever looked at it.

[enen92]

Just a question out of curiosity (do not interpret as a demand) but are you planning to add python bindings to your new library system?
Since skinning is pretty easy in python it would be cool to be able to build a local library of other stuff also from python addons. There are a lot of requests now and then for addons that require an independent library (books, sports recordings, etc). This would probably be doable by creating a library (and use the other operations) using those bindings and taking advantage of both skinning and new library system.
I understand you plan to add support for binnary addons just not sure if python will receive any love

[garbear]

Just a question out of curiosity (do not interpret as a demand) but are you planning to add python bindings to your new library system?

The new library system is based on python plugins. A content add-on is basically a python plugin, but the listitems and directory structure is scanned into a database instead of browsed in the gui

The innovation here is with unified content scrapers. Each scraper is a small python rule that describes how to derive metadata. For example, here's a few scrapers:

• If X is a video, and X has a title, and X has a year, then additional movie metadata for X can be found at http://themoviedb.org
  
• If X is a game, and X has a title, and X has a platform, then additional game metadata for X can be found at http://thegamesdb.net
  
• If X is a song, and X has a title, and X has an artist, then additional song metadata for X can be found at http://theaudiodb.com
  
• If X is a photo, and X has a facebook ID, then additional photo metadata for X can be found in the Facebook Graph API
  

Rules can have dependencies:

• If X is a game, and X is a gameboy ROM, then X has a unique ID at address $00AC
  
• If X is a game, and X has a unique ID, then game metadata can be found at http://thegamesdb.net
  

So when a content add-on is installed, every directory and listitem is scanned into the database. Kodi then applies rules to every item in the database until no more metadata can be derived.

Since skinning is pretty easy in python it would be cool to be able to build a local library of other stuff also from python addons. There are a lot of requests now and then for addons that require an independent library (books, sports recordings, etc). This would probably be doable by creating a library (and use the other operations) using those bindings and taking advantage of both skinning and new library system.
I understand you plan to add support for binnary addons just not sure if python will receive any love

With NoSQL, everything is schema-less, so alternative libraries are a possibility

[da-anda]

QOM stands for Query Object Model, so you're building a query object consisting of nested constraint objects that will get translated to the native query format by the database system used. Same can ofc be achieved by using a simple JSON structure or nested cVariants. Using objects for this has the benefit that one can't really fuck up the structure, unless he's having a typo in a property name, and that it's far nicer to deal with and read code wise. Downside is ofc that it requires an extra parsing step compared to a raw query statement in SQL for example. I'm dealing with this in a PHP framework I'm using and it's really nice to work with. I'm quite confident that it'll work for our needs, but it also has limitations ofc. Using a query abstraction language won't give you the full power and flexibility of native raw statements. Like for a project I had to use raw statements because I had to order query results of locations by distance to another location. I ofc could have done all this also in application logic (fetch all objects, loop over them and do the processing), but it would have been far to slow. So there could be some rare edge cases that might require direct access to the underlying database handle. I'm not saying that we will have them, but the interface you design should allow to bypass the abstraction. Code that's bypassing is ofc limited to certain database systems - but as said, these will be edge cases we most likely won't hit.

If you'd like to read a bit about the persistence layer of the framework I'm using feel free to read it's documentation for some inspiration for the interfaces and architecture:
- http://flowframework.readthedocs.org/en/...tence.html
- http://api.typo3.org/flow/master/ (check out the ...\Generic\Persistence stuff, the Query object is what devs mainly interact with to build queries and their constraints)

Feel free to ping me on Slack or IRC if you want to know more

edit: To access/address nested objects in the QOM, the framework I'm using is using a dot syntax, like aggregateRoot.propertyHoldingObject(s).property. Examples for this in my framework would be:

Code:

// let's assume someone would like to see all movies with Jean Connery

// a repository is the object/service to manage aggregate roots
// it usually consists of methods like "findByFoo()" for fast access to simple queries
// inside these methods the QOM is used to fetch the objects from the persistence layer
$movieRepository = new \MovieRepository();

// let's search by actor name
$result = $repository->findByActorName('Jean Connery');

// as we will most likely will have actor objects in a probably separate
// repository/microservice and not deal with their name as string, let's deal with objects
$actor = $actorRepository->findByName('Jean Connery');
if ($actor) {
  $result = $movieRepository->findByActor($actor);
}

// for more flexibility and more complex queries I usually create a Demand object that holds
// some filter criteria to be processed and converted into a QOM statement by the repository
// in our case the demand could be a smartplaylist representation (xml, json, object)
$demand = new \MovieDemand();
$demand->addActor($actor);
$demand->setLimit(50);
$result = $movieRepository->findDemanded($actor);


// inside the repository it would look like this:
function findByActorName($name) {
  $query = $this->createQuery();
  return $query->matching(
        // as we're searching for the property of a child object from the movie
        // the left operand uses the dot syntax to reference to the name property
        // of related actors, while the actors property of the movie is a object storage/array.
        // Internally the persistance layer will automatically build all required join
        // statements etc required for SQL. No idea how this would be handled with noSQL
        $query->like('actors.name', $name)
    )->execute();
}

function findByActor(\Actor $actor) {
  $query = $this->createQuery();
  return $query->matching(
        // as we're here passing an object as second operand
        // and the 'actors' property is a M:N relation, we're using
        // contains statement which internally will do a search based
        // on the UUID of the actor object in the relation tables etc
        $query->contains('actors', $actor)
    )->execute();
}

function findDemanded(\MovieDemand $demand) {
  $query = $this->createQuery();

  // build an array of constraints which is later passed to the ->logicalAnd() statement
  // to apply AND conditions between all these constraints
  $constraints = array();
  if ($demand->getActors()->count()) {
    $constraints[] = $query->contains('actors', $demand->getActors());
  }
  if ($demand->getGenres()->count()) {
   $constraints[] = ...
  }
  ...
  return $query->matching(
    $query->logicalAnd($constraints)
  )->execute();
}

[topfs2]

Requires a NoSQL backend. wsnipex helped me integrate Kyoto Cabinet and LevelDB into unified depends, so we currently have two choices.

Check out ejdb based on Tokyo Cabinet (I think they are considering moving to Kyoto Cabinet) but with a MongoDB-ish query language.

A while back I set up a test with a few different NoSQL solutions to test out what would fit us best for Kodi. I could publish the code if you want but my conclusion was that ejdb + mongodb is probably the best. Otherwise I really liked the pure key value stores though, they are easy to abstract and are plentiful but searching is a bitch

[zag]

The innovation here is with unified content scrapers. Each scraper is a small python rule that describes how to derive metadata. For example, here's a few scrapers:

• If X is a video, and X has a title, and X has a year, then additional movie metadata for X can be found at http://themoviedb.org
  
• If X is a game, and X has a title, and X has a platform, then additional game metadata for X can be found at http://thegamesdb.net
  
• If X is a song, and X has a title, and X has an artist, then additional song metadata for X can be found at http://theaudiodb.com
  
• If X is a photo, and X has a facebook ID, then additional photo metadata for X can be found in the Facebook Graph API
  

That would be very... very cool!

[da-anda]

EJDB reads quite nice and their query syntax would map nicely to a abstracted generic query language/QOM

[garbear]

@da-anda thanks for the info above, this is helpful as I'm trying to wrap my head around the problem

Check out ejdb based on Tokyo Cabinet (I think they are considering moving to Kyoto Cabinet) but with a MongoDB-ish query language.

this is almost verbatim what I planned to implement - a basic query engine on top of a Kyoto Cabinet (or LevelDB) KV store. I think my life just got easier

A while back I set up a test with a few different NoSQL solutions to test out what would fit us best for Kodi. I could publish the code if you want but my conclusion was that ejdb + mongodb is probably the best. Otherwise I really liked the pure key value stores though, they are easy to abstract and are plentiful but searching is a bitch

Right now my prospective lineup is:

• LevelDB (KV store)
  
• Kyoto Cabinet (KV store)
  
• UnQLite (document store)
  
• EJDB(dcoument store)
  

I tried integrating both MongoDB c++ drivers, the legacy one and the new one still in-planning. Both were laden with dependencies and incredibly hard to integrate into our depends. Still, it would be nice to have a network option for the MySQL crowd.

That would be very... very cool!

welcome to heimdall

[takoi]

A feature I'm currently working on is content add-ons. These are similar to python plugins, but content is scanned into the library instead of browsed through the add-ons node.

A quick perusal of the Internet Archive ROM Launcher plugin yields a delightful 10,390 games. If the "launching" part is separated from the "browsing" part, this plugin could easily become a content add-on that provides an instant game library, 10,000 games strong.

I've looked at that content branch before, but it seems to be gone now.. Could you give an overview of the design of this? How do you plan on keeping the 10000 games in sync with upstream changes? It's something I've always wanted to do for other media, though not as something separate but a replacement for plugins to kill some larger design problems with the old system in the process. That would mean keeping the old 'browsing' and other parts optional as it's still useful (and often required) for many addons to do custom vfs and ui.

[garbear]

I've looked at that content branch before, but it seems to be gone now..

I move abandoned code to https://github.com/garbear/xbmc-retroplayer

Could you give an overview of the design of this? How do you plan on keeping the 10000 games in sync with upstream changes? It's something I've always wanted to do for other media, though not as something separate but a replacement for plugins to kill some larger design problems with the old system in the process. That would mean keeping the old 'browsing' and other parts optional as it's still useful (and often required) for many addons to do custom vfs and ui.

I know I haven't explained everything properly, it's even hard for me to keep track of all the moving parts. I'll try again

Content add-ons are similar to plugins, where Kodi invokes a plugin:// url like plugin://plugin.program.iarl/Emulator/Nintendo+64+-+N64/1 and shows the directory in the GUI. Instead, content add-ons will show up at content://plugin.program.iarl/ and are scanned to the database. (OFC, they're also browsable in the GUI).

To enable local files to be scanned into the database, the content:// can also point to local media sources. For example, if your sources.xml looks like this:

Code:

<sources>
    <games>
        <source>
            <name>Main Games Collection</name>
            <path>smb://username:[email protected]/share/music/</path>
            <allowsharing>true</allowsharing>
        </source>
    </games>
</sources>

then the url content://sources/Main%20Game%20Collection/ would be translated to the location of this media source.

The corollary to content add-ons is content scrapers. These will be python-based "rules", as I mentioned earlier.

I'm turning Heimdall (a GSoC project from topfs2 back in 2012) into a service add-on. The add-on is really basic: it traverses the content:// protocol and, for each FileItem, applies rules until no more metadata can be derived. This metadata is then "written" back to the VFS.

Scrapers will be isolated from all of Kodi. They will only interact via reading and writing to the content:// protocol.

The game library will interact with this content via the the contentdb:// protocol. The contentdb:// protocol is backed by the unified content database, which stores all details of all files. The game library is defined by XML nodes, just like xml video library nodes. For example, the game titles node will map to the url contentdb://games/titles.

By "upstream changes" you mean changes to the scraper's site, e.g. https://www.themoviedb.org right? As Heimdall runs in the background, it could continuously do scans, refreshing metadata when it expires past a certain age.

does that do a better job of explaining things?

[garbear]

The innovation here is with unified content scrapers. Each scraper is a small python rule that describes how to derive metadata. For example, here's a few scrapers:

• If X is a video, and X has a title, and X has a year, then additional movie metadata for X can be found at http://themoviedb.org
  
• If X is a game, and X has a title, and X has a platform, then additional game metadata for X can be found at http://thegamesdb.net
  
• If X is a song, and X has a title, and X has an artist, then additional song metadata for X can be found at http://theaudiodb.com
  
• If X is a photo, and X has a facebook ID, then additional photo metadata for X can be found in the Facebook Graph API
  

That would be very... very cool!

Here's an example rule for a music file that derives the artist, album and title using Mutagen: https://github.com/topfs2/heimdall/blob/...io_item.py . The rule can be read "If X is a music file (item.audio), then the artist, album and title of X can be found using mutagen"

[takoi]

Content add-ons are similar to plugins, where Kodi invokes a plugin:// url like plugin://plugin.program.iarl/Emulator/Nintendo+64+-+N64/1 and shows the directory in the GUI. Instead, content add-ons will show up at content://plugin.program.iarl/ and are scanned to the database. (OFC, they're also browsable in the GUI).

To enable local files to be scanned into the database, the content:// can also point to local media sources. For example, if your sources.xml looks like this:

I was mostly interested in what the addons are the "are scanned to the database" part. With 'changes' I meant the change to the actual content (like games getting removed/added from internet archive). Besides the protocol name, what's the difference from plugins? Is it the same but non-vfs parts removed? That would certainly make more sense wrt. the scanning part and content changes.

[garbear]

I was mostly interested in what the addons are the "are scanned to the database" part. With 'changes' I meant the change to the actual content (like games getting removed/added from internet archive). Besides the protocol name, what's the difference from plugins? Is it the same but non-vfs parts removed? That would certainly make more sense wrt. the scanning part and content changes.

Yes, I purposefully renamed the protocol to "start fresh". Everything about plugins has been dropped except for generating folders and file items.

If additional functionality is required in the future, the API can be expanded, but for now a small, clean API is needed for development.

So... pictures! At the root of the game library is a link to add game content add-ons:



When selected, this shows a list of all installable add-ons with the extension point "kodi.content.game". I transformed IARL into a content add-on (content.game.internet.archive), so it shows up here:



When the Internet Archive add-on is selected, it is downloaded and the game library changes to content://content.game.internet.archive:



In the background, a Heimdall service is started with the parameter content://content.game.internet.archive. It walks the VFS, deriving metadata and writing the metadata back to the VFS.

If the content on the Internet Archive changes, the content can be re-scanned by simply calling the Heimdall service again with content://content.game.internet.archive as the parameter.

This could be put on a timer, so every 24 hours the Heimdall service can be called. Or it can be called from a context item labeled "Scan to library".

[natethomas]

This could be put on a timer, so every 24 hours the Heimdall service can be called. Or it can be called from a context item labeled "Scan to library".

Given that this is an add-on for a theoretically always-on internet service, perhaps it would make sense to trigger Heimdall any time trying to launch a game returns an error?

Though, that might not be a great idea, since a faulty internet connection could cause a total loss of your library.

[Tolriq]

Have not yet played with Retroplayer but love the concept.

Just 2 question with one directly related : Does this new content things works well for remotes (meaning browsing via webserver like vfs or things like that ?)

And unrelated but does the controllers of Retroplayer works from Event Server ? Meaning I can emulate a physical remote from a phone for example ?