[null_pointer]

I am working on some additions to the JSON-API, one of the main things we need is the ability to add items, I am working on adding a few new endpoints that allow adding items, to start with movies and eventually people and streams etc.

Does anyone have an issue with this, what I mean is, it looks like adding items was left out perhaps intentionally, does anyone have an issue with adding items from scratch using the JSON-API and bypassing the scrapers all together.

[Montellese]

Back when I first looked at adding items through JSON-RPC we didn't even have the VideoLibrary.SetFooDetails methods so simply allowing to add new items without scraper support would have left you with adding an item with only the path. That didn't make any sense so I dropped it. After that I simply never had the time to look into it again. It's probably rather simply for movies and musicvideos but could be a bit more complex for tvshows, seasons and episodes because they depend on each other. Making it work with scraper support will be quite a bit more difficult because the code that integrates our scrapers is terrifying. It usually breaks just by looking at it.

[xnappo]

Good to hear. Yes - we already ran into the interactions between the TV stuff when doing the directDB stuff - took us a while to sort out because if it isn't perfect it breaks in strange ways.

[Martijn]

Slightly related, here's an outdated PR for doing add source https://github.com/xbmc/xbmc/pull/5484

It may come in handy as reference or perhaps even include it as well

[Montellese]

Yeah I've been struggling with this for a while now in my media importing work as well. But there I need to handle adding, updating and removing items and if all episodes have been removed I also need to remove the seasons that don't make any sense anymore and if all seasons have been removed I also need to remove all tvshows etc.

[null_pointer]

Ok thanks for the confirmation. I wanted to get an idea if adding items via JSON-API and bypassing the scrappers was acceptable, it sounds like it is, I will continue to do some work on this then.

Yes I expect Shows/Seasons/Episodes will be a little more involved than movies, I was going to work on movies first and then move on to the others.

On a side note, one thing I noticed was that the request even when batch together still creates a connection to sqlite, In one of the test end points I am playing with I am passing a list of items in a single request, this means the request is a single request for say 30 items and thus the DB connection is only initialized once for that request. Do you know if this has been tried before and in your experience do you think this will help with processing speed and overall system latency (gui responsiveness).

[Montellese]

I've always found it a bit odd that we are opening and closing database connections all the time all over the Kodi core code. But I have no idea if connection-pooling would really improve performance / responsiveness.

Furthermore in case of JSON-RPC requests this would be quite tricky as requests need to be independent from each other (even when batched). Furthermore at the place where we know that we are handling a batch request we don't really know which requests require which database connection (if any at all) and you can't assume that if the first request requires a db connection so will any/all the other requests. And at the place where we handle individual requests we have absolutely no clue that there are any other requests so you don't know that you should leave the db connection open and even worse you don't know when to close it.
So you'd probably have to write very specific code into a place which is designed to be as generic as possible i.e. deserialize a JSON-RPC request, lookup the implementation, execute it, serialize the JSON-RPC response (without any knowledge about the actual method implementation).

[null_pointer]

I guess all I was trying to say was that in some requests like SetMovieDetails you could pass in an array of movies that are going to be updated, it could open the connection do all the work and close the connection. This is better than batched requests where each one would need to open set the movie and then close, then open again for the next request etc.

Anyway I will have a play and see if it makes any difference.

[Tolriq]

I've always found it a bit odd that we are opening and closing database connections all the time all over the Kodi core code. But I have no idea if connection-pooling would really improve performance / responsiveness.

This is the cause of some problems with Mysql performance, and major concern for Rpi users

You can queue an album in seconds since it's one open / close but it takes ages to queue all files 1 by 1 since it opens / close query items each times and opening a big db on slow sdcard is well very slow.

There was long discussion last year about that but nobody never took a final decision about the need.

[Razze]

I've always found it a bit odd that we are opening and closing database connections all the time all over the Kodi core code. But I have no idea if connection-pooling would really improve performance / responsiveness.

This is the cause of some problems with Mysql performance, and major concern for Rpi users

You can queue an album in seconds since it's one open / close but it takes ages to queue all files 1 by 1 since it opens / close query items each times and opening a big db on slow sdcard is well very slow.

There was long discussion last year about that but nobody never took a final decision about the need.

Sorry, but I don't think that has been proved.

I'm pretty sure it's not the connection, that is causing this, at least not for sqlite, so if your only talking about mysql you might be right. All my traces were on sqlite so far.

[angelblue05]

The way it's done in the add-on currently is one opened database connection to process multiple items and it's blazing. I am positive the way null_pointer is doing things will improve the performance overall. I've seen opening and closing database connections at large scale and it really slows things down (which is where we hit a wall with json-rpc the first time around).

[Razze]

The way it's done in the add-on currently is one opened database connection to process multiple items and it's blazing. I am positive the way null_pointer is doing things will improve the performance overall. I've seen opening and closing database connections at large scale and it really slows things down (which is where we hit a wall with json-rpc the first time around).

You keep forgetting, that your addon is not only dodging the open and close. But also alot of logic and wrappers and json-rpc etc.
So it might be the open and closing, but nobody has yet provided any hard data and I'm not sure if your looking in the right direction.

Also one time opening, doing your stuff and closing, might be faster for your plugin. But you might end up locking all other plugins and kodi out for the time that you are writing.

[Montellese]

Also one time opening, doing your stuff and closing, might be faster for your plugin. But you might end up locking all other plugins and kodi out for the time that you are writing.

This is a problem I've been seeing more and more now when working on my media importing stuff. When I set up two Kodi instances to import movies, tvshows and musicvideos from the other instance (so that both instances have all the items) and I start the instances both immediately start synchronisation. This slows at least one instance down a lot because it is trying to provide the other instance with its local items while it is already synchronising the items received from the other instance which results in a lot of simultaneous read and write accesses.
If I change the synchronisation mechanism on one instance to only be triggered manually and wait until the other instance has finished synchronisation before manually triggering the synchronisation it is a lot faster.

And in the log I can see a lot of sqlite messages about conflicted/locked database accesses.

[Tolriq]

Well writing simultaneously will always be a problem, but from an sqlite point of point of view one connection and WAL mode will always be much more efficient than normal journal mode and opening closing.
WAL mode with proper transaction would also make it way easier for you to detect locks and delay one of the sync.


I have no knowledge to show that in Kodi, but in Yatse the gains were quite big.

[Montellese]

In my media importing there are no simultaneous writes. For every imported media type (movies, tvshows, seasons, episodes, musicvideos) I first read all the previously imported items from the db, compare them to the received/imported items (doesn't have anything to do with the database) and then start a db transaction and update one item at a time in that transaction. And different media types are handled sequentially (because you need to import tvshows before seasons before episodes or you'll get a huge mess). So it definitely is also a problem for simultaneous reads and writes.

Concerning WAL mode IIRC it does not work over network filesystems and with Kodi it is possible to store your userdata on a remote server and set up path substitution.