JSON VideoLibrary.SetMovieDetails: Improve Speed?
#1
Hello,

I'm moving my add-on from direct SQL Executes to the JSON API. When I did a speed test I see a huge speed loss (200ms->800ms). That's four times slower!

Cause I want to update every movie in my DB (1100 pieces) this is a huge disadvantage. Is there any possibility to speed JSON up?

like update several movies at a time or make asynchronous JSON requests?

Any ideas?
Reply
#2
I've never used it, but the jsonrpc does support batch executions which should theoretically be faster, but no idea how much
Reply
#3
(2013-06-06, 15:55)Bstrdsmkr Wrote: I've never used it, but the jsonrpc does support batch executions which should theoretically be faster, but no idea how much

ok, thank you, but how do I do batch JSON? simple { {request1}, {request2} } ?
Reply
#4
Sorry, I don't know for sure. You might check the jsonrpc section
Reply
#5
(2013-06-06, 16:15)Bstrdsmkr Wrote: Sorry, I don't know for sure. You might check the jsonrpc section

I only found that batch requests are supported but not how.

Maybe someone knows how to do it...

I'll try some possibilities...
Reply
#6
ok here is the solution:

[{request1},{request2}]

but a dev said that this doesn't make it much faster...
I'll test it.
Reply
#7
(2013-06-06, 18:03)Jandalf Wrote: ok here is the solution:

[{request1},{request2}]

but a dev said that this doesn't make it much faster...
I'll test it.

The actual request won't be faster but you'll skip a bunch of overhead on the protocol as you send one request and them all are processed before you get a response. i.e. it should be quite a bit faster in the grand scheme of things.

Just don't assume they are processed in the order you put them, its done in order in xbmc but JSONRPC in general doesn't garantue this. But I think in your cause the order is unimportant.
If you have problems please read this before posting

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.

Image

"Well Im gonna download the code and look at it a bit but I'm certainly not a really good C/C++ programer but I'd help as much as I can, I mostly write in C#."
Reply
#8
(2013-06-06, 18:33)topfs2 Wrote: The actual request won't be faster but you'll skip a bunch of overhead on the protocol as you send one request and them all are processed before you get a response. i.e. it should be quite a bit faster in the grand scheme of things.

Just don't assume they are processed in the order you put them, its done in order in xbmc but JSONRPC in general doesn't garantue this. But I think in your cause the order is unimportant.

I don't care for the order. What batch size will be the best? 5,10,20, is there a limit?

And my other question: can I run a JSON request async, so I can collect data (JSON request to another server), start a batch request, collect the next bunch of data, wait for the first batch to complete, start thte next batch reqeust...?
this would give me 50% more speed.
Reply
#9
JSON is incredibly inefficient when updating the movie database, and batching your updates won't make a blind bit of difference.

Whenever you update one database property, JSON will delete all the rows from the database of the movie being updated (tear it down), then re-insert the same rows (re-create the movie, with the new property).

For instance, adding new artwork will result in all rows, including actor thumbnails, being removed from the database and then re-inserted. Look at the log of 205 database operations when adding just one new artwork item - it's insanely inefficient. And yet nobody seems particularly bothered about this.

If I were you, I'd avoid using JSON, it's quite nice for reading data out of the database but absolutely woeful at putting it in.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
#10
(2013-06-06, 19:44)MilhouseVH Wrote: Whenever you update one database property, JSON will delete all the rows from the database of the movie being updated (tear it down), then re-insert the same rows (re-create the movie, with the new property).

For instance, adding new artwork will result in all rows, including actor thumbnails, being removed from the database and then re-inserted. Look at the log of 205 database operations when adding just one new artwork item - it's insanely inefficient. And yet nobody seems particularly bothered about this.

If I were you, I'd avoid using JSON, it's quite nice for reading data out of the database but absolutely woeful at putting it in.

Your more than welcome to improve it if you can, anything making it speedier is very welcomed!

One gsoc project is about database improvement and hopefully it will make it easier (and faster) for JSON-rpc to interact with the database
If you have problems please read this before posting

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.

Image

"Well Im gonna download the code and look at it a bit but I'm certainly not a really good C/C++ programer but I'd help as much as I can, I mostly write in C#."
Reply
#11
(2013-06-06, 20:17)topfs2 Wrote: Your more than welcome to improve it if you can, anything making it speedier is very welcomed!

One gsoc project is about database improvement and hopefully it will make it easier (and faster) for JSON-rpc to interact with the database

I don't underestimate the amount of work involved here, and in some ways I can understand why the current approach has been taken as it is the simplest way of updating any property, but it's also the least efficient approach by a very wide margin.

However as a short term measure (assuming the GSOC project is the longer term solution), a few "quick wins" would be to optimise some of the more common JSON update queries so that a complete tear down of each media object is avoided when it isn't remotely necessary. For instance when adding a new artwork item, it shouldn't be necessary to remove every database row, a solution which keeps things simple, but which is totally nuts in terms of efficiency, particularly when cast artwork is involved - instead of 205 database operations, only one (an INSERT) is required.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
#12
(2013-06-06, 19:44)MilhouseVH Wrote: JSON is incredibly inefficient when updating the movie database, and batching your updates won't make a blind bit of difference.

Whenever you update one database property, JSON will delete all the rows from the database of the movie being updated (tear it down), then re-insert the same rows (re-create the movie, with the new property).

For instance, adding new artwork will result in all rows, including actor thumbnails, being removed from the database and then re-inserted. Look at the log of 205 database operations when adding just one new artwork item - it's insanely inefficient. And yet nobody seems particularly bothered about this.

If I were you, I'd avoid using JSON, it's quite nice for reading data out of the database but absolutely woeful at putting it in.
(2013-06-06, 20:17)topfs2 Wrote: Your more than welcome to improve it if you can, anything making it speedier is very welcomed!

One gsoc project is about database improvement and hopefully it will make it easier (and faster) for JSON-rpc to interact with the database
Ok, I can't code C++ and I know nearly nothing about Git and I dont' know how you handle development here at XBMC but I'm now how to code in general and after looking into "SetMovieDetails" code I understand MilhouseVH. If you only want to update a single field that has no relations the actual way is much away from perfect.

I don't know what was the intention behind doing it this way, I think it could be because an update of single database fields is rarely done by core XBMC or scrappers. I think only add-ons really need update single fields.

So I will dig into this topic and will see what I can do to optimize this.

Is ok when I post me ideas in the development forum?
Reply
#13
That sounds great Jandalf, many thanks.

I first noticed this problem with the Artwork Downloader addon, which adds new artwork (eg. fanart, posters, clearart, clearlogo, discart, extrafanart etc.) to movies and tvshows. Each time a new item of artwork is added to the database it hits this inefficiency, so if you are adding 6 items of new artwork for one movie that's (6 movie teardowns * however many rows need to be re-inserted) just to insert 6 new rows. In the example I linked to, that's 6 * 205 database operations required to insert the 6 new rows, and this workload isn't taking into consideration any extra storage and processing required of the client in order to recreate all the deleted rows (which could be significant on something like a Raspberry Pi). Consequently, Artwork Downloader is incredibly slow.

I would imagine that if some optimisations were employed in SetMovieDetails, most addons that perform database operations would benefit from an instant and very noticeable speed boost. It may also be worth taking a look at similar JSON calls, such as SetTVShowDetails, to confirm if anything similar is occurring there.

I doubt however that the built-in scraper uses JSON to update the database, in fact none of the core XBMC functionality seems to use JSON, instead hitting the database direct. It's easy to understand why... Smile
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
#14
JSON RPC was added after the core functionally was, it uses core functionality not the other way around. This, for obvious reasons.

The reasons why it was done this way are either a) there is some limitation in the CDatabase forcing us to do a complete reinsert or b) the most wanted feature was to completely wipe the movie (i.e. scraper thought avatar but it was star trek) or c) no-one wanted to simply add extra data to the currently scraped movie or d) simply no-one had the time to add it like this. possibly due to b) or c) it could also have been e) there exist no API in core for doing this, thus each part (scraper etc.) uses their own homebrewed SQL for it

Surely the scrapers do a complete teardown if we reset from avatar to star trek? Otherwise I'm missunderstanding the problem.

Anyways, I cannot stress d) enough. We are very few developers and it helps no-one to complain that its slow, what helps is someone stepping up and making it faster Smile

If its reason a) or e) this years GSoC might help quite a bit. If its b, c, or d this is a perfect place for a new developer to ease its way into xbmc code, its not that scary Smile
If you have problems please read this before posting

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.

Image

"Well Im gonna download the code and look at it a bit but I'm certainly not a really good C/C++ programer but I'd help as much as I can, I mostly write in C#."
Reply
#15
IMHO using JSON for core functionality would make little sense in many cases - for example, the GUI retrieving movie details when entering the movie library. This is all C code, using JSON would be unnecessary communication overhead (not to mention complication) and slow things down when you need optimum performance. For addons, JSON makes a lot of sense, and is indeed fine when retrieving most data, it's just the updates that are in need of improvement. Fingers crossed someone will be able to make some progress on that.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply

Logout Mark Read Team Forum Stats Members Help
JSON VideoLibrary.SetMovieDetails: Improve Speed?0