2014-01-25, 16:28
Hi,
I've just updated to the nightlies for XBMC (although I suspect the issue is present in 12.x) and thought I'd run the artwork downloader to update clearart, banners etc etc.
I found that it seemed to be taking a large amount of time, so I've started digging into why. Note that a factor here is that I'm using mysql on a NAS, so that I can share the db across machines. This will be exaggerating the round trip time for each sql call.
There appear to be a few issues at play when using artwork downloader (however, I'm not sure it's purely to blame, as the json-rpc api for doing an update does look to imply that internally it's just one element being updated/set):
Total time for updating one film is 15s, a chunk of which is the database deleting and recreating the movie, see pastebin:
http://pastebin.com/Tq28NS3K
The time to do one SetMovieDetails call is ~3s. (and it also looks like a ffmpeg is fired up to examine the mkv on pretty much each call)
Overall it seems inefficient to delete everything and reinsert things, as it's not just the movie object it's all the side items, eg actors, genres, that are also unlinked and recreated.
So I'm wondering how to improve this. I can think of a couple of ideas:
Relevant code from VideoLibrary:
https://github.com/xbmc/xbmc/blob/master....cpp#L1989
and json-rpc:
https://github.com/xbmc/xbmc/blob/master...y.cpp#L473
Any thoughts? Have I missed something obvious that the json-rpc/VideoLibrary could be doing instead? Or should I throw some code together and see what works?
Thanks,
Chris
I've just updated to the nightlies for XBMC (although I suspect the issue is present in 12.x) and thought I'd run the artwork downloader to update clearart, banners etc etc.
I found that it seemed to be taking a large amount of time, so I've started digging into why. Note that a factor here is that I'm using mysql on a NAS, so that I can share the db across machines. This will be exaggerating the round trip time for each sql call.
There appear to be a few issues at play when using artwork downloader (however, I'm not sure it's purely to blame, as the json-rpc api for doing an update does look to imply that internally it's just one element being updated/set):
- artwork downloader makes 5 individual JSON-RPC calls to VideoLibrary.SetMovieDetails, setting just one element for each artwork.
- the json rpc api fetches the movie information, and merges the changes in, it then calls CVideoDatabase::SetDetailsForMovie
- CVideoDatabase::SetDetailsForMovie then proceeds to delete the movie from the database, and re-adds the whole movie. This causes a number of SQL calls to happen, which are remarkably slow.
Total time for updating one film is 15s, a chunk of which is the database deleting and recreating the movie, see pastebin:
http://pastebin.com/Tq28NS3K
The time to do one SetMovieDetails call is ~3s. (and it also looks like a ffmpeg is fired up to examine the mkv on pretty much each call)
Overall it seems inefficient to delete everything and reinsert things, as it's not just the movie object it's all the side items, eg actors, genres, that are also unlinked and recreated.
So I'm wondering how to improve this. I can think of a couple of ideas:
- add a new API to Videolibrary that can be passed a movie id, and a map/table of changes - it can then be smarter and just update items that need to be updated. If it doesn't know how to do a requested update (IE it's too complex) it can error and do a full object refresh (either internally, or ask the caller to do so) I'm thinking that this allows for db/api changes, eg new fields come in, if the code doesn't know how to handle it then it falls back to the full refresh.
- Extend the SetDetailsForMovie API, and add the ability to diff the new movie details against the current movie details, and only update what's changed - while this would catch more call paths it would lose the ability to pass in the knowledge of what exactly has changed (which the json-rpc api has when called) However, it does mean that deeper understanding of how to compare movies is needed (and maintained)
Relevant code from VideoLibrary:
https://github.com/xbmc/xbmc/blob/master....cpp#L1989
and json-rpc:
https://github.com/xbmc/xbmc/blob/master...y.cpp#L473
Any thoughts? Have I missed something obvious that the json-rpc/VideoLibrary could be doing instead? Or should I throw some code together and see what works?
Thanks,
Chris