Is "application/json-rpc" even a valid MIME type? I've never heard of it (which obviously doesn't mean that it doesn't exist).
The re-caching in the thumbloader is purely backward compatibility with Eden, and you'll note it only does it a single time for each item (as no matter if we find anything or not, we record the state in the db). If someone comes up with a significantly better way to do it, feel free. The best solution I suspect is doing something similar to the music scanner - i.e. have the video scanner take a flag "update backwards compat art" that goes away and does it all. I just don't have the time to do that.
As for the general slowness, this is due to 2 things:
1. The separate queries for art.
2. The open/close of the db for each of those queries.
The second can be fixed I suspect (the thumbloader already handles it - you just need to hold onto the object throughout the list processing). The first can't really be - after all, art is a 1:many map, so you can't retrieve it all in one go. The best thing is for clients to only fetch the information when they actually need it (i.e. not when grabbing a list of 19000 songs!)
Cheers,
Jonathan
Seeing my times on a fast ssd PC I really suspect that the scan occurs all the times for music.
In the codes there's still lot's of reference to old style thumbs so quite hard to follow, but in bool CMusicThumbLoader::FillLibraryArt(CFileItem &item) I do see database updates for artists to avoid scans on next times I don't see things for albums / songs. But I must admit I'm a little lost in the code.
For testing purpose I have multiple Xbmc installs.
In one a very big database for music filled with songs that are no more accessible from Xbmc to avoid fill of the thumbs.
The art table only contains references to artists with empty values. Nothing about songs or albums.
In another one with few songs without thumbs in tags or folder.jpg ... but that do exists
The art table contains references to artists and albums with empty url but nothing for songs.
About the when to get the data, remotes that do offline sync or need to display data like to have a good speed for end users if we don't get thumbs and fanart during the sync then we need to call a getdetails that will be slow due to separate query then prepare for downloads then get the image this 19000 times while if we already have the info we know the images are the same and don't try to get info about another one.
Side effects of this are :
- we have no way to know if there's images before calling the getdetails
- we have no way to know if images have changed before calling the getdetails so whenever the user do a full sync we need to restart a full images sync to get new data.
To improve the second point mentioned by jmarshall I've submitted PR1462. On my system this brings the response time for retrieving a list of 1500 songs with thumbnail and fanart down from ~14 seconds to 3.5 seconds so quite a bit faster. It's still nowhere near the 1 second it takes to retrieve the songs without artwork but already a lot closer.
(2012-09-22, 03:36)N3MIS15 Wrote: [ -> ]you could simply change the lib.
Change line 120 in jsonrpc.py to:
Code:
connection.putheader("Content-Type", "application/json")
While changing the lib is not the best solution.. If you include the lib in your app it should be safe. This is how we got around it in Maraschino.
I also wrote my own class to behave exactly like jsonrpclib if you're interested, it does not support batch requests tho. http://forum.xbmc.org/showthread.php?tid...pid1138156
Thanks, I had to import my own jsonrpclib, instead of the easy_install jsonrpclib (which is in an egg format, don't know how to edit that).
So I changed "import jsonrpclib" to "from lib import jsonrpclib".
Also had to change "import lib.jsonrpclib" to "import jsonrpclib".
Finally my XBMC is able to be interfaced by Siri (Ac!d Siri with SiriServerCore running on my Synology DS212+).
(2012-09-22, 12:49)Montellese Wrote: [ -> ]To improve the second point mentioned by jmarshall I've submitted PR1462. On my system this brings the response time for retrieving a list of 1500 songs with thumbnail and fanart down from ~14 seconds to 3.5 seconds so quite a bit faster. It's still nowhere near the 1 second it takes to retrieve the songs without artwork but already a lot closer.
Is there a way to dl the diff from the PR ? I'm not github expert and can't find an easy way to apply this patch in my tree
And does this PR also cover the same problem that will arise with tags ?
Anyway thanks for trying to correct this
You can append a ".patch" to the URL of the PR and it will give you a patch file you can apply with "git am <path-to-patch>".
What problem with tags?
Thanks
From what I read in the code Tags also open / closes the databases and do 1 request to get them but I may be wrong just think to saw that when trying to understand where the slowness was.
So does retrieving the cast and some other extra information. But in the description of those methods in the JSON schema it explicitly states that retrieving those tags will slow things down. You can't pull one-to-many relations into a SQL view so you have to retrieve them seperately.
Yes I know that
Was just asking if the open / close database was also present for those and so get a performance boost too with your patch.
Seems date added is not present for audio.
Does the getalbums without sort send them by added order ? or is there a special way ?
Edit : How is the dateadded string generated ? Is the format host date settings dependent ?
Audio doesn't have the same dateadded functionality as videos that's why there's no dateadded there.
If you don't provide a sort method it will be returned in the order of how the items are in the database so most likely by ID.
The date added string should be in SQL format i.e. YYYY-MM-DD hh:mm
s
Ok so as ID should be auto increment I suppose this will do the trick.
Thanks for both answers.
(2012-09-22, 11:17)jmarshall Wrote: [ -> ]1. The separate queries for art.
[...]
The first can't really be - after all, art is a 1:many map, so you can't retrieve it all in one go.
But can there be multiple "thumb"'s for 1 item?
Because, mainly, the performance issue is because the remotes want to retrieve the thumb of a list of items.
If there is only 1 thumb (which seems to be the case, unless mistaken), we add it to the views and performance is great in this case.
In my remote mysql case, it adds 100ms for a ~50 items query vs. adding nearly 1sec PER ITEM.
I have an unpublished patch for this.
(2012-09-22, 11:17)jmarshall Wrote: [ -> ]2. The open/close of the db for each of those queries.
I already tried something which avoided the open/close.
Although the performances increased, it still wasn't usable, whether with Yatse or the official remote's fork which implement JSON.
BTW, asking for "resume" also fetches from the database for each item, while the info is available from the views. This also degrades performance....
I'm working on a patch.
(2012-09-24, 16:40)Koying Wrote: [ -> ] (2012-09-22, 11:17)jmarshall Wrote: [ -> ]1. The separate queries for art.
[...]
The first can't really be - after all, art is a 1:many map, so you can't retrieve it all in one go.
But can there be multiple "thumb"'s for 1 item?
Because, mainly, the performance issue is because the remotes want to retrieve the thumb of a list of items.
If there is only 1 thumb (which seems to be the case, unless mistaken), we add it to the views and performance is great in this case.
Not really multiple thumbs but multiple pieces of artwork. Currently we only have thumbnail and fanart which is why they are directly supported in the JSON-RPC calls but in the future xbmc will support clearart, logos, and all that stuff and we don't plan to add a single field for every one of those into the database view so while this approach might work right now it will utterly fail in the future.
(2012-09-24, 16:40)Koying Wrote: [ -> ]In my remote mysql case, it adds 100ms for a ~50 items query vs. adding nearly 1sec PER ITEM.
Using mysql certainly doesn't help when it comes to better performance.
(2012-09-24, 16:40)Koying Wrote: [ -> ] (2012-09-22, 11:17)jmarshall Wrote: [ -> ]2. The open/close of the db for each of those queries.
I already tried something which avoided the open/close.
Although the performances increased, it still wasn't usable, whether with Yatse or the official remote's fork which implement JSON.
As you can see in the numbers from my system it still makes quite a bit of a difference so it should certainly go in.
(2012-09-24, 16:40)Koying Wrote: [ -> ]BTW, asking for "resume" also fetches from the database for each item, while the info is available from the views. This also degrades performance....
I'm working on a patch.
Thanks for reminding me. Wanted to fix that for a while after I integrated the resume point details into the views but always forgot about it. Done in
https://github.com/xbmc/xbmc/commit/34f0...aaecde13a3