2017-03-16, 20:17
Only thing I took was that the valgrind issues all seemed to have something to do with Python calls. At that point I am out of my depth.
Options:
a) don't use LDP or skins that require it.
or
b) Power off regularly
or
c) Run on a platform with sufficient memory compatible with the use (skin, addons, library size) you want to make of it.
You know that if you stop LDP and skin.helper the memory problem (implicit behaviour) goes away.
That kind of script, making regular calls that need lots of memory just exasperate the fundamental weaknesses in C++ memory management. If the addon was redesigned to space out the JSON requests, or the requests made more memory efficient then again it would ease the situartion.
Are there also small leaks in the Python interface, or texture handling etc. that makes things worse? Well probably yes, but I'm not sure I have the skills or energy to find them, and I doubt that the issue would vanish even if I did. I have tested the JSON to music library bits exhaustively, and they allocate and release correctly. So, for now at least, I want to move on to areas I can improve.
An example of sort and limit at db being faster (and more memory efficient) is the change to how JSON request for 20 random unplayed songs works in 17.1 RC1 compared to 17.0. If you remember JSON #1 stopped showing the memory issue in your tests.
I have done some time tests on SQLite of sort at db rather than in internal memory, and was saving 15% to 25% overall on the songs node. I need to do more but I have been a bit distracted by some guy chasing a memory leak!
Yet to get a build using rapidjson to test, but I hope to get to it soon. If you get to do that with your big ass library then please report (but work with Estuary or Confluence skin and no addons!)
(2017-03-15, 23:32)Uatschitchun Wrote: I can follow your thoughts, but what would it mean if running long term?If you have a huge library accessed by Kodi on a platform with limited memory, then running polling scripts like Library Data Provider making requests that take big chunks of memory to process will mean that you need to restart fairly regularly.
Options:
a) don't use LDP or skins that require it.
or
b) Power off regularly
or
c) Run on a platform with sufficient memory compatible with the use (skin, addons, library size) you want to make of it.
You know that if you stop LDP and skin.helper the memory problem (implicit behaviour) goes away.
That kind of script, making regular calls that need lots of memory just exasperate the fundamental weaknesses in C++ memory management. If the addon was redesigned to space out the JSON requests, or the requests made more memory efficient then again it would ease the situartion.
Are there also small leaks in the Python interface, or texture handling etc. that makes things worse? Well probably yes, but I'm not sure I have the skills or energy to find them, and I doubt that the issue would vanish even if I did. I have tested the JSON to music library bits exhaustively, and they allocate and release correctly. So, for now at least, I want to move on to areas I can improve.
Quote:Sort and limit at the db is where it belongs. Have you've got 2 example queries to test for speed comparison? A test against yajl and rapidjson each with both queries would be interesting. For speed and memory behavior. Have you tested against sqlite?Yes it does, but it is a fundamental change to Kodi to start sorting at the db. All such sorting has to duplicate that done locally by Kodi when the user switches sort method on a screen full of data. That has bugs in it too (for multiple field sorting). So dealing with all of that is something I am working on for v18.
An example of sort and limit at db being faster (and more memory efficient) is the change to how JSON request for 20 random unplayed songs works in 17.1 RC1 compared to 17.0. If you remember JSON #1 stopped showing the memory issue in your tests.
I have done some time tests on SQLite of sort at db rather than in internal memory, and was saving 15% to 25% overall on the songs node. I need to do more but I have been a bit distracted by some guy chasing a memory leak!
Yet to get a build using rapidjson to test, but I hope to get to it soon. If you get to do that with your big ass library then please report (but work with Estuary or Confluence skin and no addons!)
