• 1
  • 5
  • 6
  • 7
  • 8
  • 9(current)
Problem scanning huge source (mem leak)?
Nevertheless, I'll do some tests, if I find the time today, as Milhouse pushed a release that already uses rapidjson. So maybe that makes a difference...
Reply
I'm still not getting any where on causes, but here is a plot of Kodi memory use (working private bytes) with the Library Data Provider addon running.

Image

That is v17.0, so has the inefficient fetch of all songs just to provide 20 random ones, win32 with SQLite db of 140k of songs, 11k of albums. I tweeked the LDP to request data every 2 mins (rather than 20 mins) to speed up the test.

Initailly everything is fine, 4 LDP calls and the memory returns to base level after each. Then the 5th time base level steps up to ~300MB and gradually increases but not monotonically. Sometimes the memory drops back to the <100MB starting point until the next set of requests.

As previous testing has shown:
  • Similar behaviour can also be produced using manually repeated JSON calls rather than LDP addon.
  • Happens on both SQLite and MySQL (MariaDB) databases
  • Is platform independant - happens on RPi (LibreElec), and Windows
I believe we get similar behaviour with a moderate sized music library too, but it is harder to spot because less memory is being used so the steps are smaller. In testing memory use has gone from 55MB to 100MB in 4 hours with the odd blip back to a lower level.

It looks like the issue is related to the (almost) simultaneous JSON calls requesting data, but having dug into every aspect of that processing I am beginning to wonder if it is just repeated memory use of any kind.

More soak testing to do, but if this pattern gives any ideas please mention them.
Reply
Yeap, here is a moderate music library (11k songs, 850 albums) doing a similar pattern.

Image
Reply
Nice work! Sadly I haven't had time to do further testing this weekend.
Sadly I haven't been able to reproduce the crashes with the debug enabled version Milhouse provided. Maybe that would have enlightened a little better on where to look at...
Lastly I'm glad, it's not my huge ass library being the core :-)
Reply
No its not your big ass library Smile

Of course my initial testing was all in the first few calls, where the memory behaves, no wonder I failed to repeat your experience when I first tried. Now I know I need to leave things running and repeating calls until it kicks off.

My feeling now is that it is some kind of fragmentation/general memory management issue than some leaky code, revealed by processing that uses memory (like overlapping data requests messing with a lot of data) but not anything to do with that actual processing itself.

Ah ha, just caught the first step change happening with only a single JSON request for data, but I need to run more and see what happens a few hours on.
Reply
@DaveBlake

On my Nvidia Shield TV system (753 Artists, 3k+ Albums, 32,000+ Songs - Is that a big ass library?), I haven't experienced any issues adding the source, or updating Artists. Shouldn't I be seeing some kind of issue?

I see you tested on Windows and an RPi, but what about Android?
Reply
You could do some testing with the 3 JSON calls or install Xonfluence, which installs LDP alongside, and log memory usage.

Dave, what about some testing with valgrind? Sadly, I've never done that myself. :-(
Reply
So how would I test 3 JSON calls? Is that something Kodi 17 does by default?

If the issue shows up with Xonfluence, wouldn't that be a skin issue and not necessarily a Kodi issue? As I said, I have a large library, and am using the Skin Aeon MQ7, and haven't had any of the issues you've posted.

However, I don't use a separate Mysql server, so maybe it's the interaction between Kodi and Mysql that is causing your issue.

Just throwing stuff out there, seeing if we can narrow down the issue for you.
Reply
Welcome to the "party" @Powerhouse.

Possibly time for an interim summary. Just to complicate matters there are 3 issues pursued in this thread:
a) Scanning source into library taking a very, very long time.
That was down to a deadly mix of a very, very large collection of music files (131k of songs, 8k of albums), MySQL (that version) being slower than MariaDB, and the Library Data Provider addon (installed with Xonfluence skin amongst others) invoking some very heavy queries in the middle of the process.

b) That the MySQLTuner reports join not using indexes.
Fact is MySQL is defficient, and does not apply indexes on correlated subqueries. MySQL is also 7 times slower than SQLite even if the SQL is optimised to meet MySQL limitations etc.

c) That with the Library Data Provider addon running the RPi runs out of memory and crashes if left on overnight.
This is the apparent memory leak that we are still chasing the cause of.

@Powerhouse
Roman's library is 3 to 4 times what you have and you are on SQLite, so I would not expect you to have issues a) or b). I don't know if Aeon MQ7 uses the LDP, but for interest you could check to see is this addon is installed and enabled - from My Addons list all and search for "Library Data Provider" (I think is sits under the "video addon" category even though it does more than video library stuff).

Initial difficulties we had reproducing c) on other platforms made me think it was platform and/or db specific, but I am now sure that it isn't either. That Xonfluence installed the LDP is neither here nor there, but it's presence did clutter the log. Nor is it only LDP addon releated, separate JSON commands (issued from a web browser) can cause the same issue.

Not tried it on Android, but I see no reason why that would be free of the effect if it happens on both RPi and Windows 10. So I would save your testing efforts for another day or issue, but thanks for offering.

On the other hand if you are really curious then install the LDP addon from the repo and enable it (you don't have to change skin, probably better if you don't). Leave Kodi on for many hours not doing anything but idling, and check the memory useage over time (not sure how on Android).

The issue only becomes obvious if you have a big library on memory limited hardware, but I think this is just revealing inherent behaviour rather than the cause. Chances are the cause is a tiny memory leak (one small var not being cleared) in the dynamic memory allocation that is causing fragmentation and dramatic effects. Spotting such small leaks is notoriously hard.

Never used valgrind, but I had just been reading about it. Meanwhile I have been doing some more soak tests to see how broad the scope of this is.
Reply
Dave, you've forgotten about library cleaning never ending, which is solved already. :-)

No, it's definitely not skin related! It just occurred with Xonfluence first time. I'm able to also reproduce on Linux. Haven't checked against Amlogic by now thoroughly..

@Powerhouse
It wasn't my decision to stumble upon that mem leak thing and occupying Dave on that! We've discussed already that Mysql issues are not further regarded. It's late already for 17.1, so it needs priorities, which is not my decision where to put...
It may even be, I'm not everyones darling in here, but that happens to everyone elsewhere, too...

Else I would say that some outcome is there already with influence on music, no matter what db size.
Reply
Oh yes, musn't forget the things I have fixed. Smile

It is possible that the endless increase in memory use is just a result of normal application behaviour and the way malloc in C++ works, not some coding error. Objects of various sizes and persistence (some are transient, allocated and freed regularly, while others persist for extended periods of time) are all drawn from the same memory heap. Those with a longer lifetime poke holes in the heap.

And then the 12th time Kodi tries to find a coherent space for processing 140k songs it can't find one unless it grabs more memory and makes the heap bigger. And so on...

Does Kodi simply need a different memory management strategy, say something that groups things of similar lifetimes into separate heaps?
Reply
Like to have a valgrind log?
http://sprunge.us/abbd
Reply
Thanks, although I'm not really sure what to make of it. At what point did you take the log, what had you done?

The more I think about it the more obvious it is to me that any small memory leaks (although there were none in the code I looked at) will have just the same effect as correct allocation/release of small pieces of memory that are more persistent than the bulk music data used in music library processing. More persistent or "infinitely persistent", the impact is the same.

It would be great to pin down any leaks, improve the code quality etc., but even without any leaks, anywhere in the code, we would get the same results. The mixed lifetime of memory allocation and sizes is inevitably going to result in a gradual increase in memory needed.

This means not running repeated large memory processing activities e.g. simultaneous JSON calls both requesting lots of records from a huge library, on a platform with limited memory e.g. RPi. Either no LDP or Skin.Helper addons, or adjust them to space out any data intensive requests. Changing any queries to apply sorting and limits at the db, rather than in local memory would also help.

Is this sounding reasonable? It is more understanding than fix I know, but sometimes that is as valuable.
Reply
Just running kodi with Xonfluence (as it shows it best and fastest from my experience) and do some JSON (#2} calls in a fast row. Problem is, kodi gets real slow and unresponsive (was testing on laptop). Quit from within kodi did nothing and in the end I killed kodi (not forced, just a kill to valgrind) and noticed, the JSON queries had not finished.
It's my first valgrind log tbh, so I'm not sure how to read it. I would really like to have valgrind running on pi.

I can follow your thoughts, but what would it mean if running long term?

Sort and limit at the db is where it belongs. Have you've got 2 example queries to test for speed comparison? A test against yajl and rapidjson each with both queries would be interesting. For speed and memory behavior. Have you tested against sqlite?

Btw, rpi is 32bit. All other devices here are 64bit (although amlogic is 64bit kernel with 32bit userspace)...?
Reply
Only thing I took was that the valgrind issues all seemed to have something to do with Python calls. At that point I am out of my depth.

(2017-03-15, 23:32)Uatschitchun Wrote: I can follow your thoughts, but what would it mean if running long term?
If you have a huge library accessed by Kodi on a platform with limited memory, then running polling scripts like Library Data Provider making requests that take big chunks of memory to process will mean that you need to restart fairly regularly.

Options:
a) don't use LDP or skins that require it.
or
b) Power off regularly
or
c) Run on a platform with sufficient memory compatible with the use (skin, addons, library size) you want to make of it.

You know that if you stop LDP and skin.helper the memory problem (implicit behaviour) goes away.

That kind of script, making regular calls that need lots of memory just exasperate the fundamental weaknesses in C++ memory management. If the addon was redesigned to space out the JSON requests, or the requests made more memory efficient then again it would ease the situartion.

Are there also small leaks in the Python interface, or texture handling etc. that makes things worse? Well probably yes, but I'm not sure I have the skills or energy to find them, and I doubt that the issue would vanish even if I did. I have tested the JSON to music library bits exhaustively, and they allocate and release correctly. So, for now at least, I want to move on to areas I can improve.

Quote:Sort and limit at the db is where it belongs. Have you've got 2 example queries to test for speed comparison? A test against yajl and rapidjson each with both queries would be interesting. For speed and memory behavior. Have you tested against sqlite?
Yes it does, but it is a fundamental change to Kodi to start sorting at the db. All such sorting has to duplicate that done locally by Kodi when the user switches sort method on a screen full of data. That has bugs in it too (for multiple field sorting). So dealing with all of that is something I am working on for v18.

An example of sort and limit at db being faster (and more memory efficient) is the change to how JSON request for 20 random unplayed songs works in 17.1 RC1 compared to 17.0. If you remember JSON #1 stopped showing the memory issue in your tests.

I have done some time tests on SQLite of sort at db rather than in internal memory, and was saving 15% to 25% overall on the songs node. I need to do more but I have been a bit distracted by some guy chasing a memory leak!

Yet to get a build using rapidjson to test, but I hope to get to it soon. If you get to do that with your big ass library then please report (but work with Estuary or Confluence skin and no addons!)
Reply
  • 1
  • 5
  • 6
  • 7
  • 8
  • 9(current)

Logout Mark Read Team Forum Stats Members Help
Problem scanning huge source (mem leak)?0