Problem scanning huge source (mem leak)? - Printable Version +- Kodi Community Forum (https://forum.kodi.tv) +-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33) +--- Forum: Music Support (https://forum.kodi.tv/forumdisplay.php?fid=263) +--- Thread: Problem scanning huge source (mem leak)? (/showthread.php?tid=306766) |
RE: Problem scanning huge source (mem leak)? - Uatschitchun - 2017-02-26 What I'm able to tell is that the memory leak issue until now only appeared with LE on RPi3 when using skin.widget or ldp which come with Xonfluence. Neither LE on Amlogic (s905x) or kodi on ubuntu have that issue. I haven't tested with latest 8.0.0 LE on RPi3. Amlogic and Ubuntu have had no problem with kodi 17.0 or 17.1 The library cleaning issue is still present and with 17.1 there's library cleaning after update (cleanonupdate?). Cleaning the library lead to delete query running forever RE: Problem scanning huge source (mem leak)? - DaveBlake - 2017-02-26 (2017-02-26, 12:36)Rusendusen Wrote: What I'm able to tell is that the memory leak issue until now only appeared with LE on RPi3 when using skin.widget or ldp which come with Xonfluence. Neither LE on Amlogic (s905x) or kodi on ubuntu have that issue. I haven't tested with latest 8.0.0 LE on RPi3. Amlogic and Ubuntu have had no problem with kodi 17.0 or 17.1That is very helpful to know. When 17.1 gets into LE (should be soonish) it would also be nice to know if the LDP/skin.widget related memory leak is fixed. But before that you could try if an environment setting makes any difference (if it is a fragmentation issue with repeated fetching of large amounts of data not a true leak this will show it). Set MALLOC_MMAP_THRESHOLD_=32768 [Not sure how to do that on the Pi, but you may know more than me] Quote:The library cleaning issue is still present and with 17.1 there's library cleaning after update (cleanonupdate?). Cleaning the library lead to delete query running foreverThat is on Amlogic and Ubuntu I guess, with MariaDB. OK, I'll try to get somewhere with that, got stuck with other issues (release time rush!!) RE: Problem scanning huge source (mem leak)? - Uatschitchun - 2017-02-26 I'll check regarding malloc Cleaning issue is independent from the used platform or DB (my or maria). Another one would be to see if we can optimize DB access in general with indexed searches, etc... RE: Problem scanning huge source (mem leak)? - DaveBlake - 2017-02-26 (2017-02-26, 13:06)Rusendusen Wrote: Another one would be to see if we can optimize DB access in general with indexed searches, etc...The music schema does have indexes, but you think there is a problem with them? Not so sure how well video does... Done a recent analysis of slow songs node, a mere 5% of the time goes on the actual DB query, 20% on loading results set into structures in local memory, 23% on sorting in local memory (would be so much quicker at the DB), 18% on fetching icons and 32% saving the node to cache (which gets cleared if you do anything more than look at player OSD). People have looked at indexes before, but frankly that is barking up the wrong tree. EDIT Oh and 7% messing with cuesheets even when there are none (even longer than the DB query). RE: Problem scanning huge source (mem leak)? - Uatschitchun - 2017-02-27 Optimizing DB access was not because of slowness but because of MySQLTuner always suggested a higher join_buffer_ize or indexed searches ... RE: Problem scanning huge source (mem leak)? - DaveBlake - 2017-02-27 Clean DB issues: 1) It is on the main thread, so you can't do anything else etc. 2) It is slow, so "clean" is not the best way to empty the DB (actually removing the DB from the server would be quicker, let Kodi make a new one) But quite so slow....? (2017-02-21, 20:39)Rusendusen Wrote: During the process a temporary table tmp_delartists gets created and from this point on it gets stuck.Wish you had shown more of this server log, or did it stop there? Testing was MySQL not MariaDB? The slowness of deleting from album_artist, song_artist, discography and art for each artist is confounding me. These cascade from the triggers fired after artist is deleted, so I expect them to happen, but they should be super fast because the artist cleanup happens after all the song_artist and album_artist tables have already been cleared. There is even a code comment "Create temp table to avoid 1442 trigger hell on mysql". With a lot of artists "where idArtist not in (select idArtist from tmp_delartists)" could be slow, but not the triggered deletes. At a loss for now, but wanting you to know I have thought about it. RE: Problem scanning huge source (mem leak)? - Uatschitchun - 2017-02-27 (2017-02-27, 08:54)Rusendusen Wrote: Optimizing DB access was not because of slowness but because of MySQLTuner always suggested a higher join_buffer_ize or indexed searches ...This is what mysqltuner says: Code: [!!] Joins performed without indexes: 3239 RE: Problem scanning huge source (mem leak)? - Uatschitchun - 2017-02-27 A complete log from a cleanup is coming! Here's a vid showing the "mem leak". I have environment variable "MALLOC_MMAP_THRESHOLD_=32768" set with Code: $ export MALLOC_MMAP_THRESHOLD_=32768 Code: $ echo $MALLOC_MMAP_THRESHOLD_ https://www.dropbox.com/s/irnys0nv7t2mmuz/VID_20170227_221841.mp4?dl=0 ---- And here's the log from cleanup http://sprunge.us/eTZg Kodi was stuck this morning and process still running on DB Code: 46768 | kodi | 192.168.5.7:52982 | MyMusic60 | Query | 35054 | Sending data | delete from artist where idArtist not in (select idArtist from tmp_delartists) RE: Problem scanning huge source (mem leak)? - Uatschitchun - 2017-02-28 I have to add for the cleaning part, that there wasn't any artists or folders (re) moved. Last time I scanned source and did some rearrangement in my folder structure, so the cleanup was a 'real' cleanup. The cascades came within seconds up to multiple hours in between each. RE: Problem scanning huge source (mem leak)? - Uatschitchun - 2017-02-28 On the mem leak, this is the last entry in mysql.log: Code: 363 Connect [email protected] as anonymous on http://sprunge.us/jGTU RE: Problem scanning huge source (mem leak)? - DaveBlake - 2017-03-01 Cleanup issues Thanks for the log, it does help me to see details. Of course I want more info. (2017-02-27, 23:25)Rusendusen Wrote: Kodi was stuck this morning and process still running on DB The kodi.log shows it doing the hard part - song paths, slowly but completed - yet as you said hangs on artists. I can only wonder if the LDP/skin.widgets polling queries somewhow interrupted the cleaning. Did you have the cleaning problem before installing Xonfluence? I assume that the temp table has gone now, but that the artists haven't. I wonder how many it was trying to delete? Some SQL for you to use to find out. Code: CREATE TEMPORARY TABLE tmp_delartists (idArtist integer) CHARACTER SET utf8 COLLATE utf8_general_ci; (2017-02-28, 21:28)Rusendusen Wrote: I have to add for the cleaning part, that there wasn't any artists or folders (re) moved. Last time I scanned source and did some rearrangement in my folder structure, so the cleanup was a 'real' cleanup. The cascades came within seconds up to multiple hours in between each. Just to confirm by "cascades" do you mean that the MySQL log showed repeats of the set Code: Query DELETE FROM album_artist WHERE album_artist.idArtist = old.idArtist RE: Problem scanning huge source (mem leak)? - DaveBlake - 2017-03-01 Memory Leak To summerise:
Now I may have lost the plot so please confirm these as accurate. RE: Problem scanning huge source (mem leak)? - Uatschitchun - 2017-03-01 (2017-03-01, 08:25)DaveBlake Wrote: Cleanup issues Copy & paste doesn't make an expert, but I'm learning :-) Code: MariaDB [MyMusic60]> CREATE TEMPORARY TABLE tmp_delartists (idArtist integer) CHARACTER SET utf8 COLLATE utf8_general_c; Quote:Yes.(2017-02-28, 21:28)Rusendusen Wrote: I have to add for the cleaning part, that there wasn't any artists or folders (re) moved. Last time I scanned source and did some rearrangement in my folder structure, so the cleanup was a 'real' cleanup. The cascades came within seconds up to multiple hours in between each. As the above result was 0, I could move a folder (or rename it, touch it, whatever), update DB and do a cleanup with something to clean and provide a new kodi.log and mysql.log with Estuary and no ldp & skin.helper RE: Problem scanning huge source (mem leak)? - Uatschitchun - 2017-03-01 (2017-03-01, 08:38)DaveBlake Wrote: Memory LeakExactly! It happens on RPi only since Krypton. No problem with Jarvis. No, it doesn't happen on Ubuntu with kodi from ppa (17.1rc1) and not on Amlogic s905x with LE from kszaq (17.1 82eaae9) and never happened with earlier Krypton versions on linux or amlogic Lowering MALLOC_MMAP_THRESHOLD is done within LE for 64bit builds (see: https://github.com/LibreELEC/LibreELEC.tv/blob/master/packages/mediacenter/kodi/scripts/kodi-config) but LE RPi builds are 32bit. As that kodi-config is read-only I tried setting MALLOC_MMAP_THRESHOLD as environment variable (don't know if that is the absolute correct way to do it, will ask in LE forums to be sure) Yes, reproduceable with Xonfluence, ldp & skin.widget. Even switching back to Estuary doesn't solve it, but only removing ldp and skin.widget which needs Xonfluence to be removed, too. It definitely is in conjunction to a decent DB size, as that was when I noticed it at first (scanning source with RPi and Xonfluence). Maybe I should also ask in Xonfluence forum? Will better first try to find what's causing it and I can't imagine it being a skin issue I'll try Estuary and install ldp or skin.widget alone and see RE: Problem scanning huge source (mem leak)? - DaveBlake - 2017-03-01 Memory leak: (2017-03-01, 14:40)Rusendusen Wrote: Maybe I should also ask in Xonfluence forum? Will better first try to find what's causing it and I can't imagine it being a skin issueNo point asking in skins. It is related to what the LPD addon causes to happen, but it will be interesting to see how it behaves when LE gets up to 17.1, I have changed the query it makes. I guess it could be the addon itself, but that is less likely. So next step with that is await update to LE and retest then. |