• 1
  • 2
  • 3
  • 4(current)
  • 5
  • 6
  • 9
Problem scanning huge source (mem leak)?
#46
What I'm able to tell is that the memory leak issue until now only appeared with LE on RPi3 when using skin.widget or ldp which come with Xonfluence. Neither LE on Amlogic (s905x) or kodi on ubuntu have that issue. I haven't tested with latest 8.0.0 LE on RPi3. Amlogic and Ubuntu have had no problem with kodi 17.0 or 17.1

The library cleaning issue is still present and with 17.1 there's library cleaning after update (cleanonupdate?). Cleaning the library lead to delete query running forever
Reply
#47
(2017-02-26, 12:36)Rusendusen Wrote: What I'm able to tell is that the memory leak issue until now only appeared with LE on RPi3 when using skin.widget or ldp which come with Xonfluence. Neither LE on Amlogic (s905x) or kodi on ubuntu have that issue. I haven't tested with latest 8.0.0 LE on RPi3. Amlogic and Ubuntu have had no problem with kodi 17.0 or 17.1
That is very helpful to know. Smile

When 17.1 gets into LE (should be soonish) it would also be nice to know if the LDP/skin.widget related memory leak is fixed.

But before that you could try if an environment setting makes any difference (if it is a fragmentation issue with repeated fetching of large amounts of data not a true leak this will show it). Set MALLOC_MMAP_THRESHOLD_=32768

[Not sure how to do that on the Pi, but you may know more than me]

Quote:The library cleaning issue is still present and with 17.1 there's library cleaning after update (cleanonupdate?). Cleaning the library lead to delete query running forever
That is on Amlogic and Ubuntu I guess, with MariaDB.
OK, I'll try to get somewhere with that, got stuck with other issues (release time rush!!)
Reply
#48
I'll check regarding malloc

Cleaning issue is independent from the used platform or DB (my or maria).

Another one would be to see if we can optimize DB access in general with indexed searches, etc...
Reply
#49
(2017-02-26, 13:06)Rusendusen Wrote: Another one would be to see if we can optimize DB access in general with indexed searches, etc...
The music schema does have indexes, but you think there is a problem with them? Not so sure how well video does...

Done a recent analysis of slow songs node, a mere 5% of the time goes on the actual DB query, 20% on loading results set into structures in local memory, 23% on sorting in local memory (would be so much quicker at the DB), 18% on fetching icons and 32% saving the node to cache (which gets cleared if you do anything more than look at player OSD).

People have looked at indexes before, but frankly that is barking up the wrong tree. Smile

EDIT
Oh and 7% messing with cuesheets even when there are none (even longer than the DB query).
Reply
#50
Optimizing DB access was not because of slowness but because of MySQLTuner always suggested a higher join_buffer_ize or indexed searches ...
Reply
#51
Clean DB issues:
1) It is on the main thread, so you can't do anything else etc.

2) It is slow, so "clean" is not the best way to empty the DB (actually removing the DB from the server would be quicker, let Kodi make a new one)

But quite so slow....?
(2017-02-21, 20:39)Rusendusen Wrote: During the process a temporary table tmp_delartists gets created and from this point on it gets stuck.

Quote:26089 Query CREATE TEMPORARY TABLE tmp_delartists (idArtist integer) CHARACTER SET utf8 COLLATE utf
_general_ci
26089 Query INSERT INTO tmp_delartists select idArtist from song_artist
170221 11:11:10 26089 Query INSERT INTO tmp_delartists select idArtist from album_artist
170221 11:11:11 26089 Query INSERT INTO tmp_delartists VALUES(1)
26089 Query delete from artist where idArtist not in (select idArtist from tmp_delartists)
170221 11:11:14 26089 Query DELETE FROM album_artist WHERE album_artist.idArtist = old.idArtist
26089 Query DELETE FROM song_artist WHERE song_artist.idArtist = old.idArtist
26089 Query DELETE FROM discography WHERE discography.idArtist = old.idArtist
26089 Query DELETE FROM art WHERE media_id=old.idArtist AND media_type='artist'
The bold line indicates the, from my view, beginning of the problem. After that query there are 4 queries following for each artist, I guess: DELETE FROM album_artist, song_artist, discography, and art. These individual sets of 4 queries for each artist takes from a minute up to multiple hours each :-(
Wish you had shown more of this server log, or did it stop there? Testing was MySQL not MariaDB?

The slowness of deleting from album_artist, song_artist, discography and art for each artist is confounding me. These cascade from the triggers fired after artist is deleted, so I expect them to happen, but they should be super fast because the artist cleanup happens after all the song_artist and album_artist tables have already been cleared. There is even a code comment "Create temp table to avoid 1442 trigger hell on mysql".

With a lot of artists "where idArtist not in (select idArtist from tmp_delartists)" could be slow, but not the triggered deletes.

At a loss for now, but wanting you to know I have thought about it.
Reply
#52
(2017-02-27, 08:54)Rusendusen Wrote: Optimizing DB access was not because of slowness but because of MySQLTuner always suggested a higher join_buffer_ize or indexed searches ...
This is what mysqltuner says:
Code:
[!!] Joins performed without indexes: 3239
Adjust your join queries to always utilize indexes
join_buffer_size (> 32.0M, or always use indexes with joins)
Reply
#53
A complete log from a cleanup is coming!

Here's a vid showing the "mem leak". I have environment variable "MALLOC_MMAP_THRESHOLD_=32768" set with
Code:
$ export MALLOC_MMAP_THRESHOLD_=32768
and check with
Code:
$ echo $MALLOC_MMAP_THRESHOLD_
32768

https://www.dropbox.com/s/irnys0nv7t2mmu...1.mp4?dl=0
----

And here's the log from cleanup
http://sprunge.us/eTZg
Kodi was stuck this morning and process still running on DB
Code:
46768 | kodi | 192.168.5.7:52982 | MyMusic60 | Query   | 35054 | Sending data | delete from artist where idArtist not in (select idArtist from tmp_delartists)
Reply
#54
I have to add for the cleaning part, that there wasn't any artists or folders (re) moved. Last time I scanned source and did some rearrangement in my folder structure, so the cleanup was a 'real' cleanup. The cascades came within seconds up to multiple hours in between each.
Reply
#55
On the mem leak, this is the last entry in mysql.log:
Code:
363 Connect   [email protected] as anonymous on
                  363 Query     SET NAMES utf8
                  363 Query     SET SESSION sql_mode = (SELECT REPLACE(@@SESSION.sql_mode,'O
LY_FULL_GROUP_BY',''))
                  363 Query     SELECT @@SESSION.optimizer_switch
                  363 Query     SET SESSION optimizer_switch = 'derived_merge=off'
                  363 Query     show databases like 'MyMusic60'
                  363 Query     show tables
                  363 Init DB   MyMusic60
                  363 Query     show databases like 'MyMusic60'
                  363 Query     show tables
                  363 Query     SELECT COUNT(1) FROM songview  WHERE ((CAST(songview.iTimesP
ayed as DECIMAL(5,1)) < 1))
                  363 Query     SELECT songview.*, songartistview.* FROM songview JOIN songa
tistview ON songartistview.idsong = songview.idsong  WHERE ((CAST(songview.iTimesPlayed as D
CIMAL(5,1)) < 1)) ORDER BY songartistview.idsong, songartistview.idRole, songartistview.iOrd
r
And this is kodi.log
http://sprunge.us/jGTU
Reply
#56
Cleanup issues

Thanks for the log, it does help me to see details. Of course I want more info.

(2017-02-27, 23:25)Rusendusen Wrote: Kodi was stuck this morning and process still running on DB
Code:
46768 | kodi | 192.168.5.7:52982 | MyMusic60 | Query   | 35054 | Sending data | delete from artist where idArtist not in (select idArtist from tmp_delartists)

The kodi.log shows it doing the hard part - song paths, slowly but completed - yet as you said hangs on artists.
I can only wonder if the LDP/skin.widgets polling queries somewhow interrupted the cleaning. Did you have the cleaning problem before installing Xonfluence?

I assume that the temp table has gone now, but that the artists haven't. I wonder how many it was trying to delete? Some SQL for you to use to find out.
Code:
CREATE TEMPORARY TABLE tmp_delartists (idArtist integer) CHARACTER SET utf8 COLLATE utf8_general_ci;
INSERT INTO tmp_delartists select idArtist from song_artist;
INSERT INTO tmp_delartists select idArtist from album_artist;
INSERT INTO tmp_delartists VALUES(1);
SELECT COUNT(1) FROM artist WHERE idArtist NOT IN  (SELECT idArtist FROM tmp_delartists);
I'll make a SQL expert of you eventually Smile

(2017-02-28, 21:28)Rusendusen Wrote: I have to add for the cleaning part, that there wasn't any artists or folders (re) moved. Last time I scanned source and did some rearrangement in my folder structure, so the cleanup was a 'real' cleanup. The cascades came within seconds up to multiple hours in between each.

Just to confirm by "cascades" do you mean that the MySQL log showed repeats of the set
Code:
Query DELETE FROM album_artist WHERE album_artist.idArtist = old.idArtist
Query DELETE FROM song_artist WHERE song_artist.idArtist = old.idArtist
Query DELETE FROM discography WHERE discography.idArtist = old.idArtist
Query DELETE FROM art WHERE media_id=old.idArtist AND media_type='artist'
with hours of "silence" in between each repeat?
Reply
#57
Memory Leak

To summerise:
  • It happens on RPi running LE 8.0 (hence Kodi 17.0)
  • It does not happen on Linux [is that correct?]
  • Lowering MALLOC_MMAP_THRESHOLD made no difference - so isn't the fragmentation issue
  • It only happens when Library Data Provider or Skin.widgets addon is running e.g. Xonfluence skin is installed

Now I may have lost the plot so please confirm these as accurate.
Reply
#58
(2017-03-01, 08:25)DaveBlake Wrote: Cleanup issues

Thanks for the log, it does help me to see details. Of course I want more info.

(2017-02-27, 23:25)Rusendusen Wrote: Kodi was stuck this morning and process still running on DB
Code:
46768 | kodi | 192.168.5.7:52982 | MyMusic60 | Query   | 35054 | Sending data | delete from artist where idArtist not in (select idArtist from tmp_delartists)

The kodi.log shows it doing the hard part - song paths, slowly but completed - yet as you said hangs on artists.
I can only wonder if the LDP/skin.widgets polling queries somewhow interrupted the cleaning. Did you have the cleaning problem before installing Xonfluence?

I assume that the temp table has gone now, but that the artists haven't. I wonder how many it was trying to delete? Some SQL for you to use to find out.
Code:
CREATE TEMPORARY TABLE tmp_delartists (idArtist integer) CHARACTER SET utf8 COLLATE utf8_general_ci;
INSERT INTO tmp_delartists select idArtist from song_artist;
INSERT INTO tmp_delartists select idArtist from album_artist;
INSERT INTO tmp_delartists VALUES(1);
SELECT COUNT(1) FROM artist WHERE idArtist NOT IN  (SELECT idArtist FROM tmp_delartists);
I'll make a SQL expert of you eventually Smile

Copy & paste doesn't make an expert, but I'm learning :-)
Code:
MariaDB [MyMusic60]> CREATE TEMPORARY TABLE tmp_delartists (idArtist integer) CHARACTER SET utf8 COLLATE utf8_general_c;
Query OK, 0 rows affected (0.26 sec)

MariaDB [MyMusic60]> INSERT INTO tmp_delartists select idArtist from song_artist;
Query OK, 362217 rows affected (17.83 sec)
Records: 362217  Duplicates: 0  Warnings: 0

MariaDB [MyMusic60]> INSERT INTO tmp_delartists select idArtist from album_artist;
Query OK, 8344 rows affected (0.35 sec)
Records: 8344  Duplicates: 0  Warnings: 0

MariaDB [MyMusic60]> INSERT INTO tmp_delartists VALUES(1);
Query OK, 1 row affected (0.04 sec)

MariaDB [MyMusic60]> SELECT COUNT(1) FROM artist WHERE idArtist NOT IN  (SELECT idArtist FROM tmp_delartists);
+----------+
| COUNT(1) |
+----------+
|        0 |
+----------+
1 row in set (3.64 sec)

Quote:
(2017-02-28, 21:28)Rusendusen Wrote: I have to add for the cleaning part, that there wasn't any artists or folders (re) moved. Last time I scanned source and did some rearrangement in my folder structure, so the cleanup was a 'real' cleanup. The cascades came within seconds up to multiple hours in between each.

Just to confirm by "cascades" do you mean that the MySQL log showed repeats of the set
Code:
Query DELETE FROM album_artist WHERE album_artist.idArtist = old.idArtist
Query DELETE FROM song_artist WHERE song_artist.idArtist = old.idArtist
Query DELETE FROM discography WHERE discography.idArtist = old.idArtist
Query DELETE FROM art WHERE media_id=old.idArtist AND media_type='artist'
with hours of "silence" in between each repeat?
Yes.
As the above result was 0, I could move a folder (or rename it, touch it, whatever), update DB and do a cleanup with something to clean and provide a new kodi.log and mysql.log with Estuary and no ldp & skin.helper
Reply
#59
(2017-03-01, 08:38)DaveBlake Wrote: Memory Leak

To summerise:
  • It happens on RPi running LE 8.0 (hence Kodi 17.0)
  • It does not happen on Linux [is that correct?]
  • Lowering MALLOC_MMAP_THRESHOLD made no difference - so isn't the fragmentation issue
  • It only happens when Library Data Provider or Skin.widgets addon is running e.g. Xonfluence skin is installed

Now I may have lost the plot so please confirm these as accurate.
Exactly!
It happens on RPi only since Krypton. No problem with Jarvis.
No, it doesn't happen on Ubuntu with kodi from ppa (17.1rc1) and not on Amlogic s905x with LE from kszaq (17.1 82eaae9) and never happened with earlier Krypton versions on linux or amlogic
Lowering MALLOC_MMAP_THRESHOLD is done within LE for 64bit builds (see: https://github.com/LibreELEC/LibreELEC.t...odi-config) but LE RPi builds are 32bit. As that kodi-config is read-only I tried setting MALLOC_MMAP_THRESHOLD as environment variable (don't know if that is the absolute correct way to do it, will ask in LE forums to be sure)
Yes, reproduceable with Xonfluence, ldp & skin.widget. Even switching back to Estuary doesn't solve it, but only removing ldp and skin.widget which needs Xonfluence to be removed, too. It definitely is in conjunction to a decent DB size, as that was when I noticed it at first (scanning source with RPi and Xonfluence).
Maybe I should also ask in Xonfluence forum? Will better first try to find what's causing it and I can't imagine it being a skin issue
I'll try Estuary and install ldp or skin.widget alone and see
Reply
#60
Memory leak:
(2017-03-01, 14:40)Rusendusen Wrote: Maybe I should also ask in Xonfluence forum? Will better first try to find what's causing it and I can't imagine it being a skin issue
I'll try Estuary and install ldp or skin.widget alone and see
No point asking in skins. It is related to what the LPD addon causes to happen, but it will be interesting to see how it behaves when LE gets up to 17.1, I have changed the query it makes. I guess it could be the addon itself, but that is less likely.

So next step with that is await update to LE and retest then.
Reply
  • 1
  • 2
  • 3
  • 4(current)
  • 5
  • 6
  • 9

Logout Mark Read Team Forum Stats Members Help
Problem scanning huge source (mem leak)?0