2017-03-13, 20:34
Welcome to the "party" @Powerhouse.
Possibly time for an interim summary. Just to complicate matters there are 3 issues pursued in this thread:
a) Scanning source into library taking a very, very long time.
That was down to a deadly mix of a very, very large collection of music files (131k of songs, 8k of albums), MySQL (that version) being slower than MariaDB, and the Library Data Provider addon (installed with Xonfluence skin amongst others) invoking some very heavy queries in the middle of the process.
b) That the MySQLTuner reports join not using indexes.
Fact is MySQL is defficient, and does not apply indexes on correlated subqueries. MySQL is also 7 times slower than SQLite even if the SQL is optimised to meet MySQL limitations etc.
c) That with the Library Data Provider addon running the RPi runs out of memory and crashes if left on overnight.
This is the apparent memory leak that we are still chasing the cause of.
@Powerhouse
Roman's library is 3 to 4 times what you have and you are on SQLite, so I would not expect you to have issues a) or b). I don't know if Aeon MQ7 uses the LDP, but for interest you could check to see is this addon is installed and enabled - from My Addons list all and search for "Library Data Provider" (I think is sits under the "video addon" category even though it does more than video library stuff).
Initial difficulties we had reproducing c) on other platforms made me think it was platform and/or db specific, but I am now sure that it isn't either. That Xonfluence installed the LDP is neither here nor there, but it's presence did clutter the log. Nor is it only LDP addon releated, separate JSON commands (issued from a web browser) can cause the same issue.
Not tried it on Android, but I see no reason why that would be free of the effect if it happens on both RPi and Windows 10. So I would save your testing efforts for another day or issue, but thanks for offering.
On the other hand if you are really curious then install the LDP addon from the repo and enable it (you don't have to change skin, probably better if you don't). Leave Kodi on for many hours not doing anything but idling, and check the memory useage over time (not sure how on Android).
The issue only becomes obvious if you have a big library on memory limited hardware, but I think this is just revealing inherent behaviour rather than the cause. Chances are the cause is a tiny memory leak (one small var not being cleared) in the dynamic memory allocation that is causing fragmentation and dramatic effects. Spotting such small leaks is notoriously hard.
Never used valgrind, but I had just been reading about it. Meanwhile I have been doing some more soak tests to see how broad the scope of this is.
Possibly time for an interim summary. Just to complicate matters there are 3 issues pursued in this thread:
a) Scanning source into library taking a very, very long time.
That was down to a deadly mix of a very, very large collection of music files (131k of songs, 8k of albums), MySQL (that version) being slower than MariaDB, and the Library Data Provider addon (installed with Xonfluence skin amongst others) invoking some very heavy queries in the middle of the process.
b) That the MySQLTuner reports join not using indexes.
Fact is MySQL is defficient, and does not apply indexes on correlated subqueries. MySQL is also 7 times slower than SQLite even if the SQL is optimised to meet MySQL limitations etc.
c) That with the Library Data Provider addon running the RPi runs out of memory and crashes if left on overnight.
This is the apparent memory leak that we are still chasing the cause of.
@Powerhouse
Roman's library is 3 to 4 times what you have and you are on SQLite, so I would not expect you to have issues a) or b). I don't know if Aeon MQ7 uses the LDP, but for interest you could check to see is this addon is installed and enabled - from My Addons list all and search for "Library Data Provider" (I think is sits under the "video addon" category even though it does more than video library stuff).
Initial difficulties we had reproducing c) on other platforms made me think it was platform and/or db specific, but I am now sure that it isn't either. That Xonfluence installed the LDP is neither here nor there, but it's presence did clutter the log. Nor is it only LDP addon releated, separate JSON commands (issued from a web browser) can cause the same issue.
Not tried it on Android, but I see no reason why that would be free of the effect if it happens on both RPi and Windows 10. So I would save your testing efforts for another day or issue, but thanks for offering.
On the other hand if you are really curious then install the LDP addon from the repo and enable it (you don't have to change skin, probably better if you don't). Leave Kodi on for many hours not doing anything but idling, and check the memory useage over time (not sure how on Android).
The issue only becomes obvious if you have a big library on memory limited hardware, but I think this is just revealing inherent behaviour rather than the cause. Chances are the cause is a tiny memory leak (one small var not being cleared) in the dynamic memory allocation that is causing fragmentation and dramatic effects. Spotting such small leaks is notoriously hard.
Never used valgrind, but I had just been reading about it. Meanwhile I have been doing some more soak tests to see how broad the scope of this is.
