Kodi Community Forum
v18 LibreELEC Testbuilds for RaspberryPi (Kodi 18.0) - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: General Support (https://forum.kodi.tv/forumdisplay.php?fid=111)
+---- Forum: Raspberry Pi (https://forum.kodi.tv/forumdisplay.php?fid=166)
+---- Thread: v18 LibreELEC Testbuilds for RaspberryPi (Kodi 18.0) (/showthread.php?tid=298461)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495


RE: LibreELEC Testbuilds for RaspberryPi (Kodi 18.0) - Milhouse - 2018-01-26

Last throw of the dice for OSD-troubled users, build #0125d: RPi2

This is the same as #0125 (back to 4.14.15) but does not include PR13433 and PR13435 (as the latter now has a dependency on PR13433).


RE: LibreELEC Testbuilds for RaspberryPi (Kodi 18.0) - HiassofT - 2018-01-26

Build 0125d is fine, action is OSD again:

Code:
19:29:28.477 T:1941877760   DEBUG: LIRC: Update - NEW at 25460:160 0 KEY_OK devinput (KEY_OK)
19:29:28.477 T:1941877760   DEBUG: HandleKey: 11 (0x0b, obc244) pressed, action is OSD
19:29:28.478 T:1941877760   DEBUG: ------ Window Init (VideoOSD.xml) ------

Thanks a lot for the testbuilds!

so long,

Hias


RE: LibreELEC Testbuilds for RaspberryPi (Kodi 18.0) - bleep42 - 2018-01-26

(2018-01-26, 14:35)popcornmix Wrote:
(2018-01-26, 02:43)Milhouse Wrote: To me this looks more like #0805 than #0804, but it's hard to say if #0806 is worse than #0805 - I'd probably say they're much the same, and that the problem starts with - or is made worse (a small leak becoming a much bigger leak) by - #0805.

The only major change in #0806 is the gcc/glibc bump but hopefully they're not involved, as that would be a major task to track down...and wouldn't really explain why this affects only HEVC playback and not any other video codec. 

Does the leak seem to be once per play? i.e. if you play/stop after a few seconds repeatedly does memory exhaust much faster than playing a single file that lasts an hour? (I'd guess it probably would)

How hard would it be to build head of tree with older gcc/glibc? Does that crash with out-of-memory after repeated plays?
(I suspect that #0805 is really the culprit, but the evidence is currently a bit unclear).
BTW John is on holiday this week so we won't get an immediate fix, but would be nice to be sure the bug is in the hevc code when he returns.  
Hi Popcornmix,
Looks like Milhouse beat me to it, so for what it's worth, here are the results from #0806, I've only got top to give me the memory usage accurately.
screen snaps in the drop box folder.
https://www.dropbox.com/sh/fc4chc5irjnalbu/AACKNVFIUHboS0oRPI2is3qUa?dl=0
B4Start.png  is before I do anything, after a reboot.
I then played the video in a repeat.
Free memory at start of first play was 263MB by end 204MB
Free memory at start of second play was 194MB by end 201MB
Free memory at start of third play was 157MB by end 113MB
Free memory at start of fourth play was 94MB by end 49MB
Free memory at start of fifth play was 86MB by end 47MB
End5thplay265.png is recorded just before the end of the 5th play of Elysium, so same as above at 47MB.
I then rebooted so everything was back to as it was in B4Start.png
I then played the first 5 seconds of the video and stopped, 1x5sec265.png
then again 2x5sec265.png, then twice more, 5x5sec265.png then another 5 times 10x5sec265.png
as you can see, by the end of the 10th play there is 90MB free, not good, but more than playing the whole video 10 times, which would not be possible, as it locks up after 6 or 7 plays.
Regards,
Kevin


LibreELEC Testbuilds for RaspberryPi (Kodi 18.0) - ksooo - 2018-01-26

(2018-01-26, 21:30)HiassofT Wrote: Build 0125d is fine, action is OSD again:

Code:
19:29:28.477 T:1941877760   DEBUG: LIRC: Update - NEW at 25460:160 0 KEY_OK devinput (KEY_OK)
19:29:28.477 T:1941877760   DEBUG: HandleKey: 11 (0x0b, obc244) pressed, action is OSD
19:29:28.478 T:1941877760   DEBUG: ------ Window Init (VideoOSD.xml) ------

Thanks a lot for the testbuilds!

so long,

Hias

Could you please leave a comment in the affected PRs to give @xhaggi a headsup?


RE: LibreELEC Testbuilds for RaspberryPi (Kodi 18.0) - Milhouse - 2018-01-26

(2018-01-26, 21:35)ksooo Wrote: Could you please leave a comment in the affected PRs to give @xhaggi a headsup?

Done: github

And many thanks @HiassofT!


RE: LibreELEC Testbuilds for RaspberryPi (Kodi 18.0) - Milhouse - 2018-01-26

(2018-01-26, 21:31)bleep42 Wrote: Looks like Milhouse beat me to it, so for what it's worth, here are the results from #0806, I've only got top to give me the memory usage accurately.

bcmstat.sh is included as standard in my RPi/RPi2 builds. texturecache.py is also included too (including Generic). Just in case anyone finds them useful.

I'm not saying bcmstat.sh -DAZ memory monitoring is more accurate than any other tool, but it does track GPU/RAM consumption for the whole system (and if pretty much all you're running is kodi then it makes life easier). If all you see are red allocations (and few if any green free's) then there's a good chance there's a leak.


RE: LibreELEC Testbuilds for RaspberryPi (Kodi 18.0) - bleep42 - 2018-01-26

Ok thanks Milhouse, I'll give that a go in future. :-)
Regards,
Kevin.


RE: LibreELEC Testbuilds for RaspberryPi (Kodi 18.0) - Milhouse - 2018-01-26

I've rebuilt LE master/Kodi master with gcc-6.2.0 and glibc-2.25 and the results initially look similar to #0804, with reduced leakage compared with #0805.

The following image shows the end of the 2nd and 3rd play of Elysium:
Image
where there is a deficit of 47MB after the second play, which falls to 23MB after the 3rd play.

After 5 full plays of Elysium the deficit is still only 64MB
However after a 6th play, the deficit increased to 108MB.
But after a 7th play, it reduced to 78MB.

This is more like #0804 than #0805 - there is a small leak, but it's not as significant as I saw with #0805.

But it's weird, as the gcc/glibc changes are in #0806, and #0805 showed a significant deficit after each play (even though it also used gcc-6.2.0), so I'm not sure why switching back to gcc-6.2.0/glibc-2.25 would restore the significant losses seen with #0805. Or it's another red herring, and the issue is just with #0805. That said, I have just tested a master build which is using gcc-7.3.0 and the deficit increased rapidly - first play 94MB, second play 181MB - so perhaps there is a gcc-7.x bug that is causing this increased level of leakage.

I can upload these gcc-6.2.0/gcc-7.3.0 test builds if this will be useful.


RE: LibreELEC Testbuilds for RaspberryPi (Kodi 18.0) - popcornmix - 2018-01-26

This is feeling a bit like the music database memory leak we had before in kodi (worked around with MALLOC_MMAP_THRESHOLD_).
It may not be an actual leak, but glibc's malloc heap can get fragmented with certain patterns of allocs/frees.

It feels a bit like the hevc update changed the sizes of some buffers that made things a little worse.
An updated glibc made things a lot worse.

Are you able to run with smaller settings of MALLOC_MMAP_THRESHOLD_ ?
Is LE currently using MALLOC_MMAP_THRESHOLD_=131072? (That is value kodi chose).

Can you try with, say, MALLOC_MMAP_THRESHOLD_=32768


RE: LibreELEC Testbuilds for RaspberryPi (Kodi 18.0) - HiassofT - 2018-01-26

(2018-01-26, 23:32)popcornmix Wrote: Is LE currently using MALLOC_MMAP_THRESHOLD_=131072? (That is value kodi chose).
Hmmm, interesting. I don't see this defined in kodi's environment:
Code:
xbmc:~ # tr '\0' '\n' < /proc/`pidof kodi.bin`/environ
HOSTNAME=xbmc
LD_LIBRARY_PATH=/usr/lib:/storage/.kodi/addons/script.common.plugin.cache/lib:/storage/.kodi/addons/script.module.certifi/lib:/storage/.kodi/addons/script.module.chardet/lib:/storage/.kodi/addons/script.module.idna/lib:/storage/.kodi/addons/script.module.parsedom/lib:/storage/.kodi/addons/script.module.requests/lib:/storage/.kodi/addons/script.module.routing/lib:/storage/.kodi/addons/script.module.simplejson/lib:/storage/.kodi/addons/script.module.urllib3/lib:/storage/.kodi/addons/virtual.system-tools/lib:/usr/lib/pulseaudio
SHLVL=1
HOME=/storage
PS1=\[\e[1;32m\]\h\[\e[1;32m\]:\[\e[1;34m\]\w \[\e[0m\]\$
SDL_MOUSE_RELATIVE=0
WAYLAND_DISPLAY=wayland-0
KODI_HOME=/usr/share/kodi/
JOURNAL_STREAM=8:12875
TERM=linux
KODI_ARGS=--lircdev /run/lirc/lircd
INVOCATION_ID=5e96265ed6a5477f8255e2e40cef8140
PATH=/usr/bin:/usr/sbin:/storage/.kodi/addons/service.hyperion/bin:/storage/.kodi/addons/virtual.system-tools/bin
DISPLAY=:0.0
SYSTEMD_COLORS=0
__GL_YIELD=USLEEP
KODI_TEMP=/storage/.kodi/temp
PWD=/
EDITOR=nano

__GL_YIELD, DISPLAY etc are set in /usr/lib/systemd/system/kodi.service but MALLOC_MMAP_THRESHOLD_ is missing.

so long,

Hias


RE: LibreELEC Testbuilds for RaspberryPi (Kodi 18.0) - Milhouse - 2018-01-26

(2018-01-26, 23:32)popcornmix Wrote: This is feeling a bit like the music database memory leak we had before in kodi (worked around with MALLOC_MMAP_THRESHOLD_).
It may not be an actual leak, but glibc's malloc heap can get fragmented with certain patterns of allocs/frees.

It feels a bit like the hevc update changed the sizes of some buffers that made things a little worse.
An updated glibc made things a lot worse.

Are you able to run with smaller settings of MALLOC_MMAP_THRESHOLD_ ?
Is LE currently using MALLOC_MMAP_THRESHOLD_=131072? (That is value kodi chose).

Can you try with, say, MALLOC_MMAP_THRESHOLD_=32768

I don't think we're setting this for arm:
https://github.com/LibreELEC/LibreELEC.tv/blob/master/packages/mediacenter/kodi/scripts/kodi-config#L56-L58

as this fragmentation issue first surfaced on 64-bit systems (PR for change: https://github.com/LibreELEC/LibreELEC.tv/pull/1286)

I'll run some tests with default, large and small values and let you know.


RE: LibreELEC Testbuilds for RaspberryPi (Kodi 18.0) - Milhouse - 2018-01-27

New LibreELEC.tv Leia build #0126: RPi / RPi2
(Supercedes previous build)

SHA256 Checksum: d8440c9c3674682bfb010801b4122b7bd1b9189ce8c0288f062625258ce6bae1 (RPi)
SHA256 Checksum: 4ea7109cabaf813d5d1dda01c2cb1e87d196c06a42da7300bbcad03a5c279ea2 (RPi2)

text:
# uname -a
Linux rpi512 4.14.15 #1 Fri Jan 26 21:28:03 GMT 2018 armv6l GNU/Linux

# vcgencmd version
Jan 25 2018 18:14:16
Copyright © 2012 Broadcom
version 16ab01397c559d80e3239c4d9c453a999a9121b1 (clean) (release)

# lsb_release
LibreELEC (Milhouse): devel-20180126212647-#0126-g4ff4537 [Build #0126]

# Kodi version
(18.0-ALPHA1 Git:69a6725). Platform: Linux ARM 32-bit

Based on tip of LibreELEC.tv master (4ff4537, changelog) and tip of XBMC master (6ad6c22, changelog) with the following modifications: Build Highlights:
  1. lirc: use it only for user-configured remotes
Build Details:
  1. LibreELEC.tv:
    • curl: update to curl-7.58.0 (PR:2438, 1 commit, 1 file changed)
    • bump v4l-utils to 1.14.1 and support MCE etc remotes OOTB on Odroid/WH (PR:2424, 2 commits, 4 files changed)
    • lirc: use it only for user-configured remotes (PR:2341, 3 commits, 14 files changed)
    • Move Odroid_C2 to Amlogic Project (PR:2421, 1 commit, 31 files changed)
    • libamcodec: don't build amadec (PR:2439, 1 commit, 1 file changed)
    • Amlogic: Enable Crazycat, Hauppauge dvb driver addon (PR:2405, 3 commits, 18 files changed)
  2. XBMC:
    • [PVR] Fix trac#17748. (PR:13438, 1 commit, 1 file changed)
    • [gui] confirm keyboard dialog with enter while on edit control (PR:13439, 2 commits, 1 file changed)
    • [windows] d3d11: use smart pointers (ComPtr) instead of raw pointers. (PR:13442, 7 commits, 20 files changed)
    • [xbmc][windows] Fix a bunch of warnings for windows x64 (PR:13366, 1 commit, 14 files changed)
  3. Additional commits/pull requests/changes not yet merged upstream:



RE: LibreELEC Testbuilds for RaspberryPi (Kodi 18.0) - Milhouse - 2018-01-27

LiveTV OSD users - please test build #0126b: RPi2

This includes PR:13444 which hopefully fixes the OK/Select remote control issue.


RE: LibreELEC Testbuilds for RaspberryPi (Kodi 18.0) - Milhouse - 2018-01-27

Thanks @popcornmix for having such a good memory - this definitely does look like another fragmentation issue.

I've tested various values for MALLOC_MMAP_THRESHOLD_ (with #0126 on RPi3, gcc-7.2.0, glibc-2.26, gpu_mem=320MB, arm=704MB) and these are the results:

Image

As can be seen, and probably expected, the smaller the threshold chunk size the less "loss" resulting from fragmentation and more free memory that remains available.

This is a link to the glibc tunables page, which explains MALLOC_MMAP_THRESHOLD_.

I'm not sure how low we want to go with MALLOC_MMAP_THRESHOLD_ - a value of 8192 might be a usable value. I feel that going as low as 1024 might result in an excessive number of chunks for no real advantage. But that's all a bit finger-in-the-air.

Unless there's any better idea I'll set MALLOC_MMAP_THRESHOLD_=8192 for arm in future builds (commit).


RE: LibreELEC Testbuilds for RaspberryPi (Kodi 18.0) - polo_joe - 2018-01-27

Build #0126b can't be found, error 404.