Building Kodi from source on hyper-threaded systems.
#1
I've been trying to create a "slackbuild" for Kodi-18, so that Slackware users can build and install Kodi from source using Slackware's package management tools. I have mostly succeeded - the whole saga can be found here:

Linuxquestions

but I have run in to one snag. I'm not a programmer, and not experienced with cmake, so although I have a solution, I'm not sure if its ideal - or even why its necessary!

To build a slackware package, it is necessary to build crossguid, flatbuffers and fmt internally, so first of all, I added the following to the build options (thanks to @abga who has been doing the same for Slackware on the Raspberry Pi):
Quote:#Configure build for X11: cmake ../$SRCNAM-$VERSION-$CODNAM -DCMAKE_INSTALL_PREFIX=/usr/local -DENABLE_INTERNAL_CROSSGUID=ON -DENABLE_INTERNAL_FLATBUFFERS=ON -DENABLE_INTERNAL_FMT=ON -DCORE_SYSTEM_NAME=linux

Now this works fine on non-hyper-threaded processors, such as the old AMD Bulldozer in my "workshop" machine. However, when I try and use the same code on my main machine (Intel i7), the build fails with the following (not very helpful) error:
Quote:...... -- Installing: /tmp/PC/kodi-build/build/lib64/cmake/flatbuffers/FlatcTargets-release.cmake gmake[3]: Leaving directory '/tmp/PC/kodi-build/build/flatbuffers/src/flatbuffers-build' cd /tmp/PC/kodi-build/build/flatbuffers/src/flatbuffers-build && /usr/bin/cmake -E touch /tmp/PC/kodi-build/build/flatbuffers/src/flatbuffers-stamp/flatbuffers-install [ 7%] Completed 'flatbuffers' /usr/bin/cmake -E make_directory /tmp/PC/kodi-build/CMakeFiles /usr/bin/cmake -E touch /tmp/PC/kodi-build/CMakeFiles/flatbuffers-complete /usr/bin/cmake -E touch /tmp/PC/kodi-build/build/flatbuffers/src/flatbuffers-stamp/flatbuffers-done gmake[2]: Leaving directory '/tmp/PC/kodi-build' [ 7%] Built target flatbuffers gmake[1]: Leaving directory '/tmp/PC/kodi-build' gmake: *** [Makefile:141: all] Error 2

 The i7 has 4 cores, but with hyper-threading, it looks like 8. Setting -j4 during the make process, instead of allowing the system to auto-detect the number of processors, allows the build to complete without errors. It looks as if hyper-threading is not allowing one step to complete in time during a parallel build.

@abga suggested creating a build log to determine precisely where the build was failing, thus:
Quote:cmake --build . -- VERBOSE=1 -j$(getconf _NPROCESSORS_ONLN) 2>&1 | tee kodi-build.log

but again, when adding this, the build completes perfectly! Comment out the build log, it fails! Re-instate it and it builds correctly! It seems as if creating the build log slows the system just sufficiently for the build to complete successfully.

For the moment, my solution is to set an environment variable for the number of actual processors - as opposed to virtual ones - and use that to set -j :
Quote:NUMJOBS=$(grep "^core id" /proc/cpuinfo | sort -u | wc -l)

then:

cmake --build . -- VERBOSE=1 -j$NUMJOBS

This has worked on everything I've tried so far. However, I'm NOT a programmer, and the Kodi developers may have a better way of solving the problem. I offer it here to draw their attention to the issues.

Regards,

--
Pete
Reply
#2
The error doesn't show where it failed exactly. If you can reproduce the issues with multi build threads then there might be some dependency issue in our build system. Although our Jenkins builds fine with a pretty high thread count., it might be something slackware specific. Please get a full build log where it fails
Reply
#3
I did try and produce a full build log, but when I do, the build succeeds! (See 3/4 way down my post above) Or is there another build log somewhere inside the kodi build?

--
Pete
Reply
#4
subscribed &

@wsnipex 
Is there any way to create a detailed log with cmake, other than what was already tried, piping the console output to a log file? Because, as @pchristy  described, that will influence the compilation (delay it a little) and the build will be successful.

I'm also not really experienced with cmake, but found this:
https://bytefreaks.net/programming-2/mak...ke-verbose
Code:
Option 3 – Add the variable -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON to your cmake command
Will that populate with more verbose info the Kodi CMakeError.log?

At the moment CMakeError.log does only contain some failures related to libpthreads -> pthread_create, which apparently are caused by some buggy code in Kodi:
- Slackware x86
https://www.linuxquestions.org/questions...ost5960203
- Slackware ARM
https://forum.kodi.tv/showthread.php?tid...pid2815657

Relevant info from Slackware 14.2:
Code:
$nm /lib/libpthread.so.0 | grep "pthread_create"
00006a44 t __pthread_create_2_1
00006a44 T pthread_create@@GLIBC_2.4
$ls -al /lib/libpthread.so.0
lrwxrwxrwx 1 root root 18 Apr  6  2018 /lib/libpthread.so.0 -> libpthread-2.23.so*

Additionally, is there any (minimum) version required for cmake? Here, in the building documentation for Linux, the only version requirements are for the toolchain (>= 4.9).
https://github.com/xbmc/xbmc/blob/master...E.Linux.md
In Slackware 14.2 the toolchain version is 5.5.0 and cmake 3.5.2.

@pchristy 
In case there's no other way to log the cmake build, try to start the compilation without redirecting the console output to kodi-build.log and increase the scrollback buffer of the terminal you're using - way easier under X (the text console scrollback buffer is a little bit difficult to tune and to browse). Maybe you can catch the actual error, because your small log snippet is not really helpful.
For text console scrollback buffer (difficult to tune and to browse):
https://stackoverflow.com/questions/8760...en-session
Reply
#5
Not sure if this helps, but here's the cmakeerror.log:

CMakeError.log

And here's the full cmakeoutput.log:

CMakeOutput.log

(Hope I've managed to create the links correctly!)

--
Pete
Reply
#6
@abga : I've not been ignoring your suggestions - I've just been waiting to see if any of the developers came back with any ideas / suggestions / requests!

My solution works for me, but it does suggest there may be something in the cmake files that needs tweaking to make it all work at optimum speed.

--
Pete
Reply
#7
We really need a log, since we don't see any concurrency issues ATM, although it occasionally happened in the past
Reply
#8
(2019-02-14, 23:07)wsnipex Wrote: We really need a log, since we don't see any concurrency issues ATM, although it occasionally happened in the past
 OK, but I need some advice on how to create one! I tried earlier (see my original post) but the very act of creating the log seemed to fix the problem!

Any suggestions gratefully received!

--
Pete
Reply
#9
(2019-02-15, 00:34)pchristy Wrote:
(2019-02-14, 23:07)wsnipex Wrote: We really need a log, since we don't see any concurrency issues ATM, although it occasionally happened in the past
 OK, but I need some advice on how to create one! I tried earlier (see my original post) but the very act of creating the log seemed to fix the problem!

Any suggestions gratefully received!

--
Pete 
I'm having a really hard time believing that piping the output to a log does influence this behavior. Either we have a dependency issue in cmake or we don't. If we have, it should be reproducible, even if it takes a couple of tries.
Reply
#10
(2019-02-15, 10:20)wsnipex Wrote: I'm having a really hard time believing that piping the output to a log does influence this behavior. Either we have a dependency issue in cmake or we don't. If we have, it should be reproducible, even if it takes a couple of tries. 
 I agree! But all I can say is that is what happens. It is repeatable on my machine every time.

I'm assuming that creating the log takes up one thread, reducing the threads available for the build from 8 to 7. I've been pretty tied up for the last couple of days, but I have a bit of time today. I will try using -j7. If that succeeds, it will indicate that the problem only occurs when all of the hyper-threaded (virtual) processors are in use (-j8).

In the meantime, if you have any suggestions for an alternative way of creating the desired log file, I am happy to try that.

--
Pete
Reply
#11
OK, done some more experimenting. Setting -j8 produces the error already described. Trying the build again immediately, without making any changes, and the the build fails with the same error. Set -j7, and the build succeeds. BUT, having once had a successful build, setting -j8 once again, the build succeeds!

If I close the window used for the build and re-open it, su and build again, and the build fails again.

I'm guessing that once I have managed a successful build, something somewhere is being cached, and even though my script deletes all the build directories, this allows the build to succeed on a second attempt. Closing the window must clear the cache.

I think this is a false lead, however.

The basic issue seems to be that using the maximum number of threads available on a hyper-threaded system, will cause a build failure if you attempt to build the "extra" programs internally. Reducing the number of threads by one, and the build succeeds.

For my specific purposes, I can get around the problem by limiting the the number of threads to the number of physical (as opposed to virtual) cores. But is does imply that there is a dependency issue somewhere with concurrent builds.

Cheers,

--
Pete
Reply
#12
@pchristy 
To clean the compiler cache after a build, run:
Code:
/usr/bin/ccache -C

And, I guess your only way to actually get a log and see where the compilation fails, without influencing your "quantum computer", is to increase the console buffer and browse it or dump/save it into a file (if the X terminal supports such a function).

P.S. Smile
https://en.wikipedia.org/wiki/Observer_effect_(physics)
Reply
#13
Yeah, we call it: Heisenbug
First decide what functions / features you expect from a system. Then decide for the hardware. Don't waste your money on crap.
Reply
#14
Der Werner und seine Unschärferelation...
Reply
#15
you can also try running the build inside "script", which will log all output to a file without using extra command redirection:
Code:
script kodi-build.log
cmake .......
make -j8
...
exit
Reply

Logout Mark Read Team Forum Stats Members Help
Building Kodi from source on hyper-threaded systems.0