Kodi Community Forum

Full Version: building kodi with clang/LTO unavailable... is there an internal switch?
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Pages: 1 2 3 4 5
I have been playing with compiling using clang.  How does one enable LTO with it?  Passing -DUSE_LTO=8 to cmake as I do with gcc gave this:
Code:
...
-- #---------------------------------------------#
-- SYSTEM_LDFLAGS        : 
-- CMAKE_DL_LIBS         : dl
-- CMAKE_C_FLAGS         : -D_FORTIFY_SOURCE=2  -march=haswell -mtune=haswell -O2 -pipe -fno-plt -fdiagnostics-color -Wall
-- CMAKE_CXX_FLAGS       : -D_FORTIFY_SOURCE=2  -march=haswell -mtune=haswell -O2 -pipe -fno-plt -fdiagnostics-color -Wall
-- CMAKE_EXE_LINKER_FLAGS: -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now
-- LTO_OPTIMIZATION:     : 
-- #---------------------------------------------#
-- bindir     : /usr/bin
-- includedir : /usr/include
-- libdir     : /usr/lib
-- datarootdir: /usr/share
-- #---------------------------------------------#
-- BINARY: kodi-x11
-- #---------------------------------------------#
-- GIT_REV: 20210612-nogitfound
-- Build date: 20210612
-- #---------------------------------------------#
-- CPACK_GENERATOR       : 
-- CPACK_SOURCE_GENERATOR: 
-- #---------------------------------------------#
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    USE_LTO

I guess just add the -flto=thin to the CFLAGS and CXXFLAGS?
our LTO support was added before clang had it, so please try removing https://github.com/xbmc/xbmc/blob/master....cmake#L41
@wsnipex Is this correct?
Code:
--- a/cmake/scripts/linux/ArchSetup.cmake    2021-06-12 20:21:58.000000000 -0400
+++ b/cmake/scripts/linux/ArchSetup.cmake    2021-06-13 07:40:36.552096793 -0400
@@ -37,8 +37,7 @@ else()
   endif()
endif()

-if((CMAKE_BUILD_TYPE STREQUAL Release OR CMAKE_BUILD_TYPE STREQUAL MinSizeRel)
-    AND CMAKE_COMPILER_IS_GNUCXX)
+if(CMAKE_BUILD_TYPE STREQUAL Release OR CMAKE_BUILD_TYPE STREQUAL MinSizeRel)
   # Make sure we strip binaries in Release build
   set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} -s")

When I use that, I do not see a line telling me that the system picked it up. For example, in the past I would see: LTO_OPTIMIZATION: : TRUE but after this modification, I see nothing there, just: LTO_OPTIMIZATION: :

Note I am defining clang like this:
Code:
export CC=clang
export CXX=clang++

Any thoughts?
Does anyone know why my build ends in error when I try building with -flto=thin
Code:
export CC=clang
export CXX=clang++
export CFLAGS+=" -march=haswell -mtune=haswell -flto=thin"
export CXXFLAGS="${CFLAGS}"

Complete build log: https://gist.github.com/graysky2/7396ea7...cb4ca6063e
@graysky The actual error is:
Code:
checking dynamic linker characteristics... libtool: link: gcc -shared -fPIC -DPIC lib/ac/.libs/string.o lib/.libs/downcase.o lib/.libs/fmemcmp.o lib/.libs/fmemcmpi.o lib/.libs/fstrcasecmp.o lib/.libs/fstrcasecmpi.o lib/.libs/fstrcmp.o lib/.libs/fstrcmpi.o lib/.libs/fstrcoll.o lib/.libs/fstrcolli.o lib/.libs/fwcscmp.o lib/.libs/fwcscmpi.o lib/.libs/mbs_to_wcs.o lib/.libs/nmalloc.o lib/.libs/program_name.o lib/.libs/version.o lib/.libs/version_print.o -march=x86-64 -mtune=generic -O2 -march=haswell -mtune=haswell -flto=thin -Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z -Wl,relro -Wl,-z -Wl,now -Wl,-soname -Wl,libfstrcmp.so.0 -o lib/.libs/libfstrcmp.so.0.6.0
lib/ac/.libs/string.o: file not recognized: file format not recognized
collect2: error: ld returned 1 exit status

This happens when building fstrcmp which is a very old and unmantained source package.
This is all I can tell you, I have no idea why this happens and how to fix it.
One way to work this around would be to build fstrcmp as a separate package with gcc before building Kodi with -DENABLE_INTERNAL_FSTRCMP=OFF.
Thanks for the reply, @asavah.  When I pre-build fstrcmp and have it installed as a dependency together with -DENABLE_INTERNAL_FSTRCMP=OFF, the build still errors out.

Complete build log: https://gist.github.com/graysky2/8a96484...0447f8abc1

Now the first error is with building ffmpeg it seems:
Code:
[ 13%] Building CXX object CMakeFiles/flatc.dir/src/util.cpp.o
/usr/bin/clang is unable to create an executable file.
If /usr/bin/clang is a cross-compiler, use the --enable-cross-compile option.
Only do this if you know what cross compiling means.
C compiler test failed. If you think configure made a mistake, make sure you are using the latest version from Git. If the latest version fails, report the problem to the [email protected] mailing list or IRC #ffmpeg on irc.freenode.net. Include the log file "ffbuild/config.log" produced by configure as this will help solve the problem.
make[5]: *** [CMakeFiles/ffmpeg.dir/build.make:92: ffmpeg-prefix/src/ffmpeg-stamp/ffmpeg-configure] Error 1
make[4]: *** [CMakeFiles/Makefile2:83: CMakeFiles/ffmpeg.dir/all] Error 2
make[3]: *** [Makefile:136: all] Error 2
make[2]: *** [CMakeFiles/ffmpeg.dir/build.make:86: build/ffmpeg/src/ffmpeg-stamp/ffmpeg-build] Error 2
make[1]: *** [CMakeFiles/Makefile2:4837: CMakeFiles/ffmpeg.dir/all] Error 2 make[1]: *** Waiting for unfinished jobs....
@graysky In the folder with ffmpeg sources there should be a ffbuild/config.log file.
It may contain some clues. Pastebin it please.

Edit: quick google search suggests that you may need to modify the cmake file responsible for building ffmpeg and add --as=/path/to/clang
see https://github.com/xbmc/xbmc/issues/19881 for reference and a possible solution
Thanks for the modified files in your fork, @wsnipex - I posted my failed result to build it in the github issue.  Might be better to continue conversation there.

@asavah - Are you still interested in seeing the ffbuild/config.log?  I thought I would pull back to just trying to build ffmpeg with clang and with -flto=thin and I could not get it to work either even modifying my LDFLAGS to include -flto=thin.

For pure ffmpeg NOT kodi:
Here is the build log: https://gist.github.com/graysky2/7e73b1d...8d71defc0d
Here is the ffbuild/config.log: https://gist.github.com/graysky2/243e2aa...cad4153b52

For the record, I tried building ffmpeg like this:
Code:
CFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -flto=thin"
CXXFLAGS="$CFLAGS"
LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -flto"

./configure     --prefix=/usr     --disable-debug     --disable-static     --disable-stripping     --enable-amf     --enable-avisynth     --enable-cuda-llvm     --enable-fontconfig     --enable-gmp     --enable-gnutls     --enable-gpl     --enable-ladspa     --enable-libaom     --enable-libass     --enable-libbluray     --enable-libdav1d     --enable-libdrm     --enable-libfreetype     --enable-libfribidi     --enable-libgsm     --enable-libiec61883     --enable-libjack     --enable-libmfx     --enable-libmodplug     --enable-libmp3lame     --enable-libopencore_amrnb     --enable-libopencore_amrwb     --enable-libopenjpeg     --enable-libopus     --enable-libpulse     --enable-librav1e     --enable-librsvg     --enable-libsoxr     --enable-libspeex     --enable-libsrt     --enable-libssh     --enable-libsvtav1     --enable-libtheora     --enable-libv4l2     --enable-libvidstab     --enable-libvmaf     --enable-libvorbis     --enable-libvpx     --enable-libwebp     --enable-libx264     --enable-libx265     --enable-libxcb     --enable-libxml2     --enable-libxvid     --enable-libzimg     --enable-nvdec     --enable-nvenc     --enable-shared     --enable-version3     --cc=clang --cxx=clang++
  make
@graysky sadly my understanding of clang quirks is very limited, your best bet to fix this is @wsnipex 

A small observation:
Code:
x86 assembler nasm
~
X86ASM libavcodec/x86/mdct15.o
CC libavcodec/x86/mdct15_init.o
~
libavcodec/x86/mdct15_init.o: file not recognized: file format not recognized

A wild speculation: setting assembler to clang itself --as=/usr/bin/clang may be actually needed for ffmpeg to build.
it requires the LLVM linker (at least for now) and llvms own ar, nm, ranlib. At least on ubuntu those are installed as /usr/bin/llvm-[ar|nm|ranlib]. If that's not the case for you, it's going to be more work.
Thanks for all the support with this, guys.  I added some comments to the github issue but even defining the cmake switches -DENABLE_CMAKE_CXX_COMPILER_AR=/usr/bin/llvm-ar -DENABLE_CMAKE_CXX_COMPILER_RANLIB=/usr/bin/llvm-ranlib -DENABLE_CMAKE_NM=/usr/bin/llvm-nm and also building using a pre-compiled/external fstrcmp ended in errors.  I posted this result to the github issue.  Happy to try something else.

At least there is some progress in that cmake's build is now identifying LTO whereas before it was not.
@graysky 
Code:
CMake Warning:
Manually-specified variables were not used by the project:
ENABLE_CMAKE_CXX_COMPILER_AR
ENABLE_CMAKE_CXX_COMPILER_RANLIB
ENABLE_CMAKE_NM

I think these vars should be passed without ENABLE_ eg. -DCMAKE_CXX_COMPILER_AR

 
Code:
CMAKE_EXE_LINKER_FLAGS: -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now;-fuse-ld=lld

Where does this  -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now; comes from? I don't see anything in Kodi setting -z,relro,-z,now; and I'm concerned about ";" after "now" , check your scripts/environment or Arch defaults.
Or try force resetting the var with -DCMAKE_EXE_LINKER_FLAGS="" and let Kodi buildsystem do its thing.
@graysky  I've updated my branch: https://github.com/wsnipex/xbmc/commits/clang-lto

There should be no need to set ENABLE_LLD, CMAKE_CXX_COMPILER_[AR|NM|..] anymore.
Also, as asavah said, don't set any additional LD or CXX Flags for a start.
(2021-06-15, 03:14)asavah Wrote: [ -> ]Where does this  -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now; comes from? I don't see anything in Kodi setting -z,relro,-z,now; and I'm concerned about ";" after "now" , check your scripts/environment or Arch defaults.

That's an Arch default, bit I do not know where the ";" is coming from... it's not set in my /etc/makepkg.conf which is where the build system looks to have them set.  Currently:
Code:
CPPFLAGS="-D_FORTIFY_SOURCE=2"
CFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fno-plt"
CXXFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fno-plt"
LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now"
 
(2021-06-15, 11:14)wsnipex Wrote: [ -> ]@graysky  I've updated my branch: https://github.com/wsnipex/xbmc/commits/clang-lto

There should be no need to set ENABLE_LLD, CMAKE_CXX_COMPILER_[AR|NM|..] anymore.
Also, as asavah said, don't set any additional LD or CXX Flags for a start.

OK! I pulled your latest commit and added the cmake line recommended by @asavah to clear the LDFLAGS.  The build got much further this time, but ended in errors.

Complete build log is here.

For reference, I am building like this:
Code:
  export CC=clang
  export CXX=clang++

  _args=(
    -DCMAKE_INSTALL_PREFIX=/usr
    -DCMAKE_INSTALL_LIBDIR=/usr/lib
    -DCMAKE_EXE_LINKER_FLAGS=""
    -DUSE_LTO=8
    -DENABLE_LDGOLD=OFF
    -DENABLE_EVENTCLIENTS=ON
    -DENABLE_INTERNAL_FFMPEG=ON
    -DENABLE_INTERNAL_FMT=ON
    -DENABLE_INTERNAL_SPDLOG=ON
    -DENABLE_INTERNAL_CROSSGUID=ON
    -DENABLE_INTERNAL_FSTRCMP=ON
    -DENABLE_INTERNAL_FLATBUFFERS=ON
    -DENABLE_INTERNAL_UDFREAD=ON
    -DENABLE_MYSQLCLIENT=ON
    -Dlibdvdcss_URL="/libdvdcss-1.4.2-Leia-Beta-5.tar.gz"
    -Dlibdvdnav_URL="/libdvdnav-6.0.0-Leia-Alpha-3.tar.gz"
    -Dlibdvdread_URL="/libdvdread-6.0.0-Leia-Alpha-3.tar.gz"
    -DFFMPEG_URL="/ffmpeg-4.3.2-Matrix-19.2.tar.gz"
    -DFMT_URL="/fmt-6.1.2.tar.gz"
    -DSPDLOG_URL="/spdlog-1.5.0.tar.gz"
    -DCROSSGUID_URL="/crossguid-8f399e8bd4.tar.gz"
    -DFSTRCMP_URL="/fstrcmp-0.7.D001.tar.gz"
    -DFLATBUFFERS_URL="/flatbuffers-1.12.0.tar.gz"
    -DUDFREAD_URL="/libudfread-1.1.0.tar.gz"
    -DAPP_RENDER_SYSTEM=gl
    -DCORE_PLATFORM_NAME=x11
    -DVERBOSE=ON
    -DENABLE_AIRTUNES=OFF
    -DENABLE_AVAHI=OFF
    -DENABLE_BLUETOOTH=OFF
    -DENABLE_CEC=OFF
    -DENABLE_OPTICAL=OFF
    -DENABLE_SMBCLIENT=OFF
    -DENABLE_UPNP=OFF
  )
cmake "${_args[@]}" ../xbmc-commit
make -j9
Pages: 1 2 3 4 5