Work in Progress - script.module.abi, a platform independent binary addon loader

  Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
boogiepop Offline
Junior Member
Posts: 45
Joined: Jan 2015
Reputation: 2
Post: #1
Hi

I am happy to announce abi (Application Binary Interface) module for kodi. This module's purpose is to provide an interface for python c/c++ addon packagers inside kodi / xbmc python interpreter. It opens a total new dimension for kodi addons.

The interface is pretty straight forward:
PHP Code:
import abi

binmodule 
abi.load("some.addon.name""modulename"

where some.addon.name has [email protected]_pythonid/modulename.so [or pyd, dll] directory structure. Abi module automatically detects the machine id name [os@...] automatically and imports to module in wicked ways depending to the machine.

Machine Definitions:
os: windows, linux, android,
apple products like ios, darwin appletv is not implemented yet for 3 reasons, 1) i dont use apple things, 2) i dont like about apple things, 3) i dont care about apple things. However it can be impelented and is possible.

toolchain: toolchain is the binary dependence of your interpreter to load a binary addon, in which it is your current systems operating systems ABI, and your interpreters c-library dependence if your module to be loaded is not compiled staticaly with c-libs.
a) for windows there are following toolchains:
a.1) vc2008 toolchain: where your addon is compiled with cruntime of vc 2008 (msvcrt90.dll dependency)
a.2) vc2015 toolchain: your addon is compiled vc 2015 (vcruntime140.dll) which is kodi 17 krypton. Those addons require vc redist dependency, but this dependency must already be satisfied on the kodi install, because kodi itself already depends on vc2015.
a.3) winabi: this toolchain stands for the binaries which has cruntime is statically compiled with it. ABI module first tries to load one of the above toolchain (vc2008 or vc2015) if it fails, it tries to load winabi. As you can tell winabi toolchain is more generic, but not all of the addons can be compiled with static cruntime, it has so tweaks, and quirks.

b) for linux:
b.1) pep513: which stands for manylinux1 tag standartized in https://www.python.org/dev/peps/pep-0513/ which satisfies a generic vast majority of linux distributions.

c) for android:
c.1) ndk: android bionic c-library seems to be backwards compatible (i didnt check in much in details yet)

architecture: architecture is the definition to identify on which kind of cpu that the binary addon will run. Options are: x86, x64, arm5, arm6, arm7, arm8. Rest of fancy cpus can be implemented later on.

pythonid: we know that xbmc/kodi is shipped with either python 2.6 or python 2.7, these two create 2 different dependency however this is not all, there is also UCS2, UCS4 compile time variants. UCS defines how wide is the unicode support inside the interpreter and it is an header definition which means it creates another dependency. There is also another varian of which interpreter should be used inside python but it is always cpython for kodi/xbmc so it can be dismissed generally. At the end of the day this leaves us 4 variants:
1) cp26m : cpython 2.6 UCS 2 Narrow Unicode Support
2) cp26mu : cpython 2.6 UCS 4 Wide Unicode Support
3) cp27m : cpython 2.7 UCS 2 Narrow Unicode Support
4) cp27mu : cpython 2.7 UCS 2 Wide Unicode Support

Practical Machine Ids:
As you can understand above combination of above parameters generated a bunch of targets however in practice not all of the combinations are needed. If the target operating system has generic market for kodi distribition compile-time options narrow down drastically. Ie: for windows and android there is on UCS2 builds. For windows there is never X64 distribution. For android there is only amr7 & arm8 cpu targets. So we can guess that practical machine ids are :

Desktop PCs with Windows
[email protected]_cp26m (releases before xbmc < 13 i think)
[email protected]_cp27m (releases before 13 < xbmc < 16 i think)
[email protected]_cp27m (kodi 17 krypton)

Desktop PCs with Linux:
[email protected]_cp26m
[email protected]_cp26mu
[email protected]_cp27m
[email protected]_cp27mu
[email protected]_cp26m
[email protected]_cp26mu
[email protected]_cp27m
[email protected]_cp27mu

Mobile Platforms with android:
[email protected]_cp26m (release before kodi < 17)
[email protected]_cp27m (kodi 17)
[email protected]_cp27m
[email protected]_cp27m

Apple things are ignored. Embedded Linuxes including raspberry pi or other embedded linuxes will be added later on.

Current Status:
I have first thought about this project like the rest of us complaining how slow BeautifulSoup is and how much i need lxml inside kodi, unfortunately lxml fast is because it is c written xml serializer and cant be run under kodi. Then i met tribler (see this presentation in stanford university) and i admired their effort on changing the internet. Then i here i am with the following addons required for Tribler Core. I have built below addons already for most of 32 bit platforms and tested under windows and android krypton, succesfully could use all of them:

script.module.m2crypto=0.21.1 (fast cryptography api)
script.module.cryptography=1.2.1 (fast cryptography api)
script.module.libnacl=1.4.4 (fast cryptography api)
script.module.libtorrent=1.0.8 (p2p torrent api)
script.module.apsw=3.15.0 (fast sqlite api)
script.module.plyvel=0.9 (leveldb api)
script.module.netifaces=0.10.5 (acces os networking internals)
script.module.cffi=1.10.0 (swiss knife dynamic dll loader/runtime compiler)


You can see how it works and performs from the above packages from my github https://github.com/hbiyik?tab=repositories, currently there is no kodi repo at all and there is a bunch things to work on, and meanwhile since this is still a work in progress all of the above definitions are subject to change.

you can also see the abi code in

https://github.com/hbiyik/script.module.abi
(This post was last modified: 2017-04-18 18:05 by boogiepop.)
find quote
Roman_V_M Offline
Team-Kodi Member
Posts: 1,066
Joined: Jun 2011
Reputation: 52
Location: Kyiv, Ukraine
Post: #2
Interesting. Have you actually tried this on Android? Because, as to my knowledge, importing a binary Python module from Kodi on Android is a non-conventional process.

Also, since I'm pretty well aware of all the problems with cross-compiling binary Python modules for different platforms, a gentle request to include binaries for Raspberry Pi as well.

Raspberry PI 2 + LibreELEC 8 (Kodi 17)
Acer IconiaTab A510 + Kodi 14.2 for Android
(This post was last modified: 2017-04-18 11:47 by Roman_V_M.)
find quote
boogiepop Offline
Junior Member
Posts: 45
Joined: Jan 2015
Reputation: 2
Post: #3
Yes i have tested all of them on android 5.1.1, with kodi 17, and also i have seen your blog post on pythonanywhere about it, the python shared library must be on libkodi.so.

https://github.com/hbiyik/script.module....t__.py#L81

there are permission issues on android but thanks to xbmcvfs i can override it. Loading the module with CDLL is only to check if OS raises dlimport error, if raised, abi tries to next folder possible until the base installtion of kodi, in which this place must have execution right because main kodi binary is already executed from. At last after a successfull load, abi caches this location on next load does not look for a place where os has exec rights and loads it directly from last succesfull place. (gains speed by not trying to copy binaries all around). Btw it also cleans if load fails.

This however brings some restrictions, the dll to be imported must only depend on available system libraries, because i dont use path manipulations and if the if the dll depends on some extra dll (like boost_python_xx.so or my_shared.so) it wont be able to load, because the depended libraries will be in non-execution rights directory. In short, binaries must be statically compiled.

I have used android 13 ndk from https://github.com/dockcross/dockcross dockers which are very conveinent. However i have patched my distutils manually several times for a succesfull compilation. I don't have so much knowledge on how much android bionic libc is backwards compatible, but so far it seems pretty succsfull, since ndk13b for android 3.2 and i have tested on 5.1.1. May be you can have some comments on it.

For raspberries:
It is easy to compile the binaries with existing toolchains with the setup i have however the route i want to take is a little bit more interesting. We know that raspberry works with linux oses, so PEP513 manylinux1 definition must comply with raspbian, librelece, xbian etc whatever the custom build is. I will build a custom toolchains most likely with cross-toolsng with PEP513 conformance.

GLIBC <= 2.5
CXXABI <= 3.4.8
GLIBCXX <= 3.4.9
GCC <= 4.2.0

and at the end of the day, for raspberry pi 1

[email protected]~hf~le_armv6_cp26m
[email protected]~hf~le_armv6_cp26mu
[email protected]~hf~le_armv6_cp27m
[email protected]~hf~le_armv6_cp27mu

should comply abi dependency. The good thing is not only for raspberry but *many (as stated in pep513)* of other embedded linuxes running with linux with armv6. But new definitions to be made hf (hard float), le (little endian) for arm platforms accordingly and script.module.abi should detect the current running os variants and try to load correct binary. So this will be done but in first phase i want to stick with windows/linux_x86/androids and make sure that everthing works fine. After i am sure, i can move forward to other embeddeds.
(This post was last modified: 2017-04-18 12:49 by boogiepop.)
find quote
Roman_V_M Offline
Team-Kodi Member
Posts: 1,066
Joined: Jun 2011
Reputation: 52
Location: Kyiv, Ukraine
Post: #4
Thanks for your reply. Regarding compiling for Android, in my experiments I used CrystaX NDK. Its developers claim that this NDK is more compatible with C/C++ standards than the official Android NDK, however I don't follow Android platform development, so things may have changed in this aspect.

Raspberry PI 2 + LibreELEC 8 (Kodi 17)
Acer IconiaTab A510 + Kodi 14.2 for Android
find quote
boogiepop Offline
Junior Member
Posts: 45
Joined: Jan 2015
Reputation: 2
Post: #5
thanks for your valued posts on python anywhere, may be it would be usefull for others if those posts are pinned to the forums also. I think i will have to experience backwards compatibility for android bionic libc ampirically in time.

And for those who want to contribute abi project or the packed packages above here is a basic dependency map of overall, triblerCore arhitecture for xbmc.

PHP Code:
script.module.triblercore (P)
  |||||||||||
  ||||||||||+--> 
script.module.twisted (P)
  ||||||||||      |
  ||||||||||      +--> 
script.module.zope.interface (P)
  ||||||||||      +--> 
script.module.incremental (P)
  ||||||||||      +--> 
script.module.constantly (P)
  ||||||||||      +--> 
script.module.zope.interface (P)
  ||||||||||      +--> 
script.module.pywin32ctypes (P)
  ||||||||||             |
  ||||||||||             +--> 
script.module.cffi (P/C)
  ||||||||||             |      |
  ||||||||||             |      +--> 
libffi (C)
  ||||||||||             |      +--> 
script.module.pycparser (P)
  ||||||||||             |      +------------------------------------+--> 
script.module.abi (P)
  ||||||||||             |                                           |
  |||||||||+--> 
script.module.crytography (P/C)                      |
  |||||||||       |                                                  |
  |||||||||       +--------------------------------------------------+
  |||||||||       +--> 
script.module.idna (P)                        |
  |||||||||       +--> 
script.module.pyasn1 (P)                      |
  |||||||||       +--> 
script.module.six (P)                         |
  |||||||||       +--> 
script.module.enum (P)                        |
  |||||||||       +--> 
script.module.ipaddress (P)                   |
  |||||||||       +------------------------------+--> 
OpenSSL (C)    |
  |||||||||                                      |                   |
  ||||||||+--> 
script.module.m2crypto (P/C)      |                   |
  ||||||||       |                               |                   |
  ||||||||       +-------------------------------+                   |
  ||||||||       +---------------------------------------------------+
  ||||||||                                                           |
  |||||||+--> 
script.module.libnacl (P)                              |
  |||||||       |                                                    |
  |||||||       +----------------------------------------------------+
  |||||||       +--> 
libsodium (C)                                   |
  |||||||                                                            |
  ||||||+--> 
script.module.apsw (P/C)                                |
  ||||||       |                                                     |
  ||||||       +-----------------------------------------------------+
  ||||||       +--> 
libsqlite3 (C)                                   |
  ||||||                                                             |
  |||||+--> 
script.module.plyvel (P/C++)                             |
  |||||       |                                                      |
  |||||       +------------------------------------------------------+
  |||||       +--> 
LevelDB (C++)                                     |
  |||||               |                                              |
  |||||               +--> 
Boost (C++)                               |
  |||||               |                                              |
  ||||+--> 
script.module.libtorrent (P/C++)                          |
  ||||       |                                                       |
  ||||       +-------------------------------------------------------+
  ||||                                                               |
  |||+--> 
script.module.netifaces (P/C)                              |
  |||       |                                                        |
  |||       +--------------------------------------------------------+
  ||| 
  ||+--> 
script.module.chardet (P)
  ||
  |+--> 
script.module.cherrypy (P)
  |       |
  |       +--> 
script.module.cheroot (P)
  |       +--> 
script.module.six (P)
  |       +--> 
script.module.portend (P)
  |
  +--> 
script.module.tempora (P)

(
P) : Python Code
(C) : C Code
(C++) : C++ Code 
(This post was last modified: 2017-04-19 21:53 by boogiepop.)
find quote
tribler Offline
Junior Member
Posts: 2
Joined: Mar 2015
Reputation: 2
Post: #6
(2017-04-19 21:50)boogiepop Wrote:  And for those who want to contribute abi project or the packed packages above here is a basic dependency map of overall, triblerCore arhitecture for xbmc.

Very impressive work! Would it be possible to share a screenshot of some of the code in action? We're trying to replicate all your work in our lab now.
find quote
boogiepop Offline
Junior Member
Posts: 45
Joined: Jan 2015
Reputation: 2
Post: #7
Quote: Would it be possible to share a screenshot of some of the code in action?

Sorry, i did not really understand what you mean by screenshot? You mean you want to see script.module.triblercore in action, or how i build the so binaries?

if you had problems on setting up environment for script.module.triblercore
https://github.com/hbiyik/script.module....e/issues/1

above adress is quite helpfull, if you wanna see it in action with a client, you can use my fork of einarrs client.
https://github.com/hbiyik/plugin.video.tribler

Any problems, please notify
find quote