[zachmorris]

I'm interested in a couple of python modules that also contain C, and using them in Kodi. Pandas for example.
On my desktop, it's as easy as 'pip install'

Is there a method or template to build a module addon that incorporates pip in the build process?

I see most binary addons are built with cmake, although I see there are methods to use pip with cmake, but I'm not savvy enough to understand how to set this up.

[Fuzzard]

As far as I've seen there isn't anything out of tree currently.

For in tree stuff, you can look at how pycryptodome and PIL are done
https://github.com/xbmc/xbmc/tree/master...module-pil
https://github.com/xbmc/xbmc/tree/master...cryptodome


Once you have it compiled my experience is only for iOS/tvos.
Essentially for iOS/tvos the compiled archive gets copied into the script module (https://github.com/xbmc/xbmc/tree/master...module.pil).

I can get some folder structures when I have my computer next as an example.

[wsnipex]

Everything that needs to be compiled must be shipped with kodi. So it needs to be added to depends like outlined above.
The reason is that embedded platforms have no way of using external modules.

[Fuzzard]

Just to touch base on this again. Ive been doing some work on getting a numpy module up and running.
Just had the below image for the first time, running on an ipad.

Dont want to get hopes up, still a fair bit of stuff to go (all other platforms essentially), but it works, and might be able to give others a bit of guideline on how to do other binary modules
I'll do some cleanup tomorrow and throw a link to my branch with this work in it

@Lunatixz was kind enough to provide the numpy test plugin https://github.com/fuzzard/script.numpy.test

edit: Link to my repo branch https://github.com/fuzzard/xbmc/commits/...dule-numpy extremely wip, and one of the patches definitely needs to be looked into a non hardcoded way to do what i needed, but feel free to look it over and ask any questions if you want to dive into other modules.

For other modules, you're going to want to very closely look at dependencies. From a kodi stance, any modules we ship essentially have to run on all (or at least a majority) platforms. The embedded platforms are obviously the tricky part to this (Android, and Apple). An example of something im not really going to tackle is scipy, which requires a fortran compiler as a dependency. 

[Fuzzard]

Necroing an old thread again, but just for interest.

Ive finally worked through our tools/depends system and python with all its quirks. https://github.com/xbmc/xbmc/pull/19246
Hoping this will get merged into v20 in the not too distant future

This now provides clear separation between build and target pythons, which makes adding additional modules a lot easier (imo anyway) to create.

Ive reworked the above numpy module based on the new work - https://github.com/fuzzard/xbmc/commit/b...6aecf6e169
May look like a lot, but it just adds cython as a dependency, and then sets the required build options for numpy. A lot of the changes are boilerplate that could be copied to any module.
For dependencies (eg cython), we can now even use pip to install them if desired. Ive included an example in the pythonmodule-cython example of this, but ive defaulted to manually building them.

If anyone looks to create other modules, feel free to ask any questions, happy to help out.
The above is only relevant for platforms that use tools/depends to build (all Apple, Android). Linux just uses system, so have the user pip install and then just provide the addon template with the addon.xml file. Windows has its own build system, so unfortunately i cant help out with that at all.

[zachmorris]

Necroing an old thread again, but just for interest.

Ive finally worked through our tools/depends system and python with all its quirks. https://github.com/xbmc/xbmc/pull/19246
Hoping this will get merged into v20 in the not too distant future

This now provides clear separation between build and target pythons, which makes adding additional modules a lot easier (imo anyway) to create.

Ive reworked the above numpy module based on the new work - https://github.com/fuzzard/xbmc/commit/b...6aecf6e169
May look like a lot, but it just adds cython as a dependency, and then sets the required build options for numpy. A lot of the changes are boilerplate that could be copied to any module.
For dependencies (eg cython), we can now even use pip to install them if desired. Ive included an example in the pythonmodule-cython example of this, but ive defaulted to manually building them.

If anyone looks to create other modules, feel free to ask any questions, happy to help out.
The above is only relevant for platforms that use tools/depends to build (all Apple, Android). Linux just uses system, so have the user pip install and then just provide the addon template with the addon.xml file. Windows has its own build system, so unfortunately i cant help out with that at all.

Looks awesome, I think this could extend functionality in a lot of different ways. I'll give it a try.

[Fuzzard]

I had a wip lxml one from way back, and a team member asked about that. So here is the a module for lxml

https://github.com/fuzzard/xbmc/commit/5...4a862f6623

Mind you, this one isnt fully tested. If anyone has the time to create a simple test addon to verify lxml functionality, im happy to confirm for Apple/android platforms
In the past, @Lunatixz did a test addon for numpy for me, so maybe something similar - https://github.com/fuzzard/script.numpy.test

[zachmorris]

I had a wip lxml one from way back, and a team member asked about that. So here is the a module for lxml

https://github.com/fuzzard/xbmc/commit/5...4a862f6623

Mind you, this one isnt fully tested. If anyone has the time to create a simple test addon to verify lxml functionality, im happy to confirm for Apple/android platforms
In the past, @Lunatixz did a test addon for numpy for me, so maybe something similar - https://github.com/fuzzard/script.numpy.test

I'm finally getting back to this as I think lxml would be a great alternative to pythons builtin ET (given that currently there's a bug in ET causing Kodi v20 to crash), and I'm willing to try and test it for functionality.

I've built the latest master of kodi with the lxml module you referenced, but it doesn't seem to work for some reason. It does show as installed:

python:


info <general>: Starting Kodi (21.0-ALPHA1 (20.90.101) Git:20230208-6d04737c86-dirty). Platform: macOS x86 64-bit

info <general>: Using Debug Kodi x64

info <general>: Kodi compiled 2023-02-08 by Clang 14.0.0 (clang-1400.0.29.202) for macOS x86 64-bit version 10.13 (101300)

...

info <general>: CAddonMgr::FindAddons: script.module.lxml v4.6.1 installed

info <general>: CAddonMgr::FindAddons: script.module.pil v5.1.0 installed

info <general>: CAddonMgr::FindAddons: script.module.pycryptodome v3.4.3 installed

...

But when I simply try and import i get an error:

python:


import lxml

Results in:

python:


ModuleNotFoundError: No module named 'lxml'

And when I query the internal python interpreter for what packages are installed it isn't listed:

python:


import pkg_resources

installed_packages = [(d.project_name,d.version) for d in pkg_resources.working_set]

installed_packages_str = ['{}=={}'.format(i[0],i[-1]) for i in installed_packages]

xbmc.log(msg='Python Packages:',level=xbmc.LOGINFO)

for p in sorted(installed_packages_str):

	xbmc.log(msg=p,level=xbmc.LOGINFO)

Results in:

python:


info <general>: Python Packages:

info <general>: Pillow==8.4.0

info <general>: setuptools==65.5.0.post20230208

[Fuzzard]

Been a long time since i did this, doesnt suprise me it doesnt work to be honest.
As it stands, i doubt i'll go back into more binary addons. to many issues in cpython itself with our use of sub interpreters, and some binary python module authors have shown zero interest in making their stuff work with sub interpreters, so i see it as a dead end until they come to the party.

[zachmorris]

Been a long time since i did this, doesnt suprise me it doesnt work to be honest.
As it stands, i doubt i'll go back into more binary addons. to many issues in cpython itself with our use of sub interpreters, and some binary python module authors have shown zero interest in making their stuff work with sub interpreters, so i see it as a dead end until they come to the party.

Thanks for the info. I'll keep trying at it and see how far I can get. What would the appetite be in your opinion for a PR that adds modules like this if I were to eventually open one?

[Fuzzard]

I would say minimal at this stage.

As I mentioned, unless a C based add-on understands and is built to handle sub interpreters, it's just asking for crashes and troubles. Cpython has been on a long road for a while improving their sub interpreter handling for all std modules, but last I looked, people like numpy were not wanting to do anything with their code regarding sub interpreters.
This may have changed for that particular project, but it just shows how reliant we would be on a project to understand and implement correct things, otherwise we just crash. Not good for users.