Kodi Community Forum

Hello,

I am maintaining the Kodi video add-on for PeerTube (code here - more information about PeerTube here) which requires a BitTorrent library since PeerTube is based on this protocol.
Currently the add-on uses libtorrent, a C++ implementation of the BitTorrent protocol which provides python bindings.
The python bindings are not easily installable on all the systems though: there is no official pip package yet (even though they are working on it) but even this will not help for some systems where pip cannot be used (Android, TvOS, JeOS, etc.).

There is a Kodi add-on which provides pre-built versions of libtorrent for many systems: script.module.libtorrent
It looks like a good solution (even though the maintaining effort may be quite big to support all the systems) but then I found out that binary add-ons exists after reading this and this but I was not able to find documentation about this.
So I am asking for your help since I am quite new to Kodi development eco-system.

Do you think a binary add-on based on libtorrent code is a good option for my use case? (libtorrent is used by many software so it may be of interest for other projects)
How are binary add-ons compiled for all the systems? Is it provided somehow by Kodi "infrastructure" or is the owner of the add-on responsible for it?

Thank you in advance for your help!

Yes, binary addons are built by Kodi infrastructure (Jenkins). The rule of thumb for addon submission to the official repo is that everything must be human-readable (artwork is included in that category) so binary blobs are not accepted. However, as far as Python binary libraries are concerned, those few libraries that are included with Kodi (Pillow, pycryprodome) are built with Kodi itself, except on generic Linux distributions that use Python libraries from system repos. This means that Kodi oficial repo does not contain any third-party binary Python libraries, everything Python-related is shipped with Kodi. And I don't think that something like libtorrent will be officially shipped with Kodi.

As for script.module.libtorrent, I was one of contributors to this library (I built Windows binaries). Binaries for different platforms were provided by various people who managed to create build environments for specific platforms. And that was real PITA considering that librorrent used (or maybe still uses, I haven't followed its development for some time) quite esoteric Boost Build system. 

All this means that if you want to resurrect script.module.libtorrent you will need to build binaries for all platforms that you want to support all by yourself. And note that binaries included in script.module.libtorrent were built against Python 2.7, and now you need to build them against Python 3.x.
Also, if I remember correctly Android binaries from script.module.libtorrent stopped working on latest Android versions because of some changes in Android.

Thank you for the detailed information about the binary add-ons  and script.module.libtorrent. It is much clearer for me now.

I don't know any other BitTorrent library than libtorrent which is actively maintained and feature complete. Moreover it seems it is used by many applications and torrent clients so I guess the support will continue.
So I will try to resurrect script.module.libtorrent even though it will not be easy to maintain as you said.
Based on what I read in libtorrent repository, they invested a lot in the compilation of the python bindings so I hope it will be easier than what you had to do some years ago.
I wil start small with only a few systems supported at first and then I may add additional systems like Android or tvOS.
If anybody is interested in this project and would like to help, please contact me !

The proper way would be to create a VFS binary addon, that uses libtorrent directly instead of going through python.
see https://github.com/xbmc?q=vfs&type=&language=&sort= for a few examples

Thanks for the suggestion wsnipex but it raised many questions in my head ^^

I had a look to the examples and here is what I understood:
- the library code (libtorrent in my case) has to be added in the add-on
- a customized class CMyAddon which includes libtorrent's code has to be called by ADDONCREATOR()
Is it correct? Is there any document explaining what is expected?

How different is a VFS binary add-on compared to a "classic" binary add-on?

When you said it is the "proper way" is it because a VFS binary add-on based on C++ code (libtorrent) would be accepted in Kodi's official repo contrary to a binary add-on containing a third-party python library (which was my original idea)?

Basically, a kodi binary addon consists of a C(++) library that uses Kodis C(++) API. There are several types of addons, implementing different functions inside kodi, e.g. PVR, Visualizations, Screensavers, VFS(Virtual File System)..
There is no "classic" binary addon, just different specializations.

An addon can have it's own external dependencies, independent of kodi itself.
Everything inside a binary addon source code must be human readable, including all dependencies. This is also true for python addons.
Kodi's binary addon build system handles building such external dependencies from source, so you would link against libtorrent, not include it's source code directly.


Take a look at the API docs:
https://alwinesch.github.io/group__cpp__...__vfs.html
https://alwinesch.github.io/group__cpp__..._defs.html

Example: https://github.com/xbmc/vfs.rar/blob/Mat.../RarFile.h

And here is an example how to add external library dependencies: https://github.com/xbmc/vfs.rar/tree/Mat...on/tinyxml
Since libtorrent uses cmake already, simply suppling the URL to the source tarball could be enough.

Thank you for the additional details and links.
I think I have enough information to start now even though I am not very skilled at writing C++ code so it may take a while. If anybody is interested to help, please contact me.
I will write back here when I made some progress (or if I'm stucked...).

I managed to implement a very first version of the binary add-on here: https://framagit.org/thombet/vfs.libtorrent
It is far from being ready because for now it supports only MacOS and simply downloads torrent files (not even in background) but it is a start.

I post back here because I am blocked and I ask for your help please.

Currently the torrent is downloaded in the Open() method that is to say when this code is used in a python add-on:

python:
xbmcvfs.File("torrent://<path-to-the-torrent-file>")

I plan to make the download folder configurable in the binary add-on so I want to make the path of the downloaded file available in python.
I thought to use the Read() method (because this method is not used since the download occurs in Open()) but python complains about the type of the data.
It is probably a problem of C++ cast but I'm not that good at C++...

I tried many things but I ended up using this very simple code in the Read() method of the binary add-on for testing:

cpp:
ssize_t Read(void* context, void* buffer, size_t uiBufSize) override

{

  ssize_t read = 99;

  buffer = static_cast<void*>(new std:: string("my test string"));

  return read;

} 

(link in the repo)

Then in a python video add-on I did:

python:


f = xbmcvfs.File("torrent://...")

data = f.read()

xbmc.log("read data => {}".format(data), xbmc.LOGDEBUG))

which throws the following error:


Can you please help me understand what I'm doing wrong?
Thanks!

At which point exactly does this error occur?

The exception is thrown when xbmc.log() is called. f.read() does not generate any error (but does not return the expected data).

I don’t know if it is relevant but I forgot to mention that I’m still using Kodi 18.9

Could you try: xbmc.log("read data => {}".format(repr(data)), xbmc.LOGDEBUG))

Thanks!
With repr(data) it does not fail anymore and it shows:

When I try xbmcvfs.File("file.json").read() on a standard JSON file (which does not use my binary add-on) it correctly shows the content of the file in Kodi's log without using repr() so I think there should be a way to reach the same result coding it properly in C++ I guess but I can't find the solution

First of all, I need to note that I'm a Python dev and my knowledge of C/C++ is limited, so I'm not sure about all the details but my explanation seems reasonable to me :)

When you do a static cast of your std::string object to *void in your binary addon you are returning a raw pointer to a memory area where your std::string is located. And this memory are contanis not only your text but all the bells and whistles that std::string consist of, including so called NULL bytes ('\x00') that are used as end-of-string terminators in C-strings. And when you do read() from Python you are receiving the contents of this memory area as a Python 2 str object that holds a sequence of bytes (not a text as some of inexperienced Python developers may believe).
Now let's move to xbmc.log() function. In Python 2-based Kodi API (prior to v.19) this function accepts text a UTF-8 encoded byte string that is a Python 2 str object. And such string cannot include NULL bytes '\x00'. Hence TypeError.
And the build-in repr() function converts any Python object into its "safe" textual representation so now xbmc.log() accepts it.
Botton line: you C Read() function can return raw bytes (that are then returned by xbmcvfs.File.read() method) but you cannot pass those raw bytes to xbmc.log() directly.

Note that in Python 3-based Kodi API xbmcvfs.File.read() returns a textual string by implicitly decoding the underlying raw bytes using UTF-8 encoding so in Python 3 this function can only read UTF-8-encoded textual files. For arbitrary byte sequences you need to use .readBytes() method: https://romanvm.github.io/Kodistubs/_aut....readBytes

Your explanations seemed reasonable to me too and they set me on the right track to find the solution: thanks again Roman!

For future reference one solution is:

cpp:


ssize_t Read(void* context, void* buffer, size_t uiBufSize) override

{

    std:tring my_string = "my test string";

    std::memcpy(buffer, my_string.c_str(), my_string.length());

    return my_string.length();

}



int64_t GetLength(void* context) override

{

    std:tring my_string = "my test string";

    int64_t length = my_string.length();

    return length;

}

   

In a nutshell the problems were:

• the size of the string returned by xbmcvfs.File(...).read() is defined by the returned value of the C method GetLength(). I was returning a dummy value in this method which explains why there were some extra bytes which python complained about (including null bytes). For instance changing return length; to return 1; in the code above will make the string returned by xbmcvfs.File(...).read() equal to "m".
  
• assigning the void* buffer to another pointer was not working as expected: I think it was assigning the pointer somehow to a local/temporary value. Using memcpy copies the data in the out pointer correctly.
  
• std:: string seems to be a rather useful C++ class which comes with many methods but it is not possible to use the reference of a std:: string directly because as you said the data at this address contains all the bells and whistles that make the class. This class provides the method c_str() which returns an equivalent char* which is actually only the string and can be used in memcpy.
  

I have another question about VFS binary add-ons (which is not at all related to the previous post): in the addon.xml of my binary add-on I set the attribute extensions to .torrent (link).
I thought it would make Kodi call the API of the binary add-on when a .torrent file is selected from Kodi's file manager but it does not.
Instead it seems Kodi tries to play the file because I get this in the log:

Is it possible to tell Kodi to call the binary add-on from the file manager?