[arpitn30]

I wanted to make a separate post for the proposal

ADD PYTHON3 SUPPORT

Name: Arpit Nandwani
Email: [email protected]
GitHub: https://github.com/arpitn30
LinkedIn: https://www.linkedin.com/in/arpitnandwani/

PERSONAL INFO
I’m a prefinal year undergraduate student pursuing B.Tech from Bharati Vidyapeeth’s College Of Engineering. I’m majoring in Computer Science. I reside in New Delhi, India (GMT +5:30).I started to code in C++ during my school and because of my interest in programming chose it for higher studies as well. During my sophomore year, I completed CS50 and was able to apply my knowledge of C and PHP to real world problems through its projects. I moved to Python due to its dynamic nature, easier syntax and implementation. Moreover, through use of Django and Flask, much more robust websites can be made. I’m also familiar databases like PostgreSQL and MySQL, cloud services like Azure and AWS and a bit of machine learning.

INTRODUCTION
I’ve been using Kodi Media Centre for over 4 months now for my media centre needs in Linux. I’ve contributed a bit in Open Source. I got to work on Python, Flask, PostgreSQL and Redis Server with them. Moreover, even though I knew the basics about git, I was able to learn advance git functions like rebasing and squashing. I’ve worked on projects like creating a brick breaker game or a deleted photos recovery software in C, implemented some ciphers, tries and programmed a basic server. In Python, my projects include mini scripts like a spider to crawl and search keywords on a website or a program to fetch episode titles from Wikipedia and rename them locally. I’ve also built some websites in PHP and Javascript as part of my own projects or internships. One of them is hosted on http://pocketlibrary.azurewebsites.net . Me and my team has also been selected for Smart India Hackathon organised by Govt Of India and we’ll be using React Native and Python to build an IOT embedded system. All of my work can be found on GitHub.
I usually work on Windows 10 or Linux like Ubuntu (XFCE and KDE), Antergos or Deepin depending upon the dependencies of the project. I’ve got a decent computer with a Intel Core I5, Gen 4 processor, 8GB DDR3 RAM and a dedicated 2GB NVIDIA 820m graphic card. I and am able to run multiple Virtual Machines simultaneously. For editor or compiler needs, I usually prefer Atom, Kate, Visual Studio or PyCharm.


PROJECT DETAILS
Kodi’s main code base is in C++ though it contains custom libraries for the development of add-ons in Python. Kodi currently supports Python version 2.6 and 2.7. I wish to add Python3 support to Kodi codebase so that Third Party Developers are able to code in both Python 2 and Python 3.

PHASES
• PHASE 1 – Update the C++ files with Python.h header files and the SWIG binaries to support Python 3.5
• PHASE 2 – Testing all Kodi built-in modules to check compatibility and documentation.
• PHASE 3 – Porting some of Kodi's main addons to Python 3

TIMELINE
May 05 – May 15 : Community Bonding, understanding the code base and listing files that need to be changed
May 16 – May 31 : Busy with college semester exams
June 01 – June 14 : Changing the code to add compatibility for Python 2 (2.6 and 2.7) and Python 3 (3.5 and possibly 3.6)
June 15 – June 27 : Updating the custom SWIG bindings and adding the functionality to choose version based on the individual xml files in the addons.
June 27 – June 30 : First Evaluation
July 01 – July 12 : Developing a new addon and testing the Python 3 compatibility
July 13 – July 25 : Documentation, writing tests and testing code coverage
July 26 – July 28 : Second Evaluation
July 29 – Aug 14 : Porting Kodi addons to Python 3
Aug 15 – Aug 21 : Bug fixes and code clean up
Aug 21 – Aug 29 : Submission of code and final evaluation

HOW WILL IT AFFECT KODI
This project will help the third-party developers develop their plugins and scripts in Python 3 and the programs dependent on xbmc libraries to port to python 3 as well. Moreover, I wish to continue contributing to Kodi even after the summer of code and make it a complete solution for home entertainment. I wish to work on the kodi chromium browser sometime in the future too but given the amount of ideas, I feel it’s too big of a project for the summer.

I don't have any prior engagements in summer and can dedicate my entire time to the development of this project. I will be working 7 hours a day for 6 days a week so I'll be available for communications most of the time. I've already gone through some of the code and successfully did a test build on Windows. I might need a little time to understand the working of SWIG bindings but it's nothing I can't handle.

MENTORS

[Razze]

Hey there, nice to meet you! Quiet some nice projects there and exciting proposal

• PHASE 1 – Convert all the Python 2 files to support both Python 2 and Python 3 USING Python Future Modules

Can you go a bit more into details here? I'm not really getting this.

If I would be doing this, I would probably start with your PHASE 2 and maybe do a test python 3 addon and try to get that to run.

Then expand testing to all the interfaces to python kodi offers. Then if there's time left it might be nice to move some of Team Kodis addons over to python 3, but it seems more like a plus point then a PHASE 1.

[arpitn30]

I was referring to the Python Libraries in the system/python/Lib directory. Some of the files support Python 3 functions but most of them are written in legacy Python 2 code.

[Rechi]

I was referring to the Python Libraries in the system/python/Lib directory. Some of the files support Python 3 functions but most of them are written in legacy Python 2 code.

You see those files only on Windows, if you mean the same directory I'm thinking about.
You won't need to changed them. These files are from https://github.com/python/cpython/tree/2.7/Lib and if you switch to Python3 these files should work with it as they are part of the official python code.
I'm not sure, if compiling Python3 for windows should be done by yourself, because compiling libraries for windows is a bit special.

[arpitn30]

Then expand testing to all the interfaces to python kodi offers.

Can you please explain this a bit in detail.

[Rechi]

I think Razze means the interfaces listed at http://mirrors.kodi.tv/docs/python-docs/ which are used by addons to add listings and such things.

[Roman_V_M]

I was referring to the Python Libraries in the system/python/Lib directory. Some of the files support Python 3 functions but most of them are written in legacy Python 2 code.

You don't need to touch that. This is the Python Standard Library that comes with every CPython interpreter. Each CPython interpreter has its own. If you want to include CPython 3 interpreter in Kodi, you need to use the Standard Library that is bundled with sources.

I'm not a C++ developer and have a limited experience with Python/C++ interoperability but as I understand the problem consists of 2 parts:

- We need to compile CPython 3 for Kodi. I know that Python in Kodi has its own specifics, like removed stdin and stdout/stderr redirected to the Kodi log, and I'm sure there's probably more. BTW, which minor Python 3 version do you plan to use?

- We need to port wrappers for Kodi Python API (xbmc* modules) so that it correctly works in Python 3 (str/bytes differences and such).

I don't know if it's possible to have both Python 2 and 3 (at least for some time to allow smooth transition of Python addons), but if it is, then some work will be needed to implement correct support of both versions.

BTW, although automated tools for Python 2 > 3 conversion exist, it is still mainly a manual process. General purpose Python libraries converted into Kodi addons (script.module.html5lib, script.module.requests and such) are mostly written in Python 2/3 compatible code, but most addons need to be converted to Python 3 by their developers/maintainers.

[arpitn30]

Proposal Updated

[Razze]

Good work. Your proposal looks very promissing.

[arpitn30]

I read somewhere on this forum that there were some problems with using SWIG directly so the wrapper was instead coded with an implementation similar to SWIG. I was wondering what were the problems being faced which led to the decision against using SWIG directly

[da-anda]

I think only jfcarrol (who did all the SWIG stuff) might be able to answer that. Unfortunately he hasn't be around much in the last year(s), but I try to ping him

[Montellese]

I read somewhere on this forum that there were some problems with using SWIG directly so the wrapper was instead coded with an implementation similar to SWIG. I was wondering what were the problems being faced which led to the decision against using SWIG directly

See https://sourceforge.net/p/swig/bugs/1114/ for the bug that Jim reported to the Python SWIG guys. Unfortunately there was nobody capable of looking into it.

[arpitn30]

Thanks a lot, I'll check it out

[Razze]

The pull request can be found here: https://github.com/xbmc/xbmc/pull/12705 but it ended up only beeing python3

[arpitn30]

Ported To Python3

Google Summer of Code 2017
Student: Arpit Nandwani

Link To Pull Request - https://github.com/xbmc/xbmc/pull/12705

Motivation
It's been almost 10 years since the Python 3 was introduced. As of now, the EOL date for python2 is 2020. Adding Python 3 support for Kodi has been in discussions for a long time so it was imperative to do so sooner or later. It was essential to add the support sooner as it would give developers more time to port their add-ons.

Goals
My original goal was to add support for Python 3 while still keeping support for Python 2 by building 2 versions for the xbmc modules. I planned to do it without introducing any major changes so as to make the later porting to Python 3 easier. The goal was later shifted to porting completely to Python 3 since supporting both versions didn't seem possible without changing a large chunk of the code and which would eventually have to be changed back. The progress was according to the schedule mostly, but instead of porting add-ons and documentation changes in the last month, I tried to run Python3 Kodi on all the other platforms as well.

Change Log

• PyString has been changed to PyUnicode and PyBytes accordingly.
  
• PyInt has been changed to PyLong since former is no longer supported.
  
• Obsolete Py_TPFLAGS have been removed or updated
  
• Updated ob_type to be called by multi layered PyObject
  
• tp_Compare now depreciated has been completely replaced in favor of tp_RichCompare
  
• Module Initialization has been updated
  • Py_InitModule has been replaced with PyModule_Create
    
  • PyModuleDef added
    
  • Added PyImport_AppendInittab to add modules to the import modules list
    
  • Updated swig/CMakeLists to add Python Binding directly to Core Library
    
• Added and Removed PyINCREF and PyDECREF statements
  
• Changed charset from char to wchar_t due to new functions using the latter
  
• Replaced PyFile_AsFile, now removed with a custom function
  
• Updated GIL acquisition and release statements
  
• Updated Windows package lists and directory paths to new dependency package structure
  
• Updated Python Package and Library versions
  
• Updated Posix native files and added patches wherever necessary
  
• Removed outdated and added new patches to target Python directory
  
• Updated Makefiles, dependencies and added patches for specific platforms
  

Code
All my commits are on my fork of Kodi. There might be some improvements and bug fixes needed before releasing the final version on all platforms.
I initially did my work on the Python3 branch which contains all the templates in xbmc/interface ported to Python3. From there, I divided it into 2 branches, one to test Kodi with just Python3 and one to support both Python 2 and 3 though there isn't much work done on that.

All of my commits to completely port Kodi to Python3 for my GSoC Work Product Submission can be found in this Pull Request until the commit Added Python Version Varaiable. Though Linux-Wayland still need to fixed for Python3

This pull request against the feature_python3 branch had 69 Files Changed, 825 Line Additions and 728 Line Deletions.

Development process
Since Python is dynamically generated in Kodi from C++, there was no straight forward way to know what changes needed to be made in the code. Though there were documentations available online to port from Python-C API from Python 2 to Python 3, none of them were very comprehensive and mostly focussed on function name changes. Most of the changes in the templates involved just a bit of implementation change but the major change was deciding whether a function using PyString returned Unicode string or Byte string. Most of the changes in the templates were made in the first few weeks but in order to test if the changes even worked, I had to update the dependencies on any one platform, so I started with Windows. The updated package structure gave me a bit of the trouble but after I was familiar with the CMake structure, it resolved quickly.

After Kodi was working on Windows with the changes, I had to run an add-on to check if the changes actually worked. And on the first try, they didn't, obviously.
One of the initial problems I faced was with the implementation of threads in Python C API. Python is not really multithreaded, it has a Global Interpreter Lock which needs to be acquired to perform any Python operation. And each combination of thread and sub interpreter need to have its own thread state. In Python3 with the depreciation of PyEval_AcquireLock and PyEval_ReleaseLock, the GIL is never really released and ThreadState needs to be saved every time the GIL is relinquished to some other thread and then it can be reacquired using PyEval_RestoreThread. The solution was to pass thread state between the functions releasing and acquiring the locks.
After this problem was fixed, another problem I faced was due to the changed implementation of Module Initialization. initModule which was replaced by PyInit_Module didn't add the initialized modules to the import list and had to be explicitly added at the time of execution. Moreover, some functions which returned a Borrowed Reference in Python2 returned a New Reference in Python3 and vice versa. So, all the reference had to be counted again and Py_INCREF and Py_DECREF added and removed accordingly. Other than that due to PyFile function removed, I had to find a workaround for it since native C file functions could not be used in Kodi due to linking problems with Windows

After most of the bugs were fixed and add-ons written in Python3 were correctly working on windows. I added support for the cross compile build during the last month, by updating the Makefiles, native and target dependencies, and repatching the files in Python3 downloaded source code.

During past 3 months, I was able to learn a lot about how Python is implemented on a lower level and how CMake and Makefiles are using to structure and build large projects. Finding out the bugs and knowing what needs to be changed took a lot of the time as compared to the time that went to write the code.

Acknowledgement
I'd like to thank my mentor Razze for always monitoring my progress, immediately responding and guiding me whenever I needed help. I'm also grateful to paxxi, wsnipex, romanvm and especially rechi, I know I asked him a lot of doubts and he helped me every time. I owe my wholehearted thanks and appreciation to the entire team of Kodi for selecting me for this project and for their assistance during the entire project work.