[arpitn30]

Ported To Python3

Google Summer of Code 2017
Student: Arpit Nandwani

Link To Pull Request - https://github.com/xbmc/xbmc/pull/12705

Motivation
It's been almost 10 years since the Python 3 was introduced. As of now, the EOL date for python2 is 2020. Adding Python 3 support for Kodi has been in discussions for a long time so it was imperative to do so sooner or later. It was essential to add the support sooner as it would give developers more time to port their add-ons.

Goals
My original goal was to add support for Python 3 while still keeping support for Python 2 by building 2 versions for the xbmc modules. I planned to do it without introducing any major changes so as to make the later porting to Python 3 easier. The goal was later shifted to porting completely to Python 3 since supporting both versions didn't seem possible without changing a large chunk of the code and which would eventually have to be changed back. The progress was according to the schedule mostly, but instead of porting add-ons and documentation changes in the last month, I tried to run Python3 Kodi on all the other platforms as well.

Change Log

• PyString has been changed to PyUnicode and PyBytes accordingly.
  
• PyInt has been changed to PyLong since former is no longer supported.
  
• Obsolete Py_TPFLAGS have been removed or updated
  
• Updated ob_type to be called by multi layered PyObject
  
• tp_Compare now depreciated has been completely replaced in favor of tp_RichCompare
  
• Module Initialization has been updated
  • Py_InitModule has been replaced with PyModule_Create
    
  • PyModuleDef added
    
  • Added PyImport_AppendInittab to add modules to the import modules list
    
  • Updated swig/CMakeLists to add Python Binding directly to Core Library
    
• Added and Removed PyINCREF and PyDECREF statements
  
• Changed charset from char to wchar_t due to new functions using the latter
  
• Replaced PyFile_AsFile, now removed with a custom function
  
• Updated GIL acquisition and release statements
  
• Updated Windows package lists and directory paths to new dependency package structure
  
• Updated Python Package and Library versions
  
• Updated Posix native files and added patches wherever necessary
  
• Removed outdated and added new patches to target Python directory
  
• Updated Makefiles, dependencies and added patches for specific platforms
  

Code
All my commits are on my fork of Kodi. There might be some improvements and bug fixes needed before releasing the final version on all platforms.
I initially did my work on the Python3 branch which contains all the templates in xbmc/interface ported to Python3. From there, I divided it into 2 branches, one to test Kodi with just Python3 and one to support both Python 2 and 3 though there isn't much work done on that.

All of my commits to completely port Kodi to Python3 for my GSoC Work Product Submission can be found in this Pull Request until the commit Added Python Version Varaiable. Though Linux-Wayland still need to fixed for Python3

This pull request against the feature_python3 branch had 69 Files Changed, 825 Line Additions and 728 Line Deletions.

Development process
Since Python is dynamically generated in Kodi from C++, there was no straight forward way to know what changes needed to be made in the code. Though there were documentations available online to port from Python-C API from Python 2 to Python 3, none of them were very comprehensive and mostly focussed on function name changes. Most of the changes in the templates involved just a bit of implementation change but the major change was deciding whether a function using PyString returned Unicode string or Byte string. Most of the changes in the templates were made in the first few weeks but in order to test if the changes even worked, I had to update the dependencies on any one platform, so I started with Windows. The updated package structure gave me a bit of the trouble but after I was familiar with the CMake structure, it resolved quickly.

After Kodi was working on Windows with the changes, I had to run an add-on to check if the changes actually worked. And on the first try, they didn't, obviously.
One of the initial problems I faced was with the implementation of threads in Python C API. Python is not really multithreaded, it has a Global Interpreter Lock which needs to be acquired to perform any Python operation. And each combination of thread and sub interpreter need to have its own thread state. In Python3 with the depreciation of PyEval_AcquireLock and PyEval_ReleaseLock, the GIL is never really released and ThreadState needs to be saved every time the GIL is relinquished to some other thread and then it can be reacquired using PyEval_RestoreThread. The solution was to pass thread state between the functions releasing and acquiring the locks.
After this problem was fixed, another problem I faced was due to the changed implementation of Module Initialization. initModule which was replaced by PyInit_Module didn't add the initialized modules to the import list and had to be explicitly added at the time of execution. Moreover, some functions which returned a Borrowed Reference in Python2 returned a New Reference in Python3 and vice versa. So, all the reference had to be counted again and Py_INCREF and Py_DECREF added and removed accordingly. Other than that due to PyFile function removed, I had to find a workaround for it since native C file functions could not be used in Kodi due to linking problems with Windows

After most of the bugs were fixed and add-ons written in Python3 were correctly working on windows. I added support for the cross compile build during the last month, by updating the Makefiles, native and target dependencies, and repatching the files in Python3 downloaded source code.

During past 3 months, I was able to learn a lot about how Python is implemented on a lower level and how CMake and Makefiles are using to structure and build large projects. Finding out the bugs and knowing what needs to be changed took a lot of the time as compared to the time that went to write the code.

Acknowledgement
I'd like to thank my mentor Razze for always monitoring my progress, immediately responding and guiding me whenever I needed help. I'm also grateful to paxxi, wsnipex, romanvm and especially rechi, I know I asked him a lot of doubts and he helped me every time. I owe my wholehearted thanks and appreciation to the entire team of Kodi for selecting me for this project and for their assistance during the entire project work.