GSoC'17 Proposal :- Adding Python 3 support

  Thread Rating:
  • 2 Vote(s) - 3 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
rohit-rk Offline
Junior Member
Posts: 14
Joined: Mar 2017
Reputation: 0
Post: #1
Hi,

Introduction:
My name is Roht kumar and I am an undergraduate student in Computer Science program(2nd year) at Dr. B. R. Ambedkar National Institute of Technology Jalandhar, India.I am very much interested to work on "Add Python3 Support" and “Web Browser in Kodi” projects. I joined this open source adventure from january of this year itself, it is fun as well as I am learning new things everyday. My github at https://github.com/rohit-rk. I am comfortable working with C, C++, Python, Mysql, Mongodb, Oracle 10g(Working on it as part of curriculum), prefer to work on both Linux and windows. I am very good at understanding others code, which i think makes me a good candidate.

As I mentioned two projects, but I am specifically interested in working on adding Python 3 support. The reason is strange, but makes me feel confident. When i started with python, I was confused like which version to choose. I did lot of research then came up to a point of learning both. So first i, started with python 3.5 and then python 2.7. Working on version 3 was cooler and easier. Just to mention, Python 3 removing support of u'' strings is great help as it is headache to me(personal views). I came to know that many major packages have been ported to python3 like: NumPy, SciPy, Django, Flask, PIL to Pillow, ,py2exe ,etc. Still, many prefer to use Python2. With Python 3 being the future of Python while Python 2 in active use, I think having support for both would be a good idea.
Even the python communites support for both. There is this six compatibility package, a key utility for supporting Python 2 and Python 3 in a single code for small projects. But working for an organization and a huge code base will be a difficult, interesting task to tackle as well as i’ll get to learn a lot on the way, which i like the most.

Query:
I am writing to discuss the details and ask for advice on this. Could you please let me know more info about this project and repositories as i am new to KODI ?. Specifically, what are the requirements? Also, could you please let me know how I should start to get involved and contribute?

I am aware of the fact that it is a little bit too late to discuss this but I just learned that KODI is selected for Google Summer of Code and found a perfect project for my skillset. Well, this is about getting involved and contributing kind’a query, will send proposal soon.
Thanks in advance for your help.

Rohit
(This post was last modified: 2017-04-01 22:57 by rohit-rk.)
find quote
RockerC Offline
Posting Freak
Posts: 1,515
Joined: May 2011
Reputation: 31
Post: #2
@Rohit, read all the suggestions already recommended to other students in this thread as well recommendations posted in other threads. Specifically submit seperate proposals for each feature to have less chance conflicting with proposals from other students:

http://forum.kodi.tv/showthread.php?tid=309218

Example: If two students both only submit one proposal each for Python 3 support then only one will get a student slot for GSoC. But if the same two students both only two proposals each for both Python 3 and Web Engine then both students could get a slot.
(This post was last modified: 2017-03-15 10:32 by RockerC.)
find quote
RockerC Offline
Posting Freak
Posts: 1,515
Joined: May 2011
Reputation: 31
Post: #3
(2017-03-14 18:07)rohit-rk Wrote:  With Python 3 being the future of Python while Python 2 in active use, I think having support for both would be a good idea.

Even the python communites support for both. There is this six compatibility package, a key utility for supporting Python 2 and Python 3 in a single code for small projects.
For dicussion about Python3 maybe start by also reading discussions in these two threads:

http://forum.kodi.tv/showthread.php?tid=205204

http://forum.kodi.tv/showthread.php?tid=216598

Sounds as if the "six compatibility package" is no longer the best option for supporting both Python 2 and Python 3 in Kodi.

http://forum.kodi.tv/showthread.php?tid=...pid2478984
(2016-12-15 01:56)Hedda Wrote:  Don't know about Kodi's core interface to Python, but as plain Python scripting language goes it should today be much smoother to upgrade to Python 3 and stay backwards compatible with Python 2 as there is now a new compatibility package that should make running Python 2 scripts under Python 3 easier.

Checkout: http://python-future.org/overview.html
Quote:python-future is the missing compatibility layer between Python 2 and Python 3. It allows you to use a single, clean Python 3.x-compatible codebase to support both Python 2 and Python 3 with minimal overhead.

It provides future and past packages with backports and forward ports of features from Python 3 and 2. It also comes with futurize and pasteurize, customized 2to3-based scripts that helps you to convert either Py2 or Py3 code easily to support both Python 2 and 3 in a single clean Py3-style codebase, module by module.

Notable projects that use python-future for Python 2/3 compatibility are Mezzanine and ObsPy.
As before it was adviced that if you upgraded to Python 3.x interpreter then you could still keep some backwards compatibility with Python 2.7 if you use the "six compatibility package" (for reference see http://packages.python.org/six/ ).

Now however the Python interpreter developers have made a new compatibility package that is called "python-future" (and it is also refered to as "future compatibility package"), so please see:

http://python-future.org/index.html ( and the actual package on GitHub https://github.com/PythonCharmers/python-future )

This "future compatibility package" is still in beta and doesn't support as many versions of Python as the six (it only goes back as far as Python 2.6, while six supports Python 2.4), but allows Python 2 compatible code to be written in a style that is closer to idiomatic Python 3 (for example, it includes an actual Python 2 compatible implementation of the Python 3 bytes type, rather than relying on the Python 2.x 8-bit string type that exposes a slightly different API).

Recommmed that you read both https://wiki.python.org/moin/Python2orPython3 and http://python-future.org/index.html

It suggest that much modern Python code should run largely unmodified on Python 3 using this "future compatibility package".
find quote
rohit-rk Offline
Junior Member
Posts: 14
Joined: Mar 2017
Reputation: 0
Post: #4
Thanks a lot. Checked all those links and decided to focus only on adding python3 support and fully understanding the requirements, what is to be done and making a good proposal.
I have read the threads and forked https://github.com/FernetMenta/kodi-agile . I am little confused on what is to be done and on which repo from reading recommendations from thread.

(2017-03-12 12:57)da-anda Wrote:  as for Python3 - there is currently some work done to improve our interfaces and ease the way to integrate new languages. This work is done by AlwinEsch/alwinus in the Kodi-agile fork https://github.com/FernetMenta/kodi-agile and will be merged to mainline once finished. So if you're interested in this, it might be good to check out this work instead of current master.


(2017-03-15 01:24)Razze Wrote:  
(2017-03-14 06:41)arpitn30 Wrote:  
(2017-03-12 12:57)da-anda Wrote:  as for Python3 - there is currently some work done to improve our interfaces and ease the way to integrate new languages.

I'm a bit confused. Does the entire code base including system files, add-ons and the libraries for development has to be changed to be compatible with both Python 2.7 and Python 3.6 or we have to add the python3 support to the KODI api for script and plugin development. For the latter, the API will have to be shipped with both Python 2.7 and 3.6 interpreter as well as updated xmbc modules

You will basically have to look at SWIG and how to generate a SWIG binding for python 3 additionally to the one for python 2 that we have. Then bind that to the language binaries somehow. Have something to control which language loads an addon (addon.xml probably). Make kodi able to use that and maybe an addon to proof that it works with python 3.

You can have a look at this untested branch on how the SWIG stuff might look https://github.com/Razzeee/xbmc/tree/python3-swig


Point to some modules in https://github.com/FernetMenta/kodi-agile (for check n work on ) which will be helpful for me to understand and start working on something before time starts and also to make a good proposal.
(This post was last modified: 2017-03-15 15:18 by rohit-rk.)
find quote
Montellese Offline
Team Kodi Developer
Posts: 4,842
Joined: Jan 2009
Reputation: 72
Location: Switzerland
Post: #5
Here are some of the files related to the Python API in Kodi:
You'll most likely have to implement a new interpreter which uses Python 3 instead of Python 2.

Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
find quote
rohit-rk Offline
Junior Member
Posts: 14
Joined: Mar 2017
Reputation: 0
Post: #6
Thanks a lot. I'll check them.
find quote
rohit-rk Offline
Junior Member
Posts: 14
Joined: Mar 2017
Reputation: 0
Thumbs Up  RE: GSoC'17 : Adding Python3 support project Query.
Post: #7
(2017-03-15 19:39)Montellese Wrote:  You'll most likely have to implement a new interpreter which uses Python 3 instead of Python 2.
Can you provide some details on this,
Are we completely porting to python 3 or need compatibility for both , ?
(This post was last modified: 2017-03-17 03:45 by rohit-rk.)
find quote
rohit-rk Offline
Junior Member
Posts: 14
Joined: Mar 2017
Reputation: 0
Photo  RE: GSoC'17 : Adding Python3 support project Query.
Post: #8
Will it be a good idea to approach like this, first make all python files compatible with both python2 and python3 using six, future , etc compatible code and then (Python.h included) c++ files accordingly (If need arises) .Also changing the Swig ,interface files cause there is a list of known-to-be-broken features in Python 3:
No more support for FILE* typemaps, because PyFile_AsFile has been dropped in Python 3.
The -apply command line option is removed and generating code using apply() is no longer supported. Need to change these if used.
.But even so I'm afraid we'll have a lot of #if PY_VERSION_MAJOR >= 3 everywhere. Please guide,, as me being new to kodi may get those things wrong.
Write now i am reading Swig-Python documentation to get proper understanding.
find quote
Montellese Offline
Team Kodi Developer
Posts: 4,842
Joined: Jan 2009
Reputation: 72
Location: Switzerland
Post: #9
Well I'm not really sure what the general idea is since I'm not a python coder. My guess is that we want to keep current add-ons running without having to change them but also support python 3 addons. But somebody else should confirm this assumption, don't take my word for it.

Reading the Swig-Python documentation might not help you much because we have our own transformer/mapper. The Swig-Python mapper showed to have a few really bad bugs which we couldn't work around and nobody at Swig was able to look into it. That's why we had to write our own logic (based on general swig and groovy).

One way forward might be to use the same (generic) C++ code to generate both python 2 and python 3 bindings and add a python 3 interpreter (in addition to the python 2 one) and run add-on scripts either in the python 2 or python 3 interpreter. But I've never thought about it in detail and I know too little about the differences in python 2 and 3.

Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
find quote
Roman_V_M Offline
Team-Kodi Member
Posts: 1,140
Joined: Jun 2011
Reputation: 56
Location: Kyiv, Ukraine
Post: #10
Having both Python 2 and 3 interpreters (if it's possible at all) will be good because addons have been developed over the years by many authors, many of whom are not professional coders, so it's hard to expect them all to port their addons to Py3 or refactor to universal code. If we replace the interpreter, we will inevitably loose some addons because of compatibility issues.

The Standard Library aside, the main difference between Py2 and Py3 is clear separation between textual and byte strings. Currently, Kodi API functions that take textual data accept both str and unicode Py2 types. AFAIU unicode is transparently encoded to utf-8 by SWIG and passed as std::string or char* to the Kodi core. In Py3 str type holds Unicode codepoints and bytes type is used for raw bytes. I thinks it would be totally OK if in Python 3 Kodi API functions would accept and return only str (that is, Unicode) data and not bytes.

Also, Kodi provides some binary Python libraries like PIL/Pillow that are compiled in place during Kodi building. Binary Python libraries must be compiled and linked against the target Python version, so I'm not sure if it is possible to have 2 versions of such libraries if we ever consider to support both Py2 and Py3.

Raspberry PI 2 + LibreELEC 8 (Kodi 17)
Samsung Galaxy Tab A 10.1 + Kodi 17 for Android
find quote
rohit-rk Offline
Junior Member
Posts: 14
Joined: Mar 2017
Reputation: 0
Post: #11
Thanks a lot Montellese for directing ,Learned how groovy ,swig based logic is implemented in Kodi.
find quote
rohit-rk Offline
Junior Member
Posts: 14
Joined: Mar 2017
Reputation: 0
Post: #12
Can we use distribute? With Distribute we can create setup.py file and add a few extra parameters in the setup.py file to have 2to3 run the conversion at build time. Also replacing python 2 with python 3 interpreter. This way Python 2 addons will be ported to python3 as well as python 3 addons will be supported. Don't know if it's possible. See this http://python3porting.com/strategies.html.
(This post was last modified: 2017-03-17 23:25 by rohit-rk.)
find quote
Roman_V_M Offline
Team-Kodi Member
Posts: 1,140
Joined: Jun 2011
Reputation: 56
Location: Kyiv, Ukraine
Post: #13
(2017-03-17 23:17)rohit-rk Wrote:  Can we use distribute? With Distribute we can create setup.py file and add a few extra parameters in the setup.py file to have 2to3 run the conversion at build time. Also replacing python 2 with python 3 interpreter. This way Python 2 addons will be ported to python3 as well as python 3 addons will be supported. Don't know if it's possible. See this http://python3porting.com/strategies.html.

Automated convertors are not reliable, except maybe for very simple cases. Converted code still needs manual polishing/debugging. Also distribute seems to be abandoned for a while.

Raspberry PI 2 + LibreELEC 8 (Kodi 17)
Samsung Galaxy Tab A 10.1 + Kodi 17 for Android
find quote
rohit-rk Offline
Junior Member
Posts: 14
Joined: Mar 2017
Reputation: 0
Post: #14
Having both interpreters will take extra space. So , may be support both with the python 2 interpreter using lib2to3, future, six comatibility. Do you have any suggestions for a good approach.
find quote
Roman_V_M Offline
Team-Kodi Member
Posts: 1,140
Joined: Jun 2011
Reputation: 56
Location: Kyiv, Ukraine
Post: #15
(2017-03-17 23:46)rohit-rk Wrote:  Having both interpreters will take extra space. So , may be support both with the python 2 interpreter using lib2to3, future, six comatibility. Do you have any suggestions for a good approach.

Unfortunately, there is no painless way, otherwise EOL for Python 2.7 wouldn't have been postponed several times by Python.org. Either we have both Py2 and Py3 (I don't know if it's possible at all, I'm no C++ coder although I know basics), or we break things and force addon developers to upgrade. Or we may leave things as is for now.

Raspberry PI 2 + LibreELEC 8 (Kodi 17)
Samsung Galaxy Tab A 10.1 + Kodi 17 for Android
(This post was last modified: 2017-03-18 00:15 by Roman_V_M.)
find quote
Post Reply