Support for concurrent.futures in kodi addons - Printable Version +- Kodi Community Forum (https://forum.kodi.tv) +-- Forum: Development (https://forum.kodi.tv/forumdisplay.php?fid=32) +--- Forum: Scrapers (https://forum.kodi.tv/forumdisplay.php?fid=60) +--- Thread: Support for concurrent.futures in kodi addons (/showthread.php?tid=349780) |
Support for concurrent.futures in kodi addons - BoeboeBE - 2019-12-04 Hi all, I have written a parsing library to scrape CGTN video and meta data. Now I am creating the Kodi add-on to make it available as a video add-on in Kodi. The problem is that Kodi does not seem to support the concurrent.futures module (available for both Python2 and Python3).
My scraping library is available at https://github.com/boeboe/cgtn-videos The reason I introduced concurrency is to decrease parsing time (sometimes I need to make an extra HTTP call per list item scraped to get all the data needed/wanted). This resulted scraping of some topics from 60sec to 5sec, which was exactly the performance boost needed. An example:
Any other best practices w.r.t concurrent/parallel scraping? PS: I also had to refactor my scraping library to avoid Enum, as this was also not supported, although enum is available on Python 2 and 3 and back-ported to 3.3, 3.2, 3.1, 2.7, 2.6, 2.5, and 2.4. Best regards, Boeboe RE: Support for concurrent.futures in kodi addons - BoeboeBE - 2019-12-04 Nevermind all, I have grep'ed a bit in my Kodi installation and found the following:
Keep this thread for google indexing purposes, as I could not find anything on this on the forum nor google. Best regards, Bart RE: Support for concurrent.futures in kodi addons - User 453898 - 2020-08-07 (2019-12-04, 06:37)BoeboeBE Wrote: Just noticing your post. Curious if you have looked at updating this script to python 3 capable for upcoming Kodi 19 Matrix. It will solve some problems for me later as well, as I have. a few admins that use this script. |