Support for concurrent.futures in kodi addons
#1
Hi all,

I have written a parsing library to scrape CGTN video and meta data. Now I am creating the Kodi add-on to make it available as a video add-on in Kodi.

The problem is that Kodi does not seem to support the concurrent.futures module (available for both Python2 and Python3).
  • Does anyone know whether this is available as a script module?
  • Is there an alternative if not available?

My scraping library is available at https://github.com/boeboe/cgtn-videos

The reason I introduced concurrency is to decrease parsing time (sometimes I need to make an extra HTTP call per list item scraped to get all the data needed/wanted). This resulted scraping of some topics from 60sec to 5sec, which was exactly the performance boost needed. An example:



def __process_m3u8_links(videos):
    """Helper methods to check if m3u8 links are valid and fix if not the case """
    result_videos = []
    with concurrent.futures.ThreadPoolExecutor(max_workers=len(videos)) as executor:
        future_to_video_m3u8 = {
        executor.submit(NewsParser.__check_m3u8_link, video): video
        for video in videos
        }
    for future in concurrent.futures.as_completed(future_to_video_m3u8):
        result_videos.append(future.result())
    return result_videos


Any other best practices w.r.t concurrent/parallel scraping?

PS: I also had to refactor my scraping library to avoid Enum, as this was also not supported, although enum is available on Python 2 and 3 and back-ported to 3.3, 3.2, 3.1, 2.7, 2.6, 2.5, and 2.4.

Best regards,
Boeboe
Reply
#2
Nevermind all,

I have grep'ed a bit in my Kodi installation and found the following:


<import addon="script.module.futures" version="2.2.0" />

Keep this thread for google indexing purposes, as I could not find anything on this on the forum nor google.

Best regards,
Bart
Reply
 
Thread Rating:
  • 0 Vote(s) - 0 Average



Logout Mark Read Team Forum Stats Members Help
Support for concurrent.futures in kodi addons00
This forum uses Lukasz Tkacz MyBB addons.