script.module.metadatautils - issues with bs4
#1
Hello all,

I am trying to migrate script.module.metadatautils to Python 3.  The current release uses BeautifulSoup and not BeautifulSoup4, so I am trying to replace that with using bs4.  I have modified one of the helper scripts, google.py like so:


#!/usr/bin/python
# -*- coding: utf-8 -*-

'''get images from google images'''

from .utils import DialogSelect, requests, log_exception
from bs4 import BeautifulSoup
import xbmc
import xbmcvfs
import xbmcgui
import sys
from simplecache import use_cache


class GoogleImages(object):
    '''get images from google images'''

    def __init__(self, simplecache=None):
        '''Initialize - optionaly provide simplecache object'''
        if not simplecache:
            from simplecache import SimpleCache
            self.cache = SimpleCache()
        else:
            self.cache = simplecache

    def search_images(self, search_query):
        '''search google images with the given query, returns list of all images found'''
        return self.get_data(search_query)

    def search_image(self, search_query, manual_select=False):
        '''
            search google images with the given query, returns first/best match
            optional parameter: manual_select (bool), will show selectdialog to allow manual select by user
        '''
        image = ""
        images_list = []
        for img in self.get_data(search_query):
            img = img.replace(" ", "%20")  # fix for spaces in url
            if xbmcvfs.exists(img):
                if not manual_select:
                    # just return the first image found (assuming that will be the best match)
                    return img
                else:
                    # manual lookup, list results and let user pick one
                    listitem = xbmcgui.ListItem(label=img, iconImage=img)
                    images_list.append(listitem)
        if manual_select and images_list:
            dialog = DialogSelect("DialogSelect.xml", "", listing=images_list, window_title="%s - Google"
                                  % xbmc.getLocalizedString(283))
            dialog.doModal()
            selected_item = dialog.result
            del dialog
            if selected_item != -1:
                selected_item = images_list[selected_item]
                if sys.version_info.major == 3:
                    image = selected_item.getLabel()
                else:
                    image = selected_item.getLabel().decode("utf-8")
        return image

    @use_cache(30)
    def get_data(self, search_query):
        '''helper method to get data from google images by scraping and parsing'''
        params = {"site": "imghp", "tbm": "isch", "tbs": "isz:l", "q": search_query}
        headers = {'User-agent': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows Phone OS 7.0; Trident/3.1; \
            IEMobile/7.0; LG; GW910)'}
        html = ''
        try:
            html = requests.get('https://www.google.com/search', headers=headers, params=params, timeout=5).text
        except Exception as exc:
            log_exception(__name__, exc)
        soup = BeautifulSoup(html)
        results = []
        for div in soup.findAll('div'):
            if div.get("id") == "images":
                for a_link in div.findAll("a"):
                    page = a_link.get("href")
                    try:
                        img = page.split("imgurl=")[-1]
                        img = img.split("&imgrefurl=")[0]
                        results.append(img)
                    except Exception:
                        pass
        return results

However, I always get the error in the log that there is no module bs4:


2019-12-22 13:12:10.880 T:19036 WARNING: Skin Helper Service --> b'Traceback (most recent call last):\n  File "C:\\Program Files\\Kodi\\addons\\script.skin.helper.service\\resources\\lib\\listitem_monitor.py", line 351, in set_listitem_details\n    details = merge_dict(details, self.metadatautils.get_top250_rating(details["imdbnumber"]))\n  File "C:\\Program Files\\Kodi\\addons\\script.module.metadatautils\\lib\\metadatautils.py", line 198, in get_top250_rating\n    return self.imdb.get_top250_rating(imdb_id)\n  File "C:\\Program Files\\Kodi\\addons\\script.module.metadatautils\\lib\\metadatautils.py", line 348, in imdb\n    from helpers.imdb import Imdb\n  File "C:\\Program Files\\Kodi\\addons\\script.module.metadatautils\\lib\\helpers\\imdb.py", line 11, in <module>\n    from bs4 import BeautifulSoup\nModuleNotFoundError: No module named \'bs4\'\n'

The error shown is for the imdb.py helper, but I am using the exact same syntax as in google.py.  I posted google.py since it is shorter.

I have checked that bs4 4.6.3.1 is installed and enabled.  Any ideas? Thanks for any help.

Regards,

Bart
Reply
#2
have you added the dependency to your addon.xml file as well?
Do not PM or e-mail Team-Kodi members directly asking for support.
Always read the Forum rules, Kodi online-manual, FAQ, Help and Search the forum before posting.
Reply
#3
(2019-12-22, 19:22)ronie Wrote: have you added the dependency to your addon.xml file as well?

@ronie, I have not. I will try that now. Thanks!

Regards,

Bart

EDIT: That was it! Thanks @ronie . Smile
Reply
#4
FYI, I submitted a PR for Python 3 compatibility of metadatautils:
https://github.com/kodi-community-addons...ls/pull/38
Reply
#5
(2019-12-24, 13:41)AnonTester Wrote: FYI, I submitted a PR for Python 3 compatibility of metadatautils:
https://github.com/kodi-community-addons...ls/pull/38

@AnonTester , thank you! I think you also need to add in that PR the change in the addon.xml to import beautifulsoup4 as well as beautifulsoup.  If not, you will get the error that there is no module bs4.

Regards,

Bart
Reply
#6
It's working without a change in addons.xml on my system. I don't know if adding a requirement for bs4 in the addons.xml would cause an issue with the backwards compatibility though?
Reply
#7
(2019-12-24, 17:36)AnonTester Wrote: It's working without a change in addons.xml on my system. I don't know if adding a requirement for bs4 in the addons.xml would cause an issue with the backwards compatibility though?
@AnonTester , under Windows I am getting the error.  I don't think adding bs4 will cause an issue, since it has been available for several Kodi versions.

Regards,

Bart
Reply
#8
ok in that case it's added to addons.xml now and pushed to the PR. Thanks for pointing this out!
Reply
#9
@AnonTester 

It's possible fix extendedinfo to python 3? 

https://github.com/phil65/script.extendedinfo

Regards,

Wanilton
MediaBrazil forum Website - Youtube Channel
Aeon MQ9-1.4.0 & Ace2 Skin Team
MarcosQui Website Donate and support us.
Reply
#10
@Wanilton
I tried getting extendedinfo to work for a while, but failed and gave up and just removed it as I'm not using it myself anyway.
Reply
#11
No prob, thanks.
MediaBrazil forum Website - Youtube Channel
Aeon MQ9-1.4.0 & Ace2 Skin Team
MarcosQui Website Donate and support us.
Reply



Logout Mark Read Team Forum Stats Members Help
script.module.metadatautils - issues with bs40
This forum uses Lukasz Tkacz MyBB addons.