[bossanova808]

For me it wasn't really errors in fetchpage itself, it was my crappy code - this was the first python I ever wrote and I was calling member function of str (e.g. strip.strip()) - directly, rather than on the string object. All the str functions exist for unicode as well, so I just called them directly on the string object, i.e. mystir.strip() instead.

Honestly, I looked in there and was embarrassed at what I saw!! I was quite a while ago now, but geez some ugly code is in there. Still, it works - indeed if I'd jsut written it properly to start, it would have worked even with the parsedom changes, so my bad really.

In your case, maybe pinch the fetchpage function from the older parsedom as an ugly workaround to keep going??

(At a guess on your issue: I would say you're getting strings back with embedded unicode in them - there's a well known bug for this in Python that requires a hack to get around - I came across this in XSqueeze and solve it with a function):

Code:

def unquoteUni(text):

    try:
        import urllib.parse
        return urllib.parse.unquote(text, encoding=self.charset)
    except ImportError:
        _hexdig = '0123456789ABCDEFabcdef'
        _hextochr = dict((a+b, chr(int(a+b,16))) for a in _hexdig for b in _hexdig)
        if isinstance(text, unicode):
            text = text.encode('utf-8')
        res = text.split('%')
        for i in xrange(1, len(res)):
            item = res[i]
            try:
                res[i] = _hextochr[item[:2]] + item[2:]
            except KeyError:
                res[i] = '%' + item
            except UnicodeDecodeError:
                res[i] = unichr(int(item[:2], 16)) + item[2:]
        return "".join(res)

giftie helped me find (or wrote? dude is awesome) - that function which deals with unicode in strings....this is for the case where you have a str type but with actually unicode in it like 'The message is \xe8\x91\xa3' or similar...those characters are unicode encoded (even though the type is str) - and fall outside of the ascii range hence the error you get above. The normal unquote does not work...