Win Unicode Bug in most Addons
#1
Hi,

while reading the forum I found this thread: http://forum.xbmc.org/showthread.php?tid...pid1039311

I did a few tests and can confirm that it is very easy to reproduce and effects most addons - in a not so unrealistic scenario:
- You just need an unicode letter in the xbmc path (which contains the windows-username on windows7). Not sure on the other OS's...

You can easy reproduce this (or test your addons) with starting xbmc in portable mode (add "-p" to you xbmc start shortcut) from a xbmc folder which contains an unicode letter, "ä" or "é" for example.
So just download xbmc again, install it to "C:\folder-with-unicode-ä-é\xbmc" and start "C:\folder-with-unicode-ä-é\xbmc\xbmc.exe -p".
You can also test it with starting xbmc with a local windows username which contans unicode-letters.

I cloned the eden plugins repo and got ~200 line matches without a real unicode decoding:
Code:
getAddonInfo('path')
- all usage of this path will fail without ".decode('utf-8')" added!
This effects all path's returned by getAddonInfo (because it returns string - not unicode) - you ALWAYS have to decode it properly.

In most cases the addons are using this path for icons, something like:
Code:
ICON = os.path.join(ADDON.getAddonInfo('path'), 'icon.png')
This will silently fail because xbmc skips images which aren't accessable - this won't be noticed by most users...

More bad is something like:
Code:
f = open(os.path.join( __settings__.getAddonInfo('profile'), "allmovies"),'wb')
This will fail with a script error: "WindowsError: [Error 3] The system cannot find the path specified."

But it can even avoid some addons from running with something like this:
Code:
sys.path.append(os.path.join(Addon(ADDON_ID).getAddonInfo("path"), "resources", "lib" ))
import module_in_the_above_path


Maybe a better solution would be that xbmc would return unicode-instances (and not str)...

Until that we have to do something like:
Code:
Addon.getAddonInfo('path').decode('utf-8')

Don't get me wrong - my Addons are also affected...

regards,
sphere
My GitHub. My Add-ons:
Image
Reply
#2
I don't believe it affects anything other than windows at this point, though that may possibly be due to most *nix users not having special-characters in their usernames.
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


Image
Reply
#3
Hmm haven't had any issues with Artwork Downloader concerning this. Did have some issues with show text in the dialog that threw that error and the JSON requests.

But nothing in the cases you described


EDIT:

Ok i see your point. But this only happens in portable mode?

One add-on even made from this: XBMC_Edén
This: XBMC_Edén


Seems lilke my add-on doesn't even care about this without any code intervention Smile
Read/follow the forum rules.
For troubleshooting and bug reporting, read this first
Interested in seeing some YouTube videos about Kodi? Go here and subscribe
Reply
#4
(2012-03-14, 00:26)Martijn Wrote: Ok i see your point. But this only happens in portable mode?

One add-on even made from this: XBMC_Edén
This: XBMC_Edén


Seems lilke my add-on doesn't even care about this without any code intervention Smile

Sorry to say this - but you are wrong.

1. No, this is not only happening in portable mode. This happens at least in one of these scenarios:
a: You are on windows in portable mode with having xbmc installed in any path which contains a unicode letter
b: You are on windows in "normal" mode - but your windows username contains any unicode username (think about "Renè, André, ...")

2. Of course is your addon "Artwork downloader" affected! It is even crashing in some situations because of this bug:
Code:
07:44:19 T:3044   ERROR: Error Type: <type 'exceptions.WindowsError'>
07:44:19 T:3044   ERROR: Error Contents: (3, 'Das System kann den angegebenen Pfad nicht finden', 'D:\\Progs\\XBMC-Eden\xc3\xa4\\portable_data\\userdata\\addon_data\\script.artwork.downloader\\temp/*.*')
07:44:19 T:3044   ERROR: Traceback (most recent call last):
                                              File "D:\Progs\XBMC-Eden坰ortable_data\addons\script.artwork.downloader\default.py", line 840, in <module>
                                                Main()
                                              File "D:\Progs\XBMC-Eden坰ortable_data\addons\script.artwork.downloader\default.py", line 140, in __init__
                                                self.cleanup()
                                              File "D:\Progs\XBMC-Eden坰ortable_data\addons\script.artwork.downloader\default.py", line 218, in cleanup
                                                for x in os.listdir(self.fileops.tempdir):
                                            WindowsError: (3, 'Das System kann den angegebenen Pfad nicht finden', 'D:\\Progs\\XBMC-Eden\xc3\xa4\\portable_data\\userdata\\addon_data\\script.artwork.downloader\\temp/*.*')

regards,
sphere
My GitHub. My Add-ons:
Image
Reply
#5
Well at least I had no one complaining so far Wink


I had to do this:
PHP Code:
__addonpath__   __addon__.getAddonInfo('path')
__addondir__    xbmc.translatePath__addon__.getAddonInfo('profile') ).decode('utf-8'

for it not to fail.

What about this one?
PHP Code:
THUMBS_CACHE_PATH xbmc.translatePath"special://profile/Thumbnails/Video" 

Should have the same problem I think.

Would be nice if this could be fixed in XBMC core instead of every addon is gonna need to fix this
Read/follow the forum rules.
For troubleshooting and bug reporting, read this first
Interested in seeing some YouTube videos about Kodi? Go here and subscribe
Reply
#6
this is what I did in XBMC Subtitles(thx to chadoe), but it brought in more issues ... that chadoe fixed again Smile

testing it out now on OSX to see if it breaks anything Smile

EDIT: all good on OSX with above changes
Reply
#7
What I have found my addons is that xbmc always returns utf-8 from e.g. keyboard and dialogs. By always converting to unicode for internal code use and then convert back to utf-8 for output problems are solved...
If i were to use
PHP Code:
ICON os.path.join(ADDON.getAddonInfo('path'), 'icon.png'
I would do
PHP Code:
ICON os.path.join(unicode(ADDON.getAddonInfo('path'), 'utf-8'), 'icon.png'
ICON will then be e.g. u'/some/pathto/\xD6rn/icon.png'

Also urllib.qoute and urllib.quote_plus is tricky since they don't handle uincode.
Pneumatic | SABnzbd | XBMC that just works - openelec
Reply
#8
Correct. ALL strings that XBMC gives python are currently utf8.

I suspect XBMC can make it much easier on plugins by returning unicode strings instead, but this is obviously something that will break API (i.e. post Eden).
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


Image
Reply
#9
trac it so won't be forgotten and there can be some thought about this?
Read/follow the forum rules.
For troubleshooting and bug reporting, read this first
Interested in seeing some YouTube videos about Kodi? Go here and subscribe
Reply
#10
(2012-03-14, 22:40)jmarshall Wrote: Correct. ALL strings that XBMC gives python are currently utf8.

I suspect XBMC can make it much easier on plugins by returning unicode strings instead, but this is obviously something that will break API (i.e. post Eden).

An idea, why not a xbmc return encoding tag in the addon settings.xml defaulting to utf-8 with unicode as the option?
Pneumatic | SABnzbd | XBMC that just works - openelec
Reply
#11
It doesn't only affect addon settings though (and an add-on needn't have any settings) - basically every string we pass to python is utf8.
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


Image
Reply
#12
So what is the "official recommended way"?

- Should we (addon-authors) fix this? (80% of all addons are at least affected by not working picture-path-joins)
- Ignore and add a FAQ entry to refer to "do not use windows user-names with special characters on win7 or use portable-mode instead"?
- Wait for a core fix? (I guess only for post eden)

regards,
sphere
My GitHub. My Add-ons:
Image
Reply
#13
Core fix is only post-eden, yes. Remember that it only affects paths per-se, so if you're writing to or reading from the filesystem then you'll want those paths handled. Depending on your add-on this may be minimal.
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


Image
Reply
#14
Sure, for my 6-8 add-ons it is easy and I know what to do - I was just thinking about the other add-ons. I guess most devs still don't know about this.

thanks
My GitHub. My Add-ons:
Image
Reply

Logout Mark Read Team Forum Stats Members Help
Unicode Bug in most Addons0