[finnhubb]

@fbacher thanks for the detailed post. Are you still working on this?

I don't know about Kodi's utf-8 hack for Turkish exactly, but I do notice Kodi changing reasonable locale and filesystem encoding settings. In my addon I was able to download files with non-ASCII names (from yle, not youtube) and display them correctly within Kodi. I will give a quick explanation of my approach here in becomes useful.

Overview:
- Extract filename (and download link) into python string (unicode code points).
- Set download directory, using `xbmcvfs.translatePath()` if required.
- Append filename (utf-8 encoded) to complete download path.
- Download file
- Open filename (utf-8 encoded) and write downloaded data as binary data.
- Pass filename (utf-8 encoded) and path to Kodi as list item.

python:


from urllib.parse import urlencode



filename = "non-ääsci"

url = "example.xyz/video/12345"



# Download path

target_dir = xbmcvfs.translatePath("special://temp").encode("utf-8", "surrogateescape")

filepath = os.path.join(target_dir, filename.encode("utf-8", "surrogateescape"))



# Download remote file.

res = requests.get(url, stream=True)



# Open filename (utf-8 encoded) and write downloaded data as binary data.

with open(filepath, "ab") as file:

    for chunk in res.iter_content(chunk_size=1024**2):

        file.write(chunk)



# Create list item entry for downloaded file.

list_item = xbmcgui.ListItem(label=filename.encode("utf-8", "surrogateescape"))

# From memory it is best to give unicode to urlencode

attrs = {"local_file_playback": filepath.decode("utf-8", "surrogateescape")}

param_string = urlencode(attrs, encoding="utf-8", errors="surrogateescape")

callback_url =  f"plugin://{addon_id}/?{param_string}"

listing = (callback_url, list_item, False)

Obviously this code is incomplete, and requires setting  relevant `addon_id` and `_handle` and having a function to handle "local_file_playback" parameter to play the video.

In summary:
- Changes Kodi makes to Python locale are ignored by explicitly using utf-8 for filenames.
- Changes Kodi makes to Python filesystem encoding are ignored by writing the file as binary data. i.e. video file is not a text format so encoding is not an issue.
- Filenames and paths are passed to Kodi functions are utf-8 encoded bytes.


Extra notes:
- Python os functions accept either bytes or unicode strings: return type matches input type.
- Everything in the OS is bytes (eg. filenames), so use bytes as input to os functions.
- Not all bytes have valid unicode representations when decoded from utf-8.
- Since the underlying bytes can possibly be invalid, error handling is required..
  The surrogateescape error mode represents the invalid utf-8 bytes as reserved unicode code points.
  It is the only error mode in python that provides lossless handling of the invalid utf-8 bytes.
  e.g. b"\xff" == b"\xff".decode("utf-8", "surrogateescape").encode("utf-8", "surrogateescape")
- Many Kodi functions can accept either python unicode strings or utf-8 bytestrings, but they don't accept python unicode strings containing reserved code points (i.e. unprintable utf-8 decoded using surrogateescape error handler.)