Playlists including songs with special characters wont play
#1
Hi

When saving a playlist including songs with special characters, kodi will remove the special characters and the songs will not play anymore.
For example:
Starting the song 'Śmierć Cieplna Wszechświata' from Music Library works fine.
After saving a playlist with only this song the name becomes 'mier Cieplna Wszechwiata' and the playlist wont play.

I'm using LibreELEC 9.0.2.
If needed I will provide a log file.

Thanks for your help.
Reply
#2
Yes, it's a long-standing issue and years ago I submitted a feature request asking for a change to allow Kodi to export utf-8 encoded m3u playlists.  It could be saved as m3u8 if needs be.

Kodi will read m3u playlists encoded utf-8 with no problem, if you have other software that will create them.

scott s.
.
maintainer of skin  Aeon MQ5 mods for post-Gotham Kodi releases:
Matrix
Leia see: 304472
Reply
#3
Ok, thank you scott for the information.

I'm aware that there is currently only one dev working on audio features but maybe this is something that could be fixed in the next Kodi release.

I noticed that it is possible to add songs with special characters to a playlist using the 'context.playlist.append' add-on mentioned in this thread.
As long as you don't save the playlist within Kodi it works fine...

Cheers
Reply
#4
As far as I can tell, Kodi translates the itemlabel and the filename to whatever local character set your device is using.

I have tested an easy fix on Linux by removing the lines that do that translation and thus leave everything utf-8 encoded.  I used Śmierć Cieplna Wszechświata as a filename so as to be as accurate as possible with the OP.

Results from saved playlist when Kodi core converts to local charset.

#EXTCPlayListM3U::M3U
#EXTINF:0,Plaga - mier Cieplna Wszechwiata (128  kbps).mp3
/media/sdd1/Plaga - mier Cieplna Wszechwiata (128  kbps).mp3

After change

#EXTCPlayListM3U::M3U
#EXTINF:0,Plaga - Śmierć Cieplna Wszechświata (128  kbps).mp3
/media/sdd1/Plaga - Śmierć Cieplna Wszechświata (128  kbps).mp3

@DaveBlake This works fine on Linux, but I have no dev platform to test it against any other OS's.  I think anything *nix based should be OK but Windows is a different case as it's based around utf-16 and there is a possibility that it might do some internal conversion along the way.  Perhaps this is why some conversion is done in the first place Huh
Learning Linux the hard way !!
Reply
#5
I don't know why for sure the char conversion happens on saving m3u playlists, could even go back to the early xbox days. But AFAIK .m3u files are meant by definition to contain text encoded in the local system's default non-Unicode encoding, so what Kodi does is reasonable and does not need fixing.

If someone wants to work on adding saving utf-8 encoded playlists as .m3u8 files as a improvement then I'm happy to support them. Smile

I believe generally that Kodi supports any unicode in filenames except NUL, \, /, :, *, ", <, >,  and | (the Windows invalid chars applied consistently across all platforms). So other than that limit I don't think there is any other conversion along the way.
Reply
#6
Well that's not so trivial is it ?   There is no option (in spite of core being able to save playlists in multiple formats) to select what format to save your playlist in as '.m3u' is hardcoded.  So a method of choosing the extension would be needed.
Learning Linux the hard way !!
Reply
#7
that's what I saw as well.  As Dave points out, what is being done is correct per the definition of m3u file.  As far as filename, whatever legal filename exists in the source will be saved in the m3u filename and any conversion of encoding to open the file is being done in Kodi.  As I mentioned, you can encode an m3u file with utf-8 externally and Kodi will read it and open the files.  But to be spec-correct I guess "m3u8" is the correct extension.  As it is, m3u files created in Kodi are incapable of being used in any program including Kodi when filenames contain characters that can't be encoded in the user's locale.  So to be correct IMHO Kodi should raise a notification and not create an unusable m3u file in this case.

My thought was to use an advanced setting to create m3u8 files.

scott s.
.
maintainer of skin  Aeon MQ5 mods for post-Gotham Kodi releases:
Matrix
Leia see: 304472
Reply
#8
(2019-10-31, 17:42)scott967 Wrote: My thought was to use an advanced setting to create m3u8 files.

I think a better option would be to drop creation of m3u files and just create m3u8 files on save.  That would mean that all existing playlists would still work but going forwards, any new playlists created inside kodi would be m3u8.  AS.xml has enough stuff in it already I think.
Learning Linux the hard way !!
Reply
#9
(2019-11-01, 10:55)black_eagle Wrote:
(2019-10-31, 17:42)scott967 Wrote: My thought was to use an advanced setting to create m3u8 files.

I think a better option would be to drop creation of m3u files and just create m3u8 files on save.  That would mean that all existing playlists would still work but going forwards, any new playlists created inside kodi would be m3u8.  AS.xml has enough stuff in it already I think. 
That works for me, but I was thinking a use case would be to use the playlist in other programs that might expect m3u extension.  (I don't myself.)

scott s.
.
maintainer of skin  Aeon MQ5 mods for post-Gotham Kodi releases:
Matrix
Leia see: 304472
Reply
#10
(2019-11-04, 17:55)scott967 Wrote: a use case would be to use the playlist in other programs that might expect m3u extension.

That would defeat the entire object of having m3u & m3u8 extensions though, as one is coded in the user's locale and the other in utf-8.  Besides, that would be the 'other programs' issue. Smile
Learning Linux the hard way !!
Reply
#11
Hi everyone,

Any progress on this feature? Actually  > 50% of my music files have unicode special character in their names. So I would like to see this feature actually get implemented and am even willing to do it myself.
Considering back compatibility, code changes and potential influence to other programs discussed above, I suggest this feature to be implemented in the following way:

When saving playlists, create m3u8 files (with .m3u8 extension) by default unless the name of the playlists ending with ".m3u"

I skimed kodi's GUI code. Adding new options to choose which format to "save as" requires changes to both interface message handling and skin files. And I am not sure if adding new options in the GUI brokes 3rd-party skins.

Besides, I am having this problem in a Linux, whose local charset is already configured to "UTF-8". Kodi seems to set default UserCharSet to "CP1252" (windows) ignoring my locale settings (see xbmc/LangInfo.cpp function CLangInfo::SetDefaults()). To make things worse, I cannot find UTF-8 as an option in kodi Settings->Interface->Regional->Character Set
I have checked that if I change the default charset to "UTF-8", m3u playlists work fine with my songs in my linux setup.
I would like to confirm if "ignore locate settings and force UserCharSet to be CP1252" is a bug or I configured something wrong. And why "UTF-8" is not listed as an option in kodi settings?

Thanks!
Reply
#12
(2019-12-18, 09:56)alkaid Wrote: Hi everyone,

Any progress on this feature? Actually  > 50% of my music files have unicode special character in their names. So I would like to see this feature actually get implemented and am even willing to do it myself.
Considering back compatibility, code changes and potential influence to other programs discussed above, I suggest this feature to be implemented in the following way:

When saving playlists, create m3u8 files (with .m3u8 extension) by default unless the name of the playlists ending with ".m3u"

I skimed kodi's GUI code. Adding new options to choose which format to "save as" requires changes to both interface message handling and skin files. And I am not sure if adding new options in the GUI brokes 3rd-party skins.

Besides, I am having this problem in a Linux, whose local charset is already configured to "UTF-8". Kodi seems to set default UserCharSet to "CP1252" (windows) ignoring my locale settings (see xbmc/LangInfo.cpp function CLangInfo::SetDefaults()). To make things worse, I cannot find UTF-8 as an option in kodi Settings->Interface->Regional->Character Set
I have checked that if I change the default charset to "UTF-8", m3u playlists work fine with my songs in my linux setup.
I would like to confirm if "ignore locate settings and force UserCharSet to be CP1252" is a bug or I configured something wrong. And why "UTF-8" is not listed as an option in kodi settings?

Thanks!

I'm pretty sure the system environment locale is only used for setting the collation.  Otherwise Kodi regional settings are used.  Note that for Android and maybe iOS the locale is forced to "C" as Kodi doesn't access the system's localization apis for those platforms.

I'm not sure when the user character set gets used, other than for reading subtitles (and that's a different setting than the UI setting).  Also when you set the UI language (install a language resource) the resource addon sets a default UI and subtitle character set.  But AFAIK, all the strings in strings.po are encoded utf-8.

It does seem like if a UI charset option of utf-8 was available, that would help, though still needing the m3u8 extension.

Something else to consider from a text charset decode standpoint is cue sheets.  It's been a while but ISTR that in testing cue sheets encoded in utf-8 they will be read but if there is a BOM the first line of the cue sheet is dropped.  Obviously a BOM isn't needed in UTF-8 but Windows seems to like to encode it anyway.

I suppose if the charset is set to utf-8 that would work OK if m3u are strictly ASCII (<128).

I looked at the wiki pages, and I think they have errors.  For example Language_support (wiki) says the UI character set is used in reading mp3 tags.  However, both ID3 2.3 and 2.4 specify that ISO 8859-1 must be used if not utf encoded.  And since 8859-1 != CP1252 in the range >127 it would be a bug to use that charset to decode ID3.  And the default UI and subtitle charsets are defined in addon.xml, not langinfo.xml as indicated here Language_support (wiki).

Maybe some of the intent of the user charset was to access the ttf font character map, but I think any ttf today has a unicode table in the character map so using unicode should suffice.

scott s.
.
maintainer of skin  Aeon MQ5 mods for post-Gotham Kodi releases:
Matrix
Leia see: 304472
Reply



Logout Mark Read Team Forum Stats Members Help
Playlists including songs with special characters wont play1
This forum uses Lukasz Tkacz MyBB addons.