[viljoviitanen]

I made a diff with the back converted xls file.

http://pastebin.com/2GkhGY0W

Seems ok, but I see one problem. We have the missing translations filled in with English ones. This could cause problems, as if we update back the language file to Transifex we could have them appear as translated ones, but they are only filled up with English.

Can you check what can be the problem ? Is it in the conversion or in Transifex ?

*duh* I should have realized that. It's probably because of the duplicate target elements, probably those missing the translation only have the "duplicate" (english) target element in the downloaded xliff file.

I'll fix that tomorrow.

Unless someone beats me to it the easy fix for now is to require two target elements in xliff->xml conversion (putting in empty string in the generated xml or leaving the element out). The profer fix is to get rid of the target elements in source xliff files (and fix the xliff->xml converter accordingly).

[alanwww1]

We typed the post at the same time :-)

http://forum.xbmc.org/showpost.php?p=100...stcount=60

[alanwww1]

I also created a test project here: https://www.transifex.net/projects/p/xbmc-main-test/

This time I used a source file without the target fields. This time it produces the back-translated file correctly: http://paste.kde.org/198866/

However I have some error with test-xlifftoxml.py:

Code:

File "./test-xlifftoxml.py", line 36, in <module>
    ids[t.attributes['id'].value]=getText(t.getElementsByTagName('target')[0].childNodes)
IndexError: list index out of range

Could you please check ? Thx

[viljoviitanen]

I also created a test project here: https://www.transifex.net/projects/p/xbmc-main-test/

This time I used a source file without the target fields. This time it produces the back-translated file correctly: http://paste.kde.org/198866/

However I have some error with test-xlifftoxml.py:

Code:

File "./test-xlifftoxml.py", line 36, in <module>
    ids[t.attributes['id'].value]=getText(t.getElementsByTagName('target')[0].childNodes)
IndexError: list index out of range

Could you please check ? Thx

Yep. That's because when target element was missing, [0] fails (it gets the first element of the list that was provided from getElementsByTagName, and in that case, when no target elements are there, the list is empty)

Fixed in my git, and I tested it with the the xliff you provided.

I also added test-generatesourcexliff.py to generate the source files without target elements (previously I generated the source file by making an xliff with English as target language...). I'll check my bootstrap code as well and re-do my test project.

[alanwww1]

I've got some good news. I made a little chat with Transifex support. They will have new release probably next week, for the hosted version and they warm wellcome XBMC as a new translation project.

Code:

→How can I change the UI language of Transifex.net ?
John: Hey. We're preparing a new update. We'll have more languages soon.
John: What language are you interested in?
→ALL :-) In fact we would like to set up a translation project for XBMC media player. An open source cross platform Mediacenter software.
John: Oh, great, I know about it. Well, you can go ahead and create your project on Transifex.
John: Transifex is translated through Transifex itself so you can even help by contributing translations here: https://www.transifex.net/projects/p/transifex/
John: (The languages on the UI though are unrelated to the languages your project is translated in, just to make it clear).
→Yes, that is what my idea was to urge our translation teams to help out translation Transifex itself. We have translators from all around the world so I can ask them to help out. Me personally surely can help with the Hungarian translation :-)
John: Cool. We have a Hungarian team already (https://www.transifex.net/projects/p/transifex/team/hu/). If you'd like, make a join request and I will add you as a member.
→I will, thanks. When is the new release planned to come out ?
John: Sometime next week.
→Cool. Do you recommend for a project like ours to make a standalone server or use Transifex net ? I tried to install Transifex on a Fedora 16 server machine with the Python easy_install way, but it came out as a mess unfortunately. Missing icons, occasional Python errors, strings don't show up dispite imported well. Don't know what I've done wrong.
John: It will be way easier for you to just use transifex.net. We take care of all the server management and you can just focus on the translations :)
→Thanks for the help so far :-)
John: No problem. If you guys need any help while setting up your project, just let us know :)

I'll test your code soon, thanks, Attila

[alanwww1]

I've made a quick test with xlifftoxml and works fine. Here is the diff we get back for Finnish strings:

http://pastebin.com/kY0Qww7i

Only a minor thing is two LFs at the end of the file.

There is still one thing i mentioned earlier, but seem to me as a nice thing if we could solve. Well of course translators has not had such control ability with the current XBMC translation tool, but would be good if we can solve it:

- Normally xbmc devs does not re-use the same xml ID for a completely new string, but it could happen in a long time period (One complete release cycle maybe) First they set it empty and after a longer (or sometimes not so long) time they use it for a completely new string. Also sometime they change, refine the strings with a little different meaning. So in this cases what happens:

1. We generate a new xliff file from the current fresh English GIT revision, containing this changed xml ID entry with a new English string part but the same translated language part.
2. We feed this new xliff file to Transifex and it will be treating this entry as a new translated entry. But of course this is not right, because it has not been translated at all.
3. The translators won't see this mistake as it is not showing up as a new translatable string. You can only find this problem if you walk through all the strings an check them.

To avoid I can think of two solutions.
1. Only let developers update the non-english strings in Transifex. (I don't think the would agree on this)
2. At first initial setup, use the English version of the language file which matches the version of the language string. Then with all updates, check maybe that previous xliff file if the English string has CHANGED for the SAME XML ID . If it has, we shouldn't let that entry coming into the translation, maybe only if the translated string changed as well (maybe the Dev updated that one too)

We have to check if maybe Transifex has this behavior with the changed English strings, but unchanged translated strings. I have not yet tried it.

What do you think. do you have a better idea ?

[takoi]

Not sure i understand what you mean alan. Can you give an example?

[alanwww1]

Just did a test which confirms the problem.

First I changed xml string ID 1 "Pictures" in the English Source file to "Instead of Pictures a completely different string(test)"

Then I updated Transifex with the new source file. "Instead of Pictures a completely different string(test)" appeared as a new translatable strings for all languages.

Than simulating that I update the Finnish language file from XBMC git, I modified the INPUT xliff file like this:

Code:

<trans-unit id="1">
    <context-group><context context-type="id">1</context><context            context-type="context">Sample Context</context$
    <source>Instead of Pictures a completely different string(test)</source>
    <target>Kuvat</target>
   </trans-unit>

Updated Transifex with this modified INPUTxliff file. And of course "Kuvat" is a wrong translation for the new English string.

After all this I have a wrongly translated Entry in Transifex. See last Entry here:

https://www.transifex.net/projects/p/xbm...l/fi/view/

Of course Afrikaans is still correct as having "Instead of Pictures a completely different string(test)" as a new translatable string.

Sidenote: The order of the strings got broken when updating the strings. They get to the end of the list. (Maybe that is tolerable as translators normally does not care about the xml id)

[takoi]

This is exactly why i18n tools use the English strings as keys. Conceptually, the correct keys are the template strings. It's simply not possible to properly export xbmc's non-template xml files because the true keys are not really there. Exporting from transifex works because the keys a known for every translation.

The obvious solution is to simply not import, i.e not allow translation in git repo. Personally, I don't see why anyone would need to do that.

[viljoviitanen]

How about this:

If the developers change meanings of or re-use id numbers, at the same time you change the English string, put [INVALID (wiki)] or something at the end of all changed id strings in translations in git (which then are automatically pushed to transifex as well). Then inform all translators manually about this. Those strings can then be easily searched in transifex and corrected.

Hopefully this won't happen very often, so if we have a way to deal with this, even a bad one, it's enough.

[alanwww1]

Of course I'd need to discuss this with the team, but how about having the original English string in the language xml files. Like this:

<string id="7" source="File manager">Fájlkezelő</string>

Just like "standard" translation files. This way we could write a custom parser for Transifex and we would't need a separate xlif file.

[viljoviitanen]

Of course I'd need to discuss this with the team, but how about having the original English string in the language xml files. Like this:

<string id="7" source="File manager">Fájlkezelő</string>

Just like "standard" translation files. This way we could write a custom parser for Transifex and we would't need a separate xlif file.

I still don't think it's a good idea to write a parser module in transifex, we gain essentially nothing and it's harder to do that way (auto-update of source file is so small a gain I don't count it at all cause it's trivial to do with the api, and more real time too because we can use github hooks). And we aren't really needing the xliff files as such, the idea is to generate them "on the fly" when using the api and just discard them after the api call. (as I noted before, they can be mapped 1:1 so no information is lost converting back and forth.)

Anyway I'm really not sure what adding source strings in xbmc xml accomplishes - does it help with the "changed meaning" scenario? Because in any case since xbmc is using only the id, if you do not invalidate the old translation, xbmc is using the old translation and transifex too gets updated with the old translation. And if/when a translation is invalidated, we will need some mechanism that allows strings needing retranslation detected. I think best would be to just add new id's.

About id's: I think it's "foobared up" the id's are thought to have some meaning (==i.e. derive context from id number, or have a need to re-use them), they should be just random numbers, like uuid, having no "internal" meaning. I can rewrite my tools easily to preserve the xml order. Now I've just done them the easiest way possible, not caring about order internally at all, using a python "dictionary" (similar to perl hash), and at output sort by id numerical value (this is just kind of a left-over from my first tool that detected incomplete translations...)

[takoi]

Why did we need importing again?

[alanwww1]

Anyway I'm really not sure what adding source strings in xbmc xml accomplishes - does it help with the "changed meaning" scenario? Because in any case since xbmc is using only the id, if you do not invalidate the old translation, xbmc is using the old translation and transifex too gets updated with the old translation. And if/when a translation is invalidated, we will need some mechanism that allows strings needing retranslation detected. I think best would be to just add new id's.

As I see, this is now the only way of handling the "changed meaning" problem. Either to store the used source string in a backup xml or xlif file or to put them inside the original ones,

Why did we need importing again?

Guys, I want to make the best possible solution here which is both acceptable for translators and developers. There are a lot of commits when developers change refine etc. the meaning of the string. Also when a non-English speaking developer commits something, he will do his language translation already on GIT. He won't be waiting for things to go through Transifex and translate there. Specially I am thinking of addon developers who develop, test commit to git in the most cases two languages. English +1. So we do need the ability to updated Transifex translations from git sometimes.

[takoi]

Ok, I see your point. Actually, im one of those committing English+1 for addons. (for xbmc patches i dont think the reviewer will even let you do that in the same commit) However, with respect to addons, being in the official repo is optional and there's already lots of rules in place for this repo. Will adding a few more make any difference?
Call it package translations like ubuntu if you like: You can translate your addon all you want but it wont necessarily be a part of xbmc. Translations in the addon repo are considered part of xbmc and thus distributed separately. Sure, someone is not going to like it but the same can be said for all the other rules.

Another thing that i don't think has been mentioned is how are you going to handle upstream changes? for instance, when a translation is updated in transifex, and the addon maintainer do changes without pulling from main repo, who wins? If you distinguish between upstream and package, the obliviously package trumps upstream.