Project

General

Profile

action #13730

action #11334: decomission svn.opensuse.org

Modernize the desktop file translations mechanism

Added by lnussel about 5 years ago. Updated about 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Development
Target version:
-
Start date:
2016-11-28
Due date:
% Done:

100%

Estimated time:
120.00 h

Description

Goals

  • make the mechansim work with git and weblate
  • turn the old, ugly and slow scripts into something maintainable
  • make the mechanism more generic to also support polkit actions, appdata files and mime type information
  • maybe add some sensible grouping or priorization so translators don't get overwhelmed (somehow related to package translations which has to do similar things).

brief description of the current mechanism

Translations are currently stripped off desktop files in the build
root and are output as separate build result by
https://build.opensuse.org/package/view_file/openSUSE:Factory/update-desktop-files/brp-trim-desktop.sh?expand=1

Then collected outside OBS by
https://svn.opensuse.org/svn/opensuse-i18n/trunk/lcn/50-tools/desktop-files-update.sh
The script transforms the input files to po files for translations.
The translated mo files are then put back into the distribution in
the desktop-translations package. Desktop environments then read the
English .desktop file strings and use gettext to get the
translations from mo files instead of the desktop file directly.

Links

Feature request for OBS to make collecting input files for
translation easier:
https://fate.opensuse.org/321366

There are scripts for appdata already, needs to be checked what they do actually:
https://build.opensuse.org/package/show/openSUSE:Factory/brp-extract-appdata
https://github.com/openSUSE/brp-extract-appdata

History

#1 Updated by lnussel almost 5 years ago

  • Subject changed from desktop file translations to Modernize the desktop file translations mechanism
  • Description updated (diff)

#2 Updated by lnussel almost 5 years ago

  • Description updated (diff)

#3 Updated by lnussel almost 5 years ago

  • Description updated (diff)

#4 Updated by lnussel almost 5 years ago

  • Due date set to 2016-12-31
  • Assignee set to kraih
  • Start date changed from 2016-09-14 to 2016-11-28
  • Estimated time set to 120.00 h

#5 Updated by lnussel almost 5 years ago

Some more hints:

$ osc getbinaries openSUSE:Leap:42.2 vym standard x86_64
$ cat binaries/vym.desktopfiles

to reproduce locally brp-trim-desktopfiles must be in the build root (osc build -x brp-trim-desktopfiles). Or put it as Supports: in prjconf (see osc meta prjconf openSUSE:Leap:42.2)

#6 Updated by coolo almost 5 years ago

the goal 4 is actually already achieved. there are actually 9 pots generated from desktop files.

#7 Updated by coolo almost 5 years ago

and the scripts are ugly and old - but their slowness is mainly because they have to go through packages in OBS one by one. So if you want this modern, you should start on OBS to offer a cross package build result like an on the fly generated tar */*desktopfiles

#8 Updated by coolo almost 5 years ago

Created https://github.com/openSUSE/desktop-file-translations and moved the tools,pot and translations into it - removing from SVN and announced on opensuse-translations

#9 Updated by kraih over 4 years ago

  • % Done changed from 0 to 90

My git_support branch should cover most of the requirements now. https://github.com/openSUSE/desktop-file-translations/tree/git_support

#10 Updated by kraih over 4 years ago

I've opened a pull request for review. https://github.com/openSUSE/desktop-file-translations/pull/1

#11 Updated by lnussel over 4 years ago

Stanislav, could you please add https://github.com/openSUSE/desktop-file-translations to l10n.o.o?

#12 Updated by lnussel over 4 years ago

  • Status changed from New to In Progress

#13 Updated by lnussel over 4 years ago

Example polkit file:
/usr/share/polkit-1/actions/org.freedesktop.policykit.policy

Example mime info:
/usr/share/mime/image/bmp.xml

Example Appdata:
/usr/share/appdata/firefox.appdata.xml

#14 Updated by kraih over 4 years ago

Everything should be done now, except for the point "make the mechanism more generic to also support polkit actions, appdata files and mime type information". I still have no idea what that actually means, the description is just too unspecific, and i'm getting the feeling that it might really be a different project that only slightly overlaps with this one.

My changes have been merged into the GitHub repo and the new workflow is explained here. I've also submitted an update to the desktop-translations package, which is now based on an _service file.

#15 Updated by lnussel over 4 years ago

I gave you some examples of other files (that I know of) that may be installed along with the desktop files and also contain translations. The strings in those files may appear on the desktop too. You are now familiar with how the desktop file translation mechanism works. So you are qualified enough come up with suggestions how to deal with the other files. A naive solution would probably fork your code and run three more versions of it. Clearly undesirable. The tools you worked with do most of the work already. If they are generic enough adding at least one case of processing a different file type shouldn't be hard. If that's a separate project in your opinion, then make a plan of what would need to be done. I can't spoon feed that to you.

#16 Updated by coolo over 4 years ago

i.e. the first step is changing https://build.opensuse.org/package/view_file/openSUSE:Leap:42.3/update-desktop-files/brp-trim-desktop.sh?expand=1 to copy away these files to have them available in 42.3 for extracting.

But this only makes sense if we know that e.g. polkit can read from .mo files.

#17 Updated by kraih over 4 years ago

So, Ludwig, am i understanding you correctly that you want all strings with language code extracted from those additional file types, and merged with the rest of the data from the .desktop files? So everything ends up together in the same .po files in the Git repo? And that should be done by a new program replacing brp-trim-desktop.sh, with pluggable data extraction modules for different file types?

#18 Updated by lnussel over 4 years ago

kraih wrote:

So, Ludwig, am i understanding you correctly that you want all strings with
language code extracted from those additional file types, and merged with the
rest of the data from the .desktop files? So everything ends up together in
the same .po files in the Git repo? And that should be done by a new program
replacing brp-trim-desktop.sh, with pluggable data extraction modules for
different file types?

Sounds like a possible solution.

#19 Updated by kraih over 4 years ago

  • Due date deleted (2016-12-31)

#20 Updated by sbrabec over 4 years ago

The connection with Git and Weblate it the smaller part of the work. Translation mechanism of these files was not designed for ex-post translation updates, so it will need a significant amount of work to change it.

I can imagine three possible ways to fix it:

  1. Use the SUSE desktop file approach: patch libraries and application accessing desktop files, to load external translations.

I am not sure how complicated it would be.

  1. Use the way we used for lang packages in past:

Split out all these files to a separate sub-package.

Invent an update script creating lang bundles of updated files.

It would again require patching, now we have to patch search path for these files, exactly like we have a patch in the glibc modifying search path for translations.

  1. Handle update on packaging level:

Again split out all these files to a separate sub-package.

But then invent a mechanism that picks the in-package files, processes them somehow, and creates an update package with a higher Release number. Possibly publish these packages in a separate repo.

Note: 1 and 2, as they exist currently for desktop files and language bundles, are not extensible. Only one directory and only one domain name can be used. => The mechanism is not usable for add-ons and third party packages.

I can imagine an alternative solution: Adding a TextDomain and TextDomainDir keywords. Then different packages can have different translation domain. (Normally, desktop translations remain in the po files that come with the package.)

#21 Updated by lnussel over 4 years ago

PoC for polkit to load translations via gettext as fallback

#22 Updated by favogt over 4 years ago

Attached is a POC of a script that strips all xml:lang from mimetype info, appstream metainfo and polkit actions and collects it in build result files.
(It's all just xml, so I put it into a single script for now). It can be seen in action here:

https://build.opensuse.org/project/show/home:favogt:branches:X11:common:Factory

Now the scripts for generating .pot and .po files are missing and also support for gettext in polkit and appstream (-Qt, -glib).
I'm not sure which libs all read the mimeinfo xml, but it might be many different libs that need to be patched...

#23 Updated by favogt over 4 years ago

Attachments are apparently broken, so use the linked OBS prj for now to have a look.

#24 Updated by sbrabec over 4 years ago

I don't know polkit exactly, but I will prefer more generic solution.

Looking at a random polkit and MIME files, I see that both use similar syntax:

MPEG video
MPEG مرئي
Videa MPEG
Видео — MPEG

<message>Authentication is required to run a program as another user</message>
<message xml:lang="cs">Pro spuštění programu pod jiným uživatelem je vyžadováno ověření</message>

If we define a common way to support external translation, its support can be moved to a common library (libxml2?)

I can imagine:
MPEG video

<message xml:textdomain="suse-polkit-action">Authentication is required to run a program as another user</message>

To match gettext features, we should support xml:textdomaidir as well.

Something like that would be upstream-able, code could be common for many projects, and the only change needed by SUSE would be adding small change to XML generator. Even there would be possible to invent something that could be upstreamed, e. g. configure option --with-textdomain --with-textdomaindir --disable-embedded-translation, ideally as a part of some m4 macro set.

There is a one good reason, why embedded translation was invented: Desktop files come from different projects, each desktop file would have a different textdomain; many translation files have to be loaded just to display menu. My proposal would be a half way between both. It would be a bit more complicated to implement: As textdomain name is not hardcoded, it would require to keep list of textdomains that are already loaded.

By the way, normal gettext translation of projects not only hardcodes strings to XML or desktop files, but it also keeps them in the po file. So it could work out of the box with TextDomain=@GETTEXT_PACKAGE@

#25 Updated by favogt over 4 years ago

  • Assignee changed from kraih to favogt

sbrabec wrote:

I don't know polkit exactly, but I will prefer more generic solution.

Looking at a random polkit and MIME files, I see that both use similar syntax:

This is how I implemented it here: https://github.com/Vogtinator/desktop-file-translations/blob/master/51-xml/tar2po/xml_handler.py#L34

As not every tag is translatable and not every translatable tag is translated, it's necessary to have a list
of translatable tags for each file type. I did it for polkit actions, mime info and appstream for now.

The new update-desktop-file package that is currently on the way to 42.3 uses the xsl-based stripping already.

There is a one good reason, why embedded translation was invented: Desktop files come from different projects, each desktop file would have a different textdomain; many translation files have to be loaded just to display menu. My proposal would be a half way between both. It would be a bit more complicated to implement: As textdomain name is not hardcoded, it would require to keep list of textdomains that are already loaded.

Wouldn't relying on the context work just fine as well? That's how the current/old mechanism works and changing that would mean breaking the current desktop file translations if we want to keep that similiar.

#26 Updated by favogt over 4 years ago

  • Assignee changed from favogt to sbrabec
  • % Done changed from 90 to 95

desktop-file-translations is now updated to the new mechanism, so the next tasks are:

  • Weblate needs to show the appstream.po, mimeinfo.po, polkitactions.po and polkitactions-freedesktop.po files as well (-> sbrabec)
  • desktop-translations in OBS needs to generate .mo files from the new .po files as well (-> I'll do that)
  • The polkit patch needs to be submitted (-> lnussel, as he's the patch author)

#27 Updated by lnussel about 4 years ago

  • % Done changed from 95 to 0

Ubuntu patch for polkit for reference:
https://bugs.freedesktop.org/show_bug.cgi?id=29639

#28 Updated by lnussel about 4 years ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

mechanism is in place and used for 42.3. Remaining enhancements and bugs to be solved separately.

Also available in: Atom PDF