Modify

Opened 18 months ago

Last modified 18 months ago

#16360 new defect

I18n script creates data for transifex directories

Reported by: stoecker Owned by: team
Priority: normal Milestone:
Component: unspecified Version:
Keywords: i18n transifex Cc: floscher, michael2402

Description

The I18n script only skips transifex on po file generation, not at data file step.

Attachments (1)

transifexPlugins.patch (3.5 KB) - added by floscher 18 months ago.
Suggested patch

Download all attachments as: .zip

Change History (21)

comment:1 Changed 18 months ago by Don-vip

Keywords: i18n transifex added

comment:2 Changed 18 months ago by Don-vip

Cc: floscher added

For reference: [o34023] (#8645)

comment:3 Changed 18 months ago by floscher

What is the "data file step" that should be skipped? Which i18n script, one of the perl scripts or one of the Ant targets? I don't really understand what exactly goes wrong…

comment:4 Changed 18 months ago by stoecker

The buildplugins target is calling plugintrans for all plugins. It should skip the transifex ones.

Maybe: The target singpleplugintrans could issue a warning for transifex targets.

Changed 18 months ago by floscher

Attachment: transifexPlugins.patch added

Suggested patch

comment:5 Changed 18 months ago by floscher

Hi, maybe this patch could be a solution. It wraps the contents of target plugintrans in an <if><available><then></then><else></else></if> construct.

If a Transifex configuration is present, the target only echoes a warning and does nothing else. Otherwise the target is executed like before.

This should fix both buildplugins and singleplugintrans.

comment:6 Changed 18 months ago by Don-vip

Applied in [o34257].

comment:7 Changed 18 months ago by Don-vip

I think it's fixed?

comment:8 Changed 18 months ago by floscher

I saw that there were issues with the last JOSM-i18n run of Jenkins. Only had a brief look, will examine that later today.

comment:9 in reply to:  8 ; Changed 18 months ago by Don-vip

Replying to floscher:

I saw that there were issues with the last JOSM-i18n run of Jenkins.

The i18n extraction/upload to Launchpad is done by private scripts on JOSM server every time a new "latest" version is released (automatically each night, or manually on demand). The Jenkins job is trigerred every morning after Launchpad updates the bazaar branch with new translations from contributors. The goal is to detect Java translations issues (simple quotes).

comment:10 in reply to:  9 Changed 18 months ago by floscher

Replying to Don-vip:

Replying to floscher:

I saw that there were issues with the last JOSM-i18n run of Jenkins.

The i18n extraction/upload to Launchpad is done by private scripts on JOSM server every time a new "latest" version is released (automatically each night, or manually on demand). The Jenkins job is trigerred every morning after Launchpad updates the bazaar branch with new translations from contributors. The goal is to detect Java translations issues (simple quotes).

I'm mainly concerned by https://josm.openstreetmap.de/jenkins/job/JOSM-i18n/1051/consoleFull#ant-target-314 :

plugintrans:
    [mkdir] Created dir: /var/lib/jenkins/jobs/JOSM-i18n/workspace/i18n/build/plugin_Mapillary.2.OLD
  [foreach] Exception in thread "Thread-155" The following error occurred while executing this line:
  [foreach] /var/lib/jenkins/jobs/JOSM-i18n/workspace/i18n/build.xml:173: exec returned: 2
  [foreach] 	at org.apache.tools.ant.ProjectHelper.addLocationToBuildException(ProjectHelper.java:568)
  [foreach] 	at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:443)
  [foreach] 	at org.apache.tools.ant.taskdefs.CallTarget.execute(CallTarget.java:105)
  [foreach] 	at net.sf.antcontrib.logic.ForEach$1.run(ForEach.java:125)
  [foreach] 	at net.sf.antcontrib.util.ThreadPoolThread.run(ThreadPoolThread.java:53)
  [foreach] Caused by: /var/lib/jenkins/jobs/JOSM-i18n/workspace/i18n/build.xml:173: exec returned: 2
  [foreach] 	at org.apache.tools.ant.taskdefs.ExecTask.runExecute(ExecTask.java:644)
  [foreach] 	at org.apache.tools.ant.taskdefs.ExecTask.runExec(ExecTask.java:670)
  [foreach] 	at org.apache.tools.ant.taskdefs.ExecTask.execute(ExecTask.java:496)
  [foreach] 	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:293)
  [foreach] 	at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
  [foreach] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  [foreach] 	at java.lang.reflect.Method.invoke(Method.java:498)
  [foreach] 	at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
  [foreach] 	at org.apache.tools.ant.Task.perform(Task.java:348)
  [foreach] 	at org.apache.tools.ant.taskdefs.Sequential.execute(Sequential.java:68)
  [foreach] 	at net.sf.antcontrib.logic.IfTask.execute(IfTask.java:217)
  [foreach] 	at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
  [foreach] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  [foreach] 	at java.lang.reflect.Method.invoke(Method.java:498)
  [foreach] 	at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
  [foreach] 	at org.apache.tools.ant.TaskAdapter.execute(TaskAdapter.java:155)
  [foreach] 	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:293)
  [foreach] 	at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
  [foreach] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  [foreach] 	at java.lang.reflect.Method.invoke(Method.java:498)
  [foreach] 	at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
  [foreach] 	at org.apache.tools.ant.Task.perform(Task.java:348)
  [foreach] 	at org.apache.tools.ant.Target.execute(Target.java:435)
  [foreach] 	at org.apache.tools.ant.Target.performTasks(Target.java:456)
  [foreach] 	at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1405)
  [foreach] 	at org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:38)
  [foreach] 	at org.apache.tools.ant.Project.executeTargets(Project.java:1260)
  [foreach] 	at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:441)
  [foreach] 	... 3 more

Where does Mapillary.2.OLD come from?? The plugintrans task should be skipped for Mapillary

And https://josm.openstreetmap.de/jenkins/job/JOSM-i18n/1051/consoleFull#ant-target-313:

plugintrans:
    [mkdir] Created dir: /var/lib/jenkins/jobs/JOSM-i18n/workspace/i18n/build/plugin_Mapillary
[gettext-extract] ../plugins/Mapillary
[gettext-extract] build/plugin_Mapillary
[gettext-extract] build
[gettext-extract] /tmp/srcfiles8641058713880215098.tmp
[gettext-extract] Executing: xgettext -c --from-code=utf-8 --output=/var/lib/jenkins/jobs/JOSM-i18n/workspace/i18n/build/plugin_Mapillary/josm.pot --language=Java -k -ktrc:1c,2 -kmarktrc:1c,2 -ktr -kmarktr -ktrn:1,2 -ktrnc:1c,2,3 --files-from=/tmp/srcfiles8641058713880215098.tmp
     [exec] Reading file po/sk.po (lang sk)
     [exec] Reading file po/sv.po (lang sv)
     [exec] Reading file po/uk.po (lang uk)
     [exec] Reading file po/vi.po (lang vi)
     [exec] Reading file po/zh_CN.po (lang zh_CN)
     [exec] Reading file po/zh_TW.po (lang zh_TW)
     [exec] Created file ../plugins/Create_grid_of_ways/data/ast.lang        : Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/be.lang         : Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/bg.lang         : Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/ca.lang         : Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/ca-valencia.lang: Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/cs.lang         : Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/da.lang         : Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/de.lang         : Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/el.lang         : Added     2 strings out of     7 ( 28.5%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/en_AU.lang      : Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/en_GB.lang      : Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/es.lang         : Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/et.lang         : Added     2 strings out of     7 ( 28.5%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/fi.lang         : Added     1 strings out of     7 ( 14.2%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/fr.lang         : Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/gl.lang         : Added     3 strings out of     7 ( 42.8%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/hu.lang         : Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/id.lang         : Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/it.lang         : Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/ja.lang         : Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/km.lang         : Added     2 strings out of     7 ( 28.5%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/ko.lang         : Added     1 strings out of     7 ( 14.2%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/lt.lang         : Added     3 strings out of     7 ( 42.8%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/nb.lang         : Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/nl.lang         : Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/pl.lang         : Added     4 strings out of     7 ( 57.1%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/pt.lang         : Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/pt_BR.lang      : Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/ru.lang         : Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/sk.lang         : Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/sv.lang         : Added     6 strings out of     7 ( 85.7%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/uk.lang         : Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/vi.lang         : Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/zh_CN.lang      : Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/zh_TW.lang      : Added     7 strings out of     7 (100.0%).
     [exec] Created file ../plugins/Create_grid_of_ways/data/en.lang         : Added     7 strings out of     7 (100.0%).
   [delete] Deleting directory /var/lib/jenkins/jobs/JOSM-i18n/workspace/i18n/build/plugin_Create_grid_of_ways

Here also, plugintrans should be skipped but isn't. Also why does Mapillary change to Create_grid_of_ways in the middle of the task?

comment:11 Changed 18 months ago by Don-vip

Sometimes Jenkins and SVN are confused with the svn:externals => Github links and produce stale data. I'll clean the workspace and restart the job.

comment:12 Changed 18 months ago by floscher

Can you wait a sec, I'm currently in the process of making the pt_assistant plugin an svn:external ;-) …

comment:13 Changed 18 months ago by Don-vip

Sure :) I have no access to server right now anyway.

comment:14 Changed 18 months ago by stoecker

P.S. Thanks for the fix. I had a look at it before making the ticket, but I consider build.xml syntax brain damaging and hoped you'd be faster.

comment:16 Changed 18 months ago by floscher

Seems to be at least a bit better now (the "OLD" directories are gone), but I'm still seriously confused to why inside a single plugintrans target multiple plugin names occur:

plugintrans:
    [mkdir] Created dir: /var/lib/jenkins/jobs/JOSM-i18n/workspace/i18n/build/plugin_Mapillary
     [exec] Created file ../plugins/DirectUpload/data/pt_BR.lang      : Added    28 strings out of    29 ( 96.5%).
     [exec] Created file ../plugins/DirectUpload/data/ru.lang         : Added    29 strings out of    29 (100.0%).
     [exec] Created file ../plugins/DirectUpload/data/sk.lang         : Added    29 strings out of    29 (100.0%).
     [exec] Created file ../plugins/DirectUpload/data/sv.lang         : Added    20 strings out of    29 ( 68.9%).
     [exec] Created file ../plugins/DirectUpload/data/uk.lang         : Added    29 strings out of    29 (100.0%).
     [exec] Created file ../plugins/DirectUpload/data/vi.lang         : Added    28 strings out of    29 ( 96.5%).
     [exec] Created file ../plugins/DirectUpload/data/zh_CN.lang      : Added    28 strings out of    29 ( 96.5%).
     [exec] Created file ../plugins/DirectUpload/data/zh_TW.lang      : Added    28 strings out of    29 ( 96.5%).
     [exec] Created file ../plugins/DirectUpload/data/en.lang         : Added    29 strings out of    29 (100.0%).
   [delete] Deleting directory /var/lib/jenkins/jobs/JOSM-i18n/workspace/i18n/build/plugin_DirectUpload
[gettext-extract] ../plugins/Mapillary

At least for the Mapillary and wikipedia plugins artifacts are no longer produced, but the ones for geojson are still there. So it appears to be not fully solved yet.

comment:18 Changed 18 months ago by floscher

Thanks, I already see one problem with it, will try to fix it.

comment:19 Changed 18 months ago by floscher

I addressed the issue in [o34262].

The problem was that the $path variable in plugintrans is potentially an absolute path and when I concatenated it with another path, I got something like /var/lib/jenkins/jobs/JOSM-i18n/workspace/plugins/var/lib/jenkins/jobs/JOSM-i18n/workspace/plugins/ElevationProfile/.tx/config and that file obviously always doesn't exist.

I would attribute the other issue with changing plugin names to asynchronous execution, which I guess mixed up the logging output. The separate plugintrans targets seem to execute in parallel, so the log output is not properly grouped together for each target, but the outputs of the targets get mixed together.
So hopefully this only looked like an issue, but isn't actually one.

I'm probably spoiled by Gradle, which neatly groups together the logging outputs of each task in the console or when writing logging files, even if the tasks ran in parallel.

Last edited 18 months ago by floscher (previous) (diff)

comment:20 Changed 18 months ago by michael2402

Cc: michael2402 added

Modify Ticket

Change Properties
Set your email in Preferences
Action
as new The owner will remain team.
as The resolution will be set.
to The owner will be changed from team to the specified user.
The owner will change to stoecker
as duplicate The resolution will be set to duplicate.The specified ticket will be cross-referenced with this ticket
The owner will be changed from team to anonymous.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.