Modify

Opened 2 months ago

Closed 8 weeks ago

#17173 closed enhancement (fixed)

[Patch] Detect preset wiki languages automatically

Reported by: simon04 Owned by: simon04
Priority: normal Milestone: 19.01
Component: Internal preset Version:
Keywords: Cc: Don-vip, michael2402, stoecker, naoliv, Simon, Poole

Description (last modified by simon04)

Instead of maintaining various OSM wiki languages in defaultpresets.xml, maintaining presets_lang.pl, running presets_lang.pl from time to time, still applying manual changes (cf. #17171), let JOSM detect the wiki language automatically. This features has been implemented long ago in HelpAction.displayHelp: it opens https://wiki.openstreetmap.org/wiki/[lang]:[page] if it exists and falls back to https://wiki.openstreetmap.org/wiki/[page].

So in a nutshell …

  • data/defaultpresets.xml

    diff --git a/data/defaultpresets.xml b/data/defaultpresets.xml
    index b88f3d9d4..0ec763ca8 100644
    a b  
    326326        <reference ref="voltage_frequency" />
    327327    </chunk>
    328328    <chunk id="power_generator">
    329         <link href="https://wiki.openstreetmap.org/wiki/Tag:power=generator"
    330                   cs.href="https://wiki.openstreetmap.org/wiki/Cs:Tag:power=generator"
    331                   de.href="https://wiki.openstreetmap.org/wiki/DE:Tag:power=generator"
    332                   es.href="https://wiki.openstreetmap.org/wiki/ES:Tag:power=generator"
    333                   fr.href="https://wiki.openstreetmap.org/wiki/FR:Tag:power=generator"
    334                   it.href="https://wiki.openstreetmap.org/wiki/IT:Tag:power=generator"
    335                   ja.href="https://wiki.openstreetmap.org/wiki/JA:Tag:power=generator"
    336                   pl.href="https://wiki.openstreetmap.org/wiki/Pl:Tag:power=generator"
    337                   pt.href="https://wiki.openstreetmap.org/wiki/Pt:Tag:power=generator"
    338                   ru.href="https://wiki.openstreetmap.org/wiki/RU:Tag:power=generator" />
     329        <link wiki="Tag:power=generator" />
    339330        <space />
    340331        <key key="power" value="generator" />
    341332        <reference ref="name_ref_operator" />

In total: 719 insertions(+), 5651 deletions(-)

Relates to this discussion at josm-dev@: https://lists.openstreetmap.org/pipermail/josm-dev/2018-November/008171.html (Add templated version of wiki href to link element in presets)

Attachments (1)

17173.patch.gz (64.0 KB) - added by simon04 2 months ago.

Download all attachments as: .zip

Change History (30)

Changed 2 months ago by simon04

Attachment: 17173.patch.gz added

comment:1 Changed 2 months ago by simon04

Cc: stoecker naoliv added

comment:2 Changed 2 months ago by stoecker

It's not so easy. You overlooked all these cases where the links don't have the same name.

comment:3 Changed 2 months ago by naoliv

For pt_BR it will be necessary to consider the language as pt, since pt_BR was unified into pt (ie, people using JOSM in pt or pt_BR should open the same Portuguese wiki pages).
With the patch (actually after [14617]), using pt_BR only opens the English pages.

comment:4 in reply to:  3 ; Changed 2 months ago by simon04

Replying to stoecker:

It's not so easy. You overlooked all these cases where the links don't have the same name.

Please name one. I only stumbled over https://wiki.openstreetmap.org/wiki/Link_(highway vs. https://wiki.openstreetmap.org/wiki/Cs:Highway_link, but the former redirects to https://wiki.openstreetmap.org/wiki/Highway_link. IMO we could accept 0.1% non-detected language links in favour of simplicity / elegance / reduced code / reduced maintenance burden.

Replying to naoliv:

For pt_BR it will be necessary to consider the language as pt, since pt_BR was unified into pt (ie, people using JOSM in pt or pt_BR should open the same Portuguese wiki pages).
With the patch (actually after [14617]), using pt_BR only opens the English pages.

Good to know… We should reflect the merge in LanguageInfo. The merge has taken place on the OSM wiki, the JOSM wiki, the JOSM i18n?

comment:5 in reply to:  4 Changed 2 months ago by naoliv

Replying to simon04:

Good to know… We should reflect the merge in LanguageInfo. The merge has taken place on the OSM wiki, the JOSM wiki, the JOSM i18n?

Only OSM wiki was unified. The rest are still (and I guess will stay) different.

comment:6 in reply to:  4 Changed 2 months ago by stoecker

Replying to simon04:

Replying to stoecker:

It's not so easy. You overlooked all these cases where the links don't have the same name.

Please name one. I only stumbled over https://wiki.openstreetmap.org/wiki/Link_(highway vs. https://wiki.openstreetmap.org/wiki/Cs:Highway_link, but the former redirects to https://wiki.openstreetmap.org/wiki/Highway_link. IMO we could accept 0.1% non-detected language links in favour of simplicity / elegance / reduced code / reduced maintenance burden.

Hmm, seems the wiki situation changed. E.g. Link_(highway) or Highway_link actually has references to "Pt:Tag:highway=motorway_link", but this is missing totally now. Also the wiki does not refernce redirects in the language pages. Maybe they broke the wiki in that instance? That would mean that your approach would actually be better, as the links then work again.

Can you please extract the code changes and attach them separately for review?

Replying to naoliv:

For pt_BR it will be necessary to consider the language as pt, since pt_BR was unified into pt (ie, people using JOSM in pt or pt_BR should open the same Portuguese wiki pages).
With the patch (actually after [14617]), using pt_BR only opens the English pages.

Good to know… We should reflect the merge in LanguageInfo. The merge has taken place on the OSM wiki, the JOSM wiki, the JOSM i18n?

JOSM wiki and JOSM i18n is not affected. That would only be the case, when pt_BR translation gets dropped. But we need to change JOSM code, that the OSM wiki prefix for pt_BR is "Pt:".

Note that JOSM wiki uses language code derived rules, whereas the prefix for OSM wiki seems to be manually decided. They are not identical.

comment:7 Changed 2 months ago by stoecker

Ticket #17171 has been marked as a duplicate of this ticket.

comment:8 Changed 2 months ago by stoecker

If we go this way we need to:

  • test that the added English links actually refer to non-redirect pages
  • have a way to test redirects later (i.e. modify the wiki scraper)

If not:

  • update the wiki scraper to handle redirects correct

comment:9 Changed 2 months ago by simon04

Cc: Simon Poole added
Description: modified (diff)

comment:10 Changed 2 months ago by simon04

Resolution: fixed
Status: assignedclosed

In 14646/josm:

fix #17173 - Detect preset wiki languages automatically

comment:11 Changed 2 months ago by simon04

In 14647/josm:

see #17173 - Use correct language prefixes for OSM wiki

Reference: https://wiki.openstreetmap.org/wiki/Template:Languages

comment:12 Changed 2 months ago by Don-vip

comment:13 Changed 2 months ago by simon04

In 14649/josm:

see #17173 - Fix LinkTest

comment:14 Changed 2 months ago by stoecker

Resolution: fixed
Status: closedreopened

You missed to address the comment:8, the wiki documentation is missing and this

else if (type == LocaleType.OSM_WIKI) {
  return locale.getLanguage() + ':';
}

causes lowercase results which produce a redirect always. It should not be default to produce useless redirects for each wiki call. We should use a correct uppercase/lowercase table instead.

Version 0, edited 2 months ago by stoecker (next)

comment:15 in reply to:  14 ; Changed 2 months ago by Don-vip

Replying to stoecker:

It should not be default to produce useless redirects for each wiki call.

We should add an integration test to check this.

comment:16 in reply to:  15 ; Changed 2 months ago by stoecker

Replying to Don-vip:

Replying to stoecker:

It should not be default to produce useless redirects for each wiki call.

We should add an integration test to check this.

Sidenote: We must be careful with remote integration tests. E.g. We are currently accessing all our map services 4 times a day with our JOSM user agent. I'm not sure this is a good idea. Probably that should be reduced to once a day and also use another user agent.

comment:17 in reply to:  16 ; Changed 2 months ago by Don-vip

Replying to stoecker:

I'm not sure this is a good idea.

Why?

comment:18 in reply to:  17 Changed 2 months ago by stoecker

Replying to Don-vip:

Replying to stoecker:

I'm not sure this is a good idea.

Why?

Because as administrator I tend to block user agents and servers which constantly access the same stuff. Algorithmic data access is usually not caused by humans and thus unwanted data access. Other operators handle that similar.

Actually for automatic access we'd need to follow robots.txt protocols and thus be restricted.

If some operator decides to block JOSM user agent because of this that would be a VERY unwanted effect.

comment:19 Changed 2 months ago by Don-vip

OK.

The i18n build is broken:

     [exec] /* unparsed line 7481             <link wiki="Relations/Proposed/Bridges_and_Tunnels" /> */

BUILD FAILED
/var/lib/jenkins/jobs/JOSM-i18n/workspace/i18n/build.xml:29: exec returned: 20

comment:20 in reply to:  19 ; Changed 2 months ago by simon04

Replying to stoecker:

You missed to address the comment:8

I checked my replacements of href with wiki by hand: Except for Highway_link (see above), all localized hrefs only differed by their language prefix. Thus, I consider the probability of localized pages redirecting anywhere differnt very low.

the wiki documentation is missing

Done: TaggingPresets

causes lowercase results which produce a redirect always. It should not be default to produce useless redirects for each wiki call. We should use a correct uppercase/lowercase table instead.

This is (a) not a problem and (b) not new: (a) https://wiki.openstreetmap.org/wiki/de:key:name automatically resolves to https://wiki.openstreetmap.org/wiki/DE:Key:name not involving a redirect. (b) from org.openstreetmap.josm.gui.dialogs.properties.HelpAction we gladly opened De: beforehand.

Replying to Don-vip:

The i18n build is broken:

Thanks for noticing! Fixed in [o34809].

comment:21 in reply to:  20 Changed 2 months ago by stoecker

Replying to simon04:

Replying to stoecker:

You missed to address the comment:8

I checked my replacements of href with wiki by hand: Except for Highway_link (see above), all localized hrefs only differed by their language prefix. Thus, I consider the probability of localized pages redirecting anywhere differnt very low.

As said in #17171 the script probably didn't handle redirects well. Thus I assume our links contain a lot of redirects or there wouldn't have been so many pt_BR remaining for manual fixing. This must be checked now (and later automatically) and cleaned. Redirects should only remain in seldom cases.

causes lowercase results which produce a redirect always. It should not be default to produce useless redirects for each wiki call. We should use a correct uppercase/lowercase table instead.

This is (a) not a problem and

Well, as said it is bad default behaviour.

(a) https://wiki.openstreetmap.org/wiki/de:key:name automatically resolves to https://wiki.openstreetmap.org/wiki/DE:Key:name not involving a redirect.

Check the data transmission yourself. It involves always a 301 redirect. If the URL line changes in a browser there is no way except a redirect to reach that goal (or JavaScript quirks).

(b) from org.openstreetmap.josm.gui.dialogs.properties.HelpAction we gladly opened De: beforehand.

Which tells me, that the existing code was not properly tested. Still that's no reason to leave it that way.

comment:22 Changed 2 months ago by stoecker

In 14660/josm:

see #17173 - proper prefixes for OSM wiki

comment:23 in reply to:  22 ; Changed 2 months ago by stoecker

Replying to stoecker:

In 14660/josm:

see #17173 - proper prefixes for OSM wiki

I don't have time to test each case ATM. Hope this is ok as is. 😇

comment:24 in reply to:  23 Changed 2 months ago by simon04

Replying to stoecker:

I don't have time to test each case ATM. Hope this is ok as is. 😇

The unit tests need to be adapted: https://josm.openstreetmap.de/jenkins/job/JOSM/4936/jdk=JDK8/testReport/org.openstreetmap.josm.tools/LanguageInfoTest/getWikiLanguagePrefix/

comment:25 Changed 2 months ago by stoecker

In 14664/josm:

fix test cases, see #17173

comment:26 Changed 2 months ago by anonymous

The information=map preset points to the Key:information page instead of the Tag:information=map page on the wiki.

comment:27 Changed 2 months ago by Klumbumbus

In 14703/josm:

see #17173 - improve preset wiki links

comment:28 in reply to:  15 Changed 8 weeks ago by simon04

I opened #17240 for the outstanding task.

comment:29 in reply to:  15 Changed 8 weeks ago by simon04

Resolution: fixed
Status: reopenedclosed

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain simon04.
as The resolution will be set.
The resolution will be deleted.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.